2024-09-13T14:02:10.458 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-13T14:02:10.487 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T14:02:10.595 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903555 branch: wip-aclamk-cbt-combined description: upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} email: akupczyk@redhat.com first-half-sequence: - cephadm.shell: env: - sha1 mon.a: - ceph config set mgr mgr/cephadm/daemon_cache_timeout 60 - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps - echo wait for minority of mons to upgrade - while ! ceph mon versions | grep $sha1 && ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error; do sleep 2 ; done - ceph orch ps - ceph orch upgrade pause - sleep 60 - ceph orch upgrade resume - echo wait for majority of mons to upgrade - 'while ! ceph mon versions | grep $sha1 | egrep '': [23]'' && ceph orch upgrade status | jq ''.in_progress'' | grep true && ! ceph orch upgrade status | jq ''.message'' | grep Error; do sleep 2 ; done' - ceph orch ps - ceph orch upgrade pause - sleep 60 - ceph orch upgrade resume - echo wait for all mons to upgrade - 'while ! ceph mon versions | grep $sha1 | grep '': 3'' && ceph orch upgrade status | jq ''.in_progress'' | grep true && ! ceph orch upgrade status | jq ''.message'' | grep Error; do sleep 2 ; done' - ceph orch ps - ceph orch upgrade pause - sleep 60 - ceph orch upgrade resume - echo wait for half of osds to upgrade - 'while ! ceph osd versions | grep $sha1 | egrep '': [45678]'' && ceph orch upgrade status | jq ''.in_progress'' | grep true && ! ceph orch upgrade status | jq ''.message'' | grep Error; do sleep 2 ; done' - ceph orch upgrade pause - ceph orch ps - ceph orch ps - ceph versions first-half-tasks: - full_sequential: - rados: clients: - client.0 objects: 50 op_weights: delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 write_append_excl: false - print: '**** done rados/snaps-few-objects 5-workload' first_in_suite: false job_id: '7903555' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'randomized correctness test for rados operations on a replicated pool with snapshot operations ' - desc: 'run randomized correctness test for rados operations generate write load with rados bench ' - desc: 'run basic cls tests for rbd ' - desc: 'run basic import/export cli tests for rbd ' - desc: 'librbd C and C++ api tests ' - desc: 'randomized correctness test for rados operations on a replicated pool, using only reads, writes, and deletes ' - desc: 'randomized correctness test for rados operations on a replicated pool with snapshot operations ' - desc: 'run randomized correctness test for rados operations generate write load with rados bench ' name: akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-aclamk-cbt-combined ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true create_rbd_pool: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - OSD_DOWN - mons down - mon down - MON_DOWN - out of quorum - PG_DEGRADED - Reduced data availability - Degraded data redundancy - OSDMAP_FLAGS - PG_AVAILABILITY sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-aclamk-cbt-combined sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 owner: scheduled_akupczyk@teuthology priority: 160 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b second-half-sequence: sequential: - cephadm.shell: env: - sha1 mon.a: - ceph orch upgrade resume - sleep 60 - echo wait for upgrade to complete - while ceph orch upgrade status | jq '.in_progress' | grep true ; do ceph orch ps ; ceph versions ; sleep 30 ; done - echo upgrade complete - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 second-half-tasks: - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - print: '**** done end radosbench.yaml' seed: 6545 sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 sleep_before_teardown: 0 stress-tasks: - thrashosds: aggressive_pg_num_changes: false chance_force_recovery: 0 chance_pgnum_grow: 1 chance_pgpnum_fix: 1 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 chance_thrash_pg_upmap_items: 0 disable_objectstore_tool_tests: true timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - print: '**** done end radosbench.yaml' - workunit: branch: quincy clients: client.0: - cls/test_cls_rbd.sh - print: '**** done cls/test_cls_rbd.sh 5-workload' - workunit: branch: quincy clients: client.0: - rbd/import_export.sh env: RBD_CREATE_ARGS: --new-format - print: '**** done rbd/import_export.sh 5-workload' - workunit: branch: quincy clients: client.0: - rbd/test_librbd.sh env: RBD_FEATURES: '61' - print: '**** done rbd/test_librbd.sh 7-workload' - full_sequential: - rados: clients: - client.0 objects: 500 op_weights: delete: 10 read: 45 write: 45 ops: 4000 write_append_excl: false - print: '**** done rados/readwrite 5-workload' - full_sequential: - rados: clients: - client.0 objects: 50 op_weights: delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 write_append_excl: false - print: '**** done rados/snaps-few-objects 5-workload' subset: 682/13280 suite: upgrade suite_branch: wip-aclamk-cbt-combined suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 targets: smithi151.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK9rqUDu3k0YsdCtAcvsciRIrDzdBpjzc/ti7qw62jtxJjaR1LukRnfhfHMMXfgkXmKTmkGNmEZ2W6Py+WqY9uA= smithi184.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIqvVufGunIesC1X3QZJ+YV5YR1RTLTU1wj+Q01JIJE4Y7x5lL0i0Z3WReNjlesqsKfdZ9shIlZtQbJrYbiratU= tasks: - pexec: all: - sudo dnf remove nvme-cli -y - sudo dnf install runc nvmetcli nvme-cli -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - install: branch: quincy exclude_packages: - ceph-volume - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy - cephadm.shell: mon.a: - ceph fs volume create foo - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph.healthy: null - print: '**** upgrading first half of cluster, with stress ****' - parallel: - first-half-tasks - first-half-sequence - print: '**** done upgrading first half of cluster ****' - ceph.healthy: null - print: '**** applying stress + thrashing to mixed-version cluster ****' - parallel: - stress-tasks - ceph.healthy: null - print: '**** finishing upgrade ****' - parallel: - second-half-tasks - second-half-sequence - ceph.healthy: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-13_10:00:18 tube: smithi user: akupczyk verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-13T14:02:10.595 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa; will attempt to use it 2024-09-13T14:02:10.596 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks 2024-09-13T14:02:10.596 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-13T14:02:10.598 INFO:teuthology.task.internal:Checking packages... 2024-09-13T14:02:10.618 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd58f73069a8157a9ace12a0247ee3e8497707ea9' 2024-09-13T14:02:10.619 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T14:02:10.619 INFO:teuthology.packaging:ref: None 2024-09-13T14:02:10.619 INFO:teuthology.packaging:tag: None 2024-09-13T14:02:10.619 INFO:teuthology.packaging:branch: wip-aclamk-cbt-combined 2024-09-13T14:02:10.619 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:02:10.619 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-aclamk-cbt-combined 2024-09-13T14:02:10.772 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4697.gd58f7306 2024-09-13T14:02:10.773 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-13T14:02:10.780 INFO:teuthology.task.internal:no buildpackages task found 2024-09-13T14:02:10.780 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-13T14:02:10.787 INFO:teuthology.task.internal:Saving configuration 2024-09-13T14:02:10.815 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-13T14:02:10.822 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-13T14:02:10.856 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903555', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 13:57:06.000500', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK9rqUDu3k0YsdCtAcvsciRIrDzdBpjzc/ti7qw62jtxJjaR1LukRnfhfHMMXfgkXmKTmkGNmEZ2W6Py+WqY9uA='} 2024-09-13T14:02:10.888 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi184.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903555', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 13:57:06.002200', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIqvVufGunIesC1X3QZJ+YV5YR1RTLTU1wj+Q01JIJE4Y7x5lL0i0Z3WReNjlesqsKfdZ9shIlZtQbJrYbiratU='} 2024-09-13T14:02:10.888 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-13T14:02:10.895 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-09-13T14:02:10.895 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-09-13T14:02:10.895 INFO:teuthology.run_tasks:Running task console_log... 2024-09-13T14:02:10.973 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f561f3fbbe0>, signals=[15]) 2024-09-13T14:02:10.973 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-13T14:02:10.979 INFO:teuthology.task.internal:Opening connections... 2024-09-13T14:02:10.979 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi151.front.sepia.ceph.com 2024-09-13T14:02:10.981 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:02:11.073 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi184.front.sepia.ceph.com 2024-09-13T14:02:11.074 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:02:11.159 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-13T14:02:11.166 DEBUG:teuthology.orchestra.run.smithi151:> uname -m 2024-09-13T14:02:11.184 INFO:teuthology.orchestra.run.smithi151.stdout:x86_64 2024-09-13T14:02:11.184 DEBUG:teuthology.orchestra.run.smithi151:> cat /etc/os-release 2024-09-13T14:02:11.242 INFO:teuthology.orchestra.run.smithi151.stdout:NAME="CentOS Stream" 2024-09-13T14:02:11.243 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION="9" 2024-09-13T14:02:11.243 INFO:teuthology.orchestra.run.smithi151.stdout:ID="centos" 2024-09-13T14:02:11.243 INFO:teuthology.orchestra.run.smithi151.stdout:ID_LIKE="rhel fedora" 2024-09-13T14:02:11.243 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION_ID="9" 2024-09-13T14:02:11.243 INFO:teuthology.orchestra.run.smithi151.stdout:PLATFORM_ID="platform:el9" 2024-09-13T14:02:11.243 INFO:teuthology.orchestra.run.smithi151.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T14:02:11.243 INFO:teuthology.orchestra.run.smithi151.stdout:ANSI_COLOR="0;31" 2024-09-13T14:02:11.243 INFO:teuthology.orchestra.run.smithi151.stdout:LOGO="fedora-logo-icon" 2024-09-13T14:02:11.243 INFO:teuthology.orchestra.run.smithi151.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T14:02:11.243 INFO:teuthology.orchestra.run.smithi151.stdout:HOME_URL="https://centos.org/" 2024-09-13T14:02:11.243 INFO:teuthology.orchestra.run.smithi151.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T14:02:11.243 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T14:02:11.243 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T14:02:11.244 INFO:teuthology.lock.ops:Updating smithi151.front.sepia.ceph.com on lock server 2024-09-13T14:02:11.277 DEBUG:teuthology.orchestra.run.smithi184:> uname -m 2024-09-13T14:02:11.296 INFO:teuthology.orchestra.run.smithi184.stdout:x86_64 2024-09-13T14:02:11.297 DEBUG:teuthology.orchestra.run.smithi184:> cat /etc/os-release 2024-09-13T14:02:11.355 INFO:teuthology.orchestra.run.smithi184.stdout:NAME="CentOS Stream" 2024-09-13T14:02:11.355 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION="9" 2024-09-13T14:02:11.355 INFO:teuthology.orchestra.run.smithi184.stdout:ID="centos" 2024-09-13T14:02:11.356 INFO:teuthology.orchestra.run.smithi184.stdout:ID_LIKE="rhel fedora" 2024-09-13T14:02:11.356 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_ID="9" 2024-09-13T14:02:11.356 INFO:teuthology.orchestra.run.smithi184.stdout:PLATFORM_ID="platform:el9" 2024-09-13T14:02:11.356 INFO:teuthology.orchestra.run.smithi184.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T14:02:11.356 INFO:teuthology.orchestra.run.smithi184.stdout:ANSI_COLOR="0;31" 2024-09-13T14:02:11.356 INFO:teuthology.orchestra.run.smithi184.stdout:LOGO="fedora-logo-icon" 2024-09-13T14:02:11.356 INFO:teuthology.orchestra.run.smithi184.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T14:02:11.356 INFO:teuthology.orchestra.run.smithi184.stdout:HOME_URL="https://centos.org/" 2024-09-13T14:02:11.356 INFO:teuthology.orchestra.run.smithi184.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T14:02:11.356 INFO:teuthology.orchestra.run.smithi184.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T14:02:11.356 INFO:teuthology.orchestra.run.smithi184.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T14:02:11.357 INFO:teuthology.lock.ops:Updating smithi184.front.sepia.ceph.com on lock server 2024-09-13T14:02:11.386 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-13T14:02:11.394 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-13T14:02:11.399 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-13T14:02:11.399 DEBUG:teuthology.orchestra.run.smithi151:> test '!' -e /home/ubuntu/cephtest 2024-09-13T14:02:11.401 DEBUG:teuthology.orchestra.run.smithi184:> test '!' -e /home/ubuntu/cephtest 2024-09-13T14:02:11.418 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-13T14:02:11.424 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-13T14:02:11.424 DEBUG:teuthology.orchestra.run.smithi151:> test -z $(ls -A /var/lib/ceph) 2024-09-13T14:02:11.459 DEBUG:teuthology.orchestra.run.smithi184:> test -z $(ls -A /var/lib/ceph) 2024-09-13T14:02:11.609 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-13T14:02:11.674 INFO:teuthology.run_tasks:Running task kernel... 2024-09-13T14:02:11.689 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-13T14:02:11.689 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-13T14:02:11.690 DEBUG:teuthology.orchestra.run.smithi151:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T14:02:11.690 DEBUG:teuthology.orchestra.run.smithi184:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T14:02:11.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:02:11.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:02:11.708 DEBUG:teuthology.orchestra.run.smithi151:> uname -r 2024-09-13T14:02:11.708 DEBUG:teuthology.orchestra.run.smithi184:> uname -r 2024-09-13T14:02:11.766 INFO:teuthology.orchestra.run.smithi151.stdout:5.14.0-503.el9.x86_64 2024-09-13T14:02:11.766 INFO:teuthology.task.kernel:Running kernel on smithi151: 5.14.0-503.el9.x86_64 2024-09-13T14:02:11.766 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum install -y kernel 2024-09-13T14:02:11.767 INFO:teuthology.orchestra.run.smithi184.stdout:5.14.0-503.el9.x86_64 2024-09-13T14:02:11.768 INFO:teuthology.task.kernel:Running kernel on smithi184: 5.14.0-503.el9.x86_64 2024-09-13T14:02:11.768 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum install -y kernel 2024-09-13T14:02:14.088 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.2 MB 00:00 2024-09-13T14:02:15.813 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - AppStream 34 MB/s | 20 MB 00:00 2024-09-13T14:02:16.516 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - BaseOS 2.8 MB/s | 8.2 MB 00:02 2024-09-13T14:02:21.705 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - CRB 3.2 MB/s | 6.5 MB 00:02 2024-09-13T14:02:23.448 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 19 kB 00:00 2024-09-13T14:02:25.118 INFO:teuthology.orchestra.run.smithi151.stdout:Extra Packages for Enterprise Linux 15 MB/s | 23 MB 00:01 2024-09-13T14:02:25.508 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - AppStream 2.6 MB/s | 20 MB 00:07 2024-09-13T14:02:30.184 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - CRB 7.4 MB/s | 6.5 MB 00:00 2024-09-13T14:02:30.599 INFO:teuthology.orchestra.run.smithi151.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-13T14:02:31.826 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - Extras packages 68 kB/s | 19 kB 00:00 2024-09-13T14:02:32.597 INFO:teuthology.orchestra.run.smithi151.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-13T14:02:32.597 INFO:teuthology.orchestra.run.smithi151.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-13T14:02:32.708 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-13T14:02:32.713 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:02:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-09-13T14:02:32.713 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:02:32.713 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-09-13T14:02:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-13T14:02:32.713 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-09-13T14:02:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-13T14:02:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-13T14:02:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-13T14:02:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:02:32.714 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-09-13T14:02:32.714 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:02:32.714 INFO:teuthology.orchestra.run.smithi151.stdout:Install 4 Packages 2024-09-13T14:02:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:02:32.715 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 79 M 2024-09-13T14:02:32.715 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 126 M 2024-09-13T14:02:32.715 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-09-13T14:02:32.975 INFO:teuthology.orchestra.run.smithi151.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 212 kB/s | 23 kB 00:00 2024-09-13T14:02:33.722 INFO:teuthology.orchestra.run.smithi184.stdout:Extra Packages for Enterprise Linux 13 MB/s | 23 MB 00:01 2024-09-13T14:02:35.734 INFO:teuthology.orchestra.run.smithi151.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 5.5 MB/s | 16 MB 00:02 2024-09-13T14:02:37.301 INFO:teuthology.orchestra.run.smithi151.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 6.6 MB/s | 29 MB 00:04 2024-09-13T14:02:37.968 INFO:teuthology.orchestra.run.smithi151.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 6.8 MB/s | 35 MB 00:05 2024-09-13T14:02:37.968 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:02:37.968 INFO:teuthology.orchestra.run.smithi151.stdout:Total 15 MB/s | 79 MB 00:05 2024-09-13T14:02:38.649 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-09-13T14:02:38.998 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-09-13T14:02:38.998 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-09-13T14:02:39.169 INFO:teuthology.orchestra.run.smithi184.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-13T14:02:40.038 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-09-13T14:02:40.038 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-09-13T14:02:41.109 INFO:teuthology.orchestra.run.smithi184.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-13T14:02:41.109 INFO:teuthology.orchestra.run.smithi184.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-13T14:02:41.216 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-09-13T14:02:41.231 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:02:41.232 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-09-13T14:02:41.232 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:02:41.232 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-09-13T14:02:41.232 INFO:teuthology.orchestra.run.smithi184.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-13T14:02:41.232 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-09-13T14:02:41.232 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-13T14:02:41.232 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-13T14:02:41.232 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-13T14:02:41.232 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:02:41.232 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-09-13T14:02:41.233 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:02:41.233 INFO:teuthology.orchestra.run.smithi184.stdout:Install 4 Packages 2024-09-13T14:02:41.233 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:02:41.233 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 79 M 2024-09-13T14:02:41.233 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 126 M 2024-09-13T14:02:41.233 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-09-13T14:02:41.479 INFO:teuthology.orchestra.run.smithi184.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 155 kB/s | 23 kB 00:00 2024-09-13T14:02:41.821 INFO:teuthology.orchestra.run.smithi184.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-09-13T14:02:42.263 INFO:teuthology.orchestra.run.smithi184.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 37 MB/s | 35 MB 00:00 2024-09-13T14:02:42.492 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-09-13T14:02:42.854 INFO:teuthology.orchestra.run.smithi184.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 21 MB/s | 29 MB 00:01 2024-09-13T14:02:42.855 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:02:42.855 INFO:teuthology.orchestra.run.smithi184.stdout:Total 49 MB/s | 79 MB 00:01 2024-09-13T14:02:43.302 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-13T14:02:43.499 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T14:02:43.541 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-09-13T14:02:43.895 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-09-13T14:02:43.895 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-09-13T14:02:44.853 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-09-13T14:02:44.854 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-09-13T14:02:46.322 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T14:02:46.459 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T14:02:47.283 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-09-13T14:02:47.956 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-13T14:02:48.041 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T14:02:50.836 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T14:02:50.969 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T14:02:51.302 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T14:02:51.442 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:02:55.808 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T14:02:55.919 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:02:56.307 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:03:00.802 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:03:34.569 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:03:34.609 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:03:34.987 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:03:34.987 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-13T14:03:34.988 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T14:03:34.988 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T14:03:35.862 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:03:35.862 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:03:35.862 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-09-13T14:03:35.862 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-13T14:03:35.862 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-13T14:03:35.862 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-13T14:03:35.862 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-13T14:03:35.862 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:03:35.862 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-13T14:03:36.239 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T14:03:36.325 INFO:teuthology.orchestra.run.smithi151.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T14:03:36.325 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T14:03:36.325 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-13T14:03:36.325 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T14:03:36.325 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T14:03:36.326 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-13T14:03:36.326 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi151.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T14:03:36.326 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T14:03:36.385 INFO:teuthology.orchestra.run.smithi151.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T14:03:36.385 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T14:03:36.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo rpm -qi grub2-tools 2024-09-13T14:03:36.420 INFO:teuthology.orchestra.run.smithi151.stdout:Name : grub2-tools 2024-09-13T14:03:36.420 INFO:teuthology.orchestra.run.smithi151.stdout:Epoch : 1 2024-09-13T14:03:36.420 INFO:teuthology.orchestra.run.smithi151.stdout:Version : 2.06 2024-09-13T14:03:36.420 INFO:teuthology.orchestra.run.smithi151.stdout:Release : 82.el9 2024-09-13T14:03:36.420 INFO:teuthology.orchestra.run.smithi151.stdout:Architecture: x86_64 2024-09-13T14:03:36.421 INFO:teuthology.orchestra.run.smithi151.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T14:03:36.421 INFO:teuthology.orchestra.run.smithi151.stdout:Group : Unspecified 2024-09-13T14:03:36.421 INFO:teuthology.orchestra.run.smithi151.stdout:Size : 8274670 2024-09-13T14:03:36.421 INFO:teuthology.orchestra.run.smithi151.stdout:License : GPLv3+ 2024-09-13T14:03:36.421 INFO:teuthology.orchestra.run.smithi151.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T14:03:36.421 INFO:teuthology.orchestra.run.smithi151.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T14:03:36.421 INFO:teuthology.orchestra.run.smithi151.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T14:03:36.421 INFO:teuthology.orchestra.run.smithi151.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T14:03:36.421 INFO:teuthology.orchestra.run.smithi151.stdout:Packager : builder@centos.org 2024-09-13T14:03:36.421 INFO:teuthology.orchestra.run.smithi151.stdout:Vendor : CentOS 2024-09-13T14:03:36.421 INFO:teuthology.orchestra.run.smithi151.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T14:03:36.421 INFO:teuthology.orchestra.run.smithi151.stdout:Summary : Support tools for GRUB. 2024-09-13T14:03:36.422 INFO:teuthology.orchestra.run.smithi151.stdout:Description : 2024-09-13T14:03:36.422 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:03:36.422 INFO:teuthology.orchestra.run.smithi151.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T14:03:36.422 INFO:teuthology.orchestra.run.smithi151.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T14:03:36.422 INFO:teuthology.orchestra.run.smithi151.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T14:03:36.422 INFO:teuthology.orchestra.run.smithi151.stdout:hardware devices. 2024-09-13T14:03:36.422 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:03:36.422 INFO:teuthology.orchestra.run.smithi151.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T14:03:36.423 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T14:03:36.423 INFO:teuthology.task.kernel:Updating grub on smithi151 to boot 5.14.0-505.el9.x86_64 2024-09-13T14:03:36.423 DEBUG:teuthology.orchestra.run.smithi151:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T14:03:37.143 INFO:teuthology.orchestra.run.smithi151.stderr:Generating grub configuration file ... 2024-09-13T14:03:38.456 INFO:teuthology.orchestra.run.smithi151.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T14:03:38.491 INFO:teuthology.orchestra.run.smithi151.stderr:done 2024-09-13T14:03:38.494 DEBUG:teuthology.orchestra.run.smithi151:> mktemp 2024-09-13T14:03:38.510 INFO:teuthology.orchestra.run.smithi151.stdout:/tmp/tmp.uL3VGHiIbj 2024-09-13T14:03:38.510 DEBUG:teuthology.orchestra.run.smithi151:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.uL3VGHiIbj 2024-09-13T14:03:38.577 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0666 /tmp/tmp.uL3VGHiIbj 2024-09-13T14:03:38.712 DEBUG:teuthology.orchestra.remote:smithi151:/tmp/tmp.uL3VGHiIbj is 6KB 2024-09-13T14:03:38.759 DEBUG:teuthology.orchestra.run.smithi151:> rm -fr /tmp/tmp.uL3VGHiIbj 2024-09-13T14:03:38.774 DEBUG:teuthology.orchestra.run.smithi151:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T14:03:38.837 INFO:teuthology.orchestra.run.smithi151.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T14:03:38.838 INFO:teuthology.orchestra.run.smithi151.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T14:03:38.838 INFO:teuthology.orchestra.run.smithi151.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T14:03:38.838 INFO:teuthology.orchestra.run.smithi151.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-13T14:03:38.839 DEBUG:teuthology.orchestra.run.smithi151:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-13T14:03:38.975 DEBUG:teuthology.orchestra.run.smithi151:> sudo shutdown -r now 2024-09-13T14:03:39.084 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:03:39.127 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:03:39.461 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:03:39.461 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-13T14:03:39.461 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T14:03:39.461 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T14:03:40.318 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:03:40.319 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:03:40.319 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-09-13T14:03:40.319 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-13T14:03:40.319 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-13T14:03:40.319 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-13T14:03:40.319 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-13T14:03:40.319 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:03:40.319 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-09-13T14:03:40.683 DEBUG:teuthology.orchestra.run.smithi184:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T14:03:40.756 INFO:teuthology.orchestra.run.smithi184.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T14:03:40.757 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T14:03:40.757 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-13T14:03:40.757 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T14:03:40.757 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T14:03:40.757 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-13T14:03:40.757 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi184.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T14:03:40.757 DEBUG:teuthology.orchestra.run.smithi184:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T14:03:40.816 INFO:teuthology.orchestra.run.smithi184.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T14:03:40.817 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T14:03:40.817 DEBUG:teuthology.orchestra.run.smithi184:> sudo rpm -qi grub2-tools 2024-09-13T14:03:40.858 INFO:teuthology.orchestra.run.smithi184.stdout:Name : grub2-tools 2024-09-13T14:03:40.858 INFO:teuthology.orchestra.run.smithi184.stdout:Epoch : 1 2024-09-13T14:03:40.859 INFO:teuthology.orchestra.run.smithi184.stdout:Version : 2.06 2024-09-13T14:03:40.859 INFO:teuthology.orchestra.run.smithi184.stdout:Release : 82.el9 2024-09-13T14:03:40.859 INFO:teuthology.orchestra.run.smithi184.stdout:Architecture: x86_64 2024-09-13T14:03:40.859 INFO:teuthology.orchestra.run.smithi184.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T14:03:40.859 INFO:teuthology.orchestra.run.smithi184.stdout:Group : Unspecified 2024-09-13T14:03:40.859 INFO:teuthology.orchestra.run.smithi184.stdout:Size : 8274670 2024-09-13T14:03:40.859 INFO:teuthology.orchestra.run.smithi184.stdout:License : GPLv3+ 2024-09-13T14:03:40.859 INFO:teuthology.orchestra.run.smithi184.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T14:03:40.859 INFO:teuthology.orchestra.run.smithi184.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T14:03:40.859 INFO:teuthology.orchestra.run.smithi184.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T14:03:40.860 INFO:teuthology.orchestra.run.smithi184.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T14:03:40.860 INFO:teuthology.orchestra.run.smithi184.stdout:Packager : builder@centos.org 2024-09-13T14:03:40.860 INFO:teuthology.orchestra.run.smithi184.stdout:Vendor : CentOS 2024-09-13T14:03:40.860 INFO:teuthology.orchestra.run.smithi184.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T14:03:40.860 INFO:teuthology.orchestra.run.smithi184.stdout:Summary : Support tools for GRUB. 2024-09-13T14:03:40.860 INFO:teuthology.orchestra.run.smithi184.stdout:Description : 2024-09-13T14:03:40.861 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:03:40.861 INFO:teuthology.orchestra.run.smithi184.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T14:03:40.861 INFO:teuthology.orchestra.run.smithi184.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T14:03:40.861 INFO:teuthology.orchestra.run.smithi184.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T14:03:40.861 INFO:teuthology.orchestra.run.smithi184.stdout:hardware devices. 2024-09-13T14:03:40.861 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:03:40.861 INFO:teuthology.orchestra.run.smithi184.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T14:03:40.862 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T14:03:40.862 INFO:teuthology.task.kernel:Updating grub on smithi184 to boot 5.14.0-505.el9.x86_64 2024-09-13T14:03:40.863 DEBUG:teuthology.orchestra.run.smithi184:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T14:03:41.535 INFO:teuthology.orchestra.run.smithi184.stderr:Generating grub configuration file ... 2024-09-13T14:03:42.818 INFO:teuthology.orchestra.run.smithi184.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T14:03:42.842 INFO:teuthology.orchestra.run.smithi184.stderr:done 2024-09-13T14:03:42.844 DEBUG:teuthology.orchestra.run.smithi184:> mktemp 2024-09-13T14:03:42.860 INFO:teuthology.orchestra.run.smithi184.stdout:/tmp/tmp.oU3IWblD0m 2024-09-13T14:03:42.860 DEBUG:teuthology.orchestra.run.smithi184:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.oU3IWblD0m 2024-09-13T14:03:42.926 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0666 /tmp/tmp.oU3IWblD0m 2024-09-13T14:03:43.073 DEBUG:teuthology.orchestra.remote:smithi184:/tmp/tmp.oU3IWblD0m is 6KB 2024-09-13T14:03:43.123 DEBUG:teuthology.orchestra.run.smithi184:> rm -fr /tmp/tmp.oU3IWblD0m 2024-09-13T14:03:43.138 DEBUG:teuthology.orchestra.run.smithi184:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T14:03:43.204 INFO:teuthology.orchestra.run.smithi184.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T14:03:43.204 INFO:teuthology.orchestra.run.smithi184.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T14:03:43.204 INFO:teuthology.orchestra.run.smithi184.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T14:03:43.204 INFO:teuthology.orchestra.run.smithi184.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-13T14:03:43.204 DEBUG:teuthology.orchestra.run.smithi184:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-13T14:03:43.370 DEBUG:teuthology.orchestra.run.smithi184:> sudo shutdown -r now 2024-09-13T14:04:08.999 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-13T14:04:08.999 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-09-13T14:04:09.000 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:04:13.377 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-13T14:04:13.377 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-09-13T14:04:13.378 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:04:27.325 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-09-13T14:04:31.933 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-09-13T14:04:36.330 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-09-13T14:04:36.331 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:04:39.425 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-09-13T14:04:40.935 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-09-13T14:04:40.936 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:04:51.435 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-09-13T14:04:51.436 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:05:09.821 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-09-13T14:05:24.837 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-09-13T14:05:24.838 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:05:25.474 DEBUG:teuthology.orchestra.run.smithi151:> true 2024-09-13T14:05:25.871 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-09-13T14:05:25.871 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-13T14:05:25.871 DEBUG:teuthology.orchestra.run.smithi151:> uname -r 2024-09-13T14:05:25.887 INFO:teuthology.orchestra.run.smithi151.stdout:5.14.0-505.el9.x86_64 2024-09-13T14:05:25.887 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-13T14:05:25.887 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T14:05:25.887 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T14:05:26.888 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-13T14:05:26.889 DEBUG:teuthology.orchestra.run.smithi151:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T14:05:26.953 INFO:teuthology.orchestra.run.smithi151.stdout:ttyS1 2024-09-13T14:05:26.984 DEBUG:teuthology.parallel:result is None 2024-09-13T14:05:40.946 DEBUG:teuthology.orchestra.remote:timed out 2024-09-13T14:05:52.948 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-09-13T14:05:52.949 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:05:53.347 DEBUG:teuthology.orchestra.run.smithi184:> true 2024-09-13T14:05:53.725 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-09-13T14:05:53.726 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-13T14:05:53.726 DEBUG:teuthology.orchestra.run.smithi184:> uname -r 2024-09-13T14:05:53.744 INFO:teuthology.orchestra.run.smithi184.stdout:5.14.0-505.el9.x86_64 2024-09-13T14:05:53.744 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-13T14:05:53.744 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T14:05:53.744 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T14:05:54.745 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-13T14:05:54.745 DEBUG:teuthology.orchestra.run.smithi184:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T14:05:54.824 INFO:teuthology.orchestra.run.smithi184.stdout:ttyS1 2024-09-13T14:05:54.843 DEBUG:teuthology.parallel:result is None 2024-09-13T14:05:54.843 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-13T14:05:54.853 INFO:teuthology.task.internal:Creating test directory... 2024-09-13T14:05:54.854 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T14:05:54.857 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T14:05:54.875 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-13T14:05:54.886 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-13T14:05:54.900 INFO:teuthology.task.internal:Creating archive directory... 2024-09-13T14:05:54.901 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T14:05:54.917 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T14:05:54.988 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-13T14:05:54.996 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-13T14:05:54.996 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T14:05:54.999 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T14:05:55.027 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:05:55.037 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:05:55.039 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:05:55.050 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:05:55.052 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-13T14:05:55.063 INFO:teuthology.task.internal:Configuring sudo... 2024-09-13T14:05:55.063 DEBUG:teuthology.orchestra.run.smithi151:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T14:05:55.084 DEBUG:teuthology.orchestra.run.smithi184:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T14:05:55.122 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-13T14:05:55.135 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-13T14:05:55.135 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T14:05:55.154 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T14:05:55.180 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T14:05:55.263 DEBUG:teuthology.orchestra.run.smithi151:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T14:05:55.306 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T14:05:55.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T14:05:55.430 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-13T14:05:55.430 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T14:05:55.495 DEBUG:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T14:05:55.550 DEBUG:teuthology.orchestra.run.smithi184:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T14:05:55.600 DEBUG:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T14:05:55.654 DEBUG:teuthology.orchestra.run.smithi184:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T14:05:55.690 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-13T14:05:55.690 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T14:05:55.759 DEBUG:teuthology.orchestra.run.smithi151:> sudo service rsyslog restart 2024-09-13T14:05:55.761 DEBUG:teuthology.orchestra.run.smithi184:> sudo service rsyslog restart 2024-09-13T14:05:55.818 INFO:teuthology.orchestra.run.smithi151.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T14:05:55.842 INFO:teuthology.orchestra.run.smithi184.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T14:05:56.146 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-13T14:05:56.155 INFO:teuthology.task.internal:Starting timer... 2024-09-13T14:05:56.155 INFO:teuthology.run_tasks:Running task pcp... 2024-09-13T14:05:56.214 INFO:teuthology.run_tasks:Running task selinux... 2024-09-13T14:05:56.228 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-09-13T14:05:56.229 DEBUG:teuthology.orchestra.run.smithi151:> sudo service auditd rotate 2024-09-13T14:05:56.288 INFO:teuthology.orchestra.run.smithi151.stdout:Rotating logs: 2024-09-13T14:05:56.290 DEBUG:teuthology.orchestra.run.smithi184:> sudo service auditd rotate 2024-09-13T14:05:56.334 INFO:teuthology.orchestra.run.smithi184.stdout:Rotating logs: 2024-09-13T14:05:56.336 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-13T14:05:56.337 DEBUG:teuthology.orchestra.run.smithi151:> /usr/sbin/getenforce 2024-09-13T14:05:56.357 INFO:teuthology.orchestra.run.smithi151.stdout:Permissive 2024-09-13T14:05:56.357 DEBUG:teuthology.orchestra.run.smithi184:> /usr/sbin/getenforce 2024-09-13T14:05:56.399 INFO:teuthology.orchestra.run.smithi184.stdout:Permissive 2024-09-13T14:05:56.399 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi151.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi184.front.sepia.ceph.com': 'permissive'} 2024-09-13T14:05:56.399 DEBUG:teuthology.orchestra.run.smithi151:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-13T14:05:56.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:05:56.427 DEBUG:teuthology.orchestra.run.smithi184:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-13T14:05:56.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:05:56.471 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-13T14:05:56.471 DEBUG:teuthology.orchestra.run.smithi151:> sudo /usr/sbin/setenforce permissive 2024-09-13T14:05:56.520 DEBUG:teuthology.orchestra.run.smithi184:> sudo /usr/sbin/setenforce permissive 2024-09-13T14:05:56.561 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-13T14:05:56.574 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-13T14:05:56.585 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-13T14:05:56.586 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi151.front.sepia.ceph.com,smithi184.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-13T14:09:47.282 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi151.front.sepia.ceph.com'), Remote(name='ubuntu@smithi184.front.sepia.ceph.com')] 2024-09-13T14:09:47.284 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-09-13T14:09:47.286 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:09:47.390 DEBUG:teuthology.orchestra.run.smithi151:> true 2024-09-13T14:09:47.467 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-09-13T14:09:47.468 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-09-13T14:09:47.468 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:09:47.569 DEBUG:teuthology.orchestra.run.smithi184:> true 2024-09-13T14:09:47.654 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-09-13T14:09:47.654 INFO:teuthology.run_tasks:Running task clock... 2024-09-13T14:09:47.664 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-13T14:09:47.664 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T14:09:47.665 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T14:09:47.667 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T14:09:47.667 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T14:09:47.693 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T14:09:47.708 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T14:09:47.724 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T14:09:47.733 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: ntpd: command not found 2024-09-13T14:09:47.739 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T14:09:47.743 INFO:teuthology.orchestra.run.smithi151.stdout:506 Cannot talk to daemon 2024-09-13T14:09:47.758 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T14:09:47.764 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: ntpd: command not found 2024-09-13T14:09:47.772 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T14:09:47.776 INFO:teuthology.orchestra.run.smithi184.stdout:506 Cannot talk to daemon 2024-09-13T14:09:47.790 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T14:09:47.804 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T14:09:47.828 INFO:teuthology.orchestra.run.smithi151.stderr:bash: line 1: ntpq: command not found 2024-09-13T14:09:47.833 INFO:teuthology.orchestra.run.smithi151.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T14:09:47.833 INFO:teuthology.orchestra.run.smithi151.stdout:=============================================================================== 2024-09-13T14:09:47.833 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:09:47.834 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:09:47.834 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:09:47.834 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:09:47.859 INFO:teuthology.orchestra.run.smithi184.stderr:bash: line 1: ntpq: command not found 2024-09-13T14:09:47.862 INFO:teuthology.orchestra.run.smithi184.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T14:09:47.862 INFO:teuthology.orchestra.run.smithi184.stdout:=============================================================================== 2024-09-13T14:09:47.862 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:09:47.863 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:09:47.863 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:09:47.863 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:09:47.864 INFO:teuthology.run_tasks:Running task pexec... 2024-09-13T14:09:47.873 INFO:teuthology.task.pexec:Executing custom commands... 2024-09-13T14:09:47.873 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi151.front.sepia.ceph.com 2024-09-13T14:09:47.873 DEBUG:teuthology.orchestra.run.smithi151:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-13T14:09:47.874 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi184.front.sepia.ceph.com 2024-09-13T14:09:47.874 DEBUG:teuthology.orchestra.run.smithi184:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-13T14:09:48.228 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-13T14:09:48.237 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:09:48.237 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-09-13T14:09:48.237 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:09:48.237 INFO:teuthology.orchestra.run.smithi151.stdout:Removing: 2024-09-13T14:09:48.238 INFO:teuthology.orchestra.run.smithi151.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-13T14:09:48.238 INFO:teuthology.orchestra.run.smithi151.stdout:Removing unused dependencies: 2024-09-13T14:09:48.238 INFO:teuthology.orchestra.run.smithi151.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-13T14:09:48.238 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:09:48.238 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-09-13T14:09:48.238 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:09:48.238 INFO:teuthology.orchestra.run.smithi151.stdout:Remove 2 Packages 2024-09-13T14:09:48.238 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:09:48.238 INFO:teuthology.orchestra.run.smithi151.stdout:Freed space: 5.5 M 2024-09-13T14:09:48.238 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-09-13T14:09:48.245 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-09-13T14:09:48.245 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-09-13T14:09:48.256 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-09-13T14:09:48.258 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:09:48.258 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-09-13T14:09:48.258 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:09:48.259 INFO:teuthology.orchestra.run.smithi184.stdout:Removing: 2024-09-13T14:09:48.259 INFO:teuthology.orchestra.run.smithi184.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-13T14:09:48.259 INFO:teuthology.orchestra.run.smithi184.stdout:Removing unused dependencies: 2024-09-13T14:09:48.259 INFO:teuthology.orchestra.run.smithi184.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-13T14:09:48.259 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:09:48.259 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-09-13T14:09:48.259 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:09:48.259 INFO:teuthology.orchestra.run.smithi184.stdout:Remove 2 Packages 2024-09-13T14:09:48.259 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:09:48.259 INFO:teuthology.orchestra.run.smithi184.stdout:Freed space: 5.5 M 2024-09-13T14:09:48.259 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-09-13T14:09:48.263 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-09-13T14:09:48.263 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-09-13T14:09:48.299 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-09-13T14:09:48.300 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-09-13T14:09:48.318 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-09-13T14:09:48.319 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-09-13T14:09:48.450 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-09-13T14:09:48.465 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-09-13T14:09:48.530 INFO:teuthology.orchestra.run.smithi151.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-13T14:09:48.547 INFO:teuthology.orchestra.run.smithi184.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-13T14:09:48.578 INFO:teuthology.orchestra.run.smithi151.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-13T14:09:48.602 INFO:teuthology.orchestra.run.smithi184.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-13T14:09:49.091 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-13T14:09:49.092 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-13T14:09:49.140 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-13T14:09:49.140 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-13T14:09:49.389 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-13T14:09:49.389 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:09:49.389 INFO:teuthology.orchestra.run.smithi151.stdout:Removed: 2024-09-13T14:09:49.389 INFO:teuthology.orchestra.run.smithi151.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-13T14:09:49.389 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:09:49.390 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-13T14:09:49.445 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-13T14:09:49.446 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:09:49.446 INFO:teuthology.orchestra.run.smithi184.stdout:Removed: 2024-09-13T14:09:49.446 INFO:teuthology.orchestra.run.smithi184.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-13T14:09:49.446 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:09:49.446 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-09-13T14:09:50.181 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:55 ago on Fri 13 Sep 2024 02:08:55 PM UTC. 2024-09-13T14:09:50.314 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-13T14:09:50.315 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:09:50.315 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-09-13T14:09:50.315 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:09:50.315 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-09-13T14:09:50.316 INFO:teuthology.orchestra.run.smithi151.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-13T14:09:50.316 INFO:teuthology.orchestra.run.smithi151.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-13T14:09:50.316 INFO:teuthology.orchestra.run.smithi151.stdout: runc x86_64 4:1.1.13-4.el9 appstream 3.2 M 2024-09-13T14:09:50.316 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-09-13T14:09:50.316 INFO:teuthology.orchestra.run.smithi151.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-13T14:09:50.316 INFO:teuthology.orchestra.run.smithi151.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-13T14:09:50.316 INFO:teuthology.orchestra.run.smithi151.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-13T14:09:50.316 INFO:teuthology.orchestra.run.smithi151.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-13T14:09:50.316 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:09:50.316 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-09-13T14:09:50.316 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:09:50.316 INFO:teuthology.orchestra.run.smithi151.stdout:Install 7 Packages 2024-09-13T14:09:50.316 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:09:50.317 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:46 ago on Fri 13 Sep 2024 02:09:04 PM UTC. 2024-09-13T14:09:50.325 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 5.2 M 2024-09-13T14:09:50.325 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 19 M 2024-09-13T14:09:50.325 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-09-13T14:09:50.450 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-09-13T14:09:50.451 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:09:50.451 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-09-13T14:09:50.451 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:09:50.451 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-09-13T14:09:50.451 INFO:teuthology.orchestra.run.smithi184.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-13T14:09:50.451 INFO:teuthology.orchestra.run.smithi184.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-13T14:09:50.451 INFO:teuthology.orchestra.run.smithi184.stdout: runc x86_64 4:1.1.13-4.el9 appstream 3.2 M 2024-09-13T14:09:50.451 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-09-13T14:09:50.451 INFO:teuthology.orchestra.run.smithi184.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-13T14:09:50.451 INFO:teuthology.orchestra.run.smithi184.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-13T14:09:50.451 INFO:teuthology.orchestra.run.smithi184.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-13T14:09:50.452 INFO:teuthology.orchestra.run.smithi184.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-13T14:09:50.452 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:09:50.452 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-09-13T14:09:50.452 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:09:50.452 INFO:teuthology.orchestra.run.smithi184.stdout:Install 7 Packages 2024-09-13T14:09:50.452 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:09:50.457 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 5.2 M 2024-09-13T14:09:50.457 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 19 M 2024-09-13T14:09:50.457 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-09-13T14:09:50.845 INFO:teuthology.orchestra.run.smithi151.stdout:(1/7): nvmetcli-0.7-3.el9.noarch.rpm 120 kB/s | 44 kB 00:00 2024-09-13T14:09:50.946 INFO:teuthology.orchestra.run.smithi151.stdout:(2/7): libnvme-1.9-3.el9.x86_64.rpm 225 kB/s | 106 kB 00:00 2024-09-13T14:09:50.966 INFO:teuthology.orchestra.run.smithi184.stdout:(1/7): nvmetcli-0.7-3.el9.noarch.rpm 388 kB/s | 44 kB 00:00 2024-09-13T14:09:50.992 INFO:teuthology.orchestra.run.smithi184.stdout:(2/7): libnvme-1.9-3.el9.x86_64.rpm 758 kB/s | 106 kB 00:00 2024-09-13T14:09:51.017 INFO:teuthology.orchestra.run.smithi184.stdout:(3/7): python3-configshell-1.1.30-1.el9.noarch. 1.4 MB/s | 72 kB 00:00 2024-09-13T14:09:51.046 INFO:teuthology.orchestra.run.smithi151.stdout:(3/7): python3-configshell-1.1.30-1.el9.noarch. 358 kB/s | 72 kB 00:00 2024-09-13T14:09:51.051 INFO:teuthology.orchestra.run.smithi184.stdout:(4/7): python3-kmod-0.9-32.el9.x86_64.rpm 1.4 MB/s | 84 kB 00:00 2024-09-13T14:09:51.071 INFO:teuthology.orchestra.run.smithi151.stdout:(4/7): python3-kmod-0.9-32.el9.x86_64.rpm 671 kB/s | 84 kB 00:00 2024-09-13T14:09:51.084 INFO:teuthology.orchestra.run.smithi184.stdout:(5/7): nvme-cli-2.9.1-6.el9.x86_64.rpm 3.8 MB/s | 906 kB 00:00 2024-09-13T14:09:51.151 INFO:teuthology.orchestra.run.smithi184.stdout:(6/7): python3-urwid-2.1.2-4.el9.x86_64.rpm 6.1 MB/s | 837 kB 00:00 2024-09-13T14:09:51.372 INFO:teuthology.orchestra.run.smithi151.stdout:(5/7): nvme-cli-2.9.1-6.el9.x86_64.rpm 1.0 MB/s | 906 kB 00:00 2024-09-13T14:09:51.439 INFO:teuthology.orchestra.run.smithi151.stdout:(6/7): python3-urwid-2.1.2-4.el9.x86_64.rpm 2.1 MB/s | 837 kB 00:00 2024-09-13T14:09:51.710 INFO:teuthology.orchestra.run.smithi184.stdout:(7/7): runc-1.1.13-4.el9.x86_64.rpm 4.8 MB/s | 3.2 MB 00:00 2024-09-13T14:09:51.710 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:09:51.710 INFO:teuthology.orchestra.run.smithi184.stdout:Total 4.1 MB/s | 5.2 MB 00:01 2024-09-13T14:09:51.820 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-09-13T14:09:51.835 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-09-13T14:09:51.835 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-09-13T14:09:52.129 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-09-13T14:09:52.130 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-09-13T14:09:52.347 INFO:teuthology.orchestra.run.smithi151.stdout:(7/7): runc-1.1.13-4.el9.x86_64.rpm 2.5 MB/s | 3.2 MB 00:01 2024-09-13T14:09:52.348 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:09:52.348 INFO:teuthology.orchestra.run.smithi151.stdout:Total 2.6 MB/s | 5.2 MB 00:02 2024-09-13T14:09:52.464 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-09-13T14:09:52.479 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-09-13T14:09:52.480 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-09-13T14:09:52.616 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-09-13T14:09:52.720 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/7 2024-09-13T14:09:52.755 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/7 2024-09-13T14:09:52.773 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-09-13T14:09:52.773 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-09-13T14:09:52.783 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/7 2024-09-13T14:09:52.990 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/7 2024-09-13T14:09:53.031 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-13T14:09:53.259 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-09-13T14:09:53.342 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/7 2024-09-13T14:09:53.388 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/7 2024-09-13T14:09:53.422 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/7 2024-09-13T14:09:53.460 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-13T14:09:53.484 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/7 2024-09-13T14:09:53.600 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/7 2024-09-13T14:09:53.636 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/7 2024-09-13T14:09:53.662 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-13T14:09:53.671 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-13T14:09:54.117 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-13T14:09:54.150 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/7 2024-09-13T14:09:54.260 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/7 2024-09-13T14:09:54.328 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-13T14:09:54.641 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-13T14:09:54.641 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/7 2024-09-13T14:09:54.641 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/7 2024-09-13T14:09:54.641 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/7 2024-09-13T14:09:54.641 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/7 2024-09-13T14:09:54.641 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/7 2024-09-13T14:09:54.641 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/7 2024-09-13T14:09:55.118 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-13T14:09:55.118 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:09:55.118 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-09-13T14:09:55.118 INFO:teuthology.orchestra.run.smithi184.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-13T14:09:55.118 INFO:teuthology.orchestra.run.smithi184.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-13T14:09:55.118 INFO:teuthology.orchestra.run.smithi184.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-13T14:09:55.118 INFO:teuthology.orchestra.run.smithi184.stdout: runc-4:1.1.13-4.el9.x86_64 2024-09-13T14:09:55.118 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:09:55.119 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-09-13T14:09:55.321 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-13T14:09:55.321 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/7 2024-09-13T14:09:55.321 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/7 2024-09-13T14:09:55.321 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/7 2024-09-13T14:09:55.321 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/7 2024-09-13T14:09:55.321 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/7 2024-09-13T14:09:55.321 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/7 2024-09-13T14:09:55.558 DEBUG:teuthology.parallel:result is None 2024-09-13T14:09:55.709 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-13T14:09:55.710 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:09:55.710 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-09-13T14:09:55.710 INFO:teuthology.orchestra.run.smithi151.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-13T14:09:55.710 INFO:teuthology.orchestra.run.smithi151.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-13T14:09:55.710 INFO:teuthology.orchestra.run.smithi151.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-13T14:09:55.710 INFO:teuthology.orchestra.run.smithi151.stdout: runc-4:1.1.13-4.el9.x86_64 2024-09-13T14:09:55.710 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:09:55.710 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-13T14:09:56.074 DEBUG:teuthology.parallel:result is None 2024-09-13T14:09:56.074 INFO:teuthology.run_tasks:Running task install... 2024-09-13T14:09:56.082 DEBUG:teuthology.task.install:project ceph 2024-09-13T14:09:56.082 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}} 2024-09-13T14:09:56.082 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'} 2024-09-13T14:09:56.082 INFO:teuthology.task.install:Using flavor: default 2024-09-13T14:09:56.088 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-13T14:09:56.088 INFO:teuthology.task.install:extra packages: [] 2024-09-13T14:09:56.088 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'tag': None, 'wait_for_package': False} 2024-09-13T14:09:56.089 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T14:09:56.089 INFO:teuthology.packaging:ref: None 2024-09-13T14:09:56.089 INFO:teuthology.packaging:tag: None 2024-09-13T14:09:56.089 INFO:teuthology.packaging:branch: quincy 2024-09-13T14:09:56.089 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:09:56.089 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-09-13T14:09:56.091 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'tag': None, 'wait_for_package': False} 2024-09-13T14:09:56.091 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T14:09:56.092 INFO:teuthology.packaging:ref: None 2024-09-13T14:09:56.092 INFO:teuthology.packaging:tag: None 2024-09-13T14:09:56.092 INFO:teuthology.packaging:branch: quincy 2024-09-13T14:09:56.092 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:09:56.092 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-09-13T14:09:56.315 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/bbdadfa31f8fb9afaa389358d138cac49bd69e56/centos/9/flavors/default/ 2024-09-13T14:09:56.315 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1627.gbbdadfa3 2024-09-13T14:09:56.351 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/bbdadfa31f8fb9afaa389358d138cac49bd69e56/centos/9/flavors/default/ 2024-09-13T14:09:56.351 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1627.gbbdadfa3 2024-09-13T14:09:56.450 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/quincy/bbdadfa31f8fb9afaa389358d138cac49bd69e56/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/bbdadfa31f8fb9afaa389358d138cac49bd69e56/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/bbdadfa31f8fb9afaa389358d138cac49bd69e56/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T14:09:56.450 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-13T14:09:56.450 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T14:09:56.471 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/quincy/bbdadfa31f8fb9afaa389358d138cac49bd69e56/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/bbdadfa31f8fb9afaa389358d138cac49bd69e56/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/bbdadfa31f8fb9afaa389358d138cac49bd69e56/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T14:09:56.471 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-13T14:09:56.471 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T14:09:56.477 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-13T14:09:56.477 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T14:09:56.478 INFO:teuthology.packaging:ref: None 2024-09-13T14:09:56.478 INFO:teuthology.packaging:tag: None 2024-09-13T14:09:56.478 INFO:teuthology.packaging:branch: quincy 2024-09-13T14:09:56.478 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:09:56.478 DEBUG:teuthology.orchestra.run.smithi184:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T14:09:56.498 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-13T14:09:56.498 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T14:09:56.498 INFO:teuthology.packaging:ref: None 2024-09-13T14:09:56.498 INFO:teuthology.packaging:tag: None 2024-09-13T14:09:56.498 INFO:teuthology.packaging:branch: quincy 2024-09-13T14:09:56.498 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:09:56.499 DEBUG:teuthology.orchestra.run.smithi151:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T14:09:56.544 DEBUG:teuthology.orchestra.run.smithi184:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T14:09:56.565 DEBUG:teuthology.orchestra.run.smithi151:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T14:09:56.619 DEBUG:teuthology.orchestra.run.smithi184:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T14:09:56.660 DEBUG:teuthology.orchestra.run.smithi151:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T14:09:56.683 INFO:teuthology.orchestra.run.smithi184.stdout:check_obsoletes = 1 2024-09-13T14:09:56.684 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum clean all 2024-09-13T14:09:56.685 INFO:teuthology.orchestra.run.smithi151.stdout:check_obsoletes = 1 2024-09-13T14:09:56.686 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum clean all 2024-09-13T14:09:56.905 INFO:teuthology.orchestra.run.smithi151.stdout:45 files removed 2024-09-13T14:09:56.908 INFO:teuthology.orchestra.run.smithi184.stdout:45 files removed 2024-09-13T14:09:56.932 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-radosgw 2024-09-13T14:09:56.937 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-radosgw 2024-09-13T14:09:57.485 INFO:teuthology.orchestra.run.smithi151.stdout:ceph packages for x86_64 290 kB/s | 77 kB 00:00 2024-09-13T14:09:57.500 INFO:teuthology.orchestra.run.smithi184.stdout:ceph packages for x86_64 286 kB/s | 77 kB 00:00 2024-09-13T14:09:57.769 INFO:teuthology.orchestra.run.smithi151.stdout:ceph noarch packages 45 kB/s | 11 kB 00:00 2024-09-13T14:09:57.792 INFO:teuthology.orchestra.run.smithi184.stdout:ceph noarch packages 43 kB/s | 11 kB 00:00 2024-09-13T14:09:58.045 INFO:teuthology.orchestra.run.smithi151.stdout:ceph source packages 7.5 kB/s | 1.8 kB 00:00 2024-09-13T14:09:58.068 INFO:teuthology.orchestra.run.smithi184.stdout:ceph source packages 7.5 kB/s | 1.8 kB 00:00 2024-09-13T14:09:58.487 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - BaseOS 20 MB/s | 8.2 MB 00:00 2024-09-13T14:10:00.151 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - BaseOS 4.0 MB/s | 8.2 MB 00:02 2024-09-13T14:10:00.522 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - AppStream 22 MB/s | 20 MB 00:00 2024-09-13T14:10:02.227 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - AppStream 21 MB/s | 20 MB 00:00 2024-09-13T14:10:05.257 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - CRB 7.2 MB/s | 6.5 MB 00:00 2024-09-13T14:10:06.511 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-13T14:10:06.767 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - Extras packages 64 kB/s | 19 kB 00:00 2024-09-13T14:10:07.945 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - Extras packages 90 kB/s | 19 kB 00:00 2024-09-13T14:10:08.280 INFO:teuthology.orchestra.run.smithi151.stdout:Extra Packages for Enterprise Linux 16 MB/s | 23 MB 00:01 2024-09-13T14:10:09.091 INFO:teuthology.orchestra.run.smithi184.stdout:Extra Packages for Enterprise Linux 22 MB/s | 23 MB 00:01 2024-09-13T14:10:13.736 INFO:teuthology.orchestra.run.smithi151.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-13T14:10:14.530 INFO:teuthology.orchestra.run.smithi184.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-13T14:10:16.418 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-13T14:10:16.419 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:10:16.419 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-09-13T14:10:16.419 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:10:16.419 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-09-13T14:10:16.420 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-radosgw x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 14 M 2024-09-13T14:10:16.420 INFO:teuthology.orchestra.run.smithi151.stdout:Upgrading: 2024-09-13T14:10:16.420 INFO:teuthology.orchestra.run.smithi151.stdout: librados2 x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 3.1 M 2024-09-13T14:10:16.420 INFO:teuthology.orchestra.run.smithi151.stdout: librbd1 x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 2.9 M 2024-09-13T14:10:16.420 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-09-13T14:10:16.420 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-base x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 5.0 M 2024-09-13T14:10:16.420 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-common x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 22 M 2024-09-13T14:10:16.420 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-selinux x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 25 k 2024-09-13T14:10:16.420 INFO:teuthology.orchestra.run.smithi151.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T14:10:16.420 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs2 x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 637 k 2024-09-13T14:10:16.420 INFO:teuthology.orchestra.run.smithi151.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T14:10:16.420 INFO:teuthology.orchestra.run.smithi151.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T14:10:16.421 INFO:teuthology.orchestra.run.smithi151.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T14:10:16.421 INFO:teuthology.orchestra.run.smithi151.stdout: libradosstriper1 x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 457 k 2024-09-13T14:10:16.421 INFO:teuthology.orchestra.run.smithi151.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T14:10:16.421 INFO:teuthology.orchestra.run.smithi151.stdout: librgw2 x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 4.9 M 2024-09-13T14:10:16.421 INFO:teuthology.orchestra.run.smithi151.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T14:10:16.421 INFO:teuthology.orchestra.run.smithi151.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T14:10:16.421 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-argparse x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 45 k 2024-09-13T14:10:16.421 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-common x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 120 k 2024-09-13T14:10:16.421 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cephfs x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 156 k 2024-09-13T14:10:16.421 INFO:teuthology.orchestra.run.smithi151.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T14:10:16.421 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rados x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 320 k 2024-09-13T14:10:16.421 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rbd x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 294 k 2024-09-13T14:10:16.421 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rgw x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 99 k 2024-09-13T14:10:16.421 INFO:teuthology.orchestra.run.smithi151.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T14:10:16.422 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:10:16.422 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-09-13T14:10:16.422 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:10:16.422 INFO:teuthology.orchestra.run.smithi151.stdout:Install 22 Packages 2024-09-13T14:10:16.422 INFO:teuthology.orchestra.run.smithi151.stdout:Upgrade 2 Packages 2024-09-13T14:10:16.422 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:10:16.422 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 58 M 2024-09-13T14:10:16.422 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-09-13T14:10:16.902 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-09-13T14:10:16.904 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:10:16.904 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-09-13T14:10:16.904 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:10:16.904 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-09-13T14:10:16.904 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-radosgw x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 14 M 2024-09-13T14:10:16.904 INFO:teuthology.orchestra.run.smithi184.stdout:Upgrading: 2024-09-13T14:10:16.904 INFO:teuthology.orchestra.run.smithi184.stdout: librados2 x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 3.1 M 2024-09-13T14:10:16.904 INFO:teuthology.orchestra.run.smithi184.stdout: librbd1 x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 2.9 M 2024-09-13T14:10:16.904 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-09-13T14:10:16.905 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-base x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 5.0 M 2024-09-13T14:10:16.905 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-common x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 22 M 2024-09-13T14:10:16.905 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-selinux x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 25 k 2024-09-13T14:10:16.905 INFO:teuthology.orchestra.run.smithi184.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T14:10:16.905 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs2 x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 637 k 2024-09-13T14:10:16.905 INFO:teuthology.orchestra.run.smithi184.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T14:10:16.905 INFO:teuthology.orchestra.run.smithi184.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T14:10:16.905 INFO:teuthology.orchestra.run.smithi184.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T14:10:16.905 INFO:teuthology.orchestra.run.smithi184.stdout: libradosstriper1 x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 457 k 2024-09-13T14:10:16.905 INFO:teuthology.orchestra.run.smithi184.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T14:10:16.905 INFO:teuthology.orchestra.run.smithi184.stdout: librgw2 x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 4.9 M 2024-09-13T14:10:16.905 INFO:teuthology.orchestra.run.smithi184.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T14:10:16.905 INFO:teuthology.orchestra.run.smithi184.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T14:10:16.905 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-argparse x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 45 k 2024-09-13T14:10:16.905 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-common x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 120 k 2024-09-13T14:10:16.906 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cephfs x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 156 k 2024-09-13T14:10:16.906 INFO:teuthology.orchestra.run.smithi184.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T14:10:16.906 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rados x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 320 k 2024-09-13T14:10:16.906 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rbd x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 294 k 2024-09-13T14:10:16.906 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rgw x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 99 k 2024-09-13T14:10:16.906 INFO:teuthology.orchestra.run.smithi184.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T14:10:16.906 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:10:16.906 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-09-13T14:10:16.906 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:10:16.906 INFO:teuthology.orchestra.run.smithi184.stdout:Install 22 Packages 2024-09-13T14:10:16.906 INFO:teuthology.orchestra.run.smithi184.stdout:Upgrade 2 Packages 2024-09-13T14:10:16.906 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:10:16.907 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 58 M 2024-09-13T14:10:16.907 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-09-13T14:10:17.204 INFO:teuthology.orchestra.run.smithi151.stdout:(1/24): ceph-base-17.2.7-1627.gbbdadfa3.el9.x86 9.8 MB/s | 5.0 MB 00:00 2024-09-13T14:10:17.271 INFO:teuthology.orchestra.run.smithi151.stdout:(2/24): ceph-selinux-17.2.7-1627.gbbdadfa3.el9. 377 kB/s | 25 kB 00:00 2024-09-13T14:10:17.388 INFO:teuthology.orchestra.run.smithi151.stdout:(3/24): libcephfs2-17.2.7-1627.gbbdadfa3.el9.x8 5.3 MB/s | 637 kB 00:00 2024-09-13T14:10:17.489 INFO:teuthology.orchestra.run.smithi151.stdout:(4/24): libradosstriper1-17.2.7-1627.gbbdadfa3. 4.5 MB/s | 457 kB 00:00 2024-09-13T14:10:17.797 INFO:teuthology.orchestra.run.smithi151.stdout:(5/24): ceph-radosgw-17.2.7-1627.gbbdadfa3.el9. 13 MB/s | 14 MB 00:01 2024-09-13T14:10:17.998 INFO:teuthology.orchestra.run.smithi184.stdout:(1/24): ceph-base-17.2.7-1627.gbbdadfa3.el9.x86 7.4 MB/s | 5.0 MB 00:00 2024-09-13T14:10:18.031 INFO:teuthology.orchestra.run.smithi151.stdout:(6/24): ceph-common-17.2.7-1627.gbbdadfa3.el9.x 17 MB/s | 22 MB 00:01 2024-09-13T14:10:18.056 INFO:teuthology.orchestra.run.smithi151.stdout:(7/24): python3-ceph-argparse-17.2.7-1627.gbbda 174 kB/s | 45 kB 00:00 2024-09-13T14:10:18.057 INFO:teuthology.orchestra.run.smithi184.stdout:(2/24): ceph-selinux-17.2.7-1627.gbbdadfa3.el9. 433 kB/s | 25 kB 00:00 2024-09-13T14:10:18.140 INFO:teuthology.orchestra.run.smithi151.stdout:(8/24): librgw2-17.2.7-1627.gbbdadfa3.el9.x86_6 7.6 MB/s | 4.9 MB 00:00 2024-09-13T14:10:18.157 INFO:teuthology.orchestra.run.smithi184.stdout:(3/24): libcephfs2-17.2.7-1627.gbbdadfa3.el9.x8 6.2 MB/s | 637 kB 00:00 2024-09-13T14:10:18.165 INFO:teuthology.orchestra.run.smithi151.stdout:(9/24): python3-ceph-common-17.2.7-1627.gbbdadf 899 kB/s | 120 kB 00:00 2024-09-13T14:10:18.199 INFO:teuthology.orchestra.run.smithi151.stdout:(10/24): python3-cephfs-17.2.7-1627.gbbdadfa3.e 1.1 MB/s | 156 kB 00:00 2024-09-13T14:10:18.225 INFO:teuthology.orchestra.run.smithi151.stdout:(11/24): python3-rados-17.2.7-1627.gbbdadfa3.el 3.7 MB/s | 320 kB 00:00 2024-09-13T14:10:18.250 INFO:teuthology.orchestra.run.smithi151.stdout:(12/24): python3-rbd-17.2.7-1627.gbbdadfa3.el9. 3.4 MB/s | 294 kB 00:00 2024-09-13T14:10:18.257 INFO:teuthology.orchestra.run.smithi184.stdout:(4/24): libradosstriper1-17.2.7-1627.gbbdadfa3. 4.5 MB/s | 457 kB 00:00 2024-09-13T14:10:18.284 INFO:teuthology.orchestra.run.smithi151.stdout:(13/24): python3-rgw-17.2.7-1627.gbbdadfa3.el9. 1.2 MB/s | 99 kB 00:00 2024-09-13T14:10:18.392 INFO:teuthology.orchestra.run.smithi151.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 957 kB/s | 160 kB 00:00 2024-09-13T14:10:18.418 INFO:teuthology.orchestra.run.smithi151.stdout:(15/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 270 kB/s | 45 kB 00:00 2024-09-13T14:10:18.468 INFO:teuthology.orchestra.run.smithi151.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 846 kB/s | 42 kB 00:00 2024-09-13T14:10:18.502 INFO:teuthology.orchestra.run.smithi151.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-09-13T14:10:18.552 INFO:teuthology.orchestra.run.smithi151.stdout:(18/24): librdkafka-1.6.1-102.el9.x86_64.rpm 2.4 MB/s | 662 kB 00:00 2024-09-13T14:10:18.586 INFO:teuthology.orchestra.run.smithi151.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 640 kB/s | 49 kB 00:00 2024-09-13T14:10:18.607 INFO:teuthology.orchestra.run.smithi184.stdout:(5/24): librgw2-17.2.7-1627.gbbdadfa3.el9.x86_6 14 MB/s | 4.9 MB 00:00 2024-09-13T14:10:18.611 INFO:teuthology.orchestra.run.smithi151.stdout:(20/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.1 MB/s | 308 kB 00:00 2024-09-13T14:10:18.636 INFO:teuthology.orchestra.run.smithi151.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 839 kB/s | 67 kB 00:00 2024-09-13T14:10:18.704 INFO:teuthology.orchestra.run.smithi151.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-09-13T14:10:18.758 INFO:teuthology.orchestra.run.smithi184.stdout:(6/24): ceph-radosgw-17.2.7-1627.gbbdadfa3.el9. 10 MB/s | 14 MB 00:01 2024-09-13T14:10:18.783 INFO:teuthology.orchestra.run.smithi184.stdout:(7/24): python3-ceph-argparse-17.2.7-1627.gbbda 258 kB/s | 45 kB 00:00 2024-09-13T14:10:18.808 INFO:teuthology.orchestra.run.smithi184.stdout:(8/24): python3-ceph-common-17.2.7-1627.gbbdadf 2.3 MB/s | 120 kB 00:00 2024-09-13T14:10:18.820 INFO:teuthology.orchestra.run.smithi151.stdout:(23/24): librbd1-17.2.7-1627.gbbdadfa3.el9.x86_ 16 MB/s | 2.9 MB 00:00 2024-09-13T14:10:18.834 INFO:teuthology.orchestra.run.smithi184.stdout:(9/24): python3-cephfs-17.2.7-1627.gbbdadfa3.el 3.0 MB/s | 156 kB 00:00 2024-09-13T14:10:18.859 INFO:teuthology.orchestra.run.smithi184.stdout:(10/24): python3-rados-17.2.7-1627.gbbdadfa3.el 6.2 MB/s | 320 kB 00:00 2024-09-13T14:10:18.879 INFO:teuthology.orchestra.run.smithi151.stdout:(24/24): librados2-17.2.7-1627.gbbdadfa3.el9.x8 12 MB/s | 3.1 MB 00:00 2024-09-13T14:10:18.884 INFO:teuthology.orchestra.run.smithi184.stdout:(11/24): python3-rbd-17.2.7-1627.gbbdadfa3.el9. 5.7 MB/s | 294 kB 00:00 2024-09-13T14:10:18.885 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:10:18.885 INFO:teuthology.orchestra.run.smithi151.stdout:Total 24 MB/s | 58 MB 00:02 2024-09-13T14:10:18.909 INFO:teuthology.orchestra.run.smithi184.stdout:(12/24): python3-rgw-17.2.7-1627.gbbdadfa3.el9. 1.9 MB/s | 99 kB 00:00 2024-09-13T14:10:18.949 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-09-13T14:10:18.984 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-09-13T14:10:18.984 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-09-13T14:10:19.110 INFO:teuthology.orchestra.run.smithi184.stdout:(13/24): ceph-common-17.2.7-1627.gbbdadfa3.el9. 13 MB/s | 22 MB 00:01 2024-09-13T14:10:19.218 INFO:teuthology.orchestra.run.smithi184.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 146 kB/s | 45 kB 00:00 2024-09-13T14:10:19.269 INFO:teuthology.orchestra.run.smithi184.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 417 kB/s | 160 kB 00:00 2024-09-13T14:10:19.319 INFO:teuthology.orchestra.run.smithi184.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 846 kB/s | 42 kB 00:00 2024-09-13T14:10:19.344 INFO:teuthology.orchestra.run.smithi184.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-09-13T14:10:19.386 INFO:teuthology.orchestra.run.smithi184.stdout:(18/24): librdkafka-1.6.1-102.el9.x86_64.rpm 2.3 MB/s | 662 kB 00:00 2024-09-13T14:10:19.437 INFO:teuthology.orchestra.run.smithi184.stdout:(19/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.6 MB/s | 308 kB 00:00 2024-09-13T14:10:19.446 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-09-13T14:10:19.447 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-09-13T14:10:19.462 INFO:teuthology.orchestra.run.smithi184.stdout:(20/24): liboath-2.6.7-2.el9.x86_64.rpm 441 kB/s | 49 kB 00:00 2024-09-13T14:10:19.487 INFO:teuthology.orchestra.run.smithi184.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 734 kB/s | 67 kB 00:00 2024-09-13T14:10:19.546 INFO:teuthology.orchestra.run.smithi184.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-13T14:10:19.663 INFO:teuthology.orchestra.run.smithi184.stdout:(23/24): librados2-17.2.7-1627.gbbdadfa3.el9.x8 15 MB/s | 3.1 MB 00:00 2024-09-13T14:10:19.724 INFO:teuthology.orchestra.run.smithi184.stdout:(24/24): librbd1-17.2.7-1627.gbbdadfa3.el9.x86_ 13 MB/s | 2.9 MB 00:00 2024-09-13T14:10:19.726 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:10:19.726 INFO:teuthology.orchestra.run.smithi184.stdout:Total 21 MB/s | 58 MB 00:02 2024-09-13T14:10:19.789 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-09-13T14:10:19.822 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-09-13T14:10:19.822 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-09-13T14:10:20.318 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-09-13T14:10:20.319 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-09-13T14:10:20.656 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-09-13T14:10:21.062 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-13T14:10:21.124 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : librados2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2/26 2024-09-13T14:10:21.215 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2/26 2024-09-13T14:10:21.312 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rados-2:17.2.7-1627.gbbdadfa3.el9.x86_64 3/26 2024-09-13T14:10:21.493 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephfs2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 4/26 2024-09-13T14:10:21.493 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-09-13T14:10:21.580 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephfs2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 4/26 2024-09-13T14:10:21.627 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-13T14:10:21.673 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-13T14:10:21.867 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-13T14:10:21.932 INFO:teuthology.orchestra.run.smithi184.stdout: Upgrading : librados2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2/26 2024-09-13T14:10:22.031 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librados2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2/26 2024-09-13T14:10:22.110 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rados-2:17.2.7-1627.gbbdadfa3.el9.x86_64 3/26 2024-09-13T14:10:22.116 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-13T14:10:22.133 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libcephfs2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 4/26 2024-09-13T14:10:22.146 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librgw2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 8/26 2024-09-13T14:10:22.198 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librgw2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 8/26 2024-09-13T14:10:22.221 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libcephfs2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 4/26 2024-09-13T14:10:22.226 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-13T14:10:22.269 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-13T14:10:22.277 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ceph-argparse-2:17.2.7-1627.gbbdadfa3.el9. 10/26 2024-09-13T14:10:22.296 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-13T14:10:22.323 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cephfs-2:17.2.7-1627.gbbdadfa3.el9.x86_64 11/26 2024-09-13T14:10:22.386 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 12/26 2024-09-13T14:10:22.414 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libradosstriper1-2:17.2.7-1627.gbbdadfa3.el9.x86_6 13/26 2024-09-13T14:10:22.463 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1627.gbbdadfa3.el9.x86_6 13/26 2024-09-13T14:10:22.502 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-13T14:10:22.539 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-13T14:10:22.572 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-13T14:10:22.741 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-13T14:10:22.770 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librgw2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 8/26 2024-09-13T14:10:22.816 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librgw2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 8/26 2024-09-13T14:10:22.844 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-13T14:10:22.866 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-13T14:10:22.887 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-ceph-argparse-2:17.2.7-1627.gbbdadfa3.el9. 10/26 2024-09-13T14:10:22.902 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : librbd1-2:17.2.7-1627.gbbdadfa3.el9.x86_64 18/26 2024-09-13T14:10:22.924 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cephfs-2:17.2.7-1627.gbbdadfa3.el9.x86_64 11/26 2024-09-13T14:10:22.991 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librbd1-2:17.2.7-1627.gbbdadfa3.el9.x86_64 18/26 2024-09-13T14:10:22.994 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 12/26 2024-09-13T14:10:23.031 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libradosstriper1-2:17.2.7-1627.gbbdadfa3.el9.x86_6 13/26 2024-09-13T14:10:23.066 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 19/26 2024-09-13T14:10:23.089 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1627.gbbdadfa3.el9.x86_6 13/26 2024-09-13T14:10:23.105 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x8 20/26 2024-09-13T14:10:23.136 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-13T14:10:23.189 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-13T14:10:23.231 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-13T14:10:23.525 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-13T14:10:23.568 INFO:teuthology.orchestra.run.smithi184.stdout: Upgrading : librbd1-2:17.2.7-1627.gbbdadfa3.el9.x86_64 18/26 2024-09-13T14:10:23.658 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librbd1-2:17.2.7-1627.gbbdadfa3.el9.x86_64 18/26 2024-09-13T14:10:23.732 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 19/26 2024-09-13T14:10:23.762 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x8 20/26 2024-09-13T14:10:25.575 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x86_64 21/26 2024-09-13T14:10:25.612 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x86_64 21/26 2024-09-13T14:10:26.153 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x86_64 21/26 2024-09-13T14:10:26.189 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x86_64 21/26 2024-09-13T14:10:26.199 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-base-2:17.2.7-1627.gbbdadfa3.el9.x86_64 22/26 2024-09-13T14:10:26.256 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x86_64 21/26 2024-09-13T14:10:26.300 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-base-2:17.2.7-1627.gbbdadfa3.el9.x86_64 22/26 2024-09-13T14:10:26.300 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T14:10:26.300 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T14:10:26.300 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:10:26.334 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-selinux-2:17.2.7-1627.gbbdadfa3.el9.x86_64 23/26 2024-09-13T14:10:26.815 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x86_64 21/26 2024-09-13T14:10:26.845 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-base-2:17.2.7-1627.gbbdadfa3.el9.x86_64 22/26 2024-09-13T14:10:26.935 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-base-2:17.2.7-1627.gbbdadfa3.el9.x86_64 22/26 2024-09-13T14:10:26.935 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T14:10:26.935 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T14:10:26.935 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:10:26.971 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-selinux-2:17.2.7-1627.gbbdadfa3.el9.x86_64 23/26 2024-09-13T14:10:36.116 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1627.gbbdadfa3.el9.x86_64 23/26 2024-09-13T14:10:36.117 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /sys 2024-09-13T14:10:36.117 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /proc 2024-09-13T14:10:36.117 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /mnt 2024-09-13T14:10:36.117 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /var/tmp 2024-09-13T14:10:36.117 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /home 2024-09-13T14:10:36.117 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /root 2024-09-13T14:10:36.117 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /tmp 2024-09-13T14:10:36.117 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:10:37.099 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1627.gbbdadfa3.el9.x86_64 23/26 2024-09-13T14:10:37.100 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /sys 2024-09-13T14:10:37.100 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /proc 2024-09-13T14:10:37.100 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /mnt 2024-09-13T14:10:37.100 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /var/tmp 2024-09-13T14:10:37.100 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /home 2024-09-13T14:10:37.100 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /root 2024-09-13T14:10:37.100 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /tmp 2024-09-13T14:10:37.100 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:10:37.468 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-radosgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 24/26 2024-09-13T14:10:37.546 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 24/26 2024-09-13T14:10:37.546 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:10:37.547 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T14:10:37.547 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T14:10:37.547 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T14:10:37.547 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:10:37.548 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-13T14:10:37.654 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-13T14:10:37.654 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-13T14:10:38.368 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-radosgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 24/26 2024-09-13T14:10:38.431 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 24/26 2024-09-13T14:10:38.431 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:10:38.431 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T14:10:38.431 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T14:10:38.431 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T14:10:38.431 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:10:38.432 INFO:teuthology.orchestra.run.smithi184.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-13T14:10:38.538 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-13T14:10:38.538 INFO:teuthology.orchestra.run.smithi184.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-13T14:10:38.908 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-13T14:10:38.908 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-base-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/26 2024-09-13T14:10:38.908 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2/26 2024-09-13T14:10:38.908 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-radosgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 3/26 2024-09-13T14:10:38.909 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-selinux-2:17.2.7-1627.gbbdadfa3.el9.x86_64 4/26 2024-09-13T14:10:38.909 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephfs2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 5/26 2024-09-13T14:10:38.909 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libradosstriper1-2:17.2.7-1627.gbbdadfa3.el9.x86_6 6/26 2024-09-13T14:10:38.909 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librgw2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 7/26 2024-09-13T14:10:38.909 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1627.gbbdadfa3.el9. 8/26 2024-09-13T14:10:38.909 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x8 9/26 2024-09-13T14:10:38.909 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cephfs-2:17.2.7-1627.gbbdadfa3.el9.x86_64 10/26 2024-09-13T14:10:38.909 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rados-2:17.2.7-1627.gbbdadfa3.el9.x86_64 11/26 2024-09-13T14:10:38.909 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 12/26 2024-09-13T14:10:38.909 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 13/26 2024-09-13T14:10:38.909 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-13T14:10:38.909 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-13T14:10:38.909 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-13T14:10:38.910 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-13T14:10:38.910 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-13T14:10:38.910 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-13T14:10:38.910 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-13T14:10:38.910 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-13T14:10:38.910 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-13T14:10:38.910 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 23/26 2024-09-13T14:10:38.910 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-13T14:10:38.910 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librbd1-2:17.2.7-1627.gbbdadfa3.el9.x86_64 25/26 2024-09-13T14:10:39.808 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-13T14:10:39.808 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-base-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/26 2024-09-13T14:10:39.808 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2/26 2024-09-13T14:10:39.809 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-radosgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 3/26 2024-09-13T14:10:39.809 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-selinux-2:17.2.7-1627.gbbdadfa3.el9.x86_64 4/26 2024-09-13T14:10:39.809 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libcephfs2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 5/26 2024-09-13T14:10:39.809 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libradosstriper1-2:17.2.7-1627.gbbdadfa3.el9.x86_6 6/26 2024-09-13T14:10:39.809 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librgw2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 7/26 2024-09-13T14:10:39.809 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1627.gbbdadfa3.el9. 8/26 2024-09-13T14:10:39.809 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x8 9/26 2024-09-13T14:10:39.809 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cephfs-2:17.2.7-1627.gbbdadfa3.el9.x86_64 10/26 2024-09-13T14:10:39.809 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rados-2:17.2.7-1627.gbbdadfa3.el9.x86_64 11/26 2024-09-13T14:10:39.809 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 12/26 2024-09-13T14:10:39.809 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 13/26 2024-09-13T14:10:39.809 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-13T14:10:39.809 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-13T14:10:39.810 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-13T14:10:39.810 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-13T14:10:39.810 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-13T14:10:39.810 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-13T14:10:39.810 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-13T14:10:39.810 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-13T14:10:39.810 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-13T14:10:39.810 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librados2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 23/26 2024-09-13T14:10:39.810 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-13T14:10:39.810 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librbd1-2:17.2.7-1627.gbbdadfa3.el9.x86_64 25/26 2024-09-13T14:10:39.928 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-13T14:10:39.928 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:10:39.928 INFO:teuthology.orchestra.run.smithi151.stdout:Upgraded: 2024-09-13T14:10:39.928 INFO:teuthology.orchestra.run.smithi151.stdout: librados2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:39.928 INFO:teuthology.orchestra.run.smithi151.stdout: librbd1-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:39.928 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-09-13T14:10:39.928 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-base-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:39.929 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:39.929 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-radosgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:39.929 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-selinux-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:39.929 INFO:teuthology.orchestra.run.smithi151.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T14:10:39.929 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:39.929 INFO:teuthology.orchestra.run.smithi151.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T14:10:39.929 INFO:teuthology.orchestra.run.smithi151.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T14:10:39.929 INFO:teuthology.orchestra.run.smithi151.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T14:10:39.929 INFO:teuthology.orchestra.run.smithi151.stdout: libradosstriper1-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:39.929 INFO:teuthology.orchestra.run.smithi151.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T14:10:39.929 INFO:teuthology.orchestra.run.smithi151.stdout: librgw2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:39.929 INFO:teuthology.orchestra.run.smithi151.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T14:10:39.929 INFO:teuthology.orchestra.run.smithi151.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T14:10:39.930 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-argparse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:39.930 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:39.930 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cephfs-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:39.930 INFO:teuthology.orchestra.run.smithi151.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T14:10:39.930 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rados-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:39.930 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:39.930 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:39.930 INFO:teuthology.orchestra.run.smithi151.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T14:10:39.930 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:10:39.930 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-13T14:10:40.299 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-test 2024-09-13T14:10:40.698 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-13T14:10:40.698 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:10:40.698 INFO:teuthology.orchestra.run.smithi184.stdout:Upgraded: 2024-09-13T14:10:40.698 INFO:teuthology.orchestra.run.smithi184.stdout: librados2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:40.698 INFO:teuthology.orchestra.run.smithi184.stdout: librbd1-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:40.699 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-09-13T14:10:40.699 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-base-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:40.699 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:40.699 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-radosgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:40.699 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-selinux-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:40.699 INFO:teuthology.orchestra.run.smithi184.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T14:10:40.699 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:40.699 INFO:teuthology.orchestra.run.smithi184.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T14:10:40.699 INFO:teuthology.orchestra.run.smithi184.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T14:10:40.699 INFO:teuthology.orchestra.run.smithi184.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T14:10:40.699 INFO:teuthology.orchestra.run.smithi184.stdout: libradosstriper1-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:40.699 INFO:teuthology.orchestra.run.smithi184.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T14:10:40.699 INFO:teuthology.orchestra.run.smithi184.stdout: librgw2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:40.700 INFO:teuthology.orchestra.run.smithi184.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T14:10:40.700 INFO:teuthology.orchestra.run.smithi184.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T14:10:40.700 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-argparse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:40.700 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-common-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:40.700 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cephfs-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:40.700 INFO:teuthology.orchestra.run.smithi184.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T14:10:40.700 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rados-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:40.700 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:40.700 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:40.700 INFO:teuthology.orchestra.run.smithi184.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T14:10:40.700 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:10:40.700 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-09-13T14:10:40.864 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:27 ago on Fri 13 Sep 2024 02:10:13 PM UTC. 2024-09-13T14:10:40.997 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-13T14:10:40.997 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:10:40.997 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-09-13T14:10:40.997 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:10:40.997 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-09-13T14:10:40.998 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-test x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 48 M 2024-09-13T14:10:40.998 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-09-13T14:10:40.998 INFO:teuthology.orchestra.run.smithi151.stdout: libcephsqlite x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 163 k 2024-09-13T14:10:40.998 INFO:teuthology.orchestra.run.smithi151.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T14:10:40.998 INFO:teuthology.orchestra.run.smithi151.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T14:10:40.998 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:10:40.998 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-09-13T14:10:40.998 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:10:40.998 INFO:teuthology.orchestra.run.smithi151.stdout:Install 4 Packages 2024-09-13T14:10:40.998 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:10:40.999 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 48 M 2024-09-13T14:10:40.999 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 193 M 2024-09-13T14:10:40.999 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-09-13T14:10:41.060 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-test 2024-09-13T14:10:41.329 INFO:teuthology.orchestra.run.smithi151.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.5 MB/s | 304 kB 00:00 2024-09-13T14:10:41.354 INFO:teuthology.orchestra.run.smithi151.stdout:(2/4): libcephsqlite-17.2.7-1627.gbbdadfa3.el9. 726 kB/s | 163 kB 00:00 2024-09-13T14:10:41.379 INFO:teuthology.orchestra.run.smithi151.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-13T14:10:41.629 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:27 ago on Fri 13 Sep 2024 02:10:14 PM UTC. 2024-09-13T14:10:41.763 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-09-13T14:10:41.764 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:10:41.764 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-09-13T14:10:41.764 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:10:41.764 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-09-13T14:10:41.764 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-test x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 48 M 2024-09-13T14:10:41.764 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-09-13T14:10:41.764 INFO:teuthology.orchestra.run.smithi184.stdout: libcephsqlite x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 163 k 2024-09-13T14:10:41.764 INFO:teuthology.orchestra.run.smithi184.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T14:10:41.765 INFO:teuthology.orchestra.run.smithi184.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T14:10:41.765 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:10:41.765 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-09-13T14:10:41.765 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:10:41.765 INFO:teuthology.orchestra.run.smithi184.stdout:Install 4 Packages 2024-09-13T14:10:41.765 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:10:41.765 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 48 M 2024-09-13T14:10:41.766 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 193 M 2024-09-13T14:10:41.766 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-09-13T14:10:41.958 INFO:teuthology.orchestra.run.smithi184.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.8 MB/s | 304 kB 00:00 2024-09-13T14:10:42.033 INFO:teuthology.orchestra.run.smithi184.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 851 kB/s | 64 kB 00:00 2024-09-13T14:10:42.225 INFO:teuthology.orchestra.run.smithi184.stdout:(3/4): libcephsqlite-17.2.7-1627.gbbdadfa3.el9. 374 kB/s | 163 kB 00:00 2024-09-13T14:10:42.671 INFO:teuthology.orchestra.run.smithi151.stdout:(4/4): ceph-test-17.2.7-1627.gbbdadfa3.el9.x86_ 31 MB/s | 48 MB 00:01 2024-09-13T14:10:42.674 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:10:42.674 INFO:teuthology.orchestra.run.smithi151.stdout:Total 29 MB/s | 48 MB 00:01 2024-09-13T14:10:42.702 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-09-13T14:10:42.721 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-09-13T14:10:42.721 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-09-13T14:10:43.088 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-09-13T14:10:43.089 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-09-13T14:10:43.625 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-09-13T14:10:43.721 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-13T14:10:43.777 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-13T14:10:43.813 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephsqlite-2:17.2.7-1627.gbbdadfa3.el9.x86_64 3/4 2024-09-13T14:10:44.167 INFO:teuthology.orchestra.run.smithi184.stdout:(4/4): ceph-test-17.2.7-1627.gbbdadfa3.el9.x86_ 20 MB/s | 48 MB 00:02 2024-09-13T14:10:44.169 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:10:44.169 INFO:teuthology.orchestra.run.smithi184.stdout:Total 20 MB/s | 48 MB 00:02 2024-09-13T14:10:44.196 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-09-13T14:10:44.215 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-09-13T14:10:44.215 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-09-13T14:10:44.584 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-09-13T14:10:44.584 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-09-13T14:10:45.101 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-09-13T14:10:45.184 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-13T14:10:45.235 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-13T14:10:45.260 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libcephsqlite-2:17.2.7-1627.gbbdadfa3.el9.x86_64 3/4 2024-09-13T14:10:48.207 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1627.gbbdadfa3.el9.x86_64 3/4 2024-09-13T14:10:48.306 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-test-2:17.2.7-1627.gbbdadfa3.el9.x86_64 4/4 2024-09-13T14:10:49.653 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1627.gbbdadfa3.el9.x86_64 3/4 2024-09-13T14:10:49.728 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-test-2:17.2.7-1627.gbbdadfa3.el9.x86_64 4/4 2024-09-13T14:10:50.562 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-test-2:17.2.7-1627.gbbdadfa3.el9.x86_64 4/4 2024-09-13T14:10:50.562 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-test-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/4 2024-09-13T14:10:50.562 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephsqlite-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2/4 2024-09-13T14:10:50.562 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-13T14:10:50.864 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-13T14:10:50.864 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:10:50.864 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-09-13T14:10:50.864 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-test-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:50.864 INFO:teuthology.orchestra.run.smithi151.stdout: libcephsqlite-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:50.864 INFO:teuthology.orchestra.run.smithi151.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T14:10:50.864 INFO:teuthology.orchestra.run.smithi151.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T14:10:50.864 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:10:50.864 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-13T14:10:51.132 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph 2024-09-13T14:10:51.694 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:38 ago on Fri 13 Sep 2024 02:10:13 PM UTC. 2024-09-13T14:10:51.833 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-13T14:10:51.836 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:10:51.836 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-09-13T14:10:51.836 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:10:51.836 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-09-13T14:10:51.836 INFO:teuthology.orchestra.run.smithi151.stdout: ceph x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 6.5 k 2024-09-13T14:10:51.836 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-09-13T14:10:51.836 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mds x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 2.1 M 2024-09-13T14:10:51.836 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 1.4 M 2024-09-13T14:10:51.836 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 240 k 2024-09-13T14:10:51.836 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mon x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 3.9 M 2024-09-13T14:10:51.836 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-osd x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 15 M 2024-09-13T14:10:51.837 INFO:teuthology.orchestra.run.smithi151.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T14:10:51.837 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T14:10:51.837 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T14:10:51.837 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T14:10:51.837 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T14:10:51.837 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T14:10:51.837 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T14:10:51.837 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-collections 2024-09-13T14:10:51.837 INFO:teuthology.orchestra.run.smithi151.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-13T14:10:51.837 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T14:10:51.837 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T14:10:51.837 INFO:teuthology.orchestra.run.smithi151.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T14:10:51.837 INFO:teuthology.orchestra.run.smithi151.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T14:10:51.838 INFO:teuthology.orchestra.run.smithi151.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T14:10:51.838 INFO:teuthology.orchestra.run.smithi151.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T14:10:51.838 INFO:teuthology.orchestra.run.smithi151.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T14:10:51.838 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T14:10:51.838 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T14:10:51.838 INFO:teuthology.orchestra.run.smithi151.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T14:10:51.838 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T14:10:51.838 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T14:10:51.838 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T14:10:51.838 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T14:10:51.838 INFO:teuthology.orchestra.run.smithi151.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T14:10:51.838 INFO:teuthology.orchestra.run.smithi151.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T14:10:51.838 INFO:teuthology.orchestra.run.smithi151.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T14:10:51.839 INFO:teuthology.orchestra.run.smithi151.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T14:10:51.839 INFO:teuthology.orchestra.run.smithi151.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T14:10:51.839 INFO:teuthology.orchestra.run.smithi151.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T14:10:51.839 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:10:51.839 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-09-13T14:10:51.839 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:10:51.839 INFO:teuthology.orchestra.run.smithi151.stdout:Install 34 Packages 2024-09-13T14:10:51.839 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:10:51.840 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 27 M 2024-09-13T14:10:51.840 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 96 M 2024-09-13T14:10:51.840 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-09-13T14:10:51.978 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-test-2:17.2.7-1627.gbbdadfa3.el9.x86_64 4/4 2024-09-13T14:10:51.978 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-test-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/4 2024-09-13T14:10:51.978 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libcephsqlite-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2/4 2024-09-13T14:10:51.978 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-13T14:10:52.268 INFO:teuthology.orchestra.run.smithi151.stdout:(1/34): ceph-17.2.7-1627.gbbdadfa3.el9.x86_64.r 54 kB/s | 6.5 kB 00:00 2024-09-13T14:10:52.300 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-13T14:10:52.300 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:10:52.300 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-09-13T14:10:52.300 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-test-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:52.300 INFO:teuthology.orchestra.run.smithi184.stdout: libcephsqlite-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:10:52.300 INFO:teuthology.orchestra.run.smithi184.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T14:10:52.300 INFO:teuthology.orchestra.run.smithi184.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T14:10:52.300 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:10:52.300 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-09-13T14:10:52.452 INFO:teuthology.orchestra.run.smithi151.stdout:(2/34): ceph-mgr-17.2.7-1627.gbbdadfa3.el9.x86_ 4.6 MB/s | 1.4 MB 00:00 2024-09-13T14:10:52.515 INFO:teuthology.orchestra.run.smithi151.stdout:(3/34): ceph-mds-17.2.7-1627.gbbdadfa3.el9.x86_ 5.6 MB/s | 2.1 MB 00:00 2024-09-13T14:10:52.607 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph 2024-09-13T14:10:52.607 INFO:teuthology.orchestra.run.smithi151.stdout:(4/34): ceph-mon-17.2.7-1627.gbbdadfa3.el9.x86_ 12 MB/s | 3.9 MB 00:00 2024-09-13T14:10:52.633 INFO:teuthology.orchestra.run.smithi151.stdout:(5/34): ceph-mgr-modules-core-17.2.7-1627.gbbda 2.0 MB/s | 240 kB 00:00 2024-09-13T14:10:52.783 INFO:teuthology.orchestra.run.smithi151.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 253 kB 00:00 2024-09-13T14:10:52.836 INFO:teuthology.orchestra.run.smithi151.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-13T14:10:53.001 INFO:teuthology.orchestra.run.smithi151.stdout:(8/34): ceph-osd-17.2.7-1627.gbbdadfa3.el9.x86_ 28 MB/s | 15 MB 00:00 2024-09-13T14:10:53.026 INFO:teuthology.orchestra.run.smithi151.stdout:(9/34): python3-pycparser-2.20-6.el9.noarch.rpm 704 kB/s | 135 kB 00:00 2024-09-13T14:10:53.068 INFO:teuthology.orchestra.run.smithi151.stdout:(10/34): python3-cryptography-36.0.1-4.el9.x86_ 2.8 MB/s | 1.2 MB 00:00 2024-09-13T14:10:53.093 INFO:teuthology.orchestra.run.smithi151.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-09-13T14:10:53.152 INFO:teuthology.orchestra.run.smithi151.stdout:(12/34): python3-requests-2.25.1-8.el9.noarch.r 827 kB/s | 125 kB 00:00 2024-09-13T14:10:53.176 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:39 ago on Fri 13 Sep 2024 02:10:14 PM UTC. 2024-09-13T14:10:53.177 INFO:teuthology.orchestra.run.smithi151.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 415 kB/s | 35 kB 00:00 2024-09-13T14:10:53.203 INFO:teuthology.orchestra.run.smithi151.stdout:(14/34): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-09-13T14:10:53.228 INFO:teuthology.orchestra.run.smithi151.stdout:(15/34): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-13T14:10:53.253 INFO:teuthology.orchestra.run.smithi151.stdout:(16/34): python3-toml-0.10.2-6.el9.noarch.rpm 830 kB/s | 42 kB 00:00 2024-09-13T14:10:53.279 INFO:teuthology.orchestra.run.smithi151.stdout:(17/34): python3-packaging-20.9-5.el9.noarch.rp 609 kB/s | 77 kB 00:00 2024-09-13T14:10:53.312 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-09-13T14:10:53.314 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:10:53.314 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-09-13T14:10:53.314 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:10:53.314 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-09-13T14:10:53.315 INFO:teuthology.orchestra.run.smithi184.stdout: ceph x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 6.5 k 2024-09-13T14:10:53.315 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-09-13T14:10:53.315 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mds x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 2.1 M 2024-09-13T14:10:53.315 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 1.4 M 2024-09-13T14:10:53.315 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 240 k 2024-09-13T14:10:53.315 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mon x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 3.9 M 2024-09-13T14:10:53.315 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-osd x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 15 M 2024-09-13T14:10:53.315 INFO:teuthology.orchestra.run.smithi184.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T14:10:53.315 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T14:10:53.315 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T14:10:53.315 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T14:10:53.316 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T14:10:53.316 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T14:10:53.316 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T14:10:53.316 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-collections 2024-09-13T14:10:53.316 INFO:teuthology.orchestra.run.smithi184.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-13T14:10:53.316 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T14:10:53.316 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T14:10:53.316 INFO:teuthology.orchestra.run.smithi184.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T14:10:53.316 INFO:teuthology.orchestra.run.smithi184.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T14:10:53.316 INFO:teuthology.orchestra.run.smithi184.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T14:10:53.316 INFO:teuthology.orchestra.run.smithi184.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T14:10:53.316 INFO:teuthology.orchestra.run.smithi184.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T14:10:53.316 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T14:10:53.317 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T14:10:53.317 INFO:teuthology.orchestra.run.smithi184.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T14:10:53.317 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T14:10:53.317 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T14:10:53.317 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T14:10:53.317 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T14:10:53.317 INFO:teuthology.orchestra.run.smithi184.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T14:10:53.317 INFO:teuthology.orchestra.run.smithi184.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T14:10:53.317 INFO:teuthology.orchestra.run.smithi184.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T14:10:53.317 INFO:teuthology.orchestra.run.smithi184.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T14:10:53.317 INFO:teuthology.orchestra.run.smithi184.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T14:10:53.317 INFO:teuthology.orchestra.run.smithi184.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T14:10:53.317 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:10:53.317 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-09-13T14:10:53.318 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:10:53.318 INFO:teuthology.orchestra.run.smithi184.stdout:Install 34 Packages 2024-09-13T14:10:53.318 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:10:53.318 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 27 M 2024-09-13T14:10:53.318 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 96 M 2024-09-13T14:10:53.318 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-09-13T14:10:53.321 INFO:teuthology.orchestra.run.smithi151.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 470 kB/s | 43 kB 00:00 2024-09-13T14:10:53.346 INFO:teuthology.orchestra.run.smithi151.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 457 kB/s | 11 kB 00:00 2024-09-13T14:10:53.372 INFO:teuthology.orchestra.run.smithi151.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-09-13T14:10:53.389 INFO:teuthology.orchestra.run.smithi151.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 421 kB/s | 18 kB 00:00 2024-09-13T14:10:53.422 INFO:teuthology.orchestra.run.smithi151.stdout:(22/34): python3-cherrypy-18.6.1-2.el9.noarch.r 2.7 MB/s | 358 kB 00:00 2024-09-13T14:10:53.439 INFO:teuthology.orchestra.run.smithi151.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-09-13T14:10:53.456 INFO:teuthology.orchestra.run.smithi151.stdout:(24/34): python3-jaraco-collections-3.0.0-8.el9 344 kB/s | 23 kB 00:00 2024-09-13T14:10:53.473 INFO:teuthology.orchestra.run.smithi151.stdout:(25/34): python3-jaraco-text-3.2.0-6.el9.noarch 389 kB/s | 20 kB 00:00 2024-09-13T14:10:53.490 INFO:teuthology.orchestra.run.smithi151.stdout:(26/34): python3-logutils-0.3.5-21.el9.noarch.r 912 kB/s | 46 kB 00:00 2024-09-13T14:10:53.516 INFO:teuthology.orchestra.run.smithi151.stdout:(27/34): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-09-13T14:10:53.533 INFO:teuthology.orchestra.run.smithi151.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 390 kB/s | 16 kB 00:00 2024-09-13T14:10:53.558 INFO:teuthology.orchestra.run.smithi151.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-09-13T14:10:53.575 INFO:teuthology.orchestra.run.smithi151.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 848 kB/s | 36 kB 00:00 2024-09-13T14:10:53.601 INFO:teuthology.orchestra.run.smithi151.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-09-13T14:10:53.634 INFO:teuthology.orchestra.run.smithi151.stdout:(32/34): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-13T14:10:53.651 INFO:teuthology.orchestra.run.smithi151.stdout:(33/34): python3-zc-lockfile-2.0-10.el9.noarch. 398 kB/s | 20 kB 00:00 2024-09-13T14:10:53.685 INFO:teuthology.orchestra.run.smithi151.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-13T14:10:53.688 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:10:53.688 INFO:teuthology.orchestra.run.smithi151.stdout:Total 15 MB/s | 27 MB 00:01 2024-09-13T14:10:53.813 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-09-13T14:10:53.849 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-09-13T14:10:53.850 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-09-13T14:10:53.862 INFO:teuthology.orchestra.run.smithi184.stdout:(1/34): ceph-17.2.7-1627.gbbdadfa3.el9.x86_64.r 52 kB/s | 6.5 kB 00:00 2024-09-13T14:10:54.054 INFO:teuthology.orchestra.run.smithi184.stdout:(2/34): ceph-mgr-17.2.7-1627.gbbdadfa3.el9.x86_ 4.4 MB/s | 1.4 MB 00:00 2024-09-13T14:10:54.121 INFO:teuthology.orchestra.run.smithi184.stdout:(3/34): ceph-mds-17.2.7-1627.gbbdadfa3.el9.x86_ 5.4 MB/s | 2.1 MB 00:00 2024-09-13T14:10:54.227 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-09-13T14:10:54.228 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-09-13T14:10:54.238 INFO:teuthology.orchestra.run.smithi184.stdout:(4/34): ceph-mon-17.2.7-1627.gbbdadfa3.el9.x86_ 10 MB/s | 3.9 MB 00:00 2024-09-13T14:10:54.263 INFO:teuthology.orchestra.run.smithi184.stdout:(5/34): ceph-mgr-modules-core-17.2.7-1627.gbbda 1.7 MB/s | 240 kB 00:00 2024-09-13T14:10:54.589 INFO:teuthology.orchestra.run.smithi184.stdout:(6/34): ceph-osd-17.2.7-1627.gbbdadfa3.el9.x86_ 29 MB/s | 15 MB 00:00 2024-09-13T14:10:54.881 INFO:teuthology.orchestra.run.smithi184.stdout:(7/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 394 kB/s | 253 kB 00:00 2024-09-13T14:10:54.998 INFO:teuthology.orchestra.run.smithi184.stdout:(8/34): python3-pycparser-2.20-6.el9.noarch.rpm 1.1 MB/s | 135 kB 00:00 2024-09-13T14:10:55.048 INFO:teuthology.orchestra.run.smithi184.stdout:(9/34): python3-ply-3.11-14.el9.noarch.rpm 232 kB/s | 106 kB 00:00 2024-09-13T14:10:55.106 INFO:teuthology.orchestra.run.smithi184.stdout:(10/34): python3-requests-2.25.1-8.el9.noarch.r 1.1 MB/s | 125 kB 00:00 2024-09-13T14:10:55.215 INFO:teuthology.orchestra.run.smithi184.stdout:(11/34): python3-cryptography-36.0.1-4.el9.x86_ 1.3 MB/s | 1.2 MB 00:00 2024-09-13T14:10:55.249 INFO:teuthology.orchestra.run.smithi184.stdout:(12/34): python3-urllib3-1.26.5-6.el9.noarch.rp 1.0 MB/s | 215 kB 00:00 2024-09-13T14:10:55.299 INFO:teuthology.orchestra.run.smithi184.stdout:(13/34): python3-mako-1.1.4-6.el9.noarch.rpm 894 kB/s | 172 kB 00:00 2024-09-13T14:10:55.325 INFO:teuthology.orchestra.run.smithi184.stdout:(14/34): python3-markupsafe-1.1.1-12.el9.x86_64 319 kB/s | 35 kB 00:00 2024-09-13T14:10:55.358 INFO:teuthology.orchestra.run.smithi184.stdout:(15/34): python3-pytz-2021.1-5.el9.noarch.rpm 869 kB/s | 51 kB 00:00 2024-09-13T14:10:55.400 INFO:teuthology.orchestra.run.smithi184.stdout:(16/34): python3-toml-0.10.2-6.el9.noarch.rpm 553 kB/s | 42 kB 00:00 2024-09-13T14:10:55.425 INFO:teuthology.orchestra.run.smithi184.stdout:(17/34): python3-packaging-20.9-5.el9.noarch.rp 438 kB/s | 77 kB 00:00 2024-09-13T14:10:55.459 INFO:teuthology.orchestra.run.smithi184.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 433 kB/s | 43 kB 00:00 2024-09-13T14:10:55.460 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-09-13T14:10:55.484 INFO:teuthology.orchestra.run.smithi184.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 708 kB/s | 11 kB 00:00 2024-09-13T14:10:55.501 INFO:teuthology.orchestra.run.smithi184.stdout:(20/34): python3-jaraco-classes-3.2.1-5.el9.noa 1.2 MB/s | 18 kB 00:00 2024-09-13T14:10:55.526 INFO:teuthology.orchestra.run.smithi184.stdout:(21/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-13T14:10:55.538 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-13T14:10:55.543 INFO:teuthology.orchestra.run.smithi184.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 554 kB/s | 23 kB 00:00 2024-09-13T14:10:55.560 INFO:teuthology.orchestra.run.smithi184.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 581 kB/s | 19 kB 00:00 2024-09-13T14:10:55.577 INFO:teuthology.orchestra.run.smithi184.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 590 kB/s | 20 kB 00:00 2024-09-13T14:10:55.579 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-13T14:10:55.620 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-13T14:10:55.644 INFO:teuthology.orchestra.run.smithi184.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 553 kB/s | 46 kB 00:00 2024-09-13T14:10:55.671 INFO:teuthology.orchestra.run.smithi184.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 857 kB/s | 79 kB 00:00 2024-09-13T14:10:55.695 INFO:teuthology.orchestra.run.smithi184.stdout:(27/34): python3-portend-3.1.0-2.el9.noarch.rpm 662 kB/s | 16 kB 00:00 2024-09-13T14:10:55.721 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-13T14:10:55.737 INFO:teuthology.orchestra.run.smithi184.stdout:(28/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.2 MB/s | 358 kB 00:00 2024-09-13T14:10:55.762 INFO:teuthology.orchestra.run.smithi184.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-09-13T14:10:55.787 INFO:teuthology.orchestra.run.smithi184.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-09-13T14:10:55.808 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-13T14:10:55.813 INFO:teuthology.orchestra.run.smithi184.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-09-13T14:10:55.838 INFO:teuthology.orchestra.run.smithi184.stdout:(32/34): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-13T14:10:55.857 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-13T14:10:55.863 INFO:teuthology.orchestra.run.smithi184.stdout:(33/34): python3-zc-lockfile-2.0-10.el9.noarch. 398 kB/s | 20 kB 00:00 2024-09-13T14:10:55.888 INFO:teuthology.orchestra.run.smithi184.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-13T14:10:55.893 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:10:55.893 INFO:teuthology.orchestra.run.smithi184.stdout:Total 11 MB/s | 27 MB 00:02 2024-09-13T14:10:55.901 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-13T14:10:55.930 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-13T14:10:55.965 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-13T14:10:55.999 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-13T14:10:56.026 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-09-13T14:10:56.040 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-13T14:10:56.064 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-09-13T14:10:56.064 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-09-13T14:10:56.083 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-13T14:10:56.118 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-13T14:10:56.166 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-13T14:10:56.199 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-13T14:10:56.281 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-13T14:10:56.426 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-09-13T14:10:56.427 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-09-13T14:10:56.445 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-13T14:10:56.533 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-13T14:10:56.587 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-13T14:10:56.628 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-13T14:10:56.691 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-13T14:10:56.746 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-13T14:10:56.932 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-13T14:10:56.997 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-13T14:10:57.067 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-13T14:10:57.165 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-13T14:10:57.205 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-13T14:10:57.321 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-13T14:10:57.468 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1627.gbbdadfa3.el9. 29/34 2024-09-13T14:10:57.500 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-2:17.2.7-1627.gbbdadfa3.el9.x86_64 30/34 2024-09-13T14:10:57.525 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1627.gbbdadfa3.el9.x86_64 30/34 2024-09-13T14:10:57.525 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:10:57.525 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T14:10:57.525 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T14:10:57.525 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T14:10:57.526 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:10:57.704 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-09-13T14:10:57.779 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-13T14:10:57.824 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-13T14:10:57.857 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-13T14:10:57.947 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-13T14:10:58.043 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-13T14:10:58.092 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-13T14:10:58.127 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-13T14:10:58.157 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-13T14:10:58.193 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-13T14:10:58.227 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-13T14:10:58.268 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-13T14:10:58.302 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-13T14:10:58.345 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-13T14:10:58.393 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-13T14:10:58.425 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-13T14:10:58.498 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-13T14:10:58.687 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-13T14:10:58.767 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-13T14:10:58.825 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-13T14:10:58.875 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-13T14:10:58.885 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-osd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 31/34 2024-09-13T14:10:58.914 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-osd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 31/34 2024-09-13T14:10:58.914 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:10:58.914 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T14:10:58.914 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T14:10:58.914 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T14:10:58.914 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:10:58.924 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-13T14:10:58.979 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-13T14:10:59.164 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-13T14:10:59.221 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-13T14:10:59.286 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mon-2:17.2.7-1627.gbbdadfa3.el9.x86_64 32/34 2024-09-13T14:10:59.299 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-13T14:10:59.310 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mon-2:17.2.7-1627.gbbdadfa3.el9.x86_64 32/34 2024-09-13T14:10:59.310 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:10:59.310 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T14:10:59.310 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T14:10:59.310 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T14:10:59.310 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:10:59.412 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-13T14:10:59.453 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-13T14:10:59.503 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mds-2:17.2.7-1627.gbbdadfa3.el9.x86_64 33/34 2024-09-13T14:10:59.527 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mds-2:17.2.7-1627.gbbdadfa3.el9.x86_64 33/34 2024-09-13T14:10:59.527 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:10:59.527 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T14:10:59.528 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T14:10:59.528 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T14:10:59.528 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:10:59.567 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-13T14:10:59.592 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-2:17.2.7-1627.gbbdadfa3.el9.x86_64 34/34 2024-09-13T14:10:59.725 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1627.gbbdadfa3.el9. 29/34 2024-09-13T14:10:59.756 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-2:17.2.7-1627.gbbdadfa3.el9.x86_64 30/34 2024-09-13T14:10:59.781 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1627.gbbdadfa3.el9.x86_64 30/34 2024-09-13T14:10:59.782 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:10:59.782 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T14:10:59.782 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T14:10:59.782 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T14:10:59.782 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:01.168 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-osd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 31/34 2024-09-13T14:11:01.198 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-osd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 31/34 2024-09-13T14:11:01.198 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:11:01.198 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T14:11:01.198 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T14:11:01.199 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T14:11:01.199 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:01.322 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-2:17.2.7-1627.gbbdadfa3.el9.x86_64 34/34 2024-09-13T14:11:01.322 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/34 2024-09-13T14:11:01.322 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mds-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2/34 2024-09-13T14:11:01.323 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-2:17.2.7-1627.gbbdadfa3.el9.x86_64 3/34 2024-09-13T14:11:01.323 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mon-2:17.2.7-1627.gbbdadfa3.el9.x86_64 4/34 2024-09-13T14:11:01.323 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-osd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 5/34 2024-09-13T14:11:01.323 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1627.gbbdadfa3.el9. 6/34 2024-09-13T14:11:01.323 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-13T14:11:01.323 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-13T14:11:01.323 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-13T14:11:01.323 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-13T14:11:01.323 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-13T14:11:01.324 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-13T14:11:01.324 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-13T14:11:01.324 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-13T14:11:01.324 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-13T14:11:01.324 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-13T14:11:01.324 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-13T14:11:01.324 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-13T14:11:01.324 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-13T14:11:01.324 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-13T14:11:01.325 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-13T14:11:01.325 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-13T14:11:01.325 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-13T14:11:01.325 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-13T14:11:01.325 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-13T14:11:01.325 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-13T14:11:01.325 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-13T14:11:01.325 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-13T14:11:01.325 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-13T14:11:01.325 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-13T14:11:01.326 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-13T14:11:01.326 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-13T14:11:01.326 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-13T14:11:01.568 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mon-2:17.2.7-1627.gbbdadfa3.el9.x86_64 32/34 2024-09-13T14:11:01.592 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mon-2:17.2.7-1627.gbbdadfa3.el9.x86_64 32/34 2024-09-13T14:11:01.592 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:11:01.593 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T14:11:01.593 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T14:11:01.593 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T14:11:01.593 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:01.794 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mds-2:17.2.7-1627.gbbdadfa3.el9.x86_64 33/34 2024-09-13T14:11:01.818 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mds-2:17.2.7-1627.gbbdadfa3.el9.x86_64 33/34 2024-09-13T14:11:01.818 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:11:01.818 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T14:11:01.818 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T14:11:01.818 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T14:11:01.818 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:01.882 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-2:17.2.7-1627.gbbdadfa3.el9.x86_64 34/34 2024-09-13T14:11:02.250 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-13T14:11:02.250 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:02.250 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-09-13T14:11:02.250 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:11:02.251 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mds-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:11:02.251 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:11:02.251 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-modules-core-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:11:02.251 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mon-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:11:02.251 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-osd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:11:02.251 INFO:teuthology.orchestra.run.smithi151.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T14:11:02.251 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T14:11:02.251 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T14:11:02.251 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T14:11:02.251 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T14:11:02.251 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T14:11:02.251 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T14:11:02.251 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T14:11:02.252 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T14:11:02.252 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T14:11:02.252 INFO:teuthology.orchestra.run.smithi151.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T14:11:02.252 INFO:teuthology.orchestra.run.smithi151.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T14:11:02.252 INFO:teuthology.orchestra.run.smithi151.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T14:11:02.252 INFO:teuthology.orchestra.run.smithi151.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T14:11:02.252 INFO:teuthology.orchestra.run.smithi151.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T14:11:02.252 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T14:11:02.252 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T14:11:02.252 INFO:teuthology.orchestra.run.smithi151.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T14:11:02.252 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T14:11:02.252 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T14:11:02.252 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T14:11:02.253 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T14:11:02.253 INFO:teuthology.orchestra.run.smithi151.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T14:11:02.253 INFO:teuthology.orchestra.run.smithi151.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T14:11:02.253 INFO:teuthology.orchestra.run.smithi151.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T14:11:02.253 INFO:teuthology.orchestra.run.smithi151.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T14:11:02.253 INFO:teuthology.orchestra.run.smithi151.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T14:11:02.253 INFO:teuthology.orchestra.run.smithi151.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T14:11:02.253 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:02.253 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-13T14:11:02.655 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-base 2024-09-13T14:11:03.231 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:50 ago on Fri 13 Sep 2024 02:10:13 PM UTC. 2024-09-13T14:11:03.321 INFO:teuthology.orchestra.run.smithi151.stdout:Package ceph-base-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:11:03.363 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-13T14:11:03.364 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-09-13T14:11:03.364 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-13T14:11:03.435 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install cephadm 2024-09-13T14:11:03.560 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-2:17.2.7-1627.gbbdadfa3.el9.x86_64 34/34 2024-09-13T14:11:03.560 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/34 2024-09-13T14:11:03.561 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mds-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2/34 2024-09-13T14:11:03.561 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-2:17.2.7-1627.gbbdadfa3.el9.x86_64 3/34 2024-09-13T14:11:03.561 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mon-2:17.2.7-1627.gbbdadfa3.el9.x86_64 4/34 2024-09-13T14:11:03.561 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-osd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 5/34 2024-09-13T14:11:03.561 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1627.gbbdadfa3.el9. 6/34 2024-09-13T14:11:03.561 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-13T14:11:03.561 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-13T14:11:03.561 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-13T14:11:03.561 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-13T14:11:03.561 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-13T14:11:03.561 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-13T14:11:03.561 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-13T14:11:03.562 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-13T14:11:03.562 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-13T14:11:03.562 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-13T14:11:03.562 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-13T14:11:03.562 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-13T14:11:03.562 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-13T14:11:03.562 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-13T14:11:03.562 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-13T14:11:03.562 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-13T14:11:03.562 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-13T14:11:03.562 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-13T14:11:03.562 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-13T14:11:03.562 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-13T14:11:03.563 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-13T14:11:03.563 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-13T14:11:03.563 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-13T14:11:03.563 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-13T14:11:03.563 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-13T14:11:03.563 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-13T14:11:03.563 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-13T14:11:03.993 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:50 ago on Fri 13 Sep 2024 02:10:13 PM UTC. 2024-09-13T14:11:04.128 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-13T14:11:04.129 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:04.129 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-09-13T14:11:04.129 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:04.129 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-09-13T14:11:04.130 INFO:teuthology.orchestra.run.smithi151.stdout: cephadm noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 83 k 2024-09-13T14:11:04.130 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:04.130 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-09-13T14:11:04.130 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:04.130 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-09-13T14:11:04.130 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:04.130 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 83 k 2024-09-13T14:11:04.130 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 375 k 2024-09-13T14:11:04.130 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-09-13T14:11:04.324 INFO:teuthology.orchestra.run.smithi151.stdout:cephadm-17.2.7-1627.gbbdadfa3.el9.noarch.rpm 427 kB/s | 83 kB 00:00 2024-09-13T14:11:04.325 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:11:04.325 INFO:teuthology.orchestra.run.smithi151.stdout:Total 423 kB/s | 83 kB 00:00 2024-09-13T14:11:04.327 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-09-13T14:11:04.330 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-09-13T14:11:04.330 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-09-13T14:11:04.332 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-09-13T14:11:04.332 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-09-13T14:11:04.410 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-09-13T14:11:04.455 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-13T14:11:04.455 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:04.455 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-09-13T14:11:04.455 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:11:04.455 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mds-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:11:04.456 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:11:04.456 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-modules-core-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:11:04.456 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mon-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:11:04.456 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-osd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:11:04.456 INFO:teuthology.orchestra.run.smithi184.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T14:11:04.456 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T14:11:04.457 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T14:11:04.457 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T14:11:04.457 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T14:11:04.457 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T14:11:04.457 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T14:11:04.458 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T14:11:04.458 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T14:11:04.458 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T14:11:04.458 INFO:teuthology.orchestra.run.smithi184.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T14:11:04.458 INFO:teuthology.orchestra.run.smithi184.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T14:11:04.458 INFO:teuthology.orchestra.run.smithi184.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T14:11:04.458 INFO:teuthology.orchestra.run.smithi184.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T14:11:04.458 INFO:teuthology.orchestra.run.smithi184.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T14:11:04.458 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T14:11:04.459 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T14:11:04.459 INFO:teuthology.orchestra.run.smithi184.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T14:11:04.459 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T14:11:04.459 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T14:11:04.459 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T14:11:04.459 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T14:11:04.459 INFO:teuthology.orchestra.run.smithi184.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T14:11:04.459 INFO:teuthology.orchestra.run.smithi184.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T14:11:04.459 INFO:teuthology.orchestra.run.smithi184.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T14:11:04.459 INFO:teuthology.orchestra.run.smithi184.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T14:11:04.459 INFO:teuthology.orchestra.run.smithi184.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T14:11:04.459 INFO:teuthology.orchestra.run.smithi184.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T14:11:04.459 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:04.460 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-09-13T14:11:04.866 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 1/1 2024-09-13T14:11:04.893 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-base 2024-09-13T14:11:04.970 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 1/1 2024-09-13T14:11:05.319 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 1/1 2024-09-13T14:11:05.473 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:51 ago on Fri 13 Sep 2024 02:10:14 PM UTC. 2024-09-13T14:11:05.548 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 1/1 2024-09-13T14:11:05.548 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:05.548 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-09-13T14:11:05.548 INFO:teuthology.orchestra.run.smithi151.stdout: cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:11:05.548 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:05.549 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-13T14:11:05.563 INFO:teuthology.orchestra.run.smithi184.stdout:Package ceph-base-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:11:05.605 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-09-13T14:11:05.606 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-09-13T14:11:05.606 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-09-13T14:11:05.666 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install cephadm 2024-09-13T14:11:05.755 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-immutable-object-cache 2024-09-13T14:11:06.230 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:52 ago on Fri 13 Sep 2024 02:10:14 PM UTC. 2024-09-13T14:11:06.324 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:53 ago on Fri 13 Sep 2024 02:10:13 PM UTC. 2024-09-13T14:11:06.361 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-09-13T14:11:06.362 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:06.362 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-09-13T14:11:06.362 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:06.362 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-09-13T14:11:06.362 INFO:teuthology.orchestra.run.smithi184.stdout: cephadm noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 83 k 2024-09-13T14:11:06.362 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:06.362 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-09-13T14:11:06.362 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:06.362 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-09-13T14:11:06.363 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:06.363 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 83 k 2024-09-13T14:11:06.363 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 375 k 2024-09-13T14:11:06.363 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-09-13T14:11:06.458 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-13T14:11:06.459 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:06.459 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-09-13T14:11:06.459 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:06.459 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-09-13T14:11:06.459 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 137 k 2024-09-13T14:11:06.459 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:06.459 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-09-13T14:11:06.459 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:06.459 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-09-13T14:11:06.459 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:06.459 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 137 k 2024-09-13T14:11:06.460 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 379 k 2024-09-13T14:11:06.460 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-09-13T14:11:06.560 INFO:teuthology.orchestra.run.smithi184.stdout:cephadm-17.2.7-1627.gbbdadfa3.el9.noarch.rpm 420 kB/s | 83 kB 00:00 2024-09-13T14:11:06.561 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:11:06.561 INFO:teuthology.orchestra.run.smithi184.stdout:Total 417 kB/s | 83 kB 00:00 2024-09-13T14:11:06.561 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-09-13T14:11:06.566 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-09-13T14:11:06.566 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-09-13T14:11:06.567 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-09-13T14:11:06.568 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-09-13T14:11:06.637 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-09-13T14:11:06.685 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-immutable-object-cache-17.2.7-1627.gbbdadf 610 kB/s | 137 kB 00:00 2024-09-13T14:11:06.686 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:11:06.686 INFO:teuthology.orchestra.run.smithi151.stdout:Total 606 kB/s | 137 kB 00:00 2024-09-13T14:11:06.686 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-09-13T14:11:06.693 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-09-13T14:11:06.693 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-09-13T14:11:06.751 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-09-13T14:11:06.751 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-09-13T14:11:06.898 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-09-13T14:11:06.986 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1627.gbbdadfa3. 1/1 2024-09-13T14:11:07.011 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1627.gbbdadfa3. 1/1 2024-09-13T14:11:07.012 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:11:07.012 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T14:11:07.012 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:07.110 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 1/1 2024-09-13T14:11:07.204 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 1/1 2024-09-13T14:11:07.539 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 1/1 2024-09-13T14:11:07.791 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 1/1 2024-09-13T14:11:07.791 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:07.792 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-09-13T14:11:07.792 INFO:teuthology.orchestra.run.smithi184.stdout: cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:11:07.792 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:07.792 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-09-13T14:11:07.954 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1627.gbbdadfa3. 1/1 2024-09-13T14:11:07.955 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:07.955 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-09-13T14:11:07.955 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-immutable-object-cache-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:11:07.955 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:07.955 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-13T14:11:08.006 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-immutable-object-cache 2024-09-13T14:11:08.185 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr 2024-09-13T14:11:08.583 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:54 ago on Fri 13 Sep 2024 02:10:14 PM UTC. 2024-09-13T14:11:08.717 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-09-13T14:11:08.718 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:08.718 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2024-09-13T14:11:08.718 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:08.718 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-09-13T14:11:08.718 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 137 k 2024-09-13T14:11:08.718 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:08.718 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-09-13T14:11:08.718 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:08.718 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-09-13T14:11:08.719 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:08.719 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 137 k 2024-09-13T14:11:08.719 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 379 k 2024-09-13T14:11:08.719 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-09-13T14:11:08.768 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:55 ago on Fri 13 Sep 2024 02:10:13 PM UTC. 2024-09-13T14:11:08.861 INFO:teuthology.orchestra.run.smithi151.stdout:Package ceph-mgr-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:11:08.903 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-13T14:11:08.904 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-09-13T14:11:08.904 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-13T14:11:08.932 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-immutable-object-cache-17.2.7-1627.gbbdadf 646 kB/s | 137 kB 00:00 2024-09-13T14:11:08.932 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:11:08.933 INFO:teuthology.orchestra.run.smithi184.stdout:Total 641 kB/s | 137 kB 00:00 2024-09-13T14:11:08.933 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-09-13T14:11:08.940 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-09-13T14:11:08.940 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-09-13T14:11:08.974 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr-dashboard 2024-09-13T14:11:08.997 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-09-13T14:11:08.997 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-09-13T14:11:09.148 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-09-13T14:11:09.216 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1627.gbbdadfa3. 1/1 2024-09-13T14:11:09.240 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1627.gbbdadfa3. 1/1 2024-09-13T14:11:09.241 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:11:09.241 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T14:11:09.241 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:09.545 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:56 ago on Fri 13 Sep 2024 02:10:13 PM UTC. 2024-09-13T14:11:09.677 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-13T14:11:09.678 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:09.678 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-09-13T14:11:09.678 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:09.678 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-09-13T14:11:09.678 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 3.4 M 2024-09-13T14:11:09.678 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-09-13T14:11:09.679 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 24 k 2024-09-13T14:11:09.679 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 15 k 2024-09-13T14:11:09.679 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-13T14:11:09.679 INFO:teuthology.orchestra.run.smithi151.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T14:11:09.679 INFO:teuthology.orchestra.run.smithi151.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T14:11:09.679 INFO:teuthology.orchestra.run.smithi151.stdout:Installing weak dependencies: 2024-09-13T14:11:09.679 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-13T14:11:09.679 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:09.679 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-09-13T14:11:09.679 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:09.679 INFO:teuthology.orchestra.run.smithi151.stdout:Install 7 Packages 2024-09-13T14:11:09.679 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:09.680 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 3.7 M 2024-09-13T14:11:09.680 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 80 M 2024-09-13T14:11:09.680 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-09-13T14:11:09.892 INFO:teuthology.orchestra.run.smithi151.stdout:(1/7): ceph-prometheus-alerts-17.2.7-1627.gbbda 95 kB/s | 15 kB 00:00 2024-09-13T14:11:09.926 INFO:teuthology.orchestra.run.smithi151.stdout:(2/7): ceph-grafana-dashboards-17.2.7-1627.gbbd 130 kB/s | 24 kB 00:00 2024-09-13T14:11:09.968 INFO:teuthology.orchestra.run.smithi151.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 120 kB/s | 9.0 kB 00:00 2024-09-13T14:11:10.001 INFO:teuthology.orchestra.run.smithi151.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 967 kB/s | 31 kB 00:00 2024-09-13T14:11:10.027 INFO:teuthology.orchestra.run.smithi151.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 406 kB/s | 41 kB 00:00 2024-09-13T14:11:10.061 INFO:teuthology.orchestra.run.smithi151.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.1 MB/s | 188 kB 00:00 2024-09-13T14:11:10.127 INFO:teuthology.orchestra.run.smithi151.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1627.gbbdadfa3 8.6 MB/s | 3.4 MB 00:00 2024-09-13T14:11:10.131 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:11:10.131 INFO:teuthology.orchestra.run.smithi151.stdout:Total 8.1 MB/s | 3.7 MB 00:00 2024-09-13T14:11:10.131 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-09-13T14:11:10.139 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-09-13T14:11:10.139 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-09-13T14:11:10.223 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-09-13T14:11:10.223 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-09-13T14:11:10.231 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1627.gbbdadfa3. 1/1 2024-09-13T14:11:10.231 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:10.231 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-09-13T14:11:10.231 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-immutable-object-cache-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:11:10.232 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:10.232 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-09-13T14:11:10.453 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr 2024-09-13T14:11:10.522 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-09-13T14:11:10.596 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-13T14:11:10.630 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-13T14:11:10.684 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-13T14:11:10.725 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-13T14:11:10.754 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1627.gbbdadfa3.el9.n 5/7 2024-09-13T14:11:11.035 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:57 ago on Fri 13 Sep 2024 02:10:14 PM UTC. 2024-09-13T14:11:11.125 INFO:teuthology.orchestra.run.smithi184.stdout:Package ceph-mgr-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:11:11.170 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-09-13T14:11:11.171 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-09-13T14:11:11.172 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-09-13T14:11:11.239 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr-dashboard 2024-09-13T14:11:11.747 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1627.gbbdadfa3.el9. 6/7 2024-09-13T14:11:11.800 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1627.gbbdadfa3.el9.noarc 7/7 2024-09-13T14:11:11.802 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:57 ago on Fri 13 Sep 2024 02:10:14 PM UTC. 2024-09-13T14:11:11.934 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-09-13T14:11:11.935 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:11.935 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-09-13T14:11:11.935 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:11.935 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-09-13T14:11:11.935 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 3.4 M 2024-09-13T14:11:11.936 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-09-13T14:11:11.936 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 24 k 2024-09-13T14:11:11.936 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 15 k 2024-09-13T14:11:11.936 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-13T14:11:11.936 INFO:teuthology.orchestra.run.smithi184.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T14:11:11.936 INFO:teuthology.orchestra.run.smithi184.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T14:11:11.936 INFO:teuthology.orchestra.run.smithi184.stdout:Installing weak dependencies: 2024-09-13T14:11:11.936 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-13T14:11:11.936 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:11.936 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-09-13T14:11:11.936 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:11.936 INFO:teuthology.orchestra.run.smithi184.stdout:Install 7 Packages 2024-09-13T14:11:11.936 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:11.937 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 3.7 M 2024-09-13T14:11:11.937 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 80 M 2024-09-13T14:11:11.937 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-09-13T14:11:12.144 INFO:teuthology.orchestra.run.smithi184.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1627.gbbd 159 kB/s | 24 kB 00:00 2024-09-13T14:11:12.169 INFO:teuthology.orchestra.run.smithi184.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1627.gbbda 82 kB/s | 15 kB 00:00 2024-09-13T14:11:12.211 INFO:teuthology.orchestra.run.smithi184.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2024-09-13T14:11:12.236 INFO:teuthology.orchestra.run.smithi184.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-13T14:11:12.261 INFO:teuthology.orchestra.run.smithi184.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 443 kB/s | 41 kB 00:00 2024-09-13T14:11:12.295 INFO:teuthology.orchestra.run.smithi184.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.1 MB/s | 188 kB 00:00 2024-09-13T14:11:12.387 INFO:teuthology.orchestra.run.smithi184.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1627.gbbdadfa3 8.5 MB/s | 3.4 MB 00:00 2024-09-13T14:11:12.390 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:11:12.390 INFO:teuthology.orchestra.run.smithi184.stdout:Total 8.0 MB/s | 3.7 MB 00:00 2024-09-13T14:11:12.391 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-09-13T14:11:12.398 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-09-13T14:11:12.399 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-09-13T14:11:12.482 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-09-13T14:11:12.482 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-09-13T14:11:12.789 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-09-13T14:11:12.855 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-13T14:11:12.889 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-13T14:11:12.903 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1627.gbbdadfa3.el9.noarc 7/7 2024-09-13T14:11:12.903 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1627.gbbdadfa3.el9. 1/7 2024-09-13T14:11:12.903 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1627.gbbdadfa3.el9.noarc 2/7 2024-09-13T14:11:12.904 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1627.gbbdadfa3.el9.n 3/7 2024-09-13T14:11:12.904 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-13T14:11:12.904 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-13T14:11:12.904 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-13T14:11:12.935 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-13T14:11:12.967 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-13T14:11:12.996 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1627.gbbdadfa3.el9.n 5/7 2024-09-13T14:11:13.198 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-13T14:11:13.198 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:13.198 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-09-13T14:11:13.198 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-grafana-dashboards-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:11:13.198 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-dashboard-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:11:13.198 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-prometheus-alerts-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:11:13.198 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-13T14:11:13.198 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-13T14:11:13.198 INFO:teuthology.orchestra.run.smithi151.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T14:11:13.198 INFO:teuthology.orchestra.run.smithi151.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T14:11:13.198 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:13.199 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-13T14:11:13.432 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-13T14:11:13.983 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1627.gbbdadfa3.el9. 6/7 2024-09-13T14:11:13.997 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:00 ago on Fri 13 Sep 2024 02:10:13 PM UTC. 2024-09-13T14:11:14.025 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1627.gbbdadfa3.el9.noarc 7/7 2024-09-13T14:11:14.131 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-13T14:11:14.132 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================= 2024-09-13T14:11:14.132 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-09-13T14:11:14.132 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================= 2024-09-13T14:11:14.132 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-09-13T14:11:14.133 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 7.4 M 2024-09-13T14:11:14.133 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-09-13T14:11:14.133 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T14:11:14.133 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T14:11:14.133 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T14:11:14.133 INFO:teuthology.orchestra.run.smithi151.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T14:11:14.133 INFO:teuthology.orchestra.run.smithi151.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T14:11:14.133 INFO:teuthology.orchestra.run.smithi151.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T14:11:14.133 INFO:teuthology.orchestra.run.smithi151.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T14:11:14.133 INFO:teuthology.orchestra.run.smithi151.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T14:11:14.133 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T14:11:14.134 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T14:11:14.134 INFO:teuthology.orchestra.run.smithi151.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T14:11:14.134 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:14.134 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-09-13T14:11:14.134 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================= 2024-09-13T14:11:14.134 INFO:teuthology.orchestra.run.smithi151.stdout:Install 12 Packages 2024-09-13T14:11:14.134 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:14.135 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 38 M 2024-09-13T14:11:14.135 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 212 M 2024-09-13T14:11:14.135 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-09-13T14:11:14.614 INFO:teuthology.orchestra.run.smithi151.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 191 kB 00:00 2024-09-13T14:11:14.697 INFO:teuthology.orchestra.run.smithi151.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.0 MB/s | 801 kB 00:00 2024-09-13T14:11:14.722 INFO:teuthology.orchestra.run.smithi151.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 301 kB/s | 33 kB 00:00 2024-09-13T14:11:14.773 INFO:teuthology.orchestra.run.smithi151.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-09-13T14:11:14.898 INFO:teuthology.orchestra.run.smithi151.stdout:(5/12): ceph-mgr-diskprediction-local-17.2.7-16 16 MB/s | 7.4 MB 00:00 2024-09-13T14:11:14.924 INFO:teuthology.orchestra.run.smithi151.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 255 kB/s | 38 kB 00:00 2024-09-13T14:11:15.049 INFO:teuthology.orchestra.run.smithi151.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.9 MB/s | 245 kB 00:00 2024-09-13T14:11:15.118 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1627.gbbdadfa3.el9.noarc 7/7 2024-09-13T14:11:15.118 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1627.gbbdadfa3.el9. 1/7 2024-09-13T14:11:15.118 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1627.gbbdadfa3.el9.noarc 2/7 2024-09-13T14:11:15.118 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1627.gbbdadfa3.el9.n 3/7 2024-09-13T14:11:15.118 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-13T14:11:15.118 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-13T14:11:15.119 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-13T14:11:15.405 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-13T14:11:15.405 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:15.405 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-09-13T14:11:15.405 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-grafana-dashboards-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:11:15.405 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-dashboard-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:11:15.405 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-prometheus-alerts-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:11:15.405 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-13T14:11:15.406 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-13T14:11:15.406 INFO:teuthology.orchestra.run.smithi184.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T14:11:15.406 INFO:teuthology.orchestra.run.smithi184.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T14:11:15.406 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:15.406 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-09-13T14:11:15.616 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-13T14:11:15.766 INFO:teuthology.orchestra.run.smithi151.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.8 MB/s | 3.0 MB 00:01 2024-09-13T14:11:15.908 INFO:teuthology.orchestra.run.smithi151.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.7 MB/s | 250 kB 00:00 2024-09-13T14:11:16.178 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:02 ago on Fri 13 Sep 2024 02:10:14 PM UTC. 2024-09-13T14:11:16.310 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-09-13T14:11:16.311 INFO:teuthology.orchestra.run.smithi184.stdout:======================================================================================= 2024-09-13T14:11:16.311 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-09-13T14:11:16.311 INFO:teuthology.orchestra.run.smithi184.stdout:======================================================================================= 2024-09-13T14:11:16.311 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-09-13T14:11:16.311 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 7.4 M 2024-09-13T14:11:16.311 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-09-13T14:11:16.311 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T14:11:16.312 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T14:11:16.312 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T14:11:16.312 INFO:teuthology.orchestra.run.smithi184.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T14:11:16.312 INFO:teuthology.orchestra.run.smithi184.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T14:11:16.312 INFO:teuthology.orchestra.run.smithi184.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T14:11:16.312 INFO:teuthology.orchestra.run.smithi184.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T14:11:16.312 INFO:teuthology.orchestra.run.smithi184.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T14:11:16.312 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T14:11:16.312 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T14:11:16.312 INFO:teuthology.orchestra.run.smithi184.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T14:11:16.312 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:16.312 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-09-13T14:11:16.312 INFO:teuthology.orchestra.run.smithi184.stdout:======================================================================================= 2024-09-13T14:11:16.312 INFO:teuthology.orchestra.run.smithi184.stdout:Install 12 Packages 2024-09-13T14:11:16.313 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:16.313 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 38 M 2024-09-13T14:11:16.313 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 212 M 2024-09-13T14:11:16.313 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-09-13T14:11:16.937 INFO:teuthology.orchestra.run.smithi151.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 2.4 MB/s | 4.9 MB 00:02 2024-09-13T14:11:17.037 INFO:teuthology.orchestra.run.smithi184.stdout:(1/12): ceph-mgr-diskprediction-local-17.2.7-16 17 MB/s | 7.4 MB 00:00 2024-09-13T14:11:17.154 INFO:teuthology.orchestra.run.smithi184.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 281 kB/s | 33 kB 00:00 2024-09-13T14:11:17.292 INFO:teuthology.orchestra.run.smithi151.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.3 MB/s | 5.1 MB 00:02 2024-09-13T14:11:17.404 INFO:teuthology.orchestra.run.smithi184.stdout:(3/12): libquadmath-11.5.0-2.el9.x86_64.rpm 234 kB/s | 191 kB 00:00 2024-09-13T14:11:17.479 INFO:teuthology.orchestra.run.smithi184.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 244 kB/s | 18 kB 00:00 2024-09-13T14:11:17.562 INFO:teuthology.orchestra.run.smithi184.stdout:(5/12): libgfortran-11.5.0-2.el9.x86_64.rpm 822 kB/s | 801 kB 00:00 2024-09-13T14:11:17.580 INFO:teuthology.orchestra.run.smithi184.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 384 kB/s | 38 kB 00:00 2024-09-13T14:11:17.771 INFO:teuthology.orchestra.run.smithi184.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-09-13T14:11:17.980 INFO:teuthology.orchestra.run.smithi184.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.6 MB/s | 3.0 MB 00:00 2024-09-13T14:11:18.080 INFO:teuthology.orchestra.run.smithi184.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 250 kB 00:00 2024-09-13T14:11:18.267 INFO:teuthology.orchestra.run.smithi151.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.9 MB/s | 16 MB 00:02 2024-09-13T14:11:18.268 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:11:18.268 INFO:teuthology.orchestra.run.smithi151.stdout:Total 9.2 MB/s | 38 MB 00:04 2024-09-13T14:11:18.619 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-09-13T14:11:18.622 INFO:teuthology.orchestra.run.smithi184.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.7 MB/s | 4.9 MB 00:01 2024-09-13T14:11:18.645 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-09-13T14:11:18.645 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-09-13T14:11:19.047 INFO:teuthology.orchestra.run.smithi184.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.0 MB/s | 5.1 MB 00:01 2024-09-13T14:11:19.171 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-09-13T14:11:19.171 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-09-13T14:11:19.928 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-09-13T14:11:20.023 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-13T14:11:20.065 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-13T14:11:20.172 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-13T14:11:20.217 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-13T14:11:20.403 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-13T14:11:20.433 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-13T14:11:20.528 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-13T14:11:20.756 INFO:teuthology.orchestra.run.smithi184.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.1 MB/s | 16 MB 00:02 2024-09-13T14:11:20.757 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:11:20.757 INFO:teuthology.orchestra.run.smithi184.stdout:Total 8.6 MB/s | 38 MB 00:04 2024-09-13T14:11:21.110 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-09-13T14:11:21.137 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-09-13T14:11:21.137 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-09-13T14:11:21.179 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-13T14:11:21.307 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-13T14:11:21.653 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-09-13T14:11:21.653 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-09-13T14:11:22.421 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-09-13T14:11:22.512 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-13T14:11:22.554 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-13T14:11:22.555 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T14:11:22.656 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-13T14:11:22.697 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-13T14:11:22.880 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-13T14:11:22.913 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-13T14:11:22.999 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-13T14:11:23.591 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-13T14:11:23.622 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1627.gbbdad 12/12 2024-09-13T14:11:23.663 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-13T14:11:23.777 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-13T14:11:25.065 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T14:11:26.130 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-13T14:11:26.170 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1627.gbbdad 12/12 2024-09-13T14:11:26.559 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1627.gbbdad 12/12 2024-09-13T14:11:26.559 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1627.gbbdad 1/12 2024-09-13T14:11:26.559 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-13T14:11:26.559 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-13T14:11:26.560 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-13T14:11:26.560 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-13T14:11:26.560 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-13T14:11:26.560 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-13T14:11:26.560 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-13T14:11:26.560 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-13T14:11:26.561 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T14:11:26.561 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-13T14:11:27.015 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-13T14:11:27.015 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:27.016 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-09-13T14:11:27.016 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:11:27.016 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T14:11:27.016 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T14:11:27.016 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T14:11:27.016 INFO:teuthology.orchestra.run.smithi151.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T14:11:27.016 INFO:teuthology.orchestra.run.smithi151.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T14:11:27.016 INFO:teuthology.orchestra.run.smithi151.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T14:11:27.016 INFO:teuthology.orchestra.run.smithi151.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T14:11:27.016 INFO:teuthology.orchestra.run.smithi151.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T14:11:27.016 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T14:11:27.016 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T14:11:27.017 INFO:teuthology.orchestra.run.smithi151.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T14:11:27.017 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:27.017 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-13T14:11:27.405 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr-rook 2024-09-13T14:11:27.993 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:14 ago on Fri 13 Sep 2024 02:10:13 PM UTC. 2024-09-13T14:11:28.134 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-13T14:11:28.134 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:28.135 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-09-13T14:11:28.135 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:28.135 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-09-13T14:11:28.135 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-rook noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 47 k 2024-09-13T14:11:28.135 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-09-13T14:11:28.135 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T14:11:28.135 INFO:teuthology.orchestra.run.smithi151.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T14:11:28.135 INFO:teuthology.orchestra.run.smithi151.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T14:11:28.135 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T14:11:28.135 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T14:11:28.135 INFO:teuthology.orchestra.run.smithi151.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T14:11:28.135 INFO:teuthology.orchestra.run.smithi151.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T14:11:28.135 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T14:11:28.136 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T14:11:28.136 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T14:11:28.136 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T14:11:28.136 INFO:teuthology.orchestra.run.smithi151.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T14:11:28.136 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:28.136 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-09-13T14:11:28.136 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:28.136 INFO:teuthology.orchestra.run.smithi151.stdout:Install 13 Packages 2024-09-13T14:11:28.136 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:28.137 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 2.2 M 2024-09-13T14:11:28.137 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 26 M 2024-09-13T14:11:28.137 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-09-13T14:11:28.312 INFO:teuthology.orchestra.run.smithi151.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 195 kB/s | 19 kB 00:00 2024-09-13T14:11:28.337 INFO:teuthology.orchestra.run.smithi151.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 207 kB/s | 26 kB 00:00 2024-09-13T14:11:28.396 INFO:teuthology.orchestra.run.smithi151.stdout:(3/13): ceph-mgr-rook-17.2.7-1627.gbbdadfa3.el9 260 kB/s | 47 kB 00:00 2024-09-13T14:11:28.438 INFO:teuthology.orchestra.run.smithi151.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.7 MB/s | 222 kB 00:00 2024-09-13T14:11:28.463 INFO:teuthology.orchestra.run.smithi151.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-13T14:11:28.488 INFO:teuthology.orchestra.run.smithi151.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-13T14:11:28.530 INFO:teuthology.orchestra.run.smithi151.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 479 kB/s | 32 kB 00:00 2024-09-13T14:11:28.556 INFO:teuthology.orchestra.run.smithi151.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 210 kB/s | 14 kB 00:00 2024-09-13T14:11:28.581 INFO:teuthology.orchestra.run.smithi151.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 4.4 MB/s | 220 kB 00:00 2024-09-13T14:11:28.606 INFO:teuthology.orchestra.run.smithi151.stdout:(10/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.3 MB/s | 279 kB 00:00 2024-09-13T14:11:28.632 INFO:teuthology.orchestra.run.smithi151.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-09-13T14:11:28.674 INFO:teuthology.orchestra.run.smithi151.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-09-13T14:11:28.678 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1627.gbbdad 12/12 2024-09-13T14:11:28.679 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1627.gbbdad 1/12 2024-09-13T14:11:28.679 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-13T14:11:28.679 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-13T14:11:28.679 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-13T14:11:28.679 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-13T14:11:28.679 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-13T14:11:28.679 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-13T14:11:28.679 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-13T14:11:28.679 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-13T14:11:28.679 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T14:11:28.679 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-13T14:11:28.707 INFO:teuthology.orchestra.run.smithi151.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.7 MB/s | 1.0 MB 00:00 2024-09-13T14:11:28.711 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:11:28.711 INFO:teuthology.orchestra.run.smithi151.stdout:Total 3.9 MB/s | 2.2 MB 00:00 2024-09-13T14:11:28.776 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-09-13T14:11:28.793 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-09-13T14:11:28.793 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-09-13T14:11:28.882 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-09-13T14:11:28.884 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-09-13T14:11:29.142 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-13T14:11:29.142 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:29.143 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-09-13T14:11:29.143 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:11:29.143 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T14:11:29.143 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T14:11:29.143 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T14:11:29.143 INFO:teuthology.orchestra.run.smithi184.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T14:11:29.143 INFO:teuthology.orchestra.run.smithi184.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T14:11:29.143 INFO:teuthology.orchestra.run.smithi184.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T14:11:29.143 INFO:teuthology.orchestra.run.smithi184.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T14:11:29.143 INFO:teuthology.orchestra.run.smithi184.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T14:11:29.143 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T14:11:29.144 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T14:11:29.144 INFO:teuthology.orchestra.run.smithi184.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T14:11:29.144 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:29.144 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-09-13T14:11:29.382 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-09-13T14:11:29.495 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr-rook 2024-09-13T14:11:29.590 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-13T14:11:29.651 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-13T14:11:29.715 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-13T14:11:29.753 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-13T14:11:29.794 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-13T14:11:29.906 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-13T14:11:30.007 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-13T14:11:30.063 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-13T14:11:30.080 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:16 ago on Fri 13 Sep 2024 02:10:14 PM UTC. 2024-09-13T14:11:30.222 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-09-13T14:11:30.223 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:30.223 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-09-13T14:11:30.223 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:30.223 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-09-13T14:11:30.223 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-rook noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 47 k 2024-09-13T14:11:30.224 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-09-13T14:11:30.224 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T14:11:30.224 INFO:teuthology.orchestra.run.smithi184.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T14:11:30.224 INFO:teuthology.orchestra.run.smithi184.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T14:11:30.224 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T14:11:30.224 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T14:11:30.224 INFO:teuthology.orchestra.run.smithi184.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T14:11:30.224 INFO:teuthology.orchestra.run.smithi184.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T14:11:30.224 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T14:11:30.224 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T14:11:30.224 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T14:11:30.224 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T14:11:30.224 INFO:teuthology.orchestra.run.smithi184.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T14:11:30.225 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:30.225 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-09-13T14:11:30.225 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:30.225 INFO:teuthology.orchestra.run.smithi184.stdout:Install 13 Packages 2024-09-13T14:11:30.225 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:30.225 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 2.2 M 2024-09-13T14:11:30.226 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 26 M 2024-09-13T14:11:30.226 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-09-13T14:11:30.528 INFO:teuthology.orchestra.run.smithi184.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 344 kB/s | 26 kB 00:00 2024-09-13T14:11:30.544 INFO:teuthology.orchestra.run.smithi184.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 210 kB/s | 19 kB 00:00 2024-09-13T14:11:30.629 INFO:teuthology.orchestra.run.smithi184.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 222 kB 00:00 2024-09-13T14:11:30.645 INFO:teuthology.orchestra.run.smithi184.stdout:(4/13): ceph-mgr-rook-17.2.7-1627.gbbdadfa3.el9 248 kB/s | 47 kB 00:00 2024-09-13T14:11:30.670 INFO:teuthology.orchestra.run.smithi184.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-13T14:11:30.687 INFO:teuthology.orchestra.run.smithi184.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.2 MB/s | 54 kB 00:00 2024-09-13T14:11:30.712 INFO:teuthology.orchestra.run.smithi184.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.2 MB/s | 279 kB 00:00 2024-09-13T14:11:30.754 INFO:teuthology.orchestra.run.smithi184.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 383 kB/s | 32 kB 00:00 2024-09-13T14:11:30.771 INFO:teuthology.orchestra.run.smithi184.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 168 kB/s | 14 kB 00:00 2024-09-13T14:11:30.790 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-13T14:11:30.796 INFO:teuthology.orchestra.run.smithi184.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-13T14:11:30.822 INFO:teuthology.orchestra.run.smithi184.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 2.3 MB/s | 220 kB 00:00 2024-09-13T14:11:30.845 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-13T14:11:30.847 INFO:teuthology.orchestra.run.smithi184.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-13T14:11:30.879 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-13T14:11:30.889 INFO:teuthology.orchestra.run.smithi184.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.7 MB/s | 1.0 MB 00:00 2024-09-13T14:11:30.892 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:11:30.892 INFO:teuthology.orchestra.run.smithi184.stdout:Total 3.3 MB/s | 2.2 MB 00:00 2024-09-13T14:11:30.925 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-13T14:11:30.941 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-rook-2:17.2.7-1627.gbbdadfa3.el9.noarch 13/13 2024-09-13T14:11:30.958 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-09-13T14:11:30.973 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-09-13T14:11:30.973 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-09-13T14:11:31.066 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-09-13T14:11:31.067 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-09-13T14:11:31.572 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-09-13T14:11:31.745 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-13T14:11:31.783 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1627.gbbdadfa3.el9.noarch 13/13 2024-09-13T14:11:31.783 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1627.gbbdadfa3.el9.noarch 1/13 2024-09-13T14:11:31.783 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-13T14:11:31.783 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-13T14:11:31.783 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-13T14:11:31.783 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-13T14:11:31.783 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-13T14:11:31.783 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-13T14:11:31.783 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-13T14:11:31.783 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-13T14:11:31.783 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-13T14:11:31.784 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-13T14:11:31.784 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-13T14:11:31.807 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-13T14:11:31.862 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-13T14:11:31.908 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-13T14:11:31.949 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-13T14:11:32.051 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-13T14:11:32.136 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-13T14:11:32.136 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:32.136 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-09-13T14:11:32.136 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-rook-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:11:32.136 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T14:11:32.136 INFO:teuthology.orchestra.run.smithi151.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T14:11:32.136 INFO:teuthology.orchestra.run.smithi151.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T14:11:32.136 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T14:11:32.137 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T14:11:32.137 INFO:teuthology.orchestra.run.smithi151.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T14:11:32.137 INFO:teuthology.orchestra.run.smithi151.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T14:11:32.137 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T14:11:32.137 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T14:11:32.137 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T14:11:32.137 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T14:11:32.137 INFO:teuthology.orchestra.run.smithi151.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T14:11:32.137 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:32.137 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-13T14:11:32.161 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-13T14:11:32.209 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-13T14:11:32.413 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr-cephadm 2024-09-13T14:11:32.922 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-13T14:11:32.965 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-13T14:11:32.978 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:19 ago on Fri 13 Sep 2024 02:10:13 PM UTC. 2024-09-13T14:11:32.998 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-13T14:11:33.044 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-13T14:11:33.068 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-rook-2:17.2.7-1627.gbbdadfa3.el9.noarch 13/13 2024-09-13T14:11:33.111 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-13T14:11:33.112 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:33.112 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-09-13T14:11:33.112 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:33.112 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-09-13T14:11:33.112 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 124 k 2024-09-13T14:11:33.113 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-09-13T14:11:33.113 INFO:teuthology.orchestra.run.smithi151.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T14:11:33.113 INFO:teuthology.orchestra.run.smithi151.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T14:11:33.113 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T14:11:33.113 INFO:teuthology.orchestra.run.smithi151.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T14:11:33.113 INFO:teuthology.orchestra.run.smithi151.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T14:11:33.113 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:33.113 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-09-13T14:11:33.113 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:33.113 INFO:teuthology.orchestra.run.smithi151.stdout:Install 6 Packages 2024-09-13T14:11:33.113 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:33.114 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 7.0 M 2024-09-13T14:11:33.114 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 33 M 2024-09-13T14:11:33.114 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-09-13T14:11:33.615 INFO:teuthology.orchestra.run.smithi151.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-13T14:11:33.641 INFO:teuthology.orchestra.run.smithi151.stdout:(2/6): ceph-mgr-cephadm-17.2.7-1627.gbbdadfa3.e 504 kB/s | 124 kB 00:00 2024-09-13T14:11:33.691 INFO:teuthology.orchestra.run.smithi151.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 7.1 MB/s | 548 kB 00:00 2024-09-13T14:11:33.716 INFO:teuthology.orchestra.run.smithi151.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 767 kB/s | 58 kB 00:00 2024-09-13T14:11:33.741 INFO:teuthology.orchestra.run.smithi151.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.4 MB/s | 74 kB 00:00 2024-09-13T14:11:33.842 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1627.gbbdadfa3.el9.noarch 13/13 2024-09-13T14:11:33.842 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1627.gbbdadfa3.el9.noarch 1/13 2024-09-13T14:11:33.842 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-13T14:11:33.843 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-13T14:11:33.843 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-13T14:11:33.843 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-13T14:11:33.843 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-13T14:11:33.843 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-13T14:11:33.843 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-13T14:11:33.843 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-13T14:11:33.843 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-13T14:11:33.843 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-13T14:11:33.843 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-13T14:11:33.925 INFO:teuthology.orchestra.run.smithi151.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 6.0 MB 00:00 2024-09-13T14:11:33.927 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:11:33.928 INFO:teuthology.orchestra.run.smithi151.stdout:Total 8.6 MB/s | 7.0 MB 00:00 2024-09-13T14:11:34.000 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-09-13T14:11:34.011 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-09-13T14:11:34.012 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-09-13T14:11:34.112 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-09-13T14:11:34.112 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-09-13T14:11:34.177 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-13T14:11:34.177 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:34.177 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-09-13T14:11:34.177 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-rook-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:11:34.177 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T14:11:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T14:11:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T14:11:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T14:11:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T14:11:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T14:11:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T14:11:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T14:11:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T14:11:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T14:11:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T14:11:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T14:11:34.179 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:34.179 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-09-13T14:11:34.431 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-09-13T14:11:34.436 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr-cephadm 2024-09-13T14:11:34.598 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-13T14:11:34.653 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-13T14:11:34.997 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:20 ago on Fri 13 Sep 2024 02:10:14 PM UTC. 2024-09-13T14:11:35.130 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-09-13T14:11:35.131 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:35.131 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-09-13T14:11:35.131 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:35.131 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-09-13T14:11:35.131 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1627.gbbdadfa3.el9 ceph-noarch 124 k 2024-09-13T14:11:35.131 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-09-13T14:11:35.131 INFO:teuthology.orchestra.run.smithi184.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T14:11:35.131 INFO:teuthology.orchestra.run.smithi184.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T14:11:35.131 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T14:11:35.132 INFO:teuthology.orchestra.run.smithi184.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T14:11:35.132 INFO:teuthology.orchestra.run.smithi184.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T14:11:35.132 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:35.132 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-09-13T14:11:35.132 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:35.132 INFO:teuthology.orchestra.run.smithi184.stdout:Install 6 Packages 2024-09-13T14:11:35.132 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:35.133 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 7.0 M 2024-09-13T14:11:35.133 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 33 M 2024-09-13T14:11:35.133 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-09-13T14:11:35.175 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-13T14:11:35.264 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-13T14:11:35.320 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-13T14:11:35.350 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 6/6 2024-09-13T14:11:35.565 INFO:teuthology.orchestra.run.smithi184.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-09-13T14:11:35.599 INFO:teuthology.orchestra.run.smithi184.stdout:(2/6): ceph-mgr-cephadm-17.2.7-1627.gbbdadfa3.e 575 kB/s | 124 kB 00:00 2024-09-13T14:11:35.708 INFO:teuthology.orchestra.run.smithi184.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 6.0 MB 00:00 2024-09-13T14:11:35.733 INFO:teuthology.orchestra.run.smithi184.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 438 kB/s | 58 kB 00:00 2024-09-13T14:11:35.766 INFO:teuthology.orchestra.run.smithi184.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.7 MB/s | 548 kB 00:00 2024-09-13T14:11:35.800 INFO:teuthology.orchestra.run.smithi184.stdout:(6/6): python3-typing-extensions-4.12.2-2.el9.n 814 kB/s | 74 kB 00:00 2024-09-13T14:11:35.803 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:11:35.803 INFO:teuthology.orchestra.run.smithi184.stdout:Total 10 MB/s | 7.0 MB 00:00 2024-09-13T14:11:35.873 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-09-13T14:11:35.884 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-09-13T14:11:35.884 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-09-13T14:11:35.984 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-09-13T14:11:35.985 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-09-13T14:11:36.199 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 6/6 2024-09-13T14:11:36.199 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 1/6 2024-09-13T14:11:36.199 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-13T14:11:36.199 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-13T14:11:36.199 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-13T14:11:36.199 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-13T14:11:36.286 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-09-13T14:11:36.466 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-13T14:11:36.510 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-13T14:11:36.510 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:36.510 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-09-13T14:11:36.510 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:11:36.510 INFO:teuthology.orchestra.run.smithi151.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T14:11:36.510 INFO:teuthology.orchestra.run.smithi151.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T14:11:36.511 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T14:11:36.511 INFO:teuthology.orchestra.run.smithi151.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T14:11:36.511 INFO:teuthology.orchestra.run.smithi151.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T14:11:36.511 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:36.511 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-13T14:11:36.528 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-13T14:11:36.769 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-fuse 2024-09-13T14:11:37.048 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-13T14:11:37.129 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-13T14:11:37.185 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-13T14:11:37.214 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 6/6 2024-09-13T14:11:37.337 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:24 ago on Fri 13 Sep 2024 02:10:13 PM UTC. 2024-09-13T14:11:37.471 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-13T14:11:37.472 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:37.473 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-09-13T14:11:37.473 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:37.473 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-09-13T14:11:37.473 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-fuse x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 813 k 2024-09-13T14:11:37.473 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-09-13T14:11:37.473 INFO:teuthology.orchestra.run.smithi151.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T14:11:37.474 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:37.474 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-09-13T14:11:37.474 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:37.474 INFO:teuthology.orchestra.run.smithi151.stdout:Install 2 Packages 2024-09-13T14:11:37.474 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:37.474 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 893 k 2024-09-13T14:11:37.474 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 2.6 M 2024-09-13T14:11:37.475 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-09-13T14:11:37.822 INFO:teuthology.orchestra.run.smithi151.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 865 kB/s | 80 kB 00:00 2024-09-13T14:11:38.024 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 6/6 2024-09-13T14:11:38.025 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 1/6 2024-09-13T14:11:38.025 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-13T14:11:38.025 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-13T14:11:38.025 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-13T14:11:38.025 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-13T14:11:38.038 INFO:teuthology.orchestra.run.smithi151.stdout:(2/2): ceph-fuse-17.2.7-1627.gbbdadfa3.el9.x86_ 2.6 MB/s | 813 kB 00:00 2024-09-13T14:11:38.039 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:11:38.039 INFO:teuthology.orchestra.run.smithi151.stdout:Total 1.5 MB/s | 893 kB 00:00 2024-09-13T14:11:38.051 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-09-13T14:11:38.074 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-09-13T14:11:38.074 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-09-13T14:11:38.156 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-09-13T14:11:38.157 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-09-13T14:11:38.300 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-13T14:11:38.300 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:38.300 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-09-13T14:11:38.300 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-cephadm-2:17.2.7-1627.gbbdadfa3.el9.noarch 2024-09-13T14:11:38.301 INFO:teuthology.orchestra.run.smithi184.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T14:11:38.301 INFO:teuthology.orchestra.run.smithi184.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T14:11:38.301 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T14:11:38.301 INFO:teuthology.orchestra.run.smithi184.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T14:11:38.301 INFO:teuthology.orchestra.run.smithi184.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T14:11:38.301 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:38.301 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-09-13T14:11:38.337 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-09-13T14:11:38.480 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-13T14:11:38.546 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-fuse 2024-09-13T14:11:38.555 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2/2 2024-09-13T14:11:39.112 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:25 ago on Fri 13 Sep 2024 02:10:14 PM UTC. 2024-09-13T14:11:39.247 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-09-13T14:11:39.248 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:39.248 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-09-13T14:11:39.248 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:39.248 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-09-13T14:11:39.248 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-fuse x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 813 k 2024-09-13T14:11:39.248 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-09-13T14:11:39.248 INFO:teuthology.orchestra.run.smithi184.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T14:11:39.248 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:39.249 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-09-13T14:11:39.249 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:39.249 INFO:teuthology.orchestra.run.smithi184.stdout:Install 2 Packages 2024-09-13T14:11:39.249 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:39.249 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 893 k 2024-09-13T14:11:39.249 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 2.6 M 2024-09-13T14:11:39.249 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-09-13T14:11:39.294 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2/2 2024-09-13T14:11:39.294 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/2 2024-09-13T14:11:39.411 INFO:teuthology.orchestra.run.smithi184.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 697 kB/s | 80 kB 00:00 2024-09-13T14:11:39.551 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-13T14:11:39.551 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:39.551 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-09-13T14:11:39.552 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-13T14:11:39.552 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:39.552 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-13T14:11:39.569 INFO:teuthology.orchestra.run.smithi184.stdout:(2/2): ceph-fuse-17.2.7-1627.gbbdadfa3.el9.x86_ 2.9 MB/s | 813 kB 00:00 2024-09-13T14:11:39.570 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:11:39.570 INFO:teuthology.orchestra.run.smithi184.stdout:Total 2.7 MB/s | 893 kB 00:00 2024-09-13T14:11:39.582 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-09-13T14:11:39.604 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-09-13T14:11:39.604 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-09-13T14:11:39.685 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-09-13T14:11:39.685 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-09-13T14:11:39.831 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librados-devel 2024-09-13T14:11:39.878 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-09-13T14:11:40.006 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-13T14:11:40.077 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2/2 2024-09-13T14:11:40.411 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:27 ago on Fri 13 Sep 2024 02:10:13 PM UTC. 2024-09-13T14:11:40.545 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-13T14:11:40.546 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:40.546 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-09-13T14:11:40.546 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:40.546 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-09-13T14:11:40.546 INFO:teuthology.orchestra.run.smithi151.stdout: librados-devel x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 126 k 2024-09-13T14:11:40.547 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:40.547 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-09-13T14:11:40.547 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:40.547 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-09-13T14:11:40.547 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:40.547 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 126 k 2024-09-13T14:11:40.547 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 448 k 2024-09-13T14:11:40.547 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-09-13T14:11:40.723 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2/2 2024-09-13T14:11:40.723 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/2 2024-09-13T14:11:40.772 INFO:teuthology.orchestra.run.smithi151.stdout:librados-devel-17.2.7-1627.gbbdadfa3.el9.x86_64 562 kB/s | 126 kB 00:00 2024-09-13T14:11:40.773 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:11:40.773 INFO:teuthology.orchestra.run.smithi151.stdout:Total 557 kB/s | 126 kB 00:00 2024-09-13T14:11:40.773 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-09-13T14:11:40.778 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-09-13T14:11:40.778 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-09-13T14:11:40.842 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-09-13T14:11:40.843 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-09-13T14:11:40.997 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-09-13T14:11:41.018 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-13T14:11:41.018 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:41.018 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-09-13T14:11:41.018 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-13T14:11:41.019 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:41.019 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-09-13T14:11:41.119 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librados-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:11:41.281 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install librados-devel 2024-09-13T14:11:41.613 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:11:41.858 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:27 ago on Fri 13 Sep 2024 02:10:14 PM UTC. 2024-09-13T14:11:41.869 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:11:41.869 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:41.869 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-09-13T14:11:41.869 INFO:teuthology.orchestra.run.smithi151.stdout: librados-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:11:41.870 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:41.870 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-13T14:11:41.994 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-09-13T14:11:41.995 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:41.995 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2024-09-13T14:11:41.995 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:41.996 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-09-13T14:11:41.996 INFO:teuthology.orchestra.run.smithi184.stdout: librados-devel x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 126 k 2024-09-13T14:11:41.996 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:41.996 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-09-13T14:11:41.996 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:41.996 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-09-13T14:11:41.996 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:41.996 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 126 k 2024-09-13T14:11:41.996 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 448 k 2024-09-13T14:11:41.996 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-09-13T14:11:42.106 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install libcephfs2 2024-09-13T14:11:42.182 INFO:teuthology.orchestra.run.smithi184.stdout:librados-devel-17.2.7-1627.gbbdadfa3.el9.x86_64 680 kB/s | 126 kB 00:00 2024-09-13T14:11:42.183 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:11:42.183 INFO:teuthology.orchestra.run.smithi184.stdout:Total 674 kB/s | 126 kB 00:00 2024-09-13T14:11:42.183 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-09-13T14:11:42.188 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-09-13T14:11:42.188 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-09-13T14:11:42.252 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-09-13T14:11:42.252 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-09-13T14:11:42.403 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-09-13T14:11:42.521 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librados-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:11:42.683 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:29 ago on Fri 13 Sep 2024 02:10:13 PM UTC. 2024-09-13T14:11:42.776 INFO:teuthology.orchestra.run.smithi151.stdout:Package libcephfs2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:11:42.821 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-13T14:11:42.822 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-09-13T14:11:42.822 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-13T14:11:42.884 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install libcephfs-devel 2024-09-13T14:11:42.982 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librados-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:11:43.200 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librados-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:11:43.200 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:43.200 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-09-13T14:11:43.200 INFO:teuthology.orchestra.run.smithi184.stdout: librados-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:11:43.200 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:43.200 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-09-13T14:11:43.449 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:30 ago on Fri 13 Sep 2024 02:10:13 PM UTC. 2024-09-13T14:11:43.502 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install libcephfs2 2024-09-13T14:11:43.583 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-13T14:11:43.584 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:43.584 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-09-13T14:11:43.584 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:43.584 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-09-13T14:11:43.584 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs-devel x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 25 k 2024-09-13T14:11:43.584 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:43.584 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-09-13T14:11:43.584 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:43.585 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-09-13T14:11:43.585 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:43.585 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 25 k 2024-09-13T14:11:43.585 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 110 k 2024-09-13T14:11:43.586 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-09-13T14:11:43.742 INFO:teuthology.orchestra.run.smithi151.stdout:libcephfs-devel-17.2.7-1627.gbbdadfa3.el9.x86_6 159 kB/s | 25 kB 00:00 2024-09-13T14:11:43.742 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:11:43.742 INFO:teuthology.orchestra.run.smithi151.stdout:Total 158 kB/s | 25 kB 00:00 2024-09-13T14:11:43.743 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-09-13T14:11:43.746 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-09-13T14:11:43.746 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-09-13T14:11:43.747 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-09-13T14:11:43.748 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-09-13T14:11:43.833 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-09-13T14:11:43.955 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephfs-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:11:44.084 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:30 ago on Fri 13 Sep 2024 02:10:14 PM UTC. 2024-09-13T14:11:44.177 INFO:teuthology.orchestra.run.smithi184.stdout:Package libcephfs2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:11:44.223 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-09-13T14:11:44.224 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-09-13T14:11:44.224 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-09-13T14:11:44.286 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install libcephfs-devel 2024-09-13T14:11:44.330 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:11:44.565 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephfs-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:11:44.565 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:44.565 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-09-13T14:11:44.565 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:11:44.565 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:44.566 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-13T14:11:44.782 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librados2 2024-09-13T14:11:44.853 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:30 ago on Fri 13 Sep 2024 02:10:14 PM UTC. 2024-09-13T14:11:44.987 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-09-13T14:11:44.988 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:44.988 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2024-09-13T14:11:44.988 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:44.988 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-09-13T14:11:44.988 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs-devel x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 25 k 2024-09-13T14:11:44.988 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:44.988 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-09-13T14:11:44.988 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:44.988 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-09-13T14:11:44.989 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:44.989 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 25 k 2024-09-13T14:11:44.989 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 110 k 2024-09-13T14:11:44.989 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-09-13T14:11:45.135 INFO:teuthology.orchestra.run.smithi184.stdout:libcephfs-devel-17.2.7-1627.gbbdadfa3.el9.x86_6 170 kB/s | 25 kB 00:00 2024-09-13T14:11:45.136 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:11:45.136 INFO:teuthology.orchestra.run.smithi184.stdout:Total 169 kB/s | 25 kB 00:00 2024-09-13T14:11:45.136 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-09-13T14:11:45.140 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-09-13T14:11:45.140 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-09-13T14:11:45.141 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-09-13T14:11:45.142 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-09-13T14:11:45.231 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-09-13T14:11:45.324 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libcephfs-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:11:45.354 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:32 ago on Fri 13 Sep 2024 02:10:13 PM UTC. 2024-09-13T14:11:45.447 INFO:teuthology.orchestra.run.smithi151.stdout:Package librados2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:11:45.489 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-13T14:11:45.490 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-09-13T14:11:45.491 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-13T14:11:45.549 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librbd1 2024-09-13T14:11:45.705 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:11:45.941 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libcephfs-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:11:45.941 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:45.942 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-09-13T14:11:45.942 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs-devel-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:11:45.942 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:45.942 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-09-13T14:11:46.114 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:33 ago on Fri 13 Sep 2024 02:10:13 PM UTC. 2024-09-13T14:11:46.137 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install librados2 2024-09-13T14:11:46.207 INFO:teuthology.orchestra.run.smithi151.stdout:Package librbd1-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:11:46.250 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-13T14:11:46.251 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-09-13T14:11:46.251 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-13T14:11:46.316 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rados 2024-09-13T14:11:46.710 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:32 ago on Fri 13 Sep 2024 02:10:14 PM UTC. 2024-09-13T14:11:46.801 INFO:teuthology.orchestra.run.smithi184.stdout:Package librados2-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:11:46.845 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-09-13T14:11:46.846 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-09-13T14:11:46.846 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-09-13T14:11:46.886 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:33 ago on Fri 13 Sep 2024 02:10:13 PM UTC. 2024-09-13T14:11:46.911 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install librbd1 2024-09-13T14:11:46.977 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-rados-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:11:47.020 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-13T14:11:47.021 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-09-13T14:11:47.021 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-13T14:11:47.093 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rgw 2024-09-13T14:11:47.476 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:33 ago on Fri 13 Sep 2024 02:10:14 PM UTC. 2024-09-13T14:11:47.569 INFO:teuthology.orchestra.run.smithi184.stdout:Package librbd1-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:11:47.612 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-09-13T14:11:47.613 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-09-13T14:11:47.613 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-09-13T14:11:47.659 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:34 ago on Fri 13 Sep 2024 02:10:13 PM UTC. 2024-09-13T14:11:47.677 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-rados 2024-09-13T14:11:47.750 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-rgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:11:47.794 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-13T14:11:47.794 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-09-13T14:11:47.794 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-13T14:11:47.873 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-cephfs 2024-09-13T14:11:48.246 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:34 ago on Fri 13 Sep 2024 02:10:14 PM UTC. 2024-09-13T14:11:48.336 INFO:teuthology.orchestra.run.smithi184.stdout:Package python3-rados-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:11:48.379 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-09-13T14:11:48.380 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-09-13T14:11:48.380 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-09-13T14:11:48.434 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:35 ago on Fri 13 Sep 2024 02:10:13 PM UTC. 2024-09-13T14:11:48.446 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-rgw 2024-09-13T14:11:48.524 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-cephfs-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:11:48.566 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-13T14:11:48.567 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-09-13T14:11:48.567 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-13T14:11:48.632 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rbd 2024-09-13T14:11:49.014 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:35 ago on Fri 13 Sep 2024 02:10:14 PM UTC. 2024-09-13T14:11:49.104 INFO:teuthology.orchestra.run.smithi184.stdout:Package python3-rgw-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:11:49.146 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-09-13T14:11:49.147 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-09-13T14:11:49.147 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-09-13T14:11:49.199 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:36 ago on Fri 13 Sep 2024 02:10:13 PM UTC. 2024-09-13T14:11:49.210 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-cephfs 2024-09-13T14:11:49.289 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-rbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:11:49.332 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-13T14:11:49.333 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-09-13T14:11:49.333 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-13T14:11:49.393 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-fuse 2024-09-13T14:11:49.769 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:35 ago on Fri 13 Sep 2024 02:10:14 PM UTC. 2024-09-13T14:11:49.858 INFO:teuthology.orchestra.run.smithi184.stdout:Package python3-cephfs-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:11:49.900 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-09-13T14:11:49.901 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-09-13T14:11:49.901 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-09-13T14:11:49.963 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-rbd 2024-09-13T14:11:49.965 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:36 ago on Fri 13 Sep 2024 02:10:13 PM UTC. 2024-09-13T14:11:50.100 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-13T14:11:50.101 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:50.101 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-09-13T14:11:50.101 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:50.101 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-09-13T14:11:50.101 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-fuse x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 82 k 2024-09-13T14:11:50.101 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:50.101 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-09-13T14:11:50.101 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:50.101 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-09-13T14:11:50.102 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:50.102 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 82 k 2024-09-13T14:11:50.102 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 222 k 2024-09-13T14:11:50.102 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-09-13T14:11:50.295 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-fuse-17.2.7-1627.gbbdadfa3.el9.x86_64.rpm 424 kB/s | 82 kB 00:00 2024-09-13T14:11:50.296 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:11:50.296 INFO:teuthology.orchestra.run.smithi151.stdout:Total 421 kB/s | 82 kB 00:00 2024-09-13T14:11:50.296 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-09-13T14:11:50.302 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-09-13T14:11:50.303 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-09-13T14:11:50.366 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-09-13T14:11:50.366 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-09-13T14:11:50.517 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-09-13T14:11:50.526 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:36 ago on Fri 13 Sep 2024 02:10:14 PM UTC. 2024-09-13T14:11:50.616 INFO:teuthology.orchestra.run.smithi184.stdout:Package python3-rbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 is already installed. 2024-09-13T14:11:50.627 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:11:50.658 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-09-13T14:11:50.659 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-09-13T14:11:50.660 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-09-13T14:11:50.721 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install rbd-fuse 2024-09-13T14:11:51.058 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:11:51.292 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:37 ago on Fri 13 Sep 2024 02:10:14 PM UTC. 2024-09-13T14:11:51.321 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:11:51.321 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:51.321 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-09-13T14:11:51.321 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:11:51.321 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:51.321 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-13T14:11:51.426 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-09-13T14:11:51.427 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:51.427 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-09-13T14:11:51.427 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:51.427 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-09-13T14:11:51.427 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-fuse x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 82 k 2024-09-13T14:11:51.427 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:51.427 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-09-13T14:11:51.427 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:51.427 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-09-13T14:11:51.428 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:51.428 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 82 k 2024-09-13T14:11:51.428 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 222 k 2024-09-13T14:11:51.428 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-09-13T14:11:51.546 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-mirror 2024-09-13T14:11:51.613 INFO:teuthology.orchestra.run.smithi184.stdout:rbd-fuse-17.2.7-1627.gbbdadfa3.el9.x86_64.rpm 443 kB/s | 82 kB 00:00 2024-09-13T14:11:51.613 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:11:51.614 INFO:teuthology.orchestra.run.smithi184.stdout:Total 440 kB/s | 82 kB 00:00 2024-09-13T14:11:51.614 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-09-13T14:11:51.621 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-09-13T14:11:51.621 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-09-13T14:11:51.684 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-09-13T14:11:51.684 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-09-13T14:11:51.837 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-09-13T14:11:51.953 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : rbd-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:11:52.129 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:39 ago on Fri 13 Sep 2024 02:10:13 PM UTC. 2024-09-13T14:11:52.267 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-13T14:11:52.268 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:52.268 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-09-13T14:11:52.268 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:52.268 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-09-13T14:11:52.268 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-mirror x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 2.9 M 2024-09-13T14:11:52.268 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:52.269 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-09-13T14:11:52.269 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:52.269 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-09-13T14:11:52.269 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:52.269 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 2.9 M 2024-09-13T14:11:52.269 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 12 M 2024-09-13T14:11:52.269 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-09-13T14:11:52.365 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:11:52.598 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-mirror-17.2.7-1627.gbbdadfa3.el9.x86_64.rpm 8.9 MB/s | 2.9 MB 00:00 2024-09-13T14:11:52.598 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:11:52.599 INFO:teuthology.orchestra.run.smithi151.stdout:Total 8.9 MB/s | 2.9 MB 00:00 2024-09-13T14:11:52.599 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-09-13T14:11:52.608 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-09-13T14:11:52.608 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-09-13T14:11:52.615 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : rbd-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:11:52.615 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:52.616 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-09-13T14:11:52.616 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-fuse-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:11:52.616 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:52.616 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-09-13T14:11:52.687 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-09-13T14:11:52.688 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-09-13T14:11:52.840 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install rbd-mirror 2024-09-13T14:11:53.103 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-09-13T14:11:53.165 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-mirror-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:11:53.190 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:11:53.190 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:11:53.190 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T14:11:53.190 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T14:11:53.190 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T14:11:53.190 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:53.413 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:39 ago on Fri 13 Sep 2024 02:10:14 PM UTC. 2024-09-13T14:11:53.548 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-09-13T14:11:53.549 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:53.549 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2024-09-13T14:11:53.549 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:53.549 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-09-13T14:11:53.549 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-mirror x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 2.9 M 2024-09-13T14:11:53.549 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:53.549 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-09-13T14:11:53.549 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:53.549 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-09-13T14:11:53.550 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:53.550 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 2.9 M 2024-09-13T14:11:53.550 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 12 M 2024-09-13T14:11:53.550 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-09-13T14:11:53.879 INFO:teuthology.orchestra.run.smithi184.stdout:rbd-mirror-17.2.7-1627.gbbdadfa3.el9.x86_64.rpm 8.9 MB/s | 2.9 MB 00:00 2024-09-13T14:11:53.879 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:11:53.880 INFO:teuthology.orchestra.run.smithi184.stdout:Total 8.9 MB/s | 2.9 MB 00:00 2024-09-13T14:11:53.880 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-09-13T14:11:53.889 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-09-13T14:11:53.889 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-09-13T14:11:53.968 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-09-13T14:11:53.969 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-09-13T14:11:54.274 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-mirror-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:11:54.274 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:54.274 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-09-13T14:11:54.274 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-mirror-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:11:54.274 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:54.274 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-13T14:11:54.378 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-09-13T14:11:54.442 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : rbd-mirror-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:11:54.467 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:11:54.467 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:11:54.467 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T14:11:54.467 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T14:11:54.467 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T14:11:54.467 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:54.501 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-nbd 2024-09-13T14:11:55.085 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:42 ago on Fri 13 Sep 2024 02:10:13 PM UTC. 2024-09-13T14:11:55.222 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-13T14:11:55.223 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:55.223 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-09-13T14:11:55.223 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:55.223 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-09-13T14:11:55.223 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-nbd x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 165 k 2024-09-13T14:11:55.224 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:55.224 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-09-13T14:11:55.224 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-13T14:11:55.224 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-09-13T14:11:55.224 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:55.224 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 165 k 2024-09-13T14:11:55.224 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 477 k 2024-09-13T14:11:55.224 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-09-13T14:11:55.434 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-nbd-17.2.7-1627.gbbdadfa3.el9.x86_64.rpm 787 kB/s | 165 kB 00:00 2024-09-13T14:11:55.434 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:11:55.435 INFO:teuthology.orchestra.run.smithi151.stdout:Total 781 kB/s | 165 kB 00:00 2024-09-13T14:11:55.435 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-09-13T14:11:55.442 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-09-13T14:11:55.442 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-09-13T14:11:55.506 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-09-13T14:11:55.506 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-09-13T14:11:55.584 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : rbd-mirror-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:11:55.584 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:55.584 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-09-13T14:11:55.584 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-mirror-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:11:55.584 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:55.584 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-09-13T14:11:55.662 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-09-13T14:11:55.764 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-nbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:11:55.821 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install rbd-nbd 2024-09-13T14:11:56.196 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:11:56.408 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:42 ago on Fri 13 Sep 2024 02:10:14 PM UTC. 2024-09-13T14:11:56.468 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-nbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:11:56.468 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:56.468 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-09-13T14:11:56.468 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-nbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:11:56.468 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:11:56.469 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-13T14:11:56.541 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-09-13T14:11:56.542 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:56.542 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-09-13T14:11:56.543 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:56.543 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-09-13T14:11:56.543 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-nbd x86_64 2:17.2.7-1627.gbbdadfa3.el9 ceph 165 k 2024-09-13T14:11:56.543 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:56.543 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-09-13T14:11:56.543 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-13T14:11:56.543 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-09-13T14:11:56.543 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:56.543 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 165 k 2024-09-13T14:11:56.544 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 477 k 2024-09-13T14:11:56.544 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-09-13T14:11:56.712 DEBUG:teuthology.parallel:result is None 2024-09-13T14:11:56.754 INFO:teuthology.orchestra.run.smithi184.stdout:rbd-nbd-17.2.7-1627.gbbdadfa3.el9.x86_64.rpm 786 kB/s | 165 kB 00:00 2024-09-13T14:11:56.754 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:11:56.755 INFO:teuthology.orchestra.run.smithi184.stdout:Total 780 kB/s | 165 kB 00:00 2024-09-13T14:11:56.755 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-09-13T14:11:56.762 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-09-13T14:11:56.762 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-09-13T14:11:56.825 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-09-13T14:11:56.825 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-09-13T14:11:56.999 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-09-13T14:11:57.108 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : rbd-nbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:11:57.558 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:11:57.805 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : rbd-nbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 1/1 2024-09-13T14:11:57.806 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:57.806 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-09-13T14:11:57.806 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-nbd-2:17.2.7-1627.gbbdadfa3.el9.x86_64 2024-09-13T14:11:57.806 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:11:57.806 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-09-13T14:11:58.024 DEBUG:teuthology.parallel:result is None 2024-09-13T14:11:58.025 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T14:11:58.025 INFO:teuthology.packaging:ref: None 2024-09-13T14:11:58.025 INFO:teuthology.packaging:tag: None 2024-09-13T14:11:58.025 INFO:teuthology.packaging:branch: quincy 2024-09-13T14:11:58.025 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:11:58.025 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-09-13T14:11:58.279 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T14:11:58.301 INFO:teuthology.orchestra.run.smithi151.stdout:17.2.7-1627.gbbdadfa3.el9 2024-09-13T14:11:58.301 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1627.gbbdadfa3.el9 2024-09-13T14:11:58.301 INFO:teuthology.task.install:The correct ceph version 17.2.7-1627.gbbdadfa3 is installed. 2024-09-13T14:11:58.303 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T14:11:58.303 INFO:teuthology.packaging:ref: None 2024-09-13T14:11:58.303 INFO:teuthology.packaging:tag: None 2024-09-13T14:11:58.303 INFO:teuthology.packaging:branch: quincy 2024-09-13T14:11:58.303 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:11:58.303 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-09-13T14:11:58.540 DEBUG:teuthology.orchestra.run.smithi184:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T14:11:58.562 INFO:teuthology.orchestra.run.smithi184.stdout:17.2.7-1627.gbbdadfa3.el9 2024-09-13T14:11:58.562 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1627.gbbdadfa3.el9 2024-09-13T14:11:58.562 INFO:teuthology.task.install:The correct ceph version 17.2.7-1627.gbbdadfa3 is installed. 2024-09-13T14:11:58.564 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-13T14:11:58.564 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-13T14:11:58.564 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T14:11:58.592 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-13T14:11:58.592 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T14:11:58.627 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-13T14:11:58.627 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-13T14:11:58.627 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T14:11:58.657 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T14:11:58.723 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-13T14:11:58.723 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T14:11:58.753 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T14:11:58.818 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-13T14:11:58.818 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-13T14:11:58.818 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T14:11:58.843 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T14:11:58.908 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-13T14:11:58.908 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T14:11:58.933 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T14:11:58.997 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-13T14:11:58.997 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-13T14:11:58.997 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T14:11:59.021 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T14:11:59.086 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-13T14:11:59.086 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T14:11:59.110 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T14:11:59.175 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-13T14:11:59.268 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'create_rbd_pool': True, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'OSD_DOWN', 'mons down', 'mon down', 'MON_DOWN', 'out of quorum', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSDMAP_FLAGS', 'PG_AVAILABILITY'], 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'} 2024-09-13T14:11:59.268 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-09-13T14:11:59.269 INFO:tasks.cephadm:Cluster fsid is 2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:11:59.269 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-13T14:11:59.269 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.151', 'mon.c': '[v2:172.21.15.151:3301,v1:172.21.15.151:6790]', 'mon.b': '172.21.15.184'} 2024-09-13T14:11:59.269 INFO:tasks.cephadm:First mon is mon.a on smithi151 2024-09-13T14:11:59.270 INFO:tasks.cephadm:First mgr is y 2024-09-13T14:11:59.270 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-13T14:11:59.270 DEBUG:teuthology.orchestra.run.smithi151:> sudo hostname $(hostname -s) 2024-09-13T14:11:59.295 DEBUG:teuthology.orchestra.run.smithi184:> sudo hostname $(hostname -s) 2024-09-13T14:11:59.320 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-09-13T14:11:59.321 DEBUG:teuthology.orchestra.run.smithi151:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-13T14:11:59.470 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r--. 1 ubuntu ubuntu 378585 Sep 13 14:11 /home/ubuntu/cephtest/cephadm 2024-09-13T14:11:59.470 DEBUG:teuthology.orchestra.run.smithi184:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-13T14:11:59.721 INFO:teuthology.orchestra.run.smithi184.stdout:-rw-r--r--. 1 ubuntu ubuntu 378585 Sep 13 14:11 /home/ubuntu/cephtest/cephadm 2024-09-13T14:11:59.721 DEBUG:teuthology.orchestra.run.smithi151:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-13T14:11:59.738 DEBUG:teuthology.orchestra.run.smithi184:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-13T14:11:59.764 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-09-13T14:11:59.764 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-09-13T14:11:59.780 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-09-13T14:12:01.305 INFO:teuthology.orchestra.run.smithi151.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-09-13T14:12:01.476 INFO:teuthology.orchestra.run.smithi184.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-09-13T14:12:22.708 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:12:22.708 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_version": "ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)", 2024-09-13T14:12:22.708 INFO:teuthology.orchestra.run.smithi184.stdout: "image_id": "a994b00deb3a4a8eeb484a456d3f3207aae0401a31d251b199b2c9dbc1c4036c", 2024-09-13T14:12:22.708 INFO:teuthology.orchestra.run.smithi184.stdout: "repo_digests": [ 2024-09-13T14:12:22.708 INFO:teuthology.orchestra.run.smithi184.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be" 2024-09-13T14:12:22.709 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:12:22.709 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:12:26.915 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:12:26.915 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph_version": "ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)", 2024-09-13T14:12:26.915 INFO:teuthology.orchestra.run.smithi151.stdout: "image_id": "a994b00deb3a4a8eeb484a456d3f3207aae0401a31d251b199b2c9dbc1c4036c", 2024-09-13T14:12:26.915 INFO:teuthology.orchestra.run.smithi151.stdout: "repo_digests": [ 2024-09-13T14:12:26.915 INFO:teuthology.orchestra.run.smithi151.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be" 2024-09-13T14:12:26.915 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:12:26.915 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:12:26.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /etc/ceph 2024-09-13T14:12:26.978 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /etc/ceph 2024-09-13T14:12:27.011 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 777 /etc/ceph 2024-09-13T14:12:27.036 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 777 /etc/ceph 2024-09-13T14:12:27.080 INFO:tasks.cephadm:Writing seed config... 2024-09-13T14:12:27.081 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-09-13T14:12:27.081 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-09-13T14:12:27.081 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-13T14:12:27.081 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-13T14:12:27.082 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-13T14:12:27.082 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-09-13T14:12:27.082 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-13T14:12:27.082 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-13T14:12:27.082 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-13T14:12:27.082 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-13T14:12:27.082 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-13T14:12:27.083 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-13T14:12:27.083 DEBUG:teuthology.orchestra.run.smithi151:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-13T14:12:27.098 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 2401fc50-71da-11ef-bceb-c7b262605968 mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-13T14:12:27.099 DEBUG:teuthology.orchestra.run.smithi151:mon.a> sudo journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@mon.a.service 2024-09-13T14:12:27.141 DEBUG:teuthology.orchestra.run.smithi151:mgr.y> sudo journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@mgr.y.service 2024-09-13T14:12:27.185 INFO:tasks.cephadm:Bootstrapping... 2024-09-13T14:12:27.185 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid 2401fc50-71da-11ef-bceb-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.151 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-13T14:12:27.386 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:12:27.386 INFO:teuthology.orchestra.run.smithi151.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', '2401fc50-71da-11ef-bceb-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.151', '--skip-admin-label'] 2024-09-13T14:12:27.409 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stdout 5.2.2 2024-09-13T14:12:27.410 INFO:teuthology.orchestra.run.smithi151.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-13T14:12:27.410 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying podman|docker is present... 2024-09-13T14:12:27.432 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stdout 5.2.2 2024-09-13T14:12:27.432 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying lvm2 is present... 2024-09-13T14:12:27.432 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying time synchronization is in place... 2024-09-13T14:12:27.440 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-13T14:12:27.441 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-13T14:12:27.447 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-13T14:12:27.447 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout inactive 2024-09-13T14:12:27.454 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout enabled 2024-09-13T14:12:27.461 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout active 2024-09-13T14:12:27.461 INFO:teuthology.orchestra.run.smithi151.stdout:Unit chronyd.service is enabled and running 2024-09-13T14:12:27.461 INFO:teuthology.orchestra.run.smithi151.stdout:Repeating the final host check... 2024-09-13T14:12:27.484 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stdout 5.2.2 2024-09-13T14:12:27.484 INFO:teuthology.orchestra.run.smithi151.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-13T14:12:27.484 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl is present 2024-09-13T14:12:27.484 INFO:teuthology.orchestra.run.smithi151.stdout:lvcreate is present 2024-09-13T14:12:27.491 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-13T14:12:27.491 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-13T14:12:27.498 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-13T14:12:27.498 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout inactive 2024-09-13T14:12:27.505 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout enabled 2024-09-13T14:12:27.511 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout active 2024-09-13T14:12:27.511 INFO:teuthology.orchestra.run.smithi151.stdout:Unit chronyd.service is enabled and running 2024-09-13T14:12:27.511 INFO:teuthology.orchestra.run.smithi151.stdout:Host looks OK 2024-09-13T14:12:27.511 INFO:teuthology.orchestra.run.smithi151.stdout:Cluster fsid: 2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:12:27.511 INFO:teuthology.orchestra.run.smithi151.stdout:Acquiring lock 140510608712608 on /run/cephadm/2401fc50-71da-11ef-bceb-c7b262605968.lock 2024-09-13T14:12:27.512 INFO:teuthology.orchestra.run.smithi151.stdout:Lock 140510608712608 acquired on /run/cephadm/2401fc50-71da-11ef-bceb-c7b262605968.lock 2024-09-13T14:12:27.512 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying IP 172.21.15.151 port 3300 ... 2024-09-13T14:12:27.512 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying IP 172.21.15.151 port 6789 ... 2024-09-13T14:12:27.512 INFO:teuthology.orchestra.run.smithi151.stdout:Base mon IP(s) is [172.21.15.151:3300, 172.21.15.151:6789], mon addrv is [v2:172.21.15.151:3300,v1:172.21.15.151:6789] 2024-09-13T14:12:27.516 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.151 metric 100 2024-09-13T14:12:27.516 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.151 metric 100 2024-09-13T14:12:27.518 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-13T14:12:27.519 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-13T14:12:27.519 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-13T14:12:27.521 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-13T14:12:27.521 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-13T14:12:27.521 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-13T14:12:27.521 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-13T14:12:27.522 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:6f63/64 scope link noprefixroute 2024-09-13T14:12:27.522 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-13T14:12:27.522 INFO:teuthology.orchestra.run.smithi151.stdout:Mon IP `172.21.15.151` is in CIDR network `172.21.0.0/20` 2024-09-13T14:12:27.522 INFO:teuthology.orchestra.run.smithi151.stdout:Mon IP `172.21.15.151` is in CIDR network `172.21.0.0/20` 2024-09-13T14:12:27.522 INFO:teuthology.orchestra.run.smithi151.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-13T14:12:27.523 INFO:teuthology.orchestra.run.smithi151.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-13T14:12:27.523 INFO:teuthology.orchestra.run.smithi151.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-09-13T14:12:27.915 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stdout a994b00deb3a4a8eeb484a456d3f3207aae0401a31d251b199b2c9dbc1c4036c 2024-09-13T14:12:27.915 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-09-13T14:12:27.915 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stderr Getting image source signatures 2024-09-13T14:12:27.915 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stderr Copying blob sha256:ca308b3f556da579f99cd79db684ebbf678db4c29a649b66aaf6f99bc8124cd6 2024-09-13T14:12:27.915 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-13T14:12:27.915 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stderr Copying config sha256:a994b00deb3a4a8eeb484a456d3f3207aae0401a31d251b199b2c9dbc1c4036c 2024-09-13T14:12:27.915 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-13T14:12:28.869 INFO:teuthology.orchestra.run.smithi151.stdout:ceph: stdout ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable) 2024-09-13T14:12:28.869 INFO:teuthology.orchestra.run.smithi151.stdout:Ceph version: ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable) 2024-09-13T14:12:28.869 INFO:teuthology.orchestra.run.smithi151.stdout:Extracting ceph user uid/gid from container image... 2024-09-13T14:12:29.899 INFO:teuthology.orchestra.run.smithi151.stdout:stat: stdout 167 167 2024-09-13T14:12:29.900 INFO:teuthology.orchestra.run.smithi151.stdout:Creating initial keys... 2024-09-13T14:12:30.897 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph-authtool: stdout AQBOSORmF/DDDhAAze8Pg5qMDNjd3NzMZ8r3kA== 2024-09-13T14:12:31.968 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph-authtool: stdout AQBPSORm/IaWDhAAfGUCGRF0zYbj7aZ7WbKgjw== 2024-09-13T14:12:32.966 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph-authtool: stdout AQBQSORmNkfaExAAshNKudRq5S31yfwhQg0pFg== 2024-09-13T14:12:32.967 INFO:teuthology.orchestra.run.smithi151.stdout:Creating initial monmap... 2024-09-13T14:12:33.955 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-13T14:12:33.955 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-09-13T14:12:33.955 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:12:33.955 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-13T14:12:33.955 INFO:teuthology.orchestra.run.smithi151.stdout:monmaptool for a [v2:172.21.15.151:3300,v1:172.21.15.151:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-13T14:12:33.955 INFO:teuthology.orchestra.run.smithi151.stdout:setting min_mon_release = octopus 2024-09-13T14:12:33.955 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/monmaptool: set fsid to 2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:12:33.955 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-13T14:12:33.956 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:12:33.956 INFO:teuthology.orchestra.run.smithi151.stdout:Creating mon... 2024-09-13T14:12:34.869 INFO:teuthology.orchestra.run.smithi151.stdout:create mon.a on 2024-09-13T14:12:35.069 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-13T14:12:35.209 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-13T14:12:35.361 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-2401fc50-71da-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-2401fc50-71da-11ef-bceb-c7b262605968.target. 2024-09-13T14:12:35.361 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-2401fc50-71da-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-2401fc50-71da-11ef-bceb-c7b262605968.target. 2024-09-13T14:12:35.529 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-2401fc50-71da-11ef-bceb-c7b262605968@mon.a 2024-09-13T14:12:35.530 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Failed to reset failed state of unit ceph-2401fc50-71da-11ef-bceb-c7b262605968@mon.a.service: Unit ceph-2401fc50-71da-11ef-bceb-c7b262605968@mon.a.service not loaded. 2024-09-13T14:12:35.689 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-2401fc50-71da-11ef-bceb-c7b262605968.target.wants/ceph-2401fc50-71da-11ef-bceb-c7b262605968@mon.a.service → /etc/systemd/system/ceph-2401fc50-71da-11ef-bceb-c7b262605968@.service. 2024-09-13T14:12:36.032 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:35 smithi151 podman[29302]: 2024-09-13 14:12:35.829766225 +0000 UTC m=+0.015400656 image pull a994b00deb3a4a8eeb484a456d3f3207aae0401a31d251b199b2c9dbc1c4036c quay.ceph.io/ceph-ci/ceph:quincy 2024-09-13T14:12:36.251 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-13T14:12:36.251 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout disabled 2024-09-13T14:12:36.258 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-13T14:12:36.258 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout inactive 2024-09-13T14:12:36.259 INFO:teuthology.orchestra.run.smithi151.stdout:firewalld.service is not enabled 2024-09-13T14:12:36.259 INFO:teuthology.orchestra.run.smithi151.stdout:Not possible to enable service . firewalld.service is not available 2024-09-13T14:12:36.259 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for mon to start... 2024-09-13T14:12:36.259 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for mon... 2024-09-13T14:12:36.383 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:36 smithi151 podman[29302]: 2024-09-13 14:12:36.130161691 +0000 UTC m=+0.315796112 container init 316422e66737cf17de0bb1890b4fef044e5e23b2b256d6ea4c3cc32698713e29 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=quincy-bbdadfa, GIT_CLEAN=True) 2024-09-13T14:12:36.383 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:36 smithi151 podman[29302]: 2024-09-13 14:12:36.135161196 +0000 UTC m=+0.320795617 container start 316422e66737cf17de0bb1890b4fef044e5e23b2b256d6ea4c3cc32698713e29 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=quincy-bbdadfa, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2) 2024-09-13T14:12:36.383 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:36 smithi151 ceph-mon[29340]: mkfs 2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:12:36.383 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:36 smithi151 ceph-mon[29340]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-13T14:12:36.383 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:36 smithi151 bash[29302]: 316422e66737cf17de0bb1890b4fef044e5e23b2b256d6ea4c3cc32698713e29 2024-09-13T14:12:36.383 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:36 smithi151 systemd[1]: Started Ceph mon.a for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:12:37.285 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:37 smithi151 ceph-mon[29340]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-13T14:12:37.285 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:37 smithi151 ceph-mon[29340]: monmap e1: 1 mons at {a=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T14:12:37.286 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:37 smithi151 ceph-mon[29340]: fsmap 2024-09-13T14:12:37.286 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:37 smithi151 ceph-mon[29340]: osdmap e1: 0 total, 0 up, 0 in 2024-09-13T14:12:37.286 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:37 smithi151 ceph-mon[29340]: mgrmap e1: no daemons active 2024-09-13T14:12:37.286 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:37 smithi151 ceph-mon[29340]: from='client.? 172.21.15.151:0/925678277' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-13T14:12:37.289 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout cluster: 2024-09-13T14:12:37.289 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout id: 2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:12:37.289 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-13T14:12:37.289 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-09-13T14:12:37.289 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout services: 2024-09-13T14:12:37.289 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.61755s) 2024-09-13T14:12:37.289 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-13T14:12:37.289 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-13T14:12:37.289 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-09-13T14:12:37.289 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout data: 2024-09-13T14:12:37.290 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-13T14:12:37.290 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-13T14:12:37.290 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-13T14:12:37.290 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout pgs: 2024-09-13T14:12:37.290 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-09-13T14:12:37.290 INFO:teuthology.orchestra.run.smithi151.stdout:mon is available 2024-09-13T14:12:37.290 INFO:teuthology.orchestra.run.smithi151.stdout:Assimilating anything we can from ceph.conf... 2024-09-13T14:12:38.372 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-09-13T14:12:38.372 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout [global] 2024-09-13T14:12:38.373 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout fsid = 2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:12:38.373 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-13T14:12:38.373 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.151:3300,v1:172.21.15.151:6789] 2024-09-13T14:12:38.373 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-13T14:12:38.373 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-13T14:12:38.373 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-13T14:12:38.373 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-13T14:12:38.373 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-09-13T14:12:38.373 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-13T14:12:38.373 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-13T14:12:38.373 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-09-13T14:12:38.373 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout [osd] 2024-09-13T14:12:38.373 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-13T14:12:38.374 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-13T14:12:38.374 INFO:teuthology.orchestra.run.smithi151.stdout:Generating new minimal ceph.conf... 2024-09-13T14:12:38.492 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:38 smithi151 ceph-mon[29340]: from='client.? 172.21.15.151:0/3854245431' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-13T14:12:38.493 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:38 smithi151 ceph-mon[29340]: from='client.? 172.21.15.151:0/3854245431' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-13T14:12:39.458 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:39 smithi151 ceph-mon[29340]: from='client.? 172.21.15.151:0/2592384778' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:12:39.461 INFO:teuthology.orchestra.run.smithi151.stdout:Restarting the monitor... 2024-09-13T14:12:39.739 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:39 smithi151 systemd[1]: Stopping Ceph mon.a for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:12:40.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:39 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a[29317]: 2024-09-13T14:12:39.737+0000 7fb253c70640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:12:40.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:39 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a[29317]: 2024-09-13T14:12:39.738+0000 7fb253c70640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-13T14:12:40.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:39 smithi151 podman[29657]: 2024-09-13 14:12:39.871693969 +0000 UTC m=+0.249285980 container died 316422e66737cf17de0bb1890b4fef044e5e23b2b256d6ea4c3cc32698713e29 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-bbdadfa, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-13T14:12:42.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:41 smithi151 podman[29657]: 2024-09-13 14:12:41.984042482 +0000 UTC m=+2.361634495 container cleanup 316422e66737cf17de0bb1890b4fef044e5e23b2b256d6ea4c3cc32698713e29 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=quincy-bbdadfa, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-13T14:12:42.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:41 smithi151 bash[29657]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a 2024-09-13T14:12:44.295 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:43 smithi151 podman[29674]: 2024-09-13 14:12:43.953193895 +0000 UTC m=+4.082747855 container remove 316422e66737cf17de0bb1890b4fef044e5e23b2b256d6ea4c3cc32698713e29 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-bbdadfa, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731) 2024-09-13T14:12:44.296 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:44 smithi151 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-13T14:12:44.296 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:44 smithi151 systemd[1]: Stopped Ceph mon.a for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:12:44.296 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:44 smithi151 systemd[1]: Starting Ceph mon.a for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:12:44.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:44 smithi151 podman[29769]: 2024-09-13 14:12:44.197944033 +0000 UTC m=+0.015703508 image pull a994b00deb3a4a8eeb484a456d3f3207aae0401a31d251b199b2c9dbc1c4036c quay.ceph.io/ceph-ci/ceph:quincy 2024-09-13T14:12:45.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:45 smithi151 podman[29769]: 2024-09-13 14:12:45.476646722 +0000 UTC m=+1.294406181 container create ffc441eb291c47c362a7837504a281e149b24a73556712e8015912fa0696371d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=quincy-bbdadfa, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-13T14:12:48.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 podman[29769]: 2024-09-13 14:12:47.651573345 +0000 UTC m=+3.469332812 container init ffc441eb291c47c362a7837504a281e149b24a73556712e8015912fa0696371d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-bbdadfa, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-13T14:12:48.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 podman[29769]: 2024-09-13 14:12:47.657467252 +0000 UTC m=+3.475226712 container start ffc441eb291c47c362a7837504a281e149b24a73556712e8015912fa0696371d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=quincy-bbdadfa, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD) 2024-09-13T14:12:48.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: set uid:gid to 167:167 (ceph:ceph) 2024-09-13T14:12:48.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable), process ceph-mon, pid 7 2024-09-13T14:12:48.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: pidfile_write: ignore empty --pid-file 2024-09-13T14:12:48.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: load: jerasure load: lrc 2024-09-13T14:12:48.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: RocksDB version: 6.15.5 2024-09-13T14:12:48.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-13T14:12:48.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Compile date Sep 12 2024 2024-09-13T14:12:48.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: DB SUMMARY 2024-09-13T14:12:48.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: DB Session ID: KNRVPT03LKAZG2GJIUQR 2024-09-13T14:12:48.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: CURRENT file: CURRENT 2024-09-13T14:12:48.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: IDENTITY file: IDENTITY 2024-09-13T14:12:48.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-09-13T14:12:48.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-13T14:12:48.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 83712 ; 2024-09-13T14:12:48.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.error_if_exists: 0 2024-09-13T14:12:48.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.create_if_missing: 0 2024-09-13T14:12:48.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.paranoid_checks: 1 2024-09-13T14:12:48.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-13T14:12:48.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.env: 0x555902d13440 2024-09-13T14:12:48.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.fs: Posix File System 2024-09-13T14:12:48.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.info_log: 0x55590408cec0 2024-09-13T14:12:48.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.max_file_opening_threads: 16 2024-09-13T14:12:48.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.statistics: (nil) 2024-09-13T14:12:48.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.use_fsync: 0 2024-09-13T14:12:48.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.max_log_file_size: 0 2024-09-13T14:12:48.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-13T14:12:48.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-13T14:12:48.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.keep_log_file_num: 1000 2024-09-13T14:12:48.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.recycle_log_file_num: 0 2024-09-13T14:12:48.112 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.allow_fallocate: 1 2024-09-13T14:12:48.112 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.allow_mmap_reads: 0 2024-09-13T14:12:48.112 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.allow_mmap_writes: 0 2024-09-13T14:12:48.112 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.use_direct_reads: 0 2024-09-13T14:12:48.112 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-13T14:12:48.112 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.create_missing_column_families: 0 2024-09-13T14:12:48.112 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.db_log_dir: 2024-09-13T14:12:48.112 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-09-13T14:12:48.112 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-13T14:12:48.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-13T14:12:48.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-13T14:12:48.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-13T14:12:48.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-13T14:12:48.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-13T14:12:48.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.advise_random_on_open: 1 2024-09-13T14:12:48.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.db_write_buffer_size: 0 2024-09-13T14:12:48.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.write_buffer_manager: 0x55590414a780 2024-09-13T14:12:48.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-13T14:12:48.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-13T14:12:48.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-13T14:12:48.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-13T14:12:48.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.rate_limiter: (nil) 2024-09-13T14:12:48.114 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-13T14:12:48.114 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.wal_recovery_mode: 2 2024-09-13T14:12:48.114 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.enable_thread_tracking: 0 2024-09-13T14:12:48.114 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.enable_pipelined_write: 0 2024-09-13T14:12:48.114 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.unordered_write: 0 2024-09-13T14:12:48.114 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-13T14:12:48.114 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-13T14:12:48.114 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-13T14:12:48.114 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-13T14:12:48.114 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.row_cache: None 2024-09-13T14:12:48.114 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.wal_filter: None 2024-09-13T14:12:48.115 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-13T14:12:48.115 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.allow_ingest_behind: 0 2024-09-13T14:12:48.115 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.preserve_deletes: 0 2024-09-13T14:12:48.116 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.two_write_queues: 0 2024-09-13T14:12:48.116 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.manual_wal_flush: 0 2024-09-13T14:12:48.116 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.atomic_flush: 0 2024-09-13T14:12:48.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-13T14:12:48.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-13T14:12:48.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-13T14:12:48.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.log_readahead_size: 0 2024-09-13T14:12:48.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-13T14:12:48.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.best_efforts_recovery: 0 2024-09-13T14:12:48.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-13T14:12:48.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-13T14:12:48.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.allow_data_in_errors: 0 2024-09-13T14:12:48.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.db_host_id: __hostname__ 2024-09-13T14:12:48.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.max_background_jobs: 2 2024-09-13T14:12:48.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.max_background_compactions: -1 2024-09-13T14:12:48.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.max_subcompactions: 1 2024-09-13T14:12:48.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-13T14:12:48.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-13T14:12:48.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-13T14:12:48.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.max_total_wal_size: 0 2024-09-13T14:12:48.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-13T14:12:48.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-13T14:12:48.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-13T14:12:48.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-13T14:12:48.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.max_open_files: -1 2024-09-13T14:12:48.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.bytes_per_sync: 0 2024-09-13T14:12:48.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-13T14:12:48.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-13T14:12:48.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.compaction_readahead_size: 0 2024-09-13T14:12:48.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.max_background_flushes: -1 2024-09-13T14:12:48.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Compression algorithms supported: 2024-09-13T14:12:48.119 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: kZSTD supported: 0 2024-09-13T14:12:48.119 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: kXpressCompression supported: 0 2024-09-13T14:12:48.119 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: kBZip2Compression supported: 0 2024-09-13T14:12:48.119 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-13T14:12:48.119 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: kLZ4Compression supported: 1 2024-09-13T14:12:48.119 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: kZlibCompression supported: 1 2024-09-13T14:12:48.119 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: kLZ4HCCompression supported: 1 2024-09-13T14:12:48.119 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: kSnappyCompression supported: 1 2024-09-13T14:12:48.119 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-13T14:12:48.119 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-09-13T14:12:48.119 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-09-13T14:12:48.119 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-13T14:12:48.119 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.merge_operator: 2024-09-13T14:12:48.119 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.compaction_filter: None 2024-09-13T14:12:48.120 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.compaction_filter_factory: None 2024-09-13T14:12:48.120 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.sst_partitioner_factory: None 2024-09-13T14:12:48.120 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-13T14:12:48.120 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-13T14:12:48.120 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555904033a38) 2024-09-13T14:12:48.120 INFO:journalctl@ceph.mon.a.smithi151.stdout: cache_index_and_filter_blocks: 1 2024-09-13T14:12:48.120 INFO:journalctl@ceph.mon.a.smithi151.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-13T14:12:48.120 INFO:journalctl@ceph.mon.a.smithi151.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-13T14:12:48.120 INFO:journalctl@ceph.mon.a.smithi151.stdout: pin_top_level_index_and_filter: 1 2024-09-13T14:12:48.120 INFO:journalctl@ceph.mon.a.smithi151.stdout: index_type: 0 2024-09-13T14:12:48.120 INFO:journalctl@ceph.mon.a.smithi151.stdout: data_block_index_type: 0 2024-09-13T14:12:48.120 INFO:journalctl@ceph.mon.a.smithi151.stdout: index_shortening: 1 2024-09-13T14:12:48.120 INFO:journalctl@ceph.mon.a.smithi151.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-13T14:12:48.121 INFO:journalctl@ceph.mon.a.smithi151.stdout: hash_index_allow_collision: 1 2024-09-13T14:12:48.121 INFO:journalctl@ceph.mon.a.smithi151.stdout: checksum: 1 2024-09-13T14:12:48.121 INFO:journalctl@ceph.mon.a.smithi151.stdout: no_block_cache: 0 2024-09-13T14:12:48.121 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_cache: 0x555904069350 2024-09-13T14:12:48.121 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_cache_name: BinnedLRUCache 2024-09-13T14:12:48.121 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_cache_options: 2024-09-13T14:12:48.121 INFO:journalctl@ceph.mon.a.smithi151.stdout: capacity : 536870912 2024-09-13T14:12:48.121 INFO:journalctl@ceph.mon.a.smithi151.stdout: num_shard_bits : 4 2024-09-13T14:12:48.121 INFO:journalctl@ceph.mon.a.smithi151.stdout: strict_capacity_limit : 0 2024-09-13T14:12:48.121 INFO:journalctl@ceph.mon.a.smithi151.stdout: high_pri_pool_ratio: 0.000 2024-09-13T14:12:48.121 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_cache_compressed: (nil) 2024-09-13T14:12:48.121 INFO:journalctl@ceph.mon.a.smithi151.stdout: persistent_cache: (nil) 2024-09-13T14:12:48.121 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_size: 4096 2024-09-13T14:12:48.122 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_size_deviation: 10 2024-09-13T14:12:48.122 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_restart_interval: 16 2024-09-13T14:12:48.122 INFO:journalctl@ceph.mon.a.smithi151.stdout: index_block_restart_interval: 1 2024-09-13T14:12:48.122 INFO:journalctl@ceph.mon.a.smithi151.stdout: metadata_block_size: 4096 2024-09-13T14:12:48.122 INFO:journalctl@ceph.mon.a.smithi151.stdout: partition_filters: 0 2024-09-13T14:12:48.122 INFO:journalctl@ceph.mon.a.smithi151.stdout: use_delta_encoding: 1 2024-09-13T14:12:48.122 INFO:journalctl@ceph.mon.a.smithi151.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-13T14:12:48.122 INFO:journalctl@ceph.mon.a.smithi151.stdout: whole_key_filtering: 1 2024-09-13T14:12:48.122 INFO:journalctl@ceph.mon.a.smithi151.stdout: verify_compression: 0 2024-09-13T14:12:48.122 INFO:journalctl@ceph.mon.a.smithi151.stdout: read_amp_bytes_per_bit: 0 2024-09-13T14:12:48.122 INFO:journalctl@ceph.mon.a.smithi151.stdout: format_version: 4 2024-09-13T14:12:48.122 INFO:journalctl@ceph.mon.a.smithi151.stdout: enable_index_compression: 1 2024-09-13T14:12:48.122 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_align: 0 2024-09-13T14:12:48.122 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.write_buffer_size: 33554432 2024-09-13T14:12:48.123 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.max_write_buffer_number: 2 2024-09-13T14:12:48.123 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.compression: NoCompression 2024-09-13T14:12:48.123 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.bottommost_compression: Disabled 2024-09-13T14:12:48.123 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.prefix_extractor: nullptr 2024-09-13T14:12:48.123 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-13T14:12:48.123 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.num_levels: 7 2024-09-13T14:12:48.123 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-13T14:12:48.123 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-13T14:12:48.123 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-13T14:12:48.123 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-13T14:12:48.123 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-13T14:12:48.123 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-13T14:12:48.123 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-13T14:12:48.123 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:12:48.124 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-13T14:12:48.124 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-13T14:12:48.124 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-13T14:12:48.124 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.compression_opts.level: 32767 2024-09-13T14:12:48.124 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.compression_opts.strategy: 0 2024-09-13T14:12:48.124 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-13T14:12:48.124 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:12:48.124 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-13T14:12:48.124 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.compression_opts.enabled: false 2024-09-13T14:12:48.124 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-13T14:12:48.124 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-13T14:12:48.124 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-13T14:12:48.124 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.target_file_size_base: 67108864 2024-09-13T14:12:48.124 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-13T14:12:48.125 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-13T14:12:48.125 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-13T14:12:48.125 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-13T14:12:48.125 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-13T14:12:48.125 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-13T14:12:48.125 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-13T14:12:48.125 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-13T14:12:48.125 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-13T14:12:48.125 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-13T14:12:48.125 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-13T14:12:48.125 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-13T14:12:48.125 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-13T14:12:48.125 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.arena_block_size: 4194304 2024-09-13T14:12:48.126 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-13T14:12:48.126 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-13T14:12:48.126 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-13T14:12:48.126 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.disable_auto_compactions: 0 2024-09-13T14:12:48.126 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-13T14:12:48.126 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-13T14:12:48.126 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-13T14:12:48.126 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-13T14:12:48.126 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-13T14:12:48.126 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-13T14:12:48.126 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-13T14:12:48.126 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-13T14:12:48.126 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-13T14:12:48.126 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-13T14:12:48.126 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-13T14:12:48.127 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.inplace_update_support: 0 2024-09-13T14:12:48.127 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-13T14:12:48.127 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-13T14:12:48.127 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-13T14:12:48.127 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-13T14:12:48.127 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.bloom_locality: 0 2024-09-13T14:12:48.127 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.max_successive_merges: 0 2024-09-13T14:12:48.127 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-13T14:12:48.127 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.paranoid_file_checks: 0 2024-09-13T14:12:48.127 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.force_consistency_checks: 1 2024-09-13T14:12:48.127 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.report_bg_io_stats: 0 2024-09-13T14:12:48.127 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.ttl: 2592000 2024-09-13T14:12:48.127 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-13T14:12:48.127 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.enable_blob_files: false 2024-09-13T14:12:48.128 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.min_blob_size: 0 2024-09-13T14:12:48.128 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.blob_file_size: 268435456 2024-09-13T14:12:48.128 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-13T14:12:48.128 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-13T14:12:48.128 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-13T14:12:48.128 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-13T14:12:48.128 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-09-13T14:12:48.128 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-09-13T14:12:48.128 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726236767686070, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-09-13T14:12:48.128 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-09-13T14:12:48.128 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-13T14:12:48.128 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726236767686906, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 80285, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 78527, "index_size": 220, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10130, "raw_average_key_size": 47, "raw_value_size": 72740, "raw_average_value_size": 341, "num_data_blocks": 10, "num_entries": 213, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1726236767, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "a30cc2d4-5376-406c-97fc-d2c7b106f2f7", "db_session_id": "KNRVPT03LKAZG2GJIUQR"}} 2024-09-13T14:12:48.128 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-09-13T14:12:48.129 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726236767687369, "job": 1, "event": "recovery_finished"} 2024-09-13T14:12:48.129 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-13T14:12:48.129 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55590409c700 2024-09-13T14:12:48.129 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: DB pointer 0x55590406c000 2024-09-13T14:12:48.129 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-09-13T14:12:48.129 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-13T14:12:48.129 INFO:journalctl@ceph.mon.a.smithi151.stdout: ** DB Stats ** 2024-09-13T14:12:48.129 INFO:journalctl@ceph.mon.a.smithi151.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:12:48.129 INFO:journalctl@ceph.mon.a.smithi151.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-13T14:12:48.129 INFO:journalctl@ceph.mon.a.smithi151.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:12:48.129 INFO:journalctl@ceph.mon.a.smithi151.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:12:48.129 INFO:journalctl@ceph.mon.a.smithi151.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-13T14:12:48.129 INFO:journalctl@ceph.mon.a.smithi151.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-13T14:12:48.129 INFO:journalctl@ceph.mon.a.smithi151.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:12:48.130 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-09-13T14:12:48.130 INFO:journalctl@ceph.mon.a.smithi151.stdout: ** Compaction Stats [default] ** 2024-09-13T14:12:48.130 INFO:journalctl@ceph.mon.a.smithi151.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-13T14:12:48.130 INFO:journalctl@ceph.mon.a.smithi151.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T14:12:48.130 INFO:journalctl@ceph.mon.a.smithi151.stdout: L0 2/0 80.19 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 130.0 0.00 0.00 1 0.001 0 0 2024-09-13T14:12:48.130 INFO:journalctl@ceph.mon.a.smithi151.stdout: Sum 2/0 80.19 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 130.0 0.00 0.00 1 0.001 0 0 2024-09-13T14:12:48.130 INFO:journalctl@ceph.mon.a.smithi151.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 130.0 0.00 0.00 1 0.001 0 0 2024-09-13T14:12:48.130 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-09-13T14:12:48.130 INFO:journalctl@ceph.mon.a.smithi151.stdout: ** Compaction Stats [default] ** 2024-09-13T14:12:48.130 INFO:journalctl@ceph.mon.a.smithi151.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-13T14:12:48.130 INFO:journalctl@ceph.mon.a.smithi151.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T14:12:48.130 INFO:journalctl@ceph.mon.a.smithi151.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 130.0 0.00 0.00 1 0.001 0 0 2024-09-13T14:12:48.130 INFO:journalctl@ceph.mon.a.smithi151.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:12:48.130 INFO:journalctl@ceph.mon.a.smithi151.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-13T14:12:48.131 INFO:journalctl@ceph.mon.a.smithi151.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-13T14:12:48.131 INFO:journalctl@ceph.mon.a.smithi151.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-13T14:12:48.131 INFO:journalctl@ceph.mon.a.smithi151.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-13T14:12:48.131 INFO:journalctl@ceph.mon.a.smithi151.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-13T14:12:48.131 INFO:journalctl@ceph.mon.a.smithi151.stdout: Cumulative compaction: 0.00 GB write, 19.93 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:12:48.131 INFO:journalctl@ceph.mon.a.smithi151.stdout: Interval compaction: 0.00 GB write, 19.93 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:12:48.131 INFO:journalctl@ceph.mon.a.smithi151.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-13T14:12:48.131 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-09-13T14:12:48.131 INFO:journalctl@ceph.mon.a.smithi151.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-13T14:12:48.131 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-09-13T14:12:48.131 INFO:journalctl@ceph.mon.a.smithi151.stdout: ** Compaction Stats [default] ** 2024-09-13T14:12:48.131 INFO:journalctl@ceph.mon.a.smithi151.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-13T14:12:48.131 INFO:journalctl@ceph.mon.a.smithi151.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T14:12:48.131 INFO:journalctl@ceph.mon.a.smithi151.stdout: L0 2/0 80.19 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 130.0 0.00 0.00 1 0.001 0 0 2024-09-13T14:12:48.132 INFO:journalctl@ceph.mon.a.smithi151.stdout: Sum 2/0 80.19 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 130.0 0.00 0.00 1 0.001 0 0 2024-09-13T14:12:48.132 INFO:journalctl@ceph.mon.a.smithi151.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-13T14:12:48.132 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-09-13T14:12:48.132 INFO:journalctl@ceph.mon.a.smithi151.stdout: ** Compaction Stats [default] ** 2024-09-13T14:12:48.132 INFO:journalctl@ceph.mon.a.smithi151.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-13T14:12:48.132 INFO:journalctl@ceph.mon.a.smithi151.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T14:12:48.132 INFO:journalctl@ceph.mon.a.smithi151.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 130.0 0.00 0.00 1 0.001 0 0 2024-09-13T14:12:48.132 INFO:journalctl@ceph.mon.a.smithi151.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:12:48.132 INFO:journalctl@ceph.mon.a.smithi151.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-13T14:12:48.132 INFO:journalctl@ceph.mon.a.smithi151.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-13T14:12:48.132 INFO:journalctl@ceph.mon.a.smithi151.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-13T14:12:48.132 INFO:journalctl@ceph.mon.a.smithi151.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-13T14:12:48.132 INFO:journalctl@ceph.mon.a.smithi151.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-13T14:12:48.132 INFO:journalctl@ceph.mon.a.smithi151.stdout: Cumulative compaction: 0.00 GB write, 19.75 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:12:48.133 INFO:journalctl@ceph.mon.a.smithi151.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:12:48.133 INFO:journalctl@ceph.mon.a.smithi151.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-13T14:12:48.133 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-09-13T14:12:48.133 INFO:journalctl@ceph.mon.a.smithi151.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-13T14:12:48.133 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: starting mon.a rank 0 at public addrs [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] at bind addrs [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:12:48.133 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: mon.a@-1(???) e1 preinit fsid 2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:12:48.133 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: mon.a@-1(???).mds e1 new map 2024-09-13T14:12:48.133 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: mon.a@-1(???).mds e1 print_map 2024-09-13T14:12:48.133 INFO:journalctl@ceph.mon.a.smithi151.stdout: e1 2024-09-13T14:12:48.133 INFO:journalctl@ceph.mon.a.smithi151.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-13T14:12:48.133 INFO:journalctl@ceph.mon.a.smithi151.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T14:12:48.133 INFO:journalctl@ceph.mon.a.smithi151.stdout: legacy client fscid: -1 2024-09-13T14:12:48.133 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-09-13T14:12:48.133 INFO:journalctl@ceph.mon.a.smithi151.stdout: No filesystems configured 2024-09-13T14:12:48.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-13T14:12:48.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T14:12:48.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T14:12:48.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T14:12:48.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-13T14:12:48.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-13T14:12:48.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: monmap e1: 1 mons at {a=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T14:12:48.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: fsmap 2024-09-13T14:12:48.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: osdmap e1: 0 total, 0 up, 0 in 2024-09-13T14:12:48.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:47 smithi151 ceph-mon[29807]: mgrmap e1: no daemons active 2024-09-13T14:12:48.505 INFO:teuthology.orchestra.run.smithi151.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-13T14:12:48.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:48 smithi151 bash[29769]: ffc441eb291c47c362a7837504a281e149b24a73556712e8015912fa0696371d 2024-09-13T14:12:48.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:48 smithi151 systemd[1]: Started Ceph mon.a for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:12:51.196 INFO:teuthology.orchestra.run.smithi151.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-13T14:12:51.196 INFO:teuthology.orchestra.run.smithi151.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-13T14:12:51.197 INFO:teuthology.orchestra.run.smithi151.stdout:Creating mgr... 2024-09-13T14:12:51.197 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-13T14:12:51.197 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-13T14:12:51.373 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-2401fc50-71da-11ef-bceb-c7b262605968@mgr.y 2024-09-13T14:12:51.373 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Failed to reset failed state of unit ceph-2401fc50-71da-11ef-bceb-c7b262605968@mgr.y.service: Unit ceph-2401fc50-71da-11ef-bceb-c7b262605968@mgr.y.service not loaded. 2024-09-13T14:12:51.521 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-2401fc50-71da-11ef-bceb-c7b262605968.target.wants/ceph-2401fc50-71da-11ef-bceb-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-2401fc50-71da-11ef-bceb-c7b262605968@.service. 2024-09-13T14:12:51.838 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:51 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/4153343966' entity='client.admin' 2024-09-13T14:12:51.839 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:12:51 smithi151 podman[30041]: 2024-09-13 14:12:51.653433866 +0000 UTC m=+0.015589107 image pull a994b00deb3a4a8eeb484a456d3f3207aae0401a31d251b199b2c9dbc1c4036c quay.ceph.io/ceph-ci/ceph:quincy 2024-09-13T14:12:52.060 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-13T14:12:52.060 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout disabled 2024-09-13T14:12:52.067 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-13T14:12:52.067 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout inactive 2024-09-13T14:12:52.068 INFO:teuthology.orchestra.run.smithi151.stdout:firewalld.service is not enabled 2024-09-13T14:12:52.068 INFO:teuthology.orchestra.run.smithi151.stdout:Not possible to enable service . firewalld.service is not available 2024-09-13T14:12:52.076 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-13T14:12:52.076 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout disabled 2024-09-13T14:12:52.084 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-13T14:12:52.084 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout inactive 2024-09-13T14:12:52.085 INFO:teuthology.orchestra.run.smithi151.stdout:firewalld.service is not enabled 2024-09-13T14:12:52.085 INFO:teuthology.orchestra.run.smithi151.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-13T14:12:52.085 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for mgr to start... 2024-09-13T14:12:52.085 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for mgr... 2024-09-13T14:12:52.188 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:12:51 smithi151 podman[30041]: 2024-09-13 14:12:51.937745754 +0000 UTC m=+0.299900986 container init 6ce1f75b6d5e75d4d8d6679d4c4e6fb807adff35292497c51dfb1634e8ed460e (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y, org.label-schema.license=GPLv2, RELEASE=quincy-bbdadfa, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-13T14:12:52.188 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:12:51 smithi151 podman[30041]: 2024-09-13 14:12:51.942658557 +0000 UTC m=+0.304813788 container start 6ce1f75b6d5e75d4d8d6679d4c4e6fb807adff35292497c51dfb1634e8ed460e (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=quincy-bbdadfa, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-13T14:12:52.189 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:12:52 smithi151 bash[30041]: 6ce1f75b6d5e75d4d8d6679d4c4e6fb807adff35292497c51dfb1634e8ed460e 2024-09-13T14:12:52.189 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:12:52 smithi151 systemd[1]: Started Ceph mgr.y for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:12:52.189 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:12:52 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:12:52.056+0000 7f960bd91ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T14:12:52.189 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:12:52 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:12:52.122+0000 7f960bd91ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T14:12:52.189 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:12:52 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:12:52.187+0000 7f960bd91ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T14:12:52.469 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:12:52 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:12:52.312+0000 7f960bd91ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:12:52.470 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:12:52 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:12:52.468+0000 7f960bd91ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T14:12:52.744 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:52 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/4028836852' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-13T14:12:52.995 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:12:52 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:12:52.742+0000 7f960bd91ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T14:12:52.995 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:12:52 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:12:52.804+0000 7f960bd91ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T14:12:52.995 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:12:52 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:12:52.968+0000 7f960bd91ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T14:12:53.256 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-09-13T14:12:53.262 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-09-13T14:12:53.262 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsid": "2401fc50-71da-11ef-bceb-c7b262605968", 2024-09-13T14:12:53.262 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "health": { 2024-09-13T14:12:53.262 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-13T14:12:53.262 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-13T14:12:53.262 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-13T14:12:53.263 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:12:53.263 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-13T14:12:53.263 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-13T14:12:53.263 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 0 2024-09-13T14:12:53.263 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-09-13T14:12:53.263 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-13T14:12:53.263 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "a" 2024-09-13T14:12:53.263 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-09-13T14:12:53.263 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_age": 4, 2024-09-13T14:12:53.263 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-13T14:12:53.263 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:12:53.263 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-13T14:12:53.263 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-13T14:12:53.264 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:12:53.264 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-13T14:12:53.264 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:12:53.264 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-13T14:12:53.264 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-13T14:12:53.264 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-13T14:12:53.264 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-13T14:12:53.264 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-13T14:12:53.264 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-13T14:12:53.264 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:12:53.264 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-13T14:12:53.264 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-13T14:12:53.264 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-13T14:12:53.264 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-13T14:12:53.265 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-13T14:12:53.265 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-13T14:12:53.265 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-13T14:12:53.265 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-13T14:12:53.265 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-13T14:12:53.265 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:12:53.265 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-13T14:12:53.265 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:12:53.265 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-13T14:12:53.265 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-13T14:12:53.265 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:12:53.265 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-13T14:12:53.265 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-13T14:12:53.266 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-13T14:12:53.266 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-13T14:12:53.266 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-13T14:12:53.266 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-13T14:12:53.266 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "restful" 2024-09-13T14:12:53.266 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-09-13T14:12:53.266 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T14:12:53.266 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:12:53.267 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-13T14:12:53.268 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:12:53.268 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modified": "2024-09-13T14:12:36.168625+0000", 2024-09-13T14:12:53.268 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T14:12:53.268 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:12:53.268 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-13T14:12:53.268 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-09-13T14:12:53.268 INFO:teuthology.orchestra.run.smithi151.stdout:mgr not available, waiting (1/15)... 2024-09-13T14:12:54.108 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:12:53 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:12:53.818+0000 7f960bd91ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T14:12:54.108 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:12:53 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:12:53.882+0000 7f960bd91ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T14:12:54.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:12:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:12:54.592+0000 7f960bd91ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T14:12:54.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:12:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:12:54.700+0000 7f960bd91ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T14:12:54.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:12:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:12:54.818+0000 7f960bd91ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:12:55.259 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:12:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:12:54.877+0000 7f960bd91ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T14:12:55.259 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:12:55 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:12:55.001+0000 7f960bd91ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T14:12:55.259 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:12:55 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:12:55.068+0000 7f960bd91ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T14:12:55.260 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:12:55 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:12:55.126+0000 7f960bd91ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T14:12:55.260 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:12:55 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:12:55.258+0000 7f960bd91ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T14:12:55.531 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:12:55 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:12:55.326+0000 7f960bd91ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T14:12:55.531 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:12:55 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:12:55.391+0000 7f960bd91ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T14:12:55.531 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:12:55 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:12:55.471+0000 7f960bd91ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T14:12:55.531 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:12:55 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:12:55.530+0000 7f960bd91ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T14:12:55.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:12:55 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:12:55.587+0000 7f960bd91ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T14:12:56.608 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:12:56 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:12:56.336+0000 7f960bd91ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T14:12:56.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:56 smithi151 ceph-mon[29807]: Activating manager daemon y 2024-09-13T14:12:56.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:56 smithi151 ceph-mon[29807]: mgrmap e2: y(active, starting, since 0.00286137s) 2024-09-13T14:12:56.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:56 smithi151 ceph-mon[29807]: from='mgr.14100 172.21.15.151:0/3963156127' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:12:56.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:56 smithi151 ceph-mon[29807]: from='mgr.14100 172.21.15.151:0/3963156127' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:12:56.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:56 smithi151 ceph-mon[29807]: from='mgr.14100 172.21.15.151:0/3963156127' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:12:56.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:56 smithi151 ceph-mon[29807]: from='mgr.14100 172.21.15.151:0/3963156127' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:12:56.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:56 smithi151 ceph-mon[29807]: from='mgr.14100 172.21.15.151:0/3963156127' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:12:56.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:56 smithi151 ceph-mon[29807]: Manager daemon y is now available 2024-09-13T14:12:56.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:56 smithi151 ceph-mon[29807]: from='mgr.14100 172.21.15.151:0/3963156127' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:12:56.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:56 smithi151 ceph-mon[29807]: from='mgr.14100 172.21.15.151:0/3963156127' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:12:56.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:56 smithi151 ceph-mon[29807]: from='mgr.14100 172.21.15.151:0/3963156127' entity='mgr.y' 2024-09-13T14:12:56.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:56 smithi151 ceph-mon[29807]: from='mgr.14100 172.21.15.151:0/3963156127' entity='mgr.y' 2024-09-13T14:12:56.611 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:56 smithi151 ceph-mon[29807]: from='mgr.14100 172.21.15.151:0/3963156127' entity='mgr.y' 2024-09-13T14:12:58.507 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-09-13T14:12:58.507 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-09-13T14:12:58.507 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsid": "2401fc50-71da-11ef-bceb-c7b262605968", 2024-09-13T14:12:58.507 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "health": { 2024-09-13T14:12:58.507 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-13T14:12:58.507 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-13T14:12:58.508 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-13T14:12:58.508 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:12:58.508 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-13T14:12:58.508 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-13T14:12:58.508 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 0 2024-09-13T14:12:58.508 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-09-13T14:12:58.508 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-13T14:12:58.508 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "a" 2024-09-13T14:12:58.508 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-09-13T14:12:58.508 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-09-13T14:12:58.508 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-13T14:12:58.508 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:12:58.508 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-13T14:12:58.508 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-13T14:12:58.509 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:12:58.509 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-13T14:12:58.509 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:12:58.509 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-13T14:12:58.509 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-13T14:12:58.509 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-13T14:12:58.509 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-13T14:12:58.509 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-13T14:12:58.509 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-13T14:12:58.512 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:12:58.512 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-13T14:12:58.513 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-13T14:12:58.513 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-13T14:12:58.513 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-13T14:12:58.513 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-13T14:12:58.513 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-13T14:12:58.513 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-13T14:12:58.513 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-13T14:12:58.513 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-13T14:12:58.513 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:12:58.513 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-13T14:12:58.513 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:12:58.513 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-13T14:12:58.514 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-13T14:12:58.514 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:12:58.514 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-13T14:12:58.514 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-13T14:12:58.514 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-13T14:12:58.514 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-13T14:12:58.514 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-13T14:12:58.514 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-13T14:12:58.514 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "restful" 2024-09-13T14:12:58.514 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-09-13T14:12:58.515 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T14:12:58.515 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:12:58.515 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-13T14:12:58.515 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:12:58.515 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modified": "2024-09-13T14:12:36.168625+0000", 2024-09-13T14:12:58.515 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T14:12:58.515 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:12:58.515 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-13T14:12:58.515 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-09-13T14:12:58.515 INFO:teuthology.orchestra.run.smithi151.stdout:mgr is available 2024-09-13T14:12:58.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:58 smithi151 ceph-mon[29807]: mgrmap e3: y(active, since 1.00717s) 2024-09-13T14:12:58.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:58 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/2837302228' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-13T14:12:59.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:59 smithi151 ceph-mon[29807]: mgrmap e4: y(active, since 2s) 2024-09-13T14:12:59.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:12:59 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1429941714' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-13T14:12:59.696 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-09-13T14:12:59.696 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout [global] 2024-09-13T14:12:59.696 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout fsid = 2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:12:59.696 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-13T14:12:59.696 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-13T14:12:59.697 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-13T14:12:59.697 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-13T14:12:59.697 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-13T14:12:59.697 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-09-13T14:12:59.697 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-13T14:12:59.697 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-13T14:12:59.697 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-09-13T14:12:59.697 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout [osd] 2024-09-13T14:12:59.697 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-13T14:12:59.697 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-13T14:12:59.697 INFO:teuthology.orchestra.run.smithi151.stdout:Enabling cephadm module... 2024-09-13T14:13:00.767 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:00 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/412321290' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-13T14:13:00.767 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:00 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: ignoring --setuser ceph since I am not root 2024-09-13T14:13:00.767 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:00 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: ignoring --setgroup ceph since I am not root 2024-09-13T14:13:00.767 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:00 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:00.530+0000 7f859b956ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T14:13:00.767 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:00 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:00.584+0000 7f859b956ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T14:13:00.767 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:00 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:00.648+0000 7f859b956ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T14:13:01.058 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:00 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:00.774+0000 7f859b956ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:13:01.058 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:00 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:00.910+0000 7f859b956ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T14:13:01.314 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:01 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:01.168+0000 7f859b956ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T14:13:01.314 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:01 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:01.229+0000 7f859b956ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T14:13:01.608 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:01 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:01.396+0000 7f859b956ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T14:13:01.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:01 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/412321290' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-13T14:13:01.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:01 smithi151 ceph-mon[29807]: mgrmap e5: y(active, since 4s) 2024-09-13T14:13:02.298 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:02.249+0000 7f859b956ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T14:13:02.300 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-09-13T14:13:02.300 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-13T14:13:02.301 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-13T14:13:02.301 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-13T14:13:02.301 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-13T14:13:02.301 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-09-13T14:13:02.301 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for the mgr to restart... 2024-09-13T14:13:02.301 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for mgr epoch 5... 2024-09-13T14:13:02.576 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:02 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1855408800' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-13T14:13:02.576 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:02.315+0000 7f859b956ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T14:13:03.332 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:03 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:03.044+0000 7f859b956ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T14:13:03.332 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:03 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:03.154+0000 7f859b956ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T14:13:03.332 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:03 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:03.273+0000 7f859b956ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:13:03.332 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:03 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:03.331+0000 7f859b956ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T14:13:03.608 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:03 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:03.455+0000 7f859b956ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T14:13:03.608 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:03 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:03.520+0000 7f859b956ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T14:13:03.608 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:03 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:03.577+0000 7f859b956ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T14:13:03.971 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:03 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:03.709+0000 7f859b956ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T14:13:03.971 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:03 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:03.771+0000 7f859b956ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T14:13:03.972 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:03 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:03.835+0000 7f859b956ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T14:13:03.972 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:03 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:03.914+0000 7f859b956ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T14:13:03.972 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:03 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:03.970+0000 7f859b956ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T14:13:04.358 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:04.027+0000 7f859b956ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T14:13:05.108 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:04.759+0000 7f859b956ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T14:13:05.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:04 smithi151 ceph-mon[29807]: Active manager daemon y restarted 2024-09-13T14:13:05.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:04 smithi151 ceph-mon[29807]: Activating manager daemon y 2024-09-13T14:13:05.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:04 smithi151 ceph-mon[29807]: osdmap e2: 0 total, 0 up, 0 in 2024-09-13T14:13:05.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:04 smithi151 ceph-mon[29807]: mgrmap e6: y(active, starting, since 0.00317814s) 2024-09-13T14:13:05.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:04 smithi151 ceph-mon[29807]: from='mgr.14116 172.21.15.151:0/3146213926' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:13:05.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:04 smithi151 ceph-mon[29807]: from='mgr.14116 172.21.15.151:0/3146213926' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:13:05.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:04 smithi151 ceph-mon[29807]: from='mgr.14116 172.21.15.151:0/3146213926' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:13:05.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:04 smithi151 ceph-mon[29807]: from='mgr.14116 172.21.15.151:0/3146213926' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:13:05.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:04 smithi151 ceph-mon[29807]: from='mgr.14116 172.21.15.151:0/3146213926' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:13:05.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:04 smithi151 ceph-mon[29807]: Manager daemon y is now available 2024-09-13T14:13:05.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:04 smithi151 ceph-mon[29807]: from='mgr.14116 172.21.15.151:0/3146213926' entity='mgr.y' 2024-09-13T14:13:05.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:04 smithi151 ceph-mon[29807]: from='mgr.14116 172.21.15.151:0/3146213926' entity='mgr.y' 2024-09-13T14:13:05.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:04 smithi151 ceph-mon[29807]: from='mgr.14116 172.21.15.151:0/3146213926' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:13:05.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:04 smithi151 ceph-mon[29807]: from='mgr.14116 172.21.15.151:0/3146213926' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:13:05.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:04 smithi151 ceph-mon[29807]: from='mgr.14116 172.21.15.151:0/3146213926' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:13:05.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:04 smithi151 ceph-mon[29807]: from='mgr.14116 172.21.15.151:0/3146213926' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:13:06.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:05 smithi151 ceph-mon[29807]: Found migration_current of "None". Setting to last migration. 2024-09-13T14:13:06.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:05 smithi151 ceph-mon[29807]: mgrmap e7: y(active, since 1.00578s) 2024-09-13T14:13:06.382 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-09-13T14:13:06.383 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-13T14:13:06.383 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-13T14:13:06.383 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-09-13T14:13:06.383 INFO:teuthology.orchestra.run.smithi151.stdout:mgr epoch 5 is available 2024-09-13T14:13:06.383 INFO:teuthology.orchestra.run.smithi151.stdout:Setting orchestrator backend to cephadm... 2024-09-13T14:13:06.716 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:06.391+0000 7f854dd45640 -1 log_channel(cephadm) log [ERR] : [13/Sep/2024:14:13:06] ENGINE Error in HTTPServer.serve 2024-09-13T14:13:06.716 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: Traceback (most recent call last): 2024-09-13T14:13:06.716 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:13:06.716 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: self._connections.run(self.expiration_interval) 2024-09-13T14:13:06.716 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:13:06.716 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: self._run(expiration_interval) 2024-09-13T14:13:06.716 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:13:06.717 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:13:06.717 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:13:06.717 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:13:06.717 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:13:06.717 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: s = self.context.wrap_socket( 2024-09-13T14:13:06.717 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:13:06.717 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: return self.sslsocket_class._create( 2024-09-13T14:13:06.717 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:13:06.717 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: self.do_handshake() 2024-09-13T14:13:06.717 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:13:06.717 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: self._sslobj.do_handshake() 2024-09-13T14:13:06.717 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:13:06.717 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:07.544 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:07 smithi151 ceph-mon[29807]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-13T14:13:07.544 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:07 smithi151 ceph-mon[29807]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-13T14:13:07.544 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:07 smithi151 ceph-mon[29807]: [13/Sep/2024:14:13:06] ENGINE Bus STARTING 2024-09-13T14:13:07.544 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:07 smithi151 ceph-mon[29807]: [13/Sep/2024:14:13:06] ENGINE Serving on https://172.21.15.151:7150 2024-09-13T14:13:07.545 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:07 smithi151 ceph-mon[29807]: [13/Sep/2024:14:13:06] ENGINE Bus STARTED 2024-09-13T14:13:07.545 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:07 smithi151 ceph-mon[29807]: [13/Sep/2024:14:13:06] ENGINE Error in HTTPServer.serve 2024-09-13T14:13:07.545 INFO:journalctl@ceph.mon.a.smithi151.stdout: Traceback (most recent call last): 2024-09-13T14:13:07.545 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:13:07.545 INFO:journalctl@ceph.mon.a.smithi151.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:13:07.545 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:13:07.545 INFO:journalctl@ceph.mon.a.smithi151.stdout: self._run(expiration_interval) 2024-09-13T14:13:07.545 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:13:07.545 INFO:journalctl@ceph.mon.a.smithi151.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:13:07.545 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:13:07.545 INFO:journalctl@ceph.mon.a.smithi151.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:13:07.545 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:13:07.545 INFO:journalctl@ceph.mon.a.smithi151.stdout: s = self.context.wrap_socket( 2024-09-13T14:13:07.545 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:13:07.546 INFO:journalctl@ceph.mon.a.smithi151.stdout: return self.sslsocket_class._create( 2024-09-13T14:13:07.546 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:13:07.546 INFO:journalctl@ceph.mon.a.smithi151.stdout: self.do_handshake() 2024-09-13T14:13:07.546 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:13:07.546 INFO:journalctl@ceph.mon.a.smithi151.stdout: self._sslobj.do_handshake() 2024-09-13T14:13:07.546 INFO:journalctl@ceph.mon.a.smithi151.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:13:07.546 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:07 smithi151 ceph-mon[29807]: from='mgr.14116 172.21.15.151:0/3146213926' entity='mgr.y' 2024-09-13T14:13:07.546 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:07 smithi151 ceph-mon[29807]: from='mgr.14116 172.21.15.151:0/3146213926' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:13:07.546 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:07 smithi151 ceph-mon[29807]: from='mgr.14116 172.21.15.151:0/3146213926' entity='mgr.y' 2024-09-13T14:13:07.546 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:07 smithi151 ceph-mon[29807]: from='mgr.14116 172.21.15.151:0/3146213926' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:13:08.484 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:08 smithi151 ceph-mon[29807]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:08.484 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:08 smithi151 ceph-mon[29807]: mgrmap e8: y(active, since 2s) 2024-09-13T14:13:08.753 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-13T14:13:08.753 INFO:teuthology.orchestra.run.smithi151.stdout:Generating ssh key... 2024-09-13T14:13:09.716 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:09 smithi151 ceph-mon[29807]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:10.032 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:09 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: Generating public/private rsa key pair. 2024-09-13T14:13:10.033 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:09 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: Your identification has been saved in /tmp/tmpypeecjfb/key 2024-09-13T14:13:10.033 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:09 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: Your public key has been saved in /tmp/tmpypeecjfb/key.pub 2024-09-13T14:13:10.033 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:09 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: The key fingerprint is: 2024-09-13T14:13:10.033 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:09 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: SHA256:heRnMLRBFx/9NrUwsKLJarJnn6kxNT6BUQyoE5Ex/xU ceph-2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:13:10.033 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:09 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: The key's randomart image is: 2024-09-13T14:13:10.033 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:09 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: +---[RSA 3072]----+ 2024-09-13T14:13:10.033 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:09 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: | ++ ..oEB ooo. | 2024-09-13T14:13:10.033 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:09 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: | oo. .+.B ..+. .| 2024-09-13T14:13:10.033 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:09 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: | o. . .+.+.. o.o| 2024-09-13T14:13:10.033 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:09 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: | o . = o+. oo| 2024-09-13T14:13:10.033 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:09 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: | . o BS ..| 2024-09-13T14:13:10.033 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:09 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: | + o | 2024-09-13T14:13:10.033 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:09 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: | . = o | 2024-09-13T14:13:10.033 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:09 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: | +oo + | 2024-09-13T14:13:10.034 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:09 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: | .o.o+ | 2024-09-13T14:13:10.034 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:09 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: +----[SHA256]-----+ 2024-09-13T14:13:10.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:10 smithi151 ceph-mon[29807]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:10.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:10 smithi151 ceph-mon[29807]: Generating ssh key... 2024-09-13T14:13:10.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:10 smithi151 ceph-mon[29807]: from='mgr.14116 172.21.15.151:0/3146213926' entity='mgr.y' 2024-09-13T14:13:10.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:10 smithi151 ceph-mon[29807]: from='mgr.14116 172.21.15.151:0/3146213926' entity='mgr.y' 2024-09-13T14:13:11.441 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0pEHXkpJRXn2QlgGc5+ghifHQbqtBxopZawwBzTgJP/S5soZN0zJZANtHIqc7VE8Dqd92v7erJrBX4uz3yNeLPmt0YI/M9EFbeVZnhnzuK6C+w3EZmrF7AjjPR/RKB8fuBm39tJvnCAvIqEyHf1Ezh/F/Z15f+65yJDPXGzAQl/7DpdhdL3KGf+mr8bMTk0jCVZlFpDj+1TfHaSpHULpSayznxyhQGHCJUp9YimYGkCGNujJMCT5eZHK+mey08wlf/WOgeIGafeLG1aO4Yb6lhE+LxwbjOynrMfsU8WIga12Zna8zBJVGLFm+3YskdxPAxhs6ZU4tEegFvtSMhP4rurPCkmy4/D6WB/XLe4IwWipmX+Qt4BsU9zqjdESVdatYdKY5KxVjx24mFbZtB5U4qh1BPY9jWiIwicOr1U9WJ8mu2hlSOIrL2WLyZwG+yLkPTyYrv2d6u/BpHiOKIsBX7SZfS8sMz8og/Reho0n6tBBx+cyQoO+O0nSO7leSV4U= ceph-2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:13:11.441 INFO:teuthology.orchestra.run.smithi151.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-13T14:13:11.441 INFO:teuthology.orchestra.run.smithi151.stdout:Adding key to root@localhost authorized_keys... 2024-09-13T14:13:11.451 INFO:teuthology.orchestra.run.smithi151.stdout:Adding host smithi151... 2024-09-13T14:13:11.816 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:11 smithi151 ceph-mon[29807]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:12.828 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:12 smithi151 ceph-mon[29807]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi151", "addr": "172.21.15.151", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:13.756 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:13 smithi151 ceph-mon[29807]: Deploying cephadm binary to smithi151 2024-09-13T14:13:15.299 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout Added host 'smithi151' with addr '172.21.15.151' 2024-09-13T14:13:15.299 INFO:teuthology.orchestra.run.smithi151.stdout:Deploying unmanaged mon service... 2024-09-13T14:13:15.919 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:15 smithi151 ceph-mon[29807]: from='mgr.14116 172.21.15.151:0/3146213926' entity='mgr.y' 2024-09-13T14:13:15.919 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:15 smithi151 ceph-mon[29807]: Added host smithi151 2024-09-13T14:13:15.919 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:15 smithi151 ceph-mon[29807]: from='mgr.14116 172.21.15.151:0/3146213926' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:13:16.575 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-13T14:13:16.575 INFO:teuthology.orchestra.run.smithi151.stdout:Deploying unmanaged mgr service... 2024-09-13T14:13:17.168 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:16 smithi151 ceph-mon[29807]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:17.168 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:16 smithi151 ceph-mon[29807]: Saving service mon spec with placement count:5 2024-09-13T14:13:17.168 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:16 smithi151 ceph-mon[29807]: from='mgr.14116 172.21.15.151:0/3146213926' entity='mgr.y' 2024-09-13T14:13:18.118 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-13T14:13:18.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:18 smithi151 ceph-mon[29807]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:18.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:18 smithi151 ceph-mon[29807]: Saving service mgr spec with placement count:2 2024-09-13T14:13:18.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:18 smithi151 ceph-mon[29807]: from='mgr.14116 172.21.15.151:0/3146213926' entity='mgr.y' 2024-09-13T14:13:19.984 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:19 smithi151 ceph-mon[29807]: from='mgr.14116 172.21.15.151:0/3146213926' entity='mgr.y' 2024-09-13T14:13:19.984 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:19 smithi151 ceph-mon[29807]: from='mgr.14116 172.21.15.151:0/3146213926' entity='mgr.y' 2024-09-13T14:13:19.984 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:19 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/2123020445' entity='client.admin' 2024-09-13T14:13:21.502 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:21 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1349875125' entity='client.admin' 2024-09-13T14:13:21.504 INFO:teuthology.orchestra.run.smithi151.stdout:Enabling the dashboard module... 2024-09-13T14:13:23.290 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:23 smithi151 ceph-mon[29807]: from='mgr.14116 172.21.15.151:0/3146213926' entity='mgr.y' 2024-09-13T14:13:23.290 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:23 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1419013280' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-13T14:13:23.290 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:23 smithi151 ceph-mon[29807]: from='mgr.14116 172.21.15.151:0/3146213926' entity='mgr.y' 2024-09-13T14:13:23.291 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:23 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: ignoring --setuser ceph since I am not root 2024-09-13T14:13:23.291 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:23 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: ignoring --setgroup ceph since I am not root 2024-09-13T14:13:23.291 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:23 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:23.160+0000 7f399fdc8ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T14:13:23.291 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:23 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:23.222+0000 7f399fdc8ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T14:13:23.291 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:23 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:23.288+0000 7f399fdc8ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T14:13:23.552 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:23 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:23.412+0000 7f399fdc8ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:13:23.553 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:23 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:23.551+0000 7f399fdc8ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T14:13:24.071 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:24 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1419013280' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-13T14:13:24.071 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:24 smithi151 ceph-mon[29807]: mgrmap e9: y(active, since 18s) 2024-09-13T14:13:24.071 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:23 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:23.807+0000 7f399fdc8ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T14:13:24.071 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:23 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:23.876+0000 7f399fdc8ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T14:13:24.071 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:24.040+0000 7f399fdc8ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T14:13:25.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:25 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/697480746' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-13T14:13:25.358 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:24.917+0000 7f399fdc8ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T14:13:25.358 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:24.987+0000 7f399fdc8ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T14:13:25.684 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-09-13T14:13:25.684 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-13T14:13:25.684 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-13T14:13:25.684 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-13T14:13:25.684 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-13T14:13:25.684 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-09-13T14:13:25.684 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for the mgr to restart... 2024-09-13T14:13:25.684 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for mgr epoch 9... 2024-09-13T14:13:25.948 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:25 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:25.714+0000 7f399fdc8ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T14:13:25.948 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:25 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:25.826+0000 7f399fdc8ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T14:13:26.216 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:25 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:25.950+0000 7f399fdc8ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:13:26.216 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:26 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:26.016+0000 7f399fdc8ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T14:13:26.217 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:26 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:26.144+0000 7f399fdc8ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T14:13:26.217 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:26 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:26.215+0000 7f399fdc8ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T14:13:26.471 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:26 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:26.273+0000 7f399fdc8ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T14:13:26.472 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:26 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:26.407+0000 7f399fdc8ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T14:13:26.472 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:26 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:26.470+0000 7f399fdc8ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T14:13:26.731 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:26 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:26.535+0000 7f399fdc8ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T14:13:26.731 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:26 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:26.615+0000 7f399fdc8ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T14:13:26.731 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:26 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:26.672+0000 7f399fdc8ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T14:13:26.731 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:26 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:26.730+0000 7f399fdc8ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T14:13:27.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:27 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:27.476+0000 7f399fdc8ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T14:13:27.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:27 smithi151 ceph-mon[29807]: Active manager daemon y restarted 2024-09-13T14:13:27.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:27 smithi151 ceph-mon[29807]: Activating manager daemon y 2024-09-13T14:13:27.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:27 smithi151 ceph-mon[29807]: osdmap e3: 0 total, 0 up, 0 in 2024-09-13T14:13:27.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:27 smithi151 ceph-mon[29807]: mgrmap e10: y(active, starting, since 0.00312081s) 2024-09-13T14:13:27.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:27 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:13:27.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:27 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:13:27.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:27 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:13:27.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:27 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:13:27.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:27 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:13:27.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:27 smithi151 ceph-mon[29807]: Manager daemon y is now available 2024-09-13T14:13:27.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:27 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:13:27.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:27 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:13:27.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:27 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:13:29.421 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-09-13T14:13:29.421 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-13T14:13:29.421 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-13T14:13:29.421 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-09-13T14:13:29.422 INFO:teuthology.orchestra.run.smithi151.stdout:mgr epoch 9 is available 2024-09-13T14:13:29.422 INFO:teuthology.orchestra.run.smithi151.stdout:Generating a dashboard self-signed certificate... 2024-09-13T14:13:29.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:29 smithi151 ceph-mon[29807]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-13T14:13:29.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:29 smithi151 ceph-mon[29807]: mgrmap e11: y(active, since 1.00763s) 2024-09-13T14:13:29.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:29 smithi151 ceph-mon[29807]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-13T14:13:29.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:29 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:13:29.695 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:29 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:29.510+0000 7f3950926640 -1 log_channel(cephadm) log [ERR] : [13/Sep/2024:14:13:29] ENGINE Error in HTTPServer.serve 2024-09-13T14:13:29.695 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:29 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: Traceback (most recent call last): 2024-09-13T14:13:29.695 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:29 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:13:29.695 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:29 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: self._connections.run(self.expiration_interval) 2024-09-13T14:13:29.695 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:29 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:13:29.695 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:29 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: self._run(expiration_interval) 2024-09-13T14:13:29.695 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:29 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:13:29.695 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:29 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:13:29.695 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:29 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:13:29.695 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:29 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:13:29.695 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:29 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:13:29.695 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:29 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: s = self.context.wrap_socket( 2024-09-13T14:13:29.695 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:29 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:13:29.696 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:29 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: return self.sslsocket_class._create( 2024-09-13T14:13:29.696 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:29 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:13:29.696 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:29 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: self.do_handshake() 2024-09-13T14:13:29.696 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:29 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:13:29.696 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:29 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: self._sslobj.do_handshake() 2024-09-13T14:13:29.696 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:29 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:13:29.696 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:13:29 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:13:30.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:30 smithi151 ceph-mon[29807]: [13/Sep/2024:14:13:29] ENGINE Bus STARTING 2024-09-13T14:13:30.801 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:30 smithi151 ceph-mon[29807]: [13/Sep/2024:14:13:29] ENGINE Serving on https://172.21.15.151:7150 2024-09-13T14:13:30.801 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:30 smithi151 ceph-mon[29807]: [13/Sep/2024:14:13:29] ENGINE Bus STARTED 2024-09-13T14:13:30.801 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:30 smithi151 ceph-mon[29807]: [13/Sep/2024:14:13:29] ENGINE Error in HTTPServer.serve 2024-09-13T14:13:30.801 INFO:journalctl@ceph.mon.a.smithi151.stdout: Traceback (most recent call last): 2024-09-13T14:13:30.801 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:13:30.801 INFO:journalctl@ceph.mon.a.smithi151.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:13:30.801 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:13:30.801 INFO:journalctl@ceph.mon.a.smithi151.stdout: self._run(expiration_interval) 2024-09-13T14:13:30.801 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:13:30.801 INFO:journalctl@ceph.mon.a.smithi151.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:13:30.801 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:13:30.802 INFO:journalctl@ceph.mon.a.smithi151.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:13:30.802 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:13:30.802 INFO:journalctl@ceph.mon.a.smithi151.stdout: s = self.context.wrap_socket( 2024-09-13T14:13:30.802 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:13:30.802 INFO:journalctl@ceph.mon.a.smithi151.stdout: return self.sslsocket_class._create( 2024-09-13T14:13:30.802 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:13:30.802 INFO:journalctl@ceph.mon.a.smithi151.stdout: self.do_handshake() 2024-09-13T14:13:30.802 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:13:30.802 INFO:journalctl@ceph.mon.a.smithi151.stdout: self._sslobj.do_handshake() 2024-09-13T14:13:30.802 INFO:journalctl@ceph.mon.a.smithi151.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:13:30.802 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:30 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:13:30.802 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:30 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:13:30.802 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:30 smithi151 ceph-mon[29807]: mgrmap e12: y(active, since 2s) 2024-09-13T14:13:30.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:30 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:13:30.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:30 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:13:30.803 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-13T14:13:30.803 INFO:teuthology.orchestra.run.smithi151.stdout:Creating initial admin user... 2024-09-13T14:13:31.607 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:31 smithi151 ceph-mon[29807]: from='client.14160 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:33.114 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$iN4OQjvmmtyy/MeCf2UWEeEGN.PctraO06kBN94hYCgkp3.mUa6La", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726236812, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-13T14:13:33.114 INFO:teuthology.orchestra.run.smithi151.stdout:Fetching dashboard port number... 2024-09-13T14:13:33.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:33 smithi151 ceph-mon[29807]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:33.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:33 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:13:34.338 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 8443 2024-09-13T14:13:34.349 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-13T14:13:34.349 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout disabled 2024-09-13T14:13:34.357 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-13T14:13:34.357 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout inactive 2024-09-13T14:13:34.357 INFO:teuthology.orchestra.run.smithi151.stdout:firewalld.service is not enabled 2024-09-13T14:13:34.357 INFO:teuthology.orchestra.run.smithi151.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-13T14:13:34.360 INFO:teuthology.orchestra.run.smithi151.stdout:Ceph Dashboard is now available at: 2024-09-13T14:13:34.360 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:13:34.360 INFO:teuthology.orchestra.run.smithi151.stdout: URL: https://smithi151.front.sepia.ceph.com:8443/ 2024-09-13T14:13:34.360 INFO:teuthology.orchestra.run.smithi151.stdout: User: admin 2024-09-13T14:13:34.360 INFO:teuthology.orchestra.run.smithi151.stdout: Password: ijzhfywknc 2024-09-13T14:13:34.360 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:13:34.360 INFO:teuthology.orchestra.run.smithi151.stdout:Saving cluster configuration to /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config directory 2024-09-13T14:13:34.361 INFO:teuthology.orchestra.run.smithi151.stdout:Enabling autotune for osd_memory_target 2024-09-13T14:13:34.737 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:34 smithi151 ceph-mon[29807]: mgrmap e13: y(active, since 6s) 2024-09-13T14:13:34.738 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:34 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/3112658932' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-13T14:13:36.826 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-13T14:13:36.826 INFO:teuthology.orchestra.run.smithi151.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-13T14:13:36.826 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:13:36.826 INFO:teuthology.orchestra.run.smithi151.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-13T14:13:36.826 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:13:36.827 INFO:teuthology.orchestra.run.smithi151.stdout:Or, if you are only running a single cluster on this host: 2024-09-13T14:13:36.827 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:13:36.827 INFO:teuthology.orchestra.run.smithi151.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-13T14:13:36.827 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:13:36.827 INFO:teuthology.orchestra.run.smithi151.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-13T14:13:36.827 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:13:36.827 INFO:teuthology.orchestra.run.smithi151.stdout: ceph telemetry on 2024-09-13T14:13:36.827 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:13:36.827 INFO:teuthology.orchestra.run.smithi151.stdout:For more information see: 2024-09-13T14:13:36.827 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:13:36.827 INFO:teuthology.orchestra.run.smithi151.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-09-13T14:13:36.827 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:13:36.828 INFO:teuthology.orchestra.run.smithi151.stdout:Bootstrap complete. 2024-09-13T14:13:36.863 INFO:tasks.cephadm:Fetching config... 2024-09-13T14:13:36.864 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-13T14:13:36.864 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-13T14:13:36.880 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-13T14:13:36.880 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-13T14:13:36.880 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-13T14:13:36.935 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-13T14:13:36.936 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-13T14:13:36.936 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-13T14:13:37.000 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-13T14:13:37.001 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-13T14:13:37.001 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-13T14:13:37.056 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-13T14:13:37.057 DEBUG:teuthology.orchestra.run.smithi151:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0pEHXkpJRXn2QlgGc5+ghifHQbqtBxopZawwBzTgJP/S5soZN0zJZANtHIqc7VE8Dqd92v7erJrBX4uz3yNeLPmt0YI/M9EFbeVZnhnzuK6C+w3EZmrF7AjjPR/RKB8fuBm39tJvnCAvIqEyHf1Ezh/F/Z15f+65yJDPXGzAQl/7DpdhdL3KGf+mr8bMTk0jCVZlFpDj+1TfHaSpHULpSayznxyhQGHCJUp9YimYGkCGNujJMCT5eZHK+mey08wlf/WOgeIGafeLG1aO4Yb6lhE+LxwbjOynrMfsU8WIga12Zna8zBJVGLFm+3YskdxPAxhs6ZU4tEegFvtSMhP4rurPCkmy4/D6WB/XLe4IwWipmX+Qt4BsU9zqjdESVdatYdKY5KxVjx24mFbZtB5U4qh1BPY9jWiIwicOr1U9WJ8mu2hlSOIrL2WLyZwG+yLkPTyYrv2d6u/BpHiOKIsBX7SZfS8sMz8og/Reho0n6tBBx+cyQoO+O0nSO7leSV4U= ceph-2401fc50-71da-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-13T14:13:37.130 INFO:teuthology.orchestra.run.smithi151.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0pEHXkpJRXn2QlgGc5+ghifHQbqtBxopZawwBzTgJP/S5soZN0zJZANtHIqc7VE8Dqd92v7erJrBX4uz3yNeLPmt0YI/M9EFbeVZnhnzuK6C+w3EZmrF7AjjPR/RKB8fuBm39tJvnCAvIqEyHf1Ezh/F/Z15f+65yJDPXGzAQl/7DpdhdL3KGf+mr8bMTk0jCVZlFpDj+1TfHaSpHULpSayznxyhQGHCJUp9YimYGkCGNujJMCT5eZHK+mey08wlf/WOgeIGafeLG1aO4Yb6lhE+LxwbjOynrMfsU8WIga12Zna8zBJVGLFm+3YskdxPAxhs6ZU4tEegFvtSMhP4rurPCkmy4/D6WB/XLe4IwWipmX+Qt4BsU9zqjdESVdatYdKY5KxVjx24mFbZtB5U4qh1BPY9jWiIwicOr1U9WJ8mu2hlSOIrL2WLyZwG+yLkPTyYrv2d6u/BpHiOKIsBX7SZfS8sMz8og/Reho0n6tBBx+cyQoO+O0nSO7leSV4U= ceph-2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:13:37.157 DEBUG:teuthology.orchestra.run.smithi184:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0pEHXkpJRXn2QlgGc5+ghifHQbqtBxopZawwBzTgJP/S5soZN0zJZANtHIqc7VE8Dqd92v7erJrBX4uz3yNeLPmt0YI/M9EFbeVZnhnzuK6C+w3EZmrF7AjjPR/RKB8fuBm39tJvnCAvIqEyHf1Ezh/F/Z15f+65yJDPXGzAQl/7DpdhdL3KGf+mr8bMTk0jCVZlFpDj+1TfHaSpHULpSayznxyhQGHCJUp9YimYGkCGNujJMCT5eZHK+mey08wlf/WOgeIGafeLG1aO4Yb6lhE+LxwbjOynrMfsU8WIga12Zna8zBJVGLFm+3YskdxPAxhs6ZU4tEegFvtSMhP4rurPCkmy4/D6WB/XLe4IwWipmX+Qt4BsU9zqjdESVdatYdKY5KxVjx24mFbZtB5U4qh1BPY9jWiIwicOr1U9WJ8mu2hlSOIrL2WLyZwG+yLkPTyYrv2d6u/BpHiOKIsBX7SZfS8sMz8og/Reho0n6tBBx+cyQoO+O0nSO7leSV4U= ceph-2401fc50-71da-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-13T14:13:37.233 INFO:teuthology.orchestra.run.smithi184.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0pEHXkpJRXn2QlgGc5+ghifHQbqtBxopZawwBzTgJP/S5soZN0zJZANtHIqc7VE8Dqd92v7erJrBX4uz3yNeLPmt0YI/M9EFbeVZnhnzuK6C+w3EZmrF7AjjPR/RKB8fuBm39tJvnCAvIqEyHf1Ezh/F/Z15f+65yJDPXGzAQl/7DpdhdL3KGf+mr8bMTk0jCVZlFpDj+1TfHaSpHULpSayznxyhQGHCJUp9YimYGkCGNujJMCT5eZHK+mey08wlf/WOgeIGafeLG1aO4Yb6lhE+LxwbjOynrMfsU8WIga12Zna8zBJVGLFm+3YskdxPAxhs6ZU4tEegFvtSMhP4rurPCkmy4/D6WB/XLe4IwWipmX+Qt4BsU9zqjdESVdatYdKY5KxVjx24mFbZtB5U4qh1BPY9jWiIwicOr1U9WJ8mu2hlSOIrL2WLyZwG+yLkPTyYrv2d6u/BpHiOKIsBX7SZfS8sMz8og/Reho0n6tBBx+cyQoO+O0nSO7leSV4U= ceph-2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:13:37.246 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-13T14:13:37.491 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:13:37.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:37 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1585886888' entity='client.admin' 2024-09-13T14:13:39.914 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:39 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:13:39.914 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:39 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:13:39.914 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:39 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi151", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:13:39.914 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:39 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:13:39.914 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:39 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:13:39.914 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:39 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:13:41.329 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:41 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/2147886835' entity='client.admin' 2024-09-13T14:13:41.329 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:41 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:13:41.329 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:41 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:13:41.359 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-13T14:13:41.359 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-13T14:13:41.603 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:13:43.953 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi184 2024-09-13T14:13:43.954 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-13T14:13:43.954 DEBUG:teuthology.orchestra.run.smithi184:> dd of=/etc/ceph/ceph.conf 2024-09-13T14:13:43.974 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-13T14:13:43.974 DEBUG:teuthology.orchestra.run.smithi184:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:13:44.032 INFO:tasks.cephadm:Adding host smithi184 to orchestrator... 2024-09-13T14:13:44.032 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph orch host add smithi184 2024-09-13T14:13:44.280 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:13:44.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:44 smithi151 ceph-mon[29807]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:44.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:44 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:13:44.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:44 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:13:44.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:44 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:13:44.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:44 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:13:44.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:44 smithi151 ceph-mon[29807]: Updating smithi151:/etc/ceph/ceph.conf 2024-09-13T14:13:44.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:44 smithi151 ceph-mon[29807]: Updating smithi151:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:13:44.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:44 smithi151 ceph-mon[29807]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:13:44.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:44 smithi151 ceph-mon[29807]: Updating smithi151:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:13:44.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:44 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:13:44.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:44 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:13:44.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:44 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:13:47.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:47 smithi151 ceph-mon[29807]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi184", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:48.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:48 smithi151 ceph-mon[29807]: Deploying cephadm binary to smithi184 2024-09-13T14:13:48.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:48 smithi151 ceph-mon[29807]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:13:48.898 INFO:teuthology.orchestra.run.smithi151.stdout:Added host 'smithi184' with addr '172.21.15.184' 2024-09-13T14:13:49.498 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-13T14:13:49.738 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:13:50.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:49 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:13:50.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:49 smithi151 ceph-mon[29807]: Added host smithi184 2024-09-13T14:13:50.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:49 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:13:50.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:49 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:13:50.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:49 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:13:51.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:50 smithi151 ceph-mon[29807]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:13:51.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:50 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:13:51.471 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:13:51.471 INFO:teuthology.orchestra.run.smithi151.stdout:[{"addr": "172.21.15.151", "hostname": "smithi151", "labels": [], "status": ""}, {"addr": "172.21.15.184", "hostname": "smithi184", "labels": [], "status": ""}] 2024-09-13T14:13:52.052 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-13T14:13:52.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd crush tunables default 2024-09-13T14:13:52.288 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:13:53.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:52 smithi151 ceph-mon[29807]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:13:53.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:52 smithi151 ceph-mon[29807]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:13:54.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:54 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:13:54.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:54 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:13:54.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:54 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:13:54.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:54 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:13:54.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:54 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:13:54.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:54 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:13:54.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:54 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:13:54.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:54 smithi151 ceph-mon[29807]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-13T14:13:54.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:54 smithi151 ceph-mon[29807]: Updating smithi184:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:13:54.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:54 smithi151 ceph-mon[29807]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:13:54.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:54 smithi151 ceph-mon[29807]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:13:54.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:54 smithi151 ceph-mon[29807]: Updating smithi184:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:13:54.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:54 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:13:54.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:54 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:13:54.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:54 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:13:54.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:54 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1147121664' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-13T14:13:55.045 INFO:teuthology.orchestra.run.smithi151.stderr:adjusted tunables profile to default 2024-09-13T14:13:55.689 INFO:tasks.cephadm:Adding mon.a on smithi151 2024-09-13T14:13:55.689 INFO:tasks.cephadm:Adding mon.c on smithi151 2024-09-13T14:13:55.689 INFO:tasks.cephadm:Adding mon.b on smithi184 2024-09-13T14:13:55.689 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph orch apply mon '3;smithi151:172.21.15.151=a;smithi151:[v2:172.21.15.151:3301,v1:172.21.15.151:6790]=c;smithi184:172.21.15.184=b' 2024-09-13T14:13:55.931 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:13:56.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:56 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1147121664' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-13T14:13:56.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:56 smithi151 ceph-mon[29807]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T14:13:57.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:57 smithi151 ceph-mon[29807]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:13:57.676 INFO:teuthology.orchestra.run.smithi184.stdout:Scheduled mon update... 2024-09-13T14:13:58.232 DEBUG:teuthology.orchestra.run.smithi151:mon.c> sudo journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@mon.c.service 2024-09-13T14:13:58.235 DEBUG:teuthology.orchestra.run.smithi184:mon.b> sudo journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@mon.b.service 2024-09-13T14:13:58.237 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-13T14:13:58.238 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T14:13:58.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:58 smithi151 ceph-mon[29807]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:13:58.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:58 smithi151 ceph-mon[29807]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi151:172.21.15.151=a;smithi151:[v2:172.21.15.151:3301,v1:172.21.15.151:6790]=c;smithi184:172.21.15.184=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:58.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:58 smithi151 ceph-mon[29807]: Saving service mon spec with placement smithi151:172.21.15.151=a;smithi151:[v2:172.21.15.151:3301,v1:172.21.15.151:6790]=c;smithi184:172.21.15.184=b;count:3 2024-09-13T14:13:58.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:58 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:13:58.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:58 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:13:58.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:58 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:13:58.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:58 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:13:58.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:58 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:13:58.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:58 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:13:58.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:58 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:13:58.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:13:58 smithi151 ceph-mon[29807]: Deploying daemon mon.b on smithi184 2024-09-13T14:13:58.914 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:14:00.967 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:14:00.967 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"2401fc50-71da-11ef-bceb-c7b262605968","modified":"2024-09-13T14:12:33.302475Z","created":"2024-09-13T14:12:33.302475Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-13T14:14:00.967 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-09-13T14:14:01.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:00 smithi151 ceph-mon[29807]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:02.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:01 smithi151 ceph-mon[29807]: from='client.? 172.21.15.184:0/3705869150' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T14:14:03.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:02 smithi151 ceph-mon[29807]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:03.453 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:03 smithi184 ceph-mon[33394]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-13T14:14:03.508 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-13T14:14:03.509 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T14:14:03.750 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:14:03.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:03 smithi184 bash[33357]: 8c637f147650a7fd7c59e2e5c07bb367a05134b72133dcc42f392473f12e0316 2024-09-13T14:14:03.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:03 smithi184 systemd[1]: Started Ceph mon.b for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:14:08.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:08 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:14:08.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:08 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:14:08.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:08 smithi184 ceph-mon[33394]: mon.a calling monitor election 2024-09-13T14:14:08.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:08 smithi184 ceph-mon[33394]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:08.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:08 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:14:08.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:08 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:14:08.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:08 smithi184 ceph-mon[33394]: mon.b calling monitor election 2024-09-13T14:14:08.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:08 smithi184 ceph-mon[33394]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:08.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:08 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:14:08.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:08 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:14:08.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:08 smithi184 ceph-mon[33394]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:08.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:08 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:14:08.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:08 smithi184 ceph-mon[33394]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-13T14:14:08.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:08 smithi184 ceph-mon[33394]: monmap e2: 2 mons at {a=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0],b=[v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T14:14:08.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:08 smithi184 ceph-mon[33394]: fsmap 2024-09-13T14:14:08.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:08 smithi184 ceph-mon[33394]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T14:14:08.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:08 smithi184 ceph-mon[33394]: mgrmap e13: y(active, since 40s) 2024-09-13T14:14:08.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:08 smithi184 ceph-mon[33394]: overall HEALTH_OK 2024-09-13T14:14:08.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:08 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:08.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:08 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:08.856 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:08 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:08.856 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:08 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:14:08.856 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:08 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:08.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:08 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:14:08.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:08 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:14:08.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:08 smithi151 ceph-mon[29807]: mon.a calling monitor election 2024-09-13T14:14:08.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:08 smithi151 ceph-mon[29807]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:08.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:08 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:14:08.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:08 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:14:08.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:08 smithi151 ceph-mon[29807]: mon.b calling monitor election 2024-09-13T14:14:08.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:08 smithi151 ceph-mon[29807]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:08.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:08 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:14:08.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:08 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:14:08.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:08 smithi151 ceph-mon[29807]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:08.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:08 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:14:08.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:08 smithi151 ceph-mon[29807]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-13T14:14:08.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:08 smithi151 ceph-mon[29807]: monmap e2: 2 mons at {a=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0],b=[v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T14:14:08.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:08 smithi151 ceph-mon[29807]: fsmap 2024-09-13T14:14:08.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:08 smithi151 ceph-mon[29807]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T14:14:08.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:08 smithi151 ceph-mon[29807]: mgrmap e13: y(active, since 40s) 2024-09-13T14:14:08.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:08 smithi151 ceph-mon[29807]: overall HEALTH_OK 2024-09-13T14:14:08.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:08 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:08.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:08 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:08.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:08 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:08.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:08 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:14:08.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:08 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:09.069 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:14:09.070 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":2,"fsid":"2401fc50-71da-11ef-bceb-c7b262605968","modified":"2024-09-13T14:14:03.456835Z","created":"2024-09-13T14:12:33.302475Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:3300","nonce":0},{"type":"v1","addr":"172.21.15.184:6789","nonce":0}]},"addr":"172.21.15.184:6789/0","public_addr":"172.21.15.184:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-13T14:14:09.070 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 2 2024-09-13T14:14:09.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:09 smithi184 ceph-mon[33394]: Deploying daemon mon.c on smithi151 2024-09-13T14:14:09.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:09 smithi184 ceph-mon[33394]: from='client.? 172.21.15.184:0/738129772' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T14:14:09.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:09 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:14:09.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:09 smithi151 ceph-mon[29807]: Deploying daemon mon.c on smithi151 2024-09-13T14:14:09.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:09 smithi151 ceph-mon[29807]: from='client.? 172.21.15.184:0/738129772' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T14:14:09.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:09 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:14:10.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:10 smithi184 ceph-mon[33394]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:10.682 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-13T14:14:10.682 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T14:14:10.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:10 smithi151 ceph-mon[29807]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:10.920 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:14:11.591 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 podman[41349]: 2024-09-13 14:14:11.456062546 +0000 UTC m=+0.124820577 container create dccc3a65ea4d02896222322f753527616ca0a95bcac6150ea2558e81799e67b1 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-c, RELEASE=quincy-bbdadfa, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=) 2024-09-13T14:14:12.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 podman[41349]: 2024-09-13 14:14:11.734695257 +0000 UTC m=+0.403453291 container init dccc3a65ea4d02896222322f753527616ca0a95bcac6150ea2558e81799e67b1 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=quincy-bbdadfa, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-13T14:14:12.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 podman[41349]: 2024-09-13 14:14:11.739833635 +0000 UTC m=+0.408591667 container start dccc3a65ea4d02896222322f753527616ca0a95bcac6150ea2558e81799e67b1 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-c, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-bbdadfa, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:14:12.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: set uid:gid to 167:167 (ceph:ceph) 2024-09-13T14:14:12.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable), process ceph-mon, pid 7 2024-09-13T14:14:12.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: pidfile_write: ignore empty --pid-file 2024-09-13T14:14:12.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: load: jerasure load: lrc 2024-09-13T14:14:12.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: RocksDB version: 6.15.5 2024-09-13T14:14:12.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-13T14:14:12.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Compile date Sep 12 2024 2024-09-13T14:14:12.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: DB SUMMARY 2024-09-13T14:14:12.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: DB Session ID: 6ZNTGRSZSZL8JZ2EMBB6 2024-09-13T14:14:12.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: CURRENT file: CURRENT 2024-09-13T14:14:12.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: IDENTITY file: IDENTITY 2024-09-13T14:14:12.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-09-13T14:14:12.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-13T14:14:12.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-09-13T14:14:12.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.error_if_exists: 0 2024-09-13T14:14:12.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.create_if_missing: 0 2024-09-13T14:14:12.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.paranoid_checks: 1 2024-09-13T14:14:12.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-13T14:14:12.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.env: 0x561c06686440 2024-09-13T14:14:12.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.fs: Posix File System 2024-09-13T14:14:12.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.info_log: 0x561c0804eec0 2024-09-13T14:14:12.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.max_file_opening_threads: 16 2024-09-13T14:14:12.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.statistics: (nil) 2024-09-13T14:14:12.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.use_fsync: 0 2024-09-13T14:14:12.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.max_log_file_size: 0 2024-09-13T14:14:12.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-13T14:14:12.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-13T14:14:12.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.keep_log_file_num: 1000 2024-09-13T14:14:12.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.recycle_log_file_num: 0 2024-09-13T14:14:12.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.allow_fallocate: 1 2024-09-13T14:14:12.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.allow_mmap_reads: 0 2024-09-13T14:14:12.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.allow_mmap_writes: 0 2024-09-13T14:14:12.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.use_direct_reads: 0 2024-09-13T14:14:12.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-13T14:14:12.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.create_missing_column_families: 0 2024-09-13T14:14:12.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.db_log_dir: 2024-09-13T14:14:12.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-09-13T14:14:12.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-13T14:14:12.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-13T14:14:12.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-13T14:14:12.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-13T14:14:12.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-13T14:14:12.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-13T14:14:12.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.advise_random_on_open: 1 2024-09-13T14:14:12.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.db_write_buffer_size: 0 2024-09-13T14:14:12.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.write_buffer_manager: 0x561c0810c7b0 2024-09-13T14:14:12.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-13T14:14:12.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-13T14:14:12.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-13T14:14:12.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-13T14:14:12.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.rate_limiter: (nil) 2024-09-13T14:14:12.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-13T14:14:12.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.wal_recovery_mode: 2 2024-09-13T14:14:12.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.enable_thread_tracking: 0 2024-09-13T14:14:12.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.enable_pipelined_write: 0 2024-09-13T14:14:12.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.unordered_write: 0 2024-09-13T14:14:12.114 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-13T14:14:12.114 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-13T14:14:12.114 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-13T14:14:12.114 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-13T14:14:12.114 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.row_cache: None 2024-09-13T14:14:12.114 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.wal_filter: None 2024-09-13T14:14:12.114 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-13T14:14:12.115 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.allow_ingest_behind: 0 2024-09-13T14:14:12.115 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.preserve_deletes: 0 2024-09-13T14:14:12.116 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.two_write_queues: 0 2024-09-13T14:14:12.116 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.manual_wal_flush: 0 2024-09-13T14:14:12.116 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.atomic_flush: 0 2024-09-13T14:14:12.116 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-13T14:14:12.116 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-13T14:14:12.116 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-13T14:14:12.116 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.log_readahead_size: 0 2024-09-13T14:14:12.116 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-13T14:14:12.116 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.best_efforts_recovery: 0 2024-09-13T14:14:12.116 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-13T14:14:12.116 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-13T14:14:12.116 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.allow_data_in_errors: 0 2024-09-13T14:14:12.116 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.db_host_id: __hostname__ 2024-09-13T14:14:12.117 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.max_background_jobs: 2 2024-09-13T14:14:12.117 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.max_background_compactions: -1 2024-09-13T14:14:12.117 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.max_subcompactions: 1 2024-09-13T14:14:12.117 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-13T14:14:12.117 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-13T14:14:12.117 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-13T14:14:12.117 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.max_total_wal_size: 0 2024-09-13T14:14:12.117 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-13T14:14:12.117 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-13T14:14:12.117 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-13T14:14:12.117 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-13T14:14:12.117 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.max_open_files: -1 2024-09-13T14:14:12.117 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.bytes_per_sync: 0 2024-09-13T14:14:12.118 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-13T14:14:12.118 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-13T14:14:12.118 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.compaction_readahead_size: 0 2024-09-13T14:14:12.118 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.max_background_flushes: -1 2024-09-13T14:14:12.118 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Compression algorithms supported: 2024-09-13T14:14:12.118 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: kZSTD supported: 0 2024-09-13T14:14:12.118 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: kXpressCompression supported: 0 2024-09-13T14:14:12.118 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: kBZip2Compression supported: 0 2024-09-13T14:14:12.118 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-13T14:14:12.118 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: kLZ4Compression supported: 1 2024-09-13T14:14:12.118 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: kZlibCompression supported: 1 2024-09-13T14:14:12.118 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: kLZ4HCCompression supported: 1 2024-09-13T14:14:12.119 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: kSnappyCompression supported: 1 2024-09-13T14:14:12.119 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-13T14:14:12.119 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-09-13T14:14:12.119 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-09-13T14:14:12.119 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-13T14:14:12.119 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.merge_operator: 2024-09-13T14:14:12.119 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.compaction_filter: None 2024-09-13T14:14:12.119 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.compaction_filter_factory: None 2024-09-13T14:14:12.119 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.sst_partitioner_factory: None 2024-09-13T14:14:12.119 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-13T14:14:12.119 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-13T14:14:12.119 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561c07ff5a38) 2024-09-13T14:14:12.119 INFO:journalctl@ceph.mon.c.smithi151.stdout: cache_index_and_filter_blocks: 1 2024-09-13T14:14:12.119 INFO:journalctl@ceph.mon.c.smithi151.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-13T14:14:12.120 INFO:journalctl@ceph.mon.c.smithi151.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-13T14:14:12.120 INFO:journalctl@ceph.mon.c.smithi151.stdout: pin_top_level_index_and_filter: 1 2024-09-13T14:14:12.120 INFO:journalctl@ceph.mon.c.smithi151.stdout: index_type: 0 2024-09-13T14:14:12.120 INFO:journalctl@ceph.mon.c.smithi151.stdout: data_block_index_type: 0 2024-09-13T14:14:12.120 INFO:journalctl@ceph.mon.c.smithi151.stdout: index_shortening: 1 2024-09-13T14:14:12.120 INFO:journalctl@ceph.mon.c.smithi151.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-13T14:14:12.120 INFO:journalctl@ceph.mon.c.smithi151.stdout: hash_index_allow_collision: 1 2024-09-13T14:14:12.120 INFO:journalctl@ceph.mon.c.smithi151.stdout: checksum: 1 2024-09-13T14:14:12.120 INFO:journalctl@ceph.mon.c.smithi151.stdout: no_block_cache: 0 2024-09-13T14:14:12.120 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_cache: 0x561c0802b1f0 2024-09-13T14:14:12.120 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_cache_name: BinnedLRUCache 2024-09-13T14:14:12.120 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_cache_options: 2024-09-13T14:14:12.120 INFO:journalctl@ceph.mon.c.smithi151.stdout: capacity : 536870912 2024-09-13T14:14:12.120 INFO:journalctl@ceph.mon.c.smithi151.stdout: num_shard_bits : 4 2024-09-13T14:14:12.121 INFO:journalctl@ceph.mon.c.smithi151.stdout: strict_capacity_limit : 0 2024-09-13T14:14:12.121 INFO:journalctl@ceph.mon.c.smithi151.stdout: high_pri_pool_ratio: 0.000 2024-09-13T14:14:12.121 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_cache_compressed: (nil) 2024-09-13T14:14:12.121 INFO:journalctl@ceph.mon.c.smithi151.stdout: persistent_cache: (nil) 2024-09-13T14:14:12.121 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_size: 4096 2024-09-13T14:14:12.121 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_size_deviation: 10 2024-09-13T14:14:12.121 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_restart_interval: 16 2024-09-13T14:14:12.121 INFO:journalctl@ceph.mon.c.smithi151.stdout: index_block_restart_interval: 1 2024-09-13T14:14:12.121 INFO:journalctl@ceph.mon.c.smithi151.stdout: metadata_block_size: 4096 2024-09-13T14:14:12.121 INFO:journalctl@ceph.mon.c.smithi151.stdout: partition_filters: 0 2024-09-13T14:14:12.121 INFO:journalctl@ceph.mon.c.smithi151.stdout: use_delta_encoding: 1 2024-09-13T14:14:12.121 INFO:journalctl@ceph.mon.c.smithi151.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-13T14:14:12.121 INFO:journalctl@ceph.mon.c.smithi151.stdout: whole_key_filtering: 1 2024-09-13T14:14:12.121 INFO:journalctl@ceph.mon.c.smithi151.stdout: verify_compression: 0 2024-09-13T14:14:12.122 INFO:journalctl@ceph.mon.c.smithi151.stdout: read_amp_bytes_per_bit: 0 2024-09-13T14:14:12.122 INFO:journalctl@ceph.mon.c.smithi151.stdout: format_version: 4 2024-09-13T14:14:12.122 INFO:journalctl@ceph.mon.c.smithi151.stdout: enable_index_compression: 1 2024-09-13T14:14:12.122 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_align: 0 2024-09-13T14:14:12.122 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.write_buffer_size: 33554432 2024-09-13T14:14:12.122 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.max_write_buffer_number: 2 2024-09-13T14:14:12.122 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.compression: NoCompression 2024-09-13T14:14:12.122 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.bottommost_compression: Disabled 2024-09-13T14:14:12.122 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.prefix_extractor: nullptr 2024-09-13T14:14:12.122 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-13T14:14:12.122 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.num_levels: 7 2024-09-13T14:14:12.122 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-13T14:14:12.122 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-13T14:14:12.123 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-13T14:14:12.123 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-13T14:14:12.123 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-13T14:14:12.123 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-13T14:14:12.123 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-13T14:14:12.123 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:14:12.123 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-13T14:14:12.123 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-13T14:14:12.123 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-13T14:14:12.123 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.compression_opts.level: 32767 2024-09-13T14:14:12.123 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.compression_opts.strategy: 0 2024-09-13T14:14:12.123 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-13T14:14:12.123 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:14:12.124 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-13T14:14:12.124 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.compression_opts.enabled: false 2024-09-13T14:14:12.124 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-13T14:14:12.124 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-13T14:14:12.124 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-13T14:14:12.124 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.target_file_size_base: 67108864 2024-09-13T14:14:12.124 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-13T14:14:12.124 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-13T14:14:12.124 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-13T14:14:12.124 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-13T14:14:12.124 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-13T14:14:12.124 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-13T14:14:12.125 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-13T14:14:12.125 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-13T14:14:12.125 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-13T14:14:12.125 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-13T14:14:12.125 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-13T14:14:12.125 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-13T14:14:12.125 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-13T14:14:12.125 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.arena_block_size: 4194304 2024-09-13T14:14:12.125 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-13T14:14:12.125 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-13T14:14:12.125 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-13T14:14:12.125 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.disable_auto_compactions: 0 2024-09-13T14:14:12.125 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-13T14:14:12.126 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-13T14:14:12.126 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-13T14:14:12.126 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-13T14:14:12.126 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-13T14:14:12.126 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-13T14:14:12.126 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-13T14:14:12.126 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-13T14:14:12.126 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-13T14:14:12.126 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-13T14:14:12.126 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-13T14:14:12.126 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.inplace_update_support: 0 2024-09-13T14:14:12.126 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-13T14:14:12.126 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-13T14:14:12.126 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-13T14:14:12.127 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-13T14:14:12.127 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.bloom_locality: 0 2024-09-13T14:14:12.127 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.max_successive_merges: 0 2024-09-13T14:14:12.127 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-13T14:14:12.127 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.paranoid_file_checks: 0 2024-09-13T14:14:12.127 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.force_consistency_checks: 1 2024-09-13T14:14:12.127 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.report_bg_io_stats: 0 2024-09-13T14:14:12.127 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.ttl: 2592000 2024-09-13T14:14:12.127 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-13T14:14:12.127 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.enable_blob_files: false 2024-09-13T14:14:12.127 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.min_blob_size: 0 2024-09-13T14:14:12.127 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.blob_file_size: 268435456 2024-09-13T14:14:12.127 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-13T14:14:12.127 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-13T14:14:12.128 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-13T14:14:12.128 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-13T14:14:12.128 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2024-09-13T14:14:12.128 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2024-09-13T14:14:12.128 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726236851763539, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-13T14:14:12.128 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2024-09-13T14:14:12.128 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-13T14:14:12.128 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726236851763844, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1696, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1726236851, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "c1e19c29-a9b1-46fb-82b9-f180b02a989c", "db_session_id": "6ZNTGRSZSZL8JZ2EMBB6"}} 2024-09-13T14:14:12.128 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2024-09-13T14:14:12.128 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726236851764158, "job": 1, "event": "recovery_finished"} 2024-09-13T14:14:12.128 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-13T14:14:12.128 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x561c0805e700 2024-09-13T14:14:12.128 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: DB pointer 0x561c0802e000 2024-09-13T14:14:12.129 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-09-13T14:14:12.129 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-13T14:14:12.129 INFO:journalctl@ceph.mon.c.smithi151.stdout: ** DB Stats ** 2024-09-13T14:14:12.129 INFO:journalctl@ceph.mon.c.smithi151.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:14:12.129 INFO:journalctl@ceph.mon.c.smithi151.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-13T14:14:12.129 INFO:journalctl@ceph.mon.c.smithi151.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:14:12.129 INFO:journalctl@ceph.mon.c.smithi151.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:14:12.129 INFO:journalctl@ceph.mon.c.smithi151.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-13T14:14:12.129 INFO:journalctl@ceph.mon.c.smithi151.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-13T14:14:12.129 INFO:journalctl@ceph.mon.c.smithi151.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:14:12.129 INFO:journalctl@ceph.mon.c.smithi151.stdout: 2024-09-13T14:14:12.129 INFO:journalctl@ceph.mon.c.smithi151.stdout: ** Compaction Stats [default] ** 2024-09-13T14:14:12.129 INFO:journalctl@ceph.mon.c.smithi151.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-13T14:14:12.129 INFO:journalctl@ceph.mon.c.smithi151.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T14:14:12.130 INFO:journalctl@ceph.mon.c.smithi151.stdout: L0 1/0 1.66 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-09-13T14:14:12.130 INFO:journalctl@ceph.mon.c.smithi151.stdout: Sum 1/0 1.66 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-09-13T14:14:12.130 INFO:journalctl@ceph.mon.c.smithi151.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-09-13T14:14:12.130 INFO:journalctl@ceph.mon.c.smithi151.stdout: 2024-09-13T14:14:12.130 INFO:journalctl@ceph.mon.c.smithi151.stdout: ** Compaction Stats [default] ** 2024-09-13T14:14:12.130 INFO:journalctl@ceph.mon.c.smithi151.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-13T14:14:12.130 INFO:journalctl@ceph.mon.c.smithi151.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T14:14:12.130 INFO:journalctl@ceph.mon.c.smithi151.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-09-13T14:14:12.130 INFO:journalctl@ceph.mon.c.smithi151.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:14:12.130 INFO:journalctl@ceph.mon.c.smithi151.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-13T14:14:12.130 INFO:journalctl@ceph.mon.c.smithi151.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-13T14:14:12.130 INFO:journalctl@ceph.mon.c.smithi151.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-13T14:14:12.130 INFO:journalctl@ceph.mon.c.smithi151.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-13T14:14:12.131 INFO:journalctl@ceph.mon.c.smithi151.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-13T14:14:12.131 INFO:journalctl@ceph.mon.c.smithi151.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:14:12.131 INFO:journalctl@ceph.mon.c.smithi151.stdout: Interval compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:14:12.131 INFO:journalctl@ceph.mon.c.smithi151.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-13T14:14:12.131 INFO:journalctl@ceph.mon.c.smithi151.stdout: 2024-09-13T14:14:12.131 INFO:journalctl@ceph.mon.c.smithi151.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-13T14:14:12.131 INFO:journalctl@ceph.mon.c.smithi151.stdout: 2024-09-13T14:14:12.131 INFO:journalctl@ceph.mon.c.smithi151.stdout: ** Compaction Stats [default] ** 2024-09-13T14:14:12.131 INFO:journalctl@ceph.mon.c.smithi151.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-13T14:14:12.131 INFO:journalctl@ceph.mon.c.smithi151.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T14:14:12.131 INFO:journalctl@ceph.mon.c.smithi151.stdout: L0 1/0 1.66 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-09-13T14:14:12.131 INFO:journalctl@ceph.mon.c.smithi151.stdout: Sum 1/0 1.66 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-09-13T14:14:12.131 INFO:journalctl@ceph.mon.c.smithi151.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-13T14:14:12.131 INFO:journalctl@ceph.mon.c.smithi151.stdout: 2024-09-13T14:14:12.132 INFO:journalctl@ceph.mon.c.smithi151.stdout: ** Compaction Stats [default] ** 2024-09-13T14:14:12.132 INFO:journalctl@ceph.mon.c.smithi151.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-13T14:14:12.132 INFO:journalctl@ceph.mon.c.smithi151.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T14:14:12.132 INFO:journalctl@ceph.mon.c.smithi151.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-09-13T14:14:12.132 INFO:journalctl@ceph.mon.c.smithi151.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:14:12.132 INFO:journalctl@ceph.mon.c.smithi151.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-13T14:14:12.132 INFO:journalctl@ceph.mon.c.smithi151.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-13T14:14:12.132 INFO:journalctl@ceph.mon.c.smithi151.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-13T14:14:12.132 INFO:journalctl@ceph.mon.c.smithi151.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-13T14:14:12.132 INFO:journalctl@ceph.mon.c.smithi151.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-13T14:14:12.132 INFO:journalctl@ceph.mon.c.smithi151.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:14:12.132 INFO:journalctl@ceph.mon.c.smithi151.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:14:12.132 INFO:journalctl@ceph.mon.c.smithi151.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-13T14:14:12.133 INFO:journalctl@ceph.mon.c.smithi151.stdout: 2024-09-13T14:14:12.133 INFO:journalctl@ceph.mon.c.smithi151.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-13T14:14:12.133 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-13T14:14:12.133 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: using public_addrv [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2024-09-13T14:14:12.133 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: starting mon.c rank -1 at public addrs [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] at bind addrs [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:14:12.133 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: mon.c@-1(???) e0 preinit fsid 2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:14:12.133 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: mon.c@-1(synchronizing).mds e1 new map 2024-09-13T14:14:12.133 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-13T14:14:12.133 INFO:journalctl@ceph.mon.c.smithi151.stdout: e1 2024-09-13T14:14:12.133 INFO:journalctl@ceph.mon.c.smithi151.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-13T14:14:12.133 INFO:journalctl@ceph.mon.c.smithi151.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T14:14:12.133 INFO:journalctl@ceph.mon.c.smithi151.stdout: legacy client fscid: -1 2024-09-13T14:14:12.133 INFO:journalctl@ceph.mon.c.smithi151.stdout: 2024-09-13T14:14:12.133 INFO:journalctl@ceph.mon.c.smithi151.stdout: No filesystems configured 2024-09-13T14:14:12.134 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-13T14:14:12.134 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-13T14:14:12.134 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-13T14:14:12.134 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-13T14:14:12.134 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-13T14:14:12.134 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-13T14:14:12.134 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-13T14:14:12.134 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T14:14:12.134 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T14:14:12.134 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T14:14:12.134 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: from='client.? 172.21.15.184:0/3705869150' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T14:14:12.134 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:12.134 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:14:12.135 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:14:12.135 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: mon.a calling monitor election 2024-09-13T14:14:12.135 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:12.135 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:14:12.135 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:14:12.135 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: mon.b calling monitor election 2024-09-13T14:14:12.135 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:12.135 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:14:12.135 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:14:12.135 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:12.135 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:14:12.135 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-13T14:14:12.135 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: monmap e2: 2 mons at {a=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0],b=[v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T14:14:12.135 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: fsmap 2024-09-13T14:14:12.136 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T14:14:12.136 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: mgrmap e13: y(active, since 40s) 2024-09-13T14:14:12.136 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: overall HEALTH_OK 2024-09-13T14:14:12.136 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:12.136 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:12.136 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:12.136 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:14:12.136 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:12.136 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: Deploying daemon mon.c on smithi151 2024-09-13T14:14:12.136 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: from='client.? 172.21.15.184:0/738129772' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T14:14:12.136 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:14:12.136 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:12.136 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 ceph-mon[41387]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-13T14:14:12.137 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 bash[41349]: dccc3a65ea4d02896222322f753527616ca0a95bcac6150ea2558e81799e67b1 2024-09-13T14:14:12.137 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:11 smithi151 systemd[1]: Started Ceph mon.c for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:14:17.245 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:14:17.245 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":3,"fsid":"2401fc50-71da-11ef-bceb-c7b262605968","modified":"2024-09-13T14:14:11.796948Z","created":"2024-09-13T14:12:33.302475Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:3300","nonce":0},{"type":"v1","addr":"172.21.15.184:6789","nonce":0}]},"addr":"172.21.15.184:6789/0","public_addr":"172.21.15.184:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3301","nonce":0},{"type":"v1","addr":"172.21.15.151:6790","nonce":0}]},"addr":"172.21.15.151:6790/0","public_addr":"172.21.15.151:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-13T14:14:17.245 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 3 2024-09-13T14:14:17.246 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:16 smithi184 ceph-mon[33394]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:17.246 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:16 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:14:17.246 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:16 smithi184 ceph-mon[33394]: mon.a calling monitor election 2024-09-13T14:14:17.246 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:16 smithi184 ceph-mon[33394]: mon.b calling monitor election 2024-09-13T14:14:17.246 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:16 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:14:17.246 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:16 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:14:17.246 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:16 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:14:17.246 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:16 smithi184 ceph-mon[33394]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:17.246 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:16 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:14:17.246 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:16 smithi184 ceph-mon[33394]: mon.c calling monitor election 2024-09-13T14:14:17.246 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:16 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:14:17.247 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:16 smithi184 ceph-mon[33394]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:17.247 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:16 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:14:17.247 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:16 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:14:17.247 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:16 smithi184 ceph-mon[33394]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:14:17.247 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:16 smithi184 ceph-mon[33394]: monmap e3: 3 mons at {a=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0],b=[v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0],c=[v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T14:14:17.247 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:16 smithi184 ceph-mon[33394]: fsmap 2024-09-13T14:14:17.247 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:16 smithi184 ceph-mon[33394]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T14:14:17.247 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:16 smithi184 ceph-mon[33394]: mgrmap e13: y(active, since 49s) 2024-09-13T14:14:17.247 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:16 smithi184 ceph-mon[33394]: overall HEALTH_OK 2024-09-13T14:14:17.247 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:16 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:17.247 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:16 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:17.247 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:16 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:17.247 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:16 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:17.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[29807]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:17.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:14:17.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[29807]: mon.a calling monitor election 2024-09-13T14:14:17.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[29807]: mon.b calling monitor election 2024-09-13T14:14:17.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:14:17.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:14:17.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:14:17.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[29807]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:17.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:14:17.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[29807]: mon.c calling monitor election 2024-09-13T14:14:17.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:14:17.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[29807]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:17.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:14:17.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:14:17.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[29807]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:14:17.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[29807]: monmap e3: 3 mons at {a=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0],b=[v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0],c=[v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T14:14:17.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[29807]: fsmap 2024-09-13T14:14:17.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[29807]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T14:14:17.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[29807]: mgrmap e13: y(active, since 49s) 2024-09-13T14:14:17.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[29807]: overall HEALTH_OK 2024-09-13T14:14:17.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:17.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:17.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:17.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:17.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[41387]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:17.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:14:17.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[41387]: mon.a calling monitor election 2024-09-13T14:14:17.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[41387]: mon.b calling monitor election 2024-09-13T14:14:17.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:14:17.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:14:17.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:14:17.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[41387]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:17.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:14:17.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[41387]: mon.c calling monitor election 2024-09-13T14:14:17.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:14:17.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[41387]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:17.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:14:17.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:14:17.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[41387]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:14:17.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[41387]: monmap e3: 3 mons at {a=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0],b=[v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0],c=[v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T14:14:17.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[41387]: fsmap 2024-09-13T14:14:17.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[41387]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T14:14:17.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[41387]: mgrmap e13: y(active, since 49s) 2024-09-13T14:14:17.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[41387]: overall HEALTH_OK 2024-09-13T14:14:17.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:17.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:17.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:17.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:16 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:17.844 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-13T14:14:17.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph config generate-minimal-conf 2024-09-13T14:14:18.101 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:14:18.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:17 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:18.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:17 smithi184 ceph-mon[33394]: from='client.? 172.21.15.184:0/3403834127' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T14:14:18.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:17 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:18.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:17 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:14:18.222 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:17 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:18.222 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:17 smithi151 ceph-mon[29807]: from='client.? 172.21.15.184:0/3403834127' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T14:14:18.223 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:17 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:18.223 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:17 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:14:18.223 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:17 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:18.223 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:17 smithi151 ceph-mon[41387]: from='client.? 172.21.15.184:0/3403834127' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T14:14:18.223 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:17 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:18.223 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:17 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:14:19.079 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:18 smithi151 ceph-mon[29807]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:19.079 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:18 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:19.079 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:18 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:19.079 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:18 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:19.079 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:18 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:19.079 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:18 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:19.079 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:18 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:14:19.080 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:18 smithi151 ceph-mon[41387]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:19.080 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:18 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:19.080 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:18 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:19.080 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:18 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:19.080 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:18 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:19.080 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:18 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:19.080 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:18 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:14:19.080 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:14:18 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:14:18.796+0000 7f398435b640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-13T14:14:19.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:18 smithi184 ceph-mon[33394]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:19.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:18 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:19.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:18 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:19.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:18 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:19.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:18 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:19.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:18 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:19.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:18 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:14:20.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:19 smithi184 ceph-mon[33394]: Updating smithi151:/etc/ceph/ceph.conf 2024-09-13T14:14:20.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:19 smithi184 ceph-mon[33394]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-13T14:14:20.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:19 smithi184 ceph-mon[33394]: Updating smithi184:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:14:20.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:19 smithi184 ceph-mon[33394]: Updating smithi151:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:14:20.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:19 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:20.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:19 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:20.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:19 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:20.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:19 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:20.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:19 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:20.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:19 smithi184 ceph-mon[33394]: Reconfiguring mon.a (unknown last config time)... 2024-09-13T14:14:20.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:19 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:14:20.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:19 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:14:20.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:19 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:20.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:19 smithi184 ceph-mon[33394]: Reconfiguring daemon mon.a on smithi151 2024-09-13T14:14:20.170 INFO:teuthology.orchestra.run.smithi151.stdout:# minimal ceph.conf for 2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:14:20.170 INFO:teuthology.orchestra.run.smithi151.stdout:[global] 2024-09-13T14:14:20.170 INFO:teuthology.orchestra.run.smithi151.stdout: fsid = 2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:14:20.170 INFO:teuthology.orchestra.run.smithi151.stdout: mon_host = [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2024-09-13T14:14:20.181 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:19 smithi151 ceph-mon[41387]: Updating smithi151:/etc/ceph/ceph.conf 2024-09-13T14:14:20.181 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:19 smithi151 ceph-mon[41387]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-13T14:14:20.181 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:19 smithi151 ceph-mon[41387]: Updating smithi184:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:14:20.182 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:19 smithi151 ceph-mon[41387]: Updating smithi151:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:14:20.182 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:19 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:20.182 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:19 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:20.182 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:19 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:20.182 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:19 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:20.182 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:19 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:20.182 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:19 smithi151 ceph-mon[41387]: Reconfiguring mon.a (unknown last config time)... 2024-09-13T14:14:20.182 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:19 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:14:20.182 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:19 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:14:20.182 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:19 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:20.182 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:19 smithi151 ceph-mon[41387]: Reconfiguring daemon mon.a on smithi151 2024-09-13T14:14:20.183 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:19 smithi151 ceph-mon[29807]: Updating smithi151:/etc/ceph/ceph.conf 2024-09-13T14:14:20.183 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:19 smithi151 ceph-mon[29807]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-13T14:14:20.183 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:19 smithi151 ceph-mon[29807]: Updating smithi184:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:14:20.183 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:19 smithi151 ceph-mon[29807]: Updating smithi151:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:14:20.183 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:19 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:20.183 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:19 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:20.183 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:19 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:20.183 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:19 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:20.183 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:19 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:20.183 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:19 smithi151 ceph-mon[29807]: Reconfiguring mon.a (unknown last config time)... 2024-09-13T14:14:20.183 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:19 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:14:20.183 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:19 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:14:20.183 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:19 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:20.184 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:19 smithi151 ceph-mon[29807]: Reconfiguring daemon mon.a on smithi151 2024-09-13T14:14:21.082 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:20 smithi151 ceph-mon[29807]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:21.082 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:20 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1903564835' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:21.082 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:20 smithi151 ceph-mon[41387]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:21.082 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:20 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/1903564835' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:21.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:20 smithi184 ceph-mon[33394]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:21.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:20 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/1903564835' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:21.540 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-13T14:14:21.541 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-13T14:14:21.541 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.conf 2024-09-13T14:14:21.567 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-13T14:14:21.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:14:21.633 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-13T14:14:21.633 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/ceph.conf 2024-09-13T14:14:21.666 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-13T14:14:21.667 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:14:21.737 INFO:tasks.cephadm:Adding mgr.y on smithi151 2024-09-13T14:14:21.738 INFO:tasks.cephadm:Adding mgr.x on smithi184 2024-09-13T14:14:21.738 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph orch apply mgr '2;smithi151=y;smithi184=x' 2024-09-13T14:14:22.021 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:14:22.416 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:22 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:22.416 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:22 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:22.416 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:22 smithi151 ceph-mon[29807]: Reconfiguring mon.c (monmap changed)... 2024-09-13T14:14:22.417 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:22 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:14:22.417 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:22 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:14:22.417 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:22 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:22.417 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:22 smithi151 ceph-mon[29807]: Reconfiguring daemon mon.c on smithi151 2024-09-13T14:14:22.417 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:22 smithi151 ceph-mon[29807]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:22.417 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:22 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:22.417 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:22 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:22.417 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:22 smithi151 ceph-mon[41387]: Reconfiguring mon.c (monmap changed)... 2024-09-13T14:14:22.417 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:22 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:14:22.417 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:22 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:14:22.417 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:22 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:22.418 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:22 smithi151 ceph-mon[41387]: Reconfiguring daemon mon.c on smithi151 2024-09-13T14:14:22.418 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:22 smithi151 ceph-mon[41387]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:22.502 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:22 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:22.502 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:22 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:22.502 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:22 smithi184 ceph-mon[33394]: Reconfiguring mon.c (monmap changed)... 2024-09-13T14:14:22.502 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:22 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:14:22.502 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:22 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:14:22.502 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:22 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:22.502 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:22 smithi184 ceph-mon[33394]: Reconfiguring daemon mon.c on smithi151 2024-09-13T14:14:22.502 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:22 smithi184 ceph-mon[33394]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:23.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:23 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:23.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:23 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:23.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:23 smithi184 ceph-mon[33394]: Reconfiguring mon.b (monmap changed)... 2024-09-13T14:14:23.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:23 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:14:23.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:23 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:14:23.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:23 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:23.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:23 smithi184 ceph-mon[33394]: Reconfiguring daemon mon.b on smithi184 2024-09-13T14:14:24.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:23 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:24.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:23 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:24.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:23 smithi151 ceph-mon[29807]: Reconfiguring mon.b (monmap changed)... 2024-09-13T14:14:24.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:23 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:14:24.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:23 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:14:24.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:23 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:24.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:23 smithi151 ceph-mon[29807]: Reconfiguring daemon mon.b on smithi184 2024-09-13T14:14:24.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:23 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:24.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:23 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:24.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:23 smithi151 ceph-mon[41387]: Reconfiguring mon.b (monmap changed)... 2024-09-13T14:14:24.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:23 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:14:24.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:23 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:14:24.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:23 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:24.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:23 smithi151 ceph-mon[41387]: Reconfiguring daemon mon.b on smithi184 2024-09-13T14:14:24.325 INFO:teuthology.orchestra.run.smithi184.stdout:Scheduled mgr update... 2024-09-13T14:14:25.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:24 smithi184 ceph-mon[33394]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:25.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:24 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:25.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:24 smithi151 ceph-mon[29807]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:25.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:24 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:25.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:24 smithi151 ceph-mon[41387]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:25.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:24 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:25.168 DEBUG:teuthology.orchestra.run.smithi184:mgr.x> sudo journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@mgr.x.service 2024-09-13T14:14:25.170 DEBUG:tasks.cephadm:set 0 configs 2024-09-13T14:14:25.170 INFO:tasks.cephadm:Deploying OSDs... 2024-09-13T14:14:25.170 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-13T14:14:25.171 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T14:14:25.190 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T14:14:25.191 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_1 2024-09-13T14:14:25.249 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T14:14:25.249 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:14:25.249 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 984 Links: 1 2024-09-13T14:14:25.249 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:14:25.250 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:14:25.250 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-09-13 14:14:11.173638320 +0000 2024-09-13T14:14:25.250 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-09-13 14:09:54.224729340 +0000 2024-09-13T14:14:25.250 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-09-13 14:09:54.224729340 +0000 2024-09-13T14:14:25.250 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-09-13 14:09:54.224729340 +0000 2024-09-13T14:14:25.250 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T14:14:25.318 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-09-13T14:14:25.318 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-09-13T14:14:25.318 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000792302 s, 646 kB/s 2024-09-13T14:14:25.319 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T14:14:25.378 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_2 2024-09-13T14:14:25.435 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T14:14:25.435 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:14:25.436 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 978 Links: 1 2024-09-13T14:14:25.436 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:14:25.436 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:14:25.436 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-09-13 14:14:11.173638320 +0000 2024-09-13T14:14:25.436 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-09-13 14:09:54.222729364 +0000 2024-09-13T14:14:25.436 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-09-13 14:09:54.222729364 +0000 2024-09-13T14:14:25.436 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-09-13 14:09:54.222729364 +0000 2024-09-13T14:14:25.436 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T14:14:25.502 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-09-13T14:14:25.502 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-09-13T14:14:25.502 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000287684 s, 1.8 MB/s 2024-09-13T14:14:25.503 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T14:14:25.561 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_3 2024-09-13T14:14:25.618 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T14:14:25.618 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:14:25.619 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 969 Links: 1 2024-09-13T14:14:25.619 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:14:25.619 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:14:25.619 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-09-13 14:14:11.174638308 +0000 2024-09-13T14:14:25.619 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-09-13 14:09:54.221729376 +0000 2024-09-13T14:14:25.619 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-09-13 14:09:54.221729376 +0000 2024-09-13T14:14:25.619 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-09-13 14:09:54.221729376 +0000 2024-09-13T14:14:25.619 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T14:14:25.683 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-09-13T14:14:25.683 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-09-13T14:14:25.683 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000241051 s, 2.1 MB/s 2024-09-13T14:14:25.684 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T14:14:25.741 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_4 2024-09-13T14:14:25.797 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T14:14:25.798 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:14:25.798 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 974 Links: 1 2024-09-13T14:14:25.798 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:14:25.798 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:14:25.798 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-09-13 14:14:11.174638308 +0000 2024-09-13T14:14:25.798 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-09-13 14:09:54.222729364 +0000 2024-09-13T14:14:25.798 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-09-13 14:09:54.222729364 +0000 2024-09-13T14:14:25.798 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-09-13 14:09:54.222729364 +0000 2024-09-13T14:14:25.798 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T14:14:25.862 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-09-13T14:14:25.862 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-09-13T14:14:25.862 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.00024746 s, 2.1 MB/s 2024-09-13T14:14:25.863 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T14:14:25.920 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-13T14:14:25.920 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T14:14:25.938 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T14:14:25.938 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_1 2024-09-13T14:14:25.995 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T14:14:25.995 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:14:25.995 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 984 Links: 1 2024-09-13T14:14:25.995 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:14:25.995 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:14:25.996 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-09-13 14:14:02.930302410 +0000 2024-09-13T14:14:25.996 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-09-13 14:09:53.567427310 +0000 2024-09-13T14:14:25.996 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-09-13 14:09:53.567427310 +0000 2024-09-13T14:14:25.996 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-09-13 14:09:53.567427310 +0000 2024-09-13T14:14:25.996 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T14:14:26.042 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:25 smithi184 ceph-mon[33394]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi151=y;smithi184=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:26.042 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:25 smithi184 ceph-mon[33394]: Saving service mgr spec with placement smithi151=y;smithi184=x;count:2 2024-09-13T14:14:26.042 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:25 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:26.042 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:25 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:26.043 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:25 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:26.043 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:25 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:26.043 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:25 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:14:26.043 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:25 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:26.043 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:25 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:14:26.043 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:25 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-13T14:14:26.043 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:25 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:14:26.043 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:25 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:26.043 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:25 smithi184 ceph-mon[33394]: Deploying daemon mgr.x on smithi184 2024-09-13T14:14:26.061 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-09-13T14:14:26.061 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-09-13T14:14:26.061 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000251642 s, 2.0 MB/s 2024-09-13T14:14:26.062 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T14:14:26.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:25 smithi151 ceph-mon[29807]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi151=y;smithi184=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:26.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:25 smithi151 ceph-mon[29807]: Saving service mgr spec with placement smithi151=y;smithi184=x;count:2 2024-09-13T14:14:26.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:25 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:26.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:25 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:26.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:25 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:26.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:25 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:26.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:25 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:14:26.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:25 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:26.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:25 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:14:26.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:25 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-13T14:14:26.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:25 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:14:26.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:25 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:26.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:25 smithi151 ceph-mon[29807]: Deploying daemon mgr.x on smithi184 2024-09-13T14:14:26.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:25 smithi151 ceph-mon[41387]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi151=y;smithi184=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:26.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:25 smithi151 ceph-mon[41387]: Saving service mgr spec with placement smithi151=y;smithi184=x;count:2 2024-09-13T14:14:26.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:25 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:26.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:25 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:26.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:25 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:26.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:25 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:26.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:25 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:14:26.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:25 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:26.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:25 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:14:26.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:25 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-13T14:14:26.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:25 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:14:26.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:25 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:26.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:25 smithi151 ceph-mon[41387]: Deploying daemon mgr.x on smithi184 2024-09-13T14:14:26.118 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_2 2024-09-13T14:14:26.175 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T14:14:26.175 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:14:26.175 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 979 Links: 1 2024-09-13T14:14:26.175 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:14:26.175 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:14:26.175 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-09-13 14:14:02.930302410 +0000 2024-09-13T14:14:26.175 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-09-13 14:09:53.567427310 +0000 2024-09-13T14:14:26.175 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-09-13 14:09:53.567427310 +0000 2024-09-13T14:14:26.175 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-09-13 14:09:53.567427310 +0000 2024-09-13T14:14:26.176 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T14:14:26.239 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-09-13T14:14:26.239 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-09-13T14:14:26.239 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000201246 s, 2.5 MB/s 2024-09-13T14:14:26.240 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T14:14:26.298 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_3 2024-09-13T14:14:26.355 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T14:14:26.355 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:14:26.355 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 974 Links: 1 2024-09-13T14:14:26.355 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:14:26.355 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:14:26.355 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-09-13 14:14:02.931302389 +0000 2024-09-13T14:14:26.356 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-09-13 14:09:53.565427299 +0000 2024-09-13T14:14:26.356 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-09-13 14:09:53.565427299 +0000 2024-09-13T14:14:26.356 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-09-13 14:09:53.565427299 +0000 2024-09-13T14:14:26.356 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T14:14:26.501 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-09-13T14:14:26.501 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-09-13T14:14:26.502 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000204546 s, 2.5 MB/s 2024-09-13T14:14:26.503 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T14:14:26.518 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_4 2024-09-13T14:14:26.580 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T14:14:26.580 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:14:26.580 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 968 Links: 1 2024-09-13T14:14:26.580 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:14:26.580 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:14:26.580 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-09-13 14:14:26.457819337 +0000 2024-09-13T14:14:26.580 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-09-13 14:09:53.565427299 +0000 2024-09-13T14:14:26.580 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-09-13 14:09:53.565427299 +0000 2024-09-13T14:14:26.580 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-09-13 14:09:53.565427299 +0000 2024-09-13T14:14:26.581 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T14:14:26.686 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-09-13T14:14:26.686 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-09-13T14:14:26.687 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000200096 s, 2.6 MB/s 2024-09-13T14:14:26.688 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T14:14:26.703 INFO:tasks.cephadm:Deploying osd.0 on smithi151 with /dev/vg_nvme/lv_4... 2024-09-13T14:14:26.703 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-13T14:14:26.911 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:26 smithi184 ceph-mon[33394]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:26.946 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:14:27.012 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:26 smithi151 ceph-mon[29807]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:27.012 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:26 smithi151 ceph-mon[41387]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:27.188 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:14:27 smithi184 ceph-mgr[35471]: -- 172.21.15.184:0/3748732764 <== mon.1 v2:172.21.15.184:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5653ac44e340 con 0x5653ac45ac00 2024-09-13T14:14:27.516 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:14:27 smithi184 bash[35433]: 1d88b55dd4f147cbe2877ed056139118c951135dfb05c45ae72be3fec2f8374b 2024-09-13T14:14:27.516 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:14:27 smithi184 systemd[1]: Started Ceph mgr.x for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:14:27.516 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:14:27 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:14:27.255+0000 7fe615b41ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T14:14:27.516 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:14:27 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:14:27.314+0000 7fe615b41ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T14:14:27.516 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:14:27 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:14:27.383+0000 7fe615b41ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T14:14:27.516 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:14:27 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:14:27.514+0000 7fe615b41ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:14:27.853 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:14:27 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:14:27.678+0000 7fe615b41ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T14:14:28.209 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:14:27 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:14:27.950+0000 7fe615b41ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T14:14:28.209 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:14:28 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:14:28.009+0000 7fe615b41ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T14:14:28.209 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:14:28 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:14:28.181+0000 7fe615b41ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T14:14:28.545 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:28 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:28.545 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:28 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:28.545 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:28 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:28.545 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:28 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:28.545 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:28 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:28.545 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:28 smithi151 ceph-mon[29807]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:28.546 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:28 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:28.546 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:28 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:28.546 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:28 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:28.546 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:28 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:28.546 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:28 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:28.546 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:28 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:28.546 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:28 smithi151 ceph-mon[41387]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:28.546 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:28 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:28.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:28 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:28.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:28 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:28.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:28 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:28.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:28 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:28.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:28 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:28.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:28 smithi184 ceph-mon[33394]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:28.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:28 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:29.353 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:14:29 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:14:29.040+0000 7fe615b41ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T14:14:29.353 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:14:29 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:14:29.103+0000 7fe615b41ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T14:14:29.469 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:14:29.488 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi151:vg_nvme/lv_4 2024-09-13T14:14:29.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:29 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:29.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:29 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:29.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:29 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:29.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:29 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:14:29.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:29 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:29.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:29 smithi151 ceph-mon[29807]: Reconfiguring mgr.y (unknown last config time)... 2024-09-13T14:14:29.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:29 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:14:29.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:29 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:14:29.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:29 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:29.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:29 smithi151 ceph-mon[29807]: Reconfiguring daemon mgr.y on smithi151 2024-09-13T14:14:29.765 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:29 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:29.766 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:29 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:29.766 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:29 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:29.766 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:29 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:14:29.766 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:29 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:29.766 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:29 smithi151 ceph-mon[41387]: Reconfiguring mgr.y (unknown last config time)... 2024-09-13T14:14:29.766 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:29 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:14:29.766 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:29 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:14:29.766 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:29 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:29.766 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:29 smithi151 ceph-mon[41387]: Reconfiguring daemon mgr.y on smithi151 2024-09-13T14:14:29.815 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:29 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:29.815 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:29 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:29.816 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:29 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:29.816 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:29 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:14:29.816 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:29 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:29.816 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:29 smithi184 ceph-mon[33394]: Reconfiguring mgr.y (unknown last config time)... 2024-09-13T14:14:29.816 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:29 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:14:29.816 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:29 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:14:29.816 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:29 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:29.816 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:29 smithi184 ceph-mon[33394]: Reconfiguring daemon mgr.y on smithi151 2024-09-13T14:14:30.103 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:14:29 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:14:29.814+0000 7fe615b41ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T14:14:30.103 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:14:29 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:14:29.926+0000 7fe615b41ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T14:14:30.103 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:14:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:14:30.044+0000 7fe615b41ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:14:30.207 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:14:30.482 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:14:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:14:30.102+0000 7fe615b41ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T14:14:30.483 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:14:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:14:30.227+0000 7fe615b41ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T14:14:30.483 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:14:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:14:30.292+0000 7fe615b41ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T14:14:30.483 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:14:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:14:30.349+0000 7fe615b41ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T14:14:30.483 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:14:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:14:30.481+0000 7fe615b41ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T14:14:30.605 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:30 smithi151 ceph-mon[29807]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:30.605 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:30 smithi151 ceph-mon[41387]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:30.746 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:30 smithi184 ceph-mon[33394]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:30.746 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:14:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:14:30.544+0000 7fe615b41ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T14:14:30.746 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:14:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:14:30.609+0000 7fe615b41ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T14:14:30.746 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:14:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:14:30.688+0000 7fe615b41ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T14:14:30.746 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:14:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:14:30.744+0000 7fe615b41ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T14:14:31.103 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:14:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:14:30.801+0000 7fe615b41ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T14:14:31.853 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:14:31 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:14:31.538+0000 7fe615b41ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T14:14:32.081 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:31 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:32.081 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:31 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:32.081 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:31 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:32.081 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:31 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:32.081 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:31 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:14:32.081 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:31 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:32.081 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:31 smithi151 ceph-mon[41387]: Standby manager daemon x started 2024-09-13T14:14:32.081 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:31 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.184:0/1314959472' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:14:32.082 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:31 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.184:0/1314959472' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:14:32.082 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:31 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.184:0/1314959472' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:14:32.082 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:31 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.184:0/1314959472' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:14:32.082 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:31 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:32.082 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:31 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:32.082 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:31 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:32.082 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:31 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:32.082 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:31 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:14:32.082 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:31 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:32.082 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:31 smithi151 ceph-mon[29807]: Standby manager daemon x started 2024-09-13T14:14:32.082 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:31 smithi151 ceph-mon[29807]: from='mgr.? 172.21.15.184:0/1314959472' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:14:32.083 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:31 smithi151 ceph-mon[29807]: from='mgr.? 172.21.15.184:0/1314959472' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:14:32.083 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:31 smithi151 ceph-mon[29807]: from='mgr.? 172.21.15.184:0/1314959472' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:14:32.083 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:31 smithi151 ceph-mon[29807]: from='mgr.? 172.21.15.184:0/1314959472' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:14:32.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:31 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:32.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:31 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:32.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:31 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:31 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:31 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:14:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:31 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:31 smithi184 ceph-mon[33394]: Standby manager daemon x started 2024-09-13T14:14:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:31 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.184:0/1314959472' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:14:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:31 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.184:0/1314959472' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:14:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:31 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.184:0/1314959472' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:14:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:31 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.184:0/1314959472' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:14:33.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:32 smithi151 ceph-mon[29807]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:33.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:32 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:14:33.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:32 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:14:33.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:32 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:33.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:32 smithi151 ceph-mon[29807]: mgrmap e14: y(active, since 65s), standbys: x 2024-09-13T14:14:33.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:32 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:14:33.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:32 smithi151 ceph-mon[41387]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:33.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:32 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:14:33.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:32 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:14:33.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:32 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:33.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:32 smithi151 ceph-mon[41387]: mgrmap e14: y(active, since 65s), standbys: x 2024-09-13T14:14:33.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:32 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:14:33.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:32 smithi184 ceph-mon[33394]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:33.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:32 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:14:33.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:32 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:14:33.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:32 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:33.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:32 smithi184 ceph-mon[33394]: mgrmap e14: y(active, since 65s), standbys: x 2024-09-13T14:14:33.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:32 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:14:34.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:33 smithi151 ceph-mon[29807]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:34.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:33 smithi151 ceph-mon[41387]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:34.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:33 smithi184 ceph-mon[33394]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:35.321 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:34 smithi151 ceph-mon[29807]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:35.321 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:34 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/2359383102' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ad945ee-af7d-4f26-a1e9-18ed963194ad"}]: dispatch 2024-09-13T14:14:35.321 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:34 smithi151 ceph-mon[29807]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ad945ee-af7d-4f26-a1e9-18ed963194ad"}]: dispatch 2024-09-13T14:14:35.321 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:34 smithi151 ceph-mon[29807]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0ad945ee-af7d-4f26-a1e9-18ed963194ad"}]': finished 2024-09-13T14:14:35.321 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:34 smithi151 ceph-mon[29807]: osdmap e5: 1 total, 0 up, 1 in 2024-09-13T14:14:35.321 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:34 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:14:35.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:34 smithi151 ceph-mon[41387]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:35.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:34 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/2359383102' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ad945ee-af7d-4f26-a1e9-18ed963194ad"}]: dispatch 2024-09-13T14:14:35.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:34 smithi151 ceph-mon[41387]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ad945ee-af7d-4f26-a1e9-18ed963194ad"}]: dispatch 2024-09-13T14:14:35.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:34 smithi151 ceph-mon[41387]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0ad945ee-af7d-4f26-a1e9-18ed963194ad"}]': finished 2024-09-13T14:14:35.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:34 smithi151 ceph-mon[41387]: osdmap e5: 1 total, 0 up, 1 in 2024-09-13T14:14:35.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:34 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:14:35.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:34 smithi184 ceph-mon[33394]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:35.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:34 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/2359383102' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ad945ee-af7d-4f26-a1e9-18ed963194ad"}]: dispatch 2024-09-13T14:14:35.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:34 smithi184 ceph-mon[33394]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ad945ee-af7d-4f26-a1e9-18ed963194ad"}]: dispatch 2024-09-13T14:14:35.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:34 smithi184 ceph-mon[33394]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0ad945ee-af7d-4f26-a1e9-18ed963194ad"}]': finished 2024-09-13T14:14:35.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:34 smithi184 ceph-mon[33394]: osdmap e5: 1 total, 0 up, 1 in 2024-09-13T14:14:35.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:34 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:14:36.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:35 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/1310794125' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:14:36.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:35 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1310794125' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:14:36.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:35 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/1310794125' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:14:37.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:36 smithi184 ceph-mon[33394]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:37.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:36 smithi151 ceph-mon[29807]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:37.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:36 smithi151 ceph-mon[41387]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:39.335 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:38 smithi151 ceph-mon[29807]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:39.335 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:38 smithi151 ceph-mon[41387]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:39.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:38 smithi184 ceph-mon[33394]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:41.265 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:40 smithi151 ceph-mon[41387]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:41.265 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:40 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-13T14:14:41.265 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:40 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:41.265 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:40 smithi151 ceph-mon[29807]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:41.266 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:40 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-13T14:14:41.266 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:40 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:41.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:40 smithi184 ceph-mon[33394]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:41.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:40 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-13T14:14:41.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:40 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:42.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:41 smithi151 ceph-mon[29807]: Deploying daemon osd.0 on smithi151 2024-09-13T14:14:42.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:41 smithi151 ceph-mon[41387]: Deploying daemon osd.0 on smithi151 2024-09-13T14:14:42.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:41 smithi184 ceph-mon[33394]: Deploying daemon osd.0 on smithi151 2024-09-13T14:14:43.281 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:42 smithi151 ceph-mon[29807]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:43.281 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:42 smithi151 ceph-mon[41387]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:43.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:42 smithi184 ceph-mon[33394]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:45.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:44 smithi151 ceph-mon[29807]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:45.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:44 smithi151 ceph-mon[41387]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:45.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:44 smithi184 ceph-mon[33394]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:46.277 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:46 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:46.277 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:46 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:46.277 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:46 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:46.277 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:46 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:46.278 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:46 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:46.278 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:46 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:46.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:46 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:46.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:46 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:46.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:46 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:47.333 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:47 smithi151 ceph-mon[29807]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:47.333 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:47 smithi151 ceph-mon[41387]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:47.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:47 smithi184 ceph-mon[33394]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:48.999 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 0 on host 'smithi151' 2024-09-13T14:14:49.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:49 smithi184 ceph-mon[33394]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:49.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:49 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:49.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:49 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:49.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:49 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:49.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:49 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:14:49.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:49 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:49.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:49 smithi184 ceph-mon[33394]: from='osd.0 [v2:172.21.15.151:6802/551854295,v1:172.21.15.151:6803/551854295]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-13T14:14:49.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:49 smithi184 ceph-mon[33394]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-13T14:14:49.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:49 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:49.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:49 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:49.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:49 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:49.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:49 smithi151 ceph-mon[41387]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:49.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:49 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:49.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:49 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:49.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:49 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:49.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:49 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:14:49.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:49 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:49.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:49 smithi151 ceph-mon[41387]: from='osd.0 [v2:172.21.15.151:6802/551854295,v1:172.21.15.151:6803/551854295]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-13T14:14:49.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:49 smithi151 ceph-mon[41387]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-13T14:14:49.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:49 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:49.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:49 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:49.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:49 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:49.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:49 smithi151 ceph-mon[29807]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:49.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:49 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:49.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:49 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:49.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:49 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:49.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:49 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:14:49.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:49 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:49.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:49 smithi151 ceph-mon[29807]: from='osd.0 [v2:172.21.15.151:6802/551854295,v1:172.21.15.151:6803/551854295]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-13T14:14:49.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:49 smithi151 ceph-mon[29807]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-13T14:14:49.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:49 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:49.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:49 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:49.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:49 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:49.669 DEBUG:teuthology.orchestra.run.smithi151:osd.0> sudo journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.0.service 2024-09-13T14:14:49.671 INFO:tasks.cephadm:Deploying osd.1 on smithi151 with /dev/vg_nvme/lv_3... 2024-09-13T14:14:49.672 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-13T14:14:50.405 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:14:50.525 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:50 smithi151 ceph-mon[29807]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T14:14:50.525 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:50 smithi151 ceph-mon[29807]: osdmap e6: 1 total, 0 up, 1 in 2024-09-13T14:14:50.525 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:50 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:14:50.526 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:50 smithi151 ceph-mon[29807]: from='osd.0 [v2:172.21.15.151:6802/551854295,v1:172.21.15.151:6803/551854295]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-09-13T14:14:50.526 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:50 smithi151 ceph-mon[29807]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-09-13T14:14:50.526 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:50 smithi151 ceph-mon[29807]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:50.526 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:50 smithi151 ceph-mon[41387]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T14:14:50.526 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:50 smithi151 ceph-mon[41387]: osdmap e6: 1 total, 0 up, 1 in 2024-09-13T14:14:50.526 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:50 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:14:50.526 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:50 smithi151 ceph-mon[41387]: from='osd.0 [v2:172.21.15.151:6802/551854295,v1:172.21.15.151:6803/551854295]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-09-13T14:14:50.526 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:50 smithi151 ceph-mon[41387]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-09-13T14:14:50.526 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:50 smithi151 ceph-mon[41387]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:50.527 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:14:50 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0[45764]: 2024-09-13T14:14:50.272+0000 7f9066086640 -1 osd.0 0 waiting for initial osdmap 2024-09-13T14:14:50.527 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:14:50 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0[45764]: 2024-09-13T14:14:50.274+0000 7f9061ebf640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:14:50.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:50 smithi184 ceph-mon[33394]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T14:14:50.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:50 smithi184 ceph-mon[33394]: osdmap e6: 1 total, 0 up, 1 in 2024-09-13T14:14:50.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:50 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:14:50.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:50 smithi184 ceph-mon[33394]: from='osd.0 [v2:172.21.15.151:6802/551854295,v1:172.21.15.151:6803/551854295]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-09-13T14:14:50.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:50 smithi184 ceph-mon[33394]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-09-13T14:14:50.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:50 smithi184 ceph-mon[33394]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:51.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:51 smithi151 ceph-mon[29807]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-09-13T14:14:51.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:51 smithi151 ceph-mon[29807]: osdmap e7: 1 total, 0 up, 1 in 2024-09-13T14:14:51.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:51 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:14:51.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:51 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:14:51.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:51 smithi151 ceph-mon[41387]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-09-13T14:14:51.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:51 smithi151 ceph-mon[41387]: osdmap e7: 1 total, 0 up, 1 in 2024-09-13T14:14:51.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:51 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:14:51.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:51 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:14:51.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:51 smithi184 ceph-mon[33394]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-09-13T14:14:51.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:51 smithi184 ceph-mon[33394]: osdmap e7: 1 total, 0 up, 1 in 2024-09-13T14:14:51.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:51 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:14:51.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:51 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:14:52.354 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:52 smithi151 ceph-mon[41387]: purged_snaps scrub starts 2024-09-13T14:14:52.355 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:52 smithi151 ceph-mon[41387]: purged_snaps scrub ok 2024-09-13T14:14:52.355 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:52 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:14:52.355 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:52 smithi151 ceph-mon[41387]: osd.0 [v2:172.21.15.151:6802/551854295,v1:172.21.15.151:6803/551854295] boot 2024-09-13T14:14:52.355 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:52 smithi151 ceph-mon[41387]: osdmap e8: 1 total, 1 up, 1 in 2024-09-13T14:14:52.355 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:52 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:14:52.355 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:52 smithi151 ceph-mon[41387]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:52.356 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:52 smithi151 ceph-mon[29807]: purged_snaps scrub starts 2024-09-13T14:14:52.356 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:52 smithi151 ceph-mon[29807]: purged_snaps scrub ok 2024-09-13T14:14:52.356 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:52 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:14:52.356 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:52 smithi151 ceph-mon[29807]: osd.0 [v2:172.21.15.151:6802/551854295,v1:172.21.15.151:6803/551854295] boot 2024-09-13T14:14:52.356 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:52 smithi151 ceph-mon[29807]: osdmap e8: 1 total, 1 up, 1 in 2024-09-13T14:14:52.356 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:52 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:14:52.356 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:52 smithi151 ceph-mon[29807]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:52.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:52 smithi184 ceph-mon[33394]: purged_snaps scrub starts 2024-09-13T14:14:52.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:52 smithi184 ceph-mon[33394]: purged_snaps scrub ok 2024-09-13T14:14:52.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:52 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:14:52.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:52 smithi184 ceph-mon[33394]: osd.0 [v2:172.21.15.151:6802/551854295,v1:172.21.15.151:6803/551854295] boot 2024-09-13T14:14:52.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:52 smithi184 ceph-mon[33394]: osdmap e8: 1 total, 1 up, 1 in 2024-09-13T14:14:52.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:52 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:14:52.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:52 smithi184 ceph-mon[33394]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:14:53.466 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:53 smithi151 ceph-mon[29807]: osdmap e9: 1 total, 1 up, 1 in 2024-09-13T14:14:53.467 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:53 smithi151 ceph-mon[41387]: osdmap e9: 1 total, 1 up, 1 in 2024-09-13T14:14:53.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:53 smithi184 ceph-mon[33394]: osdmap e9: 1 total, 1 up, 1 in 2024-09-13T14:14:54.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:54 smithi151 ceph-mon[29807]: Detected new or changed devices on smithi151 2024-09-13T14:14:54.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:54 smithi151 ceph-mon[29807]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:14:54.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:54 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:54.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:54 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:54.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:54 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:54.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:54 smithi151 ceph-mon[29807]: Adjusting osd_memory_target on smithi151 to 16064M 2024-09-13T14:14:54.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:54 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:54.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:54 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:54.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:54 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:14:54.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:54 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:54.842 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:54 smithi151 ceph-mon[41387]: Detected new or changed devices on smithi151 2024-09-13T14:14:54.842 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:54 smithi151 ceph-mon[41387]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:14:54.842 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:54 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:54.842 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:54 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:54.842 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:54 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:54.842 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:54 smithi151 ceph-mon[41387]: Adjusting osd_memory_target on smithi151 to 16064M 2024-09-13T14:14:54.842 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:54 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:54.842 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:54 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:54.842 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:54 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:14:54.842 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:54 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:54.846 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:14:54.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:54 smithi184 ceph-mon[33394]: Detected new or changed devices on smithi151 2024-09-13T14:14:54.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:54 smithi184 ceph-mon[33394]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:14:54.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:54 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:54.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:54 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:54.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:54 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:54.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:54 smithi184 ceph-mon[33394]: Adjusting osd_memory_target on smithi151 to 16064M 2024-09-13T14:14:54.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:54 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:54.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:54 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:54.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:54 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:14:54.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:54 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:14:54.866 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi151:vg_nvme/lv_3 2024-09-13T14:14:55.107 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:14:56.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:56 smithi151 ceph-mon[41387]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:14:56.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:56 smithi151 ceph-mon[29807]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:14:56.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:56 smithi184 ceph-mon[33394]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:14:57.630 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:57 smithi151 ceph-mon[29807]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:57.630 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:57 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:14:57.630 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:57 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:14:57.630 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:57 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:57.630 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:57 smithi151 ceph-mon[41387]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:57.630 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:57 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:14:57.630 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:57 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:14:57.631 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:57 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:57.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:57 smithi184 ceph-mon[33394]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:57.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:57 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:14:57.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:57 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:14:57.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:57 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:58.761 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:58 smithi151 ceph-mon[29807]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:14:58.762 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:58 smithi151 ceph-mon[41387]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:14:58.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:58 smithi184 ceph-mon[33394]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:14:59.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:59 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/3844042559' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09f5d705-684f-4e86-80a2-ec2292af09d3"}]: dispatch 2024-09-13T14:14:59.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:59 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/3844042559' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09f5d705-684f-4e86-80a2-ec2292af09d3"}]': finished 2024-09-13T14:14:59.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:59 smithi151 ceph-mon[29807]: osdmap e10: 2 total, 1 up, 2 in 2024-09-13T14:14:59.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:14:59 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:14:59.765 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:59 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/3844042559' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09f5d705-684f-4e86-80a2-ec2292af09d3"}]: dispatch 2024-09-13T14:14:59.766 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:59 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/3844042559' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09f5d705-684f-4e86-80a2-ec2292af09d3"}]': finished 2024-09-13T14:14:59.766 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:59 smithi151 ceph-mon[41387]: osdmap e10: 2 total, 1 up, 2 in 2024-09-13T14:14:59.766 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:14:59 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:14:59.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:59 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/3844042559' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09f5d705-684f-4e86-80a2-ec2292af09d3"}]: dispatch 2024-09-13T14:14:59.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:59 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/3844042559' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09f5d705-684f-4e86-80a2-ec2292af09d3"}]': finished 2024-09-13T14:14:59.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:59 smithi184 ceph-mon[33394]: osdmap e10: 2 total, 1 up, 2 in 2024-09-13T14:14:59.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:14:59 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:15:00.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:00 smithi184 ceph-mon[33394]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:15:00.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:00 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/2200632326' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:15:00.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:00 smithi151 ceph-mon[41387]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:15:00.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:00 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/2200632326' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:15:00.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:00 smithi151 ceph-mon[29807]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:15:00.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:00 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/2200632326' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:15:02.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:02 smithi151 ceph-mon[29807]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:15:02.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:02 smithi151 ceph-mon[41387]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:15:02.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:02 smithi184 ceph-mon[33394]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:15:04.769 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:04 smithi151 ceph-mon[29807]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:15:04.770 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:04 smithi151 ceph-mon[41387]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:15:04.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:04 smithi184 ceph-mon[33394]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:15:05.820 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:05 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-13T14:15:05.820 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:05 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:05.820 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:05 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-13T14:15:05.821 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:05 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:05.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:05 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-13T14:15:05.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:05 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:06.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:06 smithi184 ceph-mon[33394]: Deploying daemon osd.1 on smithi151 2024-09-13T14:15:06.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:06 smithi184 ceph-mon[33394]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:15:06.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:06 smithi151 ceph-mon[29807]: Deploying daemon osd.1 on smithi151 2024-09-13T14:15:06.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:06 smithi151 ceph-mon[29807]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:15:06.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:06 smithi151 ceph-mon[41387]: Deploying daemon osd.1 on smithi151 2024-09-13T14:15:06.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:06 smithi151 ceph-mon[41387]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:15:08.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:08 smithi151 ceph-mon[29807]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:15:08.693 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:08 smithi151 ceph-mon[41387]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:15:08.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:08 smithi184 ceph-mon[33394]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:15:10.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:10 smithi184 ceph-mon[33394]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:15:10.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:10 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:10.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:10 smithi151 ceph-mon[29807]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:15:10.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:10 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:10.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:10 smithi151 ceph-mon[41387]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:15:10.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:10 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:11.814 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:11 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:11.814 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:11 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:11.815 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:11 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:11.815 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:11 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:11.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:11 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:11.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:11 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:12.676 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:12 smithi151 ceph-mon[29807]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:15:12.676 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:12 smithi151 ceph-mon[41387]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:15:12.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:12 smithi184 ceph-mon[33394]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:15:14.016 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:13 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:14.016 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:13 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:14.016 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:13 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:14.017 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:13 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:15:14.017 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:13 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:14.017 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:13 smithi151 ceph-mon[29807]: from='osd.1 [v2:172.21.15.151:6810/3674059846,v1:172.21.15.151:6811/3674059846]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-13T14:15:14.017 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:13 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:14.017 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:13 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:14.017 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:13 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:14.017 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:13 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:15:14.017 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:13 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:14.017 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:13 smithi151 ceph-mon[41387]: from='osd.1 [v2:172.21.15.151:6810/3674059846,v1:172.21.15.151:6811/3674059846]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-13T14:15:14.053 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 1 on host 'smithi151' 2024-09-13T14:15:14.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:13 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:14.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:13 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:14.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:13 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:14.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:13 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:15:14.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:13 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:14.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:13 smithi184 ceph-mon[33394]: from='osd.1 [v2:172.21.15.151:6810/3674059846,v1:172.21.15.151:6811/3674059846]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-13T14:15:14.609 DEBUG:teuthology.orchestra.run.smithi151:osd.1> sudo journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.1.service 2024-09-13T14:15:14.612 INFO:tasks.cephadm:Deploying osd.2 on smithi151 with /dev/vg_nvme/lv_2... 2024-09-13T14:15:14.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-13T14:15:14.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:14 smithi151 ceph-mon[41387]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:15:14.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:14 smithi151 ceph-mon[41387]: from='osd.1 [v2:172.21.15.151:6810/3674059846,v1:172.21.15.151:6811/3674059846]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T14:15:14.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:14 smithi151 ceph-mon[41387]: osdmap e11: 2 total, 1 up, 2 in 2024-09-13T14:15:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:14 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:15:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:14 smithi151 ceph-mon[41387]: from='osd.1 [v2:172.21.15.151:6810/3674059846,v1:172.21.15.151:6811/3674059846]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-09-13T14:15:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:14 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:14 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:14 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:14.859 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:15:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1[52113]: 2024-09-13T14:15:14.793+0000 7f4e8962e640 -1 osd.1 0 waiting for initial osdmap 2024-09-13T14:15:14.859 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:15:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1[52113]: 2024-09-13T14:15:14.796+0000 7f4e8243f640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:15:14.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:14 smithi151 ceph-mon[29807]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:15:14.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:14 smithi151 ceph-mon[29807]: from='osd.1 [v2:172.21.15.151:6810/3674059846,v1:172.21.15.151:6811/3674059846]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T14:15:14.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:14 smithi151 ceph-mon[29807]: osdmap e11: 2 total, 1 up, 2 in 2024-09-13T14:15:14.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:14 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:15:14.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:14 smithi151 ceph-mon[29807]: from='osd.1 [v2:172.21.15.151:6810/3674059846,v1:172.21.15.151:6811/3674059846]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-09-13T14:15:14.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:14 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:14.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:14 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:14.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:14 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:14 smithi184 ceph-mon[33394]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:15:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:14 smithi184 ceph-mon[33394]: from='osd.1 [v2:172.21.15.151:6810/3674059846,v1:172.21.15.151:6811/3674059846]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T14:15:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:14 smithi184 ceph-mon[33394]: osdmap e11: 2 total, 1 up, 2 in 2024-09-13T14:15:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:14 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:15:15.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:14 smithi184 ceph-mon[33394]: from='osd.1 [v2:172.21.15.151:6810/3674059846,v1:172.21.15.151:6811/3674059846]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-09-13T14:15:15.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:14 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:15.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:14 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:15.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:14 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:15.388 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:15:16.061 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:15 smithi151 ceph-mon[41387]: from='osd.1 [v2:172.21.15.151:6810/3674059846,v1:172.21.15.151:6811/3674059846]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-09-13T14:15:16.061 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:15 smithi151 ceph-mon[41387]: osdmap e12: 2 total, 1 up, 2 in 2024-09-13T14:15:16.061 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:15 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:15:16.061 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:15 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:15:16.062 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:15 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:15:16.062 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:15 smithi151 ceph-mon[29807]: from='osd.1 [v2:172.21.15.151:6810/3674059846,v1:172.21.15.151:6811/3674059846]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-09-13T14:15:16.062 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:15 smithi151 ceph-mon[29807]: osdmap e12: 2 total, 1 up, 2 in 2024-09-13T14:15:16.062 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:15 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:15:16.062 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:15 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:15:16.062 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:15 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:15:16.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:15 smithi184 ceph-mon[33394]: from='osd.1 [v2:172.21.15.151:6810/3674059846,v1:172.21.15.151:6811/3674059846]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-09-13T14:15:16.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:15 smithi184 ceph-mon[33394]: osdmap e12: 2 total, 1 up, 2 in 2024-09-13T14:15:16.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:15 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:15:16.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:15 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:15:16.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:15 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:15:17.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:16 smithi184 ceph-mon[33394]: purged_snaps scrub starts 2024-09-13T14:15:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:16 smithi184 ceph-mon[33394]: purged_snaps scrub ok 2024-09-13T14:15:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:16 smithi184 ceph-mon[33394]: pgmap v56: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:15:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:16 smithi184 ceph-mon[33394]: osd.1 [v2:172.21.15.151:6810/3674059846,v1:172.21.15.151:6811/3674059846] boot 2024-09-13T14:15:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:16 smithi184 ceph-mon[33394]: osdmap e13: 2 total, 2 up, 2 in 2024-09-13T14:15:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:16 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:15:17.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:16 smithi151 ceph-mon[29807]: purged_snaps scrub starts 2024-09-13T14:15:17.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:16 smithi151 ceph-mon[29807]: purged_snaps scrub ok 2024-09-13T14:15:17.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:16 smithi151 ceph-mon[29807]: pgmap v56: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:15:17.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:16 smithi151 ceph-mon[29807]: osd.1 [v2:172.21.15.151:6810/3674059846,v1:172.21.15.151:6811/3674059846] boot 2024-09-13T14:15:17.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:16 smithi151 ceph-mon[29807]: osdmap e13: 2 total, 2 up, 2 in 2024-09-13T14:15:17.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:16 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:15:17.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:16 smithi151 ceph-mon[41387]: purged_snaps scrub starts 2024-09-13T14:15:17.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:16 smithi151 ceph-mon[41387]: purged_snaps scrub ok 2024-09-13T14:15:17.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:16 smithi151 ceph-mon[41387]: pgmap v56: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:15:17.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:16 smithi151 ceph-mon[41387]: osd.1 [v2:172.21.15.151:6810/3674059846,v1:172.21.15.151:6811/3674059846] boot 2024-09-13T14:15:17.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:16 smithi151 ceph-mon[41387]: osdmap e13: 2 total, 2 up, 2 in 2024-09-13T14:15:17.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:16 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:15:18.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:17 smithi184 ceph-mon[33394]: osdmap e14: 2 total, 2 up, 2 in 2024-09-13T14:15:18.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:17 smithi151 ceph-mon[29807]: osdmap e14: 2 total, 2 up, 2 in 2024-09-13T14:15:18.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:17 smithi151 ceph-mon[41387]: osdmap e14: 2 total, 2 up, 2 in 2024-09-13T14:15:19.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:18 smithi184 ceph-mon[33394]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:19.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:18 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:19.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:18 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:19.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:18 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:15:19.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:18 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:15:19.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:18 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:19.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:18 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:19.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:18 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:15:19.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:18 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:19.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:18 smithi151 ceph-mon[41387]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:19.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:18 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:19.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:18 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:19.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:18 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:15:19.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:18 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:15:19.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:18 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:19.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:18 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:19.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:18 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:15:19.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:18 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:19.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:18 smithi151 ceph-mon[29807]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:19.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:18 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:19.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:18 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:19.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:18 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:15:19.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:18 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:15:19.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:18 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:19.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:18 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:19.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:18 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:15:19.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:18 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:19.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:19 smithi151 ceph-mon[29807]: Detected new or changed devices on smithi151 2024-09-13T14:15:19.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:19 smithi151 ceph-mon[29807]: Adjusting osd_memory_target on smithi151 to 8032M 2024-09-13T14:15:19.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:19 smithi151 ceph-mon[41387]: Detected new or changed devices on smithi151 2024-09-13T14:15:19.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:19 smithi151 ceph-mon[41387]: Adjusting osd_memory_target on smithi151 to 8032M 2024-09-13T14:15:19.868 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:15:19.889 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi151:vg_nvme/lv_2 2024-09-13T14:15:20.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:19 smithi184 ceph-mon[33394]: Detected new or changed devices on smithi151 2024-09-13T14:15:20.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:19 smithi184 ceph-mon[33394]: Adjusting osd_memory_target on smithi151 to 8032M 2024-09-13T14:15:20.131 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:15:21.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:20 smithi184 ceph-mon[33394]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:21.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:20 smithi151 ceph-mon[41387]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:21.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:20 smithi151 ceph-mon[29807]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:22.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:21 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:15:22.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:21 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:15:22.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:21 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:22.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:21 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:15:22.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:21 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:15:22.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:21 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:22.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:21 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:15:22.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:21 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:15:22.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:21 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:23.070 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:22 smithi151 ceph-mon[29807]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:23.070 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:22 smithi151 ceph-mon[29807]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:15:23.070 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:22 smithi151 ceph-mon[41387]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:23.070 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:22 smithi151 ceph-mon[41387]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:15:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:22 smithi184 ceph-mon[33394]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:22 smithi184 ceph-mon[33394]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:15:25.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:24 smithi184 ceph-mon[33394]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:25.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:24 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/529847578' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6a63f8af-ff69-46bb-830f-03ea89fc78e3"}]: dispatch 2024-09-13T14:15:25.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:24 smithi184 ceph-mon[33394]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6a63f8af-ff69-46bb-830f-03ea89fc78e3"}]: dispatch 2024-09-13T14:15:25.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:24 smithi184 ceph-mon[33394]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6a63f8af-ff69-46bb-830f-03ea89fc78e3"}]': finished 2024-09-13T14:15:25.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:24 smithi184 ceph-mon[33394]: osdmap e15: 3 total, 2 up, 3 in 2024-09-13T14:15:25.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:24 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:15:25.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:24 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/762135394' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:15:25.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:24 smithi151 ceph-mon[41387]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:25.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:24 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/529847578' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6a63f8af-ff69-46bb-830f-03ea89fc78e3"}]: dispatch 2024-09-13T14:15:25.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:24 smithi151 ceph-mon[41387]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6a63f8af-ff69-46bb-830f-03ea89fc78e3"}]: dispatch 2024-09-13T14:15:25.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:24 smithi151 ceph-mon[41387]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6a63f8af-ff69-46bb-830f-03ea89fc78e3"}]': finished 2024-09-13T14:15:25.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:24 smithi151 ceph-mon[41387]: osdmap e15: 3 total, 2 up, 3 in 2024-09-13T14:15:25.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:24 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:15:25.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:24 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/762135394' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:15:25.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:24 smithi151 ceph-mon[29807]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:25.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:24 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/529847578' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6a63f8af-ff69-46bb-830f-03ea89fc78e3"}]: dispatch 2024-09-13T14:15:25.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:24 smithi151 ceph-mon[29807]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6a63f8af-ff69-46bb-830f-03ea89fc78e3"}]: dispatch 2024-09-13T14:15:25.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:24 smithi151 ceph-mon[29807]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6a63f8af-ff69-46bb-830f-03ea89fc78e3"}]': finished 2024-09-13T14:15:25.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:24 smithi151 ceph-mon[29807]: osdmap e15: 3 total, 2 up, 3 in 2024-09-13T14:15:25.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:24 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:15:25.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:24 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/762135394' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:15:27.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:26 smithi184 ceph-mon[33394]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:27.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:26 smithi151 ceph-mon[29807]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:27.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:26 smithi151 ceph-mon[41387]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:29.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:28 smithi184 ceph-mon[33394]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:29.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:28 smithi151 ceph-mon[29807]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:29.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:28 smithi151 ceph-mon[41387]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:31.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:30 smithi184 ceph-mon[33394]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:31.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:30 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-13T14:15:31.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:30 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:31.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:30 smithi184 ceph-mon[33394]: Deploying daemon osd.2 on smithi151 2024-09-13T14:15:31.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:30 smithi151 ceph-mon[29807]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:31.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:30 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-13T14:15:31.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:30 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:31.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:30 smithi151 ceph-mon[29807]: Deploying daemon osd.2 on smithi151 2024-09-13T14:15:31.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:30 smithi151 ceph-mon[41387]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:31.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:30 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-13T14:15:31.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:30 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:31.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:30 smithi151 ceph-mon[41387]: Deploying daemon osd.2 on smithi151 2024-09-13T14:15:33.029 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:32 smithi151 ceph-mon[41387]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:33.029 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:32 smithi151 ceph-mon[29807]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:33.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:32 smithi184 ceph-mon[33394]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:35.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:34 smithi151 ceph-mon[29807]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:35.055 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:34 smithi151 ceph-mon[41387]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:35.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:34 smithi184 ceph-mon[33394]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:35.976 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:35 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:35.976 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:35 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:35.977 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:35 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:35.977 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:35 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:35.977 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:35 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:35.977 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:35 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:36.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:35 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:36.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:35 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:36.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:35 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:37.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:36 smithi184 ceph-mon[33394]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:37.107 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:36 smithi151 ceph-mon[29807]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:37.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:36 smithi151 ceph-mon[41387]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:38.791 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 2 on host 'smithi151' 2024-09-13T14:15:39.331 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:39 smithi151 ceph-mon[29807]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:39.332 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:39 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:39.332 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:39 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:39.332 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:39 smithi151 ceph-mon[29807]: from='osd.2 [v2:172.21.15.151:6818/1646754940,v1:172.21.15.151:6819/1646754940]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-13T14:15:39.332 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:39 smithi151 ceph-mon[29807]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-13T14:15:39.332 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:39 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:39.332 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:39 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:15:39.332 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:39 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:39.332 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:39 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:39.332 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:39 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:39.332 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:39 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:39.333 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:39 smithi151 ceph-mon[41387]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:39.333 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:39 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:39.333 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:39 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:39.333 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:39 smithi151 ceph-mon[41387]: from='osd.2 [v2:172.21.15.151:6818/1646754940,v1:172.21.15.151:6819/1646754940]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-13T14:15:39.333 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:39 smithi151 ceph-mon[41387]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-13T14:15:39.333 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:39 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:39.333 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:39 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:15:39.333 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:39 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:39.333 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:39 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:39.333 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:39 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:39.333 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:39 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:39.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:39 smithi184 ceph-mon[33394]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:39.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:39 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:39.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:39 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:39.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:39 smithi184 ceph-mon[33394]: from='osd.2 [v2:172.21.15.151:6818/1646754940,v1:172.21.15.151:6819/1646754940]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-13T14:15:39.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:39 smithi184 ceph-mon[33394]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-13T14:15:39.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:39 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:39.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:39 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:15:39.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:39 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:39.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:39 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:39.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:39 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:39.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:39 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:39.412 DEBUG:teuthology.orchestra.run.smithi151:osd.2> sudo journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.2.service 2024-09-13T14:15:39.415 INFO:tasks.cephadm:Deploying osd.3 on smithi151 with /dev/vg_nvme/lv_1... 2024-09-13T14:15:39.415 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-13T14:15:40.188 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:15:40.308 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:40 smithi151 ceph-mon[29807]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T14:15:40.309 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:40 smithi151 ceph-mon[29807]: osdmap e16: 3 total, 2 up, 3 in 2024-09-13T14:15:40.309 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:40 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:15:40.309 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:40 smithi151 ceph-mon[29807]: from='osd.2 [v2:172.21.15.151:6818/1646754940,v1:172.21.15.151:6819/1646754940]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-09-13T14:15:40.309 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:40 smithi151 ceph-mon[29807]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-09-13T14:15:40.309 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:40 smithi151 ceph-mon[41387]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T14:15:40.309 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:40 smithi151 ceph-mon[41387]: osdmap e16: 3 total, 2 up, 3 in 2024-09-13T14:15:40.309 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:40 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:15:40.309 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:40 smithi151 ceph-mon[41387]: from='osd.2 [v2:172.21.15.151:6818/1646754940,v1:172.21.15.151:6819/1646754940]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-09-13T14:15:40.309 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:40 smithi151 ceph-mon[41387]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-09-13T14:15:40.310 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:15:40 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2[58488]: 2024-09-13T14:15:40.063+0000 7fa98856f640 -1 osd.2 0 waiting for initial osdmap 2024-09-13T14:15:40.310 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:15:40 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2[58488]: 2024-09-13T14:15:40.066+0000 7fa982382640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:15:40.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:40 smithi184 ceph-mon[33394]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T14:15:40.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:40 smithi184 ceph-mon[33394]: osdmap e16: 3 total, 2 up, 3 in 2024-09-13T14:15:40.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:40 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:15:40.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:40 smithi184 ceph-mon[33394]: from='osd.2 [v2:172.21.15.151:6818/1646754940,v1:172.21.15.151:6819/1646754940]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-09-13T14:15:40.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:40 smithi184 ceph-mon[33394]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-09-13T14:15:41.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:41 smithi184 ceph-mon[33394]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:41.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:41 smithi184 ceph-mon[33394]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-09-13T14:15:41.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:41 smithi184 ceph-mon[33394]: osdmap e17: 3 total, 2 up, 3 in 2024-09-13T14:15:41.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:41 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:15:41.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:41 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:15:41.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:41 smithi151 ceph-mon[29807]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:41.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:41 smithi151 ceph-mon[29807]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-09-13T14:15:41.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:41 smithi151 ceph-mon[29807]: osdmap e17: 3 total, 2 up, 3 in 2024-09-13T14:15:41.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:41 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:15:41.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:41 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:15:41.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:41 smithi151 ceph-mon[41387]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:41.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:41 smithi151 ceph-mon[41387]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-09-13T14:15:41.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:41 smithi151 ceph-mon[41387]: osdmap e17: 3 total, 2 up, 3 in 2024-09-13T14:15:41.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:41 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:15:41.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:41 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:15:42.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:42 smithi184 ceph-mon[33394]: purged_snaps scrub starts 2024-09-13T14:15:42.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:42 smithi184 ceph-mon[33394]: purged_snaps scrub ok 2024-09-13T14:15:42.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:42 smithi184 ceph-mon[33394]: osd.2 [v2:172.21.15.151:6818/1646754940,v1:172.21.15.151:6819/1646754940] boot 2024-09-13T14:15:42.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:42 smithi184 ceph-mon[33394]: osdmap e18: 3 total, 3 up, 3 in 2024-09-13T14:15:42.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:42 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:15:42.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:42 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-13T14:15:42.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:42 smithi151 ceph-mon[41387]: purged_snaps scrub starts 2024-09-13T14:15:42.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:42 smithi151 ceph-mon[41387]: purged_snaps scrub ok 2024-09-13T14:15:42.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:42 smithi151 ceph-mon[41387]: osd.2 [v2:172.21.15.151:6818/1646754940,v1:172.21.15.151:6819/1646754940] boot 2024-09-13T14:15:42.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:42 smithi151 ceph-mon[41387]: osdmap e18: 3 total, 3 up, 3 in 2024-09-13T14:15:42.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:42 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:15:42.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:42 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-13T14:15:42.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:42 smithi151 ceph-mon[29807]: purged_snaps scrub starts 2024-09-13T14:15:42.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:42 smithi151 ceph-mon[29807]: purged_snaps scrub ok 2024-09-13T14:15:42.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:42 smithi151 ceph-mon[29807]: osd.2 [v2:172.21.15.151:6818/1646754940,v1:172.21.15.151:6819/1646754940] boot 2024-09-13T14:15:42.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:42 smithi151 ceph-mon[29807]: osdmap e18: 3 total, 3 up, 3 in 2024-09-13T14:15:42.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:42 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:15:42.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:42 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-13T14:15:43.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:43 smithi151 ceph-mon[29807]: pgmap v75: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:43.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:43 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-13T14:15:43.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:43 smithi151 ceph-mon[29807]: osdmap e19: 3 total, 3 up, 3 in 2024-09-13T14:15:43.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:43 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T14:15:43.118 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:43 smithi151 ceph-mon[41387]: pgmap v75: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:43.118 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:43 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-13T14:15:43.118 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:43 smithi151 ceph-mon[41387]: osdmap e19: 3 total, 3 up, 3 in 2024-09-13T14:15:43.118 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:43 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T14:15:43.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:43 smithi184 ceph-mon[33394]: pgmap v75: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-13T14:15:43.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:43 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-13T14:15:43.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:43 smithi184 ceph-mon[33394]: osdmap e19: 3 total, 3 up, 3 in 2024-09-13T14:15:43.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:43 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T14:15:44.108 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:15:44 smithi151 sudo[62502]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T14:15:44.108 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:15:44 smithi151 sudo[62502]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:15:44.108 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:15:44 smithi151 sudo[62502]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:15:44.109 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:15:44 smithi151 sudo[62502]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:15:44.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:44 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:15:44.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:44 smithi151 ceph-mon[29807]: osdmap e20: 3 total, 3 up, 3 in 2024-09-13T14:15:44.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:44 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:44.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:44 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:44.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:44 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:15:44.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:44 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:15:44.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:44 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:15:44.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:44 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:44.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:44 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:44.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:44 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:15:44.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:44 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:44.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:44 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:15:44.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:44 smithi151 ceph-mon[41387]: osdmap e20: 3 total, 3 up, 3 in 2024-09-13T14:15:44.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:44 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:44.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:44 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:44.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:44 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:15:44.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:44 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:15:44.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:44 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:15:44.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:44 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:44.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:44 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:44.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:44 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:15:44.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:44 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:44.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:44 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:15:44.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:44 smithi184 ceph-mon[33394]: osdmap e20: 3 total, 3 up, 3 in 2024-09-13T14:15:44.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:44 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:44.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:44 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:44.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:44 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:15:44.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:44 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:15:44.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:44 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:15:44.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:44 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:44.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:44 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:44.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:44 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:15:44.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:44 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:44.558 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:15:44 smithi151 sudo[62511]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T14:15:44.558 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:15:44 smithi151 sudo[62511]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:15:44.558 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:15:44 smithi151 sudo[62511]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:15:44.558 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:15:44 smithi151 sudo[62511]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:15:44.593 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:15:44.613 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi151:vg_nvme/lv_1 2024-09-13T14:15:44.827 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:15:44 smithi151 sudo[62516]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T14:15:44.827 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:15:44 smithi151 sudo[62516]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:15:44.827 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:15:44 smithi151 sudo[62516]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:15:44.827 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:15:44 smithi151 sudo[62516]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:15:44.865 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:15:45.082 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:15:44 smithi151 sudo[62561]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T14:15:45.082 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:15:44 smithi151 sudo[62561]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:15:45.082 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:15:44 smithi151 sudo[62561]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:15:45.082 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:15:44 smithi151 sudo[62561]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:15:45.349 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:15:45 smithi151 sudo[62594]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T14:15:45.349 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:15:45 smithi151 sudo[62594]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:15:45.349 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:15:45 smithi151 sudo[62594]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:15:45.349 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:15:45 smithi151 sudo[62594]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:15:45.349 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:45 smithi151 ceph-mon[29807]: Detected new or changed devices on smithi151 2024-09-13T14:15:45.350 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:45 smithi151 ceph-mon[29807]: Adjusting osd_memory_target on smithi151 to 5354M 2024-09-13T14:15:45.350 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:45 smithi151 ceph-mon[29807]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:15:45.350 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:45 smithi151 ceph-mon[29807]: osdmap e21: 3 total, 3 up, 3 in 2024-09-13T14:15:45.350 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:45 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:45.350 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:45 smithi151 ceph-mon[41387]: Detected new or changed devices on smithi151 2024-09-13T14:15:45.350 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:45 smithi151 ceph-mon[41387]: Adjusting osd_memory_target on smithi151 to 5354M 2024-09-13T14:15:45.350 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:45 smithi151 ceph-mon[41387]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:15:45.350 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:45 smithi151 ceph-mon[41387]: osdmap e21: 3 total, 3 up, 3 in 2024-09-13T14:15:45.350 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:45 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:45.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:45 smithi184 ceph-mon[33394]: Detected new or changed devices on smithi151 2024-09-13T14:15:45.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:45 smithi184 ceph-mon[33394]: Adjusting osd_memory_target on smithi151 to 5354M 2024-09-13T14:15:45.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:45 smithi184 ceph-mon[33394]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:15:45.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:45 smithi184 ceph-mon[33394]: osdmap e21: 3 total, 3 up, 3 in 2024-09-13T14:15:45.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:45 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:45.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:45 smithi151 sudo[62638]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T14:15:45.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:45 smithi151 sudo[62638]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:15:45.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:45 smithi151 sudo[62638]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:15:45.608 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:15:45 smithi151 sudo[62635]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T14:15:45.608 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:15:45 smithi151 sudo[62635]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:15:45.609 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:15:45 smithi151 sudo[62635]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:15:45.609 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:15:45 smithi151 sudo[62635]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:15:46.006 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:45 smithi151 sudo[62638]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:15:46.006 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:45 smithi151 sudo[62647]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T14:15:46.006 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:45 smithi151 sudo[62647]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:15:46.006 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:45 smithi151 sudo[62647]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:15:46.006 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:45 smithi151 sudo[62647]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:15:46.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:46 smithi184 sudo[36122]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T14:15:46.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:46 smithi184 sudo[36122]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:15:46.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:46 smithi184 sudo[36122]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:15:46.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:46 smithi184 sudo[36122]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:15:46.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:46 smithi151 ceph-mon[41387]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:15:46.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:46 smithi151 ceph-mon[41387]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:15:46.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:46 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:15:46.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:46 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:15:46.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:46 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:15:46.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:46 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:15:46.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:46 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:15:46.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:46 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:15:46.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:46 smithi151 ceph-mon[29807]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:15:46.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:46 smithi151 ceph-mon[29807]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:15:46.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:46 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:15:46.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:46 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:15:46.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:46 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:15:46.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:46 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:15:46.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:46 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:15:46.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:46 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:15:46.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:46 smithi184 sudo[36125]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T14:15:46.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:46 smithi184 sudo[36125]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:15:46.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:46 smithi184 sudo[36125]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:15:46.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:46 smithi184 sudo[36125]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:15:46.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:46 smithi184 ceph-mon[33394]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:15:46.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:46 smithi184 ceph-mon[33394]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:15:46.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:46 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:15:46.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:46 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:15:46.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:46 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:15:46.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:46 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:15:46.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:46 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:15:46.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:46 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:15:46.891 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:46 smithi151 sudo[62717]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T14:15:46.891 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:46 smithi151 sudo[62717]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:15:46.891 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:46 smithi151 sudo[62717]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:15:46.891 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:46 smithi151 sudo[62717]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:15:47.214 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:46 smithi151 sudo[62787]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T14:15:47.214 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:46 smithi151 sudo[62787]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:15:47.214 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:46 smithi151 sudo[62787]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:15:47.214 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:46 smithi151 sudo[62787]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:15:47.482 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:47 smithi151 ceph-mon[41387]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:15:47.482 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:47 smithi151 ceph-mon[41387]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:15:47.482 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:47 smithi151 ceph-mon[41387]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:15:47.482 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:47 smithi151 ceph-mon[41387]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-13T14:15:47.482 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:47 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:47.482 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:47 smithi151 ceph-mon[41387]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:15:47.482 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:47 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:15:47.483 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:47 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:15:47.483 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:47 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:15:47.483 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:47 smithi151 ceph-mon[41387]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:15:47.483 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:47 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:15:47.483 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:47 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:15:47.483 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:47 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:47.483 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:47 smithi151 ceph-mon[41387]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:15:47.483 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:47 smithi151 ceph-mon[29807]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:15:47.483 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:47 smithi151 ceph-mon[29807]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:15:47.483 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:47 smithi151 ceph-mon[29807]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:15:47.483 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:47 smithi151 ceph-mon[29807]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-13T14:15:47.483 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:47 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:47.484 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:47 smithi151 ceph-mon[29807]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:15:47.484 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:47 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:15:47.484 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:47 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:15:47.484 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:47 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:15:47.484 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:47 smithi151 ceph-mon[29807]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:15:47.484 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:47 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:15:47.484 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:47 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:15:47.484 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:47 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:47.484 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:47 smithi151 ceph-mon[29807]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:15:47.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:47 smithi184 ceph-mon[33394]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:15:47.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:47 smithi184 ceph-mon[33394]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:15:47.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:47 smithi184 ceph-mon[33394]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:15:47.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:47 smithi184 ceph-mon[33394]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-13T14:15:47.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:47 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:15:47.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:47 smithi184 ceph-mon[33394]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:15:47.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:47 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:15:47.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:47 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:15:47.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:47 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:15:47.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:47 smithi184 ceph-mon[33394]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:15:47.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:47 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:15:47.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:47 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:15:47.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:47 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:47.856 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:47 smithi184 ceph-mon[33394]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:15:48.402 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:48 smithi151 ceph-mon[29807]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 875 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:15:48.402 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:48 smithi151 ceph-mon[41387]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 875 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:15:48.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:48 smithi184 ceph-mon[33394]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 875 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:15:49.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:49 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/876576345' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "770e8334-3b3a-46bc-9782-964b2a696660"}]: dispatch 2024-09-13T14:15:49.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:49 smithi151 ceph-mon[29807]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "770e8334-3b3a-46bc-9782-964b2a696660"}]: dispatch 2024-09-13T14:15:49.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:49 smithi151 ceph-mon[29807]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "770e8334-3b3a-46bc-9782-964b2a696660"}]': finished 2024-09-13T14:15:49.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:49 smithi151 ceph-mon[29807]: osdmap e22: 4 total, 3 up, 4 in 2024-09-13T14:15:49.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:49 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:15:49.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:49 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/876576345' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "770e8334-3b3a-46bc-9782-964b2a696660"}]: dispatch 2024-09-13T14:15:49.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:49 smithi151 ceph-mon[41387]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "770e8334-3b3a-46bc-9782-964b2a696660"}]: dispatch 2024-09-13T14:15:49.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:49 smithi151 ceph-mon[41387]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "770e8334-3b3a-46bc-9782-964b2a696660"}]': finished 2024-09-13T14:15:49.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:49 smithi151 ceph-mon[41387]: osdmap e22: 4 total, 3 up, 4 in 2024-09-13T14:15:49.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:49 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:15:49.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:49 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/876576345' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "770e8334-3b3a-46bc-9782-964b2a696660"}]: dispatch 2024-09-13T14:15:49.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:49 smithi184 ceph-mon[33394]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "770e8334-3b3a-46bc-9782-964b2a696660"}]: dispatch 2024-09-13T14:15:49.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:49 smithi184 ceph-mon[33394]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "770e8334-3b3a-46bc-9782-964b2a696660"}]': finished 2024-09-13T14:15:49.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:49 smithi184 ceph-mon[33394]: osdmap e22: 4 total, 3 up, 4 in 2024-09-13T14:15:49.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:49 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:15:50.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:50 smithi184 ceph-mon[33394]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 875 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:15:50.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:50 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/340527275' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:15:50.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:50 smithi151 ceph-mon[29807]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 875 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:15:50.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:50 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/340527275' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:15:50.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:50 smithi151 ceph-mon[41387]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 875 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:15:50.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:50 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/340527275' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:15:52.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:52 smithi184 ceph-mon[33394]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 875 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:15:52.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:52 smithi151 ceph-mon[29807]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 875 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:15:52.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:52 smithi151 ceph-mon[41387]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 875 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:15:54.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:54 smithi184 ceph-mon[33394]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-09-13T14:15:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:54 smithi151 ceph-mon[29807]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-09-13T14:15:54.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:54 smithi151 ceph-mon[41387]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-09-13T14:15:55.817 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:55 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-13T14:15:55.817 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:55 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:55.817 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:55 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-13T14:15:55.817 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:55 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:55.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:55 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-13T14:15:55.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:55 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:56.741 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:56 smithi151 ceph-mon[29807]: Deploying daemon osd.3 on smithi151 2024-09-13T14:15:56.741 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:56 smithi151 ceph-mon[29807]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-09-13T14:15:56.741 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:56 smithi151 ceph-mon[41387]: Deploying daemon osd.3 on smithi151 2024-09-13T14:15:56.742 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:56 smithi151 ceph-mon[41387]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-09-13T14:15:56.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:56 smithi184 ceph-mon[33394]: Deploying daemon osd.3 on smithi151 2024-09-13T14:15:56.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:56 smithi184 ceph-mon[33394]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-09-13T14:15:58.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:15:58 smithi151 ceph-mon[29807]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-09-13T14:15:58.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:15:58 smithi151 ceph-mon[41387]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-09-13T14:15:58.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:15:58 smithi184 ceph-mon[33394]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-09-13T14:16:00.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:00 smithi151 ceph-mon[29807]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 885 B/s rd, 25 KiB/s wr, 2 op/s 2024-09-13T14:16:00.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:00 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:00.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:00 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:00.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:00 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:00.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:00 smithi151 ceph-mon[41387]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 885 B/s rd, 25 KiB/s wr, 2 op/s 2024-09-13T14:16:00.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:00 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:00.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:00 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:00.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:00 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:00.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:00 smithi184 ceph-mon[33394]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 885 B/s rd, 25 KiB/s wr, 2 op/s 2024-09-13T14:16:00.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:00 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:00.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:00 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:00.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:00 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:02.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:02 smithi184 ceph-mon[33394]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-13T14:16:02.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:02 smithi151 ceph-mon[29807]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-13T14:16:02.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:02 smithi151 ceph-mon[41387]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-13T14:16:03.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:03 smithi151 ceph-mon[29807]: from='osd.3 [v2:172.21.15.151:6826/2203980888,v1:172.21.15.151:6827/2203980888]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-13T14:16:03.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:03 smithi151 ceph-mon[29807]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-13T14:16:03.689 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:03 smithi151 ceph-mon[41387]: from='osd.3 [v2:172.21.15.151:6826/2203980888,v1:172.21.15.151:6827/2203980888]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-13T14:16:03.689 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:03 smithi151 ceph-mon[41387]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-13T14:16:03.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:03 smithi184 ceph-mon[33394]: from='osd.3 [v2:172.21.15.151:6826/2203980888,v1:172.21.15.151:6827/2203980888]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-13T14:16:03.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:03 smithi184 ceph-mon[33394]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-13T14:16:04.063 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 3 on host 'smithi151' 2024-09-13T14:16:04.729 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:04 smithi151 ceph-mon[29807]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-13T14:16:04.729 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:04 smithi151 ceph-mon[29807]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T14:16:04.729 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:04 smithi151 ceph-mon[29807]: osdmap e23: 4 total, 3 up, 4 in 2024-09-13T14:16:04.729 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:04 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:16:04.729 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:04 smithi151 ceph-mon[29807]: from='osd.3 [v2:172.21.15.151:6826/2203980888,v1:172.21.15.151:6827/2203980888]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-09-13T14:16:04.729 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:04 smithi151 ceph-mon[29807]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-09-13T14:16:04.730 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:04 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:04.730 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:04 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:04.730 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:04 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:04.730 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:04 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:04.730 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:04 smithi151 ceph-mon[41387]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-13T14:16:04.730 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:04 smithi151 ceph-mon[41387]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T14:16:04.730 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:04 smithi151 ceph-mon[41387]: osdmap e23: 4 total, 3 up, 4 in 2024-09-13T14:16:04.730 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:04 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:16:04.730 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:04 smithi151 ceph-mon[41387]: from='osd.3 [v2:172.21.15.151:6826/2203980888,v1:172.21.15.151:6827/2203980888]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-09-13T14:16:04.730 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:04 smithi151 ceph-mon[41387]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-09-13T14:16:04.731 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:04 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:04.731 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:04 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:04.731 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:04 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:04.731 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:04 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:04.752 DEBUG:teuthology.orchestra.run.smithi151:osd.3> sudo journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.3.service 2024-09-13T14:16:04.755 INFO:tasks.cephadm:Deploying osd.4 on smithi184 with /dev/vg_nvme/lv_4... 2024-09-13T14:16:04.755 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-13T14:16:04.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:04 smithi184 ceph-mon[33394]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-13T14:16:04.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:04 smithi184 ceph-mon[33394]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T14:16:04.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:04 smithi184 ceph-mon[33394]: osdmap e23: 4 total, 3 up, 4 in 2024-09-13T14:16:04.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:04 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:16:04.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:04 smithi184 ceph-mon[33394]: from='osd.3 [v2:172.21.15.151:6826/2203980888,v1:172.21.15.151:6827/2203980888]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-09-13T14:16:04.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:04 smithi184 ceph-mon[33394]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-09-13T14:16:04.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:04 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:04.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:04 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:04.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:04 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:04.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:04 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:05.002 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:16:05.841 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:05 smithi184 ceph-mon[33394]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-09-13T14:16:05.841 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:05 smithi184 ceph-mon[33394]: osdmap e24: 4 total, 3 up, 4 in 2024-09-13T14:16:05.841 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:05 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:16:05.841 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:05 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:16:05.841 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:05 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:16:05.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:05 smithi151 ceph-mon[41387]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-09-13T14:16:05.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:05 smithi151 ceph-mon[41387]: osdmap e24: 4 total, 3 up, 4 in 2024-09-13T14:16:05.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:05 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:16:05.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:05 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:16:05.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:05 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:16:05.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:05 smithi151 ceph-mon[29807]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-09-13T14:16:05.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:05 smithi151 ceph-mon[29807]: osdmap e24: 4 total, 3 up, 4 in 2024-09-13T14:16:05.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:05 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:16:05.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:05 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:16:05.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:05 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:16:06.695 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:06 smithi184 ceph-mon[33394]: purged_snaps scrub starts 2024-09-13T14:16:06.695 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:06 smithi184 ceph-mon[33394]: purged_snaps scrub ok 2024-09-13T14:16:06.695 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:06 smithi184 ceph-mon[33394]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:16:06.695 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:06 smithi184 ceph-mon[33394]: osd.3 [v2:172.21.15.151:6826/2203980888,v1:172.21.15.151:6827/2203980888] boot 2024-09-13T14:16:06.695 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:06 smithi184 ceph-mon[33394]: osdmap e25: 4 total, 4 up, 4 in 2024-09-13T14:16:06.695 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:06 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:16:06.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:06 smithi151 ceph-mon[29807]: purged_snaps scrub starts 2024-09-13T14:16:06.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:06 smithi151 ceph-mon[29807]: purged_snaps scrub ok 2024-09-13T14:16:06.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:06 smithi151 ceph-mon[29807]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:16:06.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:06 smithi151 ceph-mon[29807]: osd.3 [v2:172.21.15.151:6826/2203980888,v1:172.21.15.151:6827/2203980888] boot 2024-09-13T14:16:06.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:06 smithi151 ceph-mon[29807]: osdmap e25: 4 total, 4 up, 4 in 2024-09-13T14:16:06.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:06 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:16:06.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:06 smithi151 ceph-mon[41387]: purged_snaps scrub starts 2024-09-13T14:16:06.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:06 smithi151 ceph-mon[41387]: purged_snaps scrub ok 2024-09-13T14:16:06.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:06 smithi151 ceph-mon[41387]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-13T14:16:06.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:06 smithi151 ceph-mon[41387]: osd.3 [v2:172.21.15.151:6826/2203980888,v1:172.21.15.151:6827/2203980888] boot 2024-09-13T14:16:06.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:06 smithi151 ceph-mon[41387]: osdmap e25: 4 total, 4 up, 4 in 2024-09-13T14:16:06.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:06 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:16:07.520 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:16:07.539 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi184:vg_nvme/lv_4 2024-09-13T14:16:07.777 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:16:07.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:07 smithi184 ceph-mon[33394]: osdmap e26: 4 total, 4 up, 4 in 2024-09-13T14:16:08.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:07 smithi151 ceph-mon[29807]: osdmap e26: 4 total, 4 up, 4 in 2024-09-13T14:16:08.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:07 smithi151 ceph-mon[41387]: osdmap e26: 4 total, 4 up, 4 in 2024-09-13T14:16:08.904 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:08 smithi184 ceph-mon[33394]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:08.905 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:08 smithi184 ceph-mon[33394]: Detected new or changed devices on smithi151 2024-09-13T14:16:08.905 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:08 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:08.905 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:08 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:08.905 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:08 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:16:08.905 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:08 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:16:08.905 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:08 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:16:08.905 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:08 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:16:08.905 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:08 smithi184 ceph-mon[33394]: Adjusting osd_memory_target on smithi151 to 4016M 2024-09-13T14:16:08.905 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:08 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:08.905 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:08 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:08.905 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:08 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:08.905 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:08 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:08.905 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:08 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:08.906 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:08 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:08.906 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:08 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:08.906 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:08 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:09.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[29807]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:09.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[29807]: Detected new or changed devices on smithi151 2024-09-13T14:16:09.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:09.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:09.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:16:09.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:16:09.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:16:09.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:16:09.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[29807]: Adjusting osd_memory_target on smithi151 to 4016M 2024-09-13T14:16:09.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:09.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:09.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:09.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:09.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:09.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:09.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:09.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:09.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[41387]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:09.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[41387]: Detected new or changed devices on smithi151 2024-09-13T14:16:09.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:09.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:09.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:16:09.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:16:09.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:16:09.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:16:09.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[41387]: Adjusting osd_memory_target on smithi151 to 4016M 2024-09-13T14:16:09.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:09.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:09.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:09.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:09.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:09.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:09.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:09.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:08 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:10.051 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:09 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:16:10.051 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:09 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:16:10.051 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:09 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:10.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:09 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:16:10.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:09 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:16:10.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:09 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:10.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:09 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:16:10.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:09 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:16:10.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:09 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:11.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:10 smithi184 ceph-mon[33394]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:11.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:10 smithi184 ceph-mon[33394]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:11.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:10 smithi151 ceph-mon[29807]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:11.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:10 smithi151 ceph-mon[29807]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:11.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:10 smithi151 ceph-mon[41387]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:11.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:10 smithi151 ceph-mon[41387]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:13.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:12 smithi184 ceph-mon[33394]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:13.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:12 smithi184 ceph-mon[33394]: from='client.? 172.21.15.184:0/2767724348' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75e258c0-1145-4aee-86aa-db2b63082cc8"}]: dispatch 2024-09-13T14:16:13.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:12 smithi184 ceph-mon[33394]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75e258c0-1145-4aee-86aa-db2b63082cc8"}]: dispatch 2024-09-13T14:16:13.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:12 smithi184 ceph-mon[33394]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75e258c0-1145-4aee-86aa-db2b63082cc8"}]': finished 2024-09-13T14:16:13.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:12 smithi184 ceph-mon[33394]: osdmap e27: 5 total, 4 up, 5 in 2024-09-13T14:16:13.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:12 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:16:13.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:12 smithi184 ceph-mon[33394]: from='client.? 172.21.15.184:0/2886991698' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:16:13.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:12 smithi151 ceph-mon[41387]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:13.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:12 smithi151 ceph-mon[41387]: from='client.? 172.21.15.184:0/2767724348' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75e258c0-1145-4aee-86aa-db2b63082cc8"}]: dispatch 2024-09-13T14:16:13.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:12 smithi151 ceph-mon[41387]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75e258c0-1145-4aee-86aa-db2b63082cc8"}]: dispatch 2024-09-13T14:16:13.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:12 smithi151 ceph-mon[41387]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75e258c0-1145-4aee-86aa-db2b63082cc8"}]': finished 2024-09-13T14:16:13.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:12 smithi151 ceph-mon[41387]: osdmap e27: 5 total, 4 up, 5 in 2024-09-13T14:16:13.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:12 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:16:13.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:12 smithi151 ceph-mon[41387]: from='client.? 172.21.15.184:0/2886991698' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:16:13.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:12 smithi151 ceph-mon[29807]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:13.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:12 smithi151 ceph-mon[29807]: from='client.? 172.21.15.184:0/2767724348' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75e258c0-1145-4aee-86aa-db2b63082cc8"}]: dispatch 2024-09-13T14:16:13.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:12 smithi151 ceph-mon[29807]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75e258c0-1145-4aee-86aa-db2b63082cc8"}]: dispatch 2024-09-13T14:16:13.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:12 smithi151 ceph-mon[29807]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75e258c0-1145-4aee-86aa-db2b63082cc8"}]': finished 2024-09-13T14:16:13.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:12 smithi151 ceph-mon[29807]: osdmap e27: 5 total, 4 up, 5 in 2024-09-13T14:16:13.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:12 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:16:13.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:12 smithi151 ceph-mon[29807]: from='client.? 172.21.15.184:0/2886991698' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:16:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:14 smithi184 ceph-mon[33394]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:15.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:14 smithi151 ceph-mon[29807]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:15.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:14 smithi151 ceph-mon[41387]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:17.044 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:16 smithi184 ceph-mon[33394]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:17.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:16 smithi151 ceph-mon[29807]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:17.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:16 smithi151 ceph-mon[41387]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:18.970 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:18 smithi184 ceph-mon[33394]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:18.970 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:18 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-13T14:16:18.970 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:18 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:19.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:18 smithi151 ceph-mon[29807]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:19.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:18 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-13T14:16:19.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:18 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:19.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:18 smithi151 ceph-mon[41387]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:19.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:18 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-13T14:16:19.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:18 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:19.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:19 smithi184 ceph-mon[33394]: Deploying daemon osd.4 on smithi184 2024-09-13T14:16:20.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:19 smithi151 ceph-mon[29807]: Deploying daemon osd.4 on smithi184 2024-09-13T14:16:20.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:19 smithi151 ceph-mon[41387]: Deploying daemon osd.4 on smithi184 2024-09-13T14:16:20.834 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:20 smithi184 ceph-mon[33394]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:21.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:20 smithi151 ceph-mon[29807]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:21.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:20 smithi151 ceph-mon[41387]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:22.960 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:22 smithi184 ceph-mon[33394]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:23.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:22 smithi151 ceph-mon[29807]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:23.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:22 smithi151 ceph-mon[41387]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:23.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:23 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:23.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:23 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:23.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:23 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:24.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:23 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:24.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:23 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:24.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:23 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:24.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:23 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:24.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:23 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:24.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:23 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:24.850 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:24 smithi184 ceph-mon[33394]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:25.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:24 smithi151 ceph-mon[29807]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:25.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:24 smithi151 ceph-mon[41387]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:26.366 INFO:teuthology.orchestra.run.smithi184.stdout:Created osd(s) 4 on host 'smithi184' 2024-09-13T14:16:26.976 DEBUG:teuthology.orchestra.run.smithi184:osd.4> sudo journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.4.service 2024-09-13T14:16:26.979 INFO:tasks.cephadm:Deploying osd.5 on smithi184 with /dev/vg_nvme/lv_3... 2024-09-13T14:16:26.980 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-13T14:16:27.216 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:26 smithi184 ceph-mon[33394]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:27.216 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:26 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:27.216 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:26 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:27.216 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:26 smithi184 ceph-mon[33394]: from='osd.4 [v2:172.21.15.184:6800/3275553935,v1:172.21.15.184:6801/3275553935]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T14:16:27.216 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:26 smithi184 ceph-mon[33394]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T14:16:27.216 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:26 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:27.216 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:26 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:27.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:26 smithi151 ceph-mon[41387]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:27.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:26 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:27.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:26 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:27.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:26 smithi151 ceph-mon[41387]: from='osd.4 [v2:172.21.15.184:6800/3275553935,v1:172.21.15.184:6801/3275553935]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T14:16:27.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:26 smithi151 ceph-mon[41387]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T14:16:27.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:26 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:27.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:26 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:27.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:26 smithi151 ceph-mon[29807]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:27.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:26 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:27.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:26 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:27.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:26 smithi151 ceph-mon[29807]: from='osd.4 [v2:172.21.15.184:6800/3275553935,v1:172.21.15.184:6801/3275553935]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T14:16:27.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:26 smithi151 ceph-mon[29807]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T14:16:27.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:26 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:27.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:26 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:27.768 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:16:28.161 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:28 smithi184 ceph-mon[33394]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-13T14:16:28.161 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:28 smithi184 ceph-mon[33394]: osdmap e28: 5 total, 4 up, 5 in 2024-09-13T14:16:28.161 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:28 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:16:28.161 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:28 smithi184 ceph-mon[33394]: from='osd.4 [v2:172.21.15.184:6800/3275553935,v1:172.21.15.184:6801/3275553935]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-13T14:16:28.161 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:28 smithi184 ceph-mon[33394]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-13T14:16:28.162 INFO:journalctl@ceph.osd.4.smithi184.stdout:Sep 13 14:16:28 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-4[38671]: 2024-09-13T14:16:28.006+0000 7fac4e028640 -1 osd.4 0 waiting for initial osdmap 2024-09-13T14:16:28.162 INFO:journalctl@ceph.osd.4.smithi184.stdout:Sep 13 14:16:28 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-4[38671]: 2024-09-13T14:16:28.012+0000 7fac4763a640 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:16:28.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:28 smithi151 ceph-mon[29807]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-13T14:16:28.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:28 smithi151 ceph-mon[29807]: osdmap e28: 5 total, 4 up, 5 in 2024-09-13T14:16:28.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:28 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:16:28.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:28 smithi151 ceph-mon[29807]: from='osd.4 [v2:172.21.15.184:6800/3275553935,v1:172.21.15.184:6801/3275553935]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-13T14:16:28.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:28 smithi151 ceph-mon[29807]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-13T14:16:28.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:28 smithi151 ceph-mon[41387]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-13T14:16:28.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:28 smithi151 ceph-mon[41387]: osdmap e28: 5 total, 4 up, 5 in 2024-09-13T14:16:28.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:28 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:16:28.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:28 smithi151 ceph-mon[41387]: from='osd.4 [v2:172.21.15.184:6800/3275553935,v1:172.21.15.184:6801/3275553935]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-13T14:16:28.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:28 smithi151 ceph-mon[41387]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-13T14:16:29.248 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:29 smithi184 ceph-mon[33394]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:29.249 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:29 smithi184 ceph-mon[33394]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-13T14:16:29.249 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:29 smithi184 ceph-mon[33394]: osdmap e29: 5 total, 4 up, 5 in 2024-09-13T14:16:29.249 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:29 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:16:29.249 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:29 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:16:29.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:29 smithi151 ceph-mon[29807]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:29.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:29 smithi151 ceph-mon[29807]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-13T14:16:29.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:29 smithi151 ceph-mon[29807]: osdmap e29: 5 total, 4 up, 5 in 2024-09-13T14:16:29.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:29 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:16:29.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:29 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:16:29.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:29 smithi151 ceph-mon[41387]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-13T14:16:29.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:29 smithi151 ceph-mon[41387]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-13T14:16:29.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:29 smithi151 ceph-mon[41387]: osdmap e29: 5 total, 4 up, 5 in 2024-09-13T14:16:29.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:29 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:16:29.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:29 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:16:30.268 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:30 smithi184 ceph-mon[33394]: purged_snaps scrub starts 2024-09-13T14:16:30.269 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:30 smithi184 ceph-mon[33394]: purged_snaps scrub ok 2024-09-13T14:16:30.269 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:30 smithi184 ceph-mon[33394]: osd.4 [v2:172.21.15.184:6800/3275553935,v1:172.21.15.184:6801/3275553935] boot 2024-09-13T14:16:30.269 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:30 smithi184 ceph-mon[33394]: osdmap e30: 5 total, 5 up, 5 in 2024-09-13T14:16:30.269 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:30 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:16:30.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:30 smithi151 ceph-mon[29807]: purged_snaps scrub starts 2024-09-13T14:16:30.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:30 smithi151 ceph-mon[29807]: purged_snaps scrub ok 2024-09-13T14:16:30.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:30 smithi151 ceph-mon[29807]: osd.4 [v2:172.21.15.184:6800/3275553935,v1:172.21.15.184:6801/3275553935] boot 2024-09-13T14:16:30.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:30 smithi151 ceph-mon[29807]: osdmap e30: 5 total, 5 up, 5 in 2024-09-13T14:16:30.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:30 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:16:30.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:30 smithi151 ceph-mon[41387]: purged_snaps scrub starts 2024-09-13T14:16:30.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:30 smithi151 ceph-mon[41387]: purged_snaps scrub ok 2024-09-13T14:16:30.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:30 smithi151 ceph-mon[41387]: osd.4 [v2:172.21.15.184:6800/3275553935,v1:172.21.15.184:6801/3275553935] boot 2024-09-13T14:16:30.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:30 smithi151 ceph-mon[41387]: osdmap e30: 5 total, 5 up, 5 in 2024-09-13T14:16:30.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:30 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:16:31.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:31 smithi184 ceph-mon[33394]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:16:31.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:31 smithi184 ceph-mon[33394]: osdmap e31: 5 total, 5 up, 5 in 2024-09-13T14:16:31.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:31 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:31.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:31 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:31.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:31 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:16:31.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:31 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:31.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:31 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:31.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:31 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:31.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:31 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:31.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:31 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:31.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:31 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:31.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:31 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:31.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:31 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:31.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:31 smithi184 ceph-mon[33394]: osdmap e32: 5 total, 5 up, 5 in 2024-09-13T14:16:31.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:31 smithi151 ceph-mon[29807]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:16:31.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:31 smithi151 ceph-mon[29807]: osdmap e31: 5 total, 5 up, 5 in 2024-09-13T14:16:31.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:31 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:31.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:31 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:31.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:31 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:16:31.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:31 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:31.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:31 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:31.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:31 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:31.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:31 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:31.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:31 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:31.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:31 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:31.361 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:31 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:31.361 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:31 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:31.361 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:31 smithi151 ceph-mon[29807]: osdmap e32: 5 total, 5 up, 5 in 2024-09-13T14:16:31.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:31 smithi151 ceph-mon[41387]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:16:31.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:31 smithi151 ceph-mon[41387]: osdmap e31: 5 total, 5 up, 5 in 2024-09-13T14:16:31.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:31 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:31.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:31 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:31.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:31 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:16:31.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:31 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:31.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:31 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:31.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:31 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:31.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:31 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:31.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:31 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:31.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:31 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:31.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:31 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:31.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:31 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:31.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:31 smithi151 ceph-mon[41387]: osdmap e32: 5 total, 5 up, 5 in 2024-09-13T14:16:32.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:32 smithi184 ceph-mon[33394]: Detected new or changed devices on smithi184 2024-09-13T14:16:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:32 smithi184 ceph-mon[33394]: Adjusting osd_memory_target on smithi184 to 17088M 2024-09-13T14:16:32.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:32 smithi151 ceph-mon[29807]: Detected new or changed devices on smithi184 2024-09-13T14:16:32.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:32 smithi151 ceph-mon[29807]: Adjusting osd_memory_target on smithi184 to 17088M 2024-09-13T14:16:32.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:32 smithi151 ceph-mon[41387]: Detected new or changed devices on smithi184 2024-09-13T14:16:32.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:32 smithi151 ceph-mon[41387]: Adjusting osd_memory_target on smithi184 to 17088M 2024-09-13T14:16:32.384 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:16:32.404 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi184:vg_nvme/lv_3 2024-09-13T14:16:32.642 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:16:33.334 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:33 smithi184 ceph-mon[33394]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:16:33.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:33 smithi151 ceph-mon[41387]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:16:33.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:33 smithi151 ceph-mon[29807]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:16:35.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:35 smithi184 ceph-mon[33394]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-09-13T14:16:35.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:35 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:16:35.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:35 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:16:35.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:35 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:35.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:35 smithi151 ceph-mon[41387]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-09-13T14:16:35.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:35 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:16:35.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:35 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:16:35.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:35 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:35.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:35 smithi151 ceph-mon[29807]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-09-13T14:16:35.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:35 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:16:35.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:35 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:16:35.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:35 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:36.297 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:36 smithi184 ceph-mon[33394]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:36.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:36 smithi151 ceph-mon[29807]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:36.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:36 smithi151 ceph-mon[41387]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:37.268 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:37 smithi184 ceph-mon[33394]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-09-13T14:16:37.268 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:37 smithi184 ceph-mon[33394]: from='client.? 172.21.15.184:0/1616900140' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f56a5366-9d22-4960-933d-340aa280fbd1"}]: dispatch 2024-09-13T14:16:37.269 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:37 smithi184 ceph-mon[33394]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f56a5366-9d22-4960-933d-340aa280fbd1"}]: dispatch 2024-09-13T14:16:37.269 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:37 smithi184 ceph-mon[33394]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f56a5366-9d22-4960-933d-340aa280fbd1"}]': finished 2024-09-13T14:16:37.269 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:37 smithi184 ceph-mon[33394]: osdmap e33: 6 total, 5 up, 6 in 2024-09-13T14:16:37.269 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:37 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:16:37.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:37 smithi151 ceph-mon[29807]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-09-13T14:16:37.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:37 smithi151 ceph-mon[29807]: from='client.? 172.21.15.184:0/1616900140' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f56a5366-9d22-4960-933d-340aa280fbd1"}]: dispatch 2024-09-13T14:16:37.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:37 smithi151 ceph-mon[29807]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f56a5366-9d22-4960-933d-340aa280fbd1"}]: dispatch 2024-09-13T14:16:37.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:37 smithi151 ceph-mon[29807]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f56a5366-9d22-4960-933d-340aa280fbd1"}]': finished 2024-09-13T14:16:37.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:37 smithi151 ceph-mon[29807]: osdmap e33: 6 total, 5 up, 6 in 2024-09-13T14:16:37.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:37 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:16:37.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:37 smithi151 ceph-mon[41387]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-09-13T14:16:37.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:37 smithi151 ceph-mon[41387]: from='client.? 172.21.15.184:0/1616900140' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f56a5366-9d22-4960-933d-340aa280fbd1"}]: dispatch 2024-09-13T14:16:37.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:37 smithi151 ceph-mon[41387]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f56a5366-9d22-4960-933d-340aa280fbd1"}]: dispatch 2024-09-13T14:16:37.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:37 smithi151 ceph-mon[41387]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f56a5366-9d22-4960-933d-340aa280fbd1"}]': finished 2024-09-13T14:16:37.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:37 smithi151 ceph-mon[41387]: osdmap e33: 6 total, 5 up, 6 in 2024-09-13T14:16:37.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:37 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:16:38.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:38 smithi184 ceph-mon[33394]: from='client.? 172.21.15.184:0/2697513' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:16:38.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:38 smithi151 ceph-mon[29807]: from='client.? 172.21.15.184:0/2697513' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:16:38.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:38 smithi151 ceph-mon[41387]: from='client.? 172.21.15.184:0/2697513' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:16:39.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:39 smithi184 ceph-mon[33394]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-09-13T14:16:39.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:39 smithi151 ceph-mon[41387]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-09-13T14:16:39.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:39 smithi151 ceph-mon[29807]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-09-13T14:16:41.200 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:41 smithi184 ceph-mon[33394]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-13T14:16:41.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:41 smithi151 ceph-mon[29807]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-13T14:16:41.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:41 smithi151 ceph-mon[41387]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-13T14:16:43.300 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:43 smithi184 ceph-mon[33394]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:16:43.300 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:43 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-13T14:16:43.300 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:43 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:43.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:43 smithi151 ceph-mon[29807]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:16:43.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:43 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-13T14:16:43.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:43 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:43.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:43 smithi151 ceph-mon[41387]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:16:43.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:43 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-13T14:16:43.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:43 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:44.188 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:44 smithi184 ceph-mon[33394]: Deploying daemon osd.5 on smithi184 2024-09-13T14:16:44.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:44 smithi151 ceph-mon[29807]: Deploying daemon osd.5 on smithi184 2024-09-13T14:16:44.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:44 smithi151 ceph-mon[41387]: Deploying daemon osd.5 on smithi184 2024-09-13T14:16:45.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:45 smithi184 ceph-mon[33394]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:16:45.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:45 smithi151 ceph-mon[29807]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:16:45.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:45 smithi151 ceph-mon[41387]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:16:47.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:47 smithi184 ceph-mon[33394]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:16:47.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:47 smithi151 ceph-mon[29807]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:16:47.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:47 smithi151 ceph-mon[41387]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:16:48.332 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:48 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:48.332 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:48 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:48.332 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:48 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:48.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:48 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:48.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:48 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:48.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:48 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:48.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:48 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:48.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:48 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:48.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:48 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:49.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:49 smithi184 ceph-mon[33394]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:16:49.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:49 smithi151 ceph-mon[29807]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:16:49.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:49 smithi151 ceph-mon[41387]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:16:51.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:51 smithi184 ceph-mon[33394]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:16:51.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:51 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:51.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:51 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:51.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:51 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:51.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:51 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:51.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:51 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:51.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:51 smithi184 ceph-mon[33394]: from='osd.5 [v2:172.21.15.184:6808/2446419621,v1:172.21.15.184:6809/2446419621]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T14:16:51.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:51 smithi184 ceph-mon[33394]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T14:16:51.186 INFO:teuthology.orchestra.run.smithi184.stdout:Created osd(s) 5 on host 'smithi184' 2024-09-13T14:16:51.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:51 smithi151 ceph-mon[29807]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:16:51.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:51 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:51.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:51 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:51.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:51 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:51.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:51 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:51.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:51 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:51.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:51 smithi151 ceph-mon[29807]: from='osd.5 [v2:172.21.15.184:6808/2446419621,v1:172.21.15.184:6809/2446419621]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T14:16:51.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:51 smithi151 ceph-mon[29807]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T14:16:51.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:51 smithi151 ceph-mon[41387]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:16:51.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:51 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:51.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:51 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:51.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:51 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:51.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:51 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:51.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:51 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:51.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:51 smithi151 ceph-mon[41387]: from='osd.5 [v2:172.21.15.184:6808/2446419621,v1:172.21.15.184:6809/2446419621]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T14:16:51.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:51 smithi151 ceph-mon[41387]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T14:16:51.883 DEBUG:teuthology.orchestra.run.smithi184:osd.5> sudo journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.5.service 2024-09-13T14:16:51.885 INFO:tasks.cephadm:Deploying osd.6 on smithi184 with /dev/vg_nvme/lv_2... 2024-09-13T14:16:51.886 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-13T14:16:52.286 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:52 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:52.286 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:52 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:52.287 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:52 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:52.287 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:52 smithi184 ceph-mon[33394]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-13T14:16:52.287 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:52 smithi184 ceph-mon[33394]: osdmap e34: 6 total, 5 up, 6 in 2024-09-13T14:16:52.287 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:52 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:16:52.287 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:52 smithi184 ceph-mon[33394]: from='osd.5 [v2:172.21.15.184:6808/2446419621,v1:172.21.15.184:6809/2446419621]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-13T14:16:52.287 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:52 smithi184 ceph-mon[33394]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-13T14:16:52.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:52 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:52.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:52 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:52.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:52 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:52.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:52 smithi151 ceph-mon[29807]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-13T14:16:52.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:52 smithi151 ceph-mon[29807]: osdmap e34: 6 total, 5 up, 6 in 2024-09-13T14:16:52.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:52 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:16:52.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:52 smithi151 ceph-mon[29807]: from='osd.5 [v2:172.21.15.184:6808/2446419621,v1:172.21.15.184:6809/2446419621]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-13T14:16:52.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:52 smithi151 ceph-mon[29807]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-13T14:16:52.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:52 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:52.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:52 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:52.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:52 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:52.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:52 smithi151 ceph-mon[41387]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-13T14:16:52.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:52 smithi151 ceph-mon[41387]: osdmap e34: 6 total, 5 up, 6 in 2024-09-13T14:16:52.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:52 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:16:52.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:52 smithi151 ceph-mon[41387]: from='osd.5 [v2:172.21.15.184:6808/2446419621,v1:172.21.15.184:6809/2446419621]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-13T14:16:52.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:52 smithi151 ceph-mon[41387]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-13T14:16:52.604 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 14:16:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T14:16:52.459+0000 7f2681e75640 -1 osd.5 0 waiting for initial osdmap 2024-09-13T14:16:52.604 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 14:16:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T14:16:52.464+0000 7f267bcaa640 -1 osd.5 35 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:16:52.754 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:16:53.352 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:53 smithi184 ceph-mon[33394]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:16:53.352 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:53 smithi184 ceph-mon[33394]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-13T14:16:53.352 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:53 smithi184 ceph-mon[33394]: osdmap e35: 6 total, 5 up, 6 in 2024-09-13T14:16:53.352 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:53 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:16:53.352 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:53 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:16:53.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:53 smithi184 ceph-mon[33394]: osd.5 [v2:172.21.15.184:6808/2446419621,v1:172.21.15.184:6809/2446419621] boot 2024-09-13T14:16:53.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:53 smithi184 ceph-mon[33394]: osdmap e36: 6 total, 6 up, 6 in 2024-09-13T14:16:53.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:53 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:16:53.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:53 smithi151 ceph-mon[29807]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:16:53.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:53 smithi151 ceph-mon[29807]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-13T14:16:53.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:53 smithi151 ceph-mon[29807]: osdmap e35: 6 total, 5 up, 6 in 2024-09-13T14:16:53.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:53 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:16:53.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:53 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:16:53.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:53 smithi151 ceph-mon[29807]: osd.5 [v2:172.21.15.184:6808/2446419621,v1:172.21.15.184:6809/2446419621] boot 2024-09-13T14:16:53.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:53 smithi151 ceph-mon[29807]: osdmap e36: 6 total, 6 up, 6 in 2024-09-13T14:16:53.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:53 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:16:53.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:53 smithi151 ceph-mon[41387]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-13T14:16:53.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:53 smithi151 ceph-mon[41387]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-13T14:16:53.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:53 smithi151 ceph-mon[41387]: osdmap e35: 6 total, 5 up, 6 in 2024-09-13T14:16:53.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:53 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:16:53.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:53 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:16:53.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:53 smithi151 ceph-mon[41387]: osd.5 [v2:172.21.15.184:6808/2446419621,v1:172.21.15.184:6809/2446419621] boot 2024-09-13T14:16:53.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:53 smithi151 ceph-mon[41387]: osdmap e36: 6 total, 6 up, 6 in 2024-09-13T14:16:53.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:53 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:16:55.071 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:54 smithi184 ceph-mon[33394]: purged_snaps scrub starts 2024-09-13T14:16:55.071 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:54 smithi184 ceph-mon[33394]: purged_snaps scrub ok 2024-09-13T14:16:55.071 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:54 smithi184 ceph-mon[33394]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:16:55.071 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:54 smithi184 ceph-mon[33394]: osdmap e37: 6 total, 6 up, 6 in 2024-09-13T14:16:55.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:54 smithi151 ceph-mon[29807]: purged_snaps scrub starts 2024-09-13T14:16:55.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:54 smithi151 ceph-mon[29807]: purged_snaps scrub ok 2024-09-13T14:16:55.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:54 smithi151 ceph-mon[29807]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:16:55.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:54 smithi151 ceph-mon[29807]: osdmap e37: 6 total, 6 up, 6 in 2024-09-13T14:16:55.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:54 smithi151 ceph-mon[41387]: purged_snaps scrub starts 2024-09-13T14:16:55.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:54 smithi151 ceph-mon[41387]: purged_snaps scrub ok 2024-09-13T14:16:55.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:54 smithi151 ceph-mon[41387]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:16:55.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:54 smithi151 ceph-mon[41387]: osdmap e37: 6 total, 6 up, 6 in 2024-09-13T14:16:56.068 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:55 smithi184 ceph-mon[33394]: osdmap e38: 6 total, 6 up, 6 in 2024-09-13T14:16:56.068 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:55 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:56.068 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:55 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:56.068 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:55 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:16:56.068 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:55 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:16:56.068 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:55 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:56.069 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:55 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:56.069 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:55 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:56.069 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:55 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:56.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:55 smithi151 ceph-mon[29807]: osdmap e38: 6 total, 6 up, 6 in 2024-09-13T14:16:56.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:55 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:56.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:55 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:56.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:55 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:16:56.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:55 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:16:56.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:55 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:56.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:55 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:56.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:55 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:56.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:55 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:56.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:55 smithi151 ceph-mon[41387]: osdmap e38: 6 total, 6 up, 6 in 2024-09-13T14:16:56.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:55 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:56.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:55 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:56.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:55 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:16:56.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:55 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:16:56.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:55 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:56.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:55 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:56.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:55 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:56.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:55 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:16:56.788 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:56 smithi184 ceph-mon[33394]: Detected new or changed devices on smithi184 2024-09-13T14:16:56.788 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:56 smithi184 ceph-mon[33394]: Adjusting osd_memory_target on smithi184 to 8544M 2024-09-13T14:16:56.789 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:56 smithi184 ceph-mon[33394]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:16:56.793 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:16:56.813 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi184:vg_nvme/lv_2 2024-09-13T14:16:57.009 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:56 smithi151 ceph-mon[29807]: Detected new or changed devices on smithi184 2024-09-13T14:16:57.010 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:56 smithi151 ceph-mon[29807]: Adjusting osd_memory_target on smithi184 to 8544M 2024-09-13T14:16:57.010 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:56 smithi151 ceph-mon[29807]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:16:57.010 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:56 smithi151 ceph-mon[41387]: Detected new or changed devices on smithi184 2024-09-13T14:16:57.010 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:56 smithi151 ceph-mon[41387]: Adjusting osd_memory_target on smithi184 to 8544M 2024-09-13T14:16:57.010 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:56 smithi151 ceph-mon[41387]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:16:57.053 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:16:58.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:58 smithi184 ceph-mon[33394]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:16:59.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:58 smithi151 ceph-mon[41387]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:16:59.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:58 smithi151 ceph-mon[29807]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:17:00.094 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:59 smithi184 ceph-mon[33394]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:00.095 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:59 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:17:00.095 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:59 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:17:00.095 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:16:59 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:00.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:59 smithi151 ceph-mon[29807]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:00.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:59 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:17:00.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:59 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:17:00.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:16:59 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:00.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:59 smithi151 ceph-mon[41387]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:59 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:17:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:59 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:17:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:16:59 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:01.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:00 smithi184 ceph-mon[33394]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:17:01.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:00 smithi151 ceph-mon[29807]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:17:01.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:00 smithi151 ceph-mon[41387]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:17:02.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:01 smithi184 ceph-mon[33394]: from='client.? 172.21.15.184:0/248507103' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9826568-a728-45ab-8180-9777fe31f93a"}]: dispatch 2024-09-13T14:17:02.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:01 smithi184 ceph-mon[33394]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9826568-a728-45ab-8180-9777fe31f93a"}]: dispatch 2024-09-13T14:17:02.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:01 smithi184 ceph-mon[33394]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9826568-a728-45ab-8180-9777fe31f93a"}]': finished 2024-09-13T14:17:02.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:01 smithi184 ceph-mon[33394]: osdmap e39: 7 total, 6 up, 7 in 2024-09-13T14:17:02.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:01 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:17:02.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:01 smithi184 ceph-mon[33394]: from='client.? 172.21.15.184:0/472361515' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:17:02.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:01 smithi151 ceph-mon[29807]: from='client.? 172.21.15.184:0/248507103' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9826568-a728-45ab-8180-9777fe31f93a"}]: dispatch 2024-09-13T14:17:02.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:01 smithi151 ceph-mon[29807]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9826568-a728-45ab-8180-9777fe31f93a"}]: dispatch 2024-09-13T14:17:02.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:01 smithi151 ceph-mon[29807]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9826568-a728-45ab-8180-9777fe31f93a"}]': finished 2024-09-13T14:17:02.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:01 smithi151 ceph-mon[29807]: osdmap e39: 7 total, 6 up, 7 in 2024-09-13T14:17:02.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:01 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:17:02.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:01 smithi151 ceph-mon[29807]: from='client.? 172.21.15.184:0/472361515' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:17:02.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:01 smithi151 ceph-mon[41387]: from='client.? 172.21.15.184:0/248507103' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9826568-a728-45ab-8180-9777fe31f93a"}]: dispatch 2024-09-13T14:17:02.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:01 smithi151 ceph-mon[41387]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9826568-a728-45ab-8180-9777fe31f93a"}]: dispatch 2024-09-13T14:17:02.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:01 smithi151 ceph-mon[41387]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9826568-a728-45ab-8180-9777fe31f93a"}]': finished 2024-09-13T14:17:02.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:01 smithi151 ceph-mon[41387]: osdmap e39: 7 total, 6 up, 7 in 2024-09-13T14:17:02.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:01 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:17:02.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:01 smithi151 ceph-mon[41387]: from='client.? 172.21.15.184:0/472361515' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:17:03.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:03 smithi184 ceph-mon[33394]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:17:03.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:03 smithi151 ceph-mon[29807]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:17:03.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:03 smithi151 ceph-mon[41387]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:17:05.089 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:05 smithi184 ceph-mon[33394]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:17:05.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:05 smithi151 ceph-mon[29807]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:17:05.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:05 smithi151 ceph-mon[41387]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:17:07.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:07 smithi184 ceph-mon[33394]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:17:07.361 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:07 smithi151 ceph-mon[29807]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:17:07.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:07 smithi151 ceph-mon[41387]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:17:08.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:08 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-13T14:17:08.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:08 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:08.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:08 smithi184 ceph-mon[33394]: Deploying daemon osd.6 on smithi184 2024-09-13T14:17:08.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:08 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-13T14:17:08.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:08 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:08.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:08 smithi151 ceph-mon[29807]: Deploying daemon osd.6 on smithi184 2024-09-13T14:17:08.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:08 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-13T14:17:08.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:08 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:08.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:08 smithi151 ceph-mon[41387]: Deploying daemon osd.6 on smithi184 2024-09-13T14:17:09.340 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:09 smithi151 ceph-mon[29807]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:17:09.340 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:09 smithi151 ceph-mon[41387]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:17:09.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:09 smithi184 ceph-mon[33394]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:17:11.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:11 smithi184 ceph-mon[33394]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:17:11.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:11 smithi151 ceph-mon[41387]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:17:11.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:11 smithi151 ceph-mon[29807]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:17:12.271 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:12 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:12.271 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:12 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:12.271 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:12 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:12.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:12 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:12.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:12 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:12.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:12 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:12.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:12 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:12.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:12 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:12.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:12 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:13.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:13 smithi184 ceph-mon[33394]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:17:13.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:13 smithi151 ceph-mon[41387]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:17:13.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:13 smithi151 ceph-mon[29807]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:17:15.151 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:15 smithi184 ceph-mon[33394]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:17:15.188 INFO:teuthology.orchestra.run.smithi184.stdout:Created osd(s) 6 on host 'smithi184' 2024-09-13T14:17:15.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:15 smithi151 ceph-mon[41387]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:17:15.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:15 smithi151 ceph-mon[29807]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:17:16.050 DEBUG:teuthology.orchestra.run.smithi184:osd.6> sudo journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.6.service 2024-09-13T14:17:16.056 INFO:tasks.cephadm:Deploying osd.7 on smithi184 with /dev/vg_nvme/lv_1... 2024-09-13T14:17:16.056 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-13T14:17:16.348 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:17:16.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:16 smithi184 ceph-mon[33394]: from='osd.6 [v2:172.21.15.184:6816/2214837203,v1:172.21.15.184:6817/2214837203]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T14:17:16.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:16 smithi184 ceph-mon[33394]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T14:17:16.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:16 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:16.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:16 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:16.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:16 smithi151 ceph-mon[41387]: from='osd.6 [v2:172.21.15.184:6816/2214837203,v1:172.21.15.184:6817/2214837203]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T14:17:16.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:16 smithi151 ceph-mon[41387]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T14:17:16.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:16 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:16.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:16 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:16.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:16 smithi151 ceph-mon[29807]: from='osd.6 [v2:172.21.15.184:6816/2214837203,v1:172.21.15.184:6817/2214837203]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T14:17:16.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:16 smithi151 ceph-mon[29807]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T14:17:16.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:16 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:16.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:16 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:17.428 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:17 smithi184 ceph-mon[33394]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:17:17.428 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:17 smithi184 ceph-mon[33394]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-13T14:17:17.428 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:17 smithi184 ceph-mon[33394]: osdmap e40: 7 total, 6 up, 7 in 2024-09-13T14:17:17.428 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:17 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:17:17.428 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:17 smithi184 ceph-mon[33394]: from='osd.6 [v2:172.21.15.184:6816/2214837203,v1:172.21.15.184:6817/2214837203]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-13T14:17:17.429 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:17 smithi184 ceph-mon[33394]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-13T14:17:17.429 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:17 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:17.429 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:17 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:17.429 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 14:17:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T14:17:17.124+0000 7f9f3d341640 -1 osd.6 0 waiting for initial osdmap 2024-09-13T14:17:17.429 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 14:17:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T14:17:17.129+0000 7f9f36975640 -1 osd.6 41 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:17:17.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:17 smithi151 ceph-mon[41387]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:17:17.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:17 smithi151 ceph-mon[41387]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-13T14:17:17.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:17 smithi151 ceph-mon[41387]: osdmap e40: 7 total, 6 up, 7 in 2024-09-13T14:17:17.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:17 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:17:17.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:17 smithi151 ceph-mon[41387]: from='osd.6 [v2:172.21.15.184:6816/2214837203,v1:172.21.15.184:6817/2214837203]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-13T14:17:17.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:17 smithi151 ceph-mon[41387]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-13T14:17:17.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:17 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:17.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:17 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:17.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:17 smithi151 ceph-mon[29807]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:17:17.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:17 smithi151 ceph-mon[29807]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-13T14:17:17.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:17 smithi151 ceph-mon[29807]: osdmap e40: 7 total, 6 up, 7 in 2024-09-13T14:17:17.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:17 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:17:17.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:17 smithi151 ceph-mon[29807]: from='osd.6 [v2:172.21.15.184:6816/2214837203,v1:172.21.15.184:6817/2214837203]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-13T14:17:17.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:17 smithi151 ceph-mon[29807]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-13T14:17:17.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:17 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:17.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:17 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:18.284 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:18 smithi184 ceph-mon[33394]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-13T14:17:18.284 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:18 smithi184 ceph-mon[33394]: osdmap e41: 7 total, 6 up, 7 in 2024-09-13T14:17:18.285 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:18 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:17:18.285 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:18 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:17:18.285 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:18 smithi184 ceph-mon[33394]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:17:18.285 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:18 smithi184 ceph-mon[33394]: osd.6 [v2:172.21.15.184:6816/2214837203,v1:172.21.15.184:6817/2214837203] boot 2024-09-13T14:17:18.285 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:18 smithi184 ceph-mon[33394]: osdmap e42: 7 total, 7 up, 7 in 2024-09-13T14:17:18.285 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:18 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:17:18.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:18 smithi151 ceph-mon[29807]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-13T14:17:18.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:18 smithi151 ceph-mon[29807]: osdmap e41: 7 total, 6 up, 7 in 2024-09-13T14:17:18.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:18 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:17:18.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:18 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:17:18.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:18 smithi151 ceph-mon[29807]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:17:18.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:18 smithi151 ceph-mon[29807]: osd.6 [v2:172.21.15.184:6816/2214837203,v1:172.21.15.184:6817/2214837203] boot 2024-09-13T14:17:18.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:18 smithi151 ceph-mon[29807]: osdmap e42: 7 total, 7 up, 7 in 2024-09-13T14:17:18.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:18 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:17:18.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:18 smithi151 ceph-mon[41387]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-13T14:17:18.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:18 smithi151 ceph-mon[41387]: osdmap e41: 7 total, 6 up, 7 in 2024-09-13T14:17:18.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:18 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:17:18.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:18 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:17:18.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:18 smithi151 ceph-mon[41387]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:17:18.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:18 smithi151 ceph-mon[41387]: osd.6 [v2:172.21.15.184:6816/2214837203,v1:172.21.15.184:6817/2214837203] boot 2024-09-13T14:17:18.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:18 smithi151 ceph-mon[41387]: osdmap e42: 7 total, 7 up, 7 in 2024-09-13T14:17:18.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:18 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:17:19.111 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-13T14:17:19.130 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi184:vg_nvme/lv_1 2024-09-13T14:17:19.400 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:19 smithi184 ceph-mon[33394]: purged_snaps scrub starts 2024-09-13T14:17:19.401 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:19 smithi184 ceph-mon[33394]: purged_snaps scrub ok 2024-09-13T14:17:19.401 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:19 smithi184 ceph-mon[33394]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:17:19.401 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:19 smithi184 ceph-mon[33394]: osdmap e43: 7 total, 7 up, 7 in 2024-09-13T14:17:19.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:19 smithi151 ceph-mon[29807]: purged_snaps scrub starts 2024-09-13T14:17:19.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:19 smithi151 ceph-mon[29807]: purged_snaps scrub ok 2024-09-13T14:17:19.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:19 smithi151 ceph-mon[29807]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:17:19.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:19 smithi151 ceph-mon[29807]: osdmap e43: 7 total, 7 up, 7 in 2024-09-13T14:17:19.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:19 smithi151 ceph-mon[41387]: purged_snaps scrub starts 2024-09-13T14:17:19.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:19 smithi151 ceph-mon[41387]: purged_snaps scrub ok 2024-09-13T14:17:19.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:19 smithi151 ceph-mon[41387]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:17:19.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:19 smithi151 ceph-mon[41387]: osdmap e43: 7 total, 7 up, 7 in 2024-09-13T14:17:20.183 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:17:20.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:20 smithi184 ceph-mon[33394]: pgmap v149: 1 pgs: 1 peering; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:17:20.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:20 smithi184 ceph-mon[33394]: osdmap e44: 7 total, 7 up, 7 in 2024-09-13T14:17:21.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:20 smithi151 ceph-mon[29807]: pgmap v149: 1 pgs: 1 peering; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:17:21.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:20 smithi151 ceph-mon[29807]: osdmap e44: 7 total, 7 up, 7 in 2024-09-13T14:17:21.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:20 smithi151 ceph-mon[41387]: pgmap v149: 1 pgs: 1 peering; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:17:21.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:20 smithi151 ceph-mon[41387]: osdmap e44: 7 total, 7 up, 7 in 2024-09-13T14:17:22.935 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:22 smithi184 ceph-mon[33394]: pgmap v151: 1 pgs: 1 peering; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:17:22.935 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:22 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:17:22.935 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:22 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:17:22.935 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:22 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:22.935 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:22 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:22.935 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:22 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:22.935 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:22 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:17:22.935 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:22 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:17:22.935 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:22 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:17:22.935 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:22 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:22.936 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:22 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:22.936 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:22 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:22.936 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:22 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:23.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:22 smithi151 ceph-mon[29807]: pgmap v151: 1 pgs: 1 peering; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:17:23.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:22 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:17:23.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:22 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:17:23.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:22 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:23.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:22 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:23.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:22 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:23.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:22 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:17:23.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:22 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:17:23.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:22 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:17:23.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:22 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:23.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:22 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:23.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:22 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:23.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:22 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:23.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:22 smithi151 ceph-mon[41387]: pgmap v151: 1 pgs: 1 peering; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:17:23.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:22 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:17:23.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:22 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:17:23.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:22 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:23.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:22 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:23.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:22 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:23.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:22 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:17:23.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:22 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:17:23.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:22 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:17:23.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:22 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:23.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:22 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:23.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:22 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:23.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:22 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:23.832 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:23 smithi184 ceph-mon[33394]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:23.832 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:23 smithi184 ceph-mon[33394]: Detected new or changed devices on smithi184 2024-09-13T14:17:23.833 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:23 smithi184 ceph-mon[33394]: Adjusting osd_memory_target on smithi184 to 5696M 2024-09-13T14:17:23.833 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:23 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:23.833 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:23 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:23.833 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:23 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:23.833 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:23 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:24.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:23 smithi151 ceph-mon[29807]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:24.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:23 smithi151 ceph-mon[29807]: Detected new or changed devices on smithi184 2024-09-13T14:17:24.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:23 smithi151 ceph-mon[29807]: Adjusting osd_memory_target on smithi184 to 5696M 2024-09-13T14:17:24.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:23 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:24.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:23 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:24.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:23 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:24.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:23 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:24.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:23 smithi151 ceph-mon[41387]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:24.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:23 smithi151 ceph-mon[41387]: Detected new or changed devices on smithi184 2024-09-13T14:17:24.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:23 smithi151 ceph-mon[41387]: Adjusting osd_memory_target on smithi184 to 5696M 2024-09-13T14:17:24.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:23 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:24.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:23 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:24.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:23 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:24.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:23 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:25.069 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:24 smithi184 ceph-mon[33394]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-13T14:17:25.069 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:24 smithi184 ceph-mon[33394]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-13T14:17:25.069 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:24 smithi184 ceph-mon[33394]: Cluster is now healthy 2024-09-13T14:17:25.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:24 smithi151 ceph-mon[29807]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-13T14:17:25.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:24 smithi151 ceph-mon[29807]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-13T14:17:25.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:24 smithi151 ceph-mon[29807]: Cluster is now healthy 2024-09-13T14:17:25.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:24 smithi151 ceph-mon[41387]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-13T14:17:25.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:24 smithi151 ceph-mon[41387]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-13T14:17:25.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:24 smithi151 ceph-mon[41387]: Cluster is now healthy 2024-09-13T14:17:26.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:25 smithi184 ceph-mon[33394]: from='client.? 172.21.15.184:0/2914872875' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d76869fe-5121-4467-8b56-1aadef0eaf65"}]: dispatch 2024-09-13T14:17:26.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:25 smithi184 ceph-mon[33394]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d76869fe-5121-4467-8b56-1aadef0eaf65"}]: dispatch 2024-09-13T14:17:26.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:25 smithi184 ceph-mon[33394]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d76869fe-5121-4467-8b56-1aadef0eaf65"}]': finished 2024-09-13T14:17:26.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:25 smithi184 ceph-mon[33394]: osdmap e45: 8 total, 7 up, 8 in 2024-09-13T14:17:26.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:25 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:17:26.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:25 smithi184 ceph-mon[33394]: from='client.? 172.21.15.184:0/882947691' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:17:26.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:25 smithi151 ceph-mon[41387]: from='client.? 172.21.15.184:0/2914872875' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d76869fe-5121-4467-8b56-1aadef0eaf65"}]: dispatch 2024-09-13T14:17:26.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:25 smithi151 ceph-mon[41387]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d76869fe-5121-4467-8b56-1aadef0eaf65"}]: dispatch 2024-09-13T14:17:26.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:25 smithi151 ceph-mon[41387]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d76869fe-5121-4467-8b56-1aadef0eaf65"}]': finished 2024-09-13T14:17:26.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:25 smithi151 ceph-mon[41387]: osdmap e45: 8 total, 7 up, 8 in 2024-09-13T14:17:26.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:25 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:17:26.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:25 smithi151 ceph-mon[41387]: from='client.? 172.21.15.184:0/882947691' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:17:26.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:25 smithi151 ceph-mon[29807]: from='client.? 172.21.15.184:0/2914872875' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d76869fe-5121-4467-8b56-1aadef0eaf65"}]: dispatch 2024-09-13T14:17:26.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:25 smithi151 ceph-mon[29807]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d76869fe-5121-4467-8b56-1aadef0eaf65"}]: dispatch 2024-09-13T14:17:26.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:25 smithi151 ceph-mon[29807]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d76869fe-5121-4467-8b56-1aadef0eaf65"}]': finished 2024-09-13T14:17:26.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:25 smithi151 ceph-mon[29807]: osdmap e45: 8 total, 7 up, 8 in 2024-09-13T14:17:26.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:25 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:17:26.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:25 smithi151 ceph-mon[29807]: from='client.? 172.21.15.184:0/882947691' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:17:27.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:26 smithi184 ceph-mon[33394]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-09-13T14:17:27.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:26 smithi151 ceph-mon[41387]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-09-13T14:17:27.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:26 smithi151 ceph-mon[29807]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-09-13T14:17:28.893 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:28 smithi184 ceph-mon[33394]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T14:17:29.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:28 smithi151 ceph-mon[29807]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T14:17:29.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:28 smithi151 ceph-mon[41387]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T14:17:31.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:30 smithi184 ceph-mon[33394]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-13T14:17:31.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:30 smithi151 ceph-mon[29807]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-13T14:17:31.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:30 smithi151 ceph-mon[41387]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-13T14:17:31.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:31 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-13T14:17:31.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:31 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:31.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:31 smithi184 ceph-mon[33394]: Deploying daemon osd.7 on smithi184 2024-09-13T14:17:32.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:31 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-13T14:17:32.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:31 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:32.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:31 smithi151 ceph-mon[29807]: Deploying daemon osd.7 on smithi184 2024-09-13T14:17:32.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:31 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-13T14:17:32.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:31 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:32.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:31 smithi151 ceph-mon[41387]: Deploying daemon osd.7 on smithi184 2024-09-13T14:17:32.949 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:32 smithi184 ceph-mon[33394]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:17:33.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:32 smithi151 ceph-mon[29807]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:17:33.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:32 smithi151 ceph-mon[41387]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:17:34.966 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:34 smithi184 ceph-mon[33394]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:17:35.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:34 smithi151 ceph-mon[29807]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:17:35.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:34 smithi151 ceph-mon[41387]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:17:35.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:35 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:35.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:35 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:35.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:35 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:36.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:35 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:36.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:35 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:36.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:35 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:36.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:35 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:36.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:35 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:36.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:35 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:37.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:36 smithi184 ceph-mon[33394]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:17:37.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:36 smithi151 ceph-mon[29807]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:17:37.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:36 smithi151 ceph-mon[41387]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:17:38.947 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:38 smithi184 ceph-mon[33394]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:17:38.947 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:38 smithi184 ceph-mon[33394]: from='osd.7 [v2:172.21.15.184:6824/3378735997,v1:172.21.15.184:6825/3378735997]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T14:17:38.947 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:38 smithi184 ceph-mon[33394]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T14:17:38.947 INFO:teuthology.orchestra.run.smithi184.stdout:Created osd(s) 7 on host 'smithi184' 2024-09-13T14:17:39.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:38 smithi151 ceph-mon[29807]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:17:39.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:38 smithi151 ceph-mon[29807]: from='osd.7 [v2:172.21.15.184:6824/3378735997,v1:172.21.15.184:6825/3378735997]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T14:17:39.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:38 smithi151 ceph-mon[29807]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T14:17:39.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:38 smithi151 ceph-mon[41387]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:17:39.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:38 smithi151 ceph-mon[41387]: from='osd.7 [v2:172.21.15.184:6824/3378735997,v1:172.21.15.184:6825/3378735997]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T14:17:39.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:38 smithi151 ceph-mon[41387]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T14:17:39.622 DEBUG:teuthology.orchestra.run.smithi184:osd.7> sudo journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.7.service 2024-09-13T14:17:39.624 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-13T14:17:39.624 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-13T14:17:39.853 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 14:17:39 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T14:17:39.825+0000 7fce62d7f640 -1 osd.7 0 waiting for initial osdmap 2024-09-13T14:17:39.854 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 14:17:39 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T14:17:39.832+0000 7fce5ebb8640 -1 osd.7 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:17:39.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:39 smithi184 ceph-mon[33394]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-13T14:17:39.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:39 smithi184 ceph-mon[33394]: osdmap e46: 8 total, 7 up, 8 in 2024-09-13T14:17:39.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:39 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:17:39.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:39 smithi184 ceph-mon[33394]: from='osd.7 [v2:172.21.15.184:6824/3378735997,v1:172.21.15.184:6825/3378735997]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-13T14:17:39.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:39 smithi184 ceph-mon[33394]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-13T14:17:39.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:39 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:39.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:39 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:39.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:39 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:39.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:39 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:39.886 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:17:40.007 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:39 smithi151 ceph-mon[41387]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-13T14:17:40.007 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:39 smithi151 ceph-mon[41387]: osdmap e46: 8 total, 7 up, 8 in 2024-09-13T14:17:40.007 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:39 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:17:40.007 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:39 smithi151 ceph-mon[41387]: from='osd.7 [v2:172.21.15.184:6824/3378735997,v1:172.21.15.184:6825/3378735997]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-13T14:17:40.007 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:39 smithi151 ceph-mon[41387]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-13T14:17:40.007 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:39 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:40.007 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:39 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:40.008 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:39 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:40.008 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:39 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:40.008 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:39 smithi151 ceph-mon[29807]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-13T14:17:40.008 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:39 smithi151 ceph-mon[29807]: osdmap e46: 8 total, 7 up, 8 in 2024-09-13T14:17:40.008 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:39 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:17:40.008 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:39 smithi151 ceph-mon[29807]: from='osd.7 [v2:172.21.15.184:6824/3378735997,v1:172.21.15.184:6825/3378735997]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-13T14:17:40.008 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:39 smithi151 ceph-mon[29807]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-13T14:17:40.008 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:39 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:40.008 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:39 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:40.008 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:39 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:40.009 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:39 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:41.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:40 smithi184 ceph-mon[33394]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:17:41.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:40 smithi184 ceph-mon[33394]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-13T14:17:41.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:40 smithi184 ceph-mon[33394]: osdmap e47: 8 total, 7 up, 8 in 2024-09-13T14:17:41.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:40 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:17:41.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:40 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:17:41.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:40 smithi184 ceph-mon[33394]: osd.7 [v2:172.21.15.184:6824/3378735997,v1:172.21.15.184:6825/3378735997] boot 2024-09-13T14:17:41.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:40 smithi184 ceph-mon[33394]: osdmap e48: 8 total, 8 up, 8 in 2024-09-13T14:17:41.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:40 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:17:41.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:40 smithi151 ceph-mon[29807]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:17:41.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:40 smithi151 ceph-mon[29807]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-13T14:17:41.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:40 smithi151 ceph-mon[29807]: osdmap e47: 8 total, 7 up, 8 in 2024-09-13T14:17:41.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:40 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:17:41.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:40 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:17:41.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:40 smithi151 ceph-mon[29807]: osd.7 [v2:172.21.15.184:6824/3378735997,v1:172.21.15.184:6825/3378735997] boot 2024-09-13T14:17:41.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:40 smithi151 ceph-mon[29807]: osdmap e48: 8 total, 8 up, 8 in 2024-09-13T14:17:41.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:40 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:17:41.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:40 smithi151 ceph-mon[41387]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:17:41.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:40 smithi151 ceph-mon[41387]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-13T14:17:41.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:40 smithi151 ceph-mon[41387]: osdmap e47: 8 total, 7 up, 8 in 2024-09-13T14:17:41.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:40 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:17:41.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:40 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:17:41.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:40 smithi151 ceph-mon[41387]: osd.7 [v2:172.21.15.184:6824/3378735997,v1:172.21.15.184:6825/3378735997] boot 2024-09-13T14:17:41.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:40 smithi151 ceph-mon[41387]: osdmap e48: 8 total, 8 up, 8 in 2024-09-13T14:17:41.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:40 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:17:41.793 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:17:42.098 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:41 smithi151 ceph-mon[29807]: purged_snaps scrub starts 2024-09-13T14:17:42.098 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:41 smithi151 ceph-mon[29807]: purged_snaps scrub ok 2024-09-13T14:17:42.099 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:41 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1671853898' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-13T14:17:42.099 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:41 smithi151 ceph-mon[41387]: purged_snaps scrub starts 2024-09-13T14:17:42.099 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:41 smithi151 ceph-mon[41387]: purged_snaps scrub ok 2024-09-13T14:17:42.099 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:41 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/1671853898' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-13T14:17:42.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:41 smithi184 ceph-mon[33394]: purged_snaps scrub starts 2024-09-13T14:17:42.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:41 smithi184 ceph-mon[33394]: purged_snaps scrub ok 2024-09-13T14:17:42.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:41 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/1671853898' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-13T14:17:42.442 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":48,"num_osds":8,"num_up_osds":8,"osd_up_since":1726237060,"num_in_osds":8,"osd_in_since":1726237044,"num_remapped_pgs":1} 2024-09-13T14:17:42.443 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T14:17:42.683 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:17:43.096 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:42 smithi151 ceph-mon[29807]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:17:43.096 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:42 smithi151 ceph-mon[29807]: osdmap e49: 8 total, 8 up, 8 in 2024-09-13T14:17:43.096 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:42 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:43.096 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:42 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:43.097 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:42 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:17:43.097 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:42 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:17:43.097 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:42 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:17:43.097 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:42 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:17:43.097 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:42 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:43.097 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:42 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:43.097 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:42 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:43.097 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:42 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:43.097 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:42 smithi151 ceph-mon[41387]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:17:43.097 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:42 smithi151 ceph-mon[41387]: osdmap e49: 8 total, 8 up, 8 in 2024-09-13T14:17:43.097 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:42 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:43.097 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:42 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:43.098 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:42 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:17:43.098 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:42 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:17:43.098 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:42 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:17:43.098 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:42 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:17:43.098 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:42 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:43.098 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:42 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:43.098 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:42 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:43.098 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:42 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:42 smithi184 ceph-mon[33394]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:17:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:42 smithi184 ceph-mon[33394]: osdmap e49: 8 total, 8 up, 8 in 2024-09-13T14:17:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:42 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:42 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:42 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:17:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:42 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:17:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:42 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:17:43.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:42 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:17:43.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:42 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:43.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:42 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:43.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:42 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:43.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:42 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:43.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:43 smithi151 ceph-mon[29807]: Detected new or changed devices on smithi184 2024-09-13T14:17:43.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:43 smithi151 ceph-mon[29807]: Adjusting osd_memory_target on smithi184 to 4272M 2024-09-13T14:17:43.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:43 smithi151 ceph-mon[29807]: osdmap e50: 8 total, 8 up, 8 in 2024-09-13T14:17:43.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:43 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:43.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:43 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:43.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:43 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:43.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:43 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:43.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:43 smithi151 ceph-mon[41387]: Detected new or changed devices on smithi184 2024-09-13T14:17:43.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:43 smithi151 ceph-mon[41387]: Adjusting osd_memory_target on smithi184 to 4272M 2024-09-13T14:17:43.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:43 smithi151 ceph-mon[41387]: osdmap e50: 8 total, 8 up, 8 in 2024-09-13T14:17:43.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:43 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:43.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:43 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:43.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:43 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:43.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:43 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:44.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:43 smithi184 ceph-mon[33394]: Detected new or changed devices on smithi184 2024-09-13T14:17:44.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:43 smithi184 ceph-mon[33394]: Adjusting osd_memory_target on smithi184 to 4272M 2024-09-13T14:17:44.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:43 smithi184 ceph-mon[33394]: osdmap e50: 8 total, 8 up, 8 in 2024-09-13T14:17:44.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:43 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:44.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:43 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:44.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:43 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:44.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:43 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:44.560 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:17:44.560 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":50,"fsid":"2401fc50-71da-11ef-bceb-c7b262605968","created":"2024-09-13T14:12:36.168205+0000","modified":"2024-09-13T14:17:42.833125+0000","last_up_change":"2024-09-13T14:17:40.817515+0000","last_in_change":"2024-09-13T14:17:24.771724+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:15:41.780748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"0ad945ee-af7d-4f26-a1e9-18ed963194ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":551854295},{"type":"v1","addr":"172.21.15.151:6803","nonce":551854295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":551854295},{"type":"v1","addr":"172.21.15.151:6805","nonce":551854295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":551854295},{"type":"v1","addr":"172.21.15.151:6809","nonce":551854295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":551854295},{"type":"v1","addr":"172.21.15.151:6807","nonce":551854295}]},"public_addr":"172.21.15.151:6803/551854295","cluster_addr":"172.21.15.151:6805/551854295","heartbeat_back_addr":"172.21.15.151:6809/551854295","heartbeat_front_addr":"172.21.15.151:6807/551854295","state":["exists","up"]},{"osd":1,"uuid":"09f5d705-684f-4e86-80a2-ec2292af09d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":3674059846},{"type":"v1","addr":"172.21.15.151:6811","nonce":3674059846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3674059846},{"type":"v1","addr":"172.21.15.151:6813","nonce":3674059846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3674059846},{"type":"v1","addr":"172.21.15.151:6817","nonce":3674059846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":3674059846},{"type":"v1","addr":"172.21.15.151:6815","nonce":3674059846}]},"public_addr":"172.21.15.151:6811/3674059846","cluster_addr":"172.21.15.151:6813/3674059846","heartbeat_back_addr":"172.21.15.151:6817/3674059846","heartbeat_front_addr":"172.21.15.151:6815/3674059846","state":["exists","up"]},{"osd":2,"uuid":"6a63f8af-ff69-46bb-830f-03ea89fc78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":1646754940},{"type":"v1","addr":"172.21.15.151:6819","nonce":1646754940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":1646754940},{"type":"v1","addr":"172.21.15.151:6821","nonce":1646754940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1646754940},{"type":"v1","addr":"172.21.15.151:6825","nonce":1646754940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":1646754940},{"type":"v1","addr":"172.21.15.151:6823","nonce":1646754940}]},"public_addr":"172.21.15.151:6819/1646754940","cluster_addr":"172.21.15.151:6821/1646754940","heartbeat_back_addr":"172.21.15.151:6825/1646754940","heartbeat_front_addr":"172.21.15.151:6823/1646754940","state":["exists","up"]},{"osd":3,"uuid":"770e8334-3b3a-46bc-9782-964b2a696660","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2203980888},{"type":"v1","addr":"172.21.15.151:6827","nonce":2203980888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":2203980888},{"type":"v1","addr":"172.21.15.151:6829","nonce":2203980888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":2203980888},{"type":"v1","addr":"172.21.15.151:6833","nonce":2203980888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":2203980888},{"type":"v1","addr":"172.21.15.151:6831","nonce":2203980888}]},"public_addr":"172.21.15.151:6827/2203980888","cluster_addr":"172.21.15.151:6829/2203980888","heartbeat_back_addr":"172.21.15.151:6833/2203980888","heartbeat_front_addr":"172.21.15.151:6831/2203980888","state":["exists","up"]},{"osd":4,"uuid":"75e258c0-1145-4aee-86aa-db2b63082cc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6801","nonce":3275553935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6803","nonce":3275553935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6807","nonce":3275553935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6805","nonce":3275553935}]},"public_addr":"172.21.15.184:6801/3275553935","cluster_addr":"172.21.15.184:6803/3275553935","heartbeat_back_addr":"172.21.15.184:6807/3275553935","heartbeat_front_addr":"172.21.15.184:6805/3275553935","state":["exists","up"]},{"osd":5,"uuid":"f56a5366-9d22-4960-933d-340aa280fbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6809","nonce":2446419621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6811","nonce":2446419621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6815","nonce":2446419621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6813","nonce":2446419621}]},"public_addr":"172.21.15.184:6809/2446419621","cluster_addr":"172.21.15.184:6811/2446419621","heartbeat_back_addr":"172.21.15.184:6815/2446419621","heartbeat_front_addr":"172.21.15.184:6813/2446419621","state":["exists","up"]},{"osd":6,"uuid":"e9826568-a728-45ab-8180-9777fe31f93a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6817","nonce":2214837203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6819","nonce":2214837203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6823","nonce":2214837203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6821","nonce":2214837203}]},"public_addr":"172.21.15.184:6817/2214837203","cluster_addr":"172.21.15.184:6819/2214837203","heartbeat_back_addr":"172.21.15.184:6823/2214837203","heartbeat_front_addr":"172.21.15.184:6821/2214837203","state":["exists","up"]},{"osd":7,"uuid":"d76869fe-5121-4467-8b56-1aadef0eaf65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6825","nonce":3378735997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6827","nonce":3378735997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6831","nonce":3378735997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6829","nonce":3378735997}]},"public_addr":"172.21.15.184:6825/3378735997","cluster_addr":"172.21.15.184:6827/3378735997","heartbeat_back_addr":"172.21.15.184:6831/3378735997","heartbeat_front_addr":"172.21.15.184:6829/3378735997","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:14:49.835715+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:15:14.682882+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:15:39.423890+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:16:04.527460+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:16:27.283661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:16:51.893868+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:16.152592+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:39.763533+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.151:0/3689756337":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/2666829207":"2024-09-14T14:13:27.478847+0000","172.21.15.151:6800/1057983082":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/1427922964":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/1182681341":"2024-09-14T14:13:04.761383+0000","172.21.15.151:0/624042464":"2024-09-14T14:13:04.761383+0000","172.21.15.151:0/3946777511":"2024-09-14T14:13:04.761383+0000","172.21.15.151:6801/1057983082":"2024-09-14T14:13:27.478847+0000","172.21.15.151:6801/3383065173":"2024-09-14T14:13:04.761383+0000","172.21.15.151:6800/3383065173":"2024-09-14T14:13:04.761383+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T14:17:45.177 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:44 smithi151 ceph-mon[41387]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:17:45.177 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:44 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/2316023022' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:17:45.177 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:44 smithi151 ceph-mon[29807]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:17:45.177 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:44 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/2316023022' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:17:45.202 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T14:15:41.780748+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-09-13T14:17:45.202 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-13T14:17:45.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:44 smithi184 ceph-mon[33394]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:17:45.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:44 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/2316023022' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:17:45.442 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:17:47.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:46 smithi151 ceph-mon[29807]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:17:47.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:46 smithi151 ceph-mon[41387]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:17:47.251 INFO:teuthology.orchestra.run.smithi151.stdout:pg_num: 1 2024-09-13T14:17:47.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:46 smithi184 ceph-mon[33394]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:17:47.869 INFO:tasks.cephadm:Adding prometheus.a on smithi184 2024-09-13T14:17:47.869 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph orch apply prometheus '1;smithi184=a' 2024-09-13T14:17:48.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:47 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/622203472' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-13T14:17:48.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:47 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/622203472' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-13T14:17:48.122 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:17:48.243 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:47 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/622203472' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-13T14:17:49.218 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:48 smithi184 ceph-mon[33394]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:17:49.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:48 smithi151 ceph-mon[29807]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:17:49.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:48 smithi151 ceph-mon[41387]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:17:49.862 INFO:teuthology.orchestra.run.smithi184.stdout:Scheduled prometheus update... 2024-09-13T14:17:50.460 DEBUG:teuthology.orchestra.run.smithi184:prometheus.a> sudo journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@prometheus.a.service 2024-09-13T14:17:50.462 INFO:tasks.cephadm:Adding node-exporter.a on smithi151 2024-09-13T14:17:50.462 INFO:tasks.cephadm:Adding node-exporter.b on smithi184 2024-09-13T14:17:50.463 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph orch apply node-exporter '2;smithi151=a;smithi184=b' 2024-09-13T14:17:50.751 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:17:51.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:50 smithi184 ceph-mon[33394]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:17:51.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:50 smithi184 ceph-mon[33394]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi184=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:51.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:50 smithi184 ceph-mon[33394]: Saving service prometheus spec with placement smithi184=a;count:1 2024-09-13T14:17:51.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:50 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:51.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:50 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:51.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:50 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:51.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:50 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:51.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:50 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:51.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:50 smithi151 ceph-mon[41387]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:17:51.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:50 smithi151 ceph-mon[41387]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi184=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:51.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:50 smithi151 ceph-mon[41387]: Saving service prometheus spec with placement smithi184=a;count:1 2024-09-13T14:17:51.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:50 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:51.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:50 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:51.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:50 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:51.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:50 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:51.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:50 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:51.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:50 smithi151 ceph-mon[29807]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:17:51.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:50 smithi151 ceph-mon[29807]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi184=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:51.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:50 smithi151 ceph-mon[29807]: Saving service prometheus spec with placement smithi184=a;count:1 2024-09-13T14:17:51.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:50 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:51.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:50 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:51.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:50 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:51.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:50 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:51.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:50 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:52.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:51 smithi184 ceph-mon[33394]: Deploying daemon prometheus.a on smithi184 2024-09-13T14:17:52.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:51 smithi151 ceph-mon[41387]: Deploying daemon prometheus.a on smithi184 2024-09-13T14:17:52.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:51 smithi151 ceph-mon[29807]: Deploying daemon prometheus.a on smithi184 2024-09-13T14:17:52.956 INFO:teuthology.orchestra.run.smithi184.stdout:Scheduled node-exporter update... 2024-09-13T14:17:53.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:52 smithi184 ceph-mon[33394]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:17:53.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:52 smithi151 ceph-mon[29807]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:17:53.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:52 smithi151 ceph-mon[41387]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:17:53.958 DEBUG:teuthology.orchestra.run.smithi151:node-exporter.a> sudo journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@node-exporter.a.service 2024-09-13T14:17:53.961 DEBUG:teuthology.orchestra.run.smithi184:node-exporter.b> sudo journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@node-exporter.b.service 2024-09-13T14:17:53.963 INFO:tasks.cephadm:Adding alertmanager.a on smithi151 2024-09-13T14:17:53.964 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph orch apply alertmanager '1;smithi151=a' 2024-09-13T14:17:54.245 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:17:54.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:53 smithi184 ceph-mon[33394]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi151=a;smithi184=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:54.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:53 smithi184 ceph-mon[33394]: Saving service node-exporter spec with placement smithi151=a;smithi184=b;count:2 2024-09-13T14:17:54.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:53 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:54.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:53 smithi151 ceph-mon[29807]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi151=a;smithi184=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:54.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:53 smithi151 ceph-mon[29807]: Saving service node-exporter spec with placement smithi151=a;smithi184=b;count:2 2024-09-13T14:17:54.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:53 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:54.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:53 smithi151 ceph-mon[41387]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi151=a;smithi184=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:54.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:53 smithi151 ceph-mon[41387]: Saving service node-exporter spec with placement smithi151=a;smithi184=b;count:2 2024-09-13T14:17:54.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:53 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:55.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:54 smithi184 ceph-mon[33394]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:17:55.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:54 smithi151 ceph-mon[29807]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:17:55.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:54 smithi151 ceph-mon[41387]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:17:56.609 INFO:teuthology.orchestra.run.smithi184.stdout:Scheduled alertmanager update... 2024-09-13T14:17:57.290 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:56 smithi184 ceph-mon[33394]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:17:57.290 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:56 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:57.314 DEBUG:teuthology.orchestra.run.smithi151:alertmanager.a> sudo journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@alertmanager.a.service 2024-09-13T14:17:57.317 INFO:tasks.cephadm:Adding grafana.a on smithi184 2024-09-13T14:17:57.317 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph orch apply grafana '1;smithi184=a' 2024-09-13T14:17:57.331 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:56 smithi151 ceph-mon[29807]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:17:57.331 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:56 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:57.331 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:56 smithi151 ceph-mon[41387]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:17:57.331 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:56 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:17:57.560 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:17:58.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:57 smithi184 ceph-mon[33394]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi151=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:58.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:57 smithi184 ceph-mon[33394]: Saving service alertmanager spec with placement smithi151=a;count:1 2024-09-13T14:17:58.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:57 smithi151 ceph-mon[29807]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi151=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:58.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:57 smithi151 ceph-mon[29807]: Saving service alertmanager spec with placement smithi151=a;count:1 2024-09-13T14:17:58.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:57 smithi151 ceph-mon[41387]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi151=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:58.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:57 smithi151 ceph-mon[41387]: Saving service alertmanager spec with placement smithi151=a;count:1 2024-09-13T14:17:59.051 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:17:58 smithi184 ceph-mon[33394]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:17:59.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:17:58 smithi151 ceph-mon[41387]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:17:59.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:17:58 smithi151 ceph-mon[29807]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:17:59.461 INFO:teuthology.orchestra.run.smithi184.stdout:Scheduled grafana update... 2024-09-13T14:18:00.105 DEBUG:teuthology.orchestra.run.smithi184:grafana.a> sudo journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@grafana.a.service 2024-09-13T14:18:00.108 INFO:tasks.cephadm:Setting up client nodes... 2024-09-13T14:18:00.108 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-13T14:18:00.364 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:18:00.758 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:00 smithi151 ceph-mon[41387]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi184=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:18:00.758 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:00 smithi151 ceph-mon[41387]: Saving service grafana spec with placement smithi184=a;count:1 2024-09-13T14:18:00.758 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:00 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:18:00.758 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:00 smithi151 ceph-mon[41387]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:00.758 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:00 smithi151 ceph-mon[29807]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi184=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:18:00.758 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:00 smithi151 ceph-mon[29807]: Saving service grafana spec with placement smithi184=a;count:1 2024-09-13T14:18:00.758 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:00 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:18:00.758 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:00 smithi151 ceph-mon[29807]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:00.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:00 smithi184 ceph-mon[33394]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi184=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:18:00.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:00 smithi184 ceph-mon[33394]: Saving service grafana spec with placement smithi184=a;count:1 2024-09-13T14:18:00.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:00 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:18:00.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:00 smithi184 ceph-mon[33394]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:02.347 INFO:teuthology.orchestra.run.smithi151.stdout:[client.0] 2024-09-13T14:18:02.347 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCaSeRmoodnFBAAu4+IWbvedE8hms3NIDEwAA== 2024-09-13T14:18:02.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:02 smithi151 ceph-mon[41387]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:02.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:02 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/3952921008' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:18:02.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:02 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/3952921008' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T14:18:02.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:02 smithi151 ceph-mon[29807]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:02.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:02 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/3952921008' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:18:02.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:02 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/3952921008' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T14:18:02.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:02 smithi184 ceph-mon[33394]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:02.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:02 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/3952921008' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:18:02.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:02 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/3952921008' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T14:18:02.964 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-13T14:18:02.964 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-13T14:18:02.964 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-13T14:18:03.004 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-13T14:18:03.260 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:18:04.740 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:04 smithi184 ceph-mon[33394]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:04.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:04 smithi151 ceph-mon[29807]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:04.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:04 smithi151 ceph-mon[41387]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:05.503 INFO:teuthology.orchestra.run.smithi184.stdout:[client.1] 2024-09-13T14:18:05.504 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQCdSeRmDcrDHRAAHoRdVe/j0r71ONkVnUlIDA== 2024-09-13T14:18:05.846 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:05 smithi184 ceph-mon[33394]: from='client.? 172.21.15.184:0/920401931' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:18:05.846 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:05 smithi184 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:18:05.846 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:05 smithi184 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T14:18:05.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:05 smithi151 ceph-mon[29807]: from='client.? 172.21.15.184:0/920401931' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:18:05.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:05 smithi151 ceph-mon[29807]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:18:05.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:05 smithi151 ceph-mon[29807]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T14:18:05.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:05 smithi151 ceph-mon[41387]: from='client.? 172.21.15.184:0/920401931' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:18:05.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:05 smithi151 ceph-mon[41387]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:18:05.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:05 smithi151 ceph-mon[41387]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T14:18:06.624 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-13T14:18:06.624 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-13T14:18:06.624 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-13T14:18:06.664 INFO:tasks.cephadm:Creating RBD pool 2024-09-13T14:18:06.664 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool create rbd 8 2024-09-13T14:18:06.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:06 smithi184 ceph-mon[33394]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:06.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:06 smithi151 ceph-mon[29807]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:06.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:06 smithi151 ceph-mon[41387]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:06.917 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:18:08.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:08 smithi184 ceph-mon[33394]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:08.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:08 smithi151 ceph-mon[29807]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:08.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:08 smithi151 ceph-mon[41387]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:09.610 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'rbd' created 2024-09-13T14:18:09.678 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:09 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/61865864' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-13T14:18:09.678 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:09 smithi184 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-13T14:18:09.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:09 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/61865864' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-13T14:18:09.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:09 smithi151 ceph-mon[29807]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-13T14:18:09.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:09 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/61865864' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-13T14:18:09.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:09 smithi151 ceph-mon[41387]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-13T14:18:10.314 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- rbd pool init rbd 2024-09-13T14:18:10.560 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:18:10.869 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:10 smithi151 ceph-mon[29807]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:10.869 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:10 smithi151 ceph-mon[29807]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-09-13T14:18:10.869 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:10 smithi151 ceph-mon[29807]: osdmap e51: 8 total, 8 up, 8 in 2024-09-13T14:18:10.869 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:10 smithi151 ceph-mon[41387]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:10.869 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:10 smithi151 ceph-mon[41387]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-09-13T14:18:10.870 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:10 smithi151 ceph-mon[41387]: osdmap e51: 8 total, 8 up, 8 in 2024-09-13T14:18:10.936 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:10 smithi184 ceph-mon[33394]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:10.936 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:10 smithi184 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-09-13T14:18:10.936 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:10 smithi184 ceph-mon[33394]: osdmap e51: 8 total, 8 up, 8 in 2024-09-13T14:18:11.698 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:11 smithi184 ceph-mon[33394]: osdmap e52: 8 total, 8 up, 8 in 2024-09-13T14:18:11.743 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:11 smithi151 ceph-mon[29807]: osdmap e52: 8 total, 8 up, 8 in 2024-09-13T14:18:11.743 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:11 smithi151 ceph-mon[41387]: osdmap e52: 8 total, 8 up, 8 in 2024-09-13T14:18:12.354 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:18:12 smithi184 podman[62919]: 2024-09-13 14:18:12.34180858 +0000 UTC m=+0.129869702 container create 66ca574fe455da469e6bad66c813569bc44f1cb745339f7b542020aaa780ebf9 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:18:12.636 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:12 smithi184 ceph-mon[33394]: pgmap v184: 9 pgs: 3 active+clean, 6 unknown; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:12.637 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:12 smithi184 ceph-mon[33394]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:18:12.637 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:12 smithi184 ceph-mon[33394]: osdmap e53: 8 total, 8 up, 8 in 2024-09-13T14:18:12.637 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:12 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/3831560250' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-13T14:18:12.637 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:12 smithi184 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-13T14:18:12.637 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:18:12 smithi184 podman[62919]: 2024-09-13 14:18:12.602446945 +0000 UTC m=+0.390508069 container init 66ca574fe455da469e6bad66c813569bc44f1cb745339f7b542020aaa780ebf9 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:18:12.637 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:18:12 smithi184 podman[62919]: 2024-09-13 14:18:12.607794672 +0000 UTC m=+0.395855794 container start 66ca574fe455da469e6bad66c813569bc44f1cb745339f7b542020aaa780ebf9 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:18:12.637 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:18:12 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:18:12.635Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-13T14:18:12.637 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:18:12 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:18:12.635Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-13T14:18:12.637 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:18:12 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:18:12.635Z caller=main.go:570 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi184 (none))" 2024-09-13T14:18:12.638 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:18:12 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:18:12.635Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-13T14:18:12.638 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:18:12 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:18:12.635Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-13T14:18:12.638 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:18:12 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:18:12.636Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-13T14:18:13.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:18:12 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:18:12.637Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-13T14:18:13.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:18:12 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:18:12.639Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-13T14:18:13.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:18:12 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:18:12.639Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-13T14:18:13.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:18:12 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:18:12.640Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-13T14:18:13.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:18:12 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:18:12.640Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.525µs 2024-09-13T14:18:13.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:18:12 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:18:12.640Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-13T14:18:13.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:18:12 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:18:12.640Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-09-13T14:18:13.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:18:12 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:18:12.640Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=17.227µs wal_replay_duration=182.628µs wbl_replay_duration=84ns total_replay_duration=217.822µs 2024-09-13T14:18:13.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:18:12 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:18:12.641Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-13T14:18:13.105 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:18:12 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:18:12.641Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-13T14:18:13.105 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:18:12 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:18:12.641Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-13T14:18:13.105 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:18:12 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:18:12.657Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.077059ms db_storage=664ns remote_storage=1µs web_handler=377ns query_engine=1µs scrape=389.087µs scrape_sd=17.267µs notify=891ns notify_sd=1.99µs rules=15.468353ms tracing=6.812µs 2024-09-13T14:18:13.105 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:18:12 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:18:12.657Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-13T14:18:13.105 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:18:12 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:18:12.657Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-13T14:18:13.105 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:18:12 smithi184 bash[62919]: 66ca574fe455da469e6bad66c813569bc44f1cb745339f7b542020aaa780ebf9 2024-09-13T14:18:13.105 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:18:12 smithi184 systemd[1]: Started Ceph prometheus.a for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:18:13.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:12 smithi151 ceph-mon[29807]: pgmap v184: 9 pgs: 3 active+clean, 6 unknown; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:13.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:12 smithi151 ceph-mon[29807]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:18:13.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:12 smithi151 ceph-mon[29807]: osdmap e53: 8 total, 8 up, 8 in 2024-09-13T14:18:13.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:12 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/3831560250' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-13T14:18:13.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:12 smithi151 ceph-mon[29807]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-13T14:18:13.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:12 smithi151 ceph-mon[41387]: pgmap v184: 9 pgs: 3 active+clean, 6 unknown; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:13.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:12 smithi151 ceph-mon[41387]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:18:13.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:12 smithi151 ceph-mon[41387]: osdmap e53: 8 total, 8 up, 8 in 2024-09-13T14:18:13.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:12 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/3831560250' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-13T14:18:13.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:12 smithi151 ceph-mon[41387]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-13T14:18:13.939 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:13 smithi184 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-09-13T14:18:13.939 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:13 smithi184 ceph-mon[33394]: osdmap e54: 8 total, 8 up, 8 in 2024-09-13T14:18:13.939 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:13 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:18:13.939 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:13 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:18:13.939 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:13 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:18:13.940 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:13 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-13T14:18:13.940 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:13 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: ignoring --setuser ceph since I am not root 2024-09-13T14:18:13.940 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:13 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: ignoring --setgroup ceph since I am not root 2024-09-13T14:18:13.940 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:13 smithi184 ceph-mgr[35471]: -- 172.21.15.184:0/2428242993 <== mon.1 v2:172.21.15.184:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f2caa6e340 con 0x55f2caa7b000 2024-09-13T14:18:13.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:13 smithi151 ceph-mon[29807]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-09-13T14:18:13.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:13 smithi151 ceph-mon[29807]: osdmap e54: 8 total, 8 up, 8 in 2024-09-13T14:18:13.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:13 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:18:13.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:13 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:18:13.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:13 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:18:13.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:13 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-13T14:18:13.942 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:13 smithi151 ceph-mon[41387]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-09-13T14:18:13.942 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:13 smithi151 ceph-mon[41387]: osdmap e54: 8 total, 8 up, 8 in 2024-09-13T14:18:13.942 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:13 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:18:13.942 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:13 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:18:13.942 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:13 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' 2024-09-13T14:18:13.942 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:13 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-13T14:18:13.942 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:13 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: ignoring --setuser ceph since I am not root 2024-09-13T14:18:13.942 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:13 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: ignoring --setgroup ceph since I am not root 2024-09-13T14:18:13.943 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:13 smithi151 ceph-mgr[30079]: -- 172.21.15.151:0/3855599428 <== mon.2 v2:172.21.15.151:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cfddab2340 con 0x55cfddabec00 2024-09-13T14:18:13.943 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:13 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:18:13.940+0000 7fdc2181aec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T14:18:14.319 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:13 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:18:13.937+0000 7f0590713ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T14:18:14.319 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:13 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:18:13.992+0000 7f0590713ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T14:18:14.319 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:14 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:18:14.057+0000 7f0590713ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T14:18:14.319 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:14 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:18:14.181+0000 7f0590713ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:18:14.319 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:14 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:18:14.318+0000 7f0590713ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T14:18:14.320 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:13 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:18:13.995+0000 7fdc2181aec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T14:18:14.320 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:18:14.060+0000 7fdc2181aec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T14:18:14.320 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:18:14.183+0000 7fdc2181aec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:18:14.321 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:18:14.319+0000 7fdc2181aec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T14:18:14.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:14 smithi184 ceph-mon[33394]: osdmap e55: 8 total, 8 up, 8 in 2024-09-13T14:18:14.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:14 smithi184 ceph-mon[33394]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:18:14.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:14 smithi184 ceph-mon[33394]: Cluster is now healthy 2024-09-13T14:18:14.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:14 smithi184 ceph-mon[33394]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-13T14:18:14.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:14 smithi184 ceph-mon[33394]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-13T14:18:14.854 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:14 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:18:14.571+0000 7f0590713ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T14:18:14.854 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:14 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:18:14.630+0000 7f0590713ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T14:18:14.854 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:14 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:18:14.795+0000 7f0590713ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T14:18:14.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:14 smithi151 ceph-mon[41387]: osdmap e55: 8 total, 8 up, 8 in 2024-09-13T14:18:14.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:14 smithi151 ceph-mon[41387]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:18:14.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:14 smithi151 ceph-mon[41387]: Cluster is now healthy 2024-09-13T14:18:14.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:14 smithi151 ceph-mon[41387]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-13T14:18:14.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:14 smithi151 ceph-mon[41387]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-13T14:18:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:14 smithi151 ceph-mon[29807]: osdmap e55: 8 total, 8 up, 8 in 2024-09-13T14:18:14.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:14 smithi151 ceph-mon[29807]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:18:14.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:14 smithi151 ceph-mon[29807]: Cluster is now healthy 2024-09-13T14:18:14.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:14 smithi151 ceph-mon[29807]: from='mgr.14148 172.21.15.151:0/2885815773' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-13T14:18:14.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:14 smithi151 ceph-mon[29807]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-13T14:18:14.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:18:14.579+0000 7fdc2181aec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T14:18:14.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:18:14.641+0000 7fdc2181aec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T14:18:14.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:18:14.807+0000 7fdc2181aec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T14:18:15.421 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph config log 1 --format=json 2024-09-13T14:18:15.678 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:18:15.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:15 smithi151 ceph-mon[29807]: osdmap e56: 8 total, 8 up, 8 in 2024-09-13T14:18:15.700 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:15 smithi151 ceph-mon[41387]: osdmap e56: 8 total, 8 up, 8 in 2024-09-13T14:18:15.962 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:15 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:18:15.698+0000 7fdc2181aec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T14:18:15.962 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:15 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:18:15.765+0000 7fdc2181aec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T14:18:16.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:15 smithi184 ceph-mon[33394]: osdmap e56: 8 total, 8 up, 8 in 2024-09-13T14:18:16.104 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:18:15.634+0000 7f0590713ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T14:18:16.104 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:18:15.699+0000 7f0590713ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T14:18:16.694 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:16 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:18:16.401+0000 7f0590713ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T14:18:16.694 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:16 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:18:16.513+0000 7f0590713ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T14:18:16.694 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:16 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:18:16.633+0000 7f0590713ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:18:16.810 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:16 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:18:16.511+0000 7fdc2181aec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T14:18:16.810 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:16 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:18:16.626+0000 7fdc2181aec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T14:18:16.810 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:16 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:18:16.748+0000 7fdc2181aec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:18:16.810 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:16 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:18:16.809+0000 7fdc2181aec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T14:18:16.945 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:16 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:18:16.692+0000 7f0590713ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T14:18:16.945 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:16 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:18:16.820+0000 7f0590713ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T14:18:16.945 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:16 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:18:16.886+0000 7f0590713ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T14:18:16.946 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:16 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:18:16.944+0000 7f0590713ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T14:18:17.078 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:16 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:18:16.948+0000 7fdc2181aec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T14:18:17.079 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:17 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:18:17.017+0000 7fdc2181aec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T14:18:17.079 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:17 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:18:17.077+0000 7fdc2181aec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T14:18:17.206 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:18:17.078+0000 7f0590713ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T14:18:17.206 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:18:17.140+0000 7f0590713ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T14:18:17.335 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:17 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:18:17.221+0000 7fdc2181aec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T14:18:17.335 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:17 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:18:17.288+0000 7fdc2181aec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T14:18:17.595 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:18:17.595 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:17 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:18:17.357+0000 7fdc2181aec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T14:18:17.596 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:17 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:18:17.444+0000 7fdc2181aec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T14:18:17.596 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:17 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:18:17.504+0000 7fdc2181aec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T14:18:17.596 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:17 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:18:17.563+0000 7fdc2181aec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T14:18:17.603 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:18:17.205+0000 7f0590713ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T14:18:17.603 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:18:17.285+0000 7f0590713ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T14:18:17.603 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:18:17.343+0000 7f0590713ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T14:18:17.603 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:18:17.400+0000 7f0590713ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T14:18:17.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:17 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/2202713658' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-13T14:18:17.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:17 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/2202713658' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-13T14:18:18.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:17 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/2202713658' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-13T14:18:18.420 INFO:teuthology.orchestra.run.smithi151.stdout:[{"version":17,"timestamp":"2024-09-13T14:17:42.814925+0000","name":"","changes":[{"name":"osd/host:smithi184/osd_memory_target","previous_value":"5972733952","new_value":"4479550464"}]}] 2024-09-13T14:18:18.421 INFO:tasks.ceph_manager:config epoch is 17 2024-09-13T14:18:18.421 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-13T14:18:18.421 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-13T14:18:18.421 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-13T14:18:18.603 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:18:18.146+0000 7f0590713ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T14:18:18.604 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: [13/Sep/2024:14:18:18] ENGINE Bus STARTING 2024-09-13T14:18:18.604 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: CherryPy Checker: 2024-09-13T14:18:18.604 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: The Application mounted at '' has an empty config. 2024-09-13T14:18:18.604 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: 2024-09-13T14:18:18.604 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: [13/Sep/2024:14:18:18] ENGINE Serving on http://:::9283 2024-09-13T14:18:18.604 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: [13/Sep/2024:14:18:18] ENGINE Bus STARTED 2024-09-13T14:18:18.608 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:18:18.342+0000 7fdc2181aec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T14:18:18.608 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:18:18] ENGINE Bus STARTING 2024-09-13T14:18:18.608 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: CherryPy Checker: 2024-09-13T14:18:18.608 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: The Application mounted at '' has an empty config. 2024-09-13T14:18:18.608 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:18:18.608 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:18:18] ENGINE Serving on http://:::9283 2024-09-13T14:18:18.608 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:18:18] ENGINE Bus STARTED 2024-09-13T14:18:18.685 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:18:18.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: Standby manager daemon x restarted 2024-09-13T14:18:18.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: Standby manager daemon x started 2024-09-13T14:18:18.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: from='mgr.? 172.21.15.184:0/4186441932' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:18:18.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: from='mgr.? 172.21.15.184:0/4186441932' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:18:18.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: from='mgr.? 172.21.15.184:0/4186441932' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:18:18.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: from='mgr.? 172.21.15.184:0/4186441932' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:18:18.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: Active manager daemon y restarted 2024-09-13T14:18:18.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: Activating manager daemon y 2024-09-13T14:18:18.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: osdmap e57: 8 total, 8 up, 8 in 2024-09-13T14:18:18.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: mgrmap e17: y(active, starting, since 0.00633246s), standbys: x 2024-09-13T14:18:18.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:18:18.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:18:18.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:18:18.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:18:18.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:18:18.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:18:18.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:18:18.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:18:18.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:18:18.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:18:18.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:18:18.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:18:18.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:18:18.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:18:18.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:18:18.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:18:18.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: Manager daemon y is now available 2024-09-13T14:18:18.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:18:18.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:18.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:18:18.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:18:18.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:18:18.952 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: Standby manager daemon x restarted 2024-09-13T14:18:18.952 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: Standby manager daemon x started 2024-09-13T14:18:18.952 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.184:0/4186441932' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:18:18.952 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.184:0/4186441932' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:18:18.952 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.184:0/4186441932' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:18:18.952 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.184:0/4186441932' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:18:18.952 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: Active manager daemon y restarted 2024-09-13T14:18:18.952 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: Activating manager daemon y 2024-09-13T14:18:18.952 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: osdmap e57: 8 total, 8 up, 8 in 2024-09-13T14:18:18.952 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: mgrmap e17: y(active, starting, since 0.00633246s), standbys: x 2024-09-13T14:18:18.952 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:18:18.952 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:18:18.952 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:18:18.952 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:18:18.953 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:18:18.953 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:18:18.953 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:18:18.953 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:18:18.953 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:18:18.953 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:18:18.953 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:18:18.953 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:18:18.953 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:18:18.953 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:18:18.953 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:18:18.953 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:18:18.953 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: Manager daemon y is now available 2024-09-13T14:18:18.953 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:18:18.953 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:18.954 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:18:18.954 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:18:18.954 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:18 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:18:19.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: Standby manager daemon x restarted 2024-09-13T14:18:19.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: Standby manager daemon x started 2024-09-13T14:18:19.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.184:0/4186441932' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:18:19.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.184:0/4186441932' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:18:19.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.184:0/4186441932' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:18:19.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.184:0/4186441932' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:18:19.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: Active manager daemon y restarted 2024-09-13T14:18:19.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: Activating manager daemon y 2024-09-13T14:18:19.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: osdmap e57: 8 total, 8 up, 8 in 2024-09-13T14:18:19.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: mgrmap e17: y(active, starting, since 0.00633246s), standbys: x 2024-09-13T14:18:19.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:18:19.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:18:19.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:18:19.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:18:19.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:18:19.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:18:19.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:18:19.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:18:19.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:18:19.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:18:19.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:18:19.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:18:19.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:18:19.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:18:19.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:18:19.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:18:19.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: Manager daemon y is now available 2024-09-13T14:18:19.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:18:19.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:19.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:18:19.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:18:19.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:18 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:18:19.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:19 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:18:19.701+0000 7fdbc45dd640 -1 log_channel(cephadm) log [ERR] : [13/Sep/2024:14:18:19] ENGINE Error in HTTPServer.serve 2024-09-13T14:18:19.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:19 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: Traceback (most recent call last): 2024-09-13T14:18:19.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:19 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:18:19.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:19 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: self._connections.run(self.expiration_interval) 2024-09-13T14:18:19.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:19 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:18:19.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:19 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: self._run(expiration_interval) 2024-09-13T14:18:19.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:19 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:18:19.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:19 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:18:19.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:19 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:18:19.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:19 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:18:19.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:19 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:18:19.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:19 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: s = self.context.wrap_socket( 2024-09-13T14:18:19.860 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:19 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:18:19.860 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:19 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: return self.sslsocket_class._create( 2024-09-13T14:18:19.860 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:19 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:18:19.860 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:19 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: self.do_handshake() 2024-09-13T14:18:19.860 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:19 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:18:19.860 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:19 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: self._sslobj.do_handshake() 2024-09-13T14:18:19.860 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:19 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:18:19.860 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:19 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:18:20.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:20 smithi184 ceph-mon[33394]: mgrmap e18: y(active, since 1.01118s), standbys: x 2024-09-13T14:18:20.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:20 smithi184 ceph-mon[33394]: [13/Sep/2024:14:18:19] ENGINE Bus STARTING 2024-09-13T14:18:20.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:20 smithi184 ceph-mon[33394]: [13/Sep/2024:14:18:19] ENGINE Serving on https://172.21.15.151:7150 2024-09-13T14:18:20.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:20 smithi184 ceph-mon[33394]: [13/Sep/2024:14:18:19] ENGINE Bus STARTED 2024-09-13T14:18:20.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:20 smithi184 ceph-mon[33394]: [13/Sep/2024:14:18:19] ENGINE Error in HTTPServer.serve 2024-09-13T14:18:20.604 INFO:journalctl@ceph.mon.b.smithi184.stdout: Traceback (most recent call last): 2024-09-13T14:18:20.604 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:18:20.604 INFO:journalctl@ceph.mon.b.smithi184.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:18:20.604 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:18:20.605 INFO:journalctl@ceph.mon.b.smithi184.stdout: self._run(expiration_interval) 2024-09-13T14:18:20.605 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:18:20.605 INFO:journalctl@ceph.mon.b.smithi184.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:18:20.605 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:18:20.605 INFO:journalctl@ceph.mon.b.smithi184.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:18:20.605 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:18:20.605 INFO:journalctl@ceph.mon.b.smithi184.stdout: s = self.context.wrap_socket( 2024-09-13T14:18:20.605 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:18:20.605 INFO:journalctl@ceph.mon.b.smithi184.stdout: return self.sslsocket_class._create( 2024-09-13T14:18:20.605 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:18:20.605 INFO:journalctl@ceph.mon.b.smithi184.stdout: self.do_handshake() 2024-09-13T14:18:20.605 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:18:20.605 INFO:journalctl@ceph.mon.b.smithi184.stdout: self._sslobj.do_handshake() 2024-09-13T14:18:20.606 INFO:journalctl@ceph.mon.b.smithi184.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:18:20.606 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:20 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:20.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:20 smithi151 ceph-mon[29807]: mgrmap e18: y(active, since 1.01118s), standbys: x 2024-09-13T14:18:20.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:20 smithi151 ceph-mon[29807]: [13/Sep/2024:14:18:19] ENGINE Bus STARTING 2024-09-13T14:18:20.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:20 smithi151 ceph-mon[29807]: [13/Sep/2024:14:18:19] ENGINE Serving on https://172.21.15.151:7150 2024-09-13T14:18:20.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:20 smithi151 ceph-mon[29807]: [13/Sep/2024:14:18:19] ENGINE Bus STARTED 2024-09-13T14:18:20.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:20 smithi151 ceph-mon[29807]: [13/Sep/2024:14:18:19] ENGINE Error in HTTPServer.serve 2024-09-13T14:18:20.608 INFO:journalctl@ceph.mon.a.smithi151.stdout: Traceback (most recent call last): 2024-09-13T14:18:20.608 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:18:20.608 INFO:journalctl@ceph.mon.a.smithi151.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:18:20.609 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:18:20.609 INFO:journalctl@ceph.mon.a.smithi151.stdout: self._run(expiration_interval) 2024-09-13T14:18:20.609 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:18:20.609 INFO:journalctl@ceph.mon.a.smithi151.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:18:20.609 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:18:20.609 INFO:journalctl@ceph.mon.a.smithi151.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:18:20.609 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:18:20.609 INFO:journalctl@ceph.mon.a.smithi151.stdout: s = self.context.wrap_socket( 2024-09-13T14:18:20.609 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:18:20.609 INFO:journalctl@ceph.mon.a.smithi151.stdout: return self.sslsocket_class._create( 2024-09-13T14:18:20.609 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:18:20.609 INFO:journalctl@ceph.mon.a.smithi151.stdout: self.do_handshake() 2024-09-13T14:18:20.609 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:18:20.609 INFO:journalctl@ceph.mon.a.smithi151.stdout: self._sslobj.do_handshake() 2024-09-13T14:18:20.610 INFO:journalctl@ceph.mon.a.smithi151.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:18:20.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:20 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:20.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:20 smithi151 ceph-mon[41387]: mgrmap e18: y(active, since 1.01118s), standbys: x 2024-09-13T14:18:20.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:20 smithi151 ceph-mon[41387]: [13/Sep/2024:14:18:19] ENGINE Bus STARTING 2024-09-13T14:18:20.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:20 smithi151 ceph-mon[41387]: [13/Sep/2024:14:18:19] ENGINE Serving on https://172.21.15.151:7150 2024-09-13T14:18:20.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:20 smithi151 ceph-mon[41387]: [13/Sep/2024:14:18:19] ENGINE Bus STARTED 2024-09-13T14:18:20.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:20 smithi151 ceph-mon[41387]: [13/Sep/2024:14:18:19] ENGINE Error in HTTPServer.serve 2024-09-13T14:18:20.610 INFO:journalctl@ceph.mon.c.smithi151.stdout: Traceback (most recent call last): 2024-09-13T14:18:20.610 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:18:20.610 INFO:journalctl@ceph.mon.c.smithi151.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:18:20.610 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:18:20.610 INFO:journalctl@ceph.mon.c.smithi151.stdout: self._run(expiration_interval) 2024-09-13T14:18:20.610 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:18:20.610 INFO:journalctl@ceph.mon.c.smithi151.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:18:20.611 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:18:20.611 INFO:journalctl@ceph.mon.c.smithi151.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:18:20.611 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:18:20.611 INFO:journalctl@ceph.mon.c.smithi151.stdout: s = self.context.wrap_socket( 2024-09-13T14:18:20.611 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:18:20.611 INFO:journalctl@ceph.mon.c.smithi151.stdout: return self.sslsocket_class._create( 2024-09-13T14:18:20.611 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:18:20.611 INFO:journalctl@ceph.mon.c.smithi151.stdout: self.do_handshake() 2024-09-13T14:18:20.611 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:18:20.611 INFO:journalctl@ceph.mon.c.smithi151.stdout: self._sslobj.do_handshake() 2024-09-13T14:18:20.611 INFO:journalctl@ceph.mon.c.smithi151.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:18:20.611 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:20 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:20.623 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:18:21.496 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":18,"flags":0,"active_gid":14460,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":4025698662},{"type":"v1","addr":"172.21.15.151:6801","nonce":4025698662}]},"active_addr":"172.21.15.151:6801/4025698662","active_change":"2024-09-13T14:18:18.344950+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":24352,"name":"x","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":57,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":2201280183}]}]} 2024-09-13T14:18:21.503 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-13T14:18:21.503 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-13T14:18:21.503 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T14:18:21.641 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:21 smithi151 ceph-mon[29807]: pgmap v4: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:21.641 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:21 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/3028553652' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-13T14:18:21.641 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:21 smithi151 ceph-mon[29807]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-13T14:18:21.641 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:21 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:21.642 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:21 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:21.642 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:21 smithi151 ceph-mon[41387]: pgmap v4: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:21.642 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:21 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/3028553652' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-13T14:18:21.642 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:21 smithi151 ceph-mon[41387]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-13T14:18:21.642 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:21 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:21.642 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:21 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:21.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:21 smithi184 ceph-mon[33394]: pgmap v4: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:21.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:21 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/3028553652' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-13T14:18:21.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:21 smithi184 ceph-mon[33394]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-13T14:18:21.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:21 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:21.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:21 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:22.084 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:18:22.498 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:22 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:22.499 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:22 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:22.499 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:22 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:22.499 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:22 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:22.499 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:22 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:22.499 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:22 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:22.499 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:22 smithi151 ceph-mon[29807]: Adjusting osd_memory_target on smithi184 to 4016M 2024-09-13T14:18:22.499 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:22 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:22.499 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:22 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:22.499 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:22 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:22.499 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:22 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:22.500 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:22 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:22.500 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:22 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:22.500 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:22 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:22.500 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:22 smithi151 ceph-mon[41387]: Adjusting osd_memory_target on smithi184 to 4016M 2024-09-13T14:18:22.500 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:22 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:22.649 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:22 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:22.649 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:22 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:22.649 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:22 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:22.649 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:22 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:22.649 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:22 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:22.650 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:22 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:18:22.650 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:22 smithi184 ceph-mon[33394]: Adjusting osd_memory_target on smithi184 to 4016M 2024-09-13T14:18:22.650 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:22 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:23.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:23 smithi184 ceph-mon[33394]: pgmap v5: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:23.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:23 smithi184 ceph-mon[33394]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-13T14:18:23.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:23 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:23.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:23 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:23.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:23 smithi184 ceph-mon[33394]: osdmap e58: 8 total, 8 up, 8 in 2024-09-13T14:18:23.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:23 smithi151 ceph-mon[41387]: pgmap v5: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:23.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:23 smithi151 ceph-mon[41387]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-13T14:18:23.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:23 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:23.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:23 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:23.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:23 smithi151 ceph-mon[41387]: osdmap e58: 8 total, 8 up, 8 in 2024-09-13T14:18:23.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:23 smithi151 ceph-mon[29807]: pgmap v5: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:23.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:23 smithi151 ceph-mon[29807]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-13T14:18:23.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:23 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:23.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:23 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:23.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:23 smithi151 ceph-mon[29807]: osdmap e58: 8 total, 8 up, 8 in 2024-09-13T14:18:24.009 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:18:24.009 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":58,"fsid":"2401fc50-71da-11ef-bceb-c7b262605968","created":"2024-09-13T14:12:36.168205+0000","modified":"2024-09-13T14:18:22.740022+0000","last_up_change":"2024-09-13T14:17:40.817515+0000","last_in_change":"2024-09-13T14:17:24.771724+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:15:41.780748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:18:09.030957+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":56,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"0ad945ee-af7d-4f26-a1e9-18ed963194ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":551854295},{"type":"v1","addr":"172.21.15.151:6803","nonce":551854295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":551854295},{"type":"v1","addr":"172.21.15.151:6805","nonce":551854295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":551854295},{"type":"v1","addr":"172.21.15.151:6809","nonce":551854295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":551854295},{"type":"v1","addr":"172.21.15.151:6807","nonce":551854295}]},"public_addr":"172.21.15.151:6803/551854295","cluster_addr":"172.21.15.151:6805/551854295","heartbeat_back_addr":"172.21.15.151:6809/551854295","heartbeat_front_addr":"172.21.15.151:6807/551854295","state":["exists","up"]},{"osd":1,"uuid":"09f5d705-684f-4e86-80a2-ec2292af09d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":3674059846},{"type":"v1","addr":"172.21.15.151:6811","nonce":3674059846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3674059846},{"type":"v1","addr":"172.21.15.151:6813","nonce":3674059846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3674059846},{"type":"v1","addr":"172.21.15.151:6817","nonce":3674059846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":3674059846},{"type":"v1","addr":"172.21.15.151:6815","nonce":3674059846}]},"public_addr":"172.21.15.151:6811/3674059846","cluster_addr":"172.21.15.151:6813/3674059846","heartbeat_back_addr":"172.21.15.151:6817/3674059846","heartbeat_front_addr":"172.21.15.151:6815/3674059846","state":["exists","up"]},{"osd":2,"uuid":"6a63f8af-ff69-46bb-830f-03ea89fc78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":1646754940},{"type":"v1","addr":"172.21.15.151:6819","nonce":1646754940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":1646754940},{"type":"v1","addr":"172.21.15.151:6821","nonce":1646754940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1646754940},{"type":"v1","addr":"172.21.15.151:6825","nonce":1646754940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":1646754940},{"type":"v1","addr":"172.21.15.151:6823","nonce":1646754940}]},"public_addr":"172.21.15.151:6819/1646754940","cluster_addr":"172.21.15.151:6821/1646754940","heartbeat_back_addr":"172.21.15.151:6825/1646754940","heartbeat_front_addr":"172.21.15.151:6823/1646754940","state":["exists","up"]},{"osd":3,"uuid":"770e8334-3b3a-46bc-9782-964b2a696660","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2203980888},{"type":"v1","addr":"172.21.15.151:6827","nonce":2203980888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":2203980888},{"type":"v1","addr":"172.21.15.151:6829","nonce":2203980888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":2203980888},{"type":"v1","addr":"172.21.15.151:6833","nonce":2203980888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":2203980888},{"type":"v1","addr":"172.21.15.151:6831","nonce":2203980888}]},"public_addr":"172.21.15.151:6827/2203980888","cluster_addr":"172.21.15.151:6829/2203980888","heartbeat_back_addr":"172.21.15.151:6833/2203980888","heartbeat_front_addr":"172.21.15.151:6831/2203980888","state":["exists","up"]},{"osd":4,"uuid":"75e258c0-1145-4aee-86aa-db2b63082cc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6801","nonce":3275553935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6803","nonce":3275553935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6807","nonce":3275553935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6805","nonce":3275553935}]},"public_addr":"172.21.15.184:6801/3275553935","cluster_addr":"172.21.15.184:6803/3275553935","heartbeat_back_addr":"172.21.15.184:6807/3275553935","heartbeat_front_addr":"172.21.15.184:6805/3275553935","state":["exists","up"]},{"osd":5,"uuid":"f56a5366-9d22-4960-933d-340aa280fbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6809","nonce":2446419621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6811","nonce":2446419621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6815","nonce":2446419621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6813","nonce":2446419621}]},"public_addr":"172.21.15.184:6809/2446419621","cluster_addr":"172.21.15.184:6811/2446419621","heartbeat_back_addr":"172.21.15.184:6815/2446419621","heartbeat_front_addr":"172.21.15.184:6813/2446419621","state":["exists","up"]},{"osd":6,"uuid":"e9826568-a728-45ab-8180-9777fe31f93a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6817","nonce":2214837203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6819","nonce":2214837203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6823","nonce":2214837203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6821","nonce":2214837203}]},"public_addr":"172.21.15.184:6817/2214837203","cluster_addr":"172.21.15.184:6819/2214837203","heartbeat_back_addr":"172.21.15.184:6823/2214837203","heartbeat_front_addr":"172.21.15.184:6821/2214837203","state":["exists","up"]},{"osd":7,"uuid":"d76869fe-5121-4467-8b56-1aadef0eaf65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6825","nonce":3378735997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6827","nonce":3378735997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6831","nonce":3378735997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6829","nonce":3378735997}]},"public_addr":"172.21.15.184:6825/3378735997","cluster_addr":"172.21.15.184:6827/3378735997","heartbeat_back_addr":"172.21.15.184:6831/3378735997","heartbeat_front_addr":"172.21.15.184:6829/3378735997","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:14:49.835715+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:15:14.682882+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:15:39.423890+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:16:04.527460+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:16:27.283661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:16:51.893868+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:16.152592+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:39.763533+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.151:6800/3383065173":"2024-09-14T14:13:04.761383+0000","172.21.15.151:6801/3383065173":"2024-09-14T14:13:04.761383+0000","172.21.15.151:0/3946777511":"2024-09-14T14:13:04.761383+0000","172.21.15.151:0/624042464":"2024-09-14T14:13:04.761383+0000","172.21.15.151:6801/1057983082":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/1182681341":"2024-09-14T14:13:04.761383+0000","172.21.15.151:0/1427922964":"2024-09-14T14:13:27.478847+0000","172.21.15.151:6800/1057983082":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/1351186656":"2024-09-14T14:18:18.344788+0000","172.21.15.151:6801/2335297718":"2024-09-14T14:18:18.344788+0000","172.21.15.151:0/2666829207":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/4115173791":"2024-09-14T14:18:18.344788+0000","172.21.15.151:6800/2335297718":"2024-09-14T14:18:18.344788+0000","172.21.15.151:0/3689756337":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/2800235450":"2024-09-14T14:18:18.344788+0000","172.21.15.151:0/2889050140":"2024-09-14T14:18:18.344788+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T14:18:24.947 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-13T14:18:24.948 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T14:18:25.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:24 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:25.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:24 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:25.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:24 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:25.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:24 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:18:25.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:24 smithi184 ceph-mon[33394]: Updating smithi151:/etc/ceph/ceph.conf 2024-09-13T14:18:25.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:24 smithi184 ceph-mon[33394]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-13T14:18:25.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:24 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/1951277762' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:18:25.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:24 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:25.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:24 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:25.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:24 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:25.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:24 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:25.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:24 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:25.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:24 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:25.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:24 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:25.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:24 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:25.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:24 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:18:25.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:24 smithi151 ceph-mon[41387]: Updating smithi151:/etc/ceph/ceph.conf 2024-09-13T14:18:25.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:24 smithi151 ceph-mon[41387]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-13T14:18:25.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:24 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/1951277762' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:18:25.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:24 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:25.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:24 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:25.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:24 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:25.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:24 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:25.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:24 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:25.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:24 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:25.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:24 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:25.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:24 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:18:25.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:24 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:18:25.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:24 smithi151 ceph-mon[29807]: Updating smithi151:/etc/ceph/ceph.conf 2024-09-13T14:18:25.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:24 smithi151 ceph-mon[29807]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-13T14:18:25.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:24 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1951277762' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:18:25.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:24 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:25.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:24 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:25.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:24 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:25.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:24 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:25.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:24 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:25.202 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:18:25.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:25 smithi151 ceph-mon[29807]: Updating smithi151:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:18:25.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:25 smithi151 ceph-mon[29807]: Updating smithi184:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:18:25.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:25 smithi151 ceph-mon[29807]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:18:25.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:25 smithi151 ceph-mon[29807]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:18:25.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:25 smithi151 ceph-mon[29807]: pgmap v7: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:25.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:25 smithi151 ceph-mon[29807]: Updating smithi151:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:18:25.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:25 smithi151 ceph-mon[29807]: Updating smithi184:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:18:25.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:25 smithi151 ceph-mon[29807]: Deploying daemon alertmanager.a on smithi151 2024-09-13T14:18:25.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:25 smithi151 ceph-mon[41387]: Updating smithi151:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:18:25.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:25 smithi151 ceph-mon[41387]: Updating smithi184:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:18:25.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:25 smithi151 ceph-mon[41387]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:18:25.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:25 smithi151 ceph-mon[41387]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:18:25.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:25 smithi151 ceph-mon[41387]: pgmap v7: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:25.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:25 smithi151 ceph-mon[41387]: Updating smithi151:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:18:25.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:25 smithi151 ceph-mon[41387]: Updating smithi184:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:18:25.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:25 smithi151 ceph-mon[41387]: Deploying daemon alertmanager.a on smithi151 2024-09-13T14:18:26.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:25 smithi184 ceph-mon[33394]: Updating smithi151:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:18:26.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:25 smithi184 ceph-mon[33394]: Updating smithi184:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:18:26.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:25 smithi184 ceph-mon[33394]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:18:26.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:25 smithi184 ceph-mon[33394]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:18:26.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:25 smithi184 ceph-mon[33394]: pgmap v7: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:26.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:25 smithi184 ceph-mon[33394]: Updating smithi151:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:18:26.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:25 smithi184 ceph-mon[33394]: Updating smithi184:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:18:26.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:25 smithi184 ceph-mon[33394]: Deploying daemon alertmanager.a on smithi151 2024-09-13T14:18:27.706 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:18:27.706 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":58,"fsid":"2401fc50-71da-11ef-bceb-c7b262605968","created":"2024-09-13T14:12:36.168205+0000","modified":"2024-09-13T14:18:22.740022+0000","last_up_change":"2024-09-13T14:17:40.817515+0000","last_in_change":"2024-09-13T14:17:24.771724+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:15:41.780748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:18:09.030957+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":56,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"0ad945ee-af7d-4f26-a1e9-18ed963194ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":551854295},{"type":"v1","addr":"172.21.15.151:6803","nonce":551854295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":551854295},{"type":"v1","addr":"172.21.15.151:6805","nonce":551854295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":551854295},{"type":"v1","addr":"172.21.15.151:6809","nonce":551854295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":551854295},{"type":"v1","addr":"172.21.15.151:6807","nonce":551854295}]},"public_addr":"172.21.15.151:6803/551854295","cluster_addr":"172.21.15.151:6805/551854295","heartbeat_back_addr":"172.21.15.151:6809/551854295","heartbeat_front_addr":"172.21.15.151:6807/551854295","state":["exists","up"]},{"osd":1,"uuid":"09f5d705-684f-4e86-80a2-ec2292af09d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":3674059846},{"type":"v1","addr":"172.21.15.151:6811","nonce":3674059846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3674059846},{"type":"v1","addr":"172.21.15.151:6813","nonce":3674059846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3674059846},{"type":"v1","addr":"172.21.15.151:6817","nonce":3674059846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":3674059846},{"type":"v1","addr":"172.21.15.151:6815","nonce":3674059846}]},"public_addr":"172.21.15.151:6811/3674059846","cluster_addr":"172.21.15.151:6813/3674059846","heartbeat_back_addr":"172.21.15.151:6817/3674059846","heartbeat_front_addr":"172.21.15.151:6815/3674059846","state":["exists","up"]},{"osd":2,"uuid":"6a63f8af-ff69-46bb-830f-03ea89fc78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":1646754940},{"type":"v1","addr":"172.21.15.151:6819","nonce":1646754940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":1646754940},{"type":"v1","addr":"172.21.15.151:6821","nonce":1646754940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1646754940},{"type":"v1","addr":"172.21.15.151:6825","nonce":1646754940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":1646754940},{"type":"v1","addr":"172.21.15.151:6823","nonce":1646754940}]},"public_addr":"172.21.15.151:6819/1646754940","cluster_addr":"172.21.15.151:6821/1646754940","heartbeat_back_addr":"172.21.15.151:6825/1646754940","heartbeat_front_addr":"172.21.15.151:6823/1646754940","state":["exists","up"]},{"osd":3,"uuid":"770e8334-3b3a-46bc-9782-964b2a696660","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2203980888},{"type":"v1","addr":"172.21.15.151:6827","nonce":2203980888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":2203980888},{"type":"v1","addr":"172.21.15.151:6829","nonce":2203980888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":2203980888},{"type":"v1","addr":"172.21.15.151:6833","nonce":2203980888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":2203980888},{"type":"v1","addr":"172.21.15.151:6831","nonce":2203980888}]},"public_addr":"172.21.15.151:6827/2203980888","cluster_addr":"172.21.15.151:6829/2203980888","heartbeat_back_addr":"172.21.15.151:6833/2203980888","heartbeat_front_addr":"172.21.15.151:6831/2203980888","state":["exists","up"]},{"osd":4,"uuid":"75e258c0-1145-4aee-86aa-db2b63082cc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6801","nonce":3275553935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6803","nonce":3275553935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6807","nonce":3275553935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6805","nonce":3275553935}]},"public_addr":"172.21.15.184:6801/3275553935","cluster_addr":"172.21.15.184:6803/3275553935","heartbeat_back_addr":"172.21.15.184:6807/3275553935","heartbeat_front_addr":"172.21.15.184:6805/3275553935","state":["exists","up"]},{"osd":5,"uuid":"f56a5366-9d22-4960-933d-340aa280fbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6809","nonce":2446419621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6811","nonce":2446419621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6815","nonce":2446419621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6813","nonce":2446419621}]},"public_addr":"172.21.15.184:6809/2446419621","cluster_addr":"172.21.15.184:6811/2446419621","heartbeat_back_addr":"172.21.15.184:6815/2446419621","heartbeat_front_addr":"172.21.15.184:6813/2446419621","state":["exists","up"]},{"osd":6,"uuid":"e9826568-a728-45ab-8180-9777fe31f93a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6817","nonce":2214837203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6819","nonce":2214837203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6823","nonce":2214837203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6821","nonce":2214837203}]},"public_addr":"172.21.15.184:6817/2214837203","cluster_addr":"172.21.15.184:6819/2214837203","heartbeat_back_addr":"172.21.15.184:6823/2214837203","heartbeat_front_addr":"172.21.15.184:6821/2214837203","state":["exists","up"]},{"osd":7,"uuid":"d76869fe-5121-4467-8b56-1aadef0eaf65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6825","nonce":3378735997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6827","nonce":3378735997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6831","nonce":3378735997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6829","nonce":3378735997}]},"public_addr":"172.21.15.184:6825/3378735997","cluster_addr":"172.21.15.184:6827/3378735997","heartbeat_back_addr":"172.21.15.184:6831/3378735997","heartbeat_front_addr":"172.21.15.184:6829/3378735997","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:14:49.835715+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:15:14.682882+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:15:39.423890+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:16:04.527460+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:16:27.283661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:16:51.893868+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:16.152592+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:39.763533+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.151:6800/3383065173":"2024-09-14T14:13:04.761383+0000","172.21.15.151:6801/3383065173":"2024-09-14T14:13:04.761383+0000","172.21.15.151:0/3946777511":"2024-09-14T14:13:04.761383+0000","172.21.15.151:0/624042464":"2024-09-14T14:13:04.761383+0000","172.21.15.151:6801/1057983082":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/1182681341":"2024-09-14T14:13:04.761383+0000","172.21.15.151:0/1427922964":"2024-09-14T14:13:27.478847+0000","172.21.15.151:6800/1057983082":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/1351186656":"2024-09-14T14:18:18.344788+0000","172.21.15.151:6801/2335297718":"2024-09-14T14:18:18.344788+0000","172.21.15.151:0/2666829207":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/4115173791":"2024-09-14T14:18:18.344788+0000","172.21.15.151:6800/2335297718":"2024-09-14T14:18:18.344788+0000","172.21.15.151:0/3689756337":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/2800235450":"2024-09-14T14:18:18.344788+0000","172.21.15.151:0/2889050140":"2024-09-14T14:18:18.344788+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T14:18:27.794 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:27 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:18:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:18:27.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:27 smithi151 ceph-mon[29807]: pgmap v8: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-13T14:18:27.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:27 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/285053002' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:18:27.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:27 smithi151 ceph-mon[41387]: pgmap v8: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-13T14:18:27.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:27 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/285053002' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:18:28.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:27 smithi184 ceph-mon[33394]: pgmap v8: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-13T14:18:28.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:27 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/285053002' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:18:29.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-13T14:18:29.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-13T14:18:29.429 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-13T14:18:29.429 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-13T14:18:29.429 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-13T14:18:29.430 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-13T14:18:29.430 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-13T14:18:29.431 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-13T14:18:30.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:29 smithi184 ceph-mon[33394]: pgmap v9: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-13T14:18:30.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:29 smithi151 ceph-mon[29807]: pgmap v9: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-13T14:18:30.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:29 smithi151 ceph-mon[41387]: pgmap v9: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-13T14:18:32.046 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:18:32.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:31 smithi184 ceph-mon[33394]: pgmap v10: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-13T14:18:32.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:31 smithi151 ceph-mon[29807]: pgmap v10: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-13T14:18:32.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:31 smithi151 ceph-mon[41387]: pgmap v10: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-13T14:18:34.004 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:18:34.004 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:18:34.005 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:18:34.009 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:18:34.009 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:18:34.009 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:18:34.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:33 smithi184 ceph-mon[33394]: pgmap v11: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-13T14:18:34.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:33 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:18:34.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:33 smithi151 ceph-mon[41387]: pgmap v11: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-13T14:18:34.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:33 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:18:34.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:33 smithi151 ceph-mon[29807]: pgmap v11: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-13T14:18:34.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:33 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:18:34.608 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:18:34 smithi151 systemd[1]: Starting Ceph alertmanager.a for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:18:34.740 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:18:36.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:35 smithi184 ceph-mon[33394]: pgmap v12: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-13T14:18:36.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:35 smithi151 ceph-mon[29807]: pgmap v12: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-13T14:18:36.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:35 smithi151 ceph-mon[41387]: pgmap v12: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-13T14:18:36.400 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:18:36 smithi151 podman[73997]: 2024-09-13 14:18:36.276701354 +0000 UTC m=+0.546439658 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-13T14:18:36.858 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:18:36 smithi151 podman[73997]: 2024-09-13 14:18:36.533745402 +0000 UTC m=+0.803483751 volume create 8c8518b5f8feadfbacd7f22e68b590102d7d5a3f230d44a6759abcd71674a6f7 2024-09-13T14:18:37.807 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:37 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:18:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:18:38.067 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:37 smithi151 ceph-mon[29807]: pgmap v13: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:18:38.067 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:37 smithi151 ceph-mon[41387]: pgmap v13: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:18:38.068 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:18:37 smithi151 podman[73997]: 2024-09-13 14:18:37.96143014 +0000 UTC m=+2.231168443 container create af9bba3abbf6956f994b2cf447b9177b10ae2470f558ef210b0b3180280aa245 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:18:38.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:37 smithi184 ceph-mon[33394]: pgmap v13: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:18:39.770 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:18:39 smithi151 podman[73997]: 2024-09-13 14:18:39.559429883 +0000 UTC m=+3.829168189 container init af9bba3abbf6956f994b2cf447b9177b10ae2470f558ef210b0b3180280aa245 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:18:39.770 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:18:39 smithi151 podman[73997]: 2024-09-13 14:18:39.564947379 +0000 UTC m=+3.834685690 container start af9bba3abbf6956f994b2cf447b9177b10ae2470f558ef210b0b3180280aa245 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:18:39.770 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:18:39 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[74160]: ts=2024-09-13T14:18:39.577Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-13T14:18:39.770 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:18:39 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[74160]: ts=2024-09-13T14:18:39.577Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-13T14:18:39.770 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:18:39 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[74160]: ts=2024-09-13T14:18:39.578Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.151 port=9094 2024-09-13T14:18:39.771 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:18:39 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[74160]: ts=2024-09-13T14:18:39.579Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-13T14:18:39.771 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:18:39 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[74160]: ts=2024-09-13T14:18:39.602Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-13T14:18:39.771 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:18:39 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[74160]: ts=2024-09-13T14:18:39.603Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-13T14:18:39.771 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:18:39 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[74160]: ts=2024-09-13T14:18:39.605Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-13T14:18:39.771 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:18:39 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[74160]: ts=2024-09-13T14:18:39.605Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-13T14:18:39.771 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:18:39 smithi151 bash[73997]: af9bba3abbf6956f994b2cf447b9177b10ae2470f558ef210b0b3180280aa245 2024-09-13T14:18:39.771 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:18:39 smithi151 systemd[1]: Started Ceph alertmanager.a for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:18:40.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:39 smithi151 ceph-mon[29807]: pgmap v14: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:40.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:39 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:40.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:39 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:40.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:39 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:40.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:39 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:40.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:39 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:40.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:39 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:40.066 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:39 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T14:18:40.066 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:39 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:40.066 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:39 smithi151 ceph-mon[41387]: pgmap v14: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:40.066 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:39 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:40.066 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:39 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:40.066 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:39 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:40.066 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:39 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:40.066 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:39 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:40.066 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:39 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:40.066 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:39 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T14:18:40.066 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:39 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:40.067 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:39 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:18:39] ENGINE Bus STOPPING 2024-09-13T14:18:40.067 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:40 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:18:40] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:18:40.067 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:40 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:18:40] ENGINE Bus STOPPED 2024-09-13T14:18:40.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:39 smithi184 ceph-mon[33394]: pgmap v14: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:40.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:39 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:40.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:39 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:40.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:39 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:40.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:39 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:40.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:39 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:40.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:39 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:40.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:39 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T14:18:40.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:39 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:40.345 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:40 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:18:40] ENGINE Bus STARTING 2024-09-13T14:18:40.345 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:40 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:18:40] ENGINE Serving on http://:::9283 2024-09-13T14:18:40.345 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:18:40 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:18:40] ENGINE Bus STARTED 2024-09-13T14:18:41.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:40 smithi151 ceph-mon[29807]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T14:18:41.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:40 smithi151 ceph-mon[29807]: Deploying daemon grafana.a on smithi184 2024-09-13T14:18:41.055 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:40 smithi151 ceph-mon[41387]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T14:18:41.055 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:40 smithi151 ceph-mon[41387]: Deploying daemon grafana.a on smithi184 2024-09-13T14:18:41.080 INFO:teuthology.orchestra.run.smithi151.stdout:55834574890 2024-09-13T14:18:41.080 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-13T14:18:41.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:40 smithi184 ceph-mon[33394]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T14:18:41.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:40 smithi184 ceph-mon[33394]: Deploying daemon grafana.a on smithi184 2024-09-13T14:18:41.820 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:18:41 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[74160]: ts=2024-09-13T14:18:41.579Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000310958s 2024-09-13T14:18:42.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:41 smithi184 ceph-mon[33394]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:42.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:41 smithi151 ceph-mon[29807]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:42.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:41 smithi151 ceph-mon[41387]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:44.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:43 smithi184 ceph-mon[33394]: pgmap v16: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:44.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:43 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:44.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:43 smithi151 ceph-mon[41387]: pgmap v16: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:44.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:43 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:44.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:43 smithi151 ceph-mon[29807]: pgmap v16: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:44.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:43 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:45.964 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:45 smithi151 ceph-mon[29807]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:45.964 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:45 smithi151 ceph-mon[41387]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:45.987 INFO:teuthology.orchestra.run.smithi151.stdout:34359738418 2024-09-13T14:18:45.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-13T14:18:46.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:45 smithi184 ceph-mon[33394]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:46.316 INFO:teuthology.orchestra.run.smithi151.stdout:180388626450 2024-09-13T14:18:46.317 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-13T14:18:47.519 INFO:teuthology.orchestra.run.smithi151.stdout:128849018908 2024-09-13T14:18:47.520 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-13T14:18:47.827 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:47 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:18:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:18:48.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:47 smithi184 ceph-mon[33394]: pgmap v18: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:48.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:47 smithi151 ceph-mon[29807]: pgmap v18: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:48.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:47 smithi151 ceph-mon[41387]: pgmap v18: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:49.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:48 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:18:49.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:48 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:18:49.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:48 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:18:49.834 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:18:49 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[74160]: ts=2024-09-13T14:18:49.582Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003152572s 2024-09-13T14:18:49.835 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:49 smithi151 ceph-mon[29807]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:49.973 INFO:teuthology.orchestra.run.smithi151.stdout:107374182435 2024-09-13T14:18:49.973 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-13T14:18:50.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:49 smithi184 ceph-mon[33394]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:50.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:49 smithi151 ceph-mon[41387]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:51.724 INFO:teuthology.orchestra.run.smithi151.stdout:154618822682 2024-09-13T14:18:51.724 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-13T14:18:51.946 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:18:51.949 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:51 smithi151 ceph-mon[41387]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:51.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:51 smithi151 ceph-mon[29807]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:51.986 INFO:teuthology.orchestra.run.smithi151.stdout:77309411368 2024-09-13T14:18:51.986 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-13T14:18:52.085 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:51 smithi184 ceph-mon[33394]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:52.086 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:51 smithi184 podman[64807]: 2024-09-13 14:18:51.889483323 +0000 UTC m=+0.016019240 image pull 954c08fa618887fe9a6fe572ae541197c77e625b05f4193df7898f5ac649ada9 quay.io/ceph/ceph-grafana:9.4.7 2024-09-13T14:18:52.442 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 podman[64807]: 2024-09-13 14:18:52.190132069 +0000 UTC m=+0.316668001 container init 20c2e676f61565ee381d83869b15f70c3f13d7ee15477569f0bf910cf8bed8f5 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a, build-date=2022-04-29T11:48:11.055897, release=239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., name=ubi8, version=8.5, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, io.buildah.version=1.32.0, description=Ceph Grafana Container, io.openshift.tags=base rhel8, architecture=x86_64, maintainer=Paul Cuzner ) 2024-09-13T14:18:52.442 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 podman[64807]: 2024-09-13 14:18:52.194953827 +0000 UTC m=+0.321489753 container start 20c2e676f61565ee381d83869b15f70c3f13d7ee15477569f0bf910cf8bed8f5 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , release=239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-04-29T11:48:11.055897, version=8.5, description=Ceph Grafana Container, io.openshift.expose-services=, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, com.redhat.component=ubi8-container, vcs-type=git, distribution-scope=public, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664) 2024-09-13T14:18:52.443 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 bash[64807]: 20c2e676f61565ee381d83869b15f70c3f13d7ee15477569f0bf910cf8bed8f5 2024-09-13T14:18:52.443 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 systemd[1]: Started Ceph grafana.a for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:18:52.443 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=settings t=2024-09-13T14:18:52.347278986Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-09-13T14:18:52.443 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=settings t=2024-09-13T14:18:52.347533908Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-13T14:18:52.443 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=settings t=2024-09-13T14:18:52.347558093Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-13T14:18:52.443 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=settings t=2024-09-13T14:18:52.347568852Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-13T14:18:52.443 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=settings t=2024-09-13T14:18:52.347579797Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-13T14:18:52.443 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=settings t=2024-09-13T14:18:52.347588546Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-13T14:18:52.443 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=settings t=2024-09-13T14:18:52.347597489Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-13T14:18:52.443 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=settings t=2024-09-13T14:18:52.347606574Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-13T14:18:52.443 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=settings t=2024-09-13T14:18:52.347632738Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-13T14:18:52.443 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=settings t=2024-09-13T14:18:52.347642196Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-13T14:18:52.444 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=settings t=2024-09-13T14:18:52.34765283Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-13T14:18:52.444 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=settings t=2024-09-13T14:18:52.347663446Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-13T14:18:52.444 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=settings t=2024-09-13T14:18:52.347673633Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-13T14:18:52.444 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=settings t=2024-09-13T14:18:52.347684869Z level=info msg="App mode production" 2024-09-13T14:18:52.444 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=sqlstore t=2024-09-13T14:18:52.347748698Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-13T14:18:52.444 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=sqlstore t=2024-09-13T14:18:52.347772757Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-13T14:18:52.444 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.348616328Z level=info msg="Starting DB migrations" 2024-09-13T14:18:52.444 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.348701179Z level=info msg="Executing migration" id="create migration_log table" 2024-09-13T14:18:52.444 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.349308938Z level=info msg="Executing migration" id="create user table" 2024-09-13T14:18:52.444 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.349773399Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-13T14:18:52.444 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.350283764Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-13T14:18:52.444 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.350905047Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-13T14:18:52.444 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.35142115Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-13T14:18:52.445 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.351795155Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-13T14:18:52.445 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.352523882Z level=info msg="Executing migration" id="create user table v2" 2024-09-13T14:18:52.445 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.352921838Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-13T14:18:52.445 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.353363834Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-13T14:18:52.446 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.353786377Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-13T14:18:52.446 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.356352997Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-13T14:18:52.446 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.359101834Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-13T14:18:52.446 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.361925897Z level=info msg="Executing migration" id="Update user table charset" 2024-09-13T14:18:52.446 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.364595381Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-13T14:18:52.446 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.367525714Z level=info msg="Executing migration" id="Add missing user data" 2024-09-13T14:18:52.446 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.367993658Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-13T14:18:52.446 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.368998099Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-13T14:18:52.446 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.369712499Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-13T14:18:52.446 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.370151879Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-13T14:18:52.446 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.371415783Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-13T14:18:52.446 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.372211156Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-13T14:18:52.446 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.372998097Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-13T14:18:52.447 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.373959589Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-13T14:18:52.447 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.37461675Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-13T14:18:52.447 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.37539721Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-13T14:18:52.447 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.376265981Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-13T14:18:52.447 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.377021209Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-13T14:18:52.447 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.37788197Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-13T14:18:52.447 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.378666672Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-13T14:18:52.447 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.379498217Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-13T14:18:52.447 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.38039493Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-13T14:18:52.447 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.381350795Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-13T14:18:52.447 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.382050378Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-13T14:18:52.447 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.382468501Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-13T14:18:52.447 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.383296011Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-13T14:18:52.447 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.383738345Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-13T14:18:52.448 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.384164268Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-13T14:18:52.448 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.385137265Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-13T14:18:52.448 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.385908256Z level=info msg="Executing migration" id="create star table" 2024-09-13T14:18:52.448 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.386890707Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-13T14:18:52.448 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.387764297Z level=info msg="Executing migration" id="create org table v1" 2024-09-13T14:18:52.448 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.388451603Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-13T14:18:52.448 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.389441931Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-13T14:18:52.448 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.390132983Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-13T14:18:52.448 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.391026435Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-13T14:18:52.448 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.391974415Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-13T14:18:52.448 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.392702837Z level=info msg="Executing migration" id="Update org table charset" 2024-09-13T14:18:52.448 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.393587184Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-13T14:18:52.448 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.394057424Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-13T14:18:52.449 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.394434856Z level=info msg="Executing migration" id="create dashboard table" 2024-09-13T14:18:52.449 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.397327654Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-13T14:18:52.449 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.399967877Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-13T14:18:52.449 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.402865893Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-13T14:18:52.449 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.405608105Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-13T14:18:52.449 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.408387127Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-13T14:18:52.449 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.411243043Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-13T14:18:52.449 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.413989185Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-13T14:18:52.449 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.416924066Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-13T14:18:52.449 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.419615057Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-13T14:18:52.449 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.422384657Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-13T14:18:52.449 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.425252198Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-13T14:18:52.449 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.428113055Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-13T14:18:52.449 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.430770877Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-13T14:18:52.450 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.433628434Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-13T14:18:52.450 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.436424019Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-13T14:18:52.450 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.439335933Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-13T14:18:52.450 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.442089981Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-13T14:18:52.694 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.444889884Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-13T14:18:52.695 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.447625392Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-13T14:18:52.695 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.450368075Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-13T14:18:52.695 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.453096852Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-13T14:18:52.695 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.455957397Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-13T14:18:52.695 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.458731559Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-13T14:18:52.695 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.461308813Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-13T14:18:52.695 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.464011306Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-13T14:18:52.695 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.466843107Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-13T14:18:52.696 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.469450675Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-13T14:18:52.696 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.472087763Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-13T14:18:52.696 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.4749365Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-13T14:18:52.696 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.47739266Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-13T14:18:52.696 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.480204759Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-13T14:18:52.696 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.482954028Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-13T14:18:52.696 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.485599804Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-13T14:18:52.696 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.488354671Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-13T14:18:52.697 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.490998697Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-13T14:18:52.697 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.491859533Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-13T14:18:52.697 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.492812986Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-13T14:18:52.697 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.49353336Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-13T14:18:52.697 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.494413931Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-13T14:18:52.697 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.495289725Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-13T14:18:52.697 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.496005542Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-13T14:18:52.697 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.496826615Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-13T14:18:52.697 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.497646302Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-13T14:18:52.697 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.4986126Z level=info msg="Executing migration" id="create data_source table" 2024-09-13T14:18:52.697 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.499350775Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-13T14:18:52.697 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.500186725Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-13T14:18:52.697 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.501131898Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-13T14:18:52.698 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.501867919Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-13T14:18:52.698 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.502734417Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-13T14:18:52.698 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.503773076Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-13T14:18:52.698 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.504695825Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-13T14:18:52.698 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.505424218Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-13T14:18:52.698 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.506386339Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-13T14:18:52.698 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.507100436Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-13T14:18:52.698 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.507955848Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-13T14:18:52.700 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.508805603Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-13T14:18:52.700 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.509679301Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-13T14:18:52.700 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.510589517Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-13T14:18:52.700 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.511245041Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-13T14:18:52.700 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.511567197Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-13T14:18:52.700 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.512421403Z level=info msg="Executing migration" id="Add uid column" 2024-09-13T14:18:52.700 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.512958394Z level=info msg="Executing migration" id="Update uid value" 2024-09-13T14:18:52.700 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.513289385Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-13T14:18:52.700 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.514121846Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-13T14:18:52.701 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.514561266Z level=info msg="Executing migration" id="create api_key table" 2024-09-13T14:18:52.701 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.514998214Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-13T14:18:52.701 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.515852404Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-13T14:18:52.701 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.516731566Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-13T14:18:52.701 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.517730012Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-13T14:18:52.701 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.518456753Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-13T14:18:52.701 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.51884131Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-13T14:18:52.701 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.519288747Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-13T14:18:52.701 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.521824456Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-13T14:18:52.701 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.524465848Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-13T14:18:52.701 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.527398677Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-13T14:18:52.701 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.527958299Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-13T14:18:52.701 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.530663213Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-13T14:18:52.702 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.533398771Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-13T14:18:52.702 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.536238258Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-13T14:18:52.702 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.53668431Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-13T14:18:52.702 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.537560426Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-13T14:18:52.702 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.538396019Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-13T14:18:52.702 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.539204661Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-13T14:18:52.702 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.539750173Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-13T14:18:52.702 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.540747791Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-13T14:18:52.702 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.541468146Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-13T14:18:52.702 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.542273546Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-13T14:18:52.702 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.542693636Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-13T14:18:52.702 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.543132692Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-13T14:18:52.702 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.544016685Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-13T14:18:52.702 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.544859736Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-13T14:18:52.703 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.54578258Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-13T14:18:52.703 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.546149193Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-13T14:18:52.703 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.547070441Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-13T14:18:52.703 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.547867128Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-13T14:18:52.703 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.548213922Z level=info msg="Executing migration" id="create quota table v1" 2024-09-13T14:18:52.703 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.548979587Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-13T14:18:52.703 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.549411715Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-13T14:18:52.703 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.549683297Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-13T14:18:52.703 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.550353401Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-13T14:18:52.704 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.551094799Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-13T14:18:52.704 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.551948749Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-13T14:18:52.704 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.55225157Z level=info msg="Executing migration" id="create session table" 2024-09-13T14:18:52.704 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.552703117Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-13T14:18:52.704 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.553477523Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-13T14:18:52.704 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.553790736Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-13T14:18:52.704 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.55423191Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-13T14:18:52.704 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.555032351Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-13T14:18:52.704 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.555882476Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-13T14:18:52.704 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.556180228Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-13T14:18:52.704 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.556454123Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-13T14:18:52.704 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.556740648Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-13T14:18:52.704 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.557143595Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-13T14:18:52.705 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.557949633Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-13T14:18:52.705 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.558560211Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-13T14:18:52.705 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.558855098Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-13T14:18:52.705 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.55945989Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-13T14:18:52.705 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.560062875Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-13T14:18:52.705 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.562959667Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-13T14:18:52.705 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.563416296Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-13T14:18:52.705 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.566268086Z level=info msg="Executing migration" id="create alert table v1" 2024-09-13T14:18:52.705 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.566728023Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-13T14:18:52.705 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.569534684Z level=info msg="Executing migration" id="add index alert state" 2024-09-13T14:18:52.705 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.572326749Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-13T14:18:52.705 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.572974711Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-13T14:18:52.705 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.573359726Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-13T14:18:52.705 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.573774361Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-13T14:18:52.705 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.57423165Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-13T14:18:52.706 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.57566975Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-13T14:18:52.706 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.57606895Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-13T14:18:52.706 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.576506811Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-13T14:18:52.706 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.576803488Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-13T14:18:52.706 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.577171698Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-13T14:18:52.706 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.577566176Z level=info msg="Executing migration" id="Add column is_default" 2024-09-13T14:18:52.706 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.578256974Z level=info msg="Executing migration" id="Add column frequency" 2024-09-13T14:18:52.706 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.578917105Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-13T14:18:52.706 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.579567422Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-13T14:18:52.706 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.580196061Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-13T14:18:52.706 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.580642133Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-13T14:18:52.706 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.580935421Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-13T14:18:52.706 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.581237965Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-13T14:18:52.706 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.581628754Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-13T14:18:52.707 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.582076436Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-13T14:18:52.707 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.582457687Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-13T14:18:52.707 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.582822993Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-13T14:18:52.707 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.583262766Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-13T14:18:52.707 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.58390534Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-13T14:18:52.707 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.584555771Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-13T14:18:52.707 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.58484725Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-13T14:18:52.707 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.585341214Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-13T14:18:52.707 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.585716273Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-13T14:18:52.707 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.586388142Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-13T14:18:52.707 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.58668724Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-13T14:18:52.707 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.587148277Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-13T14:18:52.707 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.58757091Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-13T14:18:52.708 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.587855734Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-13T14:18:52.708 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.588315244Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-13T14:18:52.708 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.591098509Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-13T14:18:52.708 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.591520166Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-13T14:18:52.708 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.591955492Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-13T14:18:52.708 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.592417598Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-13T14:18:52.708 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.59541074Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-13T14:18:52.708 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.598147067Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-13T14:18:52.708 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.599058767Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-13T14:18:52.708 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.599482889Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-13T14:18:52.708 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.602352424Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-13T14:18:52.708 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.605227948Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-13T14:18:52.709 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.608029841Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-13T14:18:52.709 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.608507341Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-13T14:18:52.709 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.611557733Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-13T14:18:52.709 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.61426999Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-13T14:18:52.709 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.6170093Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-13T14:18:52.709 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.617365328Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-13T14:18:52.709 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.620180132Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-13T14:18:52.709 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.620705552Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-13T14:18:52.709 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.623022783Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-13T14:18:52.709 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.625685958Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-13T14:18:52.709 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.626299584Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-13T14:18:52.709 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.629119864Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-13T14:18:52.710 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.631590349Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-13T14:18:52.710 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.634301929Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-13T14:18:52.710 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.634753399Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-13T14:18:52.710 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.637305323Z level=info msg="Executing migration" id="Move region to single row" 2024-09-13T14:18:52.710 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.63988468Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-13T14:18:52.710 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.642437862Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-13T14:18:52.710 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.642855225Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-13T14:18:52.710 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.645675506Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-13T14:18:52.710 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.6483795Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-13T14:18:52.710 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.650696954Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-13T14:18:52.710 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.65116518Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-13T14:18:52.710 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.652040739Z level=info msg="Executing migration" id="create test_data table" 2024-09-13T14:18:52.710 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.652836429Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-13T14:18:52.711 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.653312357Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-13T14:18:52.711 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.65378436Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-13T14:18:52.711 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.654757909Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-13T14:18:52.711 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.65558664Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-13T14:18:52.711 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.656308283Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-13T14:18:52.711 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.656623395Z level=info msg="Executing migration" id="create team table" 2024-09-13T14:18:52.711 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.65750604Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-13T14:18:52.711 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.658353811Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-13T14:18:52.711 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.659191774Z level=info msg="Executing migration" id="create team member table" 2024-09-13T14:18:52.711 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.65961873Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-13T14:18:52.711 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.66051315Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-13T14:18:52.711 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.661437613Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-13T14:18:52.711 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.662183008Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-13T14:18:52.711 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.662920163Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-13T14:18:52.712 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.663764247Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-13T14:18:52.712 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.664647481Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-13T14:18:52.712 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.665542429Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-13T14:18:52.712 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.666005835Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-13T14:18:52.712 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.666808479Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-13T14:18:52.712 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.66726389Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-13T14:18:52.712 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.668132569Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-13T14:18:52.712 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.668574115Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-13T14:18:52.712 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.669474948Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-13T14:18:52.712 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.670359041Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-13T14:18:52.712 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.671159715Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-13T14:18:52.712 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.672101296Z level=info msg="Executing migration" id="create tag table" 2024-09-13T14:18:52.712 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.672919266Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-13T14:18:52.712 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.673765786Z level=info msg="Executing migration" id="create login attempt table" 2024-09-13T14:18:52.713 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.67476536Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-13T14:18:52.713 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.677087725Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-13T14:18:52.713 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.679843927Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-13T14:18:52.713 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.682488024Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-13T14:18:52.713 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.685506297Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-13T14:18:52.713 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.688397578Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-13T14:18:52.713 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.691108838Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-13T14:18:52.713 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.694009019Z level=info msg="Executing migration" id="create user auth table" 2024-09-13T14:18:52.713 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.696630238Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-13T14:18:52.930 INFO:teuthology.orchestra.run.smithi151.stdout:206158430224 2024-09-13T14:18:52.930 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-13T14:18:52.944 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.699381854Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-13T14:18:52.945 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.702103743Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-13T14:18:52.945 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.704877794Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-13T14:18:52.945 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.707605787Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-13T14:18:52.945 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.710348362Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-13T14:18:52.945 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.713107356Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-13T14:18:52.945 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.715943323Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-13T14:18:52.945 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.718564964Z level=info msg="Executing migration" id="create server_lock table" 2024-09-13T14:18:52.945 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.721282581Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-13T14:18:52.945 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.724028792Z level=info msg="Executing migration" id="create user auth token table" 2024-09-13T14:18:52.945 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.726660222Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-13T14:18:52.945 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.729204356Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-13T14:18:52.945 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.731971837Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-13T14:18:52.945 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.734795508Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-13T14:18:52.946 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.73749625Z level=info msg="Executing migration" id="create cache_data table" 2024-09-13T14:18:52.946 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.740111196Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-13T14:18:52.946 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.740959298Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-13T14:18:52.946 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.74174344Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-13T14:18:52.946 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.742524842Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-13T14:18:52.947 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.743429654Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-13T14:18:52.947 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.744111036Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-13T14:18:52.947 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.74492493Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-13T14:18:52.947 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.745895386Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-13T14:18:52.947 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.746610753Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-13T14:18:52.947 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.747350221Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-13T14:18:52.947 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.747820275Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-13T14:18:52.948 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.748293893Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-13T14:18:52.948 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.74914308Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-13T14:18:52.948 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.749578864Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-13T14:18:52.948 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.75046676Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-13T14:18:52.948 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.751355698Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-13T14:18:52.948 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.752288182Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-13T14:18:52.948 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.752998934Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-13T14:18:52.948 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.753841107Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-13T14:18:52.948 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.754816915Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-13T14:18:52.948 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.755550373Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-13T14:18:52.948 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.756463211Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-13T14:18:52.948 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.756957165Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-13T14:18:52.948 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.757386875Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-13T14:18:52.949 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.758262101Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-13T14:18:52.949 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.759098629Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-13T14:18:52.949 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.760064082Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-13T14:18:52.949 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.761042535Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-13T14:18:52.949 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.76485918Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-13T14:18:52.949 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.768296736Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-13T14:18:52.949 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.769181084Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-13T14:18:52.949 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.770161588Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-13T14:18:52.949 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.770993041Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-13T14:18:52.949 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.771863757Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-13T14:18:52.949 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.772721665Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-13T14:18:52.949 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.773635269Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-13T14:18:52.949 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.776533613Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-13T14:18:52.949 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.779058434Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-13T14:18:52.950 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.7819894Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-13T14:18:52.950 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.784670467Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-13T14:18:52.950 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.787549047Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-13T14:18:52.950 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.790374403Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-13T14:18:52.950 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.793391734Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-13T14:18:52.950 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.796155907Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-13T14:18:52.950 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.798988063Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-13T14:18:52.950 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.801985755Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-13T14:18:52.950 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.804621574Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-13T14:18:52.950 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.807500311Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-13T14:18:52.950 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.810335612Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-13T14:18:52.950 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.813084148Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-13T14:18:52.950 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.816040342Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-13T14:18:52.950 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.818831653Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-13T14:18:52.951 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.821487098Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-13T14:18:52.951 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.824192282Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-13T14:18:52.951 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.826953171Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-13T14:18:52.951 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.829693767Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-13T14:18:52.951 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.832447486Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-13T14:18:52.951 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.835188893Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-13T14:18:52.951 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.838103908Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-13T14:18:52.951 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.840628798Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-13T14:18:52.951 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.843449062Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-13T14:18:52.951 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.843790525Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-13T14:18:52.951 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.846660099Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-13T14:18:52.951 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.849408145Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-13T14:18:52.951 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.85211315Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-13T14:18:52.952 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.852708623Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-13T14:18:52.952 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.855543596Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-13T14:18:52.952 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.858124992Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-13T14:18:52.952 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.861045582Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-13T14:18:52.952 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.861717355Z level=info msg="Executing migration" id="create alert_image table" 2024-09-13T14:18:52.952 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.864355347Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-13T14:18:52.952 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.865206327Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-13T14:18:52.952 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.866027071Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-13T14:18:52.952 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.86645217Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-13T14:18:52.952 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.867263079Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-13T14:18:52.952 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.867349328Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-13T14:18:52.952 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.867642584Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-13T14:18:52.952 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.868103404Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-13T14:18:52.953 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.868933413Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-09-13T14:18:52.953 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.869073124Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-09-13T14:18:52.953 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.869204442Z level=info msg="alerts found to migrate" alerts=0 2024-09-13T14:18:52.953 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.869607419Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-13T14:18:52.953 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.870594106Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-13T14:18:52.953 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.871258891Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-13T14:18:52.953 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.872056798Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-13T14:18:52.953 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.872515349Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-13T14:18:52.953 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.873452151Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-13T14:18:52.953 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.874143355Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-13T14:18:52.953 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.875095342Z level=info msg="Executing migration" id="create data_keys table" 2024-09-13T14:18:52.953 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.875658507Z level=info msg="Executing migration" id="create secrets table" 2024-09-13T14:18:52.953 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.87641785Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-13T14:18:52.954 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.880755647Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-13T14:18:52.954 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.881698927Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-13T14:18:52.954 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.882035041Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-13T14:18:52.954 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.886291771Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-13T14:18:52.954 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.890541347Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-13T14:18:52.954 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.891407114Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-13T14:18:52.954 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.891888476Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-13T14:18:52.954 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.892717676Z level=info msg="Executing migration" id="create permission table" 2024-09-13T14:18:52.954 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.893579847Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-13T14:18:52.954 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.894421683Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-13T14:18:52.954 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.894916231Z level=info msg="Executing migration" id="create role table" 2024-09-13T14:18:52.954 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.895785676Z level=info msg="Executing migration" id="add column display_name" 2024-09-13T14:18:52.954 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.896785802Z level=info msg="Executing migration" id="add column group_name" 2024-09-13T14:18:52.955 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.897721753Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-13T14:18:52.955 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.898199142Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-13T14:18:52.955 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.899060476Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-13T14:18:52.955 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.899917341Z level=info msg="Executing migration" id="create team role table" 2024-09-13T14:18:52.955 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.900861184Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-13T14:18:52.955 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.901423061Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-13T14:18:52.955 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.904344297Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-13T14:18:52.955 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.907122903Z level=info msg="Executing migration" id="create user role table" 2024-09-13T14:18:52.955 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.910093273Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-13T14:18:52.955 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.910549654Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-13T14:18:52.956 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.913407691Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-13T14:18:52.956 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.916206268Z level=info msg="Executing migration" id="create builtin role table" 2024-09-13T14:18:52.956 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.919096493Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-13T14:18:52.956 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.919544487Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-13T14:18:52.956 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.922392242Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-13T14:18:52.956 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.925081356Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-13T14:18:52.956 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.925751318Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-13T14:18:52.956 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.926239594Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-13T14:18:52.957 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.926668274Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-13T14:18:52.957 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.927145884Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-13T14:18:52.957 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.927537103Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-13T14:18:52.957 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.927986478Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-13T14:18:52.957 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.928960097Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-13T14:18:52.957 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.929396271Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-13T14:18:52.957 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.929887049Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-13T14:18:52.957 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.930185837Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-13T14:18:52.957 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.930596984Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-13T14:18:52.957 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.931085723Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-13T14:18:52.957 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.931491981Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-13T14:18:52.957 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.931782052Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-13T14:18:52.957 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.932166464Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-13T14:18:52.958 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.932586554Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-13T14:18:52.958 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.93304591Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-13T14:18:52.958 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.933986768Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-13T14:18:52.958 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.934308376Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-13T14:18:52.958 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.934763154Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-13T14:18:52.958 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.935238533Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-13T14:18:52.958 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.935726517Z level=info msg="Executing migration" id="add correlation config column" 2024-09-13T14:18:52.958 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.936719026Z level=info msg="Executing migration" id="create entity_events table" 2024-09-13T14:18:52.958 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.937198014Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-13T14:18:52.958 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.940103028Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-13T14:18:52.958 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.940303866Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-13T14:18:52.958 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.940628517Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-13T14:18:52.958 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.940722508Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-13T14:18:52.959 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.941025805Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-13T14:18:52.959 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.941421638Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-13T14:18:52.959 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.941824807Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-13T14:18:52.959 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.94228458Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-13T14:18:52.959 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.942734531Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-13T14:18:52.959 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.943167809Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-13T14:18:52.959 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.943604292Z level=info msg="Executing migration" id="Drop public config table" 2024-09-13T14:18:52.959 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.943974997Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-13T14:18:52.959 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.944402399Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-13T14:18:52.959 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.944834189Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-13T14:18:52.959 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.94541683Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-13T14:18:52.959 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.945907075Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-13T14:18:53.354 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.949814741Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-13T14:18:53.354 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.951031485Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-13T14:18:53.354 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.952254357Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-13T14:18:53.354 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.952616538Z level=info msg="Executing migration" id="add share column" 2024-09-13T14:18:53.354 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.953842784Z level=info msg="Executing migration" id="create default alerting folders" 2024-09-13T14:18:53.354 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.954356734Z level=info msg="Executing migration" id="create file table" 2024-09-13T14:18:53.355 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.954798036Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-13T14:18:53.355 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.955277445Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-13T14:18:53.355 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.957462619Z level=info msg="Executing migration" id="create file_meta table" 2024-09-13T14:18:53.355 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.960088063Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-13T14:18:53.355 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.962953773Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-13T14:18:53.355 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.965455055Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-13T14:18:53.355 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.966023214Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-13T14:18:53.355 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.968796939Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-13T14:18:53.355 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.971288377Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-13T14:18:53.355 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.973805571Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-13T14:18:53.355 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.974159139Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-13T14:18:53.355 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.97679999Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-13T14:18:53.355 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.979410086Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-13T14:18:53.356 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.982204984Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-13T14:18:53.356 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.982609578Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-13T14:18:53.356 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.985283432Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-13T14:18:53.356 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.986387341Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-13T14:18:53.356 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.987258506Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-13T14:18:53.356 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.997002434Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-13T14:18:53.356 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.997938686Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-13T14:18:53.356 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:52.998588805Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-13T14:18:53.357 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:53.001929654Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-13T14:18:53.357 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:53.002274231Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-13T14:18:53.357 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:53.003106352Z level=info msg="Executing migration" id="create folder table" 2024-09-13T14:18:53.357 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:53.003524404Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-13T14:18:53.357 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:53.003986427Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-13T14:18:53.357 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:53.004829949Z level=info msg="Executing migration" id="Update folder title length" 2024-09-13T14:18:53.357 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:53.005142736Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-13T14:18:53.357 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=migrator t=2024-09-13T14:18:53.005988889Z level=info msg="migrations completed" performed=485 skipped=0 duration=657.307472ms 2024-09-13T14:18:53.357 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=sqlstore t=2024-09-13T14:18:53.006378771Z level=info msg="Created default organization" 2024-09-13T14:18:53.357 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=plugin.loader t=2024-09-13T14:18:53.062836994Z level=info msg="Plugin registered" pluginID=input 2024-09-13T14:18:53.357 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=plugin.loader t=2024-09-13T14:18:53.089599494Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-09-13T14:18:53.357 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=plugin.loader t=2024-09-13T14:18:53.089616945Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-09-13T14:18:53.358 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=secrets t=2024-09-13T14:18:53.089650021Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-13T14:18:53.358 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=query_data t=2024-09-13T14:18:53.091231424Z level=info msg="Query Service initialization" 2024-09-13T14:18:53.358 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=live.push_http t=2024-09-13T14:18:53.093523939Z level=info msg="Live Push Gateway initialization" 2024-09-13T14:18:53.418 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:53 smithi151 ceph-mon[29807]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:53.418 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:53 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:53.418 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:53 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:53.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:53 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:53.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:53 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:53.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:53 smithi151 ceph-mon[29807]: Deploying daemon node-exporter.a on smithi151 2024-09-13T14:18:53.419 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:53 smithi151 ceph-mon[41387]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:53.419 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:53 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:53.419 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:53 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:53.419 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:53 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:53.419 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:53 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:53.419 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:53 smithi151 ceph-mon[41387]: Deploying daemon node-exporter.a on smithi151 2024-09-13T14:18:53.658 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:18:53.658 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:18:53.660 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:18:53.824 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-mon[33394]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:53.824 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:53.824 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:53.824 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:53.825 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:53.825 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-mon[33394]: Deploying daemon node-exporter.a on smithi151 2024-09-13T14:18:54.039 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:18:54.039 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:18:54.103 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=infra.usagestats.collector t=2024-09-13T14:18:53.824246641Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-13T14:18:54.104 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=provisioning.datasources t=2024-09-13T14:18:53.824739869Z level=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-09-13T14:18:54.104 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=provisioning.datasources t=2024-09-13T14:18:53.856595763Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-13T14:18:54.104 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=provisioning.datasources t=2024-09-13T14:18:53.888365821Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-09-13T14:18:54.104 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=provisioning.alerting t=2024-09-13T14:18:53.909784573Z level=info msg="starting to provision alerting" 2024-09-13T14:18:54.104 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=provisioning.alerting t=2024-09-13T14:18:53.909821344Z level=info msg="finished to provision alerting" 2024-09-13T14:18:54.104 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=ngalert.state.manager t=2024-09-13T14:18:53.909902094Z level=info msg="Warming state cache for startup" 2024-09-13T14:18:54.104 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=grafanaStorageLogger t=2024-09-13T14:18:53.909938419Z level=info msg="storage starting" 2024-09-13T14:18:54.104 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=http.server t=2024-09-13T14:18:53.911650312Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-13T14:18:54.104 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=sqlstore.transactions t=2024-09-13T14:18:53.920955526Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-13T14:18:54.104 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=ngalert.state.manager t=2024-09-13T14:18:53.921072138Z level=info msg="State cache has been initialized" states=0 duration=11.169306ms 2024-09-13T14:18:54.104 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=ticker t=2024-09-13T14:18:53.921148624Z level=info msg=starting first_tick=2024-09-13T14:19:00Z 2024-09-13T14:18:54.104 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=ngalert.multiorg.alertmanager t=2024-09-13T14:18:53.921154216Z level=info msg="starting MultiOrg Alertmanager" 2024-09-13T14:18:54.105 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=sqlstore.transactions t=2024-09-13T14:18:53.964536658Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-13T14:18:54.105 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:54 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=sqlstore.transactions t=2024-09-13T14:18:54.005088967Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-13T14:18:54.105 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:18:54 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[64822]: logger=sqlstore.transactions t=2024-09-13T14:18:54.015733087Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-09-13T14:18:54.677 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:54 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:54.678 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:54 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:54.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:54 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:18:55.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:55 smithi151 ceph-mon[41387]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:55.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:55 smithi151 ceph-mon[29807]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:55.716 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:18:55.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:55 smithi184 ceph-mon[33394]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:55.881 INFO:teuthology.orchestra.run.smithi151.stdout:180388626454 2024-09-13T14:18:56.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:56 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/3956185541' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T14:18:56.683 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:56 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/3956185541' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T14:18:56.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:56 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/3956185541' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T14:18:57.793 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:57 smithi151 ceph-mon[29807]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:57.793 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:57 smithi151 ceph-mon[41387]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:57.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:57 smithi184 ceph-mon[33394]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:57.853 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:18:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:18:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:18:58.805 INFO:teuthology.orchestra.run.smithi151.stdout:128849018911 2024-09-13T14:18:59.493 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:18:59.742 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:59 smithi151 ceph-mon[29807]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:59.742 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:18:59 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1434461252' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T14:18:59.742 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:59 smithi151 ceph-mon[41387]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:59.742 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:18:59 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/1434461252' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T14:18:59.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:59 smithi184 ceph-mon[33394]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:18:59.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:18:59 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/1434461252' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T14:19:00.495 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626450 got 180388626454 for osd.6 2024-09-13T14:19:00.495 DEBUG:teuthology.parallel:result is None 2024-09-13T14:19:00.847 INFO:teuthology.orchestra.run.smithi151.stdout:154618822684 2024-09-13T14:19:01.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:01 smithi151 ceph-mon[29807]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:01.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:01 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/917124812' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T14:19:01.853 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:01 smithi151 ceph-mon[41387]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:01.853 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:01 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/917124812' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T14:19:01.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:01 smithi184 ceph-mon[33394]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:01.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:01 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/917124812' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T14:19:02.664 INFO:teuthology.orchestra.run.smithi151.stdout:55834574895 2024-09-13T14:19:03.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:03 smithi151 ceph-mon[29807]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:03.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:03 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/431772974' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T14:19:03.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:03 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1031730758' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T14:19:03.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:03 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:19:03.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:03 smithi151 ceph-mon[41387]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:03.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:03 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/431772974' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T14:19:03.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:03 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/1031730758' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T14:19:03.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:03 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:19:03.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:03 smithi184 ceph-mon[33394]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:03.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:03 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/431772974' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T14:19:03.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:03 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/1031730758' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T14:19:03.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:03 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:19:04.013 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574890 got 55834574895 for osd.1 2024-09-13T14:19:04.014 DEBUG:teuthology.parallel:result is None 2024-09-13T14:19:04.905 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822682 got 154618822684 for osd.5 2024-09-13T14:19:04.906 DEBUG:teuthology.parallel:result is None 2024-09-13T14:19:05.412 INFO:teuthology.orchestra.run.smithi151.stdout:77309411371 2024-09-13T14:19:05.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:05 smithi151 ceph-mon[29807]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:05.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:05 smithi151 ceph-mon[41387]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:05.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:05 smithi184 ceph-mon[33394]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:06.004 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411368 got 77309411371 for osd.2 2024-09-13T14:19:06.004 DEBUG:teuthology.parallel:result is None 2024-09-13T14:19:06.477 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018908 got 128849018911 for osd.4 2024-09-13T14:19:06.477 DEBUG:teuthology.parallel:result is None 2024-09-13T14:19:07.501 INFO:teuthology.orchestra.run.smithi151.stdout:107374182439 2024-09-13T14:19:07.741 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:07 smithi151 ceph-mon[29807]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:07.741 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:07 smithi151 ceph-mon[41387]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:07.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:07 smithi184 ceph-mon[33394]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:07.853 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:19:07 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:19:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:19:08.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:08 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1146160156' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T14:19:08.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:08 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/353771572' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T14:19:08.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:08 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/1146160156' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T14:19:08.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:08 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/353771572' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T14:19:08.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:08 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/1146160156' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T14:19:08.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:08 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/353771572' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T14:19:08.862 INFO:teuthology.orchestra.run.smithi151.stdout:34359738422 2024-09-13T14:19:09.744 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:09 smithi151 ceph-mon[29807]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:09.744 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:09 smithi151 ceph-mon[41387]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:09.773 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738418 got 34359738422 for osd.0 2024-09-13T14:19:09.773 DEBUG:teuthology.parallel:result is None 2024-09-13T14:19:09.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:09 smithi184 ceph-mon[33394]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:10.573 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182435 got 107374182439 for osd.3 2024-09-13T14:19:10.573 DEBUG:teuthology.parallel:result is None 2024-09-13T14:19:10.898 INFO:teuthology.orchestra.run.smithi151.stdout:206158430228 2024-09-13T14:19:11.824 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:11 smithi151 ceph-mon[29807]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:11.824 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:11 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/2699793365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T14:19:11.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:11 smithi151 ceph-mon[41387]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:11.825 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:11 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/2699793365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T14:19:11.849 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430224 got 206158430228 for osd.7 2024-09-13T14:19:11.849 DEBUG:teuthology.parallel:result is None 2024-09-13T14:19:11.849 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-13T14:19:11.849 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:19:11.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:11 smithi184 ceph-mon[33394]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:11.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:11 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/2699793365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T14:19:12.108 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:19:12 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[76940]: ts=2024-09-13T14:19:12.053Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-13T14:19:12.188 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:19:12.459 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:19:12 smithi151 bash[76571]: 401be9d35c59f33d08abbec03ec8d734765876e1035fb57ca6353f7f7e15f3ea 2024-09-13T14:19:12.459 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:19:12 smithi151 systemd[1]: Started Ceph node-exporter.a for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:19:13.216 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:13 smithi184 bash[65101]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-13T14:19:13.315 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:13 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:13.315 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:13 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:13.315 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:13 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:13.315 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:13 smithi151 ceph-mon[29807]: Deploying daemon node-exporter.b on smithi184 2024-09-13T14:19:13.315 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:13 smithi151 ceph-mon[29807]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:13.316 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:13 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:13.316 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:13 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:13.316 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:13 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:13.316 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:13 smithi151 ceph-mon[41387]: Deploying daemon node-exporter.b on smithi184 2024-09-13T14:19:13.316 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:13 smithi151 ceph-mon[41387]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:13.544 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:13 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:13.545 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:13 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:13.545 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:13 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:13.545 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:13 smithi184 ceph-mon[33394]: Deploying daemon node-exporter.b on smithi184 2024-09-13T14:19:13.545 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:13 smithi184 ceph-mon[33394]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:13.853 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:13 smithi184 bash[65101]: Getting image source signatures 2024-09-13T14:19:13.853 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:13 smithi184 bash[65101]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-13T14:19:13.853 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:13 smithi184 bash[65101]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-13T14:19:13.853 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:13 smithi184 bash[65101]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-13T14:19:13.994 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:19:13.994 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T14:19:14.630 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-09-13T14:19:12.364225+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":115,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385068,"kb_used_data":6224,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747576148,"statfs":{"total":767960285184,"available":765517975552,"internally_reserved":0,"allocated":6373376,"data_stored":3705253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002261"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-13T14:18:22.827988+0000","last_change":"2024-09-13T14:18:14.645599+0000","last_active":"2024-09-13T14:18:22.827988+0000","last_peered":"2024-09-13T14:18:22.827988+0000","last_clean":"2024-09-13T14:18:22.827988+0000","last_became_active":"2024-09-13T14:18:10.617927+0000","last_became_peered":"2024-09-13T14:18:10.617927+0000","last_unstale":"2024-09-13T14:18:22.827988+0000","last_undegraded":"2024-09-13T14:18:22.827988+0000","last_fullsized":"2024-09-13T14:18:22.827988+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:51:43.437167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-13T14:18:22.749240+0000","last_change":"2024-09-13T14:18:14.641734+0000","last_active":"2024-09-13T14:18:22.749240+0000","last_peered":"2024-09-13T14:18:22.749240+0000","last_clean":"2024-09-13T14:18:22.749240+0000","last_became_active":"2024-09-13T14:18:10.614303+0000","last_became_peered":"2024-09-13T14:18:10.614303+0000","last_unstale":"2024-09-13T14:18:22.749240+0000","last_undegraded":"2024-09-13T14:18:22.749240+0000","last_fullsized":"2024-09-13T14:18:22.749240+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:28:38.174245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034192599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-13T14:18:22.828560+0000","last_change":"2024-09-13T14:18:14.639101+0000","last_active":"2024-09-13T14:18:22.828560+0000","last_peered":"2024-09-13T14:18:22.828560+0000","last_clean":"2024-09-13T14:18:22.828560+0000","last_became_active":"2024-09-13T14:18:10.718655+0000","last_became_peered":"2024-09-13T14:18:10.718655+0000","last_unstale":"2024-09-13T14:18:22.828560+0000","last_undegraded":"2024-09-13T14:18:22.828560+0000","last_fullsized":"2024-09-13T14:18:22.828560+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:59:01.637559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-13T14:18:22.749237+0000","last_change":"2024-09-13T14:18:14.641641+0000","last_active":"2024-09-13T14:18:22.749237+0000","last_peered":"2024-09-13T14:18:22.749237+0000","last_clean":"2024-09-13T14:18:22.749237+0000","last_became_active":"2024-09-13T14:18:10.718349+0000","last_became_peered":"2024-09-13T14:18:10.718349+0000","last_unstale":"2024-09-13T14:18:22.749237+0000","last_undegraded":"2024-09-13T14:18:22.749237+0000","last_fullsized":"2024-09-13T14:18:22.749237+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:23.005181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":21,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-13T14:18:22.828079+0000","last_change":"2024-09-13T14:18:14.644865+0000","last_active":"2024-09-13T14:18:22.828079+0000","last_peered":"2024-09-13T14:18:22.828079+0000","last_clean":"2024-09-13T14:18:22.828079+0000","last_became_active":"2024-09-13T14:18:10.617018+0000","last_became_peered":"2024-09-13T14:18:10.617018+0000","last_unstale":"2024-09-13T14:18:22.828079+0000","last_undegraded":"2024-09-13T14:18:22.828079+0000","last_fullsized":"2024-09-13T14:18:22.828079+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:01:22.505985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-13T14:18:22.750113+0000","last_change":"2024-09-13T14:18:14.641344+0000","last_active":"2024-09-13T14:18:22.750113+0000","last_peered":"2024-09-13T14:18:22.750113+0000","last_clean":"2024-09-13T14:18:22.750113+0000","last_became_active":"2024-09-13T14:18:10.766103+0000","last_became_peered":"2024-09-13T14:18:10.766103+0000","last_unstale":"2024-09-13T14:18:22.750113+0000","last_undegraded":"2024-09-13T14:18:22.750113+0000","last_fullsized":"2024-09-13T14:18:22.750113+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:03.456609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-13T14:18:22.827907+0000","last_change":"2024-09-13T14:18:14.638950+0000","last_active":"2024-09-13T14:18:22.827907+0000","last_peered":"2024-09-13T14:18:22.827907+0000","last_clean":"2024-09-13T14:18:22.827907+0000","last_became_active":"2024-09-13T14:18:10.718659+0000","last_became_peered":"2024-09-13T14:18:10.718659+0000","last_unstale":"2024-09-13T14:18:22.827907+0000","last_undegraded":"2024-09-13T14:18:22.827907+0000","last_fullsized":"2024-09-13T14:18:22.827907+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:38:08.688330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":20,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-13T14:18:22.827822+0000","last_change":"2024-09-13T14:18:14.644401+0000","last_active":"2024-09-13T14:18:22.827822+0000","last_peered":"2024-09-13T14:18:22.827822+0000","last_clean":"2024-09-13T14:18:22.827822+0000","last_became_active":"2024-09-13T14:18:10.616981+0000","last_became_peered":"2024-09-13T14:18:10.616981+0000","last_unstale":"2024-09-13T14:18:22.827822+0000","last_undegraded":"2024-09-13T14:18:22.827822+0000","last_fullsized":"2024-09-13T14:18:22.827822+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:00:32.657637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034405999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"57'79","reported_seq":48,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T14:18:20.411380+0000","last_change":"2024-09-13T14:17:42.845082+0000","last_active":"2024-09-13T14:18:20.411380+0000","last_peered":"2024-09-13T14:18:20.411380+0000","last_clean":"2024-09-13T14:18:20.411380+0000","last_became_active":"2024-09-13T14:17:42.839116+0000","last_became_peered":"2024-09-13T14:17:42.839116+0000","last_unstale":"2024-09-13T14:18:20.411380+0000","last_undegraded":"2024-09-13T14:18:20.411380+0000","last_fullsized":"2024-09-13T14:18:20.411380+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:15:42.069756+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:58.144190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430228,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298500,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446652,"statfs":{"total":95995035648,"available":95689371648,"internally_reserved":0,"allocated":1171456,"data_stored":832470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.38500000000000001}]},{"osd":1,"last update":"Fri Sep 13 14:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.52000000000000002}]},{"osd":2,"last update":"Fri Sep 13 14:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.49199999999999999}]},{"osd":3,"last update":"Fri Sep 13 14:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.57699999999999996}]},{"osd":4,"last update":"Fri Sep 13 14:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.628},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.42199999999999999}]},{"osd":5,"last update":"Fri Sep 13 14:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.58899999999999997}]},{"osd":6,"last update":"Fri Sep 13 14:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.55700000000000005}]}]},{"osd":6,"up_from":42,"seq":180388626457,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298496,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446656,"statfs":{"total":95995035648,"available":95689375744,"internally_reserved":0,"allocated":1167360,"data_stored":831930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.44}]},{"osd":1,"last update":"Fri Sep 13 14:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.65200000000000002}]},{"osd":2,"last update":"Fri Sep 13 14:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.57199999999999995}]},{"osd":3,"last update":"Fri Sep 13 14:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.55300000000000005}]},{"osd":4,"last update":"Fri Sep 13 14:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.72799999999999998}]},{"osd":5,"last update":"Fri Sep 13 14:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.75700000000000001}]},{"osd":7,"last update":"Fri Sep 13 14:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.70599999999999996}]}]},{"osd":5,"up_from":36,"seq":154618822686,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297916,"kb_used_data":560,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447236,"statfs":{"total":95995035648,"available":95689969664,"internally_reserved":0,"allocated":573440,"data_stored":241562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:18:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52600000000000002,"15min":0.52700000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.498,"15min":0.496},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.38400000000000001}]},{"osd":1,"last update":"Fri Sep 13 14:18:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52100000000000002,"15min":0.52000000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.51800000000000002,"15min":0.51600000000000001},"min":{"1min":0.35799999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.434}]},{"osd":2,"last update":"Fri Sep 13 14:18:56 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.35299999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52300000000000002,"15min":0.52200000000000002},"min":{"1min":0.35699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.46400000000000002}]},{"osd":3,"last update":"Fri Sep 13 14:18:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.50600000000000001,"15min":0.505},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.56100000000000005}]},{"osd":4,"last update":"Fri Sep 13 14:18:56 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.38,"15min":0.371},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.505},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.38400000000000001,"15min":0.374},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.60899999999999999}]},{"osd":6,"last update":"Fri Sep 13 14:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.54800000000000004}]},{"osd":7,"last update":"Fri Sep 13 14:18:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.621}]}]},{"osd":4,"up_from":30,"seq":128849018914,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297916,"kb_used_data":560,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447236,"statfs":{"total":95995035648,"available":95689969664,"internally_reserved":0,"allocated":573440,"data_stored":241858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:18:36 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.53300000000000003,"15min":0.54100000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.72799999999999998,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55800000000000005,"15min":0.56100000000000005},"min":{"1min":0.38200000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.78600000000000003,"5min":0.88,"15min":0.88},"last":0.52900000000000003}]},{"osd":1,"last update":"Fri Sep 13 14:18:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53900000000000003,"15min":0.54200000000000004},"min":{"1min":0.36899999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.72399999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.55500000000000005,"15min":0.56100000000000005},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.74099999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.625}]},{"osd":2,"last update":"Fri Sep 13 14:18:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.55200000000000005,"15min":0.55600000000000005},"min":{"1min":0.39400000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.69299999999999995,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.502,"5min":0.53000000000000003,"15min":0.53400000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70899999999999996,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.48499999999999999}]},{"osd":3,"last update":"Fri Sep 13 14:18:36 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.53700000000000003,"15min":0.54000000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.75700000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.495,"5min":0.53200000000000003,"15min":0.53800000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.68100000000000005,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.47699999999999998}]},{"osd":5,"last update":"Fri Sep 13 14:18:56 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.378,"15min":0.375},"min":{"1min":0.30599999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.60299999999999998,"5min":0.621,"15min":0.621},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.41799999999999998,"15min":0.41299999999999998},"min":{"1min":0.32200000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.63900000000000001,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.371}]},{"osd":6,"last update":"Fri Sep 13 14:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.32000000000000001}]},{"osd":7,"last update":"Fri Sep 13 14:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.35899999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182440,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297916,"kb_used_data":560,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447236,"statfs":{"total":95995035648,"available":95689969664,"internally_reserved":0,"allocated":573440,"data_stored":241858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:18:11 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.41299999999999998,"15min":0.41399999999999998},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.40100000000000002,"15min":0.40300000000000002},"min":{"1min":0.29399999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.52600000000000002,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.28899999999999998}]},{"osd":1,"last update":"Fri Sep 13 14:18:11 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39600000000000002,"15min":0.39700000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.435,"5min":0.45000000000000001,"15min":0.45200000000000001},"min":{"1min":0.28699999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.627,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.317}]},{"osd":2,"last update":"Fri Sep 13 14:18:11 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.32900000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.43099999999999999,"15min":0.435},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.33500000000000002}]},{"osd":4,"last update":"Fri Sep 13 14:18:30 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.52200000000000002,"15min":0.52500000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.74099999999999999,"5min":0.748,"15min":0.748},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.56200000000000006,"15min":0.56699999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.71999999999999997,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.64300000000000002}]},{"osd":5,"last update":"Fri Sep 13 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.53800000000000003,"15min":0.54400000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.66600000000000004,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.54800000000000004,"15min":0.55300000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.746,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.67800000000000005}]},{"osd":6,"last update":"Fri Sep 13 14:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.56899999999999995}]},{"osd":7,"last update":"Fri Sep 13 14:18:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.65300000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297908,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447244,"statfs":{"total":95995035648,"available":95689977856,"internally_reserved":0,"allocated":569344,"data_stored":241614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:18:49 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.39400000000000002,"15min":0.38400000000000001},"min":{"1min":0.27100000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.59099999999999997,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.376,"15min":0.36199999999999999},"min":{"1min":0.22800000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.58899999999999997,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.33100000000000002}]},{"osd":1,"last update":"Fri Sep 13 14:18:49 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.39700000000000002,"15min":0.38500000000000001},"min":{"1min":0.27600000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.53200000000000003,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.40699999999999997,"15min":0.39400000000000002},"min":{"1min":0.27800000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.36399999999999999}]},{"osd":3,"last update":"Fri Sep 13 14:18:11 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.46200000000000002,"15min":0.46000000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.441},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.46300000000000002,"15min":0.45900000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.41399999999999998}]},{"osd":4,"last update":"Fri Sep 13 14:18:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.55800000000000005,"15min":0.56200000000000006},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69199999999999995,"5min":0.751,"15min":0.751},"last":0.499},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.59199999999999997,"15min":0.59599999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.747,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.59199999999999997}]},{"osd":5,"last update":"Fri Sep 13 14:18:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.62,"15min":0.63},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76700000000000002,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58999999999999997,"15min":0.59599999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.56499999999999995}]},{"osd":6,"last update":"Fri Sep 13 14:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.51200000000000001}]},{"osd":7,"last update":"Fri Sep 13 14:18:41 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.61399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574897,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297920,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447232,"statfs":{"total":95995035648,"available":95689965568,"internally_reserved":0,"allocated":577536,"data_stored":241806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.33100000000000002,"15min":0.308},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.47199999999999998,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.34899999999999998,"15min":0.32000000000000001},"min":{"1min":0.26200000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.58299999999999996,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.50800000000000001}]},{"osd":2,"last update":"Fri Sep 13 14:18:46 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.39900000000000002,"15min":0.39100000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.502,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.38300000000000001,"15min":0.36599999999999999},"min":{"1min":0.27300000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.60399999999999998}]},{"osd":3,"last update":"Fri Sep 13 14:18:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.40899999999999997,"15min":0.40999999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.45300000000000001,"15min":0.45400000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.56200000000000006,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.624}]},{"osd":4,"last update":"Fri Sep 13 14:18:32 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54000000000000004,"15min":0.53700000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.65800000000000003,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.70199999999999996}]},{"osd":5,"last update":"Fri Sep 13 14:18:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.55800000000000005,"15min":0.56399999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.67700000000000005,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.59399999999999997,"15min":0.60199999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.71599999999999997}]},{"osd":6,"last update":"Fri Sep 13 14:18:23 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.76500000000000001}]},{"osd":7,"last update":"Fri Sep 13 14:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.746},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.81499999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738423,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298496,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446656,"statfs":{"total":95995035648,"available":95689375744,"internally_reserved":0,"allocated":1167360,"data_stored":832155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 14:18:19 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.315,"15min":0.30299999999999999},"min":{"1min":0.22600000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.46100000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.371,"5min":0.34999999999999998,"15min":0.33000000000000002},"min":{"1min":0.24099999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52800000000000002,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.47299999999999998}]},{"osd":2,"last update":"Fri Sep 13 14:18:47 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39200000000000002,"15min":0.38800000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.499},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.372,"15min":0.36599999999999999},"min":{"1min":0.26000000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.38}]},{"osd":3,"last update":"Fri Sep 13 14:18:14 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.44400000000000001,"15min":0.44800000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.432},{"interface":"front","average":{"1min":0.371,"5min":0.38800000000000001,"15min":0.39100000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.45200000000000001}]},{"osd":4,"last update":"Fri Sep 13 14:18:38 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.51000000000000001,"15min":0.50600000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.498,"15min":0.497},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.63300000000000001}]},{"osd":5,"last update":"Fri Sep 13 14:18:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.35899999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.55200000000000005,"15min":0.54700000000000004},"min":{"1min":0.42799999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.68100000000000005}]},{"osd":6,"last update":"Fri Sep 13 14:18:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.72199999999999998}]},{"osd":7,"last update":"Fri Sep 13 14:18:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.62},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.64900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:19:14.632 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:19:14.877 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:19:15.353 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:14 smithi184 bash[65101]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-13T14:19:15.353 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 bash[65101]: Writing manifest to image destination 2024-09-13T14:19:15.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:15 smithi151 ceph-mon[29807]: from='client.24512 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:19:15.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:15 smithi151 ceph-mon[29807]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:15.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:15 smithi151 ceph-mon[41387]: from='client.24512 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:19:15.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:15 smithi151 ceph-mon[41387]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:15.711 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-mon[33394]: from='client.24512 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:19:15.711 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-mon[33394]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:15.712 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 podman[65101]: 2024-09-13 14:19:15.368412415 +0000 UTC m=+2.351491887 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-13T14:19:15.712 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 podman[65101]: 2024-09-13 14:19:15.467369375 +0000 UTC m=+2.450448834 container create aca4c09ae196f0f2ee45cc936a3bc0716c5ca749e1657061c9bebd9d081aa8e4 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:19:16.104 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 podman[65101]: 2024-09-13 14:19:15.7111903 +0000 UTC m=+2.694269759 container init aca4c09ae196f0f2ee45cc936a3bc0716c5ca749e1657061c9bebd9d081aa8e4 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:19:16.104 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 podman[65101]: 2024-09-13 14:19:15.716267912 +0000 UTC m=+2.699347381 container start aca4c09ae196f0f2ee45cc936a3bc0716c5ca749e1657061c9bebd9d081aa8e4 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:19:16.104 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.720Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-13T14:19:16.104 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.720Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-13T14:19:16.104 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.720Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-13T14:19:16.104 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.720Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-13T14:19:16.105 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.720Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-13T14:19:16.105 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.720Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-13T14:19:16.105 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-13T14:19:16.105 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=arp 2024-09-13T14:19:16.105 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-13T14:19:16.105 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-13T14:19:16.105 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-13T14:19:16.105 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-13T14:19:16.105 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-13T14:19:16.105 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-13T14:19:16.105 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-13T14:19:16.105 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-13T14:19:16.105 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=edac 2024-09-13T14:19:16.105 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-13T14:19:16.106 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-13T14:19:16.106 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-13T14:19:16.106 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-13T14:19:16.106 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-13T14:19:16.106 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-13T14:19:16.106 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-13T14:19:16.106 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-13T14:19:16.106 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-13T14:19:16.106 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-13T14:19:16.106 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-13T14:19:16.106 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-13T14:19:16.106 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-13T14:19:16.106 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-13T14:19:16.106 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-13T14:19:16.106 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-13T14:19:16.107 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=os 2024-09-13T14:19:16.107 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-13T14:19:16.107 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-13T14:19:16.107 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-13T14:19:16.107 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-13T14:19:16.107 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-13T14:19:16.107 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-13T14:19:16.107 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-13T14:19:16.107 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=stat 2024-09-13T14:19:16.107 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-13T14:19:16.107 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-13T14:19:16.108 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-13T14:19:16.108 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=time 2024-09-13T14:19:16.108 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-13T14:19:16.108 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=uname 2024-09-13T14:19:16.108 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-13T14:19:16.108 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-13T14:19:16.108 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-13T14:19:16.108 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-13T14:19:16.108 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[65177]: ts=2024-09-13T14:19:15.721Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-13T14:19:16.108 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 bash[65101]: aca4c09ae196f0f2ee45cc936a3bc0716c5ca749e1657061c9bebd9d081aa8e4 2024-09-13T14:19:16.108 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:19:15 smithi184 systemd[1]: Started Ceph node-exporter.b for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:19:16.716 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:19:16.716 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T14:19:17.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:16 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:17.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:16 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:17.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:16 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:17.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:16 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:17.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:16 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:17.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:16 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:19:17.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:16 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:17.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:16 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:17.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:16 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:17.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:16 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:17.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:16 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:17.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:16 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:19:17.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:16 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:17.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:16 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:17.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:16 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:17.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:16 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:17.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:16 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:17.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:16 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:19:17.565 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-09-13T14:19:16.364857+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":115,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385068,"kb_used_data":6224,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747576148,"statfs":{"total":767960285184,"available":765517975552,"internally_reserved":0,"allocated":6373376,"data_stored":3705253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002298"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-13T14:18:22.827988+0000","last_change":"2024-09-13T14:18:14.645599+0000","last_active":"2024-09-13T14:18:22.827988+0000","last_peered":"2024-09-13T14:18:22.827988+0000","last_clean":"2024-09-13T14:18:22.827988+0000","last_became_active":"2024-09-13T14:18:10.617927+0000","last_became_peered":"2024-09-13T14:18:10.617927+0000","last_unstale":"2024-09-13T14:18:22.827988+0000","last_undegraded":"2024-09-13T14:18:22.827988+0000","last_fullsized":"2024-09-13T14:18:22.827988+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:51:43.437167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-13T14:18:22.749240+0000","last_change":"2024-09-13T14:18:14.641734+0000","last_active":"2024-09-13T14:18:22.749240+0000","last_peered":"2024-09-13T14:18:22.749240+0000","last_clean":"2024-09-13T14:18:22.749240+0000","last_became_active":"2024-09-13T14:18:10.614303+0000","last_became_peered":"2024-09-13T14:18:10.614303+0000","last_unstale":"2024-09-13T14:18:22.749240+0000","last_undegraded":"2024-09-13T14:18:22.749240+0000","last_fullsized":"2024-09-13T14:18:22.749240+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:28:38.174245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034192599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-13T14:18:22.828560+0000","last_change":"2024-09-13T14:18:14.639101+0000","last_active":"2024-09-13T14:18:22.828560+0000","last_peered":"2024-09-13T14:18:22.828560+0000","last_clean":"2024-09-13T14:18:22.828560+0000","last_became_active":"2024-09-13T14:18:10.718655+0000","last_became_peered":"2024-09-13T14:18:10.718655+0000","last_unstale":"2024-09-13T14:18:22.828560+0000","last_undegraded":"2024-09-13T14:18:22.828560+0000","last_fullsized":"2024-09-13T14:18:22.828560+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:59:01.637559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-13T14:18:22.749237+0000","last_change":"2024-09-13T14:18:14.641641+0000","last_active":"2024-09-13T14:18:22.749237+0000","last_peered":"2024-09-13T14:18:22.749237+0000","last_clean":"2024-09-13T14:18:22.749237+0000","last_became_active":"2024-09-13T14:18:10.718349+0000","last_became_peered":"2024-09-13T14:18:10.718349+0000","last_unstale":"2024-09-13T14:18:22.749237+0000","last_undegraded":"2024-09-13T14:18:22.749237+0000","last_fullsized":"2024-09-13T14:18:22.749237+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:23.005181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":21,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-13T14:18:22.828079+0000","last_change":"2024-09-13T14:18:14.644865+0000","last_active":"2024-09-13T14:18:22.828079+0000","last_peered":"2024-09-13T14:18:22.828079+0000","last_clean":"2024-09-13T14:18:22.828079+0000","last_became_active":"2024-09-13T14:18:10.617018+0000","last_became_peered":"2024-09-13T14:18:10.617018+0000","last_unstale":"2024-09-13T14:18:22.828079+0000","last_undegraded":"2024-09-13T14:18:22.828079+0000","last_fullsized":"2024-09-13T14:18:22.828079+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:01:22.505985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-13T14:18:22.750113+0000","last_change":"2024-09-13T14:18:14.641344+0000","last_active":"2024-09-13T14:18:22.750113+0000","last_peered":"2024-09-13T14:18:22.750113+0000","last_clean":"2024-09-13T14:18:22.750113+0000","last_became_active":"2024-09-13T14:18:10.766103+0000","last_became_peered":"2024-09-13T14:18:10.766103+0000","last_unstale":"2024-09-13T14:18:22.750113+0000","last_undegraded":"2024-09-13T14:18:22.750113+0000","last_fullsized":"2024-09-13T14:18:22.750113+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:03.456609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-13T14:18:22.827907+0000","last_change":"2024-09-13T14:18:14.638950+0000","last_active":"2024-09-13T14:18:22.827907+0000","last_peered":"2024-09-13T14:18:22.827907+0000","last_clean":"2024-09-13T14:18:22.827907+0000","last_became_active":"2024-09-13T14:18:10.718659+0000","last_became_peered":"2024-09-13T14:18:10.718659+0000","last_unstale":"2024-09-13T14:18:22.827907+0000","last_undegraded":"2024-09-13T14:18:22.827907+0000","last_fullsized":"2024-09-13T14:18:22.827907+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:38:08.688330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":20,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-13T14:18:22.827822+0000","last_change":"2024-09-13T14:18:14.644401+0000","last_active":"2024-09-13T14:18:22.827822+0000","last_peered":"2024-09-13T14:18:22.827822+0000","last_clean":"2024-09-13T14:18:22.827822+0000","last_became_active":"2024-09-13T14:18:10.616981+0000","last_became_peered":"2024-09-13T14:18:10.616981+0000","last_unstale":"2024-09-13T14:18:22.827822+0000","last_undegraded":"2024-09-13T14:18:22.827822+0000","last_fullsized":"2024-09-13T14:18:22.827822+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:00:32.657637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034405999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"57'79","reported_seq":48,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T14:18:20.411380+0000","last_change":"2024-09-13T14:17:42.845082+0000","last_active":"2024-09-13T14:18:20.411380+0000","last_peered":"2024-09-13T14:18:20.411380+0000","last_clean":"2024-09-13T14:18:20.411380+0000","last_became_active":"2024-09-13T14:17:42.839116+0000","last_became_peered":"2024-09-13T14:17:42.839116+0000","last_unstale":"2024-09-13T14:18:20.411380+0000","last_undegraded":"2024-09-13T14:18:20.411380+0000","last_fullsized":"2024-09-13T14:18:20.411380+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:15:42.069756+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:58.144190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430229,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298500,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446652,"statfs":{"total":95995035648,"available":95689371648,"internally_reserved":0,"allocated":1171456,"data_stored":832470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.52600000000000002}]},{"osd":1,"last update":"Fri Sep 13 14:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.379}]},{"osd":2,"last update":"Fri Sep 13 14:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.32200000000000001}]},{"osd":3,"last update":"Fri Sep 13 14:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.40899999999999997}]},{"osd":4,"last update":"Fri Sep 13 14:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.433}]},{"osd":5,"last update":"Fri Sep 13 14:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.53700000000000003}]},{"osd":6,"last update":"Fri Sep 13 14:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.503}]}]},{"osd":6,"up_from":42,"seq":180388626458,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298496,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446656,"statfs":{"total":95995035648,"available":95689375744,"internally_reserved":0,"allocated":1167360,"data_stored":831930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.30299999999999999}]},{"osd":1,"last update":"Fri Sep 13 14:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.503},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.31900000000000001}]},{"osd":2,"last update":"Fri Sep 13 14:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.45900000000000002}]},{"osd":3,"last update":"Fri Sep 13 14:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.36199999999999999}]},{"osd":4,"last update":"Fri Sep 13 14:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.378}]},{"osd":5,"last update":"Fri Sep 13 14:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.44600000000000001}]},{"osd":7,"last update":"Fri Sep 13 14:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.59599999999999997}]}]},{"osd":5,"up_from":36,"seq":154618822687,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297916,"kb_used_data":560,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447236,"statfs":{"total":95995035648,"available":95689969664,"internally_reserved":0,"allocated":573440,"data_stored":241562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:18:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52600000000000002,"15min":0.52700000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.498,"15min":0.496},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.44900000000000001}]},{"osd":1,"last update":"Fri Sep 13 14:18:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52100000000000002,"15min":0.52000000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.51800000000000002,"15min":0.51600000000000001},"min":{"1min":0.35799999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.434}]},{"osd":2,"last update":"Fri Sep 13 14:18:56 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.35299999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52300000000000002,"15min":0.52200000000000002},"min":{"1min":0.35699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.49199999999999999}]},{"osd":3,"last update":"Fri Sep 13 14:18:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.50600000000000001,"15min":0.505},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.46600000000000003}]},{"osd":4,"last update":"Fri Sep 13 14:18:56 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.38,"15min":0.371},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.314},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.38400000000000001,"15min":0.374},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.32500000000000001}]},{"osd":6,"last update":"Fri Sep 13 14:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.35699999999999998}]},{"osd":7,"last update":"Fri Sep 13 14:18:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.34599999999999997}]}]},{"osd":4,"up_from":30,"seq":128849018915,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297916,"kb_used_data":560,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447236,"statfs":{"total":95995035648,"available":95689969664,"internally_reserved":0,"allocated":573440,"data_stored":241858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:18:36 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.53300000000000003,"15min":0.54100000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.72799999999999998,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.621},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55800000000000005,"15min":0.56100000000000005},"min":{"1min":0.38200000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.78600000000000003,"5min":0.88,"15min":0.88},"last":0.60999999999999999}]},{"osd":1,"last update":"Fri Sep 13 14:18:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53900000000000003,"15min":0.54200000000000004},"min":{"1min":0.36899999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.72399999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.55500000000000005,"15min":0.56100000000000005},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.74099999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.67000000000000004}]},{"osd":2,"last update":"Fri Sep 13 14:18:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.55200000000000005,"15min":0.55600000000000005},"min":{"1min":0.39400000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.69299999999999995,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.53000000000000003,"15min":0.53400000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70899999999999996,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.59999999999999998}]},{"osd":3,"last update":"Fri Sep 13 14:18:36 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.53700000000000003,"15min":0.54000000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.75700000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.495,"5min":0.53200000000000003,"15min":0.53800000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.68100000000000005,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.56599999999999995}]},{"osd":5,"last update":"Fri Sep 13 14:18:56 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.378,"15min":0.375},"min":{"1min":0.30599999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.60299999999999998,"5min":0.621,"15min":0.621},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.41799999999999998,"15min":0.41299999999999998},"min":{"1min":0.32200000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.63900000000000001,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.44600000000000001}]},{"osd":6,"last update":"Fri Sep 13 14:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.49399999999999999}]},{"osd":7,"last update":"Fri Sep 13 14:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.54900000000000004}]}]},{"osd":3,"up_from":25,"seq":107374182441,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297916,"kb_used_data":560,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447236,"statfs":{"total":95995035648,"available":95689969664,"internally_reserved":0,"allocated":573440,"data_stored":241858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:19:12 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.39400000000000002,"15min":0.40799999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.46899999999999997,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.38500000000000001,"15min":0.39700000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.60399999999999998,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.41899999999999998}]},{"osd":1,"last update":"Fri Sep 13 14:19:12 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.38,"15min":0.39200000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.47499999999999998,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.432,"15min":0.44600000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.55600000000000005,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.34000000000000002}]},{"osd":2,"last update":"Fri Sep 13 14:19:12 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.44500000000000001,"15min":0.45700000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.56699999999999995,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.311},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.41299999999999998,"15min":0.42899999999999999},"min":{"1min":0.26900000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.52500000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.379}]},{"osd":4,"last update":"Fri Sep 13 14:18:30 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.52200000000000002,"15min":0.52500000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.74099999999999999,"5min":0.748,"15min":0.748},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.56200000000000006,"15min":0.56699999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.71999999999999997,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.52100000000000002}]},{"osd":5,"last update":"Fri Sep 13 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.53800000000000003,"15min":0.54400000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.66600000000000004,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.54800000000000004,"15min":0.55300000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.746,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.53700000000000003}]},{"osd":6,"last update":"Fri Sep 13 14:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.497}]},{"osd":7,"last update":"Fri Sep 13 14:18:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.50600000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297908,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447244,"statfs":{"total":95995035648,"available":95689977856,"internally_reserved":0,"allocated":569344,"data_stored":241614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:18:49 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.39400000000000002,"15min":0.38400000000000001},"min":{"1min":0.27100000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.59099999999999997,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.376,"15min":0.36199999999999999},"min":{"1min":0.22800000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.58899999999999997,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.39000000000000001}]},{"osd":1,"last update":"Fri Sep 13 14:18:49 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.39700000000000002,"15min":0.38500000000000001},"min":{"1min":0.27600000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.53200000000000003,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.40699999999999997,"15min":0.39400000000000002},"min":{"1min":0.27800000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.46700000000000003}]},{"osd":3,"last update":"Fri Sep 13 14:19:13 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45200000000000001,"15min":0.45600000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.439},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.45500000000000002,"15min":0.45700000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.51300000000000001}]},{"osd":4,"last update":"Fri Sep 13 14:18:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.55800000000000005,"15min":0.56200000000000006},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69199999999999995,"5min":0.751,"15min":0.751},"last":0.5},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.59199999999999997,"15min":0.59599999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.747,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.60499999999999998}]},{"osd":5,"last update":"Fri Sep 13 14:18:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.62,"15min":0.63},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76700000000000002,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.628},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58999999999999997,"15min":0.59599999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.56200000000000006}]},{"osd":6,"last update":"Fri Sep 13 14:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.54000000000000004}]},{"osd":7,"last update":"Fri Sep 13 14:18:41 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.58699999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297920,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447232,"statfs":{"total":95995035648,"available":95689965568,"internally_reserved":0,"allocated":577536,"data_stored":241806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.33100000000000002,"15min":0.308},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.47199999999999998,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.34899999999999998,"15min":0.32000000000000001},"min":{"1min":0.26200000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.58299999999999996,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.622}]},{"osd":2,"last update":"Fri Sep 13 14:18:46 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.39900000000000002,"15min":0.39100000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.502,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.38300000000000001,"15min":0.36599999999999999},"min":{"1min":0.27300000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.58599999999999997}]},{"osd":3,"last update":"Fri Sep 13 14:19:11 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.45200000000000001,"15min":0.45400000000000001},"min":{"1min":0.27700000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.65600000000000003}]},{"osd":4,"last update":"Fri Sep 13 14:18:32 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54000000000000004,"15min":0.53700000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.65800000000000003,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.80100000000000005}]},{"osd":5,"last update":"Fri Sep 13 14:18:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.55800000000000005,"15min":0.56399999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.67700000000000005,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.752},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.59399999999999997,"15min":0.60199999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.76200000000000001}]},{"osd":6,"last update":"Fri Sep 13 14:18:23 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.78800000000000003}]},{"osd":7,"last update":"Fri Sep 13 14:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.85099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298496,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446656,"statfs":{"total":95995035648,"available":95689375744,"internally_reserved":0,"allocated":1167360,"data_stored":832155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 14:18:19 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.315,"15min":0.30299999999999999},"min":{"1min":0.22600000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.46100000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.371,"5min":0.34999999999999998,"15min":0.33000000000000002},"min":{"1min":0.24099999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52800000000000002,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.20200000000000001}]},{"osd":2,"last update":"Fri Sep 13 14:18:47 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39200000000000002,"15min":0.38800000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.372,"15min":0.36599999999999999},"min":{"1min":0.26000000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.29799999999999999}]},{"osd":3,"last update":"Fri Sep 13 14:18:14 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.44400000000000001,"15min":0.44800000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.312},{"interface":"front","average":{"1min":0.371,"5min":0.38800000000000001,"15min":0.39100000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.253}]},{"osd":4,"last update":"Fri Sep 13 14:18:38 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.51000000000000001,"15min":0.50600000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.498,"15min":0.497},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.36599999999999999}]},{"osd":5,"last update":"Fri Sep 13 14:18:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.35899999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.55200000000000005,"15min":0.54700000000000004},"min":{"1min":0.42799999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.35699999999999998}]},{"osd":6,"last update":"Fri Sep 13 14:18:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.378},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.38500000000000001}]},{"osd":7,"last update":"Fri Sep 13 14:18:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.433}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:19:17.567 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-13T14:19:17.567 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-13T14:19:17.567 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-13T14:19:17.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-13T14:19:17.702 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:19:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:19:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:19:17.902 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:19:18.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:17 smithi184 ceph-mon[33394]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:18.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:17 smithi184 ceph-mon[33394]: from='client.24518 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:19:18.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:17 smithi151 ceph-mon[41387]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:18.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:17 smithi151 ceph-mon[41387]: from='client.24518 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:19:18.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:17 smithi151 ceph-mon[29807]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:18.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:17 smithi151 ceph-mon[29807]: from='client.24518 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:19:19.319 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:18 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:19:19.319 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:18 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:19.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:18 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:19:19.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:18 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:19.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:18 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:19:19.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:18 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:20.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:19 smithi151 ceph-mon[29807]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:20.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:19 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:20.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:19 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:20.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:19 smithi151 ceph-mon[41387]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:20.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:19 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:20.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:19 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:20.112 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:19:20.112 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T14:19:20.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:19 smithi184 ceph-mon[33394]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:20.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:19 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:20.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:19 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:21.212 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:20 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/3208067882' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-13T14:19:21.212 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:20 smithi151 ceph-mon[29807]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:21.213 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:20 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/3208067882' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-13T14:19:21.213 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:20 smithi151 ceph-mon[41387]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:21.238 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-13T14:19:21.238 INFO:tasks.cephadm:Setup complete, yielding 2024-09-13T14:19:21.238 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-13T14:19:21.252 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi151.front.sepia.ceph.com 2024-09-13T14:19:21.252 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- bash -c 'ceph fs volume create foo' 2024-09-13T14:19:21.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:20 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/3208067882' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-13T14:19:21.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:20 smithi184 ceph-mon[33394]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:21.815 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:19:23.033 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:22 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:23.033 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:22 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:23.033 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:22 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:23.033 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:22 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:19:23.033 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:22 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:23.033 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:22 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:23.033 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:22 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:23.033 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:22 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:23.034 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:22 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:19:23.034 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:22 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:22 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:22 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:22 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:22 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:19:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:22 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:23.968 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:23 smithi151 ceph-mon[29807]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-13T14:19:23.968 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:23 smithi151 ceph-mon[29807]: Reconfiguring daemon alertmanager.a on smithi151 2024-09-13T14:19:23.968 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:23 smithi151 ceph-mon[29807]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:23.968 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:23 smithi151 ceph-mon[41387]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-13T14:19:23.968 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:23 smithi151 ceph-mon[41387]: Reconfiguring daemon alertmanager.a on smithi151 2024-09-13T14:19:23.969 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:23 smithi151 ceph-mon[41387]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:24.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:23 smithi184 ceph-mon[33394]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-13T14:19:24.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:23 smithi184 ceph-mon[33394]: Reconfiguring daemon alertmanager.a on smithi151 2024-09-13T14:19:24.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:23 smithi184 ceph-mon[33394]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:25.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:24 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-13T14:19:25.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:24 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-13T14:19:25.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:24 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-13T14:19:26.340 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:25 smithi151 ceph-mon[41387]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:26.340 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:25 smithi151 ceph-mon[41387]: from='client.24530 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:19:26.340 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:25 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-09-13T14:19:26.340 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:25 smithi151 ceph-mon[41387]: osdmap e59: 8 total, 8 up, 8 in 2024-09-13T14:19:26.340 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:25 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-13T14:19:26.341 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:25 smithi151 ceph-mon[29807]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:26.341 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:25 smithi151 ceph-mon[29807]: from='client.24530 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:19:26.341 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:25 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-09-13T14:19:26.341 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:25 smithi151 ceph-mon[29807]: osdmap e59: 8 total, 8 up, 8 in 2024-09-13T14:19:26.341 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:25 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-13T14:19:26.341 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:25 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a[29784]: 2024-09-13T14:19:25.867+0000 7fab32340640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T14:19:26.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:25 smithi184 ceph-mon[33394]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:26.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:25 smithi184 ceph-mon[33394]: from='client.24530 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:19:26.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:25 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-09-13T14:19:26.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:25 smithi184 ceph-mon[33394]: osdmap e59: 8 total, 8 up, 8 in 2024-09-13T14:19:26.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:25 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-13T14:19:26.657 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-09-13T14:19:27.027 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:26 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-09-13T14:19:27.027 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:26 smithi151 ceph-mon[29807]: osdmap e60: 8 total, 8 up, 8 in 2024-09-13T14:19:27.027 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:26 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-13T14:19:27.027 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:26 smithi151 ceph-mon[29807]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T14:19:27.027 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:26 smithi151 ceph-mon[29807]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-13T14:19:27.027 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:26 smithi151 ceph-mon[29807]: osdmap e61: 8 total, 8 up, 8 in 2024-09-13T14:19:27.027 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:26 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-09-13T14:19:27.027 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:26 smithi151 ceph-mon[29807]: fsmap foo:0 2024-09-13T14:19:27.027 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:26 smithi151 ceph-mon[29807]: Saving service mds.foo spec with placement count:2 2024-09-13T14:19:27.027 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:26 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:27.027 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:26 smithi151 ceph-mon[29807]: pgmap v41: 73 pgs: 64 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:27.028 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:26 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-09-13T14:19:27.028 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:26 smithi151 ceph-mon[41387]: osdmap e60: 8 total, 8 up, 8 in 2024-09-13T14:19:27.028 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:26 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-13T14:19:27.028 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:26 smithi151 ceph-mon[41387]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T14:19:27.028 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:26 smithi151 ceph-mon[41387]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-13T14:19:27.028 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:26 smithi151 ceph-mon[41387]: osdmap e61: 8 total, 8 up, 8 in 2024-09-13T14:19:27.028 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:26 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-09-13T14:19:27.028 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:26 smithi151 ceph-mon[41387]: fsmap foo:0 2024-09-13T14:19:27.028 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:26 smithi151 ceph-mon[41387]: Saving service mds.foo spec with placement count:2 2024-09-13T14:19:27.028 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:26 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:27.028 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:26 smithi151 ceph-mon[41387]: pgmap v41: 73 pgs: 64 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:27.251 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:19:27.318 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:19:27 smithi151 systemd[1]: Stopping Ceph alertmanager.a for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:19:27.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:26 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-09-13T14:19:27.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:26 smithi184 ceph-mon[33394]: osdmap e60: 8 total, 8 up, 8 in 2024-09-13T14:19:27.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:26 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-13T14:19:27.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:26 smithi184 ceph-mon[33394]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T14:19:27.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:26 smithi184 ceph-mon[33394]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-13T14:19:27.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:26 smithi184 ceph-mon[33394]: osdmap e61: 8 total, 8 up, 8 in 2024-09-13T14:19:27.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:26 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-09-13T14:19:27.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:26 smithi184 ceph-mon[33394]: fsmap foo:0 2024-09-13T14:19:27.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:26 smithi184 ceph-mon[33394]: Saving service mds.foo spec with placement count:2 2024-09-13T14:19:27.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:26 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:27.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:26 smithi184 ceph-mon[33394]: pgmap v41: 73 pgs: 64 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:27.853 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:19:27 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:19:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:19:28.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:27 smithi151 ceph-mon[29807]: osdmap e62: 8 total, 8 up, 8 in 2024-09-13T14:19:28.109 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:19:27 smithi151 podman[78581]: 2024-09-13 14:19:27.763816247 +0000 UTC m=+0.562135824 container died 2d6e4859432ecf6a59f94b1ed389485c3aa17be7bbe0565bb04e9c67599d2108 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=sweet_gauss, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=quincy-bbdadfa, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-13T14:19:28.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:27 smithi151 ceph-mon[41387]: osdmap e62: 8 total, 8 up, 8 in 2024-09-13T14:19:28.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:27 smithi184 ceph-mon[33394]: osdmap e62: 8 total, 8 up, 8 in 2024-09-13T14:19:29.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:28 smithi151 ceph-mon[29807]: osdmap e63: 8 total, 8 up, 8 in 2024-09-13T14:19:29.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:28 smithi151 ceph-mon[29807]: pgmap v44: 73 pgs: 64 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:29.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:28 smithi151 ceph-mon[41387]: osdmap e63: 8 total, 8 up, 8 in 2024-09-13T14:19:29.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:28 smithi151 ceph-mon[41387]: pgmap v44: 73 pgs: 64 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:29.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:28 smithi184 ceph-mon[33394]: osdmap e63: 8 total, 8 up, 8 in 2024-09-13T14:19:29.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:28 smithi184 ceph-mon[33394]: pgmap v44: 73 pgs: 64 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:29.544 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:19:29 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[74160]: ts=2024-09-13T14:19:29.110Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-13T14:19:29.930 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-09-13T14:19:30.286 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:19:29 smithi151 podman[78738]: 2024-09-13 14:19:29.931179667 +0000 UTC m=+1.049716931 container died af9bba3abbf6956f994b2cf447b9177b10ae2470f558ef210b0b3180280aa245 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:19:30.604 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:19:30 smithi151 podman[78738]: 2024-09-13 14:19:30.394099443 +0000 UTC m=+1.512636711 container cleanup af9bba3abbf6956f994b2cf447b9177b10ae2470f558ef210b0b3180280aa245 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:19:30.604 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:19:30 smithi151 bash[78738]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a 2024-09-13T14:19:30.821 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:19:30.858 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:19:30 smithi151 podman[78778]: 2024-09-13 14:19:30.603879765 +0000 UTC m=+1.400174311 container remove af9bba3abbf6956f994b2cf447b9177b10ae2470f558ef210b0b3180280aa245 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:19:30.858 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:19:30 smithi151 podman[78778]: 2024-09-13 14:19:30.70460885 +0000 UTC m=+1.500903392 volume remove 8c8518b5f8feadfbacd7f22e68b590102d7d5a3f230d44a6759abcd71674a6f7 2024-09-13T14:19:31.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:31 smithi151 ceph-mon[29807]: pgmap v45: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:31.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:31 smithi151 ceph-mon[41387]: pgmap v45: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:31.609 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:19:31 smithi151 podman[78861]: 2024-09-13 14:19:31.311213133 +0000 UTC m=+0.890484402 container died 051facc4b1599d30ec0fb362feffd32db2e7e41f55048d35ef677c1a7229e6da (image=quay.ceph.io/ceph-ci/ceph:quincy, name=epic_noyce, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-bbdadfa, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-13T14:19:31.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:31 smithi184 ceph-mon[33394]: pgmap v45: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:32.358 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:19:32 smithi151 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-13T14:19:32.359 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:19:32 smithi151 systemd[1]: Stopped Ceph alertmanager.a for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:19:32.359 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:19:32 smithi151 systemd[1]: Starting Ceph alertmanager.a for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:19:33.107 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:19:32 smithi151 podman[79079]: 2024-09-13 14:19:32.660549401 +0000 UTC m=+0.017048064 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-13T14:19:33.107 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:19:32 smithi151 podman[79079]: 2024-09-13 14:19:32.804089437 +0000 UTC m=+0.160588087 volume create 14f9be8b814c23c16178f3a4847a8740a35d5a5bf17b0ce705dddbc73d712aa6 2024-09-13T14:19:33.107 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:19:32 smithi151 podman[79079]: 2024-09-13 14:19:32.896440694 +0000 UTC m=+0.252939388 container create 457393043edc4bd5df29aa2fcd4a3f2776d9b312de200510d3381fb081fa6130 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:19:33.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:33 smithi184 ceph-mon[33394]: pgmap v46: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:33.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:33 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:33.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:33 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:19:33.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:33 smithi151 ceph-mon[41387]: pgmap v46: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:33.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:33 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:33.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:33 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:19:33.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:33 smithi151 ceph-mon[29807]: pgmap v46: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:33.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:33 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:33.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:33 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:19:33.917 INFO:teuthology.run_tasks:Running task ceph.healthy... 2024-09-13T14:19:33.928 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-13T14:19:33.928 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-13T14:19:33.928 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-13T14:19:34.157 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:19:34 smithi151 podman[79079]: 2024-09-13 14:19:34.039420389 +0000 UTC m=+1.395919042 container init 457393043edc4bd5df29aa2fcd4a3f2776d9b312de200510d3381fb081fa6130 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:19:34.157 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:19:34 smithi151 podman[79079]: 2024-09-13 14:19:34.044970457 +0000 UTC m=+1.401469118 container start 457393043edc4bd5df29aa2fcd4a3f2776d9b312de200510d3381fb081fa6130 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:19:34.157 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:19:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[79108]: ts=2024-09-13T14:19:34.057Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-13T14:19:34.157 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:19:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[79108]: ts=2024-09-13T14:19:34.057Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-13T14:19:34.157 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:19:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[79108]: ts=2024-09-13T14:19:34.058Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.151 port=9094 2024-09-13T14:19:34.157 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:19:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[79108]: ts=2024-09-13T14:19:34.059Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-13T14:19:34.157 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:19:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[79108]: ts=2024-09-13T14:19:34.082Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-13T14:19:34.157 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:19:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[79108]: ts=2024-09-13T14:19:34.082Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-13T14:19:34.157 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:19:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[79108]: ts=2024-09-13T14:19:34.086Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-13T14:19:34.157 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:19:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[79108]: ts=2024-09-13T14:19:34.086Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-13T14:19:34.157 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:19:34 smithi151 bash[79079]: 457393043edc4bd5df29aa2fcd4a3f2776d9b312de200510d3381fb081fa6130 2024-09-13T14:19:34.244 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:19:34.498 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:19:34 smithi151 systemd[1]: Started Ceph alertmanager.a for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:19:35.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:35 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:35.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:35 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:35.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:35 smithi151 ceph-mon[29807]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:19:35.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:35 smithi151 ceph-mon[29807]: Reconfiguring daemon prometheus.a on smithi184 2024-09-13T14:19:35.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:35 smithi151 ceph-mon[29807]: pgmap v47: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:35.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:35 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:35.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:35 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:35.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:35 smithi151 ceph-mon[41387]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:19:35.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:35 smithi151 ceph-mon[41387]: Reconfiguring daemon prometheus.a on smithi184 2024-09-13T14:19:35.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:35 smithi151 ceph-mon[41387]: pgmap v47: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:35.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:35 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:35.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:35 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:35.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:35 smithi184 ceph-mon[33394]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:19:35.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:35 smithi184 ceph-mon[33394]: Reconfiguring daemon prometheus.a on smithi184 2024-09-13T14:19:35.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:35 smithi184 ceph-mon[33394]: pgmap v47: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:36.129 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:19:36.195 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:19:36 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[79108]: ts=2024-09-13T14:19:36.060Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000602908s 2024-09-13T14:19:36.343 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:36 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/3983428616' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-13T14:19:36.480 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:36 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/3983428616' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-13T14:19:36.480 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:36 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/3983428616' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-13T14:19:36.851 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":20,"flags":0,"active_gid":14460,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":4025698662},{"type":"v1","addr":"172.21.15.151:6801","nonce":4025698662}]},"active_addr":"172.21.15.151:6801/4025698662","active_change":"2024-09-13T14:18:18.344950+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":24352,"name":"x","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.151:8443/","prometheus":"http://172.21.15.151:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":57,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":514781427}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":2201280183}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":1530349337}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":1488764319}]}]} 2024-09-13T14:19:36.855 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-13T14:19:36.855 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-13T14:19:36.855 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T14:19:37.100 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:19:37.369 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:37 smithi184 ceph-mon[33394]: pgmap v48: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:37.370 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:37 smithi184 systemd[1]: Stopping Ceph prometheus.a for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:19:37.487 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:37 smithi151 ceph-mon[29807]: pgmap v48: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:37.487 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:37 smithi151 ceph-mon[41387]: pgmap v48: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:37.726 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:19:37 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:19:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:19:37.726 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:37 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:19:37.418Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-13T14:19:37.726 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:37 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:19:37.418Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-13T14:19:37.726 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:37 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:19:37.418Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-13T14:19:37.727 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:37 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:19:37.418Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-13T14:19:37.727 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:37 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:19:37.418Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-13T14:19:37.727 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:37 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:19:37.418Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-13T14:19:37.727 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:37 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:19:37.418Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-13T14:19:37.727 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:37 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:19:37.418Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-13T14:19:37.727 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:37 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:19:37.418Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-13T14:19:37.727 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:37 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:19:37.418Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-13T14:19:37.727 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:37 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:19:37.418Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-13T14:19:37.727 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:37 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[62942]: ts=2024-09-13T14:19:37.419Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-13T14:19:37.727 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:37 smithi184 podman[66052]: 2024-09-13 14:19:37.42884229 +0000 UTC m=+0.108599912 container died 66ca574fe455da469e6bad66c813569bc44f1cb745339f7b542020aaa780ebf9 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:19:38.059 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:37 smithi184 podman[66052]: 2024-09-13 14:19:37.841151155 +0000 UTC m=+0.520908767 container cleanup 66ca574fe455da469e6bad66c813569bc44f1cb745339f7b542020aaa780ebf9 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:19:38.059 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:37 smithi184 bash[66052]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a 2024-09-13T14:19:38.321 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 podman[66068]: 2024-09-13 14:19:38.059023377 +0000 UTC m=+0.629333929 container remove 66ca574fe455da469e6bad66c813569bc44f1cb745339f7b542020aaa780ebf9 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:19:38.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-13T14:19:38.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 systemd[1]: Stopped Ceph prometheus.a for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:19:38.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 1.114s CPU time. 2024-09-13T14:19:38.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 systemd[1]: Starting Ceph prometheus.a for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:19:38.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 podman[66165]: 2024-09-13 14:19:38.430311505 +0000 UTC m=+0.016558944 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-13T14:19:38.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 podman[66165]: 2024-09-13 14:19:38.551747571 +0000 UTC m=+0.137995010 container create 0d5bc4586d6c6033533a937189a66d9827ed310b742b32352b35774eb128a2ad (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:19:39.041 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:19:39.041 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":63,"fsid":"2401fc50-71da-11ef-bceb-c7b262605968","created":"2024-09-13T14:12:36.168205+0000","modified":"2024-09-13T14:19:27.879672+0000","last_up_change":"2024-09-13T14:17:40.817515+0000","last_in_change":"2024-09-13T14:17:24.771724+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:15:41.780748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:18:09.030957+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":56,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T14:19:24.791384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T14:19:25.101452+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}}}],"osds":[{"osd":0,"uuid":"0ad945ee-af7d-4f26-a1e9-18ed963194ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":551854295},{"type":"v1","addr":"172.21.15.151:6803","nonce":551854295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":551854295},{"type":"v1","addr":"172.21.15.151:6805","nonce":551854295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":551854295},{"type":"v1","addr":"172.21.15.151:6809","nonce":551854295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":551854295},{"type":"v1","addr":"172.21.15.151:6807","nonce":551854295}]},"public_addr":"172.21.15.151:6803/551854295","cluster_addr":"172.21.15.151:6805/551854295","heartbeat_back_addr":"172.21.15.151:6809/551854295","heartbeat_front_addr":"172.21.15.151:6807/551854295","state":["exists","up"]},{"osd":1,"uuid":"09f5d705-684f-4e86-80a2-ec2292af09d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":3674059846},{"type":"v1","addr":"172.21.15.151:6811","nonce":3674059846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3674059846},{"type":"v1","addr":"172.21.15.151:6813","nonce":3674059846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3674059846},{"type":"v1","addr":"172.21.15.151:6817","nonce":3674059846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":3674059846},{"type":"v1","addr":"172.21.15.151:6815","nonce":3674059846}]},"public_addr":"172.21.15.151:6811/3674059846","cluster_addr":"172.21.15.151:6813/3674059846","heartbeat_back_addr":"172.21.15.151:6817/3674059846","heartbeat_front_addr":"172.21.15.151:6815/3674059846","state":["exists","up"]},{"osd":2,"uuid":"6a63f8af-ff69-46bb-830f-03ea89fc78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":1646754940},{"type":"v1","addr":"172.21.15.151:6819","nonce":1646754940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":1646754940},{"type":"v1","addr":"172.21.15.151:6821","nonce":1646754940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1646754940},{"type":"v1","addr":"172.21.15.151:6825","nonce":1646754940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":1646754940},{"type":"v1","addr":"172.21.15.151:6823","nonce":1646754940}]},"public_addr":"172.21.15.151:6819/1646754940","cluster_addr":"172.21.15.151:6821/1646754940","heartbeat_back_addr":"172.21.15.151:6825/1646754940","heartbeat_front_addr":"172.21.15.151:6823/1646754940","state":["exists","up"]},{"osd":3,"uuid":"770e8334-3b3a-46bc-9782-964b2a696660","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2203980888},{"type":"v1","addr":"172.21.15.151:6827","nonce":2203980888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":2203980888},{"type":"v1","addr":"172.21.15.151:6829","nonce":2203980888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":2203980888},{"type":"v1","addr":"172.21.15.151:6833","nonce":2203980888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":2203980888},{"type":"v1","addr":"172.21.15.151:6831","nonce":2203980888}]},"public_addr":"172.21.15.151:6827/2203980888","cluster_addr":"172.21.15.151:6829/2203980888","heartbeat_back_addr":"172.21.15.151:6833/2203980888","heartbeat_front_addr":"172.21.15.151:6831/2203980888","state":["exists","up"]},{"osd":4,"uuid":"75e258c0-1145-4aee-86aa-db2b63082cc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6801","nonce":3275553935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6803","nonce":3275553935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6807","nonce":3275553935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6805","nonce":3275553935}]},"public_addr":"172.21.15.184:6801/3275553935","cluster_addr":"172.21.15.184:6803/3275553935","heartbeat_back_addr":"172.21.15.184:6807/3275553935","heartbeat_front_addr":"172.21.15.184:6805/3275553935","state":["exists","up"]},{"osd":5,"uuid":"f56a5366-9d22-4960-933d-340aa280fbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6809","nonce":2446419621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6811","nonce":2446419621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6815","nonce":2446419621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6813","nonce":2446419621}]},"public_addr":"172.21.15.184:6809/2446419621","cluster_addr":"172.21.15.184:6811/2446419621","heartbeat_back_addr":"172.21.15.184:6815/2446419621","heartbeat_front_addr":"172.21.15.184:6813/2446419621","state":["exists","up"]},{"osd":6,"uuid":"e9826568-a728-45ab-8180-9777fe31f93a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6817","nonce":2214837203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6819","nonce":2214837203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6823","nonce":2214837203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6821","nonce":2214837203}]},"public_addr":"172.21.15.184:6817/2214837203","cluster_addr":"172.21.15.184:6819/2214837203","heartbeat_back_addr":"172.21.15.184:6823/2214837203","heartbeat_front_addr":"172.21.15.184:6821/2214837203","state":["exists","up"]},{"osd":7,"uuid":"d76869fe-5121-4467-8b56-1aadef0eaf65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6825","nonce":3378735997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6827","nonce":3378735997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6831","nonce":3378735997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6829","nonce":3378735997}]},"public_addr":"172.21.15.184:6825/3378735997","cluster_addr":"172.21.15.184:6827/3378735997","heartbeat_back_addr":"172.21.15.184:6831/3378735997","heartbeat_front_addr":"172.21.15.184:6829/3378735997","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:14:49.835715+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:15:14.682882+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:15:39.423890+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:16:04.527460+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:16:27.283661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:16:51.893868+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:16.152592+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:39.763533+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.151:6800/3383065173":"2024-09-14T14:13:04.761383+0000","172.21.15.151:6801/3383065173":"2024-09-14T14:13:04.761383+0000","172.21.15.151:0/3946777511":"2024-09-14T14:13:04.761383+0000","172.21.15.151:0/624042464":"2024-09-14T14:13:04.761383+0000","172.21.15.151:6801/1057983082":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/1182681341":"2024-09-14T14:13:04.761383+0000","172.21.15.151:0/1427922964":"2024-09-14T14:13:27.478847+0000","172.21.15.151:6800/1057983082":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/1351186656":"2024-09-14T14:18:18.344788+0000","172.21.15.151:6801/2335297718":"2024-09-14T14:18:18.344788+0000","172.21.15.151:0/2666829207":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/4115173791":"2024-09-14T14:18:18.344788+0000","172.21.15.151:6800/2335297718":"2024-09-14T14:18:18.344788+0000","172.21.15.151:0/3689756337":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/2800235450":"2024-09-14T14:18:18.344788+0000","172.21.15.151:0/2889050140":"2024-09-14T14:18:18.344788+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T14:19:39.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 podman[66165]: 2024-09-13 14:19:38.795555242 +0000 UTC m=+0.381802682 container init 0d5bc4586d6c6033533a937189a66d9827ed310b742b32352b35774eb128a2ad (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:19:39.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 podman[66165]: 2024-09-13 14:19:38.800071086 +0000 UTC m=+0.386318539 container start 0d5bc4586d6c6033533a937189a66d9827ed310b742b32352b35774eb128a2ad (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:19:39.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:19:38.823Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-13T14:19:39.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:19:38.823Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-13T14:19:39.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:19:38.823Z caller=main.go:570 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi184 (none))" 2024-09-13T14:19:39.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:19:38.823Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-13T14:19:39.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:19:38.823Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-13T14:19:39.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:19:38.824Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-13T14:19:39.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:19:38.826Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-13T14:19:39.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:19:38.827Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-13T14:19:39.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:19:38.827Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-13T14:19:39.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:19:38.828Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-13T14:19:39.105 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:19:38.828Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.509µs 2024-09-13T14:19:39.105 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:19:38.828Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-13T14:19:39.105 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:19:38.828Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-13T14:19:39.105 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:19:38.829Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-13T14:19:39.105 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:19:38.829Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=17.421µs wal_replay_duration=891.691µs wbl_replay_duration=171ns total_replay_duration=926.693µs 2024-09-13T14:19:39.105 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:19:38.830Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-13T14:19:39.105 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:19:38.830Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-13T14:19:39.105 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:19:38.830Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-13T14:19:39.105 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:19:38.844Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=14.021053ms db_storage=686ns remote_storage=1.014µs web_handler=324ns query_engine=575ns scrape=388.005µs scrape_sd=26.868µs notify=24.159µs notify_sd=6.203µs rules=13.346056ms tracing=6.06µs 2024-09-13T14:19:39.105 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:19:38.844Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-13T14:19:39.105 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:19:38.844Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-13T14:19:39.105 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 bash[66165]: 0d5bc4586d6c6033533a937189a66d9827ed310b742b32352b35774eb128a2ad 2024-09-13T14:19:39.106 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:19:38 smithi184 systemd[1]: Started Ceph prometheus.a for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:19:39.236 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:19:38 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:19:38] ENGINE Bus STOPPING 2024-09-13T14:19:39.236 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:19:39] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:19:39.236 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:19:39] ENGINE Bus STOPPED 2024-09-13T14:19:39.552 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:19:39] ENGINE Bus STARTING 2024-09-13T14:19:39.552 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:19:39] ENGINE Serving on http://:::9283 2024-09-13T14:19:39.553 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:19:39] ENGINE Bus STARTED 2024-09-13T14:19:39.553 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:19:39] ENGINE Bus STOPPING 2024-09-13T14:19:39.553 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-mon[29807]: pgmap v49: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:39.553 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:39.553 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:39.553 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:19:39.553 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi151.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T14:19:39.553 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:39.553 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:19:39.554 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi184.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T14:19:39.554 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:39.554 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:19:39.554 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi184.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T14:19:39.554 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:39.554 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:19:39.554 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/440457117' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:19:39.554 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-mon[41387]: pgmap v49: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:39.554 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:39.554 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:39.554 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:19:39.554 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi151.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T14:19:39.555 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:39.555 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:19:39.555 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi184.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T14:19:39.555 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:39.555 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:19:39.555 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi184.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T14:19:39.555 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:39.555 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:19:39.555 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/440457117' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:19:39.714 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:39 smithi184 ceph-mon[33394]: pgmap v49: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:39.714 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:39 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:39.715 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:39 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:39.715 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:39 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:19:39.715 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:39 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi151.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T14:19:39.715 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:39 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:39.715 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:39 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:19:39.715 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:39 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi184.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T14:19:39.715 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:39 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:39.715 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:39 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:19:39.715 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:39 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi184.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T14:19:39.715 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:39 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:39.715 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:39 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:19:39.715 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:39 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/440457117' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:19:40.018 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-13T14:19:40.018 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T14:19:40.108 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:19:39] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:19:40.108 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:19:39] ENGINE Bus STOPPED 2024-09-13T14:19:40.108 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:19:39] ENGINE Bus STARTING 2024-09-13T14:19:40.108 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:19:39] ENGINE Serving on http://:::9283 2024-09-13T14:19:40.108 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:19:39] ENGINE Bus STARTED 2024-09-13T14:19:40.108 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:19:39 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:19:39] ENGINE Bus STOPPING 2024-09-13T14:19:40.746 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:40 smithi184 ceph-mon[33394]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:19:40.746 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:40 smithi184 ceph-mon[33394]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi151.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T14:19:40.746 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:40 smithi184 ceph-mon[33394]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:19:40.746 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:40 smithi184 ceph-mon[33394]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi184.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T14:19:40.746 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:40 smithi184 ceph-mon[33394]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:19:40.746 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:40 smithi184 ceph-mon[33394]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi184.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T14:19:40.856 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:19:40.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:40 smithi151 ceph-mon[29807]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:19:40.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:40 smithi151 ceph-mon[29807]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi151.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T14:19:40.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:40 smithi151 ceph-mon[29807]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:19:40.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:40 smithi151 ceph-mon[29807]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi184.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T14:19:40.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:40 smithi151 ceph-mon[29807]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:19:40.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:40 smithi151 ceph-mon[29807]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi184.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T14:19:40.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:19:40 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:19:40] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:19:40.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:19:40 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:19:40] ENGINE Bus STOPPED 2024-09-13T14:19:40.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:19:40 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:19:40] ENGINE Bus STARTING 2024-09-13T14:19:40.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:19:40 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:19:40] ENGINE Serving on http://:::9283 2024-09-13T14:19:40.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:19:40 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:19:40] ENGINE Bus STARTED 2024-09-13T14:19:40.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:40 smithi151 ceph-mon[41387]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:19:40.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:40 smithi151 ceph-mon[41387]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi151.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T14:19:40.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:40 smithi151 ceph-mon[41387]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:19:40.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:40 smithi151 ceph-mon[41387]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi184.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T14:19:40.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:40 smithi151 ceph-mon[41387]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:19:40.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:40 smithi151 ceph-mon[41387]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi184.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T14:19:41.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:41 smithi151 ceph-mon[29807]: pgmap v50: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:41.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:41 smithi151 ceph-mon[41387]: pgmap v50: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:41.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:41 smithi184 ceph-mon[33394]: pgmap v50: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:42.816 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:19:42.817 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":63,"fsid":"2401fc50-71da-11ef-bceb-c7b262605968","created":"2024-09-13T14:12:36.168205+0000","modified":"2024-09-13T14:19:27.879672+0000","last_up_change":"2024-09-13T14:17:40.817515+0000","last_in_change":"2024-09-13T14:17:24.771724+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:15:41.780748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:18:09.030957+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":56,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T14:19:24.791384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T14:19:25.101452+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}}}],"osds":[{"osd":0,"uuid":"0ad945ee-af7d-4f26-a1e9-18ed963194ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":551854295},{"type":"v1","addr":"172.21.15.151:6803","nonce":551854295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":551854295},{"type":"v1","addr":"172.21.15.151:6805","nonce":551854295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":551854295},{"type":"v1","addr":"172.21.15.151:6809","nonce":551854295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":551854295},{"type":"v1","addr":"172.21.15.151:6807","nonce":551854295}]},"public_addr":"172.21.15.151:6803/551854295","cluster_addr":"172.21.15.151:6805/551854295","heartbeat_back_addr":"172.21.15.151:6809/551854295","heartbeat_front_addr":"172.21.15.151:6807/551854295","state":["exists","up"]},{"osd":1,"uuid":"09f5d705-684f-4e86-80a2-ec2292af09d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":3674059846},{"type":"v1","addr":"172.21.15.151:6811","nonce":3674059846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3674059846},{"type":"v1","addr":"172.21.15.151:6813","nonce":3674059846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3674059846},{"type":"v1","addr":"172.21.15.151:6817","nonce":3674059846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":3674059846},{"type":"v1","addr":"172.21.15.151:6815","nonce":3674059846}]},"public_addr":"172.21.15.151:6811/3674059846","cluster_addr":"172.21.15.151:6813/3674059846","heartbeat_back_addr":"172.21.15.151:6817/3674059846","heartbeat_front_addr":"172.21.15.151:6815/3674059846","state":["exists","up"]},{"osd":2,"uuid":"6a63f8af-ff69-46bb-830f-03ea89fc78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":1646754940},{"type":"v1","addr":"172.21.15.151:6819","nonce":1646754940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":1646754940},{"type":"v1","addr":"172.21.15.151:6821","nonce":1646754940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1646754940},{"type":"v1","addr":"172.21.15.151:6825","nonce":1646754940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":1646754940},{"type":"v1","addr":"172.21.15.151:6823","nonce":1646754940}]},"public_addr":"172.21.15.151:6819/1646754940","cluster_addr":"172.21.15.151:6821/1646754940","heartbeat_back_addr":"172.21.15.151:6825/1646754940","heartbeat_front_addr":"172.21.15.151:6823/1646754940","state":["exists","up"]},{"osd":3,"uuid":"770e8334-3b3a-46bc-9782-964b2a696660","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2203980888},{"type":"v1","addr":"172.21.15.151:6827","nonce":2203980888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":2203980888},{"type":"v1","addr":"172.21.15.151:6829","nonce":2203980888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":2203980888},{"type":"v1","addr":"172.21.15.151:6833","nonce":2203980888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":2203980888},{"type":"v1","addr":"172.21.15.151:6831","nonce":2203980888}]},"public_addr":"172.21.15.151:6827/2203980888","cluster_addr":"172.21.15.151:6829/2203980888","heartbeat_back_addr":"172.21.15.151:6833/2203980888","heartbeat_front_addr":"172.21.15.151:6831/2203980888","state":["exists","up"]},{"osd":4,"uuid":"75e258c0-1145-4aee-86aa-db2b63082cc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6801","nonce":3275553935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6803","nonce":3275553935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6807","nonce":3275553935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6805","nonce":3275553935}]},"public_addr":"172.21.15.184:6801/3275553935","cluster_addr":"172.21.15.184:6803/3275553935","heartbeat_back_addr":"172.21.15.184:6807/3275553935","heartbeat_front_addr":"172.21.15.184:6805/3275553935","state":["exists","up"]},{"osd":5,"uuid":"f56a5366-9d22-4960-933d-340aa280fbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6809","nonce":2446419621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6811","nonce":2446419621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6815","nonce":2446419621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6813","nonce":2446419621}]},"public_addr":"172.21.15.184:6809/2446419621","cluster_addr":"172.21.15.184:6811/2446419621","heartbeat_back_addr":"172.21.15.184:6815/2446419621","heartbeat_front_addr":"172.21.15.184:6813/2446419621","state":["exists","up"]},{"osd":6,"uuid":"e9826568-a728-45ab-8180-9777fe31f93a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6817","nonce":2214837203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6819","nonce":2214837203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6823","nonce":2214837203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6821","nonce":2214837203}]},"public_addr":"172.21.15.184:6817/2214837203","cluster_addr":"172.21.15.184:6819/2214837203","heartbeat_back_addr":"172.21.15.184:6823/2214837203","heartbeat_front_addr":"172.21.15.184:6821/2214837203","state":["exists","up"]},{"osd":7,"uuid":"d76869fe-5121-4467-8b56-1aadef0eaf65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6825","nonce":3378735997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6827","nonce":3378735997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6831","nonce":3378735997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6829","nonce":3378735997}]},"public_addr":"172.21.15.184:6825/3378735997","cluster_addr":"172.21.15.184:6827/3378735997","heartbeat_back_addr":"172.21.15.184:6831/3378735997","heartbeat_front_addr":"172.21.15.184:6829/3378735997","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:14:49.835715+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:15:14.682882+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:15:39.423890+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:16:04.527460+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:16:27.283661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:16:51.893868+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:16.152592+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:39.763533+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.151:6800/3383065173":"2024-09-14T14:13:04.761383+0000","172.21.15.151:6801/3383065173":"2024-09-14T14:13:04.761383+0000","172.21.15.151:0/3946777511":"2024-09-14T14:13:04.761383+0000","172.21.15.151:0/624042464":"2024-09-14T14:13:04.761383+0000","172.21.15.151:6801/1057983082":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/1182681341":"2024-09-14T14:13:04.761383+0000","172.21.15.151:0/1427922964":"2024-09-14T14:13:27.478847+0000","172.21.15.151:6800/1057983082":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/1351186656":"2024-09-14T14:18:18.344788+0000","172.21.15.151:6801/2335297718":"2024-09-14T14:18:18.344788+0000","172.21.15.151:0/2666829207":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/4115173791":"2024-09-14T14:18:18.344788+0000","172.21.15.151:6800/2335297718":"2024-09-14T14:18:18.344788+0000","172.21.15.151:0/3689756337":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/2800235450":"2024-09-14T14:18:18.344788+0000","172.21.15.151:0/2889050140":"2024-09-14T14:18:18.344788+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T14:19:43.532 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-13T14:19:43.533 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-13T14:19:43.533 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-13T14:19:43.534 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-13T14:19:43.534 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-13T14:19:43.534 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-13T14:19:43.535 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-13T14:19:43.535 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-13T14:19:43.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:43 smithi184 ceph-mon[33394]: pgmap v51: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:43.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:43 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:43.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:43 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:43.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:43 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/2404704849' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:19:43.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:43 smithi151 ceph-mon[29807]: pgmap v51: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:43.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:43 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:43.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:43 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:43.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:43 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/2404704849' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:19:43.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:43 smithi151 ceph-mon[41387]: pgmap v51: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:43.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:43 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:43.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:43 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:43.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:43 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/2404704849' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:19:44.358 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:19:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[79108]: ts=2024-09-13T14:19:44.062Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002804691s 2024-09-13T14:19:44.421 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:19:44.427 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:19:44.454 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:19:44.458 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:19:44.459 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:19:45.846 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:45 smithi151 ceph-mon[29807]: pgmap v52: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:45.846 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:45 smithi151 ceph-mon[41387]: pgmap v52: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:45.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:45 smithi184 ceph-mon[33394]: pgmap v52: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:47.459 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:19:47.460 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:19:47.460 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:19:47.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:47 smithi151 ceph-mon[41387]: pgmap v53: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:47.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:47 smithi151 ceph-mon[29807]: pgmap v53: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:47.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:47 smithi184 ceph-mon[33394]: pgmap v53: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:47.853 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:19:47 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:19:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:19:49.017 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:48 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:49.017 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:48 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:49.017 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:48 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:19:49.017 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:48 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:49.017 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:48 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:49.017 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:48 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:19:49.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:48 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:49.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:48 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:49.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:48 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:19:49.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:49 smithi151 ceph-mon[41387]: pgmap v54: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:49.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:49 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:49.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:49 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:19:49.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:49 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:49.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:49 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi151.rqqklw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T14:19:49.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:49 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi151.rqqklw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T14:19:49.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:49 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:49.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:49 smithi151 ceph-mon[41387]: Deploying daemon mds.foo.smithi151.rqqklw on smithi151 2024-09-13T14:19:49.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:49 smithi151 ceph-mon[29807]: pgmap v54: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:49.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:49 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:49.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:49 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:19:49.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:49 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:49.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:49 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi151.rqqklw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T14:19:49.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:49 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi151.rqqklw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T14:19:49.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:49 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:49.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:49 smithi151 ceph-mon[29807]: Deploying daemon mds.foo.smithi151.rqqklw on smithi151 2024-09-13T14:19:50.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:49 smithi184 ceph-mon[33394]: pgmap v54: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:50.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:49 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:50.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:49 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:19:50.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:49 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:19:50.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:49 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi151.rqqklw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T14:19:50.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:49 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi151.rqqklw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T14:19:50.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:49 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:19:50.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:49 smithi184 ceph-mon[33394]: Deploying daemon mds.foo.smithi151.rqqklw on smithi151 2024-09-13T14:19:51.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:51 smithi151 ceph-mon[29807]: pgmap v55: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:51.803 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:51 smithi151 ceph-mon[41387]: pgmap v55: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:52.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:51 smithi184 ceph-mon[33394]: pgmap v55: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:53.108 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:19:52 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:19:52] "GET /metrics HTTP/1.1" 200 208798 "" "Prometheus/2.43.0" 2024-09-13T14:19:53.750 INFO:teuthology.orchestra.run.smithi151.stdout:154618822695 2024-09-13T14:19:53.750 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-13T14:19:54.077 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:53 smithi151 ceph-mon[41387]: pgmap v56: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:54.078 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:53 smithi151 ceph-mon[29807]: pgmap v56: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:54.100 INFO:teuthology.orchestra.run.smithi151.stdout:77309411382 2024-09-13T14:19:54.101 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-13T14:19:54.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:53 smithi184 ceph-mon[33394]: pgmap v56: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:56.023 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:55 smithi151 ceph-mon[29807]: pgmap v57: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:56.023 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:55 smithi151 ceph-mon[41387]: pgmap v57: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:56.047 INFO:teuthology.orchestra.run.smithi151.stdout:55834574906 2024-09-13T14:19:56.047 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-13T14:19:56.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:55 smithi184 ceph-mon[33394]: pgmap v57: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:57.715 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:19:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:19:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:19:58.102 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:57 smithi151 ceph-mon[41387]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:58.103 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:57 smithi151 ceph-mon[29807]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:58.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:57 smithi184 ceph-mon[33394]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:19:58.830 INFO:teuthology.orchestra.run.smithi151.stdout:34359738433 2024-09-13T14:19:58.830 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-13T14:20:00.001 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:19:59 smithi151 ceph-mon[29807]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:00.001 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:19:59 smithi151 ceph-mon[41387]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:00.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:19:59 smithi184 ceph-mon[33394]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:00.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:00 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a[29784]: 2024-09-13T14:19:59.999+0000 7fab34b45640 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is offline; 1 filesystem is online with fewer MDS than max_mds 2024-09-13T14:20:00.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:00 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a[29784]: 2024-09-13T14:19:59.999+0000 7fab34b45640 -1 log_channel(cluster) log [ERR] : [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-09-13T14:20:00.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:00 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a[29784]: 2024-09-13T14:19:59.999+0000 7fab34b45640 -1 log_channel(cluster) log [ERR] : fs foo is offline because no MDS is active for it. 2024-09-13T14:20:00.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:00 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a[29784]: 2024-09-13T14:19:59.999+0000 7fab34b45640 -1 log_channel(cluster) log [ERR] : [WRN] MDS_UP_LESS_THAN_MAX: 1 filesystem is online with fewer MDS than max_mds 2024-09-13T14:20:00.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:00 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a[29784]: 2024-09-13T14:19:59.999+0000 7fab34b45640 -1 log_channel(cluster) log [ERR] : fs foo has 0 MDS online, but wants 1 2024-09-13T14:20:00.992 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:00 smithi151 ceph-mon[29807]: Health detail: HEALTH_ERR 1 filesystem is offline; 1 filesystem is online with fewer MDS than max_mds 2024-09-13T14:20:00.992 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:00 smithi151 ceph-mon[29807]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-09-13T14:20:00.992 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:00 smithi151 ceph-mon[29807]: fs foo is offline because no MDS is active for it. 2024-09-13T14:20:00.992 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:00 smithi151 ceph-mon[29807]: [WRN] MDS_UP_LESS_THAN_MAX: 1 filesystem is online with fewer MDS than max_mds 2024-09-13T14:20:00.992 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:00 smithi151 ceph-mon[29807]: fs foo has 0 MDS online, but wants 1 2024-09-13T14:20:00.993 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:00 smithi151 ceph-mon[41387]: Health detail: HEALTH_ERR 1 filesystem is offline; 1 filesystem is online with fewer MDS than max_mds 2024-09-13T14:20:00.993 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:00 smithi151 ceph-mon[41387]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-09-13T14:20:00.993 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:00 smithi151 ceph-mon[41387]: fs foo is offline because no MDS is active for it. 2024-09-13T14:20:00.993 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:00 smithi151 ceph-mon[41387]: [WRN] MDS_UP_LESS_THAN_MAX: 1 filesystem is online with fewer MDS than max_mds 2024-09-13T14:20:00.993 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:00 smithi151 ceph-mon[41387]: fs foo has 0 MDS online, but wants 1 2024-09-13T14:20:01.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:00 smithi184 ceph-mon[33394]: Health detail: HEALTH_ERR 1 filesystem is offline; 1 filesystem is online with fewer MDS than max_mds 2024-09-13T14:20:01.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:00 smithi184 ceph-mon[33394]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-09-13T14:20:01.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:00 smithi184 ceph-mon[33394]: fs foo is offline because no MDS is active for it. 2024-09-13T14:20:01.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:00 smithi184 ceph-mon[33394]: [WRN] MDS_UP_LESS_THAN_MAX: 1 filesystem is online with fewer MDS than max_mds 2024-09-13T14:20:01.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:00 smithi184 ceph-mon[33394]: fs foo has 0 MDS online, but wants 1 2024-09-13T14:20:01.931 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:01 smithi151 ceph-mon[29807]: pgmap v60: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:01.931 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:01 smithi151 ceph-mon[41387]: pgmap v60: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:02.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:01 smithi184 ceph-mon[33394]: pgmap v60: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:02.496 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:03.010 INFO:teuthology.orchestra.run.smithi151.stdout:206158430239 2024-09-13T14:20:03.010 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-13T14:20:03.264 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:20:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:20:02] "GET /metrics HTTP/1.1" 200 208798 "" "Prometheus/2.43.0" 2024-09-13T14:20:03.712 INFO:teuthology.orchestra.run.smithi151.stdout:180388626468 2024-09-13T14:20:03.713 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-13T14:20:03.933 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:03 smithi151 ceph-mon[29807]: pgmap v61: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:03.933 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:03 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:20:03.933 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:03 smithi151 ceph-mon[41387]: pgmap v61: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:03.933 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:03 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:20:04.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:03 smithi184 ceph-mon[33394]: pgmap v61: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:04.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:03 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:20:04.730 INFO:teuthology.orchestra.run.smithi151.stdout:107374182450 2024-09-13T14:20:04.730 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-13T14:20:05.358 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:05.386 INFO:teuthology.orchestra.run.smithi151.stdout:128849018925 2024-09-13T14:20:05.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-13T14:20:05.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:05 smithi184 ceph-mon[33394]: pgmap v62: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:05.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:05 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:05.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:05 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:05.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:05 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:05.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:05 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi184.jismsi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T14:20:05.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:05 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi184.jismsi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T14:20:05.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:05 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:20:05.911 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:05.911 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:06.045 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:05 smithi151 ceph-mon[41387]: pgmap v62: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:06.045 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:05 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:06.046 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:05 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:06.046 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:05 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:06.046 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:05 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi184.jismsi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T14:20:06.046 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:05 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi184.jismsi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T14:20:06.046 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:05 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:20:06.046 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:05 smithi151 ceph-mon[29807]: pgmap v62: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:06.046 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:05 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:06.046 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:05 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:06.047 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:05 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:06.047 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:05 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi184.jismsi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T14:20:06.047 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:05 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi184.jismsi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T14:20:06.047 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:05 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:20:06.588 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:06.591 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:06.595 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:07.008 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:06 smithi184 ceph-mon[33394]: Deploying daemon mds.foo.smithi184.jismsi on smithi184 2024-09-13T14:20:07.008 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:06 smithi184 ceph-mon[33394]: mds.? [v2:172.21.15.151:6834/345315682,v1:172.21.15.151:6835/345315682] up:boot 2024-09-13T14:20:07.008 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:06 smithi184 ceph-mon[33394]: daemon mds.foo.smithi151.rqqklw assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-09-13T14:20:07.008 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:06 smithi184 ceph-mon[33394]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-13T14:20:07.008 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:06 smithi184 ceph-mon[33394]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-13T14:20:07.009 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:06 smithi184 ceph-mon[33394]: Cluster is now healthy 2024-09-13T14:20:07.009 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:06 smithi184 ceph-mon[33394]: fsmap foo:0 1 up:standby 2024-09-13T14:20:07.009 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:06 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi151.rqqklw"}]: dispatch 2024-09-13T14:20:07.009 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:06 smithi184 ceph-mon[33394]: fsmap foo:1 {0=foo.smithi151.rqqklw=up:creating} 2024-09-13T14:20:07.009 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:06 smithi184 ceph-mon[33394]: daemon mds.foo.smithi151.rqqklw is now active in filesystem foo as rank 0 2024-09-13T14:20:07.077 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:06 smithi151 ceph-mon[29807]: Deploying daemon mds.foo.smithi184.jismsi on smithi184 2024-09-13T14:20:07.077 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:06 smithi151 ceph-mon[29807]: mds.? [v2:172.21.15.151:6834/345315682,v1:172.21.15.151:6835/345315682] up:boot 2024-09-13T14:20:07.077 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:06 smithi151 ceph-mon[29807]: daemon mds.foo.smithi151.rqqklw assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-09-13T14:20:07.077 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:06 smithi151 ceph-mon[29807]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-13T14:20:07.077 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:06 smithi151 ceph-mon[29807]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-13T14:20:07.077 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:06 smithi151 ceph-mon[29807]: Cluster is now healthy 2024-09-13T14:20:07.077 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:06 smithi151 ceph-mon[29807]: fsmap foo:0 1 up:standby 2024-09-13T14:20:07.077 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:06 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi151.rqqklw"}]: dispatch 2024-09-13T14:20:07.078 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:06 smithi151 ceph-mon[29807]: fsmap foo:1 {0=foo.smithi151.rqqklw=up:creating} 2024-09-13T14:20:07.078 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:06 smithi151 ceph-mon[29807]: daemon mds.foo.smithi151.rqqklw is now active in filesystem foo as rank 0 2024-09-13T14:20:07.078 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:06 smithi151 ceph-mon[41387]: Deploying daemon mds.foo.smithi184.jismsi on smithi184 2024-09-13T14:20:07.078 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:06 smithi151 ceph-mon[41387]: mds.? [v2:172.21.15.151:6834/345315682,v1:172.21.15.151:6835/345315682] up:boot 2024-09-13T14:20:07.078 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:06 smithi151 ceph-mon[41387]: daemon mds.foo.smithi151.rqqklw assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-09-13T14:20:07.078 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:06 smithi151 ceph-mon[41387]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-13T14:20:07.078 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:06 smithi151 ceph-mon[41387]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-13T14:20:07.078 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:06 smithi151 ceph-mon[41387]: Cluster is now healthy 2024-09-13T14:20:07.078 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:06 smithi151 ceph-mon[41387]: fsmap foo:0 1 up:standby 2024-09-13T14:20:07.078 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:06 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi151.rqqklw"}]: dispatch 2024-09-13T14:20:07.078 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:06 smithi151 ceph-mon[41387]: fsmap foo:1 {0=foo.smithi151.rqqklw=up:creating} 2024-09-13T14:20:07.078 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:06 smithi151 ceph-mon[41387]: daemon mds.foo.smithi151.rqqklw is now active in filesystem foo as rank 0 2024-09-13T14:20:07.603 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:20:07 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:20:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:20:08.027 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:07 smithi151 ceph-mon[29807]: pgmap v63: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:08.027 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:07 smithi151 ceph-mon[29807]: mds.? [v2:172.21.15.151:6834/345315682,v1:172.21.15.151:6835/345315682] up:active 2024-09-13T14:20:08.027 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:07 smithi151 ceph-mon[29807]: fsmap foo:1 {0=foo.smithi151.rqqklw=up:active} 2024-09-13T14:20:08.028 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:07 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:08.028 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:07 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:08.028 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:07 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:08.028 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:07 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:08.028 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:07 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:08.028 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:07 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:20:08.028 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:07 smithi151 ceph-mon[41387]: pgmap v63: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:08.028 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:07 smithi151 ceph-mon[41387]: mds.? [v2:172.21.15.151:6834/345315682,v1:172.21.15.151:6835/345315682] up:active 2024-09-13T14:20:08.029 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:07 smithi151 ceph-mon[41387]: fsmap foo:1 {0=foo.smithi151.rqqklw=up:active} 2024-09-13T14:20:08.029 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:07 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:08.029 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:07 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:08.029 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:07 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:08.029 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:07 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:08.029 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:07 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:08.029 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:07 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:20:08.090 INFO:teuthology.orchestra.run.smithi151.stdout:154618822698 2024-09-13T14:20:08.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:07 smithi184 ceph-mon[33394]: pgmap v63: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:08.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:07 smithi184 ceph-mon[33394]: mds.? [v2:172.21.15.151:6834/345315682,v1:172.21.15.151:6835/345315682] up:active 2024-09-13T14:20:08.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:07 smithi184 ceph-mon[33394]: fsmap foo:1 {0=foo.smithi151.rqqklw=up:active} 2024-09-13T14:20:08.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:07 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:08.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:07 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:08.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:07 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:08.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:07 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:08.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:07 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:08.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:07 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:20:09.045 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:08 smithi151 ceph-mon[29807]: mds.? [v2:172.21.15.184:6832/4138067167,v1:172.21.15.184:6833/4138067167] up:boot 2024-09-13T14:20:09.045 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:08 smithi151 ceph-mon[29807]: fsmap foo:1 {0=foo.smithi151.rqqklw=up:active} 1 up:standby 2024-09-13T14:20:09.045 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:08 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi184.jismsi"}]: dispatch 2024-09-13T14:20:09.045 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:08 smithi151 ceph-mon[29807]: fsmap foo:1 {0=foo.smithi151.rqqklw=up:active} 1 up:standby 2024-09-13T14:20:09.045 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:08 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1087425916' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T14:20:09.045 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:08 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:09.045 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:08 smithi151 ceph-mon[41387]: mds.? [v2:172.21.15.184:6832/4138067167,v1:172.21.15.184:6833/4138067167] up:boot 2024-09-13T14:20:09.046 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:08 smithi151 ceph-mon[41387]: fsmap foo:1 {0=foo.smithi151.rqqklw=up:active} 1 up:standby 2024-09-13T14:20:09.046 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:08 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi184.jismsi"}]: dispatch 2024-09-13T14:20:09.046 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:08 smithi151 ceph-mon[41387]: fsmap foo:1 {0=foo.smithi151.rqqklw=up:active} 1 up:standby 2024-09-13T14:20:09.046 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:08 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/1087425916' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T14:20:09.046 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:08 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:09.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:08 smithi184 ceph-mon[33394]: mds.? [v2:172.21.15.184:6832/4138067167,v1:172.21.15.184:6833/4138067167] up:boot 2024-09-13T14:20:09.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:08 smithi184 ceph-mon[33394]: fsmap foo:1 {0=foo.smithi151.rqqklw=up:active} 1 up:standby 2024-09-13T14:20:09.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:08 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi184.jismsi"}]: dispatch 2024-09-13T14:20:09.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:08 smithi184 ceph-mon[33394]: fsmap foo:1 {0=foo.smithi151.rqqklw=up:active} 1 up:standby 2024-09-13T14:20:09.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:08 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/1087425916' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T14:20:09.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:08 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:09.999 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:09 smithi151 ceph-mon[29807]: pgmap v64: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:10.000 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:09 smithi151 ceph-mon[41387]: pgmap v64: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:10.025 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:09 smithi184 ceph-mon[33394]: pgmap v64: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:10.529 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822695 got 154618822698 for osd.5 2024-09-13T14:20:10.529 DEBUG:teuthology.parallel:result is None 2024-09-13T14:20:11.036 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:10 smithi151 ceph-mon[29807]: mds.? [v2:172.21.15.151:6834/345315682,v1:172.21.15.151:6835/345315682] up:active 2024-09-13T14:20:11.036 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:10 smithi151 ceph-mon[29807]: fsmap foo:1 {0=foo.smithi151.rqqklw=up:active} 1 up:standby 2024-09-13T14:20:11.036 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:10 smithi151 ceph-mon[41387]: mds.? [v2:172.21.15.151:6834/345315682,v1:172.21.15.151:6835/345315682] up:active 2024-09-13T14:20:11.036 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:10 smithi151 ceph-mon[41387]: fsmap foo:1 {0=foo.smithi151.rqqklw=up:active} 1 up:standby 2024-09-13T14:20:11.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:10 smithi184 ceph-mon[33394]: mds.? [v2:172.21.15.151:6834/345315682,v1:172.21.15.151:6835/345315682] up:active 2024-09-13T14:20:11.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:10 smithi184 ceph-mon[33394]: fsmap foo:1 {0=foo.smithi151.rqqklw=up:active} 1 up:standby 2024-09-13T14:20:11.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:11 smithi151 ceph-mon[41387]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-13T14:20:11.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:11 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:11.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:11 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:11.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:11 smithi151 ceph-mon[29807]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-13T14:20:11.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:11 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:11.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:11 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:12.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:11 smithi184 ceph-mon[33394]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-13T14:20:12.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:11 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:12.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:11 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:13.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:13 smithi151 ceph-mon[29807]: mds.? [v2:172.21.15.184:6832/4138067167,v1:172.21.15.184:6833/4138067167] up:standby 2024-09-13T14:20:13.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:13 smithi151 ceph-mon[29807]: fsmap foo:1 {0=foo.smithi151.rqqklw=up:active} 1 up:standby 2024-09-13T14:20:13.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:13 smithi151 ceph-mon[29807]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-13T14:20:13.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:13 smithi151 ceph-mon[41387]: mds.? [v2:172.21.15.184:6832/4138067167,v1:172.21.15.184:6833/4138067167] up:standby 2024-09-13T14:20:13.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:13 smithi151 ceph-mon[41387]: fsmap foo:1 {0=foo.smithi151.rqqklw=up:active} 1 up:standby 2024-09-13T14:20:13.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:13 smithi151 ceph-mon[41387]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-13T14:20:13.359 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:20:12 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:20:12] "GET /metrics HTTP/1.1" 200 208755 "" "Prometheus/2.43.0" 2024-09-13T14:20:13.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:13 smithi184 ceph-mon[33394]: mds.? [v2:172.21.15.184:6832/4138067167,v1:172.21.15.184:6833/4138067167] up:standby 2024-09-13T14:20:13.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:13 smithi184 ceph-mon[33394]: fsmap foo:1 {0=foo.smithi151.rqqklw=up:active} 1 up:standby 2024-09-13T14:20:13.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:13 smithi184 ceph-mon[33394]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-13T14:20:14.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:14 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/257723716' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T14:20:14.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:14 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/257723716' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T14:20:14.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:14 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/257723716' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T14:20:15.141 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:15 smithi151 ceph-mon[29807]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-13T14:20:15.142 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:15 smithi151 ceph-mon[41387]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-13T14:20:15.251 INFO:teuthology.orchestra.run.smithi151.stdout:206158430241 2024-09-13T14:20:15.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:15 smithi184 ceph-mon[33394]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-13T14:20:15.972 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430239 got 206158430241 for osd.7 2024-09-13T14:20:15.973 DEBUG:teuthology.parallel:result is None 2024-09-13T14:20:16.523 INFO:teuthology.orchestra.run.smithi151.stdout:77309411387 2024-09-13T14:20:16.741 INFO:teuthology.orchestra.run.smithi151.stdout:55834574911 2024-09-13T14:20:17.288 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411382 got 77309411387 for osd.2 2024-09-13T14:20:17.288 DEBUG:teuthology.parallel:result is None 2024-09-13T14:20:17.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:17 smithi151 ceph-mon[29807]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-13T14:20:17.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:17 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/267586409' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T14:20:17.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:17 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/3331996195' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T14:20:17.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:17 smithi151 ceph-mon[41387]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-13T14:20:17.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:17 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/267586409' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T14:20:17.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:17 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/3331996195' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T14:20:17.853 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:20:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:20:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:20:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:17 smithi184 ceph-mon[33394]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-13T14:20:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:17 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/267586409' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T14:20:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:17 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/3331996195' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T14:20:18.301 INFO:teuthology.orchestra.run.smithi151.stdout:34359738437 2024-09-13T14:20:18.499 INFO:teuthology.orchestra.run.smithi151.stdout:107374182454 2024-09-13T14:20:18.499 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:18 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/3174793582' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T14:20:18.500 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:18 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-09-13T14:20:18.500 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:18 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-09-13T14:20:18.500 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:18 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2024-09-13T14:20:18.500 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:18 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:18.500 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:18 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:20:18.500 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:18 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/3174793582' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T14:20:18.500 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:18 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-09-13T14:20:18.500 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:18 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-09-13T14:20:18.500 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:18 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2024-09-13T14:20:18.500 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:18 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:18.500 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:18 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:20:18.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:18 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/3174793582' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T14:20:18.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:18 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-09-13T14:20:18.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:18 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-09-13T14:20:18.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:18 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2024-09-13T14:20:18.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:18 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:18.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:18 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:20:19.196 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182450 got 107374182454 for osd.3 2024-09-13T14:20:19.196 DEBUG:teuthology.parallel:result is None 2024-09-13T14:20:19.581 INFO:teuthology.orchestra.run.smithi151.stdout:180388626471 2024-09-13T14:20:19.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:19 smithi184 ceph-mon[33394]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-13T14:20:19.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:19 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-13T14:20:19.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:19 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-13T14:20:19.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:19 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]': finished 2024-09-13T14:20:19.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:19 smithi184 ceph-mon[33394]: osdmap e64: 8 total, 8 up, 8 in 2024-09-13T14:20:19.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:19 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/1690262434' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T14:20:19.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:19 smithi151 ceph-mon[41387]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-13T14:20:19.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:19 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-13T14:20:19.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:19 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-13T14:20:19.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:19 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]': finished 2024-09-13T14:20:19.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:19 smithi151 ceph-mon[41387]: osdmap e64: 8 total, 8 up, 8 in 2024-09-13T14:20:19.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:19 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/1690262434' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T14:20:19.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:19 smithi151 ceph-mon[29807]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-13T14:20:19.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:19 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-13T14:20:19.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:19 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-13T14:20:19.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:19 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]': finished 2024-09-13T14:20:19.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:19 smithi151 ceph-mon[29807]: osdmap e64: 8 total, 8 up, 8 in 2024-09-13T14:20:19.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:19 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1690262434' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T14:20:20.453 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574906 got 55834574911 for osd.1 2024-09-13T14:20:20.454 DEBUG:teuthology.parallel:result is None 2024-09-13T14:20:20.611 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738433 got 34359738437 for osd.0 2024-09-13T14:20:20.611 DEBUG:teuthology.parallel:result is None 2024-09-13T14:20:20.817 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:20.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:20 smithi184 ceph-mon[33394]: osdmap e65: 8 total, 8 up, 8 in 2024-09-13T14:20:20.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:20 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/4016791962' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T14:20:20.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:20 smithi151 ceph-mon[29807]: osdmap e65: 8 total, 8 up, 8 in 2024-09-13T14:20:20.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:20 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/4016791962' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T14:20:20.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:20 smithi151 ceph-mon[41387]: osdmap e65: 8 total, 8 up, 8 in 2024-09-13T14:20:20.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:20 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/4016791962' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T14:20:21.332 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626468 got 180388626471 for osd.6 2024-09-13T14:20:21.332 DEBUG:teuthology.parallel:result is None 2024-09-13T14:20:21.606 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:21 smithi151 ceph-mon[29807]: pgmap v72: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:20:21.606 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:21 smithi151 ceph-mon[41387]: pgmap v72: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:20:21.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:21 smithi184 ceph-mon[33394]: pgmap v72: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:20:23.090 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:20:22 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:20:22] "GET /metrics HTTP/1.1" 200 239775 "" "Prometheus/2.43.0" 2024-09-13T14:20:23.774 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:23 smithi151 ceph-mon[29807]: pgmap v73: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:23.774 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:23 smithi151 ceph-mon[41387]: pgmap v73: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:23.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:23 smithi184 ceph-mon[33394]: pgmap v73: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:23.960 INFO:teuthology.orchestra.run.smithi151.stdout:128849018929 2024-09-13T14:20:24.570 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:24 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/187855328' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T14:20:24.570 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:24 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/187855328' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T14:20:24.593 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018925 got 128849018929 for osd.4 2024-09-13T14:20:24.593 DEBUG:teuthology.parallel:result is None 2024-09-13T14:20:24.593 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-13T14:20:24.594 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:20:24.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:24 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/187855328' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T14:20:25.279 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:25.651 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:25 smithi151 ceph-mon[29807]: pgmap v74: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:25.652 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:25 smithi151 ceph-mon[41387]: pgmap v74: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:25.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:25 smithi184 ceph-mon[33394]: pgmap v74: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:27.484 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:20:27.485 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T14:20:27.674 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:27 smithi151 ceph-mon[29807]: pgmap v75: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:27.674 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:27 smithi151 ceph-mon[41387]: pgmap v75: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:27.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:27 smithi184 ceph-mon[33394]: pgmap v75: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:27.853 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:20:27 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:20:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:20:28.527 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2024-09-13T14:20:26.379893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592725,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":160,"num_write_kb":1643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121,"ondisk_log_size":121,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385864,"kb_used_data":6992,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747575352,"statfs":{"total":767960285184,"available":765517160448,"internally_reserved":0,"allocated":7159808,"data_stored":4115619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001750"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.445376+0000","last_change":"2024-09-13T14:19:26.890506+0000","last_active":"2024-09-13T14:20:18.445376+0000","last_peered":"2024-09-13T14:20:18.445376+0000","last_clean":"2024-09-13T14:20:18.445376+0000","last_became_active":"2024-09-13T14:19:26.890223+0000","last_became_peered":"2024-09-13T14:19:26.890223+0000","last_unstale":"2024-09-13T14:20:18.445376+0000","last_undegraded":"2024-09-13T14:20:18.445376+0000","last_fullsized":"2024-09-13T14:20:18.445376+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:38:01.381959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":16,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.572650+0000","last_change":"2024-09-13T14:19:25.867510+0000","last_active":"2024-09-13T14:20:18.572650+0000","last_peered":"2024-09-13T14:20:18.572650+0000","last_clean":"2024-09-13T14:20:18.572650+0000","last_became_active":"2024-09-13T14:19:25.867195+0000","last_became_peered":"2024-09-13T14:19:25.867195+0000","last_unstale":"2024-09-13T14:20:18.572650+0000","last_undegraded":"2024-09-13T14:20:18.572650+0000","last_fullsized":"2024-09-13T14:20:18.572650+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:08:09.116221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.443313+0000","last_change":"2024-09-13T14:19:26.888945+0000","last_active":"2024-09-13T14:20:18.443313+0000","last_peered":"2024-09-13T14:20:18.443313+0000","last_clean":"2024-09-13T14:20:18.443313+0000","last_became_active":"2024-09-13T14:19:26.888587+0000","last_became_peered":"2024-09-13T14:19:26.888587+0000","last_unstale":"2024-09-13T14:20:18.443313+0000","last_undegraded":"2024-09-13T14:20:18.443313+0000","last_fullsized":"2024-09-13T14:20:18.443313+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:25.696881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.598017+0000","last_change":"2024-09-13T14:19:25.877587+0000","last_active":"2024-09-13T14:20:18.598017+0000","last_peered":"2024-09-13T14:20:18.598017+0000","last_clean":"2024-09-13T14:20:18.598017+0000","last_became_active":"2024-09-13T14:19:25.877283+0000","last_became_peered":"2024-09-13T14:19:25.877283+0000","last_unstale":"2024-09-13T14:20:18.598017+0000","last_undegraded":"2024-09-13T14:20:18.598017+0000","last_fullsized":"2024-09-13T14:20:18.598017+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:43:50.415006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.598098+0000","last_change":"2024-09-13T14:19:26.890744+0000","last_active":"2024-09-13T14:20:18.598098+0000","last_peered":"2024-09-13T14:20:18.598098+0000","last_clean":"2024-09-13T14:20:18.598098+0000","last_became_active":"2024-09-13T14:19:26.890495+0000","last_became_peered":"2024-09-13T14:19:26.890495+0000","last_unstale":"2024-09-13T14:20:18.598098+0000","last_undegraded":"2024-09-13T14:20:18.598098+0000","last_fullsized":"2024-09-13T14:20:18.598098+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:41:38.787130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.572594+0000","last_change":"2024-09-13T14:19:25.873088+0000","last_active":"2024-09-13T14:20:18.572594+0000","last_peered":"2024-09-13T14:20:18.572594+0000","last_clean":"2024-09-13T14:20:18.572594+0000","last_became_active":"2024-09-13T14:19:25.872781+0000","last_became_peered":"2024-09-13T14:19:25.872781+0000","last_unstale":"2024-09-13T14:20:18.572594+0000","last_undegraded":"2024-09-13T14:20:18.572594+0000","last_fullsized":"2024-09-13T14:20:18.572594+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:05.607382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.445817+0000","last_change":"2024-09-13T14:19:26.890640+0000","last_active":"2024-09-13T14:20:18.445817+0000","last_peered":"2024-09-13T14:20:18.445817+0000","last_clean":"2024-09-13T14:20:18.445817+0000","last_became_active":"2024-09-13T14:19:26.890377+0000","last_became_peered":"2024-09-13T14:19:26.890377+0000","last_unstale":"2024-09-13T14:20:18.445817+0000","last_undegraded":"2024-09-13T14:20:18.445817+0000","last_fullsized":"2024-09-13T14:20:18.445817+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:06:17.733701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":22,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.443550+0000","last_change":"2024-09-13T14:19:25.869377+0000","last_active":"2024-09-13T14:20:18.443550+0000","last_peered":"2024-09-13T14:20:18.443550+0000","last_clean":"2024-09-13T14:20:18.443550+0000","last_became_active":"2024-09-13T14:19:25.869081+0000","last_became_peered":"2024-09-13T14:19:25.869081+0000","last_unstale":"2024-09-13T14:20:18.443550+0000","last_undegraded":"2024-09-13T14:20:18.443550+0000","last_fullsized":"2024-09-13T14:20:18.443550+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:33:12.833243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.572940+0000","last_change":"2024-09-13T14:19:25.872435+0000","last_active":"2024-09-13T14:20:18.572940+0000","last_peered":"2024-09-13T14:20:18.572940+0000","last_clean":"2024-09-13T14:20:18.572940+0000","last_became_active":"2024-09-13T14:19:25.872148+0000","last_became_peered":"2024-09-13T14:19:25.872148+0000","last_unstale":"2024-09-13T14:20:18.572940+0000","last_undegraded":"2024-09-13T14:20:18.572940+0000","last_fullsized":"2024-09-13T14:20:18.572940+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:29.425371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.442840+0000","last_change":"2024-09-13T14:19:26.887839+0000","last_active":"2024-09-13T14:20:18.442840+0000","last_peered":"2024-09-13T14:20:18.442840+0000","last_clean":"2024-09-13T14:20:18.442840+0000","last_became_active":"2024-09-13T14:19:26.887498+0000","last_became_peered":"2024-09-13T14:19:26.887498+0000","last_unstale":"2024-09-13T14:20:18.442840+0000","last_undegraded":"2024-09-13T14:20:18.442840+0000","last_fullsized":"2024-09-13T14:20:18.442840+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:24.475263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":16,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.572936+0000","last_change":"2024-09-13T14:19:25.874105+0000","last_active":"2024-09-13T14:20:18.572936+0000","last_peered":"2024-09-13T14:20:18.572936+0000","last_clean":"2024-09-13T14:20:18.572936+0000","last_became_active":"2024-09-13T14:19:25.873714+0000","last_became_peered":"2024-09-13T14:19:25.873714+0000","last_unstale":"2024-09-13T14:20:18.572936+0000","last_undegraded":"2024-09-13T14:20:18.572936+0000","last_fullsized":"2024-09-13T14:20:18.572936+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:28:39.623535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.440545+0000","last_change":"2024-09-13T14:19:25.993445+0000","last_active":"2024-09-13T14:20:18.440545+0000","last_peered":"2024-09-13T14:20:18.440545+0000","last_clean":"2024-09-13T14:20:18.440545+0000","last_became_active":"2024-09-13T14:19:25.993153+0000","last_became_peered":"2024-09-13T14:19:25.993153+0000","last_unstale":"2024-09-13T14:20:18.440545+0000","last_undegraded":"2024-09-13T14:20:18.440545+0000","last_fullsized":"2024-09-13T14:20:18.440545+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:46:47.959833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.572915+0000","last_change":"2024-09-13T14:19:25.871006+0000","last_active":"2024-09-13T14:20:18.572915+0000","last_peered":"2024-09-13T14:20:18.572915+0000","last_clean":"2024-09-13T14:20:18.572915+0000","last_became_active":"2024-09-13T14:19:25.870713+0000","last_became_peered":"2024-09-13T14:19:25.870713+0000","last_unstale":"2024-09-13T14:20:18.572915+0000","last_undegraded":"2024-09-13T14:20:18.572915+0000","last_fullsized":"2024-09-13T14:20:18.572915+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:34.372124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.442839+0000","last_change":"2024-09-13T14:19:26.887804+0000","last_active":"2024-09-13T14:20:18.442839+0000","last_peered":"2024-09-13T14:20:18.442839+0000","last_clean":"2024-09-13T14:20:18.442839+0000","last_became_active":"2024-09-13T14:19:26.887521+0000","last_became_peered":"2024-09-13T14:19:26.887521+0000","last_unstale":"2024-09-13T14:20:18.442839+0000","last_undegraded":"2024-09-13T14:20:18.442839+0000","last_fullsized":"2024-09-13T14:20:18.442839+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:19:37.041969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.443219+0000","last_change":"2024-09-13T14:19:25.874666+0000","last_active":"2024-09-13T14:20:18.443219+0000","last_peered":"2024-09-13T14:20:18.443219+0000","last_clean":"2024-09-13T14:20:18.443219+0000","last_became_active":"2024-09-13T14:19:25.874381+0000","last_became_peered":"2024-09-13T14:19:25.874381+0000","last_unstale":"2024-09-13T14:20:18.443219+0000","last_undegraded":"2024-09-13T14:20:18.443219+0000","last_fullsized":"2024-09-13T14:20:18.443219+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:26.056377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.597954+0000","last_change":"2024-09-13T14:19:27.378254+0000","last_active":"2024-09-13T14:20:18.597954+0000","last_peered":"2024-09-13T14:20:18.597954+0000","last_clean":"2024-09-13T14:20:18.597954+0000","last_became_active":"2024-09-13T14:19:27.377992+0000","last_became_peered":"2024-09-13T14:19:27.377992+0000","last_unstale":"2024-09-13T14:20:18.597954+0000","last_undegraded":"2024-09-13T14:20:18.597954+0000","last_fullsized":"2024-09-13T14:20:18.597954+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:38.328601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":16,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.447695+0000","last_change":"2024-09-13T14:19:25.880464+0000","last_active":"2024-09-13T14:20:18.447695+0000","last_peered":"2024-09-13T14:20:18.447695+0000","last_clean":"2024-09-13T14:20:18.447695+0000","last_became_active":"2024-09-13T14:19:25.880156+0000","last_became_peered":"2024-09-13T14:19:25.880156+0000","last_unstale":"2024-09-13T14:20:18.447695+0000","last_undegraded":"2024-09-13T14:20:18.447695+0000","last_fullsized":"2024-09-13T14:20:18.447695+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:14:52.189607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.440737+0000","last_change":"2024-09-13T14:19:26.132082+0000","last_active":"2024-09-13T14:20:18.440737+0000","last_peered":"2024-09-13T14:20:18.440737+0000","last_clean":"2024-09-13T14:20:18.440737+0000","last_became_active":"2024-09-13T14:19:26.131076+0000","last_became_peered":"2024-09-13T14:19:26.131076+0000","last_unstale":"2024-09-13T14:20:18.440737+0000","last_undegraded":"2024-09-13T14:20:18.440737+0000","last_fullsized":"2024-09-13T14:20:18.440737+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:37:58.988661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.443161+0000","last_change":"2024-09-13T14:19:25.872515+0000","last_active":"2024-09-13T14:20:18.443161+0000","last_peered":"2024-09-13T14:20:18.443161+0000","last_clean":"2024-09-13T14:20:18.443161+0000","last_became_active":"2024-09-13T14:19:25.872008+0000","last_became_peered":"2024-09-13T14:19:25.872008+0000","last_unstale":"2024-09-13T14:20:18.443161+0000","last_undegraded":"2024-09-13T14:20:18.443161+0000","last_fullsized":"2024-09-13T14:20:18.443161+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:08:37.914110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.441177+0000","last_change":"2024-09-13T14:19:26.209809+0000","last_active":"2024-09-13T14:20:18.441177+0000","last_peered":"2024-09-13T14:20:18.441177+0000","last_clean":"2024-09-13T14:20:18.441177+0000","last_became_active":"2024-09-13T14:19:26.209545+0000","last_became_peered":"2024-09-13T14:19:26.209545+0000","last_unstale":"2024-09-13T14:20:18.441177+0000","last_undegraded":"2024-09-13T14:20:18.441177+0000","last_fullsized":"2024-09-13T14:20:18.441177+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:50:18.891602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.440519+0000","last_change":"2024-09-13T14:19:25.867776+0000","last_active":"2024-09-13T14:20:18.440519+0000","last_peered":"2024-09-13T14:20:18.440519+0000","last_clean":"2024-09-13T14:20:18.440519+0000","last_became_active":"2024-09-13T14:19:25.867475+0000","last_became_peered":"2024-09-13T14:19:25.867475+0000","last_unstale":"2024-09-13T14:20:18.440519+0000","last_undegraded":"2024-09-13T14:20:18.440519+0000","last_fullsized":"2024-09-13T14:20:18.440519+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:58:25.489922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.441023+0000","last_change":"2024-09-13T14:19:26.131605+0000","last_active":"2024-09-13T14:20:18.441023+0000","last_peered":"2024-09-13T14:20:18.441023+0000","last_clean":"2024-09-13T14:20:18.441023+0000","last_became_active":"2024-09-13T14:19:26.130954+0000","last_became_peered":"2024-09-13T14:19:26.130954+0000","last_unstale":"2024-09-13T14:20:18.441023+0000","last_undegraded":"2024-09-13T14:20:18.441023+0000","last_fullsized":"2024-09-13T14:20:18.441023+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:46:20.372750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.447029+0000","last_change":"2024-09-13T14:19:25.880337+0000","last_active":"2024-09-13T14:20:18.447029+0000","last_peered":"2024-09-13T14:20:18.447029+0000","last_clean":"2024-09-13T14:20:18.447029+0000","last_became_active":"2024-09-13T14:19:25.876971+0000","last_became_peered":"2024-09-13T14:19:25.876971+0000","last_unstale":"2024-09-13T14:20:18.447029+0000","last_undegraded":"2024-09-13T14:20:18.447029+0000","last_fullsized":"2024-09-13T14:20:18.447029+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:37:43.921921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.449301+0000","last_change":"2024-09-13T14:19:25.993379+0000","last_active":"2024-09-13T14:20:18.449301+0000","last_peered":"2024-09-13T14:20:18.449301+0000","last_clean":"2024-09-13T14:20:18.449301+0000","last_became_active":"2024-09-13T14:19:25.992968+0000","last_became_peered":"2024-09-13T14:19:25.992968+0000","last_unstale":"2024-09-13T14:20:18.449301+0000","last_undegraded":"2024-09-13T14:20:18.449301+0000","last_fullsized":"2024-09-13T14:20:18.449301+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:08.589526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"57'79","reported_seq":55,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.572653+0000","last_change":"2024-09-13T14:17:42.845082+0000","last_active":"2024-09-13T14:20:18.572653+0000","last_peered":"2024-09-13T14:20:18.572653+0000","last_clean":"2024-09-13T14:20:18.572653+0000","last_became_active":"2024-09-13T14:17:42.839116+0000","last_became_peered":"2024-09-13T14:17:42.839116+0000","last_unstale":"2024-09-13T14:20:18.572653+0000","last_undegraded":"2024-09-13T14:20:18.572653+0000","last_fullsized":"2024-09-13T14:20:18.572653+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:15:42.069756+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:58.144190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":27,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.443616+0000","last_change":"2024-09-13T14:18:14.644401+0000","last_active":"2024-09-13T14:20:18.443616+0000","last_peered":"2024-09-13T14:20:18.443616+0000","last_clean":"2024-09-13T14:20:18.443616+0000","last_became_active":"2024-09-13T14:18:10.616981+0000","last_became_peered":"2024-09-13T14:18:10.616981+0000","last_unstale":"2024-09-13T14:20:18.443616+0000","last_undegraded":"2024-09-13T14:20:18.443616+0000","last_fullsized":"2024-09-13T14:20:18.443616+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:00:32.657637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034405999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.445092+0000","last_change":"2024-09-13T14:19:25.873744+0000","last_active":"2024-09-13T14:20:18.445092+0000","last_peered":"2024-09-13T14:20:18.445092+0000","last_clean":"2024-09-13T14:20:18.445092+0000","last_became_active":"2024-09-13T14:19:25.873502+0000","last_became_peered":"2024-09-13T14:19:25.873502+0000","last_unstale":"2024-09-13T14:20:18.445092+0000","last_undegraded":"2024-09-13T14:20:18.445092+0000","last_fullsized":"2024-09-13T14:20:18.445092+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:21:52.217779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.443416+0000","last_change":"2024-09-13T14:19:27.379312+0000","last_active":"2024-09-13T14:20:18.443416+0000","last_peered":"2024-09-13T14:20:18.443416+0000","last_clean":"2024-09-13T14:20:18.443416+0000","last_became_active":"2024-09-13T14:19:27.378947+0000","last_became_peered":"2024-09-13T14:19:27.378947+0000","last_unstale":"2024-09-13T14:20:18.443416+0000","last_undegraded":"2024-09-13T14:20:18.443416+0000","last_fullsized":"2024-09-13T14:20:18.443416+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:54:20.706578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":26,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.572547+0000","last_change":"2024-09-13T14:18:14.638950+0000","last_active":"2024-09-13T14:20:18.572547+0000","last_peered":"2024-09-13T14:20:18.572547+0000","last_clean":"2024-09-13T14:20:18.572547+0000","last_became_active":"2024-09-13T14:18:10.718659+0000","last_became_peered":"2024-09-13T14:18:10.718659+0000","last_unstale":"2024-09-13T14:20:18.572547+0000","last_undegraded":"2024-09-13T14:20:18.572547+0000","last_fullsized":"2024-09-13T14:20:18.572547+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:38:08.688330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.597875+0000","last_change":"2024-09-13T14:19:25.877913+0000","last_active":"2024-09-13T14:20:18.597875+0000","last_peered":"2024-09-13T14:20:18.597875+0000","last_clean":"2024-09-13T14:20:18.597875+0000","last_became_active":"2024-09-13T14:19:25.877331+0000","last_became_peered":"2024-09-13T14:19:25.877331+0000","last_unstale":"2024-09-13T14:20:18.597875+0000","last_undegraded":"2024-09-13T14:20:18.597875+0000","last_fullsized":"2024-09-13T14:20:18.597875+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:38:21.235489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.598071+0000","last_change":"2024-09-13T14:19:26.889244+0000","last_active":"2024-09-13T14:20:18.598071+0000","last_peered":"2024-09-13T14:20:18.598071+0000","last_clean":"2024-09-13T14:20:18.598071+0000","last_became_active":"2024-09-13T14:19:26.888994+0000","last_became_peered":"2024-09-13T14:19:26.888994+0000","last_unstale":"2024-09-13T14:20:18.598071+0000","last_undegraded":"2024-09-13T14:20:18.598071+0000","last_fullsized":"2024-09-13T14:20:18.598071+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:42.933815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.444685+0000","last_change":"2024-09-13T14:18:14.641344+0000","last_active":"2024-09-13T14:20:18.444685+0000","last_peered":"2024-09-13T14:20:18.444685+0000","last_clean":"2024-09-13T14:20:18.444685+0000","last_became_active":"2024-09-13T14:18:10.766103+0000","last_became_peered":"2024-09-13T14:18:10.766103+0000","last_unstale":"2024-09-13T14:20:18.444685+0000","last_undegraded":"2024-09-13T14:20:18.444685+0000","last_fullsized":"2024-09-13T14:20:18.444685+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:03.456609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.449273+0000","last_change":"2024-09-13T14:19:25.867193+0000","last_active":"2024-09-13T14:20:18.449273+0000","last_peered":"2024-09-13T14:20:18.449273+0000","last_clean":"2024-09-13T14:20:18.449273+0000","last_became_active":"2024-09-13T14:19:25.866906+0000","last_became_peered":"2024-09-13T14:19:25.866906+0000","last_unstale":"2024-09-13T14:20:18.449273+0000","last_undegraded":"2024-09-13T14:20:18.449273+0000","last_fullsized":"2024-09-13T14:20:18.449273+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:24.553692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.449372+0000","last_change":"2024-09-13T14:19:25.884601+0000","last_active":"2024-09-13T14:20:18.449372+0000","last_peered":"2024-09-13T14:20:18.449372+0000","last_clean":"2024-09-13T14:20:18.449372+0000","last_became_active":"2024-09-13T14:19:25.884298+0000","last_became_peered":"2024-09-13T14:19:25.884298+0000","last_unstale":"2024-09-13T14:20:18.449372+0000","last_undegraded":"2024-09-13T14:20:18.449372+0000","last_fullsized":"2024-09-13T14:20:18.449372+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:55:51.018565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":28,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.443101+0000","last_change":"2024-09-13T14:18:14.644865+0000","last_active":"2024-09-13T14:20:18.443101+0000","last_peered":"2024-09-13T14:20:18.443101+0000","last_clean":"2024-09-13T14:20:18.443101+0000","last_became_active":"2024-09-13T14:18:10.617018+0000","last_became_peered":"2024-09-13T14:18:10.617018+0000","last_unstale":"2024-09-13T14:20:18.443101+0000","last_undegraded":"2024-09-13T14:20:18.443101+0000","last_fullsized":"2024-09-13T14:20:18.443101+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:01:22.505985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.440860+0000","last_change":"2024-09-13T14:19:25.871375+0000","last_active":"2024-09-13T14:20:18.440860+0000","last_peered":"2024-09-13T14:20:18.440860+0000","last_clean":"2024-09-13T14:20:18.440860+0000","last_became_active":"2024-09-13T14:19:25.870907+0000","last_became_peered":"2024-09-13T14:19:25.870907+0000","last_unstale":"2024-09-13T14:20:18.440860+0000","last_undegraded":"2024-09-13T14:20:18.440860+0000","last_fullsized":"2024-09-13T14:20:18.440860+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:05:00.250780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.448823+0000","last_change":"2024-09-13T14:19:26.209757+0000","last_active":"2024-09-13T14:20:18.448823+0000","last_peered":"2024-09-13T14:20:18.448823+0000","last_clean":"2024-09-13T14:20:18.448823+0000","last_became_active":"2024-09-13T14:19:26.209453+0000","last_became_peered":"2024-09-13T14:19:26.209453+0000","last_unstale":"2024-09-13T14:20:18.448823+0000","last_undegraded":"2024-09-13T14:20:18.448823+0000","last_fullsized":"2024-09-13T14:20:18.448823+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:06.060130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.443399+0000","last_change":"2024-09-13T14:19:25.874113+0000","last_active":"2024-09-13T14:20:18.443399+0000","last_peered":"2024-09-13T14:20:18.443399+0000","last_clean":"2024-09-13T14:20:18.443399+0000","last_became_active":"2024-09-13T14:19:25.873780+0000","last_became_peered":"2024-09-13T14:19:25.873780+0000","last_unstale":"2024-09-13T14:20:18.443399+0000","last_undegraded":"2024-09-13T14:20:18.443399+0000","last_fullsized":"2024-09-13T14:20:18.443399+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:14:53.429574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.448904+0000","last_change":"2024-09-13T14:18:14.641641+0000","last_active":"2024-09-13T14:20:18.448904+0000","last_peered":"2024-09-13T14:20:18.448904+0000","last_clean":"2024-09-13T14:20:18.448904+0000","last_became_active":"2024-09-13T14:18:10.718349+0000","last_became_peered":"2024-09-13T14:18:10.718349+0000","last_unstale":"2024-09-13T14:20:18.448904+0000","last_undegraded":"2024-09-13T14:20:18.448904+0000","last_fullsized":"2024-09-13T14:20:18.448904+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:23.005181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.445163+0000","last_change":"2024-09-13T14:19:25.885166+0000","last_active":"2024-09-13T14:20:18.445163+0000","last_peered":"2024-09-13T14:20:18.445163+0000","last_clean":"2024-09-13T14:20:18.445163+0000","last_became_active":"2024-09-13T14:19:25.884888+0000","last_became_peered":"2024-09-13T14:19:25.884888+0000","last_unstale":"2024-09-13T14:20:18.445163+0000","last_undegraded":"2024-09-13T14:20:18.445163+0000","last_fullsized":"2024-09-13T14:20:18.445163+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:36:38.469866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.448881+0000","last_change":"2024-09-13T14:19:25.866481+0000","last_active":"2024-09-13T14:20:18.448881+0000","last_peered":"2024-09-13T14:20:18.448881+0000","last_clean":"2024-09-13T14:20:18.448881+0000","last_became_active":"2024-09-13T14:19:25.866198+0000","last_became_peered":"2024-09-13T14:19:25.866198+0000","last_unstale":"2024-09-13T14:20:18.448881+0000","last_undegraded":"2024-09-13T14:20:18.448881+0000","last_fullsized":"2024-09-13T14:20:18.448881+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:51.651674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":26,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.572930+0000","last_change":"2024-09-13T14:18:14.639101+0000","last_active":"2024-09-13T14:20:18.572930+0000","last_peered":"2024-09-13T14:20:18.572930+0000","last_clean":"2024-09-13T14:20:18.572930+0000","last_became_active":"2024-09-13T14:18:10.718655+0000","last_became_peered":"2024-09-13T14:18:10.718655+0000","last_unstale":"2024-09-13T14:20:18.572930+0000","last_undegraded":"2024-09-13T14:20:18.572930+0000","last_fullsized":"2024-09-13T14:20:18.572930+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:59:01.637559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.598021+0000","last_change":"2024-09-13T14:19:26.887815+0000","last_active":"2024-09-13T14:20:18.598021+0000","last_peered":"2024-09-13T14:20:18.598021+0000","last_clean":"2024-09-13T14:20:18.598021+0000","last_became_active":"2024-09-13T14:19:26.887539+0000","last_became_peered":"2024-09-13T14:19:26.887539+0000","last_unstale":"2024-09-13T14:20:18.598021+0000","last_undegraded":"2024-09-13T14:20:18.598021+0000","last_fullsized":"2024-09-13T14:20:18.598021+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:23:00.385203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":16,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.447818+0000","last_change":"2024-09-13T14:19:25.879344+0000","last_active":"2024-09-13T14:20:18.447818+0000","last_peered":"2024-09-13T14:20:18.447818+0000","last_clean":"2024-09-13T14:20:18.447818+0000","last_became_active":"2024-09-13T14:19:25.879030+0000","last_became_peered":"2024-09-13T14:19:25.879030+0000","last_unstale":"2024-09-13T14:20:18.447818+0000","last_undegraded":"2024-09-13T14:20:18.447818+0000","last_fullsized":"2024-09-13T14:20:18.447818+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:51.493477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.449609+0000","last_change":"2024-09-13T14:18:14.641734+0000","last_active":"2024-09-13T14:20:18.449609+0000","last_peered":"2024-09-13T14:20:18.449609+0000","last_clean":"2024-09-13T14:20:18.449609+0000","last_became_active":"2024-09-13T14:18:10.614303+0000","last_became_peered":"2024-09-13T14:18:10.614303+0000","last_unstale":"2024-09-13T14:20:18.449609+0000","last_undegraded":"2024-09-13T14:20:18.449609+0000","last_fullsized":"2024-09-13T14:20:18.449609+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:28:38.174245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034192599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.446878+0000","last_change":"2024-09-13T14:19:26.890828+0000","last_active":"2024-09-13T14:20:18.446878+0000","last_peered":"2024-09-13T14:20:18.446878+0000","last_clean":"2024-09-13T14:20:18.446878+0000","last_became_active":"2024-09-13T14:19:26.889977+0000","last_became_peered":"2024-09-13T14:19:26.889977+0000","last_unstale":"2024-09-13T14:20:18.446878+0000","last_undegraded":"2024-09-13T14:20:18.446878+0000","last_fullsized":"2024-09-13T14:20:18.446878+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:13:19.115622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.598137+0000","last_change":"2024-09-13T14:19:25.869612+0000","last_active":"2024-09-13T14:20:18.598137+0000","last_peered":"2024-09-13T14:20:18.598137+0000","last_clean":"2024-09-13T14:20:18.598137+0000","last_became_active":"2024-09-13T14:19:25.869220+0000","last_became_peered":"2024-09-13T14:19:25.869220+0000","last_unstale":"2024-09-13T14:20:18.598137+0000","last_undegraded":"2024-09-13T14:20:18.598137+0000","last_fullsized":"2024-09-13T14:20:18.598137+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:11:25.488245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.443228+0000","last_change":"2024-09-13T14:18:14.645599+0000","last_active":"2024-09-13T14:20:18.443228+0000","last_peered":"2024-09-13T14:20:18.443228+0000","last_clean":"2024-09-13T14:20:18.443228+0000","last_became_active":"2024-09-13T14:18:10.617927+0000","last_became_peered":"2024-09-13T14:18:10.617927+0000","last_unstale":"2024-09-13T14:20:18.443228+0000","last_undegraded":"2024-09-13T14:20:18.443228+0000","last_fullsized":"2024-09-13T14:20:18.443228+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:51:43.437167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.440426+0000","last_change":"2024-09-13T14:19:26.131609+0000","last_active":"2024-09-13T14:20:18.440426+0000","last_peered":"2024-09-13T14:20:18.440426+0000","last_clean":"2024-09-13T14:20:18.440426+0000","last_became_active":"2024-09-13T14:19:26.131029+0000","last_became_peered":"2024-09-13T14:19:26.131029+0000","last_unstale":"2024-09-13T14:20:18.440426+0000","last_undegraded":"2024-09-13T14:20:18.440426+0000","last_fullsized":"2024-09-13T14:20:18.440426+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:57.933696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.440764+0000","last_change":"2024-09-13T14:19:26.131694+0000","last_active":"2024-09-13T14:20:18.440764+0000","last_peered":"2024-09-13T14:20:18.440764+0000","last_clean":"2024-09-13T14:20:18.440764+0000","last_became_active":"2024-09-13T14:19:26.131121+0000","last_became_peered":"2024-09-13T14:19:26.131121+0000","last_unstale":"2024-09-13T14:20:18.440764+0000","last_undegraded":"2024-09-13T14:20:18.440764+0000","last_fullsized":"2024-09-13T14:20:18.440764+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:10:17.349612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"63'6","reported_seq":20,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.440805+0000","last_change":"2024-09-13T14:19:25.870605+0000","last_active":"2024-09-13T14:20:18.440805+0000","last_peered":"2024-09-13T14:20:18.440805+0000","last_clean":"2024-09-13T14:20:18.440805+0000","last_became_active":"2024-09-13T14:19:25.870282+0000","last_became_peered":"2024-09-13T14:19:25.870282+0000","last_unstale":"2024-09-13T14:20:18.440805+0000","last_undegraded":"2024-09-13T14:20:18.440805+0000","last_fullsized":"2024-09-13T14:20:18.440805+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:52.514915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1552,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.445043+0000","last_change":"2024-09-13T14:19:26.209743+0000","last_active":"2024-09-13T14:20:18.445043+0000","last_peered":"2024-09-13T14:20:18.445043+0000","last_clean":"2024-09-13T14:20:18.445043+0000","last_became_active":"2024-09-13T14:19:26.209460+0000","last_became_peered":"2024-09-13T14:19:26.209460+0000","last_unstale":"2024-09-13T14:20:18.445043+0000","last_undegraded":"2024-09-13T14:20:18.445043+0000","last_fullsized":"2024-09-13T14:20:18.445043+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:15:52.778019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":17,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.572820+0000","last_change":"2024-09-13T14:19:25.875899+0000","last_active":"2024-09-13T14:20:18.572820+0000","last_peered":"2024-09-13T14:20:18.572820+0000","last_clean":"2024-09-13T14:20:18.572820+0000","last_became_active":"2024-09-13T14:19:25.875598+0000","last_became_peered":"2024-09-13T14:19:25.875598+0000","last_unstale":"2024-09-13T14:20:18.572820+0000","last_undegraded":"2024-09-13T14:20:18.572820+0000","last_fullsized":"2024-09-13T14:20:18.572820+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:45.274956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.443437+0000","last_change":"2024-09-13T14:19:25.872219+0000","last_active":"2024-09-13T14:20:18.443437+0000","last_peered":"2024-09-13T14:20:18.443437+0000","last_clean":"2024-09-13T14:20:18.443437+0000","last_became_active":"2024-09-13T14:19:25.871936+0000","last_became_peered":"2024-09-13T14:19:25.871936+0000","last_unstale":"2024-09-13T14:20:18.443437+0000","last_undegraded":"2024-09-13T14:20:18.443437+0000","last_fullsized":"2024-09-13T14:20:18.443437+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:58.476558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.444682+0000","last_change":"2024-09-13T14:19:26.890559+0000","last_active":"2024-09-13T14:20:18.444682+0000","last_peered":"2024-09-13T14:20:18.444682+0000","last_clean":"2024-09-13T14:20:18.444682+0000","last_became_active":"2024-09-13T14:19:26.890266+0000","last_became_peered":"2024-09-13T14:19:26.890266+0000","last_unstale":"2024-09-13T14:20:18.444682+0000","last_undegraded":"2024-09-13T14:20:18.444682+0000","last_fullsized":"2024-09-13T14:20:18.444682+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:27.600428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.598211+0000","last_change":"2024-09-13T14:19:25.876770+0000","last_active":"2024-09-13T14:20:18.598211+0000","last_peered":"2024-09-13T14:20:18.598211+0000","last_clean":"2024-09-13T14:20:18.598211+0000","last_became_active":"2024-09-13T14:19:25.876534+0000","last_became_peered":"2024-09-13T14:19:25.876534+0000","last_unstale":"2024-09-13T14:20:18.598211+0000","last_undegraded":"2024-09-13T14:20:18.598211+0000","last_fullsized":"2024-09-13T14:20:18.598211+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:53:21.516150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-13T14:20:19.825757+0000","last_change":"2024-09-13T14:20:19.825757+0000","last_active":"2024-09-13T14:20:19.825757+0000","last_peered":"2024-09-13T14:20:19.825757+0000","last_clean":"2024-09-13T14:20:19.825757+0000","last_became_active":"2024-09-13T14:20:19.825414+0000","last_became_peered":"2024-09-13T14:20:19.825414+0000","last_unstale":"2024-09-13T14:20:19.825757+0000","last_undegraded":"2024-09-13T14:20:19.825757+0000","last_fullsized":"2024-09-13T14:20:19.825757+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:40:37.165447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.443129+0000","last_change":"2024-09-13T14:19:26.887783+0000","last_active":"2024-09-13T14:20:18.443129+0000","last_peered":"2024-09-13T14:20:18.443129+0000","last_clean":"2024-09-13T14:20:18.443129+0000","last_became_active":"2024-09-13T14:19:26.887509+0000","last_became_peered":"2024-09-13T14:19:26.887509+0000","last_unstale":"2024-09-13T14:20:18.443129+0000","last_undegraded":"2024-09-13T14:20:18.443129+0000","last_fullsized":"2024-09-13T14:20:18.443129+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:20.774114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.446955+0000","last_change":"2024-09-13T14:19:25.877905+0000","last_active":"2024-09-13T14:20:18.446955+0000","last_peered":"2024-09-13T14:20:18.446955+0000","last_clean":"2024-09-13T14:20:18.446955+0000","last_became_active":"2024-09-13T14:19:25.877618+0000","last_became_peered":"2024-09-13T14:19:25.877618+0000","last_unstale":"2024-09-13T14:20:18.446955+0000","last_undegraded":"2024-09-13T14:20:18.446955+0000","last_fullsized":"2024-09-13T14:20:18.446955+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:30:03.430683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.598015+0000","last_change":"2024-09-13T14:19:26.891225+0000","last_active":"2024-09-13T14:20:18.598015+0000","last_peered":"2024-09-13T14:20:18.598015+0000","last_clean":"2024-09-13T14:20:18.598015+0000","last_became_active":"2024-09-13T14:19:26.890936+0000","last_became_peered":"2024-09-13T14:19:26.890936+0000","last_unstale":"2024-09-13T14:20:18.598015+0000","last_undegraded":"2024-09-13T14:20:18.598015+0000","last_fullsized":"2024-09-13T14:20:18.598015+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:27:27.983810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.444902+0000","last_change":"2024-09-13T14:19:25.867272+0000","last_active":"2024-09-13T14:20:18.444902+0000","last_peered":"2024-09-13T14:20:18.444902+0000","last_clean":"2024-09-13T14:20:18.444902+0000","last_became_active":"2024-09-13T14:19:25.866976+0000","last_became_peered":"2024-09-13T14:19:25.866976+0000","last_unstale":"2024-09-13T14:20:18.444902+0000","last_undegraded":"2024-09-13T14:20:18.444902+0000","last_fullsized":"2024-09-13T14:20:18.444902+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:40.024931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.445614+0000","last_change":"2024-09-13T14:19:27.379386+0000","last_active":"2024-09-13T14:20:18.445614+0000","last_peered":"2024-09-13T14:20:18.445614+0000","last_clean":"2024-09-13T14:20:18.445614+0000","last_became_active":"2024-09-13T14:19:27.379036+0000","last_became_peered":"2024-09-13T14:19:27.379036+0000","last_unstale":"2024-09-13T14:20:18.445614+0000","last_undegraded":"2024-09-13T14:20:18.445614+0000","last_fullsized":"2024-09-13T14:20:18.445614+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:25:55.319899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.440532+0000","last_change":"2024-09-13T14:19:25.867061+0000","last_active":"2024-09-13T14:20:18.440532+0000","last_peered":"2024-09-13T14:20:18.440532+0000","last_clean":"2024-09-13T14:20:18.440532+0000","last_became_active":"2024-09-13T14:19:25.866766+0000","last_became_peered":"2024-09-13T14:19:25.866766+0000","last_unstale":"2024-09-13T14:20:18.440532+0000","last_undegraded":"2024-09-13T14:20:18.440532+0000","last_fullsized":"2024-09-13T14:20:18.440532+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:17:30.834847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.445118+0000","last_change":"2024-09-13T14:19:26.887515+0000","last_active":"2024-09-13T14:20:18.445118+0000","last_peered":"2024-09-13T14:20:18.445118+0000","last_clean":"2024-09-13T14:20:18.445118+0000","last_became_active":"2024-09-13T14:19:26.887247+0000","last_became_peered":"2024-09-13T14:19:26.887247+0000","last_unstale":"2024-09-13T14:20:18.445118+0000","last_undegraded":"2024-09-13T14:20:18.445118+0000","last_fullsized":"2024-09-13T14:20:18.445118+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:07:32.437260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.598076+0000","last_change":"2024-09-13T14:19:25.876086+0000","last_active":"2024-09-13T14:20:18.598076+0000","last_peered":"2024-09-13T14:20:18.598076+0000","last_clean":"2024-09-13T14:20:18.598076+0000","last_became_active":"2024-09-13T14:19:25.873007+0000","last_became_peered":"2024-09-13T14:19:25.873007+0000","last_unstale":"2024-09-13T14:20:18.598076+0000","last_undegraded":"2024-09-13T14:20:18.598076+0000","last_fullsized":"2024-09-13T14:20:18.598076+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:41:18.113589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":23,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-13T14:20:19.992899+0000","last_change":"2024-09-13T14:20:19.992899+0000","last_active":"2024-09-13T14:20:19.992899+0000","last_peered":"2024-09-13T14:20:19.992899+0000","last_clean":"2024-09-13T14:20:19.992899+0000","last_became_active":"2024-09-13T14:20:19.992590+0000","last_became_peered":"2024-09-13T14:20:19.992590+0000","last_unstale":"2024-09-13T14:20:19.992899+0000","last_undegraded":"2024-09-13T14:20:19.992899+0000","last_fullsized":"2024-09-13T14:20:19.992899+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:20.912929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":16,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.443136+0000","last_change":"2024-09-13T14:19:25.873795+0000","last_active":"2024-09-13T14:20:18.443136+0000","last_peered":"2024-09-13T14:20:18.443136+0000","last_clean":"2024-09-13T14:20:18.443136+0000","last_became_active":"2024-09-13T14:19:25.873387+0000","last_became_peered":"2024-09-13T14:19:25.873387+0000","last_unstale":"2024-09-13T14:20:18.443136+0000","last_undegraded":"2024-09-13T14:20:18.443136+0000","last_fullsized":"2024-09-13T14:20:18.443136+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:55:49.807951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.440506+0000","last_change":"2024-09-13T14:19:25.993592+0000","last_active":"2024-09-13T14:20:18.440506+0000","last_peered":"2024-09-13T14:20:18.440506+0000","last_clean":"2024-09-13T14:20:18.440506+0000","last_became_active":"2024-09-13T14:19:25.993144+0000","last_became_peered":"2024-09-13T14:19:25.993144+0000","last_unstale":"2024-09-13T14:20:18.440506+0000","last_undegraded":"2024-09-13T14:20:18.440506+0000","last_fullsized":"2024-09-13T14:20:18.440506+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:09:24.672117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":16,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.443331+0000","last_change":"2024-09-13T14:19:25.873514+0000","last_active":"2024-09-13T14:20:18.443331+0000","last_peered":"2024-09-13T14:20:18.443331+0000","last_clean":"2024-09-13T14:20:18.443331+0000","last_became_active":"2024-09-13T14:19:25.873170+0000","last_became_peered":"2024-09-13T14:19:25.873170+0000","last_unstale":"2024-09-13T14:20:18.443331+0000","last_undegraded":"2024-09-13T14:20:18.443331+0000","last_fullsized":"2024-09-13T14:20:18.443331+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:48.298530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.444864+0000","last_change":"2024-09-13T14:19:26.887677+0000","last_active":"2024-09-13T14:20:18.444864+0000","last_peered":"2024-09-13T14:20:18.444864+0000","last_clean":"2024-09-13T14:20:18.444864+0000","last_became_active":"2024-09-13T14:19:26.887372+0000","last_became_peered":"2024-09-13T14:19:26.887372+0000","last_unstale":"2024-09-13T14:20:18.444864+0000","last_undegraded":"2024-09-13T14:20:18.444864+0000","last_fullsized":"2024-09-13T14:20:18.444864+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:55:43.765983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.444710+0000","last_change":"2024-09-13T14:19:25.872630+0000","last_active":"2024-09-13T14:20:18.444710+0000","last_peered":"2024-09-13T14:20:18.444710+0000","last_clean":"2024-09-13T14:20:18.444710+0000","last_became_active":"2024-09-13T14:19:25.872252+0000","last_became_peered":"2024-09-13T14:19:25.872252+0000","last_unstale":"2024-09-13T14:20:18.444710+0000","last_undegraded":"2024-09-13T14:20:18.444710+0000","last_fullsized":"2024-09-13T14:20:18.444710+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:03:03.501514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":23,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-13T14:20:19.825870+0000","last_change":"2024-09-13T14:20:19.825870+0000","last_active":"2024-09-13T14:20:19.825870+0000","last_peered":"2024-09-13T14:20:19.825870+0000","last_clean":"2024-09-13T14:20:19.825870+0000","last_became_active":"2024-09-13T14:20:19.825471+0000","last_became_peered":"2024-09-13T14:20:19.825471+0000","last_unstale":"2024-09-13T14:20:19.825870+0000","last_undegraded":"2024-09-13T14:20:19.825870+0000","last_fullsized":"2024-09-13T14:20:19.825870+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:40:23.580724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":18,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.598198+0000","last_change":"2024-09-13T14:19:25.869427+0000","last_active":"2024-09-13T14:20:18.598198+0000","last_peered":"2024-09-13T14:20:18.598198+0000","last_clean":"2024-09-13T14:20:18.598198+0000","last_became_active":"2024-09-13T14:19:25.869094+0000","last_became_peered":"2024-09-13T14:19:25.869094+0000","last_unstale":"2024-09-13T14:20:18.598198+0000","last_undegraded":"2024-09-13T14:20:18.598198+0000","last_fullsized":"2024-09-13T14:20:18.598198+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:16.636504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2338,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":7014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430244,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298616,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446536,"statfs":{"total":95995035648,"available":95689252864,"internally_reserved":0,"allocated":1286144,"data_stored":884657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51700000000000002,"15min":0.51800000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.52600000000000002,"15min":0.53400000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.57399999999999995}]},{"osd":1,"last update":"Fri Sep 13 14:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.53400000000000003,"15min":0.54200000000000004},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.629},{"interface":"front","average":{"1min":0.501,"5min":0.51500000000000001,"15min":0.51700000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.52900000000000003}]},{"osd":2,"last update":"Fri Sep 13 14:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.51200000000000001,"15min":0.51300000000000001},"min":{"1min":0.40000000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.64400000000000002,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.499,"5min":0.53300000000000003,"15min":0.53900000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.64100000000000001}]},{"osd":3,"last update":"Fri Sep 13 14:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.51200000000000001,"15min":0.51400000000000001},"min":{"1min":0.39100000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.47999999999999998}]},{"osd":4,"last update":"Fri Sep 13 14:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.44,"5min":0.46400000000000002,"15min":0.46800000000000003},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.39600000000000002}]},{"osd":5,"last update":"Fri Sep 13 14:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.45000000000000001,"15min":0.44600000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47899999999999998,"15min":0.47999999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.69499999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.442}]},{"osd":6,"last update":"Fri Sep 13 14:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.49099999999999999,"15min":0.49199999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.50900000000000001,"15min":0.51100000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.65200000000000002,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.54100000000000004}]}]},{"osd":6,"up_from":42,"seq":180388626473,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298596,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446556,"statfs":{"total":95995035648,"available":95689273344,"internally_reserved":0,"allocated":1269760,"data_stored":883971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51100000000000001,"15min":0.51000000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.72699999999999998,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.47299999999999998,"15min":0.47099999999999997},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.63200000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.65700000000000003}]},{"osd":1,"last update":"Fri Sep 13 14:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.497,"15min":0.49199999999999999},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.73799999999999999}]},{"osd":2,"last update":"Fri Sep 13 14:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.48299999999999998,"15min":0.48099999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.51600000000000001,"15min":0.51500000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.68000000000000005}]},{"osd":3,"last update":"Fri Sep 13 14:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.63400000000000001,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.746,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.69399999999999995}]},{"osd":4,"last update":"Fri Sep 13 14:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.38200000000000001,"15min":0.37},"min":{"1min":0.315,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.622},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.41599999999999998,"15min":0.40899999999999997},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.59099999999999997}]},{"osd":5,"last update":"Fri Sep 13 14:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.41599999999999998,"15min":0.40999999999999998},"min":{"1min":0.27100000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.64900000000000002,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.434,"15min":0.42399999999999999},"min":{"1min":0.315,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.497}]},{"osd":7,"last update":"Fri Sep 13 14:19:44 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.46300000000000002,"15min":0.45700000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.50600000000000001,"15min":0.498},"min":{"1min":0.34499999999999997,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.753}]}]},{"osd":5,"up_from":36,"seq":154618822702,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298012,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447140,"statfs":{"total":95995035648,"available":95689871360,"internally_reserved":0,"allocated":667648,"data_stored":292621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.52100000000000002,"15min":0.52500000000000002},"min":{"1min":0.36499999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.70299999999999996,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.498,"15min":0.496},"min":{"1min":0.30199999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.64900000000000002,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.52900000000000003}]},{"osd":1,"last update":"Fri Sep 13 14:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.51400000000000001,"15min":0.51800000000000002},"min":{"1min":0.36799999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.58099999999999996,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.498},{"interface":"front","average":{"1min":0.496,"5min":0.51400000000000001,"15min":0.51500000000000001},"min":{"1min":0.33300000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.71599999999999997,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.56999999999999995}]},{"osd":2,"last update":"Fri Sep 13 14:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.51700000000000002,"15min":0.52300000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.67300000000000004,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.34200000000000003,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.70599999999999996,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.54300000000000004}]},{"osd":3,"last update":"Fri Sep 13 14:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.50900000000000001,"15min":0.50600000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.70999999999999996,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.53200000000000003,"15min":0.53700000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.68300000000000005,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.60299999999999998}]},{"osd":4,"last update":"Fri Sep 13 14:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.38900000000000001,"15min":0.374},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.436,"5min":0.39700000000000002,"15min":0.378},"min":{"1min":0.30199999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.63600000000000001,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.432}]},{"osd":6,"last update":"Fri Sep 13 14:19:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.40100000000000002,"15min":0.38900000000000001},"min":{"1min":0.27300000000000002,"5min":0.221,"15min":0.221},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.373},{"interface":"front","average":{"1min":0.503,"5min":0.42699999999999999,"15min":0.41499999999999998},"min":{"1min":0.28999999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.44900000000000001}]},{"osd":7,"last update":"Fri Sep 13 14:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.46800000000000003,"15min":0.46400000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.74099999999999999,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.50900000000000001,"15min":0.502},"min":{"1min":0.312,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.40400000000000003}]}]},{"osd":4,"up_from":30,"seq":128849018930,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298028,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447124,"statfs":{"total":95995035648,"available":95689854976,"internally_reserved":0,"allocated":684032,"data_stored":293975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.52700000000000002,"15min":0.53900000000000003},"min":{"1min":0.33700000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.68300000000000005,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.55100000000000005,"15min":0.55900000000000005},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.79300000000000004,"5min":0.88,"15min":0.88},"last":0.495}]},{"osd":1,"last update":"Fri Sep 13 14:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53500000000000003,"15min":0.54100000000000004},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.72899999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.55000000000000004,"15min":0.56000000000000005},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.74099999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.55000000000000004}]},{"osd":2,"last update":"Fri Sep 13 14:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.54500000000000004,"15min":0.55300000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.71899999999999997,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.52500000000000002,"15min":0.53300000000000003},"min":{"1min":0.36399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.74099999999999999,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.36499999999999999}]},{"osd":3,"last update":"Fri Sep 13 14:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.53200000000000003,"15min":0.53800000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.71699999999999997,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53100000000000003,"15min":0.53800000000000003},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.47699999999999998}]},{"osd":5,"last update":"Fri Sep 13 14:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.39400000000000002,"15min":0.38},"min":{"1min":0.28199999999999997,"5min":0.254,"15min":0.254},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.443,"15min":0.42099999999999999},"min":{"1min":0.33600000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.38100000000000001}]},{"osd":6,"last update":"Fri Sep 13 14:20:22 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.44400000000000001,"15min":0.42499999999999999},"min":{"1min":0.33000000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.42199999999999999,"15min":0.40999999999999998},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.43099999999999999}]},{"osd":7,"last update":"Fri Sep 13 14:19:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.46200000000000002,"15min":0.45100000000000001},"min":{"1min":0.32300000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.45800000000000002,"15min":0.44800000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.50600000000000001}]}]},{"osd":3,"up_from":25,"seq":107374182456,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298004,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447148,"statfs":{"total":95995035648,"available":95689879552,"internally_reserved":0,"allocated":663552,"data_stored":292367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.376,"15min":0.40200000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.44600000000000001,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.377,"15min":0.39500000000000002},"min":{"1min":0.26300000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.46899999999999997,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.47999999999999998}]},{"osd":1,"last update":"Fri Sep 13 14:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.36899999999999999,"15min":0.38800000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.47299999999999998,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.41399999999999998,"15min":0.44},"min":{"1min":0.23799999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.52700000000000002,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.48699999999999999}]},{"osd":2,"last update":"Fri Sep 13 14:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.42899999999999999,"15min":0.45200000000000001},"min":{"1min":0.28000000000000003,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.48699999999999999,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.438},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.39700000000000002,"15min":0.42399999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.496,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.44700000000000001}]},{"osd":4,"last update":"Fri Sep 13 14:19:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.52800000000000002,"15min":0.52700000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.42899999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.63400000000000001}]},{"osd":5,"last update":"Fri Sep 13 14:20:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54300000000000004,"15min":0.54600000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55000000000000004,"15min":0.55400000000000005},"min":{"1min":0.436,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.75600000000000001,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.61899999999999999}]},{"osd":6,"last update":"Fri Sep 13 14:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.55300000000000005,"15min":0.55000000000000004},"min":{"1min":0.39500000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.745,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.56899999999999995,"15min":0.57199999999999995},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.73699999999999999}]},{"osd":7,"last update":"Fri Sep 13 14:19:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.55600000000000005,"15min":0.55100000000000005},"min":{"1min":0.44900000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.53500000000000003,"15min":0.52800000000000002},"min":{"1min":0.441,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.65400000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298008,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447144,"statfs":{"total":95995035648,"available":95689875456,"internally_reserved":0,"allocated":663552,"data_stored":292637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:19:53 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.38800000000000001,"15min":0.38200000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.495,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.375,"15min":0.36199999999999999},"min":{"1min":0.22800000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.51400000000000001,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.23799999999999999}]},{"osd":1,"last update":"Fri Sep 13 14:19:53 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.39300000000000002,"15min":0.38400000000000001},"min":{"1min":0.25800000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.40200000000000002,"15min":0.39200000000000002},"min":{"1min":0.26400000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.56799999999999995,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.316}]},{"osd":3,"last update":"Fri Sep 13 14:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.438,"15min":0.45200000000000001},"min":{"1min":0.25900000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.54800000000000004,"5min":0.622,"15min":0.622},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.436,"15min":0.45000000000000001},"min":{"1min":0.28399999999999997,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.54300000000000004,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.27400000000000002}]},{"osd":4,"last update":"Fri Sep 13 14:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55700000000000005,"15min":0.56200000000000006},"min":{"1min":0.38600000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.73499999999999999,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.55500000000000005}]},{"osd":5,"last update":"Fri Sep 13 14:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.60899999999999999,"15min":0.626},"min":{"1min":0.443,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.75800000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58699999999999997,"15min":0.59499999999999997},"min":{"1min":0.45700000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71099999999999997,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.70999999999999996}]},{"osd":6,"last update":"Fri Sep 13 14:19:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.59199999999999997,"15min":0.59399999999999997},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.84899999999999998,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.56999999999999995}]},{"osd":7,"last update":"Fri Sep 13 14:19:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.59399999999999997,"15min":0.59699999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.78300000000000003,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.73099999999999998,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.64500000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298008,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447144,"statfs":{"total":95995035648,"available":95689875456,"internally_reserved":0,"allocated":663552,"data_stored":292225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:20:22 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.35199999999999998,"15min":0.315},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.47899999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.36899999999999999,"15min":0.32700000000000001},"min":{"1min":0.24399999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.42999999999999999,"5min":0.622,"15min":0.622},"last":0.42899999999999999}]},{"osd":2,"last update":"Fri Sep 13 14:19:50 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.39200000000000002,"15min":0.38900000000000001},"min":{"1min":0.23300000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.56999999999999995,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.379,"15min":0.36499999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.374}]},{"osd":3,"last update":"Fri Sep 13 14:20:14 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.39500000000000002,"15min":0.40500000000000003},"min":{"1min":0.26000000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.441,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.433,"15min":0.44700000000000001},"min":{"1min":0.28599999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.47999999999999998,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.50800000000000001}]},{"osd":4,"last update":"Fri Sep 13 14:19:33 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55400000000000005,"15min":0.56200000000000006},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.52500000000000002}]},{"osd":5,"last update":"Fri Sep 13 14:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.56100000000000005,"15min":0.56499999999999995},"min":{"1min":0.373,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.58499999999999996,"15min":0.59899999999999998},"min":{"1min":0.36299999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.47699999999999998}]},{"osd":6,"last update":"Fri Sep 13 14:19:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.56499999999999995,"15min":0.56799999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.57199999999999995,"15min":0.57299999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.53700000000000003}]},{"osd":7,"last update":"Fri Sep 13 14:19:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.57799999999999996,"15min":0.58699999999999997},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.746,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.59699999999999998,"15min":0.60499999999999998},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.85099999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.49099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298592,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446560,"statfs":{"total":95995035648,"available":95689277440,"internally_reserved":0,"allocated":1261568,"data_stored":883166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 14:20:21 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.33600000000000002,"15min":0.31},"min":{"1min":0.20200000000000001,"5min":0.193,"15min":0.193},"max":{"1min":0.55400000000000005,"5min":1.218,"15min":1.218},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.36799999999999999,"15min":0.33600000000000002},"min":{"1min":0.23499999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.60699999999999998,"5min":1.284,"15min":1.284},"last":0.42599999999999999}]},{"osd":2,"last update":"Fri Sep 13 14:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.52900000000000003,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.46200000000000002,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.41399999999999998}]},{"osd":3,"last update":"Fri Sep 13 14:20:18 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.41999999999999998,"15min":0.44},"min":{"1min":0.27100000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.52700000000000002,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.379,"15min":0.38800000000000001},"min":{"1min":0.26200000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.56699999999999995,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.40400000000000003}]},{"osd":4,"last update":"Fri Sep 13 14:19:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.50600000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.71399999999999997,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.748},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.503,"15min":0.499},"min":{"1min":0.36599999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.68200000000000005}]},{"osd":5,"last update":"Fri Sep 13 14:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.51900000000000002,"15min":0.51100000000000001},"min":{"1min":0.41899999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54700000000000004,"15min":0.54500000000000004},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.73599999999999999,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.64300000000000002}]},{"osd":6,"last update":"Fri Sep 13 14:20:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.56200000000000006,"15min":0.56899999999999995},"min":{"1min":0.42599999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.71899999999999997,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.629},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.73099999999999998}]},{"osd":7,"last update":"Fri Sep 13 14:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55700000000000005,"15min":0.56100000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73299999999999998,"5min":1.141,"15min":1.141},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.58499999999999996,"15min":0.59199999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.69099999999999995,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.72099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:20:28.531 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:20:28.837 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:28 smithi184 ceph-mon[33394]: from='client.24674 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:20:28.837 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:28 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:28.837 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:28 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:28.838 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:28 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:20:28.838 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:28 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:20:28.838 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:28 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:28.838 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:28 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:20:28.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:28 smithi151 ceph-mon[41387]: from='client.24674 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:20:28.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:28 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:28.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:28 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:28.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:28 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:20:28.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:28 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:20:28.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:28 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:28.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:28 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:20:28.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:28 smithi151 ceph-mon[29807]: from='client.24674 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:20:28.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:28 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:28.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:28 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:28.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:28 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:20:28.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:28 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:20:28.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:28 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:28.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:28 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:20:29.302 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:29.733 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:29 smithi151 ceph-mon[29807]: pgmap v76: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:29.733 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:29 smithi151 ceph-mon[41387]: pgmap v76: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:29.786 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:29 smithi184 ceph-mon[33394]: pgmap v76: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:31.289 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:20:31.290 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T14:20:31.762 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:31 smithi151 ceph-mon[29807]: pgmap v77: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:31.763 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:31 smithi151 ceph-mon[41387]: pgmap v77: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:31.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:31 smithi184 ceph-mon[33394]: pgmap v77: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:32.089 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2024-09-13T14:20:30.380782+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592725,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":160,"num_write_kb":1643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121,"ondisk_log_size":121,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385864,"kb_used_data":6992,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747575352,"statfs":{"total":767960285184,"available":765517160448,"internally_reserved":0,"allocated":7159808,"data_stored":4115619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.937921"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.445376+0000","last_change":"2024-09-13T14:19:26.890506+0000","last_active":"2024-09-13T14:20:18.445376+0000","last_peered":"2024-09-13T14:20:18.445376+0000","last_clean":"2024-09-13T14:20:18.445376+0000","last_became_active":"2024-09-13T14:19:26.890223+0000","last_became_peered":"2024-09-13T14:19:26.890223+0000","last_unstale":"2024-09-13T14:20:18.445376+0000","last_undegraded":"2024-09-13T14:20:18.445376+0000","last_fullsized":"2024-09-13T14:20:18.445376+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:38:01.381959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":16,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.572650+0000","last_change":"2024-09-13T14:19:25.867510+0000","last_active":"2024-09-13T14:20:18.572650+0000","last_peered":"2024-09-13T14:20:18.572650+0000","last_clean":"2024-09-13T14:20:18.572650+0000","last_became_active":"2024-09-13T14:19:25.867195+0000","last_became_peered":"2024-09-13T14:19:25.867195+0000","last_unstale":"2024-09-13T14:20:18.572650+0000","last_undegraded":"2024-09-13T14:20:18.572650+0000","last_fullsized":"2024-09-13T14:20:18.572650+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:08:09.116221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.443313+0000","last_change":"2024-09-13T14:19:26.888945+0000","last_active":"2024-09-13T14:20:18.443313+0000","last_peered":"2024-09-13T14:20:18.443313+0000","last_clean":"2024-09-13T14:20:18.443313+0000","last_became_active":"2024-09-13T14:19:26.888587+0000","last_became_peered":"2024-09-13T14:19:26.888587+0000","last_unstale":"2024-09-13T14:20:18.443313+0000","last_undegraded":"2024-09-13T14:20:18.443313+0000","last_fullsized":"2024-09-13T14:20:18.443313+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:25.696881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.598017+0000","last_change":"2024-09-13T14:19:25.877587+0000","last_active":"2024-09-13T14:20:18.598017+0000","last_peered":"2024-09-13T14:20:18.598017+0000","last_clean":"2024-09-13T14:20:18.598017+0000","last_became_active":"2024-09-13T14:19:25.877283+0000","last_became_peered":"2024-09-13T14:19:25.877283+0000","last_unstale":"2024-09-13T14:20:18.598017+0000","last_undegraded":"2024-09-13T14:20:18.598017+0000","last_fullsized":"2024-09-13T14:20:18.598017+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:43:50.415006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.598098+0000","last_change":"2024-09-13T14:19:26.890744+0000","last_active":"2024-09-13T14:20:18.598098+0000","last_peered":"2024-09-13T14:20:18.598098+0000","last_clean":"2024-09-13T14:20:18.598098+0000","last_became_active":"2024-09-13T14:19:26.890495+0000","last_became_peered":"2024-09-13T14:19:26.890495+0000","last_unstale":"2024-09-13T14:20:18.598098+0000","last_undegraded":"2024-09-13T14:20:18.598098+0000","last_fullsized":"2024-09-13T14:20:18.598098+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:41:38.787130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.572594+0000","last_change":"2024-09-13T14:19:25.873088+0000","last_active":"2024-09-13T14:20:18.572594+0000","last_peered":"2024-09-13T14:20:18.572594+0000","last_clean":"2024-09-13T14:20:18.572594+0000","last_became_active":"2024-09-13T14:19:25.872781+0000","last_became_peered":"2024-09-13T14:19:25.872781+0000","last_unstale":"2024-09-13T14:20:18.572594+0000","last_undegraded":"2024-09-13T14:20:18.572594+0000","last_fullsized":"2024-09-13T14:20:18.572594+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:05.607382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.445817+0000","last_change":"2024-09-13T14:19:26.890640+0000","last_active":"2024-09-13T14:20:18.445817+0000","last_peered":"2024-09-13T14:20:18.445817+0000","last_clean":"2024-09-13T14:20:18.445817+0000","last_became_active":"2024-09-13T14:19:26.890377+0000","last_became_peered":"2024-09-13T14:19:26.890377+0000","last_unstale":"2024-09-13T14:20:18.445817+0000","last_undegraded":"2024-09-13T14:20:18.445817+0000","last_fullsized":"2024-09-13T14:20:18.445817+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:06:17.733701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":22,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.443550+0000","last_change":"2024-09-13T14:19:25.869377+0000","last_active":"2024-09-13T14:20:18.443550+0000","last_peered":"2024-09-13T14:20:18.443550+0000","last_clean":"2024-09-13T14:20:18.443550+0000","last_became_active":"2024-09-13T14:19:25.869081+0000","last_became_peered":"2024-09-13T14:19:25.869081+0000","last_unstale":"2024-09-13T14:20:18.443550+0000","last_undegraded":"2024-09-13T14:20:18.443550+0000","last_fullsized":"2024-09-13T14:20:18.443550+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:33:12.833243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.572940+0000","last_change":"2024-09-13T14:19:25.872435+0000","last_active":"2024-09-13T14:20:18.572940+0000","last_peered":"2024-09-13T14:20:18.572940+0000","last_clean":"2024-09-13T14:20:18.572940+0000","last_became_active":"2024-09-13T14:19:25.872148+0000","last_became_peered":"2024-09-13T14:19:25.872148+0000","last_unstale":"2024-09-13T14:20:18.572940+0000","last_undegraded":"2024-09-13T14:20:18.572940+0000","last_fullsized":"2024-09-13T14:20:18.572940+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:29.425371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.442840+0000","last_change":"2024-09-13T14:19:26.887839+0000","last_active":"2024-09-13T14:20:18.442840+0000","last_peered":"2024-09-13T14:20:18.442840+0000","last_clean":"2024-09-13T14:20:18.442840+0000","last_became_active":"2024-09-13T14:19:26.887498+0000","last_became_peered":"2024-09-13T14:19:26.887498+0000","last_unstale":"2024-09-13T14:20:18.442840+0000","last_undegraded":"2024-09-13T14:20:18.442840+0000","last_fullsized":"2024-09-13T14:20:18.442840+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:24.475263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":16,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.572936+0000","last_change":"2024-09-13T14:19:25.874105+0000","last_active":"2024-09-13T14:20:18.572936+0000","last_peered":"2024-09-13T14:20:18.572936+0000","last_clean":"2024-09-13T14:20:18.572936+0000","last_became_active":"2024-09-13T14:19:25.873714+0000","last_became_peered":"2024-09-13T14:19:25.873714+0000","last_unstale":"2024-09-13T14:20:18.572936+0000","last_undegraded":"2024-09-13T14:20:18.572936+0000","last_fullsized":"2024-09-13T14:20:18.572936+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:28:39.623535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.440545+0000","last_change":"2024-09-13T14:19:25.993445+0000","last_active":"2024-09-13T14:20:18.440545+0000","last_peered":"2024-09-13T14:20:18.440545+0000","last_clean":"2024-09-13T14:20:18.440545+0000","last_became_active":"2024-09-13T14:19:25.993153+0000","last_became_peered":"2024-09-13T14:19:25.993153+0000","last_unstale":"2024-09-13T14:20:18.440545+0000","last_undegraded":"2024-09-13T14:20:18.440545+0000","last_fullsized":"2024-09-13T14:20:18.440545+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:46:47.959833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.572915+0000","last_change":"2024-09-13T14:19:25.871006+0000","last_active":"2024-09-13T14:20:18.572915+0000","last_peered":"2024-09-13T14:20:18.572915+0000","last_clean":"2024-09-13T14:20:18.572915+0000","last_became_active":"2024-09-13T14:19:25.870713+0000","last_became_peered":"2024-09-13T14:19:25.870713+0000","last_unstale":"2024-09-13T14:20:18.572915+0000","last_undegraded":"2024-09-13T14:20:18.572915+0000","last_fullsized":"2024-09-13T14:20:18.572915+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:34.372124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.442839+0000","last_change":"2024-09-13T14:19:26.887804+0000","last_active":"2024-09-13T14:20:18.442839+0000","last_peered":"2024-09-13T14:20:18.442839+0000","last_clean":"2024-09-13T14:20:18.442839+0000","last_became_active":"2024-09-13T14:19:26.887521+0000","last_became_peered":"2024-09-13T14:19:26.887521+0000","last_unstale":"2024-09-13T14:20:18.442839+0000","last_undegraded":"2024-09-13T14:20:18.442839+0000","last_fullsized":"2024-09-13T14:20:18.442839+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:19:37.041969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.443219+0000","last_change":"2024-09-13T14:19:25.874666+0000","last_active":"2024-09-13T14:20:18.443219+0000","last_peered":"2024-09-13T14:20:18.443219+0000","last_clean":"2024-09-13T14:20:18.443219+0000","last_became_active":"2024-09-13T14:19:25.874381+0000","last_became_peered":"2024-09-13T14:19:25.874381+0000","last_unstale":"2024-09-13T14:20:18.443219+0000","last_undegraded":"2024-09-13T14:20:18.443219+0000","last_fullsized":"2024-09-13T14:20:18.443219+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:26.056377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.597954+0000","last_change":"2024-09-13T14:19:27.378254+0000","last_active":"2024-09-13T14:20:18.597954+0000","last_peered":"2024-09-13T14:20:18.597954+0000","last_clean":"2024-09-13T14:20:18.597954+0000","last_became_active":"2024-09-13T14:19:27.377992+0000","last_became_peered":"2024-09-13T14:19:27.377992+0000","last_unstale":"2024-09-13T14:20:18.597954+0000","last_undegraded":"2024-09-13T14:20:18.597954+0000","last_fullsized":"2024-09-13T14:20:18.597954+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:38.328601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":16,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.447695+0000","last_change":"2024-09-13T14:19:25.880464+0000","last_active":"2024-09-13T14:20:18.447695+0000","last_peered":"2024-09-13T14:20:18.447695+0000","last_clean":"2024-09-13T14:20:18.447695+0000","last_became_active":"2024-09-13T14:19:25.880156+0000","last_became_peered":"2024-09-13T14:19:25.880156+0000","last_unstale":"2024-09-13T14:20:18.447695+0000","last_undegraded":"2024-09-13T14:20:18.447695+0000","last_fullsized":"2024-09-13T14:20:18.447695+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:14:52.189607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.440737+0000","last_change":"2024-09-13T14:19:26.132082+0000","last_active":"2024-09-13T14:20:18.440737+0000","last_peered":"2024-09-13T14:20:18.440737+0000","last_clean":"2024-09-13T14:20:18.440737+0000","last_became_active":"2024-09-13T14:19:26.131076+0000","last_became_peered":"2024-09-13T14:19:26.131076+0000","last_unstale":"2024-09-13T14:20:18.440737+0000","last_undegraded":"2024-09-13T14:20:18.440737+0000","last_fullsized":"2024-09-13T14:20:18.440737+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:37:58.988661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.443161+0000","last_change":"2024-09-13T14:19:25.872515+0000","last_active":"2024-09-13T14:20:18.443161+0000","last_peered":"2024-09-13T14:20:18.443161+0000","last_clean":"2024-09-13T14:20:18.443161+0000","last_became_active":"2024-09-13T14:19:25.872008+0000","last_became_peered":"2024-09-13T14:19:25.872008+0000","last_unstale":"2024-09-13T14:20:18.443161+0000","last_undegraded":"2024-09-13T14:20:18.443161+0000","last_fullsized":"2024-09-13T14:20:18.443161+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:08:37.914110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.441177+0000","last_change":"2024-09-13T14:19:26.209809+0000","last_active":"2024-09-13T14:20:18.441177+0000","last_peered":"2024-09-13T14:20:18.441177+0000","last_clean":"2024-09-13T14:20:18.441177+0000","last_became_active":"2024-09-13T14:19:26.209545+0000","last_became_peered":"2024-09-13T14:19:26.209545+0000","last_unstale":"2024-09-13T14:20:18.441177+0000","last_undegraded":"2024-09-13T14:20:18.441177+0000","last_fullsized":"2024-09-13T14:20:18.441177+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:50:18.891602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.440519+0000","last_change":"2024-09-13T14:19:25.867776+0000","last_active":"2024-09-13T14:20:18.440519+0000","last_peered":"2024-09-13T14:20:18.440519+0000","last_clean":"2024-09-13T14:20:18.440519+0000","last_became_active":"2024-09-13T14:19:25.867475+0000","last_became_peered":"2024-09-13T14:19:25.867475+0000","last_unstale":"2024-09-13T14:20:18.440519+0000","last_undegraded":"2024-09-13T14:20:18.440519+0000","last_fullsized":"2024-09-13T14:20:18.440519+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:58:25.489922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.441023+0000","last_change":"2024-09-13T14:19:26.131605+0000","last_active":"2024-09-13T14:20:18.441023+0000","last_peered":"2024-09-13T14:20:18.441023+0000","last_clean":"2024-09-13T14:20:18.441023+0000","last_became_active":"2024-09-13T14:19:26.130954+0000","last_became_peered":"2024-09-13T14:19:26.130954+0000","last_unstale":"2024-09-13T14:20:18.441023+0000","last_undegraded":"2024-09-13T14:20:18.441023+0000","last_fullsized":"2024-09-13T14:20:18.441023+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:46:20.372750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.447029+0000","last_change":"2024-09-13T14:19:25.880337+0000","last_active":"2024-09-13T14:20:18.447029+0000","last_peered":"2024-09-13T14:20:18.447029+0000","last_clean":"2024-09-13T14:20:18.447029+0000","last_became_active":"2024-09-13T14:19:25.876971+0000","last_became_peered":"2024-09-13T14:19:25.876971+0000","last_unstale":"2024-09-13T14:20:18.447029+0000","last_undegraded":"2024-09-13T14:20:18.447029+0000","last_fullsized":"2024-09-13T14:20:18.447029+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:37:43.921921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.449301+0000","last_change":"2024-09-13T14:19:25.993379+0000","last_active":"2024-09-13T14:20:18.449301+0000","last_peered":"2024-09-13T14:20:18.449301+0000","last_clean":"2024-09-13T14:20:18.449301+0000","last_became_active":"2024-09-13T14:19:25.992968+0000","last_became_peered":"2024-09-13T14:19:25.992968+0000","last_unstale":"2024-09-13T14:20:18.449301+0000","last_undegraded":"2024-09-13T14:20:18.449301+0000","last_fullsized":"2024-09-13T14:20:18.449301+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:08.589526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"57'79","reported_seq":55,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.572653+0000","last_change":"2024-09-13T14:17:42.845082+0000","last_active":"2024-09-13T14:20:18.572653+0000","last_peered":"2024-09-13T14:20:18.572653+0000","last_clean":"2024-09-13T14:20:18.572653+0000","last_became_active":"2024-09-13T14:17:42.839116+0000","last_became_peered":"2024-09-13T14:17:42.839116+0000","last_unstale":"2024-09-13T14:20:18.572653+0000","last_undegraded":"2024-09-13T14:20:18.572653+0000","last_fullsized":"2024-09-13T14:20:18.572653+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:15:42.069756+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:58.144190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":27,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.443616+0000","last_change":"2024-09-13T14:18:14.644401+0000","last_active":"2024-09-13T14:20:18.443616+0000","last_peered":"2024-09-13T14:20:18.443616+0000","last_clean":"2024-09-13T14:20:18.443616+0000","last_became_active":"2024-09-13T14:18:10.616981+0000","last_became_peered":"2024-09-13T14:18:10.616981+0000","last_unstale":"2024-09-13T14:20:18.443616+0000","last_undegraded":"2024-09-13T14:20:18.443616+0000","last_fullsized":"2024-09-13T14:20:18.443616+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:00:32.657637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034405999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.445092+0000","last_change":"2024-09-13T14:19:25.873744+0000","last_active":"2024-09-13T14:20:18.445092+0000","last_peered":"2024-09-13T14:20:18.445092+0000","last_clean":"2024-09-13T14:20:18.445092+0000","last_became_active":"2024-09-13T14:19:25.873502+0000","last_became_peered":"2024-09-13T14:19:25.873502+0000","last_unstale":"2024-09-13T14:20:18.445092+0000","last_undegraded":"2024-09-13T14:20:18.445092+0000","last_fullsized":"2024-09-13T14:20:18.445092+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:21:52.217779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.443416+0000","last_change":"2024-09-13T14:19:27.379312+0000","last_active":"2024-09-13T14:20:18.443416+0000","last_peered":"2024-09-13T14:20:18.443416+0000","last_clean":"2024-09-13T14:20:18.443416+0000","last_became_active":"2024-09-13T14:19:27.378947+0000","last_became_peered":"2024-09-13T14:19:27.378947+0000","last_unstale":"2024-09-13T14:20:18.443416+0000","last_undegraded":"2024-09-13T14:20:18.443416+0000","last_fullsized":"2024-09-13T14:20:18.443416+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:54:20.706578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":26,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.572547+0000","last_change":"2024-09-13T14:18:14.638950+0000","last_active":"2024-09-13T14:20:18.572547+0000","last_peered":"2024-09-13T14:20:18.572547+0000","last_clean":"2024-09-13T14:20:18.572547+0000","last_became_active":"2024-09-13T14:18:10.718659+0000","last_became_peered":"2024-09-13T14:18:10.718659+0000","last_unstale":"2024-09-13T14:20:18.572547+0000","last_undegraded":"2024-09-13T14:20:18.572547+0000","last_fullsized":"2024-09-13T14:20:18.572547+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:38:08.688330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.597875+0000","last_change":"2024-09-13T14:19:25.877913+0000","last_active":"2024-09-13T14:20:18.597875+0000","last_peered":"2024-09-13T14:20:18.597875+0000","last_clean":"2024-09-13T14:20:18.597875+0000","last_became_active":"2024-09-13T14:19:25.877331+0000","last_became_peered":"2024-09-13T14:19:25.877331+0000","last_unstale":"2024-09-13T14:20:18.597875+0000","last_undegraded":"2024-09-13T14:20:18.597875+0000","last_fullsized":"2024-09-13T14:20:18.597875+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:38:21.235489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.598071+0000","last_change":"2024-09-13T14:19:26.889244+0000","last_active":"2024-09-13T14:20:18.598071+0000","last_peered":"2024-09-13T14:20:18.598071+0000","last_clean":"2024-09-13T14:20:18.598071+0000","last_became_active":"2024-09-13T14:19:26.888994+0000","last_became_peered":"2024-09-13T14:19:26.888994+0000","last_unstale":"2024-09-13T14:20:18.598071+0000","last_undegraded":"2024-09-13T14:20:18.598071+0000","last_fullsized":"2024-09-13T14:20:18.598071+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:42.933815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.444685+0000","last_change":"2024-09-13T14:18:14.641344+0000","last_active":"2024-09-13T14:20:18.444685+0000","last_peered":"2024-09-13T14:20:18.444685+0000","last_clean":"2024-09-13T14:20:18.444685+0000","last_became_active":"2024-09-13T14:18:10.766103+0000","last_became_peered":"2024-09-13T14:18:10.766103+0000","last_unstale":"2024-09-13T14:20:18.444685+0000","last_undegraded":"2024-09-13T14:20:18.444685+0000","last_fullsized":"2024-09-13T14:20:18.444685+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:03.456609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.449273+0000","last_change":"2024-09-13T14:19:25.867193+0000","last_active":"2024-09-13T14:20:18.449273+0000","last_peered":"2024-09-13T14:20:18.449273+0000","last_clean":"2024-09-13T14:20:18.449273+0000","last_became_active":"2024-09-13T14:19:25.866906+0000","last_became_peered":"2024-09-13T14:19:25.866906+0000","last_unstale":"2024-09-13T14:20:18.449273+0000","last_undegraded":"2024-09-13T14:20:18.449273+0000","last_fullsized":"2024-09-13T14:20:18.449273+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:24.553692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.449372+0000","last_change":"2024-09-13T14:19:25.884601+0000","last_active":"2024-09-13T14:20:18.449372+0000","last_peered":"2024-09-13T14:20:18.449372+0000","last_clean":"2024-09-13T14:20:18.449372+0000","last_became_active":"2024-09-13T14:19:25.884298+0000","last_became_peered":"2024-09-13T14:19:25.884298+0000","last_unstale":"2024-09-13T14:20:18.449372+0000","last_undegraded":"2024-09-13T14:20:18.449372+0000","last_fullsized":"2024-09-13T14:20:18.449372+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:55:51.018565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":28,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.443101+0000","last_change":"2024-09-13T14:18:14.644865+0000","last_active":"2024-09-13T14:20:18.443101+0000","last_peered":"2024-09-13T14:20:18.443101+0000","last_clean":"2024-09-13T14:20:18.443101+0000","last_became_active":"2024-09-13T14:18:10.617018+0000","last_became_peered":"2024-09-13T14:18:10.617018+0000","last_unstale":"2024-09-13T14:20:18.443101+0000","last_undegraded":"2024-09-13T14:20:18.443101+0000","last_fullsized":"2024-09-13T14:20:18.443101+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:01:22.505985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.440860+0000","last_change":"2024-09-13T14:19:25.871375+0000","last_active":"2024-09-13T14:20:18.440860+0000","last_peered":"2024-09-13T14:20:18.440860+0000","last_clean":"2024-09-13T14:20:18.440860+0000","last_became_active":"2024-09-13T14:19:25.870907+0000","last_became_peered":"2024-09-13T14:19:25.870907+0000","last_unstale":"2024-09-13T14:20:18.440860+0000","last_undegraded":"2024-09-13T14:20:18.440860+0000","last_fullsized":"2024-09-13T14:20:18.440860+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:05:00.250780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.448823+0000","last_change":"2024-09-13T14:19:26.209757+0000","last_active":"2024-09-13T14:20:18.448823+0000","last_peered":"2024-09-13T14:20:18.448823+0000","last_clean":"2024-09-13T14:20:18.448823+0000","last_became_active":"2024-09-13T14:19:26.209453+0000","last_became_peered":"2024-09-13T14:19:26.209453+0000","last_unstale":"2024-09-13T14:20:18.448823+0000","last_undegraded":"2024-09-13T14:20:18.448823+0000","last_fullsized":"2024-09-13T14:20:18.448823+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:06.060130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.443399+0000","last_change":"2024-09-13T14:19:25.874113+0000","last_active":"2024-09-13T14:20:18.443399+0000","last_peered":"2024-09-13T14:20:18.443399+0000","last_clean":"2024-09-13T14:20:18.443399+0000","last_became_active":"2024-09-13T14:19:25.873780+0000","last_became_peered":"2024-09-13T14:19:25.873780+0000","last_unstale":"2024-09-13T14:20:18.443399+0000","last_undegraded":"2024-09-13T14:20:18.443399+0000","last_fullsized":"2024-09-13T14:20:18.443399+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:14:53.429574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.448904+0000","last_change":"2024-09-13T14:18:14.641641+0000","last_active":"2024-09-13T14:20:18.448904+0000","last_peered":"2024-09-13T14:20:18.448904+0000","last_clean":"2024-09-13T14:20:18.448904+0000","last_became_active":"2024-09-13T14:18:10.718349+0000","last_became_peered":"2024-09-13T14:18:10.718349+0000","last_unstale":"2024-09-13T14:20:18.448904+0000","last_undegraded":"2024-09-13T14:20:18.448904+0000","last_fullsized":"2024-09-13T14:20:18.448904+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:23.005181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.445163+0000","last_change":"2024-09-13T14:19:25.885166+0000","last_active":"2024-09-13T14:20:18.445163+0000","last_peered":"2024-09-13T14:20:18.445163+0000","last_clean":"2024-09-13T14:20:18.445163+0000","last_became_active":"2024-09-13T14:19:25.884888+0000","last_became_peered":"2024-09-13T14:19:25.884888+0000","last_unstale":"2024-09-13T14:20:18.445163+0000","last_undegraded":"2024-09-13T14:20:18.445163+0000","last_fullsized":"2024-09-13T14:20:18.445163+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:36:38.469866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.448881+0000","last_change":"2024-09-13T14:19:25.866481+0000","last_active":"2024-09-13T14:20:18.448881+0000","last_peered":"2024-09-13T14:20:18.448881+0000","last_clean":"2024-09-13T14:20:18.448881+0000","last_became_active":"2024-09-13T14:19:25.866198+0000","last_became_peered":"2024-09-13T14:19:25.866198+0000","last_unstale":"2024-09-13T14:20:18.448881+0000","last_undegraded":"2024-09-13T14:20:18.448881+0000","last_fullsized":"2024-09-13T14:20:18.448881+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:51.651674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":26,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.572930+0000","last_change":"2024-09-13T14:18:14.639101+0000","last_active":"2024-09-13T14:20:18.572930+0000","last_peered":"2024-09-13T14:20:18.572930+0000","last_clean":"2024-09-13T14:20:18.572930+0000","last_became_active":"2024-09-13T14:18:10.718655+0000","last_became_peered":"2024-09-13T14:18:10.718655+0000","last_unstale":"2024-09-13T14:20:18.572930+0000","last_undegraded":"2024-09-13T14:20:18.572930+0000","last_fullsized":"2024-09-13T14:20:18.572930+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:59:01.637559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.598021+0000","last_change":"2024-09-13T14:19:26.887815+0000","last_active":"2024-09-13T14:20:18.598021+0000","last_peered":"2024-09-13T14:20:18.598021+0000","last_clean":"2024-09-13T14:20:18.598021+0000","last_became_active":"2024-09-13T14:19:26.887539+0000","last_became_peered":"2024-09-13T14:19:26.887539+0000","last_unstale":"2024-09-13T14:20:18.598021+0000","last_undegraded":"2024-09-13T14:20:18.598021+0000","last_fullsized":"2024-09-13T14:20:18.598021+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:23:00.385203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":16,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.447818+0000","last_change":"2024-09-13T14:19:25.879344+0000","last_active":"2024-09-13T14:20:18.447818+0000","last_peered":"2024-09-13T14:20:18.447818+0000","last_clean":"2024-09-13T14:20:18.447818+0000","last_became_active":"2024-09-13T14:19:25.879030+0000","last_became_peered":"2024-09-13T14:19:25.879030+0000","last_unstale":"2024-09-13T14:20:18.447818+0000","last_undegraded":"2024-09-13T14:20:18.447818+0000","last_fullsized":"2024-09-13T14:20:18.447818+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:51.493477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.449609+0000","last_change":"2024-09-13T14:18:14.641734+0000","last_active":"2024-09-13T14:20:18.449609+0000","last_peered":"2024-09-13T14:20:18.449609+0000","last_clean":"2024-09-13T14:20:18.449609+0000","last_became_active":"2024-09-13T14:18:10.614303+0000","last_became_peered":"2024-09-13T14:18:10.614303+0000","last_unstale":"2024-09-13T14:20:18.449609+0000","last_undegraded":"2024-09-13T14:20:18.449609+0000","last_fullsized":"2024-09-13T14:20:18.449609+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:28:38.174245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034192599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.446878+0000","last_change":"2024-09-13T14:19:26.890828+0000","last_active":"2024-09-13T14:20:18.446878+0000","last_peered":"2024-09-13T14:20:18.446878+0000","last_clean":"2024-09-13T14:20:18.446878+0000","last_became_active":"2024-09-13T14:19:26.889977+0000","last_became_peered":"2024-09-13T14:19:26.889977+0000","last_unstale":"2024-09-13T14:20:18.446878+0000","last_undegraded":"2024-09-13T14:20:18.446878+0000","last_fullsized":"2024-09-13T14:20:18.446878+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:13:19.115622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.598137+0000","last_change":"2024-09-13T14:19:25.869612+0000","last_active":"2024-09-13T14:20:18.598137+0000","last_peered":"2024-09-13T14:20:18.598137+0000","last_clean":"2024-09-13T14:20:18.598137+0000","last_became_active":"2024-09-13T14:19:25.869220+0000","last_became_peered":"2024-09-13T14:19:25.869220+0000","last_unstale":"2024-09-13T14:20:18.598137+0000","last_undegraded":"2024-09-13T14:20:18.598137+0000","last_fullsized":"2024-09-13T14:20:18.598137+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:11:25.488245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.443228+0000","last_change":"2024-09-13T14:18:14.645599+0000","last_active":"2024-09-13T14:20:18.443228+0000","last_peered":"2024-09-13T14:20:18.443228+0000","last_clean":"2024-09-13T14:20:18.443228+0000","last_became_active":"2024-09-13T14:18:10.617927+0000","last_became_peered":"2024-09-13T14:18:10.617927+0000","last_unstale":"2024-09-13T14:20:18.443228+0000","last_undegraded":"2024-09-13T14:20:18.443228+0000","last_fullsized":"2024-09-13T14:20:18.443228+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:51:43.437167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.440426+0000","last_change":"2024-09-13T14:19:26.131609+0000","last_active":"2024-09-13T14:20:18.440426+0000","last_peered":"2024-09-13T14:20:18.440426+0000","last_clean":"2024-09-13T14:20:18.440426+0000","last_became_active":"2024-09-13T14:19:26.131029+0000","last_became_peered":"2024-09-13T14:19:26.131029+0000","last_unstale":"2024-09-13T14:20:18.440426+0000","last_undegraded":"2024-09-13T14:20:18.440426+0000","last_fullsized":"2024-09-13T14:20:18.440426+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:57.933696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.440764+0000","last_change":"2024-09-13T14:19:26.131694+0000","last_active":"2024-09-13T14:20:18.440764+0000","last_peered":"2024-09-13T14:20:18.440764+0000","last_clean":"2024-09-13T14:20:18.440764+0000","last_became_active":"2024-09-13T14:19:26.131121+0000","last_became_peered":"2024-09-13T14:19:26.131121+0000","last_unstale":"2024-09-13T14:20:18.440764+0000","last_undegraded":"2024-09-13T14:20:18.440764+0000","last_fullsized":"2024-09-13T14:20:18.440764+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:10:17.349612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"63'6","reported_seq":20,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.440805+0000","last_change":"2024-09-13T14:19:25.870605+0000","last_active":"2024-09-13T14:20:18.440805+0000","last_peered":"2024-09-13T14:20:18.440805+0000","last_clean":"2024-09-13T14:20:18.440805+0000","last_became_active":"2024-09-13T14:19:25.870282+0000","last_became_peered":"2024-09-13T14:19:25.870282+0000","last_unstale":"2024-09-13T14:20:18.440805+0000","last_undegraded":"2024-09-13T14:20:18.440805+0000","last_fullsized":"2024-09-13T14:20:18.440805+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:52.514915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1552,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.445043+0000","last_change":"2024-09-13T14:19:26.209743+0000","last_active":"2024-09-13T14:20:18.445043+0000","last_peered":"2024-09-13T14:20:18.445043+0000","last_clean":"2024-09-13T14:20:18.445043+0000","last_became_active":"2024-09-13T14:19:26.209460+0000","last_became_peered":"2024-09-13T14:19:26.209460+0000","last_unstale":"2024-09-13T14:20:18.445043+0000","last_undegraded":"2024-09-13T14:20:18.445043+0000","last_fullsized":"2024-09-13T14:20:18.445043+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:15:52.778019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":17,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.572820+0000","last_change":"2024-09-13T14:19:25.875899+0000","last_active":"2024-09-13T14:20:18.572820+0000","last_peered":"2024-09-13T14:20:18.572820+0000","last_clean":"2024-09-13T14:20:18.572820+0000","last_became_active":"2024-09-13T14:19:25.875598+0000","last_became_peered":"2024-09-13T14:19:25.875598+0000","last_unstale":"2024-09-13T14:20:18.572820+0000","last_undegraded":"2024-09-13T14:20:18.572820+0000","last_fullsized":"2024-09-13T14:20:18.572820+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:45.274956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.443437+0000","last_change":"2024-09-13T14:19:25.872219+0000","last_active":"2024-09-13T14:20:18.443437+0000","last_peered":"2024-09-13T14:20:18.443437+0000","last_clean":"2024-09-13T14:20:18.443437+0000","last_became_active":"2024-09-13T14:19:25.871936+0000","last_became_peered":"2024-09-13T14:19:25.871936+0000","last_unstale":"2024-09-13T14:20:18.443437+0000","last_undegraded":"2024-09-13T14:20:18.443437+0000","last_fullsized":"2024-09-13T14:20:18.443437+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:58.476558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.444682+0000","last_change":"2024-09-13T14:19:26.890559+0000","last_active":"2024-09-13T14:20:18.444682+0000","last_peered":"2024-09-13T14:20:18.444682+0000","last_clean":"2024-09-13T14:20:18.444682+0000","last_became_active":"2024-09-13T14:19:26.890266+0000","last_became_peered":"2024-09-13T14:19:26.890266+0000","last_unstale":"2024-09-13T14:20:18.444682+0000","last_undegraded":"2024-09-13T14:20:18.444682+0000","last_fullsized":"2024-09-13T14:20:18.444682+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:27.600428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.598211+0000","last_change":"2024-09-13T14:19:25.876770+0000","last_active":"2024-09-13T14:20:18.598211+0000","last_peered":"2024-09-13T14:20:18.598211+0000","last_clean":"2024-09-13T14:20:18.598211+0000","last_became_active":"2024-09-13T14:19:25.876534+0000","last_became_peered":"2024-09-13T14:19:25.876534+0000","last_unstale":"2024-09-13T14:20:18.598211+0000","last_undegraded":"2024-09-13T14:20:18.598211+0000","last_fullsized":"2024-09-13T14:20:18.598211+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:53:21.516150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-13T14:20:19.825757+0000","last_change":"2024-09-13T14:20:19.825757+0000","last_active":"2024-09-13T14:20:19.825757+0000","last_peered":"2024-09-13T14:20:19.825757+0000","last_clean":"2024-09-13T14:20:19.825757+0000","last_became_active":"2024-09-13T14:20:19.825414+0000","last_became_peered":"2024-09-13T14:20:19.825414+0000","last_unstale":"2024-09-13T14:20:19.825757+0000","last_undegraded":"2024-09-13T14:20:19.825757+0000","last_fullsized":"2024-09-13T14:20:19.825757+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:40:37.165447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.443129+0000","last_change":"2024-09-13T14:19:26.887783+0000","last_active":"2024-09-13T14:20:18.443129+0000","last_peered":"2024-09-13T14:20:18.443129+0000","last_clean":"2024-09-13T14:20:18.443129+0000","last_became_active":"2024-09-13T14:19:26.887509+0000","last_became_peered":"2024-09-13T14:19:26.887509+0000","last_unstale":"2024-09-13T14:20:18.443129+0000","last_undegraded":"2024-09-13T14:20:18.443129+0000","last_fullsized":"2024-09-13T14:20:18.443129+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:20.774114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.446955+0000","last_change":"2024-09-13T14:19:25.877905+0000","last_active":"2024-09-13T14:20:18.446955+0000","last_peered":"2024-09-13T14:20:18.446955+0000","last_clean":"2024-09-13T14:20:18.446955+0000","last_became_active":"2024-09-13T14:19:25.877618+0000","last_became_peered":"2024-09-13T14:19:25.877618+0000","last_unstale":"2024-09-13T14:20:18.446955+0000","last_undegraded":"2024-09-13T14:20:18.446955+0000","last_fullsized":"2024-09-13T14:20:18.446955+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:30:03.430683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.598015+0000","last_change":"2024-09-13T14:19:26.891225+0000","last_active":"2024-09-13T14:20:18.598015+0000","last_peered":"2024-09-13T14:20:18.598015+0000","last_clean":"2024-09-13T14:20:18.598015+0000","last_became_active":"2024-09-13T14:19:26.890936+0000","last_became_peered":"2024-09-13T14:19:26.890936+0000","last_unstale":"2024-09-13T14:20:18.598015+0000","last_undegraded":"2024-09-13T14:20:18.598015+0000","last_fullsized":"2024-09-13T14:20:18.598015+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:27:27.983810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.444902+0000","last_change":"2024-09-13T14:19:25.867272+0000","last_active":"2024-09-13T14:20:18.444902+0000","last_peered":"2024-09-13T14:20:18.444902+0000","last_clean":"2024-09-13T14:20:18.444902+0000","last_became_active":"2024-09-13T14:19:25.866976+0000","last_became_peered":"2024-09-13T14:19:25.866976+0000","last_unstale":"2024-09-13T14:20:18.444902+0000","last_undegraded":"2024-09-13T14:20:18.444902+0000","last_fullsized":"2024-09-13T14:20:18.444902+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:40.024931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.445614+0000","last_change":"2024-09-13T14:19:27.379386+0000","last_active":"2024-09-13T14:20:18.445614+0000","last_peered":"2024-09-13T14:20:18.445614+0000","last_clean":"2024-09-13T14:20:18.445614+0000","last_became_active":"2024-09-13T14:19:27.379036+0000","last_became_peered":"2024-09-13T14:19:27.379036+0000","last_unstale":"2024-09-13T14:20:18.445614+0000","last_undegraded":"2024-09-13T14:20:18.445614+0000","last_fullsized":"2024-09-13T14:20:18.445614+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:25:55.319899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.440532+0000","last_change":"2024-09-13T14:19:25.867061+0000","last_active":"2024-09-13T14:20:18.440532+0000","last_peered":"2024-09-13T14:20:18.440532+0000","last_clean":"2024-09-13T14:20:18.440532+0000","last_became_active":"2024-09-13T14:19:25.866766+0000","last_became_peered":"2024-09-13T14:19:25.866766+0000","last_unstale":"2024-09-13T14:20:18.440532+0000","last_undegraded":"2024-09-13T14:20:18.440532+0000","last_fullsized":"2024-09-13T14:20:18.440532+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:17:30.834847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.445118+0000","last_change":"2024-09-13T14:19:26.887515+0000","last_active":"2024-09-13T14:20:18.445118+0000","last_peered":"2024-09-13T14:20:18.445118+0000","last_clean":"2024-09-13T14:20:18.445118+0000","last_became_active":"2024-09-13T14:19:26.887247+0000","last_became_peered":"2024-09-13T14:19:26.887247+0000","last_unstale":"2024-09-13T14:20:18.445118+0000","last_undegraded":"2024-09-13T14:20:18.445118+0000","last_fullsized":"2024-09-13T14:20:18.445118+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:07:32.437260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.598076+0000","last_change":"2024-09-13T14:19:25.876086+0000","last_active":"2024-09-13T14:20:18.598076+0000","last_peered":"2024-09-13T14:20:18.598076+0000","last_clean":"2024-09-13T14:20:18.598076+0000","last_became_active":"2024-09-13T14:19:25.873007+0000","last_became_peered":"2024-09-13T14:19:25.873007+0000","last_unstale":"2024-09-13T14:20:18.598076+0000","last_undegraded":"2024-09-13T14:20:18.598076+0000","last_fullsized":"2024-09-13T14:20:18.598076+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:41:18.113589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":23,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-13T14:20:19.992899+0000","last_change":"2024-09-13T14:20:19.992899+0000","last_active":"2024-09-13T14:20:19.992899+0000","last_peered":"2024-09-13T14:20:19.992899+0000","last_clean":"2024-09-13T14:20:19.992899+0000","last_became_active":"2024-09-13T14:20:19.992590+0000","last_became_peered":"2024-09-13T14:20:19.992590+0000","last_unstale":"2024-09-13T14:20:19.992899+0000","last_undegraded":"2024-09-13T14:20:19.992899+0000","last_fullsized":"2024-09-13T14:20:19.992899+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:20.912929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":16,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.443136+0000","last_change":"2024-09-13T14:19:25.873795+0000","last_active":"2024-09-13T14:20:18.443136+0000","last_peered":"2024-09-13T14:20:18.443136+0000","last_clean":"2024-09-13T14:20:18.443136+0000","last_became_active":"2024-09-13T14:19:25.873387+0000","last_became_peered":"2024-09-13T14:19:25.873387+0000","last_unstale":"2024-09-13T14:20:18.443136+0000","last_undegraded":"2024-09-13T14:20:18.443136+0000","last_fullsized":"2024-09-13T14:20:18.443136+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:55:49.807951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.440506+0000","last_change":"2024-09-13T14:19:25.993592+0000","last_active":"2024-09-13T14:20:18.440506+0000","last_peered":"2024-09-13T14:20:18.440506+0000","last_clean":"2024-09-13T14:20:18.440506+0000","last_became_active":"2024-09-13T14:19:25.993144+0000","last_became_peered":"2024-09-13T14:19:25.993144+0000","last_unstale":"2024-09-13T14:20:18.440506+0000","last_undegraded":"2024-09-13T14:20:18.440506+0000","last_fullsized":"2024-09-13T14:20:18.440506+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:09:24.672117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":16,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.443331+0000","last_change":"2024-09-13T14:19:25.873514+0000","last_active":"2024-09-13T14:20:18.443331+0000","last_peered":"2024-09-13T14:20:18.443331+0000","last_clean":"2024-09-13T14:20:18.443331+0000","last_became_active":"2024-09-13T14:19:25.873170+0000","last_became_peered":"2024-09-13T14:19:25.873170+0000","last_unstale":"2024-09-13T14:20:18.443331+0000","last_undegraded":"2024-09-13T14:20:18.443331+0000","last_fullsized":"2024-09-13T14:20:18.443331+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:48.298530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.444864+0000","last_change":"2024-09-13T14:19:26.887677+0000","last_active":"2024-09-13T14:20:18.444864+0000","last_peered":"2024-09-13T14:20:18.444864+0000","last_clean":"2024-09-13T14:20:18.444864+0000","last_became_active":"2024-09-13T14:19:26.887372+0000","last_became_peered":"2024-09-13T14:19:26.887372+0000","last_unstale":"2024-09-13T14:20:18.444864+0000","last_undegraded":"2024-09-13T14:20:18.444864+0000","last_fullsized":"2024-09-13T14:20:18.444864+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:55:43.765983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.444710+0000","last_change":"2024-09-13T14:19:25.872630+0000","last_active":"2024-09-13T14:20:18.444710+0000","last_peered":"2024-09-13T14:20:18.444710+0000","last_clean":"2024-09-13T14:20:18.444710+0000","last_became_active":"2024-09-13T14:19:25.872252+0000","last_became_peered":"2024-09-13T14:19:25.872252+0000","last_unstale":"2024-09-13T14:20:18.444710+0000","last_undegraded":"2024-09-13T14:20:18.444710+0000","last_fullsized":"2024-09-13T14:20:18.444710+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:03:03.501514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":23,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-13T14:20:19.825870+0000","last_change":"2024-09-13T14:20:19.825870+0000","last_active":"2024-09-13T14:20:19.825870+0000","last_peered":"2024-09-13T14:20:19.825870+0000","last_clean":"2024-09-13T14:20:19.825870+0000","last_became_active":"2024-09-13T14:20:19.825471+0000","last_became_peered":"2024-09-13T14:20:19.825471+0000","last_unstale":"2024-09-13T14:20:19.825870+0000","last_undegraded":"2024-09-13T14:20:19.825870+0000","last_fullsized":"2024-09-13T14:20:19.825870+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:40:23.580724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":18,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T14:20:18.598198+0000","last_change":"2024-09-13T14:19:25.869427+0000","last_active":"2024-09-13T14:20:18.598198+0000","last_peered":"2024-09-13T14:20:18.598198+0000","last_clean":"2024-09-13T14:20:18.598198+0000","last_became_active":"2024-09-13T14:19:25.869094+0000","last_became_peered":"2024-09-13T14:19:25.869094+0000","last_unstale":"2024-09-13T14:20:18.598198+0000","last_undegraded":"2024-09-13T14:20:18.598198+0000","last_fullsized":"2024-09-13T14:20:18.598198+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:16.636504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2338,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":7014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430245,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298616,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446536,"statfs":{"total":95995035648,"available":95689252864,"internally_reserved":0,"allocated":1286144,"data_stored":884657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51700000000000002,"15min":0.51800000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.52600000000000002,"15min":0.53400000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.34699999999999998}]},{"osd":1,"last update":"Fri Sep 13 14:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.53400000000000003,"15min":0.54200000000000004},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.501,"5min":0.51500000000000001,"15min":0.51700000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.58999999999999997}]},{"osd":2,"last update":"Fri Sep 13 14:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.51200000000000001,"15min":0.51300000000000001},"min":{"1min":0.40000000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.64400000000000002,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.53300000000000003,"15min":0.53900000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.36099999999999999}]},{"osd":3,"last update":"Fri Sep 13 14:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.51200000000000001,"15min":0.51400000000000001},"min":{"1min":0.39100000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.499},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.42699999999999999}]},{"osd":4,"last update":"Fri Sep 13 14:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.44,"5min":0.46400000000000002,"15min":0.46800000000000003},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.57399999999999995}]},{"osd":5,"last update":"Fri Sep 13 14:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.45000000000000001,"15min":0.44600000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47899999999999998,"15min":0.47999999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.69499999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.56100000000000005}]},{"osd":6,"last update":"Fri Sep 13 14:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.49099999999999999,"15min":0.49199999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.50900000000000001,"15min":0.51100000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.65200000000000002,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.60799999999999998}]}]},{"osd":6,"up_from":42,"seq":180388626474,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298596,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446556,"statfs":{"total":95995035648,"available":95689273344,"internally_reserved":0,"allocated":1269760,"data_stored":883971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:20:26 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51200000000000001,"15min":0.51000000000000001},"min":{"1min":0.38,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.64000000000000001,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.48899999999999999,"15min":0.47599999999999998},"min":{"1min":0.377,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.501}]},{"osd":1,"last update":"Fri Sep 13 14:20:26 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.50900000000000001,"15min":0.496},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.375,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.40500000000000003}]},{"osd":2,"last update":"Fri Sep 13 14:20:26 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.48799999999999999,"15min":0.48299999999999998},"min":{"1min":0.36299999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.63500000000000001,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.623},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.52100000000000002,"15min":0.51700000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.68999999999999995,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.42599999999999999}]},{"osd":3,"last update":"Fri Sep 13 14:20:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.496,"15min":0.48799999999999999},"min":{"1min":0.40600000000000003,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.70399999999999996,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.53200000000000003,"15min":0.52600000000000002},"min":{"1min":0.432,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.75900000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.57399999999999995}]},{"osd":4,"last update":"Fri Sep 13 14:20:26 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.40799999999999997,"15min":0.379},"min":{"1min":0.32300000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.441,"15min":0.41699999999999998},"min":{"1min":0.33000000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.66400000000000003,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.60599999999999998}]},{"osd":5,"last update":"Fri Sep 13 14:20:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.44,"15min":0.41799999999999998},"min":{"1min":0.33700000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.72199999999999998,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.46000000000000002,"15min":0.432},"min":{"1min":0.33200000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.71399999999999997,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.56299999999999994}]},{"osd":7,"last update":"Fri Sep 13 14:19:44 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.46300000000000002,"15min":0.45700000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.50600000000000001,"15min":0.498},"min":{"1min":0.34499999999999997,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.63600000000000001}]}]},{"osd":5,"up_from":36,"seq":154618822703,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298012,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447140,"statfs":{"total":95995035648,"available":95689871360,"internally_reserved":0,"allocated":667648,"data_stored":292621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.52100000000000002,"15min":0.52500000000000002},"min":{"1min":0.36499999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.70299999999999996,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.498,"15min":0.496},"min":{"1min":0.30199999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.64900000000000002,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.45900000000000002}]},{"osd":1,"last update":"Fri Sep 13 14:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.51400000000000001,"15min":0.51800000000000002},"min":{"1min":0.36799999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.58099999999999996,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.51400000000000001,"15min":0.51500000000000001},"min":{"1min":0.33300000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.71599999999999997,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.497}]},{"osd":2,"last update":"Fri Sep 13 14:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.51700000000000002,"15min":0.52300000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.67300000000000004,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.34200000000000003,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.70599999999999996,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.55000000000000004}]},{"osd":3,"last update":"Fri Sep 13 14:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.50900000000000001,"15min":0.50600000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.70999999999999996,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.437},{"interface":"front","average":{"1min":0.505,"5min":0.53200000000000003,"15min":0.53700000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.68300000000000005,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.53500000000000003}]},{"osd":4,"last update":"Fri Sep 13 14:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.38900000000000001,"15min":0.374},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.38},{"interface":"front","average":{"1min":0.436,"5min":0.39700000000000002,"15min":0.378},"min":{"1min":0.30199999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.63600000000000001,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.34699999999999998}]},{"osd":6,"last update":"Fri Sep 13 14:20:28 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.41199999999999998,"15min":0.39200000000000002},"min":{"1min":0.28000000000000003,"5min":0.221,"15min":0.221},"max":{"1min":0.67300000000000004,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.438,"15min":0.41799999999999998},"min":{"1min":0.32800000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.68400000000000005,"5min":0.75,"15min":0.75},"last":0.41899999999999998}]},{"osd":7,"last update":"Fri Sep 13 14:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.46800000000000003,"15min":0.46400000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.74099999999999999,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.50900000000000001,"15min":0.502},"min":{"1min":0.312,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.51200000000000001}]}]},{"osd":4,"up_from":30,"seq":128849018931,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298028,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447124,"statfs":{"total":95995035648,"available":95689854976,"internally_reserved":0,"allocated":684032,"data_stored":293975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.52700000000000002,"15min":0.53900000000000003},"min":{"1min":0.33700000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.68300000000000005,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.55100000000000005,"15min":0.55900000000000005},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.79300000000000004,"5min":0.88,"15min":0.88},"last":0.53200000000000003}]},{"osd":1,"last update":"Fri Sep 13 14:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53500000000000003,"15min":0.54100000000000004},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.72899999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.55000000000000004,"15min":0.56000000000000005},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.74099999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.64400000000000002}]},{"osd":2,"last update":"Fri Sep 13 14:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.54500000000000004,"15min":0.55300000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.71899999999999997,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.624},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.52500000000000002,"15min":0.53300000000000003},"min":{"1min":0.36399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.74099999999999999,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.58299999999999996}]},{"osd":3,"last update":"Fri Sep 13 14:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.53200000000000003,"15min":0.53800000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.71699999999999997,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53100000000000003,"15min":0.53800000000000003},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.65700000000000003}]},{"osd":5,"last update":"Fri Sep 13 14:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.39400000000000002,"15min":0.38},"min":{"1min":0.28199999999999997,"5min":0.254,"15min":0.254},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.436},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.443,"15min":0.42099999999999999},"min":{"1min":0.33600000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.45500000000000002}]},{"osd":6,"last update":"Fri Sep 13 14:20:22 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.44400000000000001,"15min":0.42499999999999999},"min":{"1min":0.33000000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.378},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.42199999999999999,"15min":0.40999999999999998},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.496}]},{"osd":7,"last update":"Fri Sep 13 14:19:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.46200000000000002,"15min":0.45100000000000001},"min":{"1min":0.32300000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.45800000000000002,"15min":0.44800000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.54400000000000004}]}]},{"osd":3,"up_from":25,"seq":107374182457,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298004,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447148,"statfs":{"total":95995035648,"available":95689879552,"internally_reserved":0,"allocated":663552,"data_stored":292367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.376,"15min":0.40200000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.44600000000000001,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.377,"15min":0.39500000000000002},"min":{"1min":0.26300000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.46899999999999997,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.32100000000000001}]},{"osd":1,"last update":"Fri Sep 13 14:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.36899999999999999,"15min":0.38800000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.47299999999999998,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.41399999999999998,"15min":0.44},"min":{"1min":0.23799999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.52700000000000002,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.309}]},{"osd":2,"last update":"Fri Sep 13 14:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.42899999999999999,"15min":0.45200000000000001},"min":{"1min":0.28000000000000003,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.48699999999999999,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.39700000000000002,"15min":0.42399999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.496,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.29699999999999999}]},{"osd":4,"last update":"Fri Sep 13 14:19:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.52800000000000002,"15min":0.52700000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.42899999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.47799999999999998}]},{"osd":5,"last update":"Fri Sep 13 14:20:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54300000000000004,"15min":0.54600000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55000000000000004,"15min":0.55400000000000005},"min":{"1min":0.436,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.75600000000000001,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.505}]},{"osd":6,"last update":"Fri Sep 13 14:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.55300000000000005,"15min":0.55000000000000004},"min":{"1min":0.39500000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.745,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.434},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.56899999999999995,"15min":0.57199999999999995},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.45700000000000002}]},{"osd":7,"last update":"Fri Sep 13 14:19:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.55600000000000005,"15min":0.55100000000000005},"min":{"1min":0.44900000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.53500000000000003,"15min":0.52800000000000002},"min":{"1min":0.441,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.53500000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298008,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447144,"statfs":{"total":95995035648,"available":95689875456,"internally_reserved":0,"allocated":663552,"data_stored":292637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:19:53 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.38800000000000001,"15min":0.38200000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.495,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.375,"15min":0.36199999999999999},"min":{"1min":0.22800000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.51400000000000001,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.42799999999999999}]},{"osd":1,"last update":"Fri Sep 13 14:19:53 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.39300000000000002,"15min":0.38400000000000001},"min":{"1min":0.25800000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.441},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.40200000000000002,"15min":0.39200000000000002},"min":{"1min":0.26400000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.56799999999999995,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.41099999999999998}]},{"osd":3,"last update":"Fri Sep 13 14:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.438,"15min":0.45200000000000001},"min":{"1min":0.25900000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.54800000000000004,"5min":0.622,"15min":0.622},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.436,"15min":0.45000000000000001},"min":{"1min":0.28399999999999997,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.54300000000000004,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.45700000000000002}]},{"osd":4,"last update":"Fri Sep 13 14:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55700000000000005,"15min":0.56200000000000006},"min":{"1min":0.38600000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.73499999999999999,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.66300000000000003}]},{"osd":5,"last update":"Fri Sep 13 14:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.60899999999999999,"15min":0.626},"min":{"1min":0.443,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.75800000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58699999999999997,"15min":0.59499999999999997},"min":{"1min":0.45700000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71099999999999997,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.59799999999999998}]},{"osd":6,"last update":"Fri Sep 13 14:20:25 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59199999999999997,"15min":0.59399999999999997},"min":{"1min":0.43099999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74399999999999999,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.61799999999999999,"15min":0.63600000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.77500000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.70299999999999996}]},{"osd":7,"last update":"Fri Sep 13 14:19:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.59399999999999997,"15min":0.59699999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.78300000000000003,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.73099999999999998,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.57499999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298008,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447144,"statfs":{"total":95995035648,"available":95689875456,"internally_reserved":0,"allocated":663552,"data_stored":292225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:20:22 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.35199999999999998,"15min":0.315},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.47899999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.16900000000000001},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.36899999999999999,"15min":0.32700000000000001},"min":{"1min":0.24399999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.42999999999999999,"5min":0.622,"15min":0.622},"last":0.24099999999999999}]},{"osd":2,"last update":"Fri Sep 13 14:19:50 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.39200000000000002,"15min":0.38900000000000001},"min":{"1min":0.23300000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.56999999999999995,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.187},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.379,"15min":0.36499999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.29699999999999999}]},{"osd":3,"last update":"Fri Sep 13 14:20:14 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.39500000000000002,"15min":0.40500000000000003},"min":{"1min":0.26000000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.441,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.254},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.433,"15min":0.44700000000000001},"min":{"1min":0.28599999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.47999999999999998,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.26600000000000001}]},{"osd":4,"last update":"Fri Sep 13 14:19:33 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55400000000000005,"15min":0.56200000000000006},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.47799999999999998}]},{"osd":5,"last update":"Fri Sep 13 14:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.56100000000000005,"15min":0.56499999999999995},"min":{"1min":0.373,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.58499999999999996,"15min":0.59899999999999998},"min":{"1min":0.36299999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.46500000000000002}]},{"osd":6,"last update":"Fri Sep 13 14:20:25 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.56100000000000005,"15min":0.56699999999999995},"min":{"1min":0.40500000000000003,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.72999999999999998,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.439,"5min":0.376,"15min":0.376},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.54900000000000004}]},{"osd":7,"last update":"Fri Sep 13 14:19:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.57799999999999996,"15min":0.58699999999999997},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.746,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.59699999999999998,"15min":0.60499999999999998},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.85099999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.52500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298592,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446560,"statfs":{"total":95995035648,"available":95689277440,"internally_reserved":0,"allocated":1261568,"data_stored":883166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 14:20:21 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.33600000000000002,"15min":0.31},"min":{"1min":0.20200000000000001,"5min":0.193,"15min":0.193},"max":{"1min":0.55400000000000005,"5min":1.218,"15min":1.218},"last":0.314},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.36799999999999999,"15min":0.33600000000000002},"min":{"1min":0.23499999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.60699999999999998,"5min":1.284,"15min":1.284},"last":0.36199999999999999}]},{"osd":2,"last update":"Fri Sep 13 14:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.52900000000000003,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.46200000000000002,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.34499999999999997}]},{"osd":3,"last update":"Fri Sep 13 14:20:18 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.41999999999999998,"15min":0.44},"min":{"1min":0.27100000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.52700000000000002,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.436},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.379,"15min":0.38800000000000001},"min":{"1min":0.26200000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.56699999999999995,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.28899999999999998}]},{"osd":4,"last update":"Fri Sep 13 14:19:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.50600000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.71399999999999997,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.503,"15min":0.499},"min":{"1min":0.36599999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.40100000000000002}]},{"osd":5,"last update":"Fri Sep 13 14:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.51900000000000002,"15min":0.51100000000000001},"min":{"1min":0.41899999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54700000000000004,"15min":0.54500000000000004},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.73599999999999999,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.57499999999999996}]},{"osd":6,"last update":"Fri Sep 13 14:20:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.56200000000000006,"15min":0.56899999999999995},"min":{"1min":0.42599999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.71899999999999997,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.51100000000000001}]},{"osd":7,"last update":"Fri Sep 13 14:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55700000000000005,"15min":0.56100000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73299999999999998,"5min":1.141,"15min":1.141},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.58499999999999996,"15min":0.59199999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.69099999999999995,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.54800000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:20:32.094 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-13T14:20:32.094 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-13T14:20:32.094 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-13T14:20:32.094 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-13T14:20:32.688 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:32.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:32 smithi184 ceph-mon[33394]: from='client.24680 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:20:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:32 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:32 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:32.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:32 smithi151 ceph-mon[41387]: from='client.24680 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:20:32.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:32 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:32.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:32 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:32.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:32 smithi151 ceph-mon[29807]: from='client.24680 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:20:32.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:32 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:32.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:32 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:33.232 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:20:32 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:20:32] "GET /metrics HTTP/1.1" 200 239775 "" "Prometheus/2.43.0" 2024-09-13T14:20:33.774 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:33 smithi151 ceph-mon[29807]: pgmap v78: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:33.774 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:33 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:20:33.775 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:33 smithi151 ceph-mon[41387]: pgmap v78: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:33.775 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:33 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:20:33.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:33 smithi184 ceph-mon[33394]: pgmap v78: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:33.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:33 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:20:35.149 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:35 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:35.149 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:35 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:35.149 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:35 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:20:35.149 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:35 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:20:35.149 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:35 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:35.149 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:35 smithi151 ceph-mon[29807]: pgmap v79: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:35.149 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:35 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:35.149 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:35 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:35.150 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:35 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:20:35.150 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:35 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:20:35.150 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:35 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:35.150 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:35 smithi151 ceph-mon[41387]: pgmap v79: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:35.318 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:20:35.318 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T14:20:35.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:35 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:35.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:35 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:35.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:35 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:20:35.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:35 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:20:35.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:35 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:35.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:35 smithi184 ceph-mon[33394]: pgmap v79: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:35.964 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-13T14:20:35.964 INFO:teuthology.run_tasks:Running task print... 2024-09-13T14:20:35.976 INFO:teuthology.task.print:**** upgrading first half of cluster, with stress **** 2024-09-13T14:20:35.977 INFO:teuthology.run_tasks:Running task parallel... 2024-09-13T14:20:36.044 INFO:teuthology.task.parallel:starting parallel... 2024-09-13T14:20:36.044 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-13T14:20:36.045 INFO:teuthology.task.sequential:In sequential, running task full_sequential... 2024-09-13T14:20:36.045 INFO:teuthology.task.full_sequential:In full_sequential, running task rados... 2024-09-13T14:20:36.049 INFO:tasks.rados:Beginning rados... 2024-09-13T14:20:36.049 INFO:tasks.rados:config is {'clients': ['client.0'], 'objects': 50, 'op_weights': {'delete': 50, 'read': 100, 'rollback': 50, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000, 'write_append_excl': False} 2024-09-13T14:20:36.049 INFO:tasks.rados:overrides is {'admin_socket': {'branch': 'wip-aclamk-cbt-combined'}, 'ceph': {'conf': {'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'create_rbd_pool': True, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'OSD_DOWN', 'mons down', 'mon down', 'MON_DOWN', 'out of quorum', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSDMAP_FLAGS', 'PG_AVAILABILITY'], 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-aclamk-cbt-combined', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}} 2024-09-13T14:20:36.049 INFO:tasks.rados:config is {'clients': ['client.0'], 'objects': 50, 'op_weights': {'delete': 50, 'read': 100, 'rollback': 50, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000, 'write_append_excl': False} 2024-09-13T14:20:36.049 INFO:tasks.rados:joining rados 2024-09-13T14:20:36.049 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-13T14:20:36.049 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-09-13T14:20:36.050 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi151.front.sepia.ceph.com 2024-09-13T14:20:36.050 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph config set mgr mgr/cephadm/daemon_cache_timeout 60' 2024-09-13T14:20:36.051 INFO:tasks.rados:clients are ['client.0', 'client.1'] 2024-09-13T14:20:36.051 INFO:tasks.rados:starting run 0 out of 1 2024-09-13T14:20:36.051 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-13T14:20:36.051 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-09-13T14:20:36.326 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:36.326 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:36.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:36 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/2487848573' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-13T14:20:36.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:36 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/2487848573' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-13T14:20:36.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:36 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/2487848573' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-13T14:20:37.459 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:37 smithi151 ceph-mon[29807]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:37.460 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:37 smithi151 ceph-mon[41387]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:37.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:37 smithi184 ceph-mon[33394]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:37.604 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:20:37 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:20:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:20:39.087 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:20:38 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:20:38] ENGINE Bus STOPPING 2024-09-13T14:20:39.087 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:20:38 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:20:38] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:20:39.088 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:20:38 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:20:38] ENGINE Bus STOPPED 2024-09-13T14:20:39.088 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:20:38 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:20:38] ENGINE Bus STARTING 2024-09-13T14:20:39.088 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:20:39 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:20:39] ENGINE Serving on http://:::9283 2024-09-13T14:20:39.088 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:20:39 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:20:39] ENGINE Bus STARTED 2024-09-13T14:20:40.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:39 smithi184 ceph-mon[33394]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:40.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:39 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/514194081' entity='client.admin' 2024-09-13T14:20:40.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:39 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:20:40.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:39 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:20:40.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:39 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:20:40.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:39 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:40.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:39 smithi151 ceph-mon[29807]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:40.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:39 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/514194081' entity='client.admin' 2024-09-13T14:20:40.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:39 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:20:40.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:39 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:20:40.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:39 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:20:40.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:39 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:40.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:39 smithi151 ceph-mon[41387]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:40.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:39 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/514194081' entity='client.admin' 2024-09-13T14:20:40.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:39 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:20:40.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:39 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:20:40.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:39 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:20:40.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:39 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:40.249 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph config set global log_to_journald false --force' 2024-09-13T14:20:40.523 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:40.929 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_0' created 2024-09-13T14:20:41.343 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:40 smithi151 ceph-mon[29807]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:41.343 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:40 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1683450517' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-13T14:20:41.343 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:40 smithi151 ceph-mon[29807]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-13T14:20:41.343 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:40 smithi151 ceph-mon[41387]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:41.343 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:40 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/1683450517' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-13T14:20:41.343 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:40 smithi151 ceph-mon[41387]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-13T14:20:41.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:40 smithi184 ceph-mon[33394]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:41.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:40 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/1683450517' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-13T14:20:41.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:40 smithi184 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-13T14:20:42.043 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:41 smithi151 ceph-mon[29807]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-13T14:20:42.043 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:41 smithi151 ceph-mon[29807]: osdmap e66: 8 total, 8 up, 8 in 2024-09-13T14:20:42.043 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:41 smithi151 ceph-mon[29807]: osdmap e67: 8 total, 8 up, 8 in 2024-09-13T14:20:42.044 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:41 smithi151 ceph-mon[41387]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-13T14:20:42.044 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:41 smithi151 ceph-mon[41387]: osdmap e66: 8 total, 8 up, 8 in 2024-09-13T14:20:42.044 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:41 smithi151 ceph-mon[41387]: osdmap e67: 8 total, 8 up, 8 in 2024-09-13T14:20:42.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:41 smithi184 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-13T14:20:42.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:41 smithi184 ceph-mon[33394]: osdmap e66: 8 total, 8 up, 8 in 2024-09-13T14:20:42.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:41 smithi184 ceph-mon[33394]: osdmap e67: 8 total, 8 up, 8 in 2024-09-13T14:20:42.959 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-13T14:20:43.269 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:42 smithi151 ceph-mon[29807]: pgmap v85: 89 pgs: 16 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:43.270 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:42 smithi151 ceph-mon[41387]: pgmap v85: 89 pgs: 16 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:43.270 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:20:42 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:20:42] "GET /metrics HTTP/1.1" 200 239818 "" "Prometheus/2.43.0" 2024-09-13T14:20:43.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:42 smithi184 ceph-mon[33394]: pgmap v85: 89 pgs: 16 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:43.730 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-09-13T14:20:43.737 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:44.029 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:43 smithi151 ceph-mon[29807]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:20:44.029 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:43 smithi151 ceph-mon[29807]: osdmap e68: 8 total, 8 up, 8 in 2024-09-13T14:20:44.029 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:43 smithi151 ceph-mon[41387]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:20:44.030 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:43 smithi151 ceph-mon[41387]: osdmap e68: 8 total, 8 up, 8 in 2024-09-13T14:20:44.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:43 smithi184 ceph-mon[33394]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:20:44.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:43 smithi184 ceph-mon[33394]: osdmap e68: 8 total, 8 up, 8 in 2024-09-13T14:20:44.484 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:45.297 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:44 smithi151 ceph-mon[29807]: pgmap v87: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:45.297 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:44 smithi151 ceph-mon[41387]: pgmap v87: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:45.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:44 smithi184 ceph-mon[33394]: pgmap v87: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:47.195 INFO:teuthology.orchestra.run.smithi151.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:20:47.454 INFO:teuthology.orchestra.run.smithi151.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-13T14:20:47.455 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:47 smithi151 ceph-mon[29807]: pgmap v88: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:47.455 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:47 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/451517796' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T14:20:47.455 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:47 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:47.455 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:47 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:20:47.455 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:47 smithi151 ceph-mon[41387]: pgmap v88: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:47.455 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:47 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/451517796' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T14:20:47.455 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:47 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:47.456 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:47 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:20:47.853 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:20:47 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:20:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:20:47.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:47 smithi184 ceph-mon[33394]: pgmap v88: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:47.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:47 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/451517796' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T14:20:47.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:47 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:47.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:47 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:20:47.980 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch ps' 2024-09-13T14:20:48.668 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:48 smithi151 ceph-mon[29807]: from='client.24716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:48.668 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:48 smithi151 ceph-mon[29807]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:20:48.668 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:48 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/451517796' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:20:48.668 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:48 smithi151 ceph-mon[29807]: osdmap e69: 8 total, 8 up, 8 in 2024-09-13T14:20:48.669 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:48 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:20:48.669 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:48 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:20:48.669 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:48 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:48.669 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:48 smithi151 ceph-mon[29807]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:20:48.669 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:48 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:48.669 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:48 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:20:48.669 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:48 smithi151 ceph-mon[41387]: from='client.24716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:48.669 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:48 smithi151 ceph-mon[41387]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:20:48.669 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:48 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/451517796' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:20:48.669 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:48 smithi151 ceph-mon[41387]: osdmap e69: 8 total, 8 up, 8 in 2024-09-13T14:20:48.669 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:48 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:20:48.670 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:48 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:20:48.670 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:48 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:48.670 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:48 smithi151 ceph-mon[41387]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:20:48.670 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:48 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:48.670 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:48 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:20:48.810 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:48.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:48 smithi184 ceph-mon[33394]: from='client.24716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:48.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:48 smithi184 ceph-mon[33394]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:20:48.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:48 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/451517796' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:20:48.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:48 smithi184 ceph-mon[33394]: osdmap e69: 8 total, 8 up, 8 in 2024-09-13T14:20:48.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:48 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:20:48.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:48 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:20:48.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:48 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:48.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:48 smithi184 ceph-mon[33394]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:20:48.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:48 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:20:48.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:48 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:20:49.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:49 smithi151 ceph-mon[41387]: pgmap v90: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:49.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:49 smithi151 ceph-mon[41387]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:20:49.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:49 smithi151 ceph-mon[41387]: Cluster is now healthy 2024-09-13T14:20:49.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:49 smithi151 ceph-mon[29807]: pgmap v90: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:49.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:49 smithi151 ceph-mon[29807]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:20:49.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:49 smithi151 ceph-mon[29807]: Cluster is now healthy 2024-09-13T14:20:49.694 DEBUG:teuthology.orchestra.run.smithi151:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 100 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --pool unique_pool_0 2024-09-13T14:20:49.741 INFO:tasks.rados.rados.0.smithi151.stdout:adding op weight read -> 100 2024-09-13T14:20:49.741 INFO:tasks.rados.rados.0.smithi151.stdout:adding op weight write -> 100 2024-09-13T14:20:49.741 INFO:tasks.rados.rados.0.smithi151.stdout:adding op weight delete -> 50 2024-09-13T14:20:49.741 INFO:tasks.rados.rados.0.smithi151.stdout:adding op weight snap_create -> 50 2024-09-13T14:20:49.741 INFO:tasks.rados.rados.0.smithi151.stdout:adding op weight snap_remove -> 50 2024-09-13T14:20:49.741 INFO:tasks.rados.rados.0.smithi151.stdout:adding op weight rollback -> 50 2024-09-13T14:20:49.741 INFO:tasks.rados.rados.0.smithi151.stdout:ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable) 2024-09-13T14:20:49.741 INFO:tasks.rados.rados.0.smithi151.stdout:Configuration: 2024-09-13T14:20:49.741 INFO:tasks.rados.rados.0.smithi151.stdout: Number of operations: 4000 2024-09-13T14:20:49.741 INFO:tasks.rados.rados.0.smithi151.stdout: Number of objects: 50 2024-09-13T14:20:49.741 INFO:tasks.rados.rados.0.smithi151.stdout: Max in flight operations: 16 2024-09-13T14:20:49.742 INFO:tasks.rados.rados.0.smithi151.stdout: Object size (in bytes): 4000000 2024-09-13T14:20:49.742 INFO:tasks.rados.rados.0.smithi151.stdout: Write stride min: 400000 2024-09-13T14:20:49.742 INFO:tasks.rados.rados.0.smithi151.stdout: Write stride max: 800000 2024-09-13T14:20:49.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:49 smithi184 ceph-mon[33394]: pgmap v90: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:49.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:49 smithi184 ceph-mon[33394]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:20:49.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:49 smithi184 ceph-mon[33394]: Cluster is now healthy 2024-09-13T14:20:50.463 INFO:tasks.rados.rados.0.smithi151.stdout:1: write initial oid 1 2024-09-13T14:20:50.463 INFO:tasks.rados.rados.0.smithi151.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2024-09-13T14:20:50.468 INFO:tasks.rados.rados.0.smithi151.stdout:1: writing smithi15186934-1 from 416807 to 892056 tid 1 2024-09-13T14:20:50.475 INFO:tasks.rados.rados.0.smithi151.stdout:1: writing smithi15186934-1 from 1542129 to 2016807 tid 2 2024-09-13T14:20:50.475 INFO:tasks.rados.rados.0.smithi151.stdout:2: write initial oid 2 2024-09-13T14:20:50.475 INFO:tasks.rados.rados.0.smithi151.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2024-09-13T14:20:50.480 INFO:tasks.rados.rados.0.smithi151.stdout:2: writing smithi15186934-2 from 416807 to 892056 tid 1 2024-09-13T14:20:50.487 INFO:tasks.rados.rados.0.smithi151.stdout:2: writing smithi15186934-2 from 1542129 to 2016807 tid 2 2024-09-13T14:20:50.488 INFO:tasks.rados.rados.0.smithi151.stdout:3: write initial oid 3 2024-09-13T14:20:50.488 INFO:tasks.rados.rados.0.smithi151.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2024-09-13T14:20:50.493 INFO:tasks.rados.rados.0.smithi151.stdout:3: writing smithi15186934-3 from 433614 to 984112 tid 1 2024-09-13T14:20:50.499 INFO:tasks.rados.rados.0.smithi151.stdout:3: writing smithi15186934-3 from 1600611 to 2033614 tid 2 2024-09-13T14:20:50.499 INFO:tasks.rados.rados.0.smithi151.stdout:4: write initial oid 4 2024-09-13T14:20:50.499 INFO:tasks.rados.rados.0.smithi151.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2024-09-13T14:20:50.505 INFO:tasks.rados.rados.0.smithi151.stdout:4: writing smithi15186934-4 from 450421 to 1076168 tid 1 2024-09-13T14:20:50.511 INFO:tasks.rados.rados.0.smithi151.stdout:4: writing smithi15186934-4 from 1659093 to 2050421 tid 2 2024-09-13T14:20:50.511 INFO:tasks.rados.rados.0.smithi151.stdout:5: write initial oid 5 2024-09-13T14:20:50.512 INFO:tasks.rados.rados.0.smithi151.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2024-09-13T14:20:50.518 INFO:tasks.rados.rados.0.smithi151.stdout:5: writing smithi15186934-5 from 467228 to 1168224 tid 1 2024-09-13T14:20:50.524 INFO:tasks.rados.rados.0.smithi151.stdout:5: writing smithi15186934-5 from 1717575 to 2067228 tid 2 2024-09-13T14:20:50.524 INFO:tasks.rados.rados.0.smithi151.stdout:6: write initial oid 6 2024-09-13T14:20:50.524 INFO:tasks.rados.rados.0.smithi151.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2024-09-13T14:20:50.531 INFO:tasks.rados.rados.0.smithi151.stdout:6: writing smithi15186934-6 from 484035 to 1260280 tid 1 2024-09-13T14:20:50.536 INFO:tasks.rados.rados.0.smithi151.stdout:6: writing smithi15186934-6 from 2059704 to 2084035 tid 2 2024-09-13T14:20:50.536 INFO:tasks.rados.rados.0.smithi151.stdout:7: write initial oid 7 2024-09-13T14:20:50.536 INFO:tasks.rados.rados.0.smithi151.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2024-09-13T14:20:50.541 INFO:tasks.rados.rados.0.smithi151.stdout:7: writing smithi15186934-7 from 500842 to 952336 tid 1 2024-09-13T14:20:50.547 INFO:tasks.rados.rados.0.smithi151.stdout:7: writing smithi15186934-7 from 1718186 to 2100842 tid 2 2024-09-13T14:20:50.547 INFO:tasks.rados.rados.0.smithi151.stdout:8: write initial oid 8 2024-09-13T14:20:50.547 INFO:tasks.rados.rados.0.smithi151.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2024-09-13T14:20:50.553 INFO:tasks.rados.rados.0.smithi151.stdout:8: writing smithi15186934-8 from 517649 to 1044392 tid 1 2024-09-13T14:20:50.560 INFO:tasks.rados.rados.0.smithi151.stdout:8: writing smithi15186934-8 from 1776668 to 2117649 tid 2 2024-09-13T14:20:50.560 INFO:tasks.rados.rados.0.smithi151.stdout:9: write initial oid 9 2024-09-13T14:20:50.560 INFO:tasks.rados.rados.0.smithi151.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2024-09-13T14:20:50.569 INFO:tasks.rados.rados.0.smithi151.stdout:9: writing smithi15186934-9 from 534456 to 1252801 tid 1 2024-09-13T14:20:50.572 INFO:tasks.rados.rados.0.smithi151.stdout:9: writing smithi15186934-9 from 1951503 to 2134456 tid 2 2024-09-13T14:20:50.572 INFO:tasks.rados.rados.0.smithi151.stdout:10: write initial oid 10 2024-09-13T14:20:50.572 INFO:tasks.rados.rados.0.smithi151.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2024-09-13T14:20:50.579 INFO:tasks.rados.rados.0.smithi151.stdout:10: writing smithi15186934-10 from 551263 to 1344857 tid 1 2024-09-13T14:20:50.584 INFO:tasks.rados.rados.0.smithi151.stdout:10: writing smithi15186934-10 from 1893632 to 2151263 tid 2 2024-09-13T14:20:50.584 INFO:tasks.rados.rados.0.smithi151.stdout:11: write initial oid 11 2024-09-13T14:20:50.584 INFO:tasks.rados.rados.0.smithi151.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2024-09-13T14:20:50.585 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:50 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1096909476' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2024-09-13T14:20:50.586 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:50 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/1096909476' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2024-09-13T14:20:50.590 INFO:tasks.rados.rados.0.smithi151.stdout:11: writing smithi15186934-11 from 568070 to 1036913 tid 1 2024-09-13T14:20:50.596 INFO:tasks.rados.rados.0.smithi151.stdout:11: writing smithi15186934-11 from 1552114 to 2168070 tid 2 2024-09-13T14:20:50.597 INFO:tasks.rados.rados.0.smithi151.stdout:12: write initial oid 12 2024-09-13T14:20:50.597 INFO:tasks.rados.rados.0.smithi151.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2024-09-13T14:20:50.602 INFO:tasks.rados.rados.0.smithi151.stdout:12: writing smithi15186934-12 from 584877 to 1128969 tid 1 2024-09-13T14:20:50.608 INFO:tasks.rados.rados.0.smithi151.stdout:12: writing smithi15186934-12 from 1610596 to 2172599 tid 2 2024-09-13T14:20:50.609 INFO:tasks.rados.rados.0.smithi151.stdout:13: write initial oid 13 2024-09-13T14:20:50.609 INFO:tasks.rados.rados.0.smithi151.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2024-09-13T14:20:50.615 INFO:tasks.rados.rados.0.smithi151.stdout:13: writing smithi15186934-13 from 601684 to 1221025 tid 1 2024-09-13T14:20:50.621 INFO:tasks.rados.rados.0.smithi151.stdout:13: writing smithi15186934-13 from 1669078 to 2201684 tid 2 2024-09-13T14:20:50.621 INFO:tasks.rados.rados.0.smithi151.stdout:14: write initial oid 14 2024-09-13T14:20:50.621 INFO:tasks.rados.rados.0.smithi151.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2024-09-13T14:20:50.628 INFO:tasks.rados.rados.0.smithi151.stdout:14: writing smithi15186934-14 from 618491 to 1313081 tid 1 2024-09-13T14:20:50.633 INFO:tasks.rados.rados.0.smithi151.stdout:14: writing smithi15186934-14 from 2011207 to 2218491 tid 2 2024-09-13T14:20:50.634 INFO:tasks.rados.rados.0.smithi151.stdout:15: write initial oid 15 2024-09-13T14:20:50.634 INFO:tasks.rados.rados.0.smithi151.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2024-09-13T14:20:50.641 INFO:tasks.rados.rados.0.smithi151.stdout:15: writing smithi15186934-15 from 635298 to 1405137 tid 1 2024-09-13T14:20:50.646 INFO:tasks.rados.rados.0.smithi151.stdout:15: writing smithi15186934-15 from 2069689 to 2235298 tid 2 2024-09-13T14:20:50.647 INFO:tasks.rados.rados.0.smithi151.stdout:16: write initial oid 16 2024-09-13T14:20:50.647 INFO:tasks.rados.rados.0.smithi151.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2024-09-13T14:20:50.652 INFO:tasks.rados.rados.0.smithi151.stdout:16: writing smithi15186934-16 from 652105 to 1097193 tid 1 2024-09-13T14:20:50.660 INFO:tasks.rados.rados.0.smithi151.stdout:16: writing smithi15186934-16 from 1728171 to 2181159 tid 2 2024-09-13T14:20:50.660 INFO:tasks.rados.rados.0.smithi151.stdout: waiting on 16 2024-09-13T14:20:50.660 INFO:tasks.rados.rados.0.smithi151.stdout:1: finishing write tid 1 to smithi15186934-1 2024-09-13T14:20:50.660 INFO:tasks.rados.rados.0.smithi151.stdout:1: finishing write tid 2 to smithi15186934-1 2024-09-13T14:20:50.660 INFO:tasks.rados.rados.0.smithi151.stdout:1: finishing write tid 3 to smithi15186934-1 2024-09-13T14:20:50.661 INFO:tasks.rados.rados.0.smithi151.stdout:1: finishing write tid 4 to smithi15186934-1 2024-09-13T14:20:50.661 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-13T14:20:50.661 INFO:tasks.rados.rados.0.smithi151.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-09-13T14:20:50.661 INFO:tasks.rados.rados.0.smithi151.stdout:2: finishing write tid 1 to smithi15186934-2 2024-09-13T14:20:50.661 INFO:tasks.rados.rados.0.smithi151.stdout:1: done (15 left) 2024-09-13T14:20:50.661 INFO:tasks.rados.rados.0.smithi151.stdout:17: write initial oid 17 2024-09-13T14:20:50.661 INFO:tasks.rados.rados.0.smithi151.stdout:2: finishing write tid 2 to smithi15186934-2 2024-09-13T14:20:50.661 INFO:tasks.rados.rados.0.smithi151.stdout:2: finishing write tid 3 to smithi15186934-2 2024-09-13T14:20:50.661 INFO:tasks.rados.rados.0.smithi151.stdout:2: finishing write tid 4 to smithi15186934-2 2024-09-13T14:20:50.661 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-13T14:20:50.661 INFO:tasks.rados.rados.0.smithi151.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-09-13T14:20:50.661 INFO:tasks.rados.rados.0.smithi151.stdout:3: finishing write tid 1 to smithi15186934-3 2024-09-13T14:20:50.661 INFO:tasks.rados.rados.0.smithi151.stdout:3: finishing write tid 2 to smithi15186934-3 2024-09-13T14:20:50.661 INFO:tasks.rados.rados.0.smithi151.stdout:3: finishing write tid 3 to smithi15186934-3 2024-09-13T14:20:50.662 INFO:tasks.rados.rados.0.smithi151.stdout:3: finishing write tid 4 to smithi15186934-3 2024-09-13T14:20:50.662 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T14:20:50.662 INFO:tasks.rados.rados.0.smithi151.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-09-13T14:20:50.662 INFO:tasks.rados.rados.0.smithi151.stdout:4: finishing write tid 1 to smithi15186934-4 2024-09-13T14:20:50.662 INFO:tasks.rados.rados.0.smithi151.stdout:4: finishing write tid 2 to smithi15186934-4 2024-09-13T14:20:50.662 INFO:tasks.rados.rados.0.smithi151.stdout:4: finishing write tid 3 to smithi15186934-4 2024-09-13T14:20:50.662 INFO:tasks.rados.rados.0.smithi151.stdout:4: finishing write tid 4 to smithi15186934-4 2024-09-13T14:20:50.662 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-13T14:20:50.662 INFO:tasks.rados.rados.0.smithi151.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-09-13T14:20:50.662 INFO:tasks.rados.rados.0.smithi151.stdout:5: finishing write tid 1 to smithi15186934-5 2024-09-13T14:20:50.662 INFO:tasks.rados.rados.0.smithi151.stdout:5: finishing write tid 2 to smithi15186934-5 2024-09-13T14:20:50.662 INFO:tasks.rados.rados.0.smithi151.stdout:5: finishing write tid 3 to smithi15186934-5 2024-09-13T14:20:50.662 INFO:tasks.rados.rados.0.smithi151.stdout:5: finishing write tid 4 to smithi15186934-5 2024-09-13T14:20:50.663 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 6 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-13T14:20:50.663 INFO:tasks.rados.rados.0.smithi151.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-09-13T14:20:50.663 INFO:tasks.rados.rados.0.smithi151.stdout:6: finishing write tid 1 to smithi15186934-6 2024-09-13T14:20:50.663 INFO:tasks.rados.rados.0.smithi151.stdout:6: finishing write tid 2 to smithi15186934-6 2024-09-13T14:20:50.663 INFO:tasks.rados.rados.0.smithi151.stdout:6: finishing write tid 3 to smithi15186934-6 2024-09-13T14:20:50.663 INFO:tasks.rados.rados.0.smithi151.stdout:6: finishing write tid 4 to smithi15186934-6 2024-09-13T14:20:50.663 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-13T14:20:50.663 INFO:tasks.rados.rados.0.smithi151.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-09-13T14:20:50.663 INFO:tasks.rados.rados.0.smithi151.stdout:7: finishing write tid 1 to smithi15186934-7 2024-09-13T14:20:50.663 INFO:tasks.rados.rados.0.smithi151.stdout:7: finishing write tid 2 to smithi15186934-7 2024-09-13T14:20:50.663 INFO:tasks.rados.rados.0.smithi151.stdout:7: finishing write tid 3 to smithi15186934-7 2024-09-13T14:20:50.663 INFO:tasks.rados.rados.0.smithi151.stdout:7: finishing write tid 4 to smithi15186934-7 2024-09-13T14:20:50.663 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-13T14:20:50.663 INFO:tasks.rados.rados.0.smithi151.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-09-13T14:20:50.664 INFO:tasks.rados.rados.0.smithi151.stdout:8: finishing write tid 1 to smithi15186934-8 2024-09-13T14:20:50.664 INFO:tasks.rados.rados.0.smithi151.stdout:8: finishing write tid 2 to smithi15186934-8 2024-09-13T14:20:50.664 INFO:tasks.rados.rados.0.smithi151.stdout:8: finishing write tid 3 to smithi15186934-8 2024-09-13T14:20:50.664 INFO:tasks.rados.rados.0.smithi151.stdout:8: finishing write tid 4 to smithi15186934-8 2024-09-13T14:20:50.664 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 6 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-13T14:20:50.664 INFO:tasks.rados.rados.0.smithi151.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-09-13T14:20:50.664 INFO:tasks.rados.rados.0.smithi151.stdout:9: finishing write tid 1 to smithi15186934-9 2024-09-13T14:20:50.664 INFO:tasks.rados.rados.0.smithi151.stdout:9: finishing write tid 2 to smithi15186934-9 2024-09-13T14:20:50.664 INFO:tasks.rados.rados.0.smithi151.stdout:9: finishing write tid 3 to smithi15186934-9 2024-09-13T14:20:50.664 INFO:tasks.rados.rados.0.smithi151.stdout:9: finishing write tid 4 to smithi15186934-9 2024-09-13T14:20:50.664 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-13T14:20:50.665 INFO:tasks.rados.rados.0.smithi151.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-09-13T14:20:50.665 INFO:tasks.rados.rados.0.smithi151.stdout:10: finishing write tid 1 to smithi15186934-10 2024-09-13T14:20:50.665 INFO:tasks.rados.rados.0.smithi151.stdout:10: finishing write tid 2 to smithi15186934-10 2024-09-13T14:20:50.665 INFO:tasks.rados.rados.0.smithi151.stdout:10: finishing write tid 3 to smithi15186934-10 2024-09-13T14:20:50.665 INFO:tasks.rados.rados.0.smithi151.stdout:10: finishing write tid 4 to smithi15186934-10 2024-09-13T14:20:50.665 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-13T14:20:50.665 INFO:tasks.rados.rados.0.smithi151.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-09-13T14:20:50.665 INFO:tasks.rados.rados.0.smithi151.stdout:11: finishing write tid 1 to smithi15186934-11 2024-09-13T14:20:50.665 INFO:tasks.rados.rados.0.smithi151.stdout:11: finishing write tid 2 to smithi15186934-11 2024-09-13T14:20:50.665 INFO:tasks.rados.rados.0.smithi151.stdout:11: finishing write tid 3 to smithi15186934-11 2024-09-13T14:20:50.665 INFO:tasks.rados.rados.0.smithi151.stdout:11: finishing write tid 4 to smithi15186934-11 2024-09-13T14:20:50.665 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-13T14:20:50.666 INFO:tasks.rados.rados.0.smithi151.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-09-13T14:20:50.666 INFO:tasks.rados.rados.0.smithi151.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2024-09-13T14:20:50.666 INFO:tasks.rados.rados.0.smithi151.stdout:17: writing smithi15186934-17 from 668912 to 1305602 tid 1 2024-09-13T14:20:50.671 INFO:tasks.rados.rados.0.smithi151.stdout:17: writing smithi15186934-17 from 1903006 to 2268912 tid 2 2024-09-13T14:20:50.672 INFO:tasks.rados.rados.0.smithi151.stdout:2: done (15 left) 2024-09-13T14:20:50.672 INFO:tasks.rados.rados.0.smithi151.stdout:3: done (14 left) 2024-09-13T14:20:50.672 INFO:tasks.rados.rados.0.smithi151.stdout:4: done (13 left) 2024-09-13T14:20:50.672 INFO:tasks.rados.rados.0.smithi151.stdout:5: done (12 left) 2024-09-13T14:20:50.672 INFO:tasks.rados.rados.0.smithi151.stdout:6: done (11 left) 2024-09-13T14:20:50.672 INFO:tasks.rados.rados.0.smithi151.stdout:7: done (10 left) 2024-09-13T14:20:50.672 INFO:tasks.rados.rados.0.smithi151.stdout:8: done (9 left) 2024-09-13T14:20:50.672 INFO:tasks.rados.rados.0.smithi151.stdout:9: done (8 left) 2024-09-13T14:20:50.672 INFO:tasks.rados.rados.0.smithi151.stdout:10: done (7 left) 2024-09-13T14:20:50.672 INFO:tasks.rados.rados.0.smithi151.stdout:11: done (6 left) 2024-09-13T14:20:50.672 INFO:tasks.rados.rados.0.smithi151.stdout:18: write initial oid 18 2024-09-13T14:20:50.672 INFO:tasks.rados.rados.0.smithi151.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2024-09-13T14:20:50.679 INFO:tasks.rados.rados.0.smithi151.stdout:18: writing smithi15186934-18 from 685719 to 1397658 tid 1 2024-09-13T14:20:50.684 INFO:tasks.rados.rados.0.smithi151.stdout:18: writing smithi15186934-18 from 1845135 to 2285719 tid 2 2024-09-13T14:20:50.684 INFO:tasks.rados.rados.0.smithi151.stdout:19: write initial oid 19 2024-09-13T14:20:50.684 INFO:tasks.rados.rados.0.smithi151.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2024-09-13T14:20:50.692 INFO:tasks.rados.rados.0.smithi151.stdout:19: writing smithi15186934-19 from 702526 to 1489714 tid 1 2024-09-13T14:20:50.697 INFO:tasks.rados.rados.0.smithi151.stdout:19: writing smithi15186934-19 from 1903617 to 2302526 tid 2 2024-09-13T14:20:50.697 INFO:tasks.rados.rados.0.smithi151.stdout:20: write initial oid 20 2024-09-13T14:20:50.697 INFO:tasks.rados.rados.0.smithi151.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2024-09-13T14:20:50.704 INFO:tasks.rados.rados.0.smithi151.stdout:20: writing smithi15186934-20 from 719333 to 1181770 tid 1 2024-09-13T14:20:50.710 INFO:tasks.rados.rados.0.smithi151.stdout:20: writing smithi15186934-20 from 1962099 to 2319333 tid 2 2024-09-13T14:20:50.710 INFO:tasks.rados.rados.0.smithi151.stdout:21: write initial oid 21 2024-09-13T14:20:50.710 INFO:tasks.rados.rados.0.smithi151.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2024-09-13T14:20:50.717 INFO:tasks.rados.rados.0.smithi151.stdout:21: writing smithi15186934-21 from 736140 to 1273826 tid 1 2024-09-13T14:20:50.723 INFO:tasks.rados.rados.0.smithi151.stdout:21: writing smithi15186934-21 from 2020581 to 2336140 tid 2 2024-09-13T14:20:50.723 INFO:tasks.rados.rados.0.smithi151.stdout:22: write initial oid 22 2024-09-13T14:20:50.723 INFO:tasks.rados.rados.0.smithi151.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2024-09-13T14:20:50.731 INFO:tasks.rados.rados.0.smithi151.stdout:22: writing smithi15186934-22 from 752947 to 1365882 tid 1 2024-09-13T14:20:50.736 INFO:tasks.rados.rados.0.smithi151.stdout:22: writing smithi15186934-22 from 1962710 to 2352947 tid 2 2024-09-13T14:20:50.736 INFO:tasks.rados.rados.0.smithi151.stdout:23: write initial oid 23 2024-09-13T14:20:50.737 INFO:tasks.rados.rados.0.smithi151.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2024-09-13T14:20:50.744 INFO:tasks.rados.rados.0.smithi151.stdout:23: writing smithi15186934-23 from 769754 to 1457938 tid 1 2024-09-13T14:20:50.750 INFO:tasks.rados.rados.0.smithi151.stdout:23: writing smithi15186934-23 from 2021192 to 2369754 tid 2 2024-09-13T14:20:50.750 INFO:tasks.rados.rados.0.smithi151.stdout:24: write initial oid 24 2024-09-13T14:20:50.750 INFO:tasks.rados.rados.0.smithi151.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2024-09-13T14:20:50.757 INFO:tasks.rados.rados.0.smithi151.stdout:24: writing smithi15186934-24 from 786561 to 1266347 tid 1 2024-09-13T14:20:50.762 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:20:50.762 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.a smithi151 *:9093,9094 running (78s) 16s ago 2m 23.8M - 0.25.0 c8568f914cd2 457393043edc 2024-09-13T14:20:50.762 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.a smithi184 *:3000 running (118s) 19s ago 118s 80.2M - 9.4.7 954c08fa6188 20c2e676f615 2024-09-13T14:20:50.762 INFO:teuthology.orchestra.run.smithi151.stdout:mds.foo.smithi151.rqqklw smithi151 running (46s) 16s ago 45s 13.7M - 17.2.7-1627-gbbdadfa3 a994b00deb3a c960b57fc84b 2024-09-13T14:20:50.762 INFO:teuthology.orchestra.run.smithi151.stdout:mds.foo.smithi184.jismsi smithi184 running (43s) 19s ago 43s 12.7M - 17.2.7-1627-gbbdadfa3 a994b00deb3a 0cc31a3a5339 2024-09-13T14:20:50.762 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.x smithi184 *:8443 running (6m) 19s ago 6m 399M - 17.2.7-1627-gbbdadfa3 a994b00deb3a 1d88b55dd4f1 2024-09-13T14:20:50.762 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.y smithi151 *:9283,8765 running (7m) 16s ago 7m 447M - 17.2.7-1627-gbbdadfa3 a994b00deb3a 6ce1f75b6d5e 2024-09-13T14:20:50.762 INFO:teuthology.orchestra.run.smithi151.stdout:mon.a smithi151 running (8m) 16s ago 8m 47.0M 2048M 17.2.7-1627-gbbdadfa3 a994b00deb3a ffc441eb291c 2024-09-13T14:20:50.762 INFO:teuthology.orchestra.run.smithi151.stdout:mon.b smithi184 running (6m) 19s ago 6m 39.3M 2048M 17.2.7-1627-gbbdadfa3 a994b00deb3a 8c637f147650 2024-09-13T14:20:50.762 INFO:teuthology.orchestra.run.smithi151.stdout:mon.c smithi151 running (6m) 16s ago 6m 41.1M 2048M 17.2.7-1627-gbbdadfa3 a994b00deb3a dccc3a65ea4d 2024-09-13T14:20:50.762 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.a smithi151 *:9100 running (101s) 16s ago 98s 15.4M - 1.5.0 0da6a335fe13 401be9d35c59 2024-09-13T14:20:50.763 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.b smithi184 *:9100 running (95s) 19s ago 94s 8384k - 1.5.0 0da6a335fe13 aca4c09ae196 2024-09-13T14:20:50.763 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (6m) 16s ago 6m 43.8M 4016M 17.2.7-1627-gbbdadfa3 a994b00deb3a e6951a1a86d9 2024-09-13T14:20:50.763 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (5m) 16s ago 5m 42.2M 4016M 17.2.7-1627-gbbdadfa3 a994b00deb3a 94bc726b3eb5 2024-09-13T14:20:50.763 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (5m) 16s ago 5m 40.6M 4016M 17.2.7-1627-gbbdadfa3 a994b00deb3a 122660181e46 2024-09-13T14:20:50.763 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi151 running (4m) 16s ago 4m 41.9M 4016M 17.2.7-1627-gbbdadfa3 a994b00deb3a 495241a87df6 2024-09-13T14:20:50.763 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi184 running (4m) 19s ago 4m 44.4M 4016M 17.2.7-1627-gbbdadfa3 a994b00deb3a 6e70699b8084 2024-09-13T14:20:50.763 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi184 running (4m) 19s ago 4m 41.4M 4016M 17.2.7-1627-gbbdadfa3 a994b00deb3a 36b82c9e7230 2024-09-13T14:20:50.763 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6 smithi184 running (3m) 19s ago 3m 40.4M 4016M 17.2.7-1627-gbbdadfa3 a994b00deb3a f9379cec7504 2024-09-13T14:20:50.763 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7 smithi184 running (3m) 19s ago 3m 41.0M 4016M 17.2.7-1627-gbbdadfa3 a994b00deb3a f37cc947097b 2024-09-13T14:20:50.763 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.a smithi184 *:9095 running (72s) 19s ago 2m 39.8M - 2.43.0 a07b618ecd1d 0d5bc4586d6c 2024-09-13T14:20:50.764 INFO:tasks.rados.rados.0.smithi151.stdout:24: writing smithi15186934-24 from 1796027 to 2263691 tid 2 2024-09-13T14:20:50.764 INFO:tasks.rados.rados.0.smithi151.stdout:25: write initial oid 25 2024-09-13T14:20:50.764 INFO:tasks.rados.rados.0.smithi151.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2024-09-13T14:20:50.767 INFO:tasks.rados.rados.0.smithi151.stdout:25: writing smithi15186934-25 from 403368 to 958403 tid 1 2024-09-13T14:20:50.774 INFO:tasks.rados.rados.0.smithi151.stdout:25: writing smithi15186934-25 from 1454509 to 2182184 tid 2 2024-09-13T14:20:50.774 INFO:tasks.rados.rados.0.smithi151.stdout:26: write initial oid 26 2024-09-13T14:20:50.775 INFO:tasks.rados.rados.0.smithi151.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2024-09-13T14:20:50.781 INFO:tasks.rados.rados.0.smithi151.stdout:26: writing smithi15186934-26 from 420175 to 1050459 tid 1 2024-09-13T14:20:50.787 INFO:tasks.rados.rados.0.smithi151.stdout:26: writing smithi15186934-26 from 1796638 to 2267971 tid 2 2024-09-13T14:20:50.788 INFO:tasks.rados.rados.0.smithi151.stdout:27: write initial oid 27 2024-09-13T14:20:50.788 INFO:tasks.rados.rados.0.smithi151.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2024-09-13T14:20:50.794 INFO:tasks.rados.rados.0.smithi151.stdout:27: writing smithi15186934-27 from 436982 to 1142515 tid 1 2024-09-13T14:20:50.803 INFO:tasks.rados.rados.0.smithi151.stdout:27: writing smithi15186934-27 from 1855120 to 2436982 tid 2 2024-09-13T14:20:50.803 INFO:tasks.rados.rados.0.smithi151.stdout: waiting on 16 2024-09-13T14:20:50.803 INFO:tasks.rados.rados.0.smithi151.stdout:12: finishing write tid 1 to smithi15186934-12 2024-09-13T14:20:50.803 INFO:tasks.rados.rados.0.smithi151.stdout:12: finishing write tid 2 to smithi15186934-12 2024-09-13T14:20:50.803 INFO:tasks.rados.rados.0.smithi151.stdout:12: finishing write tid 3 to smithi15186934-12 2024-09-13T14:20:50.803 INFO:tasks.rados.rados.0.smithi151.stdout:12: finishing write tid 4 to smithi15186934-12 2024-09-13T14:20:50.803 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-13T14:20:50.803 INFO:tasks.rados.rados.0.smithi151.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-09-13T14:20:50.804 INFO:tasks.rados.rados.0.smithi151.stdout:13: finishing write tid 1 to smithi15186934-13 2024-09-13T14:20:50.804 INFO:tasks.rados.rados.0.smithi151.stdout:13: finishing write tid 2 to smithi15186934-13 2024-09-13T14:20:50.804 INFO:tasks.rados.rados.0.smithi151.stdout:13: finishing write tid 3 to smithi15186934-13 2024-09-13T14:20:50.804 INFO:tasks.rados.rados.0.smithi151.stdout:13: finishing write tid 4 to smithi15186934-13 2024-09-13T14:20:50.804 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 13 v 9 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-13T14:20:50.804 INFO:tasks.rados.rados.0.smithi151.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-09-13T14:20:50.804 INFO:tasks.rados.rados.0.smithi151.stdout:14: finishing write tid 1 to smithi15186934-14 2024-09-13T14:20:50.804 INFO:tasks.rados.rados.0.smithi151.stdout:14: finishing write tid 2 to smithi15186934-14 2024-09-13T14:20:50.804 INFO:tasks.rados.rados.0.smithi151.stdout:14: finishing write tid 3 to smithi15186934-14 2024-09-13T14:20:50.804 INFO:tasks.rados.rados.0.smithi151.stdout:14: finishing write tid 4 to smithi15186934-14 2024-09-13T14:20:50.804 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 6 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-13T14:20:50.804 INFO:tasks.rados.rados.0.smithi151.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-09-13T14:20:50.804 INFO:tasks.rados.rados.0.smithi151.stdout:15: finishing write tid 1 to smithi15186934-15 2024-09-13T14:20:50.804 INFO:tasks.rados.rados.0.smithi151.stdout:15: finishing write tid 2 to smithi15186934-15 2024-09-13T14:20:50.805 INFO:tasks.rados.rados.0.smithi151.stdout:15: finishing write tid 3 to smithi15186934-15 2024-09-13T14:20:50.805 INFO:tasks.rados.rados.0.smithi151.stdout:15: finishing write tid 4 to smithi15186934-15 2024-09-13T14:20:50.805 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 15 v 3 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-13T14:20:50.805 INFO:tasks.rados.rados.0.smithi151.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-09-13T14:20:50.805 INFO:tasks.rados.rados.0.smithi151.stdout:16: finishing write tid 1 to smithi15186934-16 2024-09-13T14:20:50.805 INFO:tasks.rados.rados.0.smithi151.stdout:16: finishing write tid 2 to smithi15186934-16 2024-09-13T14:20:50.805 INFO:tasks.rados.rados.0.smithi151.stdout:16: finishing write tid 3 to smithi15186934-16 2024-09-13T14:20:50.805 INFO:tasks.rados.rados.0.smithi151.stdout:16: finishing write tid 4 to smithi15186934-16 2024-09-13T14:20:50.805 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 16 v 3 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-13T14:20:50.805 INFO:tasks.rados.rados.0.smithi151.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-09-13T14:20:50.805 INFO:tasks.rados.rados.0.smithi151.stdout:17: finishing write tid 1 to smithi15186934-17 2024-09-13T14:20:50.805 INFO:tasks.rados.rados.0.smithi151.stdout:17: finishing write tid 2 to smithi15186934-17 2024-09-13T14:20:50.805 INFO:tasks.rados.rados.0.smithi151.stdout:17: finishing write tid 3 to smithi15186934-17 2024-09-13T14:20:50.805 INFO:tasks.rados.rados.0.smithi151.stdout:17: finishing write tid 4 to smithi15186934-17 2024-09-13T14:20:50.805 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 6 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-13T14:20:50.806 INFO:tasks.rados.rados.0.smithi151.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-09-13T14:20:50.806 INFO:tasks.rados.rados.0.smithi151.stdout:18: finishing write tid 1 to smithi15186934-18 2024-09-13T14:20:50.806 INFO:tasks.rados.rados.0.smithi151.stdout:18: finishing write tid 2 to smithi15186934-18 2024-09-13T14:20:50.806 INFO:tasks.rados.rados.0.smithi151.stdout:18: finishing write tid 3 to smithi15186934-18 2024-09-13T14:20:50.806 INFO:tasks.rados.rados.0.smithi151.stdout:18: finishing write tid 4 to smithi15186934-18 2024-09-13T14:20:50.806 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-13T14:20:50.806 INFO:tasks.rados.rados.0.smithi151.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-09-13T14:20:50.806 INFO:tasks.rados.rados.0.smithi151.stdout:19: finishing write tid 1 to smithi15186934-19 2024-09-13T14:20:50.806 INFO:tasks.rados.rados.0.smithi151.stdout:19: finishing write tid 2 to smithi15186934-19 2024-09-13T14:20:50.806 INFO:tasks.rados.rados.0.smithi151.stdout:19: finishing write tid 3 to smithi15186934-19 2024-09-13T14:20:50.806 INFO:tasks.rados.rados.0.smithi151.stdout:19: finishing write tid 4 to smithi15186934-19 2024-09-13T14:20:50.806 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 19 v 6 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-13T14:20:50.806 INFO:tasks.rados.rados.0.smithi151.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-09-13T14:20:50.806 INFO:tasks.rados.rados.0.smithi151.stdout:20: finishing write tid 1 to smithi15186934-20 2024-09-13T14:20:50.806 INFO:tasks.rados.rados.0.smithi151.stdout:20: finishing write tid 2 to smithi15186934-20 2024-09-13T14:20:50.806 INFO:tasks.rados.rados.0.smithi151.stdout:12: done (15 left) 2024-09-13T14:20:50.807 INFO:tasks.rados.rados.0.smithi151.stdout:13: done (14 left) 2024-09-13T14:20:50.807 INFO:tasks.rados.rados.0.smithi151.stdout:14: done (13 left) 2024-09-13T14:20:50.807 INFO:tasks.rados.rados.0.smithi151.stdout:15: done (12 left) 2024-09-13T14:20:50.807 INFO:tasks.rados.rados.0.smithi151.stdout:16: done (11 left) 2024-09-13T14:20:50.807 INFO:tasks.rados.rados.0.smithi151.stdout:17: done (10 left) 2024-09-13T14:20:50.807 INFO:tasks.rados.rados.0.smithi151.stdout:18: done (9 left) 2024-09-13T14:20:50.807 INFO:tasks.rados.rados.0.smithi151.stdout:19: done (8 left) 2024-09-13T14:20:50.807 INFO:tasks.rados.rados.0.smithi151.stdout:28: write initial oid 28 2024-09-13T14:20:50.807 INFO:tasks.rados.rados.0.smithi151.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2024-09-13T14:20:50.808 INFO:tasks.rados.rados.0.smithi151.stdout:28: writing smithi15186934-28 from 453789 to 1234571 tid 1 2024-09-13T14:20:50.815 INFO:tasks.rados.rados.0.smithi151.stdout:28: writing smithi15186934-28 from 1913602 to 2388604 tid 2 2024-09-13T14:20:50.815 INFO:tasks.rados.rados.0.smithi151.stdout:29: write initial oid 29 2024-09-13T14:20:50.815 INFO:tasks.rados.rados.0.smithi151.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2024-09-13T14:20:50.819 INFO:tasks.rados.rados.0.smithi151.stdout:29: writing smithi15186934-29 from 470596 to 926627 tid 1 2024-09-13T14:20:50.827 INFO:tasks.rados.rados.0.smithi151.stdout:29: writing smithi15186934-29 from 1572084 to 2190744 tid 2 2024-09-13T14:20:50.827 INFO:tasks.rados.rados.0.smithi151.stdout:30: write initial oid 30 2024-09-13T14:20:50.827 INFO:tasks.rados.rados.0.smithi151.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2024-09-13T14:20:50.832 INFO:tasks.rados.rados.0.smithi151.stdout:30: writing smithi15186934-30 from 487403 to 1018683 tid 1 2024-09-13T14:20:50.838 INFO:tasks.rados.rados.0.smithi151.stdout:30: writing smithi15186934-30 from 1514213 to 1992884 tid 2 2024-09-13T14:20:50.838 INFO:tasks.rados.rados.0.smithi151.stdout:31: write initial oid 31 2024-09-13T14:20:50.838 INFO:tasks.rados.rados.0.smithi151.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2024-09-13T14:20:50.844 INFO:tasks.rados.rados.0.smithi151.stdout:31: writing smithi15186934-31 from 504210 to 1110739 tid 1 2024-09-13T14:20:50.850 INFO:tasks.rados.rados.0.smithi151.stdout:31: writing smithi15186934-31 from 1572695 to 2195024 tid 2 2024-09-13T14:20:50.851 INFO:tasks.rados.rados.0.smithi151.stdout:32: write initial oid 32 2024-09-13T14:20:50.851 INFO:tasks.rados.rados.0.smithi151.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2024-09-13T14:20:50.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:50 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/1096909476' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2024-09-13T14:20:50.858 INFO:tasks.rados.rados.0.smithi151.stdout:32: writing smithi15186934-32 from 521017 to 1319148 tid 1 2024-09-13T14:20:50.863 INFO:tasks.rados.rados.0.smithi151.stdout:32: writing smithi15186934-32 from 1747530 to 2229870 tid 2 2024-09-13T14:20:50.863 INFO:tasks.rados.rados.0.smithi151.stdout:33: write initial oid 33 2024-09-13T14:20:50.863 INFO:tasks.rados.rados.0.smithi151.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2024-09-13T14:20:50.868 INFO:tasks.rados.rados.0.smithi151.stdout:33: writing smithi15186934-33 from 537824 to 1011204 tid 1 2024-09-13T14:20:50.876 INFO:tasks.rados.rados.0.smithi151.stdout:33: writing smithi15186934-33 from 1806012 to 2432010 tid 2 2024-09-13T14:20:50.876 INFO:tasks.rados.rados.0.smithi151.stdout:34: write initial oid 34 2024-09-13T14:20:50.876 INFO:tasks.rados.rados.0.smithi151.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2024-09-13T14:20:50.883 INFO:tasks.rados.rados.0.smithi151.stdout:34: writing smithi15186934-34 from 554631 to 1103260 tid 1 2024-09-13T14:20:50.889 INFO:tasks.rados.rados.0.smithi151.stdout:34: writing smithi15186934-34 from 1748141 to 2234150 tid 2 2024-09-13T14:20:50.889 INFO:tasks.rados.rados.0.smithi151.stdout:35: write initial oid 35 2024-09-13T14:20:50.889 INFO:tasks.rados.rados.0.smithi151.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2024-09-13T14:20:50.896 INFO:tasks.rados.rados.0.smithi151.stdout:35: writing smithi15186934-35 from 571438 to 1195316 tid 1 2024-09-13T14:20:50.903 INFO:tasks.rados.rados.0.smithi151.stdout:35: writing smithi15186934-35 from 1806623 to 2436290 tid 2 2024-09-13T14:20:50.903 INFO:tasks.rados.rados.0.smithi151.stdout: waiting on 16 2024-09-13T14:20:50.903 INFO:tasks.rados.rados.0.smithi151.stdout:20: finishing write tid 3 to smithi15186934-20 2024-09-13T14:20:50.903 INFO:tasks.rados.rados.0.smithi151.stdout:20: finishing write tid 4 to smithi15186934-20 2024-09-13T14:20:50.903 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 6 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-13T14:20:50.903 INFO:tasks.rados.rados.0.smithi151.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-09-13T14:20:50.903 INFO:tasks.rados.rados.0.smithi151.stdout:21: finishing write tid 1 to smithi15186934-21 2024-09-13T14:20:50.903 INFO:tasks.rados.rados.0.smithi151.stdout:21: finishing write tid 2 to smithi15186934-21 2024-09-13T14:20:50.904 INFO:tasks.rados.rados.0.smithi151.stdout:21: finishing write tid 3 to smithi15186934-21 2024-09-13T14:20:50.904 INFO:tasks.rados.rados.0.smithi151.stdout:20: done (15 left) 2024-09-13T14:20:50.904 INFO:tasks.rados.rados.0.smithi151.stdout:36: write initial oid 36 2024-09-13T14:20:50.904 INFO:tasks.rados.rados.0.smithi151.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2024-09-13T14:20:50.910 INFO:tasks.rados.rados.0.smithi151.stdout:36: writing smithi15186934-36 from 588245 to 1287372 tid 1 2024-09-13T14:20:50.916 INFO:tasks.rados.rados.0.smithi151.stdout:36: writing smithi15186934-36 from 1865105 to 2354783 tid 2 2024-09-13T14:20:50.917 INFO:tasks.rados.rados.0.smithi151.stdout: waiting on 16 2024-09-13T14:20:50.917 INFO:tasks.rados.rados.0.smithi151.stdout:21: finishing write tid 4 to smithi15186934-21 2024-09-13T14:20:50.917 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 21 v 3 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-13T14:20:50.917 INFO:tasks.rados.rados.0.smithi151.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-09-13T14:20:50.917 INFO:tasks.rados.rados.0.smithi151.stdout:22: finishing write tid 1 to smithi15186934-22 2024-09-13T14:20:50.917 INFO:tasks.rados.rados.0.smithi151.stdout:22: finishing write tid 2 to smithi15186934-22 2024-09-13T14:20:50.917 INFO:tasks.rados.rados.0.smithi151.stdout:22: finishing write tid 3 to smithi15186934-22 2024-09-13T14:20:50.917 INFO:tasks.rados.rados.0.smithi151.stdout:22: finishing write tid 4 to smithi15186934-22 2024-09-13T14:20:50.917 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 22 v 6 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-13T14:20:50.917 INFO:tasks.rados.rados.0.smithi151.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-09-13T14:20:50.917 INFO:tasks.rados.rados.0.smithi151.stdout:23: finishing write tid 1 to smithi15186934-23 2024-09-13T14:20:50.918 INFO:tasks.rados.rados.0.smithi151.stdout:23: finishing write tid 2 to smithi15186934-23 2024-09-13T14:20:50.918 INFO:tasks.rados.rados.0.smithi151.stdout:23: finishing write tid 3 to smithi15186934-23 2024-09-13T14:20:50.918 INFO:tasks.rados.rados.0.smithi151.stdout:23: finishing write tid 4 to smithi15186934-23 2024-09-13T14:20:50.918 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 23 v 3 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-13T14:20:50.918 INFO:tasks.rados.rados.0.smithi151.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-09-13T14:20:50.918 INFO:tasks.rados.rados.0.smithi151.stdout:24: finishing write tid 1 to smithi15186934-24 2024-09-13T14:20:50.918 INFO:tasks.rados.rados.0.smithi151.stdout:24: finishing write tid 2 to smithi15186934-24 2024-09-13T14:20:50.918 INFO:tasks.rados.rados.0.smithi151.stdout:24: finishing write tid 3 to smithi15186934-24 2024-09-13T14:20:50.918 INFO:tasks.rados.rados.0.smithi151.stdout:24: finishing write tid 4 to smithi15186934-24 2024-09-13T14:20:50.918 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 24 v 9 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-13T14:20:50.919 INFO:tasks.rados.rados.0.smithi151.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-09-13T14:20:50.919 INFO:tasks.rados.rados.0.smithi151.stdout:25: finishing write tid 1 to smithi15186934-25 2024-09-13T14:20:50.919 INFO:tasks.rados.rados.0.smithi151.stdout:25: finishing write tid 2 to smithi15186934-25 2024-09-13T14:20:50.919 INFO:tasks.rados.rados.0.smithi151.stdout:25: finishing write tid 3 to smithi15186934-25 2024-09-13T14:20:50.919 INFO:tasks.rados.rados.0.smithi151.stdout:25: finishing write tid 4 to smithi15186934-25 2024-09-13T14:20:50.919 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 6 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-13T14:20:50.919 INFO:tasks.rados.rados.0.smithi151.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-09-13T14:20:50.919 INFO:tasks.rados.rados.0.smithi151.stdout:26: finishing write tid 1 to smithi15186934-26 2024-09-13T14:20:50.919 INFO:tasks.rados.rados.0.smithi151.stdout:26: finishing write tid 2 to smithi15186934-26 2024-09-13T14:20:50.919 INFO:tasks.rados.rados.0.smithi151.stdout:26: finishing write tid 3 to smithi15186934-26 2024-09-13T14:20:50.919 INFO:tasks.rados.rados.0.smithi151.stdout:26: finishing write tid 4 to smithi15186934-26 2024-09-13T14:20:50.919 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-13T14:20:50.920 INFO:tasks.rados.rados.0.smithi151.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-09-13T14:20:50.920 INFO:tasks.rados.rados.0.smithi151.stdout:27: finishing write tid 1 to smithi15186934-27 2024-09-13T14:20:50.920 INFO:tasks.rados.rados.0.smithi151.stdout:21: done (15 left) 2024-09-13T14:20:50.920 INFO:tasks.rados.rados.0.smithi151.stdout:22: done (14 left) 2024-09-13T14:20:50.920 INFO:tasks.rados.rados.0.smithi151.stdout:23: done (13 left) 2024-09-13T14:20:50.920 INFO:tasks.rados.rados.0.smithi151.stdout:24: done (12 left) 2024-09-13T14:20:50.920 INFO:tasks.rados.rados.0.smithi151.stdout:25: done (11 left) 2024-09-13T14:20:50.920 INFO:tasks.rados.rados.0.smithi151.stdout:26: done (10 left) 2024-09-13T14:20:50.920 INFO:tasks.rados.rados.0.smithi151.stdout:37: write initial oid 37 2024-09-13T14:20:50.920 INFO:tasks.rados.rados.0.smithi151.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2024-09-13T14:20:50.925 INFO:tasks.rados.rados.0.smithi151.stdout:37: writing smithi15186934-37 from 605052 to 1379428 tid 1 2024-09-13T14:20:50.930 INFO:tasks.rados.rados.0.smithi151.stdout:37: writing smithi15186934-37 from 1923587 to 2556923 tid 2 2024-09-13T14:20:50.930 INFO:tasks.rados.rados.0.smithi151.stdout:38: write initial oid 38 2024-09-13T14:20:50.930 INFO:tasks.rados.rados.0.smithi151.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2024-09-13T14:20:50.936 INFO:tasks.rados.rados.0.smithi151.stdout:38: writing smithi15186934-38 from 621859 to 1071484 tid 1 2024-09-13T14:20:50.945 INFO:tasks.rados.rados.0.smithi151.stdout:38: writing smithi15186934-38 from 1865716 to 2621859 tid 2 2024-09-13T14:20:50.945 INFO:tasks.rados.rados.0.smithi151.stdout:39: write initial oid 39 2024-09-13T14:20:50.945 INFO:tasks.rados.rados.0.smithi151.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2024-09-13T14:20:50.951 INFO:tasks.rados.rados.0.smithi151.stdout:39: writing smithi15186934-39 from 638666 to 1163540 tid 1 2024-09-13T14:20:50.960 INFO:tasks.rados.rados.0.smithi151.stdout:39: writing smithi15186934-39 from 1924198 to 2561203 tid 2 2024-09-13T14:20:50.960 INFO:tasks.rados.rados.0.smithi151.stdout:40: write initial oid 40 2024-09-13T14:20:50.960 INFO:tasks.rados.rados.0.smithi151.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2024-09-13T14:20:50.967 INFO:tasks.rados.rados.0.smithi151.stdout:40: writing smithi15186934-40 from 655473 to 1371949 tid 1 2024-09-13T14:20:50.977 INFO:tasks.rados.rados.0.smithi151.stdout:40: writing smithi15186934-40 from 2099033 to 2655473 tid 2 2024-09-13T14:20:50.977 INFO:tasks.rados.rados.0.smithi151.stdout:41: write initial oid 41 2024-09-13T14:20:50.977 INFO:tasks.rados.rados.0.smithi151.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2024-09-13T14:20:50.983 INFO:tasks.rados.rados.0.smithi151.stdout:41: writing smithi15186934-41 from 672280 to 1464005 tid 1 2024-09-13T14:20:50.990 INFO:tasks.rados.rados.0.smithi151.stdout:41: writing smithi15186934-41 from 2157515 to 2672280 tid 2 2024-09-13T14:20:50.990 INFO:tasks.rados.rados.0.smithi151.stdout:42: write initial oid 42 2024-09-13T14:20:50.990 INFO:tasks.rados.rados.0.smithi151.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2024-09-13T14:20:50.996 INFO:tasks.rados.rados.0.smithi151.stdout:42: writing smithi15186934-42 from 689087 to 1156061 tid 1 2024-09-13T14:20:51.005 INFO:tasks.rados.rados.0.smithi151.stdout:42: writing smithi15186934-42 from 1699644 to 2483976 tid 2 2024-09-13T14:20:51.005 INFO:tasks.rados.rados.0.smithi151.stdout: waiting on 16 2024-09-13T14:20:51.005 INFO:tasks.rados.rados.0.smithi151.stdout:27: finishing write tid 2 to smithi15186934-27 2024-09-13T14:20:51.006 INFO:tasks.rados.rados.0.smithi151.stdout:27: finishing write tid 3 to smithi15186934-27 2024-09-13T14:20:51.006 INFO:tasks.rados.rados.0.smithi151.stdout:27: finishing write tid 4 to smithi15186934-27 2024-09-13T14:20:51.006 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 27 v 9 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-13T14:20:51.006 INFO:tasks.rados.rados.0.smithi151.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-09-13T14:20:51.006 INFO:tasks.rados.rados.0.smithi151.stdout:28: finishing write tid 1 to smithi15186934-28 2024-09-13T14:20:51.006 INFO:tasks.rados.rados.0.smithi151.stdout:27: done (15 left) 2024-09-13T14:20:51.007 INFO:tasks.rados.rados.0.smithi151.stdout:43: write initial oid 43 2024-09-13T14:20:51.007 INFO:tasks.rados.rados.0.smithi151.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2024-09-13T14:20:51.011 INFO:tasks.rados.rados.0.smithi151.stdout:43: writing smithi15186934-43 from 705894 to 1248117 tid 1 2024-09-13T14:20:51.017 INFO:tasks.rados.rados.0.smithi151.stdout:43: writing smithi15186934-43 from 1758126 to 2402469 tid 2 2024-09-13T14:20:51.017 INFO:tasks.rados.rados.0.smithi151.stdout: waiting on 16 2024-09-13T14:20:51.017 INFO:tasks.rados.rados.0.smithi151.stdout:28: finishing write tid 2 to smithi15186934-28 2024-09-13T14:20:51.018 INFO:tasks.rados.rados.0.smithi151.stdout:28: finishing write tid 3 to smithi15186934-28 2024-09-13T14:20:51.018 INFO:tasks.rados.rados.0.smithi151.stdout:28: finishing write tid 4 to smithi15186934-28 2024-09-13T14:20:51.018 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 28 v 6 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-13T14:20:51.018 INFO:tasks.rados.rados.0.smithi151.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-09-13T14:20:51.018 INFO:tasks.rados.rados.0.smithi151.stdout:29: finishing write tid 1 to smithi15186934-29 2024-09-13T14:20:51.018 INFO:tasks.rados.rados.0.smithi151.stdout:29: finishing write tid 2 to smithi15186934-29 2024-09-13T14:20:51.018 INFO:tasks.rados.rados.0.smithi151.stdout:29: finishing write tid 3 to smithi15186934-29 2024-09-13T14:20:51.018 INFO:tasks.rados.rados.0.smithi151.stdout:28: done (15 left) 2024-09-13T14:20:51.018 INFO:tasks.rados.rados.0.smithi151.stdout:44: write initial oid 44 2024-09-13T14:20:51.018 INFO:tasks.rados.rados.0.smithi151.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2024-09-13T14:20:51.024 INFO:tasks.rados.rados.0.smithi151.stdout:44: writing smithi15186934-44 from 722701 to 1340173 tid 1 2024-09-13T14:20:51.032 INFO:tasks.rados.rados.0.smithi151.stdout:44: writing smithi15186934-44 from 1816608 to 2604609 tid 2 2024-09-13T14:20:51.032 INFO:tasks.rados.rados.0.smithi151.stdout: waiting on 16 2024-09-13T14:20:51.032 INFO:tasks.rados.rados.0.smithi151.stdout:29: finishing write tid 4 to smithi15186934-29 2024-09-13T14:20:51.032 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 29 v 9 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-13T14:20:51.032 INFO:tasks.rados.rados.0.smithi151.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-09-13T14:20:51.033 INFO:tasks.rados.rados.0.smithi151.stdout:30: finishing write tid 1 to smithi15186934-30 2024-09-13T14:20:51.033 INFO:tasks.rados.rados.0.smithi151.stdout:30: finishing write tid 2 to smithi15186934-30 2024-09-13T14:20:51.033 INFO:tasks.rados.rados.0.smithi151.stdout:30: finishing write tid 3 to smithi15186934-30 2024-09-13T14:20:51.033 INFO:tasks.rados.rados.0.smithi151.stdout:30: finishing write tid 4 to smithi15186934-30 2024-09-13T14:20:51.033 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 30 v 6 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-13T14:20:51.033 INFO:tasks.rados.rados.0.smithi151.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-09-13T14:20:51.033 INFO:tasks.rados.rados.0.smithi151.stdout:31: finishing write tid 1 to smithi15186934-31 2024-09-13T14:20:51.033 INFO:tasks.rados.rados.0.smithi151.stdout:31: finishing write tid 2 to smithi15186934-31 2024-09-13T14:20:51.033 INFO:tasks.rados.rados.0.smithi151.stdout:29: done (15 left) 2024-09-13T14:20:51.033 INFO:tasks.rados.rados.0.smithi151.stdout:30: done (14 left) 2024-09-13T14:20:51.033 INFO:tasks.rados.rados.0.smithi151.stdout:45: write initial oid 45 2024-09-13T14:20:51.033 INFO:tasks.rados.rados.0.smithi151.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2024-09-13T14:20:51.040 INFO:tasks.rados.rados.0.smithi151.stdout:45: writing smithi15186934-45 from 739508 to 1432229 tid 1 2024-09-13T14:20:51.046 INFO:tasks.rados.rados.0.smithi151.stdout:45: writing smithi15186934-45 from 1875090 to 2523102 tid 2 2024-09-13T14:20:51.046 INFO:tasks.rados.rados.0.smithi151.stdout:46: write initial oid 46 2024-09-13T14:20:51.046 INFO:tasks.rados.rados.0.smithi151.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2024-09-13T14:20:51.054 INFO:tasks.rados.rados.0.smithi151.stdout:46: writing smithi15186934-46 from 756315 to 1524285 tid 1 2024-09-13T14:20:51.061 INFO:tasks.rados.rados.0.smithi151.stdout:46: writing smithi15186934-46 from 1933572 to 2725242 tid 2 2024-09-13T14:20:51.061 INFO:tasks.rados.rados.0.smithi151.stdout: waiting on 16 2024-09-13T14:20:51.061 INFO:tasks.rados.rados.0.smithi151.stdout:31: finishing write tid 3 to smithi15186934-31 2024-09-13T14:20:51.062 INFO:tasks.rados.rados.0.smithi151.stdout:31: finishing write tid 4 to smithi15186934-31 2024-09-13T14:20:51.062 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 31 v 9 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-13T14:20:51.062 INFO:tasks.rados.rados.0.smithi151.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-09-13T14:20:51.062 INFO:tasks.rados.rados.0.smithi151.stdout:32: finishing write tid 1 to smithi15186934-32 2024-09-13T14:20:51.062 INFO:tasks.rados.rados.0.smithi151.stdout:32: finishing write tid 2 to smithi15186934-32 2024-09-13T14:20:51.062 INFO:tasks.rados.rados.0.smithi151.stdout:31: done (15 left) 2024-09-13T14:20:51.062 INFO:tasks.rados.rados.0.smithi151.stdout:47: write initial oid 47 2024-09-13T14:20:51.062 INFO:tasks.rados.rados.0.smithi151.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2024-09-13T14:20:51.068 INFO:tasks.rados.rados.0.smithi151.stdout:47: writing smithi15186934-47 from 773122 to 1332694 tid 1 2024-09-13T14:20:51.076 INFO:tasks.rados.rados.0.smithi151.stdout:47: writing smithi15186934-47 from 1992054 to 2643735 tid 2 2024-09-13T14:20:51.076 INFO:tasks.rados.rados.0.smithi151.stdout: waiting on 16 2024-09-13T14:20:51.076 INFO:tasks.rados.rados.0.smithi151.stdout:32: finishing write tid 3 to smithi15186934-32 2024-09-13T14:20:51.076 INFO:tasks.rados.rados.0.smithi151.stdout:32: finishing write tid 4 to smithi15186934-32 2024-09-13T14:20:51.076 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 32 v 9 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-13T14:20:51.076 INFO:tasks.rados.rados.0.smithi151.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-09-13T14:20:51.076 INFO:tasks.rados.rados.0.smithi151.stdout:33: finishing write tid 1 to smithi15186934-33 2024-09-13T14:20:51.077 INFO:tasks.rados.rados.0.smithi151.stdout:33: finishing write tid 2 to smithi15186934-33 2024-09-13T14:20:51.077 INFO:tasks.rados.rados.0.smithi151.stdout:32: done (15 left) 2024-09-13T14:20:51.077 INFO:tasks.rados.rados.0.smithi151.stdout:48: write initial oid 48 2024-09-13T14:20:51.077 INFO:tasks.rados.rados.0.smithi151.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2024-09-13T14:20:51.083 INFO:tasks.rados.rados.0.smithi151.stdout:48: writing smithi15186934-48 from 789929 to 1424750 tid 1 2024-09-13T14:20:51.092 INFO:tasks.rados.rados.0.smithi151.stdout:48: writing smithi15186934-48 from 2050536 to 2789929 tid 2 2024-09-13T14:20:51.092 INFO:tasks.rados.rados.0.smithi151.stdout: waiting on 16 2024-09-13T14:20:51.092 INFO:tasks.rados.rados.0.smithi151.stdout:33: finishing write tid 3 to smithi15186934-33 2024-09-13T14:20:51.092 INFO:tasks.rados.rados.0.smithi151.stdout:33: finishing write tid 4 to smithi15186934-33 2024-09-13T14:20:51.092 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 33 v 12 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-13T14:20:51.092 INFO:tasks.rados.rados.0.smithi151.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-09-13T14:20:51.093 INFO:tasks.rados.rados.0.smithi151.stdout:34: finishing write tid 1 to smithi15186934-34 2024-09-13T14:20:51.093 INFO:tasks.rados.rados.0.smithi151.stdout:34: finishing write tid 2 to smithi15186934-34 2024-09-13T14:20:51.093 INFO:tasks.rados.rados.0.smithi151.stdout:34: finishing write tid 3 to smithi15186934-34 2024-09-13T14:20:51.093 INFO:tasks.rados.rados.0.smithi151.stdout:34: finishing write tid 4 to smithi15186934-34 2024-09-13T14:20:51.093 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 9 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-13T14:20:51.093 INFO:tasks.rados.rados.0.smithi151.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-09-13T14:20:51.093 INFO:tasks.rados.rados.0.smithi151.stdout:35: finishing write tid 1 to smithi15186934-35 2024-09-13T14:20:51.093 INFO:tasks.rados.rados.0.smithi151.stdout:33: done (15 left) 2024-09-13T14:20:51.093 INFO:tasks.rados.rados.0.smithi151.stdout:34: done (14 left) 2024-09-13T14:20:51.093 INFO:tasks.rados.rados.0.smithi151.stdout:49: write initial oid 49 2024-09-13T14:20:51.093 INFO:tasks.rados.rados.0.smithi151.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2024-09-13T14:20:51.099 INFO:tasks.rados.rados.0.smithi151.stdout:49: writing smithi15186934-49 from 406736 to 1116806 tid 1 2024-09-13T14:20:51.106 INFO:tasks.rados.rados.0.smithi151.stdout:49: writing smithi15186934-49 from 1709018 to 2364368 tid 2 2024-09-13T14:20:51.106 INFO:tasks.rados.rados.0.smithi151.stdout:50: write initial oid 50 2024-09-13T14:20:51.106 INFO:tasks.rados.rados.0.smithi151.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2024-09-13T14:20:51.112 INFO:tasks.rados.rados.0.smithi151.stdout:50: writing smithi15186934-50 from 423543 to 1208862 tid 1 2024-09-13T14:20:51.122 INFO:tasks.rados.rados.0.smithi151.stdout:50: writing smithi15186934-50 from 1767500 to 2566508 tid 2 2024-09-13T14:20:51.122 INFO:tasks.rados.rados.0.smithi151.stdout: waiting on 16 2024-09-13T14:20:51.122 INFO:tasks.rados.rados.0.smithi151.stdout:35: finishing write tid 2 to smithi15186934-35 2024-09-13T14:20:51.122 INFO:tasks.rados.rados.0.smithi151.stdout:35: finishing write tid 3 to smithi15186934-35 2024-09-13T14:20:51.122 INFO:tasks.rados.rados.0.smithi151.stdout:35: finishing write tid 4 to smithi15186934-35 2024-09-13T14:20:51.122 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 35 v 12 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-13T14:20:51.122 INFO:tasks.rados.rados.0.smithi151.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-09-13T14:20:51.123 INFO:tasks.rados.rados.0.smithi151.stdout:36: finishing write tid 1 to smithi15186934-36 2024-09-13T14:20:51.123 INFO:tasks.rados.rados.0.smithi151.stdout:35: done (15 left) 2024-09-13T14:20:51.123 INFO:tasks.rados.rados.0.smithi151.stdout:51: write oid 2 current snap is 0 2024-09-13T14:20:51.123 INFO:tasks.rados.rados.0.smithi151.stdout:36: finishing write tid 2 to smithi15186934-36 2024-09-13T14:20:51.123 INFO:tasks.rados.rados.0.smithi151.stdout:36: finishing write tid 3 to smithi15186934-36 2024-09-13T14:20:51.123 INFO:tasks.rados.rados.0.smithi151.stdout:36: finishing write tid 4 to smithi15186934-36 2024-09-13T14:20:51.123 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 36 v 6 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-13T14:20:51.123 INFO:tasks.rados.rados.0.smithi151.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-09-13T14:20:51.123 INFO:tasks.rados.rados.0.smithi151.stdout:37: finishing write tid 1 to smithi15186934-37 2024-09-13T14:20:51.123 INFO:tasks.rados.rados.0.smithi151.stdout:37: finishing write tid 2 to smithi15186934-37 2024-09-13T14:20:51.123 INFO:tasks.rados.rados.0.smithi151.stdout:37: finishing write tid 3 to smithi15186934-37 2024-09-13T14:20:51.124 INFO:tasks.rados.rados.0.smithi151.stdout:37: finishing write tid 4 to smithi15186934-37 2024-09-13T14:20:51.124 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 37 v 6 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-13T14:20:51.124 INFO:tasks.rados.rados.0.smithi151.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-09-13T14:20:51.124 INFO:tasks.rados.rados.0.smithi151.stdout:38: finishing write tid 1 to smithi15186934-38 2024-09-13T14:20:51.124 INFO:tasks.rados.rados.0.smithi151.stdout:38: finishing write tid 2 to smithi15186934-38 2024-09-13T14:20:51.124 INFO:tasks.rados.rados.0.smithi151.stdout:38: finishing write tid 3 to smithi15186934-38 2024-09-13T14:20:51.124 INFO:tasks.rados.rados.0.smithi151.stdout:38: finishing write tid 4 to smithi15186934-38 2024-09-13T14:20:51.124 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 3 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-13T14:20:51.124 INFO:tasks.rados.rados.0.smithi151.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-09-13T14:20:51.124 INFO:tasks.rados.rados.0.smithi151.stdout:39: finishing write tid 1 to smithi15186934-39 2024-09-13T14:20:51.124 INFO:tasks.rados.rados.0.smithi151.stdout:39: finishing write tid 2 to smithi15186934-39 2024-09-13T14:20:51.124 INFO:tasks.rados.rados.0.smithi151.stdout:39: finishing write tid 3 to smithi15186934-39 2024-09-13T14:20:51.124 INFO:tasks.rados.rados.0.smithi151.stdout:51: seq_num 50 ranges {440350=460568,1309629=542666,2323973=516377} 2024-09-13T14:20:51.126 INFO:tasks.rados.rados.0.smithi151.stdout:51: writing smithi15186934-2 from 440350 to 900918 tid 1 2024-09-13T14:20:51.131 INFO:tasks.rados.rados.0.smithi151.stdout:51: writing smithi15186934-2 from 1309629 to 1852295 tid 2 2024-09-13T14:20:51.137 INFO:tasks.rados.rados.0.smithi151.stdout:51: writing smithi15186934-2 from 2323973 to 2840350 tid 3 2024-09-13T14:20:51.137 INFO:tasks.rados.rados.0.smithi151.stdout:36: done (15 left) 2024-09-13T14:20:51.137 INFO:tasks.rados.rados.0.smithi151.stdout:37: done (14 left) 2024-09-13T14:20:51.137 INFO:tasks.rados.rados.0.smithi151.stdout:38: done (13 left) 2024-09-13T14:20:51.138 INFO:tasks.rados.rados.0.smithi151.stdout:52: rollback oid 18 current snap is 0 2024-09-13T14:20:51.138 INFO:tasks.rados.rados.0.smithi151.stdout:52: done (13 left) 2024-09-13T14:20:51.138 INFO:tasks.rados.rados.0.smithi151.stdout:53: write oid 8 current snap is 0 2024-09-13T14:20:51.138 INFO:tasks.rados.rados.0.smithi151.stdout:39: finishing write tid 4 to smithi15186934-39 2024-09-13T14:20:51.138 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 6 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-13T14:20:51.138 INFO:tasks.rados.rados.0.smithi151.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-09-13T14:20:51.138 INFO:tasks.rados.rados.0.smithi151.stdout:40: finishing write tid 1 to smithi15186934-40 2024-09-13T14:20:51.138 INFO:tasks.rados.rados.0.smithi151.stdout:53: seq_num 51 ranges {457157=535817,1768111=402677} 2024-09-13T14:20:51.142 INFO:tasks.rados.rados.0.smithi151.stdout:53: writing smithi15186934-8 from 457157 to 992974 tid 1 2024-09-13T14:20:51.149 INFO:tasks.rados.rados.0.smithi151.stdout:53: writing smithi15186934-8 from 1768111 to 2170788 tid 2 2024-09-13T14:20:51.149 INFO:tasks.rados.rados.0.smithi151.stdout:39: done (13 left) 2024-09-13T14:20:51.149 INFO:tasks.rados.rados.0.smithi151.stdout:54: write oid 24 current snap is 0 2024-09-13T14:20:51.149 INFO:tasks.rados.rados.0.smithi151.stdout:40: finishing write tid 2 to smithi15186934-40 2024-09-13T14:20:51.149 INFO:tasks.rados.rados.0.smithi151.stdout:40: finishing write tid 3 to smithi15186934-40 2024-09-13T14:20:51.149 INFO:tasks.rados.rados.0.smithi151.stdout:40: finishing write tid 4 to smithi15186934-40 2024-09-13T14:20:51.149 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 40 v 12 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-13T14:20:51.149 INFO:tasks.rados.rados.0.smithi151.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-09-13T14:20:51.149 INFO:tasks.rados.rados.0.smithi151.stdout:41: finishing write tid 1 to smithi15186934-41 2024-09-13T14:20:51.150 INFO:tasks.rados.rados.0.smithi151.stdout:41: finishing write tid 2 to smithi15186934-41 2024-09-13T14:20:51.150 INFO:tasks.rados.rados.0.smithi151.stdout:54: seq_num 52 ranges {473964=611066,1826593=546335,2778819=95145} 2024-09-13T14:20:51.155 INFO:tasks.rados.rados.0.smithi151.stdout:54: writing smithi15186934-24 from 473964 to 1085030 tid 1 2024-09-13T14:20:51.161 INFO:tasks.rados.rados.0.smithi151.stdout:54: writing smithi15186934-24 from 1826593 to 2372928 tid 2 2024-09-13T14:20:51.167 INFO:tasks.rados.rados.0.smithi151.stdout:54: writing smithi15186934-24 from 2778819 to 2873964 tid 3 2024-09-13T14:20:51.167 INFO:tasks.rados.rados.0.smithi151.stdout:40: done (13 left) 2024-09-13T14:20:51.167 INFO:tasks.rados.rados.0.smithi151.stdout:55: delete oid 4 current snap is 0 2024-09-13T14:20:51.167 INFO:tasks.rados.rados.0.smithi151.stdout:41: finishing write tid 3 to smithi15186934-41 2024-09-13T14:20:51.167 INFO:tasks.rados.rados.0.smithi151.stdout:41: finishing write tid 4 to smithi15186934-41 2024-09-13T14:20:51.167 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 3 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-13T14:20:51.167 INFO:tasks.rados.rados.0.smithi151.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-09-13T14:20:51.167 INFO:tasks.rados.rados.0.smithi151.stdout:42: finishing write tid 1 to smithi15186934-42 2024-09-13T14:20:51.167 INFO:tasks.rados.rados.0.smithi151.stdout:42: finishing write tid 2 to smithi15186934-42 2024-09-13T14:20:51.167 INFO:tasks.rados.rados.0.smithi151.stdout:42: finishing write tid 3 to smithi15186934-42 2024-09-13T14:20:51.167 INFO:tasks.rados.rados.0.smithi151.stdout:42: finishing write tid 4 to smithi15186934-42 2024-09-13T14:20:51.168 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 12 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-13T14:20:51.168 INFO:tasks.rados.rados.0.smithi151.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-09-13T14:20:51.168 INFO:tasks.rados.rados.0.smithi151.stdout:43: finishing write tid 1 to smithi15186934-43 2024-09-13T14:20:51.168 INFO:tasks.rados.rados.0.smithi151.stdout:43: finishing write tid 2 to smithi15186934-43 2024-09-13T14:20:51.168 INFO:tasks.rados.rados.0.smithi151.stdout:43: finishing write tid 3 to smithi15186934-43 2024-09-13T14:20:51.168 INFO:tasks.rados.rados.0.smithi151.stdout:43: finishing write tid 4 to smithi15186934-43 2024-09-13T14:20:51.168 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 43 v 15 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-13T14:20:51.168 INFO:tasks.rados.rados.0.smithi151.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-09-13T14:20:51.168 INFO:tasks.rados.rados.0.smithi151.stdout:44: finishing write tid 1 to smithi15186934-44 2024-09-13T14:20:51.168 INFO:tasks.rados.rados.0.smithi151.stdout:44: finishing write tid 2 to smithi15186934-44 2024-09-13T14:20:51.168 INFO:tasks.rados.rados.0.smithi151.stdout:44: finishing write tid 3 to smithi15186934-44 2024-09-13T14:20:51.168 INFO:tasks.rados.rados.0.smithi151.stdout:44: finishing write tid 4 to smithi15186934-44 2024-09-13T14:20:51.168 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 9 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-13T14:20:51.169 INFO:tasks.rados.rados.0.smithi151.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-09-13T14:20:51.169 INFO:tasks.rados.rados.0.smithi151.stdout:45: finishing write tid 1 to smithi15186934-45 2024-09-13T14:20:51.169 INFO:tasks.rados.rados.0.smithi151.stdout:45: finishing write tid 2 to smithi15186934-45 2024-09-13T14:20:51.169 INFO:tasks.rados.rados.0.smithi151.stdout:45: finishing write tid 3 to smithi15186934-45 2024-09-13T14:20:51.169 INFO:tasks.rados.rados.0.smithi151.stdout:45: finishing write tid 4 to smithi15186934-45 2024-09-13T14:20:51.169 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 9 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-13T14:20:51.169 INFO:tasks.rados.rados.0.smithi151.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-09-13T14:20:51.169 INFO:tasks.rados.rados.0.smithi151.stdout:46: finishing write tid 1 to smithi15186934-46 2024-09-13T14:20:51.169 INFO:tasks.rados.rados.0.smithi151.stdout:46: finishing write tid 2 to smithi15186934-46 2024-09-13T14:20:51.169 INFO:tasks.rados.rados.0.smithi151.stdout:46: finishing write tid 3 to smithi15186934-46 2024-09-13T14:20:51.169 INFO:tasks.rados.rados.0.smithi151.stdout:46: finishing write tid 4 to smithi15186934-46 2024-09-13T14:20:51.169 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 46 v 9 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-13T14:20:51.169 INFO:tasks.rados.rados.0.smithi151.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-09-13T14:20:51.169 INFO:tasks.rados.rados.0.smithi151.stdout:47: finishing write tid 1 to smithi15186934-47 2024-09-13T14:20:51.169 INFO:tasks.rados.rados.0.smithi151.stdout:47: finishing write tid 2 to smithi15186934-47 2024-09-13T14:20:51.170 INFO:tasks.rados.rados.0.smithi151.stdout:47: finishing write tid 3 to smithi15186934-47 2024-09-13T14:20:51.170 INFO:tasks.rados.rados.0.smithi151.stdout:47: finishing write tid 4 to smithi15186934-47 2024-09-13T14:20:51.170 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 9 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-13T14:20:51.170 INFO:tasks.rados.rados.0.smithi151.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-09-13T14:20:51.170 INFO:tasks.rados.rados.0.smithi151.stdout:48: finishing write tid 1 to smithi15186934-48 2024-09-13T14:20:51.170 INFO:tasks.rados.rados.0.smithi151.stdout:48: finishing write tid 2 to smithi15186934-48 2024-09-13T14:20:51.170 INFO:tasks.rados.rados.0.smithi151.stdout:48: finishing write tid 3 to smithi15186934-48 2024-09-13T14:20:51.170 INFO:tasks.rados.rados.0.smithi151.stdout:48: finishing write tid 4 to smithi15186934-48 2024-09-13T14:20:51.170 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 48 v 12 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-13T14:20:51.170 INFO:tasks.rados.rados.0.smithi151.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-09-13T14:20:51.170 INFO:tasks.rados.rados.0.smithi151.stdout:49: finishing write tid 1 to smithi15186934-49 2024-09-13T14:20:51.170 INFO:tasks.rados.rados.0.smithi151.stdout:49: finishing write tid 2 to smithi15186934-49 2024-09-13T14:20:51.170 INFO:tasks.rados.rados.0.smithi151.stdout:49: finishing write tid 3 to smithi15186934-49 2024-09-13T14:20:51.170 INFO:tasks.rados.rados.0.smithi151.stdout:49: finishing write tid 4 to smithi15186934-49 2024-09-13T14:20:51.171 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 49 v 12 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-13T14:20:51.171 INFO:tasks.rados.rados.0.smithi151.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-09-13T14:20:51.171 INFO:tasks.rados.rados.0.smithi151.stdout:50: finishing write tid 1 to smithi15186934-50 2024-09-13T14:20:51.171 INFO:tasks.rados.rados.0.smithi151.stdout:50: finishing write tid 2 to smithi15186934-50 2024-09-13T14:20:51.171 INFO:tasks.rados.rados.0.smithi151.stdout:50: finishing write tid 3 to smithi15186934-50 2024-09-13T14:20:51.171 INFO:tasks.rados.rados.0.smithi151.stdout:50: finishing write tid 4 to smithi15186934-50 2024-09-13T14:20:51.171 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 50 v 9 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-13T14:20:51.171 INFO:tasks.rados.rados.0.smithi151.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-09-13T14:20:51.171 INFO:tasks.rados.rados.0.smithi151.stdout:51: finishing write tid 1 to smithi15186934-2 2024-09-13T14:20:51.171 INFO:tasks.rados.rados.0.smithi151.stdout:51: finishing write tid 2 to smithi15186934-2 2024-09-13T14:20:51.171 INFO:tasks.rados.rados.0.smithi151.stdout:51: finishing write tid 3 to smithi15186934-2 2024-09-13T14:20:51.171 INFO:tasks.rados.rados.0.smithi151.stdout:51: finishing write tid 4 to smithi15186934-2 2024-09-13T14:20:51.171 INFO:tasks.rados.rados.0.smithi151.stdout:51: finishing write tid 5 to smithi15186934-2 2024-09-13T14:20:51.171 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 2 v 16 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-13T14:20:51.171 INFO:tasks.rados.rados.0.smithi151.stdout:51: left oid 2 (ObjNum 50 snap 0 seq_num 50) 2024-09-13T14:20:51.172 INFO:tasks.rados.rados.0.smithi151.stdout:53: finishing write tid 1 to smithi15186934-8 2024-09-13T14:20:51.172 INFO:tasks.rados.rados.0.smithi151.stdout:53: finishing write tid 2 to smithi15186934-8 2024-09-13T14:20:51.172 INFO:tasks.rados.rados.0.smithi151.stdout:53: finishing write tid 3 to smithi15186934-8 2024-09-13T14:20:51.172 INFO:tasks.rados.rados.0.smithi151.stdout:53: finishing write tid 4 to smithi15186934-8 2024-09-13T14:20:51.172 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 12 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-09-13T14:20:51.172 INFO:tasks.rados.rados.0.smithi151.stdout:53: left oid 8 (ObjNum 51 snap 0 seq_num 51) 2024-09-13T14:20:51.172 INFO:tasks.rados.rados.0.smithi151.stdout:54: finishing write tid 1 to smithi15186934-24 2024-09-13T14:20:51.172 INFO:tasks.rados.rados.0.smithi151.stdout:54: finishing write tid 2 to smithi15186934-24 2024-09-13T14:20:51.172 INFO:tasks.rados.rados.0.smithi151.stdout:41: done (13 left) 2024-09-13T14:20:51.172 INFO:tasks.rados.rados.0.smithi151.stdout:42: done (12 left) 2024-09-13T14:20:51.172 INFO:tasks.rados.rados.0.smithi151.stdout:43: done (11 left) 2024-09-13T14:20:51.172 INFO:tasks.rados.rados.0.smithi151.stdout:44: done (10 left) 2024-09-13T14:20:51.172 INFO:tasks.rados.rados.0.smithi151.stdout:45: done (9 left) 2024-09-13T14:20:51.172 INFO:tasks.rados.rados.0.smithi151.stdout:46: done (8 left) 2024-09-13T14:20:51.173 INFO:tasks.rados.rados.0.smithi151.stdout:47: done (7 left) 2024-09-13T14:20:51.173 INFO:tasks.rados.rados.0.smithi151.stdout:48: done (6 left) 2024-09-13T14:20:51.173 INFO:tasks.rados.rados.0.smithi151.stdout:49: done (5 left) 2024-09-13T14:20:51.173 INFO:tasks.rados.rados.0.smithi151.stdout:50: done (4 left) 2024-09-13T14:20:51.173 INFO:tasks.rados.rados.0.smithi151.stdout:51: done (3 left) 2024-09-13T14:20:51.173 INFO:tasks.rados.rados.0.smithi151.stdout:53: done (2 left) 2024-09-13T14:20:51.173 INFO:tasks.rados.rados.0.smithi151.stdout:55: done (1 left) 2024-09-13T14:20:51.173 INFO:tasks.rados.rados.0.smithi151.stdout:56: read oid 15 snap -1 2024-09-13T14:20:51.173 INFO:tasks.rados.rados.0.smithi151.stdout:56: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-13T14:20:51.173 INFO:tasks.rados.rados.0.smithi151.stdout:57: delete oid 13 current snap is 0 2024-09-13T14:20:51.173 INFO:tasks.rados.rados.0.smithi151.stdout:54: finishing write tid 3 to smithi15186934-24 2024-09-13T14:20:51.173 INFO:tasks.rados.rados.0.smithi151.stdout:54: finishing write tid 4 to smithi15186934-24 2024-09-13T14:20:51.173 INFO:tasks.rados.rados.0.smithi151.stdout:54: finishing write tid 5 to smithi15186934-24 2024-09-13T14:20:51.173 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 24 v 19 (ObjNum 52 snap 0 seq_num 52) dirty exists 2024-09-13T14:20:51.174 INFO:tasks.rados.rados.0.smithi151.stdout:54: left oid 24 (ObjNum 52 snap 0 seq_num 52) 2024-09-13T14:20:51.174 INFO:tasks.rados.rados.0.smithi151.stdout:54: done (2 left) 2024-09-13T14:20:51.174 INFO:tasks.rados.rados.0.smithi151.stdout:57: done (1 left) 2024-09-13T14:20:51.174 INFO:tasks.rados.rados.0.smithi151.stdout:58: read oid 3 snap -1 2024-09-13T14:20:51.174 INFO:tasks.rados.rados.0.smithi151.stdout:58: expect (ObjNum 2 snap 0 seq_num 2) 2024-09-13T14:20:51.174 INFO:tasks.rados.rados.0.smithi151.stdout:59: read oid 28 snap -1 2024-09-13T14:20:51.174 INFO:tasks.rados.rados.0.smithi151.stdout:59: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-13T14:20:51.174 INFO:tasks.rados.rados.0.smithi151.stdout:60: read oid 34 snap -1 2024-09-13T14:20:51.174 INFO:tasks.rados.rados.0.smithi151.stdout:60: expect (ObjNum 33 snap 0 seq_num 33) 2024-09-13T14:20:51.174 INFO:tasks.rados.rados.0.smithi151.stdout:61: delete oid 2 current snap is 0 2024-09-13T14:20:51.175 INFO:tasks.rados.rados.0.smithi151.stdout:61: done (4 left) 2024-09-13T14:20:51.175 INFO:tasks.rados.rados.0.smithi151.stdout:62: write oid 7 current snap is 0 2024-09-13T14:20:51.175 INFO:tasks.rados.rados.0.smithi151.stdout:62: seq_num 56 ranges {541192=628415,1660521=553673,2888511=52681} 2024-09-13T14:20:51.175 INFO:tasks.rados.rados.0.smithi151.stdout:62: writing smithi15186934-7 from 541192 to 1169607 tid 1 2024-09-13T14:20:51.181 INFO:tasks.rados.rados.0.smithi151.stdout:62: writing smithi15186934-7 from 1660521 to 2214194 tid 2 2024-09-13T14:20:51.185 INFO:tasks.rados.rados.0.smithi151.stdout:62: writing smithi15186934-7 from 2888511 to 2941192 tid 3 2024-09-13T14:20:51.185 INFO:tasks.rados.rados.0.smithi151.stdout:63: write oid 23 current snap is 0 2024-09-13T14:20:51.185 INFO:tasks.rados.rados.0.smithi151.stdout:63: seq_num 57 ranges {557999=703664,1719003=413684,2632287=325712} 2024-09-13T14:20:51.192 INFO:tasks.rados.rados.0.smithi151.stdout:63: writing smithi15186934-23 from 557999 to 1261663 tid 1 2024-09-13T14:20:51.197 INFO:tasks.rados.rados.0.smithi151.stdout:63: writing smithi15186934-23 from 1719003 to 2132687 tid 2 2024-09-13T14:20:51.202 INFO:tasks.rados.rados.0.smithi151.stdout:63: writing smithi15186934-23 from 2632287 to 2957999 tid 3 2024-09-13T14:20:51.202 INFO:tasks.rados.rados.0.smithi151.stdout:64: write oid 38 current snap is 0 2024-09-13T14:20:51.203 INFO:tasks.rados.rados.0.smithi151.stdout:64: seq_num 58 ranges {574806=778913,2061132=557342} 2024-09-13T14:20:51.210 INFO:tasks.rados.rados.0.smithi151.stdout:64: writing smithi15186934-38 from 574806 to 1353719 tid 1 2024-09-13T14:20:51.217 INFO:tasks.rados.rados.0.smithi151.stdout:64: writing smithi15186934-38 from 2061132 to 2618474 tid 2 2024-09-13T14:20:51.217 INFO:tasks.rados.rados.0.smithi151.stdout:65: rollback oid 25 current snap is 0 2024-09-13T14:20:51.218 INFO:tasks.rados.rados.0.smithi151.stdout:65: done (7 left) 2024-09-13T14:20:51.218 INFO:tasks.rados.rados.0.smithi151.stdout:66: read oid 29 snap -1 2024-09-13T14:20:51.218 INFO:tasks.rados.rados.0.smithi151.stdout:66: expect (ObjNum 28 snap 0 seq_num 28) 2024-09-13T14:20:51.218 INFO:tasks.rados.rados.0.smithi151.stdout:67: write oid 44 current snap is 0 2024-09-13T14:20:51.218 INFO:tasks.rados.rados.0.smithi151.stdout:56: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-13T14:20:51.296 INFO:tasks.rados.rados.0.smithi151.stdout:58: expect (ObjNum 2 snap 0 seq_num 2) 2024-09-13T14:20:51.364 INFO:tasks.rados.rados.0.smithi151.stdout:59: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-13T14:20:51.447 INFO:tasks.rados.rados.0.smithi151.stdout:60: expect (ObjNum 33 snap 0 seq_num 33) 2024-09-13T14:20:51.539 INFO:tasks.rados.rados.0.smithi151.stdout:62: finishing write tid 1 to smithi15186934-7 2024-09-13T14:20:51.539 INFO:tasks.rados.rados.0.smithi151.stdout:62: finishing write tid 2 to smithi15186934-7 2024-09-13T14:20:51.539 INFO:tasks.rados.rados.0.smithi151.stdout:67: seq_num 59 ranges {591613=454162,1719614=417353,2570780=420833} 2024-09-13T14:20:51.544 INFO:tasks.rados.rados.0.smithi151.stdout:67: writing smithi15186934-44 from 591613 to 1045775 tid 1 2024-09-13T14:20:51.550 INFO:tasks.rados.rados.0.smithi151.stdout:67: writing smithi15186934-44 from 1719614 to 2136967 tid 2 2024-09-13T14:20:51.555 INFO:tasks.rados.rados.0.smithi151.stdout:67: writing smithi15186934-44 from 2570780 to 2991613 tid 3 2024-09-13T14:20:51.555 INFO:tasks.rados.rados.0.smithi151.stdout:56: done (8 left) 2024-09-13T14:20:51.556 INFO:tasks.rados.rados.0.smithi151.stdout:58: done (7 left) 2024-09-13T14:20:51.556 INFO:tasks.rados.rados.0.smithi151.stdout:59: done (6 left) 2024-09-13T14:20:51.557 INFO:tasks.rados.rados.0.smithi151.stdout:60: done (5 left) 2024-09-13T14:20:51.557 INFO:tasks.rados.rados.0.smithi151.stdout:68: write oid 22 current snap is 0 2024-09-13T14:20:51.557 INFO:tasks.rados.rados.0.smithi151.stdout:68: seq_num 60 ranges {608420=529411,1778096=561011,2881850=126570} 2024-09-13T14:20:51.561 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'echo wait for minority of mons to upgrade' 2024-09-13T14:20:51.563 INFO:tasks.rados.rados.0.smithi151.stdout:68: writing smithi15186934-22 from 608420 to 1137831 tid 1 2024-09-13T14:20:51.569 INFO:tasks.rados.rados.0.smithi151.stdout:68: writing smithi15186934-22 from 1778096 to 2339107 tid 2 2024-09-13T14:20:51.573 INFO:tasks.rados.rados.0.smithi151.stdout:68: writing smithi15186934-22 from 2881850 to 3008420 tid 3 2024-09-13T14:20:51.573 INFO:tasks.rados.rados.0.smithi151.stdout:69: write oid 19 current snap is 0 2024-09-13T14:20:51.573 INFO:tasks.rados.rados.0.smithi151.stdout:62: finishing write tid 3 to smithi15186934-7 2024-09-13T14:20:51.573 INFO:tasks.rados.rados.0.smithi151.stdout:62: finishing write tid 4 to smithi15186934-7 2024-09-13T14:20:51.573 INFO:tasks.rados.rados.0.smithi151.stdout:69: seq_num 61 ranges {625227=721013,1952931=704669} 2024-09-13T14:20:51.581 INFO:tasks.rados.rados.0.smithi151.stdout:69: writing smithi15186934-19 from 625227 to 1346240 tid 1 2024-09-13T14:20:51.589 INFO:tasks.rados.rados.0.smithi151.stdout:69: writing smithi15186934-19 from 1952931 to 2657600 tid 2 2024-09-13T14:20:51.590 INFO:tasks.rados.rados.0.smithi151.stdout:70: write oid 42 current snap is 0 2024-09-13T14:20:51.590 INFO:tasks.rados.rados.0.smithi151.stdout:62: finishing write tid 5 to smithi15186934-7 2024-09-13T14:20:51.590 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 23 (ObjNum 56 snap 0 seq_num 56) dirty exists 2024-09-13T14:20:51.590 INFO:tasks.rados.rados.0.smithi151.stdout:62: left oid 7 (ObjNum 56 snap 0 seq_num 56) 2024-09-13T14:20:51.590 INFO:tasks.rados.rados.0.smithi151.stdout:63: finishing write tid 1 to smithi15186934-23 2024-09-13T14:20:51.590 INFO:tasks.rados.rados.0.smithi151.stdout:63: finishing write tid 2 to smithi15186934-23 2024-09-13T14:20:51.590 INFO:tasks.rados.rados.0.smithi151.stdout:63: finishing write tid 3 to smithi15186934-23 2024-09-13T14:20:51.590 INFO:tasks.rados.rados.0.smithi151.stdout:63: finishing write tid 4 to smithi15186934-23 2024-09-13T14:20:51.590 INFO:tasks.rados.rados.0.smithi151.stdout:63: finishing write tid 5 to smithi15186934-23 2024-09-13T14:20:51.590 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 23 v 16 (ObjNum 57 snap 0 seq_num 57) dirty exists 2024-09-13T14:20:51.590 INFO:tasks.rados.rados.0.smithi151.stdout:63: left oid 23 (ObjNum 57 snap 0 seq_num 57) 2024-09-13T14:20:51.590 INFO:tasks.rados.rados.0.smithi151.stdout:64: finishing write tid 1 to smithi15186934-38 2024-09-13T14:20:51.590 INFO:tasks.rados.rados.0.smithi151.stdout:70: seq_num 62 ranges {642034=796262,1895060=564680} 2024-09-13T14:20:51.597 INFO:tasks.rados.rados.0.smithi151.stdout:70: writing smithi15186934-42 from 642034 to 1438296 tid 1 2024-09-13T14:20:51.604 INFO:tasks.rados.rados.0.smithi151.stdout:70: writing smithi15186934-42 from 1895060 to 2459740 tid 2 2024-09-13T14:20:51.604 INFO:tasks.rados.rados.0.smithi151.stdout:62: done (7 left) 2024-09-13T14:20:51.604 INFO:tasks.rados.rados.0.smithi151.stdout:63: done (6 left) 2024-09-13T14:20:51.604 INFO:tasks.rados.rados.0.smithi151.stdout:71: snap_create 2024-09-13T14:20:51.604 INFO:tasks.rados.rados.0.smithi151.stdout:64: finishing write tid 2 to smithi15186934-38 2024-09-13T14:20:51.604 INFO:tasks.rados.rados.0.smithi151.stdout:64: finishing write tid 3 to smithi15186934-38 2024-09-13T14:20:51.604 INFO:tasks.rados.rados.0.smithi151.stdout:64: finishing write tid 4 to smithi15186934-38 2024-09-13T14:20:51.604 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 12 (ObjNum 58 snap 0 seq_num 58) dirty exists 2024-09-13T14:20:51.604 INFO:tasks.rados.rados.0.smithi151.stdout:64: left oid 38 (ObjNum 58 snap 0 seq_num 58) 2024-09-13T14:20:51.604 INFO:tasks.rados.rados.0.smithi151.stdout:66: expect (ObjNum 28 snap 0 seq_num 28) 2024-09-13T14:20:51.691 INFO:tasks.rados.rados.0.smithi151.stdout:67: finishing write tid 1 to smithi15186934-44 2024-09-13T14:20:51.691 INFO:tasks.rados.rados.0.smithi151.stdout:67: finishing write tid 2 to smithi15186934-44 2024-09-13T14:20:51.691 INFO:tasks.rados.rados.0.smithi151.stdout:67: finishing write tid 3 to smithi15186934-44 2024-09-13T14:20:51.691 INFO:tasks.rados.rados.0.smithi151.stdout:67: finishing write tid 4 to smithi15186934-44 2024-09-13T14:20:51.691 INFO:tasks.rados.rados.0.smithi151.stdout:67: finishing write tid 5 to smithi15186934-44 2024-09-13T14:20:51.692 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 16 (ObjNum 59 snap 0 seq_num 59) dirty exists 2024-09-13T14:20:51.692 INFO:tasks.rados.rados.0.smithi151.stdout:67: left oid 44 (ObjNum 59 snap 0 seq_num 59) 2024-09-13T14:20:51.692 INFO:tasks.rados.rados.0.smithi151.stdout:68: finishing write tid 1 to smithi15186934-22 2024-09-13T14:20:51.692 INFO:tasks.rados.rados.0.smithi151.stdout:68: finishing write tid 2 to smithi15186934-22 2024-09-13T14:20:51.692 INFO:tasks.rados.rados.0.smithi151.stdout:68: finishing write tid 3 to smithi15186934-22 2024-09-13T14:20:51.692 INFO:tasks.rados.rados.0.smithi151.stdout:68: finishing write tid 4 to smithi15186934-22 2024-09-13T14:20:51.692 INFO:tasks.rados.rados.0.smithi151.stdout:68: finishing write tid 5 to smithi15186934-22 2024-09-13T14:20:51.692 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 22 v 10 (ObjNum 60 snap 0 seq_num 60) dirty exists 2024-09-13T14:20:51.692 INFO:tasks.rados.rados.0.smithi151.stdout:68: left oid 22 (ObjNum 60 snap 0 seq_num 60) 2024-09-13T14:20:51.692 INFO:tasks.rados.rados.0.smithi151.stdout:69: finishing write tid 1 to smithi15186934-19 2024-09-13T14:20:51.692 INFO:tasks.rados.rados.0.smithi151.stdout:69: finishing write tid 2 to smithi15186934-19 2024-09-13T14:20:51.692 INFO:tasks.rados.rados.0.smithi151.stdout:69: finishing write tid 3 to smithi15186934-19 2024-09-13T14:20:51.692 INFO:tasks.rados.rados.0.smithi151.stdout:69: finishing write tid 4 to smithi15186934-19 2024-09-13T14:20:51.693 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 19 v 9 (ObjNum 61 snap 0 seq_num 61) dirty exists 2024-09-13T14:20:51.693 INFO:tasks.rados.rados.0.smithi151.stdout:69: left oid 19 (ObjNum 61 snap 0 seq_num 61) 2024-09-13T14:20:51.693 INFO:tasks.rados.rados.0.smithi151.stdout:70: finishing write tid 1 to smithi15186934-42 2024-09-13T14:20:51.693 INFO:tasks.rados.rados.0.smithi151.stdout:70: finishing write tid 2 to smithi15186934-42 2024-09-13T14:20:51.693 INFO:tasks.rados.rados.0.smithi151.stdout:70: finishing write tid 3 to smithi15186934-42 2024-09-13T14:20:51.693 INFO:tasks.rados.rados.0.smithi151.stdout:70: finishing write tid 4 to smithi15186934-42 2024-09-13T14:20:51.693 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 15 (ObjNum 62 snap 0 seq_num 62) dirty exists 2024-09-13T14:20:51.693 INFO:tasks.rados.rados.0.smithi151.stdout:70: left oid 42 (ObjNum 62 snap 0 seq_num 62) 2024-09-13T14:20:51.829 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:51.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:51 smithi184 ceph-mon[33394]: pgmap v91: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:51.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:51 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/1096909476' entity='client.0' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2024-09-13T14:20:51.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:51 smithi184 ceph-mon[33394]: osdmap e70: 8 total, 8 up, 8 in 2024-09-13T14:20:51.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:51 smithi184 ceph-mon[33394]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:51.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:51 smithi151 ceph-mon[29807]: pgmap v91: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:51.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:51 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1096909476' entity='client.0' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2024-09-13T14:20:51.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:51 smithi151 ceph-mon[29807]: osdmap e70: 8 total, 8 up, 8 in 2024-09-13T14:20:51.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:51 smithi151 ceph-mon[29807]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:51.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:51 smithi151 ceph-mon[41387]: pgmap v91: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:51.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:51 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/1096909476' entity='client.0' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2024-09-13T14:20:51.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:51 smithi151 ceph-mon[41387]: osdmap e70: 8 total, 8 up, 8 in 2024-09-13T14:20:51.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:51 smithi151 ceph-mon[41387]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:52.468 INFO:tasks.rados.rados.0.smithi151.stdout:64: done (6 left) 2024-09-13T14:20:52.468 INFO:tasks.rados.rados.0.smithi151.stdout:66: done (5 left) 2024-09-13T14:20:52.468 INFO:tasks.rados.rados.0.smithi151.stdout:67: done (4 left) 2024-09-13T14:20:52.468 INFO:tasks.rados.rados.0.smithi151.stdout:68: done (3 left) 2024-09-13T14:20:52.468 INFO:tasks.rados.rados.0.smithi151.stdout:69: done (2 left) 2024-09-13T14:20:52.468 INFO:tasks.rados.rados.0.smithi151.stdout:70: done (1 left) 2024-09-13T14:20:52.468 INFO:tasks.rados.rados.0.smithi151.stdout:71: done (0 left) 2024-09-13T14:20:52.468 INFO:tasks.rados.rados.0.smithi151.stdout:72: write oid 50 current snap is 1 2024-09-13T14:20:52.469 INFO:tasks.rados.rados.0.smithi151.stdout:72: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2024-09-13T14:20:52.474 INFO:tasks.rados.rados.0.smithi151.stdout:72: writing smithi15186934-50 from 658841 to 1130352 tid 1 2024-09-13T14:20:52.480 INFO:tasks.rados.rados.0.smithi151.stdout:72: writing smithi15186934-50 from 1553542 to 2261880 tid 2 2024-09-13T14:20:52.485 INFO:tasks.rados.rados.0.smithi151.stdout:72: writing smithi15186934-50 from 2964119 to 3058841 tid 3 2024-09-13T14:20:52.485 INFO:tasks.rados.rados.0.smithi151.stdout:73: rollback oid 38 current snap is 1 2024-09-13T14:20:52.485 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 38 to 0 2024-09-13T14:20:52.486 INFO:tasks.rados.rados.0.smithi151.stdout:72: finishing write tid 1 to smithi15186934-50 2024-09-13T14:20:52.486 INFO:tasks.rados.rados.0.smithi151.stdout:72: finishing write tid 2 to smithi15186934-50 2024-09-13T14:20:52.486 INFO:tasks.rados.rados.0.smithi151.stdout:74: write oid 33 current snap is 1 2024-09-13T14:20:52.486 INFO:tasks.rados.rados.0.smithi151.stdout:74: seq_num 64 ranges {675648=546760,2012024=568349} 2024-09-13T14:20:52.492 INFO:tasks.rados.rados.0.smithi151.stdout:74: writing smithi15186934-33 from 675648 to 1222408 tid 1 2024-09-13T14:20:52.500 INFO:tasks.rados.rados.0.smithi151.stdout:74: writing smithi15186934-33 from 2012024 to 2580373 tid 2 2024-09-13T14:20:52.500 INFO:tasks.rados.rados.0.smithi151.stdout:75: rollback oid 37 current snap is 1 2024-09-13T14:20:52.500 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 37 to 0 2024-09-13T14:20:52.501 INFO:tasks.rados.rados.0.smithi151.stdout:72: finishing write tid 3 to smithi15186934-50 2024-09-13T14:20:52.501 INFO:tasks.rados.rados.0.smithi151.stdout:72: finishing write tid 4 to smithi15186934-50 2024-09-13T14:20:52.501 INFO:tasks.rados.rados.0.smithi151.stdout:72: finishing write tid 5 to smithi15186934-50 2024-09-13T14:20:52.501 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 50 v 17 (ObjNum 63 snap 1 seq_num 63) dirty exists 2024-09-13T14:20:52.501 INFO:tasks.rados.rados.0.smithi151.stdout:72: left oid 50 (ObjNum 63 snap 1 seq_num 63) 2024-09-13T14:20:52.501 INFO:tasks.rados.rados.0.smithi151.stdout:73: finishing rollback tid 0 to smithi15186934-38 2024-09-13T14:20:52.501 INFO:tasks.rados.rados.0.smithi151.stdout:73: finishing rollback tid 1 to smithi15186934-38 2024-09-13T14:20:52.501 INFO:tasks.rados.rados.0.smithi151.stdout:73: finishing rollback tid 2 to smithi15186934-38 2024-09-13T14:20:52.501 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 21 (ObjNum 58 snap 0 seq_num 58) dirty exists 2024-09-13T14:20:52.501 INFO:tasks.rados.rados.0.smithi151.stdout:74: finishing write tid 1 to smithi15186934-33 2024-09-13T14:20:52.501 INFO:tasks.rados.rados.0.smithi151.stdout:72: done (3 left) 2024-09-13T14:20:52.501 INFO:tasks.rados.rados.0.smithi151.stdout:73: done (2 left) 2024-09-13T14:20:52.501 INFO:tasks.rados.rados.0.smithi151.stdout:76: snap_create 2024-09-13T14:20:52.509 INFO:tasks.rados.rados.0.smithi151.stdout:74: finishing write tid 2 to smithi15186934-33 2024-09-13T14:20:52.509 INFO:tasks.rados.rados.0.smithi151.stdout:74: finishing write tid 3 to smithi15186934-33 2024-09-13T14:20:52.509 INFO:tasks.rados.rados.0.smithi151.stdout:74: finishing write tid 4 to smithi15186934-33 2024-09-13T14:20:52.509 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 33 v 22 (ObjNum 64 snap 1 seq_num 64) dirty exists 2024-09-13T14:20:52.509 INFO:tasks.rados.rados.0.smithi151.stdout:74: left oid 33 (ObjNum 64 snap 1 seq_num 64) 2024-09-13T14:20:52.510 INFO:tasks.rados.rados.0.smithi151.stdout:75: finishing rollback tid 0 to smithi15186934-37 2024-09-13T14:20:52.510 INFO:tasks.rados.rados.0.smithi151.stdout:75: finishing rollback tid 1 to smithi15186934-37 2024-09-13T14:20:52.510 INFO:tasks.rados.rados.0.smithi151.stdout:75: finishing rollback tid 2 to smithi15186934-37 2024-09-13T14:20:52.511 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 37 v 20 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-13T14:20:53.306 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:20:52 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:20:52] "GET /metrics HTTP/1.1" 200 242179 "" "Prometheus/2.43.0" 2024-09-13T14:20:53.474 INFO:tasks.rados.rados.0.smithi151.stdout:74: done (2 left) 2024-09-13T14:20:53.474 INFO:tasks.rados.rados.0.smithi151.stdout:75: done (1 left) 2024-09-13T14:20:53.474 INFO:tasks.rados.rados.0.smithi151.stdout:76: done (0 left) 2024-09-13T14:20:53.474 INFO:tasks.rados.rados.0.smithi151.stdout:77: read oid 33 snap -1 2024-09-13T14:20:53.474 INFO:tasks.rados.rados.0.smithi151.stdout:77: expect (ObjNum 64 snap 1 seq_num 64) 2024-09-13T14:20:53.474 INFO:tasks.rados.rados.0.smithi151.stdout:78: snap_create 2024-09-13T14:20:53.483 INFO:tasks.rados.rados.0.smithi151.stdout:77: expect (ObjNum 64 snap 1 seq_num 64) 2024-09-13T14:20:53.537 INFO:teuthology.orchestra.run.smithi151.stdout:wait for minority of mons to upgrade 2024-09-13T14:20:53.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:53 smithi151 ceph-mon[29807]: pgmap v93: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:53.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:53 smithi151 ceph-mon[29807]: osdmap e71: 8 total, 8 up, 8 in 2024-09-13T14:20:53.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:53 smithi151 ceph-mon[41387]: pgmap v93: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:53.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:53 smithi151 ceph-mon[41387]: osdmap e71: 8 total, 8 up, 8 in 2024-09-13T14:20:53.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:53 smithi184 ceph-mon[33394]: pgmap v93: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:53.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:53 smithi184 ceph-mon[33394]: osdmap e71: 8 total, 8 up, 8 in 2024-09-13T14:20:54.357 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'while ! ceph mon versions | grep $sha1 && ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error; do sleep 2 ; done' 2024-09-13T14:20:54.484 INFO:tasks.rados.rados.0.smithi151.stdout:77: done (1 left) 2024-09-13T14:20:54.485 INFO:tasks.rados.rados.0.smithi151.stdout:78: done (0 left) 2024-09-13T14:20:54.485 INFO:tasks.rados.rados.0.smithi151.stdout:79: rollback oid 9 current snap is 3 2024-09-13T14:20:54.485 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 9 to 0 2024-09-13T14:20:54.485 INFO:tasks.rados.rados.0.smithi151.stdout:80: snap_create 2024-09-13T14:20:54.486 INFO:tasks.rados.rados.0.smithi151.stdout:79: finishing rollback tid 0 to smithi15186934-9 2024-09-13T14:20:54.487 INFO:tasks.rados.rados.0.smithi151.stdout:79: finishing rollback tid 1 to smithi15186934-9 2024-09-13T14:20:54.487 INFO:tasks.rados.rados.0.smithi151.stdout:79: finishing rollback tid 2 to smithi15186934-9 2024-09-13T14:20:54.487 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 13 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-13T14:20:54.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:54 smithi151 ceph-mon[29807]: osdmap e72: 8 total, 8 up, 8 in 2024-09-13T14:20:54.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:54 smithi151 ceph-mon[41387]: osdmap e72: 8 total, 8 up, 8 in 2024-09-13T14:20:54.653 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:54.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:54 smithi184 ceph-mon[33394]: osdmap e72: 8 total, 8 up, 8 in 2024-09-13T14:20:55.483 INFO:tasks.rados.rados.0.smithi151.stdout:79: done (1 left) 2024-09-13T14:20:55.484 INFO:tasks.rados.rados.0.smithi151.stdout:80: done (0 left) 2024-09-13T14:20:55.484 INFO:tasks.rados.rados.0.smithi151.stdout:81: read oid 26 snap -1 2024-09-13T14:20:55.484 INFO:tasks.rados.rados.0.smithi151.stdout:81: expect (ObjNum 25 snap 0 seq_num 25) 2024-09-13T14:20:55.484 INFO:tasks.rados.rados.0.smithi151.stdout:82: read oid 36 snap -1 2024-09-13T14:20:55.484 INFO:tasks.rados.rados.0.smithi151.stdout:82: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-13T14:20:55.484 INFO:tasks.rados.rados.0.smithi151.stdout:83: snap_create 2024-09-13T14:20:55.490 INFO:tasks.rados.rados.0.smithi151.stdout:81: expect (ObjNum 25 snap 0 seq_num 25) 2024-09-13T14:20:55.574 INFO:tasks.rados.rados.0.smithi151.stdout:82: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-13T14:20:55.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:55 smithi184 ceph-mon[33394]: pgmap v96: 89 pgs: 89 active+clean; 117 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.1 MiB/s rd, 11 MiB/s wr, 70 op/s 2024-09-13T14:20:55.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:55 smithi184 ceph-mon[33394]: osdmap e73: 8 total, 8 up, 8 in 2024-09-13T14:20:55.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:55 smithi151 ceph-mon[29807]: pgmap v96: 89 pgs: 89 active+clean; 117 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.1 MiB/s rd, 11 MiB/s wr, 70 op/s 2024-09-13T14:20:55.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:55 smithi151 ceph-mon[29807]: osdmap e73: 8 total, 8 up, 8 in 2024-09-13T14:20:55.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:55 smithi151 ceph-mon[41387]: pgmap v96: 89 pgs: 89 active+clean; 117 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.1 MiB/s rd, 11 MiB/s wr, 70 op/s 2024-09-13T14:20:55.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:55 smithi151 ceph-mon[41387]: osdmap e73: 8 total, 8 up, 8 in 2024-09-13T14:20:56.486 INFO:tasks.rados.rados.0.smithi151.stdout:81: done (2 left) 2024-09-13T14:20:56.486 INFO:tasks.rados.rados.0.smithi151.stdout:82: done (1 left) 2024-09-13T14:20:56.486 INFO:tasks.rados.rados.0.smithi151.stdout:83: done (0 left) 2024-09-13T14:20:56.486 INFO:tasks.rados.rados.0.smithi151.stdout:84: delete oid 7 current snap is 5 2024-09-13T14:20:56.488 INFO:tasks.rados.rados.0.smithi151.stdout:84: done (0 left) 2024-09-13T14:20:56.488 INFO:tasks.rados.rados.0.smithi151.stdout:85: write oid 48 current snap is 5 2024-09-13T14:20:56.489 INFO:tasks.rados.rados.0.smithi151.stdout:85: seq_num 66 ranges {709262=697258,2012635=572018,3046388=62874} 2024-09-13T14:20:56.497 INFO:tasks.rados.rados.0.smithi151.stdout:85: writing smithi15186934-48 from 709262 to 1406520 tid 1 2024-09-13T14:20:56.504 INFO:tasks.rados.rados.0.smithi151.stdout:85: writing smithi15186934-48 from 2012635 to 2584653 tid 2 2024-09-13T14:20:56.507 INFO:tasks.rados.rados.0.smithi151.stdout:85: writing smithi15186934-48 from 3046388 to 3109262 tid 3 2024-09-13T14:20:56.507 INFO:tasks.rados.rados.0.smithi151.stdout:86: snap_create 2024-09-13T14:20:56.507 INFO:tasks.rados.rados.0.smithi151.stdout:85: finishing write tid 1 to smithi15186934-48 2024-09-13T14:20:56.507 INFO:tasks.rados.rados.0.smithi151.stdout:85: finishing write tid 2 to smithi15186934-48 2024-09-13T14:20:56.508 INFO:tasks.rados.rados.0.smithi151.stdout:85: finishing write tid 3 to smithi15186934-48 2024-09-13T14:20:56.508 INFO:tasks.rados.rados.0.smithi151.stdout:85: finishing write tid 4 to smithi15186934-48 2024-09-13T14:20:56.509 INFO:tasks.rados.rados.0.smithi151.stdout:85: finishing write tid 5 to smithi15186934-48 2024-09-13T14:20:56.509 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 48 v 18 (ObjNum 66 snap 5 seq_num 66) dirty exists 2024-09-13T14:20:56.509 INFO:tasks.rados.rados.0.smithi151.stdout:85: left oid 48 (ObjNum 66 snap 5 seq_num 66) 2024-09-13T14:20:56.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:56 smithi151 ceph-mon[29807]: osdmap e74: 8 total, 8 up, 8 in 2024-09-13T14:20:56.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:56 smithi151 ceph-mon[41387]: osdmap e74: 8 total, 8 up, 8 in 2024-09-13T14:20:56.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:56 smithi184 ceph-mon[33394]: osdmap e74: 8 total, 8 up, 8 in 2024-09-13T14:20:57.022 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:20:57.503 INFO:tasks.rados.rados.0.smithi151.stdout:85: done (1 left) 2024-09-13T14:20:57.503 INFO:tasks.rados.rados.0.smithi151.stdout:86: done (0 left) 2024-09-13T14:20:57.503 INFO:tasks.rados.rados.0.smithi151.stdout:87: rollback oid 10 current snap is 6 2024-09-13T14:20:57.503 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 10 to 2 2024-09-13T14:20:57.503 INFO:tasks.rados.rados.0.smithi151.stdout:88: snap_create 2024-09-13T14:20:57.506 INFO:tasks.rados.rados.0.smithi151.stdout:87: finishing rollback tid 0 to smithi15186934-10 2024-09-13T14:20:57.507 INFO:tasks.rados.rados.0.smithi151.stdout:87: finishing rollback tid 1 to smithi15186934-10 2024-09-13T14:20:57.507 INFO:tasks.rados.rados.0.smithi151.stdout:87: finishing rollback tid 2 to smithi15186934-10 2024-09-13T14:20:57.507 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 13 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-13T14:20:57.635 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:57 smithi151 ceph-mon[29807]: pgmap v99: 89 pgs: 89 active+clean; 117 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 17 MiB/s wr, 106 op/s 2024-09-13T14:20:57.636 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:57 smithi151 ceph-mon[29807]: osdmap e75: 8 total, 8 up, 8 in 2024-09-13T14:20:57.636 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:57 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/793036308' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:20:57.636 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:57 smithi151 ceph-mon[41387]: pgmap v99: 89 pgs: 89 active+clean; 117 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 17 MiB/s wr, 106 op/s 2024-09-13T14:20:57.636 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:57 smithi151 ceph-mon[41387]: osdmap e75: 8 total, 8 up, 8 in 2024-09-13T14:20:57.636 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:57 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/793036308' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:20:57.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:57 smithi184 ceph-mon[33394]: pgmap v99: 89 pgs: 89 active+clean; 117 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 17 MiB/s wr, 106 op/s 2024-09-13T14:20:57.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:57 smithi184 ceph-mon[33394]: osdmap e75: 8 total, 8 up, 8 in 2024-09-13T14:20:57.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:57 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/793036308' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:20:57.854 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:20:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:20:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:20:58.510 INFO:tasks.rados.rados.0.smithi151.stdout:87: done (1 left) 2024-09-13T14:20:58.510 INFO:tasks.rados.rados.0.smithi151.stdout:88: done (0 left) 2024-09-13T14:20:58.510 INFO:tasks.rados.rados.0.smithi151.stdout:89: delete oid 23 current snap is 7 2024-09-13T14:20:58.514 INFO:tasks.rados.rados.0.smithi151.stdout:89: done (0 left) 2024-09-13T14:20:58.514 INFO:tasks.rados.rados.0.smithi151.stdout:90: read oid 27 snap 3 2024-09-13T14:20:58.514 INFO:tasks.rados.rados.0.smithi151.stdout:90: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-13T14:20:58.514 INFO:tasks.rados.rados.0.smithi151.stdout:91: write oid 12 current snap is 7 2024-09-13T14:20:58.514 INFO:tasks.rados.rados.0.smithi151.stdout:91: seq_num 68 ranges {742876=447756,1729599=575687,3101234=41642} 2024-09-13T14:20:58.521 INFO:tasks.rados.rados.0.smithi151.stdout:91: writing smithi15186934-12 from 742876 to 1190632 tid 1 2024-09-13T14:20:58.527 INFO:tasks.rados.rados.0.smithi151.stdout:91: writing smithi15186934-12 from 1729599 to 2305286 tid 2 2024-09-13T14:20:58.533 INFO:tasks.rados.rados.0.smithi151.stdout:91: writing smithi15186934-12 from 3101234 to 3142876 tid 3 2024-09-13T14:20:58.533 INFO:tasks.rados.rados.0.smithi151.stdout:92: snap_create 2024-09-13T14:20:58.533 INFO:tasks.rados.rados.0.smithi151.stdout:91: finishing write tid 1 to smithi15186934-12 2024-09-13T14:20:58.533 INFO:tasks.rados.rados.0.smithi151.stdout:90: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-13T14:20:58.617 INFO:tasks.rados.rados.0.smithi151.stdout:91: finishing write tid 2 to smithi15186934-12 2024-09-13T14:20:58.618 INFO:tasks.rados.rados.0.smithi151.stdout:91: finishing write tid 3 to smithi15186934-12 2024-09-13T14:20:58.618 INFO:tasks.rados.rados.0.smithi151.stdout:91: finishing write tid 4 to smithi15186934-12 2024-09-13T14:20:58.618 INFO:tasks.rados.rados.0.smithi151.stdout:91: finishing write tid 5 to smithi15186934-12 2024-09-13T14:20:58.618 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 12 v 11 (ObjNum 68 snap 7 seq_num 68) dirty exists 2024-09-13T14:20:58.618 INFO:tasks.rados.rados.0.smithi151.stdout:91: left oid 12 (ObjNum 68 snap 7 seq_num 68) 2024-09-13T14:20:58.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:58 smithi184 ceph-mon[33394]: from='client.14838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:58.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:58 smithi184 ceph-mon[33394]: from='client.24746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:58.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:58 smithi184 ceph-mon[33394]: osdmap e76: 8 total, 8 up, 8 in 2024-09-13T14:20:58.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:58 smithi151 ceph-mon[29807]: from='client.14838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:58.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:58 smithi151 ceph-mon[29807]: from='client.24746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:58.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:58 smithi151 ceph-mon[29807]: osdmap e76: 8 total, 8 up, 8 in 2024-09-13T14:20:58.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:58 smithi151 ceph-mon[41387]: from='client.14838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:58.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:58 smithi151 ceph-mon[41387]: from='client.24746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:58.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:58 smithi151 ceph-mon[41387]: osdmap e76: 8 total, 8 up, 8 in 2024-09-13T14:20:59.512 INFO:tasks.rados.rados.0.smithi151.stdout:90: done (2 left) 2024-09-13T14:20:59.512 INFO:tasks.rados.rados.0.smithi151.stdout:91: done (1 left) 2024-09-13T14:20:59.512 INFO:tasks.rados.rados.0.smithi151.stdout:92: done (0 left) 2024-09-13T14:20:59.513 INFO:tasks.rados.rados.0.smithi151.stdout:93: snap_remove snap 7 2024-09-13T14:20:59.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:59 smithi184 ceph-mon[33394]: pgmap v102: 89 pgs: 89 active+clean; 117 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:59.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:20:59 smithi184 ceph-mon[33394]: osdmap e77: 8 total, 8 up, 8 in 2024-09-13T14:20:59.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:59 smithi151 ceph-mon[41387]: pgmap v102: 89 pgs: 89 active+clean; 117 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:59.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:20:59 smithi151 ceph-mon[41387]: osdmap e77: 8 total, 8 up, 8 in 2024-09-13T14:20:59.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:59 smithi151 ceph-mon[29807]: pgmap v102: 89 pgs: 89 active+clean; 117 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:20:59.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:20:59 smithi151 ceph-mon[29807]: osdmap e77: 8 total, 8 up, 8 in 2024-09-13T14:20:59.956 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:21:00.519 INFO:tasks.rados.rados.0.smithi151.stdout:93: done (0 left) 2024-09-13T14:21:00.519 INFO:tasks.rados.rados.0.smithi151.stdout:94: write oid 15 current snap is 8 2024-09-13T14:21:00.519 INFO:tasks.rados.rados.0.smithi151.stdout:94: seq_num 69 ranges {759683=639358,1904434=719345,3128657=31026} 2024-09-13T14:21:00.527 INFO:tasks.rados.rados.0.smithi151.stdout:94: writing smithi15186934-15 from 759683 to 1399041 tid 1 2024-09-13T14:21:00.535 INFO:tasks.rados.rados.0.smithi151.stdout:94: writing smithi15186934-15 from 1904434 to 2623779 tid 2 2024-09-13T14:21:00.538 INFO:tasks.rados.rados.0.smithi151.stdout:94: writing smithi15186934-15 from 3128657 to 3159683 tid 3 2024-09-13T14:21:00.538 INFO:tasks.rados.rados.0.smithi151.stdout:94: finishing write tid 1 to smithi15186934-15 2024-09-13T14:21:00.538 INFO:tasks.rados.rados.0.smithi151.stdout:95: snap_create 2024-09-13T14:21:00.540 INFO:tasks.rados.rados.0.smithi151.stdout:94: finishing write tid 2 to smithi15186934-15 2024-09-13T14:21:00.540 INFO:tasks.rados.rados.0.smithi151.stdout:94: finishing write tid 3 to smithi15186934-15 2024-09-13T14:21:00.540 INFO:tasks.rados.rados.0.smithi151.stdout:94: finishing write tid 4 to smithi15186934-15 2024-09-13T14:21:00.540 INFO:tasks.rados.rados.0.smithi151.stdout:94: finishing write tid 5 to smithi15186934-15 2024-09-13T14:21:00.541 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 15 v 20 (ObjNum 69 snap 8 seq_num 69) dirty exists 2024-09-13T14:21:00.541 INFO:tasks.rados.rados.0.smithi151.stdout:94: left oid 15 (ObjNum 69 snap 8 seq_num 69) 2024-09-13T14:21:00.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:00 smithi184 ceph-mon[33394]: osdmap e78: 8 total, 8 up, 8 in 2024-09-13T14:21:00.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:00 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/1670507648' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:00.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:00 smithi151 ceph-mon[41387]: osdmap e78: 8 total, 8 up, 8 in 2024-09-13T14:21:00.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:00 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/1670507648' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:00.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:00 smithi151 ceph-mon[29807]: osdmap e78: 8 total, 8 up, 8 in 2024-09-13T14:21:00.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:00 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1670507648' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:01.532 INFO:tasks.rados.rados.0.smithi151.stdout:94: done (1 left) 2024-09-13T14:21:01.532 INFO:tasks.rados.rados.0.smithi151.stdout:95: done (0 left) 2024-09-13T14:21:01.532 INFO:tasks.rados.rados.0.smithi151.stdout:96: rollback oid 42 current snap is 9 2024-09-13T14:21:01.532 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 42 to 5 2024-09-13T14:21:01.532 INFO:tasks.rados.rados.0.smithi151.stdout:97: write oid 3 current snap is 9 2024-09-13T14:21:01.532 INFO:tasks.rados.rados.0.smithi151.stdout:97: seq_num 70 ranges {776490=714607,2246563=579356} 2024-09-13T14:21:01.540 INFO:tasks.rados.rados.0.smithi151.stdout:97: writing smithi15186934-3 from 776490 to 1491097 tid 1 2024-09-13T14:21:01.548 INFO:tasks.rados.rados.0.smithi151.stdout:97: writing smithi15186934-3 from 2246563 to 2825919 tid 2 2024-09-13T14:21:01.548 INFO:tasks.rados.rados.0.smithi151.stdout:98: delete oid 4 current snap is 9 2024-09-13T14:21:01.548 INFO:tasks.rados.rados.0.smithi151.stdout:96: finishing rollback tid 0 to smithi15186934-42 2024-09-13T14:21:01.548 INFO:tasks.rados.rados.0.smithi151.stdout:96: finishing rollback tid 1 to smithi15186934-42 2024-09-13T14:21:01.548 INFO:tasks.rados.rados.0.smithi151.stdout:96: finishing rollback tid 2 to smithi15186934-42 2024-09-13T14:21:01.548 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 24 (ObjNum 62 snap 0 seq_num 62) dirty exists 2024-09-13T14:21:01.549 INFO:tasks.rados.rados.0.smithi151.stdout:97: finishing write tid 1 to smithi15186934-3 2024-09-13T14:21:01.552 INFO:tasks.rados.rados.0.smithi151.stdout:97: finishing write tid 2 to smithi15186934-3 2024-09-13T14:21:01.552 INFO:tasks.rados.rados.0.smithi151.stdout:97: finishing write tid 3 to smithi15186934-3 2024-09-13T14:21:01.552 INFO:tasks.rados.rados.0.smithi151.stdout:97: finishing write tid 4 to smithi15186934-3 2024-09-13T14:21:01.552 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 16 (ObjNum 70 snap 9 seq_num 70) dirty exists 2024-09-13T14:21:01.552 INFO:tasks.rados.rados.0.smithi151.stdout:97: left oid 3 (ObjNum 70 snap 9 seq_num 70) 2024-09-13T14:21:01.552 INFO:tasks.rados.rados.0.smithi151.stdout:96: done (2 left) 2024-09-13T14:21:01.552 INFO:tasks.rados.rados.0.smithi151.stdout:97: done (1 left) 2024-09-13T14:21:01.553 INFO:tasks.rados.rados.0.smithi151.stdout:98: done (0 left) 2024-09-13T14:21:01.553 INFO:tasks.rados.rados.0.smithi151.stdout:99: read oid 39 snap -1 2024-09-13T14:21:01.553 INFO:tasks.rados.rados.0.smithi151.stdout:99: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-13T14:21:01.553 INFO:tasks.rados.rados.0.smithi151.stdout:100: write oid 6 current snap is 9 2024-09-13T14:21:01.553 INFO:tasks.rados.rados.0.smithi151.stdout:100: seq_num 72 ranges {410104=465105,1563527=583025,2810926=399178} 2024-09-13T14:21:01.556 INFO:tasks.rados.rados.0.smithi151.stdout:100: writing smithi15186934-6 from 410104 to 875209 tid 1 2024-09-13T14:21:01.563 INFO:tasks.rados.rados.0.smithi151.stdout:100: writing smithi15186934-6 from 1563527 to 2146552 tid 2 2024-09-13T14:21:01.569 INFO:tasks.rados.rados.0.smithi151.stdout:100: writing smithi15186934-6 from 2810926 to 3210104 tid 3 2024-09-13T14:21:01.569 INFO:tasks.rados.rados.0.smithi151.stdout:101: snap_remove snap 4 2024-09-13T14:21:01.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:01 smithi184 ceph-mon[33394]: from='client.24758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:01.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:01 smithi184 ceph-mon[33394]: from='client.14862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:01.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:01 smithi184 ceph-mon[33394]: pgmap v105: 89 pgs: 89 active+clean; 121 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 586 KiB/s wr, 10 op/s 2024-09-13T14:21:01.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:01 smithi184 ceph-mon[33394]: osdmap e79: 8 total, 8 up, 8 in 2024-09-13T14:21:01.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:01 smithi151 ceph-mon[41387]: from='client.24758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:01.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:01 smithi151 ceph-mon[41387]: from='client.14862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:01.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:01 smithi151 ceph-mon[41387]: pgmap v105: 89 pgs: 89 active+clean; 121 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 586 KiB/s wr, 10 op/s 2024-09-13T14:21:01.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:01 smithi151 ceph-mon[41387]: osdmap e79: 8 total, 8 up, 8 in 2024-09-13T14:21:01.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:01 smithi151 ceph-mon[29807]: from='client.24758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:01.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:01 smithi151 ceph-mon[29807]: from='client.14862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:01.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:01 smithi151 ceph-mon[29807]: pgmap v105: 89 pgs: 89 active+clean; 121 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 586 KiB/s wr, 10 op/s 2024-09-13T14:21:01.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:01 smithi151 ceph-mon[29807]: osdmap e79: 8 total, 8 up, 8 in 2024-09-13T14:21:02.541 INFO:tasks.rados.rados.0.smithi151.stdout:101: done (2 left) 2024-09-13T14:21:02.541 INFO:tasks.rados.rados.0.smithi151.stdout:102: write oid 43 current snap is 9 2024-09-13T14:21:02.541 INFO:tasks.rados.rados.0.smithi151.stdout:102: seq_num 73 ranges {426911=540354,1622009=726683,3121996=104915} 2024-09-13T14:21:02.544 INFO:tasks.rados.rados.0.smithi151.stdout:102: writing smithi15186934-43 from 426911 to 967265 tid 1 2024-09-13T14:21:02.552 INFO:tasks.rados.rados.0.smithi151.stdout:102: writing smithi15186934-43 from 1622009 to 2348692 tid 2 2024-09-13T14:21:02.557 INFO:tasks.rados.rados.0.smithi151.stdout:102: writing smithi15186934-43 from 3121996 to 3226911 tid 3 2024-09-13T14:21:02.557 INFO:tasks.rados.rados.0.smithi151.stdout:103: snap_remove snap 5 2024-09-13T14:21:02.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:02 smithi184 ceph-mon[33394]: osdmap e80: 8 total, 8 up, 8 in 2024-09-13T14:21:02.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:02 smithi151 ceph-mon[29807]: osdmap e80: 8 total, 8 up, 8 in 2024-09-13T14:21:02.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:02 smithi151 ceph-mon[41387]: osdmap e80: 8 total, 8 up, 8 in 2024-09-13T14:21:02.901 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:21:03.358 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:21:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:21:02] "GET /metrics HTTP/1.1" 200 242179 "" "Prometheus/2.43.0" 2024-09-13T14:21:03.550 INFO:tasks.rados.rados.0.smithi151.stdout:103: done (3 left) 2024-09-13T14:21:03.551 INFO:tasks.rados.rados.0.smithi151.stdout:104: delete oid 14 current snap is 9 2024-09-13T14:21:03.551 INFO:tasks.rados.rados.0.smithi151.stdout:100: finishing write tid 1 to smithi15186934-6 2024-09-13T14:21:03.551 INFO:tasks.rados.rados.0.smithi151.stdout:99: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-13T14:21:03.650 INFO:tasks.rados.rados.0.smithi151.stdout:100: finishing write tid 2 to smithi15186934-6 2024-09-13T14:21:03.650 INFO:tasks.rados.rados.0.smithi151.stdout:100: finishing write tid 3 to smithi15186934-6 2024-09-13T14:21:03.650 INFO:tasks.rados.rados.0.smithi151.stdout:100: finishing write tid 4 to smithi15186934-6 2024-09-13T14:21:03.650 INFO:tasks.rados.rados.0.smithi151.stdout:100: finishing write tid 5 to smithi15186934-6 2024-09-13T14:21:03.650 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 6 v 14 (ObjNum 72 snap 9 seq_num 72) dirty exists 2024-09-13T14:21:03.650 INFO:tasks.rados.rados.0.smithi151.stdout:100: left oid 6 (ObjNum 72 snap 9 seq_num 72) 2024-09-13T14:21:03.650 INFO:tasks.rados.rados.0.smithi151.stdout:102: finishing write tid 1 to smithi15186934-43 2024-09-13T14:21:03.650 INFO:tasks.rados.rados.0.smithi151.stdout:102: finishing write tid 2 to smithi15186934-43 2024-09-13T14:21:03.650 INFO:tasks.rados.rados.0.smithi151.stdout:102: finishing write tid 3 to smithi15186934-43 2024-09-13T14:21:03.650 INFO:tasks.rados.rados.0.smithi151.stdout:102: finishing write tid 4 to smithi15186934-43 2024-09-13T14:21:03.650 INFO:tasks.rados.rados.0.smithi151.stdout:102: finishing write tid 5 to smithi15186934-43 2024-09-13T14:21:03.651 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 43 v 32 (ObjNum 73 snap 9 seq_num 73) dirty exists 2024-09-13T14:21:03.651 INFO:tasks.rados.rados.0.smithi151.stdout:102: left oid 43 (ObjNum 73 snap 9 seq_num 73) 2024-09-13T14:21:03.651 INFO:tasks.rados.rados.0.smithi151.stdout:99: done (3 left) 2024-09-13T14:21:03.651 INFO:tasks.rados.rados.0.smithi151.stdout:100: done (2 left) 2024-09-13T14:21:03.651 INFO:tasks.rados.rados.0.smithi151.stdout:102: done (1 left) 2024-09-13T14:21:03.651 INFO:tasks.rados.rados.0.smithi151.stdout:104: done (0 left) 2024-09-13T14:21:03.651 INFO:tasks.rados.rados.0.smithi151.stdout:105: write oid 14 current snap is 9 2024-09-13T14:21:03.651 INFO:tasks.rados.rados.0.smithi151.stdout:105: seq_num 75 ranges {460525=690852,1622620=730352,3060489=200036} 2024-09-13T14:21:03.655 INFO:tasks.rados.rados.0.smithi151.stdout:105: writing smithi15186934-14 from 460525 to 1151377 tid 1 2024-09-13T14:21:03.663 INFO:tasks.rados.rados.0.smithi151.stdout:105: writing smithi15186934-14 from 1622620 to 2352972 tid 2 2024-09-13T14:21:03.668 INFO:tasks.rados.rados.0.smithi151.stdout:105: writing smithi15186934-14 from 3060489 to 3260525 tid 3 2024-09-13T14:21:03.668 INFO:tasks.rados.rados.0.smithi151.stdout:106: rollback oid 30 current snap is 9 2024-09-13T14:21:03.668 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 30 to 2 2024-09-13T14:21:03.668 INFO:tasks.rados.rados.0.smithi151.stdout:105: finishing write tid 1 to smithi15186934-14 2024-09-13T14:21:03.669 INFO:tasks.rados.rados.0.smithi151.stdout:105: finishing write tid 2 to smithi15186934-14 2024-09-13T14:21:03.669 INFO:tasks.rados.rados.0.smithi151.stdout:107: delete oid 10 current snap is 9 2024-09-13T14:21:03.671 INFO:tasks.rados.rados.0.smithi151.stdout:105: finishing write tid 3 to smithi15186934-14 2024-09-13T14:21:03.671 INFO:tasks.rados.rados.0.smithi151.stdout:105: finishing write tid 4 to smithi15186934-14 2024-09-13T14:21:03.671 INFO:tasks.rados.rados.0.smithi151.stdout:107: done (2 left) 2024-09-13T14:21:03.671 INFO:tasks.rados.rados.0.smithi151.stdout:108: rollback oid 37 current snap is 9 2024-09-13T14:21:03.671 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 37 to 2 2024-09-13T14:21:03.671 INFO:tasks.rados.rados.0.smithi151.stdout:105: finishing write tid 5 to smithi15186934-14 2024-09-13T14:21:03.671 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 19 (ObjNum 75 snap 9 seq_num 75) dirty exists 2024-09-13T14:21:03.672 INFO:tasks.rados.rados.0.smithi151.stdout:105: left oid 14 (ObjNum 75 snap 9 seq_num 75) 2024-09-13T14:21:03.672 INFO:tasks.rados.rados.0.smithi151.stdout:105: done (2 left) 2024-09-13T14:21:03.672 INFO:tasks.rados.rados.0.smithi151.stdout:109: write oid 20 current snap is 9 2024-09-13T14:21:03.672 INFO:tasks.rados.rados.0.smithi151.stdout:109: seq_num 77 ranges {494139=557703,1455937=734021,2948041=346098} 2024-09-13T14:21:03.677 INFO:tasks.rados.rados.0.smithi151.stdout:109: writing smithi15186934-20 from 494139 to 1051842 tid 1 2024-09-13T14:21:03.683 INFO:tasks.rados.rados.0.smithi151.stdout:109: writing smithi15186934-20 from 1455937 to 2189958 tid 2 2024-09-13T14:21:03.689 INFO:tasks.rados.rados.0.smithi151.stdout:109: writing smithi15186934-20 from 2948041 to 3294139 tid 3 2024-09-13T14:21:03.689 INFO:tasks.rados.rados.0.smithi151.stdout:110: write oid 40 current snap is 9 2024-09-13T14:21:03.690 INFO:tasks.rados.rados.0.smithi151.stdout:110: seq_num 78 ranges {510946=632952,1798066=477679,2742758=457217} 2024-09-13T14:21:03.695 INFO:tasks.rados.rados.0.smithi151.stdout:110: writing smithi15186934-40 from 510946 to 1143898 tid 1 2024-09-13T14:21:03.702 INFO:tasks.rados.rados.0.smithi151.stdout:110: writing smithi15186934-40 from 1798066 to 2275745 tid 2 2024-09-13T14:21:03.709 INFO:tasks.rados.rados.0.smithi151.stdout:110: writing smithi15186934-40 from 2742758 to 3199975 tid 3 2024-09-13T14:21:03.709 INFO:tasks.rados.rados.0.smithi151.stdout:111: snap_create 2024-09-13T14:21:03.709 INFO:tasks.rados.rados.0.smithi151.stdout:106: finishing rollback tid 0 to smithi15186934-30 2024-09-13T14:21:03.709 INFO:tasks.rados.rados.0.smithi151.stdout:106: finishing rollback tid 1 to smithi15186934-30 2024-09-13T14:21:03.709 INFO:tasks.rados.rados.0.smithi151.stdout:106: finishing rollback tid 2 to smithi15186934-30 2024-09-13T14:21:03.709 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 30 v 24 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-13T14:21:03.709 INFO:tasks.rados.rados.0.smithi151.stdout:108: finishing rollback tid 0 to smithi15186934-37 2024-09-13T14:21:03.709 INFO:tasks.rados.rados.0.smithi151.stdout:108: finishing rollback tid 1 to smithi15186934-37 2024-09-13T14:21:03.710 INFO:tasks.rados.rados.0.smithi151.stdout:108: finishing rollback tid 2 to smithi15186934-37 2024-09-13T14:21:03.710 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 37 v 30 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-13T14:21:03.710 INFO:tasks.rados.rados.0.smithi151.stdout:109: finishing write tid 1 to smithi15186934-20 2024-09-13T14:21:03.710 INFO:tasks.rados.rados.0.smithi151.stdout:109: finishing write tid 2 to smithi15186934-20 2024-09-13T14:21:03.710 INFO:tasks.rados.rados.0.smithi151.stdout:109: finishing write tid 3 to smithi15186934-20 2024-09-13T14:21:03.710 INFO:tasks.rados.rados.0.smithi151.stdout:109: finishing write tid 4 to smithi15186934-20 2024-09-13T14:21:03.710 INFO:tasks.rados.rados.0.smithi151.stdout:109: finishing write tid 5 to smithi15186934-20 2024-09-13T14:21:03.710 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 14 (ObjNum 77 snap 9 seq_num 77) dirty exists 2024-09-13T14:21:03.710 INFO:tasks.rados.rados.0.smithi151.stdout:109: left oid 20 (ObjNum 77 snap 9 seq_num 77) 2024-09-13T14:21:03.710 INFO:tasks.rados.rados.0.smithi151.stdout:110: finishing write tid 1 to smithi15186934-40 2024-09-13T14:21:03.710 INFO:tasks.rados.rados.0.smithi151.stdout:110: finishing write tid 2 to smithi15186934-40 2024-09-13T14:21:03.711 INFO:tasks.rados.rados.0.smithi151.stdout:110: finishing write tid 3 to smithi15186934-40 2024-09-13T14:21:03.711 INFO:tasks.rados.rados.0.smithi151.stdout:110: finishing write tid 4 to smithi15186934-40 2024-09-13T14:21:03.711 INFO:tasks.rados.rados.0.smithi151.stdout:110: finishing write tid 5 to smithi15186934-40 2024-09-13T14:21:03.711 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 40 v 29 (ObjNum 78 snap 9 seq_num 78) dirty exists 2024-09-13T14:21:03.711 INFO:tasks.rados.rados.0.smithi151.stdout:110: left oid 40 (ObjNum 78 snap 9 seq_num 78) 2024-09-13T14:21:03.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:03 smithi184 ceph-mon[33394]: pgmap v108: 89 pgs: 89 active+clean; 121 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 586 KiB/s wr, 10 op/s 2024-09-13T14:21:03.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:03 smithi184 ceph-mon[33394]: osdmap e81: 8 total, 8 up, 8 in 2024-09-13T14:21:03.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:03 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/2243571402' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:03.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:03 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:21:03.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:03 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:21:03.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:03 smithi151 ceph-mon[29807]: pgmap v108: 89 pgs: 89 active+clean; 121 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 586 KiB/s wr, 10 op/s 2024-09-13T14:21:03.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:03 smithi151 ceph-mon[29807]: osdmap e81: 8 total, 8 up, 8 in 2024-09-13T14:21:03.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:03 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/2243571402' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:03.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:03 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:21:03.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:03 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:21:03.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:03 smithi151 ceph-mon[41387]: pgmap v108: 89 pgs: 89 active+clean; 121 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 586 KiB/s wr, 10 op/s 2024-09-13T14:21:03.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:03 smithi151 ceph-mon[41387]: osdmap e81: 8 total, 8 up, 8 in 2024-09-13T14:21:03.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:03 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/2243571402' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:03.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:03 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:21:03.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:03 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:21:04.553 INFO:tasks.rados.rados.0.smithi151.stdout:106: done (4 left) 2024-09-13T14:21:04.553 INFO:tasks.rados.rados.0.smithi151.stdout:108: done (3 left) 2024-09-13T14:21:04.553 INFO:tasks.rados.rados.0.smithi151.stdout:109: done (2 left) 2024-09-13T14:21:04.553 INFO:tasks.rados.rados.0.smithi151.stdout:110: done (1 left) 2024-09-13T14:21:04.553 INFO:tasks.rados.rados.0.smithi151.stdout:111: done (0 left) 2024-09-13T14:21:04.553 INFO:tasks.rados.rados.0.smithi151.stdout:112: write oid 47 current snap is 10 2024-09-13T14:21:04.553 INFO:tasks.rados.rados.0.smithi151.stdout:112: seq_num 79 ranges {527753=708201,1856548=737690,3286534=41219} 2024-09-13T14:21:04.559 INFO:tasks.rados.rados.0.smithi151.stdout:112: writing smithi15186934-47 from 527753 to 1235954 tid 1 2024-09-13T14:21:04.567 INFO:tasks.rados.rados.0.smithi151.stdout:112: writing smithi15186934-47 from 1856548 to 2594238 tid 2 2024-09-13T14:21:04.571 INFO:tasks.rados.rados.0.smithi151.stdout:112: writing smithi15186934-47 from 3286534 to 3327753 tid 3 2024-09-13T14:21:04.571 INFO:tasks.rados.rados.0.smithi151.stdout:113: delete oid 17 current snap is 10 2024-09-13T14:21:04.571 INFO:tasks.rados.rados.0.smithi151.stdout:112: finishing write tid 1 to smithi15186934-47 2024-09-13T14:21:04.572 INFO:tasks.rados.rados.0.smithi151.stdout:112: finishing write tid 2 to smithi15186934-47 2024-09-13T14:21:04.573 INFO:tasks.rados.rados.0.smithi151.stdout:112: finishing write tid 3 to smithi15186934-47 2024-09-13T14:21:04.573 INFO:tasks.rados.rados.0.smithi151.stdout:112: finishing write tid 4 to smithi15186934-47 2024-09-13T14:21:04.573 INFO:tasks.rados.rados.0.smithi151.stdout:112: finishing write tid 5 to smithi15186934-47 2024-09-13T14:21:04.573 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 35 (ObjNum 79 snap 10 seq_num 79) dirty exists 2024-09-13T14:21:04.573 INFO:tasks.rados.rados.0.smithi151.stdout:112: left oid 47 (ObjNum 79 snap 10 seq_num 79) 2024-09-13T14:21:04.575 INFO:tasks.rados.rados.0.smithi151.stdout:112: done (1 left) 2024-09-13T14:21:04.575 INFO:tasks.rados.rados.0.smithi151.stdout:113: done (0 left) 2024-09-13T14:21:04.575 INFO:tasks.rados.rados.0.smithi151.stdout:114: snap_remove snap 2 2024-09-13T14:21:04.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:04 smithi184 ceph-mon[33394]: from='client.14874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:04.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:04 smithi184 ceph-mon[33394]: from='client.24782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:04.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:04 smithi184 ceph-mon[33394]: osdmap e82: 8 total, 8 up, 8 in 2024-09-13T14:21:04.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:04 smithi151 ceph-mon[29807]: from='client.14874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:04.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:04 smithi151 ceph-mon[29807]: from='client.24782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:04.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:04 smithi151 ceph-mon[29807]: osdmap e82: 8 total, 8 up, 8 in 2024-09-13T14:21:04.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:04 smithi151 ceph-mon[41387]: from='client.14874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:04.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:04 smithi151 ceph-mon[41387]: from='client.24782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:04.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:04 smithi151 ceph-mon[41387]: osdmap e82: 8 total, 8 up, 8 in 2024-09-13T14:21:05.562 INFO:tasks.rados.rados.0.smithi151.stdout:114: done (0 left) 2024-09-13T14:21:05.562 INFO:tasks.rados.rados.0.smithi151.stdout:115: write oid 43 current snap is 10 2024-09-13T14:21:05.562 INFO:tasks.rados.rados.0.smithi151.stdout:115: seq_num 81 ranges {561367=458699,1573512=741359,2941380=419987} 2024-09-13T14:21:05.567 INFO:tasks.rados.rados.0.smithi151.stdout:115: writing smithi15186934-43 from 561367 to 1020066 tid 1 2024-09-13T14:21:05.574 INFO:tasks.rados.rados.0.smithi151.stdout:115: writing smithi15186934-43 from 1573512 to 2314871 tid 2 2024-09-13T14:21:05.580 INFO:tasks.rados.rados.0.smithi151.stdout:115: writing smithi15186934-43 from 2941380 to 3361367 tid 3 2024-09-13T14:21:05.581 INFO:tasks.rados.rados.0.smithi151.stdout:116: read oid 11 snap -1 2024-09-13T14:21:05.581 INFO:tasks.rados.rados.0.smithi151.stdout:116: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-13T14:21:05.581 INFO:tasks.rados.rados.0.smithi151.stdout:117: rollback oid 37 current snap is 10 2024-09-13T14:21:05.581 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 37 to 1 2024-09-13T14:21:05.581 INFO:tasks.rados.rados.0.smithi151.stdout:115: finishing write tid 1 to smithi15186934-43 2024-09-13T14:21:05.581 INFO:tasks.rados.rados.0.smithi151.stdout:115: finishing write tid 2 to smithi15186934-43 2024-09-13T14:21:05.581 INFO:tasks.rados.rados.0.smithi151.stdout:118: delete oid 38 current snap is 10 2024-09-13T14:21:05.583 INFO:tasks.rados.rados.0.smithi151.stdout:117: finishing rollback tid 0 to smithi15186934-37 2024-09-13T14:21:05.583 INFO:tasks.rados.rados.0.smithi151.stdout:115: finishing write tid 3 to smithi15186934-43 2024-09-13T14:21:05.583 INFO:tasks.rados.rados.0.smithi151.stdout:118: done (3 left) 2024-09-13T14:21:05.584 INFO:tasks.rados.rados.0.smithi151.stdout:119: read oid 13 snap -1 2024-09-13T14:21:05.584 INFO:tasks.rados.rados.0.smithi151.stdout:119: expect deleted 2024-09-13T14:21:05.584 INFO:tasks.rados.rados.0.smithi151.stdout:120: read oid 14 snap -1 2024-09-13T14:21:05.584 INFO:tasks.rados.rados.0.smithi151.stdout:120: expect (ObjNum 75 snap 9 seq_num 75) 2024-09-13T14:21:05.584 INFO:tasks.rados.rados.0.smithi151.stdout:121: read oid 45 snap -1 2024-09-13T14:21:05.584 INFO:tasks.rados.rados.0.smithi151.stdout:121: expect (ObjNum 44 snap 0 seq_num 44) 2024-09-13T14:21:05.584 INFO:tasks.rados.rados.0.smithi151.stdout:122: snap_remove snap 9 2024-09-13T14:21:05.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:05 smithi184 ceph-mon[33394]: pgmap v111: 89 pgs: 89 active+clean; 135 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 14 op/s 2024-09-13T14:21:05.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:05 smithi184 ceph-mon[33394]: osdmap e83: 8 total, 8 up, 8 in 2024-09-13T14:21:05.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:05 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/2931581287' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:05.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:05 smithi151 ceph-mon[29807]: pgmap v111: 89 pgs: 89 active+clean; 135 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 14 op/s 2024-09-13T14:21:05.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:05 smithi151 ceph-mon[29807]: osdmap e83: 8 total, 8 up, 8 in 2024-09-13T14:21:05.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:05 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/2931581287' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:05.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:05 smithi151 ceph-mon[41387]: pgmap v111: 89 pgs: 89 active+clean; 135 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 14 op/s 2024-09-13T14:21:05.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:05 smithi151 ceph-mon[41387]: osdmap e83: 8 total, 8 up, 8 in 2024-09-13T14:21:05.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:05 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/2931581287' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:05.866 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:21:06.572 INFO:tasks.rados.rados.0.smithi151.stdout:122: done (6 left) 2024-09-13T14:21:06.573 INFO:tasks.rados.rados.0.smithi151.stdout:123: write oid 2 current snap is 10 2024-09-13T14:21:06.573 INFO:tasks.rados.rados.0.smithi151.stdout:117: finishing rollback tid 1 to smithi15186934-37 2024-09-13T14:21:06.573 INFO:tasks.rados.rados.0.smithi151.stdout:123: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2024-09-13T14:21:06.579 INFO:tasks.rados.rados.0.smithi151.stdout:123: writing smithi15186934-2 from 594981 to 1204178 tid 1 2024-09-13T14:21:06.587 INFO:tasks.rados.rados.0.smithi151.stdout:123: writing smithi15186934-2 from 1974123 to 2719151 tid 2 2024-09-13T14:21:06.592 INFO:tasks.rados.rados.0.smithi151.stdout:123: writing smithi15186934-2 from 3279873 to 3394981 tid 3 2024-09-13T14:21:06.592 INFO:tasks.rados.rados.0.smithi151.stdout:124: write oid 38 current snap is 10 2024-09-13T14:21:06.592 INFO:tasks.rados.rados.0.smithi151.stdout:117: finishing rollback tid 2 to smithi15186934-37 2024-09-13T14:21:06.592 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 37 v 45 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-13T14:21:06.592 INFO:tasks.rados.rados.0.smithi151.stdout:115: finishing write tid 4 to smithi15186934-43 2024-09-13T14:21:06.592 INFO:tasks.rados.rados.0.smithi151.stdout:115: finishing write tid 5 to smithi15186934-43 2024-09-13T14:21:06.592 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 43 v 47 (ObjNum 81 snap 10 seq_num 81) dirty exists 2024-09-13T14:21:06.592 INFO:tasks.rados.rados.0.smithi151.stdout:115: left oid 43 (ObjNum 81 snap 10 seq_num 81) 2024-09-13T14:21:06.592 INFO:tasks.rados.rados.0.smithi151.stdout:116: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-13T14:21:06.672 INFO:tasks.rados.rados.0.smithi151.stdout:120: expect (ObjNum 75 snap 9 seq_num 75) 2024-09-13T14:21:06.789 INFO:tasks.rados.rados.0.smithi151.stdout:121: expect (ObjNum 44 snap 0 seq_num 44) 2024-09-13T14:21:06.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:06 smithi184 ceph-mon[33394]: osdmap e84: 8 total, 8 up, 8 in 2024-09-13T14:21:06.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:06 smithi184 ceph-mon[33394]: from='client.24794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:06.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:06 smithi151 ceph-mon[29807]: osdmap e84: 8 total, 8 up, 8 in 2024-09-13T14:21:06.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:06 smithi151 ceph-mon[29807]: from='client.24794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:06.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:06 smithi151 ceph-mon[41387]: osdmap e84: 8 total, 8 up, 8 in 2024-09-13T14:21:06.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:06 smithi151 ceph-mon[41387]: from='client.24794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:06.884 INFO:tasks.rados.rados.0.smithi151.stdout:123: finishing write tid 1 to smithi15186934-2 2024-09-13T14:21:06.885 INFO:tasks.rados.rados.0.smithi151.stdout:123: finishing write tid 2 to smithi15186934-2 2024-09-13T14:21:06.885 INFO:tasks.rados.rados.0.smithi151.stdout:123: finishing write tid 3 to smithi15186934-2 2024-09-13T14:21:06.885 INFO:tasks.rados.rados.0.smithi151.stdout:123: finishing write tid 4 to smithi15186934-2 2024-09-13T14:21:06.885 INFO:tasks.rados.rados.0.smithi151.stdout:124: seq_num 84 ranges {611788=400799,1748958=488686,2907296=504492} 2024-09-13T14:21:06.890 INFO:tasks.rados.rados.0.smithi151.stdout:124: writing smithi15186934-38 from 611788 to 1012587 tid 1 2024-09-13T14:21:06.897 INFO:tasks.rados.rados.0.smithi151.stdout:124: writing smithi15186934-38 from 1748958 to 2237644 tid 2 2024-09-13T14:21:06.904 INFO:tasks.rados.rados.0.smithi151.stdout:124: writing smithi15186934-38 from 2907296 to 3411788 tid 3 2024-09-13T14:21:06.904 INFO:tasks.rados.rados.0.smithi151.stdout:115: done (7 left) 2024-09-13T14:21:06.904 INFO:tasks.rados.rados.0.smithi151.stdout:116: done (6 left) 2024-09-13T14:21:06.904 INFO:tasks.rados.rados.0.smithi151.stdout:117: done (5 left) 2024-09-13T14:21:06.904 INFO:tasks.rados.rados.0.smithi151.stdout:119: done (4 left) 2024-09-13T14:21:06.904 INFO:tasks.rados.rados.0.smithi151.stdout:120: done (3 left) 2024-09-13T14:21:06.904 INFO:tasks.rados.rados.0.smithi151.stdout:121: done (2 left) 2024-09-13T14:21:06.905 INFO:tasks.rados.rados.0.smithi151.stdout:125: write oid 34 current snap is 10 2024-09-13T14:21:06.905 INFO:tasks.rados.rados.0.smithi151.stdout:123: finishing write tid 5 to smithi15186934-2 2024-09-13T14:21:06.905 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 2 v 26 (ObjNum 83 snap 10 seq_num 83) dirty exists 2024-09-13T14:21:06.905 INFO:tasks.rados.rados.0.smithi151.stdout:123: left oid 2 (ObjNum 83 snap 10 seq_num 83) 2024-09-13T14:21:06.905 INFO:tasks.rados.rados.0.smithi151.stdout:125: seq_num 85 ranges {628595=476048,1807440=632344,2934719=452474} 2024-09-13T14:21:06.910 INFO:tasks.rados.rados.0.smithi151.stdout:125: writing smithi15186934-34 from 628595 to 1104643 tid 1 2024-09-13T14:21:06.917 INFO:tasks.rados.rados.0.smithi151.stdout:125: writing smithi15186934-34 from 1807440 to 2439784 tid 2 2024-09-13T14:21:06.923 INFO:tasks.rados.rados.0.smithi151.stdout:125: writing smithi15186934-34 from 2934719 to 3387193 tid 3 2024-09-13T14:21:06.923 INFO:tasks.rados.rados.0.smithi151.stdout:123: done (2 left) 2024-09-13T14:21:06.923 INFO:tasks.rados.rados.0.smithi151.stdout:126: snap_create 2024-09-13T14:21:06.923 INFO:tasks.rados.rados.0.smithi151.stdout:124: finishing write tid 1 to smithi15186934-38 2024-09-13T14:21:06.923 INFO:tasks.rados.rados.0.smithi151.stdout:124: finishing write tid 2 to smithi15186934-38 2024-09-13T14:21:06.923 INFO:tasks.rados.rados.0.smithi151.stdout:124: finishing write tid 3 to smithi15186934-38 2024-09-13T14:21:06.923 INFO:tasks.rados.rados.0.smithi151.stdout:124: finishing write tid 4 to smithi15186934-38 2024-09-13T14:21:06.923 INFO:tasks.rados.rados.0.smithi151.stdout:124: finishing write tid 5 to smithi15186934-38 2024-09-13T14:21:06.924 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 29 (ObjNum 84 snap 10 seq_num 84) dirty exists 2024-09-13T14:21:06.924 INFO:tasks.rados.rados.0.smithi151.stdout:124: left oid 38 (ObjNum 84 snap 10 seq_num 84) 2024-09-13T14:21:06.924 INFO:tasks.rados.rados.0.smithi151.stdout:125: finishing write tid 1 to smithi15186934-34 2024-09-13T14:21:06.924 INFO:tasks.rados.rados.0.smithi151.stdout:125: finishing write tid 2 to smithi15186934-34 2024-09-13T14:21:06.925 INFO:tasks.rados.rados.0.smithi151.stdout:125: finishing write tid 3 to smithi15186934-34 2024-09-13T14:21:06.925 INFO:tasks.rados.rados.0.smithi151.stdout:125: finishing write tid 4 to smithi15186934-34 2024-09-13T14:21:06.925 INFO:tasks.rados.rados.0.smithi151.stdout:125: finishing write tid 5 to smithi15186934-34 2024-09-13T14:21:06.925 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 40 (ObjNum 85 snap 10 seq_num 85) dirty exists 2024-09-13T14:21:06.925 INFO:tasks.rados.rados.0.smithi151.stdout:125: left oid 34 (ObjNum 85 snap 10 seq_num 85) 2024-09-13T14:21:07.575 INFO:tasks.rados.rados.0.smithi151.stdout:124: done (2 left) 2024-09-13T14:21:07.575 INFO:tasks.rados.rados.0.smithi151.stdout:125: done (1 left) 2024-09-13T14:21:07.575 INFO:tasks.rados.rados.0.smithi151.stdout:126: done (0 left) 2024-09-13T14:21:07.575 INFO:tasks.rados.rados.0.smithi151.stdout:127: write oid 32 current snap is 11 2024-09-13T14:21:07.575 INFO:tasks.rados.rados.0.smithi151.stdout:127: seq_num 86 ranges {645402=551297,1749569=492355,2845789=599613} 2024-09-13T14:21:07.581 INFO:tasks.rados.rados.0.smithi151.stdout:127: writing smithi15186934-32 from 645402 to 1196699 tid 1 2024-09-13T14:21:07.587 INFO:tasks.rados.rados.0.smithi151.stdout:127: writing smithi15186934-32 from 1749569 to 2241924 tid 2 2024-09-13T14:21:07.594 INFO:tasks.rados.rados.0.smithi151.stdout:127: writing smithi15186934-32 from 2845789 to 3445402 tid 3 2024-09-13T14:21:07.595 INFO:tasks.rados.rados.0.smithi151.stdout:128: read oid 35 snap -1 2024-09-13T14:21:07.595 INFO:tasks.rados.rados.0.smithi151.stdout:128: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-13T14:21:07.595 INFO:tasks.rados.rados.0.smithi151.stdout:129: read oid 11 snap -1 2024-09-13T14:21:07.595 INFO:tasks.rados.rados.0.smithi151.stdout:129: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-13T14:21:07.595 INFO:tasks.rados.rados.0.smithi151.stdout:130: write oid 42 current snap is 11 2024-09-13T14:21:07.595 INFO:tasks.rados.rados.0.smithi151.stdout:127: finishing write tid 1 to smithi15186934-32 2024-09-13T14:21:07.595 INFO:tasks.rados.rados.0.smithi151.stdout:127: finishing write tid 2 to smithi15186934-32 2024-09-13T14:21:07.595 INFO:tasks.rados.rados.0.smithi151.stdout:130: seq_num 87 ranges {662209=626546,1808051=636013,2873212=588997} 2024-09-13T14:21:07.602 INFO:tasks.rados.rados.0.smithi151.stdout:130: writing smithi15186934-42 from 662209 to 1288755 tid 1 2024-09-13T14:21:07.608 INFO:tasks.rados.rados.0.smithi151.stdout:130: writing smithi15186934-42 from 1808051 to 2444064 tid 2 2024-09-13T14:21:07.614 INFO:tasks.rados.rados.0.smithi151.stdout:130: writing smithi15186934-42 from 2873212 to 3462209 tid 3 2024-09-13T14:21:07.615 INFO:tasks.rados.rados.0.smithi151.stdout:131: snap_remove snap 10 2024-09-13T14:21:07.771 INFO:tasks.rados.rados.0.smithi151.stdout:131: done (4 left) 2024-09-13T14:21:07.771 INFO:tasks.rados.rados.0.smithi151.stdout:132: read oid 16 snap -1 2024-09-13T14:21:07.771 INFO:tasks.rados.rados.0.smithi151.stdout:132: expect (ObjNum 15 snap 0 seq_num 15) 2024-09-13T14:21:07.771 INFO:tasks.rados.rados.0.smithi151.stdout:133: snap_remove snap 3 2024-09-13T14:21:07.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:07 smithi184 ceph-mon[33394]: from='client.24800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:07.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:07 smithi184 ceph-mon[33394]: pgmap v114: 89 pgs: 89 active+clean; 135 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 14 op/s 2024-09-13T14:21:07.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:07 smithi184 ceph-mon[33394]: osdmap e85: 8 total, 8 up, 8 in 2024-09-13T14:21:07.854 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:21:07 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:21:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:21:07.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:07 smithi151 ceph-mon[29807]: from='client.24800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:07.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:07 smithi151 ceph-mon[29807]: pgmap v114: 89 pgs: 89 active+clean; 135 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 14 op/s 2024-09-13T14:21:07.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:07 smithi151 ceph-mon[29807]: osdmap e85: 8 total, 8 up, 8 in 2024-09-13T14:21:07.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:07 smithi151 ceph-mon[41387]: from='client.24800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:07.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:07 smithi151 ceph-mon[41387]: pgmap v114: 89 pgs: 89 active+clean; 135 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 14 op/s 2024-09-13T14:21:07.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:07 smithi151 ceph-mon[41387]: osdmap e85: 8 total, 8 up, 8 in 2024-09-13T14:21:08.778 INFO:tasks.rados.rados.0.smithi151.stdout:133: done (5 left) 2024-09-13T14:21:08.778 INFO:tasks.rados.rados.0.smithi151.stdout:134: write oid 20 current snap is 11 2024-09-13T14:21:08.778 INFO:tasks.rados.rados.0.smithi151.stdout:134: seq_num 88 ranges {679016=701795,1866533=496024,2900635=402643} 2024-09-13T14:21:08.786 INFO:tasks.rados.rados.0.smithi151.stdout:134: writing smithi15186934-20 from 679016 to 1380811 tid 1 2024-09-13T14:21:08.792 INFO:tasks.rados.rados.0.smithi151.stdout:134: writing smithi15186934-20 from 1866533 to 2362557 tid 2 2024-09-13T14:21:08.798 INFO:tasks.rados.rados.0.smithi151.stdout:134: writing smithi15186934-20 from 2900635 to 3303278 tid 3 2024-09-13T14:21:08.798 INFO:tasks.rados.rados.0.smithi151.stdout:129: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-13T14:21:08.841 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:21:08.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:08 smithi151 ceph-mon[29807]: osdmap e86: 8 total, 8 up, 8 in 2024-09-13T14:21:08.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:08 smithi151 ceph-mon[29807]: osdmap e87: 8 total, 8 up, 8 in 2024-09-13T14:21:08.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:08 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/792030774' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:08.841 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:08 smithi151 ceph-mon[41387]: osdmap e86: 8 total, 8 up, 8 in 2024-09-13T14:21:08.842 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:08 smithi151 ceph-mon[41387]: osdmap e87: 8 total, 8 up, 8 in 2024-09-13T14:21:08.842 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:08 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/792030774' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:08.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:08 smithi184 ceph-mon[33394]: osdmap e86: 8 total, 8 up, 8 in 2024-09-13T14:21:08.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:08 smithi184 ceph-mon[33394]: osdmap e87: 8 total, 8 up, 8 in 2024-09-13T14:21:08.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:08 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/792030774' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:08.876 INFO:tasks.rados.rados.0.smithi151.stdout:127: finishing write tid 3 to smithi15186934-32 2024-09-13T14:21:08.876 INFO:tasks.rados.rados.0.smithi151.stdout:127: finishing write tid 4 to smithi15186934-32 2024-09-13T14:21:08.876 INFO:tasks.rados.rados.0.smithi151.stdout:127: finishing write tid 5 to smithi15186934-32 2024-09-13T14:21:08.876 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 32 v 28 (ObjNum 86 snap 11 seq_num 86) dirty exists 2024-09-13T14:21:08.876 INFO:tasks.rados.rados.0.smithi151.stdout:127: left oid 32 (ObjNum 86 snap 11 seq_num 86) 2024-09-13T14:21:08.876 INFO:tasks.rados.rados.0.smithi151.stdout:135: read oid 10 snap -1 2024-09-13T14:21:08.876 INFO:tasks.rados.rados.0.smithi151.stdout:135: expect deleted 2024-09-13T14:21:08.877 INFO:tasks.rados.rados.0.smithi151.stdout:127: done (6 left) 2024-09-13T14:21:08.877 INFO:tasks.rados.rados.0.smithi151.stdout:129: done (5 left) 2024-09-13T14:21:08.877 INFO:tasks.rados.rados.0.smithi151.stdout:136: write oid 37 current snap is 11 2024-09-13T14:21:08.877 INFO:tasks.rados.rados.0.smithi151.stdout:128: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-13T14:21:08.992 INFO:tasks.rados.rados.0.smithi151.stdout:130: finishing write tid 1 to smithi15186934-42 2024-09-13T14:21:08.992 INFO:tasks.rados.rados.0.smithi151.stdout:130: finishing write tid 2 to smithi15186934-42 2024-09-13T14:21:08.992 INFO:tasks.rados.rados.0.smithi151.stdout:130: finishing write tid 3 to smithi15186934-42 2024-09-13T14:21:08.992 INFO:tasks.rados.rados.0.smithi151.stdout:130: finishing write tid 4 to smithi15186934-42 2024-09-13T14:21:08.992 INFO:tasks.rados.rados.0.smithi151.stdout:130: finishing write tid 5 to smithi15186934-42 2024-09-13T14:21:08.992 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 41 (ObjNum 87 snap 11 seq_num 87) dirty exists 2024-09-13T14:21:08.992 INFO:tasks.rados.rados.0.smithi151.stdout:130: left oid 42 (ObjNum 87 snap 11 seq_num 87) 2024-09-13T14:21:08.992 INFO:tasks.rados.rados.0.smithi151.stdout:136: seq_num 89 ranges {695823=777044,1925015=639682,3328058=167765} 2024-09-13T14:21:08.999 INFO:tasks.rados.rados.0.smithi151.stdout:136: writing smithi15186934-37 from 695823 to 1472867 tid 1 2024-09-13T14:21:09.006 INFO:tasks.rados.rados.0.smithi151.stdout:136: writing smithi15186934-37 from 1925015 to 2564697 tid 2 2024-09-13T14:21:09.012 INFO:tasks.rados.rados.0.smithi151.stdout:136: writing smithi15186934-37 from 3328058 to 3495823 tid 3 2024-09-13T14:21:09.012 INFO:tasks.rados.rados.0.smithi151.stdout:128: done (5 left) 2024-09-13T14:21:09.012 INFO:tasks.rados.rados.0.smithi151.stdout:130: done (4 left) 2024-09-13T14:21:09.012 INFO:tasks.rados.rados.0.smithi151.stdout:137: delete oid 34 current snap is 11 2024-09-13T14:21:09.012 INFO:tasks.rados.rados.0.smithi151.stdout:132: expect (ObjNum 15 snap 0 seq_num 15) 2024-09-13T14:21:09.095 INFO:tasks.rados.rados.0.smithi151.stdout:132: done (4 left) 2024-09-13T14:21:09.095 INFO:tasks.rados.rados.0.smithi151.stdout:137: done (3 left) 2024-09-13T14:21:09.095 INFO:tasks.rados.rados.0.smithi151.stdout:138: snap_create 2024-09-13T14:21:09.095 INFO:tasks.rados.rados.0.smithi151.stdout:134: finishing write tid 1 to smithi15186934-20 2024-09-13T14:21:09.095 INFO:tasks.rados.rados.0.smithi151.stdout:134: finishing write tid 2 to smithi15186934-20 2024-09-13T14:21:09.095 INFO:tasks.rados.rados.0.smithi151.stdout:134: finishing write tid 3 to smithi15186934-20 2024-09-13T14:21:09.096 INFO:tasks.rados.rados.0.smithi151.stdout:134: finishing write tid 4 to smithi15186934-20 2024-09-13T14:21:09.096 INFO:tasks.rados.rados.0.smithi151.stdout:134: finishing write tid 5 to smithi15186934-20 2024-09-13T14:21:09.096 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 22 (ObjNum 88 snap 11 seq_num 88) dirty exists 2024-09-13T14:21:09.096 INFO:tasks.rados.rados.0.smithi151.stdout:134: left oid 20 (ObjNum 88 snap 11 seq_num 88) 2024-09-13T14:21:09.096 INFO:tasks.rados.rados.0.smithi151.stdout:136: finishing write tid 1 to smithi15186934-37 2024-09-13T14:21:09.096 INFO:tasks.rados.rados.0.smithi151.stdout:136: finishing write tid 2 to smithi15186934-37 2024-09-13T14:21:09.096 INFO:tasks.rados.rados.0.smithi151.stdout:136: finishing write tid 3 to smithi15186934-37 2024-09-13T14:21:09.096 INFO:tasks.rados.rados.0.smithi151.stdout:136: finishing write tid 4 to smithi15186934-37 2024-09-13T14:21:09.096 INFO:tasks.rados.rados.0.smithi151.stdout:136: finishing write tid 5 to smithi15186934-37 2024-09-13T14:21:09.096 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 37 v 59 (ObjNum 89 snap 11 seq_num 89) dirty exists 2024-09-13T14:21:09.096 INFO:tasks.rados.rados.0.smithi151.stdout:136: left oid 37 (ObjNum 89 snap 11 seq_num 89) 2024-09-13T14:21:09.790 INFO:tasks.rados.rados.0.smithi151.stdout:134: done (3 left) 2024-09-13T14:21:09.790 INFO:tasks.rados.rados.0.smithi151.stdout:135: done (2 left) 2024-09-13T14:21:09.791 INFO:tasks.rados.rados.0.smithi151.stdout:136: done (1 left) 2024-09-13T14:21:09.791 INFO:tasks.rados.rados.0.smithi151.stdout:138: done (0 left) 2024-09-13T14:21:09.791 INFO:tasks.rados.rados.0.smithi151.stdout:139: snap_remove snap 1 2024-09-13T14:21:10.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:09 smithi184 ceph-mon[33394]: pgmap v118: 89 pgs: 89 active+clean; 135 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:10.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:09 smithi184 ceph-mon[33394]: osdmap e88: 8 total, 8 up, 8 in 2024-09-13T14:21:10.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:09 smithi184 ceph-mon[33394]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:10.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:09 smithi151 ceph-mon[29807]: pgmap v118: 89 pgs: 89 active+clean; 135 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:10.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:09 smithi151 ceph-mon[29807]: osdmap e88: 8 total, 8 up, 8 in 2024-09-13T14:21:10.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:09 smithi151 ceph-mon[29807]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:10.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:09 smithi151 ceph-mon[41387]: pgmap v118: 89 pgs: 89 active+clean; 135 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:10.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:09 smithi151 ceph-mon[41387]: osdmap e88: 8 total, 8 up, 8 in 2024-09-13T14:21:10.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:09 smithi151 ceph-mon[41387]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:10.801 INFO:tasks.rados.rados.0.smithi151.stdout:139: done (0 left) 2024-09-13T14:21:10.801 INFO:tasks.rados.rados.0.smithi151.stdout:140: snap_create 2024-09-13T14:21:11.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:10 smithi184 ceph-mon[33394]: from='client.24818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:11.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:10 smithi184 ceph-mon[33394]: osdmap e89: 8 total, 8 up, 8 in 2024-09-13T14:21:11.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:10 smithi151 ceph-mon[29807]: from='client.24818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:11.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:10 smithi151 ceph-mon[29807]: osdmap e89: 8 total, 8 up, 8 in 2024-09-13T14:21:11.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:10 smithi151 ceph-mon[41387]: from='client.24818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:11.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:10 smithi151 ceph-mon[41387]: osdmap e89: 8 total, 8 up, 8 in 2024-09-13T14:21:11.797 INFO:tasks.rados.rados.0.smithi151.stdout:140: done (0 left) 2024-09-13T14:21:11.797 INFO:tasks.rados.rados.0.smithi151.stdout:141: read oid 31 snap -1 2024-09-13T14:21:11.797 INFO:tasks.rados.rados.0.smithi151.stdout:141: expect (ObjNum 30 snap 0 seq_num 30) 2024-09-13T14:21:11.798 INFO:tasks.rados.rados.0.smithi151.stdout:142: read oid 13 snap -1 2024-09-13T14:21:11.798 INFO:tasks.rados.rados.0.smithi151.stdout:142: expect deleted 2024-09-13T14:21:11.798 INFO:tasks.rados.rados.0.smithi151.stdout:143: read oid 7 snap -1 2024-09-13T14:21:11.798 INFO:tasks.rados.rados.0.smithi151.stdout:143: expect deleted 2024-09-13T14:21:11.798 INFO:tasks.rados.rados.0.smithi151.stdout:144: snap_create 2024-09-13T14:21:11.802 INFO:tasks.rados.rados.0.smithi151.stdout:141: expect (ObjNum 30 snap 0 seq_num 30) 2024-09-13T14:21:11.810 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:21:12.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:11 smithi184 ceph-mon[33394]: pgmap v121: 89 pgs: 89 active+clean; 150 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 3.5 MiB/s wr, 28 op/s 2024-09-13T14:21:12.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:11 smithi184 ceph-mon[33394]: osdmap e90: 8 total, 8 up, 8 in 2024-09-13T14:21:12.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:11 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/2817562358' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:12.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:11 smithi151 ceph-mon[29807]: pgmap v121: 89 pgs: 89 active+clean; 150 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 3.5 MiB/s wr, 28 op/s 2024-09-13T14:21:12.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:11 smithi151 ceph-mon[29807]: osdmap e90: 8 total, 8 up, 8 in 2024-09-13T14:21:12.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:11 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/2817562358' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:12.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:11 smithi151 ceph-mon[41387]: pgmap v121: 89 pgs: 89 active+clean; 150 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 3.5 MiB/s wr, 28 op/s 2024-09-13T14:21:12.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:11 smithi151 ceph-mon[41387]: osdmap e90: 8 total, 8 up, 8 in 2024-09-13T14:21:12.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:11 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/2817562358' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:12.772 INFO:tasks.rados.rados.0.smithi151.stdout:141: done (3 left) 2024-09-13T14:21:12.772 INFO:tasks.rados.rados.0.smithi151.stdout:142: done (2 left) 2024-09-13T14:21:12.772 INFO:tasks.rados.rados.0.smithi151.stdout:143: done (1 left) 2024-09-13T14:21:12.772 INFO:tasks.rados.rados.0.smithi151.stdout:144: done (0 left) 2024-09-13T14:21:12.773 INFO:tasks.rados.rados.0.smithi151.stdout:145: snap_create 2024-09-13T14:21:13.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:12 smithi184 ceph-mon[33394]: osdmap e91: 8 total, 8 up, 8 in 2024-09-13T14:21:13.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:12 smithi184 ceph-mon[33394]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:13.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:12 smithi184 ceph-mon[33394]: osdmap e92: 8 total, 8 up, 8 in 2024-09-13T14:21:13.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:12 smithi151 ceph-mon[29807]: osdmap e91: 8 total, 8 up, 8 in 2024-09-13T14:21:13.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:12 smithi151 ceph-mon[29807]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:13.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:12 smithi151 ceph-mon[29807]: osdmap e92: 8 total, 8 up, 8 in 2024-09-13T14:21:13.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:12 smithi151 ceph-mon[41387]: osdmap e91: 8 total, 8 up, 8 in 2024-09-13T14:21:13.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:12 smithi151 ceph-mon[41387]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:13.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:12 smithi151 ceph-mon[41387]: osdmap e92: 8 total, 8 up, 8 in 2024-09-13T14:21:13.109 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:21:12 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:21:12] "GET /metrics HTTP/1.1" 200 243016 "" "Prometheus/2.43.0" 2024-09-13T14:21:13.773 INFO:tasks.rados.rados.0.smithi151.stdout:145: done (0 left) 2024-09-13T14:21:13.774 INFO:tasks.rados.rados.0.smithi151.stdout:146: write oid 9 current snap is 15 2024-09-13T14:21:13.774 INFO:tasks.rados.rados.0.smithi151.stdout:146: seq_num 91 ranges {729437=527542,1925626=643351,3266551=262886} 2024-09-13T14:21:13.780 INFO:tasks.rados.rados.0.smithi151.stdout:146: writing smithi15186934-9 from 729437 to 1256979 tid 1 2024-09-13T14:21:13.788 INFO:tasks.rados.rados.0.smithi151.stdout:146: writing smithi15186934-9 from 1925626 to 2568977 tid 2 2024-09-13T14:21:13.794 INFO:tasks.rados.rados.0.smithi151.stdout:146: writing smithi15186934-9 from 3266551 to 3529437 tid 3 2024-09-13T14:21:13.794 INFO:tasks.rados.rados.0.smithi151.stdout:147: rollback oid 14 current snap is 15 2024-09-13T14:21:13.794 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 14 to 13 2024-09-13T14:21:13.794 INFO:tasks.rados.rados.0.smithi151.stdout:146: finishing write tid 1 to smithi15186934-9 2024-09-13T14:21:13.794 INFO:tasks.rados.rados.0.smithi151.stdout:146: finishing write tid 2 to smithi15186934-9 2024-09-13T14:21:13.794 INFO:tasks.rados.rados.0.smithi151.stdout:148: write oid 6 current snap is 15 2024-09-13T14:21:13.794 INFO:tasks.rados.rados.0.smithi151.stdout:148: seq_num 92 ranges {746244=719144,2100461=503362,3126680=419564} 2024-09-13T14:21:13.802 INFO:tasks.rados.rados.0.smithi151.stdout:148: writing smithi15186934-6 from 746244 to 1465388 tid 1 2024-09-13T14:21:13.808 INFO:tasks.rados.rados.0.smithi151.stdout:148: writing smithi15186934-6 from 2100461 to 2603823 tid 2 2024-09-13T14:21:13.814 INFO:tasks.rados.rados.0.smithi151.stdout:148: writing smithi15186934-6 from 3126680 to 3546244 tid 3 2024-09-13T14:21:13.814 INFO:tasks.rados.rados.0.smithi151.stdout:149: read oid 42 snap -1 2024-09-13T14:21:13.814 INFO:tasks.rados.rados.0.smithi151.stdout:149: expect (ObjNum 87 snap 11 seq_num 87) 2024-09-13T14:21:13.814 INFO:tasks.rados.rados.0.smithi151.stdout:150: snap_create 2024-09-13T14:21:13.814 INFO:tasks.rados.rados.0.smithi151.stdout:146: finishing write tid 3 to smithi15186934-9 2024-09-13T14:21:13.814 INFO:tasks.rados.rados.0.smithi151.stdout:146: finishing write tid 4 to smithi15186934-9 2024-09-13T14:21:13.814 INFO:tasks.rados.rados.0.smithi151.stdout:146: finishing write tid 5 to smithi15186934-9 2024-09-13T14:21:13.815 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 45 (ObjNum 91 snap 15 seq_num 91) dirty exists 2024-09-13T14:21:13.815 INFO:tasks.rados.rados.0.smithi151.stdout:146: left oid 9 (ObjNum 91 snap 15 seq_num 91) 2024-09-13T14:21:13.815 INFO:tasks.rados.rados.0.smithi151.stdout:147: finishing rollback tid 0 to smithi15186934-14 2024-09-13T14:21:13.815 INFO:tasks.rados.rados.0.smithi151.stdout:147: finishing rollback tid 1 to smithi15186934-14 2024-09-13T14:21:13.815 INFO:tasks.rados.rados.0.smithi151.stdout:147: finishing rollback tid 2 to smithi15186934-14 2024-09-13T14:21:13.815 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 49 (ObjNum 75 snap 9 seq_num 75) dirty exists 2024-09-13T14:21:13.815 INFO:tasks.rados.rados.0.smithi151.stdout:148: finishing write tid 1 to smithi15186934-6 2024-09-13T14:21:13.815 INFO:tasks.rados.rados.0.smithi151.stdout:148: finishing write tid 2 to smithi15186934-6 2024-09-13T14:21:13.816 INFO:tasks.rados.rados.0.smithi151.stdout:148: finishing write tid 3 to smithi15186934-6 2024-09-13T14:21:13.816 INFO:tasks.rados.rados.0.smithi151.stdout:148: finishing write tid 4 to smithi15186934-6 2024-09-13T14:21:13.816 INFO:tasks.rados.rados.0.smithi151.stdout:148: finishing write tid 5 to smithi15186934-6 2024-09-13T14:21:13.817 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 6 v 29 (ObjNum 92 snap 15 seq_num 92) dirty exists 2024-09-13T14:21:13.817 INFO:tasks.rados.rados.0.smithi151.stdout:148: left oid 6 (ObjNum 92 snap 15 seq_num 92) 2024-09-13T14:21:13.821 INFO:tasks.rados.rados.0.smithi151.stdout:149: expect (ObjNum 87 snap 11 seq_num 87) 2024-09-13T14:21:14.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:13 smithi184 ceph-mon[33394]: from='client.24836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:14.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:13 smithi184 ceph-mon[33394]: pgmap v124: 89 pgs: 89 active+clean; 150 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 3.4 MiB/s wr, 27 op/s 2024-09-13T14:21:14.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:13 smithi184 ceph-mon[33394]: osdmap e93: 8 total, 8 up, 8 in 2024-09-13T14:21:14.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:13 smithi151 ceph-mon[29807]: from='client.24836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:14.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:13 smithi151 ceph-mon[29807]: pgmap v124: 89 pgs: 89 active+clean; 150 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 3.4 MiB/s wr, 27 op/s 2024-09-13T14:21:14.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:13 smithi151 ceph-mon[29807]: osdmap e93: 8 total, 8 up, 8 in 2024-09-13T14:21:14.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:13 smithi151 ceph-mon[41387]: from='client.24836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:14.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:13 smithi151 ceph-mon[41387]: pgmap v124: 89 pgs: 89 active+clean; 150 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 3.4 MiB/s wr, 27 op/s 2024-09-13T14:21:14.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:13 smithi151 ceph-mon[41387]: osdmap e93: 8 total, 8 up, 8 in 2024-09-13T14:21:14.731 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:21:14.817 INFO:tasks.rados.rados.0.smithi151.stdout:146: done (4 left) 2024-09-13T14:21:14.817 INFO:tasks.rados.rados.0.smithi151.stdout:147: done (3 left) 2024-09-13T14:21:14.818 INFO:tasks.rados.rados.0.smithi151.stdout:148: done (2 left) 2024-09-13T14:21:14.818 INFO:tasks.rados.rados.0.smithi151.stdout:149: done (1 left) 2024-09-13T14:21:14.818 INFO:tasks.rados.rados.0.smithi151.stdout:150: done (0 left) 2024-09-13T14:21:14.818 INFO:tasks.rados.rados.0.smithi151.stdout:151: write oid 46 current snap is 16 2024-09-13T14:21:14.818 INFO:tasks.rados.rados.0.smithi151.stdout:151: seq_num 93 ranges {763051=794393,2158943=647020,3437750=125301} 2024-09-13T14:21:14.827 INFO:tasks.rados.rados.0.smithi151.stdout:151: writing smithi15186934-46 from 763051 to 1557444 tid 1 2024-09-13T14:21:14.835 INFO:tasks.rados.rados.0.smithi151.stdout:151: writing smithi15186934-46 from 2158943 to 2805963 tid 2 2024-09-13T14:21:14.840 INFO:tasks.rados.rados.0.smithi151.stdout:151: writing smithi15186934-46 from 3437750 to 3563051 tid 3 2024-09-13T14:21:14.841 INFO:tasks.rados.rados.0.smithi151.stdout:152: read oid 25 snap -1 2024-09-13T14:21:14.841 INFO:tasks.rados.rados.0.smithi151.stdout:152: expect (ObjNum 24 snap 0 seq_num 24) 2024-09-13T14:21:14.841 INFO:tasks.rados.rados.0.smithi151.stdout:153: write oid 50 current snap is 16 2024-09-13T14:21:14.841 INFO:tasks.rados.rados.0.smithi151.stdout:151: finishing write tid 1 to smithi15186934-46 2024-09-13T14:21:14.841 INFO:tasks.rados.rados.0.smithi151.stdout:151: finishing write tid 2 to smithi15186934-46 2024-09-13T14:21:14.841 INFO:tasks.rados.rados.0.smithi151.stdout:153: seq_num 94 ranges {779858=469642,1817425=507031,2781526=586628} 2024-09-13T14:21:14.848 INFO:tasks.rados.rados.0.smithi151.stdout:153: writing smithi15186934-50 from 779858 to 1249500 tid 1 2024-09-13T14:21:14.855 INFO:tasks.rados.rados.0.smithi151.stdout:153: writing smithi15186934-50 from 1817425 to 2324456 tid 2 2024-09-13T14:21:14.861 INFO:tasks.rados.rados.0.smithi151.stdout:153: writing smithi15186934-50 from 2781526 to 3368154 tid 3 2024-09-13T14:21:14.861 INFO:tasks.rados.rados.0.smithi151.stdout:154: read oid 6 snap 12 2024-09-13T14:21:14.861 INFO:tasks.rados.rados.0.smithi151.stdout:154: expect (ObjNum 72 snap 9 seq_num 72) 2024-09-13T14:21:14.861 INFO:tasks.rados.rados.0.smithi151.stdout:155: snap_remove snap 8 2024-09-13T14:21:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:14 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/447650218' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:15.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:14 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/447650218' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:15.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:14 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/447650218' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:15.822 INFO:tasks.rados.rados.0.smithi151.stdout:155: done (4 left) 2024-09-13T14:21:15.822 INFO:tasks.rados.rados.0.smithi151.stdout:156: snap_remove snap 6 2024-09-13T14:21:16.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:15 smithi184 ceph-mon[33394]: pgmap v127: 89 pgs: 89 active+clean; 153 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.6 MiB/s rd, 752 KiB/s wr, 7 op/s 2024-09-13T14:21:16.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:15 smithi184 ceph-mon[33394]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:16.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:15 smithi184 ceph-mon[33394]: osdmap e94: 8 total, 8 up, 8 in 2024-09-13T14:21:16.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:15 smithi151 ceph-mon[29807]: pgmap v127: 89 pgs: 89 active+clean; 153 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.6 MiB/s rd, 752 KiB/s wr, 7 op/s 2024-09-13T14:21:16.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:15 smithi151 ceph-mon[29807]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:16.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:15 smithi151 ceph-mon[29807]: osdmap e94: 8 total, 8 up, 8 in 2024-09-13T14:21:16.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:15 smithi151 ceph-mon[41387]: pgmap v127: 89 pgs: 89 active+clean; 153 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.6 MiB/s rd, 752 KiB/s wr, 7 op/s 2024-09-13T14:21:16.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:15 smithi151 ceph-mon[41387]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:16.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:15 smithi151 ceph-mon[41387]: osdmap e94: 8 total, 8 up, 8 in 2024-09-13T14:21:16.836 INFO:tasks.rados.rados.0.smithi151.stdout:156: done (4 left) 2024-09-13T14:21:16.836 INFO:tasks.rados.rados.0.smithi151.stdout:157: delete oid 16 current snap is 16 2024-09-13T14:21:16.836 INFO:tasks.rados.rados.0.smithi151.stdout:151: finishing write tid 3 to smithi15186934-46 2024-09-13T14:21:16.836 INFO:tasks.rados.rados.0.smithi151.stdout:151: finishing write tid 4 to smithi15186934-46 2024-09-13T14:21:16.836 INFO:tasks.rados.rados.0.smithi151.stdout:151: finishing write tid 5 to smithi15186934-46 2024-09-13T14:21:16.836 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 46 v 30 (ObjNum 93 snap 16 seq_num 93) dirty exists 2024-09-13T14:21:16.836 INFO:tasks.rados.rados.0.smithi151.stdout:151: left oid 46 (ObjNum 93 snap 16 seq_num 93) 2024-09-13T14:21:16.836 INFO:tasks.rados.rados.0.smithi151.stdout:152: expect (ObjNum 24 snap 0 seq_num 24) 2024-09-13T14:21:16.925 INFO:tasks.rados.rados.0.smithi151.stdout:153: finishing write tid 1 to smithi15186934-50 2024-09-13T14:21:16.926 INFO:tasks.rados.rados.0.smithi151.stdout:153: finishing write tid 2 to smithi15186934-50 2024-09-13T14:21:16.926 INFO:tasks.rados.rados.0.smithi151.stdout:151: done (4 left) 2024-09-13T14:21:16.926 INFO:tasks.rados.rados.0.smithi151.stdout:152: done (3 left) 2024-09-13T14:21:16.926 INFO:tasks.rados.rados.0.smithi151.stdout:157: done (2 left) 2024-09-13T14:21:16.926 INFO:tasks.rados.rados.0.smithi151.stdout:158: write oid 32 current snap is 16 2024-09-13T14:21:16.926 INFO:tasks.rados.rados.0.smithi151.stdout:153: finishing write tid 3 to smithi15186934-50 2024-09-13T14:21:16.926 INFO:tasks.rados.rados.0.smithi151.stdout:153: finishing write tid 4 to smithi15186934-50 2024-09-13T14:21:16.926 INFO:tasks.rados.rados.0.smithi151.stdout:153: finishing write tid 5 to smithi15186934-50 2024-09-13T14:21:16.926 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 50 v 38 (ObjNum 94 snap 16 seq_num 94) dirty exists 2024-09-13T14:21:16.926 INFO:tasks.rados.rados.0.smithi151.stdout:153: left oid 50 (ObjNum 94 snap 16 seq_num 94) 2024-09-13T14:21:16.926 INFO:tasks.rados.rados.0.smithi151.stdout:154: expect (ObjNum 72 snap 9 seq_num 72) 2024-09-13T14:21:17.075 INFO:tasks.rados.rados.0.smithi151.stdout:158: seq_num 96 ranges {413472=620140,1818036=510700,3120019=493453} 2024-09-13T14:21:17.081 INFO:tasks.rados.rados.0.smithi151.stdout:158: writing smithi15186934-32 from 413472 to 1033612 tid 1 2024-09-13T14:21:17.089 INFO:tasks.rados.rados.0.smithi151.stdout:158: writing smithi15186934-32 from 1818036 to 2328736 tid 2 2024-09-13T14:21:17.097 INFO:tasks.rados.rados.0.smithi151.stdout:158: writing smithi15186934-32 from 3120019 to 3613472 tid 3 2024-09-13T14:21:17.097 INFO:tasks.rados.rados.0.smithi151.stdout:153: done (2 left) 2024-09-13T14:21:17.097 INFO:tasks.rados.rados.0.smithi151.stdout:154: done (1 left) 2024-09-13T14:21:17.097 INFO:tasks.rados.rados.0.smithi151.stdout:159: read oid 41 snap -1 2024-09-13T14:21:17.097 INFO:tasks.rados.rados.0.smithi151.stdout:159: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-13T14:21:17.097 INFO:tasks.rados.rados.0.smithi151.stdout:160: delete oid 28 current snap is 16 2024-09-13T14:21:17.097 INFO:tasks.rados.rados.0.smithi151.stdout:158: finishing write tid 1 to smithi15186934-32 2024-09-13T14:21:17.097 INFO:tasks.rados.rados.0.smithi151.stdout:158: finishing write tid 2 to smithi15186934-32 2024-09-13T14:21:17.099 INFO:tasks.rados.rados.0.smithi151.stdout:160: done (2 left) 2024-09-13T14:21:17.099 INFO:tasks.rados.rados.0.smithi151.stdout:161: write oid 44 current snap is 16 2024-09-13T14:21:17.099 INFO:tasks.rados.rados.0.smithi151.stdout:161: seq_num 98 ranges {447086=770638,1935000=798016,3458512=188574} 2024-09-13T14:21:17.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:16 smithi184 ceph-mon[33394]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:17.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:16 smithi184 ceph-mon[33394]: osdmap e95: 8 total, 8 up, 8 in 2024-09-13T14:21:17.106 INFO:tasks.rados.rados.0.smithi151.stdout:161: writing smithi15186934-44 from 447086 to 1217724 tid 1 2024-09-13T14:21:17.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:16 smithi151 ceph-mon[41387]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:17.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:16 smithi151 ceph-mon[41387]: osdmap e95: 8 total, 8 up, 8 in 2024-09-13T14:21:17.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:16 smithi151 ceph-mon[29807]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:17.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:16 smithi151 ceph-mon[29807]: osdmap e95: 8 total, 8 up, 8 in 2024-09-13T14:21:17.115 INFO:tasks.rados.rados.0.smithi151.stdout:161: writing smithi15186934-44 from 1935000 to 2733016 tid 2 2024-09-13T14:21:17.120 INFO:tasks.rados.rados.0.smithi151.stdout:161: writing smithi15186934-44 from 3458512 to 3647086 tid 3 2024-09-13T14:21:17.120 INFO:tasks.rados.rados.0.smithi151.stdout:162: snap_remove snap 12 2024-09-13T14:21:17.774 INFO:tasks.rados.rados.0.smithi151.stdout:162: done (3 left) 2024-09-13T14:21:17.774 INFO:tasks.rados.rados.0.smithi151.stdout:163: rollback oid 15 current snap is 16 2024-09-13T14:21:17.774 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 15 to 0 2024-09-13T14:21:17.775 INFO:tasks.rados.rados.0.smithi151.stdout:164: read oid 21 snap -1 2024-09-13T14:21:17.775 INFO:tasks.rados.rados.0.smithi151.stdout:164: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-13T14:21:17.775 INFO:tasks.rados.rados.0.smithi151.stdout:165: read oid 17 snap 13 2024-09-13T14:21:17.775 INFO:tasks.rados.rados.0.smithi151.stdout:165: expect deleted 2024-09-13T14:21:17.775 INFO:tasks.rados.rados.0.smithi151.stdout:166: read oid 18 snap 13 2024-09-13T14:21:17.775 INFO:tasks.rados.rados.0.smithi151.stdout:166: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-13T14:21:17.775 INFO:tasks.rados.rados.0.smithi151.stdout:167: write oid 24 current snap is 16 2024-09-13T14:21:17.775 INFO:tasks.rados.rados.0.smithi151.stdout:158: finishing write tid 3 to smithi15186934-32 2024-09-13T14:21:17.775 INFO:tasks.rados.rados.0.smithi151.stdout:158: finishing write tid 4 to smithi15186934-32 2024-09-13T14:21:17.775 INFO:tasks.rados.rados.0.smithi151.stdout:158: finishing write tid 5 to smithi15186934-32 2024-09-13T14:21:17.776 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 32 v 66 (ObjNum 96 snap 16 seq_num 96) dirty exists 2024-09-13T14:21:17.776 INFO:tasks.rados.rados.0.smithi151.stdout:158: left oid 32 (ObjNum 96 snap 16 seq_num 96) 2024-09-13T14:21:17.776 INFO:tasks.rados.rados.0.smithi151.stdout:159: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-13T14:21:17.825 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:21:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:21:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:21:17.836 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:21:17.881 INFO:tasks.rados.rados.0.smithi151.stdout:161: finishing write tid 1 to smithi15186934-44 2024-09-13T14:21:17.881 INFO:tasks.rados.rados.0.smithi151.stdout:161: finishing write tid 2 to smithi15186934-44 2024-09-13T14:21:17.881 INFO:tasks.rados.rados.0.smithi151.stdout:161: finishing write tid 3 to smithi15186934-44 2024-09-13T14:21:17.881 INFO:tasks.rados.rados.0.smithi151.stdout:161: finishing write tid 4 to smithi15186934-44 2024-09-13T14:21:17.881 INFO:tasks.rados.rados.0.smithi151.stdout:161: finishing write tid 5 to smithi15186934-44 2024-09-13T14:21:17.881 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 21 (ObjNum 98 snap 16 seq_num 98) dirty exists 2024-09-13T14:21:17.881 INFO:tasks.rados.rados.0.smithi151.stdout:161: left oid 44 (ObjNum 98 snap 16 seq_num 98) 2024-09-13T14:21:17.881 INFO:tasks.rados.rados.0.smithi151.stdout:167: seq_num 99 ranges {463893=562240,1593482=658027,2685935=559341} 2024-09-13T14:21:17.886 INFO:tasks.rados.rados.0.smithi151.stdout:167: writing smithi15186934-24 from 463893 to 1026133 tid 1 2024-09-13T14:21:17.894 INFO:tasks.rados.rados.0.smithi151.stdout:167: writing smithi15186934-24 from 1593482 to 2251509 tid 2 2024-09-13T14:21:17.900 INFO:tasks.rados.rados.0.smithi151.stdout:167: writing smithi15186934-24 from 2685935 to 3245276 tid 3 2024-09-13T14:21:17.900 INFO:tasks.rados.rados.0.smithi151.stdout:158: done (7 left) 2024-09-13T14:21:17.900 INFO:tasks.rados.rados.0.smithi151.stdout:159: done (6 left) 2024-09-13T14:21:17.901 INFO:tasks.rados.rados.0.smithi151.stdout:161: done (5 left) 2024-09-13T14:21:17.901 INFO:tasks.rados.rados.0.smithi151.stdout:168: write oid 9 current snap is 16 2024-09-13T14:21:17.901 INFO:tasks.rados.rados.0.smithi151.stdout:168: seq_num 100 ranges {480700=637489,1651964=401685,2713358=770613} 2024-09-13T14:21:17.906 INFO:tasks.rados.rados.0.smithi151.stdout:168: writing smithi15186934-9 from 480700 to 1118189 tid 1 2024-09-13T14:21:17.912 INFO:tasks.rados.rados.0.smithi151.stdout:168: writing smithi15186934-9 from 1651964 to 2053649 tid 2 2024-09-13T14:21:17.920 INFO:tasks.rados.rados.0.smithi151.stdout:168: writing smithi15186934-9 from 2713358 to 3483971 tid 3 2024-09-13T14:21:17.920 INFO:tasks.rados.rados.0.smithi151.stdout:169: write oid 19 current snap is 16 2024-09-13T14:21:17.920 INFO:tasks.rados.rados.0.smithi151.stdout:163: finishing rollback tid 0 to smithi15186934-15 2024-09-13T14:21:17.920 INFO:tasks.rados.rados.0.smithi151.stdout:163: finishing rollback tid 1 to smithi15186934-15 2024-09-13T14:21:17.920 INFO:tasks.rados.rados.0.smithi151.stdout:169: seq_num 101 ranges {497507=712738,1710446=661696,3140781=556726} 2024-09-13T14:21:17.928 INFO:tasks.rados.rados.0.smithi151.stdout:169: writing smithi15186934-19 from 497507 to 1210245 tid 1 2024-09-13T14:21:17.935 INFO:tasks.rados.rados.0.smithi151.stdout:169: writing smithi15186934-19 from 1710446 to 2372142 tid 2 2024-09-13T14:21:17.943 INFO:tasks.rados.rados.0.smithi151.stdout:169: writing smithi15186934-19 from 3140781 to 3697507 tid 3 2024-09-13T14:21:17.943 INFO:tasks.rados.rados.0.smithi151.stdout:165: done (6 left) 2024-09-13T14:21:17.943 INFO:tasks.rados.rados.0.smithi151.stdout:170: write oid 4 current snap is 16 2024-09-13T14:21:17.944 INFO:tasks.rados.rados.0.smithi151.stdout:163: finishing rollback tid 2 to smithi15186934-15 2024-09-13T14:21:17.944 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 15 v 63 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-13T14:21:17.944 INFO:tasks.rados.rados.0.smithi151.stdout:166: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-13T14:21:18.030 INFO:tasks.rados.rados.0.smithi151.stdout:164: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-13T14:21:18.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:17 smithi184 ceph-mon[33394]: pgmap v130: 89 pgs: 89 active+clean; 153 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.6 MiB/s rd, 752 KiB/s wr, 7 op/s 2024-09-13T14:21:18.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:17 smithi184 ceph-mon[33394]: osdmap e96: 8 total, 8 up, 8 in 2024-09-13T14:21:18.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:17 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/2416652720' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:18.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:17 smithi184 ceph-mon[33394]: osdmap e97: 8 total, 8 up, 8 in 2024-09-13T14:21:18.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:17 smithi151 ceph-mon[29807]: pgmap v130: 89 pgs: 89 active+clean; 153 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.6 MiB/s rd, 752 KiB/s wr, 7 op/s 2024-09-13T14:21:18.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:17 smithi151 ceph-mon[29807]: osdmap e96: 8 total, 8 up, 8 in 2024-09-13T14:21:18.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:17 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/2416652720' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:18.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:17 smithi151 ceph-mon[29807]: osdmap e97: 8 total, 8 up, 8 in 2024-09-13T14:21:18.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:17 smithi151 ceph-mon[41387]: pgmap v130: 89 pgs: 89 active+clean; 153 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.6 MiB/s rd, 752 KiB/s wr, 7 op/s 2024-09-13T14:21:18.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:17 smithi151 ceph-mon[41387]: osdmap e96: 8 total, 8 up, 8 in 2024-09-13T14:21:18.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:17 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/2416652720' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:18.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:17 smithi151 ceph-mon[41387]: osdmap e97: 8 total, 8 up, 8 in 2024-09-13T14:21:18.142 INFO:tasks.rados.rados.0.smithi151.stdout:170: seq_num 102 ranges {514314=787987,1768928=405354,2768204=509510} 2024-09-13T14:21:18.150 INFO:tasks.rados.rados.0.smithi151.stdout:170: writing smithi15186934-4 from 514314 to 1302301 tid 1 2024-09-13T14:21:18.155 INFO:tasks.rados.rados.0.smithi151.stdout:170: writing smithi15186934-4 from 1768928 to 2174282 tid 2 2024-09-13T14:21:18.163 INFO:tasks.rados.rados.0.smithi151.stdout:170: writing smithi15186934-4 from 2768204 to 3277714 tid 3 2024-09-13T14:21:18.164 INFO:tasks.rados.rados.0.smithi151.stdout:163: done (6 left) 2024-09-13T14:21:18.164 INFO:tasks.rados.rados.0.smithi151.stdout:164: done (5 left) 2024-09-13T14:21:18.164 INFO:tasks.rados.rados.0.smithi151.stdout:166: done (4 left) 2024-09-13T14:21:18.164 INFO:tasks.rados.rados.0.smithi151.stdout:171: rollback oid 47 current snap is 16 2024-09-13T14:21:18.164 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 47 to 15 2024-09-13T14:21:18.164 INFO:tasks.rados.rados.0.smithi151.stdout:167: finishing write tid 1 to smithi15186934-24 2024-09-13T14:21:18.164 INFO:tasks.rados.rados.0.smithi151.stdout:167: finishing write tid 2 to smithi15186934-24 2024-09-13T14:21:18.164 INFO:tasks.rados.rados.0.smithi151.stdout:167: finishing write tid 3 to smithi15186934-24 2024-09-13T14:21:18.164 INFO:tasks.rados.rados.0.smithi151.stdout:167: finishing write tid 4 to smithi15186934-24 2024-09-13T14:21:18.164 INFO:tasks.rados.rados.0.smithi151.stdout:167: finishing write tid 5 to smithi15186934-24 2024-09-13T14:21:18.164 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 24 v 76 (ObjNum 99 snap 16 seq_num 99) dirty exists 2024-09-13T14:21:18.165 INFO:tasks.rados.rados.0.smithi151.stdout:167: left oid 24 (ObjNum 99 snap 16 seq_num 99) 2024-09-13T14:21:18.165 INFO:tasks.rados.rados.0.smithi151.stdout:168: finishing write tid 1 to smithi15186934-9 2024-09-13T14:21:18.165 INFO:tasks.rados.rados.0.smithi151.stdout:168: finishing write tid 2 to smithi15186934-9 2024-09-13T14:21:18.165 INFO:tasks.rados.rados.0.smithi151.stdout:168: finishing write tid 3 to smithi15186934-9 2024-09-13T14:21:18.165 INFO:tasks.rados.rados.0.smithi151.stdout:168: finishing write tid 4 to smithi15186934-9 2024-09-13T14:21:18.165 INFO:tasks.rados.rados.0.smithi151.stdout:168: finishing write tid 5 to smithi15186934-9 2024-09-13T14:21:18.165 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 77 (ObjNum 100 snap 16 seq_num 100) dirty exists 2024-09-13T14:21:18.165 INFO:tasks.rados.rados.0.smithi151.stdout:168: left oid 9 (ObjNum 100 snap 16 seq_num 100) 2024-09-13T14:21:18.165 INFO:tasks.rados.rados.0.smithi151.stdout:169: finishing write tid 1 to smithi15186934-19 2024-09-13T14:21:18.165 INFO:tasks.rados.rados.0.smithi151.stdout:169: finishing write tid 2 to smithi15186934-19 2024-09-13T14:21:18.165 INFO:tasks.rados.rados.0.smithi151.stdout:169: finishing write tid 3 to smithi15186934-19 2024-09-13T14:21:18.165 INFO:tasks.rados.rados.0.smithi151.stdout:169: finishing write tid 4 to smithi15186934-19 2024-09-13T14:21:18.165 INFO:tasks.rados.rados.0.smithi151.stdout:169: finishing write tid 5 to smithi15186934-19 2024-09-13T14:21:18.165 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 19 v 40 (ObjNum 101 snap 16 seq_num 101) dirty exists 2024-09-13T14:21:18.166 INFO:tasks.rados.rados.0.smithi151.stdout:169: left oid 19 (ObjNum 101 snap 16 seq_num 101) 2024-09-13T14:21:18.166 INFO:tasks.rados.rados.0.smithi151.stdout:170: finishing write tid 1 to smithi15186934-4 2024-09-13T14:21:18.166 INFO:tasks.rados.rados.0.smithi151.stdout:170: finishing write tid 2 to smithi15186934-4 2024-09-13T14:21:18.166 INFO:tasks.rados.rados.0.smithi151.stdout:167: done (4 left) 2024-09-13T14:21:18.166 INFO:tasks.rados.rados.0.smithi151.stdout:168: done (3 left) 2024-09-13T14:21:18.166 INFO:tasks.rados.rados.0.smithi151.stdout:169: done (2 left) 2024-09-13T14:21:18.166 INFO:tasks.rados.rados.0.smithi151.stdout:172: rollback oid 25 current snap is 16 2024-09-13T14:21:18.166 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 25 to 14 2024-09-13T14:21:18.166 INFO:tasks.rados.rados.0.smithi151.stdout:173: write oid 17 current snap is 16 2024-09-13T14:21:18.166 INFO:tasks.rados.rados.0.smithi151.stdout:173: seq_num 103 ranges {531121=463236,1711057=665365,3079274=651847} 2024-09-13T14:21:18.168 INFO:tasks.rados.rados.0.smithi151.stdout:173: writing smithi15186934-17 from 531121 to 994357 tid 1 2024-09-13T14:21:18.176 INFO:tasks.rados.rados.0.smithi151.stdout:173: writing smithi15186934-17 from 1711057 to 2376422 tid 2 2024-09-13T14:21:18.185 INFO:tasks.rados.rados.0.smithi151.stdout:173: writing smithi15186934-17 from 3079274 to 3731121 tid 3 2024-09-13T14:21:18.186 INFO:tasks.rados.rados.0.smithi151.stdout:174: read oid 8 snap -1 2024-09-13T14:21:18.186 INFO:tasks.rados.rados.0.smithi151.stdout:174: expect (ObjNum 51 snap 0 seq_num 51) 2024-09-13T14:21:18.186 INFO:tasks.rados.rados.0.smithi151.stdout:175: read oid 50 snap -1 2024-09-13T14:21:18.186 INFO:tasks.rados.rados.0.smithi151.stdout:175: expect (ObjNum 94 snap 16 seq_num 94) 2024-09-13T14:21:18.186 INFO:tasks.rados.rados.0.smithi151.stdout:176: rollback oid 39 current snap is 16 2024-09-13T14:21:18.186 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 39 to 15 2024-09-13T14:21:18.186 INFO:tasks.rados.rados.0.smithi151.stdout:171: finishing rollback tid 0 to smithi15186934-47 2024-09-13T14:21:18.186 INFO:tasks.rados.rados.0.smithi151.stdout:171: finishing rollback tid 1 to smithi15186934-47 2024-09-13T14:21:18.186 INFO:tasks.rados.rados.0.smithi151.stdout:171: finishing rollback tid 2 to smithi15186934-47 2024-09-13T14:21:18.186 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 79 (ObjNum 79 snap 10 seq_num 79) dirty exists 2024-09-13T14:21:18.186 INFO:tasks.rados.rados.0.smithi151.stdout:170: finishing write tid 3 to smithi15186934-4 2024-09-13T14:21:18.186 INFO:tasks.rados.rados.0.smithi151.stdout:170: finishing write tid 4 to smithi15186934-4 2024-09-13T14:21:18.187 INFO:tasks.rados.rados.0.smithi151.stdout:170: finishing write tid 5 to smithi15186934-4 2024-09-13T14:21:18.187 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 31 (ObjNum 102 snap 16 seq_num 102) dirty exists 2024-09-13T14:21:18.187 INFO:tasks.rados.rados.0.smithi151.stdout:170: left oid 4 (ObjNum 102 snap 16 seq_num 102) 2024-09-13T14:21:18.187 INFO:tasks.rados.rados.0.smithi151.stdout:172: finishing rollback tid 0 to smithi15186934-25 2024-09-13T14:21:18.187 INFO:tasks.rados.rados.0.smithi151.stdout:172: finishing rollback tid 1 to smithi15186934-25 2024-09-13T14:21:18.187 INFO:tasks.rados.rados.0.smithi151.stdout:172: finishing rollback tid 2 to smithi15186934-25 2024-09-13T14:21:18.187 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 35 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-13T14:21:18.187 INFO:tasks.rados.rados.0.smithi151.stdout:173: finishing write tid 1 to smithi15186934-17 2024-09-13T14:21:18.187 INFO:tasks.rados.rados.0.smithi151.stdout:173: finishing write tid 2 to smithi15186934-17 2024-09-13T14:21:18.187 INFO:tasks.rados.rados.0.smithi151.stdout:170: done (6 left) 2024-09-13T14:21:18.187 INFO:tasks.rados.rados.0.smithi151.stdout:171: done (5 left) 2024-09-13T14:21:18.187 INFO:tasks.rados.rados.0.smithi151.stdout:172: done (4 left) 2024-09-13T14:21:18.187 INFO:tasks.rados.rados.0.smithi151.stdout:177: write oid 18 current snap is 16 2024-09-13T14:21:18.187 INFO:tasks.rados.rados.0.smithi151.stdout:177: seq_num 104 ranges {547928=538485,1769539=409023,2706697=532054} 2024-09-13T14:21:18.194 INFO:tasks.rados.rados.0.smithi151.stdout:177: writing smithi15186934-18 from 547928 to 1086413 tid 1 2024-09-13T14:21:18.200 INFO:tasks.rados.rados.0.smithi151.stdout:177: writing smithi15186934-18 from 1769539 to 2178562 tid 2 2024-09-13T14:21:18.208 INFO:tasks.rados.rados.0.smithi151.stdout:177: writing smithi15186934-18 from 2706697 to 3238751 tid 3 2024-09-13T14:21:18.208 INFO:tasks.rados.rados.0.smithi151.stdout:178: read oid 21 snap 11 2024-09-13T14:21:18.208 INFO:tasks.rados.rados.0.smithi151.stdout:178: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-13T14:21:18.208 INFO:tasks.rados.rados.0.smithi151.stdout:179: rollback oid 42 current snap is 16 2024-09-13T14:21:18.208 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 42 to 14 2024-09-13T14:21:18.208 INFO:tasks.rados.rados.0.smithi151.stdout:173: finishing write tid 3 to smithi15186934-17 2024-09-13T14:21:18.208 INFO:tasks.rados.rados.0.smithi151.stdout:173: finishing write tid 4 to smithi15186934-17 2024-09-13T14:21:18.208 INFO:tasks.rados.rados.0.smithi151.stdout:173: finishing write tid 5 to smithi15186934-17 2024-09-13T14:21:18.209 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 80 (ObjNum 103 snap 16 seq_num 103) dirty exists 2024-09-13T14:21:18.209 INFO:tasks.rados.rados.0.smithi151.stdout:173: left oid 17 (ObjNum 103 snap 16 seq_num 103) 2024-09-13T14:21:18.209 INFO:tasks.rados.rados.0.smithi151.stdout:174: expect (ObjNum 51 snap 0 seq_num 51) 2024-09-13T14:21:18.376 INFO:tasks.rados.rados.0.smithi151.stdout:173: done (6 left) 2024-09-13T14:21:18.377 INFO:tasks.rados.rados.0.smithi151.stdout:174: done (5 left) 2024-09-13T14:21:18.377 INFO:tasks.rados.rados.0.smithi151.stdout:180: rollback oid 28 current snap is 16 2024-09-13T14:21:18.377 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 28 to 11 2024-09-13T14:21:18.377 INFO:tasks.rados.rados.0.smithi151.stdout:177: finishing write tid 1 to smithi15186934-18 2024-09-13T14:21:18.377 INFO:tasks.rados.rados.0.smithi151.stdout:175: expect (ObjNum 94 snap 16 seq_num 94) 2024-09-13T14:21:18.610 INFO:tasks.rados.rados.0.smithi151.stdout:176: finishing rollback tid 0 to smithi15186934-39 2024-09-13T14:21:18.610 INFO:tasks.rados.rados.0.smithi151.stdout:176: finishing rollback tid 1 to smithi15186934-39 2024-09-13T14:21:18.610 INFO:tasks.rados.rados.0.smithi151.stdout:176: finishing rollback tid 2 to smithi15186934-39 2024-09-13T14:21:18.611 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 52 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-13T14:21:18.611 INFO:tasks.rados.rados.0.smithi151.stdout:177: finishing write tid 2 to smithi15186934-18 2024-09-13T14:21:18.611 INFO:tasks.rados.rados.0.smithi151.stdout:175: done (5 left) 2024-09-13T14:21:18.611 INFO:tasks.rados.rados.0.smithi151.stdout:176: done (4 left) 2024-09-13T14:21:18.611 INFO:tasks.rados.rados.0.smithi151.stdout:181: read oid 37 snap 14 2024-09-13T14:21:18.611 INFO:tasks.rados.rados.0.smithi151.stdout:181: expect (ObjNum 89 snap 11 seq_num 89) 2024-09-13T14:21:18.611 INFO:tasks.rados.rados.0.smithi151.stdout:182: read oid 34 snap -1 2024-09-13T14:21:18.611 INFO:tasks.rados.rados.0.smithi151.stdout:182: expect deleted 2024-09-13T14:21:18.611 INFO:tasks.rados.rados.0.smithi151.stdout:183: write oid 15 current snap is 16 2024-09-13T14:21:18.611 INFO:tasks.rados.rados.0.smithi151.stdout:179: finishing rollback tid 0 to smithi15186934-42 2024-09-13T14:21:18.611 INFO:tasks.rados.rados.0.smithi151.stdout:178: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-13T14:21:18.709 INFO:tasks.rados.rados.0.smithi151.stdout:179: finishing rollback tid 1 to smithi15186934-42 2024-09-13T14:21:18.709 INFO:tasks.rados.rados.0.smithi151.stdout:179: finishing rollback tid 2 to smithi15186934-42 2024-09-13T14:21:18.709 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 69 (ObjNum 87 snap 11 seq_num 87) dirty exists 2024-09-13T14:21:18.709 INFO:tasks.rados.rados.0.smithi151.stdout:177: finishing write tid 3 to smithi15186934-18 2024-09-13T14:21:18.709 INFO:tasks.rados.rados.0.smithi151.stdout:177: finishing write tid 4 to smithi15186934-18 2024-09-13T14:21:18.709 INFO:tasks.rados.rados.0.smithi151.stdout:177: finishing write tid 5 to smithi15186934-18 2024-09-13T14:21:18.709 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 18 v 14 (ObjNum 104 snap 16 seq_num 104) dirty exists 2024-09-13T14:21:18.709 INFO:tasks.rados.rados.0.smithi151.stdout:177: left oid 18 (ObjNum 104 snap 16 seq_num 104) 2024-09-13T14:21:18.709 INFO:tasks.rados.rados.0.smithi151.stdout:180: finishing rollback tid 1 to smithi15186934-28 2024-09-13T14:21:18.709 INFO:tasks.rados.rados.0.smithi151.stdout:180: finishing rollback tid 2 to smithi15186934-28 2024-09-13T14:21:18.709 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 28 v 71 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-13T14:21:18.710 INFO:tasks.rados.rados.0.smithi151.stdout:183: seq_num 105 ranges {564735=613734,1828021=669034,3134120=630615} 2024-09-13T14:21:18.715 INFO:tasks.rados.rados.0.smithi151.stdout:183: writing smithi15186934-15 from 564735 to 1178469 tid 1 2024-09-13T14:21:18.723 INFO:tasks.rados.rados.0.smithi151.stdout:183: writing smithi15186934-15 from 1828021 to 2497055 tid 2 2024-09-13T14:21:18.730 INFO:tasks.rados.rados.0.smithi151.stdout:183: writing smithi15186934-15 from 3134120 to 3764735 tid 3 2024-09-13T14:21:18.731 INFO:tasks.rados.rados.0.smithi151.stdout:177: done (6 left) 2024-09-13T14:21:18.731 INFO:tasks.rados.rados.0.smithi151.stdout:178: done (5 left) 2024-09-13T14:21:18.731 INFO:tasks.rados.rados.0.smithi151.stdout:179: done (4 left) 2024-09-13T14:21:18.731 INFO:tasks.rados.rados.0.smithi151.stdout:180: done (3 left) 2024-09-13T14:21:18.731 INFO:tasks.rados.rados.0.smithi151.stdout:184: write oid 35 current snap is 16 2024-09-13T14:21:18.731 INFO:tasks.rados.rados.0.smithi151.stdout:184: seq_num 106 ranges {581542=688983,1886503=412692,2761543=670951} 2024-09-13T14:21:18.738 INFO:tasks.rados.rados.0.smithi151.stdout:184: writing smithi15186934-35 from 581542 to 1270525 tid 1 2024-09-13T14:21:18.744 INFO:tasks.rados.rados.0.smithi151.stdout:184: writing smithi15186934-35 from 1886503 to 2299195 tid 2 2024-09-13T14:21:18.752 INFO:tasks.rados.rados.0.smithi151.stdout:184: writing smithi15186934-35 from 2761543 to 3432494 tid 3 2024-09-13T14:21:18.752 INFO:tasks.rados.rados.0.smithi151.stdout:185: delete oid 13 current snap is 16 2024-09-13T14:21:18.752 INFO:tasks.rados.rados.0.smithi151.stdout:181: expect (ObjNum 89 snap 11 seq_num 89) 2024-09-13T14:21:18.928 INFO:tasks.rados.rados.0.smithi151.stdout:181: done (4 left) 2024-09-13T14:21:18.929 INFO:tasks.rados.rados.0.smithi151.stdout:182: done (3 left) 2024-09-13T14:21:18.929 INFO:tasks.rados.rados.0.smithi151.stdout:185: done (2 left) 2024-09-13T14:21:18.929 INFO:tasks.rados.rados.0.smithi151.stdout:186: delete oid 12 current snap is 16 2024-09-13T14:21:18.929 INFO:tasks.rados.rados.0.smithi151.stdout:183: finishing write tid 1 to smithi15186934-15 2024-09-13T14:21:18.929 INFO:tasks.rados.rados.0.smithi151.stdout:183: finishing write tid 2 to smithi15186934-15 2024-09-13T14:21:18.929 INFO:tasks.rados.rados.0.smithi151.stdout:183: finishing write tid 3 to smithi15186934-15 2024-09-13T14:21:18.929 INFO:tasks.rados.rados.0.smithi151.stdout:183: finishing write tid 4 to smithi15186934-15 2024-09-13T14:21:18.929 INFO:tasks.rados.rados.0.smithi151.stdout:183: finishing write tid 5 to smithi15186934-15 2024-09-13T14:21:18.929 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 15 v 75 (ObjNum 105 snap 16 seq_num 105) dirty exists 2024-09-13T14:21:18.929 INFO:tasks.rados.rados.0.smithi151.stdout:183: left oid 15 (ObjNum 105 snap 16 seq_num 105) 2024-09-13T14:21:18.929 INFO:tasks.rados.rados.0.smithi151.stdout:184: finishing write tid 1 to smithi15186934-35 2024-09-13T14:21:18.929 INFO:tasks.rados.rados.0.smithi151.stdout:184: finishing write tid 2 to smithi15186934-35 2024-09-13T14:21:18.929 INFO:tasks.rados.rados.0.smithi151.stdout:184: finishing write tid 3 to smithi15186934-35 2024-09-13T14:21:18.930 INFO:tasks.rados.rados.0.smithi151.stdout:184: finishing write tid 4 to smithi15186934-35 2024-09-13T14:21:18.930 INFO:tasks.rados.rados.0.smithi151.stdout:184: finishing write tid 5 to smithi15186934-35 2024-09-13T14:21:18.930 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 35 v 85 (ObjNum 106 snap 16 seq_num 106) dirty exists 2024-09-13T14:21:18.930 INFO:tasks.rados.rados.0.smithi151.stdout:184: left oid 35 (ObjNum 106 snap 16 seq_num 106) 2024-09-13T14:21:18.930 INFO:tasks.rados.rados.0.smithi151.stdout:183: done (2 left) 2024-09-13T14:21:18.930 INFO:tasks.rados.rados.0.smithi151.stdout:184: done (1 left) 2024-09-13T14:21:18.930 INFO:tasks.rados.rados.0.smithi151.stdout:186: done (0 left) 2024-09-13T14:21:18.931 INFO:tasks.rados.rados.0.smithi151.stdout:187: snap_create 2024-09-13T14:21:19.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:18 smithi184 ceph-mon[33394]: from='client.14964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:19.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:18 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:21:19.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:18 smithi151 ceph-mon[29807]: from='client.14964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:19.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:18 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:21:19.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:18 smithi151 ceph-mon[41387]: from='client.14964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:19.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:18 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:21:19.841 INFO:tasks.rados.rados.0.smithi151.stdout:187: done (0 left) 2024-09-13T14:21:19.841 INFO:tasks.rados.rados.0.smithi151.stdout:188: read oid 42 snap -1 2024-09-13T14:21:19.841 INFO:tasks.rados.rados.0.smithi151.stdout:188: expect (ObjNum 87 snap 11 seq_num 87) 2024-09-13T14:21:19.842 INFO:tasks.rados.rados.0.smithi151.stdout:189: rollback oid 30 current snap is 17 2024-09-13T14:21:19.842 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 30 to 14 2024-09-13T14:21:19.842 INFO:tasks.rados.rados.0.smithi151.stdout:190: delete oid 20 current snap is 17 2024-09-13T14:21:19.845 INFO:tasks.rados.rados.0.smithi151.stdout:189: finishing rollback tid 0 to smithi15186934-30 2024-09-13T14:21:19.845 INFO:tasks.rados.rados.0.smithi151.stdout:190: done (2 left) 2024-09-13T14:21:19.845 INFO:tasks.rados.rados.0.smithi151.stdout:191: snap_remove snap 0 2024-09-13T14:21:20.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:19 smithi184 ceph-mon[33394]: from='client.24872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:20.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:19 smithi184 ceph-mon[33394]: pgmap v133: 89 pgs: 89 active+clean; 153 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:20.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:19 smithi151 ceph-mon[29807]: from='client.24872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:20.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:19 smithi151 ceph-mon[29807]: pgmap v133: 89 pgs: 89 active+clean; 153 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:20.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:19 smithi151 ceph-mon[41387]: from='client.24872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:20.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:19 smithi151 ceph-mon[41387]: pgmap v133: 89 pgs: 89 active+clean; 153 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:20.806 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:21:20.850 INFO:tasks.rados.rados.0.smithi151.stdout:191: done (2 left) 2024-09-13T14:21:20.850 INFO:tasks.rados.rados.0.smithi151.stdout:192: snap_remove snap 15 2024-09-13T14:21:21.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:20 smithi184 ceph-mon[33394]: osdmap e98: 8 total, 8 up, 8 in 2024-09-13T14:21:21.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:20 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/439162257' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:21.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:20 smithi151 ceph-mon[29807]: osdmap e98: 8 total, 8 up, 8 in 2024-09-13T14:21:21.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:20 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/439162257' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:21.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:20 smithi151 ceph-mon[41387]: osdmap e98: 8 total, 8 up, 8 in 2024-09-13T14:21:21.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:20 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/439162257' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:21.852 INFO:tasks.rados.rados.0.smithi151.stdout:192: done (2 left) 2024-09-13T14:21:21.852 INFO:tasks.rados.rados.0.smithi151.stdout:193: read oid 11 snap 11 2024-09-13T14:21:21.852 INFO:tasks.rados.rados.0.smithi151.stdout:193: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-13T14:21:21.852 INFO:tasks.rados.rados.0.smithi151.stdout:194: snap_create 2024-09-13T14:21:21.852 INFO:tasks.rados.rados.0.smithi151.stdout:189: finishing rollback tid 1 to smithi15186934-30 2024-09-13T14:21:21.852 INFO:tasks.rados.rados.0.smithi151.stdout:189: finishing rollback tid 2 to smithi15186934-30 2024-09-13T14:21:21.852 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 30 v 77 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-13T14:21:21.852 INFO:tasks.rados.rados.0.smithi151.stdout:188: expect (ObjNum 87 snap 11 seq_num 87) 2024-09-13T14:21:22.042 INFO:tasks.rados.rados.0.smithi151.stdout:193: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-13T14:21:22.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:21 smithi151 ceph-mon[29807]: pgmap v135: 89 pgs: 89 active+clean; 183 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 4.3 MiB/s wr, 36 op/s 2024-09-13T14:21:22.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:21 smithi151 ceph-mon[29807]: from='client.24884 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:22.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:21 smithi151 ceph-mon[29807]: osdmap e99: 8 total, 8 up, 8 in 2024-09-13T14:21:22.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:21 smithi151 ceph-mon[41387]: pgmap v135: 89 pgs: 89 active+clean; 183 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 4.3 MiB/s wr, 36 op/s 2024-09-13T14:21:22.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:21 smithi151 ceph-mon[41387]: from='client.24884 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:22.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:21 smithi151 ceph-mon[41387]: osdmap e99: 8 total, 8 up, 8 in 2024-09-13T14:21:22.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:21 smithi184 ceph-mon[33394]: pgmap v135: 89 pgs: 89 active+clean; 183 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 4.3 MiB/s wr, 36 op/s 2024-09-13T14:21:22.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:21 smithi184 ceph-mon[33394]: from='client.24884 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:22.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:21 smithi184 ceph-mon[33394]: osdmap e99: 8 total, 8 up, 8 in 2024-09-13T14:21:22.775 INFO:tasks.rados.rados.0.smithi151.stdout:188: done (3 left) 2024-09-13T14:21:22.775 INFO:tasks.rados.rados.0.smithi151.stdout:189: done (2 left) 2024-09-13T14:21:22.775 INFO:tasks.rados.rados.0.smithi151.stdout:193: done (1 left) 2024-09-13T14:21:22.775 INFO:tasks.rados.rados.0.smithi151.stdout:194: done (0 left) 2024-09-13T14:21:22.775 INFO:tasks.rados.rados.0.smithi151.stdout:195: delete oid 45 current snap is 18 2024-09-13T14:21:22.777 INFO:tasks.rados.rados.0.smithi151.stdout:195: done (0 left) 2024-09-13T14:21:22.778 INFO:tasks.rados.rados.0.smithi151.stdout:196: read oid 12 snap 13 2024-09-13T14:21:22.778 INFO:tasks.rados.rados.0.smithi151.stdout:196: expect (ObjNum 68 snap 7 seq_num 68) 2024-09-13T14:21:22.778 INFO:tasks.rados.rados.0.smithi151.stdout:197: rollback oid 44 current snap is 18 2024-09-13T14:21:22.778 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 44 to 17 2024-09-13T14:21:22.778 INFO:tasks.rados.rados.0.smithi151.stdout:198: snap_remove snap 14 2024-09-13T14:21:22.915 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:22 smithi151 ceph-mon[29807]: from='client.24890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:22.915 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:22 smithi151 ceph-mon[29807]: osdmap e100: 8 total, 8 up, 8 in 2024-09-13T14:21:22.915 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:22 smithi151 ceph-mon[29807]: osdmap e101: 8 total, 8 up, 8 in 2024-09-13T14:21:22.915 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:22 smithi151 ceph-mon[41387]: from='client.24890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:22.915 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:22 smithi151 ceph-mon[41387]: osdmap e100: 8 total, 8 up, 8 in 2024-09-13T14:21:22.915 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:22 smithi151 ceph-mon[41387]: osdmap e101: 8 total, 8 up, 8 in 2024-09-13T14:21:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:22 smithi184 ceph-mon[33394]: from='client.24890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:22 smithi184 ceph-mon[33394]: osdmap e100: 8 total, 8 up, 8 in 2024-09-13T14:21:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:22 smithi184 ceph-mon[33394]: osdmap e101: 8 total, 8 up, 8 in 2024-09-13T14:21:23.358 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:21:22 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:21:22] "GET /metrics HTTP/1.1" 200 243175 "" "Prometheus/2.43.0" 2024-09-13T14:21:23.678 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:21:23.779 INFO:tasks.rados.rados.0.smithi151.stdout:198: done (2 left) 2024-09-13T14:21:23.780 INFO:tasks.rados.rados.0.smithi151.stdout:199: delete oid 24 current snap is 18 2024-09-13T14:21:23.780 INFO:tasks.rados.rados.0.smithi151.stdout:197: finishing rollback tid 0 to smithi15186934-44 2024-09-13T14:21:23.780 INFO:tasks.rados.rados.0.smithi151.stdout:197: finishing rollback tid 1 to smithi15186934-44 2024-09-13T14:21:23.780 INFO:tasks.rados.rados.0.smithi151.stdout:197: finishing rollback tid 2 to smithi15186934-44 2024-09-13T14:21:23.780 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 31 (ObjNum 98 snap 16 seq_num 98) dirty exists 2024-09-13T14:21:23.780 INFO:tasks.rados.rados.0.smithi151.stdout:196: expect (ObjNum 68 snap 7 seq_num 68) 2024-09-13T14:21:23.927 INFO:tasks.rados.rados.0.smithi151.stdout:196: done (2 left) 2024-09-13T14:21:23.927 INFO:tasks.rados.rados.0.smithi151.stdout:197: done (1 left) 2024-09-13T14:21:23.927 INFO:tasks.rados.rados.0.smithi151.stdout:199: done (0 left) 2024-09-13T14:21:23.927 INFO:tasks.rados.rados.0.smithi151.stdout:200: read oid 9 snap 13 2024-09-13T14:21:23.928 INFO:tasks.rados.rados.0.smithi151.stdout:200: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-13T14:21:23.928 INFO:tasks.rados.rados.0.smithi151.stdout:201: rollback oid 45 current snap is 18 2024-09-13T14:21:23.928 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 45 to 16 2024-09-13T14:21:23.928 INFO:tasks.rados.rados.0.smithi151.stdout:202: rollback oid 14 current snap is 18 2024-09-13T14:21:23.928 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 14 to 13 2024-09-13T14:21:23.928 INFO:tasks.rados.rados.0.smithi151.stdout:203: write oid 34 current snap is 18 2024-09-13T14:21:23.928 INFO:tasks.rados.rados.0.smithi151.stdout:203: seq_num 112 ranges {682384=456830,1721042=423699,2809728=454936,3731357=151027} 2024-09-13T14:21:23.933 INFO:tasks.rados.rados.0.smithi151.stdout:203: writing smithi15186934-34 from 682384 to 1139214 tid 1 2024-09-13T14:21:23.939 INFO:tasks.rados.rados.0.smithi151.stdout:203: writing smithi15186934-34 from 1721042 to 2144741 tid 2 2024-09-13T14:21:23.945 INFO:tasks.rados.rados.0.smithi151.stdout:203: writing smithi15186934-34 from 2809728 to 3264664 tid 3 2024-09-13T14:21:23.949 INFO:tasks.rados.rados.0.smithi151.stdout:203: writing smithi15186934-34 from 3731357 to 3882384 tid 4 2024-09-13T14:21:23.949 INFO:tasks.rados.rados.0.smithi151.stdout:204: snap_create 2024-09-13T14:21:23.949 INFO:tasks.rados.rados.0.smithi151.stdout:201: finishing rollback tid 1 to smithi15186934-45 2024-09-13T14:21:23.949 INFO:tasks.rados.rados.0.smithi151.stdout:201: finishing rollback tid 2 to smithi15186934-45 2024-09-13T14:21:23.949 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 49 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-13T14:21:23.949 INFO:tasks.rados.rados.0.smithi151.stdout:200: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-13T14:21:24.025 INFO:tasks.rados.rados.0.smithi151.stdout:202: finishing rollback tid 0 to smithi15186934-14 2024-09-13T14:21:24.025 INFO:tasks.rados.rados.0.smithi151.stdout:202: finishing rollback tid 1 to smithi15186934-14 2024-09-13T14:21:24.025 INFO:tasks.rados.rados.0.smithi151.stdout:202: finishing rollback tid 2 to smithi15186934-14 2024-09-13T14:21:24.025 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 97 (ObjNum 75 snap 9 seq_num 75) dirty exists 2024-09-13T14:21:24.025 INFO:tasks.rados.rados.0.smithi151.stdout:203: finishing write tid 1 to smithi15186934-34 2024-09-13T14:21:24.025 INFO:tasks.rados.rados.0.smithi151.stdout:203: finishing write tid 2 to smithi15186934-34 2024-09-13T14:21:24.025 INFO:tasks.rados.rados.0.smithi151.stdout:203: finishing write tid 3 to smithi15186934-34 2024-09-13T14:21:24.025 INFO:tasks.rados.rados.0.smithi151.stdout:203: finishing write tid 4 to smithi15186934-34 2024-09-13T14:21:24.025 INFO:tasks.rados.rados.0.smithi151.stdout:203: finishing write tid 5 to smithi15186934-34 2024-09-13T14:21:24.025 INFO:tasks.rados.rados.0.smithi151.stdout:203: finishing write tid 6 to smithi15186934-34 2024-09-13T14:21:24.026 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 94 (ObjNum 112 snap 18 seq_num 112) dirty exists 2024-09-13T14:21:24.026 INFO:tasks.rados.rados.0.smithi151.stdout:203: left oid 34 (ObjNum 112 snap 18 seq_num 112) 2024-09-13T14:21:24.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:23 smithi184 ceph-mon[33394]: pgmap v138: 89 pgs: 89 active+clean; 183 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 4.3 MiB/s wr, 35 op/s 2024-09-13T14:21:24.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:23 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/2079995224' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:24.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:23 smithi184 ceph-mon[33394]: osdmap e102: 8 total, 8 up, 8 in 2024-09-13T14:21:24.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:23 smithi151 ceph-mon[29807]: pgmap v138: 89 pgs: 89 active+clean; 183 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 4.3 MiB/s wr, 35 op/s 2024-09-13T14:21:24.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:23 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/2079995224' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:24.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:23 smithi151 ceph-mon[29807]: osdmap e102: 8 total, 8 up, 8 in 2024-09-13T14:21:24.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:23 smithi151 ceph-mon[41387]: pgmap v138: 89 pgs: 89 active+clean; 183 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 4.3 MiB/s wr, 35 op/s 2024-09-13T14:21:24.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:23 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/2079995224' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:24.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:23 smithi151 ceph-mon[41387]: osdmap e102: 8 total, 8 up, 8 in 2024-09-13T14:21:24.917 INFO:tasks.rados.rados.0.smithi151.stdout:200: done (4 left) 2024-09-13T14:21:24.917 INFO:tasks.rados.rados.0.smithi151.stdout:201: done (3 left) 2024-09-13T14:21:24.917 INFO:tasks.rados.rados.0.smithi151.stdout:202: done (2 left) 2024-09-13T14:21:24.917 INFO:tasks.rados.rados.0.smithi151.stdout:203: done (1 left) 2024-09-13T14:21:24.917 INFO:tasks.rados.rados.0.smithi151.stdout:204: done (0 left) 2024-09-13T14:21:24.918 INFO:tasks.rados.rados.0.smithi151.stdout:205: write oid 49 current snap is 19 2024-09-13T14:21:24.918 INFO:tasks.rados.rados.0.smithi151.stdout:205: seq_num 113 ranges {699191=532079,1779524=567357,2837151=666208} 2024-09-13T14:21:24.923 INFO:tasks.rados.rados.0.smithi151.stdout:205: writing smithi15186934-49 from 699191 to 1231270 tid 1 2024-09-13T14:21:24.930 INFO:tasks.rados.rados.0.smithi151.stdout:205: writing smithi15186934-49 from 1779524 to 2346881 tid 2 2024-09-13T14:21:24.936 INFO:tasks.rados.rados.0.smithi151.stdout:205: writing smithi15186934-49 from 2837151 to 3503359 tid 3 2024-09-13T14:21:24.936 INFO:tasks.rados.rados.0.smithi151.stdout:206: snap_remove snap 18 2024-09-13T14:21:25.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:24 smithi184 ceph-mon[33394]: from='client.15000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:25.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:24 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:21:25.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:24 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:21:25.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:24 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:21:25.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:24 smithi151 ceph-mon[29807]: from='client.15000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:25.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:24 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:21:25.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:24 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:21:25.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:24 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:21:25.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:24 smithi151 ceph-mon[41387]: from='client.15000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:25.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:24 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:21:25.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:24 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:21:25.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:24 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:21:25.924 INFO:tasks.rados.rados.0.smithi151.stdout:206: done (1 left) 2024-09-13T14:21:25.924 INFO:tasks.rados.rados.0.smithi151.stdout:207: rollback oid 12 current snap is 19 2024-09-13T14:21:25.924 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 12 to 11 2024-09-13T14:21:25.924 INFO:tasks.rados.rados.0.smithi151.stdout:205: finishing write tid 1 to smithi15186934-49 2024-09-13T14:21:25.924 INFO:tasks.rados.rados.0.smithi151.stdout:205: finishing write tid 2 to smithi15186934-49 2024-09-13T14:21:25.924 INFO:tasks.rados.rados.0.smithi151.stdout:205: finishing write tid 3 to smithi15186934-49 2024-09-13T14:21:25.925 INFO:tasks.rados.rados.0.smithi151.stdout:205: finishing write tid 4 to smithi15186934-49 2024-09-13T14:21:25.925 INFO:tasks.rados.rados.0.smithi151.stdout:205: finishing write tid 5 to smithi15186934-49 2024-09-13T14:21:25.925 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 49 v 94 (ObjNum 113 snap 19 seq_num 113) dirty exists 2024-09-13T14:21:25.925 INFO:tasks.rados.rados.0.smithi151.stdout:205: left oid 49 (ObjNum 113 snap 19 seq_num 113) 2024-09-13T14:21:25.925 INFO:tasks.rados.rados.0.smithi151.stdout:205: done (1 left) 2024-09-13T14:21:25.925 INFO:tasks.rados.rados.0.smithi151.stdout:208: snap_remove snap 13 2024-09-13T14:21:26.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:25 smithi184 ceph-mon[33394]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (unknown) 2024-09-13T14:21:26.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:25 smithi184 ceph-mon[33394]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:21:26.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:25 smithi184 ceph-mon[33394]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-13T14:21:26.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:25 smithi184 ceph-mon[33394]: from='client.15006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:26.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:25 smithi184 ceph-mon[33394]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f on smithi184 2024-09-13T14:21:26.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:25 smithi184 ceph-mon[33394]: pgmap v141: 89 pgs: 89 active+clean; 155 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 363 KiB/s wr, 12 op/s 2024-09-13T14:21:26.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:25 smithi184 ceph-mon[33394]: osdmap e103: 8 total, 8 up, 8 in 2024-09-13T14:21:26.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:25 smithi151 ceph-mon[29807]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (unknown) 2024-09-13T14:21:26.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:25 smithi151 ceph-mon[29807]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:21:26.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:25 smithi151 ceph-mon[29807]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-13T14:21:26.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:25 smithi151 ceph-mon[29807]: from='client.15006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:26.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:25 smithi151 ceph-mon[29807]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f on smithi184 2024-09-13T14:21:26.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:25 smithi151 ceph-mon[29807]: pgmap v141: 89 pgs: 89 active+clean; 155 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 363 KiB/s wr, 12 op/s 2024-09-13T14:21:26.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:25 smithi151 ceph-mon[29807]: osdmap e103: 8 total, 8 up, 8 in 2024-09-13T14:21:26.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:25 smithi151 ceph-mon[41387]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (unknown) 2024-09-13T14:21:26.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:25 smithi151 ceph-mon[41387]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:21:26.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:25 smithi151 ceph-mon[41387]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-13T14:21:26.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:25 smithi151 ceph-mon[41387]: from='client.15006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:26.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:25 smithi151 ceph-mon[41387]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f on smithi184 2024-09-13T14:21:26.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:25 smithi151 ceph-mon[41387]: pgmap v141: 89 pgs: 89 active+clean; 155 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 363 KiB/s wr, 12 op/s 2024-09-13T14:21:26.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:25 smithi151 ceph-mon[41387]: osdmap e103: 8 total, 8 up, 8 in 2024-09-13T14:21:26.537 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:21:26.934 INFO:tasks.rados.rados.0.smithi151.stdout:208: done (1 left) 2024-09-13T14:21:26.934 INFO:tasks.rados.rados.0.smithi151.stdout:209: delete oid 21 current snap is 19 2024-09-13T14:21:26.934 INFO:tasks.rados.rados.0.smithi151.stdout:207: finishing rollback tid 1 to smithi15186934-12 2024-09-13T14:21:26.934 INFO:tasks.rados.rados.0.smithi151.stdout:207: finishing rollback tid 2 to smithi15186934-12 2024-09-13T14:21:26.934 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 12 v 33 (ObjNum 68 snap 7 seq_num 68) dirty exists 2024-09-13T14:21:26.941 INFO:tasks.rados.rados.0.smithi151.stdout:207: done (1 left) 2024-09-13T14:21:26.941 INFO:tasks.rados.rados.0.smithi151.stdout:209: done (0 left) 2024-09-13T14:21:26.941 INFO:tasks.rados.rados.0.smithi151.stdout:210: read oid 19 snap -1 2024-09-13T14:21:26.941 INFO:tasks.rados.rados.0.smithi151.stdout:210: expect (ObjNum 101 snap 16 seq_num 101) 2024-09-13T14:21:26.941 INFO:tasks.rados.rados.0.smithi151.stdout:211: snap_remove snap 11 2024-09-13T14:21:27.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:26 smithi184 ceph-mon[33394]: osdmap e104: 8 total, 8 up, 8 in 2024-09-13T14:21:27.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:26 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/2293853724' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:27.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:26 smithi184 ceph-mon[33394]: pgmap v144: 89 pgs: 89 active+clean; 155 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 363 KiB/s wr, 12 op/s 2024-09-13T14:21:27.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:26 smithi184 ceph-mon[33394]: from='client.24920 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:27.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:26 smithi184 ceph-mon[33394]: from='client.15024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:27.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:26 smithi151 ceph-mon[29807]: osdmap e104: 8 total, 8 up, 8 in 2024-09-13T14:21:27.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:26 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/2293853724' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:27.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:26 smithi151 ceph-mon[29807]: pgmap v144: 89 pgs: 89 active+clean; 155 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 363 KiB/s wr, 12 op/s 2024-09-13T14:21:27.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:26 smithi151 ceph-mon[29807]: from='client.24920 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:27.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:26 smithi151 ceph-mon[29807]: from='client.15024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:27.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:26 smithi151 ceph-mon[41387]: osdmap e104: 8 total, 8 up, 8 in 2024-09-13T14:21:27.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:26 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/2293853724' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:27.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:26 smithi151 ceph-mon[41387]: pgmap v144: 89 pgs: 89 active+clean; 155 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 363 KiB/s wr, 12 op/s 2024-09-13T14:21:27.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:26 smithi151 ceph-mon[41387]: from='client.24920 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:27.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:26 smithi151 ceph-mon[41387]: from='client.15024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:27.776 INFO:tasks.rados.rados.0.smithi151.stdout:211: done (1 left) 2024-09-13T14:21:27.777 INFO:tasks.rados.rados.0.smithi151.stdout:212: delete oid 2 current snap is 19 2024-09-13T14:21:27.777 INFO:tasks.rados.rados.0.smithi151.stdout:210: expect (ObjNum 101 snap 16 seq_num 101) 2024-09-13T14:21:27.853 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:21:27 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:21:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:21:27.985 INFO:tasks.rados.rados.0.smithi151.stdout:210: done (1 left) 2024-09-13T14:21:27.985 INFO:tasks.rados.rados.0.smithi151.stdout:212: done (0 left) 2024-09-13T14:21:27.985 INFO:tasks.rados.rados.0.smithi151.stdout:213: snap_create 2024-09-13T14:21:28.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:27 smithi184 ceph-mon[33394]: osdmap e105: 8 total, 8 up, 8 in 2024-09-13T14:21:28.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:27 smithi184 ceph-mon[33394]: osdmap e106: 8 total, 8 up, 8 in 2024-09-13T14:21:28.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:27 smithi151 ceph-mon[29807]: osdmap e105: 8 total, 8 up, 8 in 2024-09-13T14:21:28.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:27 smithi151 ceph-mon[29807]: osdmap e106: 8 total, 8 up, 8 in 2024-09-13T14:21:28.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:27 smithi151 ceph-mon[41387]: osdmap e105: 8 total, 8 up, 8 in 2024-09-13T14:21:28.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:27 smithi151 ceph-mon[41387]: osdmap e106: 8 total, 8 up, 8 in 2024-09-13T14:21:28.937 INFO:tasks.rados.rados.0.smithi151.stdout:213: done (0 left) 2024-09-13T14:21:28.937 INFO:tasks.rados.rados.0.smithi151.stdout:214: delete oid 46 current snap is 20 2024-09-13T14:21:28.938 INFO:tasks.rados.rados.0.smithi151.stdout:214: done (0 left) 2024-09-13T14:21:28.939 INFO:tasks.rados.rados.0.smithi151.stdout:215: write oid 16 current snap is 20 2024-09-13T14:21:28.939 INFO:tasks.rados.rados.0.smithi151.stdout:215: seq_num 117 ranges {766419=549428,2013452=574695,3346843=427649} 2024-09-13T14:21:28.945 INFO:tasks.rados.rados.0.smithi151.stdout:215: writing smithi15186934-16 from 766419 to 1315847 tid 1 2024-09-13T14:21:28.952 INFO:tasks.rados.rados.0.smithi151.stdout:215: writing smithi15186934-16 from 2013452 to 2588147 tid 2 2024-09-13T14:21:28.960 INFO:tasks.rados.rados.0.smithi151.stdout:215: writing smithi15186934-16 from 3346843 to 3774492 tid 3 2024-09-13T14:21:28.960 INFO:tasks.rados.rados.0.smithi151.stdout:216: read oid 30 snap -1 2024-09-13T14:21:28.960 INFO:tasks.rados.rados.0.smithi151.stdout:216: expect (ObjNum 29 snap 0 seq_num 29) 2024-09-13T14:21:28.960 INFO:tasks.rados.rados.0.smithi151.stdout:217: snap_remove snap 16 2024-09-13T14:21:29.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:28 smithi184 ceph-mon[33394]: pgmap v147: 89 pgs: 89 active+clean; 155 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:29.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:28 smithi151 ceph-mon[29807]: pgmap v147: 89 pgs: 89 active+clean; 155 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:29.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:28 smithi151 ceph-mon[41387]: pgmap v147: 89 pgs: 89 active+clean; 155 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:29.366 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:21:29.940 INFO:tasks.rados.rados.0.smithi151.stdout:217: done (2 left) 2024-09-13T14:21:29.941 INFO:tasks.rados.rados.0.smithi151.stdout:218: snap_remove snap 17 2024-09-13T14:21:30.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:29 smithi184 ceph-mon[33394]: osdmap e107: 8 total, 8 up, 8 in 2024-09-13T14:21:30.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:29 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/734093434' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:30.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:29 smithi184 ceph-mon[33394]: from='client.24938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:30.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:29 smithi184 ceph-mon[33394]: from='client.15042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:30.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:29 smithi151 ceph-mon[29807]: osdmap e107: 8 total, 8 up, 8 in 2024-09-13T14:21:30.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:29 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/734093434' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:30.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:29 smithi151 ceph-mon[29807]: from='client.24938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:30.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:29 smithi151 ceph-mon[29807]: from='client.15042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:30.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:29 smithi151 ceph-mon[41387]: osdmap e107: 8 total, 8 up, 8 in 2024-09-13T14:21:30.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:29 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/734093434' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:30.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:29 smithi151 ceph-mon[41387]: from='client.24938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:30.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:29 smithi151 ceph-mon[41387]: from='client.15042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:30.947 INFO:tasks.rados.rados.0.smithi151.stdout:218: done (2 left) 2024-09-13T14:21:30.948 INFO:tasks.rados.rados.0.smithi151.stdout:219: write oid 18 current snap is 20 2024-09-13T14:21:30.948 INFO:tasks.rados.rados.0.smithi151.stdout:215: finishing write tid 1 to smithi15186934-16 2024-09-13T14:21:30.948 INFO:tasks.rados.rados.0.smithi151.stdout:215: finishing write tid 2 to smithi15186934-16 2024-09-13T14:21:30.948 INFO:tasks.rados.rados.0.smithi151.stdout:215: finishing write tid 3 to smithi15186934-16 2024-09-13T14:21:30.948 INFO:tasks.rados.rados.0.smithi151.stdout:215: finishing write tid 4 to smithi15186934-16 2024-09-13T14:21:30.948 INFO:tasks.rados.rados.0.smithi151.stdout:215: finishing write tid 5 to smithi15186934-16 2024-09-13T14:21:30.948 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 16 v 58 (ObjNum 117 snap 20 seq_num 117) dirty exists 2024-09-13T14:21:30.948 INFO:tasks.rados.rados.0.smithi151.stdout:215: left oid 16 (ObjNum 117 snap 20 seq_num 117) 2024-09-13T14:21:30.948 INFO:tasks.rados.rados.0.smithi151.stdout:216: expect (ObjNum 29 snap 0 seq_num 29) 2024-09-13T14:21:31.031 INFO:tasks.rados.rados.0.smithi151.stdout:219: seq_num 118 ranges {783226=624677,2071934=434706,2974266=638921} 2024-09-13T14:21:31.038 INFO:tasks.rados.rados.0.smithi151.stdout:219: writing smithi15186934-18 from 783226 to 1407903 tid 1 2024-09-13T14:21:31.044 INFO:tasks.rados.rados.0.smithi151.stdout:219: writing smithi15186934-18 from 2071934 to 2506640 tid 2 2024-09-13T14:21:31.051 INFO:tasks.rados.rados.0.smithi151.stdout:219: writing smithi15186934-18 from 2974266 to 3613187 tid 3 2024-09-13T14:21:31.051 INFO:tasks.rados.rados.0.smithi151.stdout:215: done (2 left) 2024-09-13T14:21:31.051 INFO:tasks.rados.rados.0.smithi151.stdout:216: done (1 left) 2024-09-13T14:21:31.051 INFO:tasks.rados.rados.0.smithi151.stdout:220: rollback oid 11 current snap is 20 2024-09-13T14:21:31.051 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 11 to 19 2024-09-13T14:21:31.051 INFO:tasks.rados.rados.0.smithi151.stdout:219: finishing write tid 1 to smithi15186934-18 2024-09-13T14:21:31.051 INFO:tasks.rados.rados.0.smithi151.stdout:219: finishing write tid 2 to smithi15186934-18 2024-09-13T14:21:31.052 INFO:tasks.rados.rados.0.smithi151.stdout:221: snap_create 2024-09-13T14:21:31.054 INFO:tasks.rados.rados.0.smithi151.stdout:219: finishing write tid 3 to smithi15186934-18 2024-09-13T14:21:31.054 INFO:tasks.rados.rados.0.smithi151.stdout:219: finishing write tid 4 to smithi15186934-18 2024-09-13T14:21:31.054 INFO:tasks.rados.rados.0.smithi151.stdout:219: finishing write tid 5 to smithi15186934-18 2024-09-13T14:21:31.054 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 18 v 29 (ObjNum 118 snap 20 seq_num 118) dirty exists 2024-09-13T14:21:31.054 INFO:tasks.rados.rados.0.smithi151.stdout:219: left oid 18 (ObjNum 118 snap 20 seq_num 118) 2024-09-13T14:21:31.054 INFO:tasks.rados.rados.0.smithi151.stdout:220: finishing rollback tid 0 to smithi15186934-11 2024-09-13T14:21:31.054 INFO:tasks.rados.rados.0.smithi151.stdout:220: finishing rollback tid 1 to smithi15186934-11 2024-09-13T14:21:31.055 INFO:tasks.rados.rados.0.smithi151.stdout:220: finishing rollback tid 2 to smithi15186934-11 2024-09-13T14:21:31.055 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 11 v 33 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-13T14:21:31.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:30 smithi184 ceph-mon[33394]: osdmap e108: 8 total, 8 up, 8 in 2024-09-13T14:21:31.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:30 smithi184 ceph-mon[33394]: pgmap v150: 89 pgs: 89 active+clean; 142 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.0 MiB/s rd, 811 KiB/s wr, 8 op/s 2024-09-13T14:21:31.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:30 smithi151 ceph-mon[29807]: osdmap e108: 8 total, 8 up, 8 in 2024-09-13T14:21:31.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:30 smithi151 ceph-mon[29807]: pgmap v150: 89 pgs: 89 active+clean; 142 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.0 MiB/s rd, 811 KiB/s wr, 8 op/s 2024-09-13T14:21:31.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:30 smithi151 ceph-mon[41387]: osdmap e108: 8 total, 8 up, 8 in 2024-09-13T14:21:31.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:30 smithi151 ceph-mon[41387]: pgmap v150: 89 pgs: 89 active+clean; 142 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.0 MiB/s rd, 811 KiB/s wr, 8 op/s 2024-09-13T14:21:31.956 INFO:tasks.rados.rados.0.smithi151.stdout:219: done (2 left) 2024-09-13T14:21:31.957 INFO:tasks.rados.rados.0.smithi151.stdout:220: done (1 left) 2024-09-13T14:21:31.957 INFO:tasks.rados.rados.0.smithi151.stdout:221: done (0 left) 2024-09-13T14:21:31.957 INFO:tasks.rados.rados.0.smithi151.stdout:222: delete oid 28 current snap is 21 2024-09-13T14:21:31.963 INFO:tasks.rados.rados.0.smithi151.stdout:222: done (0 left) 2024-09-13T14:21:31.963 INFO:tasks.rados.rados.0.smithi151.stdout:223: snap_create 2024-09-13T14:21:32.204 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:21:32.204 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:31 smithi151 ceph-mon[29807]: osdmap e109: 8 total, 8 up, 8 in 2024-09-13T14:21:32.204 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:31 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/4186903428' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:32.205 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:31 smithi151 ceph-mon[41387]: osdmap e109: 8 total, 8 up, 8 in 2024-09-13T14:21:32.205 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:31 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/4186903428' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:32.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:31 smithi184 ceph-mon[33394]: osdmap e109: 8 total, 8 up, 8 in 2024-09-13T14:21:32.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:31 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/4186903428' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:32.778 INFO:tasks.rados.rados.0.smithi151.stdout:223: done (0 left) 2024-09-13T14:21:32.778 INFO:tasks.rados.rados.0.smithi151.stdout:224: read oid 49 snap -1 2024-09-13T14:21:32.778 INFO:tasks.rados.rados.0.smithi151.stdout:224: expect (ObjNum 113 snap 19 seq_num 113) 2024-09-13T14:21:32.778 INFO:tasks.rados.rados.0.smithi151.stdout:225: read oid 23 snap 21 2024-09-13T14:21:32.778 INFO:tasks.rados.rados.0.smithi151.stdout:225: expect deleted 2024-09-13T14:21:32.779 INFO:tasks.rados.rados.0.smithi151.stdout:226: write oid 7 current snap is 22 2024-09-13T14:21:32.779 INFO:tasks.rados.rados.0.smithi151.stdout:226: seq_num 120 ranges {416840=775175,1672545=344295} 2024-09-13T14:21:32.784 INFO:tasks.rados.rados.0.smithi151.stdout:226: writing smithi15186934-7 from 416840 to 1192015 tid 1 2024-09-13T14:21:32.789 INFO:tasks.rados.rados.0.smithi151.stdout:226: writing smithi15186934-7 from 1672545 to 2016840 tid 2 2024-09-13T14:21:32.789 INFO:tasks.rados.rados.0.smithi151.stdout:227: snap_create 2024-09-13T14:21:32.791 INFO:tasks.rados.rados.0.smithi151.stdout:226: finishing write tid 1 to smithi15186934-7 2024-09-13T14:21:32.791 INFO:tasks.rados.rados.0.smithi151.stdout:226: finishing write tid 2 to smithi15186934-7 2024-09-13T14:21:32.792 INFO:tasks.rados.rados.0.smithi151.stdout:226: finishing write tid 3 to smithi15186934-7 2024-09-13T14:21:32.792 INFO:tasks.rados.rados.0.smithi151.stdout:226: finishing write tid 4 to smithi15186934-7 2024-09-13T14:21:32.792 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 120 (ObjNum 120 snap 22 seq_num 120) dirty exists 2024-09-13T14:21:32.792 INFO:tasks.rados.rados.0.smithi151.stdout:226: left oid 7 (ObjNum 120 snap 22 seq_num 120) 2024-09-13T14:21:32.793 INFO:tasks.rados.rados.0.smithi151.stdout:224: expect (ObjNum 113 snap 19 seq_num 113) 2024-09-13T14:21:33.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:32 smithi184 ceph-mon[33394]: osdmap e110: 8 total, 8 up, 8 in 2024-09-13T14:21:33.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:32 smithi184 ceph-mon[33394]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:33.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:32 smithi184 ceph-mon[33394]: pgmap v153: 89 pgs: 89 active+clean; 142 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.0 MiB/s rd, 811 KiB/s wr, 8 op/s 2024-09-13T14:21:33.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:32 smithi184 ceph-mon[33394]: from='client.24962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:33.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:32 smithi184 ceph-mon[33394]: osdmap e111: 8 total, 8 up, 8 in 2024-09-13T14:21:33.358 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:21:32 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:21:32] "GET /metrics HTTP/1.1" 200 243175 "" "Prometheus/2.43.0" 2024-09-13T14:21:33.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:32 smithi151 ceph-mon[29807]: osdmap e110: 8 total, 8 up, 8 in 2024-09-13T14:21:33.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:32 smithi151 ceph-mon[29807]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:33.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:32 smithi151 ceph-mon[29807]: pgmap v153: 89 pgs: 89 active+clean; 142 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.0 MiB/s rd, 811 KiB/s wr, 8 op/s 2024-09-13T14:21:33.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:32 smithi151 ceph-mon[29807]: from='client.24962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:33.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:32 smithi151 ceph-mon[29807]: osdmap e111: 8 total, 8 up, 8 in 2024-09-13T14:21:33.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:32 smithi151 ceph-mon[41387]: osdmap e110: 8 total, 8 up, 8 in 2024-09-13T14:21:33.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:32 smithi151 ceph-mon[41387]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:33.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:32 smithi151 ceph-mon[41387]: pgmap v153: 89 pgs: 89 active+clean; 142 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.0 MiB/s rd, 811 KiB/s wr, 8 op/s 2024-09-13T14:21:33.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:32 smithi151 ceph-mon[41387]: from='client.24962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:33.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:32 smithi151 ceph-mon[41387]: osdmap e111: 8 total, 8 up, 8 in 2024-09-13T14:21:33.779 INFO:tasks.rados.rados.0.smithi151.stdout:224: done (3 left) 2024-09-13T14:21:33.779 INFO:tasks.rados.rados.0.smithi151.stdout:225: done (2 left) 2024-09-13T14:21:33.779 INFO:tasks.rados.rados.0.smithi151.stdout:226: done (1 left) 2024-09-13T14:21:33.779 INFO:tasks.rados.rados.0.smithi151.stdout:227: done (0 left) 2024-09-13T14:21:33.779 INFO:tasks.rados.rados.0.smithi151.stdout:228: rollback oid 47 current snap is 23 2024-09-13T14:21:33.779 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 47 to 19 2024-09-13T14:21:33.779 INFO:tasks.rados.rados.0.smithi151.stdout:229: delete oid 28 current snap is 23 2024-09-13T14:21:33.781 INFO:tasks.rados.rados.0.smithi151.stdout:229: done (1 left) 2024-09-13T14:21:33.781 INFO:tasks.rados.rados.0.smithi151.stdout:230: snap_remove snap 21 2024-09-13T14:21:34.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:33 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:21:34.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:33 smithi184 ceph-mon[33394]: osdmap e112: 8 total, 8 up, 8 in 2024-09-13T14:21:34.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:33 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:21:34.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:33 smithi151 ceph-mon[29807]: osdmap e112: 8 total, 8 up, 8 in 2024-09-13T14:21:34.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:33 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:21:34.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:33 smithi151 ceph-mon[41387]: osdmap e112: 8 total, 8 up, 8 in 2024-09-13T14:21:34.781 INFO:tasks.rados.rados.0.smithi151.stdout:230: done (1 left) 2024-09-13T14:21:34.781 INFO:tasks.rados.rados.0.smithi151.stdout:231: write oid 28 current snap is 23 2024-09-13T14:21:34.781 INFO:tasks.rados.rados.0.smithi151.stdout:231: seq_num 122 ranges {450454=642026,1505862=544592} 2024-09-13T14:21:34.786 INFO:tasks.rados.rados.0.smithi151.stdout:231: writing smithi15186934-28 from 450454 to 1092480 tid 1 2024-09-13T14:21:34.792 INFO:tasks.rados.rados.0.smithi151.stdout:231: writing smithi15186934-28 from 1505862 to 2050454 tid 2 2024-09-13T14:21:34.792 INFO:tasks.rados.rados.0.smithi151.stdout:232: read oid 40 snap -1 2024-09-13T14:21:34.793 INFO:tasks.rados.rados.0.smithi151.stdout:232: expect (ObjNum 78 snap 9 seq_num 78) 2024-09-13T14:21:34.793 INFO:tasks.rados.rados.0.smithi151.stdout:233: read oid 46 snap -1 2024-09-13T14:21:34.793 INFO:tasks.rados.rados.0.smithi151.stdout:233: expect deleted 2024-09-13T14:21:34.793 INFO:tasks.rados.rados.0.smithi151.stdout:234: read oid 15 snap -1 2024-09-13T14:21:34.793 INFO:tasks.rados.rados.0.smithi151.stdout:234: expect (ObjNum 105 snap 16 seq_num 105) 2024-09-13T14:21:34.793 INFO:tasks.rados.rados.0.smithi151.stdout:235: delete oid 27 current snap is 23 2024-09-13T14:21:34.793 INFO:tasks.rados.rados.0.smithi151.stdout:228: finishing rollback tid 0 to smithi15186934-47 2024-09-13T14:21:34.793 INFO:tasks.rados.rados.0.smithi151.stdout:228: finishing rollback tid 1 to smithi15186934-47 2024-09-13T14:21:34.793 INFO:tasks.rados.rados.0.smithi151.stdout:228: finishing rollback tid 2 to smithi15186934-47 2024-09-13T14:21:34.793 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 112 (ObjNum 79 snap 10 seq_num 79) dirty exists 2024-09-13T14:21:34.793 INFO:tasks.rados.rados.0.smithi151.stdout:231: finishing write tid 1 to smithi15186934-28 2024-09-13T14:21:34.796 INFO:tasks.rados.rados.0.smithi151.stdout:228: done (5 left) 2024-09-13T14:21:34.796 INFO:tasks.rados.rados.0.smithi151.stdout:233: done (4 left) 2024-09-13T14:21:34.796 INFO:tasks.rados.rados.0.smithi151.stdout:235: done (3 left) 2024-09-13T14:21:34.796 INFO:tasks.rados.rados.0.smithi151.stdout:236: write oid 8 current snap is 23 2024-09-13T14:21:34.796 INFO:tasks.rados.rados.0.smithi151.stdout:236: seq_num 124 ranges {484068=792524,1906473=177595} 2024-09-13T14:21:34.804 INFO:tasks.rados.rados.0.smithi151.stdout:236: writing smithi15186934-8 from 484068 to 1276592 tid 1 2024-09-13T14:21:34.809 INFO:tasks.rados.rados.0.smithi151.stdout:236: writing smithi15186934-8 from 1906473 to 2084068 tid 2 2024-09-13T14:21:34.809 INFO:tasks.rados.rados.0.smithi151.stdout:237: write oid 23 current snap is 23 2024-09-13T14:21:34.810 INFO:tasks.rados.rados.0.smithi151.stdout:237: seq_num 125 ranges {500875=467773,1564955=535920} 2024-09-13T14:21:34.814 INFO:tasks.rados.rados.0.smithi151.stdout:237: writing smithi15186934-23 from 500875 to 968648 tid 1 2024-09-13T14:21:34.821 INFO:tasks.rados.rados.0.smithi151.stdout:237: writing smithi15186934-23 from 1564955 to 2100875 tid 2 2024-09-13T14:21:34.821 INFO:tasks.rados.rados.0.smithi151.stdout:238: snap_create 2024-09-13T14:21:34.821 INFO:tasks.rados.rados.0.smithi151.stdout:232: expect (ObjNum 78 snap 9 seq_num 78) 2024-09-13T14:21:34.975 INFO:tasks.rados.rados.0.smithi151.stdout:231: finishing write tid 2 to smithi15186934-28 2024-09-13T14:21:34.975 INFO:tasks.rados.rados.0.smithi151.stdout:231: finishing write tid 3 to smithi15186934-28 2024-09-13T14:21:34.975 INFO:tasks.rados.rados.0.smithi151.stdout:231: finishing write tid 4 to smithi15186934-28 2024-09-13T14:21:34.975 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 28 v 111 (ObjNum 122 snap 23 seq_num 122) dirty exists 2024-09-13T14:21:34.975 INFO:tasks.rados.rados.0.smithi151.stdout:231: left oid 28 (ObjNum 122 snap 23 seq_num 122) 2024-09-13T14:21:34.975 INFO:tasks.rados.rados.0.smithi151.stdout:234: expect (ObjNum 105 snap 16 seq_num 105) 2024-09-13T14:21:35.051 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:21:35.139 INFO:tasks.rados.rados.0.smithi151.stdout:236: finishing write tid 1 to smithi15186934-8 2024-09-13T14:21:35.139 INFO:tasks.rados.rados.0.smithi151.stdout:236: finishing write tid 2 to smithi15186934-8 2024-09-13T14:21:35.139 INFO:tasks.rados.rados.0.smithi151.stdout:236: finishing write tid 3 to smithi15186934-8 2024-09-13T14:21:35.139 INFO:tasks.rados.rados.0.smithi151.stdout:236: finishing write tid 4 to smithi15186934-8 2024-09-13T14:21:35.139 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 45 (ObjNum 124 snap 23 seq_num 124) dirty exists 2024-09-13T14:21:35.140 INFO:tasks.rados.rados.0.smithi151.stdout:236: left oid 8 (ObjNum 124 snap 23 seq_num 124) 2024-09-13T14:21:35.140 INFO:tasks.rados.rados.0.smithi151.stdout:237: finishing write tid 1 to smithi15186934-23 2024-09-13T14:21:35.140 INFO:tasks.rados.rados.0.smithi151.stdout:237: finishing write tid 2 to smithi15186934-23 2024-09-13T14:21:35.140 INFO:tasks.rados.rados.0.smithi151.stdout:237: finishing write tid 3 to smithi15186934-23 2024-09-13T14:21:35.140 INFO:tasks.rados.rados.0.smithi151.stdout:237: finishing write tid 4 to smithi15186934-23 2024-09-13T14:21:35.140 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 23 v 117 (ObjNum 125 snap 23 seq_num 125) dirty exists 2024-09-13T14:21:35.140 INFO:tasks.rados.rados.0.smithi151.stdout:237: left oid 23 (ObjNum 125 snap 23 seq_num 125) 2024-09-13T14:21:35.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:34 smithi184 ceph-mon[33394]: pgmap v156: 89 pgs: 89 active+clean; 130 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 689 KiB/s wr, 6 op/s 2024-09-13T14:21:35.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:34 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/3318934067' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:35.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:34 smithi184 ceph-mon[33394]: osdmap e113: 8 total, 8 up, 8 in 2024-09-13T14:21:35.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:34 smithi151 ceph-mon[29807]: pgmap v156: 89 pgs: 89 active+clean; 130 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 689 KiB/s wr, 6 op/s 2024-09-13T14:21:35.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:34 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/3318934067' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:35.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:34 smithi151 ceph-mon[29807]: osdmap e113: 8 total, 8 up, 8 in 2024-09-13T14:21:35.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:34 smithi151 ceph-mon[41387]: pgmap v156: 89 pgs: 89 active+clean; 130 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 689 KiB/s wr, 6 op/s 2024-09-13T14:21:35.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:34 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/3318934067' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:35.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:34 smithi151 ceph-mon[41387]: osdmap e113: 8 total, 8 up, 8 in 2024-09-13T14:21:35.782 INFO:tasks.rados.rados.0.smithi151.stdout:231: done (5 left) 2024-09-13T14:21:35.782 INFO:tasks.rados.rados.0.smithi151.stdout:232: done (4 left) 2024-09-13T14:21:35.782 INFO:tasks.rados.rados.0.smithi151.stdout:234: done (3 left) 2024-09-13T14:21:35.782 INFO:tasks.rados.rados.0.smithi151.stdout:236: done (2 left) 2024-09-13T14:21:35.782 INFO:tasks.rados.rados.0.smithi151.stdout:237: done (1 left) 2024-09-13T14:21:35.782 INFO:tasks.rados.rados.0.smithi151.stdout:238: done (0 left) 2024-09-13T14:21:35.783 INFO:tasks.rados.rados.0.smithi151.stdout:239: read oid 44 snap -1 2024-09-13T14:21:35.783 INFO:tasks.rados.rados.0.smithi151.stdout:239: expect (ObjNum 98 snap 16 seq_num 98) 2024-09-13T14:21:35.783 INFO:tasks.rados.rados.0.smithi151.stdout:240: write oid 22 current snap is 24 2024-09-13T14:21:35.783 INFO:tasks.rados.rados.0.smithi151.stdout:240: seq_num 126 ranges {517682=543022,1623437=494245} 2024-09-13T14:21:35.787 INFO:tasks.rados.rados.0.smithi151.stdout:240: writing smithi15186934-22 from 517682 to 1060704 tid 1 2024-09-13T14:21:35.793 INFO:tasks.rados.rados.0.smithi151.stdout:240: writing smithi15186934-22 from 1623437 to 2117682 tid 2 2024-09-13T14:21:35.793 INFO:tasks.rados.rados.0.smithi151.stdout:241: snap_create 2024-09-13T14:21:35.793 INFO:tasks.rados.rados.0.smithi151.stdout:239: expect (ObjNum 98 snap 16 seq_num 98) 2024-09-13T14:21:35.968 INFO:tasks.rados.rados.0.smithi151.stdout:240: finishing write tid 1 to smithi15186934-22 2024-09-13T14:21:35.969 INFO:tasks.rados.rados.0.smithi151.stdout:240: finishing write tid 2 to smithi15186934-22 2024-09-13T14:21:35.969 INFO:tasks.rados.rados.0.smithi151.stdout:240: finishing write tid 3 to smithi15186934-22 2024-09-13T14:21:35.969 INFO:tasks.rados.rados.0.smithi151.stdout:240: finishing write tid 4 to smithi15186934-22 2024-09-13T14:21:35.969 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 22 v 22 (ObjNum 126 snap 24 seq_num 126) dirty exists 2024-09-13T14:21:35.969 INFO:tasks.rados.rados.0.smithi151.stdout:240: left oid 22 (ObjNum 126 snap 24 seq_num 126) 2024-09-13T14:21:36.791 INFO:tasks.rados.rados.0.smithi151.stdout:239: done (2 left) 2024-09-13T14:21:36.791 INFO:tasks.rados.rados.0.smithi151.stdout:240: done (1 left) 2024-09-13T14:21:36.791 INFO:tasks.rados.rados.0.smithi151.stdout:241: done (0 left) 2024-09-13T14:21:36.791 INFO:tasks.rados.rados.0.smithi151.stdout:242: rollback oid 28 current snap is 25 2024-09-13T14:21:36.792 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 28 to 24 2024-09-13T14:21:36.792 INFO:tasks.rados.rados.0.smithi151.stdout:243: delete oid 29 current snap is 25 2024-09-13T14:21:36.793 INFO:tasks.rados.rados.0.smithi151.stdout:243: done (1 left) 2024-09-13T14:21:36.793 INFO:tasks.rados.rados.0.smithi151.stdout:244: rollback oid 26 current snap is 25 2024-09-13T14:21:36.793 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 26 to 24 2024-09-13T14:21:36.793 INFO:tasks.rados.rados.0.smithi151.stdout:245: rollback oid 16 current snap is 25 2024-09-13T14:21:36.793 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 16 to 20 2024-09-13T14:21:36.793 INFO:tasks.rados.rados.0.smithi151.stdout:246: snap_create 2024-09-13T14:21:36.795 INFO:tasks.rados.rados.0.smithi151.stdout:242: finishing rollback tid 0 to smithi15186934-28 2024-09-13T14:21:36.795 INFO:tasks.rados.rados.0.smithi151.stdout:242: finishing rollback tid 1 to smithi15186934-28 2024-09-13T14:21:36.796 INFO:tasks.rados.rados.0.smithi151.stdout:242: finishing rollback tid 2 to smithi15186934-28 2024-09-13T14:21:36.796 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 28 v 115 (ObjNum 122 snap 23 seq_num 122) dirty exists 2024-09-13T14:21:36.797 INFO:tasks.rados.rados.0.smithi151.stdout:244: finishing rollback tid 0 to smithi15186934-26 2024-09-13T14:21:36.797 INFO:tasks.rados.rados.0.smithi151.stdout:244: finishing rollback tid 1 to smithi15186934-26 2024-09-13T14:21:36.797 INFO:tasks.rados.rados.0.smithi151.stdout:245: finishing rollback tid 0 to smithi15186934-16 2024-09-13T14:21:36.797 INFO:tasks.rados.rados.0.smithi151.stdout:244: finishing rollback tid 2 to smithi15186934-26 2024-09-13T14:21:36.798 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 62 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-13T14:21:36.798 INFO:tasks.rados.rados.0.smithi151.stdout:245: finishing rollback tid 1 to smithi15186934-16 2024-09-13T14:21:36.798 INFO:tasks.rados.rados.0.smithi151.stdout:245: finishing rollback tid 2 to smithi15186934-16 2024-09-13T14:21:36.798 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 16 v 66 (ObjNum 117 snap 20 seq_num 117) dirty exists 2024-09-13T14:21:37.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:36 smithi184 ceph-mon[33394]: from='client.15072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:37.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:36 smithi184 ceph-mon[33394]: from='client.24980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:37.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:36 smithi184 ceph-mon[33394]: osdmap e114: 8 total, 8 up, 8 in 2024-09-13T14:21:37.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:36 smithi151 ceph-mon[29807]: from='client.15072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:37.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:36 smithi151 ceph-mon[29807]: from='client.24980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:37.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:36 smithi151 ceph-mon[29807]: osdmap e114: 8 total, 8 up, 8 in 2024-09-13T14:21:37.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:36 smithi151 ceph-mon[41387]: from='client.15072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:37.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:36 smithi151 ceph-mon[41387]: from='client.24980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:37.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:36 smithi151 ceph-mon[41387]: osdmap e114: 8 total, 8 up, 8 in 2024-09-13T14:21:37.779 INFO:tasks.rados.rados.0.smithi151.stdout:242: done (3 left) 2024-09-13T14:21:37.780 INFO:tasks.rados.rados.0.smithi151.stdout:244: done (2 left) 2024-09-13T14:21:37.780 INFO:tasks.rados.rados.0.smithi151.stdout:245: done (1 left) 2024-09-13T14:21:37.780 INFO:tasks.rados.rados.0.smithi151.stdout:246: done (0 left) 2024-09-13T14:21:37.780 INFO:tasks.rados.rados.0.smithi151.stdout:247: snap_create 2024-09-13T14:21:37.783 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:21:37 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:21:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:21:37.894 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:21:38.080 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:37 smithi151 ceph-mon[29807]: pgmap v159: 89 pgs: 89 active+clean; 130 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 689 KiB/s wr, 6 op/s 2024-09-13T14:21:38.081 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:37 smithi151 ceph-mon[29807]: osdmap e115: 8 total, 8 up, 8 in 2024-09-13T14:21:38.081 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:37 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/2638522548' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:38.081 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:37 smithi151 ceph-mon[29807]: osdmap e116: 8 total, 8 up, 8 in 2024-09-13T14:21:38.081 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:37 smithi151 ceph-mon[41387]: pgmap v159: 89 pgs: 89 active+clean; 130 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 689 KiB/s wr, 6 op/s 2024-09-13T14:21:38.081 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:37 smithi151 ceph-mon[41387]: osdmap e115: 8 total, 8 up, 8 in 2024-09-13T14:21:38.081 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:37 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/2638522548' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:38.081 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:37 smithi151 ceph-mon[41387]: osdmap e116: 8 total, 8 up, 8 in 2024-09-13T14:21:38.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:37 smithi184 ceph-mon[33394]: pgmap v159: 89 pgs: 89 active+clean; 130 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 689 KiB/s wr, 6 op/s 2024-09-13T14:21:38.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:37 smithi184 ceph-mon[33394]: osdmap e115: 8 total, 8 up, 8 in 2024-09-13T14:21:38.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:37 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/2638522548' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:38.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:37 smithi184 ceph-mon[33394]: osdmap e116: 8 total, 8 up, 8 in 2024-09-13T14:21:38.780 INFO:tasks.rados.rados.0.smithi151.stdout:247: done (0 left) 2024-09-13T14:21:38.780 INFO:tasks.rados.rados.0.smithi151.stdout:248: write oid 48 current snap is 27 2024-09-13T14:21:38.780 INFO:tasks.rados.rados.0.smithi151.stdout:248: seq_num 128 ranges {551296=693520,2024048=127248} 2024-09-13T14:21:38.787 INFO:tasks.rados.rados.0.smithi151.stdout:248: writing smithi15186934-48 from 551296 to 1244816 tid 1 2024-09-13T14:21:38.793 INFO:tasks.rados.rados.0.smithi151.stdout:248: writing smithi15186934-48 from 2024048 to 2151296 tid 2 2024-09-13T14:21:38.793 INFO:tasks.rados.rados.0.smithi151.stdout:249: write oid 44 current snap is 27 2024-09-13T14:21:38.793 INFO:tasks.rados.rados.0.smithi151.stdout:249: seq_num 129 ranges {568103=768769,2082530=85573} 2024-09-13T14:21:38.800 INFO:tasks.rados.rados.0.smithi151.stdout:249: writing smithi15186934-44 from 568103 to 1336872 tid 1 2024-09-13T14:21:38.804 INFO:tasks.rados.rados.0.smithi151.stdout:249: writing smithi15186934-44 from 2082530 to 2168103 tid 2 2024-09-13T14:21:38.804 INFO:tasks.rados.rados.0.smithi151.stdout:250: snap_remove snap 26 2024-09-13T14:21:39.790 INFO:tasks.rados.rados.0.smithi151.stdout:250: done (2 left) 2024-09-13T14:21:39.790 INFO:tasks.rados.rados.0.smithi151.stdout:251: write oid 8 current snap is 27 2024-09-13T14:21:39.790 INFO:tasks.rados.rados.0.smithi151.stdout:248: finishing write tid 1 to smithi15186934-48 2024-09-13T14:21:39.790 INFO:tasks.rados.rados.0.smithi151.stdout:248: finishing write tid 2 to smithi15186934-48 2024-09-13T14:21:39.790 INFO:tasks.rados.rados.0.smithi151.stdout:248: finishing write tid 3 to smithi15186934-48 2024-09-13T14:21:39.790 INFO:tasks.rados.rados.0.smithi151.stdout:248: finishing write tid 4 to smithi15186934-48 2024-09-13T14:21:39.791 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 48 v 65 (ObjNum 128 snap 27 seq_num 128) dirty exists 2024-09-13T14:21:39.791 INFO:tasks.rados.rados.0.smithi151.stdout:248: left oid 48 (ObjNum 128 snap 27 seq_num 128) 2024-09-13T14:21:39.791 INFO:tasks.rados.rados.0.smithi151.stdout:249: finishing write tid 1 to smithi15186934-44 2024-09-13T14:21:39.791 INFO:tasks.rados.rados.0.smithi151.stdout:249: finishing write tid 2 to smithi15186934-44 2024-09-13T14:21:39.791 INFO:tasks.rados.rados.0.smithi151.stdout:249: finishing write tid 3 to smithi15186934-44 2024-09-13T14:21:39.791 INFO:tasks.rados.rados.0.smithi151.stdout:249: finishing write tid 4 to smithi15186934-44 2024-09-13T14:21:39.791 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 49 (ObjNum 129 snap 27 seq_num 129) dirty exists 2024-09-13T14:21:39.791 INFO:tasks.rados.rados.0.smithi151.stdout:249: left oid 44 (ObjNum 129 snap 27 seq_num 129) 2024-09-13T14:21:39.791 INFO:tasks.rados.rados.0.smithi151.stdout:251: seq_num 130 ranges {584910=560371,1857365=327545} 2024-09-13T14:21:39.796 INFO:tasks.rados.rados.0.smithi151.stdout:251: writing smithi15186934-8 from 584910 to 1145281 tid 1 2024-09-13T14:21:39.802 INFO:tasks.rados.rados.0.smithi151.stdout:251: writing smithi15186934-8 from 1857365 to 2184910 tid 2 2024-09-13T14:21:39.802 INFO:tasks.rados.rados.0.smithi151.stdout:248: done (2 left) 2024-09-13T14:21:39.802 INFO:tasks.rados.rados.0.smithi151.stdout:249: done (1 left) 2024-09-13T14:21:39.802 INFO:tasks.rados.rados.0.smithi151.stdout:252: snap_create 2024-09-13T14:21:39.803 INFO:tasks.rados.rados.0.smithi151.stdout:251: finishing write tid 1 to smithi15186934-8 2024-09-13T14:21:39.804 INFO:tasks.rados.rados.0.smithi151.stdout:251: finishing write tid 2 to smithi15186934-8 2024-09-13T14:21:39.805 INFO:tasks.rados.rados.0.smithi151.stdout:251: finishing write tid 3 to smithi15186934-8 2024-09-13T14:21:39.805 INFO:tasks.rados.rados.0.smithi151.stdout:251: finishing write tid 4 to smithi15186934-8 2024-09-13T14:21:39.805 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 55 (ObjNum 130 snap 27 seq_num 130) dirty exists 2024-09-13T14:21:39.805 INFO:tasks.rados.rados.0.smithi151.stdout:251: left oid 8 (ObjNum 130 snap 27 seq_num 130) 2024-09-13T14:21:40.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:39 smithi184 ceph-mon[33394]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:40.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:39 smithi184 ceph-mon[33394]: from='client.24998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:40.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:39 smithi184 ceph-mon[33394]: pgmap v162: 89 pgs: 89 active+clean; 130 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:40.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:39 smithi184 ceph-mon[33394]: osdmap e117: 8 total, 8 up, 8 in 2024-09-13T14:21:40.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:39 smithi151 ceph-mon[29807]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:40.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:39 smithi151 ceph-mon[29807]: from='client.24998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:40.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:39 smithi151 ceph-mon[29807]: pgmap v162: 89 pgs: 89 active+clean; 130 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:40.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:39 smithi151 ceph-mon[29807]: osdmap e117: 8 total, 8 up, 8 in 2024-09-13T14:21:40.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:39 smithi151 ceph-mon[41387]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:40.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:39 smithi151 ceph-mon[41387]: from='client.24998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:40.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:39 smithi151 ceph-mon[41387]: pgmap v162: 89 pgs: 89 active+clean; 130 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:40.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:39 smithi151 ceph-mon[41387]: osdmap e117: 8 total, 8 up, 8 in 2024-09-13T14:21:40.731 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:21:40.802 INFO:tasks.rados.rados.0.smithi151.stdout:251: done (1 left) 2024-09-13T14:21:40.802 INFO:tasks.rados.rados.0.smithi151.stdout:252: done (0 left) 2024-09-13T14:21:40.802 INFO:tasks.rados.rados.0.smithi151.stdout:253: snap_remove snap 27 2024-09-13T14:21:41.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:40 smithi184 ceph-mon[33394]: osdmap e118: 8 total, 8 up, 8 in 2024-09-13T14:21:41.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:40 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/3933502770' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:41.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:40 smithi151 ceph-mon[29807]: osdmap e118: 8 total, 8 up, 8 in 2024-09-13T14:21:41.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:40 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/3933502770' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:41.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:40 smithi151 ceph-mon[41387]: osdmap e118: 8 total, 8 up, 8 in 2024-09-13T14:21:41.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:40 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/3933502770' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:41.800 INFO:tasks.rados.rados.0.smithi151.stdout:253: done (0 left) 2024-09-13T14:21:41.800 INFO:tasks.rados.rados.0.smithi151.stdout:254: read oid 29 snap -1 2024-09-13T14:21:41.800 INFO:tasks.rados.rados.0.smithi151.stdout:254: expect deleted 2024-09-13T14:21:41.800 INFO:tasks.rados.rados.0.smithi151.stdout:255: read oid 15 snap -1 2024-09-13T14:21:41.800 INFO:tasks.rados.rados.0.smithi151.stdout:255: expect (ObjNum 105 snap 16 seq_num 105) 2024-09-13T14:21:41.800 INFO:tasks.rados.rados.0.smithi151.stdout:256: delete oid 36 current snap is 28 2024-09-13T14:21:41.806 INFO:tasks.rados.rados.0.smithi151.stdout:255: expect (ObjNum 105 snap 16 seq_num 105) 2024-09-13T14:21:41.969 INFO:tasks.rados.rados.0.smithi151.stdout:254: done (2 left) 2024-09-13T14:21:41.969 INFO:tasks.rados.rados.0.smithi151.stdout:255: done (1 left) 2024-09-13T14:21:41.969 INFO:tasks.rados.rados.0.smithi151.stdout:256: done (0 left) 2024-09-13T14:21:41.969 INFO:tasks.rados.rados.0.smithi151.stdout:257: delete oid 44 current snap is 28 2024-09-13T14:21:41.970 INFO:tasks.rados.rados.0.smithi151.stdout:257: done (0 left) 2024-09-13T14:21:41.970 INFO:tasks.rados.rados.0.smithi151.stdout:258: read oid 29 snap -1 2024-09-13T14:21:41.970 INFO:tasks.rados.rados.0.smithi151.stdout:258: expect deleted 2024-09-13T14:21:41.970 INFO:tasks.rados.rados.0.smithi151.stdout:259: write oid 19 current snap is 28 2024-09-13T14:21:41.970 INFO:tasks.rados.rados.0.smithi151.stdout:259: seq_num 133 ranges {635331=786118,1916458=318873} 2024-09-13T14:21:41.978 INFO:tasks.rados.rados.0.smithi151.stdout:259: writing smithi15186934-19 from 635331 to 1421449 tid 1 2024-09-13T14:21:41.982 INFO:tasks.rados.rados.0.smithi151.stdout:259: writing smithi15186934-19 from 1916458 to 2235331 tid 2 2024-09-13T14:21:41.982 INFO:tasks.rados.rados.0.smithi151.stdout:260: snap_remove snap 19 2024-09-13T14:21:42.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:41 smithi184 ceph-mon[33394]: pgmap v165: 89 pgs: 89 active+clean; 138 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.4 MiB/s wr, 16 op/s 2024-09-13T14:21:42.112 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:41 smithi184 ceph-mon[33394]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:42.112 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:41 smithi184 ceph-mon[33394]: osdmap e119: 8 total, 8 up, 8 in 2024-09-13T14:21:42.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:41 smithi151 ceph-mon[41387]: pgmap v165: 89 pgs: 89 active+clean; 138 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.4 MiB/s wr, 16 op/s 2024-09-13T14:21:42.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:41 smithi151 ceph-mon[41387]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:42.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:41 smithi151 ceph-mon[41387]: osdmap e119: 8 total, 8 up, 8 in 2024-09-13T14:21:42.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:41 smithi151 ceph-mon[29807]: pgmap v165: 89 pgs: 89 active+clean; 138 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.4 MiB/s wr, 16 op/s 2024-09-13T14:21:42.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:41 smithi151 ceph-mon[29807]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:42.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:41 smithi151 ceph-mon[29807]: osdmap e119: 8 total, 8 up, 8 in 2024-09-13T14:21:42.785 INFO:tasks.rados.rados.0.smithi151.stdout:259: finishing write tid 1 to smithi15186934-19 2024-09-13T14:21:42.785 INFO:tasks.rados.rados.0.smithi151.stdout:259: finishing write tid 2 to smithi15186934-19 2024-09-13T14:21:42.785 INFO:tasks.rados.rados.0.smithi151.stdout:259: finishing write tid 3 to smithi15186934-19 2024-09-13T14:21:42.785 INFO:tasks.rados.rados.0.smithi151.stdout:259: finishing write tid 4 to smithi15186934-19 2024-09-13T14:21:42.785 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 19 v 62 (ObjNum 133 snap 28 seq_num 133) dirty exists 2024-09-13T14:21:42.785 INFO:tasks.rados.rados.0.smithi151.stdout:259: left oid 19 (ObjNum 133 snap 28 seq_num 133) 2024-09-13T14:21:42.785 INFO:tasks.rados.rados.0.smithi151.stdout:258: done (2 left) 2024-09-13T14:21:42.785 INFO:tasks.rados.rados.0.smithi151.stdout:259: done (1 left) 2024-09-13T14:21:42.786 INFO:tasks.rados.rados.0.smithi151.stdout:260: done (0 left) 2024-09-13T14:21:42.786 INFO:tasks.rados.rados.0.smithi151.stdout:261: write oid 4 current snap is 28 2024-09-13T14:21:42.786 INFO:tasks.rados.rados.0.smithi151.stdout:261: seq_num 134 ranges {652138=461367,1574940=677198} 2024-09-13T14:21:42.791 INFO:tasks.rados.rados.0.smithi151.stdout:261: writing smithi15186934-4 from 652138 to 1113505 tid 1 2024-09-13T14:21:42.798 INFO:tasks.rados.rados.0.smithi151.stdout:261: writing smithi15186934-4 from 1574940 to 2252138 tid 2 2024-09-13T14:21:42.798 INFO:tasks.rados.rados.0.smithi151.stdout:262: read oid 39 snap -1 2024-09-13T14:21:42.798 INFO:tasks.rados.rados.0.smithi151.stdout:262: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-13T14:21:42.798 INFO:tasks.rados.rados.0.smithi151.stdout:263: write oid 8 current snap is 28 2024-09-13T14:21:42.798 INFO:tasks.rados.rados.0.smithi151.stdout:261: finishing write tid 1 to smithi15186934-4 2024-09-13T14:21:42.798 INFO:tasks.rados.rados.0.smithi151.stdout:263: seq_num 135 ranges {668945=536616,1633422=491363} 2024-09-13T14:21:42.805 INFO:tasks.rados.rados.0.smithi151.stdout:263: writing smithi15186934-8 from 668945 to 1205561 tid 1 2024-09-13T14:21:42.810 INFO:tasks.rados.rados.0.smithi151.stdout:263: writing smithi15186934-8 from 1633422 to 2124785 tid 2 2024-09-13T14:21:42.811 INFO:tasks.rados.rados.0.smithi151.stdout:264: read oid 14 snap -1 2024-09-13T14:21:42.811 INFO:tasks.rados.rados.0.smithi151.stdout:264: expect (ObjNum 75 snap 9 seq_num 75) 2024-09-13T14:21:42.811 INFO:tasks.rados.rados.0.smithi151.stdout:265: read oid 20 snap -1 2024-09-13T14:21:42.811 INFO:tasks.rados.rados.0.smithi151.stdout:265: expect deleted 2024-09-13T14:21:42.811 INFO:tasks.rados.rados.0.smithi151.stdout:266: write oid 19 current snap is 28 2024-09-13T14:21:42.811 INFO:tasks.rados.rados.0.smithi151.stdout:266: seq_num 136 ranges {685752=611865,1975551=310201} 2024-09-13T14:21:42.817 INFO:tasks.rados.rados.0.smithi151.stdout:266: writing smithi15186934-19 from 685752 to 1297617 tid 1 2024-09-13T14:21:42.823 INFO:tasks.rados.rados.0.smithi151.stdout:266: writing smithi15186934-19 from 1975551 to 2285752 tid 2 2024-09-13T14:21:42.823 INFO:tasks.rados.rados.0.smithi151.stdout:267: rollback oid 18 current snap is 28 2024-09-13T14:21:42.823 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 18 to 24 2024-09-13T14:21:42.823 INFO:tasks.rados.rados.0.smithi151.stdout:261: finishing write tid 2 to smithi15186934-4 2024-09-13T14:21:42.823 INFO:tasks.rados.rados.0.smithi151.stdout:261: finishing write tid 3 to smithi15186934-4 2024-09-13T14:21:42.823 INFO:tasks.rados.rados.0.smithi151.stdout:261: finishing write tid 4 to smithi15186934-4 2024-09-13T14:21:42.824 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 73 (ObjNum 134 snap 28 seq_num 134) dirty exists 2024-09-13T14:21:42.824 INFO:tasks.rados.rados.0.smithi151.stdout:261: left oid 4 (ObjNum 134 snap 28 seq_num 134) 2024-09-13T14:21:42.824 INFO:tasks.rados.rados.0.smithi151.stdout:262: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-13T14:21:42.911 INFO:tasks.rados.rados.0.smithi151.stdout:263: finishing write tid 1 to smithi15186934-8 2024-09-13T14:21:42.911 INFO:tasks.rados.rados.0.smithi151.stdout:261: done (6 left) 2024-09-13T14:21:42.911 INFO:tasks.rados.rados.0.smithi151.stdout:262: done (5 left) 2024-09-13T14:21:42.911 INFO:tasks.rados.rados.0.smithi151.stdout:268: write oid 1 current snap is 28 2024-09-13T14:21:42.911 INFO:tasks.rados.rados.0.smithi151.stdout:268: seq_num 137 ranges {702559=403467,1750386=495032} 2024-09-13T14:21:42.917 INFO:tasks.rados.rados.0.smithi151.stdout:268: writing smithi15186934-1 from 702559 to 1106026 tid 1 2024-09-13T14:21:42.923 INFO:tasks.rados.rados.0.smithi151.stdout:268: writing smithi15186934-1 from 1750386 to 2245418 tid 2 2024-09-13T14:21:42.923 INFO:tasks.rados.rados.0.smithi151.stdout:269: read oid 7 snap -1 2024-09-13T14:21:42.924 INFO:tasks.rados.rados.0.smithi151.stdout:269: expect (ObjNum 120 snap 22 seq_num 120) 2024-09-13T14:21:42.924 INFO:tasks.rados.rados.0.smithi151.stdout:270: rollback oid 27 current snap is 28 2024-09-13T14:21:42.924 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 27 to 24 2024-09-13T14:21:42.924 INFO:tasks.rados.rados.0.smithi151.stdout:263: finishing write tid 2 to smithi15186934-8 2024-09-13T14:21:42.924 INFO:tasks.rados.rados.0.smithi151.stdout:263: finishing write tid 3 to smithi15186934-8 2024-09-13T14:21:42.924 INFO:tasks.rados.rados.0.smithi151.stdout:265: done (7 left) 2024-09-13T14:21:42.924 INFO:tasks.rados.rados.0.smithi151.stdout:271: snap_create 2024-09-13T14:21:42.924 INFO:tasks.rados.rados.0.smithi151.stdout:263: finishing write tid 4 to smithi15186934-8 2024-09-13T14:21:42.924 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 63 (ObjNum 135 snap 28 seq_num 135) dirty exists 2024-09-13T14:21:42.924 INFO:tasks.rados.rados.0.smithi151.stdout:263: left oid 8 (ObjNum 135 snap 28 seq_num 135) 2024-09-13T14:21:42.924 INFO:tasks.rados.rados.0.smithi151.stdout:264: expect (ObjNum 75 snap 9 seq_num 75) 2024-09-13T14:21:43.029 INFO:tasks.rados.rados.0.smithi151.stdout:266: finishing write tid 1 to smithi15186934-19 2024-09-13T14:21:43.029 INFO:tasks.rados.rados.0.smithi151.stdout:267: finishing rollback tid 0 to smithi15186934-18 2024-09-13T14:21:43.030 INFO:tasks.rados.rados.0.smithi151.stdout:266: finishing write tid 2 to smithi15186934-19 2024-09-13T14:21:43.030 INFO:tasks.rados.rados.0.smithi151.stdout:267: finishing rollback tid 1 to smithi15186934-18 2024-09-13T14:21:43.030 INFO:tasks.rados.rados.0.smithi151.stdout:266: finishing write tid 3 to smithi15186934-19 2024-09-13T14:21:43.030 INFO:tasks.rados.rados.0.smithi151.stdout:267: finishing rollback tid 2 to smithi15186934-18 2024-09-13T14:21:43.030 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 18 v 45 (ObjNum 118 snap 20 seq_num 118) dirty exists 2024-09-13T14:21:43.030 INFO:tasks.rados.rados.0.smithi151.stdout:266: finishing write tid 4 to smithi15186934-19 2024-09-13T14:21:43.030 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 19 v 67 (ObjNum 136 snap 28 seq_num 136) dirty exists 2024-09-13T14:21:43.030 INFO:tasks.rados.rados.0.smithi151.stdout:266: left oid 19 (ObjNum 136 snap 28 seq_num 136) 2024-09-13T14:21:43.030 INFO:tasks.rados.rados.0.smithi151.stdout:268: finishing write tid 1 to smithi15186934-1 2024-09-13T14:21:43.030 INFO:tasks.rados.rados.0.smithi151.stdout:268: finishing write tid 2 to smithi15186934-1 2024-09-13T14:21:43.030 INFO:tasks.rados.rados.0.smithi151.stdout:268: finishing write tid 3 to smithi15186934-1 2024-09-13T14:21:43.030 INFO:tasks.rados.rados.0.smithi151.stdout:268: finishing write tid 4 to smithi15186934-1 2024-09-13T14:21:43.030 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 1 v 67 (ObjNum 137 snap 28 seq_num 137) dirty exists 2024-09-13T14:21:43.031 INFO:tasks.rados.rados.0.smithi151.stdout:268: left oid 1 (ObjNum 137 snap 28 seq_num 137) 2024-09-13T14:21:43.031 INFO:tasks.rados.rados.0.smithi151.stdout:270: finishing rollback tid 1 to smithi15186934-27 2024-09-13T14:21:43.031 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 27 v 62 (ObjNum 129 snap 0 seq_num 3093055600) dirty dne 2024-09-13T14:21:43.031 INFO:tasks.rados.rados.0.smithi151.stdout:269: expect (ObjNum 120 snap 22 seq_num 120) 2024-09-13T14:21:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:42 smithi184 ceph-mon[33394]: from='client.25016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:42 smithi184 ceph-mon[33394]: osdmap e120: 8 total, 8 up, 8 in 2024-09-13T14:21:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:42 smithi184 ceph-mon[33394]: osdmap e121: 8 total, 8 up, 8 in 2024-09-13T14:21:43.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:42 smithi151 ceph-mon[29807]: from='client.25016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:43.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:42 smithi151 ceph-mon[29807]: osdmap e120: 8 total, 8 up, 8 in 2024-09-13T14:21:43.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:42 smithi151 ceph-mon[29807]: osdmap e121: 8 total, 8 up, 8 in 2024-09-13T14:21:43.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:42 smithi151 ceph-mon[41387]: from='client.25016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:43.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:42 smithi151 ceph-mon[41387]: osdmap e120: 8 total, 8 up, 8 in 2024-09-13T14:21:43.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:42 smithi151 ceph-mon[41387]: osdmap e121: 8 total, 8 up, 8 in 2024-09-13T14:21:43.109 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:21:42 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:21:42] "GET /metrics HTTP/1.1" 200 243280 "" "Prometheus/2.43.0" 2024-09-13T14:21:43.560 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:21:43.821 INFO:tasks.rados.rados.0.smithi151.stdout:263: done (7 left) 2024-09-13T14:21:43.821 INFO:tasks.rados.rados.0.smithi151.stdout:264: done (6 left) 2024-09-13T14:21:43.821 INFO:tasks.rados.rados.0.smithi151.stdout:266: done (5 left) 2024-09-13T14:21:43.821 INFO:tasks.rados.rados.0.smithi151.stdout:267: done (4 left) 2024-09-13T14:21:43.821 INFO:tasks.rados.rados.0.smithi151.stdout:268: done (3 left) 2024-09-13T14:21:43.821 INFO:tasks.rados.rados.0.smithi151.stdout:269: done (2 left) 2024-09-13T14:21:43.822 INFO:tasks.rados.rados.0.smithi151.stdout:270: done (1 left) 2024-09-13T14:21:43.822 INFO:tasks.rados.rados.0.smithi151.stdout:271: done (0 left) 2024-09-13T14:21:43.822 INFO:tasks.rados.rados.0.smithi151.stdout:272: snap_create 2024-09-13T14:21:44.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:43 smithi184 ceph-mon[33394]: pgmap v168: 89 pgs: 89 active+clean; 138 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.4 MiB/s wr, 16 op/s 2024-09-13T14:21:44.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:43 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/1390229279' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:44.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:43 smithi151 ceph-mon[41387]: pgmap v168: 89 pgs: 89 active+clean; 138 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.4 MiB/s wr, 16 op/s 2024-09-13T14:21:44.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:43 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/1390229279' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:44.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:43 smithi151 ceph-mon[29807]: pgmap v168: 89 pgs: 89 active+clean; 138 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.4 MiB/s wr, 16 op/s 2024-09-13T14:21:44.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:43 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1390229279' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:44.823 INFO:tasks.rados.rados.0.smithi151.stdout:272: done (0 left) 2024-09-13T14:21:44.824 INFO:tasks.rados.rados.0.smithi151.stdout:273: write oid 31 current snap is 30 2024-09-13T14:21:44.824 INFO:tasks.rados.rados.0.smithi151.stdout:273: seq_num 138 ranges {719366=478716,1808868=510498} 2024-09-13T14:21:44.830 INFO:tasks.rados.rados.0.smithi151.stdout:273: writing smithi15186934-31 from 719366 to 1198082 tid 1 2024-09-13T14:21:44.836 INFO:tasks.rados.rados.0.smithi151.stdout:273: writing smithi15186934-31 from 1808868 to 2319366 tid 2 2024-09-13T14:21:44.837 INFO:tasks.rados.rados.0.smithi151.stdout:274: delete oid 38 current snap is 30 2024-09-13T14:21:44.837 INFO:tasks.rados.rados.0.smithi151.stdout:273: finishing write tid 1 to smithi15186934-31 2024-09-13T14:21:44.839 INFO:tasks.rados.rados.0.smithi151.stdout:273: finishing write tid 2 to smithi15186934-31 2024-09-13T14:21:44.839 INFO:tasks.rados.rados.0.smithi151.stdout:273: finishing write tid 3 to smithi15186934-31 2024-09-13T14:21:44.839 INFO:tasks.rados.rados.0.smithi151.stdout:273: finishing write tid 4 to smithi15186934-31 2024-09-13T14:21:44.839 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 31 v 121 (ObjNum 138 snap 30 seq_num 138) dirty exists 2024-09-13T14:21:44.839 INFO:tasks.rados.rados.0.smithi151.stdout:273: left oid 31 (ObjNum 138 snap 30 seq_num 138) 2024-09-13T14:21:44.840 INFO:tasks.rados.rados.0.smithi151.stdout:273: done (1 left) 2024-09-13T14:21:44.840 INFO:tasks.rados.rados.0.smithi151.stdout:274: done (0 left) 2024-09-13T14:21:44.840 INFO:tasks.rados.rados.0.smithi151.stdout:275: rollback oid 20 current snap is 30 2024-09-13T14:21:44.840 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 20 to 25 2024-09-13T14:21:44.840 INFO:tasks.rados.rados.0.smithi151.stdout:276: snap_create 2024-09-13T14:21:44.842 INFO:tasks.rados.rados.0.smithi151.stdout:275: finishing rollback tid 1 to smithi15186934-20 2024-09-13T14:21:44.842 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 0 (ObjNum 129 snap 0 seq_num 3093011664) dirty dne 2024-09-13T14:21:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:44 smithi184 ceph-mon[33394]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:44 smithi184 ceph-mon[33394]: from='client.25034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:44 smithi184 ceph-mon[33394]: osdmap e122: 8 total, 8 up, 8 in 2024-09-13T14:21:45.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:44 smithi151 ceph-mon[29807]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:45.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:44 smithi151 ceph-mon[29807]: from='client.25034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:45.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:44 smithi151 ceph-mon[29807]: osdmap e122: 8 total, 8 up, 8 in 2024-09-13T14:21:45.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:44 smithi151 ceph-mon[41387]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:45.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:44 smithi151 ceph-mon[41387]: from='client.25034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:45.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:44 smithi151 ceph-mon[41387]: osdmap e122: 8 total, 8 up, 8 in 2024-09-13T14:21:45.841 INFO:tasks.rados.rados.0.smithi151.stdout:275: done (1 left) 2024-09-13T14:21:45.841 INFO:tasks.rados.rados.0.smithi151.stdout:276: done (0 left) 2024-09-13T14:21:45.842 INFO:tasks.rados.rados.0.smithi151.stdout:277: write oid 19 current snap is 31 2024-09-13T14:21:45.842 INFO:tasks.rados.rados.0.smithi151.stdout:277: seq_num 140 ranges {752980=629214,1809479=543501} 2024-09-13T14:21:45.849 INFO:tasks.rados.rados.0.smithi151.stdout:277: writing smithi15186934-19 from 752980 to 1382194 tid 1 2024-09-13T14:21:45.854 INFO:tasks.rados.rados.0.smithi151.stdout:277: writing smithi15186934-19 from 1809479 to 2352980 tid 2 2024-09-13T14:21:45.854 INFO:tasks.rados.rados.0.smithi151.stdout:278: snap_remove snap 29 2024-09-13T14:21:46.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:45 smithi184 ceph-mon[33394]: pgmap v171: 89 pgs: 89 active+clean; 136 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 1.4 MiB/s wr, 17 op/s 2024-09-13T14:21:46.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:45 smithi184 ceph-mon[33394]: osdmap e123: 8 total, 8 up, 8 in 2024-09-13T14:21:46.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:45 smithi151 ceph-mon[29807]: pgmap v171: 89 pgs: 89 active+clean; 136 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 1.4 MiB/s wr, 17 op/s 2024-09-13T14:21:46.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:45 smithi151 ceph-mon[29807]: osdmap e123: 8 total, 8 up, 8 in 2024-09-13T14:21:46.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:45 smithi151 ceph-mon[41387]: pgmap v171: 89 pgs: 89 active+clean; 136 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 1.4 MiB/s wr, 17 op/s 2024-09-13T14:21:46.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:45 smithi151 ceph-mon[41387]: osdmap e123: 8 total, 8 up, 8 in 2024-09-13T14:21:46.410 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:21:46.847 INFO:tasks.rados.rados.0.smithi151.stdout:278: done (1 left) 2024-09-13T14:21:46.847 INFO:tasks.rados.rados.0.smithi151.stdout:279: delete oid 33 current snap is 31 2024-09-13T14:21:46.847 INFO:tasks.rados.rados.0.smithi151.stdout:277: finishing write tid 1 to smithi15186934-19 2024-09-13T14:21:46.847 INFO:tasks.rados.rados.0.smithi151.stdout:277: finishing write tid 2 to smithi15186934-19 2024-09-13T14:21:46.847 INFO:tasks.rados.rados.0.smithi151.stdout:277: finishing write tid 3 to smithi15186934-19 2024-09-13T14:21:46.847 INFO:tasks.rados.rados.0.smithi151.stdout:277: finishing write tid 4 to smithi15186934-19 2024-09-13T14:21:46.848 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 19 v 71 (ObjNum 140 snap 31 seq_num 140) dirty exists 2024-09-13T14:21:46.848 INFO:tasks.rados.rados.0.smithi151.stdout:277: left oid 19 (ObjNum 140 snap 31 seq_num 140) 2024-09-13T14:21:46.849 INFO:tasks.rados.rados.0.smithi151.stdout:277: done (1 left) 2024-09-13T14:21:46.849 INFO:tasks.rados.rados.0.smithi151.stdout:279: done (0 left) 2024-09-13T14:21:46.849 INFO:tasks.rados.rados.0.smithi151.stdout:280: snap_remove snap 24 2024-09-13T14:21:47.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:46 smithi184 ceph-mon[33394]: osdmap e124: 8 total, 8 up, 8 in 2024-09-13T14:21:47.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:46 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/4240652930' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:47.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:46 smithi151 ceph-mon[29807]: osdmap e124: 8 total, 8 up, 8 in 2024-09-13T14:21:47.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:46 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/4240652930' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:47.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:46 smithi151 ceph-mon[41387]: osdmap e124: 8 total, 8 up, 8 in 2024-09-13T14:21:47.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:46 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/4240652930' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:47.784 INFO:tasks.rados.rados.0.smithi151.stdout:280: done (0 left) 2024-09-13T14:21:47.784 INFO:tasks.rados.rados.0.smithi151.stdout:281: rollback oid 49 current snap is 31 2024-09-13T14:21:47.784 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 49 to 22 2024-09-13T14:21:47.784 INFO:tasks.rados.rados.0.smithi151.stdout:282: snap_remove snap 25 2024-09-13T14:21:47.844 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:21:47 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[35448]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:21:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:21:48.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:47 smithi184 ceph-mon[33394]: pgmap v174: 89 pgs: 89 active+clean; 136 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 1.4 MiB/s wr, 17 op/s 2024-09-13T14:21:48.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:47 smithi184 ceph-mon[33394]: from='client.25046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:48.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:47 smithi184 ceph-mon[33394]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:48.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:47 smithi184 ceph-mon[33394]: osdmap e125: 8 total, 8 up, 8 in 2024-09-13T14:21:48.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:47 smithi184 ceph-mon[33394]: osdmap e126: 8 total, 8 up, 8 in 2024-09-13T14:21:48.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:47 smithi151 ceph-mon[29807]: pgmap v174: 89 pgs: 89 active+clean; 136 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 1.4 MiB/s wr, 17 op/s 2024-09-13T14:21:48.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:47 smithi151 ceph-mon[29807]: from='client.25046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:48.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:47 smithi151 ceph-mon[29807]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:48.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:47 smithi151 ceph-mon[29807]: osdmap e125: 8 total, 8 up, 8 in 2024-09-13T14:21:48.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:47 smithi151 ceph-mon[29807]: osdmap e126: 8 total, 8 up, 8 in 2024-09-13T14:21:48.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:47 smithi151 ceph-mon[41387]: pgmap v174: 89 pgs: 89 active+clean; 136 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 1.4 MiB/s wr, 17 op/s 2024-09-13T14:21:48.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:47 smithi151 ceph-mon[41387]: from='client.25046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:48.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:47 smithi151 ceph-mon[41387]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:48.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:47 smithi151 ceph-mon[41387]: osdmap e125: 8 total, 8 up, 8 in 2024-09-13T14:21:48.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:47 smithi151 ceph-mon[41387]: osdmap e126: 8 total, 8 up, 8 in 2024-09-13T14:21:48.785 INFO:tasks.rados.rados.0.smithi151.stdout:281: finishing rollback tid 0 to smithi15186934-49 2024-09-13T14:21:48.785 INFO:tasks.rados.rados.0.smithi151.stdout:281: finishing rollback tid 1 to smithi15186934-49 2024-09-13T14:21:48.785 INFO:tasks.rados.rados.0.smithi151.stdout:281: finishing rollback tid 2 to smithi15186934-49 2024-09-13T14:21:48.786 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 49 v 123 (ObjNum 113 snap 19 seq_num 113) dirty exists 2024-09-13T14:21:48.786 INFO:tasks.rados.rados.0.smithi151.stdout:281: done (1 left) 2024-09-13T14:21:48.786 INFO:tasks.rados.rados.0.smithi151.stdout:282: done (0 left) 2024-09-13T14:21:48.786 INFO:tasks.rados.rados.0.smithi151.stdout:283: read oid 19 snap -1 2024-09-13T14:21:48.786 INFO:tasks.rados.rados.0.smithi151.stdout:283: expect (ObjNum 140 snap 31 seq_num 140) 2024-09-13T14:21:48.786 INFO:tasks.rados.rados.0.smithi151.stdout:284: snap_create 2024-09-13T14:21:48.793 INFO:tasks.rados.rados.0.smithi151.stdout:283: expect (ObjNum 140 snap 31 seq_num 140) 2024-09-13T14:21:49.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:48 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:21:49.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:48 smithi184 ceph-mon[33394]: osdmap e127: 8 total, 8 up, 8 in 2024-09-13T14:21:49.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:48 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:21:49.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:48 smithi151 ceph-mon[29807]: osdmap e127: 8 total, 8 up, 8 in 2024-09-13T14:21:49.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:48 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:21:49.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:48 smithi151 ceph-mon[41387]: osdmap e127: 8 total, 8 up, 8 in 2024-09-13T14:21:49.260 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:21:49.785 INFO:tasks.rados.rados.0.smithi151.stdout:283: done (1 left) 2024-09-13T14:21:49.785 INFO:tasks.rados.rados.0.smithi151.stdout:284: done (0 left) 2024-09-13T14:21:49.786 INFO:tasks.rados.rados.0.smithi151.stdout:285: snap_create 2024-09-13T14:21:50.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:49 smithi184 ceph-mon[33394]: pgmap v177: 89 pgs: 89 active+clean; 136 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:50.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:49 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/740731871' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:50.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:49 smithi184 ceph-mon[33394]: osdmap e128: 8 total, 8 up, 8 in 2024-09-13T14:21:50.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:49 smithi151 ceph-mon[29807]: pgmap v177: 89 pgs: 89 active+clean; 136 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:50.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:49 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/740731871' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:50.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:49 smithi151 ceph-mon[29807]: osdmap e128: 8 total, 8 up, 8 in 2024-09-13T14:21:50.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:49 smithi151 ceph-mon[41387]: pgmap v177: 89 pgs: 89 active+clean; 136 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:21:50.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:49 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/740731871' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:50.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:49 smithi151 ceph-mon[41387]: osdmap e128: 8 total, 8 up, 8 in 2024-09-13T14:21:50.789 INFO:tasks.rados.rados.0.smithi151.stdout:285: done (0 left) 2024-09-13T14:21:50.789 INFO:tasks.rados.rados.0.smithi151.stdout:286: snap_create 2024-09-13T14:21:51.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:50 smithi151 ceph-mon[29807]: from='client.25064 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:51.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:50 smithi151 ceph-mon[29807]: from='client.25070 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:51.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:50 smithi151 ceph-mon[29807]: osdmap e129: 8 total, 8 up, 8 in 2024-09-13T14:21:51.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:50 smithi151 ceph-mon[41387]: from='client.25064 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:51.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:50 smithi151 ceph-mon[41387]: from='client.25070 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:51.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:50 smithi151 ceph-mon[41387]: osdmap e129: 8 total, 8 up, 8 in 2024-09-13T14:21:51.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:50 smithi184 ceph-mon[33394]: from='client.25064 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:51.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:50 smithi184 ceph-mon[33394]: from='client.25070 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:51.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:50 smithi184 ceph-mon[33394]: osdmap e129: 8 total, 8 up, 8 in 2024-09-13T14:21:51.788 INFO:tasks.rados.rados.0.smithi151.stdout:286: done (0 left) 2024-09-13T14:21:51.788 INFO:tasks.rados.rados.0.smithi151.stdout:287: write oid 13 current snap is 34 2024-09-13T14:21:51.788 INFO:tasks.rados.rados.0.smithi151.stdout:287: seq_num 142 ranges {786594=779712,2326443=60151} 2024-09-13T14:21:51.798 INFO:tasks.rados.rados.0.smithi151.stdout:287: writing smithi15186934-13 from 786594 to 1566306 tid 1 2024-09-13T14:21:51.803 INFO:tasks.rados.rados.0.smithi151.stdout:287: writing smithi15186934-13 from 2326443 to 2386594 tid 2 2024-09-13T14:21:51.803 INFO:tasks.rados.rados.0.smithi151.stdout:288: write oid 29 current snap is 34 2024-09-13T14:21:51.803 INFO:tasks.rados.rados.0.smithi151.stdout:288: seq_num 143 ranges {403401=454961,1584925=506039} 2024-09-13T14:21:51.809 INFO:tasks.rados.rados.0.smithi151.stdout:288: writing smithi15186934-29 from 403401 to 858362 tid 1 2024-09-13T14:21:51.817 INFO:tasks.rados.rados.0.smithi151.stdout:288: writing smithi15186934-29 from 1584925 to 2090964 tid 2 2024-09-13T14:21:51.818 INFO:tasks.rados.rados.0.smithi151.stdout:289: read oid 30 snap -1 2024-09-13T14:21:51.818 INFO:tasks.rados.rados.0.smithi151.stdout:289: expect (ObjNum 29 snap 0 seq_num 29) 2024-09-13T14:21:51.818 INFO:tasks.rados.rados.0.smithi151.stdout:290: delete oid 22 current snap is 34 2024-09-13T14:21:51.818 INFO:tasks.rados.rados.0.smithi151.stdout:287: finishing write tid 1 to smithi15186934-13 2024-09-13T14:21:51.818 INFO:tasks.rados.rados.0.smithi151.stdout:287: finishing write tid 2 to smithi15186934-13 2024-09-13T14:21:51.818 INFO:tasks.rados.rados.0.smithi151.stdout:287: finishing write tid 3 to smithi15186934-13 2024-09-13T14:21:51.818 INFO:tasks.rados.rados.0.smithi151.stdout:287: finishing write tid 4 to smithi15186934-13 2024-09-13T14:21:51.818 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 13 v 44 (ObjNum 142 snap 34 seq_num 142) dirty exists 2024-09-13T14:21:51.818 INFO:tasks.rados.rados.0.smithi151.stdout:287: left oid 13 (ObjNum 142 snap 34 seq_num 142) 2024-09-13T14:21:51.818 INFO:tasks.rados.rados.0.smithi151.stdout:288: finishing write tid 1 to smithi15186934-29 2024-09-13T14:21:51.820 INFO:tasks.rados.rados.0.smithi151.stdout:287: done (3 left) 2024-09-13T14:21:51.820 INFO:tasks.rados.rados.0.smithi151.stdout:290: done (2 left) 2024-09-13T14:21:51.820 INFO:tasks.rados.rados.0.smithi151.stdout:291: write oid 7 current snap is 34 2024-09-13T14:21:51.820 INFO:tasks.rados.rados.0.smithi151.stdout:291: seq_num 145 ranges {437015=721812,1701889=509708} 2024-09-13T14:21:51.826 INFO:tasks.rados.rados.0.smithi151.stdout:291: writing smithi15186934-7 from 437015 to 1158827 tid 1 2024-09-13T14:21:51.832 INFO:tasks.rados.rados.0.smithi151.stdout:291: writing smithi15186934-7 from 1701889 to 2211597 tid 2 2024-09-13T14:21:51.833 INFO:tasks.rados.rados.0.smithi151.stdout:292: delete oid 38 current snap is 34 2024-09-13T14:21:51.833 INFO:tasks.rados.rados.0.smithi151.stdout:288: finishing write tid 2 to smithi15186934-29 2024-09-13T14:21:51.833 INFO:tasks.rados.rados.0.smithi151.stdout:288: finishing write tid 3 to smithi15186934-29 2024-09-13T14:21:51.833 INFO:tasks.rados.rados.0.smithi151.stdout:289: expect (ObjNum 29 snap 0 seq_num 29) 2024-09-13T14:21:51.921 INFO:tasks.rados.rados.0.smithi151.stdout:288: finishing write tid 4 to smithi15186934-29 2024-09-13T14:21:51.921 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 29 v 54 (ObjNum 143 snap 34 seq_num 143) dirty exists 2024-09-13T14:21:51.921 INFO:tasks.rados.rados.0.smithi151.stdout:288: left oid 29 (ObjNum 143 snap 34 seq_num 143) 2024-09-13T14:21:51.921 INFO:tasks.rados.rados.0.smithi151.stdout:291: finishing write tid 1 to smithi15186934-7 2024-09-13T14:21:51.922 INFO:tasks.rados.rados.0.smithi151.stdout:291: finishing write tid 2 to smithi15186934-7 2024-09-13T14:21:51.922 INFO:tasks.rados.rados.0.smithi151.stdout:288: done (3 left) 2024-09-13T14:21:51.922 INFO:tasks.rados.rados.0.smithi151.stdout:289: done (2 left) 2024-09-13T14:21:51.922 INFO:tasks.rados.rados.0.smithi151.stdout:292: done (1 left) 2024-09-13T14:21:51.922 INFO:tasks.rados.rados.0.smithi151.stdout:293: write oid 6 current snap is 34 2024-09-13T14:21:51.922 INFO:tasks.rados.rados.0.smithi151.stdout:291: finishing write tid 3 to smithi15186934-7 2024-09-13T14:21:51.922 INFO:tasks.rados.rados.0.smithi151.stdout:291: finishing write tid 4 to smithi15186934-7 2024-09-13T14:21:51.922 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 124 (ObjNum 145 snap 34 seq_num 145) dirty exists 2024-09-13T14:21:51.922 INFO:tasks.rados.rados.0.smithi151.stdout:291: left oid 7 (ObjNum 145 snap 34 seq_num 145) 2024-09-13T14:21:51.922 INFO:tasks.rados.rados.0.smithi151.stdout:293: seq_num 147 ranges {470629=472310,1418853=513377} 2024-09-13T14:21:51.926 INFO:tasks.rados.rados.0.smithi151.stdout:293: writing smithi15186934-6 from 470629 to 942939 tid 1 2024-09-13T14:21:51.932 INFO:tasks.rados.rados.0.smithi151.stdout:293: writing smithi15186934-6 from 1418853 to 1932230 tid 2 2024-09-13T14:21:51.932 INFO:tasks.rados.rados.0.smithi151.stdout:291: done (1 left) 2024-09-13T14:21:51.932 INFO:tasks.rados.rados.0.smithi151.stdout:294: write oid 26 current snap is 34 2024-09-13T14:21:51.932 INFO:tasks.rados.rados.0.smithi151.stdout:293: finishing write tid 1 to smithi15186934-6 2024-09-13T14:21:51.932 INFO:tasks.rados.rados.0.smithi151.stdout:294: seq_num 148 ranges {487436=547559,1760982=657035} 2024-09-13T14:21:51.937 INFO:tasks.rados.rados.0.smithi151.stdout:294: writing smithi15186934-26 from 487436 to 1034995 tid 1 2024-09-13T14:21:51.945 INFO:tasks.rados.rados.0.smithi151.stdout:294: writing smithi15186934-26 from 1760982 to 2418017 tid 2 2024-09-13T14:21:51.945 INFO:tasks.rados.rados.0.smithi151.stdout:295: rollback oid 7 current snap is 34 2024-09-13T14:21:51.946 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 7 to 30 2024-09-13T14:21:51.946 INFO:tasks.rados.rados.0.smithi151.stdout:293: finishing write tid 2 to smithi15186934-6 2024-09-13T14:21:51.946 INFO:tasks.rados.rados.0.smithi151.stdout:293: finishing write tid 3 to smithi15186934-6 2024-09-13T14:21:51.946 INFO:tasks.rados.rados.0.smithi151.stdout:293: finishing write tid 4 to smithi15186934-6 2024-09-13T14:21:51.946 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 6 v 81 (ObjNum 147 snap 34 seq_num 147) dirty exists 2024-09-13T14:21:51.946 INFO:tasks.rados.rados.0.smithi151.stdout:293: left oid 6 (ObjNum 147 snap 34 seq_num 147) 2024-09-13T14:21:51.946 INFO:tasks.rados.rados.0.smithi151.stdout:294: finishing write tid 1 to smithi15186934-26 2024-09-13T14:21:51.946 INFO:tasks.rados.rados.0.smithi151.stdout:293: done (2 left) 2024-09-13T14:21:51.946 INFO:tasks.rados.rados.0.smithi151.stdout:296: rollback oid 10 current snap is 34 2024-09-13T14:21:51.946 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 10 to 20 2024-09-13T14:21:51.946 INFO:tasks.rados.rados.0.smithi151.stdout:297: delete oid 8 current snap is 34 2024-09-13T14:21:51.947 INFO:tasks.rados.rados.0.smithi151.stdout:297: done (3 left) 2024-09-13T14:21:51.947 INFO:tasks.rados.rados.0.smithi151.stdout:298: snap_remove snap 28 2024-09-13T14:21:52.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:51 smithi151 ceph-mon[29807]: pgmap v180: 89 pgs: 89 active+clean; 138 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 528 KiB/s wr, 6 op/s 2024-09-13T14:21:52.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:51 smithi151 ceph-mon[29807]: osdmap e130: 8 total, 8 up, 8 in 2024-09-13T14:21:52.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:51 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1455240336' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:52.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:51 smithi151 ceph-mon[41387]: pgmap v180: 89 pgs: 89 active+clean; 138 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 528 KiB/s wr, 6 op/s 2024-09-13T14:21:52.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:51 smithi151 ceph-mon[41387]: osdmap e130: 8 total, 8 up, 8 in 2024-09-13T14:21:52.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:51 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/1455240336' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:52.120 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:21:52.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:51 smithi184 ceph-mon[33394]: pgmap v180: 89 pgs: 89 active+clean; 138 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 528 KiB/s wr, 6 op/s 2024-09-13T14:21:52.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:51 smithi184 ceph-mon[33394]: osdmap e130: 8 total, 8 up, 8 in 2024-09-13T14:21:52.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:51 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/1455240336' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:52.784 INFO:tasks.rados.rados.0.smithi151.stdout:298: done (3 left) 2024-09-13T14:21:52.784 INFO:tasks.rados.rados.0.smithi151.stdout:299: snap_remove snap 32 2024-09-13T14:21:53.358 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:21:52 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:21:52] "GET /metrics HTTP/1.1" 200 243277 "" "Prometheus/2.43.0" 2024-09-13T14:21:53.791 INFO:tasks.rados.rados.0.smithi151.stdout:299: done (3 left) 2024-09-13T14:21:53.791 INFO:tasks.rados.rados.0.smithi151.stdout:300: rollback oid 47 current snap is 34 2024-09-13T14:21:53.791 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 47 to 30 2024-09-13T14:21:53.791 INFO:tasks.rados.rados.0.smithi151.stdout:296: finishing rollback tid 1 to smithi15186934-10 2024-09-13T14:21:53.791 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 0 (ObjNum 129 snap 0 seq_num 3092627520) dirty dne 2024-09-13T14:21:53.791 INFO:tasks.rados.rados.0.smithi151.stdout:295: finishing rollback tid 0 to smithi15186934-7 2024-09-13T14:21:53.791 INFO:tasks.rados.rados.0.smithi151.stdout:295: finishing rollback tid 1 to smithi15186934-7 2024-09-13T14:21:53.792 INFO:tasks.rados.rados.0.smithi151.stdout:295: finishing rollback tid 2 to smithi15186934-7 2024-09-13T14:21:53.792 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 127 (ObjNum 120 snap 22 seq_num 120) dirty exists 2024-09-13T14:21:53.792 INFO:tasks.rados.rados.0.smithi151.stdout:294: finishing write tid 2 to smithi15186934-26 2024-09-13T14:21:53.792 INFO:tasks.rados.rados.0.smithi151.stdout:294: finishing write tid 3 to smithi15186934-26 2024-09-13T14:21:53.792 INFO:tasks.rados.rados.0.smithi151.stdout:294: finishing write tid 4 to smithi15186934-26 2024-09-13T14:21:53.792 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 72 (ObjNum 148 snap 34 seq_num 148) dirty exists 2024-09-13T14:21:53.792 INFO:tasks.rados.rados.0.smithi151.stdout:294: left oid 26 (ObjNum 148 snap 34 seq_num 148) 2024-09-13T14:21:53.792 INFO:tasks.rados.rados.0.smithi151.stdout:294: done (3 left) 2024-09-13T14:21:53.792 INFO:tasks.rados.rados.0.smithi151.stdout:295: done (2 left) 2024-09-13T14:21:53.792 INFO:tasks.rados.rados.0.smithi151.stdout:296: done (1 left) 2024-09-13T14:21:53.792 INFO:tasks.rados.rados.0.smithi151.stdout:301: snap_create 2024-09-13T14:21:53.794 INFO:tasks.rados.rados.0.smithi151.stdout:300: finishing rollback tid 0 to smithi15186934-47 2024-09-13T14:21:53.794 INFO:tasks.rados.rados.0.smithi151.stdout:300: finishing rollback tid 1 to smithi15186934-47 2024-09-13T14:21:53.794 INFO:tasks.rados.rados.0.smithi151.stdout:300: finishing rollback tid 2 to smithi15186934-47 2024-09-13T14:21:53.795 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 131 (ObjNum 79 snap 10 seq_num 79) dirty exists 2024-09-13T14:21:53.926 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:53 smithi184 ceph-mon[33394]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:53.926 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:53 smithi184 ceph-mon[33394]: from='client.25088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:53.926 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:53 smithi184 ceph-mon[33394]: pgmap v183: 89 pgs: 89 active+clean; 138 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 528 KiB/s wr, 6 op/s 2024-09-13T14:21:53.926 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:53 smithi184 ceph-mon[33394]: osdmap e131: 8 total, 8 up, 8 in 2024-09-13T14:21:54.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:53 smithi151 ceph-mon[29807]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:54.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:53 smithi151 ceph-mon[29807]: from='client.25088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:54.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:53 smithi151 ceph-mon[29807]: pgmap v183: 89 pgs: 89 active+clean; 138 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 528 KiB/s wr, 6 op/s 2024-09-13T14:21:54.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:53 smithi151 ceph-mon[29807]: osdmap e131: 8 total, 8 up, 8 in 2024-09-13T14:21:54.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:53 smithi151 ceph-mon[41387]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:54.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:53 smithi151 ceph-mon[41387]: from='client.25088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:54.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:53 smithi151 ceph-mon[41387]: pgmap v183: 89 pgs: 89 active+clean; 138 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 528 KiB/s wr, 6 op/s 2024-09-13T14:21:54.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:53 smithi151 ceph-mon[41387]: osdmap e131: 8 total, 8 up, 8 in 2024-09-13T14:21:54.799 INFO:tasks.rados.rados.0.smithi151.stdout:300: done (1 left) 2024-09-13T14:21:54.799 INFO:tasks.rados.rados.0.smithi151.stdout:301: done (0 left) 2024-09-13T14:21:54.799 INFO:tasks.rados.rados.0.smithi151.stdout:302: write oid 3 current snap is 35 2024-09-13T14:21:54.799 INFO:tasks.rados.rados.0.smithi151.stdout:302: seq_num 150 ranges {521050=698057,1877946=643104} 2024-09-13T14:21:54.805 INFO:tasks.rados.rados.0.smithi151.stdout:302: writing smithi15186934-3 from 521050 to 1219107 tid 1 2024-09-13T14:21:54.813 INFO:tasks.rados.rados.0.smithi151.stdout:302: writing smithi15186934-3 from 1877946 to 2521050 tid 2 2024-09-13T14:21:54.813 INFO:tasks.rados.rados.0.smithi151.stdout:303: read oid 25 snap 30 2024-09-13T14:21:54.813 INFO:tasks.rados.rados.0.smithi151.stdout:303: expect (ObjNum 24 snap 0 seq_num 24) 2024-09-13T14:21:54.813 INFO:tasks.rados.rados.0.smithi151.stdout:304: snap_create 2024-09-13T14:21:54.813 INFO:tasks.rados.rados.0.smithi151.stdout:302: finishing write tid 1 to smithi15186934-3 2024-09-13T14:21:54.816 INFO:tasks.rados.rados.0.smithi151.stdout:302: finishing write tid 2 to smithi15186934-3 2024-09-13T14:21:54.816 INFO:tasks.rados.rados.0.smithi151.stdout:302: finishing write tid 3 to smithi15186934-3 2024-09-13T14:21:54.816 INFO:tasks.rados.rados.0.smithi151.stdout:302: finishing write tid 4 to smithi15186934-3 2024-09-13T14:21:54.816 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 104 (ObjNum 150 snap 35 seq_num 150) dirty exists 2024-09-13T14:21:54.816 INFO:tasks.rados.rados.0.smithi151.stdout:302: left oid 3 (ObjNum 150 snap 35 seq_num 150) 2024-09-13T14:21:54.819 INFO:tasks.rados.rados.0.smithi151.stdout:303: expect (ObjNum 24 snap 0 seq_num 24) 2024-09-13T14:21:54.957 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:21:55.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:54 smithi184 ceph-mon[33394]: osdmap e132: 8 total, 8 up, 8 in 2024-09-13T14:21:55.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:54 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/1291146133' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:55.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:54 smithi151 ceph-mon[41387]: osdmap e132: 8 total, 8 up, 8 in 2024-09-13T14:21:55.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:54 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/1291146133' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:55.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:54 smithi151 ceph-mon[29807]: osdmap e132: 8 total, 8 up, 8 in 2024-09-13T14:21:55.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:54 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1291146133' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:55.800 INFO:tasks.rados.rados.0.smithi151.stdout:302: done (2 left) 2024-09-13T14:21:55.800 INFO:tasks.rados.rados.0.smithi151.stdout:303: done (1 left) 2024-09-13T14:21:55.801 INFO:tasks.rados.rados.0.smithi151.stdout:304: done (0 left) 2024-09-13T14:21:55.801 INFO:tasks.rados.rados.0.smithi151.stdout:305: snap_remove snap 34 2024-09-13T14:21:55.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:55 smithi184 ceph-mon[33394]: pgmap v186: 89 pgs: 89 active+clean; 146 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:21:55.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:55 smithi184 ceph-mon[33394]: osdmap e133: 8 total, 8 up, 8 in 2024-09-13T14:21:55.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:55 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:21:55.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:55 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:21:55.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:55 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:21:55.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:55 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:21:56.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:55 smithi151 ceph-mon[29807]: pgmap v186: 89 pgs: 89 active+clean; 146 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:21:56.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:55 smithi151 ceph-mon[29807]: osdmap e133: 8 total, 8 up, 8 in 2024-09-13T14:21:56.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:55 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:21:56.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:55 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:21:56.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:55 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:21:56.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:55 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:21:56.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:55 smithi151 ceph-mon[41387]: pgmap v186: 89 pgs: 89 active+clean; 146 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:21:56.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:55 smithi151 ceph-mon[41387]: osdmap e133: 8 total, 8 up, 8 in 2024-09-13T14:21:56.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:55 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:21:56.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:55 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:21:56.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:55 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:21:56.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:55 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:21:56.803 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:21:56 smithi184 systemd[1]: Stopping Ceph mgr.x for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:21:56.807 INFO:tasks.rados.rados.0.smithi151.stdout:305: done (0 left) 2024-09-13T14:21:56.808 INFO:tasks.rados.rados.0.smithi151.stdout:306: read oid 30 snap -1 2024-09-13T14:21:56.808 INFO:tasks.rados.rados.0.smithi151.stdout:306: expect (ObjNum 29 snap 0 seq_num 29) 2024-09-13T14:21:56.808 INFO:tasks.rados.rados.0.smithi151.stdout:307: write oid 9 current snap is 36 2024-09-13T14:21:56.808 INFO:tasks.rados.rados.0.smithi151.stdout:307: seq_num 151 ranges {537857=773306,1936428=520715} 2024-09-13T14:21:56.815 INFO:tasks.rados.rados.0.smithi151.stdout:307: writing smithi15186934-9 from 537857 to 1311163 tid 1 2024-09-13T14:21:56.822 INFO:tasks.rados.rados.0.smithi151.stdout:307: writing smithi15186934-9 from 1936428 to 2457143 tid 2 2024-09-13T14:21:56.822 INFO:tasks.rados.rados.0.smithi151.stdout:308: delete oid 1 current snap is 36 2024-09-13T14:21:56.822 INFO:tasks.rados.rados.0.smithi151.stdout:306: expect (ObjNum 29 snap 0 seq_num 29) 2024-09-13T14:21:56.905 INFO:tasks.rados.rados.0.smithi151.stdout:307: finishing write tid 1 to smithi15186934-9 2024-09-13T14:21:56.905 INFO:tasks.rados.rados.0.smithi151.stdout:307: finishing write tid 2 to smithi15186934-9 2024-09-13T14:21:56.906 INFO:tasks.rados.rados.0.smithi151.stdout:307: finishing write tid 3 to smithi15186934-9 2024-09-13T14:21:56.906 INFO:tasks.rados.rados.0.smithi151.stdout:307: finishing write tid 4 to smithi15186934-9 2024-09-13T14:21:56.906 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 117 (ObjNum 151 snap 36 seq_num 151) dirty exists 2024-09-13T14:21:56.906 INFO:tasks.rados.rados.0.smithi151.stdout:307: left oid 9 (ObjNum 151 snap 36 seq_num 151) 2024-09-13T14:21:56.906 INFO:tasks.rados.rados.0.smithi151.stdout:306: done (2 left) 2024-09-13T14:21:56.906 INFO:tasks.rados.rados.0.smithi151.stdout:307: done (1 left) 2024-09-13T14:21:56.906 INFO:tasks.rados.rados.0.smithi151.stdout:308: done (0 left) 2024-09-13T14:21:56.906 INFO:tasks.rados.rados.0.smithi151.stdout:309: write oid 49 current snap is 36 2024-09-13T14:21:56.906 INFO:tasks.rados.rados.0.smithi151.stdout:309: seq_num 153 ranges {571471=640157,1653392=524384} 2024-09-13T14:21:56.911 INFO:tasks.rados.rados.0.smithi151.stdout:309: writing smithi15186934-49 from 571471 to 1211628 tid 1 2024-09-13T14:21:56.917 INFO:tasks.rados.rados.0.smithi151.stdout:309: writing smithi15186934-49 from 1653392 to 2177776 tid 2 2024-09-13T14:21:56.917 INFO:tasks.rados.rados.0.smithi151.stdout:310: snap_create 2024-09-13T14:21:56.917 INFO:tasks.rados.rados.0.smithi151.stdout:309: finishing write tid 1 to smithi15186934-49 2024-09-13T14:21:56.920 INFO:tasks.rados.rados.0.smithi151.stdout:309: finishing write tid 2 to smithi15186934-49 2024-09-13T14:21:56.920 INFO:tasks.rados.rados.0.smithi151.stdout:309: finishing write tid 3 to smithi15186934-49 2024-09-13T14:21:56.920 INFO:tasks.rados.rados.0.smithi151.stdout:309: finishing write tid 4 to smithi15186934-49 2024-09-13T14:21:56.920 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 49 v 135 (ObjNum 153 snap 36 seq_num 153) dirty exists 2024-09-13T14:21:56.920 INFO:tasks.rados.rados.0.smithi151.stdout:309: left oid 49 (ObjNum 153 snap 36 seq_num 153) 2024-09-13T14:21:57.064 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:56 smithi184 ceph-mon[33394]: from='client.25100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:57.065 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:56 smithi184 ceph-mon[33394]: Upgrade: Updating mgr.x 2024-09-13T14:21:57.065 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:56 smithi184 ceph-mon[33394]: Deploying daemon mgr.x on smithi184 2024-09-13T14:21:57.065 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:56 smithi184 ceph-mon[33394]: from='client.25106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:57.065 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:56 smithi184 ceph-mon[33394]: osdmap e134: 8 total, 8 up, 8 in 2024-09-13T14:21:57.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:56 smithi151 ceph-mon[41387]: from='client.25100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:57.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:56 smithi151 ceph-mon[41387]: Upgrade: Updating mgr.x 2024-09-13T14:21:57.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:56 smithi151 ceph-mon[41387]: Deploying daemon mgr.x on smithi184 2024-09-13T14:21:57.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:56 smithi151 ceph-mon[41387]: from='client.25106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:57.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:56 smithi151 ceph-mon[41387]: osdmap e134: 8 total, 8 up, 8 in 2024-09-13T14:21:57.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:56 smithi151 ceph-mon[29807]: from='client.25100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:57.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:56 smithi151 ceph-mon[29807]: Upgrade: Updating mgr.x 2024-09-13T14:21:57.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:56 smithi151 ceph-mon[29807]: Deploying daemon mgr.x on smithi184 2024-09-13T14:21:57.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:56 smithi151 ceph-mon[29807]: from='client.25106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:57.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:56 smithi151 ceph-mon[29807]: osdmap e134: 8 total, 8 up, 8 in 2024-09-13T14:21:57.353 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:21:57 smithi184 podman[69055]: 2024-09-13 14:21:57.064089407 +0000 UTC m=+0.256835605 container died 1d88b55dd4f147cbe2877ed056139118c951135dfb05c45ae72be3fec2f8374b (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=quincy-bbdadfa, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-09-13T14:21:57.785 INFO:tasks.rados.rados.0.smithi151.stdout:309: done (1 left) 2024-09-13T14:21:57.785 INFO:tasks.rados.rados.0.smithi151.stdout:310: done (0 left) 2024-09-13T14:21:57.785 INFO:tasks.rados.rados.0.smithi151.stdout:311: snap_create 2024-09-13T14:21:57.803 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:21:57 smithi184 podman[69055]: 2024-09-13 14:21:57.508016515 +0000 UTC m=+0.700762754 container cleanup 1d88b55dd4f147cbe2877ed056139118c951135dfb05c45ae72be3fec2f8374b (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, RELEASE=quincy-bbdadfa, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:21:57.803 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:21:57 smithi184 bash[69055]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x 2024-09-13T14:21:57.810 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:21:58.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:57 smithi184 ceph-mon[33394]: pgmap v189: 89 pgs: 89 active+clean; 146 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:21:58.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:57 smithi184 ceph-mon[33394]: osdmap e135: 8 total, 8 up, 8 in 2024-09-13T14:21:58.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:57 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/2717956665' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:58.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:57 smithi184 ceph-mon[33394]: osdmap e136: 8 total, 8 up, 8 in 2024-09-13T14:21:58.104 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:21:57 smithi184 podman[69074]: 2024-09-13 14:21:57.809461067 +0000 UTC m=+0.742859979 container remove 1d88b55dd4f147cbe2877ed056139118c951135dfb05c45ae72be3fec2f8374b (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=quincy-bbdadfa, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-13T14:21:58.104 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:21:57 smithi184 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-13T14:21:58.104 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:21:57 smithi184 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-09-13T14:21:58.104 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:21:57 smithi184 systemd[1]: Stopped Ceph mgr.x for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:21:58.104 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:21:57 smithi184 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@mgr.x.service: Consumed 10.090s CPU time. 2024-09-13T14:21:58.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:57 smithi151 ceph-mon[41387]: pgmap v189: 89 pgs: 89 active+clean; 146 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:21:58.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:57 smithi151 ceph-mon[41387]: osdmap e135: 8 total, 8 up, 8 in 2024-09-13T14:21:58.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:57 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/2717956665' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:58.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:57 smithi151 ceph-mon[41387]: osdmap e136: 8 total, 8 up, 8 in 2024-09-13T14:21:58.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:57 smithi151 ceph-mon[29807]: pgmap v189: 89 pgs: 89 active+clean; 146 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:21:58.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:57 smithi151 ceph-mon[29807]: osdmap e135: 8 total, 8 up, 8 in 2024-09-13T14:21:58.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:57 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/2717956665' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:21:58.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:57 smithi151 ceph-mon[29807]: osdmap e136: 8 total, 8 up, 8 in 2024-09-13T14:21:58.499 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:21:58 smithi184 systemd[1]: Starting Ceph mgr.x for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:21:58.500 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:21:58 smithi184 podman[69210]: 2024-09-13 14:21:58.296187959 +0000 UTC m=+0.015890001 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:21:58.500 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:21:58 smithi184 podman[69210]: 2024-09-13 14:21:58.394131868 +0000 UTC m=+0.113833903 container create 6c42b96484232d0d71c5b936fa80ee86c8b44d8dec39cc4fbe694ff58ac68132 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:21:58.786 INFO:tasks.rados.rados.0.smithi151.stdout:311: done (0 left) 2024-09-13T14:21:58.786 INFO:tasks.rados.rados.0.smithi151.stdout:312: read oid 35 snap -1 2024-09-13T14:21:58.786 INFO:tasks.rados.rados.0.smithi151.stdout:312: expect (ObjNum 106 snap 16 seq_num 106) 2024-09-13T14:21:58.787 INFO:tasks.rados.rados.0.smithi151.stdout:313: delete oid 29 current snap is 38 2024-09-13T14:21:58.790 INFO:tasks.rados.rados.0.smithi151.stdout:313: done (1 left) 2024-09-13T14:21:58.790 INFO:tasks.rados.rados.0.smithi151.stdout:314: snap_remove snap 22 2024-09-13T14:21:59.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:58 smithi184 ceph-mon[33394]: from='client.25118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:59.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:58 smithi184 ceph-mon[33394]: osdmap e137: 8 total, 8 up, 8 in 2024-09-13T14:21:59.104 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:21:58 smithi184 podman[69210]: 2024-09-13 14:21:58.638223902 +0000 UTC m=+0.357925933 container init 6c42b96484232d0d71c5b936fa80ee86c8b44d8dec39cc4fbe694ff58ac68132 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-13T14:21:59.104 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:21:58 smithi184 podman[69210]: 2024-09-13 14:21:58.643283848 +0000 UTC m=+0.362985880 container start 6c42b96484232d0d71c5b936fa80ee86c8b44d8dec39cc4fbe694ff58ac68132 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-13T14:21:59.104 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:21:58 smithi184 ceph-mgr[69248]: -- 172.21.15.184:0/1985793909 <== mon.1 v2:172.21.15.184:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564d4b5a50e0 con 0x564d4b582800 2024-09-13T14:21:59.104 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:21:58 smithi184 bash[69210]: 6c42b96484232d0d71c5b936fa80ee86c8b44d8dec39cc4fbe694ff58ac68132 2024-09-13T14:21:59.105 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:21:58 smithi184 systemd[1]: Started Ceph mgr.x for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:21:59.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:58 smithi151 ceph-mon[29807]: from='client.25118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:59.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:58 smithi151 ceph-mon[29807]: osdmap e137: 8 total, 8 up, 8 in 2024-09-13T14:21:59.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:58 smithi151 ceph-mon[41387]: from='client.25118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:59.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:58 smithi151 ceph-mon[41387]: osdmap e137: 8 total, 8 up, 8 in 2024-09-13T14:21:59.796 INFO:tasks.rados.rados.0.smithi151.stdout:314: done (1 left) 2024-09-13T14:21:59.796 INFO:tasks.rados.rados.0.smithi151.stdout:315: snap_remove snap 30 2024-09-13T14:22:00.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:59 smithi184 ceph-mon[33394]: from='client.25124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:00.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:59 smithi184 ceph-mon[33394]: pgmap v192: 89 pgs: 89 active+clean; 146 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:00.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:59 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:00.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:59 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:00.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:59 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:22:00.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:21:59 smithi184 ceph-mon[33394]: osdmap e138: 8 total, 8 up, 8 in 2024-09-13T14:22:00.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:59 smithi151 ceph-mon[29807]: from='client.25124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:00.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:59 smithi151 ceph-mon[29807]: pgmap v192: 89 pgs: 89 active+clean; 146 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:00.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:59 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:00.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:59 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:00.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:59 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:22:00.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:21:59 smithi151 ceph-mon[29807]: osdmap e138: 8 total, 8 up, 8 in 2024-09-13T14:22:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:59 smithi151 ceph-mon[41387]: from='client.25124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:59 smithi151 ceph-mon[41387]: pgmap v192: 89 pgs: 89 active+clean; 146 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:59 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:59 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:59 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:22:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:21:59 smithi151 ceph-mon[41387]: osdmap e138: 8 total, 8 up, 8 in 2024-09-13T14:22:00.675 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:22:00.799 INFO:tasks.rados.rados.0.smithi151.stdout:315: done (1 left) 2024-09-13T14:22:00.799 INFO:tasks.rados.rados.0.smithi151.stdout:316: write oid 47 current snap is 38 2024-09-13T14:22:00.799 INFO:tasks.rados.rados.0.smithi151.stdout:316: seq_num 155 ranges {605085=790655,2170356=434729} 2024-09-13T14:22:00.805 INFO:tasks.rados.rados.0.smithi151.stdout:316: writing smithi15186934-47 from 605085 to 1395740 tid 1 2024-09-13T14:22:00.812 INFO:tasks.rados.rados.0.smithi151.stdout:316: writing smithi15186934-47 from 2170356 to 2605085 tid 2 2024-09-13T14:22:00.812 INFO:tasks.rados.rados.0.smithi151.stdout:317: snap_create 2024-09-13T14:22:00.812 INFO:tasks.rados.rados.0.smithi151.stdout:312: expect (ObjNum 106 snap 16 seq_num 106) 2024-09-13T14:22:01.009 INFO:tasks.rados.rados.0.smithi151.stdout:316: finishing write tid 1 to smithi15186934-47 2024-09-13T14:22:01.009 INFO:tasks.rados.rados.0.smithi151.stdout:316: finishing write tid 2 to smithi15186934-47 2024-09-13T14:22:01.009 INFO:tasks.rados.rados.0.smithi151.stdout:316: finishing write tid 3 to smithi15186934-47 2024-09-13T14:22:01.009 INFO:tasks.rados.rados.0.smithi151.stdout:316: finishing write tid 4 to smithi15186934-47 2024-09-13T14:22:01.009 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 147 (ObjNum 155 snap 38 seq_num 155) dirty exists 2024-09-13T14:22:01.009 INFO:tasks.rados.rados.0.smithi151.stdout:316: left oid 47 (ObjNum 155 snap 38 seq_num 155) 2024-09-13T14:22:01.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:00 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/1542669865' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:01.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:00 smithi184 ceph-mon[33394]: osdmap e139: 8 total, 8 up, 8 in 2024-09-13T14:22:01.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:00 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1542669865' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:01.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:00 smithi151 ceph-mon[29807]: osdmap e139: 8 total, 8 up, 8 in 2024-09-13T14:22:01.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:00 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/1542669865' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:01.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:00 smithi151 ceph-mon[41387]: osdmap e139: 8 total, 8 up, 8 in 2024-09-13T14:22:01.792 INFO:tasks.rados.rados.0.smithi151.stdout:312: done (2 left) 2024-09-13T14:22:01.792 INFO:tasks.rados.rados.0.smithi151.stdout:316: done (1 left) 2024-09-13T14:22:01.792 INFO:tasks.rados.rados.0.smithi151.stdout:317: done (0 left) 2024-09-13T14:22:01.792 INFO:tasks.rados.rados.0.smithi151.stdout:318: write oid 7 current snap is 39 2024-09-13T14:22:01.792 INFO:tasks.rados.rados.0.smithi151.stdout:318: seq_num 156 ranges {621892=465904,1712485=671711} 2024-09-13T14:22:01.798 INFO:tasks.rados.rados.0.smithi151.stdout:318: writing smithi15186934-7 from 621892 to 1087796 tid 1 2024-09-13T14:22:01.805 INFO:tasks.rados.rados.0.smithi151.stdout:318: writing smithi15186934-7 from 1712485 to 2384196 tid 2 2024-09-13T14:22:01.805 INFO:tasks.rados.rados.0.smithi151.stdout:319: rollback oid 50 current snap is 39 2024-09-13T14:22:01.806 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 50 to 36 2024-09-13T14:22:01.806 INFO:tasks.rados.rados.0.smithi151.stdout:318: finishing write tid 1 to smithi15186934-7 2024-09-13T14:22:01.806 INFO:tasks.rados.rados.0.smithi151.stdout:320: read oid 36 snap -1 2024-09-13T14:22:01.806 INFO:tasks.rados.rados.0.smithi151.stdout:320: expect deleted 2024-09-13T14:22:01.806 INFO:tasks.rados.rados.0.smithi151.stdout:321: rollback oid 44 current snap is 39 2024-09-13T14:22:01.806 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 44 to 37 2024-09-13T14:22:01.806 INFO:tasks.rados.rados.0.smithi151.stdout:322: write oid 17 current snap is 39 2024-09-13T14:22:01.806 INFO:tasks.rados.rados.0.smithi151.stdout:322: seq_num 157 ranges {638699=541153,1770967=531722} 2024-09-13T14:22:01.812 INFO:tasks.rados.rados.0.smithi151.stdout:322: writing smithi15186934-17 from 638699 to 1179852 tid 1 2024-09-13T14:22:01.818 INFO:tasks.rados.rados.0.smithi151.stdout:322: writing smithi15186934-17 from 1770967 to 2302689 tid 2 2024-09-13T14:22:01.818 INFO:tasks.rados.rados.0.smithi151.stdout:323: snap_create 2024-09-13T14:22:01.818 INFO:tasks.rados.rados.0.smithi151.stdout:321: finishing rollback tid 1 to smithi15186934-44 2024-09-13T14:22:01.819 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 56 (ObjNum 129 snap 0 seq_num 3093055560) dirty dne 2024-09-13T14:22:01.819 INFO:tasks.rados.rados.0.smithi151.stdout:318: finishing write tid 2 to smithi15186934-7 2024-09-13T14:22:01.819 INFO:tasks.rados.rados.0.smithi151.stdout:318: finishing write tid 3 to smithi15186934-7 2024-09-13T14:22:01.819 INFO:tasks.rados.rados.0.smithi151.stdout:318: finishing write tid 4 to smithi15186934-7 2024-09-13T14:22:01.819 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 139 (ObjNum 156 snap 39 seq_num 156) dirty exists 2024-09-13T14:22:01.819 INFO:tasks.rados.rados.0.smithi151.stdout:318: left oid 7 (ObjNum 156 snap 39 seq_num 156) 2024-09-13T14:22:01.819 INFO:tasks.rados.rados.0.smithi151.stdout:319: finishing rollback tid 0 to smithi15186934-50 2024-09-13T14:22:01.819 INFO:tasks.rados.rados.0.smithi151.stdout:319: finishing rollback tid 1 to smithi15186934-50 2024-09-13T14:22:01.819 INFO:tasks.rados.rados.0.smithi151.stdout:319: finishing rollback tid 2 to smithi15186934-50 2024-09-13T14:22:01.819 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 50 v 91 (ObjNum 94 snap 16 seq_num 94) dirty exists 2024-09-13T14:22:01.819 INFO:tasks.rados.rados.0.smithi151.stdout:322: finishing write tid 1 to smithi15186934-17 2024-09-13T14:22:01.822 INFO:tasks.rados.rados.0.smithi151.stdout:322: finishing write tid 2 to smithi15186934-17 2024-09-13T14:22:01.822 INFO:tasks.rados.rados.0.smithi151.stdout:322: finishing write tid 3 to smithi15186934-17 2024-09-13T14:22:01.822 INFO:tasks.rados.rados.0.smithi151.stdout:322: finishing write tid 4 to smithi15186934-17 2024-09-13T14:22:01.822 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 143 (ObjNum 157 snap 39 seq_num 157) dirty exists 2024-09-13T14:22:01.822 INFO:tasks.rados.rados.0.smithi151.stdout:322: left oid 17 (ObjNum 157 snap 39 seq_num 157) 2024-09-13T14:22:02.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:01 smithi184 ceph-mon[33394]: pgmap v195: 89 pgs: 89 active+clean; 149 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.3 MiB/s rd, 929 KiB/s wr, 10 op/s 2024-09-13T14:22:02.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:01 smithi184 ceph-mon[33394]: from='client.15237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:02.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:01 smithi184 ceph-mon[33394]: osdmap e140: 8 total, 8 up, 8 in 2024-09-13T14:22:02.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:01 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:02.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:01 smithi151 ceph-mon[29807]: pgmap v195: 89 pgs: 89 active+clean; 149 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.3 MiB/s rd, 929 KiB/s wr, 10 op/s 2024-09-13T14:22:02.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:01 smithi151 ceph-mon[29807]: from='client.15237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:02.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:01 smithi151 ceph-mon[29807]: osdmap e140: 8 total, 8 up, 8 in 2024-09-13T14:22:02.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:01 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:02.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:01 smithi151 ceph-mon[41387]: pgmap v195: 89 pgs: 89 active+clean; 149 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.3 MiB/s rd, 929 KiB/s wr, 10 op/s 2024-09-13T14:22:02.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:01 smithi151 ceph-mon[41387]: from='client.15237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:02.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:01 smithi151 ceph-mon[41387]: osdmap e140: 8 total, 8 up, 8 in 2024-09-13T14:22:02.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:01 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:02.787 INFO:tasks.rados.rados.0.smithi151.stdout:318: done (5 left) 2024-09-13T14:22:02.787 INFO:tasks.rados.rados.0.smithi151.stdout:319: done (4 left) 2024-09-13T14:22:02.787 INFO:tasks.rados.rados.0.smithi151.stdout:320: done (3 left) 2024-09-13T14:22:02.787 INFO:tasks.rados.rados.0.smithi151.stdout:321: done (2 left) 2024-09-13T14:22:02.787 INFO:tasks.rados.rados.0.smithi151.stdout:322: done (1 left) 2024-09-13T14:22:02.788 INFO:tasks.rados.rados.0.smithi151.stdout:323: done (0 left) 2024-09-13T14:22:02.788 INFO:tasks.rados.rados.0.smithi151.stdout:324: write oid 40 current snap is 40 2024-09-13T14:22:02.788 INFO:tasks.rados.rados.0.smithi151.stdout:324: seq_num 158 ranges {655506=616402,1829449=675380} 2024-09-13T14:22:02.793 INFO:tasks.rados.rados.0.smithi151.stdout:324: writing smithi15186934-40 from 655506 to 1271908 tid 1 2024-09-13T14:22:02.801 INFO:tasks.rados.rados.0.smithi151.stdout:324: writing smithi15186934-40 from 1829449 to 2504829 tid 2 2024-09-13T14:22:02.801 INFO:tasks.rados.rados.0.smithi151.stdout:325: snap_create 2024-09-13T14:22:02.801 INFO:tasks.rados.rados.0.smithi151.stdout:324: finishing write tid 1 to smithi15186934-40 2024-09-13T14:22:02.804 INFO:tasks.rados.rados.0.smithi151.stdout:324: finishing write tid 2 to smithi15186934-40 2024-09-13T14:22:02.804 INFO:tasks.rados.rados.0.smithi151.stdout:324: finishing write tid 3 to smithi15186934-40 2024-09-13T14:22:02.805 INFO:tasks.rados.rados.0.smithi151.stdout:324: finishing write tid 4 to smithi15186934-40 2024-09-13T14:22:02.805 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 40 v 114 (ObjNum 158 snap 40 seq_num 158) dirty exists 2024-09-13T14:22:02.805 INFO:tasks.rados.rados.0.smithi151.stdout:324: left oid 40 (ObjNum 158 snap 40 seq_num 158) 2024-09-13T14:22:03.083 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:02 smithi184 ceph-mon[33394]: from='client.25145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:03.083 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:02 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:03.083 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:02 smithi184 ceph-mon[33394]: osdmap e141: 8 total, 8 up, 8 in 2024-09-13T14:22:03.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:02 smithi151 ceph-mon[41387]: from='client.25145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:03.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:02 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:03.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:02 smithi151 ceph-mon[41387]: osdmap e141: 8 total, 8 up, 8 in 2024-09-13T14:22:03.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:02 smithi151 ceph-mon[29807]: from='client.25145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:03.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:02 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:03.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:02 smithi151 ceph-mon[29807]: osdmap e141: 8 total, 8 up, 8 in 2024-09-13T14:22:03.109 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:22:02] "GET /metrics HTTP/1.1" 200 243277 "" "Prometheus/2.43.0" 2024-09-13T14:22:03.544 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:22:03.788 INFO:tasks.rados.rados.0.smithi151.stdout:324: done (1 left) 2024-09-13T14:22:03.788 INFO:tasks.rados.rados.0.smithi151.stdout:325: done (0 left) 2024-09-13T14:22:03.788 INFO:tasks.rados.rados.0.smithi151.stdout:326: write oid 33 current snap is 41 2024-09-13T14:22:03.789 INFO:tasks.rados.rados.0.smithi151.stdout:326: seq_num 159 ranges {672313=691651,1887931=419038} 2024-09-13T14:22:03.795 INFO:tasks.rados.rados.0.smithi151.stdout:326: writing smithi15186934-33 from 672313 to 1363964 tid 1 2024-09-13T14:22:03.801 INFO:tasks.rados.rados.0.smithi151.stdout:326: writing smithi15186934-33 from 1887931 to 2306969 tid 2 2024-09-13T14:22:03.802 INFO:tasks.rados.rados.0.smithi151.stdout:327: read oid 18 snap 23 2024-09-13T14:22:03.802 INFO:tasks.rados.rados.0.smithi151.stdout:327: expect (ObjNum 118 snap 20 seq_num 118) 2024-09-13T14:22:03.802 INFO:tasks.rados.rados.0.smithi151.stdout:328: read oid 48 snap 40 2024-09-13T14:22:03.802 INFO:tasks.rados.rados.0.smithi151.stdout:328: expect (ObjNum 128 snap 27 seq_num 128) 2024-09-13T14:22:03.802 INFO:tasks.rados.rados.0.smithi151.stdout:329: write oid 49 current snap is 41 2024-09-13T14:22:03.802 INFO:tasks.rados.rados.0.smithi151.stdout:329: seq_num 160 ranges {689120=483253,1946413=679049} 2024-09-13T14:22:03.808 INFO:tasks.rados.rados.0.smithi151.stdout:329: writing smithi15186934-49 from 689120 to 1172373 tid 1 2024-09-13T14:22:03.817 INFO:tasks.rados.rados.0.smithi151.stdout:329: writing smithi15186934-49 from 1946413 to 2625462 tid 2 2024-09-13T14:22:03.817 INFO:tasks.rados.rados.0.smithi151.stdout:330: snap_create 2024-09-13T14:22:03.817 INFO:tasks.rados.rados.0.smithi151.stdout:326: finishing write tid 1 to smithi15186934-33 2024-09-13T14:22:03.817 INFO:tasks.rados.rados.0.smithi151.stdout:326: finishing write tid 2 to smithi15186934-33 2024-09-13T14:22:03.817 INFO:tasks.rados.rados.0.smithi151.stdout:326: finishing write tid 3 to smithi15186934-33 2024-09-13T14:22:03.817 INFO:tasks.rados.rados.0.smithi151.stdout:326: finishing write tid 4 to smithi15186934-33 2024-09-13T14:22:03.817 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 33 v 53 (ObjNum 159 snap 41 seq_num 159) dirty exists 2024-09-13T14:22:03.817 INFO:tasks.rados.rados.0.smithi151.stdout:326: left oid 33 (ObjNum 159 snap 41 seq_num 159) 2024-09-13T14:22:03.817 INFO:tasks.rados.rados.0.smithi151.stdout:328: expect (ObjNum 128 snap 27 seq_num 128) 2024-09-13T14:22:03.943 INFO:tasks.rados.rados.0.smithi151.stdout:327: expect (ObjNum 118 snap 20 seq_num 118) 2024-09-13T14:22:04.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:03 smithi184 ceph-mon[33394]: pgmap v198: 89 pgs: 89 active+clean; 149 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.3 MiB/s rd, 929 KiB/s wr, 10 op/s 2024-09-13T14:22:04.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:03 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/4236573796' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:04.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:03 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:22:04.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:03 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:04.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:03 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:04.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:03 smithi184 ceph-mon[33394]: osdmap e142: 8 total, 8 up, 8 in 2024-09-13T14:22:04.104 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:03 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: [13/Sep/2024:14:22:03] ENGINE Bus STARTING 2024-09-13T14:22:04.104 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:03 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: CherryPy Checker: 2024-09-13T14:22:04.104 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:03 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: The Application mounted at '' has an empty config. 2024-09-13T14:22:04.104 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:03 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: 2024-09-13T14:22:04.104 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:04 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: [13/Sep/2024:14:22:04] ENGINE Serving on http://:::9283 2024-09-13T14:22:04.104 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:04 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: [13/Sep/2024:14:22:04] ENGINE Bus STARTED 2024-09-13T14:22:04.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:03 smithi151 ceph-mon[41387]: pgmap v198: 89 pgs: 89 active+clean; 149 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.3 MiB/s rd, 929 KiB/s wr, 10 op/s 2024-09-13T14:22:04.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:03 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/4236573796' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:04.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:03 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:22:04.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:03 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:04.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:03 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:04.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:03 smithi151 ceph-mon[41387]: osdmap e142: 8 total, 8 up, 8 in 2024-09-13T14:22:04.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:03 smithi151 ceph-mon[29807]: pgmap v198: 89 pgs: 89 active+clean; 149 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.3 MiB/s rd, 929 KiB/s wr, 10 op/s 2024-09-13T14:22:04.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:03 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/4236573796' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:04.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:03 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:22:04.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:03 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:04.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:03 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:04.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:03 smithi151 ceph-mon[29807]: osdmap e142: 8 total, 8 up, 8 in 2024-09-13T14:22:04.181 INFO:tasks.rados.rados.0.smithi151.stdout:329: finishing write tid 1 to smithi15186934-49 2024-09-13T14:22:04.181 INFO:tasks.rados.rados.0.smithi151.stdout:329: finishing write tid 2 to smithi15186934-49 2024-09-13T14:22:04.181 INFO:tasks.rados.rados.0.smithi151.stdout:329: finishing write tid 3 to smithi15186934-49 2024-09-13T14:22:04.181 INFO:tasks.rados.rados.0.smithi151.stdout:329: finishing write tid 4 to smithi15186934-49 2024-09-13T14:22:04.181 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 49 v 151 (ObjNum 160 snap 41 seq_num 160) dirty exists 2024-09-13T14:22:04.181 INFO:tasks.rados.rados.0.smithi151.stdout:329: left oid 49 (ObjNum 160 snap 41 seq_num 160) 2024-09-13T14:22:04.791 INFO:tasks.rados.rados.0.smithi151.stdout:326: done (4 left) 2024-09-13T14:22:04.791 INFO:tasks.rados.rados.0.smithi151.stdout:327: done (3 left) 2024-09-13T14:22:04.791 INFO:tasks.rados.rados.0.smithi151.stdout:328: done (2 left) 2024-09-13T14:22:04.791 INFO:tasks.rados.rados.0.smithi151.stdout:329: done (1 left) 2024-09-13T14:22:04.791 INFO:tasks.rados.rados.0.smithi151.stdout:330: done (0 left) 2024-09-13T14:22:04.791 INFO:tasks.rados.rados.0.smithi151.stdout:331: snap_remove snap 39 2024-09-13T14:22:04.872 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:04 smithi184 ceph-mon[33394]: from='client.25157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:04.872 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:04 smithi184 ceph-mon[33394]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:04.872 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:04 smithi184 ceph-mon[33394]: Standby manager daemon x restarted 2024-09-13T14:22:04.872 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:04 smithi184 ceph-mon[33394]: Standby manager daemon x started 2024-09-13T14:22:04.873 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:04 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:22:04.873 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:04 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:22:04.873 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:04 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:22:04.873 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:04 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:22:04.873 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:04 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:04.873 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:04 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:04.873 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:04 smithi184 ceph-mon[33394]: osdmap e143: 8 total, 8 up, 8 in 2024-09-13T14:22:05.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:04 smithi151 ceph-mon[29807]: from='client.25157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:05.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:04 smithi151 ceph-mon[29807]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:05.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:04 smithi151 ceph-mon[29807]: Standby manager daemon x restarted 2024-09-13T14:22:05.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:04 smithi151 ceph-mon[29807]: Standby manager daemon x started 2024-09-13T14:22:05.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:04 smithi151 ceph-mon[29807]: from='mgr.? 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:22:05.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:04 smithi151 ceph-mon[29807]: from='mgr.? 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:22:05.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:04 smithi151 ceph-mon[29807]: from='mgr.? 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:22:05.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:04 smithi151 ceph-mon[29807]: from='mgr.? 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:22:05.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:04 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:05.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:04 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:05.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:04 smithi151 ceph-mon[29807]: osdmap e143: 8 total, 8 up, 8 in 2024-09-13T14:22:05.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:04 smithi151 ceph-mon[41387]: from='client.25157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:05.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:04 smithi151 ceph-mon[41387]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:05.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:04 smithi151 ceph-mon[41387]: Standby manager daemon x restarted 2024-09-13T14:22:05.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:04 smithi151 ceph-mon[41387]: Standby manager daemon x started 2024-09-13T14:22:05.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:04 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:22:05.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:04 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:22:05.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:04 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:22:05.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:04 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:22:05.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:04 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:05.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:04 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:05.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:04 smithi151 ceph-mon[41387]: osdmap e143: 8 total, 8 up, 8 in 2024-09-13T14:22:05.793 INFO:tasks.rados.rados.0.smithi151.stdout:331: done (0 left) 2024-09-13T14:22:05.793 INFO:tasks.rados.rados.0.smithi151.stdout:332: rollback oid 33 current snap is 42 2024-09-13T14:22:05.793 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 33 to 38 2024-09-13T14:22:05.793 INFO:tasks.rados.rados.0.smithi151.stdout:333: snap_remove snap 33 2024-09-13T14:22:06.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:05 smithi184 ceph-mon[33394]: pgmap v201: 89 pgs: 89 active+clean; 157 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.7 MiB/s wr, 12 op/s 2024-09-13T14:22:06.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:05 smithi184 ceph-mon[33394]: mgrmap e21: y(active, since 3m), standbys: x 2024-09-13T14:22:06.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:05 smithi184 ceph-mon[33394]: osdmap e144: 8 total, 8 up, 8 in 2024-09-13T14:22:06.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:05 smithi151 ceph-mon[41387]: pgmap v201: 89 pgs: 89 active+clean; 157 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.7 MiB/s wr, 12 op/s 2024-09-13T14:22:06.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:05 smithi151 ceph-mon[41387]: mgrmap e21: y(active, since 3m), standbys: x 2024-09-13T14:22:06.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:05 smithi151 ceph-mon[41387]: osdmap e144: 8 total, 8 up, 8 in 2024-09-13T14:22:06.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:05 smithi151 ceph-mon[29807]: pgmap v201: 89 pgs: 89 active+clean; 157 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.7 MiB/s wr, 12 op/s 2024-09-13T14:22:06.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:05 smithi151 ceph-mon[29807]: mgrmap e21: y(active, since 3m), standbys: x 2024-09-13T14:22:06.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:05 smithi151 ceph-mon[29807]: osdmap e144: 8 total, 8 up, 8 in 2024-09-13T14:22:06.388 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:22:06.794 INFO:tasks.rados.rados.0.smithi151.stdout:332: finishing rollback tid 0 to smithi15186934-33 2024-09-13T14:22:06.794 INFO:tasks.rados.rados.0.smithi151.stdout:332: finishing rollback tid 1 to smithi15186934-33 2024-09-13T14:22:06.794 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 33 v 56 (ObjNum 129 snap 0 seq_num 1664845238) dirty dne 2024-09-13T14:22:06.795 INFO:tasks.rados.rados.0.smithi151.stdout:332: done (1 left) 2024-09-13T14:22:06.795 INFO:tasks.rados.rados.0.smithi151.stdout:333: done (0 left) 2024-09-13T14:22:06.795 INFO:tasks.rados.rados.0.smithi151.stdout:334: rollback oid 38 current snap is 42 2024-09-13T14:22:06.795 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 38 to 37 2024-09-13T14:22:06.795 INFO:tasks.rados.rados.0.smithi151.stdout:335: write oid 48 current snap is 42 2024-09-13T14:22:06.795 INFO:tasks.rados.rados.0.smithi151.stdout:335: seq_num 161 ranges {705927=558502,2004895=422707} 2024-09-13T14:22:06.802 INFO:tasks.rados.rados.0.smithi151.stdout:335: writing smithi15186934-48 from 705927 to 1264429 tid 1 2024-09-13T14:22:06.809 INFO:tasks.rados.rados.0.smithi151.stdout:335: writing smithi15186934-48 from 2004895 to 2427602 tid 2 2024-09-13T14:22:06.809 INFO:tasks.rados.rados.0.smithi151.stdout:336: snap_create 2024-09-13T14:22:06.809 INFO:tasks.rados.rados.0.smithi151.stdout:334: finishing rollback tid 1 to smithi15186934-38 2024-09-13T14:22:06.809 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 76 (ObjNum 129 snap 0 seq_num 3093055560) dirty dne 2024-09-13T14:22:06.809 INFO:tasks.rados.rados.0.smithi151.stdout:335: finishing write tid 1 to smithi15186934-48 2024-09-13T14:22:06.811 INFO:tasks.rados.rados.0.smithi151.stdout:335: finishing write tid 2 to smithi15186934-48 2024-09-13T14:22:06.811 INFO:tasks.rados.rados.0.smithi151.stdout:335: finishing write tid 3 to smithi15186934-48 2024-09-13T14:22:06.811 INFO:tasks.rados.rados.0.smithi151.stdout:335: finishing write tid 4 to smithi15186934-48 2024-09-13T14:22:06.811 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 48 v 89 (ObjNum 161 snap 42 seq_num 161) dirty exists 2024-09-13T14:22:06.811 INFO:tasks.rados.rados.0.smithi151.stdout:335: left oid 48 (ObjNum 161 snap 42 seq_num 161) 2024-09-13T14:22:07.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:06 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/2961045574' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:07.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:06 smithi184 ceph-mon[33394]: osdmap e145: 8 total, 8 up, 8 in 2024-09-13T14:22:07.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:06 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/2961045574' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:07.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:06 smithi151 ceph-mon[29807]: osdmap e145: 8 total, 8 up, 8 in 2024-09-13T14:22:07.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:06 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/2961045574' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:07.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:06 smithi151 ceph-mon[41387]: osdmap e145: 8 total, 8 up, 8 in 2024-09-13T14:22:07.791 INFO:tasks.rados.rados.0.smithi151.stdout:334: done (2 left) 2024-09-13T14:22:07.791 INFO:tasks.rados.rados.0.smithi151.stdout:335: done (1 left) 2024-09-13T14:22:07.791 INFO:tasks.rados.rados.0.smithi151.stdout:336: done (0 left) 2024-09-13T14:22:07.791 INFO:tasks.rados.rados.0.smithi151.stdout:337: read oid 36 snap -1 2024-09-13T14:22:07.791 INFO:tasks.rados.rados.0.smithi151.stdout:337: expect deleted 2024-09-13T14:22:07.791 INFO:tasks.rados.rados.0.smithi151.stdout:338: delete oid 35 current snap is 43 2024-09-13T14:22:07.793 INFO:tasks.rados.rados.0.smithi151.stdout:337: done (1 left) 2024-09-13T14:22:07.793 INFO:tasks.rados.rados.0.smithi151.stdout:338: done (0 left) 2024-09-13T14:22:07.793 INFO:tasks.rados.rados.0.smithi151.stdout:339: rollback oid 38 current snap is 43 2024-09-13T14:22:07.793 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 38 to 35 2024-09-13T14:22:07.793 INFO:tasks.rados.rados.0.smithi151.stdout:340: write oid 36 current snap is 43 2024-09-13T14:22:07.794 INFO:tasks.rados.rados.0.smithi151.stdout:340: seq_num 163 ranges {739541=709000,2121859=426376} 2024-09-13T14:22:07.801 INFO:tasks.rados.rados.0.smithi151.stdout:340: writing smithi15186934-36 from 739541 to 1448541 tid 1 2024-09-13T14:22:07.808 INFO:tasks.rados.rados.0.smithi151.stdout:340: writing smithi15186934-36 from 2121859 to 2548235 tid 2 2024-09-13T14:22:07.808 INFO:tasks.rados.rados.0.smithi151.stdout:341: rollback oid 14 current snap is 43 2024-09-13T14:22:07.808 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 14 to 40 2024-09-13T14:22:07.808 INFO:tasks.rados.rados.0.smithi151.stdout:339: finishing rollback tid 1 to smithi15186934-38 2024-09-13T14:22:07.808 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 76 (ObjNum -1201950576 snap 22067 seq_num 304) dirty dne 2024-09-13T14:22:07.808 INFO:tasks.rados.rados.0.smithi151.stdout:340: finishing write tid 1 to smithi15186934-36 2024-09-13T14:22:07.808 INFO:tasks.rados.rados.0.smithi151.stdout:339: done (2 left) 2024-09-13T14:22:07.808 INFO:tasks.rados.rados.0.smithi151.stdout:342: delete oid 25 current snap is 43 2024-09-13T14:22:07.810 INFO:tasks.rados.rados.0.smithi151.stdout:340: finishing write tid 2 to smithi15186934-36 2024-09-13T14:22:07.810 INFO:tasks.rados.rados.0.smithi151.stdout:340: finishing write tid 3 to smithi15186934-36 2024-09-13T14:22:07.811 INFO:tasks.rados.rados.0.smithi151.stdout:340: finishing write tid 4 to smithi15186934-36 2024-09-13T14:22:07.811 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 36 v 50 (ObjNum 163 snap 43 seq_num 163) dirty exists 2024-09-13T14:22:07.811 INFO:tasks.rados.rados.0.smithi151.stdout:340: left oid 36 (ObjNum 163 snap 43 seq_num 163) 2024-09-13T14:22:07.811 INFO:tasks.rados.rados.0.smithi151.stdout:341: finishing rollback tid 0 to smithi15186934-14 2024-09-13T14:22:07.811 INFO:tasks.rados.rados.0.smithi151.stdout:341: finishing rollback tid 1 to smithi15186934-14 2024-09-13T14:22:07.811 INFO:tasks.rados.rados.0.smithi151.stdout:341: finishing rollback tid 2 to smithi15186934-14 2024-09-13T14:22:07.811 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 127 (ObjNum 75 snap 9 seq_num 75) dirty exists 2024-09-13T14:22:07.812 INFO:tasks.rados.rados.0.smithi151.stdout:340: done (2 left) 2024-09-13T14:22:07.812 INFO:tasks.rados.rados.0.smithi151.stdout:341: done (1 left) 2024-09-13T14:22:07.812 INFO:tasks.rados.rados.0.smithi151.stdout:342: done (0 left) 2024-09-13T14:22:07.812 INFO:tasks.rados.rados.0.smithi151.stdout:343: delete oid 3 current snap is 43 2024-09-13T14:22:07.813 INFO:tasks.rados.rados.0.smithi151.stdout:343: done (0 left) 2024-09-13T14:22:07.813 INFO:tasks.rados.rados.0.smithi151.stdout:344: write oid 7 current snap is 43 2024-09-13T14:22:07.813 INFO:tasks.rados.rados.0.smithi151.stdout:344: seq_num 166 ranges {789962=534747,1780952=690056} 2024-09-13T14:22:07.820 INFO:tasks.rados.rados.0.smithi151.stdout:344: writing smithi15186934-7 from 789962 to 1324709 tid 1 2024-09-13T14:22:07.827 INFO:tasks.rados.rados.0.smithi151.stdout:344: writing smithi15186934-7 from 1780952 to 2471008 tid 2 2024-09-13T14:22:07.827 INFO:tasks.rados.rados.0.smithi151.stdout:345: read oid 31 snap -1 2024-09-13T14:22:07.827 INFO:tasks.rados.rados.0.smithi151.stdout:345: expect (ObjNum 138 snap 30 seq_num 138) 2024-09-13T14:22:07.827 INFO:tasks.rados.rados.0.smithi151.stdout:346: write oid 12 current snap is 43 2024-09-13T14:22:07.827 INFO:tasks.rados.rados.0.smithi151.stdout:344: finishing write tid 1 to smithi15186934-7 2024-09-13T14:22:07.827 INFO:tasks.rados.rados.0.smithi151.stdout:346: seq_num 167 ranges {406769=609996,1439434=433714,2303522=503247} 2024-09-13T14:22:07.832 INFO:tasks.rados.rados.0.smithi151.stdout:346: writing smithi15186934-12 from 406769 to 1016765 tid 1 2024-09-13T14:22:07.837 INFO:tasks.rados.rados.0.smithi151.stdout:346: writing smithi15186934-12 from 1439434 to 1873148 tid 2 2024-09-13T14:22:07.842 INFO:tasks.rados.rados.0.smithi151.stdout:346: writing smithi15186934-12 from 2303522 to 2806769 tid 3 2024-09-13T14:22:07.843 INFO:tasks.rados.rados.0.smithi151.stdout:347: rollback oid 36 current snap is 43 2024-09-13T14:22:07.843 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 36 to 40 2024-09-13T14:22:07.843 INFO:tasks.rados.rados.0.smithi151.stdout:344: finishing write tid 2 to smithi15186934-7 2024-09-13T14:22:07.843 INFO:tasks.rados.rados.0.smithi151.stdout:344: finishing write tid 3 to smithi15186934-7 2024-09-13T14:22:07.843 INFO:tasks.rados.rados.0.smithi151.stdout:344: finishing write tid 4 to smithi15186934-7 2024-09-13T14:22:07.843 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 153 (ObjNum 166 snap 43 seq_num 166) dirty exists 2024-09-13T14:22:07.843 INFO:tasks.rados.rados.0.smithi151.stdout:344: left oid 7 (ObjNum 166 snap 43 seq_num 166) 2024-09-13T14:22:07.843 INFO:tasks.rados.rados.0.smithi151.stdout:345: expect (ObjNum 138 snap 30 seq_num 138) 2024-09-13T14:22:07.844 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:07 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:22:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:22:07.944 INFO:tasks.rados.rados.0.smithi151.stdout:346: finishing write tid 1 to smithi15186934-12 2024-09-13T14:22:07.944 INFO:tasks.rados.rados.0.smithi151.stdout:346: finishing write tid 2 to smithi15186934-12 2024-09-13T14:22:07.944 INFO:tasks.rados.rados.0.smithi151.stdout:346: finishing write tid 3 to smithi15186934-12 2024-09-13T14:22:07.944 INFO:tasks.rados.rados.0.smithi151.stdout:346: finishing write tid 4 to smithi15186934-12 2024-09-13T14:22:07.945 INFO:tasks.rados.rados.0.smithi151.stdout:346: finishing write tid 5 to smithi15186934-12 2024-09-13T14:22:07.945 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 12 v 55 (ObjNum 167 snap 43 seq_num 167) dirty exists 2024-09-13T14:22:07.945 INFO:tasks.rados.rados.0.smithi151.stdout:346: left oid 12 (ObjNum 167 snap 43 seq_num 167) 2024-09-13T14:22:07.945 INFO:tasks.rados.rados.0.smithi151.stdout:347: finishing rollback tid 0 to smithi15186934-36 2024-09-13T14:22:07.945 INFO:tasks.rados.rados.0.smithi151.stdout:347: finishing rollback tid 1 to smithi15186934-36 2024-09-13T14:22:07.945 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 36 v 57 (ObjNum 81 snap 0 seq_num 0) dirty dne 2024-09-13T14:22:07.945 INFO:tasks.rados.rados.0.smithi151.stdout:344: done (3 left) 2024-09-13T14:22:07.945 INFO:tasks.rados.rados.0.smithi151.stdout:345: done (2 left) 2024-09-13T14:22:07.945 INFO:tasks.rados.rados.0.smithi151.stdout:346: done (1 left) 2024-09-13T14:22:07.945 INFO:tasks.rados.rados.0.smithi151.stdout:347: done (0 left) 2024-09-13T14:22:07.945 INFO:tasks.rados.rados.0.smithi151.stdout:348: snap_create 2024-09-13T14:22:08.094 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:07 smithi184 ceph-mon[33394]: from='client.25175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:08.094 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:07 smithi184 ceph-mon[33394]: pgmap v204: 89 pgs: 89 active+clean; 157 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.7 MiB/s wr, 12 op/s 2024-09-13T14:22:08.094 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:07 smithi184 ceph-mon[33394]: from='client.15279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:08.095 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:07 smithi184 ceph-mon[33394]: osdmap e146: 8 total, 8 up, 8 in 2024-09-13T14:22:08.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:07 smithi151 ceph-mon[29807]: from='client.25175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:08.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:07 smithi151 ceph-mon[29807]: pgmap v204: 89 pgs: 89 active+clean; 157 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.7 MiB/s wr, 12 op/s 2024-09-13T14:22:08.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:07 smithi151 ceph-mon[29807]: from='client.15279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:08.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:07 smithi151 ceph-mon[29807]: osdmap e146: 8 total, 8 up, 8 in 2024-09-13T14:22:08.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:07 smithi151 ceph-mon[41387]: from='client.25175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:08.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:07 smithi151 ceph-mon[41387]: pgmap v204: 89 pgs: 89 active+clean; 157 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.7 MiB/s wr, 12 op/s 2024-09-13T14:22:08.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:07 smithi151 ceph-mon[41387]: from='client.15279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:08.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:07 smithi151 ceph-mon[41387]: osdmap e146: 8 total, 8 up, 8 in 2024-09-13T14:22:08.855 INFO:tasks.rados.rados.0.smithi151.stdout:348: done (0 left) 2024-09-13T14:22:08.855 INFO:tasks.rados.rados.0.smithi151.stdout:349: write oid 13 current snap is 44 2024-09-13T14:22:08.855 INFO:tasks.rados.rados.0.smithi151.stdout:349: seq_num 168 ranges {423576=401598,1497916=693725} 2024-09-13T14:22:08.861 INFO:tasks.rados.rados.0.smithi151.stdout:349: writing smithi15186934-13 from 423576 to 825174 tid 1 2024-09-13T14:22:08.870 INFO:tasks.rados.rados.0.smithi151.stdout:349: writing smithi15186934-13 from 1497916 to 2191641 tid 2 2024-09-13T14:22:08.870 INFO:tasks.rados.rados.0.smithi151.stdout:350: write oid 44 current snap is 44 2024-09-13T14:22:08.870 INFO:tasks.rados.rados.0.smithi151.stdout:350: seq_num 169 ranges {440383=476847,1556398=437383,2474721=365662} 2024-09-13T14:22:08.876 INFO:tasks.rados.rados.0.smithi151.stdout:350: writing smithi15186934-44 from 440383 to 917230 tid 1 2024-09-13T14:22:08.883 INFO:tasks.rados.rados.0.smithi151.stdout:350: writing smithi15186934-44 from 1556398 to 1993781 tid 2 2024-09-13T14:22:08.889 INFO:tasks.rados.rados.0.smithi151.stdout:350: writing smithi15186934-44 from 2474721 to 2840383 tid 3 2024-09-13T14:22:08.889 INFO:tasks.rados.rados.0.smithi151.stdout:351: write oid 21 current snap is 44 2024-09-13T14:22:08.889 INFO:tasks.rados.rados.0.smithi151.stdout:349: finishing write tid 1 to smithi15186934-13 2024-09-13T14:22:08.889 INFO:tasks.rados.rados.0.smithi151.stdout:349: finishing write tid 2 to smithi15186934-13 2024-09-13T14:22:08.889 INFO:tasks.rados.rados.0.smithi151.stdout:349: finishing write tid 3 to smithi15186934-13 2024-09-13T14:22:08.889 INFO:tasks.rados.rados.0.smithi151.stdout:349: finishing write tid 4 to smithi15186934-13 2024-09-13T14:22:08.890 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 13 v 60 (ObjNum 168 snap 44 seq_num 168) dirty exists 2024-09-13T14:22:08.890 INFO:tasks.rados.rados.0.smithi151.stdout:349: left oid 13 (ObjNum 168 snap 44 seq_num 168) 2024-09-13T14:22:08.890 INFO:tasks.rados.rados.0.smithi151.stdout:350: finishing write tid 1 to smithi15186934-44 2024-09-13T14:22:08.890 INFO:tasks.rados.rados.0.smithi151.stdout:351: seq_num 170 ranges {457190=552096,1614880=581041,2785791=71399} 2024-09-13T14:22:08.896 INFO:tasks.rados.rados.0.smithi151.stdout:351: writing smithi15186934-21 from 457190 to 1009286 tid 1 2024-09-13T14:22:08.903 INFO:tasks.rados.rados.0.smithi151.stdout:351: writing smithi15186934-21 from 1614880 to 2195921 tid 2 2024-09-13T14:22:08.907 INFO:tasks.rados.rados.0.smithi151.stdout:351: writing smithi15186934-21 from 2785791 to 2857190 tid 3 2024-09-13T14:22:08.908 INFO:tasks.rados.rados.0.smithi151.stdout:349: done (2 left) 2024-09-13T14:22:08.908 INFO:tasks.rados.rados.0.smithi151.stdout:352: write oid 7 current snap is 44 2024-09-13T14:22:08.908 INFO:tasks.rados.rados.0.smithi151.stdout:350: finishing write tid 2 to smithi15186934-44 2024-09-13T14:22:08.908 INFO:tasks.rados.rados.0.smithi151.stdout:350: finishing write tid 3 to smithi15186934-44 2024-09-13T14:22:08.908 INFO:tasks.rados.rados.0.smithi151.stdout:350: finishing write tid 4 to smithi15186934-44 2024-09-13T14:22:08.908 INFO:tasks.rados.rados.0.smithi151.stdout:350: finishing write tid 5 to smithi15186934-44 2024-09-13T14:22:08.908 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 106 (ObjNum 169 snap 44 seq_num 169) dirty exists 2024-09-13T14:22:08.908 INFO:tasks.rados.rados.0.smithi151.stdout:350: left oid 44 (ObjNum 169 snap 44 seq_num 169) 2024-09-13T14:22:08.908 INFO:tasks.rados.rados.0.smithi151.stdout:351: finishing write tid 1 to smithi15186934-21 2024-09-13T14:22:08.908 INFO:tasks.rados.rados.0.smithi151.stdout:352: seq_num 171 ranges {473997=627345,1673362=441052,2529567=344430} 2024-09-13T14:22:08.913 INFO:tasks.rados.rados.0.smithi151.stdout:352: writing smithi15186934-7 from 473997 to 1101342 tid 1 2024-09-13T14:22:08.919 INFO:tasks.rados.rados.0.smithi151.stdout:352: writing smithi15186934-7 from 1673362 to 2114414 tid 2 2024-09-13T14:22:08.924 INFO:tasks.rados.rados.0.smithi151.stdout:352: writing smithi15186934-7 from 2529567 to 2873997 tid 3 2024-09-13T14:22:08.924 INFO:tasks.rados.rados.0.smithi151.stdout:350: done (2 left) 2024-09-13T14:22:08.924 INFO:tasks.rados.rados.0.smithi151.stdout:353: snap_create 2024-09-13T14:22:08.924 INFO:tasks.rados.rados.0.smithi151.stdout:351: finishing write tid 2 to smithi15186934-21 2024-09-13T14:22:08.924 INFO:tasks.rados.rados.0.smithi151.stdout:351: finishing write tid 3 to smithi15186934-21 2024-09-13T14:22:08.924 INFO:tasks.rados.rados.0.smithi151.stdout:351: finishing write tid 4 to smithi15186934-21 2024-09-13T14:22:08.924 INFO:tasks.rados.rados.0.smithi151.stdout:351: finishing write tid 5 to smithi15186934-21 2024-09-13T14:22:08.924 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 21 v 40 (ObjNum 170 snap 44 seq_num 170) dirty exists 2024-09-13T14:22:08.924 INFO:tasks.rados.rados.0.smithi151.stdout:351: left oid 21 (ObjNum 170 snap 44 seq_num 170) 2024-09-13T14:22:08.925 INFO:tasks.rados.rados.0.smithi151.stdout:352: finishing write tid 1 to smithi15186934-7 2024-09-13T14:22:08.925 INFO:tasks.rados.rados.0.smithi151.stdout:352: finishing write tid 2 to smithi15186934-7 2024-09-13T14:22:08.925 INFO:tasks.rados.rados.0.smithi151.stdout:352: finishing write tid 3 to smithi15186934-7 2024-09-13T14:22:08.925 INFO:tasks.rados.rados.0.smithi151.stdout:352: finishing write tid 4 to smithi15186934-7 2024-09-13T14:22:08.925 INFO:tasks.rados.rados.0.smithi151.stdout:352: finishing write tid 5 to smithi15186934-7 2024-09-13T14:22:08.926 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 158 (ObjNum 171 snap 44 seq_num 171) dirty exists 2024-09-13T14:22:08.926 INFO:tasks.rados.rados.0.smithi151.stdout:352: left oid 7 (ObjNum 171 snap 44 seq_num 171) 2024-09-13T14:22:09.237 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:22:09.859 INFO:tasks.rados.rados.0.smithi151.stdout:351: done (2 left) 2024-09-13T14:22:09.860 INFO:tasks.rados.rados.0.smithi151.stdout:352: done (1 left) 2024-09-13T14:22:09.860 INFO:tasks.rados.rados.0.smithi151.stdout:353: done (0 left) 2024-09-13T14:22:09.860 INFO:tasks.rados.rados.0.smithi151.stdout:354: rollback oid 19 current snap is 45 2024-09-13T14:22:09.860 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 19 to 38 2024-09-13T14:22:09.860 INFO:tasks.rados.rados.0.smithi151.stdout:355: read oid 36 snap -1 2024-09-13T14:22:09.860 INFO:tasks.rados.rados.0.smithi151.stdout:355: expect deleted 2024-09-13T14:22:09.860 INFO:tasks.rados.rados.0.smithi151.stdout:356: write oid 27 current snap is 45 2024-09-13T14:22:09.860 INFO:tasks.rados.rados.0.smithi151.stdout:356: seq_num 172 ranges {490804=702594,1615491=584710,2724284=166520} 2024-09-13T14:22:09.866 INFO:tasks.rados.rados.0.smithi151.stdout:356: writing smithi15186934-27 from 490804 to 1193398 tid 1 2024-09-13T14:22:09.871 INFO:tasks.rados.rados.0.smithi151.stdout:356: writing smithi15186934-27 from 1615491 to 2200201 tid 2 2024-09-13T14:22:09.876 INFO:tasks.rados.rados.0.smithi151.stdout:356: writing smithi15186934-27 from 2724284 to 2890804 tid 3 2024-09-13T14:22:09.876 INFO:tasks.rados.rados.0.smithi151.stdout:357: snap_create 2024-09-13T14:22:09.876 INFO:tasks.rados.rados.0.smithi151.stdout:354: finishing rollback tid 0 to smithi15186934-19 2024-09-13T14:22:09.876 INFO:tasks.rados.rados.0.smithi151.stdout:354: finishing rollback tid 1 to smithi15186934-19 2024-09-13T14:22:09.876 INFO:tasks.rados.rados.0.smithi151.stdout:354: finishing rollback tid 2 to smithi15186934-19 2024-09-13T14:22:09.876 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 19 v 101 (ObjNum 140 snap 31 seq_num 140) dirty exists 2024-09-13T14:22:09.876 INFO:tasks.rados.rados.0.smithi151.stdout:356: finishing write tid 1 to smithi15186934-27 2024-09-13T14:22:09.876 INFO:tasks.rados.rados.0.smithi151.stdout:356: finishing write tid 2 to smithi15186934-27 2024-09-13T14:22:09.877 INFO:tasks.rados.rados.0.smithi151.stdout:356: finishing write tid 3 to smithi15186934-27 2024-09-13T14:22:09.878 INFO:tasks.rados.rados.0.smithi151.stdout:356: finishing write tid 4 to smithi15186934-27 2024-09-13T14:22:09.878 INFO:tasks.rados.rados.0.smithi151.stdout:356: finishing write tid 5 to smithi15186934-27 2024-09-13T14:22:09.878 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 27 v 79 (ObjNum 172 snap 45 seq_num 172) dirty exists 2024-09-13T14:22:09.878 INFO:tasks.rados.rados.0.smithi151.stdout:356: left oid 27 (ObjNum 172 snap 45 seq_num 172) 2024-09-13T14:22:10.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:09 smithi184 ceph-mon[33394]: pgmap v207: 89 pgs: 89 active+clean; 157 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:10.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:09 smithi184 ceph-mon[33394]: osdmap e147: 8 total, 8 up, 8 in 2024-09-13T14:22:10.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:09 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/2443105626' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:10.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:09 smithi151 ceph-mon[29807]: pgmap v207: 89 pgs: 89 active+clean; 157 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:10.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:09 smithi151 ceph-mon[29807]: osdmap e147: 8 total, 8 up, 8 in 2024-09-13T14:22:10.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:09 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/2443105626' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:10.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:09 smithi151 ceph-mon[41387]: pgmap v207: 89 pgs: 89 active+clean; 157 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:10.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:09 smithi151 ceph-mon[41387]: osdmap e147: 8 total, 8 up, 8 in 2024-09-13T14:22:10.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:09 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/2443105626' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:10.867 INFO:tasks.rados.rados.0.smithi151.stdout:354: done (3 left) 2024-09-13T14:22:10.867 INFO:tasks.rados.rados.0.smithi151.stdout:355: done (2 left) 2024-09-13T14:22:10.867 INFO:tasks.rados.rados.0.smithi151.stdout:356: done (1 left) 2024-09-13T14:22:10.867 INFO:tasks.rados.rados.0.smithi151.stdout:357: done (0 left) 2024-09-13T14:22:10.867 INFO:tasks.rados.rados.0.smithi151.stdout:358: write oid 28 current snap is 46 2024-09-13T14:22:10.867 INFO:tasks.rados.rados.0.smithi151.stdout:358: seq_num 173 ranges {507611=777843,2073973=444721} 2024-09-13T14:22:10.874 INFO:tasks.rados.rados.0.smithi151.stdout:358: writing smithi15186934-28 from 507611 to 1285454 tid 1 2024-09-13T14:22:10.881 INFO:tasks.rados.rados.0.smithi151.stdout:358: writing smithi15186934-28 from 2073973 to 2518694 tid 2 2024-09-13T14:22:10.881 INFO:tasks.rados.rados.0.smithi151.stdout:359: snap_create 2024-09-13T14:22:10.881 INFO:tasks.rados.rados.0.smithi151.stdout:358: finishing write tid 1 to smithi15186934-28 2024-09-13T14:22:10.883 INFO:tasks.rados.rados.0.smithi151.stdout:358: finishing write tid 2 to smithi15186934-28 2024-09-13T14:22:10.884 INFO:tasks.rados.rados.0.smithi151.stdout:358: finishing write tid 3 to smithi15186934-28 2024-09-13T14:22:10.884 INFO:tasks.rados.rados.0.smithi151.stdout:358: finishing write tid 4 to smithi15186934-28 2024-09-13T14:22:10.884 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 28 v 137 (ObjNum 173 snap 46 seq_num 173) dirty exists 2024-09-13T14:22:10.884 INFO:tasks.rados.rados.0.smithi151.stdout:358: left oid 28 (ObjNum 173 snap 46 seq_num 173) 2024-09-13T14:22:11.289 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:10 smithi184 ceph-mon[33394]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:11.289 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:10 smithi184 ceph-mon[33394]: from='client.25199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:11.289 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:10 smithi184 ceph-mon[33394]: osdmap e148: 8 total, 8 up, 8 in 2024-09-13T14:22:11.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:10 smithi151 ceph-mon[41387]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:11.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:10 smithi151 ceph-mon[41387]: from='client.25199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:11.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:10 smithi151 ceph-mon[41387]: osdmap e148: 8 total, 8 up, 8 in 2024-09-13T14:22:11.361 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:10 smithi151 ceph-mon[29807]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:11.361 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:10 smithi151 ceph-mon[29807]: from='client.25199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:11.361 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:10 smithi151 ceph-mon[29807]: osdmap e148: 8 total, 8 up, 8 in 2024-09-13T14:22:11.873 INFO:tasks.rados.rados.0.smithi151.stdout:358: done (1 left) 2024-09-13T14:22:11.873 INFO:tasks.rados.rados.0.smithi151.stdout:359: done (0 left) 2024-09-13T14:22:11.873 INFO:tasks.rados.rados.0.smithi151.stdout:360: snap_remove snap 45 2024-09-13T14:22:12.077 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:22:12.273 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:11 smithi184 ceph-mon[33394]: pgmap v210: 89 pgs: 89 active+clean; 169 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 15 op/s 2024-09-13T14:22:12.273 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:11 smithi184 ceph-mon[33394]: osdmap e149: 8 total, 8 up, 8 in 2024-09-13T14:22:12.274 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:11 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/2661306614' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:12.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:11 smithi151 ceph-mon[29807]: pgmap v210: 89 pgs: 89 active+clean; 169 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 15 op/s 2024-09-13T14:22:12.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:11 smithi151 ceph-mon[29807]: osdmap e149: 8 total, 8 up, 8 in 2024-09-13T14:22:12.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:11 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/2661306614' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:12.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:11 smithi151 ceph-mon[41387]: pgmap v210: 89 pgs: 89 active+clean; 169 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 15 op/s 2024-09-13T14:22:12.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:11 smithi151 ceph-mon[41387]: osdmap e149: 8 total, 8 up, 8 in 2024-09-13T14:22:12.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:11 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/2661306614' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:12.790 INFO:tasks.rados.rados.0.smithi151.stdout:360: done (0 left) 2024-09-13T14:22:12.790 INFO:tasks.rados.rados.0.smithi151.stdout:361: read oid 32 snap -1 2024-09-13T14:22:12.790 INFO:tasks.rados.rados.0.smithi151.stdout:361: expect (ObjNum 96 snap 16 seq_num 96) 2024-09-13T14:22:12.790 INFO:tasks.rados.rados.0.smithi151.stdout:362: read oid 10 snap 43 2024-09-13T14:22:12.790 INFO:tasks.rados.rados.0.smithi151.stdout:362: expect deleted 2024-09-13T14:22:12.790 INFO:tasks.rados.rados.0.smithi151.stdout:363: read oid 26 snap -1 2024-09-13T14:22:12.790 INFO:tasks.rados.rados.0.smithi151.stdout:363: expect (ObjNum 148 snap 34 seq_num 148) 2024-09-13T14:22:12.790 INFO:tasks.rados.rados.0.smithi151.stdout:364: read oid 2 snap -1 2024-09-13T14:22:12.790 INFO:tasks.rados.rados.0.smithi151.stdout:364: expect deleted 2024-09-13T14:22:12.790 INFO:tasks.rados.rados.0.smithi151.stdout:365: delete oid 40 current snap is 47 2024-09-13T14:22:12.797 INFO:tasks.rados.rados.0.smithi151.stdout:362: done (4 left) 2024-09-13T14:22:12.797 INFO:tasks.rados.rados.0.smithi151.stdout:364: done (3 left) 2024-09-13T14:22:12.797 INFO:tasks.rados.rados.0.smithi151.stdout:365: done (2 left) 2024-09-13T14:22:12.797 INFO:tasks.rados.rados.0.smithi151.stdout:366: read oid 36 snap -1 2024-09-13T14:22:12.797 INFO:tasks.rados.rados.0.smithi151.stdout:366: expect deleted 2024-09-13T14:22:12.797 INFO:tasks.rados.rados.0.smithi151.stdout:367: rollback oid 44 current snap is 47 2024-09-13T14:22:12.797 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 44 to 38 2024-09-13T14:22:12.797 INFO:tasks.rados.rados.0.smithi151.stdout:368: rollback oid 18 current snap is 47 2024-09-13T14:22:12.797 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 18 to 43 2024-09-13T14:22:12.797 INFO:tasks.rados.rados.0.smithi151.stdout:369: write oid 45 current snap is 47 2024-09-13T14:22:12.797 INFO:tasks.rados.rados.0.smithi151.stdout:369: seq_num 175 ranges {541225=644694,1907290=448390} 2024-09-13T14:22:12.802 INFO:tasks.rados.rados.0.smithi151.stdout:369: writing smithi15186934-45 from 541225 to 1185919 tid 1 2024-09-13T14:22:12.808 INFO:tasks.rados.rados.0.smithi151.stdout:369: writing smithi15186934-45 from 1907290 to 2355680 tid 2 2024-09-13T14:22:12.809 INFO:tasks.rados.rados.0.smithi151.stdout:370: write oid 33 current snap is 47 2024-09-13T14:22:12.809 INFO:tasks.rados.rados.0.smithi151.stdout:370: seq_num 176 ranges {558032=719943,1849419=592048} 2024-09-13T14:22:12.815 INFO:tasks.rados.rados.0.smithi151.stdout:370: writing smithi15186934-33 from 558032 to 1277975 tid 1 2024-09-13T14:22:12.822 INFO:tasks.rados.rados.0.smithi151.stdout:370: writing smithi15186934-33 from 1849419 to 2441467 tid 2 2024-09-13T14:22:12.822 INFO:tasks.rados.rados.0.smithi151.stdout:371: read oid 21 snap -1 2024-09-13T14:22:12.822 INFO:tasks.rados.rados.0.smithi151.stdout:371: expect (ObjNum 170 snap 44 seq_num 170) 2024-09-13T14:22:12.822 INFO:tasks.rados.rados.0.smithi151.stdout:372: delete oid 41 current snap is 47 2024-09-13T14:22:12.822 INFO:tasks.rados.rados.0.smithi151.stdout:363: expect (ObjNum 148 snap 34 seq_num 148) 2024-09-13T14:22:12.941 INFO:tasks.rados.rados.0.smithi151.stdout:367: finishing rollback tid 0 to smithi15186934-44 2024-09-13T14:22:12.942 INFO:tasks.rados.rados.0.smithi151.stdout:361: expect (ObjNum 96 snap 16 seq_num 96) 2024-09-13T14:22:13.138 INFO:tasks.rados.rados.0.smithi151.stdout:368: finishing rollback tid 0 to smithi15186934-18 2024-09-13T14:22:13.138 INFO:tasks.rados.rados.0.smithi151.stdout:368: finishing rollback tid 1 to smithi15186934-18 2024-09-13T14:22:13.138 INFO:tasks.rados.rados.0.smithi151.stdout:367: finishing rollback tid 1 to smithi15186934-44 2024-09-13T14:22:13.138 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 109 (ObjNum 49 snap 0 seq_num 3681403705) dirty dne 2024-09-13T14:22:13.138 INFO:tasks.rados.rados.0.smithi151.stdout:368: finishing rollback tid 2 to smithi15186934-18 2024-09-13T14:22:13.138 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 18 v 64 (ObjNum 118 snap 20 seq_num 118) dirty exists 2024-09-13T14:22:13.139 INFO:tasks.rados.rados.0.smithi151.stdout:369: finishing write tid 1 to smithi15186934-45 2024-09-13T14:22:13.139 INFO:tasks.rados.rados.0.smithi151.stdout:361: done (8 left) 2024-09-13T14:22:13.139 INFO:tasks.rados.rados.0.smithi151.stdout:363: done (7 left) 2024-09-13T14:22:13.139 INFO:tasks.rados.rados.0.smithi151.stdout:366: done (6 left) 2024-09-13T14:22:13.139 INFO:tasks.rados.rados.0.smithi151.stdout:367: done (5 left) 2024-09-13T14:22:13.139 INFO:tasks.rados.rados.0.smithi151.stdout:368: done (4 left) 2024-09-13T14:22:13.139 INFO:tasks.rados.rados.0.smithi151.stdout:372: done (3 left) 2024-09-13T14:22:13.139 INFO:tasks.rados.rados.0.smithi151.stdout:373: write oid 26 current snap is 47 2024-09-13T14:22:13.139 INFO:tasks.rados.rados.0.smithi151.stdout:369: finishing write tid 2 to smithi15186934-45 2024-09-13T14:22:13.139 INFO:tasks.rados.rados.0.smithi151.stdout:369: finishing write tid 3 to smithi15186934-45 2024-09-13T14:22:13.139 INFO:tasks.rados.rados.0.smithi151.stdout:373: seq_num 178 ranges {591646=470441,1566383=595717,2888822=102824} 2024-09-13T14:22:13.143 INFO:tasks.rados.rados.0.smithi151.stdout:373: writing smithi15186934-26 from 591646 to 1062087 tid 1 2024-09-13T14:22:13.149 INFO:tasks.rados.rados.0.smithi151.stdout:373: writing smithi15186934-26 from 1566383 to 2162100 tid 2 2024-09-13T14:22:13.154 INFO:tasks.rados.rados.0.smithi151.stdout:373: writing smithi15186934-26 from 2888822 to 2991646 tid 3 2024-09-13T14:22:13.154 INFO:tasks.rados.rados.0.smithi151.stdout:374: write oid 37 current snap is 47 2024-09-13T14:22:13.154 INFO:tasks.rados.rados.0.smithi151.stdout:374: seq_num 179 ranges {608453=545690,1624865=455728,2632598=375855} 2024-09-13T14:22:13.160 INFO:tasks.rados.rados.0.smithi151.stdout:374: writing smithi15186934-37 from 608453 to 1154143 tid 1 2024-09-13T14:22:13.165 INFO:tasks.rados.rados.0.smithi151.stdout:374: writing smithi15186934-37 from 1624865 to 2080593 tid 2 2024-09-13T14:22:13.171 INFO:tasks.rados.rados.0.smithi151.stdout:374: writing smithi15186934-37 from 2632598 to 3008453 tid 3 2024-09-13T14:22:13.171 INFO:tasks.rados.rados.0.smithi151.stdout:375: rollback oid 3 current snap is 47 2024-09-13T14:22:13.171 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 3 to 46 2024-09-13T14:22:13.171 INFO:tasks.rados.rados.0.smithi151.stdout:376: delete oid 23 current snap is 47 2024-09-13T14:22:13.171 INFO:tasks.rados.rados.0.smithi151.stdout:369: finishing write tid 4 to smithi15186934-45 2024-09-13T14:22:13.172 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 95 (ObjNum 175 snap 47 seq_num 175) dirty exists 2024-09-13T14:22:13.172 INFO:tasks.rados.rados.0.smithi151.stdout:369: left oid 45 (ObjNum 175 snap 47 seq_num 175) 2024-09-13T14:22:13.172 INFO:tasks.rados.rados.0.smithi151.stdout:370: finishing write tid 1 to smithi15186934-33 2024-09-13T14:22:13.172 INFO:tasks.rados.rados.0.smithi151.stdout:370: finishing write tid 2 to smithi15186934-33 2024-09-13T14:22:13.172 INFO:tasks.rados.rados.0.smithi151.stdout:370: finishing write tid 3 to smithi15186934-33 2024-09-13T14:22:13.172 INFO:tasks.rados.rados.0.smithi151.stdout:370: finishing write tid 4 to smithi15186934-33 2024-09-13T14:22:13.172 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 33 v 63 (ObjNum 176 snap 47 seq_num 176) dirty exists 2024-09-13T14:22:13.172 INFO:tasks.rados.rados.0.smithi151.stdout:370: left oid 33 (ObjNum 176 snap 47 seq_num 176) 2024-09-13T14:22:13.172 INFO:tasks.rados.rados.0.smithi151.stdout:371: expect (ObjNum 170 snap 44 seq_num 170) 2024-09-13T14:22:13.252 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:12 smithi184 ceph-mon[33394]: osdmap e150: 8 total, 8 up, 8 in 2024-09-13T14:22:13.252 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:12 smithi184 ceph-mon[33394]: osdmap e151: 8 total, 8 up, 8 in 2024-09-13T14:22:13.265 INFO:tasks.rados.rados.0.smithi151.stdout:373: finishing write tid 1 to smithi15186934-26 2024-09-13T14:22:13.265 INFO:tasks.rados.rados.0.smithi151.stdout:373: finishing write tid 2 to smithi15186934-26 2024-09-13T14:22:13.265 INFO:tasks.rados.rados.0.smithi151.stdout:373: finishing write tid 3 to smithi15186934-26 2024-09-13T14:22:13.265 INFO:tasks.rados.rados.0.smithi151.stdout:373: finishing write tid 4 to smithi15186934-26 2024-09-13T14:22:13.265 INFO:tasks.rados.rados.0.smithi151.stdout:369: done (6 left) 2024-09-13T14:22:13.265 INFO:tasks.rados.rados.0.smithi151.stdout:370: done (5 left) 2024-09-13T14:22:13.265 INFO:tasks.rados.rados.0.smithi151.stdout:371: done (4 left) 2024-09-13T14:22:13.265 INFO:tasks.rados.rados.0.smithi151.stdout:376: done (3 left) 2024-09-13T14:22:13.265 INFO:tasks.rados.rados.0.smithi151.stdout:377: snap_remove snap 37 2024-09-13T14:22:13.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:12 smithi151 ceph-mon[29807]: osdmap e150: 8 total, 8 up, 8 in 2024-09-13T14:22:13.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:12 smithi151 ceph-mon[29807]: osdmap e151: 8 total, 8 up, 8 in 2024-09-13T14:22:13.358 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:12 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:22:12] "GET /metrics HTTP/1.1" 200 243327 "" "Prometheus/2.43.0" 2024-09-13T14:22:13.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:12 smithi151 ceph-mon[41387]: osdmap e150: 8 total, 8 up, 8 in 2024-09-13T14:22:13.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:12 smithi151 ceph-mon[41387]: osdmap e151: 8 total, 8 up, 8 in 2024-09-13T14:22:13.886 INFO:tasks.rados.rados.0.smithi151.stdout:377: done (3 left) 2024-09-13T14:22:13.886 INFO:tasks.rados.rados.0.smithi151.stdout:378: read oid 35 snap -1 2024-09-13T14:22:13.886 INFO:tasks.rados.rados.0.smithi151.stdout:378: expect deleted 2024-09-13T14:22:13.886 INFO:tasks.rados.rados.0.smithi151.stdout:379: delete oid 10 current snap is 47 2024-09-13T14:22:13.886 INFO:tasks.rados.rados.0.smithi151.stdout:373: finishing write tid 5 to smithi15186934-26 2024-09-13T14:22:13.886 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 87 (ObjNum 178 snap 47 seq_num 178) dirty exists 2024-09-13T14:22:13.886 INFO:tasks.rados.rados.0.smithi151.stdout:373: left oid 26 (ObjNum 178 snap 47 seq_num 178) 2024-09-13T14:22:13.886 INFO:tasks.rados.rados.0.smithi151.stdout:374: finishing write tid 1 to smithi15186934-37 2024-09-13T14:22:13.886 INFO:tasks.rados.rados.0.smithi151.stdout:374: finishing write tid 2 to smithi15186934-37 2024-09-13T14:22:13.886 INFO:tasks.rados.rados.0.smithi151.stdout:375: finishing rollback tid 1 to smithi15186934-3 2024-09-13T14:22:13.886 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 122 (ObjNum 33 snap 0 seq_num 3092741760) dirty dne 2024-09-13T14:22:13.886 INFO:tasks.rados.rados.0.smithi151.stdout:374: finishing write tid 3 to smithi15186934-37 2024-09-13T14:22:13.887 INFO:tasks.rados.rados.0.smithi151.stdout:374: finishing write tid 4 to smithi15186934-37 2024-09-13T14:22:13.887 INFO:tasks.rados.rados.0.smithi151.stdout:374: finishing write tid 5 to smithi15186934-37 2024-09-13T14:22:13.887 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 37 v 162 (ObjNum 179 snap 47 seq_num 179) dirty exists 2024-09-13T14:22:13.887 INFO:tasks.rados.rados.0.smithi151.stdout:374: left oid 37 (ObjNum 179 snap 47 seq_num 179) 2024-09-13T14:22:13.889 INFO:tasks.rados.rados.0.smithi151.stdout:373: done (4 left) 2024-09-13T14:22:13.889 INFO:tasks.rados.rados.0.smithi151.stdout:374: done (3 left) 2024-09-13T14:22:13.889 INFO:tasks.rados.rados.0.smithi151.stdout:375: done (2 left) 2024-09-13T14:22:13.889 INFO:tasks.rados.rados.0.smithi151.stdout:379: done (1 left) 2024-09-13T14:22:13.889 INFO:tasks.rados.rados.0.smithi151.stdout:380: snap_create 2024-09-13T14:22:14.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:13 smithi184 ceph-mon[33394]: from='client.15309 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:14.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:13 smithi184 ceph-mon[33394]: from='client.25217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:14.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:13 smithi184 ceph-mon[33394]: pgmap v213: 89 pgs: 89 active+clean; 169 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 15 op/s 2024-09-13T14:22:14.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:13 smithi151 ceph-mon[41387]: from='client.15309 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:14.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:13 smithi151 ceph-mon[41387]: from='client.25217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:14.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:13 smithi151 ceph-mon[41387]: pgmap v213: 89 pgs: 89 active+clean; 169 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 15 op/s 2024-09-13T14:22:14.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:13 smithi151 ceph-mon[29807]: from='client.15309 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:14.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:13 smithi151 ceph-mon[29807]: from='client.25217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:14.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:13 smithi151 ceph-mon[29807]: pgmap v213: 89 pgs: 89 active+clean; 169 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 15 op/s 2024-09-13T14:22:14.895 INFO:tasks.rados.rados.0.smithi151.stdout:378: done (1 left) 2024-09-13T14:22:14.895 INFO:tasks.rados.rados.0.smithi151.stdout:380: done (0 left) 2024-09-13T14:22:14.895 INFO:tasks.rados.rados.0.smithi151.stdout:381: read oid 34 snap -1 2024-09-13T14:22:14.895 INFO:tasks.rados.rados.0.smithi151.stdout:381: expect (ObjNum 112 snap 18 seq_num 112) 2024-09-13T14:22:14.895 INFO:tasks.rados.rados.0.smithi151.stdout:382: read oid 47 snap -1 2024-09-13T14:22:14.895 INFO:tasks.rados.rados.0.smithi151.stdout:382: expect (ObjNum 155 snap 38 seq_num 155) 2024-09-13T14:22:14.895 INFO:tasks.rados.rados.0.smithi151.stdout:383: write oid 25 current snap is 48 2024-09-13T14:22:14.895 INFO:tasks.rados.rados.0.smithi151.stdout:383: seq_num 182 ranges {658874=771437,2083958=603055} 2024-09-13T14:22:14.902 INFO:tasks.rados.rados.0.smithi151.stdout:383: writing smithi15186934-25 from 658874 to 1430311 tid 1 2024-09-13T14:22:14.910 INFO:tasks.rados.rados.0.smithi151.stdout:383: writing smithi15186934-25 from 2083958 to 2687013 tid 2 2024-09-13T14:22:14.911 INFO:tasks.rados.rados.0.smithi151.stdout:384: rollback oid 1 current snap is 48 2024-09-13T14:22:14.911 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 1 to 35 2024-09-13T14:22:14.911 INFO:tasks.rados.rados.0.smithi151.stdout:381: expect (ObjNum 112 snap 18 seq_num 112) 2024-09-13T14:22:14.921 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:22:15.049 INFO:tasks.rados.rados.0.smithi151.stdout:383: finishing write tid 1 to smithi15186934-25 2024-09-13T14:22:15.049 INFO:tasks.rados.rados.0.smithi151.stdout:381: done (3 left) 2024-09-13T14:22:15.050 INFO:tasks.rados.rados.0.smithi151.stdout:385: write oid 10 current snap is 48 2024-09-13T14:22:15.050 INFO:tasks.rados.rados.0.smithi151.stdout:382: expect (ObjNum 155 snap 38 seq_num 155) 2024-09-13T14:22:15.166 INFO:tasks.rados.rados.0.smithi151.stdout:384: finishing rollback tid 1 to smithi15186934-1 2024-09-13T14:22:15.166 INFO:tasks.rados.rados.0.smithi151.stdout:384: finishing rollback tid 2 to smithi15186934-1 2024-09-13T14:22:15.166 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 1 v 111 (ObjNum 137 snap 28 seq_num 137) dirty exists 2024-09-13T14:22:15.167 INFO:tasks.rados.rados.0.smithi151.stdout:383: finishing write tid 2 to smithi15186934-25 2024-09-13T14:22:15.167 INFO:tasks.rados.rados.0.smithi151.stdout:383: finishing write tid 3 to smithi15186934-25 2024-09-13T14:22:15.167 INFO:tasks.rados.rados.0.smithi151.stdout:385: seq_num 183 ranges {675681=563039,1858793=746713,3025937=49744} 2024-09-13T14:22:15.173 INFO:tasks.rados.rados.0.smithi151.stdout:385: writing smithi15186934-10 from 675681 to 1238720 tid 1 2024-09-13T14:22:15.181 INFO:tasks.rados.rados.0.smithi151.stdout:385: writing smithi15186934-10 from 1858793 to 2605506 tid 2 2024-09-13T14:22:15.184 INFO:tasks.rados.rados.0.smithi151.stdout:385: writing smithi15186934-10 from 3025937 to 3075681 tid 3 2024-09-13T14:22:15.184 INFO:tasks.rados.rados.0.smithi151.stdout:382: done (3 left) 2024-09-13T14:22:15.184 INFO:tasks.rados.rados.0.smithi151.stdout:384: done (2 left) 2024-09-13T14:22:15.184 INFO:tasks.rados.rados.0.smithi151.stdout:386: snap_create 2024-09-13T14:22:15.184 INFO:tasks.rados.rados.0.smithi151.stdout:383: finishing write tid 4 to smithi15186934-25 2024-09-13T14:22:15.184 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 104 (ObjNum 182 snap 48 seq_num 182) dirty exists 2024-09-13T14:22:15.184 INFO:tasks.rados.rados.0.smithi151.stdout:383: left oid 25 (ObjNum 182 snap 48 seq_num 182) 2024-09-13T14:22:15.184 INFO:tasks.rados.rados.0.smithi151.stdout:385: finishing write tid 1 to smithi15186934-10 2024-09-13T14:22:15.185 INFO:tasks.rados.rados.0.smithi151.stdout:385: finishing write tid 2 to smithi15186934-10 2024-09-13T14:22:15.185 INFO:tasks.rados.rados.0.smithi151.stdout:385: finishing write tid 3 to smithi15186934-10 2024-09-13T14:22:15.185 INFO:tasks.rados.rados.0.smithi151.stdout:385: finishing write tid 4 to smithi15186934-10 2024-09-13T14:22:15.185 INFO:tasks.rados.rados.0.smithi151.stdout:385: finishing write tid 5 to smithi15186934-10 2024-09-13T14:22:15.186 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 94 (ObjNum 183 snap 48 seq_num 183) dirty exists 2024-09-13T14:22:15.186 INFO:tasks.rados.rados.0.smithi151.stdout:385: left oid 10 (ObjNum 183 snap 48 seq_num 183) 2024-09-13T14:22:15.270 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:14 smithi184 ceph-mon[33394]: osdmap e152: 8 total, 8 up, 8 in 2024-09-13T14:22:15.270 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:14 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/2865068020' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:15.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:14 smithi151 ceph-mon[29807]: osdmap e152: 8 total, 8 up, 8 in 2024-09-13T14:22:15.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:14 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/2865068020' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:15.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:14 smithi151 ceph-mon[41387]: osdmap e152: 8 total, 8 up, 8 in 2024-09-13T14:22:15.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:14 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/2865068020' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:15.399 INFO:tasks.rados.rados.0.smithi151.stdout:383: done (2 left) 2024-09-13T14:22:15.399 INFO:tasks.rados.rados.0.smithi151.stdout:385: done (1 left) 2024-09-13T14:22:15.399 INFO:tasks.rados.rados.0.smithi151.stdout:386: done (0 left) 2024-09-13T14:22:15.399 INFO:tasks.rados.rados.0.smithi151.stdout:387: read oid 39 snap -1 2024-09-13T14:22:15.399 INFO:tasks.rados.rados.0.smithi151.stdout:387: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-13T14:22:15.399 INFO:tasks.rados.rados.0.smithi151.stdout:388: delete oid 22 current snap is 49 2024-09-13T14:22:15.407 INFO:tasks.rados.rados.0.smithi151.stdout:388: done (1 left) 2024-09-13T14:22:15.408 INFO:tasks.rados.rados.0.smithi151.stdout:389: rollback oid 29 current snap is 49 2024-09-13T14:22:15.408 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 29 to 38 2024-09-13T14:22:15.408 INFO:tasks.rados.rados.0.smithi151.stdout:390: read oid 28 snap -1 2024-09-13T14:22:15.408 INFO:tasks.rados.rados.0.smithi151.stdout:390: expect (ObjNum 173 snap 46 seq_num 173) 2024-09-13T14:22:15.408 INFO:tasks.rados.rados.0.smithi151.stdout:391: read oid 33 snap -1 2024-09-13T14:22:15.408 INFO:tasks.rados.rados.0.smithi151.stdout:391: expect (ObjNum 176 snap 47 seq_num 176) 2024-09-13T14:22:15.408 INFO:tasks.rados.rados.0.smithi151.stdout:392: rollback oid 20 current snap is 49 2024-09-13T14:22:15.408 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 20 to 48 2024-09-13T14:22:15.408 INFO:tasks.rados.rados.0.smithi151.stdout:393: snap_remove snap 23 2024-09-13T14:22:15.604 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: [13/Sep/2024:14:22:15] ENGINE Bus STOPPING 2024-09-13T14:22:15.655 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:22:15.393+0000 7fdc1c7bb640 -1 mgr handle_mgr_map I was active but no longer am 2024-09-13T14:22:15.655 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: ignoring --setuser ceph since I am not root 2024-09-13T14:22:15.655 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: ignoring --setgroup ceph since I am not root 2024-09-13T14:22:15.655 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mgr[30079]: -- 172.21.15.151:0/4119678286 <== mon.0 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d9c37ce340 con 0x55d9c37da800 2024-09-13T14:22:15.655 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:22:15.534+0000 7f6954db3ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T14:22:15.655 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:22:15.589+0000 7f6954db3ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T14:22:15.655 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:22:15.654+0000 7f6954db3ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T14:22:15.915 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:22:15.777+0000 7f6954db3ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:22:15.916 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:22:15.914+0000 7f6954db3ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T14:22:15.916 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[29807]: pgmap v216: 89 pgs: 89 active+clean; 180 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.2 MiB/s rd, 1.8 MiB/s wr, 16 op/s 2024-09-13T14:22:15.916 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[29807]: osdmap e153: 8 total, 8 up, 8 in 2024-09-13T14:22:15.916 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:15.916 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:15.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:22:15.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:22:15.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:15.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:22:15.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-13T14:22:15.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[29807]: Activating manager daemon x 2024-09-13T14:22:15.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[29807]: osdmap e154: 8 total, 8 up, 8 in 2024-09-13T14:22:15.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[29807]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-13T14:22:15.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[29807]: mgrmap e22: x(active, starting, since 0.00932587s) 2024-09-13T14:22:15.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:22:15.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:22:15.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:22:15.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi184.jismsi"}]: dispatch 2024-09-13T14:22:15.918 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi151.rqqklw"}]: dispatch 2024-09-13T14:22:15.918 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:22:15.918 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:22:15.918 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:22:15.918 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:22:15.918 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:22:15.918 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:22:15.918 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:22:15.918 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:22:15.918 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:22:15.918 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:22:15.918 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:22:15.918 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:22:15.919 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[41387]: pgmap v216: 89 pgs: 89 active+clean; 180 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.2 MiB/s rd, 1.8 MiB/s wr, 16 op/s 2024-09-13T14:22:15.919 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[41387]: osdmap e153: 8 total, 8 up, 8 in 2024-09-13T14:22:15.919 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:15.919 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:15.919 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:22:15.919 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:22:15.919 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:15.919 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:22:15.919 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-13T14:22:15.919 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[41387]: Activating manager daemon x 2024-09-13T14:22:15.920 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[41387]: osdmap e154: 8 total, 8 up, 8 in 2024-09-13T14:22:15.920 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[41387]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-13T14:22:15.920 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[41387]: mgrmap e22: x(active, starting, since 0.00932587s) 2024-09-13T14:22:15.920 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:22:15.920 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:22:15.920 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:22:15.920 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi184.jismsi"}]: dispatch 2024-09-13T14:22:15.920 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi151.rqqklw"}]: dispatch 2024-09-13T14:22:15.920 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:22:15.920 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:22:15.921 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:22:15.921 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:22:15.921 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:22:15.921 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:22:15.921 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:22:15.921 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:22:15.921 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:22:15.921 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:22:15.921 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:22:15.921 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:15 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:22:16.353 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: [13/Sep/2024:14:22:15] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:22:16.353 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: [13/Sep/2024:14:22:15] ENGINE Bus STOPPED 2024-09-13T14:22:16.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-mon[33394]: pgmap v216: 89 pgs: 89 active+clean; 180 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.2 MiB/s rd, 1.8 MiB/s wr, 16 op/s 2024-09-13T14:22:16.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-mon[33394]: osdmap e153: 8 total, 8 up, 8 in 2024-09-13T14:22:16.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:16.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:16.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:22:16.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:22:16.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' 2024-09-13T14:22:16.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:22:16.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-13T14:22:16.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-mon[33394]: Activating manager daemon x 2024-09-13T14:22:16.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-mon[33394]: osdmap e154: 8 total, 8 up, 8 in 2024-09-13T14:22:16.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-mon[33394]: from='mgr.14460 172.21.15.151:0/2204401699' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-13T14:22:16.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-mon[33394]: mgrmap e22: x(active, starting, since 0.00932587s) 2024-09-13T14:22:16.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:22:16.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:22:16.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:22:16.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi184.jismsi"}]: dispatch 2024-09-13T14:22:16.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi151.rqqklw"}]: dispatch 2024-09-13T14:22:16.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:22:16.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:22:16.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:22:16.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:22:16.356 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:22:16.356 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:22:16.356 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:22:16.356 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:22:16.356 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:22:16.356 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:22:16.356 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:22:16.356 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:15 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:22:16.397 INFO:tasks.rados.rados.0.smithi151.stdout:393: done (5 left) 2024-09-13T14:22:16.397 INFO:tasks.rados.rados.0.smithi151.stdout:394: write oid 17 current snap is 49 2024-09-13T14:22:16.397 INFO:tasks.rados.rados.0.smithi151.stdout:387: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-13T14:22:16.494 INFO:tasks.rados.rados.0.smithi151.stdout:392: finishing rollback tid 1 to smithi15186934-20 2024-09-13T14:22:16.494 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 0 (ObjNum 33 snap 0 seq_num 3064833400) dirty dne 2024-09-13T14:22:16.494 INFO:tasks.rados.rados.0.smithi151.stdout:389: finishing rollback tid 1 to smithi15186934-29 2024-09-13T14:22:16.494 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 29 v 56 (ObjNum 0 snap 0 seq_num 304) dirty dne 2024-09-13T14:22:16.494 INFO:tasks.rados.rados.0.smithi151.stdout:391: expect (ObjNum 176 snap 47 seq_num 176) 2024-09-13T14:22:16.595 INFO:tasks.rados.rados.0.smithi151.stdout:390: expect (ObjNum 173 snap 46 seq_num 173) 2024-09-13T14:22:16.608 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:22:16.171+0000 7f6954db3ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T14:22:16.608 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:22:16.231+0000 7f6954db3ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T14:22:16.608 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:22:16.391+0000 7f6954db3ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T14:22:16.728 INFO:tasks.rados.rados.0.smithi151.stdout:394: seq_num 185 ranges {709295=713537,1859404=750382} 2024-09-13T14:22:16.735 INFO:tasks.rados.rados.0.smithi151.stdout:394: writing smithi15186934-17 from 709295 to 1422832 tid 1 2024-09-13T14:22:16.742 INFO:tasks.rados.rados.0.smithi151.stdout:394: writing smithi15186934-17 from 1859404 to 2609786 tid 2 2024-09-13T14:22:16.742 INFO:tasks.rados.rados.0.smithi151.stdout:387: done (5 left) 2024-09-13T14:22:16.742 INFO:tasks.rados.rados.0.smithi151.stdout:389: done (4 left) 2024-09-13T14:22:16.742 INFO:tasks.rados.rados.0.smithi151.stdout:390: done (3 left) 2024-09-13T14:22:16.742 INFO:tasks.rados.rados.0.smithi151.stdout:391: done (2 left) 2024-09-13T14:22:16.742 INFO:tasks.rados.rados.0.smithi151.stdout:392: done (1 left) 2024-09-13T14:22:16.743 INFO:tasks.rados.rados.0.smithi151.stdout:395: write oid 23 current snap is 49 2024-09-13T14:22:16.743 INFO:tasks.rados.rados.0.smithi151.stdout:394: finishing write tid 1 to smithi15186934-17 2024-09-13T14:22:16.743 INFO:tasks.rados.rados.0.smithi151.stdout:395: seq_num 186 ranges {726102=788786,1917886=610393,3108206=17896} 2024-09-13T14:22:16.751 INFO:tasks.rados.rados.0.smithi151.stdout:395: writing smithi15186934-23 from 726102 to 1514888 tid 1 2024-09-13T14:22:16.756 INFO:tasks.rados.rados.0.smithi151.stdout:395: writing smithi15186934-23 from 1917886 to 2528279 tid 2 2024-09-13T14:22:16.760 INFO:tasks.rados.rados.0.smithi151.stdout:395: writing smithi15186934-23 from 3108206 to 3126102 tid 3 2024-09-13T14:22:16.760 INFO:tasks.rados.rados.0.smithi151.stdout:396: write oid 35 current snap is 49 2024-09-13T14:22:16.760 INFO:tasks.rados.rados.0.smithi151.stdout:394: finishing write tid 2 to smithi15186934-17 2024-09-13T14:22:16.760 INFO:tasks.rados.rados.0.smithi151.stdout:394: finishing write tid 3 to smithi15186934-17 2024-09-13T14:22:16.760 INFO:tasks.rados.rados.0.smithi151.stdout:396: seq_num 187 ranges {742909=464035,1976368=754051} 2024-09-13T14:22:16.766 INFO:tasks.rados.rados.0.smithi151.stdout:396: writing smithi15186934-35 from 742909 to 1206944 tid 1 2024-09-13T14:22:16.775 INFO:tasks.rados.rados.0.smithi151.stdout:396: writing smithi15186934-35 from 1976368 to 2730419 tid 2 2024-09-13T14:22:16.775 INFO:tasks.rados.rados.0.smithi151.stdout:397: read oid 15 snap 47 2024-09-13T14:22:16.775 INFO:tasks.rados.rados.0.smithi151.stdout:397: expect (ObjNum 105 snap 16 seq_num 105) 2024-09-13T14:22:16.776 INFO:tasks.rados.rados.0.smithi151.stdout:398: snap_create 2024-09-13T14:22:16.776 INFO:tasks.rados.rados.0.smithi151.stdout:394: finishing write tid 4 to smithi15186934-17 2024-09-13T14:22:16.776 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 174 (ObjNum 185 snap 49 seq_num 185) dirty exists 2024-09-13T14:22:16.776 INFO:tasks.rados.rados.0.smithi151.stdout:394: left oid 17 (ObjNum 185 snap 49 seq_num 185) 2024-09-13T14:22:16.776 INFO:tasks.rados.rados.0.smithi151.stdout:395: finishing write tid 1 to smithi15186934-23 2024-09-13T14:22:16.776 INFO:tasks.rados.rados.0.smithi151.stdout:395: finishing write tid 2 to smithi15186934-23 2024-09-13T14:22:16.776 INFO:tasks.rados.rados.0.smithi151.stdout:395: finishing write tid 3 to smithi15186934-23 2024-09-13T14:22:16.776 INFO:tasks.rados.rados.0.smithi151.stdout:395: finishing write tid 4 to smithi15186934-23 2024-09-13T14:22:16.776 INFO:tasks.rados.rados.0.smithi151.stdout:395: finishing write tid 5 to smithi15186934-23 2024-09-13T14:22:16.776 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 23 v 184 (ObjNum 186 snap 49 seq_num 186) dirty exists 2024-09-13T14:22:16.776 INFO:tasks.rados.rados.0.smithi151.stdout:395: left oid 23 (ObjNum 186 snap 49 seq_num 186) 2024-09-13T14:22:16.776 INFO:tasks.rados.rados.0.smithi151.stdout:396: finishing write tid 1 to smithi15186934-35 2024-09-13T14:22:16.777 INFO:tasks.rados.rados.0.smithi151.stdout:397: expect (ObjNum 105 snap 16 seq_num 105) 2024-09-13T14:22:16.852 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:16 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: [13/Sep/2024:14:22:16] ENGINE Bus STARTING 2024-09-13T14:22:16.852 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:16 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: 2024-09-13T14:22:16.535+0000 7f201e6e4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:22:16.852 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:16 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: 2024-09-13T14:22:16.535+0000 7f201e6e4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:22:16.852 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:16 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: 2024-09-13T14:22:16.535+0000 7f201e6e4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:22:16.852 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:16 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: 2024-09-13T14:22:16.535+0000 7f201e6e4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:22:16.852 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:16 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: 2024-09-13T14:22:16.535+0000 7f201e6e4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:22:16.852 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:16 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: 2024-09-13T14:22:16.536+0000 7f2022eed640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:22:16.853 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:16 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: 2024-09-13T14:22:16.536+0000 7f2022eed640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:22:16.853 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:16 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: 2024-09-13T14:22:16.536+0000 7f2022eed640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:22:16.853 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:16 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: 2024-09-13T14:22:16.536+0000 7f2022eed640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:22:16.853 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:16 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: 2024-09-13T14:22:16.536+0000 7f2022eed640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:22:16.853 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:16 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: [13/Sep/2024:14:22:16] ENGINE Serving on http://:::9283 2024-09-13T14:22:16.853 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:16 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: [13/Sep/2024:14:22:16] ENGINE Bus STARTED 2024-09-13T14:22:16.982 INFO:tasks.rados.rados.0.smithi151.stdout:396: finishing write tid 2 to smithi15186934-35 2024-09-13T14:22:16.983 INFO:tasks.rados.rados.0.smithi151.stdout:396: finishing write tid 3 to smithi15186934-35 2024-09-13T14:22:16.983 INFO:tasks.rados.rados.0.smithi151.stdout:396: finishing write tid 4 to smithi15186934-35 2024-09-13T14:22:16.983 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 35 v 177 (ObjNum 187 snap 49 seq_num 187) dirty exists 2024-09-13T14:22:16.983 INFO:tasks.rados.rados.0.smithi151.stdout:396: left oid 35 (ObjNum 187 snap 49 seq_num 187) 2024-09-13T14:22:17.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:16 smithi184 ceph-mon[33394]: Manager daemon x is now available 2024-09-13T14:22:17.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:16 smithi184 ceph-mon[33394]: osdmap e155: 8 total, 8 up, 8 in 2024-09-13T14:22:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:16 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:16 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:16 smithi184 ceph-mon[33394]: Checking for cert/key for grafana.a 2024-09-13T14:22:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:16 smithi184 ceph-mon[33394]: Migrating grafana.a cert to cert store 2024-09-13T14:22:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:16 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:16 smithi184 ceph-mon[33394]: Migrating grafana.a key to cert store 2024-09-13T14:22:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:16 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:16 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:16 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:16 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:22:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:16 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:22:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:16 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:22:17.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:16 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:22:17.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:16 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-13T14:22:17.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:16 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-13T14:22:17.348 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[41387]: Manager daemon x is now available 2024-09-13T14:22:17.348 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[41387]: osdmap e155: 8 total, 8 up, 8 in 2024-09-13T14:22:17.348 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:17.348 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:17.348 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[41387]: Checking for cert/key for grafana.a 2024-09-13T14:22:17.348 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[41387]: Migrating grafana.a cert to cert store 2024-09-13T14:22:17.349 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:17.349 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[41387]: Migrating grafana.a key to cert store 2024-09-13T14:22:17.349 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:17.349 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:17.349 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:17.349 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:22:17.349 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:22:17.349 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:22:17.349 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:22:17.349 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-13T14:22:17.349 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-13T14:22:17.350 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[29807]: Manager daemon x is now available 2024-09-13T14:22:17.350 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[29807]: osdmap e155: 8 total, 8 up, 8 in 2024-09-13T14:22:17.350 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:17.350 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:17.350 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[29807]: Checking for cert/key for grafana.a 2024-09-13T14:22:17.350 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[29807]: Migrating grafana.a cert to cert store 2024-09-13T14:22:17.350 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:17.350 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[29807]: Migrating grafana.a key to cert store 2024-09-13T14:22:17.350 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:17.350 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:17.350 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:17.350 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:22:17.351 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:22:17.351 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:22:17.351 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:22:17.351 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-13T14:22:17.351 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:16 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-13T14:22:17.351 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:17 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:22:17.346+0000 7f6954db3ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T14:22:17.463 INFO:tasks.rados.rados.0.smithi151.stdout:394: done (4 left) 2024-09-13T14:22:17.463 INFO:tasks.rados.rados.0.smithi151.stdout:395: done (3 left) 2024-09-13T14:22:17.463 INFO:tasks.rados.rados.0.smithi151.stdout:396: done (2 left) 2024-09-13T14:22:17.463 INFO:tasks.rados.rados.0.smithi151.stdout:397: done (1 left) 2024-09-13T14:22:17.463 INFO:tasks.rados.rados.0.smithi151.stdout:398: done (0 left) 2024-09-13T14:22:17.463 INFO:tasks.rados.rados.0.smithi151.stdout:399: read oid 47 snap 46 2024-09-13T14:22:17.463 INFO:tasks.rados.rados.0.smithi151.stdout:399: expect (ObjNum 155 snap 38 seq_num 155) 2024-09-13T14:22:17.463 INFO:tasks.rados.rados.0.smithi151.stdout:400: rollback oid 24 current snap is 50 2024-09-13T14:22:17.463 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 24 to 20 2024-09-13T14:22:17.463 INFO:tasks.rados.rados.0.smithi151.stdout:401: delete oid 8 current snap is 50 2024-09-13T14:22:17.464 INFO:tasks.rados.rados.0.smithi151.stdout:400: finishing rollback tid 1 to smithi15186934-24 2024-09-13T14:22:17.464 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 24 v 0 (ObjNum 129 snap 0 seq_num 3092666528) dirty dne 2024-09-13T14:22:17.466 INFO:tasks.rados.rados.0.smithi151.stdout:400: done (2 left) 2024-09-13T14:22:17.466 INFO:tasks.rados.rados.0.smithi151.stdout:401: done (1 left) 2024-09-13T14:22:17.466 INFO:tasks.rados.rados.0.smithi151.stdout:402: snap_remove snap 20 2024-09-13T14:22:17.608 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:17 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:22:17.414+0000 7f6954db3ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T14:22:17.792 INFO:tasks.rados.rados.0.smithi151.stdout:402: done (1 left) 2024-09-13T14:22:17.792 INFO:tasks.rados.rados.0.smithi151.stdout:403: snap_create 2024-09-13T14:22:17.792 INFO:tasks.rados.rados.0.smithi151.stdout:399: expect (ObjNum 155 snap 38 seq_num 155) 2024-09-13T14:22:17.853 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:22:17.854 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:22:17] "GET /metrics HTTP/1.1" 200 33449 "" "Prometheus/2.43.0" 2024-09-13T14:22:17.854 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: 2024-09-13T14:22:17.572+0000 7f2006437640 -1 log_channel(cephadm) log [ERR] : [13/Sep/2024:14:22:17] ENGINE Error in HTTPServer.serve 2024-09-13T14:22:17.854 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: Traceback (most recent call last): 2024-09-13T14:22:17.854 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:22:17.854 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: self._connections.run(self.expiration_interval) 2024-09-13T14:22:17.854 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:22:17.854 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: self._run(expiration_interval) 2024-09-13T14:22:17.855 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:22:17.855 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:22:17.855 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:22:17.855 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:22:17.855 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:22:17.855 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: s = self.context.wrap_socket( 2024-09-13T14:22:17.855 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:22:17.855 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: return self.sslsocket_class._create( 2024-09-13T14:22:17.855 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:22:17.855 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: self.do_handshake() 2024-09-13T14:22:17.855 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:22:17.855 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: self._sslobj.do_handshake() 2024-09-13T14:22:17.855 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:22:17.855 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: 2024-09-13T14:22:18.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:17 smithi151 ceph-mon[41387]: mgrmap e23: x(active, since 1.51055s) 2024-09-13T14:22:18.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:17 smithi151 ceph-mon[41387]: pgmap v4: 89 pgs: 89 active+clean; 195 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:18.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:17 smithi151 ceph-mon[41387]: Deploying cephadm binary to smithi151 2024-09-13T14:22:18.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:17 smithi151 ceph-mon[41387]: Deploying cephadm binary to smithi184 2024-09-13T14:22:18.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:17 smithi151 ceph-mon[41387]: [13/Sep/2024:14:22:17] ENGINE Bus STARTING 2024-09-13T14:22:18.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:17 smithi151 ceph-mon[41387]: pgmap v5: 89 pgs: 89 active+clean; 195 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:18.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:17 smithi151 ceph-mon[41387]: [13/Sep/2024:14:22:17] ENGINE Serving on http://172.21.15.184:8765 2024-09-13T14:22:18.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:17 smithi151 ceph-mon[41387]: osdmap e156: 8 total, 8 up, 8 in 2024-09-13T14:22:18.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:17 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/3125551364' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:18.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:17 smithi151 ceph-mon[41387]: [13/Sep/2024:14:22:17] ENGINE Serving on https://172.21.15.184:7150 2024-09-13T14:22:18.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:17 smithi151 ceph-mon[41387]: [13/Sep/2024:14:22:17] ENGINE Bus STARTED 2024-09-13T14:22:18.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:17 smithi151 ceph-mon[41387]: [13/Sep/2024:14:22:17] ENGINE Error in HTTPServer.serve 2024-09-13T14:22:18.109 INFO:journalctl@ceph.mon.c.smithi151.stdout: Traceback (most recent call last): 2024-09-13T14:22:18.109 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:22:18.109 INFO:journalctl@ceph.mon.c.smithi151.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:22:18.109 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:22:18.110 INFO:journalctl@ceph.mon.c.smithi151.stdout: self._run(expiration_interval) 2024-09-13T14:22:18.110 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:22:18.110 INFO:journalctl@ceph.mon.c.smithi151.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:22:18.110 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:22:18.110 INFO:journalctl@ceph.mon.c.smithi151.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:22:18.110 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:22:18.110 INFO:journalctl@ceph.mon.c.smithi151.stdout: s = self.context.wrap_socket( 2024-09-13T14:22:18.110 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:22:18.110 INFO:journalctl@ceph.mon.c.smithi151.stdout: return self.sslsocket_class._create( 2024-09-13T14:22:18.110 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:22:18.110 INFO:journalctl@ceph.mon.c.smithi151.stdout: self.do_handshake() 2024-09-13T14:22:18.110 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:22:18.110 INFO:journalctl@ceph.mon.c.smithi151.stdout: self._sslobj.do_handshake() 2024-09-13T14:22:18.110 INFO:journalctl@ceph.mon.c.smithi151.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:22:18.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:17 smithi151 ceph-mon[41387]: osdmap e157: 8 total, 8 up, 8 in 2024-09-13T14:22:18.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:17 smithi151 ceph-mon[29807]: mgrmap e23: x(active, since 1.51055s) 2024-09-13T14:22:18.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:17 smithi151 ceph-mon[29807]: pgmap v4: 89 pgs: 89 active+clean; 195 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:18.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:17 smithi151 ceph-mon[29807]: Deploying cephadm binary to smithi151 2024-09-13T14:22:18.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:17 smithi151 ceph-mon[29807]: Deploying cephadm binary to smithi184 2024-09-13T14:22:18.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:17 smithi151 ceph-mon[29807]: [13/Sep/2024:14:22:17] ENGINE Bus STARTING 2024-09-13T14:22:18.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:17 smithi151 ceph-mon[29807]: pgmap v5: 89 pgs: 89 active+clean; 195 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:18.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:17 smithi151 ceph-mon[29807]: [13/Sep/2024:14:22:17] ENGINE Serving on http://172.21.15.184:8765 2024-09-13T14:22:18.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:17 smithi151 ceph-mon[29807]: osdmap e156: 8 total, 8 up, 8 in 2024-09-13T14:22:18.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:17 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/3125551364' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:18.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:17 smithi151 ceph-mon[29807]: [13/Sep/2024:14:22:17] ENGINE Serving on https://172.21.15.184:7150 2024-09-13T14:22:18.112 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:17 smithi151 ceph-mon[29807]: [13/Sep/2024:14:22:17] ENGINE Bus STARTED 2024-09-13T14:22:18.112 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:17 smithi151 ceph-mon[29807]: [13/Sep/2024:14:22:17] ENGINE Error in HTTPServer.serve 2024-09-13T14:22:18.112 INFO:journalctl@ceph.mon.a.smithi151.stdout: Traceback (most recent call last): 2024-09-13T14:22:18.112 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:22:18.112 INFO:journalctl@ceph.mon.a.smithi151.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:22:18.112 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:22:18.112 INFO:journalctl@ceph.mon.a.smithi151.stdout: self._run(expiration_interval) 2024-09-13T14:22:18.112 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:22:18.112 INFO:journalctl@ceph.mon.a.smithi151.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:22:18.112 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:22:18.112 INFO:journalctl@ceph.mon.a.smithi151.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:22:18.112 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:22:18.112 INFO:journalctl@ceph.mon.a.smithi151.stdout: s = self.context.wrap_socket( 2024-09-13T14:22:18.113 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:22:18.113 INFO:journalctl@ceph.mon.a.smithi151.stdout: return self.sslsocket_class._create( 2024-09-13T14:22:18.113 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:22:18.113 INFO:journalctl@ceph.mon.a.smithi151.stdout: self.do_handshake() 2024-09-13T14:22:18.113 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:22:18.113 INFO:journalctl@ceph.mon.a.smithi151.stdout: self._sslobj.do_handshake() 2024-09-13T14:22:18.113 INFO:journalctl@ceph.mon.a.smithi151.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:22:18.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:17 smithi151 ceph-mon[29807]: osdmap e157: 8 total, 8 up, 8 in 2024-09-13T14:22:18.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-mon[33394]: mgrmap e23: x(active, since 1.51055s) 2024-09-13T14:22:18.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-mon[33394]: pgmap v4: 89 pgs: 89 active+clean; 195 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:18.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-mon[33394]: Deploying cephadm binary to smithi151 2024-09-13T14:22:18.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-mon[33394]: Deploying cephadm binary to smithi184 2024-09-13T14:22:18.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-mon[33394]: [13/Sep/2024:14:22:17] ENGINE Bus STARTING 2024-09-13T14:22:18.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-mon[33394]: pgmap v5: 89 pgs: 89 active+clean; 195 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:18.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-mon[33394]: [13/Sep/2024:14:22:17] ENGINE Serving on http://172.21.15.184:8765 2024-09-13T14:22:18.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-mon[33394]: osdmap e156: 8 total, 8 up, 8 in 2024-09-13T14:22:18.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/3125551364' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:18.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-mon[33394]: [13/Sep/2024:14:22:17] ENGINE Serving on https://172.21.15.184:7150 2024-09-13T14:22:18.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-mon[33394]: [13/Sep/2024:14:22:17] ENGINE Bus STARTED 2024-09-13T14:22:18.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-mon[33394]: [13/Sep/2024:14:22:17] ENGINE Error in HTTPServer.serve 2024-09-13T14:22:18.355 INFO:journalctl@ceph.mon.b.smithi184.stdout: Traceback (most recent call last): 2024-09-13T14:22:18.355 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:22:18.355 INFO:journalctl@ceph.mon.b.smithi184.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:22:18.355 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:22:18.355 INFO:journalctl@ceph.mon.b.smithi184.stdout: self._run(expiration_interval) 2024-09-13T14:22:18.355 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:22:18.355 INFO:journalctl@ceph.mon.b.smithi184.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:22:18.355 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:22:18.355 INFO:journalctl@ceph.mon.b.smithi184.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:22:18.355 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:22:18.355 INFO:journalctl@ceph.mon.b.smithi184.stdout: s = self.context.wrap_socket( 2024-09-13T14:22:18.355 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:22:18.356 INFO:journalctl@ceph.mon.b.smithi184.stdout: return self.sslsocket_class._create( 2024-09-13T14:22:18.356 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:22:18.356 INFO:journalctl@ceph.mon.b.smithi184.stdout: self.do_handshake() 2024-09-13T14:22:18.356 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:22:18.356 INFO:journalctl@ceph.mon.b.smithi184.stdout: self._sslobj.do_handshake() 2024-09-13T14:22:18.356 INFO:journalctl@ceph.mon.b.smithi184.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:22:18.356 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:17 smithi184 ceph-mon[33394]: osdmap e157: 8 total, 8 up, 8 in 2024-09-13T14:22:18.467 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:18 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:22:18.174+0000 7f6954db3ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T14:22:18.468 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:18 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:22:18.285+0000 7f6954db3ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T14:22:18.468 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:18 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:22:18.406+0000 7f6954db3ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:22:18.468 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:18 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:22:18.466+0000 7f6954db3ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T14:22:18.732 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:18 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:22:18.595+0000 7f6954db3ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T14:22:18.732 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:18 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:22:18.667+0000 7f6954db3ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T14:22:18.733 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:18 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:22:18.728+0000 7f6954db3ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T14:22:18.792 INFO:tasks.rados.rados.0.smithi151.stdout:399: done (1 left) 2024-09-13T14:22:18.792 INFO:tasks.rados.rados.0.smithi151.stdout:403: done (0 left) 2024-09-13T14:22:18.792 INFO:tasks.rados.rados.0.smithi151.stdout:404: write oid 45 current snap is 51 2024-09-13T14:22:18.792 INFO:tasks.rados.rados.0.smithi151.stdout:404: seq_num 189 ranges {776523=614533,1976979=757720} 2024-09-13T14:22:18.800 INFO:tasks.rados.rados.0.smithi151.stdout:404: writing smithi15186934-45 from 776523 to 1391056 tid 1 2024-09-13T14:22:18.808 INFO:tasks.rados.rados.0.smithi151.stdout:404: writing smithi15186934-45 from 1976979 to 2734699 tid 2 2024-09-13T14:22:18.808 INFO:tasks.rados.rados.0.smithi151.stdout:405: write oid 9 current snap is 51 2024-09-13T14:22:18.808 INFO:tasks.rados.rados.0.smithi151.stdout:404: finishing write tid 1 to smithi15186934-45 2024-09-13T14:22:18.808 INFO:tasks.rados.rados.0.smithi151.stdout:405: seq_num 190 ranges {793330=689782,2035461=617731,3101545=91785} 2024-09-13T14:22:18.816 INFO:tasks.rados.rados.0.smithi151.stdout:405: writing smithi15186934-9 from 793330 to 1483112 tid 1 2024-09-13T14:22:18.823 INFO:tasks.rados.rados.0.smithi151.stdout:405: writing smithi15186934-9 from 2035461 to 2653192 tid 2 2024-09-13T14:22:18.826 INFO:tasks.rados.rados.0.smithi151.stdout:405: writing smithi15186934-9 from 3101545 to 3193330 tid 3 2024-09-13T14:22:18.827 INFO:tasks.rados.rados.0.smithi151.stdout:404: finishing write tid 2 to smithi15186934-45 2024-09-13T14:22:18.827 INFO:tasks.rados.rados.0.smithi151.stdout:406: write oid 21 current snap is 51 2024-09-13T14:22:18.827 INFO:tasks.rados.rados.0.smithi151.stdout:406: seq_num 191 ranges {410137=481384,1410296=761389,2728968=481169} 2024-09-13T14:22:18.831 INFO:tasks.rados.rados.0.smithi151.stdout:406: writing smithi15186934-21 from 410137 to 891521 tid 1 2024-09-13T14:22:18.838 INFO:tasks.rados.rados.0.smithi151.stdout:406: writing smithi15186934-21 from 1410296 to 2171685 tid 2 2024-09-13T14:22:18.845 INFO:tasks.rados.rados.0.smithi151.stdout:406: writing smithi15186934-21 from 2728968 to 3210137 tid 3 2024-09-13T14:22:18.845 INFO:tasks.rados.rados.0.smithi151.stdout:407: rollback oid 26 current snap is 51 2024-09-13T14:22:18.845 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 26 to 46 2024-09-13T14:22:18.845 INFO:tasks.rados.rados.0.smithi151.stdout:404: finishing write tid 3 to smithi15186934-45 2024-09-13T14:22:18.845 INFO:tasks.rados.rados.0.smithi151.stdout:404: finishing write tid 4 to smithi15186934-45 2024-09-13T14:22:18.845 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 124 (ObjNum 189 snap 51 seq_num 189) dirty exists 2024-09-13T14:22:18.845 INFO:tasks.rados.rados.0.smithi151.stdout:404: left oid 45 (ObjNum 189 snap 51 seq_num 189) 2024-09-13T14:22:18.846 INFO:tasks.rados.rados.0.smithi151.stdout:405: finishing write tid 1 to smithi15186934-9 2024-09-13T14:22:18.846 INFO:tasks.rados.rados.0.smithi151.stdout:405: finishing write tid 2 to smithi15186934-9 2024-09-13T14:22:18.846 INFO:tasks.rados.rados.0.smithi151.stdout:405: finishing write tid 3 to smithi15186934-9 2024-09-13T14:22:18.846 INFO:tasks.rados.rados.0.smithi151.stdout:405: finishing write tid 4 to smithi15186934-9 2024-09-13T14:22:18.846 INFO:tasks.rados.rados.0.smithi151.stdout:405: finishing write tid 5 to smithi15186934-9 2024-09-13T14:22:18.846 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 142 (ObjNum 190 snap 51 seq_num 190) dirty exists 2024-09-13T14:22:18.846 INFO:tasks.rados.rados.0.smithi151.stdout:405: left oid 9 (ObjNum 190 snap 51 seq_num 190) 2024-09-13T14:22:18.846 INFO:tasks.rados.rados.0.smithi151.stdout:406: finishing write tid 1 to smithi15186934-21 2024-09-13T14:22:18.846 INFO:tasks.rados.rados.0.smithi151.stdout:406: finishing write tid 2 to smithi15186934-21 2024-09-13T14:22:18.846 INFO:tasks.rados.rados.0.smithi151.stdout:404: done (3 left) 2024-09-13T14:22:18.846 INFO:tasks.rados.rados.0.smithi151.stdout:405: done (2 left) 2024-09-13T14:22:18.846 INFO:tasks.rados.rados.0.smithi151.stdout:408: write oid 4 current snap is 51 2024-09-13T14:22:18.846 INFO:tasks.rados.rados.0.smithi151.stdout:408: seq_num 192 ranges {426944=556633,1468778=621400,2872744=354200} 2024-09-13T14:22:18.850 INFO:tasks.rados.rados.0.smithi151.stdout:408: writing smithi15186934-4 from 426944 to 983577 tid 1 2024-09-13T14:22:18.857 INFO:tasks.rados.rados.0.smithi151.stdout:408: writing smithi15186934-4 from 1468778 to 2090178 tid 2 2024-09-13T14:22:18.864 INFO:tasks.rados.rados.0.smithi151.stdout:408: writing smithi15186934-4 from 2872744 to 3226944 tid 3 2024-09-13T14:22:18.865 INFO:tasks.rados.rados.0.smithi151.stdout:409: snap_create 2024-09-13T14:22:18.865 INFO:tasks.rados.rados.0.smithi151.stdout:406: finishing write tid 3 to smithi15186934-21 2024-09-13T14:22:18.865 INFO:tasks.rados.rados.0.smithi151.stdout:406: finishing write tid 4 to smithi15186934-21 2024-09-13T14:22:18.865 INFO:tasks.rados.rados.0.smithi151.stdout:406: finishing write tid 5 to smithi15186934-21 2024-09-13T14:22:18.865 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 21 v 50 (ObjNum 191 snap 51 seq_num 191) dirty exists 2024-09-13T14:22:18.865 INFO:tasks.rados.rados.0.smithi151.stdout:406: left oid 21 (ObjNum 191 snap 51 seq_num 191) 2024-09-13T14:22:18.865 INFO:tasks.rados.rados.0.smithi151.stdout:407: finishing rollback tid 0 to smithi15186934-26 2024-09-13T14:22:18.865 INFO:tasks.rados.rados.0.smithi151.stdout:407: finishing rollback tid 1 to smithi15186934-26 2024-09-13T14:22:18.865 INFO:tasks.rados.rados.0.smithi151.stdout:407: finishing rollback tid 2 to smithi15186934-26 2024-09-13T14:22:18.865 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 102 (ObjNum 148 snap 34 seq_num 148) dirty exists 2024-09-13T14:22:18.865 INFO:tasks.rados.rados.0.smithi151.stdout:408: finishing write tid 1 to smithi15186934-4 2024-09-13T14:22:18.865 INFO:tasks.rados.rados.0.smithi151.stdout:408: finishing write tid 2 to smithi15186934-4 2024-09-13T14:22:18.866 INFO:tasks.rados.rados.0.smithi151.stdout:408: finishing write tid 3 to smithi15186934-4 2024-09-13T14:22:18.866 INFO:tasks.rados.rados.0.smithi151.stdout:408: finishing write tid 4 to smithi15186934-4 2024-09-13T14:22:18.866 INFO:tasks.rados.rados.0.smithi151.stdout:408: finishing write tid 5 to smithi15186934-4 2024-09-13T14:22:18.866 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 129 (ObjNum 192 snap 51 seq_num 192) dirty exists 2024-09-13T14:22:18.866 INFO:tasks.rados.rados.0.smithi151.stdout:408: left oid 4 (ObjNum 192 snap 51 seq_num 192) 2024-09-13T14:22:19.033 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:18 smithi151 ceph-mon[29807]: from='client.15378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:19.033 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:18 smithi151 ceph-mon[29807]: from='client.15384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:19.033 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:18 smithi151 ceph-mon[29807]: osdmap e158: 8 total, 8 up, 8 in 2024-09-13T14:22:19.033 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:18 smithi151 ceph-mon[41387]: from='client.15378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:19.034 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:18 smithi151 ceph-mon[41387]: from='client.15384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:19.034 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:18 smithi151 ceph-mon[41387]: osdmap e158: 8 total, 8 up, 8 in 2024-09-13T14:22:19.034 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:18 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:22:18.884+0000 7f6954db3ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T14:22:19.034 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:18 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:22:18.964+0000 7f6954db3ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T14:22:19.034 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:19 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:22:19.032+0000 7f6954db3ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T14:22:19.237 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:18 smithi184 ceph-mon[33394]: from='client.15378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:19.237 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:18 smithi184 ceph-mon[33394]: from='client.15384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:19.237 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:18 smithi184 ceph-mon[33394]: osdmap e158: 8 total, 8 up, 8 in 2024-09-13T14:22:19.357 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:19 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:22:19.114+0000 7f6954db3ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T14:22:19.357 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:19 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:22:19.172+0000 7f6954db3ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T14:22:19.357 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:19 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:22:19.234+0000 7f6954db3ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T14:22:19.793 INFO:tasks.rados.rados.0.smithi151.stdout:406: done (3 left) 2024-09-13T14:22:19.793 INFO:tasks.rados.rados.0.smithi151.stdout:407: done (2 left) 2024-09-13T14:22:19.793 INFO:tasks.rados.rados.0.smithi151.stdout:408: done (1 left) 2024-09-13T14:22:19.793 INFO:tasks.rados.rados.0.smithi151.stdout:409: done (0 left) 2024-09-13T14:22:19.794 INFO:tasks.rados.rados.0.smithi151.stdout:410: snap_create 2024-09-13T14:22:20.211 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:19 smithi184 ceph-mon[33394]: mgrmap e24: x(active, since 3s) 2024-09-13T14:22:20.211 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:19 smithi184 ceph-mon[33394]: pgmap v9: 89 pgs: 89 active+clean; 195 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:20.211 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:19 smithi184 ceph-mon[33394]: osdmap e159: 8 total, 8 up, 8 in 2024-09-13T14:22:20.335 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:19 smithi151 ceph-mon[29807]: mgrmap e24: x(active, since 3s) 2024-09-13T14:22:20.336 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:19 smithi151 ceph-mon[29807]: pgmap v9: 89 pgs: 89 active+clean; 195 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:20.336 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:19 smithi151 ceph-mon[29807]: osdmap e159: 8 total, 8 up, 8 in 2024-09-13T14:22:20.336 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:19 smithi151 ceph-mon[41387]: mgrmap e24: x(active, since 3s) 2024-09-13T14:22:20.336 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:19 smithi151 ceph-mon[41387]: pgmap v9: 89 pgs: 89 active+clean; 195 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:20.336 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:19 smithi151 ceph-mon[41387]: osdmap e159: 8 total, 8 up, 8 in 2024-09-13T14:22:20.336 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:20 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:22:20.055+0000 7f6954db3ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T14:22:20.337 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:20 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:22:20] ENGINE Bus STARTING 2024-09-13T14:22:20.337 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:20 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: CherryPy Checker: 2024-09-13T14:22:20.337 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:20 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: The Application mounted at '' has an empty config. 2024-09-13T14:22:20.337 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:20 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: 2024-09-13T14:22:20.337 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:20 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:22:20] ENGINE Serving on http://:::9283 2024-09-13T14:22:20.337 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:20 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: [13/Sep/2024:14:22:20] ENGINE Bus STARTED 2024-09-13T14:22:20.777 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:22:20.795 INFO:tasks.rados.rados.0.smithi151.stdout:410: done (0 left) 2024-09-13T14:22:20.795 INFO:tasks.rados.rados.0.smithi151.stdout:411: write oid 8 current snap is 53 2024-09-13T14:22:20.795 INFO:tasks.rados.rados.0.smithi151.stdout:411: seq_num 193 ranges {443751=631882,1810907=765058,3067461=176290} 2024-09-13T14:22:20.801 INFO:tasks.rados.rados.0.smithi151.stdout:411: writing smithi15186934-8 from 443751 to 1075633 tid 1 2024-09-13T14:22:20.809 INFO:tasks.rados.rados.0.smithi151.stdout:411: writing smithi15186934-8 from 1810907 to 2575965 tid 2 2024-09-13T14:22:20.813 INFO:tasks.rados.rados.0.smithi151.stdout:411: writing smithi15186934-8 from 3067461 to 3243751 tid 3 2024-09-13T14:22:20.813 INFO:tasks.rados.rados.0.smithi151.stdout:412: snap_create 2024-09-13T14:22:20.813 INFO:tasks.rados.rados.0.smithi151.stdout:411: finishing write tid 1 to smithi15186934-8 2024-09-13T14:22:20.815 INFO:tasks.rados.rados.0.smithi151.stdout:411: finishing write tid 2 to smithi15186934-8 2024-09-13T14:22:20.815 INFO:tasks.rados.rados.0.smithi151.stdout:411: finishing write tid 3 to smithi15186934-8 2024-09-13T14:22:20.815 INFO:tasks.rados.rados.0.smithi151.stdout:411: finishing write tid 4 to smithi15186934-8 2024-09-13T14:22:20.815 INFO:tasks.rados.rados.0.smithi151.stdout:411: finishing write tid 5 to smithi15186934-8 2024-09-13T14:22:20.815 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 128 (ObjNum 193 snap 53 seq_num 193) dirty exists 2024-09-13T14:22:20.815 INFO:tasks.rados.rados.0.smithi151.stdout:411: left oid 8 (ObjNum 193 snap 53 seq_num 193) 2024-09-13T14:22:21.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:20 smithi151 ceph-mon[41387]: Standby manager daemon y started 2024-09-13T14:22:21.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:20 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.151:0/2933669429' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-13T14:22:21.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:20 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.151:0/2933669429' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:22:21.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:20 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.151:0/2933669429' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-13T14:22:21.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:20 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.151:0/2933669429' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:22:21.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:20 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/157554160' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:21.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:20 smithi151 ceph-mon[41387]: osdmap e160: 8 total, 8 up, 8 in 2024-09-13T14:22:21.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:20 smithi151 ceph-mon[29807]: Standby manager daemon y started 2024-09-13T14:22:21.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:20 smithi151 ceph-mon[29807]: from='mgr.? 172.21.15.151:0/2933669429' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-13T14:22:21.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:20 smithi151 ceph-mon[29807]: from='mgr.? 172.21.15.151:0/2933669429' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:22:21.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:20 smithi151 ceph-mon[29807]: from='mgr.? 172.21.15.151:0/2933669429' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-13T14:22:21.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:20 smithi151 ceph-mon[29807]: from='mgr.? 172.21.15.151:0/2933669429' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:22:21.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:20 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/157554160' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:21.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:20 smithi151 ceph-mon[29807]: osdmap e160: 8 total, 8 up, 8 in 2024-09-13T14:22:21.182 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:20 smithi184 ceph-mon[33394]: Standby manager daemon y started 2024-09-13T14:22:21.182 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:20 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.151:0/2933669429' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-13T14:22:21.182 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:20 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.151:0/2933669429' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:22:21.182 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:20 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.151:0/2933669429' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-13T14:22:21.182 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:20 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.151:0/2933669429' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:22:21.182 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:20 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/157554160' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:21.182 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:20 smithi184 ceph-mon[33394]: osdmap e160: 8 total, 8 up, 8 in 2024-09-13T14:22:21.795 INFO:tasks.rados.rados.0.smithi151.stdout:411: done (1 left) 2024-09-13T14:22:21.795 INFO:tasks.rados.rados.0.smithi151.stdout:412: done (0 left) 2024-09-13T14:22:21.795 INFO:tasks.rados.rados.0.smithi151.stdout:413: snap_create 2024-09-13T14:22:22.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:21 smithi184 ceph-mon[33394]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:22.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:21 smithi184 ceph-mon[33394]: mgrmap e25: x(active, since 5s), standbys: y 2024-09-13T14:22:22.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:21 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:22:22.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:21 smithi184 ceph-mon[33394]: from='client.25310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:22.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:21 smithi184 ceph-mon[33394]: pgmap v12: 89 pgs: 89 active+clean; 195 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:22.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:21 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:22.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:21 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:22.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:21 smithi184 ceph-mon[33394]: osdmap e161: 8 total, 8 up, 8 in 2024-09-13T14:22:22.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:21 smithi151 ceph-mon[41387]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:22.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:21 smithi151 ceph-mon[41387]: mgrmap e25: x(active, since 5s), standbys: y 2024-09-13T14:22:22.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:21 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:22:22.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:21 smithi151 ceph-mon[41387]: from='client.25310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:22.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:21 smithi151 ceph-mon[41387]: pgmap v12: 89 pgs: 89 active+clean; 195 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:22.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:21 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:22.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:21 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:22.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:21 smithi151 ceph-mon[41387]: osdmap e161: 8 total, 8 up, 8 in 2024-09-13T14:22:22.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:21 smithi151 ceph-mon[29807]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:22.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:21 smithi151 ceph-mon[29807]: mgrmap e25: x(active, since 5s), standbys: y 2024-09-13T14:22:22.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:21 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:22:22.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:21 smithi151 ceph-mon[29807]: from='client.25310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:22.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:21 smithi151 ceph-mon[29807]: pgmap v12: 89 pgs: 89 active+clean; 195 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:22.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:21 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:22.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:21 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:22.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:21 smithi151 ceph-mon[29807]: osdmap e161: 8 total, 8 up, 8 in 2024-09-13T14:22:22.793 INFO:tasks.rados.rados.0.smithi151.stdout:413: done (0 left) 2024-09-13T14:22:22.793 INFO:tasks.rados.rados.0.smithi151.stdout:414: snap_remove snap 41 2024-09-13T14:22:23.279 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:22 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:22:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:22:23.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:23 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:23.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:23 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:23.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:23 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:23.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:23 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:23.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:23 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:23.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:23 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:23.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:23 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:23.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:23 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:23.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:23 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:23.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:23 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:23.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:23 smithi184 ceph-mon[33394]: Adjusting osd_memory_target on smithi151 to 2480M 2024-09-13T14:22:23.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:23 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:23.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:23 smithi184 ceph-mon[33394]: osdmap e162: 8 total, 8 up, 8 in 2024-09-13T14:22:23.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:23 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:23.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:23 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:23.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:23 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:23.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:23 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:23.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:23 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:23.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:23 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:23.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:23 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:23.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:23 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:23.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:23 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:23.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:23 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:23.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:23 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:23.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:23 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:23.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:23 smithi151 ceph-mon[41387]: Adjusting osd_memory_target on smithi151 to 2480M 2024-09-13T14:22:23.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:23 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:23.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:23 smithi151 ceph-mon[41387]: osdmap e162: 8 total, 8 up, 8 in 2024-09-13T14:22:23.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:23 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:23.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:23 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:23.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:23 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:23.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:23 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:23.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:23 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:23.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:23 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:23.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:23 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:23.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:23 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:23.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:23 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:23.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:23 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:23.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:23 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:23.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:23 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:23.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:23 smithi151 ceph-mon[29807]: Adjusting osd_memory_target on smithi151 to 2480M 2024-09-13T14:22:23.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:23 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:23.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:23 smithi151 ceph-mon[29807]: osdmap e162: 8 total, 8 up, 8 in 2024-09-13T14:22:23.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:23 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:23.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:23 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:23.695 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:22:23.800 INFO:tasks.rados.rados.0.smithi151.stdout:414: done (0 left) 2024-09-13T14:22:23.800 INFO:tasks.rados.rados.0.smithi151.stdout:415: rollback oid 9 current snap is 55 2024-09-13T14:22:23.800 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 9 to 35 2024-09-13T14:22:23.801 INFO:tasks.rados.rados.0.smithi151.stdout:416: rollback oid 11 current snap is 55 2024-09-13T14:22:23.801 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 11 to 49 2024-09-13T14:22:23.801 INFO:tasks.rados.rados.0.smithi151.stdout:417: write oid 25 current snap is 55 2024-09-13T14:22:23.801 INFO:tasks.rados.rados.0.smithi151.stdout:417: seq_num 194 ranges {460558=707131,1869389=508716,3094884=165674} 2024-09-13T14:22:23.804 INFO:tasks.rados.rados.0.smithi151.stdout:417: writing smithi15186934-25 from 460558 to 1167689 tid 1 2024-09-13T14:22:23.811 INFO:tasks.rados.rados.0.smithi151.stdout:417: writing smithi15186934-25 from 1869389 to 2378105 tid 2 2024-09-13T14:22:23.816 INFO:tasks.rados.rados.0.smithi151.stdout:417: writing smithi15186934-25 from 3094884 to 3260558 tid 3 2024-09-13T14:22:23.816 INFO:tasks.rados.rados.0.smithi151.stdout:418: write oid 5 current snap is 55 2024-09-13T14:22:23.816 INFO:tasks.rados.rados.0.smithi151.stdout:418: seq_num 195 ranges {477365=782380,1927871=768727,3122307=155058} 2024-09-13T14:22:23.823 INFO:tasks.rados.rados.0.smithi151.stdout:418: writing smithi15186934-5 from 477365 to 1259745 tid 1 2024-09-13T14:22:23.831 INFO:tasks.rados.rados.0.smithi151.stdout:418: writing smithi15186934-5 from 1927871 to 2696598 tid 2 2024-09-13T14:22:23.834 INFO:tasks.rados.rados.0.smithi151.stdout:418: writing smithi15186934-5 from 3122307 to 3277365 tid 3 2024-09-13T14:22:23.835 INFO:tasks.rados.rados.0.smithi151.stdout:419: read oid 6 snap -1 2024-09-13T14:22:23.835 INFO:tasks.rados.rados.0.smithi151.stdout:419: expect (ObjNum 147 snap 34 seq_num 147) 2024-09-13T14:22:23.835 INFO:tasks.rados.rados.0.smithi151.stdout:420: write oid 35 current snap is 55 2024-09-13T14:22:23.835 INFO:tasks.rados.rados.0.smithi151.stdout:416: finishing rollback tid 0 to smithi15186934-11 2024-09-13T14:22:23.835 INFO:tasks.rados.rados.0.smithi151.stdout:415: finishing rollback tid 0 to smithi15186934-9 2024-09-13T14:22:23.835 INFO:tasks.rados.rados.0.smithi151.stdout:415: finishing rollback tid 1 to smithi15186934-9 2024-09-13T14:22:23.835 INFO:tasks.rados.rados.0.smithi151.stdout:415: finishing rollback tid 2 to smithi15186934-9 2024-09-13T14:22:23.835 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 150 (ObjNum 100 snap 16 seq_num 100) dirty exists 2024-09-13T14:22:23.835 INFO:tasks.rados.rados.0.smithi151.stdout:416: finishing rollback tid 1 to smithi15186934-11 2024-09-13T14:22:23.835 INFO:tasks.rados.rados.0.smithi151.stdout:416: finishing rollback tid 2 to smithi15186934-11 2024-09-13T14:22:23.835 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 11 v 83 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-13T14:22:23.835 INFO:tasks.rados.rados.0.smithi151.stdout:417: finishing write tid 1 to smithi15186934-25 2024-09-13T14:22:23.835 INFO:tasks.rados.rados.0.smithi151.stdout:417: finishing write tid 2 to smithi15186934-25 2024-09-13T14:22:23.836 INFO:tasks.rados.rados.0.smithi151.stdout:417: finishing write tid 3 to smithi15186934-25 2024-09-13T14:22:23.836 INFO:tasks.rados.rados.0.smithi151.stdout:417: finishing write tid 4 to smithi15186934-25 2024-09-13T14:22:23.836 INFO:tasks.rados.rados.0.smithi151.stdout:417: finishing write tid 5 to smithi15186934-25 2024-09-13T14:22:23.836 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 142 (ObjNum 194 snap 55 seq_num 194) dirty exists 2024-09-13T14:22:23.836 INFO:tasks.rados.rados.0.smithi151.stdout:417: left oid 25 (ObjNum 194 snap 55 seq_num 194) 2024-09-13T14:22:23.836 INFO:tasks.rados.rados.0.smithi151.stdout:418: finishing write tid 1 to smithi15186934-5 2024-09-13T14:22:23.836 INFO:tasks.rados.rados.0.smithi151.stdout:420: seq_num 196 ranges {494172=457629,1586353=512385,2749730=544442} 2024-09-13T14:22:23.839 INFO:tasks.rados.rados.0.smithi151.stdout:420: writing smithi15186934-35 from 494172 to 951801 tid 1 2024-09-13T14:22:23.846 INFO:tasks.rados.rados.0.smithi151.stdout:420: writing smithi15186934-35 from 1586353 to 2098738 tid 2 2024-09-13T14:22:23.853 INFO:tasks.rados.rados.0.smithi151.stdout:420: writing smithi15186934-35 from 2749730 to 3294172 tid 3 2024-09-13T14:22:23.853 INFO:tasks.rados.rados.0.smithi151.stdout:415: done (5 left) 2024-09-13T14:22:23.853 INFO:tasks.rados.rados.0.smithi151.stdout:416: done (4 left) 2024-09-13T14:22:23.853 INFO:tasks.rados.rados.0.smithi151.stdout:417: done (3 left) 2024-09-13T14:22:23.853 INFO:tasks.rados.rados.0.smithi151.stdout:421: rollback oid 26 current snap is 55 2024-09-13T14:22:23.853 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 26 to 35 2024-09-13T14:22:23.853 INFO:tasks.rados.rados.0.smithi151.stdout:418: finishing write tid 2 to smithi15186934-5 2024-09-13T14:22:23.854 INFO:tasks.rados.rados.0.smithi151.stdout:418: finishing write tid 3 to smithi15186934-5 2024-09-13T14:22:23.854 INFO:tasks.rados.rados.0.smithi151.stdout:418: finishing write tid 4 to smithi15186934-5 2024-09-13T14:22:23.854 INFO:tasks.rados.rados.0.smithi151.stdout:418: finishing write tid 5 to smithi15186934-5 2024-09-13T14:22:23.854 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 78 (ObjNum 195 snap 55 seq_num 195) dirty exists 2024-09-13T14:22:23.854 INFO:tasks.rados.rados.0.smithi151.stdout:418: left oid 5 (ObjNum 195 snap 55 seq_num 195) 2024-09-13T14:22:23.854 INFO:tasks.rados.rados.0.smithi151.stdout:419: expect (ObjNum 147 snap 34 seq_num 147) 2024-09-13T14:22:24.011 INFO:tasks.rados.rados.0.smithi151.stdout:420: finishing write tid 1 to smithi15186934-35 2024-09-13T14:22:24.011 INFO:tasks.rados.rados.0.smithi151.stdout:420: finishing write tid 2 to smithi15186934-35 2024-09-13T14:22:24.012 INFO:tasks.rados.rados.0.smithi151.stdout:418: done (3 left) 2024-09-13T14:22:24.012 INFO:tasks.rados.rados.0.smithi151.stdout:419: done (2 left) 2024-09-13T14:22:24.012 INFO:tasks.rados.rados.0.smithi151.stdout:422: write oid 11 current snap is 55 2024-09-13T14:22:24.012 INFO:tasks.rados.rados.0.smithi151.stdout:421: finishing rollback tid 0 to smithi15186934-26 2024-09-13T14:22:24.012 INFO:tasks.rados.rados.0.smithi151.stdout:422: seq_num 197 ranges {510979=532878,1528482=772396,3060800=250179} 2024-09-13T14:22:24.017 INFO:tasks.rados.rados.0.smithi151.stdout:422: writing smithi15186934-11 from 510979 to 1043857 tid 1 2024-09-13T14:22:24.024 INFO:tasks.rados.rados.0.smithi151.stdout:422: writing smithi15186934-11 from 1528482 to 2300878 tid 2 2024-09-13T14:22:24.030 INFO:tasks.rados.rados.0.smithi151.stdout:422: writing smithi15186934-11 from 3060800 to 3310979 tid 3 2024-09-13T14:22:24.030 INFO:tasks.rados.rados.0.smithi151.stdout:423: snap_create 2024-09-13T14:22:24.030 INFO:tasks.rados.rados.0.smithi151.stdout:421: finishing rollback tid 1 to smithi15186934-26 2024-09-13T14:22:24.030 INFO:tasks.rados.rados.0.smithi151.stdout:420: finishing write tid 3 to smithi15186934-35 2024-09-13T14:22:24.030 INFO:tasks.rados.rados.0.smithi151.stdout:421: finishing rollback tid 2 to smithi15186934-26 2024-09-13T14:22:24.030 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 108 (ObjNum 148 snap 34 seq_num 148) dirty exists 2024-09-13T14:22:24.030 INFO:tasks.rados.rados.0.smithi151.stdout:420: finishing write tid 4 to smithi15186934-35 2024-09-13T14:22:24.030 INFO:tasks.rados.rados.0.smithi151.stdout:420: finishing write tid 5 to smithi15186934-35 2024-09-13T14:22:24.030 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 35 v 192 (ObjNum 196 snap 55 seq_num 196) dirty exists 2024-09-13T14:22:24.030 INFO:tasks.rados.rados.0.smithi151.stdout:420: left oid 35 (ObjNum 196 snap 55 seq_num 196) 2024-09-13T14:22:24.030 INFO:tasks.rados.rados.0.smithi151.stdout:422: finishing write tid 1 to smithi15186934-11 2024-09-13T14:22:24.031 INFO:tasks.rados.rados.0.smithi151.stdout:422: finishing write tid 2 to smithi15186934-11 2024-09-13T14:22:24.031 INFO:tasks.rados.rados.0.smithi151.stdout:422: finishing write tid 3 to smithi15186934-11 2024-09-13T14:22:24.031 INFO:tasks.rados.rados.0.smithi151.stdout:422: finishing write tid 4 to smithi15186934-11 2024-09-13T14:22:24.031 INFO:tasks.rados.rados.0.smithi151.stdout:422: finishing write tid 5 to smithi15186934-11 2024-09-13T14:22:24.031 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 11 v 87 (ObjNum 197 snap 55 seq_num 197) dirty exists 2024-09-13T14:22:24.032 INFO:tasks.rados.rados.0.smithi151.stdout:422: left oid 11 (ObjNum 197 snap 55 seq_num 197) 2024-09-13T14:22:24.559 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:24 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/3653134914' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:24.559 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:24 smithi184 ceph-mon[33394]: pgmap v15: 89 pgs: 89 active+clean; 175 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 27 op/s 2024-09-13T14:22:24.559 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:24 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:24.559 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:24 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:24.559 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:24 smithi184 ceph-mon[33394]: osdmap e163: 8 total, 8 up, 8 in 2024-09-13T14:22:24.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:24 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/3653134914' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:24.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:24 smithi151 ceph-mon[29807]: pgmap v15: 89 pgs: 89 active+clean; 175 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 27 op/s 2024-09-13T14:22:24.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:24 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:24.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:24 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:24.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:24 smithi151 ceph-mon[29807]: osdmap e163: 8 total, 8 up, 8 in 2024-09-13T14:22:24.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:24 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/3653134914' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:24.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:24 smithi151 ceph-mon[41387]: pgmap v15: 89 pgs: 89 active+clean; 175 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 27 op/s 2024-09-13T14:22:24.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:24 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:24.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:24 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:24.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:24 smithi151 ceph-mon[41387]: osdmap e163: 8 total, 8 up, 8 in 2024-09-13T14:22:24.797 INFO:tasks.rados.rados.0.smithi151.stdout:420: done (3 left) 2024-09-13T14:22:24.798 INFO:tasks.rados.rados.0.smithi151.stdout:421: done (2 left) 2024-09-13T14:22:24.798 INFO:tasks.rados.rados.0.smithi151.stdout:422: done (1 left) 2024-09-13T14:22:24.798 INFO:tasks.rados.rados.0.smithi151.stdout:423: done (0 left) 2024-09-13T14:22:24.798 INFO:tasks.rados.rados.0.smithi151.stdout:424: rollback oid 48 current snap is 56 2024-09-13T14:22:24.798 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 48 to 36 2024-09-13T14:22:24.798 INFO:tasks.rados.rados.0.smithi151.stdout:425: snap_create 2024-09-13T14:22:24.801 INFO:tasks.rados.rados.0.smithi151.stdout:424: finishing rollback tid 0 to smithi15186934-48 2024-09-13T14:22:24.801 INFO:tasks.rados.rados.0.smithi151.stdout:424: finishing rollback tid 1 to smithi15186934-48 2024-09-13T14:22:24.801 INFO:tasks.rados.rados.0.smithi151.stdout:424: finishing rollback tid 2 to smithi15186934-48 2024-09-13T14:22:24.801 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 48 v 146 (ObjNum 128 snap 27 seq_num 128) dirty exists 2024-09-13T14:22:25.449 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:25 smithi184 ceph-mon[33394]: from='client.25322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:25.450 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:25 smithi184 ceph-mon[33394]: from='client.25328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:25.450 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:25 smithi184 ceph-mon[33394]: osdmap e164: 8 total, 8 up, 8 in 2024-09-13T14:22:25.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:25 smithi151 ceph-mon[29807]: from='client.25322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:25.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:25 smithi151 ceph-mon[29807]: from='client.25328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:25.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:25 smithi151 ceph-mon[29807]: osdmap e164: 8 total, 8 up, 8 in 2024-09-13T14:22:25.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:25 smithi151 ceph-mon[41387]: from='client.25322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:25.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:25 smithi151 ceph-mon[41387]: from='client.25328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:25.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:25 smithi151 ceph-mon[41387]: osdmap e164: 8 total, 8 up, 8 in 2024-09-13T14:22:25.802 INFO:tasks.rados.rados.0.smithi151.stdout:424: done (1 left) 2024-09-13T14:22:25.802 INFO:tasks.rados.rados.0.smithi151.stdout:425: done (0 left) 2024-09-13T14:22:25.802 INFO:tasks.rados.rados.0.smithi151.stdout:426: snap_remove snap 56 2024-09-13T14:22:26.476 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:26 smithi184 ceph-mon[33394]: pgmap v18: 89 pgs: 89 active+clean; 175 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 27 op/s 2024-09-13T14:22:26.476 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:26 smithi184 ceph-mon[33394]: osdmap e165: 8 total, 8 up, 8 in 2024-09-13T14:22:26.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:26 smithi151 ceph-mon[29807]: pgmap v18: 89 pgs: 89 active+clean; 175 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 27 op/s 2024-09-13T14:22:26.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:26 smithi151 ceph-mon[29807]: osdmap e165: 8 total, 8 up, 8 in 2024-09-13T14:22:26.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:26 smithi151 ceph-mon[41387]: pgmap v18: 89 pgs: 89 active+clean; 175 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 27 op/s 2024-09-13T14:22:26.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:26 smithi151 ceph-mon[41387]: osdmap e165: 8 total, 8 up, 8 in 2024-09-13T14:22:26.650 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:22:26.802 INFO:tasks.rados.rados.0.smithi151.stdout:426: done (0 left) 2024-09-13T14:22:26.803 INFO:tasks.rados.rados.0.smithi151.stdout:427: delete oid 27 current snap is 57 2024-09-13T14:22:26.807 INFO:tasks.rados.rados.0.smithi151.stdout:427: done (0 left) 2024-09-13T14:22:26.807 INFO:tasks.rados.rados.0.smithi151.stdout:428: rollback oid 33 current snap is 57 2024-09-13T14:22:26.808 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 33 to 51 2024-09-13T14:22:26.808 INFO:tasks.rados.rados.0.smithi151.stdout:429: read oid 45 snap -1 2024-09-13T14:22:26.808 INFO:tasks.rados.rados.0.smithi151.stdout:429: expect (ObjNum 189 snap 51 seq_num 189) 2024-09-13T14:22:26.808 INFO:tasks.rados.rados.0.smithi151.stdout:430: read oid 26 snap -1 2024-09-13T14:22:26.808 INFO:tasks.rados.rados.0.smithi151.stdout:430: expect (ObjNum 148 snap 34 seq_num 148) 2024-09-13T14:22:26.808 INFO:tasks.rados.rados.0.smithi151.stdout:431: write oid 43 current snap is 57 2024-09-13T14:22:26.808 INFO:tasks.rados.rados.0.smithi151.stdout:431: seq_num 199 ranges {544593=799729,1761799=776065,2948352=396241} 2024-09-13T14:22:26.816 INFO:tasks.rados.rados.0.smithi151.stdout:431: writing smithi15186934-43 from 544593 to 1344322 tid 1 2024-09-13T14:22:26.823 INFO:tasks.rados.rados.0.smithi151.stdout:431: writing smithi15186934-43 from 1761799 to 2537864 tid 2 2024-09-13T14:22:26.828 INFO:tasks.rados.rados.0.smithi151.stdout:431: writing smithi15186934-43 from 2948352 to 3344593 tid 3 2024-09-13T14:22:26.828 INFO:tasks.rados.rados.0.smithi151.stdout:432: read oid 47 snap -1 2024-09-13T14:22:26.828 INFO:tasks.rados.rados.0.smithi151.stdout:432: expect (ObjNum 155 snap 38 seq_num 155) 2024-09-13T14:22:26.828 INFO:tasks.rados.rados.0.smithi151.stdout:433: read oid 31 snap -1 2024-09-13T14:22:26.828 INFO:tasks.rados.rados.0.smithi151.stdout:433: expect (ObjNum 138 snap 30 seq_num 138) 2024-09-13T14:22:26.828 INFO:tasks.rados.rados.0.smithi151.stdout:434: write oid 30 current snap is 57 2024-09-13T14:22:26.828 INFO:tasks.rados.rados.0.smithi151.stdout:434: seq_num 200 ranges {561400=474978,1820281=519723,2859422=457579} 2024-09-13T14:22:26.834 INFO:tasks.rados.rados.0.smithi151.stdout:434: writing smithi15186934-30 from 561400 to 1036378 tid 1 2024-09-13T14:22:26.841 INFO:tasks.rados.rados.0.smithi151.stdout:434: writing smithi15186934-30 from 1820281 to 2340004 tid 2 2024-09-13T14:22:26.847 INFO:tasks.rados.rados.0.smithi151.stdout:434: writing smithi15186934-30 from 2859422 to 3317001 tid 3 2024-09-13T14:22:26.847 INFO:tasks.rados.rados.0.smithi151.stdout:435: write oid 21 current snap is 57 2024-09-13T14:22:26.847 INFO:tasks.rados.rados.0.smithi151.stdout:435: seq_num 201 ranges {578207=550227,1762410=779734,3286845=91362} 2024-09-13T14:22:26.853 INFO:tasks.rados.rados.0.smithi151.stdout:435: writing smithi15186934-21 from 578207 to 1128434 tid 1 2024-09-13T14:22:26.861 INFO:tasks.rados.rados.0.smithi151.stdout:435: writing smithi15186934-21 from 1762410 to 2542144 tid 2 2024-09-13T14:22:26.866 INFO:tasks.rados.rados.0.smithi151.stdout:435: writing smithi15186934-21 from 3286845 to 3378207 tid 3 2024-09-13T14:22:26.866 INFO:tasks.rados.rados.0.smithi151.stdout:436: rollback oid 1 current snap is 57 2024-09-13T14:22:26.866 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 1 to 43 2024-09-13T14:22:26.866 INFO:tasks.rados.rados.0.smithi151.stdout:430: expect (ObjNum 148 snap 34 seq_num 148) 2024-09-13T14:22:26.988 INFO:tasks.rados.rados.0.smithi151.stdout:428: finishing rollback tid 0 to smithi15186934-33 2024-09-13T14:22:26.989 INFO:tasks.rados.rados.0.smithi151.stdout:428: finishing rollback tid 1 to smithi15186934-33 2024-09-13T14:22:26.989 INFO:tasks.rados.rados.0.smithi151.stdout:428: finishing rollback tid 2 to smithi15186934-33 2024-09-13T14:22:26.989 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 33 v 82 (ObjNum 176 snap 47 seq_num 176) dirty exists 2024-09-13T14:22:26.989 INFO:tasks.rados.rados.0.smithi151.stdout:429: expect (ObjNum 189 snap 51 seq_num 189) 2024-09-13T14:22:27.187 INFO:tasks.rados.rados.0.smithi151.stdout:431: finishing write tid 1 to smithi15186934-43 2024-09-13T14:22:27.187 INFO:tasks.rados.rados.0.smithi151.stdout:431: finishing write tid 2 to smithi15186934-43 2024-09-13T14:22:27.187 INFO:tasks.rados.rados.0.smithi151.stdout:428: done (8 left) 2024-09-13T14:22:27.187 INFO:tasks.rados.rados.0.smithi151.stdout:429: done (7 left) 2024-09-13T14:22:27.187 INFO:tasks.rados.rados.0.smithi151.stdout:430: done (6 left) 2024-09-13T14:22:27.187 INFO:tasks.rados.rados.0.smithi151.stdout:437: snap_remove snap 42 2024-09-13T14:22:27.379 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/3648530226' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:27.379 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:27.379 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:27.379 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:27.379 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:27.380 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:27.380 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:27.380 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:27.380 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:27.380 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:27.380 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:27.380 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[29807]: Adjusting osd_memory_target on smithi184 to 2480M 2024-09-13T14:22:27.380 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:27.380 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:22:27.380 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:22:27.380 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[29807]: Updating smithi151:/etc/ceph/ceph.conf 2024-09-13T14:22:27.380 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[29807]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-13T14:22:27.380 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[29807]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:27.381 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[29807]: osdmap e166: 8 total, 8 up, 8 in 2024-09-13T14:22:27.381 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/3648530226' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:27.381 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:27.381 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:27.381 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:27.381 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:27.381 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:27.381 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:27.381 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:27.381 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:27.381 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:27.381 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:27.382 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[41387]: Adjusting osd_memory_target on smithi184 to 2480M 2024-09-13T14:22:27.382 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:27.382 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:22:27.382 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:22:27.382 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[41387]: Updating smithi151:/etc/ceph/ceph.conf 2024-09-13T14:22:27.382 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[41387]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-13T14:22:27.382 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[41387]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:27.382 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:27 smithi151 ceph-mon[41387]: osdmap e166: 8 total, 8 up, 8 in 2024-09-13T14:22:27.603 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:27 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:22:27] "GET /metrics HTTP/1.1" 200 33449 "" "Prometheus/2.43.0" 2024-09-13T14:22:27.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:27 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/3648530226' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:27.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:27 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:27.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:27 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:27.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:27 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:27.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:27 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:27.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:27 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:27.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:27 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:27.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:27 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:27.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:27 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:27.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:27 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:27.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:27 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:27.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:27 smithi184 ceph-mon[33394]: Adjusting osd_memory_target on smithi184 to 2480M 2024-09-13T14:22:27.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:27 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:27.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:27 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:22:27.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:27 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:22:27.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:27 smithi184 ceph-mon[33394]: Updating smithi151:/etc/ceph/ceph.conf 2024-09-13T14:22:27.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:27 smithi184 ceph-mon[33394]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-13T14:22:27.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:27 smithi184 ceph-mon[33394]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:27.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:27 smithi184 ceph-mon[33394]: osdmap e166: 8 total, 8 up, 8 in 2024-09-13T14:22:27.804 INFO:tasks.rados.rados.0.smithi151.stdout:437: done (6 left) 2024-09-13T14:22:27.805 INFO:tasks.rados.rados.0.smithi151.stdout:438: write oid 37 current snap is 57 2024-09-13T14:22:27.805 INFO:tasks.rados.rados.0.smithi151.stdout:438: seq_num 202 ranges {595014=625476,1820892=523392,2797915=596476} 2024-09-13T14:22:27.813 INFO:tasks.rados.rados.0.smithi151.stdout:438: writing smithi15186934-37 from 595014 to 1220490 tid 1 2024-09-13T14:22:27.820 INFO:tasks.rados.rados.0.smithi151.stdout:438: writing smithi15186934-37 from 1820892 to 2344284 tid 2 2024-09-13T14:22:27.827 INFO:tasks.rados.rados.0.smithi151.stdout:438: writing smithi15186934-37 from 2797915 to 3394391 tid 3 2024-09-13T14:22:27.828 INFO:tasks.rados.rados.0.smithi151.stdout:431: finishing write tid 3 to smithi15186934-43 2024-09-13T14:22:27.828 INFO:tasks.rados.rados.0.smithi151.stdout:431: finishing write tid 4 to smithi15186934-43 2024-09-13T14:22:27.828 INFO:tasks.rados.rados.0.smithi151.stdout:431: finishing write tid 5 to smithi15186934-43 2024-09-13T14:22:27.829 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 43 v 197 (ObjNum 199 snap 57 seq_num 199) dirty exists 2024-09-13T14:22:27.829 INFO:tasks.rados.rados.0.smithi151.stdout:431: left oid 43 (ObjNum 199 snap 57 seq_num 199) 2024-09-13T14:22:27.829 INFO:tasks.rados.rados.0.smithi151.stdout:433: expect (ObjNum 138 snap 30 seq_num 138) 2024-09-13T14:22:27.932 INFO:tasks.rados.rados.0.smithi151.stdout:432: expect (ObjNum 155 snap 38 seq_num 155) 2024-09-13T14:22:28.050 INFO:tasks.rados.rados.0.smithi151.stdout:434: finishing write tid 1 to smithi15186934-30 2024-09-13T14:22:28.050 INFO:tasks.rados.rados.0.smithi151.stdout:434: finishing write tid 2 to smithi15186934-30 2024-09-13T14:22:28.050 INFO:tasks.rados.rados.0.smithi151.stdout:434: finishing write tid 3 to smithi15186934-30 2024-09-13T14:22:28.050 INFO:tasks.rados.rados.0.smithi151.stdout:434: finishing write tid 4 to smithi15186934-30 2024-09-13T14:22:28.050 INFO:tasks.rados.rados.0.smithi151.stdout:434: finishing write tid 5 to smithi15186934-30 2024-09-13T14:22:28.050 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 30 v 146 (ObjNum 200 snap 57 seq_num 200) dirty exists 2024-09-13T14:22:28.050 INFO:tasks.rados.rados.0.smithi151.stdout:434: left oid 30 (ObjNum 200 snap 57 seq_num 200) 2024-09-13T14:22:28.050 INFO:tasks.rados.rados.0.smithi151.stdout:435: finishing write tid 1 to smithi15186934-21 2024-09-13T14:22:28.050 INFO:tasks.rados.rados.0.smithi151.stdout:431: done (6 left) 2024-09-13T14:22:28.050 INFO:tasks.rados.rados.0.smithi151.stdout:432: done (5 left) 2024-09-13T14:22:28.050 INFO:tasks.rados.rados.0.smithi151.stdout:433: done (4 left) 2024-09-13T14:22:28.051 INFO:tasks.rados.rados.0.smithi151.stdout:434: done (3 left) 2024-09-13T14:22:28.051 INFO:tasks.rados.rados.0.smithi151.stdout:439: write oid 10 current snap is 57 2024-09-13T14:22:28.051 INFO:tasks.rados.rados.0.smithi151.stdout:435: finishing write tid 2 to smithi15186934-21 2024-09-13T14:22:28.051 INFO:tasks.rados.rados.0.smithi151.stdout:435: finishing write tid 3 to smithi15186934-21 2024-09-13T14:22:28.051 INFO:tasks.rados.rados.0.smithi151.stdout:439: seq_num 203 ranges {611821=700725,1879374=783403,3341691=70130} 2024-09-13T14:22:28.056 INFO:tasks.rados.rados.0.smithi151.stdout:439: writing smithi15186934-10 from 611821 to 1312546 tid 1 2024-09-13T14:22:28.064 INFO:tasks.rados.rados.0.smithi151.stdout:439: writing smithi15186934-10 from 1879374 to 2662777 tid 2 2024-09-13T14:22:28.069 INFO:tasks.rados.rados.0.smithi151.stdout:439: writing smithi15186934-10 from 3341691 to 3411821 tid 3 2024-09-13T14:22:28.069 INFO:tasks.rados.rados.0.smithi151.stdout:440: snap_create 2024-09-13T14:22:28.069 INFO:tasks.rados.rados.0.smithi151.stdout:436: finishing rollback tid 0 to smithi15186934-1 2024-09-13T14:22:28.069 INFO:tasks.rados.rados.0.smithi151.stdout:435: finishing write tid 4 to smithi15186934-21 2024-09-13T14:22:28.069 INFO:tasks.rados.rados.0.smithi151.stdout:436: finishing rollback tid 1 to smithi15186934-1 2024-09-13T14:22:28.070 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 1 v 131 (ObjNum 33 snap 0 seq_num 3681424361) dirty dne 2024-09-13T14:22:28.070 INFO:tasks.rados.rados.0.smithi151.stdout:435: finishing write tid 5 to smithi15186934-21 2024-09-13T14:22:28.070 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 21 v 55 (ObjNum 201 snap 57 seq_num 201) dirty exists 2024-09-13T14:22:28.070 INFO:tasks.rados.rados.0.smithi151.stdout:435: left oid 21 (ObjNum 201 snap 57 seq_num 201) 2024-09-13T14:22:28.070 INFO:tasks.rados.rados.0.smithi151.stdout:438: finishing write tid 1 to smithi15186934-37 2024-09-13T14:22:28.070 INFO:tasks.rados.rados.0.smithi151.stdout:438: finishing write tid 2 to smithi15186934-37 2024-09-13T14:22:28.070 INFO:tasks.rados.rados.0.smithi151.stdout:438: finishing write tid 3 to smithi15186934-37 2024-09-13T14:22:28.070 INFO:tasks.rados.rados.0.smithi151.stdout:438: finishing write tid 4 to smithi15186934-37 2024-09-13T14:22:28.070 INFO:tasks.rados.rados.0.smithi151.stdout:438: finishing write tid 5 to smithi15186934-37 2024-09-13T14:22:28.070 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 37 v 203 (ObjNum 202 snap 57 seq_num 202) dirty exists 2024-09-13T14:22:28.070 INFO:tasks.rados.rados.0.smithi151.stdout:438: left oid 37 (ObjNum 202 snap 57 seq_num 202) 2024-09-13T14:22:28.070 INFO:tasks.rados.rados.0.smithi151.stdout:439: finishing write tid 1 to smithi15186934-10 2024-09-13T14:22:28.070 INFO:tasks.rados.rados.0.smithi151.stdout:439: finishing write tid 2 to smithi15186934-10 2024-09-13T14:22:28.071 INFO:tasks.rados.rados.0.smithi151.stdout:439: finishing write tid 3 to smithi15186934-10 2024-09-13T14:22:28.071 INFO:tasks.rados.rados.0.smithi151.stdout:439: finishing write tid 4 to smithi15186934-10 2024-09-13T14:22:28.071 INFO:tasks.rados.rados.0.smithi151.stdout:439: finishing write tid 5 to smithi15186934-10 2024-09-13T14:22:28.071 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 115 (ObjNum 203 snap 57 seq_num 203) dirty exists 2024-09-13T14:22:28.071 INFO:tasks.rados.rados.0.smithi151.stdout:439: left oid 10 (ObjNum 203 snap 57 seq_num 203) 2024-09-13T14:22:28.811 INFO:tasks.rados.rados.0.smithi151.stdout:435: done (4 left) 2024-09-13T14:22:28.812 INFO:tasks.rados.rados.0.smithi151.stdout:436: done (3 left) 2024-09-13T14:22:28.812 INFO:tasks.rados.rados.0.smithi151.stdout:438: done (2 left) 2024-09-13T14:22:28.812 INFO:tasks.rados.rados.0.smithi151.stdout:439: done (1 left) 2024-09-13T14:22:28.812 INFO:tasks.rados.rados.0.smithi151.stdout:440: done (0 left) 2024-09-13T14:22:28.812 INFO:tasks.rados.rados.0.smithi151.stdout:441: snap_remove snap 46 2024-09-13T14:22:28.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:28 smithi184 ceph-mon[33394]: Updating smithi184:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:22:28.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:28 smithi184 ceph-mon[33394]: Updating smithi151:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:22:28.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:28 smithi184 ceph-mon[33394]: from='client.15444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:28.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:28 smithi184 ceph-mon[33394]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:22:28.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:28 smithi184 ceph-mon[33394]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:22:28.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:28 smithi184 ceph-mon[33394]: Updating smithi184:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:22:28.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:28 smithi184 ceph-mon[33394]: Updating smithi151:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:22:28.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:28 smithi184 ceph-mon[33394]: pgmap v21: 89 pgs: 89 active+clean; 189 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 2.6 MiB/s wr, 24 op/s 2024-09-13T14:22:28.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:28 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:28.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:28 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:28.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:28 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:28.856 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:28 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:28.856 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:28 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:28.856 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:28 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:28.856 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:28 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:28.856 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:28 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:28.857 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:28 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:28.857 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:28 smithi184 ceph-mon[33394]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-13T14:22:28.857 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:28 smithi184 ceph-mon[33394]: Deploying daemon alertmanager.a on smithi151 2024-09-13T14:22:28.857 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:28 smithi184 ceph-mon[33394]: osdmap e167: 8 total, 8 up, 8 in 2024-09-13T14:22:28.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[29807]: Updating smithi184:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:22:28.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[29807]: Updating smithi151:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:22:28.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[29807]: from='client.15444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:28.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[29807]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:22:28.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[29807]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:22:28.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[29807]: Updating smithi184:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:22:28.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[29807]: Updating smithi151:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:22:28.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[29807]: pgmap v21: 89 pgs: 89 active+clean; 189 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 2.6 MiB/s wr, 24 op/s 2024-09-13T14:22:28.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:28.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:28.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:28.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:28.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:28.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:28.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:28.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:28.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:28.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[29807]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-13T14:22:28.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[29807]: Deploying daemon alertmanager.a on smithi151 2024-09-13T14:22:28.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[29807]: osdmap e167: 8 total, 8 up, 8 in 2024-09-13T14:22:28.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[41387]: Updating smithi184:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:22:28.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[41387]: Updating smithi151:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:22:28.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[41387]: from='client.15444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:28.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[41387]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:22:28.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[41387]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:22:28.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[41387]: Updating smithi184:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:22:28.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[41387]: Updating smithi151:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:22:28.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[41387]: pgmap v21: 89 pgs: 89 active+clean; 189 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 2.6 MiB/s wr, 24 op/s 2024-09-13T14:22:28.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:28.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:28.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:28.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:28.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:28.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:28.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:28.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:28.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:28.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[41387]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-13T14:22:28.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[41387]: Deploying daemon alertmanager.a on smithi151 2024-09-13T14:22:28.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:28 smithi151 ceph-mon[41387]: osdmap e167: 8 total, 8 up, 8 in 2024-09-13T14:22:29.590 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:22:29.826 INFO:tasks.rados.rados.0.smithi151.stdout:441: done (0 left) 2024-09-13T14:22:29.826 INFO:tasks.rados.rados.0.smithi151.stdout:442: read oid 36 snap -1 2024-09-13T14:22:29.826 INFO:tasks.rados.rados.0.smithi151.stdout:442: expect deleted 2024-09-13T14:22:29.826 INFO:tasks.rados.rados.0.smithi151.stdout:443: read oid 35 snap 54 2024-09-13T14:22:29.826 INFO:tasks.rados.rados.0.smithi151.stdout:443: expect (ObjNum 187 snap 49 seq_num 187) 2024-09-13T14:22:29.826 INFO:tasks.rados.rados.0.smithi151.stdout:444: snap_remove snap 51 2024-09-13T14:22:29.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:29 smithi151 ceph-mon[29807]: osdmap e168: 8 total, 8 up, 8 in 2024-09-13T14:22:29.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:29 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1076054437' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:29.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:29 smithi151 ceph-mon[29807]: pgmap v24: 89 pgs: 89 active+clean; 189 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 2.6 MiB/s wr, 24 op/s 2024-09-13T14:22:29.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:29 smithi151 ceph-mon[29807]: from='client.15456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:29.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:29 smithi151 ceph-mon[41387]: osdmap e168: 8 total, 8 up, 8 in 2024-09-13T14:22:29.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:29 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/1076054437' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:29.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:29 smithi151 ceph-mon[41387]: pgmap v24: 89 pgs: 89 active+clean; 189 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 2.6 MiB/s wr, 24 op/s 2024-09-13T14:22:29.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:29 smithi151 ceph-mon[41387]: from='client.15456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:30.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:29 smithi184 ceph-mon[33394]: osdmap e168: 8 total, 8 up, 8 in 2024-09-13T14:22:30.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:29 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/1076054437' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:30.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:29 smithi184 ceph-mon[33394]: pgmap v24: 89 pgs: 89 active+clean; 189 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 2.6 MiB/s wr, 24 op/s 2024-09-13T14:22:30.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:29 smithi184 ceph-mon[33394]: from='client.15456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:30.821 INFO:tasks.rados.rados.0.smithi151.stdout:444: done (2 left) 2024-09-13T14:22:30.821 INFO:tasks.rados.rados.0.smithi151.stdout:445: rollback oid 37 current snap is 58 2024-09-13T14:22:30.821 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 37 to 53 2024-09-13T14:22:30.822 INFO:tasks.rados.rados.0.smithi151.stdout:443: expect (ObjNum 187 snap 49 seq_num 187) 2024-09-13T14:22:30.931 INFO:tasks.rados.rados.0.smithi151.stdout:445: finishing rollback tid 0 to smithi15186934-37 2024-09-13T14:22:30.931 INFO:tasks.rados.rados.0.smithi151.stdout:445: finishing rollback tid 1 to smithi15186934-37 2024-09-13T14:22:30.932 INFO:tasks.rados.rados.0.smithi151.stdout:445: finishing rollback tid 2 to smithi15186934-37 2024-09-13T14:22:30.932 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 37 v 211 (ObjNum 179 snap 47 seq_num 179) dirty exists 2024-09-13T14:22:30.932 INFO:tasks.rados.rados.0.smithi151.stdout:442: done (2 left) 2024-09-13T14:22:30.932 INFO:tasks.rados.rados.0.smithi151.stdout:443: done (1 left) 2024-09-13T14:22:30.932 INFO:tasks.rados.rados.0.smithi151.stdout:445: done (0 left) 2024-09-13T14:22:30.932 INFO:tasks.rados.rados.0.smithi151.stdout:446: snap_remove snap 50 2024-09-13T14:22:31.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:30 smithi184 ceph-mon[33394]: osdmap e169: 8 total, 8 up, 8 in 2024-09-13T14:22:31.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:30 smithi184 ceph-mon[33394]: from='client.15462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:31.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:30 smithi151 ceph-mon[29807]: osdmap e169: 8 total, 8 up, 8 in 2024-09-13T14:22:31.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:30 smithi151 ceph-mon[29807]: from='client.15462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:31.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:30 smithi151 ceph-mon[41387]: osdmap e169: 8 total, 8 up, 8 in 2024-09-13T14:22:31.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:30 smithi151 ceph-mon[41387]: from='client.15462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:31.835 INFO:tasks.rados.rados.0.smithi151.stdout:446: done (0 left) 2024-09-13T14:22:31.835 INFO:tasks.rados.rados.0.smithi151.stdout:447: write oid 49 current snap is 58 2024-09-13T14:22:31.835 INFO:tasks.rados.rados.0.smithi151.stdout:447: seq_num 204 ranges {628628=775974,1937856=527061,3252761=175867} 2024-09-13T14:22:31.843 INFO:tasks.rados.rados.0.smithi151.stdout:447: writing smithi15186934-49 from 628628 to 1404602 tid 1 2024-09-13T14:22:31.849 INFO:tasks.rados.rados.0.smithi151.stdout:447: writing smithi15186934-49 from 1937856 to 2464917 tid 2 2024-09-13T14:22:31.855 INFO:tasks.rados.rados.0.smithi151.stdout:447: writing smithi15186934-49 from 3252761 to 3428628 tid 3 2024-09-13T14:22:31.855 INFO:tasks.rados.rados.0.smithi151.stdout:448: read oid 34 snap 43 2024-09-13T14:22:31.855 INFO:tasks.rados.rados.0.smithi151.stdout:448: expect (ObjNum 112 snap 18 seq_num 112) 2024-09-13T14:22:31.855 INFO:tasks.rados.rados.0.smithi151.stdout:447: finishing write tid 1 to smithi15186934-49 2024-09-13T14:22:31.855 INFO:tasks.rados.rados.0.smithi151.stdout:447: finishing write tid 2 to smithi15186934-49 2024-09-13T14:22:31.855 INFO:tasks.rados.rados.0.smithi151.stdout:449: read oid 8 snap -1 2024-09-13T14:22:31.856 INFO:tasks.rados.rados.0.smithi151.stdout:449: expect (ObjNum 193 snap 53 seq_num 193) 2024-09-13T14:22:31.856 INFO:tasks.rados.rados.0.smithi151.stdout:450: write oid 42 current snap is 58 2024-09-13T14:22:31.856 INFO:tasks.rados.rados.0.smithi151.stdout:450: seq_num 205 ranges {645435=451223,1879985=787072,3280184=165251} 2024-09-13T14:22:31.861 INFO:tasks.rados.rados.0.smithi151.stdout:450: writing smithi15186934-42 from 645435 to 1096658 tid 1 2024-09-13T14:22:31.870 INFO:tasks.rados.rados.0.smithi151.stdout:450: writing smithi15186934-42 from 1879985 to 2667057 tid 2 2024-09-13T14:22:31.874 INFO:tasks.rados.rados.0.smithi151.stdout:450: writing smithi15186934-42 from 3280184 to 3445435 tid 3 2024-09-13T14:22:31.874 INFO:tasks.rados.rados.0.smithi151.stdout:451: snap_create 2024-09-13T14:22:31.875 INFO:tasks.rados.rados.0.smithi151.stdout:447: finishing write tid 3 to smithi15186934-49 2024-09-13T14:22:31.875 INFO:tasks.rados.rados.0.smithi151.stdout:447: finishing write tid 4 to smithi15186934-49 2024-09-13T14:22:31.875 INFO:tasks.rados.rados.0.smithi151.stdout:447: finishing write tid 5 to smithi15186934-49 2024-09-13T14:22:31.875 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 49 v 220 (ObjNum 204 snap 58 seq_num 204) dirty exists 2024-09-13T14:22:31.875 INFO:tasks.rados.rados.0.smithi151.stdout:447: left oid 49 (ObjNum 204 snap 58 seq_num 204) 2024-09-13T14:22:31.875 INFO:tasks.rados.rados.0.smithi151.stdout:448: expect (ObjNum 112 snap 18 seq_num 112) 2024-09-13T14:22:32.014 INFO:tasks.rados.rados.0.smithi151.stdout:449: expect (ObjNum 193 snap 53 seq_num 193) 2024-09-13T14:22:32.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:31 smithi184 ceph-mon[33394]: osdmap e170: 8 total, 8 up, 8 in 2024-09-13T14:22:32.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:31 smithi184 ceph-mon[33394]: pgmap v27: 89 pgs: 89 active+clean; 189 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:32.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:31 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:22:32.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:31 smithi151 ceph-mon[41387]: osdmap e170: 8 total, 8 up, 8 in 2024-09-13T14:22:32.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:31 smithi151 ceph-mon[41387]: pgmap v27: 89 pgs: 89 active+clean; 189 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:32.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:31 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:22:32.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:31 smithi151 ceph-mon[29807]: osdmap e170: 8 total, 8 up, 8 in 2024-09-13T14:22:32.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:31 smithi151 ceph-mon[29807]: pgmap v27: 89 pgs: 89 active+clean; 189 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:32.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:31 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:22:32.122 INFO:tasks.rados.rados.0.smithi151.stdout:450: finishing write tid 1 to smithi15186934-42 2024-09-13T14:22:32.123 INFO:tasks.rados.rados.0.smithi151.stdout:450: finishing write tid 2 to smithi15186934-42 2024-09-13T14:22:32.123 INFO:tasks.rados.rados.0.smithi151.stdout:450: finishing write tid 3 to smithi15186934-42 2024-09-13T14:22:32.123 INFO:tasks.rados.rados.0.smithi151.stdout:450: finishing write tid 4 to smithi15186934-42 2024-09-13T14:22:32.123 INFO:tasks.rados.rados.0.smithi151.stdout:450: finishing write tid 5 to smithi15186934-42 2024-09-13T14:22:32.123 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 158 (ObjNum 205 snap 58 seq_num 205) dirty exists 2024-09-13T14:22:32.123 INFO:tasks.rados.rados.0.smithi151.stdout:450: left oid 42 (ObjNum 205 snap 58 seq_num 205) 2024-09-13T14:22:32.523 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:22:32.842 INFO:tasks.rados.rados.0.smithi151.stdout:447: done (4 left) 2024-09-13T14:22:32.842 INFO:tasks.rados.rados.0.smithi151.stdout:448: done (3 left) 2024-09-13T14:22:32.842 INFO:tasks.rados.rados.0.smithi151.stdout:449: done (2 left) 2024-09-13T14:22:32.842 INFO:tasks.rados.rados.0.smithi151.stdout:450: done (1 left) 2024-09-13T14:22:32.842 INFO:tasks.rados.rados.0.smithi151.stdout:451: done (0 left) 2024-09-13T14:22:32.842 INFO:tasks.rados.rados.0.smithi151.stdout:452: snap_create 2024-09-13T14:22:32.932 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:32 smithi151 ceph-mon[41387]: osdmap e171: 8 total, 8 up, 8 in 2024-09-13T14:22:32.932 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:32 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/3311127280' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:32.932 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:32 smithi151 ceph-mon[41387]: from='client.15474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:32.932 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:32 smithi151 ceph-mon[29807]: osdmap e171: 8 total, 8 up, 8 in 2024-09-13T14:22:32.932 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:32 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/3311127280' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:32.932 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:32 smithi151 ceph-mon[29807]: from='client.15474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:33.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:32 smithi184 ceph-mon[33394]: osdmap e171: 8 total, 8 up, 8 in 2024-09-13T14:22:33.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:32 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/3311127280' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:33.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:32 smithi184 ceph-mon[33394]: from='client.15474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:33.253 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:32 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:22:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:22:33.845 INFO:tasks.rados.rados.0.smithi151.stdout:452: done (0 left) 2024-09-13T14:22:33.845 INFO:tasks.rados.rados.0.smithi151.stdout:453: delete oid 44 current snap is 60 2024-09-13T14:22:33.848 INFO:tasks.rados.rados.0.smithi151.stdout:453: done (0 left) 2024-09-13T14:22:33.848 INFO:tasks.rados.rados.0.smithi151.stdout:454: rollback oid 27 current snap is 60 2024-09-13T14:22:33.848 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 27 to 35 2024-09-13T14:22:33.848 INFO:tasks.rados.rados.0.smithi151.stdout:455: rollback oid 7 current snap is 60 2024-09-13T14:22:33.848 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 7 to 57 2024-09-13T14:22:33.849 INFO:tasks.rados.rados.0.smithi151.stdout:456: read oid 25 snap -1 2024-09-13T14:22:33.849 INFO:tasks.rados.rados.0.smithi151.stdout:456: expect (ObjNum 194 snap 55 seq_num 194) 2024-09-13T14:22:33.849 INFO:tasks.rados.rados.0.smithi151.stdout:457: read oid 48 snap -1 2024-09-13T14:22:33.849 INFO:tasks.rados.rados.0.smithi151.stdout:457: expect (ObjNum 128 snap 27 seq_num 128) 2024-09-13T14:22:33.849 INFO:tasks.rados.rados.0.smithi151.stdout:458: write oid 8 current snap is 60 2024-09-13T14:22:33.849 INFO:tasks.rados.rados.0.smithi151.stdout:458: seq_num 207 ranges {679049=718074,2113302=674388,3335030=144019} 2024-09-13T14:22:33.856 INFO:tasks.rados.rados.0.smithi151.stdout:458: writing smithi15186934-8 from 679049 to 1397123 tid 1 2024-09-13T14:22:33.864 INFO:tasks.rados.rados.0.smithi151.stdout:458: writing smithi15186934-8 from 2113302 to 2787690 tid 2 2024-09-13T14:22:33.868 INFO:tasks.rados.rados.0.smithi151.stdout:458: writing smithi15186934-8 from 3335030 to 3479049 tid 3 2024-09-13T14:22:33.868 INFO:tasks.rados.rados.0.smithi151.stdout:459: snap_remove snap 55 2024-09-13T14:22:34.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:33 smithi184 ceph-mon[33394]: from='client.25382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:34.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:33 smithi184 ceph-mon[33394]: osdmap e172: 8 total, 8 up, 8 in 2024-09-13T14:22:34.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:33 smithi184 ceph-mon[33394]: pgmap v30: 89 pgs: 89 active+clean; 197 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-13T14:22:34.106 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:33 smithi151 ceph-mon[29807]: from='client.25382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:34.107 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:33 smithi151 ceph-mon[29807]: osdmap e172: 8 total, 8 up, 8 in 2024-09-13T14:22:34.107 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:33 smithi151 ceph-mon[29807]: pgmap v30: 89 pgs: 89 active+clean; 197 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-13T14:22:34.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:33 smithi151 ceph-mon[41387]: from='client.25382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:34.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:33 smithi151 ceph-mon[41387]: osdmap e172: 8 total, 8 up, 8 in 2024-09-13T14:22:34.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:33 smithi151 ceph-mon[41387]: pgmap v30: 89 pgs: 89 active+clean; 197 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-13T14:22:34.868 INFO:tasks.rados.rados.0.smithi151.stdout:455: finishing rollback tid 0 to smithi15186934-7 2024-09-13T14:22:34.868 INFO:tasks.rados.rados.0.smithi151.stdout:455: finishing rollback tid 1 to smithi15186934-7 2024-09-13T14:22:34.869 INFO:tasks.rados.rados.0.smithi151.stdout:454: finishing rollback tid 1 to smithi15186934-27 2024-09-13T14:22:34.869 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 27 v 88 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-13T14:22:34.869 INFO:tasks.rados.rados.0.smithi151.stdout:455: finishing rollback tid 2 to smithi15186934-7 2024-09-13T14:22:34.869 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 221 (ObjNum 171 snap 44 seq_num 171) dirty exists 2024-09-13T14:22:34.869 INFO:tasks.rados.rados.0.smithi151.stdout:458: finishing write tid 1 to smithi15186934-8 2024-09-13T14:22:34.869 INFO:tasks.rados.rados.0.smithi151.stdout:456: expect (ObjNum 194 snap 55 seq_num 194) 2024-09-13T14:22:35.006 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:34 smithi151 ceph-mon[29807]: osdmap e173: 8 total, 8 up, 8 in 2024-09-13T14:22:35.006 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:34 smithi151 ceph-mon[41387]: osdmap e173: 8 total, 8 up, 8 in 2024-09-13T14:22:35.023 INFO:tasks.rados.rados.0.smithi151.stdout:457: expect (ObjNum 128 snap 27 seq_num 128) 2024-09-13T14:22:35.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:34 smithi184 ceph-mon[33394]: osdmap e173: 8 total, 8 up, 8 in 2024-09-13T14:22:35.151 INFO:tasks.rados.rados.0.smithi151.stdout:458: finishing write tid 2 to smithi15186934-8 2024-09-13T14:22:35.151 INFO:tasks.rados.rados.0.smithi151.stdout:458: finishing write tid 3 to smithi15186934-8 2024-09-13T14:22:35.151 INFO:tasks.rados.rados.0.smithi151.stdout:458: finishing write tid 4 to smithi15186934-8 2024-09-13T14:22:35.151 INFO:tasks.rados.rados.0.smithi151.stdout:454: done (5 left) 2024-09-13T14:22:35.151 INFO:tasks.rados.rados.0.smithi151.stdout:455: done (4 left) 2024-09-13T14:22:35.151 INFO:tasks.rados.rados.0.smithi151.stdout:456: done (3 left) 2024-09-13T14:22:35.151 INFO:tasks.rados.rados.0.smithi151.stdout:457: done (2 left) 2024-09-13T14:22:35.152 INFO:tasks.rados.rados.0.smithi151.stdout:459: done (1 left) 2024-09-13T14:22:35.152 INFO:tasks.rados.rados.0.smithi151.stdout:460: snap_remove snap 52 2024-09-13T14:22:35.499 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:22:35.689 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:22:35 smithi151 systemd[1]: Stopping Ceph alertmanager.a for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:22:35.869 INFO:tasks.rados.rados.0.smithi151.stdout:460: done (1 left) 2024-09-13T14:22:35.869 INFO:tasks.rados.rados.0.smithi151.stdout:461: snap_create 2024-09-13T14:22:35.869 INFO:tasks.rados.rados.0.smithi151.stdout:458: finishing write tid 5 to smithi15186934-8 2024-09-13T14:22:35.869 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 143 (ObjNum 207 snap 60 seq_num 207) dirty exists 2024-09-13T14:22:35.869 INFO:tasks.rados.rados.0.smithi151.stdout:458: left oid 8 (ObjNum 207 snap 60 seq_num 207) 2024-09-13T14:22:36.040 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:35 smithi151 ceph-mon[29807]: osdmap e174: 8 total, 8 up, 8 in 2024-09-13T14:22:36.040 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:35 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/3226348781' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:36.040 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:35 smithi151 ceph-mon[29807]: pgmap v33: 89 pgs: 89 active+clean; 197 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-13T14:22:36.040 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:35 smithi151 ceph-mon[29807]: from='client.15492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:36.040 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:22:35 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[79108]: ts=2024-09-13T14:22:35.688Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-13T14:22:36.040 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:22:35 smithi151 podman[92832]: 2024-09-13 14:22:35.729496131 +0000 UTC m=+0.153900502 container died 457393043edc4bd5df29aa2fcd4a3f2776d9b312de200510d3381fb081fa6130 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:22:36.041 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:35 smithi151 ceph-mon[41387]: osdmap e174: 8 total, 8 up, 8 in 2024-09-13T14:22:36.041 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:35 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/3226348781' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:36.041 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:35 smithi151 ceph-mon[41387]: pgmap v33: 89 pgs: 89 active+clean; 197 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-13T14:22:36.041 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:35 smithi151 ceph-mon[41387]: from='client.15492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:36.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:35 smithi184 ceph-mon[33394]: osdmap e174: 8 total, 8 up, 8 in 2024-09-13T14:22:36.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:35 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/3226348781' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:36.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:35 smithi184 ceph-mon[33394]: pgmap v33: 89 pgs: 89 active+clean; 197 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-13T14:22:36.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:35 smithi184 ceph-mon[33394]: from='client.15492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:36.358 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:22:36 smithi151 podman[92832]: 2024-09-13 14:22:36.138358268 +0000 UTC m=+0.562762633 container cleanup 457393043edc4bd5df29aa2fcd4a3f2776d9b312de200510d3381fb081fa6130 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:22:36.358 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:22:36 smithi151 bash[92832]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a 2024-09-13T14:22:36.693 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:22:36 smithi151 podman[92868]: 2024-09-13 14:22:36.394123816 +0000 UTC m=+0.658016767 container remove 457393043edc4bd5df29aa2fcd4a3f2776d9b312de200510d3381fb081fa6130 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:22:36.693 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:22:36 smithi151 podman[92868]: 2024-09-13 14:22:36.49090039 +0000 UTC m=+0.754793296 volume remove 14f9be8b814c23c16178f3a4847a8740a35d5a5bf17b0ce705dddbc73d712aa6 2024-09-13T14:22:36.867 INFO:tasks.rados.rados.0.smithi151.stdout:458: done (1 left) 2024-09-13T14:22:36.867 INFO:tasks.rados.rados.0.smithi151.stdout:461: done (0 left) 2024-09-13T14:22:36.867 INFO:tasks.rados.rados.0.smithi151.stdout:462: delete oid 40 current snap is 61 2024-09-13T14:22:36.878 INFO:tasks.rados.rados.0.smithi151.stdout:462: done (0 left) 2024-09-13T14:22:36.878 INFO:tasks.rados.rados.0.smithi151.stdout:463: rollback oid 3 current snap is 61 2024-09-13T14:22:36.878 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 3 to 47 2024-09-13T14:22:36.878 INFO:tasks.rados.rados.0.smithi151.stdout:464: rollback oid 19 current snap is 61 2024-09-13T14:22:36.878 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 19 to 60 2024-09-13T14:22:36.878 INFO:tasks.rados.rados.0.smithi151.stdout:465: snap_remove snap 53 2024-09-13T14:22:37.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:36 smithi151 ceph-mon[29807]: from='client.15498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:37.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:36 smithi151 ceph-mon[29807]: osdmap e175: 8 total, 8 up, 8 in 2024-09-13T14:22:37.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:36 smithi151 ceph-mon[41387]: from='client.15498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:37.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:36 smithi151 ceph-mon[41387]: osdmap e175: 8 total, 8 up, 8 in 2024-09-13T14:22:37.109 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:22:36 smithi151 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-13T14:22:37.109 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:22:36 smithi151 systemd[1]: Stopped Ceph alertmanager.a for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:22:37.109 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:22:36 smithi151 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@alertmanager.a.service: Consumed 1.202s CPU time. 2024-09-13T14:22:37.109 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:22:36 smithi151 systemd[1]: Starting Ceph alertmanager.a for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:22:37.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:36 smithi184 ceph-mon[33394]: from='client.15498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:37.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:36 smithi184 ceph-mon[33394]: osdmap e175: 8 total, 8 up, 8 in 2024-09-13T14:22:37.608 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:22:37 smithi151 podman[93005]: 2024-09-13 14:22:37.032995368 +0000 UTC m=+0.015966215 image pull 11f11916f8cdfb17c1da74b20deb2075d778357b4d60474d82453303f8d461f5 quay.io/prometheus/alertmanager:v0.27.0 2024-09-13T14:22:37.805 INFO:tasks.rados.rados.0.smithi151.stdout:465: done (2 left) 2024-09-13T14:22:37.806 INFO:tasks.rados.rados.0.smithi151.stdout:466: delete oid 29 current snap is 61 2024-09-13T14:22:37.806 INFO:tasks.rados.rados.0.smithi151.stdout:463: finishing rollback tid 1 to smithi15186934-3 2024-09-13T14:22:37.806 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 122 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-13T14:22:37.806 INFO:tasks.rados.rados.0.smithi151.stdout:464: finishing rollback tid 0 to smithi15186934-19 2024-09-13T14:22:37.806 INFO:tasks.rados.rados.0.smithi151.stdout:464: finishing rollback tid 1 to smithi15186934-19 2024-09-13T14:22:37.806 INFO:tasks.rados.rados.0.smithi151.stdout:464: finishing rollback tid 2 to smithi15186934-19 2024-09-13T14:22:37.806 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 19 v 119 (ObjNum 140 snap 31 seq_num 140) dirty exists 2024-09-13T14:22:37.809 INFO:tasks.rados.rados.0.smithi151.stdout:463: done (2 left) 2024-09-13T14:22:37.809 INFO:tasks.rados.rados.0.smithi151.stdout:464: done (1 left) 2024-09-13T14:22:37.809 INFO:tasks.rados.rados.0.smithi151.stdout:466: done (0 left) 2024-09-13T14:22:37.809 INFO:tasks.rados.rados.0.smithi151.stdout:467: snap_create 2024-09-13T14:22:37.853 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:37 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:22:37] "GET /metrics HTTP/1.1" 200 36409 "" "Prometheus/2.43.0" 2024-09-13T14:22:37.880 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:22:37 smithi151 podman[93005]: 2024-09-13 14:22:37.612827152 +0000 UTC m=+0.595798033 volume create c6414bb6d522b88697c95b79ed90224b825961ea410115f794924cf09daade91 2024-09-13T14:22:37.880 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:22:37 smithi151 podman[93005]: 2024-09-13 14:22:37.704990703 +0000 UTC m=+0.687961534 container create ed70c3d751072ecc08fe78b35eb20e6407c7ae03a131f6a3030aaef8d0b11d2d (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:22:38.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:37 smithi184 ceph-mon[33394]: osdmap e176: 8 total, 8 up, 8 in 2024-09-13T14:22:38.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:37 smithi184 ceph-mon[33394]: pgmap v36: 89 pgs: 89 active+clean; 199 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 375 KiB/s wr, 6 op/s 2024-09-13T14:22:38.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:37 smithi184 ceph-mon[33394]: osdmap e177: 8 total, 8 up, 8 in 2024-09-13T14:22:38.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:37 smithi151 ceph-mon[29807]: osdmap e176: 8 total, 8 up, 8 in 2024-09-13T14:22:38.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:37 smithi151 ceph-mon[29807]: pgmap v36: 89 pgs: 89 active+clean; 199 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 375 KiB/s wr, 6 op/s 2024-09-13T14:22:38.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:37 smithi151 ceph-mon[29807]: osdmap e177: 8 total, 8 up, 8 in 2024-09-13T14:22:38.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:37 smithi151 ceph-mon[41387]: osdmap e176: 8 total, 8 up, 8 in 2024-09-13T14:22:38.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:37 smithi151 ceph-mon[41387]: pgmap v36: 89 pgs: 89 active+clean; 199 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 375 KiB/s wr, 6 op/s 2024-09-13T14:22:38.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:37 smithi151 ceph-mon[41387]: osdmap e177: 8 total, 8 up, 8 in 2024-09-13T14:22:38.359 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:22:38 smithi151 podman[93005]: 2024-09-13 14:22:38.000238634 +0000 UTC m=+0.983209478 container init ed70c3d751072ecc08fe78b35eb20e6407c7ae03a131f6a3030aaef8d0b11d2d (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:22:38.359 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:22:38 smithi151 podman[93005]: 2024-09-13 14:22:38.004914169 +0000 UTC m=+0.987885012 container start ed70c3d751072ecc08fe78b35eb20e6407c7ae03a131f6a3030aaef8d0b11d2d (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:22:38.359 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:22:38 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[93024]: ts=2024-09-13T14:22:38.014Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)" 2024-09-13T14:22:38.359 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:22:38 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[93024]: ts=2024-09-13T14:22:38.015Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)" 2024-09-13T14:22:38.359 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:22:38 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[93024]: ts=2024-09-13T14:22:38.016Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.151 port=9094 2024-09-13T14:22:38.359 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:22:38 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[93024]: ts=2024-09-13T14:22:38.017Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-13T14:22:38.359 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:22:38 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[93024]: ts=2024-09-13T14:22:38.040Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-13T14:22:38.359 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:22:38 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[93024]: ts=2024-09-13T14:22:38.041Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-13T14:22:38.359 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:22:38 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[93024]: ts=2024-09-13T14:22:38.044Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093 2024-09-13T14:22:38.359 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:22:38 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[93024]: ts=2024-09-13T14:22:38.044Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-13T14:22:38.360 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:22:38 smithi151 bash[93005]: ed70c3d751072ecc08fe78b35eb20e6407c7ae03a131f6a3030aaef8d0b11d2d 2024-09-13T14:22:38.360 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:22:38 smithi151 systemd[1]: Started Ceph alertmanager.a for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:22:38.477 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:22:38.801 INFO:tasks.rados.rados.0.smithi151.stdout:467: done (0 left) 2024-09-13T14:22:38.801 INFO:tasks.rados.rados.0.smithi151.stdout:468: read oid 26 snap -1 2024-09-13T14:22:38.801 INFO:tasks.rados.rados.0.smithi151.stdout:468: expect (ObjNum 148 snap 34 seq_num 148) 2024-09-13T14:22:38.801 INFO:tasks.rados.rados.0.smithi151.stdout:469: delete oid 16 current snap is 62 2024-09-13T14:22:38.805 INFO:tasks.rados.rados.0.smithi151.stdout:468: expect (ObjNum 148 snap 34 seq_num 148) 2024-09-13T14:22:38.858 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:38 smithi151 systemd[1]: Stopping Ceph node-exporter.a for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:22:38.932 INFO:tasks.rados.rados.0.smithi151.stdout:468: done (1 left) 2024-09-13T14:22:38.932 INFO:tasks.rados.rados.0.smithi151.stdout:469: done (0 left) 2024-09-13T14:22:38.933 INFO:tasks.rados.rados.0.smithi151.stdout:470: snap_remove snap 38 2024-09-13T14:22:39.325 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:38 smithi151 podman[93272]: 2024-09-13 14:22:38.994149666 +0000 UTC m=+0.137984855 container died 401be9d35c59f33d08abbec03ec8d734765876e1035fb57ca6353f7f7e15f3ea (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:22:39.325 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:39 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:39.325 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:39 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:39.325 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:39 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/725271270' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:39.325 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:39 smithi151 ceph-mon[29807]: Reconfiguring node-exporter.a (dependencies changed)... 2024-09-13T14:22:39.326 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:39 smithi151 ceph-mon[29807]: Deploying daemon node-exporter.a on smithi151 2024-09-13T14:22:39.326 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:39 smithi151 ceph-mon[29807]: from='client.25412 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:39.326 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:39 smithi151 ceph-mon[29807]: osdmap e178: 8 total, 8 up, 8 in 2024-09-13T14:22:39.326 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:39 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:39.326 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:39 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:39.326 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:39 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/725271270' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:39.326 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:39 smithi151 ceph-mon[41387]: Reconfiguring node-exporter.a (dependencies changed)... 2024-09-13T14:22:39.326 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:39 smithi151 ceph-mon[41387]: Deploying daemon node-exporter.a on smithi151 2024-09-13T14:22:39.326 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:39 smithi151 ceph-mon[41387]: from='client.25412 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:39.326 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:39 smithi151 ceph-mon[41387]: osdmap e178: 8 total, 8 up, 8 in 2024-09-13T14:22:39.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:39 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:39.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:39 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:39.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:39 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/725271270' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:39.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:39 smithi184 ceph-mon[33394]: Reconfiguring node-exporter.a (dependencies changed)... 2024-09-13T14:22:39.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:39 smithi184 ceph-mon[33394]: Deploying daemon node-exporter.a on smithi151 2024-09-13T14:22:39.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:39 smithi184 ceph-mon[33394]: from='client.25412 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:39.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:39 smithi184 ceph-mon[33394]: osdmap e178: 8 total, 8 up, 8 in 2024-09-13T14:22:39.608 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:39 smithi151 podman[93272]: 2024-09-13 14:22:39.423595605 +0000 UTC m=+0.567430794 container cleanup 401be9d35c59f33d08abbec03ec8d734765876e1035fb57ca6353f7f7e15f3ea (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:22:39.608 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:39 smithi151 bash[93272]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a 2024-09-13T14:22:39.802 INFO:tasks.rados.rados.0.smithi151.stdout:470: done (0 left) 2024-09-13T14:22:39.802 INFO:tasks.rados.rados.0.smithi151.stdout:471: snap_remove snap 44 2024-09-13T14:22:39.902 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:39 smithi151 podman[93289]: 2024-09-13 14:22:39.65053855 +0000 UTC m=+0.659354675 container remove 401be9d35c59f33d08abbec03ec8d734765876e1035fb57ca6353f7f7e15f3ea (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:22:39.902 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:39 smithi151 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-13T14:22:39.903 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:39 smithi151 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-13T14:22:39.903 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:39 smithi151 systemd[1]: Stopped Ceph node-exporter.a for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:22:39.903 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:39 smithi151 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@node-exporter.a.service: Consumed 2.268s CPU time. 2024-09-13T14:22:40.159 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:22:40 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[93024]: ts=2024-09-13T14:22:40.018Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000816087s 2024-09-13T14:22:40.159 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:40 smithi151 systemd[1]: Starting Ceph node-exporter.a for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:22:40.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:40 smithi184 ceph-mon[33394]: from='client.25168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:40.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:40 smithi184 ceph-mon[33394]: pgmap v39: 89 pgs: 89 active+clean; 199 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 375 KiB/s wr, 6 op/s 2024-09-13T14:22:40.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:40 smithi184 ceph-mon[33394]: osdmap e179: 8 total, 8 up, 8 in 2024-09-13T14:22:40.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:40 smithi151 ceph-mon[41387]: from='client.25168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:40.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:40 smithi151 ceph-mon[41387]: pgmap v39: 89 pgs: 89 active+clean; 199 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 375 KiB/s wr, 6 op/s 2024-09-13T14:22:40.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:40 smithi151 ceph-mon[41387]: osdmap e179: 8 total, 8 up, 8 in 2024-09-13T14:22:40.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:40 smithi151 ceph-mon[29807]: from='client.25168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:40.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:40 smithi151 ceph-mon[29807]: pgmap v39: 89 pgs: 89 active+clean; 199 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 375 KiB/s wr, 6 op/s 2024-09-13T14:22:40.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:40 smithi151 ceph-mon[29807]: osdmap e179: 8 total, 8 up, 8 in 2024-09-13T14:22:40.609 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:40 smithi151 bash[93424]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-09-13T14:22:40.803 INFO:tasks.rados.rados.0.smithi151.stdout:471: done (0 left) 2024-09-13T14:22:40.803 INFO:tasks.rados.rados.0.smithi151.stdout:472: snap_create 2024-09-13T14:22:41.108 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:40 smithi151 bash[93424]: Getting image source signatures 2024-09-13T14:22:41.108 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:40 smithi151 bash[93424]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-09-13T14:22:41.108 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:40 smithi151 bash[93424]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-13T14:22:41.108 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:40 smithi151 bash[93424]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-09-13T14:22:41.480 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:22:41.812 INFO:tasks.rados.rados.0.smithi151.stdout:472: done (0 left) 2024-09-13T14:22:41.812 INFO:tasks.rados.rados.0.smithi151.stdout:473: read oid 27 snap -1 2024-09-13T14:22:41.812 INFO:tasks.rados.rados.0.smithi151.stdout:473: expect deleted 2024-09-13T14:22:41.812 INFO:tasks.rados.rados.0.smithi151.stdout:474: rollback oid 48 current snap is 63 2024-09-13T14:22:41.812 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 48 to 31 2024-09-13T14:22:41.812 INFO:tasks.rados.rados.0.smithi151.stdout:475: read oid 17 snap 35 2024-09-13T14:22:41.812 INFO:tasks.rados.rados.0.smithi151.stdout:475: expect (ObjNum 103 snap 16 seq_num 103) 2024-09-13T14:22:41.812 INFO:tasks.rados.rados.0.smithi151.stdout:476: read oid 14 snap -1 2024-09-13T14:22:41.812 INFO:tasks.rados.rados.0.smithi151.stdout:476: expect (ObjNum 75 snap 9 seq_num 75) 2024-09-13T14:22:41.812 INFO:tasks.rados.rados.0.smithi151.stdout:477: write oid 46 current snap is 63 2024-09-13T14:22:41.813 INFO:tasks.rados.rados.0.smithi151.stdout:477: seq_num 211 ranges {746277=619070,1830877=681726,2928369=617908} 2024-09-13T14:22:41.820 INFO:tasks.rados.rados.0.smithi151.stdout:477: writing smithi15186934-46 from 746277 to 1365347 tid 1 2024-09-13T14:22:41.828 INFO:tasks.rados.rados.0.smithi151.stdout:477: writing smithi15186934-46 from 1830877 to 2512603 tid 2 2024-09-13T14:22:41.834 INFO:tasks.rados.rados.0.smithi151.stdout:477: writing smithi15186934-46 from 2928369 to 3546277 tid 3 2024-09-13T14:22:41.834 INFO:tasks.rados.rados.0.smithi151.stdout:478: snap_remove snap 54 2024-09-13T14:22:42.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:41 smithi184 ceph-mon[33394]: osdmap e180: 8 total, 8 up, 8 in 2024-09-13T14:22:42.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:41 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/747505411' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:42.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:41 smithi184 ceph-mon[33394]: pgmap v42: 89 pgs: 89 active+clean; 199 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:42.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:41 smithi184 ceph-mon[33394]: from='client.25430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:42.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:41 smithi151 ceph-mon[29807]: osdmap e180: 8 total, 8 up, 8 in 2024-09-13T14:22:42.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:41 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/747505411' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:42.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:41 smithi151 ceph-mon[29807]: pgmap v42: 89 pgs: 89 active+clean; 199 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:42.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:41 smithi151 ceph-mon[29807]: from='client.25430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:42.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:41 smithi151 ceph-mon[41387]: osdmap e180: 8 total, 8 up, 8 in 2024-09-13T14:22:42.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:41 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/747505411' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:42.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:41 smithi151 ceph-mon[41387]: pgmap v42: 89 pgs: 89 active+clean; 199 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:42.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:41 smithi151 ceph-mon[41387]: from='client.25430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:42.109 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:42 smithi151 bash[93424]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-13T14:22:42.701 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:42 smithi151 bash[93424]: Writing manifest to image destination 2024-09-13T14:22:42.814 INFO:tasks.rados.rados.0.smithi151.stdout:478: done (5 left) 2024-09-13T14:22:42.814 INFO:tasks.rados.rados.0.smithi151.stdout:479: read oid 4 snap -1 2024-09-13T14:22:42.814 INFO:tasks.rados.rados.0.smithi151.stdout:479: expect (ObjNum 192 snap 51 seq_num 192) 2024-09-13T14:22:42.814 INFO:tasks.rados.rados.0.smithi151.stdout:480: rollback oid 7 current snap is 63 2024-09-13T14:22:42.814 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 7 to 57 2024-09-13T14:22:42.815 INFO:tasks.rados.rados.0.smithi151.stdout:474: finishing rollback tid 0 to smithi15186934-48 2024-09-13T14:22:42.815 INFO:tasks.rados.rados.0.smithi151.stdout:474: finishing rollback tid 1 to smithi15186934-48 2024-09-13T14:22:42.815 INFO:tasks.rados.rados.0.smithi151.stdout:474: finishing rollback tid 2 to smithi15186934-48 2024-09-13T14:22:42.815 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 48 v 200 (ObjNum 128 snap 27 seq_num 128) dirty exists 2024-09-13T14:22:42.815 INFO:tasks.rados.rados.0.smithi151.stdout:476: expect (ObjNum 75 snap 9 seq_num 75) 2024-09-13T14:22:42.927 INFO:tasks.rados.rados.0.smithi151.stdout:477: finishing write tid 1 to smithi15186934-46 2024-09-13T14:22:42.927 INFO:tasks.rados.rados.0.smithi151.stdout:475: expect (ObjNum 103 snap 16 seq_num 103) 2024-09-13T14:22:42.974 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:42 smithi151 ceph-mon[29807]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:42.974 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:42 smithi151 ceph-mon[29807]: osdmap e181: 8 total, 8 up, 8 in 2024-09-13T14:22:42.974 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:42 smithi151 ceph-mon[41387]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:42.975 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:42 smithi151 ceph-mon[41387]: osdmap e181: 8 total, 8 up, 8 in 2024-09-13T14:22:42.975 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:42 smithi151 podman[93424]: 2024-09-13 14:22:42.678451798 +0000 UTC m=+2.498579025 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-13T14:22:42.975 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:42 smithi151 podman[93424]: 2024-09-13 14:22:42.785449038 +0000 UTC m=+2.605576265 container create d91e4cb09e9abc98c127ea34498db567a9a6dbd00d90a8be381e26fffd4beeba (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:22:43.051 INFO:tasks.rados.rados.0.smithi151.stdout:477: finishing write tid 2 to smithi15186934-46 2024-09-13T14:22:43.051 INFO:tasks.rados.rados.0.smithi151.stdout:477: finishing write tid 3 to smithi15186934-46 2024-09-13T14:22:43.051 INFO:tasks.rados.rados.0.smithi151.stdout:477: finishing write tid 4 to smithi15186934-46 2024-09-13T14:22:43.051 INFO:tasks.rados.rados.0.smithi151.stdout:473: done (6 left) 2024-09-13T14:22:43.051 INFO:tasks.rados.rados.0.smithi151.stdout:474: done (5 left) 2024-09-13T14:22:43.051 INFO:tasks.rados.rados.0.smithi151.stdout:475: done (4 left) 2024-09-13T14:22:43.051 INFO:tasks.rados.rados.0.smithi151.stdout:476: done (3 left) 2024-09-13T14:22:43.051 INFO:tasks.rados.rados.0.smithi151.stdout:481: rollback oid 14 current snap is 63 2024-09-13T14:22:43.051 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 14 to 57 2024-09-13T14:22:43.051 INFO:tasks.rados.rados.0.smithi151.stdout:477: finishing write tid 5 to smithi15186934-46 2024-09-13T14:22:43.051 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 46 v 143 (ObjNum 211 snap 63 seq_num 211) dirty exists 2024-09-13T14:22:43.051 INFO:tasks.rados.rados.0.smithi151.stdout:477: left oid 46 (ObjNum 211 snap 63 seq_num 211) 2024-09-13T14:22:43.052 INFO:tasks.rados.rados.0.smithi151.stdout:480: finishing rollback tid 0 to smithi15186934-7 2024-09-13T14:22:43.052 INFO:tasks.rados.rados.0.smithi151.stdout:480: finishing rollback tid 1 to smithi15186934-7 2024-09-13T14:22:43.052 INFO:tasks.rados.rados.0.smithi151.stdout:480: finishing rollback tid 2 to smithi15186934-7 2024-09-13T14:22:43.052 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 255 (ObjNum 171 snap 44 seq_num 171) dirty exists 2024-09-13T14:22:43.052 INFO:tasks.rados.rados.0.smithi151.stdout:479: expect (ObjNum 192 snap 51 seq_num 192) 2024-09-13T14:22:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:42 smithi184 ceph-mon[33394]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:42 smithi184 ceph-mon[33394]: osdmap e181: 8 total, 8 up, 8 in 2024-09-13T14:22:43.211 INFO:tasks.rados.rados.0.smithi151.stdout:481: finishing rollback tid 0 to smithi15186934-14 2024-09-13T14:22:43.211 INFO:tasks.rados.rados.0.smithi151.stdout:481: finishing rollback tid 1 to smithi15186934-14 2024-09-13T14:22:43.211 INFO:tasks.rados.rados.0.smithi151.stdout:477: done (3 left) 2024-09-13T14:22:43.211 INFO:tasks.rados.rados.0.smithi151.stdout:479: done (2 left) 2024-09-13T14:22:43.211 INFO:tasks.rados.rados.0.smithi151.stdout:480: done (1 left) 2024-09-13T14:22:43.211 INFO:tasks.rados.rados.0.smithi151.stdout:482: rollback oid 3 current snap is 63 2024-09-13T14:22:43.211 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 3 to 59 2024-09-13T14:22:43.211 INFO:tasks.rados.rados.0.smithi151.stdout:481: finishing rollback tid 2 to smithi15186934-14 2024-09-13T14:22:43.211 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 176 (ObjNum 75 snap 9 seq_num 75) dirty exists 2024-09-13T14:22:43.212 INFO:tasks.rados.rados.0.smithi151.stdout:481: done (1 left) 2024-09-13T14:22:43.212 INFO:tasks.rados.rados.0.smithi151.stdout:483: snap_remove snap 31 2024-09-13T14:22:43.358 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:42 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:22:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:22:43.360 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 podman[93424]: 2024-09-13 14:22:43.05593016 +0000 UTC m=+2.876057397 container init d91e4cb09e9abc98c127ea34498db567a9a6dbd00d90a8be381e26fffd4beeba (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:22:43.360 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 podman[93424]: 2024-09-13 14:22:43.060943783 +0000 UTC m=+2.881071020 container start d91e4cb09e9abc98c127ea34498db567a9a6dbd00d90a8be381e26fffd4beeba (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:22:43.360 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.063Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-13T14:22:43.360 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.064Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-13T14:22:43.360 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.064Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-13T14:22:43.361 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.064Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-13T14:22:43.361 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.064Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-13T14:22:43.361 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-13T14:22:43.361 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-13T14:22:43.361 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=arp 2024-09-13T14:22:43.362 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-13T14:22:43.362 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-13T14:22:43.362 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-13T14:22:43.362 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-13T14:22:43.363 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-13T14:22:43.363 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-13T14:22:43.363 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-13T14:22:43.363 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-13T14:22:43.363 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=edac 2024-09-13T14:22:43.364 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-13T14:22:43.364 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-13T14:22:43.364 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-13T14:22:43.364 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-13T14:22:43.364 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-13T14:22:43.365 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-13T14:22:43.365 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-13T14:22:43.365 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-13T14:22:43.365 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-13T14:22:43.365 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-13T14:22:43.366 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-13T14:22:43.366 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-13T14:22:43.366 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-13T14:22:43.366 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-13T14:22:43.366 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-13T14:22:43.366 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-13T14:22:43.366 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=os 2024-09-13T14:22:43.366 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-13T14:22:43.367 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-13T14:22:43.367 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-13T14:22:43.367 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-13T14:22:43.367 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-13T14:22:43.367 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-13T14:22:43.367 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-13T14:22:43.367 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=stat 2024-09-13T14:22:43.367 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-13T14:22:43.368 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-13T14:22:43.368 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-13T14:22:43.368 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=time 2024-09-13T14:22:43.368 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-13T14:22:43.368 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=uname 2024-09-13T14:22:43.368 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-13T14:22:43.368 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-13T14:22:43.368 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-13T14:22:43.369 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-13T14:22:43.369 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a[93571]: ts=2024-09-13T14:22:43.065Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-13T14:22:43.369 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 bash[93424]: d91e4cb09e9abc98c127ea34498db567a9a6dbd00d90a8be381e26fffd4beeba 2024-09-13T14:22:43.369 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 14:22:43 smithi151 systemd[1]: Started Ceph node-exporter.a for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:22:44.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:43 smithi184 ceph-mon[33394]: osdmap e182: 8 total, 8 up, 8 in 2024-09-13T14:22:44.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:43 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:44.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:43 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:44.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:43 smithi184 ceph-mon[33394]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:22:44.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:43 smithi184 ceph-mon[33394]: pgmap v45: 89 pgs: 89 active+clean; 200 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 469 KiB/s wr, 9 op/s 2024-09-13T14:22:44.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:43 smithi184 ceph-mon[33394]: Deploying daemon prometheus.a on smithi184 2024-09-13T14:22:44.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-mon[29807]: osdmap e182: 8 total, 8 up, 8 in 2024-09-13T14:22:44.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:44.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:44.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-mon[29807]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:22:44.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-mon[29807]: pgmap v45: 89 pgs: 89 active+clean; 200 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 469 KiB/s wr, 9 op/s 2024-09-13T14:22:44.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-mon[29807]: Deploying daemon prometheus.a on smithi184 2024-09-13T14:22:44.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-mon[41387]: osdmap e182: 8 total, 8 up, 8 in 2024-09-13T14:22:44.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:44.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:44.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-mon[41387]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:22:44.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-mon[41387]: pgmap v45: 89 pgs: 89 active+clean; 200 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 469 KiB/s wr, 9 op/s 2024-09-13T14:22:44.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:43 smithi151 ceph-mon[41387]: Deploying daemon prometheus.a on smithi184 2024-09-13T14:22:44.207 INFO:tasks.rados.rados.0.smithi151.stdout:483: done (1 left) 2024-09-13T14:22:44.207 INFO:tasks.rados.rados.0.smithi151.stdout:484: delete oid 16 current snap is 63 2024-09-13T14:22:44.207 INFO:tasks.rados.rados.0.smithi151.stdout:482: finishing rollback tid 1 to smithi15186934-3 2024-09-13T14:22:44.207 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 122 (ObjNum 129 snap 0 seq_num 3091268352) dirty dne 2024-09-13T14:22:44.213 INFO:tasks.rados.rados.0.smithi151.stdout:482: done (1 left) 2024-09-13T14:22:44.213 INFO:tasks.rados.rados.0.smithi151.stdout:484: done (0 left) 2024-09-13T14:22:44.213 INFO:tasks.rados.rados.0.smithi151.stdout:485: read oid 37 snap -1 2024-09-13T14:22:44.213 INFO:tasks.rados.rados.0.smithi151.stdout:485: expect (ObjNum 179 snap 47 seq_num 179) 2024-09-13T14:22:44.213 INFO:tasks.rados.rados.0.smithi151.stdout:486: read oid 47 snap 48 2024-09-13T14:22:44.213 INFO:tasks.rados.rados.0.smithi151.stdout:486: expect (ObjNum 155 snap 38 seq_num 155) 2024-09-13T14:22:44.213 INFO:tasks.rados.rados.0.smithi151.stdout:487: snap_remove snap 49 2024-09-13T14:22:44.407 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:22:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:44 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/462271719' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:44 smithi184 ceph-mon[33394]: osdmap e183: 8 total, 8 up, 8 in 2024-09-13T14:22:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:44 smithi184 ceph-mon[33394]: from='client.15546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:44 smithi184 ceph-mon[33394]: from='client.15552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:45.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:44 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/462271719' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:45.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:44 smithi151 ceph-mon[41387]: osdmap e183: 8 total, 8 up, 8 in 2024-09-13T14:22:45.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:44 smithi151 ceph-mon[41387]: from='client.15546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:45.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:44 smithi151 ceph-mon[41387]: from='client.15552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:45.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:44 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/462271719' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:45.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:44 smithi151 ceph-mon[29807]: osdmap e183: 8 total, 8 up, 8 in 2024-09-13T14:22:45.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:44 smithi151 ceph-mon[29807]: from='client.15546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:45.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:44 smithi151 ceph-mon[29807]: from='client.15552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:45.210 INFO:tasks.rados.rados.0.smithi151.stdout:487: done (2 left) 2024-09-13T14:22:45.211 INFO:tasks.rados.rados.0.smithi151.stdout:488: rollback oid 17 current snap is 63 2024-09-13T14:22:45.211 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 17 to 36 2024-09-13T14:22:45.211 INFO:tasks.rados.rados.0.smithi151.stdout:485: expect (ObjNum 179 snap 47 seq_num 179) 2024-09-13T14:22:45.381 INFO:tasks.rados.rados.0.smithi151.stdout:486: expect (ObjNum 155 snap 38 seq_num 155) 2024-09-13T14:22:45.493 INFO:tasks.rados.rados.0.smithi151.stdout:488: finishing rollback tid 0 to smithi15186934-17 2024-09-13T14:22:45.493 INFO:tasks.rados.rados.0.smithi151.stdout:488: finishing rollback tid 1 to smithi15186934-17 2024-09-13T14:22:45.493 INFO:tasks.rados.rados.0.smithi151.stdout:485: done (2 left) 2024-09-13T14:22:45.493 INFO:tasks.rados.rados.0.smithi151.stdout:486: done (1 left) 2024-09-13T14:22:45.493 INFO:tasks.rados.rados.0.smithi151.stdout:489: write oid 22 current snap is 63 2024-09-13T14:22:45.493 INFO:tasks.rados.rados.0.smithi151.stdout:488: finishing rollback tid 2 to smithi15186934-17 2024-09-13T14:22:45.493 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 277 (ObjNum 103 snap 16 seq_num 103) dirty exists 2024-09-13T14:22:45.493 INFO:tasks.rados.rados.0.smithi151.stdout:489: seq_num 213 ranges {779891=485921,1947841=685395,3383215=196676} 2024-09-13T14:22:45.500 INFO:tasks.rados.rados.0.smithi151.stdout:489: writing smithi15186934-22 from 779891 to 1265812 tid 1 2024-09-13T14:22:45.506 INFO:tasks.rados.rados.0.smithi151.stdout:489: writing smithi15186934-22 from 1947841 to 2633236 tid 2 2024-09-13T14:22:45.511 INFO:tasks.rados.rados.0.smithi151.stdout:489: writing smithi15186934-22 from 3383215 to 3579891 tid 3 2024-09-13T14:22:45.512 INFO:tasks.rados.rados.0.smithi151.stdout:488: done (1 left) 2024-09-13T14:22:45.512 INFO:tasks.rados.rados.0.smithi151.stdout:490: delete oid 18 current snap is 63 2024-09-13T14:22:45.512 INFO:tasks.rados.rados.0.smithi151.stdout:489: finishing write tid 1 to smithi15186934-22 2024-09-13T14:22:45.512 INFO:tasks.rados.rados.0.smithi151.stdout:489: finishing write tid 2 to smithi15186934-22 2024-09-13T14:22:45.513 INFO:tasks.rados.rados.0.smithi151.stdout:489: finishing write tid 3 to smithi15186934-22 2024-09-13T14:22:45.513 INFO:tasks.rados.rados.0.smithi151.stdout:489: finishing write tid 4 to smithi15186934-22 2024-09-13T14:22:45.513 INFO:tasks.rados.rados.0.smithi151.stdout:490: done (1 left) 2024-09-13T14:22:45.514 INFO:tasks.rados.rados.0.smithi151.stdout:491: write oid 42 current snap is 63 2024-09-13T14:22:45.514 INFO:tasks.rados.rados.0.smithi151.stdout:491: seq_num 215 ranges {413505=636419,1664805=689064,3038061=492071} 2024-09-13T14:22:45.519 INFO:tasks.rados.rados.0.smithi151.stdout:491: writing smithi15186934-42 from 413505 to 1049924 tid 1 2024-09-13T14:22:45.526 INFO:tasks.rados.rados.0.smithi151.stdout:491: writing smithi15186934-42 from 1664805 to 2353869 tid 2 2024-09-13T14:22:45.533 INFO:tasks.rados.rados.0.smithi151.stdout:491: writing smithi15186934-42 from 3038061 to 3530132 tid 3 2024-09-13T14:22:45.533 INFO:tasks.rados.rados.0.smithi151.stdout:492: read oid 6 snap 62 2024-09-13T14:22:45.533 INFO:tasks.rados.rados.0.smithi151.stdout:492: expect (ObjNum 147 snap 34 seq_num 147) 2024-09-13T14:22:45.533 INFO:tasks.rados.rados.0.smithi151.stdout:493: write oid 46 current snap is 63 2024-09-13T14:22:45.533 INFO:tasks.rados.rados.0.smithi151.stdout:489: finishing write tid 5 to smithi15186934-22 2024-09-13T14:22:45.533 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 22 v 79 (ObjNum 213 snap 63 seq_num 213) dirty exists 2024-09-13T14:22:45.533 INFO:tasks.rados.rados.0.smithi151.stdout:489: left oid 22 (ObjNum 213 snap 63 seq_num 213) 2024-09-13T14:22:45.533 INFO:tasks.rados.rados.0.smithi151.stdout:491: finishing write tid 1 to smithi15186934-42 2024-09-13T14:22:45.533 INFO:tasks.rados.rados.0.smithi151.stdout:491: finishing write tid 2 to smithi15186934-42 2024-09-13T14:22:45.533 INFO:tasks.rados.rados.0.smithi151.stdout:493: seq_num 216 ranges {430312=711668,1723287=549075,2781837=703343} 2024-09-13T14:22:45.539 INFO:tasks.rados.rados.0.smithi151.stdout:493: writing smithi15186934-46 from 430312 to 1141980 tid 1 2024-09-13T14:22:45.545 INFO:tasks.rados.rados.0.smithi151.stdout:493: writing smithi15186934-46 from 1723287 to 2272362 tid 2 2024-09-13T14:22:45.552 INFO:tasks.rados.rados.0.smithi151.stdout:493: writing smithi15186934-46 from 2781837 to 3485180 tid 3 2024-09-13T14:22:45.552 INFO:tasks.rados.rados.0.smithi151.stdout:489: done (3 left) 2024-09-13T14:22:45.552 INFO:tasks.rados.rados.0.smithi151.stdout:494: read oid 26 snap -1 2024-09-13T14:22:45.552 INFO:tasks.rados.rados.0.smithi151.stdout:494: expect (ObjNum 148 snap 34 seq_num 148) 2024-09-13T14:22:45.552 INFO:tasks.rados.rados.0.smithi151.stdout:495: rollback oid 4 current snap is 63 2024-09-13T14:22:45.552 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 4 to 60 2024-09-13T14:22:45.552 INFO:tasks.rados.rados.0.smithi151.stdout:491: finishing write tid 3 to smithi15186934-42 2024-09-13T14:22:45.553 INFO:tasks.rados.rados.0.smithi151.stdout:491: finishing write tid 4 to smithi15186934-42 2024-09-13T14:22:45.553 INFO:tasks.rados.rados.0.smithi151.stdout:492: expect (ObjNum 147 snap 34 seq_num 147) 2024-09-13T14:22:45.691 INFO:tasks.rados.rados.0.smithi151.stdout:491: finishing write tid 5 to smithi15186934-42 2024-09-13T14:22:45.692 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 185 (ObjNum 215 snap 63 seq_num 215) dirty exists 2024-09-13T14:22:45.692 INFO:tasks.rados.rados.0.smithi151.stdout:491: left oid 42 (ObjNum 215 snap 63 seq_num 215) 2024-09-13T14:22:45.692 INFO:tasks.rados.rados.0.smithi151.stdout:493: finishing write tid 1 to smithi15186934-46 2024-09-13T14:22:45.692 INFO:tasks.rados.rados.0.smithi151.stdout:493: finishing write tid 2 to smithi15186934-46 2024-09-13T14:22:45.692 INFO:tasks.rados.rados.0.smithi151.stdout:491: done (4 left) 2024-09-13T14:22:45.693 INFO:tasks.rados.rados.0.smithi151.stdout:492: done (3 left) 2024-09-13T14:22:45.693 INFO:tasks.rados.rados.0.smithi151.stdout:496: delete oid 1 current snap is 63 2024-09-13T14:22:45.693 INFO:tasks.rados.rados.0.smithi151.stdout:495: finishing rollback tid 0 to smithi15186934-4 2024-09-13T14:22:45.693 INFO:tasks.rados.rados.0.smithi151.stdout:495: finishing rollback tid 1 to smithi15186934-4 2024-09-13T14:22:45.694 INFO:tasks.rados.rados.0.smithi151.stdout:493: finishing write tid 3 to smithi15186934-46 2024-09-13T14:22:45.694 INFO:tasks.rados.rados.0.smithi151.stdout:493: finishing write tid 4 to smithi15186934-46 2024-09-13T14:22:45.694 INFO:tasks.rados.rados.0.smithi151.stdout:493: finishing write tid 5 to smithi15186934-46 2024-09-13T14:22:45.694 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 46 v 157 (ObjNum 216 snap 63 seq_num 216) dirty exists 2024-09-13T14:22:45.694 INFO:tasks.rados.rados.0.smithi151.stdout:493: left oid 46 (ObjNum 216 snap 63 seq_num 216) 2024-09-13T14:22:45.694 INFO:tasks.rados.rados.0.smithi151.stdout:495: finishing rollback tid 2 to smithi15186934-4 2024-09-13T14:22:45.695 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 224 (ObjNum 192 snap 51 seq_num 192) dirty exists 2024-09-13T14:22:45.695 INFO:tasks.rados.rados.0.smithi151.stdout:494: expect (ObjNum 148 snap 34 seq_num 148) 2024-09-13T14:22:45.812 INFO:tasks.rados.rados.0.smithi151.stdout:493: done (3 left) 2024-09-13T14:22:45.812 INFO:tasks.rados.rados.0.smithi151.stdout:494: done (2 left) 2024-09-13T14:22:45.813 INFO:tasks.rados.rados.0.smithi151.stdout:495: done (1 left) 2024-09-13T14:22:45.813 INFO:tasks.rados.rados.0.smithi151.stdout:496: done (0 left) 2024-09-13T14:22:45.813 INFO:tasks.rados.rados.0.smithi151.stdout:497: write oid 5 current snap is 63 2024-09-13T14:22:45.813 INFO:tasks.rados.rados.0.smithi151.stdout:497: seq_num 218 ranges {463926=462166,1723898=436391,2603977=725887} 2024-09-13T14:22:45.817 INFO:tasks.rados.rados.0.smithi151.stdout:497: writing smithi15186934-5 from 463926 to 926092 tid 1 2024-09-13T14:22:45.824 INFO:tasks.rados.rados.0.smithi151.stdout:497: writing smithi15186934-5 from 1723898 to 2160289 tid 2 2024-09-13T14:22:45.830 INFO:tasks.rados.rados.0.smithi151.stdout:497: writing smithi15186934-5 from 2603977 to 3329864 tid 3 2024-09-13T14:22:45.831 INFO:tasks.rados.rados.0.smithi151.stdout:498: rollback oid 41 current snap is 63 2024-09-13T14:22:45.831 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 41 to 48 2024-09-13T14:22:45.831 INFO:tasks.rados.rados.0.smithi151.stdout:497: finishing write tid 1 to smithi15186934-5 2024-09-13T14:22:45.831 INFO:tasks.rados.rados.0.smithi151.stdout:497: finishing write tid 2 to smithi15186934-5 2024-09-13T14:22:45.831 INFO:tasks.rados.rados.0.smithi151.stdout:499: write oid 9 current snap is 63 2024-09-13T14:22:45.831 INFO:tasks.rados.rados.0.smithi151.stdout:499: seq_num 219 ranges {480733=537415,1782380=696402,3031400=537159} 2024-09-13T14:22:45.836 INFO:tasks.rados.rados.0.smithi151.stdout:499: writing smithi15186934-9 from 480733 to 1018148 tid 1 2024-09-13T14:22:45.844 INFO:tasks.rados.rados.0.smithi151.stdout:499: writing smithi15186934-9 from 1782380 to 2478782 tid 2 2024-09-13T14:22:45.850 INFO:tasks.rados.rados.0.smithi151.stdout:499: writing smithi15186934-9 from 3031400 to 3568559 tid 3 2024-09-13T14:22:45.850 INFO:tasks.rados.rados.0.smithi151.stdout:500: read oid 43 snap -1 2024-09-13T14:22:45.850 INFO:tasks.rados.rados.0.smithi151.stdout:500: expect (ObjNum 199 snap 57 seq_num 199) 2024-09-13T14:22:45.850 INFO:tasks.rados.rados.0.smithi151.stdout:501: write oid 38 current snap is 63 2024-09-13T14:22:45.850 INFO:tasks.rados.rados.0.smithi151.stdout:498: finishing rollback tid 1 to smithi15186934-41 2024-09-13T14:22:45.850 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 5 (ObjNum 593 snap 0 seq_num 3092788056) dirty dne 2024-09-13T14:22:45.850 INFO:tasks.rados.rados.0.smithi151.stdout:497: finishing write tid 3 to smithi15186934-5 2024-09-13T14:22:45.850 INFO:tasks.rados.rados.0.smithi151.stdout:497: finishing write tid 4 to smithi15186934-5 2024-09-13T14:22:45.851 INFO:tasks.rados.rados.0.smithi151.stdout:501: seq_num 220 ranges {497540=612664,1840862=440060,3058823=464784} 2024-09-13T14:22:45.856 INFO:tasks.rados.rados.0.smithi151.stdout:501: writing smithi15186934-38 from 497540 to 1110204 tid 1 2024-09-13T14:22:45.862 INFO:tasks.rados.rados.0.smithi151.stdout:501: writing smithi15186934-38 from 1840862 to 2280922 tid 2 2024-09-13T14:22:45.869 INFO:tasks.rados.rados.0.smithi151.stdout:501: writing smithi15186934-38 from 3058823 to 3523607 tid 3 2024-09-13T14:22:45.870 INFO:tasks.rados.rados.0.smithi151.stdout:498: done (4 left) 2024-09-13T14:22:45.870 INFO:tasks.rados.rados.0.smithi151.stdout:502: snap_remove snap 48 2024-09-13T14:22:46.216 INFO:tasks.rados.rados.0.smithi151.stdout:502: done (4 left) 2024-09-13T14:22:46.216 INFO:tasks.rados.rados.0.smithi151.stdout:503: delete oid 23 current snap is 63 2024-09-13T14:22:46.216 INFO:tasks.rados.rados.0.smithi151.stdout:497: finishing write tid 5 to smithi15186934-5 2024-09-13T14:22:46.216 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 127 (ObjNum 218 snap 63 seq_num 218) dirty exists 2024-09-13T14:22:46.216 INFO:tasks.rados.rados.0.smithi151.stdout:497: left oid 5 (ObjNum 218 snap 63 seq_num 218) 2024-09-13T14:22:46.216 INFO:tasks.rados.rados.0.smithi151.stdout:499: finishing write tid 1 to smithi15186934-9 2024-09-13T14:22:46.216 INFO:tasks.rados.rados.0.smithi151.stdout:499: finishing write tid 2 to smithi15186934-9 2024-09-13T14:22:46.216 INFO:tasks.rados.rados.0.smithi151.stdout:499: finishing write tid 3 to smithi15186934-9 2024-09-13T14:22:46.216 INFO:tasks.rados.rados.0.smithi151.stdout:499: finishing write tid 4 to smithi15186934-9 2024-09-13T14:22:46.217 INFO:tasks.rados.rados.0.smithi151.stdout:499: finishing write tid 5 to smithi15186934-9 2024-09-13T14:22:46.217 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 189 (ObjNum 219 snap 63 seq_num 219) dirty exists 2024-09-13T14:22:46.217 INFO:tasks.rados.rados.0.smithi151.stdout:499: left oid 9 (ObjNum 219 snap 63 seq_num 219) 2024-09-13T14:22:46.217 INFO:tasks.rados.rados.0.smithi151.stdout:501: finishing write tid 1 to smithi15186934-38 2024-09-13T14:22:46.217 INFO:tasks.rados.rados.0.smithi151.stdout:500: expect (ObjNum 199 snap 57 seq_num 199) 2024-09-13T14:22:46.412 INFO:tasks.rados.rados.0.smithi151.stdout:501: finishing write tid 2 to smithi15186934-38 2024-09-13T14:22:46.412 INFO:tasks.rados.rados.0.smithi151.stdout:501: finishing write tid 3 to smithi15186934-38 2024-09-13T14:22:46.412 INFO:tasks.rados.rados.0.smithi151.stdout:501: finishing write tid 4 to smithi15186934-38 2024-09-13T14:22:46.412 INFO:tasks.rados.rados.0.smithi151.stdout:501: finishing write tid 5 to smithi15186934-38 2024-09-13T14:22:46.412 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 113 (ObjNum 220 snap 63 seq_num 220) dirty exists 2024-09-13T14:22:46.413 INFO:tasks.rados.rados.0.smithi151.stdout:501: left oid 38 (ObjNum 220 snap 63 seq_num 220) 2024-09-13T14:22:46.413 INFO:tasks.rados.rados.0.smithi151.stdout:497: done (4 left) 2024-09-13T14:22:46.413 INFO:tasks.rados.rados.0.smithi151.stdout:499: done (3 left) 2024-09-13T14:22:46.413 INFO:tasks.rados.rados.0.smithi151.stdout:500: done (2 left) 2024-09-13T14:22:46.413 INFO:tasks.rados.rados.0.smithi151.stdout:501: done (1 left) 2024-09-13T14:22:46.413 INFO:tasks.rados.rados.0.smithi151.stdout:503: done (0 left) 2024-09-13T14:22:46.413 INFO:tasks.rados.rados.0.smithi151.stdout:504: snap_create 2024-09-13T14:22:46.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:46 smithi184 ceph-mon[33394]: osdmap e184: 8 total, 8 up, 8 in 2024-09-13T14:22:46.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:46 smithi184 ceph-mon[33394]: pgmap v48: 89 pgs: 89 active+clean; 200 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 469 KiB/s wr, 9 op/s 2024-09-13T14:22:46.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:46 smithi151 ceph-mon[29807]: osdmap e184: 8 total, 8 up, 8 in 2024-09-13T14:22:46.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:46 smithi151 ceph-mon[29807]: pgmap v48: 89 pgs: 89 active+clean; 200 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 469 KiB/s wr, 9 op/s 2024-09-13T14:22:46.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:46 smithi151 ceph-mon[41387]: osdmap e184: 8 total, 8 up, 8 in 2024-09-13T14:22:46.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:46 smithi151 ceph-mon[41387]: pgmap v48: 89 pgs: 89 active+clean; 200 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 469 KiB/s wr, 9 op/s 2024-09-13T14:22:47.225 INFO:tasks.rados.rados.0.smithi151.stdout:504: done (0 left) 2024-09-13T14:22:47.225 INFO:tasks.rados.rados.0.smithi151.stdout:505: write oid 32 current snap is 64 2024-09-13T14:22:47.225 INFO:tasks.rados.rados.0.smithi151.stdout:505: seq_num 222 ranges {531154=479515,1557826=443729,2713669=487328,3679295=51859} 2024-09-13T14:22:47.230 INFO:tasks.rados.rados.0.smithi151.stdout:505: writing smithi15186934-32 from 531154 to 1010669 tid 1 2024-09-13T14:22:47.236 INFO:tasks.rados.rados.0.smithi151.stdout:505: writing smithi15186934-32 from 1557826 to 2001555 tid 2 2024-09-13T14:22:47.242 INFO:tasks.rados.rados.0.smithi151.stdout:505: writing smithi15186934-32 from 2713669 to 3200997 tid 3 2024-09-13T14:22:47.246 INFO:tasks.rados.rados.0.smithi151.stdout:505: writing smithi15186934-32 from 3679295 to 3731154 tid 4 2024-09-13T14:22:47.246 INFO:tasks.rados.rados.0.smithi151.stdout:506: read oid 47 snap -1 2024-09-13T14:22:47.246 INFO:tasks.rados.rados.0.smithi151.stdout:506: expect (ObjNum 155 snap 38 seq_num 155) 2024-09-13T14:22:47.246 INFO:tasks.rados.rados.0.smithi151.stdout:507: write oid 21 current snap is 64 2024-09-13T14:22:47.246 INFO:tasks.rados.rados.0.smithi151.stdout:505: finishing write tid 1 to smithi15186934-32 2024-09-13T14:22:47.246 INFO:tasks.rados.rados.0.smithi151.stdout:507: seq_num 223 ranges {547961=554764,1616308=703740,2741092=698600} 2024-09-13T14:22:47.252 INFO:tasks.rados.rados.0.smithi151.stdout:507: writing smithi15186934-21 from 547961 to 1102725 tid 1 2024-09-13T14:22:47.258 INFO:tasks.rados.rados.0.smithi151.stdout:507: writing smithi15186934-21 from 1616308 to 2320048 tid 2 2024-09-13T14:22:47.265 INFO:tasks.rados.rados.0.smithi151.stdout:507: writing smithi15186934-21 from 2741092 to 3439692 tid 3 2024-09-13T14:22:47.265 INFO:tasks.rados.rados.0.smithi151.stdout:508: rollback oid 11 current snap is 64 2024-09-13T14:22:47.265 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 11 to 61 2024-09-13T14:22:47.265 INFO:tasks.rados.rados.0.smithi151.stdout:505: finishing write tid 2 to smithi15186934-32 2024-09-13T14:22:47.265 INFO:tasks.rados.rados.0.smithi151.stdout:505: finishing write tid 3 to smithi15186934-32 2024-09-13T14:22:47.265 INFO:tasks.rados.rados.0.smithi151.stdout:505: finishing write tid 4 to smithi15186934-32 2024-09-13T14:22:47.265 INFO:tasks.rados.rados.0.smithi151.stdout:505: finishing write tid 5 to smithi15186934-32 2024-09-13T14:22:47.265 INFO:tasks.rados.rados.0.smithi151.stdout:505: finishing write tid 6 to smithi15186934-32 2024-09-13T14:22:47.265 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 32 v 199 (ObjNum 222 snap 64 seq_num 222) dirty exists 2024-09-13T14:22:47.265 INFO:tasks.rados.rados.0.smithi151.stdout:505: left oid 32 (ObjNum 222 snap 64 seq_num 222) 2024-09-13T14:22:47.265 INFO:tasks.rados.rados.0.smithi151.stdout:506: expect (ObjNum 155 snap 38 seq_num 155) 2024-09-13T14:22:47.308 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:22:47.382 INFO:tasks.rados.rados.0.smithi151.stdout:507: finishing write tid 1 to smithi15186934-21 2024-09-13T14:22:47.382 INFO:tasks.rados.rados.0.smithi151.stdout:507: finishing write tid 2 to smithi15186934-21 2024-09-13T14:22:47.382 INFO:tasks.rados.rados.0.smithi151.stdout:505: done (3 left) 2024-09-13T14:22:47.383 INFO:tasks.rados.rados.0.smithi151.stdout:506: done (2 left) 2024-09-13T14:22:47.383 INFO:tasks.rados.rados.0.smithi151.stdout:509: snap_create 2024-09-13T14:22:47.383 INFO:tasks.rados.rados.0.smithi151.stdout:508: finishing rollback tid 0 to smithi15186934-11 2024-09-13T14:22:47.383 INFO:tasks.rados.rados.0.smithi151.stdout:508: finishing rollback tid 1 to smithi15186934-11 2024-09-13T14:22:47.383 INFO:tasks.rados.rados.0.smithi151.stdout:508: finishing rollback tid 2 to smithi15186934-11 2024-09-13T14:22:47.383 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 11 v 130 (ObjNum 197 snap 55 seq_num 197) dirty exists 2024-09-13T14:22:47.383 INFO:tasks.rados.rados.0.smithi151.stdout:507: finishing write tid 3 to smithi15186934-21 2024-09-13T14:22:47.383 INFO:tasks.rados.rados.0.smithi151.stdout:507: finishing write tid 4 to smithi15186934-21 2024-09-13T14:22:47.383 INFO:tasks.rados.rados.0.smithi151.stdout:507: finishing write tid 5 to smithi15186934-21 2024-09-13T14:22:47.383 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 21 v 86 (ObjNum 223 snap 64 seq_num 223) dirty exists 2024-09-13T14:22:47.383 INFO:tasks.rados.rados.0.smithi151.stdout:507: left oid 21 (ObjNum 223 snap 64 seq_num 223) 2024-09-13T14:22:47.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:47 smithi184 ceph-mon[33394]: osdmap e185: 8 total, 8 up, 8 in 2024-09-13T14:22:47.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:47 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:22:47.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:47 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/1391690257' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:47.604 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:22:47 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:22:47] "GET /metrics HTTP/1.1" 200 36409 "" "Prometheus/2.43.0" 2024-09-13T14:22:47.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:47 smithi151 ceph-mon[29807]: osdmap e185: 8 total, 8 up, 8 in 2024-09-13T14:22:47.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:47 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:22:47.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:47 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1391690257' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:47.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:47 smithi151 ceph-mon[41387]: osdmap e185: 8 total, 8 up, 8 in 2024-09-13T14:22:47.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:47 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:22:47.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:47 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/1391690257' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:47.802 INFO:tasks.rados.rados.0.smithi151.stdout:507: done (2 left) 2024-09-13T14:22:47.803 INFO:tasks.rados.rados.0.smithi151.stdout:508: done (1 left) 2024-09-13T14:22:47.803 INFO:tasks.rados.rados.0.smithi151.stdout:509: done (0 left) 2024-09-13T14:22:47.803 INFO:tasks.rados.rados.0.smithi151.stdout:510: delete oid 45 current snap is 65 2024-09-13T14:22:47.806 INFO:tasks.rados.rados.0.smithi151.stdout:510: done (0 left) 2024-09-13T14:22:47.806 INFO:tasks.rados.rados.0.smithi151.stdout:511: snap_create 2024-09-13T14:22:48.358 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:22:48 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[93024]: ts=2024-09-13T14:22:48.020Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003116803s 2024-09-13T14:22:48.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:48 smithi151 ceph-mon[41387]: osdmap e186: 8 total, 8 up, 8 in 2024-09-13T14:22:48.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:48 smithi151 ceph-mon[41387]: from='client.15564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:48.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:48 smithi151 ceph-mon[41387]: pgmap v51: 89 pgs: 89 active+clean; 195 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 2.4 MiB/s wr, 24 op/s 2024-09-13T14:22:48.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:48 smithi151 ceph-mon[41387]: from='client.25472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:48.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:48 smithi151 ceph-mon[41387]: osdmap e187: 8 total, 8 up, 8 in 2024-09-13T14:22:48.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:48 smithi151 ceph-mon[29807]: osdmap e186: 8 total, 8 up, 8 in 2024-09-13T14:22:48.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:48 smithi151 ceph-mon[29807]: from='client.15564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:48.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:48 smithi151 ceph-mon[29807]: pgmap v51: 89 pgs: 89 active+clean; 195 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 2.4 MiB/s wr, 24 op/s 2024-09-13T14:22:48.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:48 smithi151 ceph-mon[29807]: from='client.25472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:48.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:48 smithi151 ceph-mon[29807]: osdmap e187: 8 total, 8 up, 8 in 2024-09-13T14:22:48.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:48 smithi184 ceph-mon[33394]: osdmap e186: 8 total, 8 up, 8 in 2024-09-13T14:22:48.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:48 smithi184 ceph-mon[33394]: from='client.15564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:48.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:48 smithi184 ceph-mon[33394]: pgmap v51: 89 pgs: 89 active+clean; 195 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 2.4 MiB/s wr, 24 op/s 2024-09-13T14:22:48.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:48 smithi184 ceph-mon[33394]: from='client.25472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:48.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:48 smithi184 ceph-mon[33394]: osdmap e187: 8 total, 8 up, 8 in 2024-09-13T14:22:48.803 INFO:tasks.rados.rados.0.smithi151.stdout:511: done (0 left) 2024-09-13T14:22:48.803 INFO:tasks.rados.rados.0.smithi151.stdout:512: read oid 2 snap -1 2024-09-13T14:22:48.803 INFO:tasks.rados.rados.0.smithi151.stdout:512: expect deleted 2024-09-13T14:22:48.803 INFO:tasks.rados.rados.0.smithi151.stdout:513: rollback oid 25 current snap is 66 2024-09-13T14:22:48.803 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 25 to 64 2024-09-13T14:22:48.804 INFO:tasks.rados.rados.0.smithi151.stdout:514: write oid 50 current snap is 66 2024-09-13T14:22:48.804 INFO:tasks.rados.rados.0.smithi151.stdout:514: seq_num 225 ranges {581575=705262,1733272=707409,3195938=437497} 2024-09-13T14:22:48.810 INFO:tasks.rados.rados.0.smithi151.stdout:514: writing smithi15186934-50 from 581575 to 1286837 tid 1 2024-09-13T14:22:48.816 INFO:tasks.rados.rados.0.smithi151.stdout:514: writing smithi15186934-50 from 1733272 to 2440681 tid 2 2024-09-13T14:22:48.824 INFO:tasks.rados.rados.0.smithi151.stdout:514: writing smithi15186934-50 from 3195938 to 3633435 tid 3 2024-09-13T14:22:48.824 INFO:tasks.rados.rados.0.smithi151.stdout:515: read oid 34 snap -1 2024-09-13T14:22:48.824 INFO:tasks.rados.rados.0.smithi151.stdout:515: expect (ObjNum 112 snap 18 seq_num 112) 2024-09-13T14:22:48.824 INFO:tasks.rados.rados.0.smithi151.stdout:516: read oid 1 snap 43 2024-09-13T14:22:48.824 INFO:tasks.rados.rados.0.smithi151.stdout:516: expect deleted 2024-09-13T14:22:48.824 INFO:tasks.rados.rados.0.smithi151.stdout:517: read oid 39 snap -1 2024-09-13T14:22:48.824 INFO:tasks.rados.rados.0.smithi151.stdout:517: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-13T14:22:48.824 INFO:tasks.rados.rados.0.smithi151.stdout:518: write oid 27 current snap is 66 2024-09-13T14:22:48.824 INFO:tasks.rados.rados.0.smithi151.stdout:518: seq_num 226 ranges {598382=780511,2075401=451067,3107008=648769} 2024-09-13T14:22:48.832 INFO:tasks.rados.rados.0.smithi151.stdout:518: writing smithi15186934-27 from 598382 to 1378893 tid 1 2024-09-13T14:22:48.838 INFO:tasks.rados.rados.0.smithi151.stdout:518: writing smithi15186934-27 from 2075401 to 2526468 tid 2 2024-09-13T14:22:48.845 INFO:tasks.rados.rados.0.smithi151.stdout:518: writing smithi15186934-27 from 3107008 to 3755777 tid 3 2024-09-13T14:22:48.845 INFO:tasks.rados.rados.0.smithi151.stdout:519: write oid 44 current snap is 66 2024-09-13T14:22:48.845 INFO:tasks.rados.rados.0.smithi151.stdout:519: seq_num 227 ranges {615189=455760,1733883=711078,3134431=460041} 2024-09-13T14:22:48.850 INFO:tasks.rados.rados.0.smithi151.stdout:519: writing smithi15186934-44 from 615189 to 1070949 tid 1 2024-09-13T14:22:48.858 INFO:tasks.rados.rados.0.smithi151.stdout:519: writing smithi15186934-44 from 1733883 to 2444961 tid 2 2024-09-13T14:22:48.865 INFO:tasks.rados.rados.0.smithi151.stdout:519: writing smithi15186934-44 from 3134431 to 3594472 tid 3 2024-09-13T14:22:48.865 INFO:tasks.rados.rados.0.smithi151.stdout:520: read oid 3 snap 59 2024-09-13T14:22:48.865 INFO:tasks.rados.rados.0.smithi151.stdout:520: expect deleted 2024-09-13T14:22:48.865 INFO:tasks.rados.rados.0.smithi151.stdout:521: rollback oid 9 current snap is 66 2024-09-13T14:22:48.865 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 9 to 40 2024-09-13T14:22:48.865 INFO:tasks.rados.rados.0.smithi151.stdout:513: finishing rollback tid 0 to smithi15186934-25 2024-09-13T14:22:48.865 INFO:tasks.rados.rados.0.smithi151.stdout:513: finishing rollback tid 1 to smithi15186934-25 2024-09-13T14:22:48.865 INFO:tasks.rados.rados.0.smithi151.stdout:513: finishing rollback tid 2 to smithi15186934-25 2024-09-13T14:22:48.866 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 238 (ObjNum 194 snap 55 seq_num 194) dirty exists 2024-09-13T14:22:48.866 INFO:tasks.rados.rados.0.smithi151.stdout:514: finishing write tid 1 to smithi15186934-50 2024-09-13T14:22:48.866 INFO:tasks.rados.rados.0.smithi151.stdout:514: finishing write tid 2 to smithi15186934-50 2024-09-13T14:22:48.866 INFO:tasks.rados.rados.0.smithi151.stdout:512: done (9 left) 2024-09-13T14:22:48.866 INFO:tasks.rados.rados.0.smithi151.stdout:513: done (8 left) 2024-09-13T14:22:48.866 INFO:tasks.rados.rados.0.smithi151.stdout:516: done (7 left) 2024-09-13T14:22:48.866 INFO:tasks.rados.rados.0.smithi151.stdout:522: snap_create 2024-09-13T14:22:48.866 INFO:tasks.rados.rados.0.smithi151.stdout:515: expect (ObjNum 112 snap 18 seq_num 112) 2024-09-13T14:22:48.993 INFO:tasks.rados.rados.0.smithi151.stdout:514: finishing write tid 3 to smithi15186934-50 2024-09-13T14:22:48.993 INFO:tasks.rados.rados.0.smithi151.stdout:514: finishing write tid 4 to smithi15186934-50 2024-09-13T14:22:48.994 INFO:tasks.rados.rados.0.smithi151.stdout:514: finishing write tid 5 to smithi15186934-50 2024-09-13T14:22:48.994 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 50 v 118 (ObjNum 225 snap 66 seq_num 225) dirty exists 2024-09-13T14:22:48.994 INFO:tasks.rados.rados.0.smithi151.stdout:514: left oid 50 (ObjNum 225 snap 66 seq_num 225) 2024-09-13T14:22:48.994 INFO:tasks.rados.rados.0.smithi151.stdout:517: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-13T14:22:49.079 INFO:tasks.rados.rados.0.smithi151.stdout:518: finishing write tid 1 to smithi15186934-27 2024-09-13T14:22:49.079 INFO:tasks.rados.rados.0.smithi151.stdout:518: finishing write tid 2 to smithi15186934-27 2024-09-13T14:22:49.079 INFO:tasks.rados.rados.0.smithi151.stdout:518: finishing write tid 3 to smithi15186934-27 2024-09-13T14:22:49.079 INFO:tasks.rados.rados.0.smithi151.stdout:518: finishing write tid 4 to smithi15186934-27 2024-09-13T14:22:49.079 INFO:tasks.rados.rados.0.smithi151.stdout:518: finishing write tid 5 to smithi15186934-27 2024-09-13T14:22:49.079 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 27 v 126 (ObjNum 226 snap 66 seq_num 226) dirty exists 2024-09-13T14:22:49.079 INFO:tasks.rados.rados.0.smithi151.stdout:518: left oid 27 (ObjNum 226 snap 66 seq_num 226) 2024-09-13T14:22:49.079 INFO:tasks.rados.rados.0.smithi151.stdout:519: finishing write tid 1 to smithi15186934-44 2024-09-13T14:22:49.079 INFO:tasks.rados.rados.0.smithi151.stdout:519: finishing write tid 2 to smithi15186934-44 2024-09-13T14:22:49.079 INFO:tasks.rados.rados.0.smithi151.stdout:521: finishing rollback tid 0 to smithi15186934-9 2024-09-13T14:22:49.079 INFO:tasks.rados.rados.0.smithi151.stdout:519: finishing write tid 3 to smithi15186934-44 2024-09-13T14:22:49.080 INFO:tasks.rados.rados.0.smithi151.stdout:519: finishing write tid 4 to smithi15186934-44 2024-09-13T14:22:49.080 INFO:tasks.rados.rados.0.smithi151.stdout:521: finishing rollback tid 1 to smithi15186934-9 2024-09-13T14:22:49.080 INFO:tasks.rados.rados.0.smithi151.stdout:521: finishing rollback tid 2 to smithi15186934-9 2024-09-13T14:22:49.080 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 203 (ObjNum 151 snap 36 seq_num 151) dirty exists 2024-09-13T14:22:49.080 INFO:tasks.rados.rados.0.smithi151.stdout:519: finishing write tid 5 to smithi15186934-44 2024-09-13T14:22:49.080 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 172 (ObjNum 227 snap 66 seq_num 227) dirty exists 2024-09-13T14:22:49.080 INFO:tasks.rados.rados.0.smithi151.stdout:519: left oid 44 (ObjNum 227 snap 66 seq_num 227) 2024-09-13T14:22:49.808 INFO:tasks.rados.rados.0.smithi151.stdout:514: done (7 left) 2024-09-13T14:22:49.808 INFO:tasks.rados.rados.0.smithi151.stdout:515: done (6 left) 2024-09-13T14:22:49.809 INFO:tasks.rados.rados.0.smithi151.stdout:517: done (5 left) 2024-09-13T14:22:49.809 INFO:tasks.rados.rados.0.smithi151.stdout:518: done (4 left) 2024-09-13T14:22:49.809 INFO:tasks.rados.rados.0.smithi151.stdout:519: done (3 left) 2024-09-13T14:22:49.809 INFO:tasks.rados.rados.0.smithi151.stdout:520: done (2 left) 2024-09-13T14:22:49.809 INFO:tasks.rados.rados.0.smithi151.stdout:521: done (1 left) 2024-09-13T14:22:49.809 INFO:tasks.rados.rados.0.smithi151.stdout:522: done (0 left) 2024-09-13T14:22:49.809 INFO:tasks.rados.rados.0.smithi151.stdout:523: delete oid 22 current snap is 67 2024-09-13T14:22:49.811 INFO:tasks.rados.rados.0.smithi151.stdout:523: done (0 left) 2024-09-13T14:22:49.811 INFO:tasks.rados.rados.0.smithi151.stdout:524: snap_create 2024-09-13T14:22:50.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:49 smithi184 ceph-mon[33394]: osdmap e188: 8 total, 8 up, 8 in 2024-09-13T14:22:50.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:49 smithi184 ceph-mon[33394]: pgmap v54: 89 pgs: 89 active+clean; 195 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 2.4 MiB/s wr, 24 op/s 2024-09-13T14:22:50.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:49 smithi151 ceph-mon[29807]: osdmap e188: 8 total, 8 up, 8 in 2024-09-13T14:22:50.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:49 smithi151 ceph-mon[29807]: pgmap v54: 89 pgs: 89 active+clean; 195 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 2.4 MiB/s wr, 24 op/s 2024-09-13T14:22:50.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:49 smithi151 ceph-mon[41387]: osdmap e188: 8 total, 8 up, 8 in 2024-09-13T14:22:50.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:49 smithi151 ceph-mon[41387]: pgmap v54: 89 pgs: 89 active+clean; 195 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 2.4 MiB/s wr, 24 op/s 2024-09-13T14:22:50.209 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:22:50.809 INFO:tasks.rados.rados.0.smithi151.stdout:524: done (0 left) 2024-09-13T14:22:50.809 INFO:tasks.rados.rados.0.smithi151.stdout:525: snap_remove snap 67 2024-09-13T14:22:51.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:50 smithi184 ceph-mon[33394]: osdmap e189: 8 total, 8 up, 8 in 2024-09-13T14:22:51.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:50 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/1747178243' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:51.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:50 smithi184 ceph-mon[33394]: from='client.15582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:51.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:50 smithi184 ceph-mon[33394]: from='client.25490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:51.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:50 smithi151 ceph-mon[29807]: osdmap e189: 8 total, 8 up, 8 in 2024-09-13T14:22:51.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:50 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1747178243' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:51.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:50 smithi151 ceph-mon[29807]: from='client.15582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:51.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:50 smithi151 ceph-mon[29807]: from='client.25490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:51.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:50 smithi151 ceph-mon[41387]: osdmap e189: 8 total, 8 up, 8 in 2024-09-13T14:22:51.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:50 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/1747178243' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:51.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:50 smithi151 ceph-mon[41387]: from='client.15582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:51.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:50 smithi151 ceph-mon[41387]: from='client.25490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:51.814 INFO:tasks.rados.rados.0.smithi151.stdout:525: done (0 left) 2024-09-13T14:22:51.814 INFO:tasks.rados.rados.0.smithi151.stdout:526: write oid 45 current snap is 68 2024-09-13T14:22:51.814 INFO:tasks.rados.rados.0.smithi151.stdout:526: seq_num 229 ranges {648803=722611,1967200=714747,3421983=426820} 2024-09-13T14:22:51.822 INFO:tasks.rados.rados.0.smithi151.stdout:526: writing smithi15186934-45 from 648803 to 1371414 tid 1 2024-09-13T14:22:51.830 INFO:tasks.rados.rados.0.smithi151.stdout:526: writing smithi15186934-45 from 1967200 to 2681947 tid 2 2024-09-13T14:22:51.837 INFO:tasks.rados.rados.0.smithi151.stdout:526: writing smithi15186934-45 from 3421983 to 3848803 tid 3 2024-09-13T14:22:51.837 INFO:tasks.rados.rados.0.smithi151.stdout:527: snap_remove snap 62 2024-09-13T14:22:52.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:51 smithi184 ceph-mon[33394]: osdmap e190: 8 total, 8 up, 8 in 2024-09-13T14:22:52.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:51 smithi184 ceph-mon[33394]: pgmap v57: 89 pgs: 89 active+clean; 195 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:52.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:51 smithi151 ceph-mon[29807]: osdmap e190: 8 total, 8 up, 8 in 2024-09-13T14:22:52.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:51 smithi151 ceph-mon[29807]: pgmap v57: 89 pgs: 89 active+clean; 195 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:52.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:51 smithi151 ceph-mon[41387]: osdmap e190: 8 total, 8 up, 8 in 2024-09-13T14:22:52.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:51 smithi151 ceph-mon[41387]: pgmap v57: 89 pgs: 89 active+clean; 195 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:22:52.805 INFO:tasks.rados.rados.0.smithi151.stdout:527: done (1 left) 2024-09-13T14:22:52.805 INFO:tasks.rados.rados.0.smithi151.stdout:528: rollback oid 12 current snap is 68 2024-09-13T14:22:52.805 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 12 to 66 2024-09-13T14:22:52.805 INFO:tasks.rados.rados.0.smithi151.stdout:526: finishing write tid 1 to smithi15186934-45 2024-09-13T14:22:52.806 INFO:tasks.rados.rados.0.smithi151.stdout:526: finishing write tid 2 to smithi15186934-45 2024-09-13T14:22:52.806 INFO:tasks.rados.rados.0.smithi151.stdout:526: finishing write tid 3 to smithi15186934-45 2024-09-13T14:22:52.806 INFO:tasks.rados.rados.0.smithi151.stdout:526: finishing write tid 4 to smithi15186934-45 2024-09-13T14:22:52.806 INFO:tasks.rados.rados.0.smithi151.stdout:526: finishing write tid 5 to smithi15186934-45 2024-09-13T14:22:52.806 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 242 (ObjNum 229 snap 68 seq_num 229) dirty exists 2024-09-13T14:22:52.806 INFO:tasks.rados.rados.0.smithi151.stdout:526: left oid 45 (ObjNum 229 snap 68 seq_num 229) 2024-09-13T14:22:52.806 INFO:tasks.rados.rados.0.smithi151.stdout:526: done (1 left) 2024-09-13T14:22:52.806 INFO:tasks.rados.rados.0.smithi151.stdout:529: snap_create 2024-09-13T14:22:52.811 INFO:tasks.rados.rados.0.smithi151.stdout:528: finishing rollback tid 0 to smithi15186934-12 2024-09-13T14:22:52.811 INFO:tasks.rados.rados.0.smithi151.stdout:528: finishing rollback tid 1 to smithi15186934-12 2024-09-13T14:22:52.812 INFO:tasks.rados.rados.0.smithi151.stdout:528: finishing rollback tid 2 to smithi15186934-12 2024-09-13T14:22:52.812 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 12 v 79 (ObjNum 167 snap 43 seq_num 167) dirty exists 2024-09-13T14:22:53.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:52 smithi184 ceph-mon[33394]: osdmap e191: 8 total, 8 up, 8 in 2024-09-13T14:22:53.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:52 smithi184 ceph-mon[33394]: osdmap e192: 8 total, 8 up, 8 in 2024-09-13T14:22:53.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:52 smithi151 ceph-mon[29807]: osdmap e191: 8 total, 8 up, 8 in 2024-09-13T14:22:53.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:52 smithi151 ceph-mon[29807]: osdmap e192: 8 total, 8 up, 8 in 2024-09-13T14:22:53.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:52 smithi151 ceph-mon[41387]: osdmap e191: 8 total, 8 up, 8 in 2024-09-13T14:22:53.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:52 smithi151 ceph-mon[41387]: osdmap e192: 8 total, 8 up, 8 in 2024-09-13T14:22:53.108 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:22:52 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[30056]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:22:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:22:53.122 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:22:53.806 INFO:tasks.rados.rados.0.smithi151.stdout:528: done (1 left) 2024-09-13T14:22:53.904 INFO:tasks.rados.rados.0.smithi151.stdout:529: done (0 left) 2024-09-13T14:22:53.904 INFO:tasks.rados.rados.0.smithi151.stdout:530: rollback oid 7 current snap is 69 2024-09-13T14:22:53.904 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 7 to 59 2024-09-13T14:22:53.904 INFO:tasks.rados.rados.0.smithi151.stdout:531: read oid 4 snap -1 2024-09-13T14:22:53.904 INFO:tasks.rados.rados.0.smithi151.stdout:531: expect (ObjNum 192 snap 51 seq_num 192) 2024-09-13T14:22:53.904 INFO:tasks.rados.rados.0.smithi151.stdout:532: write oid 34 current snap is 69 2024-09-13T14:22:53.904 INFO:tasks.rados.rados.0.smithi151.stdout:532: seq_num 230 ranges {665610=797860,1909329=458405,2816700=410210} 2024-09-13T14:22:53.905 INFO:tasks.rados.rados.0.smithi151.stdout:532: writing smithi15186934-34 from 665610 to 1463470 tid 1 2024-09-13T14:22:53.906 INFO:tasks.rados.rados.0.smithi151.stdout:532: writing smithi15186934-34 from 1909329 to 2367734 tid 2 2024-09-13T14:22:53.906 INFO:tasks.rados.rados.0.smithi151.stdout:532: writing smithi15186934-34 from 2816700 to 3226910 tid 3 2024-09-13T14:22:53.906 INFO:tasks.rados.rados.0.smithi151.stdout:533: snap_create 2024-09-13T14:22:53.906 INFO:tasks.rados.rados.0.smithi151.stdout:530: finishing rollback tid 0 to smithi15186934-7 2024-09-13T14:22:53.906 INFO:tasks.rados.rados.0.smithi151.stdout:530: finishing rollback tid 1 to smithi15186934-7 2024-09-13T14:22:53.906 INFO:tasks.rados.rados.0.smithi151.stdout:530: finishing rollback tid 2 to smithi15186934-7 2024-09-13T14:22:53.906 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 293 (ObjNum 171 snap 44 seq_num 171) dirty exists 2024-09-13T14:22:53.906 INFO:tasks.rados.rados.0.smithi151.stdout:532: finishing write tid 1 to smithi15186934-34 2024-09-13T14:22:53.906 INFO:tasks.rados.rados.0.smithi151.stdout:531: expect (ObjNum 192 snap 51 seq_num 192) 2024-09-13T14:22:53.981 INFO:tasks.rados.rados.0.smithi151.stdout:532: finishing write tid 2 to smithi15186934-34 2024-09-13T14:22:53.982 INFO:tasks.rados.rados.0.smithi151.stdout:532: finishing write tid 3 to smithi15186934-34 2024-09-13T14:22:53.982 INFO:tasks.rados.rados.0.smithi151.stdout:532: finishing write tid 4 to smithi15186934-34 2024-09-13T14:22:53.982 INFO:tasks.rados.rados.0.smithi151.stdout:532: finishing write tid 5 to smithi15186934-34 2024-09-13T14:22:53.982 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 196 (ObjNum 230 snap 69 seq_num 230) dirty exists 2024-09-13T14:22:53.982 INFO:tasks.rados.rados.0.smithi151.stdout:532: left oid 34 (ObjNum 230 snap 69 seq_num 230) 2024-09-13T14:22:54.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:53 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/3742427317' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:54.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:53 smithi184 ceph-mon[33394]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:54.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:53 smithi184 ceph-mon[33394]: from='client.15606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:54.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:53 smithi184 ceph-mon[33394]: pgmap v60: 89 pgs: 89 active+clean; 213 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.5 MiB/s rd, 2.5 MiB/s wr, 17 op/s 2024-09-13T14:22:54.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:53 smithi184 ceph-mon[33394]: osdmap e193: 8 total, 8 up, 8 in 2024-09-13T14:22:54.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:53 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/3742427317' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:54.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:53 smithi151 ceph-mon[29807]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:54.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:53 smithi151 ceph-mon[29807]: from='client.15606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:54.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:53 smithi151 ceph-mon[29807]: pgmap v60: 89 pgs: 89 active+clean; 213 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.5 MiB/s rd, 2.5 MiB/s wr, 17 op/s 2024-09-13T14:22:54.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:53 smithi151 ceph-mon[29807]: osdmap e193: 8 total, 8 up, 8 in 2024-09-13T14:22:54.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:53 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/3742427317' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:54.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:53 smithi151 ceph-mon[41387]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:54.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:53 smithi151 ceph-mon[41387]: from='client.15606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:54.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:53 smithi151 ceph-mon[41387]: pgmap v60: 89 pgs: 89 active+clean; 213 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.5 MiB/s rd, 2.5 MiB/s wr, 17 op/s 2024-09-13T14:22:54.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:53 smithi151 ceph-mon[41387]: osdmap e193: 8 total, 8 up, 8 in 2024-09-13T14:22:54.386 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:54 smithi184 systemd[1]: Stopping Ceph prometheus.a for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:22:54.387 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:54 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:22:54.375Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-13T14:22:54.387 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:54 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:22:54.375Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-13T14:22:54.387 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:54 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:22:54.375Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-13T14:22:54.387 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:54 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:22:54.375Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-13T14:22:54.387 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:54 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:22:54.375Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-13T14:22:54.387 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:54 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:22:54.375Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-13T14:22:54.387 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:54 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:22:54.375Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-13T14:22:54.387 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:54 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:22:54.375Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-13T14:22:54.387 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:54 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:22:54.375Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-13T14:22:54.387 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:54 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:22:54.376Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-13T14:22:54.387 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:54 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:22:54.376Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-13T14:22:54.387 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:54 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[66180]: ts=2024-09-13T14:22:54.376Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-13T14:22:54.388 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:54 smithi184 podman[77775]: 2024-09-13 14:22:54.385907896 +0000 UTC m=+0.109948120 container died 0d5bc4586d6c6033533a937189a66d9827ed310b742b32352b35774eb128a2ad (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:22:54.824 INFO:tasks.rados.rados.0.smithi151.stdout:530: done (3 left) 2024-09-13T14:22:54.824 INFO:tasks.rados.rados.0.smithi151.stdout:531: done (2 left) 2024-09-13T14:22:54.825 INFO:tasks.rados.rados.0.smithi151.stdout:532: done (1 left) 2024-09-13T14:22:54.825 INFO:tasks.rados.rados.0.smithi151.stdout:533: done (0 left) 2024-09-13T14:22:54.825 INFO:tasks.rados.rados.0.smithi151.stdout:534: delete oid 49 current snap is 70 2024-09-13T14:22:54.828 INFO:tasks.rados.rados.0.smithi151.stdout:534: done (0 left) 2024-09-13T14:22:54.828 INFO:tasks.rados.rados.0.smithi151.stdout:535: snap_remove snap 60 2024-09-13T14:22:55.103 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:54 smithi184 podman[77775]: 2024-09-13 14:22:54.898593569 +0000 UTC m=+0.622633839 container cleanup 0d5bc4586d6c6033533a937189a66d9827ed310b742b32352b35774eb128a2ad (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:22:55.103 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:54 smithi184 bash[77775]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a 2024-09-13T14:22:55.590 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:55 smithi184 podman[77792]: 2024-09-13 14:22:55.335575194 +0000 UTC m=+0.951885650 container remove 0d5bc4586d6c6033533a937189a66d9827ed310b742b32352b35774eb128a2ad (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:22:55.590 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:55 smithi184 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-13T14:22:55.590 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:55 smithi184 systemd[1]: Stopped Ceph prometheus.a for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:22:55.590 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:55 smithi184 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 1.724s CPU time. 2024-09-13T14:22:55.833 INFO:tasks.rados.rados.0.smithi151.stdout:535: done (0 left) 2024-09-13T14:22:55.833 INFO:tasks.rados.rados.0.smithi151.stdout:536: write oid 45 current snap is 70 2024-09-13T14:22:55.833 INFO:tasks.rados.rados.0.smithi151.stdout:536: seq_num 232 ranges {699224=548358,2026293=462074,3271546=432754} 2024-09-13T14:22:55.841 INFO:tasks.rados.rados.0.smithi151.stdout:536: writing smithi15186934-45 from 699224 to 1247582 tid 1 2024-09-13T14:22:55.849 INFO:tasks.rados.rados.0.smithi151.stdout:536: writing smithi15186934-45 from 2026293 to 2488367 tid 2 2024-09-13T14:22:55.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:55 smithi184 ceph-mon[33394]: osdmap e194: 8 total, 8 up, 8 in 2024-09-13T14:22:55.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:55 smithi184 ceph-mon[33394]: pgmap v63: 89 pgs: 89 active+clean; 213 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.5 MiB/s rd, 2.5 MiB/s wr, 17 op/s 2024-09-13T14:22:55.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:55 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/1581287258' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:55.857 INFO:tasks.rados.rados.0.smithi151.stdout:536: writing smithi15186934-45 from 3271546 to 3704300 tid 3 2024-09-13T14:22:55.858 INFO:tasks.rados.rados.0.smithi151.stdout:537: delete oid 10 current snap is 70 2024-09-13T14:22:55.858 INFO:tasks.rados.rados.0.smithi151.stdout:536: finishing write tid 1 to smithi15186934-45 2024-09-13T14:22:55.858 INFO:tasks.rados.rados.0.smithi151.stdout:536: finishing write tid 2 to smithi15186934-45 2024-09-13T14:22:55.859 INFO:tasks.rados.rados.0.smithi151.stdout:537: done (1 left) 2024-09-13T14:22:55.859 INFO:tasks.rados.rados.0.smithi151.stdout:538: write oid 48 current snap is 70 2024-09-13T14:22:55.859 INFO:tasks.rados.rados.0.smithi151.stdout:538: seq_num 234 ranges {732838=698856,2026904=465743,3210039=571651} 2024-09-13T14:22:55.869 INFO:tasks.rados.rados.0.smithi151.stdout:538: writing smithi15186934-48 from 732838 to 1431694 tid 1 2024-09-13T14:22:55.876 INFO:tasks.rados.rados.0.smithi151.stdout:538: writing smithi15186934-48 from 2026904 to 2492647 tid 2 2024-09-13T14:22:55.885 INFO:tasks.rados.rados.0.smithi151.stdout:538: writing smithi15186934-48 from 3210039 to 3781690 tid 3 2024-09-13T14:22:55.885 INFO:tasks.rados.rados.0.smithi151.stdout:539: snap_create 2024-09-13T14:22:55.885 INFO:tasks.rados.rados.0.smithi151.stdout:536: finishing write tid 3 to smithi15186934-45 2024-09-13T14:22:55.885 INFO:tasks.rados.rados.0.smithi151.stdout:536: finishing write tid 4 to smithi15186934-45 2024-09-13T14:22:55.885 INFO:tasks.rados.rados.0.smithi151.stdout:536: finishing write tid 5 to smithi15186934-45 2024-09-13T14:22:55.885 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 263 (ObjNum 232 snap 70 seq_num 232) dirty exists 2024-09-13T14:22:55.885 INFO:tasks.rados.rados.0.smithi151.stdout:536: left oid 45 (ObjNum 232 snap 70 seq_num 232) 2024-09-13T14:22:55.886 INFO:tasks.rados.rados.0.smithi151.stdout:538: finishing write tid 1 to smithi15186934-48 2024-09-13T14:22:55.886 INFO:tasks.rados.rados.0.smithi151.stdout:538: finishing write tid 2 to smithi15186934-48 2024-09-13T14:22:55.888 INFO:tasks.rados.rados.0.smithi151.stdout:538: finishing write tid 3 to smithi15186934-48 2024-09-13T14:22:55.888 INFO:tasks.rados.rados.0.smithi151.stdout:538: finishing write tid 4 to smithi15186934-48 2024-09-13T14:22:55.888 INFO:tasks.rados.rados.0.smithi151.stdout:538: finishing write tid 5 to smithi15186934-48 2024-09-13T14:22:55.888 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 48 v 268 (ObjNum 234 snap 70 seq_num 234) dirty exists 2024-09-13T14:22:55.888 INFO:tasks.rados.rados.0.smithi151.stdout:538: left oid 48 (ObjNum 234 snap 70 seq_num 234) 2024-09-13T14:22:56.051 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:22:56.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:55 smithi151 ceph-mon[29807]: osdmap e194: 8 total, 8 up, 8 in 2024-09-13T14:22:56.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:55 smithi151 ceph-mon[29807]: pgmap v63: 89 pgs: 89 active+clean; 213 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.5 MiB/s rd, 2.5 MiB/s wr, 17 op/s 2024-09-13T14:22:56.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:55 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1581287258' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:56.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:55 smithi151 ceph-mon[41387]: osdmap e194: 8 total, 8 up, 8 in 2024-09-13T14:22:56.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:55 smithi151 ceph-mon[41387]: pgmap v63: 89 pgs: 89 active+clean; 213 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.5 MiB/s rd, 2.5 MiB/s wr, 17 op/s 2024-09-13T14:22:56.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:55 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/1581287258' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:56.354 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:55 smithi184 systemd[1]: Starting Ceph prometheus.a for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:22:56.354 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:56 smithi184 podman[77926]: 2024-09-13 14:22:55.974752084 +0000 UTC m=+0.016795403 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-13T14:22:56.836 INFO:tasks.rados.rados.0.smithi151.stdout:536: done (2 left) 2024-09-13T14:22:56.837 INFO:tasks.rados.rados.0.smithi151.stdout:538: done (1 left) 2024-09-13T14:22:56.837 INFO:tasks.rados.rados.0.smithi151.stdout:539: done (0 left) 2024-09-13T14:22:56.837 INFO:tasks.rados.rados.0.smithi151.stdout:540: read oid 42 snap -1 2024-09-13T14:22:56.837 INFO:tasks.rados.rados.0.smithi151.stdout:540: expect (ObjNum 215 snap 63 seq_num 215) 2024-09-13T14:22:56.837 INFO:tasks.rados.rados.0.smithi151.stdout:541: snap_create 2024-09-13T14:22:56.844 INFO:tasks.rados.rados.0.smithi151.stdout:540: expect (ObjNum 215 snap 63 seq_num 215) 2024-09-13T14:22:57.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:56 smithi184 ceph-mon[33394]: osdmap e195: 8 total, 8 up, 8 in 2024-09-13T14:22:57.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:56 smithi184 ceph-mon[33394]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:57.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:56 smithi184 ceph-mon[33394]: from='client.25526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:57.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:56 smithi184 podman[77926]: 2024-09-13 14:22:56.988549447 +0000 UTC m=+1.030592766 container create 35d43c151982cfbfdafb8f16405bd62ec2caab8dd8e8fd25c4ad26ce2195188e (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:22:57.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:56 smithi151 ceph-mon[29807]: osdmap e195: 8 total, 8 up, 8 in 2024-09-13T14:22:57.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:56 smithi151 ceph-mon[29807]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:57.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:56 smithi151 ceph-mon[29807]: from='client.25526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:57.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:56 smithi151 ceph-mon[41387]: osdmap e195: 8 total, 8 up, 8 in 2024-09-13T14:22:57.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:56 smithi151 ceph-mon[41387]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:57.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:56 smithi151 ceph-mon[41387]: from='client.25526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:57.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:57 smithi184 podman[77926]: 2024-09-13 14:22:57.274077884 +0000 UTC m=+1.316121204 container init 35d43c151982cfbfdafb8f16405bd62ec2caab8dd8e8fd25c4ad26ce2195188e (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:22:57.605 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:57 smithi184 podman[77926]: 2024-09-13 14:22:57.279394201 +0000 UTC m=+1.321437521 container start 35d43c151982cfbfdafb8f16405bd62ec2caab8dd8e8fd25c4ad26ce2195188e (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:22:57.605 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:22:57.301Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-13T14:22:57.605 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:22:57.301Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-13T14:22:57.605 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:22:57.301Z caller=main.go:623 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi184 (none))" 2024-09-13T14:22:57.605 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:22:57.301Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-13T14:22:57.605 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:22:57.301Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-13T14:22:57.605 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:22:57.303Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-13T14:22:57.605 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:22:57.303Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-13T14:22:57.605 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:22:57.305Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-13T14:22:57.606 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:22:57.305Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-13T14:22:57.606 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:22:57.307Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-13T14:22:57.606 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:22:57.307Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.979µs 2024-09-13T14:22:57.606 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:22:57.307Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-13T14:22:57.606 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:22:57.308Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-09-13T14:22:57.606 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:22:57.320Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-09-13T14:22:57.606 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:22:57.320Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-09-13T14:22:57.606 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:22:57.320Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=26.249µs wal_replay_duration=12.706034ms wbl_replay_duration=146ns total_replay_duration=12.751321ms 2024-09-13T14:22:57.606 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:22:57.323Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-13T14:22:57.607 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:22:57.323Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-13T14:22:57.607 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:22:57.323Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-13T14:22:57.607 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:22:57.341Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=17.808636ms db_storage=850ns remote_storage=1.234µs web_handler=449ns query_engine=793ns scrape=423.638µs scrape_sd=114.151µs notify=10.42µs notify_sd=7.915µs rules=16.55822ms tracing=24.81µs 2024-09-13T14:22:57.607 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:22:57.341Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-13T14:22:57.607 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:22:57.341Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-13T14:22:57.607 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:57 smithi184 bash[77926]: 35d43c151982cfbfdafb8f16405bd62ec2caab8dd8e8fd25c4ad26ce2195188e 2024-09-13T14:22:57.607 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:22:57 smithi184 systemd[1]: Started Ceph prometheus.a for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:22:57.837 INFO:tasks.rados.rados.0.smithi151.stdout:540: done (1 left) 2024-09-13T14:22:57.837 INFO:tasks.rados.rados.0.smithi151.stdout:541: done (0 left) 2024-09-13T14:22:57.837 INFO:tasks.rados.rados.0.smithi151.stdout:542: delete oid 14 current snap is 72 2024-09-13T14:22:57.839 INFO:tasks.rados.rados.0.smithi151.stdout:542: done (0 left) 2024-09-13T14:22:57.840 INFO:tasks.rados.rados.0.smithi151.stdout:543: write oid 24 current snap is 72 2024-09-13T14:22:57.840 INFO:tasks.rados.rados.0.smithi151.stdout:543: seq_num 236 ranges {766452=565707,1860221=469412,2981238=594195} 2024-09-13T14:22:57.847 INFO:tasks.rados.rados.0.smithi151.stdout:543: writing smithi15186934-24 from 766452 to 1332159 tid 1 2024-09-13T14:22:57.852 INFO:tasks.rados.rados.0.smithi151.stdout:543: writing smithi15186934-24 from 1860221 to 2329633 tid 2 2024-09-13T14:22:57.860 INFO:tasks.rados.rados.0.smithi151.stdout:543: writing smithi15186934-24 from 2981238 to 3575433 tid 3 2024-09-13T14:22:57.860 INFO:tasks.rados.rados.0.smithi151.stdout:544: read oid 16 snap 43 2024-09-13T14:22:57.860 INFO:tasks.rados.rados.0.smithi151.stdout:544: expect (ObjNum 117 snap 20 seq_num 117) 2024-09-13T14:22:57.860 INFO:tasks.rados.rados.0.smithi151.stdout:545: snap_remove snap 61 2024-09-13T14:22:58.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:57 smithi184 ceph-mon[33394]: osdmap e196: 8 total, 8 up, 8 in 2024-09-13T14:22:58.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:57 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:58.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:57 smithi184 ceph-mon[33394]: pgmap v66: 89 pgs: 89 active+clean; 219 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.6 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-13T14:22:58.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:57 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:58.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:57 smithi184 ceph-mon[33394]: Reconfiguring grafana.a (dependencies changed)... 2024-09-13T14:22:58.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:57 smithi184 ceph-mon[33394]: Reconfiguring daemon grafana.a on smithi184 2024-09-13T14:22:58.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:57 smithi151 ceph-mon[29807]: osdmap e196: 8 total, 8 up, 8 in 2024-09-13T14:22:58.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:57 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:58.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:57 smithi151 ceph-mon[29807]: pgmap v66: 89 pgs: 89 active+clean; 219 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.6 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-13T14:22:58.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:57 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:58.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:57 smithi151 ceph-mon[29807]: Reconfiguring grafana.a (dependencies changed)... 2024-09-13T14:22:58.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:57 smithi151 ceph-mon[29807]: Reconfiguring daemon grafana.a on smithi184 2024-09-13T14:22:58.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:57 smithi151 ceph-mon[41387]: osdmap e196: 8 total, 8 up, 8 in 2024-09-13T14:22:58.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:57 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:58.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:57 smithi151 ceph-mon[41387]: pgmap v66: 89 pgs: 89 active+clean; 219 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.6 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-13T14:22:58.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:57 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:22:58.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:57 smithi151 ceph-mon[41387]: Reconfiguring grafana.a (dependencies changed)... 2024-09-13T14:22:58.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:57 smithi151 ceph-mon[41387]: Reconfiguring daemon grafana.a on smithi184 2024-09-13T14:22:58.845 INFO:tasks.rados.rados.0.smithi151.stdout:545: done (2 left) 2024-09-13T14:22:58.845 INFO:tasks.rados.rados.0.smithi151.stdout:546: snap_create 2024-09-13T14:22:58.845 INFO:tasks.rados.rados.0.smithi151.stdout:543: finishing write tid 1 to smithi15186934-24 2024-09-13T14:22:58.845 INFO:tasks.rados.rados.0.smithi151.stdout:543: finishing write tid 2 to smithi15186934-24 2024-09-13T14:22:58.845 INFO:tasks.rados.rados.0.smithi151.stdout:543: finishing write tid 3 to smithi15186934-24 2024-09-13T14:22:58.845 INFO:tasks.rados.rados.0.smithi151.stdout:543: finishing write tid 4 to smithi15186934-24 2024-09-13T14:22:58.845 INFO:tasks.rados.rados.0.smithi151.stdout:543: finishing write tid 5 to smithi15186934-24 2024-09-13T14:22:58.845 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 24 v 301 (ObjNum 236 snap 72 seq_num 236) dirty exists 2024-09-13T14:22:58.845 INFO:tasks.rados.rados.0.smithi151.stdout:543: left oid 24 (ObjNum 236 snap 72 seq_num 236) 2024-09-13T14:22:58.846 INFO:tasks.rados.rados.0.smithi151.stdout:544: expect (ObjNum 117 snap 20 seq_num 117) 2024-09-13T14:22:58.957 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:22:59.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:58 smithi184 ceph-mon[33394]: osdmap e197: 8 total, 8 up, 8 in 2024-09-13T14:22:59.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:58 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/2786279157' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:59.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:58 smithi151 ceph-mon[29807]: osdmap e197: 8 total, 8 up, 8 in 2024-09-13T14:22:59.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:58 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/2786279157' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:59.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:58 smithi151 ceph-mon[41387]: osdmap e197: 8 total, 8 up, 8 in 2024-09-13T14:22:59.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:58 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/2786279157' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:22:59.845 INFO:tasks.rados.rados.0.smithi151.stdout:543: done (2 left) 2024-09-13T14:22:59.845 INFO:tasks.rados.rados.0.smithi151.stdout:544: done (1 left) 2024-09-13T14:22:59.846 INFO:tasks.rados.rados.0.smithi151.stdout:546: done (0 left) 2024-09-13T14:22:59.846 INFO:tasks.rados.rados.0.smithi151.stdout:547: write oid 42 current snap is 73 2024-09-13T14:22:59.846 INFO:tasks.rados.rados.0.smithi151.stdout:547: seq_num 237 ranges {783259=640956,1918703=613070,3008661=405467,3821939=161320} 2024-09-13T14:22:59.853 INFO:tasks.rados.rados.0.smithi151.stdout:547: writing smithi15186934-42 from 783259 to 1424215 tid 1 2024-09-13T14:22:59.859 INFO:tasks.rados.rados.0.smithi151.stdout:547: writing smithi15186934-42 from 1918703 to 2531773 tid 2 2024-09-13T14:22:59.864 INFO:tasks.rados.rados.0.smithi151.stdout:547: writing smithi15186934-42 from 3008661 to 3414128 tid 3 2024-09-13T14:22:59.868 INFO:tasks.rados.rados.0.smithi151.stdout:547: writing smithi15186934-42 from 3821939 to 3983259 tid 4 2024-09-13T14:22:59.868 INFO:tasks.rados.rados.0.smithi151.stdout:548: snap_remove snap 47 2024-09-13T14:23:00.099 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:59 smithi184 ceph-mon[33394]: osdmap e198: 8 total, 8 up, 8 in 2024-09-13T14:23:00.099 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:59 smithi184 ceph-mon[33394]: from='client.15636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:00.099 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:59 smithi184 ceph-mon[33394]: from='client.25544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:00.099 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:59 smithi184 ceph-mon[33394]: pgmap v69: 89 pgs: 89 active+clean; 219 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.6 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-13T14:23:00.099 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:22:59 smithi184 ceph-mon[33394]: osdmap e199: 8 total, 8 up, 8 in 2024-09-13T14:23:00.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:59 smithi151 ceph-mon[29807]: osdmap e198: 8 total, 8 up, 8 in 2024-09-13T14:23:00.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:59 smithi151 ceph-mon[29807]: from='client.15636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:00.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:59 smithi151 ceph-mon[29807]: from='client.25544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:00.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:59 smithi151 ceph-mon[29807]: pgmap v69: 89 pgs: 89 active+clean; 219 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.6 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-13T14:23:00.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:22:59 smithi151 ceph-mon[29807]: osdmap e199: 8 total, 8 up, 8 in 2024-09-13T14:23:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:59 smithi151 ceph-mon[41387]: osdmap e198: 8 total, 8 up, 8 in 2024-09-13T14:23:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:59 smithi151 ceph-mon[41387]: from='client.15636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:59 smithi151 ceph-mon[41387]: from='client.25544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:59 smithi151 ceph-mon[41387]: pgmap v69: 89 pgs: 89 active+clean; 219 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.6 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-13T14:23:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:22:59 smithi151 ceph-mon[41387]: osdmap e199: 8 total, 8 up, 8 in 2024-09-13T14:23:00.773 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:00 smithi184 systemd[1]: Stopping Ceph grafana.a for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:23:00.860 INFO:tasks.rados.rados.0.smithi151.stdout:548: done (1 left) 2024-09-13T14:23:00.860 INFO:tasks.rados.rados.0.smithi151.stdout:549: snap_create 2024-09-13T14:23:00.860 INFO:tasks.rados.rados.0.smithi151.stdout:547: finishing write tid 1 to smithi15186934-42 2024-09-13T14:23:00.860 INFO:tasks.rados.rados.0.smithi151.stdout:547: finishing write tid 2 to smithi15186934-42 2024-09-13T14:23:00.860 INFO:tasks.rados.rados.0.smithi151.stdout:547: finishing write tid 3 to smithi15186934-42 2024-09-13T14:23:00.860 INFO:tasks.rados.rados.0.smithi151.stdout:547: finishing write tid 4 to smithi15186934-42 2024-09-13T14:23:00.860 INFO:tasks.rados.rados.0.smithi151.stdout:547: finishing write tid 5 to smithi15186934-42 2024-09-13T14:23:00.861 INFO:tasks.rados.rados.0.smithi151.stdout:547: finishing write tid 6 to smithi15186934-42 2024-09-13T14:23:00.861 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 199 (ObjNum 237 snap 73 seq_num 237) dirty exists 2024-09-13T14:23:00.861 INFO:tasks.rados.rados.0.smithi151.stdout:547: left oid 42 (ObjNum 237 snap 73 seq_num 237) 2024-09-13T14:23:01.070 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:00 smithi184 podman[78219]: 2024-09-13 14:23:00.772317209 +0000 UTC m=+0.119474414 container died 20c2e676f61565ee381d83869b15f70c3f13d7ee15477569f0bf910cf8bed8f5 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, vcs-type=git, com.redhat.component=ubi8-container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , name=ubi8, release=239.1651231664, io.openshift.expose-services=, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-04-29T11:48:11.055897, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.buildah.version=1.32.0) 2024-09-13T14:23:01.353 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:01 smithi184 podman[78219]: 2024-09-13 14:23:01.159919978 +0000 UTC m=+0.507077185 container cleanup 20c2e676f61565ee381d83869b15f70c3f13d7ee15477569f0bf910cf8bed8f5 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a, vcs-type=git, description=Ceph Grafana Container, build-date=2022-04-29T11:48:11.055897, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, io.openshift.expose-services=, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=239.1651231664, distribution-scope=public, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.32.0, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, version=8.5) 2024-09-13T14:23:01.353 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:01 smithi184 bash[78219]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a 2024-09-13T14:23:01.784 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:01 smithi184 podman[78235]: 2024-09-13 14:23:01.411494322 +0000 UTC m=+0.639905194 container remove 20c2e676f61565ee381d83869b15f70c3f13d7ee15477569f0bf910cf8bed8f5 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a, vendor=Red Hat, Inc., maintainer=Paul Cuzner , name=ubi8, version=8.5, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-04-29T11:48:11.055897) 2024-09-13T14:23:01.784 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:01 smithi184 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-09-13T14:23:01.784 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:01 smithi184 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-09-13T14:23:01.785 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:01 smithi184 systemd[1]: Stopped Ceph grafana.a for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:23:01.785 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:01 smithi184 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@grafana.a.service: Consumed 5.410s CPU time. 2024-09-13T14:23:01.785 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:01 smithi184 systemd[1]: Starting Ceph grafana.a for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:23:01.860 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:23:01.864 INFO:tasks.rados.rados.0.smithi151.stdout:547: done (1 left) 2024-09-13T14:23:01.864 INFO:tasks.rados.rados.0.smithi151.stdout:549: done (0 left) 2024-09-13T14:23:01.864 INFO:tasks.rados.rados.0.smithi151.stdout:550: snap_remove snap 69 2024-09-13T14:23:02.073 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:01 smithi184 ceph-mon[33394]: osdmap e200: 8 total, 8 up, 8 in 2024-09-13T14:23:02.073 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:01 smithi184 ceph-mon[33394]: pgmap v72: 89 pgs: 89 active+clean; 219 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:23:02.074 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:01 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:23:02.074 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:01 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/89610671' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:02.074 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:01 smithi184 podman[78334]: 2024-09-13 14:23:01.685676911 +0000 UTC m=+0.015154137 image pull 954c08fa618887fe9a6fe572ae541197c77e625b05f4193df7898f5ac649ada9 quay.io/ceph/ceph-grafana:9.4.7 2024-09-13T14:23:02.074 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:01 smithi184 podman[78334]: 2024-09-13 14:23:01.82106837 +0000 UTC m=+0.150545589 container create 4ba18fb1f1fc82cd0aa5b0d7bd519a9b28340496730b2a332a9c65a8349e2af8 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a, io.buildah.version=1.32.0, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, release=239.1651231664, com.redhat.component=ubi8-container, build-date=2022-04-29T11:48:11.055897, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, architecture=x86_64, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, name=ubi8, vcs-type=git, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664) 2024-09-13T14:23:02.074 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:02 smithi184 podman[78334]: 2024-09-13 14:23:02.073383084 +0000 UTC m=+0.402860304 container init 4ba18fb1f1fc82cd0aa5b0d7bd519a9b28340496730b2a332a9c65a8349e2af8 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a, io.openshift.expose-services=, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, release=239.1651231664, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, build-date=2022-04-29T11:48:11.055897, architecture=x86_64, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.32.0) 2024-09-13T14:23:02.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:01 smithi151 ceph-mon[29807]: osdmap e200: 8 total, 8 up, 8 in 2024-09-13T14:23:02.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:01 smithi151 ceph-mon[29807]: pgmap v72: 89 pgs: 89 active+clean; 219 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:23:02.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:01 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:23:02.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:01 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/89610671' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:02.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:01 smithi151 ceph-mon[41387]: osdmap e200: 8 total, 8 up, 8 in 2024-09-13T14:23:02.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:01 smithi151 ceph-mon[41387]: pgmap v72: 89 pgs: 89 active+clean; 219 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:23:02.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:01 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:23:02.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:01 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/89610671' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:02.340 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:02 smithi184 podman[78334]: 2024-09-13 14:23:02.078261864 +0000 UTC m=+0.407739084 container start 4ba18fb1f1fc82cd0aa5b0d7bd519a9b28340496730b2a332a9c65a8349e2af8 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a, vendor=Red Hat, Inc., build-date=2022-04-29T11:48:11.055897, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, vcs-type=git, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.32.0, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, maintainer=Paul Cuzner , release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container) 2024-09-13T14:23:02.340 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:02 smithi184 bash[78334]: 4ba18fb1f1fc82cd0aa5b0d7bd519a9b28340496730b2a332a9c65a8349e2af8 2024-09-13T14:23:02.340 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:02 smithi184 systemd[1]: Started Ceph grafana.a for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:23:02.340 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=settings t=2024-09-13T14:23:02.23215602Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-09-13T14:23:02.340 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=settings t=2024-09-13T14:23:02.232333385Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-13T14:23:02.340 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=settings t=2024-09-13T14:23:02.232352247Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-13T14:23:02.340 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=settings t=2024-09-13T14:23:02.232358904Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-13T14:23:02.340 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=settings t=2024-09-13T14:23:02.232364941Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-13T14:23:02.341 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=settings t=2024-09-13T14:23:02.232372866Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-13T14:23:02.341 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=settings t=2024-09-13T14:23:02.23237864Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-13T14:23:02.341 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=settings t=2024-09-13T14:23:02.232384448Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-13T14:23:02.341 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=settings t=2024-09-13T14:23:02.232391153Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-13T14:23:02.341 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=settings t=2024-09-13T14:23:02.23239701Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-13T14:23:02.341 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=settings t=2024-09-13T14:23:02.232402487Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-13T14:23:02.341 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=settings t=2024-09-13T14:23:02.232407935Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-13T14:23:02.341 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=settings t=2024-09-13T14:23:02.232414264Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-13T14:23:02.341 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=settings t=2024-09-13T14:23:02.232420143Z level=info msg="App mode production" 2024-09-13T14:23:02.341 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=sqlstore t=2024-09-13T14:23:02.232462984Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-13T14:23:02.341 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=sqlstore t=2024-09-13T14:23:02.232480567Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-13T14:23:02.341 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=migrator t=2024-09-13T14:23:02.246789651Z level=info msg="Starting DB migrations" 2024-09-13T14:23:02.341 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=migrator t=2024-09-13T14:23:02.252325532Z level=info msg="migrations completed" performed=0 skipped=481 duration=540.287µs 2024-09-13T14:23:02.342 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=sqlstore t=2024-09-13T14:23:02.252804492Z level=info msg="Created default organization" 2024-09-13T14:23:02.342 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=plugin.loader t=2024-09-13T14:23:02.310100678Z level=info msg="Plugin registered" pluginID=input 2024-09-13T14:23:02.342 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=plugin.loader t=2024-09-13T14:23:02.339084545Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-09-13T14:23:02.342 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=plugin.loader t=2024-09-13T14:23:02.339119307Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-09-13T14:23:02.342 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=secrets t=2024-09-13T14:23:02.339163051Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-13T14:23:02.603 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=query_data t=2024-09-13T14:23:02.340972209Z level=info msg="Query Service initialization" 2024-09-13T14:23:02.603 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=live.push_http t=2024-09-13T14:23:02.343666974Z level=info msg="Live Push Gateway initialization" 2024-09-13T14:23:02.803 INFO:tasks.rados.rados.0.smithi151.stdout:550: done (0 left) 2024-09-13T14:23:02.804 INFO:tasks.rados.rados.0.smithi151.stdout:551: rollback oid 39 current snap is 74 2024-09-13T14:23:02.804 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 39 to 68 2024-09-13T14:23:02.804 INFO:tasks.rados.rados.0.smithi151.stdout:552: read oid 9 snap -1 2024-09-13T14:23:02.804 INFO:tasks.rados.rados.0.smithi151.stdout:552: expect (ObjNum 151 snap 36 seq_num 151) 2024-09-13T14:23:02.804 INFO:tasks.rados.rados.0.smithi151.stdout:553: snap_create 2024-09-13T14:23:02.808 INFO:tasks.rados.rados.0.smithi151.stdout:551: finishing rollback tid 0 to smithi15186934-39 2024-09-13T14:23:02.808 INFO:tasks.rados.rados.0.smithi151.stdout:551: finishing rollback tid 1 to smithi15186934-39 2024-09-13T14:23:02.808 INFO:tasks.rados.rados.0.smithi151.stdout:551: finishing rollback tid 2 to smithi15186934-39 2024-09-13T14:23:02.808 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 130 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-13T14:23:02.813 INFO:tasks.rados.rados.0.smithi151.stdout:552: expect (ObjNum 151 snap 36 seq_num 151) 2024-09-13T14:23:03.060 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:02 smithi184 systemd[1]: Stopping Ceph node-exporter.b for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:23:03.061 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-mon[33394]: from='client.15654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:03.061 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-mon[33394]: osdmap e201: 8 total, 8 up, 8 in 2024-09-13T14:23:03.061 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-mon[33394]: from='client.15660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:03.061 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:03.061 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:03.061 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-mon[33394]: Reconfiguring node-exporter.b (dependencies changed)... 2024-09-13T14:23:03.061 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-mon[33394]: Deploying daemon node-exporter.b on smithi184 2024-09-13T14:23:03.061 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:02 smithi184 ceph-mon[33394]: osdmap e202: 8 total, 8 up, 8 in 2024-09-13T14:23:03.353 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:03 smithi184 podman[78529]: 2024-09-13 14:23:03.059505152 +0000 UTC m=+0.137570922 container died aca4c09ae196f0f2ee45cc936a3bc0716c5ca749e1657061c9bebd9d081aa8e4 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:23:03.354 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:03 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=infra.usagestats.collector t=2024-09-13T14:23:03.111775206Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-13T14:23:03.354 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:03 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=provisioning.datasources t=2024-09-13T14:23:03.198670334Z level=info msg="deleted correlations based on configuration" ds_name=Dashboard1 2024-09-13T14:23:03.354 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:03 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=provisioning.datasources t=2024-09-13T14:23:03.198711307Z level=info msg="deleted datasource based on configuration" name=Dashboard1 2024-09-13T14:23:03.354 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:03 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=provisioning.datasources t=2024-09-13T14:23:03.209042514Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-13T14:23:03.354 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:03 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=provisioning.alerting t=2024-09-13T14:23:03.289089816Z level=info msg="starting to provision alerting" 2024-09-13T14:23:03.354 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:03 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=provisioning.alerting t=2024-09-13T14:23:03.289146915Z level=info msg="finished to provision alerting" 2024-09-13T14:23:03.354 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:03 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=grafanaStorageLogger t=2024-09-13T14:23:03.289349835Z level=info msg="storage starting" 2024-09-13T14:23:03.354 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:03 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=ngalert.state.manager t=2024-09-13T14:23:03.289338968Z level=info msg="Warming state cache for startup" 2024-09-13T14:23:03.354 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:03 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=http.server t=2024-09-13T14:23:03.291555433Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-13T14:23:03.354 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:03 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=ngalert.state.manager t=2024-09-13T14:23:03.299876584Z level=info msg="State cache has been initialized" states=0 duration=10.532429ms 2024-09-13T14:23:03.354 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:03 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=ticker t=2024-09-13T14:23:03.299931024Z level=info msg=starting first_tick=2024-09-13T14:23:10Z 2024-09-13T14:23:03.354 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:23:03 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=ngalert.multiorg.alertmanager t=2024-09-13T14:23:03.299950575Z level=info msg="starting MultiOrg Alertmanager" 2024-09-13T14:23:03.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:02 smithi151 ceph-mon[29807]: from='client.15654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:03.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:02 smithi151 ceph-mon[29807]: osdmap e201: 8 total, 8 up, 8 in 2024-09-13T14:23:03.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:02 smithi151 ceph-mon[29807]: from='client.15660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:03.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:02 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:03.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:02 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:03.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:02 smithi151 ceph-mon[29807]: Reconfiguring node-exporter.b (dependencies changed)... 2024-09-13T14:23:03.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:02 smithi151 ceph-mon[29807]: Deploying daemon node-exporter.b on smithi184 2024-09-13T14:23:03.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:02 smithi151 ceph-mon[29807]: osdmap e202: 8 total, 8 up, 8 in 2024-09-13T14:23:03.363 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:02 smithi151 ceph-mon[41387]: from='client.15654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:03.363 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:02 smithi151 ceph-mon[41387]: osdmap e201: 8 total, 8 up, 8 in 2024-09-13T14:23:03.363 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:02 smithi151 ceph-mon[41387]: from='client.15660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:03.363 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:02 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:03.363 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:02 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:03.363 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:02 smithi151 ceph-mon[41387]: Reconfiguring node-exporter.b (dependencies changed)... 2024-09-13T14:23:03.363 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:02 smithi151 ceph-mon[41387]: Deploying daemon node-exporter.b on smithi184 2024-09-13T14:23:03.363 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:02 smithi151 ceph-mon[41387]: osdmap e202: 8 total, 8 up, 8 in 2024-09-13T14:23:03.656 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:03 smithi184 podman[78529]: 2024-09-13 14:23:03.487727337 +0000 UTC m=+0.565793111 container cleanup aca4c09ae196f0f2ee45cc936a3bc0716c5ca749e1657061c9bebd9d081aa8e4 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:23:03.656 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:03 smithi184 bash[78529]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b 2024-09-13T14:23:03.805 INFO:tasks.rados.rados.0.smithi151.stdout:551: done (2 left) 2024-09-13T14:23:03.805 INFO:tasks.rados.rados.0.smithi151.stdout:552: done (1 left) 2024-09-13T14:23:03.805 INFO:tasks.rados.rados.0.smithi151.stdout:553: done (0 left) 2024-09-13T14:23:03.805 INFO:tasks.rados.rados.0.smithi151.stdout:554: write oid 40 current snap is 75 2024-09-13T14:23:03.805 INFO:tasks.rados.rados.0.smithi151.stdout:554: seq_num 238 ranges {400066=716205,1860832=139234} 2024-09-13T14:23:03.810 INFO:tasks.rados.rados.0.smithi151.stdout:554: writing smithi15186934-40 from 400066 to 1116271 tid 1 2024-09-13T14:23:03.816 INFO:tasks.rados.rados.0.smithi151.stdout:554: writing smithi15186934-40 from 1860832 to 2000066 tid 2 2024-09-13T14:23:03.816 INFO:tasks.rados.rados.0.smithi151.stdout:555: read oid 4 snap 74 2024-09-13T14:23:03.816 INFO:tasks.rados.rados.0.smithi151.stdout:555: expect (ObjNum 192 snap 51 seq_num 192) 2024-09-13T14:23:03.816 INFO:tasks.rados.rados.0.smithi151.stdout:556: write oid 24 current snap is 75 2024-09-13T14:23:03.816 INFO:tasks.rados.rados.0.smithi151.stdout:554: finishing write tid 1 to smithi15186934-40 2024-09-13T14:23:03.816 INFO:tasks.rados.rados.0.smithi151.stdout:556: seq_num 239 ranges {416873=791454,1919314=97559} 2024-09-13T14:23:03.822 INFO:tasks.rados.rados.0.smithi151.stdout:556: writing smithi15186934-24 from 416873 to 1208327 tid 1 2024-09-13T14:23:03.828 INFO:tasks.rados.rados.0.smithi151.stdout:556: writing smithi15186934-24 from 1919314 to 2016873 tid 2 2024-09-13T14:23:03.828 INFO:tasks.rados.rados.0.smithi151.stdout:557: rollback oid 23 current snap is 75 2024-09-13T14:23:03.828 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 23 to 66 2024-09-13T14:23:03.828 INFO:tasks.rados.rados.0.smithi151.stdout:554: finishing write tid 2 to smithi15186934-40 2024-09-13T14:23:03.828 INFO:tasks.rados.rados.0.smithi151.stdout:554: finishing write tid 3 to smithi15186934-40 2024-09-13T14:23:03.828 INFO:tasks.rados.rados.0.smithi151.stdout:554: finishing write tid 4 to smithi15186934-40 2024-09-13T14:23:03.828 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 40 v 207 (ObjNum 238 snap 75 seq_num 238) dirty exists 2024-09-13T14:23:03.828 INFO:tasks.rados.rados.0.smithi151.stdout:554: left oid 40 (ObjNum 238 snap 75 seq_num 238) 2024-09-13T14:23:03.828 INFO:tasks.rados.rados.0.smithi151.stdout:554: done (3 left) 2024-09-13T14:23:03.828 INFO:tasks.rados.rados.0.smithi151.stdout:558: read oid 10 snap -1 2024-09-13T14:23:03.828 INFO:tasks.rados.rados.0.smithi151.stdout:558: expect deleted 2024-09-13T14:23:03.829 INFO:tasks.rados.rados.0.smithi151.stdout:559: write oid 20 current snap is 75 2024-09-13T14:23:03.829 INFO:tasks.rados.rados.0.smithi151.stdout:559: seq_num 240 ranges {433680=466703,1577796=455884} 2024-09-13T14:23:03.832 INFO:tasks.rados.rados.0.smithi151.stdout:559: writing smithi15186934-20 from 433680 to 900383 tid 1 2024-09-13T14:23:03.838 INFO:tasks.rados.rados.0.smithi151.stdout:559: writing smithi15186934-20 from 1577796 to 2033680 tid 2 2024-09-13T14:23:03.839 INFO:tasks.rados.rados.0.smithi151.stdout:560: delete oid 11 current snap is 75 2024-09-13T14:23:03.839 INFO:tasks.rados.rados.0.smithi151.stdout:555: expect (ObjNum 192 snap 51 seq_num 192) 2024-09-13T14:23:03.915 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:03 smithi184 ceph-mon[33394]: pgmap v75: 89 pgs: 89 active+clean; 217 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 843 KiB/s wr, 5 op/s 2024-09-13T14:23:03.916 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:03 smithi184 ceph-mon[33394]: osdmap e203: 8 total, 8 up, 8 in 2024-09-13T14:23:03.916 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:03 smithi184 podman[78547]: 2024-09-13 14:23:03.655868127 +0000 UTC m=+0.597366439 container remove aca4c09ae196f0f2ee45cc936a3bc0716c5ca749e1657061c9bebd9d081aa8e4 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:23:03.916 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:03 smithi184 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-13T14:23:03.993 INFO:tasks.rados.rados.0.smithi151.stdout:557: finishing rollback tid 1 to smithi15186934-23 2024-09-13T14:23:03.993 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 23 v 266 (ObjNum 49 snap 0 seq_num 3064833136) dirty dne 2024-09-13T14:23:03.993 INFO:tasks.rados.rados.0.smithi151.stdout:556: finishing write tid 1 to smithi15186934-24 2024-09-13T14:23:03.993 INFO:tasks.rados.rados.0.smithi151.stdout:556: finishing write tid 2 to smithi15186934-24 2024-09-13T14:23:03.993 INFO:tasks.rados.rados.0.smithi151.stdout:556: finishing write tid 3 to smithi15186934-24 2024-09-13T14:23:03.993 INFO:tasks.rados.rados.0.smithi151.stdout:556: finishing write tid 4 to smithi15186934-24 2024-09-13T14:23:03.993 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 24 v 315 (ObjNum 239 snap 75 seq_num 239) dirty exists 2024-09-13T14:23:03.993 INFO:tasks.rados.rados.0.smithi151.stdout:556: left oid 24 (ObjNum 239 snap 75 seq_num 239) 2024-09-13T14:23:03.993 INFO:tasks.rados.rados.0.smithi151.stdout:559: finishing write tid 1 to smithi15186934-20 2024-09-13T14:23:03.993 INFO:tasks.rados.rados.0.smithi151.stdout:555: done (5 left) 2024-09-13T14:23:03.994 INFO:tasks.rados.rados.0.smithi151.stdout:556: done (4 left) 2024-09-13T14:23:03.994 INFO:tasks.rados.rados.0.smithi151.stdout:557: done (3 left) 2024-09-13T14:23:03.994 INFO:tasks.rados.rados.0.smithi151.stdout:558: done (2 left) 2024-09-13T14:23:03.994 INFO:tasks.rados.rados.0.smithi151.stdout:560: done (1 left) 2024-09-13T14:23:03.994 INFO:tasks.rados.rados.0.smithi151.stdout:561: write oid 27 current snap is 75 2024-09-13T14:23:03.994 INFO:tasks.rados.rados.0.smithi151.stdout:559: finishing write tid 2 to smithi15186934-20 2024-09-13T14:23:03.994 INFO:tasks.rados.rados.0.smithi151.stdout:559: finishing write tid 3 to smithi15186934-20 2024-09-13T14:23:03.994 INFO:tasks.rados.rados.0.smithi151.stdout:561: seq_num 242 ranges {467294=617201,1578407=488887} 2024-09-13T14:23:03.998 INFO:tasks.rados.rados.0.smithi151.stdout:561: writing smithi15186934-27 from 467294 to 1084495 tid 1 2024-09-13T14:23:04.004 INFO:tasks.rados.rados.0.smithi151.stdout:561: writing smithi15186934-27 from 1578407 to 2067294 tid 2 2024-09-13T14:23:04.004 INFO:tasks.rados.rados.0.smithi151.stdout:562: rollback oid 35 current snap is 75 2024-09-13T14:23:04.004 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 35 to 73 2024-09-13T14:23:04.004 INFO:tasks.rados.rados.0.smithi151.stdout:559: finishing write tid 4 to smithi15186934-20 2024-09-13T14:23:04.004 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 137 (ObjNum 240 snap 75 seq_num 240) dirty exists 2024-09-13T14:23:04.004 INFO:tasks.rados.rados.0.smithi151.stdout:559: left oid 20 (ObjNum 240 snap 75 seq_num 240) 2024-09-13T14:23:04.004 INFO:tasks.rados.rados.0.smithi151.stdout:561: finishing write tid 1 to smithi15186934-27 2024-09-13T14:23:04.004 INFO:tasks.rados.rados.0.smithi151.stdout:559: done (2 left) 2024-09-13T14:23:04.004 INFO:tasks.rados.rados.0.smithi151.stdout:563: delete oid 44 current snap is 75 2024-09-13T14:23:04.005 INFO:tasks.rados.rados.0.smithi151.stdout:563: done (2 left) 2024-09-13T14:23:04.005 INFO:tasks.rados.rados.0.smithi151.stdout:564: delete oid 10 current snap is 75 2024-09-13T14:23:04.006 INFO:tasks.rados.rados.0.smithi151.stdout:564: done (2 left) 2024-09-13T14:23:04.006 INFO:tasks.rados.rados.0.smithi151.stdout:565: delete oid 8 current snap is 75 2024-09-13T14:23:04.007 INFO:tasks.rados.rados.0.smithi151.stdout:562: finishing rollback tid 0 to smithi15186934-35 2024-09-13T14:23:04.007 INFO:tasks.rados.rados.0.smithi151.stdout:562: finishing rollback tid 1 to smithi15186934-35 2024-09-13T14:23:04.007 INFO:tasks.rados.rados.0.smithi151.stdout:561: finishing write tid 2 to smithi15186934-27 2024-09-13T14:23:04.008 INFO:tasks.rados.rados.0.smithi151.stdout:562: finishing rollback tid 2 to smithi15186934-35 2024-09-13T14:23:04.008 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 35 v 319 (ObjNum 196 snap 55 seq_num 196) dirty exists 2024-09-13T14:23:04.008 INFO:tasks.rados.rados.0.smithi151.stdout:561: finishing write tid 3 to smithi15186934-27 2024-09-13T14:23:04.008 INFO:tasks.rados.rados.0.smithi151.stdout:561: finishing write tid 4 to smithi15186934-27 2024-09-13T14:23:04.008 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 27 v 141 (ObjNum 242 snap 75 seq_num 242) dirty exists 2024-09-13T14:23:04.008 INFO:tasks.rados.rados.0.smithi151.stdout:561: left oid 27 (ObjNum 242 snap 75 seq_num 242) 2024-09-13T14:23:04.008 INFO:tasks.rados.rados.0.smithi151.stdout:561: done (2 left) 2024-09-13T14:23:04.008 INFO:tasks.rados.rados.0.smithi151.stdout:562: done (1 left) 2024-09-13T14:23:04.009 INFO:tasks.rados.rados.0.smithi151.stdout:565: done (0 left) 2024-09-13T14:23:04.009 INFO:tasks.rados.rados.0.smithi151.stdout:566: write oid 4 current snap is 75 2024-09-13T14:23:04.009 INFO:tasks.rados.rados.0.smithi151.stdout:566: seq_num 246 ranges {534522=634550,1812335=322187} 2024-09-13T14:23:04.014 INFO:tasks.rados.rados.0.smithi151.stdout:566: writing smithi15186934-4 from 534522 to 1169072 tid 1 2024-09-13T14:23:04.019 INFO:tasks.rados.rados.0.smithi151.stdout:566: writing smithi15186934-4 from 1812335 to 2134522 tid 2 2024-09-13T14:23:04.019 INFO:tasks.rados.rados.0.smithi151.stdout:567: read oid 31 snap -1 2024-09-13T14:23:04.019 INFO:tasks.rados.rados.0.smithi151.stdout:567: expect (ObjNum 138 snap 30 seq_num 138) 2024-09-13T14:23:04.020 INFO:tasks.rados.rados.0.smithi151.stdout:568: snap_create 2024-09-13T14:23:04.020 INFO:tasks.rados.rados.0.smithi151.stdout:566: finishing write tid 1 to smithi15186934-4 2024-09-13T14:23:04.022 INFO:tasks.rados.rados.0.smithi151.stdout:566: finishing write tid 2 to smithi15186934-4 2024-09-13T14:23:04.022 INFO:tasks.rados.rados.0.smithi151.stdout:566: finishing write tid 3 to smithi15186934-4 2024-09-13T14:23:04.022 INFO:tasks.rados.rados.0.smithi151.stdout:566: finishing write tid 4 to smithi15186934-4 2024-09-13T14:23:04.022 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 290 (ObjNum 246 snap 75 seq_num 246) dirty exists 2024-09-13T14:23:04.022 INFO:tasks.rados.rados.0.smithi151.stdout:566: left oid 4 (ObjNum 246 snap 75 seq_num 246) 2024-09-13T14:23:04.023 INFO:tasks.rados.rados.0.smithi151.stdout:567: expect (ObjNum 138 snap 30 seq_num 138) 2024-09-13T14:23:04.307 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:03 smithi184 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-13T14:23:04.307 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:03 smithi184 systemd[1]: Stopped Ceph node-exporter.b for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:23:04.307 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:03 smithi184 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@node-exporter.b.service: Consumed 2.187s CPU time. 2024-09-13T14:23:04.307 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:04 smithi184 systemd[1]: Starting Ceph node-exporter.b for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:23:04.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:03 smithi151 ceph-mon[41387]: pgmap v75: 89 pgs: 89 active+clean; 217 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 843 KiB/s wr, 5 op/s 2024-09-13T14:23:04.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:03 smithi151 ceph-mon[41387]: osdmap e203: 8 total, 8 up, 8 in 2024-09-13T14:23:04.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:03 smithi151 ceph-mon[29807]: pgmap v75: 89 pgs: 89 active+clean; 217 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 843 KiB/s wr, 5 op/s 2024-09-13T14:23:04.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:03 smithi151 ceph-mon[29807]: osdmap e203: 8 total, 8 up, 8 in 2024-09-13T14:23:04.603 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:04 smithi184 bash[78679]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-09-13T14:23:04.760 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:23:04.876 INFO:tasks.rados.rados.0.smithi151.stdout:566: done (2 left) 2024-09-13T14:23:04.876 INFO:tasks.rados.rados.0.smithi151.stdout:567: done (1 left) 2024-09-13T14:23:04.877 INFO:tasks.rados.rados.0.smithi151.stdout:568: done (0 left) 2024-09-13T14:23:04.877 INFO:tasks.rados.rados.0.smithi151.stdout:569: write oid 36 current snap is 76 2024-09-13T14:23:04.877 INFO:tasks.rados.rados.0.smithi151.stdout:569: seq_num 247 ranges {551329=709799,1870817=280512} 2024-09-13T14:23:04.883 INFO:tasks.rados.rados.0.smithi151.stdout:569: writing smithi15186934-36 from 551329 to 1261128 tid 1 2024-09-13T14:23:04.888 INFO:tasks.rados.rados.0.smithi151.stdout:569: writing smithi15186934-36 from 1870817 to 2151329 tid 2 2024-09-13T14:23:04.888 INFO:tasks.rados.rados.0.smithi151.stdout:570: rollback oid 12 current snap is 76 2024-09-13T14:23:04.888 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 12 to 66 2024-09-13T14:23:04.888 INFO:tasks.rados.rados.0.smithi151.stdout:569: finishing write tid 1 to smithi15186934-36 2024-09-13T14:23:04.888 INFO:tasks.rados.rados.0.smithi151.stdout:571: write oid 18 current snap is 76 2024-09-13T14:23:04.889 INFO:tasks.rados.rados.0.smithi151.stdout:571: seq_num 248 ranges {568136=785048,1929299=238837} 2024-09-13T14:23:04.896 INFO:tasks.rados.rados.0.smithi151.stdout:571: writing smithi15186934-18 from 568136 to 1353184 tid 1 2024-09-13T14:23:04.901 INFO:tasks.rados.rados.0.smithi151.stdout:571: writing smithi15186934-18 from 1929299 to 2168136 tid 2 2024-09-13T14:23:04.901 INFO:tasks.rados.rados.0.smithi151.stdout:572: rollback oid 21 current snap is 76 2024-09-13T14:23:04.901 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 21 to 66 2024-09-13T14:23:04.901 INFO:tasks.rados.rados.0.smithi151.stdout:569: finishing write tid 2 to smithi15186934-36 2024-09-13T14:23:04.901 INFO:tasks.rados.rados.0.smithi151.stdout:569: finishing write tid 3 to smithi15186934-36 2024-09-13T14:23:04.901 INFO:tasks.rados.rados.0.smithi151.stdout:569: finishing write tid 4 to smithi15186934-36 2024-09-13T14:23:04.901 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 36 v 88 (ObjNum 247 snap 76 seq_num 247) dirty exists 2024-09-13T14:23:04.902 INFO:tasks.rados.rados.0.smithi151.stdout:569: left oid 36 (ObjNum 247 snap 76 seq_num 247) 2024-09-13T14:23:04.902 INFO:tasks.rados.rados.0.smithi151.stdout:570: finishing rollback tid 0 to smithi15186934-12 2024-09-13T14:23:04.902 INFO:tasks.rados.rados.0.smithi151.stdout:570: finishing rollback tid 1 to smithi15186934-12 2024-09-13T14:23:04.902 INFO:tasks.rados.rados.0.smithi151.stdout:569: done (3 left) 2024-09-13T14:23:04.902 INFO:tasks.rados.rados.0.smithi151.stdout:573: snap_remove snap 72 2024-09-13T14:23:05.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:04 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/3741282196' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:05.103 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:04 smithi184 bash[78679]: Getting image source signatures 2024-09-13T14:23:05.104 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:04 smithi184 bash[78679]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-13T14:23:05.104 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:04 smithi184 bash[78679]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-09-13T14:23:05.104 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:04 smithi184 bash[78679]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-09-13T14:23:05.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:04 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/3741282196' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:05.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:04 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/3741282196' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:05.884 INFO:tasks.rados.rados.0.smithi151.stdout:573: done (3 left) 2024-09-13T14:23:05.885 INFO:tasks.rados.rados.0.smithi151.stdout:574: read oid 11 snap -1 2024-09-13T14:23:05.885 INFO:tasks.rados.rados.0.smithi151.stdout:574: expect deleted 2024-09-13T14:23:05.885 INFO:tasks.rados.rados.0.smithi151.stdout:575: snap_remove snap 64 2024-09-13T14:23:06.195 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:05 smithi184 ceph-mon[33394]: from='client.25574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:06.196 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:05 smithi184 ceph-mon[33394]: osdmap e204: 8 total, 8 up, 8 in 2024-09-13T14:23:06.196 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:05 smithi184 ceph-mon[33394]: from='client.25580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:06.196 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:05 smithi184 ceph-mon[33394]: pgmap v78: 89 pgs: 89 active+clean; 217 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 843 KiB/s wr, 5 op/s 2024-09-13T14:23:06.196 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 bash[78679]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-13T14:23:06.196 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 bash[78679]: Writing manifest to image destination 2024-09-13T14:23:06.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:05 smithi151 ceph-mon[29807]: from='client.25574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:06.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:05 smithi151 ceph-mon[29807]: osdmap e204: 8 total, 8 up, 8 in 2024-09-13T14:23:06.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:05 smithi151 ceph-mon[29807]: from='client.25580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:06.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:05 smithi151 ceph-mon[29807]: pgmap v78: 89 pgs: 89 active+clean; 217 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 843 KiB/s wr, 5 op/s 2024-09-13T14:23:06.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:05 smithi151 ceph-mon[41387]: from='client.25574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:06.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:05 smithi151 ceph-mon[41387]: osdmap e204: 8 total, 8 up, 8 in 2024-09-13T14:23:06.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:05 smithi151 ceph-mon[41387]: from='client.25580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:06.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:05 smithi151 ceph-mon[41387]: pgmap v78: 89 pgs: 89 active+clean; 217 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 843 KiB/s wr, 5 op/s 2024-09-13T14:23:06.794 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 podman[78679]: 2024-09-13 14:23:06.451111715 +0000 UTC m=+2.237391857 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-13T14:23:06.794 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 podman[78679]: 2024-09-13 14:23:06.566445034 +0000 UTC m=+2.352725175 container create 1e1a9dd643a0fb3d8c295b4aa417b9292f72e15efd8cc3c20df9f5514ac076b4 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:23:06.794 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 podman[78679]: 2024-09-13 14:23:06.793972603 +0000 UTC m=+2.580252746 container init 1e1a9dd643a0fb3d8c295b4aa417b9292f72e15efd8cc3c20df9f5514ac076b4 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:23:06.890 INFO:tasks.rados.rados.0.smithi151.stdout:575: done (4 left) 2024-09-13T14:23:06.891 INFO:tasks.rados.rados.0.smithi151.stdout:576: snap_create 2024-09-13T14:23:06.891 INFO:tasks.rados.rados.0.smithi151.stdout:570: finishing rollback tid 2 to smithi15186934-12 2024-09-13T14:23:06.891 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 12 v 92 (ObjNum 167 snap 43 seq_num 167) dirty exists 2024-09-13T14:23:06.891 INFO:tasks.rados.rados.0.smithi151.stdout:571: finishing write tid 1 to smithi15186934-18 2024-09-13T14:23:06.891 INFO:tasks.rados.rados.0.smithi151.stdout:571: finishing write tid 2 to smithi15186934-18 2024-09-13T14:23:06.891 INFO:tasks.rados.rados.0.smithi151.stdout:571: finishing write tid 3 to smithi15186934-18 2024-09-13T14:23:06.891 INFO:tasks.rados.rados.0.smithi151.stdout:571: finishing write tid 4 to smithi15186934-18 2024-09-13T14:23:06.891 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 18 v 151 (ObjNum 248 snap 76 seq_num 248) dirty exists 2024-09-13T14:23:06.891 INFO:tasks.rados.rados.0.smithi151.stdout:571: left oid 18 (ObjNum 248 snap 76 seq_num 248) 2024-09-13T14:23:06.891 INFO:tasks.rados.rados.0.smithi151.stdout:572: finishing rollback tid 0 to smithi15186934-21 2024-09-13T14:23:06.891 INFO:tasks.rados.rados.0.smithi151.stdout:572: finishing rollback tid 1 to smithi15186934-21 2024-09-13T14:23:06.891 INFO:tasks.rados.rados.0.smithi151.stdout:572: finishing rollback tid 2 to smithi15186934-21 2024-09-13T14:23:06.892 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 21 v 100 (ObjNum 223 snap 64 seq_num 223) dirty exists 2024-09-13T14:23:07.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-mon[33394]: osdmap e205: 8 total, 8 up, 8 in 2024-09-13T14:23:07.104 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 podman[78679]: 2024-09-13 14:23:06.798826765 +0000 UTC m=+2.585106906 container start 1e1a9dd643a0fb3d8c295b4aa417b9292f72e15efd8cc3c20df9f5514ac076b4 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:23:07.104 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.802Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-13T14:23:07.104 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.802Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-13T14:23:07.104 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.802Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-13T14:23:07.104 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.802Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-13T14:23:07.104 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-13T14:23:07.104 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-13T14:23:07.104 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-13T14:23:07.104 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=arp 2024-09-13T14:23:07.104 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-13T14:23:07.105 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-13T14:23:07.105 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-13T14:23:07.105 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-13T14:23:07.105 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-13T14:23:07.105 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-13T14:23:07.105 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-13T14:23:07.105 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-13T14:23:07.106 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=edac 2024-09-13T14:23:07.106 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-13T14:23:07.106 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-13T14:23:07.106 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-13T14:23:07.106 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-13T14:23:07.106 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-13T14:23:07.106 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-13T14:23:07.106 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-13T14:23:07.106 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-13T14:23:07.106 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-13T14:23:07.106 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-13T14:23:07.106 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-13T14:23:07.106 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-13T14:23:07.107 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-13T14:23:07.107 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-13T14:23:07.107 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-13T14:23:07.107 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-13T14:23:07.107 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=os 2024-09-13T14:23:07.107 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-13T14:23:07.107 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-13T14:23:07.107 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-13T14:23:07.107 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-13T14:23:07.107 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-13T14:23:07.107 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-13T14:23:07.107 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-13T14:23:07.107 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=stat 2024-09-13T14:23:07.107 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-13T14:23:07.108 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-13T14:23:07.108 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-13T14:23:07.108 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=time 2024-09-13T14:23:07.108 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-13T14:23:07.108 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=uname 2024-09-13T14:23:07.108 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-13T14:23:07.108 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-13T14:23:07.108 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.803Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-13T14:23:07.108 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.804Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-13T14:23:07.108 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b[78755]: ts=2024-09-13T14:23:06.804Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-13T14:23:07.108 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 bash[78679]: 1e1a9dd643a0fb3d8c295b4aa417b9292f72e15efd8cc3c20df9f5514ac076b4 2024-09-13T14:23:07.108 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 14:23:06 smithi184 systemd[1]: Started Ceph node-exporter.b for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:23:07.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:06 smithi151 ceph-mon[29807]: osdmap e205: 8 total, 8 up, 8 in 2024-09-13T14:23:07.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:06 smithi151 ceph-mon[41387]: osdmap e205: 8 total, 8 up, 8 in 2024-09-13T14:23:07.688 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:23:07.804 INFO:tasks.rados.rados.0.smithi151.stdout:570: done (4 left) 2024-09-13T14:23:07.804 INFO:tasks.rados.rados.0.smithi151.stdout:571: done (3 left) 2024-09-13T14:23:07.804 INFO:tasks.rados.rados.0.smithi151.stdout:572: done (2 left) 2024-09-13T14:23:07.805 INFO:tasks.rados.rados.0.smithi151.stdout:574: done (1 left) 2024-09-13T14:23:07.805 INFO:tasks.rados.rados.0.smithi151.stdout:576: done (0 left) 2024-09-13T14:23:07.805 INFO:tasks.rados.rados.0.smithi151.stdout:577: snap_create 2024-09-13T14:23:08.042 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[29807]: osdmap e206: 8 total, 8 up, 8 in 2024-09-13T14:23:08.043 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:08.043 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:08.043 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:23:08.043 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[29807]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:23:08.043 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:23:08.044 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[29807]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:23:08.044 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:23:08.044 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[29807]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:23:08.044 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:08.044 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[29807]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-13T14:23:08.044 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[29807]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-13T14:23:08.044 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/3712752170' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:08.044 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[29807]: pgmap v81: 89 pgs: 89 active+clean; 227 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.4 MiB/s rd, 1.6 MiB/s wr, 18 op/s 2024-09-13T14:23:08.044 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[29807]: from='client.15690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:08.044 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[29807]: osdmap e207: 8 total, 8 up, 8 in 2024-09-13T14:23:08.044 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[41387]: osdmap e206: 8 total, 8 up, 8 in 2024-09-13T14:23:08.045 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:08.045 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:08.045 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:23:08.045 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[41387]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:23:08.045 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:23:08.045 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[41387]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:23:08.045 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:23:08.045 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[41387]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:23:08.045 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:08.045 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[41387]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-13T14:23:08.045 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[41387]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-13T14:23:08.045 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/3712752170' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:08.045 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[41387]: pgmap v81: 89 pgs: 89 active+clean; 227 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.4 MiB/s rd, 1.6 MiB/s wr, 18 op/s 2024-09-13T14:23:08.046 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[41387]: from='client.15690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:08.046 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:07 smithi151 ceph-mon[41387]: osdmap e207: 8 total, 8 up, 8 in 2024-09-13T14:23:08.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:07 smithi184 ceph-mon[33394]: osdmap e206: 8 total, 8 up, 8 in 2024-09-13T14:23:08.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:07 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:08.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:07 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:08.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:07 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:23:08.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:07 smithi184 ceph-mon[33394]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:23:08.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:07 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:23:08.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:07 smithi184 ceph-mon[33394]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:23:08.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:07 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:23:08.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:07 smithi184 ceph-mon[33394]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:23:08.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:07 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:08.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:07 smithi184 ceph-mon[33394]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-13T14:23:08.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:07 smithi184 ceph-mon[33394]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-13T14:23:08.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:07 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/3712752170' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:08.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:07 smithi184 ceph-mon[33394]: pgmap v81: 89 pgs: 89 active+clean; 227 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.4 MiB/s rd, 1.6 MiB/s wr, 18 op/s 2024-09-13T14:23:08.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:07 smithi184 ceph-mon[33394]: from='client.15690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:08.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:07 smithi184 ceph-mon[33394]: osdmap e207: 8 total, 8 up, 8 in 2024-09-13T14:23:08.807 INFO:tasks.rados.rados.0.smithi151.stdout:577: done (0 left) 2024-09-13T14:23:08.807 INFO:tasks.rados.rados.0.smithi151.stdout:578: snap_remove snap 65 2024-09-13T14:23:09.321 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:08 smithi184 ceph-mon[33394]: from='client.15696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:09.321 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:08 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:09.321 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:08 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:23:09.321 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:08 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:23:09.321 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:08 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:23:09.321 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:08 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:09.322 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:08 smithi184 ceph-mon[33394]: osdmap e208: 8 total, 8 up, 8 in 2024-09-13T14:23:09.357 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:08 smithi151 ceph-mon[29807]: from='client.15696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:09.357 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:08 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:09.357 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:08 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:23:09.357 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:08 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:23:09.357 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:08 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:23:09.357 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:08 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:09.357 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:08 smithi151 ceph-mon[29807]: osdmap e208: 8 total, 8 up, 8 in 2024-09-13T14:23:09.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:08 smithi151 ceph-mon[41387]: from='client.15696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:09.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:08 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:09.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:08 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:23:09.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:08 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:23:09.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:08 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:23:09.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:08 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:09.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:08 smithi151 ceph-mon[41387]: osdmap e208: 8 total, 8 up, 8 in 2024-09-13T14:23:09.603 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:09 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:23:09] "GET /metrics HTTP/1.1" 200 36403 "" "Prometheus/2.51.0" 2024-09-13T14:23:09.815 INFO:tasks.rados.rados.0.smithi151.stdout:578: done (0 left) 2024-09-13T14:23:09.815 INFO:tasks.rados.rados.0.smithi151.stdout:579: delete oid 39 current snap is 78 2024-09-13T14:23:09.817 INFO:tasks.rados.rados.0.smithi151.stdout:579: done (0 left) 2024-09-13T14:23:09.817 INFO:tasks.rados.rados.0.smithi151.stdout:580: delete oid 19 current snap is 78 2024-09-13T14:23:09.819 INFO:tasks.rados.rados.0.smithi151.stdout:580: done (0 left) 2024-09-13T14:23:09.819 INFO:tasks.rados.rados.0.smithi151.stdout:581: read oid 6 snap -1 2024-09-13T14:23:09.819 INFO:tasks.rados.rados.0.smithi151.stdout:581: expect (ObjNum 147 snap 34 seq_num 147) 2024-09-13T14:23:09.820 INFO:tasks.rados.rados.0.smithi151.stdout:582: write oid 45 current snap is 78 2024-09-13T14:23:09.820 INFO:tasks.rados.rados.0.smithi151.stdout:582: seq_num 251 ranges {618557=727148,2104745=113812} 2024-09-13T14:23:09.827 INFO:tasks.rados.rados.0.smithi151.stdout:582: writing smithi15186934-45 from 618557 to 1345705 tid 1 2024-09-13T14:23:09.832 INFO:tasks.rados.rados.0.smithi151.stdout:582: writing smithi15186934-45 from 2104745 to 2218557 tid 2 2024-09-13T14:23:09.832 INFO:tasks.rados.rados.0.smithi151.stdout:583: snap_create 2024-09-13T14:23:09.832 INFO:tasks.rados.rados.0.smithi151.stdout:581: expect (ObjNum 147 snap 34 seq_num 147) 2024-09-13T14:23:09.974 INFO:tasks.rados.rados.0.smithi151.stdout:582: finishing write tid 1 to smithi15186934-45 2024-09-13T14:23:09.974 INFO:tasks.rados.rados.0.smithi151.stdout:582: finishing write tid 2 to smithi15186934-45 2024-09-13T14:23:09.974 INFO:tasks.rados.rados.0.smithi151.stdout:582: finishing write tid 3 to smithi15186934-45 2024-09-13T14:23:09.974 INFO:tasks.rados.rados.0.smithi151.stdout:582: finishing write tid 4 to smithi15186934-45 2024-09-13T14:23:09.975 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 310 (ObjNum 251 snap 78 seq_num 251) dirty exists 2024-09-13T14:23:09.975 INFO:tasks.rados.rados.0.smithi151.stdout:582: left oid 45 (ObjNum 251 snap 78 seq_num 251) 2024-09-13T14:23:10.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:09 smithi151 ceph-mon[29807]: Upgrade: Updating mgr.y 2024-09-13T14:23:10.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:09 smithi151 ceph-mon[29807]: Deploying daemon mgr.y on smithi151 2024-09-13T14:23:10.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:09 smithi151 ceph-mon[29807]: pgmap v84: 89 pgs: 89 active+clean; 227 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.4 MiB/s rd, 1.6 MiB/s wr, 18 op/s 2024-09-13T14:23:10.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:09 smithi151 ceph-mon[29807]: osdmap e209: 8 total, 8 up, 8 in 2024-09-13T14:23:10.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:09 smithi151 ceph-mon[41387]: Upgrade: Updating mgr.y 2024-09-13T14:23:10.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:09 smithi151 ceph-mon[41387]: Deploying daemon mgr.y on smithi151 2024-09-13T14:23:10.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:09 smithi151 ceph-mon[41387]: pgmap v84: 89 pgs: 89 active+clean; 227 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.4 MiB/s rd, 1.6 MiB/s wr, 18 op/s 2024-09-13T14:23:10.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:09 smithi151 ceph-mon[41387]: osdmap e209: 8 total, 8 up, 8 in 2024-09-13T14:23:10.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:09 smithi184 ceph-mon[33394]: Upgrade: Updating mgr.y 2024-09-13T14:23:10.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:09 smithi184 ceph-mon[33394]: Deploying daemon mgr.y on smithi151 2024-09-13T14:23:10.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:09 smithi184 ceph-mon[33394]: pgmap v84: 89 pgs: 89 active+clean; 227 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.4 MiB/s rd, 1.6 MiB/s wr, 18 op/s 2024-09-13T14:23:10.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:09 smithi184 ceph-mon[33394]: osdmap e209: 8 total, 8 up, 8 in 2024-09-13T14:23:10.654 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:23:10.810 INFO:tasks.rados.rados.0.smithi151.stdout:581: done (2 left) 2024-09-13T14:23:10.810 INFO:tasks.rados.rados.0.smithi151.stdout:582: done (1 left) 2024-09-13T14:23:10.810 INFO:tasks.rados.rados.0.smithi151.stdout:583: done (0 left) 2024-09-13T14:23:10.810 INFO:tasks.rados.rados.0.smithi151.stdout:584: write oid 2 current snap is 79 2024-09-13T14:23:10.810 INFO:tasks.rados.rados.0.smithi151.stdout:584: seq_num 252 ranges {635364=402397,1763227=472137} 2024-09-13T14:23:10.815 INFO:tasks.rados.rados.0.smithi151.stdout:584: writing smithi15186934-2 from 635364 to 1037761 tid 1 2024-09-13T14:23:10.818 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:10 smithi151 systemd[1]: Stopping Ceph mgr.y for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:23:10.823 INFO:tasks.rados.rados.0.smithi151.stdout:584: writing smithi15186934-2 from 1763227 to 2235364 tid 2 2024-09-13T14:23:10.823 INFO:tasks.rados.rados.0.smithi151.stdout:585: write oid 17 current snap is 79 2024-09-13T14:23:10.823 INFO:tasks.rados.rados.0.smithi151.stdout:584: finishing write tid 1 to smithi15186934-2 2024-09-13T14:23:10.823 INFO:tasks.rados.rados.0.smithi151.stdout:585: seq_num 253 ranges {652171=477646,1821709=430462} 2024-09-13T14:23:10.829 INFO:tasks.rados.rados.0.smithi151.stdout:585: writing smithi15186934-17 from 652171 to 1129817 tid 1 2024-09-13T14:23:10.835 INFO:tasks.rados.rados.0.smithi151.stdout:585: writing smithi15186934-17 from 1821709 to 2252171 tid 2 2024-09-13T14:23:10.836 INFO:tasks.rados.rados.0.smithi151.stdout:586: write oid 30 current snap is 79 2024-09-13T14:23:10.836 INFO:tasks.rados.rados.0.smithi151.stdout:584: finishing write tid 2 to smithi15186934-2 2024-09-13T14:23:10.836 INFO:tasks.rados.rados.0.smithi151.stdout:586: seq_num 254 ranges {668978=552895,1763838=505140} 2024-09-13T14:23:10.842 INFO:tasks.rados.rados.0.smithi151.stdout:586: writing smithi15186934-30 from 668978 to 1221873 tid 1 2024-09-13T14:23:10.849 INFO:tasks.rados.rados.0.smithi151.stdout:586: writing smithi15186934-30 from 1763838 to 2268978 tid 2 2024-09-13T14:23:10.849 INFO:tasks.rados.rados.0.smithi151.stdout:587: delete oid 23 current snap is 79 2024-09-13T14:23:10.849 INFO:tasks.rados.rados.0.smithi151.stdout:584: finishing write tid 3 to smithi15186934-2 2024-09-13T14:23:10.849 INFO:tasks.rados.rados.0.smithi151.stdout:584: finishing write tid 4 to smithi15186934-2 2024-09-13T14:23:10.849 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 2 v 144 (ObjNum 252 snap 79 seq_num 252) dirty exists 2024-09-13T14:23:10.849 INFO:tasks.rados.rados.0.smithi151.stdout:584: left oid 2 (ObjNum 252 snap 79 seq_num 252) 2024-09-13T14:23:10.849 INFO:tasks.rados.rados.0.smithi151.stdout:585: finishing write tid 1 to smithi15186934-17 2024-09-13T14:23:10.850 INFO:tasks.rados.rados.0.smithi151.stdout:585: finishing write tid 2 to smithi15186934-17 2024-09-13T14:23:10.850 INFO:tasks.rados.rados.0.smithi151.stdout:585: finishing write tid 3 to smithi15186934-17 2024-09-13T14:23:10.850 INFO:tasks.rados.rados.0.smithi151.stdout:585: finishing write tid 4 to smithi15186934-17 2024-09-13T14:23:10.850 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 335 (ObjNum 253 snap 79 seq_num 253) dirty exists 2024-09-13T14:23:10.850 INFO:tasks.rados.rados.0.smithi151.stdout:585: left oid 17 (ObjNum 253 snap 79 seq_num 253) 2024-09-13T14:23:10.850 INFO:tasks.rados.rados.0.smithi151.stdout:586: finishing write tid 1 to smithi15186934-30 2024-09-13T14:23:10.851 INFO:tasks.rados.rados.0.smithi151.stdout:584: done (3 left) 2024-09-13T14:23:10.851 INFO:tasks.rados.rados.0.smithi151.stdout:585: done (2 left) 2024-09-13T14:23:10.851 INFO:tasks.rados.rados.0.smithi151.stdout:587: done (1 left) 2024-09-13T14:23:10.851 INFO:tasks.rados.rados.0.smithi151.stdout:588: read oid 39 snap 43 2024-09-13T14:23:10.851 INFO:tasks.rados.rados.0.smithi151.stdout:588: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-13T14:23:10.851 INFO:tasks.rados.rados.0.smithi151.stdout:589: snap_remove snap 70 2024-09-13T14:23:11.108 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:10 smithi151 podman[94666]: 2024-09-13 14:23:10.817346839 +0000 UTC m=+0.162028001 container died 6ce1f75b6d5e75d4d8d6679d4c4e6fb807adff35292497c51dfb1634e8ed460e (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, RELEASE=quincy-bbdadfa, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T14:23:11.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:10 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/1880166342' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:11.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:10 smithi151 ceph-mon[41387]: from='client.15708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:11.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:10 smithi151 ceph-mon[41387]: osdmap e210: 8 total, 8 up, 8 in 2024-09-13T14:23:11.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:10 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1880166342' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:11.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:10 smithi151 ceph-mon[29807]: from='client.15708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:11.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:10 smithi151 ceph-mon[29807]: osdmap e210: 8 total, 8 up, 8 in 2024-09-13T14:23:11.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:10 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/1880166342' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:11.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:10 smithi184 ceph-mon[33394]: from='client.15708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:11.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:10 smithi184 ceph-mon[33394]: osdmap e210: 8 total, 8 up, 8 in 2024-09-13T14:23:11.571 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:11 smithi151 podman[94666]: 2024-09-13 14:23:11.302246706 +0000 UTC m=+0.646927866 container cleanup 6ce1f75b6d5e75d4d8d6679d4c4e6fb807adff35292497c51dfb1634e8ed460e (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=quincy-bbdadfa, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2) 2024-09-13T14:23:11.571 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:11 smithi151 bash[94666]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y 2024-09-13T14:23:11.816 INFO:tasks.rados.rados.0.smithi151.stdout:589: done (2 left) 2024-09-13T14:23:11.816 INFO:tasks.rados.rados.0.smithi151.stdout:590: rollback oid 22 current snap is 79 2024-09-13T14:23:11.816 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 22 to 73 2024-09-13T14:23:11.816 INFO:tasks.rados.rados.0.smithi151.stdout:591: rollback oid 28 current snap is 79 2024-09-13T14:23:11.816 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 28 to 59 2024-09-13T14:23:11.816 INFO:tasks.rados.rados.0.smithi151.stdout:592: delete oid 1 current snap is 79 2024-09-13T14:23:11.817 INFO:tasks.rados.rados.0.smithi151.stdout:586: finishing write tid 2 to smithi15186934-30 2024-09-13T14:23:11.817 INFO:tasks.rados.rados.0.smithi151.stdout:586: finishing write tid 3 to smithi15186934-30 2024-09-13T14:23:11.817 INFO:tasks.rados.rados.0.smithi151.stdout:586: finishing write tid 4 to smithi15186934-30 2024-09-13T14:23:11.817 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 30 v 215 (ObjNum 254 snap 79 seq_num 254) dirty exists 2024-09-13T14:23:11.817 INFO:tasks.rados.rados.0.smithi151.stdout:586: left oid 30 (ObjNum 254 snap 79 seq_num 254) 2024-09-13T14:23:11.817 INFO:tasks.rados.rados.0.smithi151.stdout:588: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-13T14:23:11.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:11 smithi151 podman[94708]: 2024-09-13 14:23:11.571079405 +0000 UTC m=+0.749464120 container remove 6ce1f75b6d5e75d4d8d6679d4c4e6fb807adff35292497c51dfb1634e8ed460e (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=quincy-bbdadfa, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-13T14:23:11.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:11 smithi151 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-09-13T14:23:11.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:11 smithi151 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@mgr.y.service: Failed with result 'exit-code'. 2024-09-13T14:23:11.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:11 smithi151 systemd[1]: Stopped Ceph mgr.y for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:23:11.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:11 smithi151 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@mgr.y.service: Consumed 40.508s CPU time. 2024-09-13T14:23:11.907 INFO:tasks.rados.rados.0.smithi151.stdout:590: finishing rollback tid 1 to smithi15186934-22 2024-09-13T14:23:11.907 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 22 v 88 (ObjNum 129 snap 0 seq_num 3092798512) dirty dne 2024-09-13T14:23:11.907 INFO:tasks.rados.rados.0.smithi151.stdout:591: finishing rollback tid 0 to smithi15186934-28 2024-09-13T14:23:11.907 INFO:tasks.rados.rados.0.smithi151.stdout:591: finishing rollback tid 1 to smithi15186934-28 2024-09-13T14:23:11.907 INFO:tasks.rados.rados.0.smithi151.stdout:591: finishing rollback tid 2 to smithi15186934-28 2024-09-13T14:23:11.908 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 28 v 213 (ObjNum 173 snap 46 seq_num 173) dirty exists 2024-09-13T14:23:11.908 INFO:tasks.rados.rados.0.smithi151.stdout:586: done (4 left) 2024-09-13T14:23:11.908 INFO:tasks.rados.rados.0.smithi151.stdout:588: done (3 left) 2024-09-13T14:23:11.908 INFO:tasks.rados.rados.0.smithi151.stdout:590: done (2 left) 2024-09-13T14:23:11.908 INFO:tasks.rados.rados.0.smithi151.stdout:591: done (1 left) 2024-09-13T14:23:11.908 INFO:tasks.rados.rados.0.smithi151.stdout:592: done (0 left) 2024-09-13T14:23:11.908 INFO:tasks.rados.rados.0.smithi151.stdout:593: read oid 40 snap 71 2024-09-13T14:23:11.908 INFO:tasks.rados.rados.0.smithi151.stdout:593: expect deleted 2024-09-13T14:23:11.908 INFO:tasks.rados.rados.0.smithi151.stdout:594: write oid 31 current snap is 79 2024-09-13T14:23:11.908 INFO:tasks.rados.rados.0.smithi151.stdout:594: seq_num 257 ranges {719399=778642,1939284=380115} 2024-09-13T14:23:11.915 INFO:tasks.rados.rados.0.smithi151.stdout:594: writing smithi15186934-31 from 719399 to 1498041 tid 1 2024-09-13T14:23:11.920 INFO:tasks.rados.rados.0.smithi151.stdout:594: writing smithi15186934-31 from 1939284 to 2319399 tid 2 2024-09-13T14:23:11.920 INFO:tasks.rados.rados.0.smithi151.stdout:595: snap_create 2024-09-13T14:23:11.920 INFO:tasks.rados.rados.0.smithi151.stdout:594: finishing write tid 1 to smithi15186934-31 2024-09-13T14:23:11.922 INFO:tasks.rados.rados.0.smithi151.stdout:594: finishing write tid 2 to smithi15186934-31 2024-09-13T14:23:11.922 INFO:tasks.rados.rados.0.smithi151.stdout:594: finishing write tid 3 to smithi15186934-31 2024-09-13T14:23:11.922 INFO:tasks.rados.rados.0.smithi151.stdout:594: finishing write tid 4 to smithi15186934-31 2024-09-13T14:23:11.923 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 31 v 219 (ObjNum 257 snap 79 seq_num 257) dirty exists 2024-09-13T14:23:11.923 INFO:tasks.rados.rados.0.smithi151.stdout:594: left oid 31 (ObjNum 257 snap 79 seq_num 257) 2024-09-13T14:23:12.255 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:11 smithi184 ceph-mon[33394]: from='client.15714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:12.255 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:11 smithi184 ceph-mon[33394]: pgmap v87: 89 pgs: 89 active+clean; 227 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:23:12.255 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:11 smithi184 ceph-mon[33394]: osdmap e211: 8 total, 8 up, 8 in 2024-09-13T14:23:12.291 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:11 smithi151 ceph-mon[29807]: from='client.15714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:12.291 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:11 smithi151 ceph-mon[29807]: pgmap v87: 89 pgs: 89 active+clean; 227 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:23:12.291 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:11 smithi151 ceph-mon[29807]: osdmap e211: 8 total, 8 up, 8 in 2024-09-13T14:23:12.291 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:11 smithi151 ceph-mon[41387]: from='client.15714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:12.291 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:11 smithi151 ceph-mon[41387]: pgmap v87: 89 pgs: 89 active+clean; 227 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:23:12.291 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:11 smithi151 ceph-mon[41387]: osdmap e211: 8 total, 8 up, 8 in 2024-09-13T14:23:12.291 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:11 smithi151 systemd[1]: Starting Ceph mgr.y for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:23:12.292 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:12 smithi151 podman[94843]: 2024-09-13 14:23:12.155737562 +0000 UTC m=+0.110810851 container create d8c169a4f36ef4d11ce368dd05af6808863a872fc2e5e325f786a7b17a622f81 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:23:12.292 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:12 smithi151 podman[94843]: 2024-09-13 14:23:12.060669302 +0000 UTC m=+0.015742594 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:23:12.815 INFO:tasks.rados.rados.0.smithi151.stdout:593: done (2 left) 2024-09-13T14:23:12.815 INFO:tasks.rados.rados.0.smithi151.stdout:594: done (1 left) 2024-09-13T14:23:12.815 INFO:tasks.rados.rados.0.smithi151.stdout:595: done (0 left) 2024-09-13T14:23:12.815 INFO:tasks.rados.rados.0.smithi151.stdout:596: snap_create 2024-09-13T14:23:12.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:12 smithi151 podman[94843]: 2024-09-13 14:23:12.417744148 +0000 UTC m=+0.372817435 container init d8c169a4f36ef4d11ce368dd05af6808863a872fc2e5e325f786a7b17a622f81 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-13T14:23:12.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:12 smithi151 podman[94843]: 2024-09-13 14:23:12.42258767 +0000 UTC m=+0.377660955 container start d8c169a4f36ef4d11ce368dd05af6808863a872fc2e5e325f786a7b17a622f81 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS) 2024-09-13T14:23:12.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:12 smithi151 ceph-mgr[94881]: -- 172.21.15.151:0/1460174634 <== mon.0 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561d01fef0e0 con 0x561d01fcd400 2024-09-13T14:23:12.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:12 smithi151 bash[94843]: d8c169a4f36ef4d11ce368dd05af6808863a872fc2e5e325f786a7b17a622f81 2024-09-13T14:23:12.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:12 smithi151 systemd[1]: Started Ceph mgr.y for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:23:13.615 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:23:13.826 INFO:tasks.rados.rados.0.smithi151.stdout:596: done (0 left) 2024-09-13T14:23:13.826 INFO:tasks.rados.rados.0.smithi151.stdout:597: read oid 13 snap -1 2024-09-13T14:23:13.826 INFO:tasks.rados.rados.0.smithi151.stdout:597: expect (ObjNum 168 snap 44 seq_num 168) 2024-09-13T14:23:13.826 INFO:tasks.rados.rados.0.smithi151.stdout:598: snap_remove snap 35 2024-09-13T14:23:13.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:13 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:13.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:13 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:13.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:13 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:13.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:13 smithi184 ceph-mon[33394]: osdmap e212: 8 total, 8 up, 8 in 2024-09-13T14:23:13.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:13 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/2509604700' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:13.858 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:23:13 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[93024]: ts=2024-09-13T14:23:13.508Z caller=notify.go:848 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] aggrGroup="{}/{}:{alertname=\"CephNodeDiskspaceWarning\"}" msg="Notify attempt failed, will retry later" attempts=1 err="Post \"\": dial tcp 172.21.15.151:8443: connect: connection refused" 2024-09-13T14:23:13.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:13 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:13.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:13 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:13.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:13 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:13.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:13 smithi151 ceph-mon[29807]: osdmap e212: 8 total, 8 up, 8 in 2024-09-13T14:23:13.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:13 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/2509604700' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:13.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:13 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:13.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:13 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:13.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:13 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:13.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:13 smithi151 ceph-mon[41387]: osdmap e212: 8 total, 8 up, 8 in 2024-09-13T14:23:13.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:13 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/2509604700' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:14.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:14 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:23:14.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:23:14.826 INFO:tasks.rados.rados.0.smithi151.stdout:598: done (1 left) 2024-09-13T14:23:14.826 INFO:tasks.rados.rados.0.smithi151.stdout:599: write oid 12 current snap is 81 2024-09-13T14:23:14.826 INFO:tasks.rados.rados.0.smithi151.stdout:599: seq_num 258 ranges {736206=453891,1881413=454793} 2024-09-13T14:23:14.833 INFO:tasks.rados.rados.0.smithi151.stdout:599: writing smithi15186934-12 from 736206 to 1190097 tid 1 2024-09-13T14:23:14.840 INFO:tasks.rados.rados.0.smithi151.stdout:599: writing smithi15186934-12 from 1881413 to 2336206 tid 2 2024-09-13T14:23:14.841 INFO:tasks.rados.rados.0.smithi151.stdout:600: read oid 41 snap 40 2024-09-13T14:23:14.841 INFO:tasks.rados.rados.0.smithi151.stdout:600: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-13T14:23:14.841 INFO:tasks.rados.rados.0.smithi151.stdout:601: write oid 34 current snap is 81 2024-09-13T14:23:14.841 INFO:tasks.rados.rados.0.smithi151.stdout:601: seq_num 259 ranges {753013=645493,2056248=296765} 2024-09-13T14:23:14.849 INFO:tasks.rados.rados.0.smithi151.stdout:601: writing smithi15186934-34 from 753013 to 1398506 tid 1 2024-09-13T14:23:14.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:14 smithi184 ceph-mon[33394]: pgmap v90: 89 pgs: 89 active+clean; 233 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-13T14:23:14.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:14 smithi184 ceph-mon[33394]: from='client.15732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:14.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:14 smithi184 ceph-mon[33394]: osdmap e213: 8 total, 8 up, 8 in 2024-09-13T14:23:14.855 INFO:tasks.rados.rados.0.smithi151.stdout:601: writing smithi15186934-34 from 2056248 to 2353013 tid 2 2024-09-13T14:23:14.855 INFO:tasks.rados.rados.0.smithi151.stdout:602: delete oid 8 current snap is 81 2024-09-13T14:23:14.855 INFO:tasks.rados.rados.0.smithi151.stdout:597: expect (ObjNum 168 snap 44 seq_num 168) 2024-09-13T14:23:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:14 smithi151 ceph-mon[29807]: pgmap v90: 89 pgs: 89 active+clean; 233 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-13T14:23:14.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:14 smithi151 ceph-mon[29807]: from='client.15732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:14.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:14 smithi151 ceph-mon[29807]: osdmap e213: 8 total, 8 up, 8 in 2024-09-13T14:23:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:14 smithi151 ceph-mon[41387]: pgmap v90: 89 pgs: 89 active+clean; 233 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-13T14:23:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:14 smithi151 ceph-mon[41387]: from='client.15732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:14 smithi151 ceph-mon[41387]: osdmap e213: 8 total, 8 up, 8 in 2024-09-13T14:23:14.996 INFO:tasks.rados.rados.0.smithi151.stdout:599: finishing write tid 1 to smithi15186934-12 2024-09-13T14:23:14.996 INFO:tasks.rados.rados.0.smithi151.stdout:597: done (4 left) 2024-09-13T14:23:14.997 INFO:tasks.rados.rados.0.smithi151.stdout:602: done (3 left) 2024-09-13T14:23:14.997 INFO:tasks.rados.rados.0.smithi151.stdout:603: snap_remove snap 71 2024-09-13T14:23:15.827 INFO:tasks.rados.rados.0.smithi151.stdout:603: done (3 left) 2024-09-13T14:23:15.827 INFO:tasks.rados.rados.0.smithi151.stdout:604: snap_create 2024-09-13T14:23:15.827 INFO:tasks.rados.rados.0.smithi151.stdout:599: finishing write tid 2 to smithi15186934-12 2024-09-13T14:23:15.827 INFO:tasks.rados.rados.0.smithi151.stdout:599: finishing write tid 3 to smithi15186934-12 2024-09-13T14:23:15.828 INFO:tasks.rados.rados.0.smithi151.stdout:599: finishing write tid 4 to smithi15186934-12 2024-09-13T14:23:15.828 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 12 v 106 (ObjNum 258 snap 81 seq_num 258) dirty exists 2024-09-13T14:23:15.828 INFO:tasks.rados.rados.0.smithi151.stdout:599: left oid 12 (ObjNum 258 snap 81 seq_num 258) 2024-09-13T14:23:15.828 INFO:tasks.rados.rados.0.smithi151.stdout:600: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-13T14:23:15.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:15 smithi184 ceph-mon[33394]: from='client.25640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:15.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:15 smithi184 ceph-mon[33394]: osdmap e214: 8 total, 8 up, 8 in 2024-09-13T14:23:15.853 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:23:15.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:23:15.854 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:23:15.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:23:15.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:15 smithi151 ceph-mon[29807]: from='client.25640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:15.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:15 smithi151 ceph-mon[29807]: osdmap e214: 8 total, 8 up, 8 in 2024-09-13T14:23:15.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:15 smithi151 ceph-mon[41387]: from='client.25640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:15.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:15 smithi151 ceph-mon[41387]: osdmap e214: 8 total, 8 up, 8 in 2024-09-13T14:23:15.918 INFO:tasks.rados.rados.0.smithi151.stdout:601: finishing write tid 1 to smithi15186934-34 2024-09-13T14:23:15.918 INFO:tasks.rados.rados.0.smithi151.stdout:601: finishing write tid 2 to smithi15186934-34 2024-09-13T14:23:15.918 INFO:tasks.rados.rados.0.smithi151.stdout:601: finishing write tid 3 to smithi15186934-34 2024-09-13T14:23:15.918 INFO:tasks.rados.rados.0.smithi151.stdout:601: finishing write tid 4 to smithi15186934-34 2024-09-13T14:23:15.918 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 229 (ObjNum 259 snap 81 seq_num 259) dirty exists 2024-09-13T14:23:15.918 INFO:tasks.rados.rados.0.smithi151.stdout:601: left oid 34 (ObjNum 259 snap 81 seq_num 259) 2024-09-13T14:23:16.595 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:23:16.831 INFO:tasks.rados.rados.0.smithi151.stdout:599: done (3 left) 2024-09-13T14:23:16.831 INFO:tasks.rados.rados.0.smithi151.stdout:600: done (2 left) 2024-09-13T14:23:16.831 INFO:tasks.rados.rados.0.smithi151.stdout:601: done (1 left) 2024-09-13T14:23:16.832 INFO:tasks.rados.rados.0.smithi151.stdout:604: done (0 left) 2024-09-13T14:23:16.832 INFO:tasks.rados.rados.0.smithi151.stdout:605: write oid 46 current snap is 82 2024-09-13T14:23:16.832 INFO:tasks.rados.rados.0.smithi151.stdout:605: seq_num 261 ranges {786627=795991,2173212=213415} 2024-09-13T14:23:16.841 INFO:tasks.rados.rados.0.smithi151.stdout:605: writing smithi15186934-46 from 786627 to 1582618 tid 1 2024-09-13T14:23:16.846 INFO:tasks.rados.rados.0.smithi151.stdout:605: writing smithi15186934-46 from 2173212 to 2386627 tid 2 2024-09-13T14:23:16.846 INFO:tasks.rados.rados.0.smithi151.stdout:606: read oid 50 snap -1 2024-09-13T14:23:16.846 INFO:tasks.rados.rados.0.smithi151.stdout:606: expect (ObjNum 225 snap 66 seq_num 225) 2024-09-13T14:23:16.846 INFO:tasks.rados.rados.0.smithi151.stdout:607: delete oid 4 current snap is 82 2024-09-13T14:23:16.849 INFO:tasks.rados.rados.0.smithi151.stdout:605: finishing write tid 1 to smithi15186934-46 2024-09-13T14:23:16.849 INFO:tasks.rados.rados.0.smithi151.stdout:605: finishing write tid 2 to smithi15186934-46 2024-09-13T14:23:16.849 INFO:tasks.rados.rados.0.smithi151.stdout:605: finishing write tid 3 to smithi15186934-46 2024-09-13T14:23:16.850 INFO:tasks.rados.rados.0.smithi151.stdout:605: finishing write tid 4 to smithi15186934-46 2024-09-13T14:23:16.850 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 46 v 180 (ObjNum 261 snap 82 seq_num 261) dirty exists 2024-09-13T14:23:16.850 INFO:tasks.rados.rados.0.smithi151.stdout:605: left oid 46 (ObjNum 261 snap 82 seq_num 261) 2024-09-13T14:23:16.850 INFO:tasks.rados.rados.0.smithi151.stdout:605: done (2 left) 2024-09-13T14:23:16.850 INFO:tasks.rados.rados.0.smithi151.stdout:607: done (1 left) 2024-09-13T14:23:16.850 INFO:tasks.rados.rados.0.smithi151.stdout:608: write oid 43 current snap is 82 2024-09-13T14:23:16.850 INFO:tasks.rados.rados.0.smithi151.stdout:608: seq_num 263 ranges {420241=546489,1373823=544414} 2024-09-13T14:23:16.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:16 smithi184 ceph-mon[33394]: pgmap v93: 89 pgs: 89 active+clean; 233 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-13T14:23:16.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:16 smithi184 ceph-mon[33394]: osdmap e215: 8 total, 8 up, 8 in 2024-09-13T14:23:16.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:16 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/4269814203' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:16.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:16 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:23:16.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:16 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:16.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:16 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:16.856 INFO:tasks.rados.rados.0.smithi151.stdout:608: writing smithi15186934-43 from 420241 to 966730 tid 1 2024-09-13T14:23:16.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:16 smithi151 ceph-mon[41387]: pgmap v93: 89 pgs: 89 active+clean; 233 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-13T14:23:16.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:16 smithi151 ceph-mon[41387]: osdmap e215: 8 total, 8 up, 8 in 2024-09-13T14:23:16.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:16 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/4269814203' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:16.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:16 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:23:16.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:16 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:16.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:16 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:16.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:16 smithi151 ceph-mon[29807]: pgmap v93: 89 pgs: 89 active+clean; 233 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-13T14:23:16.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:16 smithi151 ceph-mon[29807]: osdmap e215: 8 total, 8 up, 8 in 2024-09-13T14:23:16.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:16 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/4269814203' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:16.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:16 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:23:16.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:16 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:16.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:16 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:16.863 INFO:tasks.rados.rados.0.smithi151.stdout:608: writing smithi15186934-43 from 1373823 to 1918237 tid 2 2024-09-13T14:23:16.863 INFO:tasks.rados.rados.0.smithi151.stdout:609: write oid 49 current snap is 82 2024-09-13T14:23:16.863 INFO:tasks.rados.rados.0.smithi151.stdout:609: seq_num 264 ranges {437048=621738,1832305=404425} 2024-09-13T14:23:16.869 INFO:tasks.rados.rados.0.smithi151.stdout:609: writing smithi15186934-49 from 437048 to 1058786 tid 1 2024-09-13T14:23:16.877 INFO:tasks.rados.rados.0.smithi151.stdout:609: writing smithi15186934-49 from 1832305 to 2236730 tid 2 2024-09-13T14:23:16.877 INFO:tasks.rados.rados.0.smithi151.stdout:610: write oid 21 current snap is 82 2024-09-13T14:23:16.877 INFO:tasks.rados.rados.0.smithi151.stdout:606: expect (ObjNum 225 snap 66 seq_num 225) 2024-09-13T14:23:17.137 INFO:tasks.rados.rados.0.smithi151.stdout:608: finishing write tid 1 to smithi15186934-43 2024-09-13T14:23:17.137 INFO:tasks.rados.rados.0.smithi151.stdout:608: finishing write tid 2 to smithi15186934-43 2024-09-13T14:23:17.137 INFO:tasks.rados.rados.0.smithi151.stdout:608: finishing write tid 3 to smithi15186934-43 2024-09-13T14:23:17.137 INFO:tasks.rados.rados.0.smithi151.stdout:608: finishing write tid 4 to smithi15186934-43 2024-09-13T14:23:17.137 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 43 v 355 (ObjNum 263 snap 82 seq_num 263) dirty exists 2024-09-13T14:23:17.137 INFO:tasks.rados.rados.0.smithi151.stdout:608: left oid 43 (ObjNum 263 snap 82 seq_num 263) 2024-09-13T14:23:17.138 INFO:tasks.rados.rados.0.smithi151.stdout:609: finishing write tid 1 to smithi15186934-49 2024-09-13T14:23:17.138 INFO:tasks.rados.rados.0.smithi151.stdout:610: seq_num 265 ranges {453855=696987,1890787=548083} 2024-09-13T14:23:17.143 INFO:tasks.rados.rados.0.smithi151.stdout:610: writing smithi15186934-21 from 453855 to 1150842 tid 1 2024-09-13T14:23:17.150 INFO:tasks.rados.rados.0.smithi151.stdout:610: writing smithi15186934-21 from 1890787 to 2438870 tid 2 2024-09-13T14:23:17.150 INFO:tasks.rados.rados.0.smithi151.stdout:606: done (3 left) 2024-09-13T14:23:17.150 INFO:tasks.rados.rados.0.smithi151.stdout:608: done (2 left) 2024-09-13T14:23:17.150 INFO:tasks.rados.rados.0.smithi151.stdout:611: snap_create 2024-09-13T14:23:17.150 INFO:tasks.rados.rados.0.smithi151.stdout:609: finishing write tid 2 to smithi15186934-49 2024-09-13T14:23:17.150 INFO:tasks.rados.rados.0.smithi151.stdout:609: finishing write tid 3 to smithi15186934-49 2024-09-13T14:23:17.150 INFO:tasks.rados.rados.0.smithi151.stdout:609: finishing write tid 4 to smithi15186934-49 2024-09-13T14:23:17.151 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 49 v 301 (ObjNum 264 snap 82 seq_num 264) dirty exists 2024-09-13T14:23:17.151 INFO:tasks.rados.rados.0.smithi151.stdout:609: left oid 49 (ObjNum 264 snap 82 seq_num 264) 2024-09-13T14:23:17.151 INFO:tasks.rados.rados.0.smithi151.stdout:610: finishing write tid 1 to smithi15186934-21 2024-09-13T14:23:17.152 INFO:tasks.rados.rados.0.smithi151.stdout:610: finishing write tid 2 to smithi15186934-21 2024-09-13T14:23:17.152 INFO:tasks.rados.rados.0.smithi151.stdout:610: finishing write tid 3 to smithi15186934-21 2024-09-13T14:23:17.153 INFO:tasks.rados.rados.0.smithi151.stdout:610: finishing write tid 4 to smithi15186934-21 2024-09-13T14:23:17.153 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 21 v 119 (ObjNum 265 snap 82 seq_num 265) dirty exists 2024-09-13T14:23:17.153 INFO:tasks.rados.rados.0.smithi151.stdout:610: left oid 21 (ObjNum 265 snap 82 seq_num 265) 2024-09-13T14:23:17.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:16 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:23:16.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi184\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:23:17.807 INFO:tasks.rados.rados.0.smithi151.stdout:609: done (2 left) 2024-09-13T14:23:17.807 INFO:tasks.rados.rados.0.smithi151.stdout:610: done (1 left) 2024-09-13T14:23:17.807 INFO:tasks.rados.rados.0.smithi151.stdout:611: done (0 left) 2024-09-13T14:23:17.807 INFO:tasks.rados.rados.0.smithi151.stdout:612: write oid 25 current snap is 83 2024-09-13T14:23:17.807 INFO:tasks.rados.rados.0.smithi151.stdout:612: seq_num 266 ranges {470662=772236,1832916=408094} 2024-09-13T14:23:17.813 INFO:tasks.rados.rados.0.smithi151.stdout:612: writing smithi15186934-25 from 470662 to 1242898 tid 1 2024-09-13T14:23:17.819 INFO:tasks.rados.rados.0.smithi151.stdout:612: writing smithi15186934-25 from 1832916 to 2241010 tid 2 2024-09-13T14:23:17.819 INFO:tasks.rados.rados.0.smithi151.stdout:613: snap_create 2024-09-13T14:23:17.819 INFO:tasks.rados.rados.0.smithi151.stdout:612: finishing write tid 1 to smithi15186934-25 2024-09-13T14:23:17.822 INFO:tasks.rados.rados.0.smithi151.stdout:612: finishing write tid 2 to smithi15186934-25 2024-09-13T14:23:17.822 INFO:tasks.rados.rados.0.smithi151.stdout:612: finishing write tid 3 to smithi15186934-25 2024-09-13T14:23:17.822 INFO:tasks.rados.rados.0.smithi151.stdout:612: finishing write tid 4 to smithi15186934-25 2024-09-13T14:23:17.822 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 332 (ObjNum 266 snap 83 seq_num 266) dirty exists 2024-09-13T14:23:17.822 INFO:tasks.rados.rados.0.smithi151.stdout:612: left oid 25 (ObjNum 266 snap 83 seq_num 266) 2024-09-13T14:23:17.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:17 smithi151 ceph-mon[29807]: from='client.25652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:17.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:17 smithi151 ceph-mon[29807]: osdmap e216: 8 total, 8 up, 8 in 2024-09-13T14:23:17.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:17 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:17.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:17 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:17.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:17 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:17.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:17 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:17.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:17 smithi151 ceph-mon[41387]: from='client.25652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:17.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:17 smithi151 ceph-mon[41387]: osdmap e216: 8 total, 8 up, 8 in 2024-09-13T14:23:17.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:17 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:17.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:17 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:17.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:17 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:17.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:17 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:17.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:17 smithi184 ceph-mon[33394]: from='client.25652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:17 smithi184 ceph-mon[33394]: osdmap e216: 8 total, 8 up, 8 in 2024-09-13T14:23:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:17 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:17 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:17 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:17 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:18.108 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:17 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: [13/Sep/2024:14:23:17] ENGINE Bus STARTING 2024-09-13T14:23:18.108 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:17 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: CherryPy Checker: 2024-09-13T14:23:18.108 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:17 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: The Application mounted at '' has an empty config. 2024-09-13T14:23:18.109 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:17 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:23:18.109 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:17 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: [13/Sep/2024:14:23:17] ENGINE Serving on http://:::9283 2024-09-13T14:23:18.109 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:17 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: [13/Sep/2024:14:23:17] ENGINE Bus STARTED 2024-09-13T14:23:18.601 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 14:23:18 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[93024]: ts=2024-09-13T14:23:18.348Z caller=notify.go:860 level=info component=dispatcher receiver=ceph-dashboard integration=webhook[1] aggrGroup="{}/{}:{alertname=\"CephNodeDiskspaceWarning\"}" msg="Notify success" attempts=5 duration=4.30599ms 2024-09-13T14:23:18.822 INFO:tasks.rados.rados.0.smithi151.stdout:612: done (1 left) 2024-09-13T14:23:18.822 INFO:tasks.rados.rados.0.smithi151.stdout:613: done (0 left) 2024-09-13T14:23:18.822 INFO:tasks.rados.rados.0.smithi151.stdout:614: delete oid 25 current snap is 84 2024-09-13T14:23:18.825 INFO:tasks.rados.rados.0.smithi151.stdout:614: done (0 left) 2024-09-13T14:23:18.825 INFO:tasks.rados.rados.0.smithi151.stdout:615: snap_remove snap 77 2024-09-13T14:23:18.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:18 smithi184 ceph-mon[33394]: from='client.15756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:18.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:18 smithi184 ceph-mon[33394]: pgmap v96: 89 pgs: 89 active+clean; 233 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 11 MiB/s rd, 1.3 MiB/s wr, 14 op/s 2024-09-13T14:23:18.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:18 smithi184 ceph-mon[33394]: osdmap e217: 8 total, 8 up, 8 in 2024-09-13T14:23:18.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:18 smithi184 ceph-mon[33394]: Standby manager daemon y restarted 2024-09-13T14:23:18.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:18 smithi184 ceph-mon[33394]: Standby manager daemon y started 2024-09-13T14:23:18.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:18 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-13T14:23:18.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:18 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:23:18.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:18 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-13T14:23:18.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:18 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:23:18.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:18 smithi151 ceph-mon[41387]: from='client.15756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:18.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:18 smithi151 ceph-mon[41387]: pgmap v96: 89 pgs: 89 active+clean; 233 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 11 MiB/s rd, 1.3 MiB/s wr, 14 op/s 2024-09-13T14:23:18.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:18 smithi151 ceph-mon[41387]: osdmap e217: 8 total, 8 up, 8 in 2024-09-13T14:23:18.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:18 smithi151 ceph-mon[41387]: Standby manager daemon y restarted 2024-09-13T14:23:18.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:18 smithi151 ceph-mon[41387]: Standby manager daemon y started 2024-09-13T14:23:18.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:18 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-13T14:23:18.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:18 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:23:18.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:18 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-13T14:23:18.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:18 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:23:18.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:18 smithi151 ceph-mon[29807]: from='client.15756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:18.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:18 smithi151 ceph-mon[29807]: pgmap v96: 89 pgs: 89 active+clean; 233 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 11 MiB/s rd, 1.3 MiB/s wr, 14 op/s 2024-09-13T14:23:18.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:18 smithi151 ceph-mon[29807]: osdmap e217: 8 total, 8 up, 8 in 2024-09-13T14:23:18.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:18 smithi151 ceph-mon[29807]: Standby manager daemon y restarted 2024-09-13T14:23:18.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:18 smithi151 ceph-mon[29807]: Standby manager daemon y started 2024-09-13T14:23:18.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:18 smithi151 ceph-mon[29807]: from='mgr.? 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-13T14:23:18.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:18 smithi151 ceph-mon[29807]: from='mgr.? 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:23:18.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:18 smithi151 ceph-mon[29807]: from='mgr.? 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-13T14:23:18.862 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:18 smithi151 ceph-mon[29807]: from='mgr.? 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:23:19.582 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:23:19.604 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:19 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:23:19] "GET /metrics HTTP/1.1" 200 36411 "" "Prometheus/2.51.0" 2024-09-13T14:23:19.831 INFO:tasks.rados.rados.0.smithi151.stdout:615: done (0 left) 2024-09-13T14:23:19.831 INFO:tasks.rados.rados.0.smithi151.stdout:616: snap_create 2024-09-13T14:23:19.845 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:19 smithi151 ceph-mon[29807]: mgrmap e26: x(active, since 63s), standbys: y 2024-09-13T14:23:19.846 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:19 smithi151 ceph-mon[29807]: osdmap e218: 8 total, 8 up, 8 in 2024-09-13T14:23:19.846 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:19 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/3846602181' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:19.846 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:19 smithi151 ceph-mon[29807]: pgmap v99: 89 pgs: 89 active+clean; 233 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 11 MiB/s rd, 1.3 MiB/s wr, 14 op/s 2024-09-13T14:23:19.846 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:19 smithi151 ceph-mon[29807]: from='client.15768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:19.846 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:19 smithi151 ceph-mon[41387]: mgrmap e26: x(active, since 63s), standbys: y 2024-09-13T14:23:19.846 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:19 smithi151 ceph-mon[41387]: osdmap e218: 8 total, 8 up, 8 in 2024-09-13T14:23:19.846 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:19 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/3846602181' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:19.846 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:19 smithi151 ceph-mon[41387]: pgmap v99: 89 pgs: 89 active+clean; 233 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 11 MiB/s rd, 1.3 MiB/s wr, 14 op/s 2024-09-13T14:23:19.846 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:19 smithi151 ceph-mon[41387]: from='client.15768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:20.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:19 smithi184 ceph-mon[33394]: mgrmap e26: x(active, since 63s), standbys: y 2024-09-13T14:23:20.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:19 smithi184 ceph-mon[33394]: osdmap e218: 8 total, 8 up, 8 in 2024-09-13T14:23:20.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:19 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/3846602181' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:20.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:19 smithi184 ceph-mon[33394]: pgmap v99: 89 pgs: 89 active+clean; 233 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 11 MiB/s rd, 1.3 MiB/s wr, 14 op/s 2024-09-13T14:23:20.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:19 smithi184 ceph-mon[33394]: from='client.15768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:20.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:23:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:23:20.779 INFO:tasks.rados.rados.0.smithi151.stdout:616: done (0 left) 2024-09-13T14:23:20.779 INFO:tasks.rados.rados.0.smithi151.stdout:617: snap_create 2024-09-13T14:23:20.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: [13/Sep/2024:14:23:20] ENGINE Bus STOPPING 2024-09-13T14:23:20.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[41387]: osdmap e219: 8 total, 8 up, 8 in 2024-09-13T14:23:20.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[41387]: from='client.25676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:20.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:20.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:20.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:20.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:23:20.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:20.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:20.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[41387]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-13T14:23:20.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-13T14:23:20.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[41387]: Activating manager daemon y 2024-09-13T14:23:20.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[41387]: osdmap e220: 8 total, 8 up, 8 in 2024-09-13T14:23:20.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[41387]: from='mgr.24916 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-13T14:23:20.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[41387]: mgrmap e27: y(active, starting, since 0.00622609s) 2024-09-13T14:23:20.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:23:20.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:23:20.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:23:20.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi184.jismsi"}]: dispatch 2024-09-13T14:23:20.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi151.rqqklw"}]: dispatch 2024-09-13T14:23:20.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:23:20.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:23:20.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:23:20.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:23:20.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:23:20.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:23:20.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:23:20.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:23:20.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:23:20.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:23:20.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:23:20.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:23:20.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[29807]: osdmap e219: 8 total, 8 up, 8 in 2024-09-13T14:23:20.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[29807]: from='client.25676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:20.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:20.862 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:20.862 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:20.862 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:23:20.862 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:20.862 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:20.862 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[29807]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-13T14:23:20.862 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-13T14:23:20.862 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[29807]: Activating manager daemon y 2024-09-13T14:23:20.862 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[29807]: osdmap e220: 8 total, 8 up, 8 in 2024-09-13T14:23:20.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[29807]: from='mgr.24916 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-13T14:23:20.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[29807]: mgrmap e27: y(active, starting, since 0.00622609s) 2024-09-13T14:23:20.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:23:20.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:23:20.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:23:20.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi184.jismsi"}]: dispatch 2024-09-13T14:23:20.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi151.rqqklw"}]: dispatch 2024-09-13T14:23:20.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:23:20.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:23:20.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:23:20.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:23:20.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:23:20.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:23:20.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:23:20.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:23:20.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:23:20.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:23:20.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:23:20.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:23:21.103 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: 2024-09-13T14:23:20.777+0000 7f20b4db8640 -1 mgr handle_mgr_map I was active but no longer am 2024-09-13T14:23:21.103 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: ignoring --setuser ceph since I am not root 2024-09-13T14:23:21.104 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: ignoring --setgroup ceph since I am not root 2024-09-13T14:23:21.104 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mgr[69248]: -- 172.21.15.184:0/3479955561 <== mon.1 v2:172.21.15.184:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564cf2b410e0 con 0x564cf2b1f400 2024-09-13T14:23:21.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mon[33394]: osdmap e219: 8 total, 8 up, 8 in 2024-09-13T14:23:21.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mon[33394]: from='client.25676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:21.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:21.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:21.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:21.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:23:21.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' 2024-09-13T14:23:21.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:21.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mon[33394]: from='mgr.24916 172.21.15.184:0/1813345893' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-13T14:23:21.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-13T14:23:21.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mon[33394]: Activating manager daemon y 2024-09-13T14:23:21.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mon[33394]: osdmap e220: 8 total, 8 up, 8 in 2024-09-13T14:23:21.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mon[33394]: from='mgr.24916 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-13T14:23:21.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mon[33394]: mgrmap e27: y(active, starting, since 0.00622609s) 2024-09-13T14:23:21.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:23:21.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:23:21.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:23:21.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi184.jismsi"}]: dispatch 2024-09-13T14:23:21.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi151.rqqklw"}]: dispatch 2024-09-13T14:23:21.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:23:21.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:23:21.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:23:21.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:23:21.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:23:21.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:23:21.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:23:21.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:23:21.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:23:21.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:23:21.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:23:21.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:20 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:23:21.270 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: [13/Sep/2024:14:23:20] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:23:21.270 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:20 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: [13/Sep/2024:14:23:20] ENGINE Bus STOPPED 2024-09-13T14:23:21.608 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:21 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:23:21.267+0000 7f5165ab9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:23:21.608 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:21 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:23:21.267+0000 7f5165ab9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:23:21.608 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:21 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:23:21.267+0000 7f5165ab9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:23:21.608 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:21 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:23:21.267+0000 7f5165ab9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:23:21.609 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:21 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:23:21.267+0000 7f5165ab9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:23:21.609 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:21 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:23:21.270+0000 7f5168abf640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:23:21.609 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:21 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:23:21.270+0000 7f5168abf640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:23:21.609 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:21 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:23:21.270+0000 7f5168abf640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:23:21.609 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:21 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:23:21.270+0000 7f5168abf640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:23:21.609 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:21 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:23:21.270+0000 7f5168abf640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:23:21.779 INFO:tasks.rados.rados.0.smithi151.stdout:617: done (0 left) 2024-09-13T14:23:21.779 INFO:tasks.rados.rados.0.smithi151.stdout:618: write oid 9 current snap is 86 2024-09-13T14:23:21.779 INFO:tasks.rados.rados.0.smithi151.stdout:618: seq_num 268 ranges {504276=639087,1666233=695410} 2024-09-13T14:23:21.785 INFO:tasks.rados.rados.0.smithi151.stdout:618: writing smithi15186934-9 from 504276 to 1143363 tid 1 2024-09-13T14:23:21.792 INFO:tasks.rados.rados.0.smithi151.stdout:618: writing smithi15186934-9 from 1666233 to 2361643 tid 2 2024-09-13T14:23:21.792 INFO:tasks.rados.rados.0.smithi151.stdout:618: finishing write tid 1 to smithi15186934-9 2024-09-13T14:23:21.792 INFO:tasks.rados.rados.0.smithi151.stdout:619: write oid 15 current snap is 86 2024-09-13T14:23:21.793 INFO:tasks.rados.rados.0.smithi151.stdout:619: seq_num 269 ranges {521083=714336,1724715=555421} 2024-09-13T14:23:21.799 INFO:tasks.rados.rados.0.smithi151.stdout:619: writing smithi15186934-15 from 521083 to 1235419 tid 1 2024-09-13T14:23:21.806 INFO:tasks.rados.rados.0.smithi151.stdout:619: writing smithi15186934-15 from 1724715 to 2280136 tid 2 2024-09-13T14:23:21.806 INFO:tasks.rados.rados.0.smithi151.stdout:620: read oid 50 snap -1 2024-09-13T14:23:21.806 INFO:tasks.rados.rados.0.smithi151.stdout:620: expect (ObjNum 225 snap 66 seq_num 225) 2024-09-13T14:23:21.807 INFO:tasks.rados.rados.0.smithi151.stdout:621: read oid 47 snap 63 2024-09-13T14:23:21.807 INFO:tasks.rados.rados.0.smithi151.stdout:621: expect (ObjNum 155 snap 38 seq_num 155) 2024-09-13T14:23:21.807 INFO:tasks.rados.rados.0.smithi151.stdout:622: read oid 34 snap -1 2024-09-13T14:23:21.807 INFO:tasks.rados.rados.0.smithi151.stdout:622: expect (ObjNum 259 snap 81 seq_num 259) 2024-09-13T14:23:21.807 INFO:tasks.rados.rados.0.smithi151.stdout:623: write oid 13 current snap is 86 2024-09-13T14:23:21.807 INFO:tasks.rados.rados.0.smithi151.stdout:618: finishing write tid 2 to smithi15186934-9 2024-09-13T14:23:21.807 INFO:tasks.rados.rados.0.smithi151.stdout:618: finishing write tid 3 to smithi15186934-9 2024-09-13T14:23:21.807 INFO:tasks.rados.rados.0.smithi151.stdout:618: finishing write tid 4 to smithi15186934-9 2024-09-13T14:23:21.807 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 253 (ObjNum 268 snap 86 seq_num 268) dirty exists 2024-09-13T14:23:21.807 INFO:tasks.rados.rados.0.smithi151.stdout:618: left oid 9 (ObjNum 268 snap 86 seq_num 268) 2024-09-13T14:23:21.807 INFO:tasks.rados.rados.0.smithi151.stdout:619: finishing write tid 1 to smithi15186934-15 2024-09-13T14:23:21.807 INFO:tasks.rados.rados.0.smithi151.stdout:623: seq_num 270 ranges {537890=789585,1783197=699079} 2024-09-13T14:23:21.814 INFO:tasks.rados.rados.0.smithi151.stdout:623: writing smithi15186934-13 from 537890 to 1327475 tid 1 2024-09-13T14:23:21.821 INFO:tasks.rados.rados.0.smithi151.stdout:623: writing smithi15186934-13 from 1783197 to 2482276 tid 2 2024-09-13T14:23:21.821 INFO:tasks.rados.rados.0.smithi151.stdout:618: done (5 left) 2024-09-13T14:23:21.821 INFO:tasks.rados.rados.0.smithi151.stdout:624: read oid 12 snap 73 2024-09-13T14:23:21.821 INFO:tasks.rados.rados.0.smithi151.stdout:624: expect (ObjNum 167 snap 43 seq_num 167) 2024-09-13T14:23:21.821 INFO:tasks.rados.rados.0.smithi151.stdout:625: snap_remove snap 36 2024-09-13T14:23:22.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:21 smithi184 ceph-mon[33394]: Manager daemon y is now available 2024-09-13T14:23:22.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:21 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:22.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:21 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:23:22.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:21 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:23:22.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:21 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:23:22.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:21 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:23:22.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:21 smithi184 ceph-mon[33394]: osdmap e221: 8 total, 8 up, 8 in 2024-09-13T14:23:22.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:21 smithi151 ceph-mon[29807]: Manager daemon y is now available 2024-09-13T14:23:22.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:21 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:22.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:21 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:23:22.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:21 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:23:22.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:21 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:23:22.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:21 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:23:22.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:21 smithi151 ceph-mon[29807]: osdmap e221: 8 total, 8 up, 8 in 2024-09-13T14:23:22.109 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:21 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: [13/Sep/2024:14:23:21] ENGINE Bus STARTING 2024-09-13T14:23:22.109 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:21 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: [13/Sep/2024:14:23:21] ENGINE Serving on http://:::9283 2024-09-13T14:23:22.109 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:21 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: [13/Sep/2024:14:23:21] ENGINE Bus STARTED 2024-09-13T14:23:22.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:21 smithi151 ceph-mon[41387]: Manager daemon y is now available 2024-09-13T14:23:22.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:21 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:22.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:21 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:23:22.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:21 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:23:22.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:21 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:23:22.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:21 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:23:22.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:21 smithi151 ceph-mon[41387]: osdmap e221: 8 total, 8 up, 8 in 2024-09-13T14:23:22.561 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:23:22.680 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:23:22.386+0000 7f514c04c640 -1 log_channel(cephadm) log [ERR] : [13/Sep/2024:14:23:22] ENGINE Error in HTTPServer.serve 2024-09-13T14:23:22.680 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: Traceback (most recent call last): 2024-09-13T14:23:22.680 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:23:22.680 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: self._connections.run(self.expiration_interval) 2024-09-13T14:23:22.680 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:23:22.681 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: self._run(expiration_interval) 2024-09-13T14:23:22.681 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:23:22.681 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:23:22.681 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:23:22.681 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:23:22.681 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:23:22.681 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: s = self.context.wrap_socket( 2024-09-13T14:23:22.681 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:23:22.681 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: return self.sslsocket_class._create( 2024-09-13T14:23:22.681 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:23:22.681 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: self.do_handshake() 2024-09-13T14:23:22.681 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:23:22.681 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: self._sslobj.do_handshake() 2024-09-13T14:23:22.682 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:23:22.682 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:23:22.783 INFO:tasks.rados.rados.0.smithi151.stdout:625: done (6 left) 2024-09-13T14:23:22.783 INFO:tasks.rados.rados.0.smithi151.stdout:626: snap_remove snap 74 2024-09-13T14:23:22.812 INFO:tasks.rados.rados.0.smithi151.stdout:626: done (6 left) 2024-09-13T14:23:22.812 INFO:tasks.rados.rados.0.smithi151.stdout:627: rollback oid 45 current snap is 86 2024-09-13T14:23:22.812 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 45 to 83 2024-09-13T14:23:22.812 INFO:tasks.rados.rados.0.smithi151.stdout:628: snap_remove snap 68 2024-09-13T14:23:23.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:22 smithi184 ceph-mon[33394]: mgrmap e28: y(active, since 1.06472s) 2024-09-13T14:23:23.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:22 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:23.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:22 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:23.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:22 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/3015102750' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:23.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:22 smithi184 ceph-mon[33394]: [13/Sep/2024:14:23:22] ENGINE Bus STARTING 2024-09-13T14:23:23.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:22 smithi184 ceph-mon[33394]: [13/Sep/2024:14:23:22] ENGINE Serving on https://172.21.15.151:7150 2024-09-13T14:23:23.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:22 smithi184 ceph-mon[33394]: [13/Sep/2024:14:23:22] ENGINE Error in HTTPServer.serve 2024-09-13T14:23:23.104 INFO:journalctl@ceph.mon.b.smithi184.stdout: Traceback (most recent call last): 2024-09-13T14:23:23.104 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:23:23.104 INFO:journalctl@ceph.mon.b.smithi184.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:23:23.104 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:23:23.105 INFO:journalctl@ceph.mon.b.smithi184.stdout: self._run(expiration_interval) 2024-09-13T14:23:23.105 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:23:23.105 INFO:journalctl@ceph.mon.b.smithi184.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:23:23.105 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:23:23.105 INFO:journalctl@ceph.mon.b.smithi184.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:23:23.105 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:23:23.105 INFO:journalctl@ceph.mon.b.smithi184.stdout: s = self.context.wrap_socket( 2024-09-13T14:23:23.105 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:23:23.105 INFO:journalctl@ceph.mon.b.smithi184.stdout: return self.sslsocket_class._create( 2024-09-13T14:23:23.105 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:23:23.105 INFO:journalctl@ceph.mon.b.smithi184.stdout: self.do_handshake() 2024-09-13T14:23:23.105 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:23:23.105 INFO:journalctl@ceph.mon.b.smithi184.stdout: self._sslobj.do_handshake() 2024-09-13T14:23:23.105 INFO:journalctl@ceph.mon.b.smithi184.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:23:23.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:22 smithi184 ceph-mon[33394]: osdmap e222: 8 total, 8 up, 8 in 2024-09-13T14:23:23.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:22 smithi184 ceph-mon[33394]: osdmap e223: 8 total, 8 up, 8 in 2024-09-13T14:23:23.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-mon[29807]: mgrmap e28: y(active, since 1.06472s) 2024-09-13T14:23:23.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:23.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:23.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/3015102750' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:23.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-mon[29807]: [13/Sep/2024:14:23:22] ENGINE Bus STARTING 2024-09-13T14:23:23.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-mon[29807]: [13/Sep/2024:14:23:22] ENGINE Serving on https://172.21.15.151:7150 2024-09-13T14:23:23.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-mon[29807]: [13/Sep/2024:14:23:22] ENGINE Error in HTTPServer.serve 2024-09-13T14:23:23.109 INFO:journalctl@ceph.mon.a.smithi151.stdout: Traceback (most recent call last): 2024-09-13T14:23:23.109 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:23:23.109 INFO:journalctl@ceph.mon.a.smithi151.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:23:23.109 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:23:23.109 INFO:journalctl@ceph.mon.a.smithi151.stdout: self._run(expiration_interval) 2024-09-13T14:23:23.109 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:23:23.109 INFO:journalctl@ceph.mon.a.smithi151.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:23:23.109 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:23:23.109 INFO:journalctl@ceph.mon.a.smithi151.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:23:23.109 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:23:23.109 INFO:journalctl@ceph.mon.a.smithi151.stdout: s = self.context.wrap_socket( 2024-09-13T14:23:23.110 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:23:23.110 INFO:journalctl@ceph.mon.a.smithi151.stdout: return self.sslsocket_class._create( 2024-09-13T14:23:23.110 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:23:23.110 INFO:journalctl@ceph.mon.a.smithi151.stdout: self.do_handshake() 2024-09-13T14:23:23.110 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:23:23.110 INFO:journalctl@ceph.mon.a.smithi151.stdout: self._sslobj.do_handshake() 2024-09-13T14:23:23.110 INFO:journalctl@ceph.mon.a.smithi151.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:23:23.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-mon[29807]: osdmap e222: 8 total, 8 up, 8 in 2024-09-13T14:23:23.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-mon[29807]: osdmap e223: 8 total, 8 up, 8 in 2024-09-13T14:23:23.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-mon[41387]: mgrmap e28: y(active, since 1.06472s) 2024-09-13T14:23:23.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:23.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:23.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/3015102750' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:23.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-mon[41387]: [13/Sep/2024:14:23:22] ENGINE Bus STARTING 2024-09-13T14:23:23.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-mon[41387]: [13/Sep/2024:14:23:22] ENGINE Serving on https://172.21.15.151:7150 2024-09-13T14:23:23.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-mon[41387]: [13/Sep/2024:14:23:22] ENGINE Error in HTTPServer.serve 2024-09-13T14:23:23.111 INFO:journalctl@ceph.mon.c.smithi151.stdout: Traceback (most recent call last): 2024-09-13T14:23:23.111 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:23:23.111 INFO:journalctl@ceph.mon.c.smithi151.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:23:23.111 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:23:23.111 INFO:journalctl@ceph.mon.c.smithi151.stdout: self._run(expiration_interval) 2024-09-13T14:23:23.111 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:23:23.111 INFO:journalctl@ceph.mon.c.smithi151.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:23:23.111 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:23:23.111 INFO:journalctl@ceph.mon.c.smithi151.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:23:23.111 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:23:23.111 INFO:journalctl@ceph.mon.c.smithi151.stdout: s = self.context.wrap_socket( 2024-09-13T14:23:23.112 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:23:23.112 INFO:journalctl@ceph.mon.c.smithi151.stdout: return self.sslsocket_class._create( 2024-09-13T14:23:23.112 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:23:23.112 INFO:journalctl@ceph.mon.c.smithi151.stdout: self.do_handshake() 2024-09-13T14:23:23.112 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:23:23.112 INFO:journalctl@ceph.mon.c.smithi151.stdout: self._sslobj.do_handshake() 2024-09-13T14:23:23.112 INFO:journalctl@ceph.mon.c.smithi151.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:23:23.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-mon[41387]: osdmap e222: 8 total, 8 up, 8 in 2024-09-13T14:23:23.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:22 smithi151 ceph-mon[41387]: osdmap e223: 8 total, 8 up, 8 in 2024-09-13T14:23:23.811 INFO:tasks.rados.rados.0.smithi151.stdout:628: done (7 left) 2024-09-13T14:23:23.811 INFO:tasks.rados.rados.0.smithi151.stdout:629: read oid 17 snap -1 2024-09-13T14:23:23.811 INFO:tasks.rados.rados.0.smithi151.stdout:629: expect (ObjNum 253 snap 79 seq_num 253) 2024-09-13T14:23:23.811 INFO:tasks.rados.rados.0.smithi151.stdout:630: write oid 22 current snap is 86 2024-09-13T14:23:23.811 INFO:tasks.rados.rados.0.smithi151.stdout:630: seq_num 271 ranges {554697=464834,1725326=559090} 2024-09-13T14:23:23.817 INFO:tasks.rados.rados.0.smithi151.stdout:630: writing smithi15186934-22 from 554697 to 1019531 tid 1 2024-09-13T14:23:23.825 INFO:tasks.rados.rados.0.smithi151.stdout:630: writing smithi15186934-22 from 1725326 to 2284416 tid 2 2024-09-13T14:23:23.825 INFO:tasks.rados.rados.0.smithi151.stdout:631: snap_remove snap 76 2024-09-13T14:23:24.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:23 smithi184 ceph-mon[33394]: [13/Sep/2024:14:23:22] ENGINE Serving on http://172.21.15.151:8765 2024-09-13T14:23:24.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:23 smithi184 ceph-mon[33394]: [13/Sep/2024:14:23:22] ENGINE Bus STARTED 2024-09-13T14:23:24.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:23 smithi184 ceph-mon[33394]: from='client.15819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:24.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:23 smithi184 ceph-mon[33394]: pgmap v6: 89 pgs: 89 active+clean; 239 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:23:24.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:23 smithi184 ceph-mon[33394]: mgrmap e29: y(active, since 2s) 2024-09-13T14:23:24.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:23 smithi184 ceph-mon[33394]: from='client.15825 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:24.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:23 smithi184 ceph-mon[33394]: osdmap e224: 8 total, 8 up, 8 in 2024-09-13T14:23:24.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:23 smithi151 ceph-mon[41387]: [13/Sep/2024:14:23:22] ENGINE Serving on http://172.21.15.151:8765 2024-09-13T14:23:24.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:23 smithi151 ceph-mon[41387]: [13/Sep/2024:14:23:22] ENGINE Bus STARTED 2024-09-13T14:23:24.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:23 smithi151 ceph-mon[41387]: from='client.15819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:24.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:23 smithi151 ceph-mon[41387]: pgmap v6: 89 pgs: 89 active+clean; 239 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:23:24.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:23 smithi151 ceph-mon[41387]: mgrmap e29: y(active, since 2s) 2024-09-13T14:23:24.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:23 smithi151 ceph-mon[41387]: from='client.15825 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:24.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:23 smithi151 ceph-mon[41387]: osdmap e224: 8 total, 8 up, 8 in 2024-09-13T14:23:24.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:23 smithi151 ceph-mon[29807]: [13/Sep/2024:14:23:22] ENGINE Serving on http://172.21.15.151:8765 2024-09-13T14:23:24.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:23 smithi151 ceph-mon[29807]: [13/Sep/2024:14:23:22] ENGINE Bus STARTED 2024-09-13T14:23:24.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:23 smithi151 ceph-mon[29807]: from='client.15819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:24.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:23 smithi151 ceph-mon[29807]: pgmap v6: 89 pgs: 89 active+clean; 239 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:23:24.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:23 smithi151 ceph-mon[29807]: mgrmap e29: y(active, since 2s) 2024-09-13T14:23:24.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:23 smithi151 ceph-mon[29807]: from='client.15825 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:24.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:23 smithi151 ceph-mon[29807]: osdmap e224: 8 total, 8 up, 8 in 2024-09-13T14:23:24.397 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:24 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:23:24.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:23:24.815 INFO:tasks.rados.rados.0.smithi151.stdout:631: done (9 left) 2024-09-13T14:23:24.815 INFO:tasks.rados.rados.0.smithi151.stdout:632: write oid 21 current snap is 86 2024-09-13T14:23:24.816 INFO:tasks.rados.rados.0.smithi151.stdout:632: seq_num 272 ranges {571504=540083,1783808=702748} 2024-09-13T14:23:24.823 INFO:tasks.rados.rados.0.smithi151.stdout:632: writing smithi15186934-21 from 571504 to 1111587 tid 1 2024-09-13T14:23:24.831 INFO:tasks.rados.rados.0.smithi151.stdout:632: writing smithi15186934-21 from 1783808 to 2486556 tid 2 2024-09-13T14:23:24.831 INFO:tasks.rados.rados.0.smithi151.stdout:633: delete oid 36 current snap is 86 2024-09-13T14:23:24.831 INFO:tasks.rados.rados.0.smithi151.stdout:619: finishing write tid 2 to smithi15186934-15 2024-09-13T14:23:24.831 INFO:tasks.rados.rados.0.smithi151.stdout:619: finishing write tid 3 to smithi15186934-15 2024-09-13T14:23:24.832 INFO:tasks.rados.rados.0.smithi151.stdout:619: finishing write tid 4 to smithi15186934-15 2024-09-13T14:23:24.832 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 15 v 239 (ObjNum 269 snap 86 seq_num 269) dirty exists 2024-09-13T14:23:24.832 INFO:tasks.rados.rados.0.smithi151.stdout:619: left oid 15 (ObjNum 269 snap 86 seq_num 269) 2024-09-13T14:23:24.832 INFO:tasks.rados.rados.0.smithi151.stdout:622: expect (ObjNum 259 snap 81 seq_num 259) 2024-09-13T14:23:24.973 INFO:tasks.rados.rados.0.smithi151.stdout:621: expect (ObjNum 155 snap 38 seq_num 155) 2024-09-13T14:23:25.093 INFO:tasks.rados.rados.0.smithi151.stdout:623: finishing write tid 1 to smithi15186934-13 2024-09-13T14:23:25.093 INFO:tasks.rados.rados.0.smithi151.stdout:620: expect (ObjNum 225 snap 66 seq_num 225) 2024-09-13T14:23:25.336 INFO:tasks.rados.rados.0.smithi151.stdout:623: finishing write tid 2 to smithi15186934-13 2024-09-13T14:23:25.337 INFO:tasks.rados.rados.0.smithi151.stdout:623: finishing write tid 3 to smithi15186934-13 2024-09-13T14:23:25.337 INFO:tasks.rados.rados.0.smithi151.stdout:619: done (10 left) 2024-09-13T14:23:25.337 INFO:tasks.rados.rados.0.smithi151.stdout:620: done (9 left) 2024-09-13T14:23:25.337 INFO:tasks.rados.rados.0.smithi151.stdout:621: done (8 left) 2024-09-13T14:23:25.337 INFO:tasks.rados.rados.0.smithi151.stdout:622: done (7 left) 2024-09-13T14:23:25.337 INFO:tasks.rados.rados.0.smithi151.stdout:633: done (6 left) 2024-09-13T14:23:25.337 INFO:tasks.rados.rados.0.smithi151.stdout:634: snap_create 2024-09-13T14:23:25.338 INFO:tasks.rados.rados.0.smithi151.stdout:623: finishing write tid 4 to smithi15186934-13 2024-09-13T14:23:25.338 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 13 v 152 (ObjNum 270 snap 86 seq_num 270) dirty exists 2024-09-13T14:23:25.338 INFO:tasks.rados.rados.0.smithi151.stdout:623: left oid 13 (ObjNum 270 snap 86 seq_num 270) 2024-09-13T14:23:25.338 INFO:tasks.rados.rados.0.smithi151.stdout:624: expect (ObjNum 167 snap 43 seq_num 167) 2024-09-13T14:23:25.457 INFO:tasks.rados.rados.0.smithi151.stdout:627: finishing rollback tid 0 to smithi15186934-45 2024-09-13T14:23:25.457 INFO:tasks.rados.rados.0.smithi151.stdout:627: finishing rollback tid 1 to smithi15186934-45 2024-09-13T14:23:25.457 INFO:tasks.rados.rados.0.smithi151.stdout:627: finishing rollback tid 2 to smithi15186934-45 2024-09-13T14:23:25.457 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 356 (ObjNum 251 snap 78 seq_num 251) dirty exists 2024-09-13T14:23:25.458 INFO:tasks.rados.rados.0.smithi151.stdout:629: expect (ObjNum 253 snap 79 seq_num 253) 2024-09-13T14:23:25.518 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:23:25.565 INFO:tasks.rados.rados.0.smithi151.stdout:630: finishing write tid 1 to smithi15186934-22 2024-09-13T14:23:25.565 INFO:tasks.rados.rados.0.smithi151.stdout:630: finishing write tid 2 to smithi15186934-22 2024-09-13T14:23:25.565 INFO:tasks.rados.rados.0.smithi151.stdout:630: finishing write tid 3 to smithi15186934-22 2024-09-13T14:23:25.565 INFO:tasks.rados.rados.0.smithi151.stdout:630: finishing write tid 4 to smithi15186934-22 2024-09-13T14:23:25.565 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 22 v 128 (ObjNum 271 snap 86 seq_num 271) dirty exists 2024-09-13T14:23:25.565 INFO:tasks.rados.rados.0.smithi151.stdout:630: left oid 22 (ObjNum 271 snap 86 seq_num 271) 2024-09-13T14:23:25.565 INFO:tasks.rados.rados.0.smithi151.stdout:632: finishing write tid 1 to smithi15186934-21 2024-09-13T14:23:25.566 INFO:tasks.rados.rados.0.smithi151.stdout:632: finishing write tid 2 to smithi15186934-21 2024-09-13T14:23:25.566 INFO:tasks.rados.rados.0.smithi151.stdout:632: finishing write tid 3 to smithi15186934-21 2024-09-13T14:23:25.566 INFO:tasks.rados.rados.0.smithi151.stdout:632: finishing write tid 4 to smithi15186934-21 2024-09-13T14:23:25.566 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 21 v 134 (ObjNum 272 snap 86 seq_num 272) dirty exists 2024-09-13T14:23:25.566 INFO:tasks.rados.rados.0.smithi151.stdout:632: left oid 21 (ObjNum 272 snap 86 seq_num 272) 2024-09-13T14:23:25.854 INFO:tasks.rados.rados.0.smithi151.stdout:623: done (6 left) 2024-09-13T14:23:25.854 INFO:tasks.rados.rados.0.smithi151.stdout:624: done (5 left) 2024-09-13T14:23:25.854 INFO:tasks.rados.rados.0.smithi151.stdout:627: done (4 left) 2024-09-13T14:23:25.854 INFO:tasks.rados.rados.0.smithi151.stdout:629: done (3 left) 2024-09-13T14:23:25.854 INFO:tasks.rados.rados.0.smithi151.stdout:630: done (2 left) 2024-09-13T14:23:25.854 INFO:tasks.rados.rados.0.smithi151.stdout:632: done (1 left) 2024-09-13T14:23:25.854 INFO:tasks.rados.rados.0.smithi151.stdout:634: done (0 left) 2024-09-13T14:23:25.854 INFO:tasks.rados.rados.0.smithi151.stdout:635: rollback oid 19 current snap is 87 2024-09-13T14:23:25.854 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 19 to 83 2024-09-13T14:23:25.854 INFO:tasks.rados.rados.0.smithi151.stdout:636: rollback oid 6 current snap is 87 2024-09-13T14:23:25.854 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 6 to 84 2024-09-13T14:23:25.854 INFO:tasks.rados.rados.0.smithi151.stdout:637: snap_remove snap 43 2024-09-13T14:23:26.043 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:25 smithi184 ceph-mon[33394]: pgmap v9: 89 pgs: 89 active+clean; 239 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:23:26.043 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:25 smithi184 ceph-mon[33394]: osdmap e225: 8 total, 8 up, 8 in 2024-09-13T14:23:26.043 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:25 smithi184 ceph-mon[33394]: mgrmap e30: y(active, since 4s) 2024-09-13T14:23:26.043 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:25 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/954386687' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:26.043 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:25 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:26.043 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:25 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:26.043 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:25 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:23:25.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:23:26.044 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:25 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:23:25.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:23:26.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:25 smithi151 ceph-mon[29807]: pgmap v9: 89 pgs: 89 active+clean; 239 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:23:26.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:25 smithi151 ceph-mon[29807]: osdmap e225: 8 total, 8 up, 8 in 2024-09-13T14:23:26.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:25 smithi151 ceph-mon[29807]: mgrmap e30: y(active, since 4s) 2024-09-13T14:23:26.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:25 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/954386687' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:26.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:25 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:26.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:25 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:26.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:25 smithi151 ceph-mon[41387]: pgmap v9: 89 pgs: 89 active+clean; 239 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:23:26.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:25 smithi151 ceph-mon[41387]: osdmap e225: 8 total, 8 up, 8 in 2024-09-13T14:23:26.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:25 smithi151 ceph-mon[41387]: mgrmap e30: y(active, since 4s) 2024-09-13T14:23:26.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:25 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/954386687' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:26.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:25 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:26.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:25 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:26.353 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:26 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: [13/Sep/2024:14:23:26] ENGINE Bus STARTING 2024-09-13T14:23:26.353 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:26 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: CherryPy Checker: 2024-09-13T14:23:26.354 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:26 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: The Application mounted at '' has an empty config. 2024-09-13T14:23:26.354 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:26 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: 2024-09-13T14:23:26.354 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:26 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: [13/Sep/2024:14:23:26] ENGINE Serving on http://:::9283 2024-09-13T14:23:26.354 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:26 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: [13/Sep/2024:14:23:26] ENGINE Bus STARTED 2024-09-13T14:23:26.862 INFO:tasks.rados.rados.0.smithi151.stdout:637: done (2 left) 2024-09-13T14:23:26.862 INFO:tasks.rados.rados.0.smithi151.stdout:638: read oid 26 snap -1 2024-09-13T14:23:26.862 INFO:tasks.rados.rados.0.smithi151.stdout:638: expect (ObjNum 148 snap 34 seq_num 148) 2024-09-13T14:23:26.862 INFO:tasks.rados.rados.0.smithi151.stdout:639: rollback oid 14 current snap is 87 2024-09-13T14:23:26.862 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 14 to 82 2024-09-13T14:23:26.863 INFO:tasks.rados.rados.0.smithi151.stdout:635: finishing rollback tid 1 to smithi15186934-19 2024-09-13T14:23:26.863 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 19 v 141 (ObjNum 209 snap 0 seq_num 3681443411) dirty dne 2024-09-13T14:23:26.863 INFO:tasks.rados.rados.0.smithi151.stdout:635: done (3 left) 2024-09-13T14:23:26.863 INFO:tasks.rados.rados.0.smithi151.stdout:640: snap_create 2024-09-13T14:23:26.863 INFO:tasks.rados.rados.0.smithi151.stdout:636: finishing rollback tid 0 to smithi15186934-6 2024-09-13T14:23:26.863 INFO:tasks.rados.rados.0.smithi151.stdout:636: finishing rollback tid 1 to smithi15186934-6 2024-09-13T14:23:26.863 INFO:tasks.rados.rados.0.smithi151.stdout:636: finishing rollback tid 2 to smithi15186934-6 2024-09-13T14:23:26.863 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 6 v 162 (ObjNum 147 snap 34 seq_num 147) dirty exists 2024-09-13T14:23:26.865 INFO:tasks.rados.rados.0.smithi151.stdout:639: finishing rollback tid 1 to smithi15186934-14 2024-09-13T14:23:26.865 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 217 (ObjNum 129 snap 0 seq_num 3680701429) dirty dne 2024-09-13T14:23:26.871 INFO:tasks.rados.rados.0.smithi151.stdout:638: expect (ObjNum 148 snap 34 seq_num 148) 2024-09-13T14:23:26.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:26 smithi151 ceph-mon[29807]: from='client.15843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:26.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:26 smithi151 ceph-mon[29807]: from='client.15849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:26.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:26 smithi151 ceph-mon[29807]: osdmap e226: 8 total, 8 up, 8 in 2024-09-13T14:23:26.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:26 smithi151 ceph-mon[29807]: Standby manager daemon x started 2024-09-13T14:23:26.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:26 smithi151 ceph-mon[29807]: from='mgr.? 172.21.15.184:0/1042712949' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:23:26.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:26 smithi151 ceph-mon[29807]: from='mgr.? 172.21.15.184:0/1042712949' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:23:26.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:26 smithi151 ceph-mon[29807]: from='mgr.? 172.21.15.184:0/1042712949' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:23:26.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:26 smithi151 ceph-mon[29807]: from='mgr.? 172.21.15.184:0/1042712949' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:23:26.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:26 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:26.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:26 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:26.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:26 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:26.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:26 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:26.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:26 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:26.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:26 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:26.941 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:26 smithi151 ceph-mon[41387]: from='client.15843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:26.942 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:26 smithi151 ceph-mon[41387]: from='client.15849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:26.942 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:26 smithi151 ceph-mon[41387]: osdmap e226: 8 total, 8 up, 8 in 2024-09-13T14:23:26.942 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:26 smithi151 ceph-mon[41387]: Standby manager daemon x started 2024-09-13T14:23:26.942 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:26 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.184:0/1042712949' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:23:26.942 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:26 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.184:0/1042712949' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:23:26.942 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:26 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.184:0/1042712949' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:23:26.942 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:26 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.184:0/1042712949' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:23:26.942 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:26 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:26.942 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:26 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:26.942 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:26 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:26.942 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:26 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:26.942 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:26 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:26.942 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:26 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:27.103 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:26 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:23:26.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi184\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:23:27.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:26 smithi184 ceph-mon[33394]: from='client.15843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:27.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:26 smithi184 ceph-mon[33394]: from='client.15849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:27.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:26 smithi184 ceph-mon[33394]: osdmap e226: 8 total, 8 up, 8 in 2024-09-13T14:23:27.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:26 smithi184 ceph-mon[33394]: Standby manager daemon x started 2024-09-13T14:23:27.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:26 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.184:0/1042712949' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:23:27.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:26 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.184:0/1042712949' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:23:27.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:26 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.184:0/1042712949' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:23:27.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:26 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.184:0/1042712949' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:23:27.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:26 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:27.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:26 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:27.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:26 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:27.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:26 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:27.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:26 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:27.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:26 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:27.861 INFO:tasks.rados.rados.0.smithi151.stdout:636: done (3 left) 2024-09-13T14:23:27.861 INFO:tasks.rados.rados.0.smithi151.stdout:638: done (2 left) 2024-09-13T14:23:27.861 INFO:tasks.rados.rados.0.smithi151.stdout:639: done (1 left) 2024-09-13T14:23:27.861 INFO:tasks.rados.rados.0.smithi151.stdout:640: done (0 left) 2024-09-13T14:23:27.861 INFO:tasks.rados.rados.0.smithi151.stdout:641: read oid 18 snap -1 2024-09-13T14:23:27.861 INFO:tasks.rados.rados.0.smithi151.stdout:641: expect (ObjNum 248 snap 76 seq_num 248) 2024-09-13T14:23:27.861 INFO:tasks.rados.rados.0.smithi151.stdout:642: snap_remove snap 87 2024-09-13T14:23:28.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:27 smithi151 ceph-mon[29807]: pgmap v12: 89 pgs: 89 active+clean; 239 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:23:28.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:27 smithi151 ceph-mon[29807]: osdmap e227: 8 total, 8 up, 8 in 2024-09-13T14:23:28.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:27 smithi151 ceph-mon[29807]: mgrmap e31: y(active, since 6s), standbys: x 2024-09-13T14:23:28.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:27 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:23:28.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:27 smithi151 ceph-mon[41387]: pgmap v12: 89 pgs: 89 active+clean; 239 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:23:28.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:27 smithi151 ceph-mon[41387]: osdmap e227: 8 total, 8 up, 8 in 2024-09-13T14:23:28.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:27 smithi151 ceph-mon[41387]: mgrmap e31: y(active, since 6s), standbys: x 2024-09-13T14:23:28.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:27 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:23:28.357 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:27 smithi184 ceph-mon[33394]: pgmap v12: 89 pgs: 89 active+clean; 239 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:23:28.357 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:27 smithi184 ceph-mon[33394]: osdmap e227: 8 total, 8 up, 8 in 2024-09-13T14:23:28.357 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:27 smithi184 ceph-mon[33394]: mgrmap e31: y(active, since 6s), standbys: x 2024-09-13T14:23:28.357 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:27 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:23:28.470 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:23:28.864 INFO:tasks.rados.rados.0.smithi151.stdout:642: done (1 left) 2024-09-13T14:23:28.864 INFO:tasks.rados.rados.0.smithi151.stdout:643: read oid 13 snap -1 2024-09-13T14:23:28.864 INFO:tasks.rados.rados.0.smithi151.stdout:643: expect (ObjNum 270 snap 86 seq_num 270) 2024-09-13T14:23:28.864 INFO:tasks.rados.rados.0.smithi151.stdout:644: delete oid 39 current snap is 88 2024-09-13T14:23:28.864 INFO:tasks.rados.rados.0.smithi151.stdout:641: expect (ObjNum 248 snap 76 seq_num 248) 2024-09-13T14:23:28.939 INFO:tasks.rados.rados.0.smithi151.stdout:643: expect (ObjNum 270 snap 86 seq_num 270) 2024-09-13T14:23:29.081 INFO:tasks.rados.rados.0.smithi151.stdout:641: done (2 left) 2024-09-13T14:23:29.082 INFO:tasks.rados.rados.0.smithi151.stdout:643: done (1 left) 2024-09-13T14:23:29.082 INFO:tasks.rados.rados.0.smithi151.stdout:644: done (0 left) 2024-09-13T14:23:29.082 INFO:tasks.rados.rados.0.smithi151.stdout:645: read oid 40 snap -1 2024-09-13T14:23:29.082 INFO:tasks.rados.rados.0.smithi151.stdout:645: expect (ObjNum 238 snap 75 seq_num 238) 2024-09-13T14:23:29.082 INFO:tasks.rados.rados.0.smithi151.stdout:646: read oid 11 snap -1 2024-09-13T14:23:29.082 INFO:tasks.rados.rados.0.smithi151.stdout:646: expect deleted 2024-09-13T14:23:29.082 INFO:tasks.rados.rados.0.smithi151.stdout:647: snap_remove snap 83 2024-09-13T14:23:29.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:28 smithi151 ceph-mon[41387]: osdmap e228: 8 total, 8 up, 8 in 2024-09-13T14:23:29.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:28 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/915144650' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:29.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:28 smithi151 ceph-mon[29807]: osdmap e228: 8 total, 8 up, 8 in 2024-09-13T14:23:29.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:28 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/915144650' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:29.322 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:28 smithi184 ceph-mon[33394]: osdmap e228: 8 total, 8 up, 8 in 2024-09-13T14:23:29.322 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:28 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/915144650' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:29.603 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:29 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[69225]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:23:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.51.0" 2024-09-13T14:23:29.889 INFO:tasks.rados.rados.0.smithi151.stdout:647: done (2 left) 2024-09-13T14:23:29.890 INFO:tasks.rados.rados.0.smithi151.stdout:648: read oid 18 snap -1 2024-09-13T14:23:29.890 INFO:tasks.rados.rados.0.smithi151.stdout:648: expect (ObjNum 248 snap 76 seq_num 248) 2024-09-13T14:23:29.890 INFO:tasks.rados.rados.0.smithi151.stdout:649: read oid 25 snap -1 2024-09-13T14:23:29.890 INFO:tasks.rados.rados.0.smithi151.stdout:649: expect deleted 2024-09-13T14:23:29.890 INFO:tasks.rados.rados.0.smithi151.stdout:650: delete oid 14 current snap is 88 2024-09-13T14:23:29.890 INFO:tasks.rados.rados.0.smithi151.stdout:645: expect (ObjNum 238 snap 75 seq_num 238) 2024-09-13T14:23:29.958 INFO:tasks.rados.rados.0.smithi151.stdout:648: expect (ObjNum 248 snap 76 seq_num 248) 2024-09-13T14:23:30.032 INFO:tasks.rados.rados.0.smithi151.stdout:645: done (4 left) 2024-09-13T14:23:30.032 INFO:tasks.rados.rados.0.smithi151.stdout:646: done (3 left) 2024-09-13T14:23:30.032 INFO:tasks.rados.rados.0.smithi151.stdout:648: done (2 left) 2024-09-13T14:23:30.032 INFO:tasks.rados.rados.0.smithi151.stdout:649: done (1 left) 2024-09-13T14:23:30.032 INFO:tasks.rados.rados.0.smithi151.stdout:650: done (0 left) 2024-09-13T14:23:30.032 INFO:tasks.rados.rados.0.smithi151.stdout:651: snap_remove snap 57 2024-09-13T14:23:30.212 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:29 smithi184 ceph-mon[33394]: from='client.15861 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:30.212 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:29 smithi184 ceph-mon[33394]: pgmap v15: 89 pgs: 89 active+clean; 232 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.5 MiB/s rd, 554 KiB/s wr, 23 op/s 2024-09-13T14:23:30.212 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:29 smithi184 ceph-mon[33394]: from='client.25772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:30.212 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:29 smithi184 ceph-mon[33394]: osdmap e229: 8 total, 8 up, 8 in 2024-09-13T14:23:30.213 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:29 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:30.213 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:29 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:30.213 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:29 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:30.213 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:29 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:23:30.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:29 smithi151 ceph-mon[41387]: from='client.15861 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:30.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:29 smithi151 ceph-mon[41387]: pgmap v15: 89 pgs: 89 active+clean; 232 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.5 MiB/s rd, 554 KiB/s wr, 23 op/s 2024-09-13T14:23:30.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:29 smithi151 ceph-mon[41387]: from='client.25772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:30.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:29 smithi151 ceph-mon[41387]: osdmap e229: 8 total, 8 up, 8 in 2024-09-13T14:23:30.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:29 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:30.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:29 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:30.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:29 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:30.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:29 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:23:30.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:29 smithi151 ceph-mon[29807]: from='client.15861 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:30.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:29 smithi151 ceph-mon[29807]: pgmap v15: 89 pgs: 89 active+clean; 232 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.5 MiB/s rd, 554 KiB/s wr, 23 op/s 2024-09-13T14:23:30.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:29 smithi151 ceph-mon[29807]: from='client.25772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:30.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:29 smithi151 ceph-mon[29807]: osdmap e229: 8 total, 8 up, 8 in 2024-09-13T14:23:30.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:29 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:30.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:29 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:30.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:29 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:30.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:29 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:23:30.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:23:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:23:30.896 INFO:tasks.rados.rados.0.smithi151.stdout:651: done (0 left) 2024-09-13T14:23:30.897 INFO:tasks.rados.rados.0.smithi151.stdout:652: write oid 29 current snap is 88 2024-09-13T14:23:30.897 INFO:tasks.rados.rados.0.smithi151.stdout:652: seq_num 276 ranges {638732=557432,1617736=710086} 2024-09-13T14:23:30.904 INFO:tasks.rados.rados.0.smithi151.stdout:652: writing smithi15186934-29 from 638732 to 1196164 tid 1 2024-09-13T14:23:30.913 INFO:tasks.rados.rados.0.smithi151.stdout:652: writing smithi15186934-29 from 1617736 to 2327822 tid 2 2024-09-13T14:23:30.913 INFO:tasks.rados.rados.0.smithi151.stdout:653: delete oid 28 current snap is 88 2024-09-13T14:23:30.913 INFO:tasks.rados.rados.0.smithi151.stdout:652: finishing write tid 1 to smithi15186934-29 2024-09-13T14:23:30.915 INFO:tasks.rados.rados.0.smithi151.stdout:653: done (1 left) 2024-09-13T14:23:30.915 INFO:tasks.rados.rados.0.smithi151.stdout:654: delete oid 7 current snap is 88 2024-09-13T14:23:30.916 INFO:tasks.rados.rados.0.smithi151.stdout:652: finishing write tid 2 to smithi15186934-29 2024-09-13T14:23:30.916 INFO:tasks.rados.rados.0.smithi151.stdout:652: finishing write tid 3 to smithi15186934-29 2024-09-13T14:23:30.916 INFO:tasks.rados.rados.0.smithi151.stdout:652: finishing write tid 4 to smithi15186934-29 2024-09-13T14:23:30.916 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 29 v 188 (ObjNum 276 snap 88 seq_num 276) dirty exists 2024-09-13T14:23:30.916 INFO:tasks.rados.rados.0.smithi151.stdout:652: left oid 29 (ObjNum 276 snap 88 seq_num 276) 2024-09-13T14:23:30.917 INFO:tasks.rados.rados.0.smithi151.stdout:652: done (1 left) 2024-09-13T14:23:30.917 INFO:tasks.rados.rados.0.smithi151.stdout:654: done (0 left) 2024-09-13T14:23:30.917 INFO:tasks.rados.rados.0.smithi151.stdout:655: read oid 4 snap -1 2024-09-13T14:23:30.917 INFO:tasks.rados.rados.0.smithi151.stdout:655: expect deleted 2024-09-13T14:23:30.917 INFO:tasks.rados.rados.0.smithi151.stdout:656: rollback oid 16 current snap is 88 2024-09-13T14:23:30.918 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 16 to 86 2024-09-13T14:23:30.918 INFO:tasks.rados.rados.0.smithi151.stdout:657: write oid 45 current snap is 88 2024-09-13T14:23:30.918 INFO:tasks.rados.rados.0.smithi151.stdout:657: seq_num 279 ranges {689153=783179,2076829=457413} 2024-09-13T14:23:30.927 INFO:tasks.rados.rados.0.smithi151.stdout:657: writing smithi15186934-45 from 689153 to 1472332 tid 1 2024-09-13T14:23:30.934 INFO:tasks.rados.rados.0.smithi151.stdout:657: writing smithi15186934-45 from 2076829 to 2534242 tid 2 2024-09-13T14:23:30.935 INFO:tasks.rados.rados.0.smithi151.stdout:658: delete oid 11 current snap is 88 2024-09-13T14:23:30.935 INFO:tasks.rados.rados.0.smithi151.stdout:656: finishing rollback tid 1 to smithi15186934-16 2024-09-13T14:23:30.935 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 16 v 103 (ObjNum 305 snap 0 seq_num 3092670960) dirty dne 2024-09-13T14:23:30.935 INFO:tasks.rados.rados.0.smithi151.stdout:657: finishing write tid 1 to smithi15186934-45 2024-09-13T14:23:30.936 INFO:tasks.rados.rados.0.smithi151.stdout:655: done (3 left) 2024-09-13T14:23:30.936 INFO:tasks.rados.rados.0.smithi151.stdout:656: done (2 left) 2024-09-13T14:23:30.936 INFO:tasks.rados.rados.0.smithi151.stdout:658: done (1 left) 2024-09-13T14:23:30.936 INFO:tasks.rados.rados.0.smithi151.stdout:659: read oid 41 snap -1 2024-09-13T14:23:30.936 INFO:tasks.rados.rados.0.smithi151.stdout:659: expect deleted 2024-09-13T14:23:30.936 INFO:tasks.rados.rados.0.smithi151.stdout:660: read oid 8 snap 63 2024-09-13T14:23:30.936 INFO:tasks.rados.rados.0.smithi151.stdout:660: expect (ObjNum 207 snap 60 seq_num 207) 2024-09-13T14:23:30.936 INFO:tasks.rados.rados.0.smithi151.stdout:661: rollback oid 10 current snap is 88 2024-09-13T14:23:30.936 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 10 to 78 2024-09-13T14:23:30.936 INFO:tasks.rados.rados.0.smithi151.stdout:662: snap_remove snap 58 2024-09-13T14:23:31.168 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:30 smithi184 ceph-mon[33394]: Updating smithi151:/etc/ceph/ceph.conf 2024-09-13T14:23:31.168 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:30 smithi184 ceph-mon[33394]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-13T14:23:31.168 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:30 smithi184 ceph-mon[33394]: Updating smithi151:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:23:31.168 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:30 smithi184 ceph-mon[33394]: Updating smithi184:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:23:31.168 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:30 smithi184 ceph-mon[33394]: osdmap e230: 8 total, 8 up, 8 in 2024-09-13T14:23:31.168 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:30 smithi184 ceph-mon[33394]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:23:31.169 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:30 smithi184 ceph-mon[33394]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:23:31.169 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:30 smithi184 ceph-mon[33394]: Updating smithi151:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:23:31.169 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:30 smithi184 ceph-mon[33394]: Updating smithi184:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:23:31.169 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:30 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:31.169 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:30 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:31.169 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:30 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:31.169 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:30 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:31.169 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:30 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:31.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:30 smithi151 ceph-mon[29807]: Updating smithi151:/etc/ceph/ceph.conf 2024-09-13T14:23:31.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:30 smithi151 ceph-mon[29807]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-13T14:23:31.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:30 smithi151 ceph-mon[29807]: Updating smithi151:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:23:31.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:30 smithi151 ceph-mon[29807]: Updating smithi184:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:23:31.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:30 smithi151 ceph-mon[29807]: osdmap e230: 8 total, 8 up, 8 in 2024-09-13T14:23:31.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:30 smithi151 ceph-mon[29807]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:23:31.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:30 smithi151 ceph-mon[29807]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:23:31.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:30 smithi151 ceph-mon[29807]: Updating smithi151:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:23:31.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:30 smithi151 ceph-mon[29807]: Updating smithi184:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:23:31.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:30 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:31.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:30 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:31.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:30 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:31.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:30 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:31.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:30 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:31.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:30 smithi151 ceph-mon[41387]: Updating smithi151:/etc/ceph/ceph.conf 2024-09-13T14:23:31.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:30 smithi151 ceph-mon[41387]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-13T14:23:31.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:30 smithi151 ceph-mon[41387]: Updating smithi151:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:23:31.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:30 smithi151 ceph-mon[41387]: Updating smithi184:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:23:31.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:30 smithi151 ceph-mon[41387]: osdmap e230: 8 total, 8 up, 8 in 2024-09-13T14:23:31.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:30 smithi151 ceph-mon[41387]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:23:31.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:30 smithi151 ceph-mon[41387]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:23:31.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:30 smithi151 ceph-mon[41387]: Updating smithi151:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:23:31.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:30 smithi151 ceph-mon[41387]: Updating smithi184:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:23:31.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:30 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:31.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:30 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:31.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:30 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:31.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:30 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:31.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:30 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:31.436 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:23:31.904 INFO:tasks.rados.rados.0.smithi151.stdout:662: done (4 left) 2024-09-13T14:23:31.905 INFO:tasks.rados.rados.0.smithi151.stdout:663: read oid 11 snap -1 2024-09-13T14:23:31.905 INFO:tasks.rados.rados.0.smithi151.stdout:663: expect deleted 2024-09-13T14:23:31.905 INFO:tasks.rados.rados.0.smithi151.stdout:664: snap_remove snap 75 2024-09-13T14:23:31.905 INFO:tasks.rados.rados.0.smithi151.stdout:661: finishing rollback tid 1 to smithi15186934-10 2024-09-13T14:23:31.905 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 163 (ObjNum 129 snap 0 seq_num 3680556972) dirty dne 2024-09-13T14:23:31.905 INFO:tasks.rados.rados.0.smithi151.stdout:657: finishing write tid 2 to smithi15186934-45 2024-09-13T14:23:31.905 INFO:tasks.rados.rados.0.smithi151.stdout:657: finishing write tid 3 to smithi15186934-45 2024-09-13T14:23:31.905 INFO:tasks.rados.rados.0.smithi151.stdout:657: finishing write tid 4 to smithi15186934-45 2024-09-13T14:23:31.905 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 394 (ObjNum 279 snap 88 seq_num 279) dirty exists 2024-09-13T14:23:31.905 INFO:tasks.rados.rados.0.smithi151.stdout:657: left oid 45 (ObjNum 279 snap 88 seq_num 279) 2024-09-13T14:23:31.905 INFO:tasks.rados.rados.0.smithi151.stdout:660: expect (ObjNum 207 snap 60 seq_num 207) 2024-09-13T14:23:32.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:31 smithi184 ceph-mon[33394]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:23:32.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:31 smithi184 ceph-mon[33394]: Reconfiguring daemon prometheus.a on smithi184 2024-09-13T14:23:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:31 smithi184 ceph-mon[33394]: pgmap v18: 89 pgs: 89 active+clean; 232 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.5 MiB/s rd, 554 KiB/s wr, 23 op/s 2024-09-13T14:23:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:31 smithi184 ceph-mon[33394]: osdmap e231: 8 total, 8 up, 8 in 2024-09-13T14:23:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:31 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/832521422' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:31 smithi184 ceph-mon[33394]: from='client.25784 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:32.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:31 smithi151 ceph-mon[29807]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:23:32.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:31 smithi151 ceph-mon[29807]: Reconfiguring daemon prometheus.a on smithi184 2024-09-13T14:23:32.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:31 smithi151 ceph-mon[29807]: pgmap v18: 89 pgs: 89 active+clean; 232 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.5 MiB/s rd, 554 KiB/s wr, 23 op/s 2024-09-13T14:23:32.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:31 smithi151 ceph-mon[29807]: osdmap e231: 8 total, 8 up, 8 in 2024-09-13T14:23:32.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:31 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/832521422' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:32.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:31 smithi151 ceph-mon[29807]: from='client.25784 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:32.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:31 smithi151 ceph-mon[41387]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:23:32.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:31 smithi151 ceph-mon[41387]: Reconfiguring daemon prometheus.a on smithi184 2024-09-13T14:23:32.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:31 smithi151 ceph-mon[41387]: pgmap v18: 89 pgs: 89 active+clean; 232 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.5 MiB/s rd, 554 KiB/s wr, 23 op/s 2024-09-13T14:23:32.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:31 smithi151 ceph-mon[41387]: osdmap e231: 8 total, 8 up, 8 in 2024-09-13T14:23:32.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:31 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/832521422' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:32.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:31 smithi151 ceph-mon[41387]: from='client.25784 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:32.811 INFO:tasks.rados.rados.0.smithi151.stdout:657: done (5 left) 2024-09-13T14:23:32.812 INFO:tasks.rados.rados.0.smithi151.stdout:659: done (4 left) 2024-09-13T14:23:32.812 INFO:tasks.rados.rados.0.smithi151.stdout:660: done (3 left) 2024-09-13T14:23:32.812 INFO:tasks.rados.rados.0.smithi151.stdout:661: done (2 left) 2024-09-13T14:23:32.812 INFO:tasks.rados.rados.0.smithi151.stdout:663: done (1 left) 2024-09-13T14:23:32.812 INFO:tasks.rados.rados.0.smithi151.stdout:664: done (0 left) 2024-09-13T14:23:32.813 INFO:tasks.rados.rados.0.smithi151.stdout:665: delete oid 28 current snap is 88 2024-09-13T14:23:32.816 INFO:tasks.rados.rados.0.smithi151.stdout:665: done (0 left) 2024-09-13T14:23:32.817 INFO:tasks.rados.rados.0.smithi151.stdout:666: rollback oid 37 current snap is 88 2024-09-13T14:23:32.817 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 37 to 66 2024-09-13T14:23:32.817 INFO:tasks.rados.rados.0.smithi151.stdout:667: rollback oid 21 current snap is 88 2024-09-13T14:23:32.817 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 21 to 66 2024-09-13T14:23:32.817 INFO:tasks.rados.rados.0.smithi151.stdout:668: write oid 40 current snap is 88 2024-09-13T14:23:32.817 INFO:tasks.rados.rados.0.smithi151.stdout:668: seq_num 282 ranges {739574=725279,1968628=721093} 2024-09-13T14:23:32.825 INFO:tasks.rados.rados.0.smithi151.stdout:668: writing smithi15186934-40 from 739574 to 1464853 tid 1 2024-09-13T14:23:32.832 INFO:tasks.rados.rados.0.smithi151.stdout:668: writing smithi15186934-40 from 1968628 to 2689721 tid 2 2024-09-13T14:23:32.832 INFO:tasks.rados.rados.0.smithi151.stdout:669: write oid 42 current snap is 88 2024-09-13T14:23:32.832 INFO:tasks.rados.rados.0.smithi151.stdout:667: finishing rollback tid 0 to smithi15186934-21 2024-09-13T14:23:32.832 INFO:tasks.rados.rados.0.smithi151.stdout:669: seq_num 283 ranges {756381=400528,1910757=464751} 2024-09-13T14:23:32.838 INFO:tasks.rados.rados.0.smithi151.stdout:669: writing smithi15186934-42 from 756381 to 1156909 tid 1 2024-09-13T14:23:32.845 INFO:tasks.rados.rados.0.smithi151.stdout:669: writing smithi15186934-42 from 1910757 to 2375508 tid 2 2024-09-13T14:23:32.845 INFO:tasks.rados.rados.0.smithi151.stdout:670: snap_create 2024-09-13T14:23:32.845 INFO:tasks.rados.rados.0.smithi151.stdout:666: finishing rollback tid 0 to smithi15186934-37 2024-09-13T14:23:32.845 INFO:tasks.rados.rados.0.smithi151.stdout:667: finishing rollback tid 1 to smithi15186934-21 2024-09-13T14:23:32.845 INFO:tasks.rados.rados.0.smithi151.stdout:666: finishing rollback tid 1 to smithi15186934-37 2024-09-13T14:23:32.845 INFO:tasks.rados.rados.0.smithi151.stdout:667: finishing rollback tid 2 to smithi15186934-21 2024-09-13T14:23:32.845 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 21 v 146 (ObjNum 223 snap 64 seq_num 223) dirty exists 2024-09-13T14:23:32.846 INFO:tasks.rados.rados.0.smithi151.stdout:666: finishing rollback tid 2 to smithi15186934-37 2024-09-13T14:23:32.846 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 37 v 331 (ObjNum 179 snap 47 seq_num 179) dirty exists 2024-09-13T14:23:32.846 INFO:tasks.rados.rados.0.smithi151.stdout:668: finishing write tid 1 to smithi15186934-40 2024-09-13T14:23:32.846 INFO:tasks.rados.rados.0.smithi151.stdout:668: finishing write tid 2 to smithi15186934-40 2024-09-13T14:23:32.846 INFO:tasks.rados.rados.0.smithi151.stdout:668: finishing write tid 3 to smithi15186934-40 2024-09-13T14:23:32.846 INFO:tasks.rados.rados.0.smithi151.stdout:668: finishing write tid 4 to smithi15186934-40 2024-09-13T14:23:32.846 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 40 v 279 (ObjNum 282 snap 88 seq_num 282) dirty exists 2024-09-13T14:23:32.846 INFO:tasks.rados.rados.0.smithi151.stdout:668: left oid 40 (ObjNum 282 snap 88 seq_num 282) 2024-09-13T14:23:32.846 INFO:tasks.rados.rados.0.smithi151.stdout:669: finishing write tid 1 to smithi15186934-42 2024-09-13T14:23:32.847 INFO:tasks.rados.rados.0.smithi151.stdout:669: finishing write tid 2 to smithi15186934-42 2024-09-13T14:23:32.847 INFO:tasks.rados.rados.0.smithi151.stdout:669: finishing write tid 3 to smithi15186934-42 2024-09-13T14:23:32.848 INFO:tasks.rados.rados.0.smithi151.stdout:669: finishing write tid 4 to smithi15186934-42 2024-09-13T14:23:32.848 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 306 (ObjNum 283 snap 88 seq_num 283) dirty exists 2024-09-13T14:23:32.848 INFO:tasks.rados.rados.0.smithi151.stdout:669: left oid 42 (ObjNum 283 snap 88 seq_num 283) 2024-09-13T14:23:33.188 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:32 smithi184 ceph-mon[33394]: from='client.15885 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:33.188 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:32 smithi184 ceph-mon[33394]: osdmap e232: 8 total, 8 up, 8 in 2024-09-13T14:23:33.188 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:32 smithi184 ceph-mon[33394]: osdmap e233: 8 total, 8 up, 8 in 2024-09-13T14:23:33.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:32 smithi151 ceph-mon[29807]: from='client.15885 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:33.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:32 smithi151 ceph-mon[29807]: osdmap e232: 8 total, 8 up, 8 in 2024-09-13T14:23:33.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:32 smithi151 ceph-mon[29807]: osdmap e233: 8 total, 8 up, 8 in 2024-09-13T14:23:33.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:32 smithi151 ceph-mon[41387]: from='client.15885 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:33.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:32 smithi151 ceph-mon[41387]: osdmap e232: 8 total, 8 up, 8 in 2024-09-13T14:23:33.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:32 smithi151 ceph-mon[41387]: osdmap e233: 8 total, 8 up, 8 in 2024-09-13T14:23:33.813 INFO:tasks.rados.rados.0.smithi151.stdout:666: done (4 left) 2024-09-13T14:23:33.813 INFO:tasks.rados.rados.0.smithi151.stdout:667: done (3 left) 2024-09-13T14:23:33.813 INFO:tasks.rados.rados.0.smithi151.stdout:668: done (2 left) 2024-09-13T14:23:33.813 INFO:tasks.rados.rados.0.smithi151.stdout:669: done (1 left) 2024-09-13T14:23:33.814 INFO:tasks.rados.rados.0.smithi151.stdout:670: done (0 left) 2024-09-13T14:23:33.814 INFO:tasks.rados.rados.0.smithi151.stdout:671: rollback oid 28 current snap is 89 2024-09-13T14:23:33.814 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 28 to 84 2024-09-13T14:23:33.814 INFO:tasks.rados.rados.0.smithi151.stdout:672: snap_remove snap 79 2024-09-13T14:23:34.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:33 smithi184 ceph-mon[33394]: pgmap v21: 89 pgs: 89 active+clean; 230 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 613 KiB/s wr, 16 op/s 2024-09-13T14:23:34.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:33 smithi184 ceph-mon[33394]: osdmap e234: 8 total, 8 up, 8 in 2024-09-13T14:23:34.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:33 smithi184 systemd[1]: Stopping Ceph prometheus.a for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:23:34.341 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:23:34.341 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:33 smithi151 ceph-mon[29807]: pgmap v21: 89 pgs: 89 active+clean; 230 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 613 KiB/s wr, 16 op/s 2024-09-13T14:23:34.341 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:33 smithi151 ceph-mon[29807]: osdmap e234: 8 total, 8 up, 8 in 2024-09-13T14:23:34.342 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:33 smithi151 ceph-mon[41387]: pgmap v21: 89 pgs: 89 active+clean; 230 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 613 KiB/s wr, 16 op/s 2024-09-13T14:23:34.342 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:33 smithi151 ceph-mon[41387]: osdmap e234: 8 total, 8 up, 8 in 2024-09-13T14:23:34.497 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:34 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:23:34.230Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-13T14:23:34.497 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:34 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:23:34.231Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-13T14:23:34.497 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:34 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:23:34.231Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-13T14:23:34.497 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:34 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:23:34.231Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-13T14:23:34.497 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:34 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:23:34.231Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-13T14:23:34.497 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:34 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:23:34.231Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-13T14:23:34.498 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:34 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:23:34.231Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-13T14:23:34.498 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:34 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:23:34.231Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-13T14:23:34.498 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:34 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:23:34.231Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-13T14:23:34.498 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:34 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:23:34.232Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-13T14:23:34.498 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:34 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:23:34.232Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-13T14:23:34.498 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:34 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[77940]: ts=2024-09-13T14:23:34.232Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-13T14:23:34.498 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:34 smithi184 podman[81362]: 2024-09-13 14:23:34.241387511 +0000 UTC m=+0.132794932 container died 35d43c151982cfbfdafb8f16405bd62ec2caab8dd8e8fd25c4ad26ce2195188e (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:23:34.816 INFO:tasks.rados.rados.0.smithi151.stdout:672: done (1 left) 2024-09-13T14:23:34.816 INFO:tasks.rados.rados.0.smithi151.stdout:673: delete oid 29 current snap is 89 2024-09-13T14:23:34.817 INFO:tasks.rados.rados.0.smithi151.stdout:671: finishing rollback tid 1 to smithi15186934-28 2024-09-13T14:23:34.817 INFO:tasks.rados.rados.0.smithi151.stdout:671: finishing rollback tid 2 to smithi15186934-28 2024-09-13T14:23:34.817 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 28 v 308 (ObjNum 173 snap 46 seq_num 173) dirty exists 2024-09-13T14:23:34.821 INFO:tasks.rados.rados.0.smithi151.stdout:671: done (1 left) 2024-09-13T14:23:34.821 INFO:tasks.rados.rados.0.smithi151.stdout:673: done (0 left) 2024-09-13T14:23:34.821 INFO:tasks.rados.rados.0.smithi151.stdout:674: read oid 5 snap -1 2024-09-13T14:23:34.821 INFO:tasks.rados.rados.0.smithi151.stdout:674: expect (ObjNum 218 snap 63 seq_num 218) 2024-09-13T14:23:34.821 INFO:tasks.rados.rados.0.smithi151.stdout:675: snap_remove snap 81 2024-09-13T14:23:34.847 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:34 smithi184 podman[81362]: 2024-09-13 14:23:34.662105471 +0000 UTC m=+0.553512890 container cleanup 35d43c151982cfbfdafb8f16405bd62ec2caab8dd8e8fd25c4ad26ce2195188e (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:23:34.847 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:34 smithi184 bash[81362]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a 2024-09-13T14:23:35.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:34 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/1201600619' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:35.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:34 smithi184 ceph-mon[33394]: from='client.15897 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:35.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:34 smithi184 ceph-mon[33394]: osdmap e235: 8 total, 8 up, 8 in 2024-09-13T14:23:35.105 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:34 smithi184 podman[81380]: 2024-09-13 14:23:34.846700668 +0000 UTC m=+0.606222355 container remove 35d43c151982cfbfdafb8f16405bd62ec2caab8dd8e8fd25c4ad26ce2195188e (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:23:35.105 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-13T14:23:35.105 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 systemd[1]: Stopped Ceph prometheus.a for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:23:35.105 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 1.141s CPU time. 2024-09-13T14:23:35.105 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 systemd[1]: Starting Ceph prometheus.a for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:23:35.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:34 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1201600619' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:35.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:34 smithi151 ceph-mon[29807]: from='client.15897 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:35.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:34 smithi151 ceph-mon[29807]: osdmap e235: 8 total, 8 up, 8 in 2024-09-13T14:23:35.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:34 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/1201600619' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:35.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:34 smithi151 ceph-mon[41387]: from='client.15897 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:35.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:34 smithi151 ceph-mon[41387]: osdmap e235: 8 total, 8 up, 8 in 2024-09-13T14:23:35.583 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 podman[81476]: 2024-09-13 14:23:35.200645332 +0000 UTC m=+0.017150682 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-13T14:23:35.583 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 podman[81476]: 2024-09-13 14:23:35.297527823 +0000 UTC m=+0.114033159 container create 3da9265b8f14fc104ad3d01a00c82acbe684385d0380509cde4281b20a65563a (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:23:35.584 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 podman[81476]: 2024-09-13 14:23:35.583423071 +0000 UTC m=+0.399928416 container init 3da9265b8f14fc104ad3d01a00c82acbe684385d0380509cde4281b20a65563a (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:23:35.823 INFO:tasks.rados.rados.0.smithi151.stdout:675: done (1 left) 2024-09-13T14:23:35.823 INFO:tasks.rados.rados.0.smithi151.stdout:676: write oid 24 current snap is 89 2024-09-13T14:23:35.823 INFO:tasks.rados.rados.0.smithi151.stdout:676: seq_num 285 ranges {789995=551026,2027721=468420} 2024-09-13T14:23:35.830 INFO:tasks.rados.rados.0.smithi151.stdout:676: writing smithi15186934-24 from 789995 to 1341021 tid 1 2024-09-13T14:23:35.837 INFO:tasks.rados.rados.0.smithi151.stdout:676: writing smithi15186934-24 from 2027721 to 2496141 tid 2 2024-09-13T14:23:35.837 INFO:tasks.rados.rados.0.smithi151.stdout:677: snap_remove snap 73 2024-09-13T14:23:35.854 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 podman[81476]: 2024-09-13 14:23:35.587978521 +0000 UTC m=+0.404483856 container start 3da9265b8f14fc104ad3d01a00c82acbe684385d0380509cde4281b20a65563a (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:23:35.854 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:35.607Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-13T14:23:35.854 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:35.607Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-13T14:23:35.854 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:35.607Z caller=main.go:623 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi184 (none))" 2024-09-13T14:23:35.854 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:35.607Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-13T14:23:35.854 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:35.607Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-13T14:23:35.854 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:35.609Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-13T14:23:35.855 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:35.609Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-13T14:23:35.855 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:35.611Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-13T14:23:35.855 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:35.611Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-13T14:23:35.855 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:35.613Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-13T14:23:35.855 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:35.613Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.725µs 2024-09-13T14:23:35.855 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:35.613Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-13T14:23:35.855 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:35.613Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-09-13T14:23:35.855 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:35.625Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-09-13T14:23:35.855 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:35.631Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-09-13T14:23:35.855 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:35.631Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-09-13T14:23:35.855 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:35.632Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=25.058µs wal_replay_duration=18.892843ms wbl_replay_duration=102ns total_replay_duration=18.936981ms 2024-09-13T14:23:35.856 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:35.635Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-13T14:23:35.856 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:35.635Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-13T14:23:35.856 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:35.635Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-13T14:23:35.856 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:35.649Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=14.142218ms db_storage=787ns remote_storage=1.216µs web_handler=344ns query_engine=521ns scrape=303.186µs scrape_sd=103.9µs notify=9.682µs notify_sd=7.871µs rules=13.388639ms tracing=5.517µs 2024-09-13T14:23:35.856 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:35.649Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-13T14:23:35.856 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:35.649Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-13T14:23:35.856 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 bash[81476]: 3da9265b8f14fc104ad3d01a00c82acbe684385d0380509cde4281b20a65563a 2024-09-13T14:23:35.856 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:35 smithi184 systemd[1]: Started Ceph prometheus.a for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:23:36.241 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:35 smithi184 ceph-mon[33394]: from='client.25808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:36.241 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:35 smithi184 ceph-mon[33394]: pgmap v24: 89 pgs: 89 active+clean; 230 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 613 KiB/s wr, 16 op/s 2024-09-13T14:23:36.241 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:35 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:36.241 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:35 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:36.241 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:35 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:23:36.241 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:35 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:36.242 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:35 smithi184 ceph-mon[33394]: osdmap e236: 8 total, 8 up, 8 in 2024-09-13T14:23:36.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:35 smithi151 ceph-mon[41387]: from='client.25808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:36.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:35 smithi151 ceph-mon[41387]: pgmap v24: 89 pgs: 89 active+clean; 230 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 613 KiB/s wr, 16 op/s 2024-09-13T14:23:36.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:35 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:36.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:35 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:36.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:35 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:23:36.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:35 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:36.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:35 smithi151 ceph-mon[41387]: osdmap e236: 8 total, 8 up, 8 in 2024-09-13T14:23:36.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:35 smithi151 ceph-mon[29807]: from='client.25808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:36.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:35 smithi151 ceph-mon[29807]: pgmap v24: 89 pgs: 89 active+clean; 230 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 613 KiB/s wr, 16 op/s 2024-09-13T14:23:36.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:35 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:36.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:35 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:36.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:35 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:23:36.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:35 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:36.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:35 smithi151 ceph-mon[29807]: osdmap e236: 8 total, 8 up, 8 in 2024-09-13T14:23:36.823 INFO:tasks.rados.rados.0.smithi151.stdout:677: done (2 left) 2024-09-13T14:23:36.823 INFO:tasks.rados.rados.0.smithi151.stdout:678: write oid 30 current snap is 89 2024-09-13T14:23:36.824 INFO:tasks.rados.rados.0.smithi151.stdout:678: seq_num 286 ranges {406802=626275,1686203=728431} 2024-09-13T14:23:36.830 INFO:tasks.rados.rados.0.smithi151.stdout:678: writing smithi15186934-30 from 406802 to 1033077 tid 1 2024-09-13T14:23:36.837 INFO:tasks.rados.rados.0.smithi151.stdout:678: writing smithi15186934-30 from 1686203 to 2414634 tid 2 2024-09-13T14:23:36.838 INFO:tasks.rados.rados.0.smithi151.stdout:679: read oid 13 snap 78 2024-09-13T14:23:36.838 INFO:tasks.rados.rados.0.smithi151.stdout:679: expect (ObjNum 168 snap 44 seq_num 168) 2024-09-13T14:23:36.838 INFO:tasks.rados.rados.0.smithi151.stdout:674: expect (ObjNum 218 snap 63 seq_num 218) 2024-09-13T14:23:37.032 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:36 smithi184 ceph-mon[33394]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:23:37.032 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:36 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:23:37.032 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:36 smithi184 ceph-mon[33394]: osdmap e237: 8 total, 8 up, 8 in 2024-09-13T14:23:37.032 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:36 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:36.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi184\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:23:37.035 INFO:tasks.rados.rados.0.smithi151.stdout:676: finishing write tid 1 to smithi15186934-24 2024-09-13T14:23:37.035 INFO:tasks.rados.rados.0.smithi151.stdout:676: finishing write tid 2 to smithi15186934-24 2024-09-13T14:23:37.035 INFO:tasks.rados.rados.0.smithi151.stdout:676: finishing write tid 3 to smithi15186934-24 2024-09-13T14:23:37.035 INFO:tasks.rados.rados.0.smithi151.stdout:676: finishing write tid 4 to smithi15186934-24 2024-09-13T14:23:37.035 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 24 v 429 (ObjNum 285 snap 89 seq_num 285) dirty exists 2024-09-13T14:23:37.035 INFO:tasks.rados.rados.0.smithi151.stdout:676: left oid 24 (ObjNum 285 snap 89 seq_num 285) 2024-09-13T14:23:37.035 INFO:tasks.rados.rados.0.smithi151.stdout:680: read oid 19 snap 85 2024-09-13T14:23:37.035 INFO:tasks.rados.rados.0.smithi151.stdout:680: expect deleted 2024-09-13T14:23:37.035 INFO:tasks.rados.rados.0.smithi151.stdout:674: done (4 left) 2024-09-13T14:23:37.036 INFO:tasks.rados.rados.0.smithi151.stdout:676: done (3 left) 2024-09-13T14:23:37.036 INFO:tasks.rados.rados.0.smithi151.stdout:681: write oid 26 current snap is 89 2024-09-13T14:23:37.036 INFO:tasks.rados.rados.0.smithi151.stdout:678: finishing write tid 1 to smithi15186934-30 2024-09-13T14:23:37.036 INFO:tasks.rados.rados.0.smithi151.stdout:681: seq_num 287 ranges {423609=701524,1628332=472089,2648987=174622} 2024-09-13T14:23:37.041 INFO:tasks.rados.rados.0.smithi151.stdout:681: writing smithi15186934-26 from 423609 to 1125133 tid 1 2024-09-13T14:23:37.046 INFO:tasks.rados.rados.0.smithi151.stdout:681: writing smithi15186934-26 from 1628332 to 2100421 tid 2 2024-09-13T14:23:37.052 INFO:tasks.rados.rados.0.smithi151.stdout:681: writing smithi15186934-26 from 2648987 to 2823609 tid 3 2024-09-13T14:23:37.052 INFO:tasks.rados.rados.0.smithi151.stdout:682: read oid 44 snap 88 2024-09-13T14:23:37.052 INFO:tasks.rados.rados.0.smithi151.stdout:682: expect deleted 2024-09-13T14:23:37.052 INFO:tasks.rados.rados.0.smithi151.stdout:683: snap_create 2024-09-13T14:23:37.052 INFO:tasks.rados.rados.0.smithi151.stdout:678: finishing write tid 2 to smithi15186934-30 2024-09-13T14:23:37.052 INFO:tasks.rados.rados.0.smithi151.stdout:678: finishing write tid 3 to smithi15186934-30 2024-09-13T14:23:37.052 INFO:tasks.rados.rados.0.smithi151.stdout:678: finishing write tid 4 to smithi15186934-30 2024-09-13T14:23:37.052 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 30 v 293 (ObjNum 286 snap 89 seq_num 286) dirty exists 2024-09-13T14:23:37.052 INFO:tasks.rados.rados.0.smithi151.stdout:678: left oid 30 (ObjNum 286 snap 89 seq_num 286) 2024-09-13T14:23:37.052 INFO:tasks.rados.rados.0.smithi151.stdout:679: expect (ObjNum 168 snap 44 seq_num 168) 2024-09-13T14:23:37.178 INFO:tasks.rados.rados.0.smithi151.stdout:681: finishing write tid 1 to smithi15186934-26 2024-09-13T14:23:37.179 INFO:tasks.rados.rados.0.smithi151.stdout:681: finishing write tid 2 to smithi15186934-26 2024-09-13T14:23:37.179 INFO:tasks.rados.rados.0.smithi151.stdout:681: finishing write tid 3 to smithi15186934-26 2024-09-13T14:23:37.179 INFO:tasks.rados.rados.0.smithi151.stdout:681: finishing write tid 4 to smithi15186934-26 2024-09-13T14:23:37.179 INFO:tasks.rados.rados.0.smithi151.stdout:681: finishing write tid 5 to smithi15186934-26 2024-09-13T14:23:37.179 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 212 (ObjNum 287 snap 89 seq_num 287) dirty exists 2024-09-13T14:23:37.179 INFO:tasks.rados.rados.0.smithi151.stdout:681: left oid 26 (ObjNum 287 snap 89 seq_num 287) 2024-09-13T14:23:37.258 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:23:37.258 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:36 smithi151 ceph-mon[29807]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:23:37.258 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:36 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:23:37.258 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:36 smithi151 ceph-mon[29807]: osdmap e237: 8 total, 8 up, 8 in 2024-09-13T14:23:37.258 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:36 smithi151 ceph-mon[41387]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:23:37.259 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:36 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:23:37.259 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:36 smithi151 ceph-mon[41387]: osdmap e237: 8 total, 8 up, 8 in 2024-09-13T14:23:37.928 INFO:tasks.rados.rados.0.smithi151.stdout:678: done (5 left) 2024-09-13T14:23:37.929 INFO:tasks.rados.rados.0.smithi151.stdout:679: done (4 left) 2024-09-13T14:23:37.929 INFO:tasks.rados.rados.0.smithi151.stdout:680: done (3 left) 2024-09-13T14:23:37.929 INFO:tasks.rados.rados.0.smithi151.stdout:681: done (2 left) 2024-09-13T14:23:37.929 INFO:tasks.rados.rados.0.smithi151.stdout:682: done (1 left) 2024-09-13T14:23:37.929 INFO:tasks.rados.rados.0.smithi151.stdout:683: done (0 left) 2024-09-13T14:23:37.929 INFO:tasks.rados.rados.0.smithi151.stdout:684: write oid 15 current snap is 90 2024-09-13T14:23:37.929 INFO:tasks.rados.rados.0.smithi151.stdout:684: seq_num 288 ranges {440416=776773,1686814=732100} 2024-09-13T14:23:37.935 INFO:tasks.rados.rados.0.smithi151.stdout:684: writing smithi15186934-15 from 440416 to 1217189 tid 1 2024-09-13T14:23:37.942 INFO:tasks.rados.rados.0.smithi151.stdout:684: writing smithi15186934-15 from 1686814 to 2418914 tid 2 2024-09-13T14:23:37.942 INFO:tasks.rados.rados.0.smithi151.stdout:685: rollback oid 5 current snap is 90 2024-09-13T14:23:37.943 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 5 to 85 2024-09-13T14:23:37.943 INFO:tasks.rados.rados.0.smithi151.stdout:684: finishing write tid 1 to smithi15186934-15 2024-09-13T14:23:37.943 INFO:tasks.rados.rados.0.smithi151.stdout:686: read oid 36 snap -1 2024-09-13T14:23:37.943 INFO:tasks.rados.rados.0.smithi151.stdout:686: expect deleted 2024-09-13T14:23:37.943 INFO:tasks.rados.rados.0.smithi151.stdout:687: snap_remove snap 59 2024-09-13T14:23:38.050 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:37 smithi184 ceph-mon[33394]: pgmap v27: 89 pgs: 89 active+clean; 230 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:23:38.050 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:37 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/510136014' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:38.050 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:37 smithi184 ceph-mon[33394]: Upgrade: Updating mgr.x 2024-09-13T14:23:38.050 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:37 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:23:38.050 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:37 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:23:38.050 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:37 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:38.050 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:37 smithi184 ceph-mon[33394]: Deploying daemon mgr.x on smithi184 2024-09-13T14:23:38.050 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:37 smithi184 ceph-mon[33394]: from='client.15915 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:38.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:37 smithi151 ceph-mon[29807]: pgmap v27: 89 pgs: 89 active+clean; 230 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:23:38.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:37 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/510136014' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:38.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:37 smithi151 ceph-mon[29807]: Upgrade: Updating mgr.x 2024-09-13T14:23:38.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:37 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:23:38.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:37 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:23:38.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:37 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:38.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:37 smithi151 ceph-mon[29807]: Deploying daemon mgr.x on smithi184 2024-09-13T14:23:38.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:37 smithi151 ceph-mon[29807]: from='client.15915 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:38.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:37 smithi151 ceph-mon[41387]: pgmap v27: 89 pgs: 89 active+clean; 230 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:23:38.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:37 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/510136014' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:38.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:37 smithi151 ceph-mon[41387]: Upgrade: Updating mgr.x 2024-09-13T14:23:38.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:37 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:23:38.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:37 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:23:38.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:37 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:38.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:37 smithi151 ceph-mon[41387]: Deploying daemon mgr.x on smithi184 2024-09-13T14:23:38.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:37 smithi151 ceph-mon[41387]: from='client.15915 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:38.853 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:38 smithi184 systemd[1]: Stopping Ceph mgr.x for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:23:38.942 INFO:tasks.rados.rados.0.smithi151.stdout:685: finishing rollback tid 0 to smithi15186934-5 2024-09-13T14:23:38.942 INFO:tasks.rados.rados.0.smithi151.stdout:685: finishing rollback tid 1 to smithi15186934-5 2024-09-13T14:23:38.942 INFO:tasks.rados.rados.0.smithi151.stdout:685: finishing rollback tid 2 to smithi15186934-5 2024-09-13T14:23:38.942 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 188 (ObjNum 218 snap 63 seq_num 218) dirty exists 2024-09-13T14:23:38.942 INFO:tasks.rados.rados.0.smithi151.stdout:684: finishing write tid 2 to smithi15186934-15 2024-09-13T14:23:38.942 INFO:tasks.rados.rados.0.smithi151.stdout:684: finishing write tid 3 to smithi15186934-15 2024-09-13T14:23:38.943 INFO:tasks.rados.rados.0.smithi151.stdout:684: finishing write tid 4 to smithi15186934-15 2024-09-13T14:23:38.943 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 15 v 332 (ObjNum 288 snap 90 seq_num 288) dirty exists 2024-09-13T14:23:38.943 INFO:tasks.rados.rados.0.smithi151.stdout:684: left oid 15 (ObjNum 288 snap 90 seq_num 288) 2024-09-13T14:23:38.943 INFO:tasks.rados.rados.0.smithi151.stdout:684: done (3 left) 2024-09-13T14:23:38.943 INFO:tasks.rados.rados.0.smithi151.stdout:685: done (2 left) 2024-09-13T14:23:38.943 INFO:tasks.rados.rados.0.smithi151.stdout:686: done (1 left) 2024-09-13T14:23:38.943 INFO:tasks.rados.rados.0.smithi151.stdout:687: done (0 left) 2024-09-13T14:23:38.943 INFO:tasks.rados.rados.0.smithi151.stdout:688: read oid 45 snap -1 2024-09-13T14:23:38.943 INFO:tasks.rados.rados.0.smithi151.stdout:688: expect (ObjNum 279 snap 88 seq_num 279) 2024-09-13T14:23:38.943 INFO:tasks.rados.rados.0.smithi151.stdout:689: snap_create 2024-09-13T14:23:38.955 INFO:tasks.rados.rados.0.smithi151.stdout:688: expect (ObjNum 279 snap 88 seq_num 279) 2024-09-13T14:23:39.197 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:38 smithi184 ceph-mon[33394]: from='client.25826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:39.197 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:38 smithi184 ceph-mon[33394]: osdmap e238: 8 total, 8 up, 8 in 2024-09-13T14:23:39.198 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:38 smithi184 podman[81882]: 2024-09-13 14:23:38.878217239 +0000 UTC m=+0.138201562 container died 6c42b96484232d0d71c5b936fa80ee86c8b44d8dec39cc4fbe694ff58ac68132 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-13T14:23:39.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:38 smithi151 ceph-mon[29807]: from='client.25826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:39.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:38 smithi151 ceph-mon[29807]: osdmap e238: 8 total, 8 up, 8 in 2024-09-13T14:23:39.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:38 smithi151 ceph-mon[41387]: from='client.25826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:39.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:38 smithi151 ceph-mon[41387]: osdmap e238: 8 total, 8 up, 8 in 2024-09-13T14:23:39.496 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:39 smithi184 podman[81882]: 2024-09-13 14:23:39.286628119 +0000 UTC m=+0.546612438 container cleanup 6c42b96484232d0d71c5b936fa80ee86c8b44d8dec39cc4fbe694ff58ac68132 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:23:39.497 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:39 smithi184 bash[81882]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x 2024-09-13T14:23:39.854 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:39 smithi184 podman[81899]: 2024-09-13 14:23:39.49629781 +0000 UTC m=+0.615351525 container remove 6c42b96484232d0d71c5b936fa80ee86c8b44d8dec39cc4fbe694ff58ac68132 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:23:39.854 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:39 smithi184 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-13T14:23:39.854 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:39 smithi184 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-09-13T14:23:39.854 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:39 smithi184 systemd[1]: Stopped Ceph mgr.x for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:23:39.854 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:39 smithi184 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@mgr.x.service: Consumed 18.030s CPU time. 2024-09-13T14:23:39.947 INFO:tasks.rados.rados.0.smithi151.stdout:688: done (1 left) 2024-09-13T14:23:39.947 INFO:tasks.rados.rados.0.smithi151.stdout:689: done (0 left) 2024-09-13T14:23:39.947 INFO:tasks.rados.rados.0.smithi151.stdout:690: write oid 27 current snap is 91 2024-09-13T14:23:39.947 INFO:tasks.rados.rados.0.smithi151.stdout:690: seq_num 289 ranges {457223=568375,1461649=475758,2420186=437037} 2024-09-13T14:23:39.952 INFO:tasks.rados.rados.0.smithi151.stdout:690: writing smithi15186934-27 from 457223 to 1025598 tid 1 2024-09-13T14:23:39.957 INFO:tasks.rados.rados.0.smithi151.stdout:690: writing smithi15186934-27 from 1461649 to 1937407 tid 2 2024-09-13T14:23:39.963 INFO:tasks.rados.rados.0.smithi151.stdout:690: writing smithi15186934-27 from 2420186 to 2857223 tid 3 2024-09-13T14:23:39.963 INFO:tasks.rados.rados.0.smithi151.stdout:691: read oid 16 snap -1 2024-09-13T14:23:39.963 INFO:tasks.rados.rados.0.smithi151.stdout:691: expect deleted 2024-09-13T14:23:39.963 INFO:tasks.rados.rados.0.smithi151.stdout:692: snap_remove snap 66 2024-09-13T14:23:40.158 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:23:40.212 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:39 smithi184 ceph-mon[33394]: pgmap v30: 89 pgs: 89 active+clean; 236 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.4 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2024-09-13T14:23:40.213 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:39 smithi184 ceph-mon[33394]: osdmap e239: 8 total, 8 up, 8 in 2024-09-13T14:23:40.213 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:39 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/3198428447' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:40.213 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:39 smithi184 systemd[1]: Starting Ceph mgr.x for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:23:40.213 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:40 smithi184 podman[82031]: 2024-09-13 14:23:40.027522979 +0000 UTC m=+0.015592863 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:23:40.213 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:40 smithi184 podman[82031]: 2024-09-13 14:23:40.139381345 +0000 UTC m=+0.127451231 container create 8a0f7d440b6e957af920c6c95c7e6875804ca27fb6c0762d585b79ba5935b226 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True) 2024-09-13T14:23:40.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:39 smithi151 ceph-mon[41387]: pgmap v30: 89 pgs: 89 active+clean; 236 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.4 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2024-09-13T14:23:40.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:39 smithi151 ceph-mon[41387]: osdmap e239: 8 total, 8 up, 8 in 2024-09-13T14:23:40.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:39 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/3198428447' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:40.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:39 smithi151 ceph-mon[29807]: pgmap v30: 89 pgs: 89 active+clean; 236 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.4 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2024-09-13T14:23:40.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:39 smithi151 ceph-mon[29807]: osdmap e239: 8 total, 8 up, 8 in 2024-09-13T14:23:40.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:39 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/3198428447' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:40.475 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:40.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:23:40.476 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:40 smithi184 podman[82031]: 2024-09-13 14:23:40.400351477 +0000 UTC m=+0.388421362 container init 8a0f7d440b6e957af920c6c95c7e6875804ca27fb6c0762d585b79ba5935b226 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730) 2024-09-13T14:23:40.476 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:40 smithi184 podman[82031]: 2024-09-13 14:23:40.405115055 +0000 UTC m=+0.393184943 container start 8a0f7d440b6e957af920c6c95c7e6875804ca27fb6c0762d585b79ba5935b226 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-13T14:23:40.476 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:40 smithi184 ceph-mgr[82069]: -- 172.21.15.184:0/2992253373 <== mon.1 v2:172.21.15.184:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5591b862d0e0 con 0x5591b860b400 2024-09-13T14:23:40.853 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:40 smithi184 bash[82031]: 8a0f7d440b6e957af920c6c95c7e6875804ca27fb6c0762d585b79ba5935b226 2024-09-13T14:23:40.854 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:40 smithi184 systemd[1]: Started Ceph mgr.x for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:23:40.952 INFO:tasks.rados.rados.0.smithi151.stdout:692: done (2 left) 2024-09-13T14:23:40.952 INFO:tasks.rados.rados.0.smithi151.stdout:693: rollback oid 3 current snap is 91 2024-09-13T14:23:40.953 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 3 to 86 2024-09-13T14:23:40.953 INFO:tasks.rados.rados.0.smithi151.stdout:690: finishing write tid 1 to smithi15186934-27 2024-09-13T14:23:40.953 INFO:tasks.rados.rados.0.smithi151.stdout:690: finishing write tid 2 to smithi15186934-27 2024-09-13T14:23:40.953 INFO:tasks.rados.rados.0.smithi151.stdout:690: finishing write tid 3 to smithi15186934-27 2024-09-13T14:23:40.953 INFO:tasks.rados.rados.0.smithi151.stdout:690: finishing write tid 4 to smithi15186934-27 2024-09-13T14:23:40.953 INFO:tasks.rados.rados.0.smithi151.stdout:690: finishing write tid 5 to smithi15186934-27 2024-09-13T14:23:40.953 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 27 v 171 (ObjNum 289 snap 91 seq_num 289) dirty exists 2024-09-13T14:23:40.953 INFO:tasks.rados.rados.0.smithi151.stdout:690: left oid 27 (ObjNum 289 snap 91 seq_num 289) 2024-09-13T14:23:40.953 INFO:tasks.rados.rados.0.smithi151.stdout:690: done (2 left) 2024-09-13T14:23:40.953 INFO:tasks.rados.rados.0.smithi151.stdout:691: done (1 left) 2024-09-13T14:23:40.953 INFO:tasks.rados.rados.0.smithi151.stdout:694: snap_create 2024-09-13T14:23:40.955 INFO:tasks.rados.rados.0.smithi151.stdout:693: finishing rollback tid 1 to smithi15186934-3 2024-09-13T14:23:40.955 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 122 (ObjNum 33 snap 0 seq_num 3092790752) dirty dne 2024-09-13T14:23:41.321 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:40 smithi184 ceph-mon[33394]: osdmap e240: 8 total, 8 up, 8 in 2024-09-13T14:23:41.321 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:40 smithi184 ceph-mon[33394]: from='client.15933 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:41.321 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:40 smithi184 ceph-mon[33394]: from='client.25844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:41.321 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:40 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:41.321 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:40 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:41.321 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:40 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:41.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:40 smithi151 ceph-mon[29807]: osdmap e240: 8 total, 8 up, 8 in 2024-09-13T14:23:41.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:40 smithi151 ceph-mon[29807]: from='client.15933 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:41.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:40 smithi151 ceph-mon[29807]: from='client.25844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:41.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:40 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:41.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:40 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:41.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:40 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:41.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:40 smithi151 ceph-mon[41387]: osdmap e240: 8 total, 8 up, 8 in 2024-09-13T14:23:41.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:40 smithi151 ceph-mon[41387]: from='client.15933 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:41.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:40 smithi151 ceph-mon[41387]: from='client.25844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:41.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:40 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:41.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:40 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:41.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:40 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:41.948 INFO:tasks.rados.rados.0.smithi151.stdout:693: done (1 left) 2024-09-13T14:23:41.948 INFO:tasks.rados.rados.0.smithi151.stdout:694: done (0 left) 2024-09-13T14:23:41.948 INFO:tasks.rados.rados.0.smithi151.stdout:695: read oid 17 snap 91 2024-09-13T14:23:41.948 INFO:tasks.rados.rados.0.smithi151.stdout:695: expect (ObjNum 253 snap 79 seq_num 253) 2024-09-13T14:23:41.948 INFO:tasks.rados.rados.0.smithi151.stdout:696: rollback oid 3 current snap is 92 2024-09-13T14:23:41.949 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 3 to 82 2024-09-13T14:23:41.949 INFO:tasks.rados.rados.0.smithi151.stdout:697: rollback oid 39 current snap is 92 2024-09-13T14:23:41.949 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 39 to 80 2024-09-13T14:23:41.949 INFO:tasks.rados.rados.0.smithi151.stdout:698: write oid 37 current snap is 92 2024-09-13T14:23:41.949 INFO:tasks.rados.rados.0.smithi151.stdout:698: seq_num 290 ranges {474030=643624,1520131=735769} 2024-09-13T14:23:41.954 INFO:tasks.rados.rados.0.smithi151.stdout:698: writing smithi15186934-37 from 474030 to 1117654 tid 1 2024-09-13T14:23:41.960 INFO:tasks.rados.rados.0.smithi151.stdout:698: writing smithi15186934-37 from 1520131 to 2255900 tid 2 2024-09-13T14:23:41.961 INFO:tasks.rados.rados.0.smithi151.stdout:699: snap_create 2024-09-13T14:23:41.961 INFO:tasks.rados.rados.0.smithi151.stdout:697: finishing rollback tid 1 to smithi15186934-39 2024-09-13T14:23:41.961 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 142 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-13T14:23:41.961 INFO:tasks.rados.rados.0.smithi151.stdout:696: finishing rollback tid 1 to smithi15186934-3 2024-09-13T14:23:41.961 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 122 (ObjNum 129 snap 0 seq_num 3681342637) dirty dne 2024-09-13T14:23:41.961 INFO:tasks.rados.rados.0.smithi151.stdout:695: expect (ObjNum 253 snap 79 seq_num 253) 2024-09-13T14:23:42.061 INFO:tasks.rados.rados.0.smithi151.stdout:698: finishing write tid 1 to smithi15186934-37 2024-09-13T14:23:42.061 INFO:tasks.rados.rados.0.smithi151.stdout:698: finishing write tid 2 to smithi15186934-37 2024-09-13T14:23:42.061 INFO:tasks.rados.rados.0.smithi151.stdout:698: finishing write tid 3 to smithi15186934-37 2024-09-13T14:23:42.062 INFO:tasks.rados.rados.0.smithi151.stdout:698: finishing write tid 4 to smithi15186934-37 2024-09-13T14:23:42.062 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 37 v 351 (ObjNum 290 snap 92 seq_num 290) dirty exists 2024-09-13T14:23:42.062 INFO:tasks.rados.rados.0.smithi151.stdout:698: left oid 37 (ObjNum 290 snap 92 seq_num 290) 2024-09-13T14:23:42.217 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:41 smithi184 ceph-mon[33394]: pgmap v33: 89 pgs: 89 active+clean; 236 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.4 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2024-09-13T14:23:42.217 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:41 smithi184 ceph-mon[33394]: osdmap e241: 8 total, 8 up, 8 in 2024-09-13T14:23:42.217 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:41 smithi184 ceph-mon[33394]: osdmap e242: 8 total, 8 up, 8 in 2024-09-13T14:23:42.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:41 smithi151 ceph-mon[29807]: pgmap v33: 89 pgs: 89 active+clean; 236 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.4 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2024-09-13T14:23:42.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:41 smithi151 ceph-mon[29807]: osdmap e241: 8 total, 8 up, 8 in 2024-09-13T14:23:42.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:41 smithi151 ceph-mon[29807]: osdmap e242: 8 total, 8 up, 8 in 2024-09-13T14:23:42.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:41 smithi151 ceph-mon[41387]: pgmap v33: 89 pgs: 89 active+clean; 236 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.4 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2024-09-13T14:23:42.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:41 smithi151 ceph-mon[41387]: osdmap e241: 8 total, 8 up, 8 in 2024-09-13T14:23:42.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:41 smithi151 ceph-mon[41387]: osdmap e242: 8 total, 8 up, 8 in 2024-09-13T14:23:42.812 INFO:tasks.rados.rados.0.smithi151.stdout:695: done (4 left) 2024-09-13T14:23:42.812 INFO:tasks.rados.rados.0.smithi151.stdout:696: done (3 left) 2024-09-13T14:23:42.812 INFO:tasks.rados.rados.0.smithi151.stdout:697: done (2 left) 2024-09-13T14:23:42.813 INFO:tasks.rados.rados.0.smithi151.stdout:698: done (1 left) 2024-09-13T14:23:42.813 INFO:tasks.rados.rados.0.smithi151.stdout:699: done (0 left) 2024-09-13T14:23:42.813 INFO:tasks.rados.rados.0.smithi151.stdout:700: rollback oid 45 current snap is 93 2024-09-13T14:23:42.813 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 45 to 78 2024-09-13T14:23:42.813 INFO:tasks.rados.rados.0.smithi151.stdout:701: read oid 9 snap -1 2024-09-13T14:23:42.813 INFO:tasks.rados.rados.0.smithi151.stdout:701: expect (ObjNum 268 snap 86 seq_num 268) 2024-09-13T14:23:42.813 INFO:tasks.rados.rados.0.smithi151.stdout:702: write oid 10 current snap is 93 2024-09-13T14:23:42.813 INFO:tasks.rados.rados.0.smithi151.stdout:702: seq_num 291 ranges {490837=718873,1862260=479427,2875032=15805} 2024-09-13T14:23:42.819 INFO:tasks.rados.rados.0.smithi151.stdout:702: writing smithi15186934-10 from 490837 to 1209710 tid 1 2024-09-13T14:23:42.825 INFO:tasks.rados.rados.0.smithi151.stdout:702: writing smithi15186934-10 from 1862260 to 2341687 tid 2 2024-09-13T14:23:42.829 INFO:tasks.rados.rados.0.smithi151.stdout:702: writing smithi15186934-10 from 2875032 to 2890837 tid 3 2024-09-13T14:23:42.829 INFO:tasks.rados.rados.0.smithi151.stdout:703: write oid 46 current snap is 93 2024-09-13T14:23:42.829 INFO:tasks.rados.rados.0.smithi151.stdout:700: finishing rollback tid 0 to smithi15186934-45 2024-09-13T14:23:42.829 INFO:tasks.rados.rados.0.smithi151.stdout:700: finishing rollback tid 1 to smithi15186934-45 2024-09-13T14:23:42.829 INFO:tasks.rados.rados.0.smithi151.stdout:700: finishing rollback tid 2 to smithi15186934-45 2024-09-13T14:23:42.829 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 444 (ObjNum 251 snap 78 seq_num 251) dirty exists 2024-09-13T14:23:42.829 INFO:tasks.rados.rados.0.smithi151.stdout:701: expect (ObjNum 268 snap 86 seq_num 268) 2024-09-13T14:23:43.011 INFO:tasks.rados.rados.0.smithi151.stdout:702: finishing write tid 1 to smithi15186934-10 2024-09-13T14:23:43.011 INFO:tasks.rados.rados.0.smithi151.stdout:702: finishing write tid 2 to smithi15186934-10 2024-09-13T14:23:43.011 INFO:tasks.rados.rados.0.smithi151.stdout:703: seq_num 292 ranges {507644=794122,1920742=623085} 2024-09-13T14:23:43.017 INFO:tasks.rados.rados.0.smithi151.stdout:703: writing smithi15186934-46 from 507644 to 1301766 tid 1 2024-09-13T14:23:43.025 INFO:tasks.rados.rados.0.smithi151.stdout:703: writing smithi15186934-46 from 1920742 to 2543827 tid 2 2024-09-13T14:23:43.025 INFO:tasks.rados.rados.0.smithi151.stdout:700: done (3 left) 2024-09-13T14:23:43.025 INFO:tasks.rados.rados.0.smithi151.stdout:701: done (2 left) 2024-09-13T14:23:43.025 INFO:tasks.rados.rados.0.smithi151.stdout:704: write oid 44 current snap is 93 2024-09-13T14:23:43.025 INFO:tasks.rados.rados.0.smithi151.stdout:702: finishing write tid 3 to smithi15186934-10 2024-09-13T14:23:43.025 INFO:tasks.rados.rados.0.smithi151.stdout:702: finishing write tid 4 to smithi15186934-10 2024-09-13T14:23:43.025 INFO:tasks.rados.rados.0.smithi151.stdout:704: seq_num 293 ranges {524451=469371,1579224=483096,2529878=394573} 2024-09-13T14:23:43.030 INFO:tasks.rados.rados.0.smithi151.stdout:704: writing smithi15186934-44 from 524451 to 993822 tid 1 2024-09-13T14:23:43.036 INFO:tasks.rados.rados.0.smithi151.stdout:704: writing smithi15186934-44 from 1579224 to 2062320 tid 2 2024-09-13T14:23:43.041 INFO:tasks.rados.rados.0.smithi151.stdout:704: writing smithi15186934-44 from 2529878 to 2924451 tid 3 2024-09-13T14:23:43.041 INFO:tasks.rados.rados.0.smithi151.stdout:705: write oid 6 current snap is 93 2024-09-13T14:23:43.041 INFO:tasks.rados.rados.0.smithi151.stdout:705: seq_num 294 ranges {541258=544620,1637706=626754,2840948=100310} 2024-09-13T14:23:43.047 INFO:tasks.rados.rados.0.smithi151.stdout:705: writing smithi15186934-6 from 541258 to 1085878 tid 1 2024-09-13T14:23:43.053 INFO:tasks.rados.rados.0.smithi151.stdout:705: writing smithi15186934-6 from 1637706 to 2264460 tid 2 2024-09-13T14:23:43.057 INFO:tasks.rados.rados.0.smithi151.stdout:705: writing smithi15186934-6 from 2840948 to 2941258 tid 3 2024-09-13T14:23:43.058 INFO:tasks.rados.rados.0.smithi151.stdout:706: read oid 32 snap -1 2024-09-13T14:23:43.058 INFO:tasks.rados.rados.0.smithi151.stdout:706: expect (ObjNum 222 snap 64 seq_num 222) 2024-09-13T14:23:43.058 INFO:tasks.rados.rados.0.smithi151.stdout:707: write oid 5 current snap is 93 2024-09-13T14:23:43.058 INFO:tasks.rados.rados.0.smithi151.stdout:702: finishing write tid 5 to smithi15186934-10 2024-09-13T14:23:43.058 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 226 (ObjNum 291 snap 93 seq_num 291) dirty exists 2024-09-13T14:23:43.058 INFO:tasks.rados.rados.0.smithi151.stdout:702: left oid 10 (ObjNum 291 snap 93 seq_num 291) 2024-09-13T14:23:43.058 INFO:tasks.rados.rados.0.smithi151.stdout:703: finishing write tid 1 to smithi15186934-46 2024-09-13T14:23:43.058 INFO:tasks.rados.rados.0.smithi151.stdout:703: finishing write tid 2 to smithi15186934-46 2024-09-13T14:23:43.058 INFO:tasks.rados.rados.0.smithi151.stdout:703: finishing write tid 3 to smithi15186934-46 2024-09-13T14:23:43.058 INFO:tasks.rados.rados.0.smithi151.stdout:707: seq_num 295 ranges {558065=619869,1579835=486765,2468371=489694} 2024-09-13T14:23:43.064 INFO:tasks.rados.rados.0.smithi151.stdout:707: writing smithi15186934-5 from 558065 to 1177934 tid 1 2024-09-13T14:23:43.067 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:23:43.069 INFO:tasks.rados.rados.0.smithi151.stdout:707: writing smithi15186934-5 from 1579835 to 2066600 tid 2 2024-09-13T14:23:43.074 INFO:tasks.rados.rados.0.smithi151.stdout:707: writing smithi15186934-5 from 2468371 to 2958065 tid 3 2024-09-13T14:23:43.075 INFO:tasks.rados.rados.0.smithi151.stdout:702: done (5 left) 2024-09-13T14:23:43.075 INFO:tasks.rados.rados.0.smithi151.stdout:708: read oid 23 snap -1 2024-09-13T14:23:43.075 INFO:tasks.rados.rados.0.smithi151.stdout:708: expect deleted 2024-09-13T14:23:43.075 INFO:tasks.rados.rados.0.smithi151.stdout:709: write oid 14 current snap is 93 2024-09-13T14:23:43.075 INFO:tasks.rados.rados.0.smithi151.stdout:709: seq_num 296 ranges {574872=695118,2038317=630423} 2024-09-13T14:23:43.081 INFO:tasks.rados.rados.0.smithi151.stdout:709: writing smithi15186934-14 from 574872 to 1269990 tid 1 2024-09-13T14:23:43.090 INFO:tasks.rados.rados.0.smithi151.stdout:709: writing smithi15186934-14 from 2038317 to 2668740 tid 2 2024-09-13T14:23:43.090 INFO:tasks.rados.rados.0.smithi151.stdout:710: write oid 20 current snap is 93 2024-09-13T14:23:43.090 INFO:tasks.rados.rados.0.smithi151.stdout:703: finishing write tid 4 to smithi15186934-46 2024-09-13T14:23:43.090 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 46 v 230 (ObjNum 292 snap 93 seq_num 292) dirty exists 2024-09-13T14:23:43.090 INFO:tasks.rados.rados.0.smithi151.stdout:703: left oid 46 (ObjNum 292 snap 93 seq_num 292) 2024-09-13T14:23:43.090 INFO:tasks.rados.rados.0.smithi151.stdout:704: finishing write tid 1 to smithi15186934-44 2024-09-13T14:23:43.090 INFO:tasks.rados.rados.0.smithi151.stdout:704: finishing write tid 2 to smithi15186934-44 2024-09-13T14:23:43.091 INFO:tasks.rados.rados.0.smithi151.stdout:704: finishing write tid 3 to smithi15186934-44 2024-09-13T14:23:43.091 INFO:tasks.rados.rados.0.smithi151.stdout:704: finishing write tid 4 to smithi15186934-44 2024-09-13T14:23:43.091 INFO:tasks.rados.rados.0.smithi151.stdout:704: finishing write tid 5 to smithi15186934-44 2024-09-13T14:23:43.091 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 222 (ObjNum 293 snap 93 seq_num 293) dirty exists 2024-09-13T14:23:43.091 INFO:tasks.rados.rados.0.smithi151.stdout:704: left oid 44 (ObjNum 293 snap 93 seq_num 293) 2024-09-13T14:23:43.091 INFO:tasks.rados.rados.0.smithi151.stdout:705: finishing write tid 1 to smithi15186934-6 2024-09-13T14:23:43.091 INFO:tasks.rados.rados.0.smithi151.stdout:705: finishing write tid 2 to smithi15186934-6 2024-09-13T14:23:43.091 INFO:tasks.rados.rados.0.smithi151.stdout:710: seq_num 297 ranges {591679=486720,1813152=490434} 2024-09-13T14:23:43.095 INFO:tasks.rados.rados.0.smithi151.stdout:710: writing smithi15186934-20 from 591679 to 1078399 tid 1 2024-09-13T14:23:43.102 INFO:tasks.rados.rados.0.smithi151.stdout:710: writing smithi15186934-20 from 1813152 to 2303586 tid 2 2024-09-13T14:23:43.102 INFO:tasks.rados.rados.0.smithi151.stdout:703: done (7 left) 2024-09-13T14:23:43.102 INFO:tasks.rados.rados.0.smithi151.stdout:704: done (6 left) 2024-09-13T14:23:43.102 INFO:tasks.rados.rados.0.smithi151.stdout:711: read oid 10 snap -1 2024-09-13T14:23:43.102 INFO:tasks.rados.rados.0.smithi151.stdout:711: expect (ObjNum 291 snap 93 seq_num 291) 2024-09-13T14:23:43.102 INFO:tasks.rados.rados.0.smithi151.stdout:712: delete oid 29 current snap is 93 2024-09-13T14:23:43.102 INFO:tasks.rados.rados.0.smithi151.stdout:705: finishing write tid 3 to smithi15186934-6 2024-09-13T14:23:43.102 INFO:tasks.rados.rados.0.smithi151.stdout:705: finishing write tid 4 to smithi15186934-6 2024-09-13T14:23:43.103 INFO:tasks.rados.rados.0.smithi151.stdout:705: finishing write tid 5 to smithi15186934-6 2024-09-13T14:23:43.103 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 6 v 201 (ObjNum 294 snap 93 seq_num 294) dirty exists 2024-09-13T14:23:43.103 INFO:tasks.rados.rados.0.smithi151.stdout:705: left oid 6 (ObjNum 294 snap 93 seq_num 294) 2024-09-13T14:23:43.103 INFO:tasks.rados.rados.0.smithi151.stdout:706: expect (ObjNum 222 snap 64 seq_num 222) 2024-09-13T14:23:43.234 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:42 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/1964512772' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:43.234 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:42 smithi184 ceph-mon[33394]: osdmap e243: 8 total, 8 up, 8 in 2024-09-13T14:23:43.342 INFO:tasks.rados.rados.0.smithi151.stdout:707: finishing write tid 1 to smithi15186934-5 2024-09-13T14:23:43.342 INFO:tasks.rados.rados.0.smithi151.stdout:707: finishing write tid 2 to smithi15186934-5 2024-09-13T14:23:43.342 INFO:tasks.rados.rados.0.smithi151.stdout:705: done (7 left) 2024-09-13T14:23:43.343 INFO:tasks.rados.rados.0.smithi151.stdout:706: done (6 left) 2024-09-13T14:23:43.343 INFO:tasks.rados.rados.0.smithi151.stdout:712: done (5 left) 2024-09-13T14:23:43.343 INFO:tasks.rados.rados.0.smithi151.stdout:713: snap_remove snap 40 2024-09-13T14:23:43.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:42 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/1964512772' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:43.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:42 smithi151 ceph-mon[41387]: osdmap e243: 8 total, 8 up, 8 in 2024-09-13T14:23:43.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:42 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1964512772' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:43.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:42 smithi151 ceph-mon[29807]: osdmap e243: 8 total, 8 up, 8 in 2024-09-13T14:23:43.969 INFO:tasks.rados.rados.0.smithi151.stdout:713: done (5 left) 2024-09-13T14:23:43.969 INFO:tasks.rados.rados.0.smithi151.stdout:714: read oid 35 snap -1 2024-09-13T14:23:43.969 INFO:tasks.rados.rados.0.smithi151.stdout:714: expect (ObjNum 196 snap 55 seq_num 196) 2024-09-13T14:23:43.969 INFO:tasks.rados.rados.0.smithi151.stdout:715: read oid 34 snap -1 2024-09-13T14:23:43.969 INFO:tasks.rados.rados.0.smithi151.stdout:715: expect (ObjNum 259 snap 81 seq_num 259) 2024-09-13T14:23:43.969 INFO:tasks.rados.rados.0.smithi151.stdout:716: rollback oid 37 current snap is 93 2024-09-13T14:23:43.969 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 37 to 91 2024-09-13T14:23:43.969 INFO:tasks.rados.rados.0.smithi151.stdout:717: write oid 39 current snap is 93 2024-09-13T14:23:43.969 INFO:tasks.rados.rados.0.smithi151.stdout:717: seq_num 299 ranges {625293=637218,1813763=494103,2978063=47230} 2024-09-13T14:23:43.976 INFO:tasks.rados.rados.0.smithi151.stdout:717: writing smithi15186934-39 from 625293 to 1262511 tid 1 2024-09-13T14:23:43.983 INFO:tasks.rados.rados.0.smithi151.stdout:717: writing smithi15186934-39 from 1813763 to 2307866 tid 2 2024-09-13T14:23:43.987 INFO:tasks.rados.rados.0.smithi151.stdout:717: writing smithi15186934-39 from 2978063 to 3025293 tid 3 2024-09-13T14:23:43.988 INFO:tasks.rados.rados.0.smithi151.stdout:718: write oid 22 current snap is 93 2024-09-13T14:23:43.988 INFO:tasks.rados.rados.0.smithi151.stdout:707: finishing write tid 3 to smithi15186934-5 2024-09-13T14:23:43.988 INFO:tasks.rados.rados.0.smithi151.stdout:707: finishing write tid 4 to smithi15186934-5 2024-09-13T14:23:43.988 INFO:tasks.rados.rados.0.smithi151.stdout:707: finishing write tid 5 to smithi15186934-5 2024-09-13T14:23:43.988 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 201 (ObjNum 295 snap 93 seq_num 295) dirty exists 2024-09-13T14:23:43.988 INFO:tasks.rados.rados.0.smithi151.stdout:707: left oid 5 (ObjNum 295 snap 93 seq_num 295) 2024-09-13T14:23:43.988 INFO:tasks.rados.rados.0.smithi151.stdout:709: finishing write tid 1 to smithi15186934-14 2024-09-13T14:23:43.988 INFO:tasks.rados.rados.0.smithi151.stdout:718: seq_num 300 ranges {642100=712467,1872245=637761} 2024-09-13T14:23:43.995 INFO:tasks.rados.rados.0.smithi151.stdout:718: writing smithi15186934-22 from 642100 to 1354567 tid 1 2024-09-13T14:23:44.001 INFO:tasks.rados.rados.0.smithi151.stdout:718: writing smithi15186934-22 from 1872245 to 2510006 tid 2 2024-09-13T14:23:44.001 INFO:tasks.rados.rados.0.smithi151.stdout:707: done (9 left) 2024-09-13T14:23:44.001 INFO:tasks.rados.rados.0.smithi151.stdout:708: done (8 left) 2024-09-13T14:23:44.001 INFO:tasks.rados.rados.0.smithi151.stdout:719: write oid 16 current snap is 93 2024-09-13T14:23:44.001 INFO:tasks.rados.rados.0.smithi151.stdout:709: finishing write tid 2 to smithi15186934-14 2024-09-13T14:23:44.002 INFO:tasks.rados.rados.0.smithi151.stdout:709: finishing write tid 3 to smithi15186934-14 2024-09-13T14:23:44.002 INFO:tasks.rados.rados.0.smithi151.stdout:709: finishing write tid 4 to smithi15186934-14 2024-09-13T14:23:44.002 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 310 (ObjNum 296 snap 93 seq_num 296) dirty exists 2024-09-13T14:23:44.002 INFO:tasks.rados.rados.0.smithi151.stdout:709: left oid 14 (ObjNum 296 snap 93 seq_num 296) 2024-09-13T14:23:44.002 INFO:tasks.rados.rados.0.smithi151.stdout:710: finishing write tid 1 to smithi15186934-20 2024-09-13T14:23:44.002 INFO:tasks.rados.rados.0.smithi151.stdout:719: seq_num 301 ranges {658907=787716,1930727=497772,3032909=25998} 2024-09-13T14:23:44.009 INFO:tasks.rados.rados.0.smithi151.stdout:719: writing smithi15186934-16 from 658907 to 1446623 tid 1 2024-09-13T14:23:44.014 INFO:tasks.rados.rados.0.smithi151.stdout:719: writing smithi15186934-16 from 1930727 to 2428499 tid 2 2024-09-13T14:23:44.018 INFO:tasks.rados.rados.0.smithi151.stdout:719: writing smithi15186934-16 from 3032909 to 3058907 tid 3 2024-09-13T14:23:44.018 INFO:tasks.rados.rados.0.smithi151.stdout:709: done (8 left) 2024-09-13T14:23:44.018 INFO:tasks.rados.rados.0.smithi151.stdout:720: write oid 38 current snap is 93 2024-09-13T14:23:44.018 INFO:tasks.rados.rados.0.smithi151.stdout:720: seq_num 302 ranges {675714=462965,1589209=641430,2943979=131735} 2024-09-13T14:23:44.024 INFO:tasks.rados.rados.0.smithi151.stdout:720: writing smithi15186934-38 from 675714 to 1138679 tid 1 2024-09-13T14:23:44.030 INFO:tasks.rados.rados.0.smithi151.stdout:720: writing smithi15186934-38 from 1589209 to 2230639 tid 2 2024-09-13T14:23:44.035 INFO:tasks.rados.rados.0.smithi151.stdout:720: writing smithi15186934-38 from 2943979 to 3075714 tid 3 2024-09-13T14:23:44.035 INFO:tasks.rados.rados.0.smithi151.stdout:721: snap_create 2024-09-13T14:23:44.035 INFO:tasks.rados.rados.0.smithi151.stdout:710: finishing write tid 2 to smithi15186934-20 2024-09-13T14:23:44.035 INFO:tasks.rados.rados.0.smithi151.stdout:710: finishing write tid 3 to smithi15186934-20 2024-09-13T14:23:44.035 INFO:tasks.rados.rados.0.smithi151.stdout:710: finishing write tid 4 to smithi15186934-20 2024-09-13T14:23:44.035 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 177 (ObjNum 297 snap 93 seq_num 297) dirty exists 2024-09-13T14:23:44.035 INFO:tasks.rados.rados.0.smithi151.stdout:710: left oid 20 (ObjNum 297 snap 93 seq_num 297) 2024-09-13T14:23:44.035 INFO:tasks.rados.rados.0.smithi151.stdout:711: expect (ObjNum 291 snap 93 seq_num 291) 2024-09-13T14:23:44.130 INFO:tasks.rados.rados.0.smithi151.stdout:715: expect (ObjNum 259 snap 81 seq_num 259) 2024-09-13T14:23:44.263 INFO:tasks.rados.rados.0.smithi151.stdout:716: finishing rollback tid 0 to smithi15186934-37 2024-09-13T14:23:44.263 INFO:tasks.rados.rados.0.smithi151.stdout:716: finishing rollback tid 1 to smithi15186934-37 2024-09-13T14:23:44.263 INFO:tasks.rados.rados.0.smithi151.stdout:716: finishing rollback tid 2 to smithi15186934-37 2024-09-13T14:23:44.263 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 37 v 359 (ObjNum 179 snap 47 seq_num 179) dirty exists 2024-09-13T14:23:44.263 INFO:tasks.rados.rados.0.smithi151.stdout:717: finishing write tid 1 to smithi15186934-39 2024-09-13T14:23:44.263 INFO:tasks.rados.rados.0.smithi151.stdout:717: finishing write tid 2 to smithi15186934-39 2024-09-13T14:23:44.263 INFO:tasks.rados.rados.0.smithi151.stdout:717: finishing write tid 3 to smithi15186934-39 2024-09-13T14:23:44.263 INFO:tasks.rados.rados.0.smithi151.stdout:717: finishing write tid 4 to smithi15186934-39 2024-09-13T14:23:44.264 INFO:tasks.rados.rados.0.smithi151.stdout:717: finishing write tid 5 to smithi15186934-39 2024-09-13T14:23:44.264 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 194 (ObjNum 299 snap 93 seq_num 299) dirty exists 2024-09-13T14:23:44.264 INFO:tasks.rados.rados.0.smithi151.stdout:717: left oid 39 (ObjNum 299 snap 93 seq_num 299) 2024-09-13T14:23:44.264 INFO:tasks.rados.rados.0.smithi151.stdout:714: expect (ObjNum 196 snap 55 seq_num 196) 2024-09-13T14:23:44.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:43 smithi184 ceph-mon[33394]: pgmap v36: 89 pgs: 89 active+clean; 224 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.0 MiB/s wr, 6 op/s 2024-09-13T14:23:44.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:43 smithi184 ceph-mon[33394]: from='client.15957 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:44.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:43 smithi184 ceph-mon[33394]: from='client.15963 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:44.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:43 smithi151 ceph-mon[41387]: pgmap v36: 89 pgs: 89 active+clean; 224 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.0 MiB/s wr, 6 op/s 2024-09-13T14:23:44.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:43 smithi151 ceph-mon[41387]: from='client.15957 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:44.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:43 smithi151 ceph-mon[41387]: from='client.15963 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:44.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:43 smithi151 ceph-mon[29807]: pgmap v36: 89 pgs: 89 active+clean; 224 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.0 MiB/s wr, 6 op/s 2024-09-13T14:23:44.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:43 smithi151 ceph-mon[29807]: from='client.15957 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:44.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:43 smithi151 ceph-mon[29807]: from='client.15963 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:44.401 INFO:tasks.rados.rados.0.smithi151.stdout:718: finishing write tid 1 to smithi15186934-22 2024-09-13T14:23:44.401 INFO:tasks.rados.rados.0.smithi151.stdout:718: finishing write tid 2 to smithi15186934-22 2024-09-13T14:23:44.401 INFO:tasks.rados.rados.0.smithi151.stdout:718: finishing write tid 3 to smithi15186934-22 2024-09-13T14:23:44.401 INFO:tasks.rados.rados.0.smithi151.stdout:718: finishing write tid 4 to smithi15186934-22 2024-09-13T14:23:44.401 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 22 v 162 (ObjNum 300 snap 93 seq_num 300) dirty exists 2024-09-13T14:23:44.401 INFO:tasks.rados.rados.0.smithi151.stdout:718: left oid 22 (ObjNum 300 snap 93 seq_num 300) 2024-09-13T14:23:44.401 INFO:tasks.rados.rados.0.smithi151.stdout:719: finishing write tid 1 to smithi15186934-16 2024-09-13T14:23:44.401 INFO:tasks.rados.rados.0.smithi151.stdout:719: finishing write tid 2 to smithi15186934-16 2024-09-13T14:23:44.402 INFO:tasks.rados.rados.0.smithi151.stdout:719: finishing write tid 3 to smithi15186934-16 2024-09-13T14:23:44.402 INFO:tasks.rados.rados.0.smithi151.stdout:719: finishing write tid 4 to smithi15186934-16 2024-09-13T14:23:44.402 INFO:tasks.rados.rados.0.smithi151.stdout:719: finishing write tid 5 to smithi15186934-16 2024-09-13T14:23:44.402 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 16 v 183 (ObjNum 301 snap 93 seq_num 301) dirty exists 2024-09-13T14:23:44.402 INFO:tasks.rados.rados.0.smithi151.stdout:719: left oid 16 (ObjNum 301 snap 93 seq_num 301) 2024-09-13T14:23:44.402 INFO:tasks.rados.rados.0.smithi151.stdout:720: finishing write tid 1 to smithi15186934-38 2024-09-13T14:23:44.402 INFO:tasks.rados.rados.0.smithi151.stdout:720: finishing write tid 2 to smithi15186934-38 2024-09-13T14:23:44.402 INFO:tasks.rados.rados.0.smithi151.stdout:720: finishing write tid 3 to smithi15186934-38 2024-09-13T14:23:44.402 INFO:tasks.rados.rados.0.smithi151.stdout:720: finishing write tid 4 to smithi15186934-38 2024-09-13T14:23:44.402 INFO:tasks.rados.rados.0.smithi151.stdout:720: finishing write tid 5 to smithi15186934-38 2024-09-13T14:23:44.402 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 199 (ObjNum 302 snap 93 seq_num 302) dirty exists 2024-09-13T14:23:44.402 INFO:tasks.rados.rados.0.smithi151.stdout:720: left oid 38 (ObjNum 302 snap 93 seq_num 302) 2024-09-13T14:23:44.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:23:44] "GET /metrics HTTP/1.1" 200 36408 "" "Prometheus/2.51.0" 2024-09-13T14:23:44.973 INFO:tasks.rados.rados.0.smithi151.stdout:710: done (9 left) 2024-09-13T14:23:44.973 INFO:tasks.rados.rados.0.smithi151.stdout:711: done (8 left) 2024-09-13T14:23:44.973 INFO:tasks.rados.rados.0.smithi151.stdout:714: done (7 left) 2024-09-13T14:23:44.973 INFO:tasks.rados.rados.0.smithi151.stdout:715: done (6 left) 2024-09-13T14:23:44.973 INFO:tasks.rados.rados.0.smithi151.stdout:716: done (5 left) 2024-09-13T14:23:44.973 INFO:tasks.rados.rados.0.smithi151.stdout:717: done (4 left) 2024-09-13T14:23:44.973 INFO:tasks.rados.rados.0.smithi151.stdout:718: done (3 left) 2024-09-13T14:23:44.973 INFO:tasks.rados.rados.0.smithi151.stdout:719: done (2 left) 2024-09-13T14:23:44.973 INFO:tasks.rados.rados.0.smithi151.stdout:720: done (1 left) 2024-09-13T14:23:44.974 INFO:tasks.rados.rados.0.smithi151.stdout:721: done (0 left) 2024-09-13T14:23:44.974 INFO:tasks.rados.rados.0.smithi151.stdout:722: snap_remove snap 63 2024-09-13T14:23:45.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:44 smithi184 ceph-mon[33394]: osdmap e244: 8 total, 8 up, 8 in 2024-09-13T14:23:45.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:44 smithi151 ceph-mon[29807]: osdmap e244: 8 total, 8 up, 8 in 2024-09-13T14:23:45.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:44 smithi151 ceph-mon[41387]: osdmap e244: 8 total, 8 up, 8 in 2024-09-13T14:23:45.985 INFO:tasks.rados.rados.0.smithi151.stdout:722: done (0 left) 2024-09-13T14:23:45.985 INFO:tasks.rados.rados.0.smithi151.stdout:723: read oid 6 snap -1 2024-09-13T14:23:45.986 INFO:tasks.rados.rados.0.smithi151.stdout:723: expect (ObjNum 294 snap 93 seq_num 294) 2024-09-13T14:23:45.986 INFO:tasks.rados.rados.0.smithi151.stdout:724: delete oid 29 current snap is 94 2024-09-13T14:23:45.994 INFO:tasks.rados.rados.0.smithi151.stdout:723: expect (ObjNum 294 snap 93 seq_num 294) 2024-09-13T14:23:46.008 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:23:46.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:45 smithi184 ceph-mon[33394]: pgmap v39: 89 pgs: 89 active+clean; 224 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.0 MiB/s wr, 6 op/s 2024-09-13T14:23:46.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:45 smithi184 ceph-mon[33394]: osdmap e245: 8 total, 8 up, 8 in 2024-09-13T14:23:46.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:45 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:46.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:45 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:46.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:45 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/920889110' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:46.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:45 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:46.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:45 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:46.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:45 smithi184 ceph-mon[33394]: Standby manager daemon x restarted 2024-09-13T14:23:46.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:45 smithi184 ceph-mon[33394]: Standby manager daemon x started 2024-09-13T14:23:46.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:45 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.184:0/665753295' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:23:46.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:45 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.184:0/665753295' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:23:46.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:45 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.184:0/665753295' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:23:46.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:45 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.184:0/665753295' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:23:46.105 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:45 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:45.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:23:46.105 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:45 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:45.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:23:46.105 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:45 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[82046]: [13/Sep/2024:14:23:45] ENGINE Bus STARTING 2024-09-13T14:23:46.105 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:45 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[82046]: CherryPy Checker: 2024-09-13T14:23:46.105 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:45 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[82046]: The Application mounted at '' has an empty config. 2024-09-13T14:23:46.105 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:45 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[82046]: 2024-09-13T14:23:46.106 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:45 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[82046]: [13/Sep/2024:14:23:45] ENGINE Serving on http://:::9283 2024-09-13T14:23:46.106 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:45 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[82046]: [13/Sep/2024:14:23:45] ENGINE Bus STARTED 2024-09-13T14:23:46.170 INFO:tasks.rados.rados.0.smithi151.stdout:723: done (1 left) 2024-09-13T14:23:46.170 INFO:tasks.rados.rados.0.smithi151.stdout:724: done (0 left) 2024-09-13T14:23:46.170 INFO:tasks.rados.rados.0.smithi151.stdout:725: write oid 32 current snap is 94 2024-09-13T14:23:46.170 INFO:tasks.rados.rados.0.smithi151.stdout:725: seq_num 304 ranges {709328=613463,1989820=645099} 2024-09-13T14:23:46.176 INFO:tasks.rados.rados.0.smithi151.stdout:725: writing smithi15186934-32 from 709328 to 1322791 tid 1 2024-09-13T14:23:46.184 INFO:tasks.rados.rados.0.smithi151.stdout:725: writing smithi15186934-32 from 1989820 to 2634919 tid 2 2024-09-13T14:23:46.184 INFO:tasks.rados.rados.0.smithi151.stdout:725: finishing write tid 1 to smithi15186934-32 2024-09-13T14:23:46.184 INFO:tasks.rados.rados.0.smithi151.stdout:726: read oid 36 snap -1 2024-09-13T14:23:46.184 INFO:tasks.rados.rados.0.smithi151.stdout:726: expect deleted 2024-09-13T14:23:46.185 INFO:tasks.rados.rados.0.smithi151.stdout:727: rollback oid 30 current snap is 94 2024-09-13T14:23:46.185 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 30 to 84 2024-09-13T14:23:46.185 INFO:tasks.rados.rados.0.smithi151.stdout:728: write oid 31 current snap is 94 2024-09-13T14:23:46.185 INFO:tasks.rados.rados.0.smithi151.stdout:728: seq_num 305 ranges {726135=405065,1764655=788757,3026248=99887} 2024-09-13T14:23:46.190 INFO:tasks.rados.rados.0.smithi151.stdout:728: writing smithi15186934-31 from 726135 to 1131200 tid 1 2024-09-13T14:23:46.198 INFO:tasks.rados.rados.0.smithi151.stdout:728: writing smithi15186934-31 from 1764655 to 2553412 tid 2 2024-09-13T14:23:46.201 INFO:tasks.rados.rados.0.smithi151.stdout:728: writing smithi15186934-31 from 3026248 to 3126135 tid 3 2024-09-13T14:23:46.201 INFO:tasks.rados.rados.0.smithi151.stdout:729: write oid 33 current snap is 94 2024-09-13T14:23:46.201 INFO:tasks.rados.rados.0.smithi151.stdout:725: finishing write tid 2 to smithi15186934-32 2024-09-13T14:23:46.202 INFO:tasks.rados.rados.0.smithi151.stdout:725: finishing write tid 3 to smithi15186934-32 2024-09-13T14:23:46.202 INFO:tasks.rados.rados.0.smithi151.stdout:729: seq_num 306 ranges {742942=480314,1823137=648768} 2024-09-13T14:23:46.208 INFO:tasks.rados.rados.0.smithi151.stdout:729: writing smithi15186934-33 from 742942 to 1223256 tid 1 2024-09-13T14:23:46.215 INFO:tasks.rados.rados.0.smithi151.stdout:729: writing smithi15186934-33 from 1823137 to 2471905 tid 2 2024-09-13T14:23:46.215 INFO:tasks.rados.rados.0.smithi151.stdout:726: done (4 left) 2024-09-13T14:23:46.215 INFO:tasks.rados.rados.0.smithi151.stdout:730: delete oid 19 current snap is 94 2024-09-13T14:23:46.216 INFO:tasks.rados.rados.0.smithi151.stdout:725: finishing write tid 4 to smithi15186934-32 2024-09-13T14:23:46.216 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 32 v 326 (ObjNum 304 snap 94 seq_num 304) dirty exists 2024-09-13T14:23:46.216 INFO:tasks.rados.rados.0.smithi151.stdout:725: left oid 32 (ObjNum 304 snap 94 seq_num 304) 2024-09-13T14:23:46.216 INFO:tasks.rados.rados.0.smithi151.stdout:727: finishing rollback tid 0 to smithi15186934-30 2024-09-13T14:23:46.216 INFO:tasks.rados.rados.0.smithi151.stdout:727: finishing rollback tid 1 to smithi15186934-30 2024-09-13T14:23:46.216 INFO:tasks.rados.rados.0.smithi151.stdout:727: finishing rollback tid 2 to smithi15186934-30 2024-09-13T14:23:46.216 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 30 v 319 (ObjNum 254 snap 79 seq_num 254) dirty exists 2024-09-13T14:23:46.216 INFO:tasks.rados.rados.0.smithi151.stdout:728: finishing write tid 1 to smithi15186934-31 2024-09-13T14:23:46.216 INFO:tasks.rados.rados.0.smithi151.stdout:728: finishing write tid 2 to smithi15186934-31 2024-09-13T14:23:46.216 INFO:tasks.rados.rados.0.smithi151.stdout:728: finishing write tid 3 to smithi15186934-31 2024-09-13T14:23:46.216 INFO:tasks.rados.rados.0.smithi151.stdout:728: finishing write tid 4 to smithi15186934-31 2024-09-13T14:23:46.216 INFO:tasks.rados.rados.0.smithi151.stdout:728: finishing write tid 5 to smithi15186934-31 2024-09-13T14:23:46.216 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 31 v 369 (ObjNum 305 snap 94 seq_num 305) dirty exists 2024-09-13T14:23:46.217 INFO:tasks.rados.rados.0.smithi151.stdout:728: left oid 31 (ObjNum 305 snap 94 seq_num 305) 2024-09-13T14:23:46.217 INFO:tasks.rados.rados.0.smithi151.stdout:729: finishing write tid 1 to smithi15186934-33 2024-09-13T14:23:46.217 INFO:tasks.rados.rados.0.smithi151.stdout:725: done (4 left) 2024-09-13T14:23:46.217 INFO:tasks.rados.rados.0.smithi151.stdout:727: done (3 left) 2024-09-13T14:23:46.217 INFO:tasks.rados.rados.0.smithi151.stdout:728: done (2 left) 2024-09-13T14:23:46.217 INFO:tasks.rados.rados.0.smithi151.stdout:730: done (1 left) 2024-09-13T14:23:46.217 INFO:tasks.rados.rados.0.smithi151.stdout:731: snap_remove snap 93 2024-09-13T14:23:46.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:45 smithi151 ceph-mon[29807]: pgmap v39: 89 pgs: 89 active+clean; 224 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.0 MiB/s wr, 6 op/s 2024-09-13T14:23:46.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:45 smithi151 ceph-mon[29807]: osdmap e245: 8 total, 8 up, 8 in 2024-09-13T14:23:46.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:45 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:46.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:45 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:46.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:45 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/920889110' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:46.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:45 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:46.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:45 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:46.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:45 smithi151 ceph-mon[29807]: Standby manager daemon x restarted 2024-09-13T14:23:46.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:45 smithi151 ceph-mon[29807]: Standby manager daemon x started 2024-09-13T14:23:46.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:45 smithi151 ceph-mon[29807]: from='mgr.? 172.21.15.184:0/665753295' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:23:46.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:45 smithi151 ceph-mon[29807]: from='mgr.? 172.21.15.184:0/665753295' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:23:46.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:45 smithi151 ceph-mon[29807]: from='mgr.? 172.21.15.184:0/665753295' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:23:46.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:45 smithi151 ceph-mon[29807]: from='mgr.? 172.21.15.184:0/665753295' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:23:46.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:45 smithi151 ceph-mon[41387]: pgmap v39: 89 pgs: 89 active+clean; 224 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.0 MiB/s wr, 6 op/s 2024-09-13T14:23:46.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:45 smithi151 ceph-mon[41387]: osdmap e245: 8 total, 8 up, 8 in 2024-09-13T14:23:46.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:45 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:46.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:45 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:46.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:45 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/920889110' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:46.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:45 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:46.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:45 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:46.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:45 smithi151 ceph-mon[41387]: Standby manager daemon x restarted 2024-09-13T14:23:46.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:45 smithi151 ceph-mon[41387]: Standby manager daemon x started 2024-09-13T14:23:46.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:45 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.184:0/665753295' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:23:46.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:45 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.184:0/665753295' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:23:46.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:45 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.184:0/665753295' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:23:46.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:45 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.184:0/665753295' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:23:46.990 INFO:tasks.rados.rados.0.smithi151.stdout:731: done (1 left) 2024-09-13T14:23:46.991 INFO:tasks.rados.rados.0.smithi151.stdout:732: write oid 47 current snap is 94 2024-09-13T14:23:46.991 INFO:tasks.rados.rados.0.smithi151.stdout:729: finishing write tid 2 to smithi15186934-33 2024-09-13T14:23:46.991 INFO:tasks.rados.rados.0.smithi151.stdout:729: finishing write tid 3 to smithi15186934-33 2024-09-13T14:23:46.991 INFO:tasks.rados.rados.0.smithi151.stdout:729: finishing write tid 4 to smithi15186934-33 2024-09-13T14:23:46.991 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 33 v 213 (ObjNum 306 snap 94 seq_num 306) dirty exists 2024-09-13T14:23:46.991 INFO:tasks.rados.rados.0.smithi151.stdout:729: left oid 33 (ObjNum 306 snap 94 seq_num 306) 2024-09-13T14:23:46.991 INFO:tasks.rados.rados.0.smithi151.stdout:732: seq_num 308 ranges {776556=630812,1823748=652437,3108517=68039} 2024-09-13T14:23:46.998 INFO:tasks.rados.rados.0.smithi151.stdout:732: writing smithi15186934-47 from 776556 to 1407368 tid 1 2024-09-13T14:23:47.004 INFO:tasks.rados.rados.0.smithi151.stdout:732: writing smithi15186934-47 from 1823748 to 2476185 tid 2 2024-09-13T14:23:47.008 INFO:tasks.rados.rados.0.smithi151.stdout:732: writing smithi15186934-47 from 3108517 to 3176556 tid 3 2024-09-13T14:23:47.008 INFO:tasks.rados.rados.0.smithi151.stdout:729: done (1 left) 2024-09-13T14:23:47.009 INFO:tasks.rados.rados.0.smithi151.stdout:733: delete oid 19 current snap is 94 2024-09-13T14:23:47.009 INFO:tasks.rados.rados.0.smithi151.stdout:732: finishing write tid 1 to smithi15186934-47 2024-09-13T14:23:47.009 INFO:tasks.rados.rados.0.smithi151.stdout:732: finishing write tid 2 to smithi15186934-47 2024-09-13T14:23:47.010 INFO:tasks.rados.rados.0.smithi151.stdout:733: done (1 left) 2024-09-13T14:23:47.010 INFO:tasks.rados.rados.0.smithi151.stdout:734: write oid 34 current snap is 94 2024-09-13T14:23:47.010 INFO:tasks.rados.rados.0.smithi151.stdout:734: seq_num 310 ranges {410170=781310,1940712=656106,3163363=46807} 2024-09-13T14:23:47.016 INFO:tasks.rados.rados.0.smithi151.stdout:734: writing smithi15186934-34 from 410170 to 1191480 tid 1 2024-09-13T14:23:47.023 INFO:tasks.rados.rados.0.smithi151.stdout:734: writing smithi15186934-34 from 1940712 to 2596818 tid 2 2024-09-13T14:23:47.027 INFO:tasks.rados.rados.0.smithi151.stdout:734: writing smithi15186934-34 from 3163363 to 3210170 tid 3 2024-09-13T14:23:47.027 INFO:tasks.rados.rados.0.smithi151.stdout:735: write oid 33 current snap is 94 2024-09-13T14:23:47.027 INFO:tasks.rados.rados.0.smithi151.stdout:732: finishing write tid 3 to smithi15186934-47 2024-09-13T14:23:47.027 INFO:tasks.rados.rados.0.smithi151.stdout:735: seq_num 311 ranges {426977=456559,1482841=799764,2958080=268897} 2024-09-13T14:23:47.032 INFO:tasks.rados.rados.0.smithi151.stdout:735: writing smithi15186934-33 from 426977 to 883536 tid 1 2024-09-13T14:23:47.039 INFO:tasks.rados.rados.0.smithi151.stdout:735: writing smithi15186934-33 from 1482841 to 2282605 tid 2 2024-09-13T14:23:47.046 INFO:tasks.rados.rados.0.smithi151.stdout:735: writing smithi15186934-33 from 2958080 to 3226977 tid 3 2024-09-13T14:23:47.046 INFO:tasks.rados.rados.0.smithi151.stdout:736: rollback oid 37 current snap is 94 2024-09-13T14:23:47.046 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 37 to 85 2024-09-13T14:23:47.046 INFO:tasks.rados.rados.0.smithi151.stdout:732: finishing write tid 4 to smithi15186934-47 2024-09-13T14:23:47.046 INFO:tasks.rados.rados.0.smithi151.stdout:732: finishing write tid 5 to smithi15186934-47 2024-09-13T14:23:47.046 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 370 (ObjNum 308 snap 94 seq_num 308) dirty exists 2024-09-13T14:23:47.046 INFO:tasks.rados.rados.0.smithi151.stdout:732: left oid 47 (ObjNum 308 snap 94 seq_num 308) 2024-09-13T14:23:47.046 INFO:tasks.rados.rados.0.smithi151.stdout:734: finishing write tid 1 to smithi15186934-34 2024-09-13T14:23:47.047 INFO:tasks.rados.rados.0.smithi151.stdout:734: finishing write tid 2 to smithi15186934-34 2024-09-13T14:23:47.047 INFO:tasks.rados.rados.0.smithi151.stdout:732: done (3 left) 2024-09-13T14:23:47.047 INFO:tasks.rados.rados.0.smithi151.stdout:737: rollback oid 46 current snap is 94 2024-09-13T14:23:47.047 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 46 to 90 2024-09-13T14:23:47.047 INFO:tasks.rados.rados.0.smithi151.stdout:734: finishing write tid 3 to smithi15186934-34 2024-09-13T14:23:47.047 INFO:tasks.rados.rados.0.smithi151.stdout:734: finishing write tid 4 to smithi15186934-34 2024-09-13T14:23:47.047 INFO:tasks.rados.rados.0.smithi151.stdout:734: finishing write tid 5 to smithi15186934-34 2024-09-13T14:23:47.047 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 326 (ObjNum 310 snap 94 seq_num 310) dirty exists 2024-09-13T14:23:47.047 INFO:tasks.rados.rados.0.smithi151.stdout:734: left oid 34 (ObjNum 310 snap 94 seq_num 310) 2024-09-13T14:23:47.047 INFO:tasks.rados.rados.0.smithi151.stdout:735: finishing write tid 1 to smithi15186934-33 2024-09-13T14:23:47.047 INFO:tasks.rados.rados.0.smithi151.stdout:735: finishing write tid 2 to smithi15186934-33 2024-09-13T14:23:47.047 INFO:tasks.rados.rados.0.smithi151.stdout:734: done (3 left) 2024-09-13T14:23:47.047 INFO:tasks.rados.rados.0.smithi151.stdout:738: read oid 17 snap 89 2024-09-13T14:23:47.047 INFO:tasks.rados.rados.0.smithi151.stdout:738: expect (ObjNum 253 snap 79 seq_num 253) 2024-09-13T14:23:47.048 INFO:tasks.rados.rados.0.smithi151.stdout:739: read oid 22 snap -1 2024-09-13T14:23:47.048 INFO:tasks.rados.rados.0.smithi151.stdout:739: expect (ObjNum 300 snap 93 seq_num 300) 2024-09-13T14:23:47.048 INFO:tasks.rados.rados.0.smithi151.stdout:740: snap_create 2024-09-13T14:23:47.049 INFO:tasks.rados.rados.0.smithi151.stdout:735: finishing write tid 3 to smithi15186934-33 2024-09-13T14:23:47.049 INFO:tasks.rados.rados.0.smithi151.stdout:735: finishing write tid 4 to smithi15186934-33 2024-09-13T14:23:47.049 INFO:tasks.rados.rados.0.smithi151.stdout:736: finishing rollback tid 0 to smithi15186934-37 2024-09-13T14:23:47.049 INFO:tasks.rados.rados.0.smithi151.stdout:735: finishing write tid 5 to smithi15186934-33 2024-09-13T14:23:47.049 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 33 v 219 (ObjNum 311 snap 94 seq_num 311) dirty exists 2024-09-13T14:23:47.049 INFO:tasks.rados.rados.0.smithi151.stdout:735: left oid 33 (ObjNum 311 snap 94 seq_num 311) 2024-09-13T14:23:47.049 INFO:tasks.rados.rados.0.smithi151.stdout:736: finishing rollback tid 1 to smithi15186934-37 2024-09-13T14:23:47.050 INFO:tasks.rados.rados.0.smithi151.stdout:736: finishing rollback tid 2 to smithi15186934-37 2024-09-13T14:23:47.050 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 37 v 373 (ObjNum 179 snap 47 seq_num 179) dirty exists 2024-09-13T14:23:47.050 INFO:tasks.rados.rados.0.smithi151.stdout:737: finishing rollback tid 0 to smithi15186934-46 2024-09-13T14:23:47.050 INFO:tasks.rados.rados.0.smithi151.stdout:737: finishing rollback tid 1 to smithi15186934-46 2024-09-13T14:23:47.050 INFO:tasks.rados.rados.0.smithi151.stdout:737: finishing rollback tid 2 to smithi15186934-46 2024-09-13T14:23:47.050 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 46 v 241 (ObjNum 261 snap 82 seq_num 261) dirty exists 2024-09-13T14:23:47.055 INFO:tasks.rados.rados.0.smithi151.stdout:739: expect (ObjNum 300 snap 93 seq_num 300) 2024-09-13T14:23:47.193 INFO:tasks.rados.rados.0.smithi151.stdout:738: expect (ObjNum 253 snap 79 seq_num 253) 2024-09-13T14:23:47.241 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:46 smithi184 ceph-mon[33394]: osdmap e246: 8 total, 8 up, 8 in 2024-09-13T14:23:47.241 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:46 smithi184 ceph-mon[33394]: from='client.15975 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:47.242 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:46 smithi184 ceph-mon[33394]: from='client.15981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:47.242 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:46 smithi184 ceph-mon[33394]: mgrmap e32: y(active, since 25s), standbys: x 2024-09-13T14:23:47.242 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:46 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:46.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi184\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:23:47.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:46 smithi151 ceph-mon[29807]: osdmap e246: 8 total, 8 up, 8 in 2024-09-13T14:23:47.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:46 smithi151 ceph-mon[29807]: from='client.15975 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:47.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:46 smithi151 ceph-mon[29807]: from='client.15981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:47.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:46 smithi151 ceph-mon[29807]: mgrmap e32: y(active, since 25s), standbys: x 2024-09-13T14:23:47.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:46 smithi151 ceph-mon[41387]: osdmap e246: 8 total, 8 up, 8 in 2024-09-13T14:23:47.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:46 smithi151 ceph-mon[41387]: from='client.15975 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:47.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:46 smithi151 ceph-mon[41387]: from='client.15981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:47.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:46 smithi151 ceph-mon[41387]: mgrmap e32: y(active, since 25s), standbys: x 2024-09-13T14:23:47.997 INFO:tasks.rados.rados.0.smithi151.stdout:735: done (5 left) 2024-09-13T14:23:47.997 INFO:tasks.rados.rados.0.smithi151.stdout:736: done (4 left) 2024-09-13T14:23:47.997 INFO:tasks.rados.rados.0.smithi151.stdout:737: done (3 left) 2024-09-13T14:23:47.997 INFO:tasks.rados.rados.0.smithi151.stdout:738: done (2 left) 2024-09-13T14:23:47.997 INFO:tasks.rados.rados.0.smithi151.stdout:739: done (1 left) 2024-09-13T14:23:47.997 INFO:tasks.rados.rados.0.smithi151.stdout:740: done (0 left) 2024-09-13T14:23:47.997 INFO:tasks.rados.rados.0.smithi151.stdout:741: snap_create 2024-09-13T14:23:48.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:47 smithi184 ceph-mon[33394]: pgmap v42: 89 pgs: 89 active+clean; 224 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:23:48.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:47 smithi184 ceph-mon[33394]: osdmap e247: 8 total, 8 up, 8 in 2024-09-13T14:23:48.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:47 smithi151 ceph-mon[41387]: pgmap v42: 89 pgs: 89 active+clean; 224 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:23:48.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:47 smithi151 ceph-mon[41387]: osdmap e247: 8 total, 8 up, 8 in 2024-09-13T14:23:48.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:47 smithi151 ceph-mon[29807]: pgmap v42: 89 pgs: 89 active+clean; 224 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:23:48.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:47 smithi151 ceph-mon[29807]: osdmap e247: 8 total, 8 up, 8 in 2024-09-13T14:23:48.912 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:23:49.001 INFO:tasks.rados.rados.0.smithi151.stdout:741: done (0 left) 2024-09-13T14:23:49.002 INFO:tasks.rados.rados.0.smithi151.stdout:742: rollback oid 17 current snap is 96 2024-09-13T14:23:49.002 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 17 to 95 2024-09-13T14:23:49.002 INFO:tasks.rados.rados.0.smithi151.stdout:743: write oid 42 current snap is 96 2024-09-13T14:23:49.002 INFO:tasks.rados.rados.0.smithi151.stdout:743: seq_num 312 ranges {443784=648161,1657676=659775,2818209=425575} 2024-09-13T14:23:49.009 INFO:tasks.rados.rados.0.smithi151.stdout:743: writing smithi15186934-42 from 443784 to 1091945 tid 1 2024-09-13T14:23:49.018 INFO:tasks.rados.rados.0.smithi151.stdout:743: writing smithi15186934-42 from 1657676 to 2317451 tid 2 2024-09-13T14:23:49.025 INFO:tasks.rados.rados.0.smithi151.stdout:743: writing smithi15186934-42 from 2818209 to 3243784 tid 3 2024-09-13T14:23:49.025 INFO:tasks.rados.rados.0.smithi151.stdout:744: rollback oid 47 current snap is 96 2024-09-13T14:23:49.025 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 47 to 86 2024-09-13T14:23:49.025 INFO:tasks.rados.rados.0.smithi151.stdout:742: finishing rollback tid 0 to smithi15186934-17 2024-09-13T14:23:49.025 INFO:tasks.rados.rados.0.smithi151.stdout:742: finishing rollback tid 1 to smithi15186934-17 2024-09-13T14:23:49.025 INFO:tasks.rados.rados.0.smithi151.stdout:742: finishing rollback tid 2 to smithi15186934-17 2024-09-13T14:23:49.025 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 475 (ObjNum 253 snap 79 seq_num 253) dirty exists 2024-09-13T14:23:49.025 INFO:tasks.rados.rados.0.smithi151.stdout:743: finishing write tid 1 to smithi15186934-42 2024-09-13T14:23:49.025 INFO:tasks.rados.rados.0.smithi151.stdout:743: finishing write tid 2 to smithi15186934-42 2024-09-13T14:23:49.025 INFO:tasks.rados.rados.0.smithi151.stdout:742: done (2 left) 2024-09-13T14:23:49.025 INFO:tasks.rados.rados.0.smithi151.stdout:745: write oid 4 current snap is 96 2024-09-13T14:23:49.025 INFO:tasks.rados.rados.0.smithi151.stdout:745: seq_num 313 ranges {460591=723410,1716158=403433,2729279=440140} 2024-09-13T14:23:49.033 INFO:tasks.rados.rados.0.smithi151.stdout:745: writing smithi15186934-4 from 460591 to 1184001 tid 1 2024-09-13T14:23:49.039 INFO:tasks.rados.rados.0.smithi151.stdout:745: writing smithi15186934-4 from 1716158 to 2119591 tid 2 2024-09-13T14:23:49.047 INFO:tasks.rados.rados.0.smithi151.stdout:745: writing smithi15186934-4 from 2729279 to 3169419 tid 3 2024-09-13T14:23:49.047 INFO:tasks.rados.rados.0.smithi151.stdout:746: write oid 36 current snap is 96 2024-09-13T14:23:49.047 INFO:tasks.rados.rados.0.smithi151.stdout:744: finishing rollback tid 0 to smithi15186934-47 2024-09-13T14:23:49.047 INFO:tasks.rados.rados.0.smithi151.stdout:744: finishing rollback tid 1 to smithi15186934-47 2024-09-13T14:23:49.047 INFO:tasks.rados.rados.0.smithi151.stdout:746: seq_num 314 ranges {477398=798659,1774640=663444,2873055=404343} 2024-09-13T14:23:49.056 INFO:tasks.rados.rados.0.smithi151.stdout:746: writing smithi15186934-36 from 477398 to 1276057 tid 1 2024-09-13T14:23:49.064 INFO:tasks.rados.rados.0.smithi151.stdout:746: writing smithi15186934-36 from 1774640 to 2438084 tid 2 2024-09-13T14:23:49.071 INFO:tasks.rados.rados.0.smithi151.stdout:746: writing smithi15186934-36 from 2873055 to 3277398 tid 3 2024-09-13T14:23:49.071 INFO:tasks.rados.rados.0.smithi151.stdout:747: read oid 38 snap -1 2024-09-13T14:23:49.071 INFO:tasks.rados.rados.0.smithi151.stdout:747: expect (ObjNum 302 snap 93 seq_num 302) 2024-09-13T14:23:49.071 INFO:tasks.rados.rados.0.smithi151.stdout:748: delete oid 31 current snap is 96 2024-09-13T14:23:49.071 INFO:tasks.rados.rados.0.smithi151.stdout:743: finishing write tid 3 to smithi15186934-42 2024-09-13T14:23:49.071 INFO:tasks.rados.rados.0.smithi151.stdout:744: finishing rollback tid 2 to smithi15186934-47 2024-09-13T14:23:49.071 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 377 (ObjNum 155 snap 38 seq_num 155) dirty exists 2024-09-13T14:23:49.072 INFO:tasks.rados.rados.0.smithi151.stdout:743: finishing write tid 4 to smithi15186934-42 2024-09-13T14:23:49.072 INFO:tasks.rados.rados.0.smithi151.stdout:743: finishing write tid 5 to smithi15186934-42 2024-09-13T14:23:49.072 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 376 (ObjNum 312 snap 96 seq_num 312) dirty exists 2024-09-13T14:23:49.072 INFO:tasks.rados.rados.0.smithi151.stdout:743: left oid 42 (ObjNum 312 snap 96 seq_num 312) 2024-09-13T14:23:49.072 INFO:tasks.rados.rados.0.smithi151.stdout:745: finishing write tid 1 to smithi15186934-4 2024-09-13T14:23:49.072 INFO:tasks.rados.rados.0.smithi151.stdout:745: finishing write tid 2 to smithi15186934-4 2024-09-13T14:23:49.072 INFO:tasks.rados.rados.0.smithi151.stdout:745: finishing write tid 3 to smithi15186934-4 2024-09-13T14:23:49.072 INFO:tasks.rados.rados.0.smithi151.stdout:745: finishing write tid 4 to smithi15186934-4 2024-09-13T14:23:49.072 INFO:tasks.rados.rados.0.smithi151.stdout:745: finishing write tid 5 to smithi15186934-4 2024-09-13T14:23:49.072 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 464 (ObjNum 313 snap 96 seq_num 313) dirty exists 2024-09-13T14:23:49.072 INFO:tasks.rados.rados.0.smithi151.stdout:745: left oid 4 (ObjNum 313 snap 96 seq_num 313) 2024-09-13T14:23:49.072 INFO:tasks.rados.rados.0.smithi151.stdout:746: finishing write tid 1 to smithi15186934-36 2024-09-13T14:23:49.072 INFO:tasks.rados.rados.0.smithi151.stdout:746: finishing write tid 2 to smithi15186934-36 2024-09-13T14:23:49.073 INFO:tasks.rados.rados.0.smithi151.stdout:743: done (5 left) 2024-09-13T14:23:49.073 INFO:tasks.rados.rados.0.smithi151.stdout:744: done (4 left) 2024-09-13T14:23:49.073 INFO:tasks.rados.rados.0.smithi151.stdout:745: done (3 left) 2024-09-13T14:23:49.074 INFO:tasks.rados.rados.0.smithi151.stdout:748: done (2 left) 2024-09-13T14:23:49.074 INFO:tasks.rados.rados.0.smithi151.stdout:749: snap_create 2024-09-13T14:23:49.074 INFO:tasks.rados.rados.0.smithi151.stdout:746: finishing write tid 3 to smithi15186934-36 2024-09-13T14:23:49.074 INFO:tasks.rados.rados.0.smithi151.stdout:746: finishing write tid 4 to smithi15186934-36 2024-09-13T14:23:49.074 INFO:tasks.rados.rados.0.smithi151.stdout:746: finishing write tid 5 to smithi15186934-36 2024-09-13T14:23:49.074 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 36 v 150 (ObjNum 314 snap 96 seq_num 314) dirty exists 2024-09-13T14:23:49.074 INFO:tasks.rados.rados.0.smithi151.stdout:746: left oid 36 (ObjNum 314 snap 96 seq_num 314) 2024-09-13T14:23:49.081 INFO:tasks.rados.rados.0.smithi151.stdout:747: expect (ObjNum 302 snap 93 seq_num 302) 2024-09-13T14:23:49.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:48 smithi184 ceph-mon[33394]: osdmap e248: 8 total, 8 up, 8 in 2024-09-13T14:23:49.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:48 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/1663689153' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:49.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:48 smithi151 ceph-mon[41387]: osdmap e248: 8 total, 8 up, 8 in 2024-09-13T14:23:49.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:48 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/1663689153' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:49.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:48 smithi151 ceph-mon[29807]: osdmap e248: 8 total, 8 up, 8 in 2024-09-13T14:23:49.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:48 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/1663689153' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:50.041 INFO:tasks.rados.rados.0.smithi151.stdout:746: done (2 left) 2024-09-13T14:23:50.041 INFO:tasks.rados.rados.0.smithi151.stdout:747: done (1 left) 2024-09-13T14:23:50.042 INFO:tasks.rados.rados.0.smithi151.stdout:749: done (0 left) 2024-09-13T14:23:50.042 INFO:tasks.rados.rados.0.smithi151.stdout:750: rollback oid 17 current snap is 97 2024-09-13T14:23:50.042 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 17 to 84 2024-09-13T14:23:50.042 INFO:tasks.rados.rados.0.smithi151.stdout:751: snap_remove snap 85 2024-09-13T14:23:50.252 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[29807]: pgmap v45: 89 pgs: 89 active+clean; 169 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 4.6 MiB/s wr, 37 op/s 2024-09-13T14:23:50.252 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[29807]: from='client.25898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:50.252 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[29807]: osdmap e249: 8 total, 8 up, 8 in 2024-09-13T14:23:50.252 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:50.252 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:50.252 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:50.252 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:23:50.252 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:50.252 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:50.253 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:23:50.253 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[29807]: Upgrade: Setting container_image for all mgr 2024-09-13T14:23:50.253 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:50.253 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-13T14:23:50.253 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-13T14:23:50.253 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-13T14:23:50.253 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-13T14:23:50.253 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-13T14:23:50.253 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-13T14:23:50.253 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[29807]: Upgrade: It appears safe to stop mon.a 2024-09-13T14:23:50.253 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[29807]: from='client.15999 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:50.254 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[41387]: pgmap v45: 89 pgs: 89 active+clean; 169 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 4.6 MiB/s wr, 37 op/s 2024-09-13T14:23:50.254 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[41387]: from='client.25898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:50.254 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[41387]: osdmap e249: 8 total, 8 up, 8 in 2024-09-13T14:23:50.254 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:50.254 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:50.254 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:50.254 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:23:50.254 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:50.254 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:50.254 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:23:50.254 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[41387]: Upgrade: Setting container_image for all mgr 2024-09-13T14:23:50.254 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:50.254 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-13T14:23:50.255 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-13T14:23:50.255 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:49 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-13T14:23:50.255 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:50 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-13T14:23:50.255 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:50 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-13T14:23:50.255 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:50 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-13T14:23:50.255 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:50 smithi151 ceph-mon[41387]: Upgrade: It appears safe to stop mon.a 2024-09-13T14:23:50.255 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:50 smithi151 ceph-mon[41387]: from='client.15999 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:50.354 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:23:50.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:49 smithi184 ceph-mon[33394]: pgmap v45: 89 pgs: 89 active+clean; 169 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 4.6 MiB/s wr, 37 op/s 2024-09-13T14:23:50.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:49 smithi184 ceph-mon[33394]: from='client.25898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:50.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:49 smithi184 ceph-mon[33394]: osdmap e249: 8 total, 8 up, 8 in 2024-09-13T14:23:50.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:49 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:50.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:49 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:50.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:49 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:50.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:49 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:23:50.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:49 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:50.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:50 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:50.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:50 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:23:50.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:50 smithi184 ceph-mon[33394]: Upgrade: Setting container_image for all mgr 2024-09-13T14:23:50.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:50 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:50.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:50 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-13T14:23:50.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:50 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-13T14:23:50.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:50 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-13T14:23:50.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:50 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-13T14:23:50.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:50 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-13T14:23:50.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:50 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-13T14:23:50.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:50 smithi184 ceph-mon[33394]: Upgrade: It appears safe to stop mon.a 2024-09-13T14:23:50.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:50 smithi184 ceph-mon[33394]: from='client.15999 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:51.044 INFO:tasks.rados.rados.0.smithi151.stdout:751: done (1 left) 2024-09-13T14:23:51.044 INFO:tasks.rados.rados.0.smithi151.stdout:752: rollback oid 3 current snap is 97 2024-09-13T14:23:51.044 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 3 to 84 2024-09-13T14:23:51.044 INFO:tasks.rados.rados.0.smithi151.stdout:753: delete oid 23 current snap is 97 2024-09-13T14:23:51.044 INFO:tasks.rados.rados.0.smithi151.stdout:750: finishing rollback tid 0 to smithi15186934-17 2024-09-13T14:23:51.044 INFO:tasks.rados.rados.0.smithi151.stdout:750: finishing rollback tid 1 to smithi15186934-17 2024-09-13T14:23:51.044 INFO:tasks.rados.rados.0.smithi151.stdout:750: finishing rollback tid 2 to smithi15186934-17 2024-09-13T14:23:51.044 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 479 (ObjNum 253 snap 79 seq_num 253) dirty exists 2024-09-13T14:23:51.045 INFO:tasks.rados.rados.0.smithi151.stdout:752: finishing rollback tid 1 to smithi15186934-3 2024-09-13T14:23:51.045 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 0 (ObjNum 161 snap 0 seq_num 1755290992) dirty dne 2024-09-13T14:23:51.051 INFO:tasks.rados.rados.0.smithi151.stdout:750: done (2 left) 2024-09-13T14:23:51.051 INFO:tasks.rados.rados.0.smithi151.stdout:752: done (1 left) 2024-09-13T14:23:51.051 INFO:tasks.rados.rados.0.smithi151.stdout:753: done (0 left) 2024-09-13T14:23:51.051 INFO:tasks.rados.rados.0.smithi151.stdout:754: delete oid 17 current snap is 97 2024-09-13T14:23:51.054 INFO:tasks.rados.rados.0.smithi151.stdout:754: done (0 left) 2024-09-13T14:23:51.054 INFO:tasks.rados.rados.0.smithi151.stdout:755: write oid 27 current snap is 97 2024-09-13T14:23:51.054 INFO:tasks.rados.rados.0.smithi151.stdout:755: seq_num 318 ranges {544626=699655,1892215=554429,3150041=194585} 2024-09-13T14:23:51.061 INFO:tasks.rados.rados.0.smithi151.stdout:755: writing smithi15186934-27 from 544626 to 1244281 tid 1 2024-09-13T14:23:51.067 INFO:tasks.rados.rados.0.smithi151.stdout:755: writing smithi15186934-27 from 1892215 to 2446644 tid 2 2024-09-13T14:23:51.072 INFO:tasks.rados.rados.0.smithi151.stdout:755: writing smithi15186934-27 from 3150041 to 3344626 tid 3 2024-09-13T14:23:51.073 INFO:tasks.rados.rados.0.smithi151.stdout:756: delete oid 39 current snap is 97 2024-09-13T14:23:51.073 INFO:tasks.rados.rados.0.smithi151.stdout:755: finishing write tid 1 to smithi15186934-27 2024-09-13T14:23:51.073 INFO:tasks.rados.rados.0.smithi151.stdout:755: finishing write tid 2 to smithi15186934-27 2024-09-13T14:23:51.074 INFO:tasks.rados.rados.0.smithi151.stdout:756: done (1 left) 2024-09-13T14:23:51.074 INFO:tasks.rados.rados.0.smithi151.stdout:757: snap_create 2024-09-13T14:23:51.075 INFO:tasks.rados.rados.0.smithi151.stdout:755: finishing write tid 3 to smithi15186934-27 2024-09-13T14:23:51.075 INFO:tasks.rados.rados.0.smithi151.stdout:755: finishing write tid 4 to smithi15186934-27 2024-09-13T14:23:51.075 INFO:tasks.rados.rados.0.smithi151.stdout:755: finishing write tid 5 to smithi15186934-27 2024-09-13T14:23:51.075 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 27 v 192 (ObjNum 318 snap 97 seq_num 318) dirty exists 2024-09-13T14:23:51.075 INFO:tasks.rados.rados.0.smithi151.stdout:755: left oid 27 (ObjNum 318 snap 97 seq_num 318) 2024-09-13T14:23:51.349 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:51 smithi151 ceph-mon[29807]: osdmap e250: 8 total, 8 up, 8 in 2024-09-13T14:23:51.349 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:51 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:51.350 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:51 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:23:51.350 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:51 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:23:51.350 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:51 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:51.350 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:51 smithi151 ceph-mon[41387]: osdmap e250: 8 total, 8 up, 8 in 2024-09-13T14:23:51.350 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:51 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:51.350 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:51 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:23:51.350 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:51 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:23:51.350 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:51 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:51.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:51 smithi184 ceph-mon[33394]: osdmap e250: 8 total, 8 up, 8 in 2024-09-13T14:23:51.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:51 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' 2024-09-13T14:23:51.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:51 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:23:51.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:51 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:23:51.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:51 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:51.867 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:23:52.060 INFO:tasks.rados.rados.0.smithi151.stdout:755: done (1 left) 2024-09-13T14:23:52.060 INFO:tasks.rados.rados.0.smithi151.stdout:757: done (0 left) 2024-09-13T14:23:52.060 INFO:tasks.rados.rados.0.smithi151.stdout:758: delete oid 21 current snap is 98 2024-09-13T14:23:52.064 INFO:tasks.rados.rados.0.smithi151.stdout:758: done (0 left) 2024-09-13T14:23:52.064 INFO:tasks.rados.rados.0.smithi151.stdout:759: snap_remove snap 91 2024-09-13T14:23:52.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:52 smithi151 ceph-mon[29807]: Upgrade: Updating mon.a 2024-09-13T14:23:52.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:52 smithi151 ceph-mon[29807]: Deploying daemon mon.a on smithi151 2024-09-13T14:23:52.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:52 smithi151 ceph-mon[29807]: pgmap v48: 89 pgs: 89 active+clean; 169 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 4.6 MiB/s wr, 37 op/s 2024-09-13T14:23:52.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:52 smithi151 ceph-mon[29807]: osdmap e251: 8 total, 8 up, 8 in 2024-09-13T14:23:52.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:52 smithi151 ceph-mon[29807]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:23:52.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:52 smithi151 ceph-mon[29807]: from='client.? 172.21.15.151:0/2139081028' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:52.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:52 smithi151 ceph-mon[41387]: Upgrade: Updating mon.a 2024-09-13T14:23:52.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:52 smithi151 ceph-mon[41387]: Deploying daemon mon.a on smithi151 2024-09-13T14:23:52.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:52 smithi151 ceph-mon[41387]: pgmap v48: 89 pgs: 89 active+clean; 169 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 4.6 MiB/s wr, 37 op/s 2024-09-13T14:23:52.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:52 smithi151 ceph-mon[41387]: osdmap e251: 8 total, 8 up, 8 in 2024-09-13T14:23:52.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:52 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:23:52.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:52 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/2139081028' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:52.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:52 smithi184 ceph-mon[33394]: Upgrade: Updating mon.a 2024-09-13T14:23:52.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:52 smithi184 ceph-mon[33394]: Deploying daemon mon.a on smithi151 2024-09-13T14:23:52.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:52 smithi184 ceph-mon[33394]: pgmap v48: 89 pgs: 89 active+clean; 169 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 4.6 MiB/s wr, 37 op/s 2024-09-13T14:23:52.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:52 smithi184 ceph-mon[33394]: osdmap e251: 8 total, 8 up, 8 in 2024-09-13T14:23:52.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:52 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:23:52.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:52 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/2139081028' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:52.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:52 smithi151 systemd[1]: Stopping Ceph mon.a for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:23:53.100 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:52 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a[29784]: 2024-09-13T14:23:52.686+0000 7fab37b4b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:23:53.100 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:52 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a[29784]: 2024-09-13T14:23:52.686+0000 7fab37b4b640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-13T14:23:53.101 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:52 smithi151 podman[104231]: 2024-09-13 14:23:52.758474125 +0000 UTC m=+0.215662529 container died ffc441eb291c47c362a7837504a281e149b24a73556712e8015912fa0696371d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=quincy-bbdadfa, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-13T14:23:53.101 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:52 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: [13/Sep/2024:14:23:52] ENGINE Bus STOPPING 2024-09-13T14:23:53.101 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:53 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: [13/Sep/2024:14:23:53] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:23:53.101 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:53 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: [13/Sep/2024:14:23:53] ENGINE Bus STOPPED 2024-09-13T14:23:53.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:53 smithi151 podman[104231]: 2024-09-13 14:23:53.311314489 +0000 UTC m=+0.768502920 container cleanup ffc441eb291c47c362a7837504a281e149b24a73556712e8015912fa0696371d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=quincy-bbdadfa, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2) 2024-09-13T14:23:53.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:53 smithi151 bash[104231]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a 2024-09-13T14:23:53.359 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:53 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: [13/Sep/2024:14:23:53] ENGINE Bus STARTING 2024-09-13T14:23:53.359 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:53 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: [13/Sep/2024:14:23:53] ENGINE Serving on http://:::9283 2024-09-13T14:23:53.359 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:53 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: [13/Sep/2024:14:23:53] ENGINE Bus STARTED 2024-09-13T14:23:53.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:53 smithi151 podman[104247]: 2024-09-13 14:23:53.605088197 +0000 UTC m=+0.841212485 container remove ffc441eb291c47c362a7837504a281e149b24a73556712e8015912fa0696371d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=quincy-bbdadfa, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-13T14:23:53.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:53 smithi151 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-13T14:23:53.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:53 smithi151 systemd[1]: Stopped Ceph mon.a for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:23:53.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:53 smithi151 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@mon.a.service: Consumed 13.428s CPU time. 2024-09-13T14:23:54.263 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:53 smithi151 systemd[1]: Starting Ceph mon.a for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:23:54.263 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 podman[104391]: 2024-09-13 14:23:54.040653707 +0000 UTC m=+0.015927997 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:23:54.263 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 podman[104391]: 2024-09-13 14:23:54.148501638 +0000 UTC m=+0.123775926 container create 694de6b945cd657e48dd990549243d0260bafc754240ec51555fb5c26362a11a (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:23:54.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:54 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:54.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:23:54.781 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:23:54] "GET /metrics HTTP/1.1" 200 36402 "" "Prometheus/2.51.0" 2024-09-13T14:23:54.781 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 podman[104391]: 2024-09-13 14:23:54.376641862 +0000 UTC m=+0.351916152 container init 694de6b945cd657e48dd990549243d0260bafc754240ec51555fb5c26362a11a (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:23:54.781 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 podman[104391]: 2024-09-13 14:23:54.38121788 +0000 UTC m=+0.356492170 container start 694de6b945cd657e48dd990549243d0260bafc754240ec51555fb5c26362a11a (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD) 2024-09-13T14:23:54.781 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: set uid:gid to 167:167 (ceph:ceph) 2024-09-13T14:23:54.782 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev), process ceph-mon, pid 7 2024-09-13T14:23:54.782 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: pidfile_write: ignore empty --pid-file 2024-09-13T14:23:54.782 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: load: jerasure load: lrc 2024-09-13T14:23:54.782 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: RocksDB version: 7.9.2 2024-09-13T14:23:54.782 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Git sha 0 2024-09-13T14:23:54.782 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Compile date 2024-09-12 20:46:14 2024-09-13T14:23:54.782 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: DB SUMMARY 2024-09-13T14:23:54.782 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: DB Session ID: W2A270W4BM93HL2XIOSK 2024-09-13T14:23:54.782 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: CURRENT file: CURRENT 2024-09-13T14:23:54.782 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: IDENTITY file: IDENTITY 2024-09-13T14:23:54.782 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: MANIFEST file: MANIFEST-000015 size: 973 Bytes 2024-09-13T14:23:54.783 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000030.sst 2024-09-13T14:23:54.783 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000028.log size: 3717114 ; 2024-09-13T14:23:54.783 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.error_if_exists: 0 2024-09-13T14:23:54.783 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.create_if_missing: 0 2024-09-13T14:23:54.783 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.paranoid_checks: 1 2024-09-13T14:23:54.783 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-13T14:23:54.783 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-13T14:23:54.783 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-13T14:23:54.783 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.env: 0x55fa81764160 2024-09-13T14:23:54.783 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.fs: PosixFileSystem 2024-09-13T14:23:54.783 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.info_log: 0x55fa822d1880 2024-09-13T14:23:54.784 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.max_file_opening_threads: 16 2024-09-13T14:23:54.784 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:23:54.786 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.statistics: (nil) 2024-09-13T14:23:54.786 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.use_fsync: 0 2024-09-13T14:23:54.786 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.max_log_file_size: 0 2024-09-13T14:23:54.786 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-13T14:23:54.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-13T14:23:54.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.keep_log_file_num: 1000 2024-09-13T14:23:54.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.recycle_log_file_num: 0 2024-09-13T14:23:54.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.allow_fallocate: 1 2024-09-13T14:23:54.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.allow_mmap_reads: 0 2024-09-13T14:23:54.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.allow_mmap_writes: 0 2024-09-13T14:23:54.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.use_direct_reads: 0 2024-09-13T14:23:54.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-13T14:23:54.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.create_missing_column_families: 0 2024-09-13T14:23:54.788 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.db_log_dir: 2024-09-13T14:23:54.788 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.wal_dir: 2024-09-13T14:23:54.788 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-13T14:23:54.788 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-13T14:23:54.788 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-13T14:23:54.788 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-13T14:23:54.788 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-13T14:23:54.788 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-13T14:23:54.788 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.advise_random_on_open: 1 2024-09-13T14:23:54.788 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.db_write_buffer_size: 0 2024-09-13T14:23:54.788 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.write_buffer_manager: 0x55fa822d4960 2024-09-13T14:23:54.789 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-13T14:23:54.789 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-13T14:23:54.789 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-13T14:23:54.789 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.rate_limiter: (nil) 2024-09-13T14:23:54.789 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-13T14:23:54.789 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.wal_recovery_mode: 2 2024-09-13T14:23:54.789 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.enable_thread_tracking: 0 2024-09-13T14:23:54.789 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.enable_pipelined_write: 0 2024-09-13T14:23:54.789 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.unordered_write: 0 2024-09-13T14:23:54.789 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-13T14:23:54.789 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-13T14:23:54.789 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-13T14:23:54.790 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-13T14:23:54.790 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.row_cache: None 2024-09-13T14:23:54.790 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.wal_filter: None 2024-09-13T14:23:54.790 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-13T14:23:54.790 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.allow_ingest_behind: 0 2024-09-13T14:23:54.790 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.two_write_queues: 0 2024-09-13T14:23:54.790 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.manual_wal_flush: 0 2024-09-13T14:23:54.790 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.wal_compression: 0 2024-09-13T14:23:54.790 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.atomic_flush: 0 2024-09-13T14:23:54.790 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-13T14:23:54.790 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-13T14:23:54.790 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-13T14:23:54.790 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.log_readahead_size: 0 2024-09-13T14:23:54.791 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-13T14:23:54.791 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.best_efforts_recovery: 0 2024-09-13T14:23:54.791 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-13T14:23:54.791 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-13T14:23:54.791 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.allow_data_in_errors: 0 2024-09-13T14:23:54.791 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.db_host_id: __hostname__ 2024-09-13T14:23:54.791 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-13T14:23:54.791 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.max_background_jobs: 2 2024-09-13T14:23:54.791 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.max_background_compactions: -1 2024-09-13T14:23:54.791 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.max_subcompactions: 1 2024-09-13T14:23:54.791 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-13T14:23:54.791 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-13T14:23:54.791 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-13T14:23:54.791 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.max_total_wal_size: 0 2024-09-13T14:23:54.791 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-13T14:23:54.792 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-13T14:23:54.792 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-13T14:23:54.792 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-13T14:23:54.792 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.max_open_files: -1 2024-09-13T14:23:54.792 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.bytes_per_sync: 0 2024-09-13T14:23:54.792 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-13T14:23:54.792 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-13T14:23:54.792 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.compaction_readahead_size: 0 2024-09-13T14:23:54.792 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.max_background_flushes: -1 2024-09-13T14:23:54.792 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Compression algorithms supported: 2024-09-13T14:23:54.792 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: kZSTD supported: 0 2024-09-13T14:23:54.792 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: kXpressCompression supported: 0 2024-09-13T14:23:54.792 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: kBZip2Compression supported: 0 2024-09-13T14:23:54.792 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-13T14:23:54.793 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: kLZ4Compression supported: 1 2024-09-13T14:23:54.793 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: kZlibCompression supported: 1 2024-09-13T14:23:54.793 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: kLZ4HCCompression supported: 1 2024-09-13T14:23:54.793 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: kSnappyCompression supported: 1 2024-09-13T14:23:54.793 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-13T14:23:54.793 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-13T14:23:54.793 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-09-13T14:23:54.793 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-13T14:23:54.793 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-13T14:23:54.793 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.merge_operator: 2024-09-13T14:23:54.793 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.compaction_filter: None 2024-09-13T14:23:54.793 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.compaction_filter_factory: None 2024-09-13T14:23:54.793 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.sst_partitioner_factory: None 2024-09-13T14:23:54.793 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-13T14:23:54.794 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-13T14:23:54.794 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fa822d14e0) 2024-09-13T14:23:54.794 INFO:journalctl@ceph.mon.a.smithi151.stdout: cache_index_and_filter_blocks: 1 2024-09-13T14:23:54.794 INFO:journalctl@ceph.mon.a.smithi151.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-13T14:23:54.794 INFO:journalctl@ceph.mon.a.smithi151.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-13T14:23:54.794 INFO:journalctl@ceph.mon.a.smithi151.stdout: pin_top_level_index_and_filter: 1 2024-09-13T14:23:54.794 INFO:journalctl@ceph.mon.a.smithi151.stdout: index_type: 0 2024-09-13T14:23:54.794 INFO:journalctl@ceph.mon.a.smithi151.stdout: data_block_index_type: 0 2024-09-13T14:23:54.794 INFO:journalctl@ceph.mon.a.smithi151.stdout: index_shortening: 1 2024-09-13T14:23:54.794 INFO:journalctl@ceph.mon.a.smithi151.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-13T14:23:54.794 INFO:journalctl@ceph.mon.a.smithi151.stdout: checksum: 4 2024-09-13T14:23:54.794 INFO:journalctl@ceph.mon.a.smithi151.stdout: no_block_cache: 0 2024-09-13T14:23:54.795 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_cache: 0x55fa822c4850 2024-09-13T14:23:54.795 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_cache_name: BinnedLRUCache 2024-09-13T14:23:54.795 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_cache_options: 2024-09-13T14:23:54.795 INFO:journalctl@ceph.mon.a.smithi151.stdout: capacity : 536870912 2024-09-13T14:23:54.795 INFO:journalctl@ceph.mon.a.smithi151.stdout: num_shard_bits : 4 2024-09-13T14:23:54.795 INFO:journalctl@ceph.mon.a.smithi151.stdout: strict_capacity_limit : 0 2024-09-13T14:23:54.795 INFO:journalctl@ceph.mon.a.smithi151.stdout: high_pri_pool_ratio: 0.000 2024-09-13T14:23:54.795 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_cache_compressed: (nil) 2024-09-13T14:23:54.795 INFO:journalctl@ceph.mon.a.smithi151.stdout: persistent_cache: (nil) 2024-09-13T14:23:54.795 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_size: 4096 2024-09-13T14:23:54.795 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_size_deviation: 10 2024-09-13T14:23:54.795 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_restart_interval: 16 2024-09-13T14:23:54.795 INFO:journalctl@ceph.mon.a.smithi151.stdout: index_block_restart_interval: 1 2024-09-13T14:23:54.795 INFO:journalctl@ceph.mon.a.smithi151.stdout: metadata_block_size: 4096 2024-09-13T14:23:54.796 INFO:journalctl@ceph.mon.a.smithi151.stdout: partition_filters: 0 2024-09-13T14:23:54.796 INFO:journalctl@ceph.mon.a.smithi151.stdout: use_delta_encoding: 1 2024-09-13T14:23:54.796 INFO:journalctl@ceph.mon.a.smithi151.stdout: filter_policy: bloomfilter 2024-09-13T14:23:54.796 INFO:journalctl@ceph.mon.a.smithi151.stdout: whole_key_filtering: 1 2024-09-13T14:23:54.796 INFO:journalctl@ceph.mon.a.smithi151.stdout: verify_compression: 0 2024-09-13T14:23:54.796 INFO:journalctl@ceph.mon.a.smithi151.stdout: read_amp_bytes_per_bit: 0 2024-09-13T14:23:54.796 INFO:journalctl@ceph.mon.a.smithi151.stdout: format_version: 5 2024-09-13T14:23:54.796 INFO:journalctl@ceph.mon.a.smithi151.stdout: enable_index_compression: 1 2024-09-13T14:23:54.796 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_align: 0 2024-09-13T14:23:54.796 INFO:journalctl@ceph.mon.a.smithi151.stdout: max_auto_readahead_size: 262144 2024-09-13T14:23:54.796 INFO:journalctl@ceph.mon.a.smithi151.stdout: prepopulate_block_cache: 0 2024-09-13T14:23:54.796 INFO:journalctl@ceph.mon.a.smithi151.stdout: initial_auto_readahead_size: 8192 2024-09-13T14:23:54.796 INFO:journalctl@ceph.mon.a.smithi151.stdout: num_file_reads_for_auto_readahead: 2 2024-09-13T14:23:54.797 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.write_buffer_size: 33554432 2024-09-13T14:23:54.797 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.max_write_buffer_number: 2 2024-09-13T14:23:54.797 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.compression: NoCompression 2024-09-13T14:23:54.797 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.bottommost_compression: Disabled 2024-09-13T14:23:54.797 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.prefix_extractor: nullptr 2024-09-13T14:23:54.797 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-13T14:23:54.797 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.num_levels: 7 2024-09-13T14:23:54.797 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-13T14:23:54.797 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-13T14:23:54.797 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-13T14:23:54.797 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-13T14:23:54.797 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-13T14:23:54.797 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-13T14:23:54.797 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-13T14:23:54.798 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:23:54.798 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-13T14:23:54.798 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-13T14:23:54.798 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-13T14:23:54.798 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-13T14:23:54.798 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-13T14:23:54.798 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.compression_opts.level: 32767 2024-09-13T14:23:54.798 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.compression_opts.strategy: 0 2024-09-13T14:23:54.798 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-13T14:23:54.798 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:23:54.798 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-13T14:23:54.798 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-13T14:23:54.798 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.compression_opts.enabled: false 2024-09-13T14:23:54.799 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-13T14:23:54.799 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-13T14:23:54.799 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-13T14:23:54.799 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-13T14:23:54.799 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.target_file_size_base: 67108864 2024-09-13T14:23:54.799 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-13T14:23:54.799 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-13T14:23:54.799 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-13T14:23:54.799 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-13T14:23:54.799 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-13T14:23:54.799 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-13T14:23:54.799 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-13T14:23:54.799 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-13T14:23:54.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-13T14:23:54.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-13T14:23:54.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-13T14:23:54.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-13T14:23:54.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-13T14:23:54.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-13T14:23:54.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.arena_block_size: 1048576 2024-09-13T14:23:54.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-13T14:23:54.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-13T14:23:54.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.disable_auto_compactions: 0 2024-09-13T14:23:54.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-13T14:23:54.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-13T14:23:54.801 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-13T14:23:54.801 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-13T14:23:54.801 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-13T14:23:54.801 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-13T14:23:54.801 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-13T14:23:54.801 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-13T14:23:54.801 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-13T14:23:54.801 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-13T14:23:54.801 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-13T14:23:54.801 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.inplace_update_support: 0 2024-09-13T14:23:54.801 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-13T14:23:54.801 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-13T14:23:54.802 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-13T14:23:54.802 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-13T14:23:54.802 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.bloom_locality: 0 2024-09-13T14:23:54.802 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.max_successive_merges: 0 2024-09-13T14:23:54.802 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-13T14:23:54.802 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.paranoid_file_checks: 0 2024-09-13T14:23:54.802 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.force_consistency_checks: 1 2024-09-13T14:23:54.802 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.report_bg_io_stats: 0 2024-09-13T14:23:54.802 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.ttl: 2592000 2024-09-13T14:23:54.802 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-13T14:23:54.802 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-13T14:23:54.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-13T14:23:54.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.enable_blob_files: false 2024-09-13T14:23:54.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.min_blob_size: 0 2024-09-13T14:23:54.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.blob_file_size: 268435456 2024-09-13T14:23:54.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-13T14:23:54.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-13T14:23:54.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-13T14:23:54.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-13T14:23:54.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-13T14:23:54.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.blob_file_starting_level: 0 2024-09-13T14:23:54.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-13T14:23:54.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-09-13T14:23:54.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-13T14:23:54.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 12809, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-13T14:23:54.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-09-13T14:23:54.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a30cc2d4-5376-406c-97fc-d2c7b106f2f7 2024-09-13T14:23:54.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726237434408196, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-09-13T14:23:54.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-09-13T14:23:54.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726237434418433, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 3472706, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12811, "largest_seqno": 14557, "table_properties": {"data_size": 3463412, "index_size": 6264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1989, "raw_key_size": 19452, "raw_average_key_size": 24, "raw_value_size": 3445738, "raw_average_value_size": 4367, "num_data_blocks": 275, "num_entries": 789, "num_filter_entries": 789, "num_deletions": 36, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726237434, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a30cc2d4-5376-406c-97fc-d2c7b106f2f7", "db_session_id": "W2A270W4BM93HL2XIOSK", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-09-13T14:23:54.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726237434418534, "job": 1, "event": "recovery_finished"} 2024-09-13T14:23:54.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-09-13T14:23:54.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-13T14:23:54.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-13T14:23:54.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55fa822f4e00 2024-09-13T14:23:54.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: DB pointer 0x55fa823f4000 2024-09-13T14:23:54.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-13T14:23:54.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-13T14:23:54.805 INFO:journalctl@ceph.mon.a.smithi151.stdout: ** DB Stats ** 2024-09-13T14:23:54.805 INFO:journalctl@ceph.mon.a.smithi151.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:23:54.805 INFO:journalctl@ceph.mon.a.smithi151.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-13T14:23:54.805 INFO:journalctl@ceph.mon.a.smithi151.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:23:54.805 INFO:journalctl@ceph.mon.a.smithi151.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:23:54.805 INFO:journalctl@ceph.mon.a.smithi151.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-13T14:23:54.806 INFO:journalctl@ceph.mon.a.smithi151.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:23:54.806 INFO:journalctl@ceph.mon.a.smithi151.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:23:54.806 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-09-13T14:23:54.806 INFO:journalctl@ceph.mon.a.smithi151.stdout: ** Compaction Stats [default] ** 2024-09-13T14:23:54.806 INFO:journalctl@ceph.mon.a.smithi151.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T14:23:54.806 INFO:journalctl@ceph.mon.a.smithi151.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-13T14:23:54.806 INFO:journalctl@ceph.mon.a.smithi151.stdout: L0 1/0 3.31 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 668.9 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-09-13T14:23:54.806 INFO:journalctl@ceph.mon.a.smithi151.stdout: L6 1/0 7.36 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-13T14:23:54.806 INFO:journalctl@ceph.mon.a.smithi151.stdout: Sum 2/0 10.67 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 668.9 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-09-13T14:23:54.806 INFO:journalctl@ceph.mon.a.smithi151.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 668.9 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-09-13T14:23:54.806 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-09-13T14:23:54.806 INFO:journalctl@ceph.mon.a.smithi151.stdout: ** Compaction Stats [default] ** 2024-09-13T14:23:54.807 INFO:journalctl@ceph.mon.a.smithi151.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T14:23:54.807 INFO:journalctl@ceph.mon.a.smithi151.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T14:23:54.807 INFO:journalctl@ceph.mon.a.smithi151.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 668.9 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-09-13T14:23:54.807 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-09-13T14:23:54.807 INFO:journalctl@ceph.mon.a.smithi151.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-13T14:23:54.807 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-09-13T14:23:54.807 INFO:journalctl@ceph.mon.a.smithi151.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:23:54.807 INFO:journalctl@ceph.mon.a.smithi151.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-09-13T14:23:54.807 INFO:journalctl@ceph.mon.a.smithi151.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-13T14:23:54.807 INFO:journalctl@ceph.mon.a.smithi151.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-13T14:23:54.807 INFO:journalctl@ceph.mon.a.smithi151.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-13T14:23:54.807 INFO:journalctl@ceph.mon.a.smithi151.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-13T14:23:54.808 INFO:journalctl@ceph.mon.a.smithi151.stdout: Cumulative compaction: 0.00 GB write, 194.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:23:54.808 INFO:journalctl@ceph.mon.a.smithi151.stdout: Interval compaction: 0.00 GB write, 194.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:23:54.808 INFO:journalctl@ceph.mon.a.smithi151.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-13T14:23:54.808 INFO:journalctl@ceph.mon.a.smithi151.stdout: Block cache BinnedLRUCache@0x55fa822c4850#7 capacity: 512.00 MB usage: 8.61 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-09-13T14:23:54.808 INFO:journalctl@ceph.mon.a.smithi151.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.03 KB,0.00038743%) IndexBlock(1,6.58 KB,0.00125468%) Misc(1,0.00 KB,0%) 2024-09-13T14:23:54.808 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-09-13T14:23:54.808 INFO:journalctl@ceph.mon.a.smithi151.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-13T14:23:54.808 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: starting mon.a rank 0 at public addrs [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] at bind addrs [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:23:54.808 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: mon.a@-1(???) e3 preinit fsid 2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:23:54.808 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: mon.a@-1(???).mds e9 new map 2024-09-13T14:23:54.808 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: mon.a@-1(???).mds e9 print_map 2024-09-13T14:23:54.808 INFO:journalctl@ceph.mon.a.smithi151.stdout: e9 2024-09-13T14:23:54.809 INFO:journalctl@ceph.mon.a.smithi151.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-13T14:23:54.809 INFO:journalctl@ceph.mon.a.smithi151.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-13T14:23:54.809 INFO:journalctl@ceph.mon.a.smithi151.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T14:23:54.809 INFO:journalctl@ceph.mon.a.smithi151.stdout: legacy client fscid: 1 2024-09-13T14:23:54.809 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-09-13T14:23:54.809 INFO:journalctl@ceph.mon.a.smithi151.stdout: Filesystem 'foo' (1) 2024-09-13T14:23:54.809 INFO:journalctl@ceph.mon.a.smithi151.stdout: fs_name foo 2024-09-13T14:23:54.809 INFO:journalctl@ceph.mon.a.smithi151.stdout: epoch 8 2024-09-13T14:23:54.809 INFO:journalctl@ceph.mon.a.smithi151.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-13T14:23:54.809 INFO:journalctl@ceph.mon.a.smithi151.stdout: created 2024-09-13T14:19:25.869068+0000 2024-09-13T14:23:54.809 INFO:journalctl@ceph.mon.a.smithi151.stdout: modified 2024-09-13T14:20:09.750989+0000 2024-09-13T14:23:54.809 INFO:journalctl@ceph.mon.a.smithi151.stdout: tableserver 0 2024-09-13T14:23:54.809 INFO:journalctl@ceph.mon.a.smithi151.stdout: root 0 2024-09-13T14:23:54.810 INFO:journalctl@ceph.mon.a.smithi151.stdout: session_timeout 60 2024-09-13T14:23:54.810 INFO:journalctl@ceph.mon.a.smithi151.stdout: session_autoclose 300 2024-09-13T14:23:54.810 INFO:journalctl@ceph.mon.a.smithi151.stdout: max_file_size 1099511627776 2024-09-13T14:23:54.810 INFO:journalctl@ceph.mon.a.smithi151.stdout: max_xattr_size 65536 2024-09-13T14:23:54.810 INFO:journalctl@ceph.mon.a.smithi151.stdout: required_client_features {} 2024-09-13T14:23:54.810 INFO:journalctl@ceph.mon.a.smithi151.stdout: last_failure 0 2024-09-13T14:23:54.810 INFO:journalctl@ceph.mon.a.smithi151.stdout: last_failure_osd_epoch 0 2024-09-13T14:23:54.810 INFO:journalctl@ceph.mon.a.smithi151.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T14:23:54.810 INFO:journalctl@ceph.mon.a.smithi151.stdout: max_mds 1 2024-09-13T14:23:54.810 INFO:journalctl@ceph.mon.a.smithi151.stdout: in 0 2024-09-13T14:23:54.810 INFO:journalctl@ceph.mon.a.smithi151.stdout: up {0=24614} 2024-09-13T14:23:54.810 INFO:journalctl@ceph.mon.a.smithi151.stdout: failed 2024-09-13T14:23:54.810 INFO:journalctl@ceph.mon.a.smithi151.stdout: damaged 2024-09-13T14:23:54.810 INFO:journalctl@ceph.mon.a.smithi151.stdout: stopped 2024-09-13T14:23:54.811 INFO:journalctl@ceph.mon.a.smithi151.stdout: data_pools [4] 2024-09-13T14:23:54.811 INFO:journalctl@ceph.mon.a.smithi151.stdout: metadata_pool 3 2024-09-13T14:23:54.811 INFO:journalctl@ceph.mon.a.smithi151.stdout: inline_data disabled 2024-09-13T14:23:54.811 INFO:journalctl@ceph.mon.a.smithi151.stdout: balancer 2024-09-13T14:23:54.811 INFO:journalctl@ceph.mon.a.smithi151.stdout: bal_rank_mask -1 2024-09-13T14:23:54.811 INFO:journalctl@ceph.mon.a.smithi151.stdout: standby_count_wanted 1 2024-09-13T14:23:54.811 INFO:journalctl@ceph.mon.a.smithi151.stdout: qdb_cluster leader: 0 members: 2024-09-13T14:23:54.811 INFO:journalctl@ceph.mon.a.smithi151.stdout: [mds.foo.smithi151.rqqklw{0:24614} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.151:6834/345315682,v1:172.21.15.151:6835/345315682] compat {c=[1],r=[1],i=[7ff]}] 2024-09-13T14:23:54.811 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-09-13T14:23:54.811 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-09-13T14:23:54.811 INFO:journalctl@ceph.mon.a.smithi151.stdout: Standby daemons: 2024-09-13T14:23:54.811 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-09-13T14:23:54.811 INFO:journalctl@ceph.mon.a.smithi151.stdout: [mds.foo.smithi184.jismsi{-1:24529} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.184:6832/4138067167,v1:172.21.15.184:6833/4138067167] compat {c=[1],r=[1],i=[7ff]}] 2024-09-13T14:23:54.812 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: mon.a@-1(???).osd e252 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-13T14:23:54.812 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: mon.a@-1(???).osd e252 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:23:54.812 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: mon.a@-1(???).osd e252 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:23:54.812 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: mon.a@-1(???).osd e252 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:23:54.812 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 ceph-mon[104450]: mon.a@-1(???).paxosservice(auth 1..18) refresh upgraded, format 0 -> 3 2024-09-13T14:23:54.812 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 bash[104391]: 694de6b945cd657e48dd990549243d0260bafc754240ec51555fb5c26362a11a 2024-09-13T14:23:54.812 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:54 smithi151 systemd[1]: Started Ceph mon.a for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:23:55.906 INFO:tasks.rados.rados.0.smithi151.stdout:759: done (0 left) 2024-09-13T14:23:55.906 INFO:tasks.rados.rados.0.smithi151.stdout:760: read oid 41 snap 80 2024-09-13T14:23:55.906 INFO:tasks.rados.rados.0.smithi151.stdout:760: expect deleted 2024-09-13T14:23:55.906 INFO:tasks.rados.rados.0.smithi151.stdout:761: write oid 14 current snap is 98 2024-09-13T14:23:55.906 INFO:tasks.rados.rados.0.smithi151.stdout:761: seq_num 321 ranges {595047=641755,1667661=418109,2548663=763022} 2024-09-13T14:23:55.913 INFO:tasks.rados.rados.0.smithi151.stdout:761: writing smithi15186934-14 from 595047 to 1236802 tid 1 2024-09-13T14:23:55.918 INFO:tasks.rados.rados.0.smithi151.stdout:761: writing smithi15186934-14 from 1667661 to 2085770 tid 2 2024-09-13T14:23:55.925 INFO:tasks.rados.rados.0.smithi151.stdout:761: writing smithi15186934-14 from 2548663 to 3311685 tid 3 2024-09-13T14:23:55.925 INFO:tasks.rados.rados.0.smithi151.stdout:762: read oid 10 snap 94 2024-09-13T14:23:55.926 INFO:tasks.rados.rados.0.smithi151.stdout:762: expect (ObjNum 291 snap 93 seq_num 291) 2024-09-13T14:23:55.926 INFO:tasks.rados.rados.0.smithi151.stdout:763: read oid 6 snap -1 2024-09-13T14:23:55.926 INFO:tasks.rados.rados.0.smithi151.stdout:763: expect (ObjNum 294 snap 93 seq_num 294) 2024-09-13T14:23:55.926 INFO:tasks.rados.rados.0.smithi151.stdout:764: snap_create 2024-09-13T14:23:55.926 INFO:tasks.rados.rados.0.smithi151.stdout:761: finishing write tid 1 to smithi15186934-14 2024-09-13T14:23:55.926 INFO:tasks.rados.rados.0.smithi151.stdout:761: finishing write tid 2 to smithi15186934-14 2024-09-13T14:23:55.931 INFO:tasks.rados.rados.0.smithi151.stdout:761: finishing write tid 3 to smithi15186934-14 2024-09-13T14:23:55.931 INFO:tasks.rados.rados.0.smithi151.stdout:761: finishing write tid 4 to smithi15186934-14 2024-09-13T14:23:55.931 INFO:tasks.rados.rados.0.smithi151.stdout:761: finishing write tid 5 to smithi15186934-14 2024-09-13T14:23:55.931 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 339 (ObjNum 321 snap 98 seq_num 321) dirty exists 2024-09-13T14:23:55.931 INFO:tasks.rados.rados.0.smithi151.stdout:761: left oid 14 (ObjNum 321 snap 98 seq_num 321) 2024-09-13T14:23:55.931 INFO:tasks.rados.rados.0.smithi151.stdout:762: expect (ObjNum 291 snap 93 seq_num 291) 2024-09-13T14:23:56.030 INFO:tasks.rados.rados.0.smithi151.stdout:763: expect (ObjNum 294 snap 93 seq_num 294) 2024-09-13T14:23:56.103 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[82046]: ignoring --setuser ceph since I am not root 2024-09-13T14:23:56.104 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[82046]: ignoring --setgroup ceph since I am not root 2024-09-13T14:23:56.104 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-mgr[82069]: -- 172.21.15.184:0/770379031 <== mon.1 v2:172.21.15.184:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556ad9a490e0 con 0x556ad9a27000 2024-09-13T14:23:56.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:55.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:23:56.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:55.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:23:56.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:23:56.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:23:56.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:23:56.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-mon[33394]: pgmap v51: 89 pgs: 89 active+clean; 184 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.8 MiB/s rd, 2.6 MiB/s wr, 20 op/s 2024-09-13T14:23:56.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-mon[33394]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:23:56.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-mon[33394]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:23:56.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/4292148050' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:56.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-mon[33394]: from='client.25934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:56.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-mon[33394]: pgmap v52: 89 pgs: 89 active+clean; 184 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 1.8 MiB/s wr, 14 op/s 2024-09-13T14:23:56.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-mon[33394]: mon.a calling monitor election 2024-09-13T14:23:56.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-mon[33394]: from='client.25940 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:56.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-mon[33394]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:23:56.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-mon[33394]: monmap epoch 3 2024-09-13T14:23:56.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-mon[33394]: fsid 2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:23:56.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-mon[33394]: last_changed 2024-09-13T14:14:11.796948+0000 2024-09-13T14:23:56.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-mon[33394]: created 2024-09-13T14:12:33.302475+0000 2024-09-13T14:23:56.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-mon[33394]: min_mon_release 17 (quincy) 2024-09-13T14:23:56.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-mon[33394]: election_strategy: 1 2024-09-13T14:23:56.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-mon[33394]: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2024-09-13T14:23:56.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-mon[33394]: 1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2024-09-13T14:23:56.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-mon[33394]: 2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2024-09-13T14:23:56.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-mon[33394]: fsmap foo:1 {0=foo.smithi151.rqqklw=up:active} 1 up:standby 2024-09-13T14:23:56.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-mon[33394]: osdmap e252: 8 total, 8 up, 8 in 2024-09-13T14:23:56.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-mon[33394]: mgrmap e32: y(active, since 35s), standbys: x 2024-09-13T14:23:56.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-mon[33394]: overall HEALTH_OK 2024-09-13T14:23:56.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-mon[33394]: from='mgr.15720 ' entity='' 2024-09-13T14:23:56.107 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:55 smithi184 ceph-mon[33394]: mgrmap e33: y(active, since 35s), standbys: x 2024-09-13T14:23:56.358 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ignoring --setuser ceph since I am not root 2024-09-13T14:23:56.359 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ignoring --setgroup ceph since I am not root 2024-09-13T14:23:56.359 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mgr[94881]: -- 172.21.15.151:0/959865683 <== mon.0 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55769c85b0e0 con 0x55769c839400 2024-09-13T14:23:56.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:23:56.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:23:56.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:23:56.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[41387]: pgmap v51: 89 pgs: 89 active+clean; 184 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.8 MiB/s rd, 2.6 MiB/s wr, 20 op/s 2024-09-13T14:23:56.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[41387]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:23:56.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[41387]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:23:56.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/4292148050' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:56.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[41387]: from='client.25934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:56.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[41387]: pgmap v52: 89 pgs: 89 active+clean; 184 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 1.8 MiB/s wr, 14 op/s 2024-09-13T14:23:56.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[41387]: mon.a calling monitor election 2024-09-13T14:23:56.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[41387]: from='client.25940 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:56.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[41387]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:23:56.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[41387]: monmap epoch 3 2024-09-13T14:23:56.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[41387]: fsid 2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:23:56.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[41387]: last_changed 2024-09-13T14:14:11.796948+0000 2024-09-13T14:23:56.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[41387]: created 2024-09-13T14:12:33.302475+0000 2024-09-13T14:23:56.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[41387]: min_mon_release 17 (quincy) 2024-09-13T14:23:56.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[41387]: election_strategy: 1 2024-09-13T14:23:56.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[41387]: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2024-09-13T14:23:56.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[41387]: 1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2024-09-13T14:23:56.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[41387]: 2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2024-09-13T14:23:56.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[41387]: fsmap foo:1 {0=foo.smithi151.rqqklw=up:active} 1 up:standby 2024-09-13T14:23:56.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[41387]: osdmap e252: 8 total, 8 up, 8 in 2024-09-13T14:23:56.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[41387]: mgrmap e32: y(active, since 35s), standbys: x 2024-09-13T14:23:56.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[41387]: overall HEALTH_OK 2024-09-13T14:23:56.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[41387]: from='mgr.15720 ' entity='' 2024-09-13T14:23:56.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[41387]: mgrmap e33: y(active, since 35s), standbys: x 2024-09-13T14:23:56.361 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[104450]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:23:56.361 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[104450]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:23:56.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[104450]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:23:56.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[104450]: pgmap v51: 89 pgs: 89 active+clean; 184 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.8 MiB/s rd, 2.6 MiB/s wr, 20 op/s 2024-09-13T14:23:56.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[104450]: from='mgr.15720 172.21.15.151:0/2209716439' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:23:56.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[104450]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:23:56.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/4292148050' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:56.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[104450]: from='client.25934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:56.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[104450]: pgmap v52: 89 pgs: 89 active+clean; 184 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 1.8 MiB/s wr, 14 op/s 2024-09-13T14:23:56.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[104450]: mon.a calling monitor election 2024-09-13T14:23:56.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[104450]: from='client.25940 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:56.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[104450]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:23:56.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[104450]: monmap epoch 3 2024-09-13T14:23:56.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[104450]: fsid 2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:23:56.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[104450]: last_changed 2024-09-13T14:14:11.796948+0000 2024-09-13T14:23:56.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[104450]: created 2024-09-13T14:12:33.302475+0000 2024-09-13T14:23:56.363 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[104450]: min_mon_release 17 (quincy) 2024-09-13T14:23:56.363 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[104450]: election_strategy: 1 2024-09-13T14:23:56.363 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[104450]: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2024-09-13T14:23:56.363 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[104450]: 1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2024-09-13T14:23:56.363 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[104450]: 2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2024-09-13T14:23:56.363 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[104450]: fsmap foo:1 {0=foo.smithi151.rqqklw=up:active} 1 up:standby 2024-09-13T14:23:56.363 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[104450]: osdmap e252: 8 total, 8 up, 8 in 2024-09-13T14:23:56.363 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[104450]: mgrmap e32: y(active, since 35s), standbys: x 2024-09-13T14:23:56.363 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[104450]: overall HEALTH_OK 2024-09-13T14:23:56.363 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[104450]: from='mgr.15720 ' entity='' 2024-09-13T14:23:56.363 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:55 smithi151 ceph-mon[104450]: mgrmap e33: y(active, since 35s), standbys: x 2024-09-13T14:23:56.910 INFO:tasks.rados.rados.0.smithi151.stdout:760: done (4 left) 2024-09-13T14:23:56.910 INFO:tasks.rados.rados.0.smithi151.stdout:761: done (3 left) 2024-09-13T14:23:56.910 INFO:tasks.rados.rados.0.smithi151.stdout:762: done (2 left) 2024-09-13T14:23:56.910 INFO:tasks.rados.rados.0.smithi151.stdout:763: done (1 left) 2024-09-13T14:23:56.910 INFO:tasks.rados.rados.0.smithi151.stdout:764: done (0 left) 2024-09-13T14:23:56.910 INFO:tasks.rados.rados.0.smithi151.stdout:765: snap_create 2024-09-13T14:23:57.259 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:56 smithi151 ceph-mon[104450]: osdmap e253: 8 total, 8 up, 8 in 2024-09-13T14:23:57.259 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:56 smithi151 ceph-mon[41387]: osdmap e253: 8 total, 8 up, 8 in 2024-09-13T14:23:57.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:56 smithi184 ceph-mon[33394]: osdmap e253: 8 total, 8 up, 8 in 2024-09-13T14:23:57.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:23:56 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:23:56.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi184\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:23:57.407 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 1 2024-09-13T14:23:57.917 INFO:tasks.rados.rados.0.smithi151.stdout:765: done (0 left) 2024-09-13T14:23:57.917 INFO:tasks.rados.rados.0.smithi151.stdout:766: write oid 42 current snap is 100 2024-09-13T14:23:57.918 INFO:tasks.rados.rados.0.smithi151.stdout:766: seq_num 322 ranges {611854=717004,2126143=561767,3259733=152121} 2024-09-13T14:23:57.925 INFO:tasks.rados.rados.0.smithi151.stdout:766: writing smithi15186934-42 from 611854 to 1328858 tid 1 2024-09-13T14:23:57.933 INFO:tasks.rados.rados.0.smithi151.stdout:766: writing smithi15186934-42 from 2126143 to 2687910 tid 2 2024-09-13T14:23:57.938 INFO:tasks.rados.rados.0.smithi151.stdout:766: writing smithi15186934-42 from 3259733 to 3411854 tid 3 2024-09-13T14:23:57.938 INFO:tasks.rados.rados.0.smithi151.stdout:767: read oid 4 snap 89 2024-09-13T14:23:57.938 INFO:tasks.rados.rados.0.smithi151.stdout:767: expect deleted 2024-09-13T14:23:57.938 INFO:tasks.rados.rados.0.smithi151.stdout:768: rollback oid 35 current snap is 100 2024-09-13T14:23:57.938 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 35 to 97 2024-09-13T14:23:57.938 INFO:tasks.rados.rados.0.smithi151.stdout:766: finishing write tid 1 to smithi15186934-42 2024-09-13T14:23:57.938 INFO:tasks.rados.rados.0.smithi151.stdout:766: finishing write tid 2 to smithi15186934-42 2024-09-13T14:23:57.938 INFO:tasks.rados.rados.0.smithi151.stdout:769: snap_create 2024-09-13T14:23:57.939 INFO:tasks.rados.rados.0.smithi151.stdout:766: finishing write tid 3 to smithi15186934-42 2024-09-13T14:23:57.939 INFO:tasks.rados.rados.0.smithi151.stdout:766: finishing write tid 4 to smithi15186934-42 2024-09-13T14:23:57.940 INFO:tasks.rados.rados.0.smithi151.stdout:766: finishing write tid 5 to smithi15186934-42 2024-09-13T14:23:57.940 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 395 (ObjNum 322 snap 100 seq_num 322) dirty exists 2024-09-13T14:23:57.940 INFO:tasks.rados.rados.0.smithi151.stdout:766: left oid 42 (ObjNum 322 snap 100 seq_num 322) 2024-09-13T14:23:57.941 INFO:tasks.rados.rados.0.smithi151.stdout:768: finishing rollback tid 0 to smithi15186934-35 2024-09-13T14:23:57.941 INFO:tasks.rados.rados.0.smithi151.stdout:768: finishing rollback tid 1 to smithi15186934-35 2024-09-13T14:23:57.941 INFO:tasks.rados.rados.0.smithi151.stdout:768: finishing rollback tid 2 to smithi15186934-35 2024-09-13T14:23:57.941 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 35 v 492 (ObjNum 196 snap 55 seq_num 196) dirty exists 2024-09-13T14:23:58.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:57 smithi151 ceph-mon[41387]: osdmap e254: 8 total, 8 up, 8 in 2024-09-13T14:23:58.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:57 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/713077486' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:58.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:57 smithi151 ceph-mon[104450]: osdmap e254: 8 total, 8 up, 8 in 2024-09-13T14:23:58.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:57 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/713077486' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:58.158 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch ps' 2024-09-13T14:23:58.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:57 smithi184 ceph-mon[33394]: osdmap e254: 8 total, 8 up, 8 in 2024-09-13T14:23:58.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:57 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/713077486' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:23:58.424 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:23:58.933 INFO:tasks.rados.rados.0.smithi151.stdout:766: done (3 left) 2024-09-13T14:23:58.933 INFO:tasks.rados.rados.0.smithi151.stdout:767: done (2 left) 2024-09-13T14:23:58.933 INFO:tasks.rados.rados.0.smithi151.stdout:768: done (1 left) 2024-09-13T14:23:58.933 INFO:tasks.rados.rados.0.smithi151.stdout:769: done (0 left) 2024-09-13T14:23:58.933 INFO:tasks.rados.rados.0.smithi151.stdout:770: write oid 19 current snap is 101 2024-09-13T14:23:58.933 INFO:tasks.rados.rados.0.smithi151.stdout:770: seq_num 323 ranges {628661=792253,2184625=421778,3403509=25152} 2024-09-13T14:23:58.940 INFO:tasks.rados.rados.0.smithi151.stdout:770: writing smithi15186934-19 from 628661 to 1420914 tid 1 2024-09-13T14:23:58.947 INFO:tasks.rados.rados.0.smithi151.stdout:770: writing smithi15186934-19 from 2184625 to 2606403 tid 2 2024-09-13T14:23:58.952 INFO:tasks.rados.rados.0.smithi151.stdout:770: writing smithi15186934-19 from 3403509 to 3428661 tid 3 2024-09-13T14:23:58.952 INFO:tasks.rados.rados.0.smithi151.stdout:771: write oid 44 current snap is 101 2024-09-13T14:23:58.952 INFO:tasks.rados.rados.0.smithi151.stdout:770: finishing write tid 1 to smithi15186934-19 2024-09-13T14:23:58.952 INFO:tasks.rados.rados.0.smithi151.stdout:770: finishing write tid 2 to smithi15186934-19 2024-09-13T14:23:58.952 INFO:tasks.rados.rados.0.smithi151.stdout:771: seq_num 324 ranges {645468=467502,1726754=565436,2798226=596838} 2024-09-13T14:23:58.958 INFO:tasks.rados.rados.0.smithi151.stdout:771: writing smithi15186934-44 from 645468 to 1112970 tid 1 2024-09-13T14:23:58.964 INFO:tasks.rados.rados.0.smithi151.stdout:771: writing smithi15186934-44 from 1726754 to 2292190 tid 2 2024-09-13T14:23:58.971 INFO:tasks.rados.rados.0.smithi151.stdout:771: writing smithi15186934-44 from 2798226 to 3395064 tid 3 2024-09-13T14:23:58.971 INFO:tasks.rados.rados.0.smithi151.stdout:772: snap_remove snap 94 2024-09-13T14:23:59.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:58 smithi151 ceph-mon[104450]: osdmap e255: 8 total, 8 up, 8 in 2024-09-13T14:23:59.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:58 smithi151 ceph-mon[41387]: osdmap e255: 8 total, 8 up, 8 in 2024-09-13T14:23:59.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:58 smithi184 ceph-mon[33394]: osdmap e255: 8 total, 8 up, 8 in 2024-09-13T14:23:59.438 INFO:tasks.rados.rados.0.smithi151.stdout:770: finishing write tid 3 to smithi15186934-19 2024-09-13T14:23:59.438 INFO:tasks.rados.rados.0.smithi151.stdout:770: finishing write tid 4 to smithi15186934-19 2024-09-13T14:23:59.438 INFO:tasks.rados.rados.0.smithi151.stdout:770: finishing write tid 5 to smithi15186934-19 2024-09-13T14:23:59.438 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 19 v 219 (ObjNum 323 snap 101 seq_num 323) dirty exists 2024-09-13T14:23:59.438 INFO:tasks.rados.rados.0.smithi151.stdout:770: left oid 19 (ObjNum 323 snap 101 seq_num 323) 2024-09-13T14:23:59.438 INFO:tasks.rados.rados.0.smithi151.stdout:771: finishing write tid 1 to smithi15186934-44 2024-09-13T14:23:59.438 INFO:tasks.rados.rados.0.smithi151.stdout:771: finishing write tid 2 to smithi15186934-44 2024-09-13T14:23:59.439 INFO:tasks.rados.rados.0.smithi151.stdout:771: finishing write tid 3 to smithi15186934-44 2024-09-13T14:23:59.439 INFO:tasks.rados.rados.0.smithi151.stdout:771: finishing write tid 4 to smithi15186934-44 2024-09-13T14:23:59.439 INFO:tasks.rados.rados.0.smithi151.stdout:771: finishing write tid 5 to smithi15186934-44 2024-09-13T14:23:59.439 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 229 (ObjNum 324 snap 101 seq_num 324) dirty exists 2024-09-13T14:23:59.439 INFO:tasks.rados.rados.0.smithi151.stdout:771: left oid 44 (ObjNum 324 snap 101 seq_num 324) 2024-09-13T14:23:59.439 INFO:tasks.rados.rados.0.smithi151.stdout:770: done (2 left) 2024-09-13T14:23:59.439 INFO:tasks.rados.rados.0.smithi151.stdout:771: done (1 left) 2024-09-13T14:23:59.439 INFO:tasks.rados.rados.0.smithi151.stdout:772: done (0 left) 2024-09-13T14:23:59.439 INFO:tasks.rados.rados.0.smithi151.stdout:773: delete oid 33 current snap is 101 2024-09-13T14:23:59.441 INFO:tasks.rados.rados.0.smithi151.stdout:773: done (0 left) 2024-09-13T14:23:59.442 INFO:tasks.rados.rados.0.smithi151.stdout:774: snap_remove snap 96 2024-09-13T14:24:00.206 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:59 smithi151 ceph-mon[104450]: osdmap e256: 8 total, 8 up, 8 in 2024-09-13T14:24:00.206 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:23:59 smithi151 ceph-mon[104450]: osdmap e257: 8 total, 8 up, 8 in 2024-09-13T14:24:00.207 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:59 smithi151 ceph-mon[41387]: osdmap e256: 8 total, 8 up, 8 in 2024-09-13T14:24:00.207 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:23:59 smithi151 ceph-mon[41387]: osdmap e257: 8 total, 8 up, 8 in 2024-09-13T14:24:00.212 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:59 smithi184 ceph-mon[33394]: osdmap e256: 8 total, 8 up, 8 in 2024-09-13T14:24:00.212 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:23:59 smithi184 ceph-mon[33394]: osdmap e257: 8 total, 8 up, 8 in 2024-09-13T14:24:00.445 INFO:tasks.rados.rados.0.smithi151.stdout:774: done (0 left) 2024-09-13T14:24:00.445 INFO:tasks.rados.rados.0.smithi151.stdout:775: write oid 24 current snap is 101 2024-09-13T14:24:00.445 INFO:tasks.rados.rados.0.smithi151.stdout:775: seq_num 326 ranges {679082=618000,1843718=569105,2853072=626010} 2024-09-13T14:24:00.451 INFO:tasks.rados.rados.0.smithi151.stdout:775: writing smithi15186934-24 from 679082 to 1297082 tid 1 2024-09-13T14:24:00.457 INFO:tasks.rados.rados.0.smithi151.stdout:775: writing smithi15186934-24 from 1843718 to 2412823 tid 2 2024-09-13T14:24:00.464 INFO:tasks.rados.rados.0.smithi151.stdout:775: writing smithi15186934-24 from 2853072 to 3479082 tid 3 2024-09-13T14:24:00.465 INFO:tasks.rados.rados.0.smithi151.stdout:776: read oid 19 snap 90 2024-09-13T14:24:00.465 INFO:tasks.rados.rados.0.smithi151.stdout:776: expect deleted 2024-09-13T14:24:00.465 INFO:tasks.rados.rados.0.smithi151.stdout:777: read oid 3 snap 99 2024-09-13T14:24:00.465 INFO:tasks.rados.rados.0.smithi151.stdout:777: expect deleted 2024-09-13T14:24:00.465 INFO:tasks.rados.rados.0.smithi151.stdout:778: read oid 43 snap -1 2024-09-13T14:24:00.465 INFO:tasks.rados.rados.0.smithi151.stdout:778: expect (ObjNum 263 snap 82 seq_num 263) 2024-09-13T14:24:00.465 INFO:tasks.rados.rados.0.smithi151.stdout:775: finishing write tid 1 to smithi15186934-24 2024-09-13T14:24:00.465 INFO:tasks.rados.rados.0.smithi151.stdout:779: read oid 46 snap -1 2024-09-13T14:24:00.465 INFO:tasks.rados.rados.0.smithi151.stdout:779: expect (ObjNum 261 snap 82 seq_num 261) 2024-09-13T14:24:00.465 INFO:tasks.rados.rados.0.smithi151.stdout:780: rollback oid 4 current snap is 101 2024-09-13T14:24:00.465 INFO:tasks.rados.rados.0.smithi151.stdout:775: finishing write tid 2 to smithi15186934-24 2024-09-13T14:24:00.465 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 4 to 80 2024-09-13T14:24:00.465 INFO:tasks.rados.rados.0.smithi151.stdout:781: write oid 18 current snap is 101 2024-09-13T14:24:00.465 INFO:tasks.rados.rados.0.smithi151.stdout:781: seq_num 327 ranges {695889=409602,1618553=712763,2996848=499041} 2024-09-13T14:24:00.470 INFO:tasks.rados.rados.0.smithi151.stdout:781: writing smithi15186934-18 from 695889 to 1105491 tid 1 2024-09-13T14:24:00.477 INFO:tasks.rados.rados.0.smithi151.stdout:781: writing smithi15186934-18 from 1618553 to 2331316 tid 2 2024-09-13T14:24:00.484 INFO:tasks.rados.rados.0.smithi151.stdout:781: writing smithi15186934-18 from 2996848 to 3495889 tid 3 2024-09-13T14:24:00.484 INFO:tasks.rados.rados.0.smithi151.stdout:782: snap_create 2024-09-13T14:24:00.485 INFO:tasks.rados.rados.0.smithi151.stdout:779: expect (ObjNum 261 snap 82 seq_num 261) 2024-09-13T14:24:00.590 INFO:tasks.rados.rados.0.smithi151.stdout:780: finishing rollback tid 0 to smithi15186934-4 2024-09-13T14:24:00.590 INFO:tasks.rados.rados.0.smithi151.stdout:780: finishing rollback tid 1 to smithi15186934-4 2024-09-13T14:24:00.590 INFO:tasks.rados.rados.0.smithi151.stdout:780: finishing rollback tid 2 to smithi15186934-4 2024-09-13T14:24:00.590 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 472 (ObjNum 246 snap 75 seq_num 246) dirty exists 2024-09-13T14:24:00.590 INFO:tasks.rados.rados.0.smithi151.stdout:775: finishing write tid 3 to smithi15186934-24 2024-09-13T14:24:00.590 INFO:tasks.rados.rados.0.smithi151.stdout:775: finishing write tid 4 to smithi15186934-24 2024-09-13T14:24:00.590 INFO:tasks.rados.rados.0.smithi151.stdout:775: finishing write tid 5 to smithi15186934-24 2024-09-13T14:24:00.590 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 24 v 505 (ObjNum 326 snap 101 seq_num 326) dirty exists 2024-09-13T14:24:00.590 INFO:tasks.rados.rados.0.smithi151.stdout:775: left oid 24 (ObjNum 326 snap 101 seq_num 326) 2024-09-13T14:24:00.591 INFO:tasks.rados.rados.0.smithi151.stdout:781: finishing write tid 1 to smithi15186934-18 2024-09-13T14:24:00.591 INFO:tasks.rados.rados.0.smithi151.stdout:778: expect (ObjNum 263 snap 82 seq_num 263) 2024-09-13T14:24:00.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:24:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:24:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:24:00.699 INFO:tasks.rados.rados.0.smithi151.stdout:781: finishing write tid 2 to smithi15186934-18 2024-09-13T14:24:00.699 INFO:tasks.rados.rados.0.smithi151.stdout:781: finishing write tid 3 to smithi15186934-18 2024-09-13T14:24:00.699 INFO:tasks.rados.rados.0.smithi151.stdout:781: finishing write tid 4 to smithi15186934-18 2024-09-13T14:24:00.699 INFO:tasks.rados.rados.0.smithi151.stdout:781: finishing write tid 5 to smithi15186934-18 2024-09-13T14:24:00.699 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 18 v 216 (ObjNum 327 snap 101 seq_num 327) dirty exists 2024-09-13T14:24:00.699 INFO:tasks.rados.rados.0.smithi151.stdout:781: left oid 18 (ObjNum 327 snap 101 seq_num 327) 2024-09-13T14:24:01.085 INFO:tasks.rados.rados.0.smithi151.stdout:775: done (7 left) 2024-09-13T14:24:01.085 INFO:tasks.rados.rados.0.smithi151.stdout:776: done (6 left) 2024-09-13T14:24:01.085 INFO:tasks.rados.rados.0.smithi151.stdout:777: done (5 left) 2024-09-13T14:24:01.085 INFO:tasks.rados.rados.0.smithi151.stdout:778: done (4 left) 2024-09-13T14:24:01.086 INFO:tasks.rados.rados.0.smithi151.stdout:779: done (3 left) 2024-09-13T14:24:01.086 INFO:tasks.rados.rados.0.smithi151.stdout:780: done (2 left) 2024-09-13T14:24:01.086 INFO:tasks.rados.rados.0.smithi151.stdout:781: done (1 left) 2024-09-13T14:24:01.086 INFO:tasks.rados.rados.0.smithi151.stdout:782: done (0 left) 2024-09-13T14:24:01.086 INFO:tasks.rados.rados.0.smithi151.stdout:783: snap_remove snap 78 2024-09-13T14:24:01.353 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:24:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[82046]: [13/Sep/2024:14:24:00] ENGINE Bus STARTING 2024-09-13T14:24:01.354 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:24:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[82046]: CherryPy Checker: 2024-09-13T14:24:01.354 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:24:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[82046]: The Application mounted at '' has an empty config. 2024-09-13T14:24:01.354 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:24:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[82046]: 2024-09-13T14:24:01.354 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:24:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[82046]: [13/Sep/2024:14:24:00] ENGINE Serving on http://:::9283 2024-09-13T14:24:01.354 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 14:24:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x[82046]: [13/Sep/2024:14:24:00] ENGINE Bus STARTED 2024-09-13T14:24:01.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: osdmap e258: 8 total, 8 up, 8 in 2024-09-13T14:24:01.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: Standby manager daemon x restarted 2024-09-13T14:24:01.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: Standby manager daemon x started 2024-09-13T14:24:01.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.184:0/4100685425' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:24:01.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.184:0/4100685425' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:24:01.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.184:0/4100685425' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:24:01.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: from='mgr.? 172.21.15.184:0/4100685425' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:24:01.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: Active manager daemon y restarted 2024-09-13T14:24:01.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: Activating manager daemon y 2024-09-13T14:24:01.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: osdmap e259: 8 total, 8 up, 8 in 2024-09-13T14:24:01.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: mgrmap e34: y(active, starting, since 0.00829774s), standbys: x 2024-09-13T14:24:01.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T14:24:01.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T14:24:01.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T14:24:01.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi184.jismsi"} : dispatch 2024-09-13T14:24:01.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi151.rqqklw"} : dispatch 2024-09-13T14:24:01.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-13T14:24:01.856 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-13T14:24:01.856 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T14:24:01.856 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T14:24:01.856 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T14:24:01.856 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T14:24:01.856 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-13T14:24:01.856 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-13T14:24:01.856 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T14:24:01.856 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-13T14:24:01.856 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-13T14:24:01.856 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-13T14:24:01.856 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-13T14:24:01.857 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: Manager daemon y is now available 2024-09-13T14:24:01.857 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:24:01.857 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:24:01.857 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-13T14:24:01.857 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-13T14:24:01.857 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:01 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-13T14:24:01.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: osdmap e258: 8 total, 8 up, 8 in 2024-09-13T14:24:01.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: Standby manager daemon x restarted 2024-09-13T14:24:01.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: Standby manager daemon x started 2024-09-13T14:24:01.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: from='mgr.? 172.21.15.184:0/4100685425' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:24:01.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: from='mgr.? 172.21.15.184:0/4100685425' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:24:01.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: from='mgr.? 172.21.15.184:0/4100685425' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:24:01.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: from='mgr.? 172.21.15.184:0/4100685425' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:24:01.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: Active manager daemon y restarted 2024-09-13T14:24:01.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: Activating manager daemon y 2024-09-13T14:24:01.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: osdmap e259: 8 total, 8 up, 8 in 2024-09-13T14:24:01.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: mgrmap e34: y(active, starting, since 0.00829774s), standbys: x 2024-09-13T14:24:01.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T14:24:01.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T14:24:01.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T14:24:01.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi184.jismsi"} : dispatch 2024-09-13T14:24:01.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi151.rqqklw"} : dispatch 2024-09-13T14:24:01.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-13T14:24:01.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-13T14:24:01.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T14:24:01.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T14:24:01.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T14:24:01.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T14:24:01.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-13T14:24:01.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-13T14:24:01.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T14:24:01.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-13T14:24:01.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-13T14:24:01.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-13T14:24:01.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-13T14:24:01.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: Manager daemon y is now available 2024-09-13T14:24:01.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:24:01.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:24:01.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-13T14:24:01.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-13T14:24:01.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-13T14:24:01.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: osdmap e258: 8 total, 8 up, 8 in 2024-09-13T14:24:01.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: Standby manager daemon x restarted 2024-09-13T14:24:01.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: Standby manager daemon x started 2024-09-13T14:24:01.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.184:0/4100685425' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:24:01.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.184:0/4100685425' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:24:01.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.184:0/4100685425' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:24:01.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: from='mgr.? 172.21.15.184:0/4100685425' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:24:01.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: Active manager daemon y restarted 2024-09-13T14:24:01.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: Activating manager daemon y 2024-09-13T14:24:01.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: osdmap e259: 8 total, 8 up, 8 in 2024-09-13T14:24:01.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: mgrmap e34: y(active, starting, since 0.00829774s), standbys: x 2024-09-13T14:24:01.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T14:24:01.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T14:24:01.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T14:24:01.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi184.jismsi"} : dispatch 2024-09-13T14:24:01.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi151.rqqklw"} : dispatch 2024-09-13T14:24:01.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-13T14:24:01.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-13T14:24:01.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T14:24:01.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T14:24:01.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T14:24:01.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T14:24:01.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-13T14:24:01.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-13T14:24:01.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T14:24:01.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-13T14:24:01.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-13T14:24:01.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-13T14:24:01.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-13T14:24:01.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: Manager daemon y is now available 2024-09-13T14:24:01.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:24:01.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:24:01.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-13T14:24:01.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-13T14:24:01.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-13T14:24:01.865 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:24:01.422+0000 7fc91b624640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:24:01.865 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:24:01.422+0000 7fc91b624640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:24:01.865 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:24:01.422+0000 7fc91b624640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:24:01.865 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:24:01.422+0000 7fc91b624640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:24:01.865 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:24:01.422+0000 7fc91b624640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:24:01.865 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:24:01.423+0000 7fc91861e640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:24:01.865 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:24:01.423+0000 7fc91861e640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:24:01.865 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:24:01.423+0000 7fc91861e640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:24:01.865 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:24:01.423+0000 7fc91861e640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:24:01.865 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:01 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:24:01.423+0000 7fc91861e640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:24:02.112 INFO:tasks.rados.rados.0.smithi151.stdout:783: done (0 left) 2024-09-13T14:24:02.112 INFO:tasks.rados.rados.0.smithi151.stdout:784: read oid 47 snap 97 2024-09-13T14:24:02.112 INFO:tasks.rados.rados.0.smithi151.stdout:784: expect (ObjNum 155 snap 38 seq_num 155) 2024-09-13T14:24:02.112 INFO:tasks.rados.rados.0.smithi151.stdout:785: read oid 28 snap 86 2024-09-13T14:24:02.112 INFO:tasks.rados.rados.0.smithi151.stdout:785: expect (ObjNum 173 snap 46 seq_num 173) 2024-09-13T14:24:02.113 INFO:tasks.rados.rados.0.smithi151.stdout:786: snap_remove snap 88 2024-09-13T14:24:02.120 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:24:02.120 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.a smithi151 *:9093,9094 running (85s) 36s ago 5m 16.0M - 0.27.0 11f11916f8cd ed70c3d75107 2024-09-13T14:24:02.120 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.a smithi184 *:3000 running (60s) 17s ago 5m 80.3M - 9.4.7 954c08fa6188 4ba18fb1f1fc 2024-09-13T14:24:02.120 INFO:teuthology.orchestra.run.smithi151.stdout:mds.foo.smithi151.rqqklw smithi151 running (3m) 36s ago 3m 16.1M - 17.2.7-1627-gbbdadfa3 a994b00deb3a c960b57fc84b 2024-09-13T14:24:02.120 INFO:teuthology.orchestra.run.smithi151.stdout:mds.foo.smithi184.jismsi smithi184 running (3m) 17s ago 3m 14.3M - 17.2.7-1627-gbbdadfa3 a994b00deb3a 0cc31a3a5339 2024-09-13T14:24:02.120 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.x smithi184 *:8443,9283,8765 running (22s) 17s ago 9m 65.2M - 19.3.0-4697-gd58f7306 fec07016a2b0 8a0f7d440b6e 2024-09-13T14:24:02.120 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.y smithi151 *:8443,9283,8765 running (50s) 36s ago 11m 490M - 19.3.0-4697-gd58f7306 fec07016a2b0 d8c169a4f36e 2024-09-13T14:24:02.120 INFO:teuthology.orchestra.run.smithi151.stdout:mon.a smithi151 running (11m) 36s ago 11m 55.7M 2048M 17.2.7-1627-gbbdadfa3 a994b00deb3a ffc441eb291c 2024-09-13T14:24:02.120 INFO:teuthology.orchestra.run.smithi151.stdout:mon.b smithi184 running (9m) 17s ago 9m 48.5M 2048M 17.2.7-1627-gbbdadfa3 a994b00deb3a 8c637f147650 2024-09-13T14:24:02.120 INFO:teuthology.orchestra.run.smithi151.stdout:mon.c smithi151 running (9m) 36s ago 9m 45.8M 2048M 17.2.7-1627-gbbdadfa3 a994b00deb3a dccc3a65ea4d 2024-09-13T14:24:02.121 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.a smithi151 *:9100 running (79s) 36s ago 4m 8220k - 1.7.0 72c9c2088986 d91e4cb09e9a 2024-09-13T14:24:02.121 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.b smithi184 *:9100 running (55s) 17s ago 4m 8274k - 1.7.0 72c9c2088986 1e1a9dd643a0 2024-09-13T14:24:02.121 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (9m) 36s ago 9m 76.3M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a e6951a1a86d9 2024-09-13T14:24:02.121 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (8m) 36s ago 8m 63.1M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a 94bc726b3eb5 2024-09-13T14:24:02.121 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (8m) 36s ago 8m 82.9M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a 122660181e46 2024-09-13T14:24:02.121 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi151 running (8m) 36s ago 8m 55.6M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a 495241a87df6 2024-09-13T14:24:02.121 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi184 running (7m) 17s ago 7m 76.0M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a 6e70699b8084 2024-09-13T14:24:02.121 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi184 running (7m) 17s ago 7m 75.6M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a 36b82c9e7230 2024-09-13T14:24:02.121 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6 smithi184 running (6m) 17s ago 6m 62.7M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a f9379cec7504 2024-09-13T14:24:02.121 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7 smithi184 running (6m) 17s ago 6m 67.8M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a f37cc947097b 2024-09-13T14:24:02.121 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.a smithi184 *:9095 running (26s) 17s ago 5m 41.5M - 2.51.0 1d3b7f56885b 3da9265b8f14 2024-09-13T14:24:02.358 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:24:02.095+0000 7fc94c153640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-13T14:24:02.358 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: [13/Sep/2024:14:24:02] ENGINE Bus STARTING 2024-09-13T14:24:02.358 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: CherryPy Checker: 2024-09-13T14:24:02.359 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: The Application mounted at '' has an empty config. 2024-09-13T14:24:02.359 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:24:02.359 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: [13/Sep/2024:14:24:02] ENGINE Serving on http://:::9283 2024-09-13T14:24:02.359 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: [13/Sep/2024:14:24:02] ENGINE Bus STARTED 2024-09-13T14:24:02.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:24:02.607+0000 7fc8fa428640 -1 log_channel(cephadm) log [ERR] : [13/Sep/2024:14:24:02] ENGINE Error in HTTPServer.serve 2024-09-13T14:24:02.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: Traceback (most recent call last): 2024-09-13T14:24:02.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:24:02.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: self._connections.run(self.expiration_interval) 2024-09-13T14:24:02.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:24:02.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: self._run(expiration_interval) 2024-09-13T14:24:02.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:24:02.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:24:02.860 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:24:02.860 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:24:02.860 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:24:02.860 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: s = self.context.wrap_socket( 2024-09-13T14:24:02.860 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:24:02.860 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: return self.sslsocket_class._create( 2024-09-13T14:24:02.860 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:24:02.860 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: self.do_handshake() 2024-09-13T14:24:02.860 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:24:02.860 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: self._sslobj.do_handshake() 2024-09-13T14:24:02.860 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:24:02.861 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:24:02.939 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch upgrade pause' 2024-09-13T14:24:03.105 INFO:tasks.rados.rados.0.smithi151.stdout:786: done (2 left) 2024-09-13T14:24:03.105 INFO:tasks.rados.rados.0.smithi151.stdout:787: write oid 36 current snap is 102 2024-09-13T14:24:03.105 INFO:tasks.rados.rados.0.smithi151.stdout:787: seq_num 328 ranges {712696=484851,1960682=572774,3307918=204778} 2024-09-13T14:24:03.113 INFO:tasks.rados.rados.0.smithi151.stdout:787: writing smithi15186934-36 from 712696 to 1197547 tid 1 2024-09-13T14:24:03.121 INFO:tasks.rados.rados.0.smithi151.stdout:787: writing smithi15186934-36 from 1960682 to 2533456 tid 2 2024-09-13T14:24:03.126 INFO:tasks.rados.rados.0.smithi151.stdout:787: writing smithi15186934-36 from 3307918 to 3512696 tid 3 2024-09-13T14:24:03.126 INFO:tasks.rados.rados.0.smithi151.stdout:788: snap_create 2024-09-13T14:24:03.126 INFO:tasks.rados.rados.0.smithi151.stdout:785: expect (ObjNum 173 snap 46 seq_num 173) 2024-09-13T14:24:03.208 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:03 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:03.208 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:03 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:03.209 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:03 smithi151 ceph-mon[41387]: mgrmap e35: y(active, since 1.01449s), standbys: x 2024-09-13T14:24:03.209 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:03 smithi151 ceph-mon[41387]: osdmap e260: 8 total, 8 up, 8 in 2024-09-13T14:24:03.209 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:03 smithi151 ceph-mon[41387]: [13/Sep/2024:14:24:02] ENGINE Bus STARTING 2024-09-13T14:24:03.209 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:03 smithi151 ceph-mon[41387]: [13/Sep/2024:14:24:02] ENGINE Serving on http://172.21.15.151:8765 2024-09-13T14:24:03.209 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:03 smithi151 ceph-mon[41387]: [13/Sep/2024:14:24:02] ENGINE Serving on https://172.21.15.151:7150 2024-09-13T14:24:03.209 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:03 smithi151 ceph-mon[41387]: [13/Sep/2024:14:24:02] ENGINE Bus STARTED 2024-09-13T14:24:03.209 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:03 smithi151 ceph-mon[41387]: [13/Sep/2024:14:24:02] ENGINE Error in HTTPServer.serve 2024-09-13T14:24:03.209 INFO:journalctl@ceph.mon.c.smithi151.stdout: Traceback (most recent call last): 2024-09-13T14:24:03.209 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:24:03.209 INFO:journalctl@ceph.mon.c.smithi151.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:24:03.209 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:24:03.209 INFO:journalctl@ceph.mon.c.smithi151.stdout: self._run(expiration_interval) 2024-09-13T14:24:03.210 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:24:03.210 INFO:journalctl@ceph.mon.c.smithi151.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:24:03.210 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:24:03.210 INFO:journalctl@ceph.mon.c.smithi151.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:24:03.210 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:24:03.210 INFO:journalctl@ceph.mon.c.smithi151.stdout: s = self.context.wrap_socket( 2024-09-13T14:24:03.210 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:24:03.210 INFO:journalctl@ceph.mon.c.smithi151.stdout: return self.sslsocket_class._create( 2024-09-13T14:24:03.210 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:24:03.210 INFO:journalctl@ceph.mon.c.smithi151.stdout: self.do_handshake() 2024-09-13T14:24:03.210 INFO:journalctl@ceph.mon.c.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:24:03.210 INFO:journalctl@ceph.mon.c.smithi151.stdout: self._sslobj.do_handshake() 2024-09-13T14:24:03.210 INFO:journalctl@ceph.mon.c.smithi151.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:24:03.211 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:03 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:03.211 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:03 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:03.211 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:03 smithi151 ceph-mon[104450]: mgrmap e35: y(active, since 1.01449s), standbys: x 2024-09-13T14:24:03.211 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:03 smithi151 ceph-mon[104450]: osdmap e260: 8 total, 8 up, 8 in 2024-09-13T14:24:03.211 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:03 smithi151 ceph-mon[104450]: [13/Sep/2024:14:24:02] ENGINE Bus STARTING 2024-09-13T14:24:03.211 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:03 smithi151 ceph-mon[104450]: [13/Sep/2024:14:24:02] ENGINE Serving on http://172.21.15.151:8765 2024-09-13T14:24:03.211 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:03 smithi151 ceph-mon[104450]: [13/Sep/2024:14:24:02] ENGINE Serving on https://172.21.15.151:7150 2024-09-13T14:24:03.211 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:03 smithi151 ceph-mon[104450]: [13/Sep/2024:14:24:02] ENGINE Bus STARTED 2024-09-13T14:24:03.211 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:03 smithi151 ceph-mon[104450]: [13/Sep/2024:14:24:02] ENGINE Error in HTTPServer.serve 2024-09-13T14:24:03.211 INFO:journalctl@ceph.mon.a.smithi151.stdout: Traceback (most recent call last): 2024-09-13T14:24:03.212 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:24:03.212 INFO:journalctl@ceph.mon.a.smithi151.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:24:03.212 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:24:03.212 INFO:journalctl@ceph.mon.a.smithi151.stdout: self._run(expiration_interval) 2024-09-13T14:24:03.212 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:24:03.212 INFO:journalctl@ceph.mon.a.smithi151.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:24:03.212 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:24:03.212 INFO:journalctl@ceph.mon.a.smithi151.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:24:03.212 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:24:03.213 INFO:journalctl@ceph.mon.a.smithi151.stdout: s = self.context.wrap_socket( 2024-09-13T14:24:03.213 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:24:03.213 INFO:journalctl@ceph.mon.a.smithi151.stdout: return self.sslsocket_class._create( 2024-09-13T14:24:03.213 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:24:03.213 INFO:journalctl@ceph.mon.a.smithi151.stdout: self.do_handshake() 2024-09-13T14:24:03.213 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:24:03.213 INFO:journalctl@ceph.mon.a.smithi151.stdout: self._sslobj.do_handshake() 2024-09-13T14:24:03.213 INFO:journalctl@ceph.mon.a.smithi151.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:24:03.261 INFO:tasks.rados.rados.0.smithi151.stdout:784: expect (ObjNum 155 snap 38 seq_num 155) 2024-09-13T14:24:03.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:03 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:03.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:03 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:03.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:03 smithi184 ceph-mon[33394]: mgrmap e35: y(active, since 1.01449s), standbys: x 2024-09-13T14:24:03.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:03 smithi184 ceph-mon[33394]: osdmap e260: 8 total, 8 up, 8 in 2024-09-13T14:24:03.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:03 smithi184 ceph-mon[33394]: [13/Sep/2024:14:24:02] ENGINE Bus STARTING 2024-09-13T14:24:03.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:03 smithi184 ceph-mon[33394]: [13/Sep/2024:14:24:02] ENGINE Serving on http://172.21.15.151:8765 2024-09-13T14:24:03.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:03 smithi184 ceph-mon[33394]: [13/Sep/2024:14:24:02] ENGINE Serving on https://172.21.15.151:7150 2024-09-13T14:24:03.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:03 smithi184 ceph-mon[33394]: [13/Sep/2024:14:24:02] ENGINE Bus STARTED 2024-09-13T14:24:03.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:03 smithi184 ceph-mon[33394]: [13/Sep/2024:14:24:02] ENGINE Error in HTTPServer.serve 2024-09-13T14:24:03.354 INFO:journalctl@ceph.mon.b.smithi184.stdout: Traceback (most recent call last): 2024-09-13T14:24:03.354 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:24:03.354 INFO:journalctl@ceph.mon.b.smithi184.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:24:03.354 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:24:03.354 INFO:journalctl@ceph.mon.b.smithi184.stdout: self._run(expiration_interval) 2024-09-13T14:24:03.354 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:24:03.354 INFO:journalctl@ceph.mon.b.smithi184.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:24:03.354 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:24:03.355 INFO:journalctl@ceph.mon.b.smithi184.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:24:03.355 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:24:03.355 INFO:journalctl@ceph.mon.b.smithi184.stdout: s = self.context.wrap_socket( 2024-09-13T14:24:03.355 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:24:03.355 INFO:journalctl@ceph.mon.b.smithi184.stdout: return self.sslsocket_class._create( 2024-09-13T14:24:03.355 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:24:03.355 INFO:journalctl@ceph.mon.b.smithi184.stdout: self.do_handshake() 2024-09-13T14:24:03.355 INFO:journalctl@ceph.mon.b.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:24:03.355 INFO:journalctl@ceph.mon.b.smithi184.stdout: self._sslobj.do_handshake() 2024-09-13T14:24:03.355 INFO:journalctl@ceph.mon.b.smithi184.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:24:03.386 INFO:tasks.rados.rados.0.smithi151.stdout:787: finishing write tid 1 to smithi15186934-36 2024-09-13T14:24:03.386 INFO:tasks.rados.rados.0.smithi151.stdout:787: finishing write tid 2 to smithi15186934-36 2024-09-13T14:24:03.386 INFO:tasks.rados.rados.0.smithi151.stdout:787: finishing write tid 3 to smithi15186934-36 2024-09-13T14:24:03.386 INFO:tasks.rados.rados.0.smithi151.stdout:787: finishing write tid 4 to smithi15186934-36 2024-09-13T14:24:03.386 INFO:tasks.rados.rados.0.smithi151.stdout:787: finishing write tid 5 to smithi15186934-36 2024-09-13T14:24:03.386 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 36 v 161 (ObjNum 328 snap 102 seq_num 328) dirty exists 2024-09-13T14:24:03.386 INFO:tasks.rados.rados.0.smithi151.stdout:787: left oid 36 (ObjNum 328 snap 102 seq_num 328) 2024-09-13T14:24:03.641 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:24:04.110 INFO:tasks.rados.rados.0.smithi151.stdout:784: done (3 left) 2024-09-13T14:24:04.111 INFO:tasks.rados.rados.0.smithi151.stdout:785: done (2 left) 2024-09-13T14:24:04.111 INFO:tasks.rados.rados.0.smithi151.stdout:787: done (1 left) 2024-09-13T14:24:04.111 INFO:tasks.rados.rados.0.smithi151.stdout:788: done (0 left) 2024-09-13T14:24:04.111 INFO:tasks.rados.rados.0.smithi151.stdout:789: delete oid 3 current snap is 103 2024-09-13T14:24:04.113 INFO:tasks.rados.rados.0.smithi151.stdout:789: done (0 left) 2024-09-13T14:24:04.113 INFO:tasks.rados.rados.0.smithi151.stdout:790: delete oid 4 current snap is 103 2024-09-13T14:24:04.118 INFO:tasks.rados.rados.0.smithi151.stdout:790: done (0 left) 2024-09-13T14:24:04.118 INFO:tasks.rados.rados.0.smithi151.stdout:791: read oid 46 snap 82 2024-09-13T14:24:04.118 INFO:tasks.rados.rados.0.smithi151.stdout:791: expect (ObjNum 261 snap 82 seq_num 261) 2024-09-13T14:24:04.118 INFO:tasks.rados.rados.0.smithi151.stdout:792: delete oid 9 current snap is 103 2024-09-13T14:24:04.120 INFO:tasks.rados.rados.0.smithi151.stdout:792: done (1 left) 2024-09-13T14:24:04.120 INFO:tasks.rados.rados.0.smithi151.stdout:793: read oid 3 snap 101 2024-09-13T14:24:04.120 INFO:tasks.rados.rados.0.smithi151.stdout:793: expect deleted 2024-09-13T14:24:04.120 INFO:tasks.rados.rados.0.smithi151.stdout:794: read oid 39 snap -1 2024-09-13T14:24:04.120 INFO:tasks.rados.rados.0.smithi151.stdout:794: expect deleted 2024-09-13T14:24:04.120 INFO:tasks.rados.rados.0.smithi151.stdout:795: snap_remove snap 80 2024-09-13T14:24:04.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:04 smithi184 ceph-mon[33394]: pgmap v5: 89 pgs: 1 active+clean+snaptrim, 88 active+clean; 198 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:24:04.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:04 smithi184 ceph-mon[33394]: osdmap e261: 8 total, 8 up, 8 in 2024-09-13T14:24:04.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:04 smithi184 ceph-mon[33394]: mgrmap e36: y(active, since 3s), standbys: x 2024-09-13T14:24:04.354 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:24:04 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:24:04.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:24:04.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:04 smithi151 ceph-mon[104450]: pgmap v5: 89 pgs: 1 active+clean+snaptrim, 88 active+clean; 198 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:24:04.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:04 smithi151 ceph-mon[104450]: osdmap e261: 8 total, 8 up, 8 in 2024-09-13T14:24:04.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:04 smithi151 ceph-mon[104450]: mgrmap e36: y(active, since 3s), standbys: x 2024-09-13T14:24:04.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:04 smithi151 ceph-mon[41387]: pgmap v5: 89 pgs: 1 active+clean+snaptrim, 88 active+clean; 198 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:24:04.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:04 smithi151 ceph-mon[41387]: osdmap e261: 8 total, 8 up, 8 in 2024-09-13T14:24:04.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:04 smithi151 ceph-mon[41387]: mgrmap e36: y(active, since 3s), standbys: x 2024-09-13T14:24:04.437 INFO:tasks.rados.rados.0.smithi151.stdout:795: done (3 left) 2024-09-13T14:24:04.437 INFO:tasks.rados.rados.0.smithi151.stdout:796: rollback oid 26 current snap is 103 2024-09-13T14:24:04.437 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 26 to 86 2024-09-13T14:24:04.438 INFO:tasks.rados.rados.0.smithi151.stdout:791: expect (ObjNum 261 snap 82 seq_num 261) 2024-09-13T14:24:04.545 INFO:tasks.rados.rados.0.smithi151.stdout:796: finishing rollback tid 0 to smithi15186934-26 2024-09-13T14:24:04.545 INFO:tasks.rados.rados.0.smithi151.stdout:796: finishing rollback tid 1 to smithi15186934-26 2024-09-13T14:24:04.545 INFO:tasks.rados.rados.0.smithi151.stdout:796: finishing rollback tid 2 to smithi15186934-26 2024-09-13T14:24:04.545 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 259 (ObjNum 148 snap 34 seq_num 148) dirty exists 2024-09-13T14:24:04.545 INFO:tasks.rados.rados.0.smithi151.stdout:791: done (3 left) 2024-09-13T14:24:04.545 INFO:tasks.rados.rados.0.smithi151.stdout:793: done (2 left) 2024-09-13T14:24:04.545 INFO:tasks.rados.rados.0.smithi151.stdout:794: done (1 left) 2024-09-13T14:24:04.545 INFO:tasks.rados.rados.0.smithi151.stdout:796: done (0 left) 2024-09-13T14:24:04.546 INFO:tasks.rados.rados.0.smithi151.stdout:797: write oid 36 current snap is 103 2024-09-13T14:24:04.546 INFO:tasks.rados.rados.0.smithi151.stdout:797: seq_num 332 ranges {779924=785847,2078257=580112,3301257=278667} 2024-09-13T14:24:04.553 INFO:tasks.rados.rados.0.smithi151.stdout:797: writing smithi15186934-36 from 779924 to 1565771 tid 1 2024-09-13T14:24:04.559 INFO:tasks.rados.rados.0.smithi151.stdout:797: writing smithi15186934-36 from 2078257 to 2658369 tid 2 2024-09-13T14:24:04.564 INFO:tasks.rados.rados.0.smithi151.stdout:797: writing smithi15186934-36 from 3301257 to 3579924 tid 3 2024-09-13T14:24:04.564 INFO:tasks.rados.rados.0.smithi151.stdout:798: snap_remove snap 97 2024-09-13T14:24:04.820 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:24:04] "GET /metrics HTTP/1.1" 200 33658 "" "Prometheus/2.51.0" 2024-09-13T14:24:05.431 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:05 smithi151 ceph-mon[104450]: osdmap e262: 8 total, 8 up, 8 in 2024-09-13T14:24:05.431 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:05 smithi151 ceph-mon[104450]: osdmap e263: 8 total, 8 up, 8 in 2024-09-13T14:24:05.432 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:05 smithi151 ceph-mon[41387]: osdmap e262: 8 total, 8 up, 8 in 2024-09-13T14:24:05.432 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:05 smithi151 ceph-mon[41387]: osdmap e263: 8 total, 8 up, 8 in 2024-09-13T14:24:05.452 INFO:tasks.rados.rados.0.smithi151.stdout:798: done (1 left) 2024-09-13T14:24:05.453 INFO:tasks.rados.rados.0.smithi151.stdout:799: write oid 47 current snap is 103 2024-09-13T14:24:05.453 INFO:tasks.rados.rados.0.smithi151.stdout:797: finishing write tid 1 to smithi15186934-36 2024-09-13T14:24:05.453 INFO:tasks.rados.rados.0.smithi151.stdout:799: seq_num 333 ranges {796731=461096,1736739=723770,2928680=668051} 2024-09-13T14:24:05.459 INFO:tasks.rados.rados.0.smithi151.stdout:799: writing smithi15186934-47 from 796731 to 1257827 tid 1 2024-09-13T14:24:05.466 INFO:tasks.rados.rados.0.smithi151.stdout:799: writing smithi15186934-47 from 1736739 to 2460509 tid 2 2024-09-13T14:24:05.473 INFO:tasks.rados.rados.0.smithi151.stdout:799: writing smithi15186934-47 from 2928680 to 3596731 tid 3 2024-09-13T14:24:05.474 INFO:tasks.rados.rados.0.smithi151.stdout:797: finishing write tid 2 to smithi15186934-36 2024-09-13T14:24:05.474 INFO:tasks.rados.rados.0.smithi151.stdout:797: finishing write tid 3 to smithi15186934-36 2024-09-13T14:24:05.474 INFO:tasks.rados.rados.0.smithi151.stdout:797: finishing write tid 4 to smithi15186934-36 2024-09-13T14:24:05.474 INFO:tasks.rados.rados.0.smithi151.stdout:797: finishing write tid 5 to smithi15186934-36 2024-09-13T14:24:05.474 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 36 v 170 (ObjNum 332 snap 103 seq_num 332) dirty exists 2024-09-13T14:24:05.474 INFO:tasks.rados.rados.0.smithi151.stdout:797: left oid 36 (ObjNum 332 snap 103 seq_num 332) 2024-09-13T14:24:05.474 INFO:tasks.rados.rados.0.smithi151.stdout:800: read oid 19 snap 99 2024-09-13T14:24:05.474 INFO:tasks.rados.rados.0.smithi151.stdout:800: expect deleted 2024-09-13T14:24:05.474 INFO:tasks.rados.rados.0.smithi151.stdout:797: done (2 left) 2024-09-13T14:24:05.475 INFO:tasks.rados.rados.0.smithi151.stdout:801: snap_remove snap 90 2024-09-13T14:24:05.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:05 smithi184 ceph-mon[33394]: osdmap e262: 8 total, 8 up, 8 in 2024-09-13T14:24:05.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:05 smithi184 ceph-mon[33394]: osdmap e263: 8 total, 8 up, 8 in 2024-09-13T14:24:06.038 INFO:teuthology.orchestra.run.smithi151.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:24:06.061 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:24:05 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:24:05.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:24:06.061 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:24:05 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:24:05.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:24:06.455 INFO:tasks.rados.rados.0.smithi151.stdout:801: done (2 left) 2024-09-13T14:24:06.455 INFO:tasks.rados.rados.0.smithi151.stdout:802: write oid 3 current snap is 103 2024-09-13T14:24:06.455 INFO:tasks.rados.rados.0.smithi151.stdout:802: seq_num 334 ranges {413538=536345,1395221=583781,2556103=658617} 2024-09-13T14:24:06.460 INFO:tasks.rados.rados.0.smithi151.stdout:802: writing smithi15186934-3 from 413538 to 949883 tid 1 2024-09-13T14:24:06.466 INFO:tasks.rados.rados.0.smithi151.stdout:802: writing smithi15186934-3 from 1395221 to 1979002 tid 2 2024-09-13T14:24:06.473 INFO:tasks.rados.rados.0.smithi151.stdout:802: writing smithi15186934-3 from 2556103 to 3214720 tid 3 2024-09-13T14:24:06.473 INFO:tasks.rados.rados.0.smithi151.stdout:803: write oid 15 current snap is 103 2024-09-13T14:24:06.473 INFO:tasks.rados.rados.0.smithi151.stdout:803: seq_num 335 ranges {430345=727947,1570056=727439,2699879=469889} 2024-09-13T14:24:06.480 INFO:tasks.rados.rados.0.smithi151.stdout:803: writing smithi15186934-15 from 430345 to 1158292 tid 1 2024-09-13T14:24:06.486 INFO:tasks.rados.rados.0.smithi151.stdout:803: writing smithi15186934-15 from 1570056 to 2297495 tid 2 2024-09-13T14:24:06.491 INFO:tasks.rados.rados.0.smithi151.stdout:803: writing smithi15186934-15 from 2699879 to 3169768 tid 3 2024-09-13T14:24:06.491 INFO:tasks.rados.rados.0.smithi151.stdout:804: snap_create 2024-09-13T14:24:06.491 INFO:tasks.rados.rados.0.smithi151.stdout:799: finishing write tid 1 to smithi15186934-47 2024-09-13T14:24:06.491 INFO:tasks.rados.rados.0.smithi151.stdout:799: finishing write tid 2 to smithi15186934-47 2024-09-13T14:24:06.491 INFO:tasks.rados.rados.0.smithi151.stdout:799: finishing write tid 3 to smithi15186934-47 2024-09-13T14:24:06.492 INFO:tasks.rados.rados.0.smithi151.stdout:799: finishing write tid 4 to smithi15186934-47 2024-09-13T14:24:06.492 INFO:tasks.rados.rados.0.smithi151.stdout:799: finishing write tid 5 to smithi15186934-47 2024-09-13T14:24:06.492 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 405 (ObjNum 333 snap 103 seq_num 333) dirty exists 2024-09-13T14:24:06.492 INFO:tasks.rados.rados.0.smithi151.stdout:799: left oid 47 (ObjNum 333 snap 103 seq_num 333) 2024-09-13T14:24:06.492 INFO:tasks.rados.rados.0.smithi151.stdout:802: finishing write tid 1 to smithi15186934-3 2024-09-13T14:24:06.492 INFO:tasks.rados.rados.0.smithi151.stdout:802: finishing write tid 2 to smithi15186934-3 2024-09-13T14:24:06.492 INFO:tasks.rados.rados.0.smithi151.stdout:802: finishing write tid 3 to smithi15186934-3 2024-09-13T14:24:06.492 INFO:tasks.rados.rados.0.smithi151.stdout:802: finishing write tid 4 to smithi15186934-3 2024-09-13T14:24:06.492 INFO:tasks.rados.rados.0.smithi151.stdout:802: finishing write tid 5 to smithi15186934-3 2024-09-13T14:24:06.492 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 361 (ObjNum 334 snap 103 seq_num 334) dirty exists 2024-09-13T14:24:06.492 INFO:tasks.rados.rados.0.smithi151.stdout:802: left oid 3 (ObjNum 334 snap 103 seq_num 334) 2024-09-13T14:24:06.492 INFO:tasks.rados.rados.0.smithi151.stdout:803: finishing write tid 1 to smithi15186934-15 2024-09-13T14:24:06.492 INFO:tasks.rados.rados.0.smithi151.stdout:803: finishing write tid 2 to smithi15186934-15 2024-09-13T14:24:06.493 INFO:tasks.rados.rados.0.smithi151.stdout:803: finishing write tid 3 to smithi15186934-15 2024-09-13T14:24:06.493 INFO:tasks.rados.rados.0.smithi151.stdout:803: finishing write tid 4 to smithi15186934-15 2024-09-13T14:24:06.493 INFO:tasks.rados.rados.0.smithi151.stdout:803: finishing write tid 5 to smithi15186934-15 2024-09-13T14:24:06.493 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 15 v 434 (ObjNum 335 snap 103 seq_num 335) dirty exists 2024-09-13T14:24:06.493 INFO:tasks.rados.rados.0.smithi151.stdout:803: left oid 15 (ObjNum 335 snap 103 seq_num 335) 2024-09-13T14:24:06.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:06 smithi151 ceph-mon[104450]: pgmap v9: 89 pgs: 1 active+clean+snaptrim, 88 active+clean; 198 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:24:06.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:06 smithi151 ceph-mon[104450]: osdmap e264: 8 total, 8 up, 8 in 2024-09-13T14:24:06.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:06 smithi151 ceph-mon[104450]: mgrmap e37: y(active, since 4s), standbys: x 2024-09-13T14:24:06.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:06 smithi151 ceph-mon[104450]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:06.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:06 smithi151 ceph-mon[104450]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:24:06.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:06 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:06.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:06 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:06.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:06 smithi151 ceph-mon[41387]: pgmap v9: 89 pgs: 1 active+clean+snaptrim, 88 active+clean; 198 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:24:06.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:06 smithi151 ceph-mon[41387]: osdmap e264: 8 total, 8 up, 8 in 2024-09-13T14:24:06.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:06 smithi151 ceph-mon[41387]: mgrmap e37: y(active, since 4s), standbys: x 2024-09-13T14:24:06.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:06 smithi151 ceph-mon[41387]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:06.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:06 smithi151 ceph-mon[41387]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:24:06.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:06 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:06.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:06 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:06.726 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'sleep 60' 2024-09-13T14:24:06.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:06 smithi184 ceph-mon[33394]: pgmap v9: 89 pgs: 1 active+clean+snaptrim, 88 active+clean; 198 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:24:06.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:06 smithi184 ceph-mon[33394]: osdmap e264: 8 total, 8 up, 8 in 2024-09-13T14:24:06.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:06 smithi184 ceph-mon[33394]: mgrmap e37: y(active, since 4s), standbys: x 2024-09-13T14:24:06.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:06 smithi184 ceph-mon[33394]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:06.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:06 smithi184 ceph-mon[33394]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:24:06.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:06 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:06.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:06 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:07.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:24:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:24:06.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi184\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:24:07.460 INFO:tasks.rados.rados.0.smithi151.stdout:799: done (4 left) 2024-09-13T14:24:07.461 INFO:tasks.rados.rados.0.smithi151.stdout:800: done (3 left) 2024-09-13T14:24:07.461 INFO:tasks.rados.rados.0.smithi151.stdout:802: done (2 left) 2024-09-13T14:24:07.461 INFO:tasks.rados.rados.0.smithi151.stdout:803: done (1 left) 2024-09-13T14:24:07.461 INFO:tasks.rados.rados.0.smithi151.stdout:804: done (0 left) 2024-09-13T14:24:07.461 INFO:tasks.rados.rados.0.smithi151.stdout:805: delete oid 35 current snap is 104 2024-09-13T14:24:07.462 INFO:tasks.rados.rados.0.smithi151.stdout:805: done (0 left) 2024-09-13T14:24:07.463 INFO:tasks.rados.rados.0.smithi151.stdout:806: write oid 17 current snap is 104 2024-09-13T14:24:07.463 INFO:tasks.rados.rados.0.smithi151.stdout:806: seq_num 337 ranges {463959=478445,1570667=731108,3038372=492433} 2024-09-13T14:24:07.467 INFO:tasks.rados.rados.0.smithi151.stdout:806: writing smithi15186934-17 from 463959 to 942404 tid 1 2024-09-13T14:24:07.475 INFO:tasks.rados.rados.0.smithi151.stdout:806: writing smithi15186934-17 from 1570667 to 2301775 tid 2 2024-09-13T14:24:07.482 INFO:tasks.rados.rados.0.smithi151.stdout:806: writing smithi15186934-17 from 3038372 to 3530805 tid 3 2024-09-13T14:24:07.482 INFO:tasks.rados.rados.0.smithi151.stdout:807: read oid 50 snap -1 2024-09-13T14:24:07.482 INFO:tasks.rados.rados.0.smithi151.stdout:807: expect (ObjNum 225 snap 66 seq_num 225) 2024-09-13T14:24:07.482 INFO:tasks.rados.rados.0.smithi151.stdout:808: write oid 1 current snap is 104 2024-09-13T14:24:07.482 INFO:tasks.rados.rados.0.smithi151.stdout:806: finishing write tid 1 to smithi15186934-17 2024-09-13T14:24:07.482 INFO:tasks.rados.rados.0.smithi151.stdout:806: finishing write tid 2 to smithi15186934-17 2024-09-13T14:24:07.482 INFO:tasks.rados.rados.0.smithi151.stdout:808: seq_num 338 ranges {480766=553694,1629149=591119,2782148=420058} 2024-09-13T14:24:07.488 INFO:tasks.rados.rados.0.smithi151.stdout:808: writing smithi15186934-1 from 480766 to 1034460 tid 1 2024-09-13T14:24:07.494 INFO:tasks.rados.rados.0.smithi151.stdout:808: writing smithi15186934-1 from 1629149 to 2220268 tid 2 2024-09-13T14:24:07.499 INFO:tasks.rados.rados.0.smithi151.stdout:808: writing smithi15186934-1 from 2782148 to 3202206 tid 3 2024-09-13T14:24:07.499 INFO:tasks.rados.rados.0.smithi151.stdout:809: snap_remove snap 82 2024-09-13T14:24:07.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:07 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:07.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:07 smithi151 ceph-mon[104450]: osdmap e265: 8 total, 8 up, 8 in 2024-09-13T14:24:07.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:07 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:07.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:07 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:07.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:07 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:07.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:07 smithi151 ceph-mon[41387]: osdmap e265: 8 total, 8 up, 8 in 2024-09-13T14:24:07.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:07 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:07.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:07 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:07.711 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:24:07.737 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:07 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:07.737 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:07 smithi184 ceph-mon[33394]: osdmap e265: 8 total, 8 up, 8 in 2024-09-13T14:24:07.738 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:07 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:07.738 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:07 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:08.465 INFO:tasks.rados.rados.0.smithi151.stdout:809: done (3 left) 2024-09-13T14:24:08.466 INFO:tasks.rados.rados.0.smithi151.stdout:810: read oid 14 snap -1 2024-09-13T14:24:08.466 INFO:tasks.rados.rados.0.smithi151.stdout:810: expect (ObjNum 321 snap 98 seq_num 321) 2024-09-13T14:24:08.466 INFO:tasks.rados.rados.0.smithi151.stdout:811: snap_remove snap 84 2024-09-13T14:24:08.558 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:08 smithi184 ceph-mon[33394]: pgmap v12: 89 pgs: 1 active+clean+snaptrim, 88 active+clean; 198 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:24:08.558 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:08 smithi184 ceph-mon[33394]: osdmap e266: 8 total, 8 up, 8 in 2024-09-13T14:24:08.775 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:08 smithi151 ceph-mon[104450]: pgmap v12: 89 pgs: 1 active+clean+snaptrim, 88 active+clean; 198 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:24:08.775 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:08 smithi151 ceph-mon[104450]: osdmap e266: 8 total, 8 up, 8 in 2024-09-13T14:24:08.775 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:08 smithi151 ceph-mon[41387]: pgmap v12: 89 pgs: 1 active+clean+snaptrim, 88 active+clean; 198 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:24:08.775 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:08 smithi151 ceph-mon[41387]: osdmap e266: 8 total, 8 up, 8 in 2024-09-13T14:24:09.482 INFO:tasks.rados.rados.0.smithi151.stdout:811: done (4 left) 2024-09-13T14:24:09.482 INFO:tasks.rados.rados.0.smithi151.stdout:812: rollback oid 7 current snap is 104 2024-09-13T14:24:09.482 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 7 to 89 2024-09-13T14:24:09.482 INFO:tasks.rados.rados.0.smithi151.stdout:813: read oid 35 snap 92 2024-09-13T14:24:09.483 INFO:tasks.rados.rados.0.smithi151.stdout:813: expect (ObjNum 196 snap 55 seq_num 196) 2024-09-13T14:24:09.483 INFO:tasks.rados.rados.0.smithi151.stdout:814: rollback oid 10 current snap is 104 2024-09-13T14:24:09.483 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 10 to 89 2024-09-13T14:24:09.483 INFO:tasks.rados.rados.0.smithi151.stdout:806: finishing write tid 3 to smithi15186934-17 2024-09-13T14:24:09.483 INFO:tasks.rados.rados.0.smithi151.stdout:806: finishing write tid 4 to smithi15186934-17 2024-09-13T14:24:09.483 INFO:tasks.rados.rados.0.smithi151.stdout:806: finishing write tid 5 to smithi15186934-17 2024-09-13T14:24:09.483 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 547 (ObjNum 337 snap 104 seq_num 337) dirty exists 2024-09-13T14:24:09.483 INFO:tasks.rados.rados.0.smithi151.stdout:806: left oid 17 (ObjNum 337 snap 104 seq_num 337) 2024-09-13T14:24:09.483 INFO:tasks.rados.rados.0.smithi151.stdout:808: finishing write tid 1 to smithi15186934-1 2024-09-13T14:24:09.483 INFO:tasks.rados.rados.0.smithi151.stdout:807: expect (ObjNum 225 snap 66 seq_num 225) 2024-09-13T14:24:09.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:09 smithi151 ceph-mon[104450]: osdmap e267: 8 total, 8 up, 8 in 2024-09-13T14:24:09.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:09 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:09.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:09 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:09.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:09 smithi151 ceph-mon[41387]: osdmap e267: 8 total, 8 up, 8 in 2024-09-13T14:24:09.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:09 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:09.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:09 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:09.724 INFO:tasks.rados.rados.0.smithi151.stdout:808: finishing write tid 2 to smithi15186934-1 2024-09-13T14:24:09.724 INFO:tasks.rados.rados.0.smithi151.stdout:808: finishing write tid 3 to smithi15186934-1 2024-09-13T14:24:09.724 INFO:tasks.rados.rados.0.smithi151.stdout:806: done (6 left) 2024-09-13T14:24:09.724 INFO:tasks.rados.rados.0.smithi151.stdout:807: done (5 left) 2024-09-13T14:24:09.724 INFO:tasks.rados.rados.0.smithi151.stdout:815: rollback oid 13 current snap is 104 2024-09-13T14:24:09.724 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 13 to 102 2024-09-13T14:24:09.724 INFO:tasks.rados.rados.0.smithi151.stdout:808: finishing write tid 4 to smithi15186934-1 2024-09-13T14:24:09.724 INFO:tasks.rados.rados.0.smithi151.stdout:808: finishing write tid 5 to smithi15186934-1 2024-09-13T14:24:09.724 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 1 v 239 (ObjNum 338 snap 104 seq_num 338) dirty exists 2024-09-13T14:24:09.725 INFO:tasks.rados.rados.0.smithi151.stdout:808: left oid 1 (ObjNum 338 snap 104 seq_num 338) 2024-09-13T14:24:09.725 INFO:tasks.rados.rados.0.smithi151.stdout:810: expect (ObjNum 321 snap 98 seq_num 321) 2024-09-13T14:24:09.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:09 smithi184 ceph-mon[33394]: osdmap e267: 8 total, 8 up, 8 in 2024-09-13T14:24:09.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:09 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:09.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:09 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:09.877 INFO:tasks.rados.rados.0.smithi151.stdout:814: finishing rollback tid 0 to smithi15186934-10 2024-09-13T14:24:09.877 INFO:tasks.rados.rados.0.smithi151.stdout:814: finishing rollback tid 1 to smithi15186934-10 2024-09-13T14:24:09.877 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 280 (ObjNum 593 snap 0 seq_num 3092881784) dirty dne 2024-09-13T14:24:09.877 INFO:tasks.rados.rados.0.smithi151.stdout:812: finishing rollback tid 1 to smithi15186934-7 2024-09-13T14:24:09.878 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 403 (ObjNum 529 snap 0 seq_num 1755291360) dirty dne 2024-09-13T14:24:09.878 INFO:tasks.rados.rados.0.smithi151.stdout:813: expect (ObjNum 196 snap 55 seq_num 196) 2024-09-13T14:24:10.023 INFO:tasks.rados.rados.0.smithi151.stdout:815: finishing rollback tid 0 to smithi15186934-13 2024-09-13T14:24:10.023 INFO:tasks.rados.rados.0.smithi151.stdout:815: finishing rollback tid 1 to smithi15186934-13 2024-09-13T14:24:10.023 INFO:tasks.rados.rados.0.smithi151.stdout:815: finishing rollback tid 2 to smithi15186934-13 2024-09-13T14:24:10.023 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 13 v 271 (ObjNum 270 snap 86 seq_num 270) dirty exists 2024-09-13T14:24:10.023 INFO:tasks.rados.rados.0.smithi151.stdout:808: done (5 left) 2024-09-13T14:24:10.023 INFO:tasks.rados.rados.0.smithi151.stdout:810: done (4 left) 2024-09-13T14:24:10.023 INFO:tasks.rados.rados.0.smithi151.stdout:812: done (3 left) 2024-09-13T14:24:10.023 INFO:tasks.rados.rados.0.smithi151.stdout:813: done (2 left) 2024-09-13T14:24:10.023 INFO:tasks.rados.rados.0.smithi151.stdout:814: done (1 left) 2024-09-13T14:24:10.024 INFO:tasks.rados.rados.0.smithi151.stdout:815: done (0 left) 2024-09-13T14:24:10.024 INFO:tasks.rados.rados.0.smithi151.stdout:816: snap_remove snap 92 2024-09-13T14:24:10.258 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:24:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:24:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:24:10.481 INFO:tasks.rados.rados.0.smithi151.stdout:816: done (0 left) 2024-09-13T14:24:10.481 INFO:tasks.rados.rados.0.smithi151.stdout:817: write oid 4 current snap is 104 2024-09-13T14:24:10.481 INFO:tasks.rados.rados.0.smithi151.stdout:817: seq_num 339 ranges {497573=628943,1687631=734777,3093218=604355} 2024-09-13T14:24:10.483 INFO:tasks.rados.rados.0.smithi151.stdout:817: writing smithi15186934-4 from 497573 to 1126516 tid 1 2024-09-13T14:24:10.491 INFO:tasks.rados.rados.0.smithi151.stdout:817: writing smithi15186934-4 from 1687631 to 2422408 tid 2 2024-09-13T14:24:10.498 INFO:tasks.rados.rados.0.smithi151.stdout:817: writing smithi15186934-4 from 3093218 to 3697573 tid 3 2024-09-13T14:24:10.498 INFO:tasks.rados.rados.0.smithi151.stdout:818: snap_remove snap 89 2024-09-13T14:24:10.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:10 smithi184 ceph-mon[33394]: pgmap v15: 89 pgs: 89 active+clean; 189 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.5 MiB/s rd, 2.0 MiB/s wr, 32 op/s 2024-09-13T14:24:10.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:10 smithi184 ceph-mon[33394]: osdmap e268: 8 total, 8 up, 8 in 2024-09-13T14:24:10.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:10 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:10.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:10 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:10.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:10 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:10.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:10 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:10.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:10 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:24:10.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:10 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:24:10.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:10 smithi151 ceph-mon[104450]: pgmap v15: 89 pgs: 89 active+clean; 189 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.5 MiB/s rd, 2.0 MiB/s wr, 32 op/s 2024-09-13T14:24:10.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:10 smithi151 ceph-mon[104450]: osdmap e268: 8 total, 8 up, 8 in 2024-09-13T14:24:10.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:10 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:10.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:10 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:10.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:10 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:10.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:10 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:10.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:10 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:24:10.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:10 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:24:10.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:10 smithi151 ceph-mon[41387]: pgmap v15: 89 pgs: 89 active+clean; 189 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.5 MiB/s rd, 2.0 MiB/s wr, 32 op/s 2024-09-13T14:24:10.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:10 smithi151 ceph-mon[41387]: osdmap e268: 8 total, 8 up, 8 in 2024-09-13T14:24:10.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:10 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:10.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:10 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:10.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:10 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:10.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:10 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:10.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:10 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:24:10.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:10 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:24:11.490 INFO:tasks.rados.rados.0.smithi151.stdout:818: done (1 left) 2024-09-13T14:24:11.491 INFO:tasks.rados.rados.0.smithi151.stdout:819: delete oid 29 current snap is 104 2024-09-13T14:24:11.491 INFO:tasks.rados.rados.0.smithi151.stdout:817: finishing write tid 1 to smithi15186934-4 2024-09-13T14:24:11.491 INFO:tasks.rados.rados.0.smithi151.stdout:817: finishing write tid 2 to smithi15186934-4 2024-09-13T14:24:11.491 INFO:tasks.rados.rados.0.smithi151.stdout:817: finishing write tid 3 to smithi15186934-4 2024-09-13T14:24:11.491 INFO:tasks.rados.rados.0.smithi151.stdout:817: finishing write tid 4 to smithi15186934-4 2024-09-13T14:24:11.491 INFO:tasks.rados.rados.0.smithi151.stdout:817: finishing write tid 5 to smithi15186934-4 2024-09-13T14:24:11.491 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 504 (ObjNum 339 snap 104 seq_num 339) dirty exists 2024-09-13T14:24:11.491 INFO:tasks.rados.rados.0.smithi151.stdout:817: left oid 4 (ObjNum 339 snap 104 seq_num 339) 2024-09-13T14:24:11.494 INFO:tasks.rados.rados.0.smithi151.stdout:817: done (1 left) 2024-09-13T14:24:11.495 INFO:tasks.rados.rados.0.smithi151.stdout:819: done (0 left) 2024-09-13T14:24:11.495 INFO:tasks.rados.rados.0.smithi151.stdout:820: read oid 39 snap 103 2024-09-13T14:24:11.495 INFO:tasks.rados.rados.0.smithi151.stdout:820: expect deleted 2024-09-13T14:24:11.495 INFO:tasks.rados.rados.0.smithi151.stdout:821: read oid 43 snap 95 2024-09-13T14:24:11.495 INFO:tasks.rados.rados.0.smithi151.stdout:821: expect (ObjNum 263 snap 82 seq_num 263) 2024-09-13T14:24:11.495 INFO:tasks.rados.rados.0.smithi151.stdout:822: snap_create 2024-09-13T14:24:11.501 INFO:tasks.rados.rados.0.smithi151.stdout:821: expect (ObjNum 263 snap 82 seq_num 263) 2024-09-13T14:24:11.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:11 smithi184 ceph-mon[33394]: Updating smithi151:/etc/ceph/ceph.conf 2024-09-13T14:24:11.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:11 smithi184 ceph-mon[33394]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-13T14:24:11.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:11 smithi184 ceph-mon[33394]: osdmap e269: 8 total, 8 up, 8 in 2024-09-13T14:24:11.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:11 smithi184 ceph-mon[33394]: Updating smithi151:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:24:11.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:11 smithi184 ceph-mon[33394]: Updating smithi184:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:24:11.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:11 smithi184 ceph-mon[33394]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:24:11.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:11 smithi184 ceph-mon[33394]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:24:11.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:11 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:11.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:11 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:11.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:11 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:11.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:11 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:11.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:11 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:11.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:11 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:24:11.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:11 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:24:11.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:11 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:24:11.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:11 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:11.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[104450]: Updating smithi151:/etc/ceph/ceph.conf 2024-09-13T14:24:11.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[104450]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-13T14:24:11.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[104450]: osdmap e269: 8 total, 8 up, 8 in 2024-09-13T14:24:11.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[104450]: Updating smithi151:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:24:11.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[104450]: Updating smithi184:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:24:11.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[104450]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:24:11.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[104450]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:24:11.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:11.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:11.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:11.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:11.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:11.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:24:11.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:24:11.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:24:11.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:11.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[41387]: Updating smithi151:/etc/ceph/ceph.conf 2024-09-13T14:24:11.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[41387]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-13T14:24:11.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[41387]: osdmap e269: 8 total, 8 up, 8 in 2024-09-13T14:24:11.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[41387]: Updating smithi151:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:24:11.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[41387]: Updating smithi184:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:24:11.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[41387]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:24:11.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[41387]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:24:11.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:11.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:11.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:11.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:11.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:11.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:24:11.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:24:11.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:24:11.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:11 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:24:12.493 INFO:tasks.rados.rados.0.smithi151.stdout:820: done (2 left) 2024-09-13T14:24:12.494 INFO:tasks.rados.rados.0.smithi151.stdout:821: done (1 left) 2024-09-13T14:24:12.494 INFO:tasks.rados.rados.0.smithi151.stdout:822: done (0 left) 2024-09-13T14:24:12.494 INFO:tasks.rados.rados.0.smithi151.stdout:823: write oid 35 current snap is 105 2024-09-13T14:24:12.494 INFO:tasks.rados.rados.0.smithi151.stdout:823: seq_num 341 ranges {531187=779441,2088242=738446,3431711=299476} 2024-09-13T14:24:12.500 INFO:tasks.rados.rados.0.smithi151.stdout:823: writing smithi15186934-35 from 531187 to 1310628 tid 1 2024-09-13T14:24:12.508 INFO:tasks.rados.rados.0.smithi151.stdout:823: writing smithi15186934-35 from 2088242 to 2826688 tid 2 2024-09-13T14:24:12.514 INFO:tasks.rados.rados.0.smithi151.stdout:823: writing smithi15186934-35 from 3431711 to 3731187 tid 3 2024-09-13T14:24:12.514 INFO:tasks.rados.rados.0.smithi151.stdout:824: write oid 48 current snap is 105 2024-09-13T14:24:12.514 INFO:tasks.rados.rados.0.smithi151.stdout:824: seq_num 342 ranges {547994=454690,1746724=482104,2659134=465146,3605976=142018} 2024-09-13T14:24:12.519 INFO:tasks.rados.rados.0.smithi151.stdout:824: writing smithi15186934-48 from 547994 to 1002684 tid 1 2024-09-13T14:24:12.526 INFO:tasks.rados.rados.0.smithi151.stdout:824: writing smithi15186934-48 from 1746724 to 2228828 tid 2 2024-09-13T14:24:12.531 INFO:tasks.rados.rados.0.smithi151.stdout:824: writing smithi15186934-48 from 2659134 to 3124280 tid 3 2024-09-13T14:24:12.534 INFO:tasks.rados.rados.0.smithi151.stdout:824: writing smithi15186934-48 from 3605976 to 3747994 tid 4 2024-09-13T14:24:12.534 INFO:tasks.rados.rados.0.smithi151.stdout:825: write oid 49 current snap is 105 2024-09-13T14:24:12.535 INFO:tasks.rados.rados.0.smithi151.stdout:823: finishing write tid 1 to smithi15186934-35 2024-09-13T14:24:12.535 INFO:tasks.rados.rados.0.smithi151.stdout:825: seq_num 343 ranges {564801=646292,1921559=742115,3202910=561891} 2024-09-13T14:24:12.540 INFO:tasks.rados.rados.0.smithi151.stdout:825: writing smithi15186934-49 from 564801 to 1211093 tid 1 2024-09-13T14:24:12.549 INFO:tasks.rados.rados.0.smithi151.stdout:825: writing smithi15186934-49 from 1921559 to 2663674 tid 2 2024-09-13T14:24:12.555 INFO:tasks.rados.rados.0.smithi151.stdout:825: writing smithi15186934-49 from 3202910 to 3764801 tid 3 2024-09-13T14:24:12.555 INFO:tasks.rados.rados.0.smithi151.stdout:826: read oid 32 snap -1 2024-09-13T14:24:12.555 INFO:tasks.rados.rados.0.smithi151.stdout:826: expect (ObjNum 304 snap 94 seq_num 304) 2024-09-13T14:24:12.555 INFO:tasks.rados.rados.0.smithi151.stdout:827: delete oid 27 current snap is 105 2024-09-13T14:24:12.556 INFO:tasks.rados.rados.0.smithi151.stdout:823: finishing write tid 2 to smithi15186934-35 2024-09-13T14:24:12.556 INFO:tasks.rados.rados.0.smithi151.stdout:823: finishing write tid 3 to smithi15186934-35 2024-09-13T14:24:12.556 INFO:tasks.rados.rados.0.smithi151.stdout:823: finishing write tid 4 to smithi15186934-35 2024-09-13T14:24:12.556 INFO:tasks.rados.rados.0.smithi151.stdout:823: finishing write tid 5 to smithi15186934-35 2024-09-13T14:24:12.556 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 35 v 580 (ObjNum 341 snap 105 seq_num 341) dirty exists 2024-09-13T14:24:12.556 INFO:tasks.rados.rados.0.smithi151.stdout:823: left oid 35 (ObjNum 341 snap 105 seq_num 341) 2024-09-13T14:24:12.556 INFO:tasks.rados.rados.0.smithi151.stdout:824: finishing write tid 1 to smithi15186934-48 2024-09-13T14:24:12.556 INFO:tasks.rados.rados.0.smithi151.stdout:824: finishing write tid 2 to smithi15186934-48 2024-09-13T14:24:12.556 INFO:tasks.rados.rados.0.smithi151.stdout:824: finishing write tid 3 to smithi15186934-48 2024-09-13T14:24:12.556 INFO:tasks.rados.rados.0.smithi151.stdout:824: finishing write tid 4 to smithi15186934-48 2024-09-13T14:24:12.556 INFO:tasks.rados.rados.0.smithi151.stdout:824: finishing write tid 5 to smithi15186934-48 2024-09-13T14:24:12.557 INFO:tasks.rados.rados.0.smithi151.stdout:824: finishing write tid 6 to smithi15186934-48 2024-09-13T14:24:12.557 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 48 v 512 (ObjNum 342 snap 105 seq_num 342) dirty exists 2024-09-13T14:24:12.557 INFO:tasks.rados.rados.0.smithi151.stdout:824: left oid 48 (ObjNum 342 snap 105 seq_num 342) 2024-09-13T14:24:12.557 INFO:tasks.rados.rados.0.smithi151.stdout:825: finishing write tid 1 to smithi15186934-49 2024-09-13T14:24:12.557 INFO:tasks.rados.rados.0.smithi151.stdout:825: finishing write tid 2 to smithi15186934-49 2024-09-13T14:24:12.558 INFO:tasks.rados.rados.0.smithi151.stdout:823: done (4 left) 2024-09-13T14:24:12.558 INFO:tasks.rados.rados.0.smithi151.stdout:824: done (3 left) 2024-09-13T14:24:12.558 INFO:tasks.rados.rados.0.smithi151.stdout:827: done (2 left) 2024-09-13T14:24:12.558 INFO:tasks.rados.rados.0.smithi151.stdout:828: write oid 21 current snap is 105 2024-09-13T14:24:12.558 INFO:tasks.rados.rados.0.smithi151.stdout:828: seq_num 345 ranges {598415=796790,1922170=745784,3141403=415315} 2024-09-13T14:24:12.565 INFO:tasks.rados.rados.0.smithi151.stdout:828: writing smithi15186934-21 from 598415 to 1395205 tid 1 2024-09-13T14:24:12.572 INFO:tasks.rados.rados.0.smithi151.stdout:828: writing smithi15186934-21 from 1922170 to 2667954 tid 2 2024-09-13T14:24:12.576 INFO:tasks.rados.rados.0.smithi151.stdout:828: writing smithi15186934-21 from 3141403 to 3556718 tid 3 2024-09-13T14:24:12.577 INFO:tasks.rados.rados.0.smithi151.stdout:829: write oid 39 current snap is 105 2024-09-13T14:24:12.577 INFO:tasks.rados.rados.0.smithi151.stdout:829: seq_num 346 ranges {615222=472039,1580652=489442,2768826=626587} 2024-09-13T14:24:12.582 INFO:tasks.rados.rados.0.smithi151.stdout:829: writing smithi15186934-39 from 615222 to 1087261 tid 1 2024-09-13T14:24:12.588 INFO:tasks.rados.rados.0.smithi151.stdout:829: writing smithi15186934-39 from 1580652 to 2070094 tid 2 2024-09-13T14:24:12.596 INFO:tasks.rados.rados.0.smithi151.stdout:829: writing smithi15186934-39 from 2768826 to 3395413 tid 3 2024-09-13T14:24:12.597 INFO:tasks.rados.rados.0.smithi151.stdout:830: read oid 37 snap 104 2024-09-13T14:24:12.597 INFO:tasks.rados.rados.0.smithi151.stdout:830: expect (ObjNum 179 snap 47 seq_num 179) 2024-09-13T14:24:12.597 INFO:tasks.rados.rados.0.smithi151.stdout:831: rollback oid 1 current snap is 105 2024-09-13T14:24:12.597 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 1 to 99 2024-09-13T14:24:12.597 INFO:tasks.rados.rados.0.smithi151.stdout:825: finishing write tid 3 to smithi15186934-49 2024-09-13T14:24:12.597 INFO:tasks.rados.rados.0.smithi151.stdout:825: finishing write tid 4 to smithi15186934-49 2024-09-13T14:24:12.597 INFO:tasks.rados.rados.0.smithi151.stdout:825: finishing write tid 5 to smithi15186934-49 2024-09-13T14:24:12.597 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 49 v 430 (ObjNum 343 snap 105 seq_num 343) dirty exists 2024-09-13T14:24:12.597 INFO:tasks.rados.rados.0.smithi151.stdout:825: left oid 49 (ObjNum 343 snap 105 seq_num 343) 2024-09-13T14:24:12.597 INFO:tasks.rados.rados.0.smithi151.stdout:826: expect (ObjNum 304 snap 94 seq_num 304) 2024-09-13T14:24:12.755 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:12 smithi184 ceph-mon[33394]: Updating smithi151:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:24:12.755 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:12 smithi184 ceph-mon[33394]: Updating smithi184:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:24:12.755 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:12 smithi184 ceph-mon[33394]: pgmap v18: 89 pgs: 89 active+clean; 189 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.5 MiB/s rd, 2.0 MiB/s wr, 32 op/s 2024-09-13T14:24:12.755 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:12 smithi184 ceph-mon[33394]: osdmap e270: 8 total, 8 up, 8 in 2024-09-13T14:24:12.781 INFO:tasks.rados.rados.0.smithi151.stdout:828: finishing write tid 1 to smithi15186934-21 2024-09-13T14:24:12.781 INFO:tasks.rados.rados.0.smithi151.stdout:828: finishing write tid 2 to smithi15186934-21 2024-09-13T14:24:12.781 INFO:tasks.rados.rados.0.smithi151.stdout:828: finishing write tid 3 to smithi15186934-21 2024-09-13T14:24:12.782 INFO:tasks.rados.rados.0.smithi151.stdout:828: finishing write tid 4 to smithi15186934-21 2024-09-13T14:24:12.782 INFO:tasks.rados.rados.0.smithi151.stdout:828: finishing write tid 5 to smithi15186934-21 2024-09-13T14:24:12.782 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 21 v 208 (ObjNum 345 snap 105 seq_num 345) dirty exists 2024-09-13T14:24:12.782 INFO:tasks.rados.rados.0.smithi151.stdout:828: left oid 21 (ObjNum 345 snap 105 seq_num 345) 2024-09-13T14:24:12.782 INFO:tasks.rados.rados.0.smithi151.stdout:829: finishing write tid 1 to smithi15186934-39 2024-09-13T14:24:12.782 INFO:tasks.rados.rados.0.smithi151.stdout:825: done (5 left) 2024-09-13T14:24:12.782 INFO:tasks.rados.rados.0.smithi151.stdout:826: done (4 left) 2024-09-13T14:24:12.782 INFO:tasks.rados.rados.0.smithi151.stdout:828: done (3 left) 2024-09-13T14:24:12.782 INFO:tasks.rados.rados.0.smithi151.stdout:832: delete oid 28 current snap is 105 2024-09-13T14:24:12.782 INFO:tasks.rados.rados.0.smithi151.stdout:829: finishing write tid 2 to smithi15186934-39 2024-09-13T14:24:12.782 INFO:tasks.rados.rados.0.smithi151.stdout:831: finishing rollback tid 0 to smithi15186934-1 2024-09-13T14:24:12.782 INFO:tasks.rados.rados.0.smithi151.stdout:831: finishing rollback tid 1 to smithi15186934-1 2024-09-13T14:24:12.782 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 1 v 242 (ObjNum 33 snap 0 seq_num 3064838224) dirty dne 2024-09-13T14:24:12.782 INFO:tasks.rados.rados.0.smithi151.stdout:829: finishing write tid 3 to smithi15186934-39 2024-09-13T14:24:12.783 INFO:tasks.rados.rados.0.smithi151.stdout:829: finishing write tid 4 to smithi15186934-39 2024-09-13T14:24:12.783 INFO:tasks.rados.rados.0.smithi151.stdout:829: finishing write tid 5 to smithi15186934-39 2024-09-13T14:24:12.783 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 235 (ObjNum 346 snap 105 seq_num 346) dirty exists 2024-09-13T14:24:12.783 INFO:tasks.rados.rados.0.smithi151.stdout:829: left oid 39 (ObjNum 346 snap 105 seq_num 346) 2024-09-13T14:24:12.784 INFO:tasks.rados.rados.0.smithi151.stdout:830: expect (ObjNum 179 snap 47 seq_num 179) 2024-09-13T14:24:12.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:12 smithi151 ceph-mon[104450]: Updating smithi151:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:24:12.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:12 smithi151 ceph-mon[104450]: Updating smithi184:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:24:12.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:12 smithi151 ceph-mon[104450]: pgmap v18: 89 pgs: 89 active+clean; 189 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.5 MiB/s rd, 2.0 MiB/s wr, 32 op/s 2024-09-13T14:24:12.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:12 smithi151 ceph-mon[104450]: osdmap e270: 8 total, 8 up, 8 in 2024-09-13T14:24:12.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:12 smithi151 ceph-mon[41387]: Updating smithi151:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:24:12.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:12 smithi151 ceph-mon[41387]: Updating smithi184:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:24:12.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:12 smithi151 ceph-mon[41387]: pgmap v18: 89 pgs: 89 active+clean; 189 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.5 MiB/s rd, 2.0 MiB/s wr, 32 op/s 2024-09-13T14:24:12.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:12 smithi151 ceph-mon[41387]: osdmap e270: 8 total, 8 up, 8 in 2024-09-13T14:24:12.948 INFO:tasks.rados.rados.0.smithi151.stdout:829: done (3 left) 2024-09-13T14:24:12.948 INFO:tasks.rados.rados.0.smithi151.stdout:830: done (2 left) 2024-09-13T14:24:12.948 INFO:tasks.rados.rados.0.smithi151.stdout:831: done (1 left) 2024-09-13T14:24:12.948 INFO:tasks.rados.rados.0.smithi151.stdout:832: done (0 left) 2024-09-13T14:24:12.948 INFO:tasks.rados.rados.0.smithi151.stdout:833: read oid 43 snap -1 2024-09-13T14:24:12.949 INFO:tasks.rados.rados.0.smithi151.stdout:833: expect (ObjNum 263 snap 82 seq_num 263) 2024-09-13T14:24:12.949 INFO:tasks.rados.rados.0.smithi151.stdout:834: snap_create 2024-09-13T14:24:12.955 INFO:tasks.rados.rados.0.smithi151.stdout:833: expect (ObjNum 263 snap 82 seq_num 263) 2024-09-13T14:24:13.498 INFO:tasks.rados.rados.0.smithi151.stdout:833: done (1 left) 2024-09-13T14:24:13.498 INFO:tasks.rados.rados.0.smithi151.stdout:834: done (0 left) 2024-09-13T14:24:13.499 INFO:tasks.rados.rados.0.smithi151.stdout:835: rollback oid 18 current snap is 106 2024-09-13T14:24:13.499 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 18 to 104 2024-09-13T14:24:13.499 INFO:tasks.rados.rados.0.smithi151.stdout:836: snap_create 2024-09-13T14:24:13.501 INFO:tasks.rados.rados.0.smithi151.stdout:835: finishing rollback tid 0 to smithi15186934-18 2024-09-13T14:24:13.501 INFO:tasks.rados.rados.0.smithi151.stdout:835: finishing rollback tid 1 to smithi15186934-18 2024-09-13T14:24:13.501 INFO:tasks.rados.rados.0.smithi151.stdout:835: finishing rollback tid 2 to smithi15186934-18 2024-09-13T14:24:13.501 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 18 v 241 (ObjNum 327 snap 101 seq_num 327) dirty exists 2024-09-13T14:24:13.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:13 smithi184 ceph-mon[33394]: osdmap e271: 8 total, 8 up, 8 in 2024-09-13T14:24:13.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:13 smithi151 ceph-mon[104450]: osdmap e271: 8 total, 8 up, 8 in 2024-09-13T14:24:13.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:13 smithi151 ceph-mon[41387]: osdmap e271: 8 total, 8 up, 8 in 2024-09-13T14:24:14.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:24:14 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:24:14.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:24:14.440 INFO:tasks.rados.rados.0.smithi151.stdout:835: done (1 left) 2024-09-13T14:24:14.441 INFO:tasks.rados.rados.0.smithi151.stdout:836: done (0 left) 2024-09-13T14:24:14.441 INFO:tasks.rados.rados.0.smithi151.stdout:837: rollback oid 38 current snap is 107 2024-09-13T14:24:14.441 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 38 to 103 2024-09-13T14:24:14.441 INFO:tasks.rados.rados.0.smithi151.stdout:838: read oid 40 snap 104 2024-09-13T14:24:14.441 INFO:tasks.rados.rados.0.smithi151.stdout:838: expect (ObjNum 282 snap 88 seq_num 282) 2024-09-13T14:24:14.441 INFO:tasks.rados.rados.0.smithi151.stdout:839: write oid 39 current snap is 107 2024-09-13T14:24:14.441 INFO:tasks.rados.rados.0.smithi151.stdout:839: seq_num 348 ranges {648836=622537,1981263=493111,3107319=741517} 2024-09-13T14:24:14.448 INFO:tasks.rados.rados.0.smithi151.stdout:839: writing smithi15186934-39 from 648836 to 1271373 tid 1 2024-09-13T14:24:14.455 INFO:tasks.rados.rados.0.smithi151.stdout:839: writing smithi15186934-39 from 1981263 to 2474374 tid 2 2024-09-13T14:24:14.462 INFO:tasks.rados.rados.0.smithi151.stdout:839: writing smithi15186934-39 from 3107319 to 3848836 tid 3 2024-09-13T14:24:14.463 INFO:tasks.rados.rados.0.smithi151.stdout:840: write oid 36 current snap is 107 2024-09-13T14:24:14.463 INFO:tasks.rados.rados.0.smithi151.stdout:837: finishing rollback tid 0 to smithi15186934-38 2024-09-13T14:24:14.463 INFO:tasks.rados.rados.0.smithi151.stdout:837: finishing rollback tid 1 to smithi15186934-38 2024-09-13T14:24:14.463 INFO:tasks.rados.rados.0.smithi151.stdout:837: finishing rollback tid 2 to smithi15186934-38 2024-09-13T14:24:14.463 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 239 (ObjNum 302 snap 93 seq_num 302) dirty exists 2024-09-13T14:24:14.463 INFO:tasks.rados.rados.0.smithi151.stdout:838: expect (ObjNum 282 snap 88 seq_num 282) 2024-09-13T14:24:14.589 INFO:tasks.rados.rados.0.smithi151.stdout:839: finishing write tid 1 to smithi15186934-39 2024-09-13T14:24:14.589 INFO:tasks.rados.rados.0.smithi151.stdout:839: finishing write tid 2 to smithi15186934-39 2024-09-13T14:24:14.589 INFO:tasks.rados.rados.0.smithi151.stdout:840: seq_num 349 ranges {665643=697786,2039745=753122,3534742=330901} 2024-09-13T14:24:14.596 INFO:tasks.rados.rados.0.smithi151.stdout:840: writing smithi15186934-36 from 665643 to 1363429 tid 1 2024-09-13T14:24:14.604 INFO:tasks.rados.rados.0.smithi151.stdout:840: writing smithi15186934-36 from 2039745 to 2792867 tid 2 2024-09-13T14:24:14.610 INFO:tasks.rados.rados.0.smithi151.stdout:840: writing smithi15186934-36 from 3534742 to 3865643 tid 3 2024-09-13T14:24:14.610 INFO:tasks.rados.rados.0.smithi151.stdout:837: done (3 left) 2024-09-13T14:24:14.610 INFO:tasks.rados.rados.0.smithi151.stdout:838: done (2 left) 2024-09-13T14:24:14.610 INFO:tasks.rados.rados.0.smithi151.stdout:841: snap_remove snap 95 2024-09-13T14:24:14.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:14 smithi184 ceph-mon[33394]: pgmap v21: 89 pgs: 89 active+clean; 184 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-09-13T14:24:14.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:14 smithi184 ceph-mon[33394]: osdmap e272: 8 total, 8 up, 8 in 2024-09-13T14:24:14.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:14 smithi184 ceph-mon[33394]: osdmap e273: 8 total, 8 up, 8 in 2024-09-13T14:24:14.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:24:14] "GET /metrics HTTP/1.1" 200 33658 "" "Prometheus/2.51.0" 2024-09-13T14:24:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:14 smithi151 ceph-mon[104450]: pgmap v21: 89 pgs: 89 active+clean; 184 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-09-13T14:24:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:14 smithi151 ceph-mon[104450]: osdmap e272: 8 total, 8 up, 8 in 2024-09-13T14:24:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:14 smithi151 ceph-mon[104450]: osdmap e273: 8 total, 8 up, 8 in 2024-09-13T14:24:14.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:14 smithi151 ceph-mon[41387]: pgmap v21: 89 pgs: 89 active+clean; 184 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-09-13T14:24:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:14 smithi151 ceph-mon[41387]: osdmap e272: 8 total, 8 up, 8 in 2024-09-13T14:24:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:14 smithi151 ceph-mon[41387]: osdmap e273: 8 total, 8 up, 8 in 2024-09-13T14:24:15.511 INFO:tasks.rados.rados.0.smithi151.stdout:841: done (2 left) 2024-09-13T14:24:15.511 INFO:tasks.rados.rados.0.smithi151.stdout:842: write oid 25 current snap is 107 2024-09-13T14:24:15.511 INFO:tasks.rados.rados.0.smithi151.stdout:839: finishing write tid 3 to smithi15186934-39 2024-09-13T14:24:15.511 INFO:tasks.rados.rados.0.smithi151.stdout:839: finishing write tid 4 to smithi15186934-39 2024-09-13T14:24:15.511 INFO:tasks.rados.rados.0.smithi151.stdout:839: finishing write tid 5 to smithi15186934-39 2024-09-13T14:24:15.511 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 244 (ObjNum 348 snap 107 seq_num 348) dirty exists 2024-09-13T14:24:15.511 INFO:tasks.rados.rados.0.smithi151.stdout:839: left oid 39 (ObjNum 348 snap 107 seq_num 348) 2024-09-13T14:24:15.511 INFO:tasks.rados.rados.0.smithi151.stdout:840: finishing write tid 1 to smithi15186934-36 2024-09-13T14:24:15.511 INFO:tasks.rados.rados.0.smithi151.stdout:840: finishing write tid 2 to smithi15186934-36 2024-09-13T14:24:15.512 INFO:tasks.rados.rados.0.smithi151.stdout:840: finishing write tid 3 to smithi15186934-36 2024-09-13T14:24:15.512 INFO:tasks.rados.rados.0.smithi151.stdout:840: finishing write tid 4 to smithi15186934-36 2024-09-13T14:24:15.512 INFO:tasks.rados.rados.0.smithi151.stdout:840: finishing write tid 5 to smithi15186934-36 2024-09-13T14:24:15.512 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 36 v 181 (ObjNum 349 snap 107 seq_num 349) dirty exists 2024-09-13T14:24:15.512 INFO:tasks.rados.rados.0.smithi151.stdout:840: left oid 36 (ObjNum 349 snap 107 seq_num 349) 2024-09-13T14:24:15.512 INFO:tasks.rados.rados.0.smithi151.stdout:842: seq_num 350 ranges {682450=489388,1814580=496780,2878518=788028} 2024-09-13T14:24:15.518 INFO:tasks.rados.rados.0.smithi151.stdout:842: writing smithi15186934-25 from 682450 to 1171838 tid 1 2024-09-13T14:24:15.524 INFO:tasks.rados.rados.0.smithi151.stdout:842: writing smithi15186934-25 from 1814580 to 2311360 tid 2 2024-09-13T14:24:15.532 INFO:tasks.rados.rados.0.smithi151.stdout:842: writing smithi15186934-25 from 2878518 to 3666546 tid 3 2024-09-13T14:24:15.532 INFO:tasks.rados.rados.0.smithi151.stdout:839: done (2 left) 2024-09-13T14:24:15.532 INFO:tasks.rados.rados.0.smithi151.stdout:840: done (1 left) 2024-09-13T14:24:15.532 INFO:tasks.rados.rados.0.smithi151.stdout:843: write oid 21 current snap is 107 2024-09-13T14:24:15.532 INFO:tasks.rados.rados.0.smithi151.stdout:843: seq_num 351 ranges {699257=564637,1873062=640438,3305941=593316} 2024-09-13T14:24:15.538 INFO:tasks.rados.rados.0.smithi151.stdout:843: writing smithi15186934-21 from 699257 to 1263894 tid 1 2024-09-13T14:24:15.545 INFO:tasks.rados.rados.0.smithi151.stdout:843: writing smithi15186934-21 from 1873062 to 2513500 tid 2 2024-09-13T14:24:15.553 INFO:tasks.rados.rados.0.smithi151.stdout:843: writing smithi15186934-21 from 3305941 to 3899257 tid 3 2024-09-13T14:24:15.553 INFO:tasks.rados.rados.0.smithi151.stdout:844: write oid 10 current snap is 107 2024-09-13T14:24:15.553 INFO:tasks.rados.rados.0.smithi151.stdout:844: seq_num 352 ranges {716064=639886,1815191=500449,2817011=526925} 2024-09-13T14:24:15.560 INFO:tasks.rados.rados.0.smithi151.stdout:844: writing smithi15186934-10 from 716064 to 1355950 tid 1 2024-09-13T14:24:15.566 INFO:tasks.rados.rados.0.smithi151.stdout:844: writing smithi15186934-10 from 1815191 to 2315640 tid 2 2024-09-13T14:24:15.572 INFO:tasks.rados.rados.0.smithi151.stdout:844: writing smithi15186934-10 from 2817011 to 3343936 tid 3 2024-09-13T14:24:15.572 INFO:tasks.rados.rados.0.smithi151.stdout:845: snap_create 2024-09-13T14:24:15.572 INFO:tasks.rados.rados.0.smithi151.stdout:842: finishing write tid 1 to smithi15186934-25 2024-09-13T14:24:15.572 INFO:tasks.rados.rados.0.smithi151.stdout:842: finishing write tid 2 to smithi15186934-25 2024-09-13T14:24:15.572 INFO:tasks.rados.rados.0.smithi151.stdout:842: finishing write tid 3 to smithi15186934-25 2024-09-13T14:24:15.572 INFO:tasks.rados.rados.0.smithi151.stdout:842: finishing write tid 4 to smithi15186934-25 2024-09-13T14:24:15.573 INFO:tasks.rados.rados.0.smithi151.stdout:842: finishing write tid 5 to smithi15186934-25 2024-09-13T14:24:15.573 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 518 (ObjNum 350 snap 107 seq_num 350) dirty exists 2024-09-13T14:24:15.573 INFO:tasks.rados.rados.0.smithi151.stdout:842: left oid 25 (ObjNum 350 snap 107 seq_num 350) 2024-09-13T14:24:15.573 INFO:tasks.rados.rados.0.smithi151.stdout:843: finishing write tid 1 to smithi15186934-21 2024-09-13T14:24:15.573 INFO:tasks.rados.rados.0.smithi151.stdout:843: finishing write tid 2 to smithi15186934-21 2024-09-13T14:24:15.573 INFO:tasks.rados.rados.0.smithi151.stdout:843: finishing write tid 3 to smithi15186934-21 2024-09-13T14:24:15.573 INFO:tasks.rados.rados.0.smithi151.stdout:843: finishing write tid 4 to smithi15186934-21 2024-09-13T14:24:15.573 INFO:tasks.rados.rados.0.smithi151.stdout:843: finishing write tid 5 to smithi15186934-21 2024-09-13T14:24:15.573 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 21 v 215 (ObjNum 351 snap 107 seq_num 351) dirty exists 2024-09-13T14:24:15.573 INFO:tasks.rados.rados.0.smithi151.stdout:843: left oid 21 (ObjNum 351 snap 107 seq_num 351) 2024-09-13T14:24:15.574 INFO:tasks.rados.rados.0.smithi151.stdout:844: finishing write tid 1 to smithi15186934-10 2024-09-13T14:24:15.574 INFO:tasks.rados.rados.0.smithi151.stdout:844: finishing write tid 2 to smithi15186934-10 2024-09-13T14:24:15.574 INFO:tasks.rados.rados.0.smithi151.stdout:844: finishing write tid 3 to smithi15186934-10 2024-09-13T14:24:15.575 INFO:tasks.rados.rados.0.smithi151.stdout:844: finishing write tid 4 to smithi15186934-10 2024-09-13T14:24:15.575 INFO:tasks.rados.rados.0.smithi151.stdout:844: finishing write tid 5 to smithi15186934-10 2024-09-13T14:24:15.575 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 296 (ObjNum 352 snap 107 seq_num 352) dirty exists 2024-09-13T14:24:15.575 INFO:tasks.rados.rados.0.smithi151.stdout:844: left oid 10 (ObjNum 352 snap 107 seq_num 352) 2024-09-13T14:24:16.103 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:24:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:24:15.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:24:16.103 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:24:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:24:15.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:24:16.514 INFO:tasks.rados.rados.0.smithi151.stdout:842: done (3 left) 2024-09-13T14:24:16.514 INFO:tasks.rados.rados.0.smithi151.stdout:843: done (2 left) 2024-09-13T14:24:16.514 INFO:tasks.rados.rados.0.smithi151.stdout:844: done (1 left) 2024-09-13T14:24:16.515 INFO:tasks.rados.rados.0.smithi151.stdout:845: done (0 left) 2024-09-13T14:24:16.515 INFO:tasks.rados.rados.0.smithi151.stdout:846: rollback oid 4 current snap is 108 2024-09-13T14:24:16.515 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 4 to 100 2024-09-13T14:24:16.515 INFO:tasks.rados.rados.0.smithi151.stdout:847: write oid 5 current snap is 108 2024-09-13T14:24:16.515 INFO:tasks.rados.rados.0.smithi151.stdout:847: seq_num 353 ranges {732871=715135,1873673=644107,3244434=688437} 2024-09-13T14:24:16.522 INFO:tasks.rados.rados.0.smithi151.stdout:847: writing smithi15186934-5 from 732871 to 1448006 tid 1 2024-09-13T14:24:16.528 INFO:tasks.rados.rados.0.smithi151.stdout:847: writing smithi15186934-5 from 1873673 to 2517780 tid 2 2024-09-13T14:24:16.536 INFO:tasks.rados.rados.0.smithi151.stdout:847: writing smithi15186934-5 from 3244434 to 3932871 tid 3 2024-09-13T14:24:16.536 INFO:tasks.rados.rados.0.smithi151.stdout:848: snap_create 2024-09-13T14:24:16.536 INFO:tasks.rados.rados.0.smithi151.stdout:846: finishing rollback tid 0 to smithi15186934-4 2024-09-13T14:24:16.536 INFO:tasks.rados.rados.0.smithi151.stdout:846: finishing rollback tid 1 to smithi15186934-4 2024-09-13T14:24:16.536 INFO:tasks.rados.rados.0.smithi151.stdout:846: finishing rollback tid 2 to smithi15186934-4 2024-09-13T14:24:16.536 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 522 (ObjNum 313 snap 96 seq_num 313) dirty exists 2024-09-13T14:24:16.536 INFO:tasks.rados.rados.0.smithi151.stdout:847: finishing write tid 1 to smithi15186934-5 2024-09-13T14:24:16.536 INFO:tasks.rados.rados.0.smithi151.stdout:847: finishing write tid 2 to smithi15186934-5 2024-09-13T14:24:16.539 INFO:tasks.rados.rados.0.smithi151.stdout:847: finishing write tid 3 to smithi15186934-5 2024-09-13T14:24:16.539 INFO:tasks.rados.rados.0.smithi151.stdout:847: finishing write tid 4 to smithi15186934-5 2024-09-13T14:24:16.539 INFO:tasks.rados.rados.0.smithi151.stdout:847: finishing write tid 5 to smithi15186934-5 2024-09-13T14:24:16.539 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 292 (ObjNum 353 snap 108 seq_num 353) dirty exists 2024-09-13T14:24:16.539 INFO:tasks.rados.rados.0.smithi151.stdout:847: left oid 5 (ObjNum 353 snap 108 seq_num 353) 2024-09-13T14:24:16.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:16 smithi184 ceph-mon[33394]: pgmap v24: 89 pgs: 89 active+clean; 184 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-09-13T14:24:16.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:16 smithi184 ceph-mon[33394]: osdmap e274: 8 total, 8 up, 8 in 2024-09-13T14:24:16.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:16 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:24:16.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:16 smithi151 ceph-mon[104450]: pgmap v24: 89 pgs: 89 active+clean; 184 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-09-13T14:24:16.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:16 smithi151 ceph-mon[104450]: osdmap e274: 8 total, 8 up, 8 in 2024-09-13T14:24:16.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:24:16.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:16 smithi151 ceph-mon[41387]: pgmap v24: 89 pgs: 89 active+clean; 184 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-09-13T14:24:16.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:16 smithi151 ceph-mon[41387]: osdmap e274: 8 total, 8 up, 8 in 2024-09-13T14:24:16.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:16 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:24:17.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:24:16 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:24:16.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi184\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:24:17.531 INFO:tasks.rados.rados.0.smithi151.stdout:846: done (2 left) 2024-09-13T14:24:17.531 INFO:tasks.rados.rados.0.smithi151.stdout:847: done (1 left) 2024-09-13T14:24:17.531 INFO:tasks.rados.rados.0.smithi151.stdout:848: done (0 left) 2024-09-13T14:24:17.531 INFO:tasks.rados.rados.0.smithi151.stdout:849: read oid 12 snap -1 2024-09-13T14:24:17.531 INFO:tasks.rados.rados.0.smithi151.stdout:849: expect (ObjNum 258 snap 81 seq_num 258) 2024-09-13T14:24:17.531 INFO:tasks.rados.rados.0.smithi151.stdout:850: rollback oid 9 current snap is 109 2024-09-13T14:24:17.531 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 9 to 104 2024-09-13T14:24:17.531 INFO:tasks.rados.rados.0.smithi151.stdout:851: snap_remove snap 108 2024-09-13T14:24:17.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:17 smithi184 ceph-mon[33394]: osdmap e275: 8 total, 8 up, 8 in 2024-09-13T14:24:17.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:17 smithi151 ceph-mon[104450]: osdmap e275: 8 total, 8 up, 8 in 2024-09-13T14:24:17.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:17 smithi151 ceph-mon[41387]: osdmap e275: 8 total, 8 up, 8 in 2024-09-13T14:24:18.530 INFO:tasks.rados.rados.0.smithi151.stdout:851: done (2 left) 2024-09-13T14:24:18.530 INFO:tasks.rados.rados.0.smithi151.stdout:852: write oid 25 current snap is 109 2024-09-13T14:24:18.530 INFO:tasks.rados.rados.0.smithi151.stdout:852: seq_num 354 ranges {749678=790384,2332155=504118,3271857=549469} 2024-09-13T14:24:18.538 INFO:tasks.rados.rados.0.smithi151.stdout:852: writing smithi15186934-25 from 749678 to 1540062 tid 1 2024-09-13T14:24:18.545 INFO:tasks.rados.rados.0.smithi151.stdout:852: writing smithi15186934-25 from 2332155 to 2836273 tid 2 2024-09-13T14:24:18.551 INFO:tasks.rados.rados.0.smithi151.stdout:852: writing smithi15186934-25 from 3271857 to 3821326 tid 3 2024-09-13T14:24:18.551 INFO:tasks.rados.rados.0.smithi151.stdout:853: delete oid 34 current snap is 109 2024-09-13T14:24:18.551 INFO:tasks.rados.rados.0.smithi151.stdout:850: finishing rollback tid 1 to smithi15186934-9 2024-09-13T14:24:18.551 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 351 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-13T14:24:18.551 INFO:tasks.rados.rados.0.smithi151.stdout:849: expect (ObjNum 258 snap 81 seq_num 258) 2024-09-13T14:24:18.679 INFO:tasks.rados.rados.0.smithi151.stdout:852: finishing write tid 1 to smithi15186934-25 2024-09-13T14:24:18.679 INFO:tasks.rados.rados.0.smithi151.stdout:852: finishing write tid 2 to smithi15186934-25 2024-09-13T14:24:18.680 INFO:tasks.rados.rados.0.smithi151.stdout:852: finishing write tid 3 to smithi15186934-25 2024-09-13T14:24:18.680 INFO:tasks.rados.rados.0.smithi151.stdout:852: finishing write tid 4 to smithi15186934-25 2024-09-13T14:24:18.680 INFO:tasks.rados.rados.0.smithi151.stdout:852: finishing write tid 5 to smithi15186934-25 2024-09-13T14:24:18.680 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 527 (ObjNum 354 snap 109 seq_num 354) dirty exists 2024-09-13T14:24:18.680 INFO:tasks.rados.rados.0.smithi151.stdout:852: left oid 25 (ObjNum 354 snap 109 seq_num 354) 2024-09-13T14:24:18.680 INFO:tasks.rados.rados.0.smithi151.stdout:849: done (3 left) 2024-09-13T14:24:18.680 INFO:tasks.rados.rados.0.smithi151.stdout:850: done (2 left) 2024-09-13T14:24:18.680 INFO:tasks.rados.rados.0.smithi151.stdout:852: done (1 left) 2024-09-13T14:24:18.680 INFO:tasks.rados.rados.0.smithi151.stdout:853: done (0 left) 2024-09-13T14:24:18.680 INFO:tasks.rados.rados.0.smithi151.stdout:854: read oid 2 snap -1 2024-09-13T14:24:18.680 INFO:tasks.rados.rados.0.smithi151.stdout:854: expect (ObjNum 252 snap 79 seq_num 252) 2024-09-13T14:24:18.680 INFO:tasks.rados.rados.0.smithi151.stdout:855: rollback oid 32 current snap is 109 2024-09-13T14:24:18.681 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 32 to 107 2024-09-13T14:24:18.681 INFO:tasks.rados.rados.0.smithi151.stdout:856: snap_create 2024-09-13T14:24:18.682 INFO:tasks.rados.rados.0.smithi151.stdout:855: finishing rollback tid 0 to smithi15186934-32 2024-09-13T14:24:18.682 INFO:tasks.rados.rados.0.smithi151.stdout:855: finishing rollback tid 1 to smithi15186934-32 2024-09-13T14:24:18.683 INFO:tasks.rados.rados.0.smithi151.stdout:855: finishing rollback tid 2 to smithi15186934-32 2024-09-13T14:24:18.683 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 32 v 388 (ObjNum 304 snap 94 seq_num 304) dirty exists 2024-09-13T14:24:18.684 INFO:tasks.rados.rados.0.smithi151.stdout:854: expect (ObjNum 252 snap 79 seq_num 252) 2024-09-13T14:24:18.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:18 smithi184 ceph-mon[33394]: pgmap v27: 89 pgs: 89 active+clean; 184 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:24:18.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:18 smithi184 ceph-mon[33394]: osdmap e276: 8 total, 8 up, 8 in 2024-09-13T14:24:18.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:18 smithi151 ceph-mon[104450]: pgmap v27: 89 pgs: 89 active+clean; 184 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:24:18.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:18 smithi151 ceph-mon[104450]: osdmap e276: 8 total, 8 up, 8 in 2024-09-13T14:24:18.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:18 smithi151 ceph-mon[41387]: pgmap v27: 89 pgs: 89 active+clean; 184 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:24:18.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:18 smithi151 ceph-mon[41387]: osdmap e276: 8 total, 8 up, 8 in 2024-09-13T14:24:19.545 INFO:tasks.rados.rados.0.smithi151.stdout:854: done (2 left) 2024-09-13T14:24:19.545 INFO:tasks.rados.rados.0.smithi151.stdout:855: done (1 left) 2024-09-13T14:24:19.545 INFO:tasks.rados.rados.0.smithi151.stdout:856: done (0 left) 2024-09-13T14:24:19.545 INFO:tasks.rados.rados.0.smithi151.stdout:857: read oid 25 snap -1 2024-09-13T14:24:19.545 INFO:tasks.rados.rados.0.smithi151.stdout:857: expect (ObjNum 354 snap 109 seq_num 354) 2024-09-13T14:24:19.546 INFO:tasks.rados.rados.0.smithi151.stdout:858: snap_remove snap 98 2024-09-13T14:24:19.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:19 smithi184 ceph-mon[33394]: osdmap e277: 8 total, 8 up, 8 in 2024-09-13T14:24:19.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:19 smithi151 ceph-mon[104450]: osdmap e277: 8 total, 8 up, 8 in 2024-09-13T14:24:19.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:19 smithi151 ceph-mon[41387]: osdmap e277: 8 total, 8 up, 8 in 2024-09-13T14:24:20.545 INFO:tasks.rados.rados.0.smithi151.stdout:858: done (1 left) 2024-09-13T14:24:20.545 INFO:tasks.rados.rados.0.smithi151.stdout:859: write oid 12 current snap is 110 2024-09-13T14:24:20.545 INFO:tasks.rados.rados.0.smithi151.stdout:859: seq_num 356 ranges {783292=540882,1932766=507787,3210350=572013} 2024-09-13T14:24:20.549 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:24:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:24:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:24:20.552 INFO:tasks.rados.rados.0.smithi151.stdout:859: writing smithi15186934-12 from 783292 to 1324174 tid 1 2024-09-13T14:24:20.559 INFO:tasks.rados.rados.0.smithi151.stdout:859: writing smithi15186934-12 from 1932766 to 2440553 tid 2 2024-09-13T14:24:20.567 INFO:tasks.rados.rados.0.smithi151.stdout:859: writing smithi15186934-12 from 3210350 to 3782363 tid 3 2024-09-13T14:24:20.567 INFO:tasks.rados.rados.0.smithi151.stdout:860: delete oid 11 current snap is 110 2024-09-13T14:24:20.567 INFO:tasks.rados.rados.0.smithi151.stdout:857: expect (ObjNum 354 snap 109 seq_num 354) 2024-09-13T14:24:20.743 INFO:tasks.rados.rados.0.smithi151.stdout:859: finishing write tid 1 to smithi15186934-12 2024-09-13T14:24:20.744 INFO:tasks.rados.rados.0.smithi151.stdout:859: finishing write tid 2 to smithi15186934-12 2024-09-13T14:24:20.744 INFO:tasks.rados.rados.0.smithi151.stdout:857: done (2 left) 2024-09-13T14:24:20.744 INFO:tasks.rados.rados.0.smithi151.stdout:860: done (1 left) 2024-09-13T14:24:20.744 INFO:tasks.rados.rados.0.smithi151.stdout:861: read oid 26 snap -1 2024-09-13T14:24:20.744 INFO:tasks.rados.rados.0.smithi151.stdout:861: expect (ObjNum 148 snap 34 seq_num 148) 2024-09-13T14:24:20.744 INFO:tasks.rados.rados.0.smithi151.stdout:862: read oid 9 snap -1 2024-09-13T14:24:20.744 INFO:tasks.rados.rados.0.smithi151.stdout:862: expect deleted 2024-09-13T14:24:20.744 INFO:tasks.rados.rados.0.smithi151.stdout:863: snap_remove snap 103 2024-09-13T14:24:20.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:20 smithi184 ceph-mon[33394]: pgmap v30: 89 pgs: 89 active+clean; 205 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 4.7 MiB/s wr, 27 op/s 2024-09-13T14:24:20.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:20 smithi184 ceph-mon[33394]: osdmap e278: 8 total, 8 up, 8 in 2024-09-13T14:24:20.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:20 smithi151 ceph-mon[104450]: pgmap v30: 89 pgs: 89 active+clean; 205 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 4.7 MiB/s wr, 27 op/s 2024-09-13T14:24:20.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:20 smithi151 ceph-mon[104450]: osdmap e278: 8 total, 8 up, 8 in 2024-09-13T14:24:20.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:20 smithi151 ceph-mon[41387]: pgmap v30: 89 pgs: 89 active+clean; 205 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 4.7 MiB/s wr, 27 op/s 2024-09-13T14:24:20.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:20 smithi151 ceph-mon[41387]: osdmap e278: 8 total, 8 up, 8 in 2024-09-13T14:24:21.567 INFO:tasks.rados.rados.0.smithi151.stdout:863: done (3 left) 2024-09-13T14:24:21.567 INFO:tasks.rados.rados.0.smithi151.stdout:864: delete oid 40 current snap is 110 2024-09-13T14:24:21.567 INFO:tasks.rados.rados.0.smithi151.stdout:859: finishing write tid 3 to smithi15186934-12 2024-09-13T14:24:21.567 INFO:tasks.rados.rados.0.smithi151.stdout:859: finishing write tid 4 to smithi15186934-12 2024-09-13T14:24:21.567 INFO:tasks.rados.rados.0.smithi151.stdout:859: finishing write tid 5 to smithi15186934-12 2024-09-13T14:24:21.567 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 12 v 188 (ObjNum 356 snap 110 seq_num 356) dirty exists 2024-09-13T14:24:21.567 INFO:tasks.rados.rados.0.smithi151.stdout:859: left oid 12 (ObjNum 356 snap 110 seq_num 356) 2024-09-13T14:24:21.567 INFO:tasks.rados.rados.0.smithi151.stdout:861: expect (ObjNum 148 snap 34 seq_num 148) 2024-09-13T14:24:21.685 INFO:tasks.rados.rados.0.smithi151.stdout:859: done (3 left) 2024-09-13T14:24:21.685 INFO:tasks.rados.rados.0.smithi151.stdout:861: done (2 left) 2024-09-13T14:24:21.686 INFO:tasks.rados.rados.0.smithi151.stdout:862: done (1 left) 2024-09-13T14:24:21.686 INFO:tasks.rados.rados.0.smithi151.stdout:864: done (0 left) 2024-09-13T14:24:21.686 INFO:tasks.rados.rados.0.smithi151.stdout:865: read oid 11 snap 105 2024-09-13T14:24:21.686 INFO:tasks.rados.rados.0.smithi151.stdout:865: expect deleted 2024-09-13T14:24:21.686 INFO:tasks.rados.rados.0.smithi151.stdout:866: rollback oid 3 current snap is 110 2024-09-13T14:24:21.686 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 3 to 105 2024-09-13T14:24:21.686 INFO:tasks.rados.rados.0.smithi151.stdout:867: rollback oid 41 current snap is 110 2024-09-13T14:24:21.686 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 41 to 86 2024-09-13T14:24:21.686 INFO:tasks.rados.rados.0.smithi151.stdout:868: snap_remove snap 99 2024-09-13T14:24:21.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:21 smithi184 ceph-mon[33394]: osdmap e279: 8 total, 8 up, 8 in 2024-09-13T14:24:21.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:21 smithi151 ceph-mon[104450]: osdmap e279: 8 total, 8 up, 8 in 2024-09-13T14:24:21.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:21 smithi151 ceph-mon[41387]: osdmap e279: 8 total, 8 up, 8 in 2024-09-13T14:24:22.577 INFO:tasks.rados.rados.0.smithi151.stdout:867: finishing rollback tid 1 to smithi15186934-41 2024-09-13T14:24:22.577 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 0 (ObjNum 33 snap 0 seq_num 3064838224) dirty dne 2024-09-13T14:24:22.578 INFO:tasks.rados.rados.0.smithi151.stdout:866: finishing rollback tid 0 to smithi15186934-3 2024-09-13T14:24:22.578 INFO:tasks.rados.rados.0.smithi151.stdout:866: finishing rollback tid 1 to smithi15186934-3 2024-09-13T14:24:22.578 INFO:tasks.rados.rados.0.smithi151.stdout:866: finishing rollback tid 2 to smithi15186934-3 2024-09-13T14:24:22.578 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 393 (ObjNum 334 snap 103 seq_num 334) dirty exists 2024-09-13T14:24:22.578 INFO:tasks.rados.rados.0.smithi151.stdout:865: done (3 left) 2024-09-13T14:24:22.578 INFO:tasks.rados.rados.0.smithi151.stdout:866: done (2 left) 2024-09-13T14:24:22.578 INFO:tasks.rados.rados.0.smithi151.stdout:867: done (1 left) 2024-09-13T14:24:22.578 INFO:tasks.rados.rados.0.smithi151.stdout:868: done (0 left) 2024-09-13T14:24:22.578 INFO:tasks.rados.rados.0.smithi151.stdout:869: snap_remove snap 86 2024-09-13T14:24:22.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:22 smithi184 ceph-mon[33394]: pgmap v33: 89 pgs: 89 active+clean; 205 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 4.7 MiB/s wr, 27 op/s 2024-09-13T14:24:22.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:22 smithi184 ceph-mon[33394]: osdmap e280: 8 total, 8 up, 8 in 2024-09-13T14:24:22.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:22 smithi151 ceph-mon[104450]: pgmap v33: 89 pgs: 89 active+clean; 205 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 4.7 MiB/s wr, 27 op/s 2024-09-13T14:24:22.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:22 smithi151 ceph-mon[104450]: osdmap e280: 8 total, 8 up, 8 in 2024-09-13T14:24:22.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:22 smithi151 ceph-mon[41387]: pgmap v33: 89 pgs: 89 active+clean; 205 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 4.7 MiB/s wr, 27 op/s 2024-09-13T14:24:22.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:22 smithi151 ceph-mon[41387]: osdmap e280: 8 total, 8 up, 8 in 2024-09-13T14:24:23.587 INFO:tasks.rados.rados.0.smithi151.stdout:869: done (0 left) 2024-09-13T14:24:23.587 INFO:tasks.rados.rados.0.smithi151.stdout:870: read oid 26 snap -1 2024-09-13T14:24:23.587 INFO:tasks.rados.rados.0.smithi151.stdout:870: expect (ObjNum 148 snap 34 seq_num 148) 2024-09-13T14:24:23.587 INFO:tasks.rados.rados.0.smithi151.stdout:871: write oid 47 current snap is 110 2024-09-13T14:24:23.587 INFO:tasks.rados.rados.0.smithi151.stdout:871: seq_num 359 ranges {433713=482982,1424565=609148} 2024-09-13T14:24:23.592 INFO:tasks.rados.rados.0.smithi151.stdout:871: writing smithi15186934-47 from 433713 to 916695 tid 1 2024-09-13T14:24:23.599 INFO:tasks.rados.rados.0.smithi151.stdout:871: writing smithi15186934-47 from 1424565 to 2033713 tid 2 2024-09-13T14:24:23.599 INFO:tasks.rados.rados.0.smithi151.stdout:872: write oid 32 current snap is 110 2024-09-13T14:24:23.600 INFO:tasks.rados.rados.0.smithi151.stdout:872: seq_num 360 ranges {450520=558231,1483047=515125} 2024-09-13T14:24:23.605 INFO:tasks.rados.rados.0.smithi151.stdout:872: writing smithi15186934-32 from 450520 to 1008751 tid 1 2024-09-13T14:24:23.611 INFO:tasks.rados.rados.0.smithi151.stdout:872: writing smithi15186934-32 from 1483047 to 1998172 tid 2 2024-09-13T14:24:23.611 INFO:tasks.rados.rados.0.smithi151.stdout:873: read oid 31 snap 105 2024-09-13T14:24:23.611 INFO:tasks.rados.rados.0.smithi151.stdout:873: expect deleted 2024-09-13T14:24:23.611 INFO:tasks.rados.rados.0.smithi151.stdout:874: write oid 25 current snap is 110 2024-09-13T14:24:23.611 INFO:tasks.rados.rados.0.smithi151.stdout:871: finishing write tid 1 to smithi15186934-47 2024-09-13T14:24:23.611 INFO:tasks.rados.rados.0.smithi151.stdout:870: expect (ObjNum 148 snap 34 seq_num 148) 2024-09-13T14:24:23.728 INFO:tasks.rados.rados.0.smithi151.stdout:871: finishing write tid 2 to smithi15186934-47 2024-09-13T14:24:23.728 INFO:tasks.rados.rados.0.smithi151.stdout:871: finishing write tid 3 to smithi15186934-47 2024-09-13T14:24:23.728 INFO:tasks.rados.rados.0.smithi151.stdout:871: finishing write tid 4 to smithi15186934-47 2024-09-13T14:24:23.728 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 454 (ObjNum 359 snap 110 seq_num 359) dirty exists 2024-09-13T14:24:23.728 INFO:tasks.rados.rados.0.smithi151.stdout:871: left oid 47 (ObjNum 359 snap 110 seq_num 359) 2024-09-13T14:24:23.728 INFO:tasks.rados.rados.0.smithi151.stdout:872: finishing write tid 1 to smithi15186934-32 2024-09-13T14:24:23.728 INFO:tasks.rados.rados.0.smithi151.stdout:874: seq_num 361 ranges {467327=633480,1825176=242151} 2024-09-13T14:24:23.733 INFO:tasks.rados.rados.0.smithi151.stdout:874: writing smithi15186934-25 from 467327 to 1100807 tid 1 2024-09-13T14:24:23.739 INFO:tasks.rados.rados.0.smithi151.stdout:874: writing smithi15186934-25 from 1825176 to 2067327 tid 2 2024-09-13T14:24:23.740 INFO:tasks.rados.rados.0.smithi151.stdout:870: done (4 left) 2024-09-13T14:24:23.740 INFO:tasks.rados.rados.0.smithi151.stdout:871: done (3 left) 2024-09-13T14:24:23.740 INFO:tasks.rados.rados.0.smithi151.stdout:875: read oid 40 snap -1 2024-09-13T14:24:23.740 INFO:tasks.rados.rados.0.smithi151.stdout:875: expect deleted 2024-09-13T14:24:23.740 INFO:tasks.rados.rados.0.smithi151.stdout:876: read oid 17 snap 104 2024-09-13T14:24:23.740 INFO:tasks.rados.rados.0.smithi151.stdout:876: expect (ObjNum 337 snap 104 seq_num 337) 2024-09-13T14:24:23.740 INFO:tasks.rados.rados.0.smithi151.stdout:877: write oid 4 current snap is 110 2024-09-13T14:24:23.740 INFO:tasks.rados.rados.0.smithi151.stdout:872: finishing write tid 2 to smithi15186934-32 2024-09-13T14:24:23.740 INFO:tasks.rados.rados.0.smithi151.stdout:872: finishing write tid 3 to smithi15186934-32 2024-09-13T14:24:23.740 INFO:tasks.rados.rados.0.smithi151.stdout:872: finishing write tid 4 to smithi15186934-32 2024-09-13T14:24:23.740 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 32 v 406 (ObjNum 360 snap 110 seq_num 360) dirty exists 2024-09-13T14:24:23.741 INFO:tasks.rados.rados.0.smithi151.stdout:872: left oid 32 (ObjNum 360 snap 110 seq_num 360) 2024-09-13T14:24:23.741 INFO:tasks.rados.rados.0.smithi151.stdout:874: finishing write tid 1 to smithi15186934-25 2024-09-13T14:24:23.741 INFO:tasks.rados.rados.0.smithi151.stdout:877: seq_num 362 ranges {484134=708729,1883658=200476} 2024-09-13T14:24:23.746 INFO:tasks.rados.rados.0.smithi151.stdout:877: writing smithi15186934-4 from 484134 to 1192863 tid 1 2024-09-13T14:24:23.752 INFO:tasks.rados.rados.0.smithi151.stdout:877: writing smithi15186934-4 from 1883658 to 2084134 tid 2 2024-09-13T14:24:23.752 INFO:tasks.rados.rados.0.smithi151.stdout:872: done (5 left) 2024-09-13T14:24:23.752 INFO:tasks.rados.rados.0.smithi151.stdout:873: done (4 left) 2024-09-13T14:24:23.752 INFO:tasks.rados.rados.0.smithi151.stdout:878: snap_create 2024-09-13T14:24:23.752 INFO:tasks.rados.rados.0.smithi151.stdout:874: finishing write tid 2 to smithi15186934-25 2024-09-13T14:24:23.753 INFO:tasks.rados.rados.0.smithi151.stdout:874: finishing write tid 3 to smithi15186934-25 2024-09-13T14:24:23.753 INFO:tasks.rados.rados.0.smithi151.stdout:874: finishing write tid 4 to smithi15186934-25 2024-09-13T14:24:23.753 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 545 (ObjNum 361 snap 110 seq_num 361) dirty exists 2024-09-13T14:24:23.753 INFO:tasks.rados.rados.0.smithi151.stdout:874: left oid 25 (ObjNum 361 snap 110 seq_num 361) 2024-09-13T14:24:23.753 INFO:tasks.rados.rados.0.smithi151.stdout:877: finishing write tid 1 to smithi15186934-4 2024-09-13T14:24:23.753 INFO:tasks.rados.rados.0.smithi151.stdout:876: expect (ObjNum 337 snap 104 seq_num 337) 2024-09-13T14:24:23.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:23 smithi184 ceph-mon[33394]: osdmap e281: 8 total, 8 up, 8 in 2024-09-13T14:24:23.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:23 smithi151 ceph-mon[41387]: osdmap e281: 8 total, 8 up, 8 in 2024-09-13T14:24:23.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:23 smithi151 ceph-mon[104450]: osdmap e281: 8 total, 8 up, 8 in 2024-09-13T14:24:23.870 INFO:tasks.rados.rados.0.smithi151.stdout:877: finishing write tid 2 to smithi15186934-4 2024-09-13T14:24:23.871 INFO:tasks.rados.rados.0.smithi151.stdout:877: finishing write tid 3 to smithi15186934-4 2024-09-13T14:24:23.871 INFO:tasks.rados.rados.0.smithi151.stdout:877: finishing write tid 4 to smithi15186934-4 2024-09-13T14:24:23.871 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 549 (ObjNum 362 snap 110 seq_num 362) dirty exists 2024-09-13T14:24:23.871 INFO:tasks.rados.rados.0.smithi151.stdout:877: left oid 4 (ObjNum 362 snap 110 seq_num 362) 2024-09-13T14:24:24.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:24:24 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:24:24.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:24:24.442 INFO:tasks.rados.rados.0.smithi151.stdout:874: done (4 left) 2024-09-13T14:24:24.442 INFO:tasks.rados.rados.0.smithi151.stdout:875: done (3 left) 2024-09-13T14:24:24.442 INFO:tasks.rados.rados.0.smithi151.stdout:876: done (2 left) 2024-09-13T14:24:24.442 INFO:tasks.rados.rados.0.smithi151.stdout:877: done (1 left) 2024-09-13T14:24:24.442 INFO:tasks.rados.rados.0.smithi151.stdout:878: done (0 left) 2024-09-13T14:24:24.442 INFO:tasks.rados.rados.0.smithi151.stdout:879: write oid 45 current snap is 111 2024-09-13T14:24:24.443 INFO:tasks.rados.rados.0.smithi151.stdout:879: seq_num 363 ranges {500941=783978,1942140=158801} 2024-09-13T14:24:24.449 INFO:tasks.rados.rados.0.smithi151.stdout:879: writing smithi15186934-45 from 500941 to 1284919 tid 1 2024-09-13T14:24:24.454 INFO:tasks.rados.rados.0.smithi151.stdout:879: writing smithi15186934-45 from 1942140 to 2100941 tid 2 2024-09-13T14:24:24.454 INFO:tasks.rados.rados.0.smithi151.stdout:880: rollback oid 30 current snap is 111 2024-09-13T14:24:24.454 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 30 to 106 2024-09-13T14:24:24.454 INFO:tasks.rados.rados.0.smithi151.stdout:881: read oid 50 snap -1 2024-09-13T14:24:24.454 INFO:tasks.rados.rados.0.smithi151.stdout:881: expect (ObjNum 225 snap 66 seq_num 225) 2024-09-13T14:24:24.454 INFO:tasks.rados.rados.0.smithi151.stdout:882: delete oid 22 current snap is 111 2024-09-13T14:24:24.456 INFO:tasks.rados.rados.0.smithi151.stdout:879: finishing write tid 1 to smithi15186934-45 2024-09-13T14:24:24.456 INFO:tasks.rados.rados.0.smithi151.stdout:879: finishing write tid 2 to smithi15186934-45 2024-09-13T14:24:24.457 INFO:tasks.rados.rados.0.smithi151.stdout:879: finishing write tid 3 to smithi15186934-45 2024-09-13T14:24:24.457 INFO:tasks.rados.rados.0.smithi151.stdout:879: finishing write tid 4 to smithi15186934-45 2024-09-13T14:24:24.457 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 553 (ObjNum 363 snap 111 seq_num 363) dirty exists 2024-09-13T14:24:24.457 INFO:tasks.rados.rados.0.smithi151.stdout:879: left oid 45 (ObjNum 363 snap 111 seq_num 363) 2024-09-13T14:24:24.460 INFO:tasks.rados.rados.0.smithi151.stdout:880: finishing rollback tid 0 to smithi15186934-30 2024-09-13T14:24:24.460 INFO:tasks.rados.rados.0.smithi151.stdout:880: finishing rollback tid 1 to smithi15186934-30 2024-09-13T14:24:24.460 INFO:tasks.rados.rados.0.smithi151.stdout:880: finishing rollback tid 2 to smithi15186934-30 2024-09-13T14:24:24.460 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 30 v 407 (ObjNum 254 snap 79 seq_num 254) dirty exists 2024-09-13T14:24:24.464 INFO:tasks.rados.rados.0.smithi151.stdout:881: expect (ObjNum 225 snap 66 seq_num 225) 2024-09-13T14:24:24.699 INFO:tasks.rados.rados.0.smithi151.stdout:879: done (3 left) 2024-09-13T14:24:24.699 INFO:tasks.rados.rados.0.smithi151.stdout:880: done (2 left) 2024-09-13T14:24:24.699 INFO:tasks.rados.rados.0.smithi151.stdout:881: done (1 left) 2024-09-13T14:24:24.699 INFO:tasks.rados.rados.0.smithi151.stdout:882: done (0 left) 2024-09-13T14:24:24.699 INFO:tasks.rados.rados.0.smithi151.stdout:883: snap_create 2024-09-13T14:24:24.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:24 smithi184 ceph-mon[33394]: pgmap v36: 89 pgs: 89 active+clean; 209 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 10 MiB/s rd, 846 KiB/s wr, 13 op/s 2024-09-13T14:24:24.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:24 smithi184 ceph-mon[33394]: osdmap e282: 8 total, 8 up, 8 in 2024-09-13T14:24:24.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:24 smithi184 ceph-mon[33394]: osdmap e283: 8 total, 8 up, 8 in 2024-09-13T14:24:24.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:24 smithi151 ceph-mon[41387]: pgmap v36: 89 pgs: 89 active+clean; 209 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 10 MiB/s rd, 846 KiB/s wr, 13 op/s 2024-09-13T14:24:24.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:24 smithi151 ceph-mon[41387]: osdmap e282: 8 total, 8 up, 8 in 2024-09-13T14:24:24.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:24 smithi151 ceph-mon[41387]: osdmap e283: 8 total, 8 up, 8 in 2024-09-13T14:24:24.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:24 smithi151 ceph-mon[104450]: pgmap v36: 89 pgs: 89 active+clean; 209 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 10 MiB/s rd, 846 KiB/s wr, 13 op/s 2024-09-13T14:24:24.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:24 smithi151 ceph-mon[104450]: osdmap e282: 8 total, 8 up, 8 in 2024-09-13T14:24:24.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:24 smithi151 ceph-mon[104450]: osdmap e283: 8 total, 8 up, 8 in 2024-09-13T14:24:24.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:24:24] "GET /metrics HTTP/1.1" 200 36404 "" "Prometheus/2.51.0" 2024-09-13T14:24:25.593 INFO:tasks.rados.rados.0.smithi151.stdout:883: done (0 left) 2024-09-13T14:24:25.593 INFO:tasks.rados.rados.0.smithi151.stdout:884: snap_create 2024-09-13T14:24:26.103 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:24:25 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:24:25.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:24:26.103 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:24:25 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:24:25.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:24:26.597 INFO:tasks.rados.rados.0.smithi151.stdout:884: done (0 left) 2024-09-13T14:24:26.598 INFO:tasks.rados.rados.0.smithi151.stdout:885: snap_create 2024-09-13T14:24:26.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:26 smithi184 ceph-mon[33394]: pgmap v39: 89 pgs: 89 active+clean; 209 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 10 MiB/s rd, 846 KiB/s wr, 13 op/s 2024-09-13T14:24:26.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:26 smithi184 ceph-mon[33394]: osdmap e284: 8 total, 8 up, 8 in 2024-09-13T14:24:26.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:26 smithi151 ceph-mon[104450]: pgmap v39: 89 pgs: 89 active+clean; 209 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 10 MiB/s rd, 846 KiB/s wr, 13 op/s 2024-09-13T14:24:26.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:26 smithi151 ceph-mon[104450]: osdmap e284: 8 total, 8 up, 8 in 2024-09-13T14:24:26.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:26 smithi151 ceph-mon[41387]: pgmap v39: 89 pgs: 89 active+clean; 209 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 10 MiB/s rd, 846 KiB/s wr, 13 op/s 2024-09-13T14:24:26.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:26 smithi151 ceph-mon[41387]: osdmap e284: 8 total, 8 up, 8 in 2024-09-13T14:24:27.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:24:26 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:24:26.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi184\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:24:27.617 INFO:tasks.rados.rados.0.smithi151.stdout:885: done (0 left) 2024-09-13T14:24:27.617 INFO:tasks.rados.rados.0.smithi151.stdout:886: delete oid 27 current snap is 114 2024-09-13T14:24:27.620 INFO:tasks.rados.rados.0.smithi151.stdout:886: done (0 left) 2024-09-13T14:24:27.620 INFO:tasks.rados.rados.0.smithi151.stdout:887: read oid 34 snap -1 2024-09-13T14:24:27.620 INFO:tasks.rados.rados.0.smithi151.stdout:887: expect deleted 2024-09-13T14:24:27.620 INFO:tasks.rados.rados.0.smithi151.stdout:888: delete oid 17 current snap is 114 2024-09-13T14:24:27.623 INFO:tasks.rados.rados.0.smithi151.stdout:887: done (1 left) 2024-09-13T14:24:27.623 INFO:tasks.rados.rados.0.smithi151.stdout:888: done (0 left) 2024-09-13T14:24:27.623 INFO:tasks.rados.rados.0.smithi151.stdout:889: snap_remove snap 110 2024-09-13T14:24:27.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:27 smithi184 ceph-mon[33394]: osdmap e285: 8 total, 8 up, 8 in 2024-09-13T14:24:27.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:27 smithi151 ceph-mon[104450]: osdmap e285: 8 total, 8 up, 8 in 2024-09-13T14:24:27.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:27 smithi151 ceph-mon[41387]: osdmap e285: 8 total, 8 up, 8 in 2024-09-13T14:24:28.625 INFO:tasks.rados.rados.0.smithi151.stdout:889: done (0 left) 2024-09-13T14:24:28.626 INFO:tasks.rados.rados.0.smithi151.stdout:890: read oid 21 snap 107 2024-09-13T14:24:28.626 INFO:tasks.rados.rados.0.smithi151.stdout:890: expect (ObjNum 351 snap 107 seq_num 351) 2024-09-13T14:24:28.626 INFO:tasks.rados.rados.0.smithi151.stdout:891: snap_create 2024-09-13T14:24:28.635 INFO:tasks.rados.rados.0.smithi151.stdout:890: expect (ObjNum 351 snap 107 seq_num 351) 2024-09-13T14:24:29.020 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:28 smithi151 ceph-mon[104450]: pgmap v42: 89 pgs: 89 active+clean; 209 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:24:29.021 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:28 smithi151 ceph-mon[104450]: osdmap e286: 8 total, 8 up, 8 in 2024-09-13T14:24:29.021 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:28 smithi151 ceph-mon[41387]: pgmap v42: 89 pgs: 89 active+clean; 209 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:24:29.021 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:28 smithi151 ceph-mon[41387]: osdmap e286: 8 total, 8 up, 8 in 2024-09-13T14:24:29.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:28 smithi184 ceph-mon[33394]: pgmap v42: 89 pgs: 89 active+clean; 209 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:24:29.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:28 smithi184 ceph-mon[33394]: osdmap e286: 8 total, 8 up, 8 in 2024-09-13T14:24:29.628 INFO:tasks.rados.rados.0.smithi151.stdout:890: done (1 left) 2024-09-13T14:24:29.628 INFO:tasks.rados.rados.0.smithi151.stdout:891: done (0 left) 2024-09-13T14:24:29.628 INFO:tasks.rados.rados.0.smithi151.stdout:892: read oid 5 snap -1 2024-09-13T14:24:29.628 INFO:tasks.rados.rados.0.smithi151.stdout:892: expect (ObjNum 353 snap 108 seq_num 353) 2024-09-13T14:24:29.628 INFO:tasks.rados.rados.0.smithi151.stdout:893: read oid 36 snap -1 2024-09-13T14:24:29.628 INFO:tasks.rados.rados.0.smithi151.stdout:893: expect (ObjNum 349 snap 107 seq_num 349) 2024-09-13T14:24:29.628 INFO:tasks.rados.rados.0.smithi151.stdout:894: snap_remove snap 111 2024-09-13T14:24:29.995 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:29 smithi151 ceph-mon[104450]: osdmap e287: 8 total, 8 up, 8 in 2024-09-13T14:24:29.995 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:29 smithi151 ceph-mon[41387]: osdmap e287: 8 total, 8 up, 8 in 2024-09-13T14:24:30.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:29 smithi184 ceph-mon[33394]: osdmap e287: 8 total, 8 up, 8 in 2024-09-13T14:24:30.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:24:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:24:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:24:30.631 INFO:tasks.rados.rados.0.smithi151.stdout:894: done (2 left) 2024-09-13T14:24:30.632 INFO:tasks.rados.rados.0.smithi151.stdout:895: snap_create 2024-09-13T14:24:30.632 INFO:tasks.rados.rados.0.smithi151.stdout:893: expect (ObjNum 349 snap 107 seq_num 349) 2024-09-13T14:24:30.878 INFO:tasks.rados.rados.0.smithi151.stdout:892: expect (ObjNum 353 snap 108 seq_num 353) 2024-09-13T14:24:31.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:30 smithi184 ceph-mon[33394]: pgmap v45: 89 pgs: 89 active+clean; 175 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.9 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-09-13T14:24:31.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:30 smithi184 ceph-mon[33394]: osdmap e288: 8 total, 8 up, 8 in 2024-09-13T14:24:31.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:30 smithi151 ceph-mon[104450]: pgmap v45: 89 pgs: 89 active+clean; 175 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.9 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-09-13T14:24:31.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:30 smithi151 ceph-mon[104450]: osdmap e288: 8 total, 8 up, 8 in 2024-09-13T14:24:31.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:30 smithi151 ceph-mon[41387]: pgmap v45: 89 pgs: 89 active+clean; 175 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.9 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-09-13T14:24:31.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:30 smithi151 ceph-mon[41387]: osdmap e288: 8 total, 8 up, 8 in 2024-09-13T14:24:31.649 INFO:tasks.rados.rados.0.smithi151.stdout:892: done (2 left) 2024-09-13T14:24:31.649 INFO:tasks.rados.rados.0.smithi151.stdout:893: done (1 left) 2024-09-13T14:24:31.649 INFO:tasks.rados.rados.0.smithi151.stdout:895: done (0 left) 2024-09-13T14:24:31.649 INFO:tasks.rados.rados.0.smithi151.stdout:896: write oid 35 current snap is 116 2024-09-13T14:24:31.649 INFO:tasks.rados.rados.0.smithi151.stdout:896: seq_num 367 ranges {568169=401327,1376068=669790} 2024-09-13T14:24:31.654 INFO:tasks.rados.rados.0.smithi151.stdout:896: writing smithi15186934-35 from 568169 to 969496 tid 1 2024-09-13T14:24:31.661 INFO:tasks.rados.rados.0.smithi151.stdout:896: writing smithi15186934-35 from 1376068 to 2045858 tid 2 2024-09-13T14:24:31.661 INFO:tasks.rados.rados.0.smithi151.stdout:897: read oid 45 snap -1 2024-09-13T14:24:31.661 INFO:tasks.rados.rados.0.smithi151.stdout:897: expect (ObjNum 363 snap 111 seq_num 363) 2024-09-13T14:24:31.661 INFO:tasks.rados.rados.0.smithi151.stdout:898: snap_remove snap 102 2024-09-13T14:24:32.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:31 smithi184 ceph-mon[33394]: osdmap e289: 8 total, 8 up, 8 in 2024-09-13T14:24:32.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:31 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:24:32.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:31 smithi151 ceph-mon[41387]: osdmap e289: 8 total, 8 up, 8 in 2024-09-13T14:24:32.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:31 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:24:32.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:31 smithi151 ceph-mon[104450]: osdmap e289: 8 total, 8 up, 8 in 2024-09-13T14:24:32.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:24:32.651 INFO:tasks.rados.rados.0.smithi151.stdout:898: done (2 left) 2024-09-13T14:24:32.651 INFO:tasks.rados.rados.0.smithi151.stdout:899: write oid 18 current snap is 116 2024-09-13T14:24:32.651 INFO:tasks.rados.rados.0.smithi151.stdout:899: seq_num 368 ranges {584976=476576,1834550=350426} 2024-09-13T14:24:32.658 INFO:tasks.rados.rados.0.smithi151.stdout:899: writing smithi15186934-18 from 584976 to 1061552 tid 1 2024-09-13T14:24:32.664 INFO:tasks.rados.rados.0.smithi151.stdout:899: writing smithi15186934-18 from 1834550 to 2184976 tid 2 2024-09-13T14:24:32.664 INFO:tasks.rados.rados.0.smithi151.stdout:900: snap_remove snap 109 2024-09-13T14:24:33.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:32 smithi184 ceph-mon[33394]: pgmap v48: 89 pgs: 89 active+clean; 175 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.9 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-09-13T14:24:33.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:32 smithi184 ceph-mon[33394]: osdmap e290: 8 total, 8 up, 8 in 2024-09-13T14:24:33.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:32 smithi151 ceph-mon[104450]: pgmap v48: 89 pgs: 89 active+clean; 175 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.9 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-09-13T14:24:33.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:32 smithi151 ceph-mon[104450]: osdmap e290: 8 total, 8 up, 8 in 2024-09-13T14:24:33.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:32 smithi151 ceph-mon[41387]: pgmap v48: 89 pgs: 89 active+clean; 175 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.9 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-09-13T14:24:33.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:32 smithi151 ceph-mon[41387]: osdmap e290: 8 total, 8 up, 8 in 2024-09-13T14:24:33.662 INFO:tasks.rados.rados.0.smithi151.stdout:900: done (3 left) 2024-09-13T14:24:33.662 INFO:tasks.rados.rados.0.smithi151.stdout:901: rollback oid 15 current snap is 116 2024-09-13T14:24:33.662 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 15 to 115 2024-09-13T14:24:33.662 INFO:tasks.rados.rados.0.smithi151.stdout:896: finishing write tid 1 to smithi15186934-35 2024-09-13T14:24:33.662 INFO:tasks.rados.rados.0.smithi151.stdout:896: finishing write tid 2 to smithi15186934-35 2024-09-13T14:24:33.663 INFO:tasks.rados.rados.0.smithi151.stdout:896: finishing write tid 3 to smithi15186934-35 2024-09-13T14:24:33.663 INFO:tasks.rados.rados.0.smithi151.stdout:896: finishing write tid 4 to smithi15186934-35 2024-09-13T14:24:33.663 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 35 v 614 (ObjNum 367 snap 116 seq_num 367) dirty exists 2024-09-13T14:24:33.663 INFO:tasks.rados.rados.0.smithi151.stdout:896: left oid 35 (ObjNum 367 snap 116 seq_num 367) 2024-09-13T14:24:33.663 INFO:tasks.rados.rados.0.smithi151.stdout:897: expect (ObjNum 363 snap 111 seq_num 363) 2024-09-13T14:24:33.776 INFO:tasks.rados.rados.0.smithi151.stdout:899: finishing write tid 1 to smithi15186934-18 2024-09-13T14:24:33.776 INFO:tasks.rados.rados.0.smithi151.stdout:899: finishing write tid 2 to smithi15186934-18 2024-09-13T14:24:33.776 INFO:tasks.rados.rados.0.smithi151.stdout:899: finishing write tid 3 to smithi15186934-18 2024-09-13T14:24:33.776 INFO:tasks.rados.rados.0.smithi151.stdout:899: finishing write tid 4 to smithi15186934-18 2024-09-13T14:24:33.776 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 18 v 258 (ObjNum 368 snap 116 seq_num 368) dirty exists 2024-09-13T14:24:33.776 INFO:tasks.rados.rados.0.smithi151.stdout:899: left oid 18 (ObjNum 368 snap 116 seq_num 368) 2024-09-13T14:24:33.777 INFO:tasks.rados.rados.0.smithi151.stdout:901: finishing rollback tid 0 to smithi15186934-15 2024-09-13T14:24:33.777 INFO:tasks.rados.rados.0.smithi151.stdout:896: done (3 left) 2024-09-13T14:24:33.777 INFO:tasks.rados.rados.0.smithi151.stdout:897: done (2 left) 2024-09-13T14:24:33.777 INFO:tasks.rados.rados.0.smithi151.stdout:899: done (1 left) 2024-09-13T14:24:33.777 INFO:tasks.rados.rados.0.smithi151.stdout:902: read oid 33 snap -1 2024-09-13T14:24:33.777 INFO:tasks.rados.rados.0.smithi151.stdout:902: expect deleted 2024-09-13T14:24:33.777 INFO:tasks.rados.rados.0.smithi151.stdout:903: snap_create 2024-09-13T14:24:33.777 INFO:tasks.rados.rados.0.smithi151.stdout:901: finishing rollback tid 1 to smithi15186934-15 2024-09-13T14:24:33.777 INFO:tasks.rados.rados.0.smithi151.stdout:901: finishing rollback tid 2 to smithi15186934-15 2024-09-13T14:24:33.777 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 15 v 502 (ObjNum 335 snap 103 seq_num 335) dirty exists 2024-09-13T14:24:34.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:33 smithi184 ceph-mon[33394]: osdmap e291: 8 total, 8 up, 8 in 2024-09-13T14:24:34.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:33 smithi151 ceph-mon[104450]: osdmap e291: 8 total, 8 up, 8 in 2024-09-13T14:24:34.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:33 smithi151 ceph-mon[41387]: osdmap e291: 8 total, 8 up, 8 in 2024-09-13T14:24:34.450 INFO:tasks.rados.rados.0.smithi151.stdout:901: done (2 left) 2024-09-13T14:24:34.450 INFO:tasks.rados.rados.0.smithi151.stdout:902: done (1 left) 2024-09-13T14:24:34.450 INFO:tasks.rados.rados.0.smithi151.stdout:903: done (0 left) 2024-09-13T14:24:34.450 INFO:tasks.rados.rados.0.smithi151.stdout:904: write oid 12 current snap is 117 2024-09-13T14:24:34.450 INFO:tasks.rados.rados.0.smithi151.stdout:904: seq_num 369 ranges {601783=551825,1776679=425104} 2024-09-13T14:24:34.456 INFO:tasks.rados.rados.0.smithi151.stdout:904: writing smithi15186934-12 from 601783 to 1153608 tid 1 2024-09-13T14:24:34.462 INFO:tasks.rados.rados.0.smithi151.stdout:904: writing smithi15186934-12 from 1776679 to 2201783 tid 2 2024-09-13T14:24:34.462 INFO:tasks.rados.rados.0.smithi151.stdout:905: delete oid 36 current snap is 117 2024-09-13T14:24:34.462 INFO:tasks.rados.rados.0.smithi151.stdout:904: finishing write tid 1 to smithi15186934-12 2024-09-13T14:24:34.466 INFO:tasks.rados.rados.0.smithi151.stdout:904: finishing write tid 2 to smithi15186934-12 2024-09-13T14:24:34.466 INFO:tasks.rados.rados.0.smithi151.stdout:904: finishing write tid 3 to smithi15186934-12 2024-09-13T14:24:34.466 INFO:tasks.rados.rados.0.smithi151.stdout:904: finishing write tid 4 to smithi15186934-12 2024-09-13T14:24:34.466 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 12 v 208 (ObjNum 369 snap 117 seq_num 369) dirty exists 2024-09-13T14:24:34.466 INFO:tasks.rados.rados.0.smithi151.stdout:904: left oid 12 (ObjNum 369 snap 117 seq_num 369) 2024-09-13T14:24:34.466 INFO:tasks.rados.rados.0.smithi151.stdout:904: done (1 left) 2024-09-13T14:24:34.466 INFO:tasks.rados.rados.0.smithi151.stdout:905: done (0 left) 2024-09-13T14:24:34.466 INFO:tasks.rados.rados.0.smithi151.stdout:906: read oid 33 snap -1 2024-09-13T14:24:34.467 INFO:tasks.rados.rados.0.smithi151.stdout:906: expect deleted 2024-09-13T14:24:34.467 INFO:tasks.rados.rados.0.smithi151.stdout:907: delete oid 34 current snap is 117 2024-09-13T14:24:34.467 INFO:tasks.rados.rados.0.smithi151.stdout:907: done (1 left) 2024-09-13T14:24:34.467 INFO:tasks.rados.rados.0.smithi151.stdout:908: snap_create 2024-09-13T14:24:34.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:24:34 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:24:34.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:24:34.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:34 smithi151 ceph-mon[104450]: pgmap v51: 89 pgs: 89 active+clean; 176 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 262 KiB/s wr, 9 op/s 2024-09-13T14:24:34.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:34 smithi151 ceph-mon[104450]: osdmap e292: 8 total, 8 up, 8 in 2024-09-13T14:24:34.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:34 smithi151 ceph-mon[104450]: osdmap e293: 8 total, 8 up, 8 in 2024-09-13T14:24:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:34 smithi151 ceph-mon[41387]: pgmap v51: 89 pgs: 89 active+clean; 176 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 262 KiB/s wr, 9 op/s 2024-09-13T14:24:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:34 smithi151 ceph-mon[41387]: osdmap e292: 8 total, 8 up, 8 in 2024-09-13T14:24:34.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:34 smithi151 ceph-mon[41387]: osdmap e293: 8 total, 8 up, 8 in 2024-09-13T14:24:34.860 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:24:34] "GET /metrics HTTP/1.1" 200 36406 "" "Prometheus/2.51.0" 2024-09-13T14:24:35.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:34 smithi184 ceph-mon[33394]: pgmap v51: 89 pgs: 89 active+clean; 176 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 262 KiB/s wr, 9 op/s 2024-09-13T14:24:35.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:34 smithi184 ceph-mon[33394]: osdmap e292: 8 total, 8 up, 8 in 2024-09-13T14:24:35.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:34 smithi184 ceph-mon[33394]: osdmap e293: 8 total, 8 up, 8 in 2024-09-13T14:24:35.452 INFO:tasks.rados.rados.0.smithi151.stdout:906: done (1 left) 2024-09-13T14:24:35.452 INFO:tasks.rados.rados.0.smithi151.stdout:908: done (0 left) 2024-09-13T14:24:35.452 INFO:tasks.rados.rados.0.smithi151.stdout:909: delete oid 23 current snap is 118 2024-09-13T14:24:35.456 INFO:tasks.rados.rados.0.smithi151.stdout:909: done (0 left) 2024-09-13T14:24:35.456 INFO:tasks.rados.rados.0.smithi151.stdout:910: delete oid 30 current snap is 118 2024-09-13T14:24:35.458 INFO:tasks.rados.rados.0.smithi151.stdout:910: done (0 left) 2024-09-13T14:24:35.458 INFO:tasks.rados.rados.0.smithi151.stdout:911: write oid 39 current snap is 118 2024-09-13T14:24:35.458 INFO:tasks.rados.rados.0.smithi151.stdout:911: seq_num 374 ranges {685818=644423,2069089=216729} 2024-09-13T14:24:35.465 INFO:tasks.rados.rados.0.smithi151.stdout:911: writing smithi15186934-39 from 685818 to 1330241 tid 1 2024-09-13T14:24:35.471 INFO:tasks.rados.rados.0.smithi151.stdout:911: writing smithi15186934-39 from 2069089 to 2285818 tid 2 2024-09-13T14:24:35.471 INFO:tasks.rados.rados.0.smithi151.stdout:912: snap_create 2024-09-13T14:24:35.471 INFO:tasks.rados.rados.0.smithi151.stdout:911: finishing write tid 1 to smithi15186934-39 2024-09-13T14:24:35.473 INFO:tasks.rados.rados.0.smithi151.stdout:911: finishing write tid 2 to smithi15186934-39 2024-09-13T14:24:35.473 INFO:tasks.rados.rados.0.smithi151.stdout:911: finishing write tid 3 to smithi15186934-39 2024-09-13T14:24:35.473 INFO:tasks.rados.rados.0.smithi151.stdout:911: finishing write tid 4 to smithi15186934-39 2024-09-13T14:24:35.473 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 262 (ObjNum 374 snap 118 seq_num 374) dirty exists 2024-09-13T14:24:35.474 INFO:tasks.rados.rados.0.smithi151.stdout:911: left oid 39 (ObjNum 374 snap 118 seq_num 374) 2024-09-13T14:24:36.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:24:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:24:35.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:24:36.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:24:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:24:35.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:24:36.457 INFO:tasks.rados.rados.0.smithi151.stdout:911: done (1 left) 2024-09-13T14:24:36.457 INFO:tasks.rados.rados.0.smithi151.stdout:912: done (0 left) 2024-09-13T14:24:36.457 INFO:tasks.rados.rados.0.smithi151.stdout:913: delete oid 43 current snap is 119 2024-09-13T14:24:36.461 INFO:tasks.rados.rados.0.smithi151.stdout:913: done (0 left) 2024-09-13T14:24:36.462 INFO:tasks.rados.rados.0.smithi151.stdout:914: delete oid 5 current snap is 119 2024-09-13T14:24:36.465 INFO:tasks.rados.rados.0.smithi151.stdout:914: done (0 left) 2024-09-13T14:24:36.465 INFO:tasks.rados.rados.0.smithi151.stdout:915: read oid 3 snap -1 2024-09-13T14:24:36.465 INFO:tasks.rados.rados.0.smithi151.stdout:915: expect (ObjNum 334 snap 103 seq_num 334) 2024-09-13T14:24:36.466 INFO:tasks.rados.rados.0.smithi151.stdout:916: rollback oid 37 current snap is 119 2024-09-13T14:24:36.466 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 37 to 115 2024-09-13T14:24:36.466 INFO:tasks.rados.rados.0.smithi151.stdout:917: snap_remove snap 104 2024-09-13T14:24:36.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:36 smithi184 ceph-mon[33394]: pgmap v54: 89 pgs: 89 active+clean; 176 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 262 KiB/s wr, 9 op/s 2024-09-13T14:24:36.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:36 smithi184 ceph-mon[33394]: osdmap e294: 8 total, 8 up, 8 in 2024-09-13T14:24:36.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:36 smithi151 ceph-mon[104450]: pgmap v54: 89 pgs: 89 active+clean; 176 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 262 KiB/s wr, 9 op/s 2024-09-13T14:24:36.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:36 smithi151 ceph-mon[104450]: osdmap e294: 8 total, 8 up, 8 in 2024-09-13T14:24:36.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:36 smithi151 ceph-mon[41387]: pgmap v54: 89 pgs: 89 active+clean; 176 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 262 KiB/s wr, 9 op/s 2024-09-13T14:24:36.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:36 smithi151 ceph-mon[41387]: osdmap e294: 8 total, 8 up, 8 in 2024-09-13T14:24:37.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:24:36 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:24:36.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi184\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:24:37.462 INFO:tasks.rados.rados.0.smithi151.stdout:917: done (2 left) 2024-09-13T14:24:37.462 INFO:tasks.rados.rados.0.smithi151.stdout:918: rollback oid 40 current snap is 119 2024-09-13T14:24:37.462 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 40 to 115 2024-09-13T14:24:37.462 INFO:tasks.rados.rados.0.smithi151.stdout:919: write oid 2 current snap is 119 2024-09-13T14:24:37.463 INFO:tasks.rados.rados.0.smithi151.stdout:919: seq_num 377 ranges {736239=470170,1728182=571782} 2024-09-13T14:24:37.469 INFO:tasks.rados.rados.0.smithi151.stdout:919: writing smithi15186934-2 from 736239 to 1206409 tid 1 2024-09-13T14:24:37.476 INFO:tasks.rados.rados.0.smithi151.stdout:919: writing smithi15186934-2 from 1728182 to 2299964 tid 2 2024-09-13T14:24:37.476 INFO:tasks.rados.rados.0.smithi151.stdout:920: delete oid 25 current snap is 119 2024-09-13T14:24:37.476 INFO:tasks.rados.rados.0.smithi151.stdout:916: finishing rollback tid 0 to smithi15186934-37 2024-09-13T14:24:37.476 INFO:tasks.rados.rados.0.smithi151.stdout:916: finishing rollback tid 1 to smithi15186934-37 2024-09-13T14:24:37.476 INFO:tasks.rados.rados.0.smithi151.stdout:916: finishing rollback tid 2 to smithi15186934-37 2024-09-13T14:24:37.477 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 37 v 465 (ObjNum 179 snap 47 seq_num 179) dirty exists 2024-09-13T14:24:37.477 INFO:tasks.rados.rados.0.smithi151.stdout:915: expect (ObjNum 334 snap 103 seq_num 334) 2024-09-13T14:24:37.592 INFO:tasks.rados.rados.0.smithi151.stdout:918: finishing rollback tid 1 to smithi15186934-40 2024-09-13T14:24:37.592 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 40 v 389 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-13T14:24:37.592 INFO:tasks.rados.rados.0.smithi151.stdout:919: finishing write tid 1 to smithi15186934-2 2024-09-13T14:24:37.592 INFO:tasks.rados.rados.0.smithi151.stdout:919: finishing write tid 2 to smithi15186934-2 2024-09-13T14:24:37.592 INFO:tasks.rados.rados.0.smithi151.stdout:919: finishing write tid 3 to smithi15186934-2 2024-09-13T14:24:37.593 INFO:tasks.rados.rados.0.smithi151.stdout:919: finishing write tid 4 to smithi15186934-2 2024-09-13T14:24:37.593 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 2 v 326 (ObjNum 377 snap 119 seq_num 377) dirty exists 2024-09-13T14:24:37.593 INFO:tasks.rados.rados.0.smithi151.stdout:919: left oid 2 (ObjNum 377 snap 119 seq_num 377) 2024-09-13T14:24:37.593 INFO:tasks.rados.rados.0.smithi151.stdout:915: done (4 left) 2024-09-13T14:24:37.593 INFO:tasks.rados.rados.0.smithi151.stdout:916: done (3 left) 2024-09-13T14:24:37.593 INFO:tasks.rados.rados.0.smithi151.stdout:918: done (2 left) 2024-09-13T14:24:37.593 INFO:tasks.rados.rados.0.smithi151.stdout:919: done (1 left) 2024-09-13T14:24:37.593 INFO:tasks.rados.rados.0.smithi151.stdout:920: done (0 left) 2024-09-13T14:24:37.593 INFO:tasks.rados.rados.0.smithi151.stdout:921: rollback oid 9 current snap is 119 2024-09-13T14:24:37.593 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 9 to 105 2024-09-13T14:24:37.593 INFO:tasks.rados.rados.0.smithi151.stdout:922: read oid 1 snap -1 2024-09-13T14:24:37.593 INFO:tasks.rados.rados.0.smithi151.stdout:922: expect deleted 2024-09-13T14:24:37.593 INFO:tasks.rados.rados.0.smithi151.stdout:923: read oid 7 snap -1 2024-09-13T14:24:37.594 INFO:tasks.rados.rados.0.smithi151.stdout:923: expect deleted 2024-09-13T14:24:37.594 INFO:tasks.rados.rados.0.smithi151.stdout:924: rollback oid 44 current snap is 119 2024-09-13T14:24:37.594 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 44 to 118 2024-09-13T14:24:37.594 INFO:tasks.rados.rados.0.smithi151.stdout:925: snap_remove snap 112 2024-09-13T14:24:37.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:37 smithi184 ceph-mon[33394]: osdmap e295: 8 total, 8 up, 8 in 2024-09-13T14:24:37.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:37 smithi151 ceph-mon[104450]: osdmap e295: 8 total, 8 up, 8 in 2024-09-13T14:24:37.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:37 smithi151 ceph-mon[41387]: osdmap e295: 8 total, 8 up, 8 in 2024-09-13T14:24:38.467 INFO:tasks.rados.rados.0.smithi151.stdout:921: finishing rollback tid 1 to smithi15186934-9 2024-09-13T14:24:38.467 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 351 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-13T14:24:38.467 INFO:tasks.rados.rados.0.smithi151.stdout:924: finishing rollback tid 0 to smithi15186934-44 2024-09-13T14:24:38.467 INFO:tasks.rados.rados.0.smithi151.stdout:924: finishing rollback tid 1 to smithi15186934-44 2024-09-13T14:24:38.467 INFO:tasks.rados.rados.0.smithi151.stdout:924: finishing rollback tid 2 to smithi15186934-44 2024-09-13T14:24:38.467 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 254 (ObjNum 324 snap 101 seq_num 324) dirty exists 2024-09-13T14:24:38.467 INFO:tasks.rados.rados.0.smithi151.stdout:921: done (4 left) 2024-09-13T14:24:38.467 INFO:tasks.rados.rados.0.smithi151.stdout:922: done (3 left) 2024-09-13T14:24:38.467 INFO:tasks.rados.rados.0.smithi151.stdout:923: done (2 left) 2024-09-13T14:24:38.467 INFO:tasks.rados.rados.0.smithi151.stdout:924: done (1 left) 2024-09-13T14:24:38.468 INFO:tasks.rados.rados.0.smithi151.stdout:925: done (0 left) 2024-09-13T14:24:38.468 INFO:tasks.rados.rados.0.smithi151.stdout:926: rollback oid 41 current snap is 119 2024-09-13T14:24:38.468 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 41 to 100 2024-09-13T14:24:38.468 INFO:tasks.rados.rados.0.smithi151.stdout:927: snap_remove snap 115 2024-09-13T14:24:38.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:38 smithi184 ceph-mon[33394]: pgmap v57: 89 pgs: 89 active+clean; 176 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:24:38.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:38 smithi184 ceph-mon[33394]: osdmap e296: 8 total, 8 up, 8 in 2024-09-13T14:24:38.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:38 smithi151 ceph-mon[104450]: pgmap v57: 89 pgs: 89 active+clean; 176 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:24:38.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:38 smithi151 ceph-mon[104450]: osdmap e296: 8 total, 8 up, 8 in 2024-09-13T14:24:38.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:38 smithi151 ceph-mon[41387]: pgmap v57: 89 pgs: 89 active+clean; 176 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:24:38.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:38 smithi151 ceph-mon[41387]: osdmap e296: 8 total, 8 up, 8 in 2024-09-13T14:24:39.487 INFO:tasks.rados.rados.0.smithi151.stdout:927: done (1 left) 2024-09-13T14:24:39.487 INFO:tasks.rados.rados.0.smithi151.stdout:928: read oid 27 snap -1 2024-09-13T14:24:39.487 INFO:tasks.rados.rados.0.smithi151.stdout:928: expect deleted 2024-09-13T14:24:39.487 INFO:tasks.rados.rados.0.smithi151.stdout:929: write oid 43 current snap is 119 2024-09-13T14:24:39.487 INFO:tasks.rados.rados.0.smithi151.stdout:926: finishing rollback tid 1 to smithi15186934-41 2024-09-13T14:24:39.487 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 0 (ObjNum 129 snap 0 seq_num 3092793392) dirty dne 2024-09-13T14:24:39.487 INFO:tasks.rados.rados.0.smithi151.stdout:929: seq_num 379 ranges {769853=620668,1845146=524707} 2024-09-13T14:24:39.494 INFO:tasks.rados.rados.0.smithi151.stdout:929: writing smithi15186934-43 from 769853 to 1390521 tid 1 2024-09-13T14:24:39.500 INFO:tasks.rados.rados.0.smithi151.stdout:929: writing smithi15186934-43 from 1845146 to 2369853 tid 2 2024-09-13T14:24:39.500 INFO:tasks.rados.rados.0.smithi151.stdout:926: done (2 left) 2024-09-13T14:24:39.500 INFO:tasks.rados.rados.0.smithi151.stdout:930: write oid 17 current snap is 119 2024-09-13T14:24:39.500 INFO:tasks.rados.rados.0.smithi151.stdout:930: seq_num 380 ranges {786660=695917,1903628=435462} 2024-09-13T14:24:39.508 INFO:tasks.rados.rados.0.smithi151.stdout:930: writing smithi15186934-17 from 786660 to 1482577 tid 1 2024-09-13T14:24:39.513 INFO:tasks.rados.rados.0.smithi151.stdout:930: writing smithi15186934-17 from 1903628 to 2339090 tid 2 2024-09-13T14:24:39.513 INFO:tasks.rados.rados.0.smithi151.stdout:931: read oid 31 snap -1 2024-09-13T14:24:39.513 INFO:tasks.rados.rados.0.smithi151.stdout:931: expect deleted 2024-09-13T14:24:39.513 INFO:tasks.rados.rados.0.smithi151.stdout:932: snap_remove snap 116 2024-09-13T14:24:39.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:39 smithi184 ceph-mon[33394]: osdmap e297: 8 total, 8 up, 8 in 2024-09-13T14:24:39.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:39 smithi151 ceph-mon[104450]: osdmap e297: 8 total, 8 up, 8 in 2024-09-13T14:24:39.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:39 smithi151 ceph-mon[41387]: osdmap e297: 8 total, 8 up, 8 in 2024-09-13T14:24:40.484 INFO:tasks.rados.rados.0.smithi151.stdout:929: finishing write tid 1 to smithi15186934-43 2024-09-13T14:24:40.484 INFO:tasks.rados.rados.0.smithi151.stdout:929: finishing write tid 2 to smithi15186934-43 2024-09-13T14:24:40.485 INFO:tasks.rados.rados.0.smithi151.stdout:929: finishing write tid 3 to smithi15186934-43 2024-09-13T14:24:40.485 INFO:tasks.rados.rados.0.smithi151.stdout:929: finishing write tid 4 to smithi15186934-43 2024-09-13T14:24:40.485 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 43 v 639 (ObjNum 379 snap 119 seq_num 379) dirty exists 2024-09-13T14:24:40.485 INFO:tasks.rados.rados.0.smithi151.stdout:929: left oid 43 (ObjNum 379 snap 119 seq_num 379) 2024-09-13T14:24:40.485 INFO:tasks.rados.rados.0.smithi151.stdout:930: finishing write tid 1 to smithi15186934-17 2024-09-13T14:24:40.485 INFO:tasks.rados.rados.0.smithi151.stdout:930: finishing write tid 2 to smithi15186934-17 2024-09-13T14:24:40.485 INFO:tasks.rados.rados.0.smithi151.stdout:930: finishing write tid 3 to smithi15186934-17 2024-09-13T14:24:40.485 INFO:tasks.rados.rados.0.smithi151.stdout:930: finishing write tid 4 to smithi15186934-17 2024-09-13T14:24:40.485 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 642 (ObjNum 380 snap 119 seq_num 380) dirty exists 2024-09-13T14:24:40.485 INFO:tasks.rados.rados.0.smithi151.stdout:930: left oid 17 (ObjNum 380 snap 119 seq_num 380) 2024-09-13T14:24:40.485 INFO:tasks.rados.rados.0.smithi151.stdout:928: done (4 left) 2024-09-13T14:24:40.485 INFO:tasks.rados.rados.0.smithi151.stdout:929: done (3 left) 2024-09-13T14:24:40.485 INFO:tasks.rados.rados.0.smithi151.stdout:930: done (2 left) 2024-09-13T14:24:40.485 INFO:tasks.rados.rados.0.smithi151.stdout:931: done (1 left) 2024-09-13T14:24:40.486 INFO:tasks.rados.rados.0.smithi151.stdout:932: done (0 left) 2024-09-13T14:24:40.486 INFO:tasks.rados.rados.0.smithi151.stdout:933: read oid 48 snap -1 2024-09-13T14:24:40.486 INFO:tasks.rados.rados.0.smithi151.stdout:933: expect (ObjNum 342 snap 105 seq_num 342) 2024-09-13T14:24:40.486 INFO:tasks.rados.rados.0.smithi151.stdout:934: snap_create 2024-09-13T14:24:40.487 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:24:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:24:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:24:40.501 INFO:tasks.rados.rados.0.smithi151.stdout:933: expect (ObjNum 342 snap 105 seq_num 342) 2024-09-13T14:24:40.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:40 smithi184 ceph-mon[33394]: pgmap v60: 89 pgs: 89 active+clean; 174 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 651 KiB/s wr, 7 op/s 2024-09-13T14:24:40.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:40 smithi184 ceph-mon[33394]: osdmap e298: 8 total, 8 up, 8 in 2024-09-13T14:24:40.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:40 smithi151 ceph-mon[104450]: pgmap v60: 89 pgs: 89 active+clean; 174 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 651 KiB/s wr, 7 op/s 2024-09-13T14:24:40.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:40 smithi151 ceph-mon[104450]: osdmap e298: 8 total, 8 up, 8 in 2024-09-13T14:24:40.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:40 smithi151 ceph-mon[41387]: pgmap v60: 89 pgs: 89 active+clean; 174 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 651 KiB/s wr, 7 op/s 2024-09-13T14:24:40.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:40 smithi151 ceph-mon[41387]: osdmap e298: 8 total, 8 up, 8 in 2024-09-13T14:24:41.496 INFO:tasks.rados.rados.0.smithi151.stdout:933: done (1 left) 2024-09-13T14:24:41.496 INFO:tasks.rados.rados.0.smithi151.stdout:934: done (0 left) 2024-09-13T14:24:41.497 INFO:tasks.rados.rados.0.smithi151.stdout:935: rollback oid 28 current snap is 120 2024-09-13T14:24:41.497 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 28 to 117 2024-09-13T14:24:41.497 INFO:tasks.rados.rados.0.smithi151.stdout:936: read oid 29 snap -1 2024-09-13T14:24:41.497 INFO:tasks.rados.rados.0.smithi151.stdout:936: expect deleted 2024-09-13T14:24:41.497 INFO:tasks.rados.rados.0.smithi151.stdout:937: rollback oid 42 current snap is 120 2024-09-13T14:24:41.497 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 42 to 119 2024-09-13T14:24:41.497 INFO:tasks.rados.rados.0.smithi151.stdout:938: read oid 46 snap -1 2024-09-13T14:24:41.497 INFO:tasks.rados.rados.0.smithi151.stdout:938: expect (ObjNum 261 snap 82 seq_num 261) 2024-09-13T14:24:41.497 INFO:tasks.rados.rados.0.smithi151.stdout:939: write oid 1 current snap is 120 2024-09-13T14:24:41.497 INFO:tasks.rados.rados.0.smithi151.stdout:939: seq_num 381 ranges {403467=487519,1562110=579120} 2024-09-13T14:24:41.502 INFO:tasks.rados.rados.0.smithi151.stdout:939: writing smithi15186934-1 from 403467 to 890986 tid 1 2024-09-13T14:24:41.509 INFO:tasks.rados.rados.0.smithi151.stdout:939: writing smithi15186934-1 from 1562110 to 2141230 tid 2 2024-09-13T14:24:41.509 INFO:tasks.rados.rados.0.smithi151.stdout:940: read oid 30 snap -1 2024-09-13T14:24:41.509 INFO:tasks.rados.rados.0.smithi151.stdout:940: expect deleted 2024-09-13T14:24:41.509 INFO:tasks.rados.rados.0.smithi151.stdout:941: delete oid 48 current snap is 120 2024-09-13T14:24:41.510 INFO:tasks.rados.rados.0.smithi151.stdout:935: finishing rollback tid 1 to smithi15186934-28 2024-09-13T14:24:41.510 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 28 v 464 (ObjNum 305 snap 0 seq_num 0) dirty dne 2024-09-13T14:24:41.510 INFO:tasks.rados.rados.0.smithi151.stdout:937: finishing rollback tid 0 to smithi15186934-42 2024-09-13T14:24:41.510 INFO:tasks.rados.rados.0.smithi151.stdout:937: finishing rollback tid 1 to smithi15186934-42 2024-09-13T14:24:41.510 INFO:tasks.rados.rados.0.smithi151.stdout:937: finishing rollback tid 2 to smithi15186934-42 2024-09-13T14:24:41.510 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 515 (ObjNum 322 snap 100 seq_num 322) dirty exists 2024-09-13T14:24:41.510 INFO:tasks.rados.rados.0.smithi151.stdout:938: expect (ObjNum 261 snap 82 seq_num 261) 2024-09-13T14:24:41.611 INFO:tasks.rados.rados.0.smithi151.stdout:939: finishing write tid 1 to smithi15186934-1 2024-09-13T14:24:41.611 INFO:tasks.rados.rados.0.smithi151.stdout:939: finishing write tid 2 to smithi15186934-1 2024-09-13T14:24:41.611 INFO:tasks.rados.rados.0.smithi151.stdout:939: finishing write tid 3 to smithi15186934-1 2024-09-13T14:24:41.611 INFO:tasks.rados.rados.0.smithi151.stdout:939: finishing write tid 4 to smithi15186934-1 2024-09-13T14:24:41.611 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 1 v 263 (ObjNum 381 snap 120 seq_num 381) dirty exists 2024-09-13T14:24:41.611 INFO:tasks.rados.rados.0.smithi151.stdout:939: left oid 1 (ObjNum 381 snap 120 seq_num 381) 2024-09-13T14:24:41.612 INFO:tasks.rados.rados.0.smithi151.stdout:935: done (6 left) 2024-09-13T14:24:41.612 INFO:tasks.rados.rados.0.smithi151.stdout:936: done (5 left) 2024-09-13T14:24:41.612 INFO:tasks.rados.rados.0.smithi151.stdout:937: done (4 left) 2024-09-13T14:24:41.612 INFO:tasks.rados.rados.0.smithi151.stdout:938: done (3 left) 2024-09-13T14:24:41.612 INFO:tasks.rados.rados.0.smithi151.stdout:939: done (2 left) 2024-09-13T14:24:41.612 INFO:tasks.rados.rados.0.smithi151.stdout:940: done (1 left) 2024-09-13T14:24:41.612 INFO:tasks.rados.rados.0.smithi151.stdout:941: done (0 left) 2024-09-13T14:24:41.612 INFO:tasks.rados.rados.0.smithi151.stdout:942: snap_remove snap 114 2024-09-13T14:24:41.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:41 smithi184 ceph-mon[33394]: osdmap e299: 8 total, 8 up, 8 in 2024-09-13T14:24:41.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:41 smithi151 ceph-mon[104450]: osdmap e299: 8 total, 8 up, 8 in 2024-09-13T14:24:41.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:41 smithi151 ceph-mon[41387]: osdmap e299: 8 total, 8 up, 8 in 2024-09-13T14:24:42.503 INFO:tasks.rados.rados.0.smithi151.stdout:942: done (0 left) 2024-09-13T14:24:42.503 INFO:tasks.rados.rados.0.smithi151.stdout:943: read oid 5 snap -1 2024-09-13T14:24:42.503 INFO:tasks.rados.rados.0.smithi151.stdout:943: expect deleted 2024-09-13T14:24:42.503 INFO:tasks.rados.rados.0.smithi151.stdout:944: delete oid 37 current snap is 120 2024-09-13T14:24:42.506 INFO:tasks.rados.rados.0.smithi151.stdout:944: done (1 left) 2024-09-13T14:24:42.506 INFO:tasks.rados.rados.0.smithi151.stdout:945: write oid 45 current snap is 120 2024-09-13T14:24:42.506 INFO:tasks.rados.rados.0.smithi151.stdout:945: seq_num 384 ranges {453888=713266,1737556=442800} 2024-09-13T14:24:42.513 INFO:tasks.rados.rados.0.smithi151.stdout:945: writing smithi15186934-45 from 453888 to 1167154 tid 1 2024-09-13T14:24:42.519 INFO:tasks.rados.rados.0.smithi151.stdout:945: writing smithi15186934-45 from 1737556 to 2180356 tid 2 2024-09-13T14:24:42.519 INFO:tasks.rados.rados.0.smithi151.stdout:946: read oid 33 snap -1 2024-09-13T14:24:42.519 INFO:tasks.rados.rados.0.smithi151.stdout:946: expect deleted 2024-09-13T14:24:42.519 INFO:tasks.rados.rados.0.smithi151.stdout:947: snap_remove snap 100 2024-09-13T14:24:42.755 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:42 smithi184 ceph-mon[33394]: pgmap v63: 89 pgs: 89 active+clean; 174 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 651 KiB/s wr, 7 op/s 2024-09-13T14:24:42.755 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:42 smithi184 ceph-mon[33394]: osdmap e300: 8 total, 8 up, 8 in 2024-09-13T14:24:42.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:42 smithi151 ceph-mon[41387]: pgmap v63: 89 pgs: 89 active+clean; 174 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 651 KiB/s wr, 7 op/s 2024-09-13T14:24:42.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:42 smithi151 ceph-mon[41387]: osdmap e300: 8 total, 8 up, 8 in 2024-09-13T14:24:42.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:42 smithi151 ceph-mon[104450]: pgmap v63: 89 pgs: 89 active+clean; 174 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 651 KiB/s wr, 7 op/s 2024-09-13T14:24:42.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:42 smithi151 ceph-mon[104450]: osdmap e300: 8 total, 8 up, 8 in 2024-09-13T14:24:43.513 INFO:tasks.rados.rados.0.smithi151.stdout:947: done (3 left) 2024-09-13T14:24:43.514 INFO:tasks.rados.rados.0.smithi151.stdout:948: write oid 42 current snap is 120 2024-09-13T14:24:43.514 INFO:tasks.rados.rados.0.smithi151.stdout:948: seq_num 385 ranges {470695=788515,1679685=586458} 2024-09-13T14:24:43.521 INFO:tasks.rados.rados.0.smithi151.stdout:948: writing smithi15186934-42 from 470695 to 1259210 tid 1 2024-09-13T14:24:43.527 INFO:tasks.rados.rados.0.smithi151.stdout:948: writing smithi15186934-42 from 1679685 to 2266143 tid 2 2024-09-13T14:24:43.527 INFO:tasks.rados.rados.0.smithi151.stdout:949: write oid 22 current snap is 120 2024-09-13T14:24:43.527 INFO:tasks.rados.rados.0.smithi151.stdout:949: seq_num 386 ranges {487502=463764,1738167=446469} 2024-09-13T14:24:43.532 INFO:tasks.rados.rados.0.smithi151.stdout:949: writing smithi15186934-22 from 487502 to 951266 tid 1 2024-09-13T14:24:43.540 INFO:tasks.rados.rados.0.smithi151.stdout:949: writing smithi15186934-22 from 1738167 to 2184636 tid 2 2024-09-13T14:24:43.540 INFO:tasks.rados.rados.0.smithi151.stdout:950: rollback oid 25 current snap is 120 2024-09-13T14:24:43.540 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 25 to 117 2024-09-13T14:24:43.540 INFO:tasks.rados.rados.0.smithi151.stdout:945: finishing write tid 1 to smithi15186934-45 2024-09-13T14:24:43.540 INFO:tasks.rados.rados.0.smithi151.stdout:945: finishing write tid 2 to smithi15186934-45 2024-09-13T14:24:43.540 INFO:tasks.rados.rados.0.smithi151.stdout:945: finishing write tid 3 to smithi15186934-45 2024-09-13T14:24:43.540 INFO:tasks.rados.rados.0.smithi151.stdout:945: finishing write tid 4 to smithi15186934-45 2024-09-13T14:24:43.540 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 591 (ObjNum 384 snap 120 seq_num 384) dirty exists 2024-09-13T14:24:43.540 INFO:tasks.rados.rados.0.smithi151.stdout:945: left oid 45 (ObjNum 384 snap 120 seq_num 384) 2024-09-13T14:24:43.540 INFO:tasks.rados.rados.0.smithi151.stdout:943: done (5 left) 2024-09-13T14:24:43.541 INFO:tasks.rados.rados.0.smithi151.stdout:945: done (4 left) 2024-09-13T14:24:43.541 INFO:tasks.rados.rados.0.smithi151.stdout:946: done (3 left) 2024-09-13T14:24:43.541 INFO:tasks.rados.rados.0.smithi151.stdout:951: write oid 45 current snap is 120 2024-09-13T14:24:43.541 INFO:tasks.rados.rados.0.smithi151.stdout:948: finishing write tid 1 to smithi15186934-42 2024-09-13T14:24:43.541 INFO:tasks.rados.rados.0.smithi151.stdout:948: finishing write tid 2 to smithi15186934-42 2024-09-13T14:24:43.541 INFO:tasks.rados.rados.0.smithi151.stdout:948: finishing write tid 3 to smithi15186934-42 2024-09-13T14:24:43.541 INFO:tasks.rados.rados.0.smithi151.stdout:948: finishing write tid 4 to smithi15186934-42 2024-09-13T14:24:43.541 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 528 (ObjNum 385 snap 120 seq_num 385) dirty exists 2024-09-13T14:24:43.541 INFO:tasks.rados.rados.0.smithi151.stdout:948: left oid 42 (ObjNum 385 snap 120 seq_num 385) 2024-09-13T14:24:43.541 INFO:tasks.rados.rados.0.smithi151.stdout:949: finishing write tid 1 to smithi15186934-22 2024-09-13T14:24:43.541 INFO:tasks.rados.rados.0.smithi151.stdout:951: seq_num 387 ranges {504309=539013,1796649=590127} 2024-09-13T14:24:43.545 INFO:tasks.rados.rados.0.smithi151.stdout:951: writing smithi15186934-45 from 504309 to 1043322 tid 1 2024-09-13T14:24:43.552 INFO:tasks.rados.rados.0.smithi151.stdout:951: writing smithi15186934-45 from 1796649 to 2386776 tid 2 2024-09-13T14:24:43.552 INFO:tasks.rados.rados.0.smithi151.stdout:948: done (3 left) 2024-09-13T14:24:43.552 INFO:tasks.rados.rados.0.smithi151.stdout:952: snap_remove snap 106 2024-09-13T14:24:43.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:43 smithi184 ceph-mon[33394]: osdmap e301: 8 total, 8 up, 8 in 2024-09-13T14:24:43.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:43 smithi151 ceph-mon[104450]: osdmap e301: 8 total, 8 up, 8 in 2024-09-13T14:24:43.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:43 smithi151 ceph-mon[41387]: osdmap e301: 8 total, 8 up, 8 in 2024-09-13T14:24:44.452 INFO:tasks.rados.rados.0.smithi151.stdout:950: finishing rollback tid 1 to smithi15186934-25 2024-09-13T14:24:44.452 INFO:tasks.rados.rados.0.smithi151.stdout:950: finishing rollback tid 2 to smithi15186934-25 2024-09-13T14:24:44.452 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 599 (ObjNum 361 snap 110 seq_num 361) dirty exists 2024-09-13T14:24:44.452 INFO:tasks.rados.rados.0.smithi151.stdout:949: finishing write tid 2 to smithi15186934-22 2024-09-13T14:24:44.452 INFO:tasks.rados.rados.0.smithi151.stdout:949: finishing write tid 3 to smithi15186934-22 2024-09-13T14:24:44.453 INFO:tasks.rados.rados.0.smithi151.stdout:949: finishing write tid 4 to smithi15186934-22 2024-09-13T14:24:44.453 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 22 v 234 (ObjNum 386 snap 120 seq_num 386) dirty exists 2024-09-13T14:24:44.453 INFO:tasks.rados.rados.0.smithi151.stdout:949: left oid 22 (ObjNum 386 snap 120 seq_num 386) 2024-09-13T14:24:44.453 INFO:tasks.rados.rados.0.smithi151.stdout:951: finishing write tid 1 to smithi15186934-45 2024-09-13T14:24:44.453 INFO:tasks.rados.rados.0.smithi151.stdout:951: finishing write tid 2 to smithi15186934-45 2024-09-13T14:24:44.453 INFO:tasks.rados.rados.0.smithi151.stdout:951: finishing write tid 3 to smithi15186934-45 2024-09-13T14:24:44.453 INFO:tasks.rados.rados.0.smithi151.stdout:951: finishing write tid 4 to smithi15186934-45 2024-09-13T14:24:44.453 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 602 (ObjNum 387 snap 120 seq_num 387) dirty exists 2024-09-13T14:24:44.453 INFO:tasks.rados.rados.0.smithi151.stdout:951: left oid 45 (ObjNum 387 snap 120 seq_num 387) 2024-09-13T14:24:44.453 INFO:tasks.rados.rados.0.smithi151.stdout:949: done (3 left) 2024-09-13T14:24:44.453 INFO:tasks.rados.rados.0.smithi151.stdout:950: done (2 left) 2024-09-13T14:24:44.453 INFO:tasks.rados.rados.0.smithi151.stdout:951: done (1 left) 2024-09-13T14:24:44.453 INFO:tasks.rados.rados.0.smithi151.stdout:952: done (0 left) 2024-09-13T14:24:44.454 INFO:tasks.rados.rados.0.smithi151.stdout:953: snap_create 2024-09-13T14:24:44.507 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:24:44 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:24:44.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:24:44.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:44 smithi184 ceph-mon[33394]: pgmap v66: 89 pgs: 89 active+clean; 178 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.0 MiB/s wr, 10 op/s 2024-09-13T14:24:44.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:44 smithi184 ceph-mon[33394]: osdmap e302: 8 total, 8 up, 8 in 2024-09-13T14:24:44.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:44 smithi184 ceph-mon[33394]: osdmap e303: 8 total, 8 up, 8 in 2024-09-13T14:24:44.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:44 smithi151 ceph-mon[104450]: pgmap v66: 89 pgs: 89 active+clean; 178 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.0 MiB/s wr, 10 op/s 2024-09-13T14:24:44.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:44 smithi151 ceph-mon[104450]: osdmap e302: 8 total, 8 up, 8 in 2024-09-13T14:24:44.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:44 smithi151 ceph-mon[104450]: osdmap e303: 8 total, 8 up, 8 in 2024-09-13T14:24:44.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:44 smithi151 ceph-mon[41387]: pgmap v66: 89 pgs: 89 active+clean; 178 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.0 MiB/s wr, 10 op/s 2024-09-13T14:24:44.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:44 smithi151 ceph-mon[41387]: osdmap e302: 8 total, 8 up, 8 in 2024-09-13T14:24:44.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:44 smithi151 ceph-mon[41387]: osdmap e303: 8 total, 8 up, 8 in 2024-09-13T14:24:44.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:24:44] "GET /metrics HTTP/1.1" 200 36406 "" "Prometheus/2.51.0" 2024-09-13T14:24:45.450 INFO:tasks.rados.rados.0.smithi151.stdout:953: done (0 left) 2024-09-13T14:24:45.451 INFO:tasks.rados.rados.0.smithi151.stdout:954: snap_remove snap 117 2024-09-13T14:24:46.103 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:24:45 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:24:45.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:24:46.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:24:45 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:24:45.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:24:46.456 INFO:tasks.rados.rados.0.smithi151.stdout:954: done (0 left) 2024-09-13T14:24:46.456 INFO:tasks.rados.rados.0.smithi151.stdout:955: delete oid 50 current snap is 121 2024-09-13T14:24:46.459 INFO:tasks.rados.rados.0.smithi151.stdout:955: done (0 left) 2024-09-13T14:24:46.460 INFO:tasks.rados.rados.0.smithi151.stdout:956: delete oid 37 current snap is 121 2024-09-13T14:24:46.463 INFO:tasks.rados.rados.0.smithi151.stdout:956: done (0 left) 2024-09-13T14:24:46.463 INFO:tasks.rados.rados.0.smithi151.stdout:957: write oid 39 current snap is 121 2024-09-13T14:24:46.463 INFO:tasks.rados.rados.0.smithi151.stdout:957: seq_num 390 ranges {554730=481113,1572095=737454} 2024-09-13T14:24:46.469 INFO:tasks.rados.rados.0.smithi151.stdout:957: writing smithi15186934-39 from 554730 to 1035843 tid 1 2024-09-13T14:24:46.476 INFO:tasks.rados.rados.0.smithi151.stdout:957: writing smithi15186934-39 from 1572095 to 2309549 tid 2 2024-09-13T14:24:46.476 INFO:tasks.rados.rados.0.smithi151.stdout:958: rollback oid 32 current snap is 121 2024-09-13T14:24:46.476 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 32 to 120 2024-09-13T14:24:46.476 INFO:tasks.rados.rados.0.smithi151.stdout:957: finishing write tid 1 to smithi15186934-39 2024-09-13T14:24:46.476 INFO:tasks.rados.rados.0.smithi151.stdout:959: snap_remove snap 118 2024-09-13T14:24:46.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:46 smithi184 ceph-mon[33394]: pgmap v69: 89 pgs: 89 active+clean; 178 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.0 MiB/s wr, 10 op/s 2024-09-13T14:24:46.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:46 smithi184 ceph-mon[33394]: osdmap e304: 8 total, 8 up, 8 in 2024-09-13T14:24:46.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:46 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:24:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:46 smithi151 ceph-mon[104450]: pgmap v69: 89 pgs: 89 active+clean; 178 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.0 MiB/s wr, 10 op/s 2024-09-13T14:24:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:46 smithi151 ceph-mon[104450]: osdmap e304: 8 total, 8 up, 8 in 2024-09-13T14:24:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:24:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:46 smithi151 ceph-mon[41387]: pgmap v69: 89 pgs: 89 active+clean; 178 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.0 MiB/s wr, 10 op/s 2024-09-13T14:24:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:46 smithi151 ceph-mon[41387]: osdmap e304: 8 total, 8 up, 8 in 2024-09-13T14:24:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:46 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:24:47.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:24:46 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:24:46.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi184\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:24:47.469 INFO:tasks.rados.rados.0.smithi151.stdout:958: finishing rollback tid 0 to smithi15186934-32 2024-09-13T14:24:47.469 INFO:tasks.rados.rados.0.smithi151.stdout:958: finishing rollback tid 1 to smithi15186934-32 2024-09-13T14:24:47.469 INFO:tasks.rados.rados.0.smithi151.stdout:958: finishing rollback tid 2 to smithi15186934-32 2024-09-13T14:24:47.469 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 32 v 425 (ObjNum 360 snap 110 seq_num 360) dirty exists 2024-09-13T14:24:47.470 INFO:tasks.rados.rados.0.smithi151.stdout:957: finishing write tid 2 to smithi15186934-39 2024-09-13T14:24:47.470 INFO:tasks.rados.rados.0.smithi151.stdout:958: done (2 left) 2024-09-13T14:24:47.470 INFO:tasks.rados.rados.0.smithi151.stdout:959: done (1 left) 2024-09-13T14:24:47.470 INFO:tasks.rados.rados.0.smithi151.stdout:960: write oid 24 current snap is 121 2024-09-13T14:24:47.470 INFO:tasks.rados.rados.0.smithi151.stdout:960: seq_num 391 ranges {571537=556362,1630577=597465} 2024-09-13T14:24:47.475 INFO:tasks.rados.rados.0.smithi151.stdout:960: writing smithi15186934-24 from 571537 to 1127899 tid 1 2024-09-13T14:24:47.482 INFO:tasks.rados.rados.0.smithi151.stdout:960: writing smithi15186934-24 from 1630577 to 2228042 tid 2 2024-09-13T14:24:47.482 INFO:tasks.rados.rados.0.smithi151.stdout:961: snap_remove snap 119 2024-09-13T14:24:47.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:47 smithi184 ceph-mon[33394]: osdmap e305: 8 total, 8 up, 8 in 2024-09-13T14:24:47.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:47 smithi151 ceph-mon[104450]: osdmap e305: 8 total, 8 up, 8 in 2024-09-13T14:24:47.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:47 smithi151 ceph-mon[41387]: osdmap e305: 8 total, 8 up, 8 in 2024-09-13T14:24:48.467 INFO:tasks.rados.rados.0.smithi151.stdout:957: finishing write tid 3 to smithi15186934-39 2024-09-13T14:24:48.468 INFO:tasks.rados.rados.0.smithi151.stdout:957: finishing write tid 4 to smithi15186934-39 2024-09-13T14:24:48.468 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 286 (ObjNum 390 snap 121 seq_num 390) dirty exists 2024-09-13T14:24:48.468 INFO:tasks.rados.rados.0.smithi151.stdout:957: left oid 39 (ObjNum 390 snap 121 seq_num 390) 2024-09-13T14:24:48.468 INFO:tasks.rados.rados.0.smithi151.stdout:960: finishing write tid 1 to smithi15186934-24 2024-09-13T14:24:48.468 INFO:tasks.rados.rados.0.smithi151.stdout:960: finishing write tid 2 to smithi15186934-24 2024-09-13T14:24:48.468 INFO:tasks.rados.rados.0.smithi151.stdout:960: finishing write tid 3 to smithi15186934-24 2024-09-13T14:24:48.468 INFO:tasks.rados.rados.0.smithi151.stdout:960: finishing write tid 4 to smithi15186934-24 2024-09-13T14:24:48.468 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 24 v 668 (ObjNum 391 snap 121 seq_num 391) dirty exists 2024-09-13T14:24:48.468 INFO:tasks.rados.rados.0.smithi151.stdout:960: left oid 24 (ObjNum 391 snap 121 seq_num 391) 2024-09-13T14:24:48.468 INFO:tasks.rados.rados.0.smithi151.stdout:957: done (2 left) 2024-09-13T14:24:48.468 INFO:tasks.rados.rados.0.smithi151.stdout:960: done (1 left) 2024-09-13T14:24:48.469 INFO:tasks.rados.rados.0.smithi151.stdout:961: done (0 left) 2024-09-13T14:24:48.469 INFO:tasks.rados.rados.0.smithi151.stdout:962: rollback oid 29 current snap is 121 2024-09-13T14:24:48.469 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 29 to 101 2024-09-13T14:24:48.469 INFO:tasks.rados.rados.0.smithi151.stdout:963: write oid 17 current snap is 121 2024-09-13T14:24:48.469 INFO:tasks.rados.rados.0.smithi151.stdout:963: seq_num 392 ranges {588344=631611,1689059=741123} 2024-09-13T14:24:48.475 INFO:tasks.rados.rados.0.smithi151.stdout:963: writing smithi15186934-17 from 588344 to 1219955 tid 1 2024-09-13T14:24:48.482 INFO:tasks.rados.rados.0.smithi151.stdout:963: writing smithi15186934-17 from 1689059 to 2430182 tid 2 2024-09-13T14:24:48.482 INFO:tasks.rados.rados.0.smithi151.stdout:964: delete oid 5 current snap is 121 2024-09-13T14:24:48.482 INFO:tasks.rados.rados.0.smithi151.stdout:962: finishing rollback tid 1 to smithi15186934-29 2024-09-13T14:24:48.482 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 29 v 0 (ObjNum 113 snap 0 seq_num 3680511042) dirty dne 2024-09-13T14:24:48.482 INFO:tasks.rados.rados.0.smithi151.stdout:963: finishing write tid 1 to smithi15186934-17 2024-09-13T14:24:48.483 INFO:tasks.rados.rados.0.smithi151.stdout:962: done (2 left) 2024-09-13T14:24:48.484 INFO:tasks.rados.rados.0.smithi151.stdout:964: done (1 left) 2024-09-13T14:24:48.484 INFO:tasks.rados.rados.0.smithi151.stdout:965: write oid 11 current snap is 121 2024-09-13T14:24:48.484 INFO:tasks.rados.rados.0.smithi151.stdout:965: seq_num 394 ranges {621958=782109,2089670=532288} 2024-09-13T14:24:48.491 INFO:tasks.rados.rados.0.smithi151.stdout:965: writing smithi15186934-11 from 621958 to 1404067 tid 1 2024-09-13T14:24:48.498 INFO:tasks.rados.rados.0.smithi151.stdout:965: writing smithi15186934-11 from 2089670 to 2621958 tid 2 2024-09-13T14:24:48.498 INFO:tasks.rados.rados.0.smithi151.stdout:966: snap_remove snap 120 2024-09-13T14:24:48.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:48 smithi184 ceph-mon[33394]: pgmap v72: 89 pgs: 89 active+clean; 177 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:24:48.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:48 smithi184 ceph-mon[33394]: osdmap e306: 8 total, 8 up, 8 in 2024-09-13T14:24:48.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:48 smithi151 ceph-mon[104450]: pgmap v72: 89 pgs: 89 active+clean; 177 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:24:48.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:48 smithi151 ceph-mon[104450]: osdmap e306: 8 total, 8 up, 8 in 2024-09-13T14:24:48.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:48 smithi151 ceph-mon[41387]: pgmap v72: 89 pgs: 89 active+clean; 177 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:24:48.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:48 smithi151 ceph-mon[41387]: osdmap e306: 8 total, 8 up, 8 in 2024-09-13T14:24:49.480 INFO:tasks.rados.rados.0.smithi151.stdout:963: finishing write tid 2 to smithi15186934-17 2024-09-13T14:24:49.480 INFO:tasks.rados.rados.0.smithi151.stdout:963: finishing write tid 3 to smithi15186934-17 2024-09-13T14:24:49.481 INFO:tasks.rados.rados.0.smithi151.stdout:963: finishing write tid 4 to smithi15186934-17 2024-09-13T14:24:49.481 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 674 (ObjNum 392 snap 121 seq_num 392) dirty exists 2024-09-13T14:24:49.481 INFO:tasks.rados.rados.0.smithi151.stdout:963: left oid 17 (ObjNum 392 snap 121 seq_num 392) 2024-09-13T14:24:49.481 INFO:tasks.rados.rados.0.smithi151.stdout:965: finishing write tid 1 to smithi15186934-11 2024-09-13T14:24:49.481 INFO:tasks.rados.rados.0.smithi151.stdout:965: finishing write tid 2 to smithi15186934-11 2024-09-13T14:24:49.481 INFO:tasks.rados.rados.0.smithi151.stdout:965: finishing write tid 3 to smithi15186934-11 2024-09-13T14:24:49.481 INFO:tasks.rados.rados.0.smithi151.stdout:965: finishing write tid 4 to smithi15186934-11 2024-09-13T14:24:49.481 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 11 v 275 (ObjNum 394 snap 121 seq_num 394) dirty exists 2024-09-13T14:24:49.481 INFO:tasks.rados.rados.0.smithi151.stdout:965: left oid 11 (ObjNum 394 snap 121 seq_num 394) 2024-09-13T14:24:49.481 INFO:tasks.rados.rados.0.smithi151.stdout:963: done (2 left) 2024-09-13T14:24:49.481 INFO:tasks.rados.rados.0.smithi151.stdout:965: done (1 left) 2024-09-13T14:24:49.481 INFO:tasks.rados.rados.0.smithi151.stdout:966: done (0 left) 2024-09-13T14:24:49.481 INFO:tasks.rados.rados.0.smithi151.stdout:967: snap_create 2024-09-13T14:24:49.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:49 smithi184 ceph-mon[33394]: osdmap e307: 8 total, 8 up, 8 in 2024-09-13T14:24:49.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:49 smithi151 ceph-mon[104450]: osdmap e307: 8 total, 8 up, 8 in 2024-09-13T14:24:49.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:49 smithi151 ceph-mon[41387]: osdmap e307: 8 total, 8 up, 8 in 2024-09-13T14:24:50.483 INFO:tasks.rados.rados.0.smithi151.stdout:967: done (0 left) 2024-09-13T14:24:50.484 INFO:tasks.rados.rados.0.smithi151.stdout:968: snap_create 2024-09-13T14:24:50.484 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:24:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:24:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:24:50.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:50 smithi184 ceph-mon[33394]: pgmap v75: 89 pgs: 89 active+clean; 177 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 MiB/s wr, 7 op/s 2024-09-13T14:24:50.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:50 smithi184 ceph-mon[33394]: osdmap e308: 8 total, 8 up, 8 in 2024-09-13T14:24:50.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:50 smithi151 ceph-mon[104450]: pgmap v75: 89 pgs: 89 active+clean; 177 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 MiB/s wr, 7 op/s 2024-09-13T14:24:50.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:50 smithi151 ceph-mon[104450]: osdmap e308: 8 total, 8 up, 8 in 2024-09-13T14:24:50.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:50 smithi151 ceph-mon[41387]: pgmap v75: 89 pgs: 89 active+clean; 177 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 MiB/s wr, 7 op/s 2024-09-13T14:24:50.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:50 smithi151 ceph-mon[41387]: osdmap e308: 8 total, 8 up, 8 in 2024-09-13T14:24:51.501 INFO:tasks.rados.rados.0.smithi151.stdout:968: done (0 left) 2024-09-13T14:24:51.501 INFO:tasks.rados.rados.0.smithi151.stdout:969: write oid 31 current snap is 123 2024-09-13T14:24:51.501 INFO:tasks.rados.rados.0.smithi151.stdout:969: seq_num 395 ranges {638765=457358,1748152=604803} 2024-09-13T14:24:51.507 INFO:tasks.rados.rados.0.smithi151.stdout:969: writing smithi15186934-31 from 638765 to 1096123 tid 1 2024-09-13T14:24:51.515 INFO:tasks.rados.rados.0.smithi151.stdout:969: writing smithi15186934-31 from 1748152 to 2352955 tid 2 2024-09-13T14:24:51.515 INFO:tasks.rados.rados.0.smithi151.stdout:970: write oid 7 current snap is 123 2024-09-13T14:24:51.515 INFO:tasks.rados.rados.0.smithi151.stdout:969: finishing write tid 1 to smithi15186934-31 2024-09-13T14:24:51.515 INFO:tasks.rados.rados.0.smithi151.stdout:970: seq_num 396 ranges {655572=648960,1922987=732585} 2024-09-13T14:24:51.521 INFO:tasks.rados.rados.0.smithi151.stdout:970: writing smithi15186934-7 from 655572 to 1304532 tid 1 2024-09-13T14:24:51.529 INFO:tasks.rados.rados.0.smithi151.stdout:970: writing smithi15186934-7 from 1922987 to 2655572 tid 2 2024-09-13T14:24:51.529 INFO:tasks.rados.rados.0.smithi151.stdout:971: write oid 45 current snap is 123 2024-09-13T14:24:51.529 INFO:tasks.rados.rados.0.smithi151.stdout:969: finishing write tid 2 to smithi15186934-31 2024-09-13T14:24:51.530 INFO:tasks.rados.rados.0.smithi151.stdout:969: finishing write tid 3 to smithi15186934-31 2024-09-13T14:24:51.530 INFO:tasks.rados.rados.0.smithi151.stdout:969: finishing write tid 4 to smithi15186934-31 2024-09-13T14:24:51.530 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 31 v 541 (ObjNum 395 snap 123 seq_num 395) dirty exists 2024-09-13T14:24:51.530 INFO:tasks.rados.rados.0.smithi151.stdout:969: left oid 31 (ObjNum 395 snap 123 seq_num 395) 2024-09-13T14:24:51.530 INFO:tasks.rados.rados.0.smithi151.stdout:970: finishing write tid 1 to smithi15186934-7 2024-09-13T14:24:51.530 INFO:tasks.rados.rados.0.smithi151.stdout:971: seq_num 397 ranges {672379=724209,1865116=608472} 2024-09-13T14:24:51.537 INFO:tasks.rados.rados.0.smithi151.stdout:971: writing smithi15186934-45 from 672379 to 1396588 tid 1 2024-09-13T14:24:51.544 INFO:tasks.rados.rados.0.smithi151.stdout:971: writing smithi15186934-45 from 1865116 to 2473588 tid 2 2024-09-13T14:24:51.544 INFO:tasks.rados.rados.0.smithi151.stdout:969: done (2 left) 2024-09-13T14:24:51.544 INFO:tasks.rados.rados.0.smithi151.stdout:972: read oid 34 snap -1 2024-09-13T14:24:51.544 INFO:tasks.rados.rados.0.smithi151.stdout:972: expect deleted 2024-09-13T14:24:51.544 INFO:tasks.rados.rados.0.smithi151.stdout:973: write oid 36 current snap is 123 2024-09-13T14:24:51.544 INFO:tasks.rados.rados.0.smithi151.stdout:970: finishing write tid 2 to smithi15186934-7 2024-09-13T14:24:51.544 INFO:tasks.rados.rados.0.smithi151.stdout:970: finishing write tid 3 to smithi15186934-7 2024-09-13T14:24:51.545 INFO:tasks.rados.rados.0.smithi151.stdout:970: finishing write tid 4 to smithi15186934-7 2024-09-13T14:24:51.545 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 681 (ObjNum 396 snap 123 seq_num 396) dirty exists 2024-09-13T14:24:51.545 INFO:tasks.rados.rados.0.smithi151.stdout:970: left oid 7 (ObjNum 396 snap 123 seq_num 396) 2024-09-13T14:24:51.545 INFO:tasks.rados.rados.0.smithi151.stdout:971: finishing write tid 1 to smithi15186934-45 2024-09-13T14:24:51.545 INFO:tasks.rados.rados.0.smithi151.stdout:973: seq_num 398 ranges {689186=799458,1923598=752130} 2024-09-13T14:24:51.551 INFO:tasks.rados.rados.0.smithi151.stdout:973: writing smithi15186934-36 from 689186 to 1488644 tid 1 2024-09-13T14:24:51.559 INFO:tasks.rados.rados.0.smithi151.stdout:973: writing smithi15186934-36 from 1923598 to 2675728 tid 2 2024-09-13T14:24:51.559 INFO:tasks.rados.rados.0.smithi151.stdout:970: done (3 left) 2024-09-13T14:24:51.559 INFO:tasks.rados.rados.0.smithi151.stdout:974: read oid 11 snap -1 2024-09-13T14:24:51.559 INFO:tasks.rados.rados.0.smithi151.stdout:974: expect (ObjNum 394 snap 121 seq_num 394) 2024-09-13T14:24:51.559 INFO:tasks.rados.rados.0.smithi151.stdout:975: read oid 4 snap -1 2024-09-13T14:24:51.559 INFO:tasks.rados.rados.0.smithi151.stdout:975: expect (ObjNum 362 snap 110 seq_num 362) 2024-09-13T14:24:51.559 INFO:tasks.rados.rados.0.smithi151.stdout:976: rollback oid 8 current snap is 123 2024-09-13T14:24:51.559 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 8 to 101 2024-09-13T14:24:51.559 INFO:tasks.rados.rados.0.smithi151.stdout:971: finishing write tid 2 to smithi15186934-45 2024-09-13T14:24:51.559 INFO:tasks.rados.rados.0.smithi151.stdout:971: finishing write tid 3 to smithi15186934-45 2024-09-13T14:24:51.559 INFO:tasks.rados.rados.0.smithi151.stdout:971: finishing write tid 4 to smithi15186934-45 2024-09-13T14:24:51.559 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 628 (ObjNum 397 snap 123 seq_num 397) dirty exists 2024-09-13T14:24:51.560 INFO:tasks.rados.rados.0.smithi151.stdout:971: left oid 45 (ObjNum 397 snap 123 seq_num 397) 2024-09-13T14:24:51.560 INFO:tasks.rados.rados.0.smithi151.stdout:973: finishing write tid 1 to smithi15186934-36 2024-09-13T14:24:51.560 INFO:tasks.rados.rados.0.smithi151.stdout:971: done (5 left) 2024-09-13T14:24:51.560 INFO:tasks.rados.rados.0.smithi151.stdout:972: done (4 left) 2024-09-13T14:24:51.560 INFO:tasks.rados.rados.0.smithi151.stdout:977: delete oid 40 current snap is 123 2024-09-13T14:24:51.560 INFO:tasks.rados.rados.0.smithi151.stdout:976: finishing rollback tid 1 to smithi15186934-8 2024-09-13T14:24:51.560 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-13T14:24:51.562 INFO:tasks.rados.rados.0.smithi151.stdout:976: done (4 left) 2024-09-13T14:24:51.562 INFO:tasks.rados.rados.0.smithi151.stdout:977: done (3 left) 2024-09-13T14:24:51.562 INFO:tasks.rados.rados.0.smithi151.stdout:978: read oid 28 snap -1 2024-09-13T14:24:51.562 INFO:tasks.rados.rados.0.smithi151.stdout:978: expect deleted 2024-09-13T14:24:51.562 INFO:tasks.rados.rados.0.smithi151.stdout:979: rollback oid 12 current snap is 123 2024-09-13T14:24:51.562 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 12 to 101 2024-09-13T14:24:51.562 INFO:tasks.rados.rados.0.smithi151.stdout:980: snap_remove snap 113 2024-09-13T14:24:51.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:51 smithi184 ceph-mon[33394]: osdmap e309: 8 total, 8 up, 8 in 2024-09-13T14:24:51.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:51 smithi151 ceph-mon[104450]: osdmap e309: 8 total, 8 up, 8 in 2024-09-13T14:24:51.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:51 smithi151 ceph-mon[41387]: osdmap e309: 8 total, 8 up, 8 in 2024-09-13T14:24:52.500 INFO:tasks.rados.rados.0.smithi151.stdout:980: done (5 left) 2024-09-13T14:24:52.500 INFO:tasks.rados.rados.0.smithi151.stdout:981: snap_remove snap 121 2024-09-13T14:24:52.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:52 smithi184 ceph-mon[33394]: pgmap v78: 89 pgs: 89 active+clean; 177 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 MiB/s wr, 7 op/s 2024-09-13T14:24:52.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:52 smithi184 ceph-mon[33394]: osdmap e310: 8 total, 8 up, 8 in 2024-09-13T14:24:52.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:52 smithi151 ceph-mon[104450]: pgmap v78: 89 pgs: 89 active+clean; 177 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 MiB/s wr, 7 op/s 2024-09-13T14:24:52.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:52 smithi151 ceph-mon[104450]: osdmap e310: 8 total, 8 up, 8 in 2024-09-13T14:24:52.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:52 smithi151 ceph-mon[41387]: pgmap v78: 89 pgs: 89 active+clean; 177 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 MiB/s wr, 7 op/s 2024-09-13T14:24:52.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:52 smithi151 ceph-mon[41387]: osdmap e310: 8 total, 8 up, 8 in 2024-09-13T14:24:53.505 INFO:tasks.rados.rados.0.smithi151.stdout:981: done (5 left) 2024-09-13T14:24:53.506 INFO:tasks.rados.rados.0.smithi151.stdout:982: write oid 6 current snap is 123 2024-09-13T14:24:53.506 INFO:tasks.rados.rados.0.smithi151.stdout:982: seq_num 400 ranges {722800=549956,2040562=682238} 2024-09-13T14:24:53.512 INFO:tasks.rados.rados.0.smithi151.stdout:982: writing smithi15186934-6 from 722800 to 1272756 tid 1 2024-09-13T14:24:53.521 INFO:tasks.rados.rados.0.smithi151.stdout:982: writing smithi15186934-6 from 2040562 to 2722800 tid 2 2024-09-13T14:24:53.521 INFO:tasks.rados.rados.0.smithi151.stdout:983: snap_remove snap 122 2024-09-13T14:24:53.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:53 smithi184 ceph-mon[33394]: osdmap e311: 8 total, 8 up, 8 in 2024-09-13T14:24:53.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:53 smithi151 ceph-mon[104450]: osdmap e311: 8 total, 8 up, 8 in 2024-09-13T14:24:53.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:53 smithi151 ceph-mon[41387]: osdmap e311: 8 total, 8 up, 8 in 2024-09-13T14:24:54.455 INFO:tasks.rados.rados.0.smithi151.stdout:983: done (6 left) 2024-09-13T14:24:54.455 INFO:tasks.rados.rados.0.smithi151.stdout:984: snap_remove snap 105 2024-09-13T14:24:54.502 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:24:54 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:24:54.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:24:54.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:54 smithi184 ceph-mon[33394]: pgmap v81: 89 pgs: 89 active+clean; 191 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.2 MiB/s wr, 13 op/s 2024-09-13T14:24:54.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:54 smithi184 ceph-mon[33394]: osdmap e312: 8 total, 8 up, 8 in 2024-09-13T14:24:54.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:54 smithi184 ceph-mon[33394]: osdmap e313: 8 total, 8 up, 8 in 2024-09-13T14:24:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:54 smithi151 ceph-mon[104450]: pgmap v81: 89 pgs: 89 active+clean; 191 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.2 MiB/s wr, 13 op/s 2024-09-13T14:24:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:54 smithi151 ceph-mon[104450]: osdmap e312: 8 total, 8 up, 8 in 2024-09-13T14:24:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:54 smithi151 ceph-mon[104450]: osdmap e313: 8 total, 8 up, 8 in 2024-09-13T14:24:54.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:24:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:24:54] "GET /metrics HTTP/1.1" 200 36403 "" "Prometheus/2.51.0" 2024-09-13T14:24:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:54 smithi151 ceph-mon[41387]: pgmap v81: 89 pgs: 89 active+clean; 191 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.2 MiB/s wr, 13 op/s 2024-09-13T14:24:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:54 smithi151 ceph-mon[41387]: osdmap e312: 8 total, 8 up, 8 in 2024-09-13T14:24:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:54 smithi151 ceph-mon[41387]: osdmap e313: 8 total, 8 up, 8 in 2024-09-13T14:24:55.455 INFO:tasks.rados.rados.0.smithi151.stdout:984: done (6 left) 2024-09-13T14:24:55.455 INFO:tasks.rados.rados.0.smithi151.stdout:985: read oid 32 snap -1 2024-09-13T14:24:55.455 INFO:tasks.rados.rados.0.smithi151.stdout:985: expect (ObjNum 360 snap 110 seq_num 360) 2024-09-13T14:24:55.455 INFO:tasks.rados.rados.0.smithi151.stdout:986: rollback oid 14 current snap is 123 2024-09-13T14:24:55.455 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 14 to 101 2024-09-13T14:24:55.455 INFO:tasks.rados.rados.0.smithi151.stdout:973: finishing write tid 2 to smithi15186934-36 2024-09-13T14:24:55.455 INFO:tasks.rados.rados.0.smithi151.stdout:974: expect (ObjNum 394 snap 121 seq_num 394) 2024-09-13T14:24:55.544 INFO:tasks.rados.rados.0.smithi151.stdout:975: expect (ObjNum 362 snap 110 seq_num 362) 2024-09-13T14:24:55.639 INFO:tasks.rados.rados.0.smithi151.stdout:973: finishing write tid 3 to smithi15186934-36 2024-09-13T14:24:55.639 INFO:tasks.rados.rados.0.smithi151.stdout:973: finishing write tid 4 to smithi15186934-36 2024-09-13T14:24:55.639 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 36 v 241 (ObjNum 398 snap 123 seq_num 398) dirty exists 2024-09-13T14:24:55.639 INFO:tasks.rados.rados.0.smithi151.stdout:973: left oid 36 (ObjNum 398 snap 123 seq_num 398) 2024-09-13T14:24:55.639 INFO:tasks.rados.rados.0.smithi151.stdout:979: finishing rollback tid 0 to smithi15186934-12 2024-09-13T14:24:55.639 INFO:tasks.rados.rados.0.smithi151.stdout:979: finishing rollback tid 1 to smithi15186934-12 2024-09-13T14:24:55.639 INFO:tasks.rados.rados.0.smithi151.stdout:979: finishing rollback tid 2 to smithi15186934-12 2024-09-13T14:24:55.639 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 12 v 245 (ObjNum 258 snap 81 seq_num 258) dirty exists 2024-09-13T14:24:55.640 INFO:tasks.rados.rados.0.smithi151.stdout:973: done (7 left) 2024-09-13T14:24:55.640 INFO:tasks.rados.rados.0.smithi151.stdout:974: done (6 left) 2024-09-13T14:24:55.640 INFO:tasks.rados.rados.0.smithi151.stdout:975: done (5 left) 2024-09-13T14:24:55.640 INFO:tasks.rados.rados.0.smithi151.stdout:978: done (4 left) 2024-09-13T14:24:55.640 INFO:tasks.rados.rados.0.smithi151.stdout:979: done (3 left) 2024-09-13T14:24:55.640 INFO:tasks.rados.rados.0.smithi151.stdout:987: read oid 10 snap 107 2024-09-13T14:24:55.640 INFO:tasks.rados.rados.0.smithi151.stdout:987: expect (ObjNum 352 snap 107 seq_num 352) 2024-09-13T14:24:55.640 INFO:tasks.rados.rados.0.smithi151.stdout:988: read oid 7 snap 101 2024-09-13T14:24:55.640 INFO:tasks.rados.rados.0.smithi151.stdout:988: expect deleted 2024-09-13T14:24:55.640 INFO:tasks.rados.rados.0.smithi151.stdout:989: write oid 30 current snap is 123 2024-09-13T14:24:55.640 INFO:tasks.rados.rados.0.smithi151.stdout:982: finishing write tid 1 to smithi15186934-6 2024-09-13T14:24:55.641 INFO:tasks.rados.rados.0.smithi151.stdout:982: finishing write tid 2 to smithi15186934-6 2024-09-13T14:24:55.641 INFO:tasks.rados.rados.0.smithi151.stdout:982: finishing write tid 3 to smithi15186934-6 2024-09-13T14:24:55.641 INFO:tasks.rados.rados.0.smithi151.stdout:989: seq_num 401 ranges {739607=625205,1982691=499457} 2024-09-13T14:24:55.646 INFO:tasks.rados.rados.0.smithi151.stdout:989: writing smithi15186934-30 from 739607 to 1364812 tid 1 2024-09-13T14:24:55.654 INFO:tasks.rados.rados.0.smithi151.stdout:989: writing smithi15186934-30 from 1982691 to 2482148 tid 2 2024-09-13T14:24:55.654 INFO:tasks.rados.rados.0.smithi151.stdout:990: read oid 42 snap -1 2024-09-13T14:24:55.654 INFO:tasks.rados.rados.0.smithi151.stdout:990: expect (ObjNum 385 snap 120 seq_num 385) 2024-09-13T14:24:55.654 INFO:tasks.rados.rados.0.smithi151.stdout:991: read oid 25 snap -1 2024-09-13T14:24:55.654 INFO:tasks.rados.rados.0.smithi151.stdout:991: expect (ObjNum 361 snap 110 seq_num 361) 2024-09-13T14:24:55.654 INFO:tasks.rados.rados.0.smithi151.stdout:992: rollback oid 35 current snap is 123 2024-09-13T14:24:55.654 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 35 to 107 2024-09-13T14:24:55.654 INFO:tasks.rados.rados.0.smithi151.stdout:982: finishing write tid 4 to smithi15186934-6 2024-09-13T14:24:55.654 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 6 v 241 (ObjNum 400 snap 123 seq_num 400) dirty exists 2024-09-13T14:24:55.654 INFO:tasks.rados.rados.0.smithi151.stdout:982: left oid 6 (ObjNum 400 snap 123 seq_num 400) 2024-09-13T14:24:55.655 INFO:tasks.rados.rados.0.smithi151.stdout:982: done (8 left) 2024-09-13T14:24:55.655 INFO:tasks.rados.rados.0.smithi151.stdout:993: write oid 11 current snap is 123 2024-09-13T14:24:55.655 INFO:tasks.rados.rados.0.smithi151.stdout:993: seq_num 402 ranges {756414=700454,2041173=715241} 2024-09-13T14:24:55.661 INFO:tasks.rados.rados.0.smithi151.stdout:993: writing smithi15186934-11 from 756414 to 1456868 tid 1 2024-09-13T14:24:55.668 INFO:tasks.rados.rados.0.smithi151.stdout:993: writing smithi15186934-11 from 2041173 to 2756414 tid 2 2024-09-13T14:24:55.669 INFO:tasks.rados.rados.0.smithi151.stdout:994: write oid 17 current snap is 123 2024-09-13T14:24:55.669 INFO:tasks.rados.rados.0.smithi151.stdout:994: seq_num 403 ranges {773221=492056,1816008=503126} 2024-09-13T14:24:55.675 INFO:tasks.rados.rados.0.smithi151.stdout:994: writing smithi15186934-17 from 773221 to 1265277 tid 1 2024-09-13T14:24:55.681 INFO:tasks.rados.rados.0.smithi151.stdout:994: writing smithi15186934-17 from 1816008 to 2319134 tid 2 2024-09-13T14:24:55.681 INFO:tasks.rados.rados.0.smithi151.stdout:995: read oid 31 snap -1 2024-09-13T14:24:55.681 INFO:tasks.rados.rados.0.smithi151.stdout:995: expect (ObjNum 395 snap 123 seq_num 395) 2024-09-13T14:24:55.681 INFO:tasks.rados.rados.0.smithi151.stdout:985: expect (ObjNum 360 snap 110 seq_num 360) 2024-09-13T14:24:55.780 INFO:tasks.rados.rados.0.smithi151.stdout:986: finishing rollback tid 0 to smithi15186934-14 2024-09-13T14:24:55.780 INFO:tasks.rados.rados.0.smithi151.stdout:986: finishing rollback tid 1 to smithi15186934-14 2024-09-13T14:24:55.780 INFO:tasks.rados.rados.0.smithi151.stdout:986: finishing rollback tid 2 to smithi15186934-14 2024-09-13T14:24:55.780 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 437 (ObjNum 321 snap 98 seq_num 321) dirty exists 2024-09-13T14:24:55.780 INFO:tasks.rados.rados.0.smithi151.stdout:985: done (10 left) 2024-09-13T14:24:55.780 INFO:tasks.rados.rados.0.smithi151.stdout:986: done (9 left) 2024-09-13T14:24:55.780 INFO:tasks.rados.rados.0.smithi151.stdout:996: write oid 4 current snap is 123 2024-09-13T14:24:55.781 INFO:tasks.rados.rados.0.smithi151.stdout:987: expect (ObjNum 352 snap 107 seq_num 352) 2024-09-13T14:24:55.907 INFO:tasks.rados.rados.0.smithi151.stdout:989: finishing write tid 1 to smithi15186934-30 2024-09-13T14:24:55.907 INFO:tasks.rados.rados.0.smithi151.stdout:989: finishing write tid 2 to smithi15186934-30 2024-09-13T14:24:55.907 INFO:tasks.rados.rados.0.smithi151.stdout:989: finishing write tid 3 to smithi15186934-30 2024-09-13T14:24:55.907 INFO:tasks.rados.rados.0.smithi151.stdout:989: finishing write tid 4 to smithi15186934-30 2024-09-13T14:24:55.907 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 30 v 471 (ObjNum 401 snap 123 seq_num 401) dirty exists 2024-09-13T14:24:55.907 INFO:tasks.rados.rados.0.smithi151.stdout:989: left oid 30 (ObjNum 401 snap 123 seq_num 401) 2024-09-13T14:24:55.907 INFO:tasks.rados.rados.0.smithi151.stdout:992: finishing rollback tid 0 to smithi15186934-35 2024-09-13T14:24:55.908 INFO:tasks.rados.rados.0.smithi151.stdout:990: expect (ObjNum 385 snap 120 seq_num 385) 2024-09-13T14:24:56.091 INFO:tasks.rados.rados.0.smithi151.stdout:992: finishing rollback tid 1 to smithi15186934-35 2024-09-13T14:24:56.092 INFO:tasks.rados.rados.0.smithi151.stdout:992: finishing rollback tid 2 to smithi15186934-35 2024-09-13T14:24:56.092 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 35 v 700 (ObjNum 341 snap 105 seq_num 341) dirty exists 2024-09-13T14:24:56.092 INFO:tasks.rados.rados.0.smithi151.stdout:991: expect (ObjNum 361 snap 110 seq_num 361) 2024-09-13T14:24:56.103 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:24:55 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:24:55.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:24:56.103 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:24:55 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:24:55.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:24:56.205 INFO:tasks.rados.rados.0.smithi151.stdout:993: finishing write tid 1 to smithi15186934-11 2024-09-13T14:24:56.205 INFO:tasks.rados.rados.0.smithi151.stdout:993: finishing write tid 2 to smithi15186934-11 2024-09-13T14:24:56.205 INFO:tasks.rados.rados.0.smithi151.stdout:993: finishing write tid 3 to smithi15186934-11 2024-09-13T14:24:56.205 INFO:tasks.rados.rados.0.smithi151.stdout:993: finishing write tid 4 to smithi15186934-11 2024-09-13T14:24:56.205 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 11 v 282 (ObjNum 402 snap 123 seq_num 402) dirty exists 2024-09-13T14:24:56.205 INFO:tasks.rados.rados.0.smithi151.stdout:993: left oid 11 (ObjNum 402 snap 123 seq_num 402) 2024-09-13T14:24:56.206 INFO:tasks.rados.rados.0.smithi151.stdout:994: finishing write tid 1 to smithi15186934-17 2024-09-13T14:24:56.206 INFO:tasks.rados.rados.0.smithi151.stdout:995: expect (ObjNum 395 snap 123 seq_num 395) 2024-09-13T14:24:56.291 INFO:tasks.rados.rados.0.smithi151.stdout:994: finishing write tid 2 to smithi15186934-17 2024-09-13T14:24:56.291 INFO:tasks.rados.rados.0.smithi151.stdout:994: finishing write tid 3 to smithi15186934-17 2024-09-13T14:24:56.291 INFO:tasks.rados.rados.0.smithi151.stdout:994: finishing write tid 4 to smithi15186934-17 2024-09-13T14:24:56.291 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 703 (ObjNum 403 snap 123 seq_num 403) dirty exists 2024-09-13T14:24:56.291 INFO:tasks.rados.rados.0.smithi151.stdout:994: left oid 17 (ObjNum 403 snap 123 seq_num 403) 2024-09-13T14:24:56.292 INFO:tasks.rados.rados.0.smithi151.stdout:996: seq_num 404 ranges {790028=567305,1874490=763137} 2024-09-13T14:24:56.298 INFO:tasks.rados.rados.0.smithi151.stdout:996: writing smithi15186934-4 from 790028 to 1357333 tid 1 2024-09-13T14:24:56.306 INFO:tasks.rados.rados.0.smithi151.stdout:996: writing smithi15186934-4 from 1874490 to 2637627 tid 2 2024-09-13T14:24:56.306 INFO:tasks.rados.rados.0.smithi151.stdout:987: done (9 left) 2024-09-13T14:24:56.306 INFO:tasks.rados.rados.0.smithi151.stdout:988: done (8 left) 2024-09-13T14:24:56.306 INFO:tasks.rados.rados.0.smithi151.stdout:989: done (7 left) 2024-09-13T14:24:56.306 INFO:tasks.rados.rados.0.smithi151.stdout:990: done (6 left) 2024-09-13T14:24:56.307 INFO:tasks.rados.rados.0.smithi151.stdout:991: done (5 left) 2024-09-13T14:24:56.307 INFO:tasks.rados.rados.0.smithi151.stdout:992: done (4 left) 2024-09-13T14:24:56.307 INFO:tasks.rados.rados.0.smithi151.stdout:993: done (3 left) 2024-09-13T14:24:56.307 INFO:tasks.rados.rados.0.smithi151.stdout:994: done (2 left) 2024-09-13T14:24:56.307 INFO:tasks.rados.rados.0.smithi151.stdout:995: done (1 left) 2024-09-13T14:24:56.307 INFO:tasks.rados.rados.0.smithi151.stdout:997: read oid 26 snap 101 2024-09-13T14:24:56.308 INFO:tasks.rados.rados.0.smithi151.stdout:997: expect (ObjNum 287 snap 89 seq_num 287) 2024-09-13T14:24:56.308 INFO:tasks.rados.rados.0.smithi151.stdout:998: snap_create 2024-09-13T14:24:56.308 INFO:tasks.rados.rados.0.smithi151.stdout:996: finishing write tid 1 to smithi15186934-4 2024-09-13T14:24:56.309 INFO:tasks.rados.rados.0.smithi151.stdout:996: finishing write tid 2 to smithi15186934-4 2024-09-13T14:24:56.309 INFO:tasks.rados.rados.0.smithi151.stdout:996: finishing write tid 3 to smithi15186934-4 2024-09-13T14:24:56.309 INFO:tasks.rados.rados.0.smithi151.stdout:996: finishing write tid 4 to smithi15186934-4 2024-09-13T14:24:56.309 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 647 (ObjNum 404 snap 123 seq_num 404) dirty exists 2024-09-13T14:24:56.309 INFO:tasks.rados.rados.0.smithi151.stdout:996: left oid 4 (ObjNum 404 snap 123 seq_num 404) 2024-09-13T14:24:56.310 INFO:tasks.rados.rados.0.smithi151.stdout:997: expect (ObjNum 287 snap 89 seq_num 287) 2024-09-13T14:24:56.510 INFO:tasks.rados.rados.0.smithi151.stdout:996: done (2 left) 2024-09-13T14:24:56.510 INFO:tasks.rados.rados.0.smithi151.stdout:997: done (1 left) 2024-09-13T14:24:56.511 INFO:tasks.rados.rados.0.smithi151.stdout:998: done (0 left) 2024-09-13T14:24:56.511 INFO:tasks.rados.rados.0.smithi151.stdout:999: snap_create 2024-09-13T14:24:56.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:56 smithi184 ceph-mon[33394]: pgmap v84: 89 pgs: 89 active+clean; 191 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.2 MiB/s wr, 13 op/s 2024-09-13T14:24:56.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:56 smithi184 ceph-mon[33394]: osdmap e314: 8 total, 8 up, 8 in 2024-09-13T14:24:56.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:56 smithi151 ceph-mon[104450]: pgmap v84: 89 pgs: 89 active+clean; 191 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.2 MiB/s wr, 13 op/s 2024-09-13T14:24:56.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:56 smithi151 ceph-mon[104450]: osdmap e314: 8 total, 8 up, 8 in 2024-09-13T14:24:56.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:56 smithi151 ceph-mon[41387]: pgmap v84: 89 pgs: 89 active+clean; 191 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 2.2 MiB/s wr, 13 op/s 2024-09-13T14:24:56.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:56 smithi151 ceph-mon[41387]: osdmap e314: 8 total, 8 up, 8 in 2024-09-13T14:24:57.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:24:56 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:24:56.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi184\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:24:57.517 INFO:tasks.rados.rados.0.smithi151.stdout:999: done (0 left) 2024-09-13T14:24:57.517 INFO:tasks.rados.rados.0.smithi151.stdout:1000: rollback oid 43 current snap is 125 2024-09-13T14:24:57.517 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 43 to 124 2024-09-13T14:24:57.518 INFO:tasks.rados.rados.0.smithi151.stdout:1001: rollback oid 11 current snap is 125 2024-09-13T14:24:57.518 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 11 to 123 2024-09-13T14:24:57.518 INFO:tasks.rados.rados.0.smithi151.stdout:1002: snap_create 2024-09-13T14:24:57.522 INFO:tasks.rados.rados.0.smithi151.stdout:1001: finishing rollback tid 0 to smithi15186934-11 2024-09-13T14:24:57.522 INFO:tasks.rados.rados.0.smithi151.stdout:1000: finishing rollback tid 0 to smithi15186934-43 2024-09-13T14:24:57.522 INFO:tasks.rados.rados.0.smithi151.stdout:1001: finishing rollback tid 1 to smithi15186934-11 2024-09-13T14:24:57.522 INFO:tasks.rados.rados.0.smithi151.stdout:1000: finishing rollback tid 1 to smithi15186934-43 2024-09-13T14:24:57.523 INFO:tasks.rados.rados.0.smithi151.stdout:1001: finishing rollback tid 2 to smithi15186934-11 2024-09-13T14:24:57.523 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 11 v 286 (ObjNum 402 snap 123 seq_num 402) dirty exists 2024-09-13T14:24:57.523 INFO:tasks.rados.rados.0.smithi151.stdout:1000: finishing rollback tid 2 to smithi15186934-43 2024-09-13T14:24:57.523 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 43 v 707 (ObjNum 379 snap 119 seq_num 379) dirty exists 2024-09-13T14:24:57.773 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:57 smithi151 ceph-mon[41387]: osdmap e315: 8 total, 8 up, 8 in 2024-09-13T14:24:57.774 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:57 smithi151 ceph-mon[104450]: osdmap e315: 8 total, 8 up, 8 in 2024-09-13T14:24:57.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:57 smithi184 ceph-mon[33394]: osdmap e315: 8 total, 8 up, 8 in 2024-09-13T14:24:58.516 INFO:tasks.rados.rados.0.smithi151.stdout:1000: done (2 left) 2024-09-13T14:24:58.517 INFO:tasks.rados.rados.0.smithi151.stdout:1001: done (1 left) 2024-09-13T14:24:58.517 INFO:tasks.rados.rados.0.smithi151.stdout:1002: done (0 left) 2024-09-13T14:24:58.517 INFO:tasks.rados.rados.0.smithi151.stdout:1003: delete oid 21 current snap is 126 2024-09-13T14:24:58.521 INFO:tasks.rados.rados.0.smithi151.stdout:1003: done (0 left) 2024-09-13T14:24:58.521 INFO:tasks.rados.rados.0.smithi151.stdout:1004: snap_remove snap 125 2024-09-13T14:24:58.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:58 smithi184 ceph-mon[33394]: pgmap v87: 89 pgs: 89 active+clean; 190 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 568 KiB/s wr, 4 op/s 2024-09-13T14:24:58.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:58 smithi184 ceph-mon[33394]: osdmap e316: 8 total, 8 up, 8 in 2024-09-13T14:24:58.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:58 smithi151 ceph-mon[104450]: pgmap v87: 89 pgs: 89 active+clean; 190 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 568 KiB/s wr, 4 op/s 2024-09-13T14:24:58.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:58 smithi151 ceph-mon[104450]: osdmap e316: 8 total, 8 up, 8 in 2024-09-13T14:24:58.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:58 smithi151 ceph-mon[41387]: pgmap v87: 89 pgs: 89 active+clean; 190 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 568 KiB/s wr, 4 op/s 2024-09-13T14:24:58.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:58 smithi151 ceph-mon[41387]: osdmap e316: 8 total, 8 up, 8 in 2024-09-13T14:24:59.527 INFO:tasks.rados.rados.0.smithi151.stdout:1004: done (0 left) 2024-09-13T14:24:59.527 INFO:tasks.rados.rados.0.smithi151.stdout:1005: read oid 31 snap -1 2024-09-13T14:24:59.527 INFO:tasks.rados.rados.0.smithi151.stdout:1005: expect (ObjNum 395 snap 123 seq_num 395) 2024-09-13T14:24:59.527 INFO:tasks.rados.rados.0.smithi151.stdout:1006: snap_create 2024-09-13T14:24:59.531 INFO:tasks.rados.rados.0.smithi151.stdout:1005: expect (ObjNum 395 snap 123 seq_num 395) 2024-09-13T14:24:59.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:24:59 smithi184 ceph-mon[33394]: osdmap e317: 8 total, 8 up, 8 in 2024-09-13T14:24:59.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:24:59 smithi151 ceph-mon[104450]: osdmap e317: 8 total, 8 up, 8 in 2024-09-13T14:24:59.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:24:59 smithi151 ceph-mon[41387]: osdmap e317: 8 total, 8 up, 8 in 2024-09-13T14:25:00.528 INFO:tasks.rados.rados.0.smithi151.stdout:1005: done (1 left) 2024-09-13T14:25:00.528 INFO:tasks.rados.rados.0.smithi151.stdout:1006: done (0 left) 2024-09-13T14:25:00.528 INFO:tasks.rados.rados.0.smithi151.stdout:1007: read oid 29 snap 124 2024-09-13T14:25:00.528 INFO:tasks.rados.rados.0.smithi151.stdout:1007: expect deleted 2024-09-13T14:25:00.529 INFO:tasks.rados.rados.0.smithi151.stdout:1008: delete oid 9 current snap is 127 2024-09-13T14:25:00.529 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:25:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:25:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:25:00.535 INFO:tasks.rados.rados.0.smithi151.stdout:1007: done (1 left) 2024-09-13T14:25:00.535 INFO:tasks.rados.rados.0.smithi151.stdout:1008: done (0 left) 2024-09-13T14:25:00.535 INFO:tasks.rados.rados.0.smithi151.stdout:1009: read oid 39 snap 124 2024-09-13T14:25:00.535 INFO:tasks.rados.rados.0.smithi151.stdout:1009: expect (ObjNum 390 snap 121 seq_num 390) 2024-09-13T14:25:00.535 INFO:tasks.rados.rados.0.smithi151.stdout:1010: read oid 16 snap -1 2024-09-13T14:25:00.535 INFO:tasks.rados.rados.0.smithi151.stdout:1010: expect (ObjNum 301 snap 93 seq_num 301) 2024-09-13T14:25:00.536 INFO:tasks.rados.rados.0.smithi151.stdout:1011: rollback oid 14 current snap is 127 2024-09-13T14:25:00.536 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 14 to 124 2024-09-13T14:25:00.536 INFO:tasks.rados.rados.0.smithi151.stdout:1012: delete oid 48 current snap is 127 2024-09-13T14:25:00.537 INFO:tasks.rados.rados.0.smithi151.stdout:1011: finishing rollback tid 0 to smithi15186934-14 2024-09-13T14:25:00.538 INFO:tasks.rados.rados.0.smithi151.stdout:1011: finishing rollback tid 1 to smithi15186934-14 2024-09-13T14:25:00.538 INFO:tasks.rados.rados.0.smithi151.stdout:1011: finishing rollback tid 2 to smithi15186934-14 2024-09-13T14:25:00.538 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 444 (ObjNum 321 snap 98 seq_num 321) dirty exists 2024-09-13T14:25:00.539 INFO:tasks.rados.rados.0.smithi151.stdout:1011: done (3 left) 2024-09-13T14:25:00.540 INFO:tasks.rados.rados.0.smithi151.stdout:1012: done (2 left) 2024-09-13T14:25:00.540 INFO:tasks.rados.rados.0.smithi151.stdout:1013: read oid 46 snap -1 2024-09-13T14:25:00.540 INFO:tasks.rados.rados.0.smithi151.stdout:1013: expect (ObjNum 261 snap 82 seq_num 261) 2024-09-13T14:25:00.540 INFO:tasks.rados.rados.0.smithi151.stdout:1014: write oid 38 current snap is 127 2024-09-13T14:25:00.540 INFO:tasks.rados.rados.0.smithi151.stdout:1014: seq_num 408 ranges {457256=468301,1592065=770475,2854581=2675} 2024-09-13T14:25:00.545 INFO:tasks.rados.rados.0.smithi151.stdout:1014: writing smithi15186934-38 from 457256 to 925557 tid 1 2024-09-13T14:25:00.553 INFO:tasks.rados.rados.0.smithi151.stdout:1014: writing smithi15186934-38 from 1592065 to 2362540 tid 2 2024-09-13T14:25:00.556 INFO:tasks.rados.rados.0.smithi151.stdout:1014: writing smithi15186934-38 from 2854581 to 2857256 tid 3 2024-09-13T14:25:00.556 INFO:tasks.rados.rados.0.smithi151.stdout:1015: rollback oid 29 current snap is 127 2024-09-13T14:25:00.556 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 29 to 101 2024-09-13T14:25:00.556 INFO:tasks.rados.rados.0.smithi151.stdout:1010: expect (ObjNum 301 snap 93 seq_num 301) 2024-09-13T14:25:00.657 INFO:tasks.rados.rados.0.smithi151.stdout:1013: expect (ObjNum 261 snap 82 seq_num 261) 2024-09-13T14:25:00.759 INFO:tasks.rados.rados.0.smithi151.stdout:1009: expect (ObjNum 390 snap 121 seq_num 390) 2024-09-13T14:25:00.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:00 smithi184 ceph-mon[33394]: pgmap v90: 89 pgs: 89 active+clean; 184 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 1.5 MiB/s wr, 20 op/s 2024-09-13T14:25:00.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:00 smithi184 ceph-mon[33394]: osdmap e318: 8 total, 8 up, 8 in 2024-09-13T14:25:00.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:00 smithi151 ceph-mon[104450]: pgmap v90: 89 pgs: 89 active+clean; 184 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 1.5 MiB/s wr, 20 op/s 2024-09-13T14:25:00.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:00 smithi151 ceph-mon[104450]: osdmap e318: 8 total, 8 up, 8 in 2024-09-13T14:25:00.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:00 smithi151 ceph-mon[41387]: pgmap v90: 89 pgs: 89 active+clean; 184 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 1.5 MiB/s wr, 20 op/s 2024-09-13T14:25:00.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:00 smithi151 ceph-mon[41387]: osdmap e318: 8 total, 8 up, 8 in 2024-09-13T14:25:00.874 INFO:tasks.rados.rados.0.smithi151.stdout:1014: finishing write tid 1 to smithi15186934-38 2024-09-13T14:25:00.874 INFO:tasks.rados.rados.0.smithi151.stdout:1009: done (4 left) 2024-09-13T14:25:00.874 INFO:tasks.rados.rados.0.smithi151.stdout:1010: done (3 left) 2024-09-13T14:25:00.874 INFO:tasks.rados.rados.0.smithi151.stdout:1013: done (2 left) 2024-09-13T14:25:00.874 INFO:tasks.rados.rados.0.smithi151.stdout:1016: read oid 19 snap 123 2024-09-13T14:25:00.874 INFO:tasks.rados.rados.0.smithi151.stdout:1016: expect (ObjNum 323 snap 101 seq_num 323) 2024-09-13T14:25:00.874 INFO:tasks.rados.rados.0.smithi151.stdout:1017: delete oid 49 current snap is 127 2024-09-13T14:25:00.874 INFO:tasks.rados.rados.0.smithi151.stdout:1014: finishing write tid 2 to smithi15186934-38 2024-09-13T14:25:00.874 INFO:tasks.rados.rados.0.smithi151.stdout:1015: finishing rollback tid 1 to smithi15186934-29 2024-09-13T14:25:00.874 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 29 v 0 (ObjNum 49 snap 0 seq_num 1664845187) dirty dne 2024-09-13T14:25:00.874 INFO:tasks.rados.rados.0.smithi151.stdout:1014: finishing write tid 3 to smithi15186934-38 2024-09-13T14:25:00.874 INFO:tasks.rados.rados.0.smithi151.stdout:1014: finishing write tid 4 to smithi15186934-38 2024-09-13T14:25:00.874 INFO:tasks.rados.rados.0.smithi151.stdout:1014: finishing write tid 5 to smithi15186934-38 2024-09-13T14:25:00.875 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 313 (ObjNum 408 snap 127 seq_num 408) dirty exists 2024-09-13T14:25:00.875 INFO:tasks.rados.rados.0.smithi151.stdout:1014: left oid 38 (ObjNum 408 snap 127 seq_num 408) 2024-09-13T14:25:00.875 INFO:tasks.rados.rados.0.smithi151.stdout:1014: done (3 left) 2024-09-13T14:25:00.875 INFO:tasks.rados.rados.0.smithi151.stdout:1015: done (2 left) 2024-09-13T14:25:00.875 INFO:tasks.rados.rados.0.smithi151.stdout:1017: done (1 left) 2024-09-13T14:25:00.876 INFO:tasks.rados.rados.0.smithi151.stdout:1018: rollback oid 26 current snap is 127 2024-09-13T14:25:00.876 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 26 to 107 2024-09-13T14:25:00.876 INFO:tasks.rados.rados.0.smithi151.stdout:1019: rollback oid 28 current snap is 127 2024-09-13T14:25:00.876 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 28 to 101 2024-09-13T14:25:00.876 INFO:tasks.rados.rados.0.smithi151.stdout:1020: write oid 45 current snap is 127 2024-09-13T14:25:00.876 INFO:tasks.rados.rados.0.smithi151.stdout:1020: seq_num 410 ranges {490870=618799,1592676=774144,2793074=97796} 2024-09-13T14:25:00.882 INFO:tasks.rados.rados.0.smithi151.stdout:1020: writing smithi15186934-45 from 490870 to 1109669 tid 1 2024-09-13T14:25:00.889 INFO:tasks.rados.rados.0.smithi151.stdout:1020: writing smithi15186934-45 from 1592676 to 2366820 tid 2 2024-09-13T14:25:00.892 INFO:tasks.rados.rados.0.smithi151.stdout:1020: writing smithi15186934-45 from 2793074 to 2890870 tid 3 2024-09-13T14:25:00.892 INFO:tasks.rados.rados.0.smithi151.stdout:1021: read oid 14 snap -1 2024-09-13T14:25:00.892 INFO:tasks.rados.rados.0.smithi151.stdout:1021: expect (ObjNum 321 snap 98 seq_num 321) 2024-09-13T14:25:00.892 INFO:tasks.rados.rados.0.smithi151.stdout:1022: snap_remove snap 126 2024-09-13T14:25:01.537 INFO:tasks.rados.rados.0.smithi151.stdout:1022: done (5 left) 2024-09-13T14:25:01.537 INFO:tasks.rados.rados.0.smithi151.stdout:1023: snap_create 2024-09-13T14:25:01.537 INFO:tasks.rados.rados.0.smithi151.stdout:1019: finishing rollback tid 1 to smithi15186934-28 2024-09-13T14:25:01.537 INFO:tasks.rados.rados.0.smithi151.stdout:1019: finishing rollback tid 2 to smithi15186934-28 2024-09-13T14:25:01.537 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 28 v 551 (ObjNum 173 snap 46 seq_num 173) dirty exists 2024-09-13T14:25:01.538 INFO:tasks.rados.rados.0.smithi151.stdout:1018: finishing rollback tid 0 to smithi15186934-26 2024-09-13T14:25:01.538 INFO:tasks.rados.rados.0.smithi151.stdout:1018: finishing rollback tid 1 to smithi15186934-26 2024-09-13T14:25:01.538 INFO:tasks.rados.rados.0.smithi151.stdout:1018: finishing rollback tid 2 to smithi15186934-26 2024-09-13T14:25:01.538 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 322 (ObjNum 148 snap 34 seq_num 148) dirty exists 2024-09-13T14:25:01.538 INFO:tasks.rados.rados.0.smithi151.stdout:1016: expect (ObjNum 323 snap 101 seq_num 323) 2024-09-13T14:25:01.652 INFO:tasks.rados.rados.0.smithi151.stdout:1020: finishing write tid 1 to smithi15186934-45 2024-09-13T14:25:01.652 INFO:tasks.rados.rados.0.smithi151.stdout:1020: finishing write tid 2 to smithi15186934-45 2024-09-13T14:25:01.652 INFO:tasks.rados.rados.0.smithi151.stdout:1020: finishing write tid 3 to smithi15186934-45 2024-09-13T14:25:01.652 INFO:tasks.rados.rados.0.smithi151.stdout:1020: finishing write tid 4 to smithi15186934-45 2024-09-13T14:25:01.653 INFO:tasks.rados.rados.0.smithi151.stdout:1020: finishing write tid 5 to smithi15186934-45 2024-09-13T14:25:01.653 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 653 (ObjNum 410 snap 127 seq_num 410) dirty exists 2024-09-13T14:25:01.653 INFO:tasks.rados.rados.0.smithi151.stdout:1020: left oid 45 (ObjNum 410 snap 127 seq_num 410) 2024-09-13T14:25:01.653 INFO:tasks.rados.rados.0.smithi151.stdout:1021: expect (ObjNum 321 snap 98 seq_num 321) 2024-09-13T14:25:01.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:01 smithi184 ceph-mon[33394]: osdmap e319: 8 total, 8 up, 8 in 2024-09-13T14:25:01.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:01 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:25:01.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:01 smithi151 ceph-mon[104450]: osdmap e319: 8 total, 8 up, 8 in 2024-09-13T14:25:01.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:25:01.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:01 smithi151 ceph-mon[41387]: osdmap e319: 8 total, 8 up, 8 in 2024-09-13T14:25:01.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:01 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:25:02.539 INFO:tasks.rados.rados.0.smithi151.stdout:1016: done (5 left) 2024-09-13T14:25:02.539 INFO:tasks.rados.rados.0.smithi151.stdout:1018: done (4 left) 2024-09-13T14:25:02.539 INFO:tasks.rados.rados.0.smithi151.stdout:1019: done (3 left) 2024-09-13T14:25:02.539 INFO:tasks.rados.rados.0.smithi151.stdout:1020: done (2 left) 2024-09-13T14:25:02.539 INFO:tasks.rados.rados.0.smithi151.stdout:1021: done (1 left) 2024-09-13T14:25:02.539 INFO:tasks.rados.rados.0.smithi151.stdout:1023: done (0 left) 2024-09-13T14:25:02.539 INFO:tasks.rados.rados.0.smithi151.stdout:1024: write oid 4 current snap is 128 2024-09-13T14:25:02.540 INFO:tasks.rados.rados.0.smithi151.stdout:1024: seq_num 411 ranges {507677=410401,1367511=517802,2420497=487180} 2024-09-13T14:25:02.544 INFO:tasks.rados.rados.0.smithi151.stdout:1024: writing smithi15186934-4 from 507677 to 918078 tid 1 2024-09-13T14:25:02.549 INFO:tasks.rados.rados.0.smithi151.stdout:1024: writing smithi15186934-4 from 1367511 to 1885313 tid 2 2024-09-13T14:25:02.555 INFO:tasks.rados.rados.0.smithi151.stdout:1024: writing smithi15186934-4 from 2420497 to 2907677 tid 3 2024-09-13T14:25:02.555 INFO:tasks.rados.rados.0.smithi151.stdout:1025: delete oid 48 current snap is 128 2024-09-13T14:25:02.555 INFO:tasks.rados.rados.0.smithi151.stdout:1024: finishing write tid 1 to smithi15186934-4 2024-09-13T14:25:02.555 INFO:tasks.rados.rados.0.smithi151.stdout:1024: finishing write tid 2 to smithi15186934-4 2024-09-13T14:25:02.559 INFO:tasks.rados.rados.0.smithi151.stdout:1024: finishing write tid 3 to smithi15186934-4 2024-09-13T14:25:02.559 INFO:tasks.rados.rados.0.smithi151.stdout:1024: finishing write tid 4 to smithi15186934-4 2024-09-13T14:25:02.559 INFO:tasks.rados.rados.0.smithi151.stdout:1024: finishing write tid 5 to smithi15186934-4 2024-09-13T14:25:02.559 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 660 (ObjNum 411 snap 128 seq_num 411) dirty exists 2024-09-13T14:25:02.559 INFO:tasks.rados.rados.0.smithi151.stdout:1024: left oid 4 (ObjNum 411 snap 128 seq_num 411) 2024-09-13T14:25:02.559 INFO:tasks.rados.rados.0.smithi151.stdout:1024: done (1 left) 2024-09-13T14:25:02.559 INFO:tasks.rados.rados.0.smithi151.stdout:1025: done (0 left) 2024-09-13T14:25:02.559 INFO:tasks.rados.rados.0.smithi151.stdout:1026: read oid 14 snap 101 2024-09-13T14:25:02.559 INFO:tasks.rados.rados.0.smithi151.stdout:1026: expect (ObjNum 321 snap 98 seq_num 321) 2024-09-13T14:25:02.559 INFO:tasks.rados.rados.0.smithi151.stdout:1027: write oid 15 current snap is 128 2024-09-13T14:25:02.560 INFO:tasks.rados.rados.0.smithi151.stdout:1027: seq_num 413 ranges {541291=560899,1884475=521471} 2024-09-13T14:25:02.565 INFO:tasks.rados.rados.0.smithi151.stdout:1027: writing smithi15186934-15 from 541291 to 1102190 tid 1 2024-09-13T14:25:02.572 INFO:tasks.rados.rados.0.smithi151.stdout:1027: writing smithi15186934-15 from 1884475 to 2405946 tid 2 2024-09-13T14:25:02.572 INFO:tasks.rados.rados.0.smithi151.stdout:1028: read oid 9 snap -1 2024-09-13T14:25:02.573 INFO:tasks.rados.rados.0.smithi151.stdout:1028: expect deleted 2024-09-13T14:25:02.573 INFO:tasks.rados.rados.0.smithi151.stdout:1029: rollback oid 27 current snap is 128 2024-09-13T14:25:02.573 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 27 to 123 2024-09-13T14:25:02.573 INFO:tasks.rados.rados.0.smithi151.stdout:1026: expect (ObjNum 321 snap 98 seq_num 321) 2024-09-13T14:25:02.725 INFO:tasks.rados.rados.0.smithi151.stdout:1027: finishing write tid 1 to smithi15186934-15 2024-09-13T14:25:02.725 INFO:tasks.rados.rados.0.smithi151.stdout:1026: done (3 left) 2024-09-13T14:25:02.725 INFO:tasks.rados.rados.0.smithi151.stdout:1030: snap_create 2024-09-13T14:25:02.725 INFO:tasks.rados.rados.0.smithi151.stdout:1027: finishing write tid 2 to smithi15186934-15 2024-09-13T14:25:02.725 INFO:tasks.rados.rados.0.smithi151.stdout:1027: finishing write tid 3 to smithi15186934-15 2024-09-13T14:25:02.725 INFO:tasks.rados.rados.0.smithi151.stdout:1027: finishing write tid 4 to smithi15186934-15 2024-09-13T14:25:02.725 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 15 v 555 (ObjNum 413 snap 128 seq_num 413) dirty exists 2024-09-13T14:25:02.725 INFO:tasks.rados.rados.0.smithi151.stdout:1027: left oid 15 (ObjNum 413 snap 128 seq_num 413) 2024-09-13T14:25:02.725 INFO:tasks.rados.rados.0.smithi151.stdout:1029: finishing rollback tid 1 to smithi15186934-27 2024-09-13T14:25:02.725 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 27 v 234 (ObjNum 33 snap 0 seq_num 3064833400) dirty dne 2024-09-13T14:25:02.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:02 smithi184 ceph-mon[33394]: pgmap v93: 89 pgs: 89 active+clean; 184 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 922 KiB/s wr, 15 op/s 2024-09-13T14:25:02.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:02 smithi184 ceph-mon[33394]: osdmap e320: 8 total, 8 up, 8 in 2024-09-13T14:25:02.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:02 smithi151 ceph-mon[41387]: pgmap v93: 89 pgs: 89 active+clean; 184 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 922 KiB/s wr, 15 op/s 2024-09-13T14:25:02.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:02 smithi151 ceph-mon[41387]: osdmap e320: 8 total, 8 up, 8 in 2024-09-13T14:25:02.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:02 smithi151 ceph-mon[104450]: pgmap v93: 89 pgs: 89 active+clean; 184 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 922 KiB/s wr, 15 op/s 2024-09-13T14:25:02.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:02 smithi151 ceph-mon[104450]: osdmap e320: 8 total, 8 up, 8 in 2024-09-13T14:25:03.553 INFO:tasks.rados.rados.0.smithi151.stdout:1027: done (3 left) 2024-09-13T14:25:03.553 INFO:tasks.rados.rados.0.smithi151.stdout:1028: done (2 left) 2024-09-13T14:25:03.553 INFO:tasks.rados.rados.0.smithi151.stdout:1029: done (1 left) 2024-09-13T14:25:03.553 INFO:tasks.rados.rados.0.smithi151.stdout:1030: done (0 left) 2024-09-13T14:25:03.553 INFO:tasks.rados.rados.0.smithi151.stdout:1031: read oid 1 snap 127 2024-09-13T14:25:03.554 INFO:tasks.rados.rados.0.smithi151.stdout:1031: expect (ObjNum 381 snap 120 seq_num 381) 2024-09-13T14:25:03.554 INFO:tasks.rados.rados.0.smithi151.stdout:1032: write oid 28 current snap is 129 2024-09-13T14:25:03.554 INFO:tasks.rados.rados.0.smithi151.stdout:1032: seq_num 414 ranges {558098=636148,1826604=665129} 2024-09-13T14:25:03.559 INFO:tasks.rados.rados.0.smithi151.stdout:1032: writing smithi15186934-28 from 558098 to 1194246 tid 1 2024-09-13T14:25:03.566 INFO:tasks.rados.rados.0.smithi151.stdout:1032: writing smithi15186934-28 from 1826604 to 2491733 tid 2 2024-09-13T14:25:03.566 INFO:tasks.rados.rados.0.smithi151.stdout:1033: write oid 26 current snap is 129 2024-09-13T14:25:03.566 INFO:tasks.rados.rados.0.smithi151.stdout:1033: seq_num 415 ranges {574905=711397,1885086=525140,2930189=44716} 2024-09-13T14:25:03.573 INFO:tasks.rados.rados.0.smithi151.stdout:1033: writing smithi15186934-26 from 574905 to 1286302 tid 1 2024-09-13T14:25:03.579 INFO:tasks.rados.rados.0.smithi151.stdout:1033: writing smithi15186934-26 from 1885086 to 2410226 tid 2 2024-09-13T14:25:03.583 INFO:tasks.rados.rados.0.smithi151.stdout:1033: writing smithi15186934-26 from 2930189 to 2974905 tid 3 2024-09-13T14:25:03.583 INFO:tasks.rados.rados.0.smithi151.stdout:1034: snap_create 2024-09-13T14:25:03.583 INFO:tasks.rados.rados.0.smithi151.stdout:1031: expect (ObjNum 381 snap 120 seq_num 381) 2024-09-13T14:25:03.661 INFO:tasks.rados.rados.0.smithi151.stdout:1032: finishing write tid 1 to smithi15186934-28 2024-09-13T14:25:03.661 INFO:tasks.rados.rados.0.smithi151.stdout:1032: finishing write tid 2 to smithi15186934-28 2024-09-13T14:25:03.661 INFO:tasks.rados.rados.0.smithi151.stdout:1032: finishing write tid 3 to smithi15186934-28 2024-09-13T14:25:03.661 INFO:tasks.rados.rados.0.smithi151.stdout:1032: finishing write tid 4 to smithi15186934-28 2024-09-13T14:25:03.661 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 28 v 559 (ObjNum 414 snap 129 seq_num 414) dirty exists 2024-09-13T14:25:03.661 INFO:tasks.rados.rados.0.smithi151.stdout:1032: left oid 28 (ObjNum 414 snap 129 seq_num 414) 2024-09-13T14:25:03.661 INFO:tasks.rados.rados.0.smithi151.stdout:1033: finishing write tid 1 to smithi15186934-26 2024-09-13T14:25:03.662 INFO:tasks.rados.rados.0.smithi151.stdout:1033: finishing write tid 2 to smithi15186934-26 2024-09-13T14:25:03.662 INFO:tasks.rados.rados.0.smithi151.stdout:1033: finishing write tid 3 to smithi15186934-26 2024-09-13T14:25:03.662 INFO:tasks.rados.rados.0.smithi151.stdout:1033: finishing write tid 4 to smithi15186934-26 2024-09-13T14:25:03.662 INFO:tasks.rados.rados.0.smithi151.stdout:1033: finishing write tid 5 to smithi15186934-26 2024-09-13T14:25:03.662 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 329 (ObjNum 415 snap 129 seq_num 415) dirty exists 2024-09-13T14:25:03.662 INFO:tasks.rados.rados.0.smithi151.stdout:1033: left oid 26 (ObjNum 415 snap 129 seq_num 415) 2024-09-13T14:25:03.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:03 smithi184 ceph-mon[33394]: osdmap e321: 8 total, 8 up, 8 in 2024-09-13T14:25:03.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:03 smithi151 ceph-mon[104450]: osdmap e321: 8 total, 8 up, 8 in 2024-09-13T14:25:03.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:03 smithi151 ceph-mon[41387]: osdmap e321: 8 total, 8 up, 8 in 2024-09-13T14:25:04.456 INFO:tasks.rados.rados.0.smithi151.stdout:1031: done (3 left) 2024-09-13T14:25:04.457 INFO:tasks.rados.rados.0.smithi151.stdout:1032: done (2 left) 2024-09-13T14:25:04.457 INFO:tasks.rados.rados.0.smithi151.stdout:1033: done (1 left) 2024-09-13T14:25:04.457 INFO:tasks.rados.rados.0.smithi151.stdout:1034: done (0 left) 2024-09-13T14:25:04.457 INFO:tasks.rados.rados.0.smithi151.stdout:1035: delete oid 26 current snap is 130 2024-09-13T14:25:04.460 INFO:tasks.rados.rados.0.smithi151.stdout:1035: done (0 left) 2024-09-13T14:25:04.460 INFO:tasks.rados.rados.0.smithi151.stdout:1036: write oid 47 current snap is 130 2024-09-13T14:25:04.460 INFO:tasks.rados.rados.0.smithi151.stdout:1036: seq_num 417 ranges {608519=461895,1602050=528809,2585035=423484} 2024-09-13T14:25:04.466 INFO:tasks.rados.rados.0.smithi151.stdout:1036: writing smithi15186934-47 from 608519 to 1070414 tid 1 2024-09-13T14:25:04.472 INFO:tasks.rados.rados.0.smithi151.stdout:1036: writing smithi15186934-47 from 1602050 to 2130859 tid 2 2024-09-13T14:25:04.477 INFO:tasks.rados.rados.0.smithi151.stdout:1036: writing smithi15186934-47 from 2585035 to 3008519 tid 3 2024-09-13T14:25:04.477 INFO:tasks.rados.rados.0.smithi151.stdout:1037: write oid 20 current snap is 130 2024-09-13T14:25:04.478 INFO:tasks.rados.rados.0.smithi151.stdout:1036: finishing write tid 1 to smithi15186934-47 2024-09-13T14:25:04.478 INFO:tasks.rados.rados.0.smithi151.stdout:1036: finishing write tid 2 to smithi15186934-47 2024-09-13T14:25:04.478 INFO:tasks.rados.rados.0.smithi151.stdout:1037: seq_num 418 ranges {625326=537144,1944179=672467} 2024-09-13T14:25:04.484 INFO:tasks.rados.rados.0.smithi151.stdout:1037: writing smithi15186934-20 from 625326 to 1162470 tid 1 2024-09-13T14:25:04.491 INFO:tasks.rados.rados.0.smithi151.stdout:1037: writing smithi15186934-20 from 1944179 to 2616646 tid 2 2024-09-13T14:25:04.492 INFO:tasks.rados.rados.0.smithi151.stdout:1038: snap_create 2024-09-13T14:25:04.492 INFO:tasks.rados.rados.0.smithi151.stdout:1036: finishing write tid 3 to smithi15186934-47 2024-09-13T14:25:04.492 INFO:tasks.rados.rados.0.smithi151.stdout:1036: finishing write tid 4 to smithi15186934-47 2024-09-13T14:25:04.492 INFO:tasks.rados.rados.0.smithi151.stdout:1036: finishing write tid 5 to smithi15186934-47 2024-09-13T14:25:04.492 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 513 (ObjNum 417 snap 130 seq_num 417) dirty exists 2024-09-13T14:25:04.492 INFO:tasks.rados.rados.0.smithi151.stdout:1036: left oid 47 (ObjNum 417 snap 130 seq_num 417) 2024-09-13T14:25:04.492 INFO:tasks.rados.rados.0.smithi151.stdout:1037: finishing write tid 1 to smithi15186934-20 2024-09-13T14:25:04.494 INFO:tasks.rados.rados.0.smithi151.stdout:1037: finishing write tid 2 to smithi15186934-20 2024-09-13T14:25:04.495 INFO:tasks.rados.rados.0.smithi151.stdout:1037: finishing write tid 3 to smithi15186934-20 2024-09-13T14:25:04.495 INFO:tasks.rados.rados.0.smithi151.stdout:1037: finishing write tid 4 to smithi15186934-20 2024-09-13T14:25:04.495 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 258 (ObjNum 418 snap 130 seq_num 418) dirty exists 2024-09-13T14:25:04.495 INFO:tasks.rados.rados.0.smithi151.stdout:1037: left oid 20 (ObjNum 418 snap 130 seq_num 418) 2024-09-13T14:25:04.547 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:25:04 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:25:04.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:25:04.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:04 smithi184 ceph-mon[33394]: pgmap v96: 89 pgs: 89 active+clean; 189 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 668 KiB/s wr, 18 op/s 2024-09-13T14:25:04.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:04 smithi184 ceph-mon[33394]: osdmap e322: 8 total, 8 up, 8 in 2024-09-13T14:25:04.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:04 smithi184 ceph-mon[33394]: osdmap e323: 8 total, 8 up, 8 in 2024-09-13T14:25:04.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:04 smithi151 ceph-mon[104450]: pgmap v96: 89 pgs: 89 active+clean; 189 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 668 KiB/s wr, 18 op/s 2024-09-13T14:25:04.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:04 smithi151 ceph-mon[104450]: osdmap e322: 8 total, 8 up, 8 in 2024-09-13T14:25:04.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:04 smithi151 ceph-mon[104450]: osdmap e323: 8 total, 8 up, 8 in 2024-09-13T14:25:04.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:25:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:25:04] "GET /metrics HTTP/1.1" 200 36402 "" "Prometheus/2.51.0" 2024-09-13T14:25:04.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:04 smithi151 ceph-mon[41387]: pgmap v96: 89 pgs: 89 active+clean; 189 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 668 KiB/s wr, 18 op/s 2024-09-13T14:25:04.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:04 smithi151 ceph-mon[41387]: osdmap e322: 8 total, 8 up, 8 in 2024-09-13T14:25:04.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:04 smithi151 ceph-mon[41387]: osdmap e323: 8 total, 8 up, 8 in 2024-09-13T14:25:05.456 INFO:tasks.rados.rados.0.smithi151.stdout:1036: done (2 left) 2024-09-13T14:25:05.457 INFO:tasks.rados.rados.0.smithi151.stdout:1037: done (1 left) 2024-09-13T14:25:05.457 INFO:tasks.rados.rados.0.smithi151.stdout:1038: done (0 left) 2024-09-13T14:25:05.457 INFO:tasks.rados.rados.0.smithi151.stdout:1039: read oid 36 snap 129 2024-09-13T14:25:05.457 INFO:tasks.rados.rados.0.smithi151.stdout:1039: expect (ObjNum 398 snap 123 seq_num 398) 2024-09-13T14:25:05.457 INFO:tasks.rados.rados.0.smithi151.stdout:1040: write oid 8 current snap is 131 2024-09-13T14:25:05.457 INFO:tasks.rados.rados.0.smithi151.stdout:1040: seq_num 419 ranges {642133=728746,2119014=532478} 2024-09-13T14:25:05.463 INFO:tasks.rados.rados.0.smithi151.stdout:1040: writing smithi15186934-8 from 642133 to 1370879 tid 1 2024-09-13T14:25:05.471 INFO:tasks.rados.rados.0.smithi151.stdout:1040: writing smithi15186934-8 from 2119014 to 2651492 tid 2 2024-09-13T14:25:05.471 INFO:tasks.rados.rados.0.smithi151.stdout:1041: write oid 16 current snap is 131 2024-09-13T14:25:05.471 INFO:tasks.rados.rados.0.smithi151.stdout:1041: seq_num 420 ranges {658940=403995,1777496=676136,2950951=107989} 2024-09-13T14:25:05.477 INFO:tasks.rados.rados.0.smithi151.stdout:1041: writing smithi15186934-16 from 658940 to 1062935 tid 1 2024-09-13T14:25:05.484 INFO:tasks.rados.rados.0.smithi151.stdout:1041: writing smithi15186934-16 from 1777496 to 2453632 tid 2 2024-09-13T14:25:05.488 INFO:tasks.rados.rados.0.smithi151.stdout:1041: writing smithi15186934-16 from 2950951 to 3058940 tid 3 2024-09-13T14:25:05.488 INFO:tasks.rados.rados.0.smithi151.stdout:1042: rollback oid 9 current snap is 131 2024-09-13T14:25:05.488 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 9 to 101 2024-09-13T14:25:05.488 INFO:tasks.rados.rados.0.smithi151.stdout:1039: expect (ObjNum 398 snap 123 seq_num 398) 2024-09-13T14:25:05.572 INFO:tasks.rados.rados.0.smithi151.stdout:1040: finishing write tid 1 to smithi15186934-8 2024-09-13T14:25:05.572 INFO:tasks.rados.rados.0.smithi151.stdout:1039: done (3 left) 2024-09-13T14:25:05.572 INFO:tasks.rados.rados.0.smithi151.stdout:1043: read oid 2 snap -1 2024-09-13T14:25:05.572 INFO:tasks.rados.rados.0.smithi151.stdout:1043: expect (ObjNum 377 snap 119 seq_num 377) 2024-09-13T14:25:05.572 INFO:tasks.rados.rados.0.smithi151.stdout:1044: write oid 17 current snap is 131 2024-09-13T14:25:05.572 INFO:tasks.rados.rados.0.smithi151.stdout:1040: finishing write tid 2 to smithi15186934-8 2024-09-13T14:25:05.573 INFO:tasks.rados.rados.0.smithi151.stdout:1040: finishing write tid 3 to smithi15186934-8 2024-09-13T14:25:05.573 INFO:tasks.rados.rados.0.smithi151.stdout:1040: finishing write tid 4 to smithi15186934-8 2024-09-13T14:25:05.573 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 280 (ObjNum 419 snap 131 seq_num 419) dirty exists 2024-09-13T14:25:05.573 INFO:tasks.rados.rados.0.smithi151.stdout:1040: left oid 8 (ObjNum 419 snap 131 seq_num 419) 2024-09-13T14:25:05.573 INFO:tasks.rados.rados.0.smithi151.stdout:1041: finishing write tid 1 to smithi15186934-16 2024-09-13T14:25:05.573 INFO:tasks.rados.rados.0.smithi151.stdout:1044: seq_num 421 ranges {675747=479244,1835978=536147} 2024-09-13T14:25:05.578 INFO:tasks.rados.rados.0.smithi151.stdout:1044: writing smithi15186934-17 from 675747 to 1154991 tid 1 2024-09-13T14:25:05.585 INFO:tasks.rados.rados.0.smithi151.stdout:1044: writing smithi15186934-17 from 1835978 to 2372125 tid 2 2024-09-13T14:25:05.585 INFO:tasks.rados.rados.0.smithi151.stdout:1040: done (4 left) 2024-09-13T14:25:05.585 INFO:tasks.rados.rados.0.smithi151.stdout:1045: snap_create 2024-09-13T14:25:05.585 INFO:tasks.rados.rados.0.smithi151.stdout:1041: finishing write tid 2 to smithi15186934-16 2024-09-13T14:25:05.585 INFO:tasks.rados.rados.0.smithi151.stdout:1042: finishing rollback tid 1 to smithi15186934-9 2024-09-13T14:25:05.585 INFO:tasks.rados.rados.0.smithi151.stdout:1041: finishing write tid 3 to smithi15186934-16 2024-09-13T14:25:05.586 INFO:tasks.rados.rados.0.smithi151.stdout:1041: finishing write tid 4 to smithi15186934-16 2024-09-13T14:25:05.586 INFO:tasks.rados.rados.0.smithi151.stdout:1042: finishing rollback tid 2 to smithi15186934-9 2024-09-13T14:25:05.586 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 448 (ObjNum 268 snap 86 seq_num 268) dirty exists 2024-09-13T14:25:05.586 INFO:tasks.rados.rados.0.smithi151.stdout:1041: finishing write tid 5 to smithi15186934-16 2024-09-13T14:25:05.586 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 16 v 263 (ObjNum 420 snap 131 seq_num 420) dirty exists 2024-09-13T14:25:05.586 INFO:tasks.rados.rados.0.smithi151.stdout:1041: left oid 16 (ObjNum 420 snap 131 seq_num 420) 2024-09-13T14:25:05.586 INFO:tasks.rados.rados.0.smithi151.stdout:1043: expect (ObjNum 377 snap 119 seq_num 377) 2024-09-13T14:25:05.695 INFO:tasks.rados.rados.0.smithi151.stdout:1044: finishing write tid 1 to smithi15186934-17 2024-09-13T14:25:05.695 INFO:tasks.rados.rados.0.smithi151.stdout:1044: finishing write tid 2 to smithi15186934-17 2024-09-13T14:25:05.695 INFO:tasks.rados.rados.0.smithi151.stdout:1044: finishing write tid 3 to smithi15186934-17 2024-09-13T14:25:05.695 INFO:tasks.rados.rados.0.smithi151.stdout:1044: finishing write tid 4 to smithi15186934-17 2024-09-13T14:25:05.695 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 711 (ObjNum 421 snap 131 seq_num 421) dirty exists 2024-09-13T14:25:05.695 INFO:tasks.rados.rados.0.smithi151.stdout:1044: left oid 17 (ObjNum 421 snap 131 seq_num 421) 2024-09-13T14:25:06.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:25:05 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:25:05.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:25:06.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:25:05 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:25:05.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:25:06.558 INFO:tasks.rados.rados.0.smithi151.stdout:1041: done (4 left) 2024-09-13T14:25:06.558 INFO:tasks.rados.rados.0.smithi151.stdout:1042: done (3 left) 2024-09-13T14:25:06.558 INFO:tasks.rados.rados.0.smithi151.stdout:1043: done (2 left) 2024-09-13T14:25:06.558 INFO:tasks.rados.rados.0.smithi151.stdout:1044: done (1 left) 2024-09-13T14:25:06.558 INFO:tasks.rados.rados.0.smithi151.stdout:1045: done (0 left) 2024-09-13T14:25:06.558 INFO:tasks.rados.rados.0.smithi151.stdout:1046: rollback oid 27 current snap is 132 2024-09-13T14:25:06.558 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 27 to 101 2024-09-13T14:25:06.558 INFO:tasks.rados.rados.0.smithi151.stdout:1047: write oid 39 current snap is 132 2024-09-13T14:25:06.559 INFO:tasks.rados.rados.0.smithi151.stdout:1047: seq_num 422 ranges {692554=554493,1778107=679805,2889444=203110} 2024-09-13T14:25:06.564 INFO:tasks.rados.rados.0.smithi151.stdout:1047: writing smithi15186934-39 from 692554 to 1247047 tid 1 2024-09-13T14:25:06.571 INFO:tasks.rados.rados.0.smithi151.stdout:1047: writing smithi15186934-39 from 1778107 to 2457912 tid 2 2024-09-13T14:25:06.575 INFO:tasks.rados.rados.0.smithi151.stdout:1047: writing smithi15186934-39 from 2889444 to 3092554 tid 3 2024-09-13T14:25:06.575 INFO:tasks.rados.rados.0.smithi151.stdout:1048: write oid 38 current snap is 132 2024-09-13T14:25:06.576 INFO:tasks.rados.rados.0.smithi151.stdout:1046: finishing rollback tid 1 to smithi15186934-27 2024-09-13T14:25:06.576 INFO:tasks.rados.rados.0.smithi151.stdout:1048: seq_num 423 ranges {709361=629742,1836589=539816,3033220=76141} 2024-09-13T14:25:06.582 INFO:tasks.rados.rados.0.smithi151.stdout:1048: writing smithi15186934-38 from 709361 to 1339103 tid 1 2024-09-13T14:25:06.588 INFO:tasks.rados.rados.0.smithi151.stdout:1048: writing smithi15186934-38 from 1836589 to 2376405 tid 2 2024-09-13T14:25:06.593 INFO:tasks.rados.rados.0.smithi151.stdout:1048: writing smithi15186934-38 from 3033220 to 3109361 tid 3 2024-09-13T14:25:06.593 INFO:tasks.rados.rados.0.smithi151.stdout:1049: write oid 32 current snap is 132 2024-09-13T14:25:06.593 INFO:tasks.rados.rados.0.smithi151.stdout:1046: finishing rollback tid 2 to smithi15186934-27 2024-09-13T14:25:06.593 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 27 v 265 (ObjNum 318 snap 97 seq_num 318) dirty exists 2024-09-13T14:25:06.593 INFO:tasks.rados.rados.0.smithi151.stdout:1049: seq_num 424 ranges {726168=704991,1895071=683474} 2024-09-13T14:25:06.601 INFO:tasks.rados.rados.0.smithi151.stdout:1049: writing smithi15186934-32 from 726168 to 1431159 tid 1 2024-09-13T14:25:06.608 INFO:tasks.rados.rados.0.smithi151.stdout:1049: writing smithi15186934-32 from 1895071 to 2578545 tid 2 2024-09-13T14:25:06.608 INFO:tasks.rados.rados.0.smithi151.stdout:1046: done (3 left) 2024-09-13T14:25:06.608 INFO:tasks.rados.rados.0.smithi151.stdout:1050: read oid 16 snap 131 2024-09-13T14:25:06.608 INFO:tasks.rados.rados.0.smithi151.stdout:1050: expect (ObjNum 420 snap 131 seq_num 420) 2024-09-13T14:25:06.608 INFO:tasks.rados.rados.0.smithi151.stdout:1051: rollback oid 8 current snap is 132 2024-09-13T14:25:06.608 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 8 to 128 2024-09-13T14:25:06.608 INFO:tasks.rados.rados.0.smithi151.stdout:1047: finishing write tid 1 to smithi15186934-39 2024-09-13T14:25:06.608 INFO:tasks.rados.rados.0.smithi151.stdout:1047: finishing write tid 2 to smithi15186934-39 2024-09-13T14:25:06.609 INFO:tasks.rados.rados.0.smithi151.stdout:1047: finishing write tid 3 to smithi15186934-39 2024-09-13T14:25:06.609 INFO:tasks.rados.rados.0.smithi151.stdout:1047: finishing write tid 4 to smithi15186934-39 2024-09-13T14:25:06.609 INFO:tasks.rados.rados.0.smithi151.stdout:1047: finishing write tid 5 to smithi15186934-39 2024-09-13T14:25:06.609 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 320 (ObjNum 422 snap 132 seq_num 422) dirty exists 2024-09-13T14:25:06.609 INFO:tasks.rados.rados.0.smithi151.stdout:1047: left oid 39 (ObjNum 422 snap 132 seq_num 422) 2024-09-13T14:25:06.609 INFO:tasks.rados.rados.0.smithi151.stdout:1048: finishing write tid 1 to smithi15186934-38 2024-09-13T14:25:06.609 INFO:tasks.rados.rados.0.smithi151.stdout:1048: finishing write tid 2 to smithi15186934-38 2024-09-13T14:25:06.609 INFO:tasks.rados.rados.0.smithi151.stdout:1047: done (4 left) 2024-09-13T14:25:06.609 INFO:tasks.rados.rados.0.smithi151.stdout:1052: snap_create 2024-09-13T14:25:06.609 INFO:tasks.rados.rados.0.smithi151.stdout:1048: finishing write tid 3 to smithi15186934-38 2024-09-13T14:25:06.609 INFO:tasks.rados.rados.0.smithi151.stdout:1048: finishing write tid 4 to smithi15186934-38 2024-09-13T14:25:06.609 INFO:tasks.rados.rados.0.smithi151.stdout:1048: finishing write tid 5 to smithi15186934-38 2024-09-13T14:25:06.609 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 325 (ObjNum 423 snap 132 seq_num 423) dirty exists 2024-09-13T14:25:06.609 INFO:tasks.rados.rados.0.smithi151.stdout:1048: left oid 38 (ObjNum 423 snap 132 seq_num 423) 2024-09-13T14:25:06.610 INFO:tasks.rados.rados.0.smithi151.stdout:1049: finishing write tid 1 to smithi15186934-32 2024-09-13T14:25:06.610 INFO:tasks.rados.rados.0.smithi151.stdout:1051: finishing rollback tid 0 to smithi15186934-8 2024-09-13T14:25:06.611 INFO:tasks.rados.rados.0.smithi151.stdout:1051: finishing rollback tid 1 to smithi15186934-8 2024-09-13T14:25:06.611 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 283 (ObjNum 33 snap 0 seq_num 3680656941) dirty dne 2024-09-13T14:25:06.611 INFO:tasks.rados.rados.0.smithi151.stdout:1049: finishing write tid 2 to smithi15186934-32 2024-09-13T14:25:06.611 INFO:tasks.rados.rados.0.smithi151.stdout:1049: finishing write tid 3 to smithi15186934-32 2024-09-13T14:25:06.611 INFO:tasks.rados.rados.0.smithi151.stdout:1049: finishing write tid 4 to smithi15186934-32 2024-09-13T14:25:06.611 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 32 v 452 (ObjNum 424 snap 132 seq_num 424) dirty exists 2024-09-13T14:25:06.611 INFO:tasks.rados.rados.0.smithi151.stdout:1049: left oid 32 (ObjNum 424 snap 132 seq_num 424) 2024-09-13T14:25:06.617 INFO:tasks.rados.rados.0.smithi151.stdout:1050: expect (ObjNum 420 snap 131 seq_num 420) 2024-09-13T14:25:06.858 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:06 smithi184 ceph-mon[33394]: pgmap v99: 89 pgs: 89 active+clean; 189 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 668 KiB/s wr, 18 op/s 2024-09-13T14:25:06.858 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:06 smithi184 ceph-mon[33394]: osdmap e324: 8 total, 8 up, 8 in 2024-09-13T14:25:06.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:06 smithi151 ceph-mon[104450]: pgmap v99: 89 pgs: 89 active+clean; 189 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 668 KiB/s wr, 18 op/s 2024-09-13T14:25:06.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:06 smithi151 ceph-mon[104450]: osdmap e324: 8 total, 8 up, 8 in 2024-09-13T14:25:06.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:06 smithi151 ceph-mon[41387]: pgmap v99: 89 pgs: 89 active+clean; 189 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 668 KiB/s wr, 18 op/s 2024-09-13T14:25:06.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:06 smithi151 ceph-mon[41387]: osdmap e324: 8 total, 8 up, 8 in 2024-09-13T14:25:07.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:25:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:25:06.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi184\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:25:07.568 INFO:tasks.rados.rados.0.smithi151.stdout:1048: done (4 left) 2024-09-13T14:25:07.568 INFO:tasks.rados.rados.0.smithi151.stdout:1049: done (3 left) 2024-09-13T14:25:07.568 INFO:tasks.rados.rados.0.smithi151.stdout:1050: done (2 left) 2024-09-13T14:25:07.568 INFO:tasks.rados.rados.0.smithi151.stdout:1051: done (1 left) 2024-09-13T14:25:07.568 INFO:tasks.rados.rados.0.smithi151.stdout:1052: done (0 left) 2024-09-13T14:25:07.568 INFO:tasks.rados.rados.0.smithi151.stdout:1053: snap_remove snap 132 2024-09-13T14:25:07.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:07 smithi184 ceph-mon[33394]: osdmap e325: 8 total, 8 up, 8 in 2024-09-13T14:25:07.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:07 smithi151 ceph-mon[104450]: osdmap e325: 8 total, 8 up, 8 in 2024-09-13T14:25:07.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:07 smithi151 ceph-mon[41387]: osdmap e325: 8 total, 8 up, 8 in 2024-09-13T14:25:08.567 INFO:tasks.rados.rados.0.smithi151.stdout:1053: done (0 left) 2024-09-13T14:25:08.568 INFO:tasks.rados.rados.0.smithi151.stdout:1054: snap_create 2024-09-13T14:25:08.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:08 smithi184 ceph-mon[33394]: pgmap v102: 89 pgs: 89 active+clean; 195 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 1.2 MiB/s wr, 5 op/s 2024-09-13T14:25:08.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:08 smithi184 ceph-mon[33394]: osdmap e326: 8 total, 8 up, 8 in 2024-09-13T14:25:08.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:08 smithi151 ceph-mon[104450]: pgmap v102: 89 pgs: 89 active+clean; 195 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 1.2 MiB/s wr, 5 op/s 2024-09-13T14:25:08.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:08 smithi151 ceph-mon[104450]: osdmap e326: 8 total, 8 up, 8 in 2024-09-13T14:25:08.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:08 smithi151 ceph-mon[41387]: pgmap v102: 89 pgs: 89 active+clean; 195 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 1.2 MiB/s wr, 5 op/s 2024-09-13T14:25:08.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:08 smithi151 ceph-mon[41387]: osdmap e326: 8 total, 8 up, 8 in 2024-09-13T14:25:09.581 INFO:tasks.rados.rados.0.smithi151.stdout:1054: done (0 left) 2024-09-13T14:25:09.581 INFO:tasks.rados.rados.0.smithi151.stdout:1055: snap_remove snap 128 2024-09-13T14:25:09.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:09 smithi184 ceph-mon[33394]: osdmap e327: 8 total, 8 up, 8 in 2024-09-13T14:25:09.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:09 smithi151 ceph-mon[104450]: osdmap e327: 8 total, 8 up, 8 in 2024-09-13T14:25:09.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:09 smithi151 ceph-mon[41387]: osdmap e327: 8 total, 8 up, 8 in 2024-09-13T14:25:10.586 INFO:tasks.rados.rados.0.smithi151.stdout:1055: done (0 left) 2024-09-13T14:25:10.586 INFO:tasks.rados.rados.0.smithi151.stdout:1056: delete oid 17 current snap is 134 2024-09-13T14:25:10.586 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:25:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:25:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:25:10.591 INFO:tasks.rados.rados.0.smithi151.stdout:1056: done (0 left) 2024-09-13T14:25:10.592 INFO:tasks.rados.rados.0.smithi151.stdout:1057: read oid 16 snap -1 2024-09-13T14:25:10.592 INFO:tasks.rados.rados.0.smithi151.stdout:1057: expect (ObjNum 420 snap 131 seq_num 420) 2024-09-13T14:25:10.592 INFO:tasks.rados.rados.0.smithi151.stdout:1058: read oid 42 snap -1 2024-09-13T14:25:10.592 INFO:tasks.rados.rados.0.smithi151.stdout:1058: expect (ObjNum 385 snap 120 seq_num 385) 2024-09-13T14:25:10.592 INFO:tasks.rados.rados.0.smithi151.stdout:1059: snap_remove snap 127 2024-09-13T14:25:10.794 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:10 smithi151 ceph-mon[41387]: pgmap v105: 89 pgs: 89 active+clean; 213 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 3.6 MiB/s wr, 27 op/s 2024-09-13T14:25:10.794 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:10 smithi151 ceph-mon[41387]: osdmap e328: 8 total, 8 up, 8 in 2024-09-13T14:25:10.794 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:10 smithi151 ceph-mon[104450]: pgmap v105: 89 pgs: 89 active+clean; 213 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 3.6 MiB/s wr, 27 op/s 2024-09-13T14:25:10.794 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:10 smithi151 ceph-mon[104450]: osdmap e328: 8 total, 8 up, 8 in 2024-09-13T14:25:10.821 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch upgrade resume' 2024-09-13T14:25:10.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:10 smithi184 ceph-mon[33394]: pgmap v105: 89 pgs: 89 active+clean; 213 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 3.6 MiB/s wr, 27 op/s 2024-09-13T14:25:10.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:10 smithi184 ceph-mon[33394]: osdmap e328: 8 total, 8 up, 8 in 2024-09-13T14:25:11.075 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:25:11.603 INFO:tasks.rados.rados.0.smithi151.stdout:1059: done (2 left) 2024-09-13T14:25:11.603 INFO:tasks.rados.rados.0.smithi151.stdout:1060: snap_create 2024-09-13T14:25:11.604 INFO:tasks.rados.rados.0.smithi151.stdout:1058: expect (ObjNum 385 snap 120 seq_num 385) 2024-09-13T14:25:11.805 INFO:tasks.rados.rados.0.smithi151.stdout:1057: expect (ObjNum 420 snap 131 seq_num 420) 2024-09-13T14:25:11.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:11 smithi184 ceph-mon[33394]: osdmap e329: 8 total, 8 up, 8 in 2024-09-13T14:25:11.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:11 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:25:11.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:11 smithi151 ceph-mon[41387]: osdmap e329: 8 total, 8 up, 8 in 2024-09-13T14:25:11.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:11 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:25:11.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:11 smithi151 ceph-mon[104450]: osdmap e329: 8 total, 8 up, 8 in 2024-09-13T14:25:11.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:11 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:25:12.605 INFO:tasks.rados.rados.0.smithi151.stdout:1057: done (2 left) 2024-09-13T14:25:12.606 INFO:tasks.rados.rados.0.smithi151.stdout:1058: done (1 left) 2024-09-13T14:25:12.606 INFO:tasks.rados.rados.0.smithi151.stdout:1060: done (0 left) 2024-09-13T14:25:12.606 INFO:tasks.rados.rados.0.smithi151.stdout:1061: read oid 27 snap -1 2024-09-13T14:25:12.606 INFO:tasks.rados.rados.0.smithi151.stdout:1061: expect (ObjNum 318 snap 97 seq_num 318) 2024-09-13T14:25:12.606 INFO:tasks.rados.rados.0.smithi151.stdout:1062: read oid 41 snap -1 2024-09-13T14:25:12.606 INFO:tasks.rados.rados.0.smithi151.stdout:1062: expect deleted 2024-09-13T14:25:12.606 INFO:tasks.rados.rados.0.smithi151.stdout:1063: write oid 6 current snap is 135 2024-09-13T14:25:12.606 INFO:tasks.rados.rados.0.smithi151.stdout:1063: seq_num 426 ranges {759782=571842,2012035=687143} 2024-09-13T14:25:12.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:12 smithi151 ceph-mon[41387]: pgmap v108: 89 pgs: 89 active+clean; 213 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 2.4 MiB/s wr, 22 op/s 2024-09-13T14:25:12.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:12 smithi151 ceph-mon[41387]: osdmap e330: 8 total, 8 up, 8 in 2024-09-13T14:25:12.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:12 smithi151 ceph-mon[104450]: pgmap v108: 89 pgs: 89 active+clean; 213 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 2.4 MiB/s wr, 22 op/s 2024-09-13T14:25:12.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:12 smithi151 ceph-mon[104450]: osdmap e330: 8 total, 8 up, 8 in 2024-09-13T14:25:12.612 INFO:tasks.rados.rados.0.smithi151.stdout:1063: writing smithi15186934-6 from 759782 to 1331624 tid 1 2024-09-13T14:25:12.621 INFO:tasks.rados.rados.0.smithi151.stdout:1063: writing smithi15186934-6 from 2012035 to 2699178 tid 2 2024-09-13T14:25:12.621 INFO:tasks.rados.rados.0.smithi151.stdout:1064: delete oid 23 current snap is 135 2024-09-13T14:25:12.621 INFO:tasks.rados.rados.0.smithi151.stdout:1063: finishing write tid 1 to smithi15186934-6 2024-09-13T14:25:12.621 INFO:tasks.rados.rados.0.smithi151.stdout:1061: expect (ObjNum 318 snap 97 seq_num 318) 2024-09-13T14:25:12.792 INFO:tasks.rados.rados.0.smithi151.stdout:1063: finishing write tid 2 to smithi15186934-6 2024-09-13T14:25:12.793 INFO:tasks.rados.rados.0.smithi151.stdout:1063: finishing write tid 3 to smithi15186934-6 2024-09-13T14:25:12.793 INFO:tasks.rados.rados.0.smithi151.stdout:1061: done (3 left) 2024-09-13T14:25:12.793 INFO:tasks.rados.rados.0.smithi151.stdout:1062: done (2 left) 2024-09-13T14:25:12.793 INFO:tasks.rados.rados.0.smithi151.stdout:1064: done (1 left) 2024-09-13T14:25:12.793 INFO:tasks.rados.rados.0.smithi151.stdout:1065: read oid 19 snap 133 2024-09-13T14:25:12.793 INFO:tasks.rados.rados.0.smithi151.stdout:1065: expect (ObjNum 323 snap 101 seq_num 323) 2024-09-13T14:25:12.793 INFO:tasks.rados.rados.0.smithi151.stdout:1066: read oid 9 snap -1 2024-09-13T14:25:12.793 INFO:tasks.rados.rados.0.smithi151.stdout:1066: expect (ObjNum 268 snap 86 seq_num 268) 2024-09-13T14:25:12.793 INFO:tasks.rados.rados.0.smithi151.stdout:1067: read oid 16 snap -1 2024-09-13T14:25:12.793 INFO:tasks.rados.rados.0.smithi151.stdout:1067: expect (ObjNum 420 snap 131 seq_num 420) 2024-09-13T14:25:12.793 INFO:tasks.rados.rados.0.smithi151.stdout:1068: write oid 8 current snap is 135 2024-09-13T14:25:12.793 INFO:tasks.rados.rados.0.smithi151.stdout:1063: finishing write tid 4 to smithi15186934-6 2024-09-13T14:25:12.794 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 6 v 249 (ObjNum 426 snap 135 seq_num 426) dirty exists 2024-09-13T14:25:12.794 INFO:tasks.rados.rados.0.smithi151.stdout:1063: left oid 6 (ObjNum 426 snap 135 seq_num 426) 2024-09-13T14:25:12.794 INFO:tasks.rados.rados.0.smithi151.stdout:1068: seq_num 428 ranges {793396=722340,2128999=690812} 2024-09-13T14:25:12.800 INFO:tasks.rados.rados.0.smithi151.stdout:1068: writing smithi15186934-8 from 793396 to 1515736 tid 1 2024-09-13T14:25:12.808 INFO:tasks.rados.rados.0.smithi151.stdout:1068: writing smithi15186934-8 from 2128999 to 2819811 tid 2 2024-09-13T14:25:12.808 INFO:tasks.rados.rados.0.smithi151.stdout:1063: done (4 left) 2024-09-13T14:25:12.808 INFO:tasks.rados.rados.0.smithi151.stdout:1069: read oid 46 snap -1 2024-09-13T14:25:12.808 INFO:tasks.rados.rados.0.smithi151.stdout:1069: expect (ObjNum 261 snap 82 seq_num 261) 2024-09-13T14:25:12.808 INFO:tasks.rados.rados.0.smithi151.stdout:1070: rollback oid 24 current snap is 135 2024-09-13T14:25:12.808 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 24 to 123 2024-09-13T14:25:12.808 INFO:tasks.rados.rados.0.smithi151.stdout:1066: expect (ObjNum 268 snap 86 seq_num 268) 2024-09-13T14:25:12.822 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:12 smithi184 ceph-mon[33394]: pgmap v108: 89 pgs: 89 active+clean; 213 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 2.4 MiB/s wr, 22 op/s 2024-09-13T14:25:12.822 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:12 smithi184 ceph-mon[33394]: osdmap e330: 8 total, 8 up, 8 in 2024-09-13T14:25:13.014 INFO:tasks.rados.rados.0.smithi151.stdout:1065: expect (ObjNum 323 snap 101 seq_num 323) 2024-09-13T14:25:13.131 INFO:teuthology.orchestra.run.smithi151.stdout:Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:25:13.133 INFO:tasks.rados.rados.0.smithi151.stdout:1067: expect (ObjNum 420 snap 131 seq_num 420) 2024-09-13T14:25:13.262 INFO:tasks.rados.rados.0.smithi151.stdout:1068: finishing write tid 1 to smithi15186934-8 2024-09-13T14:25:13.263 INFO:tasks.rados.rados.0.smithi151.stdout:1065: done (5 left) 2024-09-13T14:25:13.263 INFO:tasks.rados.rados.0.smithi151.stdout:1066: done (4 left) 2024-09-13T14:25:13.263 INFO:tasks.rados.rados.0.smithi151.stdout:1067: done (3 left) 2024-09-13T14:25:13.263 INFO:tasks.rados.rados.0.smithi151.stdout:1071: read oid 28 snap -1 2024-09-13T14:25:13.263 INFO:tasks.rados.rados.0.smithi151.stdout:1071: expect (ObjNum 414 snap 129 seq_num 414) 2024-09-13T14:25:13.263 INFO:tasks.rados.rados.0.smithi151.stdout:1072: rollback oid 7 current snap is 135 2024-09-13T14:25:13.263 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 7 to 101 2024-09-13T14:25:13.263 INFO:tasks.rados.rados.0.smithi151.stdout:1070: finishing rollback tid 0 to smithi15186934-24 2024-09-13T14:25:13.263 INFO:tasks.rados.rados.0.smithi151.stdout:1070: finishing rollback tid 1 to smithi15186934-24 2024-09-13T14:25:13.263 INFO:tasks.rados.rados.0.smithi151.stdout:1068: finishing write tid 2 to smithi15186934-8 2024-09-13T14:25:13.263 INFO:tasks.rados.rados.0.smithi151.stdout:1070: finishing rollback tid 2 to smithi15186934-24 2024-09-13T14:25:13.263 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 24 v 721 (ObjNum 391 snap 121 seq_num 391) dirty exists 2024-09-13T14:25:13.264 INFO:tasks.rados.rados.0.smithi151.stdout:1068: finishing write tid 3 to smithi15186934-8 2024-09-13T14:25:13.264 INFO:tasks.rados.rados.0.smithi151.stdout:1069: expect (ObjNum 261 snap 82 seq_num 261) 2024-09-13T14:25:13.365 INFO:tasks.rados.rados.0.smithi151.stdout:1068: finishing write tid 4 to smithi15186934-8 2024-09-13T14:25:13.365 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 286 (ObjNum 428 snap 135 seq_num 428) dirty exists 2024-09-13T14:25:13.366 INFO:tasks.rados.rados.0.smithi151.stdout:1068: left oid 8 (ObjNum 428 snap 135 seq_num 428) 2024-09-13T14:25:13.366 INFO:tasks.rados.rados.0.smithi151.stdout:1068: done (4 left) 2024-09-13T14:25:13.366 INFO:tasks.rados.rados.0.smithi151.stdout:1069: done (3 left) 2024-09-13T14:25:13.366 INFO:tasks.rados.rados.0.smithi151.stdout:1070: done (2 left) 2024-09-13T14:25:13.366 INFO:tasks.rados.rados.0.smithi151.stdout:1073: write oid 42 current snap is 135 2024-09-13T14:25:13.366 INFO:tasks.rados.rados.0.smithi151.stdout:1072: finishing rollback tid 0 to smithi15186934-7 2024-09-13T14:25:13.366 INFO:tasks.rados.rados.0.smithi151.stdout:1073: seq_num 429 ranges {410203=797589,1787481=434470,2681405=528798} 2024-09-13T14:25:13.371 INFO:tasks.rados.rados.0.smithi151.stdout:1073: writing smithi15186934-42 from 410203 to 1207792 tid 1 2024-09-13T14:25:13.377 INFO:tasks.rados.rados.0.smithi151.stdout:1073: writing smithi15186934-42 from 1787481 to 2221951 tid 2 2024-09-13T14:25:13.383 INFO:tasks.rados.rados.0.smithi151.stdout:1073: writing smithi15186934-42 from 2681405 to 3210203 tid 3 2024-09-13T14:25:13.383 INFO:tasks.rados.rados.0.smithi151.stdout:1074: snap_create 2024-09-13T14:25:13.383 INFO:tasks.rados.rados.0.smithi151.stdout:1072: finishing rollback tid 1 to smithi15186934-7 2024-09-13T14:25:13.383 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 724 (ObjNum 97 snap 0 seq_num 1) dirty dne 2024-09-13T14:25:13.384 INFO:tasks.rados.rados.0.smithi151.stdout:1071: expect (ObjNum 414 snap 129 seq_num 414) 2024-09-13T14:25:13.558 INFO:tasks.rados.rados.0.smithi151.stdout:1073: finishing write tid 1 to smithi15186934-42 2024-09-13T14:25:13.558 INFO:tasks.rados.rados.0.smithi151.stdout:1073: finishing write tid 2 to smithi15186934-42 2024-09-13T14:25:13.558 INFO:tasks.rados.rados.0.smithi151.stdout:1073: finishing write tid 3 to smithi15186934-42 2024-09-13T14:25:13.558 INFO:tasks.rados.rados.0.smithi151.stdout:1073: finishing write tid 4 to smithi15186934-42 2024-09-13T14:25:13.558 INFO:tasks.rados.rados.0.smithi151.stdout:1073: finishing write tid 5 to smithi15186934-42 2024-09-13T14:25:13.558 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 570 (ObjNum 429 snap 135 seq_num 429) dirty exists 2024-09-13T14:25:13.559 INFO:tasks.rados.rados.0.smithi151.stdout:1073: left oid 42 (ObjNum 429 snap 135 seq_num 429) 2024-09-13T14:25:13.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:13 smithi184 ceph-mon[33394]: osdmap e331: 8 total, 8 up, 8 in 2024-09-13T14:25:13.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:13 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:13.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:13 smithi151 ceph-mon[104450]: osdmap e331: 8 total, 8 up, 8 in 2024-09-13T14:25:13.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:13 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:13.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:13 smithi151 ceph-mon[41387]: osdmap e331: 8 total, 8 up, 8 in 2024-09-13T14:25:13.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:13 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:14.135 INFO:tasks.rados.rados.0.smithi151.stdout:1071: done (3 left) 2024-09-13T14:25:14.135 INFO:tasks.rados.rados.0.smithi151.stdout:1072: done (2 left) 2024-09-13T14:25:14.135 INFO:tasks.rados.rados.0.smithi151.stdout:1073: done (1 left) 2024-09-13T14:25:14.135 INFO:tasks.rados.rados.0.smithi151.stdout:1074: done (0 left) 2024-09-13T14:25:14.135 INFO:tasks.rados.rados.0.smithi151.stdout:1075: snap_create 2024-09-13T14:25:14.184 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'echo wait for majority of mons to upgrade' 2024-09-13T14:25:14.462 INFO:tasks.rados.rados.0.smithi151.stdout:1075: done (0 left) 2024-09-13T14:25:14.462 INFO:tasks.rados.rados.0.smithi151.stdout:1076: read oid 47 snap -1 2024-09-13T14:25:14.463 INFO:tasks.rados.rados.0.smithi151.stdout:1076: expect (ObjNum 417 snap 130 seq_num 417) 2024-09-13T14:25:14.463 INFO:tasks.rados.rados.0.smithi151.stdout:1077: write oid 26 current snap is 137 2024-09-13T14:25:14.463 INFO:tasks.rados.rados.0.smithi151.stdout:1077: seq_num 430 ranges {427010=472838,1329610=694481,2708828=518182} 2024-09-13T14:25:14.466 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:25:14 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:25:14.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:25:14.467 INFO:tasks.rados.rados.0.smithi151.stdout:1077: writing smithi15186934-26 from 427010 to 899848 tid 1 2024-09-13T14:25:14.475 INFO:tasks.rados.rados.0.smithi151.stdout:1077: writing smithi15186934-26 from 1329610 to 2024091 tid 2 2024-09-13T14:25:14.482 INFO:tasks.rados.rados.0.smithi151.stdout:1077: writing smithi15186934-26 from 2708828 to 3227010 tid 3 2024-09-13T14:25:14.482 INFO:tasks.rados.rados.0.smithi151.stdout:1078: write oid 49 current snap is 137 2024-09-13T14:25:14.482 INFO:tasks.rados.rados.0.smithi151.stdout:1078: seq_num 431 ranges {443817=548087,1788092=438139,3019898=223919} 2024-09-13T14:25:14.487 INFO:tasks.rados.rados.0.smithi151.stdout:1078: writing smithi15186934-49 from 443817 to 991904 tid 1 2024-09-13T14:25:14.494 INFO:tasks.rados.rados.0.smithi151.stdout:1078: writing smithi15186934-49 from 1788092 to 2226231 tid 2 2024-09-13T14:25:14.500 INFO:tasks.rados.rados.0.smithi151.stdout:1078: writing smithi15186934-49 from 3019898 to 3243817 tid 3 2024-09-13T14:25:14.500 INFO:tasks.rados.rados.0.smithi151.stdout:1079: write oid 2 current snap is 137 2024-09-13T14:25:14.500 INFO:tasks.rados.rados.0.smithi151.stdout:1077: finishing write tid 1 to smithi15186934-26 2024-09-13T14:25:14.500 INFO:tasks.rados.rados.0.smithi151.stdout:1076: expect (ObjNum 417 snap 130 seq_num 417) 2024-09-13T14:25:14.672 INFO:tasks.rados.rados.0.smithi151.stdout:1077: finishing write tid 2 to smithi15186934-26 2024-09-13T14:25:14.672 INFO:tasks.rados.rados.0.smithi151.stdout:1079: seq_num 432 ranges {460624=623336,1846574=698150,3163674=96950} 2024-09-13T14:25:14.677 INFO:tasks.rados.rados.0.smithi151.stdout:1079: writing smithi15186934-2 from 460624 to 1083960 tid 1 2024-09-13T14:25:14.686 INFO:tasks.rados.rados.0.smithi151.stdout:1079: writing smithi15186934-2 from 1846574 to 2544724 tid 2 2024-09-13T14:25:14.691 INFO:tasks.rados.rados.0.smithi151.stdout:1079: writing smithi15186934-2 from 3163674 to 3260624 tid 3 2024-09-13T14:25:14.691 INFO:tasks.rados.rados.0.smithi151.stdout:1076: done (3 left) 2024-09-13T14:25:14.691 INFO:tasks.rados.rados.0.smithi151.stdout:1080: read oid 37 snap -1 2024-09-13T14:25:14.691 INFO:tasks.rados.rados.0.smithi151.stdout:1080: expect deleted 2024-09-13T14:25:14.691 INFO:tasks.rados.rados.0.smithi151.stdout:1081: read oid 14 snap -1 2024-09-13T14:25:14.691 INFO:tasks.rados.rados.0.smithi151.stdout:1081: expect (ObjNum 321 snap 98 seq_num 321) 2024-09-13T14:25:14.691 INFO:tasks.rados.rados.0.smithi151.stdout:1082: rollback oid 7 current snap is 137 2024-09-13T14:25:14.691 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 7 to 134 2024-09-13T14:25:14.691 INFO:tasks.rados.rados.0.smithi151.stdout:1077: finishing write tid 3 to smithi15186934-26 2024-09-13T14:25:14.691 INFO:tasks.rados.rados.0.smithi151.stdout:1077: finishing write tid 4 to smithi15186934-26 2024-09-13T14:25:14.692 INFO:tasks.rados.rados.0.smithi151.stdout:1077: finishing write tid 5 to smithi15186934-26 2024-09-13T14:25:14.692 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 339 (ObjNum 430 snap 137 seq_num 430) dirty exists 2024-09-13T14:25:14.692 INFO:tasks.rados.rados.0.smithi151.stdout:1077: left oid 26 (ObjNum 430 snap 137 seq_num 430) 2024-09-13T14:25:14.692 INFO:tasks.rados.rados.0.smithi151.stdout:1078: finishing write tid 1 to smithi15186934-49 2024-09-13T14:25:14.692 INFO:tasks.rados.rados.0.smithi151.stdout:1078: finishing write tid 2 to smithi15186934-49 2024-09-13T14:25:14.692 INFO:tasks.rados.rados.0.smithi151.stdout:1078: finishing write tid 3 to smithi15186934-49 2024-09-13T14:25:14.692 INFO:tasks.rados.rados.0.smithi151.stdout:1078: finishing write tid 4 to smithi15186934-49 2024-09-13T14:25:14.692 INFO:tasks.rados.rados.0.smithi151.stdout:1078: finishing write tid 5 to smithi15186934-49 2024-09-13T14:25:14.692 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 49 v 522 (ObjNum 431 snap 137 seq_num 431) dirty exists 2024-09-13T14:25:14.692 INFO:tasks.rados.rados.0.smithi151.stdout:1078: left oid 49 (ObjNum 431 snap 137 seq_num 431) 2024-09-13T14:25:14.692 INFO:tasks.rados.rados.0.smithi151.stdout:1079: finishing write tid 1 to smithi15186934-2 2024-09-13T14:25:14.692 INFO:tasks.rados.rados.0.smithi151.stdout:1079: finishing write tid 2 to smithi15186934-2 2024-09-13T14:25:14.692 INFO:tasks.rados.rados.0.smithi151.stdout:1077: done (5 left) 2024-09-13T14:25:14.693 INFO:tasks.rados.rados.0.smithi151.stdout:1078: done (4 left) 2024-09-13T14:25:14.693 INFO:tasks.rados.rados.0.smithi151.stdout:1083: write oid 11 current snap is 137 2024-09-13T14:25:14.693 INFO:tasks.rados.rados.0.smithi151.stdout:1083: seq_num 433 ranges {477431=698585,1905056=441808,3074744=202687} 2024-09-13T14:25:14.696 INFO:tasks.rados.rados.0.smithi151.stdout:1083: writing smithi15186934-11 from 477431 to 1176016 tid 1 2024-09-13T14:25:14.703 INFO:tasks.rados.rados.0.smithi151.stdout:1083: writing smithi15186934-11 from 1905056 to 2346864 tid 2 2024-09-13T14:25:14.709 INFO:tasks.rados.rados.0.smithi151.stdout:1083: writing smithi15186934-11 from 3074744 to 3277431 tid 3 2024-09-13T14:25:14.709 INFO:tasks.rados.rados.0.smithi151.stdout:1084: snap_create 2024-09-13T14:25:14.709 INFO:tasks.rados.rados.0.smithi151.stdout:1082: finishing rollback tid 1 to smithi15186934-7 2024-09-13T14:25:14.709 INFO:tasks.rados.rados.0.smithi151.stdout:1079: finishing write tid 3 to smithi15186934-2 2024-09-13T14:25:14.709 INFO:tasks.rados.rados.0.smithi151.stdout:1079: finishing write tid 4 to smithi15186934-2 2024-09-13T14:25:14.709 INFO:tasks.rados.rados.0.smithi151.stdout:1082: finishing rollback tid 2 to smithi15186934-7 2024-09-13T14:25:14.709 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 726 (ObjNum 396 snap 123 seq_num 396) dirty exists 2024-09-13T14:25:14.709 INFO:tasks.rados.rados.0.smithi151.stdout:1079: finishing write tid 5 to smithi15186934-2 2024-09-13T14:25:14.709 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 2 v 376 (ObjNum 432 snap 137 seq_num 432) dirty exists 2024-09-13T14:25:14.709 INFO:tasks.rados.rados.0.smithi151.stdout:1079: left oid 2 (ObjNum 432 snap 137 seq_num 432) 2024-09-13T14:25:14.709 INFO:tasks.rados.rados.0.smithi151.stdout:1081: expect (ObjNum 321 snap 98 seq_num 321) 2024-09-13T14:25:14.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:14 smithi184 ceph-mon[33394]: pgmap v111: 89 pgs: 89 active+clean; 212 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.3 MiB/s rd, 0 B/s wr, 4 op/s 2024-09-13T14:25:14.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:14 smithi184 ceph-mon[33394]: from='client.26012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:25:14.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:14 smithi184 ceph-mon[33394]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:25:14.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:14 smithi184 ceph-mon[33394]: osdmap e332: 8 total, 8 up, 8 in 2024-09-13T14:25:14.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:14 smithi184 ceph-mon[33394]: osdmap e333: 8 total, 8 up, 8 in 2024-09-13T14:25:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:14 smithi151 ceph-mon[104450]: pgmap v111: 89 pgs: 89 active+clean; 212 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.3 MiB/s rd, 0 B/s wr, 4 op/s 2024-09-13T14:25:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:14 smithi151 ceph-mon[104450]: from='client.26012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:25:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:14 smithi151 ceph-mon[104450]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:25:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:14 smithi151 ceph-mon[104450]: osdmap e332: 8 total, 8 up, 8 in 2024-09-13T14:25:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:14 smithi151 ceph-mon[104450]: osdmap e333: 8 total, 8 up, 8 in 2024-09-13T14:25:14.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:25:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:25:14] "GET /metrics HTTP/1.1" 200 36402 "" "Prometheus/2.51.0" 2024-09-13T14:25:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:14 smithi151 ceph-mon[41387]: pgmap v111: 89 pgs: 89 active+clean; 212 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.3 MiB/s rd, 0 B/s wr, 4 op/s 2024-09-13T14:25:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:14 smithi151 ceph-mon[41387]: from='client.26012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:25:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:14 smithi151 ceph-mon[41387]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:25:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:14 smithi151 ceph-mon[41387]: osdmap e332: 8 total, 8 up, 8 in 2024-09-13T14:25:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:14 smithi151 ceph-mon[41387]: osdmap e333: 8 total, 8 up, 8 in 2024-09-13T14:25:14.861 INFO:tasks.rados.rados.0.smithi151.stdout:1083: finishing write tid 1 to smithi15186934-11 2024-09-13T14:25:14.862 INFO:tasks.rados.rados.0.smithi151.stdout:1083: finishing write tid 2 to smithi15186934-11 2024-09-13T14:25:14.862 INFO:tasks.rados.rados.0.smithi151.stdout:1083: finishing write tid 3 to smithi15186934-11 2024-09-13T14:25:14.862 INFO:tasks.rados.rados.0.smithi151.stdout:1083: finishing write tid 4 to smithi15186934-11 2024-09-13T14:25:14.862 INFO:tasks.rados.rados.0.smithi151.stdout:1083: finishing write tid 5 to smithi15186934-11 2024-09-13T14:25:14.862 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 11 v 292 (ObjNum 433 snap 137 seq_num 433) dirty exists 2024-09-13T14:25:14.862 INFO:tasks.rados.rados.0.smithi151.stdout:1083: left oid 11 (ObjNum 433 snap 137 seq_num 433) 2024-09-13T14:25:14.937 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:25:15.615 INFO:tasks.rados.rados.0.smithi151.stdout:1079: done (5 left) 2024-09-13T14:25:15.615 INFO:tasks.rados.rados.0.smithi151.stdout:1080: done (4 left) 2024-09-13T14:25:15.615 INFO:tasks.rados.rados.0.smithi151.stdout:1081: done (3 left) 2024-09-13T14:25:15.615 INFO:tasks.rados.rados.0.smithi151.stdout:1082: done (2 left) 2024-09-13T14:25:15.615 INFO:tasks.rados.rados.0.smithi151.stdout:1083: done (1 left) 2024-09-13T14:25:15.615 INFO:tasks.rados.rados.0.smithi151.stdout:1084: done (0 left) 2024-09-13T14:25:15.615 INFO:tasks.rados.rados.0.smithi151.stdout:1085: rollback oid 17 current snap is 138 2024-09-13T14:25:15.615 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 17 to 130 2024-09-13T14:25:15.615 INFO:tasks.rados.rados.0.smithi151.stdout:1086: rollback oid 23 current snap is 138 2024-09-13T14:25:15.616 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 23 to 131 2024-09-13T14:25:15.616 INFO:tasks.rados.rados.0.smithi151.stdout:1087: read oid 24 snap 131 2024-09-13T14:25:15.616 INFO:tasks.rados.rados.0.smithi151.stdout:1087: expect (ObjNum 391 snap 121 seq_num 391) 2024-09-13T14:25:15.616 INFO:tasks.rados.rados.0.smithi151.stdout:1088: read oid 20 snap -1 2024-09-13T14:25:15.616 INFO:tasks.rados.rados.0.smithi151.stdout:1088: expect (ObjNum 418 snap 130 seq_num 418) 2024-09-13T14:25:15.616 INFO:tasks.rados.rados.0.smithi151.stdout:1089: read oid 21 snap -1 2024-09-13T14:25:15.616 INFO:tasks.rados.rados.0.smithi151.stdout:1089: expect deleted 2024-09-13T14:25:15.616 INFO:tasks.rados.rados.0.smithi151.stdout:1090: read oid 34 snap -1 2024-09-13T14:25:15.616 INFO:tasks.rados.rados.0.smithi151.stdout:1090: expect deleted 2024-09-13T14:25:15.616 INFO:tasks.rados.rados.0.smithi151.stdout:1091: read oid 1 snap -1 2024-09-13T14:25:15.616 INFO:tasks.rados.rados.0.smithi151.stdout:1091: expect (ObjNum 381 snap 120 seq_num 381) 2024-09-13T14:25:15.616 INFO:tasks.rados.rados.0.smithi151.stdout:1092: write oid 48 current snap is 138 2024-09-13T14:25:15.616 INFO:tasks.rados.rados.0.smithi151.stdout:1092: seq_num 434 ranges {494238=490187,1563538=701819,2818520=475718} 2024-09-13T14:25:15.619 INFO:tasks.rados.rados.0.smithi151.stdout:1092: writing smithi15186934-48 from 494238 to 984425 tid 1 2024-09-13T14:25:15.627 INFO:tasks.rados.rados.0.smithi151.stdout:1092: writing smithi15186934-48 from 1563538 to 2265357 tid 2 2024-09-13T14:25:15.633 INFO:tasks.rados.rados.0.smithi151.stdout:1092: writing smithi15186934-48 from 2818520 to 3294238 tid 3 2024-09-13T14:25:15.633 INFO:tasks.rados.rados.0.smithi151.stdout:1093: rollback oid 2 current snap is 138 2024-09-13T14:25:15.633 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 2 to 124 2024-09-13T14:25:15.633 INFO:tasks.rados.rados.0.smithi151.stdout:1091: expect (ObjNum 381 snap 120 seq_num 381) 2024-09-13T14:25:15.712 INFO:tasks.rados.rados.0.smithi151.stdout:1086: finishing rollback tid 1 to smithi15186934-23 2024-09-13T14:25:15.712 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 23 v 0 (ObjNum 33 snap 0 seq_num 3092875168) dirty dne 2024-09-13T14:25:15.712 INFO:tasks.rados.rados.0.smithi151.stdout:1092: finishing write tid 1 to smithi15186934-48 2024-09-13T14:25:15.712 INFO:tasks.rados.rados.0.smithi151.stdout:1087: expect (ObjNum 391 snap 121 seq_num 391) 2024-09-13T14:25:15.907 INFO:tasks.rados.rados.0.smithi151.stdout:1085: finishing rollback tid 1 to smithi15186934-17 2024-09-13T14:25:15.907 INFO:tasks.rados.rados.0.smithi151.stdout:1085: finishing rollback tid 2 to smithi15186934-17 2024-09-13T14:25:15.907 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 728 (ObjNum 403 snap 123 seq_num 403) dirty exists 2024-09-13T14:25:15.907 INFO:tasks.rados.rados.0.smithi151.stdout:1092: finishing write tid 2 to smithi15186934-48 2024-09-13T14:25:15.907 INFO:tasks.rados.rados.0.smithi151.stdout:1085: done (8 left) 2024-09-13T14:25:15.907 INFO:tasks.rados.rados.0.smithi151.stdout:1086: done (7 left) 2024-09-13T14:25:15.907 INFO:tasks.rados.rados.0.smithi151.stdout:1087: done (6 left) 2024-09-13T14:25:15.907 INFO:tasks.rados.rados.0.smithi151.stdout:1089: done (5 left) 2024-09-13T14:25:15.907 INFO:tasks.rados.rados.0.smithi151.stdout:1090: done (4 left) 2024-09-13T14:25:15.908 INFO:tasks.rados.rados.0.smithi151.stdout:1091: done (3 left) 2024-09-13T14:25:15.908 INFO:tasks.rados.rados.0.smithi151.stdout:1094: delete oid 36 current snap is 138 2024-09-13T14:25:15.908 INFO:tasks.rados.rados.0.smithi151.stdout:1088: expect (ObjNum 418 snap 130 seq_num 418) 2024-09-13T14:25:16.088 INFO:tasks.rados.rados.0.smithi151.stdout:1093: finishing rollback tid 0 to smithi15186934-2 2024-09-13T14:25:16.088 INFO:tasks.rados.rados.0.smithi151.stdout:1092: finishing write tid 3 to smithi15186934-48 2024-09-13T14:25:16.088 INFO:tasks.rados.rados.0.smithi151.stdout:1093: finishing rollback tid 1 to smithi15186934-2 2024-09-13T14:25:16.088 INFO:tasks.rados.rados.0.smithi151.stdout:1092: finishing write tid 4 to smithi15186934-48 2024-09-13T14:25:16.088 INFO:tasks.rados.rados.0.smithi151.stdout:1093: finishing rollback tid 2 to smithi15186934-2 2024-09-13T14:25:16.088 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 2 v 380 (ObjNum 377 snap 119 seq_num 377) dirty exists 2024-09-13T14:25:16.088 INFO:tasks.rados.rados.0.smithi151.stdout:1092: finishing write tid 5 to smithi15186934-48 2024-09-13T14:25:16.089 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 48 v 667 (ObjNum 434 snap 138 seq_num 434) dirty exists 2024-09-13T14:25:16.089 INFO:tasks.rados.rados.0.smithi151.stdout:1092: left oid 48 (ObjNum 434 snap 138 seq_num 434) 2024-09-13T14:25:16.089 INFO:tasks.rados.rados.0.smithi151.stdout:1088: done (3 left) 2024-09-13T14:25:16.089 INFO:tasks.rados.rados.0.smithi151.stdout:1092: done (2 left) 2024-09-13T14:25:16.089 INFO:tasks.rados.rados.0.smithi151.stdout:1093: done (1 left) 2024-09-13T14:25:16.089 INFO:tasks.rados.rados.0.smithi151.stdout:1094: done (0 left) 2024-09-13T14:25:16.089 INFO:tasks.rados.rados.0.smithi151.stdout:1095: rollback oid 7 current snap is 138 2024-09-13T14:25:16.089 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 7 to 130 2024-09-13T14:25:16.089 INFO:tasks.rados.rados.0.smithi151.stdout:1096: snap_create 2024-09-13T14:25:16.090 INFO:tasks.rados.rados.0.smithi151.stdout:1095: finishing rollback tid 0 to smithi15186934-7 2024-09-13T14:25:16.090 INFO:tasks.rados.rados.0.smithi151.stdout:1095: finishing rollback tid 1 to smithi15186934-7 2024-09-13T14:25:16.091 INFO:tasks.rados.rados.0.smithi151.stdout:1095: finishing rollback tid 2 to smithi15186934-7 2024-09-13T14:25:16.091 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 732 (ObjNum 396 snap 123 seq_num 396) dirty exists 2024-09-13T14:25:16.103 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:25:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:25:15.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:25:16.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:25:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:25:15.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:25:16.648 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:16 smithi151 ceph-mon[104450]: pgmap v114: 89 pgs: 89 active+clean; 212 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.3 MiB/s rd, 0 B/s wr, 4 op/s 2024-09-13T14:25:16.648 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:16 smithi151 ceph-mon[104450]: osdmap e334: 8 total, 8 up, 8 in 2024-09-13T14:25:16.648 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:16.648 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:16.648 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:25:16.649 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:16 smithi151 ceph-mon[41387]: pgmap v114: 89 pgs: 89 active+clean; 212 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.3 MiB/s rd, 0 B/s wr, 4 op/s 2024-09-13T14:25:16.649 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:16 smithi151 ceph-mon[41387]: osdmap e334: 8 total, 8 up, 8 in 2024-09-13T14:25:16.649 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:16 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:16.649 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:16 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:16.649 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:16 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:25:16.795 INFO:tasks.rados.rados.0.smithi151.stdout:1095: done (1 left) 2024-09-13T14:25:16.796 INFO:tasks.rados.rados.0.smithi151.stdout:1096: done (0 left) 2024-09-13T14:25:16.796 INFO:tasks.rados.rados.0.smithi151.stdout:1097: rollback oid 11 current snap is 139 2024-09-13T14:25:16.796 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 11 to 107 2024-09-13T14:25:16.796 INFO:tasks.rados.rados.0.smithi151.stdout:1098: snap_remove snap 131 2024-09-13T14:25:16.900 INFO:teuthology.orchestra.run.smithi151.stdout:wait for majority of mons to upgrade 2024-09-13T14:25:16.949 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:16 smithi184 ceph-mon[33394]: pgmap v114: 89 pgs: 89 active+clean; 212 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.3 MiB/s rd, 0 B/s wr, 4 op/s 2024-09-13T14:25:16.949 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:16 smithi184 ceph-mon[33394]: osdmap e334: 8 total, 8 up, 8 in 2024-09-13T14:25:16.949 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:16 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:16.950 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:16 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:16.950 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:16 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:25:17.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:25:16 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:25:16.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi184\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:25:17.638 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'while ! ceph mon versions | grep $sha1 | egrep '"'"': [23]'"'"' && ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error; do sleep 2 ; done' 2024-09-13T14:25:17.805 INFO:tasks.rados.rados.0.smithi151.stdout:1098: done (1 left) 2024-09-13T14:25:17.805 INFO:tasks.rados.rados.0.smithi151.stdout:1099: write oid 26 current snap is 139 2024-09-13T14:25:17.805 INFO:tasks.rados.rados.0.smithi151.stdout:1097: finishing rollback tid 0 to smithi15186934-11 2024-09-13T14:25:17.805 INFO:tasks.rados.rados.0.smithi151.stdout:1097: finishing rollback tid 1 to smithi15186934-11 2024-09-13T14:25:17.805 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 11 v 295 (ObjNum -1202093696 snap 22067 seq_num 304) dirty dne 2024-09-13T14:25:17.806 INFO:tasks.rados.rados.0.smithi151.stdout:1099: seq_num 436 ranges {527852=640685,1680502=589135,2757013=570839} 2024-09-13T14:25:17.811 INFO:tasks.rados.rados.0.smithi151.stdout:1099: writing smithi15186934-26 from 527852 to 1168537 tid 1 2024-09-13T14:25:17.817 INFO:tasks.rados.rados.0.smithi151.stdout:1099: writing smithi15186934-26 from 1680502 to 2269637 tid 2 2024-09-13T14:25:17.824 INFO:tasks.rados.rados.0.smithi151.stdout:1099: writing smithi15186934-26 from 2757013 to 3327852 tid 3 2024-09-13T14:25:17.824 INFO:tasks.rados.rados.0.smithi151.stdout:1097: done (1 left) 2024-09-13T14:25:17.824 INFO:tasks.rados.rados.0.smithi151.stdout:1100: read oid 30 snap -1 2024-09-13T14:25:17.824 INFO:tasks.rados.rados.0.smithi151.stdout:1100: expect (ObjNum 401 snap 123 seq_num 401) 2024-09-13T14:25:17.824 INFO:tasks.rados.rados.0.smithi151.stdout:1101: write oid 10 current snap is 139 2024-09-13T14:25:17.824 INFO:tasks.rados.rados.0.smithi151.stdout:1099: finishing write tid 1 to smithi15186934-26 2024-09-13T14:25:17.824 INFO:tasks.rados.rados.0.smithi151.stdout:1101: seq_num 437 ranges {544659=715934,1738984=449146,2784436=560223} 2024-09-13T14:25:17.830 INFO:tasks.rados.rados.0.smithi151.stdout:1101: writing smithi15186934-10 from 544659 to 1260593 tid 1 2024-09-13T14:25:17.836 INFO:tasks.rados.rados.0.smithi151.stdout:1101: writing smithi15186934-10 from 1738984 to 2188130 tid 2 2024-09-13T14:25:17.843 INFO:tasks.rados.rados.0.smithi151.stdout:1101: writing smithi15186934-10 from 2784436 to 3344659 tid 3 2024-09-13T14:25:17.843 INFO:tasks.rados.rados.0.smithi151.stdout:1102: write oid 5 current snap is 139 2024-09-13T14:25:17.843 INFO:tasks.rados.rados.0.smithi151.stdout:1099: finishing write tid 2 to smithi15186934-26 2024-09-13T14:25:17.843 INFO:tasks.rados.rados.0.smithi151.stdout:1099: finishing write tid 3 to smithi15186934-26 2024-09-13T14:25:17.843 INFO:tasks.rados.rados.0.smithi151.stdout:1102: seq_num 438 ranges {561466=791183,2081113=592804,3095506=265960} 2024-09-13T14:25:17.850 INFO:tasks.rados.rados.0.smithi151.stdout:1102: writing smithi15186934-5 from 561466 to 1352649 tid 1 2024-09-13T14:25:17.857 INFO:tasks.rados.rados.0.smithi151.stdout:1102: writing smithi15186934-5 from 2081113 to 2673917 tid 2 2024-09-13T14:25:17.862 INFO:tasks.rados.rados.0.smithi151.stdout:1102: writing smithi15186934-5 from 3095506 to 3361466 tid 3 2024-09-13T14:25:17.862 INFO:tasks.rados.rados.0.smithi151.stdout:1103: read oid 7 snap -1 2024-09-13T14:25:17.862 INFO:tasks.rados.rados.0.smithi151.stdout:1103: expect (ObjNum 396 snap 123 seq_num 396) 2024-09-13T14:25:17.862 INFO:tasks.rados.rados.0.smithi151.stdout:1104: rollback oid 24 current snap is 139 2024-09-13T14:25:17.862 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 24 to 124 2024-09-13T14:25:17.862 INFO:tasks.rados.rados.0.smithi151.stdout:1099: finishing write tid 4 to smithi15186934-26 2024-09-13T14:25:17.862 INFO:tasks.rados.rados.0.smithi151.stdout:1099: finishing write tid 5 to smithi15186934-26 2024-09-13T14:25:17.862 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 344 (ObjNum 436 snap 139 seq_num 436) dirty exists 2024-09-13T14:25:17.862 INFO:tasks.rados.rados.0.smithi151.stdout:1099: left oid 26 (ObjNum 436 snap 139 seq_num 436) 2024-09-13T14:25:17.863 INFO:tasks.rados.rados.0.smithi151.stdout:1100: expect (ObjNum 401 snap 123 seq_num 401) 2024-09-13T14:25:17.955 INFO:tasks.rados.rados.0.smithi151.stdout:1101: finishing write tid 1 to smithi15186934-10 2024-09-13T14:25:17.956 INFO:tasks.rados.rados.0.smithi151.stdout:1101: finishing write tid 2 to smithi15186934-10 2024-09-13T14:25:17.956 INFO:tasks.rados.rados.0.smithi151.stdout:1099: done (5 left) 2024-09-13T14:25:17.956 INFO:tasks.rados.rados.0.smithi151.stdout:1100: done (4 left) 2024-09-13T14:25:17.956 INFO:tasks.rados.rados.0.smithi151.stdout:1105: delete oid 14 current snap is 139 2024-09-13T14:25:17.956 INFO:tasks.rados.rados.0.smithi151.stdout:1101: finishing write tid 3 to smithi15186934-10 2024-09-13T14:25:17.956 INFO:tasks.rados.rados.0.smithi151.stdout:1101: finishing write tid 4 to smithi15186934-10 2024-09-13T14:25:17.956 INFO:tasks.rados.rados.0.smithi151.stdout:1101: finishing write tid 5 to smithi15186934-10 2024-09-13T14:25:17.956 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 349 (ObjNum 437 snap 139 seq_num 437) dirty exists 2024-09-13T14:25:17.956 INFO:tasks.rados.rados.0.smithi151.stdout:1101: left oid 10 (ObjNum 437 snap 139 seq_num 437) 2024-09-13T14:25:17.956 INFO:tasks.rados.rados.0.smithi151.stdout:1102: finishing write tid 1 to smithi15186934-5 2024-09-13T14:25:17.956 INFO:tasks.rados.rados.0.smithi151.stdout:1102: finishing write tid 2 to smithi15186934-5 2024-09-13T14:25:17.957 INFO:tasks.rados.rados.0.smithi151.stdout:1102: finishing write tid 3 to smithi15186934-5 2024-09-13T14:25:17.957 INFO:tasks.rados.rados.0.smithi151.stdout:1102: finishing write tid 4 to smithi15186934-5 2024-09-13T14:25:17.957 INFO:tasks.rados.rados.0.smithi151.stdout:1102: finishing write tid 5 to smithi15186934-5 2024-09-13T14:25:17.957 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 386 (ObjNum 438 snap 139 seq_num 438) dirty exists 2024-09-13T14:25:17.957 INFO:tasks.rados.rados.0.smithi151.stdout:1102: left oid 5 (ObjNum 438 snap 139 seq_num 438) 2024-09-13T14:25:17.957 INFO:tasks.rados.rados.0.smithi151.stdout:1103: expect (ObjNum 396 snap 123 seq_num 396) 2024-09-13T14:25:18.044 INFO:tasks.rados.rados.0.smithi151.stdout:1104: finishing rollback tid 0 to smithi15186934-24 2024-09-13T14:25:18.045 INFO:tasks.rados.rados.0.smithi151.stdout:1104: finishing rollback tid 1 to smithi15186934-24 2024-09-13T14:25:18.045 INFO:tasks.rados.rados.0.smithi151.stdout:1104: finishing rollback tid 2 to smithi15186934-24 2024-09-13T14:25:18.045 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 24 v 742 (ObjNum 391 snap 121 seq_num 391) dirty exists 2024-09-13T14:25:18.045 INFO:tasks.rados.rados.0.smithi151.stdout:1101: done (4 left) 2024-09-13T14:25:18.045 INFO:tasks.rados.rados.0.smithi151.stdout:1102: done (3 left) 2024-09-13T14:25:18.045 INFO:tasks.rados.rados.0.smithi151.stdout:1103: done (2 left) 2024-09-13T14:25:18.045 INFO:tasks.rados.rados.0.smithi151.stdout:1104: done (1 left) 2024-09-13T14:25:18.045 INFO:tasks.rados.rados.0.smithi151.stdout:1105: done (0 left) 2024-09-13T14:25:18.045 INFO:tasks.rados.rados.0.smithi151.stdout:1106: write oid 14 current snap is 139 2024-09-13T14:25:18.045 INFO:tasks.rados.rados.0.smithi151.stdout:1106: seq_num 440 ranges {595080=541681,1798077=596473,3150352=244728} 2024-09-13T14:25:18.048 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:17 smithi151 ceph-mon[104450]: osdmap e335: 8 total, 8 up, 8 in 2024-09-13T14:25:18.048 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:17 smithi151 ceph-mon[41387]: osdmap e335: 8 total, 8 up, 8 in 2024-09-13T14:25:18.050 INFO:tasks.rados.rados.0.smithi151.stdout:1106: writing smithi15186934-14 from 595080 to 1136761 tid 1 2024-09-13T14:25:18.057 INFO:tasks.rados.rados.0.smithi151.stdout:1106: writing smithi15186934-14 from 1798077 to 2394550 tid 2 2024-09-13T14:25:18.064 INFO:tasks.rados.rados.0.smithi151.stdout:1106: writing smithi15186934-14 from 3150352 to 3395080 tid 3 2024-09-13T14:25:18.064 INFO:tasks.rados.rados.0.smithi151.stdout:1107: write oid 31 current snap is 139 2024-09-13T14:25:18.064 INFO:tasks.rados.rados.0.smithi151.stdout:1106: finishing write tid 1 to smithi15186934-14 2024-09-13T14:25:18.064 INFO:tasks.rados.rados.0.smithi151.stdout:1106: finishing write tid 2 to smithi15186934-14 2024-09-13T14:25:18.064 INFO:tasks.rados.rados.0.smithi151.stdout:1107: seq_num 441 ranges {611887=733283,1972912=456484,2894128=517759} 2024-09-13T14:25:18.072 INFO:tasks.rados.rados.0.smithi151.stdout:1107: writing smithi15186934-31 from 611887 to 1345170 tid 1 2024-09-13T14:25:18.078 INFO:tasks.rados.rados.0.smithi151.stdout:1107: writing smithi15186934-31 from 1972912 to 2429396 tid 2 2024-09-13T14:25:18.084 INFO:tasks.rados.rados.0.smithi151.stdout:1107: writing smithi15186934-31 from 2894128 to 3411887 tid 3 2024-09-13T14:25:18.084 INFO:tasks.rados.rados.0.smithi151.stdout:1108: snap_create 2024-09-13T14:25:18.084 INFO:tasks.rados.rados.0.smithi151.stdout:1106: finishing write tid 3 to smithi15186934-14 2024-09-13T14:25:18.084 INFO:tasks.rados.rados.0.smithi151.stdout:1106: finishing write tid 4 to smithi15186934-14 2024-09-13T14:25:18.084 INFO:tasks.rados.rados.0.smithi151.stdout:1106: finishing write tid 5 to smithi15186934-14 2024-09-13T14:25:18.084 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 464 (ObjNum 440 snap 139 seq_num 440) dirty exists 2024-09-13T14:25:18.085 INFO:tasks.rados.rados.0.smithi151.stdout:1106: left oid 14 (ObjNum 440 snap 139 seq_num 440) 2024-09-13T14:25:18.085 INFO:tasks.rados.rados.0.smithi151.stdout:1107: finishing write tid 1 to smithi15186934-31 2024-09-13T14:25:18.085 INFO:tasks.rados.rados.0.smithi151.stdout:1107: finishing write tid 2 to smithi15186934-31 2024-09-13T14:25:18.086 INFO:tasks.rados.rados.0.smithi151.stdout:1107: finishing write tid 3 to smithi15186934-31 2024-09-13T14:25:18.086 INFO:tasks.rados.rados.0.smithi151.stdout:1107: finishing write tid 4 to smithi15186934-31 2024-09-13T14:25:18.086 INFO:tasks.rados.rados.0.smithi151.stdout:1107: finishing write tid 5 to smithi15186934-31 2024-09-13T14:25:18.086 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 31 v 577 (ObjNum 441 snap 139 seq_num 441) dirty exists 2024-09-13T14:25:18.086 INFO:tasks.rados.rados.0.smithi151.stdout:1107: left oid 31 (ObjNum 441 snap 139 seq_num 441) 2024-09-13T14:25:18.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:17 smithi184 ceph-mon[33394]: osdmap e335: 8 total, 8 up, 8 in 2024-09-13T14:25:18.547 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:25:18.810 INFO:tasks.rados.rados.0.smithi151.stdout:1106: done (2 left) 2024-09-13T14:25:18.810 INFO:tasks.rados.rados.0.smithi151.stdout:1107: done (1 left) 2024-09-13T14:25:18.810 INFO:tasks.rados.rados.0.smithi151.stdout:1108: done (0 left) 2024-09-13T14:25:18.811 INFO:tasks.rados.rados.0.smithi151.stdout:1109: read oid 45 snap -1 2024-09-13T14:25:18.811 INFO:tasks.rados.rados.0.smithi151.stdout:1109: expect (ObjNum 410 snap 127 seq_num 410) 2024-09-13T14:25:18.811 INFO:tasks.rados.rados.0.smithi151.stdout:1110: snap_create 2024-09-13T14:25:18.825 INFO:tasks.rados.rados.0.smithi151.stdout:1109: expect (ObjNum 410 snap 127 seq_num 410) 2024-09-13T14:25:19.088 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:18 smithi151 ceph-mon[104450]: pgmap v117: 89 pgs: 89 active+clean; 228 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 1011 KiB/s wr, 18 op/s 2024-09-13T14:25:19.088 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:18 smithi151 ceph-mon[104450]: osdmap e336: 8 total, 8 up, 8 in 2024-09-13T14:25:19.088 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:18 smithi151 ceph-mon[41387]: pgmap v117: 89 pgs: 89 active+clean; 228 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 1011 KiB/s wr, 18 op/s 2024-09-13T14:25:19.088 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:18 smithi151 ceph-mon[41387]: osdmap e336: 8 total, 8 up, 8 in 2024-09-13T14:25:19.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:18 smithi184 ceph-mon[33394]: pgmap v117: 89 pgs: 89 active+clean; 228 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 1011 KiB/s wr, 18 op/s 2024-09-13T14:25:19.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:18 smithi184 ceph-mon[33394]: osdmap e336: 8 total, 8 up, 8 in 2024-09-13T14:25:19.825 INFO:tasks.rados.rados.0.smithi151.stdout:1109: done (1 left) 2024-09-13T14:25:19.825 INFO:tasks.rados.rados.0.smithi151.stdout:1110: done (0 left) 2024-09-13T14:25:19.826 INFO:tasks.rados.rados.0.smithi151.stdout:1111: read oid 37 snap -1 2024-09-13T14:25:19.826 INFO:tasks.rados.rados.0.smithi151.stdout:1111: expect deleted 2024-09-13T14:25:19.826 INFO:tasks.rados.rados.0.smithi151.stdout:1112: rollback oid 38 current snap is 141 2024-09-13T14:25:19.826 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 38 to 138 2024-09-13T14:25:19.826 INFO:tasks.rados.rados.0.smithi151.stdout:1113: write oid 1 current snap is 141 2024-09-13T14:25:19.826 INFO:tasks.rados.rados.0.smithi151.stdout:1113: seq_num 442 ranges {628694=408532,1515041=600142,2805198=552112} 2024-09-13T14:25:19.831 INFO:tasks.rados.rados.0.smithi151.stdout:1113: writing smithi15186934-1 from 628694 to 1037226 tid 1 2024-09-13T14:25:19.837 INFO:tasks.rados.rados.0.smithi151.stdout:1113: writing smithi15186934-1 from 1515041 to 2115183 tid 2 2024-09-13T14:25:19.844 INFO:tasks.rados.rados.0.smithi151.stdout:1113: writing smithi15186934-1 from 2805198 to 3357310 tid 3 2024-09-13T14:25:19.844 INFO:tasks.rados.rados.0.smithi151.stdout:1114: delete oid 7 current snap is 141 2024-09-13T14:25:19.844 INFO:tasks.rados.rados.0.smithi151.stdout:1112: finishing rollback tid 0 to smithi15186934-38 2024-09-13T14:25:19.844 INFO:tasks.rados.rados.0.smithi151.stdout:1112: finishing rollback tid 1 to smithi15186934-38 2024-09-13T14:25:19.844 INFO:tasks.rados.rados.0.smithi151.stdout:1112: finishing rollback tid 2 to smithi15186934-38 2024-09-13T14:25:19.844 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 341 (ObjNum 423 snap 132 seq_num 423) dirty exists 2024-09-13T14:25:19.844 INFO:tasks.rados.rados.0.smithi151.stdout:1113: finishing write tid 1 to smithi15186934-1 2024-09-13T14:25:19.844 INFO:tasks.rados.rados.0.smithi151.stdout:1113: finishing write tid 2 to smithi15186934-1 2024-09-13T14:25:19.846 INFO:tasks.rados.rados.0.smithi151.stdout:1111: done (3 left) 2024-09-13T14:25:19.846 INFO:tasks.rados.rados.0.smithi151.stdout:1112: done (2 left) 2024-09-13T14:25:19.846 INFO:tasks.rados.rados.0.smithi151.stdout:1114: done (1 left) 2024-09-13T14:25:19.846 INFO:tasks.rados.rados.0.smithi151.stdout:1115: rollback oid 48 current snap is 141 2024-09-13T14:25:19.846 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 48 to 107 2024-09-13T14:25:19.846 INFO:tasks.rados.rados.0.smithi151.stdout:1116: write oid 5 current snap is 141 2024-09-13T14:25:19.846 INFO:tasks.rados.rados.0.smithi151.stdout:1116: seq_num 444 ranges {662308=559030,1632005=603811,2860044=602264} 2024-09-13T14:25:19.853 INFO:tasks.rados.rados.0.smithi151.stdout:1116: writing smithi15186934-5 from 662308 to 1221338 tid 1 2024-09-13T14:25:19.858 INFO:tasks.rados.rados.0.smithi151.stdout:1116: writing smithi15186934-5 from 1632005 to 2235816 tid 2 2024-09-13T14:25:19.865 INFO:tasks.rados.rados.0.smithi151.stdout:1116: writing smithi15186934-5 from 2860044 to 3462308 tid 3 2024-09-13T14:25:19.865 INFO:tasks.rados.rados.0.smithi151.stdout:1117: write oid 29 current snap is 141 2024-09-13T14:25:19.865 INFO:tasks.rados.rados.0.smithi151.stdout:1113: finishing write tid 3 to smithi15186934-1 2024-09-13T14:25:19.865 INFO:tasks.rados.rados.0.smithi151.stdout:1117: seq_num 445 ranges {679115=634279,2090487=463822,3003820=475295} 2024-09-13T14:25:19.872 INFO:tasks.rados.rados.0.smithi151.stdout:1117: writing smithi15186934-29 from 679115 to 1313394 tid 1 2024-09-13T14:25:19.879 INFO:tasks.rados.rados.0.smithi151.stdout:1117: writing smithi15186934-29 from 2090487 to 2554309 tid 2 2024-09-13T14:25:19.885 INFO:tasks.rados.rados.0.smithi151.stdout:1117: writing smithi15186934-29 from 3003820 to 3479115 tid 3 2024-09-13T14:25:19.885 INFO:tasks.rados.rados.0.smithi151.stdout:1118: read oid 15 snap -1 2024-09-13T14:25:19.885 INFO:tasks.rados.rados.0.smithi151.stdout:1118: expect (ObjNum 413 snap 128 seq_num 413) 2024-09-13T14:25:19.885 INFO:tasks.rados.rados.0.smithi151.stdout:1119: read oid 16 snap -1 2024-09-13T14:25:19.885 INFO:tasks.rados.rados.0.smithi151.stdout:1119: expect (ObjNum 420 snap 131 seq_num 420) 2024-09-13T14:25:19.885 INFO:tasks.rados.rados.0.smithi151.stdout:1120: snap_create 2024-09-13T14:25:19.885 INFO:tasks.rados.rados.0.smithi151.stdout:1113: finishing write tid 4 to smithi15186934-1 2024-09-13T14:25:19.886 INFO:tasks.rados.rados.0.smithi151.stdout:1113: finishing write tid 5 to smithi15186934-1 2024-09-13T14:25:19.886 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 1 v 293 (ObjNum 442 snap 141 seq_num 442) dirty exists 2024-09-13T14:25:19.886 INFO:tasks.rados.rados.0.smithi151.stdout:1113: left oid 1 (ObjNum 442 snap 141 seq_num 442) 2024-09-13T14:25:19.886 INFO:tasks.rados.rados.0.smithi151.stdout:1115: finishing rollback tid 0 to smithi15186934-48 2024-09-13T14:25:19.886 INFO:tasks.rados.rados.0.smithi151.stdout:1115: finishing rollback tid 1 to smithi15186934-48 2024-09-13T14:25:19.886 INFO:tasks.rados.rados.0.smithi151.stdout:1115: finishing rollback tid 2 to smithi15186934-48 2024-09-13T14:25:19.886 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 48 v 671 (ObjNum 342 snap 105 seq_num 342) dirty exists 2024-09-13T14:25:19.886 INFO:tasks.rados.rados.0.smithi151.stdout:1116: finishing write tid 1 to smithi15186934-5 2024-09-13T14:25:19.886 INFO:tasks.rados.rados.0.smithi151.stdout:1116: finishing write tid 2 to smithi15186934-5 2024-09-13T14:25:19.886 INFO:tasks.rados.rados.0.smithi151.stdout:1116: finishing write tid 3 to smithi15186934-5 2024-09-13T14:25:19.886 INFO:tasks.rados.rados.0.smithi151.stdout:1116: finishing write tid 4 to smithi15186934-5 2024-09-13T14:25:19.886 INFO:tasks.rados.rados.0.smithi151.stdout:1116: finishing write tid 5 to smithi15186934-5 2024-09-13T14:25:19.887 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 391 (ObjNum 444 snap 141 seq_num 444) dirty exists 2024-09-13T14:25:19.887 INFO:tasks.rados.rados.0.smithi151.stdout:1116: left oid 5 (ObjNum 444 snap 141 seq_num 444) 2024-09-13T14:25:19.887 INFO:tasks.rados.rados.0.smithi151.stdout:1117: finishing write tid 1 to smithi15186934-29 2024-09-13T14:25:19.887 INFO:tasks.rados.rados.0.smithi151.stdout:1117: finishing write tid 2 to smithi15186934-29 2024-09-13T14:25:19.888 INFO:tasks.rados.rados.0.smithi151.stdout:1117: finishing write tid 3 to smithi15186934-29 2024-09-13T14:25:19.888 INFO:tasks.rados.rados.0.smithi151.stdout:1117: finishing write tid 4 to smithi15186934-29 2024-09-13T14:25:19.888 INFO:tasks.rados.rados.0.smithi151.stdout:1117: finishing write tid 5 to smithi15186934-29 2024-09-13T14:25:19.888 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 29 v 301 (ObjNum 445 snap 141 seq_num 445) dirty exists 2024-09-13T14:25:19.888 INFO:tasks.rados.rados.0.smithi151.stdout:1117: left oid 29 (ObjNum 445 snap 141 seq_num 445) 2024-09-13T14:25:19.893 INFO:tasks.rados.rados.0.smithi151.stdout:1118: expect (ObjNum 413 snap 128 seq_num 413) 2024-09-13T14:25:20.054 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:19 smithi151 ceph-mon[104450]: osdmap e337: 8 total, 8 up, 8 in 2024-09-13T14:25:20.054 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:19 smithi151 ceph-mon[41387]: osdmap e337: 8 total, 8 up, 8 in 2024-09-13T14:25:20.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:19 smithi184 ceph-mon[33394]: osdmap e337: 8 total, 8 up, 8 in 2024-09-13T14:25:20.197 INFO:tasks.rados.rados.0.smithi151.stdout:1119: expect (ObjNum 420 snap 131 seq_num 420) 2024-09-13T14:25:20.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:25:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:25:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:25:20.827 INFO:tasks.rados.rados.0.smithi151.stdout:1113: done (6 left) 2024-09-13T14:25:20.828 INFO:tasks.rados.rados.0.smithi151.stdout:1115: done (5 left) 2024-09-13T14:25:20.828 INFO:tasks.rados.rados.0.smithi151.stdout:1116: done (4 left) 2024-09-13T14:25:20.828 INFO:tasks.rados.rados.0.smithi151.stdout:1117: done (3 left) 2024-09-13T14:25:20.828 INFO:tasks.rados.rados.0.smithi151.stdout:1118: done (2 left) 2024-09-13T14:25:20.828 INFO:tasks.rados.rados.0.smithi151.stdout:1119: done (1 left) 2024-09-13T14:25:20.828 INFO:tasks.rados.rados.0.smithi151.stdout:1120: done (0 left) 2024-09-13T14:25:20.828 INFO:tasks.rados.rados.0.smithi151.stdout:1121: rollback oid 34 current snap is 142 2024-09-13T14:25:20.828 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 34 to 141 2024-09-13T14:25:20.828 INFO:tasks.rados.rados.0.smithi151.stdout:1122: snap_create 2024-09-13T14:25:20.830 INFO:tasks.rados.rados.0.smithi151.stdout:1121: finishing rollback tid 1 to smithi15186934-34 2024-09-13T14:25:20.830 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 383 (ObjNum 0 snap 0 seq_num 304) dirty dne 2024-09-13T14:25:21.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:20 smithi151 ceph-mon[104450]: pgmap v120: 89 pgs: 89 active+clean; 240 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 28 MiB/s rd, 2.8 MiB/s wr, 37 op/s 2024-09-13T14:25:21.066 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:20 smithi151 ceph-mon[104450]: osdmap e338: 8 total, 8 up, 8 in 2024-09-13T14:25:21.066 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:20 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:21.066 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:20 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:21.066 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:20 smithi151 ceph-mon[41387]: pgmap v120: 89 pgs: 89 active+clean; 240 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 28 MiB/s rd, 2.8 MiB/s wr, 37 op/s 2024-09-13T14:25:21.066 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:20 smithi151 ceph-mon[41387]: osdmap e338: 8 total, 8 up, 8 in 2024-09-13T14:25:21.066 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:20 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:21.066 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:20 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:21.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:20 smithi184 ceph-mon[33394]: pgmap v120: 89 pgs: 89 active+clean; 240 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 28 MiB/s rd, 2.8 MiB/s wr, 37 op/s 2024-09-13T14:25:21.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:20 smithi184 ceph-mon[33394]: osdmap e338: 8 total, 8 up, 8 in 2024-09-13T14:25:21.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:20 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:21.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:20 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:21.566 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:25:21.833 INFO:tasks.rados.rados.0.smithi151.stdout:1121: done (1 left) 2024-09-13T14:25:21.833 INFO:tasks.rados.rados.0.smithi151.stdout:1122: done (0 left) 2024-09-13T14:25:21.833 INFO:tasks.rados.rados.0.smithi151.stdout:1123: snap_create 2024-09-13T14:25:22.089 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:21 smithi151 ceph-mon[104450]: osdmap e339: 8 total, 8 up, 8 in 2024-09-13T14:25:22.090 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:21 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:25:22.090 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:21 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:25:22.090 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:21 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:22.090 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:21 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:25:22.090 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:21 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:25:22.090 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:21 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-13T14:25:22.090 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:21 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["c"]} : dispatch 2024-09-13T14:25:22.090 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:21 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/535544457' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-13T14:25:22.090 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:21 smithi151 ceph-mon[41387]: osdmap e339: 8 total, 8 up, 8 in 2024-09-13T14:25:22.090 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:21 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:25:22.091 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:21 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:25:22.091 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:21 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:22.091 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:21 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:25:22.091 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:21 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:25:22.091 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:21 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-13T14:25:22.091 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:21 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["c"]} : dispatch 2024-09-13T14:25:22.091 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:21 smithi151 ceph-mon[41387]: from='client.? 172.21.15.151:0/535544457' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-13T14:25:22.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:21 smithi184 ceph-mon[33394]: osdmap e339: 8 total, 8 up, 8 in 2024-09-13T14:25:22.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:21 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:25:22.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:21 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:25:22.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:21 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:22.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:21 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:25:22.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:21 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:25:22.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:21 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-13T14:25:22.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:21 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["c"]} : dispatch 2024-09-13T14:25:22.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:21 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/535544457' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-13T14:25:22.852 INFO:tasks.rados.rados.0.smithi151.stdout:1123: done (0 left) 2024-09-13T14:25:22.852 INFO:tasks.rados.rados.0.smithi151.stdout:1124: write oid 16 current snap is 144 2024-09-13T14:25:22.852 INFO:tasks.rados.rados.0.smithi151.stdout:1124: seq_num 446 ranges {695922=709528,2032616=607480,3198537=297385} 2024-09-13T14:25:22.860 INFO:tasks.rados.rados.0.smithi151.stdout:1124: writing smithi15186934-16 from 695922 to 1405450 tid 1 2024-09-13T14:25:22.866 INFO:tasks.rados.rados.0.smithi151.stdout:1124: writing smithi15186934-16 from 2032616 to 2640096 tid 2 2024-09-13T14:25:22.871 INFO:tasks.rados.rados.0.smithi151.stdout:1124: writing smithi15186934-16 from 3198537 to 3495922 tid 3 2024-09-13T14:25:22.871 INFO:tasks.rados.rados.0.smithi151.stdout:1125: read oid 3 snap -1 2024-09-13T14:25:22.871 INFO:tasks.rados.rados.0.smithi151.stdout:1125: expect (ObjNum 334 snap 103 seq_num 334) 2024-09-13T14:25:22.871 INFO:tasks.rados.rados.0.smithi151.stdout:1126: snap_remove snap 138 2024-09-13T14:25:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:22 smithi184 ceph-mon[33394]: Upgrade: It appears safe to stop mon.c 2024-09-13T14:25:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:22 smithi184 ceph-mon[33394]: pgmap v123: 89 pgs: 89 active+clean; 240 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 1.8 MiB/s wr, 18 op/s 2024-09-13T14:25:23.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:22 smithi184 ceph-mon[33394]: from='client.26024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:25:23.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:22 smithi184 ceph-mon[33394]: osdmap e340: 8 total, 8 up, 8 in 2024-09-13T14:25:23.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:22 smithi184 ceph-mon[33394]: from='client.26030 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:25:23.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:22 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:23.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:22 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:25:23.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:22 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:25:23.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:22 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:25:23.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:22 smithi151 ceph-mon[104450]: Upgrade: It appears safe to stop mon.c 2024-09-13T14:25:23.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:22 smithi151 ceph-mon[104450]: pgmap v123: 89 pgs: 89 active+clean; 240 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 1.8 MiB/s wr, 18 op/s 2024-09-13T14:25:23.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:22 smithi151 ceph-mon[104450]: from='client.26024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:25:23.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:22 smithi151 ceph-mon[104450]: osdmap e340: 8 total, 8 up, 8 in 2024-09-13T14:25:23.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:22 smithi151 ceph-mon[104450]: from='client.26030 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:25:23.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:22 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:23.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:22 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:25:23.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:22 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:25:23.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:22 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:25:23.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:22 smithi151 ceph-mon[41387]: Upgrade: It appears safe to stop mon.c 2024-09-13T14:25:23.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:22 smithi151 ceph-mon[41387]: pgmap v123: 89 pgs: 89 active+clean; 240 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 1.8 MiB/s wr, 18 op/s 2024-09-13T14:25:23.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:22 smithi151 ceph-mon[41387]: from='client.26024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:25:23.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:22 smithi151 ceph-mon[41387]: osdmap e340: 8 total, 8 up, 8 in 2024-09-13T14:25:23.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:22 smithi151 ceph-mon[41387]: from='client.26030 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:25:23.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:22 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:23.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:22 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:25:23.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:22 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:25:23.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:22 smithi151 ceph-mon[41387]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:25:23.863 INFO:tasks.rados.rados.0.smithi151.stdout:1126: done (2 left) 2024-09-13T14:25:23.863 INFO:tasks.rados.rados.0.smithi151.stdout:1127: read oid 48 snap -1 2024-09-13T14:25:23.863 INFO:tasks.rados.rados.0.smithi151.stdout:1127: expect (ObjNum 342 snap 105 seq_num 342) 2024-09-13T14:25:23.863 INFO:tasks.rados.rados.0.smithi151.stdout:1124: finishing write tid 1 to smithi15186934-16 2024-09-13T14:25:23.864 INFO:tasks.rados.rados.0.smithi151.stdout:1124: finishing write tid 2 to smithi15186934-16 2024-09-13T14:25:23.864 INFO:tasks.rados.rados.0.smithi151.stdout:1128: delete oid 45 current snap is 144 2024-09-13T14:25:23.864 INFO:tasks.rados.rados.0.smithi151.stdout:1124: finishing write tid 3 to smithi15186934-16 2024-09-13T14:25:23.864 INFO:tasks.rados.rados.0.smithi151.stdout:1124: finishing write tid 4 to smithi15186934-16 2024-09-13T14:25:23.864 INFO:tasks.rados.rados.0.smithi151.stdout:1124: finishing write tid 5 to smithi15186934-16 2024-09-13T14:25:23.864 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 16 v 278 (ObjNum 446 snap 144 seq_num 446) dirty exists 2024-09-13T14:25:23.864 INFO:tasks.rados.rados.0.smithi151.stdout:1124: left oid 16 (ObjNum 446 snap 144 seq_num 446) 2024-09-13T14:25:23.864 INFO:tasks.rados.rados.0.smithi151.stdout:1125: expect (ObjNum 334 snap 103 seq_num 334) 2024-09-13T14:25:23.990 INFO:tasks.rados.rados.0.smithi151.stdout:1127: expect (ObjNum 342 snap 105 seq_num 342) 2024-09-13T14:25:24.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:23 smithi151 ceph-mon[104450]: Upgrade: Updating mon.c 2024-09-13T14:25:24.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:23 smithi151 ceph-mon[104450]: Deploying daemon mon.c on smithi151 2024-09-13T14:25:24.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:23 smithi151 ceph-mon[104450]: osdmap e341: 8 total, 8 up, 8 in 2024-09-13T14:25:24.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:23 smithi151 ceph-mon[41387]: Upgrade: Updating mon.c 2024-09-13T14:25:24.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:23 smithi151 ceph-mon[41387]: Deploying daemon mon.c on smithi151 2024-09-13T14:25:24.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:23 smithi151 ceph-mon[41387]: osdmap e341: 8 total, 8 up, 8 in 2024-09-13T14:25:24.147 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:23 smithi184 ceph-mon[33394]: Upgrade: Updating mon.c 2024-09-13T14:25:24.148 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:23 smithi184 ceph-mon[33394]: Deploying daemon mon.c on smithi151 2024-09-13T14:25:24.148 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:23 smithi184 ceph-mon[33394]: osdmap e341: 8 total, 8 up, 8 in 2024-09-13T14:25:24.233 INFO:tasks.rados.rados.0.smithi151.stdout:1124: done (3 left) 2024-09-13T14:25:24.233 INFO:tasks.rados.rados.0.smithi151.stdout:1125: done (2 left) 2024-09-13T14:25:24.233 INFO:tasks.rados.rados.0.smithi151.stdout:1127: done (1 left) 2024-09-13T14:25:24.233 INFO:tasks.rados.rados.0.smithi151.stdout:1128: done (0 left) 2024-09-13T14:25:24.233 INFO:tasks.rados.rados.0.smithi151.stdout:1129: snap_create 2024-09-13T14:25:24.457 INFO:tasks.rados.rados.0.smithi151.stdout:1129: done (0 left) 2024-09-13T14:25:24.457 INFO:tasks.rados.rados.0.smithi151.stdout:1130: delete oid 24 current snap is 145 2024-09-13T14:25:24.459 INFO:tasks.rados.rados.0.smithi151.stdout:1130: done (0 left) 2024-09-13T14:25:24.460 INFO:tasks.rados.rados.0.smithi151.stdout:1131: read oid 43 snap 101 2024-09-13T14:25:24.460 INFO:tasks.rados.rados.0.smithi151.stdout:1131: expect (ObjNum 263 snap 82 seq_num 263) 2024-09-13T14:25:24.460 INFO:tasks.rados.rados.0.smithi151.stdout:1132: delete oid 39 current snap is 145 2024-09-13T14:25:24.464 INFO:tasks.rados.rados.0.smithi151.stdout:1132: done (1 left) 2024-09-13T14:25:24.464 INFO:tasks.rados.rados.0.smithi151.stdout:1133: snap_create 2024-09-13T14:25:24.469 INFO:tasks.rados.rados.0.smithi151.stdout:1131: expect (ObjNum 263 snap 82 seq_num 263) 2024-09-13T14:25:24.502 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:24 smithi151 systemd[1]: Stopping Ceph mon.c for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:25:24.552 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:25:24.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:25:24 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:25:24.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:25:24.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:25:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:25:24] "GET /metrics HTTP/1.1" 200 36399 "" "Prometheus/2.51.0" 2024-09-13T14:25:24.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-c[41364]: 2024-09-13T14:25:24.495+0000 7fb587d11640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:25:24.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-c[41364]: 2024-09-13T14:25:24.495+0000 7fb587d11640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-13T14:25:24.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:24 smithi151 podman[108748]: 2024-09-13 14:25:24.699100042 +0000 UTC m=+0.335794818 container died dccc3a65ea4d02896222322f753527616ca0a95bcac6150ea2558e81799e67b1 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-c, RELEASE=quincy-bbdadfa, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-13T14:25:25.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:25 smithi151 podman[108748]: 2024-09-13 14:25:25.176246262 +0000 UTC m=+0.812941069 container cleanup dccc3a65ea4d02896222322f753527616ca0a95bcac6150ea2558e81799e67b1 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-c, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=quincy-bbdadfa) 2024-09-13T14:25:25.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:25 smithi151 bash[108748]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-c 2024-09-13T14:25:25.644 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:25 smithi151 podman[108791]: 2024-09-13 14:25:25.393682011 +0000 UTC m=+0.694120642 container remove dccc3a65ea4d02896222322f753527616ca0a95bcac6150ea2558e81799e67b1 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-c, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-bbdadfa, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-13T14:25:25.644 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:25 smithi151 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@mon.c.service: Deactivated successfully. 2024-09-13T14:25:25.644 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:25 smithi151 systemd[1]: Stopped Ceph mon.c for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:25:25.644 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:25 smithi151 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@mon.c.service: Consumed 8.773s CPU time. 2024-09-13T14:25:26.057 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:25 smithi151 systemd[1]: Starting Ceph mon.c for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:25:26.103 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:25:25 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:25:25.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:25:26.103 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:25:25 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:25:25.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:25:26.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 podman[108928]: 2024-09-13 14:25:25.958721763 +0000 UTC m=+0.015677367 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:25:26.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 podman[108928]: 2024-09-13 14:25:26.162418722 +0000 UTC m=+0.219374322 container create e548fceed2f9a4804e53d2f49ccc63cb9cc0cd3fb315e9189c6066ba685cf405 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-c, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-13T14:25:26.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 podman[108928]: 2024-09-13 14:25:26.373305248 +0000 UTC m=+0.430260853 container init e548fceed2f9a4804e53d2f49ccc63cb9cc0cd3fb315e9189c6066ba685cf405 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-c, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-13T14:25:26.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 podman[108928]: 2024-09-13 14:25:26.378806766 +0000 UTC m=+0.435762370 container start e548fceed2f9a4804e53d2f49ccc63cb9cc0cd3fb315e9189c6066ba685cf405 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-c, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:25:26.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: set uid:gid to 167:167 (ceph:ceph) 2024-09-13T14:25:26.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev), process ceph-mon, pid 7 2024-09-13T14:25:26.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: pidfile_write: ignore empty --pid-file 2024-09-13T14:25:26.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: load: jerasure load: lrc 2024-09-13T14:25:26.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: RocksDB version: 7.9.2 2024-09-13T14:25:26.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Git sha 0 2024-09-13T14:25:26.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Compile date 2024-09-12 20:46:14 2024-09-13T14:25:26.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: DB SUMMARY 2024-09-13T14:25:26.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: DB Session ID: 21I9A7A114IWKIFHNUY1 2024-09-13T14:25:26.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: CURRENT file: CURRENT 2024-09-13T14:25:26.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: IDENTITY file: IDENTITY 2024-09-13T14:25:26.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: MANIFEST file: MANIFEST-000009 size: 897 Bytes 2024-09-13T14:25:26.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000024.sst 2024-09-13T14:25:26.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000022.log size: 11955846 ; 2024-09-13T14:25:26.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.error_if_exists: 0 2024-09-13T14:25:26.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.create_if_missing: 0 2024-09-13T14:25:26.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.paranoid_checks: 1 2024-09-13T14:25:26.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-13T14:25:26.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-13T14:25:26.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-13T14:25:26.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.env: 0x55b33d13f160 2024-09-13T14:25:26.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.fs: PosixFileSystem 2024-09-13T14:25:26.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.info_log: 0x55b33ed3b880 2024-09-13T14:25:26.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.max_file_opening_threads: 16 2024-09-13T14:25:26.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.statistics: (nil) 2024-09-13T14:25:26.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.use_fsync: 0 2024-09-13T14:25:26.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.max_log_file_size: 0 2024-09-13T14:25:26.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-13T14:25:26.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-13T14:25:26.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.keep_log_file_num: 1000 2024-09-13T14:25:26.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.recycle_log_file_num: 0 2024-09-13T14:25:26.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.allow_fallocate: 1 2024-09-13T14:25:26.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.allow_mmap_reads: 0 2024-09-13T14:25:26.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.allow_mmap_writes: 0 2024-09-13T14:25:26.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.use_direct_reads: 0 2024-09-13T14:25:26.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-13T14:25:26.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.create_missing_column_families: 0 2024-09-13T14:25:26.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.db_log_dir: 2024-09-13T14:25:26.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.wal_dir: 2024-09-13T14:25:26.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-13T14:25:26.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-13T14:25:26.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-13T14:25:26.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-13T14:25:26.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-13T14:25:26.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-13T14:25:26.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.advise_random_on_open: 1 2024-09-13T14:25:26.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.db_write_buffer_size: 0 2024-09-13T14:25:26.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.write_buffer_manager: 0x55b33ed3e960 2024-09-13T14:25:26.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-13T14:25:26.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-13T14:25:26.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-13T14:25:26.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.rate_limiter: (nil) 2024-09-13T14:25:26.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-13T14:25:26.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.wal_recovery_mode: 2 2024-09-13T14:25:26.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.enable_thread_tracking: 0 2024-09-13T14:25:26.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.enable_pipelined_write: 0 2024-09-13T14:25:26.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.unordered_write: 0 2024-09-13T14:25:26.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-13T14:25:26.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-13T14:25:26.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-13T14:25:26.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-13T14:25:26.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.row_cache: None 2024-09-13T14:25:26.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.wal_filter: None 2024-09-13T14:25:26.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-13T14:25:26.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.allow_ingest_behind: 0 2024-09-13T14:25:26.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.two_write_queues: 0 2024-09-13T14:25:26.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.manual_wal_flush: 0 2024-09-13T14:25:26.866 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.wal_compression: 0 2024-09-13T14:25:26.866 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.atomic_flush: 0 2024-09-13T14:25:26.866 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-13T14:25:26.866 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-13T14:25:26.866 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-13T14:25:26.866 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.log_readahead_size: 0 2024-09-13T14:25:26.866 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-13T14:25:26.866 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.best_efforts_recovery: 0 2024-09-13T14:25:26.866 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-13T14:25:26.866 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-13T14:25:26.866 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.allow_data_in_errors: 0 2024-09-13T14:25:26.866 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.db_host_id: __hostname__ 2024-09-13T14:25:26.866 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-13T14:25:26.867 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.max_background_jobs: 2 2024-09-13T14:25:26.867 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.max_background_compactions: -1 2024-09-13T14:25:26.867 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.max_subcompactions: 1 2024-09-13T14:25:26.867 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-13T14:25:26.867 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-13T14:25:26.867 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-13T14:25:26.867 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.max_total_wal_size: 0 2024-09-13T14:25:26.867 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-13T14:25:26.867 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-13T14:25:26.867 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-13T14:25:26.867 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-13T14:25:26.867 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.max_open_files: -1 2024-09-13T14:25:26.867 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.bytes_per_sync: 0 2024-09-13T14:25:26.867 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-13T14:25:26.868 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-13T14:25:26.868 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.compaction_readahead_size: 0 2024-09-13T14:25:26.868 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.max_background_flushes: -1 2024-09-13T14:25:26.868 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Compression algorithms supported: 2024-09-13T14:25:26.868 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: kZSTD supported: 0 2024-09-13T14:25:26.868 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: kXpressCompression supported: 0 2024-09-13T14:25:26.868 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: kBZip2Compression supported: 0 2024-09-13T14:25:26.868 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-13T14:25:26.868 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: kLZ4Compression supported: 1 2024-09-13T14:25:26.868 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: kZlibCompression supported: 1 2024-09-13T14:25:26.868 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: kLZ4HCCompression supported: 1 2024-09-13T14:25:26.868 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: kSnappyCompression supported: 1 2024-09-13T14:25:26.868 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-13T14:25:26.868 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-13T14:25:26.869 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-09-13T14:25:26.869 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-13T14:25:26.869 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-13T14:25:26.869 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.merge_operator: 2024-09-13T14:25:26.869 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.compaction_filter: None 2024-09-13T14:25:26.869 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.compaction_filter_factory: None 2024-09-13T14:25:26.869 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.sst_partitioner_factory: None 2024-09-13T14:25:26.869 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-13T14:25:26.869 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-13T14:25:26.869 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b33ed3b4e0) 2024-09-13T14:25:26.869 INFO:journalctl@ceph.mon.c.smithi151.stdout: cache_index_and_filter_blocks: 1 2024-09-13T14:25:26.870 INFO:journalctl@ceph.mon.c.smithi151.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-13T14:25:26.870 INFO:journalctl@ceph.mon.c.smithi151.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-13T14:25:26.870 INFO:journalctl@ceph.mon.c.smithi151.stdout: pin_top_level_index_and_filter: 1 2024-09-13T14:25:26.870 INFO:journalctl@ceph.mon.c.smithi151.stdout: index_type: 0 2024-09-13T14:25:26.870 INFO:journalctl@ceph.mon.c.smithi151.stdout: data_block_index_type: 0 2024-09-13T14:25:26.870 INFO:journalctl@ceph.mon.c.smithi151.stdout: index_shortening: 1 2024-09-13T14:25:26.870 INFO:journalctl@ceph.mon.c.smithi151.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-13T14:25:26.870 INFO:journalctl@ceph.mon.c.smithi151.stdout: checksum: 4 2024-09-13T14:25:26.870 INFO:journalctl@ceph.mon.c.smithi151.stdout: no_block_cache: 0 2024-09-13T14:25:26.870 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_cache: 0x55b33ed2e850 2024-09-13T14:25:26.870 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_cache_name: BinnedLRUCache 2024-09-13T14:25:26.870 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_cache_options: 2024-09-13T14:25:26.870 INFO:journalctl@ceph.mon.c.smithi151.stdout: capacity : 536870912 2024-09-13T14:25:26.870 INFO:journalctl@ceph.mon.c.smithi151.stdout: num_shard_bits : 4 2024-09-13T14:25:26.871 INFO:journalctl@ceph.mon.c.smithi151.stdout: strict_capacity_limit : 0 2024-09-13T14:25:26.871 INFO:journalctl@ceph.mon.c.smithi151.stdout: high_pri_pool_ratio: 0.000 2024-09-13T14:25:26.871 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_cache_compressed: (nil) 2024-09-13T14:25:26.871 INFO:journalctl@ceph.mon.c.smithi151.stdout: persistent_cache: (nil) 2024-09-13T14:25:26.871 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_size: 4096 2024-09-13T14:25:26.871 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_size_deviation: 10 2024-09-13T14:25:26.871 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_restart_interval: 16 2024-09-13T14:25:26.871 INFO:journalctl@ceph.mon.c.smithi151.stdout: index_block_restart_interval: 1 2024-09-13T14:25:26.871 INFO:journalctl@ceph.mon.c.smithi151.stdout: metadata_block_size: 4096 2024-09-13T14:25:26.871 INFO:journalctl@ceph.mon.c.smithi151.stdout: partition_filters: 0 2024-09-13T14:25:26.871 INFO:journalctl@ceph.mon.c.smithi151.stdout: use_delta_encoding: 1 2024-09-13T14:25:26.871 INFO:journalctl@ceph.mon.c.smithi151.stdout: filter_policy: bloomfilter 2024-09-13T14:25:26.871 INFO:journalctl@ceph.mon.c.smithi151.stdout: whole_key_filtering: 1 2024-09-13T14:25:26.871 INFO:journalctl@ceph.mon.c.smithi151.stdout: verify_compression: 0 2024-09-13T14:25:26.871 INFO:journalctl@ceph.mon.c.smithi151.stdout: read_amp_bytes_per_bit: 0 2024-09-13T14:25:26.872 INFO:journalctl@ceph.mon.c.smithi151.stdout: format_version: 5 2024-09-13T14:25:26.872 INFO:journalctl@ceph.mon.c.smithi151.stdout: enable_index_compression: 1 2024-09-13T14:25:26.872 INFO:journalctl@ceph.mon.c.smithi151.stdout: block_align: 0 2024-09-13T14:25:26.872 INFO:journalctl@ceph.mon.c.smithi151.stdout: max_auto_readahead_size: 262144 2024-09-13T14:25:26.872 INFO:journalctl@ceph.mon.c.smithi151.stdout: prepopulate_block_cache: 0 2024-09-13T14:25:26.872 INFO:journalctl@ceph.mon.c.smithi151.stdout: initial_auto_readahead_size: 8192 2024-09-13T14:25:26.872 INFO:journalctl@ceph.mon.c.smithi151.stdout: num_file_reads_for_auto_readahead: 2 2024-09-13T14:25:26.872 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.write_buffer_size: 33554432 2024-09-13T14:25:26.872 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.max_write_buffer_number: 2 2024-09-13T14:25:26.872 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.compression: NoCompression 2024-09-13T14:25:26.872 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.bottommost_compression: Disabled 2024-09-13T14:25:26.872 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.prefix_extractor: nullptr 2024-09-13T14:25:26.872 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-13T14:25:26.873 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.num_levels: 7 2024-09-13T14:25:26.873 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-13T14:25:26.873 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-13T14:25:26.873 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-13T14:25:26.873 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-13T14:25:26.873 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-13T14:25:26.873 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-13T14:25:26.873 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-13T14:25:26.873 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:25:26.873 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-13T14:25:26.873 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-13T14:25:26.873 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-13T14:25:26.873 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-13T14:25:26.873 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-13T14:25:26.874 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.compression_opts.level: 32767 2024-09-13T14:25:26.874 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.compression_opts.strategy: 0 2024-09-13T14:25:26.874 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-13T14:25:26.874 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:25:26.874 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-13T14:25:26.874 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-13T14:25:26.874 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.compression_opts.enabled: false 2024-09-13T14:25:26.874 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-13T14:25:26.874 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-13T14:25:26.874 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-13T14:25:26.874 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-13T14:25:26.874 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.target_file_size_base: 67108864 2024-09-13T14:25:26.874 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-13T14:25:26.875 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-13T14:25:26.875 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-13T14:25:26.875 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-13T14:25:26.876 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-13T14:25:26.876 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-13T14:25:26.876 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-13T14:25:26.876 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-13T14:25:26.876 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-13T14:25:26.876 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-13T14:25:26.876 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-13T14:25:26.876 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-13T14:25:26.877 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-13T14:25:26.877 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-13T14:25:26.878 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.arena_block_size: 1048576 2024-09-13T14:25:26.878 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-13T14:25:26.878 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-13T14:25:26.878 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.disable_auto_compactions: 0 2024-09-13T14:25:26.878 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-13T14:25:26.878 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-13T14:25:26.878 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-13T14:25:26.878 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-13T14:25:26.878 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-13T14:25:26.878 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-13T14:25:26.879 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-13T14:25:26.879 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-13T14:25:26.879 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-13T14:25:26.879 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-13T14:25:26.879 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-13T14:25:26.879 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.inplace_update_support: 0 2024-09-13T14:25:26.879 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-13T14:25:26.880 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-13T14:25:26.880 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-13T14:25:26.880 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-13T14:25:26.880 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.bloom_locality: 0 2024-09-13T14:25:26.880 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.max_successive_merges: 0 2024-09-13T14:25:26.880 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-13T14:25:26.880 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.paranoid_file_checks: 0 2024-09-13T14:25:26.881 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.force_consistency_checks: 1 2024-09-13T14:25:26.881 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.report_bg_io_stats: 0 2024-09-13T14:25:26.881 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.ttl: 2592000 2024-09-13T14:25:26.881 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-13T14:25:26.881 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-13T14:25:26.881 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-13T14:25:26.881 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.enable_blob_files: false 2024-09-13T14:25:26.881 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.min_blob_size: 0 2024-09-13T14:25:26.881 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.blob_file_size: 268435456 2024-09-13T14:25:26.882 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-13T14:25:26.882 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-13T14:25:26.882 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-13T14:25:26.882 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-13T14:25:26.882 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-13T14:25:26.882 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.blob_file_starting_level: 0 2024-09-13T14:25:26.882 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-13T14:25:26.882 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 24.sst 2024-09-13T14:25:26.882 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-13T14:25:26.883 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 26, last_sequence is 13434, log_number is 22,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-13T14:25:26.883 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 22 2024-09-13T14:25:26.883 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: c1e19c29-a9b1-46fb-82b9-f180b02a989c 2024-09-13T14:25:26.883 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726237526405995, "job": 1, "event": "recovery_started", "wal_files": [22]} 2024-09-13T14:25:26.883 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #22 mode 2 2024-09-13T14:25:26.883 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726237526435234, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 27, "file_size": 7452877, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 13439, "largest_seqno": 17982, "table_properties": {"data_size": 7432593, "index_size": 14437, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 4805, "raw_key_size": 43167, "raw_average_key_size": 22, "raw_value_size": 7391789, "raw_average_value_size": 3888, "num_data_blocks": 622, "num_entries": 1901, "num_filter_entries": 1901, "num_deletions": 316, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726237526, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "c1e19c29-a9b1-46fb-82b9-f180b02a989c", "db_session_id": "21I9A7A114IWKIFHNUY1", "orig_file_number": 27, "seqno_to_time_mapping": "N/A"}} 2024-09-13T14:25:26.883 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726237526435376, "job": 1, "event": "recovery_finished"} 2024-09-13T14:25:26.884 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: [db/version_set.cc:5047] Creating manifest 29 2024-09-13T14:25:26.884 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-13T14:25:26.884 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000022.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-13T14:25:26.885 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55b33ed5ee00 2024-09-13T14:25:26.885 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: DB pointer 0x55b33ee5e000 2024-09-13T14:25:26.885 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-13T14:25:26.885 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-13T14:25:26.885 INFO:journalctl@ceph.mon.c.smithi151.stdout: ** DB Stats ** 2024-09-13T14:25:26.885 INFO:journalctl@ceph.mon.c.smithi151.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:25:26.885 INFO:journalctl@ceph.mon.c.smithi151.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-13T14:25:26.885 INFO:journalctl@ceph.mon.c.smithi151.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:25:26.886 INFO:journalctl@ceph.mon.c.smithi151.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:25:26.886 INFO:journalctl@ceph.mon.c.smithi151.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-13T14:25:26.886 INFO:journalctl@ceph.mon.c.smithi151.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:25:26.886 INFO:journalctl@ceph.mon.c.smithi151.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:25:26.886 INFO:journalctl@ceph.mon.c.smithi151.stdout: 2024-09-13T14:25:26.886 INFO:journalctl@ceph.mon.c.smithi151.stdout: ** Compaction Stats [default] ** 2024-09-13T14:25:26.886 INFO:journalctl@ceph.mon.c.smithi151.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T14:25:26.887 INFO:journalctl@ceph.mon.c.smithi151.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-13T14:25:26.887 INFO:journalctl@ceph.mon.c.smithi151.stdout: L0 1/0 7.11 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 551.5 0.01 0.00 1 0.013 0 0 0.0 0.0 2024-09-13T14:25:26.887 INFO:journalctl@ceph.mon.c.smithi151.stdout: L6 1/0 7.36 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-13T14:25:26.887 INFO:journalctl@ceph.mon.c.smithi151.stdout: Sum 2/0 14.46 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 551.5 0.01 0.00 1 0.013 0 0 0.0 0.0 2024-09-13T14:25:26.887 INFO:journalctl@ceph.mon.c.smithi151.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 551.5 0.01 0.00 1 0.013 0 0 0.0 0.0 2024-09-13T14:25:26.887 INFO:journalctl@ceph.mon.c.smithi151.stdout: 2024-09-13T14:25:26.887 INFO:journalctl@ceph.mon.c.smithi151.stdout: ** Compaction Stats [default] ** 2024-09-13T14:25:26.887 INFO:journalctl@ceph.mon.c.smithi151.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T14:25:26.888 INFO:journalctl@ceph.mon.c.smithi151.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T14:25:26.888 INFO:journalctl@ceph.mon.c.smithi151.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 551.5 0.01 0.00 1 0.013 0 0 0.0 0.0 2024-09-13T14:25:26.888 INFO:journalctl@ceph.mon.c.smithi151.stdout: 2024-09-13T14:25:26.888 INFO:journalctl@ceph.mon.c.smithi151.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-13T14:25:26.888 INFO:journalctl@ceph.mon.c.smithi151.stdout: 2024-09-13T14:25:26.888 INFO:journalctl@ceph.mon.c.smithi151.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:25:26.888 INFO:journalctl@ceph.mon.c.smithi151.stdout: Flush(GB): cumulative 0.007, interval 0.007 2024-09-13T14:25:26.888 INFO:journalctl@ceph.mon.c.smithi151.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-13T14:25:26.889 INFO:journalctl@ceph.mon.c.smithi151.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-13T14:25:26.889 INFO:journalctl@ceph.mon.c.smithi151.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-13T14:25:26.889 INFO:journalctl@ceph.mon.c.smithi151.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-13T14:25:26.889 INFO:journalctl@ceph.mon.c.smithi151.stdout: Cumulative compaction: 0.01 GB write, 219.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:25:26.889 INFO:journalctl@ceph.mon.c.smithi151.stdout: Interval compaction: 0.01 GB write, 219.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:25:26.889 INFO:journalctl@ceph.mon.c.smithi151.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-13T14:25:26.890 INFO:journalctl@ceph.mon.c.smithi151.stdout: Block cache BinnedLRUCache@0x55b33ed2e850#7 capacity: 512.00 MB usage: 21.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-09-13T14:25:26.890 INFO:journalctl@ceph.mon.c.smithi151.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,5.03 KB,0.000959635%) IndexBlock(1,16.08 KB,0.00306666%) Misc(1,0.00 KB,0%) 2024-09-13T14:25:26.890 INFO:journalctl@ceph.mon.c.smithi151.stdout: 2024-09-13T14:25:26.890 INFO:journalctl@ceph.mon.c.smithi151.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-13T14:25:26.890 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: starting mon.c rank 2 at public addrs [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] at bind addrs [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:25:26.890 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: mon.c@-1(???) e3 preinit fsid 2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:25:26.890 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: mon.c@-1(???).mds e9 new map 2024-09-13T14:25:26.890 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: mon.c@-1(???).mds e9 print_map 2024-09-13T14:25:26.891 INFO:journalctl@ceph.mon.c.smithi151.stdout: e9 2024-09-13T14:25:26.891 INFO:journalctl@ceph.mon.c.smithi151.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-13T14:25:26.891 INFO:journalctl@ceph.mon.c.smithi151.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-13T14:25:26.891 INFO:journalctl@ceph.mon.c.smithi151.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T14:25:26.891 INFO:journalctl@ceph.mon.c.smithi151.stdout: legacy client fscid: 1 2024-09-13T14:25:26.891 INFO:journalctl@ceph.mon.c.smithi151.stdout: 2024-09-13T14:25:26.891 INFO:journalctl@ceph.mon.c.smithi151.stdout: Filesystem 'foo' (1) 2024-09-13T14:25:26.891 INFO:journalctl@ceph.mon.c.smithi151.stdout: fs_name foo 2024-09-13T14:25:26.892 INFO:journalctl@ceph.mon.c.smithi151.stdout: epoch 8 2024-09-13T14:25:26.892 INFO:journalctl@ceph.mon.c.smithi151.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-13T14:25:26.892 INFO:journalctl@ceph.mon.c.smithi151.stdout: created 2024-09-13T14:19:25.869068+0000 2024-09-13T14:25:26.892 INFO:journalctl@ceph.mon.c.smithi151.stdout: modified 2024-09-13T14:20:09.750989+0000 2024-09-13T14:25:26.892 INFO:journalctl@ceph.mon.c.smithi151.stdout: tableserver 0 2024-09-13T14:25:26.892 INFO:journalctl@ceph.mon.c.smithi151.stdout: root 0 2024-09-13T14:25:26.892 INFO:journalctl@ceph.mon.c.smithi151.stdout: session_timeout 60 2024-09-13T14:25:26.892 INFO:journalctl@ceph.mon.c.smithi151.stdout: session_autoclose 300 2024-09-13T14:25:26.892 INFO:journalctl@ceph.mon.c.smithi151.stdout: max_file_size 1099511627776 2024-09-13T14:25:26.893 INFO:journalctl@ceph.mon.c.smithi151.stdout: max_xattr_size 65536 2024-09-13T14:25:26.893 INFO:journalctl@ceph.mon.c.smithi151.stdout: required_client_features {} 2024-09-13T14:25:26.893 INFO:journalctl@ceph.mon.c.smithi151.stdout: last_failure 0 2024-09-13T14:25:26.893 INFO:journalctl@ceph.mon.c.smithi151.stdout: last_failure_osd_epoch 0 2024-09-13T14:25:26.894 INFO:journalctl@ceph.mon.c.smithi151.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T14:25:26.894 INFO:journalctl@ceph.mon.c.smithi151.stdout: max_mds 1 2024-09-13T14:25:26.894 INFO:journalctl@ceph.mon.c.smithi151.stdout: in 0 2024-09-13T14:25:26.894 INFO:journalctl@ceph.mon.c.smithi151.stdout: up {0=24614} 2024-09-13T14:25:26.894 INFO:journalctl@ceph.mon.c.smithi151.stdout: failed 2024-09-13T14:25:26.895 INFO:journalctl@ceph.mon.c.smithi151.stdout: damaged 2024-09-13T14:25:26.895 INFO:journalctl@ceph.mon.c.smithi151.stdout: stopped 2024-09-13T14:25:26.895 INFO:journalctl@ceph.mon.c.smithi151.stdout: data_pools [4] 2024-09-13T14:25:26.895 INFO:journalctl@ceph.mon.c.smithi151.stdout: metadata_pool 3 2024-09-13T14:25:26.895 INFO:journalctl@ceph.mon.c.smithi151.stdout: inline_data disabled 2024-09-13T14:25:26.895 INFO:journalctl@ceph.mon.c.smithi151.stdout: balancer 2024-09-13T14:25:26.895 INFO:journalctl@ceph.mon.c.smithi151.stdout: bal_rank_mask -1 2024-09-13T14:25:26.895 INFO:journalctl@ceph.mon.c.smithi151.stdout: standby_count_wanted 1 2024-09-13T14:25:26.896 INFO:journalctl@ceph.mon.c.smithi151.stdout: qdb_cluster leader: 0 members: 2024-09-13T14:25:26.896 INFO:journalctl@ceph.mon.c.smithi151.stdout: [mds.foo.smithi151.rqqklw{0:24614} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.151:6834/345315682,v1:172.21.15.151:6835/345315682] compat {c=[1],r=[1],i=[7ff]}] 2024-09-13T14:25:26.896 INFO:journalctl@ceph.mon.c.smithi151.stdout: 2024-09-13T14:25:26.896 INFO:journalctl@ceph.mon.c.smithi151.stdout: 2024-09-13T14:25:26.896 INFO:journalctl@ceph.mon.c.smithi151.stdout: Standby daemons: 2024-09-13T14:25:26.896 INFO:journalctl@ceph.mon.c.smithi151.stdout: 2024-09-13T14:25:26.896 INFO:journalctl@ceph.mon.c.smithi151.stdout: [mds.foo.smithi184.jismsi{-1:24529} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.184:6832/4138067167,v1:172.21.15.184:6833/4138067167] compat {c=[1],r=[1],i=[7ff]}] 2024-09-13T14:25:26.896 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: mon.c@-1(???).osd e343 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-13T14:25:26.896 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: mon.c@-1(???).osd e343 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:25:26.896 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: mon.c@-1(???).osd e343 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:25:26.896 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: mon.c@-1(???).osd e343 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:25:26.896 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 ceph-mon[108966]: mon.c@-1(???).paxosservice(auth 1..20) refresh upgraded, format 0 -> 3 2024-09-13T14:25:26.896 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 bash[108928]: e548fceed2f9a4804e53d2f49ccc63cb9cc0cd3fb315e9189c6066ba685cf405 2024-09-13T14:25:26.897 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:26 smithi151 systemd[1]: Started Ceph mon.c for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:25:27.354 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:25:26 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:25:26.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi184\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:25:30.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:25:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:25:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:25:32.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:31 smithi184 ceph-mon[33394]: pgmap v126: 89 pgs: 89 active+clean; 259 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 17 MiB/s rd, 3.1 MiB/s wr, 25 op/s 2024-09-13T14:25:32.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:31 smithi184 ceph-mon[33394]: osdmap e342: 8 total, 8 up, 8 in 2024-09-13T14:25:32.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:31 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/222246908' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:25:32.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:31 smithi184 ceph-mon[33394]: osdmap e343: 8 total, 8 up, 8 in 2024-09-13T14:25:32.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:31 smithi151 ceph-mon[104450]: pgmap v126: 89 pgs: 89 active+clean; 259 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 17 MiB/s rd, 3.1 MiB/s wr, 25 op/s 2024-09-13T14:25:32.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:31 smithi151 ceph-mon[104450]: osdmap e342: 8 total, 8 up, 8 in 2024-09-13T14:25:32.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:31 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/222246908' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:25:32.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:31 smithi151 ceph-mon[104450]: osdmap e343: 8 total, 8 up, 8 in 2024-09-13T14:25:32.689 INFO:tasks.rados.rados.0.smithi151.stdout:1131: done (1 left) 2024-09-13T14:25:32.689 INFO:tasks.rados.rados.0.smithi151.stdout:1133: done (0 left) 2024-09-13T14:25:32.689 INFO:tasks.rados.rados.0.smithi151.stdout:1134: read oid 6 snap -1 2024-09-13T14:25:32.689 INFO:tasks.rados.rados.0.smithi151.stdout:1134: expect (ObjNum 426 snap 135 seq_num 426) 2024-09-13T14:25:32.689 INFO:tasks.rados.rados.0.smithi151.stdout:1135: read oid 21 snap -1 2024-09-13T14:25:32.689 INFO:tasks.rados.rados.0.smithi151.stdout:1135: expect deleted 2024-09-13T14:25:32.689 INFO:tasks.rados.rados.0.smithi151.stdout:1136: snap_create 2024-09-13T14:25:32.699 INFO:tasks.rados.rados.0.smithi151.stdout:1134: expect (ObjNum 426 snap 135 seq_num 426) 2024-09-13T14:25:33.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:32 smithi184 ceph-mon[33394]: mon.c calling monitor election 2024-09-13T14:25:33.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:32 smithi184 ceph-mon[33394]: mon.b calling monitor election 2024-09-13T14:25:33.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:32 smithi184 ceph-mon[33394]: mon.a calling monitor election 2024-09-13T14:25:33.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:32 smithi184 ceph-mon[33394]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:25:33.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:32 smithi184 ceph-mon[33394]: monmap epoch 3 2024-09-13T14:25:33.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:32 smithi184 ceph-mon[33394]: fsid 2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:25:33.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:32 smithi184 ceph-mon[33394]: last_changed 2024-09-13T14:14:11.796948+0000 2024-09-13T14:25:33.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:32 smithi184 ceph-mon[33394]: created 2024-09-13T14:12:33.302475+0000 2024-09-13T14:25:33.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:32 smithi184 ceph-mon[33394]: min_mon_release 17 (quincy) 2024-09-13T14:25:33.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:32 smithi184 ceph-mon[33394]: election_strategy: 1 2024-09-13T14:25:33.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:32 smithi184 ceph-mon[33394]: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2024-09-13T14:25:33.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:32 smithi184 ceph-mon[33394]: 1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2024-09-13T14:25:33.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:32 smithi184 ceph-mon[33394]: 2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2024-09-13T14:25:33.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:32 smithi184 ceph-mon[33394]: fsmap foo:1 {0=foo.smithi151.rqqklw=up:active} 1 up:standby 2024-09-13T14:25:33.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:32 smithi184 ceph-mon[33394]: osdmap e343: 8 total, 8 up, 8 in 2024-09-13T14:25:33.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:32 smithi184 ceph-mon[33394]: mgrmap e37: y(active, since 91s), standbys: x 2024-09-13T14:25:33.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:32 smithi184 ceph-mon[33394]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-13T14:25:33.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:32 smithi184 ceph-mon[33394]: Cluster is now healthy 2024-09-13T14:25:33.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:32 smithi184 ceph-mon[33394]: overall HEALTH_OK 2024-09-13T14:25:33.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[104450]: mon.c calling monitor election 2024-09-13T14:25:33.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[104450]: mon.b calling monitor election 2024-09-13T14:25:33.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[104450]: mon.a calling monitor election 2024-09-13T14:25:33.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[104450]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:25:33.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[104450]: monmap epoch 3 2024-09-13T14:25:33.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[104450]: fsid 2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:25:33.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[104450]: last_changed 2024-09-13T14:14:11.796948+0000 2024-09-13T14:25:33.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[104450]: created 2024-09-13T14:12:33.302475+0000 2024-09-13T14:25:33.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[104450]: min_mon_release 17 (quincy) 2024-09-13T14:25:33.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[104450]: election_strategy: 1 2024-09-13T14:25:33.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[104450]: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2024-09-13T14:25:33.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[104450]: 1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2024-09-13T14:25:33.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[104450]: 2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2024-09-13T14:25:33.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[104450]: fsmap foo:1 {0=foo.smithi151.rqqklw=up:active} 1 up:standby 2024-09-13T14:25:33.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[104450]: osdmap e343: 8 total, 8 up, 8 in 2024-09-13T14:25:33.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[104450]: mgrmap e37: y(active, since 91s), standbys: x 2024-09-13T14:25:33.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[104450]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-13T14:25:33.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[104450]: Cluster is now healthy 2024-09-13T14:25:33.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[104450]: overall HEALTH_OK 2024-09-13T14:25:33.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[108966]: pgmap v126: 89 pgs: 89 active+clean; 259 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 17 MiB/s rd, 3.1 MiB/s wr, 25 op/s 2024-09-13T14:25:33.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[108966]: osdmap e342: 8 total, 8 up, 8 in 2024-09-13T14:25:33.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/222246908' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-13T14:25:33.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[108966]: osdmap e343: 8 total, 8 up, 8 in 2024-09-13T14:25:33.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[108966]: mon.c calling monitor election 2024-09-13T14:25:33.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[108966]: mon.b calling monitor election 2024-09-13T14:25:33.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[108966]: mon.a calling monitor election 2024-09-13T14:25:33.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[108966]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:25:33.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[108966]: monmap epoch 3 2024-09-13T14:25:33.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[108966]: fsid 2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:25:33.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[108966]: last_changed 2024-09-13T14:14:11.796948+0000 2024-09-13T14:25:33.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[108966]: created 2024-09-13T14:12:33.302475+0000 2024-09-13T14:25:33.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[108966]: min_mon_release 17 (quincy) 2024-09-13T14:25:33.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[108966]: election_strategy: 1 2024-09-13T14:25:33.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[108966]: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2024-09-13T14:25:33.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[108966]: 1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2024-09-13T14:25:33.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[108966]: 2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2024-09-13T14:25:33.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[108966]: fsmap foo:1 {0=foo.smithi151.rqqklw=up:active} 1 up:standby 2024-09-13T14:25:33.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[108966]: osdmap e343: 8 total, 8 up, 8 in 2024-09-13T14:25:33.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[108966]: mgrmap e37: y(active, since 91s), standbys: x 2024-09-13T14:25:33.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[108966]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-13T14:25:33.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[108966]: Cluster is now healthy 2024-09-13T14:25:33.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:32 smithi151 ceph-mon[108966]: overall HEALTH_OK 2024-09-13T14:25:33.231 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:25:33.690 INFO:tasks.rados.rados.0.smithi151.stdout:1134: done (2 left) 2024-09-13T14:25:33.690 INFO:tasks.rados.rados.0.smithi151.stdout:1135: done (1 left) 2024-09-13T14:25:33.690 INFO:tasks.rados.rados.0.smithi151.stdout:1136: done (0 left) 2024-09-13T14:25:33.690 INFO:tasks.rados.rados.0.smithi151.stdout:1137: write oid 49 current snap is 147 2024-09-13T14:25:33.690 INFO:tasks.rados.rados.0.smithi151.stdout:1137: seq_num 450 ranges {763150=726877,1982897=614818,3024582=474994} 2024-09-13T14:25:33.698 INFO:tasks.rados.rados.0.smithi151.stdout:1137: writing smithi15186934-49 from 763150 to 1490027 tid 1 2024-09-13T14:25:33.704 INFO:tasks.rados.rados.0.smithi151.stdout:1137: writing smithi15186934-49 from 1982897 to 2597715 tid 2 2024-09-13T14:25:33.710 INFO:tasks.rados.rados.0.smithi151.stdout:1137: writing smithi15186934-49 from 3024582 to 3499576 tid 3 2024-09-13T14:25:33.710 INFO:tasks.rados.rados.0.smithi151.stdout:1138: snap_remove snap 136 2024-09-13T14:25:33.968 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:33 smithi151 ceph-mon[104450]: osdmap e344: 8 total, 8 up, 8 in 2024-09-13T14:25:33.968 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:33 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3608058042' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-13T14:25:33.968 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:33 smithi151 ceph-mon[104450]: osdmap e345: 8 total, 8 up, 8 in 2024-09-13T14:25:33.969 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:33 smithi151 ceph-mon[108966]: osdmap e344: 8 total, 8 up, 8 in 2024-09-13T14:25:33.969 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:33 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3608058042' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-13T14:25:33.969 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:33 smithi151 ceph-mon[108966]: osdmap e345: 8 total, 8 up, 8 in 2024-09-13T14:25:33.993 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch ps' 2024-09-13T14:25:34.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:33 smithi184 ceph-mon[33394]: osdmap e344: 8 total, 8 up, 8 in 2024-09-13T14:25:34.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:33 smithi184 ceph-mon[33394]: from='client.? 172.21.15.151:0/3608058042' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-13T14:25:34.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:33 smithi184 ceph-mon[33394]: osdmap e345: 8 total, 8 up, 8 in 2024-09-13T14:25:34.426 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:25:34.463 INFO:tasks.rados.rados.0.smithi151.stdout:1138: done (1 left) 2024-09-13T14:25:34.463 INFO:tasks.rados.rados.0.smithi151.stdout:1139: read oid 1 snap -1 2024-09-13T14:25:34.463 INFO:tasks.rados.rados.0.smithi151.stdout:1139: expect (ObjNum 442 snap 141 seq_num 442) 2024-09-13T14:25:34.463 INFO:tasks.rados.rados.0.smithi151.stdout:1140: read oid 40 snap 129 2024-09-13T14:25:34.463 INFO:tasks.rados.rados.0.smithi151.stdout:1140: expect deleted 2024-09-13T14:25:34.463 INFO:tasks.rados.rados.0.smithi151.stdout:1141: snap_remove snap 142 2024-09-13T14:25:34.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:25:34 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:25:34.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:25:34.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:34 smithi151 ceph-mon[104450]: pgmap v134: 89 pgs: 89 active+clean; 261 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.2 MiB/s rd, 183 KiB/s wr, 4 op/s 2024-09-13T14:25:34.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:34 smithi151 ceph-mon[104450]: mgrmap e38: y(active, since 92s), standbys: x 2024-09-13T14:25:34.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:34 smithi151 ceph-mon[104450]: osdmap e346: 8 total, 8 up, 8 in 2024-09-13T14:25:34.840 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:25:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:25:34] "GET /metrics HTTP/1.1" 200 36399 "" "Prometheus/2.51.0" 2024-09-13T14:25:34.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:34 smithi151 ceph-mon[108966]: pgmap v134: 89 pgs: 89 active+clean; 261 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.2 MiB/s rd, 183 KiB/s wr, 4 op/s 2024-09-13T14:25:34.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:34 smithi151 ceph-mon[108966]: mgrmap e38: y(active, since 92s), standbys: x 2024-09-13T14:25:34.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:34 smithi151 ceph-mon[108966]: osdmap e346: 8 total, 8 up, 8 in 2024-09-13T14:25:35.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:34 smithi184 ceph-mon[33394]: pgmap v134: 89 pgs: 89 active+clean; 261 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.2 MiB/s rd, 183 KiB/s wr, 4 op/s 2024-09-13T14:25:35.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:34 smithi184 ceph-mon[33394]: mgrmap e38: y(active, since 92s), standbys: x 2024-09-13T14:25:35.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:34 smithi184 ceph-mon[33394]: osdmap e346: 8 total, 8 up, 8 in 2024-09-13T14:25:35.464 INFO:tasks.rados.rados.0.smithi151.stdout:1141: done (3 left) 2024-09-13T14:25:35.464 INFO:tasks.rados.rados.0.smithi151.stdout:1142: read oid 37 snap 140 2024-09-13T14:25:35.464 INFO:tasks.rados.rados.0.smithi151.stdout:1142: expect deleted 2024-09-13T14:25:35.464 INFO:tasks.rados.rados.0.smithi151.stdout:1137: finishing write tid 1 to smithi15186934-49 2024-09-13T14:25:35.465 INFO:tasks.rados.rados.0.smithi151.stdout:1137: finishing write tid 2 to smithi15186934-49 2024-09-13T14:25:35.465 INFO:tasks.rados.rados.0.smithi151.stdout:1137: finishing write tid 3 to smithi15186934-49 2024-09-13T14:25:35.465 INFO:tasks.rados.rados.0.smithi151.stdout:1137: finishing write tid 4 to smithi15186934-49 2024-09-13T14:25:35.465 INFO:tasks.rados.rados.0.smithi151.stdout:1137: finishing write tid 5 to smithi15186934-49 2024-09-13T14:25:35.465 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 49 v 527 (ObjNum 450 snap 147 seq_num 450) dirty exists 2024-09-13T14:25:35.465 INFO:tasks.rados.rados.0.smithi151.stdout:1137: left oid 49 (ObjNum 450 snap 147 seq_num 450) 2024-09-13T14:25:35.465 INFO:tasks.rados.rados.0.smithi151.stdout:1139: expect (ObjNum 442 snap 141 seq_num 442) 2024-09-13T14:25:35.628 INFO:tasks.rados.rados.0.smithi151.stdout:1143: read oid 4 snap -1 2024-09-13T14:25:35.628 INFO:tasks.rados.rados.0.smithi151.stdout:1143: expect (ObjNum 411 snap 128 seq_num 411) 2024-09-13T14:25:35.628 INFO:tasks.rados.rados.0.smithi151.stdout:1137: done (4 left) 2024-09-13T14:25:35.628 INFO:tasks.rados.rados.0.smithi151.stdout:1139: done (3 left) 2024-09-13T14:25:35.628 INFO:tasks.rados.rados.0.smithi151.stdout:1140: done (2 left) 2024-09-13T14:25:35.628 INFO:tasks.rados.rados.0.smithi151.stdout:1142: done (1 left) 2024-09-13T14:25:35.628 INFO:tasks.rados.rados.0.smithi151.stdout:1144: read oid 32 snap -1 2024-09-13T14:25:35.628 INFO:tasks.rados.rados.0.smithi151.stdout:1144: expect (ObjNum 424 snap 132 seq_num 424) 2024-09-13T14:25:35.628 INFO:tasks.rados.rados.0.smithi151.stdout:1145: write oid 46 current snap is 147 2024-09-13T14:25:35.628 INFO:tasks.rados.rados.0.smithi151.stdout:1145: seq_num 451 ranges {779957=402126,1925026=758476,3335652=244305} 2024-09-13T14:25:35.634 INFO:tasks.rados.rados.0.smithi151.stdout:1145: writing smithi15186934-46 from 779957 to 1182083 tid 1 2024-09-13T14:25:35.642 INFO:tasks.rados.rados.0.smithi151.stdout:1145: writing smithi15186934-46 from 1925026 to 2683502 tid 2 2024-09-13T14:25:35.647 INFO:tasks.rados.rados.0.smithi151.stdout:1145: writing smithi15186934-46 from 3335652 to 3579957 tid 3 2024-09-13T14:25:35.647 INFO:tasks.rados.rados.0.smithi151.stdout:1146: read oid 18 snap -1 2024-09-13T14:25:35.647 INFO:tasks.rados.rados.0.smithi151.stdout:1146: expect (ObjNum 368 snap 116 seq_num 368) 2024-09-13T14:25:35.647 INFO:tasks.rados.rados.0.smithi151.stdout:1147: snap_remove snap 124 2024-09-13T14:25:36.103 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:25:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:25:35.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:25:36.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:25:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:25:35.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:25:36.480 INFO:tasks.rados.rados.0.smithi151.stdout:1147: done (4 left) 2024-09-13T14:25:36.480 INFO:tasks.rados.rados.0.smithi151.stdout:1148: read oid 21 snap -1 2024-09-13T14:25:36.480 INFO:tasks.rados.rados.0.smithi151.stdout:1148: expect deleted 2024-09-13T14:25:36.480 INFO:tasks.rados.rados.0.smithi151.stdout:1149: read oid 34 snap -1 2024-09-13T14:25:36.480 INFO:tasks.rados.rados.0.smithi151.stdout:1149: expect deleted 2024-09-13T14:25:36.480 INFO:tasks.rados.rados.0.smithi151.stdout:1150: delete oid 12 current snap is 147 2024-09-13T14:25:36.480 INFO:tasks.rados.rados.0.smithi151.stdout:1144: expect (ObjNum 424 snap 132 seq_num 424) 2024-09-13T14:25:36.528 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:36 smithi151 ceph-mon[104450]: pgmap v137: 89 pgs: 89 active+clean; 261 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-09-13T14:25:36.528 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:36 smithi151 ceph-mon[104450]: osdmap e347: 8 total, 8 up, 8 in 2024-09-13T14:25:36.528 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:36 smithi151 ceph-mon[108966]: pgmap v137: 89 pgs: 89 active+clean; 261 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-09-13T14:25:36.528 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:36 smithi151 ceph-mon[108966]: osdmap e347: 8 total, 8 up, 8 in 2024-09-13T14:25:36.528 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:25:36 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:25:36.445+0000 7fc94c153640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-13T14:25:36.624 INFO:tasks.rados.rados.0.smithi151.stdout:1143: expect (ObjNum 411 snap 128 seq_num 411) 2024-09-13T14:25:36.691 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:25:36.691 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.a smithi151 *:9093,9094 running (2m) 16s ago 6m 20.0M - 0.27.0 11f11916f8cd ed70c3d75107 2024-09-13T14:25:36.691 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.a smithi184 *:3000 running (2m) 20s ago 6m 80.6M - 9.4.7 954c08fa6188 4ba18fb1f1fc 2024-09-13T14:25:36.691 INFO:teuthology.orchestra.run.smithi151.stdout:mds.foo.smithi151.rqqklw smithi151 running (5m) 16s ago 5m 17.4M - 17.2.7-1627-gbbdadfa3 a994b00deb3a c960b57fc84b 2024-09-13T14:25:36.691 INFO:teuthology.orchestra.run.smithi151.stdout:mds.foo.smithi184.jismsi smithi184 running (5m) 20s ago 5m 14.8M - 17.2.7-1627-gbbdadfa3 a994b00deb3a 0cc31a3a5339 2024-09-13T14:25:36.691 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.x smithi184 *:8443,9283,8765 running (116s) 20s ago 11m 453M - 19.3.0-4697-gd58f7306 fec07016a2b0 8a0f7d440b6e 2024-09-13T14:25:36.691 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.y smithi151 *:8443,9283,8765 running (2m) 16s ago 12m 504M - 19.3.0-4697-gd58f7306 fec07016a2b0 d8c169a4f36e 2024-09-13T14:25:36.692 INFO:teuthology.orchestra.run.smithi151.stdout:mon.a smithi151 running (102s) 16s ago 13m 42.1M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 694de6b945cd 2024-09-13T14:25:36.692 INFO:teuthology.orchestra.run.smithi151.stdout:mon.b smithi184 running (11m) 20s ago 11m 59.1M 2048M 17.2.7-1627-gbbdadfa3 a994b00deb3a 8c637f147650 2024-09-13T14:25:36.692 INFO:teuthology.orchestra.run.smithi151.stdout:mon.c smithi151 starting - - - 2048M 2024-09-13T14:25:36.692 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.a smithi151 *:9100 running (2m) 16s ago 6m 11.6M - 1.7.0 72c9c2088986 d91e4cb09e9a 2024-09-13T14:25:36.692 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.b smithi184 *:9100 running (2m) 20s ago 6m 10.2M - 1.7.0 72c9c2088986 1e1a9dd643a0 2024-09-13T14:25:36.692 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (10m) 16s ago 10m 88.7M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a e6951a1a86d9 2024-09-13T14:25:36.692 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (10m) 16s ago 10m 71.8M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a 94bc726b3eb5 2024-09-13T14:25:36.692 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (10m) 16s ago 10m 92.9M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a 122660181e46 2024-09-13T14:25:36.692 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi151 running (9m) 16s ago 9m 67.2M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a 495241a87df6 2024-09-13T14:25:36.692 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi184 running (9m) 20s ago 9m 83.8M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a 6e70699b8084 2024-09-13T14:25:36.692 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi184 running (8m) 20s ago 8m 87.9M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a 36b82c9e7230 2024-09-13T14:25:36.692 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6 smithi184 running (8m) 20s ago 8m 68.5M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a f9379cec7504 2024-09-13T14:25:36.692 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7 smithi184 running (8m) 20s ago 8m 75.6M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a f37cc947097b 2024-09-13T14:25:36.692 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.a smithi184 *:9095 running (2m) 20s ago 7m 46.3M - 2.51.0 1d3b7f56885b 3da9265b8f14 2024-09-13T14:25:36.762 INFO:tasks.rados.rados.0.smithi151.stdout:1145: finishing write tid 1 to smithi15186934-46 2024-09-13T14:25:36.762 INFO:tasks.rados.rados.0.smithi151.stdout:1145: finishing write tid 2 to smithi15186934-46 2024-09-13T14:25:36.762 INFO:tasks.rados.rados.0.smithi151.stdout:1145: finishing write tid 3 to smithi15186934-46 2024-09-13T14:25:36.762 INFO:tasks.rados.rados.0.smithi151.stdout:1143: done (6 left) 2024-09-13T14:25:36.762 INFO:tasks.rados.rados.0.smithi151.stdout:1144: done (5 left) 2024-09-13T14:25:36.762 INFO:tasks.rados.rados.0.smithi151.stdout:1150: done (4 left) 2024-09-13T14:25:36.762 INFO:tasks.rados.rados.0.smithi151.stdout:1151: snap_create 2024-09-13T14:25:36.762 INFO:tasks.rados.rados.0.smithi151.stdout:1145: finishing write tid 4 to smithi15186934-46 2024-09-13T14:25:36.763 INFO:tasks.rados.rados.0.smithi151.stdout:1145: finishing write tid 5 to smithi15186934-46 2024-09-13T14:25:36.763 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 46 v 360 (ObjNum 451 snap 147 seq_num 451) dirty exists 2024-09-13T14:25:36.763 INFO:tasks.rados.rados.0.smithi151.stdout:1145: left oid 46 (ObjNum 451 snap 147 seq_num 451) 2024-09-13T14:25:36.763 INFO:tasks.rados.rados.0.smithi151.stdout:1146: expect (ObjNum 368 snap 116 seq_num 368) 2024-09-13T14:25:36.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:36 smithi184 ceph-mon[33394]: pgmap v137: 89 pgs: 89 active+clean; 261 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-09-13T14:25:36.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:36 smithi184 ceph-mon[33394]: osdmap e347: 8 total, 8 up, 8 in 2024-09-13T14:25:37.354 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:25:36 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:25:36.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi184\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:25:37.485 INFO:tasks.rados.rados.0.smithi151.stdout:1145: done (4 left) 2024-09-13T14:25:37.485 INFO:tasks.rados.rados.0.smithi151.stdout:1146: done (3 left) 2024-09-13T14:25:37.485 INFO:tasks.rados.rados.0.smithi151.stdout:1148: done (2 left) 2024-09-13T14:25:37.486 INFO:tasks.rados.rados.0.smithi151.stdout:1149: done (1 left) 2024-09-13T14:25:37.486 INFO:tasks.rados.rados.0.smithi151.stdout:1151: done (0 left) 2024-09-13T14:25:37.486 INFO:tasks.rados.rados.0.smithi151.stdout:1152: snap_remove snap 146 2024-09-13T14:25:37.529 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch upgrade pause' 2024-09-13T14:25:37.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:37 smithi184 ceph-mon[33394]: osdmap e348: 8 total, 8 up, 8 in 2024-09-13T14:25:37.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:37 smithi184 ceph-mon[33394]: from='client.44102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:25:37.857 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:37 smithi151 ceph-mon[104450]: osdmap e348: 8 total, 8 up, 8 in 2024-09-13T14:25:37.857 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:37 smithi151 ceph-mon[104450]: from='client.44102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:25:37.857 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:37 smithi151 ceph-mon[108966]: osdmap e348: 8 total, 8 up, 8 in 2024-09-13T14:25:37.857 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:37 smithi151 ceph-mon[108966]: from='client.44102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:25:38.267 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:25:38.486 INFO:tasks.rados.rados.0.smithi151.stdout:1152: done (0 left) 2024-09-13T14:25:38.487 INFO:tasks.rados.rados.0.smithi151.stdout:1153: rollback oid 42 current snap is 148 2024-09-13T14:25:38.487 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 42 to 143 2024-09-13T14:25:38.487 INFO:tasks.rados.rados.0.smithi151.stdout:1154: read oid 44 snap -1 2024-09-13T14:25:38.487 INFO:tasks.rados.rados.0.smithi151.stdout:1154: expect (ObjNum 324 snap 101 seq_num 324) 2024-09-13T14:25:38.487 INFO:tasks.rados.rados.0.smithi151.stdout:1155: read oid 43 snap -1 2024-09-13T14:25:38.487 INFO:tasks.rados.rados.0.smithi151.stdout:1155: expect (ObjNum 379 snap 119 seq_num 379) 2024-09-13T14:25:38.487 INFO:tasks.rados.rados.0.smithi151.stdout:1156: write oid 22 current snap is 148 2024-09-13T14:25:38.487 INFO:tasks.rados.rados.0.smithi151.stdout:1156: seq_num 453 ranges {413571=552624,1641990=762145,2990498=425163} 2024-09-13T14:25:38.492 INFO:tasks.rados.rados.0.smithi151.stdout:1156: writing smithi15186934-22 from 413571 to 966195 tid 1 2024-09-13T14:25:38.500 INFO:tasks.rados.rados.0.smithi151.stdout:1156: writing smithi15186934-22 from 1641990 to 2404135 tid 2 2024-09-13T14:25:38.506 INFO:tasks.rados.rados.0.smithi151.stdout:1156: writing smithi15186934-22 from 2990498 to 3415661 tid 3 2024-09-13T14:25:38.506 INFO:tasks.rados.rados.0.smithi151.stdout:1157: read oid 21 snap -1 2024-09-13T14:25:38.506 INFO:tasks.rados.rados.0.smithi151.stdout:1157: expect deleted 2024-09-13T14:25:38.506 INFO:tasks.rados.rados.0.smithi151.stdout:1158: write oid 48 current snap is 148 2024-09-13T14:25:38.506 INFO:tasks.rados.rados.0.smithi151.stdout:1158: seq_num 454 ranges {430378=627873,1700472=622156,3017921=612457} 2024-09-13T14:25:38.511 INFO:tasks.rados.rados.0.smithi151.stdout:1158: writing smithi15186934-48 from 430378 to 1058251 tid 1 2024-09-13T14:25:38.519 INFO:tasks.rados.rados.0.smithi151.stdout:1158: writing smithi15186934-48 from 1700472 to 2322628 tid 2 2024-09-13T14:25:38.526 INFO:tasks.rados.rados.0.smithi151.stdout:1158: writing smithi15186934-48 from 3017921 to 3630378 tid 3 2024-09-13T14:25:38.526 INFO:tasks.rados.rados.0.smithi151.stdout:1159: snap_create 2024-09-13T14:25:38.526 INFO:tasks.rados.rados.0.smithi151.stdout:1155: expect (ObjNum 379 snap 119 seq_num 379) 2024-09-13T14:25:38.611 INFO:tasks.rados.rados.0.smithi151.stdout:1154: expect (ObjNum 324 snap 101 seq_num 324) 2024-09-13T14:25:38.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:38 smithi151 ceph-mon[104450]: pgmap v140: 89 pgs: 89 active+clean; 263 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.1 MiB/s rd, 444 KiB/s wr, 5 op/s 2024-09-13T14:25:38.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:38 smithi151 ceph-mon[104450]: osdmap e349: 8 total, 8 up, 8 in 2024-09-13T14:25:38.696 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:38 smithi151 ceph-mon[108966]: pgmap v140: 89 pgs: 89 active+clean; 263 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.1 MiB/s rd, 444 KiB/s wr, 5 op/s 2024-09-13T14:25:38.696 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:38 smithi151 ceph-mon[108966]: osdmap e349: 8 total, 8 up, 8 in 2024-09-13T14:25:38.781 INFO:tasks.rados.rados.0.smithi151.stdout:1156: finishing write tid 1 to smithi15186934-22 2024-09-13T14:25:38.781 INFO:tasks.rados.rados.0.smithi151.stdout:1153: finishing rollback tid 0 to smithi15186934-42 2024-09-13T14:25:38.781 INFO:tasks.rados.rados.0.smithi151.stdout:1153: finishing rollback tid 1 to smithi15186934-42 2024-09-13T14:25:38.781 INFO:tasks.rados.rados.0.smithi151.stdout:1153: finishing rollback tid 2 to smithi15186934-42 2024-09-13T14:25:38.781 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 591 (ObjNum 429 snap 135 seq_num 429) dirty exists 2024-09-13T14:25:38.781 INFO:tasks.rados.rados.0.smithi151.stdout:1156: finishing write tid 2 to smithi15186934-22 2024-09-13T14:25:38.781 INFO:tasks.rados.rados.0.smithi151.stdout:1156: finishing write tid 3 to smithi15186934-22 2024-09-13T14:25:38.782 INFO:tasks.rados.rados.0.smithi151.stdout:1156: finishing write tid 4 to smithi15186934-22 2024-09-13T14:25:38.782 INFO:tasks.rados.rados.0.smithi151.stdout:1156: finishing write tid 5 to smithi15186934-22 2024-09-13T14:25:38.782 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 22 v 253 (ObjNum 453 snap 148 seq_num 453) dirty exists 2024-09-13T14:25:38.782 INFO:tasks.rados.rados.0.smithi151.stdout:1156: left oid 22 (ObjNum 453 snap 148 seq_num 453) 2024-09-13T14:25:38.782 INFO:tasks.rados.rados.0.smithi151.stdout:1158: finishing write tid 1 to smithi15186934-48 2024-09-13T14:25:38.782 INFO:tasks.rados.rados.0.smithi151.stdout:1158: finishing write tid 2 to smithi15186934-48 2024-09-13T14:25:38.782 INFO:tasks.rados.rados.0.smithi151.stdout:1158: finishing write tid 3 to smithi15186934-48 2024-09-13T14:25:38.782 INFO:tasks.rados.rados.0.smithi151.stdout:1158: finishing write tid 4 to smithi15186934-48 2024-09-13T14:25:38.782 INFO:tasks.rados.rados.0.smithi151.stdout:1158: finishing write tid 5 to smithi15186934-48 2024-09-13T14:25:38.782 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 48 v 688 (ObjNum 454 snap 148 seq_num 454) dirty exists 2024-09-13T14:25:38.782 INFO:tasks.rados.rados.0.smithi151.stdout:1158: left oid 48 (ObjNum 454 snap 148 seq_num 454) 2024-09-13T14:25:38.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:38 smithi184 ceph-mon[33394]: pgmap v140: 89 pgs: 89 active+clean; 263 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.1 MiB/s rd, 444 KiB/s wr, 5 op/s 2024-09-13T14:25:38.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:38 smithi184 ceph-mon[33394]: osdmap e349: 8 total, 8 up, 8 in 2024-09-13T14:25:39.501 INFO:tasks.rados.rados.0.smithi151.stdout:1153: done (6 left) 2024-09-13T14:25:39.501 INFO:tasks.rados.rados.0.smithi151.stdout:1154: done (5 left) 2024-09-13T14:25:39.501 INFO:tasks.rados.rados.0.smithi151.stdout:1155: done (4 left) 2024-09-13T14:25:39.501 INFO:tasks.rados.rados.0.smithi151.stdout:1156: done (3 left) 2024-09-13T14:25:39.501 INFO:tasks.rados.rados.0.smithi151.stdout:1157: done (2 left) 2024-09-13T14:25:39.501 INFO:tasks.rados.rados.0.smithi151.stdout:1158: done (1 left) 2024-09-13T14:25:39.501 INFO:tasks.rados.rados.0.smithi151.stdout:1159: done (0 left) 2024-09-13T14:25:39.501 INFO:tasks.rados.rados.0.smithi151.stdout:1160: write oid 7 current snap is 149 2024-09-13T14:25:39.501 INFO:tasks.rados.rados.0.smithi151.stdout:1160: seq_num 455 ranges {447185=703122,1642601=765814,2928991=447707} 2024-09-13T14:25:39.507 INFO:tasks.rados.rados.0.smithi151.stdout:1160: writing smithi15186934-7 from 447185 to 1150307 tid 1 2024-09-13T14:25:39.514 INFO:tasks.rados.rados.0.smithi151.stdout:1160: writing smithi15186934-7 from 1642601 to 2408415 tid 2 2024-09-13T14:25:39.519 INFO:tasks.rados.rados.0.smithi151.stdout:1160: writing smithi15186934-7 from 2928991 to 3376698 tid 3 2024-09-13T14:25:39.519 INFO:tasks.rados.rados.0.smithi151.stdout:1161: write oid 19 current snap is 149 2024-09-13T14:25:39.519 INFO:tasks.rados.rados.0.smithi151.stdout:1160: finishing write tid 1 to smithi15186934-7 2024-09-13T14:25:39.519 INFO:tasks.rados.rados.0.smithi151.stdout:1160: finishing write tid 2 to smithi15186934-7 2024-09-13T14:25:39.519 INFO:tasks.rados.rados.0.smithi151.stdout:1161: seq_num 456 ranges {463992=778371,1701083=625825,2956414=658979} 2024-09-13T14:25:39.526 INFO:tasks.rados.rados.0.smithi151.stdout:1161: writing smithi15186934-19 from 463992 to 1242363 tid 1 2024-09-13T14:25:39.532 INFO:tasks.rados.rados.0.smithi151.stdout:1161: writing smithi15186934-19 from 1701083 to 2326908 tid 2 2024-09-13T14:25:39.541 INFO:tasks.rados.rados.0.smithi151.stdout:1161: writing smithi15186934-19 from 2956414 to 3615393 tid 3 2024-09-13T14:25:39.541 INFO:tasks.rados.rados.0.smithi151.stdout:1162: snap_create 2024-09-13T14:25:39.541 INFO:tasks.rados.rados.0.smithi151.stdout:1160: finishing write tid 3 to smithi15186934-7 2024-09-13T14:25:39.541 INFO:tasks.rados.rados.0.smithi151.stdout:1160: finishing write tid 4 to smithi15186934-7 2024-09-13T14:25:39.542 INFO:tasks.rados.rados.0.smithi151.stdout:1160: finishing write tid 5 to smithi15186934-7 2024-09-13T14:25:39.542 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 766 (ObjNum 455 snap 149 seq_num 455) dirty exists 2024-09-13T14:25:39.542 INFO:tasks.rados.rados.0.smithi151.stdout:1160: left oid 7 (ObjNum 455 snap 149 seq_num 455) 2024-09-13T14:25:39.542 INFO:tasks.rados.rados.0.smithi151.stdout:1161: finishing write tid 1 to smithi15186934-19 2024-09-13T14:25:39.542 INFO:tasks.rados.rados.0.smithi151.stdout:1161: finishing write tid 2 to smithi15186934-19 2024-09-13T14:25:39.543 INFO:tasks.rados.rados.0.smithi151.stdout:1161: finishing write tid 3 to smithi15186934-19 2024-09-13T14:25:39.544 INFO:tasks.rados.rados.0.smithi151.stdout:1161: finishing write tid 4 to smithi15186934-19 2024-09-13T14:25:39.544 INFO:tasks.rados.rados.0.smithi151.stdout:1161: finishing write tid 5 to smithi15186934-19 2024-09-13T14:25:39.544 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 19 v 258 (ObjNum 456 snap 149 seq_num 456) dirty exists 2024-09-13T14:25:39.544 INFO:tasks.rados.rados.0.smithi151.stdout:1161: left oid 19 (ObjNum 456 snap 149 seq_num 456) 2024-09-13T14:25:39.810 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:39 smithi151 ceph-mon[104450]: osdmap e350: 8 total, 8 up, 8 in 2024-09-13T14:25:39.810 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:39 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:39.810 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:39 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:39.810 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:39 smithi151 ceph-mon[108966]: osdmap e350: 8 total, 8 up, 8 in 2024-09-13T14:25:39.810 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:39 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:39.811 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:39 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:39.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:39 smithi184 ceph-mon[33394]: osdmap e350: 8 total, 8 up, 8 in 2024-09-13T14:25:39.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:39 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:39.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:39 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:40.239 INFO:teuthology.orchestra.run.smithi151.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:25:40.497 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:25:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:25:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:25:40.509 INFO:tasks.rados.rados.0.smithi151.stdout:1160: done (2 left) 2024-09-13T14:25:40.510 INFO:tasks.rados.rados.0.smithi151.stdout:1161: done (1 left) 2024-09-13T14:25:40.510 INFO:tasks.rados.rados.0.smithi151.stdout:1162: done (0 left) 2024-09-13T14:25:40.510 INFO:tasks.rados.rados.0.smithi151.stdout:1163: read oid 47 snap 147 2024-09-13T14:25:40.510 INFO:tasks.rados.rados.0.smithi151.stdout:1163: expect (ObjNum 417 snap 130 seq_num 417) 2024-09-13T14:25:40.510 INFO:tasks.rados.rados.0.smithi151.stdout:1164: snap_remove snap 101 2024-09-13T14:25:40.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:40 smithi184 ceph-mon[33394]: pgmap v143: 89 pgs: 89 active+clean; 265 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 15 MiB/s rd, 787 KiB/s wr, 13 op/s 2024-09-13T14:25:40.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:40 smithi184 ceph-mon[33394]: osdmap e351: 8 total, 8 up, 8 in 2024-09-13T14:25:40.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:40 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:40.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:40 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:40.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:40 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:40.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:40 smithi151 ceph-mon[108966]: pgmap v143: 89 pgs: 89 active+clean; 265 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 15 MiB/s rd, 787 KiB/s wr, 13 op/s 2024-09-13T14:25:40.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:40 smithi151 ceph-mon[108966]: osdmap e351: 8 total, 8 up, 8 in 2024-09-13T14:25:40.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:40 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:40.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:40 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:40.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:40 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:40.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:40 smithi151 ceph-mon[104450]: pgmap v143: 89 pgs: 89 active+clean; 265 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 15 MiB/s rd, 787 KiB/s wr, 13 op/s 2024-09-13T14:25:40.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:40 smithi151 ceph-mon[104450]: osdmap e351: 8 total, 8 up, 8 in 2024-09-13T14:25:40.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:40 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:40.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:40 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:40.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:40 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:40.976 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'sleep 60' 2024-09-13T14:25:41.525 INFO:tasks.rados.rados.0.smithi151.stdout:1164: done (1 left) 2024-09-13T14:25:41.525 INFO:tasks.rados.rados.0.smithi151.stdout:1165: snap_remove snap 144 2024-09-13T14:25:41.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:41 smithi184 ceph-mon[33394]: from='client.44108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:25:41.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:41 smithi184 ceph-mon[33394]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:25:41.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:41 smithi184 ceph-mon[33394]: osdmap e352: 8 total, 8 up, 8 in 2024-09-13T14:25:41.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:41 smithi151 ceph-mon[104450]: from='client.44108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:25:41.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:41 smithi151 ceph-mon[104450]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:25:41.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:41 smithi151 ceph-mon[104450]: osdmap e352: 8 total, 8 up, 8 in 2024-09-13T14:25:41.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:41 smithi151 ceph-mon[108966]: from='client.44108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:25:41.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:41 smithi151 ceph-mon[108966]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:25:41.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:41 smithi151 ceph-mon[108966]: osdmap e352: 8 total, 8 up, 8 in 2024-09-13T14:25:42.517 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:25:42.522 INFO:tasks.rados.rados.0.smithi151.stdout:1165: done (1 left) 2024-09-13T14:25:42.522 INFO:tasks.rados.rados.0.smithi151.stdout:1166: delete oid 21 current snap is 150 2024-09-13T14:25:42.522 INFO:tasks.rados.rados.0.smithi151.stdout:1163: expect (ObjNum 417 snap 130 seq_num 417) 2024-09-13T14:25:42.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:42 smithi151 ceph-mon[104450]: pgmap v146: 89 pgs: 89 active+clean; 265 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.9 MiB/s rd, 343 KiB/s wr, 7 op/s 2024-09-13T14:25:42.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:42 smithi151 ceph-mon[104450]: osdmap e353: 8 total, 8 up, 8 in 2024-09-13T14:25:42.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:42 smithi151 ceph-mon[108966]: pgmap v146: 89 pgs: 89 active+clean; 265 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.9 MiB/s rd, 343 KiB/s wr, 7 op/s 2024-09-13T14:25:42.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:42 smithi151 ceph-mon[108966]: osdmap e353: 8 total, 8 up, 8 in 2024-09-13T14:25:42.696 INFO:tasks.rados.rados.0.smithi151.stdout:1163: done (1 left) 2024-09-13T14:25:42.696 INFO:tasks.rados.rados.0.smithi151.stdout:1166: done (0 left) 2024-09-13T14:25:42.696 INFO:tasks.rados.rados.0.smithi151.stdout:1167: rollback oid 19 current snap is 150 2024-09-13T14:25:42.696 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 19 to 123 2024-09-13T14:25:42.696 INFO:tasks.rados.rados.0.smithi151.stdout:1168: read oid 5 snap -1 2024-09-13T14:25:42.696 INFO:tasks.rados.rados.0.smithi151.stdout:1168: expect (ObjNum 444 snap 141 seq_num 444) 2024-09-13T14:25:42.697 INFO:tasks.rados.rados.0.smithi151.stdout:1169: write oid 1 current snap is 150 2024-09-13T14:25:42.697 INFO:tasks.rados.rados.0.smithi151.stdout:1169: seq_num 458 ranges {497606=645222,1934400=629494,3243966=453640} 2024-09-13T14:25:42.702 INFO:tasks.rados.rados.0.smithi151.stdout:1169: writing smithi15186934-1 from 497606 to 1142828 tid 1 2024-09-13T14:25:42.710 INFO:tasks.rados.rados.0.smithi151.stdout:1169: writing smithi15186934-1 from 1934400 to 2563894 tid 2 2024-09-13T14:25:42.717 INFO:tasks.rados.rados.0.smithi151.stdout:1169: writing smithi15186934-1 from 3243966 to 3697606 tid 3 2024-09-13T14:25:42.717 INFO:tasks.rados.rados.0.smithi151.stdout:1170: rollback oid 10 current snap is 150 2024-09-13T14:25:42.717 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 10 to 143 2024-09-13T14:25:42.717 INFO:tasks.rados.rados.0.smithi151.stdout:1167: finishing rollback tid 0 to smithi15186934-19 2024-09-13T14:25:42.717 INFO:tasks.rados.rados.0.smithi151.stdout:1167: finishing rollback tid 1 to smithi15186934-19 2024-09-13T14:25:42.717 INFO:tasks.rados.rados.0.smithi151.stdout:1167: finishing rollback tid 2 to smithi15186934-19 2024-09-13T14:25:42.717 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 19 v 268 (ObjNum 323 snap 101 seq_num 323) dirty exists 2024-09-13T14:25:42.717 INFO:tasks.rados.rados.0.smithi151.stdout:1168: expect (ObjNum 444 snap 141 seq_num 444) 2024-09-13T14:25:42.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:42 smithi184 ceph-mon[33394]: pgmap v146: 89 pgs: 89 active+clean; 265 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.9 MiB/s rd, 343 KiB/s wr, 7 op/s 2024-09-13T14:25:42.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:42 smithi184 ceph-mon[33394]: osdmap e353: 8 total, 8 up, 8 in 2024-09-13T14:25:42.872 INFO:tasks.rados.rados.0.smithi151.stdout:1169: finishing write tid 1 to smithi15186934-1 2024-09-13T14:25:42.872 INFO:tasks.rados.rados.0.smithi151.stdout:1169: finishing write tid 2 to smithi15186934-1 2024-09-13T14:25:42.872 INFO:tasks.rados.rados.0.smithi151.stdout:1167: done (3 left) 2024-09-13T14:25:42.872 INFO:tasks.rados.rados.0.smithi151.stdout:1168: done (2 left) 2024-09-13T14:25:42.872 INFO:tasks.rados.rados.0.smithi151.stdout:1171: read oid 14 snap -1 2024-09-13T14:25:42.872 INFO:tasks.rados.rados.0.smithi151.stdout:1171: expect (ObjNum 440 snap 139 seq_num 440) 2024-09-13T14:25:42.872 INFO:tasks.rados.rados.0.smithi151.stdout:1172: write oid 20 current snap is 150 2024-09-13T14:25:42.872 INFO:tasks.rados.rados.0.smithi151.stdout:1170: finishing rollback tid 0 to smithi15186934-10 2024-09-13T14:25:42.872 INFO:tasks.rados.rados.0.smithi151.stdout:1172: seq_num 459 ranges {514413=720471,1876529=773152,3438683=275730} 2024-09-13T14:25:42.878 INFO:tasks.rados.rados.0.smithi151.stdout:1172: writing smithi15186934-20 from 514413 to 1234884 tid 1 2024-09-13T14:25:42.887 INFO:tasks.rados.rados.0.smithi151.stdout:1172: writing smithi15186934-20 from 1876529 to 2649681 tid 2 2024-09-13T14:25:42.893 INFO:tasks.rados.rados.0.smithi151.stdout:1172: writing smithi15186934-20 from 3438683 to 3714413 tid 3 2024-09-13T14:25:42.893 INFO:tasks.rados.rados.0.smithi151.stdout:1173: write oid 45 current snap is 150 2024-09-13T14:25:42.893 INFO:tasks.rados.rados.0.smithi151.stdout:1173: seq_num 460 ranges {531220=795720,1935011=516810,3066106=420420} 2024-09-13T14:25:42.901 INFO:tasks.rados.rados.0.smithi151.stdout:1173: writing smithi15186934-45 from 531220 to 1326940 tid 1 2024-09-13T14:25:42.907 INFO:tasks.rados.rados.0.smithi151.stdout:1173: writing smithi15186934-45 from 1935011 to 2451821 tid 2 2024-09-13T14:25:42.913 INFO:tasks.rados.rados.0.smithi151.stdout:1173: writing smithi15186934-45 from 3066106 to 3486526 tid 3 2024-09-13T14:25:42.913 INFO:tasks.rados.rados.0.smithi151.stdout:1174: write oid 37 current snap is 150 2024-09-13T14:25:42.913 INFO:tasks.rados.rados.0.smithi151.stdout:1170: finishing rollback tid 1 to smithi15186934-10 2024-09-13T14:25:42.913 INFO:tasks.rados.rados.0.smithi151.stdout:1169: finishing write tid 3 to smithi15186934-1 2024-09-13T14:25:42.913 INFO:tasks.rados.rados.0.smithi151.stdout:1170: finishing rollback tid 2 to smithi15186934-10 2024-09-13T14:25:42.914 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 380 (ObjNum 437 snap 139 seq_num 437) dirty exists 2024-09-13T14:25:42.914 INFO:tasks.rados.rados.0.smithi151.stdout:1169: finishing write tid 4 to smithi15186934-1 2024-09-13T14:25:42.914 INFO:tasks.rados.rados.0.smithi151.stdout:1174: seq_num 461 ranges {548027=470969,1593493=776821,3093529=631692} 2024-09-13T14:25:42.918 INFO:tasks.rados.rados.0.smithi151.stdout:1174: writing smithi15186934-37 from 548027 to 1018996 tid 1 2024-09-13T14:25:42.926 INFO:tasks.rados.rados.0.smithi151.stdout:1174: writing smithi15186934-37 from 1593493 to 2370314 tid 2 2024-09-13T14:25:42.934 INFO:tasks.rados.rados.0.smithi151.stdout:1174: writing smithi15186934-37 from 3093529 to 3725221 tid 3 2024-09-13T14:25:42.934 INFO:tasks.rados.rados.0.smithi151.stdout:1170: done (5 left) 2024-09-13T14:25:42.935 INFO:tasks.rados.rados.0.smithi151.stdout:1175: rollback oid 48 current snap is 150 2024-09-13T14:25:42.935 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 48 to 107 2024-09-13T14:25:42.935 INFO:tasks.rados.rados.0.smithi151.stdout:1169: finishing write tid 5 to smithi15186934-1 2024-09-13T14:25:42.935 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 1 v 306 (ObjNum 458 snap 150 seq_num 458) dirty exists 2024-09-13T14:25:42.935 INFO:tasks.rados.rados.0.smithi151.stdout:1169: left oid 1 (ObjNum 458 snap 150 seq_num 458) 2024-09-13T14:25:42.935 INFO:tasks.rados.rados.0.smithi151.stdout:1169: done (5 left) 2024-09-13T14:25:42.935 INFO:tasks.rados.rados.0.smithi151.stdout:1176: write oid 49 current snap is 150 2024-09-13T14:25:42.935 INFO:tasks.rados.rados.0.smithi151.stdout:1171: expect (ObjNum 440 snap 139 seq_num 440) 2024-09-13T14:25:43.071 INFO:tasks.rados.rados.0.smithi151.stdout:1172: finishing write tid 1 to smithi15186934-20 2024-09-13T14:25:43.071 INFO:tasks.rados.rados.0.smithi151.stdout:1172: finishing write tid 2 to smithi15186934-20 2024-09-13T14:25:43.071 INFO:tasks.rados.rados.0.smithi151.stdout:1172: finishing write tid 3 to smithi15186934-20 2024-09-13T14:25:43.071 INFO:tasks.rados.rados.0.smithi151.stdout:1172: finishing write tid 4 to smithi15186934-20 2024-09-13T14:25:43.071 INFO:tasks.rados.rados.0.smithi151.stdout:1172: finishing write tid 5 to smithi15186934-20 2024-09-13T14:25:43.072 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 299 (ObjNum 459 snap 150 seq_num 459) dirty exists 2024-09-13T14:25:43.072 INFO:tasks.rados.rados.0.smithi151.stdout:1172: left oid 20 (ObjNum 459 snap 150 seq_num 459) 2024-09-13T14:25:43.072 INFO:tasks.rados.rados.0.smithi151.stdout:1173: finishing write tid 1 to smithi15186934-45 2024-09-13T14:25:43.072 INFO:tasks.rados.rados.0.smithi151.stdout:1173: finishing write tid 2 to smithi15186934-45 2024-09-13T14:25:43.072 INFO:tasks.rados.rados.0.smithi151.stdout:1173: finishing write tid 3 to smithi15186934-45 2024-09-13T14:25:43.072 INFO:tasks.rados.rados.0.smithi151.stdout:1173: finishing write tid 4 to smithi15186934-45 2024-09-13T14:25:43.072 INFO:tasks.rados.rados.0.smithi151.stdout:1173: finishing write tid 5 to smithi15186934-45 2024-09-13T14:25:43.072 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 700 (ObjNum 460 snap 150 seq_num 460) dirty exists 2024-09-13T14:25:43.072 INFO:tasks.rados.rados.0.smithi151.stdout:1173: left oid 45 (ObjNum 460 snap 150 seq_num 460) 2024-09-13T14:25:43.072 INFO:tasks.rados.rados.0.smithi151.stdout:1174: finishing write tid 1 to smithi15186934-37 2024-09-13T14:25:43.072 INFO:tasks.rados.rados.0.smithi151.stdout:1176: seq_num 462 ranges {564834=546218,1651975=520479,2720952=559317} 2024-09-13T14:25:43.077 INFO:tasks.rados.rados.0.smithi151.stdout:1176: writing smithi15186934-49 from 564834 to 1111052 tid 1 2024-09-13T14:25:43.083 INFO:tasks.rados.rados.0.smithi151.stdout:1176: writing smithi15186934-49 from 1651975 to 2172454 tid 2 2024-09-13T14:25:43.090 INFO:tasks.rados.rados.0.smithi151.stdout:1176: writing smithi15186934-49 from 2720952 to 3280269 tid 3 2024-09-13T14:25:43.090 INFO:tasks.rados.rados.0.smithi151.stdout:1171: done (5 left) 2024-09-13T14:25:43.090 INFO:tasks.rados.rados.0.smithi151.stdout:1172: done (4 left) 2024-09-13T14:25:43.090 INFO:tasks.rados.rados.0.smithi151.stdout:1173: done (3 left) 2024-09-13T14:25:43.090 INFO:tasks.rados.rados.0.smithi151.stdout:1177: snap_create 2024-09-13T14:25:43.090 INFO:tasks.rados.rados.0.smithi151.stdout:1174: finishing write tid 2 to smithi15186934-37 2024-09-13T14:25:43.090 INFO:tasks.rados.rados.0.smithi151.stdout:1174: finishing write tid 3 to smithi15186934-37 2024-09-13T14:25:43.090 INFO:tasks.rados.rados.0.smithi151.stdout:1174: finishing write tid 4 to smithi15186934-37 2024-09-13T14:25:43.091 INFO:tasks.rados.rados.0.smithi151.stdout:1175: finishing rollback tid 0 to smithi15186934-48 2024-09-13T14:25:43.091 INFO:tasks.rados.rados.0.smithi151.stdout:1174: finishing write tid 5 to smithi15186934-37 2024-09-13T14:25:43.091 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 37 v 547 (ObjNum 461 snap 150 seq_num 461) dirty exists 2024-09-13T14:25:43.091 INFO:tasks.rados.rados.0.smithi151.stdout:1174: left oid 37 (ObjNum 461 snap 150 seq_num 461) 2024-09-13T14:25:43.091 INFO:tasks.rados.rados.0.smithi151.stdout:1175: finishing rollback tid 1 to smithi15186934-48 2024-09-13T14:25:43.091 INFO:tasks.rados.rados.0.smithi151.stdout:1175: finishing rollback tid 2 to smithi15186934-48 2024-09-13T14:25:43.091 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 48 v 704 (ObjNum 342 snap 105 seq_num 342) dirty exists 2024-09-13T14:25:43.091 INFO:tasks.rados.rados.0.smithi151.stdout:1176: finishing write tid 1 to smithi15186934-49 2024-09-13T14:25:43.091 INFO:tasks.rados.rados.0.smithi151.stdout:1176: finishing write tid 2 to smithi15186934-49 2024-09-13T14:25:43.092 INFO:tasks.rados.rados.0.smithi151.stdout:1176: finishing write tid 3 to smithi15186934-49 2024-09-13T14:25:43.092 INFO:tasks.rados.rados.0.smithi151.stdout:1176: finishing write tid 4 to smithi15186934-49 2024-09-13T14:25:43.092 INFO:tasks.rados.rados.0.smithi151.stdout:1176: finishing write tid 5 to smithi15186934-49 2024-09-13T14:25:43.093 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 49 v 552 (ObjNum 462 snap 150 seq_num 462) dirty exists 2024-09-13T14:25:43.093 INFO:tasks.rados.rados.0.smithi151.stdout:1176: left oid 49 (ObjNum 462 snap 150 seq_num 462) 2024-09-13T14:25:43.531 INFO:tasks.rados.rados.0.smithi151.stdout:1174: done (3 left) 2024-09-13T14:25:43.531 INFO:tasks.rados.rados.0.smithi151.stdout:1175: done (2 left) 2024-09-13T14:25:43.531 INFO:tasks.rados.rados.0.smithi151.stdout:1176: done (1 left) 2024-09-13T14:25:43.531 INFO:tasks.rados.rados.0.smithi151.stdout:1177: done (0 left) 2024-09-13T14:25:43.531 INFO:tasks.rados.rados.0.smithi151.stdout:1178: rollback oid 13 current snap is 151 2024-09-13T14:25:43.531 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 13 to 139 2024-09-13T14:25:43.532 INFO:tasks.rados.rados.0.smithi151.stdout:1179: read oid 18 snap 123 2024-09-13T14:25:43.532 INFO:tasks.rados.rados.0.smithi151.stdout:1179: expect (ObjNum 368 snap 116 seq_num 368) 2024-09-13T14:25:43.532 INFO:tasks.rados.rados.0.smithi151.stdout:1180: delete oid 24 current snap is 151 2024-09-13T14:25:43.533 INFO:tasks.rados.rados.0.smithi151.stdout:1179: expect (ObjNum 368 snap 116 seq_num 368) 2024-09-13T14:25:43.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:43 smithi151 ceph-mon[108966]: osdmap e354: 8 total, 8 up, 8 in 2024-09-13T14:25:43.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:43 smithi151 ceph-mon[104450]: osdmap e354: 8 total, 8 up, 8 in 2024-09-13T14:25:43.654 INFO:tasks.rados.rados.0.smithi151.stdout:1178: finishing rollback tid 0 to smithi15186934-13 2024-09-13T14:25:43.654 INFO:tasks.rados.rados.0.smithi151.stdout:1178: finishing rollback tid 1 to smithi15186934-13 2024-09-13T14:25:43.654 INFO:tasks.rados.rados.0.smithi151.stdout:1178: finishing rollback tid 2 to smithi15186934-13 2024-09-13T14:25:43.654 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 13 v 405 (ObjNum 270 snap 86 seq_num 270) dirty exists 2024-09-13T14:25:43.654 INFO:tasks.rados.rados.0.smithi151.stdout:1178: done (2 left) 2024-09-13T14:25:43.654 INFO:tasks.rados.rados.0.smithi151.stdout:1179: done (1 left) 2024-09-13T14:25:43.654 INFO:tasks.rados.rados.0.smithi151.stdout:1180: done (0 left) 2024-09-13T14:25:43.655 INFO:tasks.rados.rados.0.smithi151.stdout:1181: snap_create 2024-09-13T14:25:43.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:43 smithi184 ceph-mon[33394]: osdmap e354: 8 total, 8 up, 8 in 2024-09-13T14:25:44.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:25:44 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:25:44.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:25:44.461 INFO:tasks.rados.rados.0.smithi151.stdout:1181: done (0 left) 2024-09-13T14:25:44.461 INFO:tasks.rados.rados.0.smithi151.stdout:1182: snap_remove snap 150 2024-09-13T14:25:44.734 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:25:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:25:44] "GET /metrics HTTP/1.1" 200 36399 "" "Prometheus/2.51.0" 2024-09-13T14:25:44.735 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:44 smithi151 ceph-mon[104450]: pgmap v149: 89 pgs: 89 active+clean; 250 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.8 MiB/s wr, 12 op/s 2024-09-13T14:25:44.735 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:44 smithi151 ceph-mon[104450]: osdmap e355: 8 total, 8 up, 8 in 2024-09-13T14:25:44.735 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:44 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:44.735 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:44 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:44.735 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:44 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:25:44.735 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:44 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:25:44.735 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:44 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:44.735 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:44 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:25:44.735 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:44 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:25:44.735 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:44 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:25:44.735 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:44 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:44.735 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:44 smithi151 ceph-mon[104450]: osdmap e356: 8 total, 8 up, 8 in 2024-09-13T14:25:44.736 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:44 smithi151 ceph-mon[108966]: pgmap v149: 89 pgs: 89 active+clean; 250 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.8 MiB/s wr, 12 op/s 2024-09-13T14:25:44.736 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:44 smithi151 ceph-mon[108966]: osdmap e355: 8 total, 8 up, 8 in 2024-09-13T14:25:44.736 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:44 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:44.736 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:44 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:44.736 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:44 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:25:44.736 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:44 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:25:44.736 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:44 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:44.736 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:44 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:25:44.736 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:44 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:25:44.736 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:44 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:25:44.737 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:44 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:44.737 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:44 smithi151 ceph-mon[108966]: osdmap e356: 8 total, 8 up, 8 in 2024-09-13T14:25:44.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:44 smithi184 ceph-mon[33394]: pgmap v149: 89 pgs: 89 active+clean; 250 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.8 MiB/s wr, 12 op/s 2024-09-13T14:25:44.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:44 smithi184 ceph-mon[33394]: osdmap e355: 8 total, 8 up, 8 in 2024-09-13T14:25:44.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:44 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:44.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:44 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:44.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:44 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:25:44.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:44 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:25:44.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:44 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:44.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:44 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:25:44.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:44 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:25:44.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:44 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:25:44.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:44 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:25:44.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:44 smithi184 ceph-mon[33394]: osdmap e356: 8 total, 8 up, 8 in 2024-09-13T14:25:45.463 INFO:tasks.rados.rados.0.smithi151.stdout:1182: done (0 left) 2024-09-13T14:25:45.464 INFO:tasks.rados.rados.0.smithi151.stdout:1183: snap_remove snap 135 2024-09-13T14:25:46.103 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:25:45 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:25:45.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:25:46.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:25:45 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:25:45.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:25:46.466 INFO:tasks.rados.rados.0.smithi151.stdout:1183: done (0 left) 2024-09-13T14:25:46.466 INFO:tasks.rados.rados.0.smithi151.stdout:1184: rollback oid 2 current snap is 152 2024-09-13T14:25:46.466 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 2 to 151 2024-09-13T14:25:46.466 INFO:tasks.rados.rados.0.smithi151.stdout:1185: write oid 26 current snap is 152 2024-09-13T14:25:46.466 INFO:tasks.rados.rados.0.smithi151.stdout:1185: seq_num 464 ranges {598448=413069,1768939=524148,2775798=581861,3781488=16960} 2024-09-13T14:25:46.472 INFO:tasks.rados.rados.0.smithi151.stdout:1185: writing smithi15186934-26 from 598448 to 1011517 tid 1 2024-09-13T14:25:46.480 INFO:tasks.rados.rados.0.smithi151.stdout:1185: writing smithi15186934-26 from 1768939 to 2293087 tid 2 2024-09-13T14:25:46.486 INFO:tasks.rados.rados.0.smithi151.stdout:1185: writing smithi15186934-26 from 2775798 to 3357659 tid 3 2024-09-13T14:25:46.489 INFO:tasks.rados.rados.0.smithi151.stdout:1185: writing smithi15186934-26 from 3781488 to 3798448 tid 4 2024-09-13T14:25:46.489 INFO:tasks.rados.rados.0.smithi151.stdout:1186: rollback oid 44 current snap is 152 2024-09-13T14:25:46.489 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 44 to 129 2024-09-13T14:25:46.489 INFO:tasks.rados.rados.0.smithi151.stdout:1184: finishing rollback tid 0 to smithi15186934-2 2024-09-13T14:25:46.489 INFO:tasks.rados.rados.0.smithi151.stdout:1184: finishing rollback tid 1 to smithi15186934-2 2024-09-13T14:25:46.489 INFO:tasks.rados.rados.0.smithi151.stdout:1184: finishing rollback tid 2 to smithi15186934-2 2024-09-13T14:25:46.489 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 2 v 411 (ObjNum 377 snap 119 seq_num 377) dirty exists 2024-09-13T14:25:46.489 INFO:tasks.rados.rados.0.smithi151.stdout:1185: finishing write tid 1 to smithi15186934-26 2024-09-13T14:25:46.489 INFO:tasks.rados.rados.0.smithi151.stdout:1185: finishing write tid 2 to smithi15186934-26 2024-09-13T14:25:46.489 INFO:tasks.rados.rados.0.smithi151.stdout:1185: finishing write tid 3 to smithi15186934-26 2024-09-13T14:25:46.489 INFO:tasks.rados.rados.0.smithi151.stdout:1184: done (2 left) 2024-09-13T14:25:46.490 INFO:tasks.rados.rados.0.smithi151.stdout:1187: rollback oid 14 current snap is 152 2024-09-13T14:25:46.490 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 14 to 140 2024-09-13T14:25:46.490 INFO:tasks.rados.rados.0.smithi151.stdout:1188: snap_remove snap 137 2024-09-13T14:25:46.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:46 smithi184 ceph-mon[33394]: pgmap v152: 89 pgs: 89 active+clean; 250 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.8 MiB/s wr, 12 op/s 2024-09-13T14:25:46.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:46 smithi184 ceph-mon[33394]: osdmap e357: 8 total, 8 up, 8 in 2024-09-13T14:25:46.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:46 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:25:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:46 smithi151 ceph-mon[104450]: pgmap v152: 89 pgs: 89 active+clean; 250 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.8 MiB/s wr, 12 op/s 2024-09-13T14:25:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:46 smithi151 ceph-mon[104450]: osdmap e357: 8 total, 8 up, 8 in 2024-09-13T14:25:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:25:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:46 smithi151 ceph-mon[108966]: pgmap v152: 89 pgs: 89 active+clean; 250 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.8 MiB/s wr, 12 op/s 2024-09-13T14:25:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:46 smithi151 ceph-mon[108966]: osdmap e357: 8 total, 8 up, 8 in 2024-09-13T14:25:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:25:47.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:25:46 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:25:46.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi184\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:25:47.489 INFO:tasks.rados.rados.0.smithi151.stdout:1188: done (3 left) 2024-09-13T14:25:47.489 INFO:tasks.rados.rados.0.smithi151.stdout:1189: write oid 29 current snap is 152 2024-09-13T14:25:47.489 INFO:tasks.rados.rados.0.smithi151.stdout:1185: finishing write tid 4 to smithi15186934-26 2024-09-13T14:25:47.489 INFO:tasks.rados.rados.0.smithi151.stdout:1185: finishing write tid 5 to smithi15186934-26 2024-09-13T14:25:47.489 INFO:tasks.rados.rados.0.smithi151.stdout:1189: seq_num 465 ranges {615255=488318,1827421=784159,3203221=612034} 2024-09-13T14:25:47.495 INFO:tasks.rados.rados.0.smithi151.stdout:1189: writing smithi15186934-29 from 615255 to 1103573 tid 1 2024-09-13T14:25:47.503 INFO:tasks.rados.rados.0.smithi151.stdout:1189: writing smithi15186934-29 from 1827421 to 2611580 tid 2 2024-09-13T14:25:47.510 INFO:tasks.rados.rados.0.smithi151.stdout:1189: writing smithi15186934-29 from 3203221 to 3815255 tid 3 2024-09-13T14:25:47.510 INFO:tasks.rados.rados.0.smithi151.stdout:1190: snap_remove snap 147 2024-09-13T14:25:47.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:47 smithi184 ceph-mon[33394]: osdmap e358: 8 total, 8 up, 8 in 2024-09-13T14:25:47.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:47 smithi151 ceph-mon[104450]: osdmap e358: 8 total, 8 up, 8 in 2024-09-13T14:25:47.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:47 smithi151 ceph-mon[108966]: osdmap e358: 8 total, 8 up, 8 in 2024-09-13T14:25:48.489 INFO:tasks.rados.rados.0.smithi151.stdout:1190: done (4 left) 2024-09-13T14:25:48.489 INFO:tasks.rados.rados.0.smithi151.stdout:1191: rollback oid 41 current snap is 152 2024-09-13T14:25:48.489 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 41 to 149 2024-09-13T14:25:48.489 INFO:tasks.rados.rados.0.smithi151.stdout:1185: finishing write tid 6 to smithi15186934-26 2024-09-13T14:25:48.489 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 390 (ObjNum 464 snap 152 seq_num 464) dirty exists 2024-09-13T14:25:48.489 INFO:tasks.rados.rados.0.smithi151.stdout:1185: left oid 26 (ObjNum 464 snap 152 seq_num 464) 2024-09-13T14:25:48.489 INFO:tasks.rados.rados.0.smithi151.stdout:1186: finishing rollback tid 0 to smithi15186934-44 2024-09-13T14:25:48.489 INFO:tasks.rados.rados.0.smithi151.stdout:1187: finishing rollback tid 0 to smithi15186934-14 2024-09-13T14:25:48.489 INFO:tasks.rados.rados.0.smithi151.stdout:1186: finishing rollback tid 1 to smithi15186934-44 2024-09-13T14:25:48.489 INFO:tasks.rados.rados.0.smithi151.stdout:1187: finishing rollback tid 1 to smithi15186934-14 2024-09-13T14:25:48.489 INFO:tasks.rados.rados.0.smithi151.stdout:1186: finishing rollback tid 2 to smithi15186934-44 2024-09-13T14:25:48.490 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 312 (ObjNum 324 snap 101 seq_num 324) dirty exists 2024-09-13T14:25:48.490 INFO:tasks.rados.rados.0.smithi151.stdout:1187: finishing rollback tid 2 to smithi15186934-14 2024-09-13T14:25:48.490 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 484 (ObjNum 440 snap 139 seq_num 440) dirty exists 2024-09-13T14:25:48.490 INFO:tasks.rados.rados.0.smithi151.stdout:1189: finishing write tid 1 to smithi15186934-29 2024-09-13T14:25:48.490 INFO:tasks.rados.rados.0.smithi151.stdout:1189: finishing write tid 2 to smithi15186934-29 2024-09-13T14:25:48.490 INFO:tasks.rados.rados.0.smithi151.stdout:1189: finishing write tid 3 to smithi15186934-29 2024-09-13T14:25:48.490 INFO:tasks.rados.rados.0.smithi151.stdout:1189: finishing write tid 4 to smithi15186934-29 2024-09-13T14:25:48.490 INFO:tasks.rados.rados.0.smithi151.stdout:1189: finishing write tid 5 to smithi15186934-29 2024-09-13T14:25:48.490 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 29 v 318 (ObjNum 465 snap 152 seq_num 465) dirty exists 2024-09-13T14:25:48.490 INFO:tasks.rados.rados.0.smithi151.stdout:1189: left oid 29 (ObjNum 465 snap 152 seq_num 465) 2024-09-13T14:25:48.490 INFO:tasks.rados.rados.0.smithi151.stdout:1185: done (4 left) 2024-09-13T14:25:48.490 INFO:tasks.rados.rados.0.smithi151.stdout:1186: done (3 left) 2024-09-13T14:25:48.490 INFO:tasks.rados.rados.0.smithi151.stdout:1187: done (2 left) 2024-09-13T14:25:48.490 INFO:tasks.rados.rados.0.smithi151.stdout:1189: done (1 left) 2024-09-13T14:25:48.490 INFO:tasks.rados.rados.0.smithi151.stdout:1192: delete oid 2 current snap is 152 2024-09-13T14:25:48.495 INFO:tasks.rados.rados.0.smithi151.stdout:1191: finishing rollback tid 1 to smithi15186934-41 2024-09-13T14:25:48.495 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 0 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-13T14:25:48.498 INFO:tasks.rados.rados.0.smithi151.stdout:1191: done (1 left) 2024-09-13T14:25:48.498 INFO:tasks.rados.rados.0.smithi151.stdout:1192: done (0 left) 2024-09-13T14:25:48.498 INFO:tasks.rados.rados.0.smithi151.stdout:1193: rollback oid 48 current snap is 152 2024-09-13T14:25:48.498 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 48 to 139 2024-09-13T14:25:48.498 INFO:tasks.rados.rados.0.smithi151.stdout:1194: snap_create 2024-09-13T14:25:48.501 INFO:tasks.rados.rados.0.smithi151.stdout:1193: finishing rollback tid 0 to smithi15186934-48 2024-09-13T14:25:48.501 INFO:tasks.rados.rados.0.smithi151.stdout:1193: finishing rollback tid 1 to smithi15186934-48 2024-09-13T14:25:48.501 INFO:tasks.rados.rados.0.smithi151.stdout:1193: finishing rollback tid 2 to smithi15186934-48 2024-09-13T14:25:48.501 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 48 v 714 (ObjNum 434 snap 138 seq_num 434) dirty exists 2024-09-13T14:25:48.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:48 smithi184 ceph-mon[33394]: pgmap v155: 89 pgs: 89 active+clean; 268 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.9 MiB/s rd, 2.1 MiB/s wr, 17 op/s 2024-09-13T14:25:48.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:48 smithi184 ceph-mon[33394]: osdmap e359: 8 total, 8 up, 8 in 2024-09-13T14:25:48.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:48 smithi151 ceph-mon[104450]: pgmap v155: 89 pgs: 89 active+clean; 268 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.9 MiB/s rd, 2.1 MiB/s wr, 17 op/s 2024-09-13T14:25:48.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:48 smithi151 ceph-mon[104450]: osdmap e359: 8 total, 8 up, 8 in 2024-09-13T14:25:48.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:48 smithi151 ceph-mon[108966]: pgmap v155: 89 pgs: 89 active+clean; 268 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.9 MiB/s rd, 2.1 MiB/s wr, 17 op/s 2024-09-13T14:25:48.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:48 smithi151 ceph-mon[108966]: osdmap e359: 8 total, 8 up, 8 in 2024-09-13T14:25:49.496 INFO:tasks.rados.rados.0.smithi151.stdout:1193: done (1 left) 2024-09-13T14:25:49.496 INFO:tasks.rados.rados.0.smithi151.stdout:1194: done (0 left) 2024-09-13T14:25:49.496 INFO:tasks.rados.rados.0.smithi151.stdout:1195: delete oid 11 current snap is 153 2024-09-13T14:25:49.497 INFO:tasks.rados.rados.0.smithi151.stdout:1195: done (0 left) 2024-09-13T14:25:49.497 INFO:tasks.rados.rados.0.smithi151.stdout:1196: delete oid 37 current snap is 153 2024-09-13T14:25:49.501 INFO:tasks.rados.rados.0.smithi151.stdout:1196: done (0 left) 2024-09-13T14:25:49.501 INFO:tasks.rados.rados.0.smithi151.stdout:1197: read oid 46 snap -1 2024-09-13T14:25:49.501 INFO:tasks.rados.rados.0.smithi151.stdout:1197: expect (ObjNum 451 snap 147 seq_num 451) 2024-09-13T14:25:49.501 INFO:tasks.rados.rados.0.smithi151.stdout:1198: rollback oid 6 current snap is 153 2024-09-13T14:25:49.502 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 6 to 140 2024-09-13T14:25:49.502 INFO:tasks.rados.rados.0.smithi151.stdout:1199: snap_create 2024-09-13T14:25:49.504 INFO:tasks.rados.rados.0.smithi151.stdout:1198: finishing rollback tid 0 to smithi15186934-6 2024-09-13T14:25:49.504 INFO:tasks.rados.rados.0.smithi151.stdout:1198: finishing rollback tid 1 to smithi15186934-6 2024-09-13T14:25:49.505 INFO:tasks.rados.rados.0.smithi151.stdout:1198: finishing rollback tid 2 to smithi15186934-6 2024-09-13T14:25:49.505 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 6 v 278 (ObjNum 426 snap 135 seq_num 426) dirty exists 2024-09-13T14:25:49.507 INFO:tasks.rados.rados.0.smithi151.stdout:1197: expect (ObjNum 451 snap 147 seq_num 451) 2024-09-13T14:25:49.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:49 smithi184 ceph-mon[33394]: osdmap e360: 8 total, 8 up, 8 in 2024-09-13T14:25:49.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:49 smithi151 ceph-mon[104450]: osdmap e360: 8 total, 8 up, 8 in 2024-09-13T14:25:49.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:49 smithi151 ceph-mon[108966]: osdmap e360: 8 total, 8 up, 8 in 2024-09-13T14:25:50.493 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:25:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:25:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:25:50.503 INFO:tasks.rados.rados.0.smithi151.stdout:1197: done (2 left) 2024-09-13T14:25:50.503 INFO:tasks.rados.rados.0.smithi151.stdout:1198: done (1 left) 2024-09-13T14:25:50.504 INFO:tasks.rados.rados.0.smithi151.stdout:1199: done (0 left) 2024-09-13T14:25:50.504 INFO:tasks.rados.rados.0.smithi151.stdout:1200: write oid 29 current snap is 154 2024-09-13T14:25:50.504 INFO:tasks.rados.rados.0.smithi151.stdout:1200: seq_num 469 ranges {682483=789314,1944996=791497,3196560=554574} 2024-09-13T14:25:50.510 INFO:tasks.rados.rados.0.smithi151.stdout:1200: writing smithi15186934-29 from 682483 to 1471797 tid 1 2024-09-13T14:25:50.518 INFO:tasks.rados.rados.0.smithi151.stdout:1200: writing smithi15186934-29 from 1944996 to 2736493 tid 2 2024-09-13T14:25:50.523 INFO:tasks.rados.rados.0.smithi151.stdout:1200: writing smithi15186934-29 from 3196560 to 3751134 tid 3 2024-09-13T14:25:50.524 INFO:tasks.rados.rados.0.smithi151.stdout:1201: write oid 41 current snap is 154 2024-09-13T14:25:50.524 INFO:tasks.rados.rados.0.smithi151.stdout:1200: finishing write tid 1 to smithi15186934-29 2024-09-13T14:25:50.524 INFO:tasks.rados.rados.0.smithi151.stdout:1200: finishing write tid 2 to smithi15186934-29 2024-09-13T14:25:50.524 INFO:tasks.rados.rados.0.smithi151.stdout:1201: seq_num 470 ranges {699290=464563,1603478=535155,2823983=765846} 2024-09-13T14:25:50.530 INFO:tasks.rados.rados.0.smithi151.stdout:1201: writing smithi15186934-41 from 699290 to 1163853 tid 1 2024-09-13T14:25:50.535 INFO:tasks.rados.rados.0.smithi151.stdout:1201: writing smithi15186934-41 from 1603478 to 2138633 tid 2 2024-09-13T14:25:50.543 INFO:tasks.rados.rados.0.smithi151.stdout:1201: writing smithi15186934-41 from 2823983 to 3589829 tid 3 2024-09-13T14:25:50.543 INFO:tasks.rados.rados.0.smithi151.stdout:1202: read oid 24 snap -1 2024-09-13T14:25:50.543 INFO:tasks.rados.rados.0.smithi151.stdout:1202: expect deleted 2024-09-13T14:25:50.543 INFO:tasks.rados.rados.0.smithi151.stdout:1203: snap_remove snap 143 2024-09-13T14:25:50.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:50 smithi184 ceph-mon[33394]: pgmap v158: 89 pgs: 89 active+clean; 270 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.9 MiB/s rd, 2.5 MiB/s wr, 19 op/s 2024-09-13T14:25:50.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:50 smithi184 ceph-mon[33394]: osdmap e361: 8 total, 8 up, 8 in 2024-09-13T14:25:50.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:50 smithi151 ceph-mon[104450]: pgmap v158: 89 pgs: 89 active+clean; 270 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.9 MiB/s rd, 2.5 MiB/s wr, 19 op/s 2024-09-13T14:25:50.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:50 smithi151 ceph-mon[104450]: osdmap e361: 8 total, 8 up, 8 in 2024-09-13T14:25:50.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:50 smithi151 ceph-mon[108966]: pgmap v158: 89 pgs: 89 active+clean; 270 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.9 MiB/s rd, 2.5 MiB/s wr, 19 op/s 2024-09-13T14:25:50.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:50 smithi151 ceph-mon[108966]: osdmap e361: 8 total, 8 up, 8 in 2024-09-13T14:25:51.510 INFO:tasks.rados.rados.0.smithi151.stdout:1200: finishing write tid 3 to smithi15186934-29 2024-09-13T14:25:51.510 INFO:tasks.rados.rados.0.smithi151.stdout:1200: finishing write tid 4 to smithi15186934-29 2024-09-13T14:25:51.510 INFO:tasks.rados.rados.0.smithi151.stdout:1200: finishing write tid 5 to smithi15186934-29 2024-09-13T14:25:51.511 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 29 v 326 (ObjNum 469 snap 154 seq_num 469) dirty exists 2024-09-13T14:25:51.511 INFO:tasks.rados.rados.0.smithi151.stdout:1200: left oid 29 (ObjNum 469 snap 154 seq_num 469) 2024-09-13T14:25:51.511 INFO:tasks.rados.rados.0.smithi151.stdout:1201: finishing write tid 1 to smithi15186934-41 2024-09-13T14:25:51.511 INFO:tasks.rados.rados.0.smithi151.stdout:1201: finishing write tid 2 to smithi15186934-41 2024-09-13T14:25:51.511 INFO:tasks.rados.rados.0.smithi151.stdout:1201: finishing write tid 3 to smithi15186934-41 2024-09-13T14:25:51.511 INFO:tasks.rados.rados.0.smithi151.stdout:1201: finishing write tid 4 to smithi15186934-41 2024-09-13T14:25:51.511 INFO:tasks.rados.rados.0.smithi151.stdout:1201: finishing write tid 5 to smithi15186934-41 2024-09-13T14:25:51.511 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 38 (ObjNum 470 snap 154 seq_num 470) dirty exists 2024-09-13T14:25:51.511 INFO:tasks.rados.rados.0.smithi151.stdout:1201: left oid 41 (ObjNum 470 snap 154 seq_num 470) 2024-09-13T14:25:51.511 INFO:tasks.rados.rados.0.smithi151.stdout:1200: done (3 left) 2024-09-13T14:25:51.511 INFO:tasks.rados.rados.0.smithi151.stdout:1201: done (2 left) 2024-09-13T14:25:51.511 INFO:tasks.rados.rados.0.smithi151.stdout:1202: done (1 left) 2024-09-13T14:25:51.511 INFO:tasks.rados.rados.0.smithi151.stdout:1203: done (0 left) 2024-09-13T14:25:51.511 INFO:tasks.rados.rados.0.smithi151.stdout:1204: snap_create 2024-09-13T14:25:51.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:51 smithi184 ceph-mon[33394]: osdmap e362: 8 total, 8 up, 8 in 2024-09-13T14:25:51.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:51 smithi151 ceph-mon[104450]: osdmap e362: 8 total, 8 up, 8 in 2024-09-13T14:25:51.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:51 smithi151 ceph-mon[108966]: osdmap e362: 8 total, 8 up, 8 in 2024-09-13T14:25:52.508 INFO:tasks.rados.rados.0.smithi151.stdout:1204: done (0 left) 2024-09-13T14:25:52.508 INFO:tasks.rados.rados.0.smithi151.stdout:1205: read oid 37 snap -1 2024-09-13T14:25:52.508 INFO:tasks.rados.rados.0.smithi151.stdout:1205: expect deleted 2024-09-13T14:25:52.508 INFO:tasks.rados.rados.0.smithi151.stdout:1206: snap_remove snap 148 2024-09-13T14:25:52.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:52 smithi184 ceph-mon[33394]: pgmap v161: 89 pgs: 89 active+clean; 270 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 255 B/s rd, 375 KiB/s wr, 2 op/s 2024-09-13T14:25:52.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:52 smithi184 ceph-mon[33394]: osdmap e363: 8 total, 8 up, 8 in 2024-09-13T14:25:52.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:52 smithi151 ceph-mon[104450]: pgmap v161: 89 pgs: 89 active+clean; 270 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 255 B/s rd, 375 KiB/s wr, 2 op/s 2024-09-13T14:25:52.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:52 smithi151 ceph-mon[104450]: osdmap e363: 8 total, 8 up, 8 in 2024-09-13T14:25:52.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:52 smithi151 ceph-mon[108966]: pgmap v161: 89 pgs: 89 active+clean; 270 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 255 B/s rd, 375 KiB/s wr, 2 op/s 2024-09-13T14:25:52.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:52 smithi151 ceph-mon[108966]: osdmap e363: 8 total, 8 up, 8 in 2024-09-13T14:25:53.523 INFO:tasks.rados.rados.0.smithi151.stdout:1205: done (1 left) 2024-09-13T14:25:53.524 INFO:tasks.rados.rados.0.smithi151.stdout:1206: done (0 left) 2024-09-13T14:25:53.524 INFO:tasks.rados.rados.0.smithi151.stdout:1207: rollback oid 9 current snap is 155 2024-09-13T14:25:53.524 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 9 to 149 2024-09-13T14:25:53.524 INFO:tasks.rados.rados.0.smithi151.stdout:1208: rollback oid 10 current snap is 155 2024-09-13T14:25:53.524 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 10 to 129 2024-09-13T14:25:53.524 INFO:tasks.rados.rados.0.smithi151.stdout:1209: snap_remove snap 107 2024-09-13T14:25:53.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:53 smithi184 ceph-mon[33394]: osdmap e364: 8 total, 8 up, 8 in 2024-09-13T14:25:53.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:53 smithi151 ceph-mon[104450]: osdmap e364: 8 total, 8 up, 8 in 2024-09-13T14:25:53.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:53 smithi151 ceph-mon[108966]: osdmap e364: 8 total, 8 up, 8 in 2024-09-13T14:25:54.469 INFO:tasks.rados.rados.0.smithi151.stdout:1209: done (2 left) 2024-09-13T14:25:54.469 INFO:tasks.rados.rados.0.smithi151.stdout:1210: write oid 45 current snap is 155 2024-09-13T14:25:54.469 INFO:tasks.rados.rados.0.smithi151.stdout:1210: seq_num 471 ranges {716097=539812,1945607=795166,3535053=381044} 2024-09-13T14:25:54.477 INFO:tasks.rados.rados.0.smithi151.stdout:1210: writing smithi15186934-45 from 716097 to 1255909 tid 1 2024-09-13T14:25:54.486 INFO:tasks.rados.rados.0.smithi151.stdout:1210: writing smithi15186934-45 from 1945607 to 2740773 tid 2 2024-09-13T14:25:54.493 INFO:tasks.rados.rados.0.smithi151.stdout:1210: writing smithi15186934-45 from 3535053 to 3916097 tid 3 2024-09-13T14:25:54.493 INFO:tasks.rados.rados.0.smithi151.stdout:1211: snap_create 2024-09-13T14:25:54.493 INFO:tasks.rados.rados.0.smithi151.stdout:1207: finishing rollback tid 0 to smithi15186934-9 2024-09-13T14:25:54.493 INFO:tasks.rados.rados.0.smithi151.stdout:1207: finishing rollback tid 1 to smithi15186934-9 2024-09-13T14:25:54.493 INFO:tasks.rados.rados.0.smithi151.stdout:1207: finishing rollback tid 2 to smithi15186934-9 2024-09-13T14:25:54.493 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 494 (ObjNum 268 snap 86 seq_num 268) dirty exists 2024-09-13T14:25:54.493 INFO:tasks.rados.rados.0.smithi151.stdout:1208: finishing rollback tid 0 to smithi15186934-10 2024-09-13T14:25:54.494 INFO:tasks.rados.rados.0.smithi151.stdout:1208: finishing rollback tid 1 to smithi15186934-10 2024-09-13T14:25:54.494 INFO:tasks.rados.rados.0.smithi151.stdout:1208: finishing rollback tid 2 to smithi15186934-10 2024-09-13T14:25:54.494 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 410 (ObjNum 352 snap 107 seq_num 352) dirty exists 2024-09-13T14:25:54.494 INFO:tasks.rados.rados.0.smithi151.stdout:1210: finishing write tid 1 to smithi15186934-45 2024-09-13T14:25:54.494 INFO:tasks.rados.rados.0.smithi151.stdout:1210: finishing write tid 2 to smithi15186934-45 2024-09-13T14:25:54.495 INFO:tasks.rados.rados.0.smithi151.stdout:1210: finishing write tid 3 to smithi15186934-45 2024-09-13T14:25:54.495 INFO:tasks.rados.rados.0.smithi151.stdout:1210: finishing write tid 4 to smithi15186934-45 2024-09-13T14:25:54.495 INFO:tasks.rados.rados.0.smithi151.stdout:1210: finishing write tid 5 to smithi15186934-45 2024-09-13T14:25:54.495 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 733 (ObjNum 471 snap 155 seq_num 471) dirty exists 2024-09-13T14:25:54.495 INFO:tasks.rados.rados.0.smithi151.stdout:1210: left oid 45 (ObjNum 471 snap 155 seq_num 471) 2024-09-13T14:25:54.501 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:25:54 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:25:54.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:25:54.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:54 smithi184 ceph-mon[33394]: pgmap v164: 89 pgs: 89 active+clean; 273 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 7 op/s 2024-09-13T14:25:54.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:54 smithi184 ceph-mon[33394]: osdmap e365: 8 total, 8 up, 8 in 2024-09-13T14:25:54.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:54 smithi184 ceph-mon[33394]: osdmap e366: 8 total, 8 up, 8 in 2024-09-13T14:25:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:54 smithi151 ceph-mon[104450]: pgmap v164: 89 pgs: 89 active+clean; 273 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 7 op/s 2024-09-13T14:25:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:54 smithi151 ceph-mon[104450]: osdmap e365: 8 total, 8 up, 8 in 2024-09-13T14:25:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:54 smithi151 ceph-mon[104450]: osdmap e366: 8 total, 8 up, 8 in 2024-09-13T14:25:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:54 smithi151 ceph-mon[108966]: pgmap v164: 89 pgs: 89 active+clean; 273 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 7 op/s 2024-09-13T14:25:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:54 smithi151 ceph-mon[108966]: osdmap e365: 8 total, 8 up, 8 in 2024-09-13T14:25:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:54 smithi151 ceph-mon[108966]: osdmap e366: 8 total, 8 up, 8 in 2024-09-13T14:25:54.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:25:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:25:54] "GET /metrics HTTP/1.1" 200 36397 "" "Prometheus/2.51.0" 2024-09-13T14:25:55.470 INFO:tasks.rados.rados.0.smithi151.stdout:1207: done (3 left) 2024-09-13T14:25:55.470 INFO:tasks.rados.rados.0.smithi151.stdout:1208: done (2 left) 2024-09-13T14:25:55.470 INFO:tasks.rados.rados.0.smithi151.stdout:1210: done (1 left) 2024-09-13T14:25:55.470 INFO:tasks.rados.rados.0.smithi151.stdout:1211: done (0 left) 2024-09-13T14:25:55.471 INFO:tasks.rados.rados.0.smithi151.stdout:1212: rollback oid 29 current snap is 156 2024-09-13T14:25:55.471 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 29 to 145 2024-09-13T14:25:55.471 INFO:tasks.rados.rados.0.smithi151.stdout:1213: rollback oid 34 current snap is 156 2024-09-13T14:25:55.471 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 34 to 141 2024-09-13T14:25:55.471 INFO:tasks.rados.rados.0.smithi151.stdout:1214: delete oid 2 current snap is 156 2024-09-13T14:25:55.471 INFO:tasks.rados.rados.0.smithi151.stdout:1213: finishing rollback tid 1 to smithi15186934-34 2024-09-13T14:25:55.471 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 0 (ObjNum 129 snap 0 seq_num 3092838816) dirty dne 2024-09-13T14:25:55.472 INFO:tasks.rados.rados.0.smithi151.stdout:1213: done (2 left) 2024-09-13T14:25:55.472 INFO:tasks.rados.rados.0.smithi151.stdout:1214: done (1 left) 2024-09-13T14:25:55.472 INFO:tasks.rados.rados.0.smithi151.stdout:1215: write oid 1 current snap is 156 2024-09-13T14:25:55.472 INFO:tasks.rados.rados.0.smithi151.stdout:1215: seq_num 473 ranges {749711=406663,1778924=682482,3189899=716015} 2024-09-13T14:25:55.478 INFO:tasks.rados.rados.0.smithi151.stdout:1215: writing smithi15186934-1 from 749711 to 1156374 tid 1 2024-09-13T14:25:55.485 INFO:tasks.rados.rados.0.smithi151.stdout:1215: writing smithi15186934-1 from 1778924 to 2461406 tid 2 2024-09-13T14:25:55.493 INFO:tasks.rados.rados.0.smithi151.stdout:1215: writing smithi15186934-1 from 3189899 to 3905914 tid 3 2024-09-13T14:25:55.493 INFO:tasks.rados.rados.0.smithi151.stdout:1216: snap_remove snap 140 2024-09-13T14:25:56.103 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:25:55 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:25:55.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:25:56.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:25:55 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:25:55.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:25:56.475 INFO:tasks.rados.rados.0.smithi151.stdout:1212: finishing rollback tid 0 to smithi15186934-29 2024-09-13T14:25:56.475 INFO:tasks.rados.rados.0.smithi151.stdout:1212: finishing rollback tid 1 to smithi15186934-29 2024-09-13T14:25:56.475 INFO:tasks.rados.rados.0.smithi151.stdout:1212: finishing rollback tid 2 to smithi15186934-29 2024-09-13T14:25:56.475 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 29 v 336 (ObjNum 445 snap 141 seq_num 445) dirty exists 2024-09-13T14:25:56.475 INFO:tasks.rados.rados.0.smithi151.stdout:1215: finishing write tid 1 to smithi15186934-1 2024-09-13T14:25:56.475 INFO:tasks.rados.rados.0.smithi151.stdout:1215: finishing write tid 2 to smithi15186934-1 2024-09-13T14:25:56.475 INFO:tasks.rados.rados.0.smithi151.stdout:1215: finishing write tid 3 to smithi15186934-1 2024-09-13T14:25:56.475 INFO:tasks.rados.rados.0.smithi151.stdout:1215: finishing write tid 4 to smithi15186934-1 2024-09-13T14:25:56.476 INFO:tasks.rados.rados.0.smithi151.stdout:1215: finishing write tid 5 to smithi15186934-1 2024-09-13T14:25:56.476 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 1 v 335 (ObjNum 473 snap 156 seq_num 473) dirty exists 2024-09-13T14:25:56.476 INFO:tasks.rados.rados.0.smithi151.stdout:1215: left oid 1 (ObjNum 473 snap 156 seq_num 473) 2024-09-13T14:25:56.476 INFO:tasks.rados.rados.0.smithi151.stdout:1212: done (2 left) 2024-09-13T14:25:56.476 INFO:tasks.rados.rados.0.smithi151.stdout:1215: done (1 left) 2024-09-13T14:25:56.476 INFO:tasks.rados.rados.0.smithi151.stdout:1216: done (0 left) 2024-09-13T14:25:56.476 INFO:tasks.rados.rados.0.smithi151.stdout:1217: write oid 44 current snap is 156 2024-09-13T14:25:56.476 INFO:tasks.rados.rados.0.smithi151.stdout:1217: seq_num 474 ranges {766518=481912,1837406=542493,2933675=527287,3876584=89934} 2024-09-13T14:25:56.482 INFO:tasks.rados.rados.0.smithi151.stdout:1217: writing smithi15186934-44 from 766518 to 1248430 tid 1 2024-09-13T14:25:56.489 INFO:tasks.rados.rados.0.smithi151.stdout:1217: writing smithi15186934-44 from 1837406 to 2379899 tid 2 2024-09-13T14:25:56.494 INFO:tasks.rados.rados.0.smithi151.stdout:1217: writing smithi15186934-44 from 2933675 to 3460962 tid 3 2024-09-13T14:25:56.498 INFO:tasks.rados.rados.0.smithi151.stdout:1217: writing smithi15186934-44 from 3876584 to 3966518 tid 4 2024-09-13T14:25:56.498 INFO:tasks.rados.rados.0.smithi151.stdout:1218: rollback oid 42 current snap is 156 2024-09-13T14:25:56.498 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 42 to 151 2024-09-13T14:25:56.498 INFO:tasks.rados.rados.0.smithi151.stdout:1217: finishing write tid 1 to smithi15186934-44 2024-09-13T14:25:56.498 INFO:tasks.rados.rados.0.smithi151.stdout:1217: finishing write tid 2 to smithi15186934-44 2024-09-13T14:25:56.498 INFO:tasks.rados.rados.0.smithi151.stdout:1217: finishing write tid 3 to smithi15186934-44 2024-09-13T14:25:56.498 INFO:tasks.rados.rados.0.smithi151.stdout:1219: snap_remove snap 152 2024-09-13T14:25:56.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:56 smithi184 ceph-mon[33394]: pgmap v167: 89 pgs: 89 active+clean; 273 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 7 op/s 2024-09-13T14:25:56.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:56 smithi184 ceph-mon[33394]: osdmap e367: 8 total, 8 up, 8 in 2024-09-13T14:25:56.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:56 smithi151 ceph-mon[104450]: pgmap v167: 89 pgs: 89 active+clean; 273 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 7 op/s 2024-09-13T14:25:56.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:56 smithi151 ceph-mon[104450]: osdmap e367: 8 total, 8 up, 8 in 2024-09-13T14:25:56.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:56 smithi151 ceph-mon[108966]: pgmap v167: 89 pgs: 89 active+clean; 273 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 7 op/s 2024-09-13T14:25:56.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:56 smithi151 ceph-mon[108966]: osdmap e367: 8 total, 8 up, 8 in 2024-09-13T14:25:57.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:25:56 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:25:56.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi184\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:25:57.494 INFO:tasks.rados.rados.0.smithi151.stdout:1219: done (2 left) 2024-09-13T14:25:57.495 INFO:tasks.rados.rados.0.smithi151.stdout:1220: delete oid 45 current snap is 156 2024-09-13T14:25:57.495 INFO:tasks.rados.rados.0.smithi151.stdout:1217: finishing write tid 4 to smithi15186934-44 2024-09-13T14:25:57.495 INFO:tasks.rados.rados.0.smithi151.stdout:1217: finishing write tid 5 to smithi15186934-44 2024-09-13T14:25:57.495 INFO:tasks.rados.rados.0.smithi151.stdout:1217: finishing write tid 6 to smithi15186934-44 2024-09-13T14:25:57.495 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 345 (ObjNum 474 snap 156 seq_num 474) dirty exists 2024-09-13T14:25:57.495 INFO:tasks.rados.rados.0.smithi151.stdout:1217: left oid 44 (ObjNum 474 snap 156 seq_num 474) 2024-09-13T14:25:57.495 INFO:tasks.rados.rados.0.smithi151.stdout:1218: finishing rollback tid 0 to smithi15186934-42 2024-09-13T14:25:57.495 INFO:tasks.rados.rados.0.smithi151.stdout:1218: finishing rollback tid 1 to smithi15186934-42 2024-09-13T14:25:57.495 INFO:tasks.rados.rados.0.smithi151.stdout:1218: finishing rollback tid 2 to smithi15186934-42 2024-09-13T14:25:57.495 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 621 (ObjNum 429 snap 135 seq_num 429) dirty exists 2024-09-13T14:25:57.497 INFO:tasks.rados.rados.0.smithi151.stdout:1217: done (2 left) 2024-09-13T14:25:57.497 INFO:tasks.rados.rados.0.smithi151.stdout:1218: done (1 left) 2024-09-13T14:25:57.497 INFO:tasks.rados.rados.0.smithi151.stdout:1220: done (0 left) 2024-09-13T14:25:57.497 INFO:tasks.rados.rados.0.smithi151.stdout:1221: write oid 50 current snap is 156 2024-09-13T14:25:57.497 INFO:tasks.rados.rados.0.smithi151.stdout:1221: seq_num 476 ranges {400132=632410,1438017=546162} 2024-09-13T14:25:57.502 INFO:tasks.rados.rados.0.smithi151.stdout:1221: writing smithi15186934-50 from 400132 to 1032542 tid 1 2024-09-13T14:25:57.508 INFO:tasks.rados.rados.0.smithi151.stdout:1221: writing smithi15186934-50 from 1438017 to 1984179 tid 2 2024-09-13T14:25:57.508 INFO:tasks.rados.rados.0.smithi151.stdout:1222: read oid 31 snap 134 2024-09-13T14:25:57.508 INFO:tasks.rados.rados.0.smithi151.stdout:1222: expect (ObjNum 395 snap 123 seq_num 395) 2024-09-13T14:25:57.508 INFO:tasks.rados.rados.0.smithi151.stdout:1223: write oid 34 current snap is 156 2024-09-13T14:25:57.508 INFO:tasks.rados.rados.0.smithi151.stdout:1221: finishing write tid 1 to smithi15186934-50 2024-09-13T14:25:57.508 INFO:tasks.rados.rados.0.smithi151.stdout:1223: seq_num 477 ranges {416939=707659,1896499=120440} 2024-09-13T14:25:57.514 INFO:tasks.rados.rados.0.smithi151.stdout:1223: writing smithi15186934-34 from 416939 to 1124598 tid 1 2024-09-13T14:25:57.520 INFO:tasks.rados.rados.0.smithi151.stdout:1223: writing smithi15186934-34 from 1896499 to 2016939 tid 2 2024-09-13T14:25:57.520 INFO:tasks.rados.rados.0.smithi151.stdout:1224: write oid 10 current snap is 156 2024-09-13T14:25:57.520 INFO:tasks.rados.rados.0.smithi151.stdout:1224: seq_num 478 ranges {433746=782908,1954981=78765} 2024-09-13T14:25:57.526 INFO:tasks.rados.rados.0.smithi151.stdout:1224: writing smithi15186934-10 from 433746 to 1216654 tid 1 2024-09-13T14:25:57.531 INFO:tasks.rados.rados.0.smithi151.stdout:1224: writing smithi15186934-10 from 1954981 to 2033746 tid 2 2024-09-13T14:25:57.531 INFO:tasks.rados.rados.0.smithi151.stdout:1225: write oid 42 current snap is 156 2024-09-13T14:25:57.531 INFO:tasks.rados.rados.0.smithi151.stdout:1221: finishing write tid 2 to smithi15186934-50 2024-09-13T14:25:57.532 INFO:tasks.rados.rados.0.smithi151.stdout:1221: finishing write tid 3 to smithi15186934-50 2024-09-13T14:25:57.532 INFO:tasks.rados.rados.0.smithi151.stdout:1221: finishing write tid 4 to smithi15186934-50 2024-09-13T14:25:57.532 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 50 v 384 (ObjNum 476 snap 156 seq_num 476) dirty exists 2024-09-13T14:25:57.532 INFO:tasks.rados.rados.0.smithi151.stdout:1221: left oid 50 (ObjNum 476 snap 156 seq_num 476) 2024-09-13T14:25:57.532 INFO:tasks.rados.rados.0.smithi151.stdout:1222: expect (ObjNum 395 snap 123 seq_num 395) 2024-09-13T14:25:57.619 INFO:tasks.rados.rados.0.smithi151.stdout:1223: finishing write tid 1 to smithi15186934-34 2024-09-13T14:25:57.619 INFO:tasks.rados.rados.0.smithi151.stdout:1223: finishing write tid 2 to smithi15186934-34 2024-09-13T14:25:57.619 INFO:tasks.rados.rados.0.smithi151.stdout:1223: finishing write tid 3 to smithi15186934-34 2024-09-13T14:25:57.619 INFO:tasks.rados.rados.0.smithi151.stdout:1223: finishing write tid 4 to smithi15186934-34 2024-09-13T14:25:57.620 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 489 (ObjNum 477 snap 156 seq_num 477) dirty exists 2024-09-13T14:25:57.620 INFO:tasks.rados.rados.0.smithi151.stdout:1223: left oid 34 (ObjNum 477 snap 156 seq_num 477) 2024-09-13T14:25:57.620 INFO:tasks.rados.rados.0.smithi151.stdout:1224: finishing write tid 1 to smithi15186934-10 2024-09-13T14:25:57.620 INFO:tasks.rados.rados.0.smithi151.stdout:1224: finishing write tid 2 to smithi15186934-10 2024-09-13T14:25:57.620 INFO:tasks.rados.rados.0.smithi151.stdout:1224: finishing write tid 3 to smithi15186934-10 2024-09-13T14:25:57.620 INFO:tasks.rados.rados.0.smithi151.stdout:1224: finishing write tid 4 to smithi15186934-10 2024-09-13T14:25:57.620 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 432 (ObjNum 478 snap 156 seq_num 478) dirty exists 2024-09-13T14:25:57.620 INFO:tasks.rados.rados.0.smithi151.stdout:1224: left oid 10 (ObjNum 478 snap 156 seq_num 478) 2024-09-13T14:25:57.620 INFO:tasks.rados.rados.0.smithi151.stdout:1225: seq_num 479 ranges {450553=574510,1613463=437090} 2024-09-13T14:25:57.624 INFO:tasks.rados.rados.0.smithi151.stdout:1225: writing smithi15186934-42 from 450553 to 1025063 tid 1 2024-09-13T14:25:57.630 INFO:tasks.rados.rados.0.smithi151.stdout:1225: writing smithi15186934-42 from 1613463 to 2050553 tid 2 2024-09-13T14:25:57.630 INFO:tasks.rados.rados.0.smithi151.stdout:1221: done (4 left) 2024-09-13T14:25:57.630 INFO:tasks.rados.rados.0.smithi151.stdout:1222: done (3 left) 2024-09-13T14:25:57.630 INFO:tasks.rados.rados.0.smithi151.stdout:1223: done (2 left) 2024-09-13T14:25:57.630 INFO:tasks.rados.rados.0.smithi151.stdout:1224: done (1 left) 2024-09-13T14:25:57.630 INFO:tasks.rados.rados.0.smithi151.stdout:1226: snap_remove snap 155 2024-09-13T14:25:57.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:57 smithi184 ceph-mon[33394]: osdmap e368: 8 total, 8 up, 8 in 2024-09-13T14:25:57.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:57 smithi151 ceph-mon[104450]: osdmap e368: 8 total, 8 up, 8 in 2024-09-13T14:25:57.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:57 smithi151 ceph-mon[108966]: osdmap e368: 8 total, 8 up, 8 in 2024-09-13T14:25:58.490 INFO:tasks.rados.rados.0.smithi151.stdout:1226: done (1 left) 2024-09-13T14:25:58.490 INFO:tasks.rados.rados.0.smithi151.stdout:1227: read oid 37 snap -1 2024-09-13T14:25:58.490 INFO:tasks.rados.rados.0.smithi151.stdout:1227: expect deleted 2024-09-13T14:25:58.490 INFO:tasks.rados.rados.0.smithi151.stdout:1225: finishing write tid 1 to smithi15186934-42 2024-09-13T14:25:58.491 INFO:tasks.rados.rados.0.smithi151.stdout:1225: finishing write tid 2 to smithi15186934-42 2024-09-13T14:25:58.491 INFO:tasks.rados.rados.0.smithi151.stdout:1225: finishing write tid 3 to smithi15186934-42 2024-09-13T14:25:58.491 INFO:tasks.rados.rados.0.smithi151.stdout:1225: finishing write tid 4 to smithi15186934-42 2024-09-13T14:25:58.491 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 626 (ObjNum 479 snap 156 seq_num 479) dirty exists 2024-09-13T14:25:58.491 INFO:tasks.rados.rados.0.smithi151.stdout:1225: left oid 42 (ObjNum 479 snap 156 seq_num 479) 2024-09-13T14:25:58.491 INFO:tasks.rados.rados.0.smithi151.stdout:1225: done (1 left) 2024-09-13T14:25:58.491 INFO:tasks.rados.rados.0.smithi151.stdout:1228: delete oid 34 current snap is 156 2024-09-13T14:25:58.492 INFO:tasks.rados.rados.0.smithi151.stdout:1227: done (1 left) 2024-09-13T14:25:58.492 INFO:tasks.rados.rados.0.smithi151.stdout:1228: done (0 left) 2024-09-13T14:25:58.492 INFO:tasks.rados.rados.0.smithi151.stdout:1229: write oid 2 current snap is 156 2024-09-13T14:25:58.493 INFO:tasks.rados.rados.0.smithi151.stdout:1229: seq_num 481 ranges {484167=725008,1730427=353740} 2024-09-13T14:25:58.499 INFO:tasks.rados.rados.0.smithi151.stdout:1229: writing smithi15186934-2 from 484167 to 1209175 tid 1 2024-09-13T14:25:58.504 INFO:tasks.rados.rados.0.smithi151.stdout:1229: writing smithi15186934-2 from 1730427 to 2084167 tid 2 2024-09-13T14:25:58.504 INFO:tasks.rados.rados.0.smithi151.stdout:1230: snap_remove snap 139 2024-09-13T14:25:58.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:58 smithi184 ceph-mon[33394]: pgmap v170: 89 pgs: 89 active+clean; 219 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 767 B/s rd, 1.2 MiB/s wr, 6 op/s 2024-09-13T14:25:58.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:58 smithi184 ceph-mon[33394]: osdmap e369: 8 total, 8 up, 8 in 2024-09-13T14:25:58.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:58 smithi151 ceph-mon[104450]: pgmap v170: 89 pgs: 89 active+clean; 219 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 767 B/s rd, 1.2 MiB/s wr, 6 op/s 2024-09-13T14:25:58.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:58 smithi151 ceph-mon[104450]: osdmap e369: 8 total, 8 up, 8 in 2024-09-13T14:25:58.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:58 smithi151 ceph-mon[108966]: pgmap v170: 89 pgs: 89 active+clean; 219 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 767 B/s rd, 1.2 MiB/s wr, 6 op/s 2024-09-13T14:25:58.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:58 smithi151 ceph-mon[108966]: osdmap e369: 8 total, 8 up, 8 in 2024-09-13T14:25:59.510 INFO:tasks.rados.rados.0.smithi151.stdout:1229: finishing write tid 1 to smithi15186934-2 2024-09-13T14:25:59.510 INFO:tasks.rados.rados.0.smithi151.stdout:1229: finishing write tid 2 to smithi15186934-2 2024-09-13T14:25:59.510 INFO:tasks.rados.rados.0.smithi151.stdout:1229: finishing write tid 3 to smithi15186934-2 2024-09-13T14:25:59.511 INFO:tasks.rados.rados.0.smithi151.stdout:1229: finishing write tid 4 to smithi15186934-2 2024-09-13T14:25:59.511 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 2 v 448 (ObjNum 481 snap 156 seq_num 481) dirty exists 2024-09-13T14:25:59.511 INFO:tasks.rados.rados.0.smithi151.stdout:1229: left oid 2 (ObjNum 481 snap 156 seq_num 481) 2024-09-13T14:25:59.511 INFO:tasks.rados.rados.0.smithi151.stdout:1229: done (1 left) 2024-09-13T14:25:59.511 INFO:tasks.rados.rados.0.smithi151.stdout:1230: done (0 left) 2024-09-13T14:25:59.511 INFO:tasks.rados.rados.0.smithi151.stdout:1231: rollback oid 3 current snap is 156 2024-09-13T14:25:59.511 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 3 to 153 2024-09-13T14:25:59.511 INFO:tasks.rados.rados.0.smithi151.stdout:1232: read oid 15 snap -1 2024-09-13T14:25:59.512 INFO:tasks.rados.rados.0.smithi151.stdout:1232: expect (ObjNum 413 snap 128 seq_num 413) 2024-09-13T14:25:59.512 INFO:tasks.rados.rados.0.smithi151.stdout:1233: rollback oid 32 current snap is 156 2024-09-13T14:25:59.512 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 32 to 153 2024-09-13T14:25:59.512 INFO:tasks.rados.rados.0.smithi151.stdout:1234: read oid 25 snap -1 2024-09-13T14:25:59.512 INFO:tasks.rados.rados.0.smithi151.stdout:1234: expect (ObjNum 361 snap 110 seq_num 361) 2024-09-13T14:25:59.512 INFO:tasks.rados.rados.0.smithi151.stdout:1235: rollback oid 17 current snap is 156 2024-09-13T14:25:59.512 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 17 to 151 2024-09-13T14:25:59.513 INFO:tasks.rados.rados.0.smithi151.stdout:1236: write oid 46 current snap is 156 2024-09-13T14:25:59.513 INFO:tasks.rados.rados.0.smithi151.stdout:1236: seq_num 482 ranges {500974=400257,1388909=557169} 2024-09-13T14:25:59.515 INFO:tasks.rados.rados.0.smithi151.stdout:1236: writing smithi15186934-46 from 500974 to 901231 tid 1 2024-09-13T14:25:59.522 INFO:tasks.rados.rados.0.smithi151.stdout:1236: writing smithi15186934-46 from 1388909 to 1946078 tid 2 2024-09-13T14:25:59.522 INFO:tasks.rados.rados.0.smithi151.stdout:1237: read oid 27 snap -1 2024-09-13T14:25:59.522 INFO:tasks.rados.rados.0.smithi151.stdout:1237: expect (ObjNum 318 snap 97 seq_num 318) 2024-09-13T14:25:59.522 INFO:tasks.rados.rados.0.smithi151.stdout:1235: finishing rollback tid 0 to smithi15186934-17 2024-09-13T14:25:59.522 INFO:tasks.rados.rados.0.smithi151.stdout:1235: finishing rollback tid 1 to smithi15186934-17 2024-09-13T14:25:59.522 INFO:tasks.rados.rados.0.smithi151.stdout:1235: finishing rollback tid 2 to smithi15186934-17 2024-09-13T14:25:59.522 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 803 (ObjNum 403 snap 123 seq_num 403) dirty exists 2024-09-13T14:25:59.522 INFO:tasks.rados.rados.0.smithi151.stdout:1231: finishing rollback tid 0 to smithi15186934-3 2024-09-13T14:25:59.523 INFO:tasks.rados.rados.0.smithi151.stdout:1234: expect (ObjNum 361 snap 110 seq_num 361) 2024-09-13T14:25:59.637 INFO:tasks.rados.rados.0.smithi151.stdout:1233: finishing rollback tid 0 to smithi15186934-32 2024-09-13T14:25:59.637 INFO:tasks.rados.rados.0.smithi151.stdout:1231: finishing rollback tid 1 to smithi15186934-3 2024-09-13T14:25:59.637 INFO:tasks.rados.rados.0.smithi151.stdout:1233: finishing rollback tid 1 to smithi15186934-32 2024-09-13T14:25:59.637 INFO:tasks.rados.rados.0.smithi151.stdout:1231: finishing rollback tid 2 to smithi15186934-3 2024-09-13T14:25:59.637 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 494 (ObjNum 334 snap 103 seq_num 334) dirty exists 2024-09-13T14:25:59.637 INFO:tasks.rados.rados.0.smithi151.stdout:1233: finishing rollback tid 2 to smithi15186934-32 2024-09-13T14:25:59.637 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 32 v 514 (ObjNum 424 snap 132 seq_num 424) dirty exists 2024-09-13T14:25:59.638 INFO:tasks.rados.rados.0.smithi151.stdout:1236: finishing write tid 1 to smithi15186934-46 2024-09-13T14:25:59.638 INFO:tasks.rados.rados.0.smithi151.stdout:1231: done (6 left) 2024-09-13T14:25:59.638 INFO:tasks.rados.rados.0.smithi151.stdout:1233: done (5 left) 2024-09-13T14:25:59.638 INFO:tasks.rados.rados.0.smithi151.stdout:1234: done (4 left) 2024-09-13T14:25:59.638 INFO:tasks.rados.rados.0.smithi151.stdout:1235: done (3 left) 2024-09-13T14:25:59.638 INFO:tasks.rados.rados.0.smithi151.stdout:1238: write oid 11 current snap is 156 2024-09-13T14:25:59.638 INFO:tasks.rados.rados.0.smithi151.stdout:1232: expect (ObjNum 413 snap 128 seq_num 413) 2024-09-13T14:25:59.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:25:59 smithi184 ceph-mon[33394]: osdmap e370: 8 total, 8 up, 8 in 2024-09-13T14:25:59.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:25:59 smithi151 ceph-mon[104450]: osdmap e370: 8 total, 8 up, 8 in 2024-09-13T14:25:59.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:25:59 smithi151 ceph-mon[108966]: osdmap e370: 8 total, 8 up, 8 in 2024-09-13T14:25:59.924 INFO:tasks.rados.rados.0.smithi151.stdout:1236: finishing write tid 2 to smithi15186934-46 2024-09-13T14:25:59.925 INFO:tasks.rados.rados.0.smithi151.stdout:1236: finishing write tid 3 to smithi15186934-46 2024-09-13T14:25:59.925 INFO:tasks.rados.rados.0.smithi151.stdout:1236: finishing write tid 4 to smithi15186934-46 2024-09-13T14:25:59.925 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 46 v 444 (ObjNum 482 snap 156 seq_num 482) dirty exists 2024-09-13T14:25:59.925 INFO:tasks.rados.rados.0.smithi151.stdout:1236: left oid 46 (ObjNum 482 snap 156 seq_num 482) 2024-09-13T14:25:59.925 INFO:tasks.rados.rados.0.smithi151.stdout:1237: expect (ObjNum 318 snap 97 seq_num 318) 2024-09-13T14:26:00.084 INFO:tasks.rados.rados.0.smithi151.stdout:1238: seq_num 483 ranges {517781=475506,1731038=386743} 2024-09-13T14:26:00.089 INFO:tasks.rados.rados.0.smithi151.stdout:1238: writing smithi15186934-11 from 517781 to 993287 tid 1 2024-09-13T14:26:00.098 INFO:tasks.rados.rados.0.smithi151.stdout:1238: writing smithi15186934-11 from 1731038 to 2117781 tid 2 2024-09-13T14:26:00.098 INFO:tasks.rados.rados.0.smithi151.stdout:1232: done (3 left) 2024-09-13T14:26:00.098 INFO:tasks.rados.rados.0.smithi151.stdout:1236: done (2 left) 2024-09-13T14:26:00.098 INFO:tasks.rados.rados.0.smithi151.stdout:1237: done (1 left) 2024-09-13T14:26:00.099 INFO:tasks.rados.rados.0.smithi151.stdout:1239: write oid 19 current snap is 156 2024-09-13T14:26:00.099 INFO:tasks.rados.rados.0.smithi151.stdout:1238: finishing write tid 1 to smithi15186934-11 2024-09-13T14:26:00.099 INFO:tasks.rados.rados.0.smithi151.stdout:1239: seq_num 484 ranges {534588=550755,1789520=345068} 2024-09-13T14:26:00.101 INFO:tasks.rados.rados.0.smithi151.stdout:1239: writing smithi15186934-19 from 534588 to 1085343 tid 1 2024-09-13T14:26:00.107 INFO:tasks.rados.rados.0.smithi151.stdout:1239: writing smithi15186934-19 from 1789520 to 2134588 tid 2 2024-09-13T14:26:00.107 INFO:tasks.rados.rados.0.smithi151.stdout:1240: write oid 41 current snap is 156 2024-09-13T14:26:00.107 INFO:tasks.rados.rados.0.smithi151.stdout:1238: finishing write tid 2 to smithi15186934-11 2024-09-13T14:26:00.107 INFO:tasks.rados.rados.0.smithi151.stdout:1238: finishing write tid 3 to smithi15186934-11 2024-09-13T14:26:00.107 INFO:tasks.rados.rados.0.smithi151.stdout:1240: seq_num 485 ranges {551395=626004,1848002=303393} 2024-09-13T14:26:00.113 INFO:tasks.rados.rados.0.smithi151.stdout:1240: writing smithi15186934-41 from 551395 to 1177399 tid 1 2024-09-13T14:26:00.119 INFO:tasks.rados.rados.0.smithi151.stdout:1240: writing smithi15186934-41 from 1848002 to 2151395 tid 2 2024-09-13T14:26:00.120 INFO:tasks.rados.rados.0.smithi151.stdout:1241: snap_create 2024-09-13T14:26:00.120 INFO:tasks.rados.rados.0.smithi151.stdout:1238: finishing write tid 4 to smithi15186934-11 2024-09-13T14:26:00.120 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 11 v 343 (ObjNum 483 snap 156 seq_num 483) dirty exists 2024-09-13T14:26:00.120 INFO:tasks.rados.rados.0.smithi151.stdout:1238: left oid 11 (ObjNum 483 snap 156 seq_num 483) 2024-09-13T14:26:00.120 INFO:tasks.rados.rados.0.smithi151.stdout:1239: finishing write tid 1 to smithi15186934-19 2024-09-13T14:26:00.120 INFO:tasks.rados.rados.0.smithi151.stdout:1239: finishing write tid 2 to smithi15186934-19 2024-09-13T14:26:00.120 INFO:tasks.rados.rados.0.smithi151.stdout:1239: finishing write tid 3 to smithi15186934-19 2024-09-13T14:26:00.120 INFO:tasks.rados.rados.0.smithi151.stdout:1239: finishing write tid 4 to smithi15186934-19 2024-09-13T14:26:00.120 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 19 v 304 (ObjNum 484 snap 156 seq_num 484) dirty exists 2024-09-13T14:26:00.120 INFO:tasks.rados.rados.0.smithi151.stdout:1239: left oid 19 (ObjNum 484 snap 156 seq_num 484) 2024-09-13T14:26:00.121 INFO:tasks.rados.rados.0.smithi151.stdout:1240: finishing write tid 1 to smithi15186934-41 2024-09-13T14:26:00.121 INFO:tasks.rados.rados.0.smithi151.stdout:1240: finishing write tid 2 to smithi15186934-41 2024-09-13T14:26:00.122 INFO:tasks.rados.rados.0.smithi151.stdout:1240: finishing write tid 3 to smithi15186934-41 2024-09-13T14:26:00.122 INFO:tasks.rados.rados.0.smithi151.stdout:1240: finishing write tid 4 to smithi15186934-41 2024-09-13T14:26:00.122 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 42 (ObjNum 485 snap 156 seq_num 485) dirty exists 2024-09-13T14:26:00.122 INFO:tasks.rados.rados.0.smithi151.stdout:1240: left oid 41 (ObjNum 485 snap 156 seq_num 485) 2024-09-13T14:26:00.506 INFO:tasks.rados.rados.0.smithi151.stdout:1238: done (3 left) 2024-09-13T14:26:00.507 INFO:tasks.rados.rados.0.smithi151.stdout:1239: done (2 left) 2024-09-13T14:26:00.507 INFO:tasks.rados.rados.0.smithi151.stdout:1240: done (1 left) 2024-09-13T14:26:00.507 INFO:tasks.rados.rados.0.smithi151.stdout:1241: done (0 left) 2024-09-13T14:26:00.507 INFO:tasks.rados.rados.0.smithi151.stdout:1242: delete oid 32 current snap is 157 2024-09-13T14:26:00.509 INFO:tasks.rados.rados.0.smithi151.stdout:1242: done (0 left) 2024-09-13T14:26:00.509 INFO:tasks.rados.rados.0.smithi151.stdout:1243: delete oid 41 current snap is 157 2024-09-13T14:26:00.510 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:26:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:26:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:00.516 INFO:tasks.rados.rados.0.smithi151.stdout:1243: done (0 left) 2024-09-13T14:26:00.516 INFO:tasks.rados.rados.0.smithi151.stdout:1244: snap_create 2024-09-13T14:26:00.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:00 smithi184 ceph-mon[33394]: pgmap v173: 89 pgs: 89 active+clean; 221 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 767 B/s rd, 1.2 MiB/s wr, 6 op/s 2024-09-13T14:26:00.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:00 smithi184 ceph-mon[33394]: osdmap e371: 8 total, 8 up, 8 in 2024-09-13T14:26:00.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:00 smithi151 ceph-mon[104450]: pgmap v173: 89 pgs: 89 active+clean; 221 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 767 B/s rd, 1.2 MiB/s wr, 6 op/s 2024-09-13T14:26:00.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:00 smithi151 ceph-mon[104450]: osdmap e371: 8 total, 8 up, 8 in 2024-09-13T14:26:00.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:00 smithi151 ceph-mon[108966]: pgmap v173: 89 pgs: 89 active+clean; 221 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 767 B/s rd, 1.2 MiB/s wr, 6 op/s 2024-09-13T14:26:00.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:00 smithi151 ceph-mon[108966]: osdmap e371: 8 total, 8 up, 8 in 2024-09-13T14:26:01.523 INFO:tasks.rados.rados.0.smithi151.stdout:1244: done (0 left) 2024-09-13T14:26:01.524 INFO:tasks.rados.rados.0.smithi151.stdout:1245: delete oid 34 current snap is 158 2024-09-13T14:26:01.525 INFO:tasks.rados.rados.0.smithi151.stdout:1245: done (0 left) 2024-09-13T14:26:01.525 INFO:tasks.rados.rados.0.smithi151.stdout:1246: write oid 46 current snap is 158 2024-09-13T14:26:01.525 INFO:tasks.rados.rados.0.smithi151.stdout:1246: seq_num 489 ranges {618623=643353,1681930=536693} 2024-09-13T14:26:01.532 INFO:tasks.rados.rados.0.smithi151.stdout:1246: writing smithi15186934-46 from 618623 to 1261976 tid 1 2024-09-13T14:26:01.538 INFO:tasks.rados.rados.0.smithi151.stdout:1246: writing smithi15186934-46 from 1681930 to 2218623 tid 2 2024-09-13T14:26:01.538 INFO:tasks.rados.rados.0.smithi151.stdout:1247: write oid 48 current snap is 158 2024-09-13T14:26:01.538 INFO:tasks.rados.rados.0.smithi151.stdout:1246: finishing write tid 1 to smithi15186934-46 2024-09-13T14:26:01.538 INFO:tasks.rados.rados.0.smithi151.stdout:1247: seq_num 490 ranges {635430=718602,2140412=95018} 2024-09-13T14:26:01.545 INFO:tasks.rados.rados.0.smithi151.stdout:1247: writing smithi15186934-48 from 635430 to 1354032 tid 1 2024-09-13T14:26:01.550 INFO:tasks.rados.rados.0.smithi151.stdout:1247: writing smithi15186934-48 from 2140412 to 2235430 tid 2 2024-09-13T14:26:01.550 INFO:tasks.rados.rados.0.smithi151.stdout:1248: write oid 18 current snap is 158 2024-09-13T14:26:01.550 INFO:tasks.rados.rados.0.smithi151.stdout:1248: seq_num 491 ranges {652237=793851,2198894=53343} 2024-09-13T14:26:01.558 INFO:tasks.rados.rados.0.smithi151.stdout:1248: writing smithi15186934-18 from 652237 to 1446088 tid 1 2024-09-13T14:26:01.562 INFO:tasks.rados.rados.0.smithi151.stdout:1248: writing smithi15186934-18 from 2198894 to 2252237 tid 2 2024-09-13T14:26:01.562 INFO:tasks.rados.rados.0.smithi151.stdout:1249: snap_remove snap 123 2024-09-13T14:26:01.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:01 smithi184 ceph-mon[33394]: osdmap e372: 8 total, 8 up, 8 in 2024-09-13T14:26:01.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:01 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:26:01.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:01 smithi151 ceph-mon[108966]: osdmap e372: 8 total, 8 up, 8 in 2024-09-13T14:26:01.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:01 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:26:01.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:01 smithi151 ceph-mon[104450]: osdmap e372: 8 total, 8 up, 8 in 2024-09-13T14:26:01.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:26:02.528 INFO:tasks.rados.rados.0.smithi151.stdout:1246: finishing write tid 2 to smithi15186934-46 2024-09-13T14:26:02.528 INFO:tasks.rados.rados.0.smithi151.stdout:1246: finishing write tid 3 to smithi15186934-46 2024-09-13T14:26:02.529 INFO:tasks.rados.rados.0.smithi151.stdout:1246: finishing write tid 4 to smithi15186934-46 2024-09-13T14:26:02.529 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 46 v 448 (ObjNum 489 snap 158 seq_num 489) dirty exists 2024-09-13T14:26:02.529 INFO:tasks.rados.rados.0.smithi151.stdout:1246: left oid 46 (ObjNum 489 snap 158 seq_num 489) 2024-09-13T14:26:02.529 INFO:tasks.rados.rados.0.smithi151.stdout:1247: finishing write tid 1 to smithi15186934-48 2024-09-13T14:26:02.529 INFO:tasks.rados.rados.0.smithi151.stdout:1247: finishing write tid 2 to smithi15186934-48 2024-09-13T14:26:02.529 INFO:tasks.rados.rados.0.smithi151.stdout:1247: finishing write tid 3 to smithi15186934-48 2024-09-13T14:26:02.529 INFO:tasks.rados.rados.0.smithi151.stdout:1247: finishing write tid 4 to smithi15186934-48 2024-09-13T14:26:02.529 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 48 v 751 (ObjNum 490 snap 158 seq_num 490) dirty exists 2024-09-13T14:26:02.529 INFO:tasks.rados.rados.0.smithi151.stdout:1247: left oid 48 (ObjNum 490 snap 158 seq_num 490) 2024-09-13T14:26:02.529 INFO:tasks.rados.rados.0.smithi151.stdout:1248: finishing write tid 1 to smithi15186934-18 2024-09-13T14:26:02.529 INFO:tasks.rados.rados.0.smithi151.stdout:1248: finishing write tid 2 to smithi15186934-18 2024-09-13T14:26:02.529 INFO:tasks.rados.rados.0.smithi151.stdout:1248: finishing write tid 3 to smithi15186934-18 2024-09-13T14:26:02.529 INFO:tasks.rados.rados.0.smithi151.stdout:1248: finishing write tid 4 to smithi15186934-18 2024-09-13T14:26:02.530 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 18 v 347 (ObjNum 491 snap 158 seq_num 491) dirty exists 2024-09-13T14:26:02.530 INFO:tasks.rados.rados.0.smithi151.stdout:1248: left oid 18 (ObjNum 491 snap 158 seq_num 491) 2024-09-13T14:26:02.530 INFO:tasks.rados.rados.0.smithi151.stdout:1246: done (3 left) 2024-09-13T14:26:02.530 INFO:tasks.rados.rados.0.smithi151.stdout:1247: done (2 left) 2024-09-13T14:26:02.530 INFO:tasks.rados.rados.0.smithi151.stdout:1248: done (1 left) 2024-09-13T14:26:02.530 INFO:tasks.rados.rados.0.smithi151.stdout:1249: done (0 left) 2024-09-13T14:26:02.530 INFO:tasks.rados.rados.0.smithi151.stdout:1250: write oid 4 current snap is 158 2024-09-13T14:26:02.530 INFO:tasks.rados.rados.0.smithi151.stdout:1250: seq_num 492 ranges {669044=469100,1741023=459161} 2024-09-13T14:26:02.535 INFO:tasks.rados.rados.0.smithi151.stdout:1250: writing smithi15186934-4 from 669044 to 1138144 tid 1 2024-09-13T14:26:02.541 INFO:tasks.rados.rados.0.smithi151.stdout:1250: writing smithi15186934-4 from 1741023 to 2200184 tid 2 2024-09-13T14:26:02.542 INFO:tasks.rados.rados.0.smithi151.stdout:1251: snap_create 2024-09-13T14:26:02.542 INFO:tasks.rados.rados.0.smithi151.stdout:1250: finishing write tid 1 to smithi15186934-4 2024-09-13T14:26:02.545 INFO:tasks.rados.rados.0.smithi151.stdout:1250: finishing write tid 2 to smithi15186934-4 2024-09-13T14:26:02.545 INFO:tasks.rados.rados.0.smithi151.stdout:1250: finishing write tid 3 to smithi15186934-4 2024-09-13T14:26:02.545 INFO:tasks.rados.rados.0.smithi151.stdout:1250: finishing write tid 4 to smithi15186934-4 2024-09-13T14:26:02.545 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 759 (ObjNum 492 snap 158 seq_num 492) dirty exists 2024-09-13T14:26:02.548 INFO:tasks.rados.rados.0.smithi151.stdout:1250: left oid 4 (ObjNum 492 snap 158 seq_num 492) 2024-09-13T14:26:02.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:02 smithi184 ceph-mon[33394]: pgmap v176: 89 pgs: 89 active+clean; 221 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:26:02.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:02 smithi184 ceph-mon[33394]: osdmap e373: 8 total, 8 up, 8 in 2024-09-13T14:26:02.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:02 smithi151 ceph-mon[104450]: pgmap v176: 89 pgs: 89 active+clean; 221 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:26:02.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:02 smithi151 ceph-mon[104450]: osdmap e373: 8 total, 8 up, 8 in 2024-09-13T14:26:02.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:02 smithi151 ceph-mon[108966]: pgmap v176: 89 pgs: 89 active+clean; 221 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:26:02.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:02 smithi151 ceph-mon[108966]: osdmap e373: 8 total, 8 up, 8 in 2024-09-13T14:26:03.532 INFO:tasks.rados.rados.0.smithi151.stdout:1250: done (1 left) 2024-09-13T14:26:03.532 INFO:tasks.rados.rados.0.smithi151.stdout:1251: done (0 left) 2024-09-13T14:26:03.532 INFO:tasks.rados.rados.0.smithi151.stdout:1252: snap_remove snap 154 2024-09-13T14:26:03.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:03 smithi184 ceph-mon[33394]: osdmap e374: 8 total, 8 up, 8 in 2024-09-13T14:26:03.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:03 smithi151 ceph-mon[104450]: osdmap e374: 8 total, 8 up, 8 in 2024-09-13T14:26:03.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:03 smithi151 ceph-mon[108966]: osdmap e374: 8 total, 8 up, 8 in 2024-09-13T14:26:04.468 INFO:tasks.rados.rados.0.smithi151.stdout:1252: done (0 left) 2024-09-13T14:26:04.468 INFO:tasks.rados.rados.0.smithi151.stdout:1253: read oid 18 snap -1 2024-09-13T14:26:04.468 INFO:tasks.rados.rados.0.smithi151.stdout:1253: expect (ObjNum 491 snap 158 seq_num 491) 2024-09-13T14:26:04.468 INFO:tasks.rados.rados.0.smithi151.stdout:1254: snap_remove snap 134 2024-09-13T14:26:04.529 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:26:04 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:26:04.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:04.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:04 smithi184 ceph-mon[33394]: pgmap v179: 89 pgs: 89 active+clean; 223 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.3 MiB/s rd, 2.7 MiB/s wr, 25 op/s 2024-09-13T14:26:04.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:04 smithi184 ceph-mon[33394]: osdmap e375: 8 total, 8 up, 8 in 2024-09-13T14:26:04.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:04 smithi184 ceph-mon[33394]: osdmap e376: 8 total, 8 up, 8 in 2024-09-13T14:26:04.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:04 smithi151 ceph-mon[108966]: pgmap v179: 89 pgs: 89 active+clean; 223 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.3 MiB/s rd, 2.7 MiB/s wr, 25 op/s 2024-09-13T14:26:04.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:04 smithi151 ceph-mon[108966]: osdmap e375: 8 total, 8 up, 8 in 2024-09-13T14:26:04.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:04 smithi151 ceph-mon[108966]: osdmap e376: 8 total, 8 up, 8 in 2024-09-13T14:26:04.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:04 smithi151 ceph-mon[104450]: pgmap v179: 89 pgs: 89 active+clean; 223 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.3 MiB/s rd, 2.7 MiB/s wr, 25 op/s 2024-09-13T14:26:04.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:04 smithi151 ceph-mon[104450]: osdmap e375: 8 total, 8 up, 8 in 2024-09-13T14:26:04.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:04 smithi151 ceph-mon[104450]: osdmap e376: 8 total, 8 up, 8 in 2024-09-13T14:26:04.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:26:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:26:04] "GET /metrics HTTP/1.1" 200 36389 "" "Prometheus/2.51.0" 2024-09-13T14:26:05.474 INFO:tasks.rados.rados.0.smithi151.stdout:1253: expect (ObjNum 491 snap 158 seq_num 491) 2024-09-13T14:26:05.615 INFO:tasks.rados.rados.0.smithi151.stdout:1253: done (1 left) 2024-09-13T14:26:05.615 INFO:tasks.rados.rados.0.smithi151.stdout:1254: done (0 left) 2024-09-13T14:26:05.615 INFO:tasks.rados.rados.0.smithi151.stdout:1255: rollback oid 50 current snap is 159 2024-09-13T14:26:05.615 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 50 to 157 2024-09-13T14:26:05.615 INFO:tasks.rados.rados.0.smithi151.stdout:1256: write oid 7 current snap is 159 2024-09-13T14:26:05.615 INFO:tasks.rados.rados.0.smithi151.stdout:1256: seq_num 493 ranges {685851=544349,1799505=486346} 2024-09-13T14:26:05.621 INFO:tasks.rados.rados.0.smithi151.stdout:1256: writing smithi15186934-7 from 685851 to 1230200 tid 1 2024-09-13T14:26:05.627 INFO:tasks.rados.rados.0.smithi151.stdout:1256: writing smithi15186934-7 from 1799505 to 2285851 tid 2 2024-09-13T14:26:05.627 INFO:tasks.rados.rados.0.smithi151.stdout:1257: write oid 25 current snap is 159 2024-09-13T14:26:05.628 INFO:tasks.rados.rados.0.smithi151.stdout:1255: finishing rollback tid 0 to smithi15186934-50 2024-09-13T14:26:05.628 INFO:tasks.rados.rados.0.smithi151.stdout:1257: seq_num 494 ranges {702658=619598,1857987=444671} 2024-09-13T14:26:05.634 INFO:tasks.rados.rados.0.smithi151.stdout:1257: writing smithi15186934-25 from 702658 to 1322256 tid 1 2024-09-13T14:26:05.640 INFO:tasks.rados.rados.0.smithi151.stdout:1257: writing smithi15186934-25 from 1857987 to 2302658 tid 2 2024-09-13T14:26:05.640 INFO:tasks.rados.rados.0.smithi151.stdout:1258: delete oid 30 current snap is 159 2024-09-13T14:26:05.640 INFO:tasks.rados.rados.0.smithi151.stdout:1255: finishing rollback tid 1 to smithi15186934-50 2024-09-13T14:26:05.641 INFO:tasks.rados.rados.0.smithi151.stdout:1255: finishing rollback tid 2 to smithi15186934-50 2024-09-13T14:26:05.641 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 50 v 400 (ObjNum 476 snap 156 seq_num 476) dirty exists 2024-09-13T14:26:05.641 INFO:tasks.rados.rados.0.smithi151.stdout:1256: finishing write tid 1 to smithi15186934-7 2024-09-13T14:26:05.641 INFO:tasks.rados.rados.0.smithi151.stdout:1256: finishing write tid 2 to smithi15186934-7 2024-09-13T14:26:05.641 INFO:tasks.rados.rados.0.smithi151.stdout:1256: finishing write tid 3 to smithi15186934-7 2024-09-13T14:26:05.641 INFO:tasks.rados.rados.0.smithi151.stdout:1256: finishing write tid 4 to smithi15186934-7 2024-09-13T14:26:05.641 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 821 (ObjNum 493 snap 159 seq_num 493) dirty exists 2024-09-13T14:26:05.641 INFO:tasks.rados.rados.0.smithi151.stdout:1256: left oid 7 (ObjNum 493 snap 159 seq_num 493) 2024-09-13T14:26:05.641 INFO:tasks.rados.rados.0.smithi151.stdout:1257: finishing write tid 1 to smithi15186934-25 2024-09-13T14:26:05.642 INFO:tasks.rados.rados.0.smithi151.stdout:1255: done (3 left) 2024-09-13T14:26:05.642 INFO:tasks.rados.rados.0.smithi151.stdout:1256: done (2 left) 2024-09-13T14:26:05.642 INFO:tasks.rados.rados.0.smithi151.stdout:1258: done (1 left) 2024-09-13T14:26:05.642 INFO:tasks.rados.rados.0.smithi151.stdout:1259: rollback oid 26 current snap is 159 2024-09-13T14:26:05.642 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 26 to 133 2024-09-13T14:26:05.642 INFO:tasks.rados.rados.0.smithi151.stdout:1260: read oid 7 snap -1 2024-09-13T14:26:05.642 INFO:tasks.rados.rados.0.smithi151.stdout:1260: expect (ObjNum 493 snap 159 seq_num 493) 2024-09-13T14:26:05.642 INFO:tasks.rados.rados.0.smithi151.stdout:1261: delete oid 34 current snap is 159 2024-09-13T14:26:05.643 INFO:tasks.rados.rados.0.smithi151.stdout:1257: finishing write tid 2 to smithi15186934-25 2024-09-13T14:26:05.643 INFO:tasks.rados.rados.0.smithi151.stdout:1257: finishing write tid 3 to smithi15186934-25 2024-09-13T14:26:05.643 INFO:tasks.rados.rados.0.smithi151.stdout:1261: done (3 left) 2024-09-13T14:26:05.643 INFO:tasks.rados.rados.0.smithi151.stdout:1262: write oid 43 current snap is 159 2024-09-13T14:26:05.643 INFO:tasks.rados.rados.0.smithi151.stdout:1262: seq_num 497 ranges {753079=561698,2033433=319646} 2024-09-13T14:26:05.651 INFO:tasks.rados.rados.0.smithi151.stdout:1262: writing smithi15186934-43 from 753079 to 1314777 tid 1 2024-09-13T14:26:05.657 INFO:tasks.rados.rados.0.smithi151.stdout:1262: writing smithi15186934-43 from 2033433 to 2353079 tid 2 2024-09-13T14:26:05.657 INFO:tasks.rados.rados.0.smithi151.stdout:1263: snap_create 2024-09-13T14:26:05.657 INFO:tasks.rados.rados.0.smithi151.stdout:1257: finishing write tid 4 to smithi15186934-25 2024-09-13T14:26:05.657 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 773 (ObjNum 494 snap 159 seq_num 494) dirty exists 2024-09-13T14:26:05.657 INFO:tasks.rados.rados.0.smithi151.stdout:1257: left oid 25 (ObjNum 494 snap 159 seq_num 494) 2024-09-13T14:26:05.657 INFO:tasks.rados.rados.0.smithi151.stdout:1259: finishing rollback tid 0 to smithi15186934-26 2024-09-13T14:26:05.657 INFO:tasks.rados.rados.0.smithi151.stdout:1259: finishing rollback tid 1 to smithi15186934-26 2024-09-13T14:26:05.657 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 465 (ObjNum 33 snap 0 seq_num 2) dirty dne 2024-09-13T14:26:05.657 INFO:tasks.rados.rados.0.smithi151.stdout:1260: expect (ObjNum 493 snap 159 seq_num 493) 2024-09-13T14:26:05.753 INFO:tasks.rados.rados.0.smithi151.stdout:1262: finishing write tid 1 to smithi15186934-43 2024-09-13T14:26:05.753 INFO:tasks.rados.rados.0.smithi151.stdout:1262: finishing write tid 2 to smithi15186934-43 2024-09-13T14:26:05.753 INFO:tasks.rados.rados.0.smithi151.stdout:1262: finishing write tid 3 to smithi15186934-43 2024-09-13T14:26:05.753 INFO:tasks.rados.rados.0.smithi151.stdout:1262: finishing write tid 4 to smithi15186934-43 2024-09-13T14:26:05.753 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 43 v 825 (ObjNum 497 snap 159 seq_num 497) dirty exists 2024-09-13T14:26:05.754 INFO:tasks.rados.rados.0.smithi151.stdout:1262: left oid 43 (ObjNum 497 snap 159 seq_num 497) 2024-09-13T14:26:06.103 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:26:05 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:26:05.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:06.103 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:26:05 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:26:05.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:06.537 INFO:tasks.rados.rados.0.smithi151.stdout:1257: done (4 left) 2024-09-13T14:26:06.537 INFO:tasks.rados.rados.0.smithi151.stdout:1259: done (3 left) 2024-09-13T14:26:06.537 INFO:tasks.rados.rados.0.smithi151.stdout:1260: done (2 left) 2024-09-13T14:26:06.537 INFO:tasks.rados.rados.0.smithi151.stdout:1262: done (1 left) 2024-09-13T14:26:06.538 INFO:tasks.rados.rados.0.smithi151.stdout:1263: done (0 left) 2024-09-13T14:26:06.538 INFO:tasks.rados.rados.0.smithi151.stdout:1264: write oid 44 current snap is 160 2024-09-13T14:26:06.538 INFO:tasks.rados.rados.0.smithi151.stdout:1264: seq_num 498 ranges {769886=636947,2091915=277971} 2024-09-13T14:26:06.545 INFO:tasks.rados.rados.0.smithi151.stdout:1264: writing smithi15186934-44 from 769886 to 1406833 tid 1 2024-09-13T14:26:06.550 INFO:tasks.rados.rados.0.smithi151.stdout:1264: writing smithi15186934-44 from 2091915 to 2369886 tid 2 2024-09-13T14:26:06.550 INFO:tasks.rados.rados.0.smithi151.stdout:1265: read oid 49 snap 151 2024-09-13T14:26:06.551 INFO:tasks.rados.rados.0.smithi151.stdout:1265: expect (ObjNum 462 snap 150 seq_num 462) 2024-09-13T14:26:06.551 INFO:tasks.rados.rados.0.smithi151.stdout:1266: rollback oid 32 current snap is 160 2024-09-13T14:26:06.551 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 32 to 159 2024-09-13T14:26:06.551 INFO:tasks.rados.rados.0.smithi151.stdout:1267: read oid 5 snap -1 2024-09-13T14:26:06.551 INFO:tasks.rados.rados.0.smithi151.stdout:1267: expect (ObjNum 444 snap 141 seq_num 444) 2024-09-13T14:26:06.551 INFO:tasks.rados.rados.0.smithi151.stdout:1268: snap_remove snap 153 2024-09-13T14:26:06.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:06 smithi184 ceph-mon[33394]: pgmap v182: 89 pgs: 89 active+clean; 223 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.3 MiB/s rd, 2.7 MiB/s wr, 25 op/s 2024-09-13T14:26:06.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:06 smithi184 ceph-mon[33394]: osdmap e377: 8 total, 8 up, 8 in 2024-09-13T14:26:06.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:06 smithi151 ceph-mon[108966]: pgmap v182: 89 pgs: 89 active+clean; 223 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.3 MiB/s rd, 2.7 MiB/s wr, 25 op/s 2024-09-13T14:26:06.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:06 smithi151 ceph-mon[108966]: osdmap e377: 8 total, 8 up, 8 in 2024-09-13T14:26:06.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:06 smithi151 ceph-mon[104450]: pgmap v182: 89 pgs: 89 active+clean; 223 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.3 MiB/s rd, 2.7 MiB/s wr, 25 op/s 2024-09-13T14:26:06.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:06 smithi151 ceph-mon[104450]: osdmap e377: 8 total, 8 up, 8 in 2024-09-13T14:26:07.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:26:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:26:06.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi184\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:07.552 INFO:tasks.rados.rados.0.smithi151.stdout:1268: done (4 left) 2024-09-13T14:26:07.552 INFO:tasks.rados.rados.0.smithi151.stdout:1269: delete oid 37 current snap is 160 2024-09-13T14:26:07.552 INFO:tasks.rados.rados.0.smithi151.stdout:1264: finishing write tid 1 to smithi15186934-44 2024-09-13T14:26:07.552 INFO:tasks.rados.rados.0.smithi151.stdout:1266: finishing rollback tid 1 to smithi15186934-32 2024-09-13T14:26:07.552 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 32 v 516 (ObjNum 49 snap 0 seq_num 3064833136) dirty dne 2024-09-13T14:26:07.552 INFO:tasks.rados.rados.0.smithi151.stdout:1264: finishing write tid 2 to smithi15186934-44 2024-09-13T14:26:07.553 INFO:tasks.rados.rados.0.smithi151.stdout:1264: finishing write tid 3 to smithi15186934-44 2024-09-13T14:26:07.553 INFO:tasks.rados.rados.0.smithi151.stdout:1264: finishing write tid 4 to smithi15186934-44 2024-09-13T14:26:07.553 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 373 (ObjNum 498 snap 160 seq_num 498) dirty exists 2024-09-13T14:26:07.553 INFO:tasks.rados.rados.0.smithi151.stdout:1264: left oid 44 (ObjNum 498 snap 160 seq_num 498) 2024-09-13T14:26:07.553 INFO:tasks.rados.rados.0.smithi151.stdout:1267: expect (ObjNum 444 snap 141 seq_num 444) 2024-09-13T14:26:07.697 INFO:tasks.rados.rados.0.smithi151.stdout:1265: expect (ObjNum 462 snap 150 seq_num 462) 2024-09-13T14:26:07.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:07 smithi184 ceph-mon[33394]: osdmap e378: 8 total, 8 up, 8 in 2024-09-13T14:26:07.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:07 smithi151 ceph-mon[104450]: osdmap e378: 8 total, 8 up, 8 in 2024-09-13T14:26:07.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:07 smithi151 ceph-mon[108966]: osdmap e378: 8 total, 8 up, 8 in 2024-09-13T14:26:07.916 INFO:tasks.rados.rados.0.smithi151.stdout:1264: done (4 left) 2024-09-13T14:26:07.916 INFO:tasks.rados.rados.0.smithi151.stdout:1265: done (3 left) 2024-09-13T14:26:07.916 INFO:tasks.rados.rados.0.smithi151.stdout:1266: done (2 left) 2024-09-13T14:26:07.916 INFO:tasks.rados.rados.0.smithi151.stdout:1267: done (1 left) 2024-09-13T14:26:07.917 INFO:tasks.rados.rados.0.smithi151.stdout:1269: done (0 left) 2024-09-13T14:26:07.917 INFO:tasks.rados.rados.0.smithi151.stdout:1270: snap_remove snap 151 2024-09-13T14:26:08.549 INFO:tasks.rados.rados.0.smithi151.stdout:1270: done (0 left) 2024-09-13T14:26:08.549 INFO:tasks.rados.rados.0.smithi151.stdout:1271: snap_create 2024-09-13T14:26:08.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:08 smithi184 ceph-mon[33394]: pgmap v185: 89 pgs: 89 active+clean; 210 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 13 MiB/s rd, 1.2 MiB/s wr, 15 op/s 2024-09-13T14:26:08.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:08 smithi184 ceph-mon[33394]: osdmap e379: 8 total, 8 up, 8 in 2024-09-13T14:26:08.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:08 smithi151 ceph-mon[104450]: pgmap v185: 89 pgs: 89 active+clean; 210 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 13 MiB/s rd, 1.2 MiB/s wr, 15 op/s 2024-09-13T14:26:08.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:08 smithi151 ceph-mon[104450]: osdmap e379: 8 total, 8 up, 8 in 2024-09-13T14:26:08.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:08 smithi151 ceph-mon[108966]: pgmap v185: 89 pgs: 89 active+clean; 210 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 13 MiB/s rd, 1.2 MiB/s wr, 15 op/s 2024-09-13T14:26:08.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:08 smithi151 ceph-mon[108966]: osdmap e379: 8 total, 8 up, 8 in 2024-09-13T14:26:09.565 INFO:tasks.rados.rados.0.smithi151.stdout:1271: done (0 left) 2024-09-13T14:26:09.565 INFO:tasks.rados.rados.0.smithi151.stdout:1272: delete oid 27 current snap is 161 2024-09-13T14:26:09.569 INFO:tasks.rados.rados.0.smithi151.stdout:1272: done (0 left) 2024-09-13T14:26:09.569 INFO:tasks.rados.rados.0.smithi151.stdout:1273: read oid 6 snap 158 2024-09-13T14:26:09.569 INFO:tasks.rados.rados.0.smithi151.stdout:1273: expect (ObjNum 426 snap 135 seq_num 426) 2024-09-13T14:26:09.569 INFO:tasks.rados.rados.0.smithi151.stdout:1274: write oid 24 current snap is 161 2024-09-13T14:26:09.569 INFO:tasks.rados.rados.0.smithi151.stdout:1274: seq_num 501 ranges {420307=462694,1351008=617495} 2024-09-13T14:26:09.574 INFO:tasks.rados.rados.0.smithi151.stdout:1274: writing smithi15186934-24 from 420307 to 883001 tid 1 2024-09-13T14:26:09.580 INFO:tasks.rados.rados.0.smithi151.stdout:1274: writing smithi15186934-24 from 1351008 to 1968503 tid 2 2024-09-13T14:26:09.580 INFO:tasks.rados.rados.0.smithi151.stdout:1275: write oid 7 current snap is 161 2024-09-13T14:26:09.580 INFO:tasks.rados.rados.0.smithi151.stdout:1275: seq_num 502 ranges {437114=654296,1525843=477506} 2024-09-13T14:26:09.586 INFO:tasks.rados.rados.0.smithi151.stdout:1275: writing smithi15186934-7 from 437114 to 1091410 tid 1 2024-09-13T14:26:09.591 INFO:tasks.rados.rados.0.smithi151.stdout:1275: writing smithi15186934-7 from 1525843 to 2003349 tid 2 2024-09-13T14:26:09.592 INFO:tasks.rados.rados.0.smithi151.stdout:1276: snap_create 2024-09-13T14:26:09.592 INFO:tasks.rados.rados.0.smithi151.stdout:1273: expect (ObjNum 426 snap 135 seq_num 426) 2024-09-13T14:26:09.812 INFO:tasks.rados.rados.0.smithi151.stdout:1274: finishing write tid 1 to smithi15186934-24 2024-09-13T14:26:09.813 INFO:tasks.rados.rados.0.smithi151.stdout:1274: finishing write tid 2 to smithi15186934-24 2024-09-13T14:26:09.813 INFO:tasks.rados.rados.0.smithi151.stdout:1274: finishing write tid 3 to smithi15186934-24 2024-09-13T14:26:09.813 INFO:tasks.rados.rados.0.smithi151.stdout:1274: finishing write tid 4 to smithi15186934-24 2024-09-13T14:26:09.813 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 24 v 840 (ObjNum 501 snap 161 seq_num 501) dirty exists 2024-09-13T14:26:09.813 INFO:tasks.rados.rados.0.smithi151.stdout:1274: left oid 24 (ObjNum 501 snap 161 seq_num 501) 2024-09-13T14:26:09.813 INFO:tasks.rados.rados.0.smithi151.stdout:1275: finishing write tid 1 to smithi15186934-7 2024-09-13T14:26:09.813 INFO:tasks.rados.rados.0.smithi151.stdout:1275: finishing write tid 2 to smithi15186934-7 2024-09-13T14:26:09.813 INFO:tasks.rados.rados.0.smithi151.stdout:1275: finishing write tid 3 to smithi15186934-7 2024-09-13T14:26:09.813 INFO:tasks.rados.rados.0.smithi151.stdout:1275: finishing write tid 4 to smithi15186934-7 2024-09-13T14:26:09.813 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 844 (ObjNum 502 snap 161 seq_num 502) dirty exists 2024-09-13T14:26:09.813 INFO:tasks.rados.rados.0.smithi151.stdout:1275: left oid 7 (ObjNum 502 snap 161 seq_num 502) 2024-09-13T14:26:09.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:09 smithi184 ceph-mon[33394]: osdmap e380: 8 total, 8 up, 8 in 2024-09-13T14:26:09.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:09 smithi151 ceph-mon[104450]: osdmap e380: 8 total, 8 up, 8 in 2024-09-13T14:26:09.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:09 smithi151 ceph-mon[108966]: osdmap e380: 8 total, 8 up, 8 in 2024-09-13T14:26:10.560 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:26:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:26:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:10.568 INFO:tasks.rados.rados.0.smithi151.stdout:1273: done (3 left) 2024-09-13T14:26:10.568 INFO:tasks.rados.rados.0.smithi151.stdout:1274: done (2 left) 2024-09-13T14:26:10.568 INFO:tasks.rados.rados.0.smithi151.stdout:1275: done (1 left) 2024-09-13T14:26:10.568 INFO:tasks.rados.rados.0.smithi151.stdout:1276: done (0 left) 2024-09-13T14:26:10.568 INFO:tasks.rados.rados.0.smithi151.stdout:1277: rollback oid 42 current snap is 162 2024-09-13T14:26:10.569 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 42 to 161 2024-09-13T14:26:10.569 INFO:tasks.rados.rados.0.smithi151.stdout:1278: rollback oid 39 current snap is 162 2024-09-13T14:26:10.569 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 39 to 130 2024-09-13T14:26:10.569 INFO:tasks.rados.rados.0.smithi151.stdout:1279: write oid 21 current snap is 162 2024-09-13T14:26:10.569 INFO:tasks.rados.rados.0.smithi151.stdout:1279: seq_num 503 ranges {453921=729545,1584325=621164} 2024-09-13T14:26:10.575 INFO:tasks.rados.rados.0.smithi151.stdout:1279: writing smithi15186934-21 from 453921 to 1183466 tid 1 2024-09-13T14:26:10.581 INFO:tasks.rados.rados.0.smithi151.stdout:1279: writing smithi15186934-21 from 1584325 to 2205489 tid 2 2024-09-13T14:26:10.581 INFO:tasks.rados.rados.0.smithi151.stdout:1280: read oid 50 snap -1 2024-09-13T14:26:10.581 INFO:tasks.rados.rados.0.smithi151.stdout:1280: expect (ObjNum 476 snap 156 seq_num 476) 2024-09-13T14:26:10.581 INFO:tasks.rados.rados.0.smithi151.stdout:1281: write oid 38 current snap is 162 2024-09-13T14:26:10.581 INFO:tasks.rados.rados.0.smithi151.stdout:1277: finishing rollback tid 0 to smithi15186934-42 2024-09-13T14:26:10.581 INFO:tasks.rados.rados.0.smithi151.stdout:1278: finishing rollback tid 1 to smithi15186934-39 2024-09-13T14:26:10.581 INFO:tasks.rados.rados.0.smithi151.stdout:1277: finishing rollback tid 1 to smithi15186934-42 2024-09-13T14:26:10.581 INFO:tasks.rados.rados.0.smithi151.stdout:1281: seq_num 504 ranges {470728=404794,1526454=481175} 2024-09-13T14:26:10.585 INFO:tasks.rados.rados.0.smithi151.stdout:1281: writing smithi15186934-38 from 470728 to 875522 tid 1 2024-09-13T14:26:10.592 INFO:tasks.rados.rados.0.smithi151.stdout:1281: writing smithi15186934-38 from 1526454 to 2007629 tid 2 2024-09-13T14:26:10.592 INFO:tasks.rados.rados.0.smithi151.stdout:1282: read oid 29 snap -1 2024-09-13T14:26:10.592 INFO:tasks.rados.rados.0.smithi151.stdout:1282: expect (ObjNum 445 snap 141 seq_num 445) 2024-09-13T14:26:10.592 INFO:tasks.rados.rados.0.smithi151.stdout:1283: write oid 17 current snap is 162 2024-09-13T14:26:10.592 INFO:tasks.rados.rados.0.smithi151.stdout:1278: finishing rollback tid 2 to smithi15186934-39 2024-09-13T14:26:10.592 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 402 (ObjNum 390 snap 121 seq_num 390) dirty exists 2024-09-13T14:26:10.593 INFO:tasks.rados.rados.0.smithi151.stdout:1277: finishing rollback tid 2 to smithi15186934-42 2024-09-13T14:26:10.593 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 650 (ObjNum 479 snap 156 seq_num 479) dirty exists 2024-09-13T14:26:10.593 INFO:tasks.rados.rados.0.smithi151.stdout:1279: finishing write tid 1 to smithi15186934-21 2024-09-13T14:26:10.593 INFO:tasks.rados.rados.0.smithi151.stdout:1279: finishing write tid 2 to smithi15186934-21 2024-09-13T14:26:10.593 INFO:tasks.rados.rados.0.smithi151.stdout:1283: seq_num 505 ranges {487535=480043,1584936=624833} 2024-09-13T14:26:10.597 INFO:tasks.rados.rados.0.smithi151.stdout:1283: writing smithi15186934-17 from 487535 to 967578 tid 1 2024-09-13T14:26:10.604 INFO:tasks.rados.rados.0.smithi151.stdout:1283: writing smithi15186934-17 from 1584936 to 2209769 tid 2 2024-09-13T14:26:10.604 INFO:tasks.rados.rados.0.smithi151.stdout:1277: done (6 left) 2024-09-13T14:26:10.604 INFO:tasks.rados.rados.0.smithi151.stdout:1278: done (5 left) 2024-09-13T14:26:10.604 INFO:tasks.rados.rados.0.smithi151.stdout:1284: write oid 23 current snap is 162 2024-09-13T14:26:10.605 INFO:tasks.rados.rados.0.smithi151.stdout:1284: seq_num 506 ranges {504342=555292,1643418=484844} 2024-09-13T14:26:10.610 INFO:tasks.rados.rados.0.smithi151.stdout:1284: writing smithi15186934-23 from 504342 to 1059634 tid 1 2024-09-13T14:26:10.616 INFO:tasks.rados.rados.0.smithi151.stdout:1284: writing smithi15186934-23 from 1643418 to 2128262 tid 2 2024-09-13T14:26:10.616 INFO:tasks.rados.rados.0.smithi151.stdout:1285: read oid 24 snap 158 2024-09-13T14:26:10.616 INFO:tasks.rados.rados.0.smithi151.stdout:1285: expect deleted 2024-09-13T14:26:10.616 INFO:tasks.rados.rados.0.smithi151.stdout:1279: finishing write tid 3 to smithi15186934-21 2024-09-13T14:26:10.616 INFO:tasks.rados.rados.0.smithi151.stdout:1279: finishing write tid 4 to smithi15186934-21 2024-09-13T14:26:10.617 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 21 v 283 (ObjNum 503 snap 162 seq_num 503) dirty exists 2024-09-13T14:26:10.617 INFO:tasks.rados.rados.0.smithi151.stdout:1279: left oid 21 (ObjNum 503 snap 162 seq_num 503) 2024-09-13T14:26:10.617 INFO:tasks.rados.rados.0.smithi151.stdout:1280: expect (ObjNum 476 snap 156 seq_num 476) 2024-09-13T14:26:10.678 INFO:tasks.rados.rados.0.smithi151.stdout:1281: finishing write tid 1 to smithi15186934-38 2024-09-13T14:26:10.678 INFO:tasks.rados.rados.0.smithi151.stdout:1281: finishing write tid 2 to smithi15186934-38 2024-09-13T14:26:10.679 INFO:tasks.rados.rados.0.smithi151.stdout:1281: finishing write tid 3 to smithi15186934-38 2024-09-13T14:26:10.679 INFO:tasks.rados.rados.0.smithi151.stdout:1281: finishing write tid 4 to smithi15186934-38 2024-09-13T14:26:10.679 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 406 (ObjNum 504 snap 162 seq_num 504) dirty exists 2024-09-13T14:26:10.679 INFO:tasks.rados.rados.0.smithi151.stdout:1281: left oid 38 (ObjNum 504 snap 162 seq_num 504) 2024-09-13T14:26:10.679 INFO:tasks.rados.rados.0.smithi151.stdout:1279: done (6 left) 2024-09-13T14:26:10.679 INFO:tasks.rados.rados.0.smithi151.stdout:1280: done (5 left) 2024-09-13T14:26:10.679 INFO:tasks.rados.rados.0.smithi151.stdout:1281: done (4 left) 2024-09-13T14:26:10.679 INFO:tasks.rados.rados.0.smithi151.stdout:1286: snap_create 2024-09-13T14:26:10.679 INFO:tasks.rados.rados.0.smithi151.stdout:1282: expect (ObjNum 445 snap 141 seq_num 445) 2024-09-13T14:26:10.791 INFO:tasks.rados.rados.0.smithi151.stdout:1283: finishing write tid 1 to smithi15186934-17 2024-09-13T14:26:10.791 INFO:tasks.rados.rados.0.smithi151.stdout:1283: finishing write tid 2 to smithi15186934-17 2024-09-13T14:26:10.791 INFO:tasks.rados.rados.0.smithi151.stdout:1283: finishing write tid 3 to smithi15186934-17 2024-09-13T14:26:10.791 INFO:tasks.rados.rados.0.smithi151.stdout:1283: finishing write tid 4 to smithi15186934-17 2024-09-13T14:26:10.791 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 848 (ObjNum 505 snap 162 seq_num 505) dirty exists 2024-09-13T14:26:10.792 INFO:tasks.rados.rados.0.smithi151.stdout:1283: left oid 17 (ObjNum 505 snap 162 seq_num 505) 2024-09-13T14:26:10.792 INFO:tasks.rados.rados.0.smithi151.stdout:1284: finishing write tid 1 to smithi15186934-23 2024-09-13T14:26:10.792 INFO:tasks.rados.rados.0.smithi151.stdout:1284: finishing write tid 2 to smithi15186934-23 2024-09-13T14:26:10.792 INFO:tasks.rados.rados.0.smithi151.stdout:1284: finishing write tid 3 to smithi15186934-23 2024-09-13T14:26:10.792 INFO:tasks.rados.rados.0.smithi151.stdout:1284: finishing write tid 4 to smithi15186934-23 2024-09-13T14:26:10.792 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 23 v 584 (ObjNum 506 snap 162 seq_num 506) dirty exists 2024-09-13T14:26:10.792 INFO:tasks.rados.rados.0.smithi151.stdout:1284: left oid 23 (ObjNum 506 snap 162 seq_num 506) 2024-09-13T14:26:10.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:10 smithi184 ceph-mon[33394]: pgmap v188: 89 pgs: 89 active+clean; 209 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 13 MiB/s rd, 1.2 MiB/s wr, 16 op/s 2024-09-13T14:26:10.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:10 smithi184 ceph-mon[33394]: osdmap e381: 8 total, 8 up, 8 in 2024-09-13T14:26:10.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:10 smithi151 ceph-mon[104450]: pgmap v188: 89 pgs: 89 active+clean; 209 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 13 MiB/s rd, 1.2 MiB/s wr, 16 op/s 2024-09-13T14:26:10.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:10 smithi151 ceph-mon[104450]: osdmap e381: 8 total, 8 up, 8 in 2024-09-13T14:26:10.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:10 smithi151 ceph-mon[108966]: pgmap v188: 89 pgs: 89 active+clean; 209 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 13 MiB/s rd, 1.2 MiB/s wr, 16 op/s 2024-09-13T14:26:10.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:10 smithi151 ceph-mon[108966]: osdmap e381: 8 total, 8 up, 8 in 2024-09-13T14:26:11.580 INFO:tasks.rados.rados.0.smithi151.stdout:1282: done (4 left) 2024-09-13T14:26:11.580 INFO:tasks.rados.rados.0.smithi151.stdout:1283: done (3 left) 2024-09-13T14:26:11.581 INFO:tasks.rados.rados.0.smithi151.stdout:1284: done (2 left) 2024-09-13T14:26:11.581 INFO:tasks.rados.rados.0.smithi151.stdout:1285: done (1 left) 2024-09-13T14:26:11.581 INFO:tasks.rados.rados.0.smithi151.stdout:1286: done (0 left) 2024-09-13T14:26:11.581 INFO:tasks.rados.rados.0.smithi151.stdout:1287: snap_create 2024-09-13T14:26:11.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:11 smithi184 ceph-mon[33394]: osdmap e382: 8 total, 8 up, 8 in 2024-09-13T14:26:11.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:11 smithi151 ceph-mon[104450]: osdmap e382: 8 total, 8 up, 8 in 2024-09-13T14:26:11.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:11 smithi151 ceph-mon[108966]: osdmap e382: 8 total, 8 up, 8 in 2024-09-13T14:26:12.580 INFO:tasks.rados.rados.0.smithi151.stdout:1287: done (0 left) 2024-09-13T14:26:12.580 INFO:tasks.rados.rados.0.smithi151.stdout:1288: read oid 13 snap -1 2024-09-13T14:26:12.580 INFO:tasks.rados.rados.0.smithi151.stdout:1288: expect (ObjNum 270 snap 86 seq_num 270) 2024-09-13T14:26:12.580 INFO:tasks.rados.rados.0.smithi151.stdout:1289: read oid 44 snap -1 2024-09-13T14:26:12.581 INFO:tasks.rados.rados.0.smithi151.stdout:1289: expect (ObjNum 498 snap 160 seq_num 498) 2024-09-13T14:26:12.581 INFO:tasks.rados.rados.0.smithi151.stdout:1290: snap_remove snap 159 2024-09-13T14:26:12.756 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:12 smithi184 ceph-mon[33394]: pgmap v191: 89 pgs: 89 active+clean; 209 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:26:12.756 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:12 smithi184 ceph-mon[33394]: osdmap e383: 8 total, 8 up, 8 in 2024-09-13T14:26:12.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:12 smithi151 ceph-mon[104450]: pgmap v191: 89 pgs: 89 active+clean; 209 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:26:12.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:12 smithi151 ceph-mon[104450]: osdmap e383: 8 total, 8 up, 8 in 2024-09-13T14:26:12.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:12 smithi151 ceph-mon[108966]: pgmap v191: 89 pgs: 89 active+clean; 209 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:26:12.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:12 smithi151 ceph-mon[108966]: osdmap e383: 8 total, 8 up, 8 in 2024-09-13T14:26:13.596 INFO:tasks.rados.rados.0.smithi151.stdout:1290: done (2 left) 2024-09-13T14:26:13.596 INFO:tasks.rados.rados.0.smithi151.stdout:1291: read oid 35 snap -1 2024-09-13T14:26:13.596 INFO:tasks.rados.rados.0.smithi151.stdout:1291: expect (ObjNum 341 snap 105 seq_num 341) 2024-09-13T14:26:13.596 INFO:tasks.rados.rados.0.smithi151.stdout:1292: rollback oid 29 current snap is 164 2024-09-13T14:26:13.596 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 29 to 129 2024-09-13T14:26:13.596 INFO:tasks.rados.rados.0.smithi151.stdout:1289: expect (ObjNum 498 snap 160 seq_num 498) 2024-09-13T14:26:13.755 INFO:tasks.rados.rados.0.smithi151.stdout:1288: expect (ObjNum 270 snap 86 seq_num 270) 2024-09-13T14:26:13.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:13 smithi184 ceph-mon[33394]: osdmap e384: 8 total, 8 up, 8 in 2024-09-13T14:26:13.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:13 smithi151 ceph-mon[104450]: osdmap e384: 8 total, 8 up, 8 in 2024-09-13T14:26:13.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:13 smithi151 ceph-mon[108966]: osdmap e384: 8 total, 8 up, 8 in 2024-09-13T14:26:13.894 INFO:tasks.rados.rados.0.smithi151.stdout:1292: finishing rollback tid 0 to smithi15186934-29 2024-09-13T14:26:13.894 INFO:tasks.rados.rados.0.smithi151.stdout:1292: finishing rollback tid 1 to smithi15186934-29 2024-09-13T14:26:13.894 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 29 v 370 (ObjNum 81 snap 0 seq_num 0) dirty dne 2024-09-13T14:26:13.894 INFO:tasks.rados.rados.0.smithi151.stdout:1291: expect (ObjNum 341 snap 105 seq_num 341) 2024-09-13T14:26:14.014 INFO:tasks.rados.rados.0.smithi151.stdout:1288: done (3 left) 2024-09-13T14:26:14.014 INFO:tasks.rados.rados.0.smithi151.stdout:1289: done (2 left) 2024-09-13T14:26:14.014 INFO:tasks.rados.rados.0.smithi151.stdout:1291: done (1 left) 2024-09-13T14:26:14.014 INFO:tasks.rados.rados.0.smithi151.stdout:1292: done (0 left) 2024-09-13T14:26:14.014 INFO:tasks.rados.rados.0.smithi151.stdout:1293: write oid 30 current snap is 164 2024-09-13T14:26:14.014 INFO:tasks.rados.rados.0.smithi151.stdout:1293: seq_num 507 ranges {521149=630541,1701900=628502} 2024-09-13T14:26:14.020 INFO:tasks.rados.rados.0.smithi151.stdout:1293: writing smithi15186934-30 from 521149 to 1151690 tid 1 2024-09-13T14:26:14.026 INFO:tasks.rados.rados.0.smithi151.stdout:1293: writing smithi15186934-30 from 1701900 to 2330402 tid 2 2024-09-13T14:26:14.026 INFO:tasks.rados.rados.0.smithi151.stdout:1294: write oid 33 current snap is 164 2024-09-13T14:26:14.026 INFO:tasks.rados.rados.0.smithi151.stdout:1293: finishing write tid 1 to smithi15186934-30 2024-09-13T14:26:14.027 INFO:tasks.rados.rados.0.smithi151.stdout:1294: seq_num 508 ranges {537956=705790,1644029=772160} 2024-09-13T14:26:14.033 INFO:tasks.rados.rados.0.smithi151.stdout:1294: writing smithi15186934-33 from 537956 to 1243746 tid 1 2024-09-13T14:26:14.040 INFO:tasks.rados.rados.0.smithi151.stdout:1294: writing smithi15186934-33 from 1644029 to 2416189 tid 2 2024-09-13T14:26:14.040 INFO:tasks.rados.rados.0.smithi151.stdout:1295: write oid 10 current snap is 164 2024-09-13T14:26:14.040 INFO:tasks.rados.rados.0.smithi151.stdout:1295: seq_num 509 ranges {554763=781039,2102511=452252} 2024-09-13T14:26:14.047 INFO:tasks.rados.rados.0.smithi151.stdout:1295: writing smithi15186934-10 from 554763 to 1335802 tid 1 2024-09-13T14:26:14.055 INFO:tasks.rados.rados.0.smithi151.stdout:1295: writing smithi15186934-10 from 2102511 to 2554763 tid 2 2024-09-13T14:26:14.055 INFO:tasks.rados.rados.0.smithi151.stdout:1296: read oid 18 snap -1 2024-09-13T14:26:14.055 INFO:tasks.rados.rados.0.smithi151.stdout:1296: expect (ObjNum 491 snap 158 seq_num 491) 2024-09-13T14:26:14.055 INFO:tasks.rados.rados.0.smithi151.stdout:1297: rollback oid 48 current snap is 164 2024-09-13T14:26:14.056 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 48 to 133 2024-09-13T14:26:14.056 INFO:tasks.rados.rados.0.smithi151.stdout:1293: finishing write tid 2 to smithi15186934-30 2024-09-13T14:26:14.056 INFO:tasks.rados.rados.0.smithi151.stdout:1293: finishing write tid 3 to smithi15186934-30 2024-09-13T14:26:14.056 INFO:tasks.rados.rados.0.smithi151.stdout:1293: finishing write tid 4 to smithi15186934-30 2024-09-13T14:26:14.056 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 30 v 515 (ObjNum 507 snap 164 seq_num 507) dirty exists 2024-09-13T14:26:14.056 INFO:tasks.rados.rados.0.smithi151.stdout:1293: left oid 30 (ObjNum 507 snap 164 seq_num 507) 2024-09-13T14:26:14.056 INFO:tasks.rados.rados.0.smithi151.stdout:1294: finishing write tid 1 to smithi15186934-33 2024-09-13T14:26:14.056 INFO:tasks.rados.rados.0.smithi151.stdout:1293: done (4 left) 2024-09-13T14:26:14.056 INFO:tasks.rados.rados.0.smithi151.stdout:1298: rollback oid 46 current snap is 164 2024-09-13T14:26:14.056 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 46 to 163 2024-09-13T14:26:14.056 INFO:tasks.rados.rados.0.smithi151.stdout:1294: finishing write tid 2 to smithi15186934-33 2024-09-13T14:26:14.056 INFO:tasks.rados.rados.0.smithi151.stdout:1294: finishing write tid 3 to smithi15186934-33 2024-09-13T14:26:14.056 INFO:tasks.rados.rados.0.smithi151.stdout:1294: finishing write tid 4 to smithi15186934-33 2024-09-13T14:26:14.057 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 33 v 467 (ObjNum 508 snap 164 seq_num 508) dirty exists 2024-09-13T14:26:14.057 INFO:tasks.rados.rados.0.smithi151.stdout:1294: left oid 33 (ObjNum 508 snap 164 seq_num 508) 2024-09-13T14:26:14.057 INFO:tasks.rados.rados.0.smithi151.stdout:1295: finishing write tid 1 to smithi15186934-10 2024-09-13T14:26:14.057 INFO:tasks.rados.rados.0.smithi151.stdout:1294: done (4 left) 2024-09-13T14:26:14.057 INFO:tasks.rados.rados.0.smithi151.stdout:1299: read oid 36 snap 141 2024-09-13T14:26:14.057 INFO:tasks.rados.rados.0.smithi151.stdout:1299: expect deleted 2024-09-13T14:26:14.057 INFO:tasks.rados.rados.0.smithi151.stdout:1300: read oid 14 snap -1 2024-09-13T14:26:14.057 INFO:tasks.rados.rados.0.smithi151.stdout:1300: expect (ObjNum 440 snap 139 seq_num 440) 2024-09-13T14:26:14.057 INFO:tasks.rados.rados.0.smithi151.stdout:1301: write oid 45 current snap is 164 2024-09-13T14:26:14.057 INFO:tasks.rados.rados.0.smithi151.stdout:1301: seq_num 510 ranges {571570=572641,1877346=694224} 2024-09-13T14:26:14.061 INFO:tasks.rados.rados.0.smithi151.stdout:1301: writing smithi15186934-45 from 571570 to 1144211 tid 1 2024-09-13T14:26:14.069 INFO:tasks.rados.rados.0.smithi151.stdout:1301: writing smithi15186934-45 from 1877346 to 2571570 tid 2 2024-09-13T14:26:14.069 INFO:tasks.rados.rados.0.smithi151.stdout:1302: read oid 20 snap -1 2024-09-13T14:26:14.070 INFO:tasks.rados.rados.0.smithi151.stdout:1302: expect (ObjNum 459 snap 150 seq_num 459) 2024-09-13T14:26:14.070 INFO:tasks.rados.rados.0.smithi151.stdout:1303: read oid 12 snap 133 2024-09-13T14:26:14.070 INFO:tasks.rados.rados.0.smithi151.stdout:1303: expect (ObjNum 258 snap 81 seq_num 258) 2024-09-13T14:26:14.070 INFO:tasks.rados.rados.0.smithi151.stdout:1304: snap_remove snap 145 2024-09-13T14:26:14.477 INFO:tasks.rados.rados.0.smithi151.stdout:1304: done (9 left) 2024-09-13T14:26:14.477 INFO:tasks.rados.rados.0.smithi151.stdout:1305: write oid 1 current snap is 164 2024-09-13T14:26:14.477 INFO:tasks.rados.rados.0.smithi151.stdout:1305: seq_num 511 ranges {588377=647890,1935828=635840} 2024-09-13T14:26:14.484 INFO:tasks.rados.rados.0.smithi151.stdout:1305: writing smithi15186934-1 from 588377 to 1236267 tid 1 2024-09-13T14:26:14.492 INFO:tasks.rados.rados.0.smithi151.stdout:1305: writing smithi15186934-1 from 1935828 to 2571668 tid 2 2024-09-13T14:26:14.492 INFO:tasks.rados.rados.0.smithi151.stdout:1306: snap_create 2024-09-13T14:26:14.492 INFO:tasks.rados.rados.0.smithi151.stdout:1300: expect (ObjNum 440 snap 139 seq_num 440) 2024-09-13T14:26:14.589 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:26:14 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:26:14.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:14.607 INFO:tasks.rados.rados.0.smithi151.stdout:1297: finishing rollback tid 0 to smithi15186934-48 2024-09-13T14:26:14.607 INFO:tasks.rados.rados.0.smithi151.stdout:1297: finishing rollback tid 1 to smithi15186934-48 2024-09-13T14:26:14.608 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 48 v 792 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-13T14:26:14.608 INFO:tasks.rados.rados.0.smithi151.stdout:1295: finishing write tid 2 to smithi15186934-10 2024-09-13T14:26:14.608 INFO:tasks.rados.rados.0.smithi151.stdout:1296: expect (ObjNum 491 snap 158 seq_num 491) 2024-09-13T14:26:14.745 INFO:tasks.rados.rados.0.smithi151.stdout:1295: finishing write tid 3 to smithi15186934-10 2024-09-13T14:26:14.745 INFO:tasks.rados.rados.0.smithi151.stdout:1295: finishing write tid 4 to smithi15186934-10 2024-09-13T14:26:14.745 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 481 (ObjNum 509 snap 164 seq_num 509) dirty exists 2024-09-13T14:26:14.745 INFO:tasks.rados.rados.0.smithi151.stdout:1295: left oid 10 (ObjNum 509 snap 164 seq_num 509) 2024-09-13T14:26:14.745 INFO:tasks.rados.rados.0.smithi151.stdout:1298: finishing rollback tid 0 to smithi15186934-46 2024-09-13T14:26:14.745 INFO:tasks.rados.rados.0.smithi151.stdout:1298: finishing rollback tid 1 to smithi15186934-46 2024-09-13T14:26:14.745 INFO:tasks.rados.rados.0.smithi151.stdout:1298: finishing rollback tid 2 to smithi15186934-46 2024-09-13T14:26:14.745 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 46 v 485 (ObjNum 489 snap 158 seq_num 489) dirty exists 2024-09-13T14:26:14.745 INFO:tasks.rados.rados.0.smithi151.stdout:1301: finishing write tid 1 to smithi15186934-45 2024-09-13T14:26:14.746 INFO:tasks.rados.rados.0.smithi151.stdout:1301: finishing write tid 2 to smithi15186934-45 2024-09-13T14:26:14.746 INFO:tasks.rados.rados.0.smithi151.stdout:1301: finishing write tid 3 to smithi15186934-45 2024-09-13T14:26:14.746 INFO:tasks.rados.rados.0.smithi151.stdout:1301: finishing write tid 4 to smithi15186934-45 2024-09-13T14:26:14.746 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 795 (ObjNum 510 snap 164 seq_num 510) dirty exists 2024-09-13T14:26:14.746 INFO:tasks.rados.rados.0.smithi151.stdout:1301: left oid 45 (ObjNum 510 snap 164 seq_num 510) 2024-09-13T14:26:14.746 INFO:tasks.rados.rados.0.smithi151.stdout:1303: expect (ObjNum 258 snap 81 seq_num 258) 2024-09-13T14:26:14.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:14 smithi184 ceph-mon[33394]: pgmap v194: 89 pgs: 89 active+clean; 206 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.3 MiB/s rd, 1.6 MiB/s wr, 14 op/s 2024-09-13T14:26:14.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:14 smithi184 ceph-mon[33394]: osdmap e385: 8 total, 8 up, 8 in 2024-09-13T14:26:14.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:14 smithi184 ceph-mon[33394]: osdmap e386: 8 total, 8 up, 8 in 2024-09-13T14:26:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:14 smithi151 ceph-mon[104450]: pgmap v194: 89 pgs: 89 active+clean; 206 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.3 MiB/s rd, 1.6 MiB/s wr, 14 op/s 2024-09-13T14:26:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:14 smithi151 ceph-mon[104450]: osdmap e385: 8 total, 8 up, 8 in 2024-09-13T14:26:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:14 smithi151 ceph-mon[104450]: osdmap e386: 8 total, 8 up, 8 in 2024-09-13T14:26:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:14 smithi151 ceph-mon[108966]: pgmap v194: 89 pgs: 89 active+clean; 206 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.3 MiB/s rd, 1.6 MiB/s wr, 14 op/s 2024-09-13T14:26:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:14 smithi151 ceph-mon[108966]: osdmap e385: 8 total, 8 up, 8 in 2024-09-13T14:26:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:14 smithi151 ceph-mon[108966]: osdmap e386: 8 total, 8 up, 8 in 2024-09-13T14:26:14.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:26:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:26:14] "GET /metrics HTTP/1.1" 200 36389 "" "Prometheus/2.51.0" 2024-09-13T14:26:14.871 INFO:tasks.rados.rados.0.smithi151.stdout:1302: expect (ObjNum 459 snap 150 seq_num 459) 2024-09-13T14:26:15.124 INFO:tasks.rados.rados.0.smithi151.stdout:1305: finishing write tid 1 to smithi15186934-1 2024-09-13T14:26:15.125 INFO:tasks.rados.rados.0.smithi151.stdout:1305: finishing write tid 2 to smithi15186934-1 2024-09-13T14:26:15.125 INFO:tasks.rados.rados.0.smithi151.stdout:1305: finishing write tid 3 to smithi15186934-1 2024-09-13T14:26:15.125 INFO:tasks.rados.rados.0.smithi151.stdout:1305: finishing write tid 4 to smithi15186934-1 2024-09-13T14:26:15.125 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 1 v 391 (ObjNum 511 snap 164 seq_num 511) dirty exists 2024-09-13T14:26:15.125 INFO:tasks.rados.rados.0.smithi151.stdout:1305: left oid 1 (ObjNum 511 snap 164 seq_num 511) 2024-09-13T14:26:15.473 INFO:tasks.rados.rados.0.smithi151.stdout:1295: done (10 left) 2024-09-13T14:26:15.474 INFO:tasks.rados.rados.0.smithi151.stdout:1296: done (9 left) 2024-09-13T14:26:15.474 INFO:tasks.rados.rados.0.smithi151.stdout:1297: done (8 left) 2024-09-13T14:26:15.474 INFO:tasks.rados.rados.0.smithi151.stdout:1298: done (7 left) 2024-09-13T14:26:15.474 INFO:tasks.rados.rados.0.smithi151.stdout:1299: done (6 left) 2024-09-13T14:26:15.474 INFO:tasks.rados.rados.0.smithi151.stdout:1300: done (5 left) 2024-09-13T14:26:15.474 INFO:tasks.rados.rados.0.smithi151.stdout:1301: done (4 left) 2024-09-13T14:26:15.474 INFO:tasks.rados.rados.0.smithi151.stdout:1302: done (3 left) 2024-09-13T14:26:15.474 INFO:tasks.rados.rados.0.smithi151.stdout:1303: done (2 left) 2024-09-13T14:26:15.474 INFO:tasks.rados.rados.0.smithi151.stdout:1305: done (1 left) 2024-09-13T14:26:15.474 INFO:tasks.rados.rados.0.smithi151.stdout:1306: done (0 left) 2024-09-13T14:26:15.475 INFO:tasks.rados.rados.0.smithi151.stdout:1307: write oid 25 current snap is 165 2024-09-13T14:26:15.475 INFO:tasks.rados.rados.0.smithi151.stdout:1307: seq_num 512 ranges {605184=723139,1877957=727227} 2024-09-13T14:26:15.480 INFO:tasks.rados.rados.0.smithi151.stdout:1307: writing smithi15186934-25 from 605184 to 1328323 tid 1 2024-09-13T14:26:15.488 INFO:tasks.rados.rados.0.smithi151.stdout:1307: writing smithi15186934-25 from 1877957 to 2605184 tid 2 2024-09-13T14:26:15.488 INFO:tasks.rados.rados.0.smithi151.stdout:1308: read oid 21 snap -1 2024-09-13T14:26:15.488 INFO:tasks.rados.rados.0.smithi151.stdout:1308: expect (ObjNum 503 snap 162 seq_num 503) 2024-09-13T14:26:15.488 INFO:tasks.rados.rados.0.smithi151.stdout:1309: rollback oid 5 current snap is 165 2024-09-13T14:26:15.488 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 5 to 149 2024-09-13T14:26:15.488 INFO:tasks.rados.rados.0.smithi151.stdout:1307: finishing write tid 1 to smithi15186934-25 2024-09-13T14:26:15.488 INFO:tasks.rados.rados.0.smithi151.stdout:1310: write oid 26 current snap is 165 2024-09-13T14:26:15.488 INFO:tasks.rados.rados.0.smithi151.stdout:1310: seq_num 513 ranges {621991=798388,1936439=639509} 2024-09-13T14:26:15.496 INFO:tasks.rados.rados.0.smithi151.stdout:1310: writing smithi15186934-26 from 621991 to 1420379 tid 1 2024-09-13T14:26:15.503 INFO:tasks.rados.rados.0.smithi151.stdout:1310: writing smithi15186934-26 from 1936439 to 2575948 tid 2 2024-09-13T14:26:15.503 INFO:tasks.rados.rados.0.smithi151.stdout:1311: write oid 42 current snap is 165 2024-09-13T14:26:15.503 INFO:tasks.rados.rados.0.smithi151.stdout:1309: finishing rollback tid 0 to smithi15186934-5 2024-09-13T14:26:15.503 INFO:tasks.rados.rados.0.smithi151.stdout:1309: finishing rollback tid 1 to smithi15186934-5 2024-09-13T14:26:15.503 INFO:tasks.rados.rados.0.smithi151.stdout:1309: finishing rollback tid 2 to smithi15186934-5 2024-09-13T14:26:15.503 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 475 (ObjNum 444 snap 141 seq_num 444) dirty exists 2024-09-13T14:26:15.503 INFO:tasks.rados.rados.0.smithi151.stdout:1307: finishing write tid 2 to smithi15186934-25 2024-09-13T14:26:15.503 INFO:tasks.rados.rados.0.smithi151.stdout:1311: seq_num 514 ranges {638798=473637,1594921=783167} 2024-09-13T14:26:15.508 INFO:tasks.rados.rados.0.smithi151.stdout:1311: writing smithi15186934-42 from 638798 to 1112435 tid 1 2024-09-13T14:26:15.516 INFO:tasks.rados.rados.0.smithi151.stdout:1311: writing smithi15186934-42 from 1594921 to 2378088 tid 2 2024-09-13T14:26:15.516 INFO:tasks.rados.rados.0.smithi151.stdout:1309: done (4 left) 2024-09-13T14:26:15.516 INFO:tasks.rados.rados.0.smithi151.stdout:1312: snap_create 2024-09-13T14:26:15.516 INFO:tasks.rados.rados.0.smithi151.stdout:1307: finishing write tid 3 to smithi15186934-25 2024-09-13T14:26:15.516 INFO:tasks.rados.rados.0.smithi151.stdout:1307: finishing write tid 4 to smithi15186934-25 2024-09-13T14:26:15.516 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 805 (ObjNum 512 snap 165 seq_num 512) dirty exists 2024-09-13T14:26:15.516 INFO:tasks.rados.rados.0.smithi151.stdout:1307: left oid 25 (ObjNum 512 snap 165 seq_num 512) 2024-09-13T14:26:15.516 INFO:tasks.rados.rados.0.smithi151.stdout:1308: expect (ObjNum 503 snap 162 seq_num 503) 2024-09-13T14:26:15.593 INFO:tasks.rados.rados.0.smithi151.stdout:1310: finishing write tid 1 to smithi15186934-26 2024-09-13T14:26:15.594 INFO:tasks.rados.rados.0.smithi151.stdout:1310: finishing write tid 2 to smithi15186934-26 2024-09-13T14:26:15.594 INFO:tasks.rados.rados.0.smithi151.stdout:1310: finishing write tid 3 to smithi15186934-26 2024-09-13T14:26:15.594 INFO:tasks.rados.rados.0.smithi151.stdout:1310: finishing write tid 4 to smithi15186934-26 2024-09-13T14:26:15.594 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 494 (ObjNum 513 snap 165 seq_num 513) dirty exists 2024-09-13T14:26:15.594 INFO:tasks.rados.rados.0.smithi151.stdout:1310: left oid 26 (ObjNum 513 snap 165 seq_num 513) 2024-09-13T14:26:15.594 INFO:tasks.rados.rados.0.smithi151.stdout:1311: finishing write tid 1 to smithi15186934-42 2024-09-13T14:26:15.594 INFO:tasks.rados.rados.0.smithi151.stdout:1311: finishing write tid 2 to smithi15186934-42 2024-09-13T14:26:15.594 INFO:tasks.rados.rados.0.smithi151.stdout:1311: finishing write tid 3 to smithi15186934-42 2024-09-13T14:26:15.594 INFO:tasks.rados.rados.0.smithi151.stdout:1311: finishing write tid 4 to smithi15186934-42 2024-09-13T14:26:15.594 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 658 (ObjNum 514 snap 165 seq_num 514) dirty exists 2024-09-13T14:26:15.594 INFO:tasks.rados.rados.0.smithi151.stdout:1311: left oid 42 (ObjNum 514 snap 165 seq_num 514) 2024-09-13T14:26:16.103 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:26:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:26:15.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:16.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:26:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:26:15.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:16.481 INFO:tasks.rados.rados.0.smithi151.stdout:1307: done (4 left) 2024-09-13T14:26:16.481 INFO:tasks.rados.rados.0.smithi151.stdout:1308: done (3 left) 2024-09-13T14:26:16.481 INFO:tasks.rados.rados.0.smithi151.stdout:1310: done (2 left) 2024-09-13T14:26:16.481 INFO:tasks.rados.rados.0.smithi151.stdout:1311: done (1 left) 2024-09-13T14:26:16.481 INFO:tasks.rados.rados.0.smithi151.stdout:1312: done (0 left) 2024-09-13T14:26:16.481 INFO:tasks.rados.rados.0.smithi151.stdout:1313: read oid 24 snap -1 2024-09-13T14:26:16.481 INFO:tasks.rados.rados.0.smithi151.stdout:1313: expect (ObjNum 501 snap 161 seq_num 501) 2024-09-13T14:26:16.481 INFO:tasks.rados.rados.0.smithi151.stdout:1314: delete oid 31 current snap is 166 2024-09-13T14:26:16.484 INFO:tasks.rados.rados.0.smithi151.stdout:1314: done (1 left) 2024-09-13T14:26:16.485 INFO:tasks.rados.rados.0.smithi151.stdout:1315: delete oid 19 current snap is 166 2024-09-13T14:26:16.487 INFO:tasks.rados.rados.0.smithi151.stdout:1315: done (1 left) 2024-09-13T14:26:16.487 INFO:tasks.rados.rados.0.smithi151.stdout:1316: snap_remove snap 149 2024-09-13T14:26:16.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:16 smithi184 ceph-mon[33394]: pgmap v197: 89 pgs: 89 active+clean; 206 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.3 MiB/s rd, 1.6 MiB/s wr, 14 op/s 2024-09-13T14:26:16.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:16 smithi184 ceph-mon[33394]: osdmap e387: 8 total, 8 up, 8 in 2024-09-13T14:26:16.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:16 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:26:16.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:16 smithi151 ceph-mon[108966]: pgmap v197: 89 pgs: 89 active+clean; 206 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.3 MiB/s rd, 1.6 MiB/s wr, 14 op/s 2024-09-13T14:26:16.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:16 smithi151 ceph-mon[108966]: osdmap e387: 8 total, 8 up, 8 in 2024-09-13T14:26:16.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:26:16.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:16 smithi151 ceph-mon[104450]: pgmap v197: 89 pgs: 89 active+clean; 206 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.3 MiB/s rd, 1.6 MiB/s wr, 14 op/s 2024-09-13T14:26:16.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:16 smithi151 ceph-mon[104450]: osdmap e387: 8 total, 8 up, 8 in 2024-09-13T14:26:16.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:26:17.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:26:16 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:26:16.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi184\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:17.494 INFO:tasks.rados.rados.0.smithi151.stdout:1316: done (1 left) 2024-09-13T14:26:17.494 INFO:tasks.rados.rados.0.smithi151.stdout:1317: snap_create 2024-09-13T14:26:17.494 INFO:tasks.rados.rados.0.smithi151.stdout:1313: expect (ObjNum 501 snap 161 seq_num 501) 2024-09-13T14:26:17.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:17 smithi184 ceph-mon[33394]: osdmap e388: 8 total, 8 up, 8 in 2024-09-13T14:26:17.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:17 smithi151 ceph-mon[104450]: osdmap e388: 8 total, 8 up, 8 in 2024-09-13T14:26:17.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:17 smithi151 ceph-mon[108966]: osdmap e388: 8 total, 8 up, 8 in 2024-09-13T14:26:18.492 INFO:tasks.rados.rados.0.smithi151.stdout:1313: done (1 left) 2024-09-13T14:26:18.492 INFO:tasks.rados.rados.0.smithi151.stdout:1317: done (0 left) 2024-09-13T14:26:18.492 INFO:tasks.rados.rados.0.smithi151.stdout:1318: read oid 32 snap -1 2024-09-13T14:26:18.493 INFO:tasks.rados.rados.0.smithi151.stdout:1318: expect deleted 2024-09-13T14:26:18.493 INFO:tasks.rados.rados.0.smithi151.stdout:1319: rollback oid 49 current snap is 167 2024-09-13T14:26:18.493 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 49 to 164 2024-09-13T14:26:18.493 INFO:tasks.rados.rados.0.smithi151.stdout:1320: delete oid 50 current snap is 167 2024-09-13T14:26:18.497 INFO:tasks.rados.rados.0.smithi151.stdout:1318: done (2 left) 2024-09-13T14:26:18.497 INFO:tasks.rados.rados.0.smithi151.stdout:1320: done (1 left) 2024-09-13T14:26:18.497 INFO:tasks.rados.rados.0.smithi151.stdout:1321: read oid 45 snap 157 2024-09-13T14:26:18.498 INFO:tasks.rados.rados.0.smithi151.stdout:1321: expect deleted 2024-09-13T14:26:18.498 INFO:tasks.rados.rados.0.smithi151.stdout:1322: delete oid 28 current snap is 167 2024-09-13T14:26:18.498 INFO:tasks.rados.rados.0.smithi151.stdout:1319: finishing rollback tid 0 to smithi15186934-49 2024-09-13T14:26:18.499 INFO:tasks.rados.rados.0.smithi151.stdout:1319: finishing rollback tid 1 to smithi15186934-49 2024-09-13T14:26:18.499 INFO:tasks.rados.rados.0.smithi151.stdout:1319: finishing rollback tid 2 to smithi15186934-49 2024-09-13T14:26:18.499 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 49 v 592 (ObjNum 462 snap 150 seq_num 462) dirty exists 2024-09-13T14:26:18.501 INFO:tasks.rados.rados.0.smithi151.stdout:1319: done (2 left) 2024-09-13T14:26:18.501 INFO:tasks.rados.rados.0.smithi151.stdout:1321: done (1 left) 2024-09-13T14:26:18.501 INFO:tasks.rados.rados.0.smithi151.stdout:1322: done (0 left) 2024-09-13T14:26:18.501 INFO:tasks.rados.rados.0.smithi151.stdout:1323: read oid 3 snap 133 2024-09-13T14:26:18.501 INFO:tasks.rados.rados.0.smithi151.stdout:1323: expect (ObjNum 334 snap 103 seq_num 334) 2024-09-13T14:26:18.501 INFO:tasks.rados.rados.0.smithi151.stdout:1324: rollback oid 19 current snap is 167 2024-09-13T14:26:18.501 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 19 to 158 2024-09-13T14:26:18.502 INFO:tasks.rados.rados.0.smithi151.stdout:1325: write oid 23 current snap is 167 2024-09-13T14:26:18.502 INFO:tasks.rados.rados.0.smithi151.stdout:1325: seq_num 519 ranges {722833=566235,1887331=650516} 2024-09-13T14:26:18.509 INFO:tasks.rados.rados.0.smithi151.stdout:1325: writing smithi15186934-23 from 722833 to 1289068 tid 1 2024-09-13T14:26:18.516 INFO:tasks.rados.rados.0.smithi151.stdout:1325: writing smithi15186934-23 from 1887331 to 2537847 tid 2 2024-09-13T14:26:18.516 INFO:tasks.rados.rados.0.smithi151.stdout:1326: snap_create 2024-09-13T14:26:18.516 INFO:tasks.rados.rados.0.smithi151.stdout:1324: finishing rollback tid 1 to smithi15186934-19 2024-09-13T14:26:18.516 INFO:tasks.rados.rados.0.smithi151.stdout:1324: finishing rollback tid 2 to smithi15186934-19 2024-09-13T14:26:18.516 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 19 v 332 (ObjNum 484 snap 156 seq_num 484) dirty exists 2024-09-13T14:26:18.516 INFO:tasks.rados.rados.0.smithi151.stdout:1323: expect (ObjNum 334 snap 103 seq_num 334) 2024-09-13T14:26:18.632 INFO:tasks.rados.rados.0.smithi151.stdout:1325: finishing write tid 1 to smithi15186934-23 2024-09-13T14:26:18.632 INFO:tasks.rados.rados.0.smithi151.stdout:1325: finishing write tid 2 to smithi15186934-23 2024-09-13T14:26:18.632 INFO:tasks.rados.rados.0.smithi151.stdout:1325: finishing write tid 3 to smithi15186934-23 2024-09-13T14:26:18.633 INFO:tasks.rados.rados.0.smithi151.stdout:1325: finishing write tid 4 to smithi15186934-23 2024-09-13T14:26:18.633 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 23 v 596 (ObjNum 519 snap 167 seq_num 519) dirty exists 2024-09-13T14:26:18.633 INFO:tasks.rados.rados.0.smithi151.stdout:1325: left oid 23 (ObjNum 519 snap 167 seq_num 519) 2024-09-13T14:26:18.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:18 smithi184 ceph-mon[33394]: pgmap v200: 89 pgs: 89 active+clean; 217 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 23 MiB/s rd, 1.9 MiB/s wr, 29 op/s 2024-09-13T14:26:18.980 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:18 smithi184 ceph-mon[33394]: osdmap e389: 8 total, 8 up, 8 in 2024-09-13T14:26:18.981 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:18 smithi151 ceph-mon[104450]: pgmap v200: 89 pgs: 89 active+clean; 217 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 23 MiB/s rd, 1.9 MiB/s wr, 29 op/s 2024-09-13T14:26:18.981 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:18 smithi151 ceph-mon[104450]: osdmap e389: 8 total, 8 up, 8 in 2024-09-13T14:26:18.981 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:18 smithi151 ceph-mon[108966]: pgmap v200: 89 pgs: 89 active+clean; 217 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 23 MiB/s rd, 1.9 MiB/s wr, 29 op/s 2024-09-13T14:26:18.981 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:18 smithi151 ceph-mon[108966]: osdmap e389: 8 total, 8 up, 8 in 2024-09-13T14:26:19.497 INFO:tasks.rados.rados.0.smithi151.stdout:1323: done (3 left) 2024-09-13T14:26:19.497 INFO:tasks.rados.rados.0.smithi151.stdout:1324: done (2 left) 2024-09-13T14:26:19.497 INFO:tasks.rados.rados.0.smithi151.stdout:1325: done (1 left) 2024-09-13T14:26:19.497 INFO:tasks.rados.rados.0.smithi151.stdout:1326: done (0 left) 2024-09-13T14:26:19.497 INFO:tasks.rados.rados.0.smithi151.stdout:1327: delete oid 41 current snap is 168 2024-09-13T14:26:19.501 INFO:tasks.rados.rados.0.smithi151.stdout:1327: done (0 left) 2024-09-13T14:26:19.501 INFO:tasks.rados.rados.0.smithi151.stdout:1328: delete oid 45 current snap is 168 2024-09-13T14:26:19.504 INFO:tasks.rados.rados.0.smithi151.stdout:1328: done (0 left) 2024-09-13T14:26:19.504 INFO:tasks.rados.rados.0.smithi151.stdout:1329: snap_create 2024-09-13T14:26:19.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:19 smithi184 ceph-mon[33394]: osdmap e390: 8 total, 8 up, 8 in 2024-09-13T14:26:19.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:19 smithi151 ceph-mon[104450]: osdmap e390: 8 total, 8 up, 8 in 2024-09-13T14:26:19.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:19 smithi151 ceph-mon[108966]: osdmap e390: 8 total, 8 up, 8 in 2024-09-13T14:26:20.499 INFO:tasks.rados.rados.0.smithi151.stdout:1329: done (0 left) 2024-09-13T14:26:20.499 INFO:tasks.rados.rados.0.smithi151.stdout:1330: write oid 34 current snap is 169 2024-09-13T14:26:20.499 INFO:tasks.rados.rados.0.smithi151.stdout:1330: seq_num 522 ranges {773254=791982,2346424=426830} 2024-09-13T14:26:20.501 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:26:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:26:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:20.508 INFO:tasks.rados.rados.0.smithi151.stdout:1330: writing smithi15186934-34 from 773254 to 1565236 tid 1 2024-09-13T14:26:20.516 INFO:tasks.rados.rados.0.smithi151.stdout:1330: writing smithi15186934-34 from 2346424 to 2773254 tid 2 2024-09-13T14:26:20.516 INFO:tasks.rados.rados.0.smithi151.stdout:1331: read oid 50 snap -1 2024-09-13T14:26:20.516 INFO:tasks.rados.rados.0.smithi151.stdout:1331: expect deleted 2024-09-13T14:26:20.516 INFO:tasks.rados.rados.0.smithi151.stdout:1332: read oid 19 snap -1 2024-09-13T14:26:20.516 INFO:tasks.rados.rados.0.smithi151.stdout:1332: expect (ObjNum 484 snap 156 seq_num 484) 2024-09-13T14:26:20.516 INFO:tasks.rados.rados.0.smithi151.stdout:1333: snap_remove snap 158 2024-09-13T14:26:20.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:20 smithi184 ceph-mon[33394]: pgmap v203: 89 pgs: 89 active+clean; 221 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 23 MiB/s rd, 2.5 MiB/s wr, 32 op/s 2024-09-13T14:26:20.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:20 smithi184 ceph-mon[33394]: osdmap e391: 8 total, 8 up, 8 in 2024-09-13T14:26:20.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:20 smithi151 ceph-mon[104450]: pgmap v203: 89 pgs: 89 active+clean; 221 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 23 MiB/s rd, 2.5 MiB/s wr, 32 op/s 2024-09-13T14:26:20.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:20 smithi151 ceph-mon[104450]: osdmap e391: 8 total, 8 up, 8 in 2024-09-13T14:26:20.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:20 smithi151 ceph-mon[108966]: pgmap v203: 89 pgs: 89 active+clean; 221 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 23 MiB/s rd, 2.5 MiB/s wr, 32 op/s 2024-09-13T14:26:20.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:20 smithi151 ceph-mon[108966]: osdmap e391: 8 total, 8 up, 8 in 2024-09-13T14:26:21.523 INFO:tasks.rados.rados.0.smithi151.stdout:1330: finishing write tid 1 to smithi15186934-34 2024-09-13T14:26:21.524 INFO:tasks.rados.rados.0.smithi151.stdout:1332: expect (ObjNum 484 snap 156 seq_num 484) 2024-09-13T14:26:21.613 INFO:tasks.rados.rados.0.smithi151.stdout:1330: finishing write tid 2 to smithi15186934-34 2024-09-13T14:26:21.613 INFO:tasks.rados.rados.0.smithi151.stdout:1330: finishing write tid 3 to smithi15186934-34 2024-09-13T14:26:21.613 INFO:tasks.rados.rados.0.smithi151.stdout:1330: finishing write tid 4 to smithi15186934-34 2024-09-13T14:26:21.613 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 526 (ObjNum 522 snap 169 seq_num 522) dirty exists 2024-09-13T14:26:21.614 INFO:tasks.rados.rados.0.smithi151.stdout:1330: left oid 34 (ObjNum 522 snap 169 seq_num 522) 2024-09-13T14:26:21.614 INFO:tasks.rados.rados.0.smithi151.stdout:1330: done (3 left) 2024-09-13T14:26:21.614 INFO:tasks.rados.rados.0.smithi151.stdout:1331: done (2 left) 2024-09-13T14:26:21.614 INFO:tasks.rados.rados.0.smithi151.stdout:1332: done (1 left) 2024-09-13T14:26:21.614 INFO:tasks.rados.rados.0.smithi151.stdout:1333: done (0 left) 2024-09-13T14:26:21.614 INFO:tasks.rados.rados.0.smithi151.stdout:1334: write oid 50 current snap is 169 2024-09-13T14:26:21.614 INFO:tasks.rados.rados.0.smithi151.stdout:1334: seq_num 523 ranges {790061=467231,2004906=541501} 2024-09-13T14:26:21.619 INFO:tasks.rados.rados.0.smithi151.stdout:1334: writing smithi15186934-50 from 790061 to 1257292 tid 1 2024-09-13T14:26:21.628 INFO:tasks.rados.rados.0.smithi151.stdout:1334: writing smithi15186934-50 from 2004906 to 2546407 tid 2 2024-09-13T14:26:21.628 INFO:tasks.rados.rados.0.smithi151.stdout:1335: read oid 22 snap -1 2024-09-13T14:26:21.629 INFO:tasks.rados.rados.0.smithi151.stdout:1335: expect (ObjNum 453 snap 148 seq_num 453) 2024-09-13T14:26:21.629 INFO:tasks.rados.rados.0.smithi151.stdout:1336: read oid 13 snap -1 2024-09-13T14:26:21.629 INFO:tasks.rados.rados.0.smithi151.stdout:1336: expect (ObjNum 270 snap 86 seq_num 270) 2024-09-13T14:26:21.629 INFO:tasks.rados.rados.0.smithi151.stdout:1337: read oid 20 snap -1 2024-09-13T14:26:21.629 INFO:tasks.rados.rados.0.smithi151.stdout:1337: expect (ObjNum 459 snap 150 seq_num 459) 2024-09-13T14:26:21.629 INFO:tasks.rados.rados.0.smithi151.stdout:1338: snap_create 2024-09-13T14:26:21.630 INFO:tasks.rados.rados.0.smithi151.stdout:1334: finishing write tid 1 to smithi15186934-50 2024-09-13T14:26:21.631 INFO:tasks.rados.rados.0.smithi151.stdout:1336: expect (ObjNum 270 snap 86 seq_num 270) 2024-09-13T14:26:21.770 INFO:tasks.rados.rados.0.smithi151.stdout:1334: finishing write tid 2 to smithi15186934-50 2024-09-13T14:26:21.770 INFO:tasks.rados.rados.0.smithi151.stdout:1334: finishing write tid 3 to smithi15186934-50 2024-09-13T14:26:21.770 INFO:tasks.rados.rados.0.smithi151.stdout:1334: finishing write tid 4 to smithi15186934-50 2024-09-13T14:26:21.770 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 50 v 421 (ObjNum 523 snap 169 seq_num 523) dirty exists 2024-09-13T14:26:21.771 INFO:tasks.rados.rados.0.smithi151.stdout:1334: left oid 50 (ObjNum 523 snap 169 seq_num 523) 2024-09-13T14:26:21.771 INFO:tasks.rados.rados.0.smithi151.stdout:1337: expect (ObjNum 459 snap 150 seq_num 459) 2024-09-13T14:26:21.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:21 smithi184 ceph-mon[33394]: osdmap e392: 8 total, 8 up, 8 in 2024-09-13T14:26:21.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:21 smithi151 ceph-mon[104450]: osdmap e392: 8 total, 8 up, 8 in 2024-09-13T14:26:21.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:21 smithi151 ceph-mon[108966]: osdmap e392: 8 total, 8 up, 8 in 2024-09-13T14:26:22.021 INFO:tasks.rados.rados.0.smithi151.stdout:1335: expect (ObjNum 453 snap 148 seq_num 453) 2024-09-13T14:26:22.521 INFO:tasks.rados.rados.0.smithi151.stdout:1334: done (4 left) 2024-09-13T14:26:22.522 INFO:tasks.rados.rados.0.smithi151.stdout:1335: done (3 left) 2024-09-13T14:26:22.522 INFO:tasks.rados.rados.0.smithi151.stdout:1336: done (2 left) 2024-09-13T14:26:22.522 INFO:tasks.rados.rados.0.smithi151.stdout:1337: done (1 left) 2024-09-13T14:26:22.522 INFO:tasks.rados.rados.0.smithi151.stdout:1338: done (0 left) 2024-09-13T14:26:22.522 INFO:tasks.rados.rados.0.smithi151.stdout:1339: write oid 19 current snap is 170 2024-09-13T14:26:22.522 INFO:tasks.rados.rados.0.smithi151.stdout:1339: seq_num 524 ranges {406868=542480,1547035=401512,2690354=116514} 2024-09-13T14:26:22.526 INFO:tasks.rados.rados.0.smithi151.stdout:1339: writing smithi15186934-19 from 406868 to 949348 tid 1 2024-09-13T14:26:22.532 INFO:tasks.rados.rados.0.smithi151.stdout:1339: writing smithi15186934-19 from 1547035 to 1948547 tid 2 2024-09-13T14:26:22.537 INFO:tasks.rados.rados.0.smithi151.stdout:1339: writing smithi15186934-19 from 2690354 to 2806868 tid 3 2024-09-13T14:26:22.538 INFO:tasks.rados.rados.0.smithi151.stdout:1340: rollback oid 15 current snap is 170 2024-09-13T14:26:22.538 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 15 to 156 2024-09-13T14:26:22.538 INFO:tasks.rados.rados.0.smithi151.stdout:1339: finishing write tid 1 to smithi15186934-19 2024-09-13T14:26:22.538 INFO:tasks.rados.rados.0.smithi151.stdout:1339: finishing write tid 2 to smithi15186934-19 2024-09-13T14:26:22.538 INFO:tasks.rados.rados.0.smithi151.stdout:1341: read oid 38 snap -1 2024-09-13T14:26:22.538 INFO:tasks.rados.rados.0.smithi151.stdout:1341: expect (ObjNum 504 snap 162 seq_num 504) 2024-09-13T14:26:22.538 INFO:tasks.rados.rados.0.smithi151.stdout:1342: read oid 2 snap -1 2024-09-13T14:26:22.538 INFO:tasks.rados.rados.0.smithi151.stdout:1342: expect (ObjNum 481 snap 156 seq_num 481) 2024-09-13T14:26:22.538 INFO:tasks.rados.rados.0.smithi151.stdout:1343: delete oid 5 current snap is 170 2024-09-13T14:26:22.539 INFO:tasks.rados.rados.0.smithi151.stdout:1339: finishing write tid 3 to smithi15186934-19 2024-09-13T14:26:22.540 INFO:tasks.rados.rados.0.smithi151.stdout:1339: finishing write tid 4 to smithi15186934-19 2024-09-13T14:26:22.540 INFO:tasks.rados.rados.0.smithi151.stdout:1342: expect (ObjNum 481 snap 156 seq_num 481) 2024-09-13T14:26:22.609 INFO:tasks.rados.rados.0.smithi151.stdout:1339: finishing write tid 5 to smithi15186934-19 2024-09-13T14:26:22.609 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 19 v 339 (ObjNum 524 snap 170 seq_num 524) dirty exists 2024-09-13T14:26:22.610 INFO:tasks.rados.rados.0.smithi151.stdout:1339: left oid 19 (ObjNum 524 snap 170 seq_num 524) 2024-09-13T14:26:22.610 INFO:tasks.rados.rados.0.smithi151.stdout:1339: done (4 left) 2024-09-13T14:26:22.610 INFO:tasks.rados.rados.0.smithi151.stdout:1342: done (3 left) 2024-09-13T14:26:22.610 INFO:tasks.rados.rados.0.smithi151.stdout:1343: done (2 left) 2024-09-13T14:26:22.610 INFO:tasks.rados.rados.0.smithi151.stdout:1344: read oid 50 snap 161 2024-09-13T14:26:22.610 INFO:tasks.rados.rados.0.smithi151.stdout:1344: expect (ObjNum 476 snap 156 seq_num 476) 2024-09-13T14:26:22.610 INFO:tasks.rados.rados.0.smithi151.stdout:1340: finishing rollback tid 0 to smithi15186934-15 2024-09-13T14:26:22.610 INFO:tasks.rados.rados.0.smithi151.stdout:1340: finishing rollback tid 1 to smithi15186934-15 2024-09-13T14:26:22.610 INFO:tasks.rados.rados.0.smithi151.stdout:1340: finishing rollback tid 2 to smithi15186934-15 2024-09-13T14:26:22.610 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 15 v 676 (ObjNum 413 snap 128 seq_num 413) dirty exists 2024-09-13T14:26:22.610 INFO:tasks.rados.rados.0.smithi151.stdout:1341: expect (ObjNum 504 snap 162 seq_num 504) 2024-09-13T14:26:22.722 INFO:tasks.rados.rados.0.smithi151.stdout:1344: expect (ObjNum 476 snap 156 seq_num 476) 2024-09-13T14:26:22.784 INFO:tasks.rados.rados.0.smithi151.stdout:1345: read oid 31 snap -1 2024-09-13T14:26:22.784 INFO:tasks.rados.rados.0.smithi151.stdout:1345: expect deleted 2024-09-13T14:26:22.784 INFO:tasks.rados.rados.0.smithi151.stdout:1340: done (3 left) 2024-09-13T14:26:22.785 INFO:tasks.rados.rados.0.smithi151.stdout:1341: done (2 left) 2024-09-13T14:26:22.785 INFO:tasks.rados.rados.0.smithi151.stdout:1344: done (1 left) 2024-09-13T14:26:22.785 INFO:tasks.rados.rados.0.smithi151.stdout:1346: rollback oid 15 current snap is 170 2024-09-13T14:26:22.785 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 15 to 167 2024-09-13T14:26:22.785 INFO:tasks.rados.rados.0.smithi151.stdout:1347: rollback oid 8 current snap is 170 2024-09-13T14:26:22.785 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 8 to 165 2024-09-13T14:26:22.785 INFO:tasks.rados.rados.0.smithi151.stdout:1348: snap_remove snap 130 2024-09-13T14:26:22.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:22 smithi184 ceph-mon[33394]: pgmap v206: 89 pgs: 89 active+clean; 221 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 652 KiB/s wr, 2 op/s 2024-09-13T14:26:22.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:22 smithi184 ceph-mon[33394]: osdmap e393: 8 total, 8 up, 8 in 2024-09-13T14:26:22.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:22 smithi151 ceph-mon[108966]: pgmap v206: 89 pgs: 89 active+clean; 221 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 652 KiB/s wr, 2 op/s 2024-09-13T14:26:22.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:22 smithi151 ceph-mon[108966]: osdmap e393: 8 total, 8 up, 8 in 2024-09-13T14:26:22.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:22 smithi151 ceph-mon[104450]: pgmap v206: 89 pgs: 89 active+clean; 221 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 652 KiB/s wr, 2 op/s 2024-09-13T14:26:22.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:22 smithi151 ceph-mon[104450]: osdmap e393: 8 total, 8 up, 8 in 2024-09-13T14:26:23.535 INFO:tasks.rados.rados.0.smithi151.stdout:1348: done (3 left) 2024-09-13T14:26:23.535 INFO:tasks.rados.rados.0.smithi151.stdout:1349: delete oid 12 current snap is 170 2024-09-13T14:26:23.535 INFO:tasks.rados.rados.0.smithi151.stdout:1347: finishing rollback tid 0 to smithi15186934-8 2024-09-13T14:26:23.535 INFO:tasks.rados.rados.0.smithi151.stdout:1346: finishing rollback tid 0 to smithi15186934-15 2024-09-13T14:26:23.535 INFO:tasks.rados.rados.0.smithi151.stdout:1347: finishing rollback tid 1 to smithi15186934-8 2024-09-13T14:26:23.535 INFO:tasks.rados.rados.0.smithi151.stdout:1346: finishing rollback tid 1 to smithi15186934-15 2024-09-13T14:26:23.535 INFO:tasks.rados.rados.0.smithi151.stdout:1346: finishing rollback tid 2 to smithi15186934-15 2024-09-13T14:26:23.535 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 15 v 679 (ObjNum 413 snap 128 seq_num 413) dirty exists 2024-09-13T14:26:23.536 INFO:tasks.rados.rados.0.smithi151.stdout:1347: finishing rollback tid 2 to smithi15186934-8 2024-09-13T14:26:23.536 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 403 (ObjNum 428 snap 135 seq_num 428) dirty exists 2024-09-13T14:26:23.539 INFO:tasks.rados.rados.0.smithi151.stdout:1345: done (3 left) 2024-09-13T14:26:23.539 INFO:tasks.rados.rados.0.smithi151.stdout:1346: done (2 left) 2024-09-13T14:26:23.539 INFO:tasks.rados.rados.0.smithi151.stdout:1347: done (1 left) 2024-09-13T14:26:23.540 INFO:tasks.rados.rados.0.smithi151.stdout:1349: done (0 left) 2024-09-13T14:26:23.540 INFO:tasks.rados.rados.0.smithi151.stdout:1350: snap_remove snap 168 2024-09-13T14:26:23.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:23 smithi184 ceph-mon[33394]: osdmap e394: 8 total, 8 up, 8 in 2024-09-13T14:26:23.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:23 smithi151 ceph-mon[104450]: osdmap e394: 8 total, 8 up, 8 in 2024-09-13T14:26:23.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:23 smithi151 ceph-mon[108966]: osdmap e394: 8 total, 8 up, 8 in 2024-09-13T14:26:24.478 INFO:tasks.rados.rados.0.smithi151.stdout:1350: done (0 left) 2024-09-13T14:26:24.479 INFO:tasks.rados.rados.0.smithi151.stdout:1351: read oid 19 snap 157 2024-09-13T14:26:24.479 INFO:tasks.rados.rados.0.smithi151.stdout:1351: expect (ObjNum 484 snap 156 seq_num 484) 2024-09-13T14:26:24.479 INFO:tasks.rados.rados.0.smithi151.stdout:1352: write oid 21 current snap is 170 2024-09-13T14:26:24.479 INFO:tasks.rados.rados.0.smithi151.stdout:1352: seq_num 527 ranges {457289=484580,1438834=548839,2772623=84666} 2024-09-13T14:26:24.483 INFO:tasks.rados.rados.0.smithi151.stdout:1352: writing smithi15186934-21 from 457289 to 941869 tid 1 2024-09-13T14:26:24.490 INFO:tasks.rados.rados.0.smithi151.stdout:1352: writing smithi15186934-21 from 1438834 to 1987673 tid 2 2024-09-13T14:26:24.495 INFO:tasks.rados.rados.0.smithi151.stdout:1352: writing smithi15186934-21 from 2772623 to 2857289 tid 3 2024-09-13T14:26:24.495 INFO:tasks.rados.rados.0.smithi151.stdout:1353: write oid 44 current snap is 170 2024-09-13T14:26:24.495 INFO:tasks.rados.rados.0.smithi151.stdout:1353: seq_num 528 ranges {474096=559829,1780963=408850,2800046=74050} 2024-09-13T14:26:24.500 INFO:tasks.rados.rados.0.smithi151.stdout:1353: writing smithi15186934-44 from 474096 to 1033925 tid 1 2024-09-13T14:26:24.507 INFO:tasks.rados.rados.0.smithi151.stdout:1353: writing smithi15186934-44 from 1780963 to 2189813 tid 2 2024-09-13T14:26:24.511 INFO:tasks.rados.rados.0.smithi151.stdout:1353: writing smithi15186934-44 from 2800046 to 2874096 tid 3 2024-09-13T14:26:24.511 INFO:tasks.rados.rados.0.smithi151.stdout:1354: read oid 35 snap -1 2024-09-13T14:26:24.512 INFO:tasks.rados.rados.0.smithi151.stdout:1354: expect (ObjNum 341 snap 105 seq_num 341) 2024-09-13T14:26:24.512 INFO:tasks.rados.rados.0.smithi151.stdout:1355: read oid 50 snap -1 2024-09-13T14:26:24.512 INFO:tasks.rados.rados.0.smithi151.stdout:1355: expect (ObjNum 523 snap 169 seq_num 523) 2024-09-13T14:26:24.512 INFO:tasks.rados.rados.0.smithi151.stdout:1356: write oid 26 current snap is 170 2024-09-13T14:26:24.512 INFO:tasks.rados.rados.0.smithi151.stdout:1356: seq_num 529 ranges {490903=635078,1839445=552508} 2024-09-13T14:26:24.517 INFO:tasks.rados.rados.0.smithi151.stdout:1356: writing smithi15186934-26 from 490903 to 1125981 tid 1 2024-09-13T14:26:24.524 INFO:tasks.rados.rados.0.smithi151.stdout:1356: writing smithi15186934-26 from 1839445 to 2391953 tid 2 2024-09-13T14:26:24.524 INFO:tasks.rados.rados.0.smithi151.stdout:1357: snap_remove snap 162 2024-09-13T14:26:24.530 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:26:24 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:26:24.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:24.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:24 smithi184 ceph-mon[33394]: pgmap v209: 89 pgs: 89 active+clean; 224 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 20 MiB/s rd, 841 KiB/s wr, 17 op/s 2024-09-13T14:26:24.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:24 smithi184 ceph-mon[33394]: osdmap e395: 8 total, 8 up, 8 in 2024-09-13T14:26:24.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:24 smithi184 ceph-mon[33394]: osdmap e396: 8 total, 8 up, 8 in 2024-09-13T14:26:24.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:24 smithi151 ceph-mon[104450]: pgmap v209: 89 pgs: 89 active+clean; 224 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 20 MiB/s rd, 841 KiB/s wr, 17 op/s 2024-09-13T14:26:24.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:24 smithi151 ceph-mon[104450]: osdmap e395: 8 total, 8 up, 8 in 2024-09-13T14:26:24.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:24 smithi151 ceph-mon[104450]: osdmap e396: 8 total, 8 up, 8 in 2024-09-13T14:26:24.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:24 smithi151 ceph-mon[108966]: pgmap v209: 89 pgs: 89 active+clean; 224 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 20 MiB/s rd, 841 KiB/s wr, 17 op/s 2024-09-13T14:26:24.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:24 smithi151 ceph-mon[108966]: osdmap e395: 8 total, 8 up, 8 in 2024-09-13T14:26:24.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:24 smithi151 ceph-mon[108966]: osdmap e396: 8 total, 8 up, 8 in 2024-09-13T14:26:24.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:26:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:26:24] "GET /metrics HTTP/1.1" 200 36406 "" "Prometheus/2.51.0" 2024-09-13T14:26:25.478 INFO:tasks.rados.rados.0.smithi151.stdout:1357: done (6 left) 2024-09-13T14:26:25.478 INFO:tasks.rados.rados.0.smithi151.stdout:1358: rollback oid 29 current snap is 170 2024-09-13T14:26:25.478 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 29 to 129 2024-09-13T14:26:25.478 INFO:tasks.rados.rados.0.smithi151.stdout:1359: read oid 3 snap -1 2024-09-13T14:26:25.478 INFO:tasks.rados.rados.0.smithi151.stdout:1359: expect (ObjNum 334 snap 103 seq_num 334) 2024-09-13T14:26:25.478 INFO:tasks.rados.rados.0.smithi151.stdout:1360: write oid 12 current snap is 170 2024-09-13T14:26:25.478 INFO:tasks.rados.rados.0.smithi151.stdout:1360: seq_num 530 ranges {507710=710327,1897927=412519,2854892=52818} 2024-09-13T14:26:25.485 INFO:tasks.rados.rados.0.smithi151.stdout:1360: writing smithi15186934-12 from 507710 to 1218037 tid 1 2024-09-13T14:26:25.492 INFO:tasks.rados.rados.0.smithi151.stdout:1360: writing smithi15186934-12 from 1897927 to 2310446 tid 2 2024-09-13T14:26:25.495 INFO:tasks.rados.rados.0.smithi151.stdout:1360: writing smithi15186934-12 from 2854892 to 2907710 tid 3 2024-09-13T14:26:25.495 INFO:tasks.rados.rados.0.smithi151.stdout:1361: write oid 31 current snap is 170 2024-09-13T14:26:25.495 INFO:tasks.rados.rados.0.smithi151.stdout:1351: expect (ObjNum 484 snap 156 seq_num 484) 2024-09-13T14:26:25.584 INFO:tasks.rados.rados.0.smithi151.stdout:1352: finishing write tid 1 to smithi15186934-21 2024-09-13T14:26:25.585 INFO:tasks.rados.rados.0.smithi151.stdout:1352: finishing write tid 2 to smithi15186934-21 2024-09-13T14:26:25.585 INFO:tasks.rados.rados.0.smithi151.stdout:1352: finishing write tid 3 to smithi15186934-21 2024-09-13T14:26:25.585 INFO:tasks.rados.rados.0.smithi151.stdout:1352: finishing write tid 4 to smithi15186934-21 2024-09-13T14:26:25.585 INFO:tasks.rados.rados.0.smithi151.stdout:1352: finishing write tid 5 to smithi15186934-21 2024-09-13T14:26:25.585 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 21 v 292 (ObjNum 527 snap 170 seq_num 527) dirty exists 2024-09-13T14:26:25.585 INFO:tasks.rados.rados.0.smithi151.stdout:1352: left oid 21 (ObjNum 527 snap 170 seq_num 527) 2024-09-13T14:26:25.585 INFO:tasks.rados.rados.0.smithi151.stdout:1353: finishing write tid 1 to smithi15186934-44 2024-09-13T14:26:25.585 INFO:tasks.rados.rados.0.smithi151.stdout:1353: finishing write tid 2 to smithi15186934-44 2024-09-13T14:26:25.585 INFO:tasks.rados.rados.0.smithi151.stdout:1353: finishing write tid 3 to smithi15186934-44 2024-09-13T14:26:25.585 INFO:tasks.rados.rados.0.smithi151.stdout:1353: finishing write tid 4 to smithi15186934-44 2024-09-13T14:26:25.585 INFO:tasks.rados.rados.0.smithi151.stdout:1353: finishing write tid 5 to smithi15186934-44 2024-09-13T14:26:25.586 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 414 (ObjNum 528 snap 170 seq_num 528) dirty exists 2024-09-13T14:26:25.586 INFO:tasks.rados.rados.0.smithi151.stdout:1353: left oid 44 (ObjNum 528 snap 170 seq_num 528) 2024-09-13T14:26:25.586 INFO:tasks.rados.rados.0.smithi151.stdout:1354: expect (ObjNum 341 snap 105 seq_num 341) 2024-09-13T14:26:25.702 INFO:tasks.rados.rados.0.smithi151.stdout:1356: finishing write tid 1 to smithi15186934-26 2024-09-13T14:26:25.702 INFO:tasks.rados.rados.0.smithi151.stdout:1355: expect (ObjNum 523 snap 169 seq_num 523) 2024-09-13T14:26:25.797 INFO:tasks.rados.rados.0.smithi151.stdout:1356: finishing write tid 2 to smithi15186934-26 2024-09-13T14:26:25.797 INFO:tasks.rados.rados.0.smithi151.stdout:1356: finishing write tid 3 to smithi15186934-26 2024-09-13T14:26:25.797 INFO:tasks.rados.rados.0.smithi151.stdout:1356: finishing write tid 4 to smithi15186934-26 2024-09-13T14:26:25.797 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 514 (ObjNum 529 snap 170 seq_num 529) dirty exists 2024-09-13T14:26:25.798 INFO:tasks.rados.rados.0.smithi151.stdout:1356: left oid 26 (ObjNum 529 snap 170 seq_num 529) 2024-09-13T14:26:25.798 INFO:tasks.rados.rados.0.smithi151.stdout:1359: expect (ObjNum 334 snap 103 seq_num 334) 2024-09-13T14:26:25.911 INFO:tasks.rados.rados.0.smithi151.stdout:1358: finishing rollback tid 1 to smithi15186934-29 2024-09-13T14:26:25.911 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 29 v 370 (ObjNum 129 snap 0 seq_num 3092956120) dirty dne 2024-09-13T14:26:25.911 INFO:tasks.rados.rados.0.smithi151.stdout:1360: finishing write tid 1 to smithi15186934-12 2024-09-13T14:26:25.911 INFO:tasks.rados.rados.0.smithi151.stdout:1361: seq_num 531 ranges {524517=785576,1956409=556177} 2024-09-13T14:26:25.918 INFO:tasks.rados.rados.0.smithi151.stdout:1361: writing smithi15186934-31 from 524517 to 1310093 tid 1 2024-09-13T14:26:25.925 INFO:tasks.rados.rados.0.smithi151.stdout:1361: writing smithi15186934-31 from 1956409 to 2512586 tid 2 2024-09-13T14:26:25.925 INFO:tasks.rados.rados.0.smithi151.stdout:1351: done (9 left) 2024-09-13T14:26:25.925 INFO:tasks.rados.rados.0.smithi151.stdout:1352: done (8 left) 2024-09-13T14:26:25.925 INFO:tasks.rados.rados.0.smithi151.stdout:1353: done (7 left) 2024-09-13T14:26:25.925 INFO:tasks.rados.rados.0.smithi151.stdout:1354: done (6 left) 2024-09-13T14:26:25.925 INFO:tasks.rados.rados.0.smithi151.stdout:1355: done (5 left) 2024-09-13T14:26:25.925 INFO:tasks.rados.rados.0.smithi151.stdout:1356: done (4 left) 2024-09-13T14:26:25.925 INFO:tasks.rados.rados.0.smithi151.stdout:1358: done (3 left) 2024-09-13T14:26:25.926 INFO:tasks.rados.rados.0.smithi151.stdout:1359: done (2 left) 2024-09-13T14:26:25.926 INFO:tasks.rados.rados.0.smithi151.stdout:1362: delete oid 10 current snap is 170 2024-09-13T14:26:25.926 INFO:tasks.rados.rados.0.smithi151.stdout:1360: finishing write tid 2 to smithi15186934-12 2024-09-13T14:26:25.926 INFO:tasks.rados.rados.0.smithi151.stdout:1360: finishing write tid 3 to smithi15186934-12 2024-09-13T14:26:25.926 INFO:tasks.rados.rados.0.smithi151.stdout:1360: finishing write tid 4 to smithi15186934-12 2024-09-13T14:26:25.926 INFO:tasks.rados.rados.0.smithi151.stdout:1360: finishing write tid 5 to smithi15186934-12 2024-09-13T14:26:25.926 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 12 v 312 (ObjNum 530 snap 170 seq_num 530) dirty exists 2024-09-13T14:26:25.926 INFO:tasks.rados.rados.0.smithi151.stdout:1360: left oid 12 (ObjNum 530 snap 170 seq_num 530) 2024-09-13T14:26:25.926 INFO:tasks.rados.rados.0.smithi151.stdout:1361: finishing write tid 1 to smithi15186934-31 2024-09-13T14:26:25.927 INFO:tasks.rados.rados.0.smithi151.stdout:1360: done (2 left) 2024-09-13T14:26:25.927 INFO:tasks.rados.rados.0.smithi151.stdout:1362: done (1 left) 2024-09-13T14:26:25.927 INFO:tasks.rados.rados.0.smithi151.stdout:1363: read oid 12 snap -1 2024-09-13T14:26:25.927 INFO:tasks.rados.rados.0.smithi151.stdout:1363: expect (ObjNum 530 snap 170 seq_num 530) 2024-09-13T14:26:25.927 INFO:tasks.rados.rados.0.smithi151.stdout:1364: read oid 28 snap 164 2024-09-13T14:26:25.927 INFO:tasks.rados.rados.0.smithi151.stdout:1364: expect (ObjNum 414 snap 129 seq_num 414) 2024-09-13T14:26:25.927 INFO:tasks.rados.rados.0.smithi151.stdout:1365: snap_create 2024-09-13T14:26:25.932 INFO:tasks.rados.rados.0.smithi151.stdout:1361: finishing write tid 2 to smithi15186934-31 2024-09-13T14:26:25.932 INFO:tasks.rados.rados.0.smithi151.stdout:1361: finishing write tid 3 to smithi15186934-31 2024-09-13T14:26:25.932 INFO:tasks.rados.rados.0.smithi151.stdout:1361: finishing write tid 4 to smithi15186934-31 2024-09-13T14:26:25.932 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 31 v 700 (ObjNum 531 snap 170 seq_num 531) dirty exists 2024-09-13T14:26:25.933 INFO:tasks.rados.rados.0.smithi151.stdout:1361: left oid 31 (ObjNum 531 snap 170 seq_num 531) 2024-09-13T14:26:25.934 INFO:tasks.rados.rados.0.smithi151.stdout:1364: expect (ObjNum 414 snap 129 seq_num 414) 2024-09-13T14:26:26.103 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:26:25 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:26:25.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:26.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:26:25 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:26:25.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:26.109 INFO:tasks.rados.rados.0.smithi151.stdout:1363: expect (ObjNum 530 snap 170 seq_num 530) 2024-09-13T14:26:26.540 INFO:tasks.rados.rados.0.smithi151.stdout:1361: done (3 left) 2024-09-13T14:26:26.540 INFO:tasks.rados.rados.0.smithi151.stdout:1363: done (2 left) 2024-09-13T14:26:26.540 INFO:tasks.rados.rados.0.smithi151.stdout:1364: done (1 left) 2024-09-13T14:26:26.541 INFO:tasks.rados.rados.0.smithi151.stdout:1365: done (0 left) 2024-09-13T14:26:26.541 INFO:tasks.rados.rados.0.smithi151.stdout:1366: read oid 17 snap -1 2024-09-13T14:26:26.541 INFO:tasks.rados.rados.0.smithi151.stdout:1366: expect (ObjNum 505 snap 162 seq_num 505) 2024-09-13T14:26:26.541 INFO:tasks.rados.rados.0.smithi151.stdout:1367: write oid 41 current snap is 171 2024-09-13T14:26:26.541 INFO:tasks.rados.rados.0.smithi151.stdout:1367: seq_num 533 ranges {558131=652427,1673373=559846,2820808=137323} 2024-09-13T14:26:26.546 INFO:tasks.rados.rados.0.smithi151.stdout:1367: writing smithi15186934-41 from 558131 to 1210558 tid 1 2024-09-13T14:26:26.552 INFO:tasks.rados.rados.0.smithi151.stdout:1367: writing smithi15186934-41 from 1673373 to 2233219 tid 2 2024-09-13T14:26:26.556 INFO:tasks.rados.rados.0.smithi151.stdout:1367: writing smithi15186934-41 from 2820808 to 2958131 tid 3 2024-09-13T14:26:26.556 INFO:tasks.rados.rados.0.smithi151.stdout:1368: snap_create 2024-09-13T14:26:26.556 INFO:tasks.rados.rados.0.smithi151.stdout:1366: expect (ObjNum 505 snap 162 seq_num 505) 2024-09-13T14:26:26.740 INFO:tasks.rados.rados.0.smithi151.stdout:1367: finishing write tid 1 to smithi15186934-41 2024-09-13T14:26:26.740 INFO:tasks.rados.rados.0.smithi151.stdout:1367: finishing write tid 2 to smithi15186934-41 2024-09-13T14:26:26.740 INFO:tasks.rados.rados.0.smithi151.stdout:1367: finishing write tid 3 to smithi15186934-41 2024-09-13T14:26:26.741 INFO:tasks.rados.rados.0.smithi151.stdout:1367: finishing write tid 4 to smithi15186934-41 2024-09-13T14:26:26.741 INFO:tasks.rados.rados.0.smithi151.stdout:1367: finishing write tid 5 to smithi15186934-41 2024-09-13T14:26:26.741 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 51 (ObjNum 533 snap 171 seq_num 533) dirty exists 2024-09-13T14:26:26.741 INFO:tasks.rados.rados.0.smithi151.stdout:1367: left oid 41 (ObjNum 533 snap 171 seq_num 533) 2024-09-13T14:26:26.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:26 smithi184 ceph-mon[33394]: pgmap v212: 89 pgs: 89 active+clean; 224 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 20 MiB/s rd, 841 KiB/s wr, 17 op/s 2024-09-13T14:26:26.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:26 smithi184 ceph-mon[33394]: osdmap e397: 8 total, 8 up, 8 in 2024-09-13T14:26:26.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:26 smithi151 ceph-mon[108966]: pgmap v212: 89 pgs: 89 active+clean; 224 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 20 MiB/s rd, 841 KiB/s wr, 17 op/s 2024-09-13T14:26:26.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:26 smithi151 ceph-mon[108966]: osdmap e397: 8 total, 8 up, 8 in 2024-09-13T14:26:26.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:26 smithi151 ceph-mon[104450]: pgmap v212: 89 pgs: 89 active+clean; 224 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 20 MiB/s rd, 841 KiB/s wr, 17 op/s 2024-09-13T14:26:26.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:26 smithi151 ceph-mon[104450]: osdmap e397: 8 total, 8 up, 8 in 2024-09-13T14:26:27.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:26:26 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:26:26.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi184\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:27.549 INFO:tasks.rados.rados.0.smithi151.stdout:1366: done (2 left) 2024-09-13T14:26:27.549 INFO:tasks.rados.rados.0.smithi151.stdout:1367: done (1 left) 2024-09-13T14:26:27.549 INFO:tasks.rados.rados.0.smithi151.stdout:1368: done (0 left) 2024-09-13T14:26:27.550 INFO:tasks.rados.rados.0.smithi151.stdout:1369: write oid 10 current snap is 172 2024-09-13T14:26:27.550 INFO:tasks.rados.rados.0.smithi151.stdout:1369: seq_num 534 ranges {574938=727676,1731855=703504,2848231=126707} 2024-09-13T14:26:27.556 INFO:tasks.rados.rados.0.smithi151.stdout:1369: writing smithi15186934-10 from 574938 to 1302614 tid 1 2024-09-13T14:26:27.562 INFO:tasks.rados.rados.0.smithi151.stdout:1369: writing smithi15186934-10 from 1731855 to 2435359 tid 2 2024-09-13T14:26:27.565 INFO:tasks.rados.rados.0.smithi151.stdout:1369: writing smithi15186934-10 from 2848231 to 2974938 tid 3 2024-09-13T14:26:27.566 INFO:tasks.rados.rados.0.smithi151.stdout:1370: delete oid 29 current snap is 172 2024-09-13T14:26:27.566 INFO:tasks.rados.rados.0.smithi151.stdout:1369: finishing write tid 1 to smithi15186934-10 2024-09-13T14:26:27.566 INFO:tasks.rados.rados.0.smithi151.stdout:1369: finishing write tid 2 to smithi15186934-10 2024-09-13T14:26:27.568 INFO:tasks.rados.rados.0.smithi151.stdout:1370: done (1 left) 2024-09-13T14:26:27.568 INFO:tasks.rados.rados.0.smithi151.stdout:1371: read oid 32 snap -1 2024-09-13T14:26:27.568 INFO:tasks.rados.rados.0.smithi151.stdout:1371: expect deleted 2024-09-13T14:26:27.568 INFO:tasks.rados.rados.0.smithi151.stdout:1372: snap_create 2024-09-13T14:26:27.569 INFO:tasks.rados.rados.0.smithi151.stdout:1369: finishing write tid 3 to smithi15186934-10 2024-09-13T14:26:27.569 INFO:tasks.rados.rados.0.smithi151.stdout:1369: finishing write tid 4 to smithi15186934-10 2024-09-13T14:26:27.569 INFO:tasks.rados.rados.0.smithi151.stdout:1369: finishing write tid 5 to smithi15186934-10 2024-09-13T14:26:27.569 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 524 (ObjNum 534 snap 172 seq_num 534) dirty exists 2024-09-13T14:26:27.569 INFO:tasks.rados.rados.0.smithi151.stdout:1369: left oid 10 (ObjNum 534 snap 172 seq_num 534) 2024-09-13T14:26:27.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:27 smithi184 ceph-mon[33394]: osdmap e398: 8 total, 8 up, 8 in 2024-09-13T14:26:27.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:27 smithi151 ceph-mon[104450]: osdmap e398: 8 total, 8 up, 8 in 2024-09-13T14:26:27.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:27 smithi151 ceph-mon[108966]: osdmap e398: 8 total, 8 up, 8 in 2024-09-13T14:26:28.547 INFO:tasks.rados.rados.0.smithi151.stdout:1369: done (2 left) 2024-09-13T14:26:28.548 INFO:tasks.rados.rados.0.smithi151.stdout:1371: done (1 left) 2024-09-13T14:26:28.548 INFO:tasks.rados.rados.0.smithi151.stdout:1372: done (0 left) 2024-09-13T14:26:28.548 INFO:tasks.rados.rados.0.smithi151.stdout:1373: read oid 41 snap -1 2024-09-13T14:26:28.548 INFO:tasks.rados.rados.0.smithi151.stdout:1373: expect (ObjNum 533 snap 171 seq_num 533) 2024-09-13T14:26:28.548 INFO:tasks.rados.rados.0.smithi151.stdout:1374: delete oid 20 current snap is 173 2024-09-13T14:26:28.554 INFO:tasks.rados.rados.0.smithi151.stdout:1374: done (1 left) 2024-09-13T14:26:28.554 INFO:tasks.rados.rados.0.smithi151.stdout:1375: write oid 17 current snap is 173 2024-09-13T14:26:28.554 INFO:tasks.rados.rados.0.smithi151.stdout:1375: seq_num 537 ranges {625359=553423,1790948=567184,2930500=94859} 2024-09-13T14:26:28.561 INFO:tasks.rados.rados.0.smithi151.stdout:1375: writing smithi15186934-17 from 625359 to 1178782 tid 1 2024-09-13T14:26:28.567 INFO:tasks.rados.rados.0.smithi151.stdout:1375: writing smithi15186934-17 from 1790948 to 2358132 tid 2 2024-09-13T14:26:28.571 INFO:tasks.rados.rados.0.smithi151.stdout:1375: writing smithi15186934-17 from 2930500 to 3025359 tid 3 2024-09-13T14:26:28.572 INFO:tasks.rados.rados.0.smithi151.stdout:1376: write oid 8 current snap is 173 2024-09-13T14:26:28.572 INFO:tasks.rados.rados.0.smithi151.stdout:1376: seq_num 538 ranges {642166=628672,1849430=710842} 2024-09-13T14:26:28.578 INFO:tasks.rados.rados.0.smithi151.stdout:1376: writing smithi15186934-8 from 642166 to 1270838 tid 1 2024-09-13T14:26:28.586 INFO:tasks.rados.rados.0.smithi151.stdout:1376: writing smithi15186934-8 from 1849430 to 2560272 tid 2 2024-09-13T14:26:28.586 INFO:tasks.rados.rados.0.smithi151.stdout:1377: rollback oid 40 current snap is 173 2024-09-13T14:26:28.586 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 40 to 166 2024-09-13T14:26:28.586 INFO:tasks.rados.rados.0.smithi151.stdout:1373: expect (ObjNum 533 snap 171 seq_num 533) 2024-09-13T14:26:28.684 INFO:tasks.rados.rados.0.smithi151.stdout:1375: finishing write tid 1 to smithi15186934-17 2024-09-13T14:26:28.684 INFO:tasks.rados.rados.0.smithi151.stdout:1375: finishing write tid 2 to smithi15186934-17 2024-09-13T14:26:28.684 INFO:tasks.rados.rados.0.smithi151.stdout:1375: finishing write tid 3 to smithi15186934-17 2024-09-13T14:26:28.684 INFO:tasks.rados.rados.0.smithi151.stdout:1375: finishing write tid 4 to smithi15186934-17 2024-09-13T14:26:28.685 INFO:tasks.rados.rados.0.smithi151.stdout:1375: finishing write tid 5 to smithi15186934-17 2024-09-13T14:26:28.685 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 881 (ObjNum 537 snap 173 seq_num 537) dirty exists 2024-09-13T14:26:28.685 INFO:tasks.rados.rados.0.smithi151.stdout:1375: left oid 17 (ObjNum 537 snap 173 seq_num 537) 2024-09-13T14:26:28.685 INFO:tasks.rados.rados.0.smithi151.stdout:1376: finishing write tid 1 to smithi15186934-8 2024-09-13T14:26:28.685 INFO:tasks.rados.rados.0.smithi151.stdout:1377: finishing rollback tid 1 to smithi15186934-40 2024-09-13T14:26:28.685 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 40 v 0 (ObjNum 33 snap 0 seq_num 3064833400) dirty dne 2024-09-13T14:26:28.685 INFO:tasks.rados.rados.0.smithi151.stdout:1376: finishing write tid 2 to smithi15186934-8 2024-09-13T14:26:28.685 INFO:tasks.rados.rados.0.smithi151.stdout:1373: done (3 left) 2024-09-13T14:26:28.685 INFO:tasks.rados.rados.0.smithi151.stdout:1375: done (2 left) 2024-09-13T14:26:28.685 INFO:tasks.rados.rados.0.smithi151.stdout:1377: done (1 left) 2024-09-13T14:26:28.685 INFO:tasks.rados.rados.0.smithi151.stdout:1378: write oid 34 current snap is 173 2024-09-13T14:26:28.685 INFO:tasks.rados.rados.0.smithi151.stdout:1376: finishing write tid 3 to smithi15186934-8 2024-09-13T14:26:28.685 INFO:tasks.rados.rados.0.smithi151.stdout:1376: finishing write tid 4 to smithi15186934-8 2024-09-13T14:26:28.686 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 424 (ObjNum 538 snap 173 seq_num 538) dirty exists 2024-09-13T14:26:28.686 INFO:tasks.rados.rados.0.smithi151.stdout:1376: left oid 8 (ObjNum 538 snap 173 seq_num 538) 2024-09-13T14:26:28.686 INFO:tasks.rados.rados.0.smithi151.stdout:1378: seq_num 539 ranges {658973=703921,1907912=570853,2985346=73627} 2024-09-13T14:26:28.691 INFO:tasks.rados.rados.0.smithi151.stdout:1378: writing smithi15186934-34 from 658973 to 1362894 tid 1 2024-09-13T14:26:28.697 INFO:tasks.rados.rados.0.smithi151.stdout:1378: writing smithi15186934-34 from 1907912 to 2478765 tid 2 2024-09-13T14:26:28.700 INFO:tasks.rados.rados.0.smithi151.stdout:1378: writing smithi15186934-34 from 2985346 to 3058973 tid 3 2024-09-13T14:26:28.700 INFO:tasks.rados.rados.0.smithi151.stdout:1376: done (1 left) 2024-09-13T14:26:28.701 INFO:tasks.rados.rados.0.smithi151.stdout:1379: snap_create 2024-09-13T14:26:28.701 INFO:tasks.rados.rados.0.smithi151.stdout:1378: finishing write tid 1 to smithi15186934-34 2024-09-13T14:26:28.701 INFO:tasks.rados.rados.0.smithi151.stdout:1378: finishing write tid 2 to smithi15186934-34 2024-09-13T14:26:28.702 INFO:tasks.rados.rados.0.smithi151.stdout:1378: finishing write tid 3 to smithi15186934-34 2024-09-13T14:26:28.702 INFO:tasks.rados.rados.0.smithi151.stdout:1378: finishing write tid 4 to smithi15186934-34 2024-09-13T14:26:28.703 INFO:tasks.rados.rados.0.smithi151.stdout:1378: finishing write tid 5 to smithi15186934-34 2024-09-13T14:26:28.703 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 537 (ObjNum 539 snap 173 seq_num 539) dirty exists 2024-09-13T14:26:28.703 INFO:tasks.rados.rados.0.smithi151.stdout:1378: left oid 34 (ObjNum 539 snap 173 seq_num 539) 2024-09-13T14:26:28.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:28 smithi184 ceph-mon[33394]: pgmap v215: 89 pgs: 89 active+clean; 238 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 22 MiB/s rd, 2.0 MiB/s wr, 32 op/s 2024-09-13T14:26:28.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:28 smithi184 ceph-mon[33394]: osdmap e399: 8 total, 8 up, 8 in 2024-09-13T14:26:28.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:28 smithi151 ceph-mon[104450]: pgmap v215: 89 pgs: 89 active+clean; 238 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 22 MiB/s rd, 2.0 MiB/s wr, 32 op/s 2024-09-13T14:26:28.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:28 smithi151 ceph-mon[104450]: osdmap e399: 8 total, 8 up, 8 in 2024-09-13T14:26:28.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:28 smithi151 ceph-mon[108966]: pgmap v215: 89 pgs: 89 active+clean; 238 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 22 MiB/s rd, 2.0 MiB/s wr, 32 op/s 2024-09-13T14:26:28.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:28 smithi151 ceph-mon[108966]: osdmap e399: 8 total, 8 up, 8 in 2024-09-13T14:26:29.474 INFO:tasks.rados.rados.0.smithi151.stdout:1378: done (1 left) 2024-09-13T14:26:29.474 INFO:tasks.rados.rados.0.smithi151.stdout:1379: done (0 left) 2024-09-13T14:26:29.474 INFO:tasks.rados.rados.0.smithi151.stdout:1380: delete oid 36 current snap is 174 2024-09-13T14:26:29.476 INFO:tasks.rados.rados.0.smithi151.stdout:1380: done (0 left) 2024-09-13T14:26:29.476 INFO:tasks.rados.rados.0.smithi151.stdout:1381: read oid 1 snap 167 2024-09-13T14:26:29.476 INFO:tasks.rados.rados.0.smithi151.stdout:1381: expect (ObjNum 511 snap 164 seq_num 511) 2024-09-13T14:26:29.476 INFO:tasks.rados.rados.0.smithi151.stdout:1382: write oid 35 current snap is 174 2024-09-13T14:26:29.476 INFO:tasks.rados.rados.0.smithi151.stdout:1382: seq_num 541 ranges {692587=570772,2024876=574522,3040192=52395} 2024-09-13T14:26:29.483 INFO:tasks.rados.rados.0.smithi151.stdout:1382: writing smithi15186934-35 from 692587 to 1263359 tid 1 2024-09-13T14:26:29.490 INFO:tasks.rados.rados.0.smithi151.stdout:1382: writing smithi15186934-35 from 2024876 to 2599398 tid 2 2024-09-13T14:26:29.493 INFO:tasks.rados.rados.0.smithi151.stdout:1382: writing smithi15186934-35 from 3040192 to 3092587 tid 3 2024-09-13T14:26:29.493 INFO:tasks.rados.rados.0.smithi151.stdout:1383: read oid 45 snap -1 2024-09-13T14:26:29.493 INFO:tasks.rados.rados.0.smithi151.stdout:1383: expect deleted 2024-09-13T14:26:29.493 INFO:tasks.rados.rados.0.smithi151.stdout:1384: write oid 41 current snap is 174 2024-09-13T14:26:29.494 INFO:tasks.rados.rados.0.smithi151.stdout:1384: seq_num 542 ranges {709394=646021,2083358=718180} 2024-09-13T14:26:29.500 INFO:tasks.rados.rados.0.smithi151.stdout:1384: writing smithi15186934-41 from 709394 to 1355415 tid 1 2024-09-13T14:26:29.508 INFO:tasks.rados.rados.0.smithi151.stdout:1384: writing smithi15186934-41 from 2083358 to 2801538 tid 2 2024-09-13T14:26:29.509 INFO:tasks.rados.rados.0.smithi151.stdout:1385: delete oid 24 current snap is 174 2024-09-13T14:26:29.509 INFO:tasks.rados.rados.0.smithi151.stdout:1381: expect (ObjNum 511 snap 164 seq_num 511) 2024-09-13T14:26:29.694 INFO:tasks.rados.rados.0.smithi151.stdout:1382: finishing write tid 1 to smithi15186934-35 2024-09-13T14:26:29.694 INFO:tasks.rados.rados.0.smithi151.stdout:1382: finishing write tid 2 to smithi15186934-35 2024-09-13T14:26:29.694 INFO:tasks.rados.rados.0.smithi151.stdout:1381: done (4 left) 2024-09-13T14:26:29.694 INFO:tasks.rados.rados.0.smithi151.stdout:1385: done (3 left) 2024-09-13T14:26:29.694 INFO:tasks.rados.rados.0.smithi151.stdout:1386: snap_remove snap 169 2024-09-13T14:26:29.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:29 smithi184 ceph-mon[33394]: osdmap e400: 8 total, 8 up, 8 in 2024-09-13T14:26:29.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:29 smithi184 ceph-mon[33394]: osdmap e401: 8 total, 8 up, 8 in 2024-09-13T14:26:29.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:29 smithi151 ceph-mon[104450]: osdmap e400: 8 total, 8 up, 8 in 2024-09-13T14:26:29.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:29 smithi151 ceph-mon[104450]: osdmap e401: 8 total, 8 up, 8 in 2024-09-13T14:26:29.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:29 smithi151 ceph-mon[108966]: osdmap e400: 8 total, 8 up, 8 in 2024-09-13T14:26:29.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:29 smithi151 ceph-mon[108966]: osdmap e401: 8 total, 8 up, 8 in 2024-09-13T14:26:30.555 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:26:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:26:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:30.564 INFO:tasks.rados.rados.0.smithi151.stdout:1386: done (3 left) 2024-09-13T14:26:30.565 INFO:tasks.rados.rados.0.smithi151.stdout:1387: snap_create 2024-09-13T14:26:30.565 INFO:tasks.rados.rados.0.smithi151.stdout:1382: finishing write tid 3 to smithi15186934-35 2024-09-13T14:26:30.565 INFO:tasks.rados.rados.0.smithi151.stdout:1382: finishing write tid 4 to smithi15186934-35 2024-09-13T14:26:30.565 INFO:tasks.rados.rados.0.smithi151.stdout:1382: finishing write tid 5 to smithi15186934-35 2024-09-13T14:26:30.565 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 35 v 886 (ObjNum 541 snap 174 seq_num 541) dirty exists 2024-09-13T14:26:30.565 INFO:tasks.rados.rados.0.smithi151.stdout:1382: left oid 35 (ObjNum 541 snap 174 seq_num 541) 2024-09-13T14:26:30.565 INFO:tasks.rados.rados.0.smithi151.stdout:1384: finishing write tid 1 to smithi15186934-41 2024-09-13T14:26:30.565 INFO:tasks.rados.rados.0.smithi151.stdout:1384: finishing write tid 2 to smithi15186934-41 2024-09-13T14:26:30.565 INFO:tasks.rados.rados.0.smithi151.stdout:1384: finishing write tid 3 to smithi15186934-41 2024-09-13T14:26:30.565 INFO:tasks.rados.rados.0.smithi151.stdout:1384: finishing write tid 4 to smithi15186934-41 2024-09-13T14:26:30.565 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 55 (ObjNum 542 snap 174 seq_num 542) dirty exists 2024-09-13T14:26:30.565 INFO:tasks.rados.rados.0.smithi151.stdout:1384: left oid 41 (ObjNum 542 snap 174 seq_num 542) 2024-09-13T14:26:30.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:30 smithi184 ceph-mon[33394]: pgmap v218: 89 pgs: 89 active+clean; 238 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 22 MiB/s rd, 2.0 MiB/s wr, 32 op/s 2024-09-13T14:26:30.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:30 smithi151 ceph-mon[104450]: pgmap v218: 89 pgs: 89 active+clean; 238 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 22 MiB/s rd, 2.0 MiB/s wr, 32 op/s 2024-09-13T14:26:30.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:30 smithi151 ceph-mon[108966]: pgmap v218: 89 pgs: 89 active+clean; 238 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 22 MiB/s rd, 2.0 MiB/s wr, 32 op/s 2024-09-13T14:26:31.574 INFO:tasks.rados.rados.0.smithi151.stdout:1382: done (3 left) 2024-09-13T14:26:31.574 INFO:tasks.rados.rados.0.smithi151.stdout:1383: done (2 left) 2024-09-13T14:26:31.574 INFO:tasks.rados.rados.0.smithi151.stdout:1384: done (1 left) 2024-09-13T14:26:31.574 INFO:tasks.rados.rados.0.smithi151.stdout:1387: done (0 left) 2024-09-13T14:26:31.574 INFO:tasks.rados.rados.0.smithi151.stdout:1388: delete oid 44 current snap is 175 2024-09-13T14:26:31.577 INFO:tasks.rados.rados.0.smithi151.stdout:1388: done (0 left) 2024-09-13T14:26:31.577 INFO:tasks.rados.rados.0.smithi151.stdout:1389: write oid 12 current snap is 175 2024-09-13T14:26:31.577 INFO:tasks.rados.rados.0.smithi151.stdout:1389: seq_num 545 ranges {759815=471768,1742451=465507,2917178=242637} 2024-09-13T14:26:31.583 INFO:tasks.rados.rados.0.smithi151.stdout:1389: writing smithi15186934-12 from 759815 to 1231583 tid 1 2024-09-13T14:26:31.589 INFO:tasks.rados.rados.0.smithi151.stdout:1389: writing smithi15186934-12 from 1742451 to 2207958 tid 2 2024-09-13T14:26:31.595 INFO:tasks.rados.rados.0.smithi151.stdout:1389: writing smithi15186934-12 from 2917178 to 3159815 tid 3 2024-09-13T14:26:31.595 INFO:tasks.rados.rados.0.smithi151.stdout:1390: read oid 40 snap -1 2024-09-13T14:26:31.595 INFO:tasks.rados.rados.0.smithi151.stdout:1390: expect deleted 2024-09-13T14:26:31.595 INFO:tasks.rados.rados.0.smithi151.stdout:1391: rollback oid 2 current snap is 175 2024-09-13T14:26:31.595 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 2 to 167 2024-09-13T14:26:31.595 INFO:tasks.rados.rados.0.smithi151.stdout:1389: finishing write tid 1 to smithi15186934-12 2024-09-13T14:26:31.595 INFO:tasks.rados.rados.0.smithi151.stdout:1389: finishing write tid 2 to smithi15186934-12 2024-09-13T14:26:31.595 INFO:tasks.rados.rados.0.smithi151.stdout:1392: rollback oid 42 current snap is 175 2024-09-13T14:26:31.595 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 42 to 164 2024-09-13T14:26:31.595 INFO:tasks.rados.rados.0.smithi151.stdout:1393: read oid 45 snap -1 2024-09-13T14:26:31.595 INFO:tasks.rados.rados.0.smithi151.stdout:1393: expect deleted 2024-09-13T14:26:31.595 INFO:tasks.rados.rados.0.smithi151.stdout:1394: write oid 23 current snap is 175 2024-09-13T14:26:31.595 INFO:tasks.rados.rados.0.smithi151.stdout:1394: seq_num 546 ranges {776622=547017,1800933=725518,2944601=232021} 2024-09-13T14:26:31.602 INFO:tasks.rados.rados.0.smithi151.stdout:1394: writing smithi15186934-23 from 776622 to 1323639 tid 1 2024-09-13T14:26:31.609 INFO:tasks.rados.rados.0.smithi151.stdout:1394: writing smithi15186934-23 from 1800933 to 2526451 tid 2 2024-09-13T14:26:31.613 INFO:tasks.rados.rados.0.smithi151.stdout:1394: writing smithi15186934-23 from 2944601 to 3176622 tid 3 2024-09-13T14:26:31.613 INFO:tasks.rados.rados.0.smithi151.stdout:1395: snap_remove snap 160 2024-09-13T14:26:31.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:31 smithi184 ceph-mon[33394]: osdmap e402: 8 total, 8 up, 8 in 2024-09-13T14:26:31.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:31 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:26:31.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:31 smithi151 ceph-mon[104450]: osdmap e402: 8 total, 8 up, 8 in 2024-09-13T14:26:31.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:26:31.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:31 smithi151 ceph-mon[108966]: osdmap e402: 8 total, 8 up, 8 in 2024-09-13T14:26:31.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:26:32.579 INFO:tasks.rados.rados.0.smithi151.stdout:1389: finishing write tid 3 to smithi15186934-12 2024-09-13T14:26:32.579 INFO:tasks.rados.rados.0.smithi151.stdout:1389: finishing write tid 4 to smithi15186934-12 2024-09-13T14:26:32.580 INFO:tasks.rados.rados.0.smithi151.stdout:1390: done (6 left) 2024-09-13T14:26:32.580 INFO:tasks.rados.rados.0.smithi151.stdout:1393: done (5 left) 2024-09-13T14:26:32.580 INFO:tasks.rados.rados.0.smithi151.stdout:1395: done (4 left) 2024-09-13T14:26:32.580 INFO:tasks.rados.rados.0.smithi151.stdout:1396: delete oid 19 current snap is 175 2024-09-13T14:26:32.580 INFO:tasks.rados.rados.0.smithi151.stdout:1389: finishing write tid 5 to smithi15186934-12 2024-09-13T14:26:32.580 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 12 v 318 (ObjNum 545 snap 175 seq_num 545) dirty exists 2024-09-13T14:26:32.580 INFO:tasks.rados.rados.0.smithi151.stdout:1389: left oid 12 (ObjNum 545 snap 175 seq_num 545) 2024-09-13T14:26:32.580 INFO:tasks.rados.rados.0.smithi151.stdout:1391: finishing rollback tid 0 to smithi15186934-2 2024-09-13T14:26:32.580 INFO:tasks.rados.rados.0.smithi151.stdout:1391: finishing rollback tid 1 to smithi15186934-2 2024-09-13T14:26:32.580 INFO:tasks.rados.rados.0.smithi151.stdout:1391: finishing rollback tid 2 to smithi15186934-2 2024-09-13T14:26:32.580 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 2 v 499 (ObjNum 481 snap 156 seq_num 481) dirty exists 2024-09-13T14:26:32.580 INFO:tasks.rados.rados.0.smithi151.stdout:1392: finishing rollback tid 0 to smithi15186934-42 2024-09-13T14:26:32.580 INFO:tasks.rados.rados.0.smithi151.stdout:1392: finishing rollback tid 1 to smithi15186934-42 2024-09-13T14:26:32.581 INFO:tasks.rados.rados.0.smithi151.stdout:1392: finishing rollback tid 2 to smithi15186934-42 2024-09-13T14:26:32.581 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 706 (ObjNum 479 snap 156 seq_num 479) dirty exists 2024-09-13T14:26:32.581 INFO:tasks.rados.rados.0.smithi151.stdout:1394: finishing write tid 1 to smithi15186934-23 2024-09-13T14:26:32.581 INFO:tasks.rados.rados.0.smithi151.stdout:1394: finishing write tid 2 to smithi15186934-23 2024-09-13T14:26:32.581 INFO:tasks.rados.rados.0.smithi151.stdout:1394: finishing write tid 3 to smithi15186934-23 2024-09-13T14:26:32.581 INFO:tasks.rados.rados.0.smithi151.stdout:1394: finishing write tid 4 to smithi15186934-23 2024-09-13T14:26:32.581 INFO:tasks.rados.rados.0.smithi151.stdout:1394: finishing write tid 5 to smithi15186934-23 2024-09-13T14:26:32.581 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 23 v 607 (ObjNum 546 snap 175 seq_num 546) dirty exists 2024-09-13T14:26:32.581 INFO:tasks.rados.rados.0.smithi151.stdout:1394: left oid 23 (ObjNum 546 snap 175 seq_num 546) 2024-09-13T14:26:32.587 INFO:tasks.rados.rados.0.smithi151.stdout:1389: done (4 left) 2024-09-13T14:26:32.587 INFO:tasks.rados.rados.0.smithi151.stdout:1391: done (3 left) 2024-09-13T14:26:32.587 INFO:tasks.rados.rados.0.smithi151.stdout:1392: done (2 left) 2024-09-13T14:26:32.587 INFO:tasks.rados.rados.0.smithi151.stdout:1394: done (1 left) 2024-09-13T14:26:32.587 INFO:tasks.rados.rados.0.smithi151.stdout:1396: done (0 left) 2024-09-13T14:26:32.587 INFO:tasks.rados.rados.0.smithi151.stdout:1397: read oid 9 snap -1 2024-09-13T14:26:32.587 INFO:tasks.rados.rados.0.smithi151.stdout:1397: expect (ObjNum 268 snap 86 seq_num 268) 2024-09-13T14:26:32.587 INFO:tasks.rados.rados.0.smithi151.stdout:1398: read oid 16 snap -1 2024-09-13T14:26:32.587 INFO:tasks.rados.rados.0.smithi151.stdout:1398: expect (ObjNum 446 snap 144 seq_num 446) 2024-09-13T14:26:32.587 INFO:tasks.rados.rados.0.smithi151.stdout:1399: write oid 24 current snap is 175 2024-09-13T14:26:32.588 INFO:tasks.rados.rados.0.smithi151.stdout:1399: seq_num 548 ranges {410236=413868,1234250=729187,2715800=423063} 2024-09-13T14:26:32.591 INFO:tasks.rados.rados.0.smithi151.stdout:1399: writing smithi15186934-24 from 410236 to 824104 tid 1 2024-09-13T14:26:32.598 INFO:tasks.rados.rados.0.smithi151.stdout:1399: writing smithi15186934-24 from 1234250 to 1963437 tid 2 2024-09-13T14:26:32.604 INFO:tasks.rados.rados.0.smithi151.stdout:1399: writing smithi15186934-24 from 2715800 to 3138863 tid 3 2024-09-13T14:26:32.605 INFO:tasks.rados.rados.0.smithi151.stdout:1400: snap_remove snap 174 2024-09-13T14:26:32.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:32 smithi184 ceph-mon[33394]: pgmap v221: 89 pgs: 89 active+clean; 238 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:26:32.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:32 smithi184 ceph-mon[33394]: osdmap e403: 8 total, 8 up, 8 in 2024-09-13T14:26:32.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:32 smithi151 ceph-mon[108966]: pgmap v221: 89 pgs: 89 active+clean; 238 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:26:32.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:32 smithi151 ceph-mon[108966]: osdmap e403: 8 total, 8 up, 8 in 2024-09-13T14:26:32.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:32 smithi151 ceph-mon[104450]: pgmap v221: 89 pgs: 89 active+clean; 238 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:26:32.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:32 smithi151 ceph-mon[104450]: osdmap e403: 8 total, 8 up, 8 in 2024-09-13T14:26:33.588 INFO:tasks.rados.rados.0.smithi151.stdout:1400: done (3 left) 2024-09-13T14:26:33.588 INFO:tasks.rados.rados.0.smithi151.stdout:1401: snap_remove snap 173 2024-09-13T14:26:33.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:33 smithi184 ceph-mon[33394]: osdmap e404: 8 total, 8 up, 8 in 2024-09-13T14:26:33.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:33 smithi151 ceph-mon[108966]: osdmap e404: 8 total, 8 up, 8 in 2024-09-13T14:26:33.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:33 smithi151 ceph-mon[104450]: osdmap e404: 8 total, 8 up, 8 in 2024-09-13T14:26:34.478 INFO:tasks.rados.rados.0.smithi151.stdout:1401: done (3 left) 2024-09-13T14:26:34.478 INFO:tasks.rados.rados.0.smithi151.stdout:1402: write oid 36 current snap is 175 2024-09-13T14:26:34.478 INFO:tasks.rados.rados.0.smithi151.stdout:1402: seq_num 549 ranges {427043=489117,1576379=472845,2626870=600173} 2024-09-13T14:26:34.483 INFO:tasks.rados.rados.0.smithi151.stdout:1402: writing smithi15186934-36 from 427043 to 916160 tid 1 2024-09-13T14:26:34.490 INFO:tasks.rados.rados.0.smithi151.stdout:1402: writing smithi15186934-36 from 1576379 to 2049224 tid 2 2024-09-13T14:26:34.497 INFO:tasks.rados.rados.0.smithi151.stdout:1402: writing smithi15186934-36 from 2626870 to 3227043 tid 3 2024-09-13T14:26:34.497 INFO:tasks.rados.rados.0.smithi151.stdout:1403: snap_create 2024-09-13T14:26:34.497 INFO:tasks.rados.rados.0.smithi151.stdout:1397: expect (ObjNum 268 snap 86 seq_num 268) 2024-09-13T14:26:34.579 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:26:34 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:26:34.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:34.675 INFO:tasks.rados.rados.0.smithi151.stdout:1399: finishing write tid 1 to smithi15186934-24 2024-09-13T14:26:34.675 INFO:tasks.rados.rados.0.smithi151.stdout:1398: expect (ObjNum 446 snap 144 seq_num 446) 2024-09-13T14:26:34.852 INFO:tasks.rados.rados.0.smithi151.stdout:1399: finishing write tid 2 to smithi15186934-24 2024-09-13T14:26:34.852 INFO:tasks.rados.rados.0.smithi151.stdout:1399: finishing write tid 3 to smithi15186934-24 2024-09-13T14:26:34.852 INFO:tasks.rados.rados.0.smithi151.stdout:1399: finishing write tid 4 to smithi15186934-24 2024-09-13T14:26:34.852 INFO:tasks.rados.rados.0.smithi151.stdout:1399: finishing write tid 5 to smithi15186934-24 2024-09-13T14:26:34.852 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 24 v 904 (ObjNum 548 snap 175 seq_num 548) dirty exists 2024-09-13T14:26:34.852 INFO:tasks.rados.rados.0.smithi151.stdout:1399: left oid 24 (ObjNum 548 snap 175 seq_num 548) 2024-09-13T14:26:34.852 INFO:tasks.rados.rados.0.smithi151.stdout:1402: finishing write tid 1 to smithi15186934-36 2024-09-13T14:26:34.852 INFO:tasks.rados.rados.0.smithi151.stdout:1402: finishing write tid 2 to smithi15186934-36 2024-09-13T14:26:34.852 INFO:tasks.rados.rados.0.smithi151.stdout:1402: finishing write tid 3 to smithi15186934-36 2024-09-13T14:26:34.852 INFO:tasks.rados.rados.0.smithi151.stdout:1402: finishing write tid 4 to smithi15186934-36 2024-09-13T14:26:34.852 INFO:tasks.rados.rados.0.smithi151.stdout:1402: finishing write tid 5 to smithi15186934-36 2024-09-13T14:26:34.853 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 36 v 326 (ObjNum 549 snap 175 seq_num 549) dirty exists 2024-09-13T14:26:34.853 INFO:tasks.rados.rados.0.smithi151.stdout:1402: left oid 36 (ObjNum 549 snap 175 seq_num 549) 2024-09-13T14:26:34.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:34 smithi184 ceph-mon[33394]: pgmap v224: 89 pgs: 89 active+clean; 251 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 17 op/s 2024-09-13T14:26:34.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:34 smithi184 ceph-mon[33394]: osdmap e405: 8 total, 8 up, 8 in 2024-09-13T14:26:34.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:34 smithi184 ceph-mon[33394]: osdmap e406: 8 total, 8 up, 8 in 2024-09-13T14:26:34.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:34 smithi151 ceph-mon[108966]: pgmap v224: 89 pgs: 89 active+clean; 251 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 17 op/s 2024-09-13T14:26:34.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:34 smithi151 ceph-mon[108966]: osdmap e405: 8 total, 8 up, 8 in 2024-09-13T14:26:34.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:34 smithi151 ceph-mon[108966]: osdmap e406: 8 total, 8 up, 8 in 2024-09-13T14:26:34.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:34 smithi151 ceph-mon[104450]: pgmap v224: 89 pgs: 89 active+clean; 251 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 17 op/s 2024-09-13T14:26:34.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:34 smithi151 ceph-mon[104450]: osdmap e405: 8 total, 8 up, 8 in 2024-09-13T14:26:34.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:34 smithi151 ceph-mon[104450]: osdmap e406: 8 total, 8 up, 8 in 2024-09-13T14:26:34.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:26:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:26:34] "GET /metrics HTTP/1.1" 200 36406 "" "Prometheus/2.51.0" 2024-09-13T14:26:35.476 INFO:tasks.rados.rados.0.smithi151.stdout:1397: done (4 left) 2024-09-13T14:26:35.476 INFO:tasks.rados.rados.0.smithi151.stdout:1398: done (3 left) 2024-09-13T14:26:35.477 INFO:tasks.rados.rados.0.smithi151.stdout:1399: done (2 left) 2024-09-13T14:26:35.477 INFO:tasks.rados.rados.0.smithi151.stdout:1402: done (1 left) 2024-09-13T14:26:35.477 INFO:tasks.rados.rados.0.smithi151.stdout:1403: done (0 left) 2024-09-13T14:26:35.477 INFO:tasks.rados.rados.0.smithi151.stdout:1404: write oid 24 current snap is 176 2024-09-13T14:26:35.477 INFO:tasks.rados.rados.0.smithi151.stdout:1404: seq_num 550 ranges {443850=564366,1634861=732856,2770646=473204} 2024-09-13T14:26:35.482 INFO:tasks.rados.rados.0.smithi151.stdout:1404: writing smithi15186934-24 from 443850 to 1008216 tid 1 2024-09-13T14:26:35.489 INFO:tasks.rados.rados.0.smithi151.stdout:1404: writing smithi15186934-24 from 1634861 to 2367717 tid 2 2024-09-13T14:26:35.494 INFO:tasks.rados.rados.0.smithi151.stdout:1404: writing smithi15186934-24 from 2770646 to 3243850 tid 3 2024-09-13T14:26:35.494 INFO:tasks.rados.rados.0.smithi151.stdout:1405: read oid 49 snap -1 2024-09-13T14:26:35.494 INFO:tasks.rados.rados.0.smithi151.stdout:1405: expect (ObjNum 462 snap 150 seq_num 462) 2024-09-13T14:26:35.495 INFO:tasks.rados.rados.0.smithi151.stdout:1406: write oid 9 current snap is 176 2024-09-13T14:26:35.495 INFO:tasks.rados.rados.0.smithi151.stdout:1404: finishing write tid 1 to smithi15186934-24 2024-09-13T14:26:35.495 INFO:tasks.rados.rados.0.smithi151.stdout:1404: finishing write tid 2 to smithi15186934-24 2024-09-13T14:26:35.495 INFO:tasks.rados.rados.0.smithi151.stdout:1406: seq_num 551 ranges {460657=639615,1693343=476514,2681716=578941} 2024-09-13T14:26:35.500 INFO:tasks.rados.rados.0.smithi151.stdout:1406: writing smithi15186934-9 from 460657 to 1100272 tid 1 2024-09-13T14:26:35.506 INFO:tasks.rados.rados.0.smithi151.stdout:1406: writing smithi15186934-9 from 1693343 to 2169857 tid 2 2024-09-13T14:26:35.513 INFO:tasks.rados.rados.0.smithi151.stdout:1406: writing smithi15186934-9 from 2681716 to 3260657 tid 3 2024-09-13T14:26:35.513 INFO:tasks.rados.rados.0.smithi151.stdout:1407: read oid 12 snap 141 2024-09-13T14:26:35.513 INFO:tasks.rados.rados.0.smithi151.stdout:1407: expect (ObjNum 258 snap 81 seq_num 258) 2024-09-13T14:26:35.513 INFO:tasks.rados.rados.0.smithi151.stdout:1408: delete oid 10 current snap is 176 2024-09-13T14:26:35.513 INFO:tasks.rados.rados.0.smithi151.stdout:1404: finishing write tid 3 to smithi15186934-24 2024-09-13T14:26:35.513 INFO:tasks.rados.rados.0.smithi151.stdout:1404: finishing write tid 4 to smithi15186934-24 2024-09-13T14:26:35.513 INFO:tasks.rados.rados.0.smithi151.stdout:1404: finishing write tid 5 to smithi15186934-24 2024-09-13T14:26:35.514 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 24 v 913 (ObjNum 550 snap 176 seq_num 550) dirty exists 2024-09-13T14:26:35.514 INFO:tasks.rados.rados.0.smithi151.stdout:1404: left oid 24 (ObjNum 550 snap 176 seq_num 550) 2024-09-13T14:26:35.514 INFO:tasks.rados.rados.0.smithi151.stdout:1405: expect (ObjNum 462 snap 150 seq_num 462) 2024-09-13T14:26:35.735 INFO:tasks.rados.rados.0.smithi151.stdout:1406: finishing write tid 1 to smithi15186934-9 2024-09-13T14:26:35.735 INFO:tasks.rados.rados.0.smithi151.stdout:1406: finishing write tid 2 to smithi15186934-9 2024-09-13T14:26:35.735 INFO:tasks.rados.rados.0.smithi151.stdout:1407: expect (ObjNum 258 snap 81 seq_num 258) 2024-09-13T14:26:35.861 INFO:tasks.rados.rados.0.smithi151.stdout:1406: finishing write tid 3 to smithi15186934-9 2024-09-13T14:26:35.861 INFO:tasks.rados.rados.0.smithi151.stdout:1406: finishing write tid 4 to smithi15186934-9 2024-09-13T14:26:35.861 INFO:tasks.rados.rados.0.smithi151.stdout:1406: finishing write tid 5 to smithi15186934-9 2024-09-13T14:26:35.862 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 562 (ObjNum 551 snap 176 seq_num 551) dirty exists 2024-09-13T14:26:35.862 INFO:tasks.rados.rados.0.smithi151.stdout:1406: left oid 9 (ObjNum 551 snap 176 seq_num 551) 2024-09-13T14:26:35.862 INFO:tasks.rados.rados.0.smithi151.stdout:1404: done (4 left) 2024-09-13T14:26:35.862 INFO:tasks.rados.rados.0.smithi151.stdout:1405: done (3 left) 2024-09-13T14:26:35.862 INFO:tasks.rados.rados.0.smithi151.stdout:1406: done (2 left) 2024-09-13T14:26:35.862 INFO:tasks.rados.rados.0.smithi151.stdout:1407: done (1 left) 2024-09-13T14:26:35.862 INFO:tasks.rados.rados.0.smithi151.stdout:1408: done (0 left) 2024-09-13T14:26:35.862 INFO:tasks.rados.rados.0.smithi151.stdout:1409: snap_remove snap 175 2024-09-13T14:26:36.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:26:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:26:35.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:36.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:26:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:26:35.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:36.586 INFO:tasks.rados.rados.0.smithi151.stdout:1409: done (0 left) 2024-09-13T14:26:36.586 INFO:tasks.rados.rados.0.smithi151.stdout:1410: write oid 21 current snap is 176 2024-09-13T14:26:36.586 INFO:tasks.rados.rados.0.smithi151.stdout:1410: seq_num 553 ranges {494271=790113,1693954=480183,2620209=512129} 2024-09-13T14:26:36.593 INFO:tasks.rados.rados.0.smithi151.stdout:1410: writing smithi15186934-21 from 494271 to 1284384 tid 1 2024-09-13T14:26:36.598 INFO:tasks.rados.rados.0.smithi151.stdout:1410: writing smithi15186934-21 from 1693954 to 2174137 tid 2 2024-09-13T14:26:36.603 INFO:tasks.rados.rados.0.smithi151.stdout:1410: writing smithi15186934-21 from 2620209 to 3132338 tid 3 2024-09-13T14:26:36.603 INFO:tasks.rados.rados.0.smithi151.stdout:1411: read oid 25 snap -1 2024-09-13T14:26:36.603 INFO:tasks.rados.rados.0.smithi151.stdout:1411: expect (ObjNum 512 snap 165 seq_num 512) 2024-09-13T14:26:36.603 INFO:tasks.rados.rados.0.smithi151.stdout:1412: delete oid 19 current snap is 176 2024-09-13T14:26:36.603 INFO:tasks.rados.rados.0.smithi151.stdout:1410: finishing write tid 1 to smithi15186934-21 2024-09-13T14:26:36.604 INFO:tasks.rados.rados.0.smithi151.stdout:1410: finishing write tid 2 to smithi15186934-21 2024-09-13T14:26:36.605 INFO:tasks.rados.rados.0.smithi151.stdout:1412: done (2 left) 2024-09-13T14:26:36.605 INFO:tasks.rados.rados.0.smithi151.stdout:1413: delete oid 46 current snap is 176 2024-09-13T14:26:36.606 INFO:tasks.rados.rados.0.smithi151.stdout:1410: finishing write tid 3 to smithi15186934-21 2024-09-13T14:26:36.606 INFO:tasks.rados.rados.0.smithi151.stdout:1410: finishing write tid 4 to smithi15186934-21 2024-09-13T14:26:36.606 INFO:tasks.rados.rados.0.smithi151.stdout:1410: finishing write tid 5 to smithi15186934-21 2024-09-13T14:26:36.606 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 21 v 301 (ObjNum 553 snap 176 seq_num 553) dirty exists 2024-09-13T14:26:36.606 INFO:tasks.rados.rados.0.smithi151.stdout:1410: left oid 21 (ObjNum 553 snap 176 seq_num 553) 2024-09-13T14:26:36.607 INFO:tasks.rados.rados.0.smithi151.stdout:1410: done (2 left) 2024-09-13T14:26:36.607 INFO:tasks.rados.rados.0.smithi151.stdout:1413: done (1 left) 2024-09-13T14:26:36.607 INFO:tasks.rados.rados.0.smithi151.stdout:1414: write oid 13 current snap is 176 2024-09-13T14:26:36.607 INFO:tasks.rados.rados.0.smithi151.stdout:1414: seq_num 556 ranges {544692=732213,1985753=743863,3335184=9508} 2024-09-13T14:26:36.614 INFO:tasks.rados.rados.0.smithi151.stdout:1414: writing smithi15186934-13 from 544692 to 1276905 tid 1 2024-09-13T14:26:36.622 INFO:tasks.rados.rados.0.smithi151.stdout:1414: writing smithi15186934-13 from 1985753 to 2729616 tid 2 2024-09-13T14:26:36.625 INFO:tasks.rados.rados.0.smithi151.stdout:1414: writing smithi15186934-13 from 3335184 to 3344692 tid 3 2024-09-13T14:26:36.625 INFO:tasks.rados.rados.0.smithi151.stdout:1415: write oid 36 current snap is 176 2024-09-13T14:26:36.625 INFO:tasks.rados.rados.0.smithi151.stdout:1415: seq_num 557 ranges {561499=407462,1527882=487521,2729901=557217} 2024-09-13T14:26:36.630 INFO:tasks.rados.rados.0.smithi151.stdout:1415: writing smithi15186934-36 from 561499 to 968961 tid 1 2024-09-13T14:26:36.636 INFO:tasks.rados.rados.0.smithi151.stdout:1415: writing smithi15186934-36 from 1527882 to 2015403 tid 2 2024-09-13T14:26:36.643 INFO:tasks.rados.rados.0.smithi151.stdout:1415: writing smithi15186934-36 from 2729901 to 3287118 tid 3 2024-09-13T14:26:36.643 INFO:tasks.rados.rados.0.smithi151.stdout:1416: read oid 33 snap -1 2024-09-13T14:26:36.644 INFO:tasks.rados.rados.0.smithi151.stdout:1416: expect (ObjNum 508 snap 164 seq_num 508) 2024-09-13T14:26:36.644 INFO:tasks.rados.rados.0.smithi151.stdout:1417: snap_remove snap 156 2024-09-13T14:26:36.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:36 smithi184 ceph-mon[33394]: pgmap v227: 89 pgs: 89 active+clean; 251 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 17 op/s 2024-09-13T14:26:36.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:36 smithi184 ceph-mon[33394]: osdmap e407: 8 total, 8 up, 8 in 2024-09-13T14:26:36.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:36 smithi151 ceph-mon[104450]: pgmap v227: 89 pgs: 89 active+clean; 251 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 17 op/s 2024-09-13T14:26:36.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:36 smithi151 ceph-mon[104450]: osdmap e407: 8 total, 8 up, 8 in 2024-09-13T14:26:36.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:36 smithi151 ceph-mon[108966]: pgmap v227: 89 pgs: 89 active+clean; 251 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 17 op/s 2024-09-13T14:26:36.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:36 smithi151 ceph-mon[108966]: osdmap e407: 8 total, 8 up, 8 in 2024-09-13T14:26:37.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:26:36 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:26:36.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi184\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:37.593 INFO:tasks.rados.rados.0.smithi151.stdout:1417: done (4 left) 2024-09-13T14:26:37.593 INFO:tasks.rados.rados.0.smithi151.stdout:1418: read oid 35 snap -1 2024-09-13T14:26:37.593 INFO:tasks.rados.rados.0.smithi151.stdout:1418: expect (ObjNum 541 snap 174 seq_num 541) 2024-09-13T14:26:37.593 INFO:tasks.rados.rados.0.smithi151.stdout:1419: snap_remove snap 170 2024-09-13T14:26:37.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:37 smithi184 ceph-mon[33394]: osdmap e408: 8 total, 8 up, 8 in 2024-09-13T14:26:37.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:37 smithi151 ceph-mon[104450]: osdmap e408: 8 total, 8 up, 8 in 2024-09-13T14:26:37.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:37 smithi151 ceph-mon[108966]: osdmap e408: 8 total, 8 up, 8 in 2024-09-13T14:26:38.597 INFO:tasks.rados.rados.0.smithi151.stdout:1419: done (5 left) 2024-09-13T14:26:38.598 INFO:tasks.rados.rados.0.smithi151.stdout:1420: snap_create 2024-09-13T14:26:38.598 INFO:tasks.rados.rados.0.smithi151.stdout:1411: expect (ObjNum 512 snap 165 seq_num 512) 2024-09-13T14:26:38.787 INFO:tasks.rados.rados.0.smithi151.stdout:1414: finishing write tid 1 to smithi15186934-13 2024-09-13T14:26:38.788 INFO:tasks.rados.rados.0.smithi151.stdout:1414: finishing write tid 2 to smithi15186934-13 2024-09-13T14:26:38.788 INFO:tasks.rados.rados.0.smithi151.stdout:1414: finishing write tid 3 to smithi15186934-13 2024-09-13T14:26:38.788 INFO:tasks.rados.rados.0.smithi151.stdout:1414: finishing write tid 4 to smithi15186934-13 2024-09-13T14:26:38.788 INFO:tasks.rados.rados.0.smithi151.stdout:1414: finishing write tid 5 to smithi15186934-13 2024-09-13T14:26:38.788 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 13 v 512 (ObjNum 556 snap 176 seq_num 556) dirty exists 2024-09-13T14:26:38.788 INFO:tasks.rados.rados.0.smithi151.stdout:1414: left oid 13 (ObjNum 556 snap 176 seq_num 556) 2024-09-13T14:26:38.788 INFO:tasks.rados.rados.0.smithi151.stdout:1415: finishing write tid 1 to smithi15186934-36 2024-09-13T14:26:38.788 INFO:tasks.rados.rados.0.smithi151.stdout:1415: finishing write tid 2 to smithi15186934-36 2024-09-13T14:26:38.788 INFO:tasks.rados.rados.0.smithi151.stdout:1415: finishing write tid 3 to smithi15186934-36 2024-09-13T14:26:38.788 INFO:tasks.rados.rados.0.smithi151.stdout:1416: expect (ObjNum 508 snap 164 seq_num 508) 2024-09-13T14:26:38.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:38 smithi184 ceph-mon[33394]: pgmap v230: 89 pgs: 89 active+clean; 263 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 14 MiB/s rd, 2.7 MiB/s wr, 23 op/s 2024-09-13T14:26:38.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:38 smithi184 ceph-mon[33394]: osdmap e409: 8 total, 8 up, 8 in 2024-09-13T14:26:38.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:38 smithi151 ceph-mon[104450]: pgmap v230: 89 pgs: 89 active+clean; 263 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 14 MiB/s rd, 2.7 MiB/s wr, 23 op/s 2024-09-13T14:26:38.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:38 smithi151 ceph-mon[104450]: osdmap e409: 8 total, 8 up, 8 in 2024-09-13T14:26:38.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:38 smithi151 ceph-mon[108966]: pgmap v230: 89 pgs: 89 active+clean; 263 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 14 MiB/s rd, 2.7 MiB/s wr, 23 op/s 2024-09-13T14:26:38.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:38 smithi151 ceph-mon[108966]: osdmap e409: 8 total, 8 up, 8 in 2024-09-13T14:26:38.865 INFO:tasks.rados.rados.0.smithi151.stdout:1415: finishing write tid 4 to smithi15186934-36 2024-09-13T14:26:38.866 INFO:tasks.rados.rados.0.smithi151.stdout:1415: finishing write tid 5 to smithi15186934-36 2024-09-13T14:26:38.866 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 36 v 330 (ObjNum 557 snap 176 seq_num 557) dirty exists 2024-09-13T14:26:38.866 INFO:tasks.rados.rados.0.smithi151.stdout:1415: left oid 36 (ObjNum 557 snap 176 seq_num 557) 2024-09-13T14:26:38.866 INFO:tasks.rados.rados.0.smithi151.stdout:1418: expect (ObjNum 541 snap 174 seq_num 541) 2024-09-13T14:26:39.476 INFO:tasks.rados.rados.0.smithi151.stdout:1411: done (5 left) 2024-09-13T14:26:39.477 INFO:tasks.rados.rados.0.smithi151.stdout:1414: done (4 left) 2024-09-13T14:26:39.477 INFO:tasks.rados.rados.0.smithi151.stdout:1415: done (3 left) 2024-09-13T14:26:39.477 INFO:tasks.rados.rados.0.smithi151.stdout:1416: done (2 left) 2024-09-13T14:26:39.477 INFO:tasks.rados.rados.0.smithi151.stdout:1418: done (1 left) 2024-09-13T14:26:39.477 INFO:tasks.rados.rados.0.smithi151.stdout:1420: done (0 left) 2024-09-13T14:26:39.477 INFO:tasks.rados.rados.0.smithi151.stdout:1421: delete oid 36 current snap is 177 2024-09-13T14:26:39.478 INFO:tasks.rados.rados.0.smithi151.stdout:1421: done (0 left) 2024-09-13T14:26:39.479 INFO:tasks.rados.rados.0.smithi151.stdout:1422: read oid 29 snap 133 2024-09-13T14:26:39.479 INFO:tasks.rados.rados.0.smithi151.stdout:1422: expect deleted 2024-09-13T14:26:39.479 INFO:tasks.rados.rados.0.smithi151.stdout:1423: write oid 27 current snap is 177 2024-09-13T14:26:39.479 INFO:tasks.rados.rados.0.smithi151.stdout:1423: seq_num 559 ranges {595113=557960,1644846=491190,2784747=610366} 2024-09-13T14:26:39.484 INFO:tasks.rados.rados.0.smithi151.stdout:1423: writing smithi15186934-27 from 595113 to 1153073 tid 1 2024-09-13T14:26:39.489 INFO:tasks.rados.rados.0.smithi151.stdout:1423: writing smithi15186934-27 from 1644846 to 2136036 tid 2 2024-09-13T14:26:39.496 INFO:tasks.rados.rados.0.smithi151.stdout:1423: writing smithi15186934-27 from 2784747 to 3395113 tid 3 2024-09-13T14:26:39.497 INFO:tasks.rados.rados.0.smithi151.stdout:1424: read oid 43 snap -1 2024-09-13T14:26:39.497 INFO:tasks.rados.rados.0.smithi151.stdout:1424: expect (ObjNum 497 snap 159 seq_num 497) 2024-09-13T14:26:39.497 INFO:tasks.rados.rados.0.smithi151.stdout:1425: delete oid 44 current snap is 177 2024-09-13T14:26:39.497 INFO:tasks.rados.rados.0.smithi151.stdout:1423: finishing write tid 1 to smithi15186934-27 2024-09-13T14:26:39.497 INFO:tasks.rados.rados.0.smithi151.stdout:1423: finishing write tid 2 to smithi15186934-27 2024-09-13T14:26:39.498 INFO:tasks.rados.rados.0.smithi151.stdout:1422: done (3 left) 2024-09-13T14:26:39.498 INFO:tasks.rados.rados.0.smithi151.stdout:1425: done (2 left) 2024-09-13T14:26:39.498 INFO:tasks.rados.rados.0.smithi151.stdout:1426: write oid 14 current snap is 177 2024-09-13T14:26:39.498 INFO:tasks.rados.rados.0.smithi151.stdout:1426: seq_num 561 ranges {628727=708458,2045457=494859,3123240=305487} 2024-09-13T14:26:39.505 INFO:tasks.rados.rados.0.smithi151.stdout:1426: writing smithi15186934-14 from 628727 to 1337185 tid 1 2024-09-13T14:26:39.512 INFO:tasks.rados.rados.0.smithi151.stdout:1426: writing smithi15186934-14 from 2045457 to 2540316 tid 2 2024-09-13T14:26:39.517 INFO:tasks.rados.rados.0.smithi151.stdout:1426: writing smithi15186934-14 from 3123240 to 3428727 tid 3 2024-09-13T14:26:39.517 INFO:tasks.rados.rados.0.smithi151.stdout:1427: snap_remove snap 165 2024-09-13T14:26:39.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:39 smithi184 ceph-mon[33394]: osdmap e410: 8 total, 8 up, 8 in 2024-09-13T14:26:39.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:39 smithi184 ceph-mon[33394]: osdmap e411: 8 total, 8 up, 8 in 2024-09-13T14:26:39.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:39 smithi151 ceph-mon[104450]: osdmap e410: 8 total, 8 up, 8 in 2024-09-13T14:26:39.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:39 smithi151 ceph-mon[104450]: osdmap e411: 8 total, 8 up, 8 in 2024-09-13T14:26:39.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:39 smithi151 ceph-mon[108966]: osdmap e410: 8 total, 8 up, 8 in 2024-09-13T14:26:39.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:39 smithi151 ceph-mon[108966]: osdmap e411: 8 total, 8 up, 8 in 2024-09-13T14:26:40.483 INFO:tasks.rados.rados.0.smithi151.stdout:1423: finishing write tid 3 to smithi15186934-27 2024-09-13T14:26:40.483 INFO:tasks.rados.rados.0.smithi151.stdout:1423: finishing write tid 4 to smithi15186934-27 2024-09-13T14:26:40.483 INFO:tasks.rados.rados.0.smithi151.stdout:1423: finishing write tid 5 to smithi15186934-27 2024-09-13T14:26:40.483 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 27 v 371 (ObjNum 559 snap 177 seq_num 559) dirty exists 2024-09-13T14:26:40.483 INFO:tasks.rados.rados.0.smithi151.stdout:1423: left oid 27 (ObjNum 559 snap 177 seq_num 559) 2024-09-13T14:26:40.483 INFO:tasks.rados.rados.0.smithi151.stdout:1424: expect (ObjNum 497 snap 159 seq_num 497) 2024-09-13T14:26:40.593 INFO:tasks.rados.rados.0.smithi151.stdout:1426: finishing write tid 1 to smithi15186934-14 2024-09-13T14:26:40.593 INFO:tasks.rados.rados.0.smithi151.stdout:1426: finishing write tid 2 to smithi15186934-14 2024-09-13T14:26:40.593 INFO:tasks.rados.rados.0.smithi151.stdout:1426: finishing write tid 3 to smithi15186934-14 2024-09-13T14:26:40.593 INFO:tasks.rados.rados.0.smithi151.stdout:1426: finishing write tid 4 to smithi15186934-14 2024-09-13T14:26:40.594 INFO:tasks.rados.rados.0.smithi151.stdout:1426: finishing write tid 5 to smithi15186934-14 2024-09-13T14:26:40.594 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 575 (ObjNum 561 snap 177 seq_num 561) dirty exists 2024-09-13T14:26:40.594 INFO:tasks.rados.rados.0.smithi151.stdout:1426: left oid 14 (ObjNum 561 snap 177 seq_num 561) 2024-09-13T14:26:40.594 INFO:tasks.rados.rados.0.smithi151.stdout:1423: done (3 left) 2024-09-13T14:26:40.594 INFO:tasks.rados.rados.0.smithi151.stdout:1424: done (2 left) 2024-09-13T14:26:40.594 INFO:tasks.rados.rados.0.smithi151.stdout:1426: done (1 left) 2024-09-13T14:26:40.594 INFO:tasks.rados.rados.0.smithi151.stdout:1427: done (0 left) 2024-09-13T14:26:40.594 INFO:tasks.rados.rados.0.smithi151.stdout:1428: read oid 4 snap -1 2024-09-13T14:26:40.594 INFO:tasks.rados.rados.0.smithi151.stdout:1428: expect (ObjNum 492 snap 158 seq_num 492) 2024-09-13T14:26:40.594 INFO:tasks.rados.rados.0.smithi151.stdout:1429: write oid 21 current snap is 177 2024-09-13T14:26:40.594 INFO:tasks.rados.rados.0.smithi151.stdout:1429: seq_num 562 ranges {645534=783707,2103939=638517,3150663=294871} 2024-09-13T14:26:40.601 INFO:tasks.rados.rados.0.smithi151.stdout:1429: writing smithi15186934-21 from 645534 to 1429241 tid 1 2024-09-13T14:26:40.601 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:26:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:26:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:40.608 INFO:tasks.rados.rados.0.smithi151.stdout:1429: writing smithi15186934-21 from 2103939 to 2742456 tid 2 2024-09-13T14:26:40.612 INFO:tasks.rados.rados.0.smithi151.stdout:1429: writing smithi15186934-21 from 3150663 to 3445534 tid 3 2024-09-13T14:26:40.612 INFO:tasks.rados.rados.0.smithi151.stdout:1430: write oid 9 current snap is 177 2024-09-13T14:26:40.612 INFO:tasks.rados.rados.0.smithi151.stdout:1430: seq_num 563 ranges {662341=575309,1878774=498528,2894439=457555} 2024-09-13T14:26:40.619 INFO:tasks.rados.rados.0.smithi151.stdout:1430: writing smithi15186934-9 from 662341 to 1237650 tid 1 2024-09-13T14:26:40.625 INFO:tasks.rados.rados.0.smithi151.stdout:1430: writing smithi15186934-9 from 1878774 to 2377302 tid 2 2024-09-13T14:26:40.631 INFO:tasks.rados.rados.0.smithi151.stdout:1430: writing smithi15186934-9 from 2894439 to 3351994 tid 3 2024-09-13T14:26:40.631 INFO:tasks.rados.rados.0.smithi151.stdout:1431: read oid 27 snap -1 2024-09-13T14:26:40.632 INFO:tasks.rados.rados.0.smithi151.stdout:1431: expect (ObjNum 559 snap 177 seq_num 559) 2024-09-13T14:26:40.632 INFO:tasks.rados.rados.0.smithi151.stdout:1432: write oid 5 current snap is 177 2024-09-13T14:26:40.632 INFO:tasks.rados.rados.0.smithi151.stdout:1432: seq_num 564 ranges {679148=650558,1937256=642186,3321862=157286} 2024-09-13T14:26:40.638 INFO:tasks.rados.rados.0.smithi151.stdout:1432: writing smithi15186934-5 from 679148 to 1329706 tid 1 2024-09-13T14:26:40.645 INFO:tasks.rados.rados.0.smithi151.stdout:1432: writing smithi15186934-5 from 1937256 to 2579442 tid 2 2024-09-13T14:26:40.650 INFO:tasks.rados.rados.0.smithi151.stdout:1432: writing smithi15186934-5 from 3321862 to 3479148 tid 3 2024-09-13T14:26:40.650 INFO:tasks.rados.rados.0.smithi151.stdout:1433: rollback oid 48 current snap is 177 2024-09-13T14:26:40.651 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 48 to 133 2024-09-13T14:26:40.651 INFO:tasks.rados.rados.0.smithi151.stdout:1428: expect (ObjNum 492 snap 158 seq_num 492) 2024-09-13T14:26:40.791 INFO:tasks.rados.rados.0.smithi151.stdout:1429: finishing write tid 1 to smithi15186934-21 2024-09-13T14:26:40.791 INFO:tasks.rados.rados.0.smithi151.stdout:1429: finishing write tid 2 to smithi15186934-21 2024-09-13T14:26:40.791 INFO:tasks.rados.rados.0.smithi151.stdout:1428: done (5 left) 2024-09-13T14:26:40.792 INFO:tasks.rados.rados.0.smithi151.stdout:1434: write oid 29 current snap is 177 2024-09-13T14:26:40.792 INFO:tasks.rados.rados.0.smithi151.stdout:1429: finishing write tid 3 to smithi15186934-21 2024-09-13T14:26:40.792 INFO:tasks.rados.rados.0.smithi151.stdout:1434: seq_num 565 ranges {695955=725807,1879385=502197,2949285=480099} 2024-09-13T14:26:40.799 INFO:tasks.rados.rados.0.smithi151.stdout:1434: writing smithi15186934-29 from 695955 to 1421762 tid 1 2024-09-13T14:26:40.804 INFO:tasks.rados.rados.0.smithi151.stdout:1434: writing smithi15186934-29 from 1879385 to 2381582 tid 2 2024-09-13T14:26:40.810 INFO:tasks.rados.rados.0.smithi151.stdout:1434: writing smithi15186934-29 from 2949285 to 3429384 tid 3 2024-09-13T14:26:40.810 INFO:tasks.rados.rados.0.smithi151.stdout:1435: write oid 37 current snap is 177 2024-09-13T14:26:40.810 INFO:tasks.rados.rados.0.smithi151.stdout:1429: finishing write tid 4 to smithi15186934-21 2024-09-13T14:26:40.810 INFO:tasks.rados.rados.0.smithi151.stdout:1429: finishing write tid 5 to smithi15186934-21 2024-09-13T14:26:40.811 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 21 v 310 (ObjNum 562 snap 177 seq_num 562) dirty exists 2024-09-13T14:26:40.811 INFO:tasks.rados.rados.0.smithi151.stdout:1429: left oid 21 (ObjNum 562 snap 177 seq_num 562) 2024-09-13T14:26:40.811 INFO:tasks.rados.rados.0.smithi151.stdout:1430: finishing write tid 1 to smithi15186934-9 2024-09-13T14:26:40.811 INFO:tasks.rados.rados.0.smithi151.stdout:1435: seq_num 566 ranges {712762=401056,1537867=645855,2860355=652407} 2024-09-13T14:26:40.816 INFO:tasks.rados.rados.0.smithi151.stdout:1435: writing smithi15186934-37 from 712762 to 1113818 tid 1 2024-09-13T14:26:40.822 INFO:tasks.rados.rados.0.smithi151.stdout:1435: writing smithi15186934-37 from 1537867 to 2183722 tid 2 2024-09-13T14:26:40.829 INFO:tasks.rados.rados.0.smithi151.stdout:1435: writing smithi15186934-37 from 2860355 to 3512762 tid 3 2024-09-13T14:26:40.829 INFO:tasks.rados.rados.0.smithi151.stdout:1429: done (6 left) 2024-09-13T14:26:40.829 INFO:tasks.rados.rados.0.smithi151.stdout:1436: snap_remove snap 167 2024-09-13T14:26:40.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:40 smithi184 ceph-mon[33394]: pgmap v233: 89 pgs: 89 active+clean; 263 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 14 MiB/s rd, 2.7 MiB/s wr, 23 op/s 2024-09-13T14:26:40.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:40 smithi184 ceph-mon[33394]: osdmap e412: 8 total, 8 up, 8 in 2024-09-13T14:26:40.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:40 smithi151 ceph-mon[104450]: pgmap v233: 89 pgs: 89 active+clean; 263 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 14 MiB/s rd, 2.7 MiB/s wr, 23 op/s 2024-09-13T14:26:40.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:40 smithi151 ceph-mon[104450]: osdmap e412: 8 total, 8 up, 8 in 2024-09-13T14:26:40.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:40 smithi151 ceph-mon[108966]: pgmap v233: 89 pgs: 89 active+clean; 263 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 14 MiB/s rd, 2.7 MiB/s wr, 23 op/s 2024-09-13T14:26:40.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:40 smithi151 ceph-mon[108966]: osdmap e412: 8 total, 8 up, 8 in 2024-09-13T14:26:41.617 INFO:tasks.rados.rados.0.smithi151.stdout:1436: done (6 left) 2024-09-13T14:26:41.618 INFO:tasks.rados.rados.0.smithi151.stdout:1437: read oid 23 snap -1 2024-09-13T14:26:41.618 INFO:tasks.rados.rados.0.smithi151.stdout:1437: expect (ObjNum 546 snap 175 seq_num 546) 2024-09-13T14:26:41.618 INFO:tasks.rados.rados.0.smithi151.stdout:1438: rollback oid 21 current snap is 177 2024-09-13T14:26:41.618 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 21 to 171 2024-09-13T14:26:41.618 INFO:tasks.rados.rados.0.smithi151.stdout:1430: finishing write tid 2 to smithi15186934-9 2024-09-13T14:26:41.618 INFO:tasks.rados.rados.0.smithi151.stdout:1430: finishing write tid 3 to smithi15186934-9 2024-09-13T14:26:41.618 INFO:tasks.rados.rados.0.smithi151.stdout:1430: finishing write tid 4 to smithi15186934-9 2024-09-13T14:26:41.618 INFO:tasks.rados.rados.0.smithi151.stdout:1430: finishing write tid 5 to smithi15186934-9 2024-09-13T14:26:41.618 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 584 (ObjNum 563 snap 177 seq_num 563) dirty exists 2024-09-13T14:26:41.618 INFO:tasks.rados.rados.0.smithi151.stdout:1430: left oid 9 (ObjNum 563 snap 177 seq_num 563) 2024-09-13T14:26:41.618 INFO:tasks.rados.rados.0.smithi151.stdout:1430: done (7 left) 2024-09-13T14:26:41.619 INFO:tasks.rados.rados.0.smithi151.stdout:1439: rollback oid 19 current snap is 177 2024-09-13T14:26:41.619 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 19 to 133 2024-09-13T14:26:41.619 INFO:tasks.rados.rados.0.smithi151.stdout:1432: finishing write tid 1 to smithi15186934-5 2024-09-13T14:26:41.619 INFO:tasks.rados.rados.0.smithi151.stdout:1431: expect (ObjNum 559 snap 177 seq_num 559) 2024-09-13T14:26:41.729 INFO:tasks.rados.rados.0.smithi151.stdout:1432: finishing write tid 2 to smithi15186934-5 2024-09-13T14:26:41.729 INFO:tasks.rados.rados.0.smithi151.stdout:1432: finishing write tid 3 to smithi15186934-5 2024-09-13T14:26:41.729 INFO:tasks.rados.rados.0.smithi151.stdout:1431: done (7 left) 2024-09-13T14:26:41.729 INFO:tasks.rados.rados.0.smithi151.stdout:1440: write oid 49 current snap is 177 2024-09-13T14:26:41.729 INFO:tasks.rados.rados.0.smithi151.stdout:1433: finishing rollback tid 1 to smithi15186934-48 2024-09-13T14:26:41.729 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 48 v 792 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-13T14:26:41.729 INFO:tasks.rados.rados.0.smithi151.stdout:1432: finishing write tid 4 to smithi15186934-5 2024-09-13T14:26:41.730 INFO:tasks.rados.rados.0.smithi151.stdout:1440: seq_num 567 ranges {729569=476305,1996349=505866,3004131=525438} 2024-09-13T14:26:41.735 INFO:tasks.rados.rados.0.smithi151.stdout:1440: writing smithi15186934-49 from 729569 to 1205874 tid 1 2024-09-13T14:26:41.742 INFO:tasks.rados.rados.0.smithi151.stdout:1440: writing smithi15186934-49 from 1996349 to 2502215 tid 2 2024-09-13T14:26:41.749 INFO:tasks.rados.rados.0.smithi151.stdout:1440: writing smithi15186934-49 from 3004131 to 3529569 tid 3 2024-09-13T14:26:41.749 INFO:tasks.rados.rados.0.smithi151.stdout:1433: done (7 left) 2024-09-13T14:26:41.749 INFO:tasks.rados.rados.0.smithi151.stdout:1441: rollback oid 9 current snap is 177 2024-09-13T14:26:41.749 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 9 to 164 2024-09-13T14:26:41.749 INFO:tasks.rados.rados.0.smithi151.stdout:1432: finishing write tid 5 to smithi15186934-5 2024-09-13T14:26:41.749 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 532 (ObjNum 564 snap 177 seq_num 564) dirty exists 2024-09-13T14:26:41.749 INFO:tasks.rados.rados.0.smithi151.stdout:1432: left oid 5 (ObjNum 564 snap 177 seq_num 564) 2024-09-13T14:26:41.750 INFO:tasks.rados.rados.0.smithi151.stdout:1434: finishing write tid 1 to smithi15186934-29 2024-09-13T14:26:41.750 INFO:tasks.rados.rados.0.smithi151.stdout:1434: finishing write tid 2 to smithi15186934-29 2024-09-13T14:26:41.750 INFO:tasks.rados.rados.0.smithi151.stdout:1434: finishing write tid 3 to smithi15186934-29 2024-09-13T14:26:41.750 INFO:tasks.rados.rados.0.smithi151.stdout:1434: finishing write tid 4 to smithi15186934-29 2024-09-13T14:26:41.750 INFO:tasks.rados.rados.0.smithi151.stdout:1434: finishing write tid 5 to smithi15186934-29 2024-09-13T14:26:41.750 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 29 v 398 (ObjNum 565 snap 177 seq_num 565) dirty exists 2024-09-13T14:26:41.750 INFO:tasks.rados.rados.0.smithi151.stdout:1434: left oid 29 (ObjNum 565 snap 177 seq_num 565) 2024-09-13T14:26:41.750 INFO:tasks.rados.rados.0.smithi151.stdout:1435: finishing write tid 1 to smithi15186934-37 2024-09-13T14:26:41.750 INFO:tasks.rados.rados.0.smithi151.stdout:1435: finishing write tid 2 to smithi15186934-37 2024-09-13T14:26:41.750 INFO:tasks.rados.rados.0.smithi151.stdout:1435: finishing write tid 3 to smithi15186934-37 2024-09-13T14:26:41.750 INFO:tasks.rados.rados.0.smithi151.stdout:1432: done (7 left) 2024-09-13T14:26:41.750 INFO:tasks.rados.rados.0.smithi151.stdout:1434: done (6 left) 2024-09-13T14:26:41.750 INFO:tasks.rados.rados.0.smithi151.stdout:1442: rollback oid 3 current snap is 177 2024-09-13T14:26:41.750 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 3 to 164 2024-09-13T14:26:41.751 INFO:tasks.rados.rados.0.smithi151.stdout:1435: finishing write tid 4 to smithi15186934-37 2024-09-13T14:26:41.751 INFO:tasks.rados.rados.0.smithi151.stdout:1435: finishing write tid 5 to smithi15186934-37 2024-09-13T14:26:41.751 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 37 v 625 (ObjNum 566 snap 177 seq_num 566) dirty exists 2024-09-13T14:26:41.751 INFO:tasks.rados.rados.0.smithi151.stdout:1435: left oid 37 (ObjNum 566 snap 177 seq_num 566) 2024-09-13T14:26:41.751 INFO:tasks.rados.rados.0.smithi151.stdout:1439: finishing rollback tid 1 to smithi15186934-19 2024-09-13T14:26:41.751 INFO:tasks.rados.rados.0.smithi151.stdout:1439: finishing rollback tid 2 to smithi15186934-19 2024-09-13T14:26:41.751 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 19 v 368 (ObjNum 323 snap 101 seq_num 323) dirty exists 2024-09-13T14:26:41.751 INFO:tasks.rados.rados.0.smithi151.stdout:1438: finishing rollback tid 0 to smithi15186934-21 2024-09-13T14:26:41.751 INFO:tasks.rados.rados.0.smithi151.stdout:1438: finishing rollback tid 1 to smithi15186934-21 2024-09-13T14:26:41.751 INFO:tasks.rados.rados.0.smithi151.stdout:1438: finishing rollback tid 2 to smithi15186934-21 2024-09-13T14:26:41.751 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 21 v 315 (ObjNum 527 snap 170 seq_num 527) dirty exists 2024-09-13T14:26:41.751 INFO:tasks.rados.rados.0.smithi151.stdout:1437: expect (ObjNum 546 snap 175 seq_num 546) 2024-09-13T14:26:41.928 INFO:tasks.rados.rados.0.smithi151.stdout:1440: finishing write tid 1 to smithi15186934-49 2024-09-13T14:26:41.928 INFO:tasks.rados.rados.0.smithi151.stdout:1440: finishing write tid 2 to smithi15186934-49 2024-09-13T14:26:41.928 INFO:tasks.rados.rados.0.smithi151.stdout:1442: finishing rollback tid 0 to smithi15186934-3 2024-09-13T14:26:41.928 INFO:tasks.rados.rados.0.smithi151.stdout:1441: finishing rollback tid 0 to smithi15186934-9 2024-09-13T14:26:41.928 INFO:tasks.rados.rados.0.smithi151.stdout:1442: finishing rollback tid 1 to smithi15186934-3 2024-09-13T14:26:41.928 INFO:tasks.rados.rados.0.smithi151.stdout:1441: finishing rollback tid 1 to smithi15186934-9 2024-09-13T14:26:41.928 INFO:tasks.rados.rados.0.smithi151.stdout:1442: finishing rollback tid 2 to smithi15186934-3 2024-09-13T14:26:41.928 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 547 (ObjNum 334 snap 103 seq_num 334) dirty exists 2024-09-13T14:26:41.928 INFO:tasks.rados.rados.0.smithi151.stdout:1440: finishing write tid 3 to smithi15186934-49 2024-09-13T14:26:41.928 INFO:tasks.rados.rados.0.smithi151.stdout:1441: finishing rollback tid 2 to smithi15186934-9 2024-09-13T14:26:41.929 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 591 (ObjNum 268 snap 86 seq_num 268) dirty exists 2024-09-13T14:26:41.929 INFO:tasks.rados.rados.0.smithi151.stdout:1440: finishing write tid 4 to smithi15186934-49 2024-09-13T14:26:41.929 INFO:tasks.rados.rados.0.smithi151.stdout:1440: finishing write tid 5 to smithi15186934-49 2024-09-13T14:26:41.929 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 49 v 632 (ObjNum 567 snap 177 seq_num 567) dirty exists 2024-09-13T14:26:41.929 INFO:tasks.rados.rados.0.smithi151.stdout:1440: left oid 49 (ObjNum 567 snap 177 seq_num 567) 2024-09-13T14:26:41.929 INFO:tasks.rados.rados.0.smithi151.stdout:1435: done (6 left) 2024-09-13T14:26:41.929 INFO:tasks.rados.rados.0.smithi151.stdout:1437: done (5 left) 2024-09-13T14:26:41.929 INFO:tasks.rados.rados.0.smithi151.stdout:1438: done (4 left) 2024-09-13T14:26:41.929 INFO:tasks.rados.rados.0.smithi151.stdout:1439: done (3 left) 2024-09-13T14:26:41.929 INFO:tasks.rados.rados.0.smithi151.stdout:1440: done (2 left) 2024-09-13T14:26:41.929 INFO:tasks.rados.rados.0.smithi151.stdout:1441: done (1 left) 2024-09-13T14:26:41.929 INFO:tasks.rados.rados.0.smithi151.stdout:1442: done (0 left) 2024-09-13T14:26:41.929 INFO:tasks.rados.rados.0.smithi151.stdout:1443: read oid 49 snap -1 2024-09-13T14:26:41.929 INFO:tasks.rados.rados.0.smithi151.stdout:1443: expect (ObjNum 567 snap 177 seq_num 567) 2024-09-13T14:26:41.930 INFO:tasks.rados.rados.0.smithi151.stdout:1444: snap_remove snap 164 2024-09-13T14:26:42.616 INFO:tasks.rados.rados.0.smithi151.stdout:1443: expect (ObjNum 567 snap 177 seq_num 567) 2024-09-13T14:26:42.813 INFO:tasks.rados.rados.0.smithi151.stdout:1443: done (1 left) 2024-09-13T14:26:42.814 INFO:tasks.rados.rados.0.smithi151.stdout:1444: done (0 left) 2024-09-13T14:26:42.814 INFO:tasks.rados.rados.0.smithi151.stdout:1445: read oid 44 snap -1 2024-09-13T14:26:42.814 INFO:tasks.rados.rados.0.smithi151.stdout:1445: expect deleted 2024-09-13T14:26:42.814 INFO:tasks.rados.rados.0.smithi151.stdout:1446: snap_create 2024-09-13T14:26:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:42 smithi184 ceph-mon[33394]: pgmap v236: 89 pgs: 89 active+clean; 263 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:26:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:42 smithi184 ceph-mon[33394]: osdmap e413: 8 total, 8 up, 8 in 2024-09-13T14:26:43.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:42 smithi151 ceph-mon[104450]: pgmap v236: 89 pgs: 89 active+clean; 263 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:26:43.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:42 smithi151 ceph-mon[104450]: osdmap e413: 8 total, 8 up, 8 in 2024-09-13T14:26:43.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:42 smithi151 ceph-mon[108966]: pgmap v236: 89 pgs: 89 active+clean; 263 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:26:43.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:42 smithi151 ceph-mon[108966]: osdmap e413: 8 total, 8 up, 8 in 2024-09-13T14:26:43.629 INFO:tasks.rados.rados.0.smithi151.stdout:1445: done (1 left) 2024-09-13T14:26:43.629 INFO:tasks.rados.rados.0.smithi151.stdout:1446: done (0 left) 2024-09-13T14:26:43.629 INFO:tasks.rados.rados.0.smithi151.stdout:1447: snap_create 2024-09-13T14:26:44.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:43 smithi184 ceph-mon[33394]: osdmap e414: 8 total, 8 up, 8 in 2024-09-13T14:26:44.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:43 smithi151 ceph-mon[104450]: osdmap e414: 8 total, 8 up, 8 in 2024-09-13T14:26:44.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:43 smithi151 ceph-mon[108966]: osdmap e414: 8 total, 8 up, 8 in 2024-09-13T14:26:44.479 INFO:tasks.rados.rados.0.smithi151.stdout:1447: done (0 left) 2024-09-13T14:26:44.479 INFO:tasks.rados.rados.0.smithi151.stdout:1448: write oid 17 current snap is 179 2024-09-13T14:26:44.479 INFO:tasks.rados.rados.0.smithi151.stdout:1448: seq_num 568 ranges {746376=551554,2054831=649524,3315201=231175} 2024-09-13T14:26:44.486 INFO:tasks.rados.rados.0.smithi151.stdout:1448: writing smithi15186934-17 from 746376 to 1297930 tid 1 2024-09-13T14:26:44.494 INFO:tasks.rados.rados.0.smithi151.stdout:1448: writing smithi15186934-17 from 2054831 to 2704355 tid 2 2024-09-13T14:26:44.499 INFO:tasks.rados.rados.0.smithi151.stdout:1448: writing smithi15186934-17 from 3315201 to 3546376 tid 3 2024-09-13T14:26:44.499 INFO:tasks.rados.rados.0.smithi151.stdout:1449: delete oid 4 current snap is 179 2024-09-13T14:26:44.499 INFO:tasks.rados.rados.0.smithi151.stdout:1448: finishing write tid 1 to smithi15186934-17 2024-09-13T14:26:44.499 INFO:tasks.rados.rados.0.smithi151.stdout:1448: finishing write tid 2 to smithi15186934-17 2024-09-13T14:26:44.502 INFO:tasks.rados.rados.0.smithi151.stdout:1448: finishing write tid 3 to smithi15186934-17 2024-09-13T14:26:44.502 INFO:tasks.rados.rados.0.smithi151.stdout:1448: finishing write tid 4 to smithi15186934-17 2024-09-13T14:26:44.502 INFO:tasks.rados.rados.0.smithi151.stdout:1448: finishing write tid 5 to smithi15186934-17 2024-09-13T14:26:44.502 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 952 (ObjNum 568 snap 179 seq_num 568) dirty exists 2024-09-13T14:26:44.502 INFO:tasks.rados.rados.0.smithi151.stdout:1448: left oid 17 (ObjNum 568 snap 179 seq_num 568) 2024-09-13T14:26:44.504 INFO:tasks.rados.rados.0.smithi151.stdout:1448: done (1 left) 2024-09-13T14:26:44.504 INFO:tasks.rados.rados.0.smithi151.stdout:1449: done (0 left) 2024-09-13T14:26:44.505 INFO:tasks.rados.rados.0.smithi151.stdout:1450: snap_create 2024-09-13T14:26:44.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:26:44 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:26:44.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:44.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:44 smithi151 ceph-mon[104450]: pgmap v239: 89 pgs: 89 active+clean; 282 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 3.0 MiB/s wr, 26 op/s 2024-09-13T14:26:44.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:44 smithi151 ceph-mon[104450]: osdmap e415: 8 total, 8 up, 8 in 2024-09-13T14:26:44.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:44 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:26:44.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:44 smithi151 ceph-mon[104450]: osdmap e416: 8 total, 8 up, 8 in 2024-09-13T14:26:44.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:44 smithi151 ceph-mon[108966]: pgmap v239: 89 pgs: 89 active+clean; 282 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 3.0 MiB/s wr, 26 op/s 2024-09-13T14:26:44.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:44 smithi151 ceph-mon[108966]: osdmap e415: 8 total, 8 up, 8 in 2024-09-13T14:26:44.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:44 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:26:44.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:44 smithi151 ceph-mon[108966]: osdmap e416: 8 total, 8 up, 8 in 2024-09-13T14:26:44.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:26:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:26:44] "GET /metrics HTTP/1.1" 200 36406 "" "Prometheus/2.51.0" 2024-09-13T14:26:44.878 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:44 smithi184 ceph-mon[33394]: pgmap v239: 89 pgs: 89 active+clean; 282 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 3.0 MiB/s wr, 26 op/s 2024-09-13T14:26:44.879 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:44 smithi184 ceph-mon[33394]: osdmap e415: 8 total, 8 up, 8 in 2024-09-13T14:26:44.879 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:44 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:26:44.879 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:44 smithi184 ceph-mon[33394]: osdmap e416: 8 total, 8 up, 8 in 2024-09-13T14:26:45.481 INFO:tasks.rados.rados.0.smithi151.stdout:1450: done (0 left) 2024-09-13T14:26:45.481 INFO:tasks.rados.rados.0.smithi151.stdout:1451: snap_create 2024-09-13T14:26:45.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch upgrade resume' 2024-09-13T14:26:45.986 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:46.103 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:26:45 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:26:45.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:46.103 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:26:45 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:26:45.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:46.483 INFO:tasks.rados.rados.0.smithi151.stdout:1451: done (0 left) 2024-09-13T14:26:46.483 INFO:tasks.rados.rados.0.smithi151.stdout:1452: rollback oid 17 current snap is 181 2024-09-13T14:26:46.483 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 17 to 177 2024-09-13T14:26:46.483 INFO:tasks.rados.rados.0.smithi151.stdout:1453: read oid 29 snap -1 2024-09-13T14:26:46.483 INFO:tasks.rados.rados.0.smithi151.stdout:1453: expect (ObjNum 565 snap 177 seq_num 565) 2024-09-13T14:26:46.483 INFO:tasks.rados.rados.0.smithi151.stdout:1454: snap_create 2024-09-13T14:26:46.487 INFO:tasks.rados.rados.0.smithi151.stdout:1452: finishing rollback tid 0 to smithi15186934-17 2024-09-13T14:26:46.487 INFO:tasks.rados.rados.0.smithi151.stdout:1452: finishing rollback tid 1 to smithi15186934-17 2024-09-13T14:26:46.488 INFO:tasks.rados.rados.0.smithi151.stdout:1452: finishing rollback tid 2 to smithi15186934-17 2024-09-13T14:26:46.488 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 956 (ObjNum 537 snap 173 seq_num 537) dirty exists 2024-09-13T14:26:46.488 INFO:tasks.rados.rados.0.smithi151.stdout:1453: expect (ObjNum 565 snap 177 seq_num 565) 2024-09-13T14:26:46.763 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:46 smithi184 ceph-mon[33394]: pgmap v242: 89 pgs: 89 active+clean; 282 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 3.0 MiB/s wr, 26 op/s 2024-09-13T14:26:46.763 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:46 smithi184 ceph-mon[33394]: osdmap e417: 8 total, 8 up, 8 in 2024-09-13T14:26:46.763 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:46 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:26:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:46 smithi151 ceph-mon[104450]: pgmap v242: 89 pgs: 89 active+clean; 282 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 3.0 MiB/s wr, 26 op/s 2024-09-13T14:26:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:46 smithi151 ceph-mon[104450]: osdmap e417: 8 total, 8 up, 8 in 2024-09-13T14:26:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:26:46.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:46 smithi151 ceph-mon[108966]: pgmap v242: 89 pgs: 89 active+clean; 282 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 3.0 MiB/s wr, 26 op/s 2024-09-13T14:26:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:46 smithi151 ceph-mon[108966]: osdmap e417: 8 total, 8 up, 8 in 2024-09-13T14:26:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:26:47.103 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:26:46 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:26:46.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi184\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:47.492 INFO:tasks.rados.rados.0.smithi151.stdout:1452: done (2 left) 2024-09-13T14:26:47.492 INFO:tasks.rados.rados.0.smithi151.stdout:1453: done (1 left) 2024-09-13T14:26:47.492 INFO:tasks.rados.rados.0.smithi151.stdout:1454: done (0 left) 2024-09-13T14:26:47.492 INFO:tasks.rados.rados.0.smithi151.stdout:1455: delete oid 6 current snap is 182 2024-09-13T14:26:47.495 INFO:tasks.rados.rados.0.smithi151.stdout:1455: done (0 left) 2024-09-13T14:26:47.496 INFO:tasks.rados.rados.0.smithi151.stdout:1456: rollback oid 25 current snap is 182 2024-09-13T14:26:47.496 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 25 to 163 2024-09-13T14:26:47.496 INFO:tasks.rados.rados.0.smithi151.stdout:1457: read oid 45 snap -1 2024-09-13T14:26:47.496 INFO:tasks.rados.rados.0.smithi151.stdout:1457: expect deleted 2024-09-13T14:26:47.496 INFO:tasks.rados.rados.0.smithi151.stdout:1458: read oid 21 snap -1 2024-09-13T14:26:47.496 INFO:tasks.rados.rados.0.smithi151.stdout:1458: expect (ObjNum 527 snap 170 seq_num 527) 2024-09-13T14:26:47.496 INFO:tasks.rados.rados.0.smithi151.stdout:1459: read oid 46 snap -1 2024-09-13T14:26:47.496 INFO:tasks.rados.rados.0.smithi151.stdout:1459: expect deleted 2024-09-13T14:26:47.496 INFO:tasks.rados.rados.0.smithi151.stdout:1460: rollback oid 40 current snap is 182 2024-09-13T14:26:47.496 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 40 to 166 2024-09-13T14:26:47.496 INFO:tasks.rados.rados.0.smithi151.stdout:1461: snap_remove snap 172 2024-09-13T14:26:47.599 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:47 smithi151 ceph-mon[104450]: osdmap e418: 8 total, 8 up, 8 in 2024-09-13T14:26:47.600 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:47 smithi151 ceph-mon[108966]: osdmap e418: 8 total, 8 up, 8 in 2024-09-13T14:26:47.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:47 smithi184 ceph-mon[33394]: osdmap e418: 8 total, 8 up, 8 in 2024-09-13T14:26:48.469 INFO:teuthology.orchestra.run.smithi151.stdout:Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:26:48.503 INFO:tasks.rados.rados.0.smithi151.stdout:1461: done (5 left) 2024-09-13T14:26:48.503 INFO:tasks.rados.rados.0.smithi151.stdout:1462: read oid 49 snap -1 2024-09-13T14:26:48.503 INFO:tasks.rados.rados.0.smithi151.stdout:1462: expect (ObjNum 567 snap 177 seq_num 567) 2024-09-13T14:26:48.503 INFO:tasks.rados.rados.0.smithi151.stdout:1463: snap_remove snap 179 2024-09-13T14:26:48.739 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:48 smithi151 ceph-mon[104450]: pgmap v245: 89 pgs: 89 active+clean; 285 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.5 MiB/s rd, 350 KiB/s wr, 4 op/s 2024-09-13T14:26:48.740 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:48 smithi151 ceph-mon[104450]: osdmap e419: 8 total, 8 up, 8 in 2024-09-13T14:26:48.740 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:48 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:26:48.740 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:48 smithi151 ceph-mon[108966]: pgmap v245: 89 pgs: 89 active+clean; 285 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.5 MiB/s rd, 350 KiB/s wr, 4 op/s 2024-09-13T14:26:48.740 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:48 smithi151 ceph-mon[108966]: osdmap e419: 8 total, 8 up, 8 in 2024-09-13T14:26:48.741 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:48 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:26:48.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:48 smithi184 ceph-mon[33394]: pgmap v245: 89 pgs: 89 active+clean; 285 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.5 MiB/s rd, 350 KiB/s wr, 4 op/s 2024-09-13T14:26:48.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:48 smithi184 ceph-mon[33394]: osdmap e419: 8 total, 8 up, 8 in 2024-09-13T14:26:48.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:48 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:26:49.481 INFO:tasks.rados.rados.0.smithi151.stdout:1463: done (6 left) 2024-09-13T14:26:49.481 INFO:tasks.rados.rados.0.smithi151.stdout:1464: rollback oid 27 current snap is 182 2024-09-13T14:26:49.481 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 27 to 178 2024-09-13T14:26:49.481 INFO:tasks.rados.rados.0.smithi151.stdout:1460: finishing rollback tid 1 to smithi15186934-40 2024-09-13T14:26:49.481 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 40 v 0 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-13T14:26:49.481 INFO:tasks.rados.rados.0.smithi151.stdout:1456: finishing rollback tid 0 to smithi15186934-25 2024-09-13T14:26:49.481 INFO:tasks.rados.rados.0.smithi151.stdout:1456: finishing rollback tid 1 to smithi15186934-25 2024-09-13T14:26:49.481 INFO:tasks.rados.rados.0.smithi151.stdout:1456: finishing rollback tid 2 to smithi15186934-25 2024-09-13T14:26:49.481 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 852 (ObjNum 494 snap 159 seq_num 494) dirty exists 2024-09-13T14:26:49.481 INFO:tasks.rados.rados.0.smithi151.stdout:1458: expect (ObjNum 527 snap 170 seq_num 527) 2024-09-13T14:26:49.613 INFO:tasks.rados.rados.0.smithi151.stdout:1462: expect (ObjNum 567 snap 177 seq_num 567) 2024-09-13T14:26:49.762 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:49 smithi151 ceph-mon[104450]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:49.762 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:49 smithi151 ceph-mon[104450]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:26:49.762 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:49 smithi151 ceph-mon[104450]: osdmap e420: 8 total, 8 up, 8 in 2024-09-13T14:26:49.762 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:49 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:26:49.762 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:49 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:26:49.763 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:49 smithi151 ceph-mon[104450]: osdmap e421: 8 total, 8 up, 8 in 2024-09-13T14:26:49.763 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:49 smithi151 ceph-mon[108966]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:49.763 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:49 smithi151 ceph-mon[108966]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:26:49.763 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:49 smithi151 ceph-mon[108966]: osdmap e420: 8 total, 8 up, 8 in 2024-09-13T14:26:49.763 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:49 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:26:49.763 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:49 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:26:49.763 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:49 smithi151 ceph-mon[108966]: osdmap e421: 8 total, 8 up, 8 in 2024-09-13T14:26:49.785 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'echo wait for all mons to upgrade' 2024-09-13T14:26:49.817 INFO:tasks.rados.rados.0.smithi151.stdout:1464: finishing rollback tid 0 to smithi15186934-27 2024-09-13T14:26:49.817 INFO:tasks.rados.rados.0.smithi151.stdout:1464: finishing rollback tid 1 to smithi15186934-27 2024-09-13T14:26:49.818 INFO:tasks.rados.rados.0.smithi151.stdout:1464: finishing rollback tid 2 to smithi15186934-27 2024-09-13T14:26:49.818 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 27 v 383 (ObjNum 559 snap 177 seq_num 559) dirty exists 2024-09-13T14:26:49.818 INFO:tasks.rados.rados.0.smithi151.stdout:1456: done (6 left) 2024-09-13T14:26:49.818 INFO:tasks.rados.rados.0.smithi151.stdout:1457: done (5 left) 2024-09-13T14:26:49.818 INFO:tasks.rados.rados.0.smithi151.stdout:1458: done (4 left) 2024-09-13T14:26:49.818 INFO:tasks.rados.rados.0.smithi151.stdout:1459: done (3 left) 2024-09-13T14:26:49.819 INFO:tasks.rados.rados.0.smithi151.stdout:1460: done (2 left) 2024-09-13T14:26:49.819 INFO:tasks.rados.rados.0.smithi151.stdout:1462: done (1 left) 2024-09-13T14:26:49.819 INFO:tasks.rados.rados.0.smithi151.stdout:1464: done (0 left) 2024-09-13T14:26:49.819 INFO:tasks.rados.rados.0.smithi151.stdout:1465: read oid 18 snap -1 2024-09-13T14:26:49.819 INFO:tasks.rados.rados.0.smithi151.stdout:1465: expect (ObjNum 491 snap 158 seq_num 491) 2024-09-13T14:26:49.819 INFO:tasks.rados.rados.0.smithi151.stdout:1466: snap_remove snap 129 2024-09-13T14:26:49.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:49 smithi184 ceph-mon[33394]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:49.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:49 smithi184 ceph-mon[33394]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:26:49.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:49 smithi184 ceph-mon[33394]: osdmap e420: 8 total, 8 up, 8 in 2024-09-13T14:26:49.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:49 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:26:49.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:49 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:26:49.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:49 smithi184 ceph-mon[33394]: osdmap e421: 8 total, 8 up, 8 in 2024-09-13T14:26:50.522 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:50.544 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:26:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:26:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:50.553 INFO:tasks.rados.rados.0.smithi151.stdout:1466: done (1 left) 2024-09-13T14:26:50.553 INFO:tasks.rados.rados.0.smithi151.stdout:1467: delete oid 31 current snap is 182 2024-09-13T14:26:50.553 INFO:tasks.rados.rados.0.smithi151.stdout:1465: expect (ObjNum 491 snap 158 seq_num 491) 2024-09-13T14:26:50.698 INFO:tasks.rados.rados.0.smithi151.stdout:1465: done (1 left) 2024-09-13T14:26:50.699 INFO:tasks.rados.rados.0.smithi151.stdout:1467: done (0 left) 2024-09-13T14:26:50.699 INFO:tasks.rados.rados.0.smithi151.stdout:1468: write oid 14 current snap is 182 2024-09-13T14:26:50.699 INFO:tasks.rados.rados.0.smithi151.stdout:1468: seq_num 572 ranges {413604=568903,1488759=656862,2624893=591709} 2024-09-13T14:26:50.704 INFO:tasks.rados.rados.0.smithi151.stdout:1468: writing smithi15186934-14 from 413604 to 982507 tid 1 2024-09-13T14:26:50.710 INFO:tasks.rados.rados.0.smithi151.stdout:1468: writing smithi15186934-14 from 1488759 to 2145621 tid 2 2024-09-13T14:26:50.716 INFO:tasks.rados.rados.0.smithi151.stdout:1468: writing smithi15186934-14 from 2624893 to 3216602 tid 3 2024-09-13T14:26:50.716 INFO:tasks.rados.rados.0.smithi151.stdout:1469: read oid 30 snap -1 2024-09-13T14:26:50.717 INFO:tasks.rados.rados.0.smithi151.stdout:1469: expect (ObjNum 507 snap 164 seq_num 507) 2024-09-13T14:26:50.717 INFO:tasks.rados.rados.0.smithi151.stdout:1470: snap_create 2024-09-13T14:26:50.717 INFO:tasks.rados.rados.0.smithi151.stdout:1468: finishing write tid 1 to smithi15186934-14 2024-09-13T14:26:50.717 INFO:tasks.rados.rados.0.smithi151.stdout:1468: finishing write tid 2 to smithi15186934-14 2024-09-13T14:26:50.719 INFO:tasks.rados.rados.0.smithi151.stdout:1468: finishing write tid 3 to smithi15186934-14 2024-09-13T14:26:50.719 INFO:tasks.rados.rados.0.smithi151.stdout:1468: finishing write tid 4 to smithi15186934-14 2024-09-13T14:26:50.720 INFO:tasks.rados.rados.0.smithi151.stdout:1469: expect (ObjNum 507 snap 164 seq_num 507) 2024-09-13T14:26:50.803 INFO:tasks.rados.rados.0.smithi151.stdout:1468: finishing write tid 5 to smithi15186934-14 2024-09-13T14:26:50.803 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 608 (ObjNum 572 snap 182 seq_num 572) dirty exists 2024-09-13T14:26:50.803 INFO:tasks.rados.rados.0.smithi151.stdout:1468: left oid 14 (ObjNum 572 snap 182 seq_num 572) 2024-09-13T14:26:50.810 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:50 smithi151 ceph-mon[108966]: pgmap v248: 89 pgs: 89 active+clean; 285 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.5 MiB/s rd, 350 KiB/s wr, 4 op/s 2024-09-13T14:26:50.810 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:50 smithi151 ceph-mon[104450]: pgmap v248: 89 pgs: 89 active+clean; 285 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.5 MiB/s rd, 350 KiB/s wr, 4 op/s 2024-09-13T14:26:50.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:50 smithi184 ceph-mon[33394]: pgmap v248: 89 pgs: 89 active+clean; 285 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.5 MiB/s rd, 350 KiB/s wr, 4 op/s 2024-09-13T14:26:51.562 INFO:tasks.rados.rados.0.smithi151.stdout:1468: done (2 left) 2024-09-13T14:26:51.562 INFO:tasks.rados.rados.0.smithi151.stdout:1469: done (1 left) 2024-09-13T14:26:51.562 INFO:tasks.rados.rados.0.smithi151.stdout:1470: done (0 left) 2024-09-13T14:26:51.562 INFO:tasks.rados.rados.0.smithi151.stdout:1471: write oid 32 current snap is 183 2024-09-13T14:26:51.562 INFO:tasks.rados.rados.0.smithi151.stdout:1471: seq_num 573 ranges {430411=644152,1830888=400520,2935963=402981} 2024-09-13T14:26:51.567 INFO:tasks.rados.rados.0.smithi151.stdout:1471: writing smithi15186934-32 from 430411 to 1074563 tid 1 2024-09-13T14:26:51.573 INFO:tasks.rados.rados.0.smithi151.stdout:1471: writing smithi15186934-32 from 1830888 to 2231408 tid 2 2024-09-13T14:26:51.580 INFO:tasks.rados.rados.0.smithi151.stdout:1471: writing smithi15186934-32 from 2935963 to 3338944 tid 3 2024-09-13T14:26:51.580 INFO:tasks.rados.rados.0.smithi151.stdout:1472: snap_create 2024-09-13T14:26:51.580 INFO:tasks.rados.rados.0.smithi151.stdout:1471: finishing write tid 1 to smithi15186934-32 2024-09-13T14:26:51.580 INFO:tasks.rados.rados.0.smithi151.stdout:1471: finishing write tid 2 to smithi15186934-32 2024-09-13T14:26:51.583 INFO:tasks.rados.rados.0.smithi151.stdout:1471: finishing write tid 3 to smithi15186934-32 2024-09-13T14:26:51.583 INFO:tasks.rados.rados.0.smithi151.stdout:1471: finishing write tid 4 to smithi15186934-32 2024-09-13T14:26:51.583 INFO:tasks.rados.rados.0.smithi151.stdout:1471: finishing write tid 5 to smithi15186934-32 2024-09-13T14:26:51.583 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 32 v 612 (ObjNum 573 snap 183 seq_num 573) dirty exists 2024-09-13T14:26:51.583 INFO:tasks.rados.rados.0.smithi151.stdout:1471: left oid 32 (ObjNum 573 snap 183 seq_num 573) 2024-09-13T14:26:51.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:51 smithi184 ceph-mon[33394]: osdmap e422: 8 total, 8 up, 8 in 2024-09-13T14:26:51.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:51 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:26:51.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:51 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:26:51.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:51 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:26:51.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:51 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:26:51.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:51 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:26:51.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:51 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:26:51.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:51 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:26:51.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:51 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-13T14:26:51.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:51 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["b"]} : dispatch 2024-09-13T14:26:51.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:51 smithi151 ceph-mon[104450]: osdmap e422: 8 total, 8 up, 8 in 2024-09-13T14:26:51.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:51 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:26:51.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:51 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:26:51.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:51 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:26:51.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:51 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:26:51.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:51 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:26:51.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:51 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:26:51.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:51 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:26:51.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:51 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-13T14:26:51.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:51 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["b"]} : dispatch 2024-09-13T14:26:51.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:51 smithi151 ceph-mon[108966]: osdmap e422: 8 total, 8 up, 8 in 2024-09-13T14:26:51.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:51 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:26:51.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:51 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:26:51.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:51 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:26:51.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:51 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:26:51.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:51 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:26:51.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:51 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:26:51.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:51 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:26:51.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:51 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-13T14:26:51.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:51 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["b"]} : dispatch 2024-09-13T14:26:52.434 INFO:teuthology.orchestra.run.smithi151.stdout:wait for all mons to upgrade 2024-09-13T14:26:52.571 INFO:tasks.rados.rados.0.smithi151.stdout:1471: done (1 left) 2024-09-13T14:26:52.571 INFO:tasks.rados.rados.0.smithi151.stdout:1472: done (0 left) 2024-09-13T14:26:52.571 INFO:tasks.rados.rados.0.smithi151.stdout:1473: write oid 12 current snap is 184 2024-09-13T14:26:52.571 INFO:tasks.rados.rados.0.smithi151.stdout:1473: seq_num 574 ranges {447218=719401,1889370=660531,2963386=614253} 2024-09-13T14:26:52.577 INFO:tasks.rados.rados.0.smithi151.stdout:1473: writing smithi15186934-12 from 447218 to 1166619 tid 1 2024-09-13T14:26:52.585 INFO:tasks.rados.rados.0.smithi151.stdout:1473: writing smithi15186934-12 from 1889370 to 2549901 tid 2 2024-09-13T14:26:52.591 INFO:tasks.rados.rados.0.smithi151.stdout:1473: writing smithi15186934-12 from 2963386 to 3577639 tid 3 2024-09-13T14:26:52.591 INFO:tasks.rados.rados.0.smithi151.stdout:1474: snap_create 2024-09-13T14:26:52.591 INFO:tasks.rados.rados.0.smithi151.stdout:1473: finishing write tid 1 to smithi15186934-12 2024-09-13T14:26:52.591 INFO:tasks.rados.rados.0.smithi151.stdout:1473: finishing write tid 2 to smithi15186934-12 2024-09-13T14:26:52.593 INFO:tasks.rados.rados.0.smithi151.stdout:1473: finishing write tid 3 to smithi15186934-12 2024-09-13T14:26:52.593 INFO:tasks.rados.rados.0.smithi151.stdout:1473: finishing write tid 4 to smithi15186934-12 2024-09-13T14:26:52.593 INFO:tasks.rados.rados.0.smithi151.stdout:1473: finishing write tid 5 to smithi15186934-12 2024-09-13T14:26:52.593 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 12 v 345 (ObjNum 574 snap 184 seq_num 574) dirty exists 2024-09-13T14:26:52.594 INFO:tasks.rados.rados.0.smithi151.stdout:1473: left oid 12 (ObjNum 574 snap 184 seq_num 574) 2024-09-13T14:26:52.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:52 smithi184 ceph-mon[33394]: pgmap v251: 89 pgs: 89 active+clean; 285 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:26:52.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:52 smithi184 ceph-mon[33394]: Upgrade: It appears safe to stop mon.b 2024-09-13T14:26:52.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:52 smithi184 ceph-mon[33394]: osdmap e423: 8 total, 8 up, 8 in 2024-09-13T14:26:52.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:52 smithi151 ceph-mon[104450]: pgmap v251: 89 pgs: 89 active+clean; 285 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:26:52.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:52 smithi151 ceph-mon[104450]: Upgrade: It appears safe to stop mon.b 2024-09-13T14:26:52.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:52 smithi151 ceph-mon[104450]: osdmap e423: 8 total, 8 up, 8 in 2024-09-13T14:26:52.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:52 smithi151 ceph-mon[108966]: pgmap v251: 89 pgs: 89 active+clean; 285 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:26:52.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:52 smithi151 ceph-mon[108966]: Upgrade: It appears safe to stop mon.b 2024-09-13T14:26:52.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:52 smithi151 ceph-mon[108966]: osdmap e423: 8 total, 8 up, 8 in 2024-09-13T14:26:53.084 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'while ! ceph mon versions | grep $sha1 | grep '"'"': 3'"'"' && ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error; do sleep 2 ; done' 2024-09-13T14:26:53.330 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:53.578 INFO:tasks.rados.rados.0.smithi151.stdout:1473: done (1 left) 2024-09-13T14:26:53.578 INFO:tasks.rados.rados.0.smithi151.stdout:1474: done (0 left) 2024-09-13T14:26:53.578 INFO:tasks.rados.rados.0.smithi151.stdout:1475: read oid 36 snap -1 2024-09-13T14:26:53.578 INFO:tasks.rados.rados.0.smithi151.stdout:1475: expect deleted 2024-09-13T14:26:53.578 INFO:tasks.rados.rados.0.smithi151.stdout:1476: write oid 10 current snap is 185 2024-09-13T14:26:53.578 INFO:tasks.rados.rados.0.smithi151.stdout:1476: seq_num 575 ranges {464025=794650,1947852=404189,2990809=425525} 2024-09-13T14:26:53.586 INFO:tasks.rados.rados.0.smithi151.stdout:1476: writing smithi15186934-10 from 464025 to 1258675 tid 1 2024-09-13T14:26:53.592 INFO:tasks.rados.rados.0.smithi151.stdout:1476: writing smithi15186934-10 from 1947852 to 2352041 tid 2 2024-09-13T14:26:53.600 INFO:tasks.rados.rados.0.smithi151.stdout:1476: writing smithi15186934-10 from 2990809 to 3416334 tid 3 2024-09-13T14:26:53.600 INFO:tasks.rados.rados.0.smithi151.stdout:1477: rollback oid 48 current snap is 185 2024-09-13T14:26:53.600 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 48 to 171 2024-09-13T14:26:53.600 INFO:tasks.rados.rados.0.smithi151.stdout:1476: finishing write tid 1 to smithi15186934-10 2024-09-13T14:26:53.600 INFO:tasks.rados.rados.0.smithi151.stdout:1476: finishing write tid 2 to smithi15186934-10 2024-09-13T14:26:53.600 INFO:tasks.rados.rados.0.smithi151.stdout:1475: done (2 left) 2024-09-13T14:26:53.600 INFO:tasks.rados.rados.0.smithi151.stdout:1478: snap_remove snap 180 2024-09-13T14:26:53.784 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:53 smithi151 ceph-mon[104450]: osdmap e424: 8 total, 8 up, 8 in 2024-09-13T14:26:53.784 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:53 smithi151 ceph-mon[104450]: Upgrade: Updating mon.b 2024-09-13T14:26:53.784 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:53 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:26:53.784 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:53 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:26:53.784 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:53 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:26:53.784 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:53 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:26:53.784 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:53 smithi151 ceph-mon[104450]: Deploying daemon mon.b on smithi184 2024-09-13T14:26:53.785 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:53 smithi151 ceph-mon[108966]: osdmap e424: 8 total, 8 up, 8 in 2024-09-13T14:26:53.785 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:53 smithi151 ceph-mon[108966]: Upgrade: Updating mon.b 2024-09-13T14:26:53.785 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:53 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:26:53.785 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:53 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:26:53.785 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:53 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:26:53.785 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:53 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:26:53.785 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:53 smithi151 ceph-mon[108966]: Deploying daemon mon.b on smithi184 2024-09-13T14:26:53.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:53 smithi184 ceph-mon[33394]: osdmap e424: 8 total, 8 up, 8 in 2024-09-13T14:26:53.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:53 smithi184 ceph-mon[33394]: Upgrade: Updating mon.b 2024-09-13T14:26:53.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:53 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:26:53.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:53 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:26:53.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:53 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:26:53.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:53 smithi184 ceph-mon[33394]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:26:53.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:53 smithi184 ceph-mon[33394]: Deploying daemon mon.b on smithi184 2024-09-13T14:26:54.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:26:54 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:26:54.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:54.486 INFO:tasks.rados.rados.0.smithi151.stdout:1476: finishing write tid 3 to smithi15186934-10 2024-09-13T14:26:54.486 INFO:tasks.rados.rados.0.smithi151.stdout:1476: finishing write tid 4 to smithi15186934-10 2024-09-13T14:26:54.486 INFO:tasks.rados.rados.0.smithi151.stdout:1476: finishing write tid 5 to smithi15186934-10 2024-09-13T14:26:54.486 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 576 (ObjNum 575 snap 185 seq_num 575) dirty exists 2024-09-13T14:26:54.486 INFO:tasks.rados.rados.0.smithi151.stdout:1476: left oid 10 (ObjNum 575 snap 185 seq_num 575) 2024-09-13T14:26:54.487 INFO:tasks.rados.rados.0.smithi151.stdout:1477: finishing rollback tid 1 to smithi15186934-48 2024-09-13T14:26:54.487 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 48 v 792 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-13T14:26:54.487 INFO:tasks.rados.rados.0.smithi151.stdout:1476: done (2 left) 2024-09-13T14:26:54.487 INFO:tasks.rados.rados.0.smithi151.stdout:1477: done (1 left) 2024-09-13T14:26:54.487 INFO:tasks.rados.rados.0.smithi151.stdout:1478: done (0 left) 2024-09-13T14:26:54.487 INFO:tasks.rados.rados.0.smithi151.stdout:1479: snap_remove snap 183 2024-09-13T14:26:54.751 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:54 smithi184 ceph-mon[33394]: pgmap v254: 89 pgs: 89 active+clean; 279 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.7 MiB/s rd, 798 KiB/s wr, 12 op/s 2024-09-13T14:26:54.751 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:54 smithi184 ceph-mon[33394]: osdmap e425: 8 total, 8 up, 8 in 2024-09-13T14:26:54.751 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:54 smithi184 ceph-mon[33394]: osdmap e426: 8 total, 8 up, 8 in 2024-09-13T14:26:54.823 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:54 smithi151 ceph-mon[104450]: pgmap v254: 89 pgs: 89 active+clean; 279 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.7 MiB/s rd, 798 KiB/s wr, 12 op/s 2024-09-13T14:26:54.823 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:54 smithi151 ceph-mon[104450]: osdmap e425: 8 total, 8 up, 8 in 2024-09-13T14:26:54.823 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:54 smithi151 ceph-mon[104450]: osdmap e426: 8 total, 8 up, 8 in 2024-09-13T14:26:54.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:54 smithi151 ceph-mon[108966]: pgmap v254: 89 pgs: 89 active+clean; 279 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.7 MiB/s rd, 798 KiB/s wr, 12 op/s 2024-09-13T14:26:54.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:54 smithi151 ceph-mon[108966]: osdmap e425: 8 total, 8 up, 8 in 2024-09-13T14:26:54.824 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:54 smithi151 ceph-mon[108966]: osdmap e426: 8 total, 8 up, 8 in 2024-09-13T14:26:54.824 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:26:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:26:54] "GET /metrics HTTP/1.1" 200 36411 "" "Prometheus/2.51.0" 2024-09-13T14:26:55.036 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:54 smithi184 systemd[1]: Stopping Ceph mon.b for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:26:55.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:55 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-b[33371]: 2024-09-13T14:26:55.035+0000 7f061e2c5640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:26:55.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:55 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-b[33371]: 2024-09-13T14:26:55.035+0000 7f061e2c5640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-13T14:26:55.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:55 smithi184 podman[92340]: 2024-09-13 14:26:55.107227338 +0000 UTC m=+0.214112083 container died 8c637f147650a7fd7c59e2e5c07bb367a05134b72133dcc42f392473f12e0316 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=quincy-bbdadfa, ceph=True) 2024-09-13T14:26:55.599 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:26:55.745 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:55 smithi184 podman[92340]: 2024-09-13 14:26:55.566773349 +0000 UTC m=+0.673658095 container cleanup 8c637f147650a7fd7c59e2e5c07bb367a05134b72133dcc42f392473f12e0316 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=quincy-bbdadfa, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:26:55.745 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:55 smithi184 bash[92340]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-b 2024-09-13T14:26:56.001 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:26:55 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:26:55.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:56.001 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:26:55 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:26:55.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:56.001 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:55 smithi184 podman[92358]: 2024-09-13 14:26:55.85998512 +0000 UTC m=+0.746146702 container remove 8c637f147650a7fd7c59e2e5c07bb367a05134b72133dcc42f392473f12e0316 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-b, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=quincy-bbdadfa, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-13T14:26:56.001 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:55 smithi184 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@mon.b.service: Deactivated successfully. 2024-09-13T14:26:56.002 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:55 smithi184 systemd[1]: Stopped Ceph mon.b for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:26:56.002 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:55 smithi184 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@mon.b.service: Consumed 10.655s CPU time. 2024-09-13T14:26:56.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 systemd[1]: Starting Ceph mon.b for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:26:56.637 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 podman[92489]: 2024-09-13 14:26:56.288859104 +0000 UTC m=+0.015612368 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:26:56.637 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 podman[92489]: 2024-09-13 14:26:56.410932622 +0000 UTC m=+0.137685874 container create 12b6c0ccb635e1e1f30dc0613122cee5a4dd58d8baaa259c2746a4305c361546 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-b, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-13T14:26:56.637 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 podman[92489]: 2024-09-13 14:26:56.604993549 +0000 UTC m=+0.331746809 container init 12b6c0ccb635e1e1f30dc0613122cee5a4dd58d8baaa259c2746a4305c361546 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-13T14:26:56.638 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 podman[92489]: 2024-09-13 14:26:56.610032147 +0000 UTC m=+0.336785401 container start 12b6c0ccb635e1e1f30dc0613122cee5a4dd58d8baaa259c2746a4305c361546 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-b, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:26:56.638 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: set uid:gid to 167:167 (ceph:ceph) 2024-09-13T14:26:56.638 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev), process ceph-mon, pid 7 2024-09-13T14:26:56.638 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: pidfile_write: ignore empty --pid-file 2024-09-13T14:26:56.638 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: load: jerasure load: lrc 2024-09-13T14:26:56.638 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: RocksDB version: 7.9.2 2024-09-13T14:26:56.638 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Git sha 0 2024-09-13T14:26:56.638 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Compile date 2024-09-12 20:46:14 2024-09-13T14:26:56.638 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: DB SUMMARY 2024-09-13T14:26:56.638 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: DB Session ID: YN3DUXNRG4XL26VNQZIW 2024-09-13T14:26:56.638 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: CURRENT file: CURRENT 2024-09-13T14:26:56.638 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: IDENTITY file: IDENTITY 2024-09-13T14:26:56.639 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1087 Bytes 2024-09-13T14:26:56.639 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000027.sst 2024-09-13T14:26:56.639 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000025.log size: 1552116 ; 2024-09-13T14:26:56.639 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.error_if_exists: 0 2024-09-13T14:26:56.639 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.create_if_missing: 0 2024-09-13T14:26:56.639 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.paranoid_checks: 1 2024-09-13T14:26:56.639 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-13T14:26:56.639 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-13T14:26:56.639 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-13T14:26:56.639 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.env: 0x55e59039d160 2024-09-13T14:26:56.639 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.fs: PosixFileSystem 2024-09-13T14:26:56.639 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.info_log: 0x55e591e5b880 2024-09-13T14:26:56.639 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.max_file_opening_threads: 16 2024-09-13T14:26:56.640 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.statistics: (nil) 2024-09-13T14:26:56.640 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.use_fsync: 0 2024-09-13T14:26:56.640 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.max_log_file_size: 0 2024-09-13T14:26:56.640 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-13T14:26:56.640 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-13T14:26:56.640 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.keep_log_file_num: 1000 2024-09-13T14:26:56.640 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.recycle_log_file_num: 0 2024-09-13T14:26:56.640 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.allow_fallocate: 1 2024-09-13T14:26:56.640 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.allow_mmap_reads: 0 2024-09-13T14:26:56.640 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.allow_mmap_writes: 0 2024-09-13T14:26:56.640 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.use_direct_reads: 0 2024-09-13T14:26:56.640 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-13T14:26:56.640 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.create_missing_column_families: 0 2024-09-13T14:26:56.641 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.db_log_dir: 2024-09-13T14:26:56.641 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.wal_dir: 2024-09-13T14:26:56.641 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-13T14:26:56.641 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-13T14:26:56.641 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-13T14:26:56.641 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-13T14:26:56.641 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-13T14:26:56.641 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-13T14:26:56.641 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.advise_random_on_open: 1 2024-09-13T14:26:56.641 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.db_write_buffer_size: 0 2024-09-13T14:26:56.641 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.write_buffer_manager: 0x55e591e5e960 2024-09-13T14:26:56.641 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-13T14:26:56.641 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-13T14:26:56.642 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-13T14:26:56.642 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.rate_limiter: (nil) 2024-09-13T14:26:56.642 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-13T14:26:56.642 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.wal_recovery_mode: 2 2024-09-13T14:26:56.643 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.enable_thread_tracking: 0 2024-09-13T14:26:56.643 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.enable_pipelined_write: 0 2024-09-13T14:26:56.643 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.unordered_write: 0 2024-09-13T14:26:56.643 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-13T14:26:56.643 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-13T14:26:56.643 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-13T14:26:56.643 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-13T14:26:56.643 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.row_cache: None 2024-09-13T14:26:56.643 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.wal_filter: None 2024-09-13T14:26:56.643 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-13T14:26:56.643 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.allow_ingest_behind: 0 2024-09-13T14:26:56.643 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.two_write_queues: 0 2024-09-13T14:26:56.644 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.manual_wal_flush: 0 2024-09-13T14:26:56.644 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.wal_compression: 0 2024-09-13T14:26:56.644 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.atomic_flush: 0 2024-09-13T14:26:56.644 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-13T14:26:56.644 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-13T14:26:56.644 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-13T14:26:56.644 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.log_readahead_size: 0 2024-09-13T14:26:56.644 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-13T14:26:56.644 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.best_efforts_recovery: 0 2024-09-13T14:26:56.644 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-13T14:26:56.644 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-13T14:26:56.645 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.allow_data_in_errors: 0 2024-09-13T14:26:56.645 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.db_host_id: __hostname__ 2024-09-13T14:26:56.645 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-13T14:26:56.645 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.max_background_jobs: 2 2024-09-13T14:26:56.645 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.max_background_compactions: -1 2024-09-13T14:26:56.645 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.max_subcompactions: 1 2024-09-13T14:26:56.645 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-13T14:26:56.645 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-13T14:26:56.645 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-13T14:26:56.645 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.max_total_wal_size: 0 2024-09-13T14:26:56.645 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-13T14:26:56.645 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-13T14:26:56.645 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-13T14:26:56.645 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-13T14:26:56.646 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.max_open_files: -1 2024-09-13T14:26:56.646 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.bytes_per_sync: 0 2024-09-13T14:26:56.646 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-13T14:26:56.646 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-13T14:26:56.646 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.compaction_readahead_size: 0 2024-09-13T14:26:56.646 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.max_background_flushes: -1 2024-09-13T14:26:56.646 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Compression algorithms supported: 2024-09-13T14:26:56.646 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: kZSTD supported: 0 2024-09-13T14:26:56.646 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: kXpressCompression supported: 0 2024-09-13T14:26:56.646 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: kBZip2Compression supported: 0 2024-09-13T14:26:56.646 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-13T14:26:56.646 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: kLZ4Compression supported: 1 2024-09-13T14:26:56.646 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: kZlibCompression supported: 1 2024-09-13T14:26:56.647 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: kLZ4HCCompression supported: 1 2024-09-13T14:26:56.647 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: kSnappyCompression supported: 1 2024-09-13T14:26:56.647 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-13T14:26:56.647 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-13T14:26:56.647 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-09-13T14:26:56.647 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-13T14:26:56.647 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-13T14:26:56.647 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.merge_operator: 2024-09-13T14:26:56.647 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.compaction_filter: None 2024-09-13T14:26:56.647 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.compaction_filter_factory: None 2024-09-13T14:26:56.647 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.sst_partitioner_factory: None 2024-09-13T14:26:56.647 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-13T14:26:56.647 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-13T14:26:56.647 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e591e5b4e0) 2024-09-13T14:26:56.648 INFO:journalctl@ceph.mon.b.smithi184.stdout: cache_index_and_filter_blocks: 1 2024-09-13T14:26:56.648 INFO:journalctl@ceph.mon.b.smithi184.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-13T14:26:56.648 INFO:journalctl@ceph.mon.b.smithi184.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-13T14:26:56.648 INFO:journalctl@ceph.mon.b.smithi184.stdout: pin_top_level_index_and_filter: 1 2024-09-13T14:26:56.648 INFO:journalctl@ceph.mon.b.smithi184.stdout: index_type: 0 2024-09-13T14:26:56.648 INFO:journalctl@ceph.mon.b.smithi184.stdout: data_block_index_type: 0 2024-09-13T14:26:56.648 INFO:journalctl@ceph.mon.b.smithi184.stdout: index_shortening: 1 2024-09-13T14:26:56.648 INFO:journalctl@ceph.mon.b.smithi184.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-13T14:26:56.648 INFO:journalctl@ceph.mon.b.smithi184.stdout: checksum: 4 2024-09-13T14:26:56.648 INFO:journalctl@ceph.mon.b.smithi184.stdout: no_block_cache: 0 2024-09-13T14:26:56.648 INFO:journalctl@ceph.mon.b.smithi184.stdout: block_cache: 0x55e591e4e850 2024-09-13T14:26:56.648 INFO:journalctl@ceph.mon.b.smithi184.stdout: block_cache_name: BinnedLRUCache 2024-09-13T14:26:56.648 INFO:journalctl@ceph.mon.b.smithi184.stdout: block_cache_options: 2024-09-13T14:26:56.649 INFO:journalctl@ceph.mon.b.smithi184.stdout: capacity : 536870912 2024-09-13T14:26:56.649 INFO:journalctl@ceph.mon.b.smithi184.stdout: num_shard_bits : 4 2024-09-13T14:26:56.649 INFO:journalctl@ceph.mon.b.smithi184.stdout: strict_capacity_limit : 0 2024-09-13T14:26:56.649 INFO:journalctl@ceph.mon.b.smithi184.stdout: high_pri_pool_ratio: 0.000 2024-09-13T14:26:56.649 INFO:journalctl@ceph.mon.b.smithi184.stdout: block_cache_compressed: (nil) 2024-09-13T14:26:56.649 INFO:journalctl@ceph.mon.b.smithi184.stdout: persistent_cache: (nil) 2024-09-13T14:26:56.649 INFO:journalctl@ceph.mon.b.smithi184.stdout: block_size: 4096 2024-09-13T14:26:56.649 INFO:journalctl@ceph.mon.b.smithi184.stdout: block_size_deviation: 10 2024-09-13T14:26:56.649 INFO:journalctl@ceph.mon.b.smithi184.stdout: block_restart_interval: 16 2024-09-13T14:26:56.649 INFO:journalctl@ceph.mon.b.smithi184.stdout: index_block_restart_interval: 1 2024-09-13T14:26:56.649 INFO:journalctl@ceph.mon.b.smithi184.stdout: metadata_block_size: 4096 2024-09-13T14:26:56.649 INFO:journalctl@ceph.mon.b.smithi184.stdout: partition_filters: 0 2024-09-13T14:26:56.649 INFO:journalctl@ceph.mon.b.smithi184.stdout: use_delta_encoding: 1 2024-09-13T14:26:56.650 INFO:journalctl@ceph.mon.b.smithi184.stdout: filter_policy: bloomfilter 2024-09-13T14:26:56.650 INFO:journalctl@ceph.mon.b.smithi184.stdout: whole_key_filtering: 1 2024-09-13T14:26:56.650 INFO:journalctl@ceph.mon.b.smithi184.stdout: verify_compression: 0 2024-09-13T14:26:56.650 INFO:journalctl@ceph.mon.b.smithi184.stdout: read_amp_bytes_per_bit: 0 2024-09-13T14:26:56.650 INFO:journalctl@ceph.mon.b.smithi184.stdout: format_version: 5 2024-09-13T14:26:56.650 INFO:journalctl@ceph.mon.b.smithi184.stdout: enable_index_compression: 1 2024-09-13T14:26:56.650 INFO:journalctl@ceph.mon.b.smithi184.stdout: block_align: 0 2024-09-13T14:26:56.650 INFO:journalctl@ceph.mon.b.smithi184.stdout: max_auto_readahead_size: 262144 2024-09-13T14:26:56.650 INFO:journalctl@ceph.mon.b.smithi184.stdout: prepopulate_block_cache: 0 2024-09-13T14:26:56.650 INFO:journalctl@ceph.mon.b.smithi184.stdout: initial_auto_readahead_size: 8192 2024-09-13T14:26:56.650 INFO:journalctl@ceph.mon.b.smithi184.stdout: num_file_reads_for_auto_readahead: 2 2024-09-13T14:26:56.650 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.write_buffer_size: 33554432 2024-09-13T14:26:56.650 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.max_write_buffer_number: 2 2024-09-13T14:26:56.650 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.compression: NoCompression 2024-09-13T14:26:56.651 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.bottommost_compression: Disabled 2024-09-13T14:26:56.651 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.prefix_extractor: nullptr 2024-09-13T14:26:56.651 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-13T14:26:56.651 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.num_levels: 7 2024-09-13T14:26:56.651 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-13T14:26:56.651 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-13T14:26:56.651 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-13T14:26:56.651 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-13T14:26:56.651 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-13T14:26:56.651 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-13T14:26:56.651 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-13T14:26:56.651 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:26:56.651 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-13T14:26:56.652 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-13T14:26:56.652 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-13T14:26:56.652 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-13T14:26:56.652 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-13T14:26:56.652 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.compression_opts.level: 32767 2024-09-13T14:26:56.652 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.compression_opts.strategy: 0 2024-09-13T14:26:56.652 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-13T14:26:56.652 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:26:56.652 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-13T14:26:56.652 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-13T14:26:56.652 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.compression_opts.enabled: false 2024-09-13T14:26:56.652 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-13T14:26:56.652 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-13T14:26:56.652 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-13T14:26:56.653 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-13T14:26:56.653 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.target_file_size_base: 67108864 2024-09-13T14:26:56.653 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-13T14:26:56.653 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-13T14:26:56.653 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-13T14:26:56.653 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-13T14:26:56.653 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-13T14:26:56.653 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-13T14:26:56.653 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-13T14:26:56.653 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-13T14:26:56.653 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-13T14:26:56.653 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-13T14:26:56.653 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-13T14:26:56.653 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-13T14:26:56.654 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-13T14:26:56.950 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-13T14:26:56.950 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.arena_block_size: 1048576 2024-09-13T14:26:56.950 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-13T14:26:56.951 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-13T14:26:56.951 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.disable_auto_compactions: 0 2024-09-13T14:26:56.951 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-13T14:26:56.951 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-13T14:26:56.951 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-13T14:26:56.951 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-13T14:26:56.951 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-13T14:26:56.951 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-13T14:26:56.951 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-13T14:26:56.951 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-13T14:26:56.951 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-13T14:26:56.951 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-13T14:26:56.952 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-13T14:26:56.952 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.inplace_update_support: 0 2024-09-13T14:26:56.952 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-13T14:26:56.952 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-13T14:26:56.952 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-13T14:26:56.952 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-13T14:26:56.952 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.bloom_locality: 0 2024-09-13T14:26:56.952 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.max_successive_merges: 0 2024-09-13T14:26:56.952 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-13T14:26:56.952 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.paranoid_file_checks: 0 2024-09-13T14:26:56.952 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.force_consistency_checks: 1 2024-09-13T14:26:56.953 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.report_bg_io_stats: 0 2024-09-13T14:26:56.953 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.ttl: 2592000 2024-09-13T14:26:56.953 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-13T14:26:56.953 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-13T14:26:56.953 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-13T14:26:56.953 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.enable_blob_files: false 2024-09-13T14:26:56.953 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.min_blob_size: 0 2024-09-13T14:26:56.953 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.blob_file_size: 268435456 2024-09-13T14:26:56.953 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-13T14:26:56.953 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-13T14:26:56.953 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-13T14:26:56.953 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-13T14:26:56.953 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-13T14:26:56.954 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.blob_file_starting_level: 0 2024-09-13T14:26:56.954 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-13T14:26:56.954 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-09-13T14:26:56.954 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-13T14:26:56.954 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 19588, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-13T14:26:56.954 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-09-13T14:26:56.954 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7506422c-ec33-4603-9258-a9bb877c37aa 2024-09-13T14:26:56.954 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726237616636905, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-09-13T14:26:56.954 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-09-13T14:26:56.954 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726237616642292, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 1021048, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 19593, "largest_seqno": 20448, "table_properties": {"data_size": 1016266, "index_size": 2910, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 837, "raw_key_size": 7801, "raw_average_key_size": 23, "raw_value_size": 1008454, "raw_average_value_size": 3093, "num_data_blocks": 127, "num_entries": 326, "num_filter_entries": 326, "num_deletions": 7, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726237616, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7506422c-ec33-4603-9258-a9bb877c37aa", "db_session_id": "YN3DUXNRG4XL26VNQZIW", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-09-13T14:26:56.954 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726237616642401, "job": 1, "event": "recovery_finished"} 2024-09-13T14:26:56.954 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-09-13T14:26:56.955 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-13T14:26:56.955 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-13T14:26:56.955 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e591e7ee00 2024-09-13T14:26:56.955 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: DB pointer 0x55e591f7e000 2024-09-13T14:26:56.955 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-13T14:26:56.955 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-13T14:26:56.955 INFO:journalctl@ceph.mon.b.smithi184.stdout: ** DB Stats ** 2024-09-13T14:26:56.955 INFO:journalctl@ceph.mon.b.smithi184.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:26:56.955 INFO:journalctl@ceph.mon.b.smithi184.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-13T14:26:56.955 INFO:journalctl@ceph.mon.b.smithi184.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:26:56.956 INFO:journalctl@ceph.mon.b.smithi184.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:26:56.956 INFO:journalctl@ceph.mon.b.smithi184.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-13T14:26:56.956 INFO:journalctl@ceph.mon.b.smithi184.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:26:56.956 INFO:journalctl@ceph.mon.b.smithi184.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:26:56.956 INFO:journalctl@ceph.mon.b.smithi184.stdout: 2024-09-13T14:26:56.956 INFO:journalctl@ceph.mon.b.smithi184.stdout: ** Compaction Stats [default] ** 2024-09-13T14:26:56.956 INFO:journalctl@ceph.mon.b.smithi184.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T14:26:56.956 INFO:journalctl@ceph.mon.b.smithi184.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-13T14:26:56.956 INFO:journalctl@ceph.mon.b.smithi184.stdout: L0 1/0 997.12 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 410.9 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-09-13T14:26:56.956 INFO:journalctl@ceph.mon.b.smithi184.stdout: L6 1/0 12.69 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-13T14:26:56.957 INFO:journalctl@ceph.mon.b.smithi184.stdout: Sum 2/0 13.67 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 410.9 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-09-13T14:26:56.957 INFO:journalctl@ceph.mon.b.smithi184.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 410.9 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-09-13T14:26:56.957 INFO:journalctl@ceph.mon.b.smithi184.stdout: 2024-09-13T14:26:56.957 INFO:journalctl@ceph.mon.b.smithi184.stdout: ** Compaction Stats [default] ** 2024-09-13T14:26:56.957 INFO:journalctl@ceph.mon.b.smithi184.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T14:26:56.957 INFO:journalctl@ceph.mon.b.smithi184.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T14:26:56.957 INFO:journalctl@ceph.mon.b.smithi184.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 410.9 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-09-13T14:26:56.957 INFO:journalctl@ceph.mon.b.smithi184.stdout: 2024-09-13T14:26:56.957 INFO:journalctl@ceph.mon.b.smithi184.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-13T14:26:56.957 INFO:journalctl@ceph.mon.b.smithi184.stdout: 2024-09-13T14:26:56.957 INFO:journalctl@ceph.mon.b.smithi184.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:26:56.957 INFO:journalctl@ceph.mon.b.smithi184.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-09-13T14:26:56.958 INFO:journalctl@ceph.mon.b.smithi184.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-13T14:26:56.958 INFO:journalctl@ceph.mon.b.smithi184.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-13T14:26:56.958 INFO:journalctl@ceph.mon.b.smithi184.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-13T14:26:56.958 INFO:journalctl@ceph.mon.b.smithi184.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-13T14:26:56.958 INFO:journalctl@ceph.mon.b.smithi184.stdout: Cumulative compaction: 0.00 GB write, 98.03 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:26:56.958 INFO:journalctl@ceph.mon.b.smithi184.stdout: Interval compaction: 0.00 GB write, 98.03 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:26:56.958 INFO:journalctl@ceph.mon.b.smithi184.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-13T14:26:56.958 INFO:journalctl@ceph.mon.b.smithi184.stdout: Block cache BinnedLRUCache@0x55e591e4e850#7 capacity: 512.00 MB usage: 3.98 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-09-13T14:26:56.958 INFO:journalctl@ceph.mon.b.smithi184.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.91 KB,0.000172853%) IndexBlock(1,3.08 KB,0.000587106%) Misc(1,0.00 KB,0%) 2024-09-13T14:26:56.959 INFO:journalctl@ceph.mon.b.smithi184.stdout: 2024-09-13T14:26:56.959 INFO:journalctl@ceph.mon.b.smithi184.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-13T14:26:56.959 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: starting mon.b rank 1 at public addrs [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] at bind addrs [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:26:56.959 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: mon.b@-1(???) e3 preinit fsid 2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:26:56.959 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: mon.b@-1(???).mds e9 new map 2024-09-13T14:26:56.959 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: mon.b@-1(???).mds e9 print_map 2024-09-13T14:26:56.959 INFO:journalctl@ceph.mon.b.smithi184.stdout: e9 2024-09-13T14:26:56.959 INFO:journalctl@ceph.mon.b.smithi184.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-13T14:26:56.959 INFO:journalctl@ceph.mon.b.smithi184.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-13T14:26:56.959 INFO:journalctl@ceph.mon.b.smithi184.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T14:26:56.959 INFO:journalctl@ceph.mon.b.smithi184.stdout: legacy client fscid: 1 2024-09-13T14:26:56.959 INFO:journalctl@ceph.mon.b.smithi184.stdout: 2024-09-13T14:26:56.959 INFO:journalctl@ceph.mon.b.smithi184.stdout: Filesystem 'foo' (1) 2024-09-13T14:26:56.959 INFO:journalctl@ceph.mon.b.smithi184.stdout: fs_name foo 2024-09-13T14:26:56.960 INFO:journalctl@ceph.mon.b.smithi184.stdout: epoch 8 2024-09-13T14:26:56.960 INFO:journalctl@ceph.mon.b.smithi184.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-13T14:26:56.960 INFO:journalctl@ceph.mon.b.smithi184.stdout: created 2024-09-13T14:19:25.869068+0000 2024-09-13T14:26:56.960 INFO:journalctl@ceph.mon.b.smithi184.stdout: modified 2024-09-13T14:20:09.750989+0000 2024-09-13T14:26:56.960 INFO:journalctl@ceph.mon.b.smithi184.stdout: tableserver 0 2024-09-13T14:26:56.960 INFO:journalctl@ceph.mon.b.smithi184.stdout: root 0 2024-09-13T14:26:56.960 INFO:journalctl@ceph.mon.b.smithi184.stdout: session_timeout 60 2024-09-13T14:26:56.960 INFO:journalctl@ceph.mon.b.smithi184.stdout: session_autoclose 300 2024-09-13T14:26:56.960 INFO:journalctl@ceph.mon.b.smithi184.stdout: max_file_size 1099511627776 2024-09-13T14:26:56.961 INFO:journalctl@ceph.mon.b.smithi184.stdout: max_xattr_size 65536 2024-09-13T14:26:56.961 INFO:journalctl@ceph.mon.b.smithi184.stdout: required_client_features {} 2024-09-13T14:26:56.961 INFO:journalctl@ceph.mon.b.smithi184.stdout: last_failure 0 2024-09-13T14:26:56.961 INFO:journalctl@ceph.mon.b.smithi184.stdout: last_failure_osd_epoch 0 2024-09-13T14:26:56.961 INFO:journalctl@ceph.mon.b.smithi184.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T14:26:56.961 INFO:journalctl@ceph.mon.b.smithi184.stdout: max_mds 1 2024-09-13T14:26:56.961 INFO:journalctl@ceph.mon.b.smithi184.stdout: in 0 2024-09-13T14:26:56.961 INFO:journalctl@ceph.mon.b.smithi184.stdout: up {0=24614} 2024-09-13T14:26:56.961 INFO:journalctl@ceph.mon.b.smithi184.stdout: failed 2024-09-13T14:26:56.961 INFO:journalctl@ceph.mon.b.smithi184.stdout: damaged 2024-09-13T14:26:56.961 INFO:journalctl@ceph.mon.b.smithi184.stdout: stopped 2024-09-13T14:26:56.961 INFO:journalctl@ceph.mon.b.smithi184.stdout: data_pools [4] 2024-09-13T14:26:56.961 INFO:journalctl@ceph.mon.b.smithi184.stdout: metadata_pool 3 2024-09-13T14:26:56.962 INFO:journalctl@ceph.mon.b.smithi184.stdout: inline_data disabled 2024-09-13T14:26:56.962 INFO:journalctl@ceph.mon.b.smithi184.stdout: balancer 2024-09-13T14:26:56.962 INFO:journalctl@ceph.mon.b.smithi184.stdout: bal_rank_mask -1 2024-09-13T14:26:56.962 INFO:journalctl@ceph.mon.b.smithi184.stdout: standby_count_wanted 1 2024-09-13T14:26:56.962 INFO:journalctl@ceph.mon.b.smithi184.stdout: qdb_cluster leader: 0 members: 2024-09-13T14:26:56.962 INFO:journalctl@ceph.mon.b.smithi184.stdout: [mds.foo.smithi151.rqqklw{0:24614} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.151:6834/345315682,v1:172.21.15.151:6835/345315682] compat {c=[1],r=[1],i=[7ff]}] 2024-09-13T14:26:56.962 INFO:journalctl@ceph.mon.b.smithi184.stdout: 2024-09-13T14:26:56.962 INFO:journalctl@ceph.mon.b.smithi184.stdout: 2024-09-13T14:26:56.962 INFO:journalctl@ceph.mon.b.smithi184.stdout: Standby daemons: 2024-09-13T14:26:56.962 INFO:journalctl@ceph.mon.b.smithi184.stdout: 2024-09-13T14:26:56.963 INFO:journalctl@ceph.mon.b.smithi184.stdout: [mds.foo.smithi184.jismsi{-1:24529} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.184:6832/4138067167,v1:172.21.15.184:6833/4138067167] compat {c=[1],r=[1],i=[7ff]}] 2024-09-13T14:26:56.963 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: mon.b@-1(???).osd e426 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-13T14:26:56.963 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: mon.b@-1(???).osd e426 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:26:56.963 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: mon.b@-1(???).osd e426 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:26:56.963 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: mon.b@-1(???).osd e426 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:26:56.963 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-mon[92527]: mon.b@-1(???).paxosservice(auth 1..21) refresh upgraded, format 0 -> 3 2024-09-13T14:26:56.963 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 bash[92489]: 12b6c0ccb635e1e1f30dc0613122cee5a4dd58d8baaa259c2746a4305c361546 2024-09-13T14:26:56.963 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:56 smithi184 systemd[1]: Started Ceph mon.b for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:26:57.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:26:56 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:26:56.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi184\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:26:58.090 INFO:tasks.rados.rados.0.smithi151.stdout:1479: done (0 left) 2024-09-13T14:26:58.090 INFO:tasks.rados.rados.0.smithi151.stdout:1480: read oid 48 snap -1 2024-09-13T14:26:58.090 INFO:tasks.rados.rados.0.smithi151.stdout:1480: expect deleted 2024-09-13T14:26:58.090 INFO:tasks.rados.rados.0.smithi151.stdout:1481: write oid 25 current snap is 185 2024-09-13T14:26:58.090 INFO:tasks.rados.rados.0.smithi151.stdout:1481: seq_num 576 ranges {480832=469899,1606334=664200,3018232=662600} 2024-09-13T14:26:58.095 INFO:tasks.rados.rados.0.smithi151.stdout:1481: writing smithi15186934-25 from 480832 to 950731 tid 1 2024-09-13T14:26:58.104 INFO:tasks.rados.rados.0.smithi151.stdout:1481: writing smithi15186934-25 from 1606334 to 2270534 tid 2 2024-09-13T14:26:58.113 INFO:tasks.rados.rados.0.smithi151.stdout:1481: writing smithi15186934-25 from 3018232 to 3680832 tid 3 2024-09-13T14:26:58.113 INFO:tasks.rados.rados.0.smithi151.stdout:1482: read oid 33 snap -1 2024-09-13T14:26:58.113 INFO:tasks.rados.rados.0.smithi151.stdout:1482: expect (ObjNum 508 snap 164 seq_num 508) 2024-09-13T14:26:58.113 INFO:tasks.rados.rados.0.smithi151.stdout:1483: snap_create 2024-09-13T14:26:58.113 INFO:tasks.rados.rados.0.smithi151.stdout:1481: finishing write tid 1 to smithi15186934-25 2024-09-13T14:26:58.113 INFO:tasks.rados.rados.0.smithi151.stdout:1481: finishing write tid 2 to smithi15186934-25 2024-09-13T14:26:58.116 INFO:tasks.rados.rados.0.smithi151.stdout:1482: expect (ObjNum 508 snap 164 seq_num 508) 2024-09-13T14:26:58.202 INFO:tasks.rados.rados.0.smithi151.stdout:1481: finishing write tid 3 to smithi15186934-25 2024-09-13T14:26:58.203 INFO:tasks.rados.rados.0.smithi151.stdout:1481: finishing write tid 4 to smithi15186934-25 2024-09-13T14:26:58.203 INFO:tasks.rados.rados.0.smithi151.stdout:1481: finishing write tid 5 to smithi15186934-25 2024-09-13T14:26:58.203 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 872 (ObjNum 576 snap 185 seq_num 576) dirty exists 2024-09-13T14:26:58.203 INFO:tasks.rados.rados.0.smithi151.stdout:1481: left oid 25 (ObjNum 576 snap 185 seq_num 576) 2024-09-13T14:26:58.253 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 3 2024-09-13T14:26:58.819 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch ps' 2024-09-13T14:26:59.063 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:59.114 INFO:tasks.rados.rados.0.smithi151.stdout:1480: done (3 left) 2024-09-13T14:26:59.114 INFO:tasks.rados.rados.0.smithi151.stdout:1481: done (2 left) 2024-09-13T14:26:59.114 INFO:tasks.rados.rados.0.smithi151.stdout:1482: done (1 left) 2024-09-13T14:26:59.114 INFO:tasks.rados.rados.0.smithi151.stdout:1483: done (0 left) 2024-09-13T14:26:59.114 INFO:tasks.rados.rados.0.smithi151.stdout:1484: write oid 7 current snap is 186 2024-09-13T14:26:59.114 INFO:tasks.rados.rados.0.smithi151.stdout:1484: seq_num 577 ranges {497639=545148,1548463=407858,2529302=564422,3528143=169496} 2024-09-13T14:26:59.119 INFO:tasks.rados.rados.0.smithi151.stdout:1484: writing smithi15186934-7 from 497639 to 1042787 tid 1 2024-09-13T14:26:59.125 INFO:tasks.rados.rados.0.smithi151.stdout:1484: writing smithi15186934-7 from 1548463 to 1956321 tid 2 2024-09-13T14:26:59.131 INFO:tasks.rados.rados.0.smithi151.stdout:1484: writing smithi15186934-7 from 2529302 to 3093724 tid 3 2024-09-13T14:26:59.134 INFO:tasks.rados.rados.0.smithi151.stdout:1484: writing smithi15186934-7 from 3528143 to 3697639 tid 4 2024-09-13T14:26:59.135 INFO:tasks.rados.rados.0.smithi151.stdout:1485: write oid 21 current snap is 186 2024-09-13T14:26:59.135 INFO:tasks.rados.rados.0.smithi151.stdout:1484: finishing write tid 1 to smithi15186934-7 2024-09-13T14:26:59.135 INFO:tasks.rados.rados.0.smithi151.stdout:1484: finishing write tid 2 to smithi15186934-7 2024-09-13T14:26:59.135 INFO:tasks.rados.rados.0.smithi151.stdout:1484: finishing write tid 3 to smithi15186934-7 2024-09-13T14:26:59.135 INFO:tasks.rados.rados.0.smithi151.stdout:1485: seq_num 578 ranges {514446=736750,1723298=667869,3073078=641368} 2024-09-13T14:26:59.141 INFO:tasks.rados.rados.0.smithi151.stdout:1485: writing smithi15186934-21 from 514446 to 1251196 tid 1 2024-09-13T14:26:59.147 INFO:tasks.rados.rados.0.smithi151.stdout:1485: writing smithi15186934-21 from 1723298 to 2391167 tid 2 2024-09-13T14:26:59.154 INFO:tasks.rados.rados.0.smithi151.stdout:1485: writing smithi15186934-21 from 3073078 to 3714446 tid 3 2024-09-13T14:26:59.154 INFO:tasks.rados.rados.0.smithi151.stdout:1486: write oid 44 current snap is 186 2024-09-13T14:26:59.154 INFO:tasks.rados.rados.0.smithi151.stdout:1486: seq_num 579 ranges {531253=411999,1381780=411527,2300501=586966,3376974=354279} 2024-09-13T14:26:59.159 INFO:tasks.rados.rados.0.smithi151.stdout:1486: writing smithi15186934-44 from 531253 to 943252 tid 1 2024-09-13T14:26:59.164 INFO:tasks.rados.rados.0.smithi151.stdout:1486: writing smithi15186934-44 from 1381780 to 1793307 tid 2 2024-09-13T14:26:59.170 INFO:tasks.rados.rados.0.smithi151.stdout:1486: writing smithi15186934-44 from 2300501 to 2887467 tid 3 2024-09-13T14:26:59.175 INFO:tasks.rados.rados.0.smithi151.stdout:1486: writing smithi15186934-44 from 3376974 to 3731253 tid 4 2024-09-13T14:26:59.175 INFO:tasks.rados.rados.0.smithi151.stdout:1487: delete oid 12 current snap is 186 2024-09-13T14:26:59.175 INFO:tasks.rados.rados.0.smithi151.stdout:1484: finishing write tid 4 to smithi15186934-7 2024-09-13T14:26:59.175 INFO:tasks.rados.rados.0.smithi151.stdout:1484: finishing write tid 5 to smithi15186934-7 2024-09-13T14:26:59.176 INFO:tasks.rados.rados.0.smithi151.stdout:1484: finishing write tid 6 to smithi15186934-7 2024-09-13T14:26:59.176 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 982 (ObjNum 577 snap 186 seq_num 577) dirty exists 2024-09-13T14:26:59.176 INFO:tasks.rados.rados.0.smithi151.stdout:1484: left oid 7 (ObjNum 577 snap 186 seq_num 577) 2024-09-13T14:26:59.176 INFO:tasks.rados.rados.0.smithi151.stdout:1485: finishing write tid 1 to smithi15186934-21 2024-09-13T14:26:59.176 INFO:tasks.rados.rados.0.smithi151.stdout:1485: finishing write tid 2 to smithi15186934-21 2024-09-13T14:26:59.176 INFO:tasks.rados.rados.0.smithi151.stdout:1485: finishing write tid 3 to smithi15186934-21 2024-09-13T14:26:59.176 INFO:tasks.rados.rados.0.smithi151.stdout:1485: finishing write tid 4 to smithi15186934-21 2024-09-13T14:26:59.176 INFO:tasks.rados.rados.0.smithi151.stdout:1485: finishing write tid 5 to smithi15186934-21 2024-09-13T14:26:59.176 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 21 v 326 (ObjNum 578 snap 186 seq_num 578) dirty exists 2024-09-13T14:26:59.176 INFO:tasks.rados.rados.0.smithi151.stdout:1485: left oid 21 (ObjNum 578 snap 186 seq_num 578) 2024-09-13T14:26:59.176 INFO:tasks.rados.rados.0.smithi151.stdout:1486: finishing write tid 1 to smithi15186934-44 2024-09-13T14:26:59.176 INFO:tasks.rados.rados.0.smithi151.stdout:1486: finishing write tid 2 to smithi15186934-44 2024-09-13T14:26:59.176 INFO:tasks.rados.rados.0.smithi151.stdout:1486: finishing write tid 3 to smithi15186934-44 2024-09-13T14:26:59.178 INFO:tasks.rados.rados.0.smithi151.stdout:1486: finishing write tid 4 to smithi15186934-44 2024-09-13T14:26:59.178 INFO:tasks.rados.rados.0.smithi151.stdout:1484: done (3 left) 2024-09-13T14:26:59.178 INFO:tasks.rados.rados.0.smithi151.stdout:1485: done (2 left) 2024-09-13T14:26:59.179 INFO:tasks.rados.rados.0.smithi151.stdout:1487: done (1 left) 2024-09-13T14:26:59.179 INFO:tasks.rados.rados.0.smithi151.stdout:1488: write oid 39 current snap is 186 2024-09-13T14:26:59.179 INFO:tasks.rados.rados.0.smithi151.stdout:1486: finishing write tid 5 to smithi15186934-44 2024-09-13T14:26:59.179 INFO:tasks.rados.rados.0.smithi151.stdout:1488: seq_num 581 ranges {564867=562497,1898744=415196,2755347=725863} 2024-09-13T14:26:59.184 INFO:tasks.rados.rados.0.smithi151.stdout:1488: writing smithi15186934-39 from 564867 to 1127364 tid 1 2024-09-13T14:26:59.190 INFO:tasks.rados.rados.0.smithi151.stdout:1488: writing smithi15186934-39 from 1898744 to 2313940 tid 2 2024-09-13T14:26:59.197 INFO:tasks.rados.rados.0.smithi151.stdout:1488: writing smithi15186934-39 from 2755347 to 3481210 tid 3 2024-09-13T14:26:59.197 INFO:tasks.rados.rados.0.smithi151.stdout:1489: snap_remove snap 166 2024-09-13T14:26:59.345 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[104450]: 2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2024-09-13T14:26:59.345 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[104450]: fsmap foo:1 {0=foo.smithi151.rqqklw=up:active} 1 up:standby 2024-09-13T14:26:59.345 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[104450]: osdmap e427: 8 total, 8 up, 8 in 2024-09-13T14:26:59.345 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[104450]: mgrmap e38: y(active, since 2m), standbys: x 2024-09-13T14:26:59.345 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[104450]: overall HEALTH_OK 2024-09-13T14:26:59.345 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[104450]: mon.c calling monitor election 2024-09-13T14:26:59.345 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[104450]: mon.b calling monitor election 2024-09-13T14:26:59.345 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[104450]: mon.a calling monitor election 2024-09-13T14:26:59.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T14:26:59.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T14:26:59.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T14:26:59.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[104450]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:26:59.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[104450]: monmap epoch 4 2024-09-13T14:26:59.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[104450]: fsid 2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:26:59.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[104450]: last_changed 2024-09-13T14:26:58.081620+0000 2024-09-13T14:26:59.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[104450]: created 2024-09-13T14:12:33.302475+0000 2024-09-13T14:26:59.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[104450]: min_mon_release 19 (squid) 2024-09-13T14:26:59.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[104450]: election_strategy: 1 2024-09-13T14:26:59.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[104450]: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2024-09-13T14:26:59.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[104450]: 1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2024-09-13T14:26:59.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[104450]: 2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2024-09-13T14:26:59.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[104450]: fsmap foo:1 {0=foo.smithi151.rqqklw=up:active} 1 up:standby 2024-09-13T14:26:59.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[104450]: osdmap e427: 8 total, 8 up, 8 in 2024-09-13T14:26:59.347 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[104450]: mgrmap e38: y(active, since 2m), standbys: x 2024-09-13T14:26:59.347 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[104450]: overall HEALTH_OK 2024-09-13T14:26:59.347 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:26:59.347 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:26:59.347 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:26:59.347 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/450205437' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-13T14:26:59.347 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[108966]: 2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2024-09-13T14:26:59.347 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[108966]: fsmap foo:1 {0=foo.smithi151.rqqklw=up:active} 1 up:standby 2024-09-13T14:26:59.347 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[108966]: osdmap e427: 8 total, 8 up, 8 in 2024-09-13T14:26:59.347 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[108966]: mgrmap e38: y(active, since 2m), standbys: x 2024-09-13T14:26:59.348 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[108966]: overall HEALTH_OK 2024-09-13T14:26:59.348 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[108966]: mon.c calling monitor election 2024-09-13T14:26:59.348 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[108966]: mon.b calling monitor election 2024-09-13T14:26:59.348 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[108966]: mon.a calling monitor election 2024-09-13T14:26:59.348 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T14:26:59.348 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T14:26:59.348 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T14:26:59.348 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[108966]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:26:59.348 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[108966]: monmap epoch 4 2024-09-13T14:26:59.348 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[108966]: fsid 2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:26:59.348 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[108966]: last_changed 2024-09-13T14:26:58.081620+0000 2024-09-13T14:26:59.348 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[108966]: created 2024-09-13T14:12:33.302475+0000 2024-09-13T14:26:59.348 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[108966]: min_mon_release 19 (squid) 2024-09-13T14:26:59.348 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[108966]: election_strategy: 1 2024-09-13T14:26:59.349 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[108966]: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2024-09-13T14:26:59.349 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[108966]: 1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2024-09-13T14:26:59.349 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[108966]: 2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2024-09-13T14:26:59.349 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[108966]: fsmap foo:1 {0=foo.smithi151.rqqklw=up:active} 1 up:standby 2024-09-13T14:26:59.349 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[108966]: osdmap e427: 8 total, 8 up, 8 in 2024-09-13T14:26:59.349 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[108966]: mgrmap e38: y(active, since 2m), standbys: x 2024-09-13T14:26:59.349 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[108966]: overall HEALTH_OK 2024-09-13T14:26:59.349 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:26:59.349 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:26:59.349 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:26:59.349 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:26:59 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/450205437' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-13T14:26:59.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:59 smithi184 ceph-mon[92527]: 2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2024-09-13T14:26:59.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:59 smithi184 ceph-mon[92527]: fsmap foo:1 {0=foo.smithi151.rqqklw=up:active} 1 up:standby 2024-09-13T14:26:59.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:59 smithi184 ceph-mon[92527]: osdmap e427: 8 total, 8 up, 8 in 2024-09-13T14:26:59.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:59 smithi184 ceph-mon[92527]: mgrmap e38: y(active, since 2m), standbys: x 2024-09-13T14:26:59.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:59 smithi184 ceph-mon[92527]: overall HEALTH_OK 2024-09-13T14:26:59.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:59 smithi184 ceph-mon[92527]: mon.c calling monitor election 2024-09-13T14:26:59.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:59 smithi184 ceph-mon[92527]: mon.b calling monitor election 2024-09-13T14:26:59.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:59 smithi184 ceph-mon[92527]: mon.a calling monitor election 2024-09-13T14:26:59.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:59 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T14:26:59.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:59 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T14:26:59.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:59 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T14:26:59.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:59 smithi184 ceph-mon[92527]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:26:59.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:59 smithi184 ceph-mon[92527]: monmap epoch 4 2024-09-13T14:26:59.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:59 smithi184 ceph-mon[92527]: fsid 2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:26:59.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:59 smithi184 ceph-mon[92527]: last_changed 2024-09-13T14:26:58.081620+0000 2024-09-13T14:26:59.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:59 smithi184 ceph-mon[92527]: created 2024-09-13T14:12:33.302475+0000 2024-09-13T14:26:59.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:59 smithi184 ceph-mon[92527]: min_mon_release 19 (squid) 2024-09-13T14:26:59.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:59 smithi184 ceph-mon[92527]: election_strategy: 1 2024-09-13T14:26:59.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:59 smithi184 ceph-mon[92527]: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2024-09-13T14:26:59.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:59 smithi184 ceph-mon[92527]: 1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2024-09-13T14:26:59.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:59 smithi184 ceph-mon[92527]: 2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2024-09-13T14:26:59.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:59 smithi184 ceph-mon[92527]: fsmap foo:1 {0=foo.smithi151.rqqklw=up:active} 1 up:standby 2024-09-13T14:26:59.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:59 smithi184 ceph-mon[92527]: osdmap e427: 8 total, 8 up, 8 in 2024-09-13T14:26:59.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:59 smithi184 ceph-mon[92527]: mgrmap e38: y(active, since 2m), standbys: x 2024-09-13T14:26:59.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:59 smithi184 ceph-mon[92527]: overall HEALTH_OK 2024-09-13T14:26:59.356 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:59 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:26:59.356 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:59 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:26:59.356 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:59 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:26:59.356 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:26:59 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/450205437' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-13T14:27:00.120 INFO:tasks.rados.rados.0.smithi151.stdout:1489: done (2 left) 2024-09-13T14:27:00.120 INFO:tasks.rados.rados.0.smithi151.stdout:1490: read oid 41 snap -1 2024-09-13T14:27:00.120 INFO:tasks.rados.rados.0.smithi151.stdout:1490: expect (ObjNum 542 snap 174 seq_num 542) 2024-09-13T14:27:00.120 INFO:tasks.rados.rados.0.smithi151.stdout:1491: write oid 6 current snap is 186 2024-09-13T14:27:00.121 INFO:tasks.rados.rados.0.smithi151.stdout:1491: seq_num 582 ranges {581674=637746,1840873=558854,3066417=537135} 2024-09-13T14:27:00.127 INFO:tasks.rados.rados.0.smithi151.stdout:1491: writing smithi15186934-6 from 581674 to 1219420 tid 1 2024-09-13T14:27:00.134 INFO:tasks.rados.rados.0.smithi151.stdout:1491: writing smithi15186934-6 from 1840873 to 2399727 tid 2 2024-09-13T14:27:00.141 INFO:tasks.rados.rados.0.smithi151.stdout:1491: writing smithi15186934-6 from 3066417 to 3603552 tid 3 2024-09-13T14:27:00.142 INFO:tasks.rados.rados.0.smithi151.stdout:1492: snap_create 2024-09-13T14:27:00.142 INFO:tasks.rados.rados.0.smithi151.stdout:1486: finishing write tid 6 to smithi15186934-44 2024-09-13T14:27:00.142 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 476 (ObjNum 579 snap 186 seq_num 579) dirty exists 2024-09-13T14:27:00.142 INFO:tasks.rados.rados.0.smithi151.stdout:1486: left oid 44 (ObjNum 579 snap 186 seq_num 579) 2024-09-13T14:27:00.142 INFO:tasks.rados.rados.0.smithi151.stdout:1488: finishing write tid 1 to smithi15186934-39 2024-09-13T14:27:00.142 INFO:tasks.rados.rados.0.smithi151.stdout:1488: finishing write tid 2 to smithi15186934-39 2024-09-13T14:27:00.142 INFO:tasks.rados.rados.0.smithi151.stdout:1488: finishing write tid 3 to smithi15186934-39 2024-09-13T14:27:00.142 INFO:tasks.rados.rados.0.smithi151.stdout:1488: finishing write tid 4 to smithi15186934-39 2024-09-13T14:27:00.142 INFO:tasks.rados.rados.0.smithi151.stdout:1488: finishing write tid 5 to smithi15186934-39 2024-09-13T14:27:00.142 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 448 (ObjNum 581 snap 186 seq_num 581) dirty exists 2024-09-13T14:27:00.142 INFO:tasks.rados.rados.0.smithi151.stdout:1488: left oid 39 (ObjNum 581 snap 186 seq_num 581) 2024-09-13T14:27:00.142 INFO:tasks.rados.rados.0.smithi151.stdout:1491: finishing write tid 1 to smithi15186934-6 2024-09-13T14:27:00.143 INFO:tasks.rados.rados.0.smithi151.stdout:1490: expect (ObjNum 542 snap 174 seq_num 542) 2024-09-13T14:27:00.210 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:00 smithi151 ceph-mon[104450]: osdmap e428: 8 total, 8 up, 8 in 2024-09-13T14:27:00.210 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:00 smithi151 ceph-mon[104450]: pgmap v261: 89 pgs: 89 active+clean; 283 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 368 B/s rd, 637 KiB/s wr, 2 op/s 2024-09-13T14:27:00.210 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:00 smithi151 ceph-mon[108966]: osdmap e428: 8 total, 8 up, 8 in 2024-09-13T14:27:00.210 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:00 smithi151 ceph-mon[108966]: pgmap v261: 89 pgs: 89 active+clean; 283 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 368 B/s rd, 637 KiB/s wr, 2 op/s 2024-09-13T14:27:00.285 INFO:tasks.rados.rados.0.smithi151.stdout:1491: finishing write tid 2 to smithi15186934-6 2024-09-13T14:27:00.285 INFO:tasks.rados.rados.0.smithi151.stdout:1491: finishing write tid 3 to smithi15186934-6 2024-09-13T14:27:00.285 INFO:tasks.rados.rados.0.smithi151.stdout:1491: finishing write tid 4 to smithi15186934-6 2024-09-13T14:27:00.285 INFO:tasks.rados.rados.0.smithi151.stdout:1491: finishing write tid 5 to smithi15186934-6 2024-09-13T14:27:00.285 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 6 v 390 (ObjNum 582 snap 186 seq_num 582) dirty exists 2024-09-13T14:27:00.285 INFO:tasks.rados.rados.0.smithi151.stdout:1491: left oid 6 (ObjNum 582 snap 186 seq_num 582) 2024-09-13T14:27:00.559 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:00 smithi184 ceph-mon[92527]: osdmap e428: 8 total, 8 up, 8 in 2024-09-13T14:27:00.559 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:00 smithi184 ceph-mon[92527]: pgmap v261: 89 pgs: 89 active+clean; 283 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 368 B/s rd, 637 KiB/s wr, 2 op/s 2024-09-13T14:27:00.559 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:27:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:27:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:00.925 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:27:00.925 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.a smithi151 *:9093,9094 running (4m) 10s ago 8m 18.3M - 0.27.0 11f11916f8cd ed70c3d75107 2024-09-13T14:27:00.925 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.a smithi184 *:3000 running (3m) 12s ago 8m 85.5M - 9.4.7 954c08fa6188 4ba18fb1f1fc 2024-09-13T14:27:00.925 INFO:teuthology.orchestra.run.smithi151.stdout:mds.foo.smithi151.rqqklw smithi151 running (6m) 10s ago 6m 18.2M - 17.2.7-1627-gbbdadfa3 a994b00deb3a c960b57fc84b 2024-09-13T14:27:00.925 INFO:teuthology.orchestra.run.smithi151.stdout:mds.foo.smithi184.jismsi smithi184 running (6m) 12s ago 6m 15.1M - 17.2.7-1627-gbbdadfa3 a994b00deb3a 0cc31a3a5339 2024-09-13T14:27:00.925 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.x smithi184 *:8443,9283,8765 running (3m) 12s ago 12m 454M - 19.3.0-4697-gd58f7306 fec07016a2b0 8a0f7d440b6e 2024-09-13T14:27:00.925 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.y smithi151 *:8443,9283,8765 running (3m) 10s ago 14m 509M - 19.3.0-4697-gd58f7306 fec07016a2b0 d8c169a4f36e 2024-09-13T14:27:00.926 INFO:teuthology.orchestra.run.smithi151.stdout:mon.a smithi151 running (3m) 10s ago 14m 50.2M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 694de6b945cd 2024-09-13T14:27:00.926 INFO:teuthology.orchestra.run.smithi151.stdout:mon.b smithi184 starting - - - 2048M 2024-09-13T14:27:00.926 INFO:teuthology.orchestra.run.smithi151.stdout:mon.c smithi151 running (94s) 10s ago 12m 36.6M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 e548fceed2f9 2024-09-13T14:27:00.926 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.a smithi151 *:9100 running (4m) 10s ago 7m 11.1M - 1.7.0 72c9c2088986 d91e4cb09e9a 2024-09-13T14:27:00.926 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.b smithi184 *:9100 running (3m) 12s ago 7m 11.8M - 1.7.0 72c9c2088986 1e1a9dd643a0 2024-09-13T14:27:00.926 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (12m) 10s ago 12m 99.2M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a e6951a1a86d9 2024-09-13T14:27:00.926 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (11m) 10s ago 11m 79.9M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a 94bc726b3eb5 2024-09-13T14:27:00.926 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (11m) 10s ago 11m 106M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a 122660181e46 2024-09-13T14:27:00.926 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi151 running (11m) 10s ago 11m 72.2M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a 495241a87df6 2024-09-13T14:27:00.926 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi184 running (10m) 12s ago 10m 96.0M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a 6e70699b8084 2024-09-13T14:27:00.926 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi184 running (10m) 12s ago 10m 95.9M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a 36b82c9e7230 2024-09-13T14:27:00.926 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6 smithi184 running (9m) 12s ago 9m 75.5M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a f9379cec7504 2024-09-13T14:27:00.926 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7 smithi184 running (9m) 12s ago 9m 89.0M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a f37cc947097b 2024-09-13T14:27:00.926 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.a smithi184 *:9095 running (3m) 12s ago 8m 48.7M - 2.51.0 1d3b7f56885b 3da9265b8f14 2024-09-13T14:27:01.127 INFO:tasks.rados.rados.0.smithi151.stdout:1486: done (4 left) 2024-09-13T14:27:01.127 INFO:tasks.rados.rados.0.smithi151.stdout:1488: done (3 left) 2024-09-13T14:27:01.127 INFO:tasks.rados.rados.0.smithi151.stdout:1490: done (2 left) 2024-09-13T14:27:01.127 INFO:tasks.rados.rados.0.smithi151.stdout:1491: done (1 left) 2024-09-13T14:27:01.127 INFO:tasks.rados.rados.0.smithi151.stdout:1492: done (0 left) 2024-09-13T14:27:01.127 INFO:tasks.rados.rados.0.smithi151.stdout:1493: snap_create 2024-09-13T14:27:01.225 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:01 smithi151 ceph-mon[104450]: osdmap e429: 8 total, 8 up, 8 in 2024-09-13T14:27:01.226 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:01 smithi151 ceph-mon[104450]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:01.226 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:01 smithi151 ceph-mon[108966]: osdmap e429: 8 total, 8 up, 8 in 2024-09-13T14:27:01.226 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:01 smithi151 ceph-mon[108966]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:01.467 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:01 smithi184 ceph-mon[92527]: osdmap e429: 8 total, 8 up, 8 in 2024-09-13T14:27:01.467 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:01 smithi184 ceph-mon[92527]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:01.559 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch upgrade pause' 2024-09-13T14:27:01.805 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:01.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:27:01 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:27:01.654+0000 7fc94c153640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-13T14:27:02.135 INFO:tasks.rados.rados.0.smithi151.stdout:1493: done (0 left) 2024-09-13T14:27:02.135 INFO:tasks.rados.rados.0.smithi151.stdout:1494: snap_create 2024-09-13T14:27:02.220 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:02 smithi151 ceph-mon[104450]: osdmap e430: 8 total, 8 up, 8 in 2024-09-13T14:27:02.220 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:02 smithi151 ceph-mon[104450]: pgmap v264: 89 pgs: 89 active+clean; 283 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:27:02.220 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:02.221 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:02 smithi151 ceph-mon[108966]: osdmap e430: 8 total, 8 up, 8 in 2024-09-13T14:27:02.221 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:02 smithi151 ceph-mon[108966]: pgmap v264: 89 pgs: 89 active+clean; 283 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:27:02.221 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:02.434 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:02 smithi184 ceph-mon[92527]: osdmap e430: 8 total, 8 up, 8 in 2024-09-13T14:27:02.434 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:02 smithi184 ceph-mon[92527]: pgmap v264: 89 pgs: 89 active+clean; 283 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:27:02.434 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:03.132 INFO:tasks.rados.rados.0.smithi151.stdout:1494: done (0 left) 2024-09-13T14:27:03.132 INFO:tasks.rados.rados.0.smithi151.stdout:1495: snap_remove snap 163 2024-09-13T14:27:03.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:03 smithi151 ceph-mon[104450]: osdmap e431: 8 total, 8 up, 8 in 2024-09-13T14:27:03.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:03 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:03.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:03 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:03.344 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:03 smithi151 ceph-mon[108966]: osdmap e431: 8 total, 8 up, 8 in 2024-09-13T14:27:03.344 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:03 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:03.344 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:03 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:03.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:03 smithi184 ceph-mon[92527]: osdmap e431: 8 total, 8 up, 8 in 2024-09-13T14:27:03.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:03 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:03.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:03 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:03.772 INFO:teuthology.orchestra.run.smithi151.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:27:04.134 INFO:tasks.rados.rados.0.smithi151.stdout:1495: done (0 left) 2024-09-13T14:27:04.134 INFO:tasks.rados.rados.0.smithi151.stdout:1496: snap_create 2024-09-13T14:27:04.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:04 smithi184 ceph-mon[92527]: pgmap v266: 89 pgs: 89 active+clean; 294 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 MiB/s rd, 2.6 MiB/s wr, 14 op/s 2024-09-13T14:27:04.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:04 smithi184 ceph-mon[92527]: osdmap e432: 8 total, 8 up, 8 in 2024-09-13T14:27:04.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:04 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:04.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:04 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:04.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:04 smithi184 ceph-mon[92527]: from='client.44144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:04.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:04 smithi184 ceph-mon[92527]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:27:04.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:04 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:04.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:04 smithi184 ceph-mon[92527]: osdmap e433: 8 total, 8 up, 8 in 2024-09-13T14:27:04.355 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:27:04 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:27:04.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:04.399 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:04 smithi151 ceph-mon[104450]: pgmap v266: 89 pgs: 89 active+clean; 294 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 MiB/s rd, 2.6 MiB/s wr, 14 op/s 2024-09-13T14:27:04.399 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:04 smithi151 ceph-mon[104450]: osdmap e432: 8 total, 8 up, 8 in 2024-09-13T14:27:04.399 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:04 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:04.399 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:04 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:04.399 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:04 smithi151 ceph-mon[104450]: from='client.44144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:04.399 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:04 smithi151 ceph-mon[104450]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:27:04.400 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:04 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:04.400 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:04 smithi151 ceph-mon[104450]: osdmap e433: 8 total, 8 up, 8 in 2024-09-13T14:27:04.400 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:04 smithi151 ceph-mon[108966]: pgmap v266: 89 pgs: 89 active+clean; 294 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 MiB/s rd, 2.6 MiB/s wr, 14 op/s 2024-09-13T14:27:04.400 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:04 smithi151 ceph-mon[108966]: osdmap e432: 8 total, 8 up, 8 in 2024-09-13T14:27:04.400 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:04 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:04.400 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:04 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:04.400 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:04 smithi151 ceph-mon[108966]: from='client.44144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:27:04.400 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:04 smithi151 ceph-mon[108966]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:27:04.400 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:04 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:04.400 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:04 smithi151 ceph-mon[108966]: osdmap e433: 8 total, 8 up, 8 in 2024-09-13T14:27:04.424 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'sleep 60' 2024-09-13T14:27:04.482 INFO:tasks.rados.rados.0.smithi151.stdout:1496: done (0 left) 2024-09-13T14:27:04.482 INFO:tasks.rados.rados.0.smithi151.stdout:1497: snap_remove snap 185 2024-09-13T14:27:04.697 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:04.722 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:27:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:27:04] "GET /metrics HTTP/1.1" 200 36332 "" "Prometheus/2.51.0" 2024-09-13T14:27:05.488 INFO:tasks.rados.rados.0.smithi151.stdout:1497: done (0 left) 2024-09-13T14:27:05.488 INFO:tasks.rados.rados.0.smithi151.stdout:1498: rollback oid 50 current snap is 190 2024-09-13T14:27:05.488 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 50 to 182 2024-09-13T14:27:05.488 INFO:tasks.rados.rados.0.smithi151.stdout:1499: write oid 24 current snap is 190 2024-09-13T14:27:05.488 INFO:tasks.rados.rados.0.smithi151.stdout:1499: seq_num 583 ranges {598481=712995,1899355=418865,3093840=704641} 2024-09-13T14:27:05.496 INFO:tasks.rados.rados.0.smithi151.stdout:1499: writing smithi15186934-24 from 598481 to 1311476 tid 1 2024-09-13T14:27:05.501 INFO:tasks.rados.rados.0.smithi151.stdout:1499: writing smithi15186934-24 from 1899355 to 2318220 tid 2 2024-09-13T14:27:05.509 INFO:tasks.rados.rados.0.smithi151.stdout:1499: writing smithi15186934-24 from 3093840 to 3798481 tid 3 2024-09-13T14:27:05.510 INFO:tasks.rados.rados.0.smithi151.stdout:1500: write oid 49 current snap is 190 2024-09-13T14:27:05.510 INFO:tasks.rados.rados.0.smithi151.stdout:1498: finishing rollback tid 0 to smithi15186934-50 2024-09-13T14:27:05.510 INFO:tasks.rados.rados.0.smithi151.stdout:1500: seq_num 584 ranges {615288=788244,1957837=562523,3121263=559679} 2024-09-13T14:27:05.517 INFO:tasks.rados.rados.0.smithi151.stdout:1500: writing smithi15186934-49 from 615288 to 1403532 tid 1 2024-09-13T14:27:05.524 INFO:tasks.rados.rados.0.smithi151.stdout:1500: writing smithi15186934-49 from 1957837 to 2520360 tid 2 2024-09-13T14:27:05.530 INFO:tasks.rados.rados.0.smithi151.stdout:1500: writing smithi15186934-49 from 3121263 to 3680942 tid 3 2024-09-13T14:27:05.530 INFO:tasks.rados.rados.0.smithi151.stdout:1501: snap_remove snap 184 2024-09-13T14:27:05.605 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:05 smithi151 ceph-mon[104450]: osdmap e434: 8 total, 8 up, 8 in 2024-09-13T14:27:05.606 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:05 smithi151 ceph-mon[108966]: osdmap e434: 8 total, 8 up, 8 in 2024-09-13T14:27:05.741 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:05 smithi184 ceph-mon[92527]: osdmap e434: 8 total, 8 up, 8 in 2024-09-13T14:27:06.103 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:27:05 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:27:05.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:06.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:27:05 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:27:05.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:06.495 INFO:tasks.rados.rados.0.smithi151.stdout:1501: done (3 left) 2024-09-13T14:27:06.495 INFO:tasks.rados.rados.0.smithi151.stdout:1502: delete oid 36 current snap is 190 2024-09-13T14:27:06.495 INFO:tasks.rados.rados.0.smithi151.stdout:1498: finishing rollback tid 1 to smithi15186934-50 2024-09-13T14:27:06.495 INFO:tasks.rados.rados.0.smithi151.stdout:1498: finishing rollback tid 2 to smithi15186934-50 2024-09-13T14:27:06.495 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 50 v 460 (ObjNum 523 snap 169 seq_num 523) dirty exists 2024-09-13T14:27:06.495 INFO:tasks.rados.rados.0.smithi151.stdout:1499: finishing write tid 1 to smithi15186934-24 2024-09-13T14:27:06.495 INFO:tasks.rados.rados.0.smithi151.stdout:1499: finishing write tid 2 to smithi15186934-24 2024-09-13T14:27:06.495 INFO:tasks.rados.rados.0.smithi151.stdout:1499: finishing write tid 3 to smithi15186934-24 2024-09-13T14:27:06.495 INFO:tasks.rados.rados.0.smithi151.stdout:1499: finishing write tid 4 to smithi15186934-24 2024-09-13T14:27:06.496 INFO:tasks.rados.rados.0.smithi151.stdout:1499: finishing write tid 5 to smithi15186934-24 2024-09-13T14:27:06.496 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 24 v 1005 (ObjNum 583 snap 190 seq_num 583) dirty exists 2024-09-13T14:27:06.496 INFO:tasks.rados.rados.0.smithi151.stdout:1499: left oid 24 (ObjNum 583 snap 190 seq_num 583) 2024-09-13T14:27:06.496 INFO:tasks.rados.rados.0.smithi151.stdout:1500: finishing write tid 1 to smithi15186934-49 2024-09-13T14:27:06.496 INFO:tasks.rados.rados.0.smithi151.stdout:1500: finishing write tid 2 to smithi15186934-49 2024-09-13T14:27:06.496 INFO:tasks.rados.rados.0.smithi151.stdout:1500: finishing write tid 3 to smithi15186934-49 2024-09-13T14:27:06.496 INFO:tasks.rados.rados.0.smithi151.stdout:1500: finishing write tid 4 to smithi15186934-49 2024-09-13T14:27:06.496 INFO:tasks.rados.rados.0.smithi151.stdout:1500: finishing write tid 5 to smithi15186934-49 2024-09-13T14:27:06.496 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 49 v 655 (ObjNum 584 snap 190 seq_num 584) dirty exists 2024-09-13T14:27:06.496 INFO:tasks.rados.rados.0.smithi151.stdout:1500: left oid 49 (ObjNum 584 snap 190 seq_num 584) 2024-09-13T14:27:06.501 INFO:tasks.rados.rados.0.smithi151.stdout:1498: done (3 left) 2024-09-13T14:27:06.501 INFO:tasks.rados.rados.0.smithi151.stdout:1499: done (2 left) 2024-09-13T14:27:06.501 INFO:tasks.rados.rados.0.smithi151.stdout:1500: done (1 left) 2024-09-13T14:27:06.501 INFO:tasks.rados.rados.0.smithi151.stdout:1502: done (0 left) 2024-09-13T14:27:06.501 INFO:tasks.rados.rados.0.smithi151.stdout:1503: write oid 39 current snap is 190 2024-09-13T14:27:06.501 INFO:tasks.rados.rados.0.smithi151.stdout:1503: seq_num 586 ranges {648902=655095,2074801=566192,3176109=672793} 2024-09-13T14:27:06.508 INFO:tasks.rados.rados.0.smithi151.stdout:1503: writing smithi15186934-39 from 648902 to 1303997 tid 1 2024-09-13T14:27:06.515 INFO:tasks.rados.rados.0.smithi151.stdout:1503: writing smithi15186934-39 from 2074801 to 2640993 tid 2 2024-09-13T14:27:06.523 INFO:tasks.rados.rados.0.smithi151.stdout:1503: writing smithi15186934-39 from 3176109 to 3848902 tid 3 2024-09-13T14:27:06.523 INFO:tasks.rados.rados.0.smithi151.stdout:1504: read oid 49 snap 177 2024-09-13T14:27:06.523 INFO:tasks.rados.rados.0.smithi151.stdout:1504: expect (ObjNum 567 snap 177 seq_num 567) 2024-09-13T14:27:06.523 INFO:tasks.rados.rados.0.smithi151.stdout:1505: read oid 10 snap -1 2024-09-13T14:27:06.523 INFO:tasks.rados.rados.0.smithi151.stdout:1505: expect (ObjNum 575 snap 185 seq_num 575) 2024-09-13T14:27:06.523 INFO:tasks.rados.rados.0.smithi151.stdout:1506: write oid 17 current snap is 190 2024-09-13T14:27:06.523 INFO:tasks.rados.rados.0.smithi151.stdout:1503: finishing write tid 1 to smithi15186934-39 2024-09-13T14:27:06.524 INFO:tasks.rados.rados.0.smithi151.stdout:1503: finishing write tid 2 to smithi15186934-39 2024-09-13T14:27:06.524 INFO:tasks.rados.rados.0.smithi151.stdout:1506: seq_num 587 ranges {665709=730344,2133283=426203,3203532=509848} 2024-09-13T14:27:06.532 INFO:tasks.rados.rados.0.smithi151.stdout:1506: writing smithi15186934-17 from 665709 to 1396053 tid 1 2024-09-13T14:27:06.538 INFO:tasks.rados.rados.0.smithi151.stdout:1506: writing smithi15186934-17 from 2133283 to 2559486 tid 2 2024-09-13T14:27:06.545 INFO:tasks.rados.rados.0.smithi151.stdout:1506: writing smithi15186934-17 from 3203532 to 3713380 tid 3 2024-09-13T14:27:06.545 INFO:tasks.rados.rados.0.smithi151.stdout:1507: snap_remove snap 189 2024-09-13T14:27:06.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:06 smithi184 ceph-mon[92527]: pgmap v270: 89 pgs: 89 active+clean; 294 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 MiB/s rd, 2.6 MiB/s wr, 14 op/s 2024-09-13T14:27:06.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:06 smithi184 ceph-mon[92527]: osdmap e435: 8 total, 8 up, 8 in 2024-09-13T14:27:06.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:06 smithi151 ceph-mon[108966]: pgmap v270: 89 pgs: 89 active+clean; 294 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 MiB/s rd, 2.6 MiB/s wr, 14 op/s 2024-09-13T14:27:06.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:06 smithi151 ceph-mon[108966]: osdmap e435: 8 total, 8 up, 8 in 2024-09-13T14:27:06.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:06 smithi151 ceph-mon[104450]: pgmap v270: 89 pgs: 89 active+clean; 294 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 MiB/s rd, 2.6 MiB/s wr, 14 op/s 2024-09-13T14:27:06.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:06 smithi151 ceph-mon[104450]: osdmap e435: 8 total, 8 up, 8 in 2024-09-13T14:27:07.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:27:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:27:06.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi184\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:07.501 INFO:tasks.rados.rados.0.smithi151.stdout:1507: done (4 left) 2024-09-13T14:27:07.501 INFO:tasks.rados.rados.0.smithi151.stdout:1508: write oid 36 current snap is 190 2024-09-13T14:27:07.501 INFO:tasks.rados.rados.0.smithi151.stdout:1508: seq_num 588 ranges {682516=405593,1791765=569861,2830955=721120} 2024-09-13T14:27:07.506 INFO:tasks.rados.rados.0.smithi151.stdout:1508: writing smithi15186934-36 from 682516 to 1088109 tid 1 2024-09-13T14:27:07.514 INFO:tasks.rados.rados.0.smithi151.stdout:1508: writing smithi15186934-36 from 1791765 to 2361626 tid 2 2024-09-13T14:27:07.521 INFO:tasks.rados.rados.0.smithi151.stdout:1508: writing smithi15186934-36 from 2830955 to 3552075 tid 3 2024-09-13T14:27:07.521 INFO:tasks.rados.rados.0.smithi151.stdout:1509: delete oid 26 current snap is 190 2024-09-13T14:27:07.521 INFO:tasks.rados.rados.0.smithi151.stdout:1503: finishing write tid 3 to smithi15186934-39 2024-09-13T14:27:07.521 INFO:tasks.rados.rados.0.smithi151.stdout:1503: finishing write tid 4 to smithi15186934-39 2024-09-13T14:27:07.522 INFO:tasks.rados.rados.0.smithi151.stdout:1503: finishing write tid 5 to smithi15186934-39 2024-09-13T14:27:07.522 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 471 (ObjNum 586 snap 190 seq_num 586) dirty exists 2024-09-13T14:27:07.522 INFO:tasks.rados.rados.0.smithi151.stdout:1503: left oid 39 (ObjNum 586 snap 190 seq_num 586) 2024-09-13T14:27:07.522 INFO:tasks.rados.rados.0.smithi151.stdout:1505: expect (ObjNum 575 snap 185 seq_num 575) 2024-09-13T14:27:07.642 INFO:tasks.rados.rados.0.smithi151.stdout:1504: expect (ObjNum 567 snap 177 seq_num 567) 2024-09-13T14:27:07.836 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:07 smithi151 ceph-mon[104450]: osdmap e436: 8 total, 8 up, 8 in 2024-09-13T14:27:07.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:07 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:07.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:07 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:07.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:07 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:07.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:07 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:27:07.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:07 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:07.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:07 smithi151 ceph-mon[104450]: Reconfiguring mon.a (monmap changed)... 2024-09-13T14:27:07.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:07 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:27:07.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:07 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:27:07.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:07 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:07.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:07 smithi151 ceph-mon[104450]: Reconfiguring daemon mon.a on smithi151 2024-09-13T14:27:07.838 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:07 smithi151 ceph-mon[108966]: osdmap e436: 8 total, 8 up, 8 in 2024-09-13T14:27:07.838 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:07 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:07.838 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:07 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:07.838 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:07 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:07.838 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:07 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:27:07.838 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:07 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:07.838 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:07 smithi151 ceph-mon[108966]: Reconfiguring mon.a (monmap changed)... 2024-09-13T14:27:07.838 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:07 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:27:07.838 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:07 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:27:07.839 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:07 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:07.839 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:07 smithi151 ceph-mon[108966]: Reconfiguring daemon mon.a on smithi151 2024-09-13T14:27:07.841 INFO:tasks.rados.rados.0.smithi151.stdout:1506: finishing write tid 1 to smithi15186934-17 2024-09-13T14:27:07.841 INFO:tasks.rados.rados.0.smithi151.stdout:1506: finishing write tid 2 to smithi15186934-17 2024-09-13T14:27:07.841 INFO:tasks.rados.rados.0.smithi151.stdout:1506: finishing write tid 3 to smithi15186934-17 2024-09-13T14:27:07.841 INFO:tasks.rados.rados.0.smithi151.stdout:1506: finishing write tid 4 to smithi15186934-17 2024-09-13T14:27:07.841 INFO:tasks.rados.rados.0.smithi151.stdout:1503: done (5 left) 2024-09-13T14:27:07.841 INFO:tasks.rados.rados.0.smithi151.stdout:1504: done (4 left) 2024-09-13T14:27:07.841 INFO:tasks.rados.rados.0.smithi151.stdout:1505: done (3 left) 2024-09-13T14:27:07.841 INFO:tasks.rados.rados.0.smithi151.stdout:1509: done (2 left) 2024-09-13T14:27:07.841 INFO:tasks.rados.rados.0.smithi151.stdout:1510: read oid 49 snap -1 2024-09-13T14:27:07.842 INFO:tasks.rados.rados.0.smithi151.stdout:1510: expect (ObjNum 584 snap 190 seq_num 584) 2024-09-13T14:27:07.842 INFO:tasks.rados.rados.0.smithi151.stdout:1511: snap_create 2024-09-13T14:27:07.842 INFO:tasks.rados.rados.0.smithi151.stdout:1506: finishing write tid 5 to smithi15186934-17 2024-09-13T14:27:07.842 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 1014 (ObjNum 587 snap 190 seq_num 587) dirty exists 2024-09-13T14:27:07.842 INFO:tasks.rados.rados.0.smithi151.stdout:1506: left oid 17 (ObjNum 587 snap 190 seq_num 587) 2024-09-13T14:27:07.842 INFO:tasks.rados.rados.0.smithi151.stdout:1508: finishing write tid 1 to smithi15186934-36 2024-09-13T14:27:07.842 INFO:tasks.rados.rados.0.smithi151.stdout:1508: finishing write tid 2 to smithi15186934-36 2024-09-13T14:27:07.842 INFO:tasks.rados.rados.0.smithi151.stdout:1508: finishing write tid 3 to smithi15186934-36 2024-09-13T14:27:07.843 INFO:tasks.rados.rados.0.smithi151.stdout:1508: finishing write tid 4 to smithi15186934-36 2024-09-13T14:27:07.843 INFO:tasks.rados.rados.0.smithi151.stdout:1508: finishing write tid 5 to smithi15186934-36 2024-09-13T14:27:07.843 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 36 v 360 (ObjNum 588 snap 190 seq_num 588) dirty exists 2024-09-13T14:27:07.843 INFO:tasks.rados.rados.0.smithi151.stdout:1508: left oid 36 (ObjNum 588 snap 190 seq_num 588) 2024-09-13T14:27:07.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:07 smithi184 ceph-mon[92527]: osdmap e436: 8 total, 8 up, 8 in 2024-09-13T14:27:07.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:07 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:07.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:07 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:07.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:07 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:07.856 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:07 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:27:07.856 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:07 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:07.856 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:07 smithi184 ceph-mon[92527]: Reconfiguring mon.a (monmap changed)... 2024-09-13T14:27:07.856 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:07 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:27:07.856 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:07 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:27:07.856 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:07 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:07.856 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:07 smithi184 ceph-mon[92527]: Reconfiguring daemon mon.a on smithi151 2024-09-13T14:27:07.856 INFO:tasks.rados.rados.0.smithi151.stdout:1510: expect (ObjNum 584 snap 190 seq_num 584) 2024-09-13T14:27:08.718 INFO:tasks.rados.rados.0.smithi151.stdout:1506: done (3 left) 2024-09-13T14:27:08.718 INFO:tasks.rados.rados.0.smithi151.stdout:1508: done (2 left) 2024-09-13T14:27:08.718 INFO:tasks.rados.rados.0.smithi151.stdout:1510: done (1 left) 2024-09-13T14:27:08.718 INFO:tasks.rados.rados.0.smithi151.stdout:1511: done (0 left) 2024-09-13T14:27:08.718 INFO:tasks.rados.rados.0.smithi151.stdout:1512: delete oid 15 current snap is 191 2024-09-13T14:27:08.724 INFO:tasks.rados.rados.0.smithi151.stdout:1512: done (0 left) 2024-09-13T14:27:08.725 INFO:tasks.rados.rados.0.smithi151.stdout:1513: read oid 24 snap -1 2024-09-13T14:27:08.725 INFO:tasks.rados.rados.0.smithi151.stdout:1513: expect (ObjNum 583 snap 190 seq_num 583) 2024-09-13T14:27:08.725 INFO:tasks.rados.rados.0.smithi151.stdout:1514: read oid 34 snap 190 2024-09-13T14:27:08.725 INFO:tasks.rados.rados.0.smithi151.stdout:1514: expect (ObjNum 539 snap 173 seq_num 539) 2024-09-13T14:27:08.725 INFO:tasks.rados.rados.0.smithi151.stdout:1515: snap_remove snap 177 2024-09-13T14:27:08.822 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:08 smithi151 ceph-mon[104450]: pgmap v273: 89 pgs: 89 active+clean; 297 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.7 MiB/s wr, 10 op/s 2024-09-13T14:27:08.822 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:08 smithi151 ceph-mon[104450]: osdmap e437: 8 total, 8 up, 8 in 2024-09-13T14:27:08.822 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:08 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:08.822 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:08 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:08.823 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:08 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-13T14:27:08.823 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:08 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-13T14:27:08.823 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:08 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:08.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:08 smithi151 ceph-mon[108966]: pgmap v273: 89 pgs: 89 active+clean; 297 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.7 MiB/s wr, 10 op/s 2024-09-13T14:27:08.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:08 smithi151 ceph-mon[108966]: osdmap e437: 8 total, 8 up, 8 in 2024-09-13T14:27:08.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:08 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:08.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:08 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:08.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:08 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-13T14:27:08.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:08 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-13T14:27:08.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:08 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:08.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:08 smithi184 ceph-mon[92527]: pgmap v273: 89 pgs: 89 active+clean; 297 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.7 MiB/s wr, 10 op/s 2024-09-13T14:27:08.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:08 smithi184 ceph-mon[92527]: osdmap e437: 8 total, 8 up, 8 in 2024-09-13T14:27:08.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:08 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:08.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:08 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:08.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:08 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-13T14:27:08.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:08 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-13T14:27:08.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:08 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:09.485 INFO:tasks.rados.rados.0.smithi151.stdout:1515: done (2 left) 2024-09-13T14:27:09.486 INFO:tasks.rados.rados.0.smithi151.stdout:1516: write oid 29 current snap is 191 2024-09-13T14:27:09.486 INFO:tasks.rados.rados.0.smithi151.stdout:1516: seq_num 591 ranges {732937=631340,1850858=433541,2796871=671289,3888195=44742} 2024-09-13T14:27:09.494 INFO:tasks.rados.rados.0.smithi151.stdout:1516: writing smithi15186934-29 from 732937 to 1364277 tid 1 2024-09-13T14:27:09.499 INFO:tasks.rados.rados.0.smithi151.stdout:1516: writing smithi15186934-29 from 1850858 to 2284399 tid 2 2024-09-13T14:27:09.506 INFO:tasks.rados.rados.0.smithi151.stdout:1516: writing smithi15186934-29 from 2796871 to 3468160 tid 3 2024-09-13T14:27:09.508 INFO:tasks.rados.rados.0.smithi151.stdout:1516: writing smithi15186934-29 from 3888195 to 3932937 tid 4 2024-09-13T14:27:09.509 INFO:tasks.rados.rados.0.smithi151.stdout:1517: read oid 31 snap -1 2024-09-13T14:27:09.509 INFO:tasks.rados.rados.0.smithi151.stdout:1517: expect deleted 2024-09-13T14:27:09.509 INFO:tasks.rados.rados.0.smithi151.stdout:1518: snap_remove snap 187 2024-09-13T14:27:09.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:09 smithi151 ceph-mon[104450]: Reconfiguring mgr.y (monmap changed)... 2024-09-13T14:27:09.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:09 smithi151 ceph-mon[104450]: Reconfiguring daemon mgr.y on smithi151 2024-09-13T14:27:09.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:09 smithi151 ceph-mon[104450]: osdmap e438: 8 total, 8 up, 8 in 2024-09-13T14:27:09.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:09 smithi151 ceph-mon[104450]: osdmap e439: 8 total, 8 up, 8 in 2024-09-13T14:27:09.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:09 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:09.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:09 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:09.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:09 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:27:09.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:09 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:27:09.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:09 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:09.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:09 smithi151 ceph-mon[108966]: Reconfiguring mgr.y (monmap changed)... 2024-09-13T14:27:09.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:09 smithi151 ceph-mon[108966]: Reconfiguring daemon mgr.y on smithi151 2024-09-13T14:27:09.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:09 smithi151 ceph-mon[108966]: osdmap e438: 8 total, 8 up, 8 in 2024-09-13T14:27:09.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:09 smithi151 ceph-mon[108966]: osdmap e439: 8 total, 8 up, 8 in 2024-09-13T14:27:09.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:09 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:09.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:09 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:09.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:09 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:27:09.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:09 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:27:09.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:09 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:10.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:09 smithi184 ceph-mon[92527]: Reconfiguring mgr.y (monmap changed)... 2024-09-13T14:27:10.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:09 smithi184 ceph-mon[92527]: Reconfiguring daemon mgr.y on smithi151 2024-09-13T14:27:10.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:09 smithi184 ceph-mon[92527]: osdmap e438: 8 total, 8 up, 8 in 2024-09-13T14:27:10.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:09 smithi184 ceph-mon[92527]: osdmap e439: 8 total, 8 up, 8 in 2024-09-13T14:27:10.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:09 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:10.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:09 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:10.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:09 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:27:10.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:09 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:27:10.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:09 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:10.484 INFO:tasks.rados.rados.0.smithi151.stdout:1518: done (4 left) 2024-09-13T14:27:10.484 INFO:tasks.rados.rados.0.smithi151.stdout:1519: write oid 4 current snap is 191 2024-09-13T14:27:10.484 INFO:tasks.rados.rados.0.smithi151.stdout:1514: expect (ObjNum 539 snap 173 seq_num 539) 2024-09-13T14:27:10.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:27:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:27:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:10.625 INFO:tasks.rados.rados.0.smithi151.stdout:1513: expect (ObjNum 583 snap 190 seq_num 583) 2024-09-13T14:27:10.829 INFO:tasks.rados.rados.0.smithi151.stdout:1516: finishing write tid 1 to smithi15186934-29 2024-09-13T14:27:10.829 INFO:tasks.rados.rados.0.smithi151.stdout:1516: finishing write tid 2 to smithi15186934-29 2024-09-13T14:27:10.829 INFO:tasks.rados.rados.0.smithi151.stdout:1519: seq_num 592 ranges {749744=706589,1909340=577199,3224294=482561} 2024-09-13T14:27:10.836 INFO:tasks.rados.rados.0.smithi151.stdout:1519: writing smithi15186934-4 from 749744 to 1456333 tid 1 2024-09-13T14:27:10.842 INFO:tasks.rados.rados.0.smithi151.stdout:1519: writing smithi15186934-4 from 1909340 to 2486539 tid 2 2024-09-13T14:27:10.849 INFO:tasks.rados.rados.0.smithi151.stdout:1519: writing smithi15186934-4 from 3224294 to 3706855 tid 3 2024-09-13T14:27:10.849 INFO:tasks.rados.rados.0.smithi151.stdout:1513: done (4 left) 2024-09-13T14:27:10.849 INFO:tasks.rados.rados.0.smithi151.stdout:1514: done (3 left) 2024-09-13T14:27:10.849 INFO:tasks.rados.rados.0.smithi151.stdout:1517: done (2 left) 2024-09-13T14:27:10.849 INFO:tasks.rados.rados.0.smithi151.stdout:1520: read oid 3 snap -1 2024-09-13T14:27:10.850 INFO:tasks.rados.rados.0.smithi151.stdout:1520: expect (ObjNum 334 snap 103 seq_num 334) 2024-09-13T14:27:10.850 INFO:tasks.rados.rados.0.smithi151.stdout:1521: snap_remove snap 176 2024-09-13T14:27:11.034 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:10 smithi151 ceph-mon[108966]: pgmap v276: 89 pgs: 89 active+clean; 297 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.7 MiB/s wr, 10 op/s 2024-09-13T14:27:11.035 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:10 smithi151 ceph-mon[108966]: Reconfiguring mon.c (monmap changed)... 2024-09-13T14:27:11.035 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:10 smithi151 ceph-mon[108966]: Reconfiguring daemon mon.c on smithi151 2024-09-13T14:27:11.035 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:10 smithi151 ceph-mon[108966]: osdmap e440: 8 total, 8 up, 8 in 2024-09-13T14:27:11.035 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:10 smithi151 ceph-mon[104450]: pgmap v276: 89 pgs: 89 active+clean; 297 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.7 MiB/s wr, 10 op/s 2024-09-13T14:27:11.035 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:10 smithi151 ceph-mon[104450]: Reconfiguring mon.c (monmap changed)... 2024-09-13T14:27:11.035 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:10 smithi151 ceph-mon[104450]: Reconfiguring daemon mon.c on smithi151 2024-09-13T14:27:11.035 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:10 smithi151 ceph-mon[104450]: osdmap e440: 8 total, 8 up, 8 in 2024-09-13T14:27:11.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:10 smithi184 ceph-mon[92527]: pgmap v276: 89 pgs: 89 active+clean; 297 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.7 MiB/s wr, 10 op/s 2024-09-13T14:27:11.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:10 smithi184 ceph-mon[92527]: Reconfiguring mon.c (monmap changed)... 2024-09-13T14:27:11.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:10 smithi184 ceph-mon[92527]: Reconfiguring daemon mon.c on smithi151 2024-09-13T14:27:11.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:10 smithi184 ceph-mon[92527]: osdmap e440: 8 total, 8 up, 8 in 2024-09-13T14:27:11.738 INFO:tasks.rados.rados.0.smithi151.stdout:1521: done (3 left) 2024-09-13T14:27:11.738 INFO:tasks.rados.rados.0.smithi151.stdout:1522: rollback oid 27 current snap is 191 2024-09-13T14:27:11.738 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 27 to 186 2024-09-13T14:27:11.738 INFO:tasks.rados.rados.0.smithi151.stdout:1523: write oid 39 current snap is 191 2024-09-13T14:27:11.738 INFO:tasks.rados.rados.0.smithi151.stdout:1523: seq_num 593 ranges {766551=498191,1684175=720857,2851717=693833} 2024-09-13T14:27:11.744 INFO:tasks.rados.rados.0.smithi151.stdout:1523: writing smithi15186934-39 from 766551 to 1264742 tid 1 2024-09-13T14:27:11.751 INFO:tasks.rados.rados.0.smithi151.stdout:1523: writing smithi15186934-39 from 1684175 to 2405032 tid 2 2024-09-13T14:27:11.759 INFO:tasks.rados.rados.0.smithi151.stdout:1523: writing smithi15186934-39 from 2851717 to 3545550 tid 3 2024-09-13T14:27:11.759 INFO:tasks.rados.rados.0.smithi151.stdout:1524: write oid 5 current snap is 191 2024-09-13T14:27:11.759 INFO:tasks.rados.rados.0.smithi151.stdout:1516: finishing write tid 3 to smithi15186934-29 2024-09-13T14:27:11.759 INFO:tasks.rados.rados.0.smithi151.stdout:1516: finishing write tid 4 to smithi15186934-29 2024-09-13T14:27:11.759 INFO:tasks.rados.rados.0.smithi151.stdout:1524: seq_num 594 ranges {783358=573440,2026304=580868,3279140=621458} 2024-09-13T14:27:11.766 INFO:tasks.rados.rados.0.smithi151.stdout:1524: writing smithi15186934-5 from 783358 to 1356798 tid 1 2024-09-13T14:27:11.773 INFO:tasks.rados.rados.0.smithi151.stdout:1524: writing smithi15186934-5 from 2026304 to 2607172 tid 2 2024-09-13T14:27:11.781 INFO:tasks.rados.rados.0.smithi151.stdout:1524: writing smithi15186934-5 from 3279140 to 3900598 tid 3 2024-09-13T14:27:11.781 INFO:tasks.rados.rados.0.smithi151.stdout:1525: write oid 9 current snap is 191 2024-09-13T14:27:11.781 INFO:tasks.rados.rados.0.smithi151.stdout:1516: finishing write tid 5 to smithi15186934-29 2024-09-13T14:27:11.781 INFO:tasks.rados.rados.0.smithi151.stdout:1516: finishing write tid 6 to smithi15186934-29 2024-09-13T14:27:11.781 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 29 v 410 (ObjNum 591 snap 191 seq_num 591) dirty exists 2024-09-13T14:27:11.781 INFO:tasks.rados.rados.0.smithi151.stdout:1516: left oid 29 (ObjNum 591 snap 191 seq_num 591) 2024-09-13T14:27:11.781 INFO:tasks.rados.rados.0.smithi151.stdout:1525: seq_num 595 ranges {400165=648689,1684786=315379} 2024-09-13T14:27:11.786 INFO:tasks.rados.rados.0.smithi151.stdout:1525: writing smithi15186934-9 from 400165 to 1048854 tid 1 2024-09-13T14:27:11.791 INFO:tasks.rados.rados.0.smithi151.stdout:1525: writing smithi15186934-9 from 1684786 to 2000165 tid 2 2024-09-13T14:27:11.791 INFO:tasks.rados.rados.0.smithi151.stdout:1516: done (6 left) 2024-09-13T14:27:11.792 INFO:tasks.rados.rados.0.smithi151.stdout:1526: snap_remove snap 178 2024-09-13T14:27:12.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:12 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:12.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:12 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:12.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:12 smithi184 ceph-mon[92527]: Reconfiguring osd.0 (monmap changed)... 2024-09-13T14:27:12.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:12 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-13T14:27:12.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:12 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:12.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:12 smithi184 ceph-mon[92527]: Reconfiguring daemon osd.0 on smithi151 2024-09-13T14:27:12.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:12 smithi184 ceph-mon[92527]: pgmap v279: 89 pgs: 89 active+clean; 297 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:27:12.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:12 smithi184 ceph-mon[92527]: osdmap e441: 8 total, 8 up, 8 in 2024-09-13T14:27:12.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:12 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:12.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:12 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:12.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:12 smithi151 ceph-mon[108966]: Reconfiguring osd.0 (monmap changed)... 2024-09-13T14:27:12.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:12 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-13T14:27:12.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:12 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:12.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:12 smithi151 ceph-mon[108966]: Reconfiguring daemon osd.0 on smithi151 2024-09-13T14:27:12.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:12 smithi151 ceph-mon[108966]: pgmap v279: 89 pgs: 89 active+clean; 297 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:27:12.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:12 smithi151 ceph-mon[108966]: osdmap e441: 8 total, 8 up, 8 in 2024-09-13T14:27:12.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:12 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:12.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:12 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:12.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:12 smithi151 ceph-mon[104450]: Reconfiguring osd.0 (monmap changed)... 2024-09-13T14:27:12.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:12 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-13T14:27:12.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:12 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:12.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:12 smithi151 ceph-mon[104450]: Reconfiguring daemon osd.0 on smithi151 2024-09-13T14:27:12.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:12 smithi151 ceph-mon[104450]: pgmap v279: 89 pgs: 89 active+clean; 297 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:27:12.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:12 smithi151 ceph-mon[104450]: osdmap e441: 8 total, 8 up, 8 in 2024-09-13T14:27:12.732 INFO:tasks.rados.rados.0.smithi151.stdout:1526: done (6 left) 2024-09-13T14:27:12.732 INFO:tasks.rados.rados.0.smithi151.stdout:1527: delete oid 25 current snap is 191 2024-09-13T14:27:12.732 INFO:tasks.rados.rados.0.smithi151.stdout:1519: finishing write tid 1 to smithi15186934-4 2024-09-13T14:27:12.732 INFO:tasks.rados.rados.0.smithi151.stdout:1519: finishing write tid 2 to smithi15186934-4 2024-09-13T14:27:12.732 INFO:tasks.rados.rados.0.smithi151.stdout:1519: finishing write tid 3 to smithi15186934-4 2024-09-13T14:27:12.732 INFO:tasks.rados.rados.0.smithi151.stdout:1519: finishing write tid 4 to smithi15186934-4 2024-09-13T14:27:12.732 INFO:tasks.rados.rados.0.smithi151.stdout:1519: finishing write tid 5 to smithi15186934-4 2024-09-13T14:27:12.733 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 894 (ObjNum 592 snap 191 seq_num 592) dirty exists 2024-09-13T14:27:12.733 INFO:tasks.rados.rados.0.smithi151.stdout:1519: left oid 4 (ObjNum 592 snap 191 seq_num 592) 2024-09-13T14:27:12.733 INFO:tasks.rados.rados.0.smithi151.stdout:1520: expect (ObjNum 334 snap 103 seq_num 334) 2024-09-13T14:27:12.854 INFO:tasks.rados.rados.0.smithi151.stdout:1522: finishing rollback tid 0 to smithi15186934-27 2024-09-13T14:27:12.854 INFO:tasks.rados.rados.0.smithi151.stdout:1522: finishing rollback tid 1 to smithi15186934-27 2024-09-13T14:27:12.854 INFO:tasks.rados.rados.0.smithi151.stdout:1522: finishing rollback tid 2 to smithi15186934-27 2024-09-13T14:27:12.854 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 27 v 399 (ObjNum 559 snap 177 seq_num 559) dirty exists 2024-09-13T14:27:12.854 INFO:tasks.rados.rados.0.smithi151.stdout:1523: finishing write tid 1 to smithi15186934-39 2024-09-13T14:27:12.854 INFO:tasks.rados.rados.0.smithi151.stdout:1519: done (6 left) 2024-09-13T14:27:12.854 INFO:tasks.rados.rados.0.smithi151.stdout:1520: done (5 left) 2024-09-13T14:27:12.854 INFO:tasks.rados.rados.0.smithi151.stdout:1522: done (4 left) 2024-09-13T14:27:12.855 INFO:tasks.rados.rados.0.smithi151.stdout:1527: done (3 left) 2024-09-13T14:27:12.855 INFO:tasks.rados.rados.0.smithi151.stdout:1528: read oid 33 snap -1 2024-09-13T14:27:12.855 INFO:tasks.rados.rados.0.smithi151.stdout:1528: expect (ObjNum 508 snap 164 seq_num 508) 2024-09-13T14:27:12.855 INFO:tasks.rados.rados.0.smithi151.stdout:1529: write oid 7 current snap is 191 2024-09-13T14:27:12.855 INFO:tasks.rados.rados.0.smithi151.stdout:1529: seq_num 597 ranges {433779=799187,1801750=232029} 2024-09-13T14:27:12.860 INFO:tasks.rados.rados.0.smithi151.stdout:1529: writing smithi15186934-7 from 433779 to 1232966 tid 1 2024-09-13T14:27:12.866 INFO:tasks.rados.rados.0.smithi151.stdout:1529: writing smithi15186934-7 from 1801750 to 2033779 tid 2 2024-09-13T14:27:12.866 INFO:tasks.rados.rados.0.smithi151.stdout:1530: snap_create 2024-09-13T14:27:12.866 INFO:tasks.rados.rados.0.smithi151.stdout:1523: finishing write tid 2 to smithi15186934-39 2024-09-13T14:27:12.866 INFO:tasks.rados.rados.0.smithi151.stdout:1523: finishing write tid 3 to smithi15186934-39 2024-09-13T14:27:12.866 INFO:tasks.rados.rados.0.smithi151.stdout:1523: finishing write tid 4 to smithi15186934-39 2024-09-13T14:27:12.866 INFO:tasks.rados.rados.0.smithi151.stdout:1523: finishing write tid 5 to smithi15186934-39 2024-09-13T14:27:12.866 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 492 (ObjNum 593 snap 191 seq_num 593) dirty exists 2024-09-13T14:27:12.866 INFO:tasks.rados.rados.0.smithi151.stdout:1523: left oid 39 (ObjNum 593 snap 191 seq_num 593) 2024-09-13T14:27:12.866 INFO:tasks.rados.rados.0.smithi151.stdout:1524: finishing write tid 1 to smithi15186934-5 2024-09-13T14:27:12.866 INFO:tasks.rados.rados.0.smithi151.stdout:1524: finishing write tid 2 to smithi15186934-5 2024-09-13T14:27:12.866 INFO:tasks.rados.rados.0.smithi151.stdout:1524: finishing write tid 3 to smithi15186934-5 2024-09-13T14:27:12.866 INFO:tasks.rados.rados.0.smithi151.stdout:1524: finishing write tid 4 to smithi15186934-5 2024-09-13T14:27:12.867 INFO:tasks.rados.rados.0.smithi151.stdout:1524: finishing write tid 5 to smithi15186934-5 2024-09-13T14:27:12.867 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 569 (ObjNum 594 snap 191 seq_num 594) dirty exists 2024-09-13T14:27:12.867 INFO:tasks.rados.rados.0.smithi151.stdout:1524: left oid 5 (ObjNum 594 snap 191 seq_num 594) 2024-09-13T14:27:12.867 INFO:tasks.rados.rados.0.smithi151.stdout:1525: finishing write tid 1 to smithi15186934-9 2024-09-13T14:27:12.867 INFO:tasks.rados.rados.0.smithi151.stdout:1525: finishing write tid 2 to smithi15186934-9 2024-09-13T14:27:12.867 INFO:tasks.rados.rados.0.smithi151.stdout:1525: finishing write tid 3 to smithi15186934-9 2024-09-13T14:27:12.867 INFO:tasks.rados.rados.0.smithi151.stdout:1525: finishing write tid 4 to smithi15186934-9 2024-09-13T14:27:12.867 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 632 (ObjNum 595 snap 191 seq_num 595) dirty exists 2024-09-13T14:27:12.867 INFO:tasks.rados.rados.0.smithi151.stdout:1525: left oid 9 (ObjNum 595 snap 191 seq_num 595) 2024-09-13T14:27:12.867 INFO:tasks.rados.rados.0.smithi151.stdout:1528: expect (ObjNum 508 snap 164 seq_num 508) 2024-09-13T14:27:12.948 INFO:tasks.rados.rados.0.smithi151.stdout:1529: finishing write tid 1 to smithi15186934-7 2024-09-13T14:27:12.948 INFO:tasks.rados.rados.0.smithi151.stdout:1529: finishing write tid 2 to smithi15186934-7 2024-09-13T14:27:12.949 INFO:tasks.rados.rados.0.smithi151.stdout:1529: finishing write tid 3 to smithi15186934-7 2024-09-13T14:27:12.949 INFO:tasks.rados.rados.0.smithi151.stdout:1529: finishing write tid 4 to smithi15186934-7 2024-09-13T14:27:12.949 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 1044 (ObjNum 597 snap 191 seq_num 597) dirty exists 2024-09-13T14:27:12.949 INFO:tasks.rados.rados.0.smithi151.stdout:1529: left oid 7 (ObjNum 597 snap 191 seq_num 597) 2024-09-13T14:27:13.736 INFO:tasks.rados.rados.0.smithi151.stdout:1523: done (5 left) 2024-09-13T14:27:13.737 INFO:tasks.rados.rados.0.smithi151.stdout:1524: done (4 left) 2024-09-13T14:27:13.737 INFO:tasks.rados.rados.0.smithi151.stdout:1525: done (3 left) 2024-09-13T14:27:13.737 INFO:tasks.rados.rados.0.smithi151.stdout:1528: done (2 left) 2024-09-13T14:27:13.737 INFO:tasks.rados.rados.0.smithi151.stdout:1529: done (1 left) 2024-09-13T14:27:13.737 INFO:tasks.rados.rados.0.smithi151.stdout:1530: done (0 left) 2024-09-13T14:27:13.737 INFO:tasks.rados.rados.0.smithi151.stdout:1531: write oid 26 current snap is 192 2024-09-13T14:27:13.737 INFO:tasks.rados.rados.0.smithi151.stdout:1531: seq_num 598 ranges {450586=474436,1343879=588206} 2024-09-13T14:27:13.741 INFO:tasks.rados.rados.0.smithi151.stdout:1531: writing smithi15186934-26 from 450586 to 925022 tid 1 2024-09-13T14:27:13.747 INFO:tasks.rados.rados.0.smithi151.stdout:1531: writing smithi15186934-26 from 1343879 to 1932085 tid 2 2024-09-13T14:27:13.747 INFO:tasks.rados.rados.0.smithi151.stdout:1532: snap_remove snap 141 2024-09-13T14:27:13.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:13 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:13.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:13 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:13.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:13 smithi184 ceph-mon[92527]: Reconfiguring osd.1 (monmap changed)... 2024-09-13T14:27:13.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:13 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-13T14:27:13.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:13 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:13.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:13 smithi184 ceph-mon[92527]: Reconfiguring daemon osd.1 on smithi151 2024-09-13T14:27:13.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:13 smithi184 ceph-mon[92527]: osdmap e442: 8 total, 8 up, 8 in 2024-09-13T14:27:14.033 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:13 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:14.033 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:13 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:14.033 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:13 smithi151 ceph-mon[104450]: Reconfiguring osd.1 (monmap changed)... 2024-09-13T14:27:14.033 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:13 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-13T14:27:14.033 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:13 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:14.033 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:13 smithi151 ceph-mon[104450]: Reconfiguring daemon osd.1 on smithi151 2024-09-13T14:27:14.033 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:13 smithi151 ceph-mon[104450]: osdmap e442: 8 total, 8 up, 8 in 2024-09-13T14:27:14.033 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:13 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:14.033 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:13 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:14.034 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:13 smithi151 ceph-mon[108966]: Reconfiguring osd.1 (monmap changed)... 2024-09-13T14:27:14.034 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:13 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-13T14:27:14.034 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:13 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:14.034 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:13 smithi151 ceph-mon[108966]: Reconfiguring daemon osd.1 on smithi151 2024-09-13T14:27:14.034 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:13 smithi151 ceph-mon[108966]: osdmap e442: 8 total, 8 up, 8 in 2024-09-13T14:27:14.484 INFO:tasks.rados.rados.0.smithi151.stdout:1532: done (1 left) 2024-09-13T14:27:14.485 INFO:tasks.rados.rados.0.smithi151.stdout:1533: delete oid 22 current snap is 192 2024-09-13T14:27:14.485 INFO:tasks.rados.rados.0.smithi151.stdout:1531: finishing write tid 1 to smithi15186934-26 2024-09-13T14:27:14.485 INFO:tasks.rados.rados.0.smithi151.stdout:1531: finishing write tid 2 to smithi15186934-26 2024-09-13T14:27:14.485 INFO:tasks.rados.rados.0.smithi151.stdout:1531: finishing write tid 3 to smithi15186934-26 2024-09-13T14:27:14.485 INFO:tasks.rados.rados.0.smithi151.stdout:1531: finishing write tid 4 to smithi15186934-26 2024-09-13T14:27:14.485 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 599 (ObjNum 598 snap 192 seq_num 598) dirty exists 2024-09-13T14:27:14.485 INFO:tasks.rados.rados.0.smithi151.stdout:1531: left oid 26 (ObjNum 598 snap 192 seq_num 598) 2024-09-13T14:27:14.494 INFO:tasks.rados.rados.0.smithi151.stdout:1531: done (1 left) 2024-09-13T14:27:14.494 INFO:tasks.rados.rados.0.smithi151.stdout:1533: done (0 left) 2024-09-13T14:27:14.494 INFO:tasks.rados.rados.0.smithi151.stdout:1534: snap_create 2024-09-13T14:27:14.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:27:14 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:27:14.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:14.741 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:14 smithi151 ceph-mon[104450]: pgmap v282: 89 pgs: 89 active+clean; 304 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 18 MiB/s rd, 2.4 MiB/s wr, 19 op/s 2024-09-13T14:27:14.742 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:14 smithi151 ceph-mon[104450]: osdmap e443: 8 total, 8 up, 8 in 2024-09-13T14:27:14.742 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:14 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:14.742 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:14 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:14.742 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:14 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-13T14:27:14.742 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:14 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:14.742 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:14 smithi151 ceph-mon[104450]: osdmap e444: 8 total, 8 up, 8 in 2024-09-13T14:27:14.742 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:27:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:27:14] "GET /metrics HTTP/1.1" 200 36332 "" "Prometheus/2.51.0" 2024-09-13T14:27:14.742 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:14 smithi151 ceph-mon[108966]: pgmap v282: 89 pgs: 89 active+clean; 304 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 18 MiB/s rd, 2.4 MiB/s wr, 19 op/s 2024-09-13T14:27:14.743 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:14 smithi151 ceph-mon[108966]: osdmap e443: 8 total, 8 up, 8 in 2024-09-13T14:27:14.743 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:14 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:14.743 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:14 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:14.743 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:14 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-13T14:27:14.743 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:14 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:14.743 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:14 smithi151 ceph-mon[108966]: osdmap e444: 8 total, 8 up, 8 in 2024-09-13T14:27:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:14 smithi184 ceph-mon[92527]: pgmap v282: 89 pgs: 89 active+clean; 304 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 18 MiB/s rd, 2.4 MiB/s wr, 19 op/s 2024-09-13T14:27:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:14 smithi184 ceph-mon[92527]: osdmap e443: 8 total, 8 up, 8 in 2024-09-13T14:27:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:14 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:15.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:14 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:15.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:14 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-13T14:27:15.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:14 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:15.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:14 smithi184 ceph-mon[92527]: osdmap e444: 8 total, 8 up, 8 in 2024-09-13T14:27:15.486 INFO:tasks.rados.rados.0.smithi151.stdout:1534: done (0 left) 2024-09-13T14:27:15.487 INFO:tasks.rados.rados.0.smithi151.stdout:1535: snap_remove snap 192 2024-09-13T14:27:15.742 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:15 smithi151 ceph-mon[104450]: Reconfiguring osd.2 (monmap changed)... 2024-09-13T14:27:15.742 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:15 smithi151 ceph-mon[104450]: Reconfiguring daemon osd.2 on smithi151 2024-09-13T14:27:15.742 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:15 smithi151 ceph-mon[104450]: osdmap e445: 8 total, 8 up, 8 in 2024-09-13T14:27:16.000 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:15 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:16.000 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:15 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:16.000 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:15 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-13T14:27:16.000 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:15 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:16.000 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:15 smithi151 ceph-mon[108966]: Reconfiguring osd.2 (monmap changed)... 2024-09-13T14:27:16.001 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:15 smithi151 ceph-mon[108966]: Reconfiguring daemon osd.2 on smithi151 2024-09-13T14:27:16.001 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:15 smithi151 ceph-mon[108966]: osdmap e445: 8 total, 8 up, 8 in 2024-09-13T14:27:16.001 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:15 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:16.001 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:15 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:16.001 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:15 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-13T14:27:16.001 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:15 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:16.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:15 smithi184 ceph-mon[92527]: Reconfiguring osd.2 (monmap changed)... 2024-09-13T14:27:16.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:15 smithi184 ceph-mon[92527]: Reconfiguring daemon osd.2 on smithi151 2024-09-13T14:27:16.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:15 smithi184 ceph-mon[92527]: osdmap e445: 8 total, 8 up, 8 in 2024-09-13T14:27:16.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:15 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:16.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:15 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:16.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:15 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-13T14:27:16.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:15 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:16.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:27:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:27:15.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:16.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:27:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:27:15.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:16.488 INFO:tasks.rados.rados.0.smithi151.stdout:1535: done (0 left) 2024-09-13T14:27:16.488 INFO:tasks.rados.rados.0.smithi151.stdout:1536: read oid 45 snap -1 2024-09-13T14:27:16.488 INFO:tasks.rados.rados.0.smithi151.stdout:1536: expect deleted 2024-09-13T14:27:16.488 INFO:tasks.rados.rados.0.smithi151.stdout:1537: snap_remove snap 191 2024-09-13T14:27:17.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:16 smithi151 ceph-mon[104450]: pgmap v285: 89 pgs: 89 active+clean; 304 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 18 MiB/s rd, 2.4 MiB/s wr, 19 op/s 2024-09-13T14:27:17.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:16 smithi151 ceph-mon[104450]: Reconfiguring osd.3 (monmap changed)... 2024-09-13T14:27:17.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:16 smithi151 ceph-mon[104450]: Reconfiguring daemon osd.3 on smithi151 2024-09-13T14:27:17.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:17.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:16 smithi151 ceph-mon[104450]: osdmap e446: 8 total, 8 up, 8 in 2024-09-13T14:27:17.057 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:16 smithi151 ceph-mon[108966]: pgmap v285: 89 pgs: 89 active+clean; 304 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 18 MiB/s rd, 2.4 MiB/s wr, 19 op/s 2024-09-13T14:27:17.057 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:16 smithi151 ceph-mon[108966]: Reconfiguring osd.3 (monmap changed)... 2024-09-13T14:27:17.057 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:16 smithi151 ceph-mon[108966]: Reconfiguring daemon osd.3 on smithi151 2024-09-13T14:27:17.057 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:17.058 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:16 smithi151 ceph-mon[108966]: osdmap e446: 8 total, 8 up, 8 in 2024-09-13T14:27:17.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:16 smithi184 ceph-mon[92527]: pgmap v285: 89 pgs: 89 active+clean; 304 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 18 MiB/s rd, 2.4 MiB/s wr, 19 op/s 2024-09-13T14:27:17.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:16 smithi184 ceph-mon[92527]: Reconfiguring osd.3 (monmap changed)... 2024-09-13T14:27:17.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:16 smithi184 ceph-mon[92527]: Reconfiguring daemon osd.3 on smithi151 2024-09-13T14:27:17.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:16 smithi184 ceph-mon[92527]: osdmap e446: 8 total, 8 up, 8 in 2024-09-13T14:27:17.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:27:16 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:27:16.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi184\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:17.497 INFO:tasks.rados.rados.0.smithi151.stdout:1537: done (1 left) 2024-09-13T14:27:17.497 INFO:tasks.rados.rados.0.smithi151.stdout:1538: read oid 38 snap -1 2024-09-13T14:27:17.497 INFO:tasks.rados.rados.0.smithi151.stdout:1538: expect (ObjNum 504 snap 162 seq_num 504) 2024-09-13T14:27:17.497 INFO:tasks.rados.rados.0.smithi151.stdout:1539: snap_remove snap 133 2024-09-13T14:27:18.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:18 smithi151 ceph-mon[104450]: pgmap v288: 89 pgs: 89 active+clean; 292 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.8 MiB/s rd, 512 KiB/s wr, 5 op/s 2024-09-13T14:27:18.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:18 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:18.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:18 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:18.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:18 smithi151 ceph-mon[104450]: Reconfiguring mds.foo.smithi151.rqqklw (monmap changed)... 2024-09-13T14:27:18.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:18 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi151.rqqklw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-13T14:27:18.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:18 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:18.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:18 smithi151 ceph-mon[104450]: Reconfiguring daemon mds.foo.smithi151.rqqklw on smithi151 2024-09-13T14:27:18.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:18 smithi151 ceph-mon[104450]: osdmap e447: 8 total, 8 up, 8 in 2024-09-13T14:27:18.449 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:18 smithi151 ceph-mon[108966]: pgmap v288: 89 pgs: 89 active+clean; 292 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.8 MiB/s rd, 512 KiB/s wr, 5 op/s 2024-09-13T14:27:18.449 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:18 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:18.449 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:18 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:18.449 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:18 smithi151 ceph-mon[108966]: Reconfiguring mds.foo.smithi151.rqqklw (monmap changed)... 2024-09-13T14:27:18.449 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:18 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi151.rqqklw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-13T14:27:18.449 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:18 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:18.449 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:18 smithi151 ceph-mon[108966]: Reconfiguring daemon mds.foo.smithi151.rqqklw on smithi151 2024-09-13T14:27:18.449 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:18 smithi151 ceph-mon[108966]: osdmap e447: 8 total, 8 up, 8 in 2024-09-13T14:27:18.494 INFO:tasks.rados.rados.0.smithi151.stdout:1539: done (2 left) 2024-09-13T14:27:18.494 INFO:tasks.rados.rados.0.smithi151.stdout:1540: read oid 15 snap -1 2024-09-13T14:27:18.494 INFO:tasks.rados.rados.0.smithi151.stdout:1540: expect deleted 2024-09-13T14:27:18.494 INFO:tasks.rados.rados.0.smithi151.stdout:1541: read oid 28 snap 181 2024-09-13T14:27:18.494 INFO:tasks.rados.rados.0.smithi151.stdout:1541: expect deleted 2024-09-13T14:27:18.494 INFO:tasks.rados.rados.0.smithi151.stdout:1542: read oid 7 snap 181 2024-09-13T14:27:18.495 INFO:tasks.rados.rados.0.smithi151.stdout:1542: expect (ObjNum 502 snap 161 seq_num 502) 2024-09-13T14:27:18.495 INFO:tasks.rados.rados.0.smithi151.stdout:1543: read oid 29 snap 161 2024-09-13T14:27:18.495 INFO:tasks.rados.rados.0.smithi151.stdout:1543: expect (ObjNum 445 snap 141 seq_num 445) 2024-09-13T14:27:18.495 INFO:tasks.rados.rados.0.smithi151.stdout:1544: read oid 21 snap -1 2024-09-13T14:27:18.495 INFO:tasks.rados.rados.0.smithi151.stdout:1544: expect (ObjNum 578 snap 186 seq_num 578) 2024-09-13T14:27:18.495 INFO:tasks.rados.rados.0.smithi151.stdout:1545: read oid 46 snap 181 2024-09-13T14:27:18.495 INFO:tasks.rados.rados.0.smithi151.stdout:1545: expect deleted 2024-09-13T14:27:18.495 INFO:tasks.rados.rados.0.smithi151.stdout:1546: read oid 25 snap -1 2024-09-13T14:27:18.495 INFO:tasks.rados.rados.0.smithi151.stdout:1546: expect deleted 2024-09-13T14:27:18.495 INFO:tasks.rados.rados.0.smithi151.stdout:1547: delete oid 47 current snap is 193 2024-09-13T14:27:18.495 INFO:tasks.rados.rados.0.smithi151.stdout:1538: expect (ObjNum 504 snap 162 seq_num 504) 2024-09-13T14:27:18.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:18 smithi184 ceph-mon[92527]: pgmap v288: 89 pgs: 89 active+clean; 292 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.8 MiB/s rd, 512 KiB/s wr, 5 op/s 2024-09-13T14:27:18.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:18 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:18.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:18 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:18.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:18 smithi184 ceph-mon[92527]: Reconfiguring mds.foo.smithi151.rqqklw (monmap changed)... 2024-09-13T14:27:18.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:18 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi151.rqqklw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-13T14:27:18.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:18 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:18.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:18 smithi184 ceph-mon[92527]: Reconfiguring daemon mds.foo.smithi151.rqqklw on smithi151 2024-09-13T14:27:18.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:18 smithi184 ceph-mon[92527]: osdmap e447: 8 total, 8 up, 8 in 2024-09-13T14:27:18.619 INFO:tasks.rados.rados.0.smithi151.stdout:1543: expect (ObjNum 445 snap 141 seq_num 445) 2024-09-13T14:27:18.731 INFO:tasks.rados.rados.0.smithi151.stdout:1542: expect (ObjNum 502 snap 161 seq_num 502) 2024-09-13T14:27:18.839 INFO:tasks.rados.rados.0.smithi151.stdout:1544: expect (ObjNum 578 snap 186 seq_num 578) 2024-09-13T14:27:19.033 INFO:tasks.rados.rados.0.smithi151.stdout:1536: done (9 left) 2024-09-13T14:27:19.033 INFO:tasks.rados.rados.0.smithi151.stdout:1538: done (8 left) 2024-09-13T14:27:19.033 INFO:tasks.rados.rados.0.smithi151.stdout:1540: done (7 left) 2024-09-13T14:27:19.033 INFO:tasks.rados.rados.0.smithi151.stdout:1541: done (6 left) 2024-09-13T14:27:19.033 INFO:tasks.rados.rados.0.smithi151.stdout:1542: done (5 left) 2024-09-13T14:27:19.033 INFO:tasks.rados.rados.0.smithi151.stdout:1543: done (4 left) 2024-09-13T14:27:19.033 INFO:tasks.rados.rados.0.smithi151.stdout:1544: done (3 left) 2024-09-13T14:27:19.033 INFO:tasks.rados.rados.0.smithi151.stdout:1545: done (2 left) 2024-09-13T14:27:19.034 INFO:tasks.rados.rados.0.smithi151.stdout:1546: done (1 left) 2024-09-13T14:27:19.034 INFO:tasks.rados.rados.0.smithi151.stdout:1547: done (0 left) 2024-09-13T14:27:19.034 INFO:tasks.rados.rados.0.smithi151.stdout:1548: read oid 10 snap -1 2024-09-13T14:27:19.034 INFO:tasks.rados.rados.0.smithi151.stdout:1548: expect (ObjNum 575 snap 185 seq_num 575) 2024-09-13T14:27:19.034 INFO:tasks.rados.rados.0.smithi151.stdout:1549: write oid 27 current snap is 193 2024-09-13T14:27:19.034 INFO:tasks.rados.rados.0.smithi151.stdout:1549: seq_num 601 ranges {501007=416536,1635678=465329} 2024-09-13T14:27:19.037 INFO:tasks.rados.rados.0.smithi151.stdout:1549: writing smithi15186934-27 from 501007 to 917543 tid 1 2024-09-13T14:27:19.044 INFO:tasks.rados.rados.0.smithi151.stdout:1549: writing smithi15186934-27 from 1635678 to 2101007 tid 2 2024-09-13T14:27:19.044 INFO:tasks.rados.rados.0.smithi151.stdout:1550: read oid 7 snap -1 2024-09-13T14:27:19.044 INFO:tasks.rados.rados.0.smithi151.stdout:1550: expect (ObjNum 597 snap 191 seq_num 597) 2024-09-13T14:27:19.044 INFO:tasks.rados.rados.0.smithi151.stdout:1551: write oid 8 current snap is 193 2024-09-13T14:27:19.044 INFO:tasks.rados.rados.0.smithi151.stdout:1551: seq_num 602 ranges {517814=491785,1577807=540007} 2024-09-13T14:27:19.050 INFO:tasks.rados.rados.0.smithi151.stdout:1551: writing smithi15186934-8 from 517814 to 1009599 tid 1 2024-09-13T14:27:19.056 INFO:tasks.rados.rados.0.smithi151.stdout:1551: writing smithi15186934-8 from 1577807 to 2117814 tid 2 2024-09-13T14:27:19.056 INFO:tasks.rados.rados.0.smithi151.stdout:1552: delete oid 30 current snap is 193 2024-09-13T14:27:19.056 INFO:tasks.rados.rados.0.smithi151.stdout:1548: expect (ObjNum 575 snap 185 seq_num 575) 2024-09-13T14:27:19.176 INFO:tasks.rados.rados.0.smithi151.stdout:1549: finishing write tid 1 to smithi15186934-27 2024-09-13T14:27:19.177 INFO:tasks.rados.rados.0.smithi151.stdout:1549: finishing write tid 2 to smithi15186934-27 2024-09-13T14:27:19.177 INFO:tasks.rados.rados.0.smithi151.stdout:1549: finishing write tid 3 to smithi15186934-27 2024-09-13T14:27:19.177 INFO:tasks.rados.rados.0.smithi151.stdout:1549: finishing write tid 4 to smithi15186934-27 2024-09-13T14:27:19.177 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 27 v 416 (ObjNum 601 snap 193 seq_num 601) dirty exists 2024-09-13T14:27:19.177 INFO:tasks.rados.rados.0.smithi151.stdout:1549: left oid 27 (ObjNum 601 snap 193 seq_num 601) 2024-09-13T14:27:19.177 INFO:tasks.rados.rados.0.smithi151.stdout:1550: expect (ObjNum 597 snap 191 seq_num 597) 2024-09-13T14:27:19.289 INFO:tasks.rados.rados.0.smithi151.stdout:1551: finishing write tid 1 to smithi15186934-8 2024-09-13T14:27:19.289 INFO:tasks.rados.rados.0.smithi151.stdout:1551: finishing write tid 2 to smithi15186934-8 2024-09-13T14:27:19.289 INFO:tasks.rados.rados.0.smithi151.stdout:1551: finishing write tid 3 to smithi15186934-8 2024-09-13T14:27:19.289 INFO:tasks.rados.rados.0.smithi151.stdout:1551: finishing write tid 4 to smithi15186934-8 2024-09-13T14:27:19.290 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 500 (ObjNum 602 snap 193 seq_num 602) dirty exists 2024-09-13T14:27:19.290 INFO:tasks.rados.rados.0.smithi151.stdout:1551: left oid 8 (ObjNum 602 snap 193 seq_num 602) 2024-09-13T14:27:19.290 INFO:tasks.rados.rados.0.smithi151.stdout:1548: done (4 left) 2024-09-13T14:27:19.290 INFO:tasks.rados.rados.0.smithi151.stdout:1549: done (3 left) 2024-09-13T14:27:19.290 INFO:tasks.rados.rados.0.smithi151.stdout:1550: done (2 left) 2024-09-13T14:27:19.290 INFO:tasks.rados.rados.0.smithi151.stdout:1551: done (1 left) 2024-09-13T14:27:19.290 INFO:tasks.rados.rados.0.smithi151.stdout:1552: done (0 left) 2024-09-13T14:27:19.291 INFO:tasks.rados.rados.0.smithi151.stdout:1553: read oid 34 snap -1 2024-09-13T14:27:19.291 INFO:tasks.rados.rados.0.smithi151.stdout:1553: expect (ObjNum 539 snap 173 seq_num 539) 2024-09-13T14:27:19.291 INFO:tasks.rados.rados.0.smithi151.stdout:1554: read oid 20 snap -1 2024-09-13T14:27:19.291 INFO:tasks.rados.rados.0.smithi151.stdout:1554: expect deleted 2024-09-13T14:27:19.291 INFO:tasks.rados.rados.0.smithi151.stdout:1555: snap_remove snap 171 2024-09-13T14:27:19.487 INFO:tasks.rados.rados.0.smithi151.stdout:1555: done (2 left) 2024-09-13T14:27:19.488 INFO:tasks.rados.rados.0.smithi151.stdout:1556: delete oid 11 current snap is 193 2024-09-13T14:27:19.488 INFO:tasks.rados.rados.0.smithi151.stdout:1553: expect (ObjNum 539 snap 173 seq_num 539) 2024-09-13T14:27:19.630 INFO:tasks.rados.rados.0.smithi151.stdout:1553: done (2 left) 2024-09-13T14:27:19.630 INFO:tasks.rados.rados.0.smithi151.stdout:1554: done (1 left) 2024-09-13T14:27:19.630 INFO:tasks.rados.rados.0.smithi151.stdout:1556: done (0 left) 2024-09-13T14:27:19.631 INFO:tasks.rados.rados.0.smithi151.stdout:1557: write oid 29 current snap is 193 2024-09-13T14:27:19.631 INFO:tasks.rados.rados.0.smithi151.stdout:1557: seq_num 605 ranges {568235=717532,1753253=414982} 2024-09-13T14:27:19.636 INFO:tasks.rados.rados.0.smithi151.stdout:1557: writing smithi15186934-29 from 568235 to 1285767 tid 1 2024-09-13T14:27:19.641 INFO:tasks.rados.rados.0.smithi151.stdout:1557: writing smithi15186934-29 from 1753253 to 2168235 tid 2 2024-09-13T14:27:19.642 INFO:tasks.rados.rados.0.smithi151.stdout:1558: snap_remove snap 161 2024-09-13T14:27:19.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:19 smithi184 ceph-mon[92527]: osdmap e448: 8 total, 8 up, 8 in 2024-09-13T14:27:19.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:19 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:19.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:19 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:19.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:19 smithi184 ceph-mon[92527]: Reconfiguring mon.b (monmap changed)... 2024-09-13T14:27:19.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:19 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:27:19.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:19 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:27:19.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:19 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:19.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:19 smithi184 ceph-mon[92527]: Reconfiguring daemon mon.b on smithi184 2024-09-13T14:27:19.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:19 smithi184 ceph-mon[92527]: osdmap e449: 8 total, 8 up, 8 in 2024-09-13T14:27:19.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:19 smithi151 ceph-mon[104450]: osdmap e448: 8 total, 8 up, 8 in 2024-09-13T14:27:19.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:19 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:19.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:19 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:19.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:19 smithi151 ceph-mon[104450]: Reconfiguring mon.b (monmap changed)... 2024-09-13T14:27:19.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:19 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:27:19.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:19 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:27:19.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:19 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:19.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:19 smithi151 ceph-mon[104450]: Reconfiguring daemon mon.b on smithi184 2024-09-13T14:27:19.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:19 smithi151 ceph-mon[104450]: osdmap e449: 8 total, 8 up, 8 in 2024-09-13T14:27:19.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:19 smithi151 ceph-mon[108966]: osdmap e448: 8 total, 8 up, 8 in 2024-09-13T14:27:19.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:19 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:19.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:19 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:19.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:19 smithi151 ceph-mon[108966]: Reconfiguring mon.b (monmap changed)... 2024-09-13T14:27:19.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:19 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:27:19.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:19 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:27:19.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:19 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:19.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:19 smithi151 ceph-mon[108966]: Reconfiguring daemon mon.b on smithi184 2024-09-13T14:27:19.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:19 smithi151 ceph-mon[108966]: osdmap e449: 8 total, 8 up, 8 in 2024-09-13T14:27:20.354 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:27:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:27:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:20.538 INFO:tasks.rados.rados.0.smithi151.stdout:1557: finishing write tid 1 to smithi15186934-29 2024-09-13T14:27:20.538 INFO:tasks.rados.rados.0.smithi151.stdout:1557: finishing write tid 2 to smithi15186934-29 2024-09-13T14:27:20.539 INFO:tasks.rados.rados.0.smithi151.stdout:1557: finishing write tid 3 to smithi15186934-29 2024-09-13T14:27:20.539 INFO:tasks.rados.rados.0.smithi151.stdout:1557: finishing write tid 4 to smithi15186934-29 2024-09-13T14:27:20.539 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 29 v 427 (ObjNum 605 snap 193 seq_num 605) dirty exists 2024-09-13T14:27:20.539 INFO:tasks.rados.rados.0.smithi151.stdout:1557: left oid 29 (ObjNum 605 snap 193 seq_num 605) 2024-09-13T14:27:20.539 INFO:tasks.rados.rados.0.smithi151.stdout:1557: done (1 left) 2024-09-13T14:27:20.539 INFO:tasks.rados.rados.0.smithi151.stdout:1558: done (0 left) 2024-09-13T14:27:20.539 INFO:tasks.rados.rados.0.smithi151.stdout:1559: read oid 6 snap -1 2024-09-13T14:27:20.539 INFO:tasks.rados.rados.0.smithi151.stdout:1559: expect (ObjNum 582 snap 186 seq_num 582) 2024-09-13T14:27:20.539 INFO:tasks.rados.rados.0.smithi151.stdout:1560: rollback oid 32 current snap is 193 2024-09-13T14:27:20.539 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 32 to 182 2024-09-13T14:27:20.539 INFO:tasks.rados.rados.0.smithi151.stdout:1561: read oid 28 snap -1 2024-09-13T14:27:20.539 INFO:tasks.rados.rados.0.smithi151.stdout:1561: expect deleted 2024-09-13T14:27:20.539 INFO:tasks.rados.rados.0.smithi151.stdout:1562: read oid 12 snap -1 2024-09-13T14:27:20.540 INFO:tasks.rados.rados.0.smithi151.stdout:1562: expect deleted 2024-09-13T14:27:20.540 INFO:tasks.rados.rados.0.smithi151.stdout:1563: snap_create 2024-09-13T14:27:20.548 INFO:tasks.rados.rados.0.smithi151.stdout:1560: finishing rollback tid 0 to smithi15186934-32 2024-09-13T14:27:20.549 INFO:tasks.rados.rados.0.smithi151.stdout:1560: finishing rollback tid 1 to smithi15186934-32 2024-09-13T14:27:20.549 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 32 v 655 (ObjNum 961 snap 0 seq_num 1755291792) dirty dne 2024-09-13T14:27:20.551 INFO:tasks.rados.rados.0.smithi151.stdout:1559: expect (ObjNum 582 snap 186 seq_num 582) 2024-09-13T14:27:20.761 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:20 smithi184 ceph-mon[92527]: pgmap v291: 89 pgs: 89 active+clean; 292 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.8 MiB/s rd, 512 KiB/s wr, 5 op/s 2024-09-13T14:27:20.762 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:20 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:20.762 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:20 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:20.762 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:20 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-13T14:27:20.762 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:20 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-13T14:27:20.762 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:20 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:20.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:20 smithi151 ceph-mon[108966]: pgmap v291: 89 pgs: 89 active+clean; 292 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.8 MiB/s rd, 512 KiB/s wr, 5 op/s 2024-09-13T14:27:20.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:20 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:20.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:20 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:20.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:20 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-13T14:27:20.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:20 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-13T14:27:20.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:20 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:20.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:20 smithi151 ceph-mon[104450]: pgmap v291: 89 pgs: 89 active+clean; 292 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.8 MiB/s rd, 512 KiB/s wr, 5 op/s 2024-09-13T14:27:20.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:20 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:20.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:20 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:20.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:20 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-13T14:27:20.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:20 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-13T14:27:20.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:20 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:21.547 INFO:tasks.rados.rados.0.smithi151.stdout:1559: done (4 left) 2024-09-13T14:27:21.547 INFO:tasks.rados.rados.0.smithi151.stdout:1560: done (3 left) 2024-09-13T14:27:21.547 INFO:tasks.rados.rados.0.smithi151.stdout:1561: done (2 left) 2024-09-13T14:27:21.547 INFO:tasks.rados.rados.0.smithi151.stdout:1562: done (1 left) 2024-09-13T14:27:21.547 INFO:tasks.rados.rados.0.smithi151.stdout:1563: done (0 left) 2024-09-13T14:27:21.547 INFO:tasks.rados.rados.0.smithi151.stdout:1564: snap_create 2024-09-13T14:27:21.802 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:21 smithi184 ceph-mon[92527]: Reconfiguring mgr.x (monmap changed)... 2024-09-13T14:27:21.803 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:21 smithi184 ceph-mon[92527]: Reconfiguring daemon mgr.x on smithi184 2024-09-13T14:27:21.803 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:21 smithi184 ceph-mon[92527]: osdmap e450: 8 total, 8 up, 8 in 2024-09-13T14:27:21.803 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:21 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:21.803 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:21 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:21.803 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:21 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-13T14:27:21.803 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:21 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:21.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:21 smithi151 ceph-mon[108966]: Reconfiguring mgr.x (monmap changed)... 2024-09-13T14:27:21.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:21 smithi151 ceph-mon[108966]: Reconfiguring daemon mgr.x on smithi184 2024-09-13T14:27:21.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:21 smithi151 ceph-mon[108966]: osdmap e450: 8 total, 8 up, 8 in 2024-09-13T14:27:21.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:21 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:21.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:21 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:21.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:21 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-13T14:27:21.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:21 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:21.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:21 smithi151 ceph-mon[104450]: Reconfiguring mgr.x (monmap changed)... 2024-09-13T14:27:21.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:21 smithi151 ceph-mon[104450]: Reconfiguring daemon mgr.x on smithi184 2024-09-13T14:27:21.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:21 smithi151 ceph-mon[104450]: osdmap e450: 8 total, 8 up, 8 in 2024-09-13T14:27:21.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:21 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:21.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:21 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:21.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:21 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-13T14:27:21.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:21 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:22.544 INFO:tasks.rados.rados.0.smithi151.stdout:1564: done (0 left) 2024-09-13T14:27:22.545 INFO:tasks.rados.rados.0.smithi151.stdout:1565: rollback oid 6 current snap is 195 2024-09-13T14:27:22.545 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 6 to 188 2024-09-13T14:27:22.545 INFO:tasks.rados.rados.0.smithi151.stdout:1566: write oid 32 current snap is 195 2024-09-13T14:27:22.545 INFO:tasks.rados.rados.0.smithi151.stdout:1566: seq_num 606 ranges {585042=792781,2095382=89660} 2024-09-13T14:27:22.552 INFO:tasks.rados.rados.0.smithi151.stdout:1566: writing smithi15186934-32 from 585042 to 1377823 tid 1 2024-09-13T14:27:22.557 INFO:tasks.rados.rados.0.smithi151.stdout:1566: writing smithi15186934-32 from 2095382 to 2185042 tid 2 2024-09-13T14:27:22.557 INFO:tasks.rados.rados.0.smithi151.stdout:1567: rollback oid 46 current snap is 195 2024-09-13T14:27:22.558 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 46 to 181 2024-09-13T14:27:22.558 INFO:tasks.rados.rados.0.smithi151.stdout:1565: finishing rollback tid 0 to smithi15186934-6 2024-09-13T14:27:22.558 INFO:tasks.rados.rados.0.smithi151.stdout:1565: finishing rollback tid 1 to smithi15186934-6 2024-09-13T14:27:22.561 INFO:tasks.rados.rados.0.smithi151.stdout:1568: read oid 14 snap 186 2024-09-13T14:27:22.561 INFO:tasks.rados.rados.0.smithi151.stdout:1568: expect (ObjNum 572 snap 182 seq_num 572) 2024-09-13T14:27:22.561 INFO:tasks.rados.rados.0.smithi151.stdout:1569: snap_create 2024-09-13T14:27:22.561 INFO:tasks.rados.rados.0.smithi151.stdout:1565: finishing rollback tid 2 to smithi15186934-6 2024-09-13T14:27:22.561 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 6 v 420 (ObjNum 582 snap 186 seq_num 582) dirty exists 2024-09-13T14:27:22.561 INFO:tasks.rados.rados.0.smithi151.stdout:1566: finishing write tid 1 to smithi15186934-32 2024-09-13T14:27:22.562 INFO:tasks.rados.rados.0.smithi151.stdout:1567: finishing rollback tid 1 to smithi15186934-46 2024-09-13T14:27:22.562 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 46 v 538 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-13T14:27:22.565 INFO:tasks.rados.rados.0.smithi151.stdout:1566: finishing write tid 2 to smithi15186934-32 2024-09-13T14:27:22.565 INFO:tasks.rados.rados.0.smithi151.stdout:1566: finishing write tid 3 to smithi15186934-32 2024-09-13T14:27:22.565 INFO:tasks.rados.rados.0.smithi151.stdout:1566: finishing write tid 4 to smithi15186934-32 2024-09-13T14:27:22.565 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 32 v 662 (ObjNum 606 snap 195 seq_num 606) dirty exists 2024-09-13T14:27:22.565 INFO:tasks.rados.rados.0.smithi151.stdout:1566: left oid 32 (ObjNum 606 snap 195 seq_num 606) 2024-09-13T14:27:22.567 INFO:tasks.rados.rados.0.smithi151.stdout:1568: expect (ObjNum 572 snap 182 seq_num 572) 2024-09-13T14:27:22.847 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:22 smithi184 ceph-mon[92527]: pgmap v294: 89 pgs: 89 active+clean; 292 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:27:22.847 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:22 smithi184 ceph-mon[92527]: Reconfiguring osd.4 (monmap changed)... 2024-09-13T14:27:22.847 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:22 smithi184 ceph-mon[92527]: Reconfiguring daemon osd.4 on smithi184 2024-09-13T14:27:22.847 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:22 smithi184 ceph-mon[92527]: osdmap e451: 8 total, 8 up, 8 in 2024-09-13T14:27:22.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:22 smithi151 ceph-mon[104450]: pgmap v294: 89 pgs: 89 active+clean; 292 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:27:22.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:22 smithi151 ceph-mon[104450]: Reconfiguring osd.4 (monmap changed)... 2024-09-13T14:27:22.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:22 smithi151 ceph-mon[104450]: Reconfiguring daemon osd.4 on smithi184 2024-09-13T14:27:22.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:22 smithi151 ceph-mon[104450]: osdmap e451: 8 total, 8 up, 8 in 2024-09-13T14:27:22.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:22 smithi151 ceph-mon[108966]: pgmap v294: 89 pgs: 89 active+clean; 292 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:27:22.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:22 smithi151 ceph-mon[108966]: Reconfiguring osd.4 (monmap changed)... 2024-09-13T14:27:22.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:22 smithi151 ceph-mon[108966]: Reconfiguring daemon osd.4 on smithi184 2024-09-13T14:27:22.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:22 smithi151 ceph-mon[108966]: osdmap e451: 8 total, 8 up, 8 in 2024-09-13T14:27:23.559 INFO:tasks.rados.rados.0.smithi151.stdout:1565: done (4 left) 2024-09-13T14:27:23.559 INFO:tasks.rados.rados.0.smithi151.stdout:1566: done (3 left) 2024-09-13T14:27:23.559 INFO:tasks.rados.rados.0.smithi151.stdout:1567: done (2 left) 2024-09-13T14:27:23.559 INFO:tasks.rados.rados.0.smithi151.stdout:1568: done (1 left) 2024-09-13T14:27:23.559 INFO:tasks.rados.rados.0.smithi151.stdout:1569: done (0 left) 2024-09-13T14:27:23.560 INFO:tasks.rados.rados.0.smithi151.stdout:1570: read oid 17 snap 182 2024-09-13T14:27:23.560 INFO:tasks.rados.rados.0.smithi151.stdout:1570: expect (ObjNum 537 snap 173 seq_num 537) 2024-09-13T14:27:23.560 INFO:tasks.rados.rados.0.smithi151.stdout:1571: read oid 36 snap -1 2024-09-13T14:27:23.560 INFO:tasks.rados.rados.0.smithi151.stdout:1571: expect (ObjNum 588 snap 190 seq_num 588) 2024-09-13T14:27:23.560 INFO:tasks.rados.rados.0.smithi151.stdout:1572: snap_remove snap 181 2024-09-13T14:27:23.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:23 smithi184 ceph-mon[92527]: osdmap e452: 8 total, 8 up, 8 in 2024-09-13T14:27:23.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:23 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:23.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:23 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:23.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:23 smithi184 ceph-mon[92527]: Reconfiguring osd.5 (monmap changed)... 2024-09-13T14:27:23.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:23 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-13T14:27:23.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:23 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:23.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:23 smithi184 ceph-mon[92527]: Reconfiguring daemon osd.5 on smithi184 2024-09-13T14:27:23.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:23 smithi151 ceph-mon[104450]: osdmap e452: 8 total, 8 up, 8 in 2024-09-13T14:27:23.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:23 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:23.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:23 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:23.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:23 smithi151 ceph-mon[104450]: Reconfiguring osd.5 (monmap changed)... 2024-09-13T14:27:23.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:23 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-13T14:27:23.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:23 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:23.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:23 smithi151 ceph-mon[104450]: Reconfiguring daemon osd.5 on smithi184 2024-09-13T14:27:23.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:23 smithi151 ceph-mon[108966]: osdmap e452: 8 total, 8 up, 8 in 2024-09-13T14:27:23.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:23 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:23.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:23 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:23.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:23 smithi151 ceph-mon[108966]: Reconfiguring osd.5 (monmap changed)... 2024-09-13T14:27:23.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:23 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-13T14:27:23.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:23 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:23.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:23 smithi151 ceph-mon[108966]: Reconfiguring daemon osd.5 on smithi184 2024-09-13T14:27:24.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:27:24 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:27:24.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:24.487 INFO:tasks.rados.rados.0.smithi151.stdout:1572: done (2 left) 2024-09-13T14:27:24.487 INFO:tasks.rados.rados.0.smithi151.stdout:1573: snap_remove snap 194 2024-09-13T14:27:24.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:24 smithi184 ceph-mon[92527]: pgmap v297: 89 pgs: 89 active+clean; 226 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 21 MiB/s rd, 744 KiB/s wr, 22 op/s 2024-09-13T14:27:24.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:24 smithi184 ceph-mon[92527]: osdmap e453: 8 total, 8 up, 8 in 2024-09-13T14:27:24.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:24 smithi184 ceph-mon[92527]: osdmap e454: 8 total, 8 up, 8 in 2024-09-13T14:27:24.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:24 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:24.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:24 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:24.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:24 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-13T14:27:24.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:24 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:24.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:27:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:27:24] "GET /metrics HTTP/1.1" 200 36400 "" "Prometheus/2.51.0" 2024-09-13T14:27:24.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:24 smithi151 ceph-mon[104450]: pgmap v297: 89 pgs: 89 active+clean; 226 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 21 MiB/s rd, 744 KiB/s wr, 22 op/s 2024-09-13T14:27:24.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:24 smithi151 ceph-mon[104450]: osdmap e453: 8 total, 8 up, 8 in 2024-09-13T14:27:24.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:24 smithi151 ceph-mon[104450]: osdmap e454: 8 total, 8 up, 8 in 2024-09-13T14:27:24.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:24 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:24.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:24 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:24.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:24 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-13T14:27:24.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:24 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:24.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:24 smithi151 ceph-mon[108966]: pgmap v297: 89 pgs: 89 active+clean; 226 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 21 MiB/s rd, 744 KiB/s wr, 22 op/s 2024-09-13T14:27:24.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:24 smithi151 ceph-mon[108966]: osdmap e453: 8 total, 8 up, 8 in 2024-09-13T14:27:24.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:24 smithi151 ceph-mon[108966]: osdmap e454: 8 total, 8 up, 8 in 2024-09-13T14:27:24.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:24 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:24.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:24 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:24.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:24 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-13T14:27:24.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:24 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:25.490 INFO:tasks.rados.rados.0.smithi151.stdout:1573: done (2 left) 2024-09-13T14:27:25.491 INFO:tasks.rados.rados.0.smithi151.stdout:1574: write oid 29 current snap is 196 2024-09-13T14:27:25.491 INFO:tasks.rados.rados.0.smithi151.stdout:1574: seq_num 607 ranges {601849=468030,1753864=447985} 2024-09-13T14:27:25.496 INFO:tasks.rados.rados.0.smithi151.stdout:1574: writing smithi15186934-29 from 601849 to 1069879 tid 1 2024-09-13T14:27:25.503 INFO:tasks.rados.rados.0.smithi151.stdout:1574: writing smithi15186934-29 from 1753864 to 2201849 tid 2 2024-09-13T14:27:25.503 INFO:tasks.rados.rados.0.smithi151.stdout:1575: read oid 4 snap -1 2024-09-13T14:27:25.503 INFO:tasks.rados.rados.0.smithi151.stdout:1575: expect (ObjNum 592 snap 191 seq_num 592) 2024-09-13T14:27:25.504 INFO:tasks.rados.rados.0.smithi151.stdout:1576: read oid 44 snap -1 2024-09-13T14:27:25.504 INFO:tasks.rados.rados.0.smithi151.stdout:1576: expect (ObjNum 579 snap 186 seq_num 579) 2024-09-13T14:27:25.504 INFO:tasks.rados.rados.0.smithi151.stdout:1577: snap_remove snap 186 2024-09-13T14:27:25.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:25 smithi184 ceph-mon[92527]: Reconfiguring osd.6 (monmap changed)... 2024-09-13T14:27:25.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:25 smithi184 ceph-mon[92527]: Reconfiguring daemon osd.6 on smithi184 2024-09-13T14:27:25.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:25 smithi184 ceph-mon[92527]: osdmap e455: 8 total, 8 up, 8 in 2024-09-13T14:27:25.854 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:27:25 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:27:25.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:25.854 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:27:25 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:27:25.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:25.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:25 smithi151 ceph-mon[104450]: Reconfiguring osd.6 (monmap changed)... 2024-09-13T14:27:25.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:25 smithi151 ceph-mon[104450]: Reconfiguring daemon osd.6 on smithi184 2024-09-13T14:27:25.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:25 smithi151 ceph-mon[104450]: osdmap e455: 8 total, 8 up, 8 in 2024-09-13T14:27:25.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:25 smithi151 ceph-mon[108966]: Reconfiguring osd.6 (monmap changed)... 2024-09-13T14:27:25.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:25 smithi151 ceph-mon[108966]: Reconfiguring daemon osd.6 on smithi184 2024-09-13T14:27:25.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:25 smithi151 ceph-mon[108966]: osdmap e455: 8 total, 8 up, 8 in 2024-09-13T14:27:26.490 INFO:tasks.rados.rados.0.smithi151.stdout:1577: done (5 left) 2024-09-13T14:27:26.490 INFO:tasks.rados.rados.0.smithi151.stdout:1578: snap_create 2024-09-13T14:27:26.491 INFO:tasks.rados.rados.0.smithi151.stdout:1571: expect (ObjNum 588 snap 190 seq_num 588) 2024-09-13T14:27:26.621 INFO:tasks.rados.rados.0.smithi151.stdout:1570: expect (ObjNum 537 snap 173 seq_num 537) 2024-09-13T14:27:26.846 INFO:tasks.rados.rados.0.smithi151.stdout:1574: finishing write tid 1 to smithi15186934-29 2024-09-13T14:27:26.846 INFO:tasks.rados.rados.0.smithi151.stdout:1574: finishing write tid 2 to smithi15186934-29 2024-09-13T14:27:26.847 INFO:tasks.rados.rados.0.smithi151.stdout:1574: finishing write tid 3 to smithi15186934-29 2024-09-13T14:27:26.847 INFO:tasks.rados.rados.0.smithi151.stdout:1574: finishing write tid 4 to smithi15186934-29 2024-09-13T14:27:26.847 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 29 v 439 (ObjNum 607 snap 196 seq_num 607) dirty exists 2024-09-13T14:27:26.847 INFO:tasks.rados.rados.0.smithi151.stdout:1574: left oid 29 (ObjNum 607 snap 196 seq_num 607) 2024-09-13T14:27:26.847 INFO:tasks.rados.rados.0.smithi151.stdout:1576: expect (ObjNum 579 snap 186 seq_num 579) 2024-09-13T14:27:26.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:26 smithi184 ceph-mon[92527]: pgmap v300: 89 pgs: 89 active+clean; 226 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 21 MiB/s rd, 744 KiB/s wr, 22 op/s 2024-09-13T14:27:26.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:26 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:26.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:26 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:26.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:26 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-13T14:27:26.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:26 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:26.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:26 smithi184 ceph-mon[92527]: osdmap e456: 8 total, 8 up, 8 in 2024-09-13T14:27:26.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:26 smithi151 ceph-mon[108966]: pgmap v300: 89 pgs: 89 active+clean; 226 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 21 MiB/s rd, 744 KiB/s wr, 22 op/s 2024-09-13T14:27:26.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:26 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:26.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:26 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:26.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:26 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-13T14:27:26.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:26 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:26.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:26 smithi151 ceph-mon[108966]: osdmap e456: 8 total, 8 up, 8 in 2024-09-13T14:27:26.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:26 smithi151 ceph-mon[104450]: pgmap v300: 89 pgs: 89 active+clean; 226 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 21 MiB/s rd, 744 KiB/s wr, 22 op/s 2024-09-13T14:27:26.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:26 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:26.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:26 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:26.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:26 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-13T14:27:26.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:26 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:26.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:26 smithi151 ceph-mon[104450]: osdmap e456: 8 total, 8 up, 8 in 2024-09-13T14:27:26.968 INFO:tasks.rados.rados.0.smithi151.stdout:1575: expect (ObjNum 592 snap 191 seq_num 592) 2024-09-13T14:27:27.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:27:26 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:27:26.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi184\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:27.492 INFO:tasks.rados.rados.0.smithi151.stdout:1570: done (5 left) 2024-09-13T14:27:27.493 INFO:tasks.rados.rados.0.smithi151.stdout:1571: done (4 left) 2024-09-13T14:27:27.493 INFO:tasks.rados.rados.0.smithi151.stdout:1574: done (3 left) 2024-09-13T14:27:27.493 INFO:tasks.rados.rados.0.smithi151.stdout:1575: done (2 left) 2024-09-13T14:27:27.493 INFO:tasks.rados.rados.0.smithi151.stdout:1576: done (1 left) 2024-09-13T14:27:27.493 INFO:tasks.rados.rados.0.smithi151.stdout:1578: done (0 left) 2024-09-13T14:27:27.494 INFO:tasks.rados.rados.0.smithi151.stdout:1579: snap_remove snap 188 2024-09-13T14:27:27.686 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:27 smithi184 ceph-mon[92527]: Reconfiguring osd.7 (monmap changed)... 2024-09-13T14:27:27.686 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:27 smithi184 ceph-mon[92527]: Reconfiguring daemon osd.7 on smithi184 2024-09-13T14:27:27.687 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:27 smithi184 ceph-mon[92527]: osdmap e457: 8 total, 8 up, 8 in 2024-09-13T14:27:27.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:27 smithi151 ceph-mon[108966]: Reconfiguring osd.7 (monmap changed)... 2024-09-13T14:27:27.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:27 smithi151 ceph-mon[108966]: Reconfiguring daemon osd.7 on smithi184 2024-09-13T14:27:27.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:27 smithi151 ceph-mon[108966]: osdmap e457: 8 total, 8 up, 8 in 2024-09-13T14:27:27.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:27 smithi151 ceph-mon[104450]: Reconfiguring osd.7 (monmap changed)... 2024-09-13T14:27:27.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:27 smithi151 ceph-mon[104450]: Reconfiguring daemon osd.7 on smithi184 2024-09-13T14:27:27.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:27 smithi151 ceph-mon[104450]: osdmap e457: 8 total, 8 up, 8 in 2024-09-13T14:27:28.497 INFO:tasks.rados.rados.0.smithi151.stdout:1579: done (0 left) 2024-09-13T14:27:28.497 INFO:tasks.rados.rados.0.smithi151.stdout:1580: read oid 14 snap -1 2024-09-13T14:27:28.498 INFO:tasks.rados.rados.0.smithi151.stdout:1580: expect (ObjNum 572 snap 182 seq_num 572) 2024-09-13T14:27:28.498 INFO:tasks.rados.rados.0.smithi151.stdout:1581: read oid 5 snap -1 2024-09-13T14:27:28.498 INFO:tasks.rados.rados.0.smithi151.stdout:1581: expect (ObjNum 594 snap 191 seq_num 594) 2024-09-13T14:27:28.498 INFO:tasks.rados.rados.0.smithi151.stdout:1582: rollback oid 23 current snap is 197 2024-09-13T14:27:28.498 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 23 to 195 2024-09-13T14:27:28.498 INFO:tasks.rados.rados.0.smithi151.stdout:1583: read oid 1 snap 193 2024-09-13T14:27:28.498 INFO:tasks.rados.rados.0.smithi151.stdout:1583: expect (ObjNum 511 snap 164 seq_num 511) 2024-09-13T14:27:28.498 INFO:tasks.rados.rados.0.smithi151.stdout:1584: write oid 22 current snap is 197 2024-09-13T14:27:28.498 INFO:tasks.rados.rados.0.smithi151.stdout:1584: seq_num 608 ranges {618656=543279,1812346=406310} 2024-09-13T14:27:28.505 INFO:tasks.rados.rados.0.smithi151.stdout:1584: writing smithi15186934-22 from 618656 to 1161935 tid 1 2024-09-13T14:27:28.512 INFO:tasks.rados.rados.0.smithi151.stdout:1584: writing smithi15186934-22 from 1812346 to 2218656 tid 2 2024-09-13T14:27:28.512 INFO:tasks.rados.rados.0.smithi151.stdout:1585: write oid 11 current snap is 197 2024-09-13T14:27:28.512 INFO:tasks.rados.rados.0.smithi151.stdout:1582: finishing rollback tid 0 to smithi15186934-23 2024-09-13T14:27:28.512 INFO:tasks.rados.rados.0.smithi151.stdout:1582: finishing rollback tid 1 to smithi15186934-23 2024-09-13T14:27:28.512 INFO:tasks.rados.rados.0.smithi151.stdout:1582: finishing rollback tid 2 to smithi15186934-23 2024-09-13T14:27:28.512 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 23 v 697 (ObjNum 546 snap 175 seq_num 546) dirty exists 2024-09-13T14:27:28.512 INFO:tasks.rados.rados.0.smithi151.stdout:1584: finishing write tid 1 to smithi15186934-22 2024-09-13T14:27:28.512 INFO:tasks.rados.rados.0.smithi151.stdout:1583: expect (ObjNum 511 snap 164 seq_num 511) 2024-09-13T14:27:28.694 INFO:tasks.rados.rados.0.smithi151.stdout:1580: expect (ObjNum 572 snap 182 seq_num 572) 2024-09-13T14:27:28.850 INFO:tasks.rados.rados.0.smithi151.stdout:1581: expect (ObjNum 594 snap 191 seq_num 594) 2024-09-13T14:27:29.039 INFO:tasks.rados.rados.0.smithi151.stdout:1584: finishing write tid 2 to smithi15186934-22 2024-09-13T14:27:29.039 INFO:tasks.rados.rados.0.smithi151.stdout:1584: finishing write tid 3 to smithi15186934-22 2024-09-13T14:27:29.039 INFO:tasks.rados.rados.0.smithi151.stdout:1584: finishing write tid 4 to smithi15186934-22 2024-09-13T14:27:29.039 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 22 v 365 (ObjNum 608 snap 197 seq_num 608) dirty exists 2024-09-13T14:27:29.039 INFO:tasks.rados.rados.0.smithi151.stdout:1584: left oid 22 (ObjNum 608 snap 197 seq_num 608) 2024-09-13T14:27:29.039 INFO:tasks.rados.rados.0.smithi151.stdout:1585: seq_num 609 ranges {635463=734881,1987181=248282} 2024-09-13T14:27:29.046 INFO:tasks.rados.rados.0.smithi151.stdout:1585: writing smithi15186934-11 from 635463 to 1370344 tid 1 2024-09-13T14:27:29.051 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:28 smithi184 ceph-mon[92527]: pgmap v303: 89 pgs: 89 active+clean; 224 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 440 KiB/s wr, 13 op/s 2024-09-13T14:27:29.051 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:28 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:29.051 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:28 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:29.051 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:28 smithi184 ceph-mon[92527]: Reconfiguring mds.foo.smithi184.jismsi (monmap changed)... 2024-09-13T14:27:29.051 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:28 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi184.jismsi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-13T14:27:29.051 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:28 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:29.051 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:28 smithi184 ceph-mon[92527]: Reconfiguring daemon mds.foo.smithi184.jismsi on smithi184 2024-09-13T14:27:29.052 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:28 smithi184 ceph-mon[92527]: osdmap e458: 8 total, 8 up, 8 in 2024-09-13T14:27:29.052 INFO:tasks.rados.rados.0.smithi151.stdout:1585: writing smithi15186934-11 from 1987181 to 2235463 tid 2 2024-09-13T14:27:29.052 INFO:tasks.rados.rados.0.smithi151.stdout:1580: done (5 left) 2024-09-13T14:27:29.052 INFO:tasks.rados.rados.0.smithi151.stdout:1581: done (4 left) 2024-09-13T14:27:29.052 INFO:tasks.rados.rados.0.smithi151.stdout:1582: done (3 left) 2024-09-13T14:27:29.052 INFO:tasks.rados.rados.0.smithi151.stdout:1583: done (2 left) 2024-09-13T14:27:29.052 INFO:tasks.rados.rados.0.smithi151.stdout:1584: done (1 left) 2024-09-13T14:27:29.052 INFO:tasks.rados.rados.0.smithi151.stdout:1586: snap_remove snap 195 2024-09-13T14:27:29.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:28 smithi151 ceph-mon[108966]: pgmap v303: 89 pgs: 89 active+clean; 224 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 440 KiB/s wr, 13 op/s 2024-09-13T14:27:29.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:28 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:29.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:28 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:29.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:28 smithi151 ceph-mon[108966]: Reconfiguring mds.foo.smithi184.jismsi (monmap changed)... 2024-09-13T14:27:29.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:28 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi184.jismsi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-13T14:27:29.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:28 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:29.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:28 smithi151 ceph-mon[108966]: Reconfiguring daemon mds.foo.smithi184.jismsi on smithi184 2024-09-13T14:27:29.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:28 smithi151 ceph-mon[108966]: osdmap e458: 8 total, 8 up, 8 in 2024-09-13T14:27:29.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:28 smithi151 ceph-mon[104450]: pgmap v303: 89 pgs: 89 active+clean; 224 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 440 KiB/s wr, 13 op/s 2024-09-13T14:27:29.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:28 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:29.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:28 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:29.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:28 smithi151 ceph-mon[104450]: Reconfiguring mds.foo.smithi184.jismsi (monmap changed)... 2024-09-13T14:27:29.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:28 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi184.jismsi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-13T14:27:29.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:28 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:29.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:28 smithi151 ceph-mon[104450]: Reconfiguring daemon mds.foo.smithi184.jismsi on smithi184 2024-09-13T14:27:29.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:28 smithi151 ceph-mon[104450]: osdmap e458: 8 total, 8 up, 8 in 2024-09-13T14:27:29.489 INFO:tasks.rados.rados.0.smithi151.stdout:1586: done (1 left) 2024-09-13T14:27:29.489 INFO:tasks.rados.rados.0.smithi151.stdout:1587: rollback oid 43 current snap is 197 2024-09-13T14:27:29.489 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 43 to 190 2024-09-13T14:27:29.489 INFO:tasks.rados.rados.0.smithi151.stdout:1585: finishing write tid 1 to smithi15186934-11 2024-09-13T14:27:29.489 INFO:tasks.rados.rados.0.smithi151.stdout:1585: finishing write tid 2 to smithi15186934-11 2024-09-13T14:27:29.489 INFO:tasks.rados.rados.0.smithi151.stdout:1585: finishing write tid 3 to smithi15186934-11 2024-09-13T14:27:29.489 INFO:tasks.rados.rados.0.smithi151.stdout:1585: finishing write tid 4 to smithi15186934-11 2024-09-13T14:27:29.490 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 11 v 450 (ObjNum 609 snap 197 seq_num 609) dirty exists 2024-09-13T14:27:29.490 INFO:tasks.rados.rados.0.smithi151.stdout:1585: left oid 11 (ObjNum 609 snap 197 seq_num 609) 2024-09-13T14:27:29.490 INFO:tasks.rados.rados.0.smithi151.stdout:1585: done (1 left) 2024-09-13T14:27:29.490 INFO:tasks.rados.rados.0.smithi151.stdout:1588: rollback oid 28 current snap is 197 2024-09-13T14:27:29.490 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 28 to 190 2024-09-13T14:27:29.490 INFO:tasks.rados.rados.0.smithi151.stdout:1589: read oid 44 snap 157 2024-09-13T14:27:29.490 INFO:tasks.rados.rados.0.smithi151.stdout:1589: expect (ObjNum 474 snap 156 seq_num 474) 2024-09-13T14:27:29.490 INFO:tasks.rados.rados.0.smithi151.stdout:1590: delete oid 5 current snap is 197 2024-09-13T14:27:29.494 INFO:tasks.rados.rados.0.smithi151.stdout:1588: finishing rollback tid 1 to smithi15186934-28 2024-09-13T14:27:29.494 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 28 v 666 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-13T14:27:29.495 INFO:tasks.rados.rados.0.smithi151.stdout:1587: finishing rollback tid 0 to smithi15186934-43 2024-09-13T14:27:29.497 INFO:tasks.rados.rados.0.smithi151.stdout:1587: finishing rollback tid 1 to smithi15186934-43 2024-09-13T14:27:29.497 INFO:tasks.rados.rados.0.smithi151.stdout:1587: finishing rollback tid 2 to smithi15186934-43 2024-09-13T14:27:29.497 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 43 v 1100 (ObjNum 497 snap 159 seq_num 497) dirty exists 2024-09-13T14:27:29.497 INFO:tasks.rados.rados.0.smithi151.stdout:1587: done (3 left) 2024-09-13T14:27:29.497 INFO:tasks.rados.rados.0.smithi151.stdout:1588: done (2 left) 2024-09-13T14:27:29.497 INFO:tasks.rados.rados.0.smithi151.stdout:1590: done (1 left) 2024-09-13T14:27:29.497 INFO:tasks.rados.rados.0.smithi151.stdout:1591: write oid 14 current snap is 197 2024-09-13T14:27:29.497 INFO:tasks.rados.rados.0.smithi151.stdout:1591: seq_num 611 ranges {669077=485379,1587792=681285} 2024-09-13T14:27:29.503 INFO:tasks.rados.rados.0.smithi151.stdout:1591: writing smithi15186934-14 from 669077 to 1154456 tid 1 2024-09-13T14:27:29.509 INFO:tasks.rados.rados.0.smithi151.stdout:1591: writing smithi15186934-14 from 1587792 to 2269077 tid 2 2024-09-13T14:27:29.509 INFO:tasks.rados.rados.0.smithi151.stdout:1592: snap_remove snap 182 2024-09-13T14:27:30.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:27:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:27:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:30.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:30 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:30.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:30 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:30.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:30 smithi184 ceph-mon[92527]: pgmap v306: 89 pgs: 89 active+clean; 224 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 440 KiB/s wr, 13 op/s 2024-09-13T14:27:30.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:30 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:27:30.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:30 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:30.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:30 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:27:30.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:30 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:30.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:30 smithi184 ceph-mon[92527]: osdmap e459: 8 total, 8 up, 8 in 2024-09-13T14:27:30.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:30 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:30.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:30 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:30.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:30 smithi151 ceph-mon[108966]: pgmap v306: 89 pgs: 89 active+clean; 224 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 440 KiB/s wr, 13 op/s 2024-09-13T14:27:30.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:30 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:27:30.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:30 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:30.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:30 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:27:30.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:30 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:30.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:30 smithi151 ceph-mon[108966]: osdmap e459: 8 total, 8 up, 8 in 2024-09-13T14:27:30.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:30 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:30.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:30 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:30.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:30 smithi151 ceph-mon[104450]: pgmap v306: 89 pgs: 89 active+clean; 224 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 440 KiB/s wr, 13 op/s 2024-09-13T14:27:30.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:30 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:27:30.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:30 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:30.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:30 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:27:30.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:30 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:27:30.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:30 smithi151 ceph-mon[104450]: osdmap e459: 8 total, 8 up, 8 in 2024-09-13T14:27:30.493 INFO:tasks.rados.rados.0.smithi151.stdout:1592: done (2 left) 2024-09-13T14:27:30.493 INFO:tasks.rados.rados.0.smithi151.stdout:1593: snap_remove snap 193 2024-09-13T14:27:31.499 INFO:tasks.rados.rados.0.smithi151.stdout:1593: done (2 left) 2024-09-13T14:27:31.499 INFO:tasks.rados.rados.0.smithi151.stdout:1594: delete oid 7 current snap is 197 2024-09-13T14:27:31.499 INFO:tasks.rados.rados.0.smithi151.stdout:1589: expect (ObjNum 474 snap 156 seq_num 474) 2024-09-13T14:27:31.718 INFO:tasks.rados.rados.0.smithi151.stdout:1591: finishing write tid 1 to smithi15186934-14 2024-09-13T14:27:31.718 INFO:tasks.rados.rados.0.smithi151.stdout:1591: finishing write tid 2 to smithi15186934-14 2024-09-13T14:27:31.718 INFO:tasks.rados.rados.0.smithi151.stdout:1591: finishing write tid 3 to smithi15186934-14 2024-09-13T14:27:31.718 INFO:tasks.rados.rados.0.smithi151.stdout:1591: finishing write tid 4 to smithi15186934-14 2024-09-13T14:27:31.718 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 678 (ObjNum 611 snap 197 seq_num 611) dirty exists 2024-09-13T14:27:31.718 INFO:tasks.rados.rados.0.smithi151.stdout:1591: left oid 14 (ObjNum 611 snap 197 seq_num 611) 2024-09-13T14:27:31.718 INFO:tasks.rados.rados.0.smithi151.stdout:1589: done (2 left) 2024-09-13T14:27:31.718 INFO:tasks.rados.rados.0.smithi151.stdout:1591: done (1 left) 2024-09-13T14:27:31.718 INFO:tasks.rados.rados.0.smithi151.stdout:1594: done (0 left) 2024-09-13T14:27:31.719 INFO:tasks.rados.rados.0.smithi151.stdout:1595: snap_remove snap 157 2024-09-13T14:27:31.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:31 smithi184 ceph-mon[92527]: osdmap e460: 8 total, 8 up, 8 in 2024-09-13T14:27:31.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:31.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:31 smithi151 ceph-mon[108966]: osdmap e460: 8 total, 8 up, 8 in 2024-09-13T14:27:31.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:31.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:31 smithi151 ceph-mon[104450]: osdmap e460: 8 total, 8 up, 8 in 2024-09-13T14:27:31.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:32.502 INFO:tasks.rados.rados.0.smithi151.stdout:1595: done (0 left) 2024-09-13T14:27:32.502 INFO:tasks.rados.rados.0.smithi151.stdout:1596: snap_create 2024-09-13T14:27:32.760 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:32 smithi151 ceph-mon[108966]: pgmap v309: 89 pgs: 89 active+clean; 224 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:27:32.760 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:32 smithi151 ceph-mon[108966]: osdmap e461: 8 total, 8 up, 8 in 2024-09-13T14:27:32.760 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:32 smithi151 ceph-mon[104450]: pgmap v309: 89 pgs: 89 active+clean; 224 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:27:32.760 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:32 smithi151 ceph-mon[104450]: osdmap e461: 8 total, 8 up, 8 in 2024-09-13T14:27:32.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:32 smithi184 ceph-mon[92527]: pgmap v309: 89 pgs: 89 active+clean; 224 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:27:32.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:32 smithi184 ceph-mon[92527]: osdmap e461: 8 total, 8 up, 8 in 2024-09-13T14:27:33.513 INFO:tasks.rados.rados.0.smithi151.stdout:1596: done (0 left) 2024-09-13T14:27:33.513 INFO:tasks.rados.rados.0.smithi151.stdout:1597: rollback oid 25 current snap is 198 2024-09-13T14:27:33.513 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 25 to 196 2024-09-13T14:27:33.513 INFO:tasks.rados.rados.0.smithi151.stdout:1598: rollback oid 38 current snap is 198 2024-09-13T14:27:33.513 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 38 to 190 2024-09-13T14:27:33.513 INFO:tasks.rados.rados.0.smithi151.stdout:1599: rollback oid 43 current snap is 198 2024-09-13T14:27:33.513 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 43 to 197 2024-09-13T14:27:33.513 INFO:tasks.rados.rados.0.smithi151.stdout:1600: rollback oid 6 current snap is 198 2024-09-13T14:27:33.514 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 6 to 190 2024-09-13T14:27:33.514 INFO:tasks.rados.rados.0.smithi151.stdout:1601: read oid 2 snap -1 2024-09-13T14:27:33.514 INFO:tasks.rados.rados.0.smithi151.stdout:1601: expect (ObjNum 481 snap 156 seq_num 481) 2024-09-13T14:27:33.514 INFO:tasks.rados.rados.0.smithi151.stdout:1602: delete oid 9 current snap is 198 2024-09-13T14:27:33.518 INFO:tasks.rados.rados.0.smithi151.stdout:1597: finishing rollback tid 1 to smithi15186934-25 2024-09-13T14:27:33.518 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 900 (ObjNum 2133249 snap 0 seq_num 3680616213) dirty dne 2024-09-13T14:27:33.519 INFO:tasks.rados.rados.0.smithi151.stdout:1597: done (5 left) 2024-09-13T14:27:33.519 INFO:tasks.rados.rados.0.smithi151.stdout:1602: done (4 left) 2024-09-13T14:27:33.519 INFO:tasks.rados.rados.0.smithi151.stdout:1603: write oid 5 current snap is 198 2024-09-13T14:27:33.519 INFO:tasks.rados.rados.0.smithi151.stdout:1603: seq_num 614 ranges {719498=711126,2046885=272613} 2024-09-13T14:27:33.527 INFO:tasks.rados.rados.0.smithi151.stdout:1603: writing smithi15186934-5 from 719498 to 1430624 tid 1 2024-09-13T14:27:33.532 INFO:tasks.rados.rados.0.smithi151.stdout:1603: writing smithi15186934-5 from 2046885 to 2319498 tid 2 2024-09-13T14:27:33.532 INFO:tasks.rados.rados.0.smithi151.stdout:1604: rollback oid 13 current snap is 198 2024-09-13T14:27:33.533 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 13 to 196 2024-09-13T14:27:33.533 INFO:tasks.rados.rados.0.smithi151.stdout:1599: finishing rollback tid 0 to smithi15186934-43 2024-09-13T14:27:33.533 INFO:tasks.rados.rados.0.smithi151.stdout:1605: snap_create 2024-09-13T14:27:33.533 INFO:tasks.rados.rados.0.smithi151.stdout:1598: finishing rollback tid 0 to smithi15186934-38 2024-09-13T14:27:33.533 INFO:tasks.rados.rados.0.smithi151.stdout:1600: finishing rollback tid 0 to smithi15186934-6 2024-09-13T14:27:33.533 INFO:tasks.rados.rados.0.smithi151.stdout:1599: finishing rollback tid 1 to smithi15186934-43 2024-09-13T14:27:33.533 INFO:tasks.rados.rados.0.smithi151.stdout:1600: finishing rollback tid 1 to smithi15186934-6 2024-09-13T14:27:33.533 INFO:tasks.rados.rados.0.smithi151.stdout:1598: finishing rollback tid 1 to smithi15186934-38 2024-09-13T14:27:33.533 INFO:tasks.rados.rados.0.smithi151.stdout:1600: finishing rollback tid 2 to smithi15186934-6 2024-09-13T14:27:33.533 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 6 v 438 (ObjNum 582 snap 186 seq_num 582) dirty exists 2024-09-13T14:27:33.533 INFO:tasks.rados.rados.0.smithi151.stdout:1598: finishing rollback tid 2 to smithi15186934-38 2024-09-13T14:27:33.533 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 536 (ObjNum 504 snap 162 seq_num 504) dirty exists 2024-09-13T14:27:33.533 INFO:tasks.rados.rados.0.smithi151.stdout:1599: finishing rollback tid 2 to smithi15186934-43 2024-09-13T14:27:33.533 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 43 v 1124 (ObjNum 497 snap 159 seq_num 497) dirty exists 2024-09-13T14:27:33.534 INFO:tasks.rados.rados.0.smithi151.stdout:1601: expect (ObjNum 481 snap 156 seq_num 481) 2024-09-13T14:27:33.598 INFO:tasks.rados.rados.0.smithi151.stdout:1603: finishing write tid 1 to smithi15186934-5 2024-09-13T14:27:33.598 INFO:tasks.rados.rados.0.smithi151.stdout:1603: finishing write tid 2 to smithi15186934-5 2024-09-13T14:27:33.598 INFO:tasks.rados.rados.0.smithi151.stdout:1603: finishing write tid 3 to smithi15186934-5 2024-09-13T14:27:33.598 INFO:tasks.rados.rados.0.smithi151.stdout:1603: finishing write tid 4 to smithi15186934-5 2024-09-13T14:27:33.599 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 612 (ObjNum 614 snap 198 seq_num 614) dirty exists 2024-09-13T14:27:33.599 INFO:tasks.rados.rados.0.smithi151.stdout:1603: left oid 5 (ObjNum 614 snap 198 seq_num 614) 2024-09-13T14:27:33.599 INFO:tasks.rados.rados.0.smithi151.stdout:1604: finishing rollback tid 0 to smithi15186934-13 2024-09-13T14:27:33.599 INFO:tasks.rados.rados.0.smithi151.stdout:1604: finishing rollback tid 1 to smithi15186934-13 2024-09-13T14:27:33.599 INFO:tasks.rados.rados.0.smithi151.stdout:1604: finishing rollback tid 2 to smithi15186934-13 2024-09-13T14:27:33.599 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 13 v 616 (ObjNum 556 snap 176 seq_num 556) dirty exists 2024-09-13T14:27:33.775 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:33 smithi151 ceph-mon[104450]: osdmap e462: 8 total, 8 up, 8 in 2024-09-13T14:27:33.775 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:33 smithi151 ceph-mon[108966]: osdmap e462: 8 total, 8 up, 8 in 2024-09-13T14:27:33.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:33 smithi184 ceph-mon[92527]: osdmap e462: 8 total, 8 up, 8 in 2024-09-13T14:27:34.493 INFO:tasks.rados.rados.0.smithi151.stdout:1598: done (6 left) 2024-09-13T14:27:34.493 INFO:tasks.rados.rados.0.smithi151.stdout:1599: done (5 left) 2024-09-13T14:27:34.493 INFO:tasks.rados.rados.0.smithi151.stdout:1600: done (4 left) 2024-09-13T14:27:34.494 INFO:tasks.rados.rados.0.smithi151.stdout:1601: done (3 left) 2024-09-13T14:27:34.494 INFO:tasks.rados.rados.0.smithi151.stdout:1603: done (2 left) 2024-09-13T14:27:34.494 INFO:tasks.rados.rados.0.smithi151.stdout:1604: done (1 left) 2024-09-13T14:27:34.494 INFO:tasks.rados.rados.0.smithi151.stdout:1605: done (0 left) 2024-09-13T14:27:34.494 INFO:tasks.rados.rados.0.smithi151.stdout:1606: read oid 25 snap -1 2024-09-13T14:27:34.494 INFO:tasks.rados.rados.0.smithi151.stdout:1606: expect deleted 2024-09-13T14:27:34.494 INFO:tasks.rados.rados.0.smithi151.stdout:1607: snap_create 2024-09-13T14:27:34.509 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:27:34 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:27:34.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:34.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:34 smithi184 ceph-mon[92527]: pgmap v312: 89 pgs: 89 active+clean; 213 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 757 KiB/s wr, 14 op/s 2024-09-13T14:27:34.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:34 smithi184 ceph-mon[92527]: osdmap e463: 8 total, 8 up, 8 in 2024-09-13T14:27:34.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:34 smithi184 ceph-mon[92527]: osdmap e464: 8 total, 8 up, 8 in 2024-09-13T14:27:34.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:34 smithi151 ceph-mon[104450]: pgmap v312: 89 pgs: 89 active+clean; 213 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 757 KiB/s wr, 14 op/s 2024-09-13T14:27:34.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:34 smithi151 ceph-mon[104450]: osdmap e463: 8 total, 8 up, 8 in 2024-09-13T14:27:34.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:34 smithi151 ceph-mon[104450]: osdmap e464: 8 total, 8 up, 8 in 2024-09-13T14:27:34.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:27:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:27:34] "GET /metrics HTTP/1.1" 200 36399 "" "Prometheus/2.51.0" 2024-09-13T14:27:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:34 smithi151 ceph-mon[108966]: pgmap v312: 89 pgs: 89 active+clean; 213 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 757 KiB/s wr, 14 op/s 2024-09-13T14:27:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:34 smithi151 ceph-mon[108966]: osdmap e463: 8 total, 8 up, 8 in 2024-09-13T14:27:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:34 smithi151 ceph-mon[108966]: osdmap e464: 8 total, 8 up, 8 in 2024-09-13T14:27:35.491 INFO:tasks.rados.rados.0.smithi151.stdout:1606: done (1 left) 2024-09-13T14:27:35.491 INFO:tasks.rados.rados.0.smithi151.stdout:1607: done (0 left) 2024-09-13T14:27:35.491 INFO:tasks.rados.rados.0.smithi151.stdout:1608: delete oid 41 current snap is 200 2024-09-13T14:27:35.497 INFO:tasks.rados.rados.0.smithi151.stdout:1608: done (0 left) 2024-09-13T14:27:35.497 INFO:tasks.rados.rados.0.smithi151.stdout:1609: write oid 41 current snap is 200 2024-09-13T14:27:35.497 INFO:tasks.rados.rados.0.smithi151.stdout:1609: seq_num 616 ranges {753112=577977,1880202=472910} 2024-09-13T14:27:35.504 INFO:tasks.rados.rados.0.smithi151.stdout:1609: writing smithi15186934-41 from 753112 to 1331089 tid 1 2024-09-13T14:27:35.510 INFO:tasks.rados.rados.0.smithi151.stdout:1609: writing smithi15186934-41 from 1880202 to 2353112 tid 2 2024-09-13T14:27:35.510 INFO:tasks.rados.rados.0.smithi151.stdout:1610: rollback oid 43 current snap is 200 2024-09-13T14:27:35.510 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 43 to 197 2024-09-13T14:27:35.511 INFO:tasks.rados.rados.0.smithi151.stdout:1609: finishing write tid 1 to smithi15186934-41 2024-09-13T14:27:35.511 INFO:tasks.rados.rados.0.smithi151.stdout:1611: snap_create 2024-09-13T14:27:35.513 INFO:tasks.rados.rados.0.smithi151.stdout:1609: finishing write tid 2 to smithi15186934-41 2024-09-13T14:27:35.513 INFO:tasks.rados.rados.0.smithi151.stdout:1609: finishing write tid 3 to smithi15186934-41 2024-09-13T14:27:35.514 INFO:tasks.rados.rados.0.smithi151.stdout:1609: finishing write tid 4 to smithi15186934-41 2024-09-13T14:27:35.514 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 68 (ObjNum 616 snap 200 seq_num 616) dirty exists 2024-09-13T14:27:35.514 INFO:tasks.rados.rados.0.smithi151.stdout:1609: left oid 41 (ObjNum 616 snap 200 seq_num 616) 2024-09-13T14:27:35.514 INFO:tasks.rados.rados.0.smithi151.stdout:1610: finishing rollback tid 0 to smithi15186934-43 2024-09-13T14:27:35.515 INFO:tasks.rados.rados.0.smithi151.stdout:1610: finishing rollback tid 1 to smithi15186934-43 2024-09-13T14:27:35.515 INFO:tasks.rados.rados.0.smithi151.stdout:1610: finishing rollback tid 2 to smithi15186934-43 2024-09-13T14:27:35.515 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 43 v 1128 (ObjNum 497 snap 159 seq_num 497) dirty exists 2024-09-13T14:27:36.103 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:27:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:27:35.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:36.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:27:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:27:35.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:36.515 INFO:tasks.rados.rados.0.smithi151.stdout:1609: done (2 left) 2024-09-13T14:27:36.515 INFO:tasks.rados.rados.0.smithi151.stdout:1610: done (1 left) 2024-09-13T14:27:36.515 INFO:tasks.rados.rados.0.smithi151.stdout:1611: done (0 left) 2024-09-13T14:27:36.516 INFO:tasks.rados.rados.0.smithi151.stdout:1612: write oid 37 current snap is 201 2024-09-13T14:27:36.516 INFO:tasks.rados.rados.0.smithi151.stdout:1612: seq_num 617 ranges {769919=653226,1938684=431235} 2024-09-13T14:27:36.523 INFO:tasks.rados.rados.0.smithi151.stdout:1612: writing smithi15186934-37 from 769919 to 1423145 tid 1 2024-09-13T14:27:36.528 INFO:tasks.rados.rados.0.smithi151.stdout:1612: writing smithi15186934-37 from 1938684 to 2369919 tid 2 2024-09-13T14:27:36.528 INFO:tasks.rados.rados.0.smithi151.stdout:1613: snap_remove snap 196 2024-09-13T14:27:36.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:36 smithi184 ceph-mon[92527]: pgmap v315: 89 pgs: 89 active+clean; 213 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 757 KiB/s wr, 14 op/s 2024-09-13T14:27:36.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:36 smithi184 ceph-mon[92527]: osdmap e465: 8 total, 8 up, 8 in 2024-09-13T14:27:36.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:36 smithi151 ceph-mon[104450]: pgmap v315: 89 pgs: 89 active+clean; 213 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 757 KiB/s wr, 14 op/s 2024-09-13T14:27:36.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:36 smithi151 ceph-mon[104450]: osdmap e465: 8 total, 8 up, 8 in 2024-09-13T14:27:36.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:36 smithi151 ceph-mon[108966]: pgmap v315: 89 pgs: 89 active+clean; 213 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 16 MiB/s rd, 757 KiB/s wr, 14 op/s 2024-09-13T14:27:36.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:36 smithi151 ceph-mon[108966]: osdmap e465: 8 total, 8 up, 8 in 2024-09-13T14:27:37.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:27:36 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:27:36.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi184\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi184\", job=\"node\", machine=\"x86_64\", nodename=\"smithi184\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:37.522 INFO:tasks.rados.rados.0.smithi151.stdout:1613: done (1 left) 2024-09-13T14:27:37.522 INFO:tasks.rados.rados.0.smithi151.stdout:1614: read oid 48 snap -1 2024-09-13T14:27:37.522 INFO:tasks.rados.rados.0.smithi151.stdout:1614: expect deleted 2024-09-13T14:27:37.522 INFO:tasks.rados.rados.0.smithi151.stdout:1615: read oid 19 snap -1 2024-09-13T14:27:37.522 INFO:tasks.rados.rados.0.smithi151.stdout:1615: expect (ObjNum 323 snap 101 seq_num 323) 2024-09-13T14:27:37.523 INFO:tasks.rados.rados.0.smithi151.stdout:1616: snap_remove snap 197 2024-09-13T14:27:37.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:37 smithi184 ceph-mon[92527]: osdmap e466: 8 total, 8 up, 8 in 2024-09-13T14:27:37.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:37 smithi151 ceph-mon[104450]: osdmap e466: 8 total, 8 up, 8 in 2024-09-13T14:27:37.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:37 smithi151 ceph-mon[108966]: osdmap e466: 8 total, 8 up, 8 in 2024-09-13T14:27:38.527 INFO:tasks.rados.rados.0.smithi151.stdout:1616: done (3 left) 2024-09-13T14:27:38.527 INFO:tasks.rados.rados.0.smithi151.stdout:1617: write oid 41 current snap is 201 2024-09-13T14:27:38.527 INFO:tasks.rados.rados.0.smithi151.stdout:1612: finishing write tid 1 to smithi15186934-37 2024-09-13T14:27:38.527 INFO:tasks.rados.rados.0.smithi151.stdout:1617: seq_num 618 ranges {786726=728475,2280813=105913} 2024-09-13T14:27:38.535 INFO:tasks.rados.rados.0.smithi151.stdout:1617: writing smithi15186934-41 from 786726 to 1515201 tid 1 2024-09-13T14:27:38.540 INFO:tasks.rados.rados.0.smithi151.stdout:1617: writing smithi15186934-41 from 2280813 to 2386726 tid 2 2024-09-13T14:27:38.540 INFO:tasks.rados.rados.0.smithi151.stdout:1618: snap_remove snap 200 2024-09-13T14:27:38.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:38 smithi184 ceph-mon[92527]: pgmap v318: 89 pgs: 89 active+clean; 150 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 762 KiB/s wr, 9 op/s 2024-09-13T14:27:38.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:38 smithi184 ceph-mon[92527]: osdmap e467: 8 total, 8 up, 8 in 2024-09-13T14:27:38.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:38 smithi151 ceph-mon[104450]: pgmap v318: 89 pgs: 89 active+clean; 150 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 762 KiB/s wr, 9 op/s 2024-09-13T14:27:38.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:38 smithi151 ceph-mon[104450]: osdmap e467: 8 total, 8 up, 8 in 2024-09-13T14:27:38.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:38 smithi151 ceph-mon[108966]: pgmap v318: 89 pgs: 89 active+clean; 150 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 762 KiB/s wr, 9 op/s 2024-09-13T14:27:38.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:38 smithi151 ceph-mon[108966]: osdmap e467: 8 total, 8 up, 8 in 2024-09-13T14:27:39.491 INFO:tasks.rados.rados.0.smithi151.stdout:1618: done (4 left) 2024-09-13T14:27:39.491 INFO:tasks.rados.rados.0.smithi151.stdout:1619: read oid 1 snap -1 2024-09-13T14:27:39.492 INFO:tasks.rados.rados.0.smithi151.stdout:1619: expect (ObjNum 511 snap 164 seq_num 511) 2024-09-13T14:27:39.492 INFO:tasks.rados.rados.0.smithi151.stdout:1620: snap_create 2024-09-13T14:27:39.492 INFO:tasks.rados.rados.0.smithi151.stdout:1612: finishing write tid 2 to smithi15186934-37 2024-09-13T14:27:39.492 INFO:tasks.rados.rados.0.smithi151.stdout:1612: finishing write tid 3 to smithi15186934-37 2024-09-13T14:27:39.492 INFO:tasks.rados.rados.0.smithi151.stdout:1612: finishing write tid 4 to smithi15186934-37 2024-09-13T14:27:39.492 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 37 v 715 (ObjNum 617 snap 201 seq_num 617) dirty exists 2024-09-13T14:27:39.492 INFO:tasks.rados.rados.0.smithi151.stdout:1612: left oid 37 (ObjNum 617 snap 201 seq_num 617) 2024-09-13T14:27:39.492 INFO:tasks.rados.rados.0.smithi151.stdout:1615: expect (ObjNum 323 snap 101 seq_num 323) 2024-09-13T14:27:39.611 INFO:tasks.rados.rados.0.smithi151.stdout:1617: finishing write tid 1 to smithi15186934-41 2024-09-13T14:27:39.611 INFO:tasks.rados.rados.0.smithi151.stdout:1617: finishing write tid 2 to smithi15186934-41 2024-09-13T14:27:39.612 INFO:tasks.rados.rados.0.smithi151.stdout:1617: finishing write tid 3 to smithi15186934-41 2024-09-13T14:27:39.612 INFO:tasks.rados.rados.0.smithi151.stdout:1617: finishing write tid 4 to smithi15186934-41 2024-09-13T14:27:39.612 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 76 (ObjNum 618 snap 201 seq_num 618) dirty exists 2024-09-13T14:27:39.612 INFO:tasks.rados.rados.0.smithi151.stdout:1617: left oid 41 (ObjNum 618 snap 201 seq_num 618) 2024-09-13T14:27:39.612 INFO:tasks.rados.rados.0.smithi151.stdout:1619: expect (ObjNum 511 snap 164 seq_num 511) 2024-09-13T14:27:39.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:39 smithi184 ceph-mon[92527]: osdmap e468: 8 total, 8 up, 8 in 2024-09-13T14:27:39.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:39 smithi184 ceph-mon[92527]: osdmap e469: 8 total, 8 up, 8 in 2024-09-13T14:27:39.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:39 smithi151 ceph-mon[104450]: osdmap e468: 8 total, 8 up, 8 in 2024-09-13T14:27:39.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:39 smithi151 ceph-mon[104450]: osdmap e469: 8 total, 8 up, 8 in 2024-09-13T14:27:39.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:39 smithi151 ceph-mon[108966]: osdmap e468: 8 total, 8 up, 8 in 2024-09-13T14:27:39.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:39 smithi151 ceph-mon[108966]: osdmap e469: 8 total, 8 up, 8 in 2024-09-13T14:27:40.496 INFO:tasks.rados.rados.0.smithi151.stdout:1612: done (5 left) 2024-09-13T14:27:40.496 INFO:tasks.rados.rados.0.smithi151.stdout:1614: done (4 left) 2024-09-13T14:27:40.496 INFO:tasks.rados.rados.0.smithi151.stdout:1615: done (3 left) 2024-09-13T14:27:40.496 INFO:tasks.rados.rados.0.smithi151.stdout:1617: done (2 left) 2024-09-13T14:27:40.496 INFO:tasks.rados.rados.0.smithi151.stdout:1619: done (1 left) 2024-09-13T14:27:40.496 INFO:tasks.rados.rados.0.smithi151.stdout:1620: done (0 left) 2024-09-13T14:27:40.496 INFO:tasks.rados.rados.0.smithi151.stdout:1621: snap_remove snap 190 2024-09-13T14:27:40.534 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:27:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:27:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:40.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:40 smithi184 ceph-mon[92527]: pgmap v321: 89 pgs: 89 active+clean; 150 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 762 KiB/s wr, 9 op/s 2024-09-13T14:27:40.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:40 smithi184 ceph-mon[92527]: osdmap e470: 8 total, 8 up, 8 in 2024-09-13T14:27:40.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:40 smithi151 ceph-mon[104450]: pgmap v321: 89 pgs: 89 active+clean; 150 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 762 KiB/s wr, 9 op/s 2024-09-13T14:27:40.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:40 smithi151 ceph-mon[104450]: osdmap e470: 8 total, 8 up, 8 in 2024-09-13T14:27:40.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:40 smithi151 ceph-mon[108966]: pgmap v321: 89 pgs: 89 active+clean; 150 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 762 KiB/s wr, 9 op/s 2024-09-13T14:27:40.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:40 smithi151 ceph-mon[108966]: osdmap e470: 8 total, 8 up, 8 in 2024-09-13T14:27:41.497 INFO:tasks.rados.rados.0.smithi151.stdout:1621: done (0 left) 2024-09-13T14:27:41.497 INFO:tasks.rados.rados.0.smithi151.stdout:1622: read oid 34 snap -1 2024-09-13T14:27:41.497 INFO:tasks.rados.rados.0.smithi151.stdout:1622: expect (ObjNum 539 snap 173 seq_num 539) 2024-09-13T14:27:41.497 INFO:tasks.rados.rados.0.smithi151.stdout:1623: read oid 35 snap -1 2024-09-13T14:27:41.497 INFO:tasks.rados.rados.0.smithi151.stdout:1623: expect (ObjNum 541 snap 174 seq_num 541) 2024-09-13T14:27:41.497 INFO:tasks.rados.rados.0.smithi151.stdout:1624: read oid 30 snap -1 2024-09-13T14:27:41.497 INFO:tasks.rados.rados.0.smithi151.stdout:1624: expect deleted 2024-09-13T14:27:41.497 INFO:tasks.rados.rados.0.smithi151.stdout:1625: delete oid 14 current snap is 202 2024-09-13T14:27:41.505 INFO:tasks.rados.rados.0.smithi151.stdout:1622: expect (ObjNum 539 snap 173 seq_num 539) 2024-09-13T14:27:41.644 INFO:tasks.rados.rados.0.smithi151.stdout:1623: expect (ObjNum 541 snap 174 seq_num 541) 2024-09-13T14:27:41.785 INFO:tasks.rados.rados.0.smithi151.stdout:1622: done (3 left) 2024-09-13T14:27:41.785 INFO:tasks.rados.rados.0.smithi151.stdout:1623: done (2 left) 2024-09-13T14:27:41.785 INFO:tasks.rados.rados.0.smithi151.stdout:1624: done (1 left) 2024-09-13T14:27:41.785 INFO:tasks.rados.rados.0.smithi151.stdout:1625: done (0 left) 2024-09-13T14:27:41.785 INFO:tasks.rados.rados.0.smithi151.stdout:1626: delete oid 11 current snap is 202 2024-09-13T14:27:41.787 INFO:tasks.rados.rados.0.smithi151.stdout:1626: done (0 left) 2024-09-13T14:27:41.787 INFO:tasks.rados.rados.0.smithi151.stdout:1627: read oid 13 snap -1 2024-09-13T14:27:41.787 INFO:tasks.rados.rados.0.smithi151.stdout:1627: expect (ObjNum 556 snap 176 seq_num 556) 2024-09-13T14:27:41.787 INFO:tasks.rados.rados.0.smithi151.stdout:1628: read oid 46 snap -1 2024-09-13T14:27:41.787 INFO:tasks.rados.rados.0.smithi151.stdout:1628: expect deleted 2024-09-13T14:27:41.788 INFO:tasks.rados.rados.0.smithi151.stdout:1629: write oid 39 current snap is 202 2024-09-13T14:27:41.788 INFO:tasks.rados.rados.0.smithi151.stdout:1629: seq_num 621 ranges {437147=554222,1656259=655870} 2024-09-13T14:27:41.792 INFO:tasks.rados.rados.0.smithi151.stdout:1629: writing smithi15186934-39 from 437147 to 991369 tid 1 2024-09-13T14:27:41.800 INFO:tasks.rados.rados.0.smithi151.stdout:1629: writing smithi15186934-39 from 1656259 to 2312129 tid 2 2024-09-13T14:27:41.800 INFO:tasks.rados.rados.0.smithi151.stdout:1630: delete oid 22 current snap is 202 2024-09-13T14:27:41.800 INFO:tasks.rados.rados.0.smithi151.stdout:1627: expect (ObjNum 556 snap 176 seq_num 556) 2024-09-13T14:27:41.993 INFO:tasks.rados.rados.0.smithi151.stdout:1629: finishing write tid 1 to smithi15186934-39 2024-09-13T14:27:41.994 INFO:tasks.rados.rados.0.smithi151.stdout:1629: finishing write tid 2 to smithi15186934-39 2024-09-13T14:27:41.994 INFO:tasks.rados.rados.0.smithi151.stdout:1629: finishing write tid 3 to smithi15186934-39 2024-09-13T14:27:41.994 INFO:tasks.rados.rados.0.smithi151.stdout:1629: finishing write tid 4 to smithi15186934-39 2024-09-13T14:27:41.994 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 548 (ObjNum 621 snap 202 seq_num 621) dirty exists 2024-09-13T14:27:41.994 INFO:tasks.rados.rados.0.smithi151.stdout:1629: left oid 39 (ObjNum 621 snap 202 seq_num 621) 2024-09-13T14:27:41.994 INFO:tasks.rados.rados.0.smithi151.stdout:1627: done (3 left) 2024-09-13T14:27:41.994 INFO:tasks.rados.rados.0.smithi151.stdout:1628: done (2 left) 2024-09-13T14:27:41.994 INFO:tasks.rados.rados.0.smithi151.stdout:1629: done (1 left) 2024-09-13T14:27:41.994 INFO:tasks.rados.rados.0.smithi151.stdout:1630: done (0 left) 2024-09-13T14:27:41.994 INFO:tasks.rados.rados.0.smithi151.stdout:1631: read oid 36 snap 201 2024-09-13T14:27:41.994 INFO:tasks.rados.rados.0.smithi151.stdout:1631: expect (ObjNum 588 snap 190 seq_num 588) 2024-09-13T14:27:41.994 INFO:tasks.rados.rados.0.smithi151.stdout:1632: snap_remove snap 199 2024-09-13T14:27:42.539 INFO:tasks.rados.rados.0.smithi151.stdout:1632: done (1 left) 2024-09-13T14:27:42.539 INFO:tasks.rados.rados.0.smithi151.stdout:1633: delete oid 25 current snap is 202 2024-09-13T14:27:42.539 INFO:tasks.rados.rados.0.smithi151.stdout:1631: expect (ObjNum 588 snap 190 seq_num 588) 2024-09-13T14:27:42.668 INFO:tasks.rados.rados.0.smithi151.stdout:1631: done (1 left) 2024-09-13T14:27:42.668 INFO:tasks.rados.rados.0.smithi151.stdout:1633: done (0 left) 2024-09-13T14:27:42.668 INFO:tasks.rados.rados.0.smithi151.stdout:1634: read oid 27 snap 198 2024-09-13T14:27:42.669 INFO:tasks.rados.rados.0.smithi151.stdout:1634: expect (ObjNum 601 snap 193 seq_num 601) 2024-09-13T14:27:42.669 INFO:tasks.rados.rados.0.smithi151.stdout:1635: read oid 4 snap -1 2024-09-13T14:27:42.669 INFO:tasks.rados.rados.0.smithi151.stdout:1635: expect (ObjNum 592 snap 191 seq_num 592) 2024-09-13T14:27:42.669 INFO:tasks.rados.rados.0.smithi151.stdout:1636: read oid 28 snap -1 2024-09-13T14:27:42.669 INFO:tasks.rados.rados.0.smithi151.stdout:1636: expect deleted 2024-09-13T14:27:42.669 INFO:tasks.rados.rados.0.smithi151.stdout:1637: snap_remove snap 201 2024-09-13T14:27:42.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:42 smithi184 ceph-mon[92527]: pgmap v324: 89 pgs: 89 active+clean; 150 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:27:42.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:42 smithi184 ceph-mon[92527]: osdmap e471: 8 total, 8 up, 8 in 2024-09-13T14:27:42.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:42 smithi151 ceph-mon[104450]: pgmap v324: 89 pgs: 89 active+clean; 150 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:27:42.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:42 smithi151 ceph-mon[104450]: osdmap e471: 8 total, 8 up, 8 in 2024-09-13T14:27:42.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:42 smithi151 ceph-mon[108966]: pgmap v324: 89 pgs: 89 active+clean; 150 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:27:42.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:42 smithi151 ceph-mon[108966]: osdmap e471: 8 total, 8 up, 8 in 2024-09-13T14:27:43.545 INFO:tasks.rados.rados.0.smithi151.stdout:1637: done (3 left) 2024-09-13T14:27:43.545 INFO:tasks.rados.rados.0.smithi151.stdout:1638: write oid 34 current snap is 202 2024-09-13T14:27:43.545 INFO:tasks.rados.rados.0.smithi151.stdout:1638: seq_num 624 ranges {487568=496322,1431705=519550} 2024-09-13T14:27:43.550 INFO:tasks.rados.rados.0.smithi151.stdout:1638: writing smithi15186934-34 from 487568 to 983890 tid 1 2024-09-13T14:27:43.556 INFO:tasks.rados.rados.0.smithi151.stdout:1638: writing smithi15186934-34 from 1431705 to 1951255 tid 2 2024-09-13T14:27:43.556 INFO:tasks.rados.rados.0.smithi151.stdout:1639: read oid 33 snap -1 2024-09-13T14:27:43.557 INFO:tasks.rados.rados.0.smithi151.stdout:1639: expect (ObjNum 508 snap 164 seq_num 508) 2024-09-13T14:27:43.557 INFO:tasks.rados.rados.0.smithi151.stdout:1634: expect (ObjNum 601 snap 193 seq_num 601) 2024-09-13T14:27:43.642 INFO:tasks.rados.rados.0.smithi151.stdout:1635: expect (ObjNum 592 snap 191 seq_num 592) 2024-09-13T14:27:43.770 INFO:tasks.rados.rados.0.smithi151.stdout:1638: finishing write tid 1 to smithi15186934-34 2024-09-13T14:27:43.770 INFO:tasks.rados.rados.0.smithi151.stdout:1638: finishing write tid 2 to smithi15186934-34 2024-09-13T14:27:43.770 INFO:tasks.rados.rados.0.smithi151.stdout:1638: finishing write tid 3 to smithi15186934-34 2024-09-13T14:27:43.770 INFO:tasks.rados.rados.0.smithi151.stdout:1638: finishing write tid 4 to smithi15186934-34 2024-09-13T14:27:43.770 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 600 (ObjNum 624 snap 202 seq_num 624) dirty exists 2024-09-13T14:27:43.770 INFO:tasks.rados.rados.0.smithi151.stdout:1638: left oid 34 (ObjNum 624 snap 202 seq_num 624) 2024-09-13T14:27:43.770 INFO:tasks.rados.rados.0.smithi151.stdout:1634: done (4 left) 2024-09-13T14:27:43.771 INFO:tasks.rados.rados.0.smithi151.stdout:1635: done (3 left) 2024-09-13T14:27:43.771 INFO:tasks.rados.rados.0.smithi151.stdout:1636: done (2 left) 2024-09-13T14:27:43.771 INFO:tasks.rados.rados.0.smithi151.stdout:1638: done (1 left) 2024-09-13T14:27:43.771 INFO:tasks.rados.rados.0.smithi151.stdout:1640: snap_create 2024-09-13T14:27:43.773 INFO:tasks.rados.rados.0.smithi151.stdout:1639: expect (ObjNum 508 snap 164 seq_num 508) 2024-09-13T14:27:43.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:43 smithi184 ceph-mon[92527]: osdmap e472: 8 total, 8 up, 8 in 2024-09-13T14:27:43.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:43 smithi151 ceph-mon[104450]: osdmap e472: 8 total, 8 up, 8 in 2024-09-13T14:27:43.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:43 smithi151 ceph-mon[108966]: osdmap e472: 8 total, 8 up, 8 in 2024-09-13T14:27:44.492 INFO:tasks.rados.rados.0.smithi151.stdout:1639: done (1 left) 2024-09-13T14:27:44.493 INFO:tasks.rados.rados.0.smithi151.stdout:1640: done (0 left) 2024-09-13T14:27:44.493 INFO:tasks.rados.rados.0.smithi151.stdout:1641: rollback oid 21 current snap is 203 2024-09-13T14:27:44.493 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 21 to 202 2024-09-13T14:27:44.493 INFO:tasks.rados.rados.0.smithi151.stdout:1642: rollback oid 48 current snap is 203 2024-09-13T14:27:44.493 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 48 to 198 2024-09-13T14:27:44.493 INFO:tasks.rados.rados.0.smithi151.stdout:1643: snap_create 2024-09-13T14:27:44.494 INFO:tasks.rados.rados.0.smithi151.stdout:1642: finishing rollback tid 1 to smithi15186934-48 2024-09-13T14:27:44.494 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 48 v 0 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-13T14:27:44.498 INFO:tasks.rados.rados.0.smithi151.stdout:1641: finishing rollback tid 0 to smithi15186934-21 2024-09-13T14:27:44.498 INFO:tasks.rados.rados.0.smithi151.stdout:1641: finishing rollback tid 1 to smithi15186934-21 2024-09-13T14:27:44.498 INFO:tasks.rados.rados.0.smithi151.stdout:1641: finishing rollback tid 2 to smithi15186934-21 2024-09-13T14:27:44.498 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 21 v 383 (ObjNum 578 snap 186 seq_num 578) dirty exists 2024-09-13T14:27:44.542 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:27:44 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:27:44.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)\", cluster_addr=\"172.21.15.151\", device_class=\"ssd\", hostname=\"smithi151\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.151\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:44.856 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:44 smithi184 ceph-mon[92527]: pgmap v327: 89 pgs: 89 active+clean; 107 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 500 KiB/s wr, 16 op/s 2024-09-13T14:27:44.857 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:44 smithi184 ceph-mon[92527]: osdmap e473: 8 total, 8 up, 8 in 2024-09-13T14:27:44.857 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:44 smithi184 ceph-mon[92527]: osdmap e474: 8 total, 8 up, 8 in 2024-09-13T14:27:44.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:44 smithi151 ceph-mon[104450]: pgmap v327: 89 pgs: 89 active+clean; 107 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 500 KiB/s wr, 16 op/s 2024-09-13T14:27:44.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:44 smithi151 ceph-mon[104450]: osdmap e473: 8 total, 8 up, 8 in 2024-09-13T14:27:44.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:44 smithi151 ceph-mon[104450]: osdmap e474: 8 total, 8 up, 8 in 2024-09-13T14:27:44.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:44 smithi151 ceph-mon[108966]: pgmap v327: 89 pgs: 89 active+clean; 107 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 500 KiB/s wr, 16 op/s 2024-09-13T14:27:44.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:44 smithi151 ceph-mon[108966]: osdmap e473: 8 total, 8 up, 8 in 2024-09-13T14:27:44.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:44 smithi151 ceph-mon[108966]: osdmap e474: 8 total, 8 up, 8 in 2024-09-13T14:27:44.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:27:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:27:44] "GET /metrics HTTP/1.1" 200 36399 "" "Prometheus/2.51.0" 2024-09-13T14:27:45.494 INFO:tasks.rados.rados.0.smithi151.stdout:1641: done (2 left) 2024-09-13T14:27:45.495 INFO:tasks.rados.rados.0.smithi151.stdout:1642: done (1 left) 2024-09-13T14:27:45.495 INFO:tasks.rados.rados.0.smithi151.stdout:1643: done (0 left) 2024-09-13T14:27:45.495 INFO:tasks.rados.rados.0.smithi151.stdout:1644: write oid 40 current snap is 204 2024-09-13T14:27:45.495 INFO:tasks.rados.rados.0.smithi151.stdout:1644: seq_num 625 ranges {504375=571571,1490187=663208} 2024-09-13T14:27:45.500 INFO:tasks.rados.rados.0.smithi151.stdout:1644: writing smithi15186934-40 from 504375 to 1075946 tid 1 2024-09-13T14:27:45.507 INFO:tasks.rados.rados.0.smithi151.stdout:1644: writing smithi15186934-40 from 1490187 to 2153395 tid 2 2024-09-13T14:27:45.507 INFO:tasks.rados.rados.0.smithi151.stdout:1645: rollback oid 8 current snap is 204 2024-09-13T14:27:45.507 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 8 to 203 2024-09-13T14:27:45.507 INFO:tasks.rados.rados.0.smithi151.stdout:1644: finishing write tid 1 to smithi15186934-40 2024-09-13T14:27:45.507 INFO:tasks.rados.rados.0.smithi151.stdout:1646: write oid 47 current snap is 204 2024-09-13T14:27:45.507 INFO:tasks.rados.rados.0.smithi151.stdout:1646: seq_num 626 ranges {521182=646820,1948669=523219} 2024-09-13T14:27:45.513 INFO:tasks.rados.rados.0.smithi151.stdout:1646: writing smithi15186934-47 from 521182 to 1168002 tid 1 2024-09-13T14:27:45.521 INFO:tasks.rados.rados.0.smithi151.stdout:1646: writing smithi15186934-47 from 1948669 to 2471888 tid 2 2024-09-13T14:27:45.521 INFO:tasks.rados.rados.0.smithi151.stdout:1647: read oid 7 snap -1 2024-09-13T14:27:45.521 INFO:tasks.rados.rados.0.smithi151.stdout:1647: expect deleted 2024-09-13T14:27:45.521 INFO:tasks.rados.rados.0.smithi151.stdout:1648: write oid 13 current snap is 204 2024-09-13T14:27:45.521 INFO:tasks.rados.rados.0.smithi151.stdout:1644: finishing write tid 2 to smithi15186934-40 2024-09-13T14:27:45.521 INFO:tasks.rados.rados.0.smithi151.stdout:1644: finishing write tid 3 to smithi15186934-40 2024-09-13T14:27:45.521 INFO:tasks.rados.rados.0.smithi151.stdout:1644: finishing write tid 4 to smithi15186934-40 2024-09-13T14:27:45.521 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 40 v 603 (ObjNum 625 snap 204 seq_num 625) dirty exists 2024-09-13T14:27:45.521 INFO:tasks.rados.rados.0.smithi151.stdout:1644: left oid 40 (ObjNum 625 snap 204 seq_num 625) 2024-09-13T14:27:45.521 INFO:tasks.rados.rados.0.smithi151.stdout:1645: finishing rollback tid 0 to smithi15186934-8 2024-09-13T14:27:45.522 INFO:tasks.rados.rados.0.smithi151.stdout:1648: seq_num 627 ranges {537989=722069,1890798=647191} 2024-09-13T14:27:45.527 INFO:tasks.rados.rados.0.smithi151.stdout:1648: writing smithi15186934-13 from 537989 to 1260058 tid 1 2024-09-13T14:27:45.535 INFO:tasks.rados.rados.0.smithi151.stdout:1648: writing smithi15186934-13 from 1890798 to 2537989 tid 2 2024-09-13T14:27:45.535 INFO:tasks.rados.rados.0.smithi151.stdout:1644: done (4 left) 2024-09-13T14:27:45.535 INFO:tasks.rados.rados.0.smithi151.stdout:1649: write oid 23 current snap is 204 2024-09-13T14:27:45.535 INFO:tasks.rados.rados.0.smithi151.stdout:1645: finishing rollback tid 1 to smithi15186934-8 2024-09-13T14:27:45.535 INFO:tasks.rados.rados.0.smithi151.stdout:1645: finishing rollback tid 2 to smithi15186934-8 2024-09-13T14:27:45.535 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 530 (ObjNum 602 snap 193 seq_num 602) dirty exists 2024-09-13T14:27:45.536 INFO:tasks.rados.rados.0.smithi151.stdout:1646: finishing write tid 1 to smithi15186934-47 2024-09-13T14:27:45.536 INFO:tasks.rados.rados.0.smithi151.stdout:1649: seq_num 628 ranges {554796=797318,1949280=526888} 2024-09-13T14:27:45.542 INFO:tasks.rados.rados.0.smithi151.stdout:1649: writing smithi15186934-23 from 554796 to 1352114 tid 1 2024-09-13T14:27:45.549 INFO:tasks.rados.rados.0.smithi151.stdout:1649: writing smithi15186934-23 from 1949280 to 2476168 tid 2 2024-09-13T14:27:45.549 INFO:tasks.rados.rados.0.smithi151.stdout:1645: done (4 left) 2024-09-13T14:27:45.549 INFO:tasks.rados.rados.0.smithi151.stdout:1650: read oid 36 snap -1 2024-09-13T14:27:45.549 INFO:tasks.rados.rados.0.smithi151.stdout:1650: expect (ObjNum 588 snap 190 seq_num 588) 2024-09-13T14:27:45.549 INFO:tasks.rados.rados.0.smithi151.stdout:1651: snap_remove snap 203 2024-09-13T14:27:46.103 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:27:45 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:27:45.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:46.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:27:45 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:27:45.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:46.549 INFO:tasks.rados.rados.0.smithi151.stdout:1651: done (5 left) 2024-09-13T14:27:46.549 INFO:tasks.rados.rados.0.smithi151.stdout:1652: rollback oid 9 current snap is 204 2024-09-13T14:27:46.549 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 9 to 202 2024-09-13T14:27:46.550 INFO:tasks.rados.rados.0.smithi151.stdout:1646: finishing write tid 2 to smithi15186934-47 2024-09-13T14:27:46.550 INFO:tasks.rados.rados.0.smithi151.stdout:1646: finishing write tid 3 to smithi15186934-47 2024-09-13T14:27:46.550 INFO:tasks.rados.rados.0.smithi151.stdout:1646: finishing write tid 4 to smithi15186934-47 2024-09-13T14:27:46.550 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 734 (ObjNum 626 snap 204 seq_num 626) dirty exists 2024-09-13T14:27:46.550 INFO:tasks.rados.rados.0.smithi151.stdout:1646: left oid 47 (ObjNum 626 snap 204 seq_num 626) 2024-09-13T14:27:46.550 INFO:tasks.rados.rados.0.smithi151.stdout:1648: finishing write tid 1 to smithi15186934-13 2024-09-13T14:27:46.550 INFO:tasks.rados.rados.0.smithi151.stdout:1648: finishing write tid 2 to smithi15186934-13 2024-09-13T14:27:46.550 INFO:tasks.rados.rados.0.smithi151.stdout:1648: finishing write tid 3 to smithi15186934-13 2024-09-13T14:27:46.550 INFO:tasks.rados.rados.0.smithi151.stdout:1648: finishing write tid 4 to smithi15186934-13 2024-09-13T14:27:46.550 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 13 v 630 (ObjNum 627 snap 204 seq_num 627) dirty exists 2024-09-13T14:27:46.550 INFO:tasks.rados.rados.0.smithi151.stdout:1648: left oid 13 (ObjNum 627 snap 204 seq_num 627) 2024-09-13T14:27:46.550 INFO:tasks.rados.rados.0.smithi151.stdout:1649: finishing write tid 1 to smithi15186934-23 2024-09-13T14:27:46.550 INFO:tasks.rados.rados.0.smithi151.stdout:1646: done (5 left) 2024-09-13T14:27:46.550 INFO:tasks.rados.rados.0.smithi151.stdout:1647: done (4 left) 2024-09-13T14:27:46.551 INFO:tasks.rados.rados.0.smithi151.stdout:1648: done (3 left) 2024-09-13T14:27:46.551 INFO:tasks.rados.rados.0.smithi151.stdout:1653: read oid 18 snap -1 2024-09-13T14:27:46.551 INFO:tasks.rados.rados.0.smithi151.stdout:1653: expect (ObjNum 491 snap 158 seq_num 491) 2024-09-13T14:27:46.551 INFO:tasks.rados.rados.0.smithi151.stdout:1654: snap_create 2024-09-13T14:27:46.551 INFO:tasks.rados.rados.0.smithi151.stdout:1649: finishing write tid 2 to smithi15186934-23 2024-09-13T14:27:46.551 INFO:tasks.rados.rados.0.smithi151.stdout:1649: finishing write tid 3 to smithi15186934-23 2024-09-13T14:27:46.551 INFO:tasks.rados.rados.0.smithi151.stdout:1649: finishing write tid 4 to smithi15186934-23 2024-09-13T14:27:46.551 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 23 v 738 (ObjNum 628 snap 204 seq_num 628) dirty exists 2024-09-13T14:27:46.551 INFO:tasks.rados.rados.0.smithi151.stdout:1649: left oid 23 (ObjNum 628 snap 204 seq_num 628) 2024-09-13T14:27:46.551 INFO:tasks.rados.rados.0.smithi151.stdout:1650: expect (ObjNum 588 snap 190 seq_num 588) 2024-09-13T14:27:46.679 INFO:tasks.rados.rados.0.smithi151.stdout:1652: finishing rollback tid 1 to smithi15186934-9 2024-09-13T14:27:46.679 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 0 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-13T14:27:46.679 INFO:tasks.rados.rados.0.smithi151.stdout:1653: expect (ObjNum 491 snap 158 seq_num 491) 2024-09-13T14:27:46.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:46 smithi184 ceph-mon[92527]: pgmap v330: 89 pgs: 89 active+clean; 107 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 500 KiB/s wr, 16 op/s 2024-09-13T14:27:46.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:46 smithi184 ceph-mon[92527]: osdmap e475: 8 total, 8 up, 8 in 2024-09-13T14:27:46.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:46 smithi151 ceph-mon[104450]: pgmap v330: 89 pgs: 89 active+clean; 107 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 500 KiB/s wr, 16 op/s 2024-09-13T14:27:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:46 smithi151 ceph-mon[104450]: osdmap e475: 8 total, 8 up, 8 in 2024-09-13T14:27:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:46 smithi151 ceph-mon[108966]: pgmap v330: 89 pgs: 89 active+clean; 107 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 500 KiB/s wr, 16 op/s 2024-09-13T14:27:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:46 smithi151 ceph-mon[108966]: osdmap e475: 8 total, 8 up, 8 in 2024-09-13T14:27:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:47.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:27:46 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:27:46.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi151\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi151\", job=\"node\", machine=\"x86_64\", nodename=\"smithi151\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi151\", job=\"node\", machine=\"x86_64\", nodename=\"smithi151\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:47.556 INFO:tasks.rados.rados.0.smithi151.stdout:1649: done (4 left) 2024-09-13T14:27:47.557 INFO:tasks.rados.rados.0.smithi151.stdout:1650: done (3 left) 2024-09-13T14:27:47.557 INFO:tasks.rados.rados.0.smithi151.stdout:1652: done (2 left) 2024-09-13T14:27:47.557 INFO:tasks.rados.rados.0.smithi151.stdout:1653: done (1 left) 2024-09-13T14:27:47.557 INFO:tasks.rados.rados.0.smithi151.stdout:1654: done (0 left) 2024-09-13T14:27:47.557 INFO:tasks.rados.rados.0.smithi151.stdout:1655: delete oid 27 current snap is 205 2024-09-13T14:27:47.562 INFO:tasks.rados.rados.0.smithi151.stdout:1655: done (0 left) 2024-09-13T14:27:47.562 INFO:tasks.rados.rados.0.smithi151.stdout:1656: delete oid 15 current snap is 205 2024-09-13T14:27:47.566 INFO:tasks.rados.rados.0.smithi151.stdout:1656: done (0 left) 2024-09-13T14:27:47.566 INFO:tasks.rados.rados.0.smithi151.stdout:1657: snap_remove snap 204 2024-09-13T14:27:47.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:47 smithi184 ceph-mon[92527]: osdmap e476: 8 total, 8 up, 8 in 2024-09-13T14:27:47.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:47 smithi151 ceph-mon[104450]: osdmap e476: 8 total, 8 up, 8 in 2024-09-13T14:27:47.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:47 smithi151 ceph-mon[108966]: osdmap e476: 8 total, 8 up, 8 in 2024-09-13T14:27:48.557 INFO:tasks.rados.rados.0.smithi151.stdout:1657: done (0 left) 2024-09-13T14:27:48.558 INFO:tasks.rados.rados.0.smithi151.stdout:1658: read oid 37 snap 198 2024-09-13T14:27:48.558 INFO:tasks.rados.rados.0.smithi151.stdout:1658: expect (ObjNum 566 snap 177 seq_num 566) 2024-09-13T14:27:48.558 INFO:tasks.rados.rados.0.smithi151.stdout:1659: rollback oid 14 current snap is 205 2024-09-13T14:27:48.558 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 14 to 198 2024-09-13T14:27:48.558 INFO:tasks.rados.rados.0.smithi151.stdout:1660: write oid 30 current snap is 205 2024-09-13T14:27:48.558 INFO:tasks.rados.rados.0.smithi151.stdout:1660: seq_num 631 ranges {605217=739418,2124726=480491} 2024-09-13T14:27:48.565 INFO:tasks.rados.rados.0.smithi151.stdout:1660: writing smithi15186934-30 from 605217 to 1344635 tid 1 2024-09-13T14:27:48.572 INFO:tasks.rados.rados.0.smithi151.stdout:1660: writing smithi15186934-30 from 2124726 to 2605217 tid 2 2024-09-13T14:27:48.573 INFO:tasks.rados.rados.0.smithi151.stdout:1661: delete oid 9 current snap is 205 2024-09-13T14:27:48.573 INFO:tasks.rados.rados.0.smithi151.stdout:1659: finishing rollback tid 1 to smithi15186934-14 2024-09-13T14:27:48.573 INFO:tasks.rados.rados.0.smithi151.stdout:1659: finishing rollback tid 2 to smithi15186934-14 2024-09-13T14:27:48.573 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 710 (ObjNum 611 snap 197 seq_num 611) dirty exists 2024-09-13T14:27:48.573 INFO:tasks.rados.rados.0.smithi151.stdout:1658: expect (ObjNum 566 snap 177 seq_num 566) 2024-09-13T14:27:48.686 INFO:tasks.rados.rados.0.smithi151.stdout:1660: finishing write tid 1 to smithi15186934-30 2024-09-13T14:27:48.686 INFO:tasks.rados.rados.0.smithi151.stdout:1660: finishing write tid 2 to smithi15186934-30 2024-09-13T14:27:48.686 INFO:tasks.rados.rados.0.smithi151.stdout:1660: finishing write tid 3 to smithi15186934-30 2024-09-13T14:27:48.686 INFO:tasks.rados.rados.0.smithi151.stdout:1660: finishing write tid 4 to smithi15186934-30 2024-09-13T14:27:48.686 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 30 v 606 (ObjNum 631 snap 205 seq_num 631) dirty exists 2024-09-13T14:27:48.686 INFO:tasks.rados.rados.0.smithi151.stdout:1660: left oid 30 (ObjNum 631 snap 205 seq_num 631) 2024-09-13T14:27:48.686 INFO:tasks.rados.rados.0.smithi151.stdout:1658: done (3 left) 2024-09-13T14:27:48.686 INFO:tasks.rados.rados.0.smithi151.stdout:1659: done (2 left) 2024-09-13T14:27:48.686 INFO:tasks.rados.rados.0.smithi151.stdout:1660: done (1 left) 2024-09-13T14:27:48.686 INFO:tasks.rados.rados.0.smithi151.stdout:1661: done (0 left) 2024-09-13T14:27:48.687 INFO:tasks.rados.rados.0.smithi151.stdout:1662: snap_create 2024-09-13T14:27:48.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:48 smithi184 ceph-mon[92527]: pgmap v333: 89 pgs: 89 active+clean; 115 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 1.5 MiB/s wr, 17 op/s 2024-09-13T14:27:48.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:48 smithi184 ceph-mon[92527]: osdmap e477: 8 total, 8 up, 8 in 2024-09-13T14:27:48.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:48 smithi151 ceph-mon[104450]: pgmap v333: 89 pgs: 89 active+clean; 115 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 1.5 MiB/s wr, 17 op/s 2024-09-13T14:27:48.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:48 smithi151 ceph-mon[104450]: osdmap e477: 8 total, 8 up, 8 in 2024-09-13T14:27:48.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:48 smithi151 ceph-mon[108966]: pgmap v333: 89 pgs: 89 active+clean; 115 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 1.5 MiB/s wr, 17 op/s 2024-09-13T14:27:48.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:48 smithi151 ceph-mon[108966]: osdmap e477: 8 total, 8 up, 8 in 2024-09-13T14:27:49.494 INFO:tasks.rados.rados.0.smithi151.stdout:1662: done (0 left) 2024-09-13T14:27:49.495 INFO:tasks.rados.rados.0.smithi151.stdout:1663: write oid 33 current snap is 206 2024-09-13T14:27:49.495 INFO:tasks.rados.rados.0.smithi151.stdout:1663: seq_num 633 ranges {638831=489916,1841690=677884} 2024-09-13T14:27:49.500 INFO:tasks.rados.rados.0.smithi151.stdout:1663: writing smithi15186934-33 from 638831 to 1128747 tid 1 2024-09-13T14:27:49.508 INFO:tasks.rados.rados.0.smithi151.stdout:1663: writing smithi15186934-33 from 1841690 to 2519574 tid 2 2024-09-13T14:27:49.508 INFO:tasks.rados.rados.0.smithi151.stdout:1664: delete oid 15 current snap is 206 2024-09-13T14:27:49.508 INFO:tasks.rados.rados.0.smithi151.stdout:1663: finishing write tid 1 to smithi15186934-33 2024-09-13T14:27:49.511 INFO:tasks.rados.rados.0.smithi151.stdout:1664: done (1 left) 2024-09-13T14:27:49.511 INFO:tasks.rados.rados.0.smithi151.stdout:1665: write oid 49 current snap is 206 2024-09-13T14:27:49.511 INFO:tasks.rados.rados.0.smithi151.stdout:1665: seq_num 635 ranges {672445=640414,1842301=681553} 2024-09-13T14:27:49.518 INFO:tasks.rados.rados.0.smithi151.stdout:1665: writing smithi15186934-49 from 672445 to 1312859 tid 1 2024-09-13T14:27:49.526 INFO:tasks.rados.rados.0.smithi151.stdout:1665: writing smithi15186934-49 from 1842301 to 2523854 tid 2 2024-09-13T14:27:49.526 INFO:tasks.rados.rados.0.smithi151.stdout:1666: read oid 37 snap -1 2024-09-13T14:27:49.526 INFO:tasks.rados.rados.0.smithi151.stdout:1666: expect (ObjNum 617 snap 201 seq_num 617) 2024-09-13T14:27:49.526 INFO:tasks.rados.rados.0.smithi151.stdout:1667: write oid 50 current snap is 206 2024-09-13T14:27:49.526 INFO:tasks.rados.rados.0.smithi151.stdout:1663: finishing write tid 2 to smithi15186934-33 2024-09-13T14:27:49.526 INFO:tasks.rados.rados.0.smithi151.stdout:1663: finishing write tid 3 to smithi15186934-33 2024-09-13T14:27:49.526 INFO:tasks.rados.rados.0.smithi151.stdout:1663: finishing write tid 4 to smithi15186934-33 2024-09-13T14:27:49.527 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 33 v 636 (ObjNum 633 snap 206 seq_num 633) dirty exists 2024-09-13T14:27:49.527 INFO:tasks.rados.rados.0.smithi151.stdout:1663: left oid 33 (ObjNum 633 snap 206 seq_num 633) 2024-09-13T14:27:49.527 INFO:tasks.rados.rados.0.smithi151.stdout:1665: finishing write tid 1 to smithi15186934-49 2024-09-13T14:27:49.527 INFO:tasks.rados.rados.0.smithi151.stdout:1667: seq_num 636 ranges {689252=715663,1900783=425211} 2024-09-13T14:27:49.533 INFO:tasks.rados.rados.0.smithi151.stdout:1667: writing smithi15186934-50 from 689252 to 1404915 tid 1 2024-09-13T14:27:49.539 INFO:tasks.rados.rados.0.smithi151.stdout:1667: writing smithi15186934-50 from 1900783 to 2325994 tid 2 2024-09-13T14:27:49.539 INFO:tasks.rados.rados.0.smithi151.stdout:1663: done (3 left) 2024-09-13T14:27:49.539 INFO:tasks.rados.rados.0.smithi151.stdout:1668: delete oid 7 current snap is 206 2024-09-13T14:27:49.539 INFO:tasks.rados.rados.0.smithi151.stdout:1665: finishing write tid 2 to smithi15186934-49 2024-09-13T14:27:49.540 INFO:tasks.rados.rados.0.smithi151.stdout:1665: finishing write tid 3 to smithi15186934-49 2024-09-13T14:27:49.540 INFO:tasks.rados.rados.0.smithi151.stdout:1665: finishing write tid 4 to smithi15186934-49 2024-09-13T14:27:49.540 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 49 v 744 (ObjNum 635 snap 206 seq_num 635) dirty exists 2024-09-13T14:27:49.540 INFO:tasks.rados.rados.0.smithi151.stdout:1665: left oid 49 (ObjNum 635 snap 206 seq_num 635) 2024-09-13T14:27:49.540 INFO:tasks.rados.rados.0.smithi151.stdout:1666: expect (ObjNum 617 snap 201 seq_num 617) 2024-09-13T14:27:49.641 INFO:tasks.rados.rados.0.smithi151.stdout:1667: finishing write tid 1 to smithi15186934-50 2024-09-13T14:27:49.641 INFO:tasks.rados.rados.0.smithi151.stdout:1667: finishing write tid 2 to smithi15186934-50 2024-09-13T14:27:49.641 INFO:tasks.rados.rados.0.smithi151.stdout:1667: finishing write tid 3 to smithi15186934-50 2024-09-13T14:27:49.641 INFO:tasks.rados.rados.0.smithi151.stdout:1667: finishing write tid 4 to smithi15186934-50 2024-09-13T14:27:49.641 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 50 v 556 (ObjNum 636 snap 206 seq_num 636) dirty exists 2024-09-13T14:27:49.641 INFO:tasks.rados.rados.0.smithi151.stdout:1667: left oid 50 (ObjNum 636 snap 206 seq_num 636) 2024-09-13T14:27:49.641 INFO:tasks.rados.rados.0.smithi151.stdout:1665: done (3 left) 2024-09-13T14:27:49.641 INFO:tasks.rados.rados.0.smithi151.stdout:1666: done (2 left) 2024-09-13T14:27:49.641 INFO:tasks.rados.rados.0.smithi151.stdout:1667: done (1 left) 2024-09-13T14:27:49.641 INFO:tasks.rados.rados.0.smithi151.stdout:1668: done (0 left) 2024-09-13T14:27:49.641 INFO:tasks.rados.rados.0.smithi151.stdout:1669: snap_remove snap 202 2024-09-13T14:27:49.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:49 smithi184 ceph-mon[92527]: osdmap e478: 8 total, 8 up, 8 in 2024-09-13T14:27:49.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:49 smithi184 ceph-mon[92527]: osdmap e479: 8 total, 8 up, 8 in 2024-09-13T14:27:49.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:49 smithi151 ceph-mon[108966]: osdmap e478: 8 total, 8 up, 8 in 2024-09-13T14:27:49.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:49 smithi151 ceph-mon[108966]: osdmap e479: 8 total, 8 up, 8 in 2024-09-13T14:27:49.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:49 smithi151 ceph-mon[104450]: osdmap e478: 8 total, 8 up, 8 in 2024-09-13T14:27:49.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:49 smithi151 ceph-mon[104450]: osdmap e479: 8 total, 8 up, 8 in 2024-09-13T14:27:50.570 INFO:tasks.rados.rados.0.smithi151.stdout:1669: done (0 left) 2024-09-13T14:27:50.570 INFO:tasks.rados.rados.0.smithi151.stdout:1670: rollback oid 23 current snap is 206 2024-09-13T14:27:50.570 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 23 to 198 2024-09-13T14:27:50.570 INFO:tasks.rados.rados.0.smithi151.stdout:1671: read oid 11 snap -1 2024-09-13T14:27:50.570 INFO:tasks.rados.rados.0.smithi151.stdout:1671: expect deleted 2024-09-13T14:27:50.570 INFO:tasks.rados.rados.0.smithi151.stdout:1672: delete oid 44 current snap is 206 2024-09-13T14:27:50.574 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:27:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:27:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:27:50.574 INFO:tasks.rados.rados.0.smithi151.stdout:1671: done (2 left) 2024-09-13T14:27:50.574 INFO:tasks.rados.rados.0.smithi151.stdout:1672: done (1 left) 2024-09-13T14:27:50.574 INFO:tasks.rados.rados.0.smithi151.stdout:1673: snap_create 2024-09-13T14:27:50.578 INFO:tasks.rados.rados.0.smithi151.stdout:1670: finishing rollback tid 0 to smithi15186934-23 2024-09-13T14:27:50.578 INFO:tasks.rados.rados.0.smithi151.stdout:1670: finishing rollback tid 1 to smithi15186934-23 2024-09-13T14:27:50.579 INFO:tasks.rados.rados.0.smithi151.stdout:1670: finishing rollback tid 2 to smithi15186934-23 2024-09-13T14:27:50.579 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 23 v 748 (ObjNum 546 snap 175 seq_num 546) dirty exists 2024-09-13T14:27:50.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:50 smithi184 ceph-mon[92527]: pgmap v336: 89 pgs: 89 active+clean; 115 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 1.5 MiB/s wr, 17 op/s 2024-09-13T14:27:50.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:50 smithi151 ceph-mon[104450]: pgmap v336: 89 pgs: 89 active+clean; 115 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 1.5 MiB/s wr, 17 op/s 2024-09-13T14:27:50.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:50 smithi151 ceph-mon[108966]: pgmap v336: 89 pgs: 89 active+clean; 115 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 1.5 MiB/s wr, 17 op/s 2024-09-13T14:27:51.585 INFO:tasks.rados.rados.0.smithi151.stdout:1670: done (1 left) 2024-09-13T14:27:51.585 INFO:tasks.rados.rados.0.smithi151.stdout:1673: done (0 left) 2024-09-13T14:27:51.585 INFO:tasks.rados.rados.0.smithi151.stdout:1674: snap_create 2024-09-13T14:27:51.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:51 smithi184 ceph-mon[92527]: osdmap e480: 8 total, 8 up, 8 in 2024-09-13T14:27:51.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:51 smithi151 ceph-mon[104450]: osdmap e480: 8 total, 8 up, 8 in 2024-09-13T14:27:51.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:51 smithi151 ceph-mon[108966]: osdmap e480: 8 total, 8 up, 8 in 2024-09-13T14:27:52.588 INFO:tasks.rados.rados.0.smithi151.stdout:1674: done (0 left) 2024-09-13T14:27:52.588 INFO:tasks.rados.rados.0.smithi151.stdout:1675: snap_remove snap 206 2024-09-13T14:27:52.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:52 smithi184 ceph-mon[92527]: pgmap v339: 89 pgs: 89 active+clean; 115 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:27:52.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:52 smithi184 ceph-mon[92527]: osdmap e481: 8 total, 8 up, 8 in 2024-09-13T14:27:52.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:52 smithi151 ceph-mon[104450]: pgmap v339: 89 pgs: 89 active+clean; 115 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:27:52.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:52 smithi151 ceph-mon[104450]: osdmap e481: 8 total, 8 up, 8 in 2024-09-13T14:27:52.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:52 smithi151 ceph-mon[108966]: pgmap v339: 89 pgs: 89 active+clean; 115 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:27:52.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:52 smithi151 ceph-mon[108966]: osdmap e481: 8 total, 8 up, 8 in 2024-09-13T14:27:53.590 INFO:tasks.rados.rados.0.smithi151.stdout:1675: done (0 left) 2024-09-13T14:27:53.590 INFO:tasks.rados.rados.0.smithi151.stdout:1676: snap_remove snap 198 2024-09-13T14:27:53.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:53 smithi184 ceph-mon[92527]: osdmap e482: 8 total, 8 up, 8 in 2024-09-13T14:27:53.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:53 smithi184 ceph-mon[92527]: osdmap e483: 8 total, 8 up, 8 in 2024-09-13T14:27:53.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:53 smithi151 ceph-mon[104450]: osdmap e482: 8 total, 8 up, 8 in 2024-09-13T14:27:53.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:53 smithi151 ceph-mon[104450]: osdmap e483: 8 total, 8 up, 8 in 2024-09-13T14:27:53.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:53 smithi151 ceph-mon[108966]: osdmap e482: 8 total, 8 up, 8 in 2024-09-13T14:27:53.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:53 smithi151 ceph-mon[108966]: osdmap e483: 8 total, 8 up, 8 in 2024-09-13T14:27:54.499 INFO:tasks.rados.rados.0.smithi151.stdout:1676: done (0 left) 2024-09-13T14:27:54.499 INFO:tasks.rados.rados.0.smithi151.stdout:1677: write oid 25 current snap is 208 2024-09-13T14:27:54.499 INFO:tasks.rados.rados.0.smithi151.stdout:1677: seq_num 639 ranges {739673=657763,2076229=663444} 2024-09-13T14:27:54.508 INFO:tasks.rados.rados.0.smithi151.stdout:1677: writing smithi15186934-25 from 739673 to 1397436 tid 1 2024-09-13T14:27:54.515 INFO:tasks.rados.rados.0.smithi151.stdout:1677: writing smithi15186934-25 from 2076229 to 2739673 tid 2 2024-09-13T14:27:54.516 INFO:tasks.rados.rados.0.smithi151.stdout:1678: write oid 31 current snap is 208 2024-09-13T14:27:54.516 INFO:tasks.rados.rados.0.smithi151.stdout:1677: finishing write tid 1 to smithi15186934-25 2024-09-13T14:27:54.516 INFO:tasks.rados.rados.0.smithi151.stdout:1678: seq_num 640 ranges {756480=733012,2134711=432549} 2024-09-13T14:27:54.523 INFO:tasks.rados.rados.0.smithi151.stdout:1678: writing smithi15186934-31 from 756480 to 1489492 tid 1 2024-09-13T14:27:54.530 INFO:tasks.rados.rados.0.smithi151.stdout:1678: writing smithi15186934-31 from 2134711 to 2567260 tid 2 2024-09-13T14:27:54.530 INFO:tasks.rados.rados.0.smithi151.stdout:1679: snap_create 2024-09-13T14:27:54.530 INFO:tasks.rados.rados.0.smithi151.stdout:1677: finishing write tid 2 to smithi15186934-25 2024-09-13T14:27:54.530 INFO:tasks.rados.rados.0.smithi151.stdout:1677: finishing write tid 3 to smithi15186934-25 2024-09-13T14:27:54.530 INFO:tasks.rados.rados.0.smithi151.stdout:1677: finishing write tid 4 to smithi15186934-25 2024-09-13T14:27:54.530 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 949 (ObjNum 639 snap 208 seq_num 639) dirty exists 2024-09-13T14:27:54.530 INFO:tasks.rados.rados.0.smithi151.stdout:1677: left oid 25 (ObjNum 639 snap 208 seq_num 639) 2024-09-13T14:27:54.530 INFO:tasks.rados.rados.0.smithi151.stdout:1678: finishing write tid 1 to smithi15186934-31 2024-09-13T14:27:54.532 INFO:tasks.rados.rados.0.smithi151.stdout:1678: finishing write tid 2 to smithi15186934-31 2024-09-13T14:27:54.532 INFO:tasks.rados.rados.0.smithi151.stdout:1678: finishing write tid 3 to smithi15186934-31 2024-09-13T14:27:54.532 INFO:tasks.rados.rados.0.smithi151.stdout:1678: finishing write tid 4 to smithi15186934-31 2024-09-13T14:27:54.533 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 31 v 848 (ObjNum 640 snap 208 seq_num 640) dirty exists 2024-09-13T14:27:54.533 INFO:tasks.rados.rados.0.smithi151.stdout:1678: left oid 31 (ObjNum 640 snap 208 seq_num 640) 2024-09-13T14:27:54.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:54 smithi184 ceph-mon[92527]: pgmap v342: 89 pgs: 89 active+clean; 125 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.6 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:27:54.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:54 smithi184 ceph-mon[92527]: osdmap e484: 8 total, 8 up, 8 in 2024-09-13T14:27:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:54 smithi151 ceph-mon[104450]: pgmap v342: 89 pgs: 89 active+clean; 125 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.6 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:27:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:54 smithi151 ceph-mon[104450]: osdmap e484: 8 total, 8 up, 8 in 2024-09-13T14:27:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:54 smithi151 ceph-mon[108966]: pgmap v342: 89 pgs: 89 active+clean; 125 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.6 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:27:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:54 smithi151 ceph-mon[108966]: osdmap e484: 8 total, 8 up, 8 in 2024-09-13T14:27:54.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:27:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:27:54] "GET /metrics HTTP/1.1" 200 36401 "" "Prometheus/2.51.0" 2024-09-13T14:27:55.498 INFO:tasks.rados.rados.0.smithi151.stdout:1677: done (2 left) 2024-09-13T14:27:55.498 INFO:tasks.rados.rados.0.smithi151.stdout:1678: done (1 left) 2024-09-13T14:27:55.498 INFO:tasks.rados.rados.0.smithi151.stdout:1679: done (0 left) 2024-09-13T14:27:55.498 INFO:tasks.rados.rados.0.smithi151.stdout:1680: snap_remove snap 205 2024-09-13T14:27:56.502 INFO:tasks.rados.rados.0.smithi151.stdout:1680: done (0 left) 2024-09-13T14:27:56.502 INFO:tasks.rados.rados.0.smithi151.stdout:1681: delete oid 40 current snap is 209 2024-09-13T14:27:56.513 INFO:tasks.rados.rados.0.smithi151.stdout:1681: done (0 left) 2024-09-13T14:27:56.513 INFO:tasks.rados.rados.0.smithi151.stdout:1682: read oid 22 snap -1 2024-09-13T14:27:56.513 INFO:tasks.rados.rados.0.smithi151.stdout:1682: expect deleted 2024-09-13T14:27:56.513 INFO:tasks.rados.rados.0.smithi151.stdout:1683: write oid 35 current snap is 209 2024-09-13T14:27:56.513 INFO:tasks.rados.rados.0.smithi151.stdout:1683: seq_num 642 ranges {790094=483510,1851675=436218} 2024-09-13T14:27:56.518 INFO:tasks.rados.rados.0.smithi151.stdout:1683: writing smithi15186934-35 from 790094 to 1273604 tid 1 2024-09-13T14:27:56.524 INFO:tasks.rados.rados.0.smithi151.stdout:1683: writing smithi15186934-35 from 1851675 to 2287893 tid 2 2024-09-13T14:27:56.524 INFO:tasks.rados.rados.0.smithi151.stdout:1684: delete oid 20 current snap is 209 2024-09-13T14:27:56.524 INFO:tasks.rados.rados.0.smithi151.stdout:1683: finishing write tid 1 to smithi15186934-35 2024-09-13T14:27:56.526 INFO:tasks.rados.rados.0.smithi151.stdout:1682: done (2 left) 2024-09-13T14:27:56.527 INFO:tasks.rados.rados.0.smithi151.stdout:1684: done (1 left) 2024-09-13T14:27:56.527 INFO:tasks.rados.rados.0.smithi151.stdout:1685: write oid 18 current snap is 209 2024-09-13T14:27:56.527 INFO:tasks.rados.rados.0.smithi151.stdout:1685: seq_num 644 ranges {423708=634008,1852286=439887,2752172=71536} 2024-09-13T14:27:56.532 INFO:tasks.rados.rados.0.smithi151.stdout:1685: writing smithi15186934-18 from 423708 to 1057716 tid 1 2024-09-13T14:27:56.540 INFO:tasks.rados.rados.0.smithi151.stdout:1685: writing smithi15186934-18 from 1852286 to 2292173 tid 2 2024-09-13T14:27:56.543 INFO:tasks.rados.rados.0.smithi151.stdout:1685: writing smithi15186934-18 from 2752172 to 2823708 tid 3 2024-09-13T14:27:56.543 INFO:tasks.rados.rados.0.smithi151.stdout:1686: read oid 4 snap -1 2024-09-13T14:27:56.543 INFO:tasks.rados.rados.0.smithi151.stdout:1686: expect (ObjNum 592 snap 191 seq_num 592) 2024-09-13T14:27:56.543 INFO:tasks.rados.rados.0.smithi151.stdout:1687: write oid 39 current snap is 209 2024-09-13T14:27:56.543 INFO:tasks.rados.rados.0.smithi151.stdout:1683: finishing write tid 2 to smithi15186934-35 2024-09-13T14:27:56.543 INFO:tasks.rados.rados.0.smithi151.stdout:1683: finishing write tid 3 to smithi15186934-35 2024-09-13T14:27:56.543 INFO:tasks.rados.rados.0.smithi151.stdout:1687: seq_num 645 ranges {440515=709257,1910768=583545} 2024-09-13T14:27:56.549 INFO:tasks.rados.rados.0.smithi151.stdout:1687: writing smithi15186934-39 from 440515 to 1149772 tid 1 2024-09-13T14:27:56.558 INFO:tasks.rados.rados.0.smithi151.stdout:1687: writing smithi15186934-39 from 1910768 to 2494313 tid 2 2024-09-13T14:27:56.558 INFO:tasks.rados.rados.0.smithi151.stdout:1688: read oid 26 snap -1 2024-09-13T14:27:56.558 INFO:tasks.rados.rados.0.smithi151.stdout:1688: expect (ObjNum 598 snap 192 seq_num 598) 2024-09-13T14:27:56.558 INFO:tasks.rados.rados.0.smithi151.stdout:1689: delete oid 38 current snap is 209 2024-09-13T14:27:56.558 INFO:tasks.rados.rados.0.smithi151.stdout:1683: finishing write tid 4 to smithi15186934-35 2024-09-13T14:27:56.558 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 35 v 1147 (ObjNum 642 snap 209 seq_num 642) dirty exists 2024-09-13T14:27:56.558 INFO:tasks.rados.rados.0.smithi151.stdout:1683: left oid 35 (ObjNum 642 snap 209 seq_num 642) 2024-09-13T14:27:56.558 INFO:tasks.rados.rados.0.smithi151.stdout:1685: finishing write tid 1 to smithi15186934-18 2024-09-13T14:27:56.558 INFO:tasks.rados.rados.0.smithi151.stdout:1685: finishing write tid 2 to smithi15186934-18 2024-09-13T14:27:56.558 INFO:tasks.rados.rados.0.smithi151.stdout:1685: finishing write tid 3 to smithi15186934-18 2024-09-13T14:27:56.558 INFO:tasks.rados.rados.0.smithi151.stdout:1685: finishing write tid 4 to smithi15186934-18 2024-09-13T14:27:56.558 INFO:tasks.rados.rados.0.smithi151.stdout:1685: finishing write tid 5 to smithi15186934-18 2024-09-13T14:27:56.559 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 18 v 481 (ObjNum 644 snap 209 seq_num 644) dirty exists 2024-09-13T14:27:56.559 INFO:tasks.rados.rados.0.smithi151.stdout:1685: left oid 18 (ObjNum 644 snap 209 seq_num 644) 2024-09-13T14:27:56.559 INFO:tasks.rados.rados.0.smithi151.stdout:1686: expect (ObjNum 592 snap 191 seq_num 592) 2024-09-13T14:27:56.687 INFO:tasks.rados.rados.0.smithi151.stdout:1687: finishing write tid 1 to smithi15186934-39 2024-09-13T14:27:56.687 INFO:tasks.rados.rados.0.smithi151.stdout:1687: finishing write tid 2 to smithi15186934-39 2024-09-13T14:27:56.687 INFO:tasks.rados.rados.0.smithi151.stdout:1687: finishing write tid 3 to smithi15186934-39 2024-09-13T14:27:56.687 INFO:tasks.rados.rados.0.smithi151.stdout:1687: finishing write tid 4 to smithi15186934-39 2024-09-13T14:27:56.687 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 568 (ObjNum 645 snap 209 seq_num 645) dirty exists 2024-09-13T14:27:56.687 INFO:tasks.rados.rados.0.smithi151.stdout:1687: left oid 39 (ObjNum 645 snap 209 seq_num 645) 2024-09-13T14:27:56.687 INFO:tasks.rados.rados.0.smithi151.stdout:1688: expect (ObjNum 598 snap 192 seq_num 598) 2024-09-13T14:27:56.751 INFO:tasks.rados.rados.0.smithi151.stdout:1683: done (5 left) 2024-09-13T14:27:56.751 INFO:tasks.rados.rados.0.smithi151.stdout:1685: done (4 left) 2024-09-13T14:27:56.751 INFO:tasks.rados.rados.0.smithi151.stdout:1686: done (3 left) 2024-09-13T14:27:56.752 INFO:tasks.rados.rados.0.smithi151.stdout:1687: done (2 left) 2024-09-13T14:27:56.752 INFO:tasks.rados.rados.0.smithi151.stdout:1688: done (1 left) 2024-09-13T14:27:56.752 INFO:tasks.rados.rados.0.smithi151.stdout:1689: done (0 left) 2024-09-13T14:27:56.752 INFO:tasks.rados.rados.0.smithi151.stdout:1690: write oid 39 current snap is 209 2024-09-13T14:27:56.752 INFO:tasks.rados.rados.0.smithi151.stdout:1690: seq_num 647 ranges {474129=576108,1627732=587214,2718088=156041} 2024-09-13T14:27:56.757 INFO:tasks.rados.rados.0.smithi151.stdout:1690: writing smithi15186934-39 from 474129 to 1050237 tid 1 2024-09-13T14:27:56.763 INFO:tasks.rados.rados.0.smithi151.stdout:1690: writing smithi15186934-39 from 1627732 to 2214946 tid 2 2024-09-13T14:27:56.767 INFO:tasks.rados.rados.0.smithi151.stdout:1690: writing smithi15186934-39 from 2718088 to 2874129 tid 3 2024-09-13T14:27:56.767 INFO:tasks.rados.rados.0.smithi151.stdout:1691: delete oid 29 current snap is 209 2024-09-13T14:27:56.767 INFO:tasks.rados.rados.0.smithi151.stdout:1690: finishing write tid 1 to smithi15186934-39 2024-09-13T14:27:56.767 INFO:tasks.rados.rados.0.smithi151.stdout:1690: finishing write tid 2 to smithi15186934-39 2024-09-13T14:27:56.769 INFO:tasks.rados.rados.0.smithi151.stdout:1691: done (1 left) 2024-09-13T14:27:56.769 INFO:tasks.rados.rados.0.smithi151.stdout:1692: read oid 18 snap -1 2024-09-13T14:27:56.769 INFO:tasks.rados.rados.0.smithi151.stdout:1692: expect (ObjNum 644 snap 209 seq_num 644) 2024-09-13T14:27:56.770 INFO:tasks.rados.rados.0.smithi151.stdout:1693: snap_create 2024-09-13T14:27:56.770 INFO:tasks.rados.rados.0.smithi151.stdout:1690: finishing write tid 3 to smithi15186934-39 2024-09-13T14:27:56.770 INFO:tasks.rados.rados.0.smithi151.stdout:1690: finishing write tid 4 to smithi15186934-39 2024-09-13T14:27:56.770 INFO:tasks.rados.rados.0.smithi151.stdout:1690: finishing write tid 5 to smithi15186934-39 2024-09-13T14:27:56.770 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 574 (ObjNum 647 snap 209 seq_num 647) dirty exists 2024-09-13T14:27:56.770 INFO:tasks.rados.rados.0.smithi151.stdout:1690: left oid 39 (ObjNum 647 snap 209 seq_num 647) 2024-09-13T14:27:56.773 INFO:tasks.rados.rados.0.smithi151.stdout:1692: expect (ObjNum 644 snap 209 seq_num 644) 2024-09-13T14:27:56.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:56 smithi184 ceph-mon[92527]: pgmap v345: 89 pgs: 89 active+clean; 125 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.6 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:27:56.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:56 smithi184 ceph-mon[92527]: osdmap e485: 8 total, 8 up, 8 in 2024-09-13T14:27:56.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:56 smithi151 ceph-mon[104450]: pgmap v345: 89 pgs: 89 active+clean; 125 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.6 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:27:56.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:56 smithi151 ceph-mon[104450]: osdmap e485: 8 total, 8 up, 8 in 2024-09-13T14:27:56.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:56 smithi151 ceph-mon[108966]: pgmap v345: 89 pgs: 89 active+clean; 125 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.6 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:27:56.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:56 smithi151 ceph-mon[108966]: osdmap e485: 8 total, 8 up, 8 in 2024-09-13T14:27:57.518 INFO:tasks.rados.rados.0.smithi151.stdout:1690: done (2 left) 2024-09-13T14:27:57.518 INFO:tasks.rados.rados.0.smithi151.stdout:1692: done (1 left) 2024-09-13T14:27:57.518 INFO:tasks.rados.rados.0.smithi151.stdout:1693: done (0 left) 2024-09-13T14:27:57.518 INFO:tasks.rados.rados.0.smithi151.stdout:1694: read oid 48 snap 209 2024-09-13T14:27:57.518 INFO:tasks.rados.rados.0.smithi151.stdout:1694: expect deleted 2024-09-13T14:27:57.518 INFO:tasks.rados.rados.0.smithi151.stdout:1695: rollback oid 3 current snap is 210 2024-09-13T14:27:57.518 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 3 to 209 2024-09-13T14:27:57.518 INFO:tasks.rados.rados.0.smithi151.stdout:1696: read oid 15 snap -1 2024-09-13T14:27:57.518 INFO:tasks.rados.rados.0.smithi151.stdout:1696: expect deleted 2024-09-13T14:27:57.518 INFO:tasks.rados.rados.0.smithi151.stdout:1697: delete oid 42 current snap is 210 2024-09-13T14:27:57.522 INFO:tasks.rados.rados.0.smithi151.stdout:1694: done (3 left) 2024-09-13T14:27:57.522 INFO:tasks.rados.rados.0.smithi151.stdout:1696: done (2 left) 2024-09-13T14:27:57.522 INFO:tasks.rados.rados.0.smithi151.stdout:1697: done (1 left) 2024-09-13T14:27:57.522 INFO:tasks.rados.rados.0.smithi151.stdout:1698: snap_remove snap 208 2024-09-13T14:27:57.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:57 smithi184 ceph-mon[92527]: osdmap e486: 8 total, 8 up, 8 in 2024-09-13T14:27:57.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:57 smithi151 ceph-mon[104450]: osdmap e486: 8 total, 8 up, 8 in 2024-09-13T14:27:57.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:57 smithi151 ceph-mon[108966]: osdmap e486: 8 total, 8 up, 8 in 2024-09-13T14:27:58.520 INFO:tasks.rados.rados.0.smithi151.stdout:1698: done (1 left) 2024-09-13T14:27:58.521 INFO:tasks.rados.rados.0.smithi151.stdout:1699: read oid 21 snap 209 2024-09-13T14:27:58.521 INFO:tasks.rados.rados.0.smithi151.stdout:1699: expect (ObjNum 578 snap 186 seq_num 578) 2024-09-13T14:27:58.521 INFO:tasks.rados.rados.0.smithi151.stdout:1700: read oid 8 snap -1 2024-09-13T14:27:58.521 INFO:tasks.rados.rados.0.smithi151.stdout:1700: expect (ObjNum 602 snap 193 seq_num 602) 2024-09-13T14:27:58.521 INFO:tasks.rados.rados.0.smithi151.stdout:1701: snap_remove snap 207 2024-09-13T14:27:58.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:58 smithi184 ceph-mon[92527]: pgmap v348: 89 pgs: 89 active+clean; 104 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.9 MiB/s rd, 1.7 MiB/s wr, 15 op/s 2024-09-13T14:27:58.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:58 smithi184 ceph-mon[92527]: osdmap e487: 8 total, 8 up, 8 in 2024-09-13T14:27:58.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:58 smithi151 ceph-mon[104450]: pgmap v348: 89 pgs: 89 active+clean; 104 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.9 MiB/s rd, 1.7 MiB/s wr, 15 op/s 2024-09-13T14:27:58.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:58 smithi151 ceph-mon[104450]: osdmap e487: 8 total, 8 up, 8 in 2024-09-13T14:27:58.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:58 smithi151 ceph-mon[108966]: pgmap v348: 89 pgs: 89 active+clean; 104 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.9 MiB/s rd, 1.7 MiB/s wr, 15 op/s 2024-09-13T14:27:58.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:58 smithi151 ceph-mon[108966]: osdmap e487: 8 total, 8 up, 8 in 2024-09-13T14:27:59.497 INFO:tasks.rados.rados.0.smithi151.stdout:1701: done (3 left) 2024-09-13T14:27:59.497 INFO:tasks.rados.rados.0.smithi151.stdout:1702: snap_create 2024-09-13T14:27:59.498 INFO:tasks.rados.rados.0.smithi151.stdout:1695: finishing rollback tid 0 to smithi15186934-3 2024-09-13T14:27:59.498 INFO:tasks.rados.rados.0.smithi151.stdout:1695: finishing rollback tid 1 to smithi15186934-3 2024-09-13T14:27:59.498 INFO:tasks.rados.rados.0.smithi151.stdout:1695: finishing rollback tid 2 to smithi15186934-3 2024-09-13T14:27:59.498 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 614 (ObjNum 334 snap 103 seq_num 334) dirty exists 2024-09-13T14:27:59.498 INFO:tasks.rados.rados.0.smithi151.stdout:1700: expect (ObjNum 602 snap 193 seq_num 602) 2024-09-13T14:27:59.604 INFO:tasks.rados.rados.0.smithi151.stdout:1699: expect (ObjNum 578 snap 186 seq_num 578) 2024-09-13T14:27:59.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:59 smithi184 ceph-mon[92527]: osdmap e488: 8 total, 8 up, 8 in 2024-09-13T14:27:59.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:27:59 smithi184 ceph-mon[92527]: osdmap e489: 8 total, 8 up, 8 in 2024-09-13T14:27:59.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:59 smithi151 ceph-mon[104450]: osdmap e488: 8 total, 8 up, 8 in 2024-09-13T14:27:59.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:27:59 smithi151 ceph-mon[104450]: osdmap e489: 8 total, 8 up, 8 in 2024-09-13T14:27:59.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:59 smithi151 ceph-mon[108966]: osdmap e488: 8 total, 8 up, 8 in 2024-09-13T14:27:59.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:27:59 smithi151 ceph-mon[108966]: osdmap e489: 8 total, 8 up, 8 in 2024-09-13T14:28:00.502 INFO:tasks.rados.rados.0.smithi151.stdout:1695: done (3 left) 2024-09-13T14:28:00.502 INFO:tasks.rados.rados.0.smithi151.stdout:1699: done (2 left) 2024-09-13T14:28:00.502 INFO:tasks.rados.rados.0.smithi151.stdout:1700: done (1 left) 2024-09-13T14:28:00.502 INFO:tasks.rados.rados.0.smithi151.stdout:1702: done (0 left) 2024-09-13T14:28:00.502 INFO:tasks.rados.rados.0.smithi151.stdout:1703: write oid 47 current snap is 211 2024-09-13T14:28:00.502 INFO:tasks.rados.rados.0.smithi151.stdout:1703: seq_num 650 ranges {524550=401855,1403178=450894,2516710=407840} 2024-09-13T14:28:00.506 INFO:tasks.rados.rados.0.smithi151.stdout:1703: writing smithi15186934-47 from 524550 to 926405 tid 1 2024-09-13T14:28:00.512 INFO:tasks.rados.rados.0.smithi151.stdout:1703: writing smithi15186934-47 from 1403178 to 1854072 tid 2 2024-09-13T14:28:00.517 INFO:tasks.rados.rados.0.smithi151.stdout:1703: writing smithi15186934-47 from 2516710 to 2924550 tid 3 2024-09-13T14:28:00.518 INFO:tasks.rados.rados.0.smithi151.stdout:1704: snap_create 2024-09-13T14:28:00.518 INFO:tasks.rados.rados.0.smithi151.stdout:1703: finishing write tid 1 to smithi15186934-47 2024-09-13T14:28:00.518 INFO:tasks.rados.rados.0.smithi151.stdout:1703: finishing write tid 2 to smithi15186934-47 2024-09-13T14:28:00.520 INFO:tasks.rados.rados.0.smithi151.stdout:1703: finishing write tid 3 to smithi15186934-47 2024-09-13T14:28:00.520 INFO:tasks.rados.rados.0.smithi151.stdout:1703: finishing write tid 4 to smithi15186934-47 2024-09-13T14:28:00.520 INFO:tasks.rados.rados.0.smithi151.stdout:1703: finishing write tid 5 to smithi15186934-47 2024-09-13T14:28:00.520 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 767 (ObjNum 650 snap 211 seq_num 650) dirty exists 2024-09-13T14:28:00.520 INFO:tasks.rados.rados.0.smithi151.stdout:1703: left oid 47 (ObjNum 650 snap 211 seq_num 650) 2024-09-13T14:28:00.524 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:28:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:28:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:00.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:00 smithi184 ceph-mon[92527]: pgmap v351: 89 pgs: 89 active+clean; 104 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.9 MiB/s rd, 1.7 MiB/s wr, 15 op/s 2024-09-13T14:28:00.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:00 smithi184 ceph-mon[92527]: osdmap e490: 8 total, 8 up, 8 in 2024-09-13T14:28:00.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:00 smithi151 ceph-mon[104450]: pgmap v351: 89 pgs: 89 active+clean; 104 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.9 MiB/s rd, 1.7 MiB/s wr, 15 op/s 2024-09-13T14:28:00.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:00 smithi151 ceph-mon[104450]: osdmap e490: 8 total, 8 up, 8 in 2024-09-13T14:28:00.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:00 smithi151 ceph-mon[108966]: pgmap v351: 89 pgs: 89 active+clean; 104 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.9 MiB/s rd, 1.7 MiB/s wr, 15 op/s 2024-09-13T14:28:00.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:00 smithi151 ceph-mon[108966]: osdmap e490: 8 total, 8 up, 8 in 2024-09-13T14:28:01.501 INFO:tasks.rados.rados.0.smithi151.stdout:1703: done (1 left) 2024-09-13T14:28:01.501 INFO:tasks.rados.rados.0.smithi151.stdout:1704: done (0 left) 2024-09-13T14:28:01.502 INFO:tasks.rados.rados.0.smithi151.stdout:1705: write oid 37 current snap is 212 2024-09-13T14:28:01.502 INFO:tasks.rados.rados.0.smithi151.stdout:1705: seq_num 651 ranges {541357=477104,1745307=594552} 2024-09-13T14:28:01.506 INFO:tasks.rados.rados.0.smithi151.stdout:1705: writing smithi15186934-37 from 541357 to 1018461 tid 1 2024-09-13T14:28:01.515 INFO:tasks.rados.rados.0.smithi151.stdout:1705: writing smithi15186934-37 from 1745307 to 2339859 tid 2 2024-09-13T14:28:01.515 INFO:tasks.rados.rados.0.smithi151.stdout:1706: snap_remove snap 209 2024-09-13T14:28:01.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:01 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:01.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:01 smithi184 ceph-mon[92527]: osdmap e491: 8 total, 8 up, 8 in 2024-09-13T14:28:01.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:01.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:01 smithi151 ceph-mon[104450]: osdmap e491: 8 total, 8 up, 8 in 2024-09-13T14:28:01.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:01 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:01.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:01 smithi151 ceph-mon[108966]: osdmap e491: 8 total, 8 up, 8 in 2024-09-13T14:28:02.503 INFO:tasks.rados.rados.0.smithi151.stdout:1706: done (1 left) 2024-09-13T14:28:02.504 INFO:tasks.rados.rados.0.smithi151.stdout:1707: write oid 38 current snap is 212 2024-09-13T14:28:02.504 INFO:tasks.rados.rados.0.smithi151.stdout:1705: finishing write tid 1 to smithi15186934-37 2024-09-13T14:28:02.504 INFO:tasks.rados.rados.0.smithi151.stdout:1707: seq_num 652 ranges {558164=552353,1803789=454563,2855203=102961} 2024-09-13T14:28:02.510 INFO:tasks.rados.rados.0.smithi151.stdout:1707: writing smithi15186934-38 from 558164 to 1110517 tid 1 2024-09-13T14:28:02.517 INFO:tasks.rados.rados.0.smithi151.stdout:1707: writing smithi15186934-38 from 1803789 to 2258352 tid 2 2024-09-13T14:28:02.521 INFO:tasks.rados.rados.0.smithi151.stdout:1707: writing smithi15186934-38 from 2855203 to 2958164 tid 3 2024-09-13T14:28:02.521 INFO:tasks.rados.rados.0.smithi151.stdout:1708: read oid 9 snap -1 2024-09-13T14:28:02.521 INFO:tasks.rados.rados.0.smithi151.stdout:1708: expect deleted 2024-09-13T14:28:02.521 INFO:tasks.rados.rados.0.smithi151.stdout:1709: read oid 4 snap -1 2024-09-13T14:28:02.521 INFO:tasks.rados.rados.0.smithi151.stdout:1709: expect (ObjNum 592 snap 191 seq_num 592) 2024-09-13T14:28:02.522 INFO:tasks.rados.rados.0.smithi151.stdout:1710: delete oid 47 current snap is 212 2024-09-13T14:28:02.522 INFO:tasks.rados.rados.0.smithi151.stdout:1705: finishing write tid 2 to smithi15186934-37 2024-09-13T14:28:02.522 INFO:tasks.rados.rados.0.smithi151.stdout:1705: finishing write tid 3 to smithi15186934-37 2024-09-13T14:28:02.522 INFO:tasks.rados.rados.0.smithi151.stdout:1705: finishing write tid 4 to smithi15186934-37 2024-09-13T14:28:02.522 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 37 v 771 (ObjNum 651 snap 212 seq_num 651) dirty exists 2024-09-13T14:28:02.522 INFO:tasks.rados.rados.0.smithi151.stdout:1705: left oid 37 (ObjNum 651 snap 212 seq_num 651) 2024-09-13T14:28:02.522 INFO:tasks.rados.rados.0.smithi151.stdout:1707: finishing write tid 1 to smithi15186934-38 2024-09-13T14:28:02.522 INFO:tasks.rados.rados.0.smithi151.stdout:1707: finishing write tid 2 to smithi15186934-38 2024-09-13T14:28:02.523 INFO:tasks.rados.rados.0.smithi151.stdout:1707: finishing write tid 3 to smithi15186934-38 2024-09-13T14:28:02.523 INFO:tasks.rados.rados.0.smithi151.stdout:1705: done (4 left) 2024-09-13T14:28:02.523 INFO:tasks.rados.rados.0.smithi151.stdout:1708: done (3 left) 2024-09-13T14:28:02.523 INFO:tasks.rados.rados.0.smithi151.stdout:1710: done (2 left) 2024-09-13T14:28:02.523 INFO:tasks.rados.rados.0.smithi151.stdout:1711: write oid 12 current snap is 212 2024-09-13T14:28:02.523 INFO:tasks.rados.rados.0.smithi151.stdout:1711: seq_num 654 ranges {591778=419204,1637106=741879,2910049=81729} 2024-09-13T14:28:02.528 INFO:tasks.rados.rados.0.smithi151.stdout:1711: writing smithi15186934-12 from 591778 to 1010982 tid 1 2024-09-13T14:28:02.536 INFO:tasks.rados.rados.0.smithi151.stdout:1711: writing smithi15186934-12 from 1637106 to 2378985 tid 2 2024-09-13T14:28:02.540 INFO:tasks.rados.rados.0.smithi151.stdout:1711: writing smithi15186934-12 from 2910049 to 2991778 tid 3 2024-09-13T14:28:02.540 INFO:tasks.rados.rados.0.smithi151.stdout:1712: delete oid 45 current snap is 212 2024-09-13T14:28:02.540 INFO:tasks.rados.rados.0.smithi151.stdout:1707: finishing write tid 4 to smithi15186934-38 2024-09-13T14:28:02.540 INFO:tasks.rados.rados.0.smithi151.stdout:1707: finishing write tid 5 to smithi15186934-38 2024-09-13T14:28:02.541 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 586 (ObjNum 652 snap 212 seq_num 652) dirty exists 2024-09-13T14:28:02.541 INFO:tasks.rados.rados.0.smithi151.stdout:1707: left oid 38 (ObjNum 652 snap 212 seq_num 652) 2024-09-13T14:28:02.541 INFO:tasks.rados.rados.0.smithi151.stdout:1711: finishing write tid 1 to smithi15186934-12 2024-09-13T14:28:02.541 INFO:tasks.rados.rados.0.smithi151.stdout:1709: expect (ObjNum 592 snap 191 seq_num 592) 2024-09-13T14:28:02.669 INFO:tasks.rados.rados.0.smithi151.stdout:1711: finishing write tid 2 to smithi15186934-12 2024-09-13T14:28:02.669 INFO:tasks.rados.rados.0.smithi151.stdout:1711: finishing write tid 3 to smithi15186934-12 2024-09-13T14:28:02.670 INFO:tasks.rados.rados.0.smithi151.stdout:1711: finishing write tid 4 to smithi15186934-12 2024-09-13T14:28:02.670 INFO:tasks.rados.rados.0.smithi151.stdout:1711: finishing write tid 5 to smithi15186934-12 2024-09-13T14:28:02.670 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 12 v 384 (ObjNum 654 snap 212 seq_num 654) dirty exists 2024-09-13T14:28:02.670 INFO:tasks.rados.rados.0.smithi151.stdout:1711: left oid 12 (ObjNum 654 snap 212 seq_num 654) 2024-09-13T14:28:02.670 INFO:tasks.rados.rados.0.smithi151.stdout:1707: done (3 left) 2024-09-13T14:28:02.670 INFO:tasks.rados.rados.0.smithi151.stdout:1709: done (2 left) 2024-09-13T14:28:02.670 INFO:tasks.rados.rados.0.smithi151.stdout:1711: done (1 left) 2024-09-13T14:28:02.670 INFO:tasks.rados.rados.0.smithi151.stdout:1712: done (0 left) 2024-09-13T14:28:02.670 INFO:tasks.rados.rados.0.smithi151.stdout:1713: snap_create 2024-09-13T14:28:02.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:02 smithi184 ceph-mon[92527]: pgmap v354: 89 pgs: 89 active+clean; 104 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:28:02.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:02 smithi184 ceph-mon[92527]: osdmap e492: 8 total, 8 up, 8 in 2024-09-13T14:28:02.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:02 smithi151 ceph-mon[104450]: pgmap v354: 89 pgs: 89 active+clean; 104 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:28:02.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:02 smithi151 ceph-mon[104450]: osdmap e492: 8 total, 8 up, 8 in 2024-09-13T14:28:02.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:02 smithi151 ceph-mon[108966]: pgmap v354: 89 pgs: 89 active+clean; 104 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:28:02.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:02 smithi151 ceph-mon[108966]: osdmap e492: 8 total, 8 up, 8 in 2024-09-13T14:28:03.541 INFO:tasks.rados.rados.0.smithi151.stdout:1713: done (0 left) 2024-09-13T14:28:03.541 INFO:tasks.rados.rados.0.smithi151.stdout:1714: snap_remove snap 211 2024-09-13T14:28:04.498 INFO:tasks.rados.rados.0.smithi151.stdout:1714: done (0 left) 2024-09-13T14:28:04.499 INFO:tasks.rados.rados.0.smithi151.stdout:1715: delete oid 29 current snap is 213 2024-09-13T14:28:04.505 INFO:tasks.rados.rados.0.smithi151.stdout:1715: done (0 left) 2024-09-13T14:28:04.505 INFO:tasks.rados.rados.0.smithi151.stdout:1716: rollback oid 42 current snap is 213 2024-09-13T14:28:04.505 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 42 to 210 2024-09-13T14:28:04.505 INFO:tasks.rados.rados.0.smithi151.stdout:1717: write oid 17 current snap is 213 2024-09-13T14:28:04.505 INFO:tasks.rados.rados.0.smithi151.stdout:1717: seq_num 657 ranges {642199=644951,1696199=605559,2992318=49881} 2024-09-13T14:28:04.510 INFO:tasks.rados.rados.0.smithi151.stdout:1717: writing smithi15186934-17 from 642199 to 1287150 tid 1 2024-09-13T14:28:04.516 INFO:tasks.rados.rados.0.smithi151.stdout:1717: writing smithi15186934-17 from 1696199 to 2301758 tid 2 2024-09-13T14:28:04.521 INFO:tasks.rados.rados.0.smithi151.stdout:1717: writing smithi15186934-17 from 2992318 to 3042199 tid 3 2024-09-13T14:28:04.521 INFO:tasks.rados.rados.0.smithi151.stdout:1718: write oid 23 current snap is 213 2024-09-13T14:28:04.521 INFO:tasks.rados.rados.0.smithi151.stdout:1716: finishing rollback tid 1 to smithi15186934-42 2024-09-13T14:28:04.521 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 0 (ObjNum 145 snap 0 seq_num 1755290976) dirty dne 2024-09-13T14:28:04.521 INFO:tasks.rados.rados.0.smithi151.stdout:1718: seq_num 658 ranges {659006=720200,2154681=749217} 2024-09-13T14:28:04.528 INFO:tasks.rados.rados.0.smithi151.stdout:1718: writing smithi15186934-23 from 659006 to 1379206 tid 1 2024-09-13T14:28:04.537 INFO:tasks.rados.rados.0.smithi151.stdout:1718: writing smithi15186934-23 from 2154681 to 2903898 tid 2 2024-09-13T14:28:04.537 INFO:tasks.rados.rados.0.smithi151.stdout:1716: done (2 left) 2024-09-13T14:28:04.537 INFO:tasks.rados.rados.0.smithi151.stdout:1719: write oid 47 current snap is 213 2024-09-13T14:28:04.538 INFO:tasks.rados.rados.0.smithi151.stdout:1719: seq_num 659 ranges {675813=795449,2096810=609228} 2024-09-13T14:28:04.545 INFO:tasks.rados.rados.0.smithi151.stdout:1719: writing smithi15186934-47 from 675813 to 1471262 tid 1 2024-09-13T14:28:04.553 INFO:tasks.rados.rados.0.smithi151.stdout:1719: writing smithi15186934-47 from 2096810 to 2706038 tid 2 2024-09-13T14:28:04.553 INFO:tasks.rados.rados.0.smithi151.stdout:1720: read oid 43 snap -1 2024-09-13T14:28:04.553 INFO:tasks.rados.rados.0.smithi151.stdout:1720: expect (ObjNum 497 snap 159 seq_num 497) 2024-09-13T14:28:04.553 INFO:tasks.rados.rados.0.smithi151.stdout:1721: read oid 12 snap -1 2024-09-13T14:28:04.553 INFO:tasks.rados.rados.0.smithi151.stdout:1721: expect (ObjNum 654 snap 212 seq_num 654) 2024-09-13T14:28:04.553 INFO:tasks.rados.rados.0.smithi151.stdout:1722: write oid 3 current snap is 213 2024-09-13T14:28:04.553 INFO:tasks.rados.rados.0.smithi151.stdout:1717: finishing write tid 1 to smithi15186934-17 2024-09-13T14:28:04.553 INFO:tasks.rados.rados.0.smithi151.stdout:1717: finishing write tid 2 to smithi15186934-17 2024-09-13T14:28:04.554 INFO:tasks.rados.rados.0.smithi151.stdout:1717: finishing write tid 3 to smithi15186934-17 2024-09-13T14:28:04.554 INFO:tasks.rados.rados.0.smithi151.stdout:1717: finishing write tid 4 to smithi15186934-17 2024-09-13T14:28:04.554 INFO:tasks.rados.rados.0.smithi151.stdout:1717: finishing write tid 5 to smithi15186934-17 2024-09-13T14:28:04.554 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 1156 (ObjNum 657 snap 213 seq_num 657) dirty exists 2024-09-13T14:28:04.554 INFO:tasks.rados.rados.0.smithi151.stdout:1717: left oid 17 (ObjNum 657 snap 213 seq_num 657) 2024-09-13T14:28:04.554 INFO:tasks.rados.rados.0.smithi151.stdout:1718: finishing write tid 1 to smithi15186934-23 2024-09-13T14:28:04.554 INFO:tasks.rados.rados.0.smithi151.stdout:1722: seq_num 660 ranges {692620=470698,1755292=752886} 2024-09-13T14:28:04.560 INFO:tasks.rados.rados.0.smithi151.stdout:1722: writing smithi15186934-3 from 692620 to 1163318 tid 1 2024-09-13T14:28:04.568 INFO:tasks.rados.rados.0.smithi151.stdout:1722: writing smithi15186934-3 from 1755292 to 2508178 tid 2 2024-09-13T14:28:04.568 INFO:tasks.rados.rados.0.smithi151.stdout:1717: done (5 left) 2024-09-13T14:28:04.568 INFO:tasks.rados.rados.0.smithi151.stdout:1723: snap_remove snap 212 2024-09-13T14:28:04.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:04 smithi184 ceph-mon[92527]: pgmap v357: 89 pgs: 89 active+clean; 96 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 570 KiB/s wr, 7 op/s 2024-09-13T14:28:04.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:04 smithi184 ceph-mon[92527]: osdmap e493: 8 total, 8 up, 8 in 2024-09-13T14:28:04.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:04 smithi184 ceph-mon[92527]: osdmap e494: 8 total, 8 up, 8 in 2024-09-13T14:28:04.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:04 smithi151 ceph-mon[104450]: pgmap v357: 89 pgs: 89 active+clean; 96 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 570 KiB/s wr, 7 op/s 2024-09-13T14:28:04.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:04 smithi151 ceph-mon[104450]: osdmap e493: 8 total, 8 up, 8 in 2024-09-13T14:28:04.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:04 smithi151 ceph-mon[104450]: osdmap e494: 8 total, 8 up, 8 in 2024-09-13T14:28:04.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:04 smithi151 ceph-mon[108966]: pgmap v357: 89 pgs: 89 active+clean; 96 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 570 KiB/s wr, 7 op/s 2024-09-13T14:28:04.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:04 smithi151 ceph-mon[108966]: osdmap e493: 8 total, 8 up, 8 in 2024-09-13T14:28:04.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:04 smithi151 ceph-mon[108966]: osdmap e494: 8 total, 8 up, 8 in 2024-09-13T14:28:04.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:28:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:28:04] "GET /metrics HTTP/1.1" 200 36401 "" "Prometheus/2.51.0" 2024-09-13T14:28:05.549 INFO:tasks.rados.rados.0.smithi151.stdout:1723: done (5 left) 2024-09-13T14:28:05.549 INFO:tasks.rados.rados.0.smithi151.stdout:1724: snap_remove snap 210 2024-09-13T14:28:06.559 INFO:tasks.rados.rados.0.smithi151.stdout:1724: done (5 left) 2024-09-13T14:28:06.559 INFO:tasks.rados.rados.0.smithi151.stdout:1725: rollback oid 13 current snap is 213 2024-09-13T14:28:06.559 INFO:tasks.rados.rados.0.smithi151.stdout:1725: done (5 left) 2024-09-13T14:28:06.559 INFO:tasks.rados.rados.0.smithi151.stdout:1726: read oid 44 snap -1 2024-09-13T14:28:06.560 INFO:tasks.rados.rados.0.smithi151.stdout:1726: expect deleted 2024-09-13T14:28:06.560 INFO:tasks.rados.rados.0.smithi151.stdout:1718: finishing write tid 2 to smithi15186934-23 2024-09-13T14:28:06.560 INFO:tasks.rados.rados.0.smithi151.stdout:1718: finishing write tid 3 to smithi15186934-23 2024-09-13T14:28:06.560 INFO:tasks.rados.rados.0.smithi151.stdout:1718: finishing write tid 4 to smithi15186934-23 2024-09-13T14:28:06.560 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 23 v 785 (ObjNum 658 snap 213 seq_num 658) dirty exists 2024-09-13T14:28:06.560 INFO:tasks.rados.rados.0.smithi151.stdout:1718: left oid 23 (ObjNum 658 snap 213 seq_num 658) 2024-09-13T14:28:06.561 INFO:tasks.rados.rados.0.smithi151.stdout:1719: finishing write tid 1 to smithi15186934-47 2024-09-13T14:28:06.561 INFO:tasks.rados.rados.0.smithi151.stdout:1719: finishing write tid 2 to smithi15186934-47 2024-09-13T14:28:06.561 INFO:tasks.rados.rados.0.smithi151.stdout:1719: finishing write tid 3 to smithi15186934-47 2024-09-13T14:28:06.561 INFO:tasks.rados.rados.0.smithi151.stdout:1719: finishing write tid 4 to smithi15186934-47 2024-09-13T14:28:06.561 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 788 (ObjNum 659 snap 213 seq_num 659) dirty exists 2024-09-13T14:28:06.562 INFO:tasks.rados.rados.0.smithi151.stdout:1719: left oid 47 (ObjNum 659 snap 213 seq_num 659) 2024-09-13T14:28:06.562 INFO:tasks.rados.rados.0.smithi151.stdout:1721: expect (ObjNum 654 snap 212 seq_num 654) 2024-09-13T14:28:06.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:06 smithi151 ceph-mon[104450]: pgmap v360: 89 pgs: 89 active+clean; 96 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 570 KiB/s wr, 7 op/s 2024-09-13T14:28:06.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:06 smithi151 ceph-mon[104450]: osdmap e495: 8 total, 8 up, 8 in 2024-09-13T14:28:06.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:06 smithi151 ceph-mon[108966]: pgmap v360: 89 pgs: 89 active+clean; 96 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 570 KiB/s wr, 7 op/s 2024-09-13T14:28:06.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:06 smithi151 ceph-mon[108966]: osdmap e495: 8 total, 8 up, 8 in 2024-09-13T14:28:06.658 INFO:tasks.rados.rados.0.smithi151.stdout:1720: expect (ObjNum 497 snap 159 seq_num 497) 2024-09-13T14:28:06.768 INFO:tasks.rados.rados.0.smithi151.stdout:1722: finishing write tid 1 to smithi15186934-3 2024-09-13T14:28:06.769 INFO:tasks.rados.rados.0.smithi151.stdout:1722: finishing write tid 2 to smithi15186934-3 2024-09-13T14:28:06.769 INFO:tasks.rados.rados.0.smithi151.stdout:1722: finishing write tid 3 to smithi15186934-3 2024-09-13T14:28:06.769 INFO:tasks.rados.rados.0.smithi151.stdout:1718: done (5 left) 2024-09-13T14:28:06.769 INFO:tasks.rados.rados.0.smithi151.stdout:1719: done (4 left) 2024-09-13T14:28:06.769 INFO:tasks.rados.rados.0.smithi151.stdout:1720: done (3 left) 2024-09-13T14:28:06.769 INFO:tasks.rados.rados.0.smithi151.stdout:1721: done (2 left) 2024-09-13T14:28:06.770 INFO:tasks.rados.rados.0.smithi151.stdout:1727: delete oid 43 current snap is 213 2024-09-13T14:28:06.770 INFO:tasks.rados.rados.0.smithi151.stdout:1722: finishing write tid 4 to smithi15186934-3 2024-09-13T14:28:06.770 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 628 (ObjNum 660 snap 213 seq_num 660) dirty exists 2024-09-13T14:28:06.770 INFO:tasks.rados.rados.0.smithi151.stdout:1722: left oid 3 (ObjNum 660 snap 213 seq_num 660) 2024-09-13T14:28:06.772 INFO:tasks.rados.rados.0.smithi151.stdout:1722: done (2 left) 2024-09-13T14:28:06.772 INFO:tasks.rados.rados.0.smithi151.stdout:1726: done (1 left) 2024-09-13T14:28:06.772 INFO:tasks.rados.rados.0.smithi151.stdout:1727: done (0 left) 2024-09-13T14:28:06.772 INFO:tasks.rados.rados.0.smithi151.stdout:1728: read oid 19 snap -1 2024-09-13T14:28:06.772 INFO:tasks.rados.rados.0.smithi151.stdout:1728: expect (ObjNum 323 snap 101 seq_num 323) 2024-09-13T14:28:06.772 INFO:tasks.rados.rados.0.smithi151.stdout:1729: read oid 31 snap -1 2024-09-13T14:28:06.773 INFO:tasks.rados.rados.0.smithi151.stdout:1729: expect (ObjNum 640 snap 208 seq_num 640) 2024-09-13T14:28:06.773 INFO:tasks.rados.rados.0.smithi151.stdout:1730: read oid 21 snap -1 2024-09-13T14:28:06.774 INFO:tasks.rados.rados.0.smithi151.stdout:1730: expect (ObjNum 578 snap 186 seq_num 578) 2024-09-13T14:28:06.774 INFO:tasks.rados.rados.0.smithi151.stdout:1731: write oid 25 current snap is 213 2024-09-13T14:28:06.774 INFO:tasks.rados.rados.0.smithi151.stdout:1731: seq_num 662 ranges {726234=737549,1988609=756555} 2024-09-13T14:28:06.778 INFO:tasks.rados.rados.0.smithi151.stdout:1731: writing smithi15186934-25 from 726234 to 1463783 tid 1 2024-09-13T14:28:06.785 INFO:tasks.rados.rados.0.smithi151.stdout:1731: writing smithi15186934-25 from 1988609 to 2745164 tid 2 2024-09-13T14:28:06.786 INFO:tasks.rados.rados.0.smithi151.stdout:1732: read oid 42 snap -1 2024-09-13T14:28:06.786 INFO:tasks.rados.rados.0.smithi151.stdout:1732: expect deleted 2024-09-13T14:28:06.786 INFO:tasks.rados.rados.0.smithi151.stdout:1733: delete oid 33 current snap is 213 2024-09-13T14:28:06.786 INFO:tasks.rados.rados.0.smithi151.stdout:1728: expect (ObjNum 323 snap 101 seq_num 323) 2024-09-13T14:28:06.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:06 smithi184 ceph-mon[92527]: pgmap v360: 89 pgs: 89 active+clean; 96 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 570 KiB/s wr, 7 op/s 2024-09-13T14:28:06.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:06 smithi184 ceph-mon[92527]: osdmap e495: 8 total, 8 up, 8 in 2024-09-13T14:28:06.903 INFO:tasks.rados.rados.0.smithi151.stdout:1729: expect (ObjNum 640 snap 208 seq_num 640) 2024-09-13T14:28:06.995 INFO:tasks.rados.rados.0.smithi151.stdout:1730: expect (ObjNum 578 snap 186 seq_num 578) 2024-09-13T14:28:07.191 INFO:tasks.rados.rados.0.smithi151.stdout:1731: finishing write tid 1 to smithi15186934-25 2024-09-13T14:28:07.191 INFO:tasks.rados.rados.0.smithi151.stdout:1728: done (5 left) 2024-09-13T14:28:07.191 INFO:tasks.rados.rados.0.smithi151.stdout:1729: done (4 left) 2024-09-13T14:28:07.191 INFO:tasks.rados.rados.0.smithi151.stdout:1730: done (3 left) 2024-09-13T14:28:07.192 INFO:tasks.rados.rados.0.smithi151.stdout:1733: done (2 left) 2024-09-13T14:28:07.192 INFO:tasks.rados.rados.0.smithi151.stdout:1734: delete oid 40 current snap is 213 2024-09-13T14:28:07.192 INFO:tasks.rados.rados.0.smithi151.stdout:1731: finishing write tid 2 to smithi15186934-25 2024-09-13T14:28:07.192 INFO:tasks.rados.rados.0.smithi151.stdout:1731: finishing write tid 3 to smithi15186934-25 2024-09-13T14:28:07.192 INFO:tasks.rados.rados.0.smithi151.stdout:1731: finishing write tid 4 to smithi15186934-25 2024-09-13T14:28:07.192 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 953 (ObjNum 662 snap 213 seq_num 662) dirty exists 2024-09-13T14:28:07.192 INFO:tasks.rados.rados.0.smithi151.stdout:1731: left oid 25 (ObjNum 662 snap 213 seq_num 662) 2024-09-13T14:28:07.193 INFO:tasks.rados.rados.0.smithi151.stdout:1731: done (2 left) 2024-09-13T14:28:07.193 INFO:tasks.rados.rados.0.smithi151.stdout:1732: done (1 left) 2024-09-13T14:28:07.193 INFO:tasks.rados.rados.0.smithi151.stdout:1734: done (0 left) 2024-09-13T14:28:07.193 INFO:tasks.rados.rados.0.smithi151.stdout:1735: delete oid 27 current snap is 213 2024-09-13T14:28:07.194 INFO:tasks.rados.rados.0.smithi151.stdout:1735: done (0 left) 2024-09-13T14:28:07.194 INFO:tasks.rados.rados.0.smithi151.stdout:1736: read oid 13 snap -1 2024-09-13T14:28:07.194 INFO:tasks.rados.rados.0.smithi151.stdout:1736: expect (ObjNum 627 snap 204 seq_num 627) 2024-09-13T14:28:07.194 INFO:tasks.rados.rados.0.smithi151.stdout:1737: snap_create 2024-09-13T14:28:07.198 INFO:tasks.rados.rados.0.smithi151.stdout:1736: expect (ObjNum 627 snap 204 seq_num 627) 2024-09-13T14:28:07.201 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch upgrade resume' 2024-09-13T14:28:07.451 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:07.558 INFO:tasks.rados.rados.0.smithi151.stdout:1736: done (1 left) 2024-09-13T14:28:07.559 INFO:tasks.rados.rados.0.smithi151.stdout:1737: done (0 left) 2024-09-13T14:28:07.559 INFO:tasks.rados.rados.0.smithi151.stdout:1738: snap_create 2024-09-13T14:28:07.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:07 smithi184 ceph-mon[92527]: osdmap e496: 8 total, 8 up, 8 in 2024-09-13T14:28:07.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:07 smithi151 ceph-mon[104450]: osdmap e496: 8 total, 8 up, 8 in 2024-09-13T14:28:07.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:07 smithi151 ceph-mon[108966]: osdmap e496: 8 total, 8 up, 8 in 2024-09-13T14:28:08.559 INFO:tasks.rados.rados.0.smithi151.stdout:1738: done (0 left) 2024-09-13T14:28:08.559 INFO:tasks.rados.rados.0.smithi151.stdout:1739: write oid 50 current snap is 215 2024-09-13T14:28:08.560 INFO:tasks.rados.rados.0.smithi151.stdout:1739: seq_num 666 ranges {793462=638545,2106184=763893} 2024-09-13T14:28:08.567 INFO:tasks.rados.rados.0.smithi151.stdout:1739: writing smithi15186934-50 from 793462 to 1432007 tid 1 2024-09-13T14:28:08.577 INFO:tasks.rados.rados.0.smithi151.stdout:1739: writing smithi15186934-50 from 2106184 to 2870077 tid 2 2024-09-13T14:28:08.577 INFO:tasks.rados.rados.0.smithi151.stdout:1740: read oid 40 snap -1 2024-09-13T14:28:08.577 INFO:tasks.rados.rados.0.smithi151.stdout:1740: expect deleted 2024-09-13T14:28:08.577 INFO:tasks.rados.rados.0.smithi151.stdout:1741: snap_create 2024-09-13T14:28:08.577 INFO:tasks.rados.rados.0.smithi151.stdout:1739: finishing write tid 1 to smithi15186934-50 2024-09-13T14:28:08.580 INFO:tasks.rados.rados.0.smithi151.stdout:1739: finishing write tid 2 to smithi15186934-50 2024-09-13T14:28:08.581 INFO:tasks.rados.rados.0.smithi151.stdout:1739: finishing write tid 3 to smithi15186934-50 2024-09-13T14:28:08.581 INFO:tasks.rados.rados.0.smithi151.stdout:1739: finishing write tid 4 to smithi15186934-50 2024-09-13T14:28:08.581 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 50 v 590 (ObjNum 666 snap 215 seq_num 666) dirty exists 2024-09-13T14:28:08.581 INFO:tasks.rados.rados.0.smithi151.stdout:1739: left oid 50 (ObjNum 666 snap 215 seq_num 666) 2024-09-13T14:28:08.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:08 smithi184 ceph-mon[92527]: pgmap v363: 89 pgs: 89 active+clean; 92 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 2.2 MiB/s wr, 22 op/s 2024-09-13T14:28:08.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:08 smithi184 ceph-mon[92527]: osdmap e497: 8 total, 8 up, 8 in 2024-09-13T14:28:08.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:08 smithi151 ceph-mon[104450]: pgmap v363: 89 pgs: 89 active+clean; 92 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 2.2 MiB/s wr, 22 op/s 2024-09-13T14:28:08.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:08 smithi151 ceph-mon[104450]: osdmap e497: 8 total, 8 up, 8 in 2024-09-13T14:28:08.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:08 smithi151 ceph-mon[108966]: pgmap v363: 89 pgs: 89 active+clean; 92 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 2.2 MiB/s wr, 22 op/s 2024-09-13T14:28:08.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:08 smithi151 ceph-mon[108966]: osdmap e497: 8 total, 8 up, 8 in 2024-09-13T14:28:09.350 INFO:teuthology.orchestra.run.smithi151.stdout:Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:28:09.501 INFO:tasks.rados.rados.0.smithi151.stdout:1739: done (2 left) 2024-09-13T14:28:09.501 INFO:tasks.rados.rados.0.smithi151.stdout:1740: done (1 left) 2024-09-13T14:28:09.502 INFO:tasks.rados.rados.0.smithi151.stdout:1741: done (0 left) 2024-09-13T14:28:09.502 INFO:tasks.rados.rados.0.smithi151.stdout:1742: read oid 25 snap -1 2024-09-13T14:28:09.502 INFO:tasks.rados.rados.0.smithi151.stdout:1742: expect (ObjNum 662 snap 213 seq_num 662) 2024-09-13T14:28:09.502 INFO:tasks.rados.rados.0.smithi151.stdout:1743: read oid 1 snap -1 2024-09-13T14:28:09.502 INFO:tasks.rados.rados.0.smithi151.stdout:1743: expect (ObjNum 511 snap 164 seq_num 511) 2024-09-13T14:28:09.502 INFO:tasks.rados.rados.0.smithi151.stdout:1744: write oid 29 current snap is 216 2024-09-13T14:28:09.502 INFO:tasks.rados.rados.0.smithi151.stdout:1744: seq_num 667 ranges {410269=713794,1648313=507551,2917489=292780} 2024-09-13T14:28:09.507 INFO:tasks.rados.rados.0.smithi151.stdout:1744: writing smithi15186934-29 from 410269 to 1124063 tid 1 2024-09-13T14:28:09.513 INFO:tasks.rados.rados.0.smithi151.stdout:1744: writing smithi15186934-29 from 1648313 to 2155864 tid 2 2024-09-13T14:28:09.519 INFO:tasks.rados.rados.0.smithi151.stdout:1744: writing smithi15186934-29 from 2917489 to 3210269 tid 3 2024-09-13T14:28:09.519 INFO:tasks.rados.rados.0.smithi151.stdout:1745: delete oid 3 current snap is 216 2024-09-13T14:28:09.520 INFO:tasks.rados.rados.0.smithi151.stdout:1743: expect (ObjNum 511 snap 164 seq_num 511) 2024-09-13T14:28:09.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:09 smithi151 ceph-mon[108966]: osdmap e498: 8 total, 8 up, 8 in 2024-09-13T14:28:09.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:09 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:09.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:09 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:28:09.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:09 smithi151 ceph-mon[108966]: osdmap e499: 8 total, 8 up, 8 in 2024-09-13T14:28:09.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:09 smithi151 ceph-mon[104450]: osdmap e498: 8 total, 8 up, 8 in 2024-09-13T14:28:09.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:09 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:09.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:09 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:28:09.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:09 smithi151 ceph-mon[104450]: osdmap e499: 8 total, 8 up, 8 in 2024-09-13T14:28:09.710 INFO:tasks.rados.rados.0.smithi151.stdout:1742: expect (ObjNum 662 snap 213 seq_num 662) 2024-09-13T14:28:09.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:09 smithi184 ceph-mon[92527]: osdmap e498: 8 total, 8 up, 8 in 2024-09-13T14:28:09.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:09 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:09.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:09 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:28:09.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:09 smithi184 ceph-mon[92527]: osdmap e499: 8 total, 8 up, 8 in 2024-09-13T14:28:09.854 INFO:tasks.rados.rados.0.smithi151.stdout:1744: finishing write tid 1 to smithi15186934-29 2024-09-13T14:28:09.854 INFO:tasks.rados.rados.0.smithi151.stdout:1744: finishing write tid 2 to smithi15186934-29 2024-09-13T14:28:09.854 INFO:tasks.rados.rados.0.smithi151.stdout:1744: finishing write tid 3 to smithi15186934-29 2024-09-13T14:28:09.855 INFO:tasks.rados.rados.0.smithi151.stdout:1744: finishing write tid 4 to smithi15186934-29 2024-09-13T14:28:09.855 INFO:tasks.rados.rados.0.smithi151.stdout:1744: finishing write tid 5 to smithi15186934-29 2024-09-13T14:28:09.855 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 29 v 496 (ObjNum 667 snap 216 seq_num 667) dirty exists 2024-09-13T14:28:09.855 INFO:tasks.rados.rados.0.smithi151.stdout:1744: left oid 29 (ObjNum 667 snap 216 seq_num 667) 2024-09-13T14:28:09.855 INFO:tasks.rados.rados.0.smithi151.stdout:1742: done (3 left) 2024-09-13T14:28:09.855 INFO:tasks.rados.rados.0.smithi151.stdout:1743: done (2 left) 2024-09-13T14:28:09.855 INFO:tasks.rados.rados.0.smithi151.stdout:1744: done (1 left) 2024-09-13T14:28:09.855 INFO:tasks.rados.rados.0.smithi151.stdout:1745: done (0 left) 2024-09-13T14:28:09.855 INFO:tasks.rados.rados.0.smithi151.stdout:1746: delete oid 44 current snap is 216 2024-09-13T14:28:09.856 INFO:tasks.rados.rados.0.smithi151.stdout:1746: done (0 left) 2024-09-13T14:28:09.856 INFO:tasks.rados.rados.0.smithi151.stdout:1747: write oid 25 current snap is 216 2024-09-13T14:28:09.856 INFO:tasks.rados.rados.0.smithi151.stdout:1747: seq_num 670 ranges {460690=655894,1540112=771231,2716111=539778} 2024-09-13T14:28:09.862 INFO:tasks.rados.rados.0.smithi151.stdout:1747: writing smithi15186934-25 from 460690 to 1116584 tid 1 2024-09-13T14:28:09.868 INFO:tasks.rados.rados.0.smithi151.stdout:1747: writing smithi15186934-25 from 1540112 to 2311343 tid 2 2024-09-13T14:28:09.874 INFO:tasks.rados.rados.0.smithi151.stdout:1747: writing smithi15186934-25 from 2716111 to 3255889 tid 3 2024-09-13T14:28:09.874 INFO:tasks.rados.rados.0.smithi151.stdout:1748: snap_remove snap 215 2024-09-13T14:28:10.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'echo wait for half of osds to upgrade' 2024-09-13T14:28:10.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:28:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:28:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:10.574 INFO:tasks.rados.rados.0.smithi151.stdout:1748: done (1 left) 2024-09-13T14:28:10.574 INFO:tasks.rados.rados.0.smithi151.stdout:1749: snap_remove snap 214 2024-09-13T14:28:10.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:10 smithi184 ceph-mon[92527]: pgmap v366: 89 pgs: 89 active+clean; 92 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 2.2 MiB/s wr, 22 op/s 2024-09-13T14:28:10.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:10 smithi184 ceph-mon[92527]: from='client.44150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:10.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:10 smithi184 ceph-mon[92527]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:28:10.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:10 smithi151 ceph-mon[104450]: pgmap v366: 89 pgs: 89 active+clean; 92 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 2.2 MiB/s wr, 22 op/s 2024-09-13T14:28:10.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:10 smithi151 ceph-mon[104450]: from='client.44150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:10.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:10 smithi151 ceph-mon[104450]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:28:10.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:10 smithi151 ceph-mon[108966]: pgmap v366: 89 pgs: 89 active+clean; 92 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 2.2 MiB/s wr, 22 op/s 2024-09-13T14:28:10.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:10 smithi151 ceph-mon[108966]: from='client.44150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:10.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:10 smithi151 ceph-mon[108966]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:28:11.154 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:11.580 INFO:tasks.rados.rados.0.smithi151.stdout:1749: done (1 left) 2024-09-13T14:28:11.580 INFO:tasks.rados.rados.0.smithi151.stdout:1750: snap_remove snap 213 2024-09-13T14:28:11.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:11 smithi184 ceph-mon[92527]: osdmap e500: 8 total, 8 up, 8 in 2024-09-13T14:28:11.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:11 smithi151 ceph-mon[104450]: osdmap e500: 8 total, 8 up, 8 in 2024-09-13T14:28:11.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:11 smithi151 ceph-mon[108966]: osdmap e500: 8 total, 8 up, 8 in 2024-09-13T14:28:12.582 INFO:tasks.rados.rados.0.smithi151.stdout:1750: done (1 left) 2024-09-13T14:28:12.582 INFO:tasks.rados.rados.0.smithi151.stdout:1751: write oid 38 current snap is 216 2024-09-13T14:28:12.582 INFO:tasks.rados.rados.0.smithi151.stdout:1747: finishing write tid 1 to smithi15186934-25 2024-09-13T14:28:12.583 INFO:tasks.rados.rados.0.smithi151.stdout:1751: seq_num 671 ranges {477497=731143,1998594=514889,3143534=133963} 2024-09-13T14:28:12.589 INFO:tasks.rados.rados.0.smithi151.stdout:1751: writing smithi15186934-38 from 477497 to 1208640 tid 1 2024-09-13T14:28:12.596 INFO:tasks.rados.rados.0.smithi151.stdout:1751: writing smithi15186934-38 from 1998594 to 2513483 tid 2 2024-09-13T14:28:12.601 INFO:tasks.rados.rados.0.smithi151.stdout:1751: writing smithi15186934-38 from 3143534 to 3277497 tid 3 2024-09-13T14:28:12.601 INFO:tasks.rados.rados.0.smithi151.stdout:1752: rollback oid 29 current snap is 216 2024-09-13T14:28:12.601 INFO:tasks.rados.rados.0.smithi151.stdout:1752: done (2 left) 2024-09-13T14:28:12.601 INFO:tasks.rados.rados.0.smithi151.stdout:1753: write oid 34 current snap is 216 2024-09-13T14:28:12.601 INFO:tasks.rados.rados.0.smithi151.stdout:1747: finishing write tid 2 to smithi15186934-25 2024-09-13T14:28:12.601 INFO:tasks.rados.rados.0.smithi151.stdout:1747: finishing write tid 3 to smithi15186934-25 2024-09-13T14:28:12.601 INFO:tasks.rados.rados.0.smithi151.stdout:1747: finishing write tid 4 to smithi15186934-25 2024-09-13T14:28:12.601 INFO:tasks.rados.rados.0.smithi151.stdout:1747: finishing write tid 5 to smithi15186934-25 2024-09-13T14:28:12.601 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 958 (ObjNum 670 snap 216 seq_num 670) dirty exists 2024-09-13T14:28:12.602 INFO:tasks.rados.rados.0.smithi151.stdout:1747: left oid 25 (ObjNum 670 snap 216 seq_num 670) 2024-09-13T14:28:12.602 INFO:tasks.rados.rados.0.smithi151.stdout:1753: seq_num 672 ranges {494304=406392,1540723=774900,2770957=523347} 2024-09-13T14:28:12.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:12 smithi184 ceph-mon[92527]: pgmap v369: 89 pgs: 89 active+clean; 92 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:28:12.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:12 smithi184 ceph-mon[92527]: osdmap e501: 8 total, 8 up, 8 in 2024-09-13T14:28:12.605 INFO:tasks.rados.rados.0.smithi151.stdout:1753: writing smithi15186934-34 from 494304 to 900696 tid 1 2024-09-13T14:28:12.614 INFO:tasks.rados.rados.0.smithi151.stdout:1753: writing smithi15186934-34 from 1540723 to 2315623 tid 2 2024-09-13T14:28:12.621 INFO:tasks.rados.rados.0.smithi151.stdout:1753: writing smithi15186934-34 from 2770957 to 3294304 tid 3 2024-09-13T14:28:12.621 INFO:tasks.rados.rados.0.smithi151.stdout:1747: done (2 left) 2024-09-13T14:28:12.621 INFO:tasks.rados.rados.0.smithi151.stdout:1754: read oid 18 snap -1 2024-09-13T14:28:12.621 INFO:tasks.rados.rados.0.smithi151.stdout:1754: expect (ObjNum 644 snap 209 seq_num 644) 2024-09-13T14:28:12.621 INFO:tasks.rados.rados.0.smithi151.stdout:1755: write oid 2 current snap is 216 2024-09-13T14:28:12.621 INFO:tasks.rados.rados.0.smithi151.stdout:1751: finishing write tid 1 to smithi15186934-38 2024-09-13T14:28:12.621 INFO:tasks.rados.rados.0.smithi151.stdout:1751: finishing write tid 2 to smithi15186934-38 2024-09-13T14:28:12.621 INFO:tasks.rados.rados.0.smithi151.stdout:1755: seq_num 673 ranges {511111=481641,1599205=518558,2682027=489947} 2024-09-13T14:28:12.626 INFO:tasks.rados.rados.0.smithi151.stdout:1755: writing smithi15186934-2 from 511111 to 992752 tid 1 2024-09-13T14:28:12.632 INFO:tasks.rados.rados.0.smithi151.stdout:1755: writing smithi15186934-2 from 1599205 to 2117763 tid 2 2024-09-13T14:28:12.642 INFO:tasks.rados.rados.0.smithi151.stdout:1755: writing smithi15186934-2 from 2682027 to 3171974 tid 3 2024-09-13T14:28:12.642 INFO:tasks.rados.rados.0.smithi151.stdout:1756: write oid 17 current snap is 216 2024-09-13T14:28:12.642 INFO:tasks.rados.rados.0.smithi151.stdout:1751: finishing write tid 3 to smithi15186934-38 2024-09-13T14:28:12.642 INFO:tasks.rados.rados.0.smithi151.stdout:1751: finishing write tid 4 to smithi15186934-38 2024-09-13T14:28:12.642 INFO:tasks.rados.rados.0.smithi151.stdout:1756: seq_num 674 ranges {527918=556890,1657687=778569,3225803=102115} 2024-09-13T14:28:12.645 INFO:tasks.rados.rados.0.smithi151.stdout:1756: writing smithi15186934-17 from 527918 to 1084808 tid 1 2024-09-13T14:28:12.655 INFO:tasks.rados.rados.0.smithi151.stdout:1756: writing smithi15186934-17 from 1657687 to 2436256 tid 2 2024-09-13T14:28:12.660 INFO:tasks.rados.rados.0.smithi151.stdout:1756: writing smithi15186934-17 from 3225803 to 3327918 tid 3 2024-09-13T14:28:12.660 INFO:tasks.rados.rados.0.smithi151.stdout:1757: snap_create 2024-09-13T14:28:12.660 INFO:tasks.rados.rados.0.smithi151.stdout:1751: finishing write tid 5 to smithi15186934-38 2024-09-13T14:28:12.660 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 598 (ObjNum 671 snap 216 seq_num 671) dirty exists 2024-09-13T14:28:12.660 INFO:tasks.rados.rados.0.smithi151.stdout:1751: left oid 38 (ObjNum 671 snap 216 seq_num 671) 2024-09-13T14:28:12.661 INFO:tasks.rados.rados.0.smithi151.stdout:1753: finishing write tid 1 to smithi15186934-34 2024-09-13T14:28:12.661 INFO:tasks.rados.rados.0.smithi151.stdout:1753: finishing write tid 2 to smithi15186934-34 2024-09-13T14:28:12.661 INFO:tasks.rados.rados.0.smithi151.stdout:1754: expect (ObjNum 644 snap 209 seq_num 644) 2024-09-13T14:28:12.845 INFO:tasks.rados.rados.0.smithi151.stdout:1753: finishing write tid 3 to smithi15186934-34 2024-09-13T14:28:12.845 INFO:tasks.rados.rados.0.smithi151.stdout:1753: finishing write tid 4 to smithi15186934-34 2024-09-13T14:28:12.845 INFO:tasks.rados.rados.0.smithi151.stdout:1753: finishing write tid 5 to smithi15186934-34 2024-09-13T14:28:12.845 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 645 (ObjNum 672 snap 216 seq_num 672) dirty exists 2024-09-13T14:28:12.845 INFO:tasks.rados.rados.0.smithi151.stdout:1753: left oid 34 (ObjNum 672 snap 216 seq_num 672) 2024-09-13T14:28:12.845 INFO:tasks.rados.rados.0.smithi151.stdout:1755: finishing write tid 1 to smithi15186934-2 2024-09-13T14:28:12.845 INFO:tasks.rados.rados.0.smithi151.stdout:1755: finishing write tid 2 to smithi15186934-2 2024-09-13T14:28:12.845 INFO:tasks.rados.rados.0.smithi151.stdout:1755: finishing write tid 3 to smithi15186934-2 2024-09-13T14:28:12.846 INFO:tasks.rados.rados.0.smithi151.stdout:1755: finishing write tid 4 to smithi15186934-2 2024-09-13T14:28:12.846 INFO:tasks.rados.rados.0.smithi151.stdout:1755: finishing write tid 5 to smithi15186934-2 2024-09-13T14:28:12.846 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 2 v 651 (ObjNum 673 snap 216 seq_num 673) dirty exists 2024-09-13T14:28:12.846 INFO:tasks.rados.rados.0.smithi151.stdout:1755: left oid 2 (ObjNum 673 snap 216 seq_num 673) 2024-09-13T14:28:12.846 INFO:tasks.rados.rados.0.smithi151.stdout:1756: finishing write tid 1 to smithi15186934-17 2024-09-13T14:28:12.846 INFO:tasks.rados.rados.0.smithi151.stdout:1756: finishing write tid 2 to smithi15186934-17 2024-09-13T14:28:12.846 INFO:tasks.rados.rados.0.smithi151.stdout:1756: finishing write tid 3 to smithi15186934-17 2024-09-13T14:28:12.846 INFO:tasks.rados.rados.0.smithi151.stdout:1756: finishing write tid 4 to smithi15186934-17 2024-09-13T14:28:12.846 INFO:tasks.rados.rados.0.smithi151.stdout:1756: finishing write tid 5 to smithi15186934-17 2024-09-13T14:28:12.846 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 1165 (ObjNum 674 snap 216 seq_num 674) dirty exists 2024-09-13T14:28:12.846 INFO:tasks.rados.rados.0.smithi151.stdout:1756: left oid 17 (ObjNum 674 snap 216 seq_num 674) 2024-09-13T14:28:12.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:12 smithi151 ceph-mon[104450]: pgmap v369: 89 pgs: 89 active+clean; 92 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:28:12.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:12 smithi151 ceph-mon[104450]: osdmap e501: 8 total, 8 up, 8 in 2024-09-13T14:28:12.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:12 smithi151 ceph-mon[108966]: pgmap v369: 89 pgs: 89 active+clean; 92 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:28:12.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:12 smithi151 ceph-mon[108966]: osdmap e501: 8 total, 8 up, 8 in 2024-09-13T14:28:13.336 INFO:teuthology.orchestra.run.smithi151.stdout:wait for half of osds to upgrade 2024-09-13T14:28:13.601 INFO:tasks.rados.rados.0.smithi151.stdout:1751: done (5 left) 2024-09-13T14:28:13.601 INFO:tasks.rados.rados.0.smithi151.stdout:1753: done (4 left) 2024-09-13T14:28:13.601 INFO:tasks.rados.rados.0.smithi151.stdout:1754: done (3 left) 2024-09-13T14:28:13.601 INFO:tasks.rados.rados.0.smithi151.stdout:1755: done (2 left) 2024-09-13T14:28:13.601 INFO:tasks.rados.rados.0.smithi151.stdout:1756: done (1 left) 2024-09-13T14:28:13.601 INFO:tasks.rados.rados.0.smithi151.stdout:1757: done (0 left) 2024-09-13T14:28:13.601 INFO:tasks.rados.rados.0.smithi151.stdout:1758: snap_remove snap 216 2024-09-13T14:28:13.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:13 smithi184 ceph-mon[92527]: osdmap e502: 8 total, 8 up, 8 in 2024-09-13T14:28:13.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:13 smithi151 ceph-mon[104450]: osdmap e502: 8 total, 8 up, 8 in 2024-09-13T14:28:13.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:13 smithi151 ceph-mon[108966]: osdmap e502: 8 total, 8 up, 8 in 2024-09-13T14:28:13.986 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'while ! ceph osd versions | grep $sha1 | egrep '"'"': [45678]'"'"' && ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error; do sleep 2 ; done' 2024-09-13T14:28:14.502 INFO:tasks.rados.rados.0.smithi151.stdout:1758: done (0 left) 2024-09-13T14:28:14.502 INFO:tasks.rados.rados.0.smithi151.stdout:1759: snap_create 2024-09-13T14:28:14.628 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:14 smithi151 ceph-mon[104450]: pgmap v372: 89 pgs: 89 active+clean; 99 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-13T14:28:14.628 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:14 smithi151 ceph-mon[104450]: osdmap e503: 8 total, 8 up, 8 in 2024-09-13T14:28:14.628 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:14 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:14.628 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:14 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:14.628 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:14 smithi151 ceph-mon[104450]: osdmap e504: 8 total, 8 up, 8 in 2024-09-13T14:28:14.628 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:28:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:28:14] "GET /metrics HTTP/1.1" 200 36401 "" "Prometheus/2.51.0" 2024-09-13T14:28:14.629 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:14 smithi151 ceph-mon[108966]: pgmap v372: 89 pgs: 89 active+clean; 99 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-13T14:28:14.629 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:14 smithi151 ceph-mon[108966]: osdmap e503: 8 total, 8 up, 8 in 2024-09-13T14:28:14.629 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:14 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:14.629 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:14 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:14.629 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:14 smithi151 ceph-mon[108966]: osdmap e504: 8 total, 8 up, 8 in 2024-09-13T14:28:14.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:14 smithi184 ceph-mon[92527]: pgmap v372: 89 pgs: 89 active+clean; 99 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-13T14:28:14.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:14 smithi184 ceph-mon[92527]: osdmap e503: 8 total, 8 up, 8 in 2024-09-13T14:28:14.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:14 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:14.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:14 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:14.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:14 smithi184 ceph-mon[92527]: osdmap e504: 8 total, 8 up, 8 in 2024-09-13T14:28:14.863 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:15.503 INFO:tasks.rados.rados.0.smithi151.stdout:1759: done (0 left) 2024-09-13T14:28:15.503 INFO:tasks.rados.rados.0.smithi151.stdout:1760: read oid 19 snap -1 2024-09-13T14:28:15.503 INFO:tasks.rados.rados.0.smithi151.stdout:1760: expect (ObjNum 323 snap 101 seq_num 323) 2024-09-13T14:28:15.503 INFO:tasks.rados.rados.0.smithi151.stdout:1761: snap_create 2024-09-13T14:28:15.507 INFO:tasks.rados.rados.0.smithi151.stdout:1760: expect (ObjNum 323 snap 101 seq_num 323) 2024-09-13T14:28:16.509 INFO:tasks.rados.rados.0.smithi151.stdout:1760: done (1 left) 2024-09-13T14:28:16.509 INFO:tasks.rados.rados.0.smithi151.stdout:1761: done (0 left) 2024-09-13T14:28:16.509 INFO:tasks.rados.rados.0.smithi151.stdout:1762: delete oid 42 current snap is 219 2024-09-13T14:28:16.511 INFO:tasks.rados.rados.0.smithi151.stdout:1762: done (0 left) 2024-09-13T14:28:16.511 INFO:tasks.rados.rados.0.smithi151.stdout:1763: rollback oid 48 current snap is 219 2024-09-13T14:28:16.511 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 48 to 218 2024-09-13T14:28:16.511 INFO:tasks.rados.rados.0.smithi151.stdout:1764: write oid 24 current snap is 219 2024-09-13T14:28:16.511 INFO:tasks.rados.rados.0.smithi151.stdout:1764: seq_num 676 ranges {561532=707388,2058298=782238} 2024-09-13T14:28:16.519 INFO:tasks.rados.rados.0.smithi151.stdout:1764: writing smithi15186934-24 from 561532 to 1268920 tid 1 2024-09-13T14:28:16.529 INFO:tasks.rados.rados.0.smithi151.stdout:1764: writing smithi15186934-24 from 2058298 to 2840536 tid 2 2024-09-13T14:28:16.529 INFO:tasks.rados.rados.0.smithi151.stdout:1765: delete oid 11 current snap is 219 2024-09-13T14:28:16.529 INFO:tasks.rados.rados.0.smithi151.stdout:1763: finishing rollback tid 1 to smithi15186934-48 2024-09-13T14:28:16.529 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 48 v 0 (ObjNum 49 snap 0 seq_num 3092667088) dirty dne 2024-09-13T14:28:16.529 INFO:tasks.rados.rados.0.smithi151.stdout:1764: finishing write tid 1 to smithi15186934-24 2024-09-13T14:28:16.533 INFO:tasks.rados.rados.0.smithi151.stdout:1763: done (2 left) 2024-09-13T14:28:16.533 INFO:tasks.rados.rados.0.smithi151.stdout:1765: done (1 left) 2024-09-13T14:28:16.533 INFO:tasks.rados.rados.0.smithi151.stdout:1766: read oid 33 snap 218 2024-09-13T14:28:16.533 INFO:tasks.rados.rados.0.smithi151.stdout:1766: expect deleted 2024-09-13T14:28:16.533 INFO:tasks.rados.rados.0.smithi151.stdout:1767: write oid 31 current snap is 219 2024-09-13T14:28:16.533 INFO:tasks.rados.rados.0.smithi151.stdout:1767: seq_num 678 ranges {595146=574239,1891615=669554,3219142=176004} 2024-09-13T14:28:16.540 INFO:tasks.rados.rados.0.smithi151.stdout:1767: writing smithi15186934-31 from 595146 to 1169385 tid 1 2024-09-13T14:28:16.549 INFO:tasks.rados.rados.0.smithi151.stdout:1767: writing smithi15186934-31 from 1891615 to 2561169 tid 2 2024-09-13T14:28:16.554 INFO:tasks.rados.rados.0.smithi151.stdout:1767: writing smithi15186934-31 from 3219142 to 3395146 tid 3 2024-09-13T14:28:16.554 INFO:tasks.rados.rados.0.smithi151.stdout:1768: snap_create 2024-09-13T14:28:16.555 INFO:tasks.rados.rados.0.smithi151.stdout:1764: finishing write tid 2 to smithi15186934-24 2024-09-13T14:28:16.555 INFO:tasks.rados.rados.0.smithi151.stdout:1764: finishing write tid 3 to smithi15186934-24 2024-09-13T14:28:16.555 INFO:tasks.rados.rados.0.smithi151.stdout:1764: finishing write tid 4 to smithi15186934-24 2024-09-13T14:28:16.555 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 24 v 1169 (ObjNum 676 snap 219 seq_num 676) dirty exists 2024-09-13T14:28:16.555 INFO:tasks.rados.rados.0.smithi151.stdout:1764: left oid 24 (ObjNum 676 snap 219 seq_num 676) 2024-09-13T14:28:16.555 INFO:tasks.rados.rados.0.smithi151.stdout:1767: finishing write tid 1 to smithi15186934-31 2024-09-13T14:28:16.555 INFO:tasks.rados.rados.0.smithi151.stdout:1767: finishing write tid 2 to smithi15186934-31 2024-09-13T14:28:16.556 INFO:tasks.rados.rados.0.smithi151.stdout:1767: finishing write tid 3 to smithi15186934-31 2024-09-13T14:28:16.556 INFO:tasks.rados.rados.0.smithi151.stdout:1767: finishing write tid 4 to smithi15186934-31 2024-09-13T14:28:16.556 INFO:tasks.rados.rados.0.smithi151.stdout:1767: finishing write tid 5 to smithi15186934-31 2024-09-13T14:28:16.556 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 31 v 862 (ObjNum 678 snap 219 seq_num 678) dirty exists 2024-09-13T14:28:16.556 INFO:tasks.rados.rados.0.smithi151.stdout:1767: left oid 31 (ObjNum 678 snap 219 seq_num 678) 2024-09-13T14:28:16.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:16 smithi184 ceph-mon[92527]: pgmap v375: 89 pgs: 89 active+clean; 99 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-13T14:28:16.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:16 smithi184 ceph-mon[92527]: osdmap e505: 8 total, 8 up, 8 in 2024-09-13T14:28:16.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:16.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:16 smithi151 ceph-mon[104450]: pgmap v375: 89 pgs: 89 active+clean; 99 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-13T14:28:16.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:16 smithi151 ceph-mon[104450]: osdmap e505: 8 total, 8 up, 8 in 2024-09-13T14:28:16.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:16.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:16 smithi151 ceph-mon[108966]: pgmap v375: 89 pgs: 89 active+clean; 99 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-13T14:28:16.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:16 smithi151 ceph-mon[108966]: osdmap e505: 8 total, 8 up, 8 in 2024-09-13T14:28:16.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:17.497 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:28:17.523 INFO:tasks.rados.rados.0.smithi151.stdout:1764: done (3 left) 2024-09-13T14:28:17.523 INFO:tasks.rados.rados.0.smithi151.stdout:1766: done (2 left) 2024-09-13T14:28:17.523 INFO:tasks.rados.rados.0.smithi151.stdout:1767: done (1 left) 2024-09-13T14:28:17.523 INFO:tasks.rados.rados.0.smithi151.stdout:1768: done (0 left) 2024-09-13T14:28:17.523 INFO:tasks.rados.rados.0.smithi151.stdout:1769: rollback oid 26 current snap is 220 2024-09-13T14:28:17.523 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 26 to 217 2024-09-13T14:28:17.524 INFO:tasks.rados.rados.0.smithi151.stdout:1770: read oid 24 snap -1 2024-09-13T14:28:17.524 INFO:tasks.rados.rados.0.smithi151.stdout:1770: expect (ObjNum 676 snap 219 seq_num 676) 2024-09-13T14:28:17.524 INFO:tasks.rados.rados.0.smithi151.stdout:1771: delete oid 31 current snap is 220 2024-09-13T14:28:17.527 INFO:tasks.rados.rados.0.smithi151.stdout:1771: done (2 left) 2024-09-13T14:28:17.527 INFO:tasks.rados.rados.0.smithi151.stdout:1772: rollback oid 19 current snap is 220 2024-09-13T14:28:17.527 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 19 to 217 2024-09-13T14:28:17.527 INFO:tasks.rados.rados.0.smithi151.stdout:1773: read oid 21 snap -1 2024-09-13T14:28:17.527 INFO:tasks.rados.rados.0.smithi151.stdout:1773: expect (ObjNum 578 snap 186 seq_num 578) 2024-09-13T14:28:17.527 INFO:tasks.rados.rados.0.smithi151.stdout:1774: snap_create 2024-09-13T14:28:17.529 INFO:tasks.rados.rados.0.smithi151.stdout:1769: finishing rollback tid 0 to smithi15186934-26 2024-09-13T14:28:17.529 INFO:tasks.rados.rados.0.smithi151.stdout:1769: finishing rollback tid 1 to smithi15186934-26 2024-09-13T14:28:17.529 INFO:tasks.rados.rados.0.smithi151.stdout:1769: finishing rollback tid 2 to smithi15186934-26 2024-09-13T14:28:17.529 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 636 (ObjNum 598 snap 192 seq_num 598) dirty exists 2024-09-13T14:28:17.530 INFO:tasks.rados.rados.0.smithi151.stdout:1772: finishing rollback tid 0 to smithi15186934-19 2024-09-13T14:28:17.530 INFO:tasks.rados.rados.0.smithi151.stdout:1772: finishing rollback tid 1 to smithi15186934-19 2024-09-13T14:28:17.530 INFO:tasks.rados.rados.0.smithi151.stdout:1772: finishing rollback tid 2 to smithi15186934-19 2024-09-13T14:28:17.530 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 19 v 448 (ObjNum 323 snap 101 seq_num 323) dirty exists 2024-09-13T14:28:17.543 INFO:tasks.rados.rados.0.smithi151.stdout:1770: expect (ObjNum 676 snap 219 seq_num 676) 2024-09-13T14:28:17.775 INFO:tasks.rados.rados.0.smithi151.stdout:1773: expect (ObjNum 578 snap 186 seq_num 578) 2024-09-13T14:28:17.819 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[104450]: osdmap e506: 8 total, 8 up, 8 in 2024-09-13T14:28:17.819 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:17.819 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:17.820 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:28:17.820 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:28:17.820 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:17.820 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/114150392' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:17.820 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:28:17.820 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:28:17.820 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:28:17.820 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:17.820 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:28:17.820 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-13T14:28:17.820 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:28:17.820 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-13T14:28:17.820 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:28:17.821 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-13T14:28:17.821 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:28:17.821 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:17.821 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-13T14:28:17.821 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[108966]: osdmap e506: 8 total, 8 up, 8 in 2024-09-13T14:28:17.821 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:17.821 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:17.821 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:28:17.821 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:28:17.821 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:17.821 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/114150392' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:17.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:28:17.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:28:17.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:28:17.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:17.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:28:17.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-13T14:28:17.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:28:17.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-13T14:28:17.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:28:17.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-13T14:28:17.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:28:17.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:17.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-13T14:28:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:17 smithi184 ceph-mon[92527]: osdmap e506: 8 total, 8 up, 8 in 2024-09-13T14:28:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:28:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:28:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:17 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/114150392' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:28:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:28:17.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:28:17.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:17.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:28:17.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-13T14:28:17.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:28:17.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-13T14:28:17.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:28:17.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-13T14:28:17.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:28:17.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:17.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-13T14:28:18.533 INFO:tasks.rados.rados.0.smithi151.stdout:1769: done (4 left) 2024-09-13T14:28:18.533 INFO:tasks.rados.rados.0.smithi151.stdout:1770: done (3 left) 2024-09-13T14:28:18.533 INFO:tasks.rados.rados.0.smithi151.stdout:1772: done (2 left) 2024-09-13T14:28:18.533 INFO:tasks.rados.rados.0.smithi151.stdout:1773: done (1 left) 2024-09-13T14:28:18.533 INFO:tasks.rados.rados.0.smithi151.stdout:1774: done (0 left) 2024-09-13T14:28:18.533 INFO:tasks.rados.rados.0.smithi151.stdout:1775: write oid 35 current snap is 221 2024-09-13T14:28:18.533 INFO:tasks.rados.rados.0.smithi151.stdout:1775: seq_num 680 ranges {628760=724737,1892226=673223,3157635=271125} 2024-09-13T14:28:18.540 INFO:tasks.rados.rados.0.smithi151.stdout:1775: writing smithi15186934-35 from 628760 to 1353497 tid 1 2024-09-13T14:28:18.546 INFO:tasks.rados.rados.0.smithi151.stdout:1775: writing smithi15186934-35 from 1892226 to 2565449 tid 2 2024-09-13T14:28:18.551 INFO:tasks.rados.rados.0.smithi151.stdout:1775: writing smithi15186934-35 from 3157635 to 3428760 tid 3 2024-09-13T14:28:18.552 INFO:tasks.rados.rados.0.smithi151.stdout:1776: write oid 48 current snap is 221 2024-09-13T14:28:18.552 INFO:tasks.rados.rados.0.smithi151.stdout:1775: finishing write tid 1 to smithi15186934-35 2024-09-13T14:28:18.552 INFO:tasks.rados.rados.0.smithi151.stdout:1775: finishing write tid 2 to smithi15186934-35 2024-09-13T14:28:18.552 INFO:tasks.rados.rados.0.smithi151.stdout:1776: seq_num 681 ranges {645567=799986,1950708=533234,3185058=260509} 2024-09-13T14:28:18.559 INFO:tasks.rados.rados.0.smithi151.stdout:1776: writing smithi15186934-48 from 645567 to 1445553 tid 1 2024-09-13T14:28:18.565 INFO:tasks.rados.rados.0.smithi151.stdout:1776: writing smithi15186934-48 from 1950708 to 2483942 tid 2 2024-09-13T14:28:18.571 INFO:tasks.rados.rados.0.smithi151.stdout:1776: writing smithi15186934-48 from 3185058 to 3445567 tid 3 2024-09-13T14:28:18.571 INFO:tasks.rados.rados.0.smithi151.stdout:1777: rollback oid 20 current snap is 221 2024-09-13T14:28:18.571 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 20 to 217 2024-09-13T14:28:18.571 INFO:tasks.rados.rados.0.smithi151.stdout:1775: finishing write tid 3 to smithi15186934-35 2024-09-13T14:28:18.571 INFO:tasks.rados.rados.0.smithi151.stdout:1775: finishing write tid 4 to smithi15186934-35 2024-09-13T14:28:18.571 INFO:tasks.rados.rados.0.smithi151.stdout:1775: finishing write tid 5 to smithi15186934-35 2024-09-13T14:28:18.571 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 35 v 1174 (ObjNum 680 snap 221 seq_num 680) dirty exists 2024-09-13T14:28:18.572 INFO:tasks.rados.rados.0.smithi151.stdout:1775: left oid 35 (ObjNum 680 snap 221 seq_num 680) 2024-09-13T14:28:18.572 INFO:tasks.rados.rados.0.smithi151.stdout:1776: finishing write tid 1 to smithi15186934-48 2024-09-13T14:28:18.572 INFO:tasks.rados.rados.0.smithi151.stdout:1776: finishing write tid 2 to smithi15186934-48 2024-09-13T14:28:18.572 INFO:tasks.rados.rados.0.smithi151.stdout:1775: done (2 left) 2024-09-13T14:28:18.572 INFO:tasks.rados.rados.0.smithi151.stdout:1778: read oid 35 snap -1 2024-09-13T14:28:18.572 INFO:tasks.rados.rados.0.smithi151.stdout:1778: expect (ObjNum 680 snap 221 seq_num 680) 2024-09-13T14:28:18.572 INFO:tasks.rados.rados.0.smithi151.stdout:1779: snap_create 2024-09-13T14:28:18.572 INFO:tasks.rados.rados.0.smithi151.stdout:1777: finishing rollback tid 1 to smithi15186934-20 2024-09-13T14:28:18.572 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 0 (ObjNum 33 snap 0 seq_num 2) dirty dne 2024-09-13T14:28:18.573 INFO:tasks.rados.rados.0.smithi151.stdout:1776: finishing write tid 3 to smithi15186934-48 2024-09-13T14:28:18.573 INFO:tasks.rados.rados.0.smithi151.stdout:1776: finishing write tid 4 to smithi15186934-48 2024-09-13T14:28:18.573 INFO:tasks.rados.rados.0.smithi151.stdout:1776: finishing write tid 5 to smithi15186934-48 2024-09-13T14:28:18.573 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 48 v 969 (ObjNum 681 snap 221 seq_num 681) dirty exists 2024-09-13T14:28:18.573 INFO:tasks.rados.rados.0.smithi151.stdout:1776: left oid 48 (ObjNum 681 snap 221 seq_num 681) 2024-09-13T14:28:18.582 INFO:tasks.rados.rados.0.smithi151.stdout:1778: expect (ObjNum 680 snap 221 seq_num 680) 2024-09-13T14:28:18.743 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:18 smithi151 ceph-mon[104450]: pgmap v378: 89 pgs: 89 active+clean; 99 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.4 MiB/s rd, 2.1 MiB/s wr, 12 op/s 2024-09-13T14:28:18.743 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:18 smithi151 ceph-mon[104450]: Upgrade: Setting container_image for all mon 2024-09-13T14:28:18.744 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:18 smithi151 ceph-mon[104450]: Upgrade: Setting container_image for all crash 2024-09-13T14:28:18.744 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:18 smithi151 ceph-mon[104450]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-13T14:28:18.744 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:18 smithi151 ceph-mon[104450]: Upgrade: osd.0 is safe to restart 2024-09-13T14:28:18.744 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:18 smithi151 ceph-mon[104450]: from='client.44162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:18.744 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:18 smithi151 ceph-mon[104450]: osdmap e507: 8 total, 8 up, 8 in 2024-09-13T14:28:18.744 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:18 smithi151 ceph-mon[104450]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:18.744 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:18 smithi151 ceph-mon[108966]: pgmap v378: 89 pgs: 89 active+clean; 99 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.4 MiB/s rd, 2.1 MiB/s wr, 12 op/s 2024-09-13T14:28:18.744 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:18 smithi151 ceph-mon[108966]: Upgrade: Setting container_image for all mon 2024-09-13T14:28:18.744 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:18 smithi151 ceph-mon[108966]: Upgrade: Setting container_image for all crash 2024-09-13T14:28:18.745 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:18 smithi151 ceph-mon[108966]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-13T14:28:18.745 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:18 smithi151 ceph-mon[108966]: Upgrade: osd.0 is safe to restart 2024-09-13T14:28:18.745 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:18 smithi151 ceph-mon[108966]: from='client.44162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:18.745 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:18 smithi151 ceph-mon[108966]: osdmap e507: 8 total, 8 up, 8 in 2024-09-13T14:28:18.745 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:18 smithi151 ceph-mon[108966]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:18.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:18 smithi184 ceph-mon[92527]: pgmap v378: 89 pgs: 89 active+clean; 99 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.4 MiB/s rd, 2.1 MiB/s wr, 12 op/s 2024-09-13T14:28:18.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:18 smithi184 ceph-mon[92527]: Upgrade: Setting container_image for all mon 2024-09-13T14:28:18.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:18 smithi184 ceph-mon[92527]: Upgrade: Setting container_image for all crash 2024-09-13T14:28:18.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:18 smithi184 ceph-mon[92527]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-13T14:28:18.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:18 smithi184 ceph-mon[92527]: Upgrade: osd.0 is safe to restart 2024-09-13T14:28:18.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:18 smithi184 ceph-mon[92527]: from='client.44162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:18.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:18 smithi184 ceph-mon[92527]: osdmap e507: 8 total, 8 up, 8 in 2024-09-13T14:28:18.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:18 smithi184 ceph-mon[92527]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:19.508 INFO:tasks.rados.rados.0.smithi151.stdout:1776: done (3 left) 2024-09-13T14:28:19.508 INFO:tasks.rados.rados.0.smithi151.stdout:1777: done (2 left) 2024-09-13T14:28:19.508 INFO:tasks.rados.rados.0.smithi151.stdout:1778: done (1 left) 2024-09-13T14:28:19.508 INFO:tasks.rados.rados.0.smithi151.stdout:1779: done (0 left) 2024-09-13T14:28:19.508 INFO:tasks.rados.rados.0.smithi151.stdout:1780: delete oid 48 current snap is 222 2024-09-13T14:28:19.511 INFO:tasks.rados.rados.0.smithi151.stdout:1780: done (0 left) 2024-09-13T14:28:19.511 INFO:tasks.rados.rados.0.smithi151.stdout:1781: write oid 37 current snap is 222 2024-09-13T14:28:19.511 INFO:tasks.rados.rados.0.smithi151.stdout:1781: seq_num 683 ranges {679181=550484,1667672=536903,2839904=435373} 2024-09-13T14:28:19.518 INFO:tasks.rados.rados.0.smithi151.stdout:1781: writing smithi15186934-37 from 679181 to 1229665 tid 1 2024-09-13T14:28:19.523 INFO:tasks.rados.rados.0.smithi151.stdout:1781: writing smithi15186934-37 from 1667672 to 2204575 tid 2 2024-09-13T14:28:19.529 INFO:tasks.rados.rados.0.smithi151.stdout:1781: writing smithi15186934-37 from 2839904 to 3275277 tid 3 2024-09-13T14:28:19.530 INFO:tasks.rados.rados.0.smithi151.stdout:1782: snap_create 2024-09-13T14:28:19.530 INFO:tasks.rados.rados.0.smithi151.stdout:1781: finishing write tid 1 to smithi15186934-37 2024-09-13T14:28:19.530 INFO:tasks.rados.rados.0.smithi151.stdout:1781: finishing write tid 2 to smithi15186934-37 2024-09-13T14:28:19.533 INFO:tasks.rados.rados.0.smithi151.stdout:1781: finishing write tid 3 to smithi15186934-37 2024-09-13T14:28:19.533 INFO:tasks.rados.rados.0.smithi151.stdout:1781: finishing write tid 4 to smithi15186934-37 2024-09-13T14:28:19.533 INFO:tasks.rados.rados.0.smithi151.stdout:1781: finishing write tid 5 to smithi15186934-37 2024-09-13T14:28:19.533 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 37 v 801 (ObjNum 683 snap 222 seq_num 683) dirty exists 2024-09-13T14:28:19.533 INFO:tasks.rados.rados.0.smithi151.stdout:1781: left oid 37 (ObjNum 683 snap 222 seq_num 683) 2024-09-13T14:28:19.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:19 smithi184 ceph-mon[92527]: osdmap e508: 8 total, 8 up, 8 in 2024-09-13T14:28:19.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:19 smithi184 ceph-mon[92527]: Upgrade: Updating osd.0 2024-09-13T14:28:19.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:19 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:19.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:19 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-13T14:28:19.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:19 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:28:19.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:19 smithi184 ceph-mon[92527]: Deploying daemon osd.0 on smithi151 2024-09-13T14:28:19.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:19 smithi184 ceph-mon[92527]: osdmap e509: 8 total, 8 up, 8 in 2024-09-13T14:28:19.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:19 smithi151 ceph-mon[104450]: osdmap e508: 8 total, 8 up, 8 in 2024-09-13T14:28:19.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:19 smithi151 ceph-mon[104450]: Upgrade: Updating osd.0 2024-09-13T14:28:19.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:19 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:19.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:19 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-13T14:28:19.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:19 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:28:19.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:19 smithi151 ceph-mon[104450]: Deploying daemon osd.0 on smithi151 2024-09-13T14:28:19.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:19 smithi151 ceph-mon[104450]: osdmap e509: 8 total, 8 up, 8 in 2024-09-13T14:28:19.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:19 smithi151 ceph-mon[108966]: osdmap e508: 8 total, 8 up, 8 in 2024-09-13T14:28:19.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:19 smithi151 ceph-mon[108966]: Upgrade: Updating osd.0 2024-09-13T14:28:19.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:19 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:19.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:19 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-13T14:28:19.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:19 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:28:19.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:19 smithi151 ceph-mon[108966]: Deploying daemon osd.0 on smithi151 2024-09-13T14:28:19.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:19 smithi151 ceph-mon[108966]: osdmap e509: 8 total, 8 up, 8 in 2024-09-13T14:28:20.375 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:28:20.533 INFO:tasks.rados.rados.0.smithi151.stdout:1781: done (1 left) 2024-09-13T14:28:20.533 INFO:tasks.rados.rados.0.smithi151.stdout:1782: done (0 left) 2024-09-13T14:28:20.534 INFO:tasks.rados.rados.0.smithi151.stdout:1783: delete oid 20 current snap is 223 2024-09-13T14:28:20.537 INFO:tasks.rados.rados.0.smithi151.stdout:1783: done (0 left) 2024-09-13T14:28:20.537 INFO:tasks.rados.rados.0.smithi151.stdout:1784: write oid 16 current snap is 223 2024-09-13T14:28:20.537 INFO:tasks.rados.rados.0.smithi151.stdout:1784: seq_num 685 ranges {712795=417335,1784636=540572,2894750=457917} 2024-09-13T14:28:20.543 INFO:tasks.rados.rados.0.smithi151.stdout:1784: writing smithi15186934-16 from 712795 to 1130130 tid 1 2024-09-13T14:28:20.550 INFO:tasks.rados.rados.0.smithi151.stdout:1784: writing smithi15186934-16 from 1784636 to 2325208 tid 2 2024-09-13T14:28:20.556 INFO:tasks.rados.rados.0.smithi151.stdout:1784: writing smithi15186934-16 from 2894750 to 3352667 tid 3 2024-09-13T14:28:20.556 INFO:tasks.rados.rados.0.smithi151.stdout:1785: read oid 11 snap -1 2024-09-13T14:28:20.556 INFO:tasks.rados.rados.0.smithi151.stdout:1785: expect deleted 2024-09-13T14:28:20.556 INFO:tasks.rados.rados.0.smithi151.stdout:1786: read oid 3 snap -1 2024-09-13T14:28:20.556 INFO:tasks.rados.rados.0.smithi151.stdout:1786: expect deleted 2024-09-13T14:28:20.556 INFO:tasks.rados.rados.0.smithi151.stdout:1787: rollback oid 28 current snap is 223 2024-09-13T14:28:20.556 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 28 to 221 2024-09-13T14:28:20.557 INFO:tasks.rados.rados.0.smithi151.stdout:1784: finishing write tid 1 to smithi15186934-16 2024-09-13T14:28:20.557 INFO:tasks.rados.rados.0.smithi151.stdout:1784: finishing write tid 2 to smithi15186934-16 2024-09-13T14:28:20.557 INFO:tasks.rados.rados.0.smithi151.stdout:1788: write oid 26 current snap is 223 2024-09-13T14:28:20.557 INFO:tasks.rados.rados.0.smithi151.stdout:1788: seq_num 686 ranges {729602=492584,1843118=684230,3322173=207429} 2024-09-13T14:28:20.562 INFO:tasks.rados.rados.0.smithi151.stdout:1788: writing smithi15186934-26 from 729602 to 1222186 tid 1 2024-09-13T14:28:20.569 INFO:tasks.rados.rados.0.smithi151.stdout:1788: writing smithi15186934-26 from 1843118 to 2527348 tid 2 2024-09-13T14:28:20.576 INFO:tasks.rados.rados.0.smithi151.stdout:1788: writing smithi15186934-26 from 3322173 to 3529602 tid 3 2024-09-13T14:28:20.576 INFO:tasks.rados.rados.0.smithi151.stdout:1789: rollback oid 45 current snap is 223 2024-09-13T14:28:20.576 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 45 to 219 2024-09-13T14:28:20.576 INFO:tasks.rados.rados.0.smithi151.stdout:1787: finishing rollback tid 1 to smithi15186934-28 2024-09-13T14:28:20.576 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 28 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-13T14:28:20.576 INFO:tasks.rados.rados.0.smithi151.stdout:1784: finishing write tid 3 to smithi15186934-16 2024-09-13T14:28:20.576 INFO:tasks.rados.rados.0.smithi151.stdout:1784: finishing write tid 4 to smithi15186934-16 2024-09-13T14:28:20.577 INFO:tasks.rados.rados.0.smithi151.stdout:1784: finishing write tid 5 to smithi15186934-16 2024-09-13T14:28:20.577 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 16 v 450 (ObjNum 685 snap 223 seq_num 685) dirty exists 2024-09-13T14:28:20.577 INFO:tasks.rados.rados.0.smithi151.stdout:1784: left oid 16 (ObjNum 685 snap 223 seq_num 685) 2024-09-13T14:28:20.577 INFO:tasks.rados.rados.0.smithi151.stdout:1788: finishing write tid 1 to smithi15186934-26 2024-09-13T14:28:20.577 INFO:tasks.rados.rados.0.smithi151.stdout:1788: finishing write tid 2 to smithi15186934-26 2024-09-13T14:28:20.577 INFO:tasks.rados.rados.0.smithi151.stdout:1784: done (5 left) 2024-09-13T14:28:20.577 INFO:tasks.rados.rados.0.smithi151.stdout:1785: done (4 left) 2024-09-13T14:28:20.577 INFO:tasks.rados.rados.0.smithi151.stdout:1786: done (3 left) 2024-09-13T14:28:20.577 INFO:tasks.rados.rados.0.smithi151.stdout:1787: done (2 left) 2024-09-13T14:28:20.577 INFO:tasks.rados.rados.0.smithi151.stdout:1790: write oid 5 current snap is 223 2024-09-13T14:28:20.577 INFO:tasks.rados.rados.0.smithi151.stdout:1790: seq_num 687 ranges {746409=567833,1901600=544241,3065949=480460} 2024-09-13T14:28:20.582 INFO:tasks.rados.rados.0.smithi151.stdout:1790: writing smithi15186934-5 from 746409 to 1314242 tid 1 2024-09-13T14:28:20.589 INFO:tasks.rados.rados.0.smithi151.stdout:1790: writing smithi15186934-5 from 1901600 to 2445841 tid 2 2024-09-13T14:28:20.596 INFO:tasks.rados.rados.0.smithi151.stdout:1790: writing smithi15186934-5 from 3065949 to 3546409 tid 3 2024-09-13T14:28:20.596 INFO:tasks.rados.rados.0.smithi151.stdout:1791: read oid 37 snap -1 2024-09-13T14:28:20.596 INFO:tasks.rados.rados.0.smithi151.stdout:1791: expect (ObjNum 683 snap 222 seq_num 683) 2024-09-13T14:28:20.596 INFO:tasks.rados.rados.0.smithi151.stdout:1792: rollback oid 12 current snap is 223 2024-09-13T14:28:20.596 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 12 to 219 2024-09-13T14:28:20.596 INFO:tasks.rados.rados.0.smithi151.stdout:1789: finishing rollback tid 1 to smithi15186934-45 2024-09-13T14:28:20.596 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 0 (ObjNum 49 snap 0 seq_num 3064833136) dirty dne 2024-09-13T14:28:20.596 INFO:tasks.rados.rados.0.smithi151.stdout:1788: finishing write tid 3 to smithi15186934-26 2024-09-13T14:28:20.596 INFO:tasks.rados.rados.0.smithi151.stdout:1788: finishing write tid 4 to smithi15186934-26 2024-09-13T14:28:20.596 INFO:tasks.rados.rados.0.smithi151.stdout:1788: finishing write tid 5 to smithi15186934-26 2024-09-13T14:28:20.596 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 641 (ObjNum 686 snap 223 seq_num 686) dirty exists 2024-09-13T14:28:20.596 INFO:tasks.rados.rados.0.smithi151.stdout:1788: left oid 26 (ObjNum 686 snap 223 seq_num 686) 2024-09-13T14:28:20.597 INFO:tasks.rados.rados.0.smithi151.stdout:1790: finishing write tid 1 to smithi15186934-5 2024-09-13T14:28:20.597 INFO:tasks.rados.rados.0.smithi151.stdout:1790: finishing write tid 2 to smithi15186934-5 2024-09-13T14:28:20.597 INFO:tasks.rados.rados.0.smithi151.stdout:1788: done (4 left) 2024-09-13T14:28:20.597 INFO:tasks.rados.rados.0.smithi151.stdout:1789: done (3 left) 2024-09-13T14:28:20.597 INFO:tasks.rados.rados.0.smithi151.stdout:1793: read oid 7 snap -1 2024-09-13T14:28:20.597 INFO:tasks.rados.rados.0.smithi151.stdout:1793: expect deleted 2024-09-13T14:28:20.597 INFO:tasks.rados.rados.0.smithi151.stdout:1794: snap_remove snap 217 2024-09-13T14:28:20.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:28:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:28:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:20.855 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:20 smithi151 ceph-mon[104450]: pgmap v381: 89 pgs: 89 active+clean; 99 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.4 MiB/s rd, 2.1 MiB/s wr, 12 op/s 2024-09-13T14:28:20.855 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:20 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/211888874' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:20.855 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:20 smithi151 ceph-mon[104450]: osdmap e510: 8 total, 8 up, 8 in 2024-09-13T14:28:20.856 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:20 smithi151 ceph-mon[108966]: pgmap v381: 89 pgs: 89 active+clean; 99 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.4 MiB/s rd, 2.1 MiB/s wr, 12 op/s 2024-09-13T14:28:20.856 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:20 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/211888874' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:20.856 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:20 smithi151 ceph-mon[108966]: osdmap e510: 8 total, 8 up, 8 in 2024-09-13T14:28:21.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:20 smithi184 ceph-mon[92527]: pgmap v381: 89 pgs: 89 active+clean; 99 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.4 MiB/s rd, 2.1 MiB/s wr, 12 op/s 2024-09-13T14:28:21.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:20 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/211888874' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:21.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:20 smithi184 ceph-mon[92527]: osdmap e510: 8 total, 8 up, 8 in 2024-09-13T14:28:21.537 INFO:tasks.rados.rados.0.smithi151.stdout:1794: done (4 left) 2024-09-13T14:28:21.538 INFO:tasks.rados.rados.0.smithi151.stdout:1795: snap_create 2024-09-13T14:28:21.538 INFO:tasks.rados.rados.0.smithi151.stdout:1792: finishing rollback tid 0 to smithi15186934-12 2024-09-13T14:28:21.538 INFO:tasks.rados.rados.0.smithi151.stdout:1792: finishing rollback tid 1 to smithi15186934-12 2024-09-13T14:28:21.538 INFO:tasks.rados.rados.0.smithi151.stdout:1792: finishing rollback tid 2 to smithi15186934-12 2024-09-13T14:28:21.538 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 12 v 388 (ObjNum 654 snap 212 seq_num 654) dirty exists 2024-09-13T14:28:21.538 INFO:tasks.rados.rados.0.smithi151.stdout:1790: finishing write tid 3 to smithi15186934-5 2024-09-13T14:28:21.539 INFO:tasks.rados.rados.0.smithi151.stdout:1790: finishing write tid 4 to smithi15186934-5 2024-09-13T14:28:21.539 INFO:tasks.rados.rados.0.smithi151.stdout:1790: finishing write tid 5 to smithi15186934-5 2024-09-13T14:28:21.539 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 656 (ObjNum 687 snap 223 seq_num 687) dirty exists 2024-09-13T14:28:21.539 INFO:tasks.rados.rados.0.smithi151.stdout:1790: left oid 5 (ObjNum 687 snap 223 seq_num 687) 2024-09-13T14:28:21.539 INFO:tasks.rados.rados.0.smithi151.stdout:1791: expect (ObjNum 683 snap 222 seq_num 683) 2024-09-13T14:28:21.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:21 smithi151 ceph-mon[104450]: from='client.44180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:21.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:21 smithi151 ceph-mon[104450]: from='client.34302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:21.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:21 smithi151 ceph-mon[104450]: osdmap e511: 8 total, 8 up, 8 in 2024-09-13T14:28:21.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:21 smithi151 ceph-mon[108966]: from='client.44180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:21.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:21 smithi151 ceph-mon[108966]: from='client.34302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:21.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:21 smithi151 ceph-mon[108966]: osdmap e511: 8 total, 8 up, 8 in 2024-09-13T14:28:22.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:21 smithi184 ceph-mon[92527]: from='client.44180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:22.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:21 smithi184 ceph-mon[92527]: from='client.34302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:22.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:21 smithi184 ceph-mon[92527]: osdmap e511: 8 total, 8 up, 8 in 2024-09-13T14:28:22.166 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:21 smithi151 systemd[1]: Stopping Ceph osd.0 for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:28:22.543 INFO:tasks.rados.rados.0.smithi151.stdout:1790: done (4 left) 2024-09-13T14:28:22.543 INFO:tasks.rados.rados.0.smithi151.stdout:1791: done (3 left) 2024-09-13T14:28:22.543 INFO:tasks.rados.rados.0.smithi151.stdout:1792: done (2 left) 2024-09-13T14:28:22.543 INFO:tasks.rados.rados.0.smithi151.stdout:1793: done (1 left) 2024-09-13T14:28:22.543 INFO:tasks.rados.rados.0.smithi151.stdout:1795: done (0 left) 2024-09-13T14:28:22.543 INFO:tasks.rados.rados.0.smithi151.stdout:1796: read oid 43 snap 223 2024-09-13T14:28:22.543 INFO:tasks.rados.rados.0.smithi151.stdout:1796: expect deleted 2024-09-13T14:28:22.543 INFO:tasks.rados.rados.0.smithi151.stdout:1797: write oid 48 current snap is 224 2024-09-13T14:28:22.543 INFO:tasks.rados.rados.0.smithi151.stdout:1797: seq_num 688 ranges {763216=643082,1843729=687899,3260666=302550} 2024-09-13T14:28:22.548 INFO:tasks.rados.rados.0.smithi151.stdout:1797: writing smithi15186934-48 from 763216 to 1406298 tid 1 2024-09-13T14:28:22.554 INFO:tasks.rados.rados.0.smithi151.stdout:1797: writing smithi15186934-48 from 1843729 to 2531628 tid 2 2024-09-13T14:28:22.560 INFO:tasks.rados.rados.0.smithi151.stdout:1797: writing smithi15186934-48 from 3260666 to 3563216 tid 3 2024-09-13T14:28:22.560 INFO:tasks.rados.rados.0.smithi151.stdout:1798: write oid 7 current snap is 224 2024-09-13T14:28:22.560 INFO:tasks.rados.rados.0.smithi151.stdout:1798: seq_num 689 ranges {780023=718331,1902211=547910,3004442=575581} 2024-09-13T14:28:22.568 INFO:tasks.rados.rados.0.smithi151.stdout:1798: writing smithi15186934-7 from 780023 to 1498354 tid 1 2024-09-13T14:28:22.573 INFO:tasks.rados.rados.0.smithi151.stdout:1798: writing smithi15186934-7 from 1902211 to 2450121 tid 2 2024-09-13T14:28:22.580 INFO:tasks.rados.rados.0.smithi151.stdout:1798: writing smithi15186934-7 from 3004442 to 3580023 tid 3 2024-09-13T14:28:22.580 INFO:tasks.rados.rados.0.smithi151.stdout:1799: snap_create 2024-09-13T14:28:22.580 INFO:tasks.rados.rados.0.smithi151.stdout:1798: finishing write tid 1 to smithi15186934-7 2024-09-13T14:28:22.580 INFO:tasks.rados.rados.0.smithi151.stdout:1798: finishing write tid 2 to smithi15186934-7 2024-09-13T14:28:22.583 INFO:tasks.rados.rados.0.smithi151.stdout:1798: finishing write tid 3 to smithi15186934-7 2024-09-13T14:28:22.583 INFO:tasks.rados.rados.0.smithi151.stdout:1798: finishing write tid 4 to smithi15186934-7 2024-09-13T14:28:22.583 INFO:tasks.rados.rados.0.smithi151.stdout:1798: finishing write tid 5 to smithi15186934-7 2024-09-13T14:28:22.583 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 1182 (ObjNum 689 snap 224 seq_num 689) dirty exists 2024-09-13T14:28:22.583 INFO:tasks.rados.rados.0.smithi151.stdout:1798: left oid 7 (ObjNum 689 snap 224 seq_num 689) 2024-09-13T14:28:22.608 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:22 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0[45764]: 2024-09-13T14:28:22.164+0000 7f90670aa640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:28:22.608 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:22 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0[45764]: 2024-09-13T14:28:22.164+0000 7f90670aa640 -1 osd.0 511 *** Got signal Terminated *** 2024-09-13T14:28:22.608 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:22 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0[45764]: 2024-09-13T14:28:22.164+0000 7f90670aa640 -1 osd.0 511 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T14:28:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:22 smithi184 ceph-mon[92527]: pgmap v384: 89 pgs: 89 active+clean; 99 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:28:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:22 smithi184 ceph-mon[92527]: osd.0 marked itself down and dead 2024-09-13T14:28:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:22 smithi184 ceph-mon[92527]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:28:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:22 smithi184 ceph-mon[92527]: osdmap e512: 8 total, 7 up, 8 in 2024-09-13T14:28:23.108 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:22 smithi151 podman[118099]: 2024-09-13 14:28:22.901467971 +0000 UTC m=+0.860792858 container died e6951a1a86d93ae49f39cdf372a01a358cd45d8ea4928fa057188d9cb74480fc (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0, RELEASE=quincy-bbdadfa, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731) 2024-09-13T14:28:23.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:22 smithi151 ceph-mon[104450]: pgmap v384: 89 pgs: 89 active+clean; 99 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:28:23.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:22 smithi151 ceph-mon[104450]: osd.0 marked itself down and dead 2024-09-13T14:28:23.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:22 smithi151 ceph-mon[104450]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:28:23.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:22 smithi151 ceph-mon[104450]: osdmap e512: 8 total, 7 up, 8 in 2024-09-13T14:28:23.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:22 smithi151 ceph-mon[108966]: pgmap v384: 89 pgs: 89 active+clean; 99 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:28:23.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:22 smithi151 ceph-mon[108966]: osd.0 marked itself down and dead 2024-09-13T14:28:23.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:22 smithi151 ceph-mon[108966]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:28:23.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:22 smithi151 ceph-mon[108966]: osdmap e512: 8 total, 7 up, 8 in 2024-09-13T14:28:23.254 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:28:23.449 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:23 smithi151 podman[118099]: 2024-09-13 14:28:23.238694912 +0000 UTC m=+1.198019790 container cleanup e6951a1a86d93ae49f39cdf372a01a358cd45d8ea4928fa057188d9cb74480fc (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-bbdadfa, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-13T14:28:23.449 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:23 smithi151 bash[118099]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0 2024-09-13T14:28:23.540 INFO:tasks.rados.rados.0.smithi151.stdout:1796: done (3 left) 2024-09-13T14:28:23.540 INFO:tasks.rados.rados.0.smithi151.stdout:1798: done (2 left) 2024-09-13T14:28:23.540 INFO:tasks.rados.rados.0.smithi151.stdout:1799: done (1 left) 2024-09-13T14:28:23.540 INFO:tasks.rados.rados.0.smithi151.stdout:1800: rollback oid 37 current snap is 225 2024-09-13T14:28:23.540 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 37 to 224 2024-09-13T14:28:23.540 INFO:tasks.rados.rados.0.smithi151.stdout:1801: write oid 21 current snap is 225 2024-09-13T14:28:23.541 INFO:tasks.rados.rados.0.smithi151.stdout:1801: seq_num 690 ranges {796830=793580,2360693=691568} 2024-09-13T14:28:23.550 INFO:tasks.rados.rados.0.smithi151.stdout:1801: writing smithi15186934-21 from 796830 to 1590410 tid 1 2024-09-13T14:28:23.561 INFO:tasks.rados.rados.0.smithi151.stdout:1801: writing smithi15186934-21 from 2360693 to 3052261 tid 2 2024-09-13T14:28:23.561 INFO:tasks.rados.rados.0.smithi151.stdout:1802: write oid 32 current snap is 225 2024-09-13T14:28:23.561 INFO:tasks.rados.rados.0.smithi151.stdout:1800: finishing rollback tid 0 to smithi15186934-37 2024-09-13T14:28:23.561 INFO:tasks.rados.rados.0.smithi151.stdout:1802: seq_num 691 ranges {413637=468829,1619175=435226,2542935=758255} 2024-09-13T14:28:23.566 INFO:tasks.rados.rados.0.smithi151.stdout:1802: writing smithi15186934-32 from 413637 to 882466 tid 1 2024-09-13T14:28:23.573 INFO:tasks.rados.rados.0.smithi151.stdout:1802: writing smithi15186934-32 from 1619175 to 2054401 tid 2 2024-09-13T14:28:23.580 INFO:tasks.rados.rados.0.smithi151.stdout:1802: writing smithi15186934-32 from 2542935 to 3301190 tid 3 2024-09-13T14:28:23.580 INFO:tasks.rados.rados.0.smithi151.stdout:1803: write oid 30 current snap is 225 2024-09-13T14:28:23.580 INFO:tasks.rados.rados.0.smithi151.stdout:1800: finishing rollback tid 1 to smithi15186934-37 2024-09-13T14:28:23.580 INFO:tasks.rados.rados.0.smithi151.stdout:1800: finishing rollback tid 2 to smithi15186934-37 2024-09-13T14:28:23.581 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 37 v 807 (ObjNum 683 snap 222 seq_num 683) dirty exists 2024-09-13T14:28:23.581 INFO:tasks.rados.rados.0.smithi151.stdout:1797: finishing write tid 1 to smithi15186934-48 2024-09-13T14:28:23.581 INFO:tasks.rados.rados.0.smithi151.stdout:1801: finishing write tid 1 to smithi15186934-21 2024-09-13T14:28:23.581 INFO:tasks.rados.rados.0.smithi151.stdout:1797: finishing write tid 2 to smithi15186934-48 2024-09-13T14:28:23.581 INFO:tasks.rados.rados.0.smithi151.stdout:1797: finishing write tid 3 to smithi15186934-48 2024-09-13T14:28:23.581 INFO:tasks.rados.rados.0.smithi151.stdout:1797: finishing write tid 4 to smithi15186934-48 2024-09-13T14:28:23.581 INFO:tasks.rados.rados.0.smithi151.stdout:1803: seq_num 692 ranges {430444=660431,1677657=695237,2970358=569527} 2024-09-13T14:28:23.586 INFO:tasks.rados.rados.0.smithi151.stdout:1803: writing smithi15186934-30 from 430444 to 1090875 tid 1 2024-09-13T14:28:23.593 INFO:tasks.rados.rados.0.smithi151.stdout:1803: writing smithi15186934-30 from 1677657 to 2372894 tid 2 2024-09-13T14:28:23.600 INFO:tasks.rados.rados.0.smithi151.stdout:1803: writing smithi15186934-30 from 2970358 to 3539885 tid 3 2024-09-13T14:28:23.600 INFO:tasks.rados.rados.0.smithi151.stdout:1800: done (4 left) 2024-09-13T14:28:23.600 INFO:tasks.rados.rados.0.smithi151.stdout:1804: write oid 2 current snap is 225 2024-09-13T14:28:23.600 INFO:tasks.rados.rados.0.smithi151.stdout:1804: seq_num 693 ranges {447251=735680,1736139=438895,2597781=780799} 2024-09-13T14:28:23.606 INFO:tasks.rados.rados.0.smithi151.stdout:1804: writing smithi15186934-2 from 447251 to 1182931 tid 1 2024-09-13T14:28:23.611 INFO:tasks.rados.rados.0.smithi151.stdout:1804: writing smithi15186934-2 from 1736139 to 2175034 tid 2 2024-09-13T14:28:23.619 INFO:tasks.rados.rados.0.smithi151.stdout:1804: writing smithi15186934-2 from 2597781 to 3378580 tid 3 2024-09-13T14:28:23.619 INFO:tasks.rados.rados.0.smithi151.stdout:1805: snap_create 2024-09-13T14:28:23.619 INFO:tasks.rados.rados.0.smithi151.stdout:1797: finishing write tid 5 to smithi15186934-48 2024-09-13T14:28:23.619 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 48 v 975 (ObjNum 688 snap 224 seq_num 688) dirty exists 2024-09-13T14:28:23.620 INFO:tasks.rados.rados.0.smithi151.stdout:1797: left oid 48 (ObjNum 688 snap 224 seq_num 688) 2024-09-13T14:28:23.620 INFO:tasks.rados.rados.0.smithi151.stdout:1801: finishing write tid 2 to smithi15186934-21 2024-09-13T14:28:23.620 INFO:tasks.rados.rados.0.smithi151.stdout:1801: finishing write tid 3 to smithi15186934-21 2024-09-13T14:28:23.620 INFO:tasks.rados.rados.0.smithi151.stdout:1801: finishing write tid 4 to smithi15186934-21 2024-09-13T14:28:23.620 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 21 v 393 (ObjNum 690 snap 225 seq_num 690) dirty exists 2024-09-13T14:28:23.620 INFO:tasks.rados.rados.0.smithi151.stdout:1801: left oid 21 (ObjNum 690 snap 225 seq_num 690) 2024-09-13T14:28:23.620 INFO:tasks.rados.rados.0.smithi151.stdout:1802: finishing write tid 1 to smithi15186934-32 2024-09-13T14:28:23.620 INFO:tasks.rados.rados.0.smithi151.stdout:1802: finishing write tid 2 to smithi15186934-32 2024-09-13T14:28:23.620 INFO:tasks.rados.rados.0.smithi151.stdout:1802: finishing write tid 3 to smithi15186934-32 2024-09-13T14:28:23.620 INFO:tasks.rados.rados.0.smithi151.stdout:1802: finishing write tid 4 to smithi15186934-32 2024-09-13T14:28:23.620 INFO:tasks.rados.rados.0.smithi151.stdout:1802: finishing write tid 5 to smithi15186934-32 2024-09-13T14:28:23.620 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 32 v 718 (ObjNum 691 snap 225 seq_num 691) dirty exists 2024-09-13T14:28:23.620 INFO:tasks.rados.rados.0.smithi151.stdout:1802: left oid 32 (ObjNum 691 snap 225 seq_num 691) 2024-09-13T14:28:23.621 INFO:tasks.rados.rados.0.smithi151.stdout:1803: finishing write tid 1 to smithi15186934-30 2024-09-13T14:28:23.621 INFO:tasks.rados.rados.0.smithi151.stdout:1803: finishing write tid 2 to smithi15186934-30 2024-09-13T14:28:23.621 INFO:tasks.rados.rados.0.smithi151.stdout:1803: finishing write tid 3 to smithi15186934-30 2024-09-13T14:28:23.621 INFO:tasks.rados.rados.0.smithi151.stdout:1803: finishing write tid 4 to smithi15186934-30 2024-09-13T14:28:23.621 INFO:tasks.rados.rados.0.smithi151.stdout:1803: finishing write tid 5 to smithi15186934-30 2024-09-13T14:28:23.621 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 30 v 650 (ObjNum 692 snap 225 seq_num 692) dirty exists 2024-09-13T14:28:23.621 INFO:tasks.rados.rados.0.smithi151.stdout:1803: left oid 30 (ObjNum 692 snap 225 seq_num 692) 2024-09-13T14:28:23.621 INFO:tasks.rados.rados.0.smithi151.stdout:1804: finishing write tid 1 to smithi15186934-2 2024-09-13T14:28:23.621 INFO:tasks.rados.rados.0.smithi151.stdout:1804: finishing write tid 2 to smithi15186934-2 2024-09-13T14:28:23.622 INFO:tasks.rados.rados.0.smithi151.stdout:1804: finishing write tid 3 to smithi15186934-2 2024-09-13T14:28:23.622 INFO:tasks.rados.rados.0.smithi151.stdout:1804: finishing write tid 4 to smithi15186934-2 2024-09-13T14:28:23.622 INFO:tasks.rados.rados.0.smithi151.stdout:1804: finishing write tid 5 to smithi15186934-2 2024-09-13T14:28:23.622 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 2 v 663 (ObjNum 693 snap 225 seq_num 693) dirty exists 2024-09-13T14:28:23.622 INFO:tasks.rados.rados.0.smithi151.stdout:1804: left oid 2 (ObjNum 693 snap 225 seq_num 693) 2024-09-13T14:28:23.785 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:23 smithi151 podman[118141]: 2024-09-13 14:28:23.448654896 +0000 UTC m=+0.551206329 container remove e6951a1a86d93ae49f39cdf372a01a358cd45d8ea4928fa057188d9cb74480fc (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-bbdadfa, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-13T14:28:24.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:23 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2972593070' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:24.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:23 smithi184 ceph-mon[92527]: osdmap e513: 8 total, 7 up, 8 in 2024-09-13T14:28:24.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:23 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2972593070' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:24.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:23 smithi151 ceph-mon[104450]: osdmap e513: 8 total, 7 up, 8 in 2024-09-13T14:28:24.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:23 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2972593070' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:24.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:23 smithi151 ceph-mon[108966]: osdmap e513: 8 total, 7 up, 8 in 2024-09-13T14:28:24.109 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:23 smithi151 podman[118286]: 2024-09-13 14:28:23.762078153 +0000 UTC m=+0.015962149 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:28:24.109 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:23 smithi151 podman[118286]: 2024-09-13 14:28:23.883316162 +0000 UTC m=+0.137200191 container create 37586973cdfdd0b534fe43932eb06be2ca6efb5b2c7d77ba8b83851f5b2c26ba (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-13T14:28:24.505 INFO:tasks.rados.rados.0.smithi151.stdout:1797: done (5 left) 2024-09-13T14:28:24.505 INFO:tasks.rados.rados.0.smithi151.stdout:1801: done (4 left) 2024-09-13T14:28:24.505 INFO:tasks.rados.rados.0.smithi151.stdout:1802: done (3 left) 2024-09-13T14:28:24.505 INFO:tasks.rados.rados.0.smithi151.stdout:1803: done (2 left) 2024-09-13T14:28:24.505 INFO:tasks.rados.rados.0.smithi151.stdout:1804: done (1 left) 2024-09-13T14:28:24.505 INFO:tasks.rados.rados.0.smithi151.stdout:1805: done (0 left) 2024-09-13T14:28:24.505 INFO:tasks.rados.rados.0.smithi151.stdout:1806: snap_create 2024-09-13T14:28:24.590 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:24 smithi151 podman[118286]: 2024-09-13 14:28:24.137048934 +0000 UTC m=+0.390932924 container init 37586973cdfdd0b534fe43932eb06be2ca6efb5b2c7d77ba8b83851f5b2c26ba (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-13T14:28:24.591 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:24 smithi151 podman[118286]: 2024-09-13 14:28:24.142229615 +0000 UTC m=+0.396113606 container start 37586973cdfdd0b534fe43932eb06be2ca6efb5b2c7d77ba8b83851f5b2c26ba (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-09-13T14:28:24.591 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:24 smithi151 podman[118286]: 2024-09-13 14:28:24.227803379 +0000 UTC m=+0.481687383 container attach 37586973cdfdd0b534fe43932eb06be2ca6efb5b2c7d77ba8b83851f5b2c26ba (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-13T14:28:24.591 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:24 smithi151 podman[118329]: 2024-09-13 14:28:24.366592319 +0000 UTC m=+0.017047964 container died 37586973cdfdd0b534fe43932eb06be2ca6efb5b2c7d77ba8b83851f5b2c26ba (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True) 2024-09-13T14:28:24.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:24 smithi151 ceph-mon[104450]: pgmap v387: 89 pgs: 13 stale+active+clean, 76 active+clean; 111 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 2.2 MiB/s wr, 18 op/s 2024-09-13T14:28:24.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:24 smithi151 ceph-mon[104450]: from='client.44198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:24.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:24 smithi151 ceph-mon[104450]: from='client.34320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:24.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:24 smithi151 ceph-mon[104450]: osdmap e514: 8 total, 7 up, 8 in 2024-09-13T14:28:24.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:24 smithi151 ceph-mon[108966]: pgmap v387: 89 pgs: 13 stale+active+clean, 76 active+clean; 111 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 2.2 MiB/s wr, 18 op/s 2024-09-13T14:28:24.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:24 smithi151 ceph-mon[108966]: from='client.44198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:24.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:24 smithi151 ceph-mon[108966]: from='client.34320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:24.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:24 smithi151 ceph-mon[108966]: osdmap e514: 8 total, 7 up, 8 in 2024-09-13T14:28:24.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:28:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:28:24] "GET /metrics HTTP/1.1" 200 36400 "" "Prometheus/2.51.0" 2024-09-13T14:28:25.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:24 smithi184 ceph-mon[92527]: pgmap v387: 89 pgs: 13 stale+active+clean, 76 active+clean; 111 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 2.2 MiB/s wr, 18 op/s 2024-09-13T14:28:25.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:24 smithi184 ceph-mon[92527]: from='client.44198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:25.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:24 smithi184 ceph-mon[92527]: from='client.34320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:25.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:24 smithi184 ceph-mon[92527]: osdmap e514: 8 total, 7 up, 8 in 2024-09-13T14:28:25.359 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:25 smithi151 podman[118329]: 2024-09-13 14:28:25.040192049 +0000 UTC m=+0.690647683 container remove 37586973cdfdd0b534fe43932eb06be2ca6efb5b2c7d77ba8b83851f5b2c26ba (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0-deactivate, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-09-13T14:28:25.359 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:25 smithi151 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.0.service: Deactivated successfully. 2024-09-13T14:28:25.359 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:25 smithi151 systemd[1]: Stopped Ceph osd.0 for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:28:25.359 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:25 smithi151 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.0.service: Consumed 12.590s CPU time. 2024-09-13T14:28:25.359 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:25 smithi151 systemd[1]: Starting Ceph osd.0 for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:28:25.506 INFO:tasks.rados.rados.0.smithi151.stdout:1806: done (0 left) 2024-09-13T14:28:25.506 INFO:tasks.rados.rados.0.smithi151.stdout:1807: read oid 48 snap 225 2024-09-13T14:28:25.506 INFO:tasks.rados.rados.0.smithi151.stdout:1807: expect (ObjNum 688 snap 224 seq_num 688) 2024-09-13T14:28:25.506 INFO:tasks.rados.rados.0.smithi151.stdout:1808: read oid 5 snap 226 2024-09-13T14:28:25.506 INFO:tasks.rados.rados.0.smithi151.stdout:1808: expect (ObjNum 687 snap 223 seq_num 687) 2024-09-13T14:28:25.506 INFO:tasks.rados.rados.0.smithi151.stdout:1809: snap_create 2024-09-13T14:28:25.522 INFO:tasks.rados.rados.0.smithi151.stdout:1807: expect (ObjNum 688 snap 224 seq_num 688) 2024-09-13T14:28:25.645 INFO:tasks.rados.rados.0.smithi151.stdout:1808: expect (ObjNum 687 snap 223 seq_num 687) 2024-09-13T14:28:25.802 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:25 smithi151 podman[118433]: 2024-09-13 14:28:25.378566387 +0000 UTC m=+0.015320321 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:28:25.803 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:25 smithi151 podman[118433]: 2024-09-13 14:28:25.482873506 +0000 UTC m=+0.119627430 container create 7000b2856eb915d2ecf6421309caeec60b2dd11261987c601bc4c64d0030350e (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0-activate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-09-13T14:28:25.803 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:25 smithi151 podman[118433]: 2024-09-13 14:28:25.710665379 +0000 UTC m=+0.347419304 container init 7000b2856eb915d2ecf6421309caeec60b2dd11261987c601bc4c64d0030350e (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0-activate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True) 2024-09-13T14:28:25.803 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:25 smithi151 podman[118433]: 2024-09-13 14:28:25.717013206 +0000 UTC m=+0.353767139 container start 7000b2856eb915d2ecf6421309caeec60b2dd11261987c601bc4c64d0030350e (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0-activate, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-13T14:28:25.803 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:25 smithi151 podman[118433]: 2024-09-13 14:28:25.80218504 +0000 UTC m=+0.438938973 container attach 7000b2856eb915d2ecf6421309caeec60b2dd11261987c601bc4c64d0030350e (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0-activate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:28:26.109 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:25 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0-activate[118451]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:28:26.109 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:25 smithi151 bash[118433]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:28:26.109 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:25 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0-activate[118451]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:28:26.109 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:25 smithi151 bash[118433]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:28:26.109 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:25 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0-activate[118451]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:28:26.109 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:25 smithi151 bash[118433]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:28:26.109 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:25 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0-activate[118451]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:28:26.109 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:25 smithi151 bash[118433]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:28:26.109 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:25 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0-activate[118451]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:28:26.109 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:25 smithi151 bash[118433]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:28:26.110 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:25 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0-activate[118451]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T14:28:26.110 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:25 smithi151 bash[118433]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T14:28:26.110 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:25 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0-activate[118451]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2024-09-13T14:28:26.110 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:25 smithi151 bash[118433]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2024-09-13T14:28:26.137 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:28:26.507 INFO:tasks.rados.rados.0.smithi151.stdout:1807: done (2 left) 2024-09-13T14:28:26.508 INFO:tasks.rados.rados.0.smithi151.stdout:1808: done (1 left) 2024-09-13T14:28:26.508 INFO:tasks.rados.rados.0.smithi151.stdout:1809: done (0 left) 2024-09-13T14:28:26.508 INFO:tasks.rados.rados.0.smithi151.stdout:1810: snap_create 2024-09-13T14:28:26.508 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:26 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0-activate[118451]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-13T14:28:26.508 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:26 smithi151 bash[118433]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-13T14:28:26.508 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:26 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0-activate[118451]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2024-09-13T14:28:26.509 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:26 smithi151 bash[118433]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2024-09-13T14:28:26.509 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:26 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0-activate[118451]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-13T14:28:26.509 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:26 smithi151 bash[118433]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-13T14:28:26.509 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:26 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0-activate[118451]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T14:28:26.509 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:26 smithi151 bash[118433]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T14:28:26.509 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:26 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0-activate[118451]: --> ceph-volume lvm activate successful for osd ID: 0 2024-09-13T14:28:26.509 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:26 smithi151 bash[118433]: --> ceph-volume lvm activate successful for osd ID: 0 2024-09-13T14:28:26.509 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:26 smithi151 podman[118433]: 2024-09-13 14:28:26.277894436 +0000 UTC m=+0.914648374 container died 7000b2856eb915d2ecf6421309caeec60b2dd11261987c601bc4c64d0030350e (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0-activate, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-13T14:28:26.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:26 smithi184 ceph-mon[92527]: pgmap v390: 89 pgs: 13 stale+active+clean, 76 active+clean; 111 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 2.2 MiB/s wr, 18 op/s 2024-09-13T14:28:26.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:26 smithi184 ceph-mon[92527]: osdmap e515: 8 total, 7 up, 8 in 2024-09-13T14:28:26.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:26 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2007297032' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:26.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:26 smithi151 ceph-mon[104450]: pgmap v390: 89 pgs: 13 stale+active+clean, 76 active+clean; 111 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 2.2 MiB/s wr, 18 op/s 2024-09-13T14:28:26.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:26 smithi151 ceph-mon[104450]: osdmap e515: 8 total, 7 up, 8 in 2024-09-13T14:28:26.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:26 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2007297032' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:26.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:26 smithi151 ceph-mon[108966]: pgmap v390: 89 pgs: 13 stale+active+clean, 76 active+clean; 111 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 2.2 MiB/s wr, 18 op/s 2024-09-13T14:28:26.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:26 smithi151 ceph-mon[108966]: osdmap e515: 8 total, 7 up, 8 in 2024-09-13T14:28:26.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:26 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2007297032' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:27.322 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:26 smithi151 podman[118659]: 2024-09-13 14:28:26.981378151 +0000 UTC m=+0.697276081 container remove 7000b2856eb915d2ecf6421309caeec60b2dd11261987c601bc4c64d0030350e (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0-activate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730) 2024-09-13T14:28:27.322 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:27 smithi151 podman[118727]: 2024-09-13 14:28:27.088749748 +0000 UTC m=+0.015778086 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:28:27.322 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:27 smithi151 podman[118727]: 2024-09-13 14:28:27.215943199 +0000 UTC m=+0.142971526 container create 018b463f93c20c7f1986053c11b443c104ae16f40983b2abdd8e881257657c94 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-13T14:28:27.518 INFO:tasks.rados.rados.0.smithi151.stdout:1810: done (0 left) 2024-09-13T14:28:27.519 INFO:tasks.rados.rados.0.smithi151.stdout:1811: read oid 31 snap 223 2024-09-13T14:28:27.519 INFO:tasks.rados.rados.0.smithi151.stdout:1811: expect deleted 2024-09-13T14:28:27.519 INFO:tasks.rados.rados.0.smithi151.stdout:1812: rollback oid 3 current snap is 229 2024-09-13T14:28:27.519 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 3 to 219 2024-09-13T14:28:27.519 INFO:tasks.rados.rados.0.smithi151.stdout:1813: rollback oid 37 current snap is 229 2024-09-13T14:28:27.519 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 37 to 227 2024-09-13T14:28:27.519 INFO:tasks.rados.rados.0.smithi151.stdout:1814: write oid 34 current snap is 229 2024-09-13T14:28:27.519 INFO:tasks.rados.rados.0.smithi151.stdout:1814: seq_num 694 ranges {464058=410929,1394621=698906,2625204=592071,3656107=7951} 2024-09-13T14:28:27.523 INFO:tasks.rados.rados.0.smithi151.stdout:1814: writing smithi15186934-34 from 464058 to 874987 tid 1 2024-09-13T14:28:27.530 INFO:tasks.rados.rados.0.smithi151.stdout:1814: writing smithi15186934-34 from 1394621 to 2093527 tid 2 2024-09-13T14:28:27.536 INFO:tasks.rados.rados.0.smithi151.stdout:1814: writing smithi15186934-34 from 2625204 to 3217275 tid 3 2024-09-13T14:28:27.539 INFO:tasks.rados.rados.0.smithi151.stdout:1814: writing smithi15186934-34 from 3656107 to 3664058 tid 4 2024-09-13T14:28:27.539 INFO:tasks.rados.rados.0.smithi151.stdout:1815: snap_create 2024-09-13T14:28:27.539 INFO:tasks.rados.rados.0.smithi151.stdout:1812: finishing rollback tid 1 to smithi15186934-3 2024-09-13T14:28:27.539 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 0 (ObjNum 129 snap 0 seq_num 3092828576) dirty dne 2024-09-13T14:28:27.539 INFO:tasks.rados.rados.0.smithi151.stdout:1813: finishing rollback tid 0 to smithi15186934-37 2024-09-13T14:28:27.540 INFO:tasks.rados.rados.0.smithi151.stdout:1813: finishing rollback tid 1 to smithi15186934-37 2024-09-13T14:28:27.540 INFO:tasks.rados.rados.0.smithi151.stdout:1813: finishing rollback tid 2 to smithi15186934-37 2024-09-13T14:28:27.540 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 37 v 811 (ObjNum 683 snap 222 seq_num 683) dirty exists 2024-09-13T14:28:27.540 INFO:tasks.rados.rados.0.smithi151.stdout:1814: finishing write tid 1 to smithi15186934-34 2024-09-13T14:28:27.540 INFO:tasks.rados.rados.0.smithi151.stdout:1814: finishing write tid 2 to smithi15186934-34 2024-09-13T14:28:27.540 INFO:tasks.rados.rados.0.smithi151.stdout:1814: finishing write tid 3 to smithi15186934-34 2024-09-13T14:28:27.541 INFO:tasks.rados.rados.0.smithi151.stdout:1814: finishing write tid 4 to smithi15186934-34 2024-09-13T14:28:27.541 INFO:tasks.rados.rados.0.smithi151.stdout:1814: finishing write tid 5 to smithi15186934-34 2024-09-13T14:28:27.541 INFO:tasks.rados.rados.0.smithi151.stdout:1814: finishing write tid 6 to smithi15186934-34 2024-09-13T14:28:27.541 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 657 (ObjNum 694 snap 229 seq_num 694) dirty exists 2024-09-13T14:28:27.541 INFO:tasks.rados.rados.0.smithi151.stdout:1814: left oid 34 (ObjNum 694 snap 229 seq_num 694) 2024-09-13T14:28:27.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:27 smithi184 ceph-mon[92527]: from='client.44216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:27.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:27 smithi184 ceph-mon[92527]: from='client.34338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:27.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:27 smithi184 ceph-mon[92527]: osdmap e516: 8 total, 7 up, 8 in 2024-09-13T14:28:27.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:27 smithi151 ceph-mon[108966]: from='client.44216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:27.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:27 smithi151 ceph-mon[108966]: from='client.34338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:27.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:27 smithi151 ceph-mon[108966]: osdmap e516: 8 total, 7 up, 8 in 2024-09-13T14:28:27.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:27 smithi151 ceph-mon[104450]: from='client.44216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:27.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:27 smithi151 ceph-mon[104450]: from='client.34338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:27.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:27 smithi151 ceph-mon[104450]: osdmap e516: 8 total, 7 up, 8 in 2024-09-13T14:28:27.859 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:27 smithi151 podman[118727]: 2024-09-13 14:28:27.418420341 +0000 UTC m=+0.345448665 container init 018b463f93c20c7f1986053c11b443c104ae16f40983b2abdd8e881257657c94 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-13T14:28:27.859 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:27 smithi151 podman[118727]: 2024-09-13 14:28:27.423402018 +0000 UTC m=+0.350430345 container start 018b463f93c20c7f1986053c11b443c104ae16f40983b2abdd8e881257657c94 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-13T14:28:27.859 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:27 smithi151 ceph-osd[118765]: -- 172.21.15.151:0/1387211656 <== mon.2 v2:172.21.15.151:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555e47413a40 con 0x555e481e0400 2024-09-13T14:28:27.859 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:27 smithi151 bash[118727]: 018b463f93c20c7f1986053c11b443c104ae16f40983b2abdd8e881257657c94 2024-09-13T14:28:27.860 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:27 smithi151 systemd[1]: Started Ceph osd.0 for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:28:28.520 INFO:tasks.rados.rados.0.smithi151.stdout:1811: done (4 left) 2024-09-13T14:28:28.520 INFO:tasks.rados.rados.0.smithi151.stdout:1812: done (3 left) 2024-09-13T14:28:28.520 INFO:tasks.rados.rados.0.smithi151.stdout:1813: done (2 left) 2024-09-13T14:28:28.520 INFO:tasks.rados.rados.0.smithi151.stdout:1814: done (1 left) 2024-09-13T14:28:28.520 INFO:tasks.rados.rados.0.smithi151.stdout:1815: done (0 left) 2024-09-13T14:28:28.520 INFO:tasks.rados.rados.0.smithi151.stdout:1816: rollback oid 49 current snap is 230 2024-09-13T14:28:28.520 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 49 to 223 2024-09-13T14:28:28.520 INFO:tasks.rados.rados.0.smithi151.stdout:1817: snap_create 2024-09-13T14:28:28.525 INFO:tasks.rados.rados.0.smithi151.stdout:1816: finishing rollback tid 0 to smithi15186934-49 2024-09-13T14:28:28.525 INFO:tasks.rados.rados.0.smithi151.stdout:1816: finishing rollback tid 1 to smithi15186934-49 2024-09-13T14:28:28.525 INFO:tasks.rados.rados.0.smithi151.stdout:1816: finishing rollback tid 2 to smithi15186934-49 2024-09-13T14:28:28.525 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 49 v 815 (ObjNum 635 snap 206 seq_num 635) dirty exists 2024-09-13T14:28:28.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:28 smithi184 ceph-mon[92527]: pgmap v393: 89 pgs: 19 active+undersized, 15 active+undersized+degraded, 55 active+clean; 126 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.6 MiB/s rd, 2.5 MiB/s wr, 13 op/s; 36/234 objects degraded (15.385%) 2024-09-13T14:28:28.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:28 smithi184 ceph-mon[92527]: Health check failed: Degraded data redundancy: 36/234 objects degraded (15.385%), 15 pgs degraded (PG_DEGRADED) 2024-09-13T14:28:28.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:28 smithi184 ceph-mon[92527]: osdmap e517: 8 total, 7 up, 8 in 2024-09-13T14:28:28.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:28 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:28.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:28 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:28.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:28 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:28:28.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:28 smithi151 ceph-mon[104450]: pgmap v393: 89 pgs: 19 active+undersized, 15 active+undersized+degraded, 55 active+clean; 126 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.6 MiB/s rd, 2.5 MiB/s wr, 13 op/s; 36/234 objects degraded (15.385%) 2024-09-13T14:28:28.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:28 smithi151 ceph-mon[104450]: Health check failed: Degraded data redundancy: 36/234 objects degraded (15.385%), 15 pgs degraded (PG_DEGRADED) 2024-09-13T14:28:28.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:28 smithi151 ceph-mon[104450]: osdmap e517: 8 total, 7 up, 8 in 2024-09-13T14:28:28.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:28 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:28.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:28 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:28.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:28 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:28:28.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:28 smithi151 ceph-mon[108966]: pgmap v393: 89 pgs: 19 active+undersized, 15 active+undersized+degraded, 55 active+clean; 126 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.6 MiB/s rd, 2.5 MiB/s wr, 13 op/s; 36/234 objects degraded (15.385%) 2024-09-13T14:28:28.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:28 smithi151 ceph-mon[108966]: Health check failed: Degraded data redundancy: 36/234 objects degraded (15.385%), 15 pgs degraded (PG_DEGRADED) 2024-09-13T14:28:28.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:28 smithi151 ceph-mon[108966]: osdmap e517: 8 total, 7 up, 8 in 2024-09-13T14:28:28.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:28 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:28.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:28 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:28.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:28 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:28:29.012 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:28:29.523 INFO:tasks.rados.rados.0.smithi151.stdout:1816: done (1 left) 2024-09-13T14:28:29.523 INFO:tasks.rados.rados.0.smithi151.stdout:1817: done (0 left) 2024-09-13T14:28:29.524 INFO:tasks.rados.rados.0.smithi151.stdout:1818: delete oid 18 current snap is 231 2024-09-13T14:28:29.547 INFO:tasks.rados.rados.0.smithi151.stdout:1818: done (0 left) 2024-09-13T14:28:29.547 INFO:tasks.rados.rados.0.smithi151.stdout:1819: write oid 38 current snap is 231 2024-09-13T14:28:29.547 INFO:tasks.rados.rados.0.smithi151.stdout:1819: seq_num 696 ranges {497672=561427,1795232=702575,2963697=730968} 2024-09-13T14:28:29.548 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:29 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0[118742]: 2024-09-13T14:28:29.327+0000 7f18ba3aa640 -1 Falling back to public interface 2024-09-13T14:28:29.553 INFO:tasks.rados.rados.0.smithi151.stdout:1819: writing smithi15186934-38 from 497672 to 1059099 tid 1 2024-09-13T14:28:29.560 INFO:tasks.rados.rados.0.smithi151.stdout:1819: writing smithi15186934-38 from 1795232 to 2497807 tid 2 2024-09-13T14:28:29.568 INFO:tasks.rados.rados.0.smithi151.stdout:1819: writing smithi15186934-38 from 2963697 to 3694665 tid 3 2024-09-13T14:28:29.568 INFO:tasks.rados.rados.0.smithi151.stdout:1820: snap_create 2024-09-13T14:28:29.568 INFO:tasks.rados.rados.0.smithi151.stdout:1819: finishing write tid 1 to smithi15186934-38 2024-09-13T14:28:29.568 INFO:tasks.rados.rados.0.smithi151.stdout:1819: finishing write tid 2 to smithi15186934-38 2024-09-13T14:28:29.571 INFO:tasks.rados.rados.0.smithi151.stdout:1819: finishing write tid 3 to smithi15186934-38 2024-09-13T14:28:29.571 INFO:tasks.rados.rados.0.smithi151.stdout:1819: finishing write tid 4 to smithi15186934-38 2024-09-13T14:28:29.572 INFO:tasks.rados.rados.0.smithi151.stdout:1819: finishing write tid 5 to smithi15186934-38 2024-09-13T14:28:29.572 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 603 (ObjNum 696 snap 231 seq_num 696) dirty exists 2024-09-13T14:28:29.572 INFO:tasks.rados.rados.0.smithi151.stdout:1819: left oid 38 (ObjNum 696 snap 231 seq_num 696) 2024-09-13T14:28:29.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:29 smithi184 ceph-mon[92527]: osdmap e518: 8 total, 7 up, 8 in 2024-09-13T14:28:29.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:29 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2405730379' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:29.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:29 smithi151 ceph-mon[104450]: osdmap e518: 8 total, 7 up, 8 in 2024-09-13T14:28:29.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:29 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2405730379' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:29.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:29 smithi151 ceph-mon[108966]: osdmap e518: 8 total, 7 up, 8 in 2024-09-13T14:28:29.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:29 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2405730379' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:30.550 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:28:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:28:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:30.559 INFO:tasks.rados.rados.0.smithi151.stdout:1819: done (1 left) 2024-09-13T14:28:30.559 INFO:tasks.rados.rados.0.smithi151.stdout:1820: done (0 left) 2024-09-13T14:28:30.559 INFO:tasks.rados.rados.0.smithi151.stdout:1821: snap_remove snap 226 2024-09-13T14:28:30.816 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:30 smithi151 ceph-mon[104450]: from='client.34353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:30.817 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:30 smithi151 ceph-mon[104450]: pgmap v396: 89 pgs: 19 active+undersized, 15 active+undersized+degraded, 55 active+clean; 126 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.6 MiB/s rd, 2.5 MiB/s wr, 13 op/s; 36/234 objects degraded (15.385%) 2024-09-13T14:28:30.817 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:30 smithi151 ceph-mon[104450]: from='client.34359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:30.817 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:30 smithi151 ceph-mon[104450]: osdmap e519: 8 total, 7 up, 8 in 2024-09-13T14:28:30.817 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:30 smithi151 ceph-mon[108966]: from='client.34353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:30.817 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:30 smithi151 ceph-mon[108966]: pgmap v396: 89 pgs: 19 active+undersized, 15 active+undersized+degraded, 55 active+clean; 126 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.6 MiB/s rd, 2.5 MiB/s wr, 13 op/s; 36/234 objects degraded (15.385%) 2024-09-13T14:28:30.817 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:30 smithi151 ceph-mon[108966]: from='client.34359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:30.817 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:30 smithi151 ceph-mon[108966]: osdmap e519: 8 total, 7 up, 8 in 2024-09-13T14:28:30.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:30 smithi184 ceph-mon[92527]: from='client.34353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:30.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:30 smithi184 ceph-mon[92527]: pgmap v396: 89 pgs: 19 active+undersized, 15 active+undersized+degraded, 55 active+clean; 126 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.6 MiB/s rd, 2.5 MiB/s wr, 13 op/s; 36/234 objects degraded (15.385%) 2024-09-13T14:28:30.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:30 smithi184 ceph-mon[92527]: from='client.34359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:30.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:30 smithi184 ceph-mon[92527]: osdmap e519: 8 total, 7 up, 8 in 2024-09-13T14:28:31.566 INFO:tasks.rados.rados.0.smithi151.stdout:1821: done (0 left) 2024-09-13T14:28:31.566 INFO:tasks.rados.rados.0.smithi151.stdout:1822: read oid 4 snap -1 2024-09-13T14:28:31.566 INFO:tasks.rados.rados.0.smithi151.stdout:1822: expect (ObjNum 592 snap 191 seq_num 592) 2024-09-13T14:28:31.566 INFO:tasks.rados.rados.0.smithi151.stdout:1823: snap_remove snap 222 2024-09-13T14:28:31.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:31 smithi184 ceph-mon[92527]: osdmap e520: 8 total, 7 up, 8 in 2024-09-13T14:28:31.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:31.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:31.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:31.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:31.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:31 smithi151 ceph-mon[104450]: osdmap e520: 8 total, 7 up, 8 in 2024-09-13T14:28:31.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:31.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:31.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:31.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:31.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:31 smithi151 ceph-mon[108966]: osdmap e520: 8 total, 7 up, 8 in 2024-09-13T14:28:31.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:31.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:31.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:31.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:31.930 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:28:32.567 INFO:tasks.rados.rados.0.smithi151.stdout:1823: done (1 left) 2024-09-13T14:28:32.567 INFO:tasks.rados.rados.0.smithi151.stdout:1824: delete oid 17 current snap is 232 2024-09-13T14:28:32.567 INFO:tasks.rados.rados.0.smithi151.stdout:1822: expect (ObjNum 592 snap 191 seq_num 592) 2024-09-13T14:28:32.649 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:32 smithi151 ceph-mon[104450]: pgmap v399: 89 pgs: 19 active+undersized, 15 active+undersized+degraded, 55 active+clean; 126 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 36/234 objects degraded (15.385%) 2024-09-13T14:28:32.649 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:32 smithi151 ceph-mon[104450]: osdmap e521: 8 total, 7 up, 8 in 2024-09-13T14:28:32.649 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:32 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/4066387839' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:32.649 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:32 smithi151 ceph-mon[104450]: from='client.44255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:32.649 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:32.649 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:32.649 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:32 smithi151 ceph-mon[108966]: pgmap v399: 89 pgs: 19 active+undersized, 15 active+undersized+degraded, 55 active+clean; 126 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 36/234 objects degraded (15.385%) 2024-09-13T14:28:32.649 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:32 smithi151 ceph-mon[108966]: osdmap e521: 8 total, 7 up, 8 in 2024-09-13T14:28:32.649 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:32 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/4066387839' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:32.650 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:32 smithi151 ceph-mon[108966]: from='client.44255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:32.650 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:32.650 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:32.749 INFO:tasks.rados.rados.0.smithi151.stdout:1822: done (1 left) 2024-09-13T14:28:32.749 INFO:tasks.rados.rados.0.smithi151.stdout:1824: done (0 left) 2024-09-13T14:28:32.749 INFO:tasks.rados.rados.0.smithi151.stdout:1825: read oid 11 snap -1 2024-09-13T14:28:32.749 INFO:tasks.rados.rados.0.smithi151.stdout:1825: expect deleted 2024-09-13T14:28:32.749 INFO:tasks.rados.rados.0.smithi151.stdout:1826: rollback oid 39 current snap is 232 2024-09-13T14:28:32.749 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 39 to 218 2024-09-13T14:28:32.749 INFO:tasks.rados.rados.0.smithi151.stdout:1827: write oid 36 current snap is 232 2024-09-13T14:28:32.750 INFO:tasks.rados.rados.0.smithi151.stdout:1827: seq_num 698 ranges {531286=711925,1912196=706244,3018543=712743} 2024-09-13T14:28:32.755 INFO:tasks.rados.rados.0.smithi151.stdout:1827: writing smithi15186934-36 from 531286 to 1243211 tid 1 2024-09-13T14:28:32.763 INFO:tasks.rados.rados.0.smithi151.stdout:1827: writing smithi15186934-36 from 1912196 to 2618440 tid 2 2024-09-13T14:28:32.769 INFO:tasks.rados.rados.0.smithi151.stdout:1827: writing smithi15186934-36 from 3018543 to 3731286 tid 3 2024-09-13T14:28:32.769 INFO:tasks.rados.rados.0.smithi151.stdout:1828: write oid 9 current snap is 232 2024-09-13T14:28:32.769 INFO:tasks.rados.rados.0.smithi151.stdout:1828: seq_num 699 ranges {548093=787174,1970678=449902,3045966=681137} 2024-09-13T14:28:32.777 INFO:tasks.rados.rados.0.smithi151.stdout:1828: writing smithi15186934-9 from 548093 to 1335267 tid 1 2024-09-13T14:28:32.783 INFO:tasks.rados.rados.0.smithi151.stdout:1828: writing smithi15186934-9 from 1970678 to 2420580 tid 2 2024-09-13T14:28:32.790 INFO:tasks.rados.rados.0.smithi151.stdout:1828: writing smithi15186934-9 from 3045966 to 3727103 tid 3 2024-09-13T14:28:32.790 INFO:tasks.rados.rados.0.smithi151.stdout:1829: read oid 20 snap -1 2024-09-13T14:28:32.790 INFO:tasks.rados.rados.0.smithi151.stdout:1829: expect deleted 2024-09-13T14:28:32.791 INFO:tasks.rados.rados.0.smithi151.stdout:1830: snap_remove snap 228 2024-09-13T14:28:32.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:32 smithi184 ceph-mon[92527]: pgmap v399: 89 pgs: 19 active+undersized, 15 active+undersized+degraded, 55 active+clean; 126 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 36/234 objects degraded (15.385%) 2024-09-13T14:28:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:32 smithi184 ceph-mon[92527]: osdmap e521: 8 total, 7 up, 8 in 2024-09-13T14:28:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:32 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/4066387839' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:32 smithi184 ceph-mon[92527]: from='client.44255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:33.243 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:32 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0[118742]: 2024-09-13T14:28:32.997+0000 7f18ba3aa640 -1 osd.0 511 log_to_monitors true 2024-09-13T14:28:33.580 INFO:tasks.rados.rados.0.smithi151.stdout:1830: done (5 left) 2024-09-13T14:28:33.580 INFO:tasks.rados.rados.0.smithi151.stdout:1831: write oid 44 current snap is 232 2024-09-13T14:28:33.580 INFO:tasks.rados.rados.0.smithi151.stdout:1831: seq_num 700 ranges {564900=578776,1629160=709913,3073389=492409} 2024-09-13T14:28:33.586 INFO:tasks.rados.rados.0.smithi151.stdout:1831: writing smithi15186934-44 from 564900 to 1143676 tid 1 2024-09-13T14:28:33.593 INFO:tasks.rados.rados.0.smithi151.stdout:1831: writing smithi15186934-44 from 1629160 to 2339073 tid 2 2024-09-13T14:28:33.601 INFO:tasks.rados.rados.0.smithi151.stdout:1831: writing smithi15186934-44 from 3073389 to 3565798 tid 3 2024-09-13T14:28:33.601 INFO:tasks.rados.rados.0.smithi151.stdout:1832: rollback oid 33 current snap is 232 2024-09-13T14:28:33.602 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 33 to 219 2024-09-13T14:28:33.602 INFO:tasks.rados.rados.0.smithi151.stdout:1826: finishing rollback tid 0 to smithi15186934-39 2024-09-13T14:28:33.602 INFO:tasks.rados.rados.0.smithi151.stdout:1826: finishing rollback tid 1 to smithi15186934-39 2024-09-13T14:28:33.602 INFO:tasks.rados.rados.0.smithi151.stdout:1826: finishing rollback tid 2 to smithi15186934-39 2024-09-13T14:28:33.602 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 607 (ObjNum 647 snap 209 seq_num 647) dirty exists 2024-09-13T14:28:33.603 INFO:tasks.rados.rados.0.smithi151.stdout:1827: finishing write tid 1 to smithi15186934-36 2024-09-13T14:28:33.603 INFO:tasks.rados.rados.0.smithi151.stdout:1827: finishing write tid 2 to smithi15186934-36 2024-09-13T14:28:33.603 INFO:tasks.rados.rados.0.smithi151.stdout:1825: done (6 left) 2024-09-13T14:28:33.603 INFO:tasks.rados.rados.0.smithi151.stdout:1826: done (5 left) 2024-09-13T14:28:33.603 INFO:tasks.rados.rados.0.smithi151.stdout:1833: delete oid 8 current snap is 232 2024-09-13T14:28:33.603 INFO:tasks.rados.rados.0.smithi151.stdout:1827: finishing write tid 3 to smithi15186934-36 2024-09-13T14:28:33.604 INFO:tasks.rados.rados.0.smithi151.stdout:1827: finishing write tid 4 to smithi15186934-36 2024-09-13T14:28:33.604 INFO:tasks.rados.rados.0.smithi151.stdout:1827: finishing write tid 5 to smithi15186934-36 2024-09-13T14:28:33.604 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 36 v 397 (ObjNum 698 snap 232 seq_num 698) dirty exists 2024-09-13T14:28:33.604 INFO:tasks.rados.rados.0.smithi151.stdout:1827: left oid 36 (ObjNum 698 snap 232 seq_num 698) 2024-09-13T14:28:33.604 INFO:tasks.rados.rados.0.smithi151.stdout:1828: finishing write tid 1 to smithi15186934-9 2024-09-13T14:28:33.605 INFO:tasks.rados.rados.0.smithi151.stdout:1828: finishing write tid 2 to smithi15186934-9 2024-09-13T14:28:33.605 INFO:tasks.rados.rados.0.smithi151.stdout:1828: finishing write tid 3 to smithi15186934-9 2024-09-13T14:28:33.605 INFO:tasks.rados.rados.0.smithi151.stdout:1828: finishing write tid 4 to smithi15186934-9 2024-09-13T14:28:33.605 INFO:tasks.rados.rados.0.smithi151.stdout:1828: finishing write tid 5 to smithi15186934-9 2024-09-13T14:28:33.605 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 724 (ObjNum 699 snap 232 seq_num 699) dirty exists 2024-09-13T14:28:33.605 INFO:tasks.rados.rados.0.smithi151.stdout:1828: left oid 9 (ObjNum 699 snap 232 seq_num 699) 2024-09-13T14:28:33.606 INFO:tasks.rados.rados.0.smithi151.stdout:1832: finishing rollback tid 1 to smithi15186934-33 2024-09-13T14:28:33.606 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 33 v 0 (ObjNum 33 snap 0 seq_num 3064833400) dirty dne 2024-09-13T14:28:33.607 INFO:tasks.rados.rados.0.smithi151.stdout:1831: finishing write tid 1 to smithi15186934-44 2024-09-13T14:28:33.622 INFO:tasks.rados.rados.0.smithi151.stdout:1831: finishing write tid 2 to smithi15186934-44 2024-09-13T14:28:33.642 INFO:tasks.rados.rados.0.smithi151.stdout:1831: finishing write tid 3 to smithi15186934-44 2024-09-13T14:28:33.642 INFO:tasks.rados.rados.0.smithi151.stdout:1831: finishing write tid 4 to smithi15186934-44 2024-09-13T14:28:33.643 INFO:tasks.rados.rados.0.smithi151.stdout:1827: done (5 left) 2024-09-13T14:28:33.643 INFO:tasks.rados.rados.0.smithi151.stdout:1828: done (4 left) 2024-09-13T14:28:33.643 INFO:tasks.rados.rados.0.smithi151.stdout:1829: done (3 left) 2024-09-13T14:28:33.643 INFO:tasks.rados.rados.0.smithi151.stdout:1832: done (2 left) 2024-09-13T14:28:33.643 INFO:tasks.rados.rados.0.smithi151.stdout:1833: done (1 left) 2024-09-13T14:28:33.643 INFO:tasks.rados.rados.0.smithi151.stdout:1834: write oid 38 current snap is 232 2024-09-13T14:28:33.643 INFO:tasks.rados.rados.0.smithi151.stdout:1834: seq_num 702 ranges {598514=729274,1746124=597229,3128235=514953} 2024-09-13T14:28:33.650 INFO:tasks.rados.rados.0.smithi151.stdout:1834: writing smithi15186934-38 from 598514 to 1327788 tid 1 2024-09-13T14:28:33.656 INFO:tasks.rados.rados.0.smithi151.stdout:1834: writing smithi15186934-38 from 1746124 to 2343353 tid 2 2024-09-13T14:28:33.663 INFO:tasks.rados.rados.0.smithi151.stdout:1834: writing smithi15186934-38 from 3128235 to 3643188 tid 3 2024-09-13T14:28:33.663 INFO:tasks.rados.rados.0.smithi151.stdout:1835: write oid 10 current snap is 232 2024-09-13T14:28:33.663 INFO:tasks.rados.rados.0.smithi151.stdout:1831: finishing write tid 5 to smithi15186934-44 2024-09-13T14:28:33.664 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 547 (ObjNum 700 snap 232 seq_num 700) dirty exists 2024-09-13T14:28:33.664 INFO:tasks.rados.rados.0.smithi151.stdout:1831: left oid 44 (ObjNum 700 snap 232 seq_num 700) 2024-09-13T14:28:33.664 INFO:tasks.rados.rados.0.smithi151.stdout:1835: seq_num 703 ranges {615321=404523,1804606=457240,2755658=726225} 2024-09-13T14:28:33.669 INFO:tasks.rados.rados.0.smithi151.stdout:1835: writing smithi15186934-10 from 615321 to 1019844 tid 1 2024-09-13T14:28:33.676 INFO:tasks.rados.rados.0.smithi151.stdout:1835: writing smithi15186934-10 from 1804606 to 2261846 tid 2 2024-09-13T14:28:33.683 INFO:tasks.rados.rados.0.smithi151.stdout:1835: writing smithi15186934-10 from 2755658 to 3481883 tid 3 2024-09-13T14:28:33.683 INFO:tasks.rados.rados.0.smithi151.stdout:1831: done (2 left) 2024-09-13T14:28:33.683 INFO:tasks.rados.rados.0.smithi151.stdout:1836: rollback oid 14 current snap is 232 2024-09-13T14:28:33.683 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 14 to 221 2024-09-13T14:28:33.683 INFO:tasks.rados.rados.0.smithi151.stdout:1837: rollback oid 22 current snap is 232 2024-09-13T14:28:33.683 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 22 to 224 2024-09-13T14:28:33.683 INFO:tasks.rados.rados.0.smithi151.stdout:1838: snap_create 2024-09-13T14:28:33.684 INFO:tasks.rados.rados.0.smithi151.stdout:1837: finishing rollback tid 1 to smithi15186934-22 2024-09-13T14:28:33.684 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 22 v 0 (ObjNum 33 snap 0 seq_num 3064833400) dirty dne 2024-09-13T14:28:33.720 INFO:tasks.rados.rados.0.smithi151.stdout:1836: finishing rollback tid 0 to smithi15186934-14 2024-09-13T14:28:33.734 INFO:tasks.rados.rados.0.smithi151.stdout:1834: finishing write tid 1 to smithi15186934-38 2024-09-13T14:28:33.735 INFO:tasks.rados.rados.0.smithi151.stdout:1834: finishing write tid 2 to smithi15186934-38 2024-09-13T14:28:33.756 INFO:tasks.rados.rados.0.smithi151.stdout:1836: finishing rollback tid 1 to smithi15186934-14 2024-09-13T14:28:33.756 INFO:tasks.rados.rados.0.smithi151.stdout:1836: finishing rollback tid 2 to smithi15186934-14 2024-09-13T14:28:33.756 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 728 (ObjNum 611 snap 197 seq_num 611) dirty exists 2024-09-13T14:28:33.777 INFO:tasks.rados.rados.0.smithi151.stdout:1835: finishing write tid 1 to smithi15186934-10 2024-09-13T14:28:33.777 INFO:tasks.rados.rados.0.smithi151.stdout:1835: finishing write tid 2 to smithi15186934-10 2024-09-13T14:28:33.777 INFO:tasks.rados.rados.0.smithi151.stdout:1834: finishing write tid 3 to smithi15186934-38 2024-09-13T14:28:33.777 INFO:tasks.rados.rados.0.smithi151.stdout:1835: finishing write tid 3 to smithi15186934-10 2024-09-13T14:28:33.777 INFO:tasks.rados.rados.0.smithi151.stdout:1834: finishing write tid 4 to smithi15186934-38 2024-09-13T14:28:33.778 INFO:tasks.rados.rados.0.smithi151.stdout:1835: finishing write tid 4 to smithi15186934-10 2024-09-13T14:28:33.778 INFO:tasks.rados.rados.0.smithi151.stdout:1834: finishing write tid 5 to smithi15186934-38 2024-09-13T14:28:33.778 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 612 (ObjNum 702 snap 232 seq_num 702) dirty exists 2024-09-13T14:28:33.778 INFO:tasks.rados.rados.0.smithi151.stdout:1834: left oid 38 (ObjNum 702 snap 232 seq_num 702) 2024-09-13T14:28:33.778 INFO:tasks.rados.rados.0.smithi151.stdout:1835: finishing write tid 5 to smithi15186934-10 2024-09-13T14:28:33.778 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 650 (ObjNum 703 snap 232 seq_num 703) dirty exists 2024-09-13T14:28:33.778 INFO:tasks.rados.rados.0.smithi151.stdout:1835: left oid 10 (ObjNum 703 snap 232 seq_num 703) 2024-09-13T14:28:33.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:33 smithi151 ceph-mon[104450]: from='client.44261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:33.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:33 smithi151 ceph-mon[104450]: osdmap e522: 8 total, 7 up, 8 in 2024-09-13T14:28:33.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:33 smithi151 ceph-mon[104450]: from='osd.0 [v2:172.21.15.151:6802/2137494296,v1:172.21.15.151:6803/2137494296]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-13T14:28:33.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:33 smithi151 ceph-mon[104450]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-13T14:28:33.837 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:33 smithi151 ceph-mon[108966]: from='client.44261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:33.837 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:33 smithi151 ceph-mon[108966]: osdmap e522: 8 total, 7 up, 8 in 2024-09-13T14:28:33.837 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:33 smithi151 ceph-mon[108966]: from='osd.0 [v2:172.21.15.151:6802/2137494296,v1:172.21.15.151:6803/2137494296]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-13T14:28:33.837 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:33 smithi151 ceph-mon[108966]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-13T14:28:33.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:33 smithi184 ceph-mon[92527]: from='client.44261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:33.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:33 smithi184 ceph-mon[92527]: osdmap e522: 8 total, 7 up, 8 in 2024-09-13T14:28:33.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:33 smithi184 ceph-mon[92527]: from='osd.0 [v2:172.21.15.151:6802/2137494296,v1:172.21.15.151:6803/2137494296]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-13T14:28:33.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:33 smithi184 ceph-mon[92527]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-13T14:28:34.098 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 14:28:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0[118742]: 2024-09-13T14:28:34.040+0000 7f18ae14c640 -1 osd.0 511 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:28:34.585 INFO:tasks.rados.rados.0.smithi151.stdout:1834: done (4 left) 2024-09-13T14:28:34.585 INFO:tasks.rados.rados.0.smithi151.stdout:1835: done (3 left) 2024-09-13T14:28:34.585 INFO:tasks.rados.rados.0.smithi151.stdout:1836: done (2 left) 2024-09-13T14:28:34.585 INFO:tasks.rados.rados.0.smithi151.stdout:1837: done (1 left) 2024-09-13T14:28:34.585 INFO:tasks.rados.rados.0.smithi151.stdout:1838: done (0 left) 2024-09-13T14:28:34.585 INFO:tasks.rados.rados.0.smithi151.stdout:1839: snap_create 2024-09-13T14:28:34.828 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:28:34.828 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:28:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:28:34] "GET /metrics HTTP/1.1" 200 36523 "" "Prometheus/2.51.0" 2024-09-13T14:28:34.828 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:34 smithi151 ceph-mon[108966]: pgmap v402: 89 pgs: 19 active+undersized, 15 active+undersized+degraded, 55 active+clean; 130 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 905 KiB/s wr, 6 op/s; 39/249 objects degraded (15.663%) 2024-09-13T14:28:34.828 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:34 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 39/249 objects degraded (15.663%), 15 pgs degraded (PG_DEGRADED) 2024-09-13T14:28:34.829 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:34 smithi151 ceph-mon[108966]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T14:28:34.829 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:34 smithi151 ceph-mon[108966]: osdmap e523: 8 total, 7 up, 8 in 2024-09-13T14:28:34.829 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:34 smithi151 ceph-mon[108966]: from='osd.0 [v2:172.21.15.151:6802/2137494296,v1:172.21.15.151:6803/2137494296]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-13T14:28:34.829 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:34 smithi151 ceph-mon[108966]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-13T14:28:34.829 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:34 smithi151 ceph-mon[108966]: from='osd.0 ' entity='osd.0' 2024-09-13T14:28:34.829 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:34 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/312394703' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:34.829 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:34 smithi151 ceph-mon[104450]: pgmap v402: 89 pgs: 19 active+undersized, 15 active+undersized+degraded, 55 active+clean; 130 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 905 KiB/s wr, 6 op/s; 39/249 objects degraded (15.663%) 2024-09-13T14:28:34.829 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:34 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 39/249 objects degraded (15.663%), 15 pgs degraded (PG_DEGRADED) 2024-09-13T14:28:34.829 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:34 smithi151 ceph-mon[104450]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T14:28:34.829 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:34 smithi151 ceph-mon[104450]: osdmap e523: 8 total, 7 up, 8 in 2024-09-13T14:28:34.829 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:34 smithi151 ceph-mon[104450]: from='osd.0 [v2:172.21.15.151:6802/2137494296,v1:172.21.15.151:6803/2137494296]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-13T14:28:34.830 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:34 smithi151 ceph-mon[104450]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-13T14:28:34.830 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:34 smithi151 ceph-mon[104450]: from='osd.0 ' entity='osd.0' 2024-09-13T14:28:34.830 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:34 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/312394703' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:34.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:34 smithi184 ceph-mon[92527]: pgmap v402: 89 pgs: 19 active+undersized, 15 active+undersized+degraded, 55 active+clean; 130 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 905 KiB/s wr, 6 op/s; 39/249 objects degraded (15.663%) 2024-09-13T14:28:34.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:34 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 39/249 objects degraded (15.663%), 15 pgs degraded (PG_DEGRADED) 2024-09-13T14:28:34.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:34 smithi184 ceph-mon[92527]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T14:28:34.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:34 smithi184 ceph-mon[92527]: osdmap e523: 8 total, 7 up, 8 in 2024-09-13T14:28:34.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:34 smithi184 ceph-mon[92527]: from='osd.0 [v2:172.21.15.151:6802/2137494296,v1:172.21.15.151:6803/2137494296]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-13T14:28:34.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:34 smithi184 ceph-mon[92527]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-13T14:28:34.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:34 smithi184 ceph-mon[92527]: from='osd.0 ' entity='osd.0' 2024-09-13T14:28:34.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:34 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/312394703' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:35.595 INFO:tasks.rados.rados.0.smithi151.stdout:1839: done (0 left) 2024-09-13T14:28:35.595 INFO:tasks.rados.rados.0.smithi151.stdout:1840: snap_create 2024-09-13T14:28:35.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:35 smithi184 ceph-mon[92527]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:28:35.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:35 smithi184 ceph-mon[92527]: osd.0 [v2:172.21.15.151:6802/2137494296,v1:172.21.15.151:6803/2137494296] boot 2024-09-13T14:28:35.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:35 smithi184 ceph-mon[92527]: osdmap e524: 8 total, 8 up, 8 in 2024-09-13T14:28:35.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:35 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T14:28:35.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:35 smithi184 ceph-mon[92527]: from='client.54190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:35.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:35 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:35.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:35 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:35.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:35 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:28:35.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:35 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:28:35.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:35 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:35.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:35 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:28:35.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:35 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:28:35.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:35 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:28:35.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:35 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:28:35.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:35 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-13T14:28:35.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:28:35 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:28:35.558+0000 7fc94c153640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-09-13T14:28:35.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:35 smithi151 ceph-mon[108966]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:28:35.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:35 smithi151 ceph-mon[108966]: osd.0 [v2:172.21.15.151:6802/2137494296,v1:172.21.15.151:6803/2137494296] boot 2024-09-13T14:28:35.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:35 smithi151 ceph-mon[108966]: osdmap e524: 8 total, 8 up, 8 in 2024-09-13T14:28:35.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:35 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T14:28:35.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:35 smithi151 ceph-mon[108966]: from='client.54190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:35.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:35 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:35.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:35 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:35.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:35 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:28:35.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:35 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:28:35.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:35 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:35.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:35 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:28:35.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:35 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:28:35.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:35 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:28:35.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:35 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:28:35.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:35 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-13T14:28:35.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:35 smithi151 ceph-mon[104450]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:28:35.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:35 smithi151 ceph-mon[104450]: osd.0 [v2:172.21.15.151:6802/2137494296,v1:172.21.15.151:6803/2137494296] boot 2024-09-13T14:28:35.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:35 smithi151 ceph-mon[104450]: osdmap e524: 8 total, 8 up, 8 in 2024-09-13T14:28:35.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:35 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T14:28:35.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:35 smithi151 ceph-mon[104450]: from='client.54190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:35.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:35 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:35.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:35 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:35.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:35 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:28:35.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:35 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:28:35.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:35 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:35.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:35 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:28:35.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:35 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:28:35.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:35 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:28:35.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:35 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:28:35.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:35 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-13T14:28:36.599 INFO:tasks.rados.rados.0.smithi151.stdout:1840: done (0 left) 2024-09-13T14:28:36.599 INFO:tasks.rados.rados.0.smithi151.stdout:1841: read oid 26 snap 233 2024-09-13T14:28:36.599 INFO:tasks.rados.rados.0.smithi151.stdout:1841: expect (ObjNum 686 snap 223 seq_num 686) 2024-09-13T14:28:36.599 INFO:tasks.rados.rados.0.smithi151.stdout:1842: write oid 30 current snap is 235 2024-09-13T14:28:36.599 INFO:tasks.rados.rados.0.smithi151.stdout:1842: seq_num 704 ranges {632128=479772,1746735=600898,3066728=653850} 2024-09-13T14:28:36.605 INFO:tasks.rados.rados.0.smithi151.stdout:1842: writing smithi15186934-30 from 632128 to 1111900 tid 1 2024-09-13T14:28:36.613 INFO:tasks.rados.rados.0.smithi151.stdout:1842: writing smithi15186934-30 from 1746735 to 2347633 tid 2 2024-09-13T14:28:36.620 INFO:tasks.rados.rados.0.smithi151.stdout:1842: writing smithi15186934-30 from 3066728 to 3720578 tid 3 2024-09-13T14:28:36.620 INFO:tasks.rados.rados.0.smithi151.stdout:1843: snap_create 2024-09-13T14:28:36.621 INFO:tasks.rados.rados.0.smithi151.stdout:1841: expect (ObjNum 686 snap 223 seq_num 686) 2024-09-13T14:28:36.768 INFO:tasks.rados.rados.0.smithi151.stdout:1842: finishing write tid 1 to smithi15186934-30 2024-09-13T14:28:36.768 INFO:tasks.rados.rados.0.smithi151.stdout:1842: finishing write tid 2 to smithi15186934-30 2024-09-13T14:28:36.768 INFO:tasks.rados.rados.0.smithi151.stdout:1842: finishing write tid 3 to smithi15186934-30 2024-09-13T14:28:36.768 INFO:tasks.rados.rados.0.smithi151.stdout:1842: finishing write tid 4 to smithi15186934-30 2024-09-13T14:28:36.768 INFO:tasks.rados.rados.0.smithi151.stdout:1842: finishing write tid 5 to smithi15186934-30 2024-09-13T14:28:36.769 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 30 v 666 (ObjNum 704 snap 235 seq_num 704) dirty exists 2024-09-13T14:28:36.769 INFO:tasks.rados.rados.0.smithi151.stdout:1842: left oid 30 (ObjNum 704 snap 235 seq_num 704) 2024-09-13T14:28:36.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:36 smithi184 ceph-mon[92527]: from='client.44282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:36.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:36 smithi184 ceph-mon[92527]: pgmap v405: 89 pgs: 19 active+undersized, 15 active+undersized+degraded, 55 active+clean; 130 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 905 KiB/s wr, 6 op/s; 39/249 objects degraded (15.663%) 2024-09-13T14:28:36.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:36 smithi184 ceph-mon[92527]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-13T14:28:36.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:36 smithi184 ceph-mon[92527]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-09-13T14:28:36.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:36 smithi184 ceph-mon[92527]: osdmap e525: 8 total, 8 up, 8 in 2024-09-13T14:28:36.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:36 smithi151 ceph-mon[108966]: from='client.44282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:36.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:36 smithi151 ceph-mon[108966]: pgmap v405: 89 pgs: 19 active+undersized, 15 active+undersized+degraded, 55 active+clean; 130 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 905 KiB/s wr, 6 op/s; 39/249 objects degraded (15.663%) 2024-09-13T14:28:36.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:36 smithi151 ceph-mon[108966]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-13T14:28:36.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:36 smithi151 ceph-mon[108966]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-09-13T14:28:36.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:36 smithi151 ceph-mon[108966]: osdmap e525: 8 total, 8 up, 8 in 2024-09-13T14:28:36.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:36 smithi151 ceph-mon[104450]: from='client.44282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:36.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:36 smithi151 ceph-mon[104450]: pgmap v405: 89 pgs: 19 active+undersized, 15 active+undersized+degraded, 55 active+clean; 130 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 905 KiB/s wr, 6 op/s; 39/249 objects degraded (15.663%) 2024-09-13T14:28:36.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:36 smithi151 ceph-mon[104450]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-13T14:28:36.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:36 smithi151 ceph-mon[104450]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-09-13T14:28:36.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:36 smithi151 ceph-mon[104450]: osdmap e525: 8 total, 8 up, 8 in 2024-09-13T14:28:37.610 INFO:tasks.rados.rados.0.smithi151.stdout:1841: done (2 left) 2024-09-13T14:28:37.610 INFO:tasks.rados.rados.0.smithi151.stdout:1842: done (1 left) 2024-09-13T14:28:37.610 INFO:tasks.rados.rados.0.smithi151.stdout:1843: done (0 left) 2024-09-13T14:28:37.610 INFO:tasks.rados.rados.0.smithi151.stdout:1844: delete oid 30 current snap is 236 2024-09-13T14:28:37.614 INFO:tasks.rados.rados.0.smithi151.stdout:1844: done (0 left) 2024-09-13T14:28:37.614 INFO:tasks.rados.rados.0.smithi151.stdout:1845: read oid 24 snap -1 2024-09-13T14:28:37.614 INFO:tasks.rados.rados.0.smithi151.stdout:1845: expect (ObjNum 676 snap 219 seq_num 676) 2024-09-13T14:28:37.614 INFO:tasks.rados.rados.0.smithi151.stdout:1846: read oid 19 snap 229 2024-09-13T14:28:37.614 INFO:tasks.rados.rados.0.smithi151.stdout:1846: expect (ObjNum 323 snap 101 seq_num 323) 2024-09-13T14:28:37.614 INFO:tasks.rados.rados.0.smithi151.stdout:1847: snap_create 2024-09-13T14:28:37.619 INFO:tasks.rados.rados.0.smithi151.stdout:1846: expect (ObjNum 323 snap 101 seq_num 323) 2024-09-13T14:28:37.677 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:28:37.739 INFO:tasks.rados.rados.0.smithi151.stdout:1845: expect (ObjNum 676 snap 219 seq_num 676) 2024-09-13T14:28:37.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:37 smithi151 ceph-mon[104450]: osdmap e526: 8 total, 8 up, 8 in 2024-09-13T14:28:37.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:37 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1035133900' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:37.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:37 smithi151 ceph-mon[108966]: osdmap e526: 8 total, 8 up, 8 in 2024-09-13T14:28:37.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:37 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1035133900' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:38.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:37 smithi184 ceph-mon[92527]: osdmap e526: 8 total, 8 up, 8 in 2024-09-13T14:28:38.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:37 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1035133900' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:38.616 INFO:tasks.rados.rados.0.smithi151.stdout:1845: done (2 left) 2024-09-13T14:28:38.617 INFO:tasks.rados.rados.0.smithi151.stdout:1846: done (1 left) 2024-09-13T14:28:38.617 INFO:tasks.rados.rados.0.smithi151.stdout:1847: done (0 left) 2024-09-13T14:28:38.617 INFO:tasks.rados.rados.0.smithi151.stdout:1848: write oid 21 current snap is 237 2024-09-13T14:28:38.617 INFO:tasks.rados.rados.0.smithi151.stdout:1848: seq_num 706 ranges {665742=630270,1863699=604567,3121574=676394} 2024-09-13T14:28:38.623 INFO:tasks.rados.rados.0.smithi151.stdout:1848: writing smithi15186934-21 from 665742 to 1296012 tid 1 2024-09-13T14:28:38.629 INFO:tasks.rados.rados.0.smithi151.stdout:1848: writing smithi15186934-21 from 1863699 to 2468266 tid 2 2024-09-13T14:28:38.636 INFO:tasks.rados.rados.0.smithi151.stdout:1848: writing smithi15186934-21 from 3121574 to 3797968 tid 3 2024-09-13T14:28:38.637 INFO:tasks.rados.rados.0.smithi151.stdout:1849: write oid 30 current snap is 237 2024-09-13T14:28:38.637 INFO:tasks.rados.rados.0.smithi151.stdout:1849: seq_num 707 ranges {682549=705519,1922181=464578,3148997=487666} 2024-09-13T14:28:38.644 INFO:tasks.rados.rados.0.smithi151.stdout:1849: writing smithi15186934-30 from 682549 to 1388068 tid 1 2024-09-13T14:28:38.650 INFO:tasks.rados.rados.0.smithi151.stdout:1849: writing smithi15186934-30 from 1922181 to 2386759 tid 2 2024-09-13T14:28:38.657 INFO:tasks.rados.rados.0.smithi151.stdout:1849: writing smithi15186934-30 from 3148997 to 3636663 tid 3 2024-09-13T14:28:38.657 INFO:tasks.rados.rados.0.smithi151.stdout:1850: delete oid 4 current snap is 237 2024-09-13T14:28:38.657 INFO:tasks.rados.rados.0.smithi151.stdout:1848: finishing write tid 1 to smithi15186934-21 2024-09-13T14:28:38.657 INFO:tasks.rados.rados.0.smithi151.stdout:1848: finishing write tid 2 to smithi15186934-21 2024-09-13T14:28:38.658 INFO:tasks.rados.rados.0.smithi151.stdout:1848: finishing write tid 3 to smithi15186934-21 2024-09-13T14:28:38.658 INFO:tasks.rados.rados.0.smithi151.stdout:1848: finishing write tid 4 to smithi15186934-21 2024-09-13T14:28:38.658 INFO:tasks.rados.rados.0.smithi151.stdout:1848: finishing write tid 5 to smithi15186934-21 2024-09-13T14:28:38.658 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 21 v 400 (ObjNum 706 snap 237 seq_num 706) dirty exists 2024-09-13T14:28:38.658 INFO:tasks.rados.rados.0.smithi151.stdout:1848: left oid 21 (ObjNum 706 snap 237 seq_num 706) 2024-09-13T14:28:38.658 INFO:tasks.rados.rados.0.smithi151.stdout:1849: finishing write tid 1 to smithi15186934-30 2024-09-13T14:28:38.658 INFO:tasks.rados.rados.0.smithi151.stdout:1849: finishing write tid 2 to smithi15186934-30 2024-09-13T14:28:38.660 INFO:tasks.rados.rados.0.smithi151.stdout:1849: finishing write tid 3 to smithi15186934-30 2024-09-13T14:28:38.660 INFO:tasks.rados.rados.0.smithi151.stdout:1849: finishing write tid 4 to smithi15186934-30 2024-09-13T14:28:38.660 INFO:tasks.rados.rados.0.smithi151.stdout:1849: finishing write tid 5 to smithi15186934-30 2024-09-13T14:28:38.660 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 30 v 672 (ObjNum 707 snap 237 seq_num 707) dirty exists 2024-09-13T14:28:38.660 INFO:tasks.rados.rados.0.smithi151.stdout:1849: left oid 30 (ObjNum 707 snap 237 seq_num 707) 2024-09-13T14:28:38.660 INFO:tasks.rados.rados.0.smithi151.stdout:1848: done (2 left) 2024-09-13T14:28:38.660 INFO:tasks.rados.rados.0.smithi151.stdout:1849: done (1 left) 2024-09-13T14:28:38.660 INFO:tasks.rados.rados.0.smithi151.stdout:1850: done (0 left) 2024-09-13T14:28:38.660 INFO:tasks.rados.rados.0.smithi151.stdout:1851: snap_create 2024-09-13T14:28:38.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:38 smithi151 ceph-mon[108966]: pgmap v408: 89 pgs: 9 active+undersized, 4 active+undersized+degraded, 76 active+clean; 144 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.9 MiB/s rd, 2.6 MiB/s wr, 12 op/s; 11/279 objects degraded (3.943%); 4.4 MiB/s, 2 objects/s recovering 2024-09-13T14:28:38.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:38 smithi151 ceph-mon[108966]: osdmap e527: 8 total, 8 up, 8 in 2024-09-13T14:28:38.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:38 smithi151 ceph-mon[108966]: from='client.34410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:38.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:38 smithi151 ceph-mon[108966]: from='client.34416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:38.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:38 smithi151 ceph-mon[104450]: pgmap v408: 89 pgs: 9 active+undersized, 4 active+undersized+degraded, 76 active+clean; 144 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.9 MiB/s rd, 2.6 MiB/s wr, 12 op/s; 11/279 objects degraded (3.943%); 4.4 MiB/s, 2 objects/s recovering 2024-09-13T14:28:38.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:38 smithi151 ceph-mon[104450]: osdmap e527: 8 total, 8 up, 8 in 2024-09-13T14:28:38.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:38 smithi151 ceph-mon[104450]: from='client.34410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:38.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:38 smithi151 ceph-mon[104450]: from='client.34416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:39.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:38 smithi184 ceph-mon[92527]: pgmap v408: 89 pgs: 9 active+undersized, 4 active+undersized+degraded, 76 active+clean; 144 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.9 MiB/s rd, 2.6 MiB/s wr, 12 op/s; 11/279 objects degraded (3.943%); 4.4 MiB/s, 2 objects/s recovering 2024-09-13T14:28:39.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:38 smithi184 ceph-mon[92527]: osdmap e527: 8 total, 8 up, 8 in 2024-09-13T14:28:39.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:38 smithi184 ceph-mon[92527]: from='client.34410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:39.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:38 smithi184 ceph-mon[92527]: from='client.34416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:39.626 INFO:tasks.rados.rados.0.smithi151.stdout:1851: done (0 left) 2024-09-13T14:28:39.626 INFO:tasks.rados.rados.0.smithi151.stdout:1852: read oid 7 snap -1 2024-09-13T14:28:39.626 INFO:tasks.rados.rados.0.smithi151.stdout:1852: expect (ObjNum 689 snap 224 seq_num 689) 2024-09-13T14:28:39.626 INFO:tasks.rados.rados.0.smithi151.stdout:1853: snap_remove snap 220 2024-09-13T14:28:40.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:39 smithi184 ceph-mon[92527]: osdmap e528: 8 total, 8 up, 8 in 2024-09-13T14:28:40.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:39 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 11/279 objects degraded (3.943%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:28:40.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:39 smithi151 ceph-mon[104450]: osdmap e528: 8 total, 8 up, 8 in 2024-09-13T14:28:40.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:39 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 11/279 objects degraded (3.943%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:28:40.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:39 smithi151 ceph-mon[108966]: osdmap e528: 8 total, 8 up, 8 in 2024-09-13T14:28:40.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:39 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 11/279 objects degraded (3.943%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:28:40.536 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:28:40.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:28:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:28:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:40.634 INFO:tasks.rados.rados.0.smithi151.stdout:1852: expect (ObjNum 689 snap 224 seq_num 689) 2024-09-13T14:28:40.765 INFO:tasks.rados.rados.0.smithi151.stdout:1852: done (1 left) 2024-09-13T14:28:40.765 INFO:tasks.rados.rados.0.smithi151.stdout:1853: done (0 left) 2024-09-13T14:28:40.765 INFO:tasks.rados.rados.0.smithi151.stdout:1854: snap_create 2024-09-13T14:28:40.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:40 smithi151 ceph-mon[104450]: pgmap v411: 89 pgs: 89 active+clean; 144 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 88 KiB/s rd, 2.6 MiB/s wr, 10 op/s; 5.2 MiB/s, 3 objects/s recovering 2024-09-13T14:28:40.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:40 smithi151 ceph-mon[104450]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/279 objects degraded (3.943%), 4 pgs degraded) 2024-09-13T14:28:40.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:40 smithi151 ceph-mon[104450]: Cluster is now healthy 2024-09-13T14:28:40.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:40 smithi151 ceph-mon[104450]: osdmap e529: 8 total, 8 up, 8 in 2024-09-13T14:28:40.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:40 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3662357195' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:40.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:40 smithi151 ceph-mon[108966]: pgmap v411: 89 pgs: 89 active+clean; 144 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 88 KiB/s rd, 2.6 MiB/s wr, 10 op/s; 5.2 MiB/s, 3 objects/s recovering 2024-09-13T14:28:40.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:40 smithi151 ceph-mon[108966]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/279 objects degraded (3.943%), 4 pgs degraded) 2024-09-13T14:28:40.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:40 smithi151 ceph-mon[108966]: Cluster is now healthy 2024-09-13T14:28:40.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:40 smithi151 ceph-mon[108966]: osdmap e529: 8 total, 8 up, 8 in 2024-09-13T14:28:40.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:40 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3662357195' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:41.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:40 smithi184 ceph-mon[92527]: pgmap v411: 89 pgs: 89 active+clean; 144 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 88 KiB/s rd, 2.6 MiB/s wr, 10 op/s; 5.2 MiB/s, 3 objects/s recovering 2024-09-13T14:28:41.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:40 smithi184 ceph-mon[92527]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/279 objects degraded (3.943%), 4 pgs degraded) 2024-09-13T14:28:41.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:40 smithi184 ceph-mon[92527]: Cluster is now healthy 2024-09-13T14:28:41.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:40 smithi184 ceph-mon[92527]: osdmap e529: 8 total, 8 up, 8 in 2024-09-13T14:28:41.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:40 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3662357195' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:41.636 INFO:tasks.rados.rados.0.smithi151.stdout:1854: done (0 left) 2024-09-13T14:28:41.636 INFO:tasks.rados.rados.0.smithi151.stdout:1855: read oid 38 snap -1 2024-09-13T14:28:41.636 INFO:tasks.rados.rados.0.smithi151.stdout:1855: expect (ObjNum 702 snap 232 seq_num 702) 2024-09-13T14:28:41.636 INFO:tasks.rados.rados.0.smithi151.stdout:1856: snap_create 2024-09-13T14:28:41.647 INFO:tasks.rados.rados.0.smithi151.stdout:1855: expect (ObjNum 702 snap 232 seq_num 702) 2024-09-13T14:28:42.005 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:41 smithi184 ceph-mon[92527]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:42.005 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:41 smithi184 ceph-mon[92527]: osdmap e530: 8 total, 8 up, 8 in 2024-09-13T14:28:42.005 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:41 smithi184 ceph-mon[92527]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:42.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:41 smithi151 ceph-mon[108966]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:42.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:41 smithi151 ceph-mon[108966]: osdmap e530: 8 total, 8 up, 8 in 2024-09-13T14:28:42.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:41 smithi151 ceph-mon[108966]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:42.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:41 smithi151 ceph-mon[104450]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:42.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:41 smithi151 ceph-mon[104450]: osdmap e530: 8 total, 8 up, 8 in 2024-09-13T14:28:42.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:41 smithi151 ceph-mon[104450]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:42.636 INFO:tasks.rados.rados.0.smithi151.stdout:1855: done (1 left) 2024-09-13T14:28:42.636 INFO:tasks.rados.rados.0.smithi151.stdout:1856: done (0 left) 2024-09-13T14:28:42.637 INFO:tasks.rados.rados.0.smithi151.stdout:1857: snap_create 2024-09-13T14:28:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:42 smithi184 ceph-mon[92527]: pgmap v414: 89 pgs: 89 active+clean; 144 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 900 KiB/s, 0 objects/s recovering 2024-09-13T14:28:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:42 smithi184 ceph-mon[92527]: osdmap e531: 8 total, 8 up, 8 in 2024-09-13T14:28:43.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:42 smithi151 ceph-mon[104450]: pgmap v414: 89 pgs: 89 active+clean; 144 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 900 KiB/s, 0 objects/s recovering 2024-09-13T14:28:43.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:42 smithi151 ceph-mon[104450]: osdmap e531: 8 total, 8 up, 8 in 2024-09-13T14:28:43.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:42 smithi151 ceph-mon[108966]: pgmap v414: 89 pgs: 89 active+clean; 144 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 900 KiB/s, 0 objects/s recovering 2024-09-13T14:28:43.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:42 smithi151 ceph-mon[108966]: osdmap e531: 8 total, 8 up, 8 in 2024-09-13T14:28:43.415 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:28:43.640 INFO:tasks.rados.rados.0.smithi151.stdout:1857: done (0 left) 2024-09-13T14:28:43.640 INFO:tasks.rados.rados.0.smithi151.stdout:1858: write oid 5 current snap is 241 2024-09-13T14:28:43.640 INFO:tasks.rados.rados.0.smithi151.stdout:1858: seq_num 709 ranges {716163=572370,2039145=468247,3203843=626563} 2024-09-13T14:28:43.647 INFO:tasks.rados.rados.0.smithi151.stdout:1858: writing smithi15186934-5 from 716163 to 1288533 tid 1 2024-09-13T14:28:43.654 INFO:tasks.rados.rados.0.smithi151.stdout:1858: writing smithi15186934-5 from 2039145 to 2507392 tid 2 2024-09-13T14:28:43.662 INFO:tasks.rados.rados.0.smithi151.stdout:1858: writing smithi15186934-5 from 3203843 to 3830406 tid 3 2024-09-13T14:28:43.662 INFO:tasks.rados.rados.0.smithi151.stdout:1859: rollback oid 4 current snap is 241 2024-09-13T14:28:43.662 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 4 to 219 2024-09-13T14:28:43.662 INFO:tasks.rados.rados.0.smithi151.stdout:1858: finishing write tid 1 to smithi15186934-5 2024-09-13T14:28:43.662 INFO:tasks.rados.rados.0.smithi151.stdout:1858: finishing write tid 2 to smithi15186934-5 2024-09-13T14:28:43.662 INFO:tasks.rados.rados.0.smithi151.stdout:1860: snap_create 2024-09-13T14:28:43.664 INFO:tasks.rados.rados.0.smithi151.stdout:1859: finishing rollback tid 1 to smithi15186934-4 2024-09-13T14:28:43.665 INFO:tasks.rados.rados.0.smithi151.stdout:1858: finishing write tid 3 to smithi15186934-5 2024-09-13T14:28:43.665 INFO:tasks.rados.rados.0.smithi151.stdout:1859: finishing rollback tid 2 to smithi15186934-4 2024-09-13T14:28:43.665 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 981 (ObjNum 592 snap 191 seq_num 592) dirty exists 2024-09-13T14:28:43.666 INFO:tasks.rados.rados.0.smithi151.stdout:1858: finishing write tid 4 to smithi15186934-5 2024-09-13T14:28:43.666 INFO:tasks.rados.rados.0.smithi151.stdout:1858: finishing write tid 5 to smithi15186934-5 2024-09-13T14:28:43.666 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 677 (ObjNum 709 snap 241 seq_num 709) dirty exists 2024-09-13T14:28:43.666 INFO:tasks.rados.rados.0.smithi151.stdout:1858: left oid 5 (ObjNum 709 snap 241 seq_num 709) 2024-09-13T14:28:44.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:43 smithi184 ceph-mon[92527]: osdmap e532: 8 total, 8 up, 8 in 2024-09-13T14:28:44.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:43 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3620153535' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:44.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:43 smithi151 ceph-mon[104450]: osdmap e532: 8 total, 8 up, 8 in 2024-09-13T14:28:44.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:43 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3620153535' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:44.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:43 smithi151 ceph-mon[108966]: osdmap e532: 8 total, 8 up, 8 in 2024-09-13T14:28:44.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:43 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3620153535' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:44.511 INFO:tasks.rados.rados.0.smithi151.stdout:1858: done (2 left) 2024-09-13T14:28:44.511 INFO:tasks.rados.rados.0.smithi151.stdout:1859: done (1 left) 2024-09-13T14:28:44.511 INFO:tasks.rados.rados.0.smithi151.stdout:1860: done (0 left) 2024-09-13T14:28:44.511 INFO:tasks.rados.rados.0.smithi151.stdout:1861: read oid 32 snap -1 2024-09-13T14:28:44.511 INFO:tasks.rados.rados.0.smithi151.stdout:1861: expect (ObjNum 691 snap 225 seq_num 691) 2024-09-13T14:28:44.511 INFO:tasks.rados.rados.0.smithi151.stdout:1862: snap_create 2024-09-13T14:28:44.520 INFO:tasks.rados.rados.0.smithi151.stdout:1861: expect (ObjNum 691 snap 225 seq_num 691) 2024-09-13T14:28:44.861 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:28:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:28:44] "GET /metrics HTTP/1.1" 200 36523 "" "Prometheus/2.51.0" 2024-09-13T14:28:44.862 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:44 smithi151 ceph-mon[104450]: pgmap v417: 89 pgs: 89 active+clean; 150 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 872 KiB/s wr, 11 op/s 2024-09-13T14:28:44.862 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:44 smithi151 ceph-mon[104450]: from='client.34446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:44.862 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:44 smithi151 ceph-mon[104450]: osdmap e533: 8 total, 8 up, 8 in 2024-09-13T14:28:44.862 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:44 smithi151 ceph-mon[104450]: from='client.44336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:44.862 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:44 smithi151 ceph-mon[104450]: osdmap e534: 8 total, 8 up, 8 in 2024-09-13T14:28:44.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:44 smithi151 ceph-mon[108966]: pgmap v417: 89 pgs: 89 active+clean; 150 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 872 KiB/s wr, 11 op/s 2024-09-13T14:28:44.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:44 smithi151 ceph-mon[108966]: from='client.34446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:44.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:44 smithi151 ceph-mon[108966]: osdmap e533: 8 total, 8 up, 8 in 2024-09-13T14:28:44.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:44 smithi151 ceph-mon[108966]: from='client.44336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:44.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:44 smithi151 ceph-mon[108966]: osdmap e534: 8 total, 8 up, 8 in 2024-09-13T14:28:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:44 smithi184 ceph-mon[92527]: pgmap v417: 89 pgs: 89 active+clean; 150 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 872 KiB/s wr, 11 op/s 2024-09-13T14:28:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:44 smithi184 ceph-mon[92527]: from='client.34446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:44 smithi184 ceph-mon[92527]: osdmap e533: 8 total, 8 up, 8 in 2024-09-13T14:28:45.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:44 smithi184 ceph-mon[92527]: from='client.44336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:45.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:44 smithi184 ceph-mon[92527]: osdmap e534: 8 total, 8 up, 8 in 2024-09-13T14:28:45.515 INFO:tasks.rados.rados.0.smithi151.stdout:1861: done (1 left) 2024-09-13T14:28:45.515 INFO:tasks.rados.rados.0.smithi151.stdout:1862: done (0 left) 2024-09-13T14:28:45.515 INFO:tasks.rados.rados.0.smithi151.stdout:1863: snap_create 2024-09-13T14:28:46.274 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:28:46.516 INFO:tasks.rados.rados.0.smithi151.stdout:1863: done (0 left) 2024-09-13T14:28:46.516 INFO:tasks.rados.rados.0.smithi151.stdout:1864: write oid 39 current snap is 244 2024-09-13T14:28:46.516 INFO:tasks.rados.rados.0.smithi151.stdout:1864: seq_num 710 ranges {732970=647619,2097627=611905,3231266=437835} 2024-09-13T14:28:46.525 INFO:tasks.rados.rados.0.smithi151.stdout:1864: writing smithi15186934-39 from 732970 to 1380589 tid 1 2024-09-13T14:28:46.534 INFO:tasks.rados.rados.0.smithi151.stdout:1864: writing smithi15186934-39 from 2097627 to 2709532 tid 2 2024-09-13T14:28:46.541 INFO:tasks.rados.rados.0.smithi151.stdout:1864: writing smithi15186934-39 from 3231266 to 3669101 tid 3 2024-09-13T14:28:46.541 INFO:tasks.rados.rados.0.smithi151.stdout:1865: read oid 32 snap -1 2024-09-13T14:28:46.541 INFO:tasks.rados.rados.0.smithi151.stdout:1865: expect (ObjNum 691 snap 225 seq_num 691) 2024-09-13T14:28:46.541 INFO:tasks.rados.rados.0.smithi151.stdout:1866: snap_create 2024-09-13T14:28:46.541 INFO:tasks.rados.rados.0.smithi151.stdout:1864: finishing write tid 1 to smithi15186934-39 2024-09-13T14:28:46.542 INFO:tasks.rados.rados.0.smithi151.stdout:1864: finishing write tid 2 to smithi15186934-39 2024-09-13T14:28:46.544 INFO:tasks.rados.rados.0.smithi151.stdout:1864: finishing write tid 3 to smithi15186934-39 2024-09-13T14:28:46.544 INFO:tasks.rados.rados.0.smithi151.stdout:1864: finishing write tid 4 to smithi15186934-39 2024-09-13T14:28:46.544 INFO:tasks.rados.rados.0.smithi151.stdout:1864: finishing write tid 5 to smithi15186934-39 2024-09-13T14:28:46.545 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 625 (ObjNum 710 snap 244 seq_num 710) dirty exists 2024-09-13T14:28:46.545 INFO:tasks.rados.rados.0.smithi151.stdout:1864: left oid 39 (ObjNum 710 snap 244 seq_num 710) 2024-09-13T14:28:46.547 INFO:tasks.rados.rados.0.smithi151.stdout:1865: expect (ObjNum 691 snap 225 seq_num 691) 2024-09-13T14:28:46.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:46 smithi151 ceph-mon[104450]: pgmap v420: 89 pgs: 89 active+clean; 152 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-09-13T14:28:46.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:46 smithi151 ceph-mon[104450]: osdmap e535: 8 total, 8 up, 8 in 2024-09-13T14:28:46.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:46 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2607017761' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:46.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:46.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:46.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:46.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:46 smithi151 ceph-mon[108966]: pgmap v420: 89 pgs: 89 active+clean; 152 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-09-13T14:28:46.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:46 smithi151 ceph-mon[108966]: osdmap e535: 8 total, 8 up, 8 in 2024-09-13T14:28:46.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:46 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2607017761' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:46.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:46.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:46.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:46.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:46 smithi184 ceph-mon[92527]: pgmap v420: 89 pgs: 89 active+clean; 152 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-09-13T14:28:46.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:46 smithi184 ceph-mon[92527]: osdmap e535: 8 total, 8 up, 8 in 2024-09-13T14:28:46.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:46 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2607017761' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:46.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:46.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:46.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:47.529 INFO:tasks.rados.rados.0.smithi151.stdout:1864: done (2 left) 2024-09-13T14:28:47.529 INFO:tasks.rados.rados.0.smithi151.stdout:1865: done (1 left) 2024-09-13T14:28:47.529 INFO:tasks.rados.rados.0.smithi151.stdout:1866: done (0 left) 2024-09-13T14:28:47.529 INFO:tasks.rados.rados.0.smithi151.stdout:1867: snap_create 2024-09-13T14:28:47.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:47 smithi184 ceph-mon[92527]: from='client.44348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:47.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:47 smithi184 ceph-mon[92527]: osdmap e536: 8 total, 8 up, 8 in 2024-09-13T14:28:47.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:47 smithi184 ceph-mon[92527]: from='client.34470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:47.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:47 smithi151 ceph-mon[104450]: from='client.44348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:47.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:47 smithi151 ceph-mon[104450]: osdmap e536: 8 total, 8 up, 8 in 2024-09-13T14:28:47.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:47 smithi151 ceph-mon[104450]: from='client.34470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:47.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:47 smithi151 ceph-mon[108966]: from='client.44348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:47.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:47 smithi151 ceph-mon[108966]: osdmap e536: 8 total, 8 up, 8 in 2024-09-13T14:28:47.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:47 smithi151 ceph-mon[108966]: from='client.34470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:48.531 INFO:tasks.rados.rados.0.smithi151.stdout:1867: done (0 left) 2024-09-13T14:28:48.531 INFO:tasks.rados.rados.0.smithi151.stdout:1868: read oid 35 snap -1 2024-09-13T14:28:48.531 INFO:tasks.rados.rados.0.smithi151.stdout:1868: expect (ObjNum 680 snap 221 seq_num 680) 2024-09-13T14:28:48.532 INFO:tasks.rados.rados.0.smithi151.stdout:1869: snap_remove snap 245 2024-09-13T14:28:48.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:48 smithi184 ceph-mon[92527]: pgmap v423: 89 pgs: 89 active+clean; 158 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.8 MiB/s rd, 822 KiB/s wr, 7 op/s 2024-09-13T14:28:48.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:48 smithi184 ceph-mon[92527]: osdmap e537: 8 total, 8 up, 8 in 2024-09-13T14:28:48.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:48 smithi151 ceph-mon[104450]: pgmap v423: 89 pgs: 89 active+clean; 158 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.8 MiB/s rd, 822 KiB/s wr, 7 op/s 2024-09-13T14:28:48.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:48 smithi151 ceph-mon[104450]: osdmap e537: 8 total, 8 up, 8 in 2024-09-13T14:28:48.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:48 smithi151 ceph-mon[108966]: pgmap v423: 89 pgs: 89 active+clean; 158 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.8 MiB/s rd, 822 KiB/s wr, 7 op/s 2024-09-13T14:28:48.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:48 smithi151 ceph-mon[108966]: osdmap e537: 8 total, 8 up, 8 in 2024-09-13T14:28:49.150 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:28:49.511 INFO:tasks.rados.rados.0.smithi151.stdout:1869: done (1 left) 2024-09-13T14:28:49.511 INFO:tasks.rados.rados.0.smithi151.stdout:1870: write oid 25 current snap is 246 2024-09-13T14:28:49.511 INFO:tasks.rados.rados.0.smithi151.stdout:1870: seq_num 711 ranges {749777=722868,2156109=471916,3258689=649107} 2024-09-13T14:28:49.519 INFO:tasks.rados.rados.0.smithi151.stdout:1870: writing smithi15186934-25 from 749777 to 1472645 tid 1 2024-09-13T14:28:49.526 INFO:tasks.rados.rados.0.smithi151.stdout:1870: writing smithi15186934-25 from 2156109 to 2628025 tid 2 2024-09-13T14:28:49.533 INFO:tasks.rados.rados.0.smithi151.stdout:1870: writing smithi15186934-25 from 3258689 to 3907796 tid 3 2024-09-13T14:28:49.533 INFO:tasks.rados.rados.0.smithi151.stdout:1871: snap_remove snap 218 2024-09-13T14:28:49.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:49 smithi184 ceph-mon[92527]: osdmap e538: 8 total, 8 up, 8 in 2024-09-13T14:28:49.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:49 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/4158786532' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:49.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:49 smithi184 ceph-mon[92527]: osdmap e539: 8 total, 8 up, 8 in 2024-09-13T14:28:49.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:49 smithi151 ceph-mon[104450]: osdmap e538: 8 total, 8 up, 8 in 2024-09-13T14:28:49.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:49 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/4158786532' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:49.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:49 smithi151 ceph-mon[104450]: osdmap e539: 8 total, 8 up, 8 in 2024-09-13T14:28:49.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:49 smithi151 ceph-mon[108966]: osdmap e538: 8 total, 8 up, 8 in 2024-09-13T14:28:49.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:49 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/4158786532' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:49.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:49 smithi151 ceph-mon[108966]: osdmap e539: 8 total, 8 up, 8 in 2024-09-13T14:28:50.539 INFO:tasks.rados.rados.0.smithi151.stdout:1871: done (2 left) 2024-09-13T14:28:50.540 INFO:tasks.rados.rados.0.smithi151.stdout:1872: write oid 2 current snap is 246 2024-09-13T14:28:50.540 INFO:tasks.rados.rados.0.smithi151.stdout:1872: seq_num 712 ranges {766584=798117,2098238=615574,3169759=460379} 2024-09-13T14:28:50.544 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:28:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:28:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:28:50.549 INFO:tasks.rados.rados.0.smithi151.stdout:1872: writing smithi15186934-2 from 766584 to 1564701 tid 1 2024-09-13T14:28:50.556 INFO:tasks.rados.rados.0.smithi151.stdout:1872: writing smithi15186934-2 from 2098238 to 2713812 tid 2 2024-09-13T14:28:50.561 INFO:tasks.rados.rados.0.smithi151.stdout:1872: writing smithi15186934-2 from 3169759 to 3630138 tid 3 2024-09-13T14:28:50.561 INFO:tasks.rados.rados.0.smithi151.stdout:1873: read oid 6 snap -1 2024-09-13T14:28:50.562 INFO:tasks.rados.rados.0.smithi151.stdout:1873: expect (ObjNum 582 snap 186 seq_num 582) 2024-09-13T14:28:50.562 INFO:tasks.rados.rados.0.smithi151.stdout:1874: snap_remove snap 227 2024-09-13T14:28:50.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:50 smithi184 ceph-mon[92527]: from='client.44366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:50.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:50 smithi184 ceph-mon[92527]: pgmap v426: 89 pgs: 89 active+clean; 158 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.8 MiB/s rd, 415 KiB/s wr, 5 op/s 2024-09-13T14:28:50.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:50 smithi184 ceph-mon[92527]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:50.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:50 smithi151 ceph-mon[104450]: from='client.44366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:50.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:50 smithi151 ceph-mon[104450]: pgmap v426: 89 pgs: 89 active+clean; 158 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.8 MiB/s rd, 415 KiB/s wr, 5 op/s 2024-09-13T14:28:50.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:50 smithi151 ceph-mon[104450]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:50.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:50 smithi151 ceph-mon[108966]: from='client.44366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:50.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:50 smithi151 ceph-mon[108966]: pgmap v426: 89 pgs: 89 active+clean; 158 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.8 MiB/s rd, 415 KiB/s wr, 5 op/s 2024-09-13T14:28:50.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:50 smithi151 ceph-mon[108966]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:51.551 INFO:tasks.rados.rados.0.smithi151.stdout:1874: done (4 left) 2024-09-13T14:28:51.551 INFO:tasks.rados.rados.0.smithi151.stdout:1875: snap_create 2024-09-13T14:28:51.551 INFO:tasks.rados.rados.0.smithi151.stdout:1868: expect (ObjNum 680 snap 221 seq_num 680) 2024-09-13T14:28:51.803 INFO:tasks.rados.rados.0.smithi151.stdout:1870: finishing write tid 1 to smithi15186934-25 2024-09-13T14:28:51.803 INFO:tasks.rados.rados.0.smithi151.stdout:1870: finishing write tid 2 to smithi15186934-25 2024-09-13T14:28:51.803 INFO:tasks.rados.rados.0.smithi151.stdout:1870: finishing write tid 3 to smithi15186934-25 2024-09-13T14:28:51.803 INFO:tasks.rados.rados.0.smithi151.stdout:1870: finishing write tid 4 to smithi15186934-25 2024-09-13T14:28:51.803 INFO:tasks.rados.rados.0.smithi151.stdout:1870: finishing write tid 5 to smithi15186934-25 2024-09-13T14:28:51.803 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 986 (ObjNum 711 snap 246 seq_num 711) dirty exists 2024-09-13T14:28:51.803 INFO:tasks.rados.rados.0.smithi151.stdout:1870: left oid 25 (ObjNum 711 snap 246 seq_num 711) 2024-09-13T14:28:51.803 INFO:tasks.rados.rados.0.smithi151.stdout:1872: finishing write tid 1 to smithi15186934-2 2024-09-13T14:28:51.803 INFO:tasks.rados.rados.0.smithi151.stdout:1872: finishing write tid 2 to smithi15186934-2 2024-09-13T14:28:51.803 INFO:tasks.rados.rados.0.smithi151.stdout:1872: finishing write tid 3 to smithi15186934-2 2024-09-13T14:28:51.803 INFO:tasks.rados.rados.0.smithi151.stdout:1872: finishing write tid 4 to smithi15186934-2 2024-09-13T14:28:51.804 INFO:tasks.rados.rados.0.smithi151.stdout:1872: finishing write tid 5 to smithi15186934-2 2024-09-13T14:28:51.804 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 2 v 686 (ObjNum 712 snap 246 seq_num 712) dirty exists 2024-09-13T14:28:51.804 INFO:tasks.rados.rados.0.smithi151.stdout:1872: left oid 2 (ObjNum 712 snap 246 seq_num 712) 2024-09-13T14:28:51.804 INFO:tasks.rados.rados.0.smithi151.stdout:1873: expect (ObjNum 582 snap 186 seq_num 582) 2024-09-13T14:28:51.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:51 smithi184 ceph-mon[92527]: osdmap e540: 8 total, 8 up, 8 in 2024-09-13T14:28:51.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:51 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-13T14:28:51.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:51 smithi184 ceph-mon[92527]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-13T14:28:51.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:51 smithi184 ceph-mon[92527]: Upgrade: osd.1 is safe to restart 2024-09-13T14:28:51.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:51 smithi151 ceph-mon[104450]: osdmap e540: 8 total, 8 up, 8 in 2024-09-13T14:28:51.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:51 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-13T14:28:51.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:51 smithi151 ceph-mon[104450]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-13T14:28:51.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:51 smithi151 ceph-mon[104450]: Upgrade: osd.1 is safe to restart 2024-09-13T14:28:51.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:51 smithi151 ceph-mon[108966]: osdmap e540: 8 total, 8 up, 8 in 2024-09-13T14:28:51.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:51 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-13T14:28:51.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:51 smithi151 ceph-mon[108966]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-13T14:28:51.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:51 smithi151 ceph-mon[108966]: Upgrade: osd.1 is safe to restart 2024-09-13T14:28:52.013 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:28:52.553 INFO:tasks.rados.rados.0.smithi151.stdout:1868: done (4 left) 2024-09-13T14:28:52.553 INFO:tasks.rados.rados.0.smithi151.stdout:1870: done (3 left) 2024-09-13T14:28:52.553 INFO:tasks.rados.rados.0.smithi151.stdout:1872: done (2 left) 2024-09-13T14:28:52.553 INFO:tasks.rados.rados.0.smithi151.stdout:1873: done (1 left) 2024-09-13T14:28:52.553 INFO:tasks.rados.rados.0.smithi151.stdout:1875: done (0 left) 2024-09-13T14:28:52.553 INFO:tasks.rados.rados.0.smithi151.stdout:1876: rollback oid 36 current snap is 247 2024-09-13T14:28:52.554 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 36 to 239 2024-09-13T14:28:52.554 INFO:tasks.rados.rados.0.smithi151.stdout:1877: read oid 34 snap -1 2024-09-13T14:28:52.554 INFO:tasks.rados.rados.0.smithi151.stdout:1877: expect (ObjNum 694 snap 229 seq_num 694) 2024-09-13T14:28:52.554 INFO:tasks.rados.rados.0.smithi151.stdout:1878: rollback oid 46 current snap is 247 2024-09-13T14:28:52.554 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 46 to 221 2024-09-13T14:28:52.554 INFO:tasks.rados.rados.0.smithi151.stdout:1879: rollback oid 8 current snap is 247 2024-09-13T14:28:52.554 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 8 to 219 2024-09-13T14:28:52.554 INFO:tasks.rados.rados.0.smithi151.stdout:1880: read oid 14 snap -1 2024-09-13T14:28:52.554 INFO:tasks.rados.rados.0.smithi151.stdout:1880: expect (ObjNum 611 snap 197 seq_num 611) 2024-09-13T14:28:52.554 INFO:tasks.rados.rados.0.smithi151.stdout:1881: snap_remove snap 229 2024-09-13T14:28:52.730 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:52 smithi151 ceph-mon[104450]: pgmap v429: 89 pgs: 89 active+clean; 158 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:28:52.754 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:52 smithi151 ceph-mon[104450]: osdmap e541: 8 total, 8 up, 8 in 2024-09-13T14:28:52.754 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:52 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/446420898' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:52.754 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:52 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:52.754 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:52 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-13T14:28:52.754 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:52 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:28:52.755 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:52 smithi151 ceph-mon[108966]: pgmap v429: 89 pgs: 89 active+clean; 158 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:28:52.755 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:52 smithi151 ceph-mon[108966]: osdmap e541: 8 total, 8 up, 8 in 2024-09-13T14:28:52.755 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:52 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/446420898' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:52.755 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:52 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:52.755 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:52 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-13T14:28:52.755 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:52 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:28:52.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:52 smithi184 ceph-mon[92527]: pgmap v429: 89 pgs: 89 active+clean; 158 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:28:52.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:52 smithi184 ceph-mon[92527]: osdmap e541: 8 total, 8 up, 8 in 2024-09-13T14:28:52.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:52 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/446420898' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:52.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:52 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:28:52.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:52 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-13T14:28:52.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:52 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:28:53.555 INFO:tasks.rados.rados.0.smithi151.stdout:1881: done (5 left) 2024-09-13T14:28:53.555 INFO:tasks.rados.rados.0.smithi151.stdout:1882: write oid 6 current snap is 247 2024-09-13T14:28:53.555 INFO:tasks.rados.rados.0.smithi151.stdout:1878: finishing rollback tid 1 to smithi15186934-46 2024-09-13T14:28:53.555 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 46 v 0 (ObjNum 129 snap 0 seq_num 3681427657) dirty dne 2024-09-13T14:28:53.555 INFO:tasks.rados.rados.0.smithi151.stdout:1882: seq_num 713 ranges {783391=473366,1756720=475585,2797182=788004} 2024-09-13T14:28:53.562 INFO:tasks.rados.rados.0.smithi151.stdout:1882: writing smithi15186934-6 from 783391 to 1256757 tid 1 2024-09-13T14:28:53.569 INFO:tasks.rados.rados.0.smithi151.stdout:1882: writing smithi15186934-6 from 1756720 to 2232305 tid 2 2024-09-13T14:28:53.577 INFO:tasks.rados.rados.0.smithi151.stdout:1882: writing smithi15186934-6 from 2797182 to 3585186 tid 3 2024-09-13T14:28:53.578 INFO:tasks.rados.rados.0.smithi151.stdout:1878: done (5 left) 2024-09-13T14:28:53.578 INFO:tasks.rados.rados.0.smithi151.stdout:1883: write oid 9 current snap is 247 2024-09-13T14:28:53.578 INFO:tasks.rados.rados.0.smithi151.stdout:1883: seq_num 714 ranges {400198=548615,1415202=584996} 2024-09-13T14:28:53.582 INFO:tasks.rados.rados.0.smithi151.stdout:1883: writing smithi15186934-9 from 400198 to 948813 tid 1 2024-09-13T14:28:53.589 INFO:tasks.rados.rados.0.smithi151.stdout:1883: writing smithi15186934-9 from 1415202 to 2000198 tid 2 2024-09-13T14:28:53.589 INFO:tasks.rados.rados.0.smithi151.stdout:1884: read oid 29 snap -1 2024-09-13T14:28:53.589 INFO:tasks.rados.rados.0.smithi151.stdout:1884: expect (ObjNum 667 snap 216 seq_num 667) 2024-09-13T14:28:53.589 INFO:tasks.rados.rados.0.smithi151.stdout:1885: write oid 31 current snap is 247 2024-09-13T14:28:53.589 INFO:tasks.rados.rados.0.smithi151.stdout:1879: finishing rollback tid 1 to smithi15186934-8 2024-09-13T14:28:53.589 INFO:tasks.rados.rados.0.smithi151.stdout:1879: finishing rollback tid 2 to smithi15186934-8 2024-09-13T14:28:53.589 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 557 (ObjNum 602 snap 193 seq_num 602) dirty exists 2024-09-13T14:28:53.589 INFO:tasks.rados.rados.0.smithi151.stdout:1880: expect (ObjNum 611 snap 197 seq_num 611) 2024-09-13T14:28:53.701 INFO:tasks.rados.rados.0.smithi151.stdout:1876: finishing rollback tid 0 to smithi15186934-36 2024-09-13T14:28:53.701 INFO:tasks.rados.rados.0.smithi151.stdout:1876: finishing rollback tid 1 to smithi15186934-36 2024-09-13T14:28:53.701 INFO:tasks.rados.rados.0.smithi151.stdout:1876: finishing rollback tid 2 to smithi15186934-36 2024-09-13T14:28:53.701 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 36 v 413 (ObjNum 698 snap 232 seq_num 698) dirty exists 2024-09-13T14:28:53.701 INFO:tasks.rados.rados.0.smithi151.stdout:1877: expect (ObjNum 694 snap 229 seq_num 694) 2024-09-13T14:28:53.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:53 smithi151 ceph-mon[104450]: from='client.44384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:53.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:53 smithi151 ceph-mon[104450]: Upgrade: Updating osd.1 2024-09-13T14:28:53.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:53 smithi151 ceph-mon[104450]: Deploying daemon osd.1 on smithi151 2024-09-13T14:28:53.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:53 smithi151 ceph-mon[104450]: from='client.34506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:53.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:53 smithi151 ceph-mon[104450]: osdmap e542: 8 total, 8 up, 8 in 2024-09-13T14:28:53.806 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:53 smithi151 ceph-mon[108966]: from='client.44384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:53.807 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:53 smithi151 ceph-mon[108966]: Upgrade: Updating osd.1 2024-09-13T14:28:53.807 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:53 smithi151 ceph-mon[108966]: Deploying daemon osd.1 on smithi151 2024-09-13T14:28:53.807 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:53 smithi151 ceph-mon[108966]: from='client.34506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:53.807 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:53 smithi151 ceph-mon[108966]: osdmap e542: 8 total, 8 up, 8 in 2024-09-13T14:28:53.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:53 smithi184 ceph-mon[92527]: from='client.44384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:53.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:53 smithi184 ceph-mon[92527]: Upgrade: Updating osd.1 2024-09-13T14:28:53.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:53 smithi184 ceph-mon[92527]: Deploying daemon osd.1 on smithi151 2024-09-13T14:28:53.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:53 smithi184 ceph-mon[92527]: from='client.34506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:53.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:53 smithi184 ceph-mon[92527]: osdmap e542: 8 total, 8 up, 8 in 2024-09-13T14:28:53.983 INFO:tasks.rados.rados.0.smithi151.stdout:1882: finishing write tid 1 to smithi15186934-6 2024-09-13T14:28:53.984 INFO:tasks.rados.rados.0.smithi151.stdout:1882: finishing write tid 2 to smithi15186934-6 2024-09-13T14:28:53.984 INFO:tasks.rados.rados.0.smithi151.stdout:1882: finishing write tid 3 to smithi15186934-6 2024-09-13T14:28:53.984 INFO:tasks.rados.rados.0.smithi151.stdout:1882: finishing write tid 4 to smithi15186934-6 2024-09-13T14:28:53.984 INFO:tasks.rados.rados.0.smithi151.stdout:1882: finishing write tid 5 to smithi15186934-6 2024-09-13T14:28:53.984 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 6 v 457 (ObjNum 713 snap 247 seq_num 713) dirty exists 2024-09-13T14:28:53.984 INFO:tasks.rados.rados.0.smithi151.stdout:1882: left oid 6 (ObjNum 713 snap 247 seq_num 713) 2024-09-13T14:28:53.984 INFO:tasks.rados.rados.0.smithi151.stdout:1883: finishing write tid 1 to smithi15186934-9 2024-09-13T14:28:53.984 INFO:tasks.rados.rados.0.smithi151.stdout:1885: seq_num 715 ranges {417005=740217,1590037=426968} 2024-09-13T14:28:53.989 INFO:tasks.rados.rados.0.smithi151.stdout:1885: writing smithi15186934-31 from 417005 to 1157222 tid 1 2024-09-13T14:28:53.995 INFO:tasks.rados.rados.0.smithi151.stdout:1885: writing smithi15186934-31 from 1590037 to 2017005 tid 2 2024-09-13T14:28:53.995 INFO:tasks.rados.rados.0.smithi151.stdout:1876: done (7 left) 2024-09-13T14:28:53.995 INFO:tasks.rados.rados.0.smithi151.stdout:1877: done (6 left) 2024-09-13T14:28:53.995 INFO:tasks.rados.rados.0.smithi151.stdout:1879: done (5 left) 2024-09-13T14:28:53.995 INFO:tasks.rados.rados.0.smithi151.stdout:1880: done (4 left) 2024-09-13T14:28:53.995 INFO:tasks.rados.rados.0.smithi151.stdout:1882: done (3 left) 2024-09-13T14:28:53.995 INFO:tasks.rados.rados.0.smithi151.stdout:1886: read oid 46 snap -1 2024-09-13T14:28:53.995 INFO:tasks.rados.rados.0.smithi151.stdout:1886: expect deleted 2024-09-13T14:28:53.995 INFO:tasks.rados.rados.0.smithi151.stdout:1887: snap_remove snap 225 2024-09-13T14:28:54.576 INFO:tasks.rados.rados.0.smithi151.stdout:1887: done (4 left) 2024-09-13T14:28:54.576 INFO:tasks.rados.rados.0.smithi151.stdout:1888: snap_remove snap 244 2024-09-13T14:28:54.827 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:54 smithi151 ceph-mon[104450]: pgmap v432: 89 pgs: 89 active+clean; 158 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.8 MiB/s rd, 450 KiB/s wr, 5 op/s 2024-09-13T14:28:54.827 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:54 smithi151 ceph-mon[104450]: osdmap e543: 8 total, 8 up, 8 in 2024-09-13T14:28:54.827 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:54 smithi151 ceph-mon[108966]: pgmap v432: 89 pgs: 89 active+clean; 158 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.8 MiB/s rd, 450 KiB/s wr, 5 op/s 2024-09-13T14:28:54.827 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:54 smithi151 ceph-mon[108966]: osdmap e543: 8 total, 8 up, 8 in 2024-09-13T14:28:54.828 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:28:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:28:54] "GET /metrics HTTP/1.1" 200 36521 "" "Prometheus/2.51.0" 2024-09-13T14:28:54.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:54 smithi184 ceph-mon[92527]: pgmap v432: 89 pgs: 89 active+clean; 158 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.8 MiB/s rd, 450 KiB/s wr, 5 op/s 2024-09-13T14:28:54.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:54 smithi184 ceph-mon[92527]: osdmap e543: 8 total, 8 up, 8 in 2024-09-13T14:28:54.889 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:28:55.571 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:55 smithi151 systemd[1]: Stopping Ceph osd.1 for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:28:55.572 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:55 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1[52113]: 2024-09-13T14:28:55.432+0000 7f4e88e2d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:28:55.572 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:55 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1[52113]: 2024-09-13T14:28:55.432+0000 7f4e88e2d640 -1 osd.1 544 *** Got signal Terminated *** 2024-09-13T14:28:55.572 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:55 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1[52113]: 2024-09-13T14:28:55.432+0000 7f4e88e2d640 -1 osd.1 544 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T14:28:55.578 INFO:tasks.rados.rados.0.smithi151.stdout:1888: done (4 left) 2024-09-13T14:28:55.578 INFO:tasks.rados.rados.0.smithi151.stdout:1889: snap_remove snap 231 2024-09-13T14:28:55.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:55 smithi184 ceph-mon[92527]: osdmap e544: 8 total, 8 up, 8 in 2024-09-13T14:28:55.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:55 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1701651770' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:55.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:55 smithi184 ceph-mon[92527]: from='client.44402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:55.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:55 smithi184 ceph-mon[92527]: osd.1 marked itself down and dead 2024-09-13T14:28:55.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:55 smithi151 ceph-mon[104450]: osdmap e544: 8 total, 8 up, 8 in 2024-09-13T14:28:55.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:55 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1701651770' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:55.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:55 smithi151 ceph-mon[104450]: from='client.44402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:55.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:55 smithi151 ceph-mon[104450]: osd.1 marked itself down and dead 2024-09-13T14:28:55.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:55 smithi151 ceph-mon[108966]: osdmap e544: 8 total, 8 up, 8 in 2024-09-13T14:28:55.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:55 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1701651770' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:55.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:55 smithi151 ceph-mon[108966]: from='client.44402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:55.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:55 smithi151 ceph-mon[108966]: osd.1 marked itself down and dead 2024-09-13T14:28:56.291 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:56 smithi151 podman[124120]: 2024-09-13 14:28:56.014837865 +0000 UTC m=+0.686469230 container died 94bc726b3eb524dd554a73009e3373b412f8bc3e3c196f9e0ff6b44f65695b9b (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=quincy-bbdadfa, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-13T14:28:56.573 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:56 smithi151 podman[124120]: 2024-09-13 14:28:56.380725006 +0000 UTC m=+1.052356374 container cleanup 94bc726b3eb524dd554a73009e3373b412f8bc3e3c196f9e0ff6b44f65695b9b (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1, RELEASE=quincy-bbdadfa, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-13T14:28:56.573 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:56 smithi151 bash[124120]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1 2024-09-13T14:28:56.583 INFO:tasks.rados.rados.0.smithi151.stdout:1889: done (4 left) 2024-09-13T14:28:56.583 INFO:tasks.rados.rados.0.smithi151.stdout:1890: rollback oid 38 current snap is 247 2024-09-13T14:28:56.583 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 38 to 235 2024-09-13T14:28:56.583 INFO:tasks.rados.rados.0.smithi151.stdout:1883: finishing write tid 2 to smithi15186934-9 2024-09-13T14:28:56.584 INFO:tasks.rados.rados.0.smithi151.stdout:1883: finishing write tid 3 to smithi15186934-9 2024-09-13T14:28:56.584 INFO:tasks.rados.rados.0.smithi151.stdout:1883: finishing write tid 4 to smithi15186934-9 2024-09-13T14:28:56.584 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 744 (ObjNum 714 snap 247 seq_num 714) dirty exists 2024-09-13T14:28:56.584 INFO:tasks.rados.rados.0.smithi151.stdout:1883: left oid 9 (ObjNum 714 snap 247 seq_num 714) 2024-09-13T14:28:56.584 INFO:tasks.rados.rados.0.smithi151.stdout:1884: expect (ObjNum 667 snap 216 seq_num 667) 2024-09-13T14:28:56.695 INFO:tasks.rados.rados.0.smithi151.stdout:1885: finishing write tid 1 to smithi15186934-31 2024-09-13T14:28:56.696 INFO:tasks.rados.rados.0.smithi151.stdout:1883: done (4 left) 2024-09-13T14:28:56.696 INFO:tasks.rados.rados.0.smithi151.stdout:1884: done (3 left) 2024-09-13T14:28:56.696 INFO:tasks.rados.rados.0.smithi151.stdout:1891: rollback oid 20 current snap is 247 2024-09-13T14:28:56.696 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 20 to 240 2024-09-13T14:28:56.696 INFO:tasks.rados.rados.0.smithi151.stdout:1885: finishing write tid 2 to smithi15186934-31 2024-09-13T14:28:56.696 INFO:tasks.rados.rados.0.smithi151.stdout:1885: finishing write tid 3 to smithi15186934-31 2024-09-13T14:28:56.696 INFO:tasks.rados.rados.0.smithi151.stdout:1885: finishing write tid 4 to smithi15186934-31 2024-09-13T14:28:56.696 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 31 v 871 (ObjNum 715 snap 247 seq_num 715) dirty exists 2024-09-13T14:28:56.696 INFO:tasks.rados.rados.0.smithi151.stdout:1885: left oid 31 (ObjNum 715 snap 247 seq_num 715) 2024-09-13T14:28:56.697 INFO:tasks.rados.rados.0.smithi151.stdout:1890: finishing rollback tid 0 to smithi15186934-38 2024-09-13T14:28:56.697 INFO:tasks.rados.rados.0.smithi151.stdout:1890: finishing rollback tid 1 to smithi15186934-38 2024-09-13T14:28:56.697 INFO:tasks.rados.rados.0.smithi151.stdout:1890: finishing rollback tid 2 to smithi15186934-38 2024-09-13T14:28:56.697 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 645 (ObjNum 702 snap 232 seq_num 702) dirty exists 2024-09-13T14:28:56.697 INFO:tasks.rados.rados.0.smithi151.stdout:1885: done (3 left) 2024-09-13T14:28:56.697 INFO:tasks.rados.rados.0.smithi151.stdout:1886: done (2 left) 2024-09-13T14:28:56.697 INFO:tasks.rados.rados.0.smithi151.stdout:1890: done (1 left) 2024-09-13T14:28:56.697 INFO:tasks.rados.rados.0.smithi151.stdout:1892: rollback oid 40 current snap is 247 2024-09-13T14:28:56.697 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 40 to 221 2024-09-13T14:28:56.697 INFO:tasks.rados.rados.0.smithi151.stdout:1893: read oid 2 snap -1 2024-09-13T14:28:56.697 INFO:tasks.rados.rados.0.smithi151.stdout:1893: expect (ObjNum 712 snap 246 seq_num 712) 2024-09-13T14:28:56.697 INFO:tasks.rados.rados.0.smithi151.stdout:1894: read oid 30 snap -1 2024-09-13T14:28:56.697 INFO:tasks.rados.rados.0.smithi151.stdout:1894: expect (ObjNum 707 snap 237 seq_num 707) 2024-09-13T14:28:56.697 INFO:tasks.rados.rados.0.smithi151.stdout:1895: read oid 27 snap -1 2024-09-13T14:28:56.698 INFO:tasks.rados.rados.0.smithi151.stdout:1895: expect deleted 2024-09-13T14:28:56.698 INFO:tasks.rados.rados.0.smithi151.stdout:1896: snap_create 2024-09-13T14:28:56.698 INFO:tasks.rados.rados.0.smithi151.stdout:1891: finishing rollback tid 1 to smithi15186934-20 2024-09-13T14:28:56.698 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 0 (ObjNum 65 snap 0 seq_num 1) dirty dne 2024-09-13T14:28:56.701 INFO:tasks.rados.rados.0.smithi151.stdout:1893: expect (ObjNum 712 snap 246 seq_num 712) 2024-09-13T14:28:56.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:56 smithi184 ceph-mon[92527]: pgmap v435: 89 pgs: 89 active+clean; 159 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.8 MiB/s rd, 908 KiB/s wr, 7 op/s 2024-09-13T14:28:56.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:56 smithi184 ceph-mon[92527]: from='client.44408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:56.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:56 smithi184 ceph-mon[92527]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:28:56.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:56 smithi184 ceph-mon[92527]: osdmap e545: 8 total, 7 up, 8 in 2024-09-13T14:28:56.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:56 smithi151 ceph-mon[104450]: pgmap v435: 89 pgs: 89 active+clean; 159 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.8 MiB/s rd, 908 KiB/s wr, 7 op/s 2024-09-13T14:28:56.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:56 smithi151 ceph-mon[104450]: from='client.44408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:56.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:56 smithi151 ceph-mon[104450]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:28:56.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:56 smithi151 ceph-mon[104450]: osdmap e545: 8 total, 7 up, 8 in 2024-09-13T14:28:56.859 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:56 smithi151 podman[124140]: 2024-09-13 14:28:56.749004425 +0000 UTC m=+0.730776115 container remove 94bc726b3eb524dd554a73009e3373b412f8bc3e3c196f9e0ff6b44f65695b9b (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=quincy-bbdadfa, GIT_CLEAN=True) 2024-09-13T14:28:56.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:56 smithi151 ceph-mon[108966]: pgmap v435: 89 pgs: 89 active+clean; 159 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.8 MiB/s rd, 908 KiB/s wr, 7 op/s 2024-09-13T14:28:56.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:56 smithi151 ceph-mon[108966]: from='client.44408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:56.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:56 smithi151 ceph-mon[108966]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:28:56.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:56 smithi151 ceph-mon[108966]: osdmap e545: 8 total, 7 up, 8 in 2024-09-13T14:28:56.934 INFO:tasks.rados.rados.0.smithi151.stdout:1894: expect (ObjNum 707 snap 237 seq_num 707) 2024-09-13T14:28:57.071 INFO:tasks.rados.rados.0.smithi151.stdout:1892: finishing rollback tid 1 to smithi15186934-40 2024-09-13T14:28:57.071 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 40 v 0 (ObjNum 289 snap 0 seq_num 3680702449) dirty dne 2024-09-13T14:28:57.404 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:57 smithi151 podman[124231]: 2024-09-13 14:28:57.044922144 +0000 UTC m=+0.015966773 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:28:57.404 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:57 smithi151 podman[124231]: 2024-09-13 14:28:57.141907093 +0000 UTC m=+0.112951706 container create 254acdbe9e1d8be5572a65579a64fb6fa67ed8c524776b3e4c685228b70e58d1 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1-deactivate, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True) 2024-09-13T14:28:57.593 INFO:tasks.rados.rados.0.smithi151.stdout:1891: done (5 left) 2024-09-13T14:28:57.593 INFO:tasks.rados.rados.0.smithi151.stdout:1892: done (4 left) 2024-09-13T14:28:57.593 INFO:tasks.rados.rados.0.smithi151.stdout:1893: done (3 left) 2024-09-13T14:28:57.593 INFO:tasks.rados.rados.0.smithi151.stdout:1894: done (2 left) 2024-09-13T14:28:57.593 INFO:tasks.rados.rados.0.smithi151.stdout:1895: done (1 left) 2024-09-13T14:28:57.594 INFO:tasks.rados.rados.0.smithi151.stdout:1896: done (0 left) 2024-09-13T14:28:57.594 INFO:tasks.rados.rados.0.smithi151.stdout:1897: rollback oid 11 current snap is 248 2024-09-13T14:28:57.594 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 11 to 247 2024-09-13T14:28:57.594 INFO:tasks.rados.rados.0.smithi151.stdout:1898: snap_remove snap 237 2024-09-13T14:28:57.782 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:57 smithi151 ceph-mon[104450]: osdmap e546: 8 total, 7 up, 8 in 2024-09-13T14:28:57.782 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:57 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1482933118' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:57.782 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:57 smithi151 ceph-mon[108966]: osdmap e546: 8 total, 7 up, 8 in 2024-09-13T14:28:57.782 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:57 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1482933118' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:57.783 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:57 smithi151 podman[124231]: 2024-09-13 14:28:57.403167738 +0000 UTC m=+0.374212351 container init 254acdbe9e1d8be5572a65579a64fb6fa67ed8c524776b3e4c685228b70e58d1 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=wip-aclamk-cbt-combined-d58f730) 2024-09-13T14:28:57.783 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:57 smithi151 podman[124231]: 2024-09-13 14:28:57.40818136 +0000 UTC m=+0.379225975 container start 254acdbe9e1d8be5572a65579a64fb6fa67ed8c524776b3e4c685228b70e58d1 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-13T14:28:57.783 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:57 smithi151 podman[124231]: 2024-09-13 14:28:57.511537804 +0000 UTC m=+0.482582420 container attach 254acdbe9e1d8be5572a65579a64fb6fa67ed8c524776b3e4c685228b70e58d1 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730) 2024-09-13T14:28:57.783 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:57 smithi151 podman[124231]: 2024-09-13 14:28:57.615694031 +0000 UTC m=+0.586738655 container died 254acdbe9e1d8be5572a65579a64fb6fa67ed8c524776b3e4c685228b70e58d1 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:28:57.783 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:28:57.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:57 smithi184 ceph-mon[92527]: osdmap e546: 8 total, 7 up, 8 in 2024-09-13T14:28:57.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:57 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1482933118' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:28:58.591 INFO:tasks.rados.rados.0.smithi151.stdout:1898: done (1 left) 2024-09-13T14:28:58.591 INFO:tasks.rados.rados.0.smithi151.stdout:1899: write oid 41 current snap is 248 2024-09-13T14:28:58.591 INFO:tasks.rados.rados.0.smithi151.stdout:1897: finishing rollback tid 1 to smithi15186934-11 2024-09-13T14:28:58.591 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 11 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-13T14:28:58.591 INFO:tasks.rados.rados.0.smithi151.stdout:1899: seq_num 716 ranges {433812=415466,1648519=385293} 2024-09-13T14:28:58.596 INFO:tasks.rados.rados.0.smithi151.stdout:1899: writing smithi15186934-41 from 433812 to 849278 tid 1 2024-09-13T14:28:58.600 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:58 smithi151 podman[124320]: 2024-09-13 14:28:58.265927639 +0000 UTC m=+0.644177929 container remove 254acdbe9e1d8be5572a65579a64fb6fa67ed8c524776b3e4c685228b70e58d1 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1-deactivate, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True) 2024-09-13T14:28:58.600 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:58 smithi151 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.1.service: Deactivated successfully. 2024-09-13T14:28:58.600 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:58 smithi151 systemd[1]: Stopped Ceph osd.1 for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:28:58.600 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:58 smithi151 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.1.service: Consumed 11.289s CPU time. 2024-09-13T14:28:58.600 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:58 smithi151 systemd[1]: Starting Ceph osd.1 for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:28:58.604 INFO:tasks.rados.rados.0.smithi151.stdout:1899: writing smithi15186934-41 from 1648519 to 2033812 tid 2 2024-09-13T14:28:58.604 INFO:tasks.rados.rados.0.smithi151.stdout:1897: done (1 left) 2024-09-13T14:28:58.604 INFO:tasks.rados.rados.0.smithi151.stdout:1900: snap_remove snap 247 2024-09-13T14:28:58.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:58 smithi184 ceph-mon[92527]: pgmap v438: 89 pgs: 25 active+undersized, 1 stale+active+clean, 7 active+undersized+degraded, 56 active+clean; 163 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.1 MiB/s wr, 13 op/s; 26/303 objects degraded (8.581%) 2024-09-13T14:28:58.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:58 smithi184 ceph-mon[92527]: Health check failed: Degraded data redundancy: 26/303 objects degraded (8.581%), 7 pgs degraded (PG_DEGRADED) 2024-09-13T14:28:58.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:58 smithi184 ceph-mon[92527]: osdmap e547: 8 total, 7 up, 8 in 2024-09-13T14:28:58.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:58 smithi184 ceph-mon[92527]: from='client.44420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:58.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:58 smithi151 ceph-mon[104450]: pgmap v438: 89 pgs: 25 active+undersized, 1 stale+active+clean, 7 active+undersized+degraded, 56 active+clean; 163 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.1 MiB/s wr, 13 op/s; 26/303 objects degraded (8.581%) 2024-09-13T14:28:58.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:58 smithi151 ceph-mon[104450]: Health check failed: Degraded data redundancy: 26/303 objects degraded (8.581%), 7 pgs degraded (PG_DEGRADED) 2024-09-13T14:28:58.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:58 smithi151 ceph-mon[104450]: osdmap e547: 8 total, 7 up, 8 in 2024-09-13T14:28:58.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:58 smithi151 ceph-mon[104450]: from='client.44420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:58.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:58 smithi151 ceph-mon[108966]: pgmap v438: 89 pgs: 25 active+undersized, 1 stale+active+clean, 7 active+undersized+degraded, 56 active+clean; 163 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.1 MiB/s wr, 13 op/s; 26/303 objects degraded (8.581%) 2024-09-13T14:28:58.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:58 smithi151 ceph-mon[108966]: Health check failed: Degraded data redundancy: 26/303 objects degraded (8.581%), 7 pgs degraded (PG_DEGRADED) 2024-09-13T14:28:58.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:58 smithi151 ceph-mon[108966]: osdmap e547: 8 total, 7 up, 8 in 2024-09-13T14:28:58.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:58 smithi151 ceph-mon[108966]: from='client.44420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:58.859 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:58 smithi151 podman[124451]: 2024-09-13 14:28:58.610008416 +0000 UTC m=+0.023072601 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:28:58.859 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:58 smithi151 podman[124451]: 2024-09-13 14:28:58.733664307 +0000 UTC m=+0.146728487 container create c4a086c8af77953ab663d2c0cd409d01d2ba8981f9b08e92d1ff1ad89c61b832 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1-activate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731) 2024-09-13T14:28:59.128 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:58 smithi151 podman[124451]: 2024-09-13 14:28:58.978171085 +0000 UTC m=+0.391235272 container init c4a086c8af77953ab663d2c0cd409d01d2ba8981f9b08e92d1ff1ad89c61b832 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1-activate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:28:59.129 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:58 smithi151 podman[124451]: 2024-09-13 14:28:58.983390752 +0000 UTC m=+0.396454931 container start c4a086c8af77953ab663d2c0cd409d01d2ba8981f9b08e92d1ff1ad89c61b832 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:28:59.129 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:59 smithi151 podman[124451]: 2024-09-13 14:28:59.069700544 +0000 UTC m=+0.482764726 container attach c4a086c8af77953ab663d2c0cd409d01d2ba8981f9b08e92d1ff1ad89c61b832 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1-activate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-13T14:28:59.129 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:59 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1-activate[124465]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:28:59.129 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:59 smithi151 bash[124451]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:28:59.129 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:59 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1-activate[124465]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:28:59.129 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:59 smithi151 bash[124451]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:28:59.507 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:59 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1-activate[124465]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:28:59.507 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:59 smithi151 bash[124451]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:28:59.508 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:59 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1-activate[124465]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:28:59.508 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:59 smithi151 bash[124451]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:28:59.508 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:59 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1-activate[124465]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:28:59.508 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:59 smithi151 bash[124451]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:28:59.508 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:59 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1-activate[124465]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-13T14:28:59.508 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:59 smithi151 bash[124451]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-13T14:28:59.508 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:59 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1-activate[124465]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2024-09-13T14:28:59.508 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:59 smithi151 bash[124451]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2024-09-13T14:28:59.508 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:59 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1-activate[124465]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-13T14:28:59.508 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:59 smithi151 bash[124451]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-13T14:28:59.517 INFO:tasks.rados.rados.0.smithi151.stdout:1899: finishing write tid 1 to smithi15186934-41 2024-09-13T14:28:59.517 INFO:tasks.rados.rados.0.smithi151.stdout:1899: finishing write tid 2 to smithi15186934-41 2024-09-13T14:28:59.517 INFO:tasks.rados.rados.0.smithi151.stdout:1899: finishing write tid 3 to smithi15186934-41 2024-09-13T14:28:59.517 INFO:tasks.rados.rados.0.smithi151.stdout:1899: finishing write tid 4 to smithi15186934-41 2024-09-13T14:28:59.517 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 88 (ObjNum 716 snap 248 seq_num 716) dirty exists 2024-09-13T14:28:59.517 INFO:tasks.rados.rados.0.smithi151.stdout:1899: left oid 41 (ObjNum 716 snap 248 seq_num 716) 2024-09-13T14:28:59.517 INFO:tasks.rados.rados.0.smithi151.stdout:1899: done (1 left) 2024-09-13T14:28:59.517 INFO:tasks.rados.rados.0.smithi151.stdout:1900: done (0 left) 2024-09-13T14:28:59.518 INFO:tasks.rados.rados.0.smithi151.stdout:1901: delete oid 23 current snap is 248 2024-09-13T14:28:59.522 INFO:tasks.rados.rados.0.smithi151.stdout:1901: done (0 left) 2024-09-13T14:28:59.522 INFO:tasks.rados.rados.0.smithi151.stdout:1902: snap_create 2024-09-13T14:28:59.849 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:59 smithi151 ceph-mon[104450]: from='client.34542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:59.849 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:59 smithi151 ceph-mon[104450]: osdmap e548: 8 total, 7 up, 8 in 2024-09-13T14:28:59.849 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:28:59 smithi151 ceph-mon[104450]: osdmap e549: 8 total, 7 up, 8 in 2024-09-13T14:28:59.849 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:59 smithi151 ceph-mon[108966]: from='client.34542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:59.849 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:59 smithi151 ceph-mon[108966]: osdmap e548: 8 total, 7 up, 8 in 2024-09-13T14:28:59.850 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:28:59 smithi151 ceph-mon[108966]: osdmap e549: 8 total, 7 up, 8 in 2024-09-13T14:28:59.850 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:59 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1-activate[124465]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2024-09-13T14:28:59.850 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:59 smithi151 bash[124451]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2024-09-13T14:28:59.850 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:59 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1-activate[124465]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-13T14:28:59.850 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:59 smithi151 bash[124451]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-13T14:28:59.850 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:59 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1-activate[124465]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-13T14:28:59.850 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:59 smithi151 bash[124451]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-13T14:28:59.850 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:59 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1-activate[124465]: --> ceph-volume lvm activate successful for osd ID: 1 2024-09-13T14:28:59.850 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:59 smithi151 bash[124451]: --> ceph-volume lvm activate successful for osd ID: 1 2024-09-13T14:28:59.850 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:28:59 smithi151 podman[124451]: 2024-09-13 14:28:59.539397564 +0000 UTC m=+0.952461744 container died c4a086c8af77953ab663d2c0cd409d01d2ba8981f9b08e92d1ff1ad89c61b832 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1-activate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-13T14:29:00.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:59 smithi184 ceph-mon[92527]: from='client.34542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:00.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:59 smithi184 ceph-mon[92527]: osdmap e548: 8 total, 7 up, 8 in 2024-09-13T14:29:00.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:28:59 smithi184 ceph-mon[92527]: osdmap e549: 8 total, 7 up, 8 in 2024-09-13T14:29:00.515 INFO:tasks.rados.rados.0.smithi151.stdout:1902: done (0 left) 2024-09-13T14:29:00.515 INFO:tasks.rados.rados.0.smithi151.stdout:1903: write oid 7 current snap is 249 2024-09-13T14:29:00.515 INFO:tasks.rados.rados.0.smithi151.stdout:1903: seq_num 718 ranges {467426=565964,1649130=418296} 2024-09-13T14:29:00.521 INFO:tasks.rados.rados.0.smithi151.stdout:1903: writing smithi15186934-7 from 467426 to 1033390 tid 1 2024-09-13T14:29:00.527 INFO:tasks.rados.rados.0.smithi151.stdout:1903: writing smithi15186934-7 from 1649130 to 2067426 tid 2 2024-09-13T14:29:00.527 INFO:tasks.rados.rados.0.smithi151.stdout:1904: rollback oid 45 current snap is 249 2024-09-13T14:29:00.527 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 45 to 224 2024-09-13T14:29:00.527 INFO:tasks.rados.rados.0.smithi151.stdout:1905: read oid 44 snap -1 2024-09-13T14:29:00.527 INFO:tasks.rados.rados.0.smithi151.stdout:1905: expect (ObjNum 700 snap 232 seq_num 700) 2024-09-13T14:29:00.527 INFO:tasks.rados.rados.0.smithi151.stdout:1906: write oid 9 current snap is 249 2024-09-13T14:29:00.527 INFO:tasks.rados.rados.0.smithi151.stdout:1906: seq_num 719 ranges {484233=641213,1707612=376621} 2024-09-13T14:29:00.534 INFO:tasks.rados.rados.0.smithi151.stdout:1906: writing smithi15186934-9 from 484233 to 1125446 tid 1 2024-09-13T14:29:00.539 INFO:tasks.rados.rados.0.smithi151.stdout:1906: writing smithi15186934-9 from 1707612 to 2084233 tid 2 2024-09-13T14:29:00.539 INFO:tasks.rados.rados.0.smithi151.stdout:1907: write oid 3 current snap is 249 2024-09-13T14:29:00.539 INFO:tasks.rados.rados.0.smithi151.stdout:1907: seq_num 720 ranges {501040=716462,1766094=334946} 2024-09-13T14:29:00.546 INFO:tasks.rados.rados.0.smithi151.stdout:1907: writing smithi15186934-3 from 501040 to 1217502 tid 1 2024-09-13T14:29:00.552 INFO:tasks.rados.rados.0.smithi151.stdout:1907: writing smithi15186934-3 from 1766094 to 2101040 tid 2 2024-09-13T14:29:00.552 INFO:tasks.rados.rados.0.smithi151.stdout:1908: read oid 15 snap -1 2024-09-13T14:29:00.552 INFO:tasks.rados.rados.0.smithi151.stdout:1908: expect deleted 2024-09-13T14:29:00.552 INFO:tasks.rados.rados.0.smithi151.stdout:1909: rollback oid 47 current snap is 249 2024-09-13T14:29:00.552 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 47 to 243 2024-09-13T14:29:00.553 INFO:tasks.rados.rados.0.smithi151.stdout:1903: finishing write tid 1 to smithi15186934-7 2024-09-13T14:29:00.553 INFO:tasks.rados.rados.0.smithi151.stdout:1905: expect (ObjNum 700 snap 232 seq_num 700) 2024-09-13T14:29:00.574 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:29:00 smithi151 podman[124590]: 2024-09-13 14:29:00.249215843 +0000 UTC m=+0.704162694 container remove c4a086c8af77953ab663d2c0cd409d01d2ba8981f9b08e92d1ff1ad89c61b832 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1-activate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=) 2024-09-13T14:29:00.574 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:29:00 smithi151 podman[124676]: 2024-09-13 14:29:00.364178109 +0000 UTC m=+0.018734276 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:29:00.574 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:29:00 smithi151 podman[124676]: 2024-09-13 14:29:00.466776779 +0000 UTC m=+0.121332930 container create 04d393df5f431646e8f0f3b823329442e7889de20762f2dbe043bb127d91b0db (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:29:00.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:29:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:29:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:00.663 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:29:00.679 INFO:tasks.rados.rados.0.smithi151.stdout:1903: finishing write tid 2 to smithi15186934-7 2024-09-13T14:29:00.679 INFO:tasks.rados.rados.0.smithi151.stdout:1903: finishing write tid 3 to smithi15186934-7 2024-09-13T14:29:00.679 INFO:tasks.rados.rados.0.smithi151.stdout:1905: done (6 left) 2024-09-13T14:29:00.679 INFO:tasks.rados.rados.0.smithi151.stdout:1910: write oid 34 current snap is 249 2024-09-13T14:29:00.680 INFO:tasks.rados.rados.0.smithi151.stdout:1903: finishing write tid 4 to smithi15186934-7 2024-09-13T14:29:00.680 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 1208 (ObjNum 718 snap 249 seq_num 718) dirty exists 2024-09-13T14:29:00.680 INFO:tasks.rados.rados.0.smithi151.stdout:1903: left oid 7 (ObjNum 718 snap 249 seq_num 718) 2024-09-13T14:29:00.680 INFO:tasks.rados.rados.0.smithi151.stdout:1904: finishing rollback tid 1 to smithi15186934-45 2024-09-13T14:29:00.680 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-13T14:29:00.680 INFO:tasks.rados.rados.0.smithi151.stdout:1906: finishing write tid 1 to smithi15186934-9 2024-09-13T14:29:00.680 INFO:tasks.rados.rados.0.smithi151.stdout:1910: seq_num 721 ranges {517847=791711,2108223=9624} 2024-09-13T14:29:00.686 INFO:tasks.rados.rados.0.smithi151.stdout:1910: writing smithi15186934-34 from 517847 to 1309558 tid 1 2024-09-13T14:29:00.691 INFO:tasks.rados.rados.0.smithi151.stdout:1910: writing smithi15186934-34 from 2108223 to 2117847 tid 2 2024-09-13T14:29:00.691 INFO:tasks.rados.rados.0.smithi151.stdout:1903: done (6 left) 2024-09-13T14:29:00.691 INFO:tasks.rados.rados.0.smithi151.stdout:1904: done (5 left) 2024-09-13T14:29:00.691 INFO:tasks.rados.rados.0.smithi151.stdout:1911: write oid 2 current snap is 249 2024-09-13T14:29:00.692 INFO:tasks.rados.rados.0.smithi151.stdout:1906: finishing write tid 2 to smithi15186934-9 2024-09-13T14:29:00.692 INFO:tasks.rados.rados.0.smithi151.stdout:1906: finishing write tid 3 to smithi15186934-9 2024-09-13T14:29:00.692 INFO:tasks.rados.rados.0.smithi151.stdout:1911: seq_num 722 ranges {534654=466960,1766705=367949} 2024-09-13T14:29:00.696 INFO:tasks.rados.rados.0.smithi151.stdout:1911: writing smithi15186934-2 from 534654 to 1001614 tid 1 2024-09-13T14:29:00.703 INFO:tasks.rados.rados.0.smithi151.stdout:1911: writing smithi15186934-2 from 1766705 to 2134654 tid 2 2024-09-13T14:29:00.703 INFO:tasks.rados.rados.0.smithi151.stdout:1912: delete oid 17 current snap is 249 2024-09-13T14:29:00.703 INFO:tasks.rados.rados.0.smithi151.stdout:1906: finishing write tid 4 to smithi15186934-9 2024-09-13T14:29:00.703 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 756 (ObjNum 719 snap 249 seq_num 719) dirty exists 2024-09-13T14:29:00.703 INFO:tasks.rados.rados.0.smithi151.stdout:1906: left oid 9 (ObjNum 719 snap 249 seq_num 719) 2024-09-13T14:29:00.703 INFO:tasks.rados.rados.0.smithi151.stdout:1907: finishing write tid 1 to smithi15186934-3 2024-09-13T14:29:00.704 INFO:tasks.rados.rados.0.smithi151.stdout:1907: finishing write tid 2 to smithi15186934-3 2024-09-13T14:29:00.704 INFO:tasks.rados.rados.0.smithi151.stdout:1907: finishing write tid 3 to smithi15186934-3 2024-09-13T14:29:00.704 INFO:tasks.rados.rados.0.smithi151.stdout:1907: finishing write tid 4 to smithi15186934-3 2024-09-13T14:29:00.704 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 694 (ObjNum 720 snap 249 seq_num 720) dirty exists 2024-09-13T14:29:00.704 INFO:tasks.rados.rados.0.smithi151.stdout:1907: left oid 3 (ObjNum 720 snap 249 seq_num 720) 2024-09-13T14:29:00.704 INFO:tasks.rados.rados.0.smithi151.stdout:1909: finishing rollback tid 0 to smithi15186934-47 2024-09-13T14:29:00.704 INFO:tasks.rados.rados.0.smithi151.stdout:1909: finishing rollback tid 1 to smithi15186934-47 2024-09-13T14:29:00.704 INFO:tasks.rados.rados.0.smithi151.stdout:1909: finishing rollback tid 2 to smithi15186934-47 2024-09-13T14:29:00.704 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 851 (ObjNum 659 snap 213 seq_num 659) dirty exists 2024-09-13T14:29:00.704 INFO:tasks.rados.rados.0.smithi151.stdout:1910: finishing write tid 1 to smithi15186934-34 2024-09-13T14:29:00.704 INFO:tasks.rados.rados.0.smithi151.stdout:1910: finishing write tid 2 to smithi15186934-34 2024-09-13T14:29:00.705 INFO:tasks.rados.rados.0.smithi151.stdout:1910: finishing write tid 3 to smithi15186934-34 2024-09-13T14:29:00.705 INFO:tasks.rados.rados.0.smithi151.stdout:1910: finishing write tid 4 to smithi15186934-34 2024-09-13T14:29:00.705 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 698 (ObjNum 721 snap 249 seq_num 721) dirty exists 2024-09-13T14:29:00.705 INFO:tasks.rados.rados.0.smithi151.stdout:1910: left oid 34 (ObjNum 721 snap 249 seq_num 721) 2024-09-13T14:29:00.705 INFO:tasks.rados.rados.0.smithi151.stdout:1911: finishing write tid 1 to smithi15186934-2 2024-09-13T14:29:00.706 INFO:tasks.rados.rados.0.smithi151.stdout:1906: done (6 left) 2024-09-13T14:29:00.706 INFO:tasks.rados.rados.0.smithi151.stdout:1907: done (5 left) 2024-09-13T14:29:00.706 INFO:tasks.rados.rados.0.smithi151.stdout:1908: done (4 left) 2024-09-13T14:29:00.706 INFO:tasks.rados.rados.0.smithi151.stdout:1909: done (3 left) 2024-09-13T14:29:00.706 INFO:tasks.rados.rados.0.smithi151.stdout:1910: done (2 left) 2024-09-13T14:29:00.706 INFO:tasks.rados.rados.0.smithi151.stdout:1912: done (1 left) 2024-09-13T14:29:00.706 INFO:tasks.rados.rados.0.smithi151.stdout:1913: rollback oid 6 current snap is 249 2024-09-13T14:29:00.706 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 6 to 233 2024-09-13T14:29:00.707 INFO:tasks.rados.rados.0.smithi151.stdout:1911: finishing write tid 2 to smithi15186934-2 2024-09-13T14:29:00.707 INFO:tasks.rados.rados.0.smithi151.stdout:1914: snap_remove snap 246 2024-09-13T14:29:01.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:00 smithi184 ceph-mon[92527]: pgmap v441: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 165 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 986 KiB/s wr, 15 op/s; 29/303 objects degraded (9.571%) 2024-09-13T14:29:01.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:00 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/929671160' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:01.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:00 smithi184 ceph-mon[92527]: osdmap e550: 8 total, 7 up, 8 in 2024-09-13T14:29:01.108 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:29:00 smithi151 podman[124676]: 2024-09-13 14:29:00.711651764 +0000 UTC m=+0.366207928 container init 04d393df5f431646e8f0f3b823329442e7889de20762f2dbe043bb127d91b0db (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1, io.buildah.version=1.37.2, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-09-13T14:29:01.108 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:29:00 smithi151 podman[124676]: 2024-09-13 14:29:00.717145636 +0000 UTC m=+0.371701788 container start 04d393df5f431646e8f0f3b823329442e7889de20762f2dbe043bb127d91b0db (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-13T14:29:01.108 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:29:00 smithi151 ceph-osd[124743]: -- 172.21.15.151:0/2588651747 <== mon.2 v2:172.21.15.151:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55af5cbd1a40 con 0x55af5d9a0400 2024-09-13T14:29:01.108 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:29:00 smithi151 bash[124676]: 04d393df5f431646e8f0f3b823329442e7889de20762f2dbe043bb127d91b0db 2024-09-13T14:29:01.109 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:29:00 smithi151 systemd[1]: Started Ceph osd.1 for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:29:01.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:00 smithi151 ceph-mon[104450]: pgmap v441: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 165 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 986 KiB/s wr, 15 op/s; 29/303 objects degraded (9.571%) 2024-09-13T14:29:01.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:00 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/929671160' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:01.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:00 smithi151 ceph-mon[104450]: osdmap e550: 8 total, 7 up, 8 in 2024-09-13T14:29:01.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:00 smithi151 ceph-mon[108966]: pgmap v441: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 165 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 986 KiB/s wr, 15 op/s; 29/303 objects degraded (9.571%) 2024-09-13T14:29:01.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:00 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/929671160' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:01.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:00 smithi151 ceph-mon[108966]: osdmap e550: 8 total, 7 up, 8 in 2024-09-13T14:29:01.614 INFO:tasks.rados.rados.0.smithi151.stdout:1914: done (2 left) 2024-09-13T14:29:01.614 INFO:tasks.rados.rados.0.smithi151.stdout:1915: delete oid 29 current snap is 249 2024-09-13T14:29:01.614 INFO:tasks.rados.rados.0.smithi151.stdout:1911: finishing write tid 3 to smithi15186934-2 2024-09-13T14:29:01.614 INFO:tasks.rados.rados.0.smithi151.stdout:1911: finishing write tid 4 to smithi15186934-2 2024-09-13T14:29:01.614 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 2 v 712 (ObjNum 722 snap 249 seq_num 722) dirty exists 2024-09-13T14:29:01.614 INFO:tasks.rados.rados.0.smithi151.stdout:1911: left oid 2 (ObjNum 722 snap 249 seq_num 722) 2024-09-13T14:29:01.614 INFO:tasks.rados.rados.0.smithi151.stdout:1913: finishing rollback tid 0 to smithi15186934-6 2024-09-13T14:29:01.614 INFO:tasks.rados.rados.0.smithi151.stdout:1913: finishing rollback tid 1 to smithi15186934-6 2024-09-13T14:29:01.615 INFO:tasks.rados.rados.0.smithi151.stdout:1913: finishing rollback tid 2 to smithi15186934-6 2024-09-13T14:29:01.615 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 6 v 463 (ObjNum 582 snap 186 seq_num 582) dirty exists 2024-09-13T14:29:01.617 INFO:tasks.rados.rados.0.smithi151.stdout:1911: done (2 left) 2024-09-13T14:29:01.617 INFO:tasks.rados.rados.0.smithi151.stdout:1913: done (1 left) 2024-09-13T14:29:01.617 INFO:tasks.rados.rados.0.smithi151.stdout:1915: done (0 left) 2024-09-13T14:29:01.617 INFO:tasks.rados.rados.0.smithi151.stdout:1916: snap_remove snap 232 2024-09-13T14:29:02.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:01 smithi151 ceph-mon[104450]: from='client.44438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:02.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:02.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:02.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:29:02.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:01 smithi151 ceph-mon[104450]: from='client.44447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:02.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:02.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:02.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:01 smithi151 ceph-mon[104450]: osdmap e551: 8 total, 7 up, 8 in 2024-09-13T14:29:02.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:01 smithi151 ceph-mon[108966]: from='client.44438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:02.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:01 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:02.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:01 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:02.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:01 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:29:02.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:01 smithi151 ceph-mon[108966]: from='client.44447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:02.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:01 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:02.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:01 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:02.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:01 smithi151 ceph-mon[108966]: osdmap e551: 8 total, 7 up, 8 in 2024-09-13T14:29:02.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:01 smithi184 ceph-mon[92527]: from='client.44438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:02.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:01 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:02.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:01 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:02.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:01 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:29:02.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:01 smithi184 ceph-mon[92527]: from='client.44447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:02.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:01 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:02.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:01 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:02.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:01 smithi184 ceph-mon[92527]: osdmap e551: 8 total, 7 up, 8 in 2024-09-13T14:29:02.618 INFO:tasks.rados.rados.0.smithi151.stdout:1916: done (0 left) 2024-09-13T14:29:02.618 INFO:tasks.rados.rados.0.smithi151.stdout:1917: rollback oid 38 current snap is 249 2024-09-13T14:29:02.618 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 38 to 242 2024-09-13T14:29:02.618 INFO:tasks.rados.rados.0.smithi151.stdout:1918: read oid 36 snap -1 2024-09-13T14:29:02.618 INFO:tasks.rados.rados.0.smithi151.stdout:1918: expect (ObjNum 698 snap 232 seq_num 698) 2024-09-13T14:29:02.618 INFO:tasks.rados.rados.0.smithi151.stdout:1919: delete oid 20 current snap is 249 2024-09-13T14:29:02.624 INFO:tasks.rados.rados.0.smithi151.stdout:1919: done (2 left) 2024-09-13T14:29:02.625 INFO:tasks.rados.rados.0.smithi151.stdout:1920: read oid 28 snap -1 2024-09-13T14:29:02.625 INFO:tasks.rados.rados.0.smithi151.stdout:1920: expect deleted 2024-09-13T14:29:02.625 INFO:tasks.rados.rados.0.smithi151.stdout:1921: snap_remove snap 233 2024-09-13T14:29:02.858 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:29:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1[124715]: 2024-09-13T14:29:02.612+0000 7fd186545640 -1 Falling back to public interface 2024-09-13T14:29:03.256 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:02 smithi151 ceph-mon[104450]: pgmap v444: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 165 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 285 KiB/s wr, 3 op/s; 29/303 objects degraded (9.571%) 2024-09-13T14:29:03.256 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:02 smithi151 ceph-mon[104450]: osdmap e552: 8 total, 7 up, 8 in 2024-09-13T14:29:03.257 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:02 smithi151 ceph-mon[108966]: pgmap v444: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 165 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 285 KiB/s wr, 3 op/s; 29/303 objects degraded (9.571%) 2024-09-13T14:29:03.257 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:02 smithi151 ceph-mon[108966]: osdmap e552: 8 total, 7 up, 8 in 2024-09-13T14:29:03.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:02 smithi184 ceph-mon[92527]: pgmap v444: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 165 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 285 KiB/s wr, 3 op/s; 29/303 objects degraded (9.571%) 2024-09-13T14:29:03.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:02 smithi184 ceph-mon[92527]: osdmap e552: 8 total, 7 up, 8 in 2024-09-13T14:29:03.570 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:29:03.620 INFO:tasks.rados.rados.0.smithi151.stdout:1921: done (3 left) 2024-09-13T14:29:03.620 INFO:tasks.rados.rados.0.smithi151.stdout:1922: delete oid 17 current snap is 249 2024-09-13T14:29:03.620 INFO:tasks.rados.rados.0.smithi151.stdout:1917: finishing rollback tid 0 to smithi15186934-38 2024-09-13T14:29:03.620 INFO:tasks.rados.rados.0.smithi151.stdout:1917: finishing rollback tid 1 to smithi15186934-38 2024-09-13T14:29:03.620 INFO:tasks.rados.rados.0.smithi151.stdout:1917: finishing rollback tid 2 to smithi15186934-38 2024-09-13T14:29:03.620 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 649 (ObjNum 702 snap 232 seq_num 702) dirty exists 2024-09-13T14:29:03.620 INFO:tasks.rados.rados.0.smithi151.stdout:1918: expect (ObjNum 698 snap 232 seq_num 698) 2024-09-13T14:29:03.786 INFO:tasks.rados.rados.0.smithi151.stdout:1917: done (3 left) 2024-09-13T14:29:03.787 INFO:tasks.rados.rados.0.smithi151.stdout:1918: done (2 left) 2024-09-13T14:29:03.787 INFO:tasks.rados.rados.0.smithi151.stdout:1920: done (1 left) 2024-09-13T14:29:03.787 INFO:tasks.rados.rados.0.smithi151.stdout:1922: done (0 left) 2024-09-13T14:29:03.787 INFO:tasks.rados.rados.0.smithi151.stdout:1923: rollback oid 47 current snap is 249 2024-09-13T14:29:03.787 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 47 to 236 2024-09-13T14:29:03.787 INFO:tasks.rados.rados.0.smithi151.stdout:1924: write oid 48 current snap is 249 2024-09-13T14:29:03.787 INFO:tasks.rados.rados.0.smithi151.stdout:1924: seq_num 727 ranges {618689=559558,1659115=559574} 2024-09-13T14:29:03.794 INFO:tasks.rados.rados.0.smithi151.stdout:1924: writing smithi15186934-48 from 618689 to 1178247 tid 1 2024-09-13T14:29:03.801 INFO:tasks.rados.rados.0.smithi151.stdout:1924: writing smithi15186934-48 from 1659115 to 2218689 tid 2 2024-09-13T14:29:03.801 INFO:tasks.rados.rados.0.smithi151.stdout:1925: delete oid 35 current snap is 249 2024-09-13T14:29:03.801 INFO:tasks.rados.rados.0.smithi151.stdout:1923: finishing rollback tid 0 to smithi15186934-47 2024-09-13T14:29:03.801 INFO:tasks.rados.rados.0.smithi151.stdout:1923: finishing rollback tid 1 to smithi15186934-47 2024-09-13T14:29:03.801 INFO:tasks.rados.rados.0.smithi151.stdout:1923: finishing rollback tid 2 to smithi15186934-47 2024-09-13T14:29:03.801 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 866 (ObjNum 659 snap 213 seq_num 659) dirty exists 2024-09-13T14:29:03.801 INFO:tasks.rados.rados.0.smithi151.stdout:1924: finishing write tid 1 to smithi15186934-48 2024-09-13T14:29:03.803 INFO:tasks.rados.rados.0.smithi151.stdout:1923: done (2 left) 2024-09-13T14:29:03.803 INFO:tasks.rados.rados.0.smithi151.stdout:1925: done (1 left) 2024-09-13T14:29:03.803 INFO:tasks.rados.rados.0.smithi151.stdout:1926: delete oid 1 current snap is 249 2024-09-13T14:29:03.804 INFO:tasks.rados.rados.0.smithi151.stdout:1924: finishing write tid 2 to smithi15186934-48 2024-09-13T14:29:03.804 INFO:tasks.rados.rados.0.smithi151.stdout:1924: finishing write tid 3 to smithi15186934-48 2024-09-13T14:29:03.805 INFO:tasks.rados.rados.0.smithi151.stdout:1924: finishing write tid 4 to smithi15186934-48 2024-09-13T14:29:03.805 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 48 v 1023 (ObjNum 727 snap 249 seq_num 727) dirty exists 2024-09-13T14:29:03.805 INFO:tasks.rados.rados.0.smithi151.stdout:1924: left oid 48 (ObjNum 727 snap 249 seq_num 727) 2024-09-13T14:29:03.805 INFO:tasks.rados.rados.0.smithi151.stdout:1924: done (1 left) 2024-09-13T14:29:03.805 INFO:tasks.rados.rados.0.smithi151.stdout:1926: done (0 left) 2024-09-13T14:29:03.805 INFO:tasks.rados.rados.0.smithi151.stdout:1927: snap_create 2024-09-13T14:29:04.144 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:03 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/4230606143' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:04.144 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:03 smithi151 ceph-mon[104450]: osdmap e553: 8 total, 7 up, 8 in 2024-09-13T14:29:04.144 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:03 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/4230606143' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:04.144 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:03 smithi151 ceph-mon[108966]: osdmap e553: 8 total, 7 up, 8 in 2024-09-13T14:29:04.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:03 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/4230606143' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:04.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:03 smithi184 ceph-mon[92527]: osdmap e553: 8 total, 7 up, 8 in 2024-09-13T14:29:04.516 INFO:tasks.rados.rados.0.smithi151.stdout:1927: done (0 left) 2024-09-13T14:29:04.516 INFO:tasks.rados.rados.0.smithi151.stdout:1928: read oid 34 snap -1 2024-09-13T14:29:04.517 INFO:tasks.rados.rados.0.smithi151.stdout:1928: expect (ObjNum 721 snap 249 seq_num 721) 2024-09-13T14:29:04.517 INFO:tasks.rados.rados.0.smithi151.stdout:1929: read oid 33 snap -1 2024-09-13T14:29:04.517 INFO:tasks.rados.rados.0.smithi151.stdout:1929: expect deleted 2024-09-13T14:29:04.517 INFO:tasks.rados.rados.0.smithi151.stdout:1930: rollback oid 12 current snap is 250 2024-09-13T14:29:04.517 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 12 to 224 2024-09-13T14:29:04.517 INFO:tasks.rados.rados.0.smithi151.stdout:1931: write oid 26 current snap is 250 2024-09-13T14:29:04.517 INFO:tasks.rados.rados.0.smithi151.stdout:1931: seq_num 730 ranges {669110=501658,1834561=434549} 2024-09-13T14:29:04.524 INFO:tasks.rados.rados.0.smithi151.stdout:1931: writing smithi15186934-26 from 669110 to 1170768 tid 1 2024-09-13T14:29:04.530 INFO:tasks.rados.rados.0.smithi151.stdout:1931: writing smithi15186934-26 from 1834561 to 2269110 tid 2 2024-09-13T14:29:04.530 INFO:tasks.rados.rados.0.smithi151.stdout:1932: read oid 48 snap -1 2024-09-13T14:29:04.530 INFO:tasks.rados.rados.0.smithi151.stdout:1932: expect (ObjNum 727 snap 249 seq_num 727) 2024-09-13T14:29:04.530 INFO:tasks.rados.rados.0.smithi151.stdout:1933: read oid 9 snap 241 2024-09-13T14:29:04.530 INFO:tasks.rados.rados.0.smithi151.stdout:1933: expect (ObjNum 699 snap 232 seq_num 699) 2024-09-13T14:29:04.530 INFO:tasks.rados.rados.0.smithi151.stdout:1934: read oid 20 snap 240 2024-09-13T14:29:04.530 INFO:tasks.rados.rados.0.smithi151.stdout:1934: expect deleted 2024-09-13T14:29:04.530 INFO:tasks.rados.rados.0.smithi151.stdout:1935: snap_remove snap 230 2024-09-13T14:29:04.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:29:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:29:04] "GET /metrics HTTP/1.1" 200 36528 "" "Prometheus/2.51.0" 2024-09-13T14:29:05.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:04 smithi184 ceph-mon[92527]: pgmap v447: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 173 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 9 op/s; 32/330 objects degraded (9.697%) 2024-09-13T14:29:05.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:04 smithi184 ceph-mon[92527]: from='client.34575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:05.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:04 smithi184 ceph-mon[92527]: from='client.44465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:05.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:04 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 32/330 objects degraded (9.697%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:29:05.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:04 smithi184 ceph-mon[92527]: osdmap e554: 8 total, 7 up, 8 in 2024-09-13T14:29:05.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:04 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:05.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:04 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:05.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:04 smithi151 ceph-mon[108966]: pgmap v447: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 173 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 9 op/s; 32/330 objects degraded (9.697%) 2024-09-13T14:29:05.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:04 smithi151 ceph-mon[108966]: from='client.34575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:05.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:04 smithi151 ceph-mon[108966]: from='client.44465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:05.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:04 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 32/330 objects degraded (9.697%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:29:05.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:04 smithi151 ceph-mon[108966]: osdmap e554: 8 total, 7 up, 8 in 2024-09-13T14:29:05.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:04 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:05.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:04 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:05.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:04 smithi151 ceph-mon[104450]: pgmap v447: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 173 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 9 op/s; 32/330 objects degraded (9.697%) 2024-09-13T14:29:05.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:04 smithi151 ceph-mon[104450]: from='client.34575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:05.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:04 smithi151 ceph-mon[104450]: from='client.44465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:05.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:04 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 32/330 objects degraded (9.697%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:29:05.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:04 smithi151 ceph-mon[104450]: osdmap e554: 8 total, 7 up, 8 in 2024-09-13T14:29:05.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:04 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:05.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:04 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:05.519 INFO:tasks.rados.rados.0.smithi151.stdout:1935: done (7 left) 2024-09-13T14:29:05.519 INFO:tasks.rados.rados.0.smithi151.stdout:1936: delete oid 44 current snap is 250 2024-09-13T14:29:05.520 INFO:tasks.rados.rados.0.smithi151.stdout:1930: finishing rollback tid 0 to smithi15186934-12 2024-09-13T14:29:05.520 INFO:tasks.rados.rados.0.smithi151.stdout:1930: finishing rollback tid 1 to smithi15186934-12 2024-09-13T14:29:05.520 INFO:tasks.rados.rados.0.smithi151.stdout:1930: finishing rollback tid 2 to smithi15186934-12 2024-09-13T14:29:05.520 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 12 v 433 (ObjNum 654 snap 212 seq_num 654) dirty exists 2024-09-13T14:29:05.520 INFO:tasks.rados.rados.0.smithi151.stdout:1928: expect (ObjNum 721 snap 249 seq_num 721) 2024-09-13T14:29:05.696 INFO:tasks.rados.rados.0.smithi151.stdout:1931: finishing write tid 1 to smithi15186934-26 2024-09-13T14:29:05.696 INFO:tasks.rados.rados.0.smithi151.stdout:1931: finishing write tid 2 to smithi15186934-26 2024-09-13T14:29:05.696 INFO:tasks.rados.rados.0.smithi151.stdout:1931: finishing write tid 3 to smithi15186934-26 2024-09-13T14:29:05.696 INFO:tasks.rados.rados.0.smithi151.stdout:1931: finishing write tid 4 to smithi15186934-26 2024-09-13T14:29:05.696 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 668 (ObjNum 730 snap 250 seq_num 730) dirty exists 2024-09-13T14:29:05.696 INFO:tasks.rados.rados.0.smithi151.stdout:1931: left oid 26 (ObjNum 730 snap 250 seq_num 730) 2024-09-13T14:29:05.696 INFO:tasks.rados.rados.0.smithi151.stdout:1933: expect (ObjNum 699 snap 232 seq_num 699) 2024-09-13T14:29:05.817 INFO:tasks.rados.rados.0.smithi151.stdout:1932: expect (ObjNum 727 snap 249 seq_num 727) 2024-09-13T14:29:05.911 INFO:tasks.rados.rados.0.smithi151.stdout:1928: done (7 left) 2024-09-13T14:29:05.911 INFO:tasks.rados.rados.0.smithi151.stdout:1929: done (6 left) 2024-09-13T14:29:05.911 INFO:tasks.rados.rados.0.smithi151.stdout:1930: done (5 left) 2024-09-13T14:29:05.912 INFO:tasks.rados.rados.0.smithi151.stdout:1931: done (4 left) 2024-09-13T14:29:05.912 INFO:tasks.rados.rados.0.smithi151.stdout:1932: done (3 left) 2024-09-13T14:29:05.912 INFO:tasks.rados.rados.0.smithi151.stdout:1933: done (2 left) 2024-09-13T14:29:05.912 INFO:tasks.rados.rados.0.smithi151.stdout:1934: done (1 left) 2024-09-13T14:29:05.912 INFO:tasks.rados.rados.0.smithi151.stdout:1936: done (0 left) 2024-09-13T14:29:05.912 INFO:tasks.rados.rados.0.smithi151.stdout:1937: write oid 41 current snap is 250 2024-09-13T14:29:05.912 INFO:tasks.rados.rados.0.smithi151.stdout:1937: seq_num 732 ranges {702724=652156,1951525=351199} 2024-09-13T14:29:05.920 INFO:tasks.rados.rados.0.smithi151.stdout:1937: writing smithi15186934-41 from 702724 to 1354880 tid 1 2024-09-13T14:29:05.926 INFO:tasks.rados.rados.0.smithi151.stdout:1937: writing smithi15186934-41 from 1951525 to 2302724 tid 2 2024-09-13T14:29:05.926 INFO:tasks.rados.rados.0.smithi151.stdout:1937: finishing write tid 1 to smithi15186934-41 2024-09-13T14:29:05.926 INFO:tasks.rados.rados.0.smithi151.stdout:1938: read oid 39 snap -1 2024-09-13T14:29:05.926 INFO:tasks.rados.rados.0.smithi151.stdout:1938: expect (ObjNum 710 snap 244 seq_num 710) 2024-09-13T14:29:05.927 INFO:tasks.rados.rados.0.smithi151.stdout:1939: snap_create 2024-09-13T14:29:05.929 INFO:tasks.rados.rados.0.smithi151.stdout:1937: finishing write tid 2 to smithi15186934-41 2024-09-13T14:29:05.929 INFO:tasks.rados.rados.0.smithi151.stdout:1937: finishing write tid 3 to smithi15186934-41 2024-09-13T14:29:05.929 INFO:tasks.rados.rados.0.smithi151.stdout:1937: finishing write tid 4 to smithi15186934-41 2024-09-13T14:29:05.929 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 102 (ObjNum 732 snap 250 seq_num 732) dirty exists 2024-09-13T14:29:05.929 INFO:tasks.rados.rados.0.smithi151.stdout:1937: left oid 41 (ObjNum 732 snap 250 seq_num 732) 2024-09-13T14:29:05.940 INFO:tasks.rados.rados.0.smithi151.stdout:1938: expect (ObjNum 710 snap 244 seq_num 710) 2024-09-13T14:29:06.481 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:29:06.528 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:29:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1[124715]: 2024-09-13T14:29:06.295+0000 7fd186545640 -1 osd.1 544 log_to_monitors true 2024-09-13T14:29:06.529 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:06 smithi151 ceph-mon[108966]: pgmap v450: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 173 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 10 op/s; 32/333 objects degraded (9.610%) 2024-09-13T14:29:06.529 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:06 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:06.529 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:06 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:06.529 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:06 smithi151 ceph-mon[108966]: osdmap e555: 8 total, 7 up, 8 in 2024-09-13T14:29:06.529 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:06 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2481686776' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:06.529 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:06 smithi151 ceph-mon[104450]: pgmap v450: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 173 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 10 op/s; 32/333 objects degraded (9.610%) 2024-09-13T14:29:06.530 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:06 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:06.530 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:06 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:06.530 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:06 smithi151 ceph-mon[104450]: osdmap e555: 8 total, 7 up, 8 in 2024-09-13T14:29:06.530 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:06 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2481686776' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:06.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:06 smithi184 ceph-mon[92527]: pgmap v450: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 173 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 10 op/s; 32/333 objects degraded (9.610%) 2024-09-13T14:29:06.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:06 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:06.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:06 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:06.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:06 smithi184 ceph-mon[92527]: osdmap e555: 8 total, 7 up, 8 in 2024-09-13T14:29:06.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:06 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2481686776' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:06.891 INFO:tasks.rados.rados.0.smithi151.stdout:1937: done (2 left) 2024-09-13T14:29:06.891 INFO:tasks.rados.rados.0.smithi151.stdout:1938: done (1 left) 2024-09-13T14:29:06.891 INFO:tasks.rados.rados.0.smithi151.stdout:1939: done (0 left) 2024-09-13T14:29:06.891 INFO:tasks.rados.rados.0.smithi151.stdout:1940: read oid 49 snap -1 2024-09-13T14:29:06.891 INFO:tasks.rados.rados.0.smithi151.stdout:1940: expect (ObjNum 635 snap 206 seq_num 635) 2024-09-13T14:29:06.891 INFO:tasks.rados.rados.0.smithi151.stdout:1941: snap_create 2024-09-13T14:29:06.901 INFO:tasks.rados.rados.0.smithi151.stdout:1940: expect (ObjNum 635 snap 206 seq_num 635) 2024-09-13T14:29:07.257 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 14:29:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1[124715]: 2024-09-13T14:29:06.912+0000 7fd17e2ef640 -1 osd.1 544 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:29:07.258 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:07 smithi151 ceph-mon[104450]: from='osd.1 [v2:172.21.15.151:6810/3884881052,v1:172.21.15.151:6811/3884881052]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-13T14:29:07.258 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:07 smithi151 ceph-mon[104450]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-13T14:29:07.258 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:07 smithi151 ceph-mon[104450]: from='client.34593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:07.258 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:07 smithi151 ceph-mon[104450]: from='client.44486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:07.258 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:07 smithi151 ceph-mon[104450]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T14:29:07.258 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:07 smithi151 ceph-mon[104450]: osdmap e556: 8 total, 7 up, 8 in 2024-09-13T14:29:07.258 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:07 smithi151 ceph-mon[104450]: from='osd.1 [v2:172.21.15.151:6810/3884881052,v1:172.21.15.151:6811/3884881052]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-13T14:29:07.258 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:07 smithi151 ceph-mon[104450]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-13T14:29:07.524 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:07 smithi151 ceph-mon[108966]: from='osd.1 [v2:172.21.15.151:6810/3884881052,v1:172.21.15.151:6811/3884881052]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-13T14:29:07.525 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:07 smithi151 ceph-mon[108966]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-13T14:29:07.525 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:07 smithi151 ceph-mon[108966]: from='client.34593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:07.525 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:07 smithi151 ceph-mon[108966]: from='client.44486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:07.525 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:07 smithi151 ceph-mon[108966]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T14:29:07.525 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:07 smithi151 ceph-mon[108966]: osdmap e556: 8 total, 7 up, 8 in 2024-09-13T14:29:07.525 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:07 smithi151 ceph-mon[108966]: from='osd.1 [v2:172.21.15.151:6810/3884881052,v1:172.21.15.151:6811/3884881052]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-13T14:29:07.525 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:07 smithi151 ceph-mon[108966]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-13T14:29:07.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:07 smithi184 ceph-mon[92527]: from='osd.1 [v2:172.21.15.151:6810/3884881052,v1:172.21.15.151:6811/3884881052]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-13T14:29:07.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:07 smithi184 ceph-mon[92527]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-13T14:29:07.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:07 smithi184 ceph-mon[92527]: from='client.34593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:07.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:07 smithi184 ceph-mon[92527]: from='client.44486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:07.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:07 smithi184 ceph-mon[92527]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T14:29:07.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:07 smithi184 ceph-mon[92527]: osdmap e556: 8 total, 7 up, 8 in 2024-09-13T14:29:07.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:07 smithi184 ceph-mon[92527]: from='osd.1 [v2:172.21.15.151:6810/3884881052,v1:172.21.15.151:6811/3884881052]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-13T14:29:07.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:07 smithi184 ceph-mon[92527]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-13T14:29:07.896 INFO:tasks.rados.rados.0.smithi151.stdout:1940: done (1 left) 2024-09-13T14:29:07.896 INFO:tasks.rados.rados.0.smithi151.stdout:1941: done (0 left) 2024-09-13T14:29:07.896 INFO:tasks.rados.rados.0.smithi151.stdout:1942: rollback oid 34 current snap is 252 2024-09-13T14:29:07.897 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 34 to 224 2024-09-13T14:29:07.897 INFO:tasks.rados.rados.0.smithi151.stdout:1943: snap_create 2024-09-13T14:29:08.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:08 smithi184 ceph-mon[92527]: pgmap v453: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 176 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 951 KiB/s wr, 20 op/s; 34/357 objects degraded (9.524%) 2024-09-13T14:29:08.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:08 smithi184 ceph-mon[92527]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:29:08.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:08 smithi184 ceph-mon[92527]: osd.1 [v2:172.21.15.151:6810/3884881052,v1:172.21.15.151:6811/3884881052] boot 2024-09-13T14:29:08.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:08 smithi184 ceph-mon[92527]: osdmap e557: 8 total, 8 up, 8 in 2024-09-13T14:29:08.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:08 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T14:29:08.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:08 smithi151 ceph-mon[108966]: pgmap v453: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 176 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 951 KiB/s wr, 20 op/s; 34/357 objects degraded (9.524%) 2024-09-13T14:29:08.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:08 smithi151 ceph-mon[108966]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:29:08.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:08 smithi151 ceph-mon[108966]: osd.1 [v2:172.21.15.151:6810/3884881052,v1:172.21.15.151:6811/3884881052] boot 2024-09-13T14:29:08.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:08 smithi151 ceph-mon[108966]: osdmap e557: 8 total, 8 up, 8 in 2024-09-13T14:29:08.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:08 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T14:29:08.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:08 smithi151 ceph-mon[104450]: pgmap v453: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 176 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 951 KiB/s wr, 20 op/s; 34/357 objects degraded (9.524%) 2024-09-13T14:29:08.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:08 smithi151 ceph-mon[104450]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:29:08.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:08 smithi151 ceph-mon[104450]: osd.1 [v2:172.21.15.151:6810/3884881052,v1:172.21.15.151:6811/3884881052] boot 2024-09-13T14:29:08.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:08 smithi151 ceph-mon[104450]: osdmap e557: 8 total, 8 up, 8 in 2024-09-13T14:29:08.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:08 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T14:29:08.902 INFO:tasks.rados.rados.0.smithi151.stdout:1943: done (1 left) 2024-09-13T14:29:08.903 INFO:tasks.rados.rados.0.smithi151.stdout:1944: read oid 48 snap -1 2024-09-13T14:29:08.903 INFO:tasks.rados.rados.0.smithi151.stdout:1944: expect (ObjNum 727 snap 249 seq_num 727) 2024-09-13T14:29:08.903 INFO:tasks.rados.rados.0.smithi151.stdout:1945: write oid 23 current snap is 253 2024-09-13T14:29:08.903 INFO:tasks.rados.rados.0.smithi151.stdout:1945: seq_num 733 ranges {719531=727405,1893654=425877} 2024-09-13T14:29:08.914 INFO:tasks.rados.rados.0.smithi151.stdout:1945: writing smithi15186934-23 from 719531 to 1446936 tid 1 2024-09-13T14:29:08.920 INFO:tasks.rados.rados.0.smithi151.stdout:1945: writing smithi15186934-23 from 1893654 to 2319531 tid 2 2024-09-13T14:29:08.920 INFO:tasks.rados.rados.0.smithi151.stdout:1946: delete oid 49 current snap is 253 2024-09-13T14:29:08.921 INFO:tasks.rados.rados.0.smithi151.stdout:1944: expect (ObjNum 727 snap 249 seq_num 727) 2024-09-13T14:29:09.025 INFO:tasks.rados.rados.0.smithi151.stdout:1945: finishing write tid 1 to smithi15186934-23 2024-09-13T14:29:09.025 INFO:tasks.rados.rados.0.smithi151.stdout:1945: finishing write tid 2 to smithi15186934-23 2024-09-13T14:29:09.025 INFO:tasks.rados.rados.0.smithi151.stdout:1945: finishing write tid 3 to smithi15186934-23 2024-09-13T14:29:09.025 INFO:tasks.rados.rados.0.smithi151.stdout:1945: finishing write tid 4 to smithi15186934-23 2024-09-13T14:29:09.025 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 23 v 873 (ObjNum 733 snap 253 seq_num 733) dirty exists 2024-09-13T14:29:09.025 INFO:tasks.rados.rados.0.smithi151.stdout:1945: left oid 23 (ObjNum 733 snap 253 seq_num 733) 2024-09-13T14:29:09.025 INFO:tasks.rados.rados.0.smithi151.stdout:1942: finishing rollback tid 0 to smithi15186934-34 2024-09-13T14:29:09.025 INFO:tasks.rados.rados.0.smithi151.stdout:1942: finishing rollback tid 1 to smithi15186934-34 2024-09-13T14:29:09.026 INFO:tasks.rados.rados.0.smithi151.stdout:1942: finishing rollback tid 2 to smithi15186934-34 2024-09-13T14:29:09.026 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 702 (ObjNum 672 snap 216 seq_num 672) dirty exists 2024-09-13T14:29:09.026 INFO:tasks.rados.rados.0.smithi151.stdout:1942: done (3 left) 2024-09-13T14:29:09.026 INFO:tasks.rados.rados.0.smithi151.stdout:1944: done (2 left) 2024-09-13T14:29:09.026 INFO:tasks.rados.rados.0.smithi151.stdout:1945: done (1 left) 2024-09-13T14:29:09.026 INFO:tasks.rados.rados.0.smithi151.stdout:1946: done (0 left) 2024-09-13T14:29:09.026 INFO:tasks.rados.rados.0.smithi151.stdout:1947: read oid 12 snap -1 2024-09-13T14:29:09.026 INFO:tasks.rados.rados.0.smithi151.stdout:1947: expect (ObjNum 654 snap 212 seq_num 654) 2024-09-13T14:29:09.026 INFO:tasks.rados.rados.0.smithi151.stdout:1948: read oid 17 snap 241 2024-09-13T14:29:09.026 INFO:tasks.rados.rados.0.smithi151.stdout:1948: expect deleted 2024-09-13T14:29:09.026 INFO:tasks.rados.rados.0.smithi151.stdout:1949: read oid 23 snap -1 2024-09-13T14:29:09.026 INFO:tasks.rados.rados.0.smithi151.stdout:1949: expect (ObjNum 733 snap 253 seq_num 733) 2024-09-13T14:29:09.026 INFO:tasks.rados.rados.0.smithi151.stdout:1950: snap_create 2024-09-13T14:29:09.029 INFO:tasks.rados.rados.0.smithi151.stdout:1947: expect (ObjNum 654 snap 212 seq_num 654) 2024-09-13T14:29:09.108 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:29:09 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:29:09.010+0000 7fc94c153640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-09-13T14:29:09.131 INFO:tasks.rados.rados.0.smithi151.stdout:1949: expect (ObjNum 733 snap 253 seq_num 733) 2024-09-13T14:29:09.399 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:29:09.902 INFO:tasks.rados.rados.0.smithi151.stdout:1947: done (3 left) 2024-09-13T14:29:09.902 INFO:tasks.rados.rados.0.smithi151.stdout:1948: done (2 left) 2024-09-13T14:29:09.902 INFO:tasks.rados.rados.0.smithi151.stdout:1949: done (1 left) 2024-09-13T14:29:09.903 INFO:tasks.rados.rados.0.smithi151.stdout:1950: done (0 left) 2024-09-13T14:29:09.903 INFO:tasks.rados.rados.0.smithi151.stdout:1951: snap_remove snap 234 2024-09-13T14:29:10.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:09 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:10.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:09 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:10.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:09 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:10.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:09 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:29:10.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:09 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:10.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:09 smithi184 ceph-mon[92527]: osdmap e558: 8 total, 8 up, 8 in 2024-09-13T14:29:10.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:09 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:29:10.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:09 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:29:10.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:09 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:29:10.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:09 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:29:10.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:09 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:29:10.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:09 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3509886524' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:10.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:09 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 34/357 objects degraded (9.524%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:29:10.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:09 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:10.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:09 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:10.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:09 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:10.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:09 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:29:10.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:09 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:10.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:09 smithi151 ceph-mon[108966]: osdmap e558: 8 total, 8 up, 8 in 2024-09-13T14:29:10.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:09 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:29:10.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:09 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:29:10.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:09 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:29:10.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:09 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:29:10.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:09 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:29:10.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:09 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3509886524' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:10.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:09 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 34/357 objects degraded (9.524%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:29:10.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:09 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:10.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:09 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:10.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:09 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:10.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:09 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:29:10.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:09 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:10.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:09 smithi151 ceph-mon[104450]: osdmap e558: 8 total, 8 up, 8 in 2024-09-13T14:29:10.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:09 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:29:10.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:09 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:29:10.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:09 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:29:10.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:09 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:29:10.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:09 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:29:10.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:09 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3509886524' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:10.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:09 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 34/357 objects degraded (9.524%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:29:10.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:29:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:29:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:10.906 INFO:tasks.rados.rados.0.smithi151.stdout:1951: done (0 left) 2024-09-13T14:29:10.906 INFO:tasks.rados.rados.0.smithi151.stdout:1952: read oid 43 snap 249 2024-09-13T14:29:10.906 INFO:tasks.rados.rados.0.smithi151.stdout:1952: expect deleted 2024-09-13T14:29:10.906 INFO:tasks.rados.rados.0.smithi151.stdout:1953: delete oid 11 current snap is 254 2024-09-13T14:29:10.915 INFO:tasks.rados.rados.0.smithi151.stdout:1952: done (1 left) 2024-09-13T14:29:10.915 INFO:tasks.rados.rados.0.smithi151.stdout:1953: done (0 left) 2024-09-13T14:29:10.915 INFO:tasks.rados.rados.0.smithi151.stdout:1954: read oid 28 snap 239 2024-09-13T14:29:10.915 INFO:tasks.rados.rados.0.smithi151.stdout:1954: expect deleted 2024-09-13T14:29:10.915 INFO:tasks.rados.rados.0.smithi151.stdout:1955: snap_remove snap 251 2024-09-13T14:29:11.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:10 smithi184 ceph-mon[92527]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:29:11.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:10 smithi184 ceph-mon[92527]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-09-13T14:29:11.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:10 smithi184 ceph-mon[92527]: pgmap v456: 89 pgs: 4 peering, 26 active+undersized, 7 active+undersized+degraded, 52 active+clean; 176 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 747 KiB/s wr, 18 op/s; 31/357 objects degraded (8.683%) 2024-09-13T14:29:11.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:10 smithi184 ceph-mon[92527]: from='client.44498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:11.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:10 smithi184 ceph-mon[92527]: from='client.44504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:11.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:10 smithi184 ceph-mon[92527]: osdmap e559: 8 total, 8 up, 8 in 2024-09-13T14:29:11.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:10 smithi151 ceph-mon[104450]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:29:11.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:10 smithi151 ceph-mon[104450]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-09-13T14:29:11.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:10 smithi151 ceph-mon[104450]: pgmap v456: 89 pgs: 4 peering, 26 active+undersized, 7 active+undersized+degraded, 52 active+clean; 176 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 747 KiB/s wr, 18 op/s; 31/357 objects degraded (8.683%) 2024-09-13T14:29:11.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:10 smithi151 ceph-mon[104450]: from='client.44498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:11.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:10 smithi151 ceph-mon[104450]: from='client.44504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:11.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:10 smithi151 ceph-mon[104450]: osdmap e559: 8 total, 8 up, 8 in 2024-09-13T14:29:11.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:10 smithi151 ceph-mon[108966]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:29:11.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:10 smithi151 ceph-mon[108966]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-09-13T14:29:11.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:10 smithi151 ceph-mon[108966]: pgmap v456: 89 pgs: 4 peering, 26 active+undersized, 7 active+undersized+degraded, 52 active+clean; 176 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 747 KiB/s wr, 18 op/s; 31/357 objects degraded (8.683%) 2024-09-13T14:29:11.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:10 smithi151 ceph-mon[108966]: from='client.44498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:11.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:10 smithi151 ceph-mon[108966]: from='client.44504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:11.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:10 smithi151 ceph-mon[108966]: osdmap e559: 8 total, 8 up, 8 in 2024-09-13T14:29:11.921 INFO:tasks.rados.rados.0.smithi151.stdout:1955: done (1 left) 2024-09-13T14:29:11.921 INFO:tasks.rados.rados.0.smithi151.stdout:1956: snap_remove snap 242 2024-09-13T14:29:12.250 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:29:12.251 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:11 smithi151 ceph-mon[104450]: osdmap e560: 8 total, 8 up, 8 in 2024-09-13T14:29:12.251 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:11 smithi151 ceph-mon[108966]: osdmap e560: 8 total, 8 up, 8 in 2024-09-13T14:29:12.255 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:11 smithi184 ceph-mon[92527]: osdmap e560: 8 total, 8 up, 8 in 2024-09-13T14:29:12.923 INFO:tasks.rados.rados.0.smithi151.stdout:1956: done (1 left) 2024-09-13T14:29:12.923 INFO:tasks.rados.rados.0.smithi151.stdout:1957: write oid 15 current snap is 254 2024-09-13T14:29:12.923 INFO:tasks.rados.rados.0.smithi151.stdout:1957: seq_num 736 ranges {769952=553152,2069100=300852} 2024-09-13T14:29:12.931 INFO:tasks.rados.rados.0.smithi151.stdout:1957: writing smithi15186934-15 from 769952 to 1323104 tid 1 2024-09-13T14:29:12.937 INFO:tasks.rados.rados.0.smithi151.stdout:1957: writing smithi15186934-15 from 2069100 to 2369952 tid 2 2024-09-13T14:29:12.937 INFO:tasks.rados.rados.0.smithi151.stdout:1958: write oid 36 current snap is 254 2024-09-13T14:29:12.937 INFO:tasks.rados.rados.0.smithi151.stdout:1958: seq_num 737 ranges {786759=628401,2011229=375530} 2024-09-13T14:29:12.945 INFO:tasks.rados.rados.0.smithi151.stdout:1958: writing smithi15186934-36 from 786759 to 1415160 tid 1 2024-09-13T14:29:12.950 INFO:tasks.rados.rados.0.smithi151.stdout:1958: writing smithi15186934-36 from 2011229 to 2386759 tid 2 2024-09-13T14:29:12.950 INFO:tasks.rados.rados.0.smithi151.stdout:1959: snap_remove snap 235 2024-09-13T14:29:13.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:12 smithi184 ceph-mon[92527]: pgmap v459: 89 pgs: 4 peering, 26 active+undersized, 7 active+undersized+degraded, 52 active+clean; 176 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 31/357 objects degraded (8.683%) 2024-09-13T14:29:13.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:12 smithi184 ceph-mon[92527]: osdmap e561: 8 total, 8 up, 8 in 2024-09-13T14:29:13.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:12 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2665071353' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:13.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:12 smithi151 ceph-mon[104450]: pgmap v459: 89 pgs: 4 peering, 26 active+undersized, 7 active+undersized+degraded, 52 active+clean; 176 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 31/357 objects degraded (8.683%) 2024-09-13T14:29:13.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:12 smithi151 ceph-mon[104450]: osdmap e561: 8 total, 8 up, 8 in 2024-09-13T14:29:13.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:12 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2665071353' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:13.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:12 smithi151 ceph-mon[108966]: pgmap v459: 89 pgs: 4 peering, 26 active+undersized, 7 active+undersized+degraded, 52 active+clean; 176 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 31/357 objects degraded (8.683%) 2024-09-13T14:29:13.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:12 smithi151 ceph-mon[108966]: osdmap e561: 8 total, 8 up, 8 in 2024-09-13T14:29:13.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:12 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2665071353' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:13.938 INFO:tasks.rados.rados.0.smithi151.stdout:1959: done (3 left) 2024-09-13T14:29:13.938 INFO:tasks.rados.rados.0.smithi151.stdout:1960: read oid 13 snap 223 2024-09-13T14:29:13.938 INFO:tasks.rados.rados.0.smithi151.stdout:1960: expect (ObjNum 627 snap 204 seq_num 627) 2024-09-13T14:29:13.938 INFO:tasks.rados.rados.0.smithi151.stdout:1961: write oid 10 current snap is 254 2024-09-13T14:29:13.938 INFO:tasks.rados.rados.0.smithi151.stdout:1961: seq_num 738 ranges {403566=420003,1386064=546918} 2024-09-13T14:29:13.943 INFO:tasks.rados.rados.0.smithi151.stdout:1961: writing smithi15186934-10 from 403566 to 823569 tid 1 2024-09-13T14:29:13.950 INFO:tasks.rados.rados.0.smithi151.stdout:1961: writing smithi15186934-10 from 1386064 to 1932982 tid 2 2024-09-13T14:29:13.950 INFO:tasks.rados.rados.0.smithi151.stdout:1962: write oid 45 current snap is 254 2024-09-13T14:29:13.951 INFO:tasks.rados.rados.0.smithi151.stdout:1957: finishing write tid 1 to smithi15186934-15 2024-09-13T14:29:13.951 INFO:tasks.rados.rados.0.smithi151.stdout:1957: finishing write tid 2 to smithi15186934-15 2024-09-13T14:29:13.951 INFO:tasks.rados.rados.0.smithi151.stdout:1957: finishing write tid 3 to smithi15186934-15 2024-09-13T14:29:13.951 INFO:tasks.rados.rados.0.smithi151.stdout:1957: finishing write tid 4 to smithi15186934-15 2024-09-13T14:29:13.951 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 15 v 874 (ObjNum 736 snap 254 seq_num 736) dirty exists 2024-09-13T14:29:13.951 INFO:tasks.rados.rados.0.smithi151.stdout:1957: left oid 15 (ObjNum 736 snap 254 seq_num 736) 2024-09-13T14:29:13.951 INFO:tasks.rados.rados.0.smithi151.stdout:1958: finishing write tid 1 to smithi15186934-36 2024-09-13T14:29:13.951 INFO:tasks.rados.rados.0.smithi151.stdout:1958: finishing write tid 2 to smithi15186934-36 2024-09-13T14:29:13.951 INFO:tasks.rados.rados.0.smithi151.stdout:1958: finishing write tid 3 to smithi15186934-36 2024-09-13T14:29:13.951 INFO:tasks.rados.rados.0.smithi151.stdout:1958: finishing write tid 4 to smithi15186934-36 2024-09-13T14:29:13.951 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 36 v 449 (ObjNum 737 snap 254 seq_num 737) dirty exists 2024-09-13T14:29:13.951 INFO:tasks.rados.rados.0.smithi151.stdout:1958: left oid 36 (ObjNum 737 snap 254 seq_num 737) 2024-09-13T14:29:13.951 INFO:tasks.rados.rados.0.smithi151.stdout:1960: expect (ObjNum 627 snap 204 seq_num 627) 2024-09-13T14:29:14.122 INFO:tasks.rados.rados.0.smithi151.stdout:1961: finishing write tid 1 to smithi15186934-10 2024-09-13T14:29:14.122 INFO:tasks.rados.rados.0.smithi151.stdout:1962: seq_num 739 ranges {420373=495252,1444546=690576} 2024-09-13T14:29:14.128 INFO:tasks.rados.rados.0.smithi151.stdout:1962: writing smithi15186934-45 from 420373 to 915625 tid 1 2024-09-13T14:29:14.134 INFO:tasks.rados.rados.0.smithi151.stdout:1962: writing smithi15186934-45 from 1444546 to 2135122 tid 2 2024-09-13T14:29:14.134 INFO:tasks.rados.rados.0.smithi151.stdout:1954: done (5 left) 2024-09-13T14:29:14.134 INFO:tasks.rados.rados.0.smithi151.stdout:1957: done (4 left) 2024-09-13T14:29:14.134 INFO:tasks.rados.rados.0.smithi151.stdout:1958: done (3 left) 2024-09-13T14:29:14.134 INFO:tasks.rados.rados.0.smithi151.stdout:1960: done (2 left) 2024-09-13T14:29:14.134 INFO:tasks.rados.rados.0.smithi151.stdout:1963: snap_create 2024-09-13T14:29:14.134 INFO:tasks.rados.rados.0.smithi151.stdout:1961: finishing write tid 2 to smithi15186934-10 2024-09-13T14:29:14.134 INFO:tasks.rados.rados.0.smithi151.stdout:1961: finishing write tid 3 to smithi15186934-10 2024-09-13T14:29:14.134 INFO:tasks.rados.rados.0.smithi151.stdout:1961: finishing write tid 4 to smithi15186934-10 2024-09-13T14:29:14.135 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 682 (ObjNum 738 snap 254 seq_num 738) dirty exists 2024-09-13T14:29:14.135 INFO:tasks.rados.rados.0.smithi151.stdout:1961: left oid 10 (ObjNum 738 snap 254 seq_num 738) 2024-09-13T14:29:14.135 INFO:tasks.rados.rados.0.smithi151.stdout:1962: finishing write tid 1 to smithi15186934-45 2024-09-13T14:29:14.138 INFO:tasks.rados.rados.0.smithi151.stdout:1962: finishing write tid 2 to smithi15186934-45 2024-09-13T14:29:14.138 INFO:tasks.rados.rados.0.smithi151.stdout:1962: finishing write tid 3 to smithi15186934-45 2024-09-13T14:29:14.138 INFO:tasks.rados.rados.0.smithi151.stdout:1962: finishing write tid 4 to smithi15186934-45 2024-09-13T14:29:14.138 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 1048 (ObjNum 739 snap 254 seq_num 739) dirty exists 2024-09-13T14:29:14.138 INFO:tasks.rados.rados.0.smithi151.stdout:1962: left oid 45 (ObjNum 739 snap 254 seq_num 739) 2024-09-13T14:29:14.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:13 smithi184 ceph-mon[92527]: from='client.34632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:14.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:13 smithi184 ceph-mon[92527]: from='client.44522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:14.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:13 smithi184 ceph-mon[92527]: osdmap e562: 8 total, 8 up, 8 in 2024-09-13T14:29:14.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:13 smithi151 ceph-mon[104450]: from='client.34632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:14.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:13 smithi151 ceph-mon[104450]: from='client.44522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:14.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:13 smithi151 ceph-mon[104450]: osdmap e562: 8 total, 8 up, 8 in 2024-09-13T14:29:14.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:13 smithi151 ceph-mon[108966]: from='client.34632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:14.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:13 smithi151 ceph-mon[108966]: from='client.44522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:14.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:13 smithi151 ceph-mon[108966]: osdmap e562: 8 total, 8 up, 8 in 2024-09-13T14:29:14.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:29:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:29:14] "GET /metrics HTTP/1.1" 200 36528 "" "Prometheus/2.51.0" 2024-09-13T14:29:14.945 INFO:tasks.rados.rados.0.smithi151.stdout:1961: done (2 left) 2024-09-13T14:29:14.945 INFO:tasks.rados.rados.0.smithi151.stdout:1962: done (1 left) 2024-09-13T14:29:14.945 INFO:tasks.rados.rados.0.smithi151.stdout:1963: done (0 left) 2024-09-13T14:29:14.945 INFO:tasks.rados.rados.0.smithi151.stdout:1964: snap_remove snap 224 2024-09-13T14:29:15.096 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:29:15.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:14 smithi184 ceph-mon[92527]: pgmap v462: 89 pgs: 4 peering, 85 active+clean; 178 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 6.4 MiB/s rd, 282 KiB/s wr, 8 op/s; 2.5 MiB/s, 2 objects/s recovering 2024-09-13T14:29:15.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:14 smithi184 ceph-mon[92527]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/357 objects degraded (8.683%), 7 pgs degraded) 2024-09-13T14:29:15.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:14 smithi184 ceph-mon[92527]: Cluster is now healthy 2024-09-13T14:29:15.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:14 smithi184 ceph-mon[92527]: osdmap e563: 8 total, 8 up, 8 in 2024-09-13T14:29:15.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:14 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2200059384' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:15.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:14 smithi151 ceph-mon[104450]: pgmap v462: 89 pgs: 4 peering, 85 active+clean; 178 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 6.4 MiB/s rd, 282 KiB/s wr, 8 op/s; 2.5 MiB/s, 2 objects/s recovering 2024-09-13T14:29:15.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:14 smithi151 ceph-mon[104450]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/357 objects degraded (8.683%), 7 pgs degraded) 2024-09-13T14:29:15.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:14 smithi151 ceph-mon[104450]: Cluster is now healthy 2024-09-13T14:29:15.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:14 smithi151 ceph-mon[104450]: osdmap e563: 8 total, 8 up, 8 in 2024-09-13T14:29:15.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:14 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2200059384' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:15.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:14 smithi151 ceph-mon[108966]: pgmap v462: 89 pgs: 4 peering, 85 active+clean; 178 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 6.4 MiB/s rd, 282 KiB/s wr, 8 op/s; 2.5 MiB/s, 2 objects/s recovering 2024-09-13T14:29:15.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:14 smithi151 ceph-mon[108966]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/357 objects degraded (8.683%), 7 pgs degraded) 2024-09-13T14:29:15.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:14 smithi151 ceph-mon[108966]: Cluster is now healthy 2024-09-13T14:29:15.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:14 smithi151 ceph-mon[108966]: osdmap e563: 8 total, 8 up, 8 in 2024-09-13T14:29:15.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:14 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2200059384' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:15.953 INFO:tasks.rados.rados.0.smithi151.stdout:1964: done (0 left) 2024-09-13T14:29:15.954 INFO:tasks.rados.rados.0.smithi151.stdout:1965: snap_create 2024-09-13T14:29:16.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:15 smithi184 ceph-mon[92527]: osdmap e564: 8 total, 8 up, 8 in 2024-09-13T14:29:16.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:15 smithi151 ceph-mon[108966]: osdmap e564: 8 total, 8 up, 8 in 2024-09-13T14:29:16.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:15 smithi151 ceph-mon[104450]: osdmap e564: 8 total, 8 up, 8 in 2024-09-13T14:29:16.952 INFO:tasks.rados.rados.0.smithi151.stdout:1965: done (0 left) 2024-09-13T14:29:16.952 INFO:tasks.rados.rados.0.smithi151.stdout:1966: read oid 49 snap -1 2024-09-13T14:29:16.952 INFO:tasks.rados.rados.0.smithi151.stdout:1966: expect deleted 2024-09-13T14:29:16.952 INFO:tasks.rados.rados.0.smithi151.stdout:1967: read oid 35 snap 238 2024-09-13T14:29:16.952 INFO:tasks.rados.rados.0.smithi151.stdout:1967: expect (ObjNum 680 snap 221 seq_num 680) 2024-09-13T14:29:16.953 INFO:tasks.rados.rados.0.smithi151.stdout:1968: snap_remove snap 254 2024-09-13T14:29:17.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:16 smithi184 ceph-mon[92527]: from='client.34650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:17.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:16 smithi184 ceph-mon[92527]: pgmap v465: 89 pgs: 89 active+clean; 180 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 6.4 MiB/s rd, 490 KiB/s wr, 9 op/s; 2.5 MiB/s, 2 objects/s recovering 2024-09-13T14:29:17.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:16 smithi184 ceph-mon[92527]: from='client.44540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:17.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:16 smithi184 ceph-mon[92527]: osdmap e565: 8 total, 8 up, 8 in 2024-09-13T14:29:17.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:17.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:17.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:16 smithi151 ceph-mon[108966]: from='client.34650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:17.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:16 smithi151 ceph-mon[108966]: pgmap v465: 89 pgs: 89 active+clean; 180 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 6.4 MiB/s rd, 490 KiB/s wr, 9 op/s; 2.5 MiB/s, 2 objects/s recovering 2024-09-13T14:29:17.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:16 smithi151 ceph-mon[108966]: from='client.44540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:17.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:16 smithi151 ceph-mon[108966]: osdmap e565: 8 total, 8 up, 8 in 2024-09-13T14:29:17.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:17.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:17.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:16 smithi151 ceph-mon[104450]: from='client.34650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:17.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:16 smithi151 ceph-mon[104450]: pgmap v465: 89 pgs: 89 active+clean; 180 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 6.4 MiB/s rd, 490 KiB/s wr, 9 op/s; 2.5 MiB/s, 2 objects/s recovering 2024-09-13T14:29:17.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:16 smithi151 ceph-mon[104450]: from='client.44540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:17.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:16 smithi151 ceph-mon[104450]: osdmap e565: 8 total, 8 up, 8 in 2024-09-13T14:29:17.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:17.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:17.945 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:29:17.954 INFO:tasks.rados.rados.0.smithi151.stdout:1968: done (2 left) 2024-09-13T14:29:17.954 INFO:tasks.rados.rados.0.smithi151.stdout:1969: read oid 28 snap -1 2024-09-13T14:29:17.954 INFO:tasks.rados.rados.0.smithi151.stdout:1969: expect deleted 2024-09-13T14:29:17.954 INFO:tasks.rados.rados.0.smithi151.stdout:1970: write oid 45 current snap is 256 2024-09-13T14:29:17.954 INFO:tasks.rados.rados.0.smithi151.stdout:1970: seq_num 740 ranges {437180=570501,1503028=550587} 2024-09-13T14:29:17.960 INFO:tasks.rados.rados.0.smithi151.stdout:1970: writing smithi15186934-45 from 437180 to 1007681 tid 1 2024-09-13T14:29:17.966 INFO:tasks.rados.rados.0.smithi151.stdout:1970: writing smithi15186934-45 from 1503028 to 2053615 tid 2 2024-09-13T14:29:17.966 INFO:tasks.rados.rados.0.smithi151.stdout:1971: snap_remove snap 240 2024-09-13T14:29:18.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:17 smithi184 ceph-mon[92527]: osdmap e566: 8 total, 8 up, 8 in 2024-09-13T14:29:18.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:17 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2181293021' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:18.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:17 smithi184 ceph-mon[92527]: osdmap e567: 8 total, 8 up, 8 in 2024-09-13T14:29:18.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:17 smithi151 ceph-mon[104450]: osdmap e566: 8 total, 8 up, 8 in 2024-09-13T14:29:18.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:17 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2181293021' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:18.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:17 smithi151 ceph-mon[104450]: osdmap e567: 8 total, 8 up, 8 in 2024-09-13T14:29:18.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:17 smithi151 ceph-mon[108966]: osdmap e566: 8 total, 8 up, 8 in 2024-09-13T14:29:18.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:17 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2181293021' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:18.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:17 smithi151 ceph-mon[108966]: osdmap e567: 8 total, 8 up, 8 in 2024-09-13T14:29:18.978 INFO:tasks.rados.rados.0.smithi151.stdout:1971: done (4 left) 2024-09-13T14:29:18.978 INFO:tasks.rados.rados.0.smithi151.stdout:1972: write oid 40 current snap is 256 2024-09-13T14:29:18.978 INFO:tasks.rados.rados.0.smithi151.stdout:1967: expect (ObjNum 680 snap 221 seq_num 680) 2024-09-13T14:29:19.214 INFO:tasks.rados.rados.0.smithi151.stdout:1970: finishing write tid 1 to smithi15186934-45 2024-09-13T14:29:19.215 INFO:tasks.rados.rados.0.smithi151.stdout:1970: finishing write tid 2 to smithi15186934-45 2024-09-13T14:29:19.215 INFO:tasks.rados.rados.0.smithi151.stdout:1970: finishing write tid 3 to smithi15186934-45 2024-09-13T14:29:19.215 INFO:tasks.rados.rados.0.smithi151.stdout:1970: finishing write tid 4 to smithi15186934-45 2024-09-13T14:29:19.215 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 1058 (ObjNum 740 snap 256 seq_num 740) dirty exists 2024-09-13T14:29:19.215 INFO:tasks.rados.rados.0.smithi151.stdout:1970: left oid 45 (ObjNum 740 snap 256 seq_num 740) 2024-09-13T14:29:19.215 INFO:tasks.rados.rados.0.smithi151.stdout:1972: seq_num 741 ranges {453987=645750,1845157=608830} 2024-09-13T14:29:19.220 INFO:tasks.rados.rados.0.smithi151.stdout:1972: writing smithi15186934-40 from 453987 to 1099737 tid 1 2024-09-13T14:29:19.228 INFO:tasks.rados.rados.0.smithi151.stdout:1972: writing smithi15186934-40 from 1845157 to 2453987 tid 2 2024-09-13T14:29:19.228 INFO:tasks.rados.rados.0.smithi151.stdout:1966: done (4 left) 2024-09-13T14:29:19.228 INFO:tasks.rados.rados.0.smithi151.stdout:1967: done (3 left) 2024-09-13T14:29:19.228 INFO:tasks.rados.rados.0.smithi151.stdout:1969: done (2 left) 2024-09-13T14:29:19.228 INFO:tasks.rados.rados.0.smithi151.stdout:1970: done (1 left) 2024-09-13T14:29:19.229 INFO:tasks.rados.rados.0.smithi151.stdout:1973: rollback oid 50 current snap is 256 2024-09-13T14:29:19.229 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 50 to 239 2024-09-13T14:29:19.229 INFO:tasks.rados.rados.0.smithi151.stdout:1972: finishing write tid 1 to smithi15186934-40 2024-09-13T14:29:19.229 INFO:tasks.rados.rados.0.smithi151.stdout:1974: snap_create 2024-09-13T14:29:19.231 INFO:tasks.rados.rados.0.smithi151.stdout:1973: finishing rollback tid 0 to smithi15186934-50 2024-09-13T14:29:19.231 INFO:tasks.rados.rados.0.smithi151.stdout:1972: finishing write tid 2 to smithi15186934-40 2024-09-13T14:29:19.231 INFO:tasks.rados.rados.0.smithi151.stdout:1973: finishing rollback tid 1 to smithi15186934-50 2024-09-13T14:29:19.231 INFO:tasks.rados.rados.0.smithi151.stdout:1972: finishing write tid 3 to smithi15186934-40 2024-09-13T14:29:19.231 INFO:tasks.rados.rados.0.smithi151.stdout:1973: finishing rollback tid 2 to smithi15186934-50 2024-09-13T14:29:19.232 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 50 v 683 (ObjNum 666 snap 215 seq_num 666) dirty exists 2024-09-13T14:29:19.232 INFO:tasks.rados.rados.0.smithi151.stdout:1972: finishing write tid 4 to smithi15186934-40 2024-09-13T14:29:19.232 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 40 v 727 (ObjNum 741 snap 256 seq_num 741) dirty exists 2024-09-13T14:29:19.232 INFO:tasks.rados.rados.0.smithi151.stdout:1972: left oid 40 (ObjNum 741 snap 256 seq_num 741) 2024-09-13T14:29:19.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:18 smithi184 ceph-mon[92527]: pgmap v468: 89 pgs: 89 active+clean; 182 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 980 KiB/s wr, 6 op/s 2024-09-13T14:29:19.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:18 smithi184 ceph-mon[92527]: from='client.34668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:19.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:18 smithi151 ceph-mon[104450]: pgmap v468: 89 pgs: 89 active+clean; 182 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 980 KiB/s wr, 6 op/s 2024-09-13T14:29:19.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:18 smithi151 ceph-mon[104450]: from='client.34668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:19.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:18 smithi151 ceph-mon[108966]: pgmap v468: 89 pgs: 89 active+clean; 182 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 980 KiB/s wr, 6 op/s 2024-09-13T14:29:19.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:18 smithi151 ceph-mon[108966]: from='client.34668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:19.519 INFO:tasks.rados.rados.0.smithi151.stdout:1972: done (2 left) 2024-09-13T14:29:19.519 INFO:tasks.rados.rados.0.smithi151.stdout:1973: done (1 left) 2024-09-13T14:29:19.519 INFO:tasks.rados.rados.0.smithi151.stdout:1974: done (0 left) 2024-09-13T14:29:19.519 INFO:tasks.rados.rados.0.smithi151.stdout:1975: read oid 22 snap -1 2024-09-13T14:29:19.519 INFO:tasks.rados.rados.0.smithi151.stdout:1975: expect deleted 2024-09-13T14:29:19.519 INFO:tasks.rados.rados.0.smithi151.stdout:1976: delete oid 21 current snap is 257 2024-09-13T14:29:19.524 INFO:tasks.rados.rados.0.smithi151.stdout:1975: done (1 left) 2024-09-13T14:29:19.524 INFO:tasks.rados.rados.0.smithi151.stdout:1976: done (0 left) 2024-09-13T14:29:19.524 INFO:tasks.rados.rados.0.smithi151.stdout:1977: delete oid 36 current snap is 257 2024-09-13T14:29:19.528 INFO:tasks.rados.rados.0.smithi151.stdout:1977: done (0 left) 2024-09-13T14:29:19.528 INFO:tasks.rados.rados.0.smithi151.stdout:1978: snap_remove snap 219 2024-09-13T14:29:20.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:19 smithi184 ceph-mon[92527]: from='client.34674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:20.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:19 smithi184 ceph-mon[92527]: osdmap e568: 8 total, 8 up, 8 in 2024-09-13T14:29:20.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:19 smithi184 ceph-mon[92527]: osdmap e569: 8 total, 8 up, 8 in 2024-09-13T14:29:20.354 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:29:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:29:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:20.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:19 smithi151 ceph-mon[104450]: from='client.34674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:20.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:19 smithi151 ceph-mon[104450]: osdmap e568: 8 total, 8 up, 8 in 2024-09-13T14:29:20.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:19 smithi151 ceph-mon[104450]: osdmap e569: 8 total, 8 up, 8 in 2024-09-13T14:29:20.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:19 smithi151 ceph-mon[108966]: from='client.34674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:20.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:19 smithi151 ceph-mon[108966]: osdmap e568: 8 total, 8 up, 8 in 2024-09-13T14:29:20.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:19 smithi151 ceph-mon[108966]: osdmap e569: 8 total, 8 up, 8 in 2024-09-13T14:29:20.521 INFO:tasks.rados.rados.0.smithi151.stdout:1978: done (0 left) 2024-09-13T14:29:20.521 INFO:tasks.rados.rados.0.smithi151.stdout:1979: write oid 6 current snap is 257 2024-09-13T14:29:20.521 INFO:tasks.rados.rados.0.smithi151.stdout:1979: seq_num 744 ranges {504408=471497,1620603=557925} 2024-09-13T14:29:20.527 INFO:tasks.rados.rados.0.smithi151.stdout:1979: writing smithi15186934-6 from 504408 to 975905 tid 1 2024-09-13T14:29:20.535 INFO:tasks.rados.rados.0.smithi151.stdout:1979: writing smithi15186934-6 from 1620603 to 2178528 tid 2 2024-09-13T14:29:20.536 INFO:tasks.rados.rados.0.smithi151.stdout:1980: read oid 41 snap -1 2024-09-13T14:29:20.536 INFO:tasks.rados.rados.0.smithi151.stdout:1980: expect (ObjNum 732 snap 250 seq_num 732) 2024-09-13T14:29:20.536 INFO:tasks.rados.rados.0.smithi151.stdout:1981: read oid 27 snap 249 2024-09-13T14:29:20.536 INFO:tasks.rados.rados.0.smithi151.stdout:1981: expect deleted 2024-09-13T14:29:20.536 INFO:tasks.rados.rados.0.smithi151.stdout:1982: read oid 8 snap -1 2024-09-13T14:29:20.536 INFO:tasks.rados.rados.0.smithi151.stdout:1982: expect (ObjNum 602 snap 193 seq_num 602) 2024-09-13T14:29:20.536 INFO:tasks.rados.rados.0.smithi151.stdout:1983: delete oid 15 current snap is 257 2024-09-13T14:29:20.538 INFO:tasks.rados.rados.0.smithi151.stdout:1981: done (4 left) 2024-09-13T14:29:20.538 INFO:tasks.rados.rados.0.smithi151.stdout:1983: done (3 left) 2024-09-13T14:29:20.538 INFO:tasks.rados.rados.0.smithi151.stdout:1984: read oid 39 snap -1 2024-09-13T14:29:20.539 INFO:tasks.rados.rados.0.smithi151.stdout:1984: expect (ObjNum 710 snap 244 seq_num 710) 2024-09-13T14:29:20.539 INFO:tasks.rados.rados.0.smithi151.stdout:1985: read oid 15 snap 249 2024-09-13T14:29:20.539 INFO:tasks.rados.rados.0.smithi151.stdout:1985: expect deleted 2024-09-13T14:29:20.539 INFO:tasks.rados.rados.0.smithi151.stdout:1986: snap_remove snap 253 2024-09-13T14:29:20.809 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:29:21.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:20 smithi151 ceph-mon[104450]: pgmap v471: 89 pgs: 89 active+clean; 182 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 6.2 MiB/s rd, 771 KiB/s wr, 8 op/s 2024-09-13T14:29:21.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:20 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3070155516' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:21.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:20 smithi151 ceph-mon[104450]: osdmap e570: 8 total, 8 up, 8 in 2024-09-13T14:29:21.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:20 smithi151 ceph-mon[108966]: pgmap v471: 89 pgs: 89 active+clean; 182 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 6.2 MiB/s rd, 771 KiB/s wr, 8 op/s 2024-09-13T14:29:21.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:20 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3070155516' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:21.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:20 smithi151 ceph-mon[108966]: osdmap e570: 8 total, 8 up, 8 in 2024-09-13T14:29:21.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:20 smithi184 ceph-mon[92527]: pgmap v471: 89 pgs: 89 active+clean; 182 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 6.2 MiB/s rd, 771 KiB/s wr, 8 op/s 2024-09-13T14:29:21.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:20 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3070155516' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:21.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:20 smithi184 ceph-mon[92527]: osdmap e570: 8 total, 8 up, 8 in 2024-09-13T14:29:21.522 INFO:tasks.rados.rados.0.smithi151.stdout:1986: done (5 left) 2024-09-13T14:29:21.522 INFO:tasks.rados.rados.0.smithi151.stdout:1987: rollback oid 17 current snap is 257 2024-09-13T14:29:21.522 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 17 to 249 2024-09-13T14:29:21.523 INFO:tasks.rados.rados.0.smithi151.stdout:1979: finishing write tid 1 to smithi15186934-6 2024-09-13T14:29:21.523 INFO:tasks.rados.rados.0.smithi151.stdout:1979: finishing write tid 2 to smithi15186934-6 2024-09-13T14:29:21.523 INFO:tasks.rados.rados.0.smithi151.stdout:1979: finishing write tid 3 to smithi15186934-6 2024-09-13T14:29:21.523 INFO:tasks.rados.rados.0.smithi151.stdout:1979: finishing write tid 4 to smithi15186934-6 2024-09-13T14:29:21.523 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 6 v 483 (ObjNum 744 snap 257 seq_num 744) dirty exists 2024-09-13T14:29:21.523 INFO:tasks.rados.rados.0.smithi151.stdout:1979: left oid 6 (ObjNum 744 snap 257 seq_num 744) 2024-09-13T14:29:21.523 INFO:tasks.rados.rados.0.smithi151.stdout:1982: expect (ObjNum 602 snap 193 seq_num 602) 2024-09-13T14:29:21.628 INFO:tasks.rados.rados.0.smithi151.stdout:1980: expect (ObjNum 732 snap 250 seq_num 732) 2024-09-13T14:29:21.748 INFO:tasks.rados.rados.0.smithi151.stdout:1984: expect (ObjNum 710 snap 244 seq_num 710) 2024-09-13T14:29:21.998 INFO:tasks.rados.rados.0.smithi151.stdout:1987: finishing rollback tid 1 to smithi15186934-17 2024-09-13T14:29:21.998 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 1184 (ObjNum 129 snap 0 seq_num 3092907888) dirty dne 2024-09-13T14:29:21.998 INFO:tasks.rados.rados.0.smithi151.stdout:1979: done (5 left) 2024-09-13T14:29:21.998 INFO:tasks.rados.rados.0.smithi151.stdout:1980: done (4 left) 2024-09-13T14:29:21.998 INFO:tasks.rados.rados.0.smithi151.stdout:1982: done (3 left) 2024-09-13T14:29:21.998 INFO:tasks.rados.rados.0.smithi151.stdout:1984: done (2 left) 2024-09-13T14:29:21.998 INFO:tasks.rados.rados.0.smithi151.stdout:1985: done (1 left) 2024-09-13T14:29:21.998 INFO:tasks.rados.rados.0.smithi151.stdout:1987: done (0 left) 2024-09-13T14:29:21.999 INFO:tasks.rados.rados.0.smithi151.stdout:1988: read oid 14 snap -1 2024-09-13T14:29:21.999 INFO:tasks.rados.rados.0.smithi151.stdout:1988: expect (ObjNum 611 snap 197 seq_num 611) 2024-09-13T14:29:21.999 INFO:tasks.rados.rados.0.smithi151.stdout:1989: write oid 33 current snap is 257 2024-09-13T14:29:21.999 INFO:tasks.rados.rados.0.smithi151.stdout:1989: seq_num 746 ranges {538022=738348,1737567=561594} 2024-09-13T14:29:22.004 INFO:tasks.rados.rados.0.smithi151.stdout:1989: writing smithi15186934-33 from 538022 to 1276370 tid 1 2024-09-13T14:29:22.010 INFO:tasks.rados.rados.0.smithi151.stdout:1989: writing smithi15186934-33 from 1737567 to 2299161 tid 2 2024-09-13T14:29:22.010 INFO:tasks.rados.rados.0.smithi151.stdout:1990: snap_create 2024-09-13T14:29:22.010 INFO:tasks.rados.rados.0.smithi151.stdout:1988: expect (ObjNum 611 snap 197 seq_num 611) 2024-09-13T14:29:22.121 INFO:tasks.rados.rados.0.smithi151.stdout:1989: finishing write tid 1 to smithi15186934-33 2024-09-13T14:29:22.122 INFO:tasks.rados.rados.0.smithi151.stdout:1989: finishing write tid 2 to smithi15186934-33 2024-09-13T14:29:22.122 INFO:tasks.rados.rados.0.smithi151.stdout:1989: finishing write tid 3 to smithi15186934-33 2024-09-13T14:29:22.122 INFO:tasks.rados.rados.0.smithi151.stdout:1989: finishing write tid 4 to smithi15186934-33 2024-09-13T14:29:22.122 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 33 v 751 (ObjNum 746 snap 257 seq_num 746) dirty exists 2024-09-13T14:29:22.122 INFO:tasks.rados.rados.0.smithi151.stdout:1989: left oid 33 (ObjNum 746 snap 257 seq_num 746) 2024-09-13T14:29:22.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:21 smithi184 ceph-mon[92527]: from='client.44570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:22.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:21 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:22.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:21 smithi184 ceph-mon[92527]: osdmap e571: 8 total, 8 up, 8 in 2024-09-13T14:29:22.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:21 smithi151 ceph-mon[104450]: from='client.44570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:22.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:21 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:22.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:21 smithi151 ceph-mon[104450]: osdmap e571: 8 total, 8 up, 8 in 2024-09-13T14:29:22.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:21 smithi151 ceph-mon[108966]: from='client.44570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:22.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:21 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:22.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:21 smithi151 ceph-mon[108966]: osdmap e571: 8 total, 8 up, 8 in 2024-09-13T14:29:22.987 INFO:tasks.rados.rados.0.smithi151.stdout:1988: done (2 left) 2024-09-13T14:29:22.987 INFO:tasks.rados.rados.0.smithi151.stdout:1989: done (1 left) 2024-09-13T14:29:22.987 INFO:tasks.rados.rados.0.smithi151.stdout:1990: done (0 left) 2024-09-13T14:29:22.987 INFO:tasks.rados.rados.0.smithi151.stdout:1991: snap_remove snap 221 2024-09-13T14:29:23.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:22 smithi184 ceph-mon[92527]: from='client.34692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:23.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:22 smithi184 ceph-mon[92527]: pgmap v474: 89 pgs: 89 active+clean; 182 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.6 MiB/s rd, 2 op/s 2024-09-13T14:29:23.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:22 smithi151 ceph-mon[104450]: from='client.34692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:23.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:22 smithi151 ceph-mon[104450]: pgmap v474: 89 pgs: 89 active+clean; 182 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.6 MiB/s rd, 2 op/s 2024-09-13T14:29:23.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:22 smithi151 ceph-mon[108966]: from='client.34692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:23.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:22 smithi151 ceph-mon[108966]: pgmap v474: 89 pgs: 89 active+clean; 182 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.6 MiB/s rd, 2 op/s 2024-09-13T14:29:23.656 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:29:23.994 INFO:tasks.rados.rados.0.smithi151.stdout:1991: done (0 left) 2024-09-13T14:29:23.994 INFO:tasks.rados.rados.0.smithi151.stdout:1992: read oid 40 snap 223 2024-09-13T14:29:23.994 INFO:tasks.rados.rados.0.smithi151.stdout:1992: expect deleted 2024-09-13T14:29:23.994 INFO:tasks.rados.rados.0.smithi151.stdout:1993: delete oid 42 current snap is 258 2024-09-13T14:29:23.999 INFO:tasks.rados.rados.0.smithi151.stdout:1992: done (1 left) 2024-09-13T14:29:23.999 INFO:tasks.rados.rados.0.smithi151.stdout:1993: done (0 left) 2024-09-13T14:29:23.999 INFO:tasks.rados.rados.0.smithi151.stdout:1994: snap_create 2024-09-13T14:29:24.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:23 smithi184 ceph-mon[92527]: osdmap e572: 8 total, 8 up, 8 in 2024-09-13T14:29:24.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:23 smithi184 ceph-mon[92527]: pgmap v477: 89 pgs: 89 active+clean; 185 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 832 KiB/s wr, 14 op/s 2024-09-13T14:29:24.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:23 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2788883776' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:24.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:23 smithi184 ceph-mon[92527]: from='client.34704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:24.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:23 smithi184 ceph-mon[92527]: from='client.44594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:24.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:23 smithi151 ceph-mon[108966]: osdmap e572: 8 total, 8 up, 8 in 2024-09-13T14:29:24.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:23 smithi151 ceph-mon[108966]: pgmap v477: 89 pgs: 89 active+clean; 185 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 832 KiB/s wr, 14 op/s 2024-09-13T14:29:24.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:23 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2788883776' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:24.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:23 smithi151 ceph-mon[108966]: from='client.34704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:24.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:23 smithi151 ceph-mon[108966]: from='client.44594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:24.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:23 smithi151 ceph-mon[104450]: osdmap e572: 8 total, 8 up, 8 in 2024-09-13T14:29:24.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:23 smithi151 ceph-mon[104450]: pgmap v477: 89 pgs: 89 active+clean; 185 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 832 KiB/s wr, 14 op/s 2024-09-13T14:29:24.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:23 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2788883776' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:24.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:23 smithi151 ceph-mon[104450]: from='client.34704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:24.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:23 smithi151 ceph-mon[104450]: from='client.44594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:24.520 INFO:tasks.rados.rados.0.smithi151.stdout:1994: done (0 left) 2024-09-13T14:29:24.521 INFO:tasks.rados.rados.0.smithi151.stdout:1995: rollback oid 31 current snap is 259 2024-09-13T14:29:24.521 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 31 to 241 2024-09-13T14:29:24.521 INFO:tasks.rados.rados.0.smithi151.stdout:1996: delete oid 14 current snap is 259 2024-09-13T14:29:24.526 INFO:tasks.rados.rados.0.smithi151.stdout:1996: done (1 left) 2024-09-13T14:29:24.527 INFO:tasks.rados.rados.0.smithi151.stdout:1997: write oid 32 current snap is 259 2024-09-13T14:29:24.527 INFO:tasks.rados.rados.0.smithi151.stdout:1997: seq_num 749 ranges {588443=564095,1796660=708921} 2024-09-13T14:29:24.532 INFO:tasks.rados.rados.0.smithi151.stdout:1997: writing smithi15186934-32 from 588443 to 1152538 tid 1 2024-09-13T14:29:24.540 INFO:tasks.rados.rados.0.smithi151.stdout:1997: writing smithi15186934-32 from 1796660 to 2505581 tid 2 2024-09-13T14:29:24.541 INFO:tasks.rados.rados.0.smithi151.stdout:1998: write oid 8 current snap is 259 2024-09-13T14:29:24.541 INFO:tasks.rados.rados.0.smithi151.stdout:1995: finishing rollback tid 0 to smithi15186934-31 2024-09-13T14:29:24.541 INFO:tasks.rados.rados.0.smithi151.stdout:1995: finishing rollback tid 1 to smithi15186934-31 2024-09-13T14:29:24.541 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 31 v 882 (ObjNum 129 snap 0 seq_num 3682240159) dirty dne 2024-09-13T14:29:24.541 INFO:tasks.rados.rados.0.smithi151.stdout:1997: finishing write tid 1 to smithi15186934-32 2024-09-13T14:29:24.542 INFO:tasks.rados.rados.0.smithi151.stdout:1998: seq_num 750 ranges {605250=639344,1855142=452579} 2024-09-13T14:29:24.546 INFO:tasks.rados.rados.0.smithi151.stdout:1998: writing smithi15186934-8 from 605250 to 1244594 tid 1 2024-09-13T14:29:24.553 INFO:tasks.rados.rados.0.smithi151.stdout:1998: writing smithi15186934-8 from 1855142 to 2307721 tid 2 2024-09-13T14:29:24.553 INFO:tasks.rados.rados.0.smithi151.stdout:1995: done (2 left) 2024-09-13T14:29:24.553 INFO:tasks.rados.rados.0.smithi151.stdout:1999: snap_remove snap 257 2024-09-13T14:29:24.690 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:29:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:29:24] "GET /metrics HTTP/1.1" 200 36531 "" "Prometheus/2.51.0" 2024-09-13T14:29:25.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:24 smithi184 ceph-mon[92527]: osdmap e573: 8 total, 8 up, 8 in 2024-09-13T14:29:25.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:24 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:29:25.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:24 smithi184 ceph-mon[92527]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:29:25.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:24 smithi184 ceph-mon[92527]: Upgrade: osd.2 is safe to restart 2024-09-13T14:29:25.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:24 smithi184 ceph-mon[92527]: osdmap e574: 8 total, 8 up, 8 in 2024-09-13T14:29:25.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:24 smithi151 ceph-mon[104450]: osdmap e573: 8 total, 8 up, 8 in 2024-09-13T14:29:25.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:24 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:29:25.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:24 smithi151 ceph-mon[104450]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:29:25.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:24 smithi151 ceph-mon[104450]: Upgrade: osd.2 is safe to restart 2024-09-13T14:29:25.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:24 smithi151 ceph-mon[104450]: osdmap e574: 8 total, 8 up, 8 in 2024-09-13T14:29:25.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:24 smithi151 ceph-mon[108966]: osdmap e573: 8 total, 8 up, 8 in 2024-09-13T14:29:25.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:24 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:29:25.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:24 smithi151 ceph-mon[108966]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:29:25.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:24 smithi151 ceph-mon[108966]: Upgrade: osd.2 is safe to restart 2024-09-13T14:29:25.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:24 smithi151 ceph-mon[108966]: osdmap e574: 8 total, 8 up, 8 in 2024-09-13T14:29:25.524 INFO:tasks.rados.rados.0.smithi151.stdout:1999: done (2 left) 2024-09-13T14:29:25.524 INFO:tasks.rados.rados.0.smithi151.stdout:2000: delete oid 48 current snap is 259 2024-09-13T14:29:25.524 INFO:tasks.rados.rados.0.smithi151.stdout:1997: finishing write tid 2 to smithi15186934-32 2024-09-13T14:29:25.524 INFO:tasks.rados.rados.0.smithi151.stdout:1997: finishing write tid 3 to smithi15186934-32 2024-09-13T14:29:25.524 INFO:tasks.rados.rados.0.smithi151.stdout:1997: finishing write tid 4 to smithi15186934-32 2024-09-13T14:29:25.524 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 32 v 790 (ObjNum 749 snap 259 seq_num 749) dirty exists 2024-09-13T14:29:25.524 INFO:tasks.rados.rados.0.smithi151.stdout:1997: left oid 32 (ObjNum 749 snap 259 seq_num 749) 2024-09-13T14:29:25.524 INFO:tasks.rados.rados.0.smithi151.stdout:1998: finishing write tid 1 to smithi15186934-8 2024-09-13T14:29:25.524 INFO:tasks.rados.rados.0.smithi151.stdout:1998: finishing write tid 2 to smithi15186934-8 2024-09-13T14:29:25.524 INFO:tasks.rados.rados.0.smithi151.stdout:1998: finishing write tid 3 to smithi15186934-8 2024-09-13T14:29:25.525 INFO:tasks.rados.rados.0.smithi151.stdout:1998: finishing write tid 4 to smithi15186934-8 2024-09-13T14:29:25.525 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 603 (ObjNum 750 snap 259 seq_num 750) dirty exists 2024-09-13T14:29:25.525 INFO:tasks.rados.rados.0.smithi151.stdout:1998: left oid 8 (ObjNum 750 snap 259 seq_num 750) 2024-09-13T14:29:25.529 INFO:tasks.rados.rados.0.smithi151.stdout:1997: done (2 left) 2024-09-13T14:29:25.529 INFO:tasks.rados.rados.0.smithi151.stdout:1998: done (1 left) 2024-09-13T14:29:25.529 INFO:tasks.rados.rados.0.smithi151.stdout:2000: done (0 left) 2024-09-13T14:29:25.529 INFO:tasks.rados.rados.0.smithi151.stdout:2001: rollback oid 25 current snap is 259 2024-09-13T14:29:25.530 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 25 to 249 2024-09-13T14:29:25.530 INFO:tasks.rados.rados.0.smithi151.stdout:2002: snap_create 2024-09-13T14:29:25.533 INFO:tasks.rados.rados.0.smithi151.stdout:2001: finishing rollback tid 0 to smithi15186934-25 2024-09-13T14:29:25.534 INFO:tasks.rados.rados.0.smithi151.stdout:2001: finishing rollback tid 1 to smithi15186934-25 2024-09-13T14:29:25.534 INFO:tasks.rados.rados.0.smithi151.stdout:2001: finishing rollback tid 2 to smithi15186934-25 2024-09-13T14:29:25.534 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 1078 (ObjNum 711 snap 246 seq_num 711) dirty exists 2024-09-13T14:29:26.520 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:29:26.526 INFO:tasks.rados.rados.0.smithi151.stdout:2001: done (1 left) 2024-09-13T14:29:26.526 INFO:tasks.rados.rados.0.smithi151.stdout:2002: done (0 left) 2024-09-13T14:29:26.526 INFO:tasks.rados.rados.0.smithi151.stdout:2003: write oid 42 current snap is 260 2024-09-13T14:29:26.526 INFO:tasks.rados.rados.0.smithi151.stdout:2003: seq_num 752 ranges {638864=789842,1972106=456248} 2024-09-13T14:29:26.533 INFO:tasks.rados.rados.0.smithi151.stdout:2003: writing smithi15186934-42 from 638864 to 1428706 tid 1 2024-09-13T14:29:26.539 INFO:tasks.rados.rados.0.smithi151.stdout:2003: writing smithi15186934-42 from 1972106 to 2428354 tid 2 2024-09-13T14:29:26.540 INFO:tasks.rados.rados.0.smithi151.stdout:2004: delete oid 14 current snap is 260 2024-09-13T14:29:26.542 INFO:tasks.rados.rados.0.smithi151.stdout:2003: finishing write tid 1 to smithi15186934-42 2024-09-13T14:29:26.543 INFO:tasks.rados.rados.0.smithi151.stdout:2003: finishing write tid 2 to smithi15186934-42 2024-09-13T14:29:26.543 INFO:tasks.rados.rados.0.smithi151.stdout:2003: finishing write tid 3 to smithi15186934-42 2024-09-13T14:29:26.543 INFO:tasks.rados.rados.0.smithi151.stdout:2003: finishing write tid 4 to smithi15186934-42 2024-09-13T14:29:26.543 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 887 (ObjNum 752 snap 260 seq_num 752) dirty exists 2024-09-13T14:29:26.543 INFO:tasks.rados.rados.0.smithi151.stdout:2003: left oid 42 (ObjNum 752 snap 260 seq_num 752) 2024-09-13T14:29:26.545 INFO:tasks.rados.rados.0.smithi151.stdout:2003: done (1 left) 2024-09-13T14:29:26.545 INFO:tasks.rados.rados.0.smithi151.stdout:2004: done (0 left) 2024-09-13T14:29:26.545 INFO:tasks.rados.rados.0.smithi151.stdout:2005: read oid 11 snap -1 2024-09-13T14:29:26.545 INFO:tasks.rados.rados.0.smithi151.stdout:2005: expect deleted 2024-09-13T14:29:26.545 INFO:tasks.rados.rados.0.smithi151.stdout:2006: write oid 24 current snap is 260 2024-09-13T14:29:26.545 INFO:tasks.rados.rados.0.smithi151.stdout:2006: seq_num 754 ranges {672478=656693,2089070=459917} 2024-09-13T14:29:26.552 INFO:tasks.rados.rados.0.smithi151.stdout:2006: writing smithi15186934-24 from 672478 to 1329171 tid 1 2024-09-13T14:29:26.559 INFO:tasks.rados.rados.0.smithi151.stdout:2006: writing smithi15186934-24 from 2089070 to 2548987 tid 2 2024-09-13T14:29:26.559 INFO:tasks.rados.rados.0.smithi151.stdout:2007: write oid 3 current snap is 260 2024-09-13T14:29:26.559 INFO:tasks.rados.rados.0.smithi151.stdout:2006: finishing write tid 1 to smithi15186934-24 2024-09-13T14:29:26.559 INFO:tasks.rados.rados.0.smithi151.stdout:2007: seq_num 755 ranges {689285=731942,2147552=541733} 2024-09-13T14:29:26.567 INFO:tasks.rados.rados.0.smithi151.stdout:2007: writing smithi15186934-3 from 689285 to 1421227 tid 1 2024-09-13T14:29:26.574 INFO:tasks.rados.rados.0.smithi151.stdout:2007: writing smithi15186934-3 from 2147552 to 2689285 tid 2 2024-09-13T14:29:26.574 INFO:tasks.rados.rados.0.smithi151.stdout:2005: done (2 left) 2024-09-13T14:29:26.575 INFO:tasks.rados.rados.0.smithi151.stdout:2008: write oid 2 current snap is 260 2024-09-13T14:29:26.575 INFO:tasks.rados.rados.0.smithi151.stdout:2006: finishing write tid 2 to smithi15186934-24 2024-09-13T14:29:26.575 INFO:tasks.rados.rados.0.smithi151.stdout:2008: seq_num 756 ranges {706092=407191,1806034=463586} 2024-09-13T14:29:26.580 INFO:tasks.rados.rados.0.smithi151.stdout:2008: writing smithi15186934-2 from 706092 to 1113283 tid 1 2024-09-13T14:29:26.587 INFO:tasks.rados.rados.0.smithi151.stdout:2008: writing smithi15186934-2 from 1806034 to 2269620 tid 2 2024-09-13T14:29:26.587 INFO:tasks.rados.rados.0.smithi151.stdout:2009: read oid 44 snap -1 2024-09-13T14:29:26.587 INFO:tasks.rados.rados.0.smithi151.stdout:2009: expect deleted 2024-09-13T14:29:26.587 INFO:tasks.rados.rados.0.smithi151.stdout:2010: snap_create 2024-09-13T14:29:26.588 INFO:tasks.rados.rados.0.smithi151.stdout:2006: finishing write tid 3 to smithi15186934-24 2024-09-13T14:29:26.588 INFO:tasks.rados.rados.0.smithi151.stdout:2006: finishing write tid 4 to smithi15186934-24 2024-09-13T14:29:26.588 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 24 v 1259 (ObjNum 754 snap 260 seq_num 754) dirty exists 2024-09-13T14:29:26.588 INFO:tasks.rados.rados.0.smithi151.stdout:2006: left oid 24 (ObjNum 754 snap 260 seq_num 754) 2024-09-13T14:29:26.588 INFO:tasks.rados.rados.0.smithi151.stdout:2007: finishing write tid 1 to smithi15186934-3 2024-09-13T14:29:26.588 INFO:tasks.rados.rados.0.smithi151.stdout:2007: finishing write tid 2 to smithi15186934-3 2024-09-13T14:29:26.588 INFO:tasks.rados.rados.0.smithi151.stdout:2007: finishing write tid 3 to smithi15186934-3 2024-09-13T14:29:26.588 INFO:tasks.rados.rados.0.smithi151.stdout:2007: finishing write tid 4 to smithi15186934-3 2024-09-13T14:29:26.588 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 739 (ObjNum 755 snap 260 seq_num 755) dirty exists 2024-09-13T14:29:26.588 INFO:tasks.rados.rados.0.smithi151.stdout:2007: left oid 3 (ObjNum 755 snap 260 seq_num 755) 2024-09-13T14:29:26.588 INFO:tasks.rados.rados.0.smithi151.stdout:2008: finishing write tid 1 to smithi15186934-2 2024-09-13T14:29:26.590 INFO:tasks.rados.rados.0.smithi151.stdout:2008: finishing write tid 2 to smithi15186934-2 2024-09-13T14:29:26.590 INFO:tasks.rados.rados.0.smithi151.stdout:2008: finishing write tid 3 to smithi15186934-2 2024-09-13T14:29:26.590 INFO:tasks.rados.rados.0.smithi151.stdout:2008: finishing write tid 4 to smithi15186934-2 2024-09-13T14:29:26.590 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 2 v 759 (ObjNum 756 snap 260 seq_num 756) dirty exists 2024-09-13T14:29:26.590 INFO:tasks.rados.rados.0.smithi151.stdout:2008: left oid 2 (ObjNum 756 snap 260 seq_num 756) 2024-09-13T14:29:26.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:26 smithi184 ceph-mon[92527]: pgmap v480: 89 pgs: 89 active+clean; 184 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2024-09-13T14:29:26.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:26 smithi184 ceph-mon[92527]: Upgrade: Updating osd.2 2024-09-13T14:29:26.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:26 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:26.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:26 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-13T14:29:26.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:26 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:26.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:26 smithi184 ceph-mon[92527]: Deploying daemon osd.2 on smithi151 2024-09-13T14:29:26.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:26 smithi184 ceph-mon[92527]: osdmap e575: 8 total, 8 up, 8 in 2024-09-13T14:29:26.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:26 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1910888510' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:26.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:26 smithi151 ceph-mon[104450]: pgmap v480: 89 pgs: 89 active+clean; 184 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2024-09-13T14:29:26.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:26 smithi151 ceph-mon[104450]: Upgrade: Updating osd.2 2024-09-13T14:29:26.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:26 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:26.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:26 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-13T14:29:26.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:26 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:26.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:26 smithi151 ceph-mon[104450]: Deploying daemon osd.2 on smithi151 2024-09-13T14:29:26.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:26 smithi151 ceph-mon[104450]: osdmap e575: 8 total, 8 up, 8 in 2024-09-13T14:29:26.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:26 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1910888510' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:26.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:26 smithi151 ceph-mon[108966]: pgmap v480: 89 pgs: 89 active+clean; 184 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2024-09-13T14:29:26.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:26 smithi151 ceph-mon[108966]: Upgrade: Updating osd.2 2024-09-13T14:29:26.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:26 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:26.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:26 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-13T14:29:26.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:26 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:26.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:26 smithi151 ceph-mon[108966]: Deploying daemon osd.2 on smithi151 2024-09-13T14:29:26.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:26 smithi151 ceph-mon[108966]: osdmap e575: 8 total, 8 up, 8 in 2024-09-13T14:29:26.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:26 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1910888510' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:27.538 INFO:tasks.rados.rados.0.smithi151.stdout:2006: done (4 left) 2024-09-13T14:29:27.538 INFO:tasks.rados.rados.0.smithi151.stdout:2007: done (3 left) 2024-09-13T14:29:27.538 INFO:tasks.rados.rados.0.smithi151.stdout:2008: done (2 left) 2024-09-13T14:29:27.538 INFO:tasks.rados.rados.0.smithi151.stdout:2009: done (1 left) 2024-09-13T14:29:27.538 INFO:tasks.rados.rados.0.smithi151.stdout:2010: done (0 left) 2024-09-13T14:29:27.538 INFO:tasks.rados.rados.0.smithi151.stdout:2011: snap_remove snap 248 2024-09-13T14:29:27.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:27 smithi184 ceph-mon[92527]: from='client.34722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:27.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:27 smithi184 ceph-mon[92527]: osdmap e576: 8 total, 8 up, 8 in 2024-09-13T14:29:27.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:27 smithi184 ceph-mon[92527]: from='client.34728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:27.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:27 smithi151 ceph-mon[104450]: from='client.34722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:27.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:27 smithi151 ceph-mon[104450]: osdmap e576: 8 total, 8 up, 8 in 2024-09-13T14:29:27.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:27 smithi151 ceph-mon[104450]: from='client.34728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:27.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:27 smithi151 ceph-mon[108966]: from='client.34722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:27.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:27 smithi151 ceph-mon[108966]: osdmap e576: 8 total, 8 up, 8 in 2024-09-13T14:29:27.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:27 smithi151 ceph-mon[108966]: from='client.34728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:28.534 INFO:tasks.rados.rados.0.smithi151.stdout:2011: done (0 left) 2024-09-13T14:29:28.534 INFO:tasks.rados.rados.0.smithi151.stdout:2012: delete oid 10 current snap is 261 2024-09-13T14:29:28.549 INFO:tasks.rados.rados.0.smithi151.stdout:2012: done (0 left) 2024-09-13T14:29:28.549 INFO:tasks.rados.rados.0.smithi151.stdout:2013: read oid 18 snap -1 2024-09-13T14:29:28.549 INFO:tasks.rados.rados.0.smithi151.stdout:2013: expect deleted 2024-09-13T14:29:28.549 INFO:tasks.rados.rados.0.smithi151.stdout:2014: read oid 28 snap -1 2024-09-13T14:29:28.549 INFO:tasks.rados.rados.0.smithi151.stdout:2014: expect deleted 2024-09-13T14:29:28.549 INFO:tasks.rados.rados.0.smithi151.stdout:2015: snap_remove snap 236 2024-09-13T14:29:28.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:28 smithi151 ceph-mon[104450]: pgmap v483: 89 pgs: 89 active+clean; 183 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 MiB/s wr, 8 op/s 2024-09-13T14:29:28.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:28 smithi151 ceph-mon[104450]: osdmap e577: 8 total, 8 up, 8 in 2024-09-13T14:29:28.800 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:28 smithi151 ceph-mon[108966]: pgmap v483: 89 pgs: 89 active+clean; 183 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 MiB/s wr, 8 op/s 2024-09-13T14:29:28.800 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:28 smithi151 ceph-mon[108966]: osdmap e577: 8 total, 8 up, 8 in 2024-09-13T14:29:28.801 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:28 smithi151 systemd[1]: Stopping Ceph osd.2 for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:29:28.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:28 smithi184 ceph-mon[92527]: pgmap v483: 89 pgs: 89 active+clean; 183 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 MiB/s wr, 8 op/s 2024-09-13T14:29:28.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:28 smithi184 ceph-mon[92527]: osdmap e577: 8 total, 8 up, 8 in 2024-09-13T14:29:29.108 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:28 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2[58488]: 2024-09-13T14:29:28.798+0000 7fa987d6e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:29:29.108 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:28 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2[58488]: 2024-09-13T14:29:28.798+0000 7fa987d6e640 -1 osd.2 578 *** Got signal Terminated *** 2024-09-13T14:29:29.108 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:28 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2[58488]: 2024-09-13T14:29:28.798+0000 7fa987d6e640 -1 osd.2 578 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T14:29:29.370 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:29:29.523 INFO:tasks.rados.rados.0.smithi151.stdout:2015: done (2 left) 2024-09-13T14:29:29.523 INFO:tasks.rados.rados.0.smithi151.stdout:2016: read oid 10 snap -1 2024-09-13T14:29:29.524 INFO:tasks.rados.rados.0.smithi151.stdout:2016: expect deleted 2024-09-13T14:29:29.524 INFO:tasks.rados.rados.0.smithi151.stdout:2017: write oid 6 current snap is 261 2024-09-13T14:29:29.524 INFO:tasks.rados.rados.0.smithi151.stdout:2017: seq_num 758 ranges {739706=557689,1806645=467255} 2024-09-13T14:29:29.531 INFO:tasks.rados.rados.0.smithi151.stdout:2017: writing smithi15186934-6 from 739706 to 1297395 tid 1 2024-09-13T14:29:29.538 INFO:tasks.rados.rados.0.smithi151.stdout:2017: writing smithi15186934-6 from 1806645 to 2273900 tid 2 2024-09-13T14:29:29.538 INFO:tasks.rados.rados.0.smithi151.stdout:2018: read oid 45 snap -1 2024-09-13T14:29:29.538 INFO:tasks.rados.rados.0.smithi151.stdout:2018: expect (ObjNum 740 snap 256 seq_num 740) 2024-09-13T14:29:29.538 INFO:tasks.rados.rados.0.smithi151.stdout:2019: read oid 49 snap -1 2024-09-13T14:29:29.538 INFO:tasks.rados.rados.0.smithi151.stdout:2019: expect deleted 2024-09-13T14:29:29.539 INFO:tasks.rados.rados.0.smithi151.stdout:2020: snap_create 2024-09-13T14:29:29.541 INFO:tasks.rados.rados.0.smithi151.stdout:2017: finishing write tid 1 to smithi15186934-6 2024-09-13T14:29:29.542 INFO:tasks.rados.rados.0.smithi151.stdout:2017: finishing write tid 2 to smithi15186934-6 2024-09-13T14:29:29.542 INFO:tasks.rados.rados.0.smithi151.stdout:2017: finishing write tid 3 to smithi15186934-6 2024-09-13T14:29:29.542 INFO:tasks.rados.rados.0.smithi151.stdout:2017: finishing write tid 4 to smithi15186934-6 2024-09-13T14:29:29.542 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 6 v 495 (ObjNum 758 snap 261 seq_num 758) dirty exists 2024-09-13T14:29:29.542 INFO:tasks.rados.rados.0.smithi151.stdout:2017: left oid 6 (ObjNum 758 snap 261 seq_num 758) 2024-09-13T14:29:29.549 INFO:tasks.rados.rados.0.smithi151.stdout:2018: expect (ObjNum 740 snap 256 seq_num 740) 2024-09-13T14:29:29.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:29 smithi184 ceph-mon[92527]: osdmap e578: 8 total, 8 up, 8 in 2024-09-13T14:29:29.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:29 smithi184 ceph-mon[92527]: osd.2 marked itself down and dead 2024-09-13T14:29:29.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:29 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2092238490' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:29.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:29 smithi184 ceph-mon[92527]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:29:29.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:29 smithi184 ceph-mon[92527]: osdmap e579: 8 total, 7 up, 8 in 2024-09-13T14:29:29.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:29 smithi151 ceph-mon[108966]: osdmap e578: 8 total, 8 up, 8 in 2024-09-13T14:29:29.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:29 smithi151 ceph-mon[108966]: osd.2 marked itself down and dead 2024-09-13T14:29:29.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:29 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2092238490' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:29.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:29 smithi151 ceph-mon[108966]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:29:29.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:29 smithi151 ceph-mon[108966]: osdmap e579: 8 total, 7 up, 8 in 2024-09-13T14:29:29.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:29 smithi151 ceph-mon[104450]: osdmap e578: 8 total, 8 up, 8 in 2024-09-13T14:29:29.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:29 smithi151 ceph-mon[104450]: osd.2 marked itself down and dead 2024-09-13T14:29:29.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:29 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2092238490' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:29.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:29 smithi151 ceph-mon[104450]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:29:29.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:29 smithi151 ceph-mon[104450]: osdmap e579: 8 total, 7 up, 8 in 2024-09-13T14:29:30.323 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:30 smithi151 podman[129627]: 2024-09-13 14:29:30.064826532 +0000 UTC m=+1.371729947 container died 122660181e46b5d423ee57ee00def953a1cba99f64a5e07234dde43af23a3b77 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2, RELEASE=quincy-bbdadfa, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-13T14:29:30.528 INFO:tasks.rados.rados.0.smithi151.stdout:2013: done (6 left) 2024-09-13T14:29:30.528 INFO:tasks.rados.rados.0.smithi151.stdout:2014: done (5 left) 2024-09-13T14:29:30.528 INFO:tasks.rados.rados.0.smithi151.stdout:2016: done (4 left) 2024-09-13T14:29:30.528 INFO:tasks.rados.rados.0.smithi151.stdout:2017: done (3 left) 2024-09-13T14:29:30.529 INFO:tasks.rados.rados.0.smithi151.stdout:2018: done (2 left) 2024-09-13T14:29:30.529 INFO:tasks.rados.rados.0.smithi151.stdout:2020: done (1 left) 2024-09-13T14:29:30.529 INFO:tasks.rados.rados.0.smithi151.stdout:2021: rollback oid 20 current snap is 262 2024-09-13T14:29:30.529 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 20 to 241 2024-09-13T14:29:30.529 INFO:tasks.rados.rados.0.smithi151.stdout:2022: snap_create 2024-09-13T14:29:30.535 INFO:tasks.rados.rados.0.smithi151.stdout:2021: finishing rollback tid 1 to smithi15186934-20 2024-09-13T14:29:30.535 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 0 (ObjNum 129 snap 0 seq_num 3092914928) dirty dne 2024-09-13T14:29:30.550 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:29:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:29:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:30.608 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:30 smithi151 podman[129627]: 2024-09-13 14:29:30.413220355 +0000 UTC m=+1.720123770 container cleanup 122660181e46b5d423ee57ee00def953a1cba99f64a5e07234dde43af23a3b77 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=quincy-bbdadfa, org.label-schema.schema-version=1.0) 2024-09-13T14:29:30.608 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:30 smithi151 bash[129627]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2 2024-09-13T14:29:30.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:30 smithi151 ceph-mon[104450]: pgmap v486: 89 pgs: 89 active+clean; 184 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.6 MiB/s wr, 9 op/s 2024-09-13T14:29:30.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:30 smithi151 ceph-mon[104450]: from='client.44624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:30.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:30 smithi151 ceph-mon[104450]: from='client.34746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:30.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:30 smithi151 ceph-mon[104450]: osdmap e580: 8 total, 7 up, 8 in 2024-09-13T14:29:30.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:30 smithi151 ceph-mon[108966]: pgmap v486: 89 pgs: 89 active+clean; 184 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.6 MiB/s wr, 9 op/s 2024-09-13T14:29:30.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:30 smithi151 ceph-mon[108966]: from='client.44624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:30.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:30 smithi151 ceph-mon[108966]: from='client.34746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:30.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:30 smithi151 ceph-mon[108966]: osdmap e580: 8 total, 7 up, 8 in 2024-09-13T14:29:30.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:30 smithi184 ceph-mon[92527]: pgmap v486: 89 pgs: 89 active+clean; 184 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.6 MiB/s wr, 9 op/s 2024-09-13T14:29:30.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:30 smithi184 ceph-mon[92527]: from='client.44624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:30.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:30 smithi184 ceph-mon[92527]: from='client.34746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:30.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:30 smithi184 ceph-mon[92527]: osdmap e580: 8 total, 7 up, 8 in 2024-09-13T14:29:31.083 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:30 smithi151 podman[129720]: 2024-09-13 14:29:30.731922023 +0000 UTC m=+0.669846365 container remove 122660181e46b5d423ee57ee00def953a1cba99f64a5e07234dde43af23a3b77 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=quincy-bbdadfa, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-13T14:29:31.358 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:31 smithi151 podman[129814]: 2024-09-13 14:29:31.082699449 +0000 UTC m=+0.111697485 container create 7db9b9bcea7e8bf26beb4097ff1f9281df92b29d67f7feff3825c9a7a160b5b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-13T14:29:31.358 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:31 smithi151 podman[129814]: 2024-09-13 14:29:30.987332884 +0000 UTC m=+0.016330922 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:29:31.359 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:31 smithi151 podman[129814]: 2024-09-13 14:29:31.301783444 +0000 UTC m=+0.330781481 container init 7db9b9bcea7e8bf26beb4097ff1f9281df92b29d67f7feff3825c9a7a160b5b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:29:31.359 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:31 smithi151 podman[129814]: 2024-09-13 14:29:31.306336323 +0000 UTC m=+0.335334361 container start 7db9b9bcea7e8bf26beb4097ff1f9281df92b29d67f7feff3825c9a7a160b5b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-13T14:29:31.526 INFO:tasks.rados.rados.0.smithi151.stdout:2019: done (2 left) 2024-09-13T14:29:31.526 INFO:tasks.rados.rados.0.smithi151.stdout:2021: done (1 left) 2024-09-13T14:29:31.526 INFO:tasks.rados.rados.0.smithi151.stdout:2022: done (0 left) 2024-09-13T14:29:31.526 INFO:tasks.rados.rados.0.smithi151.stdout:2023: read oid 43 snap -1 2024-09-13T14:29:31.526 INFO:tasks.rados.rados.0.smithi151.stdout:2023: expect deleted 2024-09-13T14:29:31.526 INFO:tasks.rados.rados.0.smithi151.stdout:2024: write oid 8 current snap is 263 2024-09-13T14:29:31.526 INFO:tasks.rados.rados.0.smithi151.stdout:2024: seq_num 759 ranges {756513=632938,1865127=727266} 2024-09-13T14:29:31.533 INFO:tasks.rados.rados.0.smithi151.stdout:2024: writing smithi15186934-8 from 756513 to 1389451 tid 1 2024-09-13T14:29:31.539 INFO:tasks.rados.rados.0.smithi151.stdout:2024: writing smithi15186934-8 from 1865127 to 2592393 tid 2 2024-09-13T14:29:31.539 INFO:tasks.rados.rados.0.smithi151.stdout:2025: snap_remove snap 241 2024-09-13T14:29:31.839 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:31 smithi151 podman[129814]: 2024-09-13 14:29:31.401944228 +0000 UTC m=+0.430942278 container attach 7db9b9bcea7e8bf26beb4097ff1f9281df92b29d67f7feff3825c9a7a160b5b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2-deactivate, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-13T14:29:31.840 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:31 smithi151 podman[129814]: 2024-09-13 14:29:31.510807566 +0000 UTC m=+0.539805611 container died 7db9b9bcea7e8bf26beb4097ff1f9281df92b29d67f7feff3825c9a7a160b5b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-13T14:29:32.273 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:29:32.526 INFO:tasks.rados.rados.0.smithi151.stdout:2025: done (2 left) 2024-09-13T14:29:32.526 INFO:tasks.rados.rados.0.smithi151.stdout:2026: write oid 25 current snap is 263 2024-09-13T14:29:32.526 INFO:tasks.rados.rados.0.smithi151.stdout:2026: seq_num 760 ranges {773320=708187,1923609=470924} 2024-09-13T14:29:32.536 INFO:tasks.rados.rados.0.smithi151.stdout:2026: writing smithi15186934-25 from 773320 to 1481507 tid 1 2024-09-13T14:29:32.543 INFO:tasks.rados.rados.0.smithi151.stdout:2026: writing smithi15186934-25 from 1923609 to 2394533 tid 2 2024-09-13T14:29:32.543 INFO:tasks.rados.rados.0.smithi151.stdout:2027: rollback oid 14 current snap is 263 2024-09-13T14:29:32.543 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 14 to 256 2024-09-13T14:29:32.543 INFO:tasks.rados.rados.0.smithi151.stdout:2024: finishing write tid 1 to smithi15186934-8 2024-09-13T14:29:32.543 INFO:tasks.rados.rados.0.smithi151.stdout:2024: finishing write tid 2 to smithi15186934-8 2024-09-13T14:29:32.543 INFO:tasks.rados.rados.0.smithi151.stdout:2024: finishing write tid 3 to smithi15186934-8 2024-09-13T14:29:32.543 INFO:tasks.rados.rados.0.smithi151.stdout:2024: finishing write tid 4 to smithi15186934-8 2024-09-13T14:29:32.543 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 615 (ObjNum 759 snap 263 seq_num 759) dirty exists 2024-09-13T14:29:32.543 INFO:tasks.rados.rados.0.smithi151.stdout:2024: left oid 8 (ObjNum 759 snap 263 seq_num 759) 2024-09-13T14:29:32.543 INFO:tasks.rados.rados.0.smithi151.stdout:2023: done (3 left) 2024-09-13T14:29:32.543 INFO:tasks.rados.rados.0.smithi151.stdout:2024: done (2 left) 2024-09-13T14:29:32.543 INFO:tasks.rados.rados.0.smithi151.stdout:2028: delete oid 29 current snap is 263 2024-09-13T14:29:32.543 INFO:tasks.rados.rados.0.smithi151.stdout:2026: finishing write tid 1 to smithi15186934-25 2024-09-13T14:29:32.544 INFO:tasks.rados.rados.0.smithi151.stdout:2028: done (2 left) 2024-09-13T14:29:32.544 INFO:tasks.rados.rados.0.smithi151.stdout:2029: snap_remove snap 261 2024-09-13T14:29:32.593 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:32 smithi151 ceph-mon[104450]: pgmap v489: 89 pgs: 7 stale+active+clean, 82 active+clean; 184 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 517 KiB/s wr, 2 op/s 2024-09-13T14:29:32.593 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:32.593 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:32.593 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:32 smithi151 ceph-mon[104450]: osdmap e581: 8 total, 7 up, 8 in 2024-09-13T14:29:32.593 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:32 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/4248910505' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:32.594 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:32 smithi151 ceph-mon[108966]: pgmap v489: 89 pgs: 7 stale+active+clean, 82 active+clean; 184 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 517 KiB/s wr, 2 op/s 2024-09-13T14:29:32.594 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:32.594 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:32.594 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:32 smithi151 ceph-mon[108966]: osdmap e581: 8 total, 7 up, 8 in 2024-09-13T14:29:32.594 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:32 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/4248910505' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:32.594 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:32 smithi151 podman[129857]: 2024-09-13 14:29:32.156152164 +0000 UTC m=+0.639523808 container remove 7db9b9bcea7e8bf26beb4097ff1f9281df92b29d67f7feff3825c9a7a160b5b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-13T14:29:32.594 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:32 smithi151 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.2.service: Deactivated successfully. 2024-09-13T14:29:32.595 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:32 smithi151 systemd[1]: Stopped Ceph osd.2 for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:29:32.595 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:32 smithi151 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.2.service: Consumed 13.250s CPU time. 2024-09-13T14:29:32.595 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:32 smithi151 systemd[1]: Starting Ceph osd.2 for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:29:32.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:32 smithi184 ceph-mon[92527]: pgmap v489: 89 pgs: 7 stale+active+clean, 82 active+clean; 184 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 517 KiB/s wr, 2 op/s 2024-09-13T14:29:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:32 smithi184 ceph-mon[92527]: osdmap e581: 8 total, 7 up, 8 in 2024-09-13T14:29:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:32 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/4248910505' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:32.858 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:32 smithi151 podman[130031]: 2024-09-13 14:29:32.49570073 +0000 UTC m=+0.016172765 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:29:32.858 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:32 smithi151 podman[130031]: 2024-09-13 14:29:32.624302854 +0000 UTC m=+0.144774890 container create 62084ad8db79b79f8032b92b6965f63676ac37e3f4bc11fde12d7fd6db6915ab (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:29:32.858 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:32 smithi151 podman[130031]: 2024-09-13 14:29:32.810258233 +0000 UTC m=+0.330730268 container init 62084ad8db79b79f8032b92b6965f63676ac37e3f4bc11fde12d7fd6db6915ab (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2-activate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-13T14:29:32.858 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:32 smithi151 podman[130031]: 2024-09-13 14:29:32.815005978 +0000 UTC m=+0.335478014 container start 62084ad8db79b79f8032b92b6965f63676ac37e3f4bc11fde12d7fd6db6915ab (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2-activate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:29:33.351 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:32 smithi151 podman[130031]: 2024-09-13 14:29:32.918399206 +0000 UTC m=+0.438871255 container attach 62084ad8db79b79f8032b92b6965f63676ac37e3f4bc11fde12d7fd6db6915ab (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2-activate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-13T14:29:33.352 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:32 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2-activate[130049]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:29:33.352 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:32 smithi151 bash[130031]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:29:33.352 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:32 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2-activate[130049]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:29:33.352 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:32 smithi151 bash[130031]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:29:33.352 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:32 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2-activate[130049]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:29:33.352 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:32 smithi151 bash[130031]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:29:33.352 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:32 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2-activate[130049]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:29:33.352 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:32 smithi151 bash[130031]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:29:33.352 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:32 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2-activate[130049]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:29:33.352 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:32 smithi151 bash[130031]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:29:33.353 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:33 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2-activate[130049]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T14:29:33.353 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:33 smithi151 bash[130031]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T14:29:33.353 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:33 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2-activate[130049]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2024-09-13T14:29:33.353 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:33 smithi151 bash[130031]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2024-09-13T14:29:33.353 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:33 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2-activate[130049]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-13T14:29:33.353 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:33 smithi151 bash[130031]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-13T14:29:33.533 INFO:tasks.rados.rados.0.smithi151.stdout:2029: done (2 left) 2024-09-13T14:29:33.533 INFO:tasks.rados.rados.0.smithi151.stdout:2030: write oid 19 current snap is 263 2024-09-13T14:29:33.533 INFO:tasks.rados.rados.0.smithi151.stdout:2027: finishing rollback tid 1 to smithi15186934-14 2024-09-13T14:29:33.533 INFO:tasks.rados.rados.0.smithi151.stdout:2030: seq_num 762 ranges {406934=575038,1640573=474593,2642791=164143} 2024-09-13T14:29:33.538 INFO:tasks.rados.rados.0.smithi151.stdout:2030: writing smithi15186934-19 from 406934 to 981972 tid 1 2024-09-13T14:29:33.544 INFO:tasks.rados.rados.0.smithi151.stdout:2030: writing smithi15186934-19 from 1640573 to 2115166 tid 2 2024-09-13T14:29:33.549 INFO:tasks.rados.rados.0.smithi151.stdout:2030: writing smithi15186934-19 from 2642791 to 2806934 tid 3 2024-09-13T14:29:33.549 INFO:tasks.rados.rados.0.smithi151.stdout:2031: rollback oid 4 current snap is 263 2024-09-13T14:29:33.549 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 4 to 243 2024-09-13T14:29:33.549 INFO:tasks.rados.rados.0.smithi151.stdout:2027: finishing rollback tid 2 to smithi15186934-14 2024-09-13T14:29:33.549 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 803 (ObjNum 611 snap 197 seq_num 611) dirty exists 2024-09-13T14:29:33.549 INFO:tasks.rados.rados.0.smithi151.stdout:2026: finishing write tid 2 to smithi15186934-25 2024-09-13T14:29:33.549 INFO:tasks.rados.rados.0.smithi151.stdout:2026: finishing write tid 3 to smithi15186934-25 2024-09-13T14:29:33.550 INFO:tasks.rados.rados.0.smithi151.stdout:2026: finishing write tid 4 to smithi15186934-25 2024-09-13T14:29:33.550 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 1096 (ObjNum 760 snap 263 seq_num 760) dirty exists 2024-09-13T14:29:33.550 INFO:tasks.rados.rados.0.smithi151.stdout:2026: left oid 25 (ObjNum 760 snap 263 seq_num 760) 2024-09-13T14:29:33.550 INFO:tasks.rados.rados.0.smithi151.stdout:2030: finishing write tid 1 to smithi15186934-19 2024-09-13T14:29:33.550 INFO:tasks.rados.rados.0.smithi151.stdout:2030: finishing write tid 2 to smithi15186934-19 2024-09-13T14:29:33.550 INFO:tasks.rados.rados.0.smithi151.stdout:2026: done (3 left) 2024-09-13T14:29:33.550 INFO:tasks.rados.rados.0.smithi151.stdout:2027: done (2 left) 2024-09-13T14:29:33.550 INFO:tasks.rados.rados.0.smithi151.stdout:2032: read oid 17 snap -1 2024-09-13T14:29:33.550 INFO:tasks.rados.rados.0.smithi151.stdout:2032: expect deleted 2024-09-13T14:29:33.550 INFO:tasks.rados.rados.0.smithi151.stdout:2033: read oid 33 snap -1 2024-09-13T14:29:33.550 INFO:tasks.rados.rados.0.smithi151.stdout:2033: expect (ObjNum 746 snap 257 seq_num 746) 2024-09-13T14:29:33.550 INFO:tasks.rados.rados.0.smithi151.stdout:2034: snap_create 2024-09-13T14:29:33.551 INFO:tasks.rados.rados.0.smithi151.stdout:2030: finishing write tid 3 to smithi15186934-19 2024-09-13T14:29:33.551 INFO:tasks.rados.rados.0.smithi151.stdout:2030: finishing write tid 4 to smithi15186934-19 2024-09-13T14:29:33.551 INFO:tasks.rados.rados.0.smithi151.stdout:2030: finishing write tid 5 to smithi15186934-19 2024-09-13T14:29:33.551 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 19 v 502 (ObjNum 762 snap 263 seq_num 762) dirty exists 2024-09-13T14:29:33.551 INFO:tasks.rados.rados.0.smithi151.stdout:2030: left oid 19 (ObjNum 762 snap 263 seq_num 762) 2024-09-13T14:29:33.552 INFO:tasks.rados.rados.0.smithi151.stdout:2033: expect (ObjNum 746 snap 257 seq_num 746) 2024-09-13T14:29:33.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:33 smithi151 ceph-mon[108966]: from='client.34758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:33.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:33 smithi151 ceph-mon[108966]: osdmap e582: 8 total, 7 up, 8 in 2024-09-13T14:29:33.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:33 smithi151 ceph-mon[108966]: from='client.34764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:33.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:33 smithi151 ceph-mon[104450]: from='client.34758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:33.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:33 smithi151 ceph-mon[104450]: osdmap e582: 8 total, 7 up, 8 in 2024-09-13T14:29:33.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:33 smithi151 ceph-mon[104450]: from='client.34764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:33.609 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:33 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2-activate[130049]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2024-09-13T14:29:33.609 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:33 smithi151 bash[130031]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2024-09-13T14:29:33.609 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:33 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2-activate[130049]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-13T14:29:33.609 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:33 smithi151 bash[130031]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-13T14:29:33.609 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:33 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2-activate[130049]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T14:29:33.610 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:33 smithi151 bash[130031]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T14:29:33.610 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:33 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2-activate[130049]: --> ceph-volume lvm activate successful for osd ID: 2 2024-09-13T14:29:33.610 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:33 smithi151 bash[130031]: --> ceph-volume lvm activate successful for osd ID: 2 2024-09-13T14:29:33.610 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:33 smithi151 podman[130031]: 2024-09-13 14:29:33.379602745 +0000 UTC m=+0.900074781 container died 62084ad8db79b79f8032b92b6965f63676ac37e3f4bc11fde12d7fd6db6915ab (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2-activate, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:29:33.635 INFO:tasks.rados.rados.0.smithi151.stdout:2031: finishing rollback tid 0 to smithi15186934-4 2024-09-13T14:29:33.636 INFO:tasks.rados.rados.0.smithi151.stdout:2031: finishing rollback tid 1 to smithi15186934-4 2024-09-13T14:29:33.636 INFO:tasks.rados.rados.0.smithi151.stdout:2031: finishing rollback tid 2 to smithi15186934-4 2024-09-13T14:29:33.636 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 1102 (ObjNum 592 snap 191 seq_num 592) dirty exists 2024-09-13T14:29:33.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:33 smithi184 ceph-mon[92527]: from='client.34758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:33.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:33 smithi184 ceph-mon[92527]: osdmap e582: 8 total, 7 up, 8 in 2024-09-13T14:29:33.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:33 smithi184 ceph-mon[92527]: from='client.34764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:34.223 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:33 smithi151 podman[130190]: 2024-09-13 14:29:33.963931458 +0000 UTC m=+0.578324983 container remove 62084ad8db79b79f8032b92b6965f63676ac37e3f4bc11fde12d7fd6db6915ab (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2-activate, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:29:34.224 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:34 smithi151 podman[130256]: 2024-09-13 14:29:34.070328118 +0000 UTC m=+0.015841995 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:29:34.224 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:34 smithi151 podman[130256]: 2024-09-13 14:29:34.223513326 +0000 UTC m=+0.169027186 container create 2faaa919b127de81ef46defdf4e920eafa41865f7e62ec05a4d61ecaed748007 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730) 2024-09-13T14:29:34.526 INFO:tasks.rados.rados.0.smithi151.stdout:2030: done (4 left) 2024-09-13T14:29:34.526 INFO:tasks.rados.rados.0.smithi151.stdout:2031: done (3 left) 2024-09-13T14:29:34.527 INFO:tasks.rados.rados.0.smithi151.stdout:2032: done (2 left) 2024-09-13T14:29:34.527 INFO:tasks.rados.rados.0.smithi151.stdout:2033: done (1 left) 2024-09-13T14:29:34.527 INFO:tasks.rados.rados.0.smithi151.stdout:2034: done (0 left) 2024-09-13T14:29:34.527 INFO:tasks.rados.rados.0.smithi151.stdout:2035: write oid 20 current snap is 264 2024-09-13T14:29:34.527 INFO:tasks.rados.rados.0.smithi151.stdout:2035: seq_num 763 ranges {423741=650287,1699055=618251} 2024-09-13T14:29:34.530 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:34 smithi151 podman[130256]: 2024-09-13 14:29:34.442892778 +0000 UTC m=+0.388406643 container init 2faaa919b127de81ef46defdf4e920eafa41865f7e62ec05a4d61ecaed748007 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, org.label-schema.schema-version=1.0) 2024-09-13T14:29:34.530 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:34 smithi151 podman[130256]: 2024-09-13 14:29:34.4498885 +0000 UTC m=+0.395402357 container start 2faaa919b127de81ef46defdf4e920eafa41865f7e62ec05a4d61ecaed748007 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:29:34.530 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:34 smithi151 ceph-osd[130295]: -- 172.21.15.151:0/3066536960 <== mon.2 v2:172.21.15.151:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563f5454fa40 con 0x563f546f2000 2024-09-13T14:29:34.532 INFO:tasks.rados.rados.0.smithi151.stdout:2035: writing smithi15186934-20 from 423741 to 1074028 tid 1 2024-09-13T14:29:34.540 INFO:tasks.rados.rados.0.smithi151.stdout:2035: writing smithi15186934-20 from 1699055 to 2317306 tid 2 2024-09-13T14:29:34.540 INFO:tasks.rados.rados.0.smithi151.stdout:2036: write oid 36 current snap is 264 2024-09-13T14:29:34.540 INFO:tasks.rados.rados.0.smithi151.stdout:2036: seq_num 764 ranges {440548=725536,1757537=478262,2813990=26558} 2024-09-13T14:29:34.546 INFO:tasks.rados.rados.0.smithi151.stdout:2036: writing smithi15186934-36 from 440548 to 1166084 tid 1 2024-09-13T14:29:34.552 INFO:tasks.rados.rados.0.smithi151.stdout:2036: writing smithi15186934-36 from 1757537 to 2235799 tid 2 2024-09-13T14:29:34.556 INFO:tasks.rados.rados.0.smithi151.stdout:2036: writing smithi15186934-36 from 2813990 to 2840548 tid 3 2024-09-13T14:29:34.556 INFO:tasks.rados.rados.0.smithi151.stdout:2037: delete oid 2 current snap is 264 2024-09-13T14:29:34.557 INFO:tasks.rados.rados.0.smithi151.stdout:2035: finishing write tid 1 to smithi15186934-20 2024-09-13T14:29:34.557 INFO:tasks.rados.rados.0.smithi151.stdout:2035: finishing write tid 2 to smithi15186934-20 2024-09-13T14:29:34.557 INFO:tasks.rados.rados.0.smithi151.stdout:2035: finishing write tid 3 to smithi15186934-20 2024-09-13T14:29:34.557 INFO:tasks.rados.rados.0.smithi151.stdout:2035: finishing write tid 4 to smithi15186934-20 2024-09-13T14:29:34.557 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 468 (ObjNum 763 snap 264 seq_num 763) dirty exists 2024-09-13T14:29:34.557 INFO:tasks.rados.rados.0.smithi151.stdout:2035: left oid 20 (ObjNum 763 snap 264 seq_num 763) 2024-09-13T14:29:34.557 INFO:tasks.rados.rados.0.smithi151.stdout:2036: finishing write tid 1 to smithi15186934-36 2024-09-13T14:29:34.557 INFO:tasks.rados.rados.0.smithi151.stdout:2036: finishing write tid 2 to smithi15186934-36 2024-09-13T14:29:34.559 INFO:tasks.rados.rados.0.smithi151.stdout:2036: finishing write tid 3 to smithi15186934-36 2024-09-13T14:29:34.559 INFO:tasks.rados.rados.0.smithi151.stdout:2036: finishing write tid 4 to smithi15186934-36 2024-09-13T14:29:34.559 INFO:tasks.rados.rados.0.smithi151.stdout:2036: finishing write tid 5 to smithi15186934-36 2024-09-13T14:29:34.559 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 36 v 481 (ObjNum 764 snap 264 seq_num 764) dirty exists 2024-09-13T14:29:34.559 INFO:tasks.rados.rados.0.smithi151.stdout:2036: left oid 36 (ObjNum 764 snap 264 seq_num 764) 2024-09-13T14:29:34.559 INFO:tasks.rados.rados.0.smithi151.stdout:2035: done (2 left) 2024-09-13T14:29:34.559 INFO:tasks.rados.rados.0.smithi151.stdout:2036: done (1 left) 2024-09-13T14:29:34.559 INFO:tasks.rados.rados.0.smithi151.stdout:2037: done (0 left) 2024-09-13T14:29:34.559 INFO:tasks.rados.rados.0.smithi151.stdout:2038: read oid 11 snap -1 2024-09-13T14:29:34.559 INFO:tasks.rados.rados.0.smithi151.stdout:2038: expect deleted 2024-09-13T14:29:34.559 INFO:tasks.rados.rados.0.smithi151.stdout:2039: snap_create 2024-09-13T14:29:34.845 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:34 smithi151 ceph-mon[104450]: pgmap v492: 89 pgs: 15 active+undersized, 2 stale+active+clean, 6 active+undersized+degraded, 66 active+clean; 183 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.5 MiB/s rd, 583 KiB/s wr, 5 op/s; 35/384 objects degraded (9.115%) 2024-09-13T14:29:34.845 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:34 smithi151 ceph-mon[104450]: Health check failed: Degraded data redundancy: 35/384 objects degraded (9.115%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:29:34.846 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:34 smithi151 ceph-mon[104450]: osdmap e583: 8 total, 7 up, 8 in 2024-09-13T14:29:34.846 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:34 smithi151 ceph-mon[104450]: osdmap e584: 8 total, 7 up, 8 in 2024-09-13T14:29:34.846 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:34 smithi151 ceph-mon[108966]: pgmap v492: 89 pgs: 15 active+undersized, 2 stale+active+clean, 6 active+undersized+degraded, 66 active+clean; 183 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.5 MiB/s rd, 583 KiB/s wr, 5 op/s; 35/384 objects degraded (9.115%) 2024-09-13T14:29:34.846 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:34 smithi151 ceph-mon[108966]: Health check failed: Degraded data redundancy: 35/384 objects degraded (9.115%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:29:34.846 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:34 smithi151 ceph-mon[108966]: osdmap e583: 8 total, 7 up, 8 in 2024-09-13T14:29:34.846 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:34 smithi151 ceph-mon[108966]: osdmap e584: 8 total, 7 up, 8 in 2024-09-13T14:29:34.847 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:34 smithi151 bash[130256]: 2faaa919b127de81ef46defdf4e920eafa41865f7e62ec05a4d61ecaed748007 2024-09-13T14:29:34.847 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:34 smithi151 systemd[1]: Started Ceph osd.2 for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:29:34.847 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:29:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:29:34] "GET /metrics HTTP/1.1" 200 36520 "" "Prometheus/2.51.0" 2024-09-13T14:29:34.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:34 smithi184 ceph-mon[92527]: pgmap v492: 89 pgs: 15 active+undersized, 2 stale+active+clean, 6 active+undersized+degraded, 66 active+clean; 183 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.5 MiB/s rd, 583 KiB/s wr, 5 op/s; 35/384 objects degraded (9.115%) 2024-09-13T14:29:34.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:34 smithi184 ceph-mon[92527]: Health check failed: Degraded data redundancy: 35/384 objects degraded (9.115%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:29:34.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:34 smithi184 ceph-mon[92527]: osdmap e583: 8 total, 7 up, 8 in 2024-09-13T14:29:34.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:34 smithi184 ceph-mon[92527]: osdmap e584: 8 total, 7 up, 8 in 2024-09-13T14:29:35.209 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:29:35.537 INFO:tasks.rados.rados.0.smithi151.stdout:2038: done (1 left) 2024-09-13T14:29:35.537 INFO:tasks.rados.rados.0.smithi151.stdout:2039: done (0 left) 2024-09-13T14:29:35.537 INFO:tasks.rados.rados.0.smithi151.stdout:2040: read oid 22 snap -1 2024-09-13T14:29:35.537 INFO:tasks.rados.rados.0.smithi151.stdout:2040: expect deleted 2024-09-13T14:29:35.537 INFO:tasks.rados.rados.0.smithi151.stdout:2041: delete oid 41 current snap is 265 2024-09-13T14:29:35.538 INFO:tasks.rados.rados.0.smithi151.stdout:2040: done (1 left) 2024-09-13T14:29:35.539 INFO:tasks.rados.rados.0.smithi151.stdout:2041: done (0 left) 2024-09-13T14:29:35.539 INFO:tasks.rados.rados.0.smithi151.stdout:2042: snap_create 2024-09-13T14:29:35.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:35 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:35.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:35 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:35.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:35 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:29:35.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:35 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/607895671' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:35.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:35 smithi151 ceph-mon[108966]: osdmap e585: 8 total, 7 up, 8 in 2024-09-13T14:29:35.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:35 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:35.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:35 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:35.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:35 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:29:35.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:35 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/607895671' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:35.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:35 smithi151 ceph-mon[104450]: osdmap e585: 8 total, 7 up, 8 in 2024-09-13T14:29:36.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:35 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:36.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:35 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:36.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:35 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:29:36.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:35 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/607895671' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:36.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:35 smithi184 ceph-mon[92527]: osdmap e585: 8 total, 7 up, 8 in 2024-09-13T14:29:36.536 INFO:tasks.rados.rados.0.smithi151.stdout:2042: done (0 left) 2024-09-13T14:29:36.536 INFO:tasks.rados.rados.0.smithi151.stdout:2043: read oid 25 snap -1 2024-09-13T14:29:36.536 INFO:tasks.rados.rados.0.smithi151.stdout:2043: expect (ObjNum 760 snap 263 seq_num 760) 2024-09-13T14:29:36.536 INFO:tasks.rados.rados.0.smithi151.stdout:2044: read oid 11 snap -1 2024-09-13T14:29:36.537 INFO:tasks.rados.rados.0.smithi151.stdout:2044: expect deleted 2024-09-13T14:29:36.537 INFO:tasks.rados.rados.0.smithi151.stdout:2045: snap_remove snap 259 2024-09-13T14:29:36.608 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:36 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2[130271]: 2024-09-13T14:29:36.329+0000 7f07db227640 -1 Falling back to public interface 2024-09-13T14:29:36.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:36 smithi184 ceph-mon[92527]: pgmap v495: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 183 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.3 MiB/s rd, 583 KiB/s wr, 7 op/s; 50/384 objects degraded (13.021%) 2024-09-13T14:29:36.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:36 smithi184 ceph-mon[92527]: from='client.44663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:36.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:36 smithi184 ceph-mon[92527]: from='client.34785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:36.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:36 smithi184 ceph-mon[92527]: osdmap e586: 8 total, 7 up, 8 in 2024-09-13T14:29:36.987 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:36 smithi151 ceph-mon[108966]: pgmap v495: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 183 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.3 MiB/s rd, 583 KiB/s wr, 7 op/s; 50/384 objects degraded (13.021%) 2024-09-13T14:29:36.987 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:36 smithi151 ceph-mon[108966]: from='client.44663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:36.987 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:36 smithi151 ceph-mon[108966]: from='client.34785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:36.987 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:36 smithi151 ceph-mon[108966]: osdmap e586: 8 total, 7 up, 8 in 2024-09-13T14:29:36.988 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:36 smithi151 ceph-mon[104450]: pgmap v495: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 183 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.3 MiB/s rd, 583 KiB/s wr, 7 op/s; 50/384 objects degraded (13.021%) 2024-09-13T14:29:36.988 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:36 smithi151 ceph-mon[104450]: from='client.44663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:36.988 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:36 smithi151 ceph-mon[104450]: from='client.34785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:36.988 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:36 smithi151 ceph-mon[104450]: osdmap e586: 8 total, 7 up, 8 in 2024-09-13T14:29:37.537 INFO:tasks.rados.rados.0.smithi151.stdout:2045: done (2 left) 2024-09-13T14:29:37.537 INFO:tasks.rados.rados.0.smithi151.stdout:2046: read oid 31 snap -1 2024-09-13T14:29:37.538 INFO:tasks.rados.rados.0.smithi151.stdout:2046: expect deleted 2024-09-13T14:29:37.538 INFO:tasks.rados.rados.0.smithi151.stdout:2047: read oid 21 snap -1 2024-09-13T14:29:37.538 INFO:tasks.rados.rados.0.smithi151.stdout:2047: expect deleted 2024-09-13T14:29:37.538 INFO:tasks.rados.rados.0.smithi151.stdout:2048: read oid 47 snap -1 2024-09-13T14:29:37.538 INFO:tasks.rados.rados.0.smithi151.stdout:2048: expect (ObjNum 659 snap 213 seq_num 659) 2024-09-13T14:29:37.538 INFO:tasks.rados.rados.0.smithi151.stdout:2049: delete oid 42 current snap is 266 2024-09-13T14:29:37.538 INFO:tasks.rados.rados.0.smithi151.stdout:2043: expect (ObjNum 760 snap 263 seq_num 760) 2024-09-13T14:29:37.764 INFO:tasks.rados.rados.0.smithi151.stdout:2048: expect (ObjNum 659 snap 213 seq_num 659) 2024-09-13T14:29:37.872 INFO:tasks.rados.rados.0.smithi151.stdout:2043: done (5 left) 2024-09-13T14:29:37.872 INFO:tasks.rados.rados.0.smithi151.stdout:2044: done (4 left) 2024-09-13T14:29:37.872 INFO:tasks.rados.rados.0.smithi151.stdout:2046: done (3 left) 2024-09-13T14:29:37.873 INFO:tasks.rados.rados.0.smithi151.stdout:2047: done (2 left) 2024-09-13T14:29:37.873 INFO:tasks.rados.rados.0.smithi151.stdout:2048: done (1 left) 2024-09-13T14:29:37.873 INFO:tasks.rados.rados.0.smithi151.stdout:2049: done (0 left) 2024-09-13T14:29:37.873 INFO:tasks.rados.rados.0.smithi151.stdout:2050: write oid 46 current snap is 266 2024-09-13T14:29:37.873 INFO:tasks.rados.rados.0.smithi151.stdout:2050: seq_num 768 ranges {507776=742885,1991465=485600} 2024-09-13T14:29:37.880 INFO:tasks.rados.rados.0.smithi151.stdout:2050: writing smithi15186934-46 from 507776 to 1250661 tid 1 2024-09-13T14:29:37.887 INFO:tasks.rados.rados.0.smithi151.stdout:2050: writing smithi15186934-46 from 1991465 to 2477065 tid 2 2024-09-13T14:29:37.887 INFO:tasks.rados.rados.0.smithi151.stdout:2051: read oid 2 snap 250 2024-09-13T14:29:37.887 INFO:tasks.rados.rados.0.smithi151.stdout:2051: expect (ObjNum 722 snap 249 seq_num 722) 2024-09-13T14:29:37.888 INFO:tasks.rados.rados.0.smithi151.stdout:2052: snap_create 2024-09-13T14:29:37.888 INFO:tasks.rados.rados.0.smithi151.stdout:2050: finishing write tid 1 to smithi15186934-46 2024-09-13T14:29:37.890 INFO:tasks.rados.rados.0.smithi151.stdout:2050: finishing write tid 2 to smithi15186934-46 2024-09-13T14:29:37.890 INFO:tasks.rados.rados.0.smithi151.stdout:2050: finishing write tid 3 to smithi15186934-46 2024-09-13T14:29:37.890 INFO:tasks.rados.rados.0.smithi151.stdout:2050: finishing write tid 4 to smithi15186934-46 2024-09-13T14:29:37.890 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 46 v 719 (ObjNum 768 snap 266 seq_num 768) dirty exists 2024-09-13T14:29:37.890 INFO:tasks.rados.rados.0.smithi151.stdout:2050: left oid 46 (ObjNum 768 snap 266 seq_num 768) 2024-09-13T14:29:37.892 INFO:tasks.rados.rados.0.smithi151.stdout:2051: expect (ObjNum 722 snap 249 seq_num 722) 2024-09-13T14:29:38.098 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:29:38.623 INFO:tasks.rados.rados.0.smithi151.stdout:2050: done (2 left) 2024-09-13T14:29:38.623 INFO:tasks.rados.rados.0.smithi151.stdout:2051: done (1 left) 2024-09-13T14:29:38.623 INFO:tasks.rados.rados.0.smithi151.stdout:2052: done (0 left) 2024-09-13T14:29:38.623 INFO:tasks.rados.rados.0.smithi151.stdout:2053: snap_remove snap 263 2024-09-13T14:29:38.725 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:38 smithi184 ceph-mon[92527]: pgmap v498: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 193 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 1.2 MiB/s wr, 10 op/s; 50/399 objects degraded (12.531%) 2024-09-13T14:29:38.725 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:38 smithi184 ceph-mon[92527]: osdmap e587: 8 total, 7 up, 8 in 2024-09-13T14:29:38.725 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:38 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3107782473' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:38.725 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:38 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:38.725 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:38 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:38.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:38 smithi151 ceph-mon[104450]: pgmap v498: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 193 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 1.2 MiB/s wr, 10 op/s; 50/399 objects degraded (12.531%) 2024-09-13T14:29:38.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:38 smithi151 ceph-mon[104450]: osdmap e587: 8 total, 7 up, 8 in 2024-09-13T14:29:38.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:38 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3107782473' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:38.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:38 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:38.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:38 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:38.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:38 smithi151 ceph-mon[108966]: pgmap v498: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 193 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 1.2 MiB/s wr, 10 op/s; 50/399 objects degraded (12.531%) 2024-09-13T14:29:38.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:38 smithi151 ceph-mon[108966]: osdmap e587: 8 total, 7 up, 8 in 2024-09-13T14:29:38.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:38 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3107782473' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:38.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:38 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:38.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:38 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:39.526 INFO:tasks.rados.rados.0.smithi151.stdout:2053: done (0 left) 2024-09-13T14:29:39.526 INFO:tasks.rados.rados.0.smithi151.stdout:2054: rollback oid 10 current snap is 267 2024-09-13T14:29:39.526 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 10 to 260 2024-09-13T14:29:39.526 INFO:tasks.rados.rados.0.smithi151.stdout:2055: read oid 44 snap 223 2024-09-13T14:29:39.526 INFO:tasks.rados.rados.0.smithi151.stdout:2055: expect deleted 2024-09-13T14:29:39.526 INFO:tasks.rados.rados.0.smithi151.stdout:2056: rollback oid 43 current snap is 267 2024-09-13T14:29:39.526 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 43 to 255 2024-09-13T14:29:39.526 INFO:tasks.rados.rados.0.smithi151.stdout:2057: snap_create 2024-09-13T14:29:39.527 INFO:tasks.rados.rados.0.smithi151.stdout:2054: finishing rollback tid 1 to smithi15186934-10 2024-09-13T14:29:39.528 INFO:tasks.rados.rados.0.smithi151.stdout:2054: finishing rollback tid 2 to smithi15186934-10 2024-09-13T14:29:39.528 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 721 (ObjNum 738 snap 254 seq_num 738) dirty exists 2024-09-13T14:29:39.530 INFO:tasks.rados.rados.0.smithi151.stdout:2056: finishing rollback tid 1 to smithi15186934-43 2024-09-13T14:29:39.530 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 43 v 0 (ObjNum 129 snap 0 seq_num 3092873816) dirty dne 2024-09-13T14:29:39.680 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:39 smithi151 ceph-mon[104450]: from='client.34797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:39.681 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:39 smithi151 ceph-mon[104450]: from='client.34803 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:39.681 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:39 smithi151 ceph-mon[104450]: osdmap e588: 8 total, 7 up, 8 in 2024-09-13T14:29:39.681 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:39 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:39.681 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:39 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:39.681 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:39 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:39.681 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:39 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:39.681 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:39 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 50/399 objects degraded (12.531%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:29:39.681 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:39 smithi151 ceph-mon[104450]: osdmap e589: 8 total, 7 up, 8 in 2024-09-13T14:29:39.681 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:39 smithi151 ceph-mon[108966]: from='client.34797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:39.681 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:39 smithi151 ceph-mon[108966]: from='client.34803 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:39.682 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:39 smithi151 ceph-mon[108966]: osdmap e588: 8 total, 7 up, 8 in 2024-09-13T14:29:39.682 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:39 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:39.682 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:39 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:39.682 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:39 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:39.682 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:39 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:39.682 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:39 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 50/399 objects degraded (12.531%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:29:39.682 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:39 smithi151 ceph-mon[108966]: osdmap e589: 8 total, 7 up, 8 in 2024-09-13T14:29:40.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:39 smithi184 ceph-mon[92527]: from='client.34797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:40.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:39 smithi184 ceph-mon[92527]: from='client.34803 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:40.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:39 smithi184 ceph-mon[92527]: osdmap e588: 8 total, 7 up, 8 in 2024-09-13T14:29:40.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:39 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:40.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:39 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:40.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:39 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:40.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:39 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:40.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:39 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 50/399 objects degraded (12.531%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:29:40.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:39 smithi184 ceph-mon[92527]: osdmap e589: 8 total, 7 up, 8 in 2024-09-13T14:29:40.244 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:39 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2[130271]: 2024-09-13T14:29:39.956+0000 7f07db227640 -1 osd.2 578 log_to_monitors true 2024-09-13T14:29:40.528 INFO:tasks.rados.rados.0.smithi151.stdout:2054: done (3 left) 2024-09-13T14:29:40.528 INFO:tasks.rados.rados.0.smithi151.stdout:2055: done (2 left) 2024-09-13T14:29:40.528 INFO:tasks.rados.rados.0.smithi151.stdout:2056: done (1 left) 2024-09-13T14:29:40.528 INFO:tasks.rados.rados.0.smithi151.stdout:2057: done (0 left) 2024-09-13T14:29:40.529 INFO:tasks.rados.rados.0.smithi151.stdout:2058: snap_remove snap 266 2024-09-13T14:29:40.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:29:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:29:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:40.858 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 14:29:40 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2[130271]: 2024-09-13T14:29:40.545+0000 7f07d2fd1640 -1 osd.2 578 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:29:40.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:40 smithi151 ceph-mon[104450]: pgmap v501: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 193 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 4.0 MiB/s rd, 1.2 MiB/s wr, 11 op/s; 50/405 objects degraded (12.346%) 2024-09-13T14:29:40.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:40 smithi151 ceph-mon[104450]: from='osd.2 [v2:172.21.15.151:6818/322435630,v1:172.21.15.151:6819/322435630]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-13T14:29:40.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:40 smithi151 ceph-mon[104450]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-13T14:29:40.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:40 smithi151 ceph-mon[104450]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T14:29:40.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:40 smithi151 ceph-mon[104450]: osdmap e590: 8 total, 7 up, 8 in 2024-09-13T14:29:40.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:40 smithi151 ceph-mon[104450]: from='osd.2 [v2:172.21.15.151:6818/322435630,v1:172.21.15.151:6819/322435630]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-13T14:29:40.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:40 smithi151 ceph-mon[104450]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-13T14:29:40.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:40 smithi151 ceph-mon[108966]: pgmap v501: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 193 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 4.0 MiB/s rd, 1.2 MiB/s wr, 11 op/s; 50/405 objects degraded (12.346%) 2024-09-13T14:29:40.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:40 smithi151 ceph-mon[108966]: from='osd.2 [v2:172.21.15.151:6818/322435630,v1:172.21.15.151:6819/322435630]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-13T14:29:40.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:40 smithi151 ceph-mon[108966]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-13T14:29:40.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:40 smithi151 ceph-mon[108966]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T14:29:40.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:40 smithi151 ceph-mon[108966]: osdmap e590: 8 total, 7 up, 8 in 2024-09-13T14:29:40.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:40 smithi151 ceph-mon[108966]: from='osd.2 [v2:172.21.15.151:6818/322435630,v1:172.21.15.151:6819/322435630]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-13T14:29:40.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:40 smithi151 ceph-mon[108966]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-13T14:29:40.991 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:29:41.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:40 smithi184 ceph-mon[92527]: pgmap v501: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 193 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 4.0 MiB/s rd, 1.2 MiB/s wr, 11 op/s; 50/405 objects degraded (12.346%) 2024-09-13T14:29:41.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:40 smithi184 ceph-mon[92527]: from='osd.2 [v2:172.21.15.151:6818/322435630,v1:172.21.15.151:6819/322435630]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-13T14:29:41.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:40 smithi184 ceph-mon[92527]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-13T14:29:41.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:40 smithi184 ceph-mon[92527]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T14:29:41.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:40 smithi184 ceph-mon[92527]: osdmap e590: 8 total, 7 up, 8 in 2024-09-13T14:29:41.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:40 smithi184 ceph-mon[92527]: from='osd.2 [v2:172.21.15.151:6818/322435630,v1:172.21.15.151:6819/322435630]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-13T14:29:41.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:40 smithi184 ceph-mon[92527]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-13T14:29:41.532 INFO:tasks.rados.rados.0.smithi151.stdout:2058: done (0 left) 2024-09-13T14:29:41.533 INFO:tasks.rados.rados.0.smithi151.stdout:2059: write oid 3 current snap is 268 2024-09-13T14:29:41.533 INFO:tasks.rados.rados.0.smithi151.stdout:2059: seq_num 769 ranges {524583=418134,1649947=629258,2834752=89831} 2024-09-13T14:29:41.538 INFO:tasks.rados.rados.0.smithi151.stdout:2059: writing smithi15186934-3 from 524583 to 942717 tid 1 2024-09-13T14:29:41.547 INFO:tasks.rados.rados.0.smithi151.stdout:2059: writing smithi15186934-3 from 1649947 to 2279205 tid 2 2024-09-13T14:29:41.551 INFO:tasks.rados.rados.0.smithi151.stdout:2059: writing smithi15186934-3 from 2834752 to 2924583 tid 3 2024-09-13T14:29:41.551 INFO:tasks.rados.rados.0.smithi151.stdout:2060: snap_create 2024-09-13T14:29:42.005 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:41 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3900445445' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:42.005 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:41 smithi184 ceph-mon[92527]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:29:42.005 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:41 smithi184 ceph-mon[92527]: osd.2 [v2:172.21.15.151:6818/322435630,v1:172.21.15.151:6819/322435630] boot 2024-09-13T14:29:42.006 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:41 smithi184 ceph-mon[92527]: osdmap e591: 8 total, 8 up, 8 in 2024-09-13T14:29:42.006 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:41 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T14:29:42.018 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:41 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3900445445' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:42.018 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:41 smithi151 ceph-mon[104450]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:29:42.018 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:41 smithi151 ceph-mon[104450]: osd.2 [v2:172.21.15.151:6818/322435630,v1:172.21.15.151:6819/322435630] boot 2024-09-13T14:29:42.018 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:41 smithi151 ceph-mon[104450]: osdmap e591: 8 total, 8 up, 8 in 2024-09-13T14:29:42.018 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:41 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T14:29:42.019 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:41 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3900445445' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:42.019 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:41 smithi151 ceph-mon[108966]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:29:42.019 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:41 smithi151 ceph-mon[108966]: osd.2 [v2:172.21.15.151:6818/322435630,v1:172.21.15.151:6819/322435630] boot 2024-09-13T14:29:42.019 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:41 smithi151 ceph-mon[108966]: osdmap e591: 8 total, 8 up, 8 in 2024-09-13T14:29:42.019 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:41 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T14:29:42.531 INFO:tasks.rados.rados.0.smithi151.stdout:2060: done (1 left) 2024-09-13T14:29:42.531 INFO:tasks.rados.rados.0.smithi151.stdout:2061: delete oid 43 current snap is 269 2024-09-13T14:29:42.536 INFO:tasks.rados.rados.0.smithi151.stdout:2061: done (1 left) 2024-09-13T14:29:42.536 INFO:tasks.rados.rados.0.smithi151.stdout:2062: rollback oid 25 current snap is 269 2024-09-13T14:29:42.536 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 25 to 223 2024-09-13T14:29:42.536 INFO:tasks.rados.rados.0.smithi151.stdout:2063: snap_remove snap 249 2024-09-13T14:29:42.629 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:29:42 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:29:42.531+0000 7fc94c153640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-13T14:29:42.732 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:42 smithi184 ceph-mon[92527]: from='client.44702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:42.732 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:42 smithi184 ceph-mon[92527]: pgmap v504: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 193 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 0 B/s wr, 2 op/s; 50/405 objects degraded (12.346%) 2024-09-13T14:29:42.733 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:42 smithi184 ceph-mon[92527]: from='client.34824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:42.733 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:42 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:42.733 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:42 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:42.733 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:42 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:42.733 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:42 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:29:42.733 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:42 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:42.733 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:42 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:29:42.733 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:42 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:29:42.733 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:42 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:29:42.733 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:42 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:29:42.733 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:42 smithi184 ceph-mon[92527]: osdmap e592: 8 total, 8 up, 8 in 2024-09-13T14:29:42.733 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:42 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:29:43.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:42 smithi151 ceph-mon[108966]: from='client.44702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:43.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:42 smithi151 ceph-mon[108966]: pgmap v504: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 193 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 0 B/s wr, 2 op/s; 50/405 objects degraded (12.346%) 2024-09-13T14:29:43.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:42 smithi151 ceph-mon[108966]: from='client.34824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:43.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:42 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:43.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:42 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:43.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:42 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:43.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:42 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:29:43.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:42 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:43.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:42 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:29:43.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:42 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:29:43.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:42 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:29:43.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:42 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:29:43.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:42 smithi151 ceph-mon[108966]: osdmap e592: 8 total, 8 up, 8 in 2024-09-13T14:29:43.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:42 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:29:43.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:42 smithi151 ceph-mon[104450]: from='client.44702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:43.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:42 smithi151 ceph-mon[104450]: pgmap v504: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 193 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 0 B/s wr, 2 op/s; 50/405 objects degraded (12.346%) 2024-09-13T14:29:43.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:42 smithi151 ceph-mon[104450]: from='client.34824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:43.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:42 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:43.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:42 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:43.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:42 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:43.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:42 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:29:43.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:42 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:43.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:42 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:29:43.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:42 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:29:43.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:42 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:29:43.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:42 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:29:43.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:42 smithi151 ceph-mon[104450]: osdmap e592: 8 total, 8 up, 8 in 2024-09-13T14:29:43.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:42 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:29:43.532 INFO:tasks.rados.rados.0.smithi151.stdout:2063: done (2 left) 2024-09-13T14:29:43.532 INFO:tasks.rados.rados.0.smithi151.stdout:2064: snap_create 2024-09-13T14:29:43.532 INFO:tasks.rados.rados.0.smithi151.stdout:2062: finishing rollback tid 0 to smithi15186934-25 2024-09-13T14:29:43.532 INFO:tasks.rados.rados.0.smithi151.stdout:2062: finishing rollback tid 1 to smithi15186934-25 2024-09-13T14:29:43.532 INFO:tasks.rados.rados.0.smithi151.stdout:2062: finishing rollback tid 2 to smithi15186934-25 2024-09-13T14:29:43.533 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 1110 (ObjNum 670 snap 216 seq_num 670) dirty exists 2024-09-13T14:29:43.533 INFO:tasks.rados.rados.0.smithi151.stdout:2059: finishing write tid 1 to smithi15186934-3 2024-09-13T14:29:43.533 INFO:tasks.rados.rados.0.smithi151.stdout:2059: finishing write tid 2 to smithi15186934-3 2024-09-13T14:29:43.533 INFO:tasks.rados.rados.0.smithi151.stdout:2059: finishing write tid 3 to smithi15186934-3 2024-09-13T14:29:43.533 INFO:tasks.rados.rados.0.smithi151.stdout:2059: finishing write tid 4 to smithi15186934-3 2024-09-13T14:29:43.533 INFO:tasks.rados.rados.0.smithi151.stdout:2059: finishing write tid 5 to smithi15186934-3 2024-09-13T14:29:43.533 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 746 (ObjNum 769 snap 268 seq_num 769) dirty exists 2024-09-13T14:29:43.533 INFO:tasks.rados.rados.0.smithi151.stdout:2059: left oid 3 (ObjNum 769 snap 268 seq_num 769) 2024-09-13T14:29:43.851 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:29:44.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:43 smithi184 ceph-mon[92527]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:29:44.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:43 smithi184 ceph-mon[92527]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-13T14:29:44.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:43 smithi184 ceph-mon[92527]: osdmap e593: 8 total, 8 up, 8 in 2024-09-13T14:29:44.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:43 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2210807909' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:44.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:43 smithi151 ceph-mon[108966]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:29:44.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:43 smithi151 ceph-mon[108966]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-13T14:29:44.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:43 smithi151 ceph-mon[108966]: osdmap e593: 8 total, 8 up, 8 in 2024-09-13T14:29:44.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:43 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2210807909' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:44.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:43 smithi151 ceph-mon[104450]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:29:44.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:43 smithi151 ceph-mon[104450]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-13T14:29:44.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:43 smithi151 ceph-mon[104450]: osdmap e593: 8 total, 8 up, 8 in 2024-09-13T14:29:44.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:43 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2210807909' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:44.533 INFO:tasks.rados.rados.0.smithi151.stdout:2059: done (2 left) 2024-09-13T14:29:44.534 INFO:tasks.rados.rados.0.smithi151.stdout:2062: done (1 left) 2024-09-13T14:29:44.534 INFO:tasks.rados.rados.0.smithi151.stdout:2064: done (0 left) 2024-09-13T14:29:44.534 INFO:tasks.rados.rados.0.smithi151.stdout:2065: write oid 8 current snap is 270 2024-09-13T14:29:44.534 INFO:tasks.rados.rados.0.smithi151.stdout:2065: seq_num 771 ranges {558197=568632,1650558=632927,2773245=184952} 2024-09-13T14:29:44.539 INFO:tasks.rados.rados.0.smithi151.stdout:2065: writing smithi15186934-8 from 558197 to 1126829 tid 1 2024-09-13T14:29:44.546 INFO:tasks.rados.rados.0.smithi151.stdout:2065: writing smithi15186934-8 from 1650558 to 2283485 tid 2 2024-09-13T14:29:44.550 INFO:tasks.rados.rados.0.smithi151.stdout:2065: writing smithi15186934-8 from 2773245 to 2958197 tid 3 2024-09-13T14:29:44.550 INFO:tasks.rados.rados.0.smithi151.stdout:2066: snap_create 2024-09-13T14:29:44.550 INFO:tasks.rados.rados.0.smithi151.stdout:2065: finishing write tid 1 to smithi15186934-8 2024-09-13T14:29:44.550 INFO:tasks.rados.rados.0.smithi151.stdout:2065: finishing write tid 2 to smithi15186934-8 2024-09-13T14:29:44.552 INFO:tasks.rados.rados.0.smithi151.stdout:2065: finishing write tid 3 to smithi15186934-8 2024-09-13T14:29:44.552 INFO:tasks.rados.rados.0.smithi151.stdout:2065: finishing write tid 4 to smithi15186934-8 2024-09-13T14:29:44.552 INFO:tasks.rados.rados.0.smithi151.stdout:2065: finishing write tid 5 to smithi15186934-8 2024-09-13T14:29:44.552 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 630 (ObjNum 771 snap 270 seq_num 771) dirty exists 2024-09-13T14:29:44.552 INFO:tasks.rados.rados.0.smithi151.stdout:2065: left oid 8 (ObjNum 771 snap 270 seq_num 771) 2024-09-13T14:29:44.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:29:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:29:44] "GET /metrics HTTP/1.1" 200 36520 "" "Prometheus/2.51.0" 2024-09-13T14:29:44.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:44 smithi151 ceph-mon[108966]: pgmap v507: 89 pgs: 16 peering, 3 active+undersized, 6 active+undersized+degraded, 64 active+clean; 198 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 300 KiB/s wr, 3 op/s; 38/408 objects degraded (9.314%) 2024-09-13T14:29:44.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:44 smithi151 ceph-mon[108966]: from='client.44720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:44.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:44 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 38/408 objects degraded (9.314%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:29:44.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:44 smithi151 ceph-mon[108966]: osdmap e594: 8 total, 8 up, 8 in 2024-09-13T14:29:44.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:44 smithi151 ceph-mon[104450]: pgmap v507: 89 pgs: 16 peering, 3 active+undersized, 6 active+undersized+degraded, 64 active+clean; 198 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 300 KiB/s wr, 3 op/s; 38/408 objects degraded (9.314%) 2024-09-13T14:29:44.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:44 smithi151 ceph-mon[104450]: from='client.44720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:44.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:44 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 38/408 objects degraded (9.314%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:29:44.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:44 smithi151 ceph-mon[104450]: osdmap e594: 8 total, 8 up, 8 in 2024-09-13T14:29:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:44 smithi184 ceph-mon[92527]: pgmap v507: 89 pgs: 16 peering, 3 active+undersized, 6 active+undersized+degraded, 64 active+clean; 198 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 300 KiB/s wr, 3 op/s; 38/408 objects degraded (9.314%) 2024-09-13T14:29:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:44 smithi184 ceph-mon[92527]: from='client.44720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:44 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 38/408 objects degraded (9.314%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:29:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:44 smithi184 ceph-mon[92527]: osdmap e594: 8 total, 8 up, 8 in 2024-09-13T14:29:45.535 INFO:tasks.rados.rados.0.smithi151.stdout:2065: done (1 left) 2024-09-13T14:29:45.535 INFO:tasks.rados.rados.0.smithi151.stdout:2066: done (0 left) 2024-09-13T14:29:45.535 INFO:tasks.rados.rados.0.smithi151.stdout:2067: write oid 10 current snap is 271 2024-09-13T14:29:45.535 INFO:tasks.rados.rados.0.smithi151.stdout:2067: seq_num 772 ranges {575004=643881,1709040=492938,2917021=57983} 2024-09-13T14:29:45.541 INFO:tasks.rados.rados.0.smithi151.stdout:2067: writing smithi15186934-10 from 575004 to 1218885 tid 1 2024-09-13T14:29:45.547 INFO:tasks.rados.rados.0.smithi151.stdout:2067: writing smithi15186934-10 from 1709040 to 2201978 tid 2 2024-09-13T14:29:45.552 INFO:tasks.rados.rados.0.smithi151.stdout:2067: writing smithi15186934-10 from 2917021 to 2975004 tid 3 2024-09-13T14:29:45.552 INFO:tasks.rados.rados.0.smithi151.stdout:2068: delete oid 47 current snap is 271 2024-09-13T14:29:45.552 INFO:tasks.rados.rados.0.smithi151.stdout:2067: finishing write tid 1 to smithi15186934-10 2024-09-13T14:29:45.552 INFO:tasks.rados.rados.0.smithi151.stdout:2067: finishing write tid 2 to smithi15186934-10 2024-09-13T14:29:45.554 INFO:tasks.rados.rados.0.smithi151.stdout:2067: finishing write tid 3 to smithi15186934-10 2024-09-13T14:29:45.555 INFO:tasks.rados.rados.0.smithi151.stdout:2067: finishing write tid 4 to smithi15186934-10 2024-09-13T14:29:45.555 INFO:tasks.rados.rados.0.smithi151.stdout:2067: finishing write tid 5 to smithi15186934-10 2024-09-13T14:29:45.555 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 730 (ObjNum 772 snap 271 seq_num 772) dirty exists 2024-09-13T14:29:45.555 INFO:tasks.rados.rados.0.smithi151.stdout:2067: left oid 10 (ObjNum 772 snap 271 seq_num 772) 2024-09-13T14:29:45.555 INFO:tasks.rados.rados.0.smithi151.stdout:2067: done (1 left) 2024-09-13T14:29:45.555 INFO:tasks.rados.rados.0.smithi151.stdout:2068: done (0 left) 2024-09-13T14:29:45.555 INFO:tasks.rados.rados.0.smithi151.stdout:2069: snap_create 2024-09-13T14:29:46.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:45 smithi184 ceph-mon[92527]: from='client.34842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:46.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:45 smithi184 ceph-mon[92527]: osdmap e595: 8 total, 8 up, 8 in 2024-09-13T14:29:46.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:45 smithi151 ceph-mon[108966]: from='client.34842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:46.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:45 smithi151 ceph-mon[108966]: osdmap e595: 8 total, 8 up, 8 in 2024-09-13T14:29:46.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:45 smithi151 ceph-mon[104450]: from='client.34842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:46.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:45 smithi151 ceph-mon[104450]: osdmap e595: 8 total, 8 up, 8 in 2024-09-13T14:29:46.537 INFO:tasks.rados.rados.0.smithi151.stdout:2069: done (0 left) 2024-09-13T14:29:46.538 INFO:tasks.rados.rados.0.smithi151.stdout:2070: rollback oid 33 current snap is 272 2024-09-13T14:29:46.538 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 33 to 271 2024-09-13T14:29:46.538 INFO:tasks.rados.rados.0.smithi151.stdout:2071: read oid 44 snap -1 2024-09-13T14:29:46.538 INFO:tasks.rados.rados.0.smithi151.stdout:2071: expect deleted 2024-09-13T14:29:46.538 INFO:tasks.rados.rados.0.smithi151.stdout:2072: write oid 23 current snap is 272 2024-09-13T14:29:46.538 INFO:tasks.rados.rados.0.smithi151.stdout:2072: seq_num 774 ranges {608618=794379,2109651=780254} 2024-09-13T14:29:46.545 INFO:tasks.rados.rados.0.smithi151.stdout:2072: writing smithi15186934-23 from 608618 to 1402997 tid 1 2024-09-13T14:29:46.553 INFO:tasks.rados.rados.0.smithi151.stdout:2072: writing smithi15186934-23 from 2109651 to 2889905 tid 2 2024-09-13T14:29:46.554 INFO:tasks.rados.rados.0.smithi151.stdout:2073: snap_remove snap 238 2024-09-13T14:29:46.715 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:29:47.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:46 smithi184 ceph-mon[92527]: pgmap v510: 89 pgs: 16 peering, 2 active+undersized, 5 active+undersized+degraded, 66 active+clean; 198 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 300 KiB/s wr, 3 op/s; 36/408 objects degraded (8.824%) 2024-09-13T14:29:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:46 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2784378811' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:46 smithi184 ceph-mon[92527]: osdmap e596: 8 total, 8 up, 8 in 2024-09-13T14:29:47.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:46 smithi151 ceph-mon[104450]: pgmap v510: 89 pgs: 16 peering, 2 active+undersized, 5 active+undersized+degraded, 66 active+clean; 198 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 300 KiB/s wr, 3 op/s; 36/408 objects degraded (8.824%) 2024-09-13T14:29:47.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:47.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:47.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:46 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2784378811' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:47.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:46 smithi151 ceph-mon[104450]: osdmap e596: 8 total, 8 up, 8 in 2024-09-13T14:29:47.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:46 smithi151 ceph-mon[108966]: pgmap v510: 89 pgs: 16 peering, 2 active+undersized, 5 active+undersized+degraded, 66 active+clean; 198 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 300 KiB/s wr, 3 op/s; 36/408 objects degraded (8.824%) 2024-09-13T14:29:47.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:47.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:47.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:46 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2784378811' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:47.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:46 smithi151 ceph-mon[108966]: osdmap e596: 8 total, 8 up, 8 in 2024-09-13T14:29:47.538 INFO:tasks.rados.rados.0.smithi151.stdout:2073: done (3 left) 2024-09-13T14:29:47.538 INFO:tasks.rados.rados.0.smithi151.stdout:2074: read oid 7 snap 255 2024-09-13T14:29:47.538 INFO:tasks.rados.rados.0.smithi151.stdout:2074: expect (ObjNum 718 snap 249 seq_num 718) 2024-09-13T14:29:47.538 INFO:tasks.rados.rados.0.smithi151.stdout:2075: read oid 20 snap -1 2024-09-13T14:29:47.538 INFO:tasks.rados.rados.0.smithi151.stdout:2075: expect (ObjNum 763 snap 264 seq_num 763) 2024-09-13T14:29:47.538 INFO:tasks.rados.rados.0.smithi151.stdout:2076: delete oid 11 current snap is 272 2024-09-13T14:29:47.538 INFO:tasks.rados.rados.0.smithi151.stdout:2070: finishing rollback tid 0 to smithi15186934-33 2024-09-13T14:29:47.538 INFO:tasks.rados.rados.0.smithi151.stdout:2070: finishing rollback tid 1 to smithi15186934-33 2024-09-13T14:29:47.539 INFO:tasks.rados.rados.0.smithi151.stdout:2070: finishing rollback tid 2 to smithi15186934-33 2024-09-13T14:29:47.539 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 33 v 779 (ObjNum 746 snap 257 seq_num 746) dirty exists 2024-09-13T14:29:47.539 INFO:tasks.rados.rados.0.smithi151.stdout:2072: finishing write tid 1 to smithi15186934-23 2024-09-13T14:29:47.539 INFO:tasks.rados.rados.0.smithi151.stdout:2072: finishing write tid 2 to smithi15186934-23 2024-09-13T14:29:47.539 INFO:tasks.rados.rados.0.smithi151.stdout:2072: finishing write tid 3 to smithi15186934-23 2024-09-13T14:29:47.539 INFO:tasks.rados.rados.0.smithi151.stdout:2072: finishing write tid 4 to smithi15186934-23 2024-09-13T14:29:47.539 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 23 v 943 (ObjNum 774 snap 272 seq_num 774) dirty exists 2024-09-13T14:29:47.539 INFO:tasks.rados.rados.0.smithi151.stdout:2072: left oid 23 (ObjNum 774 snap 272 seq_num 774) 2024-09-13T14:29:47.545 INFO:tasks.rados.rados.0.smithi151.stdout:2070: done (5 left) 2024-09-13T14:29:47.545 INFO:tasks.rados.rados.0.smithi151.stdout:2071: done (4 left) 2024-09-13T14:29:47.545 INFO:tasks.rados.rados.0.smithi151.stdout:2072: done (3 left) 2024-09-13T14:29:47.545 INFO:tasks.rados.rados.0.smithi151.stdout:2076: done (2 left) 2024-09-13T14:29:47.545 INFO:tasks.rados.rados.0.smithi151.stdout:2077: snap_remove snap 267 2024-09-13T14:29:48.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:47 smithi184 ceph-mon[92527]: from='client.44738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:48.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:47 smithi184 ceph-mon[92527]: osdmap e597: 8 total, 8 up, 8 in 2024-09-13T14:29:48.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:47 smithi151 ceph-mon[104450]: from='client.44738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:48.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:47 smithi151 ceph-mon[104450]: osdmap e597: 8 total, 8 up, 8 in 2024-09-13T14:29:48.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:47 smithi151 ceph-mon[108966]: from='client.44738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:48.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:47 smithi151 ceph-mon[108966]: osdmap e597: 8 total, 8 up, 8 in 2024-09-13T14:29:48.547 INFO:tasks.rados.rados.0.smithi151.stdout:2074: expect (ObjNum 718 snap 249 seq_num 718) 2024-09-13T14:29:48.628 INFO:tasks.rados.rados.0.smithi151.stdout:2075: expect (ObjNum 763 snap 264 seq_num 763) 2024-09-13T14:29:48.720 INFO:tasks.rados.rados.0.smithi151.stdout:2074: done (2 left) 2024-09-13T14:29:48.720 INFO:tasks.rados.rados.0.smithi151.stdout:2075: done (1 left) 2024-09-13T14:29:48.720 INFO:tasks.rados.rados.0.smithi151.stdout:2077: done (0 left) 2024-09-13T14:29:48.720 INFO:tasks.rados.rados.0.smithi151.stdout:2078: snap_remove snap 264 2024-09-13T14:29:49.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:48 smithi184 ceph-mon[92527]: from='client.44744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:49.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:48 smithi184 ceph-mon[92527]: pgmap v513: 89 pgs: 89 active+clean; 206 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 1.3 MiB/s wr, 5 op/s; 1.3 MiB/s, 0 objects/s recovering 2024-09-13T14:29:49.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:48 smithi184 ceph-mon[92527]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 36/408 objects degraded (8.824%), 5 pgs degraded) 2024-09-13T14:29:49.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:48 smithi184 ceph-mon[92527]: Cluster is now healthy 2024-09-13T14:29:49.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:48 smithi184 ceph-mon[92527]: osdmap e598: 8 total, 8 up, 8 in 2024-09-13T14:29:49.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:48 smithi151 ceph-mon[104450]: from='client.44744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:49.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:48 smithi151 ceph-mon[104450]: pgmap v513: 89 pgs: 89 active+clean; 206 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 1.3 MiB/s wr, 5 op/s; 1.3 MiB/s, 0 objects/s recovering 2024-09-13T14:29:49.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:48 smithi151 ceph-mon[104450]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 36/408 objects degraded (8.824%), 5 pgs degraded) 2024-09-13T14:29:49.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:48 smithi151 ceph-mon[104450]: Cluster is now healthy 2024-09-13T14:29:49.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:48 smithi151 ceph-mon[104450]: osdmap e598: 8 total, 8 up, 8 in 2024-09-13T14:29:49.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:48 smithi151 ceph-mon[108966]: from='client.44744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:49.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:48 smithi151 ceph-mon[108966]: pgmap v513: 89 pgs: 89 active+clean; 206 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 1.3 MiB/s wr, 5 op/s; 1.3 MiB/s, 0 objects/s recovering 2024-09-13T14:29:49.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:48 smithi151 ceph-mon[108966]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 36/408 objects degraded (8.824%), 5 pgs degraded) 2024-09-13T14:29:49.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:48 smithi151 ceph-mon[108966]: Cluster is now healthy 2024-09-13T14:29:49.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:48 smithi151 ceph-mon[108966]: osdmap e598: 8 total, 8 up, 8 in 2024-09-13T14:29:49.528 INFO:tasks.rados.rados.0.smithi151.stdout:2078: done (0 left) 2024-09-13T14:29:49.528 INFO:tasks.rados.rados.0.smithi151.stdout:2079: write oid 20 current snap is 272 2024-09-13T14:29:49.528 INFO:tasks.rados.rados.0.smithi151.stdout:2079: seq_num 776 ranges {642232=661230,1942968=783923} 2024-09-13T14:29:49.536 INFO:tasks.rados.rados.0.smithi151.stdout:2079: writing smithi15186934-20 from 642232 to 1303462 tid 1 2024-09-13T14:29:49.544 INFO:tasks.rados.rados.0.smithi151.stdout:2079: writing smithi15186934-20 from 1942968 to 2726891 tid 2 2024-09-13T14:29:49.544 INFO:tasks.rados.rados.0.smithi151.stdout:2080: snap_create 2024-09-13T14:29:49.544 INFO:tasks.rados.rados.0.smithi151.stdout:2079: finishing write tid 1 to smithi15186934-20 2024-09-13T14:29:49.548 INFO:tasks.rados.rados.0.smithi151.stdout:2079: finishing write tid 2 to smithi15186934-20 2024-09-13T14:29:49.549 INFO:tasks.rados.rados.0.smithi151.stdout:2079: finishing write tid 3 to smithi15186934-20 2024-09-13T14:29:49.549 INFO:tasks.rados.rados.0.smithi151.stdout:2079: finishing write tid 4 to smithi15186934-20 2024-09-13T14:29:49.549 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 472 (ObjNum 776 snap 272 seq_num 776) dirty exists 2024-09-13T14:29:49.549 INFO:tasks.rados.rados.0.smithi151.stdout:2079: left oid 20 (ObjNum 776 snap 272 seq_num 776) 2024-09-13T14:29:49.558 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:29:49.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:49 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/121095400' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:49.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:49 smithi151 ceph-mon[104450]: osdmap e599: 8 total, 8 up, 8 in 2024-09-13T14:29:49.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:49 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/121095400' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:49.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:49 smithi151 ceph-mon[108966]: osdmap e599: 8 total, 8 up, 8 in 2024-09-13T14:29:50.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:49 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/121095400' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:50.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:49 smithi184 ceph-mon[92527]: osdmap e599: 8 total, 8 up, 8 in 2024-09-13T14:29:50.529 INFO:tasks.rados.rados.0.smithi151.stdout:2079: done (1 left) 2024-09-13T14:29:50.529 INFO:tasks.rados.rados.0.smithi151.stdout:2080: done (0 left) 2024-09-13T14:29:50.530 INFO:tasks.rados.rados.0.smithi151.stdout:2081: snap_remove snap 223 2024-09-13T14:29:50.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:29:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:29:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:29:51.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:50 smithi184 ceph-mon[92527]: pgmap v516: 89 pgs: 89 active+clean; 206 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 1.3 MiB/s wr, 6 op/s; 1.3 MiB/s, 0 objects/s recovering 2024-09-13T14:29:51.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:50 smithi184 ceph-mon[92527]: from='client.34872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:51.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:50 smithi184 ceph-mon[92527]: from='client.44762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:51.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:50 smithi184 ceph-mon[92527]: osdmap e600: 8 total, 8 up, 8 in 2024-09-13T14:29:51.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:50 smithi151 ceph-mon[104450]: pgmap v516: 89 pgs: 89 active+clean; 206 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 1.3 MiB/s wr, 6 op/s; 1.3 MiB/s, 0 objects/s recovering 2024-09-13T14:29:51.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:50 smithi151 ceph-mon[104450]: from='client.34872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:51.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:50 smithi151 ceph-mon[104450]: from='client.44762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:51.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:50 smithi151 ceph-mon[104450]: osdmap e600: 8 total, 8 up, 8 in 2024-09-13T14:29:51.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:50 smithi151 ceph-mon[108966]: pgmap v516: 89 pgs: 89 active+clean; 206 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 1.3 MiB/s wr, 6 op/s; 1.3 MiB/s, 0 objects/s recovering 2024-09-13T14:29:51.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:50 smithi151 ceph-mon[108966]: from='client.34872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:51.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:50 smithi151 ceph-mon[108966]: from='client.44762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:51.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:50 smithi151 ceph-mon[108966]: osdmap e600: 8 total, 8 up, 8 in 2024-09-13T14:29:51.533 INFO:tasks.rados.rados.0.smithi151.stdout:2081: done (0 left) 2024-09-13T14:29:51.533 INFO:tasks.rados.rados.0.smithi151.stdout:2082: write oid 8 current snap is 273 2024-09-13T14:29:51.533 INFO:tasks.rados.rados.0.smithi151.stdout:2082: seq_num 777 ranges {659039=736479,2001450=643934} 2024-09-13T14:29:51.541 INFO:tasks.rados.rados.0.smithi151.stdout:2082: writing smithi15186934-8 from 659039 to 1395518 tid 1 2024-09-13T14:29:51.550 INFO:tasks.rados.rados.0.smithi151.stdout:2082: writing smithi15186934-8 from 2001450 to 2645384 tid 2 2024-09-13T14:29:51.550 INFO:tasks.rados.rados.0.smithi151.stdout:2083: delete oid 15 current snap is 273 2024-09-13T14:29:51.550 INFO:tasks.rados.rados.0.smithi151.stdout:2082: finishing write tid 1 to smithi15186934-8 2024-09-13T14:29:51.551 INFO:tasks.rados.rados.0.smithi151.stdout:2083: done (1 left) 2024-09-13T14:29:51.552 INFO:tasks.rados.rados.0.smithi151.stdout:2084: delete oid 34 current snap is 273 2024-09-13T14:29:51.553 INFO:tasks.rados.rados.0.smithi151.stdout:2082: finishing write tid 2 to smithi15186934-8 2024-09-13T14:29:51.553 INFO:tasks.rados.rados.0.smithi151.stdout:2082: finishing write tid 3 to smithi15186934-8 2024-09-13T14:29:51.553 INFO:tasks.rados.rados.0.smithi151.stdout:2082: finishing write tid 4 to smithi15186934-8 2024-09-13T14:29:51.553 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 646 (ObjNum 777 snap 273 seq_num 777) dirty exists 2024-09-13T14:29:51.554 INFO:tasks.rados.rados.0.smithi151.stdout:2082: left oid 8 (ObjNum 777 snap 273 seq_num 777) 2024-09-13T14:29:51.554 INFO:tasks.rados.rados.0.smithi151.stdout:2082: done (1 left) 2024-09-13T14:29:51.554 INFO:tasks.rados.rados.0.smithi151.stdout:2084: done (0 left) 2024-09-13T14:29:51.554 INFO:tasks.rados.rados.0.smithi151.stdout:2085: write oid 3 current snap is 273 2024-09-13T14:29:51.554 INFO:tasks.rados.rados.0.smithi151.stdout:2085: seq_num 780 ranges {709460=562226,2060543=791261} 2024-09-13T14:29:51.560 INFO:tasks.rados.rados.0.smithi151.stdout:2085: writing smithi15186934-3 from 709460 to 1271686 tid 1 2024-09-13T14:29:51.569 INFO:tasks.rados.rados.0.smithi151.stdout:2085: writing smithi15186934-3 from 2060543 to 2851804 tid 2 2024-09-13T14:29:51.569 INFO:tasks.rados.rados.0.smithi151.stdout:2086: snap_create 2024-09-13T14:29:51.569 INFO:tasks.rados.rados.0.smithi151.stdout:2085: finishing write tid 1 to smithi15186934-3 2024-09-13T14:29:51.573 INFO:tasks.rados.rados.0.smithi151.stdout:2085: finishing write tid 2 to smithi15186934-3 2024-09-13T14:29:51.573 INFO:tasks.rados.rados.0.smithi151.stdout:2085: finishing write tid 3 to smithi15186934-3 2024-09-13T14:29:51.573 INFO:tasks.rados.rados.0.smithi151.stdout:2085: finishing write tid 4 to smithi15186934-3 2024-09-13T14:29:51.573 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 770 (ObjNum 780 snap 273 seq_num 780) dirty exists 2024-09-13T14:29:51.573 INFO:tasks.rados.rados.0.smithi151.stdout:2085: left oid 3 (ObjNum 780 snap 273 seq_num 780) 2024-09-13T14:29:52.404 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:29:52.533 INFO:tasks.rados.rados.0.smithi151.stdout:2085: done (1 left) 2024-09-13T14:29:52.533 INFO:tasks.rados.rados.0.smithi151.stdout:2086: done (0 left) 2024-09-13T14:29:52.533 INFO:tasks.rados.rados.0.smithi151.stdout:2087: write oid 18 current snap is 274 2024-09-13T14:29:52.533 INFO:tasks.rados.rados.0.smithi151.stdout:2087: seq_num 781 ranges {726267=637475,2119025=651272} 2024-09-13T14:29:52.540 INFO:tasks.rados.rados.0.smithi151.stdout:2087: writing smithi15186934-18 from 726267 to 1363742 tid 1 2024-09-13T14:29:52.548 INFO:tasks.rados.rados.0.smithi151.stdout:2087: writing smithi15186934-18 from 2119025 to 2770297 tid 2 2024-09-13T14:29:52.548 INFO:tasks.rados.rados.0.smithi151.stdout:2088: snap_remove snap 268 2024-09-13T14:29:52.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:52 smithi184 ceph-mon[92527]: pgmap v519: 89 pgs: 89 active+clean; 206 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:29:52.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:52 smithi184 ceph-mon[92527]: osdmap e601: 8 total, 8 up, 8 in 2024-09-13T14:29:52.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:52 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/614375102' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:52.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:52 smithi151 ceph-mon[104450]: pgmap v519: 89 pgs: 89 active+clean; 206 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:29:52.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:52 smithi151 ceph-mon[104450]: osdmap e601: 8 total, 8 up, 8 in 2024-09-13T14:29:52.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:52 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/614375102' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:52.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:52 smithi151 ceph-mon[108966]: pgmap v519: 89 pgs: 89 active+clean; 206 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:29:52.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:52 smithi151 ceph-mon[108966]: osdmap e601: 8 total, 8 up, 8 in 2024-09-13T14:29:52.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:52 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/614375102' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:53.550 INFO:tasks.rados.rados.0.smithi151.stdout:2088: done (1 left) 2024-09-13T14:29:53.551 INFO:tasks.rados.rados.0.smithi151.stdout:2089: delete oid 38 current snap is 274 2024-09-13T14:29:53.551 INFO:tasks.rados.rados.0.smithi151.stdout:2087: finishing write tid 1 to smithi15186934-18 2024-09-13T14:29:53.551 INFO:tasks.rados.rados.0.smithi151.stdout:2087: finishing write tid 2 to smithi15186934-18 2024-09-13T14:29:53.551 INFO:tasks.rados.rados.0.smithi151.stdout:2087: finishing write tid 3 to smithi15186934-18 2024-09-13T14:29:53.551 INFO:tasks.rados.rados.0.smithi151.stdout:2087: finishing write tid 4 to smithi15186934-18 2024-09-13T14:29:53.551 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 18 v 558 (ObjNum 781 snap 274 seq_num 781) dirty exists 2024-09-13T14:29:53.551 INFO:tasks.rados.rados.0.smithi151.stdout:2087: left oid 18 (ObjNum 781 snap 274 seq_num 781) 2024-09-13T14:29:53.556 INFO:tasks.rados.rados.0.smithi151.stdout:2087: done (1 left) 2024-09-13T14:29:53.556 INFO:tasks.rados.rados.0.smithi151.stdout:2089: done (0 left) 2024-09-13T14:29:53.557 INFO:tasks.rados.rados.0.smithi151.stdout:2090: read oid 43 snap -1 2024-09-13T14:29:53.557 INFO:tasks.rados.rados.0.smithi151.stdout:2090: expect deleted 2024-09-13T14:29:53.557 INFO:tasks.rados.rados.0.smithi151.stdout:2091: snap_remove snap 252 2024-09-13T14:29:53.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:53 smithi184 ceph-mon[92527]: from='client.34890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:53.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:53 smithi184 ceph-mon[92527]: osdmap e602: 8 total, 8 up, 8 in 2024-09-13T14:29:53.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:53 smithi184 ceph-mon[92527]: from='client.34896 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:53.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:53 smithi151 ceph-mon[104450]: from='client.34890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:53.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:53 smithi151 ceph-mon[104450]: osdmap e602: 8 total, 8 up, 8 in 2024-09-13T14:29:53.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:53 smithi151 ceph-mon[104450]: from='client.34896 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:53.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:53 smithi151 ceph-mon[108966]: from='client.34890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:53.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:53 smithi151 ceph-mon[108966]: osdmap e602: 8 total, 8 up, 8 in 2024-09-13T14:29:53.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:53 smithi151 ceph-mon[108966]: from='client.34896 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:54.532 INFO:tasks.rados.rados.0.smithi151.stdout:2091: done (1 left) 2024-09-13T14:29:54.532 INFO:tasks.rados.rados.0.smithi151.stdout:2092: rollback oid 46 current snap is 274 2024-09-13T14:29:54.532 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 46 to 262 2024-09-13T14:29:54.533 INFO:tasks.rados.rados.0.smithi151.stdout:2090: done (1 left) 2024-09-13T14:29:54.533 INFO:tasks.rados.rados.0.smithi151.stdout:2093: snap_create 2024-09-13T14:29:54.543 INFO:tasks.rados.rados.0.smithi151.stdout:2092: finishing rollback tid 0 to smithi15186934-46 2024-09-13T14:29:54.544 INFO:tasks.rados.rados.0.smithi151.stdout:2092: finishing rollback tid 1 to smithi15186934-46 2024-09-13T14:29:54.544 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 46 v 745 (ObjNum 129 snap 0 seq_num 3092866488) dirty dne 2024-09-13T14:29:54.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:54 smithi184 ceph-mon[92527]: pgmap v522: 89 pgs: 89 active+clean; 195 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.0 MiB/s rd, 353 KiB/s wr, 5 op/s 2024-09-13T14:29:54.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:54 smithi184 ceph-mon[92527]: osdmap e603: 8 total, 8 up, 8 in 2024-09-13T14:29:54.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:54 smithi184 ceph-mon[92527]: osdmap e604: 8 total, 8 up, 8 in 2024-09-13T14:29:54.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:54 smithi151 ceph-mon[108966]: pgmap v522: 89 pgs: 89 active+clean; 195 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.0 MiB/s rd, 353 KiB/s wr, 5 op/s 2024-09-13T14:29:54.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:54 smithi151 ceph-mon[108966]: osdmap e603: 8 total, 8 up, 8 in 2024-09-13T14:29:54.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:54 smithi151 ceph-mon[108966]: osdmap e604: 8 total, 8 up, 8 in 2024-09-13T14:29:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:54 smithi151 ceph-mon[104450]: pgmap v522: 89 pgs: 89 active+clean; 195 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.0 MiB/s rd, 353 KiB/s wr, 5 op/s 2024-09-13T14:29:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:54 smithi151 ceph-mon[104450]: osdmap e603: 8 total, 8 up, 8 in 2024-09-13T14:29:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:54 smithi151 ceph-mon[104450]: osdmap e604: 8 total, 8 up, 8 in 2024-09-13T14:29:54.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:29:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:29:54] "GET /metrics HTTP/1.1" 200 36515 "" "Prometheus/2.51.0" 2024-09-13T14:29:55.248 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:29:55.532 INFO:tasks.rados.rados.0.smithi151.stdout:2092: done (1 left) 2024-09-13T14:29:55.532 INFO:tasks.rados.rados.0.smithi151.stdout:2093: done (0 left) 2024-09-13T14:29:55.532 INFO:tasks.rados.rados.0.smithi151.stdout:2094: read oid 24 snap -1 2024-09-13T14:29:55.533 INFO:tasks.rados.rados.0.smithi151.stdout:2094: expect (ObjNum 754 snap 260 seq_num 754) 2024-09-13T14:29:55.533 INFO:tasks.rados.rados.0.smithi151.stdout:2095: read oid 23 snap -1 2024-09-13T14:29:55.533 INFO:tasks.rados.rados.0.smithi151.stdout:2095: expect (ObjNum 774 snap 272 seq_num 774) 2024-09-13T14:29:55.533 INFO:tasks.rados.rados.0.smithi151.stdout:2096: delete oid 46 current snap is 275 2024-09-13T14:29:55.534 INFO:tasks.rados.rados.0.smithi151.stdout:2096: done (2 left) 2024-09-13T14:29:55.534 INFO:tasks.rados.rados.0.smithi151.stdout:2097: write oid 6 current snap is 275 2024-09-13T14:29:55.534 INFO:tasks.rados.rados.0.smithi151.stdout:2097: seq_num 784 ranges {776688=579575,1894471=798599} 2024-09-13T14:29:55.543 INFO:tasks.rados.rados.0.smithi151.stdout:2097: writing smithi15186934-6 from 776688 to 1356263 tid 1 2024-09-13T14:29:55.552 INFO:tasks.rados.rados.0.smithi151.stdout:2097: writing smithi15186934-6 from 1894471 to 2693070 tid 2 2024-09-13T14:29:55.552 INFO:tasks.rados.rados.0.smithi151.stdout:2098: snap_remove snap 269 2024-09-13T14:29:55.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:55 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/78869532' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:55.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:55 smithi184 ceph-mon[92527]: osdmap e605: 8 total, 8 up, 8 in 2024-09-13T14:29:55.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:55 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/78869532' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:55.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:55 smithi151 ceph-mon[108966]: osdmap e605: 8 total, 8 up, 8 in 2024-09-13T14:29:55.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:55 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/78869532' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:55.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:55 smithi151 ceph-mon[104450]: osdmap e605: 8 total, 8 up, 8 in 2024-09-13T14:29:56.555 INFO:tasks.rados.rados.0.smithi151.stdout:2098: done (3 left) 2024-09-13T14:29:56.555 INFO:tasks.rados.rados.0.smithi151.stdout:2099: snap_remove snap 265 2024-09-13T14:29:56.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:56 smithi184 ceph-mon[92527]: pgmap v525: 89 pgs: 89 active+clean; 193 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.0 MiB/s rd, 353 KiB/s wr, 5 op/s 2024-09-13T14:29:56.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:56 smithi184 ceph-mon[92527]: from='client.34908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:56.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:56 smithi184 ceph-mon[92527]: from='client.34914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:56.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:56 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:56.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:56 smithi151 ceph-mon[104450]: pgmap v525: 89 pgs: 89 active+clean; 193 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.0 MiB/s rd, 353 KiB/s wr, 5 op/s 2024-09-13T14:29:56.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:56 smithi151 ceph-mon[104450]: from='client.34908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:56.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:56 smithi151 ceph-mon[104450]: from='client.34914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:56.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:56 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:56.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:56 smithi151 ceph-mon[108966]: pgmap v525: 89 pgs: 89 active+clean; 193 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.0 MiB/s rd, 353 KiB/s wr, 5 op/s 2024-09-13T14:29:56.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:56 smithi151 ceph-mon[108966]: from='client.34908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:56.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:56 smithi151 ceph-mon[108966]: from='client.34914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:56.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:56 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:57.559 INFO:tasks.rados.rados.0.smithi151.stdout:2099: done (3 left) 2024-09-13T14:29:57.559 INFO:tasks.rados.rados.0.smithi151.stdout:2100: rollback oid 27 current snap is 275 2024-09-13T14:29:57.559 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 27 to 239 2024-09-13T14:29:57.559 INFO:tasks.rados.rados.0.smithi151.stdout:2101: write oid 22 current snap is 275 2024-09-13T14:29:57.559 INFO:tasks.rados.rados.0.smithi151.stdout:2097: finishing write tid 1 to smithi15186934-6 2024-09-13T14:29:57.559 INFO:tasks.rados.rados.0.smithi151.stdout:2094: expect (ObjNum 754 snap 260 seq_num 754) 2024-09-13T14:29:57.775 INFO:tasks.rados.rados.0.smithi151.stdout:2095: expect (ObjNum 774 snap 272 seq_num 774) 2024-09-13T14:29:57.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:57 smithi184 ceph-mon[92527]: osdmap e606: 8 total, 8 up, 8 in 2024-09-13T14:29:57.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:57 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:29:57.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:57 smithi184 ceph-mon[92527]: osdmap e607: 8 total, 8 up, 8 in 2024-09-13T14:29:57.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:57 smithi151 ceph-mon[108966]: osdmap e606: 8 total, 8 up, 8 in 2024-09-13T14:29:57.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:57 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:29:57.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:57 smithi151 ceph-mon[108966]: osdmap e607: 8 total, 8 up, 8 in 2024-09-13T14:29:57.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:57 smithi151 ceph-mon[104450]: osdmap e606: 8 total, 8 up, 8 in 2024-09-13T14:29:57.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:57 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:29:57.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:57 smithi151 ceph-mon[104450]: osdmap e607: 8 total, 8 up, 8 in 2024-09-13T14:29:57.923 INFO:tasks.rados.rados.0.smithi151.stdout:2097: finishing write tid 2 to smithi15186934-6 2024-09-13T14:29:57.923 INFO:tasks.rados.rados.0.smithi151.stdout:2097: finishing write tid 3 to smithi15186934-6 2024-09-13T14:29:57.923 INFO:tasks.rados.rados.0.smithi151.stdout:2097: finishing write tid 4 to smithi15186934-6 2024-09-13T14:29:57.923 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 6 v 526 (ObjNum 784 snap 275 seq_num 784) dirty exists 2024-09-13T14:29:57.923 INFO:tasks.rados.rados.0.smithi151.stdout:2097: left oid 6 (ObjNum 784 snap 275 seq_num 784) 2024-09-13T14:29:57.923 INFO:tasks.rados.rados.0.smithi151.stdout:2101: seq_num 785 ranges {793495=654824,1952953=658610,3157167=36328} 2024-09-13T14:29:57.930 INFO:tasks.rados.rados.0.smithi151.stdout:2101: writing smithi15186934-22 from 793495 to 1448319 tid 1 2024-09-13T14:29:57.937 INFO:tasks.rados.rados.0.smithi151.stdout:2101: writing smithi15186934-22 from 1952953 to 2611563 tid 2 2024-09-13T14:29:57.940 INFO:tasks.rados.rados.0.smithi151.stdout:2101: writing smithi15186934-22 from 3157167 to 3193495 tid 3 2024-09-13T14:29:57.940 INFO:tasks.rados.rados.0.smithi151.stdout:2094: done (4 left) 2024-09-13T14:29:57.940 INFO:tasks.rados.rados.0.smithi151.stdout:2095: done (3 left) 2024-09-13T14:29:57.940 INFO:tasks.rados.rados.0.smithi151.stdout:2097: done (2 left) 2024-09-13T14:29:57.941 INFO:tasks.rados.rados.0.smithi151.stdout:2102: snap_remove snap 271 2024-09-13T14:29:58.139 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:29:58.583 INFO:tasks.rados.rados.0.smithi151.stdout:2102: done (2 left) 2024-09-13T14:29:58.583 INFO:tasks.rados.rados.0.smithi151.stdout:2103: read oid 5 snap -1 2024-09-13T14:29:58.583 INFO:tasks.rados.rados.0.smithi151.stdout:2103: expect (ObjNum 709 snap 241 seq_num 709) 2024-09-13T14:29:58.584 INFO:tasks.rados.rados.0.smithi151.stdout:2104: snap_remove snap 273 2024-09-13T14:29:58.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:58 smithi184 ceph-mon[92527]: pgmap v528: 89 pgs: 89 active+clean; 188 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 7.4 MiB/s rd, 1.3 MiB/s wr, 10 op/s 2024-09-13T14:29:58.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:58 smithi184 ceph-mon[92527]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:29:58.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:58 smithi184 ceph-mon[92527]: Upgrade: osd.3 is safe to restart 2024-09-13T14:29:58.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:58 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/66260014' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:58.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:58 smithi151 ceph-mon[108966]: pgmap v528: 89 pgs: 89 active+clean; 188 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 7.4 MiB/s rd, 1.3 MiB/s wr, 10 op/s 2024-09-13T14:29:58.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:58 smithi151 ceph-mon[108966]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:29:58.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:58 smithi151 ceph-mon[108966]: Upgrade: osd.3 is safe to restart 2024-09-13T14:29:58.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:58 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/66260014' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:58.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:58 smithi151 ceph-mon[104450]: pgmap v528: 89 pgs: 89 active+clean; 188 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 7.4 MiB/s rd, 1.3 MiB/s wr, 10 op/s 2024-09-13T14:29:58.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:58 smithi151 ceph-mon[104450]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:29:58.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:58 smithi151 ceph-mon[104450]: Upgrade: osd.3 is safe to restart 2024-09-13T14:29:58.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:58 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/66260014' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:29:59.534 INFO:tasks.rados.rados.0.smithi151.stdout:2104: done (3 left) 2024-09-13T14:29:59.534 INFO:tasks.rados.rados.0.smithi151.stdout:2105: read oid 40 snap -1 2024-09-13T14:29:59.534 INFO:tasks.rados.rados.0.smithi151.stdout:2105: expect (ObjNum 741 snap 256 seq_num 741) 2024-09-13T14:29:59.534 INFO:tasks.rados.rados.0.smithi151.stdout:2106: write oid 46 current snap is 275 2024-09-13T14:29:59.534 INFO:tasks.rados.rados.0.smithi151.stdout:2100: finishing rollback tid 1 to smithi15186934-27 2024-09-13T14:29:59.534 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 27 v 0 (ObjNum 129 snap 0 seq_num 3092614848) dirty dne 2024-09-13T14:29:59.534 INFO:tasks.rados.rados.0.smithi151.stdout:2106: seq_num 786 ranges {410302=730073,1895082=402268,2951884=258418} 2024-09-13T14:29:59.540 INFO:tasks.rados.rados.0.smithi151.stdout:2106: writing smithi15186934-46 from 410302 to 1140375 tid 1 2024-09-13T14:29:59.546 INFO:tasks.rados.rados.0.smithi151.stdout:2106: writing smithi15186934-46 from 1895082 to 2297350 tid 2 2024-09-13T14:29:59.552 INFO:tasks.rados.rados.0.smithi151.stdout:2106: writing smithi15186934-46 from 2951884 to 3210302 tid 3 2024-09-13T14:29:59.552 INFO:tasks.rados.rados.0.smithi151.stdout:2100: done (4 left) 2024-09-13T14:29:59.552 INFO:tasks.rados.rados.0.smithi151.stdout:2107: read oid 39 snap -1 2024-09-13T14:29:59.553 INFO:tasks.rados.rados.0.smithi151.stdout:2107: expect (ObjNum 710 snap 244 seq_num 710) 2024-09-13T14:29:59.553 INFO:tasks.rados.rados.0.smithi151.stdout:2108: delete oid 20 current snap is 275 2024-09-13T14:29:59.553 INFO:tasks.rados.rados.0.smithi151.stdout:2101: finishing write tid 1 to smithi15186934-22 2024-09-13T14:29:59.553 INFO:tasks.rados.rados.0.smithi151.stdout:2101: finishing write tid 2 to smithi15186934-22 2024-09-13T14:29:59.553 INFO:tasks.rados.rados.0.smithi151.stdout:2101: finishing write tid 3 to smithi15186934-22 2024-09-13T14:29:59.553 INFO:tasks.rados.rados.0.smithi151.stdout:2101: finishing write tid 4 to smithi15186934-22 2024-09-13T14:29:59.553 INFO:tasks.rados.rados.0.smithi151.stdout:2101: finishing write tid 5 to smithi15186934-22 2024-09-13T14:29:59.553 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 22 v 450 (ObjNum 785 snap 275 seq_num 785) dirty exists 2024-09-13T14:29:59.553 INFO:tasks.rados.rados.0.smithi151.stdout:2101: left oid 22 (ObjNum 785 snap 275 seq_num 785) 2024-09-13T14:29:59.553 INFO:tasks.rados.rados.0.smithi151.stdout:2103: expect (ObjNum 709 snap 241 seq_num 709) 2024-09-13T14:29:59.783 INFO:tasks.rados.rados.0.smithi151.stdout:2105: expect (ObjNum 741 snap 256 seq_num 741) 2024-09-13T14:29:59.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:59 smithi184 ceph-mon[92527]: from='client.44810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:59.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:59 smithi184 ceph-mon[92527]: from='client.34932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:59.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:59 smithi184 ceph-mon[92527]: osdmap e608: 8 total, 8 up, 8 in 2024-09-13T14:29:59.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:59 smithi184 ceph-mon[92527]: Upgrade: Updating osd.3 2024-09-13T14:29:59.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:59 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:59.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:59 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-13T14:29:59.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:59 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:59.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:59 smithi184 ceph-mon[92527]: Deploying daemon osd.3 on smithi151 2024-09-13T14:29:59.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:29:59 smithi184 ceph-mon[92527]: osdmap e609: 8 total, 8 up, 8 in 2024-09-13T14:29:59.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:59 smithi151 ceph-mon[104450]: from='client.44810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:59.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:59 smithi151 ceph-mon[104450]: from='client.34932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:59.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:59 smithi151 ceph-mon[104450]: osdmap e608: 8 total, 8 up, 8 in 2024-09-13T14:29:59.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:59 smithi151 ceph-mon[104450]: Upgrade: Updating osd.3 2024-09-13T14:29:59.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:59 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:59.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:59 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-13T14:29:59.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:59 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:59.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:59 smithi151 ceph-mon[104450]: Deploying daemon osd.3 on smithi151 2024-09-13T14:29:59.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:29:59 smithi151 ceph-mon[104450]: osdmap e609: 8 total, 8 up, 8 in 2024-09-13T14:29:59.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:59 smithi151 ceph-mon[108966]: from='client.44810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:59.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:59 smithi151 ceph-mon[108966]: from='client.34932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:59.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:59 smithi151 ceph-mon[108966]: osdmap e608: 8 total, 8 up, 8 in 2024-09-13T14:29:59.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:59 smithi151 ceph-mon[108966]: Upgrade: Updating osd.3 2024-09-13T14:29:59.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:59 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:29:59.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:59 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-13T14:29:59.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:59 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:29:59.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:59 smithi151 ceph-mon[108966]: Deploying daemon osd.3 on smithi151 2024-09-13T14:29:59.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:29:59 smithi151 ceph-mon[108966]: osdmap e609: 8 total, 8 up, 8 in 2024-09-13T14:29:59.861 INFO:tasks.rados.rados.0.smithi151.stdout:2106: finishing write tid 1 to smithi15186934-46 2024-09-13T14:29:59.861 INFO:tasks.rados.rados.0.smithi151.stdout:2106: finishing write tid 2 to smithi15186934-46 2024-09-13T14:29:59.861 INFO:tasks.rados.rados.0.smithi151.stdout:2106: finishing write tid 3 to smithi15186934-46 2024-09-13T14:29:59.861 INFO:tasks.rados.rados.0.smithi151.stdout:2106: finishing write tid 4 to smithi15186934-46 2024-09-13T14:29:59.861 INFO:tasks.rados.rados.0.smithi151.stdout:2106: finishing write tid 5 to smithi15186934-46 2024-09-13T14:29:59.861 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 46 v 760 (ObjNum 786 snap 275 seq_num 786) dirty exists 2024-09-13T14:29:59.861 INFO:tasks.rados.rados.0.smithi151.stdout:2106: left oid 46 (ObjNum 786 snap 275 seq_num 786) 2024-09-13T14:29:59.861 INFO:tasks.rados.rados.0.smithi151.stdout:2107: expect (ObjNum 710 snap 244 seq_num 710) 2024-09-13T14:30:00.115 INFO:tasks.rados.rados.0.smithi151.stdout:2101: done (5 left) 2024-09-13T14:30:00.115 INFO:tasks.rados.rados.0.smithi151.stdout:2103: done (4 left) 2024-09-13T14:30:00.115 INFO:tasks.rados.rados.0.smithi151.stdout:2105: done (3 left) 2024-09-13T14:30:00.115 INFO:tasks.rados.rados.0.smithi151.stdout:2106: done (2 left) 2024-09-13T14:30:00.115 INFO:tasks.rados.rados.0.smithi151.stdout:2107: done (1 left) 2024-09-13T14:30:00.115 INFO:tasks.rados.rados.0.smithi151.stdout:2108: done (0 left) 2024-09-13T14:30:00.115 INFO:tasks.rados.rados.0.smithi151.stdout:2109: write oid 26 current snap is 275 2024-09-13T14:30:00.115 INFO:tasks.rados.rados.0.smithi151.stdout:2109: seq_num 788 ranges {443916=480571,1612046=405937,2606730=495052} 2024-09-13T14:30:00.119 INFO:tasks.rados.rados.0.smithi151.stdout:2109: writing smithi15186934-26 from 443916 to 924487 tid 1 2024-09-13T14:30:00.125 INFO:tasks.rados.rados.0.smithi151.stdout:2109: writing smithi15186934-26 from 1612046 to 2017983 tid 2 2024-09-13T14:30:00.131 INFO:tasks.rados.rados.0.smithi151.stdout:2109: writing smithi15186934-26 from 2606730 to 3101782 tid 3 2024-09-13T14:30:00.132 INFO:tasks.rados.rados.0.smithi151.stdout:2110: snap_remove snap 272 2024-09-13T14:30:00.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:30:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:30:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:30:00.951 INFO:tasks.rados.rados.0.smithi151.stdout:2110: done (1 left) 2024-09-13T14:30:00.951 INFO:tasks.rados.rados.0.smithi151.stdout:2111: delete oid 10 current snap is 275 2024-09-13T14:30:00.951 INFO:tasks.rados.rados.0.smithi151.stdout:2109: finishing write tid 1 to smithi15186934-26 2024-09-13T14:30:00.951 INFO:tasks.rados.rados.0.smithi151.stdout:2109: finishing write tid 2 to smithi15186934-26 2024-09-13T14:30:00.951 INFO:tasks.rados.rados.0.smithi151.stdout:2109: finishing write tid 3 to smithi15186934-26 2024-09-13T14:30:00.951 INFO:tasks.rados.rados.0.smithi151.stdout:2109: finishing write tid 4 to smithi15186934-26 2024-09-13T14:30:00.951 INFO:tasks.rados.rados.0.smithi151.stdout:2109: finishing write tid 5 to smithi15186934-26 2024-09-13T14:30:00.951 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 765 (ObjNum 788 snap 275 seq_num 788) dirty exists 2024-09-13T14:30:00.951 INFO:tasks.rados.rados.0.smithi151.stdout:2109: left oid 26 (ObjNum 788 snap 275 seq_num 788) 2024-09-13T14:30:00.956 INFO:tasks.rados.rados.0.smithi151.stdout:2109: done (1 left) 2024-09-13T14:30:00.956 INFO:tasks.rados.rados.0.smithi151.stdout:2111: done (0 left) 2024-09-13T14:30:00.956 INFO:tasks.rados.rados.0.smithi151.stdout:2112: read oid 20 snap -1 2024-09-13T14:30:00.956 INFO:tasks.rados.rados.0.smithi151.stdout:2112: expect deleted 2024-09-13T14:30:00.956 INFO:tasks.rados.rados.0.smithi151.stdout:2113: read oid 33 snap -1 2024-09-13T14:30:00.956 INFO:tasks.rados.rados.0.smithi151.stdout:2113: expect (ObjNum 746 snap 257 seq_num 746) 2024-09-13T14:30:00.956 INFO:tasks.rados.rados.0.smithi151.stdout:2114: read oid 42 snap -1 2024-09-13T14:30:00.956 INFO:tasks.rados.rados.0.smithi151.stdout:2114: expect deleted 2024-09-13T14:30:00.956 INFO:tasks.rados.rados.0.smithi151.stdout:2115: snap_remove snap 262 2024-09-13T14:30:01.019 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:30:01.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:00 smithi184 ceph-mon[92527]: pgmap v531: 89 pgs: 89 active+clean; 188 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.3 MiB/s wr, 13 op/s 2024-09-13T14:30:01.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:00 smithi184 ceph-mon[92527]: overall HEALTH_OK 2024-09-13T14:30:01.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:00 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2502332432' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:30:01.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:00 smithi151 ceph-mon[104450]: pgmap v531: 89 pgs: 89 active+clean; 188 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.3 MiB/s wr, 13 op/s 2024-09-13T14:30:01.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:00 smithi151 ceph-mon[104450]: overall HEALTH_OK 2024-09-13T14:30:01.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:00 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2502332432' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:30:01.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:00 smithi151 ceph-mon[108966]: pgmap v531: 89 pgs: 89 active+clean; 188 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.3 MiB/s wr, 13 op/s 2024-09-13T14:30:01.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:00 smithi151 ceph-mon[108966]: overall HEALTH_OK 2024-09-13T14:30:01.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:00 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2502332432' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:30:01.962 INFO:tasks.rados.rados.0.smithi151.stdout:2115: done (3 left) 2024-09-13T14:30:01.962 INFO:tasks.rados.rados.0.smithi151.stdout:2116: delete oid 17 current snap is 275 2024-09-13T14:30:01.962 INFO:tasks.rados.rados.0.smithi151.stdout:2113: expect (ObjNum 746 snap 257 seq_num 746) 2024-09-13T14:30:02.053 INFO:tasks.rados.rados.0.smithi151.stdout:2112: done (3 left) 2024-09-13T14:30:02.053 INFO:tasks.rados.rados.0.smithi151.stdout:2113: done (2 left) 2024-09-13T14:30:02.053 INFO:tasks.rados.rados.0.smithi151.stdout:2114: done (1 left) 2024-09-13T14:30:02.054 INFO:tasks.rados.rados.0.smithi151.stdout:2116: done (0 left) 2024-09-13T14:30:02.054 INFO:tasks.rados.rados.0.smithi151.stdout:2117: snap_remove snap 258 2024-09-13T14:30:02.249 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:01 smithi151 ceph-mon[104450]: osdmap e610: 8 total, 8 up, 8 in 2024-09-13T14:30:02.249 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:02.249 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:02.249 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:01 smithi151 ceph-mon[108966]: osdmap e610: 8 total, 8 up, 8 in 2024-09-13T14:30:02.249 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:01 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:02.249 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:01 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:02.250 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:02 smithi151 systemd[1]: Stopping Ceph osd.3 for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:30:02.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:01 smithi184 ceph-mon[92527]: osdmap e610: 8 total, 8 up, 8 in 2024-09-13T14:30:02.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:01 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:02.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:01 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:02.608 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3[64774]: 2024-09-13T14:30:02.246+0000 7f87e6da0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:30:02.608 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3[64774]: 2024-09-13T14:30:02.246+0000 7f87e6da0640 -1 osd.3 611 *** Got signal Terminated *** 2024-09-13T14:30:02.608 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:02 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3[64774]: 2024-09-13T14:30:02.246+0000 7f87e6da0640 -1 osd.3 611 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T14:30:02.970 INFO:tasks.rados.rados.0.smithi151.stdout:2117: done (0 left) 2024-09-13T14:30:02.971 INFO:tasks.rados.rados.0.smithi151.stdout:2118: snap_create 2024-09-13T14:30:03.266 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:02 smithi151 ceph-mon[108966]: from='client.34944 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:03.266 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:02 smithi151 ceph-mon[108966]: pgmap v534: 89 pgs: 89 active+clean; 188 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.9 MiB/s rd, 4 op/s 2024-09-13T14:30:03.266 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:02 smithi151 ceph-mon[108966]: from='client.44834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:03.266 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:02 smithi151 ceph-mon[108966]: osdmap e611: 8 total, 8 up, 8 in 2024-09-13T14:30:03.267 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:02 smithi151 ceph-mon[108966]: osd.3 marked itself down and dead 2024-09-13T14:30:03.267 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:02 smithi151 ceph-mon[104450]: from='client.34944 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:03.267 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:02 smithi151 ceph-mon[104450]: pgmap v534: 89 pgs: 89 active+clean; 188 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.9 MiB/s rd, 4 op/s 2024-09-13T14:30:03.267 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:02 smithi151 ceph-mon[104450]: from='client.44834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:03.267 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:02 smithi151 ceph-mon[104450]: osdmap e611: 8 total, 8 up, 8 in 2024-09-13T14:30:03.267 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:02 smithi151 ceph-mon[104450]: osd.3 marked itself down and dead 2024-09-13T14:30:03.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:02 smithi184 ceph-mon[92527]: from='client.34944 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:03.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:02 smithi184 ceph-mon[92527]: pgmap v534: 89 pgs: 89 active+clean; 188 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.9 MiB/s rd, 4 op/s 2024-09-13T14:30:03.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:02 smithi184 ceph-mon[92527]: from='client.44834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:03.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:02 smithi184 ceph-mon[92527]: osdmap e611: 8 total, 8 up, 8 in 2024-09-13T14:30:03.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:02 smithi184 ceph-mon[92527]: osd.3 marked itself down and dead 2024-09-13T14:30:03.553 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:03 smithi151 podman[135358]: 2024-09-13 14:30:03.472378086 +0000 UTC m=+1.316923827 container died 495241a87df63f2adb437c32e4befefff726d7602384d3dace6c8bf22d4873a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3, maintainer=Guillaume Abrioux , RELEASE=quincy-bbdadfa, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T14:30:03.915 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:30:03.983 INFO:tasks.rados.rados.0.smithi151.stdout:2118: done (0 left) 2024-09-13T14:30:03.983 INFO:tasks.rados.rados.0.smithi151.stdout:2119: write oid 17 current snap is 276 2024-09-13T14:30:03.983 INFO:tasks.rados.rados.0.smithi151.stdout:2119: seq_num 791 ranges {494337=422671,1387492=553264,2688999=445221} 2024-09-13T14:30:03.988 INFO:tasks.rados.rados.0.smithi151.stdout:2119: writing smithi15186934-17 from 494337 to 917008 tid 1 2024-09-13T14:30:03.995 INFO:tasks.rados.rados.0.smithi151.stdout:2119: writing smithi15186934-17 from 1387492 to 1940756 tid 2 2024-09-13T14:30:04.003 INFO:tasks.rados.rados.0.smithi151.stdout:2119: writing smithi15186934-17 from 2688999 to 3134220 tid 3 2024-09-13T14:30:04.004 INFO:tasks.rados.rados.0.smithi151.stdout:2120: write oid 49 current snap is 276 2024-09-13T14:30:04.004 INFO:tasks.rados.rados.0.smithi151.stdout:2119: finishing write tid 1 to smithi15186934-17 2024-09-13T14:30:04.004 INFO:tasks.rados.rados.0.smithi151.stdout:2119: finishing write tid 2 to smithi15186934-17 2024-09-13T14:30:04.004 INFO:tasks.rados.rados.0.smithi151.stdout:2120: seq_num 792 ranges {511144=497920,1445974=413275,2316422=656493} 2024-09-13T14:30:04.009 INFO:tasks.rados.rados.0.smithi151.stdout:2120: writing smithi15186934-49 from 511144 to 1009064 tid 1 2024-09-13T14:30:04.015 INFO:tasks.rados.rados.0.smithi151.stdout:2120: writing smithi15186934-49 from 1445974 to 1859249 tid 2 2024-09-13T14:30:04.023 INFO:tasks.rados.rados.0.smithi151.stdout:2120: writing smithi15186934-49 from 2316422 to 2972915 tid 3 2024-09-13T14:30:04.023 INFO:tasks.rados.rados.0.smithi151.stdout:2121: delete oid 4 current snap is 276 2024-09-13T14:30:04.023 INFO:tasks.rados.rados.0.smithi151.stdout:2119: finishing write tid 3 to smithi15186934-17 2024-09-13T14:30:04.023 INFO:tasks.rados.rados.0.smithi151.stdout:2119: finishing write tid 4 to smithi15186934-17 2024-09-13T14:30:04.023 INFO:tasks.rados.rados.0.smithi151.stdout:2119: finishing write tid 5 to smithi15186934-17 2024-09-13T14:30:04.023 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 1302 (ObjNum 791 snap 276 seq_num 791) dirty exists 2024-09-13T14:30:04.023 INFO:tasks.rados.rados.0.smithi151.stdout:2119: left oid 17 (ObjNum 791 snap 276 seq_num 791) 2024-09-13T14:30:04.023 INFO:tasks.rados.rados.0.smithi151.stdout:2120: finishing write tid 1 to smithi15186934-49 2024-09-13T14:30:04.024 INFO:tasks.rados.rados.0.smithi151.stdout:2120: finishing write tid 2 to smithi15186934-49 2024-09-13T14:30:04.025 INFO:tasks.rados.rados.0.smithi151.stdout:2119: done (2 left) 2024-09-13T14:30:04.025 INFO:tasks.rados.rados.0.smithi151.stdout:2121: done (1 left) 2024-09-13T14:30:04.025 INFO:tasks.rados.rados.0.smithi151.stdout:2122: snap_create 2024-09-13T14:30:04.026 INFO:tasks.rados.rados.0.smithi151.stdout:2120: finishing write tid 3 to smithi15186934-49 2024-09-13T14:30:04.026 INFO:tasks.rados.rados.0.smithi151.stdout:2120: finishing write tid 4 to smithi15186934-49 2024-09-13T14:30:04.026 INFO:tasks.rados.rados.0.smithi151.stdout:2120: finishing write tid 5 to smithi15186934-49 2024-09-13T14:30:04.026 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 49 v 995 (ObjNum 792 snap 276 seq_num 792) dirty exists 2024-09-13T14:30:04.026 INFO:tasks.rados.rados.0.smithi151.stdout:2120: left oid 49 (ObjNum 792 snap 276 seq_num 792) 2024-09-13T14:30:04.108 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:03 smithi151 podman[135358]: 2024-09-13 14:30:03.92060632 +0000 UTC m=+1.765152066 container cleanup 495241a87df63f2adb437c32e4befefff726d7602384d3dace6c8bf22d4873a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=quincy-bbdadfa, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True) 2024-09-13T14:30:04.108 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:03 smithi151 bash[135358]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3 2024-09-13T14:30:04.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:03 smithi151 ceph-mon[104450]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:30:04.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:03 smithi151 ceph-mon[104450]: osdmap e612: 8 total, 7 up, 8 in 2024-09-13T14:30:04.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:03 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/257738590' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:30:04.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:03 smithi151 ceph-mon[108966]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:30:04.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:03 smithi151 ceph-mon[108966]: osdmap e612: 8 total, 7 up, 8 in 2024-09-13T14:30:04.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:03 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/257738590' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:30:04.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:03 smithi184 ceph-mon[92527]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:30:04.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:03 smithi184 ceph-mon[92527]: osdmap e612: 8 total, 7 up, 8 in 2024-09-13T14:30:04.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:03 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/257738590' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:30:04.532 INFO:tasks.rados.rados.0.smithi151.stdout:2120: done (1 left) 2024-09-13T14:30:04.532 INFO:tasks.rados.rados.0.smithi151.stdout:2122: done (0 left) 2024-09-13T14:30:04.533 INFO:tasks.rados.rados.0.smithi151.stdout:2123: read oid 4 snap 274 2024-09-13T14:30:04.533 INFO:tasks.rados.rados.0.smithi151.stdout:2123: expect (ObjNum 592 snap 191 seq_num 592) 2024-09-13T14:30:04.533 INFO:tasks.rados.rados.0.smithi151.stdout:2124: snap_create 2024-09-13T14:30:04.546 INFO:tasks.rados.rados.0.smithi151.stdout:2123: expect (ObjNum 592 snap 191 seq_num 592) 2024-09-13T14:30:04.589 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:04 smithi151 podman[135403]: 2024-09-13 14:30:04.130940202 +0000 UTC m=+0.651478572 container remove 495241a87df63f2adb437c32e4befefff726d7602384d3dace6c8bf22d4873a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=quincy-bbdadfa, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-13T14:30:04.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:30:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:30:04] "GET /metrics HTTP/1.1" 200 36513 "" "Prometheus/2.51.0" 2024-09-13T14:30:04.859 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:04 smithi151 podman[135553]: 2024-09-13 14:30:04.514301278 +0000 UTC m=+0.019314821 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:30:04.859 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:04 smithi151 podman[135553]: 2024-09-13 14:30:04.640574657 +0000 UTC m=+0.145588194 container create cfd8a7a97644ca917a43e28bbd6e581e7e5cb6bed881ee08fcbb3b3a87dd2dd5 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-13T14:30:05.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:04 smithi184 ceph-mon[92527]: pgmap v537: 89 pgs: 14 stale+active+clean, 75 active+clean; 194 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.6 MiB/s rd, 1007 KiB/s wr, 9 op/s 2024-09-13T14:30:05.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:04 smithi184 ceph-mon[92527]: from='client.34962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:05.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:04 smithi184 ceph-mon[92527]: osdmap e613: 8 total, 7 up, 8 in 2024-09-13T14:30:05.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:04 smithi184 ceph-mon[92527]: osdmap e614: 8 total, 7 up, 8 in 2024-09-13T14:30:05.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:04 smithi151 ceph-mon[104450]: pgmap v537: 89 pgs: 14 stale+active+clean, 75 active+clean; 194 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.6 MiB/s rd, 1007 KiB/s wr, 9 op/s 2024-09-13T14:30:05.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:04 smithi151 ceph-mon[104450]: from='client.34962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:05.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:04 smithi151 ceph-mon[104450]: osdmap e613: 8 total, 7 up, 8 in 2024-09-13T14:30:05.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:04 smithi151 ceph-mon[104450]: osdmap e614: 8 total, 7 up, 8 in 2024-09-13T14:30:05.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:04 smithi151 ceph-mon[108966]: pgmap v537: 89 pgs: 14 stale+active+clean, 75 active+clean; 194 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.6 MiB/s rd, 1007 KiB/s wr, 9 op/s 2024-09-13T14:30:05.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:04 smithi151 ceph-mon[108966]: from='client.34962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:05.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:04 smithi151 ceph-mon[108966]: osdmap e613: 8 total, 7 up, 8 in 2024-09-13T14:30:05.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:04 smithi151 ceph-mon[108966]: osdmap e614: 8 total, 7 up, 8 in 2024-09-13T14:30:05.359 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:04 smithi151 podman[135553]: 2024-09-13 14:30:04.901259573 +0000 UTC m=+0.406273114 container init cfd8a7a97644ca917a43e28bbd6e581e7e5cb6bed881ee08fcbb3b3a87dd2dd5 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-13T14:30:05.359 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:04 smithi151 podman[135553]: 2024-09-13 14:30:04.906421264 +0000 UTC m=+0.411434810 container start cfd8a7a97644ca917a43e28bbd6e581e7e5cb6bed881ee08fcbb3b3a87dd2dd5 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3-deactivate, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-13T14:30:05.359 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:05 smithi151 podman[135553]: 2024-09-13 14:30:05.001558819 +0000 UTC m=+0.506572371 container attach cfd8a7a97644ca917a43e28bbd6e581e7e5cb6bed881ee08fcbb3b3a87dd2dd5 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730) 2024-09-13T14:30:05.359 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:05 smithi151 podman[135553]: 2024-09-13 14:30:05.098895465 +0000 UTC m=+0.603909004 container died cfd8a7a97644ca917a43e28bbd6e581e7e5cb6bed881ee08fcbb3b3a87dd2dd5 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:30:05.533 INFO:tasks.rados.rados.0.smithi151.stdout:2123: done (1 left) 2024-09-13T14:30:05.533 INFO:tasks.rados.rados.0.smithi151.stdout:2124: done (0 left) 2024-09-13T14:30:05.533 INFO:tasks.rados.rados.0.smithi151.stdout:2125: snap_remove snap 239 2024-09-13T14:30:05.976 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:05 smithi151 podman[135597]: 2024-09-13 14:30:05.722491763 +0000 UTC m=+0.617456724 container remove cfd8a7a97644ca917a43e28bbd6e581e7e5cb6bed881ee08fcbb3b3a87dd2dd5 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-13T14:30:05.976 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:05 smithi151 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.3.service: Deactivated successfully. 2024-09-13T14:30:05.976 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:05 smithi151 systemd[1]: Stopped Ceph osd.3 for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:30:05.976 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:05 smithi151 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.3.service: Consumed 10.704s CPU time. 2024-09-13T14:30:05.976 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:05 smithi151 systemd[1]: Starting Ceph osd.3 for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T14:30:06.250 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:05 smithi151 ceph-mon[104450]: from='client.34968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:06.250 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:05 smithi151 ceph-mon[104450]: osdmap e615: 8 total, 7 up, 8 in 2024-09-13T14:30:06.250 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:05 smithi151 ceph-mon[108966]: from='client.34968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:06.251 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:05 smithi151 ceph-mon[108966]: osdmap e615: 8 total, 7 up, 8 in 2024-09-13T14:30:06.251 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:06 smithi151 podman[135701]: 2024-09-13 14:30:06.148570928 +0000 UTC m=+0.104234912 container create 3f8953bf90c4176233d9ee351660fb32db0c897fc6950891877c19fcaf8f6906 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3-activate, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-13T14:30:06.251 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:06 smithi151 podman[135701]: 2024-09-13 14:30:06.061016426 +0000 UTC m=+0.016680421 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:30:06.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:05 smithi184 ceph-mon[92527]: from='client.34968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:06.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:05 smithi184 ceph-mon[92527]: osdmap e615: 8 total, 7 up, 8 in 2024-09-13T14:30:06.509 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:06 smithi151 podman[135701]: 2024-09-13 14:30:06.325997009 +0000 UTC m=+0.281660990 container init 3f8953bf90c4176233d9ee351660fb32db0c897fc6950891877c19fcaf8f6906 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3-activate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T14:30:06.509 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:06 smithi151 podman[135701]: 2024-09-13 14:30:06.330862658 +0000 UTC m=+0.286526655 container start 3f8953bf90c4176233d9ee351660fb32db0c897fc6950891877c19fcaf8f6906 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3-activate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T14:30:06.509 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:06 smithi151 podman[135701]: 2024-09-13 14:30:06.409138425 +0000 UTC m=+0.364802419 container attach 3f8953bf90c4176233d9ee351660fb32db0c897fc6950891877c19fcaf8f6906 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-13T14:30:06.509 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3-activate[135719]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:30:06.509 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:06 smithi151 bash[135701]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:30:06.510 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3-activate[135719]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:30:06.510 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:06 smithi151 bash[135701]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:30:06.510 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3-activate[135719]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:30:06.510 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:06 smithi151 bash[135701]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:30:06.510 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3-activate[135719]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:30:06.510 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:06 smithi151 bash[135701]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:30:06.510 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3-activate[135719]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:30:06.510 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:06 smithi151 bash[135701]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:30:06.538 INFO:tasks.rados.rados.0.smithi151.stdout:2125: done (0 left) 2024-09-13T14:30:06.538 INFO:tasks.rados.rados.0.smithi151.stdout:2126: snap_remove snap 274 2024-09-13T14:30:06.801 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3-activate[135719]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-13T14:30:06.801 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:06 smithi151 bash[135701]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-13T14:30:06.801 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3-activate[135719]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-3 --no-mon-config 2024-09-13T14:30:06.801 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:06 smithi151 bash[135701]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-3 --no-mon-config 2024-09-13T14:30:06.802 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:30:07.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:06 smithi151 ceph-mon[108966]: pgmap v540: 89 pgs: 7 peering, 12 stale+active+clean, 70 active+clean; 194 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.7 MiB/s rd, 1007 KiB/s wr, 8 op/s 2024-09-13T14:30:07.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:06 smithi151 ceph-mon[108966]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:30:07.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:06 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3096734891' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:30:07.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:06 smithi151 ceph-mon[108966]: osdmap e616: 8 total, 7 up, 8 in 2024-09-13T14:30:07.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:06 smithi151 ceph-mon[104450]: pgmap v540: 89 pgs: 7 peering, 12 stale+active+clean, 70 active+clean; 194 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.7 MiB/s rd, 1007 KiB/s wr, 8 op/s 2024-09-13T14:30:07.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:06 smithi151 ceph-mon[104450]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:30:07.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:06 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3096734891' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:30:07.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:06 smithi151 ceph-mon[104450]: osdmap e616: 8 total, 7 up, 8 in 2024-09-13T14:30:07.109 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3-activate[135719]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-13T14:30:07.109 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:06 smithi151 bash[135701]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-13T14:30:07.109 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3-activate[135719]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block 2024-09-13T14:30:07.110 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:06 smithi151 bash[135701]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block 2024-09-13T14:30:07.110 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3-activate[135719]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-13T14:30:07.110 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:06 smithi151 bash[135701]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-13T14:30:07.110 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3-activate[135719]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-13T14:30:07.110 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:06 smithi151 bash[135701]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-13T14:30:07.110 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:06 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3-activate[135719]: --> ceph-volume lvm activate successful for osd ID: 3 2024-09-13T14:30:07.110 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:06 smithi151 bash[135701]: --> ceph-volume lvm activate successful for osd ID: 3 2024-09-13T14:30:07.110 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:06 smithi151 podman[135701]: 2024-09-13 14:30:06.90856839 +0000 UTC m=+0.864232383 container died 3f8953bf90c4176233d9ee351660fb32db0c897fc6950891877c19fcaf8f6906 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3-activate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-13T14:30:07.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:06 smithi184 ceph-mon[92527]: pgmap v540: 89 pgs: 7 peering, 12 stale+active+clean, 70 active+clean; 194 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.7 MiB/s rd, 1007 KiB/s wr, 8 op/s 2024-09-13T14:30:07.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:06 smithi184 ceph-mon[92527]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:30:07.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:06 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3096734891' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:30:07.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:06 smithi184 ceph-mon[92527]: osdmap e616: 8 total, 7 up, 8 in 2024-09-13T14:30:07.538 INFO:tasks.rados.rados.0.smithi151.stdout:2126: done (0 left) 2024-09-13T14:30:07.538 INFO:tasks.rados.rados.0.smithi151.stdout:2127: delete oid 34 current snap is 278 2024-09-13T14:30:07.546 INFO:tasks.rados.rados.0.smithi151.stdout:2127: done (0 left) 2024-09-13T14:30:07.546 INFO:tasks.rados.rados.0.smithi151.stdout:2128: write oid 31 current snap is 278 2024-09-13T14:30:07.546 INFO:tasks.rados.rados.0.smithi151.stdout:2128: seq_num 795 ranges {561565=723667,1905067=560602,3082338=279227} 2024-09-13T14:30:07.554 INFO:tasks.rados.rados.0.smithi151.stdout:2128: writing smithi15186934-31 from 561565 to 1285232 tid 1 2024-09-13T14:30:07.559 INFO:tasks.rados.rados.0.smithi151.stdout:2128: writing smithi15186934-31 from 1905067 to 2465669 tid 2 2024-09-13T14:30:07.565 INFO:tasks.rados.rados.0.smithi151.stdout:2128: writing smithi15186934-31 from 3082338 to 3361565 tid 3 2024-09-13T14:30:07.565 INFO:tasks.rados.rados.0.smithi151.stdout:2129: read oid 23 snap 260 2024-09-13T14:30:07.565 INFO:tasks.rados.rados.0.smithi151.stdout:2129: expect (ObjNum 733 snap 253 seq_num 733) 2024-09-13T14:30:07.565 INFO:tasks.rados.rados.0.smithi151.stdout:2130: snap_create 2024-09-13T14:30:07.565 INFO:tasks.rados.rados.0.smithi151.stdout:2128: finishing write tid 1 to smithi15186934-31 2024-09-13T14:30:07.565 INFO:tasks.rados.rados.0.smithi151.stdout:2128: finishing write tid 2 to smithi15186934-31 2024-09-13T14:30:07.567 INFO:tasks.rados.rados.0.smithi151.stdout:2128: finishing write tid 3 to smithi15186934-31 2024-09-13T14:30:07.567 INFO:tasks.rados.rados.0.smithi151.stdout:2128: finishing write tid 4 to smithi15186934-31 2024-09-13T14:30:07.567 INFO:tasks.rados.rados.0.smithi151.stdout:2128: finishing write tid 5 to smithi15186934-31 2024-09-13T14:30:07.567 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 31 v 910 (ObjNum 795 snap 278 seq_num 795) dirty exists 2024-09-13T14:30:07.568 INFO:tasks.rados.rados.0.smithi151.stdout:2128: left oid 31 (ObjNum 795 snap 278 seq_num 795) 2024-09-13T14:30:07.570 INFO:tasks.rados.rados.0.smithi151.stdout:2129: expect (ObjNum 733 snap 253 seq_num 733) 2024-09-13T14:30:07.858 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:07 smithi151 podman[135913]: 2024-09-13 14:30:07.538316599 +0000 UTC m=+0.623578198 container remove 3f8953bf90c4176233d9ee351660fb32db0c897fc6950891877c19fcaf8f6906 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3-activate, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-09-13T14:30:07.858 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:07 smithi151 podman[135981]: 2024-09-13 14:30:07.660518619 +0000 UTC m=+0.015778509 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:30:07.858 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:07 smithi151 podman[135981]: 2024-09-13 14:30:07.764330593 +0000 UTC m=+0.119590474 container create 31c786beb443da9ab78c6900e0bd0446703939d8bee7cc8d502953400544a7d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T14:30:08.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:07 smithi184 ceph-mon[92527]: from='client.34980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:08.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:07 smithi184 ceph-mon[92527]: osdmap e617: 8 total, 7 up, 8 in 2024-09-13T14:30:08.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:07 smithi151 ceph-mon[104450]: from='client.34980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:08.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:07 smithi151 ceph-mon[104450]: osdmap e617: 8 total, 7 up, 8 in 2024-09-13T14:30:08.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:07 smithi151 ceph-mon[108966]: from='client.34980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:08.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:07 smithi151 ceph-mon[108966]: osdmap e617: 8 total, 7 up, 8 in 2024-09-13T14:30:08.359 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:07 smithi151 podman[135981]: 2024-09-13 14:30:07.983389271 +0000 UTC m=+0.338649161 container init 31c786beb443da9ab78c6900e0bd0446703939d8bee7cc8d502953400544a7d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-13T14:30:08.359 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:07 smithi151 podman[135981]: 2024-09-13 14:30:07.988195078 +0000 UTC m=+0.343454960 container start 31c786beb443da9ab78c6900e0bd0446703939d8bee7cc8d502953400544a7d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-13T14:30:08.359 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:08 smithi151 ceph-osd[136020]: -- 172.21.15.151:0/2528397827 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x558fb388fba0 con 0x558fb461c000 2024-09-13T14:30:08.359 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:08 smithi151 ceph-osd[136020]: -- 172.21.15.151:0/2528397827 <== mon.0 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558fb384fa40 con 0x558fb461c000 2024-09-13T14:30:08.359 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:08 smithi151 bash[135981]: 31c786beb443da9ab78c6900e0bd0446703939d8bee7cc8d502953400544a7d8 2024-09-13T14:30:08.359 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:08 smithi151 systemd[1]: Started Ceph osd.3 for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T14:30:08.537 INFO:tasks.rados.rados.0.smithi151.stdout:2128: done (2 left) 2024-09-13T14:30:08.537 INFO:tasks.rados.rados.0.smithi151.stdout:2129: done (1 left) 2024-09-13T14:30:08.537 INFO:tasks.rados.rados.0.smithi151.stdout:2130: done (0 left) 2024-09-13T14:30:08.537 INFO:tasks.rados.rados.0.smithi151.stdout:2131: write oid 9 current snap is 279 2024-09-13T14:30:08.538 INFO:tasks.rados.rados.0.smithi151.stdout:2131: seq_num 796 ranges {578372=798916,1963549=420613,2826114=417934} 2024-09-13T14:30:08.544 INFO:tasks.rados.rados.0.smithi151.stdout:2131: writing smithi15186934-9 from 578372 to 1377288 tid 1 2024-09-13T14:30:08.550 INFO:tasks.rados.rados.0.smithi151.stdout:2131: writing smithi15186934-9 from 1963549 to 2384162 tid 2 2024-09-13T14:30:08.555 INFO:tasks.rados.rados.0.smithi151.stdout:2131: writing smithi15186934-9 from 2826114 to 3244048 tid 3 2024-09-13T14:30:08.555 INFO:tasks.rados.rados.0.smithi151.stdout:2132: read oid 10 snap 270 2024-09-13T14:30:08.555 INFO:tasks.rados.rados.0.smithi151.stdout:2132: expect (ObjNum 738 snap 254 seq_num 738) 2024-09-13T14:30:08.555 INFO:tasks.rados.rados.0.smithi151.stdout:2133: snap_create 2024-09-13T14:30:08.555 INFO:tasks.rados.rados.0.smithi151.stdout:2131: finishing write tid 1 to smithi15186934-9 2024-09-13T14:30:08.555 INFO:tasks.rados.rados.0.smithi151.stdout:2131: finishing write tid 2 to smithi15186934-9 2024-09-13T14:30:08.558 INFO:tasks.rados.rados.0.smithi151.stdout:2131: finishing write tid 3 to smithi15186934-9 2024-09-13T14:30:08.558 INFO:tasks.rados.rados.0.smithi151.stdout:2131: finishing write tid 4 to smithi15186934-9 2024-09-13T14:30:08.558 INFO:tasks.rados.rados.0.smithi151.stdout:2131: finishing write tid 5 to smithi15186934-9 2024-09-13T14:30:08.558 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 842 (ObjNum 796 snap 279 seq_num 796) dirty exists 2024-09-13T14:30:08.558 INFO:tasks.rados.rados.0.smithi151.stdout:2131: left oid 9 (ObjNum 796 snap 279 seq_num 796) 2024-09-13T14:30:08.562 INFO:tasks.rados.rados.0.smithi151.stdout:2132: expect (ObjNum 738 snap 254 seq_num 738) 2024-09-13T14:30:09.343 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:08 smithi151 ceph-mon[104450]: from='client.34986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:09.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:08 smithi151 ceph-mon[104450]: pgmap v543: 89 pgs: 22 active+undersized, 7 peering, 5 active+undersized+degraded, 55 active+clean; 201 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.6 MiB/s rd, 730 KiB/s wr, 7 op/s; 26/408 objects degraded (6.373%) 2024-09-13T14:30:09.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:08 smithi151 ceph-mon[104450]: Health check failed: Degraded data redundancy: 26/408 objects degraded (6.373%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:30:09.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:08 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:09.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:08 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:09.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:08 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:30:09.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:08 smithi151 ceph-mon[104450]: osdmap e618: 8 total, 7 up, 8 in 2024-09-13T14:30:09.344 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:08 smithi151 ceph-mon[108966]: from='client.34986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:09.344 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:08 smithi151 ceph-mon[108966]: pgmap v543: 89 pgs: 22 active+undersized, 7 peering, 5 active+undersized+degraded, 55 active+clean; 201 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.6 MiB/s rd, 730 KiB/s wr, 7 op/s; 26/408 objects degraded (6.373%) 2024-09-13T14:30:09.344 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:08 smithi151 ceph-mon[108966]: Health check failed: Degraded data redundancy: 26/408 objects degraded (6.373%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:30:09.344 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:08 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:09.344 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:08 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:09.345 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:08 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:30:09.345 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:08 smithi151 ceph-mon[108966]: osdmap e618: 8 total, 7 up, 8 in 2024-09-13T14:30:09.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:08 smithi184 ceph-mon[92527]: from='client.34986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:09.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:08 smithi184 ceph-mon[92527]: pgmap v543: 89 pgs: 22 active+undersized, 7 peering, 5 active+undersized+degraded, 55 active+clean; 201 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.6 MiB/s rd, 730 KiB/s wr, 7 op/s; 26/408 objects degraded (6.373%) 2024-09-13T14:30:09.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:08 smithi184 ceph-mon[92527]: Health check failed: Degraded data redundancy: 26/408 objects degraded (6.373%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:30:09.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:08 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:09.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:08 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:09.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:08 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:30:09.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:08 smithi184 ceph-mon[92527]: osdmap e618: 8 total, 7 up, 8 in 2024-09-13T14:30:09.536 INFO:tasks.rados.rados.0.smithi151.stdout:2131: done (2 left) 2024-09-13T14:30:09.536 INFO:tasks.rados.rados.0.smithi151.stdout:2132: done (1 left) 2024-09-13T14:30:09.536 INFO:tasks.rados.rados.0.smithi151.stdout:2133: done (0 left) 2024-09-13T14:30:09.536 INFO:tasks.rados.rados.0.smithi151.stdout:2134: read oid 3 snap -1 2024-09-13T14:30:09.537 INFO:tasks.rados.rados.0.smithi151.stdout:2134: expect (ObjNum 780 snap 273 seq_num 780) 2024-09-13T14:30:09.537 INFO:tasks.rados.rados.0.smithi151.stdout:2135: write oid 39 current snap is 280 2024-09-13T14:30:09.537 INFO:tasks.rados.rados.0.smithi151.stdout:2135: seq_num 797 ranges {595179=474165,1622031=564271,2737184=629206} 2024-09-13T14:30:09.542 INFO:tasks.rados.rados.0.smithi151.stdout:2135: writing smithi15186934-39 from 595179 to 1069344 tid 1 2024-09-13T14:30:09.549 INFO:tasks.rados.rados.0.smithi151.stdout:2135: writing smithi15186934-39 from 1622031 to 2186302 tid 2 2024-09-13T14:30:09.555 INFO:tasks.rados.rados.0.smithi151.stdout:2135: writing smithi15186934-39 from 2737184 to 3366390 tid 3 2024-09-13T14:30:09.556 INFO:tasks.rados.rados.0.smithi151.stdout:2136: read oid 5 snap 276 2024-09-13T14:30:09.556 INFO:tasks.rados.rados.0.smithi151.stdout:2136: expect (ObjNum 709 snap 241 seq_num 709) 2024-09-13T14:30:09.556 INFO:tasks.rados.rados.0.smithi151.stdout:2137: snap_create 2024-09-13T14:30:09.556 INFO:tasks.rados.rados.0.smithi151.stdout:2134: expect (ObjNum 780 snap 273 seq_num 780) 2024-09-13T14:30:09.688 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:30:09.752 INFO:tasks.rados.rados.0.smithi151.stdout:2135: finishing write tid 1 to smithi15186934-39 2024-09-13T14:30:09.752 INFO:tasks.rados.rados.0.smithi151.stdout:2135: finishing write tid 2 to smithi15186934-39 2024-09-13T14:30:09.752 INFO:tasks.rados.rados.0.smithi151.stdout:2135: finishing write tid 3 to smithi15186934-39 2024-09-13T14:30:09.752 INFO:tasks.rados.rados.0.smithi151.stdout:2135: finishing write tid 4 to smithi15186934-39 2024-09-13T14:30:09.752 INFO:tasks.rados.rados.0.smithi151.stdout:2135: finishing write tid 5 to smithi15186934-39 2024-09-13T14:30:09.752 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 758 (ObjNum 797 snap 280 seq_num 797) dirty exists 2024-09-13T14:30:09.752 INFO:tasks.rados.rados.0.smithi151.stdout:2135: left oid 39 (ObjNum 797 snap 280 seq_num 797) 2024-09-13T14:30:09.752 INFO:tasks.rados.rados.0.smithi151.stdout:2136: expect (ObjNum 709 snap 241 seq_num 709) 2024-09-13T14:30:09.988 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:09 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3[135997]: 2024-09-13T14:30:09.626+0000 7f3b99081640 -1 Falling back to public interface 2024-09-13T14:30:10.300 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:09 smithi151 ceph-mon[104450]: pgmap v546: 89 pgs: 28 active+undersized, 6 active+undersized+degraded, 55 active+clean; 203 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.1 MiB/s wr, 6 op/s; 27/405 objects degraded (6.667%) 2024-09-13T14:30:10.300 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:09 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/4030501812' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:30:10.300 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:09 smithi151 ceph-mon[104450]: osdmap e619: 8 total, 7 up, 8 in 2024-09-13T14:30:10.300 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:09 smithi151 ceph-mon[104450]: from='client.44885 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:10.300 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:09 smithi151 ceph-mon[108966]: pgmap v546: 89 pgs: 28 active+undersized, 6 active+undersized+degraded, 55 active+clean; 203 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.1 MiB/s wr, 6 op/s; 27/405 objects degraded (6.667%) 2024-09-13T14:30:10.300 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:09 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/4030501812' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:30:10.300 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:09 smithi151 ceph-mon[108966]: osdmap e619: 8 total, 7 up, 8 in 2024-09-13T14:30:10.301 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:09 smithi151 ceph-mon[108966]: from='client.44885 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:10.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:30:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:30:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:30:10.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:09 smithi184 ceph-mon[92527]: pgmap v546: 89 pgs: 28 active+undersized, 6 active+undersized+degraded, 55 active+clean; 203 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.1 MiB/s wr, 6 op/s; 27/405 objects degraded (6.667%) 2024-09-13T14:30:10.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:09 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/4030501812' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:30:10.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:09 smithi184 ceph-mon[92527]: osdmap e619: 8 total, 7 up, 8 in 2024-09-13T14:30:10.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:09 smithi184 ceph-mon[92527]: from='client.44885 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:10.534 INFO:tasks.rados.rados.0.smithi151.stdout:2134: done (3 left) 2024-09-13T14:30:10.534 INFO:tasks.rados.rados.0.smithi151.stdout:2135: done (2 left) 2024-09-13T14:30:10.534 INFO:tasks.rados.rados.0.smithi151.stdout:2136: done (1 left) 2024-09-13T14:30:10.535 INFO:tasks.rados.rados.0.smithi151.stdout:2137: done (0 left) 2024-09-13T14:30:10.535 INFO:tasks.rados.rados.0.smithi151.stdout:2138: delete oid 11 current snap is 281 2024-09-13T14:30:10.538 INFO:tasks.rados.rados.0.smithi151.stdout:2138: done (0 left) 2024-09-13T14:30:10.538 INFO:tasks.rados.rados.0.smithi151.stdout:2139: snap_create 2024-09-13T14:30:11.304 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:10 smithi151 ceph-mon[108966]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-13T14:30:11.304 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:10 smithi151 ceph-mon[108966]: from='client.35007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:11.304 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:10 smithi151 ceph-mon[108966]: osdmap e620: 8 total, 7 up, 8 in 2024-09-13T14:30:11.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:10 smithi151 ceph-mon[104450]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-13T14:30:11.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:10 smithi151 ceph-mon[104450]: from='client.35007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:11.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:10 smithi151 ceph-mon[104450]: osdmap e620: 8 total, 7 up, 8 in 2024-09-13T14:30:11.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:10 smithi184 ceph-mon[92527]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-13T14:30:11.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:10 smithi184 ceph-mon[92527]: from='client.35007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:11.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:10 smithi184 ceph-mon[92527]: osdmap e620: 8 total, 7 up, 8 in 2024-09-13T14:30:11.536 INFO:tasks.rados.rados.0.smithi151.stdout:2139: done (0 left) 2024-09-13T14:30:11.536 INFO:tasks.rados.rados.0.smithi151.stdout:2140: read oid 43 snap -1 2024-09-13T14:30:11.536 INFO:tasks.rados.rados.0.smithi151.stdout:2140: expect deleted 2024-09-13T14:30:11.536 INFO:tasks.rados.rados.0.smithi151.stdout:2141: read oid 1 snap -1 2024-09-13T14:30:11.536 INFO:tasks.rados.rados.0.smithi151.stdout:2141: expect deleted 2024-09-13T14:30:11.536 INFO:tasks.rados.rados.0.smithi151.stdout:2142: write oid 42 current snap is 282 2024-09-13T14:30:11.536 INFO:tasks.rados.rados.0.smithi151.stdout:2142: seq_num 799 ranges {628793=741016,2138995=567940,3192030=236763} 2024-09-13T14:30:11.543 INFO:tasks.rados.rados.0.smithi151.stdout:2142: writing smithi15186934-42 from 628793 to 1369809 tid 1 2024-09-13T14:30:11.551 INFO:tasks.rados.rados.0.smithi151.stdout:2142: writing smithi15186934-42 from 2138995 to 2706935 tid 2 2024-09-13T14:30:11.556 INFO:tasks.rados.rados.0.smithi151.stdout:2142: writing smithi15186934-42 from 3192030 to 3428793 tid 3 2024-09-13T14:30:11.556 INFO:tasks.rados.rados.0.smithi151.stdout:2143: delete oid 34 current snap is 282 2024-09-13T14:30:11.556 INFO:tasks.rados.rados.0.smithi151.stdout:2142: finishing write tid 1 to smithi15186934-42 2024-09-13T14:30:11.556 INFO:tasks.rados.rados.0.smithi151.stdout:2142: finishing write tid 2 to smithi15186934-42 2024-09-13T14:30:11.557 INFO:tasks.rados.rados.0.smithi151.stdout:2140: done (3 left) 2024-09-13T14:30:11.558 INFO:tasks.rados.rados.0.smithi151.stdout:2141: done (2 left) 2024-09-13T14:30:11.558 INFO:tasks.rados.rados.0.smithi151.stdout:2143: done (1 left) 2024-09-13T14:30:11.558 INFO:tasks.rados.rados.0.smithi151.stdout:2144: rollback oid 8 current snap is 282 2024-09-13T14:30:11.558 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 8 to 243 2024-09-13T14:30:11.558 INFO:tasks.rados.rados.0.smithi151.stdout:2145: snap_remove snap 256 2024-09-13T14:30:12.544 INFO:tasks.rados.rados.0.smithi151.stdout:2145: done (2 left) 2024-09-13T14:30:12.544 INFO:tasks.rados.rados.0.smithi151.stdout:2146: read oid 47 snap -1 2024-09-13T14:30:12.544 INFO:tasks.rados.rados.0.smithi151.stdout:2146: expect deleted 2024-09-13T14:30:12.544 INFO:tasks.rados.rados.0.smithi151.stdout:2147: snap_create 2024-09-13T14:30:12.544 INFO:tasks.rados.rados.0.smithi151.stdout:2142: finishing write tid 3 to smithi15186934-42 2024-09-13T14:30:12.544 INFO:tasks.rados.rados.0.smithi151.stdout:2142: finishing write tid 4 to smithi15186934-42 2024-09-13T14:30:12.544 INFO:tasks.rados.rados.0.smithi151.stdout:2142: finishing write tid 5 to smithi15186934-42 2024-09-13T14:30:12.544 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 914 (ObjNum 799 snap 282 seq_num 799) dirty exists 2024-09-13T14:30:12.544 INFO:tasks.rados.rados.0.smithi151.stdout:2142: left oid 42 (ObjNum 799 snap 282 seq_num 799) 2024-09-13T14:30:12.544 INFO:tasks.rados.rados.0.smithi151.stdout:2144: finishing rollback tid 0 to smithi15186934-8 2024-09-13T14:30:12.544 INFO:tasks.rados.rados.0.smithi151.stdout:2144: finishing rollback tid 1 to smithi15186934-8 2024-09-13T14:30:12.544 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 669 (ObjNum 129 snap 0 seq_num 3092866488) dirty dne 2024-09-13T14:30:12.622 INFO:teuthology.orchestra.run.smithi151.stdout:true 2024-09-13T14:30:12.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:12 smithi184 ceph-mon[92527]: pgmap v549: 89 pgs: 28 active+undersized, 6 active+undersized+degraded, 55 active+clean; 205 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 782 KiB/s wr, 4 op/s; 27/408 objects degraded (6.618%) 2024-09-13T14:30:12.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:12 smithi184 ceph-mon[92527]: osdmap e621: 8 total, 7 up, 8 in 2024-09-13T14:30:12.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:12 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1721912159' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:30:12.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:12 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:12.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:12 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:12.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:12 smithi151 ceph-mon[104450]: pgmap v549: 89 pgs: 28 active+undersized, 6 active+undersized+degraded, 55 active+clean; 205 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 782 KiB/s wr, 4 op/s; 27/408 objects degraded (6.618%) 2024-09-13T14:30:12.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:12 smithi151 ceph-mon[104450]: osdmap e621: 8 total, 7 up, 8 in 2024-09-13T14:30:12.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:12 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1721912159' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:30:12.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:12 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:12.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:12 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:12.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:12 smithi151 ceph-mon[108966]: pgmap v549: 89 pgs: 28 active+undersized, 6 active+undersized+degraded, 55 active+clean; 205 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 782 KiB/s wr, 4 op/s; 27/408 objects degraded (6.618%) 2024-09-13T14:30:12.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:12 smithi151 ceph-mon[108966]: osdmap e621: 8 total, 7 up, 8 in 2024-09-13T14:30:12.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:12 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1721912159' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:30:12.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:12 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:12.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:12 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:13.522 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:13 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3[135997]: 2024-09-13T14:30:13.260+0000 7f3b99081640 -1 osd.3 611 log_to_monitors true 2024-09-13T14:30:13.544 INFO:tasks.rados.rados.0.smithi151.stdout:2142: done (3 left) 2024-09-13T14:30:13.545 INFO:tasks.rados.rados.0.smithi151.stdout:2144: done (2 left) 2024-09-13T14:30:13.545 INFO:tasks.rados.rados.0.smithi151.stdout:2146: done (1 left) 2024-09-13T14:30:13.545 INFO:tasks.rados.rados.0.smithi151.stdout:2147: done (0 left) 2024-09-13T14:30:13.545 INFO:tasks.rados.rados.0.smithi151.stdout:2148: delete oid 13 current snap is 283 2024-09-13T14:30:13.549 INFO:tasks.rados.rados.0.smithi151.stdout:2148: done (0 left) 2024-09-13T14:30:13.549 INFO:tasks.rados.rados.0.smithi151.stdout:2149: write oid 20 current snap is 283 2024-09-13T14:30:13.549 INFO:tasks.rados.rados.0.smithi151.stdout:2149: seq_num 802 ranges {679214=566763,1798088=715267,3157946=321268} 2024-09-13T14:30:13.556 INFO:tasks.rados.rados.0.smithi151.stdout:2149: writing smithi15186934-20 from 679214 to 1245977 tid 1 2024-09-13T14:30:13.564 INFO:tasks.rados.rados.0.smithi151.stdout:2149: writing smithi15186934-20 from 1798088 to 2513355 tid 2 2024-09-13T14:30:13.569 INFO:tasks.rados.rados.0.smithi151.stdout:2149: writing smithi15186934-20 from 3157946 to 3479214 tid 3 2024-09-13T14:30:13.570 INFO:tasks.rados.rados.0.smithi151.stdout:2150: rollback oid 43 current snap is 283 2024-09-13T14:30:13.570 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 43 to 281 2024-09-13T14:30:13.570 INFO:tasks.rados.rados.0.smithi151.stdout:2149: finishing write tid 1 to smithi15186934-20 2024-09-13T14:30:13.570 INFO:tasks.rados.rados.0.smithi151.stdout:2149: finishing write tid 2 to smithi15186934-20 2024-09-13T14:30:13.570 INFO:tasks.rados.rados.0.smithi151.stdout:2151: rollback oid 5 current snap is 283 2024-09-13T14:30:13.570 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 5 to 270 2024-09-13T14:30:13.570 INFO:tasks.rados.rados.0.smithi151.stdout:2152: write oid 16 current snap is 283 2024-09-13T14:30:13.570 INFO:tasks.rados.rados.0.smithi151.stdout:2152: seq_num 803 ranges {696021=642012,1856570=575278,3185369=310652} 2024-09-13T14:30:13.576 INFO:tasks.rados.rados.0.smithi151.stdout:2152: writing smithi15186934-16 from 696021 to 1338033 tid 1 2024-09-13T14:30:13.582 INFO:tasks.rados.rados.0.smithi151.stdout:2152: writing smithi15186934-16 from 1856570 to 2431848 tid 2 2024-09-13T14:30:13.589 INFO:tasks.rados.rados.0.smithi151.stdout:2152: writing smithi15186934-16 from 3185369 to 3496021 tid 3 2024-09-13T14:30:13.589 INFO:tasks.rados.rados.0.smithi151.stdout:2153: read oid 42 snap 270 2024-09-13T14:30:13.589 INFO:tasks.rados.rados.0.smithi151.stdout:2153: expect deleted 2024-09-13T14:30:13.589 INFO:tasks.rados.rados.0.smithi151.stdout:2154: write oid 17 current snap is 283 2024-09-13T14:30:13.589 INFO:tasks.rados.rados.0.smithi151.stdout:2151: finishing rollback tid 0 to smithi15186934-5 2024-09-13T14:30:13.589 INFO:tasks.rados.rados.0.smithi151.stdout:2149: finishing write tid 3 to smithi15186934-20 2024-09-13T14:30:13.589 INFO:tasks.rados.rados.0.smithi151.stdout:2150: finishing rollback tid 1 to smithi15186934-43 2024-09-13T14:30:13.589 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 43 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-13T14:30:13.590 INFO:tasks.rados.rados.0.smithi151.stdout:2149: finishing write tid 4 to smithi15186934-20 2024-09-13T14:30:13.590 INFO:tasks.rados.rados.0.smithi151.stdout:2149: finishing write tid 5 to smithi15186934-20 2024-09-13T14:30:13.590 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 490 (ObjNum 802 snap 283 seq_num 802) dirty exists 2024-09-13T14:30:13.590 INFO:tasks.rados.rados.0.smithi151.stdout:2149: left oid 20 (ObjNum 802 snap 283 seq_num 802) 2024-09-13T14:30:13.590 INFO:tasks.rados.rados.0.smithi151.stdout:2151: finishing rollback tid 1 to smithi15186934-5 2024-09-13T14:30:13.590 INFO:tasks.rados.rados.0.smithi151.stdout:2151: finishing rollback tid 2 to smithi15186934-5 2024-09-13T14:30:13.590 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 821 (ObjNum 709 snap 241 seq_num 709) dirty exists 2024-09-13T14:30:13.590 INFO:tasks.rados.rados.0.smithi151.stdout:2152: finishing write tid 1 to smithi15186934-16 2024-09-13T14:30:13.590 INFO:tasks.rados.rados.0.smithi151.stdout:2152: finishing write tid 2 to smithi15186934-16 2024-09-13T14:30:13.590 INFO:tasks.rados.rados.0.smithi151.stdout:2154: seq_num 804 ranges {712828=717261,1915052=718936,3212792=300036} 2024-09-13T14:30:13.596 INFO:tasks.rados.rados.0.smithi151.stdout:2154: writing smithi15186934-17 from 712828 to 1430089 tid 1 2024-09-13T14:30:13.605 INFO:tasks.rados.rados.0.smithi151.stdout:2154: writing smithi15186934-17 from 1915052 to 2633988 tid 2 2024-09-13T14:30:13.608 INFO:tasks.rados.rados.0.smithi151.stdout:2154: writing smithi15186934-17 from 3212792 to 3512828 tid 3 2024-09-13T14:30:13.608 INFO:tasks.rados.rados.0.smithi151.stdout:2149: done (5 left) 2024-09-13T14:30:13.608 INFO:tasks.rados.rados.0.smithi151.stdout:2150: done (4 left) 2024-09-13T14:30:13.608 INFO:tasks.rados.rados.0.smithi151.stdout:2151: done (3 left) 2024-09-13T14:30:13.609 INFO:tasks.rados.rados.0.smithi151.stdout:2155: write oid 32 current snap is 283 2024-09-13T14:30:13.609 INFO:tasks.rados.rados.0.smithi151.stdout:2155: seq_num 805 ranges {729635=792510,1973534=578947,3240215=289420} 2024-09-13T14:30:13.616 INFO:tasks.rados.rados.0.smithi151.stdout:2155: writing smithi15186934-32 from 729635 to 1522145 tid 1 2024-09-13T14:30:13.622 INFO:tasks.rados.rados.0.smithi151.stdout:2155: writing smithi15186934-32 from 1973534 to 2552481 tid 2 2024-09-13T14:30:13.627 INFO:tasks.rados.rados.0.smithi151.stdout:2155: writing smithi15186934-32 from 3240215 to 3529635 tid 3 2024-09-13T14:30:13.627 INFO:tasks.rados.rados.0.smithi151.stdout:2156: write oid 20 current snap is 283 2024-09-13T14:30:13.628 INFO:tasks.rados.rados.0.smithi151.stdout:2156: seq_num 806 ranges {746442=584112,1748369=722605,2983991=562451} 2024-09-13T14:30:13.634 INFO:tasks.rados.rados.0.smithi151.stdout:2156: writing smithi15186934-20 from 746442 to 1330554 tid 1 2024-09-13T14:30:13.641 INFO:tasks.rados.rados.0.smithi151.stdout:2156: writing smithi15186934-20 from 1748369 to 2470974 tid 2 2024-09-13T14:30:13.647 INFO:tasks.rados.rados.0.smithi151.stdout:2156: writing smithi15186934-20 from 2983991 to 3546442 tid 3 2024-09-13T14:30:13.647 INFO:tasks.rados.rados.0.smithi151.stdout:2157: delete oid 5 current snap is 283 2024-09-13T14:30:13.647 INFO:tasks.rados.rados.0.smithi151.stdout:2152: finishing write tid 3 to smithi15186934-16 2024-09-13T14:30:13.647 INFO:tasks.rados.rados.0.smithi151.stdout:2152: finishing write tid 4 to smithi15186934-16 2024-09-13T14:30:13.647 INFO:tasks.rados.rados.0.smithi151.stdout:2152: finishing write tid 5 to smithi15186934-16 2024-09-13T14:30:13.647 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 16 v 495 (ObjNum 803 snap 283 seq_num 803) dirty exists 2024-09-13T14:30:13.647 INFO:tasks.rados.rados.0.smithi151.stdout:2152: left oid 16 (ObjNum 803 snap 283 seq_num 803) 2024-09-13T14:30:13.647 INFO:tasks.rados.rados.0.smithi151.stdout:2154: finishing write tid 1 to smithi15186934-17 2024-09-13T14:30:13.648 INFO:tasks.rados.rados.0.smithi151.stdout:2154: finishing write tid 2 to smithi15186934-17 2024-09-13T14:30:13.648 INFO:tasks.rados.rados.0.smithi151.stdout:2154: finishing write tid 3 to smithi15186934-17 2024-09-13T14:30:13.648 INFO:tasks.rados.rados.0.smithi151.stdout:2154: finishing write tid 4 to smithi15186934-17 2024-09-13T14:30:13.648 INFO:tasks.rados.rados.0.smithi151.stdout:2154: finishing write tid 5 to smithi15186934-17 2024-09-13T14:30:13.648 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 1308 (ObjNum 804 snap 283 seq_num 804) dirty exists 2024-09-13T14:30:13.648 INFO:tasks.rados.rados.0.smithi151.stdout:2154: left oid 17 (ObjNum 804 snap 283 seq_num 804) 2024-09-13T14:30:13.648 INFO:tasks.rados.rados.0.smithi151.stdout:2155: finishing write tid 1 to smithi15186934-32 2024-09-13T14:30:13.648 INFO:tasks.rados.rados.0.smithi151.stdout:2155: finishing write tid 2 to smithi15186934-32 2024-09-13T14:30:13.648 INFO:tasks.rados.rados.0.smithi151.stdout:2155: finishing write tid 3 to smithi15186934-32 2024-09-13T14:30:13.648 INFO:tasks.rados.rados.0.smithi151.stdout:2155: finishing write tid 4 to smithi15186934-32 2024-09-13T14:30:13.648 INFO:tasks.rados.rados.0.smithi151.stdout:2155: finishing write tid 5 to smithi15186934-32 2024-09-13T14:30:13.648 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 32 v 853 (ObjNum 805 snap 283 seq_num 805) dirty exists 2024-09-13T14:30:13.648 INFO:tasks.rados.rados.0.smithi151.stdout:2155: left oid 32 (ObjNum 805 snap 283 seq_num 805) 2024-09-13T14:30:13.649 INFO:tasks.rados.rados.0.smithi151.stdout:2156: finishing write tid 1 to smithi15186934-20 2024-09-13T14:30:13.649 INFO:tasks.rados.rados.0.smithi151.stdout:2156: finishing write tid 2 to smithi15186934-20 2024-09-13T14:30:13.649 INFO:tasks.rados.rados.0.smithi151.stdout:2152: done (5 left) 2024-09-13T14:30:13.649 INFO:tasks.rados.rados.0.smithi151.stdout:2153: done (4 left) 2024-09-13T14:30:13.649 INFO:tasks.rados.rados.0.smithi151.stdout:2154: done (3 left) 2024-09-13T14:30:13.649 INFO:tasks.rados.rados.0.smithi151.stdout:2155: done (2 left) 2024-09-13T14:30:13.649 INFO:tasks.rados.rados.0.smithi151.stdout:2157: done (1 left) 2024-09-13T14:30:13.649 INFO:tasks.rados.rados.0.smithi151.stdout:2158: delete oid 9 current snap is 283 2024-09-13T14:30:13.650 INFO:tasks.rados.rados.0.smithi151.stdout:2156: finishing write tid 3 to smithi15186934-20 2024-09-13T14:30:13.656 INFO:tasks.rados.rados.0.smithi151.stdout:2156: finishing write tid 4 to smithi15186934-20 2024-09-13T14:30:13.657 INFO:tasks.rados.rados.0.smithi151.stdout:2156: finishing write tid 5 to smithi15186934-20 2024-09-13T14:30:13.657 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 499 (ObjNum 806 snap 283 seq_num 806) dirty exists 2024-09-13T14:30:13.657 INFO:tasks.rados.rados.0.smithi151.stdout:2156: left oid 20 (ObjNum 806 snap 283 seq_num 806) 2024-09-13T14:30:13.657 INFO:tasks.rados.rados.0.smithi151.stdout:2156: done (1 left) 2024-09-13T14:30:13.657 INFO:tasks.rados.rados.0.smithi151.stdout:2158: done (0 left) 2024-09-13T14:30:13.657 INFO:tasks.rados.rados.0.smithi151.stdout:2159: read oid 43 snap -1 2024-09-13T14:30:13.657 INFO:tasks.rados.rados.0.smithi151.stdout:2159: expect deleted 2024-09-13T14:30:13.657 INFO:tasks.rados.rados.0.smithi151.stdout:2160: read oid 19 snap -1 2024-09-13T14:30:13.657 INFO:tasks.rados.rados.0.smithi151.stdout:2160: expect (ObjNum 762 snap 263 seq_num 762) 2024-09-13T14:30:13.657 INFO:tasks.rados.rados.0.smithi151.stdout:2161: read oid 35 snap -1 2024-09-13T14:30:13.657 INFO:tasks.rados.rados.0.smithi151.stdout:2161: expect deleted 2024-09-13T14:30:13.657 INFO:tasks.rados.rados.0.smithi151.stdout:2162: rollback oid 22 current snap is 283 2024-09-13T14:30:13.657 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 22 to 270 2024-09-13T14:30:13.657 INFO:tasks.rados.rados.0.smithi151.stdout:2163: read oid 31 snap -1 2024-09-13T14:30:13.658 INFO:tasks.rados.rados.0.smithi151.stdout:2163: expect (ObjNum 795 snap 278 seq_num 795) 2024-09-13T14:30:13.658 INFO:tasks.rados.rados.0.smithi151.stdout:2164: snap_remove snap 279 2024-09-13T14:30:13.789 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 14:30:13 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3[135997]: 2024-09-13T14:30:13.572+0000 7f3b90e2b640 -1 osd.3 611 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:30:13.789 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:13 smithi151 ceph-mon[108966]: osdmap e622: 8 total, 7 up, 8 in 2024-09-13T14:30:13.789 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:13 smithi151 ceph-mon[108966]: from='client.44903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:13.789 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:13 smithi151 ceph-mon[108966]: from='client.44909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:13.789 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:13 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:13.789 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:13 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:13.789 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:13 smithi151 ceph-mon[108966]: from='osd.3 [v2:172.21.15.151:6826/2997668861,v1:172.21.15.151:6827/2997668861]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-13T14:30:13.790 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:13 smithi151 ceph-mon[108966]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-13T14:30:13.790 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:13 smithi151 ceph-mon[104450]: osdmap e622: 8 total, 7 up, 8 in 2024-09-13T14:30:13.790 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:13 smithi151 ceph-mon[104450]: from='client.44903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:13.790 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:13 smithi151 ceph-mon[104450]: from='client.44909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:13.790 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:13 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:13.790 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:13 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:13.790 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:13 smithi151 ceph-mon[104450]: from='osd.3 [v2:172.21.15.151:6826/2997668861,v1:172.21.15.151:6827/2997668861]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-13T14:30:13.790 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:13 smithi151 ceph-mon[104450]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-13T14:30:13.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:13 smithi184 ceph-mon[92527]: osdmap e622: 8 total, 7 up, 8 in 2024-09-13T14:30:13.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:13 smithi184 ceph-mon[92527]: from='client.44903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:13.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:13 smithi184 ceph-mon[92527]: from='client.44909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:13.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:13 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:13.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:13 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:13.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:13 smithi184 ceph-mon[92527]: from='osd.3 [v2:172.21.15.151:6826/2997668861,v1:172.21.15.151:6827/2997668861]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-13T14:30:13.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:13 smithi184 ceph-mon[92527]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-13T14:30:14.556 INFO:tasks.rados.rados.0.smithi151.stdout:2162: finishing rollback tid 0 to smithi15186934-22 2024-09-13T14:30:14.556 INFO:tasks.rados.rados.0.smithi151.stdout:2162: finishing rollback tid 1 to smithi15186934-22 2024-09-13T14:30:14.556 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 22 v 457 (ObjNum 129 snap 0 seq_num 3092830336) dirty dne 2024-09-13T14:30:14.556 INFO:tasks.rados.rados.0.smithi151.stdout:2163: expect (ObjNum 795 snap 278 seq_num 795) 2024-09-13T14:30:14.678 INFO:tasks.rados.rados.0.smithi151.stdout:2160: expect (ObjNum 762 snap 263 seq_num 762) 2024-09-13T14:30:14.804 INFO:tasks.rados.rados.0.smithi151.stdout:2159: done (5 left) 2024-09-13T14:30:14.804 INFO:tasks.rados.rados.0.smithi151.stdout:2160: done (4 left) 2024-09-13T14:30:14.804 INFO:tasks.rados.rados.0.smithi151.stdout:2161: done (3 left) 2024-09-13T14:30:14.805 INFO:tasks.rados.rados.0.smithi151.stdout:2162: done (2 left) 2024-09-13T14:30:14.805 INFO:tasks.rados.rados.0.smithi151.stdout:2163: done (1 left) 2024-09-13T14:30:14.805 INFO:tasks.rados.rados.0.smithi151.stdout:2164: done (0 left) 2024-09-13T14:30:14.805 INFO:tasks.rados.rados.0.smithi151.stdout:2165: read oid 12 snap -1 2024-09-13T14:30:14.805 INFO:tasks.rados.rados.0.smithi151.stdout:2165: expect (ObjNum 654 snap 212 seq_num 654) 2024-09-13T14:30:14.805 INFO:tasks.rados.rados.0.smithi151.stdout:2166: write oid 32 current snap is 283 2024-09-13T14:30:14.805 INFO:tasks.rados.rados.0.smithi151.stdout:2166: seq_num 809 ranges {796863=409859,1807462=586285,3066260=530603} 2024-09-13T14:30:14.810 INFO:tasks.rados.rados.0.smithi151.stdout:2166: writing smithi15186934-32 from 796863 to 1206722 tid 1 2024-09-13T14:30:14.817 INFO:tasks.rados.rados.0.smithi151.stdout:2166: writing smithi15186934-32 from 1807462 to 2393747 tid 2 2024-09-13T14:30:14.823 INFO:tasks.rados.rados.0.smithi151.stdout:2166: writing smithi15186934-32 from 3066260 to 3596863 tid 3 2024-09-13T14:30:14.823 INFO:tasks.rados.rados.0.smithi151.stdout:2167: snap_remove snap 275 2024-09-13T14:30:14.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:14 smithi184 ceph-mon[92527]: pgmap v552: 89 pgs: 28 active+undersized, 6 active+undersized+degraded, 55 active+clean; 205 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 6.2 MiB/s rd, 808 KiB/s wr, 8 op/s; 27/408 objects degraded (6.618%) 2024-09-13T14:30:14.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:14 smithi184 ceph-mon[92527]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T14:30:14.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:14 smithi184 ceph-mon[92527]: osdmap e623: 8 total, 7 up, 8 in 2024-09-13T14:30:14.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:14 smithi184 ceph-mon[92527]: from='osd.3 [v2:172.21.15.151:6826/2997668861,v1:172.21.15.151:6827/2997668861]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-13T14:30:14.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:14 smithi184 ceph-mon[92527]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-13T14:30:14.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:14 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 27/408 objects degraded (6.618%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:30:14.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:30:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:30:14] "GET /metrics HTTP/1.1" 200 36513 "" "Prometheus/2.51.0" 2024-09-13T14:30:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:14 smithi151 ceph-mon[104450]: pgmap v552: 89 pgs: 28 active+undersized, 6 active+undersized+degraded, 55 active+clean; 205 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 6.2 MiB/s rd, 808 KiB/s wr, 8 op/s; 27/408 objects degraded (6.618%) 2024-09-13T14:30:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:14 smithi151 ceph-mon[104450]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T14:30:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:14 smithi151 ceph-mon[104450]: osdmap e623: 8 total, 7 up, 8 in 2024-09-13T14:30:14.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:14 smithi151 ceph-mon[104450]: from='osd.3 [v2:172.21.15.151:6826/2997668861,v1:172.21.15.151:6827/2997668861]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-13T14:30:14.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:14 smithi151 ceph-mon[104450]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-13T14:30:14.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:14 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 27/408 objects degraded (6.618%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:30:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:14 smithi151 ceph-mon[108966]: pgmap v552: 89 pgs: 28 active+undersized, 6 active+undersized+degraded, 55 active+clean; 205 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 6.2 MiB/s rd, 808 KiB/s wr, 8 op/s; 27/408 objects degraded (6.618%) 2024-09-13T14:30:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:14 smithi151 ceph-mon[108966]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T14:30:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:14 smithi151 ceph-mon[108966]: osdmap e623: 8 total, 7 up, 8 in 2024-09-13T14:30:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:14 smithi151 ceph-mon[108966]: from='osd.3 [v2:172.21.15.151:6826/2997668861,v1:172.21.15.151:6827/2997668861]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-13T14:30:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:14 smithi151 ceph-mon[108966]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-13T14:30:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:14 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 27/408 objects degraded (6.618%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:30:15.213 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 4 2024-09-13T14:30:15.567 INFO:tasks.rados.rados.0.smithi151.stdout:2167: done (2 left) 2024-09-13T14:30:15.567 INFO:tasks.rados.rados.0.smithi151.stdout:2168: delete oid 17 current snap is 283 2024-09-13T14:30:15.567 INFO:tasks.rados.rados.0.smithi151.stdout:2166: finishing write tid 1 to smithi15186934-32 2024-09-13T14:30:15.567 INFO:tasks.rados.rados.0.smithi151.stdout:2166: finishing write tid 2 to smithi15186934-32 2024-09-13T14:30:15.567 INFO:tasks.rados.rados.0.smithi151.stdout:2166: finishing write tid 3 to smithi15186934-32 2024-09-13T14:30:15.567 INFO:tasks.rados.rados.0.smithi151.stdout:2166: finishing write tid 4 to smithi15186934-32 2024-09-13T14:30:15.567 INFO:tasks.rados.rados.0.smithi151.stdout:2166: finishing write tid 5 to smithi15186934-32 2024-09-13T14:30:15.567 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 32 v 863 (ObjNum 809 snap 283 seq_num 809) dirty exists 2024-09-13T14:30:15.568 INFO:tasks.rados.rados.0.smithi151.stdout:2166: left oid 32 (ObjNum 809 snap 283 seq_num 809) 2024-09-13T14:30:15.589 INFO:tasks.rados.rados.0.smithi151.stdout:2165: expect (ObjNum 654 snap 212 seq_num 654) 2024-09-13T14:30:15.697 INFO:tasks.rados.rados.0.smithi151.stdout:2165: done (2 left) 2024-09-13T14:30:15.697 INFO:tasks.rados.rados.0.smithi151.stdout:2166: done (1 left) 2024-09-13T14:30:15.697 INFO:tasks.rados.rados.0.smithi151.stdout:2168: done (0 left) 2024-09-13T14:30:15.697 INFO:tasks.rados.rados.0.smithi151.stdout:2169: rollback oid 13 current snap is 283 2024-09-13T14:30:15.697 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 13 to 243 2024-09-13T14:30:15.697 INFO:tasks.rados.rados.0.smithi151.stdout:2170: delete oid 33 current snap is 283 2024-09-13T14:30:15.699 INFO:tasks.rados.rados.0.smithi151.stdout:2169: finishing rollback tid 1 to smithi15186934-13 2024-09-13T14:30:15.700 INFO:tasks.rados.rados.0.smithi151.stdout:2169: finishing rollback tid 2 to smithi15186934-13 2024-09-13T14:30:15.700 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 13 v 832 (ObjNum 627 snap 204 seq_num 627) dirty exists 2024-09-13T14:30:15.700 INFO:tasks.rados.rados.0.smithi151.stdout:2169: done (1 left) 2024-09-13T14:30:15.700 INFO:tasks.rados.rados.0.smithi151.stdout:2170: done (0 left) 2024-09-13T14:30:15.700 INFO:tasks.rados.rados.0.smithi151.stdout:2171: snap_create 2024-09-13T14:30:15.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:15 smithi184 ceph-mon[92527]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:30:15.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:15 smithi184 ceph-mon[92527]: osd.3 [v2:172.21.15.151:6826/2997668861,v1:172.21.15.151:6827/2997668861] boot 2024-09-13T14:30:15.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:15 smithi184 ceph-mon[92527]: osdmap e624: 8 total, 8 up, 8 in 2024-09-13T14:30:15.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:15 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T14:30:15.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:15 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/254526416' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:30:15.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:15 smithi151 ceph-mon[104450]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:30:15.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:15 smithi151 ceph-mon[104450]: osd.3 [v2:172.21.15.151:6826/2997668861,v1:172.21.15.151:6827/2997668861] boot 2024-09-13T14:30:15.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:15 smithi151 ceph-mon[104450]: osdmap e624: 8 total, 8 up, 8 in 2024-09-13T14:30:15.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:15 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T14:30:15.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:15 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/254526416' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:30:15.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:15 smithi151 ceph-mon[108966]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:30:15.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:15 smithi151 ceph-mon[108966]: osd.3 [v2:172.21.15.151:6826/2997668861,v1:172.21.15.151:6827/2997668861] boot 2024-09-13T14:30:15.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:15 smithi151 ceph-mon[108966]: osdmap e624: 8 total, 8 up, 8 in 2024-09-13T14:30:15.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:15 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T14:30:15.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:15 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/254526416' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-13T14:30:16.018 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch upgrade pause' 2024-09-13T14:30:16.265 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:30:16.568 INFO:tasks.rados.rados.0.smithi151.stdout:2171: done (0 left) 2024-09-13T14:30:16.569 INFO:tasks.rados.rados.0.smithi151.stdout:2172: write oid 4 current snap is 284 2024-09-13T14:30:16.569 INFO:tasks.rados.rados.0.smithi151.stdout:2172: seq_num 812 ranges {447284=635606,1866555=733612,3315823=331461} 2024-09-13T14:30:16.574 INFO:tasks.rados.rados.0.smithi151.stdout:2172: writing smithi15186934-4 from 447284 to 1082890 tid 1 2024-09-13T14:30:16.582 INFO:tasks.rados.rados.0.smithi151.stdout:2172: writing smithi15186934-4 from 1866555 to 2600167 tid 2 2024-09-13T14:30:16.589 INFO:tasks.rados.rados.0.smithi151.stdout:2172: writing smithi15186934-4 from 3315823 to 3647284 tid 3 2024-09-13T14:30:16.589 INFO:tasks.rados.rados.0.smithi151.stdout:2173: read oid 37 snap -1 2024-09-13T14:30:16.589 INFO:tasks.rados.rados.0.smithi151.stdout:2173: expect (ObjNum 683 snap 222 seq_num 683) 2024-09-13T14:30:16.589 INFO:tasks.rados.rados.0.smithi151.stdout:2174: snap_remove snap 270 2024-09-13T14:30:16.783 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:16 smithi151 ceph-mon[104450]: pgmap v555: 89 pgs: 28 active+undersized, 6 active+undersized+degraded, 55 active+clean; 208 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 785 KiB/s wr, 12 op/s; 27/414 objects degraded (6.522%) 2024-09-13T14:30:16.783 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:16 smithi151 ceph-mon[104450]: osdmap e625: 8 total, 8 up, 8 in 2024-09-13T14:30:16.783 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:16.783 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:16.784 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:16 smithi151 ceph-mon[108966]: pgmap v555: 89 pgs: 28 active+undersized, 6 active+undersized+degraded, 55 active+clean; 208 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 785 KiB/s wr, 12 op/s; 27/414 objects degraded (6.522%) 2024-09-13T14:30:16.784 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:16 smithi151 ceph-mon[108966]: osdmap e625: 8 total, 8 up, 8 in 2024-09-13T14:30:16.784 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:16.784 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:16.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:16 smithi184 ceph-mon[92527]: pgmap v555: 89 pgs: 28 active+undersized, 6 active+undersized+degraded, 55 active+clean; 208 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 785 KiB/s wr, 12 op/s; 27/414 objects degraded (6.522%) 2024-09-13T14:30:16.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:16 smithi184 ceph-mon[92527]: osdmap e625: 8 total, 8 up, 8 in 2024-09-13T14:30:16.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:16.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:17.058 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:30:16 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T14:30:16.945+0000 7fc94c153640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-09-13T14:30:17.579 INFO:tasks.rados.rados.0.smithi151.stdout:2174: done (2 left) 2024-09-13T14:30:17.579 INFO:tasks.rados.rados.0.smithi151.stdout:2175: write oid 33 current snap is 284 2024-09-13T14:30:17.579 INFO:tasks.rados.rados.0.smithi151.stdout:2172: finishing write tid 1 to smithi15186934-4 2024-09-13T14:30:17.580 INFO:tasks.rados.rados.0.smithi151.stdout:2175: seq_num 813 ranges {464091=710855,1925037=477270,2943246=474970} 2024-09-13T14:30:17.587 INFO:tasks.rados.rados.0.smithi151.stdout:2175: writing smithi15186934-33 from 464091 to 1174946 tid 1 2024-09-13T14:30:17.594 INFO:tasks.rados.rados.0.smithi151.stdout:2175: writing smithi15186934-33 from 1925037 to 2402307 tid 2 2024-09-13T14:30:17.600 INFO:tasks.rados.rados.0.smithi151.stdout:2175: writing smithi15186934-33 from 2943246 to 3418216 tid 3 2024-09-13T14:30:17.600 INFO:tasks.rados.rados.0.smithi151.stdout:2176: rollback oid 32 current snap is 284 2024-09-13T14:30:17.600 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 32 to 283 2024-09-13T14:30:17.600 INFO:tasks.rados.rados.0.smithi151.stdout:2172: finishing write tid 2 to smithi15186934-4 2024-09-13T14:30:17.601 INFO:tasks.rados.rados.0.smithi151.stdout:2172: finishing write tid 3 to smithi15186934-4 2024-09-13T14:30:17.601 INFO:tasks.rados.rados.0.smithi151.stdout:2172: finishing write tid 4 to smithi15186934-4 2024-09-13T14:30:17.601 INFO:tasks.rados.rados.0.smithi151.stdout:2172: finishing write tid 5 to smithi15186934-4 2024-09-13T14:30:17.601 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 1168 (ObjNum 812 snap 284 seq_num 812) dirty exists 2024-09-13T14:30:17.601 INFO:tasks.rados.rados.0.smithi151.stdout:2172: left oid 4 (ObjNum 812 snap 284 seq_num 812) 2024-09-13T14:30:17.601 INFO:tasks.rados.rados.0.smithi151.stdout:2173: expect (ObjNum 683 snap 222 seq_num 683) 2024-09-13T14:30:17.823 INFO:tasks.rados.rados.0.smithi151.stdout:2175: finishing write tid 1 to smithi15186934-33 2024-09-13T14:30:17.823 INFO:tasks.rados.rados.0.smithi151.stdout:2175: finishing write tid 2 to smithi15186934-33 2024-09-13T14:30:17.823 INFO:tasks.rados.rados.0.smithi151.stdout:2175: finishing write tid 3 to smithi15186934-33 2024-09-13T14:30:17.823 INFO:tasks.rados.rados.0.smithi151.stdout:2175: finishing write tid 4 to smithi15186934-33 2024-09-13T14:30:17.823 INFO:tasks.rados.rados.0.smithi151.stdout:2176: finishing rollback tid 0 to smithi15186934-32 2024-09-13T14:30:17.824 INFO:tasks.rados.rados.0.smithi151.stdout:2175: finishing write tid 5 to smithi15186934-33 2024-09-13T14:30:17.824 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 33 v 844 (ObjNum 813 snap 284 seq_num 813) dirty exists 2024-09-13T14:30:17.824 INFO:tasks.rados.rados.0.smithi151.stdout:2175: left oid 33 (ObjNum 813 snap 284 seq_num 813) 2024-09-13T14:30:17.824 INFO:tasks.rados.rados.0.smithi151.stdout:2176: finishing rollback tid 1 to smithi15186934-32 2024-09-13T14:30:17.824 INFO:tasks.rados.rados.0.smithi151.stdout:2172: done (3 left) 2024-09-13T14:30:17.824 INFO:tasks.rados.rados.0.smithi151.stdout:2173: done (2 left) 2024-09-13T14:30:17.824 INFO:tasks.rados.rados.0.smithi151.stdout:2175: done (1 left) 2024-09-13T14:30:17.824 INFO:tasks.rados.rados.0.smithi151.stdout:2177: snap_create 2024-09-13T14:30:17.824 INFO:tasks.rados.rados.0.smithi151.stdout:2176: finishing rollback tid 2 to smithi15186934-32 2024-09-13T14:30:17.824 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 32 v 875 (ObjNum 809 snap 283 seq_num 809) dirty exists 2024-09-13T14:30:17.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:17 smithi184 ceph-mon[92527]: osdmap e626: 8 total, 8 up, 8 in 2024-09-13T14:30:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:30:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:30:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:30:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:30:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:30:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-13T14:30:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:17 smithi184 ceph-mon[92527]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-13T14:30:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:17 smithi184 ceph-mon[92527]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-09-13T14:30:17.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:17 smithi151 ceph-mon[104450]: osdmap e626: 8 total, 8 up, 8 in 2024-09-13T14:30:17.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:17.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:17.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:17.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:30:17.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:17.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:30:17.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:30:17.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:30:17.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:30:17.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-13T14:30:17.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:17 smithi151 ceph-mon[104450]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-13T14:30:17.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:17 smithi151 ceph-mon[104450]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-09-13T14:30:17.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:17 smithi151 ceph-mon[108966]: osdmap e626: 8 total, 8 up, 8 in 2024-09-13T14:30:17.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:17.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:17.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:17.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:30:17.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:17.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:30:17.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:30:17.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:30:17.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:30:17.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-13T14:30:17.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:17 smithi151 ceph-mon[108966]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-13T14:30:17.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:17 smithi151 ceph-mon[108966]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-09-13T14:30:18.582 INFO:tasks.rados.rados.0.smithi151.stdout:2176: done (1 left) 2024-09-13T14:30:18.582 INFO:tasks.rados.rados.0.smithi151.stdout:2177: done (0 left) 2024-09-13T14:30:18.582 INFO:tasks.rados.rados.0.smithi151.stdout:2178: write oid 29 current snap is 285 2024-09-13T14:30:18.582 INFO:tasks.rados.rados.0.smithi151.stdout:2178: seq_num 814 ranges {480898=502457,1699872=737281,3087022=593876} 2024-09-13T14:30:18.587 INFO:tasks.rados.rados.0.smithi151.stdout:2178: writing smithi15186934-29 from 480898 to 983355 tid 1 2024-09-13T14:30:18.595 INFO:tasks.rados.rados.0.smithi151.stdout:2178: writing smithi15186934-29 from 1699872 to 2437153 tid 2 2024-09-13T14:30:18.603 INFO:tasks.rados.rados.0.smithi151.stdout:2178: writing smithi15186934-29 from 3087022 to 3680898 tid 3 2024-09-13T14:30:18.603 INFO:tasks.rados.rados.0.smithi151.stdout:2179: read oid 21 snap 250 2024-09-13T14:30:18.603 INFO:tasks.rados.rados.0.smithi151.stdout:2179: expect (ObjNum 706 snap 237 seq_num 706) 2024-09-13T14:30:18.603 INFO:tasks.rados.rados.0.smithi151.stdout:2180: rollback oid 46 current snap is 285 2024-09-13T14:30:18.603 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 46 to 276 2024-09-13T14:30:18.603 INFO:tasks.rados.rados.0.smithi151.stdout:2178: finishing write tid 1 to smithi15186934-29 2024-09-13T14:30:18.603 INFO:tasks.rados.rados.0.smithi151.stdout:2178: finishing write tid 2 to smithi15186934-29 2024-09-13T14:30:18.603 INFO:tasks.rados.rados.0.smithi151.stdout:2181: write oid 34 current snap is 285 2024-09-13T14:30:18.603 INFO:tasks.rados.rados.0.smithi151.stdout:2181: seq_num 815 ranges {497705=577706,1642001=480939,2598092=497514} 2024-09-13T14:30:18.609 INFO:tasks.rados.rados.0.smithi151.stdout:2181: writing smithi15186934-34 from 497705 to 1075411 tid 1 2024-09-13T14:30:18.616 INFO:tasks.rados.rados.0.smithi151.stdout:2181: writing smithi15186934-34 from 1642001 to 2122940 tid 2 2024-09-13T14:30:18.622 INFO:tasks.rados.rados.0.smithi151.stdout:2181: writing smithi15186934-34 from 2598092 to 3095606 tid 3 2024-09-13T14:30:18.622 INFO:tasks.rados.rados.0.smithi151.stdout:2182: delete oid 28 current snap is 285 2024-09-13T14:30:18.622 INFO:tasks.rados.rados.0.smithi151.stdout:2178: finishing write tid 3 to smithi15186934-29 2024-09-13T14:30:18.622 INFO:tasks.rados.rados.0.smithi151.stdout:2178: finishing write tid 4 to smithi15186934-29 2024-09-13T14:30:18.622 INFO:tasks.rados.rados.0.smithi151.stdout:2178: finishing write tid 5 to smithi15186934-29 2024-09-13T14:30:18.623 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 29 v 565 (ObjNum 814 snap 285 seq_num 814) dirty exists 2024-09-13T14:30:18.623 INFO:tasks.rados.rados.0.smithi151.stdout:2178: left oid 29 (ObjNum 814 snap 285 seq_num 814) 2024-09-13T14:30:18.623 INFO:tasks.rados.rados.0.smithi151.stdout:2180: finishing rollback tid 0 to smithi15186934-46 2024-09-13T14:30:18.623 INFO:tasks.rados.rados.0.smithi151.stdout:2180: finishing rollback tid 1 to smithi15186934-46 2024-09-13T14:30:18.623 INFO:tasks.rados.rados.0.smithi151.stdout:2180: finishing rollback tid 2 to smithi15186934-46 2024-09-13T14:30:18.623 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 46 v 793 (ObjNum 786 snap 275 seq_num 786) dirty exists 2024-09-13T14:30:18.623 INFO:tasks.rados.rados.0.smithi151.stdout:2181: finishing write tid 1 to smithi15186934-34 2024-09-13T14:30:18.623 INFO:tasks.rados.rados.0.smithi151.stdout:2179: expect (ObjNum 706 snap 237 seq_num 706) 2024-09-13T14:30:18.692 INFO:teuthology.orchestra.run.smithi151.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:30:18.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:18 smithi184 ceph-mon[92527]: pgmap v558: 89 pgs: 3 peering, 16 active+undersized, 3 active+undersized+degraded, 67 active+clean; 219 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 3.1 MiB/s wr, 22 op/s; 12/432 objects degraded (2.778%); 1.0 MiB/s, 0 objects/s recovering 2024-09-13T14:30:18.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:18 smithi184 ceph-mon[92527]: osdmap e627: 8 total, 8 up, 8 in 2024-09-13T14:30:18.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:18 smithi151 ceph-mon[104450]: pgmap v558: 89 pgs: 3 peering, 16 active+undersized, 3 active+undersized+degraded, 67 active+clean; 219 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 3.1 MiB/s wr, 22 op/s; 12/432 objects degraded (2.778%); 1.0 MiB/s, 0 objects/s recovering 2024-09-13T14:30:18.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:18 smithi151 ceph-mon[104450]: osdmap e627: 8 total, 8 up, 8 in 2024-09-13T14:30:18.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:18 smithi151 ceph-mon[108966]: pgmap v558: 89 pgs: 3 peering, 16 active+undersized, 3 active+undersized+degraded, 67 active+clean; 219 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 3.1 MiB/s wr, 22 op/s; 12/432 objects degraded (2.778%); 1.0 MiB/s, 0 objects/s recovering 2024-09-13T14:30:18.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:18 smithi151 ceph-mon[108966]: osdmap e627: 8 total, 8 up, 8 in 2024-09-13T14:30:18.908 INFO:tasks.rados.rados.0.smithi151.stdout:2181: finishing write tid 2 to smithi15186934-34 2024-09-13T14:30:18.908 INFO:tasks.rados.rados.0.smithi151.stdout:2181: finishing write tid 3 to smithi15186934-34 2024-09-13T14:30:18.908 INFO:tasks.rados.rados.0.smithi151.stdout:2181: finishing write tid 4 to smithi15186934-34 2024-09-13T14:30:18.909 INFO:tasks.rados.rados.0.smithi151.stdout:2181: finishing write tid 5 to smithi15186934-34 2024-09-13T14:30:18.909 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 818 (ObjNum 815 snap 285 seq_num 815) dirty exists 2024-09-13T14:30:18.909 INFO:tasks.rados.rados.0.smithi151.stdout:2181: left oid 34 (ObjNum 815 snap 285 seq_num 815) 2024-09-13T14:30:18.909 INFO:tasks.rados.rados.0.smithi151.stdout:2178: done (4 left) 2024-09-13T14:30:18.909 INFO:tasks.rados.rados.0.smithi151.stdout:2179: done (3 left) 2024-09-13T14:30:18.909 INFO:tasks.rados.rados.0.smithi151.stdout:2180: done (2 left) 2024-09-13T14:30:18.909 INFO:tasks.rados.rados.0.smithi151.stdout:2181: done (1 left) 2024-09-13T14:30:18.909 INFO:tasks.rados.rados.0.smithi151.stdout:2182: done (0 left) 2024-09-13T14:30:18.909 INFO:tasks.rados.rados.0.smithi151.stdout:2183: snap_create 2024-09-13T14:30:19.447 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch ps' 2024-09-13T14:30:19.697 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:30:19.700 INFO:tasks.rados.rados.0.smithi151.stdout:2183: done (0 left) 2024-09-13T14:30:19.700 INFO:tasks.rados.rados.0.smithi151.stdout:2184: read oid 40 snap -1 2024-09-13T14:30:19.700 INFO:tasks.rados.rados.0.smithi151.stdout:2184: expect (ObjNum 741 snap 256 seq_num 741) 2024-09-13T14:30:19.701 INFO:tasks.rados.rados.0.smithi151.stdout:2185: delete oid 33 current snap is 286 2024-09-13T14:30:19.707 INFO:tasks.rados.rados.0.smithi151.stdout:2184: expect (ObjNum 741 snap 256 seq_num 741) 2024-09-13T14:30:19.789 INFO:tasks.rados.rados.0.smithi151.stdout:2184: done (1 left) 2024-09-13T14:30:19.789 INFO:tasks.rados.rados.0.smithi151.stdout:2185: done (0 left) 2024-09-13T14:30:19.789 INFO:tasks.rados.rados.0.smithi151.stdout:2186: write oid 31 current snap is 286 2024-09-13T14:30:19.789 INFO:tasks.rados.rados.0.smithi151.stdout:2186: seq_num 818 ranges {548126=403453,1417447=744619,2680361=447683,3680450=67676} 2024-09-13T14:30:19.794 INFO:tasks.rados.rados.0.smithi151.stdout:2186: writing smithi15186934-31 from 548126 to 951579 tid 1 2024-09-13T14:30:19.801 INFO:tasks.rados.rados.0.smithi151.stdout:2186: writing smithi15186934-31 from 1417447 to 2162066 tid 2 2024-09-13T14:30:19.806 INFO:tasks.rados.rados.0.smithi151.stdout:2186: writing smithi15186934-31 from 2680361 to 3128044 tid 3 2024-09-13T14:30:19.809 INFO:tasks.rados.rados.0.smithi151.stdout:2186: writing smithi15186934-31 from 3680450 to 3748126 tid 4 2024-09-13T14:30:19.810 INFO:tasks.rados.rados.0.smithi151.stdout:2187: snap_remove snap 277 2024-09-13T14:30:19.820 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:19 smithi151 ceph-mon[108966]: osdmap e628: 8 total, 8 up, 8 in 2024-09-13T14:30:19.820 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:19 smithi151 ceph-mon[108966]: from='client.35040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:19.820 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:19 smithi151 ceph-mon[108966]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:30:19.820 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:19 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:19.820 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:19 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 12/432 objects degraded (2.778%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:30:19.820 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:19 smithi151 ceph-mon[104450]: osdmap e628: 8 total, 8 up, 8 in 2024-09-13T14:30:19.821 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:19 smithi151 ceph-mon[104450]: from='client.35040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:19.821 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:19 smithi151 ceph-mon[104450]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:30:19.821 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:19 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:19.821 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:19 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 12/432 objects degraded (2.778%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:30:19.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:19 smithi184 ceph-mon[92527]: osdmap e628: 8 total, 8 up, 8 in 2024-09-13T14:30:19.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:19 smithi184 ceph-mon[92527]: from='client.35040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:19.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:19 smithi184 ceph-mon[92527]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:30:19.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:19 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:19.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:19 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 12/432 objects degraded (2.778%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:30:20.583 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:30:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:30:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:30:20.704 INFO:tasks.rados.rados.0.smithi151.stdout:2187: done (1 left) 2024-09-13T14:30:20.704 INFO:tasks.rados.rados.0.smithi151.stdout:2186: finishing write tid 1 to smithi15186934-31 2024-09-13T14:30:20.705 INFO:tasks.rados.rados.0.smithi151.stdout:2186: finishing write tid 2 to smithi15186934-31 2024-09-13T14:30:20.705 INFO:tasks.rados.rados.0.smithi151.stdout:2186: finishing write tid 3 to smithi15186934-31 2024-09-13T14:30:20.705 INFO:tasks.rados.rados.0.smithi151.stdout:2186: finishing write tid 4 to smithi15186934-31 2024-09-13T14:30:20.705 INFO:tasks.rados.rados.0.smithi151.stdout:2186: finishing write tid 5 to smithi15186934-31 2024-09-13T14:30:20.705 INFO:tasks.rados.rados.0.smithi151.stdout:2186: finishing write tid 6 to smithi15186934-31 2024-09-13T14:30:20.705 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 31 v 925 (ObjNum 818 snap 286 seq_num 818) dirty exists 2024-09-13T14:30:20.705 INFO:tasks.rados.rados.0.smithi151.stdout:2186: left oid 31 (ObjNum 818 snap 286 seq_num 818) 2024-09-13T14:30:20.705 INFO:tasks.rados.rados.0.smithi151.stdout:2188: read oid 32 snap 283 2024-09-13T14:30:20.705 INFO:tasks.rados.rados.0.smithi151.stdout:2188: expect (ObjNum 809 snap 283 seq_num 809) 2024-09-13T14:30:20.705 INFO:tasks.rados.rados.0.smithi151.stdout:2186: done (1 left) 2024-09-13T14:30:20.705 INFO:tasks.rados.rados.0.smithi151.stdout:2189: write oid 14 current snap is 286 2024-09-13T14:30:20.705 INFO:tasks.rados.rados.0.smithi151.stdout:2189: seq_num 819 ranges {564933=478702,1759576=488277,2991431=658955} 2024-09-13T14:30:20.710 INFO:tasks.rados.rados.0.smithi151.stdout:2189: writing smithi15186934-14 from 564933 to 1043635 tid 1 2024-09-13T14:30:20.717 INFO:tasks.rados.rados.0.smithi151.stdout:2189: writing smithi15186934-14 from 1759576 to 2247853 tid 2 2024-09-13T14:30:20.725 INFO:tasks.rados.rados.0.smithi151.stdout:2189: writing smithi15186934-14 from 2991431 to 3650386 tid 3 2024-09-13T14:30:20.726 INFO:tasks.rados.rados.0.smithi151.stdout:2190: read oid 31 snap -1 2024-09-13T14:30:20.726 INFO:tasks.rados.rados.0.smithi151.stdout:2190: expect (ObjNum 818 snap 286 seq_num 818) 2024-09-13T14:30:20.726 INFO:tasks.rados.rados.0.smithi151.stdout:2191: write oid 4 current snap is 286 2024-09-13T14:30:20.726 INFO:tasks.rados.rados.0.smithi151.stdout:2191: seq_num 820 ranges {581740=553951,1818058=748288,3018854=470227} 2024-09-13T14:30:20.732 INFO:tasks.rados.rados.0.smithi151.stdout:2191: writing smithi15186934-4 from 581740 to 1135691 tid 1 2024-09-13T14:30:20.740 INFO:tasks.rados.rados.0.smithi151.stdout:2191: writing smithi15186934-4 from 1818058 to 2566346 tid 2 2024-09-13T14:30:20.746 INFO:tasks.rados.rados.0.smithi151.stdout:2191: writing smithi15186934-4 from 3018854 to 3489081 tid 3 2024-09-13T14:30:20.746 INFO:tasks.rados.rados.0.smithi151.stdout:2192: delete oid 45 current snap is 286 2024-09-13T14:30:20.746 INFO:tasks.rados.rados.0.smithi151.stdout:2188: expect (ObjNum 809 snap 283 seq_num 809) 2024-09-13T14:30:20.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:20 smithi184 ceph-mon[92527]: pgmap v561: 89 pgs: 3 peering, 86 active+clean; 218 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 3.9 MiB/s rd, 3.2 MiB/s wr, 17 op/s; 1.0 MiB/s, 0 objects/s recovering 2024-09-13T14:30:20.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:20 smithi184 ceph-mon[92527]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/432 objects degraded (2.778%), 3 pgs degraded) 2024-09-13T14:30:20.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:20 smithi184 ceph-mon[92527]: Cluster is now healthy 2024-09-13T14:30:20.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:20 smithi184 ceph-mon[92527]: osdmap e629: 8 total, 8 up, 8 in 2024-09-13T14:30:20.857 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:20 smithi151 ceph-mon[104450]: pgmap v561: 89 pgs: 3 peering, 86 active+clean; 218 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 3.9 MiB/s rd, 3.2 MiB/s wr, 17 op/s; 1.0 MiB/s, 0 objects/s recovering 2024-09-13T14:30:20.857 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:20 smithi151 ceph-mon[104450]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/432 objects degraded (2.778%), 3 pgs degraded) 2024-09-13T14:30:20.857 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:20 smithi151 ceph-mon[104450]: Cluster is now healthy 2024-09-13T14:30:20.857 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:20 smithi151 ceph-mon[104450]: osdmap e629: 8 total, 8 up, 8 in 2024-09-13T14:30:20.857 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:20 smithi151 ceph-mon[108966]: pgmap v561: 89 pgs: 3 peering, 86 active+clean; 218 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 3.9 MiB/s rd, 3.2 MiB/s wr, 17 op/s; 1.0 MiB/s, 0 objects/s recovering 2024-09-13T14:30:20.857 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:20 smithi151 ceph-mon[108966]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/432 objects degraded (2.778%), 3 pgs degraded) 2024-09-13T14:30:20.857 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:20 smithi151 ceph-mon[108966]: Cluster is now healthy 2024-09-13T14:30:20.857 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:20 smithi151 ceph-mon[108966]: osdmap e629: 8 total, 8 up, 8 in 2024-09-13T14:30:20.977 INFO:tasks.rados.rados.0.smithi151.stdout:2189: finishing write tid 1 to smithi15186934-14 2024-09-13T14:30:20.977 INFO:tasks.rados.rados.0.smithi151.stdout:2189: finishing write tid 2 to smithi15186934-14 2024-09-13T14:30:20.977 INFO:tasks.rados.rados.0.smithi151.stdout:2189: finishing write tid 3 to smithi15186934-14 2024-09-13T14:30:20.977 INFO:tasks.rados.rados.0.smithi151.stdout:2189: finishing write tid 4 to smithi15186934-14 2024-09-13T14:30:20.977 INFO:tasks.rados.rados.0.smithi151.stdout:2189: finishing write tid 5 to smithi15186934-14 2024-09-13T14:30:20.977 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 884 (ObjNum 819 snap 286 seq_num 819) dirty exists 2024-09-13T14:30:20.978 INFO:tasks.rados.rados.0.smithi151.stdout:2189: left oid 14 (ObjNum 819 snap 286 seq_num 819) 2024-09-13T14:30:20.978 INFO:tasks.rados.rados.0.smithi151.stdout:2190: expect (ObjNum 818 snap 286 seq_num 818) 2024-09-13T14:30:21.145 INFO:tasks.rados.rados.0.smithi151.stdout:2191: finishing write tid 1 to smithi15186934-4 2024-09-13T14:30:21.145 INFO:tasks.rados.rados.0.smithi151.stdout:2191: finishing write tid 2 to smithi15186934-4 2024-09-13T14:30:21.145 INFO:tasks.rados.rados.0.smithi151.stdout:2191: finishing write tid 3 to smithi15186934-4 2024-09-13T14:30:21.145 INFO:tasks.rados.rados.0.smithi151.stdout:2191: finishing write tid 4 to smithi15186934-4 2024-09-13T14:30:21.145 INFO:tasks.rados.rados.0.smithi151.stdout:2191: finishing write tid 5 to smithi15186934-4 2024-09-13T14:30:21.145 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 1175 (ObjNum 820 snap 286 seq_num 820) dirty exists 2024-09-13T14:30:21.145 INFO:tasks.rados.rados.0.smithi151.stdout:2191: left oid 4 (ObjNum 820 snap 286 seq_num 820) 2024-09-13T14:30:21.145 INFO:tasks.rados.rados.0.smithi151.stdout:2188: done (4 left) 2024-09-13T14:30:21.145 INFO:tasks.rados.rados.0.smithi151.stdout:2189: done (3 left) 2024-09-13T14:30:21.145 INFO:tasks.rados.rados.0.smithi151.stdout:2190: done (2 left) 2024-09-13T14:30:21.146 INFO:tasks.rados.rados.0.smithi151.stdout:2191: done (1 left) 2024-09-13T14:30:21.146 INFO:tasks.rados.rados.0.smithi151.stdout:2192: done (0 left) 2024-09-13T14:30:21.146 INFO:tasks.rados.rados.0.smithi151.stdout:2193: snap_remove snap 243 2024-09-13T14:30:21.472 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:30:21.472 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.a smithi151 *:9093,9094 running (7m) 9s ago 11m 18.9M - 0.27.0 11f11916f8cd ed70c3d75107 2024-09-13T14:30:21.472 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.a smithi184 *:3000 running (7m) 42s ago 11m 87.0M - 9.4.7 954c08fa6188 4ba18fb1f1fc 2024-09-13T14:30:21.472 INFO:teuthology.orchestra.run.smithi151.stdout:mds.foo.smithi151.rqqklw smithi151 running (10m) 9s ago 10m 20.2M - 17.2.7-1627-gbbdadfa3 a994b00deb3a c960b57fc84b 2024-09-13T14:30:21.472 INFO:teuthology.orchestra.run.smithi151.stdout:mds.foo.smithi184.jismsi smithi184 running (10m) 42s ago 10m 16.0M - 17.2.7-1627-gbbdadfa3 a994b00deb3a 0cc31a3a5339 2024-09-13T14:30:21.473 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.x smithi184 *:8443,9283,8765 running (6m) 42s ago 15m 456M - 19.3.0-4697-gd58f7306 fec07016a2b0 8a0f7d440b6e 2024-09-13T14:30:21.473 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.y smithi151 *:8443,9283,8765 running (7m) 9s ago 17m 519M - 19.3.0-4697-gd58f7306 fec07016a2b0 d8c169a4f36e 2024-09-13T14:30:21.473 INFO:teuthology.orchestra.run.smithi151.stdout:mon.a smithi151 running (6m) 9s ago 17m 69.7M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 694de6b945cd 2024-09-13T14:30:21.473 INFO:teuthology.orchestra.run.smithi151.stdout:mon.b smithi184 running (3m) 42s ago 16m 42.3M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 12b6c0ccb635 2024-09-13T14:30:21.473 INFO:teuthology.orchestra.run.smithi151.stdout:mon.c smithi151 running (4m) 9s ago 16m 54.6M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 e548fceed2f9 2024-09-13T14:30:21.473 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.a smithi151 *:9100 running (7m) 9s ago 11m 10.6M - 1.7.0 72c9c2088986 d91e4cb09e9a 2024-09-13T14:30:21.473 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.b smithi184 *:9100 running (7m) 42s ago 11m 11.2M - 1.7.0 72c9c2088986 1e1a9dd643a0 2024-09-13T14:30:21.474 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (114s) 9s ago 15m 78.1M 2480M 19.3.0-4697-gd58f7306 fec07016a2b0 018b463f93c2 2024-09-13T14:30:21.474 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (81s) 9s ago 15m 57.0M 2480M 19.3.0-4697-gd58f7306 fec07016a2b0 04d393df5f43 2024-09-13T14:30:21.474 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (47s) 9s ago 14m 51.8M 2480M 19.3.0-4697-gd58f7306 fec07016a2b0 2faaa919b127 2024-09-13T14:30:21.474 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi151 running (13s) 9s ago 14m 12.0M 2480M 19.3.0-4697-gd58f7306 fec07016a2b0 31c786beb443 2024-09-13T14:30:21.474 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi184 running (13m) 42s ago 13m 105M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a 6e70699b8084 2024-09-13T14:30:21.474 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi184 running (13m) 42s ago 13m 107M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a 36b82c9e7230 2024-09-13T14:30:21.475 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6 smithi184 running (13m) 42s ago 13m 85.5M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a f9379cec7504 2024-09-13T14:30:21.475 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7 smithi184 running (12m) 42s ago 12m 94.2M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a f37cc947097b 2024-09-13T14:30:21.475 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.a smithi184 *:9095 running (6m) 42s ago 12m 55.8M - 2.51.0 1d3b7f56885b 3da9265b8f14 2024-09-13T14:30:21.717 INFO:tasks.rados.rados.0.smithi151.stdout:2193: done (0 left) 2024-09-13T14:30:21.717 INFO:tasks.rados.rados.0.smithi151.stdout:2194: write oid 25 current snap is 286 2024-09-13T14:30:21.718 INFO:tasks.rados.rados.0.smithi151.stdout:2194: seq_num 822 ranges {615354=420802,1651375=751957,3190053=492771} 2024-09-13T14:30:21.723 INFO:tasks.rados.rados.0.smithi151.stdout:2194: writing smithi15186934-25 from 615354 to 1036156 tid 1 2024-09-13T14:30:21.731 INFO:tasks.rados.rados.0.smithi151.stdout:2194: writing smithi15186934-25 from 1651375 to 2403332 tid 2 2024-09-13T14:30:21.739 INFO:tasks.rados.rados.0.smithi151.stdout:2194: writing smithi15186934-25 from 3190053 to 3682824 tid 3 2024-09-13T14:30:21.739 INFO:tasks.rados.rados.0.smithi151.stdout:2195: read oid 33 snap -1 2024-09-13T14:30:21.739 INFO:tasks.rados.rados.0.smithi151.stdout:2195: expect deleted 2024-09-13T14:30:21.739 INFO:tasks.rados.rados.0.smithi151.stdout:2196: write oid 50 current snap is 286 2024-09-13T14:30:21.739 INFO:tasks.rados.rados.0.smithi151.stdout:2194: finishing write tid 1 to smithi15186934-25 2024-09-13T14:30:21.739 INFO:tasks.rados.rados.0.smithi151.stdout:2194: finishing write tid 2 to smithi15186934-25 2024-09-13T14:30:21.739 INFO:tasks.rados.rados.0.smithi151.stdout:2196: seq_num 823 ranges {632161=496051,1593504=495615,2701123=420396,3611645=220516} 2024-09-13T14:30:21.744 INFO:tasks.rados.rados.0.smithi151.stdout:2196: writing smithi15186934-50 from 632161 to 1128212 tid 1 2024-09-13T14:30:21.750 INFO:tasks.rados.rados.0.smithi151.stdout:2196: writing smithi15186934-50 from 1593504 to 2089119 tid 2 2024-09-13T14:30:21.755 INFO:tasks.rados.rados.0.smithi151.stdout:2196: writing smithi15186934-50 from 2701123 to 3121519 tid 3 2024-09-13T14:30:21.760 INFO:tasks.rados.rados.0.smithi151.stdout:2196: writing smithi15186934-50 from 3611645 to 3832161 tid 4 2024-09-13T14:30:21.760 INFO:tasks.rados.rados.0.smithi151.stdout:2197: delete oid 42 current snap is 286 2024-09-13T14:30:21.761 INFO:tasks.rados.rados.0.smithi151.stdout:2194: finishing write tid 3 to smithi15186934-25 2024-09-13T14:30:21.761 INFO:tasks.rados.rados.0.smithi151.stdout:2194: finishing write tid 4 to smithi15186934-25 2024-09-13T14:30:21.761 INFO:tasks.rados.rados.0.smithi151.stdout:2194: finishing write tid 5 to smithi15186934-25 2024-09-13T14:30:21.761 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 1188 (ObjNum 822 snap 286 seq_num 822) dirty exists 2024-09-13T14:30:21.762 INFO:tasks.rados.rados.0.smithi151.stdout:2194: left oid 25 (ObjNum 822 snap 286 seq_num 822) 2024-09-13T14:30:21.762 INFO:tasks.rados.rados.0.smithi151.stdout:2196: finishing write tid 1 to smithi15186934-50 2024-09-13T14:30:21.762 INFO:tasks.rados.rados.0.smithi151.stdout:2196: finishing write tid 2 to smithi15186934-50 2024-09-13T14:30:21.762 INFO:tasks.rados.rados.0.smithi151.stdout:2196: finishing write tid 3 to smithi15186934-50 2024-09-13T14:30:21.763 INFO:tasks.rados.rados.0.smithi151.stdout:2194: done (3 left) 2024-09-13T14:30:21.763 INFO:tasks.rados.rados.0.smithi151.stdout:2195: done (2 left) 2024-09-13T14:30:21.763 INFO:tasks.rados.rados.0.smithi151.stdout:2197: done (1 left) 2024-09-13T14:30:21.764 INFO:tasks.rados.rados.0.smithi151.stdout:2198: write oid 6 current snap is 286 2024-09-13T14:30:21.764 INFO:tasks.rados.rados.0.smithi151.stdout:2198: seq_num 825 ranges {665775=646549,2110468=499284,3155969=442940} 2024-09-13T14:30:21.769 INFO:tasks.rados.rados.0.smithi151.stdout:2198: writing smithi15186934-6 from 665775 to 1312324 tid 1 2024-09-13T14:30:21.776 INFO:tasks.rados.rados.0.smithi151.stdout:2198: writing smithi15186934-6 from 2110468 to 2609752 tid 2 2024-09-13T14:30:21.782 INFO:tasks.rados.rados.0.smithi151.stdout:2198: writing smithi15186934-6 from 3155969 to 3598909 tid 3 2024-09-13T14:30:21.782 INFO:tasks.rados.rados.0.smithi151.stdout:2199: rollback oid 12 current snap is 286 2024-09-13T14:30:21.782 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 12 to 285 2024-09-13T14:30:21.782 INFO:tasks.rados.rados.0.smithi151.stdout:2196: finishing write tid 4 to smithi15186934-50 2024-09-13T14:30:21.782 INFO:tasks.rados.rados.0.smithi151.stdout:2196: finishing write tid 5 to smithi15186934-50 2024-09-13T14:30:21.783 INFO:tasks.rados.rados.0.smithi151.stdout:2196: finishing write tid 6 to smithi15186934-50 2024-09-13T14:30:21.783 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 50 v 784 (ObjNum 823 snap 286 seq_num 823) dirty exists 2024-09-13T14:30:21.783 INFO:tasks.rados.rados.0.smithi151.stdout:2196: left oid 50 (ObjNum 823 snap 286 seq_num 823) 2024-09-13T14:30:21.783 INFO:tasks.rados.rados.0.smithi151.stdout:2198: finishing write tid 1 to smithi15186934-6 2024-09-13T14:30:21.783 INFO:tasks.rados.rados.0.smithi151.stdout:2198: finishing write tid 2 to smithi15186934-6 2024-09-13T14:30:21.783 INFO:tasks.rados.rados.0.smithi151.stdout:2196: done (2 left) 2024-09-13T14:30:21.783 INFO:tasks.rados.rados.0.smithi151.stdout:2200: read oid 34 snap 281 2024-09-13T14:30:21.783 INFO:tasks.rados.rados.0.smithi151.stdout:2200: expect deleted 2024-09-13T14:30:21.783 INFO:tasks.rados.rados.0.smithi151.stdout:2201: snap_create 2024-09-13T14:30:21.784 INFO:tasks.rados.rados.0.smithi151.stdout:2198: finishing write tid 3 to smithi15186934-6 2024-09-13T14:30:21.784 INFO:tasks.rados.rados.0.smithi151.stdout:2198: finishing write tid 4 to smithi15186934-6 2024-09-13T14:30:21.785 INFO:tasks.rados.rados.0.smithi151.stdout:2198: finishing write tid 5 to smithi15186934-6 2024-09-13T14:30:21.785 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 6 v 565 (ObjNum 825 snap 286 seq_num 825) dirty exists 2024-09-13T14:30:21.785 INFO:tasks.rados.rados.0.smithi151.stdout:2198: left oid 6 (ObjNum 825 snap 286 seq_num 825) 2024-09-13T14:30:21.785 INFO:tasks.rados.rados.0.smithi151.stdout:2199: finishing rollback tid 0 to smithi15186934-12 2024-09-13T14:30:21.785 INFO:tasks.rados.rados.0.smithi151.stdout:2199: finishing rollback tid 1 to smithi15186934-12 2024-09-13T14:30:21.785 INFO:tasks.rados.rados.0.smithi151.stdout:2199: finishing rollback tid 2 to smithi15186934-12 2024-09-13T14:30:21.785 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 12 v 513 (ObjNum 654 snap 212 seq_num 654) dirty exists 2024-09-13T14:30:21.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:21 smithi151 ceph-mon[104450]: osdmap e630: 8 total, 8 up, 8 in 2024-09-13T14:30:21.787 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:21 smithi151 ceph-mon[108966]: osdmap e630: 8 total, 8 up, 8 in 2024-09-13T14:30:22.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:21 smithi184 ceph-mon[92527]: osdmap e630: 8 total, 8 up, 8 in 2024-09-13T14:30:22.173 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch ps' 2024-09-13T14:30:22.418 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:30:22.720 INFO:tasks.rados.rados.0.smithi151.stdout:2198: done (3 left) 2024-09-13T14:30:22.721 INFO:tasks.rados.rados.0.smithi151.stdout:2199: done (2 left) 2024-09-13T14:30:22.721 INFO:tasks.rados.rados.0.smithi151.stdout:2200: done (1 left) 2024-09-13T14:30:22.721 INFO:tasks.rados.rados.0.smithi151.stdout:2201: done (0 left) 2024-09-13T14:30:22.721 INFO:tasks.rados.rados.0.smithi151.stdout:2202: rollback oid 48 current snap is 287 2024-09-13T14:30:22.721 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 48 to 284 2024-09-13T14:30:22.721 INFO:tasks.rados.rados.0.smithi151.stdout:2203: snap_remove snap 278 2024-09-13T14:30:22.793 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:22 smithi151 ceph-mon[104450]: pgmap v564: 89 pgs: 89 active+clean; 223 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 3.8 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:30:22.793 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:22 smithi151 ceph-mon[104450]: from='client.44930 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:22.793 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:22 smithi151 ceph-mon[104450]: osdmap e631: 8 total, 8 up, 8 in 2024-09-13T14:30:22.794 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:22 smithi151 ceph-mon[108966]: pgmap v564: 89 pgs: 89 active+clean; 223 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 3.8 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:30:22.794 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:22 smithi151 ceph-mon[108966]: from='client.44930 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:22.794 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:22 smithi151 ceph-mon[108966]: osdmap e631: 8 total, 8 up, 8 in 2024-09-13T14:30:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:22 smithi184 ceph-mon[92527]: pgmap v564: 89 pgs: 89 active+clean; 223 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 3.8 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:30:23.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:22 smithi184 ceph-mon[92527]: from='client.44930 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:23.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:22 smithi184 ceph-mon[92527]: osdmap e631: 8 total, 8 up, 8 in 2024-09-13T14:30:23.728 INFO:tasks.rados.rados.0.smithi151.stdout:2203: done (1 left) 2024-09-13T14:30:23.728 INFO:tasks.rados.rados.0.smithi151.stdout:2204: delete oid 33 current snap is 287 2024-09-13T14:30:23.728 INFO:tasks.rados.rados.0.smithi151.stdout:2202: finishing rollback tid 1 to smithi15186934-48 2024-09-13T14:30:23.728 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 48 v 1074 (ObjNum 129 snap 0 seq_num 3092957168) dirty dne 2024-09-13T14:30:23.733 INFO:tasks.rados.rados.0.smithi151.stdout:2202: done (1 left) 2024-09-13T14:30:23.733 INFO:tasks.rados.rados.0.smithi151.stdout:2204: done (0 left) 2024-09-13T14:30:23.733 INFO:tasks.rados.rados.0.smithi151.stdout:2205: read oid 35 snap -1 2024-09-13T14:30:23.733 INFO:tasks.rados.rados.0.smithi151.stdout:2205: expect deleted 2024-09-13T14:30:23.734 INFO:tasks.rados.rados.0.smithi151.stdout:2206: read oid 30 snap -1 2024-09-13T14:30:23.734 INFO:tasks.rados.rados.0.smithi151.stdout:2206: expect (ObjNum 707 snap 237 seq_num 707) 2024-09-13T14:30:23.734 INFO:tasks.rados.rados.0.smithi151.stdout:2207: snap_create 2024-09-13T14:30:23.742 INFO:tasks.rados.rados.0.smithi151.stdout:2206: expect (ObjNum 707 snap 237 seq_num 707) 2024-09-13T14:30:23.959 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:23 smithi151 ceph-mon[104450]: osdmap e632: 8 total, 8 up, 8 in 2024-09-13T14:30:23.959 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:23 smithi151 ceph-mon[108966]: osdmap e632: 8 total, 8 up, 8 in 2024-09-13T14:30:24.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:23 smithi184 ceph-mon[92527]: osdmap e632: 8 total, 8 up, 8 in 2024-09-13T14:30:24.383 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:30:24.383 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.a smithi151 *:9093,9094 running (7m) 12s ago 11m 18.9M - 0.27.0 11f11916f8cd ed70c3d75107 2024-09-13T14:30:24.383 INFO:teuthology.orchestra.run.smithi151.stdout:grafana.a smithi184 *:3000 running (7m) 45s ago 11m 87.0M - 9.4.7 954c08fa6188 4ba18fb1f1fc 2024-09-13T14:30:24.383 INFO:teuthology.orchestra.run.smithi151.stdout:mds.foo.smithi151.rqqklw smithi151 running (10m) 12s ago 10m 20.2M - 17.2.7-1627-gbbdadfa3 a994b00deb3a c960b57fc84b 2024-09-13T14:30:24.383 INFO:teuthology.orchestra.run.smithi151.stdout:mds.foo.smithi184.jismsi smithi184 running (10m) 45s ago 10m 16.0M - 17.2.7-1627-gbbdadfa3 a994b00deb3a 0cc31a3a5339 2024-09-13T14:30:24.384 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.x smithi184 *:8443,9283,8765 running (6m) 45s ago 15m 456M - 19.3.0-4697-gd58f7306 fec07016a2b0 8a0f7d440b6e 2024-09-13T14:30:24.384 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.y smithi151 *:8443,9283,8765 running (7m) 12s ago 17m 519M - 19.3.0-4697-gd58f7306 fec07016a2b0 d8c169a4f36e 2024-09-13T14:30:24.384 INFO:teuthology.orchestra.run.smithi151.stdout:mon.a smithi151 running (6m) 12s ago 17m 69.7M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 694de6b945cd 2024-09-13T14:30:24.384 INFO:teuthology.orchestra.run.smithi151.stdout:mon.b smithi184 running (3m) 45s ago 16m 42.3M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 12b6c0ccb635 2024-09-13T14:30:24.384 INFO:teuthology.orchestra.run.smithi151.stdout:mon.c smithi151 running (4m) 12s ago 16m 54.6M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 e548fceed2f9 2024-09-13T14:30:24.384 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.a smithi151 *:9100 running (7m) 12s ago 11m 10.6M - 1.7.0 72c9c2088986 d91e4cb09e9a 2024-09-13T14:30:24.384 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter.b smithi184 *:9100 running (7m) 45s ago 11m 11.2M - 1.7.0 72c9c2088986 1e1a9dd643a0 2024-09-13T14:30:24.384 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (117s) 12s ago 15m 78.1M 2480M 19.3.0-4697-gd58f7306 fec07016a2b0 018b463f93c2 2024-09-13T14:30:24.384 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi151 running (84s) 12s ago 15m 57.0M 2480M 19.3.0-4697-gd58f7306 fec07016a2b0 04d393df5f43 2024-09-13T14:30:24.384 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi151 running (50s) 12s ago 14m 51.8M 2480M 19.3.0-4697-gd58f7306 fec07016a2b0 2faaa919b127 2024-09-13T14:30:24.384 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 smithi151 running (16s) 12s ago 14m 12.0M 2480M 19.3.0-4697-gd58f7306 fec07016a2b0 31c786beb443 2024-09-13T14:30:24.384 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 smithi184 running (14m) 45s ago 14m 105M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a 6e70699b8084 2024-09-13T14:30:24.384 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 smithi184 running (13m) 45s ago 13m 107M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a 36b82c9e7230 2024-09-13T14:30:24.384 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6 smithi184 running (13m) 45s ago 13m 85.5M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a f9379cec7504 2024-09-13T14:30:24.385 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7 smithi184 running (12m) 45s ago 12m 94.2M 2480M 17.2.7-1627-gbbdadfa3 a994b00deb3a f37cc947097b 2024-09-13T14:30:24.385 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.a smithi184 *:9095 running (6m) 45s ago 12m 55.8M - 2.51.0 1d3b7f56885b 3da9265b8f14 2024-09-13T14:30:24.537 INFO:tasks.rados.rados.0.smithi151.stdout:2205: done (2 left) 2024-09-13T14:30:24.537 INFO:tasks.rados.rados.0.smithi151.stdout:2206: done (1 left) 2024-09-13T14:30:24.537 INFO:tasks.rados.rados.0.smithi151.stdout:2207: done (0 left) 2024-09-13T14:30:24.537 INFO:tasks.rados.rados.0.smithi151.stdout:2208: snap_create 2024-09-13T14:30:24.724 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:30:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:30:24] "GET /metrics HTTP/1.1" 200 36369 "" "Prometheus/2.51.0" 2024-09-13T14:30:25.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:24 smithi184 ceph-mon[92527]: pgmap v567: 89 pgs: 89 active+clean; 210 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 8.3 MiB/s rd, 2.4 MiB/s wr, 16 op/s 2024-09-13T14:30:25.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:24 smithi184 ceph-mon[92527]: osdmap e633: 8 total, 8 up, 8 in 2024-09-13T14:30:25.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:24 smithi184 ceph-mon[92527]: osdmap e634: 8 total, 8 up, 8 in 2024-09-13T14:30:25.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:24 smithi151 ceph-mon[108966]: pgmap v567: 89 pgs: 89 active+clean; 210 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 8.3 MiB/s rd, 2.4 MiB/s wr, 16 op/s 2024-09-13T14:30:25.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:24 smithi151 ceph-mon[108966]: osdmap e633: 8 total, 8 up, 8 in 2024-09-13T14:30:25.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:24 smithi151 ceph-mon[108966]: osdmap e634: 8 total, 8 up, 8 in 2024-09-13T14:30:25.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:24 smithi151 ceph-mon[104450]: pgmap v567: 89 pgs: 89 active+clean; 210 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 8.3 MiB/s rd, 2.4 MiB/s wr, 16 op/s 2024-09-13T14:30:25.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:24 smithi151 ceph-mon[104450]: osdmap e633: 8 total, 8 up, 8 in 2024-09-13T14:30:25.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:24 smithi151 ceph-mon[104450]: osdmap e634: 8 total, 8 up, 8 in 2024-09-13T14:30:25.345 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph versions' 2024-09-13T14:30:25.538 INFO:tasks.rados.rados.0.smithi151.stdout:2208: done (0 left) 2024-09-13T14:30:25.539 INFO:tasks.rados.rados.0.smithi151.stdout:2209: rollback oid 14 current snap is 289 2024-09-13T14:30:25.539 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 14 to 286 2024-09-13T14:30:25.539 INFO:tasks.rados.rados.0.smithi151.stdout:2210: snap_create 2024-09-13T14:30:25.543 INFO:tasks.rados.rados.0.smithi151.stdout:2209: finishing rollback tid 0 to smithi15186934-14 2024-09-13T14:30:25.544 INFO:tasks.rados.rados.0.smithi151.stdout:2209: finishing rollback tid 1 to smithi15186934-14 2024-09-13T14:30:25.544 INFO:tasks.rados.rados.0.smithi151.stdout:2209: finishing rollback tid 2 to smithi15186934-14 2024-09-13T14:30:25.544 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 900 (ObjNum 819 snap 286 seq_num 819) dirty exists 2024-09-13T14:30:25.597 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:30:25.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:25 smithi151 ceph-mon[104450]: from='client.44936 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:25.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:25 smithi151 ceph-mon[104450]: osdmap e635: 8 total, 8 up, 8 in 2024-09-13T14:30:25.966 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:25 smithi151 ceph-mon[108966]: from='client.44936 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:25.966 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:25 smithi151 ceph-mon[108966]: osdmap e635: 8 total, 8 up, 8 in 2024-09-13T14:30:26.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:25 smithi184 ceph-mon[92527]: from='client.44936 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:26.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:25 smithi184 ceph-mon[92527]: osdmap e635: 8 total, 8 up, 8 in 2024-09-13T14:30:26.544 INFO:tasks.rados.rados.0.smithi151.stdout:2209: done (1 left) 2024-09-13T14:30:26.545 INFO:tasks.rados.rados.0.smithi151.stdout:2210: done (0 left) 2024-09-13T14:30:26.545 INFO:tasks.rados.rados.0.smithi151.stdout:2211: rollback oid 41 current snap is 290 2024-09-13T14:30:26.545 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 41 to 285 2024-09-13T14:30:26.545 INFO:tasks.rados.rados.0.smithi151.stdout:2212: snap_remove snap 280 2024-09-13T14:30:26.947 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:26 smithi151 ceph-mon[108966]: pgmap v570: 89 pgs: 89 active+clean; 207 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 7.5 MiB/s rd, 2.0 MiB/s wr, 13 op/s 2024-09-13T14:30:26.948 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:26 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:26.948 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:26 smithi151 ceph-mon[108966]: osdmap e636: 8 total, 8 up, 8 in 2024-09-13T14:30:26.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:26 smithi151 ceph-mon[104450]: pgmap v570: 89 pgs: 89 active+clean; 207 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 7.5 MiB/s rd, 2.0 MiB/s wr, 13 op/s 2024-09-13T14:30:26.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:26 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:26.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:26 smithi151 ceph-mon[104450]: osdmap e636: 8 total, 8 up, 8 in 2024-09-13T14:30:27.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:26 smithi184 ceph-mon[92527]: pgmap v570: 89 pgs: 89 active+clean; 207 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 7.5 MiB/s rd, 2.0 MiB/s wr, 13 op/s 2024-09-13T14:30:27.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:26 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:27.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:26 smithi184 ceph-mon[92527]: osdmap e636: 8 total, 8 up, 8 in 2024-09-13T14:30:27.418 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:30:27.419 INFO:teuthology.orchestra.run.smithi151.stdout: "mon": { 2024-09-13T14:30:27.419 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 3 2024-09-13T14:30:27.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:30:27.419 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": { 2024-09-13T14:30:27.419 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:30:27.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:30:27.419 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": { 2024-09-13T14:30:27.419 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)": 4, 2024-09-13T14:30:27.419 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 4 2024-09-13T14:30:27.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:30:27.419 INFO:teuthology.orchestra.run.smithi151.stdout: "mds": { 2024-09-13T14:30:27.419 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)": 2 2024-09-13T14:30:27.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:30:27.420 INFO:teuthology.orchestra.run.smithi151.stdout: "overall": { 2024-09-13T14:30:27.420 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 17.2.7-1627-gbbdadfa3 (bbdadfa31f8fb9afaa389358d138cac49bd69e56) quincy (stable)": 6, 2024-09-13T14:30:27.420 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 9 2024-09-13T14:30:27.420 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:30:27.420 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:30:27.547 INFO:tasks.rados.rados.0.smithi151.stdout:2212: done (1 left) 2024-09-13T14:30:27.547 INFO:tasks.rados.rados.0.smithi151.stdout:2213: read oid 21 snap -1 2024-09-13T14:30:27.547 INFO:tasks.rados.rados.0.smithi151.stdout:2213: expect deleted 2024-09-13T14:30:27.547 INFO:tasks.rados.rados.0.smithi151.stdout:2214: write oid 44 current snap is 290 2024-09-13T14:30:27.547 INFO:tasks.rados.rados.0.smithi151.stdout:2214: seq_num 827 ranges {699389=797047,2111079=502953,3094462=581837} 2024-09-13T14:30:27.556 INFO:tasks.rados.rados.0.smithi151.stdout:2214: writing smithi15186934-44 from 699389 to 1496436 tid 1 2024-09-13T14:30:27.562 INFO:tasks.rados.rados.0.smithi151.stdout:2214: writing smithi15186934-44 from 2111079 to 2614032 tid 2 2024-09-13T14:30:27.569 INFO:tasks.rados.rados.0.smithi151.stdout:2214: writing smithi15186934-44 from 3094462 to 3676299 tid 3 2024-09-13T14:30:27.569 INFO:tasks.rados.rados.0.smithi151.stdout:2215: rollback oid 29 current snap is 290 2024-09-13T14:30:27.569 INFO:tasks.rados.rados.0.smithi151.stdout:2211: finishing rollback tid 1 to smithi15186934-41 2024-09-13T14:30:27.569 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 124 (ObjNum 305 snap 0 seq_num 3681081415) dirty dne 2024-09-13T14:30:27.569 INFO:tasks.rados.rados.0.smithi151.stdout:2214: finishing write tid 1 to smithi15186934-44 2024-09-13T14:30:27.569 INFO:tasks.rados.rados.0.smithi151.stdout:2214: finishing write tid 2 to smithi15186934-44 2024-09-13T14:30:27.569 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 29 to 287 2024-09-13T14:30:27.569 INFO:tasks.rados.rados.0.smithi151.stdout:2211: done (3 left) 2024-09-13T14:30:27.570 INFO:tasks.rados.rados.0.smithi151.stdout:2213: done (2 left) 2024-09-13T14:30:27.570 INFO:tasks.rados.rados.0.smithi151.stdout:2216: write oid 43 current snap is 290 2024-09-13T14:30:27.570 INFO:tasks.rados.rados.0.smithi151.stdout:2216: seq_num 828 ranges {716196=472296,1769561=646611,3121885=793109} 2024-09-13T14:30:27.575 INFO:tasks.rados.rados.0.smithi151.stdout:2216: writing smithi15186934-43 from 716196 to 1188492 tid 1 2024-09-13T14:30:27.581 INFO:tasks.rados.rados.0.smithi151.stdout:2216: writing smithi15186934-43 from 1769561 to 2416172 tid 2 2024-09-13T14:30:27.590 INFO:tasks.rados.rados.0.smithi151.stdout:2216: writing smithi15186934-43 from 3121885 to 3914994 tid 3 2024-09-13T14:30:27.590 INFO:tasks.rados.rados.0.smithi151.stdout:2217: read oid 46 snap -1 2024-09-13T14:30:27.590 INFO:tasks.rados.rados.0.smithi151.stdout:2217: expect (ObjNum 786 snap 275 seq_num 786) 2024-09-13T14:30:27.590 INFO:tasks.rados.rados.0.smithi151.stdout:2218: snap_create 2024-09-13T14:30:27.591 INFO:tasks.rados.rados.0.smithi151.stdout:2214: finishing write tid 3 to smithi15186934-44 2024-09-13T14:30:27.591 INFO:tasks.rados.rados.0.smithi151.stdout:2214: finishing write tid 4 to smithi15186934-44 2024-09-13T14:30:27.591 INFO:tasks.rados.rados.0.smithi151.stdout:2215: finishing rollback tid 0 to smithi15186934-29 2024-09-13T14:30:27.591 INFO:tasks.rados.rados.0.smithi151.stdout:2214: finishing write tid 5 to smithi15186934-44 2024-09-13T14:30:27.591 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 691 (ObjNum 827 snap 290 seq_num 827) dirty exists 2024-09-13T14:30:27.591 INFO:tasks.rados.rados.0.smithi151.stdout:2214: left oid 44 (ObjNum 827 snap 290 seq_num 827) 2024-09-13T14:30:27.591 INFO:tasks.rados.rados.0.smithi151.stdout:2215: finishing rollback tid 1 to smithi15186934-29 2024-09-13T14:30:27.591 INFO:tasks.rados.rados.0.smithi151.stdout:2215: finishing rollback tid 2 to smithi15186934-29 2024-09-13T14:30:27.591 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 29 v 571 (ObjNum 814 snap 285 seq_num 814) dirty exists 2024-09-13T14:30:27.591 INFO:tasks.rados.rados.0.smithi151.stdout:2216: finishing write tid 1 to smithi15186934-43 2024-09-13T14:30:27.591 INFO:tasks.rados.rados.0.smithi151.stdout:2216: finishing write tid 2 to smithi15186934-43 2024-09-13T14:30:27.592 INFO:tasks.rados.rados.0.smithi151.stdout:2217: expect (ObjNum 786 snap 275 seq_num 786) 2024-09-13T14:30:27.700 INFO:tasks.rados.rados.0.smithi151.stdout:2216: finishing write tid 3 to smithi15186934-43 2024-09-13T14:30:27.700 INFO:tasks.rados.rados.0.smithi151.stdout:2216: finishing write tid 4 to smithi15186934-43 2024-09-13T14:30:27.700 INFO:tasks.rados.rados.0.smithi151.stdout:2216: finishing write tid 5 to smithi15186934-43 2024-09-13T14:30:27.700 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 43 v 1333 (ObjNum 828 snap 290 seq_num 828) dirty exists 2024-09-13T14:30:27.700 INFO:tasks.rados.rados.0.smithi151.stdout:2216: left oid 43 (ObjNum 828 snap 290 seq_num 828) 2024-09-13T14:30:28.020 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:27 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/4104358772' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:30:28.020 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:27 smithi151 ceph-mon[104450]: osdmap e637: 8 total, 8 up, 8 in 2024-09-13T14:30:28.020 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:27 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/4104358772' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:30:28.020 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:27 smithi151 ceph-mon[108966]: osdmap e637: 8 total, 8 up, 8 in 2024-09-13T14:30:28.044 DEBUG:teuthology.parallel:result is None 2024-09-13T14:30:28.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:27 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/4104358772' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:30:28.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:27 smithi184 ceph-mon[92527]: osdmap e637: 8 total, 8 up, 8 in 2024-09-13T14:30:28.545 INFO:tasks.rados.rados.0.smithi151.stdout:2214: done (4 left) 2024-09-13T14:30:28.545 INFO:tasks.rados.rados.0.smithi151.stdout:2215: done (3 left) 2024-09-13T14:30:28.546 INFO:tasks.rados.rados.0.smithi151.stdout:2216: done (2 left) 2024-09-13T14:30:28.546 INFO:tasks.rados.rados.0.smithi151.stdout:2217: done (1 left) 2024-09-13T14:30:28.546 INFO:tasks.rados.rados.0.smithi151.stdout:2218: done (0 left) 2024-09-13T14:30:28.546 INFO:tasks.rados.rados.0.smithi151.stdout:2219: write oid 44 current snap is 291 2024-09-13T14:30:28.546 INFO:tasks.rados.rados.0.smithi151.stdout:2219: seq_num 829 ranges {733003=663898,1944396=506622,2865661=604381} 2024-09-13T14:30:28.552 INFO:tasks.rados.rados.0.smithi151.stdout:2219: writing smithi15186934-44 from 733003 to 1396901 tid 1 2024-09-13T14:30:28.558 INFO:tasks.rados.rados.0.smithi151.stdout:2219: writing smithi15186934-44 from 1944396 to 2451018 tid 2 2024-09-13T14:30:28.564 INFO:tasks.rados.rados.0.smithi151.stdout:2219: writing smithi15186934-44 from 2865661 to 3470042 tid 3 2024-09-13T14:30:28.565 INFO:tasks.rados.rados.0.smithi151.stdout:2220: delete oid 13 current snap is 291 2024-09-13T14:30:28.565 INFO:tasks.rados.rados.0.smithi151.stdout:2219: finishing write tid 1 to smithi15186934-44 2024-09-13T14:30:28.565 INFO:tasks.rados.rados.0.smithi151.stdout:2219: finishing write tid 2 to smithi15186934-44 2024-09-13T14:30:28.567 INFO:tasks.rados.rados.0.smithi151.stdout:2219: finishing write tid 3 to smithi15186934-44 2024-09-13T14:30:28.567 INFO:tasks.rados.rados.0.smithi151.stdout:2219: finishing write tid 4 to smithi15186934-44 2024-09-13T14:30:28.567 INFO:tasks.rados.rados.0.smithi151.stdout:2219: finishing write tid 5 to smithi15186934-44 2024-09-13T14:30:28.568 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 696 (ObjNum 829 snap 291 seq_num 829) dirty exists 2024-09-13T14:30:28.568 INFO:tasks.rados.rados.0.smithi151.stdout:2219: left oid 44 (ObjNum 829 snap 291 seq_num 829) 2024-09-13T14:30:28.569 INFO:tasks.rados.rados.0.smithi151.stdout:2219: done (1 left) 2024-09-13T14:30:28.569 INFO:tasks.rados.rados.0.smithi151.stdout:2220: done (0 left) 2024-09-13T14:30:28.570 INFO:tasks.rados.rados.0.smithi151.stdout:2221: read oid 47 snap -1 2024-09-13T14:30:28.570 INFO:tasks.rados.rados.0.smithi151.stdout:2221: expect deleted 2024-09-13T14:30:28.570 INFO:tasks.rados.rados.0.smithi151.stdout:2222: read oid 37 snap -1 2024-09-13T14:30:28.570 INFO:tasks.rados.rados.0.smithi151.stdout:2222: expect (ObjNum 683 snap 222 seq_num 683) 2024-09-13T14:30:28.570 INFO:tasks.rados.rados.0.smithi151.stdout:2223: snap_remove snap 283 2024-09-13T14:30:29.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:28 smithi184 ceph-mon[92527]: pgmap v573: 89 pgs: 89 active+clean; 195 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 804 KiB/s wr, 11 op/s 2024-09-13T14:30:29.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:28 smithi184 ceph-mon[92527]: osdmap e638: 8 total, 8 up, 8 in 2024-09-13T14:30:29.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:28 smithi151 ceph-mon[104450]: pgmap v573: 89 pgs: 89 active+clean; 195 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 804 KiB/s wr, 11 op/s 2024-09-13T14:30:29.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:28 smithi151 ceph-mon[104450]: osdmap e638: 8 total, 8 up, 8 in 2024-09-13T14:30:29.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:28 smithi151 ceph-mon[108966]: pgmap v573: 89 pgs: 89 active+clean; 195 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 804 KiB/s wr, 11 op/s 2024-09-13T14:30:29.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:28 smithi151 ceph-mon[108966]: osdmap e638: 8 total, 8 up, 8 in 2024-09-13T14:30:29.540 INFO:tasks.rados.rados.0.smithi151.stdout:2223: done (2 left) 2024-09-13T14:30:29.540 INFO:tasks.rados.rados.0.smithi151.stdout:2224: snap_remove snap 250 2024-09-13T14:30:30.544 INFO:tasks.rados.rados.0.smithi151.stdout:2224: done (2 left) 2024-09-13T14:30:30.544 INFO:tasks.rados.rados.0.smithi151.stdout:2225: write oid 34 current snap is 291 2024-09-13T14:30:30.545 INFO:tasks.rados.rados.0.smithi151.stdout:2225: seq_num 831 ranges {766617=414396,1945007=510291,3204154=626925} 2024-09-13T14:30:30.547 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:30:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:30:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:30:30.551 INFO:tasks.rados.rados.0.smithi151.stdout:2225: writing smithi15186934-34 from 766617 to 1181013 tid 1 2024-09-13T14:30:30.559 INFO:tasks.rados.rados.0.smithi151.stdout:2225: writing smithi15186934-34 from 1945007 to 2455298 tid 2 2024-09-13T14:30:30.567 INFO:tasks.rados.rados.0.smithi151.stdout:2225: writing smithi15186934-34 from 3204154 to 3831079 tid 3 2024-09-13T14:30:30.567 INFO:tasks.rados.rados.0.smithi151.stdout:2226: snap_remove snap 281 2024-09-13T14:30:30.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:30 smithi184 ceph-mon[92527]: pgmap v576: 89 pgs: 89 active+clean; 195 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 7.7 MiB/s rd, 397 KiB/s wr, 8 op/s 2024-09-13T14:30:30.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:30 smithi184 ceph-mon[92527]: osdmap e639: 8 total, 8 up, 8 in 2024-09-13T14:30:30.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:30 smithi151 ceph-mon[104450]: pgmap v576: 89 pgs: 89 active+clean; 195 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 7.7 MiB/s rd, 397 KiB/s wr, 8 op/s 2024-09-13T14:30:30.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:30 smithi151 ceph-mon[104450]: osdmap e639: 8 total, 8 up, 8 in 2024-09-13T14:30:30.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:30 smithi151 ceph-mon[108966]: pgmap v576: 89 pgs: 89 active+clean; 195 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 7.7 MiB/s rd, 397 KiB/s wr, 8 op/s 2024-09-13T14:30:30.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:30 smithi151 ceph-mon[108966]: osdmap e639: 8 total, 8 up, 8 in 2024-09-13T14:30:31.557 INFO:tasks.rados.rados.0.smithi151.stdout:2226: done (3 left) 2024-09-13T14:30:31.557 INFO:tasks.rados.rados.0.smithi151.stdout:2227: write oid 27 current snap is 291 2024-09-13T14:30:31.557 INFO:tasks.rados.rados.0.smithi151.stdout:2222: expect (ObjNum 683 snap 222 seq_num 683) 2024-09-13T14:30:31.780 INFO:tasks.rados.rados.0.smithi151.stdout:2225: finishing write tid 1 to smithi15186934-34 2024-09-13T14:30:31.781 INFO:tasks.rados.rados.0.smithi151.stdout:2225: finishing write tid 2 to smithi15186934-34 2024-09-13T14:30:31.781 INFO:tasks.rados.rados.0.smithi151.stdout:2225: finishing write tid 3 to smithi15186934-34 2024-09-13T14:30:31.781 INFO:tasks.rados.rados.0.smithi151.stdout:2225: finishing write tid 4 to smithi15186934-34 2024-09-13T14:30:31.781 INFO:tasks.rados.rados.0.smithi151.stdout:2225: finishing write tid 5 to smithi15186934-34 2024-09-13T14:30:31.781 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 829 (ObjNum 831 snap 291 seq_num 831) dirty exists 2024-09-13T14:30:31.781 INFO:tasks.rados.rados.0.smithi151.stdout:2225: left oid 34 (ObjNum 831 snap 291 seq_num 831) 2024-09-13T14:30:31.781 INFO:tasks.rados.rados.0.smithi151.stdout:2227: seq_num 832 ranges {783424=489645,2003489=653949,3231577=554550} 2024-09-13T14:30:31.787 INFO:tasks.rados.rados.0.smithi151.stdout:2227: writing smithi15186934-27 from 783424 to 1273069 tid 1 2024-09-13T14:30:31.794 INFO:tasks.rados.rados.0.smithi151.stdout:2227: writing smithi15186934-27 from 2003489 to 2657438 tid 2 2024-09-13T14:30:31.801 INFO:tasks.rados.rados.0.smithi151.stdout:2227: writing smithi15186934-27 from 3231577 to 3786127 tid 3 2024-09-13T14:30:31.801 INFO:tasks.rados.rados.0.smithi151.stdout:2221: done (3 left) 2024-09-13T14:30:31.801 INFO:tasks.rados.rados.0.smithi151.stdout:2222: done (2 left) 2024-09-13T14:30:31.801 INFO:tasks.rados.rados.0.smithi151.stdout:2225: done (1 left) 2024-09-13T14:30:31.801 INFO:tasks.rados.rados.0.smithi151.stdout:2228: delete oid 29 current snap is 291 2024-09-13T14:30:31.802 INFO:tasks.rados.rados.0.smithi151.stdout:2227: finishing write tid 1 to smithi15186934-27 2024-09-13T14:30:31.802 INFO:tasks.rados.rados.0.smithi151.stdout:2227: finishing write tid 2 to smithi15186934-27 2024-09-13T14:30:31.803 INFO:tasks.rados.rados.0.smithi151.stdout:2228: done (1 left) 2024-09-13T14:30:31.803 INFO:tasks.rados.rados.0.smithi151.stdout:2229: read oid 22 snap -1 2024-09-13T14:30:31.803 INFO:tasks.rados.rados.0.smithi151.stdout:2229: expect deleted 2024-09-13T14:30:31.804 INFO:tasks.rados.rados.0.smithi151.stdout:2230: write oid 8 current snap is 291 2024-09-13T14:30:31.804 INFO:tasks.rados.rados.0.smithi151.stdout:2230: seq_num 834 ranges {417038=640143,1720453=296585} 2024-09-13T14:30:31.809 INFO:tasks.rados.rados.0.smithi151.stdout:2230: writing smithi15186934-8 from 417038 to 1057181 tid 1 2024-09-13T14:30:31.814 INFO:tasks.rados.rados.0.smithi151.stdout:2230: writing smithi15186934-8 from 1720453 to 2017038 tid 2 2024-09-13T14:30:31.815 INFO:tasks.rados.rados.0.smithi151.stdout:2231: snap_create 2024-09-13T14:30:31.815 INFO:tasks.rados.rados.0.smithi151.stdout:2227: finishing write tid 3 to smithi15186934-27 2024-09-13T14:30:31.815 INFO:tasks.rados.rados.0.smithi151.stdout:2227: finishing write tid 4 to smithi15186934-27 2024-09-13T14:30:31.815 INFO:tasks.rados.rados.0.smithi151.stdout:2227: finishing write tid 5 to smithi15186934-27 2024-09-13T14:30:31.815 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 27 v 523 (ObjNum 832 snap 291 seq_num 832) dirty exists 2024-09-13T14:30:31.815 INFO:tasks.rados.rados.0.smithi151.stdout:2227: left oid 27 (ObjNum 832 snap 291 seq_num 832) 2024-09-13T14:30:31.815 INFO:tasks.rados.rados.0.smithi151.stdout:2230: finishing write tid 1 to smithi15186934-8 2024-09-13T14:30:31.817 INFO:tasks.rados.rados.0.smithi151.stdout:2230: finishing write tid 2 to smithi15186934-8 2024-09-13T14:30:31.817 INFO:tasks.rados.rados.0.smithi151.stdout:2230: finishing write tid 3 to smithi15186934-8 2024-09-13T14:30:31.817 INFO:tasks.rados.rados.0.smithi151.stdout:2230: finishing write tid 4 to smithi15186934-8 2024-09-13T14:30:31.817 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 703 (ObjNum 834 snap 291 seq_num 834) dirty exists 2024-09-13T14:30:31.817 INFO:tasks.rados.rados.0.smithi151.stdout:2230: left oid 8 (ObjNum 834 snap 291 seq_num 834) 2024-09-13T14:30:31.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:31 smithi184 ceph-mon[92527]: osdmap e640: 8 total, 8 up, 8 in 2024-09-13T14:30:31.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:31.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:31.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:31 smithi151 ceph-mon[108966]: osdmap e640: 8 total, 8 up, 8 in 2024-09-13T14:30:31.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:31.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:31.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:31 smithi151 ceph-mon[104450]: osdmap e640: 8 total, 8 up, 8 in 2024-09-13T14:30:31.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:31.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:32.563 INFO:tasks.rados.rados.0.smithi151.stdout:2227: done (3 left) 2024-09-13T14:30:32.564 INFO:tasks.rados.rados.0.smithi151.stdout:2229: done (2 left) 2024-09-13T14:30:32.564 INFO:tasks.rados.rados.0.smithi151.stdout:2230: done (1 left) 2024-09-13T14:30:32.564 INFO:tasks.rados.rados.0.smithi151.stdout:2231: done (0 left) 2024-09-13T14:30:32.564 INFO:tasks.rados.rados.0.smithi151.stdout:2232: write oid 12 current snap is 292 2024-09-13T14:30:32.564 INFO:tasks.rados.rados.0.smithi151.stdout:2232: seq_num 835 ranges {433845=715392,1662582=371263} 2024-09-13T14:30:32.570 INFO:tasks.rados.rados.0.smithi151.stdout:2232: writing smithi15186934-12 from 433845 to 1149237 tid 1 2024-09-13T14:30:32.575 INFO:tasks.rados.rados.0.smithi151.stdout:2232: writing smithi15186934-12 from 1662582 to 2033845 tid 2 2024-09-13T14:30:32.575 INFO:tasks.rados.rados.0.smithi151.stdout:2233: write oid 38 current snap is 292 2024-09-13T14:30:32.575 INFO:tasks.rados.rados.0.smithi151.stdout:2233: seq_num 836 ranges {450652=790641,1721064=329588} 2024-09-13T14:30:32.582 INFO:tasks.rados.rados.0.smithi151.stdout:2233: writing smithi15186934-38 from 450652 to 1241293 tid 1 2024-09-13T14:30:32.587 INFO:tasks.rados.rados.0.smithi151.stdout:2233: writing smithi15186934-38 from 1721064 to 2050652 tid 2 2024-09-13T14:30:32.587 INFO:tasks.rados.rados.0.smithi151.stdout:2234: delete oid 39 current snap is 292 2024-09-13T14:30:32.588 INFO:tasks.rados.rados.0.smithi151.stdout:2232: finishing write tid 1 to smithi15186934-12 2024-09-13T14:30:32.588 INFO:tasks.rados.rados.0.smithi151.stdout:2232: finishing write tid 2 to smithi15186934-12 2024-09-13T14:30:32.588 INFO:tasks.rados.rados.0.smithi151.stdout:2232: finishing write tid 3 to smithi15186934-12 2024-09-13T14:30:32.588 INFO:tasks.rados.rados.0.smithi151.stdout:2232: finishing write tid 4 to smithi15186934-12 2024-09-13T14:30:32.588 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 12 v 529 (ObjNum 835 snap 292 seq_num 835) dirty exists 2024-09-13T14:30:32.589 INFO:tasks.rados.rados.0.smithi151.stdout:2232: left oid 12 (ObjNum 835 snap 292 seq_num 835) 2024-09-13T14:30:32.590 INFO:tasks.rados.rados.0.smithi151.stdout:2233: finishing write tid 1 to smithi15186934-38 2024-09-13T14:30:32.591 INFO:tasks.rados.rados.0.smithi151.stdout:2233: finishing write tid 2 to smithi15186934-38 2024-09-13T14:30:32.591 INFO:tasks.rados.rados.0.smithi151.stdout:2233: finishing write tid 3 to smithi15186934-38 2024-09-13T14:30:32.591 INFO:tasks.rados.rados.0.smithi151.stdout:2233: finishing write tid 4 to smithi15186934-38 2024-09-13T14:30:32.591 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 803 (ObjNum 836 snap 292 seq_num 836) dirty exists 2024-09-13T14:30:32.591 INFO:tasks.rados.rados.0.smithi151.stdout:2233: left oid 38 (ObjNum 836 snap 292 seq_num 836) 2024-09-13T14:30:32.591 INFO:tasks.rados.rados.0.smithi151.stdout:2232: done (2 left) 2024-09-13T14:30:32.591 INFO:tasks.rados.rados.0.smithi151.stdout:2233: done (1 left) 2024-09-13T14:30:32.591 INFO:tasks.rados.rados.0.smithi151.stdout:2234: done (0 left) 2024-09-13T14:30:32.591 INFO:tasks.rados.rados.0.smithi151.stdout:2235: write oid 42 current snap is 292 2024-09-13T14:30:32.591 INFO:tasks.rados.rados.0.smithi151.stdout:2235: seq_num 838 ranges {484266=657492,1554381=529885} 2024-09-13T14:30:32.596 INFO:tasks.rados.rados.0.smithi151.stdout:2235: writing smithi15186934-42 from 484266 to 1141758 tid 1 2024-09-13T14:30:32.602 INFO:tasks.rados.rados.0.smithi151.stdout:2235: writing smithi15186934-42 from 1554381 to 2084266 tid 2 2024-09-13T14:30:32.602 INFO:tasks.rados.rados.0.smithi151.stdout:2236: snap_create 2024-09-13T14:30:32.602 INFO:tasks.rados.rados.0.smithi151.stdout:2235: finishing write tid 1 to smithi15186934-42 2024-09-13T14:30:32.604 INFO:tasks.rados.rados.0.smithi151.stdout:2235: finishing write tid 2 to smithi15186934-42 2024-09-13T14:30:32.604 INFO:tasks.rados.rados.0.smithi151.stdout:2235: finishing write tid 3 to smithi15186934-42 2024-09-13T14:30:32.605 INFO:tasks.rados.rados.0.smithi151.stdout:2235: finishing write tid 4 to smithi15186934-42 2024-09-13T14:30:32.605 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 942 (ObjNum 838 snap 292 seq_num 838) dirty exists 2024-09-13T14:30:32.605 INFO:tasks.rados.rados.0.smithi151.stdout:2235: left oid 42 (ObjNum 838 snap 292 seq_num 838) 2024-09-13T14:30:32.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:32 smithi184 ceph-mon[92527]: pgmap v579: 89 pgs: 89 active+clean; 201 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1020 KiB/s rd, 893 KiB/s wr, 5 op/s 2024-09-13T14:30:32.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:32 smithi184 ceph-mon[92527]: osdmap e641: 8 total, 8 up, 8 in 2024-09-13T14:30:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:30:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:30:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:30:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:30:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:32.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:32 smithi151 ceph-mon[104450]: pgmap v579: 89 pgs: 89 active+clean; 201 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1020 KiB/s rd, 893 KiB/s wr, 5 op/s 2024-09-13T14:30:32.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:32 smithi151 ceph-mon[104450]: osdmap e641: 8 total, 8 up, 8 in 2024-09-13T14:30:32.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:30:32.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:32.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:30:32.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:32.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:30:32.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:32.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:30:32.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:32.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:32 smithi151 ceph-mon[108966]: pgmap v579: 89 pgs: 89 active+clean; 201 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1020 KiB/s rd, 893 KiB/s wr, 5 op/s 2024-09-13T14:30:32.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:32 smithi151 ceph-mon[108966]: osdmap e641: 8 total, 8 up, 8 in 2024-09-13T14:30:32.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:30:32.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:32.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:30:32.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:32.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:30:32.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:32.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:30:32.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:30:33.572 INFO:tasks.rados.rados.0.smithi151.stdout:2235: done (1 left) 2024-09-13T14:30:33.573 INFO:tasks.rados.rados.0.smithi151.stdout:2236: done (0 left) 2024-09-13T14:30:33.573 INFO:tasks.rados.rados.0.smithi151.stdout:2237: read oid 46 snap -1 2024-09-13T14:30:33.573 INFO:tasks.rados.rados.0.smithi151.stdout:2237: expect (ObjNum 786 snap 275 seq_num 786) 2024-09-13T14:30:33.573 INFO:tasks.rados.rados.0.smithi151.stdout:2238: write oid 5 current snap is 293 2024-09-13T14:30:33.573 INFO:tasks.rados.rados.0.smithi151.stdout:2238: seq_num 839 ranges {501073=732741,1896510=204563} 2024-09-13T14:30:33.578 INFO:tasks.rados.rados.0.smithi151.stdout:2238: writing smithi15186934-5 from 501073 to 1233814 tid 1 2024-09-13T14:30:33.583 INFO:tasks.rados.rados.0.smithi151.stdout:2238: writing smithi15186934-5 from 1896510 to 2101073 tid 2 2024-09-13T14:30:33.583 INFO:tasks.rados.rados.0.smithi151.stdout:2239: write oid 37 current snap is 293 2024-09-13T14:30:33.583 INFO:tasks.rados.rados.0.smithi151.stdout:2239: seq_num 840 ranges {517880=407990,1554992=562888} 2024-09-13T14:30:33.588 INFO:tasks.rados.rados.0.smithi151.stdout:2239: writing smithi15186934-37 from 517880 to 925870 tid 1 2024-09-13T14:30:33.595 INFO:tasks.rados.rados.0.smithi151.stdout:2239: writing smithi15186934-37 from 1554992 to 2117880 tid 2 2024-09-13T14:30:33.595 INFO:tasks.rados.rados.0.smithi151.stdout:2240: delete oid 16 current snap is 293 2024-09-13T14:30:33.595 INFO:tasks.rados.rados.0.smithi151.stdout:2237: expect (ObjNum 786 snap 275 seq_num 786) 2024-09-13T14:30:33.702 INFO:tasks.rados.rados.0.smithi151.stdout:2238: finishing write tid 1 to smithi15186934-5 2024-09-13T14:30:33.702 INFO:tasks.rados.rados.0.smithi151.stdout:2238: finishing write tid 2 to smithi15186934-5 2024-09-13T14:30:33.702 INFO:tasks.rados.rados.0.smithi151.stdout:2238: finishing write tid 3 to smithi15186934-5 2024-09-13T14:30:33.702 INFO:tasks.rados.rados.0.smithi151.stdout:2238: finishing write tid 4 to smithi15186934-5 2024-09-13T14:30:33.702 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 890 (ObjNum 839 snap 293 seq_num 839) dirty exists 2024-09-13T14:30:33.703 INFO:tasks.rados.rados.0.smithi151.stdout:2238: left oid 5 (ObjNum 839 snap 293 seq_num 839) 2024-09-13T14:30:33.703 INFO:tasks.rados.rados.0.smithi151.stdout:2239: finishing write tid 1 to smithi15186934-37 2024-09-13T14:30:33.703 INFO:tasks.rados.rados.0.smithi151.stdout:2239: finishing write tid 2 to smithi15186934-37 2024-09-13T14:30:33.703 INFO:tasks.rados.rados.0.smithi151.stdout:2239: finishing write tid 3 to smithi15186934-37 2024-09-13T14:30:33.703 INFO:tasks.rados.rados.0.smithi151.stdout:2237: done (3 left) 2024-09-13T14:30:33.703 INFO:tasks.rados.rados.0.smithi151.stdout:2238: done (2 left) 2024-09-13T14:30:33.703 INFO:tasks.rados.rados.0.smithi151.stdout:2240: done (1 left) 2024-09-13T14:30:33.703 INFO:tasks.rados.rados.0.smithi151.stdout:2241: read oid 48 snap 255 2024-09-13T14:30:33.703 INFO:tasks.rados.rados.0.smithi151.stdout:2241: expect (ObjNum 727 snap 249 seq_num 727) 2024-09-13T14:30:33.703 INFO:tasks.rados.rados.0.smithi151.stdout:2242: snap_remove snap 260 2024-09-13T14:30:33.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:33 smithi184 ceph-mon[92527]: osdmap e642: 8 total, 8 up, 8 in 2024-09-13T14:30:33.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:33 smithi151 ceph-mon[104450]: osdmap e642: 8 total, 8 up, 8 in 2024-09-13T14:30:33.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:33 smithi151 ceph-mon[108966]: osdmap e642: 8 total, 8 up, 8 in 2024-09-13T14:30:34.542 INFO:tasks.rados.rados.0.smithi151.stdout:2242: done (2 left) 2024-09-13T14:30:34.542 INFO:tasks.rados.rados.0.smithi151.stdout:2243: delete oid 49 current snap is 293 2024-09-13T14:30:34.542 INFO:tasks.rados.rados.0.smithi151.stdout:2239: finishing write tid 4 to smithi15186934-37 2024-09-13T14:30:34.542 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 37 v 1023 (ObjNum 840 snap 293 seq_num 840) dirty exists 2024-09-13T14:30:34.542 INFO:tasks.rados.rados.0.smithi151.stdout:2239: left oid 37 (ObjNum 840 snap 293 seq_num 840) 2024-09-13T14:30:34.542 INFO:tasks.rados.rados.0.smithi151.stdout:2241: expect (ObjNum 727 snap 249 seq_num 727) 2024-09-13T14:30:34.639 INFO:tasks.rados.rados.0.smithi151.stdout:2239: done (2 left) 2024-09-13T14:30:34.639 INFO:tasks.rados.rados.0.smithi151.stdout:2241: done (1 left) 2024-09-13T14:30:34.639 INFO:tasks.rados.rados.0.smithi151.stdout:2243: done (0 left) 2024-09-13T14:30:34.640 INFO:tasks.rados.rados.0.smithi151.stdout:2244: rollback oid 10 current snap is 293 2024-09-13T14:30:34.640 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 10 to 288 2024-09-13T14:30:34.640 INFO:tasks.rados.rados.0.smithi151.stdout:2245: read oid 16 snap 255 2024-09-13T14:30:34.640 INFO:tasks.rados.rados.0.smithi151.stdout:2245: expect (ObjNum 685 snap 223 seq_num 685) 2024-09-13T14:30:34.640 INFO:tasks.rados.rados.0.smithi151.stdout:2246: snap_remove snap 292 2024-09-13T14:30:34.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:34 smithi184 ceph-mon[92527]: pgmap v582: 89 pgs: 89 active+clean; 206 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 8 op/s 2024-09-13T14:30:34.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:34 smithi184 ceph-mon[92527]: osdmap e643: 8 total, 8 up, 8 in 2024-09-13T14:30:34.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:34 smithi184 ceph-mon[92527]: osdmap e644: 8 total, 8 up, 8 in 2024-09-13T14:30:34.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:34 smithi151 ceph-mon[104450]: pgmap v582: 89 pgs: 89 active+clean; 206 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 8 op/s 2024-09-13T14:30:34.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:34 smithi151 ceph-mon[104450]: osdmap e643: 8 total, 8 up, 8 in 2024-09-13T14:30:34.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:34 smithi151 ceph-mon[104450]: osdmap e644: 8 total, 8 up, 8 in 2024-09-13T14:30:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:34 smithi151 ceph-mon[108966]: pgmap v582: 89 pgs: 89 active+clean; 206 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 8 op/s 2024-09-13T14:30:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:34 smithi151 ceph-mon[108966]: osdmap e643: 8 total, 8 up, 8 in 2024-09-13T14:30:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:34 smithi151 ceph-mon[108966]: osdmap e644: 8 total, 8 up, 8 in 2024-09-13T14:30:34.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:30:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:30:34] "GET /metrics HTTP/1.1" 200 36505 "" "Prometheus/2.51.0" 2024-09-13T14:30:35.582 INFO:tasks.rados.rados.0.smithi151.stdout:2246: done (2 left) 2024-09-13T14:30:35.582 INFO:tasks.rados.rados.0.smithi151.stdout:2247: delete oid 33 current snap is 293 2024-09-13T14:30:35.582 INFO:tasks.rados.rados.0.smithi151.stdout:2244: finishing rollback tid 1 to smithi15186934-10 2024-09-13T14:30:35.582 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 767 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-13T14:30:35.583 INFO:tasks.rados.rados.0.smithi151.stdout:2245: expect (ObjNum 685 snap 223 seq_num 685) 2024-09-13T14:30:35.793 INFO:tasks.rados.rados.0.smithi151.stdout:2244: done (2 left) 2024-09-13T14:30:35.794 INFO:tasks.rados.rados.0.smithi151.stdout:2245: done (1 left) 2024-09-13T14:30:35.794 INFO:tasks.rados.rados.0.smithi151.stdout:2247: done (0 left) 2024-09-13T14:30:35.794 INFO:tasks.rados.rados.0.smithi151.stdout:2248: write oid 39 current snap is 293 2024-09-13T14:30:35.794 INFO:tasks.rados.rados.0.smithi151.stdout:2248: seq_num 844 ranges {585108=425339,1788920=396188} 2024-09-13T14:30:35.799 INFO:tasks.rados.rados.0.smithi151.stdout:2248: writing smithi15186934-39 from 585108 to 1010447 tid 1 2024-09-13T14:30:35.806 INFO:tasks.rados.rados.0.smithi151.stdout:2248: writing smithi15186934-39 from 1788920 to 2185108 tid 2 2024-09-13T14:30:35.806 INFO:tasks.rados.rados.0.smithi151.stdout:2249: rollback oid 17 current snap is 293 2024-09-13T14:30:35.806 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 17 to 291 2024-09-13T14:30:35.806 INFO:tasks.rados.rados.0.smithi151.stdout:2248: finishing write tid 1 to smithi15186934-39 2024-09-13T14:30:35.806 INFO:tasks.rados.rados.0.smithi151.stdout:2250: snap_remove snap 285 2024-09-13T14:30:36.584 INFO:tasks.rados.rados.0.smithi151.stdout:2250: done (2 left) 2024-09-13T14:30:36.585 INFO:tasks.rados.rados.0.smithi151.stdout:2251: read oid 2 snap -1 2024-09-13T14:30:36.585 INFO:tasks.rados.rados.0.smithi151.stdout:2251: expect deleted 2024-09-13T14:30:36.585 INFO:tasks.rados.rados.0.smithi151.stdout:2252: write oid 9 current snap is 293 2024-09-13T14:30:36.585 INFO:tasks.rados.rados.0.smithi151.stdout:2252: seq_num 845 ranges {601915=500588,1847402=354513} 2024-09-13T14:30:36.591 INFO:tasks.rados.rados.0.smithi151.stdout:2252: writing smithi15186934-9 from 601915 to 1102503 tid 1 2024-09-13T14:30:36.598 INFO:tasks.rados.rados.0.smithi151.stdout:2252: writing smithi15186934-9 from 1847402 to 2201915 tid 2 2024-09-13T14:30:36.598 INFO:tasks.rados.rados.0.smithi151.stdout:2253: write oid 35 current snap is 293 2024-09-13T14:30:36.598 INFO:tasks.rados.rados.0.smithi151.stdout:2249: finishing rollback tid 1 to smithi15186934-17 2024-09-13T14:30:36.599 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 1309 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-13T14:30:36.599 INFO:tasks.rados.rados.0.smithi151.stdout:2248: finishing write tid 2 to smithi15186934-39 2024-09-13T14:30:36.599 INFO:tasks.rados.rados.0.smithi151.stdout:2248: finishing write tid 3 to smithi15186934-39 2024-09-13T14:30:36.599 INFO:tasks.rados.rados.0.smithi151.stdout:2248: finishing write tid 4 to smithi15186934-39 2024-09-13T14:30:36.599 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 814 (ObjNum 844 snap 293 seq_num 844) dirty exists 2024-09-13T14:30:36.599 INFO:tasks.rados.rados.0.smithi151.stdout:2248: left oid 39 (ObjNum 844 snap 293 seq_num 844) 2024-09-13T14:30:36.599 INFO:tasks.rados.rados.0.smithi151.stdout:2252: finishing write tid 1 to smithi15186934-9 2024-09-13T14:30:36.599 INFO:tasks.rados.rados.0.smithi151.stdout:2253: seq_num 846 ranges {618722=575837,1905884=312838} 2024-09-13T14:30:36.604 INFO:tasks.rados.rados.0.smithi151.stdout:2253: writing smithi15186934-35 from 618722 to 1194559 tid 1 2024-09-13T14:30:36.611 INFO:tasks.rados.rados.0.smithi151.stdout:2253: writing smithi15186934-35 from 1905884 to 2218722 tid 2 2024-09-13T14:30:36.611 INFO:tasks.rados.rados.0.smithi151.stdout:2248: done (4 left) 2024-09-13T14:30:36.611 INFO:tasks.rados.rados.0.smithi151.stdout:2249: done (3 left) 2024-09-13T14:30:36.611 INFO:tasks.rados.rados.0.smithi151.stdout:2251: done (2 left) 2024-09-13T14:30:36.611 INFO:tasks.rados.rados.0.smithi151.stdout:2254: read oid 2 snap 288 2024-09-13T14:30:36.611 INFO:tasks.rados.rados.0.smithi151.stdout:2254: expect deleted 2024-09-13T14:30:36.612 INFO:tasks.rados.rados.0.smithi151.stdout:2255: rollback oid 14 current snap is 293 2024-09-13T14:30:36.612 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 14 to 291 2024-09-13T14:30:36.612 INFO:tasks.rados.rados.0.smithi151.stdout:2252: finishing write tid 2 to smithi15186934-9 2024-09-13T14:30:36.612 INFO:tasks.rados.rados.0.smithi151.stdout:2252: finishing write tid 3 to smithi15186934-9 2024-09-13T14:30:36.612 INFO:tasks.rados.rados.0.smithi151.stdout:2252: finishing write tid 4 to smithi15186934-9 2024-09-13T14:30:36.612 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 931 (ObjNum 845 snap 293 seq_num 845) dirty exists 2024-09-13T14:30:36.612 INFO:tasks.rados.rados.0.smithi151.stdout:2252: left oid 9 (ObjNum 845 snap 293 seq_num 845) 2024-09-13T14:30:36.612 INFO:tasks.rados.rados.0.smithi151.stdout:2253: finishing write tid 1 to smithi15186934-35 2024-09-13T14:30:36.612 INFO:tasks.rados.rados.0.smithi151.stdout:2252: done (3 left) 2024-09-13T14:30:36.612 INFO:tasks.rados.rados.0.smithi151.stdout:2256: read oid 28 snap -1 2024-09-13T14:30:36.613 INFO:tasks.rados.rados.0.smithi151.stdout:2256: expect deleted 2024-09-13T14:30:36.613 INFO:tasks.rados.rados.0.smithi151.stdout:2257: read oid 37 snap -1 2024-09-13T14:30:36.613 INFO:tasks.rados.rados.0.smithi151.stdout:2257: expect (ObjNum 840 snap 293 seq_num 840) 2024-09-13T14:30:36.613 INFO:tasks.rados.rados.0.smithi151.stdout:2258: read oid 27 snap 276 2024-09-13T14:30:36.613 INFO:tasks.rados.rados.0.smithi151.stdout:2258: expect deleted 2024-09-13T14:30:36.613 INFO:tasks.rados.rados.0.smithi151.stdout:2259: rollback oid 9 current snap is 293 2024-09-13T14:30:36.613 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 9 to 286 2024-09-13T14:30:36.613 INFO:tasks.rados.rados.0.smithi151.stdout:2260: write oid 50 current snap is 293 2024-09-13T14:30:36.614 INFO:tasks.rados.rados.0.smithi151.stdout:2260: seq_num 847 ranges {635529=651086,1848013=387516} 2024-09-13T14:30:36.617 INFO:tasks.rados.rados.0.smithi151.stdout:2260: writing smithi15186934-50 from 635529 to 1286615 tid 1 2024-09-13T14:30:36.624 INFO:tasks.rados.rados.0.smithi151.stdout:2260: writing smithi15186934-50 from 1848013 to 2235529 tid 2 2024-09-13T14:30:36.624 INFO:tasks.rados.rados.0.smithi151.stdout:2261: read oid 29 snap 288 2024-09-13T14:30:36.624 INFO:tasks.rados.rados.0.smithi151.stdout:2261: expect (ObjNum 814 snap 285 seq_num 814) 2024-09-13T14:30:36.624 INFO:tasks.rados.rados.0.smithi151.stdout:2262: read oid 17 snap -1 2024-09-13T14:30:36.624 INFO:tasks.rados.rados.0.smithi151.stdout:2262: expect deleted 2024-09-13T14:30:36.624 INFO:tasks.rados.rados.0.smithi151.stdout:2263: delete oid 33 current snap is 293 2024-09-13T14:30:36.624 INFO:tasks.rados.rados.0.smithi151.stdout:2253: finishing write tid 2 to smithi15186934-35 2024-09-13T14:30:36.624 INFO:tasks.rados.rados.0.smithi151.stdout:2253: finishing write tid 3 to smithi15186934-35 2024-09-13T14:30:36.625 INFO:tasks.rados.rados.0.smithi151.stdout:2255: finishing rollback tid 0 to smithi15186934-14 2024-09-13T14:30:36.625 INFO:tasks.rados.rados.0.smithi151.stdout:2255: finishing rollback tid 1 to smithi15186934-14 2024-09-13T14:30:36.625 INFO:tasks.rados.rados.0.smithi151.stdout:2253: finishing write tid 4 to smithi15186934-35 2024-09-13T14:30:36.625 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 35 v 1341 (ObjNum 846 snap 293 seq_num 846) dirty exists 2024-09-13T14:30:36.625 INFO:tasks.rados.rados.0.smithi151.stdout:2253: left oid 35 (ObjNum 846 snap 293 seq_num 846) 2024-09-13T14:30:36.625 INFO:tasks.rados.rados.0.smithi151.stdout:2255: finishing rollback tid 2 to smithi15186934-14 2024-09-13T14:30:36.625 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 935 (ObjNum 819 snap 286 seq_num 819) dirty exists 2024-09-13T14:30:36.625 INFO:tasks.rados.rados.0.smithi151.stdout:2259: finishing rollback tid 0 to smithi15186934-9 2024-09-13T14:30:36.625 INFO:tasks.rados.rados.0.smithi151.stdout:2259: finishing rollback tid 1 to smithi15186934-9 2024-09-13T14:30:36.625 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 937 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-13T14:30:36.625 INFO:tasks.rados.rados.0.smithi151.stdout:2257: expect (ObjNum 840 snap 293 seq_num 840) 2024-09-13T14:30:36.752 INFO:tasks.rados.rados.0.smithi151.stdout:2260: finishing write tid 1 to smithi15186934-50 2024-09-13T14:30:36.752 INFO:tasks.rados.rados.0.smithi151.stdout:2260: finishing write tid 2 to smithi15186934-50 2024-09-13T14:30:36.752 INFO:tasks.rados.rados.0.smithi151.stdout:2260: finishing write tid 3 to smithi15186934-50 2024-09-13T14:30:36.752 INFO:tasks.rados.rados.0.smithi151.stdout:2260: finishing write tid 4 to smithi15186934-50 2024-09-13T14:30:36.752 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 50 v 822 (ObjNum 847 snap 293 seq_num 847) dirty exists 2024-09-13T14:30:36.752 INFO:tasks.rados.rados.0.smithi151.stdout:2260: left oid 50 (ObjNum 847 snap 293 seq_num 847) 2024-09-13T14:30:36.752 INFO:tasks.rados.rados.0.smithi151.stdout:2261: expect (ObjNum 814 snap 285 seq_num 814) 2024-09-13T14:30:36.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:36 smithi184 ceph-mon[92527]: pgmap v585: 89 pgs: 89 active+clean; 207 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 7 op/s 2024-09-13T14:30:36.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:36 smithi184 ceph-mon[92527]: osdmap e645: 8 total, 8 up, 8 in 2024-09-13T14:30:36.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:36 smithi151 ceph-mon[104450]: pgmap v585: 89 pgs: 89 active+clean; 207 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 7 op/s 2024-09-13T14:30:36.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:36 smithi151 ceph-mon[104450]: osdmap e645: 8 total, 8 up, 8 in 2024-09-13T14:30:36.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:36 smithi151 ceph-mon[108966]: pgmap v585: 89 pgs: 89 active+clean; 207 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 7 op/s 2024-09-13T14:30:36.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:36 smithi151 ceph-mon[108966]: osdmap e645: 8 total, 8 up, 8 in 2024-09-13T14:30:36.869 INFO:tasks.rados.rados.0.smithi151.stdout:2253: done (10 left) 2024-09-13T14:30:36.869 INFO:tasks.rados.rados.0.smithi151.stdout:2254: done (9 left) 2024-09-13T14:30:36.869 INFO:tasks.rados.rados.0.smithi151.stdout:2255: done (8 left) 2024-09-13T14:30:36.869 INFO:tasks.rados.rados.0.smithi151.stdout:2256: done (7 left) 2024-09-13T14:30:36.869 INFO:tasks.rados.rados.0.smithi151.stdout:2257: done (6 left) 2024-09-13T14:30:36.870 INFO:tasks.rados.rados.0.smithi151.stdout:2258: done (5 left) 2024-09-13T14:30:36.870 INFO:tasks.rados.rados.0.smithi151.stdout:2259: done (4 left) 2024-09-13T14:30:36.870 INFO:tasks.rados.rados.0.smithi151.stdout:2260: done (3 left) 2024-09-13T14:30:36.870 INFO:tasks.rados.rados.0.smithi151.stdout:2261: done (2 left) 2024-09-13T14:30:36.870 INFO:tasks.rados.rados.0.smithi151.stdout:2262: done (1 left) 2024-09-13T14:30:36.870 INFO:tasks.rados.rados.0.smithi151.stdout:2263: done (0 left) 2024-09-13T14:30:36.870 INFO:tasks.rados.rados.0.smithi151.stdout:2264: write oid 14 current snap is 293 2024-09-13T14:30:36.870 INFO:tasks.rados.rados.0.smithi151.stdout:2264: seq_num 849 ranges {669143=401584,1564977=426959} 2024-09-13T14:30:36.874 INFO:tasks.rados.rados.0.smithi151.stdout:2264: writing smithi15186934-14 from 669143 to 1070727 tid 1 2024-09-13T14:30:36.879 INFO:tasks.rados.rados.0.smithi151.stdout:2264: writing smithi15186934-14 from 1564977 to 1991936 tid 2 2024-09-13T14:30:36.880 INFO:tasks.rados.rados.0.smithi151.stdout:2265: delete oid 41 current snap is 293 2024-09-13T14:30:36.880 INFO:tasks.rados.rados.0.smithi151.stdout:2264: finishing write tid 1 to smithi15186934-14 2024-09-13T14:30:36.881 INFO:tasks.rados.rados.0.smithi151.stdout:2265: done (1 left) 2024-09-13T14:30:36.881 INFO:tasks.rados.rados.0.smithi151.stdout:2266: read oid 46 snap -1 2024-09-13T14:30:36.881 INFO:tasks.rados.rados.0.smithi151.stdout:2266: expect (ObjNum 786 snap 275 seq_num 786) 2024-09-13T14:30:36.882 INFO:tasks.rados.rados.0.smithi151.stdout:2267: rollback oid 43 current snap is 293 2024-09-13T14:30:36.882 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 43 to 291 2024-09-13T14:30:36.882 INFO:tasks.rados.rados.0.smithi151.stdout:2268: read oid 13 snap 289 2024-09-13T14:30:36.882 INFO:tasks.rados.rados.0.smithi151.stdout:2268: expect (ObjNum 627 snap 204 seq_num 627) 2024-09-13T14:30:36.882 INFO:tasks.rados.rados.0.smithi151.stdout:2269: write oid 27 current snap is 293 2024-09-13T14:30:36.882 INFO:tasks.rados.rados.0.smithi151.stdout:2269: seq_num 851 ranges {702757=552082,1681941=430628} 2024-09-13T14:30:36.888 INFO:tasks.rados.rados.0.smithi151.stdout:2269: writing smithi15186934-27 from 702757 to 1254839 tid 1 2024-09-13T14:30:36.893 INFO:tasks.rados.rados.0.smithi151.stdout:2269: writing smithi15186934-27 from 1681941 to 2112569 tid 2 2024-09-13T14:30:36.893 INFO:tasks.rados.rados.0.smithi151.stdout:2270: write oid 29 current snap is 293 2024-09-13T14:30:36.894 INFO:tasks.rados.rados.0.smithi151.stdout:2264: finishing write tid 2 to smithi15186934-14 2024-09-13T14:30:36.894 INFO:tasks.rados.rados.0.smithi151.stdout:2270: seq_num 852 ranges {719564=743684,2140423=179141} 2024-09-13T14:30:36.901 INFO:tasks.rados.rados.0.smithi151.stdout:2270: writing smithi15186934-29 from 719564 to 1463248 tid 1 2024-09-13T14:30:36.906 INFO:tasks.rados.rados.0.smithi151.stdout:2270: writing smithi15186934-29 from 2140423 to 2319564 tid 2 2024-09-13T14:30:36.906 INFO:tasks.rados.rados.0.smithi151.stdout:2271: snap_create 2024-09-13T14:30:36.906 INFO:tasks.rados.rados.0.smithi151.stdout:2264: finishing write tid 3 to smithi15186934-14 2024-09-13T14:30:36.907 INFO:tasks.rados.rados.0.smithi151.stdout:2264: finishing write tid 4 to smithi15186934-14 2024-09-13T14:30:36.907 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 940 (ObjNum 849 snap 293 seq_num 849) dirty exists 2024-09-13T14:30:36.907 INFO:tasks.rados.rados.0.smithi151.stdout:2264: left oid 14 (ObjNum 849 snap 293 seq_num 849) 2024-09-13T14:30:36.907 INFO:tasks.rados.rados.0.smithi151.stdout:2267: finishing rollback tid 0 to smithi15186934-43 2024-09-13T14:30:36.907 INFO:tasks.rados.rados.0.smithi151.stdout:2267: finishing rollback tid 1 to smithi15186934-43 2024-09-13T14:30:36.907 INFO:tasks.rados.rados.0.smithi151.stdout:2267: finishing rollback tid 2 to smithi15186934-43 2024-09-13T14:30:36.907 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 43 v 1345 (ObjNum 828 snap 290 seq_num 828) dirty exists 2024-09-13T14:30:36.907 INFO:tasks.rados.rados.0.smithi151.stdout:2268: expect (ObjNum 627 snap 204 seq_num 627) 2024-09-13T14:30:37.079 INFO:tasks.rados.rados.0.smithi151.stdout:2266: expect (ObjNum 786 snap 275 seq_num 786) 2024-09-13T14:30:37.185 INFO:tasks.rados.rados.0.smithi151.stdout:2269: finishing write tid 1 to smithi15186934-27 2024-09-13T14:30:37.185 INFO:tasks.rados.rados.0.smithi151.stdout:2269: finishing write tid 2 to smithi15186934-27 2024-09-13T14:30:37.185 INFO:tasks.rados.rados.0.smithi151.stdout:2269: finishing write tid 3 to smithi15186934-27 2024-09-13T14:30:37.185 INFO:tasks.rados.rados.0.smithi151.stdout:2269: finishing write tid 4 to smithi15186934-27 2024-09-13T14:30:37.185 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 27 v 535 (ObjNum 851 snap 293 seq_num 851) dirty exists 2024-09-13T14:30:37.185 INFO:tasks.rados.rados.0.smithi151.stdout:2269: left oid 27 (ObjNum 851 snap 293 seq_num 851) 2024-09-13T14:30:37.185 INFO:tasks.rados.rados.0.smithi151.stdout:2270: finishing write tid 1 to smithi15186934-29 2024-09-13T14:30:37.185 INFO:tasks.rados.rados.0.smithi151.stdout:2270: finishing write tid 2 to smithi15186934-29 2024-09-13T14:30:37.185 INFO:tasks.rados.rados.0.smithi151.stdout:2270: finishing write tid 3 to smithi15186934-29 2024-09-13T14:30:37.185 INFO:tasks.rados.rados.0.smithi151.stdout:2270: finishing write tid 4 to smithi15186934-29 2024-09-13T14:30:37.185 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 29 v 578 (ObjNum 852 snap 293 seq_num 852) dirty exists 2024-09-13T14:30:37.186 INFO:tasks.rados.rados.0.smithi151.stdout:2270: left oid 29 (ObjNum 852 snap 293 seq_num 852) 2024-09-13T14:30:37.589 INFO:tasks.rados.rados.0.smithi151.stdout:2264: done (6 left) 2024-09-13T14:30:37.590 INFO:tasks.rados.rados.0.smithi151.stdout:2266: done (5 left) 2024-09-13T14:30:37.590 INFO:tasks.rados.rados.0.smithi151.stdout:2267: done (4 left) 2024-09-13T14:30:37.590 INFO:tasks.rados.rados.0.smithi151.stdout:2268: done (3 left) 2024-09-13T14:30:37.590 INFO:tasks.rados.rados.0.smithi151.stdout:2269: done (2 left) 2024-09-13T14:30:37.590 INFO:tasks.rados.rados.0.smithi151.stdout:2270: done (1 left) 2024-09-13T14:30:37.590 INFO:tasks.rados.rados.0.smithi151.stdout:2271: done (0 left) 2024-09-13T14:30:37.590 INFO:tasks.rados.rados.0.smithi151.stdout:2272: snap_create 2024-09-13T14:30:37.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:37 smithi184 ceph-mon[92527]: osdmap e646: 8 total, 8 up, 8 in 2024-09-13T14:30:37.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:37 smithi151 ceph-mon[104450]: osdmap e646: 8 total, 8 up, 8 in 2024-09-13T14:30:37.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:37 smithi151 ceph-mon[108966]: osdmap e646: 8 total, 8 up, 8 in 2024-09-13T14:30:38.591 INFO:tasks.rados.rados.0.smithi151.stdout:2272: done (0 left) 2024-09-13T14:30:38.591 INFO:tasks.rados.rados.0.smithi151.stdout:2273: delete oid 9 current snap is 295 2024-09-13T14:30:38.598 INFO:tasks.rados.rados.0.smithi151.stdout:2273: done (0 left) 2024-09-13T14:30:38.598 INFO:tasks.rados.rados.0.smithi151.stdout:2274: write oid 48 current snap is 295 2024-09-13T14:30:38.598 INFO:tasks.rados.rados.0.smithi151.stdout:2274: seq_num 854 ranges {753178=494182,1857387=495791} 2024-09-13T14:30:38.605 INFO:tasks.rados.rados.0.smithi151.stdout:2274: writing smithi15186934-48 from 753178 to 1247360 tid 1 2024-09-13T14:30:38.611 INFO:tasks.rados.rados.0.smithi151.stdout:2274: writing smithi15186934-48 from 1857387 to 2353178 tid 2 2024-09-13T14:30:38.611 INFO:tasks.rados.rados.0.smithi151.stdout:2275: snap_remove snap 288 2024-09-13T14:30:38.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:38 smithi184 ceph-mon[92527]: pgmap v588: 89 pgs: 89 active+clean; 214 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 6.8 MiB/s rd, 2.7 MiB/s wr, 22 op/s 2024-09-13T14:30:38.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:38 smithi184 ceph-mon[92527]: osdmap e647: 8 total, 8 up, 8 in 2024-09-13T14:30:38.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:38 smithi151 ceph-mon[104450]: pgmap v588: 89 pgs: 89 active+clean; 214 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 6.8 MiB/s rd, 2.7 MiB/s wr, 22 op/s 2024-09-13T14:30:38.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:38 smithi151 ceph-mon[104450]: osdmap e647: 8 total, 8 up, 8 in 2024-09-13T14:30:38.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:38 smithi151 ceph-mon[108966]: pgmap v588: 89 pgs: 89 active+clean; 214 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 6.8 MiB/s rd, 2.7 MiB/s wr, 22 op/s 2024-09-13T14:30:38.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:38 smithi151 ceph-mon[108966]: osdmap e647: 8 total, 8 up, 8 in 2024-09-13T14:30:39.543 INFO:tasks.rados.rados.0.smithi151.stdout:2275: done (1 left) 2024-09-13T14:30:39.543 INFO:tasks.rados.rados.0.smithi151.stdout:2276: read oid 34 snap -1 2024-09-13T14:30:39.543 INFO:tasks.rados.rados.0.smithi151.stdout:2276: expect (ObjNum 831 snap 291 seq_num 831) 2024-09-13T14:30:39.544 INFO:tasks.rados.rados.0.smithi151.stdout:2277: snap_remove snap 282 2024-09-13T14:30:39.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:39 smithi184 ceph-mon[92527]: osdmap e648: 8 total, 8 up, 8 in 2024-09-13T14:30:39.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:39 smithi184 ceph-mon[92527]: osdmap e649: 8 total, 8 up, 8 in 2024-09-13T14:30:39.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:39 smithi151 ceph-mon[104450]: osdmap e648: 8 total, 8 up, 8 in 2024-09-13T14:30:39.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:39 smithi151 ceph-mon[104450]: osdmap e649: 8 total, 8 up, 8 in 2024-09-13T14:30:39.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:39 smithi151 ceph-mon[108966]: osdmap e648: 8 total, 8 up, 8 in 2024-09-13T14:30:39.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:39 smithi151 ceph-mon[108966]: osdmap e649: 8 total, 8 up, 8 in 2024-09-13T14:30:40.546 INFO:tasks.rados.rados.0.smithi151.stdout:2277: done (2 left) 2024-09-13T14:30:40.546 INFO:tasks.rados.rados.0.smithi151.stdout:2278: snap_remove snap 287 2024-09-13T14:30:40.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:30:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:30:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:30:40.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:40 smithi184 ceph-mon[92527]: pgmap v591: 89 pgs: 89 active+clean; 210 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 2.2 MiB/s wr, 26 op/s 2024-09-13T14:30:40.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:40 smithi184 ceph-mon[92527]: osdmap e650: 8 total, 8 up, 8 in 2024-09-13T14:30:40.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:40 smithi151 ceph-mon[104450]: pgmap v591: 89 pgs: 89 active+clean; 210 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 2.2 MiB/s wr, 26 op/s 2024-09-13T14:30:40.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:40 smithi151 ceph-mon[104450]: osdmap e650: 8 total, 8 up, 8 in 2024-09-13T14:30:40.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:40 smithi151 ceph-mon[108966]: pgmap v591: 89 pgs: 89 active+clean; 210 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 2.2 MiB/s wr, 26 op/s 2024-09-13T14:30:40.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:40 smithi151 ceph-mon[108966]: osdmap e650: 8 total, 8 up, 8 in 2024-09-13T14:30:41.547 INFO:tasks.rados.rados.0.smithi151.stdout:2278: done (2 left) 2024-09-13T14:30:41.547 INFO:tasks.rados.rados.0.smithi151.stdout:2279: write oid 7 current snap is 295 2024-09-13T14:30:41.547 INFO:tasks.rados.rados.0.smithi151.stdout:2274: finishing write tid 1 to smithi15186934-48 2024-09-13T14:30:41.547 INFO:tasks.rados.rados.0.smithi151.stdout:2279: seq_num 855 ranges {769985=569431,1915869=437966} 2024-09-13T14:30:41.554 INFO:tasks.rados.rados.0.smithi151.stdout:2279: writing smithi15186934-7 from 769985 to 1339416 tid 1 2024-09-13T14:30:41.561 INFO:tasks.rados.rados.0.smithi151.stdout:2279: writing smithi15186934-7 from 1915869 to 2353835 tid 2 2024-09-13T14:30:41.561 INFO:tasks.rados.rados.0.smithi151.stdout:2280: snap_remove snap 255 2024-09-13T14:30:42.551 INFO:tasks.rados.rados.0.smithi151.stdout:2280: done (3 left) 2024-09-13T14:30:42.551 INFO:tasks.rados.rados.0.smithi151.stdout:2281: write oid 19 current snap is 295 2024-09-13T14:30:42.552 INFO:tasks.rados.rados.0.smithi151.stdout:2274: finishing write tid 2 to smithi15186934-48 2024-09-13T14:30:42.552 INFO:tasks.rados.rados.0.smithi151.stdout:2274: finishing write tid 3 to smithi15186934-48 2024-09-13T14:30:42.552 INFO:tasks.rados.rados.0.smithi151.stdout:2274: finishing write tid 4 to smithi15186934-48 2024-09-13T14:30:42.552 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 48 v 1226 (ObjNum 854 snap 295 seq_num 854) dirty exists 2024-09-13T14:30:42.552 INFO:tasks.rados.rados.0.smithi151.stdout:2274: left oid 48 (ObjNum 854 snap 295 seq_num 854) 2024-09-13T14:30:42.552 INFO:tasks.rados.rados.0.smithi151.stdout:2276: expect (ObjNum 831 snap 291 seq_num 831) 2024-09-13T14:30:42.733 INFO:tasks.rados.rados.0.smithi151.stdout:2279: finishing write tid 1 to smithi15186934-7 2024-09-13T14:30:42.733 INFO:tasks.rados.rados.0.smithi151.stdout:2279: finishing write tid 2 to smithi15186934-7 2024-09-13T14:30:42.733 INFO:tasks.rados.rados.0.smithi151.stdout:2279: finishing write tid 3 to smithi15186934-7 2024-09-13T14:30:42.733 INFO:tasks.rados.rados.0.smithi151.stdout:2279: finishing write tid 4 to smithi15186934-7 2024-09-13T14:30:42.733 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 1351 (ObjNum 855 snap 295 seq_num 855) dirty exists 2024-09-13T14:30:42.733 INFO:tasks.rados.rados.0.smithi151.stdout:2279: left oid 7 (ObjNum 855 snap 295 seq_num 855) 2024-09-13T14:30:42.733 INFO:tasks.rados.rados.0.smithi151.stdout:2281: seq_num 856 ranges {786792=644680,1857998=528794} 2024-09-13T14:30:42.740 INFO:tasks.rados.rados.0.smithi151.stdout:2281: writing smithi15186934-19 from 786792 to 1431472 tid 1 2024-09-13T14:30:42.746 INFO:tasks.rados.rados.0.smithi151.stdout:2281: writing smithi15186934-19 from 1857998 to 2386792 tid 2 2024-09-13T14:30:42.746 INFO:tasks.rados.rados.0.smithi151.stdout:2281: finishing write tid 1 to smithi15186934-19 2024-09-13T14:30:42.746 INFO:tasks.rados.rados.0.smithi151.stdout:2274: done (3 left) 2024-09-13T14:30:42.746 INFO:tasks.rados.rados.0.smithi151.stdout:2276: done (2 left) 2024-09-13T14:30:42.746 INFO:tasks.rados.rados.0.smithi151.stdout:2279: done (1 left) 2024-09-13T14:30:42.746 INFO:tasks.rados.rados.0.smithi151.stdout:2282: snap_create 2024-09-13T14:30:42.748 INFO:tasks.rados.rados.0.smithi151.stdout:2281: finishing write tid 2 to smithi15186934-19 2024-09-13T14:30:42.748 INFO:tasks.rados.rados.0.smithi151.stdout:2281: finishing write tid 3 to smithi15186934-19 2024-09-13T14:30:42.748 INFO:tasks.rados.rados.0.smithi151.stdout:2281: finishing write tid 4 to smithi15186934-19 2024-09-13T14:30:42.748 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 19 v 593 (ObjNum 856 snap 295 seq_num 856) dirty exists 2024-09-13T14:30:42.749 INFO:tasks.rados.rados.0.smithi151.stdout:2281: left oid 19 (ObjNum 856 snap 295 seq_num 856) 2024-09-13T14:30:42.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:42 smithi184 ceph-mon[92527]: pgmap v594: 89 pgs: 89 active+clean; 213 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 637 KiB/s wr, 15 op/s 2024-09-13T14:30:42.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:42 smithi184 ceph-mon[92527]: osdmap e651: 8 total, 8 up, 8 in 2024-09-13T14:30:42.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:42 smithi151 ceph-mon[104450]: pgmap v594: 89 pgs: 89 active+clean; 213 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 637 KiB/s wr, 15 op/s 2024-09-13T14:30:42.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:42 smithi151 ceph-mon[104450]: osdmap e651: 8 total, 8 up, 8 in 2024-09-13T14:30:42.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:42 smithi151 ceph-mon[108966]: pgmap v594: 89 pgs: 89 active+clean; 213 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 637 KiB/s wr, 15 op/s 2024-09-13T14:30:42.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:42 smithi151 ceph-mon[108966]: osdmap e651: 8 total, 8 up, 8 in 2024-09-13T14:30:43.561 INFO:tasks.rados.rados.0.smithi151.stdout:2281: done (1 left) 2024-09-13T14:30:43.561 INFO:tasks.rados.rados.0.smithi151.stdout:2282: done (0 left) 2024-09-13T14:30:43.561 INFO:tasks.rados.rados.0.smithi151.stdout:2283: delete oid 11 current snap is 296 2024-09-13T14:30:43.565 INFO:tasks.rados.rados.0.smithi151.stdout:2283: done (0 left) 2024-09-13T14:30:43.565 INFO:tasks.rados.rados.0.smithi151.stdout:2284: rollback oid 44 current snap is 296 2024-09-13T14:30:43.565 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 44 to 276 2024-09-13T14:30:43.565 INFO:tasks.rados.rados.0.smithi151.stdout:2285: write oid 45 current snap is 296 2024-09-13T14:30:43.565 INFO:tasks.rados.rados.0.smithi151.stdout:2285: seq_num 858 ranges {420406=795178,1974962=445444} 2024-09-13T14:30:43.571 INFO:tasks.rados.rados.0.smithi151.stdout:2285: writing smithi15186934-45 from 420406 to 1215584 tid 1 2024-09-13T14:30:43.579 INFO:tasks.rados.rados.0.smithi151.stdout:2285: writing smithi15186934-45 from 1974962 to 2420406 tid 2 2024-09-13T14:30:43.579 INFO:tasks.rados.rados.0.smithi151.stdout:2286: rollback oid 41 current snap is 296 2024-09-13T14:30:43.579 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 41 to 289 2024-09-13T14:30:43.579 INFO:tasks.rados.rados.0.smithi151.stdout:2284: finishing rollback tid 0 to smithi15186934-44 2024-09-13T14:30:43.579 INFO:tasks.rados.rados.0.smithi151.stdout:2284: finishing rollback tid 1 to smithi15186934-44 2024-09-13T14:30:43.579 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 710 (ObjNum 161 snap 0 seq_num 3093031536) dirty dne 2024-09-13T14:30:43.579 INFO:tasks.rados.rados.0.smithi151.stdout:2285: finishing write tid 1 to smithi15186934-45 2024-09-13T14:30:43.579 INFO:tasks.rados.rados.0.smithi151.stdout:2284: done (2 left) 2024-09-13T14:30:43.579 INFO:tasks.rados.rados.0.smithi151.stdout:2287: delete oid 17 current snap is 296 2024-09-13T14:30:43.582 INFO:tasks.rados.rados.0.smithi151.stdout:2285: finishing write tid 2 to smithi15186934-45 2024-09-13T14:30:43.582 INFO:tasks.rados.rados.0.smithi151.stdout:2285: finishing write tid 3 to smithi15186934-45 2024-09-13T14:30:43.582 INFO:tasks.rados.rados.0.smithi151.stdout:2285: finishing write tid 4 to smithi15186934-45 2024-09-13T14:30:43.582 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 1241 (ObjNum 858 snap 296 seq_num 858) dirty exists 2024-09-13T14:30:43.582 INFO:tasks.rados.rados.0.smithi151.stdout:2285: left oid 45 (ObjNum 858 snap 296 seq_num 858) 2024-09-13T14:30:43.597 INFO:tasks.rados.rados.0.smithi151.stdout:2286: finishing rollback tid 1 to smithi15186934-41 2024-09-13T14:30:43.597 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 0 (ObjNum 129 snap 0 seq_num 3092601344) dirty dne 2024-09-13T14:30:43.598 INFO:tasks.rados.rados.0.smithi151.stdout:2285: done (2 left) 2024-09-13T14:30:43.598 INFO:tasks.rados.rados.0.smithi151.stdout:2286: done (1 left) 2024-09-13T14:30:43.598 INFO:tasks.rados.rados.0.smithi151.stdout:2287: done (0 left) 2024-09-13T14:30:43.598 INFO:tasks.rados.rados.0.smithi151.stdout:2288: snap_create 2024-09-13T14:30:43.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:43 smithi184 ceph-mon[92527]: osdmap e652: 8 total, 8 up, 8 in 2024-09-13T14:30:43.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:43 smithi151 ceph-mon[104450]: osdmap e652: 8 total, 8 up, 8 in 2024-09-13T14:30:43.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:43 smithi151 ceph-mon[108966]: osdmap e652: 8 total, 8 up, 8 in 2024-09-13T14:30:44.544 INFO:tasks.rados.rados.0.smithi151.stdout:2288: done (0 left) 2024-09-13T14:30:44.544 INFO:tasks.rados.rados.0.smithi151.stdout:2289: write oid 22 current snap is 297 2024-09-13T14:30:44.544 INFO:tasks.rados.rados.0.smithi151.stdout:2289: seq_num 860 ranges {454020=662029,1691926=588962} 2024-09-13T14:30:44.549 INFO:tasks.rados.rados.0.smithi151.stdout:2289: writing smithi15186934-22 from 454020 to 1116049 tid 1 2024-09-13T14:30:44.556 INFO:tasks.rados.rados.0.smithi151.stdout:2289: writing smithi15186934-22 from 1691926 to 2280888 tid 2 2024-09-13T14:30:44.556 INFO:tasks.rados.rados.0.smithi151.stdout:2290: rollback oid 4 current snap is 297 2024-09-13T14:30:44.556 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 4 to 276 2024-09-13T14:30:44.556 INFO:tasks.rados.rados.0.smithi151.stdout:2291: delete oid 42 current snap is 297 2024-09-13T14:30:44.557 INFO:tasks.rados.rados.0.smithi151.stdout:2289: finishing write tid 1 to smithi15186934-22 2024-09-13T14:30:44.560 INFO:tasks.rados.rados.0.smithi151.stdout:2290: finishing rollback tid 0 to smithi15186934-4 2024-09-13T14:30:44.561 INFO:tasks.rados.rados.0.smithi151.stdout:2289: finishing write tid 2 to smithi15186934-22 2024-09-13T14:30:44.561 INFO:tasks.rados.rados.0.smithi151.stdout:2289: finishing write tid 3 to smithi15186934-22 2024-09-13T14:30:44.561 INFO:tasks.rados.rados.0.smithi151.stdout:2289: finishing write tid 4 to smithi15186934-22 2024-09-13T14:30:44.561 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 22 v 480 (ObjNum 860 snap 297 seq_num 860) dirty exists 2024-09-13T14:30:44.561 INFO:tasks.rados.rados.0.smithi151.stdout:2289: left oid 22 (ObjNum 860 snap 297 seq_num 860) 2024-09-13T14:30:44.564 INFO:tasks.rados.rados.0.smithi151.stdout:2289: done (2 left) 2024-09-13T14:30:44.564 INFO:tasks.rados.rados.0.smithi151.stdout:2291: done (1 left) 2024-09-13T14:30:44.564 INFO:tasks.rados.rados.0.smithi151.stdout:2292: write oid 10 current snap is 297 2024-09-13T14:30:44.565 INFO:tasks.rados.rados.0.smithi151.stdout:2292: seq_num 862 ranges {487634=412527,1408890=592631} 2024-09-13T14:30:44.569 INFO:tasks.rados.rados.0.smithi151.stdout:2292: writing smithi15186934-10 from 487634 to 900161 tid 1 2024-09-13T14:30:44.576 INFO:tasks.rados.rados.0.smithi151.stdout:2292: writing smithi15186934-10 from 1408890 to 2001521 tid 2 2024-09-13T14:30:44.576 INFO:tasks.rados.rados.0.smithi151.stdout:2293: snap_create 2024-09-13T14:30:44.576 INFO:tasks.rados.rados.0.smithi151.stdout:2290: finishing rollback tid 1 to smithi15186934-4 2024-09-13T14:30:44.576 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 1244 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-13T14:30:44.576 INFO:tasks.rados.rados.0.smithi151.stdout:2292: finishing write tid 1 to smithi15186934-10 2024-09-13T14:30:44.579 INFO:tasks.rados.rados.0.smithi151.stdout:2292: finishing write tid 2 to smithi15186934-10 2024-09-13T14:30:44.579 INFO:tasks.rados.rados.0.smithi151.stdout:2292: finishing write tid 3 to smithi15186934-10 2024-09-13T14:30:44.579 INFO:tasks.rados.rados.0.smithi151.stdout:2292: finishing write tid 4 to smithi15186934-10 2024-09-13T14:30:44.579 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 825 (ObjNum 862 snap 297 seq_num 862) dirty exists 2024-09-13T14:30:44.579 INFO:tasks.rados.rados.0.smithi151.stdout:2292: left oid 10 (ObjNum 862 snap 297 seq_num 862) 2024-09-13T14:30:44.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:44 smithi184 ceph-mon[92527]: pgmap v597: 89 pgs: 89 active+clean; 217 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 8.4 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2024-09-13T14:30:44.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:44 smithi184 ceph-mon[92527]: osdmap e653: 8 total, 8 up, 8 in 2024-09-13T14:30:44.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:44 smithi184 ceph-mon[92527]: osdmap e654: 8 total, 8 up, 8 in 2024-09-13T14:30:44.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:44 smithi151 ceph-mon[104450]: pgmap v597: 89 pgs: 89 active+clean; 217 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 8.4 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2024-09-13T14:30:44.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:44 smithi151 ceph-mon[104450]: osdmap e653: 8 total, 8 up, 8 in 2024-09-13T14:30:44.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:44 smithi151 ceph-mon[104450]: osdmap e654: 8 total, 8 up, 8 in 2024-09-13T14:30:44.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:44 smithi151 ceph-mon[108966]: pgmap v597: 89 pgs: 89 active+clean; 217 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 8.4 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2024-09-13T14:30:44.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:44 smithi151 ceph-mon[108966]: osdmap e653: 8 total, 8 up, 8 in 2024-09-13T14:30:44.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:44 smithi151 ceph-mon[108966]: osdmap e654: 8 total, 8 up, 8 in 2024-09-13T14:30:44.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:30:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:30:44] "GET /metrics HTTP/1.1" 200 36505 "" "Prometheus/2.51.0" 2024-09-13T14:30:45.570 INFO:tasks.rados.rados.0.smithi151.stdout:2290: done (2 left) 2024-09-13T14:30:45.570 INFO:tasks.rados.rados.0.smithi151.stdout:2292: done (1 left) 2024-09-13T14:30:45.571 INFO:tasks.rados.rados.0.smithi151.stdout:2293: done (0 left) 2024-09-13T14:30:45.571 INFO:tasks.rados.rados.0.smithi151.stdout:2294: read oid 3 snap -1 2024-09-13T14:30:45.571 INFO:tasks.rados.rados.0.smithi151.stdout:2294: expect (ObjNum 780 snap 273 seq_num 780) 2024-09-13T14:30:45.571 INFO:tasks.rados.rados.0.smithi151.stdout:2295: read oid 22 snap -1 2024-09-13T14:30:45.571 INFO:tasks.rados.rados.0.smithi151.stdout:2295: expect (ObjNum 860 snap 297 seq_num 860) 2024-09-13T14:30:45.571 INFO:tasks.rados.rados.0.smithi151.stdout:2296: rollback oid 35 current snap is 298 2024-09-13T14:30:45.571 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 35 to 290 2024-09-13T14:30:45.571 INFO:tasks.rados.rados.0.smithi151.stdout:2297: read oid 21 snap -1 2024-09-13T14:30:45.571 INFO:tasks.rados.rados.0.smithi151.stdout:2297: expect deleted 2024-09-13T14:30:45.571 INFO:tasks.rados.rados.0.smithi151.stdout:2298: read oid 8 snap 284 2024-09-13T14:30:45.571 INFO:tasks.rados.rados.0.smithi151.stdout:2298: expect deleted 2024-09-13T14:30:45.571 INFO:tasks.rados.rados.0.smithi151.stdout:2299: snap_create 2024-09-13T14:30:45.578 INFO:tasks.rados.rados.0.smithi151.stdout:2296: finishing rollback tid 0 to smithi15186934-35 2024-09-13T14:30:45.578 INFO:tasks.rados.rados.0.smithi151.stdout:2296: finishing rollback tid 1 to smithi15186934-35 2024-09-13T14:30:45.578 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 35 v 1359 (ObjNum 129 snap 0 seq_num 3092894728) dirty dne 2024-09-13T14:30:45.582 INFO:tasks.rados.rados.0.smithi151.stdout:2294: expect (ObjNum 780 snap 273 seq_num 780) 2024-09-13T14:30:45.749 INFO:tasks.rados.rados.0.smithi151.stdout:2295: expect (ObjNum 860 snap 297 seq_num 860) 2024-09-13T14:30:46.575 INFO:tasks.rados.rados.0.smithi151.stdout:2294: done (5 left) 2024-09-13T14:30:46.575 INFO:tasks.rados.rados.0.smithi151.stdout:2295: done (4 left) 2024-09-13T14:30:46.575 INFO:tasks.rados.rados.0.smithi151.stdout:2296: done (3 left) 2024-09-13T14:30:46.575 INFO:tasks.rados.rados.0.smithi151.stdout:2297: done (2 left) 2024-09-13T14:30:46.575 INFO:tasks.rados.rados.0.smithi151.stdout:2298: done (1 left) 2024-09-13T14:30:46.575 INFO:tasks.rados.rados.0.smithi151.stdout:2299: done (0 left) 2024-09-13T14:30:46.575 INFO:tasks.rados.rados.0.smithi151.stdout:2300: read oid 48 snap 289 2024-09-13T14:30:46.576 INFO:tasks.rados.rados.0.smithi151.stdout:2300: expect deleted 2024-09-13T14:30:46.576 INFO:tasks.rados.rados.0.smithi151.stdout:2301: delete oid 12 current snap is 299 2024-09-13T14:30:46.579 INFO:tasks.rados.rados.0.smithi151.stdout:2300: done (1 left) 2024-09-13T14:30:46.579 INFO:tasks.rados.rados.0.smithi151.stdout:2301: done (0 left) 2024-09-13T14:30:46.580 INFO:tasks.rados.rados.0.smithi151.stdout:2302: snap_create 2024-09-13T14:30:46.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:46 smithi184 ceph-mon[92527]: pgmap v600: 89 pgs: 89 active+clean; 207 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 488 KiB/s wr, 2 op/s 2024-09-13T14:30:46.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:46 smithi184 ceph-mon[92527]: osdmap e655: 8 total, 8 up, 8 in 2024-09-13T14:30:46.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:46 smithi151 ceph-mon[104450]: pgmap v600: 89 pgs: 89 active+clean; 207 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 488 KiB/s wr, 2 op/s 2024-09-13T14:30:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:46 smithi151 ceph-mon[104450]: osdmap e655: 8 total, 8 up, 8 in 2024-09-13T14:30:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:46.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:46 smithi151 ceph-mon[108966]: pgmap v600: 89 pgs: 89 active+clean; 207 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 488 KiB/s wr, 2 op/s 2024-09-13T14:30:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:46 smithi151 ceph-mon[108966]: osdmap e655: 8 total, 8 up, 8 in 2024-09-13T14:30:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:47.588 INFO:tasks.rados.rados.0.smithi151.stdout:2302: done (0 left) 2024-09-13T14:30:47.589 INFO:tasks.rados.rados.0.smithi151.stdout:2303: read oid 45 snap 284 2024-09-13T14:30:47.589 INFO:tasks.rados.rados.0.smithi151.stdout:2303: expect (ObjNum 740 snap 256 seq_num 740) 2024-09-13T14:30:47.589 INFO:tasks.rados.rados.0.smithi151.stdout:2304: delete oid 47 current snap is 300 2024-09-13T14:30:47.591 INFO:tasks.rados.rados.0.smithi151.stdout:2304: done (1 left) 2024-09-13T14:30:47.591 INFO:tasks.rados.rados.0.smithi151.stdout:2305: snap_create 2024-09-13T14:30:47.598 INFO:tasks.rados.rados.0.smithi151.stdout:2303: expect (ObjNum 740 snap 256 seq_num 740) 2024-09-13T14:30:47.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:47 smithi184 ceph-mon[92527]: osdmap e656: 8 total, 8 up, 8 in 2024-09-13T14:30:47.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:47 smithi151 ceph-mon[104450]: osdmap e656: 8 total, 8 up, 8 in 2024-09-13T14:30:47.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:47 smithi151 ceph-mon[108966]: osdmap e656: 8 total, 8 up, 8 in 2024-09-13T14:30:48.593 INFO:tasks.rados.rados.0.smithi151.stdout:2303: done (1 left) 2024-09-13T14:30:48.594 INFO:tasks.rados.rados.0.smithi151.stdout:2305: done (0 left) 2024-09-13T14:30:48.594 INFO:tasks.rados.rados.0.smithi151.stdout:2306: snap_create 2024-09-13T14:30:48.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:48 smithi184 ceph-mon[92527]: pgmap v603: 89 pgs: 89 active+clean; 170 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 896 KiB/s wr, 7 op/s 2024-09-13T14:30:48.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:48 smithi184 ceph-mon[92527]: osdmap e657: 8 total, 8 up, 8 in 2024-09-13T14:30:48.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:48 smithi151 ceph-mon[104450]: pgmap v603: 89 pgs: 89 active+clean; 170 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 896 KiB/s wr, 7 op/s 2024-09-13T14:30:48.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:48 smithi151 ceph-mon[104450]: osdmap e657: 8 total, 8 up, 8 in 2024-09-13T14:30:48.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:48 smithi151 ceph-mon[108966]: pgmap v603: 89 pgs: 89 active+clean; 170 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 896 KiB/s wr, 7 op/s 2024-09-13T14:30:48.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:48 smithi151 ceph-mon[108966]: osdmap e657: 8 total, 8 up, 8 in 2024-09-13T14:30:49.548 INFO:tasks.rados.rados.0.smithi151.stdout:2306: done (0 left) 2024-09-13T14:30:49.549 INFO:tasks.rados.rados.0.smithi151.stdout:2307: read oid 34 snap -1 2024-09-13T14:30:49.549 INFO:tasks.rados.rados.0.smithi151.stdout:2307: expect (ObjNum 831 snap 291 seq_num 831) 2024-09-13T14:30:49.549 INFO:tasks.rados.rados.0.smithi151.stdout:2308: read oid 49 snap -1 2024-09-13T14:30:49.549 INFO:tasks.rados.rados.0.smithi151.stdout:2308: expect deleted 2024-09-13T14:30:49.549 INFO:tasks.rados.rados.0.smithi151.stdout:2309: snap_create 2024-09-13T14:30:49.558 INFO:tasks.rados.rados.0.smithi151.stdout:2307: expect (ObjNum 831 snap 291 seq_num 831) 2024-09-13T14:30:49.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:49 smithi151 ceph-mon[104450]: osdmap e658: 8 total, 8 up, 8 in 2024-09-13T14:30:49.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:49 smithi151 ceph-mon[104450]: osdmap e659: 8 total, 8 up, 8 in 2024-09-13T14:30:49.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:49 smithi151 ceph-mon[108966]: osdmap e658: 8 total, 8 up, 8 in 2024-09-13T14:30:49.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:49 smithi151 ceph-mon[108966]: osdmap e659: 8 total, 8 up, 8 in 2024-09-13T14:30:50.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:49 smithi184 ceph-mon[92527]: osdmap e658: 8 total, 8 up, 8 in 2024-09-13T14:30:50.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:49 smithi184 ceph-mon[92527]: osdmap e659: 8 total, 8 up, 8 in 2024-09-13T14:30:50.547 INFO:tasks.rados.rados.0.smithi151.stdout:2307: done (2 left) 2024-09-13T14:30:50.547 INFO:tasks.rados.rados.0.smithi151.stdout:2308: done (1 left) 2024-09-13T14:30:50.547 INFO:tasks.rados.rados.0.smithi151.stdout:2309: done (0 left) 2024-09-13T14:30:50.547 INFO:tasks.rados.rados.0.smithi151.stdout:2310: read oid 40 snap 300 2024-09-13T14:30:50.547 INFO:tasks.rados.rados.0.smithi151.stdout:2310: expect (ObjNum 741 snap 256 seq_num 741) 2024-09-13T14:30:50.547 INFO:tasks.rados.rados.0.smithi151.stdout:2311: write oid 38 current snap is 303 2024-09-13T14:30:50.548 INFO:tasks.rados.rados.0.smithi151.stdout:2311: seq_num 865 ranges {538055=638274,1867983=670072} 2024-09-13T14:30:50.553 INFO:tasks.rados.rados.0.smithi151.stdout:2311: writing smithi15186934-38 from 538055 to 1176329 tid 1 2024-09-13T14:30:50.562 INFO:tasks.rados.rados.0.smithi151.stdout:2311: writing smithi15186934-38 from 1867983 to 2538055 tid 2 2024-09-13T14:30:50.562 INFO:tasks.rados.rados.0.smithi151.stdout:2312: read oid 44 snap 299 2024-09-13T14:30:50.562 INFO:tasks.rados.rados.0.smithi151.stdout:2312: expect deleted 2024-09-13T14:30:50.562 INFO:tasks.rados.rados.0.smithi151.stdout:2313: snap_create 2024-09-13T14:30:50.562 INFO:tasks.rados.rados.0.smithi151.stdout:2310: expect (ObjNum 741 snap 256 seq_num 741) 2024-09-13T14:30:50.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:30:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:30:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:30:50.641 INFO:tasks.rados.rados.0.smithi151.stdout:2311: finishing write tid 1 to smithi15186934-38 2024-09-13T14:30:50.642 INFO:tasks.rados.rados.0.smithi151.stdout:2311: finishing write tid 2 to smithi15186934-38 2024-09-13T14:30:50.642 INFO:tasks.rados.rados.0.smithi151.stdout:2311: finishing write tid 3 to smithi15186934-38 2024-09-13T14:30:50.642 INFO:tasks.rados.rados.0.smithi151.stdout:2311: finishing write tid 4 to smithi15186934-38 2024-09-13T14:30:50.642 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 844 (ObjNum 865 snap 303 seq_num 865) dirty exists 2024-09-13T14:30:50.642 INFO:tasks.rados.rados.0.smithi151.stdout:2311: left oid 38 (ObjNum 865 snap 303 seq_num 865) 2024-09-13T14:30:50.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:50 smithi151 ceph-mon[108966]: pgmap v606: 89 pgs: 89 active+clean; 172 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-13T14:30:50.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:50 smithi151 ceph-mon[108966]: osdmap e660: 8 total, 8 up, 8 in 2024-09-13T14:30:50.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:50 smithi151 ceph-mon[104450]: pgmap v606: 89 pgs: 89 active+clean; 172 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-13T14:30:50.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:50 smithi151 ceph-mon[104450]: osdmap e660: 8 total, 8 up, 8 in 2024-09-13T14:30:51.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:50 smithi184 ceph-mon[92527]: pgmap v606: 89 pgs: 89 active+clean; 172 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-13T14:30:51.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:50 smithi184 ceph-mon[92527]: osdmap e660: 8 total, 8 up, 8 in 2024-09-13T14:30:51.549 INFO:tasks.rados.rados.0.smithi151.stdout:2310: done (3 left) 2024-09-13T14:30:51.549 INFO:tasks.rados.rados.0.smithi151.stdout:2311: done (2 left) 2024-09-13T14:30:51.549 INFO:tasks.rados.rados.0.smithi151.stdout:2312: done (1 left) 2024-09-13T14:30:51.549 INFO:tasks.rados.rados.0.smithi151.stdout:2313: done (0 left) 2024-09-13T14:30:51.549 INFO:tasks.rados.rados.0.smithi151.stdout:2314: snap_create 2024-09-13T14:30:52.554 INFO:tasks.rados.rados.0.smithi151.stdout:2314: done (0 left) 2024-09-13T14:30:52.554 INFO:tasks.rados.rados.0.smithi151.stdout:2315: read oid 36 snap 298 2024-09-13T14:30:52.554 INFO:tasks.rados.rados.0.smithi151.stdout:2315: expect (ObjNum 764 snap 264 seq_num 764) 2024-09-13T14:30:52.555 INFO:tasks.rados.rados.0.smithi151.stdout:2316: write oid 21 current snap is 305 2024-09-13T14:30:52.555 INFO:tasks.rados.rados.0.smithi151.stdout:2316: seq_num 866 ranges {554862=713523,1926465=599969} 2024-09-13T14:30:52.561 INFO:tasks.rados.rados.0.smithi151.stdout:2316: writing smithi15186934-21 from 554862 to 1268385 tid 1 2024-09-13T14:30:52.568 INFO:tasks.rados.rados.0.smithi151.stdout:2316: writing smithi15186934-21 from 1926465 to 2526434 tid 2 2024-09-13T14:30:52.569 INFO:tasks.rados.rados.0.smithi151.stdout:2317: snap_create 2024-09-13T14:30:52.569 INFO:tasks.rados.rados.0.smithi151.stdout:2315: expect (ObjNum 764 snap 264 seq_num 764) 2024-09-13T14:30:52.661 INFO:tasks.rados.rados.0.smithi151.stdout:2316: finishing write tid 1 to smithi15186934-21 2024-09-13T14:30:52.661 INFO:tasks.rados.rados.0.smithi151.stdout:2316: finishing write tid 2 to smithi15186934-21 2024-09-13T14:30:52.661 INFO:tasks.rados.rados.0.smithi151.stdout:2316: finishing write tid 3 to smithi15186934-21 2024-09-13T14:30:52.661 INFO:tasks.rados.rados.0.smithi151.stdout:2316: finishing write tid 4 to smithi15186934-21 2024-09-13T14:30:52.661 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 21 v 483 (ObjNum 866 snap 305 seq_num 866) dirty exists 2024-09-13T14:30:52.661 INFO:tasks.rados.rados.0.smithi151.stdout:2316: left oid 21 (ObjNum 866 snap 305 seq_num 866) 2024-09-13T14:30:52.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:52 smithi184 ceph-mon[92527]: pgmap v609: 89 pgs: 89 active+clean; 172 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.2 MiB/s rd, 245 KiB/s wr, 6 op/s 2024-09-13T14:30:52.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:52 smithi184 ceph-mon[92527]: osdmap e661: 8 total, 8 up, 8 in 2024-09-13T14:30:52.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:52 smithi151 ceph-mon[104450]: pgmap v609: 89 pgs: 89 active+clean; 172 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.2 MiB/s rd, 245 KiB/s wr, 6 op/s 2024-09-13T14:30:52.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:52 smithi151 ceph-mon[104450]: osdmap e661: 8 total, 8 up, 8 in 2024-09-13T14:30:52.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:52 smithi151 ceph-mon[108966]: pgmap v609: 89 pgs: 89 active+clean; 172 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.2 MiB/s rd, 245 KiB/s wr, 6 op/s 2024-09-13T14:30:52.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:52 smithi151 ceph-mon[108966]: osdmap e661: 8 total, 8 up, 8 in 2024-09-13T14:30:53.565 INFO:tasks.rados.rados.0.smithi151.stdout:2315: done (2 left) 2024-09-13T14:30:53.565 INFO:tasks.rados.rados.0.smithi151.stdout:2316: done (1 left) 2024-09-13T14:30:53.565 INFO:tasks.rados.rados.0.smithi151.stdout:2317: done (0 left) 2024-09-13T14:30:53.565 INFO:tasks.rados.rados.0.smithi151.stdout:2318: snap_create 2024-09-13T14:30:53.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:53 smithi184 ceph-mon[92527]: osdmap e662: 8 total, 8 up, 8 in 2024-09-13T14:30:53.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:53 smithi151 ceph-mon[104450]: osdmap e662: 8 total, 8 up, 8 in 2024-09-13T14:30:53.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:53 smithi151 ceph-mon[108966]: osdmap e662: 8 total, 8 up, 8 in 2024-09-13T14:30:54.565 INFO:tasks.rados.rados.0.smithi151.stdout:2318: done (0 left) 2024-09-13T14:30:54.565 INFO:tasks.rados.rados.0.smithi151.stdout:2319: snap_remove snap 293 2024-09-13T14:30:54.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:54 smithi184 ceph-mon[92527]: pgmap v612: 89 pgs: 89 active+clean; 173 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 320 KiB/s wr, 8 op/s 2024-09-13T14:30:54.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:54 smithi184 ceph-mon[92527]: osdmap e663: 8 total, 8 up, 8 in 2024-09-13T14:30:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:54 smithi151 ceph-mon[104450]: pgmap v612: 89 pgs: 89 active+clean; 173 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 320 KiB/s wr, 8 op/s 2024-09-13T14:30:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:54 smithi151 ceph-mon[104450]: osdmap e663: 8 total, 8 up, 8 in 2024-09-13T14:30:54.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:54 smithi151 ceph-mon[108966]: pgmap v612: 89 pgs: 89 active+clean; 173 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 320 KiB/s wr, 8 op/s 2024-09-13T14:30:54.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:54 smithi151 ceph-mon[108966]: osdmap e663: 8 total, 8 up, 8 in 2024-09-13T14:30:54.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:30:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:30:54] "GET /metrics HTTP/1.1" 200 36510 "" "Prometheus/2.51.0" 2024-09-13T14:30:55.567 INFO:tasks.rados.rados.0.smithi151.stdout:2319: done (0 left) 2024-09-13T14:30:55.567 INFO:tasks.rados.rados.0.smithi151.stdout:2320: snap_remove snap 296 2024-09-13T14:30:55.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:55 smithi184 ceph-mon[92527]: osdmap e664: 8 total, 8 up, 8 in 2024-09-13T14:30:55.860 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:55 smithi184 ceph-mon[92527]: osdmap e665: 8 total, 8 up, 8 in 2024-09-13T14:30:55.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:55 smithi151 ceph-mon[104450]: osdmap e664: 8 total, 8 up, 8 in 2024-09-13T14:30:55.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:55 smithi151 ceph-mon[104450]: osdmap e665: 8 total, 8 up, 8 in 2024-09-13T14:30:55.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:55 smithi151 ceph-mon[108966]: osdmap e664: 8 total, 8 up, 8 in 2024-09-13T14:30:55.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:55 smithi151 ceph-mon[108966]: osdmap e665: 8 total, 8 up, 8 in 2024-09-13T14:30:56.570 INFO:tasks.rados.rados.0.smithi151.stdout:2320: done (0 left) 2024-09-13T14:30:56.570 INFO:tasks.rados.rados.0.smithi151.stdout:2321: snap_create 2024-09-13T14:30:56.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:56 smithi184 ceph-mon[92527]: pgmap v615: 89 pgs: 89 active+clean; 173 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 320 KiB/s wr, 3 op/s 2024-09-13T14:30:56.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:56 smithi184 ceph-mon[92527]: osdmap e666: 8 total, 8 up, 8 in 2024-09-13T14:30:56.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:56 smithi151 ceph-mon[104450]: pgmap v615: 89 pgs: 89 active+clean; 173 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 320 KiB/s wr, 3 op/s 2024-09-13T14:30:56.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:56 smithi151 ceph-mon[104450]: osdmap e666: 8 total, 8 up, 8 in 2024-09-13T14:30:56.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:56 smithi151 ceph-mon[108966]: pgmap v615: 89 pgs: 89 active+clean; 173 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 320 KiB/s wr, 3 op/s 2024-09-13T14:30:56.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:56 smithi151 ceph-mon[108966]: osdmap e666: 8 total, 8 up, 8 in 2024-09-13T14:30:57.571 INFO:tasks.rados.rados.0.smithi151.stdout:2321: done (0 left) 2024-09-13T14:30:57.572 INFO:tasks.rados.rados.0.smithi151.stdout:2322: snap_remove snap 291 2024-09-13T14:30:58.574 INFO:tasks.rados.rados.0.smithi151.stdout:2322: done (0 left) 2024-09-13T14:30:58.574 INFO:tasks.rados.rados.0.smithi151.stdout:2323: write oid 50 current snap is 308 2024-09-13T14:30:58.574 INFO:tasks.rados.rados.0.smithi151.stdout:2323: seq_num 867 ranges {571669=505125,1701300=743627} 2024-09-13T14:30:58.580 INFO:tasks.rados.rados.0.smithi151.stdout:2323: writing smithi15186934-50 from 571669 to 1076794 tid 1 2024-09-13T14:30:58.589 INFO:tasks.rados.rados.0.smithi151.stdout:2323: writing smithi15186934-50 from 1701300 to 2444927 tid 2 2024-09-13T14:30:58.589 INFO:tasks.rados.rados.0.smithi151.stdout:2324: read oid 4 snap -1 2024-09-13T14:30:58.589 INFO:tasks.rados.rados.0.smithi151.stdout:2324: expect deleted 2024-09-13T14:30:58.589 INFO:tasks.rados.rados.0.smithi151.stdout:2325: delete oid 12 current snap is 308 2024-09-13T14:30:58.589 INFO:tasks.rados.rados.0.smithi151.stdout:2323: finishing write tid 1 to smithi15186934-50 2024-09-13T14:30:58.591 INFO:tasks.rados.rados.0.smithi151.stdout:2324: done (2 left) 2024-09-13T14:30:58.591 INFO:tasks.rados.rados.0.smithi151.stdout:2325: done (1 left) 2024-09-13T14:30:58.591 INFO:tasks.rados.rados.0.smithi151.stdout:2326: read oid 23 snap -1 2024-09-13T14:30:58.591 INFO:tasks.rados.rados.0.smithi151.stdout:2326: expect (ObjNum 774 snap 272 seq_num 774) 2024-09-13T14:30:58.591 INFO:tasks.rados.rados.0.smithi151.stdout:2327: read oid 49 snap -1 2024-09-13T14:30:58.591 INFO:tasks.rados.rados.0.smithi151.stdout:2327: expect deleted 2024-09-13T14:30:58.592 INFO:tasks.rados.rados.0.smithi151.stdout:2328: read oid 21 snap -1 2024-09-13T14:30:58.592 INFO:tasks.rados.rados.0.smithi151.stdout:2328: expect (ObjNum 866 snap 305 seq_num 866) 2024-09-13T14:30:58.592 INFO:tasks.rados.rados.0.smithi151.stdout:2329: write oid 8 current snap is 308 2024-09-13T14:30:58.592 INFO:tasks.rados.rados.0.smithi151.stdout:2329: seq_num 869 ranges {605283=655623,1701911=747296} 2024-09-13T14:30:58.598 INFO:tasks.rados.rados.0.smithi151.stdout:2329: writing smithi15186934-8 from 605283 to 1260906 tid 1 2024-09-13T14:30:58.604 INFO:tasks.rados.rados.0.smithi151.stdout:2329: writing smithi15186934-8 from 1701911 to 2449207 tid 2 2024-09-13T14:30:58.604 INFO:tasks.rados.rados.0.smithi151.stdout:2330: write oid 9 current snap is 308 2024-09-13T14:30:58.605 INFO:tasks.rados.rados.0.smithi151.stdout:2330: seq_num 870 ranges {622090=730872,1760393=607307} 2024-09-13T14:30:58.612 INFO:tasks.rados.rados.0.smithi151.stdout:2330: writing smithi15186934-9 from 622090 to 1352962 tid 1 2024-09-13T14:30:58.618 INFO:tasks.rados.rados.0.smithi151.stdout:2330: writing smithi15186934-9 from 1760393 to 2367700 tid 2 2024-09-13T14:30:58.618 INFO:tasks.rados.rados.0.smithi151.stdout:2331: snap_remove snap 306 2024-09-13T14:30:58.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:58 smithi184 ceph-mon[92527]: pgmap v618: 89 pgs: 89 active+clean; 176 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.6 MiB/s rd, 321 KiB/s wr, 5 op/s 2024-09-13T14:30:58.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:58 smithi184 ceph-mon[92527]: osdmap e667: 8 total, 8 up, 8 in 2024-09-13T14:30:58.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:58 smithi151 ceph-mon[104450]: pgmap v618: 89 pgs: 89 active+clean; 176 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.6 MiB/s rd, 321 KiB/s wr, 5 op/s 2024-09-13T14:30:58.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:58 smithi151 ceph-mon[104450]: osdmap e667: 8 total, 8 up, 8 in 2024-09-13T14:30:58.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:58 smithi151 ceph-mon[108966]: pgmap v618: 89 pgs: 89 active+clean; 176 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.6 MiB/s rd, 321 KiB/s wr, 5 op/s 2024-09-13T14:30:58.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:58 smithi151 ceph-mon[108966]: osdmap e667: 8 total, 8 up, 8 in 2024-09-13T14:30:59.584 INFO:tasks.rados.rados.0.smithi151.stdout:2331: done (6 left) 2024-09-13T14:30:59.584 INFO:tasks.rados.rados.0.smithi151.stdout:2332: rollback oid 16 current snap is 308 2024-09-13T14:30:59.585 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 16 to 294 2024-09-13T14:30:59.585 INFO:tasks.rados.rados.0.smithi151.stdout:2323: finishing write tid 2 to smithi15186934-50 2024-09-13T14:30:59.585 INFO:tasks.rados.rados.0.smithi151.stdout:2323: finishing write tid 3 to smithi15186934-50 2024-09-13T14:30:59.585 INFO:tasks.rados.rados.0.smithi151.stdout:2323: finishing write tid 4 to smithi15186934-50 2024-09-13T14:30:59.585 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 50 v 856 (ObjNum 867 snap 308 seq_num 867) dirty exists 2024-09-13T14:30:59.585 INFO:tasks.rados.rados.0.smithi151.stdout:2323: left oid 50 (ObjNum 867 snap 308 seq_num 867) 2024-09-13T14:30:59.585 INFO:tasks.rados.rados.0.smithi151.stdout:2329: finishing write tid 1 to smithi15186934-8 2024-09-13T14:30:59.585 INFO:tasks.rados.rados.0.smithi151.stdout:2328: expect (ObjNum 866 snap 305 seq_num 866) 2024-09-13T14:30:59.667 INFO:tasks.rados.rados.0.smithi151.stdout:2326: expect (ObjNum 774 snap 272 seq_num 774) 2024-09-13T14:30:59.803 INFO:tasks.rados.rados.0.smithi151.stdout:2329: finishing write tid 2 to smithi15186934-8 2024-09-13T14:30:59.803 INFO:tasks.rados.rados.0.smithi151.stdout:2329: finishing write tid 3 to smithi15186934-8 2024-09-13T14:30:59.803 INFO:tasks.rados.rados.0.smithi151.stdout:2329: finishing write tid 4 to smithi15186934-8 2024-09-13T14:30:59.803 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 718 (ObjNum 869 snap 308 seq_num 869) dirty exists 2024-09-13T14:30:59.803 INFO:tasks.rados.rados.0.smithi151.stdout:2329: left oid 8 (ObjNum 869 snap 308 seq_num 869) 2024-09-13T14:30:59.803 INFO:tasks.rados.rados.0.smithi151.stdout:2330: finishing write tid 1 to smithi15186934-9 2024-09-13T14:30:59.803 INFO:tasks.rados.rados.0.smithi151.stdout:2330: finishing write tid 2 to smithi15186934-9 2024-09-13T14:30:59.803 INFO:tasks.rados.rados.0.smithi151.stdout:2330: finishing write tid 3 to smithi15186934-9 2024-09-13T14:30:59.803 INFO:tasks.rados.rados.0.smithi151.stdout:2330: finishing write tid 4 to smithi15186934-9 2024-09-13T14:30:59.803 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 962 (ObjNum 870 snap 308 seq_num 870) dirty exists 2024-09-13T14:30:59.804 INFO:tasks.rados.rados.0.smithi151.stdout:2330: left oid 9 (ObjNum 870 snap 308 seq_num 870) 2024-09-13T14:30:59.804 INFO:tasks.rados.rados.0.smithi151.stdout:2332: finishing rollback tid 1 to smithi15186934-16 2024-09-13T14:30:59.804 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 16 v 525 (ObjNum 129 snap 0 seq_num 3092882360) dirty dne 2024-09-13T14:30:59.804 INFO:tasks.rados.rados.0.smithi151.stdout:2323: done (6 left) 2024-09-13T14:30:59.804 INFO:tasks.rados.rados.0.smithi151.stdout:2326: done (5 left) 2024-09-13T14:30:59.804 INFO:tasks.rados.rados.0.smithi151.stdout:2327: done (4 left) 2024-09-13T14:30:59.804 INFO:tasks.rados.rados.0.smithi151.stdout:2328: done (3 left) 2024-09-13T14:30:59.804 INFO:tasks.rados.rados.0.smithi151.stdout:2329: done (2 left) 2024-09-13T14:30:59.804 INFO:tasks.rados.rados.0.smithi151.stdout:2330: done (1 left) 2024-09-13T14:30:59.804 INFO:tasks.rados.rados.0.smithi151.stdout:2332: done (0 left) 2024-09-13T14:30:59.804 INFO:tasks.rados.rados.0.smithi151.stdout:2333: rollback oid 42 current snap is 308 2024-09-13T14:30:59.804 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 42 to 305 2024-09-13T14:30:59.804 INFO:tasks.rados.rados.0.smithi151.stdout:2334: rollback oid 39 current snap is 308 2024-09-13T14:30:59.804 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 39 to 302 2024-09-13T14:30:59.805 INFO:tasks.rados.rados.0.smithi151.stdout:2335: read oid 48 snap 297 2024-09-13T14:30:59.805 INFO:tasks.rados.rados.0.smithi151.stdout:2335: expect (ObjNum 854 snap 295 seq_num 854) 2024-09-13T14:30:59.805 INFO:tasks.rados.rados.0.smithi151.stdout:2336: read oid 10 snap 295 2024-09-13T14:30:59.805 INFO:tasks.rados.rados.0.smithi151.stdout:2336: expect deleted 2024-09-13T14:30:59.805 INFO:tasks.rados.rados.0.smithi151.stdout:2337: rollback oid 16 current snap is 308 2024-09-13T14:30:59.805 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 16 to 294 2024-09-13T14:30:59.805 INFO:tasks.rados.rados.0.smithi151.stdout:2338: delete oid 18 current snap is 308 2024-09-13T14:30:59.805 INFO:tasks.rados.rados.0.smithi151.stdout:2337: finishing rollback tid 1 to smithi15186934-16 2024-09-13T14:30:59.806 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 16 v 525 (ObjNum 129 snap 0 seq_num 3092882360) dirty dne 2024-09-13T14:30:59.809 INFO:tasks.rados.rados.0.smithi151.stdout:2336: done (5 left) 2024-09-13T14:30:59.810 INFO:tasks.rados.rados.0.smithi151.stdout:2337: done (4 left) 2024-09-13T14:30:59.810 INFO:tasks.rados.rados.0.smithi151.stdout:2338: done (3 left) 2024-09-13T14:30:59.810 INFO:tasks.rados.rados.0.smithi151.stdout:2339: write oid 38 current snap is 308 2024-09-13T14:30:59.810 INFO:tasks.rados.rados.0.smithi151.stdout:2339: seq_num 872 ranges {655704=481370,1761004=494623} 2024-09-13T14:30:59.815 INFO:tasks.rados.rados.0.smithi151.stdout:2339: writing smithi15186934-38 from 655704 to 1137074 tid 1 2024-09-13T14:30:59.822 INFO:tasks.rados.rados.0.smithi151.stdout:2339: writing smithi15186934-38 from 1761004 to 2255627 tid 2 2024-09-13T14:30:59.822 INFO:tasks.rados.rados.0.smithi151.stdout:2340: rollback oid 41 current snap is 308 2024-09-13T14:30:59.822 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 41 to 294 2024-09-13T14:30:59.822 INFO:tasks.rados.rados.0.smithi151.stdout:2333: finishing rollback tid 1 to smithi15186934-42 2024-09-13T14:30:59.822 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 958 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-13T14:30:59.822 INFO:tasks.rados.rados.0.smithi151.stdout:2333: done (4 left) 2024-09-13T14:30:59.822 INFO:tasks.rados.rados.0.smithi151.stdout:2341: snap_remove snap 301 2024-09-13T14:30:59.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:30:59 smithi184 ceph-mon[92527]: osdmap e668: 8 total, 8 up, 8 in 2024-09-13T14:30:59.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:30:59 smithi151 ceph-mon[104450]: osdmap e668: 8 total, 8 up, 8 in 2024-09-13T14:30:59.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:30:59 smithi151 ceph-mon[108966]: osdmap e668: 8 total, 8 up, 8 in 2024-09-13T14:31:00.583 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:31:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:31:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:31:00.589 INFO:tasks.rados.rados.0.smithi151.stdout:2341: done (4 left) 2024-09-13T14:31:00.589 INFO:tasks.rados.rados.0.smithi151.stdout:2342: write oid 22 current snap is 308 2024-09-13T14:31:00.589 INFO:tasks.rados.rados.0.smithi151.stdout:2334: finishing rollback tid 0 to smithi15186934-39 2024-09-13T14:31:00.589 INFO:tasks.rados.rados.0.smithi151.stdout:2334: finishing rollback tid 1 to smithi15186934-39 2024-09-13T14:31:00.589 INFO:tasks.rados.rados.0.smithi151.stdout:2335: expect (ObjNum 854 snap 295 seq_num 854) 2024-09-13T14:31:00.667 INFO:tasks.rados.rados.0.smithi151.stdout:2334: finishing rollback tid 2 to smithi15186934-39 2024-09-13T14:31:00.668 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 860 (ObjNum 844 snap 293 seq_num 844) dirty exists 2024-09-13T14:31:00.668 INFO:tasks.rados.rados.0.smithi151.stdout:2339: finishing write tid 1 to smithi15186934-38 2024-09-13T14:31:00.668 INFO:tasks.rados.rados.0.smithi151.stdout:2340: finishing rollback tid 1 to smithi15186934-41 2024-09-13T14:31:00.668 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-13T14:31:00.668 INFO:tasks.rados.rados.0.smithi151.stdout:2339: finishing write tid 2 to smithi15186934-38 2024-09-13T14:31:00.668 INFO:tasks.rados.rados.0.smithi151.stdout:2339: finishing write tid 3 to smithi15186934-38 2024-09-13T14:31:00.668 INFO:tasks.rados.rados.0.smithi151.stdout:2339: finishing write tid 4 to smithi15186934-38 2024-09-13T14:31:00.668 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 866 (ObjNum 872 snap 308 seq_num 872) dirty exists 2024-09-13T14:31:00.668 INFO:tasks.rados.rados.0.smithi151.stdout:2339: left oid 38 (ObjNum 872 snap 308 seq_num 872) 2024-09-13T14:31:00.668 INFO:tasks.rados.rados.0.smithi151.stdout:2342: seq_num 873 ranges {672511=556619,1819486=754634} 2024-09-13T14:31:00.673 INFO:tasks.rados.rados.0.smithi151.stdout:2342: writing smithi15186934-22 from 672511 to 1229130 tid 1 2024-09-13T14:31:00.681 INFO:tasks.rados.rados.0.smithi151.stdout:2342: writing smithi15186934-22 from 1819486 to 2574120 tid 2 2024-09-13T14:31:00.681 INFO:tasks.rados.rados.0.smithi151.stdout:2334: done (4 left) 2024-09-13T14:31:00.681 INFO:tasks.rados.rados.0.smithi151.stdout:2335: done (3 left) 2024-09-13T14:31:00.681 INFO:tasks.rados.rados.0.smithi151.stdout:2339: done (2 left) 2024-09-13T14:31:00.681 INFO:tasks.rados.rados.0.smithi151.stdout:2340: done (1 left) 2024-09-13T14:31:00.681 INFO:tasks.rados.rados.0.smithi151.stdout:2343: snap_remove snap 300 2024-09-13T14:31:00.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:00 smithi184 ceph-mon[92527]: pgmap v621: 89 pgs: 89 active+clean; 176 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.6 MiB/s rd, 321 KiB/s wr, 5 op/s 2024-09-13T14:31:00.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:00 smithi184 ceph-mon[92527]: osdmap e669: 8 total, 8 up, 8 in 2024-09-13T14:31:00.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:00 smithi151 ceph-mon[104450]: pgmap v621: 89 pgs: 89 active+clean; 176 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.6 MiB/s rd, 321 KiB/s wr, 5 op/s 2024-09-13T14:31:00.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:00 smithi151 ceph-mon[104450]: osdmap e669: 8 total, 8 up, 8 in 2024-09-13T14:31:00.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:00 smithi151 ceph-mon[108966]: pgmap v621: 89 pgs: 89 active+clean; 176 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.6 MiB/s rd, 321 KiB/s wr, 5 op/s 2024-09-13T14:31:00.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:00 smithi151 ceph-mon[108966]: osdmap e669: 8 total, 8 up, 8 in 2024-09-13T14:31:01.600 INFO:tasks.rados.rados.0.smithi151.stdout:2343: done (1 left) 2024-09-13T14:31:01.600 INFO:tasks.rados.rados.0.smithi151.stdout:2344: snap_create 2024-09-13T14:31:01.601 INFO:tasks.rados.rados.0.smithi151.stdout:2342: finishing write tid 1 to smithi15186934-22 2024-09-13T14:31:01.601 INFO:tasks.rados.rados.0.smithi151.stdout:2342: finishing write tid 2 to smithi15186934-22 2024-09-13T14:31:01.601 INFO:tasks.rados.rados.0.smithi151.stdout:2342: finishing write tid 3 to smithi15186934-22 2024-09-13T14:31:01.601 INFO:tasks.rados.rados.0.smithi151.stdout:2342: finishing write tid 4 to smithi15186934-22 2024-09-13T14:31:01.601 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 22 v 487 (ObjNum 873 snap 308 seq_num 873) dirty exists 2024-09-13T14:31:01.601 INFO:tasks.rados.rados.0.smithi151.stdout:2342: left oid 22 (ObjNum 873 snap 308 seq_num 873) 2024-09-13T14:31:01.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:01 smithi184 ceph-mon[92527]: osdmap e670: 8 total, 8 up, 8 in 2024-09-13T14:31:01.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:01 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:01.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:01 smithi151 ceph-mon[104450]: osdmap e670: 8 total, 8 up, 8 in 2024-09-13T14:31:01.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:01.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:01 smithi151 ceph-mon[108966]: osdmap e670: 8 total, 8 up, 8 in 2024-09-13T14:31:01.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:01 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:02.599 INFO:tasks.rados.rados.0.smithi151.stdout:2342: done (1 left) 2024-09-13T14:31:02.600 INFO:tasks.rados.rados.0.smithi151.stdout:2344: done (0 left) 2024-09-13T14:31:02.600 INFO:tasks.rados.rados.0.smithi151.stdout:2345: delete oid 31 current snap is 309 2024-09-13T14:31:02.609 INFO:tasks.rados.rados.0.smithi151.stdout:2345: done (0 left) 2024-09-13T14:31:02.609 INFO:tasks.rados.rados.0.smithi151.stdout:2346: write oid 40 current snap is 309 2024-09-13T14:31:02.610 INFO:tasks.rados.rados.0.smithi151.stdout:2346: seq_num 875 ranges {706125=423470,1652803=758303} 2024-09-13T14:31:02.615 INFO:tasks.rados.rados.0.smithi151.stdout:2346: writing smithi15186934-40 from 706125 to 1129595 tid 1 2024-09-13T14:31:02.623 INFO:tasks.rados.rados.0.smithi151.stdout:2346: writing smithi15186934-40 from 1652803 to 2411106 tid 2 2024-09-13T14:31:02.623 INFO:tasks.rados.rados.0.smithi151.stdout:2347: read oid 6 snap -1 2024-09-13T14:31:02.623 INFO:tasks.rados.rados.0.smithi151.stdout:2347: expect (ObjNum 825 snap 286 seq_num 825) 2024-09-13T14:31:02.623 INFO:tasks.rados.rados.0.smithi151.stdout:2348: write oid 34 current snap is 309 2024-09-13T14:31:02.623 INFO:tasks.rados.rados.0.smithi151.stdout:2346: finishing write tid 1 to smithi15186934-40 2024-09-13T14:31:02.623 INFO:tasks.rados.rados.0.smithi151.stdout:2348: seq_num 876 ranges {722932=498719,1994932=501961} 2024-09-13T14:31:02.629 INFO:tasks.rados.rados.0.smithi151.stdout:2348: writing smithi15186934-34 from 722932 to 1221651 tid 1 2024-09-13T14:31:02.637 INFO:tasks.rados.rados.0.smithi151.stdout:2348: writing smithi15186934-34 from 1994932 to 2496893 tid 2 2024-09-13T14:31:02.637 INFO:tasks.rados.rados.0.smithi151.stdout:2349: read oid 29 snap 295 2024-09-13T14:31:02.637 INFO:tasks.rados.rados.0.smithi151.stdout:2349: expect (ObjNum 852 snap 293 seq_num 852) 2024-09-13T14:31:02.637 INFO:tasks.rados.rados.0.smithi151.stdout:2350: rollback oid 30 current snap is 309 2024-09-13T14:31:02.637 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 30 to 305 2024-09-13T14:31:02.637 INFO:tasks.rados.rados.0.smithi151.stdout:2346: finishing write tid 2 to smithi15186934-40 2024-09-13T14:31:02.638 INFO:tasks.rados.rados.0.smithi151.stdout:2347: expect (ObjNum 825 snap 286 seq_num 825) 2024-09-13T14:31:02.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:02 smithi184 ceph-mon[92527]: pgmap v624: 89 pgs: 89 active+clean; 180 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 670 KiB/s wr, 2 op/s 2024-09-13T14:31:02.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:02 smithi184 ceph-mon[92527]: osdmap e671: 8 total, 8 up, 8 in 2024-09-13T14:31:02.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:02 smithi151 ceph-mon[104450]: pgmap v624: 89 pgs: 89 active+clean; 180 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 670 KiB/s wr, 2 op/s 2024-09-13T14:31:02.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:02 smithi151 ceph-mon[104450]: osdmap e671: 8 total, 8 up, 8 in 2024-09-13T14:31:02.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:02 smithi151 ceph-mon[108966]: pgmap v624: 89 pgs: 89 active+clean; 180 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 670 KiB/s wr, 2 op/s 2024-09-13T14:31:02.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:02 smithi151 ceph-mon[108966]: osdmap e671: 8 total, 8 up, 8 in 2024-09-13T14:31:02.897 INFO:tasks.rados.rados.0.smithi151.stdout:2346: finishing write tid 3 to smithi15186934-40 2024-09-13T14:31:02.897 INFO:tasks.rados.rados.0.smithi151.stdout:2346: finishing write tid 4 to smithi15186934-40 2024-09-13T14:31:02.897 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 40 v 847 (ObjNum 875 snap 309 seq_num 875) dirty exists 2024-09-13T14:31:02.897 INFO:tasks.rados.rados.0.smithi151.stdout:2346: left oid 40 (ObjNum 875 snap 309 seq_num 875) 2024-09-13T14:31:02.897 INFO:tasks.rados.rados.0.smithi151.stdout:2348: finishing write tid 1 to smithi15186934-34 2024-09-13T14:31:02.897 INFO:tasks.rados.rados.0.smithi151.stdout:2346: done (4 left) 2024-09-13T14:31:02.897 INFO:tasks.rados.rados.0.smithi151.stdout:2347: done (3 left) 2024-09-13T14:31:02.897 INFO:tasks.rados.rados.0.smithi151.stdout:2351: delete oid 27 current snap is 309 2024-09-13T14:31:02.897 INFO:tasks.rados.rados.0.smithi151.stdout:2349: expect (ObjNum 852 snap 293 seq_num 852) 2024-09-13T14:31:02.974 INFO:tasks.rados.rados.0.smithi151.stdout:2348: finishing write tid 2 to smithi15186934-34 2024-09-13T14:31:02.974 INFO:tasks.rados.rados.0.smithi151.stdout:2348: finishing write tid 3 to smithi15186934-34 2024-09-13T14:31:02.974 INFO:tasks.rados.rados.0.smithi151.stdout:2348: finishing write tid 4 to smithi15186934-34 2024-09-13T14:31:02.974 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 851 (ObjNum 876 snap 309 seq_num 876) dirty exists 2024-09-13T14:31:02.974 INFO:tasks.rados.rados.0.smithi151.stdout:2348: left oid 34 (ObjNum 876 snap 309 seq_num 876) 2024-09-13T14:31:02.974 INFO:tasks.rados.rados.0.smithi151.stdout:2350: finishing rollback tid 0 to smithi15186934-30 2024-09-13T14:31:02.975 INFO:tasks.rados.rados.0.smithi151.stdout:2350: finishing rollback tid 1 to smithi15186934-30 2024-09-13T14:31:02.975 INFO:tasks.rados.rados.0.smithi151.stdout:2350: finishing rollback tid 2 to smithi15186934-30 2024-09-13T14:31:02.975 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 30 v 855 (ObjNum 707 snap 237 seq_num 707) dirty exists 2024-09-13T14:31:02.975 INFO:tasks.rados.rados.0.smithi151.stdout:2348: done (3 left) 2024-09-13T14:31:02.975 INFO:tasks.rados.rados.0.smithi151.stdout:2349: done (2 left) 2024-09-13T14:31:02.975 INFO:tasks.rados.rados.0.smithi151.stdout:2350: done (1 left) 2024-09-13T14:31:02.975 INFO:tasks.rados.rados.0.smithi151.stdout:2351: done (0 left) 2024-09-13T14:31:02.975 INFO:tasks.rados.rados.0.smithi151.stdout:2352: snap_create 2024-09-13T14:31:03.608 INFO:tasks.rados.rados.0.smithi151.stdout:2352: done (0 left) 2024-09-13T14:31:03.608 INFO:tasks.rados.rados.0.smithi151.stdout:2353: read oid 3 snap -1 2024-09-13T14:31:03.608 INFO:tasks.rados.rados.0.smithi151.stdout:2353: expect (ObjNum 780 snap 273 seq_num 780) 2024-09-13T14:31:03.609 INFO:tasks.rados.rados.0.smithi151.stdout:2354: snap_remove snap 289 2024-09-13T14:31:03.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:03 smithi151 ceph-mon[104450]: osdmap e672: 8 total, 8 up, 8 in 2024-09-13T14:31:03.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:03 smithi151 ceph-mon[108966]: osdmap e672: 8 total, 8 up, 8 in 2024-09-13T14:31:04.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:03 smithi184 ceph-mon[92527]: osdmap e672: 8 total, 8 up, 8 in 2024-09-13T14:31:04.548 INFO:tasks.rados.rados.0.smithi151.stdout:2354: done (1 left) 2024-09-13T14:31:04.549 INFO:tasks.rados.rados.0.smithi151.stdout:2355: read oid 10 snap 309 2024-09-13T14:31:04.549 INFO:tasks.rados.rados.0.smithi151.stdout:2355: expect (ObjNum 862 snap 297 seq_num 862) 2024-09-13T14:31:04.549 INFO:tasks.rados.rados.0.smithi151.stdout:2356: snap_remove snap 295 2024-09-13T14:31:04.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:04 smithi151 ceph-mon[104450]: pgmap v627: 89 pgs: 89 active+clean; 181 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 1.5 MiB/s wr, 13 op/s 2024-09-13T14:31:04.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:04 smithi151 ceph-mon[104450]: osdmap e673: 8 total, 8 up, 8 in 2024-09-13T14:31:04.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:04 smithi151 ceph-mon[104450]: osdmap e674: 8 total, 8 up, 8 in 2024-09-13T14:31:04.860 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:31:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:31:04] "GET /metrics HTTP/1.1" 200 36512 "" "Prometheus/2.51.0" 2024-09-13T14:31:04.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:04 smithi151 ceph-mon[108966]: pgmap v627: 89 pgs: 89 active+clean; 181 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 1.5 MiB/s wr, 13 op/s 2024-09-13T14:31:04.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:04 smithi151 ceph-mon[108966]: osdmap e673: 8 total, 8 up, 8 in 2024-09-13T14:31:04.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:04 smithi151 ceph-mon[108966]: osdmap e674: 8 total, 8 up, 8 in 2024-09-13T14:31:05.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:04 smithi184 ceph-mon[92527]: pgmap v627: 89 pgs: 89 active+clean; 181 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 1.5 MiB/s wr, 13 op/s 2024-09-13T14:31:05.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:04 smithi184 ceph-mon[92527]: osdmap e673: 8 total, 8 up, 8 in 2024-09-13T14:31:05.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:04 smithi184 ceph-mon[92527]: osdmap e674: 8 total, 8 up, 8 in 2024-09-13T14:31:05.551 INFO:tasks.rados.rados.0.smithi151.stdout:2356: done (2 left) 2024-09-13T14:31:05.551 INFO:tasks.rados.rados.0.smithi151.stdout:2357: snap_remove snap 302 2024-09-13T14:31:06.552 INFO:tasks.rados.rados.0.smithi151.stdout:2357: done (2 left) 2024-09-13T14:31:06.552 INFO:tasks.rados.rados.0.smithi151.stdout:2358: snap_create 2024-09-13T14:31:06.552 INFO:tasks.rados.rados.0.smithi151.stdout:2353: expect (ObjNum 780 snap 273 seq_num 780) 2024-09-13T14:31:06.718 INFO:tasks.rados.rados.0.smithi151.stdout:2355: expect (ObjNum 862 snap 297 seq_num 862) 2024-09-13T14:31:06.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:06 smithi184 ceph-mon[92527]: pgmap v630: 89 pgs: 89 active+clean; 181 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 864 KiB/s wr, 10 op/s 2024-09-13T14:31:06.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:06 smithi184 ceph-mon[92527]: osdmap e675: 8 total, 8 up, 8 in 2024-09-13T14:31:06.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:06 smithi151 ceph-mon[104450]: pgmap v630: 89 pgs: 89 active+clean; 181 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 864 KiB/s wr, 10 op/s 2024-09-13T14:31:06.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:06 smithi151 ceph-mon[104450]: osdmap e675: 8 total, 8 up, 8 in 2024-09-13T14:31:06.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:06 smithi151 ceph-mon[108966]: pgmap v630: 89 pgs: 89 active+clean; 181 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 864 KiB/s wr, 10 op/s 2024-09-13T14:31:06.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:06 smithi151 ceph-mon[108966]: osdmap e675: 8 total, 8 up, 8 in 2024-09-13T14:31:07.557 INFO:tasks.rados.rados.0.smithi151.stdout:2353: done (2 left) 2024-09-13T14:31:07.557 INFO:tasks.rados.rados.0.smithi151.stdout:2355: done (1 left) 2024-09-13T14:31:07.557 INFO:tasks.rados.rados.0.smithi151.stdout:2358: done (0 left) 2024-09-13T14:31:07.558 INFO:tasks.rados.rados.0.smithi151.stdout:2359: write oid 41 current snap is 311 2024-09-13T14:31:07.558 INFO:tasks.rados.rados.0.smithi151.stdout:2359: seq_num 878 ranges {756546=649217,2111896=505630} 2024-09-13T14:31:07.564 INFO:tasks.rados.rados.0.smithi151.stdout:2359: writing smithi15186934-41 from 756546 to 1405763 tid 1 2024-09-13T14:31:07.572 INFO:tasks.rados.rados.0.smithi151.stdout:2359: writing smithi15186934-41 from 2111896 to 2617526 tid 2 2024-09-13T14:31:07.572 INFO:tasks.rados.rados.0.smithi151.stdout:2360: write oid 32 current snap is 311 2024-09-13T14:31:07.572 INFO:tasks.rados.rados.0.smithi151.stdout:2360: seq_num 879 ranges {773353=724466,2170378=602975} 2024-09-13T14:31:07.580 INFO:tasks.rados.rados.0.smithi151.stdout:2360: writing smithi15186934-32 from 773353 to 1497819 tid 1 2024-09-13T14:31:07.588 INFO:tasks.rados.rados.0.smithi151.stdout:2360: writing smithi15186934-32 from 2170378 to 2773353 tid 2 2024-09-13T14:31:07.588 INFO:tasks.rados.rados.0.smithi151.stdout:2361: write oid 15 current snap is 311 2024-09-13T14:31:07.588 INFO:tasks.rados.rados.0.smithi151.stdout:2359: finishing write tid 1 to smithi15186934-41 2024-09-13T14:31:07.588 INFO:tasks.rados.rados.0.smithi151.stdout:2359: finishing write tid 2 to smithi15186934-41 2024-09-13T14:31:07.588 INFO:tasks.rados.rados.0.smithi151.stdout:2361: seq_num 880 ranges {790160=799715,2112507=509299} 2024-09-13T14:31:07.597 INFO:tasks.rados.rados.0.smithi151.stdout:2361: writing smithi15186934-15 from 790160 to 1589875 tid 1 2024-09-13T14:31:07.603 INFO:tasks.rados.rados.0.smithi151.stdout:2361: writing smithi15186934-15 from 2112507 to 2621806 tid 2 2024-09-13T14:31:07.603 INFO:tasks.rados.rados.0.smithi151.stdout:2362: read oid 38 snap -1 2024-09-13T14:31:07.603 INFO:tasks.rados.rados.0.smithi151.stdout:2362: expect (ObjNum 872 snap 308 seq_num 872) 2024-09-13T14:31:07.604 INFO:tasks.rados.rados.0.smithi151.stdout:2363: snap_create 2024-09-13T14:31:07.604 INFO:tasks.rados.rados.0.smithi151.stdout:2359: finishing write tid 3 to smithi15186934-41 2024-09-13T14:31:07.604 INFO:tasks.rados.rados.0.smithi151.stdout:2359: finishing write tid 4 to smithi15186934-41 2024-09-13T14:31:07.604 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 160 (ObjNum 878 snap 311 seq_num 878) dirty exists 2024-09-13T14:31:07.604 INFO:tasks.rados.rados.0.smithi151.stdout:2359: left oid 41 (ObjNum 878 snap 311 seq_num 878) 2024-09-13T14:31:07.604 INFO:tasks.rados.rados.0.smithi151.stdout:2360: finishing write tid 1 to smithi15186934-32 2024-09-13T14:31:07.604 INFO:tasks.rados.rados.0.smithi151.stdout:2360: finishing write tid 2 to smithi15186934-32 2024-09-13T14:31:07.604 INFO:tasks.rados.rados.0.smithi151.stdout:2360: finishing write tid 3 to smithi15186934-32 2024-09-13T14:31:07.604 INFO:tasks.rados.rados.0.smithi151.stdout:2360: finishing write tid 4 to smithi15186934-32 2024-09-13T14:31:07.604 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 32 v 968 (ObjNum 879 snap 311 seq_num 879) dirty exists 2024-09-13T14:31:07.604 INFO:tasks.rados.rados.0.smithi151.stdout:2360: left oid 32 (ObjNum 879 snap 311 seq_num 879) 2024-09-13T14:31:07.604 INFO:tasks.rados.rados.0.smithi151.stdout:2361: finishing write tid 1 to smithi15186934-15 2024-09-13T14:31:07.616 INFO:tasks.rados.rados.0.smithi151.stdout:2362: expect (ObjNum 872 snap 308 seq_num 872) 2024-09-13T14:31:07.764 INFO:tasks.rados.rados.0.smithi151.stdout:2361: finishing write tid 2 to smithi15186934-15 2024-09-13T14:31:07.764 INFO:tasks.rados.rados.0.smithi151.stdout:2361: finishing write tid 3 to smithi15186934-15 2024-09-13T14:31:07.764 INFO:tasks.rados.rados.0.smithi151.stdout:2361: finishing write tid 4 to smithi15186934-15 2024-09-13T14:31:07.764 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 15 v 976 (ObjNum 880 snap 311 seq_num 880) dirty exists 2024-09-13T14:31:07.764 INFO:tasks.rados.rados.0.smithi151.stdout:2361: left oid 15 (ObjNum 880 snap 311 seq_num 880) 2024-09-13T14:31:07.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:07 smithi184 ceph-mon[92527]: osdmap e676: 8 total, 8 up, 8 in 2024-09-13T14:31:07.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:07 smithi151 ceph-mon[104450]: osdmap e676: 8 total, 8 up, 8 in 2024-09-13T14:31:07.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:07 smithi151 ceph-mon[108966]: osdmap e676: 8 total, 8 up, 8 in 2024-09-13T14:31:08.574 INFO:tasks.rados.rados.0.smithi151.stdout:2359: done (4 left) 2024-09-13T14:31:08.574 INFO:tasks.rados.rados.0.smithi151.stdout:2360: done (3 left) 2024-09-13T14:31:08.574 INFO:tasks.rados.rados.0.smithi151.stdout:2361: done (2 left) 2024-09-13T14:31:08.574 INFO:tasks.rados.rados.0.smithi151.stdout:2362: done (1 left) 2024-09-13T14:31:08.574 INFO:tasks.rados.rados.0.smithi151.stdout:2363: done (0 left) 2024-09-13T14:31:08.575 INFO:tasks.rados.rados.0.smithi151.stdout:2364: write oid 34 current snap is 312 2024-09-13T14:31:08.575 INFO:tasks.rados.rados.0.smithi151.stdout:2364: seq_num 881 ranges {406967=474964,1370989=769310,2677186=129781} 2024-09-13T14:31:08.579 INFO:tasks.rados.rados.0.smithi151.stdout:2364: writing smithi15186934-34 from 406967 to 881931 tid 1 2024-09-13T14:31:08.587 INFO:tasks.rados.rados.0.smithi151.stdout:2364: writing smithi15186934-34 from 1370989 to 2140299 tid 2 2024-09-13T14:31:08.590 INFO:tasks.rados.rados.0.smithi151.stdout:2364: writing smithi15186934-34 from 2677186 to 2806967 tid 3 2024-09-13T14:31:08.590 INFO:tasks.rados.rados.0.smithi151.stdout:2365: write oid 13 current snap is 312 2024-09-13T14:31:08.590 INFO:tasks.rados.rados.0.smithi151.stdout:2365: seq_num 882 ranges {423774=666566,1545824=512968,2704609=119165} 2024-09-13T14:31:08.596 INFO:tasks.rados.rados.0.smithi151.stdout:2365: writing smithi15186934-13 from 423774 to 1090340 tid 1 2024-09-13T14:31:08.601 INFO:tasks.rados.rados.0.smithi151.stdout:2365: writing smithi15186934-13 from 1545824 to 2058792 tid 2 2024-09-13T14:31:08.606 INFO:tasks.rados.rados.0.smithi151.stdout:2365: writing smithi15186934-13 from 2704609 to 2823774 tid 3 2024-09-13T14:31:08.606 INFO:tasks.rados.rados.0.smithi151.stdout:2366: write oid 36 current snap is 312 2024-09-13T14:31:08.606 INFO:tasks.rados.rados.0.smithi151.stdout:2364: finishing write tid 1 to smithi15186934-34 2024-09-13T14:31:08.606 INFO:tasks.rados.rados.0.smithi151.stdout:2364: finishing write tid 2 to smithi15186934-34 2024-09-13T14:31:08.606 INFO:tasks.rados.rados.0.smithi151.stdout:2364: finishing write tid 3 to smithi15186934-34 2024-09-13T14:31:08.606 INFO:tasks.rados.rados.0.smithi151.stdout:2366: seq_num 883 ranges {440581=741815,1604306=656626,2732032=108549} 2024-09-13T14:31:08.612 INFO:tasks.rados.rados.0.smithi151.stdout:2366: writing smithi15186934-36 from 440581 to 1182396 tid 1 2024-09-13T14:31:08.618 INFO:tasks.rados.rados.0.smithi151.stdout:2366: writing smithi15186934-36 from 1604306 to 2260932 tid 2 2024-09-13T14:31:08.623 INFO:tasks.rados.rados.0.smithi151.stdout:2366: writing smithi15186934-36 from 2732032 to 2840581 tid 3 2024-09-13T14:31:08.623 INFO:tasks.rados.rados.0.smithi151.stdout:2367: read oid 15 snap -1 2024-09-13T14:31:08.623 INFO:tasks.rados.rados.0.smithi151.stdout:2367: expect (ObjNum 880 snap 311 seq_num 880) 2024-09-13T14:31:08.623 INFO:tasks.rados.rados.0.smithi151.stdout:2368: write oid 17 current snap is 312 2024-09-13T14:31:08.623 INFO:tasks.rados.rados.0.smithi151.stdout:2364: finishing write tid 4 to smithi15186934-34 2024-09-13T14:31:08.623 INFO:tasks.rados.rados.0.smithi151.stdout:2364: finishing write tid 5 to smithi15186934-34 2024-09-13T14:31:08.623 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 878 (ObjNum 881 snap 312 seq_num 881) dirty exists 2024-09-13T14:31:08.623 INFO:tasks.rados.rados.0.smithi151.stdout:2364: left oid 34 (ObjNum 881 snap 312 seq_num 881) 2024-09-13T14:31:08.624 INFO:tasks.rados.rados.0.smithi151.stdout:2365: finishing write tid 1 to smithi15186934-13 2024-09-13T14:31:08.624 INFO:tasks.rados.rados.0.smithi151.stdout:2368: seq_num 884 ranges {457388=417064,1546435=516637,2643102=214286} 2024-09-13T14:31:08.627 INFO:tasks.rados.rados.0.smithi151.stdout:2368: writing smithi15186934-17 from 457388 to 874452 tid 1 2024-09-13T14:31:08.633 INFO:tasks.rados.rados.0.smithi151.stdout:2368: writing smithi15186934-17 from 1546435 to 2063072 tid 2 2024-09-13T14:31:08.638 INFO:tasks.rados.rados.0.smithi151.stdout:2368: writing smithi15186934-17 from 2643102 to 2857388 tid 3 2024-09-13T14:31:08.638 INFO:tasks.rados.rados.0.smithi151.stdout:2364: done (4 left) 2024-09-13T14:31:08.638 INFO:tasks.rados.rados.0.smithi151.stdout:2369: write oid 8 current snap is 312 2024-09-13T14:31:08.638 INFO:tasks.rados.rados.0.smithi151.stdout:2369: seq_num 885 ranges {474195=492313,1604917=660295,2670525=203670} 2024-09-13T14:31:08.643 INFO:tasks.rados.rados.0.smithi151.stdout:2369: writing smithi15186934-8 from 474195 to 966508 tid 1 2024-09-13T14:31:08.650 INFO:tasks.rados.rados.0.smithi151.stdout:2369: writing smithi15186934-8 from 1604917 to 2265212 tid 2 2024-09-13T14:31:08.654 INFO:tasks.rados.rados.0.smithi151.stdout:2369: writing smithi15186934-8 from 2670525 to 2874195 tid 3 2024-09-13T14:31:08.654 INFO:tasks.rados.rados.0.smithi151.stdout:2370: snap_create 2024-09-13T14:31:08.654 INFO:tasks.rados.rados.0.smithi151.stdout:2365: finishing write tid 2 to smithi15186934-13 2024-09-13T14:31:08.654 INFO:tasks.rados.rados.0.smithi151.stdout:2365: finishing write tid 3 to smithi15186934-13 2024-09-13T14:31:08.654 INFO:tasks.rados.rados.0.smithi151.stdout:2365: finishing write tid 4 to smithi15186934-13 2024-09-13T14:31:08.655 INFO:tasks.rados.rados.0.smithi151.stdout:2365: finishing write tid 5 to smithi15186934-13 2024-09-13T14:31:08.655 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 13 v 926 (ObjNum 882 snap 312 seq_num 882) dirty exists 2024-09-13T14:31:08.655 INFO:tasks.rados.rados.0.smithi151.stdout:2365: left oid 13 (ObjNum 882 snap 312 seq_num 882) 2024-09-13T14:31:08.655 INFO:tasks.rados.rados.0.smithi151.stdout:2366: finishing write tid 1 to smithi15186934-36 2024-09-13T14:31:08.655 INFO:tasks.rados.rados.0.smithi151.stdout:2366: finishing write tid 2 to smithi15186934-36 2024-09-13T14:31:08.655 INFO:tasks.rados.rados.0.smithi151.stdout:2366: finishing write tid 3 to smithi15186934-36 2024-09-13T14:31:08.655 INFO:tasks.rados.rados.0.smithi151.stdout:2366: finishing write tid 4 to smithi15186934-36 2024-09-13T14:31:08.656 INFO:tasks.rados.rados.0.smithi151.stdout:2366: finishing write tid 5 to smithi15186934-36 2024-09-13T14:31:08.656 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 36 v 561 (ObjNum 883 snap 312 seq_num 883) dirty exists 2024-09-13T14:31:08.656 INFO:tasks.rados.rados.0.smithi151.stdout:2366: left oid 36 (ObjNum 883 snap 312 seq_num 883) 2024-09-13T14:31:08.656 INFO:tasks.rados.rados.0.smithi151.stdout:2367: expect (ObjNum 880 snap 311 seq_num 880) 2024-09-13T14:31:08.745 INFO:tasks.rados.rados.0.smithi151.stdout:2368: finishing write tid 1 to smithi15186934-17 2024-09-13T14:31:08.745 INFO:tasks.rados.rados.0.smithi151.stdout:2368: finishing write tid 2 to smithi15186934-17 2024-09-13T14:31:08.745 INFO:tasks.rados.rados.0.smithi151.stdout:2368: finishing write tid 3 to smithi15186934-17 2024-09-13T14:31:08.745 INFO:tasks.rados.rados.0.smithi151.stdout:2368: finishing write tid 4 to smithi15186934-17 2024-09-13T14:31:08.745 INFO:tasks.rados.rados.0.smithi151.stdout:2368: finishing write tid 5 to smithi15186934-17 2024-09-13T14:31:08.745 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 1377 (ObjNum 884 snap 312 seq_num 884) dirty exists 2024-09-13T14:31:08.745 INFO:tasks.rados.rados.0.smithi151.stdout:2368: left oid 17 (ObjNum 884 snap 312 seq_num 884) 2024-09-13T14:31:08.745 INFO:tasks.rados.rados.0.smithi151.stdout:2369: finishing write tid 1 to smithi15186934-8 2024-09-13T14:31:08.745 INFO:tasks.rados.rados.0.smithi151.stdout:2369: finishing write tid 2 to smithi15186934-8 2024-09-13T14:31:08.745 INFO:tasks.rados.rados.0.smithi151.stdout:2369: finishing write tid 3 to smithi15186934-8 2024-09-13T14:31:08.745 INFO:tasks.rados.rados.0.smithi151.stdout:2369: finishing write tid 4 to smithi15186934-8 2024-09-13T14:31:08.746 INFO:tasks.rados.rados.0.smithi151.stdout:2369: finishing write tid 5 to smithi15186934-8 2024-09-13T14:31:08.746 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 735 (ObjNum 885 snap 312 seq_num 885) dirty exists 2024-09-13T14:31:08.746 INFO:tasks.rados.rados.0.smithi151.stdout:2369: left oid 8 (ObjNum 885 snap 312 seq_num 885) 2024-09-13T14:31:08.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:08 smithi184 ceph-mon[92527]: pgmap v633: 89 pgs: 89 active+clean; 183 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 8.0 MiB/s rd, 533 KiB/s wr, 9 op/s 2024-09-13T14:31:08.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:08 smithi184 ceph-mon[92527]: osdmap e677: 8 total, 8 up, 8 in 2024-09-13T14:31:08.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:08 smithi151 ceph-mon[108966]: pgmap v633: 89 pgs: 89 active+clean; 183 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 8.0 MiB/s rd, 533 KiB/s wr, 9 op/s 2024-09-13T14:31:08.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:08 smithi151 ceph-mon[108966]: osdmap e677: 8 total, 8 up, 8 in 2024-09-13T14:31:08.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:08 smithi151 ceph-mon[104450]: pgmap v633: 89 pgs: 89 active+clean; 183 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 8.0 MiB/s rd, 533 KiB/s wr, 9 op/s 2024-09-13T14:31:08.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:08 smithi151 ceph-mon[104450]: osdmap e677: 8 total, 8 up, 8 in 2024-09-13T14:31:09.551 INFO:tasks.rados.rados.0.smithi151.stdout:2365: done (5 left) 2024-09-13T14:31:09.552 INFO:tasks.rados.rados.0.smithi151.stdout:2366: done (4 left) 2024-09-13T14:31:09.552 INFO:tasks.rados.rados.0.smithi151.stdout:2367: done (3 left) 2024-09-13T14:31:09.552 INFO:tasks.rados.rados.0.smithi151.stdout:2368: done (2 left) 2024-09-13T14:31:09.552 INFO:tasks.rados.rados.0.smithi151.stdout:2369: done (1 left) 2024-09-13T14:31:09.552 INFO:tasks.rados.rados.0.smithi151.stdout:2370: done (0 left) 2024-09-13T14:31:09.552 INFO:tasks.rados.rados.0.smithi151.stdout:2371: read oid 25 snap 305 2024-09-13T14:31:09.552 INFO:tasks.rados.rados.0.smithi151.stdout:2371: expect (ObjNum 822 snap 286 seq_num 822) 2024-09-13T14:31:09.552 INFO:tasks.rados.rados.0.smithi151.stdout:2372: write oid 39 current snap is 313 2024-09-13T14:31:09.552 INFO:tasks.rados.rados.0.smithi151.stdout:2372: seq_num 886 ranges {491002=567562,1663399=520306,2814301=76701} 2024-09-13T14:31:09.558 INFO:tasks.rados.rados.0.smithi151.stdout:2372: writing smithi15186934-39 from 491002 to 1058564 tid 1 2024-09-13T14:31:09.565 INFO:tasks.rados.rados.0.smithi151.stdout:2372: writing smithi15186934-39 from 1663399 to 2183705 tid 2 2024-09-13T14:31:09.569 INFO:tasks.rados.rados.0.smithi151.stdout:2372: writing smithi15186934-39 from 2814301 to 2891002 tid 3 2024-09-13T14:31:09.569 INFO:tasks.rados.rados.0.smithi151.stdout:2373: read oid 42 snap -1 2024-09-13T14:31:09.569 INFO:tasks.rados.rados.0.smithi151.stdout:2373: expect deleted 2024-09-13T14:31:09.569 INFO:tasks.rados.rados.0.smithi151.stdout:2374: write oid 33 current snap is 313 2024-09-13T14:31:09.569 INFO:tasks.rados.rados.0.smithi151.stdout:2374: seq_num 887 ranges {507809=642811,1721881=663964} 2024-09-13T14:31:09.576 INFO:tasks.rados.rados.0.smithi151.stdout:2374: writing smithi15186934-33 from 507809 to 1150620 tid 1 2024-09-13T14:31:09.584 INFO:tasks.rados.rados.0.smithi151.stdout:2374: writing smithi15186934-33 from 1721881 to 2385845 tid 2 2024-09-13T14:31:09.584 INFO:tasks.rados.rados.0.smithi151.stdout:2375: write oid 23 current snap is 313 2024-09-13T14:31:09.584 INFO:tasks.rados.rados.0.smithi151.stdout:2375: seq_num 888 ranges {524616=718060,1664010=523975,2752794=171822} 2024-09-13T14:31:09.592 INFO:tasks.rados.rados.0.smithi151.stdout:2375: writing smithi15186934-23 from 524616 to 1242676 tid 1 2024-09-13T14:31:09.599 INFO:tasks.rados.rados.0.smithi151.stdout:2375: writing smithi15186934-23 from 1664010 to 2187985 tid 2 2024-09-13T14:31:09.603 INFO:tasks.rados.rados.0.smithi151.stdout:2375: writing smithi15186934-23 from 2752794 to 2924616 tid 3 2024-09-13T14:31:09.603 INFO:tasks.rados.rados.0.smithi151.stdout:2376: read oid 8 snap -1 2024-09-13T14:31:09.603 INFO:tasks.rados.rados.0.smithi151.stdout:2376: expect (ObjNum 885 snap 312 seq_num 885) 2024-09-13T14:31:09.603 INFO:tasks.rados.rados.0.smithi151.stdout:2377: delete oid 21 current snap is 313 2024-09-13T14:31:09.603 INFO:tasks.rados.rados.0.smithi151.stdout:2372: finishing write tid 1 to smithi15186934-39 2024-09-13T14:31:09.603 INFO:tasks.rados.rados.0.smithi151.stdout:2371: expect (ObjNum 822 snap 286 seq_num 822) 2024-09-13T14:31:09.810 INFO:tasks.rados.rados.0.smithi151.stdout:2372: finishing write tid 2 to smithi15186934-39 2024-09-13T14:31:09.811 INFO:tasks.rados.rados.0.smithi151.stdout:2372: finishing write tid 3 to smithi15186934-39 2024-09-13T14:31:09.811 INFO:tasks.rados.rados.0.smithi151.stdout:2372: finishing write tid 4 to smithi15186934-39 2024-09-13T14:31:09.811 INFO:tasks.rados.rados.0.smithi151.stdout:2372: finishing write tid 5 to smithi15186934-39 2024-09-13T14:31:09.811 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 899 (ObjNum 886 snap 313 seq_num 886) dirty exists 2024-09-13T14:31:09.811 INFO:tasks.rados.rados.0.smithi151.stdout:2372: left oid 39 (ObjNum 886 snap 313 seq_num 886) 2024-09-13T14:31:09.811 INFO:tasks.rados.rados.0.smithi151.stdout:2371: done (6 left) 2024-09-13T14:31:09.811 INFO:tasks.rados.rados.0.smithi151.stdout:2372: done (5 left) 2024-09-13T14:31:09.812 INFO:tasks.rados.rados.0.smithi151.stdout:2373: done (4 left) 2024-09-13T14:31:09.812 INFO:tasks.rados.rados.0.smithi151.stdout:2377: done (3 left) 2024-09-13T14:31:09.812 INFO:tasks.rados.rados.0.smithi151.stdout:2378: rollback oid 20 current snap is 313 2024-09-13T14:31:09.812 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 20 to 294 2024-09-13T14:31:09.812 INFO:tasks.rados.rados.0.smithi151.stdout:2374: finishing write tid 1 to smithi15186934-33 2024-09-13T14:31:09.812 INFO:tasks.rados.rados.0.smithi151.stdout:2374: finishing write tid 2 to smithi15186934-33 2024-09-13T14:31:09.812 INFO:tasks.rados.rados.0.smithi151.stdout:2374: finishing write tid 3 to smithi15186934-33 2024-09-13T14:31:09.812 INFO:tasks.rados.rados.0.smithi151.stdout:2374: finishing write tid 4 to smithi15186934-33 2024-09-13T14:31:09.812 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 33 v 929 (ObjNum 887 snap 313 seq_num 887) dirty exists 2024-09-13T14:31:09.812 INFO:tasks.rados.rados.0.smithi151.stdout:2374: left oid 33 (ObjNum 887 snap 313 seq_num 887) 2024-09-13T14:31:09.812 INFO:tasks.rados.rados.0.smithi151.stdout:2376: expect (ObjNum 885 snap 312 seq_num 885) 2024-09-13T14:31:09.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:09 smithi184 ceph-mon[92527]: osdmap e678: 8 total, 8 up, 8 in 2024-09-13T14:31:09.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:09 smithi184 ceph-mon[92527]: osdmap e679: 8 total, 8 up, 8 in 2024-09-13T14:31:09.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:09 smithi151 ceph-mon[108966]: osdmap e678: 8 total, 8 up, 8 in 2024-09-13T14:31:09.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:09 smithi151 ceph-mon[108966]: osdmap e679: 8 total, 8 up, 8 in 2024-09-13T14:31:09.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:09 smithi151 ceph-mon[104450]: osdmap e678: 8 total, 8 up, 8 in 2024-09-13T14:31:09.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:09 smithi151 ceph-mon[104450]: osdmap e679: 8 total, 8 up, 8 in 2024-09-13T14:31:09.974 INFO:tasks.rados.rados.0.smithi151.stdout:2375: finishing write tid 1 to smithi15186934-23 2024-09-13T14:31:09.974 INFO:tasks.rados.rados.0.smithi151.stdout:2375: finishing write tid 2 to smithi15186934-23 2024-09-13T14:31:09.974 INFO:tasks.rados.rados.0.smithi151.stdout:2374: done (3 left) 2024-09-13T14:31:09.974 INFO:tasks.rados.rados.0.smithi151.stdout:2376: done (2 left) 2024-09-13T14:31:09.974 INFO:tasks.rados.rados.0.smithi151.stdout:2379: rollback oid 13 current snap is 313 2024-09-13T14:31:09.975 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 13 to 309 2024-09-13T14:31:09.975 INFO:tasks.rados.rados.0.smithi151.stdout:2375: finishing write tid 3 to smithi15186934-23 2024-09-13T14:31:09.975 INFO:tasks.rados.rados.0.smithi151.stdout:2375: finishing write tid 4 to smithi15186934-23 2024-09-13T14:31:09.975 INFO:tasks.rados.rados.0.smithi151.stdout:2375: finishing write tid 5 to smithi15186934-23 2024-09-13T14:31:09.975 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 23 v 1071 (ObjNum 888 snap 313 seq_num 888) dirty exists 2024-09-13T14:31:09.975 INFO:tasks.rados.rados.0.smithi151.stdout:2375: left oid 23 (ObjNum 888 snap 313 seq_num 888) 2024-09-13T14:31:09.975 INFO:tasks.rados.rados.0.smithi151.stdout:2378: finishing rollback tid 0 to smithi15186934-20 2024-09-13T14:31:09.975 INFO:tasks.rados.rados.0.smithi151.stdout:2378: finishing rollback tid 1 to smithi15186934-20 2024-09-13T14:31:09.975 INFO:tasks.rados.rados.0.smithi151.stdout:2378: finishing rollback tid 2 to smithi15186934-20 2024-09-13T14:31:09.975 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 561 (ObjNum 806 snap 283 seq_num 806) dirty exists 2024-09-13T14:31:09.975 INFO:tasks.rados.rados.0.smithi151.stdout:2375: done (2 left) 2024-09-13T14:31:09.976 INFO:tasks.rados.rados.0.smithi151.stdout:2378: done (1 left) 2024-09-13T14:31:09.976 INFO:tasks.rados.rados.0.smithi151.stdout:2380: write oid 10 current snap is 313 2024-09-13T14:31:09.976 INFO:tasks.rados.rados.0.smithi151.stdout:2380: seq_num 890 ranges {558230=584911,1897327=527644,2923993=34237} 2024-09-13T14:31:09.980 INFO:tasks.rados.rados.0.smithi151.stdout:2380: writing smithi15186934-10 from 558230 to 1143141 tid 1 2024-09-13T14:31:09.987 INFO:tasks.rados.rados.0.smithi151.stdout:2380: writing smithi15186934-10 from 1897327 to 2424971 tid 2 2024-09-13T14:31:09.990 INFO:tasks.rados.rados.0.smithi151.stdout:2380: writing smithi15186934-10 from 2923993 to 2958230 tid 3 2024-09-13T14:31:09.990 INFO:tasks.rados.rados.0.smithi151.stdout:2381: delete oid 17 current snap is 313 2024-09-13T14:31:09.990 INFO:tasks.rados.rados.0.smithi151.stdout:2379: finishing rollback tid 0 to smithi15186934-13 2024-09-13T14:31:09.990 INFO:tasks.rados.rados.0.smithi151.stdout:2379: finishing rollback tid 1 to smithi15186934-13 2024-09-13T14:31:09.990 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 13 v 932 (ObjNum 241 snap 0 seq_num 3092944448) dirty dne 2024-09-13T14:31:09.990 INFO:tasks.rados.rados.0.smithi151.stdout:2380: finishing write tid 1 to smithi15186934-10 2024-09-13T14:31:09.990 INFO:tasks.rados.rados.0.smithi151.stdout:2380: finishing write tid 2 to smithi15186934-10 2024-09-13T14:31:09.991 INFO:tasks.rados.rados.0.smithi151.stdout:2380: finishing write tid 3 to smithi15186934-10 2024-09-13T14:31:09.992 INFO:tasks.rados.rados.0.smithi151.stdout:2379: done (2 left) 2024-09-13T14:31:09.992 INFO:tasks.rados.rados.0.smithi151.stdout:2381: done (1 left) 2024-09-13T14:31:09.992 INFO:tasks.rados.rados.0.smithi151.stdout:2382: write oid 19 current snap is 313 2024-09-13T14:31:09.993 INFO:tasks.rados.rados.0.smithi151.stdout:2380: finishing write tid 4 to smithi15186934-10 2024-09-13T14:31:09.993 INFO:tasks.rados.rados.0.smithi151.stdout:2382: seq_num 892 ranges {591844=735409,1897938=531313,2862486=129358} 2024-09-13T14:31:09.999 INFO:tasks.rados.rados.0.smithi151.stdout:2382: writing smithi15186934-19 from 591844 to 1327253 tid 1 2024-09-13T14:31:10.005 INFO:tasks.rados.rados.0.smithi151.stdout:2382: writing smithi15186934-19 from 1897938 to 2429251 tid 2 2024-09-13T14:31:10.008 INFO:tasks.rados.rados.0.smithi151.stdout:2382: writing smithi15186934-19 from 2862486 to 2991844 tid 3 2024-09-13T14:31:10.008 INFO:tasks.rados.rados.0.smithi151.stdout:2383: read oid 41 snap -1 2024-09-13T14:31:10.009 INFO:tasks.rados.rados.0.smithi151.stdout:2383: expect (ObjNum 878 snap 311 seq_num 878) 2024-09-13T14:31:10.009 INFO:tasks.rados.rados.0.smithi151.stdout:2384: write oid 22 current snap is 313 2024-09-13T14:31:10.009 INFO:tasks.rados.rados.0.smithi151.stdout:2380: finishing write tid 5 to smithi15186934-10 2024-09-13T14:31:10.009 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 830 (ObjNum 890 snap 313 seq_num 890) dirty exists 2024-09-13T14:31:10.009 INFO:tasks.rados.rados.0.smithi151.stdout:2380: left oid 10 (ObjNum 890 snap 313 seq_num 890) 2024-09-13T14:31:10.009 INFO:tasks.rados.rados.0.smithi151.stdout:2384: seq_num 893 ranges {608651=410658,1556420=674971,2773556=235095} 2024-09-13T14:31:10.014 INFO:tasks.rados.rados.0.smithi151.stdout:2384: writing smithi15186934-22 from 608651 to 1019309 tid 1 2024-09-13T14:31:10.020 INFO:tasks.rados.rados.0.smithi151.stdout:2384: writing smithi15186934-22 from 1556420 to 2231391 tid 2 2024-09-13T14:31:10.025 INFO:tasks.rados.rados.0.smithi151.stdout:2384: writing smithi15186934-22 from 2773556 to 3008651 tid 3 2024-09-13T14:31:10.025 INFO:tasks.rados.rados.0.smithi151.stdout:2380: done (3 left) 2024-09-13T14:31:10.025 INFO:tasks.rados.rados.0.smithi151.stdout:2385: snap_create 2024-09-13T14:31:10.025 INFO:tasks.rados.rados.0.smithi151.stdout:2382: finishing write tid 1 to smithi15186934-19 2024-09-13T14:31:10.025 INFO:tasks.rados.rados.0.smithi151.stdout:2382: finishing write tid 2 to smithi15186934-19 2024-09-13T14:31:10.025 INFO:tasks.rados.rados.0.smithi151.stdout:2382: finishing write tid 3 to smithi15186934-19 2024-09-13T14:31:10.025 INFO:tasks.rados.rados.0.smithi151.stdout:2382: finishing write tid 4 to smithi15186934-19 2024-09-13T14:31:10.025 INFO:tasks.rados.rados.0.smithi151.stdout:2382: finishing write tid 5 to smithi15186934-19 2024-09-13T14:31:10.025 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 19 v 604 (ObjNum 892 snap 313 seq_num 892) dirty exists 2024-09-13T14:31:10.025 INFO:tasks.rados.rados.0.smithi151.stdout:2382: left oid 19 (ObjNum 892 snap 313 seq_num 892) 2024-09-13T14:31:10.026 INFO:tasks.rados.rados.0.smithi151.stdout:2383: expect (ObjNum 878 snap 311 seq_num 878) 2024-09-13T14:31:10.116 INFO:tasks.rados.rados.0.smithi151.stdout:2384: finishing write tid 1 to smithi15186934-22 2024-09-13T14:31:10.116 INFO:tasks.rados.rados.0.smithi151.stdout:2384: finishing write tid 2 to smithi15186934-22 2024-09-13T14:31:10.116 INFO:tasks.rados.rados.0.smithi151.stdout:2384: finishing write tid 3 to smithi15186934-22 2024-09-13T14:31:10.116 INFO:tasks.rados.rados.0.smithi151.stdout:2384: finishing write tid 4 to smithi15186934-22 2024-09-13T14:31:10.116 INFO:tasks.rados.rados.0.smithi151.stdout:2384: finishing write tid 5 to smithi15186934-22 2024-09-13T14:31:10.116 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 22 v 498 (ObjNum 893 snap 313 seq_num 893) dirty exists 2024-09-13T14:31:10.116 INFO:tasks.rados.rados.0.smithi151.stdout:2384: left oid 22 (ObjNum 893 snap 313 seq_num 893) 2024-09-13T14:31:10.585 INFO:tasks.rados.rados.0.smithi151.stdout:2382: done (3 left) 2024-09-13T14:31:10.585 INFO:tasks.rados.rados.0.smithi151.stdout:2383: done (2 left) 2024-09-13T14:31:10.585 INFO:tasks.rados.rados.0.smithi151.stdout:2384: done (1 left) 2024-09-13T14:31:10.585 INFO:tasks.rados.rados.0.smithi151.stdout:2385: done (0 left) 2024-09-13T14:31:10.585 INFO:tasks.rados.rados.0.smithi151.stdout:2386: write oid 9 current snap is 314 2024-09-13T14:31:10.585 INFO:tasks.rados.rados.0.smithi151.stdout:2386: seq_num 894 ranges {625458=485907,1614902=534982,2917332=108126} 2024-09-13T14:31:10.586 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:31:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:31:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:31:10.590 INFO:tasks.rados.rados.0.smithi151.stdout:2386: writing smithi15186934-9 from 625458 to 1111365 tid 1 2024-09-13T14:31:10.596 INFO:tasks.rados.rados.0.smithi151.stdout:2386: writing smithi15186934-9 from 1614902 to 2149884 tid 2 2024-09-13T14:31:10.601 INFO:tasks.rados.rados.0.smithi151.stdout:2386: writing smithi15186934-9 from 2917332 to 3025458 tid 3 2024-09-13T14:31:10.601 INFO:tasks.rados.rados.0.smithi151.stdout:2387: delete oid 25 current snap is 314 2024-09-13T14:31:10.601 INFO:tasks.rados.rados.0.smithi151.stdout:2386: finishing write tid 1 to smithi15186934-9 2024-09-13T14:31:10.602 INFO:tasks.rados.rados.0.smithi151.stdout:2386: finishing write tid 2 to smithi15186934-9 2024-09-13T14:31:10.603 INFO:tasks.rados.rados.0.smithi151.stdout:2386: finishing write tid 3 to smithi15186934-9 2024-09-13T14:31:10.603 INFO:tasks.rados.rados.0.smithi151.stdout:2386: finishing write tid 4 to smithi15186934-9 2024-09-13T14:31:10.604 INFO:tasks.rados.rados.0.smithi151.stdout:2386: finishing write tid 5 to smithi15186934-9 2024-09-13T14:31:10.604 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 973 (ObjNum 894 snap 314 seq_num 894) dirty exists 2024-09-13T14:31:10.604 INFO:tasks.rados.rados.0.smithi151.stdout:2386: left oid 9 (ObjNum 894 snap 314 seq_num 894) 2024-09-13T14:31:10.606 INFO:tasks.rados.rados.0.smithi151.stdout:2386: done (1 left) 2024-09-13T14:31:10.606 INFO:tasks.rados.rados.0.smithi151.stdout:2387: done (0 left) 2024-09-13T14:31:10.606 INFO:tasks.rados.rados.0.smithi151.stdout:2388: read oid 14 snap -1 2024-09-13T14:31:10.607 INFO:tasks.rados.rados.0.smithi151.stdout:2388: expect (ObjNum 849 snap 293 seq_num 849) 2024-09-13T14:31:10.607 INFO:tasks.rados.rados.0.smithi151.stdout:2389: write oid 2 current snap is 314 2024-09-13T14:31:10.607 INFO:tasks.rados.rados.0.smithi151.stdout:2389: seq_num 896 ranges {659072=636405,1731866=422298,2855825=203247} 2024-09-13T14:31:10.613 INFO:tasks.rados.rados.0.smithi151.stdout:2389: writing smithi15186934-2 from 659072 to 1295477 tid 1 2024-09-13T14:31:10.617 INFO:tasks.rados.rados.0.smithi151.stdout:2389: writing smithi15186934-2 from 1731866 to 2154164 tid 2 2024-09-13T14:31:10.623 INFO:tasks.rados.rados.0.smithi151.stdout:2389: writing smithi15186934-2 from 2855825 to 3059072 tid 3 2024-09-13T14:31:10.623 INFO:tasks.rados.rados.0.smithi151.stdout:2390: write oid 19 current snap is 314 2024-09-13T14:31:10.623 INFO:tasks.rados.rados.0.smithi151.stdout:2390: seq_num 897 ranges {675879=711654,2073995=682309} 2024-09-13T14:31:10.630 INFO:tasks.rados.rados.0.smithi151.stdout:2390: writing smithi15186934-19 from 675879 to 1387533 tid 1 2024-09-13T14:31:10.638 INFO:tasks.rados.rados.0.smithi151.stdout:2390: writing smithi15186934-19 from 2073995 to 2756304 tid 2 2024-09-13T14:31:10.638 INFO:tasks.rados.rados.0.smithi151.stdout:2391: read oid 46 snap -1 2024-09-13T14:31:10.638 INFO:tasks.rados.rados.0.smithi151.stdout:2391: expect (ObjNum 786 snap 275 seq_num 786) 2024-09-13T14:31:10.638 INFO:tasks.rados.rados.0.smithi151.stdout:2392: read oid 32 snap -1 2024-09-13T14:31:10.638 INFO:tasks.rados.rados.0.smithi151.stdout:2392: expect (ObjNum 879 snap 311 seq_num 879) 2024-09-13T14:31:10.638 INFO:tasks.rados.rados.0.smithi151.stdout:2393: read oid 45 snap -1 2024-09-13T14:31:10.639 INFO:tasks.rados.rados.0.smithi151.stdout:2393: expect (ObjNum 858 snap 296 seq_num 858) 2024-09-13T14:31:10.639 INFO:tasks.rados.rados.0.smithi151.stdout:2394: write oid 8 current snap is 314 2024-09-13T14:31:10.639 INFO:tasks.rados.rados.0.smithi151.stdout:2388: expect (ObjNum 849 snap 293 seq_num 849) 2024-09-13T14:31:10.819 INFO:tasks.rados.rados.0.smithi151.stdout:2389: finishing write tid 1 to smithi15186934-2 2024-09-13T14:31:10.819 INFO:tasks.rados.rados.0.smithi151.stdout:2389: finishing write tid 2 to smithi15186934-2 2024-09-13T14:31:10.819 INFO:tasks.rados.rados.0.smithi151.stdout:2389: finishing write tid 3 to smithi15186934-2 2024-09-13T14:31:10.819 INFO:tasks.rados.rados.0.smithi151.stdout:2389: finishing write tid 4 to smithi15186934-2 2024-09-13T14:31:10.819 INFO:tasks.rados.rados.0.smithi151.stdout:2389: finishing write tid 5 to smithi15186934-2 2024-09-13T14:31:10.819 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 2 v 936 (ObjNum 896 snap 314 seq_num 896) dirty exists 2024-09-13T14:31:10.819 INFO:tasks.rados.rados.0.smithi151.stdout:2389: left oid 2 (ObjNum 896 snap 314 seq_num 896) 2024-09-13T14:31:10.819 INFO:tasks.rados.rados.0.smithi151.stdout:2390: finishing write tid 1 to smithi15186934-19 2024-09-13T14:31:10.819 INFO:tasks.rados.rados.0.smithi151.stdout:2394: seq_num 898 ranges {692686=503256,1848830=425967,2910671=182015} 2024-09-13T14:31:10.825 INFO:tasks.rados.rados.0.smithi151.stdout:2394: writing smithi15186934-8 from 692686 to 1195942 tid 1 2024-09-13T14:31:10.831 INFO:tasks.rados.rados.0.smithi151.stdout:2394: writing smithi15186934-8 from 1848830 to 2274797 tid 2 2024-09-13T14:31:10.836 INFO:tasks.rados.rados.0.smithi151.stdout:2394: writing smithi15186934-8 from 2910671 to 3092686 tid 3 2024-09-13T14:31:10.836 INFO:tasks.rados.rados.0.smithi151.stdout:2388: done (6 left) 2024-09-13T14:31:10.836 INFO:tasks.rados.rados.0.smithi151.stdout:2389: done (5 left) 2024-09-13T14:31:10.836 INFO:tasks.rados.rados.0.smithi151.stdout:2395: delete oid 40 current snap is 314 2024-09-13T14:31:10.836 INFO:tasks.rados.rados.0.smithi151.stdout:2390: finishing write tid 2 to smithi15186934-19 2024-09-13T14:31:10.836 INFO:tasks.rados.rados.0.smithi151.stdout:2390: finishing write tid 3 to smithi15186934-19 2024-09-13T14:31:10.836 INFO:tasks.rados.rados.0.smithi151.stdout:2390: finishing write tid 4 to smithi15186934-19 2024-09-13T14:31:10.836 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 19 v 608 (ObjNum 897 snap 314 seq_num 897) dirty exists 2024-09-13T14:31:10.836 INFO:tasks.rados.rados.0.smithi151.stdout:2390: left oid 19 (ObjNum 897 snap 314 seq_num 897) 2024-09-13T14:31:10.836 INFO:tasks.rados.rados.0.smithi151.stdout:2392: expect (ObjNum 879 snap 311 seq_num 879) 2024-09-13T14:31:10.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:10 smithi184 ceph-mon[92527]: pgmap v636: 89 pgs: 89 active+clean; 188 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.1 MiB/s wr, 16 op/s 2024-09-13T14:31:10.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:10 smithi151 ceph-mon[104450]: pgmap v636: 89 pgs: 89 active+clean; 188 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.1 MiB/s wr, 16 op/s 2024-09-13T14:31:10.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:10 smithi151 ceph-mon[108966]: pgmap v636: 89 pgs: 89 active+clean; 188 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.1 MiB/s wr, 16 op/s 2024-09-13T14:31:11.026 INFO:tasks.rados.rados.0.smithi151.stdout:2391: expect (ObjNum 786 snap 275 seq_num 786) 2024-09-13T14:31:11.130 INFO:tasks.rados.rados.0.smithi151.stdout:2393: expect (ObjNum 858 snap 296 seq_num 858) 2024-09-13T14:31:11.206 INFO:tasks.rados.rados.0.smithi151.stdout:2394: finishing write tid 1 to smithi15186934-8 2024-09-13T14:31:11.228 INFO:tasks.rados.rados.0.smithi151.stdout:2394: finishing write tid 2 to smithi15186934-8 2024-09-13T14:31:11.228 INFO:tasks.rados.rados.0.smithi151.stdout:2394: finishing write tid 3 to smithi15186934-8 2024-09-13T14:31:11.228 INFO:tasks.rados.rados.0.smithi151.stdout:2394: finishing write tid 4 to smithi15186934-8 2024-09-13T14:31:11.228 INFO:tasks.rados.rados.0.smithi151.stdout:2394: finishing write tid 5 to smithi15186934-8 2024-09-13T14:31:11.229 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 740 (ObjNum 898 snap 314 seq_num 898) dirty exists 2024-09-13T14:31:11.229 INFO:tasks.rados.rados.0.smithi151.stdout:2394: left oid 8 (ObjNum 898 snap 314 seq_num 898) 2024-09-13T14:31:11.229 INFO:tasks.rados.rados.0.smithi151.stdout:2390: done (5 left) 2024-09-13T14:31:11.229 INFO:tasks.rados.rados.0.smithi151.stdout:2391: done (4 left) 2024-09-13T14:31:11.229 INFO:tasks.rados.rados.0.smithi151.stdout:2392: done (3 left) 2024-09-13T14:31:11.229 INFO:tasks.rados.rados.0.smithi151.stdout:2393: done (2 left) 2024-09-13T14:31:11.229 INFO:tasks.rados.rados.0.smithi151.stdout:2394: done (1 left) 2024-09-13T14:31:11.229 INFO:tasks.rados.rados.0.smithi151.stdout:2395: done (0 left) 2024-09-13T14:31:11.229 INFO:tasks.rados.rados.0.smithi151.stdout:2396: snap_remove snap 284 2024-09-13T14:31:11.596 INFO:tasks.rados.rados.0.smithi151.stdout:2396: done (0 left) 2024-09-13T14:31:11.596 INFO:tasks.rados.rados.0.smithi151.stdout:2397: read oid 48 snap -1 2024-09-13T14:31:11.597 INFO:tasks.rados.rados.0.smithi151.stdout:2397: expect (ObjNum 854 snap 295 seq_num 854) 2024-09-13T14:31:11.597 INFO:tasks.rados.rados.0.smithi151.stdout:2398: read oid 37 snap -1 2024-09-13T14:31:11.597 INFO:tasks.rados.rados.0.smithi151.stdout:2398: expect (ObjNum 840 snap 293 seq_num 840) 2024-09-13T14:31:11.597 INFO:tasks.rados.rados.0.smithi151.stdout:2399: read oid 31 snap 303 2024-09-13T14:31:11.597 INFO:tasks.rados.rados.0.smithi151.stdout:2399: expect (ObjNum 818 snap 286 seq_num 818) 2024-09-13T14:31:11.597 INFO:tasks.rados.rados.0.smithi151.stdout:2400: delete oid 14 current snap is 314 2024-09-13T14:31:11.602 INFO:tasks.rados.rados.0.smithi151.stdout:2400: done (3 left) 2024-09-13T14:31:11.602 INFO:tasks.rados.rados.0.smithi151.stdout:2401: write oid 34 current snap is 314 2024-09-13T14:31:11.602 INFO:tasks.rados.rados.0.smithi151.stdout:2401: seq_num 901 ranges {743107=729003,1907923=689647} 2024-09-13T14:31:11.611 INFO:tasks.rados.rados.0.smithi151.stdout:2401: writing smithi15186934-34 from 743107 to 1472110 tid 1 2024-09-13T14:31:11.617 INFO:tasks.rados.rados.0.smithi151.stdout:2401: writing smithi15186934-34 from 1907923 to 2597570 tid 2 2024-09-13T14:31:11.617 INFO:tasks.rados.rados.0.smithi151.stdout:2402: read oid 6 snap 307 2024-09-13T14:31:11.618 INFO:tasks.rados.rados.0.smithi151.stdout:2402: expect (ObjNum 825 snap 286 seq_num 825) 2024-09-13T14:31:11.618 INFO:tasks.rados.rados.0.smithi151.stdout:2403: write oid 23 current snap is 314 2024-09-13T14:31:11.618 INFO:tasks.rados.rados.0.smithi151.stdout:2403: seq_num 902 ranges {759914=404252,1566405=433305,2504010=655904} 2024-09-13T14:31:11.624 INFO:tasks.rados.rados.0.smithi151.stdout:2403: writing smithi15186934-23 from 759914 to 1164166 tid 1 2024-09-13T14:31:11.628 INFO:tasks.rados.rados.0.smithi151.stdout:2403: writing smithi15186934-23 from 1566405 to 1999710 tid 2 2024-09-13T14:31:11.635 INFO:tasks.rados.rados.0.smithi151.stdout:2403: writing smithi15186934-23 from 2504010 to 3159914 tid 3 2024-09-13T14:31:11.635 INFO:tasks.rados.rados.0.smithi151.stdout:2404: rollback oid 50 current snap is 314 2024-09-13T14:31:11.635 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 50 to 299 2024-09-13T14:31:11.635 INFO:tasks.rados.rados.0.smithi151.stdout:2399: expect (ObjNum 818 snap 286 seq_num 818) 2024-09-13T14:31:11.797 INFO:tasks.rados.rados.0.smithi151.stdout:2398: expect (ObjNum 840 snap 293 seq_num 840) 2024-09-13T14:31:11.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:11 smithi184 ceph-mon[92527]: osdmap e680: 8 total, 8 up, 8 in 2024-09-13T14:31:11.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:11 smithi151 ceph-mon[104450]: osdmap e680: 8 total, 8 up, 8 in 2024-09-13T14:31:11.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:11 smithi151 ceph-mon[108966]: osdmap e680: 8 total, 8 up, 8 in 2024-09-13T14:31:11.924 INFO:tasks.rados.rados.0.smithi151.stdout:2397: expect (ObjNum 854 snap 295 seq_num 854) 2024-09-13T14:31:12.002 INFO:tasks.rados.rados.0.smithi151.stdout:2401: finishing write tid 1 to smithi15186934-34 2024-09-13T14:31:12.002 INFO:tasks.rados.rados.0.smithi151.stdout:2401: finishing write tid 2 to smithi15186934-34 2024-09-13T14:31:12.002 INFO:tasks.rados.rados.0.smithi151.stdout:2401: finishing write tid 3 to smithi15186934-34 2024-09-13T14:31:12.002 INFO:tasks.rados.rados.0.smithi151.stdout:2397: done (6 left) 2024-09-13T14:31:12.002 INFO:tasks.rados.rados.0.smithi151.stdout:2398: done (5 left) 2024-09-13T14:31:12.002 INFO:tasks.rados.rados.0.smithi151.stdout:2399: done (4 left) 2024-09-13T14:31:12.002 INFO:tasks.rados.rados.0.smithi151.stdout:2405: read oid 46 snap -1 2024-09-13T14:31:12.003 INFO:tasks.rados.rados.0.smithi151.stdout:2405: expect (ObjNum 786 snap 275 seq_num 786) 2024-09-13T14:31:12.003 INFO:tasks.rados.rados.0.smithi151.stdout:2406: delete oid 4 current snap is 314 2024-09-13T14:31:12.003 INFO:tasks.rados.rados.0.smithi151.stdout:2401: finishing write tid 4 to smithi15186934-34 2024-09-13T14:31:12.003 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 888 (ObjNum 901 snap 314 seq_num 901) dirty exists 2024-09-13T14:31:12.003 INFO:tasks.rados.rados.0.smithi151.stdout:2401: left oid 34 (ObjNum 901 snap 314 seq_num 901) 2024-09-13T14:31:12.003 INFO:tasks.rados.rados.0.smithi151.stdout:2402: expect (ObjNum 825 snap 286 seq_num 825) 2024-09-13T14:31:12.260 INFO:tasks.rados.rados.0.smithi151.stdout:2403: finishing write tid 1 to smithi15186934-23 2024-09-13T14:31:12.260 INFO:tasks.rados.rados.0.smithi151.stdout:2403: finishing write tid 2 to smithi15186934-23 2024-09-13T14:31:12.260 INFO:tasks.rados.rados.0.smithi151.stdout:2403: finishing write tid 3 to smithi15186934-23 2024-09-13T14:31:12.260 INFO:tasks.rados.rados.0.smithi151.stdout:2404: finishing rollback tid 0 to smithi15186934-50 2024-09-13T14:31:12.260 INFO:tasks.rados.rados.0.smithi151.stdout:2404: finishing rollback tid 1 to smithi15186934-50 2024-09-13T14:31:12.260 INFO:tasks.rados.rados.0.smithi151.stdout:2401: done (5 left) 2024-09-13T14:31:12.260 INFO:tasks.rados.rados.0.smithi151.stdout:2402: done (4 left) 2024-09-13T14:31:12.260 INFO:tasks.rados.rados.0.smithi151.stdout:2406: done (3 left) 2024-09-13T14:31:12.260 INFO:tasks.rados.rados.0.smithi151.stdout:2407: snap_remove snap 294 2024-09-13T14:31:12.597 INFO:tasks.rados.rados.0.smithi151.stdout:2407: done (3 left) 2024-09-13T14:31:12.597 INFO:tasks.rados.rados.0.smithi151.stdout:2408: read oid 41 snap 312 2024-09-13T14:31:12.597 INFO:tasks.rados.rados.0.smithi151.stdout:2408: expect (ObjNum 878 snap 311 seq_num 878) 2024-09-13T14:31:12.597 INFO:tasks.rados.rados.0.smithi151.stdout:2409: delete oid 29 current snap is 314 2024-09-13T14:31:12.598 INFO:tasks.rados.rados.0.smithi151.stdout:2403: finishing write tid 4 to smithi15186934-23 2024-09-13T14:31:12.598 INFO:tasks.rados.rados.0.smithi151.stdout:2404: finishing rollback tid 2 to smithi15186934-50 2024-09-13T14:31:12.598 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 50 v 907 (ObjNum 847 snap 293 seq_num 847) dirty exists 2024-09-13T14:31:12.598 INFO:tasks.rados.rados.0.smithi151.stdout:2403: finishing write tid 5 to smithi15186934-23 2024-09-13T14:31:12.598 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 23 v 1082 (ObjNum 902 snap 314 seq_num 902) dirty exists 2024-09-13T14:31:12.598 INFO:tasks.rados.rados.0.smithi151.stdout:2403: left oid 23 (ObjNum 902 snap 314 seq_num 902) 2024-09-13T14:31:12.598 INFO:tasks.rados.rados.0.smithi151.stdout:2405: expect (ObjNum 786 snap 275 seq_num 786) 2024-09-13T14:31:12.707 INFO:tasks.rados.rados.0.smithi151.stdout:2408: expect (ObjNum 878 snap 311 seq_num 878) 2024-09-13T14:31:12.797 INFO:tasks.rados.rados.0.smithi151.stdout:2403: done (4 left) 2024-09-13T14:31:12.797 INFO:tasks.rados.rados.0.smithi151.stdout:2404: done (3 left) 2024-09-13T14:31:12.797 INFO:tasks.rados.rados.0.smithi151.stdout:2405: done (2 left) 2024-09-13T14:31:12.797 INFO:tasks.rados.rados.0.smithi151.stdout:2408: done (1 left) 2024-09-13T14:31:12.797 INFO:tasks.rados.rados.0.smithi151.stdout:2409: done (0 left) 2024-09-13T14:31:12.797 INFO:tasks.rados.rados.0.smithi151.stdout:2410: rollback oid 37 current snap is 314 2024-09-13T14:31:12.797 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 37 to 290 2024-09-13T14:31:12.798 INFO:tasks.rados.rados.0.smithi151.stdout:2411: rollback oid 34 current snap is 314 2024-09-13T14:31:12.798 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 34 to 313 2024-09-13T14:31:12.798 INFO:tasks.rados.rados.0.smithi151.stdout:2412: write oid 50 current snap is 314 2024-09-13T14:31:12.798 INFO:tasks.rados.rados.0.smithi151.stdout:2412: seq_num 905 ranges {410335=746352,1741851=696985,3102632=107703} 2024-09-13T14:31:12.803 INFO:tasks.rados.rados.0.smithi151.stdout:2412: writing smithi15186934-50 from 410335 to 1156687 tid 1 2024-09-13T14:31:12.810 INFO:tasks.rados.rados.0.smithi151.stdout:2412: writing smithi15186934-50 from 1741851 to 2438836 tid 2 2024-09-13T14:31:12.815 INFO:tasks.rados.rados.0.smithi151.stdout:2412: writing smithi15186934-50 from 3102632 to 3210335 tid 3 2024-09-13T14:31:12.816 INFO:tasks.rados.rados.0.smithi151.stdout:2413: snap_create 2024-09-13T14:31:12.816 INFO:tasks.rados.rados.0.smithi151.stdout:2411: finishing rollback tid 0 to smithi15186934-34 2024-09-13T14:31:12.816 INFO:tasks.rados.rados.0.smithi151.stdout:2411: finishing rollback tid 1 to smithi15186934-34 2024-09-13T14:31:12.816 INFO:tasks.rados.rados.0.smithi151.stdout:2411: finishing rollback tid 2 to smithi15186934-34 2024-09-13T14:31:12.816 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 897 (ObjNum 881 snap 312 seq_num 881) dirty exists 2024-09-13T14:31:12.816 INFO:tasks.rados.rados.0.smithi151.stdout:2410: finishing rollback tid 0 to smithi15186934-37 2024-09-13T14:31:12.816 INFO:tasks.rados.rados.0.smithi151.stdout:2410: finishing rollback tid 1 to smithi15186934-37 2024-09-13T14:31:12.816 INFO:tasks.rados.rados.0.smithi151.stdout:2410: finishing rollback tid 2 to smithi15186934-37 2024-09-13T14:31:12.816 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 37 v 1088 (ObjNum 683 snap 222 seq_num 683) dirty exists 2024-09-13T14:31:12.816 INFO:tasks.rados.rados.0.smithi151.stdout:2412: finishing write tid 1 to smithi15186934-50 2024-09-13T14:31:12.816 INFO:tasks.rados.rados.0.smithi151.stdout:2412: finishing write tid 2 to smithi15186934-50 2024-09-13T14:31:12.817 INFO:tasks.rados.rados.0.smithi151.stdout:2412: finishing write tid 3 to smithi15186934-50 2024-09-13T14:31:12.817 INFO:tasks.rados.rados.0.smithi151.stdout:2412: finishing write tid 4 to smithi15186934-50 2024-09-13T14:31:12.817 INFO:tasks.rados.rados.0.smithi151.stdout:2412: finishing write tid 5 to smithi15186934-50 2024-09-13T14:31:12.818 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 50 v 917 (ObjNum 905 snap 314 seq_num 905) dirty exists 2024-09-13T14:31:12.818 INFO:tasks.rados.rados.0.smithi151.stdout:2412: left oid 50 (ObjNum 905 snap 314 seq_num 905) 2024-09-13T14:31:12.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:12 smithi151 ceph-mon[104450]: pgmap v639: 89 pgs: 89 active+clean; 194 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 6.0 MiB/s rd, 1.9 MiB/s wr, 14 op/s 2024-09-13T14:31:12.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:12 smithi151 ceph-mon[104450]: osdmap e681: 8 total, 8 up, 8 in 2024-09-13T14:31:12.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:12 smithi151 ceph-mon[108966]: pgmap v639: 89 pgs: 89 active+clean; 194 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 6.0 MiB/s rd, 1.9 MiB/s wr, 14 op/s 2024-09-13T14:31:12.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:12 smithi151 ceph-mon[108966]: osdmap e681: 8 total, 8 up, 8 in 2024-09-13T14:31:13.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:12 smithi184 ceph-mon[92527]: pgmap v639: 89 pgs: 89 active+clean; 194 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 6.0 MiB/s rd, 1.9 MiB/s wr, 14 op/s 2024-09-13T14:31:13.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:12 smithi184 ceph-mon[92527]: osdmap e681: 8 total, 8 up, 8 in 2024-09-13T14:31:13.611 INFO:tasks.rados.rados.0.smithi151.stdout:2410: done (3 left) 2024-09-13T14:31:13.611 INFO:tasks.rados.rados.0.smithi151.stdout:2411: done (2 left) 2024-09-13T14:31:13.611 INFO:tasks.rados.rados.0.smithi151.stdout:2412: done (1 left) 2024-09-13T14:31:13.611 INFO:tasks.rados.rados.0.smithi151.stdout:2413: done (0 left) 2024-09-13T14:31:13.611 INFO:tasks.rados.rados.0.smithi151.stdout:2414: delete oid 24 current snap is 315 2024-09-13T14:31:13.615 INFO:tasks.rados.rados.0.smithi151.stdout:2414: done (0 left) 2024-09-13T14:31:13.615 INFO:tasks.rados.rados.0.smithi151.stdout:2415: rollback oid 18 current snap is 315 2024-09-13T14:31:13.615 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 18 to 311 2024-09-13T14:31:13.616 INFO:tasks.rados.rados.0.smithi151.stdout:2416: snap_remove snap 303 2024-09-13T14:31:13.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:13 smithi151 ceph-mon[104450]: osdmap e682: 8 total, 8 up, 8 in 2024-09-13T14:31:13.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:13 smithi151 ceph-mon[108966]: osdmap e682: 8 total, 8 up, 8 in 2024-09-13T14:31:14.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:13 smithi184 ceph-mon[92527]: osdmap e682: 8 total, 8 up, 8 in 2024-09-13T14:31:14.553 INFO:tasks.rados.rados.0.smithi151.stdout:2416: done (1 left) 2024-09-13T14:31:14.553 INFO:tasks.rados.rados.0.smithi151.stdout:2417: write oid 19 current snap is 315 2024-09-13T14:31:14.553 INFO:tasks.rados.rados.0.smithi151.stdout:2415: finishing rollback tid 1 to smithi15186934-18 2024-09-13T14:31:14.553 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 18 v 585 (ObjNum 129 snap 0 seq_num 3092711104) dirty dne 2024-09-13T14:31:14.554 INFO:tasks.rados.rados.0.smithi151.stdout:2417: seq_num 907 ranges {443949=496850,1458815=584301,2641125=602824} 2024-09-13T14:31:14.559 INFO:tasks.rados.rados.0.smithi151.stdout:2417: writing smithi15186934-19 from 443949 to 940799 tid 1 2024-09-13T14:31:14.566 INFO:tasks.rados.rados.0.smithi151.stdout:2417: writing smithi15186934-19 from 1458815 to 2043116 tid 2 2024-09-13T14:31:14.574 INFO:tasks.rados.rados.0.smithi151.stdout:2417: writing smithi15186934-19 from 2641125 to 3243949 tid 3 2024-09-13T14:31:14.574 INFO:tasks.rados.rados.0.smithi151.stdout:2415: done (1 left) 2024-09-13T14:31:14.574 INFO:tasks.rados.rados.0.smithi151.stdout:2418: write oid 14 current snap is 315 2024-09-13T14:31:14.574 INFO:tasks.rados.rados.0.smithi151.stdout:2418: seq_num 908 ranges {460756=572099,1517297=444312,2668548=472870} 2024-09-13T14:31:14.579 INFO:tasks.rados.rados.0.smithi151.stdout:2418: writing smithi15186934-14 from 460756 to 1032855 tid 1 2024-09-13T14:31:14.585 INFO:tasks.rados.rados.0.smithi151.stdout:2418: writing smithi15186934-14 from 1517297 to 1961609 tid 2 2024-09-13T14:31:14.592 INFO:tasks.rados.rados.0.smithi151.stdout:2418: writing smithi15186934-14 from 2668548 to 3141418 tid 3 2024-09-13T14:31:14.592 INFO:tasks.rados.rados.0.smithi151.stdout:2419: read oid 10 snap 299 2024-09-13T14:31:14.592 INFO:tasks.rados.rados.0.smithi151.stdout:2419: expect (ObjNum 862 snap 297 seq_num 862) 2024-09-13T14:31:14.592 INFO:tasks.rados.rados.0.smithi151.stdout:2420: rollback oid 44 current snap is 315 2024-09-13T14:31:14.592 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 44 to 298 2024-09-13T14:31:14.592 INFO:tasks.rados.rados.0.smithi151.stdout:2417: finishing write tid 1 to smithi15186934-19 2024-09-13T14:31:14.592 INFO:tasks.rados.rados.0.smithi151.stdout:2421: snap_remove snap 305 2024-09-13T14:31:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:14 smithi151 ceph-mon[104450]: pgmap v642: 89 pgs: 89 active+clean; 207 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 24 MiB/s rd, 3.8 MiB/s wr, 37 op/s 2024-09-13T14:31:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:14 smithi151 ceph-mon[104450]: osdmap e683: 8 total, 8 up, 8 in 2024-09-13T14:31:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:14 smithi151 ceph-mon[104450]: osdmap e684: 8 total, 8 up, 8 in 2024-09-13T14:31:14.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:31:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:31:14] "GET /metrics HTTP/1.1" 200 36512 "" "Prometheus/2.51.0" 2024-09-13T14:31:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:14 smithi151 ceph-mon[108966]: pgmap v642: 89 pgs: 89 active+clean; 207 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 24 MiB/s rd, 3.8 MiB/s wr, 37 op/s 2024-09-13T14:31:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:14 smithi151 ceph-mon[108966]: osdmap e683: 8 total, 8 up, 8 in 2024-09-13T14:31:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:14 smithi151 ceph-mon[108966]: osdmap e684: 8 total, 8 up, 8 in 2024-09-13T14:31:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:14 smithi184 ceph-mon[92527]: pgmap v642: 89 pgs: 89 active+clean; 207 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 24 MiB/s rd, 3.8 MiB/s wr, 37 op/s 2024-09-13T14:31:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:14 smithi184 ceph-mon[92527]: osdmap e683: 8 total, 8 up, 8 in 2024-09-13T14:31:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:14 smithi184 ceph-mon[92527]: osdmap e684: 8 total, 8 up, 8 in 2024-09-13T14:31:15.556 INFO:tasks.rados.rados.0.smithi151.stdout:2417: finishing write tid 2 to smithi15186934-19 2024-09-13T14:31:15.556 INFO:tasks.rados.rados.0.smithi151.stdout:2417: finishing write tid 3 to smithi15186934-19 2024-09-13T14:31:15.556 INFO:tasks.rados.rados.0.smithi151.stdout:2417: finishing write tid 4 to smithi15186934-19 2024-09-13T14:31:15.557 INFO:tasks.rados.rados.0.smithi151.stdout:2417: finishing write tid 5 to smithi15186934-19 2024-09-13T14:31:15.557 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 19 v 621 (ObjNum 907 snap 315 seq_num 907) dirty exists 2024-09-13T14:31:15.557 INFO:tasks.rados.rados.0.smithi151.stdout:2417: left oid 19 (ObjNum 907 snap 315 seq_num 907) 2024-09-13T14:31:15.557 INFO:tasks.rados.rados.0.smithi151.stdout:2418: finishing write tid 1 to smithi15186934-14 2024-09-13T14:31:15.557 INFO:tasks.rados.rados.0.smithi151.stdout:2418: finishing write tid 2 to smithi15186934-14 2024-09-13T14:31:15.557 INFO:tasks.rados.rados.0.smithi151.stdout:2420: finishing rollback tid 1 to smithi15186934-44 2024-09-13T14:31:15.557 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 710 (ObjNum 129 snap 0 seq_num 3092540760) dirty dne 2024-09-13T14:31:15.557 INFO:tasks.rados.rados.0.smithi151.stdout:2418: finishing write tid 3 to smithi15186934-14 2024-09-13T14:31:15.557 INFO:tasks.rados.rados.0.smithi151.stdout:2418: finishing write tid 4 to smithi15186934-14 2024-09-13T14:31:15.557 INFO:tasks.rados.rados.0.smithi151.stdout:2418: finishing write tid 5 to smithi15186934-14 2024-09-13T14:31:15.557 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 991 (ObjNum 908 snap 315 seq_num 908) dirty exists 2024-09-13T14:31:15.557 INFO:tasks.rados.rados.0.smithi151.stdout:2418: left oid 14 (ObjNum 908 snap 315 seq_num 908) 2024-09-13T14:31:15.557 INFO:tasks.rados.rados.0.smithi151.stdout:2419: expect (ObjNum 862 snap 297 seq_num 862) 2024-09-13T14:31:15.643 INFO:tasks.rados.rados.0.smithi151.stdout:2417: done (4 left) 2024-09-13T14:31:15.643 INFO:tasks.rados.rados.0.smithi151.stdout:2418: done (3 left) 2024-09-13T14:31:15.644 INFO:tasks.rados.rados.0.smithi151.stdout:2419: done (2 left) 2024-09-13T14:31:15.644 INFO:tasks.rados.rados.0.smithi151.stdout:2420: done (1 left) 2024-09-13T14:31:15.644 INFO:tasks.rados.rados.0.smithi151.stdout:2421: done (0 left) 2024-09-13T14:31:15.644 INFO:tasks.rados.rados.0.smithi151.stdout:2422: snap_remove snap 298 2024-09-13T14:31:16.617 INFO:tasks.rados.rados.0.smithi151.stdout:2422: done (0 left) 2024-09-13T14:31:16.617 INFO:tasks.rados.rados.0.smithi151.stdout:2423: read oid 45 snap 313 2024-09-13T14:31:16.617 INFO:tasks.rados.rados.0.smithi151.stdout:2423: expect (ObjNum 858 snap 296 seq_num 858) 2024-09-13T14:31:16.617 INFO:tasks.rados.rados.0.smithi151.stdout:2424: snap_remove snap 310 2024-09-13T14:31:16.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:16 smithi184 ceph-mon[92527]: pgmap v645: 89 pgs: 89 active+clean; 204 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 28 MiB/s rd, 3.4 MiB/s wr, 39 op/s 2024-09-13T14:31:16.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:16 smithi184 ceph-mon[92527]: osdmap e685: 8 total, 8 up, 8 in 2024-09-13T14:31:16.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:16.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:16 smithi151 ceph-mon[104450]: pgmap v645: 89 pgs: 89 active+clean; 204 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 28 MiB/s rd, 3.4 MiB/s wr, 39 op/s 2024-09-13T14:31:16.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:16 smithi151 ceph-mon[104450]: osdmap e685: 8 total, 8 up, 8 in 2024-09-13T14:31:16.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:16.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:16 smithi151 ceph-mon[108966]: pgmap v645: 89 pgs: 89 active+clean; 204 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 28 MiB/s rd, 3.4 MiB/s wr, 39 op/s 2024-09-13T14:31:16.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:16 smithi151 ceph-mon[108966]: osdmap e685: 8 total, 8 up, 8 in 2024-09-13T14:31:16.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:17.624 INFO:tasks.rados.rados.0.smithi151.stdout:2424: done (1 left) 2024-09-13T14:31:17.624 INFO:tasks.rados.rados.0.smithi151.stdout:2425: write oid 46 current snap is 315 2024-09-13T14:31:17.624 INFO:tasks.rados.rados.0.smithi151.stdout:2425: seq_num 909 ranges {477563=647348,1859426=587970,2979618=297945} 2024-09-13T14:31:17.631 INFO:tasks.rados.rados.0.smithi151.stdout:2425: writing smithi15186934-46 from 477563 to 1124911 tid 1 2024-09-13T14:31:17.638 INFO:tasks.rados.rados.0.smithi151.stdout:2425: writing smithi15186934-46 from 1859426 to 2447396 tid 2 2024-09-13T14:31:17.643 INFO:tasks.rados.rados.0.smithi151.stdout:2425: writing smithi15186934-46 from 2979618 to 3277563 tid 3 2024-09-13T14:31:17.643 INFO:tasks.rados.rados.0.smithi151.stdout:2426: write oid 9 current snap is 315 2024-09-13T14:31:17.643 INFO:tasks.rados.rados.0.smithi151.stdout:2426: seq_num 910 ranges {494370=722597,1917908=447981,3007041=287329} 2024-09-13T14:31:17.650 INFO:tasks.rados.rados.0.smithi151.stdout:2426: writing smithi15186934-9 from 494370 to 1216967 tid 1 2024-09-13T14:31:17.656 INFO:tasks.rados.rados.0.smithi151.stdout:2426: writing smithi15186934-9 from 1917908 to 2365889 tid 2 2024-09-13T14:31:17.661 INFO:tasks.rados.rados.0.smithi151.stdout:2426: writing smithi15186934-9 from 3007041 to 3294370 tid 3 2024-09-13T14:31:17.661 INFO:tasks.rados.rados.0.smithi151.stdout:2427: snap_create 2024-09-13T14:31:17.661 INFO:tasks.rados.rados.0.smithi151.stdout:2423: expect (ObjNum 858 snap 296 seq_num 858) 2024-09-13T14:31:17.739 INFO:tasks.rados.rados.0.smithi151.stdout:2425: finishing write tid 1 to smithi15186934-46 2024-09-13T14:31:17.739 INFO:tasks.rados.rados.0.smithi151.stdout:2425: finishing write tid 2 to smithi15186934-46 2024-09-13T14:31:17.739 INFO:tasks.rados.rados.0.smithi151.stdout:2425: finishing write tid 3 to smithi15186934-46 2024-09-13T14:31:17.739 INFO:tasks.rados.rados.0.smithi151.stdout:2425: finishing write tid 4 to smithi15186934-46 2024-09-13T14:31:17.739 INFO:tasks.rados.rados.0.smithi151.stdout:2425: finishing write tid 5 to smithi15186934-46 2024-09-13T14:31:17.739 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 46 v 845 (ObjNum 909 snap 315 seq_num 909) dirty exists 2024-09-13T14:31:17.739 INFO:tasks.rados.rados.0.smithi151.stdout:2425: left oid 46 (ObjNum 909 snap 315 seq_num 909) 2024-09-13T14:31:17.739 INFO:tasks.rados.rados.0.smithi151.stdout:2426: finishing write tid 1 to smithi15186934-9 2024-09-13T14:31:17.739 INFO:tasks.rados.rados.0.smithi151.stdout:2426: finishing write tid 2 to smithi15186934-9 2024-09-13T14:31:17.740 INFO:tasks.rados.rados.0.smithi151.stdout:2426: finishing write tid 3 to smithi15186934-9 2024-09-13T14:31:17.740 INFO:tasks.rados.rados.0.smithi151.stdout:2426: finishing write tid 4 to smithi15186934-9 2024-09-13T14:31:17.740 INFO:tasks.rados.rados.0.smithi151.stdout:2426: finishing write tid 5 to smithi15186934-9 2024-09-13T14:31:17.740 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 1010 (ObjNum 910 snap 315 seq_num 910) dirty exists 2024-09-13T14:31:17.740 INFO:tasks.rados.rados.0.smithi151.stdout:2426: left oid 9 (ObjNum 910 snap 315 seq_num 910) 2024-09-13T14:31:18.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:17 smithi184 ceph-mon[92527]: osdmap e686: 8 total, 8 up, 8 in 2024-09-13T14:31:18.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:17 smithi151 ceph-mon[104450]: osdmap e686: 8 total, 8 up, 8 in 2024-09-13T14:31:18.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:17 smithi151 ceph-mon[108966]: osdmap e686: 8 total, 8 up, 8 in 2024-09-13T14:31:18.630 INFO:tasks.rados.rados.0.smithi151.stdout:2423: done (3 left) 2024-09-13T14:31:18.631 INFO:tasks.rados.rados.0.smithi151.stdout:2425: done (2 left) 2024-09-13T14:31:18.631 INFO:tasks.rados.rados.0.smithi151.stdout:2426: done (1 left) 2024-09-13T14:31:18.631 INFO:tasks.rados.rados.0.smithi151.stdout:2427: done (0 left) 2024-09-13T14:31:18.631 INFO:tasks.rados.rados.0.smithi151.stdout:2428: write oid 38 current snap is 316 2024-09-13T14:31:18.631 INFO:tasks.rados.rados.0.smithi151.stdout:2428: seq_num 911 ranges {511177=797846,1976390=591639,3034464=276713} 2024-09-13T14:31:18.637 INFO:tasks.rados.rados.0.smithi151.stdout:2428: writing smithi15186934-38 from 511177 to 1309023 tid 1 2024-09-13T14:31:18.645 INFO:tasks.rados.rados.0.smithi151.stdout:2428: writing smithi15186934-38 from 1976390 to 2568029 tid 2 2024-09-13T14:31:18.650 INFO:tasks.rados.rados.0.smithi151.stdout:2428: writing smithi15186934-38 from 3034464 to 3311177 tid 3 2024-09-13T14:31:18.650 INFO:tasks.rados.rados.0.smithi151.stdout:2429: delete oid 17 current snap is 316 2024-09-13T14:31:18.650 INFO:tasks.rados.rados.0.smithi151.stdout:2428: finishing write tid 1 to smithi15186934-38 2024-09-13T14:31:18.650 INFO:tasks.rados.rados.0.smithi151.stdout:2428: finishing write tid 2 to smithi15186934-38 2024-09-13T14:31:18.653 INFO:tasks.rados.rados.0.smithi151.stdout:2428: finishing write tid 3 to smithi15186934-38 2024-09-13T14:31:18.653 INFO:tasks.rados.rados.0.smithi151.stdout:2429: done (1 left) 2024-09-13T14:31:18.653 INFO:tasks.rados.rados.0.smithi151.stdout:2430: rollback oid 46 current snap is 316 2024-09-13T14:31:18.653 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 46 to 312 2024-09-13T14:31:18.653 INFO:tasks.rados.rados.0.smithi151.stdout:2431: snap_create 2024-09-13T14:31:18.653 INFO:tasks.rados.rados.0.smithi151.stdout:2428: finishing write tid 4 to smithi15186934-38 2024-09-13T14:31:18.653 INFO:tasks.rados.rados.0.smithi151.stdout:2428: finishing write tid 5 to smithi15186934-38 2024-09-13T14:31:18.653 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 944 (ObjNum 911 snap 316 seq_num 911) dirty exists 2024-09-13T14:31:18.653 INFO:tasks.rados.rados.0.smithi151.stdout:2428: left oid 38 (ObjNum 911 snap 316 seq_num 911) 2024-09-13T14:31:18.655 INFO:tasks.rados.rados.0.smithi151.stdout:2430: finishing rollback tid 0 to smithi15186934-46 2024-09-13T14:31:18.655 INFO:tasks.rados.rados.0.smithi151.stdout:2430: finishing rollback tid 1 to smithi15186934-46 2024-09-13T14:31:18.655 INFO:tasks.rados.rados.0.smithi151.stdout:2430: finishing rollback tid 2 to smithi15186934-46 2024-09-13T14:31:18.655 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 46 v 849 (ObjNum 786 snap 275 seq_num 786) dirty exists 2024-09-13T14:31:19.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:18 smithi184 ceph-mon[92527]: pgmap v648: 89 pgs: 89 active+clean; 215 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 2.9 MiB/s wr, 30 op/s 2024-09-13T14:31:19.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:18 smithi184 ceph-mon[92527]: osdmap e687: 8 total, 8 up, 8 in 2024-09-13T14:31:19.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:18 smithi151 ceph-mon[104450]: pgmap v648: 89 pgs: 89 active+clean; 215 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 2.9 MiB/s wr, 30 op/s 2024-09-13T14:31:19.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:18 smithi151 ceph-mon[104450]: osdmap e687: 8 total, 8 up, 8 in 2024-09-13T14:31:19.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:18 smithi151 ceph-mon[108966]: pgmap v648: 89 pgs: 89 active+clean; 215 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 2.9 MiB/s wr, 30 op/s 2024-09-13T14:31:19.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:18 smithi151 ceph-mon[108966]: osdmap e687: 8 total, 8 up, 8 in 2024-09-13T14:31:19.555 INFO:tasks.rados.rados.0.smithi151.stdout:2428: done (2 left) 2024-09-13T14:31:19.555 INFO:tasks.rados.rados.0.smithi151.stdout:2430: done (1 left) 2024-09-13T14:31:19.555 INFO:tasks.rados.rados.0.smithi151.stdout:2431: done (0 left) 2024-09-13T14:31:19.555 INFO:tasks.rados.rados.0.smithi151.stdout:2432: read oid 24 snap -1 2024-09-13T14:31:19.555 INFO:tasks.rados.rados.0.smithi151.stdout:2432: expect deleted 2024-09-13T14:31:19.555 INFO:tasks.rados.rados.0.smithi151.stdout:2433: delete oid 2 current snap is 317 2024-09-13T14:31:19.561 INFO:tasks.rados.rados.0.smithi151.stdout:2432: done (1 left) 2024-09-13T14:31:19.561 INFO:tasks.rados.rados.0.smithi151.stdout:2433: done (0 left) 2024-09-13T14:31:19.561 INFO:tasks.rados.rados.0.smithi151.stdout:2434: delete oid 40 current snap is 317 2024-09-13T14:31:19.566 INFO:tasks.rados.rados.0.smithi151.stdout:2434: done (0 left) 2024-09-13T14:31:19.566 INFO:tasks.rados.rados.0.smithi151.stdout:2435: read oid 48 snap -1 2024-09-13T14:31:19.566 INFO:tasks.rados.rados.0.smithi151.stdout:2435: expect (ObjNum 854 snap 295 seq_num 854) 2024-09-13T14:31:19.566 INFO:tasks.rados.rados.0.smithi151.stdout:2436: write oid 41 current snap is 317 2024-09-13T14:31:19.566 INFO:tasks.rados.rados.0.smithi151.stdout:2436: seq_num 915 ranges {578405=415195,1410318=598977,2744156=584480} 2024-09-13T14:31:19.572 INFO:tasks.rados.rados.0.smithi151.stdout:2436: writing smithi15186934-41 from 578405 to 993600 tid 1 2024-09-13T14:31:19.578 INFO:tasks.rados.rados.0.smithi151.stdout:2436: writing smithi15186934-41 from 1410318 to 2009295 tid 2 2024-09-13T14:31:19.586 INFO:tasks.rados.rados.0.smithi151.stdout:2436: writing smithi15186934-41 from 2744156 to 3328636 tid 3 2024-09-13T14:31:19.586 INFO:tasks.rados.rados.0.smithi151.stdout:2437: write oid 3 current snap is 317 2024-09-13T14:31:19.586 INFO:tasks.rados.rados.0.smithi151.stdout:2437: seq_num 916 ranges {595212=490444,1868800=458988,2887932=507280} 2024-09-13T14:31:19.592 INFO:tasks.rados.rados.0.smithi151.stdout:2437: writing smithi15186934-3 from 595212 to 1085656 tid 1 2024-09-13T14:31:19.599 INFO:tasks.rados.rados.0.smithi151.stdout:2437: writing smithi15186934-3 from 1868800 to 2327788 tid 2 2024-09-13T14:31:19.606 INFO:tasks.rados.rados.0.smithi151.stdout:2437: writing smithi15186934-3 from 2887932 to 3395212 tid 3 2024-09-13T14:31:19.606 INFO:tasks.rados.rados.0.smithi151.stdout:2438: read oid 33 snap -1 2024-09-13T14:31:19.606 INFO:tasks.rados.rados.0.smithi151.stdout:2438: expect (ObjNum 887 snap 313 seq_num 887) 2024-09-13T14:31:19.606 INFO:tasks.rados.rados.0.smithi151.stdout:2439: read oid 32 snap -1 2024-09-13T14:31:19.606 INFO:tasks.rados.rados.0.smithi151.stdout:2439: expect (ObjNum 879 snap 311 seq_num 879) 2024-09-13T14:31:19.606 INFO:tasks.rados.rados.0.smithi151.stdout:2440: rollback oid 43 current snap is 317 2024-09-13T14:31:19.607 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 43 to 304 2024-09-13T14:31:19.607 INFO:tasks.rados.rados.0.smithi151.stdout:2435: expect (ObjNum 854 snap 295 seq_num 854) 2024-09-13T14:31:19.689 INFO:tasks.rados.rados.0.smithi151.stdout:2436: finishing write tid 1 to smithi15186934-41 2024-09-13T14:31:19.689 INFO:tasks.rados.rados.0.smithi151.stdout:2436: finishing write tid 2 to smithi15186934-41 2024-09-13T14:31:19.689 INFO:tasks.rados.rados.0.smithi151.stdout:2436: finishing write tid 3 to smithi15186934-41 2024-09-13T14:31:19.689 INFO:tasks.rados.rados.0.smithi151.stdout:2436: finishing write tid 4 to smithi15186934-41 2024-09-13T14:31:19.690 INFO:tasks.rados.rados.0.smithi151.stdout:2436: finishing write tid 5 to smithi15186934-41 2024-09-13T14:31:19.690 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 165 (ObjNum 915 snap 317 seq_num 915) dirty exists 2024-09-13T14:31:19.690 INFO:tasks.rados.rados.0.smithi151.stdout:2436: left oid 41 (ObjNum 915 snap 317 seq_num 915) 2024-09-13T14:31:19.690 INFO:tasks.rados.rados.0.smithi151.stdout:2437: finishing write tid 1 to smithi15186934-3 2024-09-13T14:31:19.690 INFO:tasks.rados.rados.0.smithi151.stdout:2435: done (5 left) 2024-09-13T14:31:19.690 INFO:tasks.rados.rados.0.smithi151.stdout:2436: done (4 left) 2024-09-13T14:31:19.690 INFO:tasks.rados.rados.0.smithi151.stdout:2441: delete oid 6 current snap is 317 2024-09-13T14:31:19.690 INFO:tasks.rados.rados.0.smithi151.stdout:2437: finishing write tid 2 to smithi15186934-3 2024-09-13T14:31:19.690 INFO:tasks.rados.rados.0.smithi151.stdout:2439: expect (ObjNum 879 snap 311 seq_num 879) 2024-09-13T14:31:19.880 INFO:tasks.rados.rados.0.smithi151.stdout:2440: finishing rollback tid 0 to smithi15186934-43 2024-09-13T14:31:19.880 INFO:tasks.rados.rados.0.smithi151.stdout:2440: finishing rollback tid 1 to smithi15186934-43 2024-09-13T14:31:19.880 INFO:tasks.rados.rados.0.smithi151.stdout:2440: finishing rollback tid 2 to smithi15186934-43 2024-09-13T14:31:19.880 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 43 v 1400 (ObjNum 828 snap 290 seq_num 828) dirty exists 2024-09-13T14:31:19.880 INFO:tasks.rados.rados.0.smithi151.stdout:2437: finishing write tid 3 to smithi15186934-3 2024-09-13T14:31:19.880 INFO:tasks.rados.rados.0.smithi151.stdout:2439: done (4 left) 2024-09-13T14:31:19.880 INFO:tasks.rados.rados.0.smithi151.stdout:2440: done (3 left) 2024-09-13T14:31:19.881 INFO:tasks.rados.rados.0.smithi151.stdout:2441: done (2 left) 2024-09-13T14:31:19.881 INFO:tasks.rados.rados.0.smithi151.stdout:2442: read oid 39 snap -1 2024-09-13T14:31:19.881 INFO:tasks.rados.rados.0.smithi151.stdout:2442: expect (ObjNum 886 snap 313 seq_num 886) 2024-09-13T14:31:19.881 INFO:tasks.rados.rados.0.smithi151.stdout:2443: read oid 2 snap 315 2024-09-13T14:31:19.881 INFO:tasks.rados.rados.0.smithi151.stdout:2443: expect (ObjNum 896 snap 314 seq_num 896) 2024-09-13T14:31:19.881 INFO:tasks.rados.rados.0.smithi151.stdout:2444: snap_create 2024-09-13T14:31:19.881 INFO:tasks.rados.rados.0.smithi151.stdout:2437: finishing write tid 4 to smithi15186934-3 2024-09-13T14:31:19.881 INFO:tasks.rados.rados.0.smithi151.stdout:2437: finishing write tid 5 to smithi15186934-3 2024-09-13T14:31:19.881 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 925 (ObjNum 916 snap 317 seq_num 916) dirty exists 2024-09-13T14:31:19.881 INFO:tasks.rados.rados.0.smithi151.stdout:2437: left oid 3 (ObjNum 916 snap 317 seq_num 916) 2024-09-13T14:31:19.881 INFO:tasks.rados.rados.0.smithi151.stdout:2438: expect (ObjNum 887 snap 313 seq_num 887) 2024-09-13T14:31:19.974 INFO:tasks.rados.rados.0.smithi151.stdout:2443: expect (ObjNum 896 snap 314 seq_num 896) 2024-09-13T14:31:20.075 INFO:tasks.rados.rados.0.smithi151.stdout:2442: expect (ObjNum 886 snap 313 seq_num 886) 2024-09-13T14:31:20.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:19 smithi184 ceph-mon[92527]: osdmap e688: 8 total, 8 up, 8 in 2024-09-13T14:31:20.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:19 smithi184 ceph-mon[92527]: osdmap e689: 8 total, 8 up, 8 in 2024-09-13T14:31:20.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:19 smithi151 ceph-mon[104450]: osdmap e688: 8 total, 8 up, 8 in 2024-09-13T14:31:20.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:19 smithi151 ceph-mon[104450]: osdmap e689: 8 total, 8 up, 8 in 2024-09-13T14:31:20.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:19 smithi151 ceph-mon[108966]: osdmap e688: 8 total, 8 up, 8 in 2024-09-13T14:31:20.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:19 smithi151 ceph-mon[108966]: osdmap e689: 8 total, 8 up, 8 in 2024-09-13T14:31:20.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:31:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:31:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:31:20.643 INFO:tasks.rados.rados.0.smithi151.stdout:2437: done (4 left) 2024-09-13T14:31:20.643 INFO:tasks.rados.rados.0.smithi151.stdout:2438: done (3 left) 2024-09-13T14:31:20.643 INFO:tasks.rados.rados.0.smithi151.stdout:2442: done (2 left) 2024-09-13T14:31:20.643 INFO:tasks.rados.rados.0.smithi151.stdout:2443: done (1 left) 2024-09-13T14:31:20.643 INFO:tasks.rados.rados.0.smithi151.stdout:2444: done (0 left) 2024-09-13T14:31:20.643 INFO:tasks.rados.rados.0.smithi151.stdout:2445: write oid 17 current snap is 318 2024-09-13T14:31:20.643 INFO:tasks.rados.rados.0.smithi151.stdout:2445: seq_num 918 ranges {628826=640942,1869411=462657,2826425=534649} 2024-09-13T14:31:20.649 INFO:tasks.rados.rados.0.smithi151.stdout:2445: writing smithi15186934-17 from 628826 to 1269768 tid 1 2024-09-13T14:31:20.655 INFO:tasks.rados.rados.0.smithi151.stdout:2445: writing smithi15186934-17 from 1869411 to 2332068 tid 2 2024-09-13T14:31:20.661 INFO:tasks.rados.rados.0.smithi151.stdout:2445: writing smithi15186934-17 from 2826425 to 3361074 tid 3 2024-09-13T14:31:20.662 INFO:tasks.rados.rados.0.smithi151.stdout:2446: read oid 39 snap -1 2024-09-13T14:31:20.662 INFO:tasks.rados.rados.0.smithi151.stdout:2446: expect (ObjNum 886 snap 313 seq_num 886) 2024-09-13T14:31:20.662 INFO:tasks.rados.rados.0.smithi151.stdout:2447: read oid 42 snap -1 2024-09-13T14:31:20.662 INFO:tasks.rados.rados.0.smithi151.stdout:2447: expect deleted 2024-09-13T14:31:20.662 INFO:tasks.rados.rados.0.smithi151.stdout:2448: delete oid 1 current snap is 318 2024-09-13T14:31:20.662 INFO:tasks.rados.rados.0.smithi151.stdout:2445: finishing write tid 1 to smithi15186934-17 2024-09-13T14:31:20.662 INFO:tasks.rados.rados.0.smithi151.stdout:2445: finishing write tid 2 to smithi15186934-17 2024-09-13T14:31:20.666 INFO:tasks.rados.rados.0.smithi151.stdout:2445: finishing write tid 3 to smithi15186934-17 2024-09-13T14:31:20.666 INFO:tasks.rados.rados.0.smithi151.stdout:2445: finishing write tid 4 to smithi15186934-17 2024-09-13T14:31:20.666 INFO:tasks.rados.rados.0.smithi151.stdout:2447: done (3 left) 2024-09-13T14:31:20.666 INFO:tasks.rados.rados.0.smithi151.stdout:2448: done (2 left) 2024-09-13T14:31:20.666 INFO:tasks.rados.rados.0.smithi151.stdout:2449: delete oid 9 current snap is 318 2024-09-13T14:31:20.666 INFO:tasks.rados.rados.0.smithi151.stdout:2445: finishing write tid 5 to smithi15186934-17 2024-09-13T14:31:20.666 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 1404 (ObjNum 918 snap 318 seq_num 918) dirty exists 2024-09-13T14:31:20.666 INFO:tasks.rados.rados.0.smithi151.stdout:2445: left oid 17 (ObjNum 918 snap 318 seq_num 918) 2024-09-13T14:31:20.668 INFO:tasks.rados.rados.0.smithi151.stdout:2445: done (2 left) 2024-09-13T14:31:20.668 INFO:tasks.rados.rados.0.smithi151.stdout:2449: done (1 left) 2024-09-13T14:31:20.668 INFO:tasks.rados.rados.0.smithi151.stdout:2450: snap_remove snap 313 2024-09-13T14:31:21.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:20 smithi184 ceph-mon[92527]: pgmap v651: 89 pgs: 89 active+clean; 216 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 2.4 MiB/s wr, 23 op/s 2024-09-13T14:31:21.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:20 smithi151 ceph-mon[104450]: pgmap v651: 89 pgs: 89 active+clean; 216 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 2.4 MiB/s wr, 23 op/s 2024-09-13T14:31:21.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:20 smithi151 ceph-mon[108966]: pgmap v651: 89 pgs: 89 active+clean; 216 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 2.4 MiB/s wr, 23 op/s 2024-09-13T14:31:21.662 INFO:tasks.rados.rados.0.smithi151.stdout:2450: done (1 left) 2024-09-13T14:31:21.663 INFO:tasks.rados.rados.0.smithi151.stdout:2451: snap_remove snap 314 2024-09-13T14:31:22.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:21 smithi184 ceph-mon[92527]: osdmap e690: 8 total, 8 up, 8 in 2024-09-13T14:31:22.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:21 smithi151 ceph-mon[104450]: osdmap e690: 8 total, 8 up, 8 in 2024-09-13T14:31:22.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:21 smithi151 ceph-mon[108966]: osdmap e690: 8 total, 8 up, 8 in 2024-09-13T14:31:22.660 INFO:tasks.rados.rados.0.smithi151.stdout:2451: done (1 left) 2024-09-13T14:31:22.660 INFO:tasks.rados.rados.0.smithi151.stdout:2452: read oid 37 snap -1 2024-09-13T14:31:22.660 INFO:tasks.rados.rados.0.smithi151.stdout:2452: expect (ObjNum 683 snap 222 seq_num 683) 2024-09-13T14:31:22.660 INFO:tasks.rados.rados.0.smithi151.stdout:2453: read oid 30 snap -1 2024-09-13T14:31:22.660 INFO:tasks.rados.rados.0.smithi151.stdout:2453: expect (ObjNum 707 snap 237 seq_num 707) 2024-09-13T14:31:22.661 INFO:tasks.rados.rados.0.smithi151.stdout:2454: read oid 7 snap 317 2024-09-13T14:31:22.661 INFO:tasks.rados.rados.0.smithi151.stdout:2454: expect (ObjNum 855 snap 295 seq_num 855) 2024-09-13T14:31:22.661 INFO:tasks.rados.rados.0.smithi151.stdout:2455: write oid 47 current snap is 318 2024-09-13T14:31:22.661 INFO:tasks.rados.rados.0.smithi151.stdout:2455: seq_num 921 ranges {679247=583042,2044857=609984,3192341=286906} 2024-09-13T14:31:22.668 INFO:tasks.rados.rados.0.smithi151.stdout:2455: writing smithi15186934-47 from 679247 to 1262289 tid 1 2024-09-13T14:31:22.676 INFO:tasks.rados.rados.0.smithi151.stdout:2455: writing smithi15186934-47 from 2044857 to 2654841 tid 2 2024-09-13T14:31:22.681 INFO:tasks.rados.rados.0.smithi151.stdout:2455: writing smithi15186934-47 from 3192341 to 3479247 tid 3 2024-09-13T14:31:22.681 INFO:tasks.rados.rados.0.smithi151.stdout:2456: snap_remove snap 312 2024-09-13T14:31:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:22 smithi184 ceph-mon[92527]: pgmap v654: 89 pgs: 89 active+clean; 219 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 4.1 MiB/s rd, 1.1 MiB/s wr, 8 op/s 2024-09-13T14:31:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:22 smithi184 ceph-mon[92527]: osdmap e691: 8 total, 8 up, 8 in 2024-09-13T14:31:23.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:22 smithi151 ceph-mon[104450]: pgmap v654: 89 pgs: 89 active+clean; 219 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 4.1 MiB/s rd, 1.1 MiB/s wr, 8 op/s 2024-09-13T14:31:23.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:22 smithi151 ceph-mon[104450]: osdmap e691: 8 total, 8 up, 8 in 2024-09-13T14:31:23.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:22 smithi151 ceph-mon[108966]: pgmap v654: 89 pgs: 89 active+clean; 219 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 4.1 MiB/s rd, 1.1 MiB/s wr, 8 op/s 2024-09-13T14:31:23.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:22 smithi151 ceph-mon[108966]: osdmap e691: 8 total, 8 up, 8 in 2024-09-13T14:31:23.671 INFO:tasks.rados.rados.0.smithi151.stdout:2456: done (5 left) 2024-09-13T14:31:23.671 INFO:tasks.rados.rados.0.smithi151.stdout:2457: delete oid 33 current snap is 318 2024-09-13T14:31:23.671 INFO:tasks.rados.rados.0.smithi151.stdout:2446: expect (ObjNum 886 snap 313 seq_num 886) 2024-09-13T14:31:23.804 INFO:tasks.rados.rados.0.smithi151.stdout:2453: expect (ObjNum 707 snap 237 seq_num 707) 2024-09-13T14:31:23.931 INFO:tasks.rados.rados.0.smithi151.stdout:2454: expect (ObjNum 855 snap 295 seq_num 855) 2024-09-13T14:31:24.058 INFO:tasks.rados.rados.0.smithi151.stdout:2452: expect (ObjNum 683 snap 222 seq_num 683) 2024-09-13T14:31:24.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:23 smithi184 ceph-mon[92527]: osdmap e692: 8 total, 8 up, 8 in 2024-09-13T14:31:24.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:23 smithi151 ceph-mon[104450]: osdmap e692: 8 total, 8 up, 8 in 2024-09-13T14:31:24.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:23 smithi151 ceph-mon[108966]: osdmap e692: 8 total, 8 up, 8 in 2024-09-13T14:31:24.280 INFO:tasks.rados.rados.0.smithi151.stdout:2455: finishing write tid 1 to smithi15186934-47 2024-09-13T14:31:24.281 INFO:tasks.rados.rados.0.smithi151.stdout:2455: finishing write tid 2 to smithi15186934-47 2024-09-13T14:31:24.281 INFO:tasks.rados.rados.0.smithi151.stdout:2455: finishing write tid 3 to smithi15186934-47 2024-09-13T14:31:24.281 INFO:tasks.rados.rados.0.smithi151.stdout:2455: finishing write tid 4 to smithi15186934-47 2024-09-13T14:31:24.281 INFO:tasks.rados.rados.0.smithi151.stdout:2455: finishing write tid 5 to smithi15186934-47 2024-09-13T14:31:24.281 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 1112 (ObjNum 921 snap 318 seq_num 921) dirty exists 2024-09-13T14:31:24.281 INFO:tasks.rados.rados.0.smithi151.stdout:2455: left oid 47 (ObjNum 921 snap 318 seq_num 921) 2024-09-13T14:31:24.281 INFO:tasks.rados.rados.0.smithi151.stdout:2446: done (5 left) 2024-09-13T14:31:24.281 INFO:tasks.rados.rados.0.smithi151.stdout:2452: done (4 left) 2024-09-13T14:31:24.281 INFO:tasks.rados.rados.0.smithi151.stdout:2453: done (3 left) 2024-09-13T14:31:24.282 INFO:tasks.rados.rados.0.smithi151.stdout:2454: done (2 left) 2024-09-13T14:31:24.282 INFO:tasks.rados.rados.0.smithi151.stdout:2455: done (1 left) 2024-09-13T14:31:24.282 INFO:tasks.rados.rados.0.smithi151.stdout:2457: done (0 left) 2024-09-13T14:31:24.282 INFO:tasks.rados.rados.0.smithi151.stdout:2458: write oid 15 current snap is 318 2024-09-13T14:31:24.282 INFO:tasks.rados.rados.0.smithi151.stdout:2458: seq_num 923 ranges {712861=733540,2161821=613653,3247187=265674} 2024-09-13T14:31:24.285 INFO:tasks.rados.rados.0.smithi151.stdout:2458: writing smithi15186934-15 from 712861 to 1446401 tid 1 2024-09-13T14:31:24.293 INFO:tasks.rados.rados.0.smithi151.stdout:2458: writing smithi15186934-15 from 2161821 to 2775474 tid 2 2024-09-13T14:31:24.297 INFO:tasks.rados.rados.0.smithi151.stdout:2458: writing smithi15186934-15 from 3247187 to 3512861 tid 3 2024-09-13T14:31:24.297 INFO:tasks.rados.rados.0.smithi151.stdout:2459: snap_remove snap 311 2024-09-13T14:31:24.557 INFO:tasks.rados.rados.0.smithi151.stdout:2458: finishing write tid 1 to smithi15186934-15 2024-09-13T14:31:24.558 INFO:tasks.rados.rados.0.smithi151.stdout:2458: finishing write tid 2 to smithi15186934-15 2024-09-13T14:31:24.558 INFO:tasks.rados.rados.0.smithi151.stdout:2458: finishing write tid 3 to smithi15186934-15 2024-09-13T14:31:24.558 INFO:tasks.rados.rados.0.smithi151.stdout:2458: finishing write tid 4 to smithi15186934-15 2024-09-13T14:31:24.558 INFO:tasks.rados.rados.0.smithi151.stdout:2458: finishing write tid 5 to smithi15186934-15 2024-09-13T14:31:24.558 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 15 v 995 (ObjNum 923 snap 318 seq_num 923) dirty exists 2024-09-13T14:31:24.558 INFO:tasks.rados.rados.0.smithi151.stdout:2458: left oid 15 (ObjNum 923 snap 318 seq_num 923) 2024-09-13T14:31:24.558 INFO:tasks.rados.rados.0.smithi151.stdout:2458: done (1 left) 2024-09-13T14:31:24.558 INFO:tasks.rados.rados.0.smithi151.stdout:2459: done (0 left) 2024-09-13T14:31:24.558 INFO:tasks.rados.rados.0.smithi151.stdout:2460: read oid 7 snap -1 2024-09-13T14:31:24.558 INFO:tasks.rados.rados.0.smithi151.stdout:2460: expect (ObjNum 855 snap 295 seq_num 855) 2024-09-13T14:31:24.558 INFO:tasks.rados.rados.0.smithi151.stdout:2461: delete oid 36 current snap is 318 2024-09-13T14:31:24.565 INFO:tasks.rados.rados.0.smithi151.stdout:2461: done (1 left) 2024-09-13T14:31:24.565 INFO:tasks.rados.rados.0.smithi151.stdout:2462: write oid 32 current snap is 318 2024-09-13T14:31:24.565 INFO:tasks.rados.rados.0.smithi151.stdout:2462: seq_num 925 ranges {746475=484038,1762432=617322,2785680=529906} 2024-09-13T14:31:24.572 INFO:tasks.rados.rados.0.smithi151.stdout:2462: writing smithi15186934-32 from 746475 to 1230513 tid 1 2024-09-13T14:31:24.578 INFO:tasks.rados.rados.0.smithi151.stdout:2462: writing smithi15186934-32 from 1762432 to 2379754 tid 2 2024-09-13T14:31:24.584 INFO:tasks.rados.rados.0.smithi151.stdout:2462: writing smithi15186934-32 from 2785680 to 3315586 tid 3 2024-09-13T14:31:24.584 INFO:tasks.rados.rados.0.smithi151.stdout:2463: delete oid 46 current snap is 318 2024-09-13T14:31:24.584 INFO:tasks.rados.rados.0.smithi151.stdout:2460: expect (ObjNum 855 snap 295 seq_num 855) 2024-09-13T14:31:24.713 INFO:tasks.rados.rados.0.smithi151.stdout:2462: finishing write tid 1 to smithi15186934-32 2024-09-13T14:31:24.713 INFO:tasks.rados.rados.0.smithi151.stdout:2462: finishing write tid 2 to smithi15186934-32 2024-09-13T14:31:24.713 INFO:tasks.rados.rados.0.smithi151.stdout:2462: finishing write tid 3 to smithi15186934-32 2024-09-13T14:31:24.713 INFO:tasks.rados.rados.0.smithi151.stdout:2462: finishing write tid 4 to smithi15186934-32 2024-09-13T14:31:24.714 INFO:tasks.rados.rados.0.smithi151.stdout:2462: finishing write tid 5 to smithi15186934-32 2024-09-13T14:31:24.714 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 32 v 1031 (ObjNum 925 snap 318 seq_num 925) dirty exists 2024-09-13T14:31:24.714 INFO:tasks.rados.rados.0.smithi151.stdout:2462: left oid 32 (ObjNum 925 snap 318 seq_num 925) 2024-09-13T14:31:24.714 INFO:tasks.rados.rados.0.smithi151.stdout:2460: done (2 left) 2024-09-13T14:31:24.714 INFO:tasks.rados.rados.0.smithi151.stdout:2462: done (1 left) 2024-09-13T14:31:24.714 INFO:tasks.rados.rados.0.smithi151.stdout:2463: done (0 left) 2024-09-13T14:31:24.714 INFO:tasks.rados.rados.0.smithi151.stdout:2464: read oid 2 snap -1 2024-09-13T14:31:24.714 INFO:tasks.rados.rados.0.smithi151.stdout:2464: expect deleted 2024-09-13T14:31:24.714 INFO:tasks.rados.rados.0.smithi151.stdout:2465: delete oid 35 current snap is 318 2024-09-13T14:31:24.715 INFO:tasks.rados.rados.0.smithi151.stdout:2464: done (1 left) 2024-09-13T14:31:24.715 INFO:tasks.rados.rados.0.smithi151.stdout:2465: done (0 left) 2024-09-13T14:31:24.715 INFO:tasks.rados.rados.0.smithi151.stdout:2466: delete oid 40 current snap is 318 2024-09-13T14:31:24.717 INFO:tasks.rados.rados.0.smithi151.stdout:2466: done (0 left) 2024-09-13T14:31:24.717 INFO:tasks.rados.rados.0.smithi151.stdout:2467: rollback oid 15 current snap is 318 2024-09-13T14:31:24.717 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 15 to 299 2024-09-13T14:31:24.717 INFO:tasks.rados.rados.0.smithi151.stdout:2468: delete oid 33 current snap is 318 2024-09-13T14:31:24.719 INFO:tasks.rados.rados.0.smithi151.stdout:2468: done (1 left) 2024-09-13T14:31:24.719 INFO:tasks.rados.rados.0.smithi151.stdout:2469: write oid 8 current snap is 318 2024-09-13T14:31:24.719 INFO:tasks.rados.rados.0.smithi151.stdout:2469: seq_num 930 ranges {430510=576636,1654842=768318,2922795=502619} 2024-09-13T14:31:24.724 INFO:tasks.rados.rados.0.smithi151.stdout:2469: writing smithi15186934-8 from 430510 to 1007146 tid 1 2024-09-13T14:31:24.732 INFO:tasks.rados.rados.0.smithi151.stdout:2469: writing smithi15186934-8 from 1654842 to 2423160 tid 2 2024-09-13T14:31:24.737 INFO:tasks.rados.rados.0.smithi151.stdout:2469: writing smithi15186934-8 from 2922795 to 3425414 tid 3 2024-09-13T14:31:24.737 INFO:tasks.rados.rados.0.smithi151.stdout:2470: delete oid 5 current snap is 318 2024-09-13T14:31:24.738 INFO:tasks.rados.rados.0.smithi151.stdout:2467: finishing rollback tid 0 to smithi15186934-15 2024-09-13T14:31:24.738 INFO:tasks.rados.rados.0.smithi151.stdout:2467: finishing rollback tid 1 to smithi15186934-15 2024-09-13T14:31:24.738 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 15 v 999 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-13T14:31:24.738 INFO:tasks.rados.rados.0.smithi151.stdout:2469: finishing write tid 1 to smithi15186934-8 2024-09-13T14:31:24.738 INFO:tasks.rados.rados.0.smithi151.stdout:2469: finishing write tid 2 to smithi15186934-8 2024-09-13T14:31:24.740 INFO:tasks.rados.rados.0.smithi151.stdout:2467: done (2 left) 2024-09-13T14:31:24.740 INFO:tasks.rados.rados.0.smithi151.stdout:2470: done (1 left) 2024-09-13T14:31:24.740 INFO:tasks.rados.rados.0.smithi151.stdout:2471: rollback oid 21 current snap is 318 2024-09-13T14:31:24.740 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 21 to 290 2024-09-13T14:31:24.740 INFO:tasks.rados.rados.0.smithi151.stdout:2472: read oid 20 snap 316 2024-09-13T14:31:24.740 INFO:tasks.rados.rados.0.smithi151.stdout:2472: expect (ObjNum 806 snap 283 seq_num 806) 2024-09-13T14:31:24.741 INFO:tasks.rados.rados.0.smithi151.stdout:2473: read oid 17 snap 316 2024-09-13T14:31:24.741 INFO:tasks.rados.rados.0.smithi151.stdout:2473: expect deleted 2024-09-13T14:31:24.741 INFO:tasks.rados.rados.0.smithi151.stdout:2474: write oid 10 current snap is 318 2024-09-13T14:31:24.741 INFO:tasks.rados.rados.0.smithi151.stdout:2474: seq_num 932 ranges {464124=727134,1771806=771987,2977641=641516} 2024-09-13T14:31:24.746 INFO:tasks.rados.rados.0.smithi151.stdout:2474: writing smithi15186934-10 from 464124 to 1191258 tid 1 2024-09-13T14:31:24.754 INFO:tasks.rados.rados.0.smithi151.stdout:2474: writing smithi15186934-10 from 1771806 to 2543793 tid 2 2024-09-13T14:31:24.760 INFO:tasks.rados.rados.0.smithi151.stdout:2474: writing smithi15186934-10 from 2977641 to 3619157 tid 3 2024-09-13T14:31:24.760 INFO:tasks.rados.rados.0.smithi151.stdout:2475: snap_remove snap 297 2024-09-13T14:31:24.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:31:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:31:24] "GET /metrics HTTP/1.1" 200 36506 "" "Prometheus/2.51.0" 2024-09-13T14:31:24.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:24 smithi151 ceph-mon[104450]: pgmap v657: 89 pgs: 89 active+clean; 223 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 9.9 MiB/s rd, 1.9 MiB/s wr, 16 op/s 2024-09-13T14:31:24.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:24 smithi151 ceph-mon[104450]: osdmap e693: 8 total, 8 up, 8 in 2024-09-13T14:31:24.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:24 smithi151 ceph-mon[104450]: osdmap e694: 8 total, 8 up, 8 in 2024-09-13T14:31:24.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:24 smithi151 ceph-mon[108966]: pgmap v657: 89 pgs: 89 active+clean; 223 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 9.9 MiB/s rd, 1.9 MiB/s wr, 16 op/s 2024-09-13T14:31:24.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:24 smithi151 ceph-mon[108966]: osdmap e693: 8 total, 8 up, 8 in 2024-09-13T14:31:24.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:24 smithi151 ceph-mon[108966]: osdmap e694: 8 total, 8 up, 8 in 2024-09-13T14:31:25.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:24 smithi184 ceph-mon[92527]: pgmap v657: 89 pgs: 89 active+clean; 223 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 9.9 MiB/s rd, 1.9 MiB/s wr, 16 op/s 2024-09-13T14:31:25.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:24 smithi184 ceph-mon[92527]: osdmap e693: 8 total, 8 up, 8 in 2024-09-13T14:31:25.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:24 smithi184 ceph-mon[92527]: osdmap e694: 8 total, 8 up, 8 in 2024-09-13T14:31:25.675 INFO:tasks.rados.rados.0.smithi151.stdout:2475: done (5 left) 2024-09-13T14:31:25.675 INFO:tasks.rados.rados.0.smithi151.stdout:2476: snap_create 2024-09-13T14:31:25.675 INFO:tasks.rados.rados.0.smithi151.stdout:2469: finishing write tid 3 to smithi15186934-8 2024-09-13T14:31:25.675 INFO:tasks.rados.rados.0.smithi151.stdout:2469: finishing write tid 4 to smithi15186934-8 2024-09-13T14:31:25.675 INFO:tasks.rados.rados.0.smithi151.stdout:2469: finishing write tid 5 to smithi15186934-8 2024-09-13T14:31:25.675 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 765 (ObjNum 930 snap 318 seq_num 930) dirty exists 2024-09-13T14:31:25.675 INFO:tasks.rados.rados.0.smithi151.stdout:2469: left oid 8 (ObjNum 930 snap 318 seq_num 930) 2024-09-13T14:31:25.675 INFO:tasks.rados.rados.0.smithi151.stdout:2471: finishing rollback tid 1 to smithi15186934-21 2024-09-13T14:31:25.675 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 21 v 493 (ObjNum 305 snap 0 seq_num 3680548677) dirty dne 2024-09-13T14:31:25.675 INFO:tasks.rados.rados.0.smithi151.stdout:2474: finishing write tid 1 to smithi15186934-10 2024-09-13T14:31:25.675 INFO:tasks.rados.rados.0.smithi151.stdout:2472: expect (ObjNum 806 snap 283 seq_num 806) 2024-09-13T14:31:25.827 INFO:tasks.rados.rados.0.smithi151.stdout:2474: finishing write tid 2 to smithi15186934-10 2024-09-13T14:31:25.827 INFO:tasks.rados.rados.0.smithi151.stdout:2474: finishing write tid 3 to smithi15186934-10 2024-09-13T14:31:25.827 INFO:tasks.rados.rados.0.smithi151.stdout:2474: finishing write tid 4 to smithi15186934-10 2024-09-13T14:31:25.827 INFO:tasks.rados.rados.0.smithi151.stdout:2474: finishing write tid 5 to smithi15186934-10 2024-09-13T14:31:25.827 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 868 (ObjNum 932 snap 318 seq_num 932) dirty exists 2024-09-13T14:31:25.827 INFO:tasks.rados.rados.0.smithi151.stdout:2474: left oid 10 (ObjNum 932 snap 318 seq_num 932) 2024-09-13T14:31:26.685 INFO:tasks.rados.rados.0.smithi151.stdout:2469: done (5 left) 2024-09-13T14:31:26.685 INFO:tasks.rados.rados.0.smithi151.stdout:2471: done (4 left) 2024-09-13T14:31:26.685 INFO:tasks.rados.rados.0.smithi151.stdout:2472: done (3 left) 2024-09-13T14:31:26.685 INFO:tasks.rados.rados.0.smithi151.stdout:2473: done (2 left) 2024-09-13T14:31:26.685 INFO:tasks.rados.rados.0.smithi151.stdout:2474: done (1 left) 2024-09-13T14:31:26.685 INFO:tasks.rados.rados.0.smithi151.stdout:2476: done (0 left) 2024-09-13T14:31:26.686 INFO:tasks.rados.rados.0.smithi151.stdout:2477: read oid 7 snap -1 2024-09-13T14:31:26.686 INFO:tasks.rados.rados.0.smithi151.stdout:2477: expect (ObjNum 855 snap 295 seq_num 855) 2024-09-13T14:31:26.686 INFO:tasks.rados.rados.0.smithi151.stdout:2478: rollback oid 40 current snap is 319 2024-09-13T14:31:26.686 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 40 to 318 2024-09-13T14:31:26.686 INFO:tasks.rados.rados.0.smithi151.stdout:2479: snap_remove snap 316 2024-09-13T14:31:27.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:26 smithi184 ceph-mon[92527]: pgmap v660: 89 pgs: 89 active+clean; 222 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 1.2 MiB/s wr, 17 op/s 2024-09-13T14:31:27.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:26 smithi184 ceph-mon[92527]: osdmap e695: 8 total, 8 up, 8 in 2024-09-13T14:31:27.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:26 smithi151 ceph-mon[104450]: pgmap v660: 89 pgs: 89 active+clean; 222 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 1.2 MiB/s wr, 17 op/s 2024-09-13T14:31:27.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:26 smithi151 ceph-mon[104450]: osdmap e695: 8 total, 8 up, 8 in 2024-09-13T14:31:27.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:26 smithi151 ceph-mon[108966]: pgmap v660: 89 pgs: 89 active+clean; 222 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 1.2 MiB/s wr, 17 op/s 2024-09-13T14:31:27.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:26 smithi151 ceph-mon[108966]: osdmap e695: 8 total, 8 up, 8 in 2024-09-13T14:31:27.686 INFO:tasks.rados.rados.0.smithi151.stdout:2479: done (2 left) 2024-09-13T14:31:27.686 INFO:tasks.rados.rados.0.smithi151.stdout:2480: snap_remove snap 307 2024-09-13T14:31:28.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:27 smithi184 ceph-mon[92527]: osdmap e696: 8 total, 8 up, 8 in 2024-09-13T14:31:28.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:27 smithi184 ceph-mon[92527]: osdmap e697: 8 total, 8 up, 8 in 2024-09-13T14:31:28.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:27 smithi151 ceph-mon[104450]: osdmap e696: 8 total, 8 up, 8 in 2024-09-13T14:31:28.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:27 smithi151 ceph-mon[104450]: osdmap e697: 8 total, 8 up, 8 in 2024-09-13T14:31:28.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:27 smithi151 ceph-mon[108966]: osdmap e696: 8 total, 8 up, 8 in 2024-09-13T14:31:28.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:27 smithi151 ceph-mon[108966]: osdmap e697: 8 total, 8 up, 8 in 2024-09-13T14:31:28.683 INFO:tasks.rados.rados.0.smithi151.stdout:2480: done (2 left) 2024-09-13T14:31:28.683 INFO:tasks.rados.rados.0.smithi151.stdout:2481: rollback oid 41 current snap is 319 2024-09-13T14:31:28.683 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 41 to 317 2024-09-13T14:31:28.683 INFO:tasks.rados.rados.0.smithi151.stdout:2478: finishing rollback tid 1 to smithi15186934-40 2024-09-13T14:31:28.683 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 40 v 880 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-13T14:31:28.683 INFO:tasks.rados.rados.0.smithi151.stdout:2477: expect (ObjNum 855 snap 295 seq_num 855) 2024-09-13T14:31:28.817 INFO:tasks.rados.rados.0.smithi151.stdout:2477: done (2 left) 2024-09-13T14:31:28.817 INFO:tasks.rados.rados.0.smithi151.stdout:2478: done (1 left) 2024-09-13T14:31:28.817 INFO:tasks.rados.rados.0.smithi151.stdout:2482: read oid 3 snap -1 2024-09-13T14:31:28.818 INFO:tasks.rados.rados.0.smithi151.stdout:2482: expect (ObjNum 916 snap 317 seq_num 916) 2024-09-13T14:31:28.818 INFO:tasks.rados.rados.0.smithi151.stdout:2483: snap_remove snap 290 2024-09-13T14:31:29.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:28 smithi184 ceph-mon[92527]: pgmap v663: 89 pgs: 89 active+clean; 222 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 1.2 MiB/s wr, 23 op/s 2024-09-13T14:31:29.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:28 smithi184 ceph-mon[92527]: osdmap e698: 8 total, 8 up, 8 in 2024-09-13T14:31:29.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:28 smithi151 ceph-mon[104450]: pgmap v663: 89 pgs: 89 active+clean; 222 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 1.2 MiB/s wr, 23 op/s 2024-09-13T14:31:29.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:28 smithi151 ceph-mon[104450]: osdmap e698: 8 total, 8 up, 8 in 2024-09-13T14:31:29.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:28 smithi151 ceph-mon[108966]: pgmap v663: 89 pgs: 89 active+clean; 222 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 1.2 MiB/s wr, 23 op/s 2024-09-13T14:31:29.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:28 smithi151 ceph-mon[108966]: osdmap e698: 8 total, 8 up, 8 in 2024-09-13T14:31:29.557 INFO:tasks.rados.rados.0.smithi151.stdout:2483: done (2 left) 2024-09-13T14:31:29.557 INFO:tasks.rados.rados.0.smithi151.stdout:2484: snap_create 2024-09-13T14:31:29.557 INFO:tasks.rados.rados.0.smithi151.stdout:2481: finishing rollback tid 0 to smithi15186934-41 2024-09-13T14:31:29.557 INFO:tasks.rados.rados.0.smithi151.stdout:2481: finishing rollback tid 1 to smithi15186934-41 2024-09-13T14:31:29.557 INFO:tasks.rados.rados.0.smithi151.stdout:2481: finishing rollback tid 2 to smithi15186934-41 2024-09-13T14:31:29.557 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 179 (ObjNum 915 snap 317 seq_num 915) dirty exists 2024-09-13T14:31:29.557 INFO:tasks.rados.rados.0.smithi151.stdout:2482: expect (ObjNum 916 snap 317 seq_num 916) 2024-09-13T14:31:30.560 INFO:tasks.rados.rados.0.smithi151.stdout:2481: done (2 left) 2024-09-13T14:31:30.560 INFO:tasks.rados.rados.0.smithi151.stdout:2482: done (1 left) 2024-09-13T14:31:30.560 INFO:tasks.rados.rados.0.smithi151.stdout:2484: done (0 left) 2024-09-13T14:31:30.560 INFO:tasks.rados.rados.0.smithi151.stdout:2485: write oid 39 current snap is 320 2024-09-13T14:31:30.560 INFO:tasks.rados.rados.0.smithi151.stdout:2485: seq_num 933 ranges {480931=402383,1313935=515645,2488711=452788,3443230=237701} 2024-09-13T14:31:30.563 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:31:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:31:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:31:30.565 INFO:tasks.rados.rados.0.smithi151.stdout:2485: writing smithi15186934-39 from 480931 to 883314 tid 1 2024-09-13T14:31:30.570 INFO:tasks.rados.rados.0.smithi151.stdout:2485: writing smithi15186934-39 from 1313935 to 1829580 tid 2 2024-09-13T14:31:30.576 INFO:tasks.rados.rados.0.smithi151.stdout:2485: writing smithi15186934-39 from 2488711 to 2941499 tid 3 2024-09-13T14:31:30.580 INFO:tasks.rados.rados.0.smithi151.stdout:2485: writing smithi15186934-39 from 3443230 to 3680931 tid 4 2024-09-13T14:31:30.581 INFO:tasks.rados.rados.0.smithi151.stdout:2486: snap_remove snap 315 2024-09-13T14:31:30.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:30 smithi184 ceph-mon[92527]: pgmap v666: 89 pgs: 89 active+clean; 224 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 2.1 MiB/s wr, 21 op/s 2024-09-13T14:31:30.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:30 smithi184 ceph-mon[92527]: osdmap e699: 8 total, 8 up, 8 in 2024-09-13T14:31:30.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:30 smithi151 ceph-mon[108966]: pgmap v666: 89 pgs: 89 active+clean; 224 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 2.1 MiB/s wr, 21 op/s 2024-09-13T14:31:30.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:30 smithi151 ceph-mon[108966]: osdmap e699: 8 total, 8 up, 8 in 2024-09-13T14:31:30.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:30 smithi151 ceph-mon[104450]: pgmap v666: 89 pgs: 89 active+clean; 224 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 2.1 MiB/s wr, 21 op/s 2024-09-13T14:31:30.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:30 smithi151 ceph-mon[104450]: osdmap e699: 8 total, 8 up, 8 in 2024-09-13T14:31:31.574 INFO:tasks.rados.rados.0.smithi151.stdout:2486: done (1 left) 2024-09-13T14:31:31.574 INFO:tasks.rados.rados.0.smithi151.stdout:2487: snap_create 2024-09-13T14:31:31.574 INFO:tasks.rados.rados.0.smithi151.stdout:2485: finishing write tid 1 to smithi15186934-39 2024-09-13T14:31:31.574 INFO:tasks.rados.rados.0.smithi151.stdout:2485: finishing write tid 2 to smithi15186934-39 2024-09-13T14:31:31.574 INFO:tasks.rados.rados.0.smithi151.stdout:2485: finishing write tid 3 to smithi15186934-39 2024-09-13T14:31:31.574 INFO:tasks.rados.rados.0.smithi151.stdout:2485: finishing write tid 4 to smithi15186934-39 2024-09-13T14:31:31.574 INFO:tasks.rados.rados.0.smithi151.stdout:2485: finishing write tid 5 to smithi15186934-39 2024-09-13T14:31:31.574 INFO:tasks.rados.rados.0.smithi151.stdout:2485: finishing write tid 6 to smithi15186934-39 2024-09-13T14:31:31.574 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 984 (ObjNum 933 snap 320 seq_num 933) dirty exists 2024-09-13T14:31:31.574 INFO:tasks.rados.rados.0.smithi151.stdout:2485: left oid 39 (ObjNum 933 snap 320 seq_num 933) 2024-09-13T14:31:31.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:31 smithi184 ceph-mon[92527]: osdmap e700: 8 total, 8 up, 8 in 2024-09-13T14:31:31.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:31.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:31 smithi151 ceph-mon[104450]: osdmap e700: 8 total, 8 up, 8 in 2024-09-13T14:31:31.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:31.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:31 smithi151 ceph-mon[108966]: osdmap e700: 8 total, 8 up, 8 in 2024-09-13T14:31:31.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:32.574 INFO:tasks.rados.rados.0.smithi151.stdout:2485: done (1 left) 2024-09-13T14:31:32.574 INFO:tasks.rados.rados.0.smithi151.stdout:2487: done (0 left) 2024-09-13T14:31:32.574 INFO:tasks.rados.rados.0.smithi151.stdout:2488: write oid 13 current snap is 321 2024-09-13T14:31:32.575 INFO:tasks.rados.rados.0.smithi151.stdout:2488: seq_num 934 ranges {497738=477632,1772417=775656,3316134=381604} 2024-09-13T14:31:32.580 INFO:tasks.rados.rados.0.smithi151.stdout:2488: writing smithi15186934-13 from 497738 to 975370 tid 1 2024-09-13T14:31:32.589 INFO:tasks.rados.rados.0.smithi151.stdout:2488: writing smithi15186934-13 from 1772417 to 2548073 tid 2 2024-09-13T14:31:32.595 INFO:tasks.rados.rados.0.smithi151.stdout:2488: writing smithi15186934-13 from 3316134 to 3697738 tid 3 2024-09-13T14:31:32.595 INFO:tasks.rados.rados.0.smithi151.stdout:2489: delete oid 35 current snap is 321 2024-09-13T14:31:32.595 INFO:tasks.rados.rados.0.smithi151.stdout:2488: finishing write tid 1 to smithi15186934-13 2024-09-13T14:31:32.596 INFO:tasks.rados.rados.0.smithi151.stdout:2488: finishing write tid 2 to smithi15186934-13 2024-09-13T14:31:32.599 INFO:tasks.rados.rados.0.smithi151.stdout:2488: finishing write tid 3 to smithi15186934-13 2024-09-13T14:31:32.599 INFO:tasks.rados.rados.0.smithi151.stdout:2488: finishing write tid 4 to smithi15186934-13 2024-09-13T14:31:32.600 INFO:tasks.rados.rados.0.smithi151.stdout:2488: finishing write tid 5 to smithi15186934-13 2024-09-13T14:31:32.600 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 13 v 973 (ObjNum 934 snap 321 seq_num 934) dirty exists 2024-09-13T14:31:32.600 INFO:tasks.rados.rados.0.smithi151.stdout:2488: left oid 13 (ObjNum 934 snap 321 seq_num 934) 2024-09-13T14:31:32.600 INFO:tasks.rados.rados.0.smithi151.stdout:2488: done (1 left) 2024-09-13T14:31:32.600 INFO:tasks.rados.rados.0.smithi151.stdout:2489: done (0 left) 2024-09-13T14:31:32.600 INFO:tasks.rados.rados.0.smithi151.stdout:2490: snap_create 2024-09-13T14:31:32.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:32 smithi151 ceph-mon[104450]: pgmap v669: 89 pgs: 89 active+clean; 219 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.9 MiB/s rd, 917 KiB/s wr, 6 op/s 2024-09-13T14:31:32.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:32 smithi151 ceph-mon[104450]: osdmap e701: 8 total, 8 up, 8 in 2024-09-13T14:31:32.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:31:32.853 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:32 smithi151 ceph-mon[108966]: pgmap v669: 89 pgs: 89 active+clean; 219 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.9 MiB/s rd, 917 KiB/s wr, 6 op/s 2024-09-13T14:31:32.853 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:32 smithi151 ceph-mon[108966]: osdmap e701: 8 total, 8 up, 8 in 2024-09-13T14:31:32.853 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:31:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:32 smithi184 ceph-mon[92527]: pgmap v669: 89 pgs: 89 active+clean; 219 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.9 MiB/s rd, 917 KiB/s wr, 6 op/s 2024-09-13T14:31:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:32 smithi184 ceph-mon[92527]: osdmap e701: 8 total, 8 up, 8 in 2024-09-13T14:31:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:31:33.593 INFO:tasks.rados.rados.0.smithi151.stdout:2490: done (0 left) 2024-09-13T14:31:33.593 INFO:tasks.rados.rados.0.smithi151.stdout:2491: snap_remove snap 319 2024-09-13T14:31:33.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:33 smithi184 ceph-mon[92527]: osdmap e702: 8 total, 8 up, 8 in 2024-09-13T14:31:33.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:33 smithi151 ceph-mon[104450]: osdmap e702: 8 total, 8 up, 8 in 2024-09-13T14:31:33.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:33 smithi151 ceph-mon[108966]: osdmap e702: 8 total, 8 up, 8 in 2024-09-13T14:31:34.559 INFO:tasks.rados.rados.0.smithi151.stdout:2491: done (0 left) 2024-09-13T14:31:34.559 INFO:tasks.rados.rados.0.smithi151.stdout:2492: read oid 30 snap -1 2024-09-13T14:31:34.559 INFO:tasks.rados.rados.0.smithi151.stdout:2492: expect (ObjNum 707 snap 237 seq_num 707) 2024-09-13T14:31:34.559 INFO:tasks.rados.rados.0.smithi151.stdout:2493: read oid 44 snap -1 2024-09-13T14:31:34.559 INFO:tasks.rados.rados.0.smithi151.stdout:2493: expect deleted 2024-09-13T14:31:34.559 INFO:tasks.rados.rados.0.smithi151.stdout:2494: rollback oid 20 current snap is 322 2024-09-13T14:31:34.559 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 20 to 308 2024-09-13T14:31:34.559 INFO:tasks.rados.rados.0.smithi151.stdout:2495: delete oid 23 current snap is 322 2024-09-13T14:31:34.563 INFO:tasks.rados.rados.0.smithi151.stdout:2494: finishing rollback tid 0 to smithi15186934-20 2024-09-13T14:31:34.564 INFO:tasks.rados.rados.0.smithi151.stdout:2494: finishing rollback tid 1 to smithi15186934-20 2024-09-13T14:31:34.564 INFO:tasks.rados.rados.0.smithi151.stdout:2494: finishing rollback tid 2 to smithi15186934-20 2024-09-13T14:31:34.564 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 603 (ObjNum 806 snap 283 seq_num 806) dirty exists 2024-09-13T14:31:34.564 INFO:tasks.rados.rados.0.smithi151.stdout:2493: done (3 left) 2024-09-13T14:31:34.565 INFO:tasks.rados.rados.0.smithi151.stdout:2494: done (2 left) 2024-09-13T14:31:34.565 INFO:tasks.rados.rados.0.smithi151.stdout:2495: done (1 left) 2024-09-13T14:31:34.565 INFO:tasks.rados.rados.0.smithi151.stdout:2496: read oid 25 snap 276 2024-09-13T14:31:34.565 INFO:tasks.rados.rados.0.smithi151.stdout:2496: expect (ObjNum 670 snap 216 seq_num 670) 2024-09-13T14:31:34.565 INFO:tasks.rados.rados.0.smithi151.stdout:2497: snap_create 2024-09-13T14:31:34.570 INFO:tasks.rados.rados.0.smithi151.stdout:2492: expect (ObjNum 707 snap 237 seq_num 707) 2024-09-13T14:31:34.708 INFO:tasks.rados.rados.0.smithi151.stdout:2496: expect (ObjNum 670 snap 216 seq_num 670) 2024-09-13T14:31:34.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:34 smithi184 ceph-mon[92527]: pgmap v672: 89 pgs: 89 active+clean; 220 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.9 MiB/s rd, 393 KiB/s wr, 4 op/s 2024-09-13T14:31:34.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:34 smithi184 ceph-mon[92527]: osdmap e703: 8 total, 8 up, 8 in 2024-09-13T14:31:34.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:34 smithi184 ceph-mon[92527]: osdmap e704: 8 total, 8 up, 8 in 2024-09-13T14:31:34.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:34 smithi151 ceph-mon[108966]: pgmap v672: 89 pgs: 89 active+clean; 220 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.9 MiB/s rd, 393 KiB/s wr, 4 op/s 2024-09-13T14:31:34.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:34 smithi151 ceph-mon[108966]: osdmap e703: 8 total, 8 up, 8 in 2024-09-13T14:31:34.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:34 smithi151 ceph-mon[108966]: osdmap e704: 8 total, 8 up, 8 in 2024-09-13T14:31:34.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:34 smithi151 ceph-mon[104450]: pgmap v672: 89 pgs: 89 active+clean; 220 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.9 MiB/s rd, 393 KiB/s wr, 4 op/s 2024-09-13T14:31:34.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:34 smithi151 ceph-mon[104450]: osdmap e703: 8 total, 8 up, 8 in 2024-09-13T14:31:34.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:34 smithi151 ceph-mon[104450]: osdmap e704: 8 total, 8 up, 8 in 2024-09-13T14:31:34.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:31:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:31:34] "GET /metrics HTTP/1.1" 200 36510 "" "Prometheus/2.51.0" 2024-09-13T14:31:35.562 INFO:tasks.rados.rados.0.smithi151.stdout:2492: done (2 left) 2024-09-13T14:31:35.562 INFO:tasks.rados.rados.0.smithi151.stdout:2496: done (1 left) 2024-09-13T14:31:35.562 INFO:tasks.rados.rados.0.smithi151.stdout:2497: done (0 left) 2024-09-13T14:31:35.563 INFO:tasks.rados.rados.0.smithi151.stdout:2498: write oid 21 current snap is 323 2024-09-13T14:31:35.563 INFO:tasks.rados.rados.0.smithi151.stdout:2498: seq_num 937 ranges {548159=419732,1664216=522983,2714756=614229} 2024-09-13T14:31:35.567 INFO:tasks.rados.rados.0.smithi151.stdout:2498: writing smithi15186934-21 from 548159 to 967891 tid 1 2024-09-13T14:31:35.574 INFO:tasks.rados.rados.0.smithi151.stdout:2498: writing smithi15186934-21 from 1664216 to 2187199 tid 2 2024-09-13T14:31:35.580 INFO:tasks.rados.rados.0.smithi151.stdout:2498: writing smithi15186934-21 from 2714756 to 3328985 tid 3 2024-09-13T14:31:35.581 INFO:tasks.rados.rados.0.smithi151.stdout:2499: write oid 30 current snap is 323 2024-09-13T14:31:35.581 INFO:tasks.rados.rados.0.smithi151.stdout:2499: seq_num 938 ranges {564966=494981,1606345=782994,3025826=425501} 2024-09-13T14:31:35.587 INFO:tasks.rados.rados.0.smithi151.stdout:2499: writing smithi15186934-30 from 564966 to 1059947 tid 1 2024-09-13T14:31:35.595 INFO:tasks.rados.rados.0.smithi151.stdout:2499: writing smithi15186934-30 from 1606345 to 2389339 tid 2 2024-09-13T14:31:35.601 INFO:tasks.rados.rados.0.smithi151.stdout:2499: writing smithi15186934-30 from 3025826 to 3451327 tid 3 2024-09-13T14:31:35.601 INFO:tasks.rados.rados.0.smithi151.stdout:2500: write oid 17 current snap is 323 2024-09-13T14:31:35.601 INFO:tasks.rados.rados.0.smithi151.stdout:2498: finishing write tid 1 to smithi15186934-21 2024-09-13T14:31:35.601 INFO:tasks.rados.rados.0.smithi151.stdout:2498: finishing write tid 2 to smithi15186934-21 2024-09-13T14:31:35.601 INFO:tasks.rados.rados.0.smithi151.stdout:2498: finishing write tid 3 to smithi15186934-21 2024-09-13T14:31:35.601 INFO:tasks.rados.rados.0.smithi151.stdout:2498: finishing write tid 4 to smithi15186934-21 2024-09-13T14:31:35.601 INFO:tasks.rados.rados.0.smithi151.stdout:2500: seq_num 939 ranges {581773=570230,1664827=526652,2653249=636773} 2024-09-13T14:31:35.607 INFO:tasks.rados.rados.0.smithi151.stdout:2500: writing smithi15186934-17 from 581773 to 1152003 tid 1 2024-09-13T14:31:35.613 INFO:tasks.rados.rados.0.smithi151.stdout:2500: writing smithi15186934-17 from 1664827 to 2191479 tid 2 2024-09-13T14:31:35.620 INFO:tasks.rados.rados.0.smithi151.stdout:2500: writing smithi15186934-17 from 2653249 to 3290022 tid 3 2024-09-13T14:31:35.620 INFO:tasks.rados.rados.0.smithi151.stdout:2501: rollback oid 46 current snap is 323 2024-09-13T14:31:35.620 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 46 to 320 2024-09-13T14:31:35.620 INFO:tasks.rados.rados.0.smithi151.stdout:2498: finishing write tid 5 to smithi15186934-21 2024-09-13T14:31:35.620 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 21 v 529 (ObjNum 937 snap 323 seq_num 937) dirty exists 2024-09-13T14:31:35.620 INFO:tasks.rados.rados.0.smithi151.stdout:2498: left oid 21 (ObjNum 937 snap 323 seq_num 937) 2024-09-13T14:31:35.621 INFO:tasks.rados.rados.0.smithi151.stdout:2499: finishing write tid 1 to smithi15186934-30 2024-09-13T14:31:35.621 INFO:tasks.rados.rados.0.smithi151.stdout:2499: finishing write tid 2 to smithi15186934-30 2024-09-13T14:31:35.621 INFO:tasks.rados.rados.0.smithi151.stdout:2499: finishing write tid 3 to smithi15186934-30 2024-09-13T14:31:35.621 INFO:tasks.rados.rados.0.smithi151.stdout:2499: finishing write tid 4 to smithi15186934-30 2024-09-13T14:31:35.621 INFO:tasks.rados.rados.0.smithi151.stdout:2499: finishing write tid 5 to smithi15186934-30 2024-09-13T14:31:35.621 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 30 v 980 (ObjNum 938 snap 323 seq_num 938) dirty exists 2024-09-13T14:31:35.621 INFO:tasks.rados.rados.0.smithi151.stdout:2499: left oid 30 (ObjNum 938 snap 323 seq_num 938) 2024-09-13T14:31:35.621 INFO:tasks.rados.rados.0.smithi151.stdout:2500: finishing write tid 1 to smithi15186934-17 2024-09-13T14:31:35.621 INFO:tasks.rados.rados.0.smithi151.stdout:2498: done (3 left) 2024-09-13T14:31:35.621 INFO:tasks.rados.rados.0.smithi151.stdout:2499: done (2 left) 2024-09-13T14:31:35.621 INFO:tasks.rados.rados.0.smithi151.stdout:2502: snap_create 2024-09-13T14:31:35.621 INFO:tasks.rados.rados.0.smithi151.stdout:2500: finishing write tid 2 to smithi15186934-17 2024-09-13T14:31:35.622 INFO:tasks.rados.rados.0.smithi151.stdout:2501: finishing rollback tid 1 to smithi15186934-46 2024-09-13T14:31:35.622 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 46 v 863 (ObjNum 33 snap 0 seq_num 3092860992) dirty dne 2024-09-13T14:31:35.623 INFO:tasks.rados.rados.0.smithi151.stdout:2500: finishing write tid 3 to smithi15186934-17 2024-09-13T14:31:35.623 INFO:tasks.rados.rados.0.smithi151.stdout:2500: finishing write tid 4 to smithi15186934-17 2024-09-13T14:31:35.623 INFO:tasks.rados.rados.0.smithi151.stdout:2500: finishing write tid 5 to smithi15186934-17 2024-09-13T14:31:35.623 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 1449 (ObjNum 939 snap 323 seq_num 939) dirty exists 2024-09-13T14:31:35.623 INFO:tasks.rados.rados.0.smithi151.stdout:2500: left oid 17 (ObjNum 939 snap 323 seq_num 939) 2024-09-13T14:31:36.598 INFO:tasks.rados.rados.0.smithi151.stdout:2500: done (2 left) 2024-09-13T14:31:36.598 INFO:tasks.rados.rados.0.smithi151.stdout:2501: done (1 left) 2024-09-13T14:31:36.598 INFO:tasks.rados.rados.0.smithi151.stdout:2502: done (0 left) 2024-09-13T14:31:36.598 INFO:tasks.rados.rados.0.smithi151.stdout:2503: snap_create 2024-09-13T14:31:36.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:36 smithi184 ceph-mon[92527]: pgmap v675: 89 pgs: 89 active+clean; 219 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 793 KiB/s wr, 3 op/s 2024-09-13T14:31:36.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:36 smithi184 ceph-mon[92527]: osdmap e705: 8 total, 8 up, 8 in 2024-09-13T14:31:36.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:36 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:31:36.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:36 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:31:36.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:36 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:31:36.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:36 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:31:36.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:36 smithi151 ceph-mon[104450]: pgmap v675: 89 pgs: 89 active+clean; 219 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 793 KiB/s wr, 3 op/s 2024-09-13T14:31:36.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:36 smithi151 ceph-mon[104450]: osdmap e705: 8 total, 8 up, 8 in 2024-09-13T14:31:36.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:36 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:31:36.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:36 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:31:36.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:36 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:31:36.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:36 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:31:36.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:36 smithi151 ceph-mon[108966]: pgmap v675: 89 pgs: 89 active+clean; 219 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 793 KiB/s wr, 3 op/s 2024-09-13T14:31:36.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:36 smithi151 ceph-mon[108966]: osdmap e705: 8 total, 8 up, 8 in 2024-09-13T14:31:36.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:36 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:31:36.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:36 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:31:36.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:36 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:31:36.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:36 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:31:37.597 INFO:tasks.rados.rados.0.smithi151.stdout:2503: done (0 left) 2024-09-13T14:31:37.597 INFO:tasks.rados.rados.0.smithi151.stdout:2504: write oid 35 current snap is 325 2024-09-13T14:31:37.597 INFO:tasks.rados.rados.0.smithi151.stdout:2504: seq_num 940 ranges {598580=645479,1723309=786663,3080672=448045} 2024-09-13T14:31:37.603 INFO:tasks.rados.rados.0.smithi151.stdout:2504: writing smithi15186934-35 from 598580 to 1244059 tid 1 2024-09-13T14:31:37.611 INFO:tasks.rados.rados.0.smithi151.stdout:2504: writing smithi15186934-35 from 1723309 to 2509972 tid 2 2024-09-13T14:31:37.617 INFO:tasks.rados.rados.0.smithi151.stdout:2504: writing smithi15186934-35 from 3080672 to 3528717 tid 3 2024-09-13T14:31:37.617 INFO:tasks.rados.rados.0.smithi151.stdout:2505: write oid 26 current snap is 325 2024-09-13T14:31:37.617 INFO:tasks.rados.rados.0.smithi151.stdout:2505: seq_num 941 ranges {615387=720728,1781791=530321,3108095=707292} 2024-09-13T14:31:37.624 INFO:tasks.rados.rados.0.smithi151.stdout:2505: writing smithi15186934-26 from 615387 to 1336115 tid 1 2024-09-13T14:31:37.629 INFO:tasks.rados.rados.0.smithi151.stdout:2505: writing smithi15186934-26 from 1781791 to 2312112 tid 2 2024-09-13T14:31:37.638 INFO:tasks.rados.rados.0.smithi151.stdout:2505: writing smithi15186934-26 from 3108095 to 3815387 tid 3 2024-09-13T14:31:37.638 INFO:tasks.rados.rados.0.smithi151.stdout:2506: read oid 34 snap 304 2024-09-13T14:31:37.638 INFO:tasks.rados.rados.0.smithi151.stdout:2506: expect (ObjNum 831 snap 291 seq_num 831) 2024-09-13T14:31:37.638 INFO:tasks.rados.rados.0.smithi151.stdout:2507: snap_remove snap 323 2024-09-13T14:31:37.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:37 smithi184 ceph-mon[92527]: osdmap e706: 8 total, 8 up, 8 in 2024-09-13T14:31:37.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:37 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:31:37.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:37 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:31:37.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:37 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:31:37.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:37 smithi151 ceph-mon[104450]: osdmap e706: 8 total, 8 up, 8 in 2024-09-13T14:31:37.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:37 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:31:37.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:37 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:31:37.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:37 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:31:37.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:37 smithi151 ceph-mon[108966]: osdmap e706: 8 total, 8 up, 8 in 2024-09-13T14:31:37.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:37 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:31:37.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:37 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:31:37.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:37 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:31:38.615 INFO:tasks.rados.rados.0.smithi151.stdout:2507: done (3 left) 2024-09-13T14:31:38.615 INFO:tasks.rados.rados.0.smithi151.stdout:2508: rollback oid 14 current snap is 325 2024-09-13T14:31:38.615 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 14 to 320 2024-09-13T14:31:38.615 INFO:tasks.rados.rados.0.smithi151.stdout:2504: finishing write tid 1 to smithi15186934-35 2024-09-13T14:31:38.615 INFO:tasks.rados.rados.0.smithi151.stdout:2504: finishing write tid 2 to smithi15186934-35 2024-09-13T14:31:38.615 INFO:tasks.rados.rados.0.smithi151.stdout:2504: finishing write tid 3 to smithi15186934-35 2024-09-13T14:31:38.615 INFO:tasks.rados.rados.0.smithi151.stdout:2504: finishing write tid 4 to smithi15186934-35 2024-09-13T14:31:38.615 INFO:tasks.rados.rados.0.smithi151.stdout:2504: finishing write tid 5 to smithi15186934-35 2024-09-13T14:31:38.615 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 35 v 1453 (ObjNum 940 snap 325 seq_num 940) dirty exists 2024-09-13T14:31:38.615 INFO:tasks.rados.rados.0.smithi151.stdout:2504: left oid 35 (ObjNum 940 snap 325 seq_num 940) 2024-09-13T14:31:38.615 INFO:tasks.rados.rados.0.smithi151.stdout:2505: finishing write tid 1 to smithi15186934-26 2024-09-13T14:31:38.615 INFO:tasks.rados.rados.0.smithi151.stdout:2505: finishing write tid 2 to smithi15186934-26 2024-09-13T14:31:38.615 INFO:tasks.rados.rados.0.smithi151.stdout:2505: finishing write tid 3 to smithi15186934-26 2024-09-13T14:31:38.616 INFO:tasks.rados.rados.0.smithi151.stdout:2505: finishing write tid 4 to smithi15186934-26 2024-09-13T14:31:38.616 INFO:tasks.rados.rados.0.smithi151.stdout:2505: finishing write tid 5 to smithi15186934-26 2024-09-13T14:31:38.616 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 892 (ObjNum 941 snap 325 seq_num 941) dirty exists 2024-09-13T14:31:38.616 INFO:tasks.rados.rados.0.smithi151.stdout:2505: left oid 26 (ObjNum 941 snap 325 seq_num 941) 2024-09-13T14:31:38.616 INFO:tasks.rados.rados.0.smithi151.stdout:2506: expect (ObjNum 831 snap 291 seq_num 831) 2024-09-13T14:31:38.792 INFO:tasks.rados.rados.0.smithi151.stdout:2508: finishing rollback tid 0 to smithi15186934-14 2024-09-13T14:31:38.792 INFO:tasks.rados.rados.0.smithi151.stdout:2508: finishing rollback tid 1 to smithi15186934-14 2024-09-13T14:31:38.793 INFO:tasks.rados.rados.0.smithi151.stdout:2508: finishing rollback tid 2 to smithi15186934-14 2024-09-13T14:31:38.793 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 1055 (ObjNum 908 snap 315 seq_num 908) dirty exists 2024-09-13T14:31:38.793 INFO:tasks.rados.rados.0.smithi151.stdout:2504: done (3 left) 2024-09-13T14:31:38.793 INFO:tasks.rados.rados.0.smithi151.stdout:2505: done (2 left) 2024-09-13T14:31:38.793 INFO:tasks.rados.rados.0.smithi151.stdout:2506: done (1 left) 2024-09-13T14:31:38.793 INFO:tasks.rados.rados.0.smithi151.stdout:2508: done (0 left) 2024-09-13T14:31:38.793 INFO:tasks.rados.rados.0.smithi151.stdout:2509: read oid 36 snap -1 2024-09-13T14:31:38.793 INFO:tasks.rados.rados.0.smithi151.stdout:2509: expect deleted 2024-09-13T14:31:38.793 INFO:tasks.rados.rados.0.smithi151.stdout:2510: rollback oid 7 current snap is 325 2024-09-13T14:31:38.793 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 7 to 317 2024-09-13T14:31:38.793 INFO:tasks.rados.rados.0.smithi151.stdout:2511: write oid 38 current snap is 325 2024-09-13T14:31:38.793 INFO:tasks.rados.rados.0.smithi151.stdout:2511: seq_num 942 ranges {632194=795977,2123920=790332,3419165=413029} 2024-09-13T14:31:38.800 INFO:tasks.rados.rados.0.smithi151.stdout:2511: writing smithi15186934-38 from 632194 to 1428171 tid 1 2024-09-13T14:31:38.809 INFO:tasks.rados.rados.0.smithi151.stdout:2511: writing smithi15186934-38 from 2123920 to 2914252 tid 2 2024-09-13T14:31:38.814 INFO:tasks.rados.rados.0.smithi151.stdout:2511: writing smithi15186934-38 from 3419165 to 3832194 tid 3 2024-09-13T14:31:38.815 INFO:tasks.rados.rados.0.smithi151.stdout:2512: read oid 25 snap -1 2024-09-13T14:31:38.815 INFO:tasks.rados.rados.0.smithi151.stdout:2512: expect deleted 2024-09-13T14:31:38.815 INFO:tasks.rados.rados.0.smithi151.stdout:2513: read oid 48 snap -1 2024-09-13T14:31:38.815 INFO:tasks.rados.rados.0.smithi151.stdout:2513: expect (ObjNum 854 snap 295 seq_num 854) 2024-09-13T14:31:38.815 INFO:tasks.rados.rados.0.smithi151.stdout:2514: delete oid 1 current snap is 325 2024-09-13T14:31:38.815 INFO:tasks.rados.rados.0.smithi151.stdout:2510: finishing rollback tid 0 to smithi15186934-7 2024-09-13T14:31:38.815 INFO:tasks.rados.rados.0.smithi151.stdout:2510: finishing rollback tid 1 to smithi15186934-7 2024-09-13T14:31:38.815 INFO:tasks.rados.rados.0.smithi151.stdout:2510: finishing rollback tid 2 to smithi15186934-7 2024-09-13T14:31:38.815 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 1457 (ObjNum 855 snap 295 seq_num 855) dirty exists 2024-09-13T14:31:38.815 INFO:tasks.rados.rados.0.smithi151.stdout:2511: finishing write tid 1 to smithi15186934-38 2024-09-13T14:31:38.815 INFO:tasks.rados.rados.0.smithi151.stdout:2511: finishing write tid 2 to smithi15186934-38 2024-09-13T14:31:38.816 INFO:tasks.rados.rados.0.smithi151.stdout:2509: done (5 left) 2024-09-13T14:31:38.816 INFO:tasks.rados.rados.0.smithi151.stdout:2510: done (4 left) 2024-09-13T14:31:38.816 INFO:tasks.rados.rados.0.smithi151.stdout:2514: done (3 left) 2024-09-13T14:31:38.816 INFO:tasks.rados.rados.0.smithi151.stdout:2515: write oid 20 current snap is 325 2024-09-13T14:31:38.816 INFO:tasks.rados.rados.0.smithi151.stdout:2515: seq_num 944 ranges {665808=662828,1957237=677648,3074011=609486} 2024-09-13T14:31:38.823 INFO:tasks.rados.rados.0.smithi151.stdout:2515: writing smithi15186934-20 from 665808 to 1328636 tid 1 2024-09-13T14:31:38.831 INFO:tasks.rados.rados.0.smithi151.stdout:2515: writing smithi15186934-20 from 1957237 to 2634885 tid 2 2024-09-13T14:31:38.836 INFO:tasks.rados.rados.0.smithi151.stdout:2515: writing smithi15186934-20 from 3074011 to 3683497 tid 3 2024-09-13T14:31:38.836 INFO:tasks.rados.rados.0.smithi151.stdout:2516: delete oid 34 current snap is 325 2024-09-13T14:31:38.836 INFO:tasks.rados.rados.0.smithi151.stdout:2511: finishing write tid 3 to smithi15186934-38 2024-09-13T14:31:38.837 INFO:tasks.rados.rados.0.smithi151.stdout:2511: finishing write tid 4 to smithi15186934-38 2024-09-13T14:31:38.837 INFO:tasks.rados.rados.0.smithi151.stdout:2511: finishing write tid 5 to smithi15186934-38 2024-09-13T14:31:38.837 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 995 (ObjNum 942 snap 325 seq_num 942) dirty exists 2024-09-13T14:31:38.837 INFO:tasks.rados.rados.0.smithi151.stdout:2511: left oid 38 (ObjNum 942 snap 325 seq_num 942) 2024-09-13T14:31:38.837 INFO:tasks.rados.rados.0.smithi151.stdout:2513: expect (ObjNum 854 snap 295 seq_num 854) 2024-09-13T14:31:38.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:38 smithi184 ceph-mon[92527]: pgmap v678: 89 pgs: 89 active+clean; 225 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 7.0 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:31:38.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:38 smithi184 ceph-mon[92527]: osdmap e707: 8 total, 8 up, 8 in 2024-09-13T14:31:38.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:38 smithi151 ceph-mon[104450]: pgmap v678: 89 pgs: 89 active+clean; 225 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 7.0 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:31:38.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:38 smithi151 ceph-mon[104450]: osdmap e707: 8 total, 8 up, 8 in 2024-09-13T14:31:38.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:38 smithi151 ceph-mon[108966]: pgmap v678: 89 pgs: 89 active+clean; 225 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 7.0 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:31:38.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:38 smithi151 ceph-mon[108966]: osdmap e707: 8 total, 8 up, 8 in 2024-09-13T14:31:38.914 INFO:tasks.rados.rados.0.smithi151.stdout:2515: finishing write tid 1 to smithi15186934-20 2024-09-13T14:31:38.914 INFO:tasks.rados.rados.0.smithi151.stdout:2515: finishing write tid 2 to smithi15186934-20 2024-09-13T14:31:38.914 INFO:tasks.rados.rados.0.smithi151.stdout:2515: finishing write tid 3 to smithi15186934-20 2024-09-13T14:31:38.915 INFO:tasks.rados.rados.0.smithi151.stdout:2515: finishing write tid 4 to smithi15186934-20 2024-09-13T14:31:38.915 INFO:tasks.rados.rados.0.smithi151.stdout:2515: finishing write tid 5 to smithi15186934-20 2024-09-13T14:31:38.915 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 608 (ObjNum 944 snap 325 seq_num 944) dirty exists 2024-09-13T14:31:38.915 INFO:tasks.rados.rados.0.smithi151.stdout:2515: left oid 20 (ObjNum 944 snap 325 seq_num 944) 2024-09-13T14:31:38.915 INFO:tasks.rados.rados.0.smithi151.stdout:2511: done (4 left) 2024-09-13T14:31:38.915 INFO:tasks.rados.rados.0.smithi151.stdout:2512: done (3 left) 2024-09-13T14:31:38.915 INFO:tasks.rados.rados.0.smithi151.stdout:2513: done (2 left) 2024-09-13T14:31:38.915 INFO:tasks.rados.rados.0.smithi151.stdout:2515: done (1 left) 2024-09-13T14:31:38.915 INFO:tasks.rados.rados.0.smithi151.stdout:2516: done (0 left) 2024-09-13T14:31:38.915 INFO:tasks.rados.rados.0.smithi151.stdout:2517: delete oid 11 current snap is 325 2024-09-13T14:31:38.916 INFO:tasks.rados.rados.0.smithi151.stdout:2517: done (0 left) 2024-09-13T14:31:38.916 INFO:tasks.rados.rados.0.smithi151.stdout:2518: write oid 36 current snap is 325 2024-09-13T14:31:38.916 INFO:tasks.rados.rados.0.smithi151.stdout:2518: seq_num 947 ranges {716229=488575,1616330=541328,2756280=559655} 2024-09-13T14:31:38.922 INFO:tasks.rados.rados.0.smithi151.stdout:2518: writing smithi15186934-36 from 716229 to 1204804 tid 1 2024-09-13T14:31:38.927 INFO:tasks.rados.rados.0.smithi151.stdout:2518: writing smithi15186934-36 from 1616330 to 2157658 tid 2 2024-09-13T14:31:38.934 INFO:tasks.rados.rados.0.smithi151.stdout:2518: writing smithi15186934-36 from 2756280 to 3315935 tid 3 2024-09-13T14:31:38.934 INFO:tasks.rados.rados.0.smithi151.stdout:2519: delete oid 13 current snap is 325 2024-09-13T14:31:38.934 INFO:tasks.rados.rados.0.smithi151.stdout:2518: finishing write tid 1 to smithi15186934-36 2024-09-13T14:31:38.934 INFO:tasks.rados.rados.0.smithi151.stdout:2518: finishing write tid 2 to smithi15186934-36 2024-09-13T14:31:38.936 INFO:tasks.rados.rados.0.smithi151.stdout:2519: done (1 left) 2024-09-13T14:31:38.936 INFO:tasks.rados.rados.0.smithi151.stdout:2520: read oid 45 snap -1 2024-09-13T14:31:38.936 INFO:tasks.rados.rados.0.smithi151.stdout:2520: expect (ObjNum 858 snap 296 seq_num 858) 2024-09-13T14:31:38.936 INFO:tasks.rados.rados.0.smithi151.stdout:2521: snap_remove snap 299 2024-09-13T14:31:39.561 INFO:tasks.rados.rados.0.smithi151.stdout:2521: done (2 left) 2024-09-13T14:31:39.561 INFO:tasks.rados.rados.0.smithi151.stdout:2522: delete oid 34 current snap is 325 2024-09-13T14:31:39.561 INFO:tasks.rados.rados.0.smithi151.stdout:2518: finishing write tid 3 to smithi15186934-36 2024-09-13T14:31:39.561 INFO:tasks.rados.rados.0.smithi151.stdout:2518: finishing write tid 4 to smithi15186934-36 2024-09-13T14:31:39.561 INFO:tasks.rados.rados.0.smithi151.stdout:2518: finishing write tid 5 to smithi15186934-36 2024-09-13T14:31:39.562 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 36 v 601 (ObjNum 947 snap 325 seq_num 947) dirty exists 2024-09-13T14:31:39.562 INFO:tasks.rados.rados.0.smithi151.stdout:2518: left oid 36 (ObjNum 947 snap 325 seq_num 947) 2024-09-13T14:31:39.562 INFO:tasks.rados.rados.0.smithi151.stdout:2520: expect (ObjNum 858 snap 296 seq_num 858) 2024-09-13T14:31:39.647 INFO:tasks.rados.rados.0.smithi151.stdout:2518: done (2 left) 2024-09-13T14:31:39.647 INFO:tasks.rados.rados.0.smithi151.stdout:2520: done (1 left) 2024-09-13T14:31:39.647 INFO:tasks.rados.rados.0.smithi151.stdout:2522: done (0 left) 2024-09-13T14:31:39.647 INFO:tasks.rados.rados.0.smithi151.stdout:2523: snap_create 2024-09-13T14:31:40.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:39 smithi184 ceph-mon[92527]: osdmap e708: 8 total, 8 up, 8 in 2024-09-13T14:31:40.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:39 smithi184 ceph-mon[92527]: osdmap e709: 8 total, 8 up, 8 in 2024-09-13T14:31:40.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:39 smithi151 ceph-mon[108966]: osdmap e708: 8 total, 8 up, 8 in 2024-09-13T14:31:40.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:39 smithi151 ceph-mon[108966]: osdmap e709: 8 total, 8 up, 8 in 2024-09-13T14:31:40.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:39 smithi151 ceph-mon[104450]: osdmap e708: 8 total, 8 up, 8 in 2024-09-13T14:31:40.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:39 smithi151 ceph-mon[104450]: osdmap e709: 8 total, 8 up, 8 in 2024-09-13T14:31:40.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:31:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:31:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:31:40.629 INFO:tasks.rados.rados.0.smithi151.stdout:2523: done (0 left) 2024-09-13T14:31:40.629 INFO:tasks.rados.rados.0.smithi151.stdout:2524: delete oid 11 current snap is 326 2024-09-13T14:31:40.633 INFO:tasks.rados.rados.0.smithi151.stdout:2524: done (0 left) 2024-09-13T14:31:40.633 INFO:tasks.rados.rados.0.smithi151.stdout:2525: read oid 13 snap -1 2024-09-13T14:31:40.633 INFO:tasks.rados.rados.0.smithi151.stdout:2525: expect deleted 2024-09-13T14:31:40.633 INFO:tasks.rados.rados.0.smithi151.stdout:2526: snap_create 2024-09-13T14:31:41.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:40 smithi184 ceph-mon[92527]: pgmap v681: 89 pgs: 89 active+clean; 227 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 7.0 MiB/s rd, 1.3 MiB/s wr, 9 op/s 2024-09-13T14:31:41.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:40 smithi151 ceph-mon[104450]: pgmap v681: 89 pgs: 89 active+clean; 227 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 7.0 MiB/s rd, 1.3 MiB/s wr, 9 op/s 2024-09-13T14:31:41.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:40 smithi151 ceph-mon[108966]: pgmap v681: 89 pgs: 89 active+clean; 227 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 7.0 MiB/s rd, 1.3 MiB/s wr, 9 op/s 2024-09-13T14:31:41.636 INFO:tasks.rados.rados.0.smithi151.stdout:2525: done (1 left) 2024-09-13T14:31:41.636 INFO:tasks.rados.rados.0.smithi151.stdout:2526: done (0 left) 2024-09-13T14:31:41.637 INFO:tasks.rados.rados.0.smithi151.stdout:2527: write oid 7 current snap is 327 2024-09-13T14:31:41.637 INFO:tasks.rados.rados.0.smithi151.stdout:2527: seq_num 951 ranges {783457=505924,1850258=548666,2865972=721096} 2024-09-13T14:31:41.643 INFO:tasks.rados.rados.0.smithi151.stdout:2527: writing smithi15186934-7 from 783457 to 1289381 tid 1 2024-09-13T14:31:41.650 INFO:tasks.rados.rados.0.smithi151.stdout:2527: writing smithi15186934-7 from 1850258 to 2398924 tid 2 2024-09-13T14:31:41.656 INFO:tasks.rados.rados.0.smithi151.stdout:2527: writing smithi15186934-7 from 2865972 to 3587068 tid 3 2024-09-13T14:31:41.656 INFO:tasks.rados.rados.0.smithi151.stdout:2528: read oid 10 snap -1 2024-09-13T14:31:41.657 INFO:tasks.rados.rados.0.smithi151.stdout:2528: expect (ObjNum 932 snap 318 seq_num 932) 2024-09-13T14:31:41.657 INFO:tasks.rados.rados.0.smithi151.stdout:2529: snap_remove snap 322 2024-09-13T14:31:42.004 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:41 smithi184 ceph-mon[92527]: osdmap e710: 8 total, 8 up, 8 in 2024-09-13T14:31:42.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:41 smithi151 ceph-mon[104450]: osdmap e710: 8 total, 8 up, 8 in 2024-09-13T14:31:42.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:41 smithi151 ceph-mon[108966]: osdmap e710: 8 total, 8 up, 8 in 2024-09-13T14:31:42.641 INFO:tasks.rados.rados.0.smithi151.stdout:2529: done (2 left) 2024-09-13T14:31:42.641 INFO:tasks.rados.rados.0.smithi151.stdout:2530: snap_remove snap 318 2024-09-13T14:31:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:42 smithi184 ceph-mon[92527]: pgmap v684: 89 pgs: 89 active+clean; 232 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.4 MiB/s rd, 1.3 MiB/s wr, 7 op/s 2024-09-13T14:31:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:42 smithi184 ceph-mon[92527]: osdmap e711: 8 total, 8 up, 8 in 2024-09-13T14:31:43.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:42 smithi151 ceph-mon[104450]: pgmap v684: 89 pgs: 89 active+clean; 232 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.4 MiB/s rd, 1.3 MiB/s wr, 7 op/s 2024-09-13T14:31:43.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:42 smithi151 ceph-mon[104450]: osdmap e711: 8 total, 8 up, 8 in 2024-09-13T14:31:43.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:42 smithi151 ceph-mon[108966]: pgmap v684: 89 pgs: 89 active+clean; 232 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.4 MiB/s rd, 1.3 MiB/s wr, 7 op/s 2024-09-13T14:31:43.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:42 smithi151 ceph-mon[108966]: osdmap e711: 8 total, 8 up, 8 in 2024-09-13T14:31:43.644 INFO:tasks.rados.rados.0.smithi151.stdout:2530: done (2 left) 2024-09-13T14:31:43.644 INFO:tasks.rados.rados.0.smithi151.stdout:2531: write oid 34 current snap is 327 2024-09-13T14:31:43.645 INFO:tasks.rados.rados.0.smithi151.stdout:2531: seq_num 952 ranges {400264=581173,1508740=491524} 2024-09-13T14:31:43.650 INFO:tasks.rados.rados.0.smithi151.stdout:2531: writing smithi15186934-34 from 400264 to 981437 tid 1 2024-09-13T14:31:43.656 INFO:tasks.rados.rados.0.smithi151.stdout:2531: writing smithi15186934-34 from 1508740 to 2000264 tid 2 2024-09-13T14:31:43.656 INFO:tasks.rados.rados.0.smithi151.stdout:2532: write oid 20 current snap is 327 2024-09-13T14:31:43.657 INFO:tasks.rados.rados.0.smithi151.stdout:2527: finishing write tid 1 to smithi15186934-7 2024-09-13T14:31:43.657 INFO:tasks.rados.rados.0.smithi151.stdout:2527: finishing write tid 2 to smithi15186934-7 2024-09-13T14:31:43.657 INFO:tasks.rados.rados.0.smithi151.stdout:2527: finishing write tid 3 to smithi15186934-7 2024-09-13T14:31:43.657 INFO:tasks.rados.rados.0.smithi151.stdout:2532: seq_num 953 ranges {417071=656422,1567222=449849} 2024-09-13T14:31:43.662 INFO:tasks.rados.rados.0.smithi151.stdout:2532: writing smithi15186934-20 from 417071 to 1073493 tid 1 2024-09-13T14:31:43.668 INFO:tasks.rados.rados.0.smithi151.stdout:2532: writing smithi15186934-20 from 1567222 to 2017071 tid 2 2024-09-13T14:31:43.668 INFO:tasks.rados.rados.0.smithi151.stdout:2533: read oid 48 snap -1 2024-09-13T14:31:43.668 INFO:tasks.rados.rados.0.smithi151.stdout:2533: expect (ObjNum 854 snap 295 seq_num 854) 2024-09-13T14:31:43.668 INFO:tasks.rados.rados.0.smithi151.stdout:2534: snap_remove snap 324 2024-09-13T14:31:43.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:43 smithi151 ceph-mon[104450]: osdmap e712: 8 total, 8 up, 8 in 2024-09-13T14:31:43.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:43 smithi151 ceph-mon[108966]: osdmap e712: 8 total, 8 up, 8 in 2024-09-13T14:31:44.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:43 smithi184 ceph-mon[92527]: osdmap e712: 8 total, 8 up, 8 in 2024-09-13T14:31:44.566 INFO:tasks.rados.rados.0.smithi151.stdout:2534: done (5 left) 2024-09-13T14:31:44.566 INFO:tasks.rados.rados.0.smithi151.stdout:2535: write oid 16 current snap is 327 2024-09-13T14:31:44.566 INFO:tasks.rados.rados.0.smithi151.stdout:2527: finishing write tid 4 to smithi15186934-7 2024-09-13T14:31:44.566 INFO:tasks.rados.rados.0.smithi151.stdout:2527: finishing write tid 5 to smithi15186934-7 2024-09-13T14:31:44.566 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 1468 (ObjNum 951 snap 327 seq_num 951) dirty exists 2024-09-13T14:31:44.566 INFO:tasks.rados.rados.0.smithi151.stdout:2527: left oid 7 (ObjNum 951 snap 327 seq_num 951) 2024-09-13T14:31:44.566 INFO:tasks.rados.rados.0.smithi151.stdout:2528: expect (ObjNum 932 snap 318 seq_num 932) 2024-09-13T14:31:44.773 INFO:tasks.rados.rados.0.smithi151.stdout:2531: finishing write tid 1 to smithi15186934-34 2024-09-13T14:31:44.773 INFO:tasks.rados.rados.0.smithi151.stdout:2535: seq_num 954 ranges {433878=731671,1909351=124527} 2024-09-13T14:31:44.780 INFO:tasks.rados.rados.0.smithi151.stdout:2535: writing smithi15186934-16 from 433878 to 1165549 tid 1 2024-09-13T14:31:44.785 INFO:tasks.rados.rados.0.smithi151.stdout:2535: writing smithi15186934-16 from 1909351 to 2033878 tid 2 2024-09-13T14:31:44.785 INFO:tasks.rados.rados.0.smithi151.stdout:2527: done (5 left) 2024-09-13T14:31:44.786 INFO:tasks.rados.rados.0.smithi151.stdout:2528: done (4 left) 2024-09-13T14:31:44.786 INFO:tasks.rados.rados.0.smithi151.stdout:2536: write oid 24 current snap is 327 2024-09-13T14:31:44.786 INFO:tasks.rados.rados.0.smithi151.stdout:2531: finishing write tid 2 to smithi15186934-34 2024-09-13T14:31:44.786 INFO:tasks.rados.rados.0.smithi151.stdout:2531: finishing write tid 3 to smithi15186934-34 2024-09-13T14:31:44.786 INFO:tasks.rados.rados.0.smithi151.stdout:2536: seq_num 955 ranges {450685=406920,1567833=482852} 2024-09-13T14:31:44.789 INFO:tasks.rados.rados.0.smithi151.stdout:2536: writing smithi15186934-24 from 450685 to 857605 tid 1 2024-09-13T14:31:44.796 INFO:tasks.rados.rados.0.smithi151.stdout:2536: writing smithi15186934-24 from 1567833 to 2050685 tid 2 2024-09-13T14:31:44.796 INFO:tasks.rados.rados.0.smithi151.stdout:2537: snap_remove snap 320 2024-09-13T14:31:44.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:44 smithi151 ceph-mon[108966]: pgmap v687: 89 pgs: 89 active+clean; 241 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.9 MiB/s rd, 2.6 MiB/s wr, 16 op/s 2024-09-13T14:31:44.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:44 smithi151 ceph-mon[108966]: osdmap e713: 8 total, 8 up, 8 in 2024-09-13T14:31:44.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:44 smithi151 ceph-mon[108966]: osdmap e714: 8 total, 8 up, 8 in 2024-09-13T14:31:44.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:44 smithi151 ceph-mon[104450]: pgmap v687: 89 pgs: 89 active+clean; 241 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.9 MiB/s rd, 2.6 MiB/s wr, 16 op/s 2024-09-13T14:31:44.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:44 smithi151 ceph-mon[104450]: osdmap e713: 8 total, 8 up, 8 in 2024-09-13T14:31:44.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:44 smithi151 ceph-mon[104450]: osdmap e714: 8 total, 8 up, 8 in 2024-09-13T14:31:44.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:31:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:31:44] "GET /metrics HTTP/1.1" 200 36510 "" "Prometheus/2.51.0" 2024-09-13T14:31:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:44 smithi184 ceph-mon[92527]: pgmap v687: 89 pgs: 89 active+clean; 241 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.9 MiB/s rd, 2.6 MiB/s wr, 16 op/s 2024-09-13T14:31:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:44 smithi184 ceph-mon[92527]: osdmap e713: 8 total, 8 up, 8 in 2024-09-13T14:31:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:44 smithi184 ceph-mon[92527]: osdmap e714: 8 total, 8 up, 8 in 2024-09-13T14:31:45.650 INFO:tasks.rados.rados.0.smithi151.stdout:2537: done (5 left) 2024-09-13T14:31:45.651 INFO:tasks.rados.rados.0.smithi151.stdout:2538: read oid 2 snap -1 2024-09-13T14:31:45.651 INFO:tasks.rados.rados.0.smithi151.stdout:2538: expect deleted 2024-09-13T14:31:45.651 INFO:tasks.rados.rados.0.smithi151.stdout:2539: read oid 36 snap 308 2024-09-13T14:31:45.651 INFO:tasks.rados.rados.0.smithi151.stdout:2539: expect (ObjNum 764 snap 264 seq_num 764) 2024-09-13T14:31:45.651 INFO:tasks.rados.rados.0.smithi151.stdout:2540: write oid 15 current snap is 327 2024-09-13T14:31:45.651 INFO:tasks.rados.rados.0.smithi151.stdout:2531: finishing write tid 4 to smithi15186934-34 2024-09-13T14:31:45.651 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 1002 (ObjNum 952 snap 327 seq_num 952) dirty exists 2024-09-13T14:31:45.651 INFO:tasks.rados.rados.0.smithi151.stdout:2531: left oid 34 (ObjNum 952 snap 327 seq_num 952) 2024-09-13T14:31:45.651 INFO:tasks.rados.rados.0.smithi151.stdout:2532: finishing write tid 1 to smithi15186934-20 2024-09-13T14:31:45.651 INFO:tasks.rados.rados.0.smithi151.stdout:2540: seq_num 956 ranges {467492=482169,1626315=441177} 2024-09-13T14:31:45.656 INFO:tasks.rados.rados.0.smithi151.stdout:2540: writing smithi15186934-15 from 467492 to 949661 tid 1 2024-09-13T14:31:45.663 INFO:tasks.rados.rados.0.smithi151.stdout:2540: writing smithi15186934-15 from 1626315 to 2067492 tid 2 2024-09-13T14:31:45.664 INFO:tasks.rados.rados.0.smithi151.stdout:2531: done (7 left) 2024-09-13T14:31:45.664 INFO:tasks.rados.rados.0.smithi151.stdout:2541: write oid 17 current snap is 327 2024-09-13T14:31:45.664 INFO:tasks.rados.rados.0.smithi151.stdout:2532: finishing write tid 2 to smithi15186934-20 2024-09-13T14:31:45.664 INFO:tasks.rados.rados.0.smithi151.stdout:2532: finishing write tid 3 to smithi15186934-20 2024-09-13T14:31:45.664 INFO:tasks.rados.rados.0.smithi151.stdout:2532: finishing write tid 4 to smithi15186934-20 2024-09-13T14:31:45.664 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 620 (ObjNum 953 snap 327 seq_num 953) dirty exists 2024-09-13T14:31:45.664 INFO:tasks.rados.rados.0.smithi151.stdout:2532: left oid 20 (ObjNum 953 snap 327 seq_num 953) 2024-09-13T14:31:45.664 INFO:tasks.rados.rados.0.smithi151.stdout:2533: expect (ObjNum 854 snap 295 seq_num 854) 2024-09-13T14:31:45.742 INFO:tasks.rados.rados.0.smithi151.stdout:2535: finishing write tid 1 to smithi15186934-16 2024-09-13T14:31:45.743 INFO:tasks.rados.rados.0.smithi151.stdout:2535: finishing write tid 2 to smithi15186934-16 2024-09-13T14:31:45.743 INFO:tasks.rados.rados.0.smithi151.stdout:2535: finishing write tid 3 to smithi15186934-16 2024-09-13T14:31:45.743 INFO:tasks.rados.rados.0.smithi151.stdout:2535: finishing write tid 4 to smithi15186934-16 2024-09-13T14:31:45.743 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 16 v 625 (ObjNum 954 snap 327 seq_num 954) dirty exists 2024-09-13T14:31:45.743 INFO:tasks.rados.rados.0.smithi151.stdout:2535: left oid 16 (ObjNum 954 snap 327 seq_num 954) 2024-09-13T14:31:45.743 INFO:tasks.rados.rados.0.smithi151.stdout:2536: finishing write tid 1 to smithi15186934-24 2024-09-13T14:31:45.743 INFO:tasks.rados.rados.0.smithi151.stdout:2541: seq_num 957 ranges {484299=557418,1684797=399502} 2024-09-13T14:31:45.748 INFO:tasks.rados.rados.0.smithi151.stdout:2541: writing smithi15186934-17 from 484299 to 1041717 tid 1 2024-09-13T14:31:45.754 INFO:tasks.rados.rados.0.smithi151.stdout:2541: writing smithi15186934-17 from 1684797 to 2084299 tid 2 2024-09-13T14:31:45.754 INFO:tasks.rados.rados.0.smithi151.stdout:2532: done (7 left) 2024-09-13T14:31:45.754 INFO:tasks.rados.rados.0.smithi151.stdout:2533: done (6 left) 2024-09-13T14:31:45.754 INFO:tasks.rados.rados.0.smithi151.stdout:2535: done (5 left) 2024-09-13T14:31:45.754 INFO:tasks.rados.rados.0.smithi151.stdout:2542: delete oid 30 current snap is 327 2024-09-13T14:31:45.754 INFO:tasks.rados.rados.0.smithi151.stdout:2536: finishing write tid 2 to smithi15186934-24 2024-09-13T14:31:45.755 INFO:tasks.rados.rados.0.smithi151.stdout:2536: finishing write tid 3 to smithi15186934-24 2024-09-13T14:31:45.755 INFO:tasks.rados.rados.0.smithi151.stdout:2536: finishing write tid 4 to smithi15186934-24 2024-09-13T14:31:45.755 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 24 v 1481 (ObjNum 955 snap 327 seq_num 955) dirty exists 2024-09-13T14:31:45.755 INFO:tasks.rados.rados.0.smithi151.stdout:2536: left oid 24 (ObjNum 955 snap 327 seq_num 955) 2024-09-13T14:31:45.755 INFO:tasks.rados.rados.0.smithi151.stdout:2539: expect (ObjNum 764 snap 264 seq_num 764) 2024-09-13T14:31:45.848 INFO:tasks.rados.rados.0.smithi151.stdout:2540: finishing write tid 1 to smithi15186934-15 2024-09-13T14:31:45.848 INFO:tasks.rados.rados.0.smithi151.stdout:2540: finishing write tid 2 to smithi15186934-15 2024-09-13T14:31:45.848 INFO:tasks.rados.rados.0.smithi151.stdout:2540: finishing write tid 3 to smithi15186934-15 2024-09-13T14:31:45.848 INFO:tasks.rados.rados.0.smithi151.stdout:2540: finishing write tid 4 to smithi15186934-15 2024-09-13T14:31:45.848 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 15 v 1014 (ObjNum 956 snap 327 seq_num 956) dirty exists 2024-09-13T14:31:45.848 INFO:tasks.rados.rados.0.smithi151.stdout:2540: left oid 15 (ObjNum 956 snap 327 seq_num 956) 2024-09-13T14:31:45.848 INFO:tasks.rados.rados.0.smithi151.stdout:2536: done (5 left) 2024-09-13T14:31:45.848 INFO:tasks.rados.rados.0.smithi151.stdout:2538: done (4 left) 2024-09-13T14:31:45.848 INFO:tasks.rados.rados.0.smithi151.stdout:2539: done (3 left) 2024-09-13T14:31:45.848 INFO:tasks.rados.rados.0.smithi151.stdout:2540: done (2 left) 2024-09-13T14:31:45.848 INFO:tasks.rados.rados.0.smithi151.stdout:2542: done (1 left) 2024-09-13T14:31:45.848 INFO:tasks.rados.rados.0.smithi151.stdout:2543: read oid 30 snap 321 2024-09-13T14:31:45.849 INFO:tasks.rados.rados.0.smithi151.stdout:2543: expect (ObjNum 707 snap 237 seq_num 707) 2024-09-13T14:31:45.849 INFO:tasks.rados.rados.0.smithi151.stdout:2544: delete oid 44 current snap is 327 2024-09-13T14:31:45.849 INFO:tasks.rados.rados.0.smithi151.stdout:2541: finishing write tid 1 to smithi15186934-17 2024-09-13T14:31:45.849 INFO:tasks.rados.rados.0.smithi151.stdout:2541: finishing write tid 2 to smithi15186934-17 2024-09-13T14:31:45.849 INFO:tasks.rados.rados.0.smithi151.stdout:2541: finishing write tid 3 to smithi15186934-17 2024-09-13T14:31:45.849 INFO:tasks.rados.rados.0.smithi151.stdout:2541: finishing write tid 4 to smithi15186934-17 2024-09-13T14:31:45.849 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 1489 (ObjNum 957 snap 327 seq_num 957) dirty exists 2024-09-13T14:31:45.849 INFO:tasks.rados.rados.0.smithi151.stdout:2541: left oid 17 (ObjNum 957 snap 327 seq_num 957) 2024-09-13T14:31:45.850 INFO:tasks.rados.rados.0.smithi151.stdout:2541: done (2 left) 2024-09-13T14:31:45.850 INFO:tasks.rados.rados.0.smithi151.stdout:2544: done (1 left) 2024-09-13T14:31:45.850 INFO:tasks.rados.rados.0.smithi151.stdout:2545: read oid 25 snap 286 2024-09-13T14:31:45.850 INFO:tasks.rados.rados.0.smithi151.stdout:2545: expect (ObjNum 822 snap 286 seq_num 822) 2024-09-13T14:31:45.850 INFO:tasks.rados.rados.0.smithi151.stdout:2546: snap_remove snap 308 2024-09-13T14:31:46.652 INFO:tasks.rados.rados.0.smithi151.stdout:2546: done (2 left) 2024-09-13T14:31:46.652 INFO:tasks.rados.rados.0.smithi151.stdout:2547: snap_create 2024-09-13T14:31:46.652 INFO:tasks.rados.rados.0.smithi151.stdout:2543: expect (ObjNum 707 snap 237 seq_num 707) 2024-09-13T14:31:46.786 INFO:tasks.rados.rados.0.smithi151.stdout:2545: expect (ObjNum 822 snap 286 seq_num 822) 2024-09-13T14:31:47.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:46 smithi184 ceph-mon[92527]: pgmap v690: 89 pgs: 89 active+clean; 241 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.6 MiB/s rd, 1.8 MiB/s wr, 12 op/s 2024-09-13T14:31:47.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:46 smithi184 ceph-mon[92527]: osdmap e715: 8 total, 8 up, 8 in 2024-09-13T14:31:47.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:47.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:46 smithi151 ceph-mon[104450]: pgmap v690: 89 pgs: 89 active+clean; 241 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.6 MiB/s rd, 1.8 MiB/s wr, 12 op/s 2024-09-13T14:31:47.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:46 smithi151 ceph-mon[104450]: osdmap e715: 8 total, 8 up, 8 in 2024-09-13T14:31:47.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:47.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:46 smithi151 ceph-mon[108966]: pgmap v690: 89 pgs: 89 active+clean; 241 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.6 MiB/s rd, 1.8 MiB/s wr, 12 op/s 2024-09-13T14:31:47.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:46 smithi151 ceph-mon[108966]: osdmap e715: 8 total, 8 up, 8 in 2024-09-13T14:31:47.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:47.654 INFO:tasks.rados.rados.0.smithi151.stdout:2543: done (2 left) 2024-09-13T14:31:47.654 INFO:tasks.rados.rados.0.smithi151.stdout:2545: done (1 left) 2024-09-13T14:31:47.654 INFO:tasks.rados.rados.0.smithi151.stdout:2547: done (0 left) 2024-09-13T14:31:47.654 INFO:tasks.rados.rados.0.smithi151.stdout:2548: snap_create 2024-09-13T14:31:48.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:47 smithi184 ceph-mon[92527]: osdmap e716: 8 total, 8 up, 8 in 2024-09-13T14:31:48.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:47 smithi184 ceph-mon[92527]: osdmap e717: 8 total, 8 up, 8 in 2024-09-13T14:31:48.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:47 smithi151 ceph-mon[104450]: osdmap e716: 8 total, 8 up, 8 in 2024-09-13T14:31:48.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:47 smithi151 ceph-mon[104450]: osdmap e717: 8 total, 8 up, 8 in 2024-09-13T14:31:48.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:47 smithi151 ceph-mon[108966]: osdmap e716: 8 total, 8 up, 8 in 2024-09-13T14:31:48.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:47 smithi151 ceph-mon[108966]: osdmap e717: 8 total, 8 up, 8 in 2024-09-13T14:31:48.655 INFO:tasks.rados.rados.0.smithi151.stdout:2548: done (0 left) 2024-09-13T14:31:48.655 INFO:tasks.rados.rados.0.smithi151.stdout:2549: snap_remove snap 286 2024-09-13T14:31:49.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:48 smithi184 ceph-mon[92527]: pgmap v693: 89 pgs: 89 active+clean; 249 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.8 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2024-09-13T14:31:49.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:48 smithi184 ceph-mon[92527]: osdmap e718: 8 total, 8 up, 8 in 2024-09-13T14:31:49.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:48 smithi151 ceph-mon[104450]: pgmap v693: 89 pgs: 89 active+clean; 249 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.8 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2024-09-13T14:31:49.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:48 smithi151 ceph-mon[104450]: osdmap e718: 8 total, 8 up, 8 in 2024-09-13T14:31:49.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:48 smithi151 ceph-mon[108966]: pgmap v693: 89 pgs: 89 active+clean; 249 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.8 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2024-09-13T14:31:49.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:48 smithi151 ceph-mon[108966]: osdmap e718: 8 total, 8 up, 8 in 2024-09-13T14:31:49.564 INFO:tasks.rados.rados.0.smithi151.stdout:2549: done (0 left) 2024-09-13T14:31:49.564 INFO:tasks.rados.rados.0.smithi151.stdout:2550: delete oid 4 current snap is 329 2024-09-13T14:31:49.575 INFO:tasks.rados.rados.0.smithi151.stdout:2550: done (0 left) 2024-09-13T14:31:49.575 INFO:tasks.rados.rados.0.smithi151.stdout:2551: snap_remove snap 326 2024-09-13T14:31:50.562 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:31:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:31:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:31:50.575 INFO:tasks.rados.rados.0.smithi151.stdout:2551: done (0 left) 2024-09-13T14:31:50.575 INFO:tasks.rados.rados.0.smithi151.stdout:2552: rollback oid 35 current snap is 329 2024-09-13T14:31:50.575 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 35 to 304 2024-09-13T14:31:50.575 INFO:tasks.rados.rados.0.smithi151.stdout:2553: read oid 29 snap -1 2024-09-13T14:31:50.575 INFO:tasks.rados.rados.0.smithi151.stdout:2553: expect deleted 2024-09-13T14:31:50.575 INFO:tasks.rados.rados.0.smithi151.stdout:2554: write oid 44 current snap is 329 2024-09-13T14:31:50.575 INFO:tasks.rados.rados.0.smithi151.stdout:2554: seq_num 961 ranges {551527=574767,1918725=232802} 2024-09-13T14:31:50.582 INFO:tasks.rados.rados.0.smithi151.stdout:2554: writing smithi15186934-44 from 551527 to 1126294 tid 1 2024-09-13T14:31:50.587 INFO:tasks.rados.rados.0.smithi151.stdout:2554: writing smithi15186934-44 from 1918725 to 2151527 tid 2 2024-09-13T14:31:50.588 INFO:tasks.rados.rados.0.smithi151.stdout:2555: write oid 7 current snap is 329 2024-09-13T14:31:50.588 INFO:tasks.rados.rados.0.smithi151.stdout:2555: seq_num 962 ranges {568334=650016,1860854=307480} 2024-09-13T14:31:50.594 INFO:tasks.rados.rados.0.smithi151.stdout:2555: writing smithi15186934-7 from 568334 to 1218350 tid 1 2024-09-13T14:31:50.600 INFO:tasks.rados.rados.0.smithi151.stdout:2555: writing smithi15186934-7 from 1860854 to 2168334 tid 2 2024-09-13T14:31:50.600 INFO:tasks.rados.rados.0.smithi151.stdout:2556: delete oid 15 current snap is 329 2024-09-13T14:31:50.600 INFO:tasks.rados.rados.0.smithi151.stdout:2552: finishing rollback tid 0 to smithi15186934-35 2024-09-13T14:31:50.600 INFO:tasks.rados.rados.0.smithi151.stdout:2552: finishing rollback tid 1 to smithi15186934-35 2024-09-13T14:31:50.600 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 35 v 1502 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-13T14:31:50.600 INFO:tasks.rados.rados.0.smithi151.stdout:2554: finishing write tid 1 to smithi15186934-44 2024-09-13T14:31:50.600 INFO:tasks.rados.rados.0.smithi151.stdout:2554: finishing write tid 2 to smithi15186934-44 2024-09-13T14:31:50.600 INFO:tasks.rados.rados.0.smithi151.stdout:2554: finishing write tid 3 to smithi15186934-44 2024-09-13T14:31:50.600 INFO:tasks.rados.rados.0.smithi151.stdout:2554: finishing write tid 4 to smithi15186934-44 2024-09-13T14:31:50.601 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 784 (ObjNum 961 snap 329 seq_num 961) dirty exists 2024-09-13T14:31:50.601 INFO:tasks.rados.rados.0.smithi151.stdout:2554: left oid 44 (ObjNum 961 snap 329 seq_num 961) 2024-09-13T14:31:50.601 INFO:tasks.rados.rados.0.smithi151.stdout:2555: finishing write tid 1 to smithi15186934-7 2024-09-13T14:31:50.601 INFO:tasks.rados.rados.0.smithi151.stdout:2552: done (4 left) 2024-09-13T14:31:50.601 INFO:tasks.rados.rados.0.smithi151.stdout:2553: done (3 left) 2024-09-13T14:31:50.601 INFO:tasks.rados.rados.0.smithi151.stdout:2554: done (2 left) 2024-09-13T14:31:50.602 INFO:tasks.rados.rados.0.smithi151.stdout:2556: done (1 left) 2024-09-13T14:31:50.602 INFO:tasks.rados.rados.0.smithi151.stdout:2557: snap_create 2024-09-13T14:31:50.603 INFO:tasks.rados.rados.0.smithi151.stdout:2555: finishing write tid 2 to smithi15186934-7 2024-09-13T14:31:50.603 INFO:tasks.rados.rados.0.smithi151.stdout:2555: finishing write tid 3 to smithi15186934-7 2024-09-13T14:31:50.603 INFO:tasks.rados.rados.0.smithi151.stdout:2555: finishing write tid 4 to smithi15186934-7 2024-09-13T14:31:50.603 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 1510 (ObjNum 962 snap 329 seq_num 962) dirty exists 2024-09-13T14:31:50.603 INFO:tasks.rados.rados.0.smithi151.stdout:2555: left oid 7 (ObjNum 962 snap 329 seq_num 962) 2024-09-13T14:31:50.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:50 smithi184 ceph-mon[92527]: pgmap v696: 89 pgs: 89 active+clean; 251 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 1.4 MiB/s wr, 11 op/s 2024-09-13T14:31:50.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:50 smithi184 ceph-mon[92527]: osdmap e719: 8 total, 8 up, 8 in 2024-09-13T14:31:50.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:50 smithi151 ceph-mon[108966]: pgmap v696: 89 pgs: 89 active+clean; 251 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 1.4 MiB/s wr, 11 op/s 2024-09-13T14:31:50.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:50 smithi151 ceph-mon[108966]: osdmap e719: 8 total, 8 up, 8 in 2024-09-13T14:31:50.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:50 smithi151 ceph-mon[104450]: pgmap v696: 89 pgs: 89 active+clean; 251 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 1.4 MiB/s wr, 11 op/s 2024-09-13T14:31:50.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:50 smithi151 ceph-mon[104450]: osdmap e719: 8 total, 8 up, 8 in 2024-09-13T14:31:51.574 INFO:tasks.rados.rados.0.smithi151.stdout:2555: done (1 left) 2024-09-13T14:31:51.575 INFO:tasks.rados.rados.0.smithi151.stdout:2557: done (0 left) 2024-09-13T14:31:51.575 INFO:tasks.rados.rados.0.smithi151.stdout:2558: write oid 4 current snap is 330 2024-09-13T14:31:51.575 INFO:tasks.rados.rados.0.smithi151.stdout:2558: seq_num 964 ranges {601948=400514,1577818=624130} 2024-09-13T14:31:51.580 INFO:tasks.rados.rados.0.smithi151.stdout:2558: writing smithi15186934-4 from 601948 to 1002462 tid 1 2024-09-13T14:31:51.587 INFO:tasks.rados.rados.0.smithi151.stdout:2558: writing smithi15186934-4 from 1577818 to 2201948 tid 2 2024-09-13T14:31:51.587 INFO:tasks.rados.rados.0.smithi151.stdout:2559: snap_remove snap 328 2024-09-13T14:31:51.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:51 smithi184 ceph-mon[92527]: osdmap e720: 8 total, 8 up, 8 in 2024-09-13T14:31:51.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:51 smithi151 ceph-mon[108966]: osdmap e720: 8 total, 8 up, 8 in 2024-09-13T14:31:51.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:51 smithi151 ceph-mon[104450]: osdmap e720: 8 total, 8 up, 8 in 2024-09-13T14:31:52.580 INFO:tasks.rados.rados.0.smithi151.stdout:2559: done (1 left) 2024-09-13T14:31:52.580 INFO:tasks.rados.rados.0.smithi151.stdout:2560: snap_remove snap 325 2024-09-13T14:31:52.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:52 smithi184 ceph-mon[92527]: pgmap v699: 89 pgs: 89 active+clean; 244 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.0 MiB/s rd, 225 KiB/s wr, 5 op/s 2024-09-13T14:31:52.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:52 smithi184 ceph-mon[92527]: osdmap e721: 8 total, 8 up, 8 in 2024-09-13T14:31:52.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:52 smithi151 ceph-mon[104450]: pgmap v699: 89 pgs: 89 active+clean; 244 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.0 MiB/s rd, 225 KiB/s wr, 5 op/s 2024-09-13T14:31:52.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:52 smithi151 ceph-mon[104450]: osdmap e721: 8 total, 8 up, 8 in 2024-09-13T14:31:52.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:52 smithi151 ceph-mon[108966]: pgmap v699: 89 pgs: 89 active+clean; 244 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.0 MiB/s rd, 225 KiB/s wr, 5 op/s 2024-09-13T14:31:52.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:52 smithi151 ceph-mon[108966]: osdmap e721: 8 total, 8 up, 8 in 2024-09-13T14:31:53.586 INFO:tasks.rados.rados.0.smithi151.stdout:2560: done (1 left) 2024-09-13T14:31:53.586 INFO:tasks.rados.rados.0.smithi151.stdout:2561: snap_remove snap 317 2024-09-13T14:31:53.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:53 smithi184 ceph-mon[92527]: osdmap e722: 8 total, 8 up, 8 in 2024-09-13T14:31:53.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:53 smithi151 ceph-mon[104450]: osdmap e722: 8 total, 8 up, 8 in 2024-09-13T14:31:53.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:53 smithi151 ceph-mon[108966]: osdmap e722: 8 total, 8 up, 8 in 2024-09-13T14:31:54.569 INFO:tasks.rados.rados.0.smithi151.stdout:2561: done (1 left) 2024-09-13T14:31:54.569 INFO:tasks.rados.rados.0.smithi151.stdout:2562: write oid 45 current snap is 330 2024-09-13T14:31:54.569 INFO:tasks.rados.rados.0.smithi151.stdout:2558: finishing write tid 1 to smithi15186934-4 2024-09-13T14:31:54.569 INFO:tasks.rados.rados.0.smithi151.stdout:2558: finishing write tid 2 to smithi15186934-4 2024-09-13T14:31:54.569 INFO:tasks.rados.rados.0.smithi151.stdout:2558: finishing write tid 3 to smithi15186934-4 2024-09-13T14:31:54.569 INFO:tasks.rados.rados.0.smithi151.stdout:2558: finishing write tid 4 to smithi15186934-4 2024-09-13T14:31:54.569 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 1301 (ObjNum 964 snap 330 seq_num 964) dirty exists 2024-09-13T14:31:54.570 INFO:tasks.rados.rados.0.smithi151.stdout:2558: left oid 4 (ObjNum 964 snap 330 seq_num 964) 2024-09-13T14:31:54.570 INFO:tasks.rados.rados.0.smithi151.stdout:2562: seq_num 965 ranges {618755=475763,1636300=457996} 2024-09-13T14:31:54.573 INFO:tasks.rados.rados.0.smithi151.stdout:2562: writing smithi15186934-45 from 618755 to 1094518 tid 1 2024-09-13T14:31:54.580 INFO:tasks.rados.rados.0.smithi151.stdout:2562: writing smithi15186934-45 from 1636300 to 2094296 tid 2 2024-09-13T14:31:54.581 INFO:tasks.rados.rados.0.smithi151.stdout:2558: done (1 left) 2024-09-13T14:31:54.581 INFO:tasks.rados.rados.0.smithi151.stdout:2563: snap_create 2024-09-13T14:31:54.581 INFO:tasks.rados.rados.0.smithi151.stdout:2562: finishing write tid 1 to smithi15186934-45 2024-09-13T14:31:54.583 INFO:tasks.rados.rados.0.smithi151.stdout:2562: finishing write tid 2 to smithi15186934-45 2024-09-13T14:31:54.586 INFO:tasks.rados.rados.0.smithi151.stdout:2562: finishing write tid 3 to smithi15186934-45 2024-09-13T14:31:54.586 INFO:tasks.rados.rados.0.smithi151.stdout:2562: finishing write tid 4 to smithi15186934-45 2024-09-13T14:31:54.586 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 1305 (ObjNum 965 snap 330 seq_num 965) dirty exists 2024-09-13T14:31:54.586 INFO:tasks.rados.rados.0.smithi151.stdout:2562: left oid 45 (ObjNum 965 snap 330 seq_num 965) 2024-09-13T14:31:54.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:54 smithi184 ceph-mon[92527]: pgmap v702: 89 pgs: 89 active+clean; 239 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.0 MiB/s rd, 484 KiB/s wr, 7 op/s 2024-09-13T14:31:54.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:54 smithi184 ceph-mon[92527]: osdmap e723: 8 total, 8 up, 8 in 2024-09-13T14:31:54.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:54 smithi184 ceph-mon[92527]: osdmap e724: 8 total, 8 up, 8 in 2024-09-13T14:31:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:54 smithi151 ceph-mon[104450]: pgmap v702: 89 pgs: 89 active+clean; 239 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.0 MiB/s rd, 484 KiB/s wr, 7 op/s 2024-09-13T14:31:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:54 smithi151 ceph-mon[104450]: osdmap e723: 8 total, 8 up, 8 in 2024-09-13T14:31:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:54 smithi151 ceph-mon[104450]: osdmap e724: 8 total, 8 up, 8 in 2024-09-13T14:31:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:54 smithi151 ceph-mon[108966]: pgmap v702: 89 pgs: 89 active+clean; 239 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.0 MiB/s rd, 484 KiB/s wr, 7 op/s 2024-09-13T14:31:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:54 smithi151 ceph-mon[108966]: osdmap e723: 8 total, 8 up, 8 in 2024-09-13T14:31:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:54 smithi151 ceph-mon[108966]: osdmap e724: 8 total, 8 up, 8 in 2024-09-13T14:31:54.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:31:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:31:54] "GET /metrics HTTP/1.1" 200 36514 "" "Prometheus/2.51.0" 2024-09-13T14:31:55.570 INFO:tasks.rados.rados.0.smithi151.stdout:2562: done (1 left) 2024-09-13T14:31:55.570 INFO:tasks.rados.rados.0.smithi151.stdout:2563: done (0 left) 2024-09-13T14:31:55.570 INFO:tasks.rados.rados.0.smithi151.stdout:2564: delete oid 44 current snap is 331 2024-09-13T14:31:55.572 INFO:tasks.rados.rados.0.smithi151.stdout:2564: done (0 left) 2024-09-13T14:31:55.572 INFO:tasks.rados.rados.0.smithi151.stdout:2565: write oid 25 current snap is 331 2024-09-13T14:31:55.573 INFO:tasks.rados.rados.0.smithi151.stdout:2565: seq_num 967 ranges {652369=742614,2153264=99105} 2024-09-13T14:31:55.580 INFO:tasks.rados.rados.0.smithi151.stdout:2565: writing smithi15186934-25 from 652369 to 1394983 tid 1 2024-09-13T14:31:55.585 INFO:tasks.rados.rados.0.smithi151.stdout:2565: writing smithi15186934-25 from 2153264 to 2252369 tid 2 2024-09-13T14:31:55.585 INFO:tasks.rados.rados.0.smithi151.stdout:2566: read oid 49 snap -1 2024-09-13T14:31:55.585 INFO:tasks.rados.rados.0.smithi151.stdout:2566: expect deleted 2024-09-13T14:31:55.585 INFO:tasks.rados.rados.0.smithi151.stdout:2567: snap_create 2024-09-13T14:31:55.587 INFO:tasks.rados.rados.0.smithi151.stdout:2565: finishing write tid 1 to smithi15186934-25 2024-09-13T14:31:55.590 INFO:tasks.rados.rados.0.smithi151.stdout:2565: finishing write tid 2 to smithi15186934-25 2024-09-13T14:31:55.590 INFO:tasks.rados.rados.0.smithi151.stdout:2565: finishing write tid 3 to smithi15186934-25 2024-09-13T14:31:55.590 INFO:tasks.rados.rados.0.smithi151.stdout:2565: finishing write tid 4 to smithi15186934-25 2024-09-13T14:31:55.590 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 1308 (ObjNum 967 snap 331 seq_num 967) dirty exists 2024-09-13T14:31:55.590 INFO:tasks.rados.rados.0.smithi151.stdout:2565: left oid 25 (ObjNum 967 snap 331 seq_num 967) 2024-09-13T14:31:56.589 INFO:tasks.rados.rados.0.smithi151.stdout:2565: done (2 left) 2024-09-13T14:31:56.589 INFO:tasks.rados.rados.0.smithi151.stdout:2566: done (1 left) 2024-09-13T14:31:56.589 INFO:tasks.rados.rados.0.smithi151.stdout:2567: done (0 left) 2024-09-13T14:31:56.589 INFO:tasks.rados.rados.0.smithi151.stdout:2568: snap_create 2024-09-13T14:31:56.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:56 smithi184 ceph-mon[92527]: pgmap v705: 89 pgs: 89 active+clean; 236 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 484 KiB/s wr, 3 op/s 2024-09-13T14:31:56.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:56 smithi184 ceph-mon[92527]: osdmap e725: 8 total, 8 up, 8 in 2024-09-13T14:31:56.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:56 smithi151 ceph-mon[104450]: pgmap v705: 89 pgs: 89 active+clean; 236 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 484 KiB/s wr, 3 op/s 2024-09-13T14:31:56.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:56 smithi151 ceph-mon[104450]: osdmap e725: 8 total, 8 up, 8 in 2024-09-13T14:31:56.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:56 smithi151 ceph-mon[108966]: pgmap v705: 89 pgs: 89 active+clean; 236 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 484 KiB/s wr, 3 op/s 2024-09-13T14:31:56.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:56 smithi151 ceph-mon[108966]: osdmap e725: 8 total, 8 up, 8 in 2024-09-13T14:31:57.595 INFO:tasks.rados.rados.0.smithi151.stdout:2568: done (0 left) 2024-09-13T14:31:57.595 INFO:tasks.rados.rados.0.smithi151.stdout:2569: snap_create 2024-09-13T14:31:57.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:57 smithi184 ceph-mon[92527]: osdmap e726: 8 total, 8 up, 8 in 2024-09-13T14:31:57.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:57 smithi151 ceph-mon[104450]: osdmap e726: 8 total, 8 up, 8 in 2024-09-13T14:31:57.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:57 smithi151 ceph-mon[108966]: osdmap e726: 8 total, 8 up, 8 in 2024-09-13T14:31:58.597 INFO:tasks.rados.rados.0.smithi151.stdout:2569: done (0 left) 2024-09-13T14:31:58.597 INFO:tasks.rados.rados.0.smithi151.stdout:2570: read oid 32 snap -1 2024-09-13T14:31:58.597 INFO:tasks.rados.rados.0.smithi151.stdout:2570: expect (ObjNum 925 snap 318 seq_num 925) 2024-09-13T14:31:58.597 INFO:tasks.rados.rados.0.smithi151.stdout:2571: write oid 41 current snap is 334 2024-09-13T14:31:58.598 INFO:tasks.rados.rados.0.smithi151.stdout:2571: seq_num 968 ranges {669176=417863,1811746=457430} 2024-09-13T14:31:58.603 INFO:tasks.rados.rados.0.smithi151.stdout:2571: writing smithi15186934-41 from 669176 to 1087039 tid 1 2024-09-13T14:31:58.610 INFO:tasks.rados.rados.0.smithi151.stdout:2571: writing smithi15186934-41 from 1811746 to 2269176 tid 2 2024-09-13T14:31:58.610 INFO:tasks.rados.rados.0.smithi151.stdout:2572: snap_create 2024-09-13T14:31:58.611 INFO:tasks.rados.rados.0.smithi151.stdout:2570: expect (ObjNum 925 snap 318 seq_num 925) 2024-09-13T14:31:58.824 INFO:tasks.rados.rados.0.smithi151.stdout:2571: finishing write tid 1 to smithi15186934-41 2024-09-13T14:31:58.824 INFO:tasks.rados.rados.0.smithi151.stdout:2571: finishing write tid 2 to smithi15186934-41 2024-09-13T14:31:58.824 INFO:tasks.rados.rados.0.smithi151.stdout:2571: finishing write tid 3 to smithi15186934-41 2024-09-13T14:31:58.825 INFO:tasks.rados.rados.0.smithi151.stdout:2571: finishing write tid 4 to smithi15186934-41 2024-09-13T14:31:58.825 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 189 (ObjNum 968 snap 334 seq_num 968) dirty exists 2024-09-13T14:31:58.825 INFO:tasks.rados.rados.0.smithi151.stdout:2571: left oid 41 (ObjNum 968 snap 334 seq_num 968) 2024-09-13T14:31:58.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:58 smithi184 ceph-mon[92527]: pgmap v708: 89 pgs: 89 active+clean; 230 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 631 KiB/s wr, 4 op/s 2024-09-13T14:31:58.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:58 smithi184 ceph-mon[92527]: osdmap e727: 8 total, 8 up, 8 in 2024-09-13T14:31:58.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:58 smithi151 ceph-mon[104450]: pgmap v708: 89 pgs: 89 active+clean; 230 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 631 KiB/s wr, 4 op/s 2024-09-13T14:31:58.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:58 smithi151 ceph-mon[104450]: osdmap e727: 8 total, 8 up, 8 in 2024-09-13T14:31:58.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:58 smithi151 ceph-mon[108966]: pgmap v708: 89 pgs: 89 active+clean; 230 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 631 KiB/s wr, 4 op/s 2024-09-13T14:31:58.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:58 smithi151 ceph-mon[108966]: osdmap e727: 8 total, 8 up, 8 in 2024-09-13T14:31:59.567 INFO:tasks.rados.rados.0.smithi151.stdout:2570: done (2 left) 2024-09-13T14:31:59.568 INFO:tasks.rados.rados.0.smithi151.stdout:2571: done (1 left) 2024-09-13T14:31:59.568 INFO:tasks.rados.rados.0.smithi151.stdout:2572: done (0 left) 2024-09-13T14:31:59.568 INFO:tasks.rados.rados.0.smithi151.stdout:2573: write oid 16 current snap is 335 2024-09-13T14:31:59.568 INFO:tasks.rados.rados.0.smithi151.stdout:2573: seq_num 969 ranges {685983=493112,1870228=415755} 2024-09-13T14:31:59.573 INFO:tasks.rados.rados.0.smithi151.stdout:2573: writing smithi15186934-16 from 685983 to 1179095 tid 1 2024-09-13T14:31:59.580 INFO:tasks.rados.rados.0.smithi151.stdout:2573: writing smithi15186934-16 from 1870228 to 2285983 tid 2 2024-09-13T14:31:59.580 INFO:tasks.rados.rados.0.smithi151.stdout:2574: read oid 5 snap -1 2024-09-13T14:31:59.580 INFO:tasks.rados.rados.0.smithi151.stdout:2574: expect deleted 2024-09-13T14:31:59.580 INFO:tasks.rados.rados.0.smithi151.stdout:2575: snap_remove snap 276 2024-09-13T14:31:59.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:59 smithi184 ceph-mon[92527]: osdmap e728: 8 total, 8 up, 8 in 2024-09-13T14:31:59.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:31:59 smithi184 ceph-mon[92527]: osdmap e729: 8 total, 8 up, 8 in 2024-09-13T14:31:59.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:59 smithi151 ceph-mon[104450]: osdmap e728: 8 total, 8 up, 8 in 2024-09-13T14:31:59.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:31:59 smithi151 ceph-mon[104450]: osdmap e729: 8 total, 8 up, 8 in 2024-09-13T14:31:59.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:59 smithi151 ceph-mon[108966]: osdmap e728: 8 total, 8 up, 8 in 2024-09-13T14:31:59.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:31:59 smithi151 ceph-mon[108966]: osdmap e729: 8 total, 8 up, 8 in 2024-09-13T14:32:00.570 INFO:tasks.rados.rados.0.smithi151.stdout:2575: done (2 left) 2024-09-13T14:32:00.570 INFO:tasks.rados.rados.0.smithi151.stdout:2576: write oid 34 current snap is 335 2024-09-13T14:32:00.570 INFO:tasks.rados.rados.0.smithi151.stdout:2576: seq_num 970 ranges {702790=568361,1812357=490433} 2024-09-13T14:32:00.578 INFO:tasks.rados.rados.0.smithi151.stdout:2576: writing smithi15186934-34 from 702790 to 1271151 tid 1 2024-09-13T14:32:00.585 INFO:tasks.rados.rados.0.smithi151.stdout:2576: writing smithi15186934-34 from 1812357 to 2302790 tid 2 2024-09-13T14:32:00.585 INFO:tasks.rados.rados.0.smithi151.stdout:2577: snap_create 2024-09-13T14:32:00.585 INFO:tasks.rados.rados.0.smithi151.stdout:2573: finishing write tid 1 to smithi15186934-16 2024-09-13T14:32:00.585 INFO:tasks.rados.rados.0.smithi151.stdout:2573: finishing write tid 2 to smithi15186934-16 2024-09-13T14:32:00.585 INFO:tasks.rados.rados.0.smithi151.stdout:2573: finishing write tid 3 to smithi15186934-16 2024-09-13T14:32:00.585 INFO:tasks.rados.rados.0.smithi151.stdout:2573: finishing write tid 4 to smithi15186934-16 2024-09-13T14:32:00.585 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 16 v 645 (ObjNum 969 snap 335 seq_num 969) dirty exists 2024-09-13T14:32:00.586 INFO:tasks.rados.rados.0.smithi151.stdout:2573: left oid 16 (ObjNum 969 snap 335 seq_num 969) 2024-09-13T14:32:00.587 INFO:tasks.rados.rados.0.smithi151.stdout:2576: finishing write tid 1 to smithi15186934-34 2024-09-13T14:32:00.588 INFO:tasks.rados.rados.0.smithi151.stdout:2576: finishing write tid 2 to smithi15186934-34 2024-09-13T14:32:00.588 INFO:tasks.rados.rados.0.smithi151.stdout:2576: finishing write tid 3 to smithi15186934-34 2024-09-13T14:32:00.588 INFO:tasks.rados.rados.0.smithi151.stdout:2576: finishing write tid 4 to smithi15186934-34 2024-09-13T14:32:00.588 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 1044 (ObjNum 970 snap 335 seq_num 970) dirty exists 2024-09-13T14:32:00.588 INFO:tasks.rados.rados.0.smithi151.stdout:2576: left oid 34 (ObjNum 970 snap 335 seq_num 970) 2024-09-13T14:32:00.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:32:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:32:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:32:00.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:00 smithi151 ceph-mon[104450]: pgmap v711: 89 pgs: 89 active+clean; 222 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 631 KiB/s wr, 3 op/s 2024-09-13T14:32:00.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:00 smithi151 ceph-mon[104450]: osdmap e730: 8 total, 8 up, 8 in 2024-09-13T14:32:00.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:00 smithi151 ceph-mon[108966]: pgmap v711: 89 pgs: 89 active+clean; 222 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 631 KiB/s wr, 3 op/s 2024-09-13T14:32:00.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:00 smithi151 ceph-mon[108966]: osdmap e730: 8 total, 8 up, 8 in 2024-09-13T14:32:01.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:00 smithi184 ceph-mon[92527]: pgmap v711: 89 pgs: 89 active+clean; 222 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 631 KiB/s wr, 3 op/s 2024-09-13T14:32:01.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:00 smithi184 ceph-mon[92527]: osdmap e730: 8 total, 8 up, 8 in 2024-09-13T14:32:01.571 INFO:tasks.rados.rados.0.smithi151.stdout:2573: done (3 left) 2024-09-13T14:32:01.571 INFO:tasks.rados.rados.0.smithi151.stdout:2574: done (2 left) 2024-09-13T14:32:01.571 INFO:tasks.rados.rados.0.smithi151.stdout:2576: done (1 left) 2024-09-13T14:32:01.571 INFO:tasks.rados.rados.0.smithi151.stdout:2577: done (0 left) 2024-09-13T14:32:01.571 INFO:tasks.rados.rados.0.smithi151.stdout:2578: write oid 33 current snap is 336 2024-09-13T14:32:01.571 INFO:tasks.rados.rados.0.smithi151.stdout:2578: seq_num 971 ranges {719597=643610,1870839=448758} 2024-09-13T14:32:01.578 INFO:tasks.rados.rados.0.smithi151.stdout:2578: writing smithi15186934-33 from 719597 to 1363207 tid 1 2024-09-13T14:32:01.583 INFO:tasks.rados.rados.0.smithi151.stdout:2578: writing smithi15186934-33 from 1870839 to 2319597 tid 2 2024-09-13T14:32:01.584 INFO:tasks.rados.rados.0.smithi151.stdout:2579: delete oid 20 current snap is 336 2024-09-13T14:32:01.586 INFO:tasks.rados.rados.0.smithi151.stdout:2579: done (1 left) 2024-09-13T14:32:01.586 INFO:tasks.rados.rados.0.smithi151.stdout:2580: snap_create 2024-09-13T14:32:01.586 INFO:tasks.rados.rados.0.smithi151.stdout:2578: finishing write tid 1 to smithi15186934-33 2024-09-13T14:32:01.587 INFO:tasks.rados.rados.0.smithi151.stdout:2578: finishing write tid 2 to smithi15186934-33 2024-09-13T14:32:01.587 INFO:tasks.rados.rados.0.smithi151.stdout:2578: finishing write tid 3 to smithi15186934-33 2024-09-13T14:32:01.587 INFO:tasks.rados.rados.0.smithi151.stdout:2578: finishing write tid 4 to smithi15186934-33 2024-09-13T14:32:01.587 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 33 v 998 (ObjNum 971 snap 336 seq_num 971) dirty exists 2024-09-13T14:32:01.587 INFO:tasks.rados.rados.0.smithi151.stdout:2578: left oid 33 (ObjNum 971 snap 336 seq_num 971) 2024-09-13T14:32:02.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:01 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:02.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:01 smithi184 ceph-mon[92527]: osdmap e731: 8 total, 8 up, 8 in 2024-09-13T14:32:02.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:02.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:01 smithi151 ceph-mon[104450]: osdmap e731: 8 total, 8 up, 8 in 2024-09-13T14:32:02.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:01 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:02.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:01 smithi151 ceph-mon[108966]: osdmap e731: 8 total, 8 up, 8 in 2024-09-13T14:32:02.572 INFO:tasks.rados.rados.0.smithi151.stdout:2578: done (1 left) 2024-09-13T14:32:02.573 INFO:tasks.rados.rados.0.smithi151.stdout:2580: done (0 left) 2024-09-13T14:32:02.573 INFO:tasks.rados.rados.0.smithi151.stdout:2581: rollback oid 15 current snap is 337 2024-09-13T14:32:02.573 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 15 to 336 2024-09-13T14:32:02.573 INFO:tasks.rados.rados.0.smithi151.stdout:2582: read oid 28 snap -1 2024-09-13T14:32:02.573 INFO:tasks.rados.rados.0.smithi151.stdout:2582: expect deleted 2024-09-13T14:32:02.573 INFO:tasks.rados.rados.0.smithi151.stdout:2583: read oid 11 snap -1 2024-09-13T14:32:02.573 INFO:tasks.rados.rados.0.smithi151.stdout:2583: expect deleted 2024-09-13T14:32:02.573 INFO:tasks.rados.rados.0.smithi151.stdout:2584: snap_create 2024-09-13T14:32:02.577 INFO:tasks.rados.rados.0.smithi151.stdout:2581: finishing rollback tid 1 to smithi15186934-15 2024-09-13T14:32:02.577 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 15 v 1020 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-13T14:32:03.009 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:02 smithi151 ceph-mon[104450]: pgmap v714: 89 pgs: 89 active+clean; 222 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 0 B/s wr, 2 op/s 2024-09-13T14:32:03.010 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:02 smithi151 ceph-mon[104450]: osdmap e732: 8 total, 8 up, 8 in 2024-09-13T14:32:03.010 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:02 smithi151 ceph-mon[108966]: pgmap v714: 89 pgs: 89 active+clean; 222 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 0 B/s wr, 2 op/s 2024-09-13T14:32:03.010 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:02 smithi151 ceph-mon[108966]: osdmap e732: 8 total, 8 up, 8 in 2024-09-13T14:32:03.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:02 smithi184 ceph-mon[92527]: pgmap v714: 89 pgs: 89 active+clean; 222 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 0 B/s wr, 2 op/s 2024-09-13T14:32:03.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:02 smithi184 ceph-mon[92527]: osdmap e732: 8 total, 8 up, 8 in 2024-09-13T14:32:03.575 INFO:tasks.rados.rados.0.smithi151.stdout:2581: done (3 left) 2024-09-13T14:32:03.576 INFO:tasks.rados.rados.0.smithi151.stdout:2582: done (2 left) 2024-09-13T14:32:03.576 INFO:tasks.rados.rados.0.smithi151.stdout:2583: done (1 left) 2024-09-13T14:32:03.576 INFO:tasks.rados.rados.0.smithi151.stdout:2584: done (0 left) 2024-09-13T14:32:03.576 INFO:tasks.rados.rados.0.smithi151.stdout:2585: snap_create 2024-09-13T14:32:04.568 INFO:tasks.rados.rados.0.smithi151.stdout:2585: done (0 left) 2024-09-13T14:32:04.568 INFO:tasks.rados.rados.0.smithi151.stdout:2586: read oid 7 snap -1 2024-09-13T14:32:04.568 INFO:tasks.rados.rados.0.smithi151.stdout:2586: expect (ObjNum 962 snap 329 seq_num 962) 2024-09-13T14:32:04.568 INFO:tasks.rados.rados.0.smithi151.stdout:2587: read oid 5 snap -1 2024-09-13T14:32:04.568 INFO:tasks.rados.rados.0.smithi151.stdout:2587: expect deleted 2024-09-13T14:32:04.568 INFO:tasks.rados.rados.0.smithi151.stdout:2588: write oid 35 current snap is 339 2024-09-13T14:32:04.569 INFO:tasks.rados.rados.0.smithi151.stdout:2588: seq_num 973 ranges {753211=794108,1987803=365408} 2024-09-13T14:32:04.578 INFO:tasks.rados.rados.0.smithi151.stdout:2588: writing smithi15186934-35 from 753211 to 1547319 tid 1 2024-09-13T14:32:04.583 INFO:tasks.rados.rados.0.smithi151.stdout:2588: writing smithi15186934-35 from 1987803 to 2353211 tid 2 2024-09-13T14:32:04.583 INFO:tasks.rados.rados.0.smithi151.stdout:2589: write oid 17 current snap is 339 2024-09-13T14:32:04.583 INFO:tasks.rados.rados.0.smithi151.stdout:2589: seq_num 974 ranges {770018=585710,2046285=323733} 2024-09-13T14:32:04.592 INFO:tasks.rados.rados.0.smithi151.stdout:2589: writing smithi15186934-17 from 770018 to 1355728 tid 1 2024-09-13T14:32:04.599 INFO:tasks.rados.rados.0.smithi151.stdout:2589: writing smithi15186934-17 from 2046285 to 2370018 tid 2 2024-09-13T14:32:04.599 INFO:tasks.rados.rados.0.smithi151.stdout:2590: delete oid 33 current snap is 339 2024-09-13T14:32:04.599 INFO:tasks.rados.rados.0.smithi151.stdout:2586: expect (ObjNum 962 snap 329 seq_num 962) 2024-09-13T14:32:04.771 INFO:tasks.rados.rados.0.smithi151.stdout:2588: finishing write tid 1 to smithi15186934-35 2024-09-13T14:32:04.772 INFO:tasks.rados.rados.0.smithi151.stdout:2588: finishing write tid 2 to smithi15186934-35 2024-09-13T14:32:04.772 INFO:tasks.rados.rados.0.smithi151.stdout:2586: done (4 left) 2024-09-13T14:32:04.772 INFO:tasks.rados.rados.0.smithi151.stdout:2587: done (3 left) 2024-09-13T14:32:04.772 INFO:tasks.rados.rados.0.smithi151.stdout:2590: done (2 left) 2024-09-13T14:32:04.772 INFO:tasks.rados.rados.0.smithi151.stdout:2591: read oid 1 snap -1 2024-09-13T14:32:04.772 INFO:tasks.rados.rados.0.smithi151.stdout:2591: expect deleted 2024-09-13T14:32:04.772 INFO:tasks.rados.rados.0.smithi151.stdout:2592: read oid 44 snap -1 2024-09-13T14:32:04.772 INFO:tasks.rados.rados.0.smithi151.stdout:2592: expect deleted 2024-09-13T14:32:04.772 INFO:tasks.rados.rados.0.smithi151.stdout:2593: read oid 15 snap -1 2024-09-13T14:32:04.772 INFO:tasks.rados.rados.0.smithi151.stdout:2593: expect deleted 2024-09-13T14:32:04.772 INFO:tasks.rados.rados.0.smithi151.stdout:2594: read oid 11 snap 304 2024-09-13T14:32:04.772 INFO:tasks.rados.rados.0.smithi151.stdout:2594: expect deleted 2024-09-13T14:32:04.772 INFO:tasks.rados.rados.0.smithi151.stdout:2595: read oid 32 snap 329 2024-09-13T14:32:04.772 INFO:tasks.rados.rados.0.smithi151.stdout:2595: expect (ObjNum 925 snap 318 seq_num 925) 2024-09-13T14:32:04.773 INFO:tasks.rados.rados.0.smithi151.stdout:2596: read oid 37 snap -1 2024-09-13T14:32:04.773 INFO:tasks.rados.rados.0.smithi151.stdout:2596: expect (ObjNum 683 snap 222 seq_num 683) 2024-09-13T14:32:04.773 INFO:tasks.rados.rados.0.smithi151.stdout:2597: snap_remove snap 333 2024-09-13T14:32:04.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:04 smithi184 ceph-mon[92527]: pgmap v717: 89 pgs: 89 active+clean; 196 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 436 KiB/s wr, 4 op/s 2024-09-13T14:32:04.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:04 smithi184 ceph-mon[92527]: osdmap e733: 8 total, 8 up, 8 in 2024-09-13T14:32:04.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:04 smithi184 ceph-mon[92527]: osdmap e734: 8 total, 8 up, 8 in 2024-09-13T14:32:04.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:04 smithi151 ceph-mon[108966]: pgmap v717: 89 pgs: 89 active+clean; 196 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 436 KiB/s wr, 4 op/s 2024-09-13T14:32:04.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:04 smithi151 ceph-mon[108966]: osdmap e733: 8 total, 8 up, 8 in 2024-09-13T14:32:04.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:04 smithi151 ceph-mon[108966]: osdmap e734: 8 total, 8 up, 8 in 2024-09-13T14:32:04.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:04 smithi151 ceph-mon[104450]: pgmap v717: 89 pgs: 89 active+clean; 196 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 436 KiB/s wr, 4 op/s 2024-09-13T14:32:04.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:04 smithi151 ceph-mon[104450]: osdmap e733: 8 total, 8 up, 8 in 2024-09-13T14:32:04.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:04 smithi151 ceph-mon[104450]: osdmap e734: 8 total, 8 up, 8 in 2024-09-13T14:32:04.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:32:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:32:04] "GET /metrics HTTP/1.1" 200 36508 "" "Prometheus/2.51.0" 2024-09-13T14:32:05.580 INFO:tasks.rados.rados.0.smithi151.stdout:2597: done (8 left) 2024-09-13T14:32:05.581 INFO:tasks.rados.rados.0.smithi151.stdout:2598: write oid 5 current snap is 339 2024-09-13T14:32:05.581 INFO:tasks.rados.rados.0.smithi151.stdout:2588: finishing write tid 3 to smithi15186934-35 2024-09-13T14:32:05.581 INFO:tasks.rados.rados.0.smithi151.stdout:2598: seq_num 976 ranges {403632=736208,1763249=619999} 2024-09-13T14:32:05.586 INFO:tasks.rados.rados.0.smithi151.stdout:2598: writing smithi15186934-5 from 403632 to 1139840 tid 1 2024-09-13T14:32:05.594 INFO:tasks.rados.rados.0.smithi151.stdout:2598: writing smithi15186934-5 from 1763249 to 2383248 tid 2 2024-09-13T14:32:05.594 INFO:tasks.rados.rados.0.smithi151.stdout:2599: rollback oid 38 current snap is 339 2024-09-13T14:32:05.594 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 38 to 331 2024-09-13T14:32:05.594 INFO:tasks.rados.rados.0.smithi151.stdout:2588: finishing write tid 4 to smithi15186934-35 2024-09-13T14:32:05.594 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 35 v 1531 (ObjNum 973 snap 339 seq_num 973) dirty exists 2024-09-13T14:32:05.594 INFO:tasks.rados.rados.0.smithi151.stdout:2588: left oid 35 (ObjNum 973 snap 339 seq_num 973) 2024-09-13T14:32:05.594 INFO:tasks.rados.rados.0.smithi151.stdout:2589: finishing write tid 1 to smithi15186934-17 2024-09-13T14:32:05.595 INFO:tasks.rados.rados.0.smithi151.stdout:2589: finishing write tid 2 to smithi15186934-17 2024-09-13T14:32:05.595 INFO:tasks.rados.rados.0.smithi151.stdout:2589: finishing write tid 3 to smithi15186934-17 2024-09-13T14:32:05.595 INFO:tasks.rados.rados.0.smithi151.stdout:2588: done (9 left) 2024-09-13T14:32:05.595 INFO:tasks.rados.rados.0.smithi151.stdout:2600: write oid 8 current snap is 339 2024-09-13T14:32:05.595 INFO:tasks.rados.rados.0.smithi151.stdout:2600: seq_num 977 ranges {420439=411457,1421731=480010} 2024-09-13T14:32:05.598 INFO:tasks.rados.rados.0.smithi151.stdout:2600: writing smithi15186934-8 from 420439 to 831896 tid 1 2024-09-13T14:32:05.604 INFO:tasks.rados.rados.0.smithi151.stdout:2600: writing smithi15186934-8 from 1421731 to 1901741 tid 2 2024-09-13T14:32:05.604 INFO:tasks.rados.rados.0.smithi151.stdout:2601: write oid 39 current snap is 339 2024-09-13T14:32:05.605 INFO:tasks.rados.rados.0.smithi151.stdout:2601: seq_num 978 ranges {437246=486706,1363860=623668} 2024-09-13T14:32:05.609 INFO:tasks.rados.rados.0.smithi151.stdout:2601: writing smithi15186934-39 from 437246 to 923952 tid 1 2024-09-13T14:32:05.616 INFO:tasks.rados.rados.0.smithi151.stdout:2601: writing smithi15186934-39 from 1363860 to 1987528 tid 2 2024-09-13T14:32:05.616 INFO:tasks.rados.rados.0.smithi151.stdout:2602: read oid 45 snap -1 2024-09-13T14:32:05.616 INFO:tasks.rados.rados.0.smithi151.stdout:2602: expect (ObjNum 965 snap 330 seq_num 965) 2024-09-13T14:32:05.616 INFO:tasks.rados.rados.0.smithi151.stdout:2603: rollback oid 9 current snap is 339 2024-09-13T14:32:05.616 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 9 to 309 2024-09-13T14:32:05.616 INFO:tasks.rados.rados.0.smithi151.stdout:2589: finishing write tid 4 to smithi15186934-17 2024-09-13T14:32:05.616 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 1535 (ObjNum 974 snap 339 seq_num 974) dirty exists 2024-09-13T14:32:05.616 INFO:tasks.rados.rados.0.smithi151.stdout:2589: left oid 17 (ObjNum 974 snap 339 seq_num 974) 2024-09-13T14:32:05.616 INFO:tasks.rados.rados.0.smithi151.stdout:2589: done (12 left) 2024-09-13T14:32:05.616 INFO:tasks.rados.rados.0.smithi151.stdout:2593: done (11 left) 2024-09-13T14:32:05.616 INFO:tasks.rados.rados.0.smithi151.stdout:2604: snap_remove snap 330 2024-09-13T14:32:06.593 INFO:tasks.rados.rados.0.smithi151.stdout:2604: done (11 left) 2024-09-13T14:32:06.594 INFO:tasks.rados.rados.0.smithi151.stdout:2605: read oid 18 snap 331 2024-09-13T14:32:06.594 INFO:tasks.rados.rados.0.smithi151.stdout:2605: expect deleted 2024-09-13T14:32:06.594 INFO:tasks.rados.rados.0.smithi151.stdout:2606: rollback oid 24 current snap is 339 2024-09-13T14:32:06.594 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 24 to 338 2024-09-13T14:32:06.594 INFO:tasks.rados.rados.0.smithi151.stdout:2595: expect (ObjNum 925 snap 318 seq_num 925) 2024-09-13T14:32:06.808 INFO:tasks.rados.rados.0.smithi151.stdout:2596: expect (ObjNum 683 snap 222 seq_num 683) 2024-09-13T14:32:06.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:06 smithi184 ceph-mon[92527]: pgmap v720: 89 pgs: 89 active+clean; 190 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 703 KiB/s wr, 3 op/s 2024-09-13T14:32:06.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:06 smithi184 ceph-mon[92527]: osdmap e735: 8 total, 8 up, 8 in 2024-09-13T14:32:06.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:06 smithi151 ceph-mon[104450]: pgmap v720: 89 pgs: 89 active+clean; 190 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 703 KiB/s wr, 3 op/s 2024-09-13T14:32:06.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:06 smithi151 ceph-mon[104450]: osdmap e735: 8 total, 8 up, 8 in 2024-09-13T14:32:06.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:06 smithi151 ceph-mon[108966]: pgmap v720: 89 pgs: 89 active+clean; 190 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 703 KiB/s wr, 3 op/s 2024-09-13T14:32:06.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:06 smithi151 ceph-mon[108966]: osdmap e735: 8 total, 8 up, 8 in 2024-09-13T14:32:07.027 INFO:tasks.rados.rados.0.smithi151.stdout:2598: finishing write tid 1 to smithi15186934-5 2024-09-13T14:32:07.027 INFO:tasks.rados.rados.0.smithi151.stdout:2599: finishing rollback tid 0 to smithi15186934-38 2024-09-13T14:32:07.027 INFO:tasks.rados.rados.0.smithi151.stdout:2598: finishing write tid 2 to smithi15186934-5 2024-09-13T14:32:07.027 INFO:tasks.rados.rados.0.smithi151.stdout:2599: finishing rollback tid 1 to smithi15186934-38 2024-09-13T14:32:07.027 INFO:tasks.rados.rados.0.smithi151.stdout:2598: finishing write tid 3 to smithi15186934-5 2024-09-13T14:32:07.027 INFO:tasks.rados.rados.0.smithi151.stdout:2599: finishing rollback tid 2 to smithi15186934-38 2024-09-13T14:32:07.027 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 1033 (ObjNum 942 snap 325 seq_num 942) dirty exists 2024-09-13T14:32:07.027 INFO:tasks.rados.rados.0.smithi151.stdout:2598: finishing write tid 4 to smithi15186934-5 2024-09-13T14:32:07.027 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 1003 (ObjNum 976 snap 339 seq_num 976) dirty exists 2024-09-13T14:32:07.028 INFO:tasks.rados.rados.0.smithi151.stdout:2598: left oid 5 (ObjNum 976 snap 339 seq_num 976) 2024-09-13T14:32:07.028 INFO:tasks.rados.rados.0.smithi151.stdout:2600: finishing write tid 1 to smithi15186934-8 2024-09-13T14:32:07.028 INFO:tasks.rados.rados.0.smithi151.stdout:2600: finishing write tid 2 to smithi15186934-8 2024-09-13T14:32:07.028 INFO:tasks.rados.rados.0.smithi151.stdout:2591: done (12 left) 2024-09-13T14:32:07.028 INFO:tasks.rados.rados.0.smithi151.stdout:2592: done (11 left) 2024-09-13T14:32:07.028 INFO:tasks.rados.rados.0.smithi151.stdout:2594: done (10 left) 2024-09-13T14:32:07.028 INFO:tasks.rados.rados.0.smithi151.stdout:2595: done (9 left) 2024-09-13T14:32:07.028 INFO:tasks.rados.rados.0.smithi151.stdout:2596: done (8 left) 2024-09-13T14:32:07.028 INFO:tasks.rados.rados.0.smithi151.stdout:2598: done (7 left) 2024-09-13T14:32:07.028 INFO:tasks.rados.rados.0.smithi151.stdout:2599: done (6 left) 2024-09-13T14:32:07.028 INFO:tasks.rados.rados.0.smithi151.stdout:2607: read oid 20 snap -1 2024-09-13T14:32:07.028 INFO:tasks.rados.rados.0.smithi151.stdout:2607: expect deleted 2024-09-13T14:32:07.028 INFO:tasks.rados.rados.0.smithi151.stdout:2608: read oid 35 snap -1 2024-09-13T14:32:07.028 INFO:tasks.rados.rados.0.smithi151.stdout:2608: expect (ObjNum 973 snap 339 seq_num 973) 2024-09-13T14:32:07.029 INFO:tasks.rados.rados.0.smithi151.stdout:2609: rollback oid 15 current snap is 339 2024-09-13T14:32:07.029 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 15 to 304 2024-09-13T14:32:07.029 INFO:tasks.rados.rados.0.smithi151.stdout:2600: finishing write tid 3 to smithi15186934-8 2024-09-13T14:32:07.029 INFO:tasks.rados.rados.0.smithi151.stdout:2600: finishing write tid 4 to smithi15186934-8 2024-09-13T14:32:07.029 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 794 (ObjNum 977 snap 339 seq_num 977) dirty exists 2024-09-13T14:32:07.029 INFO:tasks.rados.rados.0.smithi151.stdout:2600: left oid 8 (ObjNum 977 snap 339 seq_num 977) 2024-09-13T14:32:07.029 INFO:tasks.rados.rados.0.smithi151.stdout:2601: finishing write tid 1 to smithi15186934-39 2024-09-13T14:32:07.029 INFO:tasks.rados.rados.0.smithi151.stdout:2603: finishing rollback tid 1 to smithi15186934-9 2024-09-13T14:32:07.029 INFO:tasks.rados.rados.0.smithi151.stdout:2603: finishing rollback tid 2 to smithi15186934-9 2024-09-13T14:32:07.029 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 1091 (ObjNum 870 snap 308 seq_num 870) dirty exists 2024-09-13T14:32:07.029 INFO:tasks.rados.rados.0.smithi151.stdout:2601: finishing write tid 2 to smithi15186934-39 2024-09-13T14:32:07.029 INFO:tasks.rados.rados.0.smithi151.stdout:2600: done (8 left) 2024-09-13T14:32:07.029 INFO:tasks.rados.rados.0.smithi151.stdout:2603: done (7 left) 2024-09-13T14:32:07.029 INFO:tasks.rados.rados.0.smithi151.stdout:2610: write oid 25 current snap is 339 2024-09-13T14:32:07.029 INFO:tasks.rados.rados.0.smithi151.stdout:2601: finishing write tid 3 to smithi15186934-39 2024-09-13T14:32:07.030 INFO:tasks.rados.rados.0.smithi151.stdout:2601: finishing write tid 4 to smithi15186934-39 2024-09-13T14:32:07.030 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 1037 (ObjNum 978 snap 339 seq_num 978) dirty exists 2024-09-13T14:32:07.030 INFO:tasks.rados.rados.0.smithi151.stdout:2601: left oid 39 (ObjNum 978 snap 339 seq_num 978) 2024-09-13T14:32:07.030 INFO:tasks.rados.rados.0.smithi151.stdout:2602: expect (ObjNum 965 snap 330 seq_num 965) 2024-09-13T14:32:07.129 INFO:tasks.rados.rados.0.smithi151.stdout:2606: finishing rollback tid 0 to smithi15186934-24 2024-09-13T14:32:07.129 INFO:tasks.rados.rados.0.smithi151.stdout:2606: finishing rollback tid 1 to smithi15186934-24 2024-09-13T14:32:07.129 INFO:tasks.rados.rados.0.smithi151.stdout:2606: finishing rollback tid 2 to smithi15186934-24 2024-09-13T14:32:07.129 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 24 v 1541 (ObjNum 955 snap 327 seq_num 955) dirty exists 2024-09-13T14:32:07.129 INFO:tasks.rados.rados.0.smithi151.stdout:2609: finishing rollback tid 1 to smithi15186934-15 2024-09-13T14:32:07.129 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 15 v 1020 (ObjNum 65 snap 0 seq_num 0) dirty dne 2024-09-13T14:32:07.129 INFO:tasks.rados.rados.0.smithi151.stdout:2608: expect (ObjNum 973 snap 339 seq_num 973) 2024-09-13T14:32:07.204 INFO:tasks.rados.rados.0.smithi151.stdout:2610: seq_num 979 ranges {454053=561955,1422342=483679,2368404=85649} 2024-09-13T14:32:07.209 INFO:tasks.rados.rados.0.smithi151.stdout:2610: writing smithi15186934-25 from 454053 to 1016008 tid 1 2024-09-13T14:32:07.214 INFO:tasks.rados.rados.0.smithi151.stdout:2610: writing smithi15186934-25 from 1422342 to 1906021 tid 2 2024-09-13T14:32:07.218 INFO:tasks.rados.rados.0.smithi151.stdout:2610: writing smithi15186934-25 from 2368404 to 2454053 tid 3 2024-09-13T14:32:07.218 INFO:tasks.rados.rados.0.smithi151.stdout:2601: done (7 left) 2024-09-13T14:32:07.218 INFO:tasks.rados.rados.0.smithi151.stdout:2602: done (6 left) 2024-09-13T14:32:07.218 INFO:tasks.rados.rados.0.smithi151.stdout:2605: done (5 left) 2024-09-13T14:32:07.218 INFO:tasks.rados.rados.0.smithi151.stdout:2606: done (4 left) 2024-09-13T14:32:07.218 INFO:tasks.rados.rados.0.smithi151.stdout:2607: done (3 left) 2024-09-13T14:32:07.218 INFO:tasks.rados.rados.0.smithi151.stdout:2608: done (2 left) 2024-09-13T14:32:07.218 INFO:tasks.rados.rados.0.smithi151.stdout:2609: done (1 left) 2024-09-13T14:32:07.218 INFO:tasks.rados.rados.0.smithi151.stdout:2611: snap_create 2024-09-13T14:32:07.218 INFO:tasks.rados.rados.0.smithi151.stdout:2610: finishing write tid 1 to smithi15186934-25 2024-09-13T14:32:07.218 INFO:tasks.rados.rados.0.smithi151.stdout:2610: finishing write tid 2 to smithi15186934-25 2024-09-13T14:32:07.219 INFO:tasks.rados.rados.0.smithi151.stdout:2610: finishing write tid 3 to smithi15186934-25 2024-09-13T14:32:07.220 INFO:tasks.rados.rados.0.smithi151.stdout:2610: finishing write tid 4 to smithi15186934-25 2024-09-13T14:32:07.220 INFO:tasks.rados.rados.0.smithi151.stdout:2610: finishing write tid 5 to smithi15186934-25 2024-09-13T14:32:07.220 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 1317 (ObjNum 979 snap 339 seq_num 979) dirty exists 2024-09-13T14:32:07.220 INFO:tasks.rados.rados.0.smithi151.stdout:2610: left oid 25 (ObjNum 979 snap 339 seq_num 979) 2024-09-13T14:32:07.596 INFO:tasks.rados.rados.0.smithi151.stdout:2610: done (1 left) 2024-09-13T14:32:07.596 INFO:tasks.rados.rados.0.smithi151.stdout:2611: done (0 left) 2024-09-13T14:32:07.596 INFO:tasks.rados.rados.0.smithi151.stdout:2612: write oid 10 current snap is 340 2024-09-13T14:32:07.597 INFO:tasks.rados.rados.0.smithi151.stdout:2612: seq_num 980 ranges {470860=637204,1880824=590036} 2024-09-13T14:32:07.602 INFO:tasks.rados.rados.0.smithi151.stdout:2612: writing smithi15186934-10 from 470860 to 1108064 tid 1 2024-09-13T14:32:07.610 INFO:tasks.rados.rados.0.smithi151.stdout:2612: writing smithi15186934-10 from 1880824 to 2470860 tid 2 2024-09-13T14:32:07.610 INFO:tasks.rados.rados.0.smithi151.stdout:2613: read oid 7 snap 321 2024-09-13T14:32:07.610 INFO:tasks.rados.rados.0.smithi151.stdout:2613: expect (ObjNum 855 snap 295 seq_num 855) 2024-09-13T14:32:07.610 INFO:tasks.rados.rados.0.smithi151.stdout:2614: rollback oid 15 current snap is 340 2024-09-13T14:32:07.611 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 15 to 334 2024-09-13T14:32:07.611 INFO:tasks.rados.rados.0.smithi151.stdout:2612: finishing write tid 1 to smithi15186934-10 2024-09-13T14:32:07.611 INFO:tasks.rados.rados.0.smithi151.stdout:2615: read oid 29 snap 334 2024-09-13T14:32:07.611 INFO:tasks.rados.rados.0.smithi151.stdout:2615: expect deleted 2024-09-13T14:32:07.611 INFO:tasks.rados.rados.0.smithi151.stdout:2616: read oid 35 snap 338 2024-09-13T14:32:07.611 INFO:tasks.rados.rados.0.smithi151.stdout:2616: expect deleted 2024-09-13T14:32:07.611 INFO:tasks.rados.rados.0.smithi151.stdout:2617: delete oid 13 current snap is 340 2024-09-13T14:32:07.613 INFO:tasks.rados.rados.0.smithi151.stdout:2615: done (5 left) 2024-09-13T14:32:07.613 INFO:tasks.rados.rados.0.smithi151.stdout:2617: done (4 left) 2024-09-13T14:32:07.613 INFO:tasks.rados.rados.0.smithi151.stdout:2618: snap_remove snap 309 2024-09-13T14:32:07.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:07 smithi184 ceph-mon[92527]: osdmap e736: 8 total, 8 up, 8 in 2024-09-13T14:32:07.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:07 smithi151 ceph-mon[104450]: osdmap e736: 8 total, 8 up, 8 in 2024-09-13T14:32:07.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:07 smithi151 ceph-mon[108966]: osdmap e736: 8 total, 8 up, 8 in 2024-09-13T14:32:08.601 INFO:tasks.rados.rados.0.smithi151.stdout:2618: done (4 left) 2024-09-13T14:32:08.601 INFO:tasks.rados.rados.0.smithi151.stdout:2619: rollback oid 46 current snap is 340 2024-09-13T14:32:08.601 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 46 to 336 2024-09-13T14:32:08.601 INFO:tasks.rados.rados.0.smithi151.stdout:2614: finishing rollback tid 1 to smithi15186934-15 2024-09-13T14:32:08.601 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 15 v 1020 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-13T14:32:08.601 INFO:tasks.rados.rados.0.smithi151.stdout:2612: finishing write tid 2 to smithi15186934-10 2024-09-13T14:32:08.601 INFO:tasks.rados.rados.0.smithi151.stdout:2612: finishing write tid 3 to smithi15186934-10 2024-09-13T14:32:08.601 INFO:tasks.rados.rados.0.smithi151.stdout:2612: finishing write tid 4 to smithi15186934-10 2024-09-13T14:32:08.602 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 932 (ObjNum 980 snap 340 seq_num 980) dirty exists 2024-09-13T14:32:08.602 INFO:tasks.rados.rados.0.smithi151.stdout:2612: left oid 10 (ObjNum 980 snap 340 seq_num 980) 2024-09-13T14:32:08.602 INFO:tasks.rados.rados.0.smithi151.stdout:2613: expect (ObjNum 855 snap 295 seq_num 855) 2024-09-13T14:32:08.735 INFO:tasks.rados.rados.0.smithi151.stdout:2619: finishing rollback tid 1 to smithi15186934-46 2024-09-13T14:32:08.735 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 46 v 863 (ObjNum 129 snap 0 seq_num 3092593248) dirty dne 2024-09-13T14:32:08.735 INFO:tasks.rados.rados.0.smithi151.stdout:2612: done (4 left) 2024-09-13T14:32:08.735 INFO:tasks.rados.rados.0.smithi151.stdout:2613: done (3 left) 2024-09-13T14:32:08.736 INFO:tasks.rados.rados.0.smithi151.stdout:2614: done (2 left) 2024-09-13T14:32:08.736 INFO:tasks.rados.rados.0.smithi151.stdout:2616: done (1 left) 2024-09-13T14:32:08.736 INFO:tasks.rados.rados.0.smithi151.stdout:2619: done (0 left) 2024-09-13T14:32:08.736 INFO:tasks.rados.rados.0.smithi151.stdout:2620: delete oid 19 current snap is 340 2024-09-13T14:32:08.737 INFO:tasks.rados.rados.0.smithi151.stdout:2620: done (0 left) 2024-09-13T14:32:08.737 INFO:tasks.rados.rados.0.smithi151.stdout:2621: snap_remove snap 321 2024-09-13T14:32:08.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:08 smithi184 ceph-mon[92527]: pgmap v723: 89 pgs: 89 active+clean; 185 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.3 MiB/s wr, 17 op/s 2024-09-13T14:32:08.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:08 smithi184 ceph-mon[92527]: osdmap e737: 8 total, 8 up, 8 in 2024-09-13T14:32:08.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:08 smithi151 ceph-mon[104450]: pgmap v723: 89 pgs: 89 active+clean; 185 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.3 MiB/s wr, 17 op/s 2024-09-13T14:32:08.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:08 smithi151 ceph-mon[104450]: osdmap e737: 8 total, 8 up, 8 in 2024-09-13T14:32:08.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:08 smithi151 ceph-mon[108966]: pgmap v723: 89 pgs: 89 active+clean; 185 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.3 MiB/s wr, 17 op/s 2024-09-13T14:32:08.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:08 smithi151 ceph-mon[108966]: osdmap e737: 8 total, 8 up, 8 in 2024-09-13T14:32:09.569 INFO:tasks.rados.rados.0.smithi151.stdout:2621: done (0 left) 2024-09-13T14:32:09.569 INFO:tasks.rados.rados.0.smithi151.stdout:2622: delete oid 38 current snap is 340 2024-09-13T14:32:09.579 INFO:tasks.rados.rados.0.smithi151.stdout:2622: done (0 left) 2024-09-13T14:32:09.579 INFO:tasks.rados.rados.0.smithi151.stdout:2623: delete oid 32 current snap is 340 2024-09-13T14:32:09.582 INFO:tasks.rados.rados.0.smithi151.stdout:2623: done (0 left) 2024-09-13T14:32:09.582 INFO:tasks.rados.rados.0.smithi151.stdout:2624: snap_remove snap 334 2024-09-13T14:32:09.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:09 smithi151 ceph-mon[104450]: osdmap e738: 8 total, 8 up, 8 in 2024-09-13T14:32:09.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:09 smithi151 ceph-mon[104450]: osdmap e739: 8 total, 8 up, 8 in 2024-09-13T14:32:09.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:09 smithi151 ceph-mon[108966]: osdmap e738: 8 total, 8 up, 8 in 2024-09-13T14:32:09.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:09 smithi151 ceph-mon[108966]: osdmap e739: 8 total, 8 up, 8 in 2024-09-13T14:32:10.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:09 smithi184 ceph-mon[92527]: osdmap e738: 8 total, 8 up, 8 in 2024-09-13T14:32:10.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:09 smithi184 ceph-mon[92527]: osdmap e739: 8 total, 8 up, 8 in 2024-09-13T14:32:10.572 INFO:tasks.rados.rados.0.smithi151.stdout:2624: done (0 left) 2024-09-13T14:32:10.572 INFO:tasks.rados.rados.0.smithi151.stdout:2625: snap_create 2024-09-13T14:32:10.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:32:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:32:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:32:10.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:10 smithi151 ceph-mon[104450]: pgmap v726: 89 pgs: 89 active+clean; 188 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 18 op/s 2024-09-13T14:32:10.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:10 smithi151 ceph-mon[104450]: osdmap e740: 8 total, 8 up, 8 in 2024-09-13T14:32:10.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:10 smithi151 ceph-mon[108966]: pgmap v726: 89 pgs: 89 active+clean; 188 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 18 op/s 2024-09-13T14:32:10.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:10 smithi151 ceph-mon[108966]: osdmap e740: 8 total, 8 up, 8 in 2024-09-13T14:32:11.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:10 smithi184 ceph-mon[92527]: pgmap v726: 89 pgs: 89 active+clean; 188 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 18 op/s 2024-09-13T14:32:11.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:10 smithi184 ceph-mon[92527]: osdmap e740: 8 total, 8 up, 8 in 2024-09-13T14:32:11.574 INFO:tasks.rados.rados.0.smithi151.stdout:2625: done (0 left) 2024-09-13T14:32:11.574 INFO:tasks.rados.rados.0.smithi151.stdout:2626: snap_remove snap 335 2024-09-13T14:32:12.577 INFO:tasks.rados.rados.0.smithi151.stdout:2626: done (0 left) 2024-09-13T14:32:12.577 INFO:tasks.rados.rados.0.smithi151.stdout:2627: rollback oid 42 current snap is 341 2024-09-13T14:32:12.577 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 42 to 337 2024-09-13T14:32:12.577 INFO:tasks.rados.rados.0.smithi151.stdout:2628: rollback oid 5 current snap is 341 2024-09-13T14:32:12.578 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 5 to 332 2024-09-13T14:32:12.578 INFO:tasks.rados.rados.0.smithi151.stdout:2629: delete oid 47 current snap is 341 2024-09-13T14:32:12.579 INFO:tasks.rados.rados.0.smithi151.stdout:2627: finishing rollback tid 1 to smithi15186934-42 2024-09-13T14:32:12.579 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 0 (ObjNum 289 snap 0 seq_num 1755291120) dirty dne 2024-09-13T14:32:12.585 INFO:tasks.rados.rados.0.smithi151.stdout:2628: finishing rollback tid 0 to smithi15186934-5 2024-09-13T14:32:12.586 INFO:tasks.rados.rados.0.smithi151.stdout:2627: done (2 left) 2024-09-13T14:32:12.586 INFO:tasks.rados.rados.0.smithi151.stdout:2629: done (1 left) 2024-09-13T14:32:12.587 INFO:tasks.rados.rados.0.smithi151.stdout:2630: write oid 43 current snap is 341 2024-09-13T14:32:12.587 INFO:tasks.rados.rados.0.smithi151.stdout:2630: seq_num 986 ranges {571702=405051,1431716=638344,2560365=11337} 2024-09-13T14:32:12.591 INFO:tasks.rados.rados.0.smithi151.stdout:2630: writing smithi15186934-43 from 571702 to 976753 tid 1 2024-09-13T14:32:12.598 INFO:tasks.rados.rados.0.smithi151.stdout:2630: writing smithi15186934-43 from 1431716 to 2070060 tid 2 2024-09-13T14:32:12.604 INFO:tasks.rados.rados.0.smithi151.stdout:2630: writing smithi15186934-43 from 2560365 to 2571702 tid 3 2024-09-13T14:32:12.604 INFO:tasks.rados.rados.0.smithi151.stdout:2631: rollback oid 6 current snap is 341 2024-09-13T14:32:12.604 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 6 to 340 2024-09-13T14:32:12.605 INFO:tasks.rados.rados.0.smithi151.stdout:2628: finishing rollback tid 1 to smithi15186934-5 2024-09-13T14:32:12.605 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 1011 (ObjNum 129 snap 0 seq_num 3092548968) dirty dne 2024-09-13T14:32:12.605 INFO:tasks.rados.rados.0.smithi151.stdout:2630: finishing write tid 1 to smithi15186934-43 2024-09-13T14:32:12.605 INFO:tasks.rados.rados.0.smithi151.stdout:2628: done (2 left) 2024-09-13T14:32:12.605 INFO:tasks.rados.rados.0.smithi151.stdout:2632: rollback oid 13 current snap is 341 2024-09-13T14:32:12.605 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 13 to 327 2024-09-13T14:32:12.605 INFO:tasks.rados.rados.0.smithi151.stdout:2633: delete oid 15 current snap is 341 2024-09-13T14:32:12.605 INFO:tasks.rados.rados.0.smithi151.stdout:2632: finishing rollback tid 1 to smithi15186934-13 2024-09-13T14:32:12.605 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 13 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-13T14:32:12.605 INFO:tasks.rados.rados.0.smithi151.stdout:2631: finishing rollback tid 1 to smithi15186934-6 2024-09-13T14:32:12.605 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 6 v 629 (ObjNum 4497 snap 0 seq_num 3093096224) dirty dne 2024-09-13T14:32:12.606 INFO:tasks.rados.rados.0.smithi151.stdout:2631: done (3 left) 2024-09-13T14:32:12.606 INFO:tasks.rados.rados.0.smithi151.stdout:2632: done (2 left) 2024-09-13T14:32:12.606 INFO:tasks.rados.rados.0.smithi151.stdout:2633: done (1 left) 2024-09-13T14:32:12.606 INFO:tasks.rados.rados.0.smithi151.stdout:2634: read oid 37 snap -1 2024-09-13T14:32:12.606 INFO:tasks.rados.rados.0.smithi151.stdout:2634: expect (ObjNum 683 snap 222 seq_num 683) 2024-09-13T14:32:12.606 INFO:tasks.rados.rados.0.smithi151.stdout:2635: write oid 35 current snap is 341 2024-09-13T14:32:12.606 INFO:tasks.rados.rados.0.smithi151.stdout:2630: finishing write tid 2 to smithi15186934-43 2024-09-13T14:32:12.606 INFO:tasks.rados.rados.0.smithi151.stdout:2635: seq_num 988 ranges {605316=555549,1832327=642013} 2024-09-13T14:32:12.610 INFO:tasks.rados.rados.0.smithi151.stdout:2635: writing smithi15186934-35 from 605316 to 1160865 tid 1 2024-09-13T14:32:12.687 INFO:tasks.rados.rados.0.smithi151.stdout:2635: writing smithi15186934-35 from 1832327 to 2474340 tid 2 2024-09-13T14:32:12.687 INFO:tasks.rados.rados.0.smithi151.stdout:2636: read oid 39 snap -1 2024-09-13T14:32:12.687 INFO:tasks.rados.rados.0.smithi151.stdout:2636: expect (ObjNum 978 snap 339 seq_num 978) 2024-09-13T14:32:12.687 INFO:tasks.rados.rados.0.smithi151.stdout:2637: read oid 33 snap -1 2024-09-13T14:32:12.687 INFO:tasks.rados.rados.0.smithi151.stdout:2637: expect deleted 2024-09-13T14:32:12.687 INFO:tasks.rados.rados.0.smithi151.stdout:2638: snap_create 2024-09-13T14:32:12.688 INFO:tasks.rados.rados.0.smithi151.stdout:2630: finishing write tid 3 to smithi15186934-43 2024-09-13T14:32:12.688 INFO:tasks.rados.rados.0.smithi151.stdout:2630: finishing write tid 4 to smithi15186934-43 2024-09-13T14:32:12.688 INFO:tasks.rados.rados.0.smithi151.stdout:2630: finishing write tid 5 to smithi15186934-43 2024-09-13T14:32:12.688 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 43 v 1566 (ObjNum 986 snap 341 seq_num 986) dirty exists 2024-09-13T14:32:12.688 INFO:tasks.rados.rados.0.smithi151.stdout:2630: left oid 43 (ObjNum 986 snap 341 seq_num 986) 2024-09-13T14:32:12.688 INFO:tasks.rados.rados.0.smithi151.stdout:2634: expect (ObjNum 683 snap 222 seq_num 683) 2024-09-13T14:32:12.838 INFO:tasks.rados.rados.0.smithi151.stdout:2635: finishing write tid 1 to smithi15186934-35 2024-09-13T14:32:12.838 INFO:tasks.rados.rados.0.smithi151.stdout:2635: finishing write tid 2 to smithi15186934-35 2024-09-13T14:32:12.838 INFO:tasks.rados.rados.0.smithi151.stdout:2635: finishing write tid 3 to smithi15186934-35 2024-09-13T14:32:12.838 INFO:tasks.rados.rados.0.smithi151.stdout:2635: finishing write tid 4 to smithi15186934-35 2024-09-13T14:32:12.838 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 35 v 1570 (ObjNum 988 snap 341 seq_num 988) dirty exists 2024-09-13T14:32:12.838 INFO:tasks.rados.rados.0.smithi151.stdout:2635: left oid 35 (ObjNum 988 snap 341 seq_num 988) 2024-09-13T14:32:12.838 INFO:tasks.rados.rados.0.smithi151.stdout:2636: expect (ObjNum 978 snap 339 seq_num 978) 2024-09-13T14:32:12.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:12 smithi184 ceph-mon[92527]: pgmap v729: 89 pgs: 89 active+clean; 183 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 849 KiB/s wr, 4 op/s 2024-09-13T14:32:12.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:12 smithi184 ceph-mon[92527]: osdmap e741: 8 total, 8 up, 8 in 2024-09-13T14:32:12.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:12 smithi151 ceph-mon[104450]: pgmap v729: 89 pgs: 89 active+clean; 183 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 849 KiB/s wr, 4 op/s 2024-09-13T14:32:12.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:12 smithi151 ceph-mon[104450]: osdmap e741: 8 total, 8 up, 8 in 2024-09-13T14:32:12.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:12 smithi151 ceph-mon[108966]: pgmap v729: 89 pgs: 89 active+clean; 183 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 849 KiB/s wr, 4 op/s 2024-09-13T14:32:12.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:12 smithi151 ceph-mon[108966]: osdmap e741: 8 total, 8 up, 8 in 2024-09-13T14:32:13.589 INFO:tasks.rados.rados.0.smithi151.stdout:2630: done (5 left) 2024-09-13T14:32:13.589 INFO:tasks.rados.rados.0.smithi151.stdout:2634: done (4 left) 2024-09-13T14:32:13.589 INFO:tasks.rados.rados.0.smithi151.stdout:2635: done (3 left) 2024-09-13T14:32:13.590 INFO:tasks.rados.rados.0.smithi151.stdout:2636: done (2 left) 2024-09-13T14:32:13.590 INFO:tasks.rados.rados.0.smithi151.stdout:2637: done (1 left) 2024-09-13T14:32:13.590 INFO:tasks.rados.rados.0.smithi151.stdout:2638: done (0 left) 2024-09-13T14:32:13.590 INFO:tasks.rados.rados.0.smithi151.stdout:2639: write oid 20 current snap is 342 2024-09-13T14:32:13.590 INFO:tasks.rados.rados.0.smithi151.stdout:2639: seq_num 989 ranges {622123=747151,2007162=614961} 2024-09-13T14:32:13.596 INFO:tasks.rados.rados.0.smithi151.stdout:2639: writing smithi15186934-20 from 622123 to 1369274 tid 1 2024-09-13T14:32:13.604 INFO:tasks.rados.rados.0.smithi151.stdout:2639: writing smithi15186934-20 from 2007162 to 2622123 tid 2 2024-09-13T14:32:13.604 INFO:tasks.rados.rados.0.smithi151.stdout:2640: rollback oid 39 current snap is 342 2024-09-13T14:32:13.604 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 39 to 340 2024-09-13T14:32:13.604 INFO:tasks.rados.rados.0.smithi151.stdout:2639: finishing write tid 1 to smithi15186934-20 2024-09-13T14:32:13.604 INFO:tasks.rados.rados.0.smithi151.stdout:2641: delete oid 9 current snap is 342 2024-09-13T14:32:13.608 INFO:tasks.rados.rados.0.smithi151.stdout:2639: finishing write tid 2 to smithi15186934-20 2024-09-13T14:32:13.608 INFO:tasks.rados.rados.0.smithi151.stdout:2640: finishing rollback tid 0 to smithi15186934-39 2024-09-13T14:32:13.608 INFO:tasks.rados.rados.0.smithi151.stdout:2639: finishing write tid 3 to smithi15186934-20 2024-09-13T14:32:13.608 INFO:tasks.rados.rados.0.smithi151.stdout:2639: finishing write tid 4 to smithi15186934-20 2024-09-13T14:32:13.608 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 670 (ObjNum 989 snap 342 seq_num 989) dirty exists 2024-09-13T14:32:13.608 INFO:tasks.rados.rados.0.smithi151.stdout:2639: left oid 20 (ObjNum 989 snap 342 seq_num 989) 2024-09-13T14:32:13.609 INFO:tasks.rados.rados.0.smithi151.stdout:2640: finishing rollback tid 1 to smithi15186934-39 2024-09-13T14:32:13.609 INFO:tasks.rados.rados.0.smithi151.stdout:2640: finishing rollback tid 2 to smithi15186934-39 2024-09-13T14:32:13.609 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 1065 (ObjNum 978 snap 339 seq_num 978) dirty exists 2024-09-13T14:32:13.609 INFO:tasks.rados.rados.0.smithi151.stdout:2639: done (2 left) 2024-09-13T14:32:13.609 INFO:tasks.rados.rados.0.smithi151.stdout:2640: done (1 left) 2024-09-13T14:32:13.609 INFO:tasks.rados.rados.0.smithi151.stdout:2641: done (0 left) 2024-09-13T14:32:13.609 INFO:tasks.rados.rados.0.smithi151.stdout:2642: read oid 37 snap -1 2024-09-13T14:32:13.610 INFO:tasks.rados.rados.0.smithi151.stdout:2642: expect (ObjNum 683 snap 222 seq_num 683) 2024-09-13T14:32:13.610 INFO:tasks.rados.rados.0.smithi151.stdout:2643: write oid 25 current snap is 342 2024-09-13T14:32:13.610 INFO:tasks.rados.rados.0.smithi151.stdout:2643: seq_num 991 ranges {655737=497649,1607773=789340} 2024-09-13T14:32:13.615 INFO:tasks.rados.rados.0.smithi151.stdout:2643: writing smithi15186934-25 from 655737 to 1153386 tid 1 2024-09-13T14:32:13.623 INFO:tasks.rados.rados.0.smithi151.stdout:2643: writing smithi15186934-25 from 1607773 to 2397113 tid 2 2024-09-13T14:32:13.623 INFO:tasks.rados.rados.0.smithi151.stdout:2644: read oid 14 snap -1 2024-09-13T14:32:13.623 INFO:tasks.rados.rados.0.smithi151.stdout:2644: expect (ObjNum 908 snap 315 seq_num 908) 2024-09-13T14:32:13.623 INFO:tasks.rados.rados.0.smithi151.stdout:2645: snap_remove snap 304 2024-09-13T14:32:13.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:13 smithi184 ceph-mon[92527]: osdmap e742: 8 total, 8 up, 8 in 2024-09-13T14:32:13.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:13 smithi151 ceph-mon[104450]: osdmap e742: 8 total, 8 up, 8 in 2024-09-13T14:32:13.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:13 smithi151 ceph-mon[108966]: osdmap e742: 8 total, 8 up, 8 in 2024-09-13T14:32:14.572 INFO:tasks.rados.rados.0.smithi151.stdout:2645: done (3 left) 2024-09-13T14:32:14.572 INFO:tasks.rados.rados.0.smithi151.stdout:2646: write oid 33 current snap is 342 2024-09-13T14:32:14.572 INFO:tasks.rados.rados.0.smithi151.stdout:2646: seq_num 992 ranges {672544=572898,1666255=532998} 2024-09-13T14:32:14.579 INFO:tasks.rados.rados.0.smithi151.stdout:2646: writing smithi15186934-33 from 672544 to 1245442 tid 1 2024-09-13T14:32:14.586 INFO:tasks.rados.rados.0.smithi151.stdout:2646: writing smithi15186934-33 from 1666255 to 2199253 tid 2 2024-09-13T14:32:14.586 INFO:tasks.rados.rados.0.smithi151.stdout:2642: expect (ObjNum 683 snap 222 seq_num 683) 2024-09-13T14:32:14.810 INFO:tasks.rados.rados.0.smithi151.stdout:2643: finishing write tid 1 to smithi15186934-25 2024-09-13T14:32:14.810 INFO:tasks.rados.rados.0.smithi151.stdout:2643: finishing write tid 2 to smithi15186934-25 2024-09-13T14:32:14.810 INFO:tasks.rados.rados.0.smithi151.stdout:2643: finishing write tid 3 to smithi15186934-25 2024-09-13T14:32:14.811 INFO:tasks.rados.rados.0.smithi151.stdout:2643: finishing write tid 4 to smithi15186934-25 2024-09-13T14:32:14.811 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 1331 (ObjNum 991 snap 342 seq_num 991) dirty exists 2024-09-13T14:32:14.811 INFO:tasks.rados.rados.0.smithi151.stdout:2643: left oid 25 (ObjNum 991 snap 342 seq_num 991) 2024-09-13T14:32:14.811 INFO:tasks.rados.rados.0.smithi151.stdout:2644: expect (ObjNum 908 snap 315 seq_num 908) 2024-09-13T14:32:14.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:14 smithi184 ceph-mon[92527]: pgmap v732: 89 pgs: 89 active+clean; 173 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 494 KiB/s wr, 5 op/s 2024-09-13T14:32:14.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:14 smithi184 ceph-mon[92527]: osdmap e743: 8 total, 8 up, 8 in 2024-09-13T14:32:14.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:14 smithi184 ceph-mon[92527]: osdmap e744: 8 total, 8 up, 8 in 2024-09-13T14:32:14.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:14 smithi151 ceph-mon[108966]: pgmap v732: 89 pgs: 89 active+clean; 173 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 494 KiB/s wr, 5 op/s 2024-09-13T14:32:14.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:14 smithi151 ceph-mon[108966]: osdmap e743: 8 total, 8 up, 8 in 2024-09-13T14:32:14.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:14 smithi151 ceph-mon[108966]: osdmap e744: 8 total, 8 up, 8 in 2024-09-13T14:32:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:14 smithi151 ceph-mon[104450]: pgmap v732: 89 pgs: 89 active+clean; 173 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 494 KiB/s wr, 5 op/s 2024-09-13T14:32:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:14 smithi151 ceph-mon[104450]: osdmap e743: 8 total, 8 up, 8 in 2024-09-13T14:32:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:14 smithi151 ceph-mon[104450]: osdmap e744: 8 total, 8 up, 8 in 2024-09-13T14:32:14.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:32:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:32:14] "GET /metrics HTTP/1.1" 200 36508 "" "Prometheus/2.51.0" 2024-09-13T14:32:14.915 INFO:tasks.rados.rados.0.smithi151.stdout:2646: finishing write tid 1 to smithi15186934-33 2024-09-13T14:32:14.915 INFO:tasks.rados.rados.0.smithi151.stdout:2646: finishing write tid 2 to smithi15186934-33 2024-09-13T14:32:14.915 INFO:tasks.rados.rados.0.smithi151.stdout:2646: finishing write tid 3 to smithi15186934-33 2024-09-13T14:32:14.915 INFO:tasks.rados.rados.0.smithi151.stdout:2646: finishing write tid 4 to smithi15186934-33 2024-09-13T14:32:14.916 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 33 v 1016 (ObjNum 992 snap 342 seq_num 992) dirty exists 2024-09-13T14:32:14.916 INFO:tasks.rados.rados.0.smithi151.stdout:2646: left oid 33 (ObjNum 992 snap 342 seq_num 992) 2024-09-13T14:32:14.916 INFO:tasks.rados.rados.0.smithi151.stdout:2642: done (3 left) 2024-09-13T14:32:14.916 INFO:tasks.rados.rados.0.smithi151.stdout:2643: done (2 left) 2024-09-13T14:32:14.916 INFO:tasks.rados.rados.0.smithi151.stdout:2644: done (1 left) 2024-09-13T14:32:14.916 INFO:tasks.rados.rados.0.smithi151.stdout:2646: done (0 left) 2024-09-13T14:32:14.916 INFO:tasks.rados.rados.0.smithi151.stdout:2647: write oid 15 current snap is 342 2024-09-13T14:32:14.916 INFO:tasks.rados.rados.0.smithi151.stdout:2647: seq_num 993 ranges {689351=648147,2124737=564614} 2024-09-13T14:32:14.922 INFO:tasks.rados.rados.0.smithi151.stdout:2647: writing smithi15186934-15 from 689351 to 1337498 tid 1 2024-09-13T14:32:14.929 INFO:tasks.rados.rados.0.smithi151.stdout:2647: writing smithi15186934-15 from 2124737 to 2689351 tid 2 2024-09-13T14:32:14.929 INFO:tasks.rados.rados.0.smithi151.stdout:2648: snap_create 2024-09-13T14:32:14.929 INFO:tasks.rados.rados.0.smithi151.stdout:2647: finishing write tid 1 to smithi15186934-15 2024-09-13T14:32:14.932 INFO:tasks.rados.rados.0.smithi151.stdout:2647: finishing write tid 2 to smithi15186934-15 2024-09-13T14:32:14.932 INFO:tasks.rados.rados.0.smithi151.stdout:2647: finishing write tid 3 to smithi15186934-15 2024-09-13T14:32:14.933 INFO:tasks.rados.rados.0.smithi151.stdout:2647: finishing write tid 4 to smithi15186934-15 2024-09-13T14:32:14.933 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 15 v 1033 (ObjNum 993 snap 342 seq_num 993) dirty exists 2024-09-13T14:32:14.933 INFO:tasks.rados.rados.0.smithi151.stdout:2647: left oid 15 (ObjNum 993 snap 342 seq_num 993) 2024-09-13T14:32:15.599 INFO:tasks.rados.rados.0.smithi151.stdout:2647: done (1 left) 2024-09-13T14:32:15.599 INFO:tasks.rados.rados.0.smithi151.stdout:2648: done (0 left) 2024-09-13T14:32:15.599 INFO:tasks.rados.rados.0.smithi151.stdout:2649: delete oid 37 current snap is 343 2024-09-13T14:32:15.602 INFO:tasks.rados.rados.0.smithi151.stdout:2649: done (0 left) 2024-09-13T14:32:15.602 INFO:tasks.rados.rados.0.smithi151.stdout:2650: snap_create 2024-09-13T14:32:16.605 INFO:tasks.rados.rados.0.smithi151.stdout:2650: done (0 left) 2024-09-13T14:32:16.605 INFO:tasks.rados.rados.0.smithi151.stdout:2651: write oid 43 current snap is 344 2024-09-13T14:32:16.605 INFO:tasks.rados.rados.0.smithi151.stdout:2651: seq_num 995 ranges {722965=798645,2125348=597617} 2024-09-13T14:32:16.613 INFO:tasks.rados.rados.0.smithi151.stdout:2651: writing smithi15186934-43 from 722965 to 1521610 tid 1 2024-09-13T14:32:16.620 INFO:tasks.rados.rados.0.smithi151.stdout:2651: writing smithi15186934-43 from 2125348 to 2722965 tid 2 2024-09-13T14:32:16.620 INFO:tasks.rados.rados.0.smithi151.stdout:2652: delete oid 14 current snap is 344 2024-09-13T14:32:16.620 INFO:tasks.rados.rados.0.smithi151.stdout:2651: finishing write tid 1 to smithi15186934-43 2024-09-13T14:32:16.624 INFO:tasks.rados.rados.0.smithi151.stdout:2651: finishing write tid 2 to smithi15186934-43 2024-09-13T14:32:16.625 INFO:tasks.rados.rados.0.smithi151.stdout:2651: finishing write tid 3 to smithi15186934-43 2024-09-13T14:32:16.625 INFO:tasks.rados.rados.0.smithi151.stdout:2651: finishing write tid 4 to smithi15186934-43 2024-09-13T14:32:16.625 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 43 v 1580 (ObjNum 995 snap 344 seq_num 995) dirty exists 2024-09-13T14:32:16.625 INFO:tasks.rados.rados.0.smithi151.stdout:2651: left oid 43 (ObjNum 995 snap 344 seq_num 995) 2024-09-13T14:32:16.625 INFO:tasks.rados.rados.0.smithi151.stdout:2651: done (1 left) 2024-09-13T14:32:16.625 INFO:tasks.rados.rados.0.smithi151.stdout:2652: done (0 left) 2024-09-13T14:32:16.625 INFO:tasks.rados.rados.0.smithi151.stdout:2653: read oid 2 snap -1 2024-09-13T14:32:16.625 INFO:tasks.rados.rados.0.smithi151.stdout:2653: expect deleted 2024-09-13T14:32:16.625 INFO:tasks.rados.rados.0.smithi151.stdout:2654: read oid 44 snap -1 2024-09-13T14:32:16.625 INFO:tasks.rados.rados.0.smithi151.stdout:2654: expect deleted 2024-09-13T14:32:16.625 INFO:tasks.rados.rados.0.smithi151.stdout:2655: write oid 20 current snap is 344 2024-09-13T14:32:16.625 INFO:tasks.rados.rados.0.smithi151.stdout:2655: seq_num 997 ranges {756579=665496,1958665=400347} 2024-09-13T14:32:16.632 INFO:tasks.rados.rados.0.smithi151.stdout:2655: writing smithi15186934-20 from 756579 to 1422075 tid 1 2024-09-13T14:32:16.637 INFO:tasks.rados.rados.0.smithi151.stdout:2655: writing smithi15186934-20 from 1958665 to 2359012 tid 2 2024-09-13T14:32:16.637 INFO:tasks.rados.rados.0.smithi151.stdout:2656: delete oid 5 current snap is 344 2024-09-13T14:32:16.638 INFO:tasks.rados.rados.0.smithi151.stdout:2655: finishing write tid 1 to smithi15186934-20 2024-09-13T14:32:16.640 INFO:tasks.rados.rados.0.smithi151.stdout:2653: done (3 left) 2024-09-13T14:32:16.640 INFO:tasks.rados.rados.0.smithi151.stdout:2654: done (2 left) 2024-09-13T14:32:16.640 INFO:tasks.rados.rados.0.smithi151.stdout:2656: done (1 left) 2024-09-13T14:32:16.640 INFO:tasks.rados.rados.0.smithi151.stdout:2657: rollback oid 13 current snap is 344 2024-09-13T14:32:16.640 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 13 to 327 2024-09-13T14:32:16.640 INFO:tasks.rados.rados.0.smithi151.stdout:2658: snap_create 2024-09-13T14:32:16.641 INFO:tasks.rados.rados.0.smithi151.stdout:2655: finishing write tid 2 to smithi15186934-20 2024-09-13T14:32:16.641 INFO:tasks.rados.rados.0.smithi151.stdout:2655: finishing write tid 3 to smithi15186934-20 2024-09-13T14:32:16.641 INFO:tasks.rados.rados.0.smithi151.stdout:2655: finishing write tid 4 to smithi15186934-20 2024-09-13T14:32:16.641 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 678 (ObjNum 997 snap 344 seq_num 997) dirty exists 2024-09-13T14:32:16.642 INFO:tasks.rados.rados.0.smithi151.stdout:2655: left oid 20 (ObjNum 997 snap 344 seq_num 997) 2024-09-13T14:32:16.642 INFO:tasks.rados.rados.0.smithi151.stdout:2657: finishing rollback tid 1 to smithi15186934-13 2024-09-13T14:32:16.642 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 13 v 0 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-13T14:32:16.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:16 smithi151 ceph-mon[104450]: pgmap v735: 89 pgs: 89 active+clean; 170 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 276 KiB/s wr, 4 op/s 2024-09-13T14:32:16.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:16 smithi151 ceph-mon[104450]: osdmap e745: 8 total, 8 up, 8 in 2024-09-13T14:32:16.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:16.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:16 smithi151 ceph-mon[108966]: pgmap v735: 89 pgs: 89 active+clean; 170 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 276 KiB/s wr, 4 op/s 2024-09-13T14:32:16.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:16 smithi151 ceph-mon[108966]: osdmap e745: 8 total, 8 up, 8 in 2024-09-13T14:32:16.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:17.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:16 smithi184 ceph-mon[92527]: pgmap v735: 89 pgs: 89 active+clean; 170 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 276 KiB/s wr, 4 op/s 2024-09-13T14:32:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:16 smithi184 ceph-mon[92527]: osdmap e745: 8 total, 8 up, 8 in 2024-09-13T14:32:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:17.619 INFO:tasks.rados.rados.0.smithi151.stdout:2655: done (2 left) 2024-09-13T14:32:17.619 INFO:tasks.rados.rados.0.smithi151.stdout:2657: done (1 left) 2024-09-13T14:32:17.619 INFO:tasks.rados.rados.0.smithi151.stdout:2658: done (0 left) 2024-09-13T14:32:17.619 INFO:tasks.rados.rados.0.smithi151.stdout:2659: read oid 46 snap -1 2024-09-13T14:32:17.619 INFO:tasks.rados.rados.0.smithi151.stdout:2659: expect deleted 2024-09-13T14:32:17.619 INFO:tasks.rados.rados.0.smithi151.stdout:2660: delete oid 17 current snap is 345 2024-09-13T14:32:17.623 INFO:tasks.rados.rados.0.smithi151.stdout:2659: done (1 left) 2024-09-13T14:32:17.623 INFO:tasks.rados.rados.0.smithi151.stdout:2660: done (0 left) 2024-09-13T14:32:17.623 INFO:tasks.rados.rados.0.smithi151.stdout:2661: write oid 38 current snap is 345 2024-09-13T14:32:17.623 INFO:tasks.rados.rados.0.smithi151.stdout:2661: seq_num 1000 ranges {407000=491243,1617758=547674,2595228=211772} 2024-09-13T14:32:17.628 INFO:tasks.rados.rados.0.smithi151.stdout:2661: writing smithi15186934-38 from 407000 to 898243 tid 1 2024-09-13T14:32:17.635 INFO:tasks.rados.rados.0.smithi151.stdout:2661: writing smithi15186934-38 from 1617758 to 2165432 tid 2 2024-09-13T14:32:17.639 INFO:tasks.rados.rados.0.smithi151.stdout:2661: writing smithi15186934-38 from 2595228 to 2807000 tid 3 2024-09-13T14:32:17.639 INFO:tasks.rados.rados.0.smithi151.stdout:2662: write oid 16 current snap is 345 2024-09-13T14:32:17.639 INFO:tasks.rados.rados.0.smithi151.stdout:2661: finishing write tid 1 to smithi15186934-38 2024-09-13T14:32:17.639 INFO:tasks.rados.rados.0.smithi151.stdout:2662: seq_num 1001 ranges {423807=566492,1676240=407685,2739004=84803} 2024-09-13T14:32:17.644 INFO:tasks.rados.rados.0.smithi151.stdout:2662: writing smithi15186934-16 from 423807 to 990299 tid 1 2024-09-13T14:32:17.650 INFO:tasks.rados.rados.0.smithi151.stdout:2662: writing smithi15186934-16 from 1676240 to 2083925 tid 2 2024-09-13T14:32:17.654 INFO:tasks.rados.rados.0.smithi151.stdout:2662: writing smithi15186934-16 from 2739004 to 2823807 tid 3 2024-09-13T14:32:17.654 INFO:tasks.rados.rados.0.smithi151.stdout:2663: rollback oid 4 current snap is 345 2024-09-13T14:32:17.655 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 4 to 329 2024-09-13T14:32:17.655 INFO:tasks.rados.rados.0.smithi151.stdout:2661: finishing write tid 2 to smithi15186934-38 2024-09-13T14:32:17.655 INFO:tasks.rados.rados.0.smithi151.stdout:2661: finishing write tid 3 to smithi15186934-38 2024-09-13T14:32:17.655 INFO:tasks.rados.rados.0.smithi151.stdout:2661: finishing write tid 4 to smithi15186934-38 2024-09-13T14:32:17.655 INFO:tasks.rados.rados.0.smithi151.stdout:2661: finishing write tid 5 to smithi15186934-38 2024-09-13T14:32:17.655 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 1075 (ObjNum 1000 snap 345 seq_num 1000) dirty exists 2024-09-13T14:32:17.655 INFO:tasks.rados.rados.0.smithi151.stdout:2661: left oid 38 (ObjNum 1000 snap 345 seq_num 1000) 2024-09-13T14:32:17.655 INFO:tasks.rados.rados.0.smithi151.stdout:2662: finishing write tid 1 to smithi15186934-16 2024-09-13T14:32:17.655 INFO:tasks.rados.rados.0.smithi151.stdout:2662: finishing write tid 2 to smithi15186934-16 2024-09-13T14:32:17.655 INFO:tasks.rados.rados.0.smithi151.stdout:2661: done (2 left) 2024-09-13T14:32:17.655 INFO:tasks.rados.rados.0.smithi151.stdout:2664: read oid 27 snap 341 2024-09-13T14:32:17.655 INFO:tasks.rados.rados.0.smithi151.stdout:2664: expect deleted 2024-09-13T14:32:17.655 INFO:tasks.rados.rados.0.smithi151.stdout:2665: delete oid 1 current snap is 345 2024-09-13T14:32:17.657 INFO:tasks.rados.rados.0.smithi151.stdout:2664: done (3 left) 2024-09-13T14:32:17.657 INFO:tasks.rados.rados.0.smithi151.stdout:2665: done (2 left) 2024-09-13T14:32:17.657 INFO:tasks.rados.rados.0.smithi151.stdout:2666: read oid 50 snap -1 2024-09-13T14:32:17.657 INFO:tasks.rados.rados.0.smithi151.stdout:2666: expect (ObjNum 905 snap 314 seq_num 905) 2024-09-13T14:32:17.657 INFO:tasks.rados.rados.0.smithi151.stdout:2667: read oid 2 snap -1 2024-09-13T14:32:17.657 INFO:tasks.rados.rados.0.smithi151.stdout:2667: expect deleted 2024-09-13T14:32:17.657 INFO:tasks.rados.rados.0.smithi151.stdout:2668: write oid 38 current snap is 345 2024-09-13T14:32:17.657 INFO:tasks.rados.rados.0.smithi151.stdout:2662: finishing write tid 3 to smithi15186934-16 2024-09-13T14:32:17.658 INFO:tasks.rados.rados.0.smithi151.stdout:2668: seq_num 1003 ranges {457421=716990,1676851=411354,2677497=179924} 2024-09-13T14:32:17.663 INFO:tasks.rados.rados.0.smithi151.stdout:2668: writing smithi15186934-38 from 457421 to 1174411 tid 1 2024-09-13T14:32:17.668 INFO:tasks.rados.rados.0.smithi151.stdout:2668: writing smithi15186934-38 from 1676851 to 2088205 tid 2 2024-09-13T14:32:17.673 INFO:tasks.rados.rados.0.smithi151.stdout:2668: writing smithi15186934-38 from 2677497 to 2857421 tid 3 2024-09-13T14:32:17.673 INFO:tasks.rados.rados.0.smithi151.stdout:2669: write oid 37 current snap is 345 2024-09-13T14:32:17.673 INFO:tasks.rados.rados.0.smithi151.stdout:2663: finishing rollback tid 0 to smithi15186934-4 2024-09-13T14:32:17.673 INFO:tasks.rados.rados.0.smithi151.stdout:2669: seq_num 1004 ranges {474228=508592,1451686=555012,2704920=169308} 2024-09-13T14:32:17.678 INFO:tasks.rados.rados.0.smithi151.stdout:2669: writing smithi15186934-37 from 474228 to 982820 tid 1 2024-09-13T14:32:17.684 INFO:tasks.rados.rados.0.smithi151.stdout:2669: writing smithi15186934-37 from 1451686 to 2006698 tid 2 2024-09-13T14:32:17.689 INFO:tasks.rados.rados.0.smithi151.stdout:2669: writing smithi15186934-37 from 2704920 to 2874228 tid 3 2024-09-13T14:32:17.689 INFO:tasks.rados.rados.0.smithi151.stdout:2670: write oid 35 current snap is 345 2024-09-13T14:32:17.689 INFO:tasks.rados.rados.0.smithi151.stdout:2662: finishing write tid 4 to smithi15186934-16 2024-09-13T14:32:17.689 INFO:tasks.rados.rados.0.smithi151.stdout:2662: finishing write tid 5 to smithi15186934-16 2024-09-13T14:32:17.690 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 16 v 683 (ObjNum 1001 snap 345 seq_num 1001) dirty exists 2024-09-13T14:32:17.690 INFO:tasks.rados.rados.0.smithi151.stdout:2662: left oid 16 (ObjNum 1001 snap 345 seq_num 1001) 2024-09-13T14:32:17.690 INFO:tasks.rados.rados.0.smithi151.stdout:2663: finishing rollback tid 1 to smithi15186934-4 2024-09-13T14:32:17.690 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 1338 (ObjNum 129 snap 0 seq_num 3092983624) dirty dne 2024-09-13T14:32:17.690 INFO:tasks.rados.rados.0.smithi151.stdout:2666: expect (ObjNum 905 snap 314 seq_num 905) 2024-09-13T14:32:17.874 INFO:tasks.rados.rados.0.smithi151.stdout:2668: finishing write tid 1 to smithi15186934-38 2024-09-13T14:32:17.874 INFO:tasks.rados.rados.0.smithi151.stdout:2668: finishing write tid 2 to smithi15186934-38 2024-09-13T14:32:17.874 INFO:tasks.rados.rados.0.smithi151.stdout:2668: finishing write tid 3 to smithi15186934-38 2024-09-13T14:32:17.874 INFO:tasks.rados.rados.0.smithi151.stdout:2668: finishing write tid 4 to smithi15186934-38 2024-09-13T14:32:17.874 INFO:tasks.rados.rados.0.smithi151.stdout:2668: finishing write tid 5 to smithi15186934-38 2024-09-13T14:32:17.874 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 1079 (ObjNum 1003 snap 345 seq_num 1003) dirty exists 2024-09-13T14:32:17.874 INFO:tasks.rados.rados.0.smithi151.stdout:2668: left oid 38 (ObjNum 1003 snap 345 seq_num 1003) 2024-09-13T14:32:17.874 INFO:tasks.rados.rados.0.smithi151.stdout:2669: finishing write tid 1 to smithi15186934-37 2024-09-13T14:32:17.874 INFO:tasks.rados.rados.0.smithi151.stdout:2669: finishing write tid 2 to smithi15186934-37 2024-09-13T14:32:17.874 INFO:tasks.rados.rados.0.smithi151.stdout:2669: finishing write tid 3 to smithi15186934-37 2024-09-13T14:32:17.875 INFO:tasks.rados.rados.0.smithi151.stdout:2669: finishing write tid 4 to smithi15186934-37 2024-09-13T14:32:17.875 INFO:tasks.rados.rados.0.smithi151.stdout:2669: finishing write tid 5 to smithi15186934-37 2024-09-13T14:32:17.875 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 37 v 1180 (ObjNum 1004 snap 345 seq_num 1004) dirty exists 2024-09-13T14:32:17.875 INFO:tasks.rados.rados.0.smithi151.stdout:2669: left oid 37 (ObjNum 1004 snap 345 seq_num 1004) 2024-09-13T14:32:17.875 INFO:tasks.rados.rados.0.smithi151.stdout:2670: seq_num 1005 ranges {491035=583841,1510168=698670,2732343=158692} 2024-09-13T14:32:17.879 INFO:tasks.rados.rados.0.smithi151.stdout:2670: writing smithi15186934-35 from 491035 to 1074876 tid 1 2024-09-13T14:32:17.885 INFO:tasks.rados.rados.0.smithi151.stdout:2670: writing smithi15186934-35 from 1510168 to 2208838 tid 2 2024-09-13T14:32:17.889 INFO:tasks.rados.rados.0.smithi151.stdout:2670: writing smithi15186934-35 from 2732343 to 2891035 tid 3 2024-09-13T14:32:17.889 INFO:tasks.rados.rados.0.smithi151.stdout:2662: done (6 left) 2024-09-13T14:32:17.889 INFO:tasks.rados.rados.0.smithi151.stdout:2663: done (5 left) 2024-09-13T14:32:17.889 INFO:tasks.rados.rados.0.smithi151.stdout:2666: done (4 left) 2024-09-13T14:32:17.890 INFO:tasks.rados.rados.0.smithi151.stdout:2667: done (3 left) 2024-09-13T14:32:17.890 INFO:tasks.rados.rados.0.smithi151.stdout:2668: done (2 left) 2024-09-13T14:32:17.890 INFO:tasks.rados.rados.0.smithi151.stdout:2669: done (1 left) 2024-09-13T14:32:17.890 INFO:tasks.rados.rados.0.smithi151.stdout:2671: snap_remove snap 332 2024-09-13T14:32:18.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:17 smithi184 ceph-mon[92527]: osdmap e746: 8 total, 8 up, 8 in 2024-09-13T14:32:18.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:17 smithi151 ceph-mon[104450]: osdmap e746: 8 total, 8 up, 8 in 2024-09-13T14:32:18.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:17 smithi151 ceph-mon[108966]: osdmap e746: 8 total, 8 up, 8 in 2024-09-13T14:32:18.623 INFO:tasks.rados.rados.0.smithi151.stdout:2671: done (1 left) 2024-09-13T14:32:18.623 INFO:tasks.rados.rados.0.smithi151.stdout:2672: rollback oid 41 current snap is 345 2024-09-13T14:32:18.623 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 41 to 331 2024-09-13T14:32:18.623 INFO:tasks.rados.rados.0.smithi151.stdout:2670: finishing write tid 1 to smithi15186934-35 2024-09-13T14:32:18.623 INFO:tasks.rados.rados.0.smithi151.stdout:2670: finishing write tid 2 to smithi15186934-35 2024-09-13T14:32:18.623 INFO:tasks.rados.rados.0.smithi151.stdout:2670: finishing write tid 3 to smithi15186934-35 2024-09-13T14:32:18.623 INFO:tasks.rados.rados.0.smithi151.stdout:2670: finishing write tid 4 to smithi15186934-35 2024-09-13T14:32:18.623 INFO:tasks.rados.rados.0.smithi151.stdout:2670: finishing write tid 5 to smithi15186934-35 2024-09-13T14:32:18.623 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 35 v 1587 (ObjNum 1005 snap 345 seq_num 1005) dirty exists 2024-09-13T14:32:18.623 INFO:tasks.rados.rados.0.smithi151.stdout:2670: left oid 35 (ObjNum 1005 snap 345 seq_num 1005) 2024-09-13T14:32:18.624 INFO:tasks.rados.rados.0.smithi151.stdout:2670: done (1 left) 2024-09-13T14:32:18.624 INFO:tasks.rados.rados.0.smithi151.stdout:2673: write oid 35 current snap is 345 2024-09-13T14:32:18.624 INFO:tasks.rados.rados.0.smithi151.stdout:2673: seq_num 1006 ranges {507842=659090,1568650=558681,2759766=148076} 2024-09-13T14:32:18.630 INFO:tasks.rados.rados.0.smithi151.stdout:2673: writing smithi15186934-35 from 507842 to 1166932 tid 1 2024-09-13T14:32:18.636 INFO:tasks.rados.rados.0.smithi151.stdout:2673: writing smithi15186934-35 from 1568650 to 2127331 tid 2 2024-09-13T14:32:18.641 INFO:tasks.rados.rados.0.smithi151.stdout:2673: writing smithi15186934-35 from 2759766 to 2907842 tid 3 2024-09-13T14:32:18.641 INFO:tasks.rados.rados.0.smithi151.stdout:2674: snap_remove snap 344 2024-09-13T14:32:19.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:18 smithi184 ceph-mon[92527]: pgmap v738: 89 pgs: 89 active+clean; 127 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 1.8 MiB/s wr, 18 op/s 2024-09-13T14:32:19.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:18 smithi184 ceph-mon[92527]: osdmap e747: 8 total, 8 up, 8 in 2024-09-13T14:32:19.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:18 smithi151 ceph-mon[104450]: pgmap v738: 89 pgs: 89 active+clean; 127 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 1.8 MiB/s wr, 18 op/s 2024-09-13T14:32:19.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:18 smithi151 ceph-mon[104450]: osdmap e747: 8 total, 8 up, 8 in 2024-09-13T14:32:19.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:18 smithi151 ceph-mon[108966]: pgmap v738: 89 pgs: 89 active+clean; 127 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 1.8 MiB/s wr, 18 op/s 2024-09-13T14:32:19.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:18 smithi151 ceph-mon[108966]: osdmap e747: 8 total, 8 up, 8 in 2024-09-13T14:32:19.575 INFO:tasks.rados.rados.0.smithi151.stdout:2674: done (2 left) 2024-09-13T14:32:19.575 INFO:tasks.rados.rados.0.smithi151.stdout:2675: read oid 19 snap -1 2024-09-13T14:32:19.576 INFO:tasks.rados.rados.0.smithi151.stdout:2675: expect deleted 2024-09-13T14:32:19.576 INFO:tasks.rados.rados.0.smithi151.stdout:2676: write oid 25 current snap is 345 2024-09-13T14:32:19.576 INFO:tasks.rados.rados.0.smithi151.stdout:2672: finishing rollback tid 0 to smithi15186934-41 2024-09-13T14:32:19.576 INFO:tasks.rados.rados.0.smithi151.stdout:2672: finishing rollback tid 1 to smithi15186934-41 2024-09-13T14:32:19.576 INFO:tasks.rados.rados.0.smithi151.stdout:2672: finishing rollback tid 2 to smithi15186934-41 2024-09-13T14:32:19.576 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 201 (ObjNum 915 snap 317 seq_num 915) dirty exists 2024-09-13T14:32:19.576 INFO:tasks.rados.rados.0.smithi151.stdout:2673: finishing write tid 1 to smithi15186934-35 2024-09-13T14:32:19.576 INFO:tasks.rados.rados.0.smithi151.stdout:2673: finishing write tid 2 to smithi15186934-35 2024-09-13T14:32:19.576 INFO:tasks.rados.rados.0.smithi151.stdout:2673: finishing write tid 3 to smithi15186934-35 2024-09-13T14:32:19.576 INFO:tasks.rados.rados.0.smithi151.stdout:2673: finishing write tid 4 to smithi15186934-35 2024-09-13T14:32:19.576 INFO:tasks.rados.rados.0.smithi151.stdout:2673: finishing write tid 5 to smithi15186934-35 2024-09-13T14:32:19.576 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 35 v 1597 (ObjNum 1006 snap 345 seq_num 1006) dirty exists 2024-09-13T14:32:19.577 INFO:tasks.rados.rados.0.smithi151.stdout:2673: left oid 35 (ObjNum 1006 snap 345 seq_num 1006) 2024-09-13T14:32:19.577 INFO:tasks.rados.rados.0.smithi151.stdout:2676: seq_num 1007 ranges {524649=734339,1910779=702339} 2024-09-13T14:32:19.583 INFO:tasks.rados.rados.0.smithi151.stdout:2676: writing smithi15186934-25 from 524649 to 1258988 tid 1 2024-09-13T14:32:19.591 INFO:tasks.rados.rados.0.smithi151.stdout:2676: writing smithi15186934-25 from 1910779 to 2613118 tid 2 2024-09-13T14:32:19.591 INFO:tasks.rados.rados.0.smithi151.stdout:2672: done (3 left) 2024-09-13T14:32:19.591 INFO:tasks.rados.rados.0.smithi151.stdout:2673: done (2 left) 2024-09-13T14:32:19.591 INFO:tasks.rados.rados.0.smithi151.stdout:2677: read oid 45 snap 331 2024-09-13T14:32:19.591 INFO:tasks.rados.rados.0.smithi151.stdout:2677: expect (ObjNum 965 snap 330 seq_num 965) 2024-09-13T14:32:19.591 INFO:tasks.rados.rados.0.smithi151.stdout:2678: write oid 13 current snap is 345 2024-09-13T14:32:19.591 INFO:tasks.rados.rados.0.smithi151.stdout:2678: seq_num 1008 ranges {541456=409588,1569261=562350,2814612=126844} 2024-09-13T14:32:19.596 INFO:tasks.rados.rados.0.smithi151.stdout:2678: writing smithi15186934-13 from 541456 to 951044 tid 1 2024-09-13T14:32:19.602 INFO:tasks.rados.rados.0.smithi151.stdout:2678: writing smithi15186934-13 from 1569261 to 2131611 tid 2 2024-09-13T14:32:19.607 INFO:tasks.rados.rados.0.smithi151.stdout:2678: writing smithi15186934-13 from 2814612 to 2941456 tid 3 2024-09-13T14:32:19.607 INFO:tasks.rados.rados.0.smithi151.stdout:2679: snap_create 2024-09-13T14:32:19.607 INFO:tasks.rados.rados.0.smithi151.stdout:2676: finishing write tid 1 to smithi15186934-25 2024-09-13T14:32:19.607 INFO:tasks.rados.rados.0.smithi151.stdout:2676: finishing write tid 2 to smithi15186934-25 2024-09-13T14:32:19.607 INFO:tasks.rados.rados.0.smithi151.stdout:2676: finishing write tid 3 to smithi15186934-25 2024-09-13T14:32:19.607 INFO:tasks.rados.rados.0.smithi151.stdout:2676: finishing write tid 4 to smithi15186934-25 2024-09-13T14:32:19.607 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 1348 (ObjNum 1007 snap 345 seq_num 1007) dirty exists 2024-09-13T14:32:19.607 INFO:tasks.rados.rados.0.smithi151.stdout:2676: left oid 25 (ObjNum 1007 snap 345 seq_num 1007) 2024-09-13T14:32:19.608 INFO:tasks.rados.rados.0.smithi151.stdout:2677: expect (ObjNum 965 snap 330 seq_num 965) 2024-09-13T14:32:19.711 INFO:tasks.rados.rados.0.smithi151.stdout:2678: finishing write tid 1 to smithi15186934-13 2024-09-13T14:32:19.711 INFO:tasks.rados.rados.0.smithi151.stdout:2678: finishing write tid 2 to smithi15186934-13 2024-09-13T14:32:19.711 INFO:tasks.rados.rados.0.smithi151.stdout:2678: finishing write tid 3 to smithi15186934-13 2024-09-13T14:32:19.711 INFO:tasks.rados.rados.0.smithi151.stdout:2678: finishing write tid 4 to smithi15186934-13 2024-09-13T14:32:19.711 INFO:tasks.rados.rados.0.smithi151.stdout:2678: finishing write tid 5 to smithi15186934-13 2024-09-13T14:32:19.711 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 13 v 1022 (ObjNum 1008 snap 345 seq_num 1008) dirty exists 2024-09-13T14:32:19.712 INFO:tasks.rados.rados.0.smithi151.stdout:2678: left oid 13 (ObjNum 1008 snap 345 seq_num 1008) 2024-09-13T14:32:20.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:19 smithi184 ceph-mon[92527]: osdmap e748: 8 total, 8 up, 8 in 2024-09-13T14:32:20.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:19 smithi184 ceph-mon[92527]: osdmap e749: 8 total, 8 up, 8 in 2024-09-13T14:32:20.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:19 smithi151 ceph-mon[104450]: osdmap e748: 8 total, 8 up, 8 in 2024-09-13T14:32:20.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:19 smithi151 ceph-mon[104450]: osdmap e749: 8 total, 8 up, 8 in 2024-09-13T14:32:20.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:19 smithi151 ceph-mon[108966]: osdmap e748: 8 total, 8 up, 8 in 2024-09-13T14:32:20.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:19 smithi151 ceph-mon[108966]: osdmap e749: 8 total, 8 up, 8 in 2024-09-13T14:32:20.574 INFO:tasks.rados.rados.0.smithi151.stdout:2675: done (4 left) 2024-09-13T14:32:20.574 INFO:tasks.rados.rados.0.smithi151.stdout:2676: done (3 left) 2024-09-13T14:32:20.575 INFO:tasks.rados.rados.0.smithi151.stdout:2677: done (2 left) 2024-09-13T14:32:20.575 INFO:tasks.rados.rados.0.smithi151.stdout:2678: done (1 left) 2024-09-13T14:32:20.575 INFO:tasks.rados.rados.0.smithi151.stdout:2679: done (0 left) 2024-09-13T14:32:20.575 INFO:tasks.rados.rados.0.smithi151.stdout:2680: delete oid 4 current snap is 346 2024-09-13T14:32:20.577 INFO:tasks.rados.rados.0.smithi151.stdout:2680: done (0 left) 2024-09-13T14:32:20.577 INFO:tasks.rados.rados.0.smithi151.stdout:2681: rollback oid 2 current snap is 346 2024-09-13T14:32:20.577 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 2 to 345 2024-09-13T14:32:20.577 INFO:tasks.rados.rados.0.smithi151.stdout:2682: snap_create 2024-09-13T14:32:20.577 INFO:tasks.rados.rados.0.smithi151.stdout:2681: finishing rollback tid 1 to smithi15186934-2 2024-09-13T14:32:20.578 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 2 v 0 (ObjNum 193 snap 0 seq_num 1755291024) dirty dne 2024-09-13T14:32:20.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:32:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:32:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:32:21.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:20 smithi184 ceph-mon[92527]: pgmap v741: 89 pgs: 89 active+clean; 121 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.3 MiB/s wr, 20 op/s 2024-09-13T14:32:21.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:20 smithi184 ceph-mon[92527]: osdmap e750: 8 total, 8 up, 8 in 2024-09-13T14:32:21.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:20 smithi151 ceph-mon[104450]: pgmap v741: 89 pgs: 89 active+clean; 121 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.3 MiB/s wr, 20 op/s 2024-09-13T14:32:21.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:20 smithi151 ceph-mon[104450]: osdmap e750: 8 total, 8 up, 8 in 2024-09-13T14:32:21.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:20 smithi151 ceph-mon[108966]: pgmap v741: 89 pgs: 89 active+clean; 121 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.3 MiB/s wr, 20 op/s 2024-09-13T14:32:21.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:20 smithi151 ceph-mon[108966]: osdmap e750: 8 total, 8 up, 8 in 2024-09-13T14:32:21.576 INFO:tasks.rados.rados.0.smithi151.stdout:2681: done (1 left) 2024-09-13T14:32:21.577 INFO:tasks.rados.rados.0.smithi151.stdout:2682: done (0 left) 2024-09-13T14:32:21.577 INFO:tasks.rados.rados.0.smithi151.stdout:2683: read oid 35 snap -1 2024-09-13T14:32:21.577 INFO:tasks.rados.rados.0.smithi151.stdout:2683: expect (ObjNum 1006 snap 345 seq_num 1006) 2024-09-13T14:32:21.577 INFO:tasks.rados.rados.0.smithi151.stdout:2684: write oid 26 current snap is 347 2024-09-13T14:32:21.577 INFO:tasks.rados.rados.0.smithi151.stdout:2684: seq_num 1010 ranges {575070=560086,1686225=566019,2869458=105612} 2024-09-13T14:32:21.582 INFO:tasks.rados.rados.0.smithi151.stdout:2684: writing smithi15186934-26 from 575070 to 1135156 tid 1 2024-09-13T14:32:21.589 INFO:tasks.rados.rados.0.smithi151.stdout:2684: writing smithi15186934-26 from 1686225 to 2252244 tid 2 2024-09-13T14:32:21.593 INFO:tasks.rados.rados.0.smithi151.stdout:2684: writing smithi15186934-26 from 2869458 to 2975070 tid 3 2024-09-13T14:32:21.593 INFO:tasks.rados.rados.0.smithi151.stdout:2685: write oid 41 current snap is 347 2024-09-13T14:32:21.593 INFO:tasks.rados.rados.0.smithi151.stdout:2685: seq_num 1011 ranges {591877=635335,1628354=709677} 2024-09-13T14:32:21.600 INFO:tasks.rados.rados.0.smithi151.stdout:2685: writing smithi15186934-41 from 591877 to 1227212 tid 1 2024-09-13T14:32:21.606 INFO:tasks.rados.rados.0.smithi151.stdout:2685: writing smithi15186934-41 from 1628354 to 2338031 tid 2 2024-09-13T14:32:21.607 INFO:tasks.rados.rados.0.smithi151.stdout:2686: read oid 32 snap 342 2024-09-13T14:32:21.607 INFO:tasks.rados.rados.0.smithi151.stdout:2686: expect deleted 2024-09-13T14:32:21.607 INFO:tasks.rados.rados.0.smithi151.stdout:2687: write oid 19 current snap is 347 2024-09-13T14:32:21.607 INFO:tasks.rados.rados.0.smithi151.stdout:2687: seq_num 1012 ranges {608684=426937,1803189=569688,2924304=84380} 2024-09-13T14:32:21.612 INFO:tasks.rados.rados.0.smithi151.stdout:2687: writing smithi15186934-19 from 608684 to 1035621 tid 1 2024-09-13T14:32:21.620 INFO:tasks.rados.rados.0.smithi151.stdout:2687: writing smithi15186934-19 from 1803189 to 2372877 tid 2 2024-09-13T14:32:21.623 INFO:tasks.rados.rados.0.smithi151.stdout:2687: writing smithi15186934-19 from 2924304 to 3008684 tid 3 2024-09-13T14:32:21.623 INFO:tasks.rados.rados.0.smithi151.stdout:2688: read oid 39 snap -1 2024-09-13T14:32:21.624 INFO:tasks.rados.rados.0.smithi151.stdout:2688: expect (ObjNum 978 snap 339 seq_num 978) 2024-09-13T14:32:21.624 INFO:tasks.rados.rados.0.smithi151.stdout:2689: rollback oid 7 current snap is 347 2024-09-13T14:32:21.624 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 7 to 327 2024-09-13T14:32:21.624 INFO:tasks.rados.rados.0.smithi151.stdout:2683: expect (ObjNum 1006 snap 345 seq_num 1006) 2024-09-13T14:32:21.831 INFO:tasks.rados.rados.0.smithi151.stdout:2684: finishing write tid 1 to smithi15186934-26 2024-09-13T14:32:21.831 INFO:tasks.rados.rados.0.smithi151.stdout:2684: finishing write tid 2 to smithi15186934-26 2024-09-13T14:32:21.831 INFO:tasks.rados.rados.0.smithi151.stdout:2684: finishing write tid 3 to smithi15186934-26 2024-09-13T14:32:21.831 INFO:tasks.rados.rados.0.smithi151.stdout:2684: finishing write tid 4 to smithi15186934-26 2024-09-13T14:32:21.831 INFO:tasks.rados.rados.0.smithi151.stdout:2684: finishing write tid 5 to smithi15186934-26 2024-09-13T14:32:21.831 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 960 (ObjNum 1010 snap 347 seq_num 1010) dirty exists 2024-09-13T14:32:21.831 INFO:tasks.rados.rados.0.smithi151.stdout:2684: left oid 26 (ObjNum 1010 snap 347 seq_num 1010) 2024-09-13T14:32:21.831 INFO:tasks.rados.rados.0.smithi151.stdout:2685: finishing write tid 1 to smithi15186934-41 2024-09-13T14:32:21.832 INFO:tasks.rados.rados.0.smithi151.stdout:2683: done (6 left) 2024-09-13T14:32:21.832 INFO:tasks.rados.rados.0.smithi151.stdout:2684: done (5 left) 2024-09-13T14:32:21.832 INFO:tasks.rados.rados.0.smithi151.stdout:2690: read oid 47 snap -1 2024-09-13T14:32:21.832 INFO:tasks.rados.rados.0.smithi151.stdout:2690: expect deleted 2024-09-13T14:32:21.832 INFO:tasks.rados.rados.0.smithi151.stdout:2691: read oid 38 snap 327 2024-09-13T14:32:21.832 INFO:tasks.rados.rados.0.smithi151.stdout:2691: expect (ObjNum 942 snap 325 seq_num 942) 2024-09-13T14:32:21.832 INFO:tasks.rados.rados.0.smithi151.stdout:2692: read oid 20 snap -1 2024-09-13T14:32:21.832 INFO:tasks.rados.rados.0.smithi151.stdout:2692: expect (ObjNum 997 snap 344 seq_num 997) 2024-09-13T14:32:21.832 INFO:tasks.rados.rados.0.smithi151.stdout:2693: read oid 43 snap 342 2024-09-13T14:32:21.832 INFO:tasks.rados.rados.0.smithi151.stdout:2693: expect (ObjNum 986 snap 341 seq_num 986) 2024-09-13T14:32:21.833 INFO:tasks.rados.rados.0.smithi151.stdout:2694: write oid 44 current snap is 347 2024-09-13T14:32:21.833 INFO:tasks.rados.rados.0.smithi151.stdout:2685: finishing write tid 2 to smithi15186934-41 2024-09-13T14:32:21.833 INFO:tasks.rados.rados.0.smithi151.stdout:2685: finishing write tid 3 to smithi15186934-41 2024-09-13T14:32:21.833 INFO:tasks.rados.rados.0.smithi151.stdout:2685: finishing write tid 4 to smithi15186934-41 2024-09-13T14:32:21.833 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 207 (ObjNum 1011 snap 347 seq_num 1011) dirty exists 2024-09-13T14:32:21.833 INFO:tasks.rados.rados.0.smithi151.stdout:2685: left oid 41 (ObjNum 1011 snap 347 seq_num 1011) 2024-09-13T14:32:21.833 INFO:tasks.rados.rados.0.smithi151.stdout:2687: finishing write tid 1 to smithi15186934-19 2024-09-13T14:32:21.833 INFO:tasks.rados.rados.0.smithi151.stdout:2687: finishing write tid 2 to smithi15186934-19 2024-09-13T14:32:21.833 INFO:tasks.rados.rados.0.smithi151.stdout:2687: finishing write tid 3 to smithi15186934-19 2024-09-13T14:32:21.833 INFO:tasks.rados.rados.0.smithi151.stdout:2694: seq_num 1013 ranges {625491=502186,1861671=713346} 2024-09-13T14:32:21.836 INFO:tasks.rados.rados.0.smithi151.stdout:2694: writing smithi15186934-44 from 625491 to 1127677 tid 1 2024-09-13T14:32:21.845 INFO:tasks.rados.rados.0.smithi151.stdout:2694: writing smithi15186934-44 from 1861671 to 2575017 tid 2 2024-09-13T14:32:21.845 INFO:tasks.rados.rados.0.smithi151.stdout:2685: done (9 left) 2024-09-13T14:32:21.845 INFO:tasks.rados.rados.0.smithi151.stdout:2686: done (8 left) 2024-09-13T14:32:21.845 INFO:tasks.rados.rados.0.smithi151.stdout:2695: snap_remove snap 338 2024-09-13T14:32:22.633 INFO:tasks.rados.rados.0.smithi151.stdout:2695: done (8 left) 2024-09-13T14:32:22.633 INFO:tasks.rados.rados.0.smithi151.stdout:2696: read oid 18 snap -1 2024-09-13T14:32:22.633 INFO:tasks.rados.rados.0.smithi151.stdout:2696: expect deleted 2024-09-13T14:32:22.633 INFO:tasks.rados.rados.0.smithi151.stdout:2697: read oid 42 snap -1 2024-09-13T14:32:22.633 INFO:tasks.rados.rados.0.smithi151.stdout:2697: expect deleted 2024-09-13T14:32:22.633 INFO:tasks.rados.rados.0.smithi151.stdout:2698: snap_create 2024-09-13T14:32:22.633 INFO:tasks.rados.rados.0.smithi151.stdout:2687: finishing write tid 4 to smithi15186934-19 2024-09-13T14:32:22.633 INFO:tasks.rados.rados.0.smithi151.stdout:2687: finishing write tid 5 to smithi15186934-19 2024-09-13T14:32:22.634 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 19 v 700 (ObjNum 1012 snap 347 seq_num 1012) dirty exists 2024-09-13T14:32:22.634 INFO:tasks.rados.rados.0.smithi151.stdout:2687: left oid 19 (ObjNum 1012 snap 347 seq_num 1012) 2024-09-13T14:32:22.634 INFO:tasks.rados.rados.0.smithi151.stdout:2689: finishing rollback tid 0 to smithi15186934-7 2024-09-13T14:32:22.634 INFO:tasks.rados.rados.0.smithi151.stdout:2689: finishing rollback tid 1 to smithi15186934-7 2024-09-13T14:32:22.634 INFO:tasks.rados.rados.0.smithi151.stdout:2689: finishing rollback tid 2 to smithi15186934-7 2024-09-13T14:32:22.634 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 1605 (ObjNum 951 snap 327 seq_num 951) dirty exists 2024-09-13T14:32:22.634 INFO:tasks.rados.rados.0.smithi151.stdout:2688: expect (ObjNum 978 snap 339 seq_num 978) 2024-09-13T14:32:22.799 INFO:tasks.rados.rados.0.smithi151.stdout:2694: finishing write tid 1 to smithi15186934-44 2024-09-13T14:32:22.799 INFO:tasks.rados.rados.0.smithi151.stdout:2694: finishing write tid 2 to smithi15186934-44 2024-09-13T14:32:22.799 INFO:tasks.rados.rados.0.smithi151.stdout:2694: finishing write tid 3 to smithi15186934-44 2024-09-13T14:32:22.799 INFO:tasks.rados.rados.0.smithi151.stdout:2694: finishing write tid 4 to smithi15186934-44 2024-09-13T14:32:22.799 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 814 (ObjNum 1013 snap 347 seq_num 1013) dirty exists 2024-09-13T14:32:22.799 INFO:tasks.rados.rados.0.smithi151.stdout:2694: left oid 44 (ObjNum 1013 snap 347 seq_num 1013) 2024-09-13T14:32:22.799 INFO:tasks.rados.rados.0.smithi151.stdout:2693: expect (ObjNum 986 snap 341 seq_num 986) 2024-09-13T14:32:22.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:22 smithi184 ceph-mon[92527]: pgmap v744: 89 pgs: 89 active+clean; 121 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 566 KiB/s wr, 2 op/s 2024-09-13T14:32:22.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:22 smithi184 ceph-mon[92527]: osdmap e751: 8 total, 8 up, 8 in 2024-09-13T14:32:22.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:22 smithi151 ceph-mon[104450]: pgmap v744: 89 pgs: 89 active+clean; 121 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 566 KiB/s wr, 2 op/s 2024-09-13T14:32:22.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:22 smithi151 ceph-mon[104450]: osdmap e751: 8 total, 8 up, 8 in 2024-09-13T14:32:22.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:22 smithi151 ceph-mon[108966]: pgmap v744: 89 pgs: 89 active+clean; 121 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 566 KiB/s wr, 2 op/s 2024-09-13T14:32:22.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:22 smithi151 ceph-mon[108966]: osdmap e751: 8 total, 8 up, 8 in 2024-09-13T14:32:22.914 INFO:tasks.rados.rados.0.smithi151.stdout:2692: expect (ObjNum 997 snap 344 seq_num 997) 2024-09-13T14:32:23.041 INFO:tasks.rados.rados.0.smithi151.stdout:2691: expect (ObjNum 942 snap 325 seq_num 942) 2024-09-13T14:32:23.641 INFO:tasks.rados.rados.0.smithi151.stdout:2687: done (10 left) 2024-09-13T14:32:23.642 INFO:tasks.rados.rados.0.smithi151.stdout:2688: done (9 left) 2024-09-13T14:32:23.642 INFO:tasks.rados.rados.0.smithi151.stdout:2689: done (8 left) 2024-09-13T14:32:23.642 INFO:tasks.rados.rados.0.smithi151.stdout:2690: done (7 left) 2024-09-13T14:32:23.642 INFO:tasks.rados.rados.0.smithi151.stdout:2691: done (6 left) 2024-09-13T14:32:23.642 INFO:tasks.rados.rados.0.smithi151.stdout:2692: done (5 left) 2024-09-13T14:32:23.642 INFO:tasks.rados.rados.0.smithi151.stdout:2693: done (4 left) 2024-09-13T14:32:23.642 INFO:tasks.rados.rados.0.smithi151.stdout:2694: done (3 left) 2024-09-13T14:32:23.643 INFO:tasks.rados.rados.0.smithi151.stdout:2696: done (2 left) 2024-09-13T14:32:23.643 INFO:tasks.rados.rados.0.smithi151.stdout:2697: done (1 left) 2024-09-13T14:32:23.643 INFO:tasks.rados.rados.0.smithi151.stdout:2698: done (0 left) 2024-09-13T14:32:23.643 INFO:tasks.rados.rados.0.smithi151.stdout:2699: write oid 44 current snap is 348 2024-09-13T14:32:23.643 INFO:tasks.rados.rados.0.smithi151.stdout:2699: seq_num 1014 ranges {642298=577435,1920153=573357,2979150=63148} 2024-09-13T14:32:23.647 INFO:tasks.rados.rados.0.smithi151.stdout:2699: writing smithi15186934-44 from 642298 to 1219733 tid 1 2024-09-13T14:32:23.655 INFO:tasks.rados.rados.0.smithi151.stdout:2699: writing smithi15186934-44 from 1920153 to 2493510 tid 2 2024-09-13T14:32:23.659 INFO:tasks.rados.rados.0.smithi151.stdout:2699: writing smithi15186934-44 from 2979150 to 3042298 tid 3 2024-09-13T14:32:23.659 INFO:tasks.rados.rados.0.smithi151.stdout:2700: write oid 47 current snap is 348 2024-09-13T14:32:23.659 INFO:tasks.rados.rados.0.smithi151.stdout:2699: finishing write tid 1 to smithi15186934-44 2024-09-13T14:32:23.659 INFO:tasks.rados.rados.0.smithi151.stdout:2699: finishing write tid 2 to smithi15186934-44 2024-09-13T14:32:23.659 INFO:tasks.rados.rados.0.smithi151.stdout:2700: seq_num 1015 ranges {659105=652684,1862282=717015} 2024-09-13T14:32:23.665 INFO:tasks.rados.rados.0.smithi151.stdout:2700: writing smithi15186934-47 from 659105 to 1311789 tid 1 2024-09-13T14:32:23.673 INFO:tasks.rados.rados.0.smithi151.stdout:2700: writing smithi15186934-47 from 1862282 to 2579297 tid 2 2024-09-13T14:32:23.673 INFO:tasks.rados.rados.0.smithi151.stdout:2701: read oid 12 snap -1 2024-09-13T14:32:23.673 INFO:tasks.rados.rados.0.smithi151.stdout:2701: expect deleted 2024-09-13T14:32:23.673 INFO:tasks.rados.rados.0.smithi151.stdout:2702: write oid 14 current snap is 348 2024-09-13T14:32:23.673 INFO:tasks.rados.rados.0.smithi151.stdout:2699: finishing write tid 3 to smithi15186934-44 2024-09-13T14:32:23.673 INFO:tasks.rados.rados.0.smithi151.stdout:2699: finishing write tid 4 to smithi15186934-44 2024-09-13T14:32:23.674 INFO:tasks.rados.rados.0.smithi151.stdout:2699: finishing write tid 5 to smithi15186934-44 2024-09-13T14:32:23.674 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 821 (ObjNum 1014 snap 348 seq_num 1014) dirty exists 2024-09-13T14:32:23.674 INFO:tasks.rados.rados.0.smithi151.stdout:2699: left oid 44 (ObjNum 1014 snap 348 seq_num 1014) 2024-09-13T14:32:23.674 INFO:tasks.rados.rados.0.smithi151.stdout:2700: finishing write tid 1 to smithi15186934-47 2024-09-13T14:32:23.674 INFO:tasks.rados.rados.0.smithi151.stdout:2702: seq_num 1016 ranges {675912=727933,1920764=460673,2801290=274622} 2024-09-13T14:32:23.680 INFO:tasks.rados.rados.0.smithi151.stdout:2702: writing smithi15186934-14 from 675912 to 1403845 tid 1 2024-09-13T14:32:23.686 INFO:tasks.rados.rados.0.smithi151.stdout:2702: writing smithi15186934-14 from 1920764 to 2381437 tid 2 2024-09-13T14:32:23.690 INFO:tasks.rados.rados.0.smithi151.stdout:2702: writing smithi15186934-14 from 2801290 to 3075912 tid 3 2024-09-13T14:32:23.690 INFO:tasks.rados.rados.0.smithi151.stdout:2699: done (3 left) 2024-09-13T14:32:23.690 INFO:tasks.rados.rados.0.smithi151.stdout:2703: rollback oid 1 current snap is 348 2024-09-13T14:32:23.690 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 1 to 337 2024-09-13T14:32:23.690 INFO:tasks.rados.rados.0.smithi151.stdout:2700: finishing write tid 2 to smithi15186934-47 2024-09-13T14:32:23.690 INFO:tasks.rados.rados.0.smithi151.stdout:2700: finishing write tid 3 to smithi15186934-47 2024-09-13T14:32:23.691 INFO:tasks.rados.rados.0.smithi151.stdout:2700: finishing write tid 4 to smithi15186934-47 2024-09-13T14:32:23.691 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 1191 (ObjNum 1015 snap 348 seq_num 1015) dirty exists 2024-09-13T14:32:23.691 INFO:tasks.rados.rados.0.smithi151.stdout:2700: left oid 47 (ObjNum 1015 snap 348 seq_num 1015) 2024-09-13T14:32:23.691 INFO:tasks.rados.rados.0.smithi151.stdout:2702: finishing write tid 1 to smithi15186934-14 2024-09-13T14:32:23.691 INFO:tasks.rados.rados.0.smithi151.stdout:2702: finishing write tid 2 to smithi15186934-14 2024-09-13T14:32:23.691 INFO:tasks.rados.rados.0.smithi151.stdout:2700: done (3 left) 2024-09-13T14:32:23.691 INFO:tasks.rados.rados.0.smithi151.stdout:2701: done (2 left) 2024-09-13T14:32:23.691 INFO:tasks.rados.rados.0.smithi151.stdout:2704: snap_remove snap 336 2024-09-13T14:32:24.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:23 smithi184 ceph-mon[92527]: osdmap e752: 8 total, 8 up, 8 in 2024-09-13T14:32:24.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:23 smithi151 ceph-mon[104450]: osdmap e752: 8 total, 8 up, 8 in 2024-09-13T14:32:24.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:23 smithi151 ceph-mon[108966]: osdmap e752: 8 total, 8 up, 8 in 2024-09-13T14:32:24.576 INFO:tasks.rados.rados.0.smithi151.stdout:2704: done (2 left) 2024-09-13T14:32:24.576 INFO:tasks.rados.rados.0.smithi151.stdout:2705: rollback oid 32 current snap is 348 2024-09-13T14:32:24.576 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 32 to 347 2024-09-13T14:32:24.576 INFO:tasks.rados.rados.0.smithi151.stdout:2703: finishing rollback tid 1 to smithi15186934-1 2024-09-13T14:32:24.576 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 1 v 0 (ObjNum 81 snap 0 seq_num 1755290912) dirty dne 2024-09-13T14:32:24.576 INFO:tasks.rados.rados.0.smithi151.stdout:2702: finishing write tid 3 to smithi15186934-14 2024-09-13T14:32:24.576 INFO:tasks.rados.rados.0.smithi151.stdout:2702: finishing write tid 4 to smithi15186934-14 2024-09-13T14:32:24.576 INFO:tasks.rados.rados.0.smithi151.stdout:2702: finishing write tid 5 to smithi15186934-14 2024-09-13T14:32:24.576 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 1125 (ObjNum 1016 snap 348 seq_num 1016) dirty exists 2024-09-13T14:32:24.577 INFO:tasks.rados.rados.0.smithi151.stdout:2702: left oid 14 (ObjNum 1016 snap 348 seq_num 1016) 2024-09-13T14:32:24.577 INFO:tasks.rados.rados.0.smithi151.stdout:2702: done (2 left) 2024-09-13T14:32:24.577 INFO:tasks.rados.rados.0.smithi151.stdout:2703: done (1 left) 2024-09-13T14:32:24.577 INFO:tasks.rados.rados.0.smithi151.stdout:2706: read oid 12 snap -1 2024-09-13T14:32:24.577 INFO:tasks.rados.rados.0.smithi151.stdout:2706: expect deleted 2024-09-13T14:32:24.577 INFO:tasks.rados.rados.0.smithi151.stdout:2707: read oid 8 snap -1 2024-09-13T14:32:24.577 INFO:tasks.rados.rados.0.smithi151.stdout:2707: expect (ObjNum 977 snap 339 seq_num 977) 2024-09-13T14:32:24.577 INFO:tasks.rados.rados.0.smithi151.stdout:2708: rollback oid 26 current snap is 348 2024-09-13T14:32:24.577 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 26 to 346 2024-09-13T14:32:24.577 INFO:tasks.rados.rados.0.smithi151.stdout:2709: snap_remove snap 329 2024-09-13T14:32:24.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:24 smithi151 ceph-mon[104450]: pgmap v747: 89 pgs: 89 active+clean; 139 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 2.7 MiB/s wr, 30 op/s 2024-09-13T14:32:24.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:24 smithi151 ceph-mon[104450]: osdmap e753: 8 total, 8 up, 8 in 2024-09-13T14:32:24.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:24 smithi151 ceph-mon[104450]: osdmap e754: 8 total, 8 up, 8 in 2024-09-13T14:32:24.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:32:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:32:24] "GET /metrics HTTP/1.1" 200 36510 "" "Prometheus/2.51.0" 2024-09-13T14:32:24.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:24 smithi151 ceph-mon[108966]: pgmap v747: 89 pgs: 89 active+clean; 139 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 2.7 MiB/s wr, 30 op/s 2024-09-13T14:32:24.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:24 smithi151 ceph-mon[108966]: osdmap e753: 8 total, 8 up, 8 in 2024-09-13T14:32:24.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:24 smithi151 ceph-mon[108966]: osdmap e754: 8 total, 8 up, 8 in 2024-09-13T14:32:25.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:24 smithi184 ceph-mon[92527]: pgmap v747: 89 pgs: 89 active+clean; 139 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 2.7 MiB/s wr, 30 op/s 2024-09-13T14:32:25.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:24 smithi184 ceph-mon[92527]: osdmap e753: 8 total, 8 up, 8 in 2024-09-13T14:32:25.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:24 smithi184 ceph-mon[92527]: osdmap e754: 8 total, 8 up, 8 in 2024-09-13T14:32:25.576 INFO:tasks.rados.rados.0.smithi151.stdout:2709: done (4 left) 2024-09-13T14:32:25.576 INFO:tasks.rados.rados.0.smithi151.stdout:2710: read oid 18 snap 339 2024-09-13T14:32:25.576 INFO:tasks.rados.rados.0.smithi151.stdout:2710: expect deleted 2024-09-13T14:32:25.576 INFO:tasks.rados.rados.0.smithi151.stdout:2711: delete oid 50 current snap is 348 2024-09-13T14:32:25.577 INFO:tasks.rados.rados.0.smithi151.stdout:2707: expect (ObjNum 977 snap 339 seq_num 977) 2024-09-13T14:32:25.732 INFO:tasks.rados.rados.0.smithi151.stdout:2705: finishing rollback tid 1 to smithi15186934-32 2024-09-13T14:32:25.732 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 32 v 1101 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-13T14:32:25.733 INFO:tasks.rados.rados.0.smithi151.stdout:2708: finishing rollback tid 0 to smithi15186934-26 2024-09-13T14:32:25.733 INFO:tasks.rados.rados.0.smithi151.stdout:2708: finishing rollback tid 1 to smithi15186934-26 2024-09-13T14:32:25.733 INFO:tasks.rados.rados.0.smithi151.stdout:2708: finishing rollback tid 2 to smithi15186934-26 2024-09-13T14:32:25.733 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 968 (ObjNum 941 snap 325 seq_num 941) dirty exists 2024-09-13T14:32:25.733 INFO:tasks.rados.rados.0.smithi151.stdout:2705: done (5 left) 2024-09-13T14:32:25.733 INFO:tasks.rados.rados.0.smithi151.stdout:2706: done (4 left) 2024-09-13T14:32:25.733 INFO:tasks.rados.rados.0.smithi151.stdout:2707: done (3 left) 2024-09-13T14:32:25.733 INFO:tasks.rados.rados.0.smithi151.stdout:2708: done (2 left) 2024-09-13T14:32:25.733 INFO:tasks.rados.rados.0.smithi151.stdout:2711: done (1 left) 2024-09-13T14:32:25.733 INFO:tasks.rados.rados.0.smithi151.stdout:2712: write oid 31 current snap is 348 2024-09-13T14:32:25.733 INFO:tasks.rados.rados.0.smithi151.stdout:2712: seq_num 1018 ranges {709526=478431,1637728=464342,2856136=253390} 2024-09-13T14:32:25.738 INFO:tasks.rados.rados.0.smithi151.stdout:2712: writing smithi15186934-31 from 709526 to 1187957 tid 1 2024-09-13T14:32:25.743 INFO:tasks.rados.rados.0.smithi151.stdout:2712: writing smithi15186934-31 from 1637728 to 2102070 tid 2 2024-09-13T14:32:25.749 INFO:tasks.rados.rados.0.smithi151.stdout:2712: writing smithi15186934-31 from 2856136 to 3109526 tid 3 2024-09-13T14:32:25.749 INFO:tasks.rados.rados.0.smithi151.stdout:2710: done (1 left) 2024-09-13T14:32:25.749 INFO:tasks.rados.rados.0.smithi151.stdout:2713: snap_remove snap 340 2024-09-13T14:32:26.647 INFO:tasks.rados.rados.0.smithi151.stdout:2713: done (1 left) 2024-09-13T14:32:26.647 INFO:tasks.rados.rados.0.smithi151.stdout:2714: delete oid 2 current snap is 348 2024-09-13T14:32:26.647 INFO:tasks.rados.rados.0.smithi151.stdout:2712: finishing write tid 1 to smithi15186934-31 2024-09-13T14:32:26.647 INFO:tasks.rados.rados.0.smithi151.stdout:2712: finishing write tid 2 to smithi15186934-31 2024-09-13T14:32:26.647 INFO:tasks.rados.rados.0.smithi151.stdout:2712: finishing write tid 3 to smithi15186934-31 2024-09-13T14:32:26.647 INFO:tasks.rados.rados.0.smithi151.stdout:2712: finishing write tid 4 to smithi15186934-31 2024-09-13T14:32:26.647 INFO:tasks.rados.rados.0.smithi151.stdout:2712: finishing write tid 5 to smithi15186934-31 2024-09-13T14:32:26.647 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 31 v 1037 (ObjNum 1018 snap 348 seq_num 1018) dirty exists 2024-09-13T14:32:26.648 INFO:tasks.rados.rados.0.smithi151.stdout:2712: left oid 31 (ObjNum 1018 snap 348 seq_num 1018) 2024-09-13T14:32:26.656 INFO:tasks.rados.rados.0.smithi151.stdout:2712: done (1 left) 2024-09-13T14:32:26.657 INFO:tasks.rados.rados.0.smithi151.stdout:2714: done (0 left) 2024-09-13T14:32:26.657 INFO:tasks.rados.rados.0.smithi151.stdout:2715: read oid 38 snap 346 2024-09-13T14:32:26.657 INFO:tasks.rados.rados.0.smithi151.stdout:2715: expect (ObjNum 1003 snap 345 seq_num 1003) 2024-09-13T14:32:26.657 INFO:tasks.rados.rados.0.smithi151.stdout:2716: write oid 26 current snap is 348 2024-09-13T14:32:26.657 INFO:tasks.rados.rados.0.smithi151.stdout:2716: seq_num 1020 ranges {743140=745282,2154692=468011} 2024-09-13T14:32:26.664 INFO:tasks.rados.rados.0.smithi151.stdout:2716: writing smithi15186934-26 from 743140 to 1488422 tid 1 2024-09-13T14:32:26.671 INFO:tasks.rados.rados.0.smithi151.stdout:2716: writing smithi15186934-26 from 2154692 to 2622703 tid 2 2024-09-13T14:32:26.671 INFO:tasks.rados.rados.0.smithi151.stdout:2717: read oid 36 snap -1 2024-09-13T14:32:26.671 INFO:tasks.rados.rados.0.smithi151.stdout:2717: expect (ObjNum 947 snap 325 seq_num 947) 2024-09-13T14:32:26.671 INFO:tasks.rados.rados.0.smithi151.stdout:2718: snap_create 2024-09-13T14:32:26.671 INFO:tasks.rados.rados.0.smithi151.stdout:2715: expect (ObjNum 1003 snap 345 seq_num 1003) 2024-09-13T14:32:26.803 INFO:tasks.rados.rados.0.smithi151.stdout:2716: finishing write tid 1 to smithi15186934-26 2024-09-13T14:32:26.803 INFO:tasks.rados.rados.0.smithi151.stdout:2716: finishing write tid 2 to smithi15186934-26 2024-09-13T14:32:26.803 INFO:tasks.rados.rados.0.smithi151.stdout:2716: finishing write tid 3 to smithi15186934-26 2024-09-13T14:32:26.803 INFO:tasks.rados.rados.0.smithi151.stdout:2716: finishing write tid 4 to smithi15186934-26 2024-09-13T14:32:26.804 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 981 (ObjNum 1020 snap 348 seq_num 1020) dirty exists 2024-09-13T14:32:26.804 INFO:tasks.rados.rados.0.smithi151.stdout:2716: left oid 26 (ObjNum 1020 snap 348 seq_num 1020) 2024-09-13T14:32:26.804 INFO:tasks.rados.rados.0.smithi151.stdout:2717: expect (ObjNum 947 snap 325 seq_num 947) 2024-09-13T14:32:26.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:26 smithi184 ceph-mon[92527]: pgmap v750: 89 pgs: 89 active+clean; 143 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 3.3 MiB/s wr, 32 op/s 2024-09-13T14:32:26.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:26 smithi184 ceph-mon[92527]: osdmap e755: 8 total, 8 up, 8 in 2024-09-13T14:32:26.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:26 smithi151 ceph-mon[104450]: pgmap v750: 89 pgs: 89 active+clean; 143 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 3.3 MiB/s wr, 32 op/s 2024-09-13T14:32:26.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:26 smithi151 ceph-mon[104450]: osdmap e755: 8 total, 8 up, 8 in 2024-09-13T14:32:26.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:26 smithi151 ceph-mon[108966]: pgmap v750: 89 pgs: 89 active+clean; 143 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 3.3 MiB/s wr, 32 op/s 2024-09-13T14:32:26.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:26 smithi151 ceph-mon[108966]: osdmap e755: 8 total, 8 up, 8 in 2024-09-13T14:32:27.654 INFO:tasks.rados.rados.0.smithi151.stdout:2715: done (3 left) 2024-09-13T14:32:27.654 INFO:tasks.rados.rados.0.smithi151.stdout:2716: done (2 left) 2024-09-13T14:32:27.654 INFO:tasks.rados.rados.0.smithi151.stdout:2717: done (1 left) 2024-09-13T14:32:27.654 INFO:tasks.rados.rados.0.smithi151.stdout:2718: done (0 left) 2024-09-13T14:32:27.654 INFO:tasks.rados.rados.0.smithi151.stdout:2719: read oid 4 snap -1 2024-09-13T14:32:27.654 INFO:tasks.rados.rados.0.smithi151.stdout:2719: expect deleted 2024-09-13T14:32:27.654 INFO:tasks.rados.rados.0.smithi151.stdout:2720: snap_remove snap 346 2024-09-13T14:32:28.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:27 smithi184 ceph-mon[92527]: osdmap e756: 8 total, 8 up, 8 in 2024-09-13T14:32:28.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:27 smithi151 ceph-mon[104450]: osdmap e756: 8 total, 8 up, 8 in 2024-09-13T14:32:28.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:27 smithi151 ceph-mon[108966]: osdmap e756: 8 total, 8 up, 8 in 2024-09-13T14:32:28.657 INFO:tasks.rados.rados.0.smithi151.stdout:2720: done (1 left) 2024-09-13T14:32:28.657 INFO:tasks.rados.rados.0.smithi151.stdout:2721: snap_remove snap 337 2024-09-13T14:32:29.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:28 smithi184 ceph-mon[92527]: pgmap v753: 89 pgs: 89 active+clean; 153 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2024-09-13T14:32:29.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:28 smithi184 ceph-mon[92527]: osdmap e757: 8 total, 8 up, 8 in 2024-09-13T14:32:29.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:28 smithi151 ceph-mon[104450]: pgmap v753: 89 pgs: 89 active+clean; 153 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2024-09-13T14:32:29.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:28 smithi151 ceph-mon[104450]: osdmap e757: 8 total, 8 up, 8 in 2024-09-13T14:32:29.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:28 smithi151 ceph-mon[108966]: pgmap v753: 89 pgs: 89 active+clean; 153 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2024-09-13T14:32:29.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:28 smithi151 ceph-mon[108966]: osdmap e757: 8 total, 8 up, 8 in 2024-09-13T14:32:29.576 INFO:tasks.rados.rados.0.smithi151.stdout:2721: done (1 left) 2024-09-13T14:32:29.576 INFO:tasks.rados.rados.0.smithi151.stdout:2722: write oid 16 current snap is 349 2024-09-13T14:32:29.576 INFO:tasks.rados.rados.0.smithi151.stdout:2722: seq_num 1021 ranges {759947=420531,1813174=728022} 2024-09-13T14:32:29.583 INFO:tasks.rados.rados.0.smithi151.stdout:2722: writing smithi15186934-16 from 759947 to 1180478 tid 1 2024-09-13T14:32:29.591 INFO:tasks.rados.rados.0.smithi151.stdout:2722: writing smithi15186934-16 from 1813174 to 2541196 tid 2 2024-09-13T14:32:29.591 INFO:tasks.rados.rados.0.smithi151.stdout:2723: snap_create 2024-09-13T14:32:29.592 INFO:tasks.rados.rados.0.smithi151.stdout:2722: finishing write tid 1 to smithi15186934-16 2024-09-13T14:32:29.595 INFO:tasks.rados.rados.0.smithi151.stdout:2722: finishing write tid 2 to smithi15186934-16 2024-09-13T14:32:29.595 INFO:tasks.rados.rados.0.smithi151.stdout:2722: finishing write tid 3 to smithi15186934-16 2024-09-13T14:32:29.595 INFO:tasks.rados.rados.0.smithi151.stdout:2722: finishing write tid 4 to smithi15186934-16 2024-09-13T14:32:29.595 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 16 v 705 (ObjNum 1021 snap 349 seq_num 1021) dirty exists 2024-09-13T14:32:29.595 INFO:tasks.rados.rados.0.smithi151.stdout:2722: left oid 16 (ObjNum 1021 snap 349 seq_num 1021) 2024-09-13T14:32:30.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:29 smithi184 ceph-mon[92527]: osdmap e758: 8 total, 8 up, 8 in 2024-09-13T14:32:30.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:29 smithi184 ceph-mon[92527]: osdmap e759: 8 total, 8 up, 8 in 2024-09-13T14:32:30.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:29 smithi151 ceph-mon[104450]: osdmap e758: 8 total, 8 up, 8 in 2024-09-13T14:32:30.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:29 smithi151 ceph-mon[104450]: osdmap e759: 8 total, 8 up, 8 in 2024-09-13T14:32:30.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:29 smithi151 ceph-mon[108966]: osdmap e758: 8 total, 8 up, 8 in 2024-09-13T14:32:30.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:29 smithi151 ceph-mon[108966]: osdmap e759: 8 total, 8 up, 8 in 2024-09-13T14:32:30.579 INFO:tasks.rados.rados.0.smithi151.stdout:2719: done (2 left) 2024-09-13T14:32:30.579 INFO:tasks.rados.rados.0.smithi151.stdout:2722: done (1 left) 2024-09-13T14:32:30.579 INFO:tasks.rados.rados.0.smithi151.stdout:2723: done (0 left) 2024-09-13T14:32:30.579 INFO:tasks.rados.rados.0.smithi151.stdout:2724: read oid 27 snap 341 2024-09-13T14:32:30.579 INFO:tasks.rados.rados.0.smithi151.stdout:2724: expect deleted 2024-09-13T14:32:30.580 INFO:tasks.rados.rados.0.smithi151.stdout:2725: write oid 30 current snap is 350 2024-09-13T14:32:30.580 INFO:tasks.rados.rados.0.smithi151.stdout:2725: seq_num 1022 ranges {776754=495780,1871656=471680,2965828=210926} 2024-09-13T14:32:30.586 INFO:tasks.rados.rados.0.smithi151.stdout:2725: writing smithi15186934-30 from 776754 to 1272534 tid 1 2024-09-13T14:32:30.592 INFO:tasks.rados.rados.0.smithi151.stdout:2725: writing smithi15186934-30 from 1871656 to 2343336 tid 2 2024-09-13T14:32:30.597 INFO:tasks.rados.rados.0.smithi151.stdout:2725: writing smithi15186934-30 from 2965828 to 3176754 tid 3 2024-09-13T14:32:30.597 INFO:tasks.rados.rados.0.smithi151.stdout:2726: write oid 24 current snap is 350 2024-09-13T14:32:30.597 INFO:tasks.rados.rados.0.smithi151.stdout:2726: seq_num 1023 ranges {793561=571029,1813785=731691,2993251=200310} 2024-09-13T14:32:30.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:32:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:32:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:32:30.604 INFO:tasks.rados.rados.0.smithi151.stdout:2726: writing smithi15186934-24 from 793561 to 1364590 tid 1 2024-09-13T14:32:30.611 INFO:tasks.rados.rados.0.smithi151.stdout:2726: writing smithi15186934-24 from 1813785 to 2545476 tid 2 2024-09-13T14:32:30.614 INFO:tasks.rados.rados.0.smithi151.stdout:2726: writing smithi15186934-24 from 2993251 to 3193561 tid 3 2024-09-13T14:32:30.614 INFO:tasks.rados.rados.0.smithi151.stdout:2727: snap_remove snap 343 2024-09-13T14:32:31.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:30 smithi184 ceph-mon[92527]: pgmap v756: 89 pgs: 89 active+clean; 158 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2024-09-13T14:32:31.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:30 smithi184 ceph-mon[92527]: osdmap e760: 8 total, 8 up, 8 in 2024-09-13T14:32:31.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:30 smithi151 ceph-mon[108966]: pgmap v756: 89 pgs: 89 active+clean; 158 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2024-09-13T14:32:31.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:30 smithi151 ceph-mon[108966]: osdmap e760: 8 total, 8 up, 8 in 2024-09-13T14:32:31.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:30 smithi151 ceph-mon[104450]: pgmap v756: 89 pgs: 89 active+clean; 158 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2024-09-13T14:32:31.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:30 smithi151 ceph-mon[104450]: osdmap e760: 8 total, 8 up, 8 in 2024-09-13T14:32:31.586 INFO:tasks.rados.rados.0.smithi151.stdout:2727: done (3 left) 2024-09-13T14:32:31.586 INFO:tasks.rados.rados.0.smithi151.stdout:2728: read oid 19 snap -1 2024-09-13T14:32:31.586 INFO:tasks.rados.rados.0.smithi151.stdout:2728: expect (ObjNum 1012 snap 347 seq_num 1012) 2024-09-13T14:32:31.586 INFO:tasks.rados.rados.0.smithi151.stdout:2729: read oid 34 snap 342 2024-09-13T14:32:31.586 INFO:tasks.rados.rados.0.smithi151.stdout:2729: expect (ObjNum 970 snap 335 seq_num 970) 2024-09-13T14:32:31.586 INFO:tasks.rados.rados.0.smithi151.stdout:2730: rollback oid 5 current snap is 350 2024-09-13T14:32:31.586 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 5 to 331 2024-09-13T14:32:31.586 INFO:tasks.rados.rados.0.smithi151.stdout:2725: finishing write tid 1 to smithi15186934-30 2024-09-13T14:32:31.586 INFO:tasks.rados.rados.0.smithi151.stdout:2725: finishing write tid 2 to smithi15186934-30 2024-09-13T14:32:31.587 INFO:tasks.rados.rados.0.smithi151.stdout:2725: finishing write tid 3 to smithi15186934-30 2024-09-13T14:32:31.587 INFO:tasks.rados.rados.0.smithi151.stdout:2725: finishing write tid 4 to smithi15186934-30 2024-09-13T14:32:31.587 INFO:tasks.rados.rados.0.smithi151.stdout:2725: finishing write tid 5 to smithi15186934-30 2024-09-13T14:32:31.587 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 30 v 1078 (ObjNum 1022 snap 350 seq_num 1022) dirty exists 2024-09-13T14:32:31.587 INFO:tasks.rados.rados.0.smithi151.stdout:2725: left oid 30 (ObjNum 1022 snap 350 seq_num 1022) 2024-09-13T14:32:31.587 INFO:tasks.rados.rados.0.smithi151.stdout:2726: finishing write tid 1 to smithi15186934-24 2024-09-13T14:32:31.587 INFO:tasks.rados.rados.0.smithi151.stdout:2726: finishing write tid 2 to smithi15186934-24 2024-09-13T14:32:31.587 INFO:tasks.rados.rados.0.smithi151.stdout:2726: finishing write tid 3 to smithi15186934-24 2024-09-13T14:32:31.587 INFO:tasks.rados.rados.0.smithi151.stdout:2726: finishing write tid 4 to smithi15186934-24 2024-09-13T14:32:31.587 INFO:tasks.rados.rados.0.smithi151.stdout:2726: finishing write tid 5 to smithi15186934-24 2024-09-13T14:32:31.587 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 24 v 1652 (ObjNum 1023 snap 350 seq_num 1023) dirty exists 2024-09-13T14:32:31.587 INFO:tasks.rados.rados.0.smithi151.stdout:2726: left oid 24 (ObjNum 1023 snap 350 seq_num 1023) 2024-09-13T14:32:31.587 INFO:tasks.rados.rados.0.smithi151.stdout:2724: done (5 left) 2024-09-13T14:32:31.587 INFO:tasks.rados.rados.0.smithi151.stdout:2725: done (4 left) 2024-09-13T14:32:31.588 INFO:tasks.rados.rados.0.smithi151.stdout:2726: done (3 left) 2024-09-13T14:32:31.588 INFO:tasks.rados.rados.0.smithi151.stdout:2731: snap_create 2024-09-13T14:32:31.593 INFO:tasks.rados.rados.0.smithi151.stdout:2730: finishing rollback tid 1 to smithi15186934-5 2024-09-13T14:32:31.593 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 1011 (ObjNum 129 snap 0 seq_num 3092546584) dirty dne 2024-09-13T14:32:31.596 INFO:tasks.rados.rados.0.smithi151.stdout:2729: expect (ObjNum 970 snap 335 seq_num 970) 2024-09-13T14:32:31.691 INFO:tasks.rados.rados.0.smithi151.stdout:2728: expect (ObjNum 1012 snap 347 seq_num 1012) 2024-09-13T14:32:32.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:32.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:31 smithi184 ceph-mon[92527]: osdmap e761: 8 total, 8 up, 8 in 2024-09-13T14:32:32.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:32.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:31 smithi151 ceph-mon[108966]: osdmap e761: 8 total, 8 up, 8 in 2024-09-13T14:32:32.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:32.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:31 smithi151 ceph-mon[104450]: osdmap e761: 8 total, 8 up, 8 in 2024-09-13T14:32:32.584 INFO:tasks.rados.rados.0.smithi151.stdout:2728: done (3 left) 2024-09-13T14:32:32.584 INFO:tasks.rados.rados.0.smithi151.stdout:2729: done (2 left) 2024-09-13T14:32:32.584 INFO:tasks.rados.rados.0.smithi151.stdout:2730: done (1 left) 2024-09-13T14:32:32.584 INFO:tasks.rados.rados.0.smithi151.stdout:2731: done (0 left) 2024-09-13T14:32:32.584 INFO:tasks.rados.rados.0.smithi151.stdout:2732: rollback oid 32 current snap is 351 2024-09-13T14:32:32.585 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 32 to 339 2024-09-13T14:32:32.585 INFO:tasks.rados.rados.0.smithi151.stdout:2733: delete oid 43 current snap is 351 2024-09-13T14:32:32.589 INFO:tasks.rados.rados.0.smithi151.stdout:2732: finishing rollback tid 1 to smithi15186934-32 2024-09-13T14:32:32.589 INFO:tasks.rados.rados.0.smithi151.stdout:2732: finishing rollback tid 2 to smithi15186934-32 2024-09-13T14:32:32.589 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 32 v 1146 (ObjNum 925 snap 318 seq_num 925) dirty exists 2024-09-13T14:32:32.589 INFO:tasks.rados.rados.0.smithi151.stdout:2732: done (1 left) 2024-09-13T14:32:32.590 INFO:tasks.rados.rados.0.smithi151.stdout:2733: done (0 left) 2024-09-13T14:32:32.590 INFO:tasks.rados.rados.0.smithi151.stdout:2734: read oid 8 snap -1 2024-09-13T14:32:32.590 INFO:tasks.rados.rados.0.smithi151.stdout:2734: expect (ObjNum 977 snap 339 seq_num 977) 2024-09-13T14:32:32.590 INFO:tasks.rados.rados.0.smithi151.stdout:2735: snap_create 2024-09-13T14:32:32.591 INFO:tasks.rados.rados.0.smithi151.stdout:2734: expect (ObjNum 977 snap 339 seq_num 977) 2024-09-13T14:32:33.009 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:32 smithi151 ceph-mon[104450]: pgmap v759: 89 pgs: 89 active+clean; 156 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.6 MiB/s rd, 589 KiB/s wr, 5 op/s 2024-09-13T14:32:33.010 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:32 smithi151 ceph-mon[104450]: osdmap e762: 8 total, 8 up, 8 in 2024-09-13T14:32:33.010 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:32 smithi151 ceph-mon[108966]: pgmap v759: 89 pgs: 89 active+clean; 156 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.6 MiB/s rd, 589 KiB/s wr, 5 op/s 2024-09-13T14:32:33.010 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:32 smithi151 ceph-mon[108966]: osdmap e762: 8 total, 8 up, 8 in 2024-09-13T14:32:33.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:32 smithi184 ceph-mon[92527]: pgmap v759: 89 pgs: 89 active+clean; 156 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.6 MiB/s rd, 589 KiB/s wr, 5 op/s 2024-09-13T14:32:33.175 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:32 smithi184 ceph-mon[92527]: osdmap e762: 8 total, 8 up, 8 in 2024-09-13T14:32:33.584 INFO:tasks.rados.rados.0.smithi151.stdout:2734: done (1 left) 2024-09-13T14:32:33.584 INFO:tasks.rados.rados.0.smithi151.stdout:2735: done (0 left) 2024-09-13T14:32:33.584 INFO:tasks.rados.rados.0.smithi151.stdout:2736: read oid 7 snap 350 2024-09-13T14:32:33.585 INFO:tasks.rados.rados.0.smithi151.stdout:2736: expect (ObjNum 951 snap 327 seq_num 951) 2024-09-13T14:32:33.585 INFO:tasks.rados.rados.0.smithi151.stdout:2737: write oid 17 current snap is 352 2024-09-13T14:32:33.585 INFO:tasks.rados.rados.0.smithi151.stdout:2737: seq_num 1025 ranges {427175=721527,1930749=735360} 2024-09-13T14:32:33.590 INFO:tasks.rados.rados.0.smithi151.stdout:2737: writing smithi15186934-17 from 427175 to 1148702 tid 1 2024-09-13T14:32:33.602 INFO:tasks.rados.rados.0.smithi151.stdout:2737: writing smithi15186934-17 from 1930749 to 2666109 tid 2 2024-09-13T14:32:33.602 INFO:tasks.rados.rados.0.smithi151.stdout:2738: read oid 13 snap -1 2024-09-13T14:32:33.602 INFO:tasks.rados.rados.0.smithi151.stdout:2738: expect (ObjNum 1008 snap 345 seq_num 1008) 2024-09-13T14:32:33.602 INFO:tasks.rados.rados.0.smithi151.stdout:2739: read oid 40 snap -1 2024-09-13T14:32:33.602 INFO:tasks.rados.rados.0.smithi151.stdout:2739: expect deleted 2024-09-13T14:32:33.602 INFO:tasks.rados.rados.0.smithi151.stdout:2740: write oid 11 current snap is 352 2024-09-13T14:32:33.602 INFO:tasks.rados.rados.0.smithi151.stdout:2740: seq_num 1026 ranges {443982=796776,1989231=479018,2959167=284815} 2024-09-13T14:32:33.609 INFO:tasks.rados.rados.0.smithi151.stdout:2740: writing smithi15186934-11 from 443982 to 1240758 tid 1 2024-09-13T14:32:33.616 INFO:tasks.rados.rados.0.smithi151.stdout:2740: writing smithi15186934-11 from 1989231 to 2468249 tid 2 2024-09-13T14:32:33.620 INFO:tasks.rados.rados.0.smithi151.stdout:2740: writing smithi15186934-11 from 2959167 to 3243982 tid 3 2024-09-13T14:32:33.621 INFO:tasks.rados.rados.0.smithi151.stdout:2741: read oid 34 snap -1 2024-09-13T14:32:33.621 INFO:tasks.rados.rados.0.smithi151.stdout:2741: expect (ObjNum 970 snap 335 seq_num 970) 2024-09-13T14:32:33.621 INFO:tasks.rados.rados.0.smithi151.stdout:2742: read oid 35 snap 339 2024-09-13T14:32:33.621 INFO:tasks.rados.rados.0.smithi151.stdout:2742: expect (ObjNum 973 snap 339 seq_num 973) 2024-09-13T14:32:33.621 INFO:tasks.rados.rados.0.smithi151.stdout:2743: write oid 19 current snap is 352 2024-09-13T14:32:33.621 INFO:tasks.rados.rados.0.smithi151.stdout:2743: seq_num 1027 ranges {460789=588378,1764066=739029,3219296=41493} 2024-09-13T14:32:33.627 INFO:tasks.rados.rados.0.smithi151.stdout:2743: writing smithi15186934-19 from 460789 to 1049167 tid 1 2024-09-13T14:32:33.636 INFO:tasks.rados.rados.0.smithi151.stdout:2743: writing smithi15186934-19 from 1764066 to 2503095 tid 2 2024-09-13T14:32:33.640 INFO:tasks.rados.rados.0.smithi151.stdout:2743: writing smithi15186934-19 from 3219296 to 3260789 tid 3 2024-09-13T14:32:33.640 INFO:tasks.rados.rados.0.smithi151.stdout:2744: snap_remove snap 348 2024-09-13T14:32:34.578 INFO:tasks.rados.rados.0.smithi151.stdout:2744: done (8 left) 2024-09-13T14:32:34.578 INFO:tasks.rados.rados.0.smithi151.stdout:2745: write oid 47 current snap is 352 2024-09-13T14:32:34.579 INFO:tasks.rados.rados.0.smithi151.stdout:2736: expect (ObjNum 951 snap 327 seq_num 951) 2024-09-13T14:32:34.806 INFO:tasks.rados.rados.0.smithi151.stdout:2737: finishing write tid 1 to smithi15186934-17 2024-09-13T14:32:34.806 INFO:tasks.rados.rados.0.smithi151.stdout:2737: finishing write tid 2 to smithi15186934-17 2024-09-13T14:32:34.806 INFO:tasks.rados.rados.0.smithi151.stdout:2737: finishing write tid 3 to smithi15186934-17 2024-09-13T14:32:34.807 INFO:tasks.rados.rados.0.smithi151.stdout:2737: finishing write tid 4 to smithi15186934-17 2024-09-13T14:32:34.807 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 1667 (ObjNum 1025 snap 352 seq_num 1025) dirty exists 2024-09-13T14:32:34.807 INFO:tasks.rados.rados.0.smithi151.stdout:2737: left oid 17 (ObjNum 1025 snap 352 seq_num 1025) 2024-09-13T14:32:34.807 INFO:tasks.rados.rados.0.smithi151.stdout:2738: expect (ObjNum 1008 snap 345 seq_num 1008) 2024-09-13T14:32:34.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:34 smithi184 ceph-mon[92527]: pgmap v762: 89 pgs: 89 active+clean; 160 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 7.6 MiB/s rd, 648 KiB/s wr, 7 op/s 2024-09-13T14:32:34.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:34 smithi184 ceph-mon[92527]: osdmap e763: 8 total, 8 up, 8 in 2024-09-13T14:32:34.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:34 smithi184 ceph-mon[92527]: osdmap e764: 8 total, 8 up, 8 in 2024-09-13T14:32:34.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:34 smithi151 ceph-mon[104450]: pgmap v762: 89 pgs: 89 active+clean; 160 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 7.6 MiB/s rd, 648 KiB/s wr, 7 op/s 2024-09-13T14:32:34.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:34 smithi151 ceph-mon[104450]: osdmap e763: 8 total, 8 up, 8 in 2024-09-13T14:32:34.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:34 smithi151 ceph-mon[104450]: osdmap e764: 8 total, 8 up, 8 in 2024-09-13T14:32:34.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:34 smithi151 ceph-mon[108966]: pgmap v762: 89 pgs: 89 active+clean; 160 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 7.6 MiB/s rd, 648 KiB/s wr, 7 op/s 2024-09-13T14:32:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:34 smithi151 ceph-mon[108966]: osdmap e763: 8 total, 8 up, 8 in 2024-09-13T14:32:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:34 smithi151 ceph-mon[108966]: osdmap e764: 8 total, 8 up, 8 in 2024-09-13T14:32:34.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:32:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:32:34] "GET /metrics HTTP/1.1" 200 36511 "" "Prometheus/2.51.0" 2024-09-13T14:32:34.906 INFO:tasks.rados.rados.0.smithi151.stdout:2740: finishing write tid 1 to smithi15186934-11 2024-09-13T14:32:34.906 INFO:tasks.rados.rados.0.smithi151.stdout:2740: finishing write tid 2 to smithi15186934-11 2024-09-13T14:32:34.906 INFO:tasks.rados.rados.0.smithi151.stdout:2741: expect (ObjNum 970 snap 335 seq_num 970) 2024-09-13T14:32:34.999 INFO:tasks.rados.rados.0.smithi151.stdout:2740: finishing write tid 3 to smithi15186934-11 2024-09-13T14:32:34.999 INFO:tasks.rados.rados.0.smithi151.stdout:2740: finishing write tid 4 to smithi15186934-11 2024-09-13T14:32:34.999 INFO:tasks.rados.rados.0.smithi151.stdout:2740: finishing write tid 5 to smithi15186934-11 2024-09-13T14:32:34.999 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 11 v 660 (ObjNum 1026 snap 352 seq_num 1026) dirty exists 2024-09-13T14:32:34.999 INFO:tasks.rados.rados.0.smithi151.stdout:2740: left oid 11 (ObjNum 1026 snap 352 seq_num 1026) 2024-09-13T14:32:34.999 INFO:tasks.rados.rados.0.smithi151.stdout:2742: expect (ObjNum 973 snap 339 seq_num 973) 2024-09-13T14:32:35.074 INFO:tasks.rados.rados.0.smithi151.stdout:2743: finishing write tid 1 to smithi15186934-19 2024-09-13T14:32:35.074 INFO:tasks.rados.rados.0.smithi151.stdout:2743: finishing write tid 2 to smithi15186934-19 2024-09-13T14:32:35.074 INFO:tasks.rados.rados.0.smithi151.stdout:2743: finishing write tid 3 to smithi15186934-19 2024-09-13T14:32:35.074 INFO:tasks.rados.rados.0.smithi151.stdout:2743: finishing write tid 4 to smithi15186934-19 2024-09-13T14:32:35.074 INFO:tasks.rados.rados.0.smithi151.stdout:2743: finishing write tid 5 to smithi15186934-19 2024-09-13T14:32:35.074 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 19 v 713 (ObjNum 1027 snap 352 seq_num 1027) dirty exists 2024-09-13T14:32:35.074 INFO:tasks.rados.rados.0.smithi151.stdout:2743: left oid 19 (ObjNum 1027 snap 352 seq_num 1027) 2024-09-13T14:32:35.074 INFO:tasks.rados.rados.0.smithi151.stdout:2745: seq_num 1028 ranges {477596=663627,1706195=482687,2614013=567041} 2024-09-13T14:32:35.079 INFO:tasks.rados.rados.0.smithi151.stdout:2745: writing smithi15186934-47 from 477596 to 1141223 tid 1 2024-09-13T14:32:35.086 INFO:tasks.rados.rados.0.smithi151.stdout:2745: writing smithi15186934-47 from 1706195 to 2188882 tid 2 2024-09-13T14:32:35.092 INFO:tasks.rados.rados.0.smithi151.stdout:2745: writing smithi15186934-47 from 2614013 to 3181054 tid 3 2024-09-13T14:32:35.092 INFO:tasks.rados.rados.0.smithi151.stdout:2736: done (8 left) 2024-09-13T14:32:35.092 INFO:tasks.rados.rados.0.smithi151.stdout:2737: done (7 left) 2024-09-13T14:32:35.092 INFO:tasks.rados.rados.0.smithi151.stdout:2738: done (6 left) 2024-09-13T14:32:35.092 INFO:tasks.rados.rados.0.smithi151.stdout:2739: done (5 left) 2024-09-13T14:32:35.092 INFO:tasks.rados.rados.0.smithi151.stdout:2740: done (4 left) 2024-09-13T14:32:35.092 INFO:tasks.rados.rados.0.smithi151.stdout:2741: done (3 left) 2024-09-13T14:32:35.092 INFO:tasks.rados.rados.0.smithi151.stdout:2742: done (2 left) 2024-09-13T14:32:35.093 INFO:tasks.rados.rados.0.smithi151.stdout:2743: done (1 left) 2024-09-13T14:32:35.093 INFO:tasks.rados.rados.0.smithi151.stdout:2746: rollback oid 1 current snap is 352 2024-09-13T14:32:35.093 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 1 to 351 2024-09-13T14:32:35.093 INFO:tasks.rados.rados.0.smithi151.stdout:2745: finishing write tid 1 to smithi15186934-47 2024-09-13T14:32:35.093 INFO:tasks.rados.rados.0.smithi151.stdout:2747: read oid 29 snap -1 2024-09-13T14:32:35.093 INFO:tasks.rados.rados.0.smithi151.stdout:2747: expect deleted 2024-09-13T14:32:35.093 INFO:tasks.rados.rados.0.smithi151.stdout:2748: snap_remove snap 327 2024-09-13T14:32:35.590 INFO:tasks.rados.rados.0.smithi151.stdout:2748: done (3 left) 2024-09-13T14:32:35.590 INFO:tasks.rados.rados.0.smithi151.stdout:2749: delete oid 31 current snap is 352 2024-09-13T14:32:35.590 INFO:tasks.rados.rados.0.smithi151.stdout:2745: finishing write tid 2 to smithi15186934-47 2024-09-13T14:32:35.590 INFO:tasks.rados.rados.0.smithi151.stdout:2746: finishing rollback tid 1 to smithi15186934-1 2024-09-13T14:32:35.590 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 1 v 0 (ObjNum 129 snap 0 seq_num 3092463120) dirty dne 2024-09-13T14:32:35.590 INFO:tasks.rados.rados.0.smithi151.stdout:2745: finishing write tid 3 to smithi15186934-47 2024-09-13T14:32:35.590 INFO:tasks.rados.rados.0.smithi151.stdout:2745: finishing write tid 4 to smithi15186934-47 2024-09-13T14:32:35.590 INFO:tasks.rados.rados.0.smithi151.stdout:2745: finishing write tid 5 to smithi15186934-47 2024-09-13T14:32:35.591 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 1212 (ObjNum 1028 snap 352 seq_num 1028) dirty exists 2024-09-13T14:32:35.591 INFO:tasks.rados.rados.0.smithi151.stdout:2745: left oid 47 (ObjNum 1028 snap 352 seq_num 1028) 2024-09-13T14:32:35.598 INFO:tasks.rados.rados.0.smithi151.stdout:2745: done (3 left) 2024-09-13T14:32:35.598 INFO:tasks.rados.rados.0.smithi151.stdout:2746: done (2 left) 2024-09-13T14:32:35.598 INFO:tasks.rados.rados.0.smithi151.stdout:2747: done (1 left) 2024-09-13T14:32:35.598 INFO:tasks.rados.rados.0.smithi151.stdout:2749: done (0 left) 2024-09-13T14:32:35.598 INFO:tasks.rados.rados.0.smithi151.stdout:2750: write oid 22 current snap is 352 2024-09-13T14:32:35.598 INFO:tasks.rados.rados.0.smithi151.stdout:2750: seq_num 1030 ranges {511210=414125,1423159=486356,2668859=589585} 2024-09-13T14:32:35.603 INFO:tasks.rados.rados.0.smithi151.stdout:2750: writing smithi15186934-22 from 511210 to 925335 tid 1 2024-09-13T14:32:35.608 INFO:tasks.rados.rados.0.smithi151.stdout:2750: writing smithi15186934-22 from 1423159 to 1909515 tid 2 2024-09-13T14:32:35.616 INFO:tasks.rados.rados.0.smithi151.stdout:2750: writing smithi15186934-22 from 2668859 to 3258444 tid 3 2024-09-13T14:32:35.616 INFO:tasks.rados.rados.0.smithi151.stdout:2751: write oid 40 current snap is 352 2024-09-13T14:32:35.616 INFO:tasks.rados.rados.0.smithi151.stdout:2751: seq_num 1031 ranges {528017=489374,1481641=630014,2696282=400857} 2024-09-13T14:32:35.621 INFO:tasks.rados.rados.0.smithi151.stdout:2751: writing smithi15186934-40 from 528017 to 1017391 tid 1 2024-09-13T14:32:35.627 INFO:tasks.rados.rados.0.smithi151.stdout:2751: writing smithi15186934-40 from 1481641 to 2111655 tid 2 2024-09-13T14:32:35.633 INFO:tasks.rados.rados.0.smithi151.stdout:2751: writing smithi15186934-40 from 2696282 to 3097139 tid 3 2024-09-13T14:32:35.633 INFO:tasks.rados.rados.0.smithi151.stdout:2752: delete oid 17 current snap is 352 2024-09-13T14:32:35.633 INFO:tasks.rados.rados.0.smithi151.stdout:2750: finishing write tid 1 to smithi15186934-22 2024-09-13T14:32:35.633 INFO:tasks.rados.rados.0.smithi151.stdout:2750: finishing write tid 2 to smithi15186934-22 2024-09-13T14:32:35.634 INFO:tasks.rados.rados.0.smithi151.stdout:2750: finishing write tid 3 to smithi15186934-22 2024-09-13T14:32:35.634 INFO:tasks.rados.rados.0.smithi151.stdout:2750: finishing write tid 4 to smithi15186934-22 2024-09-13T14:32:35.634 INFO:tasks.rados.rados.0.smithi151.stdout:2750: finishing write tid 5 to smithi15186934-22 2024-09-13T14:32:35.634 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 22 v 548 (ObjNum 1030 snap 352 seq_num 1030) dirty exists 2024-09-13T14:32:35.634 INFO:tasks.rados.rados.0.smithi151.stdout:2750: left oid 22 (ObjNum 1030 snap 352 seq_num 1030) 2024-09-13T14:32:35.634 INFO:tasks.rados.rados.0.smithi151.stdout:2751: finishing write tid 1 to smithi15186934-40 2024-09-13T14:32:35.634 INFO:tasks.rados.rados.0.smithi151.stdout:2751: finishing write tid 2 to smithi15186934-40 2024-09-13T14:32:35.635 INFO:tasks.rados.rados.0.smithi151.stdout:2750: done (2 left) 2024-09-13T14:32:35.635 INFO:tasks.rados.rados.0.smithi151.stdout:2752: done (1 left) 2024-09-13T14:32:35.636 INFO:tasks.rados.rados.0.smithi151.stdout:2753: delete oid 19 current snap is 352 2024-09-13T14:32:35.636 INFO:tasks.rados.rados.0.smithi151.stdout:2751: finishing write tid 3 to smithi15186934-40 2024-09-13T14:32:35.636 INFO:tasks.rados.rados.0.smithi151.stdout:2751: finishing write tid 4 to smithi15186934-40 2024-09-13T14:32:35.637 INFO:tasks.rados.rados.0.smithi151.stdout:2751: finishing write tid 5 to smithi15186934-40 2024-09-13T14:32:35.637 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 40 v 1084 (ObjNum 1031 snap 352 seq_num 1031) dirty exists 2024-09-13T14:32:35.637 INFO:tasks.rados.rados.0.smithi151.stdout:2751: left oid 40 (ObjNum 1031 snap 352 seq_num 1031) 2024-09-13T14:32:35.637 INFO:tasks.rados.rados.0.smithi151.stdout:2751: done (1 left) 2024-09-13T14:32:35.637 INFO:tasks.rados.rados.0.smithi151.stdout:2753: done (0 left) 2024-09-13T14:32:35.637 INFO:tasks.rados.rados.0.smithi151.stdout:2754: snap_remove snap 351 2024-09-13T14:32:36.599 INFO:tasks.rados.rados.0.smithi151.stdout:2754: done (0 left) 2024-09-13T14:32:36.599 INFO:tasks.rados.rados.0.smithi151.stdout:2755: snap_remove snap 345 2024-09-13T14:32:36.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:36 smithi184 ceph-mon[92527]: pgmap v765: 89 pgs: 89 active+clean; 157 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.9 MiB/s rd, 648 KiB/s wr, 6 op/s 2024-09-13T14:32:36.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:36 smithi184 ceph-mon[92527]: osdmap e765: 8 total, 8 up, 8 in 2024-09-13T14:32:36.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:36 smithi151 ceph-mon[104450]: pgmap v765: 89 pgs: 89 active+clean; 157 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.9 MiB/s rd, 648 KiB/s wr, 6 op/s 2024-09-13T14:32:36.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:36 smithi151 ceph-mon[104450]: osdmap e765: 8 total, 8 up, 8 in 2024-09-13T14:32:36.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:36 smithi151 ceph-mon[108966]: pgmap v765: 89 pgs: 89 active+clean; 157 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.9 MiB/s rd, 648 KiB/s wr, 6 op/s 2024-09-13T14:32:36.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:36 smithi151 ceph-mon[108966]: osdmap e765: 8 total, 8 up, 8 in 2024-09-13T14:32:37.612 INFO:tasks.rados.rados.0.smithi151.stdout:2755: done (0 left) 2024-09-13T14:32:37.612 INFO:tasks.rados.rados.0.smithi151.stdout:2756: snap_create 2024-09-13T14:32:37.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:37 smithi184 ceph-mon[92527]: osdmap e766: 8 total, 8 up, 8 in 2024-09-13T14:32:37.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:37 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:32:37.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:37 smithi151 ceph-mon[104450]: osdmap e766: 8 total, 8 up, 8 in 2024-09-13T14:32:37.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:37 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:32:37.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:37 smithi151 ceph-mon[108966]: osdmap e766: 8 total, 8 up, 8 in 2024-09-13T14:32:37.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:37 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:32:38.608 INFO:tasks.rados.rados.0.smithi151.stdout:2756: done (0 left) 2024-09-13T14:32:38.608 INFO:tasks.rados.rados.0.smithi151.stdout:2757: read oid 24 snap -1 2024-09-13T14:32:38.608 INFO:tasks.rados.rados.0.smithi151.stdout:2757: expect (ObjNum 1023 snap 350 seq_num 1023) 2024-09-13T14:32:38.609 INFO:tasks.rados.rados.0.smithi151.stdout:2758: snap_remove snap 331 2024-09-13T14:32:38.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:38 smithi151 ceph-mon[104450]: pgmap v768: 89 pgs: 89 active+clean; 163 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 2.1 MiB/s wr, 21 op/s 2024-09-13T14:32:38.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:38 smithi151 ceph-mon[104450]: osdmap e767: 8 total, 8 up, 8 in 2024-09-13T14:32:38.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:38 smithi151 ceph-mon[108966]: pgmap v768: 89 pgs: 89 active+clean; 163 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 2.1 MiB/s wr, 21 op/s 2024-09-13T14:32:38.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:38 smithi151 ceph-mon[108966]: osdmap e767: 8 total, 8 up, 8 in 2024-09-13T14:32:39.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:38 smithi184 ceph-mon[92527]: pgmap v768: 89 pgs: 89 active+clean; 163 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 2.1 MiB/s wr, 21 op/s 2024-09-13T14:32:39.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:38 smithi184 ceph-mon[92527]: osdmap e767: 8 total, 8 up, 8 in 2024-09-13T14:32:39.582 INFO:tasks.rados.rados.0.smithi151.stdout:2758: done (1 left) 2024-09-13T14:32:39.582 INFO:tasks.rados.rados.0.smithi151.stdout:2759: write oid 41 current snap is 353 2024-09-13T14:32:39.582 INFO:tasks.rados.rados.0.smithi151.stdout:2759: seq_num 1034 ranges {578438=431474,1657087=493694,2778551=599887} 2024-09-13T14:32:39.588 INFO:tasks.rados.rados.0.smithi151.stdout:2759: writing smithi15186934-41 from 578438 to 1009912 tid 1 2024-09-13T14:32:39.598 INFO:tasks.rados.rados.0.smithi151.stdout:2759: writing smithi15186934-41 from 1657087 to 2150781 tid 2 2024-09-13T14:32:39.605 INFO:tasks.rados.rados.0.smithi151.stdout:2759: writing smithi15186934-41 from 2778551 to 3378438 tid 3 2024-09-13T14:32:39.605 INFO:tasks.rados.rados.0.smithi151.stdout:2760: delete oid 8 current snap is 353 2024-09-13T14:32:39.605 INFO:tasks.rados.rados.0.smithi151.stdout:2757: expect (ObjNum 1023 snap 350 seq_num 1023) 2024-09-13T14:32:39.744 INFO:tasks.rados.rados.0.smithi151.stdout:2759: finishing write tid 1 to smithi15186934-41 2024-09-13T14:32:39.744 INFO:tasks.rados.rados.0.smithi151.stdout:2759: finishing write tid 2 to smithi15186934-41 2024-09-13T14:32:39.744 INFO:tasks.rados.rados.0.smithi151.stdout:2759: finishing write tid 3 to smithi15186934-41 2024-09-13T14:32:39.744 INFO:tasks.rados.rados.0.smithi151.stdout:2759: finishing write tid 4 to smithi15186934-41 2024-09-13T14:32:39.744 INFO:tasks.rados.rados.0.smithi151.stdout:2759: finishing write tid 5 to smithi15186934-41 2024-09-13T14:32:39.745 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 232 (ObjNum 1034 snap 353 seq_num 1034) dirty exists 2024-09-13T14:32:39.745 INFO:tasks.rados.rados.0.smithi151.stdout:2759: left oid 41 (ObjNum 1034 snap 353 seq_num 1034) 2024-09-13T14:32:39.745 INFO:tasks.rados.rados.0.smithi151.stdout:2757: done (2 left) 2024-09-13T14:32:39.745 INFO:tasks.rados.rados.0.smithi151.stdout:2759: done (1 left) 2024-09-13T14:32:39.745 INFO:tasks.rados.rados.0.smithi151.stdout:2760: done (0 left) 2024-09-13T14:32:39.745 INFO:tasks.rados.rados.0.smithi151.stdout:2761: read oid 41 snap -1 2024-09-13T14:32:39.745 INFO:tasks.rados.rados.0.smithi151.stdout:2761: expect (ObjNum 1034 snap 353 seq_num 1034) 2024-09-13T14:32:39.745 INFO:tasks.rados.rados.0.smithi151.stdout:2762: read oid 4 snap -1 2024-09-13T14:32:39.745 INFO:tasks.rados.rados.0.smithi151.stdout:2762: expect deleted 2024-09-13T14:32:39.746 INFO:tasks.rados.rados.0.smithi151.stdout:2763: rollback oid 45 current snap is 353 2024-09-13T14:32:39.746 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 45 to 350 2024-09-13T14:32:39.746 INFO:tasks.rados.rados.0.smithi151.stdout:2764: snap_create 2024-09-13T14:32:39.749 INFO:tasks.rados.rados.0.smithi151.stdout:2763: finishing rollback tid 0 to smithi15186934-45 2024-09-13T14:32:39.749 INFO:tasks.rados.rados.0.smithi151.stdout:2763: finishing rollback tid 1 to smithi15186934-45 2024-09-13T14:32:39.749 INFO:tasks.rados.rados.0.smithi151.stdout:2763: finishing rollback tid 2 to smithi15186934-45 2024-09-13T14:32:39.749 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 1381 (ObjNum 965 snap 330 seq_num 965) dirty exists 2024-09-13T14:32:39.755 INFO:tasks.rados.rados.0.smithi151.stdout:2761: expect (ObjNum 1034 snap 353 seq_num 1034) 2024-09-13T14:32:39.957 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:39 smithi151 ceph-mon[104450]: osdmap e768: 8 total, 8 up, 8 in 2024-09-13T14:32:39.958 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:39 smithi151 ceph-mon[104450]: osdmap e769: 8 total, 8 up, 8 in 2024-09-13T14:32:39.958 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:39 smithi151 ceph-mon[108966]: osdmap e768: 8 total, 8 up, 8 in 2024-09-13T14:32:39.958 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:39 smithi151 ceph-mon[108966]: osdmap e769: 8 total, 8 up, 8 in 2024-09-13T14:32:39.980 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:39 smithi184 ceph-mon[92527]: osdmap e768: 8 total, 8 up, 8 in 2024-09-13T14:32:39.980 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:39 smithi184 ceph-mon[92527]: osdmap e769: 8 total, 8 up, 8 in 2024-09-13T14:32:40.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:32:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:32:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:32:40.627 INFO:tasks.rados.rados.0.smithi151.stdout:2761: done (3 left) 2024-09-13T14:32:40.627 INFO:tasks.rados.rados.0.smithi151.stdout:2762: done (2 left) 2024-09-13T14:32:40.627 INFO:tasks.rados.rados.0.smithi151.stdout:2763: done (1 left) 2024-09-13T14:32:40.627 INFO:tasks.rados.rados.0.smithi151.stdout:2764: done (0 left) 2024-09-13T14:32:40.627 INFO:tasks.rados.rados.0.smithi151.stdout:2765: read oid 42 snap -1 2024-09-13T14:32:40.627 INFO:tasks.rados.rados.0.smithi151.stdout:2765: expect deleted 2024-09-13T14:32:40.627 INFO:tasks.rados.rados.0.smithi151.stdout:2766: read oid 29 snap -1 2024-09-13T14:32:40.627 INFO:tasks.rados.rados.0.smithi151.stdout:2766: expect deleted 2024-09-13T14:32:40.627 INFO:tasks.rados.rados.0.smithi151.stdout:2767: read oid 21 snap -1 2024-09-13T14:32:40.627 INFO:tasks.rados.rados.0.smithi151.stdout:2767: expect (ObjNum 937 snap 323 seq_num 937) 2024-09-13T14:32:40.627 INFO:tasks.rados.rados.0.smithi151.stdout:2768: read oid 45 snap -1 2024-09-13T14:32:40.628 INFO:tasks.rados.rados.0.smithi151.stdout:2768: expect (ObjNum 965 snap 330 seq_num 965) 2024-09-13T14:32:40.628 INFO:tasks.rados.rados.0.smithi151.stdout:2769: rollback oid 34 current snap is 354 2024-09-13T14:32:40.628 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 34 to 353 2024-09-13T14:32:40.628 INFO:tasks.rados.rados.0.smithi151.stdout:2770: snap_remove snap 349 2024-09-13T14:32:40.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:40 smithi184 ceph-mon[92527]: pgmap v771: 89 pgs: 89 active+clean; 161 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 2.1 MiB/s wr, 20 op/s 2024-09-13T14:32:40.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:40 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:32:40.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:40 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:32:41.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:40 smithi151 ceph-mon[104450]: pgmap v771: 89 pgs: 89 active+clean; 161 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 2.1 MiB/s wr, 20 op/s 2024-09-13T14:32:41.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:40 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:32:41.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:40 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:32:41.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:40 smithi151 ceph-mon[108966]: pgmap v771: 89 pgs: 89 active+clean; 161 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 2.1 MiB/s wr, 20 op/s 2024-09-13T14:32:41.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:40 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:32:41.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:40 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:32:41.631 INFO:tasks.rados.rados.0.smithi151.stdout:2770: done (5 left) 2024-09-13T14:32:41.631 INFO:tasks.rados.rados.0.smithi151.stdout:2771: read oid 28 snap -1 2024-09-13T14:32:41.631 INFO:tasks.rados.rados.0.smithi151.stdout:2771: expect deleted 2024-09-13T14:32:41.631 INFO:tasks.rados.rados.0.smithi151.stdout:2772: write oid 41 current snap is 354 2024-09-13T14:32:41.631 INFO:tasks.rados.rados.0.smithi151.stdout:2772: seq_num 1036 ranges {612052=581972,1657698=497363,2717044=695008} 2024-09-13T14:32:41.639 INFO:tasks.rados.rados.0.smithi151.stdout:2772: writing smithi15186934-41 from 612052 to 1194024 tid 1 2024-09-13T14:32:41.644 INFO:tasks.rados.rados.0.smithi151.stdout:2772: writing smithi15186934-41 from 1657698 to 2155061 tid 2 2024-09-13T14:32:41.651 INFO:tasks.rados.rados.0.smithi151.stdout:2772: writing smithi15186934-41 from 2717044 to 3412052 tid 3 2024-09-13T14:32:41.651 INFO:tasks.rados.rados.0.smithi151.stdout:2773: write oid 47 current snap is 354 2024-09-13T14:32:41.651 INFO:tasks.rados.rados.0.smithi151.stdout:2773: seq_num 1037 ranges {628859=657221,1716180=641021,3144467=284392} 2024-09-13T14:32:41.658 INFO:tasks.rados.rados.0.smithi151.stdout:2773: writing smithi15186934-47 from 628859 to 1286080 tid 1 2024-09-13T14:32:41.664 INFO:tasks.rados.rados.0.smithi151.stdout:2773: writing smithi15186934-47 from 1716180 to 2357201 tid 2 2024-09-13T14:32:41.670 INFO:tasks.rados.rados.0.smithi151.stdout:2773: writing smithi15186934-47 from 3144467 to 3428859 tid 3 2024-09-13T14:32:41.671 INFO:tasks.rados.rados.0.smithi151.stdout:2774: rollback oid 2 current snap is 354 2024-09-13T14:32:41.671 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 2 to 342 2024-09-13T14:32:41.671 INFO:tasks.rados.rados.0.smithi151.stdout:2765: done (8 left) 2024-09-13T14:32:41.671 INFO:tasks.rados.rados.0.smithi151.stdout:2766: done (7 left) 2024-09-13T14:32:41.671 INFO:tasks.rados.rados.0.smithi151.stdout:2775: rollback oid 23 current snap is 354 2024-09-13T14:32:41.671 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 23 to 353 2024-09-13T14:32:41.671 INFO:tasks.rados.rados.0.smithi151.stdout:2776: read oid 18 snap -1 2024-09-13T14:32:41.671 INFO:tasks.rados.rados.0.smithi151.stdout:2776: expect deleted 2024-09-13T14:32:41.671 INFO:tasks.rados.rados.0.smithi151.stdout:2777: write oid 3 current snap is 354 2024-09-13T14:32:41.671 INFO:tasks.rados.rados.0.smithi151.stdout:2769: finishing rollback tid 0 to smithi15186934-34 2024-09-13T14:32:41.671 INFO:tasks.rados.rados.0.smithi151.stdout:2769: finishing rollback tid 1 to smithi15186934-34 2024-09-13T14:32:41.671 INFO:tasks.rados.rados.0.smithi151.stdout:2769: finishing rollback tid 2 to smithi15186934-34 2024-09-13T14:32:41.671 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 1090 (ObjNum 970 snap 335 seq_num 970) dirty exists 2024-09-13T14:32:41.671 INFO:tasks.rados.rados.0.smithi151.stdout:2768: expect (ObjNum 965 snap 330 seq_num 965) 2024-09-13T14:32:41.771 INFO:tasks.rados.rados.0.smithi151.stdout:2767: expect (ObjNum 937 snap 323 seq_num 937) 2024-09-13T14:32:41.894 INFO:tasks.rados.rados.0.smithi151.stdout:2772: finishing write tid 1 to smithi15186934-41 2024-09-13T14:32:41.894 INFO:tasks.rados.rados.0.smithi151.stdout:2772: finishing write tid 2 to smithi15186934-41 2024-09-13T14:32:41.895 INFO:tasks.rados.rados.0.smithi151.stdout:2772: finishing write tid 3 to smithi15186934-41 2024-09-13T14:32:41.895 INFO:tasks.rados.rados.0.smithi151.stdout:2772: finishing write tid 4 to smithi15186934-41 2024-09-13T14:32:41.895 INFO:tasks.rados.rados.0.smithi151.stdout:2772: finishing write tid 5 to smithi15186934-41 2024-09-13T14:32:41.895 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 239 (ObjNum 1036 snap 354 seq_num 1036) dirty exists 2024-09-13T14:32:41.895 INFO:tasks.rados.rados.0.smithi151.stdout:2772: left oid 41 (ObjNum 1036 snap 354 seq_num 1036) 2024-09-13T14:32:41.895 INFO:tasks.rados.rados.0.smithi151.stdout:2773: finishing write tid 1 to smithi15186934-47 2024-09-13T14:32:41.895 INFO:tasks.rados.rados.0.smithi151.stdout:2777: seq_num 1038 ranges {645666=732470,2174662=501032,3288243=157423} 2024-09-13T14:32:41.902 INFO:tasks.rados.rados.0.smithi151.stdout:2777: writing smithi15186934-3 from 645666 to 1378136 tid 1 2024-09-13T14:32:41.909 INFO:tasks.rados.rados.0.smithi151.stdout:2777: writing smithi15186934-3 from 2174662 to 2675694 tid 2 2024-09-13T14:32:41.914 INFO:tasks.rados.rados.0.smithi151.stdout:2777: writing smithi15186934-3 from 3288243 to 3445666 tid 3 2024-09-13T14:32:41.914 INFO:tasks.rados.rados.0.smithi151.stdout:2767: done (9 left) 2024-09-13T14:32:41.914 INFO:tasks.rados.rados.0.smithi151.stdout:2768: done (8 left) 2024-09-13T14:32:41.914 INFO:tasks.rados.rados.0.smithi151.stdout:2769: done (7 left) 2024-09-13T14:32:41.914 INFO:tasks.rados.rados.0.smithi151.stdout:2771: done (6 left) 2024-09-13T14:32:41.914 INFO:tasks.rados.rados.0.smithi151.stdout:2772: done (5 left) 2024-09-13T14:32:41.915 INFO:tasks.rados.rados.0.smithi151.stdout:2778: read oid 37 snap -1 2024-09-13T14:32:41.915 INFO:tasks.rados.rados.0.smithi151.stdout:2778: expect (ObjNum 1004 snap 345 seq_num 1004) 2024-09-13T14:32:41.915 INFO:tasks.rados.rados.0.smithi151.stdout:2779: write oid 36 current snap is 354 2024-09-13T14:32:41.915 INFO:tasks.rados.rados.0.smithi151.stdout:2773: finishing write tid 2 to smithi15186934-47 2024-09-13T14:32:41.915 INFO:tasks.rados.rados.0.smithi151.stdout:2774: finishing rollback tid 1 to smithi15186934-2 2024-09-13T14:32:41.915 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 2 v 0 (ObjNum 81 snap 0 seq_num 0) dirty dne 2024-09-13T14:32:41.915 INFO:tasks.rados.rados.0.smithi151.stdout:2773: finishing write tid 3 to smithi15186934-47 2024-09-13T14:32:41.915 INFO:tasks.rados.rados.0.smithi151.stdout:2773: finishing write tid 4 to smithi15186934-47 2024-09-13T14:32:41.915 INFO:tasks.rados.rados.0.smithi151.stdout:2773: finishing write tid 5 to smithi15186934-47 2024-09-13T14:32:41.915 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 1229 (ObjNum 1037 snap 354 seq_num 1037) dirty exists 2024-09-13T14:32:41.915 INFO:tasks.rados.rados.0.smithi151.stdout:2773: left oid 47 (ObjNum 1037 snap 354 seq_num 1037) 2024-09-13T14:32:41.915 INFO:tasks.rados.rados.0.smithi151.stdout:2775: finishing rollback tid 1 to smithi15186934-23 2024-09-13T14:32:41.915 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 23 v 0 (ObjNum 129 snap 0 seq_num 3092543760) dirty dne 2024-09-13T14:32:41.916 INFO:tasks.rados.rados.0.smithi151.stdout:2779: seq_num 1039 ranges {662473=407719,1833144=644690,3199313=263160} 2024-09-13T14:32:41.919 INFO:tasks.rados.rados.0.smithi151.stdout:2779: writing smithi15186934-36 from 662473 to 1070192 tid 1 2024-09-13T14:32:41.927 INFO:tasks.rados.rados.0.smithi151.stdout:2779: writing smithi15186934-36 from 1833144 to 2477834 tid 2 2024-09-13T14:32:41.933 INFO:tasks.rados.rados.0.smithi151.stdout:2779: writing smithi15186934-36 from 3199313 to 3462473 tid 3 2024-09-13T14:32:41.933 INFO:tasks.rados.rados.0.smithi151.stdout:2773: done (6 left) 2024-09-13T14:32:41.933 INFO:tasks.rados.rados.0.smithi151.stdout:2774: done (5 left) 2024-09-13T14:32:41.933 INFO:tasks.rados.rados.0.smithi151.stdout:2775: done (4 left) 2024-09-13T14:32:41.933 INFO:tasks.rados.rados.0.smithi151.stdout:2776: done (3 left) 2024-09-13T14:32:41.933 INFO:tasks.rados.rados.0.smithi151.stdout:2780: write oid 26 current snap is 354 2024-09-13T14:32:41.933 INFO:tasks.rados.rados.0.smithi151.stdout:2777: finishing write tid 1 to smithi15186934-3 2024-09-13T14:32:41.933 INFO:tasks.rados.rados.0.smithi151.stdout:2777: finishing write tid 2 to smithi15186934-3 2024-09-13T14:32:41.934 INFO:tasks.rados.rados.0.smithi151.stdout:2780: seq_num 1040 ranges {679280=482968,1775273=788348,3110383=368897} 2024-09-13T14:32:41.939 INFO:tasks.rados.rados.0.smithi151.stdout:2780: writing smithi15186934-26 from 679280 to 1162248 tid 1 2024-09-13T14:32:41.947 INFO:tasks.rados.rados.0.smithi151.stdout:2780: writing smithi15186934-26 from 1775273 to 2563621 tid 2 2024-09-13T14:32:41.953 INFO:tasks.rados.rados.0.smithi151.stdout:2780: writing smithi15186934-26 from 3110383 to 3479280 tid 3 2024-09-13T14:32:41.953 INFO:tasks.rados.rados.0.smithi151.stdout:2781: read oid 30 snap -1 2024-09-13T14:32:41.953 INFO:tasks.rados.rados.0.smithi151.stdout:2781: expect (ObjNum 1022 snap 350 seq_num 1022) 2024-09-13T14:32:41.953 INFO:tasks.rados.rados.0.smithi151.stdout:2782: rollback oid 23 current snap is 354 2024-09-13T14:32:41.953 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 23 to 353 2024-09-13T14:32:41.953 INFO:tasks.rados.rados.0.smithi151.stdout:2777: finishing write tid 3 to smithi15186934-3 2024-09-13T14:32:41.953 INFO:tasks.rados.rados.0.smithi151.stdout:2777: finishing write tid 4 to smithi15186934-3 2024-09-13T14:32:41.953 INFO:tasks.rados.rados.0.smithi151.stdout:2777: finishing write tid 5 to smithi15186934-3 2024-09-13T14:32:41.953 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 1097 (ObjNum 1038 snap 354 seq_num 1038) dirty exists 2024-09-13T14:32:41.953 INFO:tasks.rados.rados.0.smithi151.stdout:2777: left oid 3 (ObjNum 1038 snap 354 seq_num 1038) 2024-09-13T14:32:41.953 INFO:tasks.rados.rados.0.smithi151.stdout:2777: done (5 left) 2024-09-13T14:32:41.954 INFO:tasks.rados.rados.0.smithi151.stdout:2783: read oid 41 snap -1 2024-09-13T14:32:41.954 INFO:tasks.rados.rados.0.smithi151.stdout:2783: expect (ObjNum 1036 snap 354 seq_num 1036) 2024-09-13T14:32:41.954 INFO:tasks.rados.rados.0.smithi151.stdout:2784: snap_create 2024-09-13T14:32:41.954 INFO:tasks.rados.rados.0.smithi151.stdout:2778: expect (ObjNum 1004 snap 345 seq_num 1004) 2024-09-13T14:32:42.049 INFO:tasks.rados.rados.0.smithi151.stdout:2779: finishing write tid 1 to smithi15186934-36 2024-09-13T14:32:42.049 INFO:tasks.rados.rados.0.smithi151.stdout:2779: finishing write tid 2 to smithi15186934-36 2024-09-13T14:32:42.049 INFO:tasks.rados.rados.0.smithi151.stdout:2779: finishing write tid 3 to smithi15186934-36 2024-09-13T14:32:42.049 INFO:tasks.rados.rados.0.smithi151.stdout:2779: finishing write tid 4 to smithi15186934-36 2024-09-13T14:32:42.049 INFO:tasks.rados.rados.0.smithi151.stdout:2779: finishing write tid 5 to smithi15186934-36 2024-09-13T14:32:42.049 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 36 v 624 (ObjNum 1039 snap 354 seq_num 1039) dirty exists 2024-09-13T14:32:42.049 INFO:tasks.rados.rados.0.smithi151.stdout:2779: left oid 36 (ObjNum 1039 snap 354 seq_num 1039) 2024-09-13T14:32:42.050 INFO:tasks.rados.rados.0.smithi151.stdout:2780: finishing write tid 1 to smithi15186934-26 2024-09-13T14:32:42.050 INFO:tasks.rados.rados.0.smithi151.stdout:2780: finishing write tid 2 to smithi15186934-26 2024-09-13T14:32:42.050 INFO:tasks.rados.rados.0.smithi151.stdout:2780: finishing write tid 3 to smithi15186934-26 2024-09-13T14:32:42.050 INFO:tasks.rados.rados.0.smithi151.stdout:2780: finishing write tid 4 to smithi15186934-26 2024-09-13T14:32:42.050 INFO:tasks.rados.rados.0.smithi151.stdout:2782: finishing rollback tid 1 to smithi15186934-23 2024-09-13T14:32:42.050 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 23 v 0 (ObjNum 129 snap 0 seq_num 3092543760) dirty dne 2024-09-13T14:32:42.050 INFO:tasks.rados.rados.0.smithi151.stdout:2780: finishing write tid 5 to smithi15186934-26 2024-09-13T14:32:42.050 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 1004 (ObjNum 1040 snap 354 seq_num 1040) dirty exists 2024-09-13T14:32:42.050 INFO:tasks.rados.rados.0.smithi151.stdout:2780: left oid 26 (ObjNum 1040 snap 354 seq_num 1040) 2024-09-13T14:32:42.050 INFO:tasks.rados.rados.0.smithi151.stdout:2781: expect (ObjNum 1022 snap 350 seq_num 1022) 2024-09-13T14:32:42.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:41 smithi184 ceph-mon[92527]: osdmap e770: 8 total, 8 up, 8 in 2024-09-13T14:32:42.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:41 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:32:42.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:41 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:32:42.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:41 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:32:42.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:41 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:32:42.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:41 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:32:42.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:41 smithi151 ceph-mon[104450]: osdmap e770: 8 total, 8 up, 8 in 2024-09-13T14:32:42.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:41 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:32:42.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:41 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:32:42.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:41 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:32:42.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:41 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:32:42.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:41 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:32:42.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:41 smithi151 ceph-mon[108966]: osdmap e770: 8 total, 8 up, 8 in 2024-09-13T14:32:42.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:41 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:32:42.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:41 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:32:42.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:41 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:32:42.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:41 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:32:42.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:41 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:32:42.156 INFO:tasks.rados.rados.0.smithi151.stdout:2783: expect (ObjNum 1036 snap 354 seq_num 1036) 2024-09-13T14:32:42.636 INFO:tasks.rados.rados.0.smithi151.stdout:2778: done (6 left) 2024-09-13T14:32:42.637 INFO:tasks.rados.rados.0.smithi151.stdout:2779: done (5 left) 2024-09-13T14:32:42.637 INFO:tasks.rados.rados.0.smithi151.stdout:2780: done (4 left) 2024-09-13T14:32:42.637 INFO:tasks.rados.rados.0.smithi151.stdout:2781: done (3 left) 2024-09-13T14:32:42.637 INFO:tasks.rados.rados.0.smithi151.stdout:2782: done (2 left) 2024-09-13T14:32:42.637 INFO:tasks.rados.rados.0.smithi151.stdout:2783: done (1 left) 2024-09-13T14:32:42.637 INFO:tasks.rados.rados.0.smithi151.stdout:2784: done (0 left) 2024-09-13T14:32:42.637 INFO:tasks.rados.rados.0.smithi151.stdout:2785: delete oid 30 current snap is 355 2024-09-13T14:32:42.639 INFO:tasks.rados.rados.0.smithi151.stdout:2785: done (0 left) 2024-09-13T14:32:42.639 INFO:tasks.rados.rados.0.smithi151.stdout:2786: rollback oid 50 current snap is 355 2024-09-13T14:32:42.639 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 50 to 342 2024-09-13T14:32:42.639 INFO:tasks.rados.rados.0.smithi151.stdout:2787: read oid 9 snap -1 2024-09-13T14:32:42.640 INFO:tasks.rados.rados.0.smithi151.stdout:2787: expect deleted 2024-09-13T14:32:42.640 INFO:tasks.rados.rados.0.smithi151.stdout:2788: write oid 23 current snap is 355 2024-09-13T14:32:42.640 INFO:tasks.rados.rados.0.smithi151.stdout:2788: seq_num 1042 ranges {712894=749819,2008590=792017,3281582=231312} 2024-09-13T14:32:42.647 INFO:tasks.rados.rados.0.smithi151.stdout:2788: writing smithi15186934-23 from 712894 to 1462713 tid 1 2024-09-13T14:32:42.654 INFO:tasks.rados.rados.0.smithi151.stdout:2788: writing smithi15186934-23 from 2008590 to 2800607 tid 2 2024-09-13T14:32:42.658 INFO:tasks.rados.rados.0.smithi151.stdout:2788: writing smithi15186934-23 from 3281582 to 3512894 tid 3 2024-09-13T14:32:42.658 INFO:tasks.rados.rados.0.smithi151.stdout:2789: write oid 13 current snap is 355 2024-09-13T14:32:42.658 INFO:tasks.rados.rados.0.smithi151.stdout:2789: seq_num 1043 ranges {729701=425068,1667072=652028,2909005=485180} 2024-09-13T14:32:42.664 INFO:tasks.rados.rados.0.smithi151.stdout:2789: writing smithi15186934-13 from 729701 to 1154769 tid 1 2024-09-13T14:32:42.671 INFO:tasks.rados.rados.0.smithi151.stdout:2789: writing smithi15186934-13 from 1667072 to 2319100 tid 2 2024-09-13T14:32:42.678 INFO:tasks.rados.rados.0.smithi151.stdout:2789: writing smithi15186934-13 from 2909005 to 3394185 tid 3 2024-09-13T14:32:42.678 INFO:tasks.rados.rados.0.smithi151.stdout:2790: rollback oid 20 current snap is 355 2024-09-13T14:32:42.678 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 20 to 354 2024-09-13T14:32:42.678 INFO:tasks.rados.rados.0.smithi151.stdout:2786: finishing rollback tid 1 to smithi15186934-50 2024-09-13T14:32:42.678 INFO:tasks.rados.rados.0.smithi151.stdout:2786: finishing rollback tid 2 to smithi15186934-50 2024-09-13T14:32:42.678 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 50 v 1127 (ObjNum 905 snap 314 seq_num 905) dirty exists 2024-09-13T14:32:42.678 INFO:tasks.rados.rados.0.smithi151.stdout:2788: finishing write tid 1 to smithi15186934-23 2024-09-13T14:32:42.678 INFO:tasks.rados.rados.0.smithi151.stdout:2788: finishing write tid 2 to smithi15186934-23 2024-09-13T14:32:42.678 INFO:tasks.rados.rados.0.smithi151.stdout:2788: finishing write tid 3 to smithi15186934-23 2024-09-13T14:32:42.678 INFO:tasks.rados.rados.0.smithi151.stdout:2788: finishing write tid 4 to smithi15186934-23 2024-09-13T14:32:42.678 INFO:tasks.rados.rados.0.smithi151.stdout:2788: finishing write tid 5 to smithi15186934-23 2024-09-13T14:32:42.678 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 23 v 1234 (ObjNum 1042 snap 355 seq_num 1042) dirty exists 2024-09-13T14:32:42.679 INFO:tasks.rados.rados.0.smithi151.stdout:2788: left oid 23 (ObjNum 1042 snap 355 seq_num 1042) 2024-09-13T14:32:42.679 INFO:tasks.rados.rados.0.smithi151.stdout:2789: finishing write tid 1 to smithi15186934-13 2024-09-13T14:32:42.679 INFO:tasks.rados.rados.0.smithi151.stdout:2789: finishing write tid 2 to smithi15186934-13 2024-09-13T14:32:42.679 INFO:tasks.rados.rados.0.smithi151.stdout:2786: done (4 left) 2024-09-13T14:32:42.679 INFO:tasks.rados.rados.0.smithi151.stdout:2787: done (3 left) 2024-09-13T14:32:42.679 INFO:tasks.rados.rados.0.smithi151.stdout:2788: done (2 left) 2024-09-13T14:32:42.679 INFO:tasks.rados.rados.0.smithi151.stdout:2791: write oid 42 current snap is 355 2024-09-13T14:32:42.679 INFO:tasks.rados.rados.0.smithi151.stdout:2791: seq_num 1044 ranges {746508=500317,2009201=795686,3220075=326433} 2024-09-13T14:32:42.684 INFO:tasks.rados.rados.0.smithi151.stdout:2791: writing smithi15186934-42 from 746508 to 1246825 tid 1 2024-09-13T14:32:42.692 INFO:tasks.rados.rados.0.smithi151.stdout:2791: writing smithi15186934-42 from 2009201 to 2804887 tid 2 2024-09-13T14:32:42.696 INFO:tasks.rados.rados.0.smithi151.stdout:2791: writing smithi15186934-42 from 3220075 to 3546508 tid 3 2024-09-13T14:32:42.697 INFO:tasks.rados.rados.0.smithi151.stdout:2792: snap_remove snap 342 2024-09-13T14:32:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:42 smithi184 ceph-mon[92527]: pgmap v774: 89 pgs: 89 active+clean; 161 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 371 KiB/s wr, 1 op/s 2024-09-13T14:32:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:42 smithi184 ceph-mon[92527]: osdmap e771: 8 total, 8 up, 8 in 2024-09-13T14:32:43.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:42 smithi151 ceph-mon[104450]: pgmap v774: 89 pgs: 89 active+clean; 161 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 371 KiB/s wr, 1 op/s 2024-09-13T14:32:43.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:42 smithi151 ceph-mon[104450]: osdmap e771: 8 total, 8 up, 8 in 2024-09-13T14:32:43.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:42 smithi151 ceph-mon[108966]: pgmap v774: 89 pgs: 89 active+clean; 161 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 371 KiB/s wr, 1 op/s 2024-09-13T14:32:43.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:42 smithi151 ceph-mon[108966]: osdmap e771: 8 total, 8 up, 8 in 2024-09-13T14:32:43.644 INFO:tasks.rados.rados.0.smithi151.stdout:2790: finishing rollback tid 0 to smithi15186934-20 2024-09-13T14:32:43.644 INFO:tasks.rados.rados.0.smithi151.stdout:2789: finishing write tid 3 to smithi15186934-13 2024-09-13T14:32:43.645 INFO:tasks.rados.rados.0.smithi151.stdout:2789: finishing write tid 4 to smithi15186934-13 2024-09-13T14:32:43.645 INFO:tasks.rados.rados.0.smithi151.stdout:2789: finishing write tid 5 to smithi15186934-13 2024-09-13T14:32:43.645 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 13 v 1038 (ObjNum 1043 snap 355 seq_num 1043) dirty exists 2024-09-13T14:32:43.645 INFO:tasks.rados.rados.0.smithi151.stdout:2789: left oid 13 (ObjNum 1043 snap 355 seq_num 1043) 2024-09-13T14:32:43.645 INFO:tasks.rados.rados.0.smithi151.stdout:2790: finishing rollback tid 1 to smithi15186934-20 2024-09-13T14:32:43.645 INFO:tasks.rados.rados.0.smithi151.stdout:2790: finishing rollback tid 2 to smithi15186934-20 2024-09-13T14:32:43.645 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 725 (ObjNum 997 snap 344 seq_num 997) dirty exists 2024-09-13T14:32:43.645 INFO:tasks.rados.rados.0.smithi151.stdout:2791: finishing write tid 1 to smithi15186934-42 2024-09-13T14:32:43.645 INFO:tasks.rados.rados.0.smithi151.stdout:2791: finishing write tid 2 to smithi15186934-42 2024-09-13T14:32:43.645 INFO:tasks.rados.rados.0.smithi151.stdout:2791: finishing write tid 3 to smithi15186934-42 2024-09-13T14:32:43.645 INFO:tasks.rados.rados.0.smithi151.stdout:2791: finishing write tid 4 to smithi15186934-42 2024-09-13T14:32:43.645 INFO:tasks.rados.rados.0.smithi151.stdout:2791: finishing write tid 5 to smithi15186934-42 2024-09-13T14:32:43.645 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 1049 (ObjNum 1044 snap 355 seq_num 1044) dirty exists 2024-09-13T14:32:43.646 INFO:tasks.rados.rados.0.smithi151.stdout:2791: left oid 42 (ObjNum 1044 snap 355 seq_num 1044) 2024-09-13T14:32:43.646 INFO:tasks.rados.rados.0.smithi151.stdout:2789: done (3 left) 2024-09-13T14:32:43.646 INFO:tasks.rados.rados.0.smithi151.stdout:2790: done (2 left) 2024-09-13T14:32:43.646 INFO:tasks.rados.rados.0.smithi151.stdout:2791: done (1 left) 2024-09-13T14:32:43.646 INFO:tasks.rados.rados.0.smithi151.stdout:2792: done (0 left) 2024-09-13T14:32:43.646 INFO:tasks.rados.rados.0.smithi151.stdout:2793: snap_remove snap 341 2024-09-13T14:32:44.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:43 smithi184 ceph-mon[92527]: osdmap e772: 8 total, 8 up, 8 in 2024-09-13T14:32:44.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:43 smithi151 ceph-mon[104450]: osdmap e772: 8 total, 8 up, 8 in 2024-09-13T14:32:44.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:43 smithi151 ceph-mon[108966]: osdmap e772: 8 total, 8 up, 8 in 2024-09-13T14:32:44.581 INFO:tasks.rados.rados.0.smithi151.stdout:2793: done (0 left) 2024-09-13T14:32:44.581 INFO:tasks.rados.rados.0.smithi151.stdout:2794: write oid 28 current snap is 355 2024-09-13T14:32:44.581 INFO:tasks.rados.rados.0.smithi151.stdout:2794: seq_num 1045 ranges {763315=575566,2067683=655697,3247498=315817} 2024-09-13T14:32:44.589 INFO:tasks.rados.rados.0.smithi151.stdout:2794: writing smithi15186934-28 from 763315 to 1338881 tid 1 2024-09-13T14:32:44.598 INFO:tasks.rados.rados.0.smithi151.stdout:2794: writing smithi15186934-28 from 2067683 to 2723380 tid 2 2024-09-13T14:32:44.603 INFO:tasks.rados.rados.0.smithi151.stdout:2794: writing smithi15186934-28 from 3247498 to 3563315 tid 3 2024-09-13T14:32:44.603 INFO:tasks.rados.rados.0.smithi151.stdout:2795: snap_create 2024-09-13T14:32:44.603 INFO:tasks.rados.rados.0.smithi151.stdout:2794: finishing write tid 1 to smithi15186934-28 2024-09-13T14:32:44.603 INFO:tasks.rados.rados.0.smithi151.stdout:2794: finishing write tid 2 to smithi15186934-28 2024-09-13T14:32:44.605 INFO:tasks.rados.rados.0.smithi151.stdout:2794: finishing write tid 3 to smithi15186934-28 2024-09-13T14:32:44.605 INFO:tasks.rados.rados.0.smithi151.stdout:2794: finishing write tid 4 to smithi15186934-28 2024-09-13T14:32:44.606 INFO:tasks.rados.rados.0.smithi151.stdout:2794: finishing write tid 5 to smithi15186934-28 2024-09-13T14:32:44.606 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 28 v 1053 (ObjNum 1045 snap 355 seq_num 1045) dirty exists 2024-09-13T14:32:44.606 INFO:tasks.rados.rados.0.smithi151.stdout:2794: left oid 28 (ObjNum 1045 snap 355 seq_num 1045) 2024-09-13T14:32:44.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:44 smithi151 ceph-mon[104450]: pgmap v777: 89 pgs: 89 active+clean; 156 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 1.5 MiB/s wr, 19 op/s 2024-09-13T14:32:44.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:44 smithi151 ceph-mon[104450]: osdmap e773: 8 total, 8 up, 8 in 2024-09-13T14:32:44.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:44 smithi151 ceph-mon[104450]: osdmap e774: 8 total, 8 up, 8 in 2024-09-13T14:32:44.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:32:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:32:44] "GET /metrics HTTP/1.1" 200 36511 "" "Prometheus/2.51.0" 2024-09-13T14:32:44.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:44 smithi151 ceph-mon[108966]: pgmap v777: 89 pgs: 89 active+clean; 156 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 1.5 MiB/s wr, 19 op/s 2024-09-13T14:32:44.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:44 smithi151 ceph-mon[108966]: osdmap e773: 8 total, 8 up, 8 in 2024-09-13T14:32:44.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:44 smithi151 ceph-mon[108966]: osdmap e774: 8 total, 8 up, 8 in 2024-09-13T14:32:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:44 smithi184 ceph-mon[92527]: pgmap v777: 89 pgs: 89 active+clean; 156 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 1.5 MiB/s wr, 19 op/s 2024-09-13T14:32:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:44 smithi184 ceph-mon[92527]: osdmap e773: 8 total, 8 up, 8 in 2024-09-13T14:32:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:44 smithi184 ceph-mon[92527]: osdmap e774: 8 total, 8 up, 8 in 2024-09-13T14:32:45.582 INFO:tasks.rados.rados.0.smithi151.stdout:2794: done (1 left) 2024-09-13T14:32:45.582 INFO:tasks.rados.rados.0.smithi151.stdout:2795: done (0 left) 2024-09-13T14:32:45.582 INFO:tasks.rados.rados.0.smithi151.stdout:2796: delete oid 8 current snap is 356 2024-09-13T14:32:45.588 INFO:tasks.rados.rados.0.smithi151.stdout:2796: done (0 left) 2024-09-13T14:32:45.588 INFO:tasks.rados.rados.0.smithi151.stdout:2797: read oid 29 snap -1 2024-09-13T14:32:45.589 INFO:tasks.rados.rados.0.smithi151.stdout:2797: expect deleted 2024-09-13T14:32:45.589 INFO:tasks.rados.rados.0.smithi151.stdout:2798: read oid 43 snap -1 2024-09-13T14:32:45.589 INFO:tasks.rados.rados.0.smithi151.stdout:2798: expect deleted 2024-09-13T14:32:45.589 INFO:tasks.rados.rados.0.smithi151.stdout:2799: rollback oid 3 current snap is 356 2024-09-13T14:32:45.589 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 3 to 347 2024-09-13T14:32:45.589 INFO:tasks.rados.rados.0.smithi151.stdout:2800: write oid 44 current snap is 356 2024-09-13T14:32:45.589 INFO:tasks.rados.rados.0.smithi151.stdout:2800: seq_num 1047 ranges {796929=726064,2184647=659366,3302344=294585} 2024-09-13T14:32:45.597 INFO:tasks.rados.rados.0.smithi151.stdout:2800: writing smithi15186934-44 from 796929 to 1522993 tid 1 2024-09-13T14:32:45.605 INFO:tasks.rados.rados.0.smithi151.stdout:2800: writing smithi15186934-44 from 2184647 to 2844013 tid 2 2024-09-13T14:32:45.609 INFO:tasks.rados.rados.0.smithi151.stdout:2800: writing smithi15186934-44 from 3302344 to 3596929 tid 3 2024-09-13T14:32:45.609 INFO:tasks.rados.rados.0.smithi151.stdout:2801: delete oid 37 current snap is 356 2024-09-13T14:32:45.609 INFO:tasks.rados.rados.0.smithi151.stdout:2799: finishing rollback tid 0 to smithi15186934-3 2024-09-13T14:32:45.609 INFO:tasks.rados.rados.0.smithi151.stdout:2799: finishing rollback tid 1 to smithi15186934-3 2024-09-13T14:32:45.609 INFO:tasks.rados.rados.0.smithi151.stdout:2799: finishing rollback tid 2 to smithi15186934-3 2024-09-13T14:32:45.609 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 1111 (ObjNum 916 snap 317 seq_num 916) dirty exists 2024-09-13T14:32:45.609 INFO:tasks.rados.rados.0.smithi151.stdout:2800: finishing write tid 1 to smithi15186934-44 2024-09-13T14:32:45.610 INFO:tasks.rados.rados.0.smithi151.stdout:2800: finishing write tid 2 to smithi15186934-44 2024-09-13T14:32:45.611 INFO:tasks.rados.rados.0.smithi151.stdout:2797: done (4 left) 2024-09-13T14:32:45.611 INFO:tasks.rados.rados.0.smithi151.stdout:2798: done (3 left) 2024-09-13T14:32:45.611 INFO:tasks.rados.rados.0.smithi151.stdout:2799: done (2 left) 2024-09-13T14:32:45.611 INFO:tasks.rados.rados.0.smithi151.stdout:2801: done (1 left) 2024-09-13T14:32:45.611 INFO:tasks.rados.rados.0.smithi151.stdout:2802: write oid 32 current snap is 356 2024-09-13T14:32:45.611 INFO:tasks.rados.rados.0.smithi151.stdout:2802: seq_num 1049 ranges {430543=476562,1385258=663035,2840837=669165} 2024-09-13T14:32:45.616 INFO:tasks.rados.rados.0.smithi151.stdout:2802: writing smithi15186934-32 from 430543 to 907105 tid 1 2024-09-13T14:32:45.622 INFO:tasks.rados.rados.0.smithi151.stdout:2802: writing smithi15186934-32 from 1385258 to 2048293 tid 2 2024-09-13T14:32:45.630 INFO:tasks.rados.rados.0.smithi151.stdout:2802: writing smithi15186934-32 from 2840837 to 3510002 tid 3 2024-09-13T14:32:45.631 INFO:tasks.rados.rados.0.smithi151.stdout:2803: write oid 9 current snap is 356 2024-09-13T14:32:45.631 INFO:tasks.rados.rados.0.smithi151.stdout:2800: finishing write tid 3 to smithi15186934-44 2024-09-13T14:32:45.631 INFO:tasks.rados.rados.0.smithi151.stdout:2803: seq_num 1050 ranges {447350=668164,1560093=406693,2584613=480437} 2024-09-13T14:32:45.636 INFO:tasks.rados.rados.0.smithi151.stdout:2803: writing smithi15186934-9 from 447350 to 1115514 tid 1 2024-09-13T14:32:45.641 INFO:tasks.rados.rados.0.smithi151.stdout:2803: writing smithi15186934-9 from 1560093 to 1966786 tid 2 2024-09-13T14:32:45.647 INFO:tasks.rados.rados.0.smithi151.stdout:2803: writing smithi15186934-9 from 2584613 to 3065050 tid 3 2024-09-13T14:32:45.648 INFO:tasks.rados.rados.0.smithi151.stdout:2804: delete oid 19 current snap is 356 2024-09-13T14:32:45.648 INFO:tasks.rados.rados.0.smithi151.stdout:2800: finishing write tid 4 to smithi15186934-44 2024-09-13T14:32:45.648 INFO:tasks.rados.rados.0.smithi151.stdout:2800: finishing write tid 5 to smithi15186934-44 2024-09-13T14:32:45.648 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 848 (ObjNum 1047 snap 356 seq_num 1047) dirty exists 2024-09-13T14:32:45.648 INFO:tasks.rados.rados.0.smithi151.stdout:2800: left oid 44 (ObjNum 1047 snap 356 seq_num 1047) 2024-09-13T14:32:45.648 INFO:tasks.rados.rados.0.smithi151.stdout:2802: finishing write tid 1 to smithi15186934-32 2024-09-13T14:32:45.648 INFO:tasks.rados.rados.0.smithi151.stdout:2802: finishing write tid 2 to smithi15186934-32 2024-09-13T14:32:45.648 INFO:tasks.rados.rados.0.smithi151.stdout:2802: finishing write tid 3 to smithi15186934-32 2024-09-13T14:32:45.648 INFO:tasks.rados.rados.0.smithi151.stdout:2802: finishing write tid 4 to smithi15186934-32 2024-09-13T14:32:45.648 INFO:tasks.rados.rados.0.smithi151.stdout:2802: finishing write tid 5 to smithi15186934-32 2024-09-13T14:32:45.648 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 32 v 1165 (ObjNum 1049 snap 356 seq_num 1049) dirty exists 2024-09-13T14:32:45.648 INFO:tasks.rados.rados.0.smithi151.stdout:2802: left oid 32 (ObjNum 1049 snap 356 seq_num 1049) 2024-09-13T14:32:45.649 INFO:tasks.rados.rados.0.smithi151.stdout:2803: finishing write tid 1 to smithi15186934-9 2024-09-13T14:32:45.649 INFO:tasks.rados.rados.0.smithi151.stdout:2803: finishing write tid 2 to smithi15186934-9 2024-09-13T14:32:45.650 INFO:tasks.rados.rados.0.smithi151.stdout:2800: done (3 left) 2024-09-13T14:32:45.650 INFO:tasks.rados.rados.0.smithi151.stdout:2802: done (2 left) 2024-09-13T14:32:45.650 INFO:tasks.rados.rados.0.smithi151.stdout:2804: done (1 left) 2024-09-13T14:32:45.650 INFO:tasks.rados.rados.0.smithi151.stdout:2805: read oid 47 snap -1 2024-09-13T14:32:45.650 INFO:tasks.rados.rados.0.smithi151.stdout:2805: expect (ObjNum 1037 snap 354 seq_num 1037) 2024-09-13T14:32:45.650 INFO:tasks.rados.rados.0.smithi151.stdout:2806: delete oid 34 current snap is 356 2024-09-13T14:32:45.650 INFO:tasks.rados.rados.0.smithi151.stdout:2803: finishing write tid 3 to smithi15186934-9 2024-09-13T14:32:45.651 INFO:tasks.rados.rados.0.smithi151.stdout:2803: finishing write tid 4 to smithi15186934-9 2024-09-13T14:32:45.651 INFO:tasks.rados.rados.0.smithi151.stdout:2803: finishing write tid 5 to smithi15186934-9 2024-09-13T14:32:45.652 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 1169 (ObjNum 1050 snap 356 seq_num 1050) dirty exists 2024-09-13T14:32:45.652 INFO:tasks.rados.rados.0.smithi151.stdout:2803: left oid 9 (ObjNum 1050 snap 356 seq_num 1050) 2024-09-13T14:32:45.652 INFO:tasks.rados.rados.0.smithi151.stdout:2803: done (2 left) 2024-09-13T14:32:45.652 INFO:tasks.rados.rados.0.smithi151.stdout:2806: done (1 left) 2024-09-13T14:32:45.652 INFO:tasks.rados.rados.0.smithi151.stdout:2807: snap_create 2024-09-13T14:32:45.664 INFO:tasks.rados.rados.0.smithi151.stdout:2805: expect (ObjNum 1037 snap 354 seq_num 1037) 2024-09-13T14:32:46.655 INFO:tasks.rados.rados.0.smithi151.stdout:2805: done (1 left) 2024-09-13T14:32:46.655 INFO:tasks.rados.rados.0.smithi151.stdout:2807: done (0 left) 2024-09-13T14:32:46.656 INFO:tasks.rados.rados.0.smithi151.stdout:2808: snap_remove snap 352 2024-09-13T14:32:46.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:46 smithi184 ceph-mon[92527]: pgmap v780: 89 pgs: 89 active+clean; 163 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 2.3 MiB/s wr, 22 op/s 2024-09-13T14:32:46.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:46 smithi184 ceph-mon[92527]: osdmap e775: 8 total, 8 up, 8 in 2024-09-13T14:32:46.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:46 smithi151 ceph-mon[104450]: pgmap v780: 89 pgs: 89 active+clean; 163 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 2.3 MiB/s wr, 22 op/s 2024-09-13T14:32:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:46 smithi151 ceph-mon[104450]: osdmap e775: 8 total, 8 up, 8 in 2024-09-13T14:32:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:46 smithi151 ceph-mon[108966]: pgmap v780: 89 pgs: 89 active+clean; 163 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 2.3 MiB/s wr, 22 op/s 2024-09-13T14:32:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:46 smithi151 ceph-mon[108966]: osdmap e775: 8 total, 8 up, 8 in 2024-09-13T14:32:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:47.662 INFO:tasks.rados.rados.0.smithi151.stdout:2808: done (0 left) 2024-09-13T14:32:47.662 INFO:tasks.rados.rados.0.smithi151.stdout:2809: write oid 4 current snap is 357 2024-09-13T14:32:47.662 INFO:tasks.rados.rados.0.smithi151.stdout:2809: seq_num 1053 ranges {497771=493911,1619186=554020,2950529=430606} 2024-09-13T14:32:47.668 INFO:tasks.rados.rados.0.smithi151.stdout:2809: writing smithi15186934-4 from 497771 to 991682 tid 1 2024-09-13T14:32:47.675 INFO:tasks.rados.rados.0.smithi151.stdout:2809: writing smithi15186934-4 from 1619186 to 2173206 tid 2 2024-09-13T14:32:47.682 INFO:tasks.rados.rados.0.smithi151.stdout:2809: writing smithi15186934-4 from 2950529 to 3381135 tid 3 2024-09-13T14:32:47.682 INFO:tasks.rados.rados.0.smithi151.stdout:2810: write oid 11 current snap is 357 2024-09-13T14:32:47.682 INFO:tasks.rados.rados.0.smithi151.stdout:2810: seq_num 1054 ranges {514578=569160,1677668=414031,2577952=641878} 2024-09-13T14:32:47.688 INFO:tasks.rados.rados.0.smithi151.stdout:2810: writing smithi15186934-11 from 514578 to 1083738 tid 1 2024-09-13T14:32:47.693 INFO:tasks.rados.rados.0.smithi151.stdout:2810: writing smithi15186934-11 from 1677668 to 2091699 tid 2 2024-09-13T14:32:47.700 INFO:tasks.rados.rados.0.smithi151.stdout:2810: writing smithi15186934-11 from 2577952 to 3219830 tid 3 2024-09-13T14:32:47.700 INFO:tasks.rados.rados.0.smithi151.stdout:2811: write oid 39 current snap is 357 2024-09-13T14:32:47.700 INFO:tasks.rados.rados.0.smithi151.stdout:2809: finishing write tid 1 to smithi15186934-4 2024-09-13T14:32:47.701 INFO:tasks.rados.rados.0.smithi151.stdout:2809: finishing write tid 2 to smithi15186934-4 2024-09-13T14:32:47.701 INFO:tasks.rados.rados.0.smithi151.stdout:2809: finishing write tid 3 to smithi15186934-4 2024-09-13T14:32:47.701 INFO:tasks.rados.rados.0.smithi151.stdout:2809: finishing write tid 4 to smithi15186934-4 2024-09-13T14:32:47.701 INFO:tasks.rados.rados.0.smithi151.stdout:2809: finishing write tid 5 to smithi15186934-4 2024-09-13T14:32:47.701 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 1401 (ObjNum 1053 snap 357 seq_num 1053) dirty exists 2024-09-13T14:32:47.701 INFO:tasks.rados.rados.0.smithi151.stdout:2809: left oid 4 (ObjNum 1053 snap 357 seq_num 1053) 2024-09-13T14:32:47.701 INFO:tasks.rados.rados.0.smithi151.stdout:2810: finishing write tid 1 to smithi15186934-11 2024-09-13T14:32:47.701 INFO:tasks.rados.rados.0.smithi151.stdout:2810: finishing write tid 2 to smithi15186934-11 2024-09-13T14:32:47.701 INFO:tasks.rados.rados.0.smithi151.stdout:2811: seq_num 1055 ranges {531385=644409,1736150=557689,3005375=569503} 2024-09-13T14:32:47.706 INFO:tasks.rados.rados.0.smithi151.stdout:2811: writing smithi15186934-39 from 531385 to 1175794 tid 1 2024-09-13T14:32:47.713 INFO:tasks.rados.rados.0.smithi151.stdout:2811: writing smithi15186934-39 from 1736150 to 2293839 tid 2 2024-09-13T14:32:47.720 INFO:tasks.rados.rados.0.smithi151.stdout:2811: writing smithi15186934-39 from 3005375 to 3574878 tid 3 2024-09-13T14:32:47.720 INFO:tasks.rados.rados.0.smithi151.stdout:2809: done (2 left) 2024-09-13T14:32:47.720 INFO:tasks.rados.rados.0.smithi151.stdout:2812: delete oid 12 current snap is 357 2024-09-13T14:32:47.721 INFO:tasks.rados.rados.0.smithi151.stdout:2810: finishing write tid 3 to smithi15186934-11 2024-09-13T14:32:47.721 INFO:tasks.rados.rados.0.smithi151.stdout:2810: finishing write tid 4 to smithi15186934-11 2024-09-13T14:32:47.721 INFO:tasks.rados.rados.0.smithi151.stdout:2810: finishing write tid 5 to smithi15186934-11 2024-09-13T14:32:47.721 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 11 v 665 (ObjNum 1054 snap 357 seq_num 1054) dirty exists 2024-09-13T14:32:47.721 INFO:tasks.rados.rados.0.smithi151.stdout:2810: left oid 11 (ObjNum 1054 snap 357 seq_num 1054) 2024-09-13T14:32:47.721 INFO:tasks.rados.rados.0.smithi151.stdout:2811: finishing write tid 1 to smithi15186934-39 2024-09-13T14:32:47.721 INFO:tasks.rados.rados.0.smithi151.stdout:2811: finishing write tid 2 to smithi15186934-39 2024-09-13T14:32:47.722 INFO:tasks.rados.rados.0.smithi151.stdout:2810: done (2 left) 2024-09-13T14:32:47.722 INFO:tasks.rados.rados.0.smithi151.stdout:2812: done (1 left) 2024-09-13T14:32:47.722 INFO:tasks.rados.rados.0.smithi151.stdout:2813: read oid 50 snap -1 2024-09-13T14:32:47.722 INFO:tasks.rados.rados.0.smithi151.stdout:2813: expect (ObjNum 905 snap 314 seq_num 905) 2024-09-13T14:32:47.722 INFO:tasks.rados.rados.0.smithi151.stdout:2814: write oid 24 current snap is 357 2024-09-13T14:32:47.722 INFO:tasks.rados.rados.0.smithi151.stdout:2814: seq_num 1057 ranges {564999=511260,1853114=561358,3060221=592047} 2024-09-13T14:32:47.728 INFO:tasks.rados.rados.0.smithi151.stdout:2814: writing smithi15186934-24 from 564999 to 1076259 tid 1 2024-09-13T14:32:47.735 INFO:tasks.rados.rados.0.smithi151.stdout:2814: writing smithi15186934-24 from 1853114 to 2414472 tid 2 2024-09-13T14:32:47.742 INFO:tasks.rados.rados.0.smithi151.stdout:2814: writing smithi15186934-24 from 3060221 to 3652268 tid 3 2024-09-13T14:32:47.742 INFO:tasks.rados.rados.0.smithi151.stdout:2815: write oid 33 current snap is 357 2024-09-13T14:32:47.742 INFO:tasks.rados.rados.0.smithi151.stdout:2815: seq_num 1058 ranges {581806=586509,1911596=421369,3087644=403319} 2024-09-13T14:32:47.749 INFO:tasks.rados.rados.0.smithi151.stdout:2815: writing smithi15186934-33 from 581806 to 1168315 tid 1 2024-09-13T14:32:47.755 INFO:tasks.rados.rados.0.smithi151.stdout:2815: writing smithi15186934-33 from 1911596 to 2332965 tid 2 2024-09-13T14:32:47.762 INFO:tasks.rados.rados.0.smithi151.stdout:2815: writing smithi15186934-33 from 3087644 to 3490963 tid 3 2024-09-13T14:32:47.762 INFO:tasks.rados.rados.0.smithi151.stdout:2816: read oid 8 snap -1 2024-09-13T14:32:47.762 INFO:tasks.rados.rados.0.smithi151.stdout:2816: expect deleted 2024-09-13T14:32:47.762 INFO:tasks.rados.rados.0.smithi151.stdout:2817: rollback oid 32 current snap is 357 2024-09-13T14:32:47.762 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 32 to 339 2024-09-13T14:32:47.762 INFO:tasks.rados.rados.0.smithi151.stdout:2811: finishing write tid 3 to smithi15186934-39 2024-09-13T14:32:47.762 INFO:tasks.rados.rados.0.smithi151.stdout:2811: finishing write tid 4 to smithi15186934-39 2024-09-13T14:32:47.763 INFO:tasks.rados.rados.0.smithi151.stdout:2818: snap_create 2024-09-13T14:32:47.763 INFO:tasks.rados.rados.0.smithi151.stdout:2811: finishing write tid 5 to smithi15186934-39 2024-09-13T14:32:47.763 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 1138 (ObjNum 1055 snap 357 seq_num 1055) dirty exists 2024-09-13T14:32:47.763 INFO:tasks.rados.rados.0.smithi151.stdout:2811: left oid 39 (ObjNum 1055 snap 357 seq_num 1055) 2024-09-13T14:32:47.763 INFO:tasks.rados.rados.0.smithi151.stdout:2814: finishing write tid 1 to smithi15186934-24 2024-09-13T14:32:47.763 INFO:tasks.rados.rados.0.smithi151.stdout:2813: expect (ObjNum 905 snap 314 seq_num 905) 2024-09-13T14:32:47.948 INFO:tasks.rados.rados.0.smithi151.stdout:2814: finishing write tid 2 to smithi15186934-24 2024-09-13T14:32:47.948 INFO:tasks.rados.rados.0.smithi151.stdout:2814: finishing write tid 3 to smithi15186934-24 2024-09-13T14:32:47.948 INFO:tasks.rados.rados.0.smithi151.stdout:2814: finishing write tid 4 to smithi15186934-24 2024-09-13T14:32:47.948 INFO:tasks.rados.rados.0.smithi151.stdout:2814: finishing write tid 5 to smithi15186934-24 2024-09-13T14:32:47.948 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 24 v 1725 (ObjNum 1057 snap 357 seq_num 1057) dirty exists 2024-09-13T14:32:47.948 INFO:tasks.rados.rados.0.smithi151.stdout:2814: left oid 24 (ObjNum 1057 snap 357 seq_num 1057) 2024-09-13T14:32:47.948 INFO:tasks.rados.rados.0.smithi151.stdout:2815: finishing write tid 1 to smithi15186934-33 2024-09-13T14:32:47.948 INFO:tasks.rados.rados.0.smithi151.stdout:2815: finishing write tid 2 to smithi15186934-33 2024-09-13T14:32:47.949 INFO:tasks.rados.rados.0.smithi151.stdout:2815: finishing write tid 3 to smithi15186934-33 2024-09-13T14:32:47.949 INFO:tasks.rados.rados.0.smithi151.stdout:2815: finishing write tid 4 to smithi15186934-33 2024-09-13T14:32:47.949 INFO:tasks.rados.rados.0.smithi151.stdout:2815: finishing write tid 5 to smithi15186934-33 2024-09-13T14:32:47.949 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 33 v 1045 (ObjNum 1058 snap 357 seq_num 1058) dirty exists 2024-09-13T14:32:47.949 INFO:tasks.rados.rados.0.smithi151.stdout:2815: left oid 33 (ObjNum 1058 snap 357 seq_num 1058) 2024-09-13T14:32:47.949 INFO:tasks.rados.rados.0.smithi151.stdout:2817: finishing rollback tid 0 to smithi15186934-32 2024-09-13T14:32:47.949 INFO:tasks.rados.rados.0.smithi151.stdout:2817: finishing rollback tid 1 to smithi15186934-32 2024-09-13T14:32:47.949 INFO:tasks.rados.rados.0.smithi151.stdout:2817: finishing rollback tid 2 to smithi15186934-32 2024-09-13T14:32:47.949 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 32 v 1175 (ObjNum 925 snap 318 seq_num 925) dirty exists 2024-09-13T14:32:48.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:47 smithi184 ceph-mon[92527]: osdmap e776: 8 total, 8 up, 8 in 2024-09-13T14:32:48.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:47 smithi151 ceph-mon[104450]: osdmap e776: 8 total, 8 up, 8 in 2024-09-13T14:32:48.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:47 smithi151 ceph-mon[108966]: osdmap e776: 8 total, 8 up, 8 in 2024-09-13T14:32:48.670 INFO:tasks.rados.rados.0.smithi151.stdout:2811: done (6 left) 2024-09-13T14:32:48.670 INFO:tasks.rados.rados.0.smithi151.stdout:2813: done (5 left) 2024-09-13T14:32:48.670 INFO:tasks.rados.rados.0.smithi151.stdout:2814: done (4 left) 2024-09-13T14:32:48.670 INFO:tasks.rados.rados.0.smithi151.stdout:2815: done (3 left) 2024-09-13T14:32:48.670 INFO:tasks.rados.rados.0.smithi151.stdout:2816: done (2 left) 2024-09-13T14:32:48.670 INFO:tasks.rados.rados.0.smithi151.stdout:2817: done (1 left) 2024-09-13T14:32:48.670 INFO:tasks.rados.rados.0.smithi151.stdout:2818: done (0 left) 2024-09-13T14:32:48.670 INFO:tasks.rados.rados.0.smithi151.stdout:2819: snap_remove snap 355 2024-09-13T14:32:49.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:48 smithi184 ceph-mon[92527]: pgmap v783: 89 pgs: 89 active+clean; 168 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 7.0 MiB/s rd, 2.2 MiB/s wr, 15 op/s 2024-09-13T14:32:49.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:48 smithi184 ceph-mon[92527]: osdmap e777: 8 total, 8 up, 8 in 2024-09-13T14:32:49.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:48 smithi151 ceph-mon[104450]: pgmap v783: 89 pgs: 89 active+clean; 168 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 7.0 MiB/s rd, 2.2 MiB/s wr, 15 op/s 2024-09-13T14:32:49.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:48 smithi151 ceph-mon[104450]: osdmap e777: 8 total, 8 up, 8 in 2024-09-13T14:32:49.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:48 smithi151 ceph-mon[108966]: pgmap v783: 89 pgs: 89 active+clean; 168 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 7.0 MiB/s rd, 2.2 MiB/s wr, 15 op/s 2024-09-13T14:32:49.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:48 smithi151 ceph-mon[108966]: osdmap e777: 8 total, 8 up, 8 in 2024-09-13T14:32:49.585 INFO:tasks.rados.rados.0.smithi151.stdout:2819: done (0 left) 2024-09-13T14:32:49.585 INFO:tasks.rados.rados.0.smithi151.stdout:2820: rollback oid 26 current snap is 358 2024-09-13T14:32:49.585 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 26 to 347 2024-09-13T14:32:49.586 INFO:tasks.rados.rados.0.smithi151.stdout:2821: delete oid 4 current snap is 358 2024-09-13T14:32:49.591 INFO:tasks.rados.rados.0.smithi151.stdout:2821: done (1 left) 2024-09-13T14:32:49.591 INFO:tasks.rados.rados.0.smithi151.stdout:2822: read oid 4 snap 350 2024-09-13T14:32:49.591 INFO:tasks.rados.rados.0.smithi151.stdout:2822: expect deleted 2024-09-13T14:32:49.591 INFO:tasks.rados.rados.0.smithi151.stdout:2823: write oid 29 current snap is 358 2024-09-13T14:32:49.591 INFO:tasks.rados.rados.0.smithi151.stdout:2823: seq_num 1060 ranges {615420=737007,1912207=425038,3026137=542216} 2024-09-13T14:32:49.599 INFO:tasks.rados.rados.0.smithi151.stdout:2823: writing smithi15186934-29 from 615420 to 1352427 tid 1 2024-09-13T14:32:49.605 INFO:tasks.rados.rados.0.smithi151.stdout:2823: writing smithi15186934-29 from 1912207 to 2337245 tid 2 2024-09-13T14:32:49.612 INFO:tasks.rados.rados.0.smithi151.stdout:2823: writing smithi15186934-29 from 3026137 to 3568353 tid 3 2024-09-13T14:32:49.612 INFO:tasks.rados.rados.0.smithi151.stdout:2824: snap_create 2024-09-13T14:32:49.612 INFO:tasks.rados.rados.0.smithi151.stdout:2820: finishing rollback tid 0 to smithi15186934-26 2024-09-13T14:32:49.612 INFO:tasks.rados.rados.0.smithi151.stdout:2820: finishing rollback tid 1 to smithi15186934-26 2024-09-13T14:32:49.612 INFO:tasks.rados.rados.0.smithi151.stdout:2820: finishing rollback tid 2 to smithi15186934-26 2024-09-13T14:32:49.612 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 1014 (ObjNum 1010 snap 347 seq_num 1010) dirty exists 2024-09-13T14:32:49.612 INFO:tasks.rados.rados.0.smithi151.stdout:2823: finishing write tid 1 to smithi15186934-29 2024-09-13T14:32:49.612 INFO:tasks.rados.rados.0.smithi151.stdout:2823: finishing write tid 2 to smithi15186934-29 2024-09-13T14:32:49.614 INFO:tasks.rados.rados.0.smithi151.stdout:2823: finishing write tid 3 to smithi15186934-29 2024-09-13T14:32:49.615 INFO:tasks.rados.rados.0.smithi151.stdout:2823: finishing write tid 4 to smithi15186934-29 2024-09-13T14:32:49.615 INFO:tasks.rados.rados.0.smithi151.stdout:2823: finishing write tid 5 to smithi15186934-29 2024-09-13T14:32:49.615 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 29 v 671 (ObjNum 1060 snap 358 seq_num 1060) dirty exists 2024-09-13T14:32:49.615 INFO:tasks.rados.rados.0.smithi151.stdout:2823: left oid 29 (ObjNum 1060 snap 358 seq_num 1060) 2024-09-13T14:32:50.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:49 smithi184 ceph-mon[92527]: osdmap e778: 8 total, 8 up, 8 in 2024-09-13T14:32:50.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:49 smithi184 ceph-mon[92527]: osdmap e779: 8 total, 8 up, 8 in 2024-09-13T14:32:50.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:49 smithi151 ceph-mon[108966]: osdmap e778: 8 total, 8 up, 8 in 2024-09-13T14:32:50.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:49 smithi151 ceph-mon[108966]: osdmap e779: 8 total, 8 up, 8 in 2024-09-13T14:32:50.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:49 smithi151 ceph-mon[104450]: osdmap e778: 8 total, 8 up, 8 in 2024-09-13T14:32:50.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:49 smithi151 ceph-mon[104450]: osdmap e779: 8 total, 8 up, 8 in 2024-09-13T14:32:50.587 INFO:tasks.rados.rados.0.smithi151.stdout:2820: done (3 left) 2024-09-13T14:32:50.587 INFO:tasks.rados.rados.0.smithi151.stdout:2822: done (2 left) 2024-09-13T14:32:50.588 INFO:tasks.rados.rados.0.smithi151.stdout:2823: done (1 left) 2024-09-13T14:32:50.588 INFO:tasks.rados.rados.0.smithi151.stdout:2824: done (0 left) 2024-09-13T14:32:50.588 INFO:tasks.rados.rados.0.smithi151.stdout:2825: delete oid 25 current snap is 359 2024-09-13T14:32:50.591 INFO:tasks.rados.rados.0.smithi151.stdout:2825: done (0 left) 2024-09-13T14:32:50.592 INFO:tasks.rados.rados.0.smithi151.stdout:2826: snap_create 2024-09-13T14:32:50.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:32:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:32:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:32:51.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:50 smithi184 ceph-mon[92527]: pgmap v786: 89 pgs: 89 active+clean; 174 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 7.0 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2024-09-13T14:32:51.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:50 smithi184 ceph-mon[92527]: osdmap e780: 8 total, 8 up, 8 in 2024-09-13T14:32:51.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:50 smithi151 ceph-mon[104450]: pgmap v786: 89 pgs: 89 active+clean; 174 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 7.0 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2024-09-13T14:32:51.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:50 smithi151 ceph-mon[104450]: osdmap e780: 8 total, 8 up, 8 in 2024-09-13T14:32:51.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:50 smithi151 ceph-mon[108966]: pgmap v786: 89 pgs: 89 active+clean; 174 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 7.0 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2024-09-13T14:32:51.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:50 smithi151 ceph-mon[108966]: osdmap e780: 8 total, 8 up, 8 in 2024-09-13T14:32:51.589 INFO:tasks.rados.rados.0.smithi151.stdout:2826: done (0 left) 2024-09-13T14:32:51.590 INFO:tasks.rados.rados.0.smithi151.stdout:2827: rollback oid 38 current snap is 360 2024-09-13T14:32:51.590 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 38 to 358 2024-09-13T14:32:51.590 INFO:tasks.rados.rados.0.smithi151.stdout:2828: read oid 16 snap 353 2024-09-13T14:32:51.590 INFO:tasks.rados.rados.0.smithi151.stdout:2828: expect (ObjNum 1021 snap 349 seq_num 1021) 2024-09-13T14:32:51.590 INFO:tasks.rados.rados.0.smithi151.stdout:2829: read oid 33 snap -1 2024-09-13T14:32:51.590 INFO:tasks.rados.rados.0.smithi151.stdout:2829: expect (ObjNum 1058 snap 357 seq_num 1058) 2024-09-13T14:32:51.590 INFO:tasks.rados.rados.0.smithi151.stdout:2830: rollback oid 34 current snap is 360 2024-09-13T14:32:51.590 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 34 to 359 2024-09-13T14:32:51.590 INFO:tasks.rados.rados.0.smithi151.stdout:2831: write oid 3 current snap is 360 2024-09-13T14:32:51.590 INFO:tasks.rados.rados.0.smithi151.stdout:2831: seq_num 1062 ranges {649034=487505,1629171=428707,2680983=564760,3798768=50266} 2024-09-13T14:32:51.596 INFO:tasks.rados.rados.0.smithi151.stdout:2831: writing smithi15186934-3 from 649034 to 1136539 tid 1 2024-09-13T14:32:51.601 INFO:tasks.rados.rados.0.smithi151.stdout:2831: writing smithi15186934-3 from 1629171 to 2057878 tid 2 2024-09-13T14:32:51.608 INFO:tasks.rados.rados.0.smithi151.stdout:2831: writing smithi15186934-3 from 2680983 to 3245743 tid 3 2024-09-13T14:32:51.611 INFO:tasks.rados.rados.0.smithi151.stdout:2831: writing smithi15186934-3 from 3798768 to 3849034 tid 4 2024-09-13T14:32:51.612 INFO:tasks.rados.rados.0.smithi151.stdout:2832: write oid 37 current snap is 360 2024-09-13T14:32:51.612 INFO:tasks.rados.rados.0.smithi151.stdout:2830: finishing rollback tid 1 to smithi15186934-34 2024-09-13T14:32:51.612 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 1113 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-13T14:32:51.612 INFO:tasks.rados.rados.0.smithi151.stdout:2827: finishing rollback tid 0 to smithi15186934-38 2024-09-13T14:32:51.612 INFO:tasks.rados.rados.0.smithi151.stdout:2827: finishing rollback tid 1 to smithi15186934-38 2024-09-13T14:32:51.612 INFO:tasks.rados.rados.0.smithi151.stdout:2832: seq_num 1063 ranges {665841=562754,1687653=572365,2708406=776032} 2024-09-13T14:32:51.620 INFO:tasks.rados.rados.0.smithi151.stdout:2832: writing smithi15186934-37 from 665841 to 1228595 tid 1 2024-09-13T14:32:51.624 INFO:tasks.rados.rados.0.smithi151.stdout:2832: writing smithi15186934-37 from 1687653 to 2260018 tid 2 2024-09-13T14:32:51.631 INFO:tasks.rados.rados.0.smithi151.stdout:2832: writing smithi15186934-37 from 2708406 to 3484438 tid 3 2024-09-13T14:32:51.631 INFO:tasks.rados.rados.0.smithi151.stdout:2830: done (5 left) 2024-09-13T14:32:51.631 INFO:tasks.rados.rados.0.smithi151.stdout:2833: write oid 8 current snap is 360 2024-09-13T14:32:51.631 INFO:tasks.rados.rados.0.smithi151.stdout:2827: finishing rollback tid 2 to smithi15186934-38 2024-09-13T14:32:51.631 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 1144 (ObjNum 1003 snap 345 seq_num 1003) dirty exists 2024-09-13T14:32:51.631 INFO:tasks.rados.rados.0.smithi151.stdout:2829: expect (ObjNum 1058 snap 357 seq_num 1058) 2024-09-13T14:32:51.807 INFO:tasks.rados.rados.0.smithi151.stdout:2828: expect (ObjNum 1021 snap 349 seq_num 1021) 2024-09-13T14:32:51.990 INFO:tasks.rados.rados.0.smithi151.stdout:2831: finishing write tid 1 to smithi15186934-3 2024-09-13T14:32:51.990 INFO:tasks.rados.rados.0.smithi151.stdout:2831: finishing write tid 2 to smithi15186934-3 2024-09-13T14:32:51.990 INFO:tasks.rados.rados.0.smithi151.stdout:2831: finishing write tid 3 to smithi15186934-3 2024-09-13T14:32:51.990 INFO:tasks.rados.rados.0.smithi151.stdout:2831: finishing write tid 4 to smithi15186934-3 2024-09-13T14:32:51.990 INFO:tasks.rados.rados.0.smithi151.stdout:2831: finishing write tid 5 to smithi15186934-3 2024-09-13T14:32:51.990 INFO:tasks.rados.rados.0.smithi151.stdout:2831: finishing write tid 6 to smithi15186934-3 2024-09-13T14:32:51.990 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 1130 (ObjNum 1062 snap 360 seq_num 1062) dirty exists 2024-09-13T14:32:51.990 INFO:tasks.rados.rados.0.smithi151.stdout:2831: left oid 3 (ObjNum 1062 snap 360 seq_num 1062) 2024-09-13T14:32:51.990 INFO:tasks.rados.rados.0.smithi151.stdout:2832: finishing write tid 1 to smithi15186934-37 2024-09-13T14:32:51.990 INFO:tasks.rados.rados.0.smithi151.stdout:2832: finishing write tid 2 to smithi15186934-37 2024-09-13T14:32:51.990 INFO:tasks.rados.rados.0.smithi151.stdout:2832: finishing write tid 3 to smithi15186934-37 2024-09-13T14:32:51.991 INFO:tasks.rados.rados.0.smithi151.stdout:2832: finishing write tid 4 to smithi15186934-37 2024-09-13T14:32:51.991 INFO:tasks.rados.rados.0.smithi151.stdout:2832: finishing write tid 5 to smithi15186934-37 2024-09-13T14:32:51.991 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 37 v 1250 (ObjNum 1063 snap 360 seq_num 1063) dirty exists 2024-09-13T14:32:51.991 INFO:tasks.rados.rados.0.smithi151.stdout:2832: left oid 37 (ObjNum 1063 snap 360 seq_num 1063) 2024-09-13T14:32:51.991 INFO:tasks.rados.rados.0.smithi151.stdout:2833: seq_num 1064 ranges {682648=638003,2029782=432376,3019476=587304} 2024-09-13T14:32:51.996 INFO:tasks.rados.rados.0.smithi151.stdout:2833: writing smithi15186934-8 from 682648 to 1320651 tid 1 2024-09-13T14:32:52.002 INFO:tasks.rados.rados.0.smithi151.stdout:2833: writing smithi15186934-8 from 2029782 to 2462158 tid 2 2024-09-13T14:32:52.009 INFO:tasks.rados.rados.0.smithi151.stdout:2833: writing smithi15186934-8 from 3019476 to 3606780 tid 3 2024-09-13T14:32:52.009 INFO:tasks.rados.rados.0.smithi151.stdout:2827: done (5 left) 2024-09-13T14:32:52.009 INFO:tasks.rados.rados.0.smithi151.stdout:2828: done (4 left) 2024-09-13T14:32:52.009 INFO:tasks.rados.rados.0.smithi151.stdout:2829: done (3 left) 2024-09-13T14:32:52.009 INFO:tasks.rados.rados.0.smithi151.stdout:2831: done (2 left) 2024-09-13T14:32:52.009 INFO:tasks.rados.rados.0.smithi151.stdout:2832: done (1 left) 2024-09-13T14:32:52.009 INFO:tasks.rados.rados.0.smithi151.stdout:2834: snap_create 2024-09-13T14:32:52.010 INFO:tasks.rados.rados.0.smithi151.stdout:2833: finishing write tid 1 to smithi15186934-8 2024-09-13T14:32:52.010 INFO:tasks.rados.rados.0.smithi151.stdout:2833: finishing write tid 2 to smithi15186934-8 2024-09-13T14:32:52.012 INFO:tasks.rados.rados.0.smithi151.stdout:2833: finishing write tid 3 to smithi15186934-8 2024-09-13T14:32:52.012 INFO:tasks.rados.rados.0.smithi151.stdout:2833: finishing write tid 4 to smithi15186934-8 2024-09-13T14:32:52.012 INFO:tasks.rados.rados.0.smithi151.stdout:2833: finishing write tid 5 to smithi15186934-8 2024-09-13T14:32:52.013 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 858 (ObjNum 1064 snap 360 seq_num 1064) dirty exists 2024-09-13T14:32:52.013 INFO:tasks.rados.rados.0.smithi151.stdout:2833: left oid 8 (ObjNum 1064 snap 360 seq_num 1064) 2024-09-13T14:32:52.684 INFO:tasks.rados.rados.0.smithi151.stdout:2833: done (1 left) 2024-09-13T14:32:52.684 INFO:tasks.rados.rados.0.smithi151.stdout:2834: done (0 left) 2024-09-13T14:32:52.685 INFO:tasks.rados.rados.0.smithi151.stdout:2835: read oid 45 snap -1 2024-09-13T14:32:52.685 INFO:tasks.rados.rados.0.smithi151.stdout:2835: expect (ObjNum 965 snap 330 seq_num 965) 2024-09-13T14:32:52.685 INFO:tasks.rados.rados.0.smithi151.stdout:2836: write oid 8 current snap is 361 2024-09-13T14:32:52.685 INFO:tasks.rados.rados.0.smithi151.stdout:2836: seq_num 1065 ranges {699455=429605,1804617=576034,3163252=514929} 2024-09-13T14:32:52.690 INFO:tasks.rados.rados.0.smithi151.stdout:2836: writing smithi15186934-8 from 699455 to 1129060 tid 1 2024-09-13T14:32:52.697 INFO:tasks.rados.rados.0.smithi151.stdout:2836: writing smithi15186934-8 from 1804617 to 2380651 tid 2 2024-09-13T14:32:52.705 INFO:tasks.rados.rados.0.smithi151.stdout:2836: writing smithi15186934-8 from 3163252 to 3678181 tid 3 2024-09-13T14:32:52.705 INFO:tasks.rados.rados.0.smithi151.stdout:2837: read oid 39 snap 358 2024-09-13T14:32:52.705 INFO:tasks.rados.rados.0.smithi151.stdout:2837: expect (ObjNum 1055 snap 357 seq_num 1055) 2024-09-13T14:32:52.705 INFO:tasks.rados.rados.0.smithi151.stdout:2838: rollback oid 40 current snap is 361 2024-09-13T14:32:52.705 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 40 to 354 2024-09-13T14:32:52.705 INFO:tasks.rados.rados.0.smithi151.stdout:2835: expect (ObjNum 965 snap 330 seq_num 965) 2024-09-13T14:32:52.809 INFO:tasks.rados.rados.0.smithi151.stdout:2836: finishing write tid 1 to smithi15186934-8 2024-09-13T14:32:52.809 INFO:tasks.rados.rados.0.smithi151.stdout:2836: finishing write tid 2 to smithi15186934-8 2024-09-13T14:32:52.809 INFO:tasks.rados.rados.0.smithi151.stdout:2838: finishing rollback tid 0 to smithi15186934-40 2024-09-13T14:32:52.809 INFO:tasks.rados.rados.0.smithi151.stdout:2836: finishing write tid 3 to smithi15186934-8 2024-09-13T14:32:52.809 INFO:tasks.rados.rados.0.smithi151.stdout:2836: finishing write tid 4 to smithi15186934-8 2024-09-13T14:32:52.809 INFO:tasks.rados.rados.0.smithi151.stdout:2836: finishing write tid 5 to smithi15186934-8 2024-09-13T14:32:52.809 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 863 (ObjNum 1065 snap 361 seq_num 1065) dirty exists 2024-09-13T14:32:52.809 INFO:tasks.rados.rados.0.smithi151.stdout:2836: left oid 8 (ObjNum 1065 snap 361 seq_num 1065) 2024-09-13T14:32:52.810 INFO:tasks.rados.rados.0.smithi151.stdout:2838: finishing rollback tid 1 to smithi15186934-40 2024-09-13T14:32:52.810 INFO:tasks.rados.rados.0.smithi151.stdout:2835: done (3 left) 2024-09-13T14:32:52.810 INFO:tasks.rados.rados.0.smithi151.stdout:2836: done (2 left) 2024-09-13T14:32:52.810 INFO:tasks.rados.rados.0.smithi151.stdout:2839: read oid 23 snap -1 2024-09-13T14:32:52.810 INFO:tasks.rados.rados.0.smithi151.stdout:2839: expect (ObjNum 1042 snap 355 seq_num 1042) 2024-09-13T14:32:52.810 INFO:tasks.rados.rados.0.smithi151.stdout:2840: read oid 33 snap -1 2024-09-13T14:32:52.810 INFO:tasks.rados.rados.0.smithi151.stdout:2840: expect (ObjNum 1058 snap 357 seq_num 1058) 2024-09-13T14:32:52.810 INFO:tasks.rados.rados.0.smithi151.stdout:2841: rollback oid 37 current snap is 361 2024-09-13T14:32:52.810 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 37 to 360 2024-09-13T14:32:52.810 INFO:tasks.rados.rados.0.smithi151.stdout:2838: finishing rollback tid 2 to smithi15186934-40 2024-09-13T14:32:52.810 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 40 v 1134 (ObjNum 1031 snap 352 seq_num 1031) dirty exists 2024-09-13T14:32:52.810 INFO:tasks.rados.rados.0.smithi151.stdout:2837: expect (ObjNum 1055 snap 357 seq_num 1055) 2024-09-13T14:32:52.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:52 smithi184 ceph-mon[92527]: pgmap v789: 89 pgs: 89 active+clean; 191 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.7 MiB/s wr, 11 op/s 2024-09-13T14:32:52.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:52 smithi184 ceph-mon[92527]: osdmap e781: 8 total, 8 up, 8 in 2024-09-13T14:32:52.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:52 smithi151 ceph-mon[104450]: pgmap v789: 89 pgs: 89 active+clean; 191 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.7 MiB/s wr, 11 op/s 2024-09-13T14:32:52.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:52 smithi151 ceph-mon[104450]: osdmap e781: 8 total, 8 up, 8 in 2024-09-13T14:32:52.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:52 smithi151 ceph-mon[108966]: pgmap v789: 89 pgs: 89 active+clean; 191 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.7 MiB/s wr, 11 op/s 2024-09-13T14:32:52.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:52 smithi151 ceph-mon[108966]: osdmap e781: 8 total, 8 up, 8 in 2024-09-13T14:32:53.036 INFO:tasks.rados.rados.0.smithi151.stdout:2840: expect (ObjNum 1058 snap 357 seq_num 1058) 2024-09-13T14:32:53.213 INFO:tasks.rados.rados.0.smithi151.stdout:2839: expect (ObjNum 1042 snap 355 seq_num 1042) 2024-09-13T14:32:53.324 INFO:tasks.rados.rados.0.smithi151.stdout:2841: finishing rollback tid 0 to smithi15186934-37 2024-09-13T14:32:53.324 INFO:tasks.rados.rados.0.smithi151.stdout:2841: finishing rollback tid 1 to smithi15186934-37 2024-09-13T14:32:53.324 INFO:tasks.rados.rados.0.smithi151.stdout:2841: finishing rollback tid 2 to smithi15186934-37 2024-09-13T14:32:53.324 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 37 v 1254 (ObjNum 1063 snap 360 seq_num 1063) dirty exists 2024-09-13T14:32:53.324 INFO:tasks.rados.rados.0.smithi151.stdout:2837: done (4 left) 2024-09-13T14:32:53.324 INFO:tasks.rados.rados.0.smithi151.stdout:2838: done (3 left) 2024-09-13T14:32:53.324 INFO:tasks.rados.rados.0.smithi151.stdout:2839: done (2 left) 2024-09-13T14:32:53.325 INFO:tasks.rados.rados.0.smithi151.stdout:2840: done (1 left) 2024-09-13T14:32:53.325 INFO:tasks.rados.rados.0.smithi151.stdout:2841: done (0 left) 2024-09-13T14:32:53.325 INFO:tasks.rados.rados.0.smithi151.stdout:2842: read oid 2 snap 339 2024-09-13T14:32:53.325 INFO:tasks.rados.rados.0.smithi151.stdout:2842: expect deleted 2024-09-13T14:32:53.325 INFO:tasks.rados.rados.0.smithi151.stdout:2843: read oid 29 snap -1 2024-09-13T14:32:53.325 INFO:tasks.rados.rados.0.smithi151.stdout:2843: expect (ObjNum 1060 snap 358 seq_num 1060) 2024-09-13T14:32:53.325 INFO:tasks.rados.rados.0.smithi151.stdout:2844: snap_remove snap 350 2024-09-13T14:32:53.689 INFO:tasks.rados.rados.0.smithi151.stdout:2844: done (2 left) 2024-09-13T14:32:53.689 INFO:tasks.rados.rados.0.smithi151.stdout:2845: write oid 39 current snap is 361 2024-09-13T14:32:53.689 INFO:tasks.rados.rados.0.smithi151.stdout:2845: seq_num 1066 ranges {716262=504854,1863099=719692,3074322=726201} 2024-09-13T14:32:53.696 INFO:tasks.rados.rados.0.smithi151.stdout:2845: writing smithi15186934-39 from 716262 to 1221116 tid 1 2024-09-13T14:32:53.705 INFO:tasks.rados.rados.0.smithi151.stdout:2845: writing smithi15186934-39 from 1863099 to 2582791 tid 2 2024-09-13T14:32:53.712 INFO:tasks.rados.rados.0.smithi151.stdout:2845: writing smithi15186934-39 from 3074322 to 3800523 tid 3 2024-09-13T14:32:53.712 INFO:tasks.rados.rados.0.smithi151.stdout:2846: read oid 4 snap -1 2024-09-13T14:32:53.712 INFO:tasks.rados.rados.0.smithi151.stdout:2846: expect deleted 2024-09-13T14:32:53.712 INFO:tasks.rados.rados.0.smithi151.stdout:2847: snap_create 2024-09-13T14:32:53.712 INFO:tasks.rados.rados.0.smithi151.stdout:2843: expect (ObjNum 1060 snap 358 seq_num 1060) 2024-09-13T14:32:53.836 INFO:tasks.rados.rados.0.smithi151.stdout:2845: finishing write tid 1 to smithi15186934-39 2024-09-13T14:32:53.836 INFO:tasks.rados.rados.0.smithi151.stdout:2845: finishing write tid 2 to smithi15186934-39 2024-09-13T14:32:53.836 INFO:tasks.rados.rados.0.smithi151.stdout:2845: finishing write tid 3 to smithi15186934-39 2024-09-13T14:32:53.836 INFO:tasks.rados.rados.0.smithi151.stdout:2845: finishing write tid 4 to smithi15186934-39 2024-09-13T14:32:53.836 INFO:tasks.rados.rados.0.smithi151.stdout:2845: finishing write tid 5 to smithi15186934-39 2024-09-13T14:32:53.836 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 1153 (ObjNum 1066 snap 361 seq_num 1066) dirty exists 2024-09-13T14:32:53.836 INFO:tasks.rados.rados.0.smithi151.stdout:2845: left oid 39 (ObjNum 1066 snap 361 seq_num 1066) 2024-09-13T14:32:54.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:53 smithi184 ceph-mon[92527]: osdmap e782: 8 total, 8 up, 8 in 2024-09-13T14:32:54.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:53 smithi151 ceph-mon[108966]: osdmap e782: 8 total, 8 up, 8 in 2024-09-13T14:32:54.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:53 smithi151 ceph-mon[104450]: osdmap e782: 8 total, 8 up, 8 in 2024-09-13T14:32:54.584 INFO:tasks.rados.rados.0.smithi151.stdout:2842: done (4 left) 2024-09-13T14:32:54.585 INFO:tasks.rados.rados.0.smithi151.stdout:2843: done (3 left) 2024-09-13T14:32:54.585 INFO:tasks.rados.rados.0.smithi151.stdout:2845: done (2 left) 2024-09-13T14:32:54.585 INFO:tasks.rados.rados.0.smithi151.stdout:2846: done (1 left) 2024-09-13T14:32:54.585 INFO:tasks.rados.rados.0.smithi151.stdout:2847: done (0 left) 2024-09-13T14:32:54.585 INFO:tasks.rados.rados.0.smithi151.stdout:2848: write oid 10 current snap is 362 2024-09-13T14:32:54.585 INFO:tasks.rados.rados.0.smithi151.stdout:2848: seq_num 1067 ranges {733069=580103,1921581=579703,3218098=537473} 2024-09-13T14:32:54.591 INFO:tasks.rados.rados.0.smithi151.stdout:2848: writing smithi15186934-10 from 733069 to 1313172 tid 1 2024-09-13T14:32:54.598 INFO:tasks.rados.rados.0.smithi151.stdout:2848: writing smithi15186934-10 from 1921581 to 2501284 tid 2 2024-09-13T14:32:54.605 INFO:tasks.rados.rados.0.smithi151.stdout:2848: writing smithi15186934-10 from 3218098 to 3755571 tid 3 2024-09-13T14:32:54.605 INFO:tasks.rados.rados.0.smithi151.stdout:2849: snap_create 2024-09-13T14:32:54.605 INFO:tasks.rados.rados.0.smithi151.stdout:2848: finishing write tid 1 to smithi15186934-10 2024-09-13T14:32:54.605 INFO:tasks.rados.rados.0.smithi151.stdout:2848: finishing write tid 2 to smithi15186934-10 2024-09-13T14:32:54.608 INFO:tasks.rados.rados.0.smithi151.stdout:2848: finishing write tid 3 to smithi15186934-10 2024-09-13T14:32:54.608 INFO:tasks.rados.rados.0.smithi151.stdout:2848: finishing write tid 4 to smithi15186934-10 2024-09-13T14:32:54.608 INFO:tasks.rados.rados.0.smithi151.stdout:2848: finishing write tid 5 to smithi15186934-10 2024-09-13T14:32:54.608 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 1021 (ObjNum 1067 snap 362 seq_num 1067) dirty exists 2024-09-13T14:32:54.608 INFO:tasks.rados.rados.0.smithi151.stdout:2848: left oid 10 (ObjNum 1067 snap 362 seq_num 1067) 2024-09-13T14:32:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:54 smithi151 ceph-mon[104450]: pgmap v792: 89 pgs: 89 active+clean; 203 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 7.0 MiB/s rd, 3.6 MiB/s wr, 20 op/s 2024-09-13T14:32:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:54 smithi151 ceph-mon[104450]: osdmap e783: 8 total, 8 up, 8 in 2024-09-13T14:32:54.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:54 smithi151 ceph-mon[104450]: osdmap e784: 8 total, 8 up, 8 in 2024-09-13T14:32:54.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:32:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:32:54] "GET /metrics HTTP/1.1" 200 36510 "" "Prometheus/2.51.0" 2024-09-13T14:32:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:54 smithi151 ceph-mon[108966]: pgmap v792: 89 pgs: 89 active+clean; 203 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 7.0 MiB/s rd, 3.6 MiB/s wr, 20 op/s 2024-09-13T14:32:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:54 smithi151 ceph-mon[108966]: osdmap e783: 8 total, 8 up, 8 in 2024-09-13T14:32:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:54 smithi151 ceph-mon[108966]: osdmap e784: 8 total, 8 up, 8 in 2024-09-13T14:32:55.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:54 smithi184 ceph-mon[92527]: pgmap v792: 89 pgs: 89 active+clean; 203 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 7.0 MiB/s rd, 3.6 MiB/s wr, 20 op/s 2024-09-13T14:32:55.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:54 smithi184 ceph-mon[92527]: osdmap e783: 8 total, 8 up, 8 in 2024-09-13T14:32:55.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:54 smithi184 ceph-mon[92527]: osdmap e784: 8 total, 8 up, 8 in 2024-09-13T14:32:55.587 INFO:tasks.rados.rados.0.smithi151.stdout:2848: done (1 left) 2024-09-13T14:32:55.587 INFO:tasks.rados.rados.0.smithi151.stdout:2849: done (0 left) 2024-09-13T14:32:55.588 INFO:tasks.rados.rados.0.smithi151.stdout:2850: write oid 12 current snap is 363 2024-09-13T14:32:55.588 INFO:tasks.rados.rados.0.smithi151.stdout:2850: seq_num 1068 ranges {749876=655352,1863710=723361,3012815=748745} 2024-09-13T14:32:55.595 INFO:tasks.rados.rados.0.smithi151.stdout:2850: writing smithi15186934-12 from 749876 to 1405228 tid 1 2024-09-13T14:32:55.602 INFO:tasks.rados.rados.0.smithi151.stdout:2850: writing smithi15186934-12 from 1863710 to 2587071 tid 2 2024-09-13T14:32:55.609 INFO:tasks.rados.rados.0.smithi151.stdout:2850: writing smithi15186934-12 from 3012815 to 3761560 tid 3 2024-09-13T14:32:55.609 INFO:tasks.rados.rados.0.smithi151.stdout:2851: rollback oid 13 current snap is 363 2024-09-13T14:32:55.609 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 13 to 361 2024-09-13T14:32:55.609 INFO:tasks.rados.rados.0.smithi151.stdout:2850: finishing write tid 1 to smithi15186934-12 2024-09-13T14:32:55.609 INFO:tasks.rados.rados.0.smithi151.stdout:2850: finishing write tid 2 to smithi15186934-12 2024-09-13T14:32:55.609 INFO:tasks.rados.rados.0.smithi151.stdout:2852: delete oid 41 current snap is 363 2024-09-13T14:32:55.612 INFO:tasks.rados.rados.0.smithi151.stdout:2852: done (2 left) 2024-09-13T14:32:55.612 INFO:tasks.rados.rados.0.smithi151.stdout:2853: snap_create 2024-09-13T14:32:55.612 INFO:tasks.rados.rados.0.smithi151.stdout:2850: finishing write tid 3 to smithi15186934-12 2024-09-13T14:32:55.612 INFO:tasks.rados.rados.0.smithi151.stdout:2850: finishing write tid 4 to smithi15186934-12 2024-09-13T14:32:55.612 INFO:tasks.rados.rados.0.smithi151.stdout:2850: finishing write tid 5 to smithi15186934-12 2024-09-13T14:32:55.613 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 12 v 637 (ObjNum 1068 snap 363 seq_num 1068) dirty exists 2024-09-13T14:32:55.613 INFO:tasks.rados.rados.0.smithi151.stdout:2850: left oid 12 (ObjNum 1068 snap 363 seq_num 1068) 2024-09-13T14:32:55.614 INFO:tasks.rados.rados.0.smithi151.stdout:2851: finishing rollback tid 0 to smithi15186934-13 2024-09-13T14:32:55.614 INFO:tasks.rados.rados.0.smithi151.stdout:2851: finishing rollback tid 1 to smithi15186934-13 2024-09-13T14:32:55.614 INFO:tasks.rados.rados.0.smithi151.stdout:2851: finishing rollback tid 2 to smithi15186934-13 2024-09-13T14:32:55.614 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 13 v 1055 (ObjNum 1043 snap 355 seq_num 1043) dirty exists 2024-09-13T14:32:56.599 INFO:tasks.rados.rados.0.smithi151.stdout:2850: done (2 left) 2024-09-13T14:32:56.599 INFO:tasks.rados.rados.0.smithi151.stdout:2851: done (1 left) 2024-09-13T14:32:56.599 INFO:tasks.rados.rados.0.smithi151.stdout:2853: done (0 left) 2024-09-13T14:32:56.599 INFO:tasks.rados.rados.0.smithi151.stdout:2854: read oid 48 snap -1 2024-09-13T14:32:56.599 INFO:tasks.rados.rados.0.smithi151.stdout:2854: expect (ObjNum 854 snap 295 seq_num 854) 2024-09-13T14:32:56.599 INFO:tasks.rados.rados.0.smithi151.stdout:2855: delete oid 23 current snap is 364 2024-09-13T14:32:56.602 INFO:tasks.rados.rados.0.smithi151.stdout:2855: done (1 left) 2024-09-13T14:32:56.602 INFO:tasks.rados.rados.0.smithi151.stdout:2856: read oid 5 snap -1 2024-09-13T14:32:56.602 INFO:tasks.rados.rados.0.smithi151.stdout:2856: expect deleted 2024-09-13T14:32:56.602 INFO:tasks.rados.rados.0.smithi151.stdout:2857: read oid 23 snap 354 2024-09-13T14:32:56.602 INFO:tasks.rados.rados.0.smithi151.stdout:2857: expect deleted 2024-09-13T14:32:56.602 INFO:tasks.rados.rados.0.smithi151.stdout:2858: snap_remove snap 339 2024-09-13T14:32:56.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:56 smithi184 ceph-mon[92527]: pgmap v795: 89 pgs: 89 active+clean; 202 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 16 op/s 2024-09-13T14:32:56.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:56 smithi184 ceph-mon[92527]: osdmap e785: 8 total, 8 up, 8 in 2024-09-13T14:32:56.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:56 smithi151 ceph-mon[104450]: pgmap v795: 89 pgs: 89 active+clean; 202 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 16 op/s 2024-09-13T14:32:56.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:56 smithi151 ceph-mon[104450]: osdmap e785: 8 total, 8 up, 8 in 2024-09-13T14:32:56.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:56 smithi151 ceph-mon[108966]: pgmap v795: 89 pgs: 89 active+clean; 202 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 16 op/s 2024-09-13T14:32:56.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:56 smithi151 ceph-mon[108966]: osdmap e785: 8 total, 8 up, 8 in 2024-09-13T14:32:57.611 INFO:tasks.rados.rados.0.smithi151.stdout:2858: done (3 left) 2024-09-13T14:32:57.611 INFO:tasks.rados.rados.0.smithi151.stdout:2859: write oid 43 current snap is 364 2024-09-13T14:32:57.612 INFO:tasks.rados.rados.0.smithi151.stdout:2859: seq_num 1071 ranges {400297=481099,1639156=361141} 2024-09-13T14:32:57.617 INFO:tasks.rados.rados.0.smithi151.stdout:2859: writing smithi15186934-43 from 400297 to 881396 tid 1 2024-09-13T14:32:57.624 INFO:tasks.rados.rados.0.smithi151.stdout:2859: writing smithi15186934-43 from 1639156 to 2000297 tid 2 2024-09-13T14:32:57.624 INFO:tasks.rados.rados.0.smithi151.stdout:2860: snap_create 2024-09-13T14:32:57.624 INFO:tasks.rados.rados.0.smithi151.stdout:2854: expect (ObjNum 854 snap 295 seq_num 854) 2024-09-13T14:32:57.702 INFO:tasks.rados.rados.0.smithi151.stdout:2859: finishing write tid 1 to smithi15186934-43 2024-09-13T14:32:57.702 INFO:tasks.rados.rados.0.smithi151.stdout:2859: finishing write tid 2 to smithi15186934-43 2024-09-13T14:32:57.702 INFO:tasks.rados.rados.0.smithi151.stdout:2859: finishing write tid 3 to smithi15186934-43 2024-09-13T14:32:57.702 INFO:tasks.rados.rados.0.smithi151.stdout:2859: finishing write tid 4 to smithi15186934-43 2024-09-13T14:32:57.702 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 43 v 1744 (ObjNum 1071 snap 364 seq_num 1071) dirty exists 2024-09-13T14:32:57.702 INFO:tasks.rados.rados.0.smithi151.stdout:2859: left oid 43 (ObjNum 1071 snap 364 seq_num 1071) 2024-09-13T14:32:57.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:57 smithi184 ceph-mon[92527]: osdmap e786: 8 total, 8 up, 8 in 2024-09-13T14:32:57.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:57 smithi151 ceph-mon[104450]: osdmap e786: 8 total, 8 up, 8 in 2024-09-13T14:32:57.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:57 smithi151 ceph-mon[108966]: osdmap e786: 8 total, 8 up, 8 in 2024-09-13T14:32:58.613 INFO:tasks.rados.rados.0.smithi151.stdout:2854: done (4 left) 2024-09-13T14:32:58.613 INFO:tasks.rados.rados.0.smithi151.stdout:2856: done (3 left) 2024-09-13T14:32:58.613 INFO:tasks.rados.rados.0.smithi151.stdout:2857: done (2 left) 2024-09-13T14:32:58.613 INFO:tasks.rados.rados.0.smithi151.stdout:2859: done (1 left) 2024-09-13T14:32:58.613 INFO:tasks.rados.rados.0.smithi151.stdout:2860: done (0 left) 2024-09-13T14:32:58.613 INFO:tasks.rados.rados.0.smithi151.stdout:2861: snap_create 2024-09-13T14:32:58.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:58 smithi184 ceph-mon[92527]: pgmap v798: 89 pgs: 89 active+clean; 209 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 1.6 MiB/s wr, 23 op/s 2024-09-13T14:32:58.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:58 smithi184 ceph-mon[92527]: osdmap e787: 8 total, 8 up, 8 in 2024-09-13T14:32:58.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:58 smithi151 ceph-mon[104450]: pgmap v798: 89 pgs: 89 active+clean; 209 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 1.6 MiB/s wr, 23 op/s 2024-09-13T14:32:58.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:58 smithi151 ceph-mon[104450]: osdmap e787: 8 total, 8 up, 8 in 2024-09-13T14:32:58.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:58 smithi151 ceph-mon[108966]: pgmap v798: 89 pgs: 89 active+clean; 209 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 1.6 MiB/s wr, 23 op/s 2024-09-13T14:32:58.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:58 smithi151 ceph-mon[108966]: osdmap e787: 8 total, 8 up, 8 in 2024-09-13T14:32:59.587 INFO:tasks.rados.rados.0.smithi151.stdout:2861: done (0 left) 2024-09-13T14:32:59.587 INFO:tasks.rados.rados.0.smithi151.stdout:2862: rollback oid 30 current snap is 366 2024-09-13T14:32:59.587 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 30 to 347 2024-09-13T14:32:59.587 INFO:tasks.rados.rados.0.smithi151.stdout:2863: read oid 44 snap -1 2024-09-13T14:32:59.587 INFO:tasks.rados.rados.0.smithi151.stdout:2863: expect (ObjNum 1047 snap 356 seq_num 1047) 2024-09-13T14:32:59.587 INFO:tasks.rados.rados.0.smithi151.stdout:2864: snap_remove snap 353 2024-09-13T14:33:00.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:59 smithi184 ceph-mon[92527]: osdmap e788: 8 total, 8 up, 8 in 2024-09-13T14:33:00.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:32:59 smithi184 ceph-mon[92527]: osdmap e789: 8 total, 8 up, 8 in 2024-09-13T14:33:00.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:59 smithi151 ceph-mon[104450]: osdmap e788: 8 total, 8 up, 8 in 2024-09-13T14:33:00.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:32:59 smithi151 ceph-mon[104450]: osdmap e789: 8 total, 8 up, 8 in 2024-09-13T14:33:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:59 smithi151 ceph-mon[108966]: osdmap e788: 8 total, 8 up, 8 in 2024-09-13T14:33:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:32:59 smithi151 ceph-mon[108966]: osdmap e789: 8 total, 8 up, 8 in 2024-09-13T14:33:00.588 INFO:tasks.rados.rados.0.smithi151.stdout:2864: done (2 left) 2024-09-13T14:33:00.589 INFO:tasks.rados.rados.0.smithi151.stdout:2865: read oid 7 snap -1 2024-09-13T14:33:00.589 INFO:tasks.rados.rados.0.smithi151.stdout:2865: expect (ObjNum 951 snap 327 seq_num 951) 2024-09-13T14:33:00.589 INFO:tasks.rados.rados.0.smithi151.stdout:2866: read oid 48 snap -1 2024-09-13T14:33:00.589 INFO:tasks.rados.rados.0.smithi151.stdout:2866: expect (ObjNum 854 snap 295 seq_num 854) 2024-09-13T14:33:00.589 INFO:tasks.rados.rados.0.smithi151.stdout:2867: rollback oid 50 current snap is 366 2024-09-13T14:33:00.589 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 50 to 347 2024-09-13T14:33:00.589 INFO:tasks.rados.rados.0.smithi151.stdout:2863: expect (ObjNum 1047 snap 356 seq_num 1047) 2024-09-13T14:33:00.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:33:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:33:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:33:00.781 INFO:tasks.rados.rados.0.smithi151.stdout:2862: finishing rollback tid 1 to smithi15186934-30 2024-09-13T14:33:00.781 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 30 v 1099 (ObjNum 129 snap 0 seq_num 3092744192) dirty dne 2024-09-13T14:33:00.781 INFO:tasks.rados.rados.0.smithi151.stdout:2862: done (4 left) 2024-09-13T14:33:00.781 INFO:tasks.rados.rados.0.smithi151.stdout:2863: done (3 left) 2024-09-13T14:33:00.781 INFO:tasks.rados.rados.0.smithi151.stdout:2868: read oid 8 snap -1 2024-09-13T14:33:00.781 INFO:tasks.rados.rados.0.smithi151.stdout:2868: expect (ObjNum 1065 snap 361 seq_num 1065) 2024-09-13T14:33:00.781 INFO:tasks.rados.rados.0.smithi151.stdout:2869: snap_remove snap 365 2024-09-13T14:33:01.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:00 smithi184 ceph-mon[92527]: pgmap v801: 89 pgs: 89 active+clean; 205 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 2.0 MiB/s wr, 20 op/s 2024-09-13T14:33:01.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:00 smithi184 ceph-mon[92527]: osdmap e790: 8 total, 8 up, 8 in 2024-09-13T14:33:01.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:00 smithi151 ceph-mon[108966]: pgmap v801: 89 pgs: 89 active+clean; 205 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 2.0 MiB/s wr, 20 op/s 2024-09-13T14:33:01.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:00 smithi151 ceph-mon[108966]: osdmap e790: 8 total, 8 up, 8 in 2024-09-13T14:33:01.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:00 smithi151 ceph-mon[104450]: pgmap v801: 89 pgs: 89 active+clean; 205 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 2.0 MiB/s wr, 20 op/s 2024-09-13T14:33:01.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:00 smithi151 ceph-mon[104450]: osdmap e790: 8 total, 8 up, 8 in 2024-09-13T14:33:01.626 INFO:tasks.rados.rados.0.smithi151.stdout:2869: done (4 left) 2024-09-13T14:33:01.626 INFO:tasks.rados.rados.0.smithi151.stdout:2870: read oid 41 snap -1 2024-09-13T14:33:01.626 INFO:tasks.rados.rados.0.smithi151.stdout:2870: expect deleted 2024-09-13T14:33:01.627 INFO:tasks.rados.rados.0.smithi151.stdout:2871: write oid 12 current snap is 366 2024-09-13T14:33:01.627 INFO:tasks.rados.rados.0.smithi151.stdout:2871: seq_num 1072 ranges {417104=672701,1813991=203113} 2024-09-13T14:33:01.632 INFO:tasks.rados.rados.0.smithi151.stdout:2871: writing smithi15186934-12 from 417104 to 1089805 tid 1 2024-09-13T14:33:01.638 INFO:tasks.rados.rados.0.smithi151.stdout:2871: writing smithi15186934-12 from 1813991 to 2017104 tid 2 2024-09-13T14:33:01.638 INFO:tasks.rados.rados.0.smithi151.stdout:2872: read oid 40 snap 359 2024-09-13T14:33:01.638 INFO:tasks.rados.rados.0.smithi151.stdout:2872: expect (ObjNum 1031 snap 352 seq_num 1031) 2024-09-13T14:33:01.638 INFO:tasks.rados.rados.0.smithi151.stdout:2873: read oid 11 snap -1 2024-09-13T14:33:01.638 INFO:tasks.rados.rados.0.smithi151.stdout:2873: expect (ObjNum 1054 snap 357 seq_num 1054) 2024-09-13T14:33:01.638 INFO:tasks.rados.rados.0.smithi151.stdout:2874: rollback oid 26 current snap is 366 2024-09-13T14:33:01.639 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 26 to 361 2024-09-13T14:33:01.639 INFO:tasks.rados.rados.0.smithi151.stdout:2867: finishing rollback tid 0 to smithi15186934-50 2024-09-13T14:33:01.639 INFO:tasks.rados.rados.0.smithi151.stdout:2867: finishing rollback tid 1 to smithi15186934-50 2024-09-13T14:33:01.639 INFO:tasks.rados.rados.0.smithi151.stdout:2867: finishing rollback tid 2 to smithi15186934-50 2024-09-13T14:33:01.639 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 50 v 1163 (ObjNum 905 snap 314 seq_num 905) dirty exists 2024-09-13T14:33:01.639 INFO:tasks.rados.rados.0.smithi151.stdout:2866: expect (ObjNum 854 snap 295 seq_num 854) 2024-09-13T14:33:01.720 INFO:tasks.rados.rados.0.smithi151.stdout:2865: expect (ObjNum 951 snap 327 seq_num 951) 2024-09-13T14:33:01.947 INFO:tasks.rados.rados.0.smithi151.stdout:2868: expect (ObjNum 1065 snap 361 seq_num 1065) 2024-09-13T14:33:02.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:01 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:02.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:01 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:02.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:02.134 INFO:tasks.rados.rados.0.smithi151.stdout:2871: finishing write tid 1 to smithi15186934-12 2024-09-13T14:33:02.134 INFO:tasks.rados.rados.0.smithi151.stdout:2874: finishing rollback tid 0 to smithi15186934-26 2024-09-13T14:33:02.134 INFO:tasks.rados.rados.0.smithi151.stdout:2874: finishing rollback tid 1 to smithi15186934-26 2024-09-13T14:33:02.134 INFO:tasks.rados.rados.0.smithi151.stdout:2865: done (8 left) 2024-09-13T14:33:02.134 INFO:tasks.rados.rados.0.smithi151.stdout:2866: done (7 left) 2024-09-13T14:33:02.134 INFO:tasks.rados.rados.0.smithi151.stdout:2867: done (6 left) 2024-09-13T14:33:02.134 INFO:tasks.rados.rados.0.smithi151.stdout:2868: done (5 left) 2024-09-13T14:33:02.134 INFO:tasks.rados.rados.0.smithi151.stdout:2870: done (4 left) 2024-09-13T14:33:02.134 INFO:tasks.rados.rados.0.smithi151.stdout:2875: write oid 2 current snap is 366 2024-09-13T14:33:02.134 INFO:tasks.rados.rados.0.smithi151.stdout:2874: finishing rollback tid 2 to smithi15186934-26 2024-09-13T14:33:02.135 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 1033 (ObjNum 1010 snap 347 seq_num 1010) dirty exists 2024-09-13T14:33:02.135 INFO:tasks.rados.rados.0.smithi151.stdout:2871: finishing write tid 2 to smithi15186934-12 2024-09-13T14:33:02.135 INFO:tasks.rados.rados.0.smithi151.stdout:2871: finishing write tid 3 to smithi15186934-12 2024-09-13T14:33:02.135 INFO:tasks.rados.rados.0.smithi151.stdout:2871: finishing write tid 4 to smithi15186934-12 2024-09-13T14:33:02.135 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 12 v 645 (ObjNum 1072 snap 366 seq_num 1072) dirty exists 2024-09-13T14:33:02.135 INFO:tasks.rados.rados.0.smithi151.stdout:2871: left oid 12 (ObjNum 1072 snap 366 seq_num 1072) 2024-09-13T14:33:02.135 INFO:tasks.rados.rados.0.smithi151.stdout:2875: seq_num 1073 ranges {433911=747950,1756120=277791} 2024-09-13T14:33:02.139 INFO:tasks.rados.rados.0.smithi151.stdout:2875: writing smithi15186934-2 from 433911 to 1181861 tid 1 2024-09-13T14:33:02.144 INFO:tasks.rados.rados.0.smithi151.stdout:2875: writing smithi15186934-2 from 1756120 to 2033911 tid 2 2024-09-13T14:33:02.145 INFO:tasks.rados.rados.0.smithi151.stdout:2871: done (4 left) 2024-09-13T14:33:02.145 INFO:tasks.rados.rados.0.smithi151.stdout:2874: done (3 left) 2024-09-13T14:33:02.145 INFO:tasks.rados.rados.0.smithi151.stdout:2876: read oid 45 snap -1 2024-09-13T14:33:02.145 INFO:tasks.rados.rados.0.smithi151.stdout:2876: expect (ObjNum 965 snap 330 seq_num 965) 2024-09-13T14:33:02.145 INFO:tasks.rados.rados.0.smithi151.stdout:2877: delete oid 49 current snap is 366 2024-09-13T14:33:02.145 INFO:tasks.rados.rados.0.smithi151.stdout:2873: expect (ObjNum 1054 snap 357 seq_num 1054) 2024-09-13T14:33:02.311 INFO:tasks.rados.rados.0.smithi151.stdout:2872: expect (ObjNum 1031 snap 352 seq_num 1031) 2024-09-13T14:33:02.418 INFO:tasks.rados.rados.0.smithi151.stdout:2875: finishing write tid 1 to smithi15186934-2 2024-09-13T14:33:02.419 INFO:tasks.rados.rados.0.smithi151.stdout:2875: finishing write tid 2 to smithi15186934-2 2024-09-13T14:33:02.419 INFO:tasks.rados.rados.0.smithi151.stdout:2875: finishing write tid 3 to smithi15186934-2 2024-09-13T14:33:02.419 INFO:tasks.rados.rados.0.smithi151.stdout:2875: finishing write tid 4 to smithi15186934-2 2024-09-13T14:33:02.419 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 2 v 1064 (ObjNum 1073 snap 366 seq_num 1073) dirty exists 2024-09-13T14:33:02.419 INFO:tasks.rados.rados.0.smithi151.stdout:2875: left oid 2 (ObjNum 1073 snap 366 seq_num 1073) 2024-09-13T14:33:02.419 INFO:tasks.rados.rados.0.smithi151.stdout:2876: expect (ObjNum 965 snap 330 seq_num 965) 2024-09-13T14:33:02.519 INFO:tasks.rados.rados.0.smithi151.stdout:2872: done (4 left) 2024-09-13T14:33:02.520 INFO:tasks.rados.rados.0.smithi151.stdout:2873: done (3 left) 2024-09-13T14:33:02.520 INFO:tasks.rados.rados.0.smithi151.stdout:2875: done (2 left) 2024-09-13T14:33:02.520 INFO:tasks.rados.rados.0.smithi151.stdout:2876: done (1 left) 2024-09-13T14:33:02.520 INFO:tasks.rados.rados.0.smithi151.stdout:2877: done (0 left) 2024-09-13T14:33:02.520 INFO:tasks.rados.rados.0.smithi151.stdout:2878: rollback oid 38 current snap is 366 2024-09-13T14:33:02.520 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 38 to 359 2024-09-13T14:33:02.520 INFO:tasks.rados.rados.0.smithi151.stdout:2879: write oid 1 current snap is 366 2024-09-13T14:33:02.520 INFO:tasks.rados.rados.0.smithi151.stdout:2879: seq_num 1075 ranges {467525=498448,1473084=594379} 2024-09-13T14:33:02.524 INFO:tasks.rados.rados.0.smithi151.stdout:2879: writing smithi15186934-1 from 467525 to 965973 tid 1 2024-09-13T14:33:02.531 INFO:tasks.rados.rados.0.smithi151.stdout:2879: writing smithi15186934-1 from 1473084 to 2067463 tid 2 2024-09-13T14:33:02.531 INFO:tasks.rados.rados.0.smithi151.stdout:2880: snap_remove snap 354 2024-09-13T14:33:02.531 INFO:tasks.rados.rados.0.smithi151.stdout:2878: finishing rollback tid 0 to smithi15186934-38 2024-09-13T14:33:02.531 INFO:tasks.rados.rados.0.smithi151.stdout:2878: finishing rollback tid 1 to smithi15186934-38 2024-09-13T14:33:02.531 INFO:tasks.rados.rados.0.smithi151.stdout:2878: finishing rollback tid 2 to smithi15186934-38 2024-09-13T14:33:02.531 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 1173 (ObjNum 1003 snap 345 seq_num 1003) dirty exists 2024-09-13T14:33:02.531 INFO:tasks.rados.rados.0.smithi151.stdout:2879: finishing write tid 1 to smithi15186934-1 2024-09-13T14:33:02.631 INFO:tasks.rados.rados.0.smithi151.stdout:2878: done (2 left) 2024-09-13T14:33:02.631 INFO:tasks.rados.rados.0.smithi151.stdout:2880: done (1 left) 2024-09-13T14:33:02.632 INFO:tasks.rados.rados.0.smithi151.stdout:2881: delete oid 41 current snap is 366 2024-09-13T14:33:02.632 INFO:tasks.rados.rados.0.smithi151.stdout:2879: finishing write tid 2 to smithi15186934-1 2024-09-13T14:33:02.632 INFO:tasks.rados.rados.0.smithi151.stdout:2879: finishing write tid 3 to smithi15186934-1 2024-09-13T14:33:02.632 INFO:tasks.rados.rados.0.smithi151.stdout:2879: finishing write tid 4 to smithi15186934-1 2024-09-13T14:33:02.632 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 1 v 874 (ObjNum 1075 snap 366 seq_num 1075) dirty exists 2024-09-13T14:33:02.632 INFO:tasks.rados.rados.0.smithi151.stdout:2879: left oid 1 (ObjNum 1075 snap 366 seq_num 1075) 2024-09-13T14:33:02.637 INFO:tasks.rados.rados.0.smithi151.stdout:2879: done (1 left) 2024-09-13T14:33:02.637 INFO:tasks.rados.rados.0.smithi151.stdout:2881: done (0 left) 2024-09-13T14:33:02.637 INFO:tasks.rados.rados.0.smithi151.stdout:2882: snap_remove snap 362 2024-09-13T14:33:03.009 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:02 smithi151 ceph-mon[108966]: pgmap v804: 89 pgs: 89 active+clean; 200 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 934 KiB/s wr, 5 op/s 2024-09-13T14:33:03.010 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:02 smithi151 ceph-mon[108966]: osdmap e791: 8 total, 8 up, 8 in 2024-09-13T14:33:03.010 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:02 smithi151 ceph-mon[104450]: pgmap v804: 89 pgs: 89 active+clean; 200 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 934 KiB/s wr, 5 op/s 2024-09-13T14:33:03.010 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:02 smithi151 ceph-mon[104450]: osdmap e791: 8 total, 8 up, 8 in 2024-09-13T14:33:03.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:02 smithi184 ceph-mon[92527]: pgmap v804: 89 pgs: 89 active+clean; 200 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 934 KiB/s wr, 5 op/s 2024-09-13T14:33:03.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:02 smithi184 ceph-mon[92527]: osdmap e791: 8 total, 8 up, 8 in 2024-09-13T14:33:03.603 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:33:03 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=cleanup t=2024-09-13T14:33:03.30250008Z level=info msg="Completed cleanup jobs" duration=12.937149ms 2024-09-13T14:33:03.639 INFO:tasks.rados.rados.0.smithi151.stdout:2882: done (0 left) 2024-09-13T14:33:03.639 INFO:tasks.rados.rados.0.smithi151.stdout:2883: delete oid 43 current snap is 366 2024-09-13T14:33:03.644 INFO:tasks.rados.rados.0.smithi151.stdout:2883: done (0 left) 2024-09-13T14:33:03.644 INFO:tasks.rados.rados.0.smithi151.stdout:2884: write oid 30 current snap is 366 2024-09-13T14:33:03.644 INFO:tasks.rados.rados.0.smithi151.stdout:2884: seq_num 1078 ranges {517946=724195,1932177=185769} 2024-09-13T14:33:03.651 INFO:tasks.rados.rados.0.smithi151.stdout:2884: writing smithi15186934-30 from 517946 to 1242141 tid 1 2024-09-13T14:33:03.656 INFO:tasks.rados.rados.0.smithi151.stdout:2884: writing smithi15186934-30 from 1932177 to 2117946 tid 2 2024-09-13T14:33:03.656 INFO:tasks.rados.rados.0.smithi151.stdout:2885: write oid 26 current snap is 366 2024-09-13T14:33:03.656 INFO:tasks.rados.rados.0.smithi151.stdout:2884: finishing write tid 1 to smithi15186934-30 2024-09-13T14:33:03.656 INFO:tasks.rados.rados.0.smithi151.stdout:2885: seq_num 1079 ranges {534753=799444,1990659=144094} 2024-09-13T14:33:03.663 INFO:tasks.rados.rados.0.smithi151.stdout:2885: writing smithi15186934-26 from 534753 to 1334197 tid 1 2024-09-13T14:33:03.668 INFO:tasks.rados.rados.0.smithi151.stdout:2885: writing smithi15186934-26 from 1990659 to 2134753 tid 2 2024-09-13T14:33:03.668 INFO:tasks.rados.rados.0.smithi151.stdout:2886: snap_create 2024-09-13T14:33:03.669 INFO:tasks.rados.rados.0.smithi151.stdout:2884: finishing write tid 2 to smithi15186934-30 2024-09-13T14:33:03.669 INFO:tasks.rados.rados.0.smithi151.stdout:2884: finishing write tid 3 to smithi15186934-30 2024-09-13T14:33:03.669 INFO:tasks.rados.rados.0.smithi151.stdout:2884: finishing write tid 4 to smithi15186934-30 2024-09-13T14:33:03.669 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 30 v 1164 (ObjNum 1078 snap 366 seq_num 1078) dirty exists 2024-09-13T14:33:03.669 INFO:tasks.rados.rados.0.smithi151.stdout:2884: left oid 30 (ObjNum 1078 snap 366 seq_num 1078) 2024-09-13T14:33:03.669 INFO:tasks.rados.rados.0.smithi151.stdout:2885: finishing write tid 1 to smithi15186934-26 2024-09-13T14:33:03.671 INFO:tasks.rados.rados.0.smithi151.stdout:2885: finishing write tid 2 to smithi15186934-26 2024-09-13T14:33:03.671 INFO:tasks.rados.rados.0.smithi151.stdout:2885: finishing write tid 3 to smithi15186934-26 2024-09-13T14:33:03.671 INFO:tasks.rados.rados.0.smithi151.stdout:2885: finishing write tid 4 to smithi15186934-26 2024-09-13T14:33:03.671 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 1042 (ObjNum 1079 snap 366 seq_num 1079) dirty exists 2024-09-13T14:33:03.671 INFO:tasks.rados.rados.0.smithi151.stdout:2885: left oid 26 (ObjNum 1079 snap 366 seq_num 1079) 2024-09-13T14:33:04.027 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:03 smithi151 ceph-mon[104450]: osdmap e792: 8 total, 8 up, 8 in 2024-09-13T14:33:04.028 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:03 smithi151 ceph-mon[108966]: osdmap e792: 8 total, 8 up, 8 in 2024-09-13T14:33:04.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:03 smithi184 ceph-mon[92527]: osdmap e792: 8 total, 8 up, 8 in 2024-09-13T14:33:04.589 INFO:tasks.rados.rados.0.smithi151.stdout:2884: done (2 left) 2024-09-13T14:33:04.590 INFO:tasks.rados.rados.0.smithi151.stdout:2885: done (1 left) 2024-09-13T14:33:04.590 INFO:tasks.rados.rados.0.smithi151.stdout:2886: done (0 left) 2024-09-13T14:33:04.590 INFO:tasks.rados.rados.0.smithi151.stdout:2887: delete oid 34 current snap is 367 2024-09-13T14:33:04.594 INFO:tasks.rados.rados.0.smithi151.stdout:2887: done (0 left) 2024-09-13T14:33:04.594 INFO:tasks.rados.rados.0.smithi151.stdout:2888: read oid 43 snap -1 2024-09-13T14:33:04.595 INFO:tasks.rados.rados.0.smithi151.stdout:2888: expect deleted 2024-09-13T14:33:04.595 INFO:tasks.rados.rados.0.smithi151.stdout:2889: delete oid 36 current snap is 367 2024-09-13T14:33:04.598 INFO:tasks.rados.rados.0.smithi151.stdout:2888: done (1 left) 2024-09-13T14:33:04.598 INFO:tasks.rados.rados.0.smithi151.stdout:2889: done (0 left) 2024-09-13T14:33:04.598 INFO:tasks.rados.rados.0.smithi151.stdout:2890: rollback oid 17 current snap is 367 2024-09-13T14:33:04.598 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 17 to 359 2024-09-13T14:33:04.598 INFO:tasks.rados.rados.0.smithi151.stdout:2891: read oid 28 snap 358 2024-09-13T14:33:04.598 INFO:tasks.rados.rados.0.smithi151.stdout:2891: expect (ObjNum 1045 snap 355 seq_num 1045) 2024-09-13T14:33:04.598 INFO:tasks.rados.rados.0.smithi151.stdout:2892: rollback oid 31 current snap is 367 2024-09-13T14:33:04.598 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 31 to 366 2024-09-13T14:33:04.598 INFO:tasks.rados.rados.0.smithi151.stdout:2893: rollback oid 30 current snap is 367 2024-09-13T14:33:04.598 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 30 to 357 2024-09-13T14:33:04.598 INFO:tasks.rados.rados.0.smithi151.stdout:2894: write oid 42 current snap is 367 2024-09-13T14:33:04.598 INFO:tasks.rados.rados.0.smithi151.stdout:2894: seq_num 1082 ranges {585174=741544,1766105=419069} 2024-09-13T14:33:04.605 INFO:tasks.rados.rados.0.smithi151.stdout:2894: writing smithi15186934-42 from 585174 to 1326718 tid 1 2024-09-13T14:33:04.610 INFO:tasks.rados.rados.0.smithi151.stdout:2894: writing smithi15186934-42 from 1766105 to 2185174 tid 2 2024-09-13T14:33:04.610 INFO:tasks.rados.rados.0.smithi151.stdout:2895: read oid 18 snap -1 2024-09-13T14:33:04.610 INFO:tasks.rados.rados.0.smithi151.stdout:2895: expect deleted 2024-09-13T14:33:04.611 INFO:tasks.rados.rados.0.smithi151.stdout:2896: write oid 38 current snap is 367 2024-09-13T14:33:04.611 INFO:tasks.rados.rados.0.smithi151.stdout:2890: finishing rollback tid 1 to smithi15186934-17 2024-09-13T14:33:04.611 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 0 (ObjNum 129 snap 0 seq_num 3093308416) dirty dne 2024-09-13T14:33:04.611 INFO:tasks.rados.rados.0.smithi151.stdout:2896: seq_num 1083 ranges {601981=416793,1424587=492702} 2024-09-13T14:33:04.616 INFO:tasks.rados.rados.0.smithi151.stdout:2896: writing smithi15186934-38 from 601981 to 1018774 tid 1 2024-09-13T14:33:04.621 INFO:tasks.rados.rados.0.smithi151.stdout:2896: writing smithi15186934-38 from 1424587 to 1917289 tid 2 2024-09-13T14:33:04.621 INFO:tasks.rados.rados.0.smithi151.stdout:2890: done (6 left) 2024-09-13T14:33:04.621 INFO:tasks.rados.rados.0.smithi151.stdout:2897: delete oid 43 current snap is 367 2024-09-13T14:33:04.621 INFO:tasks.rados.rados.0.smithi151.stdout:2893: finishing rollback tid 0 to smithi15186934-30 2024-09-13T14:33:04.621 INFO:tasks.rados.rados.0.smithi151.stdout:2893: finishing rollback tid 1 to smithi15186934-30 2024-09-13T14:33:04.621 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 30 v 1168 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-13T14:33:04.621 INFO:tasks.rados.rados.0.smithi151.stdout:2891: expect (ObjNum 1045 snap 355 seq_num 1045) 2024-09-13T14:33:04.744 INFO:tasks.rados.rados.0.smithi151.stdout:2892: finishing rollback tid 1 to smithi15186934-31 2024-09-13T14:33:04.744 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 31 v 0 (ObjNum 129 snap 0 seq_num 3092902392) dirty dne 2024-09-13T14:33:04.744 INFO:tasks.rados.rados.0.smithi151.stdout:2894: finishing write tid 1 to smithi15186934-42 2024-09-13T14:33:04.744 INFO:tasks.rados.rados.0.smithi151.stdout:2891: done (6 left) 2024-09-13T14:33:04.744 INFO:tasks.rados.rados.0.smithi151.stdout:2892: done (5 left) 2024-09-13T14:33:04.744 INFO:tasks.rados.rados.0.smithi151.stdout:2893: done (4 left) 2024-09-13T14:33:04.744 INFO:tasks.rados.rados.0.smithi151.stdout:2897: done (3 left) 2024-09-13T14:33:04.744 INFO:tasks.rados.rados.0.smithi151.stdout:2898: write oid 46 current snap is 367 2024-09-13T14:33:04.744 INFO:tasks.rados.rados.0.smithi151.stdout:2898: seq_num 1085 ranges {635595=567291,1825198=410397} 2024-09-13T14:33:04.750 INFO:tasks.rados.rados.0.smithi151.stdout:2898: writing smithi15186934-46 from 635595 to 1202886 tid 1 2024-09-13T14:33:04.756 INFO:tasks.rados.rados.0.smithi151.stdout:2898: writing smithi15186934-46 from 1825198 to 2235595 tid 2 2024-09-13T14:33:04.756 INFO:tasks.rados.rados.0.smithi151.stdout:2899: read oid 31 snap -1 2024-09-13T14:33:04.756 INFO:tasks.rados.rados.0.smithi151.stdout:2899: expect deleted 2024-09-13T14:33:04.756 INFO:tasks.rados.rados.0.smithi151.stdout:2900: write oid 11 current snap is 367 2024-09-13T14:33:04.756 INFO:tasks.rados.rados.0.smithi151.stdout:2900: seq_num 1086 ranges {652402=642540,1883680=368722} 2024-09-13T14:33:04.763 INFO:tasks.rados.rados.0.smithi151.stdout:2900: writing smithi15186934-11 from 652402 to 1294942 tid 1 2024-09-13T14:33:04.768 INFO:tasks.rados.rados.0.smithi151.stdout:2900: writing smithi15186934-11 from 1883680 to 2252402 tid 2 2024-09-13T14:33:04.768 INFO:tasks.rados.rados.0.smithi151.stdout:2901: snap_remove snap 366 2024-09-13T14:33:04.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:04 smithi151 ceph-mon[108966]: pgmap v807: 89 pgs: 89 active+clean; 175 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 9.2 MiB/s rd, 725 KiB/s wr, 9 op/s 2024-09-13T14:33:04.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:04 smithi151 ceph-mon[108966]: osdmap e793: 8 total, 8 up, 8 in 2024-09-13T14:33:04.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:04 smithi151 ceph-mon[108966]: osdmap e794: 8 total, 8 up, 8 in 2024-09-13T14:33:04.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:04 smithi151 ceph-mon[104450]: pgmap v807: 89 pgs: 89 active+clean; 175 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 9.2 MiB/s rd, 725 KiB/s wr, 9 op/s 2024-09-13T14:33:04.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:04 smithi151 ceph-mon[104450]: osdmap e793: 8 total, 8 up, 8 in 2024-09-13T14:33:04.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:04 smithi151 ceph-mon[104450]: osdmap e794: 8 total, 8 up, 8 in 2024-09-13T14:33:04.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:33:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:33:04] "GET /metrics HTTP/1.1" 200 36511 "" "Prometheus/2.51.0" 2024-09-13T14:33:05.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:04 smithi184 ceph-mon[92527]: pgmap v807: 89 pgs: 89 active+clean; 175 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 9.2 MiB/s rd, 725 KiB/s wr, 9 op/s 2024-09-13T14:33:05.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:04 smithi184 ceph-mon[92527]: osdmap e793: 8 total, 8 up, 8 in 2024-09-13T14:33:05.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:04 smithi184 ceph-mon[92527]: osdmap e794: 8 total, 8 up, 8 in 2024-09-13T14:33:05.646 INFO:tasks.rados.rados.0.smithi151.stdout:2901: done (6 left) 2024-09-13T14:33:05.646 INFO:tasks.rados.rados.0.smithi151.stdout:2902: snap_remove snap 361 2024-09-13T14:33:06.649 INFO:tasks.rados.rados.0.smithi151.stdout:2902: done (6 left) 2024-09-13T14:33:06.649 INFO:tasks.rados.rados.0.smithi151.stdout:2903: delete oid 49 current snap is 367 2024-09-13T14:33:06.649 INFO:tasks.rados.rados.0.smithi151.stdout:2894: finishing write tid 2 to smithi15186934-42 2024-09-13T14:33:06.649 INFO:tasks.rados.rados.0.smithi151.stdout:2894: finishing write tid 3 to smithi15186934-42 2024-09-13T14:33:06.649 INFO:tasks.rados.rados.0.smithi151.stdout:2894: finishing write tid 4 to smithi15186934-42 2024-09-13T14:33:06.649 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 1059 (ObjNum 1082 snap 367 seq_num 1082) dirty exists 2024-09-13T14:33:06.649 INFO:tasks.rados.rados.0.smithi151.stdout:2894: left oid 42 (ObjNum 1082 snap 367 seq_num 1082) 2024-09-13T14:33:06.649 INFO:tasks.rados.rados.0.smithi151.stdout:2896: finishing write tid 1 to smithi15186934-38 2024-09-13T14:33:06.650 INFO:tasks.rados.rados.0.smithi151.stdout:2896: finishing write tid 2 to smithi15186934-38 2024-09-13T14:33:06.650 INFO:tasks.rados.rados.0.smithi151.stdout:2896: finishing write tid 3 to smithi15186934-38 2024-09-13T14:33:06.650 INFO:tasks.rados.rados.0.smithi151.stdout:2896: finishing write tid 4 to smithi15186934-38 2024-09-13T14:33:06.650 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 1185 (ObjNum 1083 snap 367 seq_num 1083) dirty exists 2024-09-13T14:33:06.650 INFO:tasks.rados.rados.0.smithi151.stdout:2896: left oid 38 (ObjNum 1083 snap 367 seq_num 1083) 2024-09-13T14:33:06.650 INFO:tasks.rados.rados.0.smithi151.stdout:2898: finishing write tid 1 to smithi15186934-46 2024-09-13T14:33:06.650 INFO:tasks.rados.rados.0.smithi151.stdout:2898: finishing write tid 2 to smithi15186934-46 2024-09-13T14:33:06.650 INFO:tasks.rados.rados.0.smithi151.stdout:2898: finishing write tid 3 to smithi15186934-46 2024-09-13T14:33:06.650 INFO:tasks.rados.rados.0.smithi151.stdout:2898: finishing write tid 4 to smithi15186934-46 2024-09-13T14:33:06.650 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 46 v 1045 (ObjNum 1085 snap 367 seq_num 1085) dirty exists 2024-09-13T14:33:06.650 INFO:tasks.rados.rados.0.smithi151.stdout:2898: left oid 46 (ObjNum 1085 snap 367 seq_num 1085) 2024-09-13T14:33:06.651 INFO:tasks.rados.rados.0.smithi151.stdout:2900: finishing write tid 1 to smithi15186934-11 2024-09-13T14:33:06.651 INFO:tasks.rados.rados.0.smithi151.stdout:2900: finishing write tid 2 to smithi15186934-11 2024-09-13T14:33:06.651 INFO:tasks.rados.rados.0.smithi151.stdout:2900: finishing write tid 3 to smithi15186934-11 2024-09-13T14:33:06.651 INFO:tasks.rados.rados.0.smithi151.stdout:2900: finishing write tid 4 to smithi15186934-11 2024-09-13T14:33:06.651 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 11 v 679 (ObjNum 1086 snap 367 seq_num 1086) dirty exists 2024-09-13T14:33:06.651 INFO:tasks.rados.rados.0.smithi151.stdout:2900: left oid 11 (ObjNum 1086 snap 367 seq_num 1086) 2024-09-13T14:33:06.657 INFO:tasks.rados.rados.0.smithi151.stdout:2894: done (6 left) 2024-09-13T14:33:06.657 INFO:tasks.rados.rados.0.smithi151.stdout:2895: done (5 left) 2024-09-13T14:33:06.657 INFO:tasks.rados.rados.0.smithi151.stdout:2896: done (4 left) 2024-09-13T14:33:06.657 INFO:tasks.rados.rados.0.smithi151.stdout:2898: done (3 left) 2024-09-13T14:33:06.657 INFO:tasks.rados.rados.0.smithi151.stdout:2899: done (2 left) 2024-09-13T14:33:06.657 INFO:tasks.rados.rados.0.smithi151.stdout:2900: done (1 left) 2024-09-13T14:33:06.657 INFO:tasks.rados.rados.0.smithi151.stdout:2903: done (0 left) 2024-09-13T14:33:06.657 INFO:tasks.rados.rados.0.smithi151.stdout:2904: read oid 32 snap -1 2024-09-13T14:33:06.657 INFO:tasks.rados.rados.0.smithi151.stdout:2904: expect (ObjNum 925 snap 318 seq_num 925) 2024-09-13T14:33:06.657 INFO:tasks.rados.rados.0.smithi151.stdout:2905: read oid 12 snap -1 2024-09-13T14:33:06.657 INFO:tasks.rados.rados.0.smithi151.stdout:2905: expect (ObjNum 1072 snap 366 seq_num 1072) 2024-09-13T14:33:06.657 INFO:tasks.rados.rados.0.smithi151.stdout:2906: read oid 39 snap -1 2024-09-13T14:33:06.658 INFO:tasks.rados.rados.0.smithi151.stdout:2906: expect (ObjNum 1066 snap 361 seq_num 1066) 2024-09-13T14:33:06.658 INFO:tasks.rados.rados.0.smithi151.stdout:2907: write oid 50 current snap is 367 2024-09-13T14:33:06.658 INFO:tasks.rados.rados.0.smithi151.stdout:2907: seq_num 1088 ranges {686016=509391,1716997=569019} 2024-09-13T14:33:06.663 INFO:tasks.rados.rados.0.smithi151.stdout:2907: writing smithi15186934-50 from 686016 to 1195407 tid 1 2024-09-13T14:33:06.671 INFO:tasks.rados.rados.0.smithi151.stdout:2907: writing smithi15186934-50 from 1716997 to 2286016 tid 2 2024-09-13T14:33:06.671 INFO:tasks.rados.rados.0.smithi151.stdout:2908: delete oid 11 current snap is 367 2024-09-13T14:33:06.671 INFO:tasks.rados.rados.0.smithi151.stdout:2904: expect (ObjNum 925 snap 318 seq_num 925) 2024-09-13T14:33:06.884 INFO:tasks.rados.rados.0.smithi151.stdout:2905: expect (ObjNum 1072 snap 366 seq_num 1072) 2024-09-13T14:33:06.967 INFO:tasks.rados.rados.0.smithi151.stdout:2906: expect (ObjNum 1066 snap 361 seq_num 1066) 2024-09-13T14:33:07.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:06 smithi184 ceph-mon[92527]: pgmap v810: 89 pgs: 89 active+clean; 176 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 6.5 MiB/s rd, 456 KiB/s wr, 7 op/s 2024-09-13T14:33:07.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:06 smithi184 ceph-mon[92527]: osdmap e795: 8 total, 8 up, 8 in 2024-09-13T14:33:07.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:06 smithi151 ceph-mon[104450]: pgmap v810: 89 pgs: 89 active+clean; 176 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 6.5 MiB/s rd, 456 KiB/s wr, 7 op/s 2024-09-13T14:33:07.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:06 smithi151 ceph-mon[104450]: osdmap e795: 8 total, 8 up, 8 in 2024-09-13T14:33:07.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:06 smithi151 ceph-mon[108966]: pgmap v810: 89 pgs: 89 active+clean; 176 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 6.5 MiB/s rd, 456 KiB/s wr, 7 op/s 2024-09-13T14:33:07.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:06 smithi151 ceph-mon[108966]: osdmap e795: 8 total, 8 up, 8 in 2024-09-13T14:33:07.201 INFO:tasks.rados.rados.0.smithi151.stdout:2907: finishing write tid 1 to smithi15186934-50 2024-09-13T14:33:07.201 INFO:tasks.rados.rados.0.smithi151.stdout:2907: finishing write tid 2 to smithi15186934-50 2024-09-13T14:33:07.202 INFO:tasks.rados.rados.0.smithi151.stdout:2907: finishing write tid 3 to smithi15186934-50 2024-09-13T14:33:07.202 INFO:tasks.rados.rados.0.smithi151.stdout:2907: finishing write tid 4 to smithi15186934-50 2024-09-13T14:33:07.202 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 50 v 1194 (ObjNum 1088 snap 367 seq_num 1088) dirty exists 2024-09-13T14:33:07.202 INFO:tasks.rados.rados.0.smithi151.stdout:2907: left oid 50 (ObjNum 1088 snap 367 seq_num 1088) 2024-09-13T14:33:07.202 INFO:tasks.rados.rados.0.smithi151.stdout:2904: done (4 left) 2024-09-13T14:33:07.202 INFO:tasks.rados.rados.0.smithi151.stdout:2905: done (3 left) 2024-09-13T14:33:07.202 INFO:tasks.rados.rados.0.smithi151.stdout:2906: done (2 left) 2024-09-13T14:33:07.202 INFO:tasks.rados.rados.0.smithi151.stdout:2907: done (1 left) 2024-09-13T14:33:07.202 INFO:tasks.rados.rados.0.smithi151.stdout:2908: done (0 left) 2024-09-13T14:33:07.202 INFO:tasks.rados.rados.0.smithi151.stdout:2909: write oid 13 current snap is 367 2024-09-13T14:33:07.202 INFO:tasks.rados.rados.0.smithi151.stdout:2909: seq_num 1090 ranges {719630=659889,2117608=202022} 2024-09-13T14:33:07.208 INFO:tasks.rados.rados.0.smithi151.stdout:2909: writing smithi15186934-13 from 719630 to 1379519 tid 1 2024-09-13T14:33:07.213 INFO:tasks.rados.rados.0.smithi151.stdout:2909: writing smithi15186934-13 from 2117608 to 2319630 tid 2 2024-09-13T14:33:07.214 INFO:tasks.rados.rados.0.smithi151.stdout:2910: write oid 41 current snap is 367 2024-09-13T14:33:07.214 INFO:tasks.rados.rados.0.smithi151.stdout:2910: seq_num 1091 ranges {736437=735138,2176090=160347} 2024-09-13T14:33:07.221 INFO:tasks.rados.rados.0.smithi151.stdout:2910: writing smithi15186934-41 from 736437 to 1471575 tid 1 2024-09-13T14:33:07.227 INFO:tasks.rados.rados.0.smithi151.stdout:2910: writing smithi15186934-41 from 2176090 to 2336437 tid 2 2024-09-13T14:33:07.227 INFO:tasks.rados.rados.0.smithi151.stdout:2911: rollback oid 12 current snap is 367 2024-09-13T14:33:07.227 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 12 to 357 2024-09-13T14:33:07.227 INFO:tasks.rados.rados.0.smithi151.stdout:2909: finishing write tid 1 to smithi15186934-13 2024-09-13T14:33:07.227 INFO:tasks.rados.rados.0.smithi151.stdout:2909: finishing write tid 2 to smithi15186934-13 2024-09-13T14:33:07.227 INFO:tasks.rados.rados.0.smithi151.stdout:2909: finishing write tid 3 to smithi15186934-13 2024-09-13T14:33:07.227 INFO:tasks.rados.rados.0.smithi151.stdout:2912: read oid 11 snap -1 2024-09-13T14:33:07.227 INFO:tasks.rados.rados.0.smithi151.stdout:2912: expect deleted 2024-09-13T14:33:07.227 INFO:tasks.rados.rados.0.smithi151.stdout:2913: rollback oid 16 current snap is 367 2024-09-13T14:33:07.227 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 16 to 357 2024-09-13T14:33:07.227 INFO:tasks.rados.rados.0.smithi151.stdout:2909: finishing write tid 4 to smithi15186934-13 2024-09-13T14:33:07.228 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 13 v 1076 (ObjNum 1090 snap 367 seq_num 1090) dirty exists 2024-09-13T14:33:07.228 INFO:tasks.rados.rados.0.smithi151.stdout:2909: left oid 13 (ObjNum 1090 snap 367 seq_num 1090) 2024-09-13T14:33:07.228 INFO:tasks.rados.rados.0.smithi151.stdout:2910: finishing write tid 1 to smithi15186934-41 2024-09-13T14:33:07.228 INFO:tasks.rados.rados.0.smithi151.stdout:2909: done (4 left) 2024-09-13T14:33:07.228 INFO:tasks.rados.rados.0.smithi151.stdout:2914: write oid 39 current snap is 367 2024-09-13T14:33:07.228 INFO:tasks.rados.rados.0.smithi151.stdout:2914: seq_num 1092 ranges {753244=410387,1834572=518672} 2024-09-13T14:33:07.233 INFO:tasks.rados.rados.0.smithi151.stdout:2914: writing smithi15186934-39 from 753244 to 1163631 tid 1 2024-09-13T14:33:07.240 INFO:tasks.rados.rados.0.smithi151.stdout:2914: writing smithi15186934-39 from 1834572 to 2353244 tid 2 2024-09-13T14:33:07.240 INFO:tasks.rados.rados.0.smithi151.stdout:2915: rollback oid 25 current snap is 367 2024-09-13T14:33:07.240 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 25 to 359 2024-09-13T14:33:07.240 INFO:tasks.rados.rados.0.smithi151.stdout:2916: write oid 3 current snap is 367 2024-09-13T14:33:07.241 INFO:tasks.rados.rados.0.smithi151.stdout:2916: seq_num 1093 ranges {770051=485636,1776701=511047} 2024-09-13T14:33:07.246 INFO:tasks.rados.rados.0.smithi151.stdout:2916: writing smithi15186934-3 from 770051 to 1255687 tid 1 2024-09-13T14:33:07.252 INFO:tasks.rados.rados.0.smithi151.stdout:2916: writing smithi15186934-3 from 1776701 to 2287748 tid 2 2024-09-13T14:33:07.253 INFO:tasks.rados.rados.0.smithi151.stdout:2912: done (6 left) 2024-09-13T14:33:07.253 INFO:tasks.rados.rados.0.smithi151.stdout:2917: write oid 14 current snap is 367 2024-09-13T14:33:07.253 INFO:tasks.rados.rados.0.smithi151.stdout:2911: finishing rollback tid 0 to smithi15186934-12 2024-09-13T14:33:07.253 INFO:tasks.rados.rados.0.smithi151.stdout:2910: finishing write tid 2 to smithi15186934-41 2024-09-13T14:33:07.253 INFO:tasks.rados.rados.0.smithi151.stdout:2917: seq_num 1094 ranges {786858=560885,1835183=551675} 2024-09-13T14:33:07.259 INFO:tasks.rados.rados.0.smithi151.stdout:2917: writing smithi15186934-14 from 786858 to 1347743 tid 1 2024-09-13T14:33:07.266 INFO:tasks.rados.rados.0.smithi151.stdout:2917: writing smithi15186934-14 from 1835183 to 2386858 tid 2 2024-09-13T14:33:07.266 INFO:tasks.rados.rados.0.smithi151.stdout:2918: rollback oid 2 current snap is 367 2024-09-13T14:33:07.266 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 2 to 360 2024-09-13T14:33:07.266 INFO:tasks.rados.rados.0.smithi151.stdout:2911: finishing rollback tid 1 to smithi15186934-12 2024-09-13T14:33:07.266 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 12 v 653 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-13T14:33:07.266 INFO:tasks.rados.rados.0.smithi151.stdout:2910: finishing write tid 3 to smithi15186934-41 2024-09-13T14:33:07.266 INFO:tasks.rados.rados.0.smithi151.stdout:2911: done (7 left) 2024-09-13T14:33:07.266 INFO:tasks.rados.rados.0.smithi151.stdout:2919: delete oid 35 current snap is 367 2024-09-13T14:33:07.266 INFO:tasks.rados.rados.0.smithi151.stdout:2910: finishing write tid 4 to smithi15186934-41 2024-09-13T14:33:07.266 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 263 (ObjNum 1091 snap 367 seq_num 1091) dirty exists 2024-09-13T14:33:07.266 INFO:tasks.rados.rados.0.smithi151.stdout:2910: left oid 41 (ObjNum 1091 snap 367 seq_num 1091) 2024-09-13T14:33:07.267 INFO:tasks.rados.rados.0.smithi151.stdout:2913: finishing rollback tid 0 to smithi15186934-16 2024-09-13T14:33:07.267 INFO:tasks.rados.rados.0.smithi151.stdout:2913: finishing rollback tid 1 to smithi15186934-16 2024-09-13T14:33:07.267 INFO:tasks.rados.rados.0.smithi151.stdout:2913: finishing rollback tid 2 to smithi15186934-16 2024-09-13T14:33:07.267 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 16 v 745 (ObjNum 1021 snap 349 seq_num 1021) dirty exists 2024-09-13T14:33:07.267 INFO:tasks.rados.rados.0.smithi151.stdout:2914: finishing write tid 1 to smithi15186934-39 2024-09-13T14:33:07.267 INFO:tasks.rados.rados.0.smithi151.stdout:2915: finishing rollback tid 1 to smithi15186934-25 2024-09-13T14:33:07.267 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 1405 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-13T14:33:07.267 INFO:tasks.rados.rados.0.smithi151.stdout:2914: finishing write tid 2 to smithi15186934-39 2024-09-13T14:33:07.267 INFO:tasks.rados.rados.0.smithi151.stdout:2914: finishing write tid 3 to smithi15186934-39 2024-09-13T14:33:07.267 INFO:tasks.rados.rados.0.smithi151.stdout:2914: finishing write tid 4 to smithi15186934-39 2024-09-13T14:33:07.267 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 1198 (ObjNum 1092 snap 367 seq_num 1092) dirty exists 2024-09-13T14:33:07.267 INFO:tasks.rados.rados.0.smithi151.stdout:2914: left oid 39 (ObjNum 1092 snap 367 seq_num 1092) 2024-09-13T14:33:07.267 INFO:tasks.rados.rados.0.smithi151.stdout:2916: finishing write tid 1 to smithi15186934-3 2024-09-13T14:33:07.267 INFO:tasks.rados.rados.0.smithi151.stdout:2916: finishing write tid 2 to smithi15186934-3 2024-09-13T14:33:07.268 INFO:tasks.rados.rados.0.smithi151.stdout:2916: finishing write tid 3 to smithi15186934-3 2024-09-13T14:33:07.268 INFO:tasks.rados.rados.0.smithi151.stdout:2916: finishing write tid 4 to smithi15186934-3 2024-09-13T14:33:07.268 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 1174 (ObjNum 1093 snap 367 seq_num 1093) dirty exists 2024-09-13T14:33:07.268 INFO:tasks.rados.rados.0.smithi151.stdout:2916: left oid 3 (ObjNum 1093 snap 367 seq_num 1093) 2024-09-13T14:33:07.268 INFO:tasks.rados.rados.0.smithi151.stdout:2917: finishing write tid 1 to smithi15186934-14 2024-09-13T14:33:07.269 INFO:tasks.rados.rados.0.smithi151.stdout:2918: finishing rollback tid 0 to smithi15186934-2 2024-09-13T14:33:07.269 INFO:tasks.rados.rados.0.smithi151.stdout:2918: finishing rollback tid 1 to smithi15186934-2 2024-09-13T14:33:07.270 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 2 v 1078 (ObjNum 129 snap 0 seq_num 3092864624) dirty dne 2024-09-13T14:33:07.270 INFO:tasks.rados.rados.0.smithi151.stdout:2910: done (7 left) 2024-09-13T14:33:07.270 INFO:tasks.rados.rados.0.smithi151.stdout:2913: done (6 left) 2024-09-13T14:33:07.270 INFO:tasks.rados.rados.0.smithi151.stdout:2914: done (5 left) 2024-09-13T14:33:07.270 INFO:tasks.rados.rados.0.smithi151.stdout:2915: done (4 left) 2024-09-13T14:33:07.270 INFO:tasks.rados.rados.0.smithi151.stdout:2916: done (3 left) 2024-09-13T14:33:07.270 INFO:tasks.rados.rados.0.smithi151.stdout:2918: done (2 left) 2024-09-13T14:33:07.270 INFO:tasks.rados.rados.0.smithi151.stdout:2919: done (1 left) 2024-09-13T14:33:07.270 INFO:tasks.rados.rados.0.smithi151.stdout:2920: rollback oid 31 current snap is 367 2024-09-13T14:33:07.270 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 31 to 357 2024-09-13T14:33:07.270 INFO:tasks.rados.rados.0.smithi151.stdout:2921: read oid 27 snap -1 2024-09-13T14:33:07.271 INFO:tasks.rados.rados.0.smithi151.stdout:2921: expect deleted 2024-09-13T14:33:07.271 INFO:tasks.rados.rados.0.smithi151.stdout:2922: delete oid 38 current snap is 367 2024-09-13T14:33:07.271 INFO:tasks.rados.rados.0.smithi151.stdout:2917: finishing write tid 2 to smithi15186934-14 2024-09-13T14:33:07.271 INFO:tasks.rados.rados.0.smithi151.stdout:2917: finishing write tid 3 to smithi15186934-14 2024-09-13T14:33:07.271 INFO:tasks.rados.rados.0.smithi151.stdout:2917: finishing write tid 4 to smithi15186934-14 2024-09-13T14:33:07.271 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 1191 (ObjNum 1094 snap 367 seq_num 1094) dirty exists 2024-09-13T14:33:07.271 INFO:tasks.rados.rados.0.smithi151.stdout:2917: left oid 14 (ObjNum 1094 snap 367 seq_num 1094) 2024-09-13T14:33:07.271 INFO:tasks.rados.rados.0.smithi151.stdout:2920: finishing rollback tid 1 to smithi15186934-31 2024-09-13T14:33:07.272 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 31 v 0 (ObjNum 129 snap 0 seq_num 3092902392) dirty dne 2024-09-13T14:33:07.272 INFO:tasks.rados.rados.0.smithi151.stdout:2917: done (3 left) 2024-09-13T14:33:07.272 INFO:tasks.rados.rados.0.smithi151.stdout:2920: done (2 left) 2024-09-13T14:33:07.272 INFO:tasks.rados.rados.0.smithi151.stdout:2921: done (1 left) 2024-09-13T14:33:07.272 INFO:tasks.rados.rados.0.smithi151.stdout:2922: done (0 left) 2024-09-13T14:33:07.272 INFO:tasks.rados.rados.0.smithi151.stdout:2923: snap_create 2024-09-13T14:33:07.663 INFO:tasks.rados.rados.0.smithi151.stdout:2923: done (0 left) 2024-09-13T14:33:07.663 INFO:tasks.rados.rados.0.smithi151.stdout:2924: write oid 3 current snap is 368 2024-09-13T14:33:07.663 INFO:tasks.rados.rados.0.smithi151.stdout:2924: seq_num 1097 ranges {437279=502985,1610629=518385} 2024-09-13T14:33:07.668 INFO:tasks.rados.rados.0.smithi151.stdout:2924: writing smithi15186934-3 from 437279 to 940264 tid 1 2024-09-13T14:33:07.675 INFO:tasks.rados.rados.0.smithi151.stdout:2924: writing smithi15186934-3 from 1610629 to 2129014 tid 2 2024-09-13T14:33:07.676 INFO:tasks.rados.rados.0.smithi151.stdout:2925: snap_create 2024-09-13T14:33:07.676 INFO:tasks.rados.rados.0.smithi151.stdout:2924: finishing write tid 1 to smithi15186934-3 2024-09-13T14:33:07.678 INFO:tasks.rados.rados.0.smithi151.stdout:2924: finishing write tid 2 to smithi15186934-3 2024-09-13T14:33:07.678 INFO:tasks.rados.rados.0.smithi151.stdout:2924: finishing write tid 3 to smithi15186934-3 2024-09-13T14:33:07.678 INFO:tasks.rados.rados.0.smithi151.stdout:2924: finishing write tid 4 to smithi15186934-3 2024-09-13T14:33:07.678 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 1178 (ObjNum 1097 snap 368 seq_num 1097) dirty exists 2024-09-13T14:33:07.679 INFO:tasks.rados.rados.0.smithi151.stdout:2924: left oid 3 (ObjNum 1097 snap 368 seq_num 1097) 2024-09-13T14:33:08.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:07 smithi184 ceph-mon[92527]: osdmap e796: 8 total, 8 up, 8 in 2024-09-13T14:33:08.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:07 smithi151 ceph-mon[104450]: osdmap e796: 8 total, 8 up, 8 in 2024-09-13T14:33:08.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:07 smithi151 ceph-mon[108966]: osdmap e796: 8 total, 8 up, 8 in 2024-09-13T14:33:08.669 INFO:tasks.rados.rados.0.smithi151.stdout:2924: done (1 left) 2024-09-13T14:33:08.669 INFO:tasks.rados.rados.0.smithi151.stdout:2925: done (0 left) 2024-09-13T14:33:08.669 INFO:tasks.rados.rados.0.smithi151.stdout:2926: delete oid 47 current snap is 369 2024-09-13T14:33:08.672 INFO:tasks.rados.rados.0.smithi151.stdout:2926: done (0 left) 2024-09-13T14:33:08.672 INFO:tasks.rados.rados.0.smithi151.stdout:2927: rollback oid 10 current snap is 369 2024-09-13T14:33:08.672 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 10 to 357 2024-09-13T14:33:08.672 INFO:tasks.rados.rados.0.smithi151.stdout:2928: snap_create 2024-09-13T14:33:08.677 INFO:tasks.rados.rados.0.smithi151.stdout:2927: finishing rollback tid 0 to smithi15186934-10 2024-09-13T14:33:08.677 INFO:tasks.rados.rados.0.smithi151.stdout:2927: finishing rollback tid 1 to smithi15186934-10 2024-09-13T14:33:08.677 INFO:tasks.rados.rados.0.smithi151.stdout:2927: finishing rollback tid 2 to smithi15186934-10 2024-09-13T14:33:08.677 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 1053 (ObjNum 980 snap 340 seq_num 980) dirty exists 2024-09-13T14:33:09.107 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:08 smithi184 ceph-mon[92527]: pgmap v813: 89 pgs: 89 active+clean; 175 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 1.6 MiB/s wr, 21 op/s 2024-09-13T14:33:09.107 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:08 smithi184 ceph-mon[92527]: osdmap e797: 8 total, 8 up, 8 in 2024-09-13T14:33:09.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:08 smithi151 ceph-mon[104450]: pgmap v813: 89 pgs: 89 active+clean; 175 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 1.6 MiB/s wr, 21 op/s 2024-09-13T14:33:09.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:08 smithi151 ceph-mon[104450]: osdmap e797: 8 total, 8 up, 8 in 2024-09-13T14:33:09.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:08 smithi151 ceph-mon[108966]: pgmap v813: 89 pgs: 89 active+clean; 175 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 1.6 MiB/s wr, 21 op/s 2024-09-13T14:33:09.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:08 smithi151 ceph-mon[108966]: osdmap e797: 8 total, 8 up, 8 in 2024-09-13T14:33:09.590 INFO:tasks.rados.rados.0.smithi151.stdout:2927: done (1 left) 2024-09-13T14:33:09.591 INFO:tasks.rados.rados.0.smithi151.stdout:2928: done (0 left) 2024-09-13T14:33:09.591 INFO:tasks.rados.rados.0.smithi151.stdout:2929: read oid 35 snap -1 2024-09-13T14:33:09.591 INFO:tasks.rados.rados.0.smithi151.stdout:2929: expect deleted 2024-09-13T14:33:09.591 INFO:tasks.rados.rados.0.smithi151.stdout:2930: read oid 48 snap 359 2024-09-13T14:33:09.591 INFO:tasks.rados.rados.0.smithi151.stdout:2930: expect (ObjNum 854 snap 295 seq_num 854) 2024-09-13T14:33:09.591 INFO:tasks.rados.rados.0.smithi151.stdout:2931: write oid 46 current snap is 370 2024-09-13T14:33:09.591 INFO:tasks.rados.rados.0.smithi151.stdout:2931: seq_num 1099 ranges {470893=653483,1727593=522054} 2024-09-13T14:33:09.596 INFO:tasks.rados.rados.0.smithi151.stdout:2931: writing smithi15186934-46 from 470893 to 1124376 tid 1 2024-09-13T14:33:09.603 INFO:tasks.rados.rados.0.smithi151.stdout:2931: writing smithi15186934-46 from 1727593 to 2249647 tid 2 2024-09-13T14:33:09.603 INFO:tasks.rados.rados.0.smithi151.stdout:2932: read oid 4 snap 360 2024-09-13T14:33:09.603 INFO:tasks.rados.rados.0.smithi151.stdout:2932: expect deleted 2024-09-13T14:33:09.603 INFO:tasks.rados.rados.0.smithi151.stdout:2933: write oid 22 current snap is 370 2024-09-13T14:33:09.603 INFO:tasks.rados.rados.0.smithi151.stdout:2933: seq_num 1100 ranges {487700=728732,1786075=665712} 2024-09-13T14:33:09.610 INFO:tasks.rados.rados.0.smithi151.stdout:2933: writing smithi15186934-22 from 487700 to 1216432 tid 1 2024-09-13T14:33:09.617 INFO:tasks.rados.rados.0.smithi151.stdout:2933: writing smithi15186934-22 from 1786075 to 2451787 tid 2 2024-09-13T14:33:09.617 INFO:tasks.rados.rados.0.smithi151.stdout:2934: write oid 49 current snap is 370 2024-09-13T14:33:09.617 INFO:tasks.rados.rados.0.smithi151.stdout:2930: expect (ObjNum 854 snap 295 seq_num 854) 2024-09-13T14:33:09.697 INFO:tasks.rados.rados.0.smithi151.stdout:2931: finishing write tid 1 to smithi15186934-46 2024-09-13T14:33:09.697 INFO:tasks.rados.rados.0.smithi151.stdout:2931: finishing write tid 2 to smithi15186934-46 2024-09-13T14:33:09.697 INFO:tasks.rados.rados.0.smithi151.stdout:2931: finishing write tid 3 to smithi15186934-46 2024-09-13T14:33:09.697 INFO:tasks.rados.rados.0.smithi151.stdout:2931: finishing write tid 4 to smithi15186934-46 2024-09-13T14:33:09.697 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 46 v 1057 (ObjNum 1099 snap 370 seq_num 1099) dirty exists 2024-09-13T14:33:09.697 INFO:tasks.rados.rados.0.smithi151.stdout:2931: left oid 46 (ObjNum 1099 snap 370 seq_num 1099) 2024-09-13T14:33:09.697 INFO:tasks.rados.rados.0.smithi151.stdout:2933: finishing write tid 1 to smithi15186934-22 2024-09-13T14:33:09.697 INFO:tasks.rados.rados.0.smithi151.stdout:2933: finishing write tid 2 to smithi15186934-22 2024-09-13T14:33:09.698 INFO:tasks.rados.rados.0.smithi151.stdout:2933: finishing write tid 3 to smithi15186934-22 2024-09-13T14:33:09.698 INFO:tasks.rados.rados.0.smithi151.stdout:2934: seq_num 1101 ranges {504507=403981,1328204=409370,2252362=252145} 2024-09-13T14:33:09.701 INFO:tasks.rados.rados.0.smithi151.stdout:2934: writing smithi15186934-49 from 504507 to 908488 tid 1 2024-09-13T14:33:09.706 INFO:tasks.rados.rados.0.smithi151.stdout:2934: writing smithi15186934-49 from 1328204 to 1737574 tid 2 2024-09-13T14:33:09.711 INFO:tasks.rados.rados.0.smithi151.stdout:2934: writing smithi15186934-49 from 2252362 to 2504507 tid 3 2024-09-13T14:33:09.711 INFO:tasks.rados.rados.0.smithi151.stdout:2929: done (5 left) 2024-09-13T14:33:09.711 INFO:tasks.rados.rados.0.smithi151.stdout:2930: done (4 left) 2024-09-13T14:33:09.711 INFO:tasks.rados.rados.0.smithi151.stdout:2931: done (3 left) 2024-09-13T14:33:09.711 INFO:tasks.rados.rados.0.smithi151.stdout:2932: done (2 left) 2024-09-13T14:33:09.711 INFO:tasks.rados.rados.0.smithi151.stdout:2935: read oid 24 snap -1 2024-09-13T14:33:09.711 INFO:tasks.rados.rados.0.smithi151.stdout:2935: expect (ObjNum 1057 snap 357 seq_num 1057) 2024-09-13T14:33:09.711 INFO:tasks.rados.rados.0.smithi151.stdout:2936: snap_remove snap 360 2024-09-13T14:33:10.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:09 smithi184 ceph-mon[92527]: osdmap e798: 8 total, 8 up, 8 in 2024-09-13T14:33:10.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:09 smithi184 ceph-mon[92527]: osdmap e799: 8 total, 8 up, 8 in 2024-09-13T14:33:10.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:09 smithi151 ceph-mon[104450]: osdmap e798: 8 total, 8 up, 8 in 2024-09-13T14:33:10.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:09 smithi151 ceph-mon[104450]: osdmap e799: 8 total, 8 up, 8 in 2024-09-13T14:33:10.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:09 smithi151 ceph-mon[108966]: osdmap e798: 8 total, 8 up, 8 in 2024-09-13T14:33:10.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:09 smithi151 ceph-mon[108966]: osdmap e799: 8 total, 8 up, 8 in 2024-09-13T14:33:10.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:33:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:33:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:33:10.684 INFO:tasks.rados.rados.0.smithi151.stdout:2936: done (3 left) 2024-09-13T14:33:10.685 INFO:tasks.rados.rados.0.smithi151.stdout:2937: read oid 31 snap -1 2024-09-13T14:33:10.685 INFO:tasks.rados.rados.0.smithi151.stdout:2937: expect deleted 2024-09-13T14:33:10.685 INFO:tasks.rados.rados.0.smithi151.stdout:2938: read oid 38 snap -1 2024-09-13T14:33:10.685 INFO:tasks.rados.rados.0.smithi151.stdout:2938: expect deleted 2024-09-13T14:33:10.685 INFO:tasks.rados.rados.0.smithi151.stdout:2939: delete oid 47 current snap is 370 2024-09-13T14:33:10.685 INFO:tasks.rados.rados.0.smithi151.stdout:2933: finishing write tid 4 to smithi15186934-22 2024-09-13T14:33:10.685 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 22 v 568 (ObjNum 1100 snap 370 seq_num 1100) dirty exists 2024-09-13T14:33:10.685 INFO:tasks.rados.rados.0.smithi151.stdout:2933: left oid 22 (ObjNum 1100 snap 370 seq_num 1100) 2024-09-13T14:33:10.685 INFO:tasks.rados.rados.0.smithi151.stdout:2934: finishing write tid 1 to smithi15186934-49 2024-09-13T14:33:10.685 INFO:tasks.rados.rados.0.smithi151.stdout:2934: finishing write tid 2 to smithi15186934-49 2024-09-13T14:33:10.685 INFO:tasks.rados.rados.0.smithi151.stdout:2934: finishing write tid 3 to smithi15186934-49 2024-09-13T14:33:10.686 INFO:tasks.rados.rados.0.smithi151.stdout:2934: finishing write tid 4 to smithi15186934-49 2024-09-13T14:33:10.686 INFO:tasks.rados.rados.0.smithi151.stdout:2934: finishing write tid 5 to smithi15186934-49 2024-09-13T14:33:10.686 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 49 v 1282 (ObjNum 1101 snap 370 seq_num 1101) dirty exists 2024-09-13T14:33:10.686 INFO:tasks.rados.rados.0.smithi151.stdout:2934: left oid 49 (ObjNum 1101 snap 370 seq_num 1101) 2024-09-13T14:33:10.686 INFO:tasks.rados.rados.0.smithi151.stdout:2935: expect (ObjNum 1057 snap 357 seq_num 1057) 2024-09-13T14:33:10.895 INFO:tasks.rados.rados.0.smithi151.stdout:2933: done (5 left) 2024-09-13T14:33:10.895 INFO:tasks.rados.rados.0.smithi151.stdout:2934: done (4 left) 2024-09-13T14:33:10.895 INFO:tasks.rados.rados.0.smithi151.stdout:2935: done (3 left) 2024-09-13T14:33:10.895 INFO:tasks.rados.rados.0.smithi151.stdout:2937: done (2 left) 2024-09-13T14:33:10.895 INFO:tasks.rados.rados.0.smithi151.stdout:2938: done (1 left) 2024-09-13T14:33:10.896 INFO:tasks.rados.rados.0.smithi151.stdout:2939: done (0 left) 2024-09-13T14:33:10.896 INFO:tasks.rados.rados.0.smithi151.stdout:2940: read oid 41 snap -1 2024-09-13T14:33:10.896 INFO:tasks.rados.rados.0.smithi151.stdout:2940: expect (ObjNum 1091 snap 367 seq_num 1091) 2024-09-13T14:33:10.896 INFO:tasks.rados.rados.0.smithi151.stdout:2941: write oid 50 current snap is 370 2024-09-13T14:33:10.896 INFO:tasks.rados.rados.0.smithi151.stdout:2941: seq_num 1103 ranges {538121=670832,1961521=413039} 2024-09-13T14:33:10.901 INFO:tasks.rados.rados.0.smithi151.stdout:2941: writing smithi15186934-50 from 538121 to 1208953 tid 1 2024-09-13T14:33:10.908 INFO:tasks.rados.rados.0.smithi151.stdout:2941: writing smithi15186934-50 from 1961521 to 2374560 tid 2 2024-09-13T14:33:10.908 INFO:tasks.rados.rados.0.smithi151.stdout:2942: write oid 20 current snap is 370 2024-09-13T14:33:10.908 INFO:tasks.rados.rados.0.smithi151.stdout:2942: seq_num 1104 ranges {554928=746081,2020003=534925} 2024-09-13T14:33:10.915 INFO:tasks.rados.rados.0.smithi151.stdout:2942: writing smithi15186934-20 from 554928 to 1301009 tid 1 2024-09-13T14:33:10.921 INFO:tasks.rados.rados.0.smithi151.stdout:2942: writing smithi15186934-20 from 2020003 to 2554928 tid 2 2024-09-13T14:33:10.922 INFO:tasks.rados.rados.0.smithi151.stdout:2943: delete oid 12 current snap is 370 2024-09-13T14:33:10.922 INFO:tasks.rados.rados.0.smithi151.stdout:2940: expect (ObjNum 1091 snap 367 seq_num 1091) 2024-09-13T14:33:10.999 INFO:tasks.rados.rados.0.smithi151.stdout:2941: finishing write tid 1 to smithi15186934-50 2024-09-13T14:33:10.999 INFO:tasks.rados.rados.0.smithi151.stdout:2941: finishing write tid 2 to smithi15186934-50 2024-09-13T14:33:10.999 INFO:tasks.rados.rados.0.smithi151.stdout:2941: finishing write tid 3 to smithi15186934-50 2024-09-13T14:33:10.999 INFO:tasks.rados.rados.0.smithi151.stdout:2941: finishing write tid 4 to smithi15186934-50 2024-09-13T14:33:10.999 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 50 v 1209 (ObjNum 1103 snap 370 seq_num 1103) dirty exists 2024-09-13T14:33:10.999 INFO:tasks.rados.rados.0.smithi151.stdout:2941: left oid 50 (ObjNum 1103 snap 370 seq_num 1103) 2024-09-13T14:33:10.999 INFO:tasks.rados.rados.0.smithi151.stdout:2942: finishing write tid 1 to smithi15186934-20 2024-09-13T14:33:11.000 INFO:tasks.rados.rados.0.smithi151.stdout:2942: finishing write tid 2 to smithi15186934-20 2024-09-13T14:33:11.000 INFO:tasks.rados.rados.0.smithi151.stdout:2942: finishing write tid 3 to smithi15186934-20 2024-09-13T14:33:11.000 INFO:tasks.rados.rados.0.smithi151.stdout:2940: done (3 left) 2024-09-13T14:33:11.000 INFO:tasks.rados.rados.0.smithi151.stdout:2941: done (2 left) 2024-09-13T14:33:11.000 INFO:tasks.rados.rados.0.smithi151.stdout:2943: done (1 left) 2024-09-13T14:33:11.000 INFO:tasks.rados.rados.0.smithi151.stdout:2944: write oid 4 current snap is 370 2024-09-13T14:33:11.000 INFO:tasks.rados.rados.0.smithi151.stdout:2942: finishing write tid 4 to smithi15186934-20 2024-09-13T14:33:11.000 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 751 (ObjNum 1104 snap 370 seq_num 1104) dirty exists 2024-09-13T14:33:11.000 INFO:tasks.rados.rados.0.smithi151.stdout:2942: left oid 20 (ObjNum 1104 snap 370 seq_num 1104) 2024-09-13T14:33:11.000 INFO:tasks.rados.rados.0.smithi151.stdout:2944: seq_num 1106 ranges {588542=496579,1620614=676719} 2024-09-13T14:33:11.004 INFO:tasks.rados.rados.0.smithi151.stdout:2944: writing smithi15186934-4 from 588542 to 1085121 tid 1 2024-09-13T14:33:11.011 INFO:tasks.rados.rados.0.smithi151.stdout:2944: writing smithi15186934-4 from 1620614 to 2297333 tid 2 2024-09-13T14:33:11.011 INFO:tasks.rados.rados.0.smithi151.stdout:2942: done (1 left) 2024-09-13T14:33:11.011 INFO:tasks.rados.rados.0.smithi151.stdout:2945: snap_create 2024-09-13T14:33:11.011 INFO:tasks.rados.rados.0.smithi151.stdout:2944: finishing write tid 1 to smithi15186934-4 2024-09-13T14:33:11.015 INFO:tasks.rados.rados.0.smithi151.stdout:2944: finishing write tid 2 to smithi15186934-4 2024-09-13T14:33:11.015 INFO:tasks.rados.rados.0.smithi151.stdout:2944: finishing write tid 3 to smithi15186934-4 2024-09-13T14:33:11.015 INFO:tasks.rados.rados.0.smithi151.stdout:2944: finishing write tid 4 to smithi15186934-4 2024-09-13T14:33:11.015 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 1423 (ObjNum 1106 snap 370 seq_num 1106) dirty exists 2024-09-13T14:33:11.015 INFO:tasks.rados.rados.0.smithi151.stdout:2944: left oid 4 (ObjNum 1106 snap 370 seq_num 1106) 2024-09-13T14:33:11.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:10 smithi184 ceph-mon[92527]: pgmap v816: 89 pgs: 89 active+clean; 178 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 2.3 MiB/s wr, 27 op/s 2024-09-13T14:33:11.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:10 smithi151 ceph-mon[104450]: pgmap v816: 89 pgs: 89 active+clean; 178 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 2.3 MiB/s wr, 27 op/s 2024-09-13T14:33:11.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:10 smithi151 ceph-mon[108966]: pgmap v816: 89 pgs: 89 active+clean; 178 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 2.3 MiB/s wr, 27 op/s 2024-09-13T14:33:11.698 INFO:tasks.rados.rados.0.smithi151.stdout:2944: done (1 left) 2024-09-13T14:33:11.698 INFO:tasks.rados.rados.0.smithi151.stdout:2945: done (0 left) 2024-09-13T14:33:11.698 INFO:tasks.rados.rados.0.smithi151.stdout:2946: write oid 23 current snap is 371 2024-09-13T14:33:11.698 INFO:tasks.rados.rados.0.smithi151.stdout:2946: seq_num 1107 ranges {605349=571828,1679096=420377} 2024-09-13T14:33:11.704 INFO:tasks.rados.rados.0.smithi151.stdout:2946: writing smithi15186934-23 from 605349 to 1177177 tid 1 2024-09-13T14:33:11.709 INFO:tasks.rados.rados.0.smithi151.stdout:2946: writing smithi15186934-23 from 1679096 to 2099473 tid 2 2024-09-13T14:33:11.709 INFO:tasks.rados.rados.0.smithi151.stdout:2947: snap_remove snap 368 2024-09-13T14:33:12.005 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:11 smithi184 ceph-mon[92527]: osdmap e800: 8 total, 8 up, 8 in 2024-09-13T14:33:12.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:11 smithi151 ceph-mon[108966]: osdmap e800: 8 total, 8 up, 8 in 2024-09-13T14:33:12.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:11 smithi151 ceph-mon[104450]: osdmap e800: 8 total, 8 up, 8 in 2024-09-13T14:33:12.703 INFO:tasks.rados.rados.0.smithi151.stdout:2947: done (1 left) 2024-09-13T14:33:12.703 INFO:tasks.rados.rados.0.smithi151.stdout:2948: snap_create 2024-09-13T14:33:12.703 INFO:tasks.rados.rados.0.smithi151.stdout:2946: finishing write tid 1 to smithi15186934-23 2024-09-13T14:33:12.703 INFO:tasks.rados.rados.0.smithi151.stdout:2946: finishing write tid 2 to smithi15186934-23 2024-09-13T14:33:12.703 INFO:tasks.rados.rados.0.smithi151.stdout:2946: finishing write tid 3 to smithi15186934-23 2024-09-13T14:33:12.703 INFO:tasks.rados.rados.0.smithi151.stdout:2946: finishing write tid 4 to smithi15186934-23 2024-09-13T14:33:12.703 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 23 v 1292 (ObjNum 1107 snap 371 seq_num 1107) dirty exists 2024-09-13T14:33:12.703 INFO:tasks.rados.rados.0.smithi151.stdout:2946: left oid 23 (ObjNum 1107 snap 371 seq_num 1107) 2024-09-13T14:33:13.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:12 smithi184 ceph-mon[92527]: pgmap v819: 89 pgs: 89 active+clean; 177 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 8.0 MiB/s rd, 1.7 MiB/s wr, 16 op/s 2024-09-13T14:33:13.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:12 smithi184 ceph-mon[92527]: osdmap e801: 8 total, 8 up, 8 in 2024-09-13T14:33:13.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:12 smithi151 ceph-mon[108966]: pgmap v819: 89 pgs: 89 active+clean; 177 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 8.0 MiB/s rd, 1.7 MiB/s wr, 16 op/s 2024-09-13T14:33:13.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:12 smithi151 ceph-mon[108966]: osdmap e801: 8 total, 8 up, 8 in 2024-09-13T14:33:13.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:12 smithi151 ceph-mon[104450]: pgmap v819: 89 pgs: 89 active+clean; 177 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 8.0 MiB/s rd, 1.7 MiB/s wr, 16 op/s 2024-09-13T14:33:13.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:12 smithi151 ceph-mon[104450]: osdmap e801: 8 total, 8 up, 8 in 2024-09-13T14:33:13.708 INFO:tasks.rados.rados.0.smithi151.stdout:2946: done (1 left) 2024-09-13T14:33:13.709 INFO:tasks.rados.rados.0.smithi151.stdout:2948: done (0 left) 2024-09-13T14:33:13.709 INFO:tasks.rados.rados.0.smithi151.stdout:2949: rollback oid 14 current snap is 372 2024-09-13T14:33:13.709 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 14 to 369 2024-09-13T14:33:13.709 INFO:tasks.rados.rados.0.smithi151.stdout:2950: read oid 17 snap -1 2024-09-13T14:33:13.709 INFO:tasks.rados.rados.0.smithi151.stdout:2950: expect deleted 2024-09-13T14:33:13.709 INFO:tasks.rados.rados.0.smithi151.stdout:2951: snap_remove snap 357 2024-09-13T14:33:14.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:13 smithi184 ceph-mon[92527]: osdmap e802: 8 total, 8 up, 8 in 2024-09-13T14:33:14.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:13 smithi151 ceph-mon[104450]: osdmap e802: 8 total, 8 up, 8 in 2024-09-13T14:33:14.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:13 smithi151 ceph-mon[108966]: osdmap e802: 8 total, 8 up, 8 in 2024-09-13T14:33:14.594 INFO:tasks.rados.rados.0.smithi151.stdout:2951: done (2 left) 2024-09-13T14:33:14.594 INFO:tasks.rados.rados.0.smithi151.stdout:2952: delete oid 46 current snap is 372 2024-09-13T14:33:14.594 INFO:tasks.rados.rados.0.smithi151.stdout:2949: finishing rollback tid 0 to smithi15186934-14 2024-09-13T14:33:14.594 INFO:tasks.rados.rados.0.smithi151.stdout:2949: finishing rollback tid 1 to smithi15186934-14 2024-09-13T14:33:14.594 INFO:tasks.rados.rados.0.smithi151.stdout:2949: finishing rollback tid 2 to smithi15186934-14 2024-09-13T14:33:14.594 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 1197 (ObjNum 1094 snap 367 seq_num 1094) dirty exists 2024-09-13T14:33:14.604 INFO:tasks.rados.rados.0.smithi151.stdout:2949: done (2 left) 2024-09-13T14:33:14.604 INFO:tasks.rados.rados.0.smithi151.stdout:2950: done (1 left) 2024-09-13T14:33:14.604 INFO:tasks.rados.rados.0.smithi151.stdout:2952: done (0 left) 2024-09-13T14:33:14.604 INFO:tasks.rados.rados.0.smithi151.stdout:2953: delete oid 24 current snap is 372 2024-09-13T14:33:14.609 INFO:tasks.rados.rados.0.smithi151.stdout:2953: done (0 left) 2024-09-13T14:33:14.609 INFO:tasks.rados.rados.0.smithi151.stdout:2954: delete oid 15 current snap is 372 2024-09-13T14:33:14.611 INFO:tasks.rados.rados.0.smithi151.stdout:2954: done (0 left) 2024-09-13T14:33:14.612 INFO:tasks.rados.rados.0.smithi151.stdout:2955: rollback oid 32 current snap is 372 2024-09-13T14:33:14.612 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 32 to 367 2024-09-13T14:33:14.612 INFO:tasks.rados.rados.0.smithi151.stdout:2956: write oid 48 current snap is 372 2024-09-13T14:33:14.612 INFO:tasks.rados.rados.0.smithi151.stdout:2956: seq_num 1111 ranges {672577=589177,1913024=427715} 2024-09-13T14:33:14.618 INFO:tasks.rados.rados.0.smithi151.stdout:2956: writing smithi15186934-48 from 672577 to 1261754 tid 1 2024-09-13T14:33:14.625 INFO:tasks.rados.rados.0.smithi151.stdout:2956: writing smithi15186934-48 from 1913024 to 2340739 tid 2 2024-09-13T14:33:14.625 INFO:tasks.rados.rados.0.smithi151.stdout:2957: write oid 28 current snap is 372 2024-09-13T14:33:14.625 INFO:tasks.rados.rados.0.smithi151.stdout:2955: finishing rollback tid 0 to smithi15186934-32 2024-09-13T14:33:14.625 INFO:tasks.rados.rados.0.smithi151.stdout:2955: finishing rollback tid 1 to smithi15186934-32 2024-09-13T14:33:14.625 INFO:tasks.rados.rados.0.smithi151.stdout:2955: finishing rollback tid 2 to smithi15186934-32 2024-09-13T14:33:14.625 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 32 v 1203 (ObjNum 925 snap 318 seq_num 925) dirty exists 2024-09-13T14:33:14.625 INFO:tasks.rados.rados.0.smithi151.stdout:2956: finishing write tid 1 to smithi15186934-48 2024-09-13T14:33:14.625 INFO:tasks.rados.rados.0.smithi151.stdout:2957: seq_num 1112 ranges {689384=664426,1971506=687726} 2024-09-13T14:33:14.632 INFO:tasks.rados.rados.0.smithi151.stdout:2957: writing smithi15186934-28 from 689384 to 1353810 tid 1 2024-09-13T14:33:14.640 INFO:tasks.rados.rados.0.smithi151.stdout:2957: writing smithi15186934-28 from 1971506 to 2659232 tid 2 2024-09-13T14:33:14.640 INFO:tasks.rados.rados.0.smithi151.stdout:2955: done (2 left) 2024-09-13T14:33:14.640 INFO:tasks.rados.rados.0.smithi151.stdout:2958: read oid 47 snap -1 2024-09-13T14:33:14.640 INFO:tasks.rados.rados.0.smithi151.stdout:2958: expect deleted 2024-09-13T14:33:14.640 INFO:tasks.rados.rados.0.smithi151.stdout:2959: snap_create 2024-09-13T14:33:14.640 INFO:tasks.rados.rados.0.smithi151.stdout:2956: finishing write tid 2 to smithi15186934-48 2024-09-13T14:33:14.640 INFO:tasks.rados.rados.0.smithi151.stdout:2956: finishing write tid 3 to smithi15186934-48 2024-09-13T14:33:14.640 INFO:tasks.rados.rados.0.smithi151.stdout:2956: finishing write tid 4 to smithi15186934-48 2024-09-13T14:33:14.640 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 48 v 1431 (ObjNum 1111 snap 372 seq_num 1111) dirty exists 2024-09-13T14:33:14.640 INFO:tasks.rados.rados.0.smithi151.stdout:2956: left oid 48 (ObjNum 1111 snap 372 seq_num 1111) 2024-09-13T14:33:14.657 INFO:tasks.rados.rados.0.smithi151.stdout:2957: finishing write tid 1 to smithi15186934-28 2024-09-13T14:33:14.659 INFO:tasks.rados.rados.0.smithi151.stdout:2957: finishing write tid 2 to smithi15186934-28 2024-09-13T14:33:14.659 INFO:tasks.rados.rados.0.smithi151.stdout:2957: finishing write tid 3 to smithi15186934-28 2024-09-13T14:33:14.659 INFO:tasks.rados.rados.0.smithi151.stdout:2957: finishing write tid 4 to smithi15186934-28 2024-09-13T14:33:14.659 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 28 v 1074 (ObjNum 1112 snap 372 seq_num 1112) dirty exists 2024-09-13T14:33:14.659 INFO:tasks.rados.rados.0.smithi151.stdout:2957: left oid 28 (ObjNum 1112 snap 372 seq_num 1112) 2024-09-13T14:33:14.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:14 smithi151 ceph-mon[108966]: pgmap v822: 89 pgs: 89 active+clean; 190 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.9 MiB/s wr, 26 op/s 2024-09-13T14:33:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:14 smithi151 ceph-mon[108966]: osdmap e803: 8 total, 8 up, 8 in 2024-09-13T14:33:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:14 smithi151 ceph-mon[108966]: osdmap e804: 8 total, 8 up, 8 in 2024-09-13T14:33:14.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:14 smithi151 ceph-mon[104450]: pgmap v822: 89 pgs: 89 active+clean; 190 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.9 MiB/s wr, 26 op/s 2024-09-13T14:33:14.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:14 smithi151 ceph-mon[104450]: osdmap e803: 8 total, 8 up, 8 in 2024-09-13T14:33:14.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:14 smithi151 ceph-mon[104450]: osdmap e804: 8 total, 8 up, 8 in 2024-09-13T14:33:14.860 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:33:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:33:14] "GET /metrics HTTP/1.1" 200 36511 "" "Prometheus/2.51.0" 2024-09-13T14:33:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:14 smithi184 ceph-mon[92527]: pgmap v822: 89 pgs: 89 active+clean; 190 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.9 MiB/s wr, 26 op/s 2024-09-13T14:33:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:14 smithi184 ceph-mon[92527]: osdmap e803: 8 total, 8 up, 8 in 2024-09-13T14:33:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:14 smithi184 ceph-mon[92527]: osdmap e804: 8 total, 8 up, 8 in 2024-09-13T14:33:15.608 INFO:tasks.rados.rados.0.smithi151.stdout:2956: done (3 left) 2024-09-13T14:33:15.608 INFO:tasks.rados.rados.0.smithi151.stdout:2957: done (2 left) 2024-09-13T14:33:15.608 INFO:tasks.rados.rados.0.smithi151.stdout:2958: done (1 left) 2024-09-13T14:33:15.608 INFO:tasks.rados.rados.0.smithi151.stdout:2959: done (0 left) 2024-09-13T14:33:15.608 INFO:tasks.rados.rados.0.smithi151.stdout:2960: write oid 45 current snap is 373 2024-09-13T14:33:15.608 INFO:tasks.rados.rados.0.smithi151.stdout:2960: seq_num 1113 ranges {706191=739675,1913635=431384} 2024-09-13T14:33:15.615 INFO:tasks.rados.rados.0.smithi151.stdout:2960: writing smithi15186934-45 from 706191 to 1445866 tid 1 2024-09-13T14:33:15.621 INFO:tasks.rados.rados.0.smithi151.stdout:2960: writing smithi15186934-45 from 1913635 to 2345019 tid 2 2024-09-13T14:33:15.621 INFO:tasks.rados.rados.0.smithi151.stdout:2961: write oid 4 current snap is 373 2024-09-13T14:33:15.622 INFO:tasks.rados.rados.0.smithi151.stdout:2961: seq_num 1114 ranges {722998=414924,1572117=575042} 2024-09-13T14:33:15.627 INFO:tasks.rados.rados.0.smithi151.stdout:2961: writing smithi15186934-4 from 722998 to 1137922 tid 1 2024-09-13T14:33:15.634 INFO:tasks.rados.rados.0.smithi151.stdout:2961: writing smithi15186934-4 from 1572117 to 2147159 tid 2 2024-09-13T14:33:15.634 INFO:tasks.rados.rados.0.smithi151.stdout:2962: read oid 26 snap -1 2024-09-13T14:33:15.634 INFO:tasks.rados.rados.0.smithi151.stdout:2962: expect (ObjNum 1079 snap 366 seq_num 1079) 2024-09-13T14:33:15.634 INFO:tasks.rados.rados.0.smithi151.stdout:2963: delete oid 13 current snap is 373 2024-09-13T14:33:15.634 INFO:tasks.rados.rados.0.smithi151.stdout:2960: finishing write tid 1 to smithi15186934-45 2024-09-13T14:33:15.634 INFO:tasks.rados.rados.0.smithi151.stdout:2960: finishing write tid 2 to smithi15186934-45 2024-09-13T14:33:15.634 INFO:tasks.rados.rados.0.smithi151.stdout:2960: finishing write tid 3 to smithi15186934-45 2024-09-13T14:33:15.634 INFO:tasks.rados.rados.0.smithi151.stdout:2960: finishing write tid 4 to smithi15186934-45 2024-09-13T14:33:15.634 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 1435 (ObjNum 1113 snap 373 seq_num 1113) dirty exists 2024-09-13T14:33:15.635 INFO:tasks.rados.rados.0.smithi151.stdout:2960: left oid 45 (ObjNum 1113 snap 373 seq_num 1113) 2024-09-13T14:33:15.635 INFO:tasks.rados.rados.0.smithi151.stdout:2961: finishing write tid 1 to smithi15186934-4 2024-09-13T14:33:15.636 INFO:tasks.rados.rados.0.smithi151.stdout:2960: done (3 left) 2024-09-13T14:33:15.636 INFO:tasks.rados.rados.0.smithi151.stdout:2963: done (2 left) 2024-09-13T14:33:15.636 INFO:tasks.rados.rados.0.smithi151.stdout:2964: rollback oid 6 current snap is 373 2024-09-13T14:33:15.636 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 6 to 356 2024-09-13T14:33:15.636 INFO:tasks.rados.rados.0.smithi151.stdout:2965: delete oid 8 current snap is 373 2024-09-13T14:33:15.637 INFO:tasks.rados.rados.0.smithi151.stdout:2961: finishing write tid 2 to smithi15186934-4 2024-09-13T14:33:15.637 INFO:tasks.rados.rados.0.smithi151.stdout:2961: finishing write tid 3 to smithi15186934-4 2024-09-13T14:33:15.637 INFO:tasks.rados.rados.0.smithi151.stdout:2961: finishing write tid 4 to smithi15186934-4 2024-09-13T14:33:15.638 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 1439 (ObjNum 1114 snap 373 seq_num 1114) dirty exists 2024-09-13T14:33:15.638 INFO:tasks.rados.rados.0.smithi151.stdout:2961: left oid 4 (ObjNum 1114 snap 373 seq_num 1114) 2024-09-13T14:33:15.638 INFO:tasks.rados.rados.0.smithi151.stdout:2961: done (3 left) 2024-09-13T14:33:15.638 INFO:tasks.rados.rados.0.smithi151.stdout:2965: done (2 left) 2024-09-13T14:33:15.639 INFO:tasks.rados.rados.0.smithi151.stdout:2966: delete oid 42 current snap is 373 2024-09-13T14:33:15.639 INFO:tasks.rados.rados.0.smithi151.stdout:2964: finishing rollback tid 1 to smithi15186934-6 2024-09-13T14:33:15.639 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 6 v 0 (ObjNum 81 snap 0 seq_num 0) dirty dne 2024-09-13T14:33:15.640 INFO:tasks.rados.rados.0.smithi151.stdout:2962: expect (ObjNum 1079 snap 366 seq_num 1079) 2024-09-13T14:33:15.804 INFO:tasks.rados.rados.0.smithi151.stdout:2962: done (2 left) 2024-09-13T14:33:15.804 INFO:tasks.rados.rados.0.smithi151.stdout:2964: done (1 left) 2024-09-13T14:33:15.805 INFO:tasks.rados.rados.0.smithi151.stdout:2966: done (0 left) 2024-09-13T14:33:15.805 INFO:tasks.rados.rados.0.smithi151.stdout:2967: read oid 48 snap -1 2024-09-13T14:33:15.805 INFO:tasks.rados.rados.0.smithi151.stdout:2967: expect (ObjNum 1111 snap 372 seq_num 1111) 2024-09-13T14:33:15.805 INFO:tasks.rados.rados.0.smithi151.stdout:2968: write oid 22 current snap is 373 2024-09-13T14:33:15.805 INFO:tasks.rados.rados.0.smithi151.stdout:2968: seq_num 1118 ranges {790226=432273,1806045=582380} 2024-09-13T14:33:15.810 INFO:tasks.rados.rados.0.smithi151.stdout:2968: writing smithi15186934-22 from 790226 to 1222499 tid 1 2024-09-13T14:33:15.817 INFO:tasks.rados.rados.0.smithi151.stdout:2968: writing smithi15186934-22 from 1806045 to 2388425 tid 2 2024-09-13T14:33:15.817 INFO:tasks.rados.rados.0.smithi151.stdout:2969: snap_create 2024-09-13T14:33:15.817 INFO:tasks.rados.rados.0.smithi151.stdout:2967: expect (ObjNum 1111 snap 372 seq_num 1111) 2024-09-13T14:33:15.943 INFO:tasks.rados.rados.0.smithi151.stdout:2968: finishing write tid 1 to smithi15186934-22 2024-09-13T14:33:15.943 INFO:tasks.rados.rados.0.smithi151.stdout:2968: finishing write tid 2 to smithi15186934-22 2024-09-13T14:33:15.943 INFO:tasks.rados.rados.0.smithi151.stdout:2968: finishing write tid 3 to smithi15186934-22 2024-09-13T14:33:15.943 INFO:tasks.rados.rados.0.smithi151.stdout:2968: finishing write tid 4 to smithi15186934-22 2024-09-13T14:33:15.943 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 22 v 578 (ObjNum 1118 snap 373 seq_num 1118) dirty exists 2024-09-13T14:33:15.943 INFO:tasks.rados.rados.0.smithi151.stdout:2968: left oid 22 (ObjNum 1118 snap 373 seq_num 1118) 2024-09-13T14:33:16.711 INFO:tasks.rados.rados.0.smithi151.stdout:2967: done (2 left) 2024-09-13T14:33:16.711 INFO:tasks.rados.rados.0.smithi151.stdout:2968: done (1 left) 2024-09-13T14:33:16.711 INFO:tasks.rados.rados.0.smithi151.stdout:2969: done (0 left) 2024-09-13T14:33:16.711 INFO:tasks.rados.rados.0.smithi151.stdout:2970: delete oid 36 current snap is 374 2024-09-13T14:33:16.714 INFO:tasks.rados.rados.0.smithi151.stdout:2970: done (0 left) 2024-09-13T14:33:16.714 INFO:tasks.rados.rados.0.smithi151.stdout:2971: delete oid 42 current snap is 374 2024-09-13T14:33:16.718 INFO:tasks.rados.rados.0.smithi151.stdout:2971: done (0 left) 2024-09-13T14:33:16.719 INFO:tasks.rados.rados.0.smithi151.stdout:2972: write oid 47 current snap is 374 2024-09-13T14:33:16.719 INFO:tasks.rados.rados.0.smithi151.stdout:2972: seq_num 1121 ranges {440647=658020,1581491=446060,2800822=39825} 2024-09-13T14:33:16.724 INFO:tasks.rados.rados.0.smithi151.stdout:2972: writing smithi15186934-47 from 440647 to 1098667 tid 1 2024-09-13T14:33:16.730 INFO:tasks.rados.rados.0.smithi151.stdout:2972: writing smithi15186934-47 from 1581491 to 2027551 tid 2 2024-09-13T14:33:16.734 INFO:tasks.rados.rados.0.smithi151.stdout:2972: writing smithi15186934-47 from 2800822 to 2840647 tid 3 2024-09-13T14:33:16.735 INFO:tasks.rados.rados.0.smithi151.stdout:2973: write oid 20 current snap is 374 2024-09-13T14:33:16.735 INFO:tasks.rados.rados.0.smithi151.stdout:2973: seq_num 1122 ranges {457454=733269,1923620=589718} 2024-09-13T14:33:16.740 INFO:tasks.rados.rados.0.smithi151.stdout:2973: writing smithi15186934-20 from 457454 to 1190723 tid 1 2024-09-13T14:33:16.748 INFO:tasks.rados.rados.0.smithi151.stdout:2973: writing smithi15186934-20 from 1923620 to 2513338 tid 2 2024-09-13T14:33:16.748 INFO:tasks.rados.rados.0.smithi151.stdout:2974: read oid 50 snap -1 2024-09-13T14:33:16.748 INFO:tasks.rados.rados.0.smithi151.stdout:2974: expect (ObjNum 1103 snap 370 seq_num 1103) 2024-09-13T14:33:16.748 INFO:tasks.rados.rados.0.smithi151.stdout:2975: write oid 46 current snap is 374 2024-09-13T14:33:16.749 INFO:tasks.rados.rados.0.smithi151.stdout:2972: finishing write tid 1 to smithi15186934-47 2024-09-13T14:33:16.749 INFO:tasks.rados.rados.0.smithi151.stdout:2972: finishing write tid 2 to smithi15186934-47 2024-09-13T14:33:16.749 INFO:tasks.rados.rados.0.smithi151.stdout:2972: finishing write tid 3 to smithi15186934-47 2024-09-13T14:33:16.749 INFO:tasks.rados.rados.0.smithi151.stdout:2972: finishing write tid 4 to smithi15186934-47 2024-09-13T14:33:16.749 INFO:tasks.rados.rados.0.smithi151.stdout:2975: seq_num 1123 ranges {474261=408518,1582102=449729,2739315=134946} 2024-09-13T14:33:16.752 INFO:tasks.rados.rados.0.smithi151.stdout:2975: writing smithi15186934-46 from 474261 to 882779 tid 1 2024-09-13T14:33:16.759 INFO:tasks.rados.rados.0.smithi151.stdout:2975: writing smithi15186934-46 from 1582102 to 2031831 tid 2 2024-09-13T14:33:16.764 INFO:tasks.rados.rados.0.smithi151.stdout:2975: writing smithi15186934-46 from 2739315 to 2874261 tid 3 2024-09-13T14:33:16.764 INFO:tasks.rados.rados.0.smithi151.stdout:2976: snap_create 2024-09-13T14:33:16.764 INFO:tasks.rados.rados.0.smithi151.stdout:2972: finishing write tid 5 to smithi15186934-47 2024-09-13T14:33:16.764 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 1302 (ObjNum 1121 snap 374 seq_num 1121) dirty exists 2024-09-13T14:33:16.764 INFO:tasks.rados.rados.0.smithi151.stdout:2972: left oid 47 (ObjNum 1121 snap 374 seq_num 1121) 2024-09-13T14:33:16.764 INFO:tasks.rados.rados.0.smithi151.stdout:2973: finishing write tid 1 to smithi15186934-20 2024-09-13T14:33:16.764 INFO:tasks.rados.rados.0.smithi151.stdout:2973: finishing write tid 2 to smithi15186934-20 2024-09-13T14:33:16.765 INFO:tasks.rados.rados.0.smithi151.stdout:2973: finishing write tid 3 to smithi15186934-20 2024-09-13T14:33:16.765 INFO:tasks.rados.rados.0.smithi151.stdout:2973: finishing write tid 4 to smithi15186934-20 2024-09-13T14:33:16.765 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 761 (ObjNum 1122 snap 374 seq_num 1122) dirty exists 2024-09-13T14:33:16.765 INFO:tasks.rados.rados.0.smithi151.stdout:2973: left oid 20 (ObjNum 1122 snap 374 seq_num 1122) 2024-09-13T14:33:16.765 INFO:tasks.rados.rados.0.smithi151.stdout:2975: finishing write tid 1 to smithi15186934-46 2024-09-13T14:33:16.765 INFO:tasks.rados.rados.0.smithi151.stdout:2974: expect (ObjNum 1103 snap 370 seq_num 1103) 2024-09-13T14:33:16.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:16 smithi151 ceph-mon[108966]: pgmap v825: 89 pgs: 89 active+clean; 193 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 7.4 MiB/s rd, 2.4 MiB/s wr, 19 op/s 2024-09-13T14:33:16.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:16 smithi151 ceph-mon[108966]: osdmap e805: 8 total, 8 up, 8 in 2024-09-13T14:33:16.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:16.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:16 smithi151 ceph-mon[104450]: pgmap v825: 89 pgs: 89 active+clean; 193 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 7.4 MiB/s rd, 2.4 MiB/s wr, 19 op/s 2024-09-13T14:33:16.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:16 smithi151 ceph-mon[104450]: osdmap e805: 8 total, 8 up, 8 in 2024-09-13T14:33:16.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:16.954 INFO:tasks.rados.rados.0.smithi151.stdout:2975: finishing write tid 2 to smithi15186934-46 2024-09-13T14:33:16.954 INFO:tasks.rados.rados.0.smithi151.stdout:2975: finishing write tid 3 to smithi15186934-46 2024-09-13T14:33:16.954 INFO:tasks.rados.rados.0.smithi151.stdout:2975: finishing write tid 4 to smithi15186934-46 2024-09-13T14:33:16.954 INFO:tasks.rados.rados.0.smithi151.stdout:2975: finishing write tid 5 to smithi15186934-46 2024-09-13T14:33:16.954 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 46 v 1075 (ObjNum 1123 snap 374 seq_num 1123) dirty exists 2024-09-13T14:33:16.954 INFO:tasks.rados.rados.0.smithi151.stdout:2975: left oid 46 (ObjNum 1123 snap 374 seq_num 1123) 2024-09-13T14:33:17.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:16 smithi184 ceph-mon[92527]: pgmap v825: 89 pgs: 89 active+clean; 193 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 7.4 MiB/s rd, 2.4 MiB/s wr, 19 op/s 2024-09-13T14:33:17.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:16 smithi184 ceph-mon[92527]: osdmap e805: 8 total, 8 up, 8 in 2024-09-13T14:33:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:17.716 INFO:tasks.rados.rados.0.smithi151.stdout:2972: done (4 left) 2024-09-13T14:33:17.717 INFO:tasks.rados.rados.0.smithi151.stdout:2973: done (3 left) 2024-09-13T14:33:17.717 INFO:tasks.rados.rados.0.smithi151.stdout:2974: done (2 left) 2024-09-13T14:33:17.717 INFO:tasks.rados.rados.0.smithi151.stdout:2975: done (1 left) 2024-09-13T14:33:17.717 INFO:tasks.rados.rados.0.smithi151.stdout:2976: done (0 left) 2024-09-13T14:33:17.717 INFO:tasks.rados.rados.0.smithi151.stdout:2977: snap_remove snap 374 2024-09-13T14:33:18.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:17 smithi184 ceph-mon[92527]: osdmap e806: 8 total, 8 up, 8 in 2024-09-13T14:33:18.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:17 smithi151 ceph-mon[104450]: osdmap e806: 8 total, 8 up, 8 in 2024-09-13T14:33:18.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:17 smithi151 ceph-mon[108966]: osdmap e806: 8 total, 8 up, 8 in 2024-09-13T14:33:18.719 INFO:tasks.rados.rados.0.smithi151.stdout:2977: done (0 left) 2024-09-13T14:33:18.719 INFO:tasks.rados.rados.0.smithi151.stdout:2978: read oid 11 snap -1 2024-09-13T14:33:18.719 INFO:tasks.rados.rados.0.smithi151.stdout:2978: expect deleted 2024-09-13T14:33:18.719 INFO:tasks.rados.rados.0.smithi151.stdout:2979: rollback oid 23 current snap is 375 2024-09-13T14:33:18.720 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 23 to 356 2024-09-13T14:33:18.720 INFO:tasks.rados.rados.0.smithi151.stdout:2980: snap_create 2024-09-13T14:33:18.723 INFO:tasks.rados.rados.0.smithi151.stdout:2979: finishing rollback tid 0 to smithi15186934-23 2024-09-13T14:33:18.723 INFO:tasks.rados.rados.0.smithi151.stdout:2979: finishing rollback tid 1 to smithi15186934-23 2024-09-13T14:33:18.724 INFO:tasks.rados.rados.0.smithi151.stdout:2979: finishing rollback tid 2 to smithi15186934-23 2024-09-13T14:33:18.724 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 23 v 1306 (ObjNum 1042 snap 355 seq_num 1042) dirty exists 2024-09-13T14:33:19.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:18 smithi184 ceph-mon[92527]: pgmap v828: 89 pgs: 89 active+clean; 197 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 14 op/s 2024-09-13T14:33:19.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:18 smithi184 ceph-mon[92527]: osdmap e807: 8 total, 8 up, 8 in 2024-09-13T14:33:19.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:18 smithi151 ceph-mon[104450]: pgmap v828: 89 pgs: 89 active+clean; 197 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 14 op/s 2024-09-13T14:33:19.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:18 smithi151 ceph-mon[104450]: osdmap e807: 8 total, 8 up, 8 in 2024-09-13T14:33:19.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:18 smithi151 ceph-mon[108966]: pgmap v828: 89 pgs: 89 active+clean; 197 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 14 op/s 2024-09-13T14:33:19.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:18 smithi151 ceph-mon[108966]: osdmap e807: 8 total, 8 up, 8 in 2024-09-13T14:33:19.595 INFO:tasks.rados.rados.0.smithi151.stdout:2978: done (2 left) 2024-09-13T14:33:19.595 INFO:tasks.rados.rados.0.smithi151.stdout:2979: done (1 left) 2024-09-13T14:33:19.595 INFO:tasks.rados.rados.0.smithi151.stdout:2980: done (0 left) 2024-09-13T14:33:19.595 INFO:tasks.rados.rados.0.smithi151.stdout:2981: rollback oid 9 current snap is 376 2024-09-13T14:33:19.595 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 9 to 364 2024-09-13T14:33:19.595 INFO:tasks.rados.rados.0.smithi151.stdout:2982: write oid 29 current snap is 376 2024-09-13T14:33:19.595 INFO:tasks.rados.rados.0.smithi151.stdout:2982: seq_num 1124 ranges {491068=483767,1640584=593387,2650385=240683} 2024-09-13T14:33:19.601 INFO:tasks.rados.rados.0.smithi151.stdout:2982: writing smithi15186934-29 from 491068 to 974835 tid 1 2024-09-13T14:33:19.607 INFO:tasks.rados.rados.0.smithi151.stdout:2982: writing smithi15186934-29 from 1640584 to 2233971 tid 2 2024-09-13T14:33:19.611 INFO:tasks.rados.rados.0.smithi151.stdout:2982: writing smithi15186934-29 from 2650385 to 2891068 tid 3 2024-09-13T14:33:19.611 INFO:tasks.rados.rados.0.smithi151.stdout:2983: write oid 44 current snap is 376 2024-09-13T14:33:19.611 INFO:tasks.rados.rados.0.smithi151.stdout:2981: finishing rollback tid 0 to smithi15186934-9 2024-09-13T14:33:19.611 INFO:tasks.rados.rados.0.smithi151.stdout:2983: seq_num 1125 ranges {507875=559016,1699066=737045} 2024-09-13T14:33:19.617 INFO:tasks.rados.rados.0.smithi151.stdout:2983: writing smithi15186934-44 from 507875 to 1066891 tid 1 2024-09-13T14:33:19.625 INFO:tasks.rados.rados.0.smithi151.stdout:2983: writing smithi15186934-44 from 1699066 to 2436111 tid 2 2024-09-13T14:33:19.625 INFO:tasks.rados.rados.0.smithi151.stdout:2984: snap_create 2024-09-13T14:33:19.625 INFO:tasks.rados.rados.0.smithi151.stdout:2981: finishing rollback tid 1 to smithi15186934-9 2024-09-13T14:33:19.625 INFO:tasks.rados.rados.0.smithi151.stdout:2981: finishing rollback tid 2 to smithi15186934-9 2024-09-13T14:33:19.625 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 1207 (ObjNum 1050 snap 356 seq_num 1050) dirty exists 2024-09-13T14:33:19.625 INFO:tasks.rados.rados.0.smithi151.stdout:2982: finishing write tid 1 to smithi15186934-29 2024-09-13T14:33:19.625 INFO:tasks.rados.rados.0.smithi151.stdout:2982: finishing write tid 2 to smithi15186934-29 2024-09-13T14:33:19.625 INFO:tasks.rados.rados.0.smithi151.stdout:2982: finishing write tid 3 to smithi15186934-29 2024-09-13T14:33:19.625 INFO:tasks.rados.rados.0.smithi151.stdout:2982: finishing write tid 4 to smithi15186934-29 2024-09-13T14:33:19.626 INFO:tasks.rados.rados.0.smithi151.stdout:2982: finishing write tid 5 to smithi15186934-29 2024-09-13T14:33:19.626 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 29 v 693 (ObjNum 1124 snap 376 seq_num 1124) dirty exists 2024-09-13T14:33:19.626 INFO:tasks.rados.rados.0.smithi151.stdout:2982: left oid 29 (ObjNum 1124 snap 376 seq_num 1124) 2024-09-13T14:33:19.626 INFO:tasks.rados.rados.0.smithi151.stdout:2983: finishing write tid 1 to smithi15186934-44 2024-09-13T14:33:19.628 INFO:tasks.rados.rados.0.smithi151.stdout:2983: finishing write tid 2 to smithi15186934-44 2024-09-13T14:33:19.629 INFO:tasks.rados.rados.0.smithi151.stdout:2983: finishing write tid 3 to smithi15186934-44 2024-09-13T14:33:19.629 INFO:tasks.rados.rados.0.smithi151.stdout:2983: finishing write tid 4 to smithi15186934-44 2024-09-13T14:33:19.629 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 884 (ObjNum 1125 snap 376 seq_num 1125) dirty exists 2024-09-13T14:33:19.629 INFO:tasks.rados.rados.0.smithi151.stdout:2983: left oid 44 (ObjNum 1125 snap 376 seq_num 1125) 2024-09-13T14:33:20.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:19 smithi184 ceph-mon[92527]: osdmap e808: 8 total, 8 up, 8 in 2024-09-13T14:33:20.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:19 smithi184 ceph-mon[92527]: osdmap e809: 8 total, 8 up, 8 in 2024-09-13T14:33:20.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:19 smithi151 ceph-mon[104450]: osdmap e808: 8 total, 8 up, 8 in 2024-09-13T14:33:20.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:19 smithi151 ceph-mon[104450]: osdmap e809: 8 total, 8 up, 8 in 2024-09-13T14:33:20.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:19 smithi151 ceph-mon[108966]: osdmap e808: 8 total, 8 up, 8 in 2024-09-13T14:33:20.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:19 smithi151 ceph-mon[108966]: osdmap e809: 8 total, 8 up, 8 in 2024-09-13T14:33:20.600 INFO:tasks.rados.rados.0.smithi151.stdout:2981: done (3 left) 2024-09-13T14:33:20.600 INFO:tasks.rados.rados.0.smithi151.stdout:2982: done (2 left) 2024-09-13T14:33:20.601 INFO:tasks.rados.rados.0.smithi151.stdout:2983: done (1 left) 2024-09-13T14:33:20.601 INFO:tasks.rados.rados.0.smithi151.stdout:2984: done (0 left) 2024-09-13T14:33:20.601 INFO:tasks.rados.rados.0.smithi151.stdout:2985: snap_remove snap 373 2024-09-13T14:33:20.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:33:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:33:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:33:21.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:20 smithi184 ceph-mon[92527]: pgmap v831: 89 pgs: 89 active+clean; 201 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.1 MiB/s wr, 18 op/s 2024-09-13T14:33:21.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:20 smithi184 ceph-mon[92527]: osdmap e810: 8 total, 8 up, 8 in 2024-09-13T14:33:21.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:20 smithi151 ceph-mon[104450]: pgmap v831: 89 pgs: 89 active+clean; 201 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.1 MiB/s wr, 18 op/s 2024-09-13T14:33:21.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:20 smithi151 ceph-mon[104450]: osdmap e810: 8 total, 8 up, 8 in 2024-09-13T14:33:21.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:20 smithi151 ceph-mon[108966]: pgmap v831: 89 pgs: 89 active+clean; 201 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.1 MiB/s wr, 18 op/s 2024-09-13T14:33:21.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:20 smithi151 ceph-mon[108966]: osdmap e810: 8 total, 8 up, 8 in 2024-09-13T14:33:21.600 INFO:tasks.rados.rados.0.smithi151.stdout:2985: done (0 left) 2024-09-13T14:33:21.600 INFO:tasks.rados.rados.0.smithi151.stdout:2986: snap_create 2024-09-13T14:33:22.600 INFO:tasks.rados.rados.0.smithi151.stdout:2986: done (0 left) 2024-09-13T14:33:22.600 INFO:tasks.rados.rados.0.smithi151.stdout:2987: rollback oid 7 current snap is 378 2024-09-13T14:33:22.600 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 7 to 375 2024-09-13T14:33:22.601 INFO:tasks.rados.rados.0.smithi151.stdout:2988: write oid 14 current snap is 378 2024-09-13T14:33:22.601 INFO:tasks.rados.rados.0.smithi151.stdout:2988: seq_num 1126 ranges {524682=750618,1757548=597056} 2024-09-13T14:33:22.607 INFO:tasks.rados.rados.0.smithi151.stdout:2988: writing smithi15186934-14 from 524682 to 1275300 tid 1 2024-09-13T14:33:22.614 INFO:tasks.rados.rados.0.smithi151.stdout:2988: writing smithi15186934-14 from 1757548 to 2354604 tid 2 2024-09-13T14:33:22.614 INFO:tasks.rados.rados.0.smithi151.stdout:2989: read oid 39 snap -1 2024-09-13T14:33:22.614 INFO:tasks.rados.rados.0.smithi151.stdout:2989: expect (ObjNum 1092 snap 367 seq_num 1092) 2024-09-13T14:33:22.614 INFO:tasks.rados.rados.0.smithi151.stdout:2990: delete oid 41 current snap is 378 2024-09-13T14:33:22.614 INFO:tasks.rados.rados.0.smithi151.stdout:2987: finishing rollback tid 0 to smithi15186934-7 2024-09-13T14:33:22.614 INFO:tasks.rados.rados.0.smithi151.stdout:2987: finishing rollback tid 1 to smithi15186934-7 2024-09-13T14:33:22.615 INFO:tasks.rados.rados.0.smithi151.stdout:2987: finishing rollback tid 2 to smithi15186934-7 2024-09-13T14:33:22.615 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 1763 (ObjNum 951 snap 327 seq_num 951) dirty exists 2024-09-13T14:33:22.615 INFO:tasks.rados.rados.0.smithi151.stdout:2988: finishing write tid 1 to smithi15186934-14 2024-09-13T14:33:22.617 INFO:tasks.rados.rados.0.smithi151.stdout:2988: finishing write tid 2 to smithi15186934-14 2024-09-13T14:33:22.617 INFO:tasks.rados.rados.0.smithi151.stdout:2987: done (3 left) 2024-09-13T14:33:22.617 INFO:tasks.rados.rados.0.smithi151.stdout:2990: done (2 left) 2024-09-13T14:33:22.617 INFO:tasks.rados.rados.0.smithi151.stdout:2991: write oid 7 current snap is 378 2024-09-13T14:33:22.617 INFO:tasks.rados.rados.0.smithi151.stdout:2991: seq_num 1128 ranges {558296=501116,1474512=600725,2760077=198219} 2024-09-13T14:33:22.623 INFO:tasks.rados.rados.0.smithi151.stdout:2991: writing smithi15186934-7 from 558296 to 1059412 tid 1 2024-09-13T14:33:22.628 INFO:tasks.rados.rados.0.smithi151.stdout:2991: writing smithi15186934-7 from 1474512 to 2075237 tid 2 2024-09-13T14:33:22.634 INFO:tasks.rados.rados.0.smithi151.stdout:2991: writing smithi15186934-7 from 2760077 to 2958296 tid 3 2024-09-13T14:33:22.634 INFO:tasks.rados.rados.0.smithi151.stdout:2992: write oid 37 current snap is 378 2024-09-13T14:33:22.634 INFO:tasks.rados.rados.0.smithi151.stdout:2988: finishing write tid 3 to smithi15186934-14 2024-09-13T14:33:22.634 INFO:tasks.rados.rados.0.smithi151.stdout:2988: finishing write tid 4 to smithi15186934-14 2024-09-13T14:33:22.634 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 1213 (ObjNum 1126 snap 378 seq_num 1126) dirty exists 2024-09-13T14:33:22.634 INFO:tasks.rados.rados.0.smithi151.stdout:2988: left oid 14 (ObjNum 1126 snap 378 seq_num 1126) 2024-09-13T14:33:22.634 INFO:tasks.rados.rados.0.smithi151.stdout:2992: seq_num 1129 ranges {575103=576365,1932994=744383} 2024-09-13T14:33:22.639 INFO:tasks.rados.rados.0.smithi151.stdout:2992: writing smithi15186934-37 from 575103 to 1151468 tid 1 2024-09-13T14:33:22.648 INFO:tasks.rados.rados.0.smithi151.stdout:2992: writing smithi15186934-37 from 1932994 to 2677377 tid 2 2024-09-13T14:33:22.648 INFO:tasks.rados.rados.0.smithi151.stdout:2988: done (3 left) 2024-09-13T14:33:22.648 INFO:tasks.rados.rados.0.smithi151.stdout:2993: snap_create 2024-09-13T14:33:22.648 INFO:tasks.rados.rados.0.smithi151.stdout:2989: expect (ObjNum 1092 snap 367 seq_num 1092) 2024-09-13T14:33:22.810 INFO:tasks.rados.rados.0.smithi151.stdout:2991: finishing write tid 1 to smithi15186934-7 2024-09-13T14:33:22.810 INFO:tasks.rados.rados.0.smithi151.stdout:2991: finishing write tid 2 to smithi15186934-7 2024-09-13T14:33:22.810 INFO:tasks.rados.rados.0.smithi151.stdout:2991: finishing write tid 3 to smithi15186934-7 2024-09-13T14:33:22.811 INFO:tasks.rados.rados.0.smithi151.stdout:2991: finishing write tid 4 to smithi15186934-7 2024-09-13T14:33:22.811 INFO:tasks.rados.rados.0.smithi151.stdout:2991: finishing write tid 5 to smithi15186934-7 2024-09-13T14:33:22.811 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 1767 (ObjNum 1128 snap 378 seq_num 1128) dirty exists 2024-09-13T14:33:22.811 INFO:tasks.rados.rados.0.smithi151.stdout:2991: left oid 7 (ObjNum 1128 snap 378 seq_num 1128) 2024-09-13T14:33:22.811 INFO:tasks.rados.rados.0.smithi151.stdout:2992: finishing write tid 1 to smithi15186934-37 2024-09-13T14:33:22.811 INFO:tasks.rados.rados.0.smithi151.stdout:2992: finishing write tid 2 to smithi15186934-37 2024-09-13T14:33:22.811 INFO:tasks.rados.rados.0.smithi151.stdout:2992: finishing write tid 3 to smithi15186934-37 2024-09-13T14:33:22.811 INFO:tasks.rados.rados.0.smithi151.stdout:2992: finishing write tid 4 to smithi15186934-37 2024-09-13T14:33:22.811 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 37 v 1312 (ObjNum 1129 snap 378 seq_num 1129) dirty exists 2024-09-13T14:33:22.811 INFO:tasks.rados.rados.0.smithi151.stdout:2992: left oid 37 (ObjNum 1129 snap 378 seq_num 1129) 2024-09-13T14:33:22.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:22 smithi184 ceph-mon[92527]: pgmap v834: 89 pgs: 89 active+clean; 204 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:33:22.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:22 smithi184 ceph-mon[92527]: osdmap e811: 8 total, 8 up, 8 in 2024-09-13T14:33:22.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:22 smithi151 ceph-mon[108966]: pgmap v834: 89 pgs: 89 active+clean; 204 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:33:22.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:22 smithi151 ceph-mon[108966]: osdmap e811: 8 total, 8 up, 8 in 2024-09-13T14:33:22.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:22 smithi151 ceph-mon[104450]: pgmap v834: 89 pgs: 89 active+clean; 204 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:33:22.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:22 smithi151 ceph-mon[104450]: osdmap e811: 8 total, 8 up, 8 in 2024-09-13T14:33:23.617 INFO:tasks.rados.rados.0.smithi151.stdout:2989: done (3 left) 2024-09-13T14:33:23.617 INFO:tasks.rados.rados.0.smithi151.stdout:2991: done (2 left) 2024-09-13T14:33:23.617 INFO:tasks.rados.rados.0.smithi151.stdout:2992: done (1 left) 2024-09-13T14:33:23.617 INFO:tasks.rados.rados.0.smithi151.stdout:2993: done (0 left) 2024-09-13T14:33:23.617 INFO:tasks.rados.rados.0.smithi151.stdout:2994: rollback oid 7 current snap is 379 2024-09-13T14:33:23.617 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 7 to 369 2024-09-13T14:33:23.617 INFO:tasks.rados.rados.0.smithi151.stdout:2995: write oid 26 current snap is 379 2024-09-13T14:33:23.618 INFO:tasks.rados.rados.0.smithi151.stdout:2995: seq_num 1130 ranges {591910=651614,1875123=604394} 2024-09-13T14:33:23.623 INFO:tasks.rados.rados.0.smithi151.stdout:2995: writing smithi15186934-26 from 591910 to 1243524 tid 1 2024-09-13T14:33:23.631 INFO:tasks.rados.rados.0.smithi151.stdout:2995: writing smithi15186934-26 from 1875123 to 2479517 tid 2 2024-09-13T14:33:23.631 INFO:tasks.rados.rados.0.smithi151.stdout:2996: snap_create 2024-09-13T14:33:23.631 INFO:tasks.rados.rados.0.smithi151.stdout:2994: finishing rollback tid 0 to smithi15186934-7 2024-09-13T14:33:23.631 INFO:tasks.rados.rados.0.smithi151.stdout:2994: finishing rollback tid 1 to smithi15186934-7 2024-09-13T14:33:23.631 INFO:tasks.rados.rados.0.smithi151.stdout:2994: finishing rollback tid 2 to smithi15186934-7 2024-09-13T14:33:23.631 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 1771 (ObjNum 951 snap 327 seq_num 951) dirty exists 2024-09-13T14:33:23.631 INFO:tasks.rados.rados.0.smithi151.stdout:2995: finishing write tid 1 to smithi15186934-26 2024-09-13T14:33:23.634 INFO:tasks.rados.rados.0.smithi151.stdout:2995: finishing write tid 2 to smithi15186934-26 2024-09-13T14:33:23.634 INFO:tasks.rados.rados.0.smithi151.stdout:2995: finishing write tid 3 to smithi15186934-26 2024-09-13T14:33:23.634 INFO:tasks.rados.rados.0.smithi151.stdout:2995: finishing write tid 4 to smithi15186934-26 2024-09-13T14:33:23.634 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 1079 (ObjNum 1130 snap 379 seq_num 1130) dirty exists 2024-09-13T14:33:23.634 INFO:tasks.rados.rados.0.smithi151.stdout:2995: left oid 26 (ObjNum 1130 snap 379 seq_num 1130) 2024-09-13T14:33:24.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:23 smithi184 ceph-mon[92527]: osdmap e812: 8 total, 8 up, 8 in 2024-09-13T14:33:24.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:23 smithi151 ceph-mon[104450]: osdmap e812: 8 total, 8 up, 8 in 2024-09-13T14:33:24.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:23 smithi151 ceph-mon[108966]: osdmap e812: 8 total, 8 up, 8 in 2024-09-13T14:33:24.599 INFO:tasks.rados.rados.0.smithi151.stdout:2994: done (2 left) 2024-09-13T14:33:24.600 INFO:tasks.rados.rados.0.smithi151.stdout:2995: done (1 left) 2024-09-13T14:33:24.600 INFO:tasks.rados.rados.0.smithi151.stdout:2996: done (0 left) 2024-09-13T14:33:24.600 INFO:tasks.rados.rados.0.smithi151.stdout:2997: rollback oid 2 current snap is 380 2024-09-13T14:33:24.600 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 2 to 372 2024-09-13T14:33:24.600 INFO:tasks.rados.rados.0.smithi151.stdout:2998: snap_create 2024-09-13T14:33:24.602 INFO:tasks.rados.rados.0.smithi151.stdout:2997: finishing rollback tid 1 to smithi15186934-2 2024-09-13T14:33:24.602 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 2 v 0 (ObjNum 305 snap 0 seq_num 1664845217) dirty dne 2024-09-13T14:33:24.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:24 smithi151 ceph-mon[104450]: pgmap v837: 89 pgs: 89 active+clean; 207 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 639 KiB/s wr, 3 op/s 2024-09-13T14:33:24.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:24 smithi151 ceph-mon[104450]: osdmap e813: 8 total, 8 up, 8 in 2024-09-13T14:33:24.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:24 smithi151 ceph-mon[104450]: osdmap e814: 8 total, 8 up, 8 in 2024-09-13T14:33:24.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:24 smithi151 ceph-mon[108966]: pgmap v837: 89 pgs: 89 active+clean; 207 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 639 KiB/s wr, 3 op/s 2024-09-13T14:33:24.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:24 smithi151 ceph-mon[108966]: osdmap e813: 8 total, 8 up, 8 in 2024-09-13T14:33:24.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:24 smithi151 ceph-mon[108966]: osdmap e814: 8 total, 8 up, 8 in 2024-09-13T14:33:24.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:33:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:33:24] "GET /metrics HTTP/1.1" 200 36512 "" "Prometheus/2.51.0" 2024-09-13T14:33:25.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:24 smithi184 ceph-mon[92527]: pgmap v837: 89 pgs: 89 active+clean; 207 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 639 KiB/s wr, 3 op/s 2024-09-13T14:33:25.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:24 smithi184 ceph-mon[92527]: osdmap e813: 8 total, 8 up, 8 in 2024-09-13T14:33:25.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:24 smithi184 ceph-mon[92527]: osdmap e814: 8 total, 8 up, 8 in 2024-09-13T14:33:25.599 INFO:tasks.rados.rados.0.smithi151.stdout:2997: done (1 left) 2024-09-13T14:33:25.599 INFO:tasks.rados.rados.0.smithi151.stdout:2998: done (0 left) 2024-09-13T14:33:25.600 INFO:tasks.rados.rados.0.smithi151.stdout:2999: write oid 18 current snap is 381 2024-09-13T14:33:25.600 INFO:tasks.rados.rados.0.smithi151.stdout:2999: seq_num 1131 ranges {608717=726863,1933605=748052} 2024-09-13T14:33:25.606 INFO:tasks.rados.rados.0.smithi151.stdout:2999: writing smithi15186934-18 from 608717 to 1335580 tid 1 2024-09-13T14:33:25.614 INFO:tasks.rados.rados.0.smithi151.stdout:2999: writing smithi15186934-18 from 1933605 to 2681657 tid 2 2024-09-13T14:33:25.614 INFO:tasks.rados.rados.0.smithi151.stdout:3000: delete oid 49 current snap is 381 2024-09-13T14:33:25.615 INFO:tasks.rados.rados.0.smithi151.stdout:2999: finishing write tid 1 to smithi15186934-18 2024-09-13T14:33:25.618 INFO:tasks.rados.rados.0.smithi151.stdout:3000: done (1 left) 2024-09-13T14:33:25.618 INFO:tasks.rados.rados.0.smithi151.stdout:3001: snap_remove snap 371 2024-09-13T14:33:26.620 INFO:tasks.rados.rados.0.smithi151.stdout:3001: done (1 left) 2024-09-13T14:33:26.620 INFO:tasks.rados.rados.0.smithi151.stdout:3002: write oid 27 current snap is 381 2024-09-13T14:33:26.620 INFO:tasks.rados.rados.0.smithi151.stdout:2999: finishing write tid 2 to smithi15186934-18 2024-09-13T14:33:26.620 INFO:tasks.rados.rados.0.smithi151.stdout:2999: finishing write tid 3 to smithi15186934-18 2024-09-13T14:33:26.620 INFO:tasks.rados.rados.0.smithi151.stdout:2999: finishing write tid 4 to smithi15186934-18 2024-09-13T14:33:26.620 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 18 v 698 (ObjNum 1131 snap 381 seq_num 1131) dirty exists 2024-09-13T14:33:26.620 INFO:tasks.rados.rados.0.smithi151.stdout:2999: left oid 18 (ObjNum 1131 snap 381 seq_num 1131) 2024-09-13T14:33:26.621 INFO:tasks.rados.rados.0.smithi151.stdout:3002: seq_num 1133 ranges {642331=593714,1766922=751721} 2024-09-13T14:33:26.630 INFO:tasks.rados.rados.0.smithi151.stdout:3002: writing smithi15186934-27 from 642331 to 1236045 tid 1 2024-09-13T14:33:26.636 INFO:tasks.rados.rados.0.smithi151.stdout:3002: writing smithi15186934-27 from 1766922 to 2518643 tid 2 2024-09-13T14:33:26.636 INFO:tasks.rados.rados.0.smithi151.stdout:2999: done (1 left) 2024-09-13T14:33:26.636 INFO:tasks.rados.rados.0.smithi151.stdout:3003: rollback oid 33 current snap is 381 2024-09-13T14:33:26.637 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 33 to 363 2024-09-13T14:33:26.637 INFO:tasks.rados.rados.0.smithi151.stdout:3002: finishing write tid 1 to smithi15186934-27 2024-09-13T14:33:26.637 INFO:tasks.rados.rados.0.smithi151.stdout:3004: delete oid 42 current snap is 381 2024-09-13T14:33:26.638 INFO:tasks.rados.rados.0.smithi151.stdout:3004: done (2 left) 2024-09-13T14:33:26.638 INFO:tasks.rados.rados.0.smithi151.stdout:3005: read oid 25 snap -1 2024-09-13T14:33:26.638 INFO:tasks.rados.rados.0.smithi151.stdout:3005: expect deleted 2024-09-13T14:33:26.639 INFO:tasks.rados.rados.0.smithi151.stdout:3006: write oid 46 current snap is 381 2024-09-13T14:33:26.639 INFO:tasks.rados.rados.0.smithi151.stdout:3006: seq_num 1135 ranges {675945=744212,2167533=755390} 2024-09-13T14:33:26.644 INFO:tasks.rados.rados.0.smithi151.stdout:3006: writing smithi15186934-46 from 675945 to 1420157 tid 1 2024-09-13T14:33:26.653 INFO:tasks.rados.rados.0.smithi151.stdout:3006: writing smithi15186934-46 from 2167533 to 2922923 tid 2 2024-09-13T14:33:26.653 INFO:tasks.rados.rados.0.smithi151.stdout:3007: write oid 9 current snap is 381 2024-09-13T14:33:26.653 INFO:tasks.rados.rados.0.smithi151.stdout:3007: seq_num 1136 ranges {692752=419461,1826015=615401,2979461=113291} 2024-09-13T14:33:26.659 INFO:tasks.rados.rados.0.smithi151.stdout:3007: writing smithi15186934-9 from 692752 to 1112213 tid 1 2024-09-13T14:33:26.666 INFO:tasks.rados.rados.0.smithi151.stdout:3007: writing smithi15186934-9 from 1826015 to 2441416 tid 2 2024-09-13T14:33:26.671 INFO:tasks.rados.rados.0.smithi151.stdout:3007: writing smithi15186934-9 from 2979461 to 3092752 tid 3 2024-09-13T14:33:26.671 INFO:tasks.rados.rados.0.smithi151.stdout:3008: rollback oid 38 current snap is 381 2024-09-13T14:33:26.671 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 38 to 372 2024-09-13T14:33:26.671 INFO:tasks.rados.rados.0.smithi151.stdout:3003: finishing rollback tid 0 to smithi15186934-33 2024-09-13T14:33:26.671 INFO:tasks.rados.rados.0.smithi151.stdout:3003: finishing rollback tid 1 to smithi15186934-33 2024-09-13T14:33:26.671 INFO:tasks.rados.rados.0.smithi151.stdout:3003: finishing rollback tid 2 to smithi15186934-33 2024-09-13T14:33:26.671 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 33 v 1091 (ObjNum 1058 snap 357 seq_num 1058) dirty exists 2024-09-13T14:33:26.672 INFO:tasks.rados.rados.0.smithi151.stdout:3002: finishing write tid 2 to smithi15186934-27 2024-09-13T14:33:26.672 INFO:tasks.rados.rados.0.smithi151.stdout:3002: finishing write tid 3 to smithi15186934-27 2024-09-13T14:33:26.672 INFO:tasks.rados.rados.0.smithi151.stdout:3002: finishing write tid 4 to smithi15186934-27 2024-09-13T14:33:26.672 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 27 v 768 (ObjNum 1133 snap 381 seq_num 1133) dirty exists 2024-09-13T14:33:26.672 INFO:tasks.rados.rados.0.smithi151.stdout:3002: left oid 27 (ObjNum 1133 snap 381 seq_num 1133) 2024-09-13T14:33:26.672 INFO:tasks.rados.rados.0.smithi151.stdout:3006: finishing write tid 1 to smithi15186934-46 2024-09-13T14:33:26.672 INFO:tasks.rados.rados.0.smithi151.stdout:3002: done (5 left) 2024-09-13T14:33:26.672 INFO:tasks.rados.rados.0.smithi151.stdout:3003: done (4 left) 2024-09-13T14:33:26.672 INFO:tasks.rados.rados.0.smithi151.stdout:3005: done (3 left) 2024-09-13T14:33:26.672 INFO:tasks.rados.rados.0.smithi151.stdout:3009: rollback oid 35 current snap is 381 2024-09-13T14:33:26.672 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 35 to 367 2024-09-13T14:33:26.672 INFO:tasks.rados.rados.0.smithi151.stdout:3006: finishing write tid 2 to smithi15186934-46 2024-09-13T14:33:26.672 INFO:tasks.rados.rados.0.smithi151.stdout:3006: finishing write tid 3 to smithi15186934-46 2024-09-13T14:33:26.673 INFO:tasks.rados.rados.0.smithi151.stdout:3006: finishing write tid 4 to smithi15186934-46 2024-09-13T14:33:26.673 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 46 v 1087 (ObjNum 1135 snap 381 seq_num 1135) dirty exists 2024-09-13T14:33:26.673 INFO:tasks.rados.rados.0.smithi151.stdout:3006: left oid 46 (ObjNum 1135 snap 381 seq_num 1135) 2024-09-13T14:33:26.673 INFO:tasks.rados.rados.0.smithi151.stdout:3007: finishing write tid 1 to smithi15186934-9 2024-09-13T14:33:26.673 INFO:tasks.rados.rados.0.smithi151.stdout:3007: finishing write tid 2 to smithi15186934-9 2024-09-13T14:33:26.673 INFO:tasks.rados.rados.0.smithi151.stdout:3006: done (3 left) 2024-09-13T14:33:26.673 INFO:tasks.rados.rados.0.smithi151.stdout:3010: read oid 27 snap -1 2024-09-13T14:33:26.673 INFO:tasks.rados.rados.0.smithi151.stdout:3010: expect (ObjNum 1133 snap 381 seq_num 1133) 2024-09-13T14:33:26.673 INFO:tasks.rados.rados.0.smithi151.stdout:3011: snap_remove snap 380 2024-09-13T14:33:26.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:26 smithi184 ceph-mon[92527]: pgmap v840: 89 pgs: 89 active+clean; 207 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:33:26.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:26 smithi184 ceph-mon[92527]: osdmap e815: 8 total, 8 up, 8 in 2024-09-13T14:33:26.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:26 smithi151 ceph-mon[104450]: pgmap v840: 89 pgs: 89 active+clean; 207 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:33:26.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:26 smithi151 ceph-mon[104450]: osdmap e815: 8 total, 8 up, 8 in 2024-09-13T14:33:26.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:26 smithi151 ceph-mon[108966]: pgmap v840: 89 pgs: 89 active+clean; 207 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:33:26.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:26 smithi151 ceph-mon[108966]: osdmap e815: 8 total, 8 up, 8 in 2024-09-13T14:33:27.627 INFO:tasks.rados.rados.0.smithi151.stdout:3011: done (4 left) 2024-09-13T14:33:27.627 INFO:tasks.rados.rados.0.smithi151.stdout:3012: read oid 10 snap -1 2024-09-13T14:33:27.627 INFO:tasks.rados.rados.0.smithi151.stdout:3012: expect (ObjNum 980 snap 340 seq_num 980) 2024-09-13T14:33:27.627 INFO:tasks.rados.rados.0.smithi151.stdout:3013: read oid 6 snap -1 2024-09-13T14:33:27.627 INFO:tasks.rados.rados.0.smithi151.stdout:3013: expect deleted 2024-09-13T14:33:27.627 INFO:tasks.rados.rados.0.smithi151.stdout:3014: read oid 24 snap -1 2024-09-13T14:33:27.627 INFO:tasks.rados.rados.0.smithi151.stdout:3014: expect deleted 2024-09-13T14:33:27.627 INFO:tasks.rados.rados.0.smithi151.stdout:3015: delete oid 26 current snap is 381 2024-09-13T14:33:27.628 INFO:tasks.rados.rados.0.smithi151.stdout:3008: finishing rollback tid 1 to smithi15186934-38 2024-09-13T14:33:27.628 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 1199 (ObjNum 33 snap 0 seq_num 3064833400) dirty dne 2024-09-13T14:33:27.628 INFO:tasks.rados.rados.0.smithi151.stdout:3007: finishing write tid 3 to smithi15186934-9 2024-09-13T14:33:27.628 INFO:tasks.rados.rados.0.smithi151.stdout:3007: finishing write tid 4 to smithi15186934-9 2024-09-13T14:33:27.628 INFO:tasks.rados.rados.0.smithi151.stdout:3007: finishing write tid 5 to smithi15186934-9 2024-09-13T14:33:27.628 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 1224 (ObjNum 1136 snap 381 seq_num 1136) dirty exists 2024-09-13T14:33:27.628 INFO:tasks.rados.rados.0.smithi151.stdout:3007: left oid 9 (ObjNum 1136 snap 381 seq_num 1136) 2024-09-13T14:33:27.628 INFO:tasks.rados.rados.0.smithi151.stdout:3009: finishing rollback tid 1 to smithi15186934-35 2024-09-13T14:33:27.628 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 35 v 1753 (ObjNum 209 snap 0 seq_num 3680972523) dirty dne 2024-09-13T14:33:27.628 INFO:tasks.rados.rados.0.smithi151.stdout:3010: expect (ObjNum 1133 snap 381 seq_num 1133) 2024-09-13T14:33:27.728 INFO:tasks.rados.rados.0.smithi151.stdout:3012: expect (ObjNum 980 snap 340 seq_num 980) 2024-09-13T14:33:27.856 INFO:tasks.rados.rados.0.smithi151.stdout:3007: done (7 left) 2024-09-13T14:33:27.856 INFO:tasks.rados.rados.0.smithi151.stdout:3008: done (6 left) 2024-09-13T14:33:27.856 INFO:tasks.rados.rados.0.smithi151.stdout:3009: done (5 left) 2024-09-13T14:33:27.857 INFO:tasks.rados.rados.0.smithi151.stdout:3010: done (4 left) 2024-09-13T14:33:27.857 INFO:tasks.rados.rados.0.smithi151.stdout:3012: done (3 left) 2024-09-13T14:33:27.857 INFO:tasks.rados.rados.0.smithi151.stdout:3013: done (2 left) 2024-09-13T14:33:27.857 INFO:tasks.rados.rados.0.smithi151.stdout:3014: done (1 left) 2024-09-13T14:33:27.857 INFO:tasks.rados.rados.0.smithi151.stdout:3015: done (0 left) 2024-09-13T14:33:27.857 INFO:tasks.rados.rados.0.smithi151.stdout:3016: snap_create 2024-09-13T14:33:28.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:27 smithi184 ceph-mon[92527]: osdmap e816: 8 total, 8 up, 8 in 2024-09-13T14:33:28.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:27 smithi151 ceph-mon[104450]: osdmap e816: 8 total, 8 up, 8 in 2024-09-13T14:33:28.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:27 smithi151 ceph-mon[108966]: osdmap e816: 8 total, 8 up, 8 in 2024-09-13T14:33:28.627 INFO:tasks.rados.rados.0.smithi151.stdout:3016: done (0 left) 2024-09-13T14:33:28.627 INFO:tasks.rados.rados.0.smithi151.stdout:3017: rollback oid 13 current snap is 382 2024-09-13T14:33:28.627 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 13 to 359 2024-09-13T14:33:28.627 INFO:tasks.rados.rados.0.smithi151.stdout:3018: snap_create 2024-09-13T14:33:28.632 INFO:tasks.rados.rados.0.smithi151.stdout:3017: finishing rollback tid 1 to smithi15186934-13 2024-09-13T14:33:28.632 INFO:tasks.rados.rados.0.smithi151.stdout:3017: finishing rollback tid 2 to smithi15186934-13 2024-09-13T14:33:28.632 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 13 v 1095 (ObjNum 1043 snap 355 seq_num 1043) dirty exists 2024-09-13T14:33:29.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:28 smithi184 ceph-mon[92527]: pgmap v843: 89 pgs: 89 active+clean; 216 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.1 MiB/s rd, 1.3 MiB/s wr, 10 op/s 2024-09-13T14:33:29.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:28 smithi184 ceph-mon[92527]: osdmap e817: 8 total, 8 up, 8 in 2024-09-13T14:33:29.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:28 smithi151 ceph-mon[104450]: pgmap v843: 89 pgs: 89 active+clean; 216 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.1 MiB/s rd, 1.3 MiB/s wr, 10 op/s 2024-09-13T14:33:29.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:28 smithi151 ceph-mon[104450]: osdmap e817: 8 total, 8 up, 8 in 2024-09-13T14:33:29.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:28 smithi151 ceph-mon[108966]: pgmap v843: 89 pgs: 89 active+clean; 216 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.1 MiB/s rd, 1.3 MiB/s wr, 10 op/s 2024-09-13T14:33:29.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:28 smithi151 ceph-mon[108966]: osdmap e817: 8 total, 8 up, 8 in 2024-09-13T14:33:29.597 INFO:tasks.rados.rados.0.smithi151.stdout:3017: done (1 left) 2024-09-13T14:33:29.597 INFO:tasks.rados.rados.0.smithi151.stdout:3018: done (0 left) 2024-09-13T14:33:29.597 INFO:tasks.rados.rados.0.smithi151.stdout:3019: read oid 9 snap -1 2024-09-13T14:33:29.597 INFO:tasks.rados.rados.0.smithi151.stdout:3019: expect (ObjNum 1136 snap 381 seq_num 1136) 2024-09-13T14:33:29.597 INFO:tasks.rados.rados.0.smithi151.stdout:3020: write oid 11 current snap is 383 2024-09-13T14:33:29.597 INFO:tasks.rados.rados.0.smithi151.stdout:3020: seq_num 1138 ranges {726366=569959,1826626=502717,2801601=324765} 2024-09-13T14:33:29.604 INFO:tasks.rados.rados.0.smithi151.stdout:3020: writing smithi15186934-11 from 726366 to 1296325 tid 1 2024-09-13T14:33:29.609 INFO:tasks.rados.rados.0.smithi151.stdout:3020: writing smithi15186934-11 from 1826626 to 2329343 tid 2 2024-09-13T14:33:29.614 INFO:tasks.rados.rados.0.smithi151.stdout:3020: writing smithi15186934-11 from 2801601 to 3126366 tid 3 2024-09-13T14:33:29.614 INFO:tasks.rados.rados.0.smithi151.stdout:3021: snap_remove snap 375 2024-09-13T14:33:30.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:29 smithi184 ceph-mon[92527]: osdmap e818: 8 total, 8 up, 8 in 2024-09-13T14:33:30.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:29 smithi184 ceph-mon[92527]: osdmap e819: 8 total, 8 up, 8 in 2024-09-13T14:33:30.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:29 smithi151 ceph-mon[104450]: osdmap e818: 8 total, 8 up, 8 in 2024-09-13T14:33:30.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:29 smithi151 ceph-mon[104450]: osdmap e819: 8 total, 8 up, 8 in 2024-09-13T14:33:30.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:29 smithi151 ceph-mon[108966]: osdmap e818: 8 total, 8 up, 8 in 2024-09-13T14:33:30.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:29 smithi151 ceph-mon[108966]: osdmap e819: 8 total, 8 up, 8 in 2024-09-13T14:33:30.599 INFO:tasks.rados.rados.0.smithi151.stdout:3021: done (2 left) 2024-09-13T14:33:30.600 INFO:tasks.rados.rados.0.smithi151.stdout:3022: write oid 27 current snap is 383 2024-09-13T14:33:30.600 INFO:tasks.rados.rados.0.smithi151.stdout:3022: seq_num 1139 ranges {743173=645208,1885108=762728} 2024-09-13T14:33:30.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:33:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:33:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:33:30.608 INFO:tasks.rados.rados.0.smithi151.stdout:3022: writing smithi15186934-27 from 743173 to 1388381 tid 1 2024-09-13T14:33:30.615 INFO:tasks.rados.rados.0.smithi151.stdout:3022: writing smithi15186934-27 from 1885108 to 2647836 tid 2 2024-09-13T14:33:30.615 INFO:tasks.rados.rados.0.smithi151.stdout:3023: write oid 5 current snap is 383 2024-09-13T14:33:30.615 INFO:tasks.rados.rados.0.smithi151.stdout:3023: seq_num 1140 ranges {759980=720457,1943590=506386,2856447=303533} 2024-09-13T14:33:30.623 INFO:tasks.rados.rados.0.smithi151.stdout:3023: writing smithi15186934-5 from 759980 to 1480437 tid 1 2024-09-13T14:33:30.628 INFO:tasks.rados.rados.0.smithi151.stdout:3023: writing smithi15186934-5 from 1943590 to 2449976 tid 2 2024-09-13T14:33:30.633 INFO:tasks.rados.rados.0.smithi151.stdout:3023: writing smithi15186934-5 from 2856447 to 3159980 tid 3 2024-09-13T14:33:30.633 INFO:tasks.rados.rados.0.smithi151.stdout:3024: read oid 1 snap -1 2024-09-13T14:33:30.633 INFO:tasks.rados.rados.0.smithi151.stdout:3024: expect (ObjNum 1075 snap 366 seq_num 1075) 2024-09-13T14:33:30.633 INFO:tasks.rados.rados.0.smithi151.stdout:3025: snap_remove snap 367 2024-09-13T14:33:31.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:30 smithi184 ceph-mon[92527]: pgmap v846: 89 pgs: 89 active+clean; 223 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.9 MiB/s wr, 16 op/s 2024-09-13T14:33:31.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:30 smithi184 ceph-mon[92527]: osdmap e820: 8 total, 8 up, 8 in 2024-09-13T14:33:31.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:30 smithi151 ceph-mon[104450]: pgmap v846: 89 pgs: 89 active+clean; 223 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.9 MiB/s wr, 16 op/s 2024-09-13T14:33:31.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:30 smithi151 ceph-mon[104450]: osdmap e820: 8 total, 8 up, 8 in 2024-09-13T14:33:31.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:30 smithi151 ceph-mon[108966]: pgmap v846: 89 pgs: 89 active+clean; 223 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.9 MiB/s wr, 16 op/s 2024-09-13T14:33:31.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:30 smithi151 ceph-mon[108966]: osdmap e820: 8 total, 8 up, 8 in 2024-09-13T14:33:31.601 INFO:tasks.rados.rados.0.smithi151.stdout:3025: done (5 left) 2024-09-13T14:33:31.601 INFO:tasks.rados.rados.0.smithi151.stdout:3026: snap_create 2024-09-13T14:33:31.601 INFO:tasks.rados.rados.0.smithi151.stdout:3019: expect (ObjNum 1136 snap 381 seq_num 1136) 2024-09-13T14:33:31.745 INFO:tasks.rados.rados.0.smithi151.stdout:3020: finishing write tid 1 to smithi15186934-11 2024-09-13T14:33:31.745 INFO:tasks.rados.rados.0.smithi151.stdout:3020: finishing write tid 2 to smithi15186934-11 2024-09-13T14:33:31.745 INFO:tasks.rados.rados.0.smithi151.stdout:3020: finishing write tid 3 to smithi15186934-11 2024-09-13T14:33:31.745 INFO:tasks.rados.rados.0.smithi151.stdout:3020: finishing write tid 4 to smithi15186934-11 2024-09-13T14:33:31.745 INFO:tasks.rados.rados.0.smithi151.stdout:3020: finishing write tid 5 to smithi15186934-11 2024-09-13T14:33:31.745 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 11 v 704 (ObjNum 1138 snap 383 seq_num 1138) dirty exists 2024-09-13T14:33:31.745 INFO:tasks.rados.rados.0.smithi151.stdout:3020: left oid 11 (ObjNum 1138 snap 383 seq_num 1138) 2024-09-13T14:33:31.746 INFO:tasks.rados.rados.0.smithi151.stdout:3022: finishing write tid 1 to smithi15186934-27 2024-09-13T14:33:31.746 INFO:tasks.rados.rados.0.smithi151.stdout:3022: finishing write tid 2 to smithi15186934-27 2024-09-13T14:33:31.746 INFO:tasks.rados.rados.0.smithi151.stdout:3022: finishing write tid 3 to smithi15186934-27 2024-09-13T14:33:31.746 INFO:tasks.rados.rados.0.smithi151.stdout:3022: finishing write tid 4 to smithi15186934-27 2024-09-13T14:33:31.746 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 27 v 772 (ObjNum 1139 snap 383 seq_num 1139) dirty exists 2024-09-13T14:33:31.746 INFO:tasks.rados.rados.0.smithi151.stdout:3022: left oid 27 (ObjNum 1139 snap 383 seq_num 1139) 2024-09-13T14:33:31.746 INFO:tasks.rados.rados.0.smithi151.stdout:3023: finishing write tid 1 to smithi15186934-5 2024-09-13T14:33:31.746 INFO:tasks.rados.rados.0.smithi151.stdout:3023: finishing write tid 2 to smithi15186934-5 2024-09-13T14:33:31.746 INFO:tasks.rados.rados.0.smithi151.stdout:3023: finishing write tid 3 to smithi15186934-5 2024-09-13T14:33:31.746 INFO:tasks.rados.rados.0.smithi151.stdout:3023: finishing write tid 4 to smithi15186934-5 2024-09-13T14:33:31.746 INFO:tasks.rados.rados.0.smithi151.stdout:3023: finishing write tid 5 to smithi15186934-5 2024-09-13T14:33:31.746 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 1101 (ObjNum 1140 snap 383 seq_num 1140) dirty exists 2024-09-13T14:33:31.746 INFO:tasks.rados.rados.0.smithi151.stdout:3023: left oid 5 (ObjNum 1140 snap 383 seq_num 1140) 2024-09-13T14:33:31.747 INFO:tasks.rados.rados.0.smithi151.stdout:3024: expect (ObjNum 1075 snap 366 seq_num 1075) 2024-09-13T14:33:32.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:32.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:31 smithi184 ceph-mon[92527]: osdmap e821: 8 total, 8 up, 8 in 2024-09-13T14:33:32.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:32.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:31 smithi151 ceph-mon[104450]: osdmap e821: 8 total, 8 up, 8 in 2024-09-13T14:33:32.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:32.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:31 smithi151 ceph-mon[108966]: osdmap e821: 8 total, 8 up, 8 in 2024-09-13T14:33:32.605 INFO:tasks.rados.rados.0.smithi151.stdout:3019: done (5 left) 2024-09-13T14:33:32.605 INFO:tasks.rados.rados.0.smithi151.stdout:3020: done (4 left) 2024-09-13T14:33:32.605 INFO:tasks.rados.rados.0.smithi151.stdout:3022: done (3 left) 2024-09-13T14:33:32.605 INFO:tasks.rados.rados.0.smithi151.stdout:3023: done (2 left) 2024-09-13T14:33:32.605 INFO:tasks.rados.rados.0.smithi151.stdout:3024: done (1 left) 2024-09-13T14:33:32.605 INFO:tasks.rados.rados.0.smithi151.stdout:3026: done (0 left) 2024-09-13T14:33:32.606 INFO:tasks.rados.rados.0.smithi151.stdout:3027: write oid 49 current snap is 384 2024-09-13T14:33:32.606 INFO:tasks.rados.rados.0.smithi151.stdout:3027: seq_num 1141 ranges {776787=512059,1718425=766397,3000223=176564} 2024-09-13T14:33:32.612 INFO:tasks.rados.rados.0.smithi151.stdout:3027: writing smithi15186934-49 from 776787 to 1288846 tid 1 2024-09-13T14:33:32.618 INFO:tasks.rados.rados.0.smithi151.stdout:3027: writing smithi15186934-49 from 1718425 to 2484822 tid 2 2024-09-13T14:33:32.623 INFO:tasks.rados.rados.0.smithi151.stdout:3027: writing smithi15186934-49 from 3000223 to 3176787 tid 3 2024-09-13T14:33:32.623 INFO:tasks.rados.rados.0.smithi151.stdout:3028: read oid 38 snap -1 2024-09-13T14:33:32.623 INFO:tasks.rados.rados.0.smithi151.stdout:3028: expect deleted 2024-09-13T14:33:32.623 INFO:tasks.rados.rados.0.smithi151.stdout:3029: rollback oid 21 current snap is 384 2024-09-13T14:33:32.623 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 21 to 376 2024-09-13T14:33:32.623 INFO:tasks.rados.rados.0.smithi151.stdout:3027: finishing write tid 1 to smithi15186934-49 2024-09-13T14:33:32.623 INFO:tasks.rados.rados.0.smithi151.stdout:3027: finishing write tid 2 to smithi15186934-49 2024-09-13T14:33:32.624 INFO:tasks.rados.rados.0.smithi151.stdout:3030: snap_create 2024-09-13T14:33:32.627 INFO:tasks.rados.rados.0.smithi151.stdout:3027: finishing write tid 3 to smithi15186934-49 2024-09-13T14:33:32.627 INFO:tasks.rados.rados.0.smithi151.stdout:3027: finishing write tid 4 to smithi15186934-49 2024-09-13T14:33:32.628 INFO:tasks.rados.rados.0.smithi151.stdout:3027: finishing write tid 5 to smithi15186934-49 2024-09-13T14:33:32.628 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 49 v 1334 (ObjNum 1141 snap 384 seq_num 1141) dirty exists 2024-09-13T14:33:32.628 INFO:tasks.rados.rados.0.smithi151.stdout:3027: left oid 49 (ObjNum 1141 snap 384 seq_num 1141) 2024-09-13T14:33:32.628 INFO:tasks.rados.rados.0.smithi151.stdout:3029: finishing rollback tid 0 to smithi15186934-21 2024-09-13T14:33:32.628 INFO:tasks.rados.rados.0.smithi151.stdout:3029: finishing rollback tid 1 to smithi15186934-21 2024-09-13T14:33:32.628 INFO:tasks.rados.rados.0.smithi151.stdout:3029: finishing rollback tid 2 to smithi15186934-21 2024-09-13T14:33:32.629 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 21 v 586 (ObjNum 937 snap 323 seq_num 937) dirty exists 2024-09-13T14:33:33.009 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:32 smithi151 ceph-mon[108966]: pgmap v849: 89 pgs: 89 active+clean; 230 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.2 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2024-09-13T14:33:33.010 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:32 smithi151 ceph-mon[108966]: osdmap e822: 8 total, 8 up, 8 in 2024-09-13T14:33:33.010 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:32 smithi151 ceph-mon[104450]: pgmap v849: 89 pgs: 89 active+clean; 230 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.2 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2024-09-13T14:33:33.010 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:32 smithi151 ceph-mon[104450]: osdmap e822: 8 total, 8 up, 8 in 2024-09-13T14:33:33.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:32 smithi184 ceph-mon[92527]: pgmap v849: 89 pgs: 89 active+clean; 230 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.2 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2024-09-13T14:33:33.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:32 smithi184 ceph-mon[92527]: osdmap e822: 8 total, 8 up, 8 in 2024-09-13T14:33:33.605 INFO:tasks.rados.rados.0.smithi151.stdout:3027: done (3 left) 2024-09-13T14:33:33.605 INFO:tasks.rados.rados.0.smithi151.stdout:3028: done (2 left) 2024-09-13T14:33:33.605 INFO:tasks.rados.rados.0.smithi151.stdout:3029: done (1 left) 2024-09-13T14:33:33.605 INFO:tasks.rados.rados.0.smithi151.stdout:3030: done (0 left) 2024-09-13T14:33:33.605 INFO:tasks.rados.rados.0.smithi151.stdout:3031: rollback oid 10 current snap is 385 2024-09-13T14:33:33.606 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 10 to 379 2024-09-13T14:33:33.606 INFO:tasks.rados.rados.0.smithi151.stdout:3032: snap_remove snap 381 2024-09-13T14:33:34.598 INFO:tasks.rados.rados.0.smithi151.stdout:3032: done (1 left) 2024-09-13T14:33:34.598 INFO:tasks.rados.rados.0.smithi151.stdout:3033: read oid 41 snap -1 2024-09-13T14:33:34.598 INFO:tasks.rados.rados.0.smithi151.stdout:3033: expect deleted 2024-09-13T14:33:34.598 INFO:tasks.rados.rados.0.smithi151.stdout:3034: snap_remove snap 372 2024-09-13T14:33:34.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:34 smithi184 ceph-mon[92527]: pgmap v852: 89 pgs: 89 active+clean; 231 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 7 op/s 2024-09-13T14:33:34.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:34 smithi184 ceph-mon[92527]: osdmap e823: 8 total, 8 up, 8 in 2024-09-13T14:33:34.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:34 smithi184 ceph-mon[92527]: osdmap e824: 8 total, 8 up, 8 in 2024-09-13T14:33:34.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:33:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:33:34] "GET /metrics HTTP/1.1" 200 36514 "" "Prometheus/2.51.0" 2024-09-13T14:33:34.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:34 smithi151 ceph-mon[104450]: pgmap v852: 89 pgs: 89 active+clean; 231 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 7 op/s 2024-09-13T14:33:34.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:34 smithi151 ceph-mon[104450]: osdmap e823: 8 total, 8 up, 8 in 2024-09-13T14:33:34.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:34 smithi151 ceph-mon[104450]: osdmap e824: 8 total, 8 up, 8 in 2024-09-13T14:33:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:34 smithi151 ceph-mon[108966]: pgmap v852: 89 pgs: 89 active+clean; 231 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 7 op/s 2024-09-13T14:33:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:34 smithi151 ceph-mon[108966]: osdmap e823: 8 total, 8 up, 8 in 2024-09-13T14:33:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:34 smithi151 ceph-mon[108966]: osdmap e824: 8 total, 8 up, 8 in 2024-09-13T14:33:35.601 INFO:tasks.rados.rados.0.smithi151.stdout:3034: done (2 left) 2024-09-13T14:33:35.601 INFO:tasks.rados.rados.0.smithi151.stdout:3035: write oid 45 current snap is 385 2024-09-13T14:33:35.602 INFO:tasks.rados.rados.0.smithi151.stdout:3035: seq_num 1142 ranges {793594=587308,2060554=510055} 2024-09-13T14:33:35.610 INFO:tasks.rados.rados.0.smithi151.stdout:3035: writing smithi15186934-45 from 793594 to 1380902 tid 1 2024-09-13T14:33:35.617 INFO:tasks.rados.rados.0.smithi151.stdout:3035: writing smithi15186934-45 from 2060554 to 2570609 tid 2 2024-09-13T14:33:35.618 INFO:tasks.rados.rados.0.smithi151.stdout:3036: delete oid 22 current snap is 385 2024-09-13T14:33:35.618 INFO:tasks.rados.rados.0.smithi151.stdout:3031: finishing rollback tid 0 to smithi15186934-10 2024-09-13T14:33:35.618 INFO:tasks.rados.rados.0.smithi151.stdout:3031: finishing rollback tid 1 to smithi15186934-10 2024-09-13T14:33:35.618 INFO:tasks.rados.rados.0.smithi151.stdout:3031: finishing rollback tid 2 to smithi15186934-10 2024-09-13T14:33:35.618 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 1105 (ObjNum 980 snap 340 seq_num 980) dirty exists 2024-09-13T14:33:35.618 INFO:tasks.rados.rados.0.smithi151.stdout:3035: finishing write tid 1 to smithi15186934-45 2024-09-13T14:33:35.619 INFO:tasks.rados.rados.0.smithi151.stdout:3031: done (3 left) 2024-09-13T14:33:35.619 INFO:tasks.rados.rados.0.smithi151.stdout:3033: done (2 left) 2024-09-13T14:33:35.619 INFO:tasks.rados.rados.0.smithi151.stdout:3036: done (1 left) 2024-09-13T14:33:35.619 INFO:tasks.rados.rados.0.smithi151.stdout:3037: snap_remove snap 356 2024-09-13T14:33:36.609 INFO:tasks.rados.rados.0.smithi151.stdout:3035: finishing write tid 2 to smithi15186934-45 2024-09-13T14:33:36.609 INFO:tasks.rados.rados.0.smithi151.stdout:3035: finishing write tid 3 to smithi15186934-45 2024-09-13T14:33:36.609 INFO:tasks.rados.rados.0.smithi151.stdout:3035: finishing write tid 4 to smithi15186934-45 2024-09-13T14:33:36.609 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 1457 (ObjNum 1142 snap 385 seq_num 1142) dirty exists 2024-09-13T14:33:36.609 INFO:tasks.rados.rados.0.smithi151.stdout:3035: left oid 45 (ObjNum 1142 snap 385 seq_num 1142) 2024-09-13T14:33:36.609 INFO:tasks.rados.rados.0.smithi151.stdout:3035: done (1 left) 2024-09-13T14:33:36.609 INFO:tasks.rados.rados.0.smithi151.stdout:3037: done (0 left) 2024-09-13T14:33:36.609 INFO:tasks.rados.rados.0.smithi151.stdout:3038: delete oid 22 current snap is 385 2024-09-13T14:33:36.617 INFO:tasks.rados.rados.0.smithi151.stdout:3038: done (0 left) 2024-09-13T14:33:36.617 INFO:tasks.rados.rados.0.smithi151.stdout:3039: write oid 8 current snap is 385 2024-09-13T14:33:36.617 INFO:tasks.rados.rados.0.smithi151.stdout:3039: seq_num 1145 ranges {444015=413055,1436000=773735,2709915=534100} 2024-09-13T14:33:36.622 INFO:tasks.rados.rados.0.smithi151.stdout:3039: writing smithi15186934-8 from 444015 to 857070 tid 1 2024-09-13T14:33:36.630 INFO:tasks.rados.rados.0.smithi151.stdout:3039: writing smithi15186934-8 from 1436000 to 2209735 tid 2 2024-09-13T14:33:36.636 INFO:tasks.rados.rados.0.smithi151.stdout:3039: writing smithi15186934-8 from 2709915 to 3244015 tid 3 2024-09-13T14:33:36.636 INFO:tasks.rados.rados.0.smithi151.stdout:3040: delete oid 22 current snap is 385 2024-09-13T14:33:36.636 INFO:tasks.rados.rados.0.smithi151.stdout:3039: finishing write tid 1 to smithi15186934-8 2024-09-13T14:33:36.636 INFO:tasks.rados.rados.0.smithi151.stdout:3039: finishing write tid 2 to smithi15186934-8 2024-09-13T14:33:36.637 INFO:tasks.rados.rados.0.smithi151.stdout:3040: done (1 left) 2024-09-13T14:33:36.638 INFO:tasks.rados.rados.0.smithi151.stdout:3041: read oid 31 snap -1 2024-09-13T14:33:36.638 INFO:tasks.rados.rados.0.smithi151.stdout:3041: expect deleted 2024-09-13T14:33:36.638 INFO:tasks.rados.rados.0.smithi151.stdout:3042: write oid 47 current snap is 385 2024-09-13T14:33:36.638 INFO:tasks.rados.rados.0.smithi151.stdout:3042: seq_num 1147 ranges {477629=563553,1836611=777404,3048408=229221} 2024-09-13T14:33:36.643 INFO:tasks.rados.rados.0.smithi151.stdout:3042: writing smithi15186934-47 from 477629 to 1041182 tid 1 2024-09-13T14:33:36.652 INFO:tasks.rados.rados.0.smithi151.stdout:3042: writing smithi15186934-47 from 1836611 to 2614015 tid 2 2024-09-13T14:33:36.656 INFO:tasks.rados.rados.0.smithi151.stdout:3042: writing smithi15186934-47 from 3048408 to 3277629 tid 3 2024-09-13T14:33:36.656 INFO:tasks.rados.rados.0.smithi151.stdout:3043: read oid 48 snap -1 2024-09-13T14:33:36.656 INFO:tasks.rados.rados.0.smithi151.stdout:3043: expect (ObjNum 1111 snap 372 seq_num 1111) 2024-09-13T14:33:36.657 INFO:tasks.rados.rados.0.smithi151.stdout:3044: snap_create 2024-09-13T14:33:36.657 INFO:tasks.rados.rados.0.smithi151.stdout:3039: finishing write tid 3 to smithi15186934-8 2024-09-13T14:33:36.657 INFO:tasks.rados.rados.0.smithi151.stdout:3039: finishing write tid 4 to smithi15186934-8 2024-09-13T14:33:36.657 INFO:tasks.rados.rados.0.smithi151.stdout:3039: finishing write tid 5 to smithi15186934-8 2024-09-13T14:33:36.657 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 906 (ObjNum 1145 snap 385 seq_num 1145) dirty exists 2024-09-13T14:33:36.657 INFO:tasks.rados.rados.0.smithi151.stdout:3039: left oid 8 (ObjNum 1145 snap 385 seq_num 1145) 2024-09-13T14:33:36.657 INFO:tasks.rados.rados.0.smithi151.stdout:3042: finishing write tid 1 to smithi15186934-47 2024-09-13T14:33:36.657 INFO:tasks.rados.rados.0.smithi151.stdout:3042: finishing write tid 2 to smithi15186934-47 2024-09-13T14:33:36.659 INFO:tasks.rados.rados.0.smithi151.stdout:3042: finishing write tid 3 to smithi15186934-47 2024-09-13T14:33:36.659 INFO:tasks.rados.rados.0.smithi151.stdout:3042: finishing write tid 4 to smithi15186934-47 2024-09-13T14:33:36.659 INFO:tasks.rados.rados.0.smithi151.stdout:3042: finishing write tid 5 to smithi15186934-47 2024-09-13T14:33:36.659 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 1349 (ObjNum 1147 snap 385 seq_num 1147) dirty exists 2024-09-13T14:33:36.659 INFO:tasks.rados.rados.0.smithi151.stdout:3042: left oid 47 (ObjNum 1147 snap 385 seq_num 1147) 2024-09-13T14:33:36.664 INFO:tasks.rados.rados.0.smithi151.stdout:3043: expect (ObjNum 1111 snap 372 seq_num 1111) 2024-09-13T14:33:36.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:36 smithi184 ceph-mon[92527]: pgmap v855: 89 pgs: 89 active+clean; 234 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 718 KiB/s wr, 2 op/s 2024-09-13T14:33:36.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:36 smithi184 ceph-mon[92527]: osdmap e825: 8 total, 8 up, 8 in 2024-09-13T14:33:36.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:36 smithi151 ceph-mon[104450]: pgmap v855: 89 pgs: 89 active+clean; 234 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 718 KiB/s wr, 2 op/s 2024-09-13T14:33:36.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:36 smithi151 ceph-mon[104450]: osdmap e825: 8 total, 8 up, 8 in 2024-09-13T14:33:36.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:36 smithi151 ceph-mon[108966]: pgmap v855: 89 pgs: 89 active+clean; 234 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 718 KiB/s wr, 2 op/s 2024-09-13T14:33:36.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:36 smithi151 ceph-mon[108966]: osdmap e825: 8 total, 8 up, 8 in 2024-09-13T14:33:37.614 INFO:tasks.rados.rados.0.smithi151.stdout:3039: done (4 left) 2024-09-13T14:33:37.614 INFO:tasks.rados.rados.0.smithi151.stdout:3041: done (3 left) 2024-09-13T14:33:37.614 INFO:tasks.rados.rados.0.smithi151.stdout:3042: done (2 left) 2024-09-13T14:33:37.614 INFO:tasks.rados.rados.0.smithi151.stdout:3043: done (1 left) 2024-09-13T14:33:37.614 INFO:tasks.rados.rados.0.smithi151.stdout:3044: done (0 left) 2024-09-13T14:33:37.614 INFO:tasks.rados.rados.0.smithi151.stdout:3045: snap_create 2024-09-13T14:33:38.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:37 smithi184 ceph-mon[92527]: osdmap e826: 8 total, 8 up, 8 in 2024-09-13T14:33:38.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:37 smithi151 ceph-mon[108966]: osdmap e826: 8 total, 8 up, 8 in 2024-09-13T14:33:38.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:37 smithi151 ceph-mon[104450]: osdmap e826: 8 total, 8 up, 8 in 2024-09-13T14:33:38.620 INFO:tasks.rados.rados.0.smithi151.stdout:3045: done (0 left) 2024-09-13T14:33:38.621 INFO:tasks.rados.rados.0.smithi151.stdout:3046: read oid 22 snap -1 2024-09-13T14:33:38.621 INFO:tasks.rados.rados.0.smithi151.stdout:3046: expect deleted 2024-09-13T14:33:38.621 INFO:tasks.rados.rados.0.smithi151.stdout:3047: snap_create 2024-09-13T14:33:39.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:38 smithi184 ceph-mon[92527]: pgmap v858: 89 pgs: 89 active+clean; 237 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.0 MiB/s rd, 1.3 MiB/s wr, 10 op/s 2024-09-13T14:33:39.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:38 smithi184 ceph-mon[92527]: osdmap e827: 8 total, 8 up, 8 in 2024-09-13T14:33:39.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:38 smithi151 ceph-mon[108966]: pgmap v858: 89 pgs: 89 active+clean; 237 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.0 MiB/s rd, 1.3 MiB/s wr, 10 op/s 2024-09-13T14:33:39.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:38 smithi151 ceph-mon[108966]: osdmap e827: 8 total, 8 up, 8 in 2024-09-13T14:33:39.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:38 smithi151 ceph-mon[104450]: pgmap v858: 89 pgs: 89 active+clean; 237 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.0 MiB/s rd, 1.3 MiB/s wr, 10 op/s 2024-09-13T14:33:39.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:38 smithi151 ceph-mon[104450]: osdmap e827: 8 total, 8 up, 8 in 2024-09-13T14:33:39.601 INFO:tasks.rados.rados.0.smithi151.stdout:3046: done (1 left) 2024-09-13T14:33:39.601 INFO:tasks.rados.rados.0.smithi151.stdout:3047: done (0 left) 2024-09-13T14:33:39.601 INFO:tasks.rados.rados.0.smithi151.stdout:3048: read oid 29 snap -1 2024-09-13T14:33:39.601 INFO:tasks.rados.rados.0.smithi151.stdout:3048: expect (ObjNum 1124 snap 376 seq_num 1124) 2024-09-13T14:33:39.601 INFO:tasks.rados.rados.0.smithi151.stdout:3049: snap_create 2024-09-13T14:33:39.609 INFO:tasks.rados.rados.0.smithi151.stdout:3048: expect (ObjNum 1124 snap 376 seq_num 1124) 2024-09-13T14:33:40.110 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:39 smithi184 ceph-mon[92527]: osdmap e828: 8 total, 8 up, 8 in 2024-09-13T14:33:40.110 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:39 smithi184 ceph-mon[92527]: osdmap e829: 8 total, 8 up, 8 in 2024-09-13T14:33:40.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:39 smithi151 ceph-mon[104450]: osdmap e828: 8 total, 8 up, 8 in 2024-09-13T14:33:40.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:39 smithi151 ceph-mon[104450]: osdmap e829: 8 total, 8 up, 8 in 2024-09-13T14:33:40.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:39 smithi151 ceph-mon[108966]: osdmap e828: 8 total, 8 up, 8 in 2024-09-13T14:33:40.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:39 smithi151 ceph-mon[108966]: osdmap e829: 8 total, 8 up, 8 in 2024-09-13T14:33:40.601 INFO:tasks.rados.rados.0.smithi151.stdout:3048: done (1 left) 2024-09-13T14:33:40.601 INFO:tasks.rados.rados.0.smithi151.stdout:3049: done (0 left) 2024-09-13T14:33:40.601 INFO:tasks.rados.rados.0.smithi151.stdout:3050: write oid 19 current snap is 389 2024-09-13T14:33:40.601 INFO:tasks.rados.rados.0.smithi151.stdout:3050: seq_num 1148 ranges {494436=755155,2011446=521062,3075831=218605} 2024-09-13T14:33:40.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:33:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:33:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:33:40.607 INFO:tasks.rados.rados.0.smithi151.stdout:3050: writing smithi15186934-19 from 494436 to 1249591 tid 1 2024-09-13T14:33:40.615 INFO:tasks.rados.rados.0.smithi151.stdout:3050: writing smithi15186934-19 from 2011446 to 2532508 tid 2 2024-09-13T14:33:40.619 INFO:tasks.rados.rados.0.smithi151.stdout:3050: writing smithi15186934-19 from 3075831 to 3294436 tid 3 2024-09-13T14:33:40.619 INFO:tasks.rados.rados.0.smithi151.stdout:3051: delete oid 35 current snap is 389 2024-09-13T14:33:40.619 INFO:tasks.rados.rados.0.smithi151.stdout:3050: finishing write tid 1 to smithi15186934-19 2024-09-13T14:33:40.619 INFO:tasks.rados.rados.0.smithi151.stdout:3050: finishing write tid 2 to smithi15186934-19 2024-09-13T14:33:40.621 INFO:tasks.rados.rados.0.smithi151.stdout:3050: finishing write tid 3 to smithi15186934-19 2024-09-13T14:33:40.621 INFO:tasks.rados.rados.0.smithi151.stdout:3050: finishing write tid 4 to smithi15186934-19 2024-09-13T14:33:40.621 INFO:tasks.rados.rados.0.smithi151.stdout:3050: finishing write tid 5 to smithi15186934-19 2024-09-13T14:33:40.622 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 19 v 733 (ObjNum 1148 snap 389 seq_num 1148) dirty exists 2024-09-13T14:33:40.622 INFO:tasks.rados.rados.0.smithi151.stdout:3050: left oid 19 (ObjNum 1148 snap 389 seq_num 1148) 2024-09-13T14:33:40.622 INFO:tasks.rados.rados.0.smithi151.stdout:3050: done (1 left) 2024-09-13T14:33:40.623 INFO:tasks.rados.rados.0.smithi151.stdout:3051: done (0 left) 2024-09-13T14:33:40.623 INFO:tasks.rados.rados.0.smithi151.stdout:3052: write oid 25 current snap is 389 2024-09-13T14:33:40.623 INFO:tasks.rados.rados.0.smithi151.stdout:3052: seq_num 1150 ranges {528050=505653,1612057=524731,2614324=567403} 2024-09-13T14:33:40.628 INFO:tasks.rados.rados.0.smithi151.stdout:3052: writing smithi15186934-25 from 528050 to 1033703 tid 1 2024-09-13T14:33:40.634 INFO:tasks.rados.rados.0.smithi151.stdout:3052: writing smithi15186934-25 from 1612057 to 2136788 tid 2 2024-09-13T14:33:40.640 INFO:tasks.rados.rados.0.smithi151.stdout:3052: writing smithi15186934-25 from 2614324 to 3181727 tid 3 2024-09-13T14:33:40.640 INFO:tasks.rados.rados.0.smithi151.stdout:3053: write oid 38 current snap is 389 2024-09-13T14:33:40.640 INFO:tasks.rados.rados.0.smithi151.stdout:3052: finishing write tid 1 to smithi15186934-25 2024-09-13T14:33:40.640 INFO:tasks.rados.rados.0.smithi151.stdout:3052: finishing write tid 2 to smithi15186934-25 2024-09-13T14:33:40.641 INFO:tasks.rados.rados.0.smithi151.stdout:3053: seq_num 1151 ranges {544857=580902,1670539=668389,3041747=303110} 2024-09-13T14:33:40.649 INFO:tasks.rados.rados.0.smithi151.stdout:3053: writing smithi15186934-38 from 544857 to 1125759 tid 1 2024-09-13T14:33:40.653 INFO:tasks.rados.rados.0.smithi151.stdout:3053: writing smithi15186934-38 from 1670539 to 2338928 tid 2 2024-09-13T14:33:40.659 INFO:tasks.rados.rados.0.smithi151.stdout:3053: writing smithi15186934-38 from 3041747 to 3344857 tid 3 2024-09-13T14:33:40.659 INFO:tasks.rados.rados.0.smithi151.stdout:3054: snap_create 2024-09-13T14:33:40.659 INFO:tasks.rados.rados.0.smithi151.stdout:3052: finishing write tid 3 to smithi15186934-25 2024-09-13T14:33:40.659 INFO:tasks.rados.rados.0.smithi151.stdout:3052: finishing write tid 4 to smithi15186934-25 2024-09-13T14:33:40.659 INFO:tasks.rados.rados.0.smithi151.stdout:3052: finishing write tid 5 to smithi15186934-25 2024-09-13T14:33:40.659 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 1467 (ObjNum 1150 snap 389 seq_num 1150) dirty exists 2024-09-13T14:33:40.659 INFO:tasks.rados.rados.0.smithi151.stdout:3052: left oid 25 (ObjNum 1150 snap 389 seq_num 1150) 2024-09-13T14:33:40.659 INFO:tasks.rados.rados.0.smithi151.stdout:3053: finishing write tid 1 to smithi15186934-38 2024-09-13T14:33:40.659 INFO:tasks.rados.rados.0.smithi151.stdout:3053: finishing write tid 2 to smithi15186934-38 2024-09-13T14:33:40.661 INFO:tasks.rados.rados.0.smithi151.stdout:3053: finishing write tid 3 to smithi15186934-38 2024-09-13T14:33:40.661 INFO:tasks.rados.rados.0.smithi151.stdout:3053: finishing write tid 4 to smithi15186934-38 2024-09-13T14:33:40.661 INFO:tasks.rados.rados.0.smithi151.stdout:3053: finishing write tid 5 to smithi15186934-38 2024-09-13T14:33:40.661 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 1230 (ObjNum 1151 snap 389 seq_num 1151) dirty exists 2024-09-13T14:33:40.661 INFO:tasks.rados.rados.0.smithi151.stdout:3053: left oid 38 (ObjNum 1151 snap 389 seq_num 1151) 2024-09-13T14:33:41.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:40 smithi184 ceph-mon[92527]: pgmap v861: 89 pgs: 89 active+clean; 234 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.0 MiB/s rd, 1007 KiB/s wr, 9 op/s 2024-09-13T14:33:41.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:40 smithi184 ceph-mon[92527]: osdmap e830: 8 total, 8 up, 8 in 2024-09-13T14:33:41.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:40 smithi151 ceph-mon[108966]: pgmap v861: 89 pgs: 89 active+clean; 234 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.0 MiB/s rd, 1007 KiB/s wr, 9 op/s 2024-09-13T14:33:41.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:40 smithi151 ceph-mon[108966]: osdmap e830: 8 total, 8 up, 8 in 2024-09-13T14:33:41.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:40 smithi151 ceph-mon[104450]: pgmap v861: 89 pgs: 89 active+clean; 234 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.0 MiB/s rd, 1007 KiB/s wr, 9 op/s 2024-09-13T14:33:41.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:40 smithi151 ceph-mon[104450]: osdmap e830: 8 total, 8 up, 8 in 2024-09-13T14:33:41.640 INFO:tasks.rados.rados.0.smithi151.stdout:3052: done (2 left) 2024-09-13T14:33:41.640 INFO:tasks.rados.rados.0.smithi151.stdout:3053: done (1 left) 2024-09-13T14:33:41.641 INFO:tasks.rados.rados.0.smithi151.stdout:3054: done (0 left) 2024-09-13T14:33:41.641 INFO:tasks.rados.rados.0.smithi151.stdout:3055: snap_create 2024-09-13T14:33:42.005 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:41 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:33:42.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:41 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:33:42.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:41 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:33:42.638 INFO:tasks.rados.rados.0.smithi151.stdout:3055: done (0 left) 2024-09-13T14:33:42.639 INFO:tasks.rados.rados.0.smithi151.stdout:3056: snap_create 2024-09-13T14:33:42.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:42 smithi151 ceph-mon[104450]: pgmap v864: 89 pgs: 89 active+clean; 232 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 420 KiB/s wr, 4 op/s 2024-09-13T14:33:42.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:42 smithi151 ceph-mon[104450]: osdmap e831: 8 total, 8 up, 8 in 2024-09-13T14:33:42.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:42 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:33:42.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:42 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:33:42.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:42 smithi151 ceph-mon[108966]: pgmap v864: 89 pgs: 89 active+clean; 232 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 420 KiB/s wr, 4 op/s 2024-09-13T14:33:42.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:42 smithi151 ceph-mon[108966]: osdmap e831: 8 total, 8 up, 8 in 2024-09-13T14:33:42.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:42 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:33:42.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:42 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:33:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:42 smithi184 ceph-mon[92527]: pgmap v864: 89 pgs: 89 active+clean; 232 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 420 KiB/s wr, 4 op/s 2024-09-13T14:33:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:42 smithi184 ceph-mon[92527]: osdmap e831: 8 total, 8 up, 8 in 2024-09-13T14:33:43.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:42 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:33:43.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:42 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:33:43.637 INFO:tasks.rados.rados.0.smithi151.stdout:3056: done (0 left) 2024-09-13T14:33:43.638 INFO:tasks.rados.rados.0.smithi151.stdout:3057: delete oid 36 current snap is 392 2024-09-13T14:33:43.641 INFO:tasks.rados.rados.0.smithi151.stdout:3057: done (0 left) 2024-09-13T14:33:43.641 INFO:tasks.rados.rados.0.smithi151.stdout:3058: read oid 20 snap -1 2024-09-13T14:33:43.641 INFO:tasks.rados.rados.0.smithi151.stdout:3058: expect (ObjNum 1122 snap 374 seq_num 1122) 2024-09-13T14:33:43.641 INFO:tasks.rados.rados.0.smithi151.stdout:3059: snap_remove snap 387 2024-09-13T14:33:43.793 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:43 smithi151 ceph-mon[104450]: osdmap e832: 8 total, 8 up, 8 in 2024-09-13T14:33:43.794 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:43 smithi151 ceph-mon[104450]: osdmap e833: 8 total, 8 up, 8 in 2024-09-13T14:33:43.794 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:43 smithi151 ceph-mon[108966]: osdmap e832: 8 total, 8 up, 8 in 2024-09-13T14:33:43.794 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:43 smithi151 ceph-mon[108966]: osdmap e833: 8 total, 8 up, 8 in 2024-09-13T14:33:43.991 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:43 smithi184 ceph-mon[92527]: osdmap e832: 8 total, 8 up, 8 in 2024-09-13T14:33:43.991 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:43 smithi184 ceph-mon[92527]: osdmap e833: 8 total, 8 up, 8 in 2024-09-13T14:33:44.602 INFO:tasks.rados.rados.0.smithi151.stdout:3059: done (1 left) 2024-09-13T14:33:44.602 INFO:tasks.rados.rados.0.smithi151.stdout:3060: snap_remove snap 390 2024-09-13T14:33:44.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:44 smithi151 ceph-mon[104450]: pgmap v867: 89 pgs: 89 active+clean; 242 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 1.5 MiB/s wr, 8 op/s 2024-09-13T14:33:44.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:44 smithi151 ceph-mon[104450]: osdmap e834: 8 total, 8 up, 8 in 2024-09-13T14:33:44.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:33:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:33:44] "GET /metrics HTTP/1.1" 200 36514 "" "Prometheus/2.51.0" 2024-09-13T14:33:44.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:44 smithi151 ceph-mon[108966]: pgmap v867: 89 pgs: 89 active+clean; 242 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 1.5 MiB/s wr, 8 op/s 2024-09-13T14:33:44.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:44 smithi151 ceph-mon[108966]: osdmap e834: 8 total, 8 up, 8 in 2024-09-13T14:33:44.963 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:44 smithi184 ceph-mon[92527]: pgmap v867: 89 pgs: 89 active+clean; 242 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 1.5 MiB/s wr, 8 op/s 2024-09-13T14:33:44.963 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:44 smithi184 ceph-mon[92527]: osdmap e834: 8 total, 8 up, 8 in 2024-09-13T14:33:45.604 INFO:tasks.rados.rados.0.smithi151.stdout:3060: done (1 left) 2024-09-13T14:33:45.604 INFO:tasks.rados.rados.0.smithi151.stdout:3061: read oid 48 snap -1 2024-09-13T14:33:45.604 INFO:tasks.rados.rados.0.smithi151.stdout:3061: expect (ObjNum 1111 snap 372 seq_num 1111) 2024-09-13T14:33:45.604 INFO:tasks.rados.rados.0.smithi151.stdout:3062: write oid 11 current snap is 392 2024-09-13T14:33:45.604 INFO:tasks.rados.rados.0.smithi151.stdout:3062: seq_num 1153 ranges {578471=731400,1787503=672058,3096593=281878} 2024-09-13T14:33:45.610 INFO:tasks.rados.rados.0.smithi151.stdout:3062: writing smithi15186934-11 from 578471 to 1309871 tid 1 2024-09-13T14:33:45.618 INFO:tasks.rados.rados.0.smithi151.stdout:3062: writing smithi15186934-11 from 1787503 to 2459561 tid 2 2024-09-13T14:33:45.623 INFO:tasks.rados.rados.0.smithi151.stdout:3062: writing smithi15186934-11 from 3096593 to 3378471 tid 3 2024-09-13T14:33:45.624 INFO:tasks.rados.rados.0.smithi151.stdout:3063: read oid 50 snap -1 2024-09-13T14:33:45.624 INFO:tasks.rados.rados.0.smithi151.stdout:3063: expect (ObjNum 1103 snap 370 seq_num 1103) 2024-09-13T14:33:45.624 INFO:tasks.rados.rados.0.smithi151.stdout:3064: delete oid 29 current snap is 392 2024-09-13T14:33:45.624 INFO:tasks.rados.rados.0.smithi151.stdout:3058: expect (ObjNum 1122 snap 374 seq_num 1122) 2024-09-13T14:33:45.780 INFO:tasks.rados.rados.0.smithi151.stdout:3061: expect (ObjNum 1111 snap 372 seq_num 1111) 2024-09-13T14:33:45.911 INFO:tasks.rados.rados.0.smithi151.stdout:3062: finishing write tid 1 to smithi15186934-11 2024-09-13T14:33:45.911 INFO:tasks.rados.rados.0.smithi151.stdout:3062: finishing write tid 2 to smithi15186934-11 2024-09-13T14:33:45.911 INFO:tasks.rados.rados.0.smithi151.stdout:3062: finishing write tid 3 to smithi15186934-11 2024-09-13T14:33:45.912 INFO:tasks.rados.rados.0.smithi151.stdout:3062: finishing write tid 4 to smithi15186934-11 2024-09-13T14:33:45.912 INFO:tasks.rados.rados.0.smithi151.stdout:3062: finishing write tid 5 to smithi15186934-11 2024-09-13T14:33:45.912 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 11 v 717 (ObjNum 1153 snap 392 seq_num 1153) dirty exists 2024-09-13T14:33:45.912 INFO:tasks.rados.rados.0.smithi151.stdout:3062: left oid 11 (ObjNum 1153 snap 392 seq_num 1153) 2024-09-13T14:33:45.912 INFO:tasks.rados.rados.0.smithi151.stdout:3063: expect (ObjNum 1103 snap 370 seq_num 1103) 2024-09-13T14:33:46.102 INFO:tasks.rados.rados.0.smithi151.stdout:3058: done (4 left) 2024-09-13T14:33:46.102 INFO:tasks.rados.rados.0.smithi151.stdout:3061: done (3 left) 2024-09-13T14:33:46.102 INFO:tasks.rados.rados.0.smithi151.stdout:3062: done (2 left) 2024-09-13T14:33:46.102 INFO:tasks.rados.rados.0.smithi151.stdout:3063: done (1 left) 2024-09-13T14:33:46.102 INFO:tasks.rados.rados.0.smithi151.stdout:3064: done (0 left) 2024-09-13T14:33:46.102 INFO:tasks.rados.rados.0.smithi151.stdout:3065: read oid 25 snap -1 2024-09-13T14:33:46.102 INFO:tasks.rados.rados.0.smithi151.stdout:3065: expect (ObjNum 1150 snap 389 seq_num 1150) 2024-09-13T14:33:46.102 INFO:tasks.rados.rados.0.smithi151.stdout:3066: snap_create 2024-09-13T14:33:46.109 INFO:tasks.rados.rados.0.smithi151.stdout:3065: expect (ObjNum 1150 snap 389 seq_num 1150) 2024-09-13T14:33:46.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:46 smithi184 ceph-mon[92527]: pgmap v870: 89 pgs: 89 active+clean; 242 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 1.1 MiB/s wr, 4 op/s 2024-09-13T14:33:46.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:33:46.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:33:46.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:46 smithi184 ceph-mon[92527]: osdmap e835: 8 total, 8 up, 8 in 2024-09-13T14:33:46.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:33:46.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:33:46.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:46.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:33:46.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:33:46.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:33:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:46 smithi151 ceph-mon[104450]: pgmap v870: 89 pgs: 89 active+clean; 242 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 1.1 MiB/s wr, 4 op/s 2024-09-13T14:33:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:33:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:33:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:46 smithi151 ceph-mon[104450]: osdmap e835: 8 total, 8 up, 8 in 2024-09-13T14:33:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:33:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:33:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:33:46.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:33:46.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:33:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:46 smithi151 ceph-mon[108966]: pgmap v870: 89 pgs: 89 active+clean; 242 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 1.1 MiB/s wr, 4 op/s 2024-09-13T14:33:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:33:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:33:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:46 smithi151 ceph-mon[108966]: osdmap e835: 8 total, 8 up, 8 in 2024-09-13T14:33:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:33:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:33:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:33:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:33:46.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:33:47.094 INFO:tasks.rados.rados.0.smithi151.stdout:3065: done (1 left) 2024-09-13T14:33:47.095 INFO:tasks.rados.rados.0.smithi151.stdout:3066: done (0 left) 2024-09-13T14:33:47.095 INFO:tasks.rados.rados.0.smithi151.stdout:3067: rollback oid 39 current snap is 393 2024-09-13T14:33:47.095 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 39 to 385 2024-09-13T14:33:47.095 INFO:tasks.rados.rados.0.smithi151.stdout:3068: delete oid 40 current snap is 393 2024-09-13T14:33:47.099 INFO:tasks.rados.rados.0.smithi151.stdout:3068: done (1 left) 2024-09-13T14:33:47.100 INFO:tasks.rados.rados.0.smithi151.stdout:3069: write oid 45 current snap is 393 2024-09-13T14:33:47.100 INFO:tasks.rados.rados.0.smithi151.stdout:3069: seq_num 1156 ranges {628892=673500,1962949=535738,3178862=250030} 2024-09-13T14:33:47.106 INFO:tasks.rados.rados.0.smithi151.stdout:3069: writing smithi15186934-45 from 628892 to 1302392 tid 1 2024-09-13T14:33:47.113 INFO:tasks.rados.rados.0.smithi151.stdout:3069: writing smithi15186934-45 from 1962949 to 2498687 tid 2 2024-09-13T14:33:47.119 INFO:tasks.rados.rados.0.smithi151.stdout:3069: writing smithi15186934-45 from 3178862 to 3428892 tid 3 2024-09-13T14:33:47.119 INFO:tasks.rados.rados.0.smithi151.stdout:3070: read oid 47 snap -1 2024-09-13T14:33:47.119 INFO:tasks.rados.rados.0.smithi151.stdout:3070: expect (ObjNum 1147 snap 385 seq_num 1147) 2024-09-13T14:33:47.119 INFO:tasks.rados.rados.0.smithi151.stdout:3071: read oid 36 snap -1 2024-09-13T14:33:47.119 INFO:tasks.rados.rados.0.smithi151.stdout:3071: expect deleted 2024-09-13T14:33:47.119 INFO:tasks.rados.rados.0.smithi151.stdout:3072: write oid 8 current snap is 393 2024-09-13T14:33:47.119 INFO:tasks.rados.rados.0.smithi151.stdout:3067: finishing rollback tid 0 to smithi15186934-39 2024-09-13T14:33:47.120 INFO:tasks.rados.rados.0.smithi151.stdout:3072: seq_num 1157 ranges {645699=748749,2021431=679396,3206285=239414} 2024-09-13T14:33:47.126 INFO:tasks.rados.rados.0.smithi151.stdout:3072: writing smithi15186934-8 from 645699 to 1394448 tid 1 2024-09-13T14:33:47.133 INFO:tasks.rados.rados.0.smithi151.stdout:3072: writing smithi15186934-8 from 2021431 to 2700827 tid 2 2024-09-13T14:33:47.138 INFO:tasks.rados.rados.0.smithi151.stdout:3072: writing smithi15186934-8 from 3206285 to 3445699 tid 3 2024-09-13T14:33:47.138 INFO:tasks.rados.rados.0.smithi151.stdout:3073: write oid 16 current snap is 393 2024-09-13T14:33:47.138 INFO:tasks.rados.rados.0.smithi151.stdout:3073: seq_num 1158 ranges {662506=423998,1563560=539407,2717355=490285} 2024-09-13T14:33:47.143 INFO:tasks.rados.rados.0.smithi151.stdout:3073: writing smithi15186934-16 from 662506 to 1086504 tid 1 2024-09-13T14:33:47.149 INFO:tasks.rados.rados.0.smithi151.stdout:3073: writing smithi15186934-16 from 1563560 to 2102967 tid 2 2024-09-13T14:33:47.155 INFO:tasks.rados.rados.0.smithi151.stdout:3073: writing smithi15186934-16 from 2717355 to 3207640 tid 3 2024-09-13T14:33:47.155 INFO:tasks.rados.rados.0.smithi151.stdout:3074: snap_remove snap 359 2024-09-13T14:33:48.104 INFO:tasks.rados.rados.0.smithi151.stdout:3074: done (6 left) 2024-09-13T14:33:48.104 INFO:tasks.rados.rados.0.smithi151.stdout:3075: snap_create 2024-09-13T14:33:48.104 INFO:tasks.rados.rados.0.smithi151.stdout:3067: finishing rollback tid 1 to smithi15186934-39 2024-09-13T14:33:48.104 INFO:tasks.rados.rados.0.smithi151.stdout:3067: finishing rollback tid 2 to smithi15186934-39 2024-09-13T14:33:48.104 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 1234 (ObjNum 1092 snap 367 seq_num 1092) dirty exists 2024-09-13T14:33:48.104 INFO:tasks.rados.rados.0.smithi151.stdout:3069: finishing write tid 1 to smithi15186934-45 2024-09-13T14:33:48.104 INFO:tasks.rados.rados.0.smithi151.stdout:3069: finishing write tid 2 to smithi15186934-45 2024-09-13T14:33:48.104 INFO:tasks.rados.rados.0.smithi151.stdout:3069: finishing write tid 3 to smithi15186934-45 2024-09-13T14:33:48.104 INFO:tasks.rados.rados.0.smithi151.stdout:3069: finishing write tid 4 to smithi15186934-45 2024-09-13T14:33:48.105 INFO:tasks.rados.rados.0.smithi151.stdout:3069: finishing write tid 5 to smithi15186934-45 2024-09-13T14:33:48.105 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 1472 (ObjNum 1156 snap 393 seq_num 1156) dirty exists 2024-09-13T14:33:48.105 INFO:tasks.rados.rados.0.smithi151.stdout:3069: left oid 45 (ObjNum 1156 snap 393 seq_num 1156) 2024-09-13T14:33:48.105 INFO:tasks.rados.rados.0.smithi151.stdout:3070: expect (ObjNum 1147 snap 385 seq_num 1147) 2024-09-13T14:33:48.250 INFO:tasks.rados.rados.0.smithi151.stdout:3072: finishing write tid 1 to smithi15186934-8 2024-09-13T14:33:48.250 INFO:tasks.rados.rados.0.smithi151.stdout:3072: finishing write tid 2 to smithi15186934-8 2024-09-13T14:33:48.250 INFO:tasks.rados.rados.0.smithi151.stdout:3072: finishing write tid 3 to smithi15186934-8 2024-09-13T14:33:48.250 INFO:tasks.rados.rados.0.smithi151.stdout:3072: finishing write tid 4 to smithi15186934-8 2024-09-13T14:33:48.250 INFO:tasks.rados.rados.0.smithi151.stdout:3072: finishing write tid 5 to smithi15186934-8 2024-09-13T14:33:48.251 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 911 (ObjNum 1157 snap 393 seq_num 1157) dirty exists 2024-09-13T14:33:48.251 INFO:tasks.rados.rados.0.smithi151.stdout:3072: left oid 8 (ObjNum 1157 snap 393 seq_num 1157) 2024-09-13T14:33:48.251 INFO:tasks.rados.rados.0.smithi151.stdout:3073: finishing write tid 1 to smithi15186934-16 2024-09-13T14:33:48.251 INFO:tasks.rados.rados.0.smithi151.stdout:3073: finishing write tid 2 to smithi15186934-16 2024-09-13T14:33:48.251 INFO:tasks.rados.rados.0.smithi151.stdout:3073: finishing write tid 3 to smithi15186934-16 2024-09-13T14:33:48.251 INFO:tasks.rados.rados.0.smithi151.stdout:3073: finishing write tid 4 to smithi15186934-16 2024-09-13T14:33:48.251 INFO:tasks.rados.rados.0.smithi151.stdout:3073: finishing write tid 5 to smithi15186934-16 2024-09-13T14:33:48.251 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 16 v 787 (ObjNum 1158 snap 393 seq_num 1158) dirty exists 2024-09-13T14:33:48.251 INFO:tasks.rados.rados.0.smithi151.stdout:3073: left oid 16 (ObjNum 1158 snap 393 seq_num 1158) 2024-09-13T14:33:48.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:48 smithi184 ceph-mon[92527]: osdmap e836: 8 total, 8 up, 8 in 2024-09-13T14:33:48.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:48 smithi184 ceph-mon[92527]: pgmap v873: 89 pgs: 89 active+clean; 244 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 460 KiB/s wr, 12 op/s 2024-09-13T14:33:48.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:48 smithi151 ceph-mon[108966]: osdmap e836: 8 total, 8 up, 8 in 2024-09-13T14:33:48.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:48 smithi151 ceph-mon[108966]: pgmap v873: 89 pgs: 89 active+clean; 244 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 460 KiB/s wr, 12 op/s 2024-09-13T14:33:48.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:48 smithi151 ceph-mon[104450]: osdmap e836: 8 total, 8 up, 8 in 2024-09-13T14:33:48.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:48 smithi151 ceph-mon[104450]: pgmap v873: 89 pgs: 89 active+clean; 244 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 460 KiB/s wr, 12 op/s 2024-09-13T14:33:49.100 INFO:tasks.rados.rados.0.smithi151.stdout:3067: done (6 left) 2024-09-13T14:33:49.100 INFO:tasks.rados.rados.0.smithi151.stdout:3069: done (5 left) 2024-09-13T14:33:49.100 INFO:tasks.rados.rados.0.smithi151.stdout:3070: done (4 left) 2024-09-13T14:33:49.101 INFO:tasks.rados.rados.0.smithi151.stdout:3071: done (3 left) 2024-09-13T14:33:49.101 INFO:tasks.rados.rados.0.smithi151.stdout:3072: done (2 left) 2024-09-13T14:33:49.101 INFO:tasks.rados.rados.0.smithi151.stdout:3073: done (1 left) 2024-09-13T14:33:49.101 INFO:tasks.rados.rados.0.smithi151.stdout:3075: done (0 left) 2024-09-13T14:33:49.101 INFO:tasks.rados.rados.0.smithi151.stdout:3076: write oid 7 current snap is 394 2024-09-13T14:33:49.101 INFO:tasks.rados.rados.0.smithi151.stdout:3076: seq_num 1159 ranges {679313=499247,1622042=683065,2744778=701557} 2024-09-13T14:33:49.106 INFO:tasks.rados.rados.0.smithi151.stdout:3076: writing smithi15186934-7 from 679313 to 1178560 tid 1 2024-09-13T14:33:49.113 INFO:tasks.rados.rados.0.smithi151.stdout:3076: writing smithi15186934-7 from 1622042 to 2305107 tid 2 2024-09-13T14:33:49.119 INFO:tasks.rados.rados.0.smithi151.stdout:3076: writing smithi15186934-7 from 2744778 to 3446335 tid 3 2024-09-13T14:33:49.120 INFO:tasks.rados.rados.0.smithi151.stdout:3077: read oid 21 snap -1 2024-09-13T14:33:49.120 INFO:tasks.rados.rados.0.smithi151.stdout:3077: expect (ObjNum 937 snap 323 seq_num 937) 2024-09-13T14:33:49.120 INFO:tasks.rados.rados.0.smithi151.stdout:3078: rollback oid 17 current snap is 394 2024-09-13T14:33:49.120 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 17 to 383 2024-09-13T14:33:49.120 INFO:tasks.rados.rados.0.smithi151.stdout:3076: finishing write tid 1 to smithi15186934-7 2024-09-13T14:33:49.120 INFO:tasks.rados.rados.0.smithi151.stdout:3076: finishing write tid 2 to smithi15186934-7 2024-09-13T14:33:49.120 INFO:tasks.rados.rados.0.smithi151.stdout:3079: read oid 44 snap -1 2024-09-13T14:33:49.120 INFO:tasks.rados.rados.0.smithi151.stdout:3079: expect (ObjNum 1125 snap 376 seq_num 1125) 2024-09-13T14:33:49.120 INFO:tasks.rados.rados.0.smithi151.stdout:3080: snap_remove snap 392 2024-09-13T14:33:49.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:49 smithi151 ceph-mon[104450]: osdmap e837: 8 total, 8 up, 8 in 2024-09-13T14:33:49.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:49 smithi151 ceph-mon[108966]: osdmap e837: 8 total, 8 up, 8 in 2024-09-13T14:33:49.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:49 smithi184 ceph-mon[92527]: osdmap e837: 8 total, 8 up, 8 in 2024-09-13T14:33:50.116 INFO:tasks.rados.rados.0.smithi151.stdout:3080: done (4 left) 2024-09-13T14:33:50.117 INFO:tasks.rados.rados.0.smithi151.stdout:3081: read oid 9 snap -1 2024-09-13T14:33:50.117 INFO:tasks.rados.rados.0.smithi151.stdout:3081: expect (ObjNum 1136 snap 381 seq_num 1136) 2024-09-13T14:33:50.117 INFO:tasks.rados.rados.0.smithi151.stdout:3082: rollback oid 6 current snap is 394 2024-09-13T14:33:50.117 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 6 to 385 2024-09-13T14:33:50.117 INFO:tasks.rados.rados.0.smithi151.stdout:3076: finishing write tid 3 to smithi15186934-7 2024-09-13T14:33:50.117 INFO:tasks.rados.rados.0.smithi151.stdout:3076: finishing write tid 4 to smithi15186934-7 2024-09-13T14:33:50.117 INFO:tasks.rados.rados.0.smithi151.stdout:3076: finishing write tid 5 to smithi15186934-7 2024-09-13T14:33:50.117 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 1802 (ObjNum 1159 snap 394 seq_num 1159) dirty exists 2024-09-13T14:33:50.117 INFO:tasks.rados.rados.0.smithi151.stdout:3076: left oid 7 (ObjNum 1159 snap 394 seq_num 1159) 2024-09-13T14:33:50.117 INFO:tasks.rados.rados.0.smithi151.stdout:3079: expect (ObjNum 1125 snap 376 seq_num 1125) 2024-09-13T14:33:50.246 INFO:tasks.rados.rados.0.smithi151.stdout:3078: finishing rollback tid 1 to smithi15186934-17 2024-09-13T14:33:50.246 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 0 (ObjNum 33 snap 0 seq_num 3064833400) dirty dne 2024-09-13T14:33:50.247 INFO:tasks.rados.rados.0.smithi151.stdout:3077: expect (ObjNum 937 snap 323 seq_num 937) 2024-09-13T14:33:50.369 INFO:tasks.rados.rados.0.smithi151.stdout:3082: finishing rollback tid 1 to smithi15186934-6 2024-09-13T14:33:50.369 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 6 v 0 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-13T14:33:50.370 INFO:tasks.rados.rados.0.smithi151.stdout:3081: expect (ObjNum 1136 snap 381 seq_num 1136) 2024-09-13T14:33:50.507 INFO:tasks.rados.rados.0.smithi151.stdout:3076: done (5 left) 2024-09-13T14:33:50.507 INFO:tasks.rados.rados.0.smithi151.stdout:3077: done (4 left) 2024-09-13T14:33:50.507 INFO:tasks.rados.rados.0.smithi151.stdout:3078: done (3 left) 2024-09-13T14:33:50.508 INFO:tasks.rados.rados.0.smithi151.stdout:3079: done (2 left) 2024-09-13T14:33:50.508 INFO:tasks.rados.rados.0.smithi151.stdout:3081: done (1 left) 2024-09-13T14:33:50.508 INFO:tasks.rados.rados.0.smithi151.stdout:3082: done (0 left) 2024-09-13T14:33:50.508 INFO:tasks.rados.rados.0.smithi151.stdout:3083: snap_create 2024-09-13T14:33:50.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:50 smithi184 ceph-mon[92527]: osdmap e838: 8 total, 8 up, 8 in 2024-09-13T14:33:50.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:50 smithi184 ceph-mon[92527]: pgmap v876: 89 pgs: 89 active+clean; 244 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 460 KiB/s wr, 12 op/s 2024-09-13T14:33:50.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:33:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:33:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:33:50.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:50 smithi151 ceph-mon[104450]: osdmap e838: 8 total, 8 up, 8 in 2024-09-13T14:33:50.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:50 smithi151 ceph-mon[104450]: pgmap v876: 89 pgs: 89 active+clean; 244 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 460 KiB/s wr, 12 op/s 2024-09-13T14:33:50.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:50 smithi151 ceph-mon[108966]: osdmap e838: 8 total, 8 up, 8 in 2024-09-13T14:33:50.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:50 smithi151 ceph-mon[108966]: pgmap v876: 89 pgs: 89 active+clean; 244 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 460 KiB/s wr, 12 op/s 2024-09-13T14:33:51.112 INFO:tasks.rados.rados.0.smithi151.stdout:3083: done (0 left) 2024-09-13T14:33:51.112 INFO:tasks.rados.rados.0.smithi151.stdout:3084: delete oid 11 current snap is 395 2024-09-13T14:33:51.117 INFO:tasks.rados.rados.0.smithi151.stdout:3084: done (0 left) 2024-09-13T14:33:51.117 INFO:tasks.rados.rados.0.smithi151.stdout:3085: snap_remove snap 369 2024-09-13T14:33:51.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:51 smithi184 ceph-mon[92527]: osdmap e839: 8 total, 8 up, 8 in 2024-09-13T14:33:51.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:51 smithi151 ceph-mon[104450]: osdmap e839: 8 total, 8 up, 8 in 2024-09-13T14:33:51.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:51 smithi151 ceph-mon[108966]: osdmap e839: 8 total, 8 up, 8 in 2024-09-13T14:33:52.127 INFO:tasks.rados.rados.0.smithi151.stdout:3085: done (0 left) 2024-09-13T14:33:52.127 INFO:tasks.rados.rados.0.smithi151.stdout:3086: rollback oid 30 current snap is 395 2024-09-13T14:33:52.127 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 30 to 385 2024-09-13T14:33:52.127 INFO:tasks.rados.rados.0.smithi151.stdout:3087: delete oid 35 current snap is 395 2024-09-13T14:33:52.128 INFO:tasks.rados.rados.0.smithi151.stdout:3086: finishing rollback tid 1 to smithi15186934-30 2024-09-13T14:33:52.128 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 30 v 0 (ObjNum 353 snap 0 seq_num 1755291184) dirty dne 2024-09-13T14:33:52.137 INFO:tasks.rados.rados.0.smithi151.stdout:3086: done (1 left) 2024-09-13T14:33:52.137 INFO:tasks.rados.rados.0.smithi151.stdout:3087: done (0 left) 2024-09-13T14:33:52.137 INFO:tasks.rados.rados.0.smithi151.stdout:3088: snap_create 2024-09-13T14:33:52.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:52 smithi184 ceph-mon[92527]: osdmap e840: 8 total, 8 up, 8 in 2024-09-13T14:33:52.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:52 smithi184 ceph-mon[92527]: pgmap v879: 89 pgs: 89 active+clean; 247 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.9 MiB/s rd, 819 KiB/s wr, 7 op/s 2024-09-13T14:33:52.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:52 smithi151 ceph-mon[104450]: osdmap e840: 8 total, 8 up, 8 in 2024-09-13T14:33:52.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:52 smithi151 ceph-mon[104450]: pgmap v879: 89 pgs: 89 active+clean; 247 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.9 MiB/s rd, 819 KiB/s wr, 7 op/s 2024-09-13T14:33:52.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:52 smithi151 ceph-mon[108966]: osdmap e840: 8 total, 8 up, 8 in 2024-09-13T14:33:52.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:52 smithi151 ceph-mon[108966]: pgmap v879: 89 pgs: 89 active+clean; 247 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.9 MiB/s rd, 819 KiB/s wr, 7 op/s 2024-09-13T14:33:53.133 INFO:tasks.rados.rados.0.smithi151.stdout:3088: done (0 left) 2024-09-13T14:33:53.134 INFO:tasks.rados.rados.0.smithi151.stdout:3089: read oid 34 snap -1 2024-09-13T14:33:53.134 INFO:tasks.rados.rados.0.smithi151.stdout:3089: expect deleted 2024-09-13T14:33:53.134 INFO:tasks.rados.rados.0.smithi151.stdout:3090: write oid 10 current snap is 396 2024-09-13T14:33:53.134 INFO:tasks.rados.rados.0.smithi151.stdout:3090: seq_num 1162 ranges {729734=724994,2197488=430392,3227047=302687} 2024-09-13T14:33:53.142 INFO:tasks.rados.rados.0.smithi151.stdout:3090: writing smithi15186934-10 from 729734 to 1454728 tid 1 2024-09-13T14:33:53.148 INFO:tasks.rados.rados.0.smithi151.stdout:3090: writing smithi15186934-10 from 2197488 to 2627880 tid 2 2024-09-13T14:33:53.154 INFO:tasks.rados.rados.0.smithi151.stdout:3090: writing smithi15186934-10 from 3227047 to 3529734 tid 3 2024-09-13T14:33:53.154 INFO:tasks.rados.rados.0.smithi151.stdout:3091: rollback oid 26 current snap is 396 2024-09-13T14:33:53.154 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 26 to 394 2024-09-13T14:33:53.154 INFO:tasks.rados.rados.0.smithi151.stdout:3092: read oid 12 snap -1 2024-09-13T14:33:53.154 INFO:tasks.rados.rados.0.smithi151.stdout:3092: expect deleted 2024-09-13T14:33:53.154 INFO:tasks.rados.rados.0.smithi151.stdout:3093: write oid 28 current snap is 396 2024-09-13T14:33:53.154 INFO:tasks.rados.rados.0.smithi151.stdout:3090: finishing write tid 1 to smithi15186934-10 2024-09-13T14:33:53.155 INFO:tasks.rados.rados.0.smithi151.stdout:3093: seq_num 1163 ranges {746541=400243,1739617=690403,3138117=408424} 2024-09-13T14:33:53.160 INFO:tasks.rados.rados.0.smithi151.stdout:3093: writing smithi15186934-28 from 746541 to 1146784 tid 1 2024-09-13T14:33:53.167 INFO:tasks.rados.rados.0.smithi151.stdout:3093: writing smithi15186934-28 from 1739617 to 2430020 tid 2 2024-09-13T14:33:53.174 INFO:tasks.rados.rados.0.smithi151.stdout:3093: writing smithi15186934-28 from 3138117 to 3546541 tid 3 2024-09-13T14:33:53.174 INFO:tasks.rados.rados.0.smithi151.stdout:3089: done (4 left) 2024-09-13T14:33:53.174 INFO:tasks.rados.rados.0.smithi151.stdout:3094: read oid 44 snap -1 2024-09-13T14:33:53.174 INFO:tasks.rados.rados.0.smithi151.stdout:3094: expect (ObjNum 1125 snap 376 seq_num 1125) 2024-09-13T14:33:53.174 INFO:tasks.rados.rados.0.smithi151.stdout:3095: write oid 9 current snap is 396 2024-09-13T14:33:53.174 INFO:tasks.rados.rados.0.smithi151.stdout:3090: finishing write tid 2 to smithi15186934-10 2024-09-13T14:33:53.174 INFO:tasks.rados.rados.0.smithi151.stdout:3095: seq_num 1164 ranges {763348=591845,1914452=434061,2881893=674270} 2024-09-13T14:33:53.181 INFO:tasks.rados.rados.0.smithi151.stdout:3095: writing smithi15186934-9 from 763348 to 1355193 tid 1 2024-09-13T14:33:53.186 INFO:tasks.rados.rados.0.smithi151.stdout:3095: writing smithi15186934-9 from 1914452 to 2348513 tid 2 2024-09-13T14:33:53.194 INFO:tasks.rados.rados.0.smithi151.stdout:3095: writing smithi15186934-9 from 2881893 to 3556163 tid 3 2024-09-13T14:33:53.194 INFO:tasks.rados.rados.0.smithi151.stdout:3096: read oid 18 snap 394 2024-09-13T14:33:53.194 INFO:tasks.rados.rados.0.smithi151.stdout:3096: expect (ObjNum 1131 snap 381 seq_num 1131) 2024-09-13T14:33:53.195 INFO:tasks.rados.rados.0.smithi151.stdout:3097: rollback oid 5 current snap is 396 2024-09-13T14:33:53.195 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 5 to 386 2024-09-13T14:33:53.195 INFO:tasks.rados.rados.0.smithi151.stdout:3090: finishing write tid 3 to smithi15186934-10 2024-09-13T14:33:53.195 INFO:tasks.rados.rados.0.smithi151.stdout:3090: finishing write tid 4 to smithi15186934-10 2024-09-13T14:33:53.195 INFO:tasks.rados.rados.0.smithi151.stdout:3090: finishing write tid 5 to smithi15186934-10 2024-09-13T14:33:53.195 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 1130 (ObjNum 1162 snap 396 seq_num 1162) dirty exists 2024-09-13T14:33:53.195 INFO:tasks.rados.rados.0.smithi151.stdout:3090: left oid 10 (ObjNum 1162 snap 396 seq_num 1162) 2024-09-13T14:33:53.195 INFO:tasks.rados.rados.0.smithi151.stdout:3091: finishing rollback tid 1 to smithi15186934-26 2024-09-13T14:33:53.195 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 1089 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-13T14:33:53.195 INFO:tasks.rados.rados.0.smithi151.stdout:3093: finishing write tid 1 to smithi15186934-28 2024-09-13T14:33:53.195 INFO:tasks.rados.rados.0.smithi151.stdout:3093: finishing write tid 2 to smithi15186934-28 2024-09-13T14:33:53.195 INFO:tasks.rados.rados.0.smithi151.stdout:3090: done (7 left) 2024-09-13T14:33:53.195 INFO:tasks.rados.rados.0.smithi151.stdout:3091: done (6 left) 2024-09-13T14:33:53.196 INFO:tasks.rados.rados.0.smithi151.stdout:3092: done (5 left) 2024-09-13T14:33:53.196 INFO:tasks.rados.rados.0.smithi151.stdout:3098: read oid 10 snap 394 2024-09-13T14:33:53.196 INFO:tasks.rados.rados.0.smithi151.stdout:3098: expect (ObjNum 980 snap 340 seq_num 980) 2024-09-13T14:33:53.196 INFO:tasks.rados.rados.0.smithi151.stdout:3099: write oid 50 current snap is 396 2024-09-13T14:33:53.196 INFO:tasks.rados.rados.0.smithi151.stdout:3093: finishing write tid 3 to smithi15186934-28 2024-09-13T14:33:53.196 INFO:tasks.rados.rados.0.smithi151.stdout:3093: finishing write tid 4 to smithi15186934-28 2024-09-13T14:33:53.196 INFO:tasks.rados.rados.0.smithi151.stdout:3093: finishing write tid 5 to smithi15186934-28 2024-09-13T14:33:53.196 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 28 v 1115 (ObjNum 1163 snap 396 seq_num 1163) dirty exists 2024-09-13T14:33:53.196 INFO:tasks.rados.rados.0.smithi151.stdout:3093: left oid 28 (ObjNum 1163 snap 396 seq_num 1163) 2024-09-13T14:33:53.196 INFO:tasks.rados.rados.0.smithi151.stdout:3094: expect (ObjNum 1125 snap 376 seq_num 1125) 2024-09-13T14:33:53.324 INFO:tasks.rados.rados.0.smithi151.stdout:3095: finishing write tid 1 to smithi15186934-9 2024-09-13T14:33:53.324 INFO:tasks.rados.rados.0.smithi151.stdout:3095: finishing write tid 2 to smithi15186934-9 2024-09-13T14:33:53.324 INFO:tasks.rados.rados.0.smithi151.stdout:3099: seq_num 1165 ranges {780155=667094,1972934=694072,3309316=270839} 2024-09-13T14:33:53.331 INFO:tasks.rados.rados.0.smithi151.stdout:3099: writing smithi15186934-50 from 780155 to 1447249 tid 1 2024-09-13T14:33:53.338 INFO:tasks.rados.rados.0.smithi151.stdout:3099: writing smithi15186934-50 from 1972934 to 2667006 tid 2 2024-09-13T14:33:53.346 INFO:tasks.rados.rados.0.smithi151.stdout:3099: writing smithi15186934-50 from 3309316 to 3580155 tid 3 2024-09-13T14:33:53.346 INFO:tasks.rados.rados.0.smithi151.stdout:3093: done (6 left) 2024-09-13T14:33:53.347 INFO:tasks.rados.rados.0.smithi151.stdout:3094: done (5 left) 2024-09-13T14:33:53.347 INFO:tasks.rados.rados.0.smithi151.stdout:3100: rollback oid 43 current snap is 396 2024-09-13T14:33:53.347 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 43 to 363 2024-09-13T14:33:53.347 INFO:tasks.rados.rados.0.smithi151.stdout:3097: finishing rollback tid 0 to smithi15186934-5 2024-09-13T14:33:53.347 INFO:tasks.rados.rados.0.smithi151.stdout:3097: finishing rollback tid 1 to smithi15186934-5 2024-09-13T14:33:53.347 INFO:tasks.rados.rados.0.smithi151.stdout:3097: finishing rollback tid 2 to smithi15186934-5 2024-09-13T14:33:53.347 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 1125 (ObjNum 1140 snap 383 seq_num 1140) dirty exists 2024-09-13T14:33:53.347 INFO:tasks.rados.rados.0.smithi151.stdout:3095: finishing write tid 3 to smithi15186934-9 2024-09-13T14:33:53.347 INFO:tasks.rados.rados.0.smithi151.stdout:3095: finishing write tid 4 to smithi15186934-9 2024-09-13T14:33:53.347 INFO:tasks.rados.rados.0.smithi151.stdout:3098: expect (ObjNum 980 snap 340 seq_num 980) 2024-09-13T14:33:53.472 INFO:tasks.rados.rados.0.smithi151.stdout:3095: finishing write tid 5 to smithi15186934-9 2024-09-13T14:33:53.472 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 1263 (ObjNum 1164 snap 396 seq_num 1164) dirty exists 2024-09-13T14:33:53.472 INFO:tasks.rados.rados.0.smithi151.stdout:3095: left oid 9 (ObjNum 1164 snap 396 seq_num 1164) 2024-09-13T14:33:53.472 INFO:tasks.rados.rados.0.smithi151.stdout:3096: expect (ObjNum 1131 snap 381 seq_num 1131) 2024-09-13T14:33:53.568 INFO:tasks.rados.rados.0.smithi151.stdout:3099: finishing write tid 1 to smithi15186934-50 2024-09-13T14:33:53.568 INFO:tasks.rados.rados.0.smithi151.stdout:3099: finishing write tid 2 to smithi15186934-50 2024-09-13T14:33:53.569 INFO:tasks.rados.rados.0.smithi151.stdout:3099: finishing write tid 3 to smithi15186934-50 2024-09-13T14:33:53.569 INFO:tasks.rados.rados.0.smithi151.stdout:3099: finishing write tid 4 to smithi15186934-50 2024-09-13T14:33:53.569 INFO:tasks.rados.rados.0.smithi151.stdout:3100: finishing rollback tid 1 to smithi15186934-43 2024-09-13T14:33:53.569 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 43 v 1750 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-13T14:33:53.569 INFO:tasks.rados.rados.0.smithi151.stdout:3099: finishing write tid 5 to smithi15186934-50 2024-09-13T14:33:53.569 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 50 v 1251 (ObjNum 1165 snap 396 seq_num 1165) dirty exists 2024-09-13T14:33:53.569 INFO:tasks.rados.rados.0.smithi151.stdout:3099: left oid 50 (ObjNum 1165 snap 396 seq_num 1165) 2024-09-13T14:33:53.569 INFO:tasks.rados.rados.0.smithi151.stdout:3095: done (5 left) 2024-09-13T14:33:53.570 INFO:tasks.rados.rados.0.smithi151.stdout:3096: done (4 left) 2024-09-13T14:33:53.570 INFO:tasks.rados.rados.0.smithi151.stdout:3097: done (3 left) 2024-09-13T14:33:53.570 INFO:tasks.rados.rados.0.smithi151.stdout:3098: done (2 left) 2024-09-13T14:33:53.570 INFO:tasks.rados.rados.0.smithi151.stdout:3099: done (1 left) 2024-09-13T14:33:53.570 INFO:tasks.rados.rados.0.smithi151.stdout:3100: done (0 left) 2024-09-13T14:33:53.570 INFO:tasks.rados.rados.0.smithi151.stdout:3101: snap_remove snap 389 2024-09-13T14:33:53.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:53 smithi184 ceph-mon[92527]: osdmap e841: 8 total, 8 up, 8 in 2024-09-13T14:33:53.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:53 smithi151 ceph-mon[108966]: osdmap e841: 8 total, 8 up, 8 in 2024-09-13T14:33:53.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:53 smithi151 ceph-mon[104450]: osdmap e841: 8 total, 8 up, 8 in 2024-09-13T14:33:54.143 INFO:tasks.rados.rados.0.smithi151.stdout:3101: done (0 left) 2024-09-13T14:33:54.143 INFO:tasks.rados.rados.0.smithi151.stdout:3102: write oid 27 current snap is 396 2024-09-13T14:33:54.143 INFO:tasks.rados.rados.0.smithi151.stdout:3102: seq_num 1166 ranges {796962=742343,2031416=437730,2936739=413167} 2024-09-13T14:33:54.152 INFO:tasks.rados.rados.0.smithi151.stdout:3102: writing smithi15186934-27 from 796962 to 1539305 tid 1 2024-09-13T14:33:54.157 INFO:tasks.rados.rados.0.smithi151.stdout:3102: writing smithi15186934-27 from 2031416 to 2469146 tid 2 2024-09-13T14:33:54.162 INFO:tasks.rados.rados.0.smithi151.stdout:3102: writing smithi15186934-27 from 2936739 to 3349906 tid 3 2024-09-13T14:33:54.162 INFO:tasks.rados.rados.0.smithi151.stdout:3103: write oid 11 current snap is 396 2024-09-13T14:33:54.162 INFO:tasks.rados.rados.0.smithi151.stdout:3102: finishing write tid 1 to smithi15186934-27 2024-09-13T14:33:54.162 INFO:tasks.rados.rados.0.smithi151.stdout:3103: seq_num 1167 ranges {413769=417592,1573545=697741,2847809=624439} 2024-09-13T14:33:54.166 INFO:tasks.rados.rados.0.smithi151.stdout:3103: writing smithi15186934-11 from 413769 to 831361 tid 1 2024-09-13T14:33:54.175 INFO:tasks.rados.rados.0.smithi151.stdout:3103: writing smithi15186934-11 from 1573545 to 2271286 tid 2 2024-09-13T14:33:54.183 INFO:tasks.rados.rados.0.smithi151.stdout:3103: writing smithi15186934-11 from 2847809 to 3472248 tid 3 2024-09-13T14:33:54.183 INFO:tasks.rados.rados.0.smithi151.stdout:3104: read oid 10 snap -1 2024-09-13T14:33:54.183 INFO:tasks.rados.rados.0.smithi151.stdout:3104: expect (ObjNum 1162 snap 396 seq_num 1162) 2024-09-13T14:33:54.183 INFO:tasks.rados.rados.0.smithi151.stdout:3105: rollback oid 37 current snap is 396 2024-09-13T14:33:54.183 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 37 to 388 2024-09-13T14:33:54.183 INFO:tasks.rados.rados.0.smithi151.stdout:3102: finishing write tid 2 to smithi15186934-27 2024-09-13T14:33:54.183 INFO:tasks.rados.rados.0.smithi151.stdout:3102: finishing write tid 3 to smithi15186934-27 2024-09-13T14:33:54.183 INFO:tasks.rados.rados.0.smithi151.stdout:3102: finishing write tid 4 to smithi15186934-27 2024-09-13T14:33:54.183 INFO:tasks.rados.rados.0.smithi151.stdout:3102: finishing write tid 5 to smithi15186934-27 2024-09-13T14:33:54.184 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 27 v 804 (ObjNum 1166 snap 396 seq_num 1166) dirty exists 2024-09-13T14:33:54.184 INFO:tasks.rados.rados.0.smithi151.stdout:3102: left oid 27 (ObjNum 1166 snap 396 seq_num 1166) 2024-09-13T14:33:54.184 INFO:tasks.rados.rados.0.smithi151.stdout:3103: finishing write tid 1 to smithi15186934-11 2024-09-13T14:33:54.184 INFO:tasks.rados.rados.0.smithi151.stdout:3103: finishing write tid 2 to smithi15186934-11 2024-09-13T14:33:54.184 INFO:tasks.rados.rados.0.smithi151.stdout:3102: done (3 left) 2024-09-13T14:33:54.184 INFO:tasks.rados.rados.0.smithi151.stdout:3106: write oid 15 current snap is 396 2024-09-13T14:33:54.184 INFO:tasks.rados.rados.0.smithi151.stdout:3106: seq_num 1168 ranges {430576=492841,1632027=441399,2475232=435711,3365397=265179} 2024-09-13T14:33:54.188 INFO:tasks.rados.rados.0.smithi151.stdout:3106: writing smithi15186934-15 from 430576 to 923417 tid 1 2024-09-13T14:33:54.194 INFO:tasks.rados.rados.0.smithi151.stdout:3106: writing smithi15186934-15 from 1632027 to 2073426 tid 2 2024-09-13T14:33:54.199 INFO:tasks.rados.rados.0.smithi151.stdout:3106: writing smithi15186934-15 from 2475232 to 2910943 tid 3 2024-09-13T14:33:54.203 INFO:tasks.rados.rados.0.smithi151.stdout:3106: writing smithi15186934-15 from 3365397 to 3630576 tid 4 2024-09-13T14:33:54.203 INFO:tasks.rados.rados.0.smithi151.stdout:3107: write oid 14 current snap is 396 2024-09-13T14:33:54.203 INFO:tasks.rados.rados.0.smithi151.stdout:3107: seq_num 1169 ranges {447383=568090,1690509=701410,2902655=646983} 2024-09-13T14:33:54.209 INFO:tasks.rados.rados.0.smithi151.stdout:3107: writing smithi15186934-14 from 447383 to 1015473 tid 1 2024-09-13T14:33:54.216 INFO:tasks.rados.rados.0.smithi151.stdout:3107: writing smithi15186934-14 from 1690509 to 2391919 tid 2 2024-09-13T14:33:54.223 INFO:tasks.rados.rados.0.smithi151.stdout:3107: writing smithi15186934-14 from 2902655 to 3549638 tid 3 2024-09-13T14:33:54.224 INFO:tasks.rados.rados.0.smithi151.stdout:3108: read oid 26 snap -1 2024-09-13T14:33:54.224 INFO:tasks.rados.rados.0.smithi151.stdout:3108: expect deleted 2024-09-13T14:33:54.224 INFO:tasks.rados.rados.0.smithi151.stdout:3109: rollback oid 39 current snap is 396 2024-09-13T14:33:54.224 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 39 to 388 2024-09-13T14:33:54.224 INFO:tasks.rados.rados.0.smithi151.stdout:3103: finishing write tid 3 to smithi15186934-11 2024-09-13T14:33:54.224 INFO:tasks.rados.rados.0.smithi151.stdout:3103: finishing write tid 4 to smithi15186934-11 2024-09-13T14:33:54.224 INFO:tasks.rados.rados.0.smithi151.stdout:3103: finishing write tid 5 to smithi15186934-11 2024-09-13T14:33:54.224 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 11 v 735 (ObjNum 1167 snap 396 seq_num 1167) dirty exists 2024-09-13T14:33:54.224 INFO:tasks.rados.rados.0.smithi151.stdout:3103: left oid 11 (ObjNum 1167 snap 396 seq_num 1167) 2024-09-13T14:33:54.224 INFO:tasks.rados.rados.0.smithi151.stdout:3105: finishing rollback tid 0 to smithi15186934-37 2024-09-13T14:33:54.225 INFO:tasks.rados.rados.0.smithi151.stdout:3105: finishing rollback tid 1 to smithi15186934-37 2024-09-13T14:33:54.225 INFO:tasks.rados.rados.0.smithi151.stdout:3105: finishing rollback tid 2 to smithi15186934-37 2024-09-13T14:33:54.225 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 37 v 1359 (ObjNum 1129 snap 378 seq_num 1129) dirty exists 2024-09-13T14:33:54.225 INFO:tasks.rados.rados.0.smithi151.stdout:3104: expect (ObjNum 1162 snap 396 seq_num 1162) 2024-09-13T14:33:54.426 INFO:tasks.rados.rados.0.smithi151.stdout:3106: finishing write tid 1 to smithi15186934-15 2024-09-13T14:33:54.426 INFO:tasks.rados.rados.0.smithi151.stdout:3106: finishing write tid 2 to smithi15186934-15 2024-09-13T14:33:54.427 INFO:tasks.rados.rados.0.smithi151.stdout:3106: finishing write tid 3 to smithi15186934-15 2024-09-13T14:33:54.427 INFO:tasks.rados.rados.0.smithi151.stdout:3106: finishing write tid 4 to smithi15186934-15 2024-09-13T14:33:54.427 INFO:tasks.rados.rados.0.smithi151.stdout:3106: finishing write tid 5 to smithi15186934-15 2024-09-13T14:33:54.427 INFO:tasks.rados.rados.0.smithi151.stdout:3103: done (6 left) 2024-09-13T14:33:54.427 INFO:tasks.rados.rados.0.smithi151.stdout:3104: done (5 left) 2024-09-13T14:33:54.427 INFO:tasks.rados.rados.0.smithi151.stdout:3105: done (4 left) 2024-09-13T14:33:54.428 INFO:tasks.rados.rados.0.smithi151.stdout:3110: rollback oid 44 current snap is 396 2024-09-13T14:33:54.428 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 44 to 395 2024-09-13T14:33:54.428 INFO:tasks.rados.rados.0.smithi151.stdout:3106: finishing write tid 6 to smithi15186934-15 2024-09-13T14:33:54.428 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 15 v 1122 (ObjNum 1168 snap 396 seq_num 1168) dirty exists 2024-09-13T14:33:54.428 INFO:tasks.rados.rados.0.smithi151.stdout:3106: left oid 15 (ObjNum 1168 snap 396 seq_num 1168) 2024-09-13T14:33:54.429 INFO:tasks.rados.rados.0.smithi151.stdout:3107: finishing write tid 1 to smithi15186934-14 2024-09-13T14:33:54.429 INFO:tasks.rados.rados.0.smithi151.stdout:3107: finishing write tid 2 to smithi15186934-14 2024-09-13T14:33:54.429 INFO:tasks.rados.rados.0.smithi151.stdout:3109: finishing rollback tid 0 to smithi15186934-39 2024-09-13T14:33:54.429 INFO:tasks.rados.rados.0.smithi151.stdout:3109: finishing rollback tid 1 to smithi15186934-39 2024-09-13T14:33:54.429 INFO:tasks.rados.rados.0.smithi151.stdout:3107: finishing write tid 3 to smithi15186934-14 2024-09-13T14:33:54.429 INFO:tasks.rados.rados.0.smithi151.stdout:3109: finishing rollback tid 2 to smithi15186934-39 2024-09-13T14:33:54.430 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 1259 (ObjNum 1092 snap 367 seq_num 1092) dirty exists 2024-09-13T14:33:54.430 INFO:tasks.rados.rados.0.smithi151.stdout:3107: finishing write tid 4 to smithi15186934-14 2024-09-13T14:33:54.430 INFO:tasks.rados.rados.0.smithi151.stdout:3107: finishing write tid 5 to smithi15186934-14 2024-09-13T14:33:54.430 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 1270 (ObjNum 1169 snap 396 seq_num 1169) dirty exists 2024-09-13T14:33:54.430 INFO:tasks.rados.rados.0.smithi151.stdout:3107: left oid 14 (ObjNum 1169 snap 396 seq_num 1169) 2024-09-13T14:33:54.431 INFO:tasks.rados.rados.0.smithi151.stdout:3106: done (4 left) 2024-09-13T14:33:54.431 INFO:tasks.rados.rados.0.smithi151.stdout:3107: done (3 left) 2024-09-13T14:33:54.431 INFO:tasks.rados.rados.0.smithi151.stdout:3108: done (2 left) 2024-09-13T14:33:54.431 INFO:tasks.rados.rados.0.smithi151.stdout:3109: done (1 left) 2024-09-13T14:33:54.431 INFO:tasks.rados.rados.0.smithi151.stdout:3111: snap_create 2024-09-13T14:33:54.432 INFO:tasks.rados.rados.0.smithi151.stdout:3110: finishing rollback tid 0 to smithi15186934-44 2024-09-13T14:33:54.432 INFO:tasks.rados.rados.0.smithi151.stdout:3110: finishing rollback tid 1 to smithi15186934-44 2024-09-13T14:33:54.433 INFO:tasks.rados.rados.0.smithi151.stdout:3110: finishing rollback tid 2 to smithi15186934-44 2024-09-13T14:33:54.433 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 925 (ObjNum 1125 snap 376 seq_num 1125) dirty exists 2024-09-13T14:33:54.589 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:54 smithi151 ceph-mon[104450]: osdmap e842: 8 total, 8 up, 8 in 2024-09-13T14:33:54.589 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:54 smithi151 ceph-mon[104450]: pgmap v882: 89 pgs: 89 active+clean; 249 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 7.4 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-13T14:33:54.589 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:54 smithi151 ceph-mon[108966]: osdmap e842: 8 total, 8 up, 8 in 2024-09-13T14:33:54.589 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:54 smithi151 ceph-mon[108966]: pgmap v882: 89 pgs: 89 active+clean; 249 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 7.4 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-13T14:33:54.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:54 smithi184 ceph-mon[92527]: osdmap e842: 8 total, 8 up, 8 in 2024-09-13T14:33:54.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:54 smithi184 ceph-mon[92527]: pgmap v882: 89 pgs: 89 active+clean; 249 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 7.4 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-13T14:33:54.610 INFO:tasks.rados.rados.0.smithi151.stdout:3110: done (1 left) 2024-09-13T14:33:54.610 INFO:tasks.rados.rados.0.smithi151.stdout:3111: done (0 left) 2024-09-13T14:33:54.610 INFO:tasks.rados.rados.0.smithi151.stdout:3112: snap_remove snap 385 2024-09-13T14:33:54.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:33:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:33:54] "GET /metrics HTTP/1.1" 200 36510 "" "Prometheus/2.51.0" 2024-09-13T14:33:55.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:55 smithi184 ceph-mon[92527]: osdmap e843: 8 total, 8 up, 8 in 2024-09-13T14:33:55.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:55 smithi184 ceph-mon[92527]: osdmap e844: 8 total, 8 up, 8 in 2024-09-13T14:33:55.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:55 smithi151 ceph-mon[104450]: osdmap e843: 8 total, 8 up, 8 in 2024-09-13T14:33:55.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:55 smithi151 ceph-mon[104450]: osdmap e844: 8 total, 8 up, 8 in 2024-09-13T14:33:55.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:55 smithi151 ceph-mon[108966]: osdmap e843: 8 total, 8 up, 8 in 2024-09-13T14:33:55.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:55 smithi151 ceph-mon[108966]: osdmap e844: 8 total, 8 up, 8 in 2024-09-13T14:33:55.613 INFO:tasks.rados.rados.0.smithi151.stdout:3112: done (0 left) 2024-09-13T14:33:55.613 INFO:tasks.rados.rados.0.smithi151.stdout:3113: read oid 35 snap -1 2024-09-13T14:33:55.613 INFO:tasks.rados.rados.0.smithi151.stdout:3113: expect deleted 2024-09-13T14:33:55.613 INFO:tasks.rados.rados.0.smithi151.stdout:3114: read oid 15 snap 370 2024-09-13T14:33:55.613 INFO:tasks.rados.rados.0.smithi151.stdout:3114: expect (ObjNum 993 snap 342 seq_num 993) 2024-09-13T14:33:55.613 INFO:tasks.rados.rados.0.smithi151.stdout:3115: rollback oid 13 current snap is 397 2024-09-13T14:33:55.613 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 13 to 384 2024-09-13T14:33:55.614 INFO:tasks.rados.rados.0.smithi151.stdout:3116: rollback oid 37 current snap is 397 2024-09-13T14:33:55.614 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 37 to 394 2024-09-13T14:33:55.614 INFO:tasks.rados.rados.0.smithi151.stdout:3117: read oid 46 snap -1 2024-09-13T14:33:55.614 INFO:tasks.rados.rados.0.smithi151.stdout:3117: expect (ObjNum 1135 snap 381 seq_num 1135) 2024-09-13T14:33:55.614 INFO:tasks.rados.rados.0.smithi151.stdout:3118: rollback oid 8 current snap is 397 2024-09-13T14:33:55.614 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 8 to 383 2024-09-13T14:33:55.615 INFO:tasks.rados.rados.0.smithi151.stdout:3119: write oid 12 current snap is 397 2024-09-13T14:33:55.615 INFO:tasks.rados.rados.0.smithi151.stdout:3119: seq_num 1170 ranges {464190=643339,1748991=445068,2930078=574608} 2024-09-13T14:33:55.621 INFO:tasks.rados.rados.0.smithi151.stdout:3119: writing smithi15186934-12 from 464190 to 1107529 tid 1 2024-09-13T14:33:55.628 INFO:tasks.rados.rados.0.smithi151.stdout:3119: writing smithi15186934-12 from 1748991 to 2194059 tid 2 2024-09-13T14:33:55.637 INFO:tasks.rados.rados.0.smithi151.stdout:3119: writing smithi15186934-12 from 2930078 to 3504686 tid 3 2024-09-13T14:33:55.637 INFO:tasks.rados.rados.0.smithi151.stdout:3120: delete oid 3 current snap is 397 2024-09-13T14:33:55.637 INFO:tasks.rados.rados.0.smithi151.stdout:3116: finishing rollback tid 0 to smithi15186934-37 2024-09-13T14:33:55.637 INFO:tasks.rados.rados.0.smithi151.stdout:3118: finishing rollback tid 0 to smithi15186934-8 2024-09-13T14:33:55.637 INFO:tasks.rados.rados.0.smithi151.stdout:3118: finishing rollback tid 1 to smithi15186934-8 2024-09-13T14:33:55.637 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 928 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-13T14:33:55.637 INFO:tasks.rados.rados.0.smithi151.stdout:3115: finishing rollback tid 0 to smithi15186934-13 2024-09-13T14:33:55.637 INFO:tasks.rados.rados.0.smithi151.stdout:3115: finishing rollback tid 1 to smithi15186934-13 2024-09-13T14:33:55.637 INFO:tasks.rados.rados.0.smithi151.stdout:3116: finishing rollback tid 1 to smithi15186934-37 2024-09-13T14:33:55.637 INFO:tasks.rados.rados.0.smithi151.stdout:3116: finishing rollback tid 2 to smithi15186934-37 2024-09-13T14:33:55.638 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 37 v 1365 (ObjNum 1129 snap 378 seq_num 1129) dirty exists 2024-09-13T14:33:55.638 INFO:tasks.rados.rados.0.smithi151.stdout:3115: finishing rollback tid 2 to smithi15186934-13 2024-09-13T14:33:55.638 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 13 v 1131 (ObjNum 1043 snap 355 seq_num 1043) dirty exists 2024-09-13T14:33:55.638 INFO:tasks.rados.rados.0.smithi151.stdout:3119: finishing write tid 1 to smithi15186934-12 2024-09-13T14:33:55.638 INFO:tasks.rados.rados.0.smithi151.stdout:3117: expect (ObjNum 1135 snap 381 seq_num 1135) 2024-09-13T14:33:55.772 INFO:tasks.rados.rados.0.smithi151.stdout:3119: finishing write tid 2 to smithi15186934-12 2024-09-13T14:33:55.772 INFO:tasks.rados.rados.0.smithi151.stdout:3114: expect (ObjNum 993 snap 342 seq_num 993) 2024-09-13T14:33:55.858 INFO:tasks.rados.rados.0.smithi151.stdout:3119: finishing write tid 3 to smithi15186934-12 2024-09-13T14:33:55.859 INFO:tasks.rados.rados.0.smithi151.stdout:3119: finishing write tid 4 to smithi15186934-12 2024-09-13T14:33:55.859 INFO:tasks.rados.rados.0.smithi151.stdout:3119: finishing write tid 5 to smithi15186934-12 2024-09-13T14:33:55.859 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 12 v 668 (ObjNum 1170 snap 397 seq_num 1170) dirty exists 2024-09-13T14:33:55.859 INFO:tasks.rados.rados.0.smithi151.stdout:3119: left oid 12 (ObjNum 1170 snap 397 seq_num 1170) 2024-09-13T14:33:55.859 INFO:tasks.rados.rados.0.smithi151.stdout:3113: done (7 left) 2024-09-13T14:33:55.859 INFO:tasks.rados.rados.0.smithi151.stdout:3114: done (6 left) 2024-09-13T14:33:55.859 INFO:tasks.rados.rados.0.smithi151.stdout:3115: done (5 left) 2024-09-13T14:33:55.860 INFO:tasks.rados.rados.0.smithi151.stdout:3116: done (4 left) 2024-09-13T14:33:55.860 INFO:tasks.rados.rados.0.smithi151.stdout:3117: done (3 left) 2024-09-13T14:33:55.860 INFO:tasks.rados.rados.0.smithi151.stdout:3118: done (2 left) 2024-09-13T14:33:55.860 INFO:tasks.rados.rados.0.smithi151.stdout:3119: done (1 left) 2024-09-13T14:33:55.860 INFO:tasks.rados.rados.0.smithi151.stdout:3120: done (0 left) 2024-09-13T14:33:55.860 INFO:tasks.rados.rados.0.smithi151.stdout:3121: snap_remove snap 358 2024-09-13T14:33:56.616 INFO:tasks.rados.rados.0.smithi151.stdout:3121: done (0 left) 2024-09-13T14:33:56.616 INFO:tasks.rados.rados.0.smithi151.stdout:3122: read oid 42 snap 386 2024-09-13T14:33:56.616 INFO:tasks.rados.rados.0.smithi151.stdout:3122: expect deleted 2024-09-13T14:33:56.616 INFO:tasks.rados.rados.0.smithi151.stdout:3123: write oid 14 current snap is 397 2024-09-13T14:33:56.616 INFO:tasks.rados.rados.0.smithi151.stdout:3123: seq_num 1172 ranges {497804=510190,1465955=448737,2584924=597152} 2024-09-13T14:33:56.622 INFO:tasks.rados.rados.0.smithi151.stdout:3123: writing smithi15186934-14 from 497804 to 1007994 tid 1 2024-09-13T14:33:56.628 INFO:tasks.rados.rados.0.smithi151.stdout:3123: writing smithi15186934-14 from 1465955 to 1914692 tid 2 2024-09-13T14:33:56.636 INFO:tasks.rados.rados.0.smithi151.stdout:3123: writing smithi15186934-14 from 2584924 to 3182076 tid 3 2024-09-13T14:33:56.636 INFO:tasks.rados.rados.0.smithi151.stdout:3124: rollback oid 5 current snap is 397 2024-09-13T14:33:56.636 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 5 to 376 2024-09-13T14:33:56.636 INFO:tasks.rados.rados.0.smithi151.stdout:3123: finishing write tid 1 to smithi15186934-14 2024-09-13T14:33:56.636 INFO:tasks.rados.rados.0.smithi151.stdout:3123: finishing write tid 2 to smithi15186934-14 2024-09-13T14:33:56.636 INFO:tasks.rados.rados.0.smithi151.stdout:3122: done (2 left) 2024-09-13T14:33:56.636 INFO:tasks.rados.rados.0.smithi151.stdout:3125: snap_create 2024-09-13T14:33:56.638 INFO:tasks.rados.rados.0.smithi151.stdout:3124: finishing rollback tid 0 to smithi15186934-5 2024-09-13T14:33:56.638 INFO:tasks.rados.rados.0.smithi151.stdout:3124: finishing rollback tid 1 to smithi15186934-5 2024-09-13T14:33:56.638 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 1140 (ObjNum 81 snap 0 seq_num 3681307150) dirty dne 2024-09-13T14:33:56.639 INFO:tasks.rados.rados.0.smithi151.stdout:3123: finishing write tid 3 to smithi15186934-14 2024-09-13T14:33:56.639 INFO:tasks.rados.rados.0.smithi151.stdout:3123: finishing write tid 4 to smithi15186934-14 2024-09-13T14:33:56.640 INFO:tasks.rados.rados.0.smithi151.stdout:3123: finishing write tid 5 to smithi15186934-14 2024-09-13T14:33:56.640 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 1285 (ObjNum 1172 snap 397 seq_num 1172) dirty exists 2024-09-13T14:33:56.640 INFO:tasks.rados.rados.0.smithi151.stdout:3123: left oid 14 (ObjNum 1172 snap 397 seq_num 1172) 2024-09-13T14:33:57.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:56 smithi184 ceph-mon[92527]: pgmap v885: 89 pgs: 89 active+clean; 252 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.6 MiB/s rd, 1.4 MiB/s wr, 10 op/s 2024-09-13T14:33:57.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:56 smithi184 ceph-mon[92527]: osdmap e845: 8 total, 8 up, 8 in 2024-09-13T14:33:57.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:56 smithi151 ceph-mon[104450]: pgmap v885: 89 pgs: 89 active+clean; 252 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.6 MiB/s rd, 1.4 MiB/s wr, 10 op/s 2024-09-13T14:33:57.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:56 smithi151 ceph-mon[104450]: osdmap e845: 8 total, 8 up, 8 in 2024-09-13T14:33:57.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:56 smithi151 ceph-mon[108966]: pgmap v885: 89 pgs: 89 active+clean; 252 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.6 MiB/s rd, 1.4 MiB/s wr, 10 op/s 2024-09-13T14:33:57.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:56 smithi151 ceph-mon[108966]: osdmap e845: 8 total, 8 up, 8 in 2024-09-13T14:33:57.633 INFO:tasks.rados.rados.0.smithi151.stdout:3123: done (2 left) 2024-09-13T14:33:57.633 INFO:tasks.rados.rados.0.smithi151.stdout:3124: done (1 left) 2024-09-13T14:33:57.633 INFO:tasks.rados.rados.0.smithi151.stdout:3125: done (0 left) 2024-09-13T14:33:57.633 INFO:tasks.rados.rados.0.smithi151.stdout:3126: read oid 8 snap -1 2024-09-13T14:33:57.633 INFO:tasks.rados.rados.0.smithi151.stdout:3126: expect deleted 2024-09-13T14:33:57.634 INFO:tasks.rados.rados.0.smithi151.stdout:3127: write oid 31 current snap is 398 2024-09-13T14:33:57.634 INFO:tasks.rados.rados.0.smithi151.stdout:3127: seq_num 1173 ranges {514611=585439,1524437=592395,2895994=408424} 2024-09-13T14:33:57.639 INFO:tasks.rados.rados.0.smithi151.stdout:3127: writing smithi15186934-31 from 514611 to 1100050 tid 1 2024-09-13T14:33:57.645 INFO:tasks.rados.rados.0.smithi151.stdout:3127: writing smithi15186934-31 from 1524437 to 2116832 tid 2 2024-09-13T14:33:57.652 INFO:tasks.rados.rados.0.smithi151.stdout:3127: writing smithi15186934-31 from 2895994 to 3304418 tid 3 2024-09-13T14:33:57.652 INFO:tasks.rados.rados.0.smithi151.stdout:3128: snap_create 2024-09-13T14:33:57.652 INFO:tasks.rados.rados.0.smithi151.stdout:3127: finishing write tid 1 to smithi15186934-31 2024-09-13T14:33:57.652 INFO:tasks.rados.rados.0.smithi151.stdout:3127: finishing write tid 2 to smithi15186934-31 2024-09-13T14:33:57.655 INFO:tasks.rados.rados.0.smithi151.stdout:3127: finishing write tid 3 to smithi15186934-31 2024-09-13T14:33:57.655 INFO:tasks.rados.rados.0.smithi151.stdout:3127: finishing write tid 4 to smithi15186934-31 2024-09-13T14:33:57.655 INFO:tasks.rados.rados.0.smithi151.stdout:3127: finishing write tid 5 to smithi15186934-31 2024-09-13T14:33:57.655 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 31 v 1134 (ObjNum 1173 snap 398 seq_num 1173) dirty exists 2024-09-13T14:33:57.655 INFO:tasks.rados.rados.0.smithi151.stdout:3127: left oid 31 (ObjNum 1173 snap 398 seq_num 1173) 2024-09-13T14:33:58.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:57 smithi184 ceph-mon[92527]: osdmap e846: 8 total, 8 up, 8 in 2024-09-13T14:33:58.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:57 smithi151 ceph-mon[104450]: osdmap e846: 8 total, 8 up, 8 in 2024-09-13T14:33:58.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:57 smithi151 ceph-mon[108966]: osdmap e846: 8 total, 8 up, 8 in 2024-09-13T14:33:58.633 INFO:tasks.rados.rados.0.smithi151.stdout:3126: done (2 left) 2024-09-13T14:33:58.633 INFO:tasks.rados.rados.0.smithi151.stdout:3127: done (1 left) 2024-09-13T14:33:58.633 INFO:tasks.rados.rados.0.smithi151.stdout:3128: done (0 left) 2024-09-13T14:33:58.633 INFO:tasks.rados.rados.0.smithi151.stdout:3129: snap_remove snap 376 2024-09-13T14:33:59.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:58 smithi184 ceph-mon[92527]: pgmap v888: 89 pgs: 89 active+clean; 259 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.5 MiB/s rd, 2.8 MiB/s wr, 19 op/s 2024-09-13T14:33:59.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:58 smithi184 ceph-mon[92527]: osdmap e847: 8 total, 8 up, 8 in 2024-09-13T14:33:59.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:58 smithi151 ceph-mon[104450]: pgmap v888: 89 pgs: 89 active+clean; 259 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.5 MiB/s rd, 2.8 MiB/s wr, 19 op/s 2024-09-13T14:33:59.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:58 smithi151 ceph-mon[104450]: osdmap e847: 8 total, 8 up, 8 in 2024-09-13T14:33:59.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:58 smithi151 ceph-mon[108966]: pgmap v888: 89 pgs: 89 active+clean; 259 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.5 MiB/s rd, 2.8 MiB/s wr, 19 op/s 2024-09-13T14:33:59.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:58 smithi151 ceph-mon[108966]: osdmap e847: 8 total, 8 up, 8 in 2024-09-13T14:33:59.611 INFO:tasks.rados.rados.0.smithi151.stdout:3129: done (0 left) 2024-09-13T14:33:59.611 INFO:tasks.rados.rados.0.smithi151.stdout:3130: read oid 35 snap -1 2024-09-13T14:33:59.611 INFO:tasks.rados.rados.0.smithi151.stdout:3130: expect deleted 2024-09-13T14:33:59.611 INFO:tasks.rados.rados.0.smithi151.stdout:3131: snap_remove snap 386 2024-09-13T14:34:00.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:59 smithi184 ceph-mon[92527]: osdmap e848: 8 total, 8 up, 8 in 2024-09-13T14:34:00.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:33:59 smithi184 ceph-mon[92527]: osdmap e849: 8 total, 8 up, 8 in 2024-09-13T14:34:00.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:59 smithi151 ceph-mon[104450]: osdmap e848: 8 total, 8 up, 8 in 2024-09-13T14:34:00.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:33:59 smithi151 ceph-mon[104450]: osdmap e849: 8 total, 8 up, 8 in 2024-09-13T14:34:00.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:59 smithi151 ceph-mon[108966]: osdmap e848: 8 total, 8 up, 8 in 2024-09-13T14:34:00.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:33:59 smithi151 ceph-mon[108966]: osdmap e849: 8 total, 8 up, 8 in 2024-09-13T14:34:00.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:34:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:34:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:34:00.613 INFO:tasks.rados.rados.0.smithi151.stdout:3131: done (1 left) 2024-09-13T14:34:00.613 INFO:tasks.rados.rados.0.smithi151.stdout:3132: write oid 42 current snap is 399 2024-09-13T14:34:00.613 INFO:tasks.rados.rados.0.smithi151.stdout:3132: seq_num 1174 ranges {531418=660688,1982919=452406,3039770=691648} 2024-09-13T14:34:00.621 INFO:tasks.rados.rados.0.smithi151.stdout:3132: writing smithi15186934-42 from 531418 to 1192106 tid 1 2024-09-13T14:34:00.628 INFO:tasks.rados.rados.0.smithi151.stdout:3132: writing smithi15186934-42 from 1982919 to 2435325 tid 2 2024-09-13T14:34:00.636 INFO:tasks.rados.rados.0.smithi151.stdout:3132: writing smithi15186934-42 from 3039770 to 3731418 tid 3 2024-09-13T14:34:00.637 INFO:tasks.rados.rados.0.smithi151.stdout:3133: read oid 37 snap -1 2024-09-13T14:34:00.637 INFO:tasks.rados.rados.0.smithi151.stdout:3133: expect (ObjNum 1129 snap 378 seq_num 1129) 2024-09-13T14:34:00.637 INFO:tasks.rados.rados.0.smithi151.stdout:3134: read oid 27 snap -1 2024-09-13T14:34:00.637 INFO:tasks.rados.rados.0.smithi151.stdout:3134: expect (ObjNum 1166 snap 396 seq_num 1166) 2024-09-13T14:34:00.637 INFO:tasks.rados.rados.0.smithi151.stdout:3135: write oid 24 current snap is 399 2024-09-13T14:34:00.637 INFO:tasks.rados.rados.0.smithi151.stdout:3132: finishing write tid 1 to smithi15186934-42 2024-09-13T14:34:00.637 INFO:tasks.rados.rados.0.smithi151.stdout:3132: finishing write tid 2 to smithi15186934-42 2024-09-13T14:34:00.637 INFO:tasks.rados.rados.0.smithi151.stdout:3135: seq_num 1175 ranges {548225=735937,1925048=596064,2950840=547321} 2024-09-13T14:34:00.643 INFO:tasks.rados.rados.0.smithi151.stdout:3135: writing smithi15186934-24 from 548225 to 1284162 tid 1 2024-09-13T14:34:00.651 INFO:tasks.rados.rados.0.smithi151.stdout:3135: writing smithi15186934-24 from 1925048 to 2521112 tid 2 2024-09-13T14:34:00.657 INFO:tasks.rados.rados.0.smithi151.stdout:3135: writing smithi15186934-24 from 2950840 to 3498161 tid 3 2024-09-13T14:34:00.657 INFO:tasks.rados.rados.0.smithi151.stdout:3130: done (4 left) 2024-09-13T14:34:00.657 INFO:tasks.rados.rados.0.smithi151.stdout:3136: read oid 45 snap 370 2024-09-13T14:34:00.657 INFO:tasks.rados.rados.0.smithi151.stdout:3136: expect (ObjNum 965 snap 330 seq_num 965) 2024-09-13T14:34:00.657 INFO:tasks.rados.rados.0.smithi151.stdout:3137: rollback oid 5 current snap is 399 2024-09-13T14:34:00.657 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 5 to 391 2024-09-13T14:34:00.657 INFO:tasks.rados.rados.0.smithi151.stdout:3132: finishing write tid 3 to smithi15186934-42 2024-09-13T14:34:00.657 INFO:tasks.rados.rados.0.smithi151.stdout:3132: finishing write tid 4 to smithi15186934-42 2024-09-13T14:34:00.658 INFO:tasks.rados.rados.0.smithi151.stdout:3132: finishing write tid 5 to smithi15186934-42 2024-09-13T14:34:00.658 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 1142 (ObjNum 1174 snap 399 seq_num 1174) dirty exists 2024-09-13T14:34:00.658 INFO:tasks.rados.rados.0.smithi151.stdout:3132: left oid 42 (ObjNum 1174 snap 399 seq_num 1174) 2024-09-13T14:34:00.658 INFO:tasks.rados.rados.0.smithi151.stdout:3133: expect (ObjNum 1129 snap 378 seq_num 1129) 2024-09-13T14:34:00.794 INFO:tasks.rados.rados.0.smithi151.stdout:3135: finishing write tid 1 to smithi15186934-24 2024-09-13T14:34:00.794 INFO:tasks.rados.rados.0.smithi151.stdout:3135: finishing write tid 2 to smithi15186934-24 2024-09-13T14:34:00.794 INFO:tasks.rados.rados.0.smithi151.stdout:3132: done (5 left) 2024-09-13T14:34:00.794 INFO:tasks.rados.rados.0.smithi151.stdout:3133: done (4 left) 2024-09-13T14:34:00.794 INFO:tasks.rados.rados.0.smithi151.stdout:3138: read oid 36 snap 394 2024-09-13T14:34:00.794 INFO:tasks.rados.rados.0.smithi151.stdout:3138: expect deleted 2024-09-13T14:34:00.794 INFO:tasks.rados.rados.0.smithi151.stdout:3139: rollback oid 14 current snap is 399 2024-09-13T14:34:00.795 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 14 to 396 2024-09-13T14:34:00.795 INFO:tasks.rados.rados.0.smithi151.stdout:3134: expect (ObjNum 1166 snap 396 seq_num 1166) 2024-09-13T14:34:00.985 INFO:tasks.rados.rados.0.smithi151.stdout:3137: finishing rollback tid 1 to smithi15186934-5 2024-09-13T14:34:00.985 INFO:tasks.rados.rados.0.smithi151.stdout:3137: finishing rollback tid 2 to smithi15186934-5 2024-09-13T14:34:00.985 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 1148 (ObjNum 1140 snap 383 seq_num 1140) dirty exists 2024-09-13T14:34:00.985 INFO:tasks.rados.rados.0.smithi151.stdout:3135: finishing write tid 3 to smithi15186934-24 2024-09-13T14:34:00.985 INFO:tasks.rados.rados.0.smithi151.stdout:3135: finishing write tid 4 to smithi15186934-24 2024-09-13T14:34:00.985 INFO:tasks.rados.rados.0.smithi151.stdout:3135: finishing write tid 5 to smithi15186934-24 2024-09-13T14:34:00.985 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 24 v 1829 (ObjNum 1175 snap 399 seq_num 1175) dirty exists 2024-09-13T14:34:00.985 INFO:tasks.rados.rados.0.smithi151.stdout:3135: left oid 24 (ObjNum 1175 snap 399 seq_num 1175) 2024-09-13T14:34:00.985 INFO:tasks.rados.rados.0.smithi151.stdout:3136: expect (ObjNum 965 snap 330 seq_num 965) 2024-09-13T14:34:01.085 INFO:tasks.rados.rados.0.smithi151.stdout:3139: finishing rollback tid 0 to smithi15186934-14 2024-09-13T14:34:01.085 INFO:tasks.rados.rados.0.smithi151.stdout:3139: finishing rollback tid 1 to smithi15186934-14 2024-09-13T14:34:01.086 INFO:tasks.rados.rados.0.smithi151.stdout:3139: finishing rollback tid 2 to smithi15186934-14 2024-09-13T14:34:01.086 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 1297 (ObjNum 1169 snap 396 seq_num 1169) dirty exists 2024-09-13T14:34:01.086 INFO:tasks.rados.rados.0.smithi151.stdout:3134: done (5 left) 2024-09-13T14:34:01.086 INFO:tasks.rados.rados.0.smithi151.stdout:3135: done (4 left) 2024-09-13T14:34:01.086 INFO:tasks.rados.rados.0.smithi151.stdout:3136: done (3 left) 2024-09-13T14:34:01.086 INFO:tasks.rados.rados.0.smithi151.stdout:3137: done (2 left) 2024-09-13T14:34:01.086 INFO:tasks.rados.rados.0.smithi151.stdout:3138: done (1 left) 2024-09-13T14:34:01.086 INFO:tasks.rados.rados.0.smithi151.stdout:3139: done (0 left) 2024-09-13T14:34:01.086 INFO:tasks.rados.rados.0.smithi151.stdout:3140: read oid 24 snap -1 2024-09-13T14:34:01.086 INFO:tasks.rados.rados.0.smithi151.stdout:3140: expect (ObjNum 1175 snap 399 seq_num 1175) 2024-09-13T14:34:01.086 INFO:tasks.rados.rados.0.smithi151.stdout:3141: delete oid 4 current snap is 399 2024-09-13T14:34:01.088 INFO:tasks.rados.rados.0.smithi151.stdout:3141: done (1 left) 2024-09-13T14:34:01.088 INFO:tasks.rados.rados.0.smithi151.stdout:3142: snap_remove snap 393 2024-09-13T14:34:01.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:00 smithi184 ceph-mon[92527]: pgmap v891: 89 pgs: 89 active+clean; 266 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 2.8 MiB/s wr, 24 op/s 2024-09-13T14:34:01.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:00 smithi184 ceph-mon[92527]: osdmap e850: 8 total, 8 up, 8 in 2024-09-13T14:34:01.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:00 smithi151 ceph-mon[104450]: pgmap v891: 89 pgs: 89 active+clean; 266 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 2.8 MiB/s wr, 24 op/s 2024-09-13T14:34:01.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:00 smithi151 ceph-mon[104450]: osdmap e850: 8 total, 8 up, 8 in 2024-09-13T14:34:01.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:00 smithi151 ceph-mon[108966]: pgmap v891: 89 pgs: 89 active+clean; 266 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 2.8 MiB/s wr, 24 op/s 2024-09-13T14:34:01.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:00 smithi151 ceph-mon[108966]: osdmap e850: 8 total, 8 up, 8 in 2024-09-13T14:34:01.647 INFO:tasks.rados.rados.0.smithi151.stdout:3142: done (1 left) 2024-09-13T14:34:01.647 INFO:tasks.rados.rados.0.smithi151.stdout:3143: snap_remove snap 382 2024-09-13T14:34:02.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:01 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:02.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:02.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:01 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:02.651 INFO:tasks.rados.rados.0.smithi151.stdout:3143: done (1 left) 2024-09-13T14:34:02.651 INFO:tasks.rados.rados.0.smithi151.stdout:3144: write oid 18 current snap is 399 2024-09-13T14:34:02.651 INFO:tasks.rados.rados.0.smithi151.stdout:3144: seq_num 1177 ranges {581839=486435,1642012=599733,3005686=569865} 2024-09-13T14:34:02.657 INFO:tasks.rados.rados.0.smithi151.stdout:3144: writing smithi15186934-18 from 581839 to 1068274 tid 1 2024-09-13T14:34:02.665 INFO:tasks.rados.rados.0.smithi151.stdout:3144: writing smithi15186934-18 from 1642012 to 2241745 tid 2 2024-09-13T14:34:02.673 INFO:tasks.rados.rados.0.smithi151.stdout:3144: writing smithi15186934-18 from 3005686 to 3575551 tid 3 2024-09-13T14:34:02.673 INFO:tasks.rados.rados.0.smithi151.stdout:3145: delete oid 36 current snap is 399 2024-09-13T14:34:02.673 INFO:tasks.rados.rados.0.smithi151.stdout:3140: expect (ObjNum 1175 snap 399 seq_num 1175) 2024-09-13T14:34:02.793 INFO:tasks.rados.rados.0.smithi151.stdout:3144: finishing write tid 1 to smithi15186934-18 2024-09-13T14:34:02.794 INFO:tasks.rados.rados.0.smithi151.stdout:3144: finishing write tid 2 to smithi15186934-18 2024-09-13T14:34:02.794 INFO:tasks.rados.rados.0.smithi151.stdout:3144: finishing write tid 3 to smithi15186934-18 2024-09-13T14:34:02.794 INFO:tasks.rados.rados.0.smithi151.stdout:3144: finishing write tid 4 to smithi15186934-18 2024-09-13T14:34:02.794 INFO:tasks.rados.rados.0.smithi151.stdout:3144: finishing write tid 5 to smithi15186934-18 2024-09-13T14:34:02.794 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 18 v 758 (ObjNum 1177 snap 399 seq_num 1177) dirty exists 2024-09-13T14:34:02.794 INFO:tasks.rados.rados.0.smithi151.stdout:3144: left oid 18 (ObjNum 1177 snap 399 seq_num 1177) 2024-09-13T14:34:02.794 INFO:tasks.rados.rados.0.smithi151.stdout:3140: done (2 left) 2024-09-13T14:34:02.794 INFO:tasks.rados.rados.0.smithi151.stdout:3144: done (1 left) 2024-09-13T14:34:02.795 INFO:tasks.rados.rados.0.smithi151.stdout:3145: done (0 left) 2024-09-13T14:34:02.795 INFO:tasks.rados.rados.0.smithi151.stdout:3146: delete oid 32 current snap is 399 2024-09-13T14:34:02.796 INFO:tasks.rados.rados.0.smithi151.stdout:3146: done (0 left) 2024-09-13T14:34:02.796 INFO:tasks.rados.rados.0.smithi151.stdout:3147: snap_create 2024-09-13T14:34:03.009 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:02 smithi151 ceph-mon[104450]: pgmap v894: 89 pgs: 89 active+clean; 271 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 7.1 MiB/s rd, 1.5 MiB/s wr, 14 op/s 2024-09-13T14:34:03.010 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:02 smithi151 ceph-mon[104450]: osdmap e851: 8 total, 8 up, 8 in 2024-09-13T14:34:03.010 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:02 smithi151 ceph-mon[108966]: pgmap v894: 89 pgs: 89 active+clean; 271 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 7.1 MiB/s rd, 1.5 MiB/s wr, 14 op/s 2024-09-13T14:34:03.010 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:02 smithi151 ceph-mon[108966]: osdmap e851: 8 total, 8 up, 8 in 2024-09-13T14:34:03.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:02 smithi184 ceph-mon[92527]: pgmap v894: 89 pgs: 89 active+clean; 271 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 7.1 MiB/s rd, 1.5 MiB/s wr, 14 op/s 2024-09-13T14:34:03.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:02 smithi184 ceph-mon[92527]: osdmap e851: 8 total, 8 up, 8 in 2024-09-13T14:34:03.664 INFO:tasks.rados.rados.0.smithi151.stdout:3147: done (0 left) 2024-09-13T14:34:03.664 INFO:tasks.rados.rados.0.smithi151.stdout:3148: rollback oid 4 current snap is 400 2024-09-13T14:34:03.664 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 4 to 399 2024-09-13T14:34:03.664 INFO:tasks.rados.rados.0.smithi151.stdout:3149: snap_create 2024-09-13T14:34:03.666 INFO:tasks.rados.rados.0.smithi151.stdout:3148: finishing rollback tid 1 to smithi15186934-4 2024-09-13T14:34:03.666 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 1498 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-13T14:34:04.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:03 smithi184 ceph-mon[92527]: osdmap e852: 8 total, 8 up, 8 in 2024-09-13T14:34:04.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:03 smithi151 ceph-mon[104450]: osdmap e852: 8 total, 8 up, 8 in 2024-09-13T14:34:04.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:03 smithi151 ceph-mon[108966]: osdmap e852: 8 total, 8 up, 8 in 2024-09-13T14:34:04.614 INFO:tasks.rados.rados.0.smithi151.stdout:3148: done (1 left) 2024-09-13T14:34:04.614 INFO:tasks.rados.rados.0.smithi151.stdout:3149: done (0 left) 2024-09-13T14:34:04.614 INFO:tasks.rados.rados.0.smithi151.stdout:3150: rollback oid 18 current snap is 401 2024-09-13T14:34:04.615 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 18 to 394 2024-09-13T14:34:04.615 INFO:tasks.rados.rados.0.smithi151.stdout:3151: read oid 29 snap 378 2024-09-13T14:34:04.615 INFO:tasks.rados.rados.0.smithi151.stdout:3151: expect (ObjNum 1124 snap 376 seq_num 1124) 2024-09-13T14:34:04.615 INFO:tasks.rados.rados.0.smithi151.stdout:3152: snap_create 2024-09-13T14:34:04.620 INFO:tasks.rados.rados.0.smithi151.stdout:3150: finishing rollback tid 0 to smithi15186934-18 2024-09-13T14:34:04.620 INFO:tasks.rados.rados.0.smithi151.stdout:3150: finishing rollback tid 1 to smithi15186934-18 2024-09-13T14:34:04.620 INFO:tasks.rados.rados.0.smithi151.stdout:3150: finishing rollback tid 2 to smithi15186934-18 2024-09-13T14:34:04.620 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 18 v 762 (ObjNum 1131 snap 381 seq_num 1131) dirty exists 2024-09-13T14:34:04.621 INFO:tasks.rados.rados.0.smithi151.stdout:3151: expect (ObjNum 1124 snap 376 seq_num 1124) 2024-09-13T14:34:04.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:34:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:34:04] "GET /metrics HTTP/1.1" 200 36513 "" "Prometheus/2.51.0" 2024-09-13T14:34:04.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:04 smithi151 ceph-mon[104450]: pgmap v897: 89 pgs: 89 active+clean; 274 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-09-13T14:34:04.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:04 smithi151 ceph-mon[104450]: osdmap e853: 8 total, 8 up, 8 in 2024-09-13T14:34:04.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:04 smithi151 ceph-mon[104450]: osdmap e854: 8 total, 8 up, 8 in 2024-09-13T14:34:04.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:04 smithi151 ceph-mon[108966]: pgmap v897: 89 pgs: 89 active+clean; 274 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-09-13T14:34:04.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:04 smithi151 ceph-mon[108966]: osdmap e853: 8 total, 8 up, 8 in 2024-09-13T14:34:04.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:04 smithi151 ceph-mon[108966]: osdmap e854: 8 total, 8 up, 8 in 2024-09-13T14:34:05.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:04 smithi184 ceph-mon[92527]: pgmap v897: 89 pgs: 89 active+clean; 274 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-09-13T14:34:05.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:04 smithi184 ceph-mon[92527]: osdmap e853: 8 total, 8 up, 8 in 2024-09-13T14:34:05.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:04 smithi184 ceph-mon[92527]: osdmap e854: 8 total, 8 up, 8 in 2024-09-13T14:34:05.615 INFO:tasks.rados.rados.0.smithi151.stdout:3150: done (2 left) 2024-09-13T14:34:05.615 INFO:tasks.rados.rados.0.smithi151.stdout:3151: done (1 left) 2024-09-13T14:34:05.615 INFO:tasks.rados.rados.0.smithi151.stdout:3152: done (0 left) 2024-09-13T14:34:05.615 INFO:tasks.rados.rados.0.smithi151.stdout:3153: read oid 26 snap -1 2024-09-13T14:34:05.615 INFO:tasks.rados.rados.0.smithi151.stdout:3153: expect deleted 2024-09-13T14:34:05.615 INFO:tasks.rados.rados.0.smithi151.stdout:3154: snap_remove snap 388 2024-09-13T14:34:06.626 INFO:tasks.rados.rados.0.smithi151.stdout:3154: done (1 left) 2024-09-13T14:34:06.626 INFO:tasks.rados.rados.0.smithi151.stdout:3155: write oid 47 current snap is 402 2024-09-13T14:34:06.626 INFO:tasks.rados.rados.0.smithi151.stdout:3155: seq_num 1180 ranges {632260=428535,1817458=463413,2687955=520034,3709324=122936} 2024-09-13T14:34:06.633 INFO:tasks.rados.rados.0.smithi151.stdout:3155: writing smithi15186934-47 from 632260 to 1060795 tid 1 2024-09-13T14:34:06.640 INFO:tasks.rados.rados.0.smithi151.stdout:3155: writing smithi15186934-47 from 1817458 to 2280871 tid 2 2024-09-13T14:34:06.645 INFO:tasks.rados.rados.0.smithi151.stdout:3155: writing smithi15186934-47 from 2687955 to 3207989 tid 3 2024-09-13T14:34:06.649 INFO:tasks.rados.rados.0.smithi151.stdout:3155: writing smithi15186934-47 from 3709324 to 3832260 tid 4 2024-09-13T14:34:06.649 INFO:tasks.rados.rados.0.smithi151.stdout:3156: snap_remove snap 398 2024-09-13T14:34:07.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:06 smithi184 ceph-mon[92527]: pgmap v900: 89 pgs: 89 active+clean; 281 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 900 KiB/s wr, 11 op/s 2024-09-13T14:34:07.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:06 smithi184 ceph-mon[92527]: osdmap e855: 8 total, 8 up, 8 in 2024-09-13T14:34:07.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:06 smithi151 ceph-mon[104450]: pgmap v900: 89 pgs: 89 active+clean; 281 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 900 KiB/s wr, 11 op/s 2024-09-13T14:34:07.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:06 smithi151 ceph-mon[104450]: osdmap e855: 8 total, 8 up, 8 in 2024-09-13T14:34:07.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:06 smithi151 ceph-mon[108966]: pgmap v900: 89 pgs: 89 active+clean; 281 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 900 KiB/s wr, 11 op/s 2024-09-13T14:34:07.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:06 smithi151 ceph-mon[108966]: osdmap e855: 8 total, 8 up, 8 in 2024-09-13T14:34:07.634 INFO:tasks.rados.rados.0.smithi151.stdout:3156: done (2 left) 2024-09-13T14:34:07.634 INFO:tasks.rados.rados.0.smithi151.stdout:3157: snap_remove snap 397 2024-09-13T14:34:08.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:07 smithi184 ceph-mon[92527]: osdmap e856: 8 total, 8 up, 8 in 2024-09-13T14:34:08.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:07 smithi151 ceph-mon[104450]: osdmap e856: 8 total, 8 up, 8 in 2024-09-13T14:34:08.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:07 smithi151 ceph-mon[108966]: osdmap e856: 8 total, 8 up, 8 in 2024-09-13T14:34:08.633 INFO:tasks.rados.rados.0.smithi151.stdout:3157: done (2 left) 2024-09-13T14:34:08.634 INFO:tasks.rados.rados.0.smithi151.stdout:3158: delete oid 23 current snap is 402 2024-09-13T14:34:08.634 INFO:tasks.rados.rados.0.smithi151.stdout:3155: finishing write tid 1 to smithi15186934-47 2024-09-13T14:34:08.634 INFO:tasks.rados.rados.0.smithi151.stdout:3155: finishing write tid 2 to smithi15186934-47 2024-09-13T14:34:08.634 INFO:tasks.rados.rados.0.smithi151.stdout:3155: finishing write tid 3 to smithi15186934-47 2024-09-13T14:34:08.634 INFO:tasks.rados.rados.0.smithi151.stdout:3155: finishing write tid 4 to smithi15186934-47 2024-09-13T14:34:08.634 INFO:tasks.rados.rados.0.smithi151.stdout:3155: finishing write tid 5 to smithi15186934-47 2024-09-13T14:34:08.634 INFO:tasks.rados.rados.0.smithi151.stdout:3155: finishing write tid 6 to smithi15186934-47 2024-09-13T14:34:08.634 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 1391 (ObjNum 1180 snap 402 seq_num 1180) dirty exists 2024-09-13T14:34:08.634 INFO:tasks.rados.rados.0.smithi151.stdout:3155: left oid 47 (ObjNum 1180 snap 402 seq_num 1180) 2024-09-13T14:34:08.638 INFO:tasks.rados.rados.0.smithi151.stdout:3153: done (2 left) 2024-09-13T14:34:08.638 INFO:tasks.rados.rados.0.smithi151.stdout:3155: done (1 left) 2024-09-13T14:34:08.638 INFO:tasks.rados.rados.0.smithi151.stdout:3158: done (0 left) 2024-09-13T14:34:08.638 INFO:tasks.rados.rados.0.smithi151.stdout:3159: rollback oid 49 current snap is 402 2024-09-13T14:34:08.638 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 49 to 364 2024-09-13T14:34:08.639 INFO:tasks.rados.rados.0.smithi151.stdout:3160: rollback oid 18 current snap is 402 2024-09-13T14:34:08.639 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 18 to 396 2024-09-13T14:34:08.639 INFO:tasks.rados.rados.0.smithi151.stdout:3161: rollback oid 41 current snap is 402 2024-09-13T14:34:08.639 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 41 to 396 2024-09-13T14:34:08.639 INFO:tasks.rados.rados.0.smithi151.stdout:3162: read oid 30 snap -1 2024-09-13T14:34:08.639 INFO:tasks.rados.rados.0.smithi151.stdout:3162: expect deleted 2024-09-13T14:34:08.639 INFO:tasks.rados.rados.0.smithi151.stdout:3163: read oid 25 snap -1 2024-09-13T14:34:08.639 INFO:tasks.rados.rados.0.smithi151.stdout:3163: expect (ObjNum 1150 snap 389 seq_num 1150) 2024-09-13T14:34:08.639 INFO:tasks.rados.rados.0.smithi151.stdout:3164: write oid 37 current snap is 402 2024-09-13T14:34:08.639 INFO:tasks.rados.rados.0.smithi151.stdout:3164: seq_num 1182 ranges {665874=579033,1934422=467082,3142801=542578} 2024-09-13T14:34:08.646 INFO:tasks.rados.rados.0.smithi151.stdout:3164: writing smithi15186934-37 from 665874 to 1244907 tid 1 2024-09-13T14:34:08.652 INFO:tasks.rados.rados.0.smithi151.stdout:3164: writing smithi15186934-37 from 1934422 to 2401504 tid 2 2024-09-13T14:34:08.659 INFO:tasks.rados.rados.0.smithi151.stdout:3164: writing smithi15186934-37 from 3142801 to 3685379 tid 3 2024-09-13T14:34:08.659 INFO:tasks.rados.rados.0.smithi151.stdout:3165: write oid 23 current snap is 402 2024-09-13T14:34:08.660 INFO:tasks.rados.rados.0.smithi151.stdout:3165: seq_num 1183 ranges {682681=654282,1876551=610740,3053871=470203} 2024-09-13T14:34:08.667 INFO:tasks.rados.rados.0.smithi151.stdout:3165: writing smithi15186934-23 from 682681 to 1336963 tid 1 2024-09-13T14:34:08.673 INFO:tasks.rados.rados.0.smithi151.stdout:3165: writing smithi15186934-23 from 1876551 to 2487291 tid 2 2024-09-13T14:34:08.679 INFO:tasks.rados.rados.0.smithi151.stdout:3165: writing smithi15186934-23 from 3053871 to 3524074 tid 3 2024-09-13T14:34:08.680 INFO:tasks.rados.rados.0.smithi151.stdout:3166: write oid 7 current snap is 402 2024-09-13T14:34:08.680 INFO:tasks.rados.rados.0.smithi151.stdout:3166: seq_num 1184 ranges {699488=729531,1935033=470751,3081294=681475} 2024-09-13T14:34:08.687 INFO:tasks.rados.rados.0.smithi151.stdout:3166: writing smithi15186934-7 from 699488 to 1429019 tid 1 2024-09-13T14:34:08.693 INFO:tasks.rados.rados.0.smithi151.stdout:3166: writing smithi15186934-7 from 1935033 to 2405784 tid 2 2024-09-13T14:34:08.700 INFO:tasks.rados.rados.0.smithi151.stdout:3166: writing smithi15186934-7 from 3081294 to 3762769 tid 3 2024-09-13T14:34:08.700 INFO:tasks.rados.rados.0.smithi151.stdout:3167: read oid 38 snap 347 2024-09-13T14:34:08.701 INFO:tasks.rados.rados.0.smithi151.stdout:3167: expect (ObjNum 1003 snap 345 seq_num 1003) 2024-09-13T14:34:08.701 INFO:tasks.rados.rados.0.smithi151.stdout:3168: delete oid 2 current snap is 402 2024-09-13T14:34:08.701 INFO:tasks.rados.rados.0.smithi151.stdout:3159: finishing rollback tid 0 to smithi15186934-49 2024-09-13T14:34:08.701 INFO:tasks.rados.rados.0.smithi151.stdout:3159: finishing rollback tid 1 to smithi15186934-49 2024-09-13T14:34:08.701 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 49 v 1400 (ObjNum 177 snap 0 seq_num 3680709976) dirty dne 2024-09-13T14:34:08.701 INFO:tasks.rados.rados.0.smithi151.stdout:3160: finishing rollback tid 0 to smithi15186934-18 2024-09-13T14:34:08.701 INFO:tasks.rados.rados.0.smithi151.stdout:3160: finishing rollback tid 1 to smithi15186934-18 2024-09-13T14:34:08.701 INFO:tasks.rados.rados.0.smithi151.stdout:3161: finishing rollback tid 1 to smithi15186934-41 2024-09-13T14:34:08.701 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 269 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-13T14:34:08.701 INFO:tasks.rados.rados.0.smithi151.stdout:3160: finishing rollback tid 2 to smithi15186934-18 2024-09-13T14:34:08.701 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 18 v 776 (ObjNum 1131 snap 381 seq_num 1131) dirty exists 2024-09-13T14:34:08.701 INFO:tasks.rados.rados.0.smithi151.stdout:3163: expect (ObjNum 1150 snap 389 seq_num 1150) 2024-09-13T14:34:08.808 INFO:tasks.rados.rados.0.smithi151.stdout:3164: finishing write tid 1 to smithi15186934-37 2024-09-13T14:34:08.809 INFO:tasks.rados.rados.0.smithi151.stdout:3164: finishing write tid 2 to smithi15186934-37 2024-09-13T14:34:08.809 INFO:tasks.rados.rados.0.smithi151.stdout:3164: finishing write tid 3 to smithi15186934-37 2024-09-13T14:34:08.809 INFO:tasks.rados.rados.0.smithi151.stdout:3164: finishing write tid 4 to smithi15186934-37 2024-09-13T14:34:08.809 INFO:tasks.rados.rados.0.smithi151.stdout:3164: finishing write tid 5 to smithi15186934-37 2024-09-13T14:34:08.809 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 37 v 1405 (ObjNum 1182 snap 402 seq_num 1182) dirty exists 2024-09-13T14:34:08.809 INFO:tasks.rados.rados.0.smithi151.stdout:3164: left oid 37 (ObjNum 1182 snap 402 seq_num 1182) 2024-09-13T14:34:08.809 INFO:tasks.rados.rados.0.smithi151.stdout:3165: finishing write tid 1 to smithi15186934-23 2024-09-13T14:34:08.809 INFO:tasks.rados.rados.0.smithi151.stdout:3159: done (9 left) 2024-09-13T14:34:08.809 INFO:tasks.rados.rados.0.smithi151.stdout:3160: done (8 left) 2024-09-13T14:34:08.809 INFO:tasks.rados.rados.0.smithi151.stdout:3161: done (7 left) 2024-09-13T14:34:08.809 INFO:tasks.rados.rados.0.smithi151.stdout:3162: done (6 left) 2024-09-13T14:34:08.809 INFO:tasks.rados.rados.0.smithi151.stdout:3163: done (5 left) 2024-09-13T14:34:08.810 INFO:tasks.rados.rados.0.smithi151.stdout:3164: done (4 left) 2024-09-13T14:34:08.810 INFO:tasks.rados.rados.0.smithi151.stdout:3168: done (3 left) 2024-09-13T14:34:08.810 INFO:tasks.rados.rados.0.smithi151.stdout:3169: write oid 47 current snap is 402 2024-09-13T14:34:08.810 INFO:tasks.rados.rados.0.smithi151.stdout:3165: finishing write tid 2 to smithi15186934-23 2024-09-13T14:34:08.810 INFO:tasks.rados.rados.0.smithi151.stdout:3169: seq_num 1186 ranges {733102=596382,1768350=758067,3136140=704019} 2024-09-13T14:34:08.815 INFO:tasks.rados.rados.0.smithi151.stdout:3169: writing smithi15186934-47 from 733102 to 1329484 tid 1 2024-09-13T14:34:08.821 INFO:tasks.rados.rados.0.smithi151.stdout:3169: writing smithi15186934-47 from 1768350 to 2526417 tid 2 2024-09-13T14:34:08.829 INFO:tasks.rados.rados.0.smithi151.stdout:3169: writing smithi15186934-47 from 3136140 to 3840159 tid 3 2024-09-13T14:34:08.829 INFO:tasks.rados.rados.0.smithi151.stdout:3170: read oid 37 snap 399 2024-09-13T14:34:08.830 INFO:tasks.rados.rados.0.smithi151.stdout:3170: expect (ObjNum 1129 snap 378 seq_num 1129) 2024-09-13T14:34:08.830 INFO:tasks.rados.rados.0.smithi151.stdout:3171: delete oid 22 current snap is 402 2024-09-13T14:34:08.830 INFO:tasks.rados.rados.0.smithi151.stdout:3165: finishing write tid 3 to smithi15186934-23 2024-09-13T14:34:08.830 INFO:tasks.rados.rados.0.smithi151.stdout:3165: finishing write tid 4 to smithi15186934-23 2024-09-13T14:34:08.830 INFO:tasks.rados.rados.0.smithi151.stdout:3165: finishing write tid 5 to smithi15186934-23 2024-09-13T14:34:08.830 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 23 v 1409 (ObjNum 1183 snap 402 seq_num 1183) dirty exists 2024-09-13T14:34:08.830 INFO:tasks.rados.rados.0.smithi151.stdout:3165: left oid 23 (ObjNum 1183 snap 402 seq_num 1183) 2024-09-13T14:34:08.830 INFO:tasks.rados.rados.0.smithi151.stdout:3166: finishing write tid 1 to smithi15186934-7 2024-09-13T14:34:08.830 INFO:tasks.rados.rados.0.smithi151.stdout:3166: finishing write tid 2 to smithi15186934-7 2024-09-13T14:34:08.830 INFO:tasks.rados.rados.0.smithi151.stdout:3166: finishing write tid 3 to smithi15186934-7 2024-09-13T14:34:08.830 INFO:tasks.rados.rados.0.smithi151.stdout:3166: finishing write tid 4 to smithi15186934-7 2024-09-13T14:34:08.830 INFO:tasks.rados.rados.0.smithi151.stdout:3166: finishing write tid 5 to smithi15186934-7 2024-09-13T14:34:08.830 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 1840 (ObjNum 1184 snap 402 seq_num 1184) dirty exists 2024-09-13T14:34:08.831 INFO:tasks.rados.rados.0.smithi151.stdout:3166: left oid 7 (ObjNum 1184 snap 402 seq_num 1184) 2024-09-13T14:34:08.831 INFO:tasks.rados.rados.0.smithi151.stdout:3167: expect (ObjNum 1003 snap 345 seq_num 1003) 2024-09-13T14:34:08.961 INFO:tasks.rados.rados.0.smithi151.stdout:3169: finishing write tid 1 to smithi15186934-47 2024-09-13T14:34:08.961 INFO:tasks.rados.rados.0.smithi151.stdout:3169: finishing write tid 2 to smithi15186934-47 2024-09-13T14:34:08.961 INFO:tasks.rados.rados.0.smithi151.stdout:3169: finishing write tid 3 to smithi15186934-47 2024-09-13T14:34:08.961 INFO:tasks.rados.rados.0.smithi151.stdout:3169: finishing write tid 4 to smithi15186934-47 2024-09-13T14:34:08.961 INFO:tasks.rados.rados.0.smithi151.stdout:3169: finishing write tid 5 to smithi15186934-47 2024-09-13T14:34:08.962 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 1413 (ObjNum 1186 snap 402 seq_num 1186) dirty exists 2024-09-13T14:34:08.962 INFO:tasks.rados.rados.0.smithi151.stdout:3169: left oid 47 (ObjNum 1186 snap 402 seq_num 1186) 2024-09-13T14:34:08.962 INFO:tasks.rados.rados.0.smithi151.stdout:3170: expect (ObjNum 1129 snap 378 seq_num 1129) 2024-09-13T14:34:09.097 INFO:tasks.rados.rados.0.smithi151.stdout:3165: done (5 left) 2024-09-13T14:34:09.097 INFO:tasks.rados.rados.0.smithi151.stdout:3166: done (4 left) 2024-09-13T14:34:09.097 INFO:tasks.rados.rados.0.smithi151.stdout:3167: done (3 left) 2024-09-13T14:34:09.097 INFO:tasks.rados.rados.0.smithi151.stdout:3169: done (2 left) 2024-09-13T14:34:09.097 INFO:tasks.rados.rados.0.smithi151.stdout:3170: done (1 left) 2024-09-13T14:34:09.097 INFO:tasks.rados.rados.0.smithi151.stdout:3171: done (0 left) 2024-09-13T14:34:09.098 INFO:tasks.rados.rados.0.smithi151.stdout:3172: read oid 7 snap -1 2024-09-13T14:34:09.098 INFO:tasks.rados.rados.0.smithi151.stdout:3172: expect (ObjNum 1184 snap 402 seq_num 1184) 2024-09-13T14:34:09.098 INFO:tasks.rados.rados.0.smithi151.stdout:3173: write oid 4 current snap is 402 2024-09-13T14:34:09.098 INFO:tasks.rados.rados.0.smithi151.stdout:3173: seq_num 1188 ranges {766716=746880,2168961=761736,3474633=442916} 2024-09-13T14:34:09.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:08 smithi184 ceph-mon[92527]: pgmap v903: 89 pgs: 89 active+clean; 286 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.2 MiB/s wr, 8 op/s 2024-09-13T14:34:09.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:08 smithi184 ceph-mon[92527]: osdmap e857: 8 total, 8 up, 8 in 2024-09-13T14:34:09.105 INFO:tasks.rados.rados.0.smithi151.stdout:3173: writing smithi15186934-4 from 766716 to 1513596 tid 1 2024-09-13T14:34:09.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:08 smithi151 ceph-mon[104450]: pgmap v903: 89 pgs: 89 active+clean; 286 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.2 MiB/s wr, 8 op/s 2024-09-13T14:34:09.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:08 smithi151 ceph-mon[104450]: osdmap e857: 8 total, 8 up, 8 in 2024-09-13T14:34:09.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:08 smithi151 ceph-mon[108966]: pgmap v903: 89 pgs: 89 active+clean; 286 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.2 MiB/s wr, 8 op/s 2024-09-13T14:34:09.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:08 smithi151 ceph-mon[108966]: osdmap e857: 8 total, 8 up, 8 in 2024-09-13T14:34:09.113 INFO:tasks.rados.rados.0.smithi151.stdout:3173: writing smithi15186934-4 from 2168961 to 2930697 tid 2 2024-09-13T14:34:09.119 INFO:tasks.rados.rados.0.smithi151.stdout:3173: writing smithi15186934-4 from 3474633 to 3917549 tid 3 2024-09-13T14:34:09.119 INFO:tasks.rados.rados.0.smithi151.stdout:3174: write oid 39 current snap is 402 2024-09-13T14:34:09.119 INFO:tasks.rados.rados.0.smithi151.stdout:3174: seq_num 1189 ranges {783523=422129,1827443=621747,3218409=654188} 2024-09-13T14:34:09.125 INFO:tasks.rados.rados.0.smithi151.stdout:3174: writing smithi15186934-39 from 783523 to 1205652 tid 1 2024-09-13T14:34:09.132 INFO:tasks.rados.rados.0.smithi151.stdout:3174: writing smithi15186934-39 from 1827443 to 2449190 tid 2 2024-09-13T14:34:09.140 INFO:tasks.rados.rados.0.smithi151.stdout:3174: writing smithi15186934-39 from 3218409 to 3872597 tid 3 2024-09-13T14:34:09.140 INFO:tasks.rados.rados.0.smithi151.stdout:3175: read oid 26 snap -1 2024-09-13T14:34:09.140 INFO:tasks.rados.rados.0.smithi151.stdout:3175: expect deleted 2024-09-13T14:34:09.141 INFO:tasks.rados.rados.0.smithi151.stdout:3176: write oid 2 current snap is 402 2024-09-13T14:34:09.141 INFO:tasks.rados.rados.0.smithi151.stdout:3172: expect (ObjNum 1184 snap 402 seq_num 1184) 2024-09-13T14:34:09.384 INFO:tasks.rados.rados.0.smithi151.stdout:3173: finishing write tid 1 to smithi15186934-4 2024-09-13T14:34:09.384 INFO:tasks.rados.rados.0.smithi151.stdout:3173: finishing write tid 2 to smithi15186934-4 2024-09-13T14:34:09.385 INFO:tasks.rados.rados.0.smithi151.stdout:3173: finishing write tid 3 to smithi15186934-4 2024-09-13T14:34:09.385 INFO:tasks.rados.rados.0.smithi151.stdout:3173: finishing write tid 4 to smithi15186934-4 2024-09-13T14:34:09.385 INFO:tasks.rados.rados.0.smithi151.stdout:3173: finishing write tid 5 to smithi15186934-4 2024-09-13T14:34:09.385 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 1517 (ObjNum 1188 snap 402 seq_num 1188) dirty exists 2024-09-13T14:34:09.385 INFO:tasks.rados.rados.0.smithi151.stdout:3173: left oid 4 (ObjNum 1188 snap 402 seq_num 1188) 2024-09-13T14:34:09.385 INFO:tasks.rados.rados.0.smithi151.stdout:3174: finishing write tid 1 to smithi15186934-39 2024-09-13T14:34:09.385 INFO:tasks.rados.rados.0.smithi151.stdout:3176: seq_num 1190 ranges {400330=497378,1485925=514405} 2024-09-13T14:34:09.389 INFO:tasks.rados.rados.0.smithi151.stdout:3176: writing smithi15186934-2 from 400330 to 897708 tid 1 2024-09-13T14:34:09.395 INFO:tasks.rados.rados.0.smithi151.stdout:3176: writing smithi15186934-2 from 1485925 to 2000330 tid 2 2024-09-13T14:34:09.396 INFO:tasks.rados.rados.0.smithi151.stdout:3172: done (4 left) 2024-09-13T14:34:09.396 INFO:tasks.rados.rados.0.smithi151.stdout:3173: done (3 left) 2024-09-13T14:34:09.396 INFO:tasks.rados.rados.0.smithi151.stdout:3177: read oid 13 snap 400 2024-09-13T14:34:09.396 INFO:tasks.rados.rados.0.smithi151.stdout:3177: expect (ObjNum 1043 snap 355 seq_num 1043) 2024-09-13T14:34:09.396 INFO:tasks.rados.rados.0.smithi151.stdout:3178: snap_create 2024-09-13T14:34:09.396 INFO:tasks.rados.rados.0.smithi151.stdout:3174: finishing write tid 2 to smithi15186934-39 2024-09-13T14:34:09.396 INFO:tasks.rados.rados.0.smithi151.stdout:3174: finishing write tid 3 to smithi15186934-39 2024-09-13T14:34:09.396 INFO:tasks.rados.rados.0.smithi151.stdout:3174: finishing write tid 4 to smithi15186934-39 2024-09-13T14:34:09.396 INFO:tasks.rados.rados.0.smithi151.stdout:3174: finishing write tid 5 to smithi15186934-39 2024-09-13T14:34:09.396 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 1294 (ObjNum 1189 snap 402 seq_num 1189) dirty exists 2024-09-13T14:34:09.396 INFO:tasks.rados.rados.0.smithi151.stdout:3174: left oid 39 (ObjNum 1189 snap 402 seq_num 1189) 2024-09-13T14:34:09.396 INFO:tasks.rados.rados.0.smithi151.stdout:3176: finishing write tid 1 to smithi15186934-2 2024-09-13T14:34:09.402 INFO:tasks.rados.rados.0.smithi151.stdout:3176: finishing write tid 2 to smithi15186934-2 2024-09-13T14:34:09.403 INFO:tasks.rados.rados.0.smithi151.stdout:3176: finishing write tid 3 to smithi15186934-2 2024-09-13T14:34:09.403 INFO:tasks.rados.rados.0.smithi151.stdout:3176: finishing write tid 4 to smithi15186934-2 2024-09-13T14:34:09.403 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 2 v 1162 (ObjNum 1190 snap 402 seq_num 1190) dirty exists 2024-09-13T14:34:09.403 INFO:tasks.rados.rados.0.smithi151.stdout:3176: left oid 2 (ObjNum 1190 snap 402 seq_num 1190) 2024-09-13T14:34:09.403 INFO:tasks.rados.rados.0.smithi151.stdout:3177: expect (ObjNum 1043 snap 355 seq_num 1043) 2024-09-13T14:34:09.614 INFO:tasks.rados.rados.0.smithi151.stdout:3174: done (4 left) 2024-09-13T14:34:09.614 INFO:tasks.rados.rados.0.smithi151.stdout:3175: done (3 left) 2024-09-13T14:34:09.614 INFO:tasks.rados.rados.0.smithi151.stdout:3176: done (2 left) 2024-09-13T14:34:09.614 INFO:tasks.rados.rados.0.smithi151.stdout:3177: done (1 left) 2024-09-13T14:34:09.614 INFO:tasks.rados.rados.0.smithi151.stdout:3178: done (0 left) 2024-09-13T14:34:09.614 INFO:tasks.rados.rados.0.smithi151.stdout:3179: write oid 1 current snap is 403 2024-09-13T14:34:09.614 INFO:tasks.rados.rados.0.smithi151.stdout:3179: seq_num 1191 ranges {417137=572627,1428054=589083} 2024-09-13T14:34:09.619 INFO:tasks.rados.rados.0.smithi151.stdout:3179: writing smithi15186934-1 from 417137 to 989764 tid 1 2024-09-13T14:34:09.625 INFO:tasks.rados.rados.0.smithi151.stdout:3179: writing smithi15186934-1 from 1428054 to 2017137 tid 2 2024-09-13T14:34:09.625 INFO:tasks.rados.rados.0.smithi151.stdout:3180: write oid 38 current snap is 403 2024-09-13T14:34:09.625 INFO:tasks.rados.rados.0.smithi151.stdout:3179: finishing write tid 1 to smithi15186934-1 2024-09-13T14:34:09.625 INFO:tasks.rados.rados.0.smithi151.stdout:3180: seq_num 1192 ranges {433944=647876,1486536=547408} 2024-09-13T14:34:09.631 INFO:tasks.rados.rados.0.smithi151.stdout:3180: writing smithi15186934-38 from 433944 to 1081820 tid 1 2024-09-13T14:34:09.638 INFO:tasks.rados.rados.0.smithi151.stdout:3180: writing smithi15186934-38 from 1486536 to 2033944 tid 2 2024-09-13T14:34:09.638 INFO:tasks.rados.rados.0.smithi151.stdout:3181: read oid 48 snap -1 2024-09-13T14:34:09.638 INFO:tasks.rados.rados.0.smithi151.stdout:3181: expect (ObjNum 1111 snap 372 seq_num 1111) 2024-09-13T14:34:09.639 INFO:tasks.rados.rados.0.smithi151.stdout:3182: write oid 13 current snap is 403 2024-09-13T14:34:09.639 INFO:tasks.rados.rados.0.smithi151.stdout:3179: finishing write tid 2 to smithi15186934-1 2024-09-13T14:34:09.639 INFO:tasks.rados.rados.0.smithi151.stdout:3179: finishing write tid 3 to smithi15186934-1 2024-09-13T14:34:09.639 INFO:tasks.rados.rados.0.smithi151.stdout:3179: finishing write tid 4 to smithi15186934-1 2024-09-13T14:34:09.639 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 1 v 954 (ObjNum 1191 snap 403 seq_num 1191) dirty exists 2024-09-13T14:34:09.639 INFO:tasks.rados.rados.0.smithi151.stdout:3179: left oid 1 (ObjNum 1191 snap 403 seq_num 1191) 2024-09-13T14:34:09.639 INFO:tasks.rados.rados.0.smithi151.stdout:3180: finishing write tid 1 to smithi15186934-38 2024-09-13T14:34:09.639 INFO:tasks.rados.rados.0.smithi151.stdout:3182: seq_num 1193 ranges {450751=723125,1945018=105733} 2024-09-13T14:34:09.644 INFO:tasks.rados.rados.0.smithi151.stdout:3182: writing smithi15186934-13 from 450751 to 1173876 tid 1 2024-09-13T14:34:09.650 INFO:tasks.rados.rados.0.smithi151.stdout:3182: writing smithi15186934-13 from 1945018 to 2050751 tid 2 2024-09-13T14:34:09.650 INFO:tasks.rados.rados.0.smithi151.stdout:3179: done (3 left) 2024-09-13T14:34:09.650 INFO:tasks.rados.rados.0.smithi151.stdout:3183: read oid 4 snap -1 2024-09-13T14:34:09.651 INFO:tasks.rados.rados.0.smithi151.stdout:3183: expect (ObjNum 1188 snap 402 seq_num 1188) 2024-09-13T14:34:09.651 INFO:tasks.rados.rados.0.smithi151.stdout:3184: snap_create 2024-09-13T14:34:09.651 INFO:tasks.rados.rados.0.smithi151.stdout:3180: finishing write tid 2 to smithi15186934-38 2024-09-13T14:34:09.651 INFO:tasks.rados.rados.0.smithi151.stdout:3180: finishing write tid 3 to smithi15186934-38 2024-09-13T14:34:09.651 INFO:tasks.rados.rados.0.smithi151.stdout:3180: finishing write tid 4 to smithi15186934-38 2024-09-13T14:34:09.651 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 1298 (ObjNum 1192 snap 403 seq_num 1192) dirty exists 2024-09-13T14:34:09.651 INFO:tasks.rados.rados.0.smithi151.stdout:3180: left oid 38 (ObjNum 1192 snap 403 seq_num 1192) 2024-09-13T14:34:09.651 INFO:tasks.rados.rados.0.smithi151.stdout:3181: expect (ObjNum 1111 snap 372 seq_num 1111) 2024-09-13T14:34:09.776 INFO:tasks.rados.rados.0.smithi151.stdout:3182: finishing write tid 1 to smithi15186934-13 2024-09-13T14:34:09.777 INFO:tasks.rados.rados.0.smithi151.stdout:3182: finishing write tid 2 to smithi15186934-13 2024-09-13T14:34:09.777 INFO:tasks.rados.rados.0.smithi151.stdout:3182: finishing write tid 3 to smithi15186934-13 2024-09-13T14:34:09.777 INFO:tasks.rados.rados.0.smithi151.stdout:3182: finishing write tid 4 to smithi15186934-13 2024-09-13T14:34:09.777 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 13 v 1166 (ObjNum 1193 snap 403 seq_num 1193) dirty exists 2024-09-13T14:34:09.777 INFO:tasks.rados.rados.0.smithi151.stdout:3182: left oid 13 (ObjNum 1193 snap 403 seq_num 1193) 2024-09-13T14:34:09.777 INFO:tasks.rados.rados.0.smithi151.stdout:3183: expect (ObjNum 1188 snap 402 seq_num 1188) 2024-09-13T14:34:10.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:09 smithi184 ceph-mon[92527]: osdmap e858: 8 total, 8 up, 8 in 2024-09-13T14:34:10.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:09 smithi184 ceph-mon[92527]: osdmap e859: 8 total, 8 up, 8 in 2024-09-13T14:34:10.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:09 smithi151 ceph-mon[104450]: osdmap e858: 8 total, 8 up, 8 in 2024-09-13T14:34:10.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:09 smithi151 ceph-mon[104450]: osdmap e859: 8 total, 8 up, 8 in 2024-09-13T14:34:10.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:09 smithi151 ceph-mon[108966]: osdmap e858: 8 total, 8 up, 8 in 2024-09-13T14:34:10.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:09 smithi151 ceph-mon[108966]: osdmap e859: 8 total, 8 up, 8 in 2024-09-13T14:34:10.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:34:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:34:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:34:10.649 INFO:tasks.rados.rados.0.smithi151.stdout:3180: done (4 left) 2024-09-13T14:34:10.649 INFO:tasks.rados.rados.0.smithi151.stdout:3181: done (3 left) 2024-09-13T14:34:10.649 INFO:tasks.rados.rados.0.smithi151.stdout:3182: done (2 left) 2024-09-13T14:34:10.649 INFO:tasks.rados.rados.0.smithi151.stdout:3183: done (1 left) 2024-09-13T14:34:10.649 INFO:tasks.rados.rados.0.smithi151.stdout:3184: done (0 left) 2024-09-13T14:34:10.649 INFO:tasks.rados.rados.0.smithi151.stdout:3185: write oid 19 current snap is 404 2024-09-13T14:34:10.649 INFO:tasks.rados.rados.0.smithi151.stdout:3185: seq_num 1194 ranges {467558=514727,1719853=347705} 2024-09-13T14:34:10.654 INFO:tasks.rados.rados.0.smithi151.stdout:3185: writing smithi15186934-19 from 467558 to 982285 tid 1 2024-09-13T14:34:10.660 INFO:tasks.rados.rados.0.smithi151.stdout:3185: writing smithi15186934-19 from 1719853 to 2067558 tid 2 2024-09-13T14:34:10.660 INFO:tasks.rados.rados.0.smithi151.stdout:3186: rollback oid 2 current snap is 404 2024-09-13T14:34:10.660 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 2 to 383 2024-09-13T14:34:10.660 INFO:tasks.rados.rados.0.smithi151.stdout:3185: finishing write tid 1 to smithi15186934-19 2024-09-13T14:34:10.660 INFO:tasks.rados.rados.0.smithi151.stdout:3187: delete oid 15 current snap is 404 2024-09-13T14:34:10.663 INFO:tasks.rados.rados.0.smithi151.stdout:3187: done (2 left) 2024-09-13T14:34:10.663 INFO:tasks.rados.rados.0.smithi151.stdout:3188: delete oid 8 current snap is 404 2024-09-13T14:34:10.664 INFO:tasks.rados.rados.0.smithi151.stdout:3186: finishing rollback tid 0 to smithi15186934-2 2024-09-13T14:34:10.664 INFO:tasks.rados.rados.0.smithi151.stdout:3185: finishing write tid 2 to smithi15186934-19 2024-09-13T14:34:10.664 INFO:tasks.rados.rados.0.smithi151.stdout:3185: finishing write tid 3 to smithi15186934-19 2024-09-13T14:34:10.664 INFO:tasks.rados.rados.0.smithi151.stdout:3186: finishing rollback tid 1 to smithi15186934-2 2024-09-13T14:34:10.664 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 2 v 1169 (ObjNum 0 snap 0 seq_num 304) dirty dne 2024-09-13T14:34:10.664 INFO:tasks.rados.rados.0.smithi151.stdout:3185: finishing write tid 4 to smithi15186934-19 2024-09-13T14:34:10.665 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 19 v 738 (ObjNum 1194 snap 404 seq_num 1194) dirty exists 2024-09-13T14:34:10.665 INFO:tasks.rados.rados.0.smithi151.stdout:3185: left oid 19 (ObjNum 1194 snap 404 seq_num 1194) 2024-09-13T14:34:10.667 INFO:tasks.rados.rados.0.smithi151.stdout:3185: done (2 left) 2024-09-13T14:34:10.667 INFO:tasks.rados.rados.0.smithi151.stdout:3186: done (1 left) 2024-09-13T14:34:10.667 INFO:tasks.rados.rados.0.smithi151.stdout:3188: done (0 left) 2024-09-13T14:34:10.667 INFO:tasks.rados.rados.0.smithi151.stdout:3189: write oid 32 current snap is 404 2024-09-13T14:34:10.667 INFO:tasks.rados.rados.0.smithi151.stdout:3189: seq_num 1197 ranges {517979=740474,1778946=339033} 2024-09-13T14:34:10.673 INFO:tasks.rados.rados.0.smithi151.stdout:3189: writing smithi15186934-32 from 517979 to 1258453 tid 1 2024-09-13T14:34:10.679 INFO:tasks.rados.rados.0.smithi151.stdout:3189: writing smithi15186934-32 from 1778946 to 2117979 tid 2 2024-09-13T14:34:10.679 INFO:tasks.rados.rados.0.smithi151.stdout:3190: rollback oid 35 current snap is 404 2024-09-13T14:34:10.679 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 35 to 394 2024-09-13T14:34:10.679 INFO:tasks.rados.rados.0.smithi151.stdout:3189: finishing write tid 1 to smithi15186934-32 2024-09-13T14:34:10.679 INFO:tasks.rados.rados.0.smithi151.stdout:3191: rollback oid 38 current snap is 404 2024-09-13T14:34:10.679 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 38 to 377 2024-09-13T14:34:10.679 INFO:tasks.rados.rados.0.smithi151.stdout:3192: snap_remove snap 378 2024-09-13T14:34:11.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:10 smithi184 ceph-mon[92527]: pgmap v906: 89 pgs: 89 active+clean; 286 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 780 KiB/s wr, 6 op/s 2024-09-13T14:34:11.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:10 smithi151 ceph-mon[104450]: pgmap v906: 89 pgs: 89 active+clean; 286 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 780 KiB/s wr, 6 op/s 2024-09-13T14:34:11.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:10 smithi151 ceph-mon[108966]: pgmap v906: 89 pgs: 89 active+clean; 286 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 780 KiB/s wr, 6 op/s 2024-09-13T14:34:11.655 INFO:tasks.rados.rados.0.smithi151.stdout:3192: done (3 left) 2024-09-13T14:34:11.655 INFO:tasks.rados.rados.0.smithi151.stdout:3193: snap_remove snap 395 2024-09-13T14:34:11.655 INFO:tasks.rados.rados.0.smithi151.stdout:3191: finishing rollback tid 0 to smithi15186934-38 2024-09-13T14:34:11.656 INFO:tasks.rados.rados.0.smithi151.stdout:3190: finishing rollback tid 1 to smithi15186934-35 2024-09-13T14:34:11.656 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 35 v 1753 (ObjNum 161 snap 0 seq_num 3680702892) dirty dne 2024-09-13T14:34:11.656 INFO:tasks.rados.rados.0.smithi151.stdout:3189: finishing write tid 2 to smithi15186934-32 2024-09-13T14:34:11.656 INFO:tasks.rados.rados.0.smithi151.stdout:3191: finishing rollback tid 1 to smithi15186934-38 2024-09-13T14:34:11.656 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 1301 (ObjNum 33 snap 0 seq_num 3092734768) dirty dne 2024-09-13T14:34:11.656 INFO:tasks.rados.rados.0.smithi151.stdout:3189: finishing write tid 3 to smithi15186934-32 2024-09-13T14:34:11.656 INFO:tasks.rados.rados.0.smithi151.stdout:3189: finishing write tid 4 to smithi15186934-32 2024-09-13T14:34:11.656 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 32 v 1324 (ObjNum 1197 snap 404 seq_num 1197) dirty exists 2024-09-13T14:34:11.656 INFO:tasks.rados.rados.0.smithi151.stdout:3189: left oid 32 (ObjNum 1197 snap 404 seq_num 1197) 2024-09-13T14:34:12.005 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:11 smithi184 ceph-mon[92527]: osdmap e860: 8 total, 8 up, 8 in 2024-09-13T14:34:12.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:11 smithi151 ceph-mon[104450]: osdmap e860: 8 total, 8 up, 8 in 2024-09-13T14:34:12.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:11 smithi151 ceph-mon[108966]: osdmap e860: 8 total, 8 up, 8 in 2024-09-13T14:34:12.658 INFO:tasks.rados.rados.0.smithi151.stdout:3189: done (3 left) 2024-09-13T14:34:12.658 INFO:tasks.rados.rados.0.smithi151.stdout:3190: done (2 left) 2024-09-13T14:34:12.658 INFO:tasks.rados.rados.0.smithi151.stdout:3191: done (1 left) 2024-09-13T14:34:12.658 INFO:tasks.rados.rados.0.smithi151.stdout:3193: done (0 left) 2024-09-13T14:34:12.658 INFO:tasks.rados.rados.0.smithi151.stdout:3194: delete oid 35 current snap is 404 2024-09-13T14:34:12.665 INFO:tasks.rados.rados.0.smithi151.stdout:3194: done (0 left) 2024-09-13T14:34:12.665 INFO:tasks.rados.rados.0.smithi151.stdout:3195: read oid 9 snap -1 2024-09-13T14:34:12.665 INFO:tasks.rados.rados.0.smithi151.stdout:3195: expect (ObjNum 1164 snap 396 seq_num 1164) 2024-09-13T14:34:12.666 INFO:tasks.rados.rados.0.smithi151.stdout:3196: snap_create 2024-09-13T14:34:12.675 INFO:tasks.rados.rados.0.smithi151.stdout:3195: expect (ObjNum 1164 snap 396 seq_num 1164) 2024-09-13T14:34:13.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:12 smithi184 ceph-mon[92527]: pgmap v909: 89 pgs: 89 active+clean; 286 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 284 KiB/s wr, 1 op/s 2024-09-13T14:34:13.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:12 smithi184 ceph-mon[92527]: osdmap e861: 8 total, 8 up, 8 in 2024-09-13T14:34:13.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:12 smithi151 ceph-mon[104450]: pgmap v909: 89 pgs: 89 active+clean; 286 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 284 KiB/s wr, 1 op/s 2024-09-13T14:34:13.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:12 smithi151 ceph-mon[104450]: osdmap e861: 8 total, 8 up, 8 in 2024-09-13T14:34:13.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:12 smithi151 ceph-mon[108966]: pgmap v909: 89 pgs: 89 active+clean; 286 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 284 KiB/s wr, 1 op/s 2024-09-13T14:34:13.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:12 smithi151 ceph-mon[108966]: osdmap e861: 8 total, 8 up, 8 in 2024-09-13T14:34:13.670 INFO:tasks.rados.rados.0.smithi151.stdout:3195: done (1 left) 2024-09-13T14:34:13.670 INFO:tasks.rados.rados.0.smithi151.stdout:3196: done (0 left) 2024-09-13T14:34:13.670 INFO:tasks.rados.rados.0.smithi151.stdout:3197: write oid 31 current snap is 405 2024-09-13T14:34:13.670 INFO:tasks.rados.rados.0.smithi151.stdout:3197: seq_num 1199 ranges {551593=490972,1779557=372036} 2024-09-13T14:34:13.675 INFO:tasks.rados.rados.0.smithi151.stdout:3197: writing smithi15186934-31 from 551593 to 1042565 tid 1 2024-09-13T14:34:13.681 INFO:tasks.rados.rados.0.smithi151.stdout:3197: writing smithi15186934-31 from 1779557 to 2151593 tid 2 2024-09-13T14:34:13.682 INFO:tasks.rados.rados.0.smithi151.stdout:3198: delete oid 2 current snap is 405 2024-09-13T14:34:13.682 INFO:tasks.rados.rados.0.smithi151.stdout:3197: finishing write tid 1 to smithi15186934-31 2024-09-13T14:34:13.684 INFO:tasks.rados.rados.0.smithi151.stdout:3198: done (1 left) 2024-09-13T14:34:13.684 INFO:tasks.rados.rados.0.smithi151.stdout:3199: snap_create 2024-09-13T14:34:13.685 INFO:tasks.rados.rados.0.smithi151.stdout:3197: finishing write tid 2 to smithi15186934-31 2024-09-13T14:34:13.685 INFO:tasks.rados.rados.0.smithi151.stdout:3197: finishing write tid 3 to smithi15186934-31 2024-09-13T14:34:13.685 INFO:tasks.rados.rados.0.smithi151.stdout:3197: finishing write tid 4 to smithi15186934-31 2024-09-13T14:34:13.685 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 31 v 1158 (ObjNum 1199 snap 405 seq_num 1199) dirty exists 2024-09-13T14:34:13.685 INFO:tasks.rados.rados.0.smithi151.stdout:3197: left oid 31 (ObjNum 1199 snap 405 seq_num 1199) 2024-09-13T14:34:14.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:13 smithi184 ceph-mon[92527]: osdmap e862: 8 total, 8 up, 8 in 2024-09-13T14:34:14.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:13 smithi151 ceph-mon[104450]: osdmap e862: 8 total, 8 up, 8 in 2024-09-13T14:34:14.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:13 smithi151 ceph-mon[108966]: osdmap e862: 8 total, 8 up, 8 in 2024-09-13T14:34:14.616 INFO:tasks.rados.rados.0.smithi151.stdout:3197: done (1 left) 2024-09-13T14:34:14.616 INFO:tasks.rados.rados.0.smithi151.stdout:3199: done (0 left) 2024-09-13T14:34:14.616 INFO:tasks.rados.rados.0.smithi151.stdout:3200: write oid 13 current snap is 406 2024-09-13T14:34:14.616 INFO:tasks.rados.rados.0.smithi151.stdout:3200: seq_num 1201 ranges {585207=641470,1896521=288686} 2024-09-13T14:34:14.622 INFO:tasks.rados.rados.0.smithi151.stdout:3200: writing smithi15186934-13 from 585207 to 1226677 tid 1 2024-09-13T14:34:14.627 INFO:tasks.rados.rados.0.smithi151.stdout:3200: writing smithi15186934-13 from 1896521 to 2185207 tid 2 2024-09-13T14:34:14.627 INFO:tasks.rados.rados.0.smithi151.stdout:3201: snap_create 2024-09-13T14:34:14.630 INFO:tasks.rados.rados.0.smithi151.stdout:3200: finishing write tid 1 to smithi15186934-13 2024-09-13T14:34:14.630 INFO:tasks.rados.rados.0.smithi151.stdout:3200: finishing write tid 2 to smithi15186934-13 2024-09-13T14:34:14.630 INFO:tasks.rados.rados.0.smithi151.stdout:3200: finishing write tid 3 to smithi15186934-13 2024-09-13T14:34:14.631 INFO:tasks.rados.rados.0.smithi151.stdout:3200: finishing write tid 4 to smithi15186934-13 2024-09-13T14:34:14.631 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 13 v 1180 (ObjNum 1201 snap 406 seq_num 1201) dirty exists 2024-09-13T14:34:14.631 INFO:tasks.rados.rados.0.smithi151.stdout:3200: left oid 13 (ObjNum 1201 snap 406 seq_num 1201) 2024-09-13T14:34:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:14 smithi151 ceph-mon[104450]: pgmap v912: 89 pgs: 89 active+clean; 302 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 3.4 MiB/s wr, 26 op/s 2024-09-13T14:34:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:14 smithi151 ceph-mon[104450]: osdmap e863: 8 total, 8 up, 8 in 2024-09-13T14:34:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:14 smithi151 ceph-mon[104450]: osdmap e864: 8 total, 8 up, 8 in 2024-09-13T14:34:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:14 smithi151 ceph-mon[108966]: pgmap v912: 89 pgs: 89 active+clean; 302 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 3.4 MiB/s wr, 26 op/s 2024-09-13T14:34:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:14 smithi151 ceph-mon[108966]: osdmap e863: 8 total, 8 up, 8 in 2024-09-13T14:34:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:14 smithi151 ceph-mon[108966]: osdmap e864: 8 total, 8 up, 8 in 2024-09-13T14:34:14.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:34:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:34:14] "GET /metrics HTTP/1.1" 200 36513 "" "Prometheus/2.51.0" 2024-09-13T14:34:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:14 smithi184 ceph-mon[92527]: pgmap v912: 89 pgs: 89 active+clean; 302 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 3.4 MiB/s wr, 26 op/s 2024-09-13T14:34:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:14 smithi184 ceph-mon[92527]: osdmap e863: 8 total, 8 up, 8 in 2024-09-13T14:34:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:14 smithi184 ceph-mon[92527]: osdmap e864: 8 total, 8 up, 8 in 2024-09-13T14:34:15.618 INFO:tasks.rados.rados.0.smithi151.stdout:3200: done (1 left) 2024-09-13T14:34:15.618 INFO:tasks.rados.rados.0.smithi151.stdout:3201: done (0 left) 2024-09-13T14:34:15.618 INFO:tasks.rados.rados.0.smithi151.stdout:3202: write oid 38 current snap is 407 2024-09-13T14:34:15.618 INFO:tasks.rados.rados.0.smithi151.stdout:3202: seq_num 1202 ranges {602014=433072,1671356=530658} 2024-09-13T14:34:15.623 INFO:tasks.rados.rados.0.smithi151.stdout:3202: writing smithi15186934-38 from 602014 to 1035086 tid 1 2024-09-13T14:34:15.630 INFO:tasks.rados.rados.0.smithi151.stdout:3202: writing smithi15186934-38 from 1671356 to 2202014 tid 2 2024-09-13T14:34:15.630 INFO:tasks.rados.rados.0.smithi151.stdout:3203: write oid 18 current snap is 407 2024-09-13T14:34:15.630 INFO:tasks.rados.rados.0.smithi151.stdout:3202: finishing write tid 1 to smithi15186934-38 2024-09-13T14:34:15.630 INFO:tasks.rados.rados.0.smithi151.stdout:3203: seq_num 1203 ranges {618821=508321,1613485=531077} 2024-09-13T14:34:15.636 INFO:tasks.rados.rados.0.smithi151.stdout:3203: writing smithi15186934-18 from 618821 to 1127142 tid 1 2024-09-13T14:34:15.644 INFO:tasks.rados.rados.0.smithi151.stdout:3203: writing smithi15186934-18 from 1613485 to 2144562 tid 2 2024-09-13T14:34:15.644 INFO:tasks.rados.rados.0.smithi151.stdout:3204: rollback oid 40 current snap is 407 2024-09-13T14:34:15.644 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 40 to 391 2024-09-13T14:34:15.644 INFO:tasks.rados.rados.0.smithi151.stdout:3202: finishing write tid 2 to smithi15186934-38 2024-09-13T14:34:15.644 INFO:tasks.rados.rados.0.smithi151.stdout:3202: finishing write tid 3 to smithi15186934-38 2024-09-13T14:34:15.644 INFO:tasks.rados.rados.0.smithi151.stdout:3202: finishing write tid 4 to smithi15186934-38 2024-09-13T14:34:15.644 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 1314 (ObjNum 1202 snap 407 seq_num 1202) dirty exists 2024-09-13T14:34:15.644 INFO:tasks.rados.rados.0.smithi151.stdout:3202: left oid 38 (ObjNum 1202 snap 407 seq_num 1202) 2024-09-13T14:34:15.644 INFO:tasks.rados.rados.0.smithi151.stdout:3203: finishing write tid 1 to smithi15186934-18 2024-09-13T14:34:15.644 INFO:tasks.rados.rados.0.smithi151.stdout:3202: done (2 left) 2024-09-13T14:34:15.645 INFO:tasks.rados.rados.0.smithi151.stdout:3205: read oid 7 snap 364 2024-09-13T14:34:15.645 INFO:tasks.rados.rados.0.smithi151.stdout:3205: expect (ObjNum 951 snap 327 seq_num 951) 2024-09-13T14:34:15.645 INFO:tasks.rados.rados.0.smithi151.stdout:3206: read oid 10 snap -1 2024-09-13T14:34:15.645 INFO:tasks.rados.rados.0.smithi151.stdout:3206: expect (ObjNum 1162 snap 396 seq_num 1162) 2024-09-13T14:34:15.645 INFO:tasks.rados.rados.0.smithi151.stdout:3207: rollback oid 15 current snap is 407 2024-09-13T14:34:15.645 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 15 to 384 2024-09-13T14:34:15.645 INFO:tasks.rados.rados.0.smithi151.stdout:3208: write oid 1 current snap is 407 2024-09-13T14:34:15.645 INFO:tasks.rados.rados.0.smithi151.stdout:3208: seq_num 1204 ranges {635628=583570,1671967=563661} 2024-09-13T14:34:15.650 INFO:tasks.rados.rados.0.smithi151.stdout:3208: writing smithi15186934-1 from 635628 to 1219198 tid 1 2024-09-13T14:34:15.656 INFO:tasks.rados.rados.0.smithi151.stdout:3208: writing smithi15186934-1 from 1671967 to 2235628 tid 2 2024-09-13T14:34:15.656 INFO:tasks.rados.rados.0.smithi151.stdout:3209: delete oid 19 current snap is 407 2024-09-13T14:34:15.656 INFO:tasks.rados.rados.0.smithi151.stdout:3207: finishing rollback tid 1 to smithi15186934-15 2024-09-13T14:34:15.656 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 15 v 1150 (ObjNum 33 snap 0 seq_num 3093222688) dirty dne 2024-09-13T14:34:15.656 INFO:tasks.rados.rados.0.smithi151.stdout:3203: finishing write tid 2 to smithi15186934-18 2024-09-13T14:34:15.657 INFO:tasks.rados.rados.0.smithi151.stdout:3203: finishing write tid 3 to smithi15186934-18 2024-09-13T14:34:15.657 INFO:tasks.rados.rados.0.smithi151.stdout:3204: finishing rollback tid 1 to smithi15186934-40 2024-09-13T14:34:15.657 INFO:tasks.rados.rados.0.smithi151.stdout:3203: finishing write tid 4 to smithi15186934-18 2024-09-13T14:34:15.657 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 18 v 784 (ObjNum 1203 snap 407 seq_num 1203) dirty exists 2024-09-13T14:34:15.657 INFO:tasks.rados.rados.0.smithi151.stdout:3203: left oid 18 (ObjNum 1203 snap 407 seq_num 1203) 2024-09-13T14:34:15.657 INFO:tasks.rados.rados.0.smithi151.stdout:3204: finishing rollback tid 2 to smithi15186934-40 2024-09-13T14:34:15.657 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 40 v 1238 (ObjNum 1031 snap 352 seq_num 1031) dirty exists 2024-09-13T14:34:15.657 INFO:tasks.rados.rados.0.smithi151.stdout:3206: expect (ObjNum 1162 snap 396 seq_num 1162) 2024-09-13T14:34:15.858 INFO:tasks.rados.rados.0.smithi151.stdout:3208: finishing write tid 1 to smithi15186934-1 2024-09-13T14:34:15.858 INFO:tasks.rados.rados.0.smithi151.stdout:3203: done (6 left) 2024-09-13T14:34:15.858 INFO:tasks.rados.rados.0.smithi151.stdout:3204: done (5 left) 2024-09-13T14:34:15.858 INFO:tasks.rados.rados.0.smithi151.stdout:3206: done (4 left) 2024-09-13T14:34:15.858 INFO:tasks.rados.rados.0.smithi151.stdout:3207: done (3 left) 2024-09-13T14:34:15.858 INFO:tasks.rados.rados.0.smithi151.stdout:3209: done (2 left) 2024-09-13T14:34:15.858 INFO:tasks.rados.rados.0.smithi151.stdout:3210: snap_remove snap 401 2024-09-13T14:34:16.674 INFO:tasks.rados.rados.0.smithi151.stdout:3210: done (2 left) 2024-09-13T14:34:16.674 INFO:tasks.rados.rados.0.smithi151.stdout:3211: read oid 15 snap 384 2024-09-13T14:34:16.674 INFO:tasks.rados.rados.0.smithi151.stdout:3211: expect deleted 2024-09-13T14:34:16.674 INFO:tasks.rados.rados.0.smithi151.stdout:3212: delete oid 31 current snap is 407 2024-09-13T14:34:16.674 INFO:tasks.rados.rados.0.smithi151.stdout:3205: expect (ObjNum 951 snap 327 seq_num 951) 2024-09-13T14:34:16.906 INFO:tasks.rados.rados.0.smithi151.stdout:3208: finishing write tid 2 to smithi15186934-1 2024-09-13T14:34:16.906 INFO:tasks.rados.rados.0.smithi151.stdout:3208: finishing write tid 3 to smithi15186934-1 2024-09-13T14:34:16.906 INFO:tasks.rados.rados.0.smithi151.stdout:3208: finishing write tid 4 to smithi15186934-1 2024-09-13T14:34:16.906 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 1 v 969 (ObjNum 1204 snap 407 seq_num 1204) dirty exists 2024-09-13T14:34:16.907 INFO:tasks.rados.rados.0.smithi151.stdout:3208: left oid 1 (ObjNum 1204 snap 407 seq_num 1204) 2024-09-13T14:34:16.907 INFO:tasks.rados.rados.0.smithi151.stdout:3205: done (3 left) 2024-09-13T14:34:16.907 INFO:tasks.rados.rados.0.smithi151.stdout:3208: done (2 left) 2024-09-13T14:34:16.907 INFO:tasks.rados.rados.0.smithi151.stdout:3211: done (1 left) 2024-09-13T14:34:16.907 INFO:tasks.rados.rados.0.smithi151.stdout:3212: done (0 left) 2024-09-13T14:34:16.907 INFO:tasks.rados.rados.0.smithi151.stdout:3213: write oid 11 current snap is 407 2024-09-13T14:34:16.907 INFO:tasks.rados.rados.0.smithi151.stdout:3213: seq_num 1207 ranges {686049=409317,1847413=438636} 2024-09-13T14:34:16.911 INFO:tasks.rados.rados.0.smithi151.stdout:3213: writing smithi15186934-11 from 686049 to 1095366 tid 1 2024-09-13T14:34:16.918 INFO:tasks.rados.rados.0.smithi151.stdout:3213: writing smithi15186934-11 from 1847413 to 2286049 tid 2 2024-09-13T14:34:16.918 INFO:tasks.rados.rados.0.smithi151.stdout:3214: delete oid 47 current snap is 407 2024-09-13T14:34:16.918 INFO:tasks.rados.rados.0.smithi151.stdout:3213: finishing write tid 1 to smithi15186934-11 2024-09-13T14:34:16.921 INFO:tasks.rados.rados.0.smithi151.stdout:3213: finishing write tid 2 to smithi15186934-11 2024-09-13T14:34:16.921 INFO:tasks.rados.rados.0.smithi151.stdout:3214: done (1 left) 2024-09-13T14:34:16.921 INFO:tasks.rados.rados.0.smithi151.stdout:3215: rollback oid 46 current snap is 407 2024-09-13T14:34:16.921 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 46 to 404 2024-09-13T14:34:16.921 INFO:tasks.rados.rados.0.smithi151.stdout:3216: write oid 25 current snap is 407 2024-09-13T14:34:16.921 INFO:tasks.rados.rados.0.smithi151.stdout:3216: seq_num 1209 ranges {719663=676168,1964377=355286} 2024-09-13T14:34:16.928 INFO:tasks.rados.rados.0.smithi151.stdout:3216: writing smithi15186934-25 from 719663 to 1395831 tid 1 2024-09-13T14:34:16.934 INFO:tasks.rados.rados.0.smithi151.stdout:3216: writing smithi15186934-25 from 1964377 to 2319663 tid 2 2024-09-13T14:34:16.935 INFO:tasks.rados.rados.0.smithi151.stdout:3217: read oid 21 snap -1 2024-09-13T14:34:16.935 INFO:tasks.rados.rados.0.smithi151.stdout:3217: expect (ObjNum 937 snap 323 seq_num 937) 2024-09-13T14:34:16.935 INFO:tasks.rados.rados.0.smithi151.stdout:3218: write oid 28 current snap is 407 2024-09-13T14:34:16.935 INFO:tasks.rados.rados.0.smithi151.stdout:3213: finishing write tid 3 to smithi15186934-11 2024-09-13T14:34:16.935 INFO:tasks.rados.rados.0.smithi151.stdout:3213: finishing write tid 4 to smithi15186934-11 2024-09-13T14:34:16.935 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 11 v 790 (ObjNum 1207 snap 407 seq_num 1207) dirty exists 2024-09-13T14:34:16.935 INFO:tasks.rados.rados.0.smithi151.stdout:3213: left oid 11 (ObjNum 1207 snap 407 seq_num 1207) 2024-09-13T14:34:16.935 INFO:tasks.rados.rados.0.smithi151.stdout:3215: finishing rollback tid 0 to smithi15186934-46 2024-09-13T14:34:16.935 INFO:tasks.rados.rados.0.smithi151.stdout:3215: finishing rollback tid 1 to smithi15186934-46 2024-09-13T14:34:16.935 INFO:tasks.rados.rados.0.smithi151.stdout:3215: finishing rollback tid 2 to smithi15186934-46 2024-09-13T14:34:16.935 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 46 v 1165 (ObjNum 1135 snap 381 seq_num 1135) dirty exists 2024-09-13T14:34:16.935 INFO:tasks.rados.rados.0.smithi151.stdout:3216: finishing write tid 1 to smithi15186934-25 2024-09-13T14:34:16.936 INFO:tasks.rados.rados.0.smithi151.stdout:3218: seq_num 1210 ranges {736470=751417,2022859=313611} 2024-09-13T14:34:16.943 INFO:tasks.rados.rados.0.smithi151.stdout:3218: writing smithi15186934-28 from 736470 to 1487887 tid 1 2024-09-13T14:34:16.949 INFO:tasks.rados.rados.0.smithi151.stdout:3218: writing smithi15186934-28 from 2022859 to 2336470 tid 2 2024-09-13T14:34:16.949 INFO:tasks.rados.rados.0.smithi151.stdout:3213: done (4 left) 2024-09-13T14:34:16.950 INFO:tasks.rados.rados.0.smithi151.stdout:3215: done (3 left) 2024-09-13T14:34:16.950 INFO:tasks.rados.rados.0.smithi151.stdout:3219: rollback oid 36 current snap is 407 2024-09-13T14:34:16.950 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 36 to 404 2024-09-13T14:34:16.950 INFO:tasks.rados.rados.0.smithi151.stdout:3216: finishing write tid 2 to smithi15186934-25 2024-09-13T14:34:16.950 INFO:tasks.rados.rados.0.smithi151.stdout:3216: finishing write tid 3 to smithi15186934-25 2024-09-13T14:34:16.950 INFO:tasks.rados.rados.0.smithi151.stdout:3216: finishing write tid 4 to smithi15186934-25 2024-09-13T14:34:16.950 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 1527 (ObjNum 1209 snap 407 seq_num 1209) dirty exists 2024-09-13T14:34:16.950 INFO:tasks.rados.rados.0.smithi151.stdout:3216: left oid 25 (ObjNum 1209 snap 407 seq_num 1209) 2024-09-13T14:34:16.950 INFO:tasks.rados.rados.0.smithi151.stdout:3217: expect (ObjNum 937 snap 323 seq_num 937) 2024-09-13T14:34:17.073 INFO:tasks.rados.rados.0.smithi151.stdout:3218: finishing write tid 1 to smithi15186934-28 2024-09-13T14:34:17.073 INFO:tasks.rados.rados.0.smithi151.stdout:3219: finishing rollback tid 1 to smithi15186934-36 2024-09-13T14:34:17.073 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 36 v 647 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-13T14:34:17.073 INFO:tasks.rados.rados.0.smithi151.stdout:3218: finishing write tid 2 to smithi15186934-28 2024-09-13T14:34:17.073 INFO:tasks.rados.rados.0.smithi151.stdout:3218: finishing write tid 3 to smithi15186934-28 2024-09-13T14:34:17.073 INFO:tasks.rados.rados.0.smithi151.stdout:3216: done (3 left) 2024-09-13T14:34:17.074 INFO:tasks.rados.rados.0.smithi151.stdout:3217: done (2 left) 2024-09-13T14:34:17.074 INFO:tasks.rados.rados.0.smithi151.stdout:3219: done (1 left) 2024-09-13T14:34:17.074 INFO:tasks.rados.rados.0.smithi151.stdout:3220: read oid 36 snap -1 2024-09-13T14:34:17.074 INFO:tasks.rados.rados.0.smithi151.stdout:3220: expect deleted 2024-09-13T14:34:17.074 INFO:tasks.rados.rados.0.smithi151.stdout:3221: write oid 24 current snap is 407 2024-09-13T14:34:17.074 INFO:tasks.rados.rados.0.smithi151.stdout:3218: finishing write tid 4 to smithi15186934-28 2024-09-13T14:34:17.074 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 28 v 1169 (ObjNum 1210 snap 407 seq_num 1210) dirty exists 2024-09-13T14:34:17.074 INFO:tasks.rados.rados.0.smithi151.stdout:3218: left oid 28 (ObjNum 1210 snap 407 seq_num 1210) 2024-09-13T14:34:17.074 INFO:tasks.rados.rados.0.smithi151.stdout:3221: seq_num 1211 ranges {753277=426666,1681341=545753} 2024-09-13T14:34:17.079 INFO:tasks.rados.rados.0.smithi151.stdout:3221: writing smithi15186934-24 from 753277 to 1179943 tid 1 2024-09-13T14:34:17.085 INFO:tasks.rados.rados.0.smithi151.stdout:3221: writing smithi15186934-24 from 1681341 to 2227094 tid 2 2024-09-13T14:34:17.085 INFO:tasks.rados.rados.0.smithi151.stdout:3218: done (2 left) 2024-09-13T14:34:17.085 INFO:tasks.rados.rados.0.smithi151.stdout:3222: read oid 46 snap -1 2024-09-13T14:34:17.085 INFO:tasks.rados.rados.0.smithi151.stdout:3222: expect (ObjNum 1135 snap 381 seq_num 1135) 2024-09-13T14:34:17.085 INFO:tasks.rados.rados.0.smithi151.stdout:3223: snap_remove snap 383 2024-09-13T14:34:17.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:16 smithi184 ceph-mon[92527]: pgmap v915: 89 pgs: 89 active+clean; 305 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 3.7 MiB/s wr, 32 op/s 2024-09-13T14:34:17.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:16 smithi184 ceph-mon[92527]: osdmap e865: 8 total, 8 up, 8 in 2024-09-13T14:34:17.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:17.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:16 smithi151 ceph-mon[104450]: pgmap v915: 89 pgs: 89 active+clean; 305 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 3.7 MiB/s wr, 32 op/s 2024-09-13T14:34:17.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:16 smithi151 ceph-mon[104450]: osdmap e865: 8 total, 8 up, 8 in 2024-09-13T14:34:17.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:17.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:16 smithi151 ceph-mon[108966]: pgmap v915: 89 pgs: 89 active+clean; 305 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 3.7 MiB/s wr, 32 op/s 2024-09-13T14:34:17.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:16 smithi151 ceph-mon[108966]: osdmap e865: 8 total, 8 up, 8 in 2024-09-13T14:34:17.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:17.682 INFO:tasks.rados.rados.0.smithi151.stdout:3223: done (3 left) 2024-09-13T14:34:17.682 INFO:tasks.rados.rados.0.smithi151.stdout:3224: delete oid 44 current snap is 407 2024-09-13T14:34:17.682 INFO:tasks.rados.rados.0.smithi151.stdout:3221: finishing write tid 1 to smithi15186934-24 2024-09-13T14:34:17.682 INFO:tasks.rados.rados.0.smithi151.stdout:3221: finishing write tid 2 to smithi15186934-24 2024-09-13T14:34:17.682 INFO:tasks.rados.rados.0.smithi151.stdout:3221: finishing write tid 3 to smithi15186934-24 2024-09-13T14:34:17.682 INFO:tasks.rados.rados.0.smithi151.stdout:3222: expect (ObjNum 1135 snap 381 seq_num 1135) 2024-09-13T14:34:17.821 INFO:tasks.rados.rados.0.smithi151.stdout:3221: finishing write tid 4 to smithi15186934-24 2024-09-13T14:34:17.821 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 24 v 1852 (ObjNum 1211 snap 407 seq_num 1211) dirty exists 2024-09-13T14:34:17.821 INFO:tasks.rados.rados.0.smithi151.stdout:3221: left oid 24 (ObjNum 1211 snap 407 seq_num 1211) 2024-09-13T14:34:17.821 INFO:tasks.rados.rados.0.smithi151.stdout:3220: done (3 left) 2024-09-13T14:34:17.821 INFO:tasks.rados.rados.0.smithi151.stdout:3221: done (2 left) 2024-09-13T14:34:17.821 INFO:tasks.rados.rados.0.smithi151.stdout:3222: done (1 left) 2024-09-13T14:34:17.822 INFO:tasks.rados.rados.0.smithi151.stdout:3224: done (0 left) 2024-09-13T14:34:17.822 INFO:tasks.rados.rados.0.smithi151.stdout:3225: write oid 50 current snap is 407 2024-09-13T14:34:17.822 INFO:tasks.rados.rados.0.smithi151.stdout:3225: seq_num 1213 ranges {786891=577164,2081952=304939} 2024-09-13T14:34:17.828 INFO:tasks.rados.rados.0.smithi151.stdout:3225: writing smithi15186934-50 from 786891 to 1364055 tid 1 2024-09-13T14:34:17.837 INFO:tasks.rados.rados.0.smithi151.stdout:3225: writing smithi15186934-50 from 2081952 to 2386891 tid 2 2024-09-13T14:34:17.837 INFO:tasks.rados.rados.0.smithi151.stdout:3226: snap_create 2024-09-13T14:34:17.837 INFO:tasks.rados.rados.0.smithi151.stdout:3225: finishing write tid 1 to smithi15186934-50 2024-09-13T14:34:17.837 INFO:tasks.rados.rados.0.smithi151.stdout:3225: finishing write tid 2 to smithi15186934-50 2024-09-13T14:34:17.837 INFO:tasks.rados.rados.0.smithi151.stdout:3225: finishing write tid 3 to smithi15186934-50 2024-09-13T14:34:17.837 INFO:tasks.rados.rados.0.smithi151.stdout:3225: finishing write tid 4 to smithi15186934-50 2024-09-13T14:34:17.837 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 50 v 1326 (ObjNum 1213 snap 407 seq_num 1213) dirty exists 2024-09-13T14:34:17.837 INFO:tasks.rados.rados.0.smithi151.stdout:3225: left oid 50 (ObjNum 1213 snap 407 seq_num 1213) 2024-09-13T14:34:18.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:17 smithi184 ceph-mon[92527]: osdmap e866: 8 total, 8 up, 8 in 2024-09-13T14:34:18.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:17 smithi151 ceph-mon[108966]: osdmap e866: 8 total, 8 up, 8 in 2024-09-13T14:34:18.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:17 smithi151 ceph-mon[104450]: osdmap e866: 8 total, 8 up, 8 in 2024-09-13T14:34:18.683 INFO:tasks.rados.rados.0.smithi151.stdout:3225: done (1 left) 2024-09-13T14:34:18.684 INFO:tasks.rados.rados.0.smithi151.stdout:3226: done (0 left) 2024-09-13T14:34:18.684 INFO:tasks.rados.rados.0.smithi151.stdout:3227: rollback oid 11 current snap is 408 2024-09-13T14:34:18.684 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 11 to 363 2024-09-13T14:34:18.684 INFO:tasks.rados.rados.0.smithi151.stdout:3228: read oid 22 snap 384 2024-09-13T14:34:18.684 INFO:tasks.rados.rados.0.smithi151.stdout:3228: expect (ObjNum 1118 snap 373 seq_num 1118) 2024-09-13T14:34:18.684 INFO:tasks.rados.rados.0.smithi151.stdout:3229: snap_remove snap 377 2024-09-13T14:34:19.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:18 smithi184 ceph-mon[92527]: pgmap v918: 89 pgs: 89 active+clean; 311 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 1.6 MiB/s wr, 19 op/s 2024-09-13T14:34:19.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:18 smithi184 ceph-mon[92527]: osdmap e867: 8 total, 8 up, 8 in 2024-09-13T14:34:19.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:18 smithi151 ceph-mon[104450]: pgmap v918: 89 pgs: 89 active+clean; 311 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 1.6 MiB/s wr, 19 op/s 2024-09-13T14:34:19.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:18 smithi151 ceph-mon[104450]: osdmap e867: 8 total, 8 up, 8 in 2024-09-13T14:34:19.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:18 smithi151 ceph-mon[108966]: pgmap v918: 89 pgs: 89 active+clean; 311 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 1.6 MiB/s wr, 19 op/s 2024-09-13T14:34:19.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:18 smithi151 ceph-mon[108966]: osdmap e867: 8 total, 8 up, 8 in 2024-09-13T14:34:19.618 INFO:tasks.rados.rados.0.smithi151.stdout:3229: done (2 left) 2024-09-13T14:34:19.618 INFO:tasks.rados.rados.0.smithi151.stdout:3230: snap_remove snap 370 2024-09-13T14:34:20.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:19 smithi184 ceph-mon[92527]: osdmap e868: 8 total, 8 up, 8 in 2024-09-13T14:34:20.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:19 smithi184 ceph-mon[92527]: osdmap e869: 8 total, 8 up, 8 in 2024-09-13T14:34:20.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:19 smithi151 ceph-mon[108966]: osdmap e868: 8 total, 8 up, 8 in 2024-09-13T14:34:20.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:19 smithi151 ceph-mon[108966]: osdmap e869: 8 total, 8 up, 8 in 2024-09-13T14:34:20.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:19 smithi151 ceph-mon[104450]: osdmap e868: 8 total, 8 up, 8 in 2024-09-13T14:34:20.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:19 smithi151 ceph-mon[104450]: osdmap e869: 8 total, 8 up, 8 in 2024-09-13T14:34:20.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:34:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:34:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:34:20.621 INFO:tasks.rados.rados.0.smithi151.stdout:3227: finishing rollback tid 0 to smithi15186934-11 2024-09-13T14:34:20.621 INFO:tasks.rados.rados.0.smithi151.stdout:3227: finishing rollback tid 1 to smithi15186934-11 2024-09-13T14:34:20.621 INFO:tasks.rados.rados.0.smithi151.stdout:3227: finishing rollback tid 2 to smithi15186934-11 2024-09-13T14:34:20.621 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 11 v 800 (ObjNum 1054 snap 357 seq_num 1054) dirty exists 2024-09-13T14:34:20.621 INFO:tasks.rados.rados.0.smithi151.stdout:3228: expect (ObjNum 1118 snap 373 seq_num 1118) 2024-09-13T14:34:20.821 INFO:tasks.rados.rados.0.smithi151.stdout:3227: done (2 left) 2024-09-13T14:34:20.821 INFO:tasks.rados.rados.0.smithi151.stdout:3228: done (1 left) 2024-09-13T14:34:20.821 INFO:tasks.rados.rados.0.smithi151.stdout:3230: done (0 left) 2024-09-13T14:34:20.821 INFO:tasks.rados.rados.0.smithi151.stdout:3231: write oid 23 current snap is 408 2024-09-13T14:34:20.821 INFO:tasks.rados.rados.0.smithi151.stdout:3231: seq_num 1214 ranges {403698=652413,1740434=663264} 2024-09-13T14:34:20.827 INFO:tasks.rados.rados.0.smithi151.stdout:3231: writing smithi15186934-23 from 403698 to 1056111 tid 1 2024-09-13T14:34:20.834 INFO:tasks.rados.rados.0.smithi151.stdout:3231: writing smithi15186934-23 from 1740434 to 2403698 tid 2 2024-09-13T14:34:20.834 INFO:tasks.rados.rados.0.smithi151.stdout:3232: snap_create 2024-09-13T14:34:20.834 INFO:tasks.rados.rados.0.smithi151.stdout:3231: finishing write tid 1 to smithi15186934-23 2024-09-13T14:34:20.838 INFO:tasks.rados.rados.0.smithi151.stdout:3231: finishing write tid 2 to smithi15186934-23 2024-09-13T14:34:20.838 INFO:tasks.rados.rados.0.smithi151.stdout:3231: finishing write tid 3 to smithi15186934-23 2024-09-13T14:34:20.838 INFO:tasks.rados.rados.0.smithi151.stdout:3231: finishing write tid 4 to smithi15186934-23 2024-09-13T14:34:20.838 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 23 v 1461 (ObjNum 1214 snap 408 seq_num 1214) dirty exists 2024-09-13T14:34:20.838 INFO:tasks.rados.rados.0.smithi151.stdout:3231: left oid 23 (ObjNum 1214 snap 408 seq_num 1214) 2024-09-13T14:34:21.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:20 smithi184 ceph-mon[92527]: pgmap v921: 89 pgs: 89 active+clean; 313 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 1.4 MiB/s wr, 19 op/s 2024-09-13T14:34:21.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:20 smithi184 ceph-mon[92527]: osdmap e870: 8 total, 8 up, 8 in 2024-09-13T14:34:21.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:20 smithi151 ceph-mon[104450]: pgmap v921: 89 pgs: 89 active+clean; 313 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 1.4 MiB/s wr, 19 op/s 2024-09-13T14:34:21.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:20 smithi151 ceph-mon[104450]: osdmap e870: 8 total, 8 up, 8 in 2024-09-13T14:34:21.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:20 smithi151 ceph-mon[108966]: pgmap v921: 89 pgs: 89 active+clean; 313 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 1.4 MiB/s wr, 19 op/s 2024-09-13T14:34:21.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:20 smithi151 ceph-mon[108966]: osdmap e870: 8 total, 8 up, 8 in 2024-09-13T14:34:21.708 INFO:tasks.rados.rados.0.smithi151.stdout:3231: done (1 left) 2024-09-13T14:34:21.708 INFO:tasks.rados.rados.0.smithi151.stdout:3232: done (0 left) 2024-09-13T14:34:21.708 INFO:tasks.rados.rados.0.smithi151.stdout:3233: write oid 2 current snap is 409 2024-09-13T14:34:21.708 INFO:tasks.rados.rados.0.smithi151.stdout:3233: seq_num 1215 ranges {420505=727662,1798916=553091} 2024-09-13T14:34:21.713 INFO:tasks.rados.rados.0.smithi151.stdout:3233: writing smithi15186934-2 from 420505 to 1148167 tid 1 2024-09-13T14:34:21.721 INFO:tasks.rados.rados.0.smithi151.stdout:3233: writing smithi15186934-2 from 1798916 to 2352007 tid 2 2024-09-13T14:34:21.722 INFO:tasks.rados.rados.0.smithi151.stdout:3234: read oid 41 snap 402 2024-09-13T14:34:21.722 INFO:tasks.rados.rados.0.smithi151.stdout:3234: expect deleted 2024-09-13T14:34:21.722 INFO:tasks.rados.rados.0.smithi151.stdout:3233: finishing write tid 1 to smithi15186934-2 2024-09-13T14:34:21.722 INFO:tasks.rados.rados.0.smithi151.stdout:3235: read oid 12 snap -1 2024-09-13T14:34:21.722 INFO:tasks.rados.rados.0.smithi151.stdout:3235: expect (ObjNum 1170 snap 397 seq_num 1170) 2024-09-13T14:34:21.722 INFO:tasks.rados.rados.0.smithi151.stdout:3236: snap_remove snap 391 2024-09-13T14:34:22.715 INFO:tasks.rados.rados.0.smithi151.stdout:3236: done (3 left) 2024-09-13T14:34:22.715 INFO:tasks.rados.rados.0.smithi151.stdout:3237: write oid 23 current snap is 409 2024-09-13T14:34:22.715 INFO:tasks.rados.rados.0.smithi151.stdout:3237: seq_num 1216 ranges {437312=402911,1341045=696749} 2024-09-13T14:34:22.720 INFO:tasks.rados.rados.0.smithi151.stdout:3237: writing smithi15186934-23 from 437312 to 840223 tid 1 2024-09-13T14:34:22.727 INFO:tasks.rados.rados.0.smithi151.stdout:3237: writing smithi15186934-23 from 1341045 to 2037794 tid 2 2024-09-13T14:34:22.728 INFO:tasks.rados.rados.0.smithi151.stdout:3238: delete oid 27 current snap is 409 2024-09-13T14:34:22.728 INFO:tasks.rados.rados.0.smithi151.stdout:3233: finishing write tid 2 to smithi15186934-2 2024-09-13T14:34:22.728 INFO:tasks.rados.rados.0.smithi151.stdout:3233: finishing write tid 3 to smithi15186934-2 2024-09-13T14:34:22.728 INFO:tasks.rados.rados.0.smithi151.stdout:3233: finishing write tid 4 to smithi15186934-2 2024-09-13T14:34:22.728 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 2 v 1195 (ObjNum 1215 snap 409 seq_num 1215) dirty exists 2024-09-13T14:34:22.728 INFO:tasks.rados.rados.0.smithi151.stdout:3233: left oid 2 (ObjNum 1215 snap 409 seq_num 1215) 2024-09-13T14:34:22.728 INFO:tasks.rados.rados.0.smithi151.stdout:3235: expect (ObjNum 1170 snap 397 seq_num 1170) 2024-09-13T14:34:22.850 INFO:tasks.rados.rados.0.smithi151.stdout:3237: finishing write tid 1 to smithi15186934-23 2024-09-13T14:34:22.850 INFO:tasks.rados.rados.0.smithi151.stdout:3237: finishing write tid 2 to smithi15186934-23 2024-09-13T14:34:22.850 INFO:tasks.rados.rados.0.smithi151.stdout:3237: finishing write tid 3 to smithi15186934-23 2024-09-13T14:34:22.850 INFO:tasks.rados.rados.0.smithi151.stdout:3237: finishing write tid 4 to smithi15186934-23 2024-09-13T14:34:22.850 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 23 v 1473 (ObjNum 1216 snap 409 seq_num 1216) dirty exists 2024-09-13T14:34:22.850 INFO:tasks.rados.rados.0.smithi151.stdout:3237: left oid 23 (ObjNum 1216 snap 409 seq_num 1216) 2024-09-13T14:34:22.850 INFO:tasks.rados.rados.0.smithi151.stdout:3233: done (4 left) 2024-09-13T14:34:22.850 INFO:tasks.rados.rados.0.smithi151.stdout:3234: done (3 left) 2024-09-13T14:34:22.850 INFO:tasks.rados.rados.0.smithi151.stdout:3235: done (2 left) 2024-09-13T14:34:22.851 INFO:tasks.rados.rados.0.smithi151.stdout:3237: done (1 left) 2024-09-13T14:34:22.851 INFO:tasks.rados.rados.0.smithi151.stdout:3238: done (0 left) 2024-09-13T14:34:22.851 INFO:tasks.rados.rados.0.smithi151.stdout:3239: snap_remove snap 394 2024-09-13T14:34:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:22 smithi184 ceph-mon[92527]: pgmap v924: 89 pgs: 89 active+clean; 322 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.3 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-09-13T14:34:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:22 smithi184 ceph-mon[92527]: osdmap e871: 8 total, 8 up, 8 in 2024-09-13T14:34:23.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:22 smithi151 ceph-mon[104450]: pgmap v924: 89 pgs: 89 active+clean; 322 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.3 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-09-13T14:34:23.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:22 smithi151 ceph-mon[104450]: osdmap e871: 8 total, 8 up, 8 in 2024-09-13T14:34:23.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:22 smithi151 ceph-mon[108966]: pgmap v924: 89 pgs: 89 active+clean; 322 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.3 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-09-13T14:34:23.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:22 smithi151 ceph-mon[108966]: osdmap e871: 8 total, 8 up, 8 in 2024-09-13T14:34:23.727 INFO:tasks.rados.rados.0.smithi151.stdout:3239: done (0 left) 2024-09-13T14:34:23.727 INFO:tasks.rados.rados.0.smithi151.stdout:3240: snap_create 2024-09-13T14:34:24.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:23 smithi184 ceph-mon[92527]: osdmap e872: 8 total, 8 up, 8 in 2024-09-13T14:34:24.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:23 smithi151 ceph-mon[104450]: osdmap e872: 8 total, 8 up, 8 in 2024-09-13T14:34:24.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:23 smithi151 ceph-mon[108966]: osdmap e872: 8 total, 8 up, 8 in 2024-09-13T14:34:24.623 INFO:tasks.rados.rados.0.smithi151.stdout:3240: done (0 left) 2024-09-13T14:34:24.623 INFO:tasks.rados.rados.0.smithi151.stdout:3241: write oid 30 current snap is 410 2024-09-13T14:34:24.623 INFO:tasks.rados.rados.0.smithi151.stdout:3241: seq_num 1218 ranges {470926=669762,1574362=700418} 2024-09-13T14:34:24.629 INFO:tasks.rados.rados.0.smithi151.stdout:3241: writing smithi15186934-30 from 470926 to 1140688 tid 1 2024-09-13T14:34:24.636 INFO:tasks.rados.rados.0.smithi151.stdout:3241: writing smithi15186934-30 from 1574362 to 2274780 tid 2 2024-09-13T14:34:24.637 INFO:tasks.rados.rados.0.smithi151.stdout:3242: read oid 44 snap 384 2024-09-13T14:34:24.637 INFO:tasks.rados.rados.0.smithi151.stdout:3242: expect (ObjNum 1125 snap 376 seq_num 1125) 2024-09-13T14:34:24.637 INFO:tasks.rados.rados.0.smithi151.stdout:3243: write oid 24 current snap is 410 2024-09-13T14:34:24.638 INFO:tasks.rados.rados.0.smithi151.stdout:3243: seq_num 1219 ranges {487733=745011,1632844=560429} 2024-09-13T14:34:24.644 INFO:tasks.rados.rados.0.smithi151.stdout:3243: writing smithi15186934-24 from 487733 to 1232744 tid 1 2024-09-13T14:34:24.649 INFO:tasks.rados.rados.0.smithi151.stdout:3243: writing smithi15186934-24 from 1632844 to 2193273 tid 2 2024-09-13T14:34:24.650 INFO:tasks.rados.rados.0.smithi151.stdout:3244: read oid 36 snap -1 2024-09-13T14:34:24.650 INFO:tasks.rados.rados.0.smithi151.stdout:3244: expect deleted 2024-09-13T14:34:24.650 INFO:tasks.rados.rados.0.smithi151.stdout:3245: delete oid 16 current snap is 410 2024-09-13T14:34:24.650 INFO:tasks.rados.rados.0.smithi151.stdout:3241: finishing write tid 1 to smithi15186934-30 2024-09-13T14:34:24.651 INFO:tasks.rados.rados.0.smithi151.stdout:3241: finishing write tid 2 to smithi15186934-30 2024-09-13T14:34:24.651 INFO:tasks.rados.rados.0.smithi151.stdout:3241: finishing write tid 3 to smithi15186934-30 2024-09-13T14:34:24.651 INFO:tasks.rados.rados.0.smithi151.stdout:3241: finishing write tid 4 to smithi15186934-30 2024-09-13T14:34:24.651 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 30 v 1259 (ObjNum 1218 snap 410 seq_num 1218) dirty exists 2024-09-13T14:34:24.651 INFO:tasks.rados.rados.0.smithi151.stdout:3241: left oid 30 (ObjNum 1218 snap 410 seq_num 1218) 2024-09-13T14:34:24.652 INFO:tasks.rados.rados.0.smithi151.stdout:3242: expect (ObjNum 1125 snap 376 seq_num 1125) 2024-09-13T14:34:24.781 INFO:tasks.rados.rados.0.smithi151.stdout:3243: finishing write tid 1 to smithi15186934-24 2024-09-13T14:34:24.781 INFO:tasks.rados.rados.0.smithi151.stdout:3243: finishing write tid 2 to smithi15186934-24 2024-09-13T14:34:24.781 INFO:tasks.rados.rados.0.smithi151.stdout:3243: finishing write tid 3 to smithi15186934-24 2024-09-13T14:34:24.781 INFO:tasks.rados.rados.0.smithi151.stdout:3243: finishing write tid 4 to smithi15186934-24 2024-09-13T14:34:24.781 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 24 v 1868 (ObjNum 1219 snap 410 seq_num 1219) dirty exists 2024-09-13T14:34:24.782 INFO:tasks.rados.rados.0.smithi151.stdout:3243: left oid 24 (ObjNum 1219 snap 410 seq_num 1219) 2024-09-13T14:34:24.782 INFO:tasks.rados.rados.0.smithi151.stdout:3241: done (4 left) 2024-09-13T14:34:24.782 INFO:tasks.rados.rados.0.smithi151.stdout:3242: done (3 left) 2024-09-13T14:34:24.782 INFO:tasks.rados.rados.0.smithi151.stdout:3243: done (2 left) 2024-09-13T14:34:24.782 INFO:tasks.rados.rados.0.smithi151.stdout:3244: done (1 left) 2024-09-13T14:34:24.782 INFO:tasks.rados.rados.0.smithi151.stdout:3245: done (0 left) 2024-09-13T14:34:24.783 INFO:tasks.rados.rados.0.smithi151.stdout:3246: snap_remove snap 364 2024-09-13T14:34:24.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:24 smithi151 ceph-mon[104450]: pgmap v927: 89 pgs: 89 active+clean; 318 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:34:24.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:24 smithi151 ceph-mon[104450]: osdmap e873: 8 total, 8 up, 8 in 2024-09-13T14:34:24.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:24 smithi151 ceph-mon[104450]: osdmap e874: 8 total, 8 up, 8 in 2024-09-13T14:34:24.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:34:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:34:24] "GET /metrics HTTP/1.1" 200 36511 "" "Prometheus/2.51.0" 2024-09-13T14:34:24.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:24 smithi151 ceph-mon[108966]: pgmap v927: 89 pgs: 89 active+clean; 318 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:34:24.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:24 smithi151 ceph-mon[108966]: osdmap e873: 8 total, 8 up, 8 in 2024-09-13T14:34:24.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:24 smithi151 ceph-mon[108966]: osdmap e874: 8 total, 8 up, 8 in 2024-09-13T14:34:25.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:24 smithi184 ceph-mon[92527]: pgmap v927: 89 pgs: 89 active+clean; 318 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:34:25.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:24 smithi184 ceph-mon[92527]: osdmap e873: 8 total, 8 up, 8 in 2024-09-13T14:34:25.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:24 smithi184 ceph-mon[92527]: osdmap e874: 8 total, 8 up, 8 in 2024-09-13T14:34:25.729 INFO:tasks.rados.rados.0.smithi151.stdout:3246: done (0 left) 2024-09-13T14:34:25.729 INFO:tasks.rados.rados.0.smithi151.stdout:3247: write oid 44 current snap is 410 2024-09-13T14:34:25.729 INFO:tasks.rados.rados.0.smithi151.stdout:3247: seq_num 1221 ranges {521347=495509,1633455=564098} 2024-09-13T14:34:25.736 INFO:tasks.rados.rados.0.smithi151.stdout:3247: writing smithi15186934-44 from 521347 to 1016856 tid 1 2024-09-13T14:34:25.743 INFO:tasks.rados.rados.0.smithi151.stdout:3247: writing smithi15186934-44 from 1633455 to 2197553 tid 2 2024-09-13T14:34:25.743 INFO:tasks.rados.rados.0.smithi151.stdout:3248: write oid 17 current snap is 410 2024-09-13T14:34:25.744 INFO:tasks.rados.rados.0.smithi151.stdout:3248: seq_num 1222 ranges {538154=570758,1691937=707756} 2024-09-13T14:34:25.749 INFO:tasks.rados.rados.0.smithi151.stdout:3248: writing smithi15186934-17 from 538154 to 1108912 tid 1 2024-09-13T14:34:25.757 INFO:tasks.rados.rados.0.smithi151.stdout:3248: writing smithi15186934-17 from 1691937 to 2399693 tid 2 2024-09-13T14:34:25.757 INFO:tasks.rados.rados.0.smithi151.stdout:3249: read oid 28 snap -1 2024-09-13T14:34:25.757 INFO:tasks.rados.rados.0.smithi151.stdout:3249: expect (ObjNum 1210 snap 407 seq_num 1210) 2024-09-13T14:34:25.757 INFO:tasks.rados.rados.0.smithi151.stdout:3250: write oid 21 current snap is 410 2024-09-13T14:34:25.757 INFO:tasks.rados.rados.0.smithi151.stdout:3247: finishing write tid 1 to smithi15186934-44 2024-09-13T14:34:25.757 INFO:tasks.rados.rados.0.smithi151.stdout:3247: finishing write tid 2 to smithi15186934-44 2024-09-13T14:34:25.757 INFO:tasks.rados.rados.0.smithi151.stdout:3250: seq_num 1223 ranges {554961=646007,1750419=451414} 2024-09-13T14:34:25.763 INFO:tasks.rados.rados.0.smithi151.stdout:3250: writing smithi15186934-21 from 554961 to 1200968 tid 1 2024-09-13T14:34:25.769 INFO:tasks.rados.rados.0.smithi151.stdout:3250: writing smithi15186934-21 from 1750419 to 2201833 tid 2 2024-09-13T14:34:25.770 INFO:tasks.rados.rados.0.smithi151.stdout:3251: write oid 8 current snap is 410 2024-09-13T14:34:25.770 INFO:tasks.rados.rados.0.smithi151.stdout:3247: finishing write tid 3 to smithi15186934-44 2024-09-13T14:34:25.770 INFO:tasks.rados.rados.0.smithi151.stdout:3247: finishing write tid 4 to smithi15186934-44 2024-09-13T14:34:25.770 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 1006 (ObjNum 1221 snap 410 seq_num 1221) dirty exists 2024-09-13T14:34:25.770 INFO:tasks.rados.rados.0.smithi151.stdout:3247: left oid 44 (ObjNum 1221 snap 410 seq_num 1221) 2024-09-13T14:34:25.770 INFO:tasks.rados.rados.0.smithi151.stdout:3248: finishing write tid 1 to smithi15186934-17 2024-09-13T14:34:25.770 INFO:tasks.rados.rados.0.smithi151.stdout:3248: finishing write tid 2 to smithi15186934-17 2024-09-13T14:34:25.770 INFO:tasks.rados.rados.0.smithi151.stdout:3248: finishing write tid 3 to smithi15186934-17 2024-09-13T14:34:25.770 INFO:tasks.rados.rados.0.smithi151.stdout:3248: finishing write tid 4 to smithi15186934-17 2024-09-13T14:34:25.770 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 1879 (ObjNum 1222 snap 410 seq_num 1222) dirty exists 2024-09-13T14:34:25.770 INFO:tasks.rados.rados.0.smithi151.stdout:3248: left oid 17 (ObjNum 1222 snap 410 seq_num 1222) 2024-09-13T14:34:25.771 INFO:tasks.rados.rados.0.smithi151.stdout:3249: expect (ObjNum 1210 snap 407 seq_num 1210) 2024-09-13T14:34:25.883 INFO:tasks.rados.rados.0.smithi151.stdout:3250: finishing write tid 1 to smithi15186934-21 2024-09-13T14:34:25.883 INFO:tasks.rados.rados.0.smithi151.stdout:3250: finishing write tid 2 to smithi15186934-21 2024-09-13T14:34:25.883 INFO:tasks.rados.rados.0.smithi151.stdout:3250: finishing write tid 3 to smithi15186934-21 2024-09-13T14:34:25.883 INFO:tasks.rados.rados.0.smithi151.stdout:3250: finishing write tid 4 to smithi15186934-21 2024-09-13T14:34:25.883 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 21 v 645 (ObjNum 1223 snap 410 seq_num 1223) dirty exists 2024-09-13T14:34:25.883 INFO:tasks.rados.rados.0.smithi151.stdout:3250: left oid 21 (ObjNum 1223 snap 410 seq_num 1223) 2024-09-13T14:34:25.884 INFO:tasks.rados.rados.0.smithi151.stdout:3251: seq_num 1224 ranges {571768=437609,1808901=711425} 2024-09-13T14:34:25.888 INFO:tasks.rados.rados.0.smithi151.stdout:3251: writing smithi15186934-8 from 571768 to 1009377 tid 1 2024-09-13T14:34:25.897 INFO:tasks.rados.rados.0.smithi151.stdout:3251: writing smithi15186934-8 from 1808901 to 2520326 tid 2 2024-09-13T14:34:25.897 INFO:tasks.rados.rados.0.smithi151.stdout:3247: done (4 left) 2024-09-13T14:34:25.897 INFO:tasks.rados.rados.0.smithi151.stdout:3248: done (3 left) 2024-09-13T14:34:25.897 INFO:tasks.rados.rados.0.smithi151.stdout:3249: done (2 left) 2024-09-13T14:34:25.897 INFO:tasks.rados.rados.0.smithi151.stdout:3250: done (1 left) 2024-09-13T14:34:25.897 INFO:tasks.rados.rados.0.smithi151.stdout:3252: delete oid 17 current snap is 410 2024-09-13T14:34:25.897 INFO:tasks.rados.rados.0.smithi151.stdout:3251: finishing write tid 1 to smithi15186934-8 2024-09-13T14:34:25.899 INFO:tasks.rados.rados.0.smithi151.stdout:3252: done (1 left) 2024-09-13T14:34:25.899 INFO:tasks.rados.rados.0.smithi151.stdout:3253: write oid 44 current snap is 410 2024-09-13T14:34:25.899 INFO:tasks.rados.rados.0.smithi151.stdout:3253: seq_num 1226 ranges {605382=588107,1925865=679517} 2024-09-13T14:34:25.905 INFO:tasks.rados.rados.0.smithi151.stdout:3253: writing smithi15186934-44 from 605382 to 1193489 tid 1 2024-09-13T14:34:25.913 INFO:tasks.rados.rados.0.smithi151.stdout:3253: writing smithi15186934-44 from 1925865 to 2605382 tid 2 2024-09-13T14:34:25.913 INFO:tasks.rados.rados.0.smithi151.stdout:3254: snap_remove snap 405 2024-09-13T14:34:26.735 INFO:tasks.rados.rados.0.smithi151.stdout:3254: done (2 left) 2024-09-13T14:34:26.736 INFO:tasks.rados.rados.0.smithi151.stdout:3255: write oid 40 current snap is 410 2024-09-13T14:34:26.736 INFO:tasks.rados.rados.0.smithi151.stdout:3255: seq_num 1227 ranges {622189=663356,1984347=458752} 2024-09-13T14:34:26.743 INFO:tasks.rados.rados.0.smithi151.stdout:3255: writing smithi15186934-40 from 622189 to 1285545 tid 1 2024-09-13T14:34:26.750 INFO:tasks.rados.rados.0.smithi151.stdout:3255: writing smithi15186934-40 from 1984347 to 2443099 tid 2 2024-09-13T14:34:26.750 INFO:tasks.rados.rados.0.smithi151.stdout:3256: snap_remove snap 408 2024-09-13T14:34:27.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:26 smithi184 ceph-mon[92527]: pgmap v930: 89 pgs: 89 active+clean; 316 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 850 KiB/s wr, 5 op/s 2024-09-13T14:34:27.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:26 smithi184 ceph-mon[92527]: osdmap e875: 8 total, 8 up, 8 in 2024-09-13T14:34:27.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:26 smithi151 ceph-mon[104450]: pgmap v930: 89 pgs: 89 active+clean; 316 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 850 KiB/s wr, 5 op/s 2024-09-13T14:34:27.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:26 smithi151 ceph-mon[104450]: osdmap e875: 8 total, 8 up, 8 in 2024-09-13T14:34:27.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:26 smithi151 ceph-mon[108966]: pgmap v930: 89 pgs: 89 active+clean; 316 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 850 KiB/s wr, 5 op/s 2024-09-13T14:34:27.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:26 smithi151 ceph-mon[108966]: osdmap e875: 8 total, 8 up, 8 in 2024-09-13T14:34:27.742 INFO:tasks.rados.rados.0.smithi151.stdout:3256: done (3 left) 2024-09-13T14:34:27.742 INFO:tasks.rados.rados.0.smithi151.stdout:3257: read oid 37 snap -1 2024-09-13T14:34:27.742 INFO:tasks.rados.rados.0.smithi151.stdout:3257: expect (ObjNum 1182 snap 402 seq_num 1182) 2024-09-13T14:34:27.742 INFO:tasks.rados.rados.0.smithi151.stdout:3258: write oid 47 current snap is 410 2024-09-13T14:34:27.742 INFO:tasks.rados.rados.0.smithi151.stdout:3251: finishing write tid 2 to smithi15186934-8 2024-09-13T14:34:27.742 INFO:tasks.rados.rados.0.smithi151.stdout:3251: finishing write tid 3 to smithi15186934-8 2024-09-13T14:34:27.742 INFO:tasks.rados.rados.0.smithi151.stdout:3251: finishing write tid 4 to smithi15186934-8 2024-09-13T14:34:27.742 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 1009 (ObjNum 1224 snap 410 seq_num 1224) dirty exists 2024-09-13T14:34:27.742 INFO:tasks.rados.rados.0.smithi151.stdout:3251: left oid 8 (ObjNum 1224 snap 410 seq_num 1224) 2024-09-13T14:34:27.742 INFO:tasks.rados.rados.0.smithi151.stdout:3253: finishing write tid 1 to smithi15186934-44 2024-09-13T14:34:27.743 INFO:tasks.rados.rados.0.smithi151.stdout:3253: finishing write tid 2 to smithi15186934-44 2024-09-13T14:34:27.743 INFO:tasks.rados.rados.0.smithi151.stdout:3253: finishing write tid 3 to smithi15186934-44 2024-09-13T14:34:27.743 INFO:tasks.rados.rados.0.smithi151.stdout:3258: seq_num 1228 ranges {638996=738605,1926476=712520} 2024-09-13T14:34:27.750 INFO:tasks.rados.rados.0.smithi151.stdout:3258: writing smithi15186934-47 from 638996 to 1377601 tid 1 2024-09-13T14:34:27.758 INFO:tasks.rados.rados.0.smithi151.stdout:3258: writing smithi15186934-47 from 1926476 to 2638996 tid 2 2024-09-13T14:34:27.758 INFO:tasks.rados.rados.0.smithi151.stdout:3251: done (4 left) 2024-09-13T14:34:27.758 INFO:tasks.rados.rados.0.smithi151.stdout:3259: rollback oid 12 current snap is 410 2024-09-13T14:34:27.758 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 12 to 363 2024-09-13T14:34:27.758 INFO:tasks.rados.rados.0.smithi151.stdout:3253: finishing write tid 4 to smithi15186934-44 2024-09-13T14:34:27.758 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 1012 (ObjNum 1226 snap 410 seq_num 1226) dirty exists 2024-09-13T14:34:27.759 INFO:tasks.rados.rados.0.smithi151.stdout:3253: left oid 44 (ObjNum 1226 snap 410 seq_num 1226) 2024-09-13T14:34:27.759 INFO:tasks.rados.rados.0.smithi151.stdout:3255: finishing write tid 1 to smithi15186934-40 2024-09-13T14:34:27.759 INFO:tasks.rados.rados.0.smithi151.stdout:3255: finishing write tid 2 to smithi15186934-40 2024-09-13T14:34:27.759 INFO:tasks.rados.rados.0.smithi151.stdout:3255: finishing write tid 3 to smithi15186934-40 2024-09-13T14:34:27.759 INFO:tasks.rados.rados.0.smithi151.stdout:3255: finishing write tid 4 to smithi15186934-40 2024-09-13T14:34:27.759 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 40 v 1267 (ObjNum 1227 snap 410 seq_num 1227) dirty exists 2024-09-13T14:34:27.759 INFO:tasks.rados.rados.0.smithi151.stdout:3255: left oid 40 (ObjNum 1227 snap 410 seq_num 1227) 2024-09-13T14:34:27.759 INFO:tasks.rados.rados.0.smithi151.stdout:3253: done (4 left) 2024-09-13T14:34:27.759 INFO:tasks.rados.rados.0.smithi151.stdout:3255: done (3 left) 2024-09-13T14:34:27.759 INFO:tasks.rados.rados.0.smithi151.stdout:3260: read oid 26 snap -1 2024-09-13T14:34:27.759 INFO:tasks.rados.rados.0.smithi151.stdout:3260: expect deleted 2024-09-13T14:34:27.759 INFO:tasks.rados.rados.0.smithi151.stdout:3261: snap_create 2024-09-13T14:34:27.759 INFO:tasks.rados.rados.0.smithi151.stdout:3257: expect (ObjNum 1182 snap 402 seq_num 1182) 2024-09-13T14:34:27.978 INFO:tasks.rados.rados.0.smithi151.stdout:3258: finishing write tid 1 to smithi15186934-47 2024-09-13T14:34:27.978 INFO:tasks.rados.rados.0.smithi151.stdout:3258: finishing write tid 2 to smithi15186934-47 2024-09-13T14:34:27.978 INFO:tasks.rados.rados.0.smithi151.stdout:3259: finishing rollback tid 0 to smithi15186934-12 2024-09-13T14:34:27.978 INFO:tasks.rados.rados.0.smithi151.stdout:3258: finishing write tid 3 to smithi15186934-47 2024-09-13T14:34:27.978 INFO:tasks.rados.rados.0.smithi151.stdout:3258: finishing write tid 4 to smithi15186934-47 2024-09-13T14:34:27.979 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 1494 (ObjNum 1228 snap 410 seq_num 1228) dirty exists 2024-09-13T14:34:27.979 INFO:tasks.rados.rados.0.smithi151.stdout:3258: left oid 47 (ObjNum 1228 snap 410 seq_num 1228) 2024-09-13T14:34:27.979 INFO:tasks.rados.rados.0.smithi151.stdout:3259: finishing rollback tid 1 to smithi15186934-12 2024-09-13T14:34:27.979 INFO:tasks.rados.rados.0.smithi151.stdout:3259: finishing rollback tid 2 to smithi15186934-12 2024-09-13T14:34:27.979 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 12 v 680 (ObjNum 1068 snap 363 seq_num 1068) dirty exists 2024-09-13T14:34:28.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:27 smithi184 ceph-mon[92527]: osdmap e876: 8 total, 8 up, 8 in 2024-09-13T14:34:28.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:27 smithi151 ceph-mon[104450]: osdmap e876: 8 total, 8 up, 8 in 2024-09-13T14:34:28.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:27 smithi151 ceph-mon[108966]: osdmap e876: 8 total, 8 up, 8 in 2024-09-13T14:34:28.747 INFO:tasks.rados.rados.0.smithi151.stdout:3257: done (4 left) 2024-09-13T14:34:28.747 INFO:tasks.rados.rados.0.smithi151.stdout:3258: done (3 left) 2024-09-13T14:34:28.747 INFO:tasks.rados.rados.0.smithi151.stdout:3259: done (2 left) 2024-09-13T14:34:28.747 INFO:tasks.rados.rados.0.smithi151.stdout:3260: done (1 left) 2024-09-13T14:34:28.747 INFO:tasks.rados.rados.0.smithi151.stdout:3261: done (0 left) 2024-09-13T14:34:28.747 INFO:tasks.rados.rados.0.smithi151.stdout:3262: snap_create 2024-09-13T14:34:29.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:28 smithi184 ceph-mon[92527]: pgmap v933: 89 pgs: 89 active+clean; 315 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.8 MiB/s rd, 1.8 MiB/s wr, 12 op/s 2024-09-13T14:34:29.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:28 smithi184 ceph-mon[92527]: osdmap e877: 8 total, 8 up, 8 in 2024-09-13T14:34:29.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:28 smithi151 ceph-mon[104450]: pgmap v933: 89 pgs: 89 active+clean; 315 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.8 MiB/s rd, 1.8 MiB/s wr, 12 op/s 2024-09-13T14:34:29.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:28 smithi151 ceph-mon[104450]: osdmap e877: 8 total, 8 up, 8 in 2024-09-13T14:34:29.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:28 smithi151 ceph-mon[108966]: pgmap v933: 89 pgs: 89 active+clean; 315 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.8 MiB/s rd, 1.8 MiB/s wr, 12 op/s 2024-09-13T14:34:29.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:28 smithi151 ceph-mon[108966]: osdmap e877: 8 total, 8 up, 8 in 2024-09-13T14:34:29.624 INFO:tasks.rados.rados.0.smithi151.stdout:3262: done (0 left) 2024-09-13T14:34:29.624 INFO:tasks.rados.rados.0.smithi151.stdout:3263: read oid 12 snap -1 2024-09-13T14:34:29.624 INFO:tasks.rados.rados.0.smithi151.stdout:3263: expect (ObjNum 1068 snap 363 seq_num 1068) 2024-09-13T14:34:29.625 INFO:tasks.rados.rados.0.smithi151.stdout:3264: write oid 27 current snap is 412 2024-09-13T14:34:29.625 INFO:tasks.rados.rados.0.smithi151.stdout:3264: seq_num 1229 ranges {655803=413854,1584958=462421} 2024-09-13T14:34:29.632 INFO:tasks.rados.rados.0.smithi151.stdout:3264: writing smithi15186934-27 from 655803 to 1069657 tid 1 2024-09-13T14:34:29.636 INFO:tasks.rados.rados.0.smithi151.stdout:3264: writing smithi15186934-27 from 1584958 to 2047379 tid 2 2024-09-13T14:34:29.636 INFO:tasks.rados.rados.0.smithi151.stdout:3265: rollback oid 3 current snap is 412 2024-09-13T14:34:29.636 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 3 to 396 2024-09-13T14:34:29.636 INFO:tasks.rados.rados.0.smithi151.stdout:3264: finishing write tid 1 to smithi15186934-27 2024-09-13T14:34:29.636 INFO:tasks.rados.rados.0.smithi151.stdout:3266: rollback oid 43 current snap is 412 2024-09-13T14:34:29.636 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 43 to 403 2024-09-13T14:34:29.636 INFO:tasks.rados.rados.0.smithi151.stdout:3267: snap_remove snap 404 2024-09-13T14:34:30.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:29 smithi184 ceph-mon[92527]: osdmap e878: 8 total, 8 up, 8 in 2024-09-13T14:34:30.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:29 smithi184 ceph-mon[92527]: osdmap e879: 8 total, 8 up, 8 in 2024-09-13T14:34:30.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:29 smithi151 ceph-mon[104450]: osdmap e878: 8 total, 8 up, 8 in 2024-09-13T14:34:30.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:29 smithi151 ceph-mon[104450]: osdmap e879: 8 total, 8 up, 8 in 2024-09-13T14:34:30.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:29 smithi151 ceph-mon[108966]: osdmap e878: 8 total, 8 up, 8 in 2024-09-13T14:34:30.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:29 smithi151 ceph-mon[108966]: osdmap e879: 8 total, 8 up, 8 in 2024-09-13T14:34:30.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:34:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:34:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:34:30.626 INFO:tasks.rados.rados.0.smithi151.stdout:3267: done (4 left) 2024-09-13T14:34:30.626 INFO:tasks.rados.rados.0.smithi151.stdout:3268: read oid 38 snap -1 2024-09-13T14:34:30.626 INFO:tasks.rados.rados.0.smithi151.stdout:3268: expect (ObjNum 1202 snap 407 seq_num 1202) 2024-09-13T14:34:30.627 INFO:tasks.rados.rados.0.smithi151.stdout:3269: read oid 1 snap -1 2024-09-13T14:34:30.627 INFO:tasks.rados.rados.0.smithi151.stdout:3269: expect (ObjNum 1204 snap 407 seq_num 1204) 2024-09-13T14:34:30.627 INFO:tasks.rados.rados.0.smithi151.stdout:3270: rollback oid 29 current snap is 412 2024-09-13T14:34:30.627 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 29 to 347 2024-09-13T14:34:30.627 INFO:tasks.rados.rados.0.smithi151.stdout:3271: read oid 39 snap 407 2024-09-13T14:34:30.627 INFO:tasks.rados.rados.0.smithi151.stdout:3271: expect (ObjNum 1189 snap 402 seq_num 1189) 2024-09-13T14:34:30.627 INFO:tasks.rados.rados.0.smithi151.stdout:3272: read oid 46 snap -1 2024-09-13T14:34:30.627 INFO:tasks.rados.rados.0.smithi151.stdout:3272: expect (ObjNum 1135 snap 381 seq_num 1135) 2024-09-13T14:34:30.627 INFO:tasks.rados.rados.0.smithi151.stdout:3273: snap_create 2024-09-13T14:34:30.627 INFO:tasks.rados.rados.0.smithi151.stdout:3263: expect (ObjNum 1068 snap 363 seq_num 1068) 2024-09-13T14:34:30.758 INFO:tasks.rados.rados.0.smithi151.stdout:3264: finishing write tid 2 to smithi15186934-27 2024-09-13T14:34:30.758 INFO:tasks.rados.rados.0.smithi151.stdout:3265: finishing rollback tid 1 to smithi15186934-3 2024-09-13T14:34:30.758 INFO:tasks.rados.rados.0.smithi151.stdout:3264: finishing write tid 3 to smithi15186934-27 2024-09-13T14:34:30.758 INFO:tasks.rados.rados.0.smithi151.stdout:3264: finishing write tid 4 to smithi15186934-27 2024-09-13T14:34:30.758 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 27 v 867 (ObjNum 1229 snap 412 seq_num 1229) dirty exists 2024-09-13T14:34:30.758 INFO:tasks.rados.rados.0.smithi151.stdout:3264: left oid 27 (ObjNum 1229 snap 412 seq_num 1229) 2024-09-13T14:34:30.758 INFO:tasks.rados.rados.0.smithi151.stdout:3265: finishing rollback tid 2 to smithi15186934-3 2024-09-13T14:34:30.759 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 1271 (ObjNum 1097 snap 368 seq_num 1097) dirty exists 2024-09-13T14:34:30.759 INFO:tasks.rados.rados.0.smithi151.stdout:3266: finishing rollback tid 1 to smithi15186934-43 2024-09-13T14:34:30.759 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 43 v 1750 (ObjNum 33 snap 0 seq_num 3092548368) dirty dne 2024-09-13T14:34:30.759 INFO:tasks.rados.rados.0.smithi151.stdout:3270: finishing rollback tid 1 to smithi15186934-29 2024-09-13T14:34:30.759 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 29 v 719 (ObjNum 129 snap 0 seq_num 3092812912) dirty dne 2024-09-13T14:34:30.759 INFO:tasks.rados.rados.0.smithi151.stdout:3268: expect (ObjNum 1202 snap 407 seq_num 1202) 2024-09-13T14:34:30.829 INFO:tasks.rados.rados.0.smithi151.stdout:3272: expect (ObjNum 1135 snap 381 seq_num 1135) 2024-09-13T14:34:30.963 INFO:tasks.rados.rados.0.smithi151.stdout:3269: expect (ObjNum 1204 snap 407 seq_num 1204) 2024-09-13T14:34:31.053 INFO:tasks.rados.rados.0.smithi151.stdout:3271: expect (ObjNum 1189 snap 402 seq_num 1189) 2024-09-13T14:34:31.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:30 smithi184 ceph-mon[92527]: pgmap v936: 89 pgs: 89 active+clean; 315 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 7.6 MiB/s rd, 1.5 MiB/s wr, 13 op/s 2024-09-13T14:34:31.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:30 smithi184 ceph-mon[92527]: osdmap e880: 8 total, 8 up, 8 in 2024-09-13T14:34:31.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:30 smithi151 ceph-mon[104450]: pgmap v936: 89 pgs: 89 active+clean; 315 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 7.6 MiB/s rd, 1.5 MiB/s wr, 13 op/s 2024-09-13T14:34:31.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:30 smithi151 ceph-mon[104450]: osdmap e880: 8 total, 8 up, 8 in 2024-09-13T14:34:31.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:30 smithi151 ceph-mon[108966]: pgmap v936: 89 pgs: 89 active+clean; 315 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 7.6 MiB/s rd, 1.5 MiB/s wr, 13 op/s 2024-09-13T14:34:31.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:30 smithi151 ceph-mon[108966]: osdmap e880: 8 total, 8 up, 8 in 2024-09-13T14:34:31.623 INFO:tasks.rados.rados.0.smithi151.stdout:3263: done (9 left) 2024-09-13T14:34:31.623 INFO:tasks.rados.rados.0.smithi151.stdout:3264: done (8 left) 2024-09-13T14:34:31.623 INFO:tasks.rados.rados.0.smithi151.stdout:3265: done (7 left) 2024-09-13T14:34:31.623 INFO:tasks.rados.rados.0.smithi151.stdout:3266: done (6 left) 2024-09-13T14:34:31.623 INFO:tasks.rados.rados.0.smithi151.stdout:3268: done (5 left) 2024-09-13T14:34:31.624 INFO:tasks.rados.rados.0.smithi151.stdout:3269: done (4 left) 2024-09-13T14:34:31.624 INFO:tasks.rados.rados.0.smithi151.stdout:3270: done (3 left) 2024-09-13T14:34:31.624 INFO:tasks.rados.rados.0.smithi151.stdout:3271: done (2 left) 2024-09-13T14:34:31.624 INFO:tasks.rados.rados.0.smithi151.stdout:3272: done (1 left) 2024-09-13T14:34:31.624 INFO:tasks.rados.rados.0.smithi151.stdout:3273: done (0 left) 2024-09-13T14:34:31.624 INFO:tasks.rados.rados.0.smithi151.stdout:3274: delete oid 47 current snap is 413 2024-09-13T14:34:31.628 INFO:tasks.rados.rados.0.smithi151.stdout:3274: done (0 left) 2024-09-13T14:34:31.628 INFO:tasks.rados.rados.0.smithi151.stdout:3275: rollback oid 17 current snap is 413 2024-09-13T14:34:31.628 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 17 to 410 2024-09-13T14:34:31.628 INFO:tasks.rados.rados.0.smithi151.stdout:3276: read oid 18 snap -1 2024-09-13T14:34:31.629 INFO:tasks.rados.rados.0.smithi151.stdout:3276: expect (ObjNum 1203 snap 407 seq_num 1203) 2024-09-13T14:34:31.629 INFO:tasks.rados.rados.0.smithi151.stdout:3277: read oid 19 snap -1 2024-09-13T14:34:31.629 INFO:tasks.rados.rados.0.smithi151.stdout:3277: expect deleted 2024-09-13T14:34:31.629 INFO:tasks.rados.rados.0.smithi151.stdout:3278: read oid 23 snap -1 2024-09-13T14:34:31.629 INFO:tasks.rados.rados.0.smithi151.stdout:3278: expect (ObjNum 1216 snap 409 seq_num 1216) 2024-09-13T14:34:31.629 INFO:tasks.rados.rados.0.smithi151.stdout:3279: rollback oid 40 current snap is 413 2024-09-13T14:34:31.629 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 40 to 400 2024-09-13T14:34:31.629 INFO:tasks.rados.rados.0.smithi151.stdout:3280: snap_create 2024-09-13T14:34:31.632 INFO:tasks.rados.rados.0.smithi151.stdout:3279: finishing rollback tid 0 to smithi15186934-40 2024-09-13T14:34:31.633 INFO:tasks.rados.rados.0.smithi151.stdout:3279: finishing rollback tid 1 to smithi15186934-40 2024-09-13T14:34:31.633 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 40 v 1274 (ObjNum 49 snap 0 seq_num 3680720968) dirty dne 2024-09-13T14:34:31.633 INFO:tasks.rados.rados.0.smithi151.stdout:3276: expect (ObjNum 1203 snap 407 seq_num 1203) 2024-09-13T14:34:31.736 INFO:tasks.rados.rados.0.smithi151.stdout:3275: finishing rollback tid 1 to smithi15186934-17 2024-09-13T14:34:31.736 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 0 (ObjNum 0 snap 0 seq_num 304) dirty dne 2024-09-13T14:34:31.736 INFO:tasks.rados.rados.0.smithi151.stdout:3278: expect (ObjNum 1216 snap 409 seq_num 1216) 2024-09-13T14:34:32.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:32.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:31 smithi184 ceph-mon[92527]: osdmap e881: 8 total, 8 up, 8 in 2024-09-13T14:34:32.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:32.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:31 smithi151 ceph-mon[104450]: osdmap e881: 8 total, 8 up, 8 in 2024-09-13T14:34:32.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:32.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:31 smithi151 ceph-mon[108966]: osdmap e881: 8 total, 8 up, 8 in 2024-09-13T14:34:32.628 INFO:tasks.rados.rados.0.smithi151.stdout:3275: done (5 left) 2024-09-13T14:34:32.629 INFO:tasks.rados.rados.0.smithi151.stdout:3276: done (4 left) 2024-09-13T14:34:32.629 INFO:tasks.rados.rados.0.smithi151.stdout:3277: done (3 left) 2024-09-13T14:34:32.629 INFO:tasks.rados.rados.0.smithi151.stdout:3278: done (2 left) 2024-09-13T14:34:32.629 INFO:tasks.rados.rados.0.smithi151.stdout:3279: done (1 left) 2024-09-13T14:34:32.629 INFO:tasks.rados.rados.0.smithi151.stdout:3280: done (0 left) 2024-09-13T14:34:32.629 INFO:tasks.rados.rados.0.smithi151.stdout:3281: write oid 17 current snap is 414 2024-09-13T14:34:32.629 INFO:tasks.rados.rados.0.smithi151.stdout:3281: seq_num 1231 ranges {689417=564352,1701922=466090} 2024-09-13T14:34:32.634 INFO:tasks.rados.rados.0.smithi151.stdout:3281: writing smithi15186934-17 from 689417 to 1253769 tid 1 2024-09-13T14:34:32.640 INFO:tasks.rados.rados.0.smithi151.stdout:3281: writing smithi15186934-17 from 1701922 to 2168012 tid 2 2024-09-13T14:34:32.640 INFO:tasks.rados.rados.0.smithi151.stdout:3282: write oid 21 current snap is 414 2024-09-13T14:34:32.641 INFO:tasks.rados.rados.0.smithi151.stdout:3282: seq_num 1232 ranges {706224=755954,2160404=545820} 2024-09-13T14:34:32.648 INFO:tasks.rados.rados.0.smithi151.stdout:3282: writing smithi15186934-21 from 706224 to 1462178 tid 1 2024-09-13T14:34:32.655 INFO:tasks.rados.rados.0.smithi151.stdout:3282: writing smithi15186934-21 from 2160404 to 2706224 tid 2 2024-09-13T14:34:32.655 INFO:tasks.rados.rados.0.smithi151.stdout:3283: write oid 47 current snap is 414 2024-09-13T14:34:32.655 INFO:tasks.rados.rados.0.smithi151.stdout:3283: seq_num 1233 ranges {723031=431203,1818886=469759} 2024-09-13T14:34:32.661 INFO:tasks.rados.rados.0.smithi151.stdout:3283: writing smithi15186934-47 from 723031 to 1154234 tid 1 2024-09-13T14:34:32.668 INFO:tasks.rados.rados.0.smithi151.stdout:3283: writing smithi15186934-47 from 1818886 to 2288645 tid 2 2024-09-13T14:34:32.668 INFO:tasks.rados.rados.0.smithi151.stdout:3284: delete oid 5 current snap is 414 2024-09-13T14:34:32.668 INFO:tasks.rados.rados.0.smithi151.stdout:3281: finishing write tid 1 to smithi15186934-17 2024-09-13T14:34:32.668 INFO:tasks.rados.rados.0.smithi151.stdout:3281: finishing write tid 2 to smithi15186934-17 2024-09-13T14:34:32.668 INFO:tasks.rados.rados.0.smithi151.stdout:3281: finishing write tid 3 to smithi15186934-17 2024-09-13T14:34:32.668 INFO:tasks.rados.rados.0.smithi151.stdout:3281: finishing write tid 4 to smithi15186934-17 2024-09-13T14:34:32.668 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 1891 (ObjNum 1231 snap 414 seq_num 1231) dirty exists 2024-09-13T14:34:32.669 INFO:tasks.rados.rados.0.smithi151.stdout:3281: left oid 17 (ObjNum 1231 snap 414 seq_num 1231) 2024-09-13T14:34:32.669 INFO:tasks.rados.rados.0.smithi151.stdout:3282: finishing write tid 1 to smithi15186934-21 2024-09-13T14:34:32.669 INFO:tasks.rados.rados.0.smithi151.stdout:3282: finishing write tid 2 to smithi15186934-21 2024-09-13T14:34:32.669 INFO:tasks.rados.rados.0.smithi151.stdout:3282: finishing write tid 3 to smithi15186934-21 2024-09-13T14:34:32.669 INFO:tasks.rados.rados.0.smithi151.stdout:3282: finishing write tid 4 to smithi15186934-21 2024-09-13T14:34:32.669 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 21 v 655 (ObjNum 1232 snap 414 seq_num 1232) dirty exists 2024-09-13T14:34:32.669 INFO:tasks.rados.rados.0.smithi151.stdout:3282: left oid 21 (ObjNum 1232 snap 414 seq_num 1232) 2024-09-13T14:34:32.669 INFO:tasks.rados.rados.0.smithi151.stdout:3283: finishing write tid 1 to smithi15186934-47 2024-09-13T14:34:32.671 INFO:tasks.rados.rados.0.smithi151.stdout:3283: finishing write tid 2 to smithi15186934-47 2024-09-13T14:34:32.671 INFO:tasks.rados.rados.0.smithi151.stdout:3283: finishing write tid 3 to smithi15186934-47 2024-09-13T14:34:32.671 INFO:tasks.rados.rados.0.smithi151.stdout:3283: finishing write tid 4 to smithi15186934-47 2024-09-13T14:34:32.671 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 1503 (ObjNum 1233 snap 414 seq_num 1233) dirty exists 2024-09-13T14:34:32.671 INFO:tasks.rados.rados.0.smithi151.stdout:3283: left oid 47 (ObjNum 1233 snap 414 seq_num 1233) 2024-09-13T14:34:32.672 INFO:tasks.rados.rados.0.smithi151.stdout:3281: done (3 left) 2024-09-13T14:34:32.672 INFO:tasks.rados.rados.0.smithi151.stdout:3282: done (2 left) 2024-09-13T14:34:32.672 INFO:tasks.rados.rados.0.smithi151.stdout:3283: done (1 left) 2024-09-13T14:34:32.672 INFO:tasks.rados.rados.0.smithi151.stdout:3284: done (0 left) 2024-09-13T14:34:32.673 INFO:tasks.rados.rados.0.smithi151.stdout:3285: delete oid 11 current snap is 414 2024-09-13T14:34:32.674 INFO:tasks.rados.rados.0.smithi151.stdout:3285: done (0 left) 2024-09-13T14:34:32.674 INFO:tasks.rados.rados.0.smithi151.stdout:3286: snap_create 2024-09-13T14:34:33.010 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:32 smithi151 ceph-mon[104450]: pgmap v939: 89 pgs: 89 active+clean; 327 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.1 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-13T14:34:33.010 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:32 smithi151 ceph-mon[104450]: osdmap e882: 8 total, 8 up, 8 in 2024-09-13T14:34:33.010 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:32 smithi151 ceph-mon[108966]: pgmap v939: 89 pgs: 89 active+clean; 327 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.1 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-13T14:34:33.011 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:32 smithi151 ceph-mon[108966]: osdmap e882: 8 total, 8 up, 8 in 2024-09-13T14:34:33.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:32 smithi184 ceph-mon[92527]: pgmap v939: 89 pgs: 89 active+clean; 327 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.1 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-13T14:34:33.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:32 smithi184 ceph-mon[92527]: osdmap e882: 8 total, 8 up, 8 in 2024-09-13T14:34:33.628 INFO:tasks.rados.rados.0.smithi151.stdout:3286: done (0 left) 2024-09-13T14:34:33.629 INFO:tasks.rados.rados.0.smithi151.stdout:3287: snap_create 2024-09-13T14:34:34.622 INFO:tasks.rados.rados.0.smithi151.stdout:3287: done (0 left) 2024-09-13T14:34:34.622 INFO:tasks.rados.rados.0.smithi151.stdout:3288: read oid 26 snap -1 2024-09-13T14:34:34.622 INFO:tasks.rados.rados.0.smithi151.stdout:3288: expect deleted 2024-09-13T14:34:34.623 INFO:tasks.rados.rados.0.smithi151.stdout:3289: snap_remove snap 414 2024-09-13T14:34:34.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:34 smithi151 ceph-mon[104450]: pgmap v942: 89 pgs: 89 active+clean; 331 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 1.7 MiB/s wr, 19 op/s 2024-09-13T14:34:34.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:34 smithi151 ceph-mon[104450]: osdmap e883: 8 total, 8 up, 8 in 2024-09-13T14:34:34.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:34 smithi151 ceph-mon[104450]: osdmap e884: 8 total, 8 up, 8 in 2024-09-13T14:34:34.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:34:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:34:34] "GET /metrics HTTP/1.1" 200 36507 "" "Prometheus/2.51.0" 2024-09-13T14:34:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:34 smithi151 ceph-mon[108966]: pgmap v942: 89 pgs: 89 active+clean; 331 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 1.7 MiB/s wr, 19 op/s 2024-09-13T14:34:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:34 smithi151 ceph-mon[108966]: osdmap e883: 8 total, 8 up, 8 in 2024-09-13T14:34:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:34 smithi151 ceph-mon[108966]: osdmap e884: 8 total, 8 up, 8 in 2024-09-13T14:34:35.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:34 smithi184 ceph-mon[92527]: pgmap v942: 89 pgs: 89 active+clean; 331 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 1.7 MiB/s wr, 19 op/s 2024-09-13T14:34:35.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:34 smithi184 ceph-mon[92527]: osdmap e883: 8 total, 8 up, 8 in 2024-09-13T14:34:35.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:34 smithi184 ceph-mon[92527]: osdmap e884: 8 total, 8 up, 8 in 2024-09-13T14:34:35.625 INFO:tasks.rados.rados.0.smithi151.stdout:3289: done (1 left) 2024-09-13T14:34:35.625 INFO:tasks.rados.rados.0.smithi151.stdout:3290: write oid 35 current snap is 416 2024-09-13T14:34:35.625 INFO:tasks.rados.rados.0.smithi151.stdout:3290: seq_num 1236 ranges {773452=656950,1877979=617086} 2024-09-13T14:34:35.631 INFO:tasks.rados.rados.0.smithi151.stdout:3290: writing smithi15186934-35 from 773452 to 1430402 tid 1 2024-09-13T14:34:35.638 INFO:tasks.rados.rados.0.smithi151.stdout:3290: writing smithi15186934-35 from 1877979 to 2495065 tid 2 2024-09-13T14:34:35.638 INFO:tasks.rados.rados.0.smithi151.stdout:3291: rollback oid 46 current snap is 416 2024-09-13T14:34:35.638 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 46 to 384 2024-09-13T14:34:35.638 INFO:tasks.rados.rados.0.smithi151.stdout:3288: done (2 left) 2024-09-13T14:34:35.638 INFO:tasks.rados.rados.0.smithi151.stdout:3292: rollback oid 22 current snap is 416 2024-09-13T14:34:35.639 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 22 to 399 2024-09-13T14:34:35.639 INFO:tasks.rados.rados.0.smithi151.stdout:3293: delete oid 30 current snap is 416 2024-09-13T14:34:35.639 INFO:tasks.rados.rados.0.smithi151.stdout:3290: finishing write tid 1 to smithi15186934-35 2024-09-13T14:34:35.640 INFO:tasks.rados.rados.0.smithi151.stdout:3292: finishing rollback tid 1 to smithi15186934-22 2024-09-13T14:34:35.640 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 22 v 594 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-13T14:34:35.643 INFO:tasks.rados.rados.0.smithi151.stdout:3290: finishing write tid 2 to smithi15186934-35 2024-09-13T14:34:35.646 INFO:tasks.rados.rados.0.smithi151.stdout:3290: finishing write tid 3 to smithi15186934-35 2024-09-13T14:34:35.646 INFO:tasks.rados.rados.0.smithi151.stdout:3291: finishing rollback tid 0 to smithi15186934-46 2024-09-13T14:34:35.646 INFO:tasks.rados.rados.0.smithi151.stdout:3291: finishing rollback tid 1 to smithi15186934-46 2024-09-13T14:34:35.646 INFO:tasks.rados.rados.0.smithi151.stdout:3290: finishing write tid 4 to smithi15186934-35 2024-09-13T14:34:35.646 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 35 v 1894 (ObjNum 1236 snap 416 seq_num 1236) dirty exists 2024-09-13T14:34:35.646 INFO:tasks.rados.rados.0.smithi151.stdout:3290: left oid 35 (ObjNum 1236 snap 416 seq_num 1236) 2024-09-13T14:34:35.646 INFO:tasks.rados.rados.0.smithi151.stdout:3291: finishing rollback tid 2 to smithi15186934-46 2024-09-13T14:34:35.646 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 46 v 1201 (ObjNum 1135 snap 381 seq_num 1135) dirty exists 2024-09-13T14:34:35.647 INFO:tasks.rados.rados.0.smithi151.stdout:3290: done (3 left) 2024-09-13T14:34:35.647 INFO:tasks.rados.rados.0.smithi151.stdout:3291: done (2 left) 2024-09-13T14:34:35.647 INFO:tasks.rados.rados.0.smithi151.stdout:3292: done (1 left) 2024-09-13T14:34:35.647 INFO:tasks.rados.rados.0.smithi151.stdout:3293: done (0 left) 2024-09-13T14:34:35.647 INFO:tasks.rados.rados.0.smithi151.stdout:3294: snap_remove snap 399 2024-09-13T14:34:36.633 INFO:tasks.rados.rados.0.smithi151.stdout:3294: done (0 left) 2024-09-13T14:34:36.633 INFO:tasks.rados.rados.0.smithi151.stdout:3295: snap_remove snap 415 2024-09-13T14:34:37.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:36 smithi184 ceph-mon[92527]: pgmap v945: 89 pgs: 89 active+clean; 331 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 569 KiB/s wr, 13 op/s 2024-09-13T14:34:37.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:36 smithi184 ceph-mon[92527]: osdmap e885: 8 total, 8 up, 8 in 2024-09-13T14:34:37.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:36 smithi151 ceph-mon[104450]: pgmap v945: 89 pgs: 89 active+clean; 331 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 569 KiB/s wr, 13 op/s 2024-09-13T14:34:37.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:36 smithi151 ceph-mon[104450]: osdmap e885: 8 total, 8 up, 8 in 2024-09-13T14:34:37.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:36 smithi151 ceph-mon[108966]: pgmap v945: 89 pgs: 89 active+clean; 331 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 569 KiB/s wr, 13 op/s 2024-09-13T14:34:37.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:36 smithi151 ceph-mon[108966]: osdmap e885: 8 total, 8 up, 8 in 2024-09-13T14:34:37.643 INFO:tasks.rados.rados.0.smithi151.stdout:3295: done (0 left) 2024-09-13T14:34:37.643 INFO:tasks.rados.rados.0.smithi151.stdout:3296: snap_remove snap 410 2024-09-13T14:34:38.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:37 smithi184 ceph-mon[92527]: osdmap e886: 8 total, 8 up, 8 in 2024-09-13T14:34:38.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:37 smithi151 ceph-mon[104450]: osdmap e886: 8 total, 8 up, 8 in 2024-09-13T14:34:38.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:37 smithi151 ceph-mon[108966]: osdmap e886: 8 total, 8 up, 8 in 2024-09-13T14:34:38.645 INFO:tasks.rados.rados.0.smithi151.stdout:3296: done (0 left) 2024-09-13T14:34:38.645 INFO:tasks.rados.rados.0.smithi151.stdout:3297: rollback oid 5 current snap is 416 2024-09-13T14:34:38.645 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 5 to 363 2024-09-13T14:34:38.645 INFO:tasks.rados.rados.0.smithi151.stdout:3298: rollback oid 33 current snap is 416 2024-09-13T14:34:38.645 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 33 to 396 2024-09-13T14:34:38.645 INFO:tasks.rados.rados.0.smithi151.stdout:3299: rollback oid 15 current snap is 416 2024-09-13T14:34:38.645 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 15 to 379 2024-09-13T14:34:38.645 INFO:tasks.rados.rados.0.smithi151.stdout:3300: snap_remove snap 347 2024-09-13T14:34:39.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:38 smithi184 ceph-mon[92527]: pgmap v948: 89 pgs: 89 active+clean; 340 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.4 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2024-09-13T14:34:39.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:38 smithi184 ceph-mon[92527]: osdmap e887: 8 total, 8 up, 8 in 2024-09-13T14:34:39.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:38 smithi151 ceph-mon[104450]: pgmap v948: 89 pgs: 89 active+clean; 340 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.4 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2024-09-13T14:34:39.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:38 smithi151 ceph-mon[104450]: osdmap e887: 8 total, 8 up, 8 in 2024-09-13T14:34:39.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:38 smithi151 ceph-mon[108966]: pgmap v948: 89 pgs: 89 active+clean; 340 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.4 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2024-09-13T14:34:39.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:38 smithi151 ceph-mon[108966]: osdmap e887: 8 total, 8 up, 8 in 2024-09-13T14:34:39.627 INFO:tasks.rados.rados.0.smithi151.stdout:3299: finishing rollback tid 1 to smithi15186934-15 2024-09-13T14:34:39.627 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 15 v 1150 (ObjNum 65 snap 0 seq_num 1953066355) dirty dne 2024-09-13T14:34:39.627 INFO:tasks.rados.rados.0.smithi151.stdout:3297: finishing rollback tid 1 to smithi15186934-5 2024-09-13T14:34:39.627 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 1213 (ObjNum 129 snap 0 seq_num 3092993960) dirty dne 2024-09-13T14:34:39.627 INFO:tasks.rados.rados.0.smithi151.stdout:3298: finishing rollback tid 0 to smithi15186934-33 2024-09-13T14:34:39.627 INFO:tasks.rados.rados.0.smithi151.stdout:3298: finishing rollback tid 1 to smithi15186934-33 2024-09-13T14:34:39.627 INFO:tasks.rados.rados.0.smithi151.stdout:3298: finishing rollback tid 2 to smithi15186934-33 2024-09-13T14:34:39.628 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 33 v 1222 (ObjNum 1058 snap 357 seq_num 1058) dirty exists 2024-09-13T14:34:39.628 INFO:tasks.rados.rados.0.smithi151.stdout:3297: done (3 left) 2024-09-13T14:34:39.628 INFO:tasks.rados.rados.0.smithi151.stdout:3298: done (2 left) 2024-09-13T14:34:39.628 INFO:tasks.rados.rados.0.smithi151.stdout:3299: done (1 left) 2024-09-13T14:34:39.628 INFO:tasks.rados.rados.0.smithi151.stdout:3300: done (0 left) 2024-09-13T14:34:39.628 INFO:tasks.rados.rados.0.smithi151.stdout:3301: delete oid 31 current snap is 416 2024-09-13T14:34:39.637 INFO:tasks.rados.rados.0.smithi151.stdout:3301: done (0 left) 2024-09-13T14:34:39.638 INFO:tasks.rados.rados.0.smithi151.stdout:3302: snap_remove snap 400 2024-09-13T14:34:40.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:39 smithi184 ceph-mon[92527]: osdmap e888: 8 total, 8 up, 8 in 2024-09-13T14:34:40.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:39 smithi184 ceph-mon[92527]: osdmap e889: 8 total, 8 up, 8 in 2024-09-13T14:34:40.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:39 smithi151 ceph-mon[104450]: osdmap e888: 8 total, 8 up, 8 in 2024-09-13T14:34:40.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:39 smithi151 ceph-mon[104450]: osdmap e889: 8 total, 8 up, 8 in 2024-09-13T14:34:40.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:39 smithi151 ceph-mon[108966]: osdmap e888: 8 total, 8 up, 8 in 2024-09-13T14:34:40.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:39 smithi151 ceph-mon[108966]: osdmap e889: 8 total, 8 up, 8 in 2024-09-13T14:34:40.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:34:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:34:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:34:40.640 INFO:tasks.rados.rados.0.smithi151.stdout:3302: done (0 left) 2024-09-13T14:34:40.641 INFO:tasks.rados.rados.0.smithi151.stdout:3303: rollback oid 4 current snap is 416 2024-09-13T14:34:40.641 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 4 to 403 2024-09-13T14:34:40.641 INFO:tasks.rados.rados.0.smithi151.stdout:3304: rollback oid 39 current snap is 416 2024-09-13T14:34:40.641 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 39 to 411 2024-09-13T14:34:40.641 INFO:tasks.rados.rados.0.smithi151.stdout:3305: write oid 29 current snap is 416 2024-09-13T14:34:40.641 INFO:tasks.rados.rados.0.smithi151.stdout:3305: seq_num 1239 ranges {423873=482697,1653425=480766,2691441=132432} 2024-09-13T14:34:40.646 INFO:tasks.rados.rados.0.smithi151.stdout:3305: writing smithi15186934-29 from 423873 to 906570 tid 1 2024-09-13T14:34:40.654 INFO:tasks.rados.rados.0.smithi151.stdout:3305: writing smithi15186934-29 from 1653425 to 2134191 tid 2 2024-09-13T14:34:40.659 INFO:tasks.rados.rados.0.smithi151.stdout:3305: writing smithi15186934-29 from 2691441 to 2823873 tid 3 2024-09-13T14:34:40.659 INFO:tasks.rados.rados.0.smithi151.stdout:3306: delete oid 45 current snap is 416 2024-09-13T14:34:40.659 INFO:tasks.rados.rados.0.smithi151.stdout:3304: finishing rollback tid 0 to smithi15186934-39 2024-09-13T14:34:40.659 INFO:tasks.rados.rados.0.smithi151.stdout:3303: finishing rollback tid 0 to smithi15186934-4 2024-09-13T14:34:40.659 INFO:tasks.rados.rados.0.smithi151.stdout:3303: finishing rollback tid 1 to smithi15186934-4 2024-09-13T14:34:40.659 INFO:tasks.rados.rados.0.smithi151.stdout:3304: finishing rollback tid 1 to smithi15186934-39 2024-09-13T14:34:40.659 INFO:tasks.rados.rados.0.smithi151.stdout:3304: finishing rollback tid 2 to smithi15186934-39 2024-09-13T14:34:40.659 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 1376 (ObjNum 1189 snap 402 seq_num 1189) dirty exists 2024-09-13T14:34:40.660 INFO:tasks.rados.rados.0.smithi151.stdout:3303: finishing rollback tid 2 to smithi15186934-4 2024-09-13T14:34:40.660 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 1571 (ObjNum 1188 snap 402 seq_num 1188) dirty exists 2024-09-13T14:34:40.660 INFO:tasks.rados.rados.0.smithi151.stdout:3305: finishing write tid 1 to smithi15186934-29 2024-09-13T14:34:40.660 INFO:tasks.rados.rados.0.smithi151.stdout:3305: finishing write tid 2 to smithi15186934-29 2024-09-13T14:34:40.662 INFO:tasks.rados.rados.0.smithi151.stdout:3305: finishing write tid 3 to smithi15186934-29 2024-09-13T14:34:40.662 INFO:tasks.rados.rados.0.smithi151.stdout:3303: done (3 left) 2024-09-13T14:34:40.662 INFO:tasks.rados.rados.0.smithi151.stdout:3304: done (2 left) 2024-09-13T14:34:40.662 INFO:tasks.rados.rados.0.smithi151.stdout:3306: done (1 left) 2024-09-13T14:34:40.662 INFO:tasks.rados.rados.0.smithi151.stdout:3307: write oid 27 current snap is 416 2024-09-13T14:34:40.662 INFO:tasks.rados.rados.0.smithi151.stdout:3305: finishing write tid 4 to smithi15186934-29 2024-09-13T14:34:40.663 INFO:tasks.rados.rados.0.smithi151.stdout:3307: seq_num 1241 ranges {457487=749548,1770389=484435,2746287=111200} 2024-09-13T14:34:40.668 INFO:tasks.rados.rados.0.smithi151.stdout:3307: writing smithi15186934-27 from 457487 to 1207035 tid 1 2024-09-13T14:34:40.674 INFO:tasks.rados.rados.0.smithi151.stdout:3307: writing smithi15186934-27 from 1770389 to 2254824 tid 2 2024-09-13T14:34:40.678 INFO:tasks.rados.rados.0.smithi151.stdout:3307: writing smithi15186934-27 from 2746287 to 2857487 tid 3 2024-09-13T14:34:40.678 INFO:tasks.rados.rados.0.smithi151.stdout:3308: delete oid 7 current snap is 416 2024-09-13T14:34:40.678 INFO:tasks.rados.rados.0.smithi151.stdout:3305: finishing write tid 5 to smithi15186934-29 2024-09-13T14:34:40.678 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 29 v 843 (ObjNum 1239 snap 416 seq_num 1239) dirty exists 2024-09-13T14:34:40.679 INFO:tasks.rados.rados.0.smithi151.stdout:3305: left oid 29 (ObjNum 1239 snap 416 seq_num 1239) 2024-09-13T14:34:40.679 INFO:tasks.rados.rados.0.smithi151.stdout:3307: finishing write tid 1 to smithi15186934-27 2024-09-13T14:34:40.679 INFO:tasks.rados.rados.0.smithi151.stdout:3307: finishing write tid 2 to smithi15186934-27 2024-09-13T14:34:40.680 INFO:tasks.rados.rados.0.smithi151.stdout:3307: finishing write tid 3 to smithi15186934-27 2024-09-13T14:34:40.681 INFO:tasks.rados.rados.0.smithi151.stdout:3307: finishing write tid 4 to smithi15186934-27 2024-09-13T14:34:40.681 INFO:tasks.rados.rados.0.smithi151.stdout:3307: finishing write tid 5 to smithi15186934-27 2024-09-13T14:34:40.681 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 27 v 888 (ObjNum 1241 snap 416 seq_num 1241) dirty exists 2024-09-13T14:34:40.681 INFO:tasks.rados.rados.0.smithi151.stdout:3307: left oid 27 (ObjNum 1241 snap 416 seq_num 1241) 2024-09-13T14:34:40.681 INFO:tasks.rados.rados.0.smithi151.stdout:3305: done (2 left) 2024-09-13T14:34:40.681 INFO:tasks.rados.rados.0.smithi151.stdout:3307: done (1 left) 2024-09-13T14:34:40.681 INFO:tasks.rados.rados.0.smithi151.stdout:3308: done (0 left) 2024-09-13T14:34:40.682 INFO:tasks.rados.rados.0.smithi151.stdout:3309: snap_remove snap 384 2024-09-13T14:34:41.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:40 smithi184 ceph-mon[92527]: pgmap v951: 89 pgs: 89 active+clean; 340 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.0 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2024-09-13T14:34:41.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:40 smithi184 ceph-mon[92527]: osdmap e890: 8 total, 8 up, 8 in 2024-09-13T14:34:41.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:40 smithi151 ceph-mon[104450]: pgmap v951: 89 pgs: 89 active+clean; 340 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.0 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2024-09-13T14:34:41.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:40 smithi151 ceph-mon[104450]: osdmap e890: 8 total, 8 up, 8 in 2024-09-13T14:34:41.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:40 smithi151 ceph-mon[108966]: pgmap v951: 89 pgs: 89 active+clean; 340 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.0 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2024-09-13T14:34:41.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:40 smithi151 ceph-mon[108966]: osdmap e890: 8 total, 8 up, 8 in 2024-09-13T14:34:41.668 INFO:tasks.rados.rados.0.smithi151.stdout:3309: done (0 left) 2024-09-13T14:34:41.668 INFO:tasks.rados.rados.0.smithi151.stdout:3310: snap_remove snap 407 2024-09-13T14:34:42.674 INFO:tasks.rados.rados.0.smithi151.stdout:3310: done (0 left) 2024-09-13T14:34:42.674 INFO:tasks.rados.rados.0.smithi151.stdout:3311: snap_create 2024-09-13T14:34:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:42 smithi184 ceph-mon[92527]: pgmap v954: 89 pgs: 89 active+clean; 330 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 1 op/s 2024-09-13T14:34:43.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:42 smithi184 ceph-mon[92527]: osdmap e891: 8 total, 8 up, 8 in 2024-09-13T14:34:43.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:42 smithi151 ceph-mon[104450]: pgmap v954: 89 pgs: 89 active+clean; 330 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 1 op/s 2024-09-13T14:34:43.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:42 smithi151 ceph-mon[104450]: osdmap e891: 8 total, 8 up, 8 in 2024-09-13T14:34:43.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:42 smithi151 ceph-mon[108966]: pgmap v954: 89 pgs: 89 active+clean; 330 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 1 op/s 2024-09-13T14:34:43.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:42 smithi151 ceph-mon[108966]: osdmap e891: 8 total, 8 up, 8 in 2024-09-13T14:34:43.689 INFO:tasks.rados.rados.0.smithi151.stdout:3311: done (0 left) 2024-09-13T14:34:43.689 INFO:tasks.rados.rados.0.smithi151.stdout:3312: snap_create 2024-09-13T14:34:44.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:43 smithi184 ceph-mon[92527]: osdmap e892: 8 total, 8 up, 8 in 2024-09-13T14:34:44.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:43 smithi151 ceph-mon[104450]: osdmap e892: 8 total, 8 up, 8 in 2024-09-13T14:34:44.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:43 smithi151 ceph-mon[108966]: osdmap e892: 8 total, 8 up, 8 in 2024-09-13T14:34:44.629 INFO:tasks.rados.rados.0.smithi151.stdout:3312: done (0 left) 2024-09-13T14:34:44.629 INFO:tasks.rados.rados.0.smithi151.stdout:3313: rollback oid 7 current snap is 418 2024-09-13T14:34:44.629 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 7 to 406 2024-09-13T14:34:44.629 INFO:tasks.rados.rados.0.smithi151.stdout:3314: read oid 21 snap -1 2024-09-13T14:34:44.630 INFO:tasks.rados.rados.0.smithi151.stdout:3314: expect (ObjNum 1232 snap 414 seq_num 1232) 2024-09-13T14:34:44.630 INFO:tasks.rados.rados.0.smithi151.stdout:3315: read oid 27 snap -1 2024-09-13T14:34:44.630 INFO:tasks.rados.rados.0.smithi151.stdout:3315: expect (ObjNum 1241 snap 416 seq_num 1241) 2024-09-13T14:34:44.630 INFO:tasks.rados.rados.0.smithi151.stdout:3316: write oid 15 current snap is 418 2024-09-13T14:34:44.630 INFO:tasks.rados.rados.0.smithi151.stdout:3316: seq_num 1243 ranges {491101=500046,1487353=488104,2401133=489968} 2024-09-13T14:34:44.633 INFO:tasks.rados.rados.0.smithi151.stdout:3316: writing smithi15186934-15 from 491101 to 991147 tid 1 2024-09-13T14:34:44.640 INFO:tasks.rados.rados.0.smithi151.stdout:3316: writing smithi15186934-15 from 1487353 to 1975457 tid 2 2024-09-13T14:34:44.647 INFO:tasks.rados.rados.0.smithi151.stdout:3316: writing smithi15186934-15 from 2401133 to 2891101 tid 3 2024-09-13T14:34:44.647 INFO:tasks.rados.rados.0.smithi151.stdout:3317: snap_create 2024-09-13T14:34:44.647 INFO:tasks.rados.rados.0.smithi151.stdout:3313: finishing rollback tid 1 to smithi15186934-7 2024-09-13T14:34:44.648 INFO:tasks.rados.rados.0.smithi151.stdout:3313: finishing rollback tid 2 to smithi15186934-7 2024-09-13T14:34:44.648 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 1920 (ObjNum 1184 snap 402 seq_num 1184) dirty exists 2024-09-13T14:34:44.648 INFO:tasks.rados.rados.0.smithi151.stdout:3316: finishing write tid 1 to smithi15186934-15 2024-09-13T14:34:44.648 INFO:tasks.rados.rados.0.smithi151.stdout:3314: expect (ObjNum 1232 snap 414 seq_num 1232) 2024-09-13T14:34:44.788 INFO:tasks.rados.rados.0.smithi151.stdout:3316: finishing write tid 2 to smithi15186934-15 2024-09-13T14:34:44.788 INFO:tasks.rados.rados.0.smithi151.stdout:3315: expect (ObjNum 1241 snap 416 seq_num 1241) 2024-09-13T14:34:44.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:44 smithi151 ceph-mon[108966]: pgmap v957: 89 pgs: 89 active+clean; 309 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 329 KiB/s wr, 3 op/s 2024-09-13T14:34:44.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:44 smithi151 ceph-mon[108966]: osdmap e893: 8 total, 8 up, 8 in 2024-09-13T14:34:44.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:44 smithi151 ceph-mon[108966]: osdmap e894: 8 total, 8 up, 8 in 2024-09-13T14:34:44.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:44 smithi151 ceph-mon[104450]: pgmap v957: 89 pgs: 89 active+clean; 309 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 329 KiB/s wr, 3 op/s 2024-09-13T14:34:44.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:44 smithi151 ceph-mon[104450]: osdmap e893: 8 total, 8 up, 8 in 2024-09-13T14:34:44.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:44 smithi151 ceph-mon[104450]: osdmap e894: 8 total, 8 up, 8 in 2024-09-13T14:34:44.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:34:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:34:44] "GET /metrics HTTP/1.1" 200 36507 "" "Prometheus/2.51.0" 2024-09-13T14:34:44.919 INFO:tasks.rados.rados.0.smithi151.stdout:3316: finishing write tid 3 to smithi15186934-15 2024-09-13T14:34:44.919 INFO:tasks.rados.rados.0.smithi151.stdout:3316: finishing write tid 4 to smithi15186934-15 2024-09-13T14:34:44.919 INFO:tasks.rados.rados.0.smithi151.stdout:3316: finishing write tid 5 to smithi15186934-15 2024-09-13T14:34:44.919 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 15 v 1219 (ObjNum 1243 snap 418 seq_num 1243) dirty exists 2024-09-13T14:34:44.919 INFO:tasks.rados.rados.0.smithi151.stdout:3316: left oid 15 (ObjNum 1243 snap 418 seq_num 1243) 2024-09-13T14:34:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:44 smithi184 ceph-mon[92527]: pgmap v957: 89 pgs: 89 active+clean; 309 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 329 KiB/s wr, 3 op/s 2024-09-13T14:34:45.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:44 smithi184 ceph-mon[92527]: osdmap e893: 8 total, 8 up, 8 in 2024-09-13T14:34:45.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:44 smithi184 ceph-mon[92527]: osdmap e894: 8 total, 8 up, 8 in 2024-09-13T14:34:45.627 INFO:tasks.rados.rados.0.smithi151.stdout:3313: done (4 left) 2024-09-13T14:34:45.627 INFO:tasks.rados.rados.0.smithi151.stdout:3314: done (3 left) 2024-09-13T14:34:45.627 INFO:tasks.rados.rados.0.smithi151.stdout:3315: done (2 left) 2024-09-13T14:34:45.627 INFO:tasks.rados.rados.0.smithi151.stdout:3316: done (1 left) 2024-09-13T14:34:45.628 INFO:tasks.rados.rados.0.smithi151.stdout:3317: done (0 left) 2024-09-13T14:34:45.628 INFO:tasks.rados.rados.0.smithi151.stdout:3318: write oid 20 current snap is 419 2024-09-13T14:34:45.628 INFO:tasks.rados.rados.0.smithi151.stdout:3318: seq_num 1244 ranges {507908=575295,1829482=631762} 2024-09-13T14:34:45.633 INFO:tasks.rados.rados.0.smithi151.stdout:3318: writing smithi15186934-20 from 507908 to 1083203 tid 1 2024-09-13T14:34:45.640 INFO:tasks.rados.rados.0.smithi151.stdout:3318: writing smithi15186934-20 from 1829482 to 2461244 tid 2 2024-09-13T14:34:45.641 INFO:tasks.rados.rados.0.smithi151.stdout:3319: read oid 32 snap -1 2024-09-13T14:34:45.641 INFO:tasks.rados.rados.0.smithi151.stdout:3319: expect (ObjNum 1197 snap 404 seq_num 1197) 2024-09-13T14:34:45.641 INFO:tasks.rados.rados.0.smithi151.stdout:3320: snap_create 2024-09-13T14:34:45.641 INFO:tasks.rados.rados.0.smithi151.stdout:3318: finishing write tid 1 to smithi15186934-20 2024-09-13T14:34:45.644 INFO:tasks.rados.rados.0.smithi151.stdout:3319: expect (ObjNum 1197 snap 404 seq_num 1197) 2024-09-13T14:34:45.712 INFO:tasks.rados.rados.0.smithi151.stdout:3318: finishing write tid 2 to smithi15186934-20 2024-09-13T14:34:45.712 INFO:tasks.rados.rados.0.smithi151.stdout:3318: finishing write tid 3 to smithi15186934-20 2024-09-13T14:34:45.712 INFO:tasks.rados.rados.0.smithi151.stdout:3318: finishing write tid 4 to smithi15186934-20 2024-09-13T14:34:45.713 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 900 (ObjNum 1244 snap 419 seq_num 1244) dirty exists 2024-09-13T14:34:45.713 INFO:tasks.rados.rados.0.smithi151.stdout:3318: left oid 20 (ObjNum 1244 snap 419 seq_num 1244) 2024-09-13T14:34:46.633 INFO:tasks.rados.rados.0.smithi151.stdout:3318: done (2 left) 2024-09-13T14:34:46.633 INFO:tasks.rados.rados.0.smithi151.stdout:3319: done (1 left) 2024-09-13T14:34:46.633 INFO:tasks.rados.rados.0.smithi151.stdout:3320: done (0 left) 2024-09-13T14:34:46.633 INFO:tasks.rados.rados.0.smithi151.stdout:3321: rollback oid 49 current snap is 420 2024-09-13T14:34:46.633 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 49 to 402 2024-09-13T14:34:46.633 INFO:tasks.rados.rados.0.smithi151.stdout:3322: delete oid 35 current snap is 420 2024-09-13T14:34:46.635 INFO:tasks.rados.rados.0.smithi151.stdout:3322: done (1 left) 2024-09-13T14:34:46.636 INFO:tasks.rados.rados.0.smithi151.stdout:3323: rollback oid 15 current snap is 420 2024-09-13T14:34:46.636 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 15 to 417 2024-09-13T14:34:46.636 INFO:tasks.rados.rados.0.smithi151.stdout:3324: snap_create 2024-09-13T14:34:46.636 INFO:tasks.rados.rados.0.smithi151.stdout:3321: finishing rollback tid 1 to smithi15186934-49 2024-09-13T14:34:46.636 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 49 v 1400 (ObjNum 49 snap 0 seq_num 3093250304) dirty dne 2024-09-13T14:34:46.640 INFO:tasks.rados.rados.0.smithi151.stdout:3323: finishing rollback tid 0 to smithi15186934-15 2024-09-13T14:34:46.640 INFO:tasks.rados.rados.0.smithi151.stdout:3323: finishing rollback tid 1 to smithi15186934-15 2024-09-13T14:34:46.640 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 15 v 1222 (ObjNum 49 snap 0 seq_num 3092642944) dirty dne 2024-09-13T14:34:47.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:46 smithi184 ceph-mon[92527]: pgmap v960: 89 pgs: 89 active+clean; 304 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 329 KiB/s wr, 3 op/s 2024-09-13T14:34:47.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:46 smithi184 ceph-mon[92527]: osdmap e895: 8 total, 8 up, 8 in 2024-09-13T14:34:47.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:47.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:34:47.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:46 smithi151 ceph-mon[108966]: pgmap v960: 89 pgs: 89 active+clean; 304 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 329 KiB/s wr, 3 op/s 2024-09-13T14:34:47.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:46 smithi151 ceph-mon[108966]: osdmap e895: 8 total, 8 up, 8 in 2024-09-13T14:34:47.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:47.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:34:47.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:46 smithi151 ceph-mon[104450]: pgmap v960: 89 pgs: 89 active+clean; 304 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 329 KiB/s wr, 3 op/s 2024-09-13T14:34:47.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:46 smithi151 ceph-mon[104450]: osdmap e895: 8 total, 8 up, 8 in 2024-09-13T14:34:47.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:47.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:34:47.637 INFO:tasks.rados.rados.0.smithi151.stdout:3321: done (2 left) 2024-09-13T14:34:47.637 INFO:tasks.rados.rados.0.smithi151.stdout:3323: done (1 left) 2024-09-13T14:34:47.637 INFO:tasks.rados.rados.0.smithi151.stdout:3324: done (0 left) 2024-09-13T14:34:47.638 INFO:tasks.rados.rados.0.smithi151.stdout:3325: snap_create 2024-09-13T14:34:47.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:47 smithi184 ceph-mon[92527]: osdmap e896: 8 total, 8 up, 8 in 2024-09-13T14:34:47.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:47 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:34:47.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:47 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:34:48.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:47 smithi151 ceph-mon[104450]: osdmap e896: 8 total, 8 up, 8 in 2024-09-13T14:34:48.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:47 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:34:48.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:47 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:34:48.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:47 smithi151 ceph-mon[108966]: osdmap e896: 8 total, 8 up, 8 in 2024-09-13T14:34:48.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:47 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:34:48.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:47 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:34:48.642 INFO:tasks.rados.rados.0.smithi151.stdout:3325: done (0 left) 2024-09-13T14:34:48.642 INFO:tasks.rados.rados.0.smithi151.stdout:3326: snap_remove snap 412 2024-09-13T14:34:48.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:48 smithi151 ceph-mon[104450]: pgmap v963: 89 pgs: 89 active+clean; 305 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 563 KiB/s wr, 7 op/s 2024-09-13T14:34:48.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:48 smithi151 ceph-mon[104450]: osdmap e897: 8 total, 8 up, 8 in 2024-09-13T14:34:48.947 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:48 smithi151 ceph-mon[108966]: pgmap v963: 89 pgs: 89 active+clean; 305 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 563 KiB/s wr, 7 op/s 2024-09-13T14:34:48.947 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:48 smithi151 ceph-mon[108966]: osdmap e897: 8 total, 8 up, 8 in 2024-09-13T14:34:49.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:48 smithi184 ceph-mon[92527]: pgmap v963: 89 pgs: 89 active+clean; 305 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 563 KiB/s wr, 7 op/s 2024-09-13T14:34:49.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:48 smithi184 ceph-mon[92527]: osdmap e897: 8 total, 8 up, 8 in 2024-09-13T14:34:49.627 INFO:tasks.rados.rados.0.smithi151.stdout:3326: done (0 left) 2024-09-13T14:34:49.628 INFO:tasks.rados.rados.0.smithi151.stdout:3327: snap_remove snap 411 2024-09-13T14:34:49.919 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:49 smithi184 ceph-mon[92527]: osdmap e898: 8 total, 8 up, 8 in 2024-09-13T14:34:49.919 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:49 smithi184 ceph-mon[92527]: osdmap e899: 8 total, 8 up, 8 in 2024-09-13T14:34:50.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:49 smithi151 ceph-mon[104450]: osdmap e898: 8 total, 8 up, 8 in 2024-09-13T14:34:50.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:49 smithi151 ceph-mon[104450]: osdmap e899: 8 total, 8 up, 8 in 2024-09-13T14:34:50.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:49 smithi151 ceph-mon[108966]: osdmap e898: 8 total, 8 up, 8 in 2024-09-13T14:34:50.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:49 smithi151 ceph-mon[108966]: osdmap e899: 8 total, 8 up, 8 in 2024-09-13T14:34:50.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:34:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:34:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:34:50.634 INFO:tasks.rados.rados.0.smithi151.stdout:3327: done (0 left) 2024-09-13T14:34:50.634 INFO:tasks.rados.rados.0.smithi151.stdout:3328: rollback oid 50 current snap is 422 2024-09-13T14:34:50.634 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 50 to 379 2024-09-13T14:34:50.634 INFO:tasks.rados.rados.0.smithi151.stdout:3329: rollback oid 12 current snap is 422 2024-09-13T14:34:50.634 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 12 to 403 2024-09-13T14:34:50.634 INFO:tasks.rados.rados.0.smithi151.stdout:3330: delete oid 33 current snap is 422 2024-09-13T14:34:50.638 INFO:tasks.rados.rados.0.smithi151.stdout:3329: finishing rollback tid 0 to smithi15186934-12 2024-09-13T14:34:50.639 INFO:tasks.rados.rados.0.smithi151.stdout:3329: finishing rollback tid 1 to smithi15186934-12 2024-09-13T14:34:50.639 INFO:tasks.rados.rados.0.smithi151.stdout:3329: finishing rollback tid 2 to smithi15186934-12 2024-09-13T14:34:50.639 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 12 v 694 (ObjNum 1170 snap 397 seq_num 1170) dirty exists 2024-09-13T14:34:50.641 INFO:tasks.rados.rados.0.smithi151.stdout:3329: done (2 left) 2024-09-13T14:34:50.641 INFO:tasks.rados.rados.0.smithi151.stdout:3330: done (1 left) 2024-09-13T14:34:50.641 INFO:tasks.rados.rados.0.smithi151.stdout:3331: snap_create 2024-09-13T14:34:50.641 INFO:tasks.rados.rados.0.smithi151.stdout:3328: finishing rollback tid 0 to smithi15186934-50 2024-09-13T14:34:50.642 INFO:tasks.rados.rados.0.smithi151.stdout:3328: finishing rollback tid 1 to smithi15186934-50 2024-09-13T14:34:50.642 INFO:tasks.rados.rados.0.smithi151.stdout:3328: finishing rollback tid 2 to smithi15186934-50 2024-09-13T14:34:50.642 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 50 v 1392 (ObjNum 1103 snap 370 seq_num 1103) dirty exists 2024-09-13T14:34:51.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:50 smithi184 ceph-mon[92527]: pgmap v966: 89 pgs: 89 active+clean; 308 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 924 KiB/s wr, 9 op/s 2024-09-13T14:34:51.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:50 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:34:51.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:50 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:34:51.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:50 smithi184 ceph-mon[92527]: osdmap e900: 8 total, 8 up, 8 in 2024-09-13T14:34:51.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:50 smithi151 ceph-mon[104450]: pgmap v966: 89 pgs: 89 active+clean; 308 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 924 KiB/s wr, 9 op/s 2024-09-13T14:34:51.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:50 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:34:51.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:50 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:34:51.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:50 smithi151 ceph-mon[104450]: osdmap e900: 8 total, 8 up, 8 in 2024-09-13T14:34:51.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:50 smithi151 ceph-mon[108966]: pgmap v966: 89 pgs: 89 active+clean; 308 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 924 KiB/s wr, 9 op/s 2024-09-13T14:34:51.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:50 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:34:51.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:50 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:34:51.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:50 smithi151 ceph-mon[108966]: osdmap e900: 8 total, 8 up, 8 in 2024-09-13T14:34:51.634 INFO:tasks.rados.rados.0.smithi151.stdout:3328: done (1 left) 2024-09-13T14:34:51.634 INFO:tasks.rados.rados.0.smithi151.stdout:3331: done (0 left) 2024-09-13T14:34:51.634 INFO:tasks.rados.rados.0.smithi151.stdout:3332: snap_create 2024-09-13T14:34:52.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:52 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:34:52.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:52 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:34:52.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:52 smithi184 ceph-mon[92527]: pgmap v969: 89 pgs: 89 active+clean; 308 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 361 KiB/s wr, 4 op/s 2024-09-13T14:34:52.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:52 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:34:52.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:52 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:34:52.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:52 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:34:52.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:52 smithi184 ceph-mon[92527]: osdmap e901: 8 total, 8 up, 8 in 2024-09-13T14:34:52.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:52 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:34:52.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:52 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:34:52.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:52 smithi151 ceph-mon[104450]: pgmap v969: 89 pgs: 89 active+clean; 308 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 361 KiB/s wr, 4 op/s 2024-09-13T14:34:52.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:52 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:34:52.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:52 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:34:52.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:52 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:34:52.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:52 smithi151 ceph-mon[104450]: osdmap e901: 8 total, 8 up, 8 in 2024-09-13T14:34:52.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:52 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:34:52.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:52 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:34:52.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:52 smithi151 ceph-mon[108966]: pgmap v969: 89 pgs: 89 active+clean; 308 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 361 KiB/s wr, 4 op/s 2024-09-13T14:34:52.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:52 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:34:52.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:52 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:34:52.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:52 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:34:52.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:52 smithi151 ceph-mon[108966]: osdmap e901: 8 total, 8 up, 8 in 2024-09-13T14:34:52.636 INFO:tasks.rados.rados.0.smithi151.stdout:3332: done (0 left) 2024-09-13T14:34:52.636 INFO:tasks.rados.rados.0.smithi151.stdout:3333: snap_remove snap 419 2024-09-13T14:34:53.644 INFO:tasks.rados.rados.0.smithi151.stdout:3333: done (0 left) 2024-09-13T14:34:53.644 INFO:tasks.rados.rados.0.smithi151.stdout:3334: snap_remove snap 417 2024-09-13T14:34:54.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:53 smithi184 ceph-mon[92527]: osdmap e902: 8 total, 8 up, 8 in 2024-09-13T14:34:54.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:53 smithi151 ceph-mon[104450]: osdmap e902: 8 total, 8 up, 8 in 2024-09-13T14:34:54.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:53 smithi151 ceph-mon[108966]: osdmap e902: 8 total, 8 up, 8 in 2024-09-13T14:34:54.645 INFO:tasks.rados.rados.0.smithi151.stdout:3334: done (0 left) 2024-09-13T14:34:54.645 INFO:tasks.rados.rados.0.smithi151.stdout:3335: snap_create 2024-09-13T14:34:54.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:54 smithi151 ceph-mon[108966]: pgmap v972: 89 pgs: 89 active+clean; 306 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 0 B/s wr, 2 op/s 2024-09-13T14:34:54.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:54 smithi151 ceph-mon[108966]: osdmap e903: 8 total, 8 up, 8 in 2024-09-13T14:34:54.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:54 smithi151 ceph-mon[104450]: pgmap v972: 89 pgs: 89 active+clean; 306 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 0 B/s wr, 2 op/s 2024-09-13T14:34:54.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:54 smithi151 ceph-mon[104450]: osdmap e903: 8 total, 8 up, 8 in 2024-09-13T14:34:54.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:34:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:34:54] "GET /metrics HTTP/1.1" 200 36502 "" "Prometheus/2.51.0" 2024-09-13T14:34:55.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:54 smithi184 ceph-mon[92527]: pgmap v972: 89 pgs: 89 active+clean; 306 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s rd, 0 B/s wr, 2 op/s 2024-09-13T14:34:55.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:54 smithi184 ceph-mon[92527]: osdmap e903: 8 total, 8 up, 8 in 2024-09-13T14:34:55.652 INFO:tasks.rados.rados.0.smithi151.stdout:3335: done (0 left) 2024-09-13T14:34:55.652 INFO:tasks.rados.rados.0.smithi151.stdout:3336: snap_remove snap 418 2024-09-13T14:34:56.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:55 smithi184 ceph-mon[92527]: osdmap e904: 8 total, 8 up, 8 in 2024-09-13T14:34:56.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:55 smithi151 ceph-mon[104450]: osdmap e904: 8 total, 8 up, 8 in 2024-09-13T14:34:56.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:55 smithi151 ceph-mon[108966]: osdmap e904: 8 total, 8 up, 8 in 2024-09-13T14:34:56.651 INFO:tasks.rados.rados.0.smithi151.stdout:3336: done (0 left) 2024-09-13T14:34:56.651 INFO:tasks.rados.rados.0.smithi151.stdout:3337: read oid 42 snap -1 2024-09-13T14:34:56.651 INFO:tasks.rados.rados.0.smithi151.stdout:3337: expect (ObjNum 1174 snap 399 seq_num 1174) 2024-09-13T14:34:56.651 INFO:tasks.rados.rados.0.smithi151.stdout:3338: rollback oid 23 current snap is 425 2024-09-13T14:34:56.651 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 23 to 422 2024-09-13T14:34:56.651 INFO:tasks.rados.rados.0.smithi151.stdout:3339: delete oid 41 current snap is 425 2024-09-13T14:34:56.656 INFO:tasks.rados.rados.0.smithi151.stdout:3339: done (2 left) 2024-09-13T14:34:56.656 INFO:tasks.rados.rados.0.smithi151.stdout:3340: delete oid 22 current snap is 425 2024-09-13T14:34:56.658 INFO:tasks.rados.rados.0.smithi151.stdout:3338: finishing rollback tid 0 to smithi15186934-23 2024-09-13T14:34:56.658 INFO:tasks.rados.rados.0.smithi151.stdout:3338: finishing rollback tid 1 to smithi15186934-23 2024-09-13T14:34:56.658 INFO:tasks.rados.rados.0.smithi151.stdout:3338: finishing rollback tid 2 to smithi15186934-23 2024-09-13T14:34:56.658 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 23 v 1542 (ObjNum 1216 snap 409 seq_num 1216) dirty exists 2024-09-13T14:34:56.658 INFO:tasks.rados.rados.0.smithi151.stdout:3338: done (2 left) 2024-09-13T14:34:56.659 INFO:tasks.rados.rados.0.smithi151.stdout:3340: done (1 left) 2024-09-13T14:34:56.659 INFO:tasks.rados.rados.0.smithi151.stdout:3341: read oid 19 snap 402 2024-09-13T14:34:56.659 INFO:tasks.rados.rados.0.smithi151.stdout:3341: expect (ObjNum 1148 snap 389 seq_num 1148) 2024-09-13T14:34:56.659 INFO:tasks.rados.rados.0.smithi151.stdout:3342: delete oid 20 current snap is 425 2024-09-13T14:34:56.661 INFO:tasks.rados.rados.0.smithi151.stdout:3342: done (2 left) 2024-09-13T14:34:56.661 INFO:tasks.rados.rados.0.smithi151.stdout:3343: write oid 12 current snap is 425 2024-09-13T14:34:56.661 INFO:tasks.rados.rados.0.smithi151.stdout:3343: seq_num 1250 ranges {608750=743142,1780374=642769} 2024-09-13T14:34:56.669 INFO:tasks.rados.rados.0.smithi151.stdout:3343: writing smithi15186934-12 from 608750 to 1351892 tid 1 2024-09-13T14:34:56.675 INFO:tasks.rados.rados.0.smithi151.stdout:3343: writing smithi15186934-12 from 1780374 to 2423143 tid 2 2024-09-13T14:34:56.675 INFO:tasks.rados.rados.0.smithi151.stdout:3344: snap_create 2024-09-13T14:34:56.675 INFO:tasks.rados.rados.0.smithi151.stdout:3337: expect (ObjNum 1174 snap 399 seq_num 1174) 2024-09-13T14:34:56.796 INFO:tasks.rados.rados.0.smithi151.stdout:3341: expect (ObjNum 1148 snap 389 seq_num 1148) 2024-09-13T14:34:56.901 INFO:tasks.rados.rados.0.smithi151.stdout:3343: finishing write tid 1 to smithi15186934-12 2024-09-13T14:34:56.901 INFO:tasks.rados.rados.0.smithi151.stdout:3343: finishing write tid 2 to smithi15186934-12 2024-09-13T14:34:56.901 INFO:tasks.rados.rados.0.smithi151.stdout:3343: finishing write tid 3 to smithi15186934-12 2024-09-13T14:34:56.902 INFO:tasks.rados.rados.0.smithi151.stdout:3343: finishing write tid 4 to smithi15186934-12 2024-09-13T14:34:56.902 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 12 v 704 (ObjNum 1250 snap 425 seq_num 1250) dirty exists 2024-09-13T14:34:56.902 INFO:tasks.rados.rados.0.smithi151.stdout:3343: left oid 12 (ObjNum 1250 snap 425 seq_num 1250) 2024-09-13T14:34:57.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:56 smithi184 ceph-mon[92527]: pgmap v975: 89 pgs: 89 active+clean; 306 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:34:57.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:56 smithi184 ceph-mon[92527]: osdmap e905: 8 total, 8 up, 8 in 2024-09-13T14:34:57.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:56 smithi184 ceph-mon[92527]: osdmap e906: 8 total, 8 up, 8 in 2024-09-13T14:34:57.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:56 smithi151 ceph-mon[104450]: pgmap v975: 89 pgs: 89 active+clean; 306 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:34:57.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:56 smithi151 ceph-mon[104450]: osdmap e905: 8 total, 8 up, 8 in 2024-09-13T14:34:57.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:56 smithi151 ceph-mon[104450]: osdmap e906: 8 total, 8 up, 8 in 2024-09-13T14:34:57.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:56 smithi151 ceph-mon[108966]: pgmap v975: 89 pgs: 89 active+clean; 306 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:34:57.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:56 smithi151 ceph-mon[108966]: osdmap e905: 8 total, 8 up, 8 in 2024-09-13T14:34:57.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:56 smithi151 ceph-mon[108966]: osdmap e906: 8 total, 8 up, 8 in 2024-09-13T14:34:57.671 INFO:tasks.rados.rados.0.smithi151.stdout:3337: done (3 left) 2024-09-13T14:34:57.671 INFO:tasks.rados.rados.0.smithi151.stdout:3341: done (2 left) 2024-09-13T14:34:57.671 INFO:tasks.rados.rados.0.smithi151.stdout:3343: done (1 left) 2024-09-13T14:34:57.671 INFO:tasks.rados.rados.0.smithi151.stdout:3344: done (0 left) 2024-09-13T14:34:57.671 INFO:tasks.rados.rados.0.smithi151.stdout:3345: read oid 33 snap 422 2024-09-13T14:34:57.671 INFO:tasks.rados.rados.0.smithi151.stdout:3345: expect deleted 2024-09-13T14:34:57.671 INFO:tasks.rados.rados.0.smithi151.stdout:3346: read oid 22 snap -1 2024-09-13T14:34:57.672 INFO:tasks.rados.rados.0.smithi151.stdout:3346: expect deleted 2024-09-13T14:34:57.672 INFO:tasks.rados.rados.0.smithi151.stdout:3347: write oid 8 current snap is 426 2024-09-13T14:34:57.672 INFO:tasks.rados.rados.0.smithi151.stdout:3347: seq_num 1251 ranges {625557=418391,1838856=786427} 2024-09-13T14:34:57.676 INFO:tasks.rados.rados.0.smithi151.stdout:3347: writing smithi15186934-8 from 625557 to 1043948 tid 1 2024-09-13T14:34:57.685 INFO:tasks.rados.rados.0.smithi151.stdout:3347: writing smithi15186934-8 from 1838856 to 2625283 tid 2 2024-09-13T14:34:57.686 INFO:tasks.rados.rados.0.smithi151.stdout:3348: rollback oid 42 current snap is 426 2024-09-13T14:34:57.686 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 42 to 413 2024-09-13T14:34:57.686 INFO:tasks.rados.rados.0.smithi151.stdout:3347: finishing write tid 1 to smithi15186934-8 2024-09-13T14:34:57.686 INFO:tasks.rados.rados.0.smithi151.stdout:3345: done (3 left) 2024-09-13T14:34:57.686 INFO:tasks.rados.rados.0.smithi151.stdout:3346: done (2 left) 2024-09-13T14:34:57.686 INFO:tasks.rados.rados.0.smithi151.stdout:3349: read oid 38 snap -1 2024-09-13T14:34:57.686 INFO:tasks.rados.rados.0.smithi151.stdout:3349: expect (ObjNum 1202 snap 407 seq_num 1202) 2024-09-13T14:34:57.686 INFO:tasks.rados.rados.0.smithi151.stdout:3350: snap_remove snap 403 2024-09-13T14:34:58.680 INFO:tasks.rados.rados.0.smithi151.stdout:3350: done (3 left) 2024-09-13T14:34:58.680 INFO:tasks.rados.rados.0.smithi151.stdout:3351: rollback oid 11 current snap is 426 2024-09-13T14:34:58.680 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 11 to 406 2024-09-13T14:34:58.681 INFO:tasks.rados.rados.0.smithi151.stdout:3348: finishing rollback tid 0 to smithi15186934-42 2024-09-13T14:34:58.681 INFO:tasks.rados.rados.0.smithi151.stdout:3348: finishing rollback tid 1 to smithi15186934-42 2024-09-13T14:34:58.681 INFO:tasks.rados.rados.0.smithi151.stdout:3348: finishing rollback tid 2 to smithi15186934-42 2024-09-13T14:34:58.681 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 1230 (ObjNum 1174 snap 399 seq_num 1174) dirty exists 2024-09-13T14:34:58.681 INFO:tasks.rados.rados.0.smithi151.stdout:3347: finishing write tid 2 to smithi15186934-8 2024-09-13T14:34:58.681 INFO:tasks.rados.rados.0.smithi151.stdout:3347: finishing write tid 3 to smithi15186934-8 2024-09-13T14:34:58.681 INFO:tasks.rados.rados.0.smithi151.stdout:3347: finishing write tid 4 to smithi15186934-8 2024-09-13T14:34:58.681 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 1040 (ObjNum 1251 snap 426 seq_num 1251) dirty exists 2024-09-13T14:34:58.681 INFO:tasks.rados.rados.0.smithi151.stdout:3347: left oid 8 (ObjNum 1251 snap 426 seq_num 1251) 2024-09-13T14:34:58.681 INFO:tasks.rados.rados.0.smithi151.stdout:3349: expect (ObjNum 1202 snap 407 seq_num 1202) 2024-09-13T14:34:58.760 INFO:tasks.rados.rados.0.smithi151.stdout:3351: finishing rollback tid 1 to smithi15186934-11 2024-09-13T14:34:58.760 INFO:tasks.rados.rados.0.smithi151.stdout:3351: finishing rollback tid 2 to smithi15186934-11 2024-09-13T14:34:58.760 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 11 v 857 (ObjNum 1167 snap 396 seq_num 1167) dirty exists 2024-09-13T14:34:58.760 INFO:tasks.rados.rados.0.smithi151.stdout:3347: done (3 left) 2024-09-13T14:34:58.760 INFO:tasks.rados.rados.0.smithi151.stdout:3348: done (2 left) 2024-09-13T14:34:58.760 INFO:tasks.rados.rados.0.smithi151.stdout:3349: done (1 left) 2024-09-13T14:34:58.760 INFO:tasks.rados.rados.0.smithi151.stdout:3351: done (0 left) 2024-09-13T14:34:58.760 INFO:tasks.rados.rados.0.smithi151.stdout:3352: read oid 38 snap -1 2024-09-13T14:34:58.761 INFO:tasks.rados.rados.0.smithi151.stdout:3352: expect (ObjNum 1202 snap 407 seq_num 1202) 2024-09-13T14:34:58.761 INFO:tasks.rados.rados.0.smithi151.stdout:3353: rollback oid 28 current snap is 426 2024-09-13T14:34:58.761 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 28 to 402 2024-09-13T14:34:58.761 INFO:tasks.rados.rados.0.smithi151.stdout:3354: read oid 45 snap -1 2024-09-13T14:34:58.761 INFO:tasks.rados.rados.0.smithi151.stdout:3354: expect deleted 2024-09-13T14:34:58.761 INFO:tasks.rados.rados.0.smithi151.stdout:3355: read oid 34 snap -1 2024-09-13T14:34:58.761 INFO:tasks.rados.rados.0.smithi151.stdout:3355: expect deleted 2024-09-13T14:34:58.761 INFO:tasks.rados.rados.0.smithi151.stdout:3356: delete oid 31 current snap is 426 2024-09-13T14:34:58.763 INFO:tasks.rados.rados.0.smithi151.stdout:3353: finishing rollback tid 0 to smithi15186934-28 2024-09-13T14:34:58.763 INFO:tasks.rados.rados.0.smithi151.stdout:3353: finishing rollback tid 1 to smithi15186934-28 2024-09-13T14:34:58.763 INFO:tasks.rados.rados.0.smithi151.stdout:3352: expect (ObjNum 1202 snap 407 seq_num 1202) 2024-09-13T14:34:58.836 INFO:tasks.rados.rados.0.smithi151.stdout:3353: finishing rollback tid 2 to smithi15186934-28 2024-09-13T14:34:58.836 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 28 v 1242 (ObjNum 1163 snap 396 seq_num 1163) dirty exists 2024-09-13T14:34:58.836 INFO:tasks.rados.rados.0.smithi151.stdout:3352: done (4 left) 2024-09-13T14:34:58.836 INFO:tasks.rados.rados.0.smithi151.stdout:3353: done (3 left) 2024-09-13T14:34:58.836 INFO:tasks.rados.rados.0.smithi151.stdout:3354: done (2 left) 2024-09-13T14:34:58.836 INFO:tasks.rados.rados.0.smithi151.stdout:3355: done (1 left) 2024-09-13T14:34:58.836 INFO:tasks.rados.rados.0.smithi151.stdout:3356: done (0 left) 2024-09-13T14:34:58.837 INFO:tasks.rados.rados.0.smithi151.stdout:3357: delete oid 47 current snap is 426 2024-09-13T14:34:58.838 INFO:tasks.rados.rados.0.smithi151.stdout:3357: done (0 left) 2024-09-13T14:34:58.838 INFO:tasks.rados.rados.0.smithi151.stdout:3358: snap_remove snap 420 2024-09-13T14:34:59.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:58 smithi184 ceph-mon[92527]: pgmap v978: 89 pgs: 89 active+clean; 309 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.5 MiB/s rd, 0 B/s wr, 3 op/s 2024-09-13T14:34:59.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:58 smithi184 ceph-mon[92527]: osdmap e907: 8 total, 8 up, 8 in 2024-09-13T14:34:59.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:58 smithi151 ceph-mon[104450]: pgmap v978: 89 pgs: 89 active+clean; 309 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.5 MiB/s rd, 0 B/s wr, 3 op/s 2024-09-13T14:34:59.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:58 smithi151 ceph-mon[104450]: osdmap e907: 8 total, 8 up, 8 in 2024-09-13T14:34:59.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:58 smithi151 ceph-mon[108966]: pgmap v978: 89 pgs: 89 active+clean; 309 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.5 MiB/s rd, 0 B/s wr, 3 op/s 2024-09-13T14:34:59.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:58 smithi151 ceph-mon[108966]: osdmap e907: 8 total, 8 up, 8 in 2024-09-13T14:34:59.634 INFO:tasks.rados.rados.0.smithi151.stdout:3358: done (0 left) 2024-09-13T14:34:59.634 INFO:tasks.rados.rados.0.smithi151.stdout:3359: snap_remove snap 396 2024-09-13T14:35:00.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:59 smithi184 ceph-mon[92527]: osdmap e908: 8 total, 8 up, 8 in 2024-09-13T14:35:00.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:34:59 smithi184 ceph-mon[92527]: osdmap e909: 8 total, 8 up, 8 in 2024-09-13T14:35:00.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:59 smithi151 ceph-mon[104450]: osdmap e908: 8 total, 8 up, 8 in 2024-09-13T14:35:00.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:34:59 smithi151 ceph-mon[104450]: osdmap e909: 8 total, 8 up, 8 in 2024-09-13T14:35:00.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:59 smithi151 ceph-mon[108966]: osdmap e908: 8 total, 8 up, 8 in 2024-09-13T14:35:00.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:34:59 smithi151 ceph-mon[108966]: osdmap e909: 8 total, 8 up, 8 in 2024-09-13T14:35:00.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:35:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:35:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:35:00.632 INFO:tasks.rados.rados.0.smithi151.stdout:3359: done (0 left) 2024-09-13T14:35:00.632 INFO:tasks.rados.rados.0.smithi151.stdout:3360: delete oid 3 current snap is 426 2024-09-13T14:35:00.643 INFO:tasks.rados.rados.0.smithi151.stdout:3360: done (0 left) 2024-09-13T14:35:00.644 INFO:tasks.rados.rados.0.smithi151.stdout:3361: write oid 22 current snap is 426 2024-09-13T14:35:00.644 INFO:tasks.rados.rados.0.smithi151.stdout:3361: seq_num 1255 ranges {692785=435740,1672784=793765,2897503=195282} 2024-09-13T14:35:00.650 INFO:tasks.rados.rados.0.smithi151.stdout:3361: writing smithi15186934-22 from 692785 to 1128525 tid 1 2024-09-13T14:35:00.657 INFO:tasks.rados.rados.0.smithi151.stdout:3361: writing smithi15186934-22 from 1672784 to 2466549 tid 2 2024-09-13T14:35:00.661 INFO:tasks.rados.rados.0.smithi151.stdout:3361: writing smithi15186934-22 from 2897503 to 3092785 tid 3 2024-09-13T14:35:00.661 INFO:tasks.rados.rados.0.smithi151.stdout:3362: write oid 45 current snap is 426 2024-09-13T14:35:00.661 INFO:tasks.rados.rados.0.smithi151.stdout:3362: seq_num 1256 ranges {709592=510989,1731266=653776,3041279=68313} 2024-09-13T14:35:00.667 INFO:tasks.rados.rados.0.smithi151.stdout:3362: writing smithi15186934-45 from 709592 to 1220581 tid 1 2024-09-13T14:35:00.673 INFO:tasks.rados.rados.0.smithi151.stdout:3362: writing smithi15186934-45 from 1731266 to 2385042 tid 2 2024-09-13T14:35:00.678 INFO:tasks.rados.rados.0.smithi151.stdout:3362: writing smithi15186934-45 from 3041279 to 3109592 tid 3 2024-09-13T14:35:00.678 INFO:tasks.rados.rados.0.smithi151.stdout:3363: write oid 10 current snap is 426 2024-09-13T14:35:00.678 INFO:tasks.rados.rados.0.smithi151.stdout:3361: finishing write tid 1 to smithi15186934-22 2024-09-13T14:35:00.678 INFO:tasks.rados.rados.0.smithi151.stdout:3361: finishing write tid 2 to smithi15186934-22 2024-09-13T14:35:00.678 INFO:tasks.rados.rados.0.smithi151.stdout:3361: finishing write tid 3 to smithi15186934-22 2024-09-13T14:35:00.678 INFO:tasks.rados.rados.0.smithi151.stdout:3361: finishing write tid 4 to smithi15186934-22 2024-09-13T14:35:00.678 INFO:tasks.rados.rados.0.smithi151.stdout:3361: finishing write tid 5 to smithi15186934-22 2024-09-13T14:35:00.678 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 22 v 685 (ObjNum 1255 snap 426 seq_num 1255) dirty exists 2024-09-13T14:35:00.678 INFO:tasks.rados.rados.0.smithi151.stdout:3361: left oid 22 (ObjNum 1255 snap 426 seq_num 1255) 2024-09-13T14:35:00.678 INFO:tasks.rados.rados.0.smithi151.stdout:3362: finishing write tid 1 to smithi15186934-45 2024-09-13T14:35:00.679 INFO:tasks.rados.rados.0.smithi151.stdout:3363: seq_num 1257 ranges {726399=586238,2073395=797434} 2024-09-13T14:35:00.685 INFO:tasks.rados.rados.0.smithi151.stdout:3363: writing smithi15186934-10 from 726399 to 1312637 tid 1 2024-09-13T14:35:00.694 INFO:tasks.rados.rados.0.smithi151.stdout:3363: writing smithi15186934-10 from 2073395 to 2870829 tid 2 2024-09-13T14:35:00.694 INFO:tasks.rados.rados.0.smithi151.stdout:3361: done (2 left) 2024-09-13T14:35:00.694 INFO:tasks.rados.rados.0.smithi151.stdout:3364: write oid 21 current snap is 426 2024-09-13T14:35:00.694 INFO:tasks.rados.rados.0.smithi151.stdout:3364: seq_num 1258 ranges {743206=661487,2131877=541092} 2024-09-13T14:35:00.701 INFO:tasks.rados.rados.0.smithi151.stdout:3364: writing smithi15186934-21 from 743206 to 1404693 tid 1 2024-09-13T14:35:00.709 INFO:tasks.rados.rados.0.smithi151.stdout:3364: writing smithi15186934-21 from 2131877 to 2672969 tid 2 2024-09-13T14:35:00.709 INFO:tasks.rados.rados.0.smithi151.stdout:3365: read oid 35 snap 425 2024-09-13T14:35:00.709 INFO:tasks.rados.rados.0.smithi151.stdout:3365: expect deleted 2024-09-13T14:35:00.709 INFO:tasks.rados.rados.0.smithi151.stdout:3366: delete oid 29 current snap is 426 2024-09-13T14:35:00.709 INFO:tasks.rados.rados.0.smithi151.stdout:3362: finishing write tid 2 to smithi15186934-45 2024-09-13T14:35:00.709 INFO:tasks.rados.rados.0.smithi151.stdout:3362: finishing write tid 3 to smithi15186934-45 2024-09-13T14:35:00.710 INFO:tasks.rados.rados.0.smithi151.stdout:3362: finishing write tid 4 to smithi15186934-45 2024-09-13T14:35:00.710 INFO:tasks.rados.rados.0.smithi151.stdout:3362: finishing write tid 5 to smithi15186934-45 2024-09-13T14:35:00.710 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 1607 (ObjNum 1256 snap 426 seq_num 1256) dirty exists 2024-09-13T14:35:00.710 INFO:tasks.rados.rados.0.smithi151.stdout:3362: left oid 45 (ObjNum 1256 snap 426 seq_num 1256) 2024-09-13T14:35:00.710 INFO:tasks.rados.rados.0.smithi151.stdout:3363: finishing write tid 1 to smithi15186934-10 2024-09-13T14:35:00.710 INFO:tasks.rados.rados.0.smithi151.stdout:3363: finishing write tid 2 to smithi15186934-10 2024-09-13T14:35:00.710 INFO:tasks.rados.rados.0.smithi151.stdout:3363: finishing write tid 3 to smithi15186934-10 2024-09-13T14:35:00.710 INFO:tasks.rados.rados.0.smithi151.stdout:3363: finishing write tid 4 to smithi15186934-10 2024-09-13T14:35:00.710 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 1231 (ObjNum 1257 snap 426 seq_num 1257) dirty exists 2024-09-13T14:35:00.710 INFO:tasks.rados.rados.0.smithi151.stdout:3363: left oid 10 (ObjNum 1257 snap 426 seq_num 1257) 2024-09-13T14:35:00.710 INFO:tasks.rados.rados.0.smithi151.stdout:3364: finishing write tid 1 to smithi15186934-21 2024-09-13T14:35:00.711 INFO:tasks.rados.rados.0.smithi151.stdout:3362: done (4 left) 2024-09-13T14:35:00.711 INFO:tasks.rados.rados.0.smithi151.stdout:3363: done (3 left) 2024-09-13T14:35:00.711 INFO:tasks.rados.rados.0.smithi151.stdout:3365: done (2 left) 2024-09-13T14:35:00.711 INFO:tasks.rados.rados.0.smithi151.stdout:3366: done (1 left) 2024-09-13T14:35:00.711 INFO:tasks.rados.rados.0.smithi151.stdout:3367: read oid 50 snap 406 2024-09-13T14:35:00.711 INFO:tasks.rados.rados.0.smithi151.stdout:3367: expect (ObjNum 1165 snap 396 seq_num 1165) 2024-09-13T14:35:00.711 INFO:tasks.rados.rados.0.smithi151.stdout:3368: write oid 16 current snap is 426 2024-09-13T14:35:00.711 INFO:tasks.rados.rados.0.smithi151.stdout:3368: seq_num 1260 ranges {776820=411985,1848841=544761,2918265=258555} 2024-09-13T14:35:00.717 INFO:tasks.rados.rados.0.smithi151.stdout:3368: writing smithi15186934-16 from 776820 to 1188805 tid 1 2024-09-13T14:35:00.724 INFO:tasks.rados.rados.0.smithi151.stdout:3368: writing smithi15186934-16 from 1848841 to 2393602 tid 2 2024-09-13T14:35:00.728 INFO:tasks.rados.rados.0.smithi151.stdout:3368: writing smithi15186934-16 from 2918265 to 3176820 tid 3 2024-09-13T14:35:00.728 INFO:tasks.rados.rados.0.smithi151.stdout:3369: snap_create 2024-09-13T14:35:00.728 INFO:tasks.rados.rados.0.smithi151.stdout:3364: finishing write tid 2 to smithi15186934-21 2024-09-13T14:35:00.728 INFO:tasks.rados.rados.0.smithi151.stdout:3364: finishing write tid 3 to smithi15186934-21 2024-09-13T14:35:00.728 INFO:tasks.rados.rados.0.smithi151.stdout:3367: expect (ObjNum 1165 snap 396 seq_num 1165) 2024-09-13T14:35:00.942 INFO:tasks.rados.rados.0.smithi151.stdout:3364: finishing write tid 4 to smithi15186934-21 2024-09-13T14:35:00.942 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 21 v 689 (ObjNum 1258 snap 426 seq_num 1258) dirty exists 2024-09-13T14:35:00.942 INFO:tasks.rados.rados.0.smithi151.stdout:3364: left oid 21 (ObjNum 1258 snap 426 seq_num 1258) 2024-09-13T14:35:00.942 INFO:tasks.rados.rados.0.smithi151.stdout:3368: finishing write tid 1 to smithi15186934-16 2024-09-13T14:35:00.942 INFO:tasks.rados.rados.0.smithi151.stdout:3368: finishing write tid 2 to smithi15186934-16 2024-09-13T14:35:00.942 INFO:tasks.rados.rados.0.smithi151.stdout:3368: finishing write tid 3 to smithi15186934-16 2024-09-13T14:35:00.942 INFO:tasks.rados.rados.0.smithi151.stdout:3368: finishing write tid 4 to smithi15186934-16 2024-09-13T14:35:00.943 INFO:tasks.rados.rados.0.smithi151.stdout:3368: finishing write tid 5 to smithi15186934-16 2024-09-13T14:35:00.943 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 16 v 930 (ObjNum 1260 snap 426 seq_num 1260) dirty exists 2024-09-13T14:35:00.943 INFO:tasks.rados.rados.0.smithi151.stdout:3368: left oid 16 (ObjNum 1260 snap 426 seq_num 1260) 2024-09-13T14:35:01.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:00 smithi184 ceph-mon[92527]: pgmap v981: 89 pgs: 89 active+clean; 305 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.8 MiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:35:01.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:00 smithi184 ceph-mon[92527]: osdmap e910: 8 total, 8 up, 8 in 2024-09-13T14:35:01.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:00 smithi151 ceph-mon[104450]: pgmap v981: 89 pgs: 89 active+clean; 305 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.8 MiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:35:01.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:00 smithi151 ceph-mon[104450]: osdmap e910: 8 total, 8 up, 8 in 2024-09-13T14:35:01.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:00 smithi151 ceph-mon[108966]: pgmap v981: 89 pgs: 89 active+clean; 305 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.8 MiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:35:01.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:00 smithi151 ceph-mon[108966]: osdmap e910: 8 total, 8 up, 8 in 2024-09-13T14:35:01.695 INFO:tasks.rados.rados.0.smithi151.stdout:3364: done (3 left) 2024-09-13T14:35:01.696 INFO:tasks.rados.rados.0.smithi151.stdout:3367: done (2 left) 2024-09-13T14:35:01.696 INFO:tasks.rados.rados.0.smithi151.stdout:3368: done (1 left) 2024-09-13T14:35:01.696 INFO:tasks.rados.rados.0.smithi151.stdout:3369: done (0 left) 2024-09-13T14:35:01.696 INFO:tasks.rados.rados.0.smithi151.stdout:3370: snap_remove snap 402 2024-09-13T14:35:02.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:01 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:02.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:02.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:01 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:02.700 INFO:tasks.rados.rados.0.smithi151.stdout:3370: done (0 left) 2024-09-13T14:35:02.700 INFO:tasks.rados.rados.0.smithi151.stdout:3371: read oid 7 snap 422 2024-09-13T14:35:02.700 INFO:tasks.rados.rados.0.smithi151.stdout:3371: expect (ObjNum 1184 snap 402 seq_num 1184) 2024-09-13T14:35:02.700 INFO:tasks.rados.rados.0.smithi151.stdout:3372: snap_create 2024-09-13T14:35:02.727 INFO:tasks.rados.rados.0.smithi151.stdout:3371: expect (ObjNum 1184 snap 402 seq_num 1184) 2024-09-13T14:35:03.009 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:02 smithi151 ceph-mon[104450]: pgmap v984: 89 pgs: 89 active+clean; 312 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 633 KiB/s wr, 4 op/s 2024-09-13T14:35:03.009 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:02 smithi151 ceph-mon[104450]: osdmap e911: 8 total, 8 up, 8 in 2024-09-13T14:35:03.010 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:02 smithi151 ceph-mon[108966]: pgmap v984: 89 pgs: 89 active+clean; 312 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 633 KiB/s wr, 4 op/s 2024-09-13T14:35:03.010 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:02 smithi151 ceph-mon[108966]: osdmap e911: 8 total, 8 up, 8 in 2024-09-13T14:35:03.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:02 smithi184 ceph-mon[92527]: pgmap v984: 89 pgs: 89 active+clean; 312 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 633 KiB/s wr, 4 op/s 2024-09-13T14:35:03.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:02 smithi184 ceph-mon[92527]: osdmap e911: 8 total, 8 up, 8 in 2024-09-13T14:35:03.705 INFO:tasks.rados.rados.0.smithi151.stdout:3371: done (1 left) 2024-09-13T14:35:03.705 INFO:tasks.rados.rados.0.smithi151.stdout:3372: done (0 left) 2024-09-13T14:35:03.705 INFO:tasks.rados.rados.0.smithi151.stdout:3373: read oid 18 snap -1 2024-09-13T14:35:03.705 INFO:tasks.rados.rados.0.smithi151.stdout:3373: expect (ObjNum 1203 snap 407 seq_num 1203) 2024-09-13T14:35:03.706 INFO:tasks.rados.rados.0.smithi151.stdout:3374: read oid 27 snap -1 2024-09-13T14:35:03.706 INFO:tasks.rados.rados.0.smithi151.stdout:3374: expect (ObjNum 1241 snap 416 seq_num 1241) 2024-09-13T14:35:03.706 INFO:tasks.rados.rados.0.smithi151.stdout:3375: read oid 38 snap -1 2024-09-13T14:35:03.706 INFO:tasks.rados.rados.0.smithi151.stdout:3375: expect (ObjNum 1202 snap 407 seq_num 1202) 2024-09-13T14:35:03.706 INFO:tasks.rados.rados.0.smithi151.stdout:3376: read oid 25 snap 409 2024-09-13T14:35:03.706 INFO:tasks.rados.rados.0.smithi151.stdout:3376: expect (ObjNum 1209 snap 407 seq_num 1209) 2024-09-13T14:35:03.706 INFO:tasks.rados.rados.0.smithi151.stdout:3377: read oid 49 snap 409 2024-09-13T14:35:03.706 INFO:tasks.rados.rados.0.smithi151.stdout:3377: expect deleted 2024-09-13T14:35:03.706 INFO:tasks.rados.rados.0.smithi151.stdout:3378: snap_remove snap 423 2024-09-13T14:35:04.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:03 smithi184 ceph-mon[92527]: osdmap e912: 8 total, 8 up, 8 in 2024-09-13T14:35:04.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:03 smithi151 ceph-mon[104450]: osdmap e912: 8 total, 8 up, 8 in 2024-09-13T14:35:04.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:03 smithi151 ceph-mon[108966]: osdmap e912: 8 total, 8 up, 8 in 2024-09-13T14:35:04.630 INFO:tasks.rados.rados.0.smithi151.stdout:3378: done (5 left) 2024-09-13T14:35:04.631 INFO:tasks.rados.rados.0.smithi151.stdout:3379: rollback oid 20 current snap is 428 2024-09-13T14:35:04.631 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 20 to 363 2024-09-13T14:35:04.631 INFO:tasks.rados.rados.0.smithi151.stdout:3373: expect (ObjNum 1203 snap 407 seq_num 1203) 2024-09-13T14:35:04.737 INFO:tasks.rados.rados.0.smithi151.stdout:3374: expect (ObjNum 1241 snap 416 seq_num 1241) 2024-09-13T14:35:04.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:04 smithi151 ceph-mon[104450]: pgmap v987: 89 pgs: 89 active+clean; 315 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.5 MiB/s rd, 1.8 MiB/s wr, 14 op/s 2024-09-13T14:35:04.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:04 smithi151 ceph-mon[104450]: osdmap e913: 8 total, 8 up, 8 in 2024-09-13T14:35:04.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:04 smithi151 ceph-mon[104450]: osdmap e914: 8 total, 8 up, 8 in 2024-09-13T14:35:04.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:04 smithi151 ceph-mon[108966]: pgmap v987: 89 pgs: 89 active+clean; 315 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.5 MiB/s rd, 1.8 MiB/s wr, 14 op/s 2024-09-13T14:35:04.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:04 smithi151 ceph-mon[108966]: osdmap e913: 8 total, 8 up, 8 in 2024-09-13T14:35:04.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:04 smithi151 ceph-mon[108966]: osdmap e914: 8 total, 8 up, 8 in 2024-09-13T14:35:04.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:35:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:35:04] "GET /metrics HTTP/1.1" 200 36509 "" "Prometheus/2.51.0" 2024-09-13T14:35:04.868 INFO:tasks.rados.rados.0.smithi151.stdout:3375: expect (ObjNum 1202 snap 407 seq_num 1202) 2024-09-13T14:35:04.939 INFO:tasks.rados.rados.0.smithi151.stdout:3376: expect (ObjNum 1209 snap 407 seq_num 1209) 2024-09-13T14:35:05.037 INFO:tasks.rados.rados.0.smithi151.stdout:3379: finishing rollback tid 1 to smithi15186934-20 2024-09-13T14:35:05.037 INFO:tasks.rados.rados.0.smithi151.stdout:3379: finishing rollback tid 2 to smithi15186934-20 2024-09-13T14:35:05.037 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 940 (ObjNum 997 snap 344 seq_num 997) dirty exists 2024-09-13T14:35:05.037 INFO:tasks.rados.rados.0.smithi151.stdout:3373: done (5 left) 2024-09-13T14:35:05.037 INFO:tasks.rados.rados.0.smithi151.stdout:3374: done (4 left) 2024-09-13T14:35:05.037 INFO:tasks.rados.rados.0.smithi151.stdout:3375: done (3 left) 2024-09-13T14:35:05.037 INFO:tasks.rados.rados.0.smithi151.stdout:3376: done (2 left) 2024-09-13T14:35:05.037 INFO:tasks.rados.rados.0.smithi151.stdout:3377: done (1 left) 2024-09-13T14:35:05.037 INFO:tasks.rados.rados.0.smithi151.stdout:3379: done (0 left) 2024-09-13T14:35:05.038 INFO:tasks.rados.rados.0.smithi151.stdout:3380: rollback oid 31 current snap is 428 2024-09-13T14:35:05.038 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 31 to 416 2024-09-13T14:35:05.038 INFO:tasks.rados.rados.0.smithi151.stdout:3381: snap_remove snap 422 2024-09-13T14:35:05.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:04 smithi184 ceph-mon[92527]: pgmap v987: 89 pgs: 89 active+clean; 315 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.5 MiB/s rd, 1.8 MiB/s wr, 14 op/s 2024-09-13T14:35:05.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:04 smithi184 ceph-mon[92527]: osdmap e913: 8 total, 8 up, 8 in 2024-09-13T14:35:05.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:04 smithi184 ceph-mon[92527]: osdmap e914: 8 total, 8 up, 8 in 2024-09-13T14:35:05.713 INFO:tasks.rados.rados.0.smithi151.stdout:3381: done (1 left) 2024-09-13T14:35:05.713 INFO:tasks.rados.rados.0.smithi151.stdout:3382: snap_remove snap 363 2024-09-13T14:35:06.724 INFO:tasks.rados.rados.0.smithi151.stdout:3382: done (1 left) 2024-09-13T14:35:06.725 INFO:tasks.rados.rados.0.smithi151.stdout:3383: read oid 28 snap -1 2024-09-13T14:35:06.725 INFO:tasks.rados.rados.0.smithi151.stdout:3383: expect (ObjNum 1163 snap 396 seq_num 1163) 2024-09-13T14:35:06.725 INFO:tasks.rados.rados.0.smithi151.stdout:3384: write oid 38 current snap is 428 2024-09-13T14:35:06.725 INFO:tasks.rados.rados.0.smithi151.stdout:3384: seq_num 1261 ranges {793627=487234,1790970=404772,2829335=364292} 2024-09-13T14:35:06.732 INFO:tasks.rados.rados.0.smithi151.stdout:3384: writing smithi15186934-38 from 793627 to 1280861 tid 1 2024-09-13T14:35:06.739 INFO:tasks.rados.rados.0.smithi151.stdout:3384: writing smithi15186934-38 from 1790970 to 2195742 tid 2 2024-09-13T14:35:06.745 INFO:tasks.rados.rados.0.smithi151.stdout:3384: writing smithi15186934-38 from 2829335 to 3193627 tid 3 2024-09-13T14:35:06.745 INFO:tasks.rados.rados.0.smithi151.stdout:3385: rollback oid 43 current snap is 428 2024-09-13T14:35:06.746 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 43 to 427 2024-09-13T14:35:06.746 INFO:tasks.rados.rados.0.smithi151.stdout:3380: finishing rollback tid 1 to smithi15186934-31 2024-09-13T14:35:06.746 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 31 v 1161 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-13T14:35:06.746 INFO:tasks.rados.rados.0.smithi151.stdout:3383: expect (ObjNum 1163 snap 396 seq_num 1163) 2024-09-13T14:35:06.939 INFO:tasks.rados.rados.0.smithi151.stdout:3384: finishing write tid 1 to smithi15186934-38 2024-09-13T14:35:06.939 INFO:tasks.rados.rados.0.smithi151.stdout:3380: done (3 left) 2024-09-13T14:35:06.939 INFO:tasks.rados.rados.0.smithi151.stdout:3383: done (2 left) 2024-09-13T14:35:06.939 INFO:tasks.rados.rados.0.smithi151.stdout:3386: rollback oid 30 current snap is 428 2024-09-13T14:35:06.939 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 30 to 413 2024-09-13T14:35:06.939 INFO:tasks.rados.rados.0.smithi151.stdout:3385: finishing rollback tid 1 to smithi15186934-43 2024-09-13T14:35:06.939 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 43 v 0 (ObjNum 129 snap 0 seq_num 3092666000) dirty dne 2024-09-13T14:35:06.940 INFO:tasks.rados.rados.0.smithi151.stdout:3384: finishing write tid 2 to smithi15186934-38 2024-09-13T14:35:06.940 INFO:tasks.rados.rados.0.smithi151.stdout:3384: finishing write tid 3 to smithi15186934-38 2024-09-13T14:35:06.940 INFO:tasks.rados.rados.0.smithi151.stdout:3384: finishing write tid 4 to smithi15186934-38 2024-09-13T14:35:06.940 INFO:tasks.rados.rados.0.smithi151.stdout:3384: finishing write tid 5 to smithi15186934-38 2024-09-13T14:35:06.940 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 1429 (ObjNum 1261 snap 428 seq_num 1261) dirty exists 2024-09-13T14:35:06.940 INFO:tasks.rados.rados.0.smithi151.stdout:3384: left oid 38 (ObjNum 1261 snap 428 seq_num 1261) 2024-09-13T14:35:06.940 INFO:tasks.rados.rados.0.smithi151.stdout:3384: done (2 left) 2024-09-13T14:35:06.940 INFO:tasks.rados.rados.0.smithi151.stdout:3385: done (1 left) 2024-09-13T14:35:06.940 INFO:tasks.rados.rados.0.smithi151.stdout:3387: snap_remove snap 427 2024-09-13T14:35:07.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:06 smithi184 ceph-mon[92527]: pgmap v990: 89 pgs: 89 active+clean; 306 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.5 MiB/s rd, 1.5 MiB/s wr, 13 op/s 2024-09-13T14:35:07.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:06 smithi184 ceph-mon[92527]: osdmap e915: 8 total, 8 up, 8 in 2024-09-13T14:35:07.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:06 smithi151 ceph-mon[108966]: pgmap v990: 89 pgs: 89 active+clean; 306 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.5 MiB/s rd, 1.5 MiB/s wr, 13 op/s 2024-09-13T14:35:07.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:06 smithi151 ceph-mon[108966]: osdmap e915: 8 total, 8 up, 8 in 2024-09-13T14:35:07.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:06 smithi151 ceph-mon[104450]: pgmap v990: 89 pgs: 89 active+clean; 306 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.5 MiB/s rd, 1.5 MiB/s wr, 13 op/s 2024-09-13T14:35:07.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:06 smithi151 ceph-mon[104450]: osdmap e915: 8 total, 8 up, 8 in 2024-09-13T14:35:07.727 INFO:tasks.rados.rados.0.smithi151.stdout:3387: done (1 left) 2024-09-13T14:35:07.727 INFO:tasks.rados.rados.0.smithi151.stdout:3388: snap_remove snap 416 2024-09-13T14:35:08.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:07 smithi184 ceph-mon[92527]: osdmap e916: 8 total, 8 up, 8 in 2024-09-13T14:35:08.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:07 smithi151 ceph-mon[104450]: osdmap e916: 8 total, 8 up, 8 in 2024-09-13T14:35:08.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:07 smithi151 ceph-mon[108966]: osdmap e916: 8 total, 8 up, 8 in 2024-09-13T14:35:08.729 INFO:tasks.rados.rados.0.smithi151.stdout:3388: done (1 left) 2024-09-13T14:35:08.729 INFO:tasks.rados.rados.0.smithi151.stdout:3389: delete oid 4 current snap is 428 2024-09-13T14:35:08.729 INFO:tasks.rados.rados.0.smithi151.stdout:3386: finishing rollback tid 1 to smithi15186934-30 2024-09-13T14:35:08.729 INFO:tasks.rados.rados.0.smithi151.stdout:3386: finishing rollback tid 2 to smithi15186934-30 2024-09-13T14:35:08.729 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 30 v 1314 (ObjNum 1218 snap 410 seq_num 1218) dirty exists 2024-09-13T14:35:08.739 INFO:tasks.rados.rados.0.smithi151.stdout:3386: done (1 left) 2024-09-13T14:35:08.739 INFO:tasks.rados.rados.0.smithi151.stdout:3389: done (0 left) 2024-09-13T14:35:08.739 INFO:tasks.rados.rados.0.smithi151.stdout:3390: delete oid 27 current snap is 428 2024-09-13T14:35:08.742 INFO:tasks.rados.rados.0.smithi151.stdout:3390: done (0 left) 2024-09-13T14:35:08.742 INFO:tasks.rados.rados.0.smithi151.stdout:3391: snap_create 2024-09-13T14:35:09.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:08 smithi184 ceph-mon[92527]: pgmap v993: 89 pgs: 89 active+clean; 269 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 645 KiB/s wr, 14 op/s 2024-09-13T14:35:09.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:08 smithi184 ceph-mon[92527]: osdmap e917: 8 total, 8 up, 8 in 2024-09-13T14:35:09.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:08 smithi151 ceph-mon[104450]: pgmap v993: 89 pgs: 89 active+clean; 269 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 645 KiB/s wr, 14 op/s 2024-09-13T14:35:09.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:08 smithi151 ceph-mon[104450]: osdmap e917: 8 total, 8 up, 8 in 2024-09-13T14:35:09.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:08 smithi151 ceph-mon[108966]: pgmap v993: 89 pgs: 89 active+clean; 269 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 645 KiB/s wr, 14 op/s 2024-09-13T14:35:09.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:08 smithi151 ceph-mon[108966]: osdmap e917: 8 total, 8 up, 8 in 2024-09-13T14:35:09.634 INFO:tasks.rados.rados.0.smithi151.stdout:3391: done (0 left) 2024-09-13T14:35:09.634 INFO:tasks.rados.rados.0.smithi151.stdout:3392: write oid 4 current snap is 429 2024-09-13T14:35:09.634 INFO:tasks.rados.rados.0.smithi151.stdout:3392: seq_num 1264 ranges {444048=429334,1282769=552099,2627957=616091} 2024-09-13T14:35:09.638 INFO:tasks.rados.rados.0.smithi151.stdout:3392: writing smithi15186934-4 from 444048 to 873382 tid 1 2024-09-13T14:35:09.644 INFO:tasks.rados.rados.0.smithi151.stdout:3392: writing smithi15186934-4 from 1282769 to 1834868 tid 2 2024-09-13T14:35:09.653 INFO:tasks.rados.rados.0.smithi151.stdout:3392: writing smithi15186934-4 from 2627957 to 3244048 tid 3 2024-09-13T14:35:09.653 INFO:tasks.rados.rados.0.smithi151.stdout:3392: finishing write tid 1 to smithi15186934-4 2024-09-13T14:35:09.653 INFO:tasks.rados.rados.0.smithi151.stdout:3392: finishing write tid 2 to smithi15186934-4 2024-09-13T14:35:09.653 INFO:tasks.rados.rados.0.smithi151.stdout:3393: read oid 29 snap -1 2024-09-13T14:35:09.653 INFO:tasks.rados.rados.0.smithi151.stdout:3393: expect deleted 2024-09-13T14:35:09.653 INFO:tasks.rados.rados.0.smithi151.stdout:3394: write oid 19 current snap is 429 2024-09-13T14:35:09.653 INFO:tasks.rados.rados.0.smithi151.stdout:3394: seq_num 1265 ranges {460855=504583,1624898=412110,2539027=572508} 2024-09-13T14:35:09.658 INFO:tasks.rados.rados.0.smithi151.stdout:3394: writing smithi15186934-19 from 460855 to 965438 tid 1 2024-09-13T14:35:09.664 INFO:tasks.rados.rados.0.smithi151.stdout:3394: writing smithi15186934-19 from 1624898 to 2037008 tid 2 2024-09-13T14:35:09.671 INFO:tasks.rados.rados.0.smithi151.stdout:3394: writing smithi15186934-19 from 2539027 to 3111535 tid 3 2024-09-13T14:35:09.671 INFO:tasks.rados.rados.0.smithi151.stdout:3395: read oid 49 snap -1 2024-09-13T14:35:09.671 INFO:tasks.rados.rados.0.smithi151.stdout:3395: expect deleted 2024-09-13T14:35:09.671 INFO:tasks.rados.rados.0.smithi151.stdout:3396: rollback oid 30 current snap is 429 2024-09-13T14:35:09.671 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 30 to 424 2024-09-13T14:35:09.671 INFO:tasks.rados.rados.0.smithi151.stdout:3392: finishing write tid 3 to smithi15186934-4 2024-09-13T14:35:09.671 INFO:tasks.rados.rados.0.smithi151.stdout:3392: finishing write tid 4 to smithi15186934-4 2024-09-13T14:35:09.671 INFO:tasks.rados.rados.0.smithi151.stdout:3393: done (4 left) 2024-09-13T14:35:09.671 INFO:tasks.rados.rados.0.smithi151.stdout:3397: snap_create 2024-09-13T14:35:09.672 INFO:tasks.rados.rados.0.smithi151.stdout:3392: finishing write tid 5 to smithi15186934-4 2024-09-13T14:35:09.672 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 1623 (ObjNum 1264 snap 429 seq_num 1264) dirty exists 2024-09-13T14:35:09.672 INFO:tasks.rados.rados.0.smithi151.stdout:3392: left oid 4 (ObjNum 1264 snap 429 seq_num 1264) 2024-09-13T14:35:09.672 INFO:tasks.rados.rados.0.smithi151.stdout:3394: finishing write tid 1 to smithi15186934-19 2024-09-13T14:35:09.672 INFO:tasks.rados.rados.0.smithi151.stdout:3394: finishing write tid 2 to smithi15186934-19 2024-09-13T14:35:09.674 INFO:tasks.rados.rados.0.smithi151.stdout:3394: finishing write tid 3 to smithi15186934-19 2024-09-13T14:35:09.674 INFO:tasks.rados.rados.0.smithi151.stdout:3394: finishing write tid 4 to smithi15186934-19 2024-09-13T14:35:09.674 INFO:tasks.rados.rados.0.smithi151.stdout:3396: finishing rollback tid 0 to smithi15186934-30 2024-09-13T14:35:09.674 INFO:tasks.rados.rados.0.smithi151.stdout:3396: finishing rollback tid 1 to smithi15186934-30 2024-09-13T14:35:09.674 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 30 v 1319 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-13T14:35:09.674 INFO:tasks.rados.rados.0.smithi151.stdout:3394: finishing write tid 5 to smithi15186934-19 2024-09-13T14:35:09.674 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 19 v 768 (ObjNum 1265 snap 429 seq_num 1265) dirty exists 2024-09-13T14:35:09.674 INFO:tasks.rados.rados.0.smithi151.stdout:3394: left oid 19 (ObjNum 1265 snap 429 seq_num 1265) 2024-09-13T14:35:10.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:09 smithi184 ceph-mon[92527]: osdmap e918: 8 total, 8 up, 8 in 2024-09-13T14:35:10.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:09 smithi184 ceph-mon[92527]: osdmap e919: 8 total, 8 up, 8 in 2024-09-13T14:35:10.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:09 smithi151 ceph-mon[104450]: osdmap e918: 8 total, 8 up, 8 in 2024-09-13T14:35:10.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:09 smithi151 ceph-mon[104450]: osdmap e919: 8 total, 8 up, 8 in 2024-09-13T14:35:10.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:09 smithi151 ceph-mon[108966]: osdmap e918: 8 total, 8 up, 8 in 2024-09-13T14:35:10.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:09 smithi151 ceph-mon[108966]: osdmap e919: 8 total, 8 up, 8 in 2024-09-13T14:35:10.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:35:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:35:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:35:10.639 INFO:tasks.rados.rados.0.smithi151.stdout:3392: done (4 left) 2024-09-13T14:35:10.639 INFO:tasks.rados.rados.0.smithi151.stdout:3394: done (3 left) 2024-09-13T14:35:10.640 INFO:tasks.rados.rados.0.smithi151.stdout:3395: done (2 left) 2024-09-13T14:35:10.640 INFO:tasks.rados.rados.0.smithi151.stdout:3396: done (1 left) 2024-09-13T14:35:10.640 INFO:tasks.rados.rados.0.smithi151.stdout:3397: done (0 left) 2024-09-13T14:35:10.640 INFO:tasks.rados.rados.0.smithi151.stdout:3398: read oid 3 snap 426 2024-09-13T14:35:10.640 INFO:tasks.rados.rados.0.smithi151.stdout:3398: expect deleted 2024-09-13T14:35:10.640 INFO:tasks.rados.rados.0.smithi151.stdout:3399: snap_create 2024-09-13T14:35:11.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:10 smithi184 ceph-mon[92527]: pgmap v996: 89 pgs: 89 active+clean; 253 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 307 KiB/s wr, 15 op/s 2024-09-13T14:35:11.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:10 smithi184 ceph-mon[92527]: osdmap e920: 8 total, 8 up, 8 in 2024-09-13T14:35:11.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:10 smithi151 ceph-mon[104450]: pgmap v996: 89 pgs: 89 active+clean; 253 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 307 KiB/s wr, 15 op/s 2024-09-13T14:35:11.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:10 smithi151 ceph-mon[104450]: osdmap e920: 8 total, 8 up, 8 in 2024-09-13T14:35:11.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:10 smithi151 ceph-mon[108966]: pgmap v996: 89 pgs: 89 active+clean; 253 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 307 KiB/s wr, 15 op/s 2024-09-13T14:35:11.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:10 smithi151 ceph-mon[108966]: osdmap e920: 8 total, 8 up, 8 in 2024-09-13T14:35:11.641 INFO:tasks.rados.rados.0.smithi151.stdout:3398: done (1 left) 2024-09-13T14:35:11.641 INFO:tasks.rados.rados.0.smithi151.stdout:3399: done (0 left) 2024-09-13T14:35:11.641 INFO:tasks.rados.rados.0.smithi151.stdout:3400: read oid 2 snap -1 2024-09-13T14:35:11.641 INFO:tasks.rados.rados.0.smithi151.stdout:3400: expect (ObjNum 1215 snap 409 seq_num 1215) 2024-09-13T14:35:11.641 INFO:tasks.rados.rados.0.smithi151.stdout:3401: delete oid 48 current snap is 431 2024-09-13T14:35:11.645 INFO:tasks.rados.rados.0.smithi151.stdout:3400: expect (ObjNum 1215 snap 409 seq_num 1215) 2024-09-13T14:35:11.721 INFO:tasks.rados.rados.0.smithi151.stdout:3400: done (1 left) 2024-09-13T14:35:11.722 INFO:tasks.rados.rados.0.smithi151.stdout:3401: done (0 left) 2024-09-13T14:35:11.722 INFO:tasks.rados.rados.0.smithi151.stdout:3402: delete oid 45 current snap is 431 2024-09-13T14:35:11.724 INFO:tasks.rados.rados.0.smithi151.stdout:3402: done (0 left) 2024-09-13T14:35:11.724 INFO:tasks.rados.rados.0.smithi151.stdout:3403: rollback oid 39 current snap is 431 2024-09-13T14:35:11.724 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 39 to 425 2024-09-13T14:35:11.724 INFO:tasks.rados.rados.0.smithi151.stdout:3404: write oid 5 current snap is 431 2024-09-13T14:35:11.724 INFO:tasks.rados.rados.0.smithi151.stdout:3404: seq_num 1268 ranges {511276=730330,1800344=559437,3021296=289980} 2024-09-13T14:35:11.730 INFO:tasks.rados.rados.0.smithi151.stdout:3404: writing smithi15186934-5 from 511276 to 1241606 tid 1 2024-09-13T14:35:11.737 INFO:tasks.rados.rados.0.smithi151.stdout:3404: writing smithi15186934-5 from 1800344 to 2359781 tid 2 2024-09-13T14:35:11.742 INFO:tasks.rados.rados.0.smithi151.stdout:3404: writing smithi15186934-5 from 3021296 to 3311276 tid 3 2024-09-13T14:35:11.742 INFO:tasks.rados.rados.0.smithi151.stdout:3405: snap_create 2024-09-13T14:35:11.743 INFO:tasks.rados.rados.0.smithi151.stdout:3403: finishing rollback tid 0 to smithi15186934-39 2024-09-13T14:35:11.743 INFO:tasks.rados.rados.0.smithi151.stdout:3403: finishing rollback tid 1 to smithi15186934-39 2024-09-13T14:35:11.743 INFO:tasks.rados.rados.0.smithi151.stdout:3403: finishing rollback tid 2 to smithi15186934-39 2024-09-13T14:35:11.743 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 1439 (ObjNum 1189 snap 402 seq_num 1189) dirty exists 2024-09-13T14:35:11.743 INFO:tasks.rados.rados.0.smithi151.stdout:3404: finishing write tid 1 to smithi15186934-5 2024-09-13T14:35:11.743 INFO:tasks.rados.rados.0.smithi151.stdout:3404: finishing write tid 2 to smithi15186934-5 2024-09-13T14:35:11.745 INFO:tasks.rados.rados.0.smithi151.stdout:3404: finishing write tid 3 to smithi15186934-5 2024-09-13T14:35:11.745 INFO:tasks.rados.rados.0.smithi151.stdout:3404: finishing write tid 4 to smithi15186934-5 2024-09-13T14:35:11.745 INFO:tasks.rados.rados.0.smithi151.stdout:3404: finishing write tid 5 to smithi15186934-5 2024-09-13T14:35:11.746 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 1294 (ObjNum 1268 snap 431 seq_num 1268) dirty exists 2024-09-13T14:35:11.746 INFO:tasks.rados.rados.0.smithi151.stdout:3404: left oid 5 (ObjNum 1268 snap 431 seq_num 1268) 2024-09-13T14:35:12.746 INFO:tasks.rados.rados.0.smithi151.stdout:3403: done (2 left) 2024-09-13T14:35:12.746 INFO:tasks.rados.rados.0.smithi151.stdout:3404: done (1 left) 2024-09-13T14:35:12.746 INFO:tasks.rados.rados.0.smithi151.stdout:3405: done (0 left) 2024-09-13T14:35:12.746 INFO:tasks.rados.rados.0.smithi151.stdout:3406: read oid 46 snap -1 2024-09-13T14:35:12.746 INFO:tasks.rados.rados.0.smithi151.stdout:3406: expect (ObjNum 1135 snap 381 seq_num 1135) 2024-09-13T14:35:12.747 INFO:tasks.rados.rados.0.smithi151.stdout:3407: snap_create 2024-09-13T14:35:12.753 INFO:tasks.rados.rados.0.smithi151.stdout:3406: expect (ObjNum 1135 snap 381 seq_num 1135) 2024-09-13T14:35:12.905 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:12 smithi184 ceph-mon[92527]: pgmap v999: 89 pgs: 89 active+clean; 232 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 0 B/s wr, 3 op/s 2024-09-13T14:35:12.906 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:12 smithi184 ceph-mon[92527]: osdmap e921: 8 total, 8 up, 8 in 2024-09-13T14:35:13.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:12 smithi151 ceph-mon[104450]: pgmap v999: 89 pgs: 89 active+clean; 232 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 0 B/s wr, 3 op/s 2024-09-13T14:35:13.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:12 smithi151 ceph-mon[104450]: osdmap e921: 8 total, 8 up, 8 in 2024-09-13T14:35:13.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:12 smithi151 ceph-mon[108966]: pgmap v999: 89 pgs: 89 active+clean; 232 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 0 B/s wr, 3 op/s 2024-09-13T14:35:13.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:12 smithi151 ceph-mon[108966]: osdmap e921: 8 total, 8 up, 8 in 2024-09-13T14:35:13.750 INFO:tasks.rados.rados.0.smithi151.stdout:3406: done (1 left) 2024-09-13T14:35:13.750 INFO:tasks.rados.rados.0.smithi151.stdout:3407: done (0 left) 2024-09-13T14:35:13.751 INFO:tasks.rados.rados.0.smithi151.stdout:3408: rollback oid 32 current snap is 433 2024-09-13T14:35:13.751 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 32 to 421 2024-09-13T14:35:13.751 INFO:tasks.rados.rados.0.smithi151.stdout:3409: delete oid 6 current snap is 433 2024-09-13T14:35:13.755 INFO:tasks.rados.rados.0.smithi151.stdout:3409: done (1 left) 2024-09-13T14:35:13.755 INFO:tasks.rados.rados.0.smithi151.stdout:3410: snap_create 2024-09-13T14:35:13.756 INFO:tasks.rados.rados.0.smithi151.stdout:3408: finishing rollback tid 0 to smithi15186934-32 2024-09-13T14:35:13.756 INFO:tasks.rados.rados.0.smithi151.stdout:3408: finishing rollback tid 1 to smithi15186934-32 2024-09-13T14:35:13.756 INFO:tasks.rados.rados.0.smithi151.stdout:3408: finishing rollback tid 2 to smithi15186934-32 2024-09-13T14:35:13.756 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 32 v 1392 (ObjNum 1197 snap 404 seq_num 1197) dirty exists 2024-09-13T14:35:14.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:13 smithi184 ceph-mon[92527]: osdmap e922: 8 total, 8 up, 8 in 2024-09-13T14:35:14.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:13 smithi151 ceph-mon[104450]: osdmap e922: 8 total, 8 up, 8 in 2024-09-13T14:35:14.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:13 smithi151 ceph-mon[108966]: osdmap e922: 8 total, 8 up, 8 in 2024-09-13T14:35:14.634 INFO:tasks.rados.rados.0.smithi151.stdout:3408: done (1 left) 2024-09-13T14:35:14.634 INFO:tasks.rados.rados.0.smithi151.stdout:3410: done (0 left) 2024-09-13T14:35:14.635 INFO:tasks.rados.rados.0.smithi151.stdout:3411: read oid 2 snap -1 2024-09-13T14:35:14.635 INFO:tasks.rados.rados.0.smithi151.stdout:3411: expect (ObjNum 1215 snap 409 seq_num 1215) 2024-09-13T14:35:14.635 INFO:tasks.rados.rados.0.smithi151.stdout:3412: read oid 23 snap 424 2024-09-13T14:35:14.635 INFO:tasks.rados.rados.0.smithi151.stdout:3412: expect (ObjNum 1216 snap 409 seq_num 1216) 2024-09-13T14:35:14.635 INFO:tasks.rados.rados.0.smithi151.stdout:3413: delete oid 47 current snap is 434 2024-09-13T14:35:14.642 INFO:tasks.rados.rados.0.smithi151.stdout:3412: expect (ObjNum 1216 snap 409 seq_num 1216) 2024-09-13T14:35:14.771 INFO:tasks.rados.rados.0.smithi151.stdout:3411: expect (ObjNum 1215 snap 409 seq_num 1215) 2024-09-13T14:35:14.847 INFO:tasks.rados.rados.0.smithi151.stdout:3411: done (2 left) 2024-09-13T14:35:14.847 INFO:tasks.rados.rados.0.smithi151.stdout:3412: done (1 left) 2024-09-13T14:35:14.847 INFO:tasks.rados.rados.0.smithi151.stdout:3413: done (0 left) 2024-09-13T14:35:14.847 INFO:tasks.rados.rados.0.smithi151.stdout:3414: read oid 32 snap 428 2024-09-13T14:35:14.847 INFO:tasks.rados.rados.0.smithi151.stdout:3414: expect (ObjNum 1197 snap 404 seq_num 1197) 2024-09-13T14:35:14.847 INFO:tasks.rados.rados.0.smithi151.stdout:3415: snap_remove snap 409 2024-09-13T14:35:14.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:35:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:35:14] "GET /metrics HTTP/1.1" 200 36509 "" "Prometheus/2.51.0" 2024-09-13T14:35:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:14 smithi151 ceph-mon[104450]: pgmap v1002: 89 pgs: 89 active+clean; 236 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 754 KiB/s wr, 5 op/s 2024-09-13T14:35:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:14 smithi151 ceph-mon[104450]: osdmap e923: 8 total, 8 up, 8 in 2024-09-13T14:35:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:14 smithi151 ceph-mon[104450]: osdmap e924: 8 total, 8 up, 8 in 2024-09-13T14:35:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:14 smithi151 ceph-mon[108966]: pgmap v1002: 89 pgs: 89 active+clean; 236 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 754 KiB/s wr, 5 op/s 2024-09-13T14:35:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:14 smithi151 ceph-mon[108966]: osdmap e923: 8 total, 8 up, 8 in 2024-09-13T14:35:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:14 smithi151 ceph-mon[108966]: osdmap e924: 8 total, 8 up, 8 in 2024-09-13T14:35:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:14 smithi184 ceph-mon[92527]: pgmap v1002: 89 pgs: 89 active+clean; 236 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 754 KiB/s wr, 5 op/s 2024-09-13T14:35:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:14 smithi184 ceph-mon[92527]: osdmap e923: 8 total, 8 up, 8 in 2024-09-13T14:35:15.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:14 smithi184 ceph-mon[92527]: osdmap e924: 8 total, 8 up, 8 in 2024-09-13T14:35:15.759 INFO:tasks.rados.rados.0.smithi151.stdout:3414: expect (ObjNum 1197 snap 404 seq_num 1197) 2024-09-13T14:35:15.829 INFO:tasks.rados.rados.0.smithi151.stdout:3414: done (1 left) 2024-09-13T14:35:15.829 INFO:tasks.rados.rados.0.smithi151.stdout:3415: done (0 left) 2024-09-13T14:35:15.829 INFO:tasks.rados.rados.0.smithi151.stdout:3416: snap_remove snap 429 2024-09-13T14:35:16.769 INFO:tasks.rados.rados.0.smithi151.stdout:3416: done (0 left) 2024-09-13T14:35:16.769 INFO:tasks.rados.rados.0.smithi151.stdout:3417: read oid 32 snap -1 2024-09-13T14:35:16.769 INFO:tasks.rados.rados.0.smithi151.stdout:3417: expect (ObjNum 1197 snap 404 seq_num 1197) 2024-09-13T14:35:16.769 INFO:tasks.rados.rados.0.smithi151.stdout:3418: snap_create 2024-09-13T14:35:16.781 INFO:tasks.rados.rados.0.smithi151.stdout:3417: expect (ObjNum 1197 snap 404 seq_num 1197) 2024-09-13T14:35:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:16 smithi184 ceph-mon[92527]: pgmap v1005: 89 pgs: 89 active+clean; 239 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 4.4 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2024-09-13T14:35:17.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:16 smithi184 ceph-mon[92527]: osdmap e925: 8 total, 8 up, 8 in 2024-09-13T14:35:17.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:17.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:16 smithi151 ceph-mon[104450]: pgmap v1005: 89 pgs: 89 active+clean; 239 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 4.4 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2024-09-13T14:35:17.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:16 smithi151 ceph-mon[104450]: osdmap e925: 8 total, 8 up, 8 in 2024-09-13T14:35:17.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:17.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:16 smithi151 ceph-mon[108966]: pgmap v1005: 89 pgs: 89 active+clean; 239 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 4.4 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2024-09-13T14:35:17.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:16 smithi151 ceph-mon[108966]: osdmap e925: 8 total, 8 up, 8 in 2024-09-13T14:35:17.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:17.777 INFO:tasks.rados.rados.0.smithi151.stdout:3417: done (1 left) 2024-09-13T14:35:17.777 INFO:tasks.rados.rados.0.smithi151.stdout:3418: done (0 left) 2024-09-13T14:35:17.777 INFO:tasks.rados.rados.0.smithi151.stdout:3419: read oid 6 snap -1 2024-09-13T14:35:17.778 INFO:tasks.rados.rados.0.smithi151.stdout:3419: expect deleted 2024-09-13T14:35:17.778 INFO:tasks.rados.rados.0.smithi151.stdout:3420: delete oid 45 current snap is 435 2024-09-13T14:35:17.784 INFO:tasks.rados.rados.0.smithi151.stdout:3419: done (1 left) 2024-09-13T14:35:17.785 INFO:tasks.rados.rados.0.smithi151.stdout:3420: done (0 left) 2024-09-13T14:35:17.785 INFO:tasks.rados.rados.0.smithi151.stdout:3421: read oid 44 snap -1 2024-09-13T14:35:17.785 INFO:tasks.rados.rados.0.smithi151.stdout:3421: expect (ObjNum 1226 snap 410 seq_num 1226) 2024-09-13T14:35:17.785 INFO:tasks.rados.rados.0.smithi151.stdout:3422: snap_create 2024-09-13T14:35:17.790 INFO:tasks.rados.rados.0.smithi151.stdout:3421: expect (ObjNum 1226 snap 410 seq_num 1226) 2024-09-13T14:35:18.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:17 smithi184 ceph-mon[92527]: osdmap e926: 8 total, 8 up, 8 in 2024-09-13T14:35:18.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:17 smithi151 ceph-mon[104450]: osdmap e926: 8 total, 8 up, 8 in 2024-09-13T14:35:18.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:17 smithi151 ceph-mon[108966]: osdmap e926: 8 total, 8 up, 8 in 2024-09-13T14:35:18.777 INFO:tasks.rados.rados.0.smithi151.stdout:3421: done (1 left) 2024-09-13T14:35:18.777 INFO:tasks.rados.rados.0.smithi151.stdout:3422: done (0 left) 2024-09-13T14:35:18.777 INFO:tasks.rados.rados.0.smithi151.stdout:3423: delete oid 8 current snap is 436 2024-09-13T14:35:18.783 INFO:tasks.rados.rados.0.smithi151.stdout:3423: done (0 left) 2024-09-13T14:35:18.783 INFO:tasks.rados.rados.0.smithi151.stdout:3424: delete oid 35 current snap is 436 2024-09-13T14:35:18.787 INFO:tasks.rados.rados.0.smithi151.stdout:3424: done (0 left) 2024-09-13T14:35:18.787 INFO:tasks.rados.rados.0.smithi151.stdout:3425: write oid 45 current snap is 436 2024-09-13T14:35:18.787 INFO:tasks.rados.rados.0.smithi151.stdout:3425: seq_num 1274 ranges {612118=498177,1634883=570444,2669481=706662} 2024-09-13T14:35:18.793 INFO:tasks.rados.rados.0.smithi151.stdout:3425: writing smithi15186934-45 from 612118 to 1110295 tid 1 2024-09-13T14:35:18.799 INFO:tasks.rados.rados.0.smithi151.stdout:3425: writing smithi15186934-45 from 1634883 to 2205327 tid 2 2024-09-13T14:35:18.806 INFO:tasks.rados.rados.0.smithi151.stdout:3425: writing smithi15186934-45 from 2669481 to 3376143 tid 3 2024-09-13T14:35:18.806 INFO:tasks.rados.rados.0.smithi151.stdout:3426: snap_create 2024-09-13T14:35:18.806 INFO:tasks.rados.rados.0.smithi151.stdout:3425: finishing write tid 1 to smithi15186934-45 2024-09-13T14:35:18.806 INFO:tasks.rados.rados.0.smithi151.stdout:3425: finishing write tid 2 to smithi15186934-45 2024-09-13T14:35:18.810 INFO:tasks.rados.rados.0.smithi151.stdout:3425: finishing write tid 3 to smithi15186934-45 2024-09-13T14:35:18.810 INFO:tasks.rados.rados.0.smithi151.stdout:3425: finishing write tid 4 to smithi15186934-45 2024-09-13T14:35:18.810 INFO:tasks.rados.rados.0.smithi151.stdout:3425: finishing write tid 5 to smithi15186934-45 2024-09-13T14:35:18.810 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 1642 (ObjNum 1274 snap 436 seq_num 1274) dirty exists 2024-09-13T14:35:18.810 INFO:tasks.rados.rados.0.smithi151.stdout:3425: left oid 45 (ObjNum 1274 snap 436 seq_num 1274) 2024-09-13T14:35:19.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:18 smithi184 ceph-mon[92527]: pgmap v1008: 89 pgs: 89 active+clean; 238 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 9.4 MiB/s rd, 386 KiB/s wr, 10 op/s 2024-09-13T14:35:19.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:18 smithi184 ceph-mon[92527]: osdmap e927: 8 total, 8 up, 8 in 2024-09-13T14:35:19.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:18 smithi151 ceph-mon[104450]: pgmap v1008: 89 pgs: 89 active+clean; 238 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 9.4 MiB/s rd, 386 KiB/s wr, 10 op/s 2024-09-13T14:35:19.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:18 smithi151 ceph-mon[104450]: osdmap e927: 8 total, 8 up, 8 in 2024-09-13T14:35:19.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:18 smithi151 ceph-mon[108966]: pgmap v1008: 89 pgs: 89 active+clean; 238 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 9.4 MiB/s rd, 386 KiB/s wr, 10 op/s 2024-09-13T14:35:19.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:18 smithi151 ceph-mon[108966]: osdmap e927: 8 total, 8 up, 8 in 2024-09-13T14:35:19.635 INFO:tasks.rados.rados.0.smithi151.stdout:3425: done (1 left) 2024-09-13T14:35:19.635 INFO:tasks.rados.rados.0.smithi151.stdout:3426: done (0 left) 2024-09-13T14:35:19.635 INFO:tasks.rados.rados.0.smithi151.stdout:3427: write oid 37 current snap is 437 2024-09-13T14:35:19.635 INFO:tasks.rados.rados.0.smithi151.stdout:3427: seq_num 1275 ranges {628925=573426,1693365=714102,3096904=332021} 2024-09-13T14:35:19.641 INFO:tasks.rados.rados.0.smithi151.stdout:3427: writing smithi15186934-37 from 628925 to 1202351 tid 1 2024-09-13T14:35:19.649 INFO:tasks.rados.rados.0.smithi151.stdout:3427: writing smithi15186934-37 from 1693365 to 2407467 tid 2 2024-09-13T14:35:19.655 INFO:tasks.rados.rados.0.smithi151.stdout:3427: writing smithi15186934-37 from 3096904 to 3428925 tid 3 2024-09-13T14:35:19.655 INFO:tasks.rados.rados.0.smithi151.stdout:3428: write oid 25 current snap is 437 2024-09-13T14:35:19.655 INFO:tasks.rados.rados.0.smithi151.stdout:3428: seq_num 1276 ranges {645732=648675,1751847=574113,3124327=321405} 2024-09-13T14:35:19.663 INFO:tasks.rados.rados.0.smithi151.stdout:3428: writing smithi15186934-25 from 645732 to 1294407 tid 1 2024-09-13T14:35:19.669 INFO:tasks.rados.rados.0.smithi151.stdout:3428: writing smithi15186934-25 from 1751847 to 2325960 tid 2 2024-09-13T14:35:19.676 INFO:tasks.rados.rados.0.smithi151.stdout:3428: writing smithi15186934-25 from 3124327 to 3445732 tid 3 2024-09-13T14:35:19.677 INFO:tasks.rados.rados.0.smithi151.stdout:3429: rollback oid 18 current snap is 437 2024-09-13T14:35:19.677 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 18 to 426 2024-09-13T14:35:19.677 INFO:tasks.rados.rados.0.smithi151.stdout:3427: finishing write tid 1 to smithi15186934-37 2024-09-13T14:35:19.677 INFO:tasks.rados.rados.0.smithi151.stdout:3427: finishing write tid 2 to smithi15186934-37 2024-09-13T14:35:19.677 INFO:tasks.rados.rados.0.smithi151.stdout:3427: finishing write tid 3 to smithi15186934-37 2024-09-13T14:35:19.677 INFO:tasks.rados.rados.0.smithi151.stdout:3427: finishing write tid 4 to smithi15186934-37 2024-09-13T14:35:19.677 INFO:tasks.rados.rados.0.smithi151.stdout:3427: finishing write tid 5 to smithi15186934-37 2024-09-13T14:35:19.677 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 37 v 1590 (ObjNum 1275 snap 437 seq_num 1275) dirty exists 2024-09-13T14:35:19.677 INFO:tasks.rados.rados.0.smithi151.stdout:3427: left oid 37 (ObjNum 1275 snap 437 seq_num 1275) 2024-09-13T14:35:19.677 INFO:tasks.rados.rados.0.smithi151.stdout:3428: finishing write tid 1 to smithi15186934-25 2024-09-13T14:35:19.677 INFO:tasks.rados.rados.0.smithi151.stdout:3428: finishing write tid 2 to smithi15186934-25 2024-09-13T14:35:19.677 INFO:tasks.rados.rados.0.smithi151.stdout:3427: done (2 left) 2024-09-13T14:35:19.678 INFO:tasks.rados.rados.0.smithi151.stdout:3430: read oid 13 snap -1 2024-09-13T14:35:19.678 INFO:tasks.rados.rados.0.smithi151.stdout:3430: expect (ObjNum 1201 snap 406 seq_num 1201) 2024-09-13T14:35:19.678 INFO:tasks.rados.rados.0.smithi151.stdout:3431: write oid 3 current snap is 437 2024-09-13T14:35:19.678 INFO:tasks.rados.rados.0.smithi151.stdout:3431: seq_num 1277 ranges {662539=723924,2093976=717771,3435397=27142} 2024-09-13T14:35:19.684 INFO:tasks.rados.rados.0.smithi151.stdout:3431: writing smithi15186934-3 from 662539 to 1386463 tid 1 2024-09-13T14:35:19.692 INFO:tasks.rados.rados.0.smithi151.stdout:3431: writing smithi15186934-3 from 2093976 to 2811747 tid 2 2024-09-13T14:35:19.696 INFO:tasks.rados.rados.0.smithi151.stdout:3431: writing smithi15186934-3 from 3435397 to 3462539 tid 3 2024-09-13T14:35:19.696 INFO:tasks.rados.rados.0.smithi151.stdout:3432: write oid 5 current snap is 437 2024-09-13T14:35:19.696 INFO:tasks.rados.rados.0.smithi151.stdout:3432: seq_num 1278 ranges {679346=515526,1868811=577782,3179173=300173} 2024-09-13T14:35:19.702 INFO:tasks.rados.rados.0.smithi151.stdout:3432: writing smithi15186934-5 from 679346 to 1194872 tid 1 2024-09-13T14:35:19.709 INFO:tasks.rados.rados.0.smithi151.stdout:3432: writing smithi15186934-5 from 1868811 to 2446593 tid 2 2024-09-13T14:35:19.715 INFO:tasks.rados.rados.0.smithi151.stdout:3432: writing smithi15186934-5 from 3179173 to 3479346 tid 3 2024-09-13T14:35:19.715 INFO:tasks.rados.rados.0.smithi151.stdout:3433: rollback oid 38 current snap is 437 2024-09-13T14:35:19.715 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 38 to 430 2024-09-13T14:35:19.715 INFO:tasks.rados.rados.0.smithi151.stdout:3429: finishing rollback tid 0 to smithi15186934-18 2024-09-13T14:35:19.716 INFO:tasks.rados.rados.0.smithi151.stdout:3430: expect (ObjNum 1201 snap 406 seq_num 1201) 2024-09-13T14:35:19.849 INFO:tasks.rados.rados.0.smithi151.stdout:3428: finishing write tid 3 to smithi15186934-25 2024-09-13T14:35:19.850 INFO:tasks.rados.rados.0.smithi151.stdout:3428: finishing write tid 4 to smithi15186934-25 2024-09-13T14:35:19.850 INFO:tasks.rados.rados.0.smithi151.stdout:3428: finishing write tid 5 to smithi15186934-25 2024-09-13T14:35:19.850 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 1647 (ObjNum 1276 snap 437 seq_num 1276) dirty exists 2024-09-13T14:35:19.850 INFO:tasks.rados.rados.0.smithi151.stdout:3428: left oid 25 (ObjNum 1276 snap 437 seq_num 1276) 2024-09-13T14:35:19.850 INFO:tasks.rados.rados.0.smithi151.stdout:3429: finishing rollback tid 1 to smithi15186934-18 2024-09-13T14:35:19.850 INFO:tasks.rados.rados.0.smithi151.stdout:3429: finishing rollback tid 2 to smithi15186934-18 2024-09-13T14:35:19.850 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 18 v 887 (ObjNum 1203 snap 407 seq_num 1203) dirty exists 2024-09-13T14:35:19.850 INFO:tasks.rados.rados.0.smithi151.stdout:3428: done (5 left) 2024-09-13T14:35:19.850 INFO:tasks.rados.rados.0.smithi151.stdout:3429: done (4 left) 2024-09-13T14:35:19.850 INFO:tasks.rados.rados.0.smithi151.stdout:3430: done (3 left) 2024-09-13T14:35:19.850 INFO:tasks.rados.rados.0.smithi151.stdout:3434: read oid 12 snap -1 2024-09-13T14:35:19.851 INFO:tasks.rados.rados.0.smithi151.stdout:3434: expect (ObjNum 1250 snap 425 seq_num 1250) 2024-09-13T14:35:19.851 INFO:tasks.rados.rados.0.smithi151.stdout:3435: delete oid 42 current snap is 437 2024-09-13T14:35:19.851 INFO:tasks.rados.rados.0.smithi151.stdout:3431: finishing write tid 1 to smithi15186934-3 2024-09-13T14:35:19.851 INFO:tasks.rados.rados.0.smithi151.stdout:3431: finishing write tid 2 to smithi15186934-3 2024-09-13T14:35:19.851 INFO:tasks.rados.rados.0.smithi151.stdout:3431: finishing write tid 3 to smithi15186934-3 2024-09-13T14:35:19.851 INFO:tasks.rados.rados.0.smithi151.stdout:3431: finishing write tid 4 to smithi15186934-3 2024-09-13T14:35:19.851 INFO:tasks.rados.rados.0.smithi151.stdout:3431: finishing write tid 5 to smithi15186934-3 2024-09-13T14:35:19.851 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 1325 (ObjNum 1277 snap 437 seq_num 1277) dirty exists 2024-09-13T14:35:19.851 INFO:tasks.rados.rados.0.smithi151.stdout:3431: left oid 3 (ObjNum 1277 snap 437 seq_num 1277) 2024-09-13T14:35:19.851 INFO:tasks.rados.rados.0.smithi151.stdout:3432: finishing write tid 1 to smithi15186934-5 2024-09-13T14:35:19.851 INFO:tasks.rados.rados.0.smithi151.stdout:3432: finishing write tid 2 to smithi15186934-5 2024-09-13T14:35:19.851 INFO:tasks.rados.rados.0.smithi151.stdout:3432: finishing write tid 3 to smithi15186934-5 2024-09-13T14:35:19.851 INFO:tasks.rados.rados.0.smithi151.stdout:3432: finishing write tid 4 to smithi15186934-5 2024-09-13T14:35:19.852 INFO:tasks.rados.rados.0.smithi151.stdout:3432: finishing write tid 5 to smithi15186934-5 2024-09-13T14:35:19.852 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 1303 (ObjNum 1278 snap 437 seq_num 1278) dirty exists 2024-09-13T14:35:19.852 INFO:tasks.rados.rados.0.smithi151.stdout:3432: left oid 5 (ObjNum 1278 snap 437 seq_num 1278) 2024-09-13T14:35:19.852 INFO:tasks.rados.rados.0.smithi151.stdout:3433: finishing rollback tid 0 to smithi15186934-38 2024-09-13T14:35:19.852 INFO:tasks.rados.rados.0.smithi151.stdout:3433: finishing rollback tid 1 to smithi15186934-38 2024-09-13T14:35:19.852 INFO:tasks.rados.rados.0.smithi151.stdout:3433: finishing rollback tid 2 to smithi15186934-38 2024-09-13T14:35:19.852 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 1451 (ObjNum 1261 snap 428 seq_num 1261) dirty exists 2024-09-13T14:35:19.853 INFO:tasks.rados.rados.0.smithi151.stdout:3431: done (4 left) 2024-09-13T14:35:19.853 INFO:tasks.rados.rados.0.smithi151.stdout:3432: done (3 left) 2024-09-13T14:35:19.853 INFO:tasks.rados.rados.0.smithi151.stdout:3433: done (2 left) 2024-09-13T14:35:19.853 INFO:tasks.rados.rados.0.smithi151.stdout:3435: done (1 left) 2024-09-13T14:35:19.853 INFO:tasks.rados.rados.0.smithi151.stdout:3436: write oid 6 current snap is 437 2024-09-13T14:35:19.853 INFO:tasks.rados.rados.0.smithi151.stdout:3436: seq_num 1280 ranges {712960=666024,1985775=581451,3234019=278941} 2024-09-13T14:35:19.859 INFO:tasks.rados.rados.0.smithi151.stdout:3436: writing smithi15186934-6 from 712960 to 1378984 tid 1 2024-09-13T14:35:19.866 INFO:tasks.rados.rados.0.smithi151.stdout:3436: writing smithi15186934-6 from 1985775 to 2567226 tid 2 2024-09-13T14:35:19.871 INFO:tasks.rados.rados.0.smithi151.stdout:3436: writing smithi15186934-6 from 3234019 to 3512960 tid 3 2024-09-13T14:35:19.872 INFO:tasks.rados.rados.0.smithi151.stdout:3437: read oid 49 snap 430 2024-09-13T14:35:19.872 INFO:tasks.rados.rados.0.smithi151.stdout:3437: expect deleted 2024-09-13T14:35:19.872 INFO:tasks.rados.rados.0.smithi151.stdout:3438: write oid 2 current snap is 437 2024-09-13T14:35:19.872 INFO:tasks.rados.rados.0.smithi151.stdout:3438: seq_num 1281 ranges {729767=741273,1927904=725109,3145089=384678} 2024-09-13T14:35:19.879 INFO:tasks.rados.rados.0.smithi151.stdout:3438: writing smithi15186934-2 from 729767 to 1471040 tid 1 2024-09-13T14:35:19.886 INFO:tasks.rados.rados.0.smithi151.stdout:3438: writing smithi15186934-2 from 1927904 to 2653013 tid 2 2024-09-13T14:35:19.891 INFO:tasks.rados.rados.0.smithi151.stdout:3438: writing smithi15186934-2 from 3145089 to 3529767 tid 3 2024-09-13T14:35:19.891 INFO:tasks.rados.rados.0.smithi151.stdout:3439: snap_remove snap 435 2024-09-13T14:35:20.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:19 smithi184 ceph-mon[92527]: osdmap e928: 8 total, 8 up, 8 in 2024-09-13T14:35:20.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:19 smithi184 ceph-mon[92527]: osdmap e929: 8 total, 8 up, 8 in 2024-09-13T14:35:20.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:19 smithi151 ceph-mon[104450]: osdmap e928: 8 total, 8 up, 8 in 2024-09-13T14:35:20.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:19 smithi151 ceph-mon[104450]: osdmap e929: 8 total, 8 up, 8 in 2024-09-13T14:35:20.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:19 smithi151 ceph-mon[108966]: osdmap e928: 8 total, 8 up, 8 in 2024-09-13T14:35:20.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:19 smithi151 ceph-mon[108966]: osdmap e929: 8 total, 8 up, 8 in 2024-09-13T14:35:20.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:35:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:35:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:35:20.794 INFO:tasks.rados.rados.0.smithi151.stdout:3439: done (4 left) 2024-09-13T14:35:20.794 INFO:tasks.rados.rados.0.smithi151.stdout:3440: write oid 24 current snap is 437 2024-09-13T14:35:20.794 INFO:tasks.rados.rados.0.smithi151.stdout:3434: expect (ObjNum 1250 snap 425 seq_num 1250) 2024-09-13T14:35:20.936 INFO:tasks.rados.rados.0.smithi151.stdout:3436: finishing write tid 1 to smithi15186934-6 2024-09-13T14:35:20.937 INFO:tasks.rados.rados.0.smithi151.stdout:3436: finishing write tid 2 to smithi15186934-6 2024-09-13T14:35:20.937 INFO:tasks.rados.rados.0.smithi151.stdout:3440: seq_num 1282 ranges {746574=416522,1586386=468767,2772512=629544} 2024-09-13T14:35:20.942 INFO:tasks.rados.rados.0.smithi151.stdout:3440: writing smithi15186934-24 from 746574 to 1163096 tid 1 2024-09-13T14:35:20.948 INFO:tasks.rados.rados.0.smithi151.stdout:3440: writing smithi15186934-24 from 1586386 to 2055153 tid 2 2024-09-13T14:35:20.955 INFO:tasks.rados.rados.0.smithi151.stdout:3440: writing smithi15186934-24 from 2772512 to 3402056 tid 3 2024-09-13T14:35:20.955 INFO:tasks.rados.rados.0.smithi151.stdout:3434: done (4 left) 2024-09-13T14:35:20.956 INFO:tasks.rados.rados.0.smithi151.stdout:3441: snap_create 2024-09-13T14:35:20.956 INFO:tasks.rados.rados.0.smithi151.stdout:3436: finishing write tid 3 to smithi15186934-6 2024-09-13T14:35:20.956 INFO:tasks.rados.rados.0.smithi151.stdout:3436: finishing write tid 4 to smithi15186934-6 2024-09-13T14:35:20.956 INFO:tasks.rados.rados.0.smithi151.stdout:3436: finishing write tid 5 to smithi15186934-6 2024-09-13T14:35:20.956 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 6 v 773 (ObjNum 1280 snap 437 seq_num 1280) dirty exists 2024-09-13T14:35:20.956 INFO:tasks.rados.rados.0.smithi151.stdout:3436: left oid 6 (ObjNum 1280 snap 437 seq_num 1280) 2024-09-13T14:35:20.956 INFO:tasks.rados.rados.0.smithi151.stdout:3438: finishing write tid 1 to smithi15186934-2 2024-09-13T14:35:20.956 INFO:tasks.rados.rados.0.smithi151.stdout:3438: finishing write tid 2 to smithi15186934-2 2024-09-13T14:35:20.956 INFO:tasks.rados.rados.0.smithi151.stdout:3438: finishing write tid 3 to smithi15186934-2 2024-09-13T14:35:20.956 INFO:tasks.rados.rados.0.smithi151.stdout:3438: finishing write tid 4 to smithi15186934-2 2024-09-13T14:35:20.956 INFO:tasks.rados.rados.0.smithi151.stdout:3438: finishing write tid 5 to smithi15186934-2 2024-09-13T14:35:20.956 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 2 v 1308 (ObjNum 1281 snap 437 seq_num 1281) dirty exists 2024-09-13T14:35:20.956 INFO:tasks.rados.rados.0.smithi151.stdout:3438: left oid 2 (ObjNum 1281 snap 437 seq_num 1281) 2024-09-13T14:35:20.956 INFO:tasks.rados.rados.0.smithi151.stdout:3440: finishing write tid 1 to smithi15186934-24 2024-09-13T14:35:20.957 INFO:tasks.rados.rados.0.smithi151.stdout:3440: finishing write tid 2 to smithi15186934-24 2024-09-13T14:35:20.960 INFO:tasks.rados.rados.0.smithi151.stdout:3440: finishing write tid 3 to smithi15186934-24 2024-09-13T14:35:20.960 INFO:tasks.rados.rados.0.smithi151.stdout:3440: finishing write tid 4 to smithi15186934-24 2024-09-13T14:35:20.960 INFO:tasks.rados.rados.0.smithi151.stdout:3440: finishing write tid 5 to smithi15186934-24 2024-09-13T14:35:20.960 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 24 v 1960 (ObjNum 1282 snap 437 seq_num 1282) dirty exists 2024-09-13T14:35:20.960 INFO:tasks.rados.rados.0.smithi151.stdout:3440: left oid 24 (ObjNum 1282 snap 437 seq_num 1282) 2024-09-13T14:35:21.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:20 smithi184 ceph-mon[92527]: pgmap v1011: 89 pgs: 89 active+clean; 238 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 8.5 MiB/s rd, 0 B/s wr, 7 op/s 2024-09-13T14:35:21.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:20 smithi151 ceph-mon[104450]: pgmap v1011: 89 pgs: 89 active+clean; 238 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 8.5 MiB/s rd, 0 B/s wr, 7 op/s 2024-09-13T14:35:21.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:20 smithi151 ceph-mon[108966]: pgmap v1011: 89 pgs: 89 active+clean; 238 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 8.5 MiB/s rd, 0 B/s wr, 7 op/s 2024-09-13T14:35:21.805 INFO:tasks.rados.rados.0.smithi151.stdout:3436: done (4 left) 2024-09-13T14:35:21.805 INFO:tasks.rados.rados.0.smithi151.stdout:3437: done (3 left) 2024-09-13T14:35:21.805 INFO:tasks.rados.rados.0.smithi151.stdout:3438: done (2 left) 2024-09-13T14:35:21.805 INFO:tasks.rados.rados.0.smithi151.stdout:3440: done (1 left) 2024-09-13T14:35:21.805 INFO:tasks.rados.rados.0.smithi151.stdout:3441: done (0 left) 2024-09-13T14:35:21.805 INFO:tasks.rados.rados.0.smithi151.stdout:3442: write oid 16 current snap is 438 2024-09-13T14:35:21.806 INFO:tasks.rados.rados.0.smithi151.stdout:3442: seq_num 1283 ranges {763381=491771,2044868=728778,3199935=363446} 2024-09-13T14:35:21.811 INFO:tasks.rados.rados.0.smithi151.stdout:3442: writing smithi15186934-16 from 763381 to 1255152 tid 1 2024-09-13T14:35:21.820 INFO:tasks.rados.rados.0.smithi151.stdout:3442: writing smithi15186934-16 from 2044868 to 2773646 tid 2 2024-09-13T14:35:21.824 INFO:tasks.rados.rados.0.smithi151.stdout:3442: writing smithi15186934-16 from 3199935 to 3563381 tid 3 2024-09-13T14:35:21.824 INFO:tasks.rados.rados.0.smithi151.stdout:3443: write oid 24 current snap is 438 2024-09-13T14:35:21.824 INFO:tasks.rados.rados.0.smithi151.stdout:3442: finishing write tid 1 to smithi15186934-16 2024-09-13T14:35:21.824 INFO:tasks.rados.rados.0.smithi151.stdout:3442: finishing write tid 2 to smithi15186934-16 2024-09-13T14:35:21.825 INFO:tasks.rados.rados.0.smithi151.stdout:3443: seq_num 1284 ranges {780188=567020,2103350=472436,3227358=352830} 2024-09-13T14:35:21.831 INFO:tasks.rados.rados.0.smithi151.stdout:3443: writing smithi15186934-24 from 780188 to 1347208 tid 1 2024-09-13T14:35:21.838 INFO:tasks.rados.rados.0.smithi151.stdout:3443: writing smithi15186934-24 from 2103350 to 2575786 tid 2 2024-09-13T14:35:21.844 INFO:tasks.rados.rados.0.smithi151.stdout:3443: writing smithi15186934-24 from 3227358 to 3580188 tid 3 2024-09-13T14:35:21.844 INFO:tasks.rados.rados.0.smithi151.stdout:3444: rollback oid 35 current snap is 438 2024-09-13T14:35:21.844 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 35 to 433 2024-09-13T14:35:21.844 INFO:tasks.rados.rados.0.smithi151.stdout:3442: finishing write tid 3 to smithi15186934-16 2024-09-13T14:35:21.844 INFO:tasks.rados.rados.0.smithi151.stdout:3442: finishing write tid 4 to smithi15186934-16 2024-09-13T14:35:21.844 INFO:tasks.rados.rados.0.smithi151.stdout:3442: finishing write tid 5 to smithi15186934-16 2024-09-13T14:35:21.845 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 16 v 959 (ObjNum 1283 snap 438 seq_num 1283) dirty exists 2024-09-13T14:35:21.845 INFO:tasks.rados.rados.0.smithi151.stdout:3442: left oid 16 (ObjNum 1283 snap 438 seq_num 1283) 2024-09-13T14:35:21.845 INFO:tasks.rados.rados.0.smithi151.stdout:3443: finishing write tid 1 to smithi15186934-24 2024-09-13T14:35:21.845 INFO:tasks.rados.rados.0.smithi151.stdout:3443: finishing write tid 2 to smithi15186934-24 2024-09-13T14:35:21.845 INFO:tasks.rados.rados.0.smithi151.stdout:3442: done (2 left) 2024-09-13T14:35:21.845 INFO:tasks.rados.rados.0.smithi151.stdout:3445: write oid 45 current snap is 438 2024-09-13T14:35:21.845 INFO:tasks.rados.rados.0.smithi151.stdout:3445: seq_num 1285 ranges {796995=758622,2161832=732447} 2024-09-13T14:35:21.852 INFO:tasks.rados.rados.0.smithi151.stdout:3445: writing smithi15186934-45 from 796995 to 1555617 tid 1 2024-09-13T14:35:21.859 INFO:tasks.rados.rados.0.smithi151.stdout:3445: writing smithi15186934-45 from 2161832 to 2894279 tid 2 2024-09-13T14:35:21.860 INFO:tasks.rados.rados.0.smithi151.stdout:3446: snap_create 2024-09-13T14:35:21.860 INFO:tasks.rados.rados.0.smithi151.stdout:3443: finishing write tid 3 to smithi15186934-24 2024-09-13T14:35:21.860 INFO:tasks.rados.rados.0.smithi151.stdout:3444: finishing rollback tid 1 to smithi15186934-35 2024-09-13T14:35:21.860 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 35 v 0 (ObjNum 33 snap 0 seq_num 3092775008) dirty dne 2024-09-13T14:35:21.860 INFO:tasks.rados.rados.0.smithi151.stdout:3443: finishing write tid 4 to smithi15186934-24 2024-09-13T14:35:21.860 INFO:tasks.rados.rados.0.smithi151.stdout:3443: finishing write tid 5 to smithi15186934-24 2024-09-13T14:35:21.860 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 24 v 1965 (ObjNum 1284 snap 438 seq_num 1284) dirty exists 2024-09-13T14:35:21.860 INFO:tasks.rados.rados.0.smithi151.stdout:3443: left oid 24 (ObjNum 1284 snap 438 seq_num 1284) 2024-09-13T14:35:21.860 INFO:tasks.rados.rados.0.smithi151.stdout:3445: finishing write tid 1 to smithi15186934-45 2024-09-13T14:35:21.863 INFO:tasks.rados.rados.0.smithi151.stdout:3445: finishing write tid 2 to smithi15186934-45 2024-09-13T14:35:21.863 INFO:tasks.rados.rados.0.smithi151.stdout:3445: finishing write tid 3 to smithi15186934-45 2024-09-13T14:35:21.864 INFO:tasks.rados.rados.0.smithi151.stdout:3445: finishing write tid 4 to smithi15186934-45 2024-09-13T14:35:21.864 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 1653 (ObjNum 1285 snap 438 seq_num 1285) dirty exists 2024-09-13T14:35:21.864 INFO:tasks.rados.rados.0.smithi151.stdout:3445: left oid 45 (ObjNum 1285 snap 438 seq_num 1285) 2024-09-13T14:35:22.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:21 smithi184 ceph-mon[92527]: osdmap e930: 8 total, 8 up, 8 in 2024-09-13T14:35:22.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:21 smithi151 ceph-mon[108966]: osdmap e930: 8 total, 8 up, 8 in 2024-09-13T14:35:22.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:21 smithi151 ceph-mon[104450]: osdmap e930: 8 total, 8 up, 8 in 2024-09-13T14:35:22.811 INFO:tasks.rados.rados.0.smithi151.stdout:3443: done (3 left) 2024-09-13T14:35:22.811 INFO:tasks.rados.rados.0.smithi151.stdout:3444: done (2 left) 2024-09-13T14:35:22.811 INFO:tasks.rados.rados.0.smithi151.stdout:3445: done (1 left) 2024-09-13T14:35:22.812 INFO:tasks.rados.rados.0.smithi151.stdout:3446: done (0 left) 2024-09-13T14:35:22.812 INFO:tasks.rados.rados.0.smithi151.stdout:3447: snap_remove snap 432 2024-09-13T14:35:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:22 smithi184 ceph-mon[92527]: pgmap v1014: 89 pgs: 89 active+clean; 238 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 9.1 MiB/s rd, 359 KiB/s wr, 10 op/s 2024-09-13T14:35:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:22 smithi184 ceph-mon[92527]: osdmap e931: 8 total, 8 up, 8 in 2024-09-13T14:35:23.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:22 smithi151 ceph-mon[104450]: pgmap v1014: 89 pgs: 89 active+clean; 238 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 9.1 MiB/s rd, 359 KiB/s wr, 10 op/s 2024-09-13T14:35:23.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:22 smithi151 ceph-mon[104450]: osdmap e931: 8 total, 8 up, 8 in 2024-09-13T14:35:23.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:22 smithi151 ceph-mon[108966]: pgmap v1014: 89 pgs: 89 active+clean; 238 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 9.1 MiB/s rd, 359 KiB/s wr, 10 op/s 2024-09-13T14:35:23.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:22 smithi151 ceph-mon[108966]: osdmap e931: 8 total, 8 up, 8 in 2024-09-13T14:35:23.819 INFO:tasks.rados.rados.0.smithi151.stdout:3447: done (0 left) 2024-09-13T14:35:23.819 INFO:tasks.rados.rados.0.smithi151.stdout:3448: read oid 31 snap 424 2024-09-13T14:35:23.819 INFO:tasks.rados.rados.0.smithi151.stdout:3448: expect deleted 2024-09-13T14:35:23.819 INFO:tasks.rados.rados.0.smithi151.stdout:3449: write oid 25 current snap is 439 2024-09-13T14:35:23.819 INFO:tasks.rados.rados.0.smithi151.stdout:3449: seq_num 1286 ranges {413802=433871,1420314=476105,2482204=790985} 2024-09-13T14:35:23.824 INFO:tasks.rados.rados.0.smithi151.stdout:3449: writing smithi15186934-25 from 413802 to 847673 tid 1 2024-09-13T14:35:23.830 INFO:tasks.rados.rados.0.smithi151.stdout:3449: writing smithi15186934-25 from 1420314 to 1896419 tid 2 2024-09-13T14:35:23.838 INFO:tasks.rados.rados.0.smithi151.stdout:3449: writing smithi15186934-25 from 2482204 to 3273189 tid 3 2024-09-13T14:35:23.838 INFO:tasks.rados.rados.0.smithi151.stdout:3450: snap_remove snap 379 2024-09-13T14:35:24.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:23 smithi184 ceph-mon[92527]: osdmap e932: 8 total, 8 up, 8 in 2024-09-13T14:35:24.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:23 smithi151 ceph-mon[104450]: osdmap e932: 8 total, 8 up, 8 in 2024-09-13T14:35:24.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:23 smithi151 ceph-mon[108966]: osdmap e932: 8 total, 8 up, 8 in 2024-09-13T14:35:24.636 INFO:tasks.rados.rados.0.smithi151.stdout:3450: done (2 left) 2024-09-13T14:35:24.636 INFO:tasks.rados.rados.0.smithi151.stdout:3451: read oid 21 snap -1 2024-09-13T14:35:24.637 INFO:tasks.rados.rados.0.smithi151.stdout:3451: expect (ObjNum 1258 snap 426 seq_num 1258) 2024-09-13T14:35:24.637 INFO:tasks.rados.rados.0.smithi151.stdout:3452: snap_create 2024-09-13T14:35:24.637 INFO:tasks.rados.rados.0.smithi151.stdout:3449: finishing write tid 1 to smithi15186934-25 2024-09-13T14:35:24.637 INFO:tasks.rados.rados.0.smithi151.stdout:3449: finishing write tid 2 to smithi15186934-25 2024-09-13T14:35:24.637 INFO:tasks.rados.rados.0.smithi151.stdout:3449: finishing write tid 3 to smithi15186934-25 2024-09-13T14:35:24.637 INFO:tasks.rados.rados.0.smithi151.stdout:3449: finishing write tid 4 to smithi15186934-25 2024-09-13T14:35:24.637 INFO:tasks.rados.rados.0.smithi151.stdout:3449: finishing write tid 5 to smithi15186934-25 2024-09-13T14:35:24.637 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 1660 (ObjNum 1286 snap 439 seq_num 1286) dirty exists 2024-09-13T14:35:24.637 INFO:tasks.rados.rados.0.smithi151.stdout:3449: left oid 25 (ObjNum 1286 snap 439 seq_num 1286) 2024-09-13T14:35:24.656 INFO:tasks.rados.rados.0.smithi151.stdout:3451: expect (ObjNum 1258 snap 426 seq_num 1258) 2024-09-13T14:35:24.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:24 smithi151 ceph-mon[108966]: pgmap v1017: 89 pgs: 89 active+clean; 255 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.7 MiB/s rd, 3.3 MiB/s wr, 20 op/s 2024-09-13T14:35:24.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:24 smithi151 ceph-mon[108966]: osdmap e933: 8 total, 8 up, 8 in 2024-09-13T14:35:24.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:24 smithi151 ceph-mon[108966]: osdmap e934: 8 total, 8 up, 8 in 2024-09-13T14:35:24.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:24 smithi151 ceph-mon[104450]: pgmap v1017: 89 pgs: 89 active+clean; 255 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.7 MiB/s rd, 3.3 MiB/s wr, 20 op/s 2024-09-13T14:35:24.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:24 smithi151 ceph-mon[104450]: osdmap e933: 8 total, 8 up, 8 in 2024-09-13T14:35:24.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:24 smithi151 ceph-mon[104450]: osdmap e934: 8 total, 8 up, 8 in 2024-09-13T14:35:24.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:35:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:35:24] "GET /metrics HTTP/1.1" 200 36508 "" "Prometheus/2.51.0" 2024-09-13T14:35:25.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:24 smithi184 ceph-mon[92527]: pgmap v1017: 89 pgs: 89 active+clean; 255 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.7 MiB/s rd, 3.3 MiB/s wr, 20 op/s 2024-09-13T14:35:25.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:24 smithi184 ceph-mon[92527]: osdmap e933: 8 total, 8 up, 8 in 2024-09-13T14:35:25.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:24 smithi184 ceph-mon[92527]: osdmap e934: 8 total, 8 up, 8 in 2024-09-13T14:35:25.639 INFO:tasks.rados.rados.0.smithi151.stdout:3448: done (3 left) 2024-09-13T14:35:25.639 INFO:tasks.rados.rados.0.smithi151.stdout:3449: done (2 left) 2024-09-13T14:35:25.640 INFO:tasks.rados.rados.0.smithi151.stdout:3451: done (1 left) 2024-09-13T14:35:25.640 INFO:tasks.rados.rados.0.smithi151.stdout:3452: done (0 left) 2024-09-13T14:35:25.640 INFO:tasks.rados.rados.0.smithi151.stdout:3453: snap_remove snap 425 2024-09-13T14:35:26.644 INFO:tasks.rados.rados.0.smithi151.stdout:3453: done (0 left) 2024-09-13T14:35:26.644 INFO:tasks.rados.rados.0.smithi151.stdout:3454: rollback oid 11 current snap is 440 2024-09-13T14:35:26.644 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 11 to 430 2024-09-13T14:35:26.644 INFO:tasks.rados.rados.0.smithi151.stdout:3455: snap_remove snap 421 2024-09-13T14:35:27.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:26 smithi184 ceph-mon[92527]: pgmap v1020: 89 pgs: 89 active+clean; 258 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 17 op/s 2024-09-13T14:35:27.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:26 smithi184 ceph-mon[92527]: osdmap e935: 8 total, 8 up, 8 in 2024-09-13T14:35:27.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:26 smithi151 ceph-mon[108966]: pgmap v1020: 89 pgs: 89 active+clean; 258 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 17 op/s 2024-09-13T14:35:27.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:26 smithi151 ceph-mon[108966]: osdmap e935: 8 total, 8 up, 8 in 2024-09-13T14:35:27.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:26 smithi151 ceph-mon[104450]: pgmap v1020: 89 pgs: 89 active+clean; 258 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 17 op/s 2024-09-13T14:35:27.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:26 smithi151 ceph-mon[104450]: osdmap e935: 8 total, 8 up, 8 in 2024-09-13T14:35:27.643 INFO:tasks.rados.rados.0.smithi151.stdout:3454: finishing rollback tid 0 to smithi15186934-11 2024-09-13T14:35:27.643 INFO:tasks.rados.rados.0.smithi151.stdout:3454: finishing rollback tid 1 to smithi15186934-11 2024-09-13T14:35:27.643 INFO:tasks.rados.rados.0.smithi151.stdout:3454: finishing rollback tid 2 to smithi15186934-11 2024-09-13T14:35:27.643 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 11 v 901 (ObjNum 1167 snap 396 seq_num 1167) dirty exists 2024-09-13T14:35:27.643 INFO:tasks.rados.rados.0.smithi151.stdout:3454: done (1 left) 2024-09-13T14:35:27.643 INFO:tasks.rados.rados.0.smithi151.stdout:3455: done (0 left) 2024-09-13T14:35:27.643 INFO:tasks.rados.rados.0.smithi151.stdout:3456: rollback oid 34 current snap is 440 2024-09-13T14:35:27.643 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 34 to 424 2024-09-13T14:35:27.644 INFO:tasks.rados.rados.0.smithi151.stdout:3457: delete oid 2 current snap is 440 2024-09-13T14:35:27.645 INFO:tasks.rados.rados.0.smithi151.stdout:3456: finishing rollback tid 1 to smithi15186934-34 2024-09-13T14:35:27.645 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 0 (ObjNum 209 snap 0 seq_num 3680511287) dirty dne 2024-09-13T14:35:27.654 INFO:tasks.rados.rados.0.smithi151.stdout:3456: done (1 left) 2024-09-13T14:35:27.654 INFO:tasks.rados.rados.0.smithi151.stdout:3457: done (0 left) 2024-09-13T14:35:27.654 INFO:tasks.rados.rados.0.smithi151.stdout:3458: snap_create 2024-09-13T14:35:28.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:27 smithi184 ceph-mon[92527]: osdmap e936: 8 total, 8 up, 8 in 2024-09-13T14:35:28.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:27 smithi184 ceph-mon[92527]: osdmap e937: 8 total, 8 up, 8 in 2024-09-13T14:35:28.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:27 smithi151 ceph-mon[108966]: osdmap e936: 8 total, 8 up, 8 in 2024-09-13T14:35:28.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:27 smithi151 ceph-mon[108966]: osdmap e937: 8 total, 8 up, 8 in 2024-09-13T14:35:28.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:27 smithi151 ceph-mon[104450]: osdmap e936: 8 total, 8 up, 8 in 2024-09-13T14:35:28.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:27 smithi151 ceph-mon[104450]: osdmap e937: 8 total, 8 up, 8 in 2024-09-13T14:35:28.657 INFO:tasks.rados.rados.0.smithi151.stdout:3458: done (0 left) 2024-09-13T14:35:28.657 INFO:tasks.rados.rados.0.smithi151.stdout:3459: snap_remove snap 434 2024-09-13T14:35:29.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:28 smithi184 ceph-mon[92527]: pgmap v1023: 89 pgs: 89 active+clean; 222 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 6.2 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:35:29.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:28 smithi151 ceph-mon[104450]: pgmap v1023: 89 pgs: 89 active+clean; 222 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 6.2 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:35:29.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:28 smithi151 ceph-mon[108966]: pgmap v1023: 89 pgs: 89 active+clean; 222 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 6.2 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:35:29.640 INFO:tasks.rados.rados.0.smithi151.stdout:3459: done (0 left) 2024-09-13T14:35:29.640 INFO:tasks.rados.rados.0.smithi151.stdout:3460: rollback oid 36 current snap is 441 2024-09-13T14:35:29.640 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 36 to 406 2024-09-13T14:35:29.640 INFO:tasks.rados.rados.0.smithi151.stdout:3461: snap_create 2024-09-13T14:35:29.640 INFO:tasks.rados.rados.0.smithi151.stdout:3460: finishing rollback tid 1 to smithi15186934-36 2024-09-13T14:35:29.640 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 36 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-13T14:35:30.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:29 smithi184 ceph-mon[92527]: osdmap e938: 8 total, 8 up, 8 in 2024-09-13T14:35:30.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:29 smithi184 ceph-mon[92527]: osdmap e939: 8 total, 8 up, 8 in 2024-09-13T14:35:30.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:29 smithi151 ceph-mon[104450]: osdmap e938: 8 total, 8 up, 8 in 2024-09-13T14:35:30.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:29 smithi151 ceph-mon[104450]: osdmap e939: 8 total, 8 up, 8 in 2024-09-13T14:35:30.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:29 smithi151 ceph-mon[108966]: osdmap e938: 8 total, 8 up, 8 in 2024-09-13T14:35:30.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:29 smithi151 ceph-mon[108966]: osdmap e939: 8 total, 8 up, 8 in 2024-09-13T14:35:30.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:35:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:35:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:35:30.642 INFO:tasks.rados.rados.0.smithi151.stdout:3460: done (1 left) 2024-09-13T14:35:30.642 INFO:tasks.rados.rados.0.smithi151.stdout:3461: done (0 left) 2024-09-13T14:35:30.643 INFO:tasks.rados.rados.0.smithi151.stdout:3462: rollback oid 33 current snap is 442 2024-09-13T14:35:30.643 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 33 to 426 2024-09-13T14:35:30.643 INFO:tasks.rados.rados.0.smithi151.stdout:3463: write oid 20 current snap is 442 2024-09-13T14:35:30.643 INFO:tasks.rados.rados.0.smithi151.stdout:3463: seq_num 1288 ranges {447416=584369,1537278=479774,2537050=529882,3524784=122632} 2024-09-13T14:35:30.648 INFO:tasks.rados.rados.0.smithi151.stdout:3463: writing smithi15186934-20 from 447416 to 1031785 tid 1 2024-09-13T14:35:30.654 INFO:tasks.rados.rados.0.smithi151.stdout:3463: writing smithi15186934-20 from 1537278 to 2017052 tid 2 2024-09-13T14:35:30.659 INFO:tasks.rados.rados.0.smithi151.stdout:3463: writing smithi15186934-20 from 2537050 to 3066932 tid 3 2024-09-13T14:35:30.663 INFO:tasks.rados.rados.0.smithi151.stdout:3463: writing smithi15186934-20 from 3524784 to 3647416 tid 4 2024-09-13T14:35:30.663 INFO:tasks.rados.rados.0.smithi151.stdout:3464: write oid 18 current snap is 442 2024-09-13T14:35:30.663 INFO:tasks.rados.rados.0.smithi151.stdout:3464: seq_num 1289 ranges {464223=659618,1879407=739785,3248120=416103} 2024-09-13T14:35:30.669 INFO:tasks.rados.rados.0.smithi151.stdout:3464: writing smithi15186934-18 from 464223 to 1123841 tid 1 2024-09-13T14:35:30.677 INFO:tasks.rados.rados.0.smithi151.stdout:3464: writing smithi15186934-18 from 1879407 to 2619192 tid 2 2024-09-13T14:35:30.684 INFO:tasks.rados.rados.0.smithi151.stdout:3464: writing smithi15186934-18 from 3248120 to 3664223 tid 3 2024-09-13T14:35:30.684 INFO:tasks.rados.rados.0.smithi151.stdout:3465: snap_create 2024-09-13T14:35:30.684 INFO:tasks.rados.rados.0.smithi151.stdout:3462: finishing rollback tid 1 to smithi15186934-33 2024-09-13T14:35:30.684 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 33 v 1254 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-13T14:35:30.684 INFO:tasks.rados.rados.0.smithi151.stdout:3463: finishing write tid 1 to smithi15186934-20 2024-09-13T14:35:30.684 INFO:tasks.rados.rados.0.smithi151.stdout:3463: finishing write tid 2 to smithi15186934-20 2024-09-13T14:35:30.684 INFO:tasks.rados.rados.0.smithi151.stdout:3463: finishing write tid 3 to smithi15186934-20 2024-09-13T14:35:30.684 INFO:tasks.rados.rados.0.smithi151.stdout:3463: finishing write tid 4 to smithi15186934-20 2024-09-13T14:35:30.684 INFO:tasks.rados.rados.0.smithi151.stdout:3463: finishing write tid 5 to smithi15186934-20 2024-09-13T14:35:30.685 INFO:tasks.rados.rados.0.smithi151.stdout:3463: finishing write tid 6 to smithi15186934-20 2024-09-13T14:35:30.685 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 979 (ObjNum 1288 snap 442 seq_num 1288) dirty exists 2024-09-13T14:35:30.685 INFO:tasks.rados.rados.0.smithi151.stdout:3463: left oid 20 (ObjNum 1288 snap 442 seq_num 1288) 2024-09-13T14:35:30.685 INFO:tasks.rados.rados.0.smithi151.stdout:3464: finishing write tid 1 to smithi15186934-18 2024-09-13T14:35:30.685 INFO:tasks.rados.rados.0.smithi151.stdout:3464: finishing write tid 2 to smithi15186934-18 2024-09-13T14:35:30.686 INFO:tasks.rados.rados.0.smithi151.stdout:3464: finishing write tid 3 to smithi15186934-18 2024-09-13T14:35:30.686 INFO:tasks.rados.rados.0.smithi151.stdout:3464: finishing write tid 4 to smithi15186934-18 2024-09-13T14:35:30.686 INFO:tasks.rados.rados.0.smithi151.stdout:3464: finishing write tid 5 to smithi15186934-18 2024-09-13T14:35:30.686 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 18 v 914 (ObjNum 1289 snap 442 seq_num 1289) dirty exists 2024-09-13T14:35:30.686 INFO:tasks.rados.rados.0.smithi151.stdout:3464: left oid 18 (ObjNum 1289 snap 442 seq_num 1289) 2024-09-13T14:35:31.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:30 smithi184 ceph-mon[92527]: pgmap v1026: 89 pgs: 89 active+clean; 216 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 4.5 MiB/s rd, 415 KiB/s wr, 4 op/s 2024-09-13T14:35:31.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:30 smithi184 ceph-mon[92527]: osdmap e940: 8 total, 8 up, 8 in 2024-09-13T14:35:31.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:30 smithi151 ceph-mon[104450]: pgmap v1026: 89 pgs: 89 active+clean; 216 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 4.5 MiB/s rd, 415 KiB/s wr, 4 op/s 2024-09-13T14:35:31.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:30 smithi151 ceph-mon[104450]: osdmap e940: 8 total, 8 up, 8 in 2024-09-13T14:35:31.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:30 smithi151 ceph-mon[108966]: pgmap v1026: 89 pgs: 89 active+clean; 216 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 4.5 MiB/s rd, 415 KiB/s wr, 4 op/s 2024-09-13T14:35:31.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:30 smithi151 ceph-mon[108966]: osdmap e940: 8 total, 8 up, 8 in 2024-09-13T14:35:31.678 INFO:tasks.rados.rados.0.smithi151.stdout:3462: done (3 left) 2024-09-13T14:35:31.678 INFO:tasks.rados.rados.0.smithi151.stdout:3463: done (2 left) 2024-09-13T14:35:31.678 INFO:tasks.rados.rados.0.smithi151.stdout:3464: done (1 left) 2024-09-13T14:35:31.679 INFO:tasks.rados.rados.0.smithi151.stdout:3465: done (0 left) 2024-09-13T14:35:31.679 INFO:tasks.rados.rados.0.smithi151.stdout:3466: write oid 49 current snap is 443 2024-09-13T14:35:31.679 INFO:tasks.rados.rados.0.smithi151.stdout:3466: seq_num 1290 ranges {481030=734867,1937889=483443,2875543=552426} 2024-09-13T14:35:31.685 INFO:tasks.rados.rados.0.smithi151.stdout:3466: writing smithi15186934-49 from 481030 to 1215897 tid 1 2024-09-13T14:35:31.691 INFO:tasks.rados.rados.0.smithi151.stdout:3466: writing smithi15186934-49 from 1937889 to 2421332 tid 2 2024-09-13T14:35:31.697 INFO:tasks.rados.rados.0.smithi151.stdout:3466: writing smithi15186934-49 from 2875543 to 3427969 tid 3 2024-09-13T14:35:31.697 INFO:tasks.rados.rados.0.smithi151.stdout:3467: write oid 11 current snap is 443 2024-09-13T14:35:31.697 INFO:tasks.rados.rados.0.smithi151.stdout:3467: seq_num 1291 ranges {497837=410116,1596371=743454,2902966=763698} 2024-09-13T14:35:31.702 INFO:tasks.rados.rados.0.smithi151.stdout:3467: writing smithi15186934-11 from 497837 to 907953 tid 1 2024-09-13T14:35:31.710 INFO:tasks.rados.rados.0.smithi151.stdout:3467: writing smithi15186934-11 from 1596371 to 2339825 tid 2 2024-09-13T14:35:31.717 INFO:tasks.rados.rados.0.smithi151.stdout:3467: writing smithi15186934-11 from 2902966 to 3666664 tid 3 2024-09-13T14:35:31.718 INFO:tasks.rados.rados.0.smithi151.stdout:3468: write oid 7 current snap is 443 2024-09-13T14:35:31.718 INFO:tasks.rados.rados.0.smithi151.stdout:3466: finishing write tid 1 to smithi15186934-49 2024-09-13T14:35:31.718 INFO:tasks.rados.rados.0.smithi151.stdout:3466: finishing write tid 2 to smithi15186934-49 2024-09-13T14:35:31.718 INFO:tasks.rados.rados.0.smithi151.stdout:3466: finishing write tid 3 to smithi15186934-49 2024-09-13T14:35:31.718 INFO:tasks.rados.rados.0.smithi151.stdout:3468: seq_num 1292 ranges {514644=485365,1654853=487112,2930389=574970} 2024-09-13T14:35:31.723 INFO:tasks.rados.rados.0.smithi151.stdout:3468: writing smithi15186934-7 from 514644 to 1000009 tid 1 2024-09-13T14:35:31.730 INFO:tasks.rados.rados.0.smithi151.stdout:3468: writing smithi15186934-7 from 1654853 to 2141965 tid 2 2024-09-13T14:35:31.738 INFO:tasks.rados.rados.0.smithi151.stdout:3468: writing smithi15186934-7 from 2930389 to 3505359 tid 3 2024-09-13T14:35:31.738 INFO:tasks.rados.rados.0.smithi151.stdout:3469: delete oid 28 current snap is 443 2024-09-13T14:35:31.738 INFO:tasks.rados.rados.0.smithi151.stdout:3466: finishing write tid 4 to smithi15186934-49 2024-09-13T14:35:31.738 INFO:tasks.rados.rados.0.smithi151.stdout:3466: finishing write tid 5 to smithi15186934-49 2024-09-13T14:35:31.738 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 49 v 1618 (ObjNum 1290 snap 443 seq_num 1290) dirty exists 2024-09-13T14:35:31.738 INFO:tasks.rados.rados.0.smithi151.stdout:3466: left oid 49 (ObjNum 1290 snap 443 seq_num 1290) 2024-09-13T14:35:31.738 INFO:tasks.rados.rados.0.smithi151.stdout:3467: finishing write tid 1 to smithi15186934-11 2024-09-13T14:35:31.738 INFO:tasks.rados.rados.0.smithi151.stdout:3467: finishing write tid 2 to smithi15186934-11 2024-09-13T14:35:31.738 INFO:tasks.rados.rados.0.smithi151.stdout:3467: finishing write tid 3 to smithi15186934-11 2024-09-13T14:35:31.739 INFO:tasks.rados.rados.0.smithi151.stdout:3467: finishing write tid 4 to smithi15186934-11 2024-09-13T14:35:31.739 INFO:tasks.rados.rados.0.smithi151.stdout:3467: finishing write tid 5 to smithi15186934-11 2024-09-13T14:35:31.739 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 11 v 919 (ObjNum 1291 snap 443 seq_num 1291) dirty exists 2024-09-13T14:35:31.739 INFO:tasks.rados.rados.0.smithi151.stdout:3467: left oid 11 (ObjNum 1291 snap 443 seq_num 1291) 2024-09-13T14:35:31.739 INFO:tasks.rados.rados.0.smithi151.stdout:3468: finishing write tid 1 to smithi15186934-7 2024-09-13T14:35:31.739 INFO:tasks.rados.rados.0.smithi151.stdout:3468: finishing write tid 2 to smithi15186934-7 2024-09-13T14:35:31.739 INFO:tasks.rados.rados.0.smithi151.stdout:3466: done (3 left) 2024-09-13T14:35:31.740 INFO:tasks.rados.rados.0.smithi151.stdout:3467: done (2 left) 2024-09-13T14:35:31.740 INFO:tasks.rados.rados.0.smithi151.stdout:3469: done (1 left) 2024-09-13T14:35:31.740 INFO:tasks.rados.rados.0.smithi151.stdout:3470: snap_create 2024-09-13T14:35:31.741 INFO:tasks.rados.rados.0.smithi151.stdout:3468: finishing write tid 3 to smithi15186934-7 2024-09-13T14:35:31.741 INFO:tasks.rados.rados.0.smithi151.stdout:3468: finishing write tid 4 to smithi15186934-7 2024-09-13T14:35:31.741 INFO:tasks.rados.rados.0.smithi151.stdout:3468: finishing write tid 5 to smithi15186934-7 2024-09-13T14:35:31.741 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 1980 (ObjNum 1292 snap 443 seq_num 1292) dirty exists 2024-09-13T14:35:31.741 INFO:tasks.rados.rados.0.smithi151.stdout:3468: left oid 7 (ObjNum 1292 snap 443 seq_num 1292) 2024-09-13T14:35:32.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:32.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:32.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:32.684 INFO:tasks.rados.rados.0.smithi151.stdout:3468: done (1 left) 2024-09-13T14:35:32.684 INFO:tasks.rados.rados.0.smithi151.stdout:3470: done (0 left) 2024-09-13T14:35:32.684 INFO:tasks.rados.rados.0.smithi151.stdout:3471: snap_remove snap 442 2024-09-13T14:35:33.010 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:32 smithi151 ceph-mon[104450]: pgmap v1029: 89 pgs: 89 active+clean; 216 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2024-09-13T14:35:33.010 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:32 smithi151 ceph-mon[104450]: osdmap e941: 8 total, 8 up, 8 in 2024-09-13T14:35:33.010 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:32 smithi151 ceph-mon[108966]: pgmap v1029: 89 pgs: 89 active+clean; 216 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2024-09-13T14:35:33.010 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:32 smithi151 ceph-mon[108966]: osdmap e941: 8 total, 8 up, 8 in 2024-09-13T14:35:33.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:32 smithi184 ceph-mon[92527]: pgmap v1029: 89 pgs: 89 active+clean; 216 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2024-09-13T14:35:33.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:32 smithi184 ceph-mon[92527]: osdmap e941: 8 total, 8 up, 8 in 2024-09-13T14:35:33.690 INFO:tasks.rados.rados.0.smithi151.stdout:3471: done (0 left) 2024-09-13T14:35:33.690 INFO:tasks.rados.rados.0.smithi151.stdout:3472: snap_remove snap 406 2024-09-13T14:35:34.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:33 smithi184 ceph-mon[92527]: osdmap e942: 8 total, 8 up, 8 in 2024-09-13T14:35:34.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:33 smithi151 ceph-mon[104450]: osdmap e942: 8 total, 8 up, 8 in 2024-09-13T14:35:34.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:33 smithi151 ceph-mon[108966]: osdmap e942: 8 total, 8 up, 8 in 2024-09-13T14:35:34.639 INFO:tasks.rados.rados.0.smithi151.stdout:3472: done (0 left) 2024-09-13T14:35:34.639 INFO:tasks.rados.rados.0.smithi151.stdout:3473: delete oid 7 current snap is 444 2024-09-13T14:35:34.651 INFO:tasks.rados.rados.0.smithi151.stdout:3473: done (0 left) 2024-09-13T14:35:34.651 INFO:tasks.rados.rados.0.smithi151.stdout:3474: write oid 7 current snap is 444 2024-09-13T14:35:34.651 INFO:tasks.rados.rados.0.smithi151.stdout:3474: seq_num 1295 ranges {565065=427465,1430299=634439,2496305=525139,3472104=292961} 2024-09-13T14:35:34.655 INFO:tasks.rados.rados.0.smithi151.stdout:3474: writing smithi15186934-7 from 565065 to 992530 tid 1 2024-09-13T14:35:34.662 INFO:tasks.rados.rados.0.smithi151.stdout:3474: writing smithi15186934-7 from 1430299 to 2064738 tid 2 2024-09-13T14:35:34.667 INFO:tasks.rados.rados.0.smithi151.stdout:3474: writing smithi15186934-7 from 2496305 to 3021444 tid 3 2024-09-13T14:35:34.671 INFO:tasks.rados.rados.0.smithi151.stdout:3474: writing smithi15186934-7 from 3472104 to 3765065 tid 4 2024-09-13T14:35:34.672 INFO:tasks.rados.rados.0.smithi151.stdout:3475: snap_remove snap 430 2024-09-13T14:35:34.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:34 smithi151 ceph-mon[104450]: pgmap v1032: 89 pgs: 89 active+clean; 223 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 1.2 MiB/s wr, 5 op/s 2024-09-13T14:35:34.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:34 smithi151 ceph-mon[104450]: osdmap e943: 8 total, 8 up, 8 in 2024-09-13T14:35:34.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:34 smithi151 ceph-mon[104450]: osdmap e944: 8 total, 8 up, 8 in 2024-09-13T14:35:34.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:35:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:35:34] "GET /metrics HTTP/1.1" 200 36508 "" "Prometheus/2.51.0" 2024-09-13T14:35:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:34 smithi151 ceph-mon[108966]: pgmap v1032: 89 pgs: 89 active+clean; 223 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 1.2 MiB/s wr, 5 op/s 2024-09-13T14:35:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:34 smithi151 ceph-mon[108966]: osdmap e943: 8 total, 8 up, 8 in 2024-09-13T14:35:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:34 smithi151 ceph-mon[108966]: osdmap e944: 8 total, 8 up, 8 in 2024-09-13T14:35:35.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:34 smithi184 ceph-mon[92527]: pgmap v1032: 89 pgs: 89 active+clean; 223 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 1.2 MiB/s wr, 5 op/s 2024-09-13T14:35:35.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:34 smithi184 ceph-mon[92527]: osdmap e943: 8 total, 8 up, 8 in 2024-09-13T14:35:35.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:34 smithi184 ceph-mon[92527]: osdmap e944: 8 total, 8 up, 8 in 2024-09-13T14:35:35.641 INFO:tasks.rados.rados.0.smithi151.stdout:3475: done (1 left) 2024-09-13T14:35:35.641 INFO:tasks.rados.rados.0.smithi151.stdout:3476: write oid 5 current snap is 444 2024-09-13T14:35:35.641 INFO:tasks.rados.rados.0.smithi151.stdout:3474: finishing write tid 1 to smithi15186934-7 2024-09-13T14:35:35.642 INFO:tasks.rados.rados.0.smithi151.stdout:3476: seq_num 1296 ranges {581872=502714,1488781=494450,2640081=736411} 2024-09-13T14:35:35.648 INFO:tasks.rados.rados.0.smithi151.stdout:3476: writing smithi15186934-5 from 581872 to 1084586 tid 1 2024-09-13T14:35:35.654 INFO:tasks.rados.rados.0.smithi151.stdout:3476: writing smithi15186934-5 from 1488781 to 1983231 tid 2 2024-09-13T14:35:35.662 INFO:tasks.rados.rados.0.smithi151.stdout:3476: writing smithi15186934-5 from 2640081 to 3376492 tid 3 2024-09-13T14:35:35.662 INFO:tasks.rados.rados.0.smithi151.stdout:3477: snap_remove snap 436 2024-09-13T14:35:36.648 INFO:tasks.rados.rados.0.smithi151.stdout:3477: done (2 left) 2024-09-13T14:35:36.648 INFO:tasks.rados.rados.0.smithi151.stdout:3478: snap_create 2024-09-13T14:35:36.648 INFO:tasks.rados.rados.0.smithi151.stdout:3474: finishing write tid 2 to smithi15186934-7 2024-09-13T14:35:36.648 INFO:tasks.rados.rados.0.smithi151.stdout:3474: finishing write tid 3 to smithi15186934-7 2024-09-13T14:35:36.649 INFO:tasks.rados.rados.0.smithi151.stdout:3474: finishing write tid 4 to smithi15186934-7 2024-09-13T14:35:36.649 INFO:tasks.rados.rados.0.smithi151.stdout:3474: finishing write tid 5 to smithi15186934-7 2024-09-13T14:35:36.649 INFO:tasks.rados.rados.0.smithi151.stdout:3474: finishing write tid 6 to smithi15186934-7 2024-09-13T14:35:36.649 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 1993 (ObjNum 1295 snap 444 seq_num 1295) dirty exists 2024-09-13T14:35:36.649 INFO:tasks.rados.rados.0.smithi151.stdout:3474: left oid 7 (ObjNum 1295 snap 444 seq_num 1295) 2024-09-13T14:35:36.649 INFO:tasks.rados.rados.0.smithi151.stdout:3476: finishing write tid 1 to smithi15186934-5 2024-09-13T14:35:36.649 INFO:tasks.rados.rados.0.smithi151.stdout:3476: finishing write tid 2 to smithi15186934-5 2024-09-13T14:35:36.649 INFO:tasks.rados.rados.0.smithi151.stdout:3476: finishing write tid 3 to smithi15186934-5 2024-09-13T14:35:36.649 INFO:tasks.rados.rados.0.smithi151.stdout:3476: finishing write tid 4 to smithi15186934-5 2024-09-13T14:35:36.649 INFO:tasks.rados.rados.0.smithi151.stdout:3476: finishing write tid 5 to smithi15186934-5 2024-09-13T14:35:36.649 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 1342 (ObjNum 1296 snap 444 seq_num 1296) dirty exists 2024-09-13T14:35:36.649 INFO:tasks.rados.rados.0.smithi151.stdout:3476: left oid 5 (ObjNum 1296 snap 444 seq_num 1296) 2024-09-13T14:35:37.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:36 smithi184 ceph-mon[92527]: pgmap v1035: 89 pgs: 89 active+clean; 223 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 1.2 MiB/s wr, 5 op/s 2024-09-13T14:35:37.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:36 smithi184 ceph-mon[92527]: osdmap e945: 8 total, 8 up, 8 in 2024-09-13T14:35:37.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:36 smithi151 ceph-mon[104450]: pgmap v1035: 89 pgs: 89 active+clean; 223 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 1.2 MiB/s wr, 5 op/s 2024-09-13T14:35:37.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:36 smithi151 ceph-mon[104450]: osdmap e945: 8 total, 8 up, 8 in 2024-09-13T14:35:37.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:36 smithi151 ceph-mon[108966]: pgmap v1035: 89 pgs: 89 active+clean; 223 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 1.2 MiB/s wr, 5 op/s 2024-09-13T14:35:37.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:36 smithi151 ceph-mon[108966]: osdmap e945: 8 total, 8 up, 8 in 2024-09-13T14:35:37.653 INFO:tasks.rados.rados.0.smithi151.stdout:3474: done (2 left) 2024-09-13T14:35:37.653 INFO:tasks.rados.rados.0.smithi151.stdout:3476: done (1 left) 2024-09-13T14:35:37.653 INFO:tasks.rados.rados.0.smithi151.stdout:3478: done (0 left) 2024-09-13T14:35:37.654 INFO:tasks.rados.rados.0.smithi151.stdout:3479: read oid 25 snap -1 2024-09-13T14:35:37.654 INFO:tasks.rados.rados.0.smithi151.stdout:3479: expect (ObjNum 1286 snap 439 seq_num 1286) 2024-09-13T14:35:37.654 INFO:tasks.rados.rados.0.smithi151.stdout:3480: read oid 23 snap -1 2024-09-13T14:35:37.654 INFO:tasks.rados.rados.0.smithi151.stdout:3480: expect (ObjNum 1216 snap 409 seq_num 1216) 2024-09-13T14:35:37.654 INFO:tasks.rados.rados.0.smithi151.stdout:3481: delete oid 45 current snap is 445 2024-09-13T14:35:37.660 INFO:tasks.rados.rados.0.smithi151.stdout:3480: expect (ObjNum 1216 snap 409 seq_num 1216) 2024-09-13T14:35:37.786 INFO:tasks.rados.rados.0.smithi151.stdout:3479: expect (ObjNum 1286 snap 439 seq_num 1286) 2024-09-13T14:35:37.976 INFO:tasks.rados.rados.0.smithi151.stdout:3479: done (2 left) 2024-09-13T14:35:37.977 INFO:tasks.rados.rados.0.smithi151.stdout:3480: done (1 left) 2024-09-13T14:35:37.977 INFO:tasks.rados.rados.0.smithi151.stdout:3481: done (0 left) 2024-09-13T14:35:37.977 INFO:tasks.rados.rados.0.smithi151.stdout:3482: write oid 34 current snap is 445 2024-09-13T14:35:37.977 INFO:tasks.rados.rados.0.smithi151.stdout:3482: seq_num 1298 ranges {615486=653212,1889392=498119,2978574=475308} 2024-09-13T14:35:37.983 INFO:tasks.rados.rados.0.smithi151.stdout:3482: writing smithi15186934-34 from 615486 to 1268698 tid 1 2024-09-13T14:35:37.989 INFO:tasks.rados.rados.0.smithi151.stdout:3482: writing smithi15186934-34 from 1889392 to 2387511 tid 2 2024-09-13T14:35:37.995 INFO:tasks.rados.rados.0.smithi151.stdout:3482: writing smithi15186934-34 from 2978574 to 3453882 tid 3 2024-09-13T14:35:37.996 INFO:tasks.rados.rados.0.smithi151.stdout:3483: read oid 6 snap -1 2024-09-13T14:35:37.996 INFO:tasks.rados.rados.0.smithi151.stdout:3483: expect (ObjNum 1280 snap 437 seq_num 1280) 2024-09-13T14:35:37.996 INFO:tasks.rados.rados.0.smithi151.stdout:3484: write oid 36 current snap is 445 2024-09-13T14:35:37.996 INFO:tasks.rados.rados.0.smithi151.stdout:3482: finishing write tid 1 to smithi15186934-34 2024-09-13T14:35:37.996 INFO:tasks.rados.rados.0.smithi151.stdout:3482: finishing write tid 2 to smithi15186934-34 2024-09-13T14:35:37.996 INFO:tasks.rados.rados.0.smithi151.stdout:3484: seq_num 1299 ranges {632293=728461,1947874=641777,3005997=686580} 2024-09-13T14:35:38.002 INFO:tasks.rados.rados.0.smithi151.stdout:3484: writing smithi15186934-36 from 632293 to 1360754 tid 1 2024-09-13T14:35:38.009 INFO:tasks.rados.rados.0.smithi151.stdout:3484: writing smithi15186934-36 from 1947874 to 2589651 tid 2 2024-09-13T14:35:38.016 INFO:tasks.rados.rados.0.smithi151.stdout:3484: writing smithi15186934-36 from 3005997 to 3692577 tid 3 2024-09-13T14:35:38.016 INFO:tasks.rados.rados.0.smithi151.stdout:3485: write oid 1 current snap is 445 2024-09-13T14:35:38.016 INFO:tasks.rados.rados.0.smithi151.stdout:3482: finishing write tid 3 to smithi15186934-34 2024-09-13T14:35:38.016 INFO:tasks.rados.rados.0.smithi151.stdout:3485: seq_num 1300 ranges {649100=403710,1606356=501788,2633420=497852,3636005=213095} 2024-09-13T14:35:38.021 INFO:tasks.rados.rados.0.smithi151.stdout:3485: writing smithi15186934-1 from 649100 to 1052810 tid 1 2024-09-13T14:35:38.027 INFO:tasks.rados.rados.0.smithi151.stdout:3485: writing smithi15186934-1 from 1606356 to 2108144 tid 2 2024-09-13T14:35:38.033 INFO:tasks.rados.rados.0.smithi151.stdout:3485: writing smithi15186934-1 from 2633420 to 3131272 tid 3 2024-09-13T14:35:38.037 INFO:tasks.rados.rados.0.smithi151.stdout:3485: writing smithi15186934-1 from 3636005 to 3849100 tid 4 2024-09-13T14:35:38.037 INFO:tasks.rados.rados.0.smithi151.stdout:3486: write oid 9 current snap is 445 2024-09-13T14:35:38.037 INFO:tasks.rados.rados.0.smithi151.stdout:3482: finishing write tid 4 to smithi15186934-34 2024-09-13T14:35:38.037 INFO:tasks.rados.rados.0.smithi151.stdout:3486: seq_num 1301 ranges {665907=595312,1781191=645446,3177196=688711} 2024-09-13T14:35:38.044 INFO:tasks.rados.rados.0.smithi151.stdout:3486: writing smithi15186934-9 from 665907 to 1261219 tid 1 2024-09-13T14:35:38.050 INFO:tasks.rados.rados.0.smithi151.stdout:3486: writing smithi15186934-9 from 1781191 to 2426637 tid 2 2024-09-13T14:35:38.058 INFO:tasks.rados.rados.0.smithi151.stdout:3486: writing smithi15186934-9 from 3177196 to 3865907 tid 3 2024-09-13T14:35:38.058 INFO:tasks.rados.rados.0.smithi151.stdout:3487: snap_remove snap 439 2024-09-13T14:35:38.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:37 smithi184 ceph-mon[92527]: osdmap e946: 8 total, 8 up, 8 in 2024-09-13T14:35:38.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:37 smithi151 ceph-mon[104450]: osdmap e946: 8 total, 8 up, 8 in 2024-09-13T14:35:38.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:37 smithi151 ceph-mon[108966]: osdmap e946: 8 total, 8 up, 8 in 2024-09-13T14:35:38.659 INFO:tasks.rados.rados.0.smithi151.stdout:3487: done (5 left) 2024-09-13T14:35:38.659 INFO:tasks.rados.rados.0.smithi151.stdout:3488: read oid 17 snap -1 2024-09-13T14:35:38.659 INFO:tasks.rados.rados.0.smithi151.stdout:3488: expect (ObjNum 1231 snap 414 seq_num 1231) 2024-09-13T14:35:38.659 INFO:tasks.rados.rados.0.smithi151.stdout:3489: read oid 46 snap -1 2024-09-13T14:35:38.659 INFO:tasks.rados.rados.0.smithi151.stdout:3489: expect (ObjNum 1135 snap 381 seq_num 1135) 2024-09-13T14:35:38.659 INFO:tasks.rados.rados.0.smithi151.stdout:3490: read oid 47 snap -1 2024-09-13T14:35:38.660 INFO:tasks.rados.rados.0.smithi151.stdout:3490: expect deleted 2024-09-13T14:35:38.660 INFO:tasks.rados.rados.0.smithi151.stdout:3491: rollback oid 49 current snap is 445 2024-09-13T14:35:38.660 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 49 to 431 2024-09-13T14:35:38.660 INFO:tasks.rados.rados.0.smithi151.stdout:3482: finishing write tid 5 to smithi15186934-34 2024-09-13T14:35:38.660 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 1337 (ObjNum 1298 snap 445 seq_num 1298) dirty exists 2024-09-13T14:35:38.660 INFO:tasks.rados.rados.0.smithi151.stdout:3482: left oid 34 (ObjNum 1298 snap 445 seq_num 1298) 2024-09-13T14:35:38.660 INFO:tasks.rados.rados.0.smithi151.stdout:3482: done (8 left) 2024-09-13T14:35:38.660 INFO:tasks.rados.rados.0.smithi151.stdout:3492: read oid 42 snap -1 2024-09-13T14:35:38.660 INFO:tasks.rados.rados.0.smithi151.stdout:3492: expect deleted 2024-09-13T14:35:38.660 INFO:tasks.rados.rados.0.smithi151.stdout:3493: write oid 45 current snap is 445 2024-09-13T14:35:38.660 INFO:tasks.rados.rados.0.smithi151.stdout:3493: seq_num 1302 ranges {682714=670561,2123320=505457,3088266=636749} 2024-09-13T14:35:38.667 INFO:tasks.rados.rados.0.smithi151.stdout:3493: writing smithi15186934-45 from 682714 to 1353275 tid 1 2024-09-13T14:35:38.674 INFO:tasks.rados.rados.0.smithi151.stdout:3493: writing smithi15186934-45 from 2123320 to 2628777 tid 2 2024-09-13T14:35:38.680 INFO:tasks.rados.rados.0.smithi151.stdout:3493: writing smithi15186934-45 from 3088266 to 3725015 tid 3 2024-09-13T14:35:38.680 INFO:tasks.rados.rados.0.smithi151.stdout:3494: delete oid 20 current snap is 445 2024-09-13T14:35:38.680 INFO:tasks.rados.rados.0.smithi151.stdout:3483: expect (ObjNum 1280 snap 437 seq_num 1280) 2024-09-13T14:35:38.796 INFO:tasks.rados.rados.0.smithi151.stdout:3484: finishing write tid 1 to smithi15186934-36 2024-09-13T14:35:38.796 INFO:tasks.rados.rados.0.smithi151.stdout:3484: finishing write tid 2 to smithi15186934-36 2024-09-13T14:35:38.797 INFO:tasks.rados.rados.0.smithi151.stdout:3484: finishing write tid 3 to smithi15186934-36 2024-09-13T14:35:38.797 INFO:tasks.rados.rados.0.smithi151.stdout:3484: finishing write tid 4 to smithi15186934-36 2024-09-13T14:35:38.797 INFO:tasks.rados.rados.0.smithi151.stdout:3484: finishing write tid 5 to smithi15186934-36 2024-09-13T14:35:38.797 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 36 v 730 (ObjNum 1299 snap 445 seq_num 1299) dirty exists 2024-09-13T14:35:38.797 INFO:tasks.rados.rados.0.smithi151.stdout:3484: left oid 36 (ObjNum 1299 snap 445 seq_num 1299) 2024-09-13T14:35:38.797 INFO:tasks.rados.rados.0.smithi151.stdout:3485: finishing write tid 1 to smithi15186934-1 2024-09-13T14:35:38.797 INFO:tasks.rados.rados.0.smithi151.stdout:3485: finishing write tid 2 to smithi15186934-1 2024-09-13T14:35:38.797 INFO:tasks.rados.rados.0.smithi151.stdout:3485: finishing write tid 3 to smithi15186934-1 2024-09-13T14:35:38.797 INFO:tasks.rados.rados.0.smithi151.stdout:3483: done (10 left) 2024-09-13T14:35:38.797 INFO:tasks.rados.rados.0.smithi151.stdout:3484: done (9 left) 2024-09-13T14:35:38.797 INFO:tasks.rados.rados.0.smithi151.stdout:3494: done (8 left) 2024-09-13T14:35:38.797 INFO:tasks.rados.rados.0.smithi151.stdout:3495: read oid 29 snap -1 2024-09-13T14:35:38.798 INFO:tasks.rados.rados.0.smithi151.stdout:3495: expect deleted 2024-09-13T14:35:38.798 INFO:tasks.rados.rados.0.smithi151.stdout:3496: write oid 12 current snap is 445 2024-09-13T14:35:38.798 INFO:tasks.rados.rados.0.smithi151.stdout:3485: finishing write tid 4 to smithi15186934-1 2024-09-13T14:35:38.798 INFO:tasks.rados.rados.0.smithi151.stdout:3496: seq_num 1304 ranges {716328=421059,1840284=509126,3143112=659293} 2024-09-13T14:35:38.802 INFO:tasks.rados.rados.0.smithi151.stdout:3496: writing smithi15186934-12 from 716328 to 1137387 tid 1 2024-09-13T14:35:38.808 INFO:tasks.rados.rados.0.smithi151.stdout:3496: writing smithi15186934-12 from 1840284 to 2349410 tid 2 2024-09-13T14:35:38.816 INFO:tasks.rados.rados.0.smithi151.stdout:3496: writing smithi15186934-12 from 3143112 to 3802405 tid 3 2024-09-13T14:35:38.817 INFO:tasks.rados.rados.0.smithi151.stdout:3497: write oid 32 current snap is 445 2024-09-13T14:35:38.817 INFO:tasks.rados.rados.0.smithi151.stdout:3485: finishing write tid 5 to smithi15186934-1 2024-09-13T14:35:38.817 INFO:tasks.rados.rados.0.smithi151.stdout:3485: finishing write tid 6 to smithi15186934-1 2024-09-13T14:35:38.817 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 1 v 1094 (ObjNum 1300 snap 445 seq_num 1300) dirty exists 2024-09-13T14:35:38.817 INFO:tasks.rados.rados.0.smithi151.stdout:3485: left oid 1 (ObjNum 1300 snap 445 seq_num 1300) 2024-09-13T14:35:38.817 INFO:tasks.rados.rados.0.smithi151.stdout:3486: finishing write tid 1 to smithi15186934-9 2024-09-13T14:35:38.817 INFO:tasks.rados.rados.0.smithi151.stdout:3497: seq_num 1305 ranges {733135=496308,1898766=652784,3170535=470565} 2024-09-13T14:35:38.823 INFO:tasks.rados.rados.0.smithi151.stdout:3497: writing smithi15186934-32 from 733135 to 1229443 tid 1 2024-09-13T14:35:38.830 INFO:tasks.rados.rados.0.smithi151.stdout:3497: writing smithi15186934-32 from 1898766 to 2551550 tid 2 2024-09-13T14:35:38.836 INFO:tasks.rados.rados.0.smithi151.stdout:3497: writing smithi15186934-32 from 3170535 to 3641100 tid 3 2024-09-13T14:35:38.836 INFO:tasks.rados.rados.0.smithi151.stdout:3485: done (10 left) 2024-09-13T14:35:38.836 INFO:tasks.rados.rados.0.smithi151.stdout:3498: write oid 10 current snap is 445 2024-09-13T14:35:38.837 INFO:tasks.rados.rados.0.smithi151.stdout:3486: finishing write tid 2 to smithi15186934-9 2024-09-13T14:35:38.837 INFO:tasks.rados.rados.0.smithi151.stdout:3486: finishing write tid 3 to smithi15186934-9 2024-09-13T14:35:38.837 INFO:tasks.rados.rados.0.smithi151.stdout:3486: finishing write tid 4 to smithi15186934-9 2024-09-13T14:35:38.837 INFO:tasks.rados.rados.0.smithi151.stdout:3486: finishing write tid 5 to smithi15186934-9 2024-09-13T14:35:38.837 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 1417 (ObjNum 1301 snap 445 seq_num 1301) dirty exists 2024-09-13T14:35:38.837 INFO:tasks.rados.rados.0.smithi151.stdout:3486: left oid 9 (ObjNum 1301 snap 445 seq_num 1301) 2024-09-13T14:35:38.837 INFO:tasks.rados.rados.0.smithi151.stdout:3498: seq_num 1306 ranges {749942=571557,1840895=796442,3365252=584690} 2024-09-13T14:35:38.843 INFO:tasks.rados.rados.0.smithi151.stdout:3498: writing smithi15186934-10 from 749942 to 1321499 tid 1 2024-09-13T14:35:38.850 INFO:tasks.rados.rados.0.smithi151.stdout:3498: writing smithi15186934-10 from 1840895 to 2637337 tid 2 2024-09-13T14:35:38.858 INFO:tasks.rados.rados.0.smithi151.stdout:3498: writing smithi15186934-10 from 3365252 to 3949942 tid 3 2024-09-13T14:35:38.858 INFO:tasks.rados.rados.0.smithi151.stdout:3486: done (10 left) 2024-09-13T14:35:38.858 INFO:tasks.rados.rados.0.smithi151.stdout:3499: read oid 25 snap -1 2024-09-13T14:35:38.858 INFO:tasks.rados.rados.0.smithi151.stdout:3499: expect (ObjNum 1286 snap 439 seq_num 1286) 2024-09-13T14:35:38.858 INFO:tasks.rados.rados.0.smithi151.stdout:3500: snap_create 2024-09-13T14:35:38.858 INFO:tasks.rados.rados.0.smithi151.stdout:3491: finishing rollback tid 0 to smithi15186934-49 2024-09-13T14:35:38.858 INFO:tasks.rados.rados.0.smithi151.stdout:3491: finishing rollback tid 1 to smithi15186934-49 2024-09-13T14:35:38.859 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 49 v 1631 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-13T14:35:38.859 INFO:tasks.rados.rados.0.smithi151.stdout:3489: expect (ObjNum 1135 snap 381 seq_num 1135) 2024-09-13T14:35:38.994 INFO:tasks.rados.rados.0.smithi151.stdout:3493: finishing write tid 1 to smithi15186934-45 2024-09-13T14:35:38.994 INFO:tasks.rados.rados.0.smithi151.stdout:3488: expect (ObjNum 1231 snap 414 seq_num 1231) 2024-09-13T14:35:39.083 INFO:tasks.rados.rados.0.smithi151.stdout:3493: finishing write tid 2 to smithi15186934-45 2024-09-13T14:35:39.084 INFO:tasks.rados.rados.0.smithi151.stdout:3493: finishing write tid 3 to smithi15186934-45 2024-09-13T14:35:39.084 INFO:tasks.rados.rados.0.smithi151.stdout:3493: finishing write tid 4 to smithi15186934-45 2024-09-13T14:35:39.084 INFO:tasks.rados.rados.0.smithi151.stdout:3493: finishing write tid 5 to smithi15186934-45 2024-09-13T14:35:39.084 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 1706 (ObjNum 1302 snap 445 seq_num 1302) dirty exists 2024-09-13T14:35:39.084 INFO:tasks.rados.rados.0.smithi151.stdout:3493: left oid 45 (ObjNum 1302 snap 445 seq_num 1302) 2024-09-13T14:35:39.084 INFO:tasks.rados.rados.0.smithi151.stdout:3496: finishing write tid 1 to smithi15186934-12 2024-09-13T14:35:39.084 INFO:tasks.rados.rados.0.smithi151.stdout:3496: finishing write tid 2 to smithi15186934-12 2024-09-13T14:35:39.084 INFO:tasks.rados.rados.0.smithi151.stdout:3496: finishing write tid 3 to smithi15186934-12 2024-09-13T14:35:39.084 INFO:tasks.rados.rados.0.smithi151.stdout:3496: finishing write tid 4 to smithi15186934-12 2024-09-13T14:35:39.084 INFO:tasks.rados.rados.0.smithi151.stdout:3496: finishing write tid 5 to smithi15186934-12 2024-09-13T14:35:39.085 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 12 v 735 (ObjNum 1304 snap 445 seq_num 1304) dirty exists 2024-09-13T14:35:39.085 INFO:tasks.rados.rados.0.smithi151.stdout:3496: left oid 12 (ObjNum 1304 snap 445 seq_num 1304) 2024-09-13T14:35:39.085 INFO:tasks.rados.rados.0.smithi151.stdout:3497: finishing write tid 1 to smithi15186934-32 2024-09-13T14:35:39.085 INFO:tasks.rados.rados.0.smithi151.stdout:3497: finishing write tid 2 to smithi15186934-32 2024-09-13T14:35:39.085 INFO:tasks.rados.rados.0.smithi151.stdout:3497: finishing write tid 3 to smithi15186934-32 2024-09-13T14:35:39.085 INFO:tasks.rados.rados.0.smithi151.stdout:3497: finishing write tid 4 to smithi15186934-32 2024-09-13T14:35:39.085 INFO:tasks.rados.rados.0.smithi151.stdout:3497: finishing write tid 5 to smithi15186934-32 2024-09-13T14:35:39.085 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 32 v 1424 (ObjNum 1305 snap 445 seq_num 1305) dirty exists 2024-09-13T14:35:39.085 INFO:tasks.rados.rados.0.smithi151.stdout:3497: left oid 32 (ObjNum 1305 snap 445 seq_num 1305) 2024-09-13T14:35:39.085 INFO:tasks.rados.rados.0.smithi151.stdout:3498: finishing write tid 1 to smithi15186934-10 2024-09-13T14:35:39.085 INFO:tasks.rados.rados.0.smithi151.stdout:3498: finishing write tid 2 to smithi15186934-10 2024-09-13T14:35:39.085 INFO:tasks.rados.rados.0.smithi151.stdout:3498: finishing write tid 3 to smithi15186934-10 2024-09-13T14:35:39.086 INFO:tasks.rados.rados.0.smithi151.stdout:3498: finishing write tid 4 to smithi15186934-10 2024-09-13T14:35:39.086 INFO:tasks.rados.rados.0.smithi151.stdout:3498: finishing write tid 5 to smithi15186934-10 2024-09-13T14:35:39.086 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 1268 (ObjNum 1306 snap 445 seq_num 1306) dirty exists 2024-09-13T14:35:39.086 INFO:tasks.rados.rados.0.smithi151.stdout:3498: left oid 10 (ObjNum 1306 snap 445 seq_num 1306) 2024-09-13T14:35:39.086 INFO:tasks.rados.rados.0.smithi151.stdout:3499: expect (ObjNum 1286 snap 439 seq_num 1286) 2024-09-13T14:35:39.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:38 smithi184 ceph-mon[92527]: pgmap v1038: 89 pgs: 89 active+clean; 194 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 1.3 MiB/s wr, 5 op/s 2024-09-13T14:35:39.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:38 smithi184 ceph-mon[92527]: osdmap e947: 8 total, 8 up, 8 in 2024-09-13T14:35:39.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:38 smithi151 ceph-mon[104450]: pgmap v1038: 89 pgs: 89 active+clean; 194 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 1.3 MiB/s wr, 5 op/s 2024-09-13T14:35:39.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:38 smithi151 ceph-mon[104450]: osdmap e947: 8 total, 8 up, 8 in 2024-09-13T14:35:39.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:38 smithi151 ceph-mon[108966]: pgmap v1038: 89 pgs: 89 active+clean; 194 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 1.3 MiB/s wr, 5 op/s 2024-09-13T14:35:39.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:38 smithi151 ceph-mon[108966]: osdmap e947: 8 total, 8 up, 8 in 2024-09-13T14:35:39.640 INFO:tasks.rados.rados.0.smithi151.stdout:3488: done (11 left) 2024-09-13T14:35:39.640 INFO:tasks.rados.rados.0.smithi151.stdout:3489: done (10 left) 2024-09-13T14:35:39.640 INFO:tasks.rados.rados.0.smithi151.stdout:3490: done (9 left) 2024-09-13T14:35:39.640 INFO:tasks.rados.rados.0.smithi151.stdout:3491: done (8 left) 2024-09-13T14:35:39.640 INFO:tasks.rados.rados.0.smithi151.stdout:3492: done (7 left) 2024-09-13T14:35:39.640 INFO:tasks.rados.rados.0.smithi151.stdout:3493: done (6 left) 2024-09-13T14:35:39.640 INFO:tasks.rados.rados.0.smithi151.stdout:3495: done (5 left) 2024-09-13T14:35:39.641 INFO:tasks.rados.rados.0.smithi151.stdout:3496: done (4 left) 2024-09-13T14:35:39.641 INFO:tasks.rados.rados.0.smithi151.stdout:3497: done (3 left) 2024-09-13T14:35:39.641 INFO:tasks.rados.rados.0.smithi151.stdout:3498: done (2 left) 2024-09-13T14:35:39.641 INFO:tasks.rados.rados.0.smithi151.stdout:3499: done (1 left) 2024-09-13T14:35:39.641 INFO:tasks.rados.rados.0.smithi151.stdout:3500: done (0 left) 2024-09-13T14:35:39.641 INFO:tasks.rados.rados.0.smithi151.stdout:3501: write oid 42 current snap is 446 2024-09-13T14:35:39.641 INFO:tasks.rados.rados.0.smithi151.stdout:3501: seq_num 1307 ranges {766749=646806,1899377=656453,3109028=609462} 2024-09-13T14:35:39.647 INFO:tasks.rados.rados.0.smithi151.stdout:3501: writing smithi15186934-42 from 766749 to 1413555 tid 1 2024-09-13T14:35:39.654 INFO:tasks.rados.rados.0.smithi151.stdout:3501: writing smithi15186934-42 from 1899377 to 2555830 tid 2 2024-09-13T14:35:39.662 INFO:tasks.rados.rados.0.smithi151.stdout:3501: writing smithi15186934-42 from 3109028 to 3718490 tid 3 2024-09-13T14:35:39.662 INFO:tasks.rados.rados.0.smithi151.stdout:3502: read oid 26 snap -1 2024-09-13T14:35:39.662 INFO:tasks.rados.rados.0.smithi151.stdout:3502: expect deleted 2024-09-13T14:35:39.662 INFO:tasks.rados.rados.0.smithi151.stdout:3503: write oid 2 current snap is 446 2024-09-13T14:35:39.662 INFO:tasks.rados.rados.0.smithi151.stdout:3501: finishing write tid 1 to smithi15186934-42 2024-09-13T14:35:39.662 INFO:tasks.rados.rados.0.smithi151.stdout:3501: finishing write tid 2 to smithi15186934-42 2024-09-13T14:35:39.663 INFO:tasks.rados.rados.0.smithi151.stdout:3503: seq_num 1308 ranges {783556=438408,1674212=400111,2736451=420734,3882270=101286} 2024-09-13T14:35:39.669 INFO:tasks.rados.rados.0.smithi151.stdout:3503: writing smithi15186934-2 from 783556 to 1221964 tid 1 2024-09-13T14:35:39.674 INFO:tasks.rados.rados.0.smithi151.stdout:3503: writing smithi15186934-2 from 1674212 to 2074323 tid 2 2024-09-13T14:35:39.681 INFO:tasks.rados.rados.0.smithi151.stdout:3503: writing smithi15186934-2 from 2736451 to 3157185 tid 3 2024-09-13T14:35:39.686 INFO:tasks.rados.rados.0.smithi151.stdout:3503: writing smithi15186934-2 from 3882270 to 3983556 tid 4 2024-09-13T14:35:39.686 INFO:tasks.rados.rados.0.smithi151.stdout:3504: write oid 36 current snap is 446 2024-09-13T14:35:39.686 INFO:tasks.rados.rados.0.smithi151.stdout:3504: seq_num 1309 ranges {400363=513657,1332694=660122} 2024-09-13T14:35:39.691 INFO:tasks.rados.rados.0.smithi151.stdout:3504: writing smithi15186934-36 from 400363 to 914020 tid 1 2024-09-13T14:35:39.698 INFO:tasks.rados.rados.0.smithi151.stdout:3504: writing smithi15186934-36 from 1332694 to 1992816 tid 2 2024-09-13T14:35:39.698 INFO:tasks.rados.rados.0.smithi151.stdout:3505: read oid 14 snap -1 2024-09-13T14:35:39.698 INFO:tasks.rados.rados.0.smithi151.stdout:3505: expect (ObjNum 1169 snap 396 seq_num 1169) 2024-09-13T14:35:39.698 INFO:tasks.rados.rados.0.smithi151.stdout:3506: rollback oid 19 current snap is 446 2024-09-13T14:35:39.698 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 19 to 440 2024-09-13T14:35:39.698 INFO:tasks.rados.rados.0.smithi151.stdout:3501: finishing write tid 3 to smithi15186934-42 2024-09-13T14:35:39.698 INFO:tasks.rados.rados.0.smithi151.stdout:3501: finishing write tid 4 to smithi15186934-42 2024-09-13T14:35:39.698 INFO:tasks.rados.rados.0.smithi151.stdout:3501: finishing write tid 5 to smithi15186934-42 2024-09-13T14:35:39.698 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 1324 (ObjNum 1307 snap 446 seq_num 1307) dirty exists 2024-09-13T14:35:39.698 INFO:tasks.rados.rados.0.smithi151.stdout:3501: left oid 42 (ObjNum 1307 snap 446 seq_num 1307) 2024-09-13T14:35:39.698 INFO:tasks.rados.rados.0.smithi151.stdout:3503: finishing write tid 1 to smithi15186934-2 2024-09-13T14:35:39.699 INFO:tasks.rados.rados.0.smithi151.stdout:3503: finishing write tid 2 to smithi15186934-2 2024-09-13T14:35:39.699 INFO:tasks.rados.rados.0.smithi151.stdout:3503: finishing write tid 3 to smithi15186934-2 2024-09-13T14:35:39.699 INFO:tasks.rados.rados.0.smithi151.stdout:3501: done (5 left) 2024-09-13T14:35:39.699 INFO:tasks.rados.rados.0.smithi151.stdout:3502: done (4 left) 2024-09-13T14:35:39.699 INFO:tasks.rados.rados.0.smithi151.stdout:3507: read oid 31 snap -1 2024-09-13T14:35:39.699 INFO:tasks.rados.rados.0.smithi151.stdout:3507: expect deleted 2024-09-13T14:35:39.699 INFO:tasks.rados.rados.0.smithi151.stdout:3508: write oid 15 current snap is 446 2024-09-13T14:35:39.699 INFO:tasks.rados.rados.0.smithi151.stdout:3503: finishing write tid 4 to smithi15186934-2 2024-09-13T14:35:39.699 INFO:tasks.rados.rados.0.smithi151.stdout:3503: finishing write tid 5 to smithi15186934-2 2024-09-13T14:35:39.699 INFO:tasks.rados.rados.0.smithi151.stdout:3503: finishing write tid 6 to smithi15186934-2 2024-09-13T14:35:39.699 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 2 v 1355 (ObjNum 1308 snap 446 seq_num 1308) dirty exists 2024-09-13T14:35:39.699 INFO:tasks.rados.rados.0.smithi151.stdout:3503: left oid 2 (ObjNum 1308 snap 446 seq_num 1308) 2024-09-13T14:35:39.699 INFO:tasks.rados.rados.0.smithi151.stdout:3504: finishing write tid 1 to smithi15186934-36 2024-09-13T14:35:39.699 INFO:tasks.rados.rados.0.smithi151.stdout:3508: seq_num 1310 ranges {417170=588906,1674823=342347} 2024-09-13T14:35:39.703 INFO:tasks.rados.rados.0.smithi151.stdout:3508: writing smithi15186934-15 from 417170 to 1006076 tid 1 2024-09-13T14:35:39.709 INFO:tasks.rados.rados.0.smithi151.stdout:3508: writing smithi15186934-15 from 1674823 to 2017170 tid 2 2024-09-13T14:35:39.709 INFO:tasks.rados.rados.0.smithi151.stdout:3503: done (5 left) 2024-09-13T14:35:39.709 INFO:tasks.rados.rados.0.smithi151.stdout:3509: read oid 17 snap -1 2024-09-13T14:35:39.709 INFO:tasks.rados.rados.0.smithi151.stdout:3509: expect (ObjNum 1231 snap 414 seq_num 1231) 2024-09-13T14:35:39.709 INFO:tasks.rados.rados.0.smithi151.stdout:3510: snap_remove snap 443 2024-09-13T14:35:40.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:39 smithi184 ceph-mon[92527]: osdmap e948: 8 total, 8 up, 8 in 2024-09-13T14:35:40.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:39 smithi184 ceph-mon[92527]: osdmap e949: 8 total, 8 up, 8 in 2024-09-13T14:35:40.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:39 smithi151 ceph-mon[104450]: osdmap e948: 8 total, 8 up, 8 in 2024-09-13T14:35:40.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:39 smithi151 ceph-mon[104450]: osdmap e949: 8 total, 8 up, 8 in 2024-09-13T14:35:40.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:39 smithi151 ceph-mon[108966]: osdmap e948: 8 total, 8 up, 8 in 2024-09-13T14:35:40.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:39 smithi151 ceph-mon[108966]: osdmap e949: 8 total, 8 up, 8 in 2024-09-13T14:35:40.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:35:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:35:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:35:40.673 INFO:tasks.rados.rados.0.smithi151.stdout:3510: done (6 left) 2024-09-13T14:35:40.673 INFO:tasks.rados.rados.0.smithi151.stdout:3511: write oid 26 current snap is 446 2024-09-13T14:35:40.673 INFO:tasks.rados.rados.0.smithi151.stdout:3511: seq_num 1311 ranges {433977=664155,1733305=300672} 2024-09-13T14:35:40.680 INFO:tasks.rados.rados.0.smithi151.stdout:3511: writing smithi15186934-26 from 433977 to 1098132 tid 1 2024-09-13T14:35:40.685 INFO:tasks.rados.rados.0.smithi151.stdout:3511: writing smithi15186934-26 from 1733305 to 2033977 tid 2 2024-09-13T14:35:40.685 INFO:tasks.rados.rados.0.smithi151.stdout:3512: write oid 23 current snap is 446 2024-09-13T14:35:40.685 INFO:tasks.rados.rados.0.smithi151.stdout:3512: seq_num 1312 ranges {450784=739404,1791787=258997} 2024-09-13T14:35:40.691 INFO:tasks.rados.rados.0.smithi151.stdout:3512: writing smithi15186934-23 from 450784 to 1190188 tid 1 2024-09-13T14:35:40.697 INFO:tasks.rados.rados.0.smithi151.stdout:3512: writing smithi15186934-23 from 1791787 to 2050784 tid 2 2024-09-13T14:35:40.697 INFO:tasks.rados.rados.0.smithi151.stdout:3513: write oid 41 current snap is 446 2024-09-13T14:35:40.697 INFO:tasks.rados.rados.0.smithi151.stdout:3513: seq_num 1313 ranges {467591=414653,1450269=617322} 2024-09-13T14:35:40.701 INFO:tasks.rados.rados.0.smithi151.stdout:3513: writing smithi15186934-41 from 467591 to 882244 tid 1 2024-09-13T14:35:40.708 INFO:tasks.rados.rados.0.smithi151.stdout:3513: writing smithi15186934-41 from 1450269 to 2067591 tid 2 2024-09-13T14:35:40.708 INFO:tasks.rados.rados.0.smithi151.stdout:3514: write oid 13 current snap is 446 2024-09-13T14:35:40.708 INFO:tasks.rados.rados.0.smithi151.stdout:3514: seq_num 1314 ranges {484398=489902,1392398=411118} 2024-09-13T14:35:40.713 INFO:tasks.rados.rados.0.smithi151.stdout:3514: writing smithi15186934-13 from 484398 to 974300 tid 1 2024-09-13T14:35:40.718 INFO:tasks.rados.rados.0.smithi151.stdout:3514: writing smithi15186934-13 from 1392398 to 1803516 tid 2 2024-09-13T14:35:40.719 INFO:tasks.rados.rados.0.smithi151.stdout:3515: rollback oid 25 current snap is 446 2024-09-13T14:35:40.719 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 25 to 413 2024-09-13T14:35:40.719 INFO:tasks.rados.rados.0.smithi151.stdout:3504: finishing write tid 2 to smithi15186934-36 2024-09-13T14:35:40.719 INFO:tasks.rados.rados.0.smithi151.stdout:3504: finishing write tid 3 to smithi15186934-36 2024-09-13T14:35:40.719 INFO:tasks.rados.rados.0.smithi151.stdout:3504: finishing write tid 4 to smithi15186934-36 2024-09-13T14:35:40.719 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 36 v 739 (ObjNum 1309 snap 446 seq_num 1309) dirty exists 2024-09-13T14:35:40.719 INFO:tasks.rados.rados.0.smithi151.stdout:3504: left oid 36 (ObjNum 1309 snap 446 seq_num 1309) 2024-09-13T14:35:40.719 INFO:tasks.rados.rados.0.smithi151.stdout:3506: finishing rollback tid 0 to smithi15186934-19 2024-09-13T14:35:40.719 INFO:tasks.rados.rados.0.smithi151.stdout:3506: finishing rollback tid 1 to smithi15186934-19 2024-09-13T14:35:40.719 INFO:tasks.rados.rados.0.smithi151.stdout:3506: finishing rollback tid 2 to smithi15186934-19 2024-09-13T14:35:40.719 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 19 v 779 (ObjNum 1265 snap 429 seq_num 1265) dirty exists 2024-09-13T14:35:40.719 INFO:tasks.rados.rados.0.smithi151.stdout:3504: done (10 left) 2024-09-13T14:35:40.719 INFO:tasks.rados.rados.0.smithi151.stdout:3506: done (9 left) 2024-09-13T14:35:40.720 INFO:tasks.rados.rados.0.smithi151.stdout:3507: done (8 left) 2024-09-13T14:35:40.720 INFO:tasks.rados.rados.0.smithi151.stdout:3516: write oid 20 current snap is 446 2024-09-13T14:35:40.720 INFO:tasks.rados.rados.0.smithi151.stdout:3516: seq_num 1315 ranges {501205=565151,1850880=250325} 2024-09-13T14:35:40.724 INFO:tasks.rados.rados.0.smithi151.stdout:3516: writing smithi15186934-20 from 501205 to 1066356 tid 1 2024-09-13T14:35:40.730 INFO:tasks.rados.rados.0.smithi151.stdout:3516: writing smithi15186934-20 from 1850880 to 2101205 tid 2 2024-09-13T14:35:40.730 INFO:tasks.rados.rados.0.smithi151.stdout:3517: rollback oid 43 current snap is 446 2024-09-13T14:35:40.730 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 43 to 437 2024-09-13T14:35:40.730 INFO:tasks.rados.rados.0.smithi151.stdout:3505: expect (ObjNum 1169 snap 396 seq_num 1169) 2024-09-13T14:35:40.954 INFO:tasks.rados.rados.0.smithi151.stdout:3508: finishing write tid 1 to smithi15186934-15 2024-09-13T14:35:40.955 INFO:tasks.rados.rados.0.smithi151.stdout:3508: finishing write tid 2 to smithi15186934-15 2024-09-13T14:35:40.955 INFO:tasks.rados.rados.0.smithi151.stdout:3508: finishing write tid 3 to smithi15186934-15 2024-09-13T14:35:40.955 INFO:tasks.rados.rados.0.smithi151.stdout:3508: finishing write tid 4 to smithi15186934-15 2024-09-13T14:35:40.955 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 15 v 1327 (ObjNum 1310 snap 446 seq_num 1310) dirty exists 2024-09-13T14:35:40.955 INFO:tasks.rados.rados.0.smithi151.stdout:3508: left oid 15 (ObjNum 1310 snap 446 seq_num 1310) 2024-09-13T14:35:40.955 INFO:tasks.rados.rados.0.smithi151.stdout:3505: done (9 left) 2024-09-13T14:35:40.955 INFO:tasks.rados.rados.0.smithi151.stdout:3508: done (8 left) 2024-09-13T14:35:40.955 INFO:tasks.rados.rados.0.smithi151.stdout:3518: read oid 6 snap -1 2024-09-13T14:35:40.955 INFO:tasks.rados.rados.0.smithi151.stdout:3518: expect (ObjNum 1280 snap 437 seq_num 1280) 2024-09-13T14:35:40.955 INFO:tasks.rados.rados.0.smithi151.stdout:3519: snap_create 2024-09-13T14:35:40.955 INFO:tasks.rados.rados.0.smithi151.stdout:3509: expect (ObjNum 1231 snap 414 seq_num 1231) 2024-09-13T14:35:41.044 INFO:tasks.rados.rados.0.smithi151.stdout:3511: finishing write tid 1 to smithi15186934-26 2024-09-13T14:35:41.044 INFO:tasks.rados.rados.0.smithi151.stdout:3511: finishing write tid 2 to smithi15186934-26 2024-09-13T14:35:41.044 INFO:tasks.rados.rados.0.smithi151.stdout:3511: finishing write tid 3 to smithi15186934-26 2024-09-13T14:35:41.044 INFO:tasks.rados.rados.0.smithi151.stdout:3511: finishing write tid 4 to smithi15186934-26 2024-09-13T14:35:41.044 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 1273 (ObjNum 1311 snap 446 seq_num 1311) dirty exists 2024-09-13T14:35:41.045 INFO:tasks.rados.rados.0.smithi151.stdout:3511: left oid 26 (ObjNum 1311 snap 446 seq_num 1311) 2024-09-13T14:35:41.045 INFO:tasks.rados.rados.0.smithi151.stdout:3512: finishing write tid 1 to smithi15186934-23 2024-09-13T14:35:41.045 INFO:tasks.rados.rados.0.smithi151.stdout:3512: finishing write tid 2 to smithi15186934-23 2024-09-13T14:35:41.045 INFO:tasks.rados.rados.0.smithi151.stdout:3512: finishing write tid 3 to smithi15186934-23 2024-09-13T14:35:41.045 INFO:tasks.rados.rados.0.smithi151.stdout:3512: finishing write tid 4 to smithi15186934-23 2024-09-13T14:35:41.045 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 23 v 1637 (ObjNum 1312 snap 446 seq_num 1312) dirty exists 2024-09-13T14:35:41.045 INFO:tasks.rados.rados.0.smithi151.stdout:3512: left oid 23 (ObjNum 1312 snap 446 seq_num 1312) 2024-09-13T14:35:41.045 INFO:tasks.rados.rados.0.smithi151.stdout:3513: finishing write tid 1 to smithi15186934-41 2024-09-13T14:35:41.045 INFO:tasks.rados.rados.0.smithi151.stdout:3513: finishing write tid 2 to smithi15186934-41 2024-09-13T14:35:41.045 INFO:tasks.rados.rados.0.smithi151.stdout:3513: finishing write tid 3 to smithi15186934-41 2024-09-13T14:35:41.045 INFO:tasks.rados.rados.0.smithi151.stdout:3513: finishing write tid 4 to smithi15186934-41 2024-09-13T14:35:41.045 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 298 (ObjNum 1313 snap 446 seq_num 1313) dirty exists 2024-09-13T14:35:41.045 INFO:tasks.rados.rados.0.smithi151.stdout:3513: left oid 41 (ObjNum 1313 snap 446 seq_num 1313) 2024-09-13T14:35:41.045 INFO:tasks.rados.rados.0.smithi151.stdout:3514: finishing write tid 1 to smithi15186934-13 2024-09-13T14:35:41.046 INFO:tasks.rados.rados.0.smithi151.stdout:3514: finishing write tid 2 to smithi15186934-13 2024-09-13T14:35:41.046 INFO:tasks.rados.rados.0.smithi151.stdout:3514: finishing write tid 3 to smithi15186934-13 2024-09-13T14:35:41.046 INFO:tasks.rados.rados.0.smithi151.stdout:3515: finishing rollback tid 0 to smithi15186934-25 2024-09-13T14:35:41.046 INFO:tasks.rados.rados.0.smithi151.stdout:3515: finishing rollback tid 1 to smithi15186934-25 2024-09-13T14:35:41.046 INFO:tasks.rados.rados.0.smithi151.stdout:3514: finishing write tid 4 to smithi15186934-13 2024-09-13T14:35:41.046 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 13 v 1361 (ObjNum 1314 snap 446 seq_num 1314) dirty exists 2024-09-13T14:35:41.046 INFO:tasks.rados.rados.0.smithi151.stdout:3514: left oid 13 (ObjNum 1314 snap 446 seq_num 1314) 2024-09-13T14:35:41.046 INFO:tasks.rados.rados.0.smithi151.stdout:3515: finishing rollback tid 2 to smithi15186934-25 2024-09-13T14:35:41.046 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 1712 (ObjNum 1209 snap 407 seq_num 1209) dirty exists 2024-09-13T14:35:41.046 INFO:tasks.rados.rados.0.smithi151.stdout:3516: finishing write tid 1 to smithi15186934-20 2024-09-13T14:35:41.046 INFO:tasks.rados.rados.0.smithi151.stdout:3516: finishing write tid 2 to smithi15186934-20 2024-09-13T14:35:41.046 INFO:tasks.rados.rados.0.smithi151.stdout:3516: finishing write tid 3 to smithi15186934-20 2024-09-13T14:35:41.046 INFO:tasks.rados.rados.0.smithi151.stdout:3517: finishing rollback tid 1 to smithi15186934-43 2024-09-13T14:35:41.046 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 43 v 0 (ObjNum 33 snap 0 seq_num 3093156512) dirty dne 2024-09-13T14:35:41.046 INFO:tasks.rados.rados.0.smithi151.stdout:3516: finishing write tid 4 to smithi15186934-20 2024-09-13T14:35:41.047 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 1002 (ObjNum 1315 snap 446 seq_num 1315) dirty exists 2024-09-13T14:35:41.047 INFO:tasks.rados.rados.0.smithi151.stdout:3516: left oid 20 (ObjNum 1315 snap 446 seq_num 1315) 2024-09-13T14:35:41.047 INFO:tasks.rados.rados.0.smithi151.stdout:3518: expect (ObjNum 1280 snap 437 seq_num 1280) 2024-09-13T14:35:41.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:40 smithi184 ceph-mon[92527]: pgmap v1041: 89 pgs: 89 active+clean; 192 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.8 MiB/s wr, 6 op/s 2024-09-13T14:35:41.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:40 smithi151 ceph-mon[104450]: pgmap v1041: 89 pgs: 89 active+clean; 192 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.8 MiB/s wr, 6 op/s 2024-09-13T14:35:41.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:40 smithi151 ceph-mon[108966]: pgmap v1041: 89 pgs: 89 active+clean; 192 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.8 MiB/s wr, 6 op/s 2024-09-13T14:35:41.680 INFO:tasks.rados.rados.0.smithi151.stdout:3509: done (9 left) 2024-09-13T14:35:41.681 INFO:tasks.rados.rados.0.smithi151.stdout:3511: done (8 left) 2024-09-13T14:35:41.681 INFO:tasks.rados.rados.0.smithi151.stdout:3512: done (7 left) 2024-09-13T14:35:41.681 INFO:tasks.rados.rados.0.smithi151.stdout:3513: done (6 left) 2024-09-13T14:35:41.681 INFO:tasks.rados.rados.0.smithi151.stdout:3514: done (5 left) 2024-09-13T14:35:41.681 INFO:tasks.rados.rados.0.smithi151.stdout:3515: done (4 left) 2024-09-13T14:35:41.681 INFO:tasks.rados.rados.0.smithi151.stdout:3516: done (3 left) 2024-09-13T14:35:41.681 INFO:tasks.rados.rados.0.smithi151.stdout:3517: done (2 left) 2024-09-13T14:35:41.681 INFO:tasks.rados.rados.0.smithi151.stdout:3518: done (1 left) 2024-09-13T14:35:41.681 INFO:tasks.rados.rados.0.smithi151.stdout:3519: done (0 left) 2024-09-13T14:35:41.681 INFO:tasks.rados.rados.0.smithi151.stdout:3520: rollback oid 36 current snap is 447 2024-09-13T14:35:41.681 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 36 to 444 2024-09-13T14:35:41.681 INFO:tasks.rados.rados.0.smithi151.stdout:3521: rollback oid 29 current snap is 447 2024-09-13T14:35:41.682 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 29 to 437 2024-09-13T14:35:41.682 INFO:tasks.rados.rados.0.smithi151.stdout:3522: snap_create 2024-09-13T14:35:41.684 INFO:tasks.rados.rados.0.smithi151.stdout:3520: finishing rollback tid 0 to smithi15186934-36 2024-09-13T14:35:41.684 INFO:tasks.rados.rados.0.smithi151.stdout:3521: finishing rollback tid 1 to smithi15186934-29 2024-09-13T14:35:41.684 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 29 v 867 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-13T14:35:41.685 INFO:tasks.rados.rados.0.smithi151.stdout:3520: finishing rollback tid 1 to smithi15186934-36 2024-09-13T14:35:41.685 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 36 v 744 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-13T14:35:42.005 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:41 smithi184 ceph-mon[92527]: osdmap e950: 8 total, 8 up, 8 in 2024-09-13T14:35:42.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:41 smithi151 ceph-mon[108966]: osdmap e950: 8 total, 8 up, 8 in 2024-09-13T14:35:42.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:41 smithi151 ceph-mon[104450]: osdmap e950: 8 total, 8 up, 8 in 2024-09-13T14:35:42.681 INFO:tasks.rados.rados.0.smithi151.stdout:3520: done (2 left) 2024-09-13T14:35:42.681 INFO:tasks.rados.rados.0.smithi151.stdout:3521: done (1 left) 2024-09-13T14:35:42.681 INFO:tasks.rados.rados.0.smithi151.stdout:3522: done (0 left) 2024-09-13T14:35:42.681 INFO:tasks.rados.rados.0.smithi151.stdout:3523: snap_create 2024-09-13T14:35:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:42 smithi184 ceph-mon[92527]: pgmap v1044: 89 pgs: 89 active+clean; 209 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.2 MiB/s wr, 13 op/s 2024-09-13T14:35:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:42 smithi184 ceph-mon[92527]: osdmap e951: 8 total, 8 up, 8 in 2024-09-13T14:35:43.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:42 smithi151 ceph-mon[108966]: pgmap v1044: 89 pgs: 89 active+clean; 209 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.2 MiB/s wr, 13 op/s 2024-09-13T14:35:43.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:42 smithi151 ceph-mon[108966]: osdmap e951: 8 total, 8 up, 8 in 2024-09-13T14:35:43.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:42 smithi151 ceph-mon[104450]: pgmap v1044: 89 pgs: 89 active+clean; 209 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.2 MiB/s wr, 13 op/s 2024-09-13T14:35:43.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:42 smithi151 ceph-mon[104450]: osdmap e951: 8 total, 8 up, 8 in 2024-09-13T14:35:43.683 INFO:tasks.rados.rados.0.smithi151.stdout:3523: done (0 left) 2024-09-13T14:35:43.683 INFO:tasks.rados.rados.0.smithi151.stdout:3524: delete oid 2 current snap is 449 2024-09-13T14:35:43.689 INFO:tasks.rados.rados.0.smithi151.stdout:3524: done (0 left) 2024-09-13T14:35:43.690 INFO:tasks.rados.rados.0.smithi151.stdout:3525: rollback oid 20 current snap is 449 2024-09-13T14:35:43.690 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 20 to 428 2024-09-13T14:35:43.690 INFO:tasks.rados.rados.0.smithi151.stdout:3526: write oid 39 current snap is 449 2024-09-13T14:35:43.690 INFO:tasks.rados.rados.0.smithi151.stdout:3526: seq_num 1317 ranges {534819=432002,1684197=450622} 2024-09-13T14:35:43.695 INFO:tasks.rados.rados.0.smithi151.stdout:3526: writing smithi15186934-39 from 534819 to 966821 tid 1 2024-09-13T14:35:43.701 INFO:tasks.rados.rados.0.smithi151.stdout:3526: writing smithi15186934-39 from 1684197 to 2134819 tid 2 2024-09-13T14:35:43.702 INFO:tasks.rados.rados.0.smithi151.stdout:3527: snap_remove snap 446 2024-09-13T14:35:44.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:43 smithi184 ceph-mon[92527]: osdmap e952: 8 total, 8 up, 8 in 2024-09-13T14:35:44.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:43 smithi151 ceph-mon[104450]: osdmap e952: 8 total, 8 up, 8 in 2024-09-13T14:35:44.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:43 smithi151 ceph-mon[108966]: osdmap e952: 8 total, 8 up, 8 in 2024-09-13T14:35:44.643 INFO:tasks.rados.rados.0.smithi151.stdout:3525: finishing rollback tid 0 to smithi15186934-20 2024-09-13T14:35:44.643 INFO:tasks.rados.rados.0.smithi151.stdout:3525: finishing rollback tid 1 to smithi15186934-20 2024-09-13T14:35:44.644 INFO:tasks.rados.rados.0.smithi151.stdout:3525: finishing rollback tid 2 to smithi15186934-20 2024-09-13T14:35:44.644 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 1006 (ObjNum 997 snap 344 seq_num 997) dirty exists 2024-09-13T14:35:44.644 INFO:tasks.rados.rados.0.smithi151.stdout:3526: finishing write tid 1 to smithi15186934-39 2024-09-13T14:35:44.644 INFO:tasks.rados.rados.0.smithi151.stdout:3526: finishing write tid 2 to smithi15186934-39 2024-09-13T14:35:44.644 INFO:tasks.rados.rados.0.smithi151.stdout:3526: finishing write tid 3 to smithi15186934-39 2024-09-13T14:35:44.644 INFO:tasks.rados.rados.0.smithi151.stdout:3526: finishing write tid 4 to smithi15186934-39 2024-09-13T14:35:44.644 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 1485 (ObjNum 1317 snap 449 seq_num 1317) dirty exists 2024-09-13T14:35:44.644 INFO:tasks.rados.rados.0.smithi151.stdout:3526: left oid 39 (ObjNum 1317 snap 449 seq_num 1317) 2024-09-13T14:35:44.644 INFO:tasks.rados.rados.0.smithi151.stdout:3525: done (2 left) 2024-09-13T14:35:44.644 INFO:tasks.rados.rados.0.smithi151.stdout:3526: done (1 left) 2024-09-13T14:35:44.644 INFO:tasks.rados.rados.0.smithi151.stdout:3527: done (0 left) 2024-09-13T14:35:44.644 INFO:tasks.rados.rados.0.smithi151.stdout:3528: write oid 21 current snap is 449 2024-09-13T14:35:44.644 INFO:tasks.rados.rados.0.smithi151.stdout:3528: seq_num 1318 ranges {551626=507251,1626326=418456} 2024-09-13T14:35:44.649 INFO:tasks.rados.rados.0.smithi151.stdout:3528: writing smithi15186934-21 from 551626 to 1058877 tid 1 2024-09-13T14:35:44.654 INFO:tasks.rados.rados.0.smithi151.stdout:3528: writing smithi15186934-21 from 1626326 to 2044782 tid 2 2024-09-13T14:35:44.654 INFO:tasks.rados.rados.0.smithi151.stdout:3529: rollback oid 44 current snap is 449 2024-09-13T14:35:44.655 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 44 to 438 2024-09-13T14:35:44.655 INFO:tasks.rados.rados.0.smithi151.stdout:3530: snap_create 2024-09-13T14:35:44.656 INFO:tasks.rados.rados.0.smithi151.stdout:3528: finishing write tid 1 to smithi15186934-21 2024-09-13T14:35:44.659 INFO:tasks.rados.rados.0.smithi151.stdout:3528: finishing write tid 2 to smithi15186934-21 2024-09-13T14:35:44.659 INFO:tasks.rados.rados.0.smithi151.stdout:3528: finishing write tid 3 to smithi15186934-21 2024-09-13T14:35:44.659 INFO:tasks.rados.rados.0.smithi151.stdout:3528: finishing write tid 4 to smithi15186934-21 2024-09-13T14:35:44.659 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 21 v 717 (ObjNum 1318 snap 449 seq_num 1318) dirty exists 2024-09-13T14:35:44.659 INFO:tasks.rados.rados.0.smithi151.stdout:3528: left oid 21 (ObjNum 1318 snap 449 seq_num 1318) 2024-09-13T14:35:44.660 INFO:tasks.rados.rados.0.smithi151.stdout:3529: finishing rollback tid 0 to smithi15186934-44 2024-09-13T14:35:44.660 INFO:tasks.rados.rados.0.smithi151.stdout:3529: finishing rollback tid 1 to smithi15186934-44 2024-09-13T14:35:44.660 INFO:tasks.rados.rados.0.smithi151.stdout:3529: finishing rollback tid 2 to smithi15186934-44 2024-09-13T14:35:44.660 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 1102 (ObjNum 1226 snap 410 seq_num 1226) dirty exists 2024-09-13T14:35:44.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:44 smithi151 ceph-mon[104450]: pgmap v1047: 89 pgs: 89 active+clean; 218 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 27 MiB/s rd, 3.9 MiB/s wr, 35 op/s 2024-09-13T14:35:44.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:44 smithi151 ceph-mon[104450]: osdmap e953: 8 total, 8 up, 8 in 2024-09-13T14:35:44.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:44 smithi151 ceph-mon[104450]: osdmap e954: 8 total, 8 up, 8 in 2024-09-13T14:35:44.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:44 smithi151 ceph-mon[108966]: pgmap v1047: 89 pgs: 89 active+clean; 218 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 27 MiB/s rd, 3.9 MiB/s wr, 35 op/s 2024-09-13T14:35:44.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:44 smithi151 ceph-mon[108966]: osdmap e953: 8 total, 8 up, 8 in 2024-09-13T14:35:44.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:44 smithi151 ceph-mon[108966]: osdmap e954: 8 total, 8 up, 8 in 2024-09-13T14:35:44.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:35:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:35:44] "GET /metrics HTTP/1.1" 200 36508 "" "Prometheus/2.51.0" 2024-09-13T14:35:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:44 smithi184 ceph-mon[92527]: pgmap v1047: 89 pgs: 89 active+clean; 218 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 27 MiB/s rd, 3.9 MiB/s wr, 35 op/s 2024-09-13T14:35:45.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:44 smithi184 ceph-mon[92527]: osdmap e953: 8 total, 8 up, 8 in 2024-09-13T14:35:45.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:44 smithi184 ceph-mon[92527]: osdmap e954: 8 total, 8 up, 8 in 2024-09-13T14:35:45.644 INFO:tasks.rados.rados.0.smithi151.stdout:3528: done (2 left) 2024-09-13T14:35:45.644 INFO:tasks.rados.rados.0.smithi151.stdout:3529: done (1 left) 2024-09-13T14:35:45.644 INFO:tasks.rados.rados.0.smithi151.stdout:3530: done (0 left) 2024-09-13T14:35:45.644 INFO:tasks.rados.rados.0.smithi151.stdout:3531: delete oid 23 current snap is 450 2024-09-13T14:35:45.647 INFO:tasks.rados.rados.0.smithi151.stdout:3531: done (0 left) 2024-09-13T14:35:45.647 INFO:tasks.rados.rados.0.smithi151.stdout:3532: read oid 34 snap -1 2024-09-13T14:35:45.647 INFO:tasks.rados.rados.0.smithi151.stdout:3532: expect (ObjNum 1298 snap 445 seq_num 1298) 2024-09-13T14:35:45.647 INFO:tasks.rados.rados.0.smithi151.stdout:3533: snap_remove snap 438 2024-09-13T14:35:46.651 INFO:tasks.rados.rados.0.smithi151.stdout:3532: expect (ObjNum 1298 snap 445 seq_num 1298) 2024-09-13T14:35:46.781 INFO:tasks.rados.rados.0.smithi151.stdout:3532: done (1 left) 2024-09-13T14:35:46.781 INFO:tasks.rados.rados.0.smithi151.stdout:3533: done (0 left) 2024-09-13T14:35:46.781 INFO:tasks.rados.rados.0.smithi151.stdout:3534: write oid 49 current snap is 450 2024-09-13T14:35:46.781 INFO:tasks.rados.rados.0.smithi151.stdout:3534: seq_num 1320 ranges {585240=657749,1743290=422125} 2024-09-13T14:35:46.787 INFO:tasks.rados.rados.0.smithi151.stdout:3534: writing smithi15186934-49 from 585240 to 1242989 tid 1 2024-09-13T14:35:46.792 INFO:tasks.rados.rados.0.smithi151.stdout:3534: writing smithi15186934-49 from 1743290 to 2165415 tid 2 2024-09-13T14:35:46.792 INFO:tasks.rados.rados.0.smithi151.stdout:3535: delete oid 6 current snap is 450 2024-09-13T14:35:46.792 INFO:tasks.rados.rados.0.smithi151.stdout:3534: finishing write tid 1 to smithi15186934-49 2024-09-13T14:35:46.794 INFO:tasks.rados.rados.0.smithi151.stdout:3535: done (1 left) 2024-09-13T14:35:46.794 INFO:tasks.rados.rados.0.smithi151.stdout:3536: read oid 34 snap -1 2024-09-13T14:35:46.795 INFO:tasks.rados.rados.0.smithi151.stdout:3536: expect (ObjNum 1298 snap 445 seq_num 1298) 2024-09-13T14:35:46.795 INFO:tasks.rados.rados.0.smithi151.stdout:3537: write oid 30 current snap is 450 2024-09-13T14:35:46.795 INFO:tasks.rados.rados.0.smithi151.stdout:3537: seq_num 1322 ranges {618854=408247,1743901=425794} 2024-09-13T14:35:46.800 INFO:tasks.rados.rados.0.smithi151.stdout:3537: writing smithi15186934-30 from 618854 to 1027101 tid 1 2024-09-13T14:35:46.806 INFO:tasks.rados.rados.0.smithi151.stdout:3537: writing smithi15186934-30 from 1743901 to 2169695 tid 2 2024-09-13T14:35:46.807 INFO:tasks.rados.rados.0.smithi151.stdout:3538: delete oid 2 current snap is 450 2024-09-13T14:35:46.807 INFO:tasks.rados.rados.0.smithi151.stdout:3534: finishing write tid 2 to smithi15186934-49 2024-09-13T14:35:46.807 INFO:tasks.rados.rados.0.smithi151.stdout:3534: finishing write tid 3 to smithi15186934-49 2024-09-13T14:35:46.807 INFO:tasks.rados.rados.0.smithi151.stdout:3534: finishing write tid 4 to smithi15186934-49 2024-09-13T14:35:46.807 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 49 v 1644 (ObjNum 1320 snap 450 seq_num 1320) dirty exists 2024-09-13T14:35:46.807 INFO:tasks.rados.rados.0.smithi151.stdout:3534: left oid 49 (ObjNum 1320 snap 450 seq_num 1320) 2024-09-13T14:35:46.807 INFO:tasks.rados.rados.0.smithi151.stdout:3536: expect (ObjNum 1298 snap 445 seq_num 1298) 2024-09-13T14:35:46.932 INFO:tasks.rados.rados.0.smithi151.stdout:3537: finishing write tid 1 to smithi15186934-30 2024-09-13T14:35:46.932 INFO:tasks.rados.rados.0.smithi151.stdout:3537: finishing write tid 2 to smithi15186934-30 2024-09-13T14:35:46.933 INFO:tasks.rados.rados.0.smithi151.stdout:3537: finishing write tid 3 to smithi15186934-30 2024-09-13T14:35:46.933 INFO:tasks.rados.rados.0.smithi151.stdout:3537: finishing write tid 4 to smithi15186934-30 2024-09-13T14:35:46.933 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 30 v 1340 (ObjNum 1322 snap 450 seq_num 1322) dirty exists 2024-09-13T14:35:46.933 INFO:tasks.rados.rados.0.smithi151.stdout:3537: left oid 30 (ObjNum 1322 snap 450 seq_num 1322) 2024-09-13T14:35:46.933 INFO:tasks.rados.rados.0.smithi151.stdout:3534: done (3 left) 2024-09-13T14:35:46.933 INFO:tasks.rados.rados.0.smithi151.stdout:3536: done (2 left) 2024-09-13T14:35:46.933 INFO:tasks.rados.rados.0.smithi151.stdout:3537: done (1 left) 2024-09-13T14:35:46.933 INFO:tasks.rados.rados.0.smithi151.stdout:3538: done (0 left) 2024-09-13T14:35:46.933 INFO:tasks.rados.rados.0.smithi151.stdout:3539: write oid 39 current snap is 450 2024-09-13T14:35:46.933 INFO:tasks.rados.rados.0.smithi151.stdout:3539: seq_num 1324 ranges {652468=675098,1977218=275250} 2024-09-13T14:35:46.939 INFO:tasks.rados.rados.0.smithi151.stdout:3539: writing smithi15186934-39 from 652468 to 1327566 tid 1 2024-09-13T14:35:46.949 INFO:tasks.rados.rados.0.smithi151.stdout:3539: writing smithi15186934-39 from 1977218 to 2252468 tid 2 2024-09-13T14:35:46.949 INFO:tasks.rados.rados.0.smithi151.stdout:3540: read oid 43 snap 424 2024-09-13T14:35:46.949 INFO:tasks.rados.rados.0.smithi151.stdout:3540: expect deleted 2024-09-13T14:35:46.949 INFO:tasks.rados.rados.0.smithi151.stdout:3541: rollback oid 13 current snap is 450 2024-09-13T14:35:46.949 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 13 to 433 2024-09-13T14:35:46.949 INFO:tasks.rados.rados.0.smithi151.stdout:3539: finishing write tid 1 to smithi15186934-39 2024-09-13T14:35:46.949 INFO:tasks.rados.rados.0.smithi151.stdout:3542: read oid 48 snap 428 2024-09-13T14:35:46.949 INFO:tasks.rados.rados.0.smithi151.stdout:3542: expect (ObjNum 1111 snap 372 seq_num 1111) 2024-09-13T14:35:46.950 INFO:tasks.rados.rados.0.smithi151.stdout:3543: rollback oid 12 current snap is 450 2024-09-13T14:35:46.950 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 12 to 445 2024-09-13T14:35:46.950 INFO:tasks.rados.rados.0.smithi151.stdout:3544: read oid 11 snap -1 2024-09-13T14:35:46.950 INFO:tasks.rados.rados.0.smithi151.stdout:3544: expect (ObjNum 1291 snap 443 seq_num 1291) 2024-09-13T14:35:46.950 INFO:tasks.rados.rados.0.smithi151.stdout:3545: rollback oid 29 current snap is 450 2024-09-13T14:35:46.950 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 29 to 426 2024-09-13T14:35:46.950 INFO:tasks.rados.rados.0.smithi151.stdout:3546: write oid 1 current snap is 450 2024-09-13T14:35:46.950 INFO:tasks.rados.rados.0.smithi151.stdout:3546: seq_num 1325 ranges {669275=750347,2035700=233575} 2024-09-13T14:35:46.953 INFO:tasks.rados.rados.0.smithi151.stdout:3546: writing smithi15186934-1 from 669275 to 1419622 tid 1 2024-09-13T14:35:46.958 INFO:tasks.rados.rados.0.smithi151.stdout:3546: writing smithi15186934-1 from 2035700 to 2269275 tid 2 2024-09-13T14:35:46.958 INFO:tasks.rados.rados.0.smithi151.stdout:3547: snap_remove snap 431 2024-09-13T14:35:47.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:46 smithi184 ceph-mon[92527]: pgmap v1050: 89 pgs: 89 active+clean; 232 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 27 MiB/s rd, 3.0 MiB/s wr, 33 op/s 2024-09-13T14:35:47.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:46 smithi184 ceph-mon[92527]: osdmap e955: 8 total, 8 up, 8 in 2024-09-13T14:35:47.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:47.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:46 smithi151 ceph-mon[104450]: pgmap v1050: 89 pgs: 89 active+clean; 232 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 27 MiB/s rd, 3.0 MiB/s wr, 33 op/s 2024-09-13T14:35:47.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:46 smithi151 ceph-mon[104450]: osdmap e955: 8 total, 8 up, 8 in 2024-09-13T14:35:47.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:47.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:46 smithi151 ceph-mon[108966]: pgmap v1050: 89 pgs: 89 active+clean; 232 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 27 MiB/s rd, 3.0 MiB/s wr, 33 op/s 2024-09-13T14:35:47.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:46 smithi151 ceph-mon[108966]: osdmap e955: 8 total, 8 up, 8 in 2024-09-13T14:35:47.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:47.667 INFO:tasks.rados.rados.0.smithi151.stdout:3547: done (8 left) 2024-09-13T14:35:47.668 INFO:tasks.rados.rados.0.smithi151.stdout:3548: read oid 42 snap -1 2024-09-13T14:35:47.668 INFO:tasks.rados.rados.0.smithi151.stdout:3548: expect (ObjNum 1307 snap 446 seq_num 1307) 2024-09-13T14:35:47.668 INFO:tasks.rados.rados.0.smithi151.stdout:3549: snap_create 2024-09-13T14:35:47.668 INFO:tasks.rados.rados.0.smithi151.stdout:3541: finishing rollback tid 0 to smithi15186934-13 2024-09-13T14:35:47.668 INFO:tasks.rados.rados.0.smithi151.stdout:3543: finishing rollback tid 0 to smithi15186934-12 2024-09-13T14:35:47.668 INFO:tasks.rados.rados.0.smithi151.stdout:3541: finishing rollback tid 1 to smithi15186934-13 2024-09-13T14:35:47.668 INFO:tasks.rados.rados.0.smithi151.stdout:3543: finishing rollback tid 1 to smithi15186934-12 2024-09-13T14:35:47.668 INFO:tasks.rados.rados.0.smithi151.stdout:3539: finishing write tid 2 to smithi15186934-39 2024-09-13T14:35:47.668 INFO:tasks.rados.rados.0.smithi151.stdout:3543: finishing rollback tid 2 to smithi15186934-12 2024-09-13T14:35:47.668 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 12 v 752 (ObjNum 1304 snap 445 seq_num 1304) dirty exists 2024-09-13T14:35:47.668 INFO:tasks.rados.rados.0.smithi151.stdout:3539: finishing write tid 3 to smithi15186934-39 2024-09-13T14:35:47.669 INFO:tasks.rados.rados.0.smithi151.stdout:3541: finishing rollback tid 2 to smithi15186934-13 2024-09-13T14:35:47.669 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 13 v 1376 (ObjNum 1201 snap 406 seq_num 1201) dirty exists 2024-09-13T14:35:47.669 INFO:tasks.rados.rados.0.smithi151.stdout:3539: finishing write tid 4 to smithi15186934-39 2024-09-13T14:35:47.669 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 1493 (ObjNum 1324 snap 450 seq_num 1324) dirty exists 2024-09-13T14:35:47.669 INFO:tasks.rados.rados.0.smithi151.stdout:3539: left oid 39 (ObjNum 1324 snap 450 seq_num 1324) 2024-09-13T14:35:47.669 INFO:tasks.rados.rados.0.smithi151.stdout:3544: expect (ObjNum 1291 snap 443 seq_num 1291) 2024-09-13T14:35:47.841 INFO:tasks.rados.rados.0.smithi151.stdout:3545: finishing rollback tid 1 to smithi15186934-29 2024-09-13T14:35:47.842 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 29 v 867 (ObjNum 33 snap 0 seq_num 3064833400) dirty dne 2024-09-13T14:35:47.842 INFO:tasks.rados.rados.0.smithi151.stdout:3542: expect (ObjNum 1111 snap 372 seq_num 1111) 2024-09-13T14:35:47.969 INFO:tasks.rados.rados.0.smithi151.stdout:3546: finishing write tid 1 to smithi15186934-1 2024-09-13T14:35:47.969 INFO:tasks.rados.rados.0.smithi151.stdout:3546: finishing write tid 2 to smithi15186934-1 2024-09-13T14:35:47.969 INFO:tasks.rados.rados.0.smithi151.stdout:3546: finishing write tid 3 to smithi15186934-1 2024-09-13T14:35:47.969 INFO:tasks.rados.rados.0.smithi151.stdout:3546: finishing write tid 4 to smithi15186934-1 2024-09-13T14:35:47.969 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 1 v 1110 (ObjNum 1325 snap 450 seq_num 1325) dirty exists 2024-09-13T14:35:47.969 INFO:tasks.rados.rados.0.smithi151.stdout:3546: left oid 1 (ObjNum 1325 snap 450 seq_num 1325) 2024-09-13T14:35:47.969 INFO:tasks.rados.rados.0.smithi151.stdout:3548: expect (ObjNum 1307 snap 446 seq_num 1307) 2024-09-13T14:35:48.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:47 smithi184 ceph-mon[92527]: osdmap e956: 8 total, 8 up, 8 in 2024-09-13T14:35:48.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:47 smithi151 ceph-mon[104450]: osdmap e956: 8 total, 8 up, 8 in 2024-09-13T14:35:48.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:47 smithi151 ceph-mon[108966]: osdmap e956: 8 total, 8 up, 8 in 2024-09-13T14:35:48.670 INFO:tasks.rados.rados.0.smithi151.stdout:3539: done (9 left) 2024-09-13T14:35:48.670 INFO:tasks.rados.rados.0.smithi151.stdout:3540: done (8 left) 2024-09-13T14:35:48.670 INFO:tasks.rados.rados.0.smithi151.stdout:3541: done (7 left) 2024-09-13T14:35:48.670 INFO:tasks.rados.rados.0.smithi151.stdout:3542: done (6 left) 2024-09-13T14:35:48.670 INFO:tasks.rados.rados.0.smithi151.stdout:3543: done (5 left) 2024-09-13T14:35:48.670 INFO:tasks.rados.rados.0.smithi151.stdout:3544: done (4 left) 2024-09-13T14:35:48.670 INFO:tasks.rados.rados.0.smithi151.stdout:3545: done (3 left) 2024-09-13T14:35:48.670 INFO:tasks.rados.rados.0.smithi151.stdout:3546: done (2 left) 2024-09-13T14:35:48.671 INFO:tasks.rados.rados.0.smithi151.stdout:3548: done (1 left) 2024-09-13T14:35:48.671 INFO:tasks.rados.rados.0.smithi151.stdout:3549: done (0 left) 2024-09-13T14:35:48.671 INFO:tasks.rados.rados.0.smithi151.stdout:3550: read oid 49 snap -1 2024-09-13T14:35:48.671 INFO:tasks.rados.rados.0.smithi151.stdout:3550: expect (ObjNum 1320 snap 450 seq_num 1320) 2024-09-13T14:35:48.671 INFO:tasks.rados.rados.0.smithi151.stdout:3551: snap_create 2024-09-13T14:35:48.679 INFO:tasks.rados.rados.0.smithi151.stdout:3550: expect (ObjNum 1320 snap 450 seq_num 1320) 2024-09-13T14:35:49.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:48 smithi184 ceph-mon[92527]: pgmap v1053: 89 pgs: 89 active+clean; 238 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.2 MiB/s rd, 2.8 MiB/s wr, 20 op/s 2024-09-13T14:35:49.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:48 smithi184 ceph-mon[92527]: osdmap e957: 8 total, 8 up, 8 in 2024-09-13T14:35:49.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:48 smithi151 ceph-mon[104450]: pgmap v1053: 89 pgs: 89 active+clean; 238 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.2 MiB/s rd, 2.8 MiB/s wr, 20 op/s 2024-09-13T14:35:49.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:48 smithi151 ceph-mon[104450]: osdmap e957: 8 total, 8 up, 8 in 2024-09-13T14:35:49.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:48 smithi151 ceph-mon[108966]: pgmap v1053: 89 pgs: 89 active+clean; 238 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.2 MiB/s rd, 2.8 MiB/s wr, 20 op/s 2024-09-13T14:35:49.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:48 smithi151 ceph-mon[108966]: osdmap e957: 8 total, 8 up, 8 in 2024-09-13T14:35:49.643 INFO:tasks.rados.rados.0.smithi151.stdout:3550: done (1 left) 2024-09-13T14:35:49.643 INFO:tasks.rados.rados.0.smithi151.stdout:3551: done (0 left) 2024-09-13T14:35:49.643 INFO:tasks.rados.rados.0.smithi151.stdout:3552: write oid 33 current snap is 452 2024-09-13T14:35:49.643 INFO:tasks.rados.rados.0.smithi151.stdout:3552: seq_num 1326 ranges {686082=425596,1577829=433132} 2024-09-13T14:35:49.649 INFO:tasks.rados.rados.0.smithi151.stdout:3552: writing smithi15186934-33 from 686082 to 1111678 tid 1 2024-09-13T14:35:49.654 INFO:tasks.rados.rados.0.smithi151.stdout:3552: writing smithi15186934-33 from 1577829 to 2010961 tid 2 2024-09-13T14:35:49.654 INFO:tasks.rados.rados.0.smithi151.stdout:3553: write oid 11 current snap is 452 2024-09-13T14:35:49.654 INFO:tasks.rados.rados.0.smithi151.stdout:3552: finishing write tid 1 to smithi15186934-33 2024-09-13T14:35:49.655 INFO:tasks.rados.rados.0.smithi151.stdout:3553: seq_num 1327 ranges {702889=500845,1636311=576790} 2024-09-13T14:35:49.661 INFO:tasks.rados.rados.0.smithi151.stdout:3553: writing smithi15186934-11 from 702889 to 1203734 tid 1 2024-09-13T14:35:49.667 INFO:tasks.rados.rados.0.smithi151.stdout:3553: writing smithi15186934-11 from 1636311 to 2213101 tid 2 2024-09-13T14:35:49.668 INFO:tasks.rados.rados.0.smithi151.stdout:3554: rollback oid 15 current snap is 452 2024-09-13T14:35:49.668 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 15 to 449 2024-09-13T14:35:49.668 INFO:tasks.rados.rados.0.smithi151.stdout:3552: finishing write tid 2 to smithi15186934-33 2024-09-13T14:35:49.668 INFO:tasks.rados.rados.0.smithi151.stdout:3552: finishing write tid 3 to smithi15186934-33 2024-09-13T14:35:49.668 INFO:tasks.rados.rados.0.smithi151.stdout:3552: finishing write tid 4 to smithi15186934-33 2024-09-13T14:35:49.668 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 33 v 1385 (ObjNum 1326 snap 452 seq_num 1326) dirty exists 2024-09-13T14:35:49.668 INFO:tasks.rados.rados.0.smithi151.stdout:3552: left oid 33 (ObjNum 1326 snap 452 seq_num 1326) 2024-09-13T14:35:49.668 INFO:tasks.rados.rados.0.smithi151.stdout:3553: finishing write tid 1 to smithi15186934-11 2024-09-13T14:35:49.668 INFO:tasks.rados.rados.0.smithi151.stdout:3552: done (2 left) 2024-09-13T14:35:49.668 INFO:tasks.rados.rados.0.smithi151.stdout:3555: write oid 13 current snap is 452 2024-09-13T14:35:49.668 INFO:tasks.rados.rados.0.smithi151.stdout:3555: seq_num 1328 ranges {719696=576094,2094793=224903} 2024-09-13T14:35:49.675 INFO:tasks.rados.rados.0.smithi151.stdout:3555: writing smithi15186934-13 from 719696 to 1295790 tid 1 2024-09-13T14:35:49.681 INFO:tasks.rados.rados.0.smithi151.stdout:3555: writing smithi15186934-13 from 2094793 to 2319696 tid 2 2024-09-13T14:35:49.681 INFO:tasks.rados.rados.0.smithi151.stdout:3556: rollback oid 6 current snap is 452 2024-09-13T14:35:49.681 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 6 to 433 2024-09-13T14:35:49.682 INFO:tasks.rados.rados.0.smithi151.stdout:3553: finishing write tid 2 to smithi15186934-11 2024-09-13T14:35:49.682 INFO:tasks.rados.rados.0.smithi151.stdout:3553: finishing write tid 3 to smithi15186934-11 2024-09-13T14:35:49.682 INFO:tasks.rados.rados.0.smithi151.stdout:3553: finishing write tid 4 to smithi15186934-11 2024-09-13T14:35:49.682 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 11 v 951 (ObjNum 1327 snap 452 seq_num 1327) dirty exists 2024-09-13T14:35:49.682 INFO:tasks.rados.rados.0.smithi151.stdout:3553: left oid 11 (ObjNum 1327 snap 452 seq_num 1327) 2024-09-13T14:35:49.682 INFO:tasks.rados.rados.0.smithi151.stdout:3554: finishing rollback tid 0 to smithi15186934-15 2024-09-13T14:35:49.682 INFO:tasks.rados.rados.0.smithi151.stdout:3553: done (3 left) 2024-09-13T14:35:49.682 INFO:tasks.rados.rados.0.smithi151.stdout:3557: delete oid 32 current snap is 452 2024-09-13T14:35:49.682 INFO:tasks.rados.rados.0.smithi151.stdout:3554: finishing rollback tid 1 to smithi15186934-15 2024-09-13T14:35:49.682 INFO:tasks.rados.rados.0.smithi151.stdout:3554: finishing rollback tid 2 to smithi15186934-15 2024-09-13T14:35:49.682 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 15 v 1337 (ObjNum 1310 snap 446 seq_num 1310) dirty exists 2024-09-13T14:35:49.682 INFO:tasks.rados.rados.0.smithi151.stdout:3555: finishing write tid 1 to smithi15186934-13 2024-09-13T14:35:49.683 INFO:tasks.rados.rados.0.smithi151.stdout:3556: finishing rollback tid 1 to smithi15186934-6 2024-09-13T14:35:49.683 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 6 v 785 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-13T14:35:49.684 INFO:tasks.rados.rados.0.smithi151.stdout:3554: done (3 left) 2024-09-13T14:35:49.684 INFO:tasks.rados.rados.0.smithi151.stdout:3556: done (2 left) 2024-09-13T14:35:49.684 INFO:tasks.rados.rados.0.smithi151.stdout:3557: done (1 left) 2024-09-13T14:35:49.684 INFO:tasks.rados.rados.0.smithi151.stdout:3558: write oid 19 current snap is 452 2024-09-13T14:35:49.684 INFO:tasks.rados.rados.0.smithi151.stdout:3558: seq_num 1330 ranges {753310=726592,2095404=257906} 2024-09-13T14:35:49.692 INFO:tasks.rados.rados.0.smithi151.stdout:3558: writing smithi15186934-19 from 753310 to 1479902 tid 1 2024-09-13T14:35:49.697 INFO:tasks.rados.rados.0.smithi151.stdout:3558: writing smithi15186934-19 from 2095404 to 2353310 tid 2 2024-09-13T14:35:49.697 INFO:tasks.rados.rados.0.smithi151.stdout:3559: write oid 36 current snap is 452 2024-09-13T14:35:49.697 INFO:tasks.rados.rados.0.smithi151.stdout:3555: finishing write tid 2 to smithi15186934-13 2024-09-13T14:35:49.697 INFO:tasks.rados.rados.0.smithi151.stdout:3559: seq_num 1331 ranges {770117=518194,1870239=499878} 2024-09-13T14:35:49.703 INFO:tasks.rados.rados.0.smithi151.stdout:3559: writing smithi15186934-36 from 770117 to 1288311 tid 1 2024-09-13T14:35:49.710 INFO:tasks.rados.rados.0.smithi151.stdout:3559: writing smithi15186934-36 from 1870239 to 2370117 tid 2 2024-09-13T14:35:49.710 INFO:tasks.rados.rados.0.smithi151.stdout:3560: read oid 27 snap -1 2024-09-13T14:35:49.710 INFO:tasks.rados.rados.0.smithi151.stdout:3560: expect deleted 2024-09-13T14:35:49.710 INFO:tasks.rados.rados.0.smithi151.stdout:3561: snap_remove snap 441 2024-09-13T14:35:50.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:49 smithi184 ceph-mon[92527]: osdmap e958: 8 total, 8 up, 8 in 2024-09-13T14:35:50.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:49 smithi184 ceph-mon[92527]: osdmap e959: 8 total, 8 up, 8 in 2024-09-13T14:35:50.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:49 smithi151 ceph-mon[104450]: osdmap e958: 8 total, 8 up, 8 in 2024-09-13T14:35:50.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:49 smithi151 ceph-mon[104450]: osdmap e959: 8 total, 8 up, 8 in 2024-09-13T14:35:50.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:49 smithi151 ceph-mon[108966]: osdmap e958: 8 total, 8 up, 8 in 2024-09-13T14:35:50.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:49 smithi151 ceph-mon[108966]: osdmap e959: 8 total, 8 up, 8 in 2024-09-13T14:35:50.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:35:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:35:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:35:50.684 INFO:tasks.rados.rados.0.smithi151.stdout:3561: done (4 left) 2024-09-13T14:35:50.684 INFO:tasks.rados.rados.0.smithi151.stdout:3562: snap_remove snap 444 2024-09-13T14:35:51.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:50 smithi184 ceph-mon[92527]: pgmap v1056: 89 pgs: 89 active+clean; 239 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.0 MiB/s rd, 938 KiB/s wr, 11 op/s 2024-09-13T14:35:51.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:50 smithi151 ceph-mon[104450]: pgmap v1056: 89 pgs: 89 active+clean; 239 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.0 MiB/s rd, 938 KiB/s wr, 11 op/s 2024-09-13T14:35:51.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:50 smithi151 ceph-mon[108966]: pgmap v1056: 89 pgs: 89 active+clean; 239 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.0 MiB/s rd, 938 KiB/s wr, 11 op/s 2024-09-13T14:35:51.692 INFO:tasks.rados.rados.0.smithi151.stdout:3562: done (4 left) 2024-09-13T14:35:51.692 INFO:tasks.rados.rados.0.smithi151.stdout:3563: write oid 5 current snap is 452 2024-09-13T14:35:51.692 INFO:tasks.rados.rados.0.smithi151.stdout:3563: seq_num 1332 ranges {786924=593443,1928721=458203} 2024-09-13T14:35:51.700 INFO:tasks.rados.rados.0.smithi151.stdout:3563: writing smithi15186934-5 from 786924 to 1380367 tid 1 2024-09-13T14:35:51.706 INFO:tasks.rados.rados.0.smithi151.stdout:3563: writing smithi15186934-5 from 1928721 to 2386924 tid 2 2024-09-13T14:35:51.706 INFO:tasks.rados.rados.0.smithi151.stdout:3564: write oid 30 current snap is 452 2024-09-13T14:35:51.706 INFO:tasks.rados.rados.0.smithi151.stdout:3555: finishing write tid 3 to smithi15186934-13 2024-09-13T14:35:51.706 INFO:tasks.rados.rados.0.smithi151.stdout:3555: finishing write tid 4 to smithi15186934-13 2024-09-13T14:35:51.707 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 13 v 1389 (ObjNum 1328 snap 452 seq_num 1328) dirty exists 2024-09-13T14:35:51.707 INFO:tasks.rados.rados.0.smithi151.stdout:3555: left oid 13 (ObjNum 1328 snap 452 seq_num 1328) 2024-09-13T14:35:51.707 INFO:tasks.rados.rados.0.smithi151.stdout:3558: finishing write tid 1 to smithi15186934-19 2024-09-13T14:35:51.707 INFO:tasks.rados.rados.0.smithi151.stdout:3564: seq_num 1333 ranges {403731=668692,1587203=587797} 2024-09-13T14:35:51.712 INFO:tasks.rados.rados.0.smithi151.stdout:3564: writing smithi15186934-30 from 403731 to 1072423 tid 1 2024-09-13T14:35:51.719 INFO:tasks.rados.rados.0.smithi151.stdout:3564: writing smithi15186934-30 from 1587203 to 2175000 tid 2 2024-09-13T14:35:51.719 INFO:tasks.rados.rados.0.smithi151.stdout:3555: done (5 left) 2024-09-13T14:35:51.719 INFO:tasks.rados.rados.0.smithi151.stdout:3565: delete oid 25 current snap is 452 2024-09-13T14:35:51.719 INFO:tasks.rados.rados.0.smithi151.stdout:3558: finishing write tid 2 to smithi15186934-19 2024-09-13T14:35:51.719 INFO:tasks.rados.rados.0.smithi151.stdout:3558: finishing write tid 3 to smithi15186934-19 2024-09-13T14:35:51.719 INFO:tasks.rados.rados.0.smithi151.stdout:3558: finishing write tid 4 to smithi15186934-19 2024-09-13T14:35:51.719 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 19 v 792 (ObjNum 1330 snap 452 seq_num 1330) dirty exists 2024-09-13T14:35:51.719 INFO:tasks.rados.rados.0.smithi151.stdout:3558: left oid 19 (ObjNum 1330 snap 452 seq_num 1330) 2024-09-13T14:35:51.720 INFO:tasks.rados.rados.0.smithi151.stdout:3559: finishing write tid 1 to smithi15186934-36 2024-09-13T14:35:51.720 INFO:tasks.rados.rados.0.smithi151.stdout:3559: finishing write tid 2 to smithi15186934-36 2024-09-13T14:35:51.720 INFO:tasks.rados.rados.0.smithi151.stdout:3559: finishing write tid 3 to smithi15186934-36 2024-09-13T14:35:51.720 INFO:tasks.rados.rados.0.smithi151.stdout:3559: finishing write tid 4 to smithi15186934-36 2024-09-13T14:35:51.720 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 36 v 757 (ObjNum 1331 snap 452 seq_num 1331) dirty exists 2024-09-13T14:35:51.720 INFO:tasks.rados.rados.0.smithi151.stdout:3559: left oid 36 (ObjNum 1331 snap 452 seq_num 1331) 2024-09-13T14:35:51.720 INFO:tasks.rados.rados.0.smithi151.stdout:3563: finishing write tid 1 to smithi15186934-5 2024-09-13T14:35:51.720 INFO:tasks.rados.rados.0.smithi151.stdout:3563: finishing write tid 2 to smithi15186934-5 2024-09-13T14:35:51.720 INFO:tasks.rados.rados.0.smithi151.stdout:3563: finishing write tid 3 to smithi15186934-5 2024-09-13T14:35:51.720 INFO:tasks.rados.rados.0.smithi151.stdout:3563: finishing write tid 4 to smithi15186934-5 2024-09-13T14:35:51.720 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 1399 (ObjNum 1332 snap 452 seq_num 1332) dirty exists 2024-09-13T14:35:51.720 INFO:tasks.rados.rados.0.smithi151.stdout:3563: left oid 5 (ObjNum 1332 snap 452 seq_num 1332) 2024-09-13T14:35:51.721 INFO:tasks.rados.rados.0.smithi151.stdout:3564: finishing write tid 1 to smithi15186934-30 2024-09-13T14:35:51.721 INFO:tasks.rados.rados.0.smithi151.stdout:3558: done (5 left) 2024-09-13T14:35:51.721 INFO:tasks.rados.rados.0.smithi151.stdout:3559: done (4 left) 2024-09-13T14:35:51.722 INFO:tasks.rados.rados.0.smithi151.stdout:3560: done (3 left) 2024-09-13T14:35:51.722 INFO:tasks.rados.rados.0.smithi151.stdout:3563: done (2 left) 2024-09-13T14:35:51.722 INFO:tasks.rados.rados.0.smithi151.stdout:3565: done (1 left) 2024-09-13T14:35:51.722 INFO:tasks.rados.rados.0.smithi151.stdout:3566: write oid 45 current snap is 452 2024-09-13T14:35:51.722 INFO:tasks.rados.rados.0.smithi151.stdout:3566: seq_num 1335 ranges {437345=419190,1587814=591466} 2024-09-13T14:35:51.726 INFO:tasks.rados.rados.0.smithi151.stdout:3566: writing smithi15186934-45 from 437345 to 856535 tid 1 2024-09-13T14:35:51.736 INFO:tasks.rados.rados.0.smithi151.stdout:3566: writing smithi15186934-45 from 1587814 to 2179280 tid 2 2024-09-13T14:35:51.736 INFO:tasks.rados.rados.0.smithi151.stdout:3567: delete oid 5 current snap is 452 2024-09-13T14:35:51.736 INFO:tasks.rados.rados.0.smithi151.stdout:3564: finishing write tid 2 to smithi15186934-30 2024-09-13T14:35:51.736 INFO:tasks.rados.rados.0.smithi151.stdout:3564: finishing write tid 3 to smithi15186934-30 2024-09-13T14:35:51.736 INFO:tasks.rados.rados.0.smithi151.stdout:3564: finishing write tid 4 to smithi15186934-30 2024-09-13T14:35:51.736 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 30 v 1344 (ObjNum 1333 snap 452 seq_num 1333) dirty exists 2024-09-13T14:35:51.736 INFO:tasks.rados.rados.0.smithi151.stdout:3564: left oid 30 (ObjNum 1333 snap 452 seq_num 1333) 2024-09-13T14:35:51.736 INFO:tasks.rados.rados.0.smithi151.stdout:3566: finishing write tid 1 to smithi15186934-45 2024-09-13T14:35:51.737 INFO:tasks.rados.rados.0.smithi151.stdout:3564: done (2 left) 2024-09-13T14:35:51.737 INFO:tasks.rados.rados.0.smithi151.stdout:3567: done (1 left) 2024-09-13T14:35:51.737 INFO:tasks.rados.rados.0.smithi151.stdout:3568: snap_create 2024-09-13T14:35:51.739 INFO:tasks.rados.rados.0.smithi151.stdout:3566: finishing write tid 2 to smithi15186934-45 2024-09-13T14:35:51.739 INFO:tasks.rados.rados.0.smithi151.stdout:3566: finishing write tid 3 to smithi15186934-45 2024-09-13T14:35:51.739 INFO:tasks.rados.rados.0.smithi151.stdout:3566: finishing write tid 4 to smithi15186934-45 2024-09-13T14:35:51.739 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 1732 (ObjNum 1335 snap 452 seq_num 1335) dirty exists 2024-09-13T14:35:51.739 INFO:tasks.rados.rados.0.smithi151.stdout:3566: left oid 45 (ObjNum 1335 snap 452 seq_num 1335) 2024-09-13T14:35:52.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:51 smithi184 ceph-mon[92527]: osdmap e960: 8 total, 8 up, 8 in 2024-09-13T14:35:52.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:51 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:35:52.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:51 smithi151 ceph-mon[104450]: osdmap e960: 8 total, 8 up, 8 in 2024-09-13T14:35:52.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:51 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:35:52.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:51 smithi151 ceph-mon[108966]: osdmap e960: 8 total, 8 up, 8 in 2024-09-13T14:35:52.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:51 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:35:52.695 INFO:tasks.rados.rados.0.smithi151.stdout:3566: done (1 left) 2024-09-13T14:35:52.695 INFO:tasks.rados.rados.0.smithi151.stdout:3568: done (0 left) 2024-09-13T14:35:52.695 INFO:tasks.rados.rados.0.smithi151.stdout:3569: snap_create 2024-09-13T14:35:52.970 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:52 smithi184 ceph-mon[92527]: pgmap v1059: 89 pgs: 89 active+clean; 246 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 957 KiB/s wr, 15 op/s 2024-09-13T14:35:52.970 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:52 smithi184 ceph-mon[92527]: osdmap e961: 8 total, 8 up, 8 in 2024-09-13T14:35:53.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:52 smithi151 ceph-mon[108966]: pgmap v1059: 89 pgs: 89 active+clean; 246 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 957 KiB/s wr, 15 op/s 2024-09-13T14:35:53.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:52 smithi151 ceph-mon[108966]: osdmap e961: 8 total, 8 up, 8 in 2024-09-13T14:35:53.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:52 smithi151 ceph-mon[104450]: pgmap v1059: 89 pgs: 89 active+clean; 246 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 957 KiB/s wr, 15 op/s 2024-09-13T14:35:53.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:52 smithi151 ceph-mon[104450]: osdmap e961: 8 total, 8 up, 8 in 2024-09-13T14:35:53.701 INFO:tasks.rados.rados.0.smithi151.stdout:3569: done (0 left) 2024-09-13T14:35:53.702 INFO:tasks.rados.rados.0.smithi151.stdout:3570: write oid 17 current snap is 454 2024-09-13T14:35:53.702 INFO:tasks.rados.rados.0.smithi151.stdout:3570: seq_num 1337 ranges {470959=569688,1704778=595135} 2024-09-13T14:35:53.707 INFO:tasks.rados.rados.0.smithi151.stdout:3570: writing smithi15186934-17 from 470959 to 1040647 tid 1 2024-09-13T14:35:53.715 INFO:tasks.rados.rados.0.smithi151.stdout:3570: writing smithi15186934-17 from 1704778 to 2299913 tid 2 2024-09-13T14:35:53.715 INFO:tasks.rados.rados.0.smithi151.stdout:3571: rollback oid 8 current snap is 454 2024-09-13T14:35:53.715 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 8 to 413 2024-09-13T14:35:53.715 INFO:tasks.rados.rados.0.smithi151.stdout:3570: finishing write tid 1 to smithi15186934-17 2024-09-13T14:35:53.716 INFO:tasks.rados.rados.0.smithi151.stdout:3572: write oid 34 current snap is 454 2024-09-13T14:35:53.716 INFO:tasks.rados.rados.0.smithi151.stdout:3572: seq_num 1338 ranges {487766=644937,1646907=738793} 2024-09-13T14:35:53.724 INFO:tasks.rados.rados.0.smithi151.stdout:3572: writing smithi15186934-34 from 487766 to 1132703 tid 1 2024-09-13T14:35:53.731 INFO:tasks.rados.rados.0.smithi151.stdout:3572: writing smithi15186934-34 from 1646907 to 2385700 tid 2 2024-09-13T14:35:53.732 INFO:tasks.rados.rados.0.smithi151.stdout:3573: delete oid 11 current snap is 454 2024-09-13T14:35:53.732 INFO:tasks.rados.rados.0.smithi151.stdout:3570: finishing write tid 2 to smithi15186934-17 2024-09-13T14:35:53.732 INFO:tasks.rados.rados.0.smithi151.stdout:3570: finishing write tid 3 to smithi15186934-17 2024-09-13T14:35:53.732 INFO:tasks.rados.rados.0.smithi151.stdout:3571: finishing rollback tid 1 to smithi15186934-8 2024-09-13T14:35:53.732 INFO:tasks.rados.rados.0.smithi151.stdout:3571: finishing rollback tid 2 to smithi15186934-8 2024-09-13T14:35:53.732 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 1126 (ObjNum 1224 snap 410 seq_num 1224) dirty exists 2024-09-13T14:35:53.732 INFO:tasks.rados.rados.0.smithi151.stdout:3570: finishing write tid 4 to smithi15186934-17 2024-09-13T14:35:53.732 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 2018 (ObjNum 1337 snap 454 seq_num 1337) dirty exists 2024-09-13T14:35:53.732 INFO:tasks.rados.rados.0.smithi151.stdout:3570: left oid 17 (ObjNum 1337 snap 454 seq_num 1337) 2024-09-13T14:35:53.732 INFO:tasks.rados.rados.0.smithi151.stdout:3572: finishing write tid 1 to smithi15186934-34 2024-09-13T14:35:53.735 INFO:tasks.rados.rados.0.smithi151.stdout:3570: done (3 left) 2024-09-13T14:35:53.735 INFO:tasks.rados.rados.0.smithi151.stdout:3571: done (2 left) 2024-09-13T14:35:53.735 INFO:tasks.rados.rados.0.smithi151.stdout:3573: done (1 left) 2024-09-13T14:35:53.735 INFO:tasks.rados.rados.0.smithi151.stdout:3574: read oid 27 snap -1 2024-09-13T14:35:53.735 INFO:tasks.rados.rados.0.smithi151.stdout:3574: expect deleted 2024-09-13T14:35:53.735 INFO:tasks.rados.rados.0.smithi151.stdout:3575: snap_remove snap 448 2024-09-13T14:35:53.985 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:53 smithi184 ceph-mon[92527]: osdmap e962: 8 total, 8 up, 8 in 2024-09-13T14:35:54.000 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:53 smithi151 ceph-mon[104450]: osdmap e962: 8 total, 8 up, 8 in 2024-09-13T14:35:54.001 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:53 smithi151 ceph-mon[108966]: osdmap e962: 8 total, 8 up, 8 in 2024-09-13T14:35:54.644 INFO:tasks.rados.rados.0.smithi151.stdout:3575: done (2 left) 2024-09-13T14:35:54.645 INFO:tasks.rados.rados.0.smithi151.stdout:3576: delete oid 30 current snap is 454 2024-09-13T14:35:54.645 INFO:tasks.rados.rados.0.smithi151.stdout:3572: finishing write tid 2 to smithi15186934-34 2024-09-13T14:35:54.645 INFO:tasks.rados.rados.0.smithi151.stdout:3572: finishing write tid 3 to smithi15186934-34 2024-09-13T14:35:54.645 INFO:tasks.rados.rados.0.smithi151.stdout:3572: finishing write tid 4 to smithi15186934-34 2024-09-13T14:35:54.645 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 1348 (ObjNum 1338 snap 454 seq_num 1338) dirty exists 2024-09-13T14:35:54.645 INFO:tasks.rados.rados.0.smithi151.stdout:3572: left oid 34 (ObjNum 1338 snap 454 seq_num 1338) 2024-09-13T14:35:54.649 INFO:tasks.rados.rados.0.smithi151.stdout:3572: done (2 left) 2024-09-13T14:35:54.649 INFO:tasks.rados.rados.0.smithi151.stdout:3574: done (1 left) 2024-09-13T14:35:54.649 INFO:tasks.rados.rados.0.smithi151.stdout:3576: done (0 left) 2024-09-13T14:35:54.649 INFO:tasks.rados.rados.0.smithi151.stdout:3577: read oid 21 snap -1 2024-09-13T14:35:54.649 INFO:tasks.rados.rados.0.smithi151.stdout:3577: expect (ObjNum 1318 snap 449 seq_num 1318) 2024-09-13T14:35:54.649 INFO:tasks.rados.rados.0.smithi151.stdout:3578: delete oid 36 current snap is 454 2024-09-13T14:35:54.652 INFO:tasks.rados.rados.0.smithi151.stdout:3578: done (1 left) 2024-09-13T14:35:54.652 INFO:tasks.rados.rados.0.smithi151.stdout:3579: read oid 31 snap -1 2024-09-13T14:35:54.652 INFO:tasks.rados.rados.0.smithi151.stdout:3579: expect deleted 2024-09-13T14:35:54.653 INFO:tasks.rados.rados.0.smithi151.stdout:3580: write oid 44 current snap is 454 2024-09-13T14:35:54.653 INFO:tasks.rados.rados.0.smithi151.stdout:3580: seq_num 1342 ranges {554994=662286,1997188=557806} 2024-09-13T14:35:54.660 INFO:tasks.rados.rados.0.smithi151.stdout:3580: writing smithi15186934-44 from 554994 to 1217280 tid 1 2024-09-13T14:35:54.668 INFO:tasks.rados.rados.0.smithi151.stdout:3580: writing smithi15186934-44 from 1997188 to 2554994 tid 2 2024-09-13T14:35:54.668 INFO:tasks.rados.rados.0.smithi151.stdout:3581: read oid 24 snap -1 2024-09-13T14:35:54.668 INFO:tasks.rados.rados.0.smithi151.stdout:3581: expect (ObjNum 1284 snap 438 seq_num 1284) 2024-09-13T14:35:54.668 INFO:tasks.rados.rados.0.smithi151.stdout:3582: read oid 17 snap -1 2024-09-13T14:35:54.668 INFO:tasks.rados.rados.0.smithi151.stdout:3582: expect (ObjNum 1337 snap 454 seq_num 1337) 2024-09-13T14:35:54.668 INFO:tasks.rados.rados.0.smithi151.stdout:3583: read oid 32 snap -1 2024-09-13T14:35:54.668 INFO:tasks.rados.rados.0.smithi151.stdout:3583: expect deleted 2024-09-13T14:35:54.668 INFO:tasks.rados.rados.0.smithi151.stdout:3584: read oid 9 snap -1 2024-09-13T14:35:54.669 INFO:tasks.rados.rados.0.smithi151.stdout:3584: expect (ObjNum 1301 snap 445 seq_num 1301) 2024-09-13T14:35:54.669 INFO:tasks.rados.rados.0.smithi151.stdout:3585: rollback oid 47 current snap is 454 2024-09-13T14:35:54.669 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 47 to 449 2024-09-13T14:35:54.669 INFO:tasks.rados.rados.0.smithi151.stdout:3577: expect (ObjNum 1318 snap 449 seq_num 1318) 2024-09-13T14:35:54.842 INFO:tasks.rados.rados.0.smithi151.stdout:3580: finishing write tid 1 to smithi15186934-44 2024-09-13T14:35:54.842 INFO:tasks.rados.rados.0.smithi151.stdout:3585: finishing rollback tid 1 to smithi15186934-47 2024-09-13T14:35:54.842 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 1548 (ObjNum 33 snap 0 seq_num 3064833400) dirty dne 2024-09-13T14:35:54.842 INFO:tasks.rados.rados.0.smithi151.stdout:3580: finishing write tid 2 to smithi15186934-44 2024-09-13T14:35:54.842 INFO:tasks.rados.rados.0.smithi151.stdout:3577: done (7 left) 2024-09-13T14:35:54.842 INFO:tasks.rados.rados.0.smithi151.stdout:3579: done (6 left) 2024-09-13T14:35:54.842 INFO:tasks.rados.rados.0.smithi151.stdout:3583: done (5 left) 2024-09-13T14:35:54.842 INFO:tasks.rados.rados.0.smithi151.stdout:3585: done (4 left) 2024-09-13T14:35:54.842 INFO:tasks.rados.rados.0.smithi151.stdout:3586: read oid 13 snap -1 2024-09-13T14:35:54.842 INFO:tasks.rados.rados.0.smithi151.stdout:3586: expect (ObjNum 1328 snap 452 seq_num 1328) 2024-09-13T14:35:54.842 INFO:tasks.rados.rados.0.smithi151.stdout:3587: snap_remove snap 424 2024-09-13T14:35:54.851 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:54 smithi151 ceph-mon[104450]: pgmap v1062: 89 pgs: 89 active+clean; 242 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.4 MiB/s wr, 17 op/s 2024-09-13T14:35:54.851 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:54 smithi151 ceph-mon[104450]: osdmap e963: 8 total, 8 up, 8 in 2024-09-13T14:35:54.852 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:54 smithi151 ceph-mon[104450]: osdmap e964: 8 total, 8 up, 8 in 2024-09-13T14:35:54.852 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:35:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:35:54] "GET /metrics HTTP/1.1" 200 36511 "" "Prometheus/2.51.0" 2024-09-13T14:35:54.852 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:54 smithi151 ceph-mon[108966]: pgmap v1062: 89 pgs: 89 active+clean; 242 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.4 MiB/s wr, 17 op/s 2024-09-13T14:35:54.852 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:54 smithi151 ceph-mon[108966]: osdmap e963: 8 total, 8 up, 8 in 2024-09-13T14:35:54.852 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:54 smithi151 ceph-mon[108966]: osdmap e964: 8 total, 8 up, 8 in 2024-09-13T14:35:55.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:54 smithi184 ceph-mon[92527]: pgmap v1062: 89 pgs: 89 active+clean; 242 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.4 MiB/s wr, 17 op/s 2024-09-13T14:35:55.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:54 smithi184 ceph-mon[92527]: osdmap e963: 8 total, 8 up, 8 in 2024-09-13T14:35:55.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:54 smithi184 ceph-mon[92527]: osdmap e964: 8 total, 8 up, 8 in 2024-09-13T14:35:55.708 INFO:tasks.rados.rados.0.smithi151.stdout:3587: done (5 left) 2024-09-13T14:35:55.709 INFO:tasks.rados.rados.0.smithi151.stdout:3588: write oid 48 current snap is 454 2024-09-13T14:35:55.709 INFO:tasks.rados.rados.0.smithi151.stdout:3588: seq_num 1343 ranges {571801=737535,1939317=489789} 2024-09-13T14:35:55.716 INFO:tasks.rados.rados.0.smithi151.stdout:3588: writing smithi15186934-48 from 571801 to 1309336 tid 1 2024-09-13T14:35:55.723 INFO:tasks.rados.rados.0.smithi151.stdout:3588: writing smithi15186934-48 from 1939317 to 2429106 tid 2 2024-09-13T14:35:55.723 INFO:tasks.rados.rados.0.smithi151.stdout:3589: rollback oid 47 current snap is 454 2024-09-13T14:35:55.723 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 47 to 453 2024-09-13T14:35:55.723 INFO:tasks.rados.rados.0.smithi151.stdout:3580: finishing write tid 3 to smithi15186934-44 2024-09-13T14:35:55.723 INFO:tasks.rados.rados.0.smithi151.stdout:3580: finishing write tid 4 to smithi15186934-44 2024-09-13T14:35:55.723 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 1134 (ObjNum 1342 snap 454 seq_num 1342) dirty exists 2024-09-13T14:35:55.723 INFO:tasks.rados.rados.0.smithi151.stdout:3580: left oid 44 (ObjNum 1342 snap 454 seq_num 1342) 2024-09-13T14:35:55.723 INFO:tasks.rados.rados.0.smithi151.stdout:3584: expect (ObjNum 1301 snap 445 seq_num 1301) 2024-09-13T14:35:55.965 INFO:tasks.rados.rados.0.smithi151.stdout:3581: expect (ObjNum 1284 snap 438 seq_num 1284) 2024-09-13T14:35:56.159 INFO:tasks.rados.rados.0.smithi151.stdout:3582: expect (ObjNum 1337 snap 454 seq_num 1337) 2024-09-13T14:35:56.266 INFO:tasks.rados.rados.0.smithi151.stdout:3586: expect (ObjNum 1328 snap 452 seq_num 1328) 2024-09-13T14:35:56.434 INFO:tasks.rados.rados.0.smithi151.stdout:3588: finishing write tid 1 to smithi15186934-48 2024-09-13T14:35:56.434 INFO:tasks.rados.rados.0.smithi151.stdout:3589: finishing rollback tid 1 to smithi15186934-47 2024-09-13T14:35:56.434 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 0 (ObjNum 65 snap 0 seq_num 3680676221) dirty dne 2024-09-13T14:35:56.434 INFO:tasks.rados.rados.0.smithi151.stdout:3588: finishing write tid 2 to smithi15186934-48 2024-09-13T14:35:56.434 INFO:tasks.rados.rados.0.smithi151.stdout:3588: finishing write tid 3 to smithi15186934-48 2024-09-13T14:35:56.434 INFO:tasks.rados.rados.0.smithi151.stdout:3588: finishing write tid 4 to smithi15186934-48 2024-09-13T14:35:56.434 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 48 v 1745 (ObjNum 1343 snap 454 seq_num 1343) dirty exists 2024-09-13T14:35:56.434 INFO:tasks.rados.rados.0.smithi151.stdout:3588: left oid 48 (ObjNum 1343 snap 454 seq_num 1343) 2024-09-13T14:35:56.435 INFO:tasks.rados.rados.0.smithi151.stdout:3580: done (6 left) 2024-09-13T14:35:56.435 INFO:tasks.rados.rados.0.smithi151.stdout:3581: done (5 left) 2024-09-13T14:35:56.435 INFO:tasks.rados.rados.0.smithi151.stdout:3582: done (4 left) 2024-09-13T14:35:56.435 INFO:tasks.rados.rados.0.smithi151.stdout:3584: done (3 left) 2024-09-13T14:35:56.435 INFO:tasks.rados.rados.0.smithi151.stdout:3586: done (2 left) 2024-09-13T14:35:56.435 INFO:tasks.rados.rados.0.smithi151.stdout:3588: done (1 left) 2024-09-13T14:35:56.435 INFO:tasks.rados.rados.0.smithi151.stdout:3589: done (0 left) 2024-09-13T14:35:56.436 INFO:tasks.rados.rados.0.smithi151.stdout:3590: snap_remove snap 440 2024-09-13T14:35:56.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:56 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:35:56.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:56 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:35:56.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:56 smithi184 ceph-mon[92527]: pgmap v1065: 89 pgs: 89 active+clean; 245 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-13T14:35:56.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:56 smithi184 ceph-mon[92527]: osdmap e965: 8 total, 8 up, 8 in 2024-09-13T14:35:56.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:56 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:35:56.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:56 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:35:56.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:56 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:35:56.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:56 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:35:56.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:56 smithi151 ceph-mon[104450]: pgmap v1065: 89 pgs: 89 active+clean; 245 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-13T14:35:56.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:56 smithi151 ceph-mon[104450]: osdmap e965: 8 total, 8 up, 8 in 2024-09-13T14:35:56.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:56 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:35:56.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:56 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:35:56.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:56 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:35:56.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:56 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:35:56.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:56 smithi151 ceph-mon[108966]: pgmap v1065: 89 pgs: 89 active+clean; 245 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-13T14:35:56.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:56 smithi151 ceph-mon[108966]: osdmap e965: 8 total, 8 up, 8 in 2024-09-13T14:35:56.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:56 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:35:56.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:56 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:35:57.225 INFO:tasks.rados.rados.0.smithi151.stdout:3590: done (0 left) 2024-09-13T14:35:57.225 INFO:tasks.rados.rados.0.smithi151.stdout:3591: write oid 47 current snap is 454 2024-09-13T14:35:57.226 INFO:tasks.rados.rados.0.smithi151.stdout:3591: seq_num 1344 ranges {588608=412784,1597799=749800} 2024-09-13T14:35:57.231 INFO:tasks.rados.rados.0.smithi151.stdout:3591: writing smithi15186934-47 from 588608 to 1001392 tid 1 2024-09-13T14:35:57.240 INFO:tasks.rados.rados.0.smithi151.stdout:3591: writing smithi15186934-47 from 1597799 to 2347599 tid 2 2024-09-13T14:35:57.240 INFO:tasks.rados.rados.0.smithi151.stdout:3592: snap_remove snap 426 2024-09-13T14:35:57.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:57 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:35:57.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:57 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:35:57.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:57 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:35:57.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:57 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:35:57.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:57 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:35:57.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:57 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:35:57.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:57 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:35:57.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:57 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:35:57.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:57 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:35:58.240 INFO:tasks.rados.rados.0.smithi151.stdout:3592: done (1 left) 2024-09-13T14:35:58.240 INFO:tasks.rados.rados.0.smithi151.stdout:3593: write oid 33 current snap is 454 2024-09-13T14:35:58.240 INFO:tasks.rados.rados.0.smithi151.stdout:3593: seq_num 1345 ranges {605415=488033,1656281=493458} 2024-09-13T14:35:58.247 INFO:tasks.rados.rados.0.smithi151.stdout:3593: writing smithi15186934-33 from 605415 to 1093448 tid 1 2024-09-13T14:35:58.254 INFO:tasks.rados.rados.0.smithi151.stdout:3593: writing smithi15186934-33 from 1656281 to 2149739 tid 2 2024-09-13T14:35:58.254 INFO:tasks.rados.rados.0.smithi151.stdout:3594: read oid 37 snap -1 2024-09-13T14:35:58.254 INFO:tasks.rados.rados.0.smithi151.stdout:3594: expect (ObjNum 1275 snap 437 seq_num 1275) 2024-09-13T14:35:58.254 INFO:tasks.rados.rados.0.smithi151.stdout:3595: write oid 42 current snap is 454 2024-09-13T14:35:58.254 INFO:tasks.rados.rados.0.smithi151.stdout:3591: finishing write tid 1 to smithi15186934-47 2024-09-13T14:35:58.254 INFO:tasks.rados.rados.0.smithi151.stdout:3595: seq_num 1346 ranges {622222=679635,1831116=753469} 2024-09-13T14:35:58.261 INFO:tasks.rados.rados.0.smithi151.stdout:3595: writing smithi15186934-42 from 622222 to 1301857 tid 1 2024-09-13T14:35:58.269 INFO:tasks.rados.rados.0.smithi151.stdout:3595: writing smithi15186934-42 from 1831116 to 2584585 tid 2 2024-09-13T14:35:58.269 INFO:tasks.rados.rados.0.smithi151.stdout:3596: write oid 5 current snap is 454 2024-09-13T14:35:58.269 INFO:tasks.rados.rados.0.smithi151.stdout:3596: seq_num 1347 ranges {639029=754884,2173245=465784} 2024-09-13T14:35:58.276 INFO:tasks.rados.rados.0.smithi151.stdout:3596: writing smithi15186934-5 from 639029 to 1393913 tid 1 2024-09-13T14:35:58.284 INFO:tasks.rados.rados.0.smithi151.stdout:3596: writing smithi15186934-5 from 2173245 to 2639029 tid 2 2024-09-13T14:35:58.284 INFO:tasks.rados.rados.0.smithi151.stdout:3597: delete oid 32 current snap is 454 2024-09-13T14:35:58.284 INFO:tasks.rados.rados.0.smithi151.stdout:3591: finishing write tid 2 to smithi15186934-47 2024-09-13T14:35:58.284 INFO:tasks.rados.rados.0.smithi151.stdout:3591: finishing write tid 3 to smithi15186934-47 2024-09-13T14:35:58.284 INFO:tasks.rados.rados.0.smithi151.stdout:3591: finishing write tid 4 to smithi15186934-47 2024-09-13T14:35:58.284 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 1668 (ObjNum 1344 snap 454 seq_num 1344) dirty exists 2024-09-13T14:35:58.284 INFO:tasks.rados.rados.0.smithi151.stdout:3591: left oid 47 (ObjNum 1344 snap 454 seq_num 1344) 2024-09-13T14:35:58.284 INFO:tasks.rados.rados.0.smithi151.stdout:3593: finishing write tid 1 to smithi15186934-33 2024-09-13T14:35:58.284 INFO:tasks.rados.rados.0.smithi151.stdout:3593: finishing write tid 2 to smithi15186934-33 2024-09-13T14:35:58.284 INFO:tasks.rados.rados.0.smithi151.stdout:3593: finishing write tid 3 to smithi15186934-33 2024-09-13T14:35:58.284 INFO:tasks.rados.rados.0.smithi151.stdout:3593: finishing write tid 4 to smithi15186934-33 2024-09-13T14:35:58.284 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 33 v 1422 (ObjNum 1345 snap 454 seq_num 1345) dirty exists 2024-09-13T14:35:58.285 INFO:tasks.rados.rados.0.smithi151.stdout:3593: left oid 33 (ObjNum 1345 snap 454 seq_num 1345) 2024-09-13T14:35:58.285 INFO:tasks.rados.rados.0.smithi151.stdout:3595: finishing write tid 1 to smithi15186934-42 2024-09-13T14:35:58.285 INFO:tasks.rados.rados.0.smithi151.stdout:3594: expect (ObjNum 1275 snap 437 seq_num 1275) 2024-09-13T14:35:58.497 INFO:tasks.rados.rados.0.smithi151.stdout:3595: finishing write tid 2 to smithi15186934-42 2024-09-13T14:35:58.497 INFO:tasks.rados.rados.0.smithi151.stdout:3595: finishing write tid 3 to smithi15186934-42 2024-09-13T14:35:58.497 INFO:tasks.rados.rados.0.smithi151.stdout:3595: finishing write tid 4 to smithi15186934-42 2024-09-13T14:35:58.497 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 1355 (ObjNum 1346 snap 454 seq_num 1346) dirty exists 2024-09-13T14:35:58.497 INFO:tasks.rados.rados.0.smithi151.stdout:3595: left oid 42 (ObjNum 1346 snap 454 seq_num 1346) 2024-09-13T14:35:58.497 INFO:tasks.rados.rados.0.smithi151.stdout:3596: finishing write tid 1 to smithi15186934-5 2024-09-13T14:35:58.497 INFO:tasks.rados.rados.0.smithi151.stdout:3591: done (5 left) 2024-09-13T14:35:58.498 INFO:tasks.rados.rados.0.smithi151.stdout:3593: done (4 left) 2024-09-13T14:35:58.498 INFO:tasks.rados.rados.0.smithi151.stdout:3594: done (3 left) 2024-09-13T14:35:58.498 INFO:tasks.rados.rados.0.smithi151.stdout:3595: done (2 left) 2024-09-13T14:35:58.498 INFO:tasks.rados.rados.0.smithi151.stdout:3597: done (1 left) 2024-09-13T14:35:58.498 INFO:tasks.rados.rados.0.smithi151.stdout:3598: snap_create 2024-09-13T14:35:58.498 INFO:tasks.rados.rados.0.smithi151.stdout:3596: finishing write tid 2 to smithi15186934-5 2024-09-13T14:35:58.498 INFO:tasks.rados.rados.0.smithi151.stdout:3596: finishing write tid 3 to smithi15186934-5 2024-09-13T14:35:58.498 INFO:tasks.rados.rados.0.smithi151.stdout:3596: finishing write tid 4 to smithi15186934-5 2024-09-13T14:35:58.498 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 1425 (ObjNum 1347 snap 454 seq_num 1347) dirty exists 2024-09-13T14:35:58.498 INFO:tasks.rados.rados.0.smithi151.stdout:3596: left oid 5 (ObjNum 1347 snap 454 seq_num 1347) 2024-09-13T14:35:58.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:58 smithi184 ceph-mon[92527]: osdmap e966: 8 total, 8 up, 8 in 2024-09-13T14:35:58.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:58 smithi184 ceph-mon[92527]: pgmap v1068: 89 pgs: 89 active+clean; 250 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 2.1 MiB/s wr, 21 op/s 2024-09-13T14:35:58.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:58 smithi151 ceph-mon[104450]: osdmap e966: 8 total, 8 up, 8 in 2024-09-13T14:35:58.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:58 smithi151 ceph-mon[104450]: pgmap v1068: 89 pgs: 89 active+clean; 250 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 2.1 MiB/s wr, 21 op/s 2024-09-13T14:35:58.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:58 smithi151 ceph-mon[108966]: osdmap e966: 8 total, 8 up, 8 in 2024-09-13T14:35:58.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:58 smithi151 ceph-mon[108966]: pgmap v1068: 89 pgs: 89 active+clean; 250 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 2.1 MiB/s wr, 21 op/s 2024-09-13T14:35:59.239 INFO:tasks.rados.rados.0.smithi151.stdout:3596: done (1 left) 2024-09-13T14:35:59.239 INFO:tasks.rados.rados.0.smithi151.stdout:3598: done (0 left) 2024-09-13T14:35:59.239 INFO:tasks.rados.rados.0.smithi151.stdout:3599: read oid 9 snap -1 2024-09-13T14:35:59.239 INFO:tasks.rados.rados.0.smithi151.stdout:3599: expect (ObjNum 1301 snap 445 seq_num 1301) 2024-09-13T14:35:59.239 INFO:tasks.rados.rados.0.smithi151.stdout:3600: snap_create 2024-09-13T14:35:59.251 INFO:tasks.rados.rados.0.smithi151.stdout:3599: expect (ObjNum 1301 snap 445 seq_num 1301) 2024-09-13T14:35:59.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:35:59 smithi184 ceph-mon[92527]: osdmap e967: 8 total, 8 up, 8 in 2024-09-13T14:35:59.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:35:59 smithi151 ceph-mon[104450]: osdmap e967: 8 total, 8 up, 8 in 2024-09-13T14:35:59.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:35:59 smithi151 ceph-mon[108966]: osdmap e967: 8 total, 8 up, 8 in 2024-09-13T14:35:59.650 INFO:tasks.rados.rados.0.smithi151.stdout:3599: done (1 left) 2024-09-13T14:35:59.651 INFO:tasks.rados.rados.0.smithi151.stdout:3600: done (0 left) 2024-09-13T14:35:59.651 INFO:tasks.rados.rados.0.smithi151.stdout:3601: snap_create 2024-09-13T14:36:00.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:00 smithi184 ceph-mon[92527]: osdmap e968: 8 total, 8 up, 8 in 2024-09-13T14:36:00.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:00 smithi184 ceph-mon[92527]: pgmap v1071: 89 pgs: 89 active+clean; 255 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 2.4 MiB/s wr, 22 op/s 2024-09-13T14:36:00.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:00 smithi184 ceph-mon[92527]: osdmap e969: 8 total, 8 up, 8 in 2024-09-13T14:36:00.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:36:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:36:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:36:00.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:00 smithi151 ceph-mon[104450]: osdmap e968: 8 total, 8 up, 8 in 2024-09-13T14:36:00.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:00 smithi151 ceph-mon[104450]: pgmap v1071: 89 pgs: 89 active+clean; 255 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 2.4 MiB/s wr, 22 op/s 2024-09-13T14:36:00.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:00 smithi151 ceph-mon[104450]: osdmap e969: 8 total, 8 up, 8 in 2024-09-13T14:36:00.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:00 smithi151 ceph-mon[108966]: osdmap e968: 8 total, 8 up, 8 in 2024-09-13T14:36:00.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:00 smithi151 ceph-mon[108966]: pgmap v1071: 89 pgs: 89 active+clean; 255 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 2.4 MiB/s wr, 22 op/s 2024-09-13T14:36:00.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:00 smithi151 ceph-mon[108966]: osdmap e969: 8 total, 8 up, 8 in 2024-09-13T14:36:00.654 INFO:tasks.rados.rados.0.smithi151.stdout:3601: done (0 left) 2024-09-13T14:36:00.655 INFO:tasks.rados.rados.0.smithi151.stdout:3602: snap_remove snap 433 2024-09-13T14:36:01.658 INFO:tasks.rados.rados.0.smithi151.stdout:3602: done (0 left) 2024-09-13T14:36:01.658 INFO:tasks.rados.rados.0.smithi151.stdout:3603: write oid 1 current snap is 457 2024-09-13T14:36:01.658 INFO:tasks.rados.rados.0.smithi151.stdout:3603: seq_num 1349 ranges {672643=505382,1890209=500796} 2024-09-13T14:36:01.665 INFO:tasks.rados.rados.0.smithi151.stdout:3603: writing smithi15186934-1 from 672643 to 1178025 tid 1 2024-09-13T14:36:01.673 INFO:tasks.rados.rados.0.smithi151.stdout:3603: writing smithi15186934-1 from 1890209 to 2391005 tid 2 2024-09-13T14:36:01.673 INFO:tasks.rados.rados.0.smithi151.stdout:3604: read oid 2 snap -1 2024-09-13T14:36:01.673 INFO:tasks.rados.rados.0.smithi151.stdout:3604: expect deleted 2024-09-13T14:36:01.673 INFO:tasks.rados.rados.0.smithi151.stdout:3605: write oid 13 current snap is 457 2024-09-13T14:36:01.673 INFO:tasks.rados.rados.0.smithi151.stdout:3605: seq_num 1350 ranges {689450=580631,1948691=740759} 2024-09-13T14:36:01.680 INFO:tasks.rados.rados.0.smithi151.stdout:3605: writing smithi15186934-13 from 689450 to 1270081 tid 1 2024-09-13T14:36:01.688 INFO:tasks.rados.rados.0.smithi151.stdout:3605: writing smithi15186934-13 from 1948691 to 2689450 tid 2 2024-09-13T14:36:01.689 INFO:tasks.rados.rados.0.smithi151.stdout:3606: delete oid 34 current snap is 457 2024-09-13T14:36:01.689 INFO:tasks.rados.rados.0.smithi151.stdout:3603: finishing write tid 1 to smithi15186934-1 2024-09-13T14:36:01.689 INFO:tasks.rados.rados.0.smithi151.stdout:3603: finishing write tid 2 to smithi15186934-1 2024-09-13T14:36:01.689 INFO:tasks.rados.rados.0.smithi151.stdout:3603: finishing write tid 3 to smithi15186934-1 2024-09-13T14:36:01.689 INFO:tasks.rados.rados.0.smithi151.stdout:3603: finishing write tid 4 to smithi15186934-1 2024-09-13T14:36:01.689 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 1 v 1160 (ObjNum 1349 snap 457 seq_num 1349) dirty exists 2024-09-13T14:36:01.689 INFO:tasks.rados.rados.0.smithi151.stdout:3603: left oid 1 (ObjNum 1349 snap 457 seq_num 1349) 2024-09-13T14:36:01.689 INFO:tasks.rados.rados.0.smithi151.stdout:3605: finishing write tid 1 to smithi15186934-13 2024-09-13T14:36:01.692 INFO:tasks.rados.rados.0.smithi151.stdout:3603: done (3 left) 2024-09-13T14:36:01.692 INFO:tasks.rados.rados.0.smithi151.stdout:3604: done (2 left) 2024-09-13T14:36:01.692 INFO:tasks.rados.rados.0.smithi151.stdout:3606: done (1 left) 2024-09-13T14:36:01.692 INFO:tasks.rados.rados.0.smithi151.stdout:3607: snap_remove snap 456 2024-09-13T14:36:02.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:01 smithi184 ceph-mon[92527]: osdmap e970: 8 total, 8 up, 8 in 2024-09-13T14:36:02.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:01 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:02.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:01 smithi151 ceph-mon[108966]: osdmap e970: 8 total, 8 up, 8 in 2024-09-13T14:36:02.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:01 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:02.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:01 smithi151 ceph-mon[104450]: osdmap e970: 8 total, 8 up, 8 in 2024-09-13T14:36:02.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:02.677 INFO:tasks.rados.rados.0.smithi151.stdout:3607: done (1 left) 2024-09-13T14:36:02.677 INFO:tasks.rados.rados.0.smithi151.stdout:3608: rollback oid 41 current snap is 457 2024-09-13T14:36:02.677 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 41 to 450 2024-09-13T14:36:02.677 INFO:tasks.rados.rados.0.smithi151.stdout:3605: finishing write tid 2 to smithi15186934-13 2024-09-13T14:36:02.677 INFO:tasks.rados.rados.0.smithi151.stdout:3605: finishing write tid 3 to smithi15186934-13 2024-09-13T14:36:02.677 INFO:tasks.rados.rados.0.smithi151.stdout:3605: finishing write tid 4 to smithi15186934-13 2024-09-13T14:36:02.677 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 13 v 1435 (ObjNum 1350 snap 457 seq_num 1350) dirty exists 2024-09-13T14:36:02.677 INFO:tasks.rados.rados.0.smithi151.stdout:3605: left oid 13 (ObjNum 1350 snap 457 seq_num 1350) 2024-09-13T14:36:02.677 INFO:tasks.rados.rados.0.smithi151.stdout:3605: done (1 left) 2024-09-13T14:36:02.677 INFO:tasks.rados.rados.0.smithi151.stdout:3609: write oid 16 current snap is 457 2024-09-13T14:36:02.678 INFO:tasks.rados.rados.0.smithi151.stdout:3609: seq_num 1352 ranges {723064=731129,1949302=764476} 2024-09-13T14:36:02.685 INFO:tasks.rados.rados.0.smithi151.stdout:3609: writing smithi15186934-16 from 723064 to 1454193 tid 1 2024-09-13T14:36:02.692 INFO:tasks.rados.rados.0.smithi151.stdout:3609: writing smithi15186934-16 from 1949302 to 2713778 tid 2 2024-09-13T14:36:02.692 INFO:tasks.rados.rados.0.smithi151.stdout:3610: delete oid 22 current snap is 457 2024-09-13T14:36:02.692 INFO:tasks.rados.rados.0.smithi151.stdout:3608: finishing rollback tid 0 to smithi15186934-41 2024-09-13T14:36:02.692 INFO:tasks.rados.rados.0.smithi151.stdout:3608: finishing rollback tid 1 to smithi15186934-41 2024-09-13T14:36:02.692 INFO:tasks.rados.rados.0.smithi151.stdout:3608: finishing rollback tid 2 to smithi15186934-41 2024-09-13T14:36:02.692 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 302 (ObjNum 1313 snap 446 seq_num 1313) dirty exists 2024-09-13T14:36:02.693 INFO:tasks.rados.rados.0.smithi151.stdout:3609: finishing write tid 1 to smithi15186934-16 2024-09-13T14:36:02.694 INFO:tasks.rados.rados.0.smithi151.stdout:3608: done (2 left) 2024-09-13T14:36:02.694 INFO:tasks.rados.rados.0.smithi151.stdout:3610: done (1 left) 2024-09-13T14:36:02.695 INFO:tasks.rados.rados.0.smithi151.stdout:3611: read oid 19 snap 453 2024-09-13T14:36:02.695 INFO:tasks.rados.rados.0.smithi151.stdout:3611: expect (ObjNum 1330 snap 452 seq_num 1330) 2024-09-13T14:36:02.695 INFO:tasks.rados.rados.0.smithi151.stdout:3612: snap_remove snap 449 2024-09-13T14:36:03.010 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:02 smithi151 ceph-mon[104450]: pgmap v1074: 89 pgs: 89 active+clean; 250 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.2 MiB/s rd, 888 KiB/s wr, 7 op/s 2024-09-13T14:36:03.010 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:02 smithi151 ceph-mon[104450]: osdmap e971: 8 total, 8 up, 8 in 2024-09-13T14:36:03.010 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:02 smithi151 ceph-mon[108966]: pgmap v1074: 89 pgs: 89 active+clean; 250 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.2 MiB/s rd, 888 KiB/s wr, 7 op/s 2024-09-13T14:36:03.010 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:02 smithi151 ceph-mon[108966]: osdmap e971: 8 total, 8 up, 8 in 2024-09-13T14:36:03.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:02 smithi184 ceph-mon[92527]: pgmap v1074: 89 pgs: 89 active+clean; 250 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.2 MiB/s rd, 888 KiB/s wr, 7 op/s 2024-09-13T14:36:03.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:02 smithi184 ceph-mon[92527]: osdmap e971: 8 total, 8 up, 8 in 2024-09-13T14:36:03.688 INFO:tasks.rados.rados.0.smithi151.stdout:3609: finishing write tid 2 to smithi15186934-16 2024-09-13T14:36:03.688 INFO:tasks.rados.rados.0.smithi151.stdout:3609: finishing write tid 3 to smithi15186934-16 2024-09-13T14:36:03.688 INFO:tasks.rados.rados.0.smithi151.stdout:3609: finishing write tid 4 to smithi15186934-16 2024-09-13T14:36:03.688 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 16 v 1038 (ObjNum 1352 snap 457 seq_num 1352) dirty exists 2024-09-13T14:36:03.688 INFO:tasks.rados.rados.0.smithi151.stdout:3609: left oid 16 (ObjNum 1352 snap 457 seq_num 1352) 2024-09-13T14:36:03.688 INFO:tasks.rados.rados.0.smithi151.stdout:3611: expect (ObjNum 1330 snap 452 seq_num 1330) 2024-09-13T14:36:03.790 INFO:tasks.rados.rados.0.smithi151.stdout:3609: done (2 left) 2024-09-13T14:36:03.790 INFO:tasks.rados.rados.0.smithi151.stdout:3611: done (1 left) 2024-09-13T14:36:03.791 INFO:tasks.rados.rados.0.smithi151.stdout:3612: done (0 left) 2024-09-13T14:36:03.791 INFO:tasks.rados.rados.0.smithi151.stdout:3613: read oid 33 snap -1 2024-09-13T14:36:03.791 INFO:tasks.rados.rados.0.smithi151.stdout:3613: expect (ObjNum 1345 snap 454 seq_num 1345) 2024-09-13T14:36:03.791 INFO:tasks.rados.rados.0.smithi151.stdout:3614: read oid 36 snap -1 2024-09-13T14:36:03.791 INFO:tasks.rados.rados.0.smithi151.stdout:3614: expect deleted 2024-09-13T14:36:03.791 INFO:tasks.rados.rados.0.smithi151.stdout:3615: write oid 6 current snap is 457 2024-09-13T14:36:03.791 INFO:tasks.rados.rados.0.smithi151.stdout:3615: seq_num 1354 ranges {756678=597980,1782619=768145} 2024-09-13T14:36:03.797 INFO:tasks.rados.rados.0.smithi151.stdout:3615: writing smithi15186934-6 from 756678 to 1354658 tid 1 2024-09-13T14:36:03.804 INFO:tasks.rados.rados.0.smithi151.stdout:3615: writing smithi15186934-6 from 1782619 to 2550764 tid 2 2024-09-13T14:36:03.804 INFO:tasks.rados.rados.0.smithi151.stdout:3616: write oid 31 current snap is 457 2024-09-13T14:36:03.804 INFO:tasks.rados.rados.0.smithi151.stdout:3616: seq_num 1355 ranges {773485=673229,2124748=511803} 2024-09-13T14:36:03.812 INFO:tasks.rados.rados.0.smithi151.stdout:3616: writing smithi15186934-31 from 773485 to 1446714 tid 1 2024-09-13T14:36:03.819 INFO:tasks.rados.rados.0.smithi151.stdout:3616: writing smithi15186934-31 from 2124748 to 2636551 tid 2 2024-09-13T14:36:03.819 INFO:tasks.rados.rados.0.smithi151.stdout:3617: delete oid 25 current snap is 457 2024-09-13T14:36:03.819 INFO:tasks.rados.rados.0.smithi151.stdout:3613: expect (ObjNum 1345 snap 454 seq_num 1345) 2024-09-13T14:36:03.941 INFO:tasks.rados.rados.0.smithi151.stdout:3615: finishing write tid 1 to smithi15186934-6 2024-09-13T14:36:03.941 INFO:tasks.rados.rados.0.smithi151.stdout:3615: finishing write tid 2 to smithi15186934-6 2024-09-13T14:36:03.941 INFO:tasks.rados.rados.0.smithi151.stdout:3615: finishing write tid 3 to smithi15186934-6 2024-09-13T14:36:03.941 INFO:tasks.rados.rados.0.smithi151.stdout:3615: finishing write tid 4 to smithi15186934-6 2024-09-13T14:36:03.941 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 6 v 817 (ObjNum 1354 snap 457 seq_num 1354) dirty exists 2024-09-13T14:36:03.941 INFO:tasks.rados.rados.0.smithi151.stdout:3615: left oid 6 (ObjNum 1354 snap 457 seq_num 1354) 2024-09-13T14:36:03.941 INFO:tasks.rados.rados.0.smithi151.stdout:3616: finishing write tid 1 to smithi15186934-31 2024-09-13T14:36:03.942 INFO:tasks.rados.rados.0.smithi151.stdout:3616: finishing write tid 2 to smithi15186934-31 2024-09-13T14:36:03.942 INFO:tasks.rados.rados.0.smithi151.stdout:3616: finishing write tid 3 to smithi15186934-31 2024-09-13T14:36:03.942 INFO:tasks.rados.rados.0.smithi151.stdout:3613: done (4 left) 2024-09-13T14:36:03.942 INFO:tasks.rados.rados.0.smithi151.stdout:3614: done (3 left) 2024-09-13T14:36:03.942 INFO:tasks.rados.rados.0.smithi151.stdout:3615: done (2 left) 2024-09-13T14:36:03.942 INFO:tasks.rados.rados.0.smithi151.stdout:3617: done (1 left) 2024-09-13T14:36:03.942 INFO:tasks.rados.rados.0.smithi151.stdout:3618: read oid 10 snap -1 2024-09-13T14:36:03.942 INFO:tasks.rados.rados.0.smithi151.stdout:3618: expect (ObjNum 1306 snap 445 seq_num 1306) 2024-09-13T14:36:03.942 INFO:tasks.rados.rados.0.smithi151.stdout:3619: rollback oid 43 current snap is 457 2024-09-13T14:36:03.942 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 43 to 452 2024-09-13T14:36:03.942 INFO:tasks.rados.rados.0.smithi151.stdout:3616: finishing write tid 4 to smithi15186934-31 2024-09-13T14:36:03.942 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 31 v 1366 (ObjNum 1355 snap 457 seq_num 1355) dirty exists 2024-09-13T14:36:03.942 INFO:tasks.rados.rados.0.smithi151.stdout:3616: left oid 31 (ObjNum 1355 snap 457 seq_num 1355) 2024-09-13T14:36:03.942 INFO:tasks.rados.rados.0.smithi151.stdout:3616: done (2 left) 2024-09-13T14:36:03.942 INFO:tasks.rados.rados.0.smithi151.stdout:3620: snap_create 2024-09-13T14:36:03.943 INFO:tasks.rados.rados.0.smithi151.stdout:3619: finishing rollback tid 1 to smithi15186934-43 2024-09-13T14:36:03.943 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 43 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-13T14:36:03.945 INFO:tasks.rados.rados.0.smithi151.stdout:3618: expect (ObjNum 1306 snap 445 seq_num 1306) 2024-09-13T14:36:04.031 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:03 smithi151 ceph-mon[104450]: osdmap e972: 8 total, 8 up, 8 in 2024-09-13T14:36:04.031 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:03 smithi151 ceph-mon[108966]: osdmap e972: 8 total, 8 up, 8 in 2024-09-13T14:36:04.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:03 smithi184 ceph-mon[92527]: osdmap e972: 8 total, 8 up, 8 in 2024-09-13T14:36:04.653 INFO:tasks.rados.rados.0.smithi151.stdout:3618: done (2 left) 2024-09-13T14:36:04.653 INFO:tasks.rados.rados.0.smithi151.stdout:3619: done (1 left) 2024-09-13T14:36:04.653 INFO:tasks.rados.rados.0.smithi151.stdout:3620: done (0 left) 2024-09-13T14:36:04.653 INFO:tasks.rados.rados.0.smithi151.stdout:3621: read oid 2 snap -1 2024-09-13T14:36:04.653 INFO:tasks.rados.rados.0.smithi151.stdout:3621: expect deleted 2024-09-13T14:36:04.653 INFO:tasks.rados.rados.0.smithi151.stdout:3622: snap_remove snap 452 2024-09-13T14:36:04.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:04 smithi151 ceph-mon[104450]: pgmap v1077: 89 pgs: 89 active+clean; 250 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.0 MiB/s rd, 284 KiB/s wr, 5 op/s 2024-09-13T14:36:04.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:04 smithi151 ceph-mon[104450]: osdmap e973: 8 total, 8 up, 8 in 2024-09-13T14:36:04.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:04 smithi151 ceph-mon[104450]: osdmap e974: 8 total, 8 up, 8 in 2024-09-13T14:36:04.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:04 smithi151 ceph-mon[108966]: pgmap v1077: 89 pgs: 89 active+clean; 250 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.0 MiB/s rd, 284 KiB/s wr, 5 op/s 2024-09-13T14:36:04.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:04 smithi151 ceph-mon[108966]: osdmap e973: 8 total, 8 up, 8 in 2024-09-13T14:36:04.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:04 smithi151 ceph-mon[108966]: osdmap e974: 8 total, 8 up, 8 in 2024-09-13T14:36:04.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:36:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:36:04] "GET /metrics HTTP/1.1" 200 36507 "" "Prometheus/2.51.0" 2024-09-13T14:36:05.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:04 smithi184 ceph-mon[92527]: pgmap v1077: 89 pgs: 89 active+clean; 250 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.0 MiB/s rd, 284 KiB/s wr, 5 op/s 2024-09-13T14:36:05.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:04 smithi184 ceph-mon[92527]: osdmap e973: 8 total, 8 up, 8 in 2024-09-13T14:36:05.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:04 smithi184 ceph-mon[92527]: osdmap e974: 8 total, 8 up, 8 in 2024-09-13T14:36:05.656 INFO:tasks.rados.rados.0.smithi151.stdout:3622: done (1 left) 2024-09-13T14:36:05.657 INFO:tasks.rados.rados.0.smithi151.stdout:3623: write oid 18 current snap is 458 2024-09-13T14:36:05.657 INFO:tasks.rados.rados.0.smithi151.stdout:3623: seq_num 1357 ranges {407099=423727,1441712=515472,2582060=225039} 2024-09-13T14:36:05.661 INFO:tasks.rados.rados.0.smithi151.stdout:3623: writing smithi15186934-18 from 407099 to 830826 tid 1 2024-09-13T14:36:05.668 INFO:tasks.rados.rados.0.smithi151.stdout:3623: writing smithi15186934-18 from 1441712 to 1957184 tid 2 2024-09-13T14:36:05.673 INFO:tasks.rados.rados.0.smithi151.stdout:3623: writing smithi15186934-18 from 2582060 to 2807099 tid 3 2024-09-13T14:36:05.673 INFO:tasks.rados.rados.0.smithi151.stdout:3624: delete oid 32 current snap is 458 2024-09-13T14:36:05.673 INFO:tasks.rados.rados.0.smithi151.stdout:3623: finishing write tid 1 to smithi15186934-18 2024-09-13T14:36:05.673 INFO:tasks.rados.rados.0.smithi151.stdout:3623: finishing write tid 2 to smithi15186934-18 2024-09-13T14:36:05.675 INFO:tasks.rados.rados.0.smithi151.stdout:3623: finishing write tid 3 to smithi15186934-18 2024-09-13T14:36:05.675 INFO:tasks.rados.rados.0.smithi151.stdout:3621: done (2 left) 2024-09-13T14:36:05.675 INFO:tasks.rados.rados.0.smithi151.stdout:3624: done (1 left) 2024-09-13T14:36:05.675 INFO:tasks.rados.rados.0.smithi151.stdout:3625: read oid 1 snap -1 2024-09-13T14:36:05.675 INFO:tasks.rados.rados.0.smithi151.stdout:3625: expect (ObjNum 1349 snap 457 seq_num 1349) 2024-09-13T14:36:05.676 INFO:tasks.rados.rados.0.smithi151.stdout:3626: write oid 23 current snap is 458 2024-09-13T14:36:05.676 INFO:tasks.rados.rados.0.smithi151.stdout:3623: finishing write tid 4 to smithi15186934-18 2024-09-13T14:36:05.676 INFO:tasks.rados.rados.0.smithi151.stdout:3626: seq_num 1359 ranges {440713=574225,1442323=519141,2636906=203807} 2024-09-13T14:36:05.681 INFO:tasks.rados.rados.0.smithi151.stdout:3626: writing smithi15186934-23 from 440713 to 1014938 tid 1 2024-09-13T14:36:05.686 INFO:tasks.rados.rados.0.smithi151.stdout:3626: writing smithi15186934-23 from 1442323 to 1961464 tid 2 2024-09-13T14:36:05.691 INFO:tasks.rados.rados.0.smithi151.stdout:3626: writing smithi15186934-23 from 2636906 to 2840713 tid 3 2024-09-13T14:36:05.691 INFO:tasks.rados.rados.0.smithi151.stdout:3627: delete oid 27 current snap is 458 2024-09-13T14:36:05.691 INFO:tasks.rados.rados.0.smithi151.stdout:3623: finishing write tid 5 to smithi15186934-18 2024-09-13T14:36:05.691 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 18 v 986 (ObjNum 1357 snap 458 seq_num 1357) dirty exists 2024-09-13T14:36:05.691 INFO:tasks.rados.rados.0.smithi151.stdout:3623: left oid 18 (ObjNum 1357 snap 458 seq_num 1357) 2024-09-13T14:36:05.692 INFO:tasks.rados.rados.0.smithi151.stdout:3625: expect (ObjNum 1349 snap 457 seq_num 1349) 2024-09-13T14:36:05.869 INFO:tasks.rados.rados.0.smithi151.stdout:3626: finishing write tid 1 to smithi15186934-23 2024-09-13T14:36:05.869 INFO:tasks.rados.rados.0.smithi151.stdout:3626: finishing write tid 2 to smithi15186934-23 2024-09-13T14:36:05.870 INFO:tasks.rados.rados.0.smithi151.stdout:3626: finishing write tid 3 to smithi15186934-23 2024-09-13T14:36:05.870 INFO:tasks.rados.rados.0.smithi151.stdout:3626: finishing write tid 4 to smithi15186934-23 2024-09-13T14:36:05.870 INFO:tasks.rados.rados.0.smithi151.stdout:3626: finishing write tid 5 to smithi15186934-23 2024-09-13T14:36:05.870 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 23 v 1682 (ObjNum 1359 snap 458 seq_num 1359) dirty exists 2024-09-13T14:36:05.870 INFO:tasks.rados.rados.0.smithi151.stdout:3626: left oid 23 (ObjNum 1359 snap 458 seq_num 1359) 2024-09-13T14:36:05.870 INFO:tasks.rados.rados.0.smithi151.stdout:3623: done (3 left) 2024-09-13T14:36:05.870 INFO:tasks.rados.rados.0.smithi151.stdout:3625: done (2 left) 2024-09-13T14:36:05.870 INFO:tasks.rados.rados.0.smithi151.stdout:3626: done (1 left) 2024-09-13T14:36:05.870 INFO:tasks.rados.rados.0.smithi151.stdout:3627: done (0 left) 2024-09-13T14:36:05.870 INFO:tasks.rados.rados.0.smithi151.stdout:3628: delete oid 5 current snap is 458 2024-09-13T14:36:05.872 INFO:tasks.rados.rados.0.smithi151.stdout:3628: done (0 left) 2024-09-13T14:36:05.872 INFO:tasks.rados.rados.0.smithi151.stdout:3629: write oid 32 current snap is 458 2024-09-13T14:36:05.872 INFO:tasks.rados.rados.0.smithi151.stdout:3629: seq_num 1362 ranges {491134=516325,1734122=666468} 2024-09-13T14:36:05.877 INFO:tasks.rados.rados.0.smithi151.stdout:3629: writing smithi15186934-32 from 491134 to 1007459 tid 1 2024-09-13T14:36:05.885 INFO:tasks.rados.rados.0.smithi151.stdout:3629: writing smithi15186934-32 from 1734122 to 2400590 tid 2 2024-09-13T14:36:05.885 INFO:tasks.rados.rados.0.smithi151.stdout:3630: write oid 41 current snap is 458 2024-09-13T14:36:05.885 INFO:tasks.rados.rados.0.smithi151.stdout:3629: finishing write tid 1 to smithi15186934-32 2024-09-13T14:36:05.885 INFO:tasks.rados.rados.0.smithi151.stdout:3630: seq_num 1363 ranges {507941=591574,1676251=526479,2746598=161343} 2024-09-13T14:36:05.891 INFO:tasks.rados.rados.0.smithi151.stdout:3630: writing smithi15186934-41 from 507941 to 1099515 tid 1 2024-09-13T14:36:05.897 INFO:tasks.rados.rados.0.smithi151.stdout:3630: writing smithi15186934-41 from 1676251 to 2202730 tid 2 2024-09-13T14:36:05.901 INFO:tasks.rados.rados.0.smithi151.stdout:3630: writing smithi15186934-41 from 2746598 to 2907941 tid 3 2024-09-13T14:36:05.901 INFO:tasks.rados.rados.0.smithi151.stdout:3631: snap_remove snap 437 2024-09-13T14:36:06.691 INFO:tasks.rados.rados.0.smithi151.stdout:3631: done (2 left) 2024-09-13T14:36:06.691 INFO:tasks.rados.rados.0.smithi151.stdout:3632: snap_create 2024-09-13T14:36:06.691 INFO:tasks.rados.rados.0.smithi151.stdout:3629: finishing write tid 2 to smithi15186934-32 2024-09-13T14:36:06.692 INFO:tasks.rados.rados.0.smithi151.stdout:3629: finishing write tid 3 to smithi15186934-32 2024-09-13T14:36:06.692 INFO:tasks.rados.rados.0.smithi151.stdout:3629: finishing write tid 4 to smithi15186934-32 2024-09-13T14:36:06.692 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 32 v 1471 (ObjNum 1362 snap 458 seq_num 1362) dirty exists 2024-09-13T14:36:06.692 INFO:tasks.rados.rados.0.smithi151.stdout:3629: left oid 32 (ObjNum 1362 snap 458 seq_num 1362) 2024-09-13T14:36:06.692 INFO:tasks.rados.rados.0.smithi151.stdout:3630: finishing write tid 1 to smithi15186934-41 2024-09-13T14:36:06.692 INFO:tasks.rados.rados.0.smithi151.stdout:3630: finishing write tid 2 to smithi15186934-41 2024-09-13T14:36:06.692 INFO:tasks.rados.rados.0.smithi151.stdout:3630: finishing write tid 3 to smithi15186934-41 2024-09-13T14:36:06.692 INFO:tasks.rados.rados.0.smithi151.stdout:3630: finishing write tid 4 to smithi15186934-41 2024-09-13T14:36:06.692 INFO:tasks.rados.rados.0.smithi151.stdout:3630: finishing write tid 5 to smithi15186934-41 2024-09-13T14:36:06.692 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 311 (ObjNum 1363 snap 458 seq_num 1363) dirty exists 2024-09-13T14:36:06.692 INFO:tasks.rados.rados.0.smithi151.stdout:3630: left oid 41 (ObjNum 1363 snap 458 seq_num 1363) 2024-09-13T14:36:07.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:06 smithi184 ceph-mon[92527]: pgmap v1080: 89 pgs: 89 active+clean; 250 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 607 KiB/s wr, 4 op/s 2024-09-13T14:36:07.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:06 smithi184 ceph-mon[92527]: osdmap e975: 8 total, 8 up, 8 in 2024-09-13T14:36:07.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:06 smithi151 ceph-mon[104450]: pgmap v1080: 89 pgs: 89 active+clean; 250 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 607 KiB/s wr, 4 op/s 2024-09-13T14:36:07.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:06 smithi151 ceph-mon[104450]: osdmap e975: 8 total, 8 up, 8 in 2024-09-13T14:36:07.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:06 smithi151 ceph-mon[108966]: pgmap v1080: 89 pgs: 89 active+clean; 250 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 607 KiB/s wr, 4 op/s 2024-09-13T14:36:07.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:06 smithi151 ceph-mon[108966]: osdmap e975: 8 total, 8 up, 8 in 2024-09-13T14:36:07.695 INFO:tasks.rados.rados.0.smithi151.stdout:3629: done (2 left) 2024-09-13T14:36:07.695 INFO:tasks.rados.rados.0.smithi151.stdout:3630: done (1 left) 2024-09-13T14:36:07.695 INFO:tasks.rados.rados.0.smithi151.stdout:3632: done (0 left) 2024-09-13T14:36:07.695 INFO:tasks.rados.rados.0.smithi151.stdout:3633: snap_create 2024-09-13T14:36:08.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:07 smithi184 ceph-mon[92527]: osdmap e976: 8 total, 8 up, 8 in 2024-09-13T14:36:08.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:07 smithi151 ceph-mon[104450]: osdmap e976: 8 total, 8 up, 8 in 2024-09-13T14:36:08.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:07 smithi151 ceph-mon[108966]: osdmap e976: 8 total, 8 up, 8 in 2024-09-13T14:36:08.695 INFO:tasks.rados.rados.0.smithi151.stdout:3633: done (0 left) 2024-09-13T14:36:08.695 INFO:tasks.rados.rados.0.smithi151.stdout:3634: write oid 9 current snap is 460 2024-09-13T14:36:08.695 INFO:tasks.rados.rados.0.smithi151.stdout:3634: seq_num 1364 ranges {524748=666823,1734733=670137} 2024-09-13T14:36:08.701 INFO:tasks.rados.rados.0.smithi151.stdout:3634: writing smithi15186934-9 from 524748 to 1191571 tid 1 2024-09-13T14:36:08.708 INFO:tasks.rados.rados.0.smithi151.stdout:3634: writing smithi15186934-9 from 1734733 to 2404870 tid 2 2024-09-13T14:36:08.709 INFO:tasks.rados.rados.0.smithi151.stdout:3635: write oid 7 current snap is 460 2024-09-13T14:36:08.709 INFO:tasks.rados.rados.0.smithi151.stdout:3635: seq_num 1365 ranges {541555=742072,1793215=530148,2801444=140111} 2024-09-13T14:36:08.715 INFO:tasks.rados.rados.0.smithi151.stdout:3635: writing smithi15186934-7 from 541555 to 1283627 tid 1 2024-09-13T14:36:08.722 INFO:tasks.rados.rados.0.smithi151.stdout:3635: writing smithi15186934-7 from 1793215 to 2323363 tid 2 2024-09-13T14:36:08.725 INFO:tasks.rados.rados.0.smithi151.stdout:3635: writing smithi15186934-7 from 2801444 to 2941555 tid 3 2024-09-13T14:36:08.725 INFO:tasks.rados.rados.0.smithi151.stdout:3636: delete oid 18 current snap is 460 2024-09-13T14:36:08.726 INFO:tasks.rados.rados.0.smithi151.stdout:3634: finishing write tid 1 to smithi15186934-9 2024-09-13T14:36:08.726 INFO:tasks.rados.rados.0.smithi151.stdout:3634: finishing write tid 2 to smithi15186934-9 2024-09-13T14:36:08.726 INFO:tasks.rados.rados.0.smithi151.stdout:3634: finishing write tid 3 to smithi15186934-9 2024-09-13T14:36:08.726 INFO:tasks.rados.rados.0.smithi151.stdout:3634: finishing write tid 4 to smithi15186934-9 2024-09-13T14:36:08.726 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 1479 (ObjNum 1364 snap 460 seq_num 1364) dirty exists 2024-09-13T14:36:08.726 INFO:tasks.rados.rados.0.smithi151.stdout:3634: left oid 9 (ObjNum 1364 snap 460 seq_num 1364) 2024-09-13T14:36:08.726 INFO:tasks.rados.rados.0.smithi151.stdout:3635: finishing write tid 1 to smithi15186934-7 2024-09-13T14:36:08.726 INFO:tasks.rados.rados.0.smithi151.stdout:3635: finishing write tid 2 to smithi15186934-7 2024-09-13T14:36:08.728 INFO:tasks.rados.rados.0.smithi151.stdout:3634: done (2 left) 2024-09-13T14:36:08.728 INFO:tasks.rados.rados.0.smithi151.stdout:3636: done (1 left) 2024-09-13T14:36:08.728 INFO:tasks.rados.rados.0.smithi151.stdout:3637: delete oid 31 current snap is 460 2024-09-13T14:36:08.729 INFO:tasks.rados.rados.0.smithi151.stdout:3635: finishing write tid 3 to smithi15186934-7 2024-09-13T14:36:08.729 INFO:tasks.rados.rados.0.smithi151.stdout:3635: finishing write tid 4 to smithi15186934-7 2024-09-13T14:36:08.729 INFO:tasks.rados.rados.0.smithi151.stdout:3635: finishing write tid 5 to smithi15186934-7 2024-09-13T14:36:08.729 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 2058 (ObjNum 1365 snap 460 seq_num 1365) dirty exists 2024-09-13T14:36:08.729 INFO:tasks.rados.rados.0.smithi151.stdout:3635: left oid 7 (ObjNum 1365 snap 460 seq_num 1365) 2024-09-13T14:36:08.730 INFO:tasks.rados.rados.0.smithi151.stdout:3635: done (1 left) 2024-09-13T14:36:08.730 INFO:tasks.rados.rados.0.smithi151.stdout:3637: done (0 left) 2024-09-13T14:36:08.730 INFO:tasks.rados.rados.0.smithi151.stdout:3638: snap_create 2024-09-13T14:36:09.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:08 smithi184 ceph-mon[92527]: pgmap v1083: 89 pgs: 89 active+clean; 254 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 11 op/s 2024-09-13T14:36:09.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:08 smithi184 ceph-mon[92527]: osdmap e977: 8 total, 8 up, 8 in 2024-09-13T14:36:09.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:08 smithi151 ceph-mon[108966]: pgmap v1083: 89 pgs: 89 active+clean; 254 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 11 op/s 2024-09-13T14:36:09.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:08 smithi151 ceph-mon[108966]: osdmap e977: 8 total, 8 up, 8 in 2024-09-13T14:36:09.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:08 smithi151 ceph-mon[104450]: pgmap v1083: 89 pgs: 89 active+clean; 254 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 11 op/s 2024-09-13T14:36:09.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:08 smithi151 ceph-mon[104450]: osdmap e977: 8 total, 8 up, 8 in 2024-09-13T14:36:09.651 INFO:tasks.rados.rados.0.smithi151.stdout:3638: done (0 left) 2024-09-13T14:36:09.651 INFO:tasks.rados.rados.0.smithi151.stdout:3639: snap_remove snap 458 2024-09-13T14:36:10.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:09 smithi184 ceph-mon[92527]: osdmap e978: 8 total, 8 up, 8 in 2024-09-13T14:36:10.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:09 smithi184 ceph-mon[92527]: osdmap e979: 8 total, 8 up, 8 in 2024-09-13T14:36:10.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:09 smithi151 ceph-mon[108966]: osdmap e978: 8 total, 8 up, 8 in 2024-09-13T14:36:10.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:09 smithi151 ceph-mon[108966]: osdmap e979: 8 total, 8 up, 8 in 2024-09-13T14:36:10.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:09 smithi151 ceph-mon[104450]: osdmap e978: 8 total, 8 up, 8 in 2024-09-13T14:36:10.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:09 smithi151 ceph-mon[104450]: osdmap e979: 8 total, 8 up, 8 in 2024-09-13T14:36:10.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:36:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:36:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:36:10.652 INFO:tasks.rados.rados.0.smithi151.stdout:3639: done (0 left) 2024-09-13T14:36:10.652 INFO:tasks.rados.rados.0.smithi151.stdout:3640: rollback oid 39 current snap is 461 2024-09-13T14:36:10.652 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 39 to 460 2024-09-13T14:36:10.652 INFO:tasks.rados.rados.0.smithi151.stdout:3641: delete oid 14 current snap is 461 2024-09-13T14:36:10.655 INFO:tasks.rados.rados.0.smithi151.stdout:3641: done (1 left) 2024-09-13T14:36:10.655 INFO:tasks.rados.rados.0.smithi151.stdout:3642: snap_create 2024-09-13T14:36:10.658 INFO:tasks.rados.rados.0.smithi151.stdout:3640: finishing rollback tid 0 to smithi15186934-39 2024-09-13T14:36:10.658 INFO:tasks.rados.rados.0.smithi151.stdout:3640: finishing rollback tid 1 to smithi15186934-39 2024-09-13T14:36:10.659 INFO:tasks.rados.rados.0.smithi151.stdout:3640: finishing rollback tid 2 to smithi15186934-39 2024-09-13T14:36:10.659 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 1525 (ObjNum 1324 snap 450 seq_num 1324) dirty exists 2024-09-13T14:36:11.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:10 smithi184 ceph-mon[92527]: pgmap v1086: 89 pgs: 89 active+clean; 253 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 7.8 MiB/s rd, 1.8 MiB/s wr, 15 op/s 2024-09-13T14:36:11.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:10 smithi184 ceph-mon[92527]: osdmap e980: 8 total, 8 up, 8 in 2024-09-13T14:36:11.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:10 smithi151 ceph-mon[104450]: pgmap v1086: 89 pgs: 89 active+clean; 253 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 7.8 MiB/s rd, 1.8 MiB/s wr, 15 op/s 2024-09-13T14:36:11.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:10 smithi151 ceph-mon[104450]: osdmap e980: 8 total, 8 up, 8 in 2024-09-13T14:36:11.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:10 smithi151 ceph-mon[108966]: pgmap v1086: 89 pgs: 89 active+clean; 253 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 7.8 MiB/s rd, 1.8 MiB/s wr, 15 op/s 2024-09-13T14:36:11.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:10 smithi151 ceph-mon[108966]: osdmap e980: 8 total, 8 up, 8 in 2024-09-13T14:36:11.656 INFO:tasks.rados.rados.0.smithi151.stdout:3640: done (1 left) 2024-09-13T14:36:11.656 INFO:tasks.rados.rados.0.smithi151.stdout:3642: done (0 left) 2024-09-13T14:36:11.657 INFO:tasks.rados.rados.0.smithi151.stdout:3643: read oid 43 snap 454 2024-09-13T14:36:11.657 INFO:tasks.rados.rados.0.smithi151.stdout:3643: expect deleted 2024-09-13T14:36:11.657 INFO:tasks.rados.rados.0.smithi151.stdout:3644: read oid 9 snap -1 2024-09-13T14:36:11.657 INFO:tasks.rados.rados.0.smithi151.stdout:3644: expect (ObjNum 1364 snap 460 seq_num 1364) 2024-09-13T14:36:11.657 INFO:tasks.rados.rados.0.smithi151.stdout:3645: read oid 10 snap 455 2024-09-13T14:36:11.657 INFO:tasks.rados.rados.0.smithi151.stdout:3645: expect (ObjNum 1306 snap 445 seq_num 1306) 2024-09-13T14:36:11.658 INFO:tasks.rados.rados.0.smithi151.stdout:3646: snap_create 2024-09-13T14:36:11.660 INFO:tasks.rados.rados.0.smithi151.stdout:3644: expect (ObjNum 1364 snap 460 seq_num 1364) 2024-09-13T14:36:11.912 INFO:tasks.rados.rados.0.smithi151.stdout:3645: expect (ObjNum 1306 snap 445 seq_num 1306) 2024-09-13T14:36:12.654 INFO:tasks.rados.rados.0.smithi151.stdout:3643: done (3 left) 2024-09-13T14:36:12.654 INFO:tasks.rados.rados.0.smithi151.stdout:3644: done (2 left) 2024-09-13T14:36:12.655 INFO:tasks.rados.rados.0.smithi151.stdout:3645: done (1 left) 2024-09-13T14:36:12.655 INFO:tasks.rados.rados.0.smithi151.stdout:3646: done (0 left) 2024-09-13T14:36:12.655 INFO:tasks.rados.rados.0.smithi151.stdout:3647: read oid 34 snap -1 2024-09-13T14:36:12.655 INFO:tasks.rados.rados.0.smithi151.stdout:3647: expect deleted 2024-09-13T14:36:12.655 INFO:tasks.rados.rados.0.smithi151.stdout:3648: delete oid 28 current snap is 463 2024-09-13T14:36:12.664 INFO:tasks.rados.rados.0.smithi151.stdout:3647: done (1 left) 2024-09-13T14:36:12.664 INFO:tasks.rados.rados.0.smithi151.stdout:3648: done (0 left) 2024-09-13T14:36:12.664 INFO:tasks.rados.rados.0.smithi151.stdout:3649: write oid 5 current snap is 463 2024-09-13T14:36:12.664 INFO:tasks.rados.rados.0.smithi151.stdout:3649: seq_num 1370 ranges {625590=434670,1685625=681144,2822206=203384} 2024-09-13T14:36:12.669 INFO:tasks.rados.rados.0.smithi151.stdout:3649: writing smithi15186934-5 from 625590 to 1060260 tid 1 2024-09-13T14:36:12.677 INFO:tasks.rados.rados.0.smithi151.stdout:3649: writing smithi15186934-5 from 1685625 to 2366769 tid 2 2024-09-13T14:36:12.680 INFO:tasks.rados.rados.0.smithi151.stdout:3649: writing smithi15186934-5 from 2822206 to 3025590 tid 3 2024-09-13T14:36:12.681 INFO:tasks.rados.rados.0.smithi151.stdout:3650: read oid 15 snap -1 2024-09-13T14:36:12.681 INFO:tasks.rados.rados.0.smithi151.stdout:3650: expect (ObjNum 1310 snap 446 seq_num 1310) 2024-09-13T14:36:12.681 INFO:tasks.rados.rados.0.smithi151.stdout:3651: snap_remove snap 453 2024-09-13T14:36:12.923 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:12 smithi184 ceph-mon[92527]: pgmap v1089: 89 pgs: 89 active+clean; 258 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.1 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2024-09-13T14:36:12.923 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:12 smithi184 ceph-mon[92527]: osdmap e981: 8 total, 8 up, 8 in 2024-09-13T14:36:13.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:12 smithi151 ceph-mon[104450]: pgmap v1089: 89 pgs: 89 active+clean; 258 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.1 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2024-09-13T14:36:13.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:12 smithi151 ceph-mon[104450]: osdmap e981: 8 total, 8 up, 8 in 2024-09-13T14:36:13.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:12 smithi151 ceph-mon[108966]: pgmap v1089: 89 pgs: 89 active+clean; 258 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.1 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2024-09-13T14:36:13.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:12 smithi151 ceph-mon[108966]: osdmap e981: 8 total, 8 up, 8 in 2024-09-13T14:36:13.675 INFO:tasks.rados.rados.0.smithi151.stdout:3649: finishing write tid 1 to smithi15186934-5 2024-09-13T14:36:13.675 INFO:tasks.rados.rados.0.smithi151.stdout:3649: finishing write tid 2 to smithi15186934-5 2024-09-13T14:36:13.675 INFO:tasks.rados.rados.0.smithi151.stdout:3650: expect (ObjNum 1310 snap 446 seq_num 1310) 2024-09-13T14:36:13.742 INFO:tasks.rados.rados.0.smithi151.stdout:3649: finishing write tid 3 to smithi15186934-5 2024-09-13T14:36:13.742 INFO:tasks.rados.rados.0.smithi151.stdout:3649: finishing write tid 4 to smithi15186934-5 2024-09-13T14:36:13.742 INFO:tasks.rados.rados.0.smithi151.stdout:3649: finishing write tid 5 to smithi15186934-5 2024-09-13T14:36:13.742 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 1457 (ObjNum 1370 snap 463 seq_num 1370) dirty exists 2024-09-13T14:36:13.742 INFO:tasks.rados.rados.0.smithi151.stdout:3649: left oid 5 (ObjNum 1370 snap 463 seq_num 1370) 2024-09-13T14:36:13.743 INFO:tasks.rados.rados.0.smithi151.stdout:3649: done (2 left) 2024-09-13T14:36:13.743 INFO:tasks.rados.rados.0.smithi151.stdout:3650: done (1 left) 2024-09-13T14:36:13.743 INFO:tasks.rados.rados.0.smithi151.stdout:3651: done (0 left) 2024-09-13T14:36:13.743 INFO:tasks.rados.rados.0.smithi151.stdout:3652: snap_create 2024-09-13T14:36:14.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:13 smithi184 ceph-mon[92527]: osdmap e982: 8 total, 8 up, 8 in 2024-09-13T14:36:14.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:13 smithi151 ceph-mon[104450]: osdmap e982: 8 total, 8 up, 8 in 2024-09-13T14:36:14.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:13 smithi151 ceph-mon[108966]: osdmap e982: 8 total, 8 up, 8 in 2024-09-13T14:36:14.651 INFO:tasks.rados.rados.0.smithi151.stdout:3652: done (0 left) 2024-09-13T14:36:14.651 INFO:tasks.rados.rados.0.smithi151.stdout:3653: delete oid 6 current snap is 464 2024-09-13T14:36:14.658 INFO:tasks.rados.rados.0.smithi151.stdout:3653: done (0 left) 2024-09-13T14:36:14.658 INFO:tasks.rados.rados.0.smithi151.stdout:3654: delete oid 32 current snap is 464 2024-09-13T14:36:14.661 INFO:tasks.rados.rados.0.smithi151.stdout:3654: done (0 left) 2024-09-13T14:36:14.661 INFO:tasks.rados.rados.0.smithi151.stdout:3655: read oid 33 snap -1 2024-09-13T14:36:14.661 INFO:tasks.rados.rados.0.smithi151.stdout:3655: expect (ObjNum 1345 snap 454 seq_num 1345) 2024-09-13T14:36:14.661 INFO:tasks.rados.rados.0.smithi151.stdout:3656: write oid 16 current snap is 464 2024-09-13T14:36:14.661 INFO:tasks.rados.rados.0.smithi151.stdout:3656: seq_num 1373 ranges {676011=660417,1744718=428471,2788122=287889} 2024-09-13T14:36:14.668 INFO:tasks.rados.rados.0.smithi151.stdout:3656: writing smithi15186934-16 from 676011 to 1336428 tid 1 2024-09-13T14:36:14.673 INFO:tasks.rados.rados.0.smithi151.stdout:3656: writing smithi15186934-16 from 1744718 to 2173189 tid 2 2024-09-13T14:36:14.678 INFO:tasks.rados.rados.0.smithi151.stdout:3656: writing smithi15186934-16 from 2788122 to 3076011 tid 3 2024-09-13T14:36:14.678 INFO:tasks.rados.rados.0.smithi151.stdout:3657: snap_remove snap 462 2024-09-13T14:36:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:14 smithi151 ceph-mon[104450]: pgmap v1092: 89 pgs: 89 active+clean; 259 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.2 MiB/s wr, 8 op/s 2024-09-13T14:36:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:14 smithi151 ceph-mon[104450]: osdmap e983: 8 total, 8 up, 8 in 2024-09-13T14:36:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:14 smithi151 ceph-mon[104450]: osdmap e984: 8 total, 8 up, 8 in 2024-09-13T14:36:14.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:36:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:36:14] "GET /metrics HTTP/1.1" 200 36507 "" "Prometheus/2.51.0" 2024-09-13T14:36:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:14 smithi151 ceph-mon[108966]: pgmap v1092: 89 pgs: 89 active+clean; 259 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.2 MiB/s wr, 8 op/s 2024-09-13T14:36:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:14 smithi151 ceph-mon[108966]: osdmap e983: 8 total, 8 up, 8 in 2024-09-13T14:36:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:14 smithi151 ceph-mon[108966]: osdmap e984: 8 total, 8 up, 8 in 2024-09-13T14:36:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:14 smithi184 ceph-mon[92527]: pgmap v1092: 89 pgs: 89 active+clean; 259 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.2 MiB/s wr, 8 op/s 2024-09-13T14:36:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:14 smithi184 ceph-mon[92527]: osdmap e983: 8 total, 8 up, 8 in 2024-09-13T14:36:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:14 smithi184 ceph-mon[92527]: osdmap e984: 8 total, 8 up, 8 in 2024-09-13T14:36:15.681 INFO:tasks.rados.rados.0.smithi151.stdout:3657: done (2 left) 2024-09-13T14:36:15.681 INFO:tasks.rados.rados.0.smithi151.stdout:3658: snap_remove snap 451 2024-09-13T14:36:16.682 INFO:tasks.rados.rados.0.smithi151.stdout:3658: done (2 left) 2024-09-13T14:36:16.682 INFO:tasks.rados.rados.0.smithi151.stdout:3659: delete oid 48 current snap is 464 2024-09-13T14:36:16.682 INFO:tasks.rados.rados.0.smithi151.stdout:3655: expect (ObjNum 1345 snap 454 seq_num 1345) 2024-09-13T14:36:16.807 INFO:tasks.rados.rados.0.smithi151.stdout:3656: finishing write tid 1 to smithi15186934-16 2024-09-13T14:36:16.807 INFO:tasks.rados.rados.0.smithi151.stdout:3656: finishing write tid 2 to smithi15186934-16 2024-09-13T14:36:16.808 INFO:tasks.rados.rados.0.smithi151.stdout:3656: finishing write tid 3 to smithi15186934-16 2024-09-13T14:36:16.808 INFO:tasks.rados.rados.0.smithi151.stdout:3656: finishing write tid 4 to smithi15186934-16 2024-09-13T14:36:16.808 INFO:tasks.rados.rados.0.smithi151.stdout:3656: finishing write tid 5 to smithi15186934-16 2024-09-13T14:36:16.808 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 16 v 1054 (ObjNum 1373 snap 464 seq_num 1373) dirty exists 2024-09-13T14:36:16.808 INFO:tasks.rados.rados.0.smithi151.stdout:3656: left oid 16 (ObjNum 1373 snap 464 seq_num 1373) 2024-09-13T14:36:16.808 INFO:tasks.rados.rados.0.smithi151.stdout:3655: done (2 left) 2024-09-13T14:36:16.808 INFO:tasks.rados.rados.0.smithi151.stdout:3656: done (1 left) 2024-09-13T14:36:16.808 INFO:tasks.rados.rados.0.smithi151.stdout:3659: done (0 left) 2024-09-13T14:36:16.808 INFO:tasks.rados.rados.0.smithi151.stdout:3660: snap_create 2024-09-13T14:36:17.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:16 smithi184 ceph-mon[92527]: pgmap v1095: 89 pgs: 89 active+clean; 261 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 667 KiB/s wr, 7 op/s 2024-09-13T14:36:17.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:16 smithi184 ceph-mon[92527]: osdmap e985: 8 total, 8 up, 8 in 2024-09-13T14:36:17.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:17.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:16 smithi151 ceph-mon[104450]: pgmap v1095: 89 pgs: 89 active+clean; 261 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 667 KiB/s wr, 7 op/s 2024-09-13T14:36:17.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:16 smithi151 ceph-mon[104450]: osdmap e985: 8 total, 8 up, 8 in 2024-09-13T14:36:17.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:17.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:16 smithi151 ceph-mon[108966]: pgmap v1095: 89 pgs: 89 active+clean; 261 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 667 KiB/s wr, 7 op/s 2024-09-13T14:36:17.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:16 smithi151 ceph-mon[108966]: osdmap e985: 8 total, 8 up, 8 in 2024-09-13T14:36:17.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:17.692 INFO:tasks.rados.rados.0.smithi151.stdout:3660: done (0 left) 2024-09-13T14:36:17.692 INFO:tasks.rados.rados.0.smithi151.stdout:3661: read oid 47 snap -1 2024-09-13T14:36:17.692 INFO:tasks.rados.rados.0.smithi151.stdout:3661: expect (ObjNum 1344 snap 454 seq_num 1344) 2024-09-13T14:36:17.692 INFO:tasks.rados.rados.0.smithi151.stdout:3662: write oid 4 current snap is 465 2024-09-13T14:36:17.692 INFO:tasks.rados.rados.0.smithi151.stdout:3662: seq_num 1375 ranges {709625=410915,1745329=432140,2726615=383010} 2024-09-13T14:36:17.698 INFO:tasks.rados.rados.0.smithi151.stdout:3662: writing smithi15186934-4 from 709625 to 1120540 tid 1 2024-09-13T14:36:17.704 INFO:tasks.rados.rados.0.smithi151.stdout:3662: writing smithi15186934-4 from 1745329 to 2177469 tid 2 2024-09-13T14:36:17.710 INFO:tasks.rados.rados.0.smithi151.stdout:3662: writing smithi15186934-4 from 2726615 to 3109625 tid 3 2024-09-13T14:36:17.710 INFO:tasks.rados.rados.0.smithi151.stdout:3663: write oid 13 current snap is 465 2024-09-13T14:36:17.710 INFO:tasks.rados.rados.0.smithi151.stdout:3663: seq_num 1376 ranges {726432=486164,1803811=692151} 2024-09-13T14:36:17.715 INFO:tasks.rados.rados.0.smithi151.stdout:3663: writing smithi15186934-13 from 726432 to 1212596 tid 1 2024-09-13T14:36:17.724 INFO:tasks.rados.rados.0.smithi151.stdout:3663: writing smithi15186934-13 from 1803811 to 2495962 tid 2 2024-09-13T14:36:17.724 INFO:tasks.rados.rados.0.smithi151.stdout:3664: write oid 14 current snap is 465 2024-09-13T14:36:17.724 INFO:tasks.rados.rados.0.smithi151.stdout:3664: seq_num 1377 ranges {743239=677766,1978646=435809,2897814=245425} 2024-09-13T14:36:17.730 INFO:tasks.rados.rados.0.smithi151.stdout:3664: writing smithi15186934-14 from 743239 to 1421005 tid 1 2024-09-13T14:36:17.736 INFO:tasks.rados.rados.0.smithi151.stdout:3664: writing smithi15186934-14 from 1978646 to 2414455 tid 2 2024-09-13T14:36:17.740 INFO:tasks.rados.rados.0.smithi151.stdout:3664: writing smithi15186934-14 from 2897814 to 3143239 tid 3 2024-09-13T14:36:17.740 INFO:tasks.rados.rados.0.smithi151.stdout:3665: write oid 16 current snap is 465 2024-09-13T14:36:17.740 INFO:tasks.rados.rados.0.smithi151.stdout:3665: seq_num 1378 ranges {760046=753015,2037128=695820} 2024-09-13T14:36:17.748 INFO:tasks.rados.rados.0.smithi151.stdout:3665: writing smithi15186934-16 from 760046 to 1513061 tid 1 2024-09-13T14:36:17.755 INFO:tasks.rados.rados.0.smithi151.stdout:3665: writing smithi15186934-16 from 2037128 to 2732948 tid 2 2024-09-13T14:36:17.755 INFO:tasks.rados.rados.0.smithi151.stdout:3666: snap_create 2024-09-13T14:36:17.755 INFO:tasks.rados.rados.0.smithi151.stdout:3661: expect (ObjNum 1344 snap 454 seq_num 1344) 2024-09-13T14:36:17.839 INFO:tasks.rados.rados.0.smithi151.stdout:3662: finishing write tid 1 to smithi15186934-4 2024-09-13T14:36:17.839 INFO:tasks.rados.rados.0.smithi151.stdout:3662: finishing write tid 2 to smithi15186934-4 2024-09-13T14:36:17.839 INFO:tasks.rados.rados.0.smithi151.stdout:3662: finishing write tid 3 to smithi15186934-4 2024-09-13T14:36:17.839 INFO:tasks.rados.rados.0.smithi151.stdout:3662: finishing write tid 4 to smithi15186934-4 2024-09-13T14:36:17.839 INFO:tasks.rados.rados.0.smithi151.stdout:3662: finishing write tid 5 to smithi15186934-4 2024-09-13T14:36:17.840 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 1779 (ObjNum 1375 snap 465 seq_num 1375) dirty exists 2024-09-13T14:36:17.840 INFO:tasks.rados.rados.0.smithi151.stdout:3662: left oid 4 (ObjNum 1375 snap 465 seq_num 1375) 2024-09-13T14:36:17.840 INFO:tasks.rados.rados.0.smithi151.stdout:3663: finishing write tid 1 to smithi15186934-13 2024-09-13T14:36:17.840 INFO:tasks.rados.rados.0.smithi151.stdout:3663: finishing write tid 2 to smithi15186934-13 2024-09-13T14:36:17.840 INFO:tasks.rados.rados.0.smithi151.stdout:3663: finishing write tid 3 to smithi15186934-13 2024-09-13T14:36:17.840 INFO:tasks.rados.rados.0.smithi151.stdout:3663: finishing write tid 4 to smithi15186934-13 2024-09-13T14:36:17.840 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 13 v 1469 (ObjNum 1376 snap 465 seq_num 1376) dirty exists 2024-09-13T14:36:17.840 INFO:tasks.rados.rados.0.smithi151.stdout:3663: left oid 13 (ObjNum 1376 snap 465 seq_num 1376) 2024-09-13T14:36:17.840 INFO:tasks.rados.rados.0.smithi151.stdout:3664: finishing write tid 1 to smithi15186934-14 2024-09-13T14:36:17.840 INFO:tasks.rados.rados.0.smithi151.stdout:3664: finishing write tid 2 to smithi15186934-14 2024-09-13T14:36:17.840 INFO:tasks.rados.rados.0.smithi151.stdout:3664: finishing write tid 3 to smithi15186934-14 2024-09-13T14:36:17.840 INFO:tasks.rados.rados.0.smithi151.stdout:3664: finishing write tid 4 to smithi15186934-14 2024-09-13T14:36:17.840 INFO:tasks.rados.rados.0.smithi151.stdout:3664: finishing write tid 5 to smithi15186934-14 2024-09-13T14:36:17.841 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 1501 (ObjNum 1377 snap 465 seq_num 1377) dirty exists 2024-09-13T14:36:17.841 INFO:tasks.rados.rados.0.smithi151.stdout:3664: left oid 14 (ObjNum 1377 snap 465 seq_num 1377) 2024-09-13T14:36:17.841 INFO:tasks.rados.rados.0.smithi151.stdout:3665: finishing write tid 1 to smithi15186934-16 2024-09-13T14:36:17.841 INFO:tasks.rados.rados.0.smithi151.stdout:3665: finishing write tid 2 to smithi15186934-16 2024-09-13T14:36:17.841 INFO:tasks.rados.rados.0.smithi151.stdout:3665: finishing write tid 3 to smithi15186934-16 2024-09-13T14:36:17.841 INFO:tasks.rados.rados.0.smithi151.stdout:3665: finishing write tid 4 to smithi15186934-16 2024-09-13T14:36:17.841 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 16 v 1062 (ObjNum 1378 snap 465 seq_num 1378) dirty exists 2024-09-13T14:36:17.841 INFO:tasks.rados.rados.0.smithi151.stdout:3665: left oid 16 (ObjNum 1378 snap 465 seq_num 1378) 2024-09-13T14:36:18.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:17 smithi184 ceph-mon[92527]: osdmap e986: 8 total, 8 up, 8 in 2024-09-13T14:36:18.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:17 smithi151 ceph-mon[104450]: osdmap e986: 8 total, 8 up, 8 in 2024-09-13T14:36:18.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:17 smithi151 ceph-mon[108966]: osdmap e986: 8 total, 8 up, 8 in 2024-09-13T14:36:18.696 INFO:tasks.rados.rados.0.smithi151.stdout:3661: done (5 left) 2024-09-13T14:36:18.696 INFO:tasks.rados.rados.0.smithi151.stdout:3662: done (4 left) 2024-09-13T14:36:18.697 INFO:tasks.rados.rados.0.smithi151.stdout:3663: done (3 left) 2024-09-13T14:36:18.697 INFO:tasks.rados.rados.0.smithi151.stdout:3664: done (2 left) 2024-09-13T14:36:18.697 INFO:tasks.rados.rados.0.smithi151.stdout:3665: done (1 left) 2024-09-13T14:36:18.697 INFO:tasks.rados.rados.0.smithi151.stdout:3666: done (0 left) 2024-09-13T14:36:18.697 INFO:tasks.rados.rados.0.smithi151.stdout:3667: delete oid 14 current snap is 466 2024-09-13T14:36:18.701 INFO:tasks.rados.rados.0.smithi151.stdout:3667: done (0 left) 2024-09-13T14:36:18.701 INFO:tasks.rados.rados.0.smithi151.stdout:3668: rollback oid 7 current snap is 466 2024-09-13T14:36:18.701 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 7 to 428 2024-09-13T14:36:18.701 INFO:tasks.rados.rados.0.smithi151.stdout:3669: write oid 20 current snap is 466 2024-09-13T14:36:18.701 INFO:tasks.rados.rados.0.smithi151.stdout:3669: seq_num 1380 ranges {793660=503513,2037739=583136} 2024-09-13T14:36:18.708 INFO:tasks.rados.rados.0.smithi151.stdout:3669: writing smithi15186934-20 from 793660 to 1297173 tid 1 2024-09-13T14:36:18.715 INFO:tasks.rados.rados.0.smithi151.stdout:3669: writing smithi15186934-20 from 2037739 to 2620875 tid 2 2024-09-13T14:36:18.715 INFO:tasks.rados.rados.0.smithi151.stdout:3670: snap_remove snap 459 2024-09-13T14:36:19.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:18 smithi184 ceph-mon[92527]: pgmap v1098: 89 pgs: 89 active+clean; 258 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.7 MiB/s rd, 659 KiB/s wr, 10 op/s 2024-09-13T14:36:19.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:18 smithi184 ceph-mon[92527]: osdmap e987: 8 total, 8 up, 8 in 2024-09-13T14:36:19.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:18 smithi151 ceph-mon[104450]: pgmap v1098: 89 pgs: 89 active+clean; 258 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.7 MiB/s rd, 659 KiB/s wr, 10 op/s 2024-09-13T14:36:19.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:18 smithi151 ceph-mon[104450]: osdmap e987: 8 total, 8 up, 8 in 2024-09-13T14:36:19.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:18 smithi151 ceph-mon[108966]: pgmap v1098: 89 pgs: 89 active+clean; 258 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.7 MiB/s rd, 659 KiB/s wr, 10 op/s 2024-09-13T14:36:19.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:18 smithi151 ceph-mon[108966]: osdmap e987: 8 total, 8 up, 8 in 2024-09-13T14:36:19.653 INFO:tasks.rados.rados.0.smithi151.stdout:3670: done (2 left) 2024-09-13T14:36:19.653 INFO:tasks.rados.rados.0.smithi151.stdout:3671: write oid 18 current snap is 466 2024-09-13T14:36:19.653 INFO:tasks.rados.rados.0.smithi151.stdout:3668: finishing rollback tid 0 to smithi15186934-7 2024-09-13T14:36:19.653 INFO:tasks.rados.rados.0.smithi151.stdout:3671: seq_num 1381 ranges {410467=578762,1696221=443147,2891153=319314} 2024-09-13T14:36:19.659 INFO:tasks.rados.rados.0.smithi151.stdout:3671: writing smithi15186934-18 from 410467 to 989229 tid 1 2024-09-13T14:36:19.666 INFO:tasks.rados.rados.0.smithi151.stdout:3671: writing smithi15186934-18 from 1696221 to 2139368 tid 2 2024-09-13T14:36:19.672 INFO:tasks.rados.rados.0.smithi151.stdout:3671: writing smithi15186934-18 from 2891153 to 3210467 tid 3 2024-09-13T14:36:19.672 INFO:tasks.rados.rados.0.smithi151.stdout:3672: delete oid 36 current snap is 466 2024-09-13T14:36:19.673 INFO:tasks.rados.rados.0.smithi151.stdout:3668: finishing rollback tid 1 to smithi15186934-7 2024-09-13T14:36:19.673 INFO:tasks.rados.rados.0.smithi151.stdout:3668: finishing rollback tid 2 to smithi15186934-7 2024-09-13T14:36:19.673 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 7 v 2072 (ObjNum 1184 snap 402 seq_num 1184) dirty exists 2024-09-13T14:36:19.673 INFO:tasks.rados.rados.0.smithi151.stdout:3669: finishing write tid 1 to smithi15186934-20 2024-09-13T14:36:19.673 INFO:tasks.rados.rados.0.smithi151.stdout:3669: finishing write tid 2 to smithi15186934-20 2024-09-13T14:36:19.673 INFO:tasks.rados.rados.0.smithi151.stdout:3669: finishing write tid 3 to smithi15186934-20 2024-09-13T14:36:19.673 INFO:tasks.rados.rados.0.smithi151.stdout:3669: finishing write tid 4 to smithi15186934-20 2024-09-13T14:36:19.673 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 1066 (ObjNum 1380 snap 466 seq_num 1380) dirty exists 2024-09-13T14:36:19.673 INFO:tasks.rados.rados.0.smithi151.stdout:3669: left oid 20 (ObjNum 1380 snap 466 seq_num 1380) 2024-09-13T14:36:19.673 INFO:tasks.rados.rados.0.smithi151.stdout:3671: finishing write tid 1 to smithi15186934-18 2024-09-13T14:36:19.673 INFO:tasks.rados.rados.0.smithi151.stdout:3671: finishing write tid 2 to smithi15186934-18 2024-09-13T14:36:19.674 INFO:tasks.rados.rados.0.smithi151.stdout:3668: done (3 left) 2024-09-13T14:36:19.674 INFO:tasks.rados.rados.0.smithi151.stdout:3669: done (2 left) 2024-09-13T14:36:19.675 INFO:tasks.rados.rados.0.smithi151.stdout:3672: done (1 left) 2024-09-13T14:36:19.675 INFO:tasks.rados.rados.0.smithi151.stdout:3673: write oid 22 current snap is 466 2024-09-13T14:36:19.675 INFO:tasks.rados.rados.0.smithi151.stdout:3673: seq_num 1383 ranges {444081=729260,1813185=446816,2945999=298082} 2024-09-13T14:36:19.681 INFO:tasks.rados.rados.0.smithi151.stdout:3673: writing smithi15186934-22 from 444081 to 1173341 tid 1 2024-09-13T14:36:19.687 INFO:tasks.rados.rados.0.smithi151.stdout:3673: writing smithi15186934-22 from 1813185 to 2260001 tid 2 2024-09-13T14:36:19.693 INFO:tasks.rados.rados.0.smithi151.stdout:3673: writing smithi15186934-22 from 2945999 to 3244081 tid 3 2024-09-13T14:36:19.693 INFO:tasks.rados.rados.0.smithi151.stdout:3674: read oid 33 snap 413 2024-09-13T14:36:19.694 INFO:tasks.rados.rados.0.smithi151.stdout:3674: expect (ObjNum 1058 snap 357 seq_num 1058) 2024-09-13T14:36:19.694 INFO:tasks.rados.rados.0.smithi151.stdout:3675: rollback oid 44 current snap is 466 2024-09-13T14:36:19.694 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 44 to 464 2024-09-13T14:36:19.694 INFO:tasks.rados.rados.0.smithi151.stdout:3671: finishing write tid 3 to smithi15186934-18 2024-09-13T14:36:19.694 INFO:tasks.rados.rados.0.smithi151.stdout:3671: finishing write tid 4 to smithi15186934-18 2024-09-13T14:36:19.694 INFO:tasks.rados.rados.0.smithi151.stdout:3671: finishing write tid 5 to smithi15186934-18 2024-09-13T14:36:19.694 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 18 v 1008 (ObjNum 1381 snap 466 seq_num 1381) dirty exists 2024-09-13T14:36:19.695 INFO:tasks.rados.rados.0.smithi151.stdout:3671: left oid 18 (ObjNum 1381 snap 466 seq_num 1381) 2024-09-13T14:36:19.695 INFO:tasks.rados.rados.0.smithi151.stdout:3673: finishing write tid 1 to smithi15186934-22 2024-09-13T14:36:19.695 INFO:tasks.rados.rados.0.smithi151.stdout:3671: done (3 left) 2024-09-13T14:36:19.695 INFO:tasks.rados.rados.0.smithi151.stdout:3676: write oid 47 current snap is 466 2024-09-13T14:36:19.695 INFO:tasks.rados.rados.0.smithi151.stdout:3676: seq_num 1384 ranges {460888=520862,1471667=590474,2573422=455407} 2024-09-13T14:36:19.698 INFO:tasks.rados.rados.0.smithi151.stdout:3676: writing smithi15186934-47 from 460888 to 981750 tid 1 2024-09-13T14:36:19.705 INFO:tasks.rados.rados.0.smithi151.stdout:3676: writing smithi15186934-47 from 1471667 to 2062141 tid 2 2024-09-13T14:36:19.711 INFO:tasks.rados.rados.0.smithi151.stdout:3676: writing smithi15186934-47 from 2573422 to 3028829 tid 3 2024-09-13T14:36:19.711 INFO:tasks.rados.rados.0.smithi151.stdout:3677: read oid 41 snap -1 2024-09-13T14:36:19.711 INFO:tasks.rados.rados.0.smithi151.stdout:3677: expect (ObjNum 1363 snap 458 seq_num 1363) 2024-09-13T14:36:19.711 INFO:tasks.rados.rados.0.smithi151.stdout:3678: write oid 9 current snap is 466 2024-09-13T14:36:19.711 INFO:tasks.rados.rados.0.smithi151.stdout:3673: finishing write tid 2 to smithi15186934-22 2024-09-13T14:36:19.711 INFO:tasks.rados.rados.0.smithi151.stdout:3678: seq_num 1385 ranges {477695=596111,1530149=450485,2600845=666679} 2024-09-13T14:36:19.717 INFO:tasks.rados.rados.0.smithi151.stdout:3678: writing smithi15186934-9 from 477695 to 1073806 tid 1 2024-09-13T14:36:19.722 INFO:tasks.rados.rados.0.smithi151.stdout:3678: writing smithi15186934-9 from 1530149 to 1980634 tid 2 2024-09-13T14:36:19.730 INFO:tasks.rados.rados.0.smithi151.stdout:3678: writing smithi15186934-9 from 2600845 to 3267524 tid 3 2024-09-13T14:36:19.730 INFO:tasks.rados.rados.0.smithi151.stdout:3679: read oid 40 snap -1 2024-09-13T14:36:19.730 INFO:tasks.rados.rados.0.smithi151.stdout:3679: expect deleted 2024-09-13T14:36:19.730 INFO:tasks.rados.rados.0.smithi151.stdout:3680: snap_create 2024-09-13T14:36:19.730 INFO:tasks.rados.rados.0.smithi151.stdout:3673: finishing write tid 3 to smithi15186934-22 2024-09-13T14:36:19.730 INFO:tasks.rados.rados.0.smithi151.stdout:3673: finishing write tid 4 to smithi15186934-22 2024-09-13T14:36:19.730 INFO:tasks.rados.rados.0.smithi151.stdout:3673: finishing write tid 5 to smithi15186934-22 2024-09-13T14:36:19.730 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 22 v 753 (ObjNum 1383 snap 466 seq_num 1383) dirty exists 2024-09-13T14:36:19.730 INFO:tasks.rados.rados.0.smithi151.stdout:3673: left oid 22 (ObjNum 1383 snap 466 seq_num 1383) 2024-09-13T14:36:19.730 INFO:tasks.rados.rados.0.smithi151.stdout:3675: finishing rollback tid 0 to smithi15186934-44 2024-09-13T14:36:19.730 INFO:tasks.rados.rados.0.smithi151.stdout:3675: finishing rollback tid 1 to smithi15186934-44 2024-09-13T14:36:19.730 INFO:tasks.rados.rados.0.smithi151.stdout:3675: finishing rollback tid 2 to smithi15186934-44 2024-09-13T14:36:19.731 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 1186 (ObjNum 1342 snap 454 seq_num 1342) dirty exists 2024-09-13T14:36:19.731 INFO:tasks.rados.rados.0.smithi151.stdout:3674: expect (ObjNum 1058 snap 357 seq_num 1058) 2024-09-13T14:36:19.906 INFO:tasks.rados.rados.0.smithi151.stdout:3676: finishing write tid 1 to smithi15186934-47 2024-09-13T14:36:19.906 INFO:tasks.rados.rados.0.smithi151.stdout:3676: finishing write tid 2 to smithi15186934-47 2024-09-13T14:36:19.906 INFO:tasks.rados.rados.0.smithi151.stdout:3676: finishing write tid 3 to smithi15186934-47 2024-09-13T14:36:19.906 INFO:tasks.rados.rados.0.smithi151.stdout:3676: finishing write tid 4 to smithi15186934-47 2024-09-13T14:36:19.906 INFO:tasks.rados.rados.0.smithi151.stdout:3676: finishing write tid 5 to smithi15186934-47 2024-09-13T14:36:19.906 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 1689 (ObjNum 1384 snap 466 seq_num 1384) dirty exists 2024-09-13T14:36:19.906 INFO:tasks.rados.rados.0.smithi151.stdout:3676: left oid 47 (ObjNum 1384 snap 466 seq_num 1384) 2024-09-13T14:36:19.906 INFO:tasks.rados.rados.0.smithi151.stdout:3677: expect (ObjNum 1363 snap 458 seq_num 1363) 2024-09-13T14:36:20.031 INFO:tasks.rados.rados.0.smithi151.stdout:3678: finishing write tid 1 to smithi15186934-9 2024-09-13T14:36:20.031 INFO:tasks.rados.rados.0.smithi151.stdout:3678: finishing write tid 2 to smithi15186934-9 2024-09-13T14:36:20.032 INFO:tasks.rados.rados.0.smithi151.stdout:3678: finishing write tid 3 to smithi15186934-9 2024-09-13T14:36:20.032 INFO:tasks.rados.rados.0.smithi151.stdout:3678: finishing write tid 4 to smithi15186934-9 2024-09-13T14:36:20.032 INFO:tasks.rados.rados.0.smithi151.stdout:3678: finishing write tid 5 to smithi15186934-9 2024-09-13T14:36:20.032 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 1514 (ObjNum 1385 snap 466 seq_num 1385) dirty exists 2024-09-13T14:36:20.032 INFO:tasks.rados.rados.0.smithi151.stdout:3678: left oid 9 (ObjNum 1385 snap 466 seq_num 1385) 2024-09-13T14:36:20.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:19 smithi184 ceph-mon[92527]: osdmap e988: 8 total, 8 up, 8 in 2024-09-13T14:36:20.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:19 smithi184 ceph-mon[92527]: osdmap e989: 8 total, 8 up, 8 in 2024-09-13T14:36:20.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:19 smithi151 ceph-mon[104450]: osdmap e988: 8 total, 8 up, 8 in 2024-09-13T14:36:20.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:19 smithi151 ceph-mon[104450]: osdmap e989: 8 total, 8 up, 8 in 2024-09-13T14:36:20.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:19 smithi151 ceph-mon[108966]: osdmap e988: 8 total, 8 up, 8 in 2024-09-13T14:36:20.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:19 smithi151 ceph-mon[108966]: osdmap e989: 8 total, 8 up, 8 in 2024-09-13T14:36:20.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:36:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:36:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:36:20.710 INFO:tasks.rados.rados.0.smithi151.stdout:3673: done (7 left) 2024-09-13T14:36:20.710 INFO:tasks.rados.rados.0.smithi151.stdout:3674: done (6 left) 2024-09-13T14:36:20.710 INFO:tasks.rados.rados.0.smithi151.stdout:3675: done (5 left) 2024-09-13T14:36:20.711 INFO:tasks.rados.rados.0.smithi151.stdout:3676: done (4 left) 2024-09-13T14:36:20.711 INFO:tasks.rados.rados.0.smithi151.stdout:3677: done (3 left) 2024-09-13T14:36:20.711 INFO:tasks.rados.rados.0.smithi151.stdout:3678: done (2 left) 2024-09-13T14:36:20.711 INFO:tasks.rados.rados.0.smithi151.stdout:3679: done (1 left) 2024-09-13T14:36:20.711 INFO:tasks.rados.rados.0.smithi151.stdout:3680: done (0 left) 2024-09-13T14:36:20.711 INFO:tasks.rados.rados.0.smithi151.stdout:3681: write oid 16 current snap is 467 2024-09-13T14:36:20.711 INFO:tasks.rados.rados.0.smithi151.stdout:3681: seq_num 1386 ranges {494502=671360,1588631=594143,2628268=477951} 2024-09-13T14:36:20.716 INFO:tasks.rados.rados.0.smithi151.stdout:3681: writing smithi15186934-16 from 494502 to 1165862 tid 1 2024-09-13T14:36:20.722 INFO:tasks.rados.rados.0.smithi151.stdout:3681: writing smithi15186934-16 from 1588631 to 2182774 tid 2 2024-09-13T14:36:20.728 INFO:tasks.rados.rados.0.smithi151.stdout:3681: writing smithi15186934-16 from 2628268 to 3106219 tid 3 2024-09-13T14:36:20.728 INFO:tasks.rados.rados.0.smithi151.stdout:3682: snap_remove snap 455 2024-09-13T14:36:21.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:20 smithi184 ceph-mon[92527]: pgmap v1101: 89 pgs: 89 active+clean; 260 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.3 MiB/s rd, 624 KiB/s wr, 8 op/s 2024-09-13T14:36:21.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:20 smithi151 ceph-mon[104450]: pgmap v1101: 89 pgs: 89 active+clean; 260 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.3 MiB/s rd, 624 KiB/s wr, 8 op/s 2024-09-13T14:36:21.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:20 smithi151 ceph-mon[108966]: pgmap v1101: 89 pgs: 89 active+clean; 260 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.3 MiB/s rd, 624 KiB/s wr, 8 op/s 2024-09-13T14:36:21.718 INFO:tasks.rados.rados.0.smithi151.stdout:3682: done (1 left) 2024-09-13T14:36:21.718 INFO:tasks.rados.rados.0.smithi151.stdout:3683: delete oid 28 current snap is 467 2024-09-13T14:36:21.718 INFO:tasks.rados.rados.0.smithi151.stdout:3681: finishing write tid 1 to smithi15186934-16 2024-09-13T14:36:21.718 INFO:tasks.rados.rados.0.smithi151.stdout:3681: finishing write tid 2 to smithi15186934-16 2024-09-13T14:36:21.718 INFO:tasks.rados.rados.0.smithi151.stdout:3681: finishing write tid 3 to smithi15186934-16 2024-09-13T14:36:21.718 INFO:tasks.rados.rados.0.smithi151.stdout:3681: finishing write tid 4 to smithi15186934-16 2024-09-13T14:36:21.718 INFO:tasks.rados.rados.0.smithi151.stdout:3681: finishing write tid 5 to smithi15186934-16 2024-09-13T14:36:21.718 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 16 v 1075 (ObjNum 1386 snap 467 seq_num 1386) dirty exists 2024-09-13T14:36:21.719 INFO:tasks.rados.rados.0.smithi151.stdout:3681: left oid 16 (ObjNum 1386 snap 467 seq_num 1386) 2024-09-13T14:36:21.725 INFO:tasks.rados.rados.0.smithi151.stdout:3681: done (1 left) 2024-09-13T14:36:21.725 INFO:tasks.rados.rados.0.smithi151.stdout:3683: done (0 left) 2024-09-13T14:36:21.725 INFO:tasks.rados.rados.0.smithi151.stdout:3684: snap_create 2024-09-13T14:36:22.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:21 smithi184 ceph-mon[92527]: osdmap e990: 8 total, 8 up, 8 in 2024-09-13T14:36:22.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:21 smithi151 ceph-mon[108966]: osdmap e990: 8 total, 8 up, 8 in 2024-09-13T14:36:22.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:21 smithi151 ceph-mon[104450]: osdmap e990: 8 total, 8 up, 8 in 2024-09-13T14:36:22.722 INFO:tasks.rados.rados.0.smithi151.stdout:3684: done (0 left) 2024-09-13T14:36:22.722 INFO:tasks.rados.rados.0.smithi151.stdout:3685: rollback oid 30 current snap is 468 2024-09-13T14:36:22.723 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 30 to 464 2024-09-13T14:36:22.723 INFO:tasks.rados.rados.0.smithi151.stdout:3686: snap_remove snap 445 2024-09-13T14:36:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:22 smithi184 ceph-mon[92527]: pgmap v1104: 89 pgs: 89 active+clean; 266 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.3 MiB/s wr, 8 op/s 2024-09-13T14:36:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:22 smithi184 ceph-mon[92527]: osdmap e991: 8 total, 8 up, 8 in 2024-09-13T14:36:23.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:22 smithi151 ceph-mon[104450]: pgmap v1104: 89 pgs: 89 active+clean; 266 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.3 MiB/s wr, 8 op/s 2024-09-13T14:36:23.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:22 smithi151 ceph-mon[104450]: osdmap e991: 8 total, 8 up, 8 in 2024-09-13T14:36:23.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:22 smithi151 ceph-mon[108966]: pgmap v1104: 89 pgs: 89 active+clean; 266 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.3 MiB/s wr, 8 op/s 2024-09-13T14:36:23.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:22 smithi151 ceph-mon[108966]: osdmap e991: 8 total, 8 up, 8 in 2024-09-13T14:36:23.730 INFO:tasks.rados.rados.0.smithi151.stdout:3686: done (1 left) 2024-09-13T14:36:23.730 INFO:tasks.rados.rados.0.smithi151.stdout:3687: read oid 9 snap 465 2024-09-13T14:36:23.730 INFO:tasks.rados.rados.0.smithi151.stdout:3687: expect (ObjNum 1364 snap 460 seq_num 1364) 2024-09-13T14:36:23.730 INFO:tasks.rados.rados.0.smithi151.stdout:3688: rollback oid 3 current snap is 468 2024-09-13T14:36:23.730 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 3 to 467 2024-09-13T14:36:23.730 INFO:tasks.rados.rados.0.smithi151.stdout:3685: finishing rollback tid 1 to smithi15186934-30 2024-09-13T14:36:23.730 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 30 v 1350 (ObjNum -1202163088 snap 22067 seq_num 304) dirty dne 2024-09-13T14:36:23.731 INFO:tasks.rados.rados.0.smithi151.stdout:3685: done (2 left) 2024-09-13T14:36:23.731 INFO:tasks.rados.rados.0.smithi151.stdout:3689: rollback oid 33 current snap is 468 2024-09-13T14:36:23.731 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 33 to 413 2024-09-13T14:36:23.731 INFO:tasks.rados.rados.0.smithi151.stdout:3690: snap_remove snap 460 2024-09-13T14:36:24.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:23 smithi184 ceph-mon[92527]: osdmap e992: 8 total, 8 up, 8 in 2024-09-13T14:36:24.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:23 smithi151 ceph-mon[104450]: osdmap e992: 8 total, 8 up, 8 in 2024-09-13T14:36:24.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:23 smithi151 ceph-mon[108966]: osdmap e992: 8 total, 8 up, 8 in 2024-09-13T14:36:24.655 INFO:tasks.rados.rados.0.smithi151.stdout:3690: done (3 left) 2024-09-13T14:36:24.655 INFO:tasks.rados.rados.0.smithi151.stdout:3691: rollback oid 48 current snap is 468 2024-09-13T14:36:24.655 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 48 to 465 2024-09-13T14:36:24.655 INFO:tasks.rados.rados.0.smithi151.stdout:3692: read oid 50 snap 465 2024-09-13T14:36:24.655 INFO:tasks.rados.rados.0.smithi151.stdout:3692: expect (ObjNum 1103 snap 370 seq_num 1103) 2024-09-13T14:36:24.655 INFO:tasks.rados.rados.0.smithi151.stdout:3693: write oid 36 current snap is 468 2024-09-13T14:36:24.655 INFO:tasks.rados.rados.0.smithi151.stdout:3693: seq_num 1388 ranges {528116=421858,1589242=597812,2966761=361355} 2024-09-13T14:36:24.660 INFO:tasks.rados.rados.0.smithi151.stdout:3693: writing smithi15186934-36 from 528116 to 949974 tid 1 2024-09-13T14:36:24.669 INFO:tasks.rados.rados.0.smithi151.stdout:3693: writing smithi15186934-36 from 1589242 to 2187054 tid 2 2024-09-13T14:36:24.675 INFO:tasks.rados.rados.0.smithi151.stdout:3693: writing smithi15186934-36 from 2966761 to 3328116 tid 3 2024-09-13T14:36:24.675 INFO:tasks.rados.rados.0.smithi151.stdout:3694: write oid 14 current snap is 468 2024-09-13T14:36:24.675 INFO:tasks.rados.rados.0.smithi151.stdout:3694: seq_num 1389 ranges {544923=497107,1647724=457823,2710537=428120} 2024-09-13T14:36:24.681 INFO:tasks.rados.rados.0.smithi151.stdout:3694: writing smithi15186934-14 from 544923 to 1042030 tid 1 2024-09-13T14:36:24.687 INFO:tasks.rados.rados.0.smithi151.stdout:3694: writing smithi15186934-14 from 1647724 to 2105547 tid 2 2024-09-13T14:36:24.693 INFO:tasks.rados.rados.0.smithi151.stdout:3694: writing smithi15186934-14 from 2710537 to 3138657 tid 3 2024-09-13T14:36:24.693 INFO:tasks.rados.rados.0.smithi151.stdout:3695: write oid 4 current snap is 468 2024-09-13T14:36:24.693 INFO:tasks.rados.rados.0.smithi151.stdout:3695: seq_num 1390 ranges {561730=572356,1706206=601481,3021607=340123} 2024-09-13T14:36:24.699 INFO:tasks.rados.rados.0.smithi151.stdout:3695: writing smithi15186934-4 from 561730 to 1134086 tid 1 2024-09-13T14:36:24.705 INFO:tasks.rados.rados.0.smithi151.stdout:3695: writing smithi15186934-4 from 1706206 to 2307687 tid 2 2024-09-13T14:36:24.711 INFO:tasks.rados.rados.0.smithi151.stdout:3695: writing smithi15186934-4 from 3021607 to 3361730 tid 3 2024-09-13T14:36:24.711 INFO:tasks.rados.rados.0.smithi151.stdout:3696: read oid 19 snap 457 2024-09-13T14:36:24.711 INFO:tasks.rados.rados.0.smithi151.stdout:3696: expect (ObjNum 1330 snap 452 seq_num 1330) 2024-09-13T14:36:24.711 INFO:tasks.rados.rados.0.smithi151.stdout:3697: snap_remove snap 466 2024-09-13T14:36:24.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:24 smithi151 ceph-mon[104450]: pgmap v1107: 89 pgs: 89 active+clean; 281 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.9 MiB/s rd, 3.1 MiB/s wr, 18 op/s 2024-09-13T14:36:24.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:24 smithi151 ceph-mon[104450]: osdmap e993: 8 total, 8 up, 8 in 2024-09-13T14:36:24.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:24 smithi151 ceph-mon[104450]: osdmap e994: 8 total, 8 up, 8 in 2024-09-13T14:36:24.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:24 smithi151 ceph-mon[108966]: pgmap v1107: 89 pgs: 89 active+clean; 281 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.9 MiB/s rd, 3.1 MiB/s wr, 18 op/s 2024-09-13T14:36:24.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:24 smithi151 ceph-mon[108966]: osdmap e993: 8 total, 8 up, 8 in 2024-09-13T14:36:24.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:24 smithi151 ceph-mon[108966]: osdmap e994: 8 total, 8 up, 8 in 2024-09-13T14:36:24.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:36:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:36:24] "GET /metrics HTTP/1.1" 200 36513 "" "Prometheus/2.51.0" 2024-09-13T14:36:25.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:24 smithi184 ceph-mon[92527]: pgmap v1107: 89 pgs: 89 active+clean; 281 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.9 MiB/s rd, 3.1 MiB/s wr, 18 op/s 2024-09-13T14:36:25.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:24 smithi184 ceph-mon[92527]: osdmap e993: 8 total, 8 up, 8 in 2024-09-13T14:36:25.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:24 smithi184 ceph-mon[92527]: osdmap e994: 8 total, 8 up, 8 in 2024-09-13T14:36:25.656 INFO:tasks.rados.rados.0.smithi151.stdout:3697: done (9 left) 2024-09-13T14:36:25.656 INFO:tasks.rados.rados.0.smithi151.stdout:3698: delete oid 18 current snap is 468 2024-09-13T14:36:25.656 INFO:tasks.rados.rados.0.smithi151.stdout:3687: expect (ObjNum 1364 snap 460 seq_num 1364) 2024-09-13T14:36:25.908 INFO:tasks.rados.rados.0.smithi151.stdout:3688: finishing rollback tid 0 to smithi15186934-3 2024-09-13T14:36:25.908 INFO:tasks.rados.rados.0.smithi151.stdout:3688: finishing rollback tid 1 to smithi15186934-3 2024-09-13T14:36:25.908 INFO:tasks.rados.rados.0.smithi151.stdout:3688: finishing rollback tid 2 to smithi15186934-3 2024-09-13T14:36:25.908 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 1381 (ObjNum 1277 snap 437 seq_num 1277) dirty exists 2024-09-13T14:36:25.909 INFO:tasks.rados.rados.0.smithi151.stdout:3689: finishing rollback tid 0 to smithi15186934-33 2024-09-13T14:36:25.909 INFO:tasks.rados.rados.0.smithi151.stdout:3689: finishing rollback tid 1 to smithi15186934-33 2024-09-13T14:36:25.909 INFO:tasks.rados.rados.0.smithi151.stdout:3687: done (9 left) 2024-09-13T14:36:25.909 INFO:tasks.rados.rados.0.smithi151.stdout:3688: done (8 left) 2024-09-13T14:36:25.909 INFO:tasks.rados.rados.0.smithi151.stdout:3698: done (7 left) 2024-09-13T14:36:25.909 INFO:tasks.rados.rados.0.smithi151.stdout:3699: delete oid 49 current snap is 468 2024-09-13T14:36:25.909 INFO:tasks.rados.rados.0.smithi151.stdout:3689: finishing rollback tid 2 to smithi15186934-33 2024-09-13T14:36:25.909 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 33 v 1479 (ObjNum 1058 snap 357 seq_num 1058) dirty exists 2024-09-13T14:36:25.909 INFO:tasks.rados.rados.0.smithi151.stdout:3691: finishing rollback tid 1 to smithi15186934-48 2024-09-13T14:36:25.909 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 48 v 1770 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-13T14:36:25.909 INFO:tasks.rados.rados.0.smithi151.stdout:3693: finishing write tid 1 to smithi15186934-36 2024-09-13T14:36:25.909 INFO:tasks.rados.rados.0.smithi151.stdout:3692: expect (ObjNum 1103 snap 370 seq_num 1103) 2024-09-13T14:36:26.096 INFO:tasks.rados.rados.0.smithi151.stdout:3693: finishing write tid 2 to smithi15186934-36 2024-09-13T14:36:26.096 INFO:tasks.rados.rados.0.smithi151.stdout:3693: finishing write tid 3 to smithi15186934-36 2024-09-13T14:36:26.096 INFO:tasks.rados.rados.0.smithi151.stdout:3693: finishing write tid 4 to smithi15186934-36 2024-09-13T14:36:26.097 INFO:tasks.rados.rados.0.smithi151.stdout:3693: finishing write tid 5 to smithi15186934-36 2024-09-13T14:36:26.097 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 36 v 790 (ObjNum 1388 snap 468 seq_num 1388) dirty exists 2024-09-13T14:36:26.097 INFO:tasks.rados.rados.0.smithi151.stdout:3693: left oid 36 (ObjNum 1388 snap 468 seq_num 1388) 2024-09-13T14:36:26.097 INFO:tasks.rados.rados.0.smithi151.stdout:3694: finishing write tid 1 to smithi15186934-14 2024-09-13T14:36:26.097 INFO:tasks.rados.rados.0.smithi151.stdout:3689: done (7 left) 2024-09-13T14:36:26.097 INFO:tasks.rados.rados.0.smithi151.stdout:3691: done (6 left) 2024-09-13T14:36:26.097 INFO:tasks.rados.rados.0.smithi151.stdout:3692: done (5 left) 2024-09-13T14:36:26.097 INFO:tasks.rados.rados.0.smithi151.stdout:3693: done (4 left) 2024-09-13T14:36:26.097 INFO:tasks.rados.rados.0.smithi151.stdout:3699: done (3 left) 2024-09-13T14:36:26.097 INFO:tasks.rados.rados.0.smithi151.stdout:3700: read oid 50 snap -1 2024-09-13T14:36:26.097 INFO:tasks.rados.rados.0.smithi151.stdout:3700: expect (ObjNum 1103 snap 370 seq_num 1103) 2024-09-13T14:36:26.097 INFO:tasks.rados.rados.0.smithi151.stdout:3701: write oid 11 current snap is 468 2024-09-13T14:36:26.097 INFO:tasks.rados.rados.0.smithi151.stdout:3694: finishing write tid 2 to smithi15186934-14 2024-09-13T14:36:26.097 INFO:tasks.rados.rados.0.smithi151.stdout:3694: finishing write tid 3 to smithi15186934-14 2024-09-13T14:36:26.098 INFO:tasks.rados.rados.0.smithi151.stdout:3694: finishing write tid 4 to smithi15186934-14 2024-09-13T14:36:26.098 INFO:tasks.rados.rados.0.smithi151.stdout:3694: finishing write tid 5 to smithi15186934-14 2024-09-13T14:36:26.098 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 1534 (ObjNum 1389 snap 468 seq_num 1389) dirty exists 2024-09-13T14:36:26.098 INFO:tasks.rados.rados.0.smithi151.stdout:3694: left oid 14 (ObjNum 1389 snap 468 seq_num 1389) 2024-09-13T14:36:26.098 INFO:tasks.rados.rados.0.smithi151.stdout:3695: finishing write tid 1 to smithi15186934-4 2024-09-13T14:36:26.098 INFO:tasks.rados.rados.0.smithi151.stdout:3695: finishing write tid 2 to smithi15186934-4 2024-09-13T14:36:26.098 INFO:tasks.rados.rados.0.smithi151.stdout:3701: seq_num 1393 ranges {612151=514456,1881652=748808,3103876=308275} 2024-09-13T14:36:26.101 INFO:tasks.rados.rados.0.smithi151.stdout:3701: writing smithi15186934-11 from 612151 to 1126607 tid 1 2024-09-13T14:36:26.110 INFO:tasks.rados.rados.0.smithi151.stdout:3701: writing smithi15186934-11 from 1881652 to 2630460 tid 2 2024-09-13T14:36:26.114 INFO:tasks.rados.rados.0.smithi151.stdout:3701: writing smithi15186934-11 from 3103876 to 3412151 tid 3 2024-09-13T14:36:26.114 INFO:tasks.rados.rados.0.smithi151.stdout:3694: done (4 left) 2024-09-13T14:36:26.115 INFO:tasks.rados.rados.0.smithi151.stdout:3702: delete oid 28 current snap is 468 2024-09-13T14:36:26.115 INFO:tasks.rados.rados.0.smithi151.stdout:3696: expect (ObjNum 1330 snap 452 seq_num 1330) 2024-09-13T14:36:26.215 INFO:tasks.rados.rados.0.smithi151.stdout:3695: finishing write tid 3 to smithi15186934-4 2024-09-13T14:36:26.215 INFO:tasks.rados.rados.0.smithi151.stdout:3695: finishing write tid 4 to smithi15186934-4 2024-09-13T14:36:26.215 INFO:tasks.rados.rados.0.smithi151.stdout:3696: done (4 left) 2024-09-13T14:36:26.215 INFO:tasks.rados.rados.0.smithi151.stdout:3702: done (3 left) 2024-09-13T14:36:26.215 INFO:tasks.rados.rados.0.smithi151.stdout:3703: delete oid 12 current snap is 468 2024-09-13T14:36:26.215 INFO:tasks.rados.rados.0.smithi151.stdout:3695: finishing write tid 5 to smithi15186934-4 2024-09-13T14:36:26.215 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 1803 (ObjNum 1390 snap 468 seq_num 1390) dirty exists 2024-09-13T14:36:26.216 INFO:tasks.rados.rados.0.smithi151.stdout:3695: left oid 4 (ObjNum 1390 snap 468 seq_num 1390) 2024-09-13T14:36:26.216 INFO:tasks.rados.rados.0.smithi151.stdout:3700: expect (ObjNum 1103 snap 370 seq_num 1103) 2024-09-13T14:36:26.403 INFO:tasks.rados.rados.0.smithi151.stdout:3701: finishing write tid 1 to smithi15186934-11 2024-09-13T14:36:26.403 INFO:tasks.rados.rados.0.smithi151.stdout:3701: finishing write tid 2 to smithi15186934-11 2024-09-13T14:36:26.403 INFO:tasks.rados.rados.0.smithi151.stdout:3701: finishing write tid 3 to smithi15186934-11 2024-09-13T14:36:26.403 INFO:tasks.rados.rados.0.smithi151.stdout:3701: finishing write tid 4 to smithi15186934-11 2024-09-13T14:36:26.404 INFO:tasks.rados.rados.0.smithi151.stdout:3701: finishing write tid 5 to smithi15186934-11 2024-09-13T14:36:26.404 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 11 v 1020 (ObjNum 1393 snap 468 seq_num 1393) dirty exists 2024-09-13T14:36:26.404 INFO:tasks.rados.rados.0.smithi151.stdout:3701: left oid 11 (ObjNum 1393 snap 468 seq_num 1393) 2024-09-13T14:36:26.404 INFO:tasks.rados.rados.0.smithi151.stdout:3695: done (3 left) 2024-09-13T14:36:26.404 INFO:tasks.rados.rados.0.smithi151.stdout:3700: done (2 left) 2024-09-13T14:36:26.404 INFO:tasks.rados.rados.0.smithi151.stdout:3701: done (1 left) 2024-09-13T14:36:26.404 INFO:tasks.rados.rados.0.smithi151.stdout:3703: done (0 left) 2024-09-13T14:36:26.404 INFO:tasks.rados.rados.0.smithi151.stdout:3704: delete oid 40 current snap is 468 2024-09-13T14:36:26.405 INFO:tasks.rados.rados.0.smithi151.stdout:3704: done (0 left) 2024-09-13T14:36:26.405 INFO:tasks.rados.rados.0.smithi151.stdout:3705: snap_remove snap 461 2024-09-13T14:36:26.731 INFO:tasks.rados.rados.0.smithi151.stdout:3705: done (0 left) 2024-09-13T14:36:26.731 INFO:tasks.rados.rados.0.smithi151.stdout:3706: snap_remove snap 428 2024-09-13T14:36:27.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:26 smithi184 ceph-mon[92527]: pgmap v1110: 89 pgs: 1 active+clean+snaptrim, 88 active+clean; 281 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.9 MiB/s rd, 2.0 MiB/s wr, 15 op/s 2024-09-13T14:36:27.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:26 smithi184 ceph-mon[92527]: osdmap e995: 8 total, 8 up, 8 in 2024-09-13T14:36:27.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:26 smithi151 ceph-mon[104450]: pgmap v1110: 89 pgs: 1 active+clean+snaptrim, 88 active+clean; 281 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.9 MiB/s rd, 2.0 MiB/s wr, 15 op/s 2024-09-13T14:36:27.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:26 smithi151 ceph-mon[104450]: osdmap e995: 8 total, 8 up, 8 in 2024-09-13T14:36:27.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:26 smithi151 ceph-mon[108966]: pgmap v1110: 89 pgs: 1 active+clean+snaptrim, 88 active+clean; 281 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.9 MiB/s rd, 2.0 MiB/s wr, 15 op/s 2024-09-13T14:36:27.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:26 smithi151 ceph-mon[108966]: osdmap e995: 8 total, 8 up, 8 in 2024-09-13T14:36:27.739 INFO:tasks.rados.rados.0.smithi151.stdout:3706: done (0 left) 2024-09-13T14:36:27.740 INFO:tasks.rados.rados.0.smithi151.stdout:3707: write oid 31 current snap is 468 2024-09-13T14:36:27.740 INFO:tasks.rados.rados.0.smithi151.stdout:3707: seq_num 1397 ranges {679379=415452,1599227=756146,3097215=382164} 2024-09-13T14:36:27.746 INFO:tasks.rados.rados.0.smithi151.stdout:3707: writing smithi15186934-31 from 679379 to 1094831 tid 1 2024-09-13T14:36:27.754 INFO:tasks.rados.rados.0.smithi151.stdout:3707: writing smithi15186934-31 from 1599227 to 2355373 tid 2 2024-09-13T14:36:27.761 INFO:tasks.rados.rados.0.smithi151.stdout:3707: writing smithi15186934-31 from 3097215 to 3479379 tid 3 2024-09-13T14:36:27.761 INFO:tasks.rados.rados.0.smithi151.stdout:3708: snap_create 2024-09-13T14:36:27.761 INFO:tasks.rados.rados.0.smithi151.stdout:3707: finishing write tid 1 to smithi15186934-31 2024-09-13T14:36:27.761 INFO:tasks.rados.rados.0.smithi151.stdout:3707: finishing write tid 2 to smithi15186934-31 2024-09-13T14:36:27.763 INFO:tasks.rados.rados.0.smithi151.stdout:3707: finishing write tid 3 to smithi15186934-31 2024-09-13T14:36:27.763 INFO:tasks.rados.rados.0.smithi151.stdout:3707: finishing write tid 4 to smithi15186934-31 2024-09-13T14:36:27.763 INFO:tasks.rados.rados.0.smithi151.stdout:3707: finishing write tid 5 to smithi15186934-31 2024-09-13T14:36:27.763 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 31 v 1393 (ObjNum 1397 snap 468 seq_num 1397) dirty exists 2024-09-13T14:36:27.764 INFO:tasks.rados.rados.0.smithi151.stdout:3707: left oid 31 (ObjNum 1397 snap 468 seq_num 1397) 2024-09-13T14:36:28.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:27 smithi184 ceph-mon[92527]: osdmap e996: 8 total, 8 up, 8 in 2024-09-13T14:36:28.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:27 smithi151 ceph-mon[104450]: osdmap e996: 8 total, 8 up, 8 in 2024-09-13T14:36:28.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:27 smithi151 ceph-mon[108966]: osdmap e996: 8 total, 8 up, 8 in 2024-09-13T14:36:28.747 INFO:tasks.rados.rados.0.smithi151.stdout:3707: done (1 left) 2024-09-13T14:36:28.747 INFO:tasks.rados.rados.0.smithi151.stdout:3708: done (0 left) 2024-09-13T14:36:28.747 INFO:tasks.rados.rados.0.smithi151.stdout:3709: rollback oid 14 current snap is 469 2024-09-13T14:36:28.748 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 14 to 467 2024-09-13T14:36:28.748 INFO:tasks.rados.rados.0.smithi151.stdout:3710: write oid 3 current snap is 469 2024-09-13T14:36:28.748 INFO:tasks.rados.rados.0.smithi151.stdout:3710: seq_num 1398 ranges {696186=490701,1657709=616157,2724638=562274} 2024-09-13T14:36:28.754 INFO:tasks.rados.rados.0.smithi151.stdout:3710: writing smithi15186934-3 from 696186 to 1186887 tid 1 2024-09-13T14:36:28.760 INFO:tasks.rados.rados.0.smithi151.stdout:3710: writing smithi15186934-3 from 1657709 to 2273866 tid 2 2024-09-13T14:36:28.767 INFO:tasks.rados.rados.0.smithi151.stdout:3710: writing smithi15186934-3 from 2724638 to 3286912 tid 3 2024-09-13T14:36:28.767 INFO:tasks.rados.rados.0.smithi151.stdout:3711: delete oid 10 current snap is 469 2024-09-13T14:36:28.767 INFO:tasks.rados.rados.0.smithi151.stdout:3709: finishing rollback tid 0 to smithi15186934-14 2024-09-13T14:36:28.767 INFO:tasks.rados.rados.0.smithi151.stdout:3709: finishing rollback tid 1 to smithi15186934-14 2024-09-13T14:36:28.767 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 1547 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-13T14:36:28.767 INFO:tasks.rados.rados.0.smithi151.stdout:3710: finishing write tid 1 to smithi15186934-3 2024-09-13T14:36:28.767 INFO:tasks.rados.rados.0.smithi151.stdout:3710: finishing write tid 2 to smithi15186934-3 2024-09-13T14:36:28.770 INFO:tasks.rados.rados.0.smithi151.stdout:3710: finishing write tid 3 to smithi15186934-3 2024-09-13T14:36:28.771 INFO:tasks.rados.rados.0.smithi151.stdout:3710: finishing write tid 4 to smithi15186934-3 2024-09-13T14:36:28.771 INFO:tasks.rados.rados.0.smithi151.stdout:3710: finishing write tid 5 to smithi15186934-3 2024-09-13T14:36:28.771 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 1395 (ObjNum 1398 snap 469 seq_num 1398) dirty exists 2024-09-13T14:36:28.771 INFO:tasks.rados.rados.0.smithi151.stdout:3710: left oid 3 (ObjNum 1398 snap 469 seq_num 1398) 2024-09-13T14:36:28.772 INFO:tasks.rados.rados.0.smithi151.stdout:3709: done (2 left) 2024-09-13T14:36:28.772 INFO:tasks.rados.rados.0.smithi151.stdout:3710: done (1 left) 2024-09-13T14:36:28.772 INFO:tasks.rados.rados.0.smithi151.stdout:3711: done (0 left) 2024-09-13T14:36:28.772 INFO:tasks.rados.rados.0.smithi151.stdout:3712: snap_create 2024-09-13T14:36:29.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:28 smithi184 ceph-mon[92527]: pgmap v1113: 89 pgs: 1 active+clean+snaptrim, 88 active+clean; 282 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 1.0 MiB/s wr, 16 op/s 2024-09-13T14:36:29.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:28 smithi184 ceph-mon[92527]: osdmap e997: 8 total, 8 up, 8 in 2024-09-13T14:36:29.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:28 smithi151 ceph-mon[104450]: pgmap v1113: 89 pgs: 1 active+clean+snaptrim, 88 active+clean; 282 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 1.0 MiB/s wr, 16 op/s 2024-09-13T14:36:29.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:28 smithi151 ceph-mon[104450]: osdmap e997: 8 total, 8 up, 8 in 2024-09-13T14:36:29.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:28 smithi151 ceph-mon[108966]: pgmap v1113: 89 pgs: 1 active+clean+snaptrim, 88 active+clean; 282 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 1.0 MiB/s wr, 16 op/s 2024-09-13T14:36:29.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:28 smithi151 ceph-mon[108966]: osdmap e997: 8 total, 8 up, 8 in 2024-09-13T14:36:29.656 INFO:tasks.rados.rados.0.smithi151.stdout:3712: done (0 left) 2024-09-13T14:36:29.656 INFO:tasks.rados.rados.0.smithi151.stdout:3713: read oid 18 snap -1 2024-09-13T14:36:29.656 INFO:tasks.rados.rados.0.smithi151.stdout:3713: expect deleted 2024-09-13T14:36:29.656 INFO:tasks.rados.rados.0.smithi151.stdout:3714: rollback oid 32 current snap is 470 2024-09-13T14:36:29.656 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 32 to 468 2024-09-13T14:36:29.656 INFO:tasks.rados.rados.0.smithi151.stdout:3715: write oid 8 current snap is 470 2024-09-13T14:36:29.657 INFO:tasks.rados.rados.0.smithi151.stdout:3715: seq_num 1400 ranges {729800=757552,2174673=619826} 2024-09-13T14:36:29.664 INFO:tasks.rados.rados.0.smithi151.stdout:3715: writing smithi15186934-8 from 729800 to 1487352 tid 1 2024-09-13T14:36:29.671 INFO:tasks.rados.rados.0.smithi151.stdout:3715: writing smithi15186934-8 from 2174673 to 2794499 tid 2 2024-09-13T14:36:29.671 INFO:tasks.rados.rados.0.smithi151.stdout:3716: read oid 16 snap -1 2024-09-13T14:36:29.672 INFO:tasks.rados.rados.0.smithi151.stdout:3716: expect (ObjNum 1386 snap 467 seq_num 1386) 2024-09-13T14:36:29.672 INFO:tasks.rados.rados.0.smithi151.stdout:3717: delete oid 24 current snap is 470 2024-09-13T14:36:29.672 INFO:tasks.rados.rados.0.smithi151.stdout:3714: finishing rollback tid 1 to smithi15186934-32 2024-09-13T14:36:29.672 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 32 v 1489 (ObjNum 129 snap 0 seq_num 3092574704) dirty dne 2024-09-13T14:36:29.672 INFO:tasks.rados.rados.0.smithi151.stdout:3715: finishing write tid 1 to smithi15186934-8 2024-09-13T14:36:29.676 INFO:tasks.rados.rados.0.smithi151.stdout:3715: finishing write tid 2 to smithi15186934-8 2024-09-13T14:36:29.676 INFO:tasks.rados.rados.0.smithi151.stdout:3715: finishing write tid 3 to smithi15186934-8 2024-09-13T14:36:29.676 INFO:tasks.rados.rados.0.smithi151.stdout:3715: finishing write tid 4 to smithi15186934-8 2024-09-13T14:36:29.676 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 1208 (ObjNum 1400 snap 470 seq_num 1400) dirty exists 2024-09-13T14:36:29.676 INFO:tasks.rados.rados.0.smithi151.stdout:3715: left oid 8 (ObjNum 1400 snap 470 seq_num 1400) 2024-09-13T14:36:29.677 INFO:tasks.rados.rados.0.smithi151.stdout:3713: done (4 left) 2024-09-13T14:36:29.677 INFO:tasks.rados.rados.0.smithi151.stdout:3714: done (3 left) 2024-09-13T14:36:29.678 INFO:tasks.rados.rados.0.smithi151.stdout:3715: done (2 left) 2024-09-13T14:36:29.678 INFO:tasks.rados.rados.0.smithi151.stdout:3717: done (1 left) 2024-09-13T14:36:29.678 INFO:tasks.rados.rados.0.smithi151.stdout:3718: rollback oid 14 current snap is 470 2024-09-13T14:36:29.678 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 14 to 467 2024-09-13T14:36:29.678 INFO:tasks.rados.rados.0.smithi151.stdout:3719: rollback oid 10 current snap is 470 2024-09-13T14:36:29.678 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 10 to 464 2024-09-13T14:36:29.678 INFO:tasks.rados.rados.0.smithi151.stdout:3720: write oid 28 current snap is 470 2024-09-13T14:36:29.678 INFO:tasks.rados.rados.0.smithi151.stdout:3720: seq_num 1402 ranges {763414=508050,1891637=623495,3234330=329084} 2024-09-13T14:36:29.685 INFO:tasks.rados.rados.0.smithi151.stdout:3720: writing smithi15186934-28 from 763414 to 1271464 tid 1 2024-09-13T14:36:29.692 INFO:tasks.rados.rados.0.smithi151.stdout:3720: writing smithi15186934-28 from 1891637 to 2515132 tid 2 2024-09-13T14:36:29.698 INFO:tasks.rados.rados.0.smithi151.stdout:3720: writing smithi15186934-28 from 3234330 to 3563414 tid 3 2024-09-13T14:36:29.698 INFO:tasks.rados.rados.0.smithi151.stdout:3721: read oid 5 snap -1 2024-09-13T14:36:29.698 INFO:tasks.rados.rados.0.smithi151.stdout:3721: expect (ObjNum 1370 snap 463 seq_num 1370) 2024-09-13T14:36:29.698 INFO:tasks.rados.rados.0.smithi151.stdout:3722: write oid 2 current snap is 470 2024-09-13T14:36:29.698 INFO:tasks.rados.rados.0.smithi151.stdout:3722: seq_num 1403 ranges {780221=583299,1950119=767153,3261753=318468} 2024-09-13T14:36:29.705 INFO:tasks.rados.rados.0.smithi151.stdout:3722: writing smithi15186934-2 from 780221 to 1363520 tid 1 2024-09-13T14:36:29.712 INFO:tasks.rados.rados.0.smithi151.stdout:3722: writing smithi15186934-2 from 1950119 to 2717272 tid 2 2024-09-13T14:36:29.717 INFO:tasks.rados.rados.0.smithi151.stdout:3722: writing smithi15186934-2 from 3261753 to 3580221 tid 3 2024-09-13T14:36:29.718 INFO:tasks.rados.rados.0.smithi151.stdout:3723: delete oid 13 current snap is 470 2024-09-13T14:36:29.718 INFO:tasks.rados.rados.0.smithi151.stdout:3719: finishing rollback tid 1 to smithi15186934-10 2024-09-13T14:36:29.718 INFO:tasks.rados.rados.0.smithi151.stdout:3718: finishing rollback tid 1 to smithi15186934-14 2024-09-13T14:36:29.718 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 1547 (ObjNum 65 snap 0 seq_num 993990107) dirty dne 2024-09-13T14:36:29.718 INFO:tasks.rados.rados.0.smithi151.stdout:3719: finishing rollback tid 2 to smithi15186934-10 2024-09-13T14:36:29.718 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 10 v 1297 (ObjNum 1306 snap 445 seq_num 1306) dirty exists 2024-09-13T14:36:29.718 INFO:tasks.rados.rados.0.smithi151.stdout:3716: expect (ObjNum 1386 snap 467 seq_num 1386) 2024-09-13T14:36:29.910 INFO:tasks.rados.rados.0.smithi151.stdout:3720: finishing write tid 1 to smithi15186934-28 2024-09-13T14:36:29.911 INFO:tasks.rados.rados.0.smithi151.stdout:3720: finishing write tid 2 to smithi15186934-28 2024-09-13T14:36:29.911 INFO:tasks.rados.rados.0.smithi151.stdout:3721: expect (ObjNum 1370 snap 463 seq_num 1370) 2024-09-13T14:36:30.010 INFO:tasks.rados.rados.0.smithi151.stdout:3720: finishing write tid 3 to smithi15186934-28 2024-09-13T14:36:30.010 INFO:tasks.rados.rados.0.smithi151.stdout:3720: finishing write tid 4 to smithi15186934-28 2024-09-13T14:36:30.010 INFO:tasks.rados.rados.0.smithi151.stdout:3720: finishing write tid 5 to smithi15186934-28 2024-09-13T14:36:30.010 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 28 v 1397 (ObjNum 1402 snap 470 seq_num 1402) dirty exists 2024-09-13T14:36:30.010 INFO:tasks.rados.rados.0.smithi151.stdout:3720: left oid 28 (ObjNum 1402 snap 470 seq_num 1402) 2024-09-13T14:36:30.011 INFO:tasks.rados.rados.0.smithi151.stdout:3722: finishing write tid 1 to smithi15186934-2 2024-09-13T14:36:30.011 INFO:tasks.rados.rados.0.smithi151.stdout:3722: finishing write tid 2 to smithi15186934-2 2024-09-13T14:36:30.011 INFO:tasks.rados.rados.0.smithi151.stdout:3722: finishing write tid 3 to smithi15186934-2 2024-09-13T14:36:30.011 INFO:tasks.rados.rados.0.smithi151.stdout:3722: finishing write tid 4 to smithi15186934-2 2024-09-13T14:36:30.011 INFO:tasks.rados.rados.0.smithi151.stdout:3716: done (6 left) 2024-09-13T14:36:30.011 INFO:tasks.rados.rados.0.smithi151.stdout:3718: done (5 left) 2024-09-13T14:36:30.011 INFO:tasks.rados.rados.0.smithi151.stdout:3719: done (4 left) 2024-09-13T14:36:30.011 INFO:tasks.rados.rados.0.smithi151.stdout:3720: done (3 left) 2024-09-13T14:36:30.011 INFO:tasks.rados.rados.0.smithi151.stdout:3721: done (2 left) 2024-09-13T14:36:30.011 INFO:tasks.rados.rados.0.smithi151.stdout:3723: done (1 left) 2024-09-13T14:36:30.011 INFO:tasks.rados.rados.0.smithi151.stdout:3724: rollback oid 17 current snap is 470 2024-09-13T14:36:30.011 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 17 to 465 2024-09-13T14:36:30.012 INFO:tasks.rados.rados.0.smithi151.stdout:3722: finishing write tid 5 to smithi15186934-2 2024-09-13T14:36:30.012 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 2 v 1501 (ObjNum 1403 snap 470 seq_num 1403) dirty exists 2024-09-13T14:36:30.012 INFO:tasks.rados.rados.0.smithi151.stdout:3722: left oid 2 (ObjNum 1403 snap 470 seq_num 1403) 2024-09-13T14:36:30.012 INFO:tasks.rados.rados.0.smithi151.stdout:3722: done (1 left) 2024-09-13T14:36:30.012 INFO:tasks.rados.rados.0.smithi151.stdout:3725: snap_remove snap 447 2024-09-13T14:36:30.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:29 smithi184 ceph-mon[92527]: osdmap e998: 8 total, 8 up, 8 in 2024-09-13T14:36:30.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:29 smithi184 ceph-mon[92527]: osdmap e999: 8 total, 8 up, 8 in 2024-09-13T14:36:30.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:29 smithi151 ceph-mon[104450]: osdmap e998: 8 total, 8 up, 8 in 2024-09-13T14:36:30.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:29 smithi151 ceph-mon[104450]: osdmap e999: 8 total, 8 up, 8 in 2024-09-13T14:36:30.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:29 smithi151 ceph-mon[108966]: osdmap e998: 8 total, 8 up, 8 in 2024-09-13T14:36:30.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:29 smithi151 ceph-mon[108966]: osdmap e999: 8 total, 8 up, 8 in 2024-09-13T14:36:30.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:36:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:36:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:36:30.758 INFO:tasks.rados.rados.0.smithi151.stdout:3725: done (1 left) 2024-09-13T14:36:30.758 INFO:tasks.rados.rados.0.smithi151.stdout:3726: read oid 40 snap -1 2024-09-13T14:36:30.758 INFO:tasks.rados.rados.0.smithi151.stdout:3726: expect deleted 2024-09-13T14:36:30.758 INFO:tasks.rados.rados.0.smithi151.stdout:3727: snap_remove snap 454 2024-09-13T14:36:31.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:30 smithi184 ceph-mon[92527]: pgmap v1116: 89 pgs: 89 active+clean; 286 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.4 MiB/s wr, 15 op/s 2024-09-13T14:36:31.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:30 smithi151 ceph-mon[104450]: pgmap v1116: 89 pgs: 89 active+clean; 286 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.4 MiB/s wr, 15 op/s 2024-09-13T14:36:31.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:30 smithi151 ceph-mon[108966]: pgmap v1116: 89 pgs: 89 active+clean; 286 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.4 MiB/s wr, 15 op/s 2024-09-13T14:36:31.769 INFO:tasks.rados.rados.0.smithi151.stdout:3724: finishing rollback tid 0 to smithi15186934-17 2024-09-13T14:36:31.769 INFO:tasks.rados.rados.0.smithi151.stdout:3724: finishing rollback tid 1 to smithi15186934-17 2024-09-13T14:36:31.769 INFO:tasks.rados.rados.0.smithi151.stdout:3724: finishing rollback tid 2 to smithi15186934-17 2024-09-13T14:36:31.769 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 2096 (ObjNum 1337 snap 454 seq_num 1337) dirty exists 2024-09-13T14:36:31.770 INFO:tasks.rados.rados.0.smithi151.stdout:3724: done (2 left) 2024-09-13T14:36:31.770 INFO:tasks.rados.rados.0.smithi151.stdout:3726: done (1 left) 2024-09-13T14:36:31.770 INFO:tasks.rados.rados.0.smithi151.stdout:3727: done (0 left) 2024-09-13T14:36:31.770 INFO:tasks.rados.rados.0.smithi151.stdout:3728: delete oid 1 current snap is 470 2024-09-13T14:36:31.777 INFO:tasks.rados.rados.0.smithi151.stdout:3728: done (0 left) 2024-09-13T14:36:31.777 INFO:tasks.rados.rados.0.smithi151.stdout:3729: snap_create 2024-09-13T14:36:32.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:31 smithi184 ceph-mon[92527]: osdmap e1000: 8 total, 8 up, 8 in 2024-09-13T14:36:32.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:32.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:31 smithi151 ceph-mon[104450]: osdmap e1000: 8 total, 8 up, 8 in 2024-09-13T14:36:32.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:32.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:31 smithi151 ceph-mon[108966]: osdmap e1000: 8 total, 8 up, 8 in 2024-09-13T14:36:32.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:32.778 INFO:tasks.rados.rados.0.smithi151.stdout:3729: done (0 left) 2024-09-13T14:36:32.778 INFO:tasks.rados.rados.0.smithi151.stdout:3730: rollback oid 3 current snap is 471 2024-09-13T14:36:32.778 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 3 to 463 2024-09-13T14:36:32.778 INFO:tasks.rados.rados.0.smithi151.stdout:3731: rollback oid 16 current snap is 471 2024-09-13T14:36:32.778 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 16 to 465 2024-09-13T14:36:32.778 INFO:tasks.rados.rados.0.smithi151.stdout:3732: rollback oid 33 current snap is 471 2024-09-13T14:36:32.778 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 33 to 413 2024-09-13T14:36:32.779 INFO:tasks.rados.rados.0.smithi151.stdout:3733: write oid 27 current snap is 471 2024-09-13T14:36:32.779 INFO:tasks.rados.rados.0.smithi151.stdout:3733: seq_num 1406 ranges {430642=409046,1609212=514480,2827669=768803} 2024-09-13T14:36:32.783 INFO:tasks.rados.rados.0.smithi151.stdout:3733: writing smithi15186934-27 from 430642 to 839688 tid 1 2024-09-13T14:36:32.790 INFO:tasks.rados.rados.0.smithi151.stdout:3733: writing smithi15186934-27 from 1609212 to 2123692 tid 2 2024-09-13T14:36:32.799 INFO:tasks.rados.rados.0.smithi151.stdout:3733: writing smithi15186934-27 from 2827669 to 3596472 tid 3 2024-09-13T14:36:32.799 INFO:tasks.rados.rados.0.smithi151.stdout:3734: snap_create 2024-09-13T14:36:32.799 INFO:tasks.rados.rados.0.smithi151.stdout:3731: finishing rollback tid 0 to smithi15186934-16 2024-09-13T14:36:32.799 INFO:tasks.rados.rados.0.smithi151.stdout:3731: finishing rollback tid 1 to smithi15186934-16 2024-09-13T14:36:32.799 INFO:tasks.rados.rados.0.smithi151.stdout:3731: finishing rollback tid 2 to smithi15186934-16 2024-09-13T14:36:32.799 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 16 v 1107 (ObjNum 1378 snap 465 seq_num 1378) dirty exists 2024-09-13T14:36:32.799 INFO:tasks.rados.rados.0.smithi151.stdout:3730: finishing rollback tid 0 to smithi15186934-3 2024-09-13T14:36:32.799 INFO:tasks.rados.rados.0.smithi151.stdout:3730: finishing rollback tid 1 to smithi15186934-3 2024-09-13T14:36:32.799 INFO:tasks.rados.rados.0.smithi151.stdout:3730: finishing rollback tid 2 to smithi15186934-3 2024-09-13T14:36:32.799 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 1407 (ObjNum 1277 snap 437 seq_num 1277) dirty exists 2024-09-13T14:36:32.799 INFO:tasks.rados.rados.0.smithi151.stdout:3733: finishing write tid 1 to smithi15186934-27 2024-09-13T14:36:32.800 INFO:tasks.rados.rados.0.smithi151.stdout:3732: finishing rollback tid 0 to smithi15186934-33 2024-09-13T14:36:32.800 INFO:tasks.rados.rados.0.smithi151.stdout:3732: finishing rollback tid 1 to smithi15186934-33 2024-09-13T14:36:32.800 INFO:tasks.rados.rados.0.smithi151.stdout:3732: finishing rollback tid 2 to smithi15186934-33 2024-09-13T14:36:32.800 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 33 v 1519 (ObjNum 1058 snap 357 seq_num 1058) dirty exists 2024-09-13T14:36:32.800 INFO:tasks.rados.rados.0.smithi151.stdout:3733: finishing write tid 2 to smithi15186934-27 2024-09-13T14:36:32.803 INFO:tasks.rados.rados.0.smithi151.stdout:3733: finishing write tid 3 to smithi15186934-27 2024-09-13T14:36:32.803 INFO:tasks.rados.rados.0.smithi151.stdout:3733: finishing write tid 4 to smithi15186934-27 2024-09-13T14:36:32.803 INFO:tasks.rados.rados.0.smithi151.stdout:3733: finishing write tid 5 to smithi15186934-27 2024-09-13T14:36:32.803 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 27 v 1111 (ObjNum 1406 snap 471 seq_num 1406) dirty exists 2024-09-13T14:36:32.803 INFO:tasks.rados.rados.0.smithi151.stdout:3733: left oid 27 (ObjNum 1406 snap 471 seq_num 1406) 2024-09-13T14:36:33.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:32 smithi184 ceph-mon[92527]: pgmap v1119: 89 pgs: 89 active+clean; 286 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 1.5 MiB/s wr, 6 op/s 2024-09-13T14:36:33.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:32 smithi184 ceph-mon[92527]: osdmap e1001: 8 total, 8 up, 8 in 2024-09-13T14:36:33.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:32 smithi151 ceph-mon[104450]: pgmap v1119: 89 pgs: 89 active+clean; 286 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 1.5 MiB/s wr, 6 op/s 2024-09-13T14:36:33.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:32 smithi151 ceph-mon[104450]: osdmap e1001: 8 total, 8 up, 8 in 2024-09-13T14:36:33.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:32 smithi151 ceph-mon[108966]: pgmap v1119: 89 pgs: 89 active+clean; 286 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 1.5 MiB/s wr, 6 op/s 2024-09-13T14:36:33.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:32 smithi151 ceph-mon[108966]: osdmap e1001: 8 total, 8 up, 8 in 2024-09-13T14:36:33.787 INFO:tasks.rados.rados.0.smithi151.stdout:3730: done (4 left) 2024-09-13T14:36:33.787 INFO:tasks.rados.rados.0.smithi151.stdout:3731: done (3 left) 2024-09-13T14:36:33.787 INFO:tasks.rados.rados.0.smithi151.stdout:3732: done (2 left) 2024-09-13T14:36:33.787 INFO:tasks.rados.rados.0.smithi151.stdout:3733: done (1 left) 2024-09-13T14:36:33.788 INFO:tasks.rados.rados.0.smithi151.stdout:3734: done (0 left) 2024-09-13T14:36:33.788 INFO:tasks.rados.rados.0.smithi151.stdout:3735: snap_create 2024-09-13T14:36:34.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:33 smithi184 ceph-mon[92527]: osdmap e1002: 8 total, 8 up, 8 in 2024-09-13T14:36:34.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:33 smithi151 ceph-mon[104450]: osdmap e1002: 8 total, 8 up, 8 in 2024-09-13T14:36:34.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:33 smithi151 ceph-mon[108966]: osdmap e1002: 8 total, 8 up, 8 in 2024-09-13T14:36:34.659 INFO:tasks.rados.rados.0.smithi151.stdout:3735: done (0 left) 2024-09-13T14:36:34.659 INFO:tasks.rados.rados.0.smithi151.stdout:3736: write oid 6 current snap is 473 2024-09-13T14:36:34.659 INFO:tasks.rados.rados.0.smithi151.stdout:3736: seq_num 1407 ranges {447449=600648,1784047=774491,2971445=676004} 2024-09-13T14:36:34.664 INFO:tasks.rados.rados.0.smithi151.stdout:3736: writing smithi15186934-6 from 447449 to 1048097 tid 1 2024-09-13T14:36:34.673 INFO:tasks.rados.rados.0.smithi151.stdout:3736: writing smithi15186934-6 from 1784047 to 2558538 tid 2 2024-09-13T14:36:34.679 INFO:tasks.rados.rados.0.smithi151.stdout:3736: writing smithi15186934-6 from 2971445 to 3647449 tid 3 2024-09-13T14:36:34.679 INFO:tasks.rados.rados.0.smithi151.stdout:3737: read oid 23 snap -1 2024-09-13T14:36:34.679 INFO:tasks.rados.rados.0.smithi151.stdout:3737: expect (ObjNum 1359 snap 458 seq_num 1359) 2024-09-13T14:36:34.679 INFO:tasks.rados.rados.0.smithi151.stdout:3738: snap_remove snap 472 2024-09-13T14:36:34.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:36:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:36:34] "GET /metrics HTTP/1.1" 200 36512 "" "Prometheus/2.51.0" 2024-09-13T14:36:34.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:34 smithi151 ceph-mon[104450]: pgmap v1122: 89 pgs: 89 active+clean; 268 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.1 MiB/s wr, 7 op/s 2024-09-13T14:36:34.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:34 smithi151 ceph-mon[104450]: osdmap e1003: 8 total, 8 up, 8 in 2024-09-13T14:36:34.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:34 smithi151 ceph-mon[104450]: osdmap e1004: 8 total, 8 up, 8 in 2024-09-13T14:36:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:34 smithi151 ceph-mon[108966]: pgmap v1122: 89 pgs: 89 active+clean; 268 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.1 MiB/s wr, 7 op/s 2024-09-13T14:36:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:34 smithi151 ceph-mon[108966]: osdmap e1003: 8 total, 8 up, 8 in 2024-09-13T14:36:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:34 smithi151 ceph-mon[108966]: osdmap e1004: 8 total, 8 up, 8 in 2024-09-13T14:36:35.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:34 smithi184 ceph-mon[92527]: pgmap v1122: 89 pgs: 89 active+clean; 268 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 1.1 MiB/s wr, 7 op/s 2024-09-13T14:36:35.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:34 smithi184 ceph-mon[92527]: osdmap e1003: 8 total, 8 up, 8 in 2024-09-13T14:36:35.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:34 smithi184 ceph-mon[92527]: osdmap e1004: 8 total, 8 up, 8 in 2024-09-13T14:36:35.660 INFO:tasks.rados.rados.0.smithi151.stdout:3738: done (2 left) 2024-09-13T14:36:35.660 INFO:tasks.rados.rados.0.smithi151.stdout:3739: rollback oid 48 current snap is 473 2024-09-13T14:36:35.660 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 48 to 468 2024-09-13T14:36:35.660 INFO:tasks.rados.rados.0.smithi151.stdout:3736: finishing write tid 1 to smithi15186934-6 2024-09-13T14:36:35.660 INFO:tasks.rados.rados.0.smithi151.stdout:3736: finishing write tid 2 to smithi15186934-6 2024-09-13T14:36:35.660 INFO:tasks.rados.rados.0.smithi151.stdout:3736: finishing write tid 3 to smithi15186934-6 2024-09-13T14:36:35.660 INFO:tasks.rados.rados.0.smithi151.stdout:3736: finishing write tid 4 to smithi15186934-6 2024-09-13T14:36:35.660 INFO:tasks.rados.rados.0.smithi151.stdout:3736: finishing write tid 5 to smithi15186934-6 2024-09-13T14:36:35.660 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 6 v 845 (ObjNum 1407 snap 473 seq_num 1407) dirty exists 2024-09-13T14:36:35.660 INFO:tasks.rados.rados.0.smithi151.stdout:3736: left oid 6 (ObjNum 1407 snap 473 seq_num 1407) 2024-09-13T14:36:35.661 INFO:tasks.rados.rados.0.smithi151.stdout:3737: expect (ObjNum 1359 snap 458 seq_num 1359) 2024-09-13T14:36:35.751 INFO:tasks.rados.rados.0.smithi151.stdout:3739: finishing rollback tid 1 to smithi15186934-48 2024-09-13T14:36:35.752 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 48 v 1770 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-13T14:36:35.752 INFO:tasks.rados.rados.0.smithi151.stdout:3736: done (2 left) 2024-09-13T14:36:35.752 INFO:tasks.rados.rados.0.smithi151.stdout:3737: done (1 left) 2024-09-13T14:36:35.752 INFO:tasks.rados.rados.0.smithi151.stdout:3739: done (0 left) 2024-09-13T14:36:35.752 INFO:tasks.rados.rados.0.smithi151.stdout:3740: snap_remove snap 467 2024-09-13T14:36:36.671 INFO:tasks.rados.rados.0.smithi151.stdout:3740: done (0 left) 2024-09-13T14:36:36.671 INFO:tasks.rados.rados.0.smithi151.stdout:3741: rollback oid 39 current snap is 473 2024-09-13T14:36:36.671 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 39 to 470 2024-09-13T14:36:36.672 INFO:tasks.rados.rados.0.smithi151.stdout:3742: snap_create 2024-09-13T14:36:36.680 INFO:tasks.rados.rados.0.smithi151.stdout:3741: finishing rollback tid 0 to smithi15186934-39 2024-09-13T14:36:36.682 INFO:tasks.rados.rados.0.smithi151.stdout:3741: finishing rollback tid 1 to smithi15186934-39 2024-09-13T14:36:36.682 INFO:tasks.rados.rados.0.smithi151.stdout:3741: finishing rollback tid 2 to smithi15186934-39 2024-09-13T14:36:36.682 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 1549 (ObjNum 1324 snap 450 seq_num 1324) dirty exists 2024-09-13T14:36:37.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:36 smithi184 ceph-mon[92527]: pgmap v1125: 89 pgs: 89 active+clean; 277 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.7 MiB/s rd, 765 KiB/s wr, 8 op/s 2024-09-13T14:36:37.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:36 smithi184 ceph-mon[92527]: osdmap e1005: 8 total, 8 up, 8 in 2024-09-13T14:36:37.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:36 smithi151 ceph-mon[104450]: pgmap v1125: 89 pgs: 89 active+clean; 277 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.7 MiB/s rd, 765 KiB/s wr, 8 op/s 2024-09-13T14:36:37.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:36 smithi151 ceph-mon[104450]: osdmap e1005: 8 total, 8 up, 8 in 2024-09-13T14:36:37.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:36 smithi151 ceph-mon[108966]: pgmap v1125: 89 pgs: 89 active+clean; 277 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.7 MiB/s rd, 765 KiB/s wr, 8 op/s 2024-09-13T14:36:37.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:36 smithi151 ceph-mon[108966]: osdmap e1005: 8 total, 8 up, 8 in 2024-09-13T14:36:37.677 INFO:tasks.rados.rados.0.smithi151.stdout:3741: done (1 left) 2024-09-13T14:36:37.677 INFO:tasks.rados.rados.0.smithi151.stdout:3742: done (0 left) 2024-09-13T14:36:37.677 INFO:tasks.rados.rados.0.smithi151.stdout:3743: snap_remove snap 457 2024-09-13T14:36:38.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:37 smithi184 ceph-mon[92527]: osdmap e1006: 8 total, 8 up, 8 in 2024-09-13T14:36:38.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:37 smithi151 ceph-mon[104450]: osdmap e1006: 8 total, 8 up, 8 in 2024-09-13T14:36:38.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:37 smithi151 ceph-mon[108966]: osdmap e1006: 8 total, 8 up, 8 in 2024-09-13T14:36:38.680 INFO:tasks.rados.rados.0.smithi151.stdout:3743: done (0 left) 2024-09-13T14:36:38.680 INFO:tasks.rados.rados.0.smithi151.stdout:3744: delete oid 41 current snap is 474 2024-09-13T14:36:38.692 INFO:tasks.rados.rados.0.smithi151.stdout:3744: done (0 left) 2024-09-13T14:36:38.693 INFO:tasks.rados.rados.0.smithi151.stdout:3745: write oid 36 current snap is 474 2024-09-13T14:36:38.693 INFO:tasks.rados.rados.0.smithi151.stdout:3745: seq_num 1409 ranges {481063=751146,1784658=778160,3309938=371125} 2024-09-13T14:36:38.700 INFO:tasks.rados.rados.0.smithi151.stdout:3745: writing smithi15186934-36 from 481063 to 1232209 tid 1 2024-09-13T14:36:38.707 INFO:tasks.rados.rados.0.smithi151.stdout:3745: writing smithi15186934-36 from 1784658 to 2562818 tid 2 2024-09-13T14:36:38.713 INFO:tasks.rados.rados.0.smithi151.stdout:3745: writing smithi15186934-36 from 3309938 to 3681063 tid 3 2024-09-13T14:36:38.713 INFO:tasks.rados.rados.0.smithi151.stdout:3746: write oid 9 current snap is 474 2024-09-13T14:36:38.714 INFO:tasks.rados.rados.0.smithi151.stdout:3746: seq_num 1410 ranges {497870=426395,1443140=521818,2537361=530244,3583943=113927} 2024-09-13T14:36:38.718 INFO:tasks.rados.rados.0.smithi151.stdout:3746: writing smithi15186934-9 from 497870 to 924265 tid 1 2024-09-13T14:36:38.725 INFO:tasks.rados.rados.0.smithi151.stdout:3746: writing smithi15186934-9 from 1443140 to 1964958 tid 2 2024-09-13T14:36:38.730 INFO:tasks.rados.rados.0.smithi151.stdout:3746: writing smithi15186934-9 from 2537361 to 3067605 tid 3 2024-09-13T14:36:38.734 INFO:tasks.rados.rados.0.smithi151.stdout:3746: writing smithi15186934-9 from 3583943 to 3697870 tid 4 2024-09-13T14:36:38.734 INFO:tasks.rados.rados.0.smithi151.stdout:3747: write oid 3 current snap is 474 2024-09-13T14:36:38.734 INFO:tasks.rados.rados.0.smithi151.stdout:3745: finishing write tid 1 to smithi15186934-36 2024-09-13T14:36:38.734 INFO:tasks.rados.rados.0.smithi151.stdout:3747: seq_num 1411 ranges {514677=501644,1501622=781829,2964784=741516} 2024-09-13T14:36:38.739 INFO:tasks.rados.rados.0.smithi151.stdout:3747: writing smithi15186934-3 from 514677 to 1016321 tid 1 2024-09-13T14:36:38.746 INFO:tasks.rados.rados.0.smithi151.stdout:3747: writing smithi15186934-3 from 1501622 to 2283451 tid 2 2024-09-13T14:36:38.755 INFO:tasks.rados.rados.0.smithi151.stdout:3747: writing smithi15186934-3 from 2964784 to 3706300 tid 3 2024-09-13T14:36:38.755 INFO:tasks.rados.rados.0.smithi151.stdout:3748: write oid 27 current snap is 474 2024-09-13T14:36:38.755 INFO:tasks.rados.rados.0.smithi151.stdout:3745: finishing write tid 2 to smithi15186934-36 2024-09-13T14:36:38.755 INFO:tasks.rados.rados.0.smithi151.stdout:3745: finishing write tid 3 to smithi15186934-36 2024-09-13T14:36:38.755 INFO:tasks.rados.rados.0.smithi151.stdout:3745: finishing write tid 4 to smithi15186934-36 2024-09-13T14:36:38.755 INFO:tasks.rados.rados.0.smithi151.stdout:3745: finishing write tid 5 to smithi15186934-36 2024-09-13T14:36:38.755 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 36 v 809 (ObjNum 1409 snap 474 seq_num 1409) dirty exists 2024-09-13T14:36:38.755 INFO:tasks.rados.rados.0.smithi151.stdout:3745: left oid 36 (ObjNum 1409 snap 474 seq_num 1409) 2024-09-13T14:36:38.755 INFO:tasks.rados.rados.0.smithi151.stdout:3746: finishing write tid 1 to smithi15186934-9 2024-09-13T14:36:38.756 INFO:tasks.rados.rados.0.smithi151.stdout:3748: seq_num 1412 ranges {531484=576893,1843751=525487,2875854=669141} 2024-09-13T14:36:38.760 INFO:tasks.rados.rados.0.smithi151.stdout:3748: writing smithi15186934-27 from 531484 to 1108377 tid 1 2024-09-13T14:36:38.767 INFO:tasks.rados.rados.0.smithi151.stdout:3748: writing smithi15186934-27 from 1843751 to 2369238 tid 2 2024-09-13T14:36:38.774 INFO:tasks.rados.rados.0.smithi151.stdout:3748: writing smithi15186934-27 from 2875854 to 3544995 tid 3 2024-09-13T14:36:38.774 INFO:tasks.rados.rados.0.smithi151.stdout:3745: done (3 left) 2024-09-13T14:36:38.774 INFO:tasks.rados.rados.0.smithi151.stdout:3749: delete oid 18 current snap is 474 2024-09-13T14:36:38.774 INFO:tasks.rados.rados.0.smithi151.stdout:3746: finishing write tid 2 to smithi15186934-9 2024-09-13T14:36:38.774 INFO:tasks.rados.rados.0.smithi151.stdout:3746: finishing write tid 3 to smithi15186934-9 2024-09-13T14:36:38.774 INFO:tasks.rados.rados.0.smithi151.stdout:3746: finishing write tid 4 to smithi15186934-9 2024-09-13T14:36:38.774 INFO:tasks.rados.rados.0.smithi151.stdout:3746: finishing write tid 5 to smithi15186934-9 2024-09-13T14:36:38.774 INFO:tasks.rados.rados.0.smithi151.stdout:3746: finishing write tid 6 to smithi15186934-9 2024-09-13T14:36:38.774 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 1569 (ObjNum 1410 snap 474 seq_num 1410) dirty exists 2024-09-13T14:36:38.774 INFO:tasks.rados.rados.0.smithi151.stdout:3746: left oid 9 (ObjNum 1410 snap 474 seq_num 1410) 2024-09-13T14:36:38.775 INFO:tasks.rados.rados.0.smithi151.stdout:3747: finishing write tid 1 to smithi15186934-3 2024-09-13T14:36:38.775 INFO:tasks.rados.rados.0.smithi151.stdout:3747: finishing write tid 2 to smithi15186934-3 2024-09-13T14:36:38.775 INFO:tasks.rados.rados.0.smithi151.stdout:3747: finishing write tid 3 to smithi15186934-3 2024-09-13T14:36:38.775 INFO:tasks.rados.rados.0.smithi151.stdout:3747: finishing write tid 4 to smithi15186934-3 2024-09-13T14:36:38.775 INFO:tasks.rados.rados.0.smithi151.stdout:3747: finishing write tid 5 to smithi15186934-3 2024-09-13T14:36:38.775 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 1416 (ObjNum 1411 snap 474 seq_num 1411) dirty exists 2024-09-13T14:36:38.775 INFO:tasks.rados.rados.0.smithi151.stdout:3747: left oid 3 (ObjNum 1411 snap 474 seq_num 1411) 2024-09-13T14:36:38.775 INFO:tasks.rados.rados.0.smithi151.stdout:3748: finishing write tid 1 to smithi15186934-27 2024-09-13T14:36:38.775 INFO:tasks.rados.rados.0.smithi151.stdout:3748: finishing write tid 2 to smithi15186934-27 2024-09-13T14:36:38.776 INFO:tasks.rados.rados.0.smithi151.stdout:3746: done (3 left) 2024-09-13T14:36:38.776 INFO:tasks.rados.rados.0.smithi151.stdout:3747: done (2 left) 2024-09-13T14:36:38.776 INFO:tasks.rados.rados.0.smithi151.stdout:3749: done (1 left) 2024-09-13T14:36:38.776 INFO:tasks.rados.rados.0.smithi151.stdout:3750: snap_remove snap 464 2024-09-13T14:36:39.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:38 smithi184 ceph-mon[92527]: pgmap v1128: 89 pgs: 89 active+clean; 284 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.0 MiB/s rd, 1.6 MiB/s wr, 13 op/s 2024-09-13T14:36:39.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:38 smithi184 ceph-mon[92527]: osdmap e1007: 8 total, 8 up, 8 in 2024-09-13T14:36:39.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:38 smithi151 ceph-mon[104450]: pgmap v1128: 89 pgs: 89 active+clean; 284 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.0 MiB/s rd, 1.6 MiB/s wr, 13 op/s 2024-09-13T14:36:39.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:38 smithi151 ceph-mon[104450]: osdmap e1007: 8 total, 8 up, 8 in 2024-09-13T14:36:39.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:38 smithi151 ceph-mon[108966]: pgmap v1128: 89 pgs: 89 active+clean; 284 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.0 MiB/s rd, 1.6 MiB/s wr, 13 op/s 2024-09-13T14:36:39.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:38 smithi151 ceph-mon[108966]: osdmap e1007: 8 total, 8 up, 8 in 2024-09-13T14:36:39.659 INFO:tasks.rados.rados.0.smithi151.stdout:3750: done (1 left) 2024-09-13T14:36:39.660 INFO:tasks.rados.rados.0.smithi151.stdout:3751: read oid 16 snap -1 2024-09-13T14:36:39.660 INFO:tasks.rados.rados.0.smithi151.stdout:3751: expect (ObjNum 1378 snap 465 seq_num 1378) 2024-09-13T14:36:39.660 INFO:tasks.rados.rados.0.smithi151.stdout:3752: read oid 25 snap -1 2024-09-13T14:36:39.660 INFO:tasks.rados.rados.0.smithi151.stdout:3752: expect deleted 2024-09-13T14:36:39.660 INFO:tasks.rados.rados.0.smithi151.stdout:3753: delete oid 42 current snap is 474 2024-09-13T14:36:39.660 INFO:tasks.rados.rados.0.smithi151.stdout:3748: finishing write tid 3 to smithi15186934-27 2024-09-13T14:36:39.660 INFO:tasks.rados.rados.0.smithi151.stdout:3748: finishing write tid 4 to smithi15186934-27 2024-09-13T14:36:39.660 INFO:tasks.rados.rados.0.smithi151.stdout:3748: finishing write tid 5 to smithi15186934-27 2024-09-13T14:36:39.660 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 27 v 1122 (ObjNum 1412 snap 474 seq_num 1412) dirty exists 2024-09-13T14:36:39.660 INFO:tasks.rados.rados.0.smithi151.stdout:3748: left oid 27 (ObjNum 1412 snap 474 seq_num 1412) 2024-09-13T14:36:39.670 INFO:tasks.rados.rados.0.smithi151.stdout:3748: done (3 left) 2024-09-13T14:36:39.670 INFO:tasks.rados.rados.0.smithi151.stdout:3752: done (2 left) 2024-09-13T14:36:39.670 INFO:tasks.rados.rados.0.smithi151.stdout:3753: done (1 left) 2024-09-13T14:36:39.670 INFO:tasks.rados.rados.0.smithi151.stdout:3754: read oid 42 snap -1 2024-09-13T14:36:39.670 INFO:tasks.rados.rados.0.smithi151.stdout:3754: expect deleted 2024-09-13T14:36:39.670 INFO:tasks.rados.rados.0.smithi151.stdout:3755: write oid 26 current snap is 474 2024-09-13T14:36:39.670 INFO:tasks.rados.rados.0.smithi151.stdout:3755: seq_num 1415 ranges {581905=518993,1735550=672814,2958123=502957} 2024-09-13T14:36:39.677 INFO:tasks.rados.rados.0.smithi151.stdout:3755: writing smithi15186934-26 from 581905 to 1100898 tid 1 2024-09-13T14:36:39.685 INFO:tasks.rados.rados.0.smithi151.stdout:3755: writing smithi15186934-26 from 1735550 to 2408364 tid 2 2024-09-13T14:36:39.691 INFO:tasks.rados.rados.0.smithi151.stdout:3755: writing smithi15186934-26 from 2958123 to 3461080 tid 3 2024-09-13T14:36:39.691 INFO:tasks.rados.rados.0.smithi151.stdout:3756: delete oid 46 current snap is 474 2024-09-13T14:36:39.691 INFO:tasks.rados.rados.0.smithi151.stdout:3751: expect (ObjNum 1378 snap 465 seq_num 1378) 2024-09-13T14:36:39.877 INFO:tasks.rados.rados.0.smithi151.stdout:3755: finishing write tid 1 to smithi15186934-26 2024-09-13T14:36:39.877 INFO:tasks.rados.rados.0.smithi151.stdout:3755: finishing write tid 2 to smithi15186934-26 2024-09-13T14:36:39.877 INFO:tasks.rados.rados.0.smithi151.stdout:3755: finishing write tid 3 to smithi15186934-26 2024-09-13T14:36:39.878 INFO:tasks.rados.rados.0.smithi151.stdout:3755: finishing write tid 4 to smithi15186934-26 2024-09-13T14:36:39.878 INFO:tasks.rados.rados.0.smithi151.stdout:3755: finishing write tid 5 to smithi15186934-26 2024-09-13T14:36:39.878 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 1312 (ObjNum 1415 snap 474 seq_num 1415) dirty exists 2024-09-13T14:36:39.878 INFO:tasks.rados.rados.0.smithi151.stdout:3755: left oid 26 (ObjNum 1415 snap 474 seq_num 1415) 2024-09-13T14:36:39.878 INFO:tasks.rados.rados.0.smithi151.stdout:3751: done (3 left) 2024-09-13T14:36:39.878 INFO:tasks.rados.rados.0.smithi151.stdout:3754: done (2 left) 2024-09-13T14:36:39.878 INFO:tasks.rados.rados.0.smithi151.stdout:3755: done (1 left) 2024-09-13T14:36:39.878 INFO:tasks.rados.rados.0.smithi151.stdout:3756: done (0 left) 2024-09-13T14:36:39.878 INFO:tasks.rados.rados.0.smithi151.stdout:3757: read oid 2 snap -1 2024-09-13T14:36:39.878 INFO:tasks.rados.rados.0.smithi151.stdout:3757: expect (ObjNum 1403 snap 470 seq_num 1403) 2024-09-13T14:36:39.878 INFO:tasks.rados.rados.0.smithi151.stdout:3758: read oid 13 snap 413 2024-09-13T14:36:39.878 INFO:tasks.rados.rados.0.smithi151.stdout:3758: expect (ObjNum 1201 snap 406 seq_num 1201) 2024-09-13T14:36:39.878 INFO:tasks.rados.rados.0.smithi151.stdout:3759: snap_remove snap 468 2024-09-13T14:36:40.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:39 smithi184 ceph-mon[92527]: osdmap e1008: 8 total, 8 up, 8 in 2024-09-13T14:36:40.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:39 smithi184 ceph-mon[92527]: osdmap e1009: 8 total, 8 up, 8 in 2024-09-13T14:36:40.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:39 smithi151 ceph-mon[104450]: osdmap e1008: 8 total, 8 up, 8 in 2024-09-13T14:36:40.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:39 smithi151 ceph-mon[104450]: osdmap e1009: 8 total, 8 up, 8 in 2024-09-13T14:36:40.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:39 smithi151 ceph-mon[108966]: osdmap e1008: 8 total, 8 up, 8 in 2024-09-13T14:36:40.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:39 smithi151 ceph-mon[108966]: osdmap e1009: 8 total, 8 up, 8 in 2024-09-13T14:36:40.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:36:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:36:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:36:40.694 INFO:tasks.rados.rados.0.smithi151.stdout:3759: done (2 left) 2024-09-13T14:36:40.694 INFO:tasks.rados.rados.0.smithi151.stdout:3760: read oid 22 snap -1 2024-09-13T14:36:40.695 INFO:tasks.rados.rados.0.smithi151.stdout:3760: expect (ObjNum 1383 snap 466 seq_num 1383) 2024-09-13T14:36:40.695 INFO:tasks.rados.rados.0.smithi151.stdout:3761: write oid 17 current snap is 474 2024-09-13T14:36:40.695 INFO:tasks.rados.rados.0.smithi151.stdout:3761: seq_num 1417 ranges {615519=669491,1736161=676483,2896616=641854} 2024-09-13T14:36:40.703 INFO:tasks.rados.rados.0.smithi151.stdout:3761: writing smithi15186934-17 from 615519 to 1285010 tid 1 2024-09-13T14:36:40.710 INFO:tasks.rados.rados.0.smithi151.stdout:3761: writing smithi15186934-17 from 1736161 to 2412644 tid 2 2024-09-13T14:36:40.717 INFO:tasks.rados.rados.0.smithi151.stdout:3761: writing smithi15186934-17 from 2896616 to 3538470 tid 3 2024-09-13T14:36:40.717 INFO:tasks.rados.rados.0.smithi151.stdout:3762: snap_remove snap 471 2024-09-13T14:36:41.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:40 smithi184 ceph-mon[92527]: pgmap v1131: 89 pgs: 89 active+clean; 278 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.1 MiB/s rd, 914 KiB/s wr, 7 op/s 2024-09-13T14:36:41.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:40 smithi151 ceph-mon[104450]: pgmap v1131: 89 pgs: 89 active+clean; 278 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.1 MiB/s rd, 914 KiB/s wr, 7 op/s 2024-09-13T14:36:41.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:40 smithi151 ceph-mon[108966]: pgmap v1131: 89 pgs: 89 active+clean; 278 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.1 MiB/s rd, 914 KiB/s wr, 7 op/s 2024-09-13T14:36:41.711 INFO:tasks.rados.rados.0.smithi151.stdout:3762: done (4 left) 2024-09-13T14:36:41.712 INFO:tasks.rados.rados.0.smithi151.stdout:3763: snap_create 2024-09-13T14:36:41.712 INFO:tasks.rados.rados.0.smithi151.stdout:3757: expect (ObjNum 1403 snap 470 seq_num 1403) 2024-09-13T14:36:41.832 INFO:tasks.rados.rados.0.smithi151.stdout:3758: expect (ObjNum 1201 snap 406 seq_num 1201) 2024-09-13T14:36:41.961 INFO:tasks.rados.rados.0.smithi151.stdout:3760: expect (ObjNum 1383 snap 466 seq_num 1383) 2024-09-13T14:36:42.005 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:41 smithi184 ceph-mon[92527]: osdmap e1010: 8 total, 8 up, 8 in 2024-09-13T14:36:42.066 INFO:tasks.rados.rados.0.smithi151.stdout:3761: finishing write tid 1 to smithi15186934-17 2024-09-13T14:36:42.066 INFO:tasks.rados.rados.0.smithi151.stdout:3761: finishing write tid 2 to smithi15186934-17 2024-09-13T14:36:42.066 INFO:tasks.rados.rados.0.smithi151.stdout:3761: finishing write tid 3 to smithi15186934-17 2024-09-13T14:36:42.066 INFO:tasks.rados.rados.0.smithi151.stdout:3761: finishing write tid 4 to smithi15186934-17 2024-09-13T14:36:42.066 INFO:tasks.rados.rados.0.smithi151.stdout:3761: finishing write tid 5 to smithi15186934-17 2024-09-13T14:36:42.066 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 2133 (ObjNum 1417 snap 474 seq_num 1417) dirty exists 2024-09-13T14:36:42.066 INFO:tasks.rados.rados.0.smithi151.stdout:3761: left oid 17 (ObjNum 1417 snap 474 seq_num 1417) 2024-09-13T14:36:42.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:41 smithi151 ceph-mon[104450]: osdmap e1010: 8 total, 8 up, 8 in 2024-09-13T14:36:42.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:41 smithi151 ceph-mon[108966]: osdmap e1010: 8 total, 8 up, 8 in 2024-09-13T14:36:42.716 INFO:tasks.rados.rados.0.smithi151.stdout:3757: done (4 left) 2024-09-13T14:36:42.716 INFO:tasks.rados.rados.0.smithi151.stdout:3758: done (3 left) 2024-09-13T14:36:42.716 INFO:tasks.rados.rados.0.smithi151.stdout:3760: done (2 left) 2024-09-13T14:36:42.716 INFO:tasks.rados.rados.0.smithi151.stdout:3761: done (1 left) 2024-09-13T14:36:42.716 INFO:tasks.rados.rados.0.smithi151.stdout:3763: done (0 left) 2024-09-13T14:36:42.716 INFO:tasks.rados.rados.0.smithi151.stdout:3764: read oid 2 snap -1 2024-09-13T14:36:42.716 INFO:tasks.rados.rados.0.smithi151.stdout:3764: expect (ObjNum 1403 snap 470 seq_num 1403) 2024-09-13T14:36:42.716 INFO:tasks.rados.rados.0.smithi151.stdout:3765: snap_create 2024-09-13T14:36:42.724 INFO:tasks.rados.rados.0.smithi151.stdout:3764: expect (ObjNum 1403 snap 470 seq_num 1403) 2024-09-13T14:36:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:42 smithi184 ceph-mon[92527]: pgmap v1134: 89 pgs: 89 active+clean; 281 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 1.3 MiB/s wr, 4 op/s 2024-09-13T14:36:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:42 smithi184 ceph-mon[92527]: osdmap e1011: 8 total, 8 up, 8 in 2024-09-13T14:36:43.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:42 smithi151 ceph-mon[104450]: pgmap v1134: 89 pgs: 89 active+clean; 281 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 1.3 MiB/s wr, 4 op/s 2024-09-13T14:36:43.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:42 smithi151 ceph-mon[104450]: osdmap e1011: 8 total, 8 up, 8 in 2024-09-13T14:36:43.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:42 smithi151 ceph-mon[108966]: pgmap v1134: 89 pgs: 89 active+clean; 281 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 1.3 MiB/s wr, 4 op/s 2024-09-13T14:36:43.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:42 smithi151 ceph-mon[108966]: osdmap e1011: 8 total, 8 up, 8 in 2024-09-13T14:36:43.726 INFO:tasks.rados.rados.0.smithi151.stdout:3764: done (1 left) 2024-09-13T14:36:43.726 INFO:tasks.rados.rados.0.smithi151.stdout:3765: done (0 left) 2024-09-13T14:36:43.726 INFO:tasks.rados.rados.0.smithi151.stdout:3766: snap_create 2024-09-13T14:36:44.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:43 smithi184 ceph-mon[92527]: osdmap e1012: 8 total, 8 up, 8 in 2024-09-13T14:36:44.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:43 smithi151 ceph-mon[104450]: osdmap e1012: 8 total, 8 up, 8 in 2024-09-13T14:36:44.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:43 smithi151 ceph-mon[108966]: osdmap e1012: 8 total, 8 up, 8 in 2024-09-13T14:36:44.664 INFO:tasks.rados.rados.0.smithi151.stdout:3766: done (0 left) 2024-09-13T14:36:44.664 INFO:tasks.rados.rados.0.smithi151.stdout:3767: snap_remove snap 450 2024-09-13T14:36:44.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:44 smithi151 ceph-mon[108966]: pgmap v1137: 89 pgs: 89 active+clean; 285 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.0 MiB/s rd, 2.2 MiB/s wr, 12 op/s 2024-09-13T14:36:44.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:44 smithi151 ceph-mon[108966]: osdmap e1013: 8 total, 8 up, 8 in 2024-09-13T14:36:44.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:44 smithi151 ceph-mon[108966]: osdmap e1014: 8 total, 8 up, 8 in 2024-09-13T14:36:44.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:44 smithi151 ceph-mon[104450]: pgmap v1137: 89 pgs: 89 active+clean; 285 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.0 MiB/s rd, 2.2 MiB/s wr, 12 op/s 2024-09-13T14:36:44.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:44 smithi151 ceph-mon[104450]: osdmap e1013: 8 total, 8 up, 8 in 2024-09-13T14:36:44.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:44 smithi151 ceph-mon[104450]: osdmap e1014: 8 total, 8 up, 8 in 2024-09-13T14:36:44.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:36:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:36:44] "GET /metrics HTTP/1.1" 200 36512 "" "Prometheus/2.51.0" 2024-09-13T14:36:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:44 smithi184 ceph-mon[92527]: pgmap v1137: 89 pgs: 89 active+clean; 285 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.0 MiB/s rd, 2.2 MiB/s wr, 12 op/s 2024-09-13T14:36:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:44 smithi184 ceph-mon[92527]: osdmap e1013: 8 total, 8 up, 8 in 2024-09-13T14:36:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:44 smithi184 ceph-mon[92527]: osdmap e1014: 8 total, 8 up, 8 in 2024-09-13T14:36:45.668 INFO:tasks.rados.rados.0.smithi151.stdout:3767: done (0 left) 2024-09-13T14:36:45.669 INFO:tasks.rados.rados.0.smithi151.stdout:3768: snap_remove snap 465 2024-09-13T14:36:46.666 INFO:tasks.rados.rados.0.smithi151.stdout:3768: done (0 left) 2024-09-13T14:36:46.666 INFO:tasks.rados.rados.0.smithi151.stdout:3769: write oid 39 current snap is 477 2024-09-13T14:36:46.666 INFO:tasks.rados.rados.0.smithi151.stdout:3769: seq_num 1418 ranges {632326=744740,1794643=536494,3040392=453126} 2024-09-13T14:36:46.674 INFO:tasks.rados.rados.0.smithi151.stdout:3769: writing smithi15186934-39 from 632326 to 1377066 tid 1 2024-09-13T14:36:46.680 INFO:tasks.rados.rados.0.smithi151.stdout:3769: writing smithi15186934-39 from 1794643 to 2331137 tid 2 2024-09-13T14:36:46.687 INFO:tasks.rados.rados.0.smithi151.stdout:3769: writing smithi15186934-39 from 3040392 to 3493518 tid 3 2024-09-13T14:36:46.688 INFO:tasks.rados.rados.0.smithi151.stdout:3770: write oid 30 current snap is 477 2024-09-13T14:36:46.688 INFO:tasks.rados.rados.0.smithi151.stdout:3769: finishing write tid 1 to smithi15186934-39 2024-09-13T14:36:46.688 INFO:tasks.rados.rados.0.smithi151.stdout:3769: finishing write tid 2 to smithi15186934-39 2024-09-13T14:36:46.688 INFO:tasks.rados.rados.0.smithi151.stdout:3770: seq_num 1419 ranges {649133=419989,1853125=680152,2951462=664398} 2024-09-13T14:36:46.693 INFO:tasks.rados.rados.0.smithi151.stdout:3770: writing smithi15186934-30 from 649133 to 1069122 tid 1 2024-09-13T14:36:46.701 INFO:tasks.rados.rados.0.smithi151.stdout:3770: writing smithi15186934-30 from 1853125 to 2533277 tid 2 2024-09-13T14:36:46.708 INFO:tasks.rados.rados.0.smithi151.stdout:3770: writing smithi15186934-30 from 2951462 to 3615860 tid 3 2024-09-13T14:36:46.708 INFO:tasks.rados.rados.0.smithi151.stdout:3771: write oid 1 current snap is 477 2024-09-13T14:36:46.708 INFO:tasks.rados.rados.0.smithi151.stdout:3769: finishing write tid 3 to smithi15186934-39 2024-09-13T14:36:46.708 INFO:tasks.rados.rados.0.smithi151.stdout:3769: finishing write tid 4 to smithi15186934-39 2024-09-13T14:36:46.708 INFO:tasks.rados.rados.0.smithi151.stdout:3769: finishing write tid 5 to smithi15186934-39 2024-09-13T14:36:46.708 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 1566 (ObjNum 1418 snap 477 seq_num 1418) dirty exists 2024-09-13T14:36:46.708 INFO:tasks.rados.rados.0.smithi151.stdout:3769: left oid 39 (ObjNum 1418 snap 477 seq_num 1418) 2024-09-13T14:36:46.708 INFO:tasks.rados.rados.0.smithi151.stdout:3770: finishing write tid 1 to smithi15186934-30 2024-09-13T14:36:46.709 INFO:tasks.rados.rados.0.smithi151.stdout:3770: finishing write tid 2 to smithi15186934-30 2024-09-13T14:36:46.709 INFO:tasks.rados.rados.0.smithi151.stdout:3771: seq_num 1420 ranges {665940=495238,1795254=540163,2978885=475670} 2024-09-13T14:36:46.714 INFO:tasks.rados.rados.0.smithi151.stdout:3771: writing smithi15186934-1 from 665940 to 1161178 tid 1 2024-09-13T14:36:46.721 INFO:tasks.rados.rados.0.smithi151.stdout:3771: writing smithi15186934-1 from 1795254 to 2335417 tid 2 2024-09-13T14:36:46.727 INFO:tasks.rados.rados.0.smithi151.stdout:3771: writing smithi15186934-1 from 2978885 to 3454555 tid 3 2024-09-13T14:36:46.727 INFO:tasks.rados.rados.0.smithi151.stdout:3769: done (2 left) 2024-09-13T14:36:46.727 INFO:tasks.rados.rados.0.smithi151.stdout:3772: snap_remove snap 474 2024-09-13T14:36:47.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:46 smithi184 ceph-mon[92527]: pgmap v1140: 89 pgs: 89 active+clean; 287 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 1.3 MiB/s wr, 15 op/s 2024-09-13T14:36:47.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:46 smithi184 ceph-mon[92527]: osdmap e1015: 8 total, 8 up, 8 in 2024-09-13T14:36:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:47.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:46 smithi151 ceph-mon[104450]: pgmap v1140: 89 pgs: 89 active+clean; 287 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 1.3 MiB/s wr, 15 op/s 2024-09-13T14:36:47.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:46 smithi151 ceph-mon[104450]: osdmap e1015: 8 total, 8 up, 8 in 2024-09-13T14:36:47.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:47.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:46 smithi151 ceph-mon[108966]: pgmap v1140: 89 pgs: 89 active+clean; 287 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 1.3 MiB/s wr, 15 op/s 2024-09-13T14:36:47.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:46 smithi151 ceph-mon[108966]: osdmap e1015: 8 total, 8 up, 8 in 2024-09-13T14:36:47.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:47.687 INFO:tasks.rados.rados.0.smithi151.stdout:3772: done (2 left) 2024-09-13T14:36:47.688 INFO:tasks.rados.rados.0.smithi151.stdout:3773: snap_create 2024-09-13T14:36:47.688 INFO:tasks.rados.rados.0.smithi151.stdout:3770: finishing write tid 3 to smithi15186934-30 2024-09-13T14:36:47.688 INFO:tasks.rados.rados.0.smithi151.stdout:3770: finishing write tid 4 to smithi15186934-30 2024-09-13T14:36:47.688 INFO:tasks.rados.rados.0.smithi151.stdout:3770: finishing write tid 5 to smithi15186934-30 2024-09-13T14:36:47.688 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 30 v 1434 (ObjNum 1419 snap 477 seq_num 1419) dirty exists 2024-09-13T14:36:47.688 INFO:tasks.rados.rados.0.smithi151.stdout:3770: left oid 30 (ObjNum 1419 snap 477 seq_num 1419) 2024-09-13T14:36:47.688 INFO:tasks.rados.rados.0.smithi151.stdout:3771: finishing write tid 1 to smithi15186934-1 2024-09-13T14:36:47.688 INFO:tasks.rados.rados.0.smithi151.stdout:3771: finishing write tid 2 to smithi15186934-1 2024-09-13T14:36:47.688 INFO:tasks.rados.rados.0.smithi151.stdout:3771: finishing write tid 3 to smithi15186934-1 2024-09-13T14:36:47.688 INFO:tasks.rados.rados.0.smithi151.stdout:3771: finishing write tid 4 to smithi15186934-1 2024-09-13T14:36:47.688 INFO:tasks.rados.rados.0.smithi151.stdout:3771: finishing write tid 5 to smithi15186934-1 2024-09-13T14:36:47.688 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 1 v 1254 (ObjNum 1420 snap 477 seq_num 1420) dirty exists 2024-09-13T14:36:47.689 INFO:tasks.rados.rados.0.smithi151.stdout:3771: left oid 1 (ObjNum 1420 snap 477 seq_num 1420) 2024-09-13T14:36:48.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:47 smithi184 ceph-mon[92527]: osdmap e1016: 8 total, 8 up, 8 in 2024-09-13T14:36:48.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:47 smithi151 ceph-mon[104450]: osdmap e1016: 8 total, 8 up, 8 in 2024-09-13T14:36:48.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:47 smithi151 ceph-mon[108966]: osdmap e1016: 8 total, 8 up, 8 in 2024-09-13T14:36:48.689 INFO:tasks.rados.rados.0.smithi151.stdout:3770: done (2 left) 2024-09-13T14:36:48.689 INFO:tasks.rados.rados.0.smithi151.stdout:3771: done (1 left) 2024-09-13T14:36:48.689 INFO:tasks.rados.rados.0.smithi151.stdout:3773: done (0 left) 2024-09-13T14:36:48.689 INFO:tasks.rados.rados.0.smithi151.stdout:3774: write oid 24 current snap is 478 2024-09-13T14:36:48.690 INFO:tasks.rados.rados.0.smithi151.stdout:3774: seq_num 1421 ranges {682747=570487,1853736=683821,3006308=403295} 2024-09-13T14:36:48.696 INFO:tasks.rados.rados.0.smithi151.stdout:3774: writing smithi15186934-24 from 682747 to 1253234 tid 1 2024-09-13T14:36:48.703 INFO:tasks.rados.rados.0.smithi151.stdout:3774: writing smithi15186934-24 from 1853736 to 2537557 tid 2 2024-09-13T14:36:48.708 INFO:tasks.rados.rados.0.smithi151.stdout:3774: writing smithi15186934-24 from 3006308 to 3409603 tid 3 2024-09-13T14:36:48.708 INFO:tasks.rados.rados.0.smithi151.stdout:3775: rollback oid 4 current snap is 478 2024-09-13T14:36:48.708 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 4 to 413 2024-09-13T14:36:48.708 INFO:tasks.rados.rados.0.smithi151.stdout:3774: finishing write tid 1 to smithi15186934-24 2024-09-13T14:36:48.709 INFO:tasks.rados.rados.0.smithi151.stdout:3774: finishing write tid 2 to smithi15186934-24 2024-09-13T14:36:48.709 INFO:tasks.rados.rados.0.smithi151.stdout:3776: write oid 12 current snap is 478 2024-09-13T14:36:48.709 INFO:tasks.rados.rados.0.smithi151.stdout:3776: seq_num 1422 ranges {699554=762089,2028571=543832,3150084=614567} 2024-09-13T14:36:48.716 INFO:tasks.rados.rados.0.smithi151.stdout:3776: writing smithi15186934-12 from 699554 to 1461643 tid 1 2024-09-13T14:36:48.723 INFO:tasks.rados.rados.0.smithi151.stdout:3776: writing smithi15186934-12 from 2028571 to 2572403 tid 2 2024-09-13T14:36:48.730 INFO:tasks.rados.rados.0.smithi151.stdout:3776: writing smithi15186934-12 from 3150084 to 3764651 tid 3 2024-09-13T14:36:48.730 INFO:tasks.rados.rados.0.smithi151.stdout:3777: write oid 29 current snap is 478 2024-09-13T14:36:48.730 INFO:tasks.rados.rados.0.smithi151.stdout:3774: finishing write tid 3 to smithi15186934-24 2024-09-13T14:36:48.730 INFO:tasks.rados.rados.0.smithi151.stdout:3777: seq_num 1423 ranges {716361=437338,1687053=687490,2777507=425839,3877756=38605} 2024-09-13T14:36:48.737 INFO:tasks.rados.rados.0.smithi151.stdout:3777: writing smithi15186934-29 from 716361 to 1153699 tid 1 2024-09-13T14:36:48.742 INFO:tasks.rados.rados.0.smithi151.stdout:3777: writing smithi15186934-29 from 1687053 to 2374543 tid 2 2024-09-13T14:36:48.747 INFO:tasks.rados.rados.0.smithi151.stdout:3777: writing smithi15186934-29 from 2777507 to 3203346 tid 3 2024-09-13T14:36:48.751 INFO:tasks.rados.rados.0.smithi151.stdout:3777: writing smithi15186934-29 from 3877756 to 3916361 tid 4 2024-09-13T14:36:48.751 INFO:tasks.rados.rados.0.smithi151.stdout:3778: write oid 26 current snap is 478 2024-09-13T14:36:48.751 INFO:tasks.rados.rados.0.smithi151.stdout:3778: seq_num 1424 ranges {733168=512587,2029182=547501,3088577=637111} 2024-09-13T14:36:48.758 INFO:tasks.rados.rados.0.smithi151.stdout:3778: writing smithi15186934-26 from 733168 to 1245755 tid 1 2024-09-13T14:36:48.765 INFO:tasks.rados.rados.0.smithi151.stdout:3778: writing smithi15186934-26 from 2029182 to 2576683 tid 2 2024-09-13T14:36:48.772 INFO:tasks.rados.rados.0.smithi151.stdout:3778: writing smithi15186934-26 from 3088577 to 3725688 tid 3 2024-09-13T14:36:48.772 INFO:tasks.rados.rados.0.smithi151.stdout:3779: read oid 6 snap -1 2024-09-13T14:36:48.772 INFO:tasks.rados.rados.0.smithi151.stdout:3779: expect (ObjNum 1407 snap 473 seq_num 1407) 2024-09-13T14:36:48.772 INFO:tasks.rados.rados.0.smithi151.stdout:3780: write oid 34 current snap is 478 2024-09-13T14:36:48.772 INFO:tasks.rados.rados.0.smithi151.stdout:3775: finishing rollback tid 0 to smithi15186934-4 2024-09-13T14:36:48.772 INFO:tasks.rados.rados.0.smithi151.stdout:3780: seq_num 1425 ranges {749975=587836,2087664=691159,3516000=433975} 2024-09-13T14:36:48.779 INFO:tasks.rados.rados.0.smithi151.stdout:3780: writing smithi15186934-34 from 749975 to 1337811 tid 1 2024-09-13T14:36:48.787 INFO:tasks.rados.rados.0.smithi151.stdout:3780: writing smithi15186934-34 from 2087664 to 2778823 tid 2 2024-09-13T14:36:48.795 INFO:tasks.rados.rados.0.smithi151.stdout:3780: writing smithi15186934-34 from 3516000 to 3949975 tid 3 2024-09-13T14:36:48.795 INFO:tasks.rados.rados.0.smithi151.stdout:3781: snap_create 2024-09-13T14:36:48.795 INFO:tasks.rados.rados.0.smithi151.stdout:3774: finishing write tid 4 to smithi15186934-24 2024-09-13T14:36:48.795 INFO:tasks.rados.rados.0.smithi151.stdout:3774: finishing write tid 5 to smithi15186934-24 2024-09-13T14:36:48.795 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 24 v 2151 (ObjNum 1421 snap 478 seq_num 1421) dirty exists 2024-09-13T14:36:48.795 INFO:tasks.rados.rados.0.smithi151.stdout:3774: left oid 24 (ObjNum 1421 snap 478 seq_num 1421) 2024-09-13T14:36:48.795 INFO:tasks.rados.rados.0.smithi151.stdout:3775: finishing rollback tid 1 to smithi15186934-4 2024-09-13T14:36:48.795 INFO:tasks.rados.rados.0.smithi151.stdout:3775: finishing rollback tid 2 to smithi15186934-4 2024-09-13T14:36:48.795 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 1846 (ObjNum 1188 snap 402 seq_num 1188) dirty exists 2024-09-13T14:36:48.795 INFO:tasks.rados.rados.0.smithi151.stdout:3776: finishing write tid 1 to smithi15186934-12 2024-09-13T14:36:48.796 INFO:tasks.rados.rados.0.smithi151.stdout:3776: finishing write tid 2 to smithi15186934-12 2024-09-13T14:36:48.796 INFO:tasks.rados.rados.0.smithi151.stdout:3776: finishing write tid 3 to smithi15186934-12 2024-09-13T14:36:48.796 INFO:tasks.rados.rados.0.smithi151.stdout:3776: finishing write tid 4 to smithi15186934-12 2024-09-13T14:36:48.796 INFO:tasks.rados.rados.0.smithi151.stdout:3776: finishing write tid 5 to smithi15186934-12 2024-09-13T14:36:48.796 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 12 v 823 (ObjNum 1422 snap 478 seq_num 1422) dirty exists 2024-09-13T14:36:48.796 INFO:tasks.rados.rados.0.smithi151.stdout:3776: left oid 12 (ObjNum 1422 snap 478 seq_num 1422) 2024-09-13T14:36:48.796 INFO:tasks.rados.rados.0.smithi151.stdout:3777: finishing write tid 1 to smithi15186934-29 2024-09-13T14:36:48.796 INFO:tasks.rados.rados.0.smithi151.stdout:3777: finishing write tid 2 to smithi15186934-29 2024-09-13T14:36:48.796 INFO:tasks.rados.rados.0.smithi151.stdout:3777: finishing write tid 3 to smithi15186934-29 2024-09-13T14:36:48.796 INFO:tasks.rados.rados.0.smithi151.stdout:3777: finishing write tid 4 to smithi15186934-29 2024-09-13T14:36:48.796 INFO:tasks.rados.rados.0.smithi151.stdout:3777: finishing write tid 5 to smithi15186934-29 2024-09-13T14:36:48.796 INFO:tasks.rados.rados.0.smithi151.stdout:3777: finishing write tid 6 to smithi15186934-29 2024-09-13T14:36:48.796 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 29 v 1044 (ObjNum 1423 snap 478 seq_num 1423) dirty exists 2024-09-13T14:36:48.797 INFO:tasks.rados.rados.0.smithi151.stdout:3777: left oid 29 (ObjNum 1423 snap 478 seq_num 1423) 2024-09-13T14:36:48.797 INFO:tasks.rados.rados.0.smithi151.stdout:3778: finishing write tid 1 to smithi15186934-26 2024-09-13T14:36:48.797 INFO:tasks.rados.rados.0.smithi151.stdout:3778: finishing write tid 2 to smithi15186934-26 2024-09-13T14:36:48.797 INFO:tasks.rados.rados.0.smithi151.stdout:3778: finishing write tid 3 to smithi15186934-26 2024-09-13T14:36:48.797 INFO:tasks.rados.rados.0.smithi151.stdout:3778: finishing write tid 4 to smithi15186934-26 2024-09-13T14:36:48.797 INFO:tasks.rados.rados.0.smithi151.stdout:3778: finishing write tid 5 to smithi15186934-26 2024-09-13T14:36:48.797 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 1341 (ObjNum 1424 snap 478 seq_num 1424) dirty exists 2024-09-13T14:36:48.797 INFO:tasks.rados.rados.0.smithi151.stdout:3778: left oid 26 (ObjNum 1424 snap 478 seq_num 1424) 2024-09-13T14:36:48.797 INFO:tasks.rados.rados.0.smithi151.stdout:3779: expect (ObjNum 1407 snap 473 seq_num 1407) 2024-09-13T14:36:48.908 INFO:tasks.rados.rados.0.smithi151.stdout:3780: finishing write tid 1 to smithi15186934-34 2024-09-13T14:36:48.908 INFO:tasks.rados.rados.0.smithi151.stdout:3780: finishing write tid 2 to smithi15186934-34 2024-09-13T14:36:48.908 INFO:tasks.rados.rados.0.smithi151.stdout:3780: finishing write tid 3 to smithi15186934-34 2024-09-13T14:36:48.908 INFO:tasks.rados.rados.0.smithi151.stdout:3780: finishing write tid 4 to smithi15186934-34 2024-09-13T14:36:48.909 INFO:tasks.rados.rados.0.smithi151.stdout:3780: finishing write tid 5 to smithi15186934-34 2024-09-13T14:36:48.909 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 1438 (ObjNum 1425 snap 478 seq_num 1425) dirty exists 2024-09-13T14:36:48.909 INFO:tasks.rados.rados.0.smithi151.stdout:3780: left oid 34 (ObjNum 1425 snap 478 seq_num 1425) 2024-09-13T14:36:49.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:48 smithi184 ceph-mon[92527]: pgmap v1143: 89 pgs: 89 active+clean; 269 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.0 MiB/s rd, 838 KiB/s wr, 9 op/s 2024-09-13T14:36:49.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:48 smithi184 ceph-mon[92527]: osdmap e1017: 8 total, 8 up, 8 in 2024-09-13T14:36:49.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:48 smithi151 ceph-mon[108966]: pgmap v1143: 89 pgs: 89 active+clean; 269 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.0 MiB/s rd, 838 KiB/s wr, 9 op/s 2024-09-13T14:36:49.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:48 smithi151 ceph-mon[108966]: osdmap e1017: 8 total, 8 up, 8 in 2024-09-13T14:36:49.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:48 smithi151 ceph-mon[104450]: pgmap v1143: 89 pgs: 89 active+clean; 269 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.0 MiB/s rd, 838 KiB/s wr, 9 op/s 2024-09-13T14:36:49.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:48 smithi151 ceph-mon[104450]: osdmap e1017: 8 total, 8 up, 8 in 2024-09-13T14:36:49.664 INFO:tasks.rados.rados.0.smithi151.stdout:3774: done (7 left) 2024-09-13T14:36:49.664 INFO:tasks.rados.rados.0.smithi151.stdout:3775: done (6 left) 2024-09-13T14:36:49.665 INFO:tasks.rados.rados.0.smithi151.stdout:3776: done (5 left) 2024-09-13T14:36:49.665 INFO:tasks.rados.rados.0.smithi151.stdout:3777: done (4 left) 2024-09-13T14:36:49.665 INFO:tasks.rados.rados.0.smithi151.stdout:3778: done (3 left) 2024-09-13T14:36:49.665 INFO:tasks.rados.rados.0.smithi151.stdout:3779: done (2 left) 2024-09-13T14:36:49.665 INFO:tasks.rados.rados.0.smithi151.stdout:3780: done (1 left) 2024-09-13T14:36:49.665 INFO:tasks.rados.rados.0.smithi151.stdout:3781: done (0 left) 2024-09-13T14:36:49.665 INFO:tasks.rados.rados.0.smithi151.stdout:3782: read oid 33 snap -1 2024-09-13T14:36:49.665 INFO:tasks.rados.rados.0.smithi151.stdout:3782: expect (ObjNum 1058 snap 357 seq_num 1058) 2024-09-13T14:36:49.665 INFO:tasks.rados.rados.0.smithi151.stdout:3783: snap_remove snap 477 2024-09-13T14:36:50.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:49 smithi184 ceph-mon[92527]: osdmap e1018: 8 total, 8 up, 8 in 2024-09-13T14:36:50.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:49 smithi184 ceph-mon[92527]: osdmap e1019: 8 total, 8 up, 8 in 2024-09-13T14:36:50.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:49 smithi151 ceph-mon[108966]: osdmap e1018: 8 total, 8 up, 8 in 2024-09-13T14:36:50.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:49 smithi151 ceph-mon[108966]: osdmap e1019: 8 total, 8 up, 8 in 2024-09-13T14:36:50.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:49 smithi151 ceph-mon[104450]: osdmap e1018: 8 total, 8 up, 8 in 2024-09-13T14:36:50.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:49 smithi151 ceph-mon[104450]: osdmap e1019: 8 total, 8 up, 8 in 2024-09-13T14:36:50.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:36:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:36:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:36:50.665 INFO:tasks.rados.rados.0.smithi151.stdout:3783: done (1 left) 2024-09-13T14:36:50.665 INFO:tasks.rados.rados.0.smithi151.stdout:3784: snap_remove snap 463 2024-09-13T14:36:51.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:50 smithi184 ceph-mon[92527]: pgmap v1146: 89 pgs: 89 active+clean; 262 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 424 KiB/s wr, 1 op/s 2024-09-13T14:36:51.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:50 smithi184 ceph-mon[92527]: osdmap e1020: 8 total, 8 up, 8 in 2024-09-13T14:36:51.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:50 smithi151 ceph-mon[104450]: pgmap v1146: 89 pgs: 89 active+clean; 262 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 424 KiB/s wr, 1 op/s 2024-09-13T14:36:51.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:50 smithi151 ceph-mon[104450]: osdmap e1020: 8 total, 8 up, 8 in 2024-09-13T14:36:51.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:50 smithi151 ceph-mon[108966]: pgmap v1146: 89 pgs: 89 active+clean; 262 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 424 KiB/s wr, 1 op/s 2024-09-13T14:36:51.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:50 smithi151 ceph-mon[108966]: osdmap e1020: 8 total, 8 up, 8 in 2024-09-13T14:36:51.666 INFO:tasks.rados.rados.0.smithi151.stdout:3784: done (1 left) 2024-09-13T14:36:51.666 INFO:tasks.rados.rados.0.smithi151.stdout:3785: read oid 4 snap -1 2024-09-13T14:36:51.666 INFO:tasks.rados.rados.0.smithi151.stdout:3785: expect (ObjNum 1188 snap 402 seq_num 1188) 2024-09-13T14:36:51.666 INFO:tasks.rados.rados.0.smithi151.stdout:3786: write oid 1 current snap is 479 2024-09-13T14:36:51.666 INFO:tasks.rados.rados.0.smithi151.stdout:3786: seq_num 1426 ranges {766782=663085,2146146=551170,3143423=776008} 2024-09-13T14:36:51.674 INFO:tasks.rados.rados.0.smithi151.stdout:3786: writing smithi15186934-1 from 766782 to 1429867 tid 1 2024-09-13T14:36:51.683 INFO:tasks.rados.rados.0.smithi151.stdout:3786: writing smithi15186934-1 from 2146146 to 2697316 tid 2 2024-09-13T14:36:51.690 INFO:tasks.rados.rados.0.smithi151.stdout:3786: writing smithi15186934-1 from 3143423 to 3919431 tid 3 2024-09-13T14:36:51.690 INFO:tasks.rados.rados.0.smithi151.stdout:3787: rollback oid 44 current snap is 479 2024-09-13T14:36:51.690 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 44 to 470 2024-09-13T14:36:51.690 INFO:tasks.rados.rados.0.smithi151.stdout:3782: expect (ObjNum 1058 snap 357 seq_num 1058) 2024-09-13T14:36:51.869 INFO:tasks.rados.rados.0.smithi151.stdout:3786: finishing write tid 1 to smithi15186934-1 2024-09-13T14:36:51.869 INFO:tasks.rados.rados.0.smithi151.stdout:3785: expect (ObjNum 1188 snap 402 seq_num 1188) 2024-09-13T14:36:51.994 INFO:tasks.rados.rados.0.smithi151.stdout:3786: finishing write tid 2 to smithi15186934-1 2024-09-13T14:36:51.995 INFO:tasks.rados.rados.0.smithi151.stdout:3786: finishing write tid 3 to smithi15186934-1 2024-09-13T14:36:51.995 INFO:tasks.rados.rados.0.smithi151.stdout:3782: done (3 left) 2024-09-13T14:36:51.995 INFO:tasks.rados.rados.0.smithi151.stdout:3785: done (2 left) 2024-09-13T14:36:51.995 INFO:tasks.rados.rados.0.smithi151.stdout:3788: write oid 4 current snap is 479 2024-09-13T14:36:51.995 INFO:tasks.rados.rados.0.smithi151.stdout:3786: finishing write tid 4 to smithi15186934-1 2024-09-13T14:36:51.995 INFO:tasks.rados.rados.0.smithi151.stdout:3788: seq_num 1427 ranges {783589=738334,2204628=694828,3570846=412743} 2024-09-13T14:36:52.003 INFO:tasks.rados.rados.0.smithi151.stdout:3788: writing smithi15186934-4 from 783589 to 1521923 tid 1 2024-09-13T14:36:52.010 INFO:tasks.rados.rados.0.smithi151.stdout:3788: writing smithi15186934-4 from 2204628 to 2899456 tid 2 2024-09-13T14:36:52.016 INFO:tasks.rados.rados.0.smithi151.stdout:3788: writing smithi15186934-4 from 3570846 to 3983589 tid 3 2024-09-13T14:36:52.016 INFO:tasks.rados.rados.0.smithi151.stdout:3789: write oid 36 current snap is 479 2024-09-13T14:36:52.016 INFO:tasks.rados.rados.0.smithi151.stdout:3789: seq_num 1428 ranges {400396=413583,1463110=438486} 2024-09-13T14:36:52.020 INFO:tasks.rados.rados.0.smithi151.stdout:3789: writing smithi15186934-36 from 400396 to 813979 tid 1 2024-09-13T14:36:52.027 INFO:tasks.rados.rados.0.smithi151.stdout:3789: writing smithi15186934-36 from 1463110 to 1901596 tid 2 2024-09-13T14:36:52.027 INFO:tasks.rados.rados.0.smithi151.stdout:3790: snap_create 2024-09-13T14:36:52.027 INFO:tasks.rados.rados.0.smithi151.stdout:3786: finishing write tid 5 to smithi15186934-1 2024-09-13T14:36:52.027 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 1 v 1265 (ObjNum 1426 snap 479 seq_num 1426) dirty exists 2024-09-13T14:36:52.027 INFO:tasks.rados.rados.0.smithi151.stdout:3786: left oid 1 (ObjNum 1426 snap 479 seq_num 1426) 2024-09-13T14:36:52.027 INFO:tasks.rados.rados.0.smithi151.stdout:3787: finishing rollback tid 0 to smithi15186934-44 2024-09-13T14:36:52.027 INFO:tasks.rados.rados.0.smithi151.stdout:3787: finishing rollback tid 1 to smithi15186934-44 2024-09-13T14:36:52.027 INFO:tasks.rados.rados.0.smithi151.stdout:3787: finishing rollback tid 2 to smithi15186934-44 2024-09-13T14:36:52.027 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 1269 (ObjNum 1342 snap 454 seq_num 1342) dirty exists 2024-09-13T14:36:52.027 INFO:tasks.rados.rados.0.smithi151.stdout:3788: finishing write tid 1 to smithi15186934-4 2024-09-13T14:36:52.028 INFO:tasks.rados.rados.0.smithi151.stdout:3788: finishing write tid 2 to smithi15186934-4 2024-09-13T14:36:52.028 INFO:tasks.rados.rados.0.smithi151.stdout:3788: finishing write tid 3 to smithi15186934-4 2024-09-13T14:36:52.028 INFO:tasks.rados.rados.0.smithi151.stdout:3788: finishing write tid 4 to smithi15186934-4 2024-09-13T14:36:52.028 INFO:tasks.rados.rados.0.smithi151.stdout:3788: finishing write tid 5 to smithi15186934-4 2024-09-13T14:36:52.028 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 4 v 1857 (ObjNum 1427 snap 479 seq_num 1427) dirty exists 2024-09-13T14:36:52.028 INFO:tasks.rados.rados.0.smithi151.stdout:3788: left oid 4 (ObjNum 1427 snap 479 seq_num 1427) 2024-09-13T14:36:52.028 INFO:tasks.rados.rados.0.smithi151.stdout:3789: finishing write tid 1 to smithi15186934-36 2024-09-13T14:36:52.029 INFO:tasks.rados.rados.0.smithi151.stdout:3789: finishing write tid 2 to smithi15186934-36 2024-09-13T14:36:52.029 INFO:tasks.rados.rados.0.smithi151.stdout:3789: finishing write tid 3 to smithi15186934-36 2024-09-13T14:36:52.029 INFO:tasks.rados.rados.0.smithi151.stdout:3789: finishing write tid 4 to smithi15186934-36 2024-09-13T14:36:52.030 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 36 v 829 (ObjNum 1428 snap 479 seq_num 1428) dirty exists 2024-09-13T14:36:52.030 INFO:tasks.rados.rados.0.smithi151.stdout:3789: left oid 36 (ObjNum 1428 snap 479 seq_num 1428) 2024-09-13T14:36:52.703 INFO:tasks.rados.rados.0.smithi151.stdout:3786: done (4 left) 2024-09-13T14:36:52.703 INFO:tasks.rados.rados.0.smithi151.stdout:3787: done (3 left) 2024-09-13T14:36:52.703 INFO:tasks.rados.rados.0.smithi151.stdout:3788: done (2 left) 2024-09-13T14:36:52.703 INFO:tasks.rados.rados.0.smithi151.stdout:3789: done (1 left) 2024-09-13T14:36:52.703 INFO:tasks.rados.rados.0.smithi151.stdout:3790: done (0 left) 2024-09-13T14:36:52.703 INFO:tasks.rados.rados.0.smithi151.stdout:3791: read oid 30 snap 473 2024-09-13T14:36:52.703 INFO:tasks.rados.rados.0.smithi151.stdout:3791: expect deleted 2024-09-13T14:36:52.703 INFO:tasks.rados.rados.0.smithi151.stdout:3792: write oid 29 current snap is 480 2024-09-13T14:36:52.703 INFO:tasks.rados.rados.0.smithi151.stdout:3792: seq_num 1429 ranges {417203=488832,1405239=611964} 2024-09-13T14:36:52.707 INFO:tasks.rados.rados.0.smithi151.stdout:3792: writing smithi15186934-29 from 417203 to 906035 tid 1 2024-09-13T14:36:52.714 INFO:tasks.rados.rados.0.smithi151.stdout:3792: writing smithi15186934-29 from 1405239 to 2017203 tid 2 2024-09-13T14:36:52.714 INFO:tasks.rados.rados.0.smithi151.stdout:3793: snap_remove snap 478 2024-09-13T14:36:53.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:52 smithi184 ceph-mon[92527]: pgmap v1149: 89 pgs: 89 active+clean; 261 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 MiB/s wr, 7 op/s 2024-09-13T14:36:53.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:52 smithi184 ceph-mon[92527]: osdmap e1021: 8 total, 8 up, 8 in 2024-09-13T14:36:53.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:52 smithi151 ceph-mon[104450]: pgmap v1149: 89 pgs: 89 active+clean; 261 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 MiB/s wr, 7 op/s 2024-09-13T14:36:53.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:52 smithi151 ceph-mon[104450]: osdmap e1021: 8 total, 8 up, 8 in 2024-09-13T14:36:53.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:52 smithi151 ceph-mon[108966]: pgmap v1149: 89 pgs: 89 active+clean; 261 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 MiB/s wr, 7 op/s 2024-09-13T14:36:53.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:52 smithi151 ceph-mon[108966]: osdmap e1021: 8 total, 8 up, 8 in 2024-09-13T14:36:53.713 INFO:tasks.rados.rados.0.smithi151.stdout:3793: done (2 left) 2024-09-13T14:36:53.713 INFO:tasks.rados.rados.0.smithi151.stdout:3794: write oid 11 current snap is 480 2024-09-13T14:36:53.713 INFO:tasks.rados.rados.0.smithi151.stdout:3794: seq_num 1430 ranges {434010=680434,1580074=442155} 2024-09-13T14:36:53.719 INFO:tasks.rados.rados.0.smithi151.stdout:3794: writing smithi15186934-11 from 434010 to 1114444 tid 1 2024-09-13T14:36:53.725 INFO:tasks.rados.rados.0.smithi151.stdout:3794: writing smithi15186934-11 from 1580074 to 2022229 tid 2 2024-09-13T14:36:53.725 INFO:tasks.rados.rados.0.smithi151.stdout:3795: write oid 35 current snap is 480 2024-09-13T14:36:53.725 INFO:tasks.rados.rados.0.smithi151.stdout:3795: seq_num 1431 ranges {450817=755683,1638556=412261} 2024-09-13T14:36:53.732 INFO:tasks.rados.rados.0.smithi151.stdout:3795: writing smithi15186934-35 from 450817 to 1206500 tid 1 2024-09-13T14:36:53.737 INFO:tasks.rados.rados.0.smithi151.stdout:3795: writing smithi15186934-35 from 1638556 to 2050817 tid 2 2024-09-13T14:36:53.737 INFO:tasks.rados.rados.0.smithi151.stdout:3796: write oid 25 current snap is 480 2024-09-13T14:36:53.737 INFO:tasks.rados.rados.0.smithi151.stdout:3796: seq_num 1432 ranges {467624=430932,1697038=370586} 2024-09-13T14:36:53.742 INFO:tasks.rados.rados.0.smithi151.stdout:3796: writing smithi15186934-25 from 467624 to 898556 tid 1 2024-09-13T14:36:53.750 INFO:tasks.rados.rados.0.smithi151.stdout:3796: writing smithi15186934-25 from 1697038 to 2067624 tid 2 2024-09-13T14:36:53.751 INFO:tasks.rados.rados.0.smithi151.stdout:3797: rollback oid 47 current snap is 480 2024-09-13T14:36:53.751 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 47 to 476 2024-09-13T14:36:53.751 INFO:tasks.rados.rados.0.smithi151.stdout:3792: finishing write tid 1 to smithi15186934-29 2024-09-13T14:36:53.751 INFO:tasks.rados.rados.0.smithi151.stdout:3792: finishing write tid 2 to smithi15186934-29 2024-09-13T14:36:53.751 INFO:tasks.rados.rados.0.smithi151.stdout:3792: finishing write tid 3 to smithi15186934-29 2024-09-13T14:36:53.751 INFO:tasks.rados.rados.0.smithi151.stdout:3792: finishing write tid 4 to smithi15186934-29 2024-09-13T14:36:53.751 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 29 v 1048 (ObjNum 1429 snap 480 seq_num 1429) dirty exists 2024-09-13T14:36:53.751 INFO:tasks.rados.rados.0.smithi151.stdout:3792: left oid 29 (ObjNum 1429 snap 480 seq_num 1429) 2024-09-13T14:36:53.751 INFO:tasks.rados.rados.0.smithi151.stdout:3794: finishing write tid 1 to smithi15186934-11 2024-09-13T14:36:53.751 INFO:tasks.rados.rados.0.smithi151.stdout:3791: done (5 left) 2024-09-13T14:36:53.751 INFO:tasks.rados.rados.0.smithi151.stdout:3792: done (4 left) 2024-09-13T14:36:53.751 INFO:tasks.rados.rados.0.smithi151.stdout:3798: delete oid 29 current snap is 480 2024-09-13T14:36:53.751 INFO:tasks.rados.rados.0.smithi151.stdout:3794: finishing write tid 2 to smithi15186934-11 2024-09-13T14:36:53.752 INFO:tasks.rados.rados.0.smithi151.stdout:3794: finishing write tid 3 to smithi15186934-11 2024-09-13T14:36:53.752 INFO:tasks.rados.rados.0.smithi151.stdout:3794: finishing write tid 4 to smithi15186934-11 2024-09-13T14:36:53.752 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 11 v 1054 (ObjNum 1430 snap 480 seq_num 1430) dirty exists 2024-09-13T14:36:53.752 INFO:tasks.rados.rados.0.smithi151.stdout:3794: left oid 11 (ObjNum 1430 snap 480 seq_num 1430) 2024-09-13T14:36:53.752 INFO:tasks.rados.rados.0.smithi151.stdout:3795: finishing write tid 1 to smithi15186934-35 2024-09-13T14:36:53.752 INFO:tasks.rados.rados.0.smithi151.stdout:3795: finishing write tid 2 to smithi15186934-35 2024-09-13T14:36:53.752 INFO:tasks.rados.rados.0.smithi151.stdout:3795: finishing write tid 3 to smithi15186934-35 2024-09-13T14:36:53.752 INFO:tasks.rados.rados.0.smithi151.stdout:3795: finishing write tid 4 to smithi15186934-35 2024-09-13T14:36:53.752 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 35 v 2160 (ObjNum 1431 snap 480 seq_num 1431) dirty exists 2024-09-13T14:36:53.752 INFO:tasks.rados.rados.0.smithi151.stdout:3795: left oid 35 (ObjNum 1431 snap 480 seq_num 1431) 2024-09-13T14:36:53.752 INFO:tasks.rados.rados.0.smithi151.stdout:3796: finishing write tid 1 to smithi15186934-25 2024-09-13T14:36:53.753 INFO:tasks.rados.rados.0.smithi151.stdout:3794: done (4 left) 2024-09-13T14:36:53.753 INFO:tasks.rados.rados.0.smithi151.stdout:3795: done (3 left) 2024-09-13T14:36:53.753 INFO:tasks.rados.rados.0.smithi151.stdout:3798: done (2 left) 2024-09-13T14:36:53.753 INFO:tasks.rados.rados.0.smithi151.stdout:3799: write oid 22 current snap is 480 2024-09-13T14:36:53.753 INFO:tasks.rados.rados.0.smithi151.stdout:3799: seq_num 1434 ranges {501238=581430,1697649=403589} 2024-09-13T14:36:53.757 INFO:tasks.rados.rados.0.smithi151.stdout:3799: writing smithi15186934-22 from 501238 to 1082668 tid 1 2024-09-13T14:36:53.764 INFO:tasks.rados.rados.0.smithi151.stdout:3799: writing smithi15186934-22 from 1697649 to 2101238 tid 2 2024-09-13T14:36:53.764 INFO:tasks.rados.rados.0.smithi151.stdout:3800: rollback oid 50 current snap is 480 2024-09-13T14:36:53.764 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 50 to 479 2024-09-13T14:36:53.764 INFO:tasks.rados.rados.0.smithi151.stdout:3796: finishing write tid 2 to smithi15186934-25 2024-09-13T14:36:53.764 INFO:tasks.rados.rados.0.smithi151.stdout:3796: finishing write tid 3 to smithi15186934-25 2024-09-13T14:36:53.764 INFO:tasks.rados.rados.0.smithi151.stdout:3796: finishing write tid 4 to smithi15186934-25 2024-09-13T14:36:53.764 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 1862 (ObjNum 1432 snap 480 seq_num 1432) dirty exists 2024-09-13T14:36:53.764 INFO:tasks.rados.rados.0.smithi151.stdout:3796: left oid 25 (ObjNum 1432 snap 480 seq_num 1432) 2024-09-13T14:36:53.764 INFO:tasks.rados.rados.0.smithi151.stdout:3797: finishing rollback tid 0 to smithi15186934-47 2024-09-13T14:36:53.764 INFO:tasks.rados.rados.0.smithi151.stdout:3797: finishing rollback tid 1 to smithi15186934-47 2024-09-13T14:36:53.765 INFO:tasks.rados.rados.0.smithi151.stdout:3796: done (3 left) 2024-09-13T14:36:53.765 INFO:tasks.rados.rados.0.smithi151.stdout:3801: read oid 11 snap 476 2024-09-13T14:36:53.765 INFO:tasks.rados.rados.0.smithi151.stdout:3801: expect (ObjNum 1393 snap 468 seq_num 1393) 2024-09-13T14:36:53.765 INFO:tasks.rados.rados.0.smithi151.stdout:3802: read oid 21 snap -1 2024-09-13T14:36:53.765 INFO:tasks.rados.rados.0.smithi151.stdout:3802: expect (ObjNum 1318 snap 449 seq_num 1318) 2024-09-13T14:36:53.765 INFO:tasks.rados.rados.0.smithi151.stdout:3803: rollback oid 3 current snap is 480 2024-09-13T14:36:53.765 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 3 to 470 2024-09-13T14:36:53.765 INFO:tasks.rados.rados.0.smithi151.stdout:3797: finishing rollback tid 2 to smithi15186934-47 2024-09-13T14:36:53.765 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 1735 (ObjNum 1384 snap 466 seq_num 1384) dirty exists 2024-09-13T14:36:53.765 INFO:tasks.rados.rados.0.smithi151.stdout:3799: finishing write tid 1 to smithi15186934-22 2024-09-13T14:36:53.765 INFO:tasks.rados.rados.0.smithi151.stdout:3797: done (5 left) 2024-09-13T14:36:53.765 INFO:tasks.rados.rados.0.smithi151.stdout:3804: read oid 9 snap 479 2024-09-13T14:36:53.765 INFO:tasks.rados.rados.0.smithi151.stdout:3804: expect (ObjNum 1410 snap 474 seq_num 1410) 2024-09-13T14:36:53.765 INFO:tasks.rados.rados.0.smithi151.stdout:3805: snap_create 2024-09-13T14:36:53.767 INFO:tasks.rados.rados.0.smithi151.stdout:3799: finishing write tid 2 to smithi15186934-22 2024-09-13T14:36:53.768 INFO:tasks.rados.rados.0.smithi151.stdout:3799: finishing write tid 3 to smithi15186934-22 2024-09-13T14:36:53.768 INFO:tasks.rados.rados.0.smithi151.stdout:3799: finishing write tid 4 to smithi15186934-22 2024-09-13T14:36:53.768 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 22 v 775 (ObjNum 1434 snap 480 seq_num 1434) dirty exists 2024-09-13T14:36:53.768 INFO:tasks.rados.rados.0.smithi151.stdout:3799: left oid 22 (ObjNum 1434 snap 480 seq_num 1434) 2024-09-13T14:36:53.769 INFO:tasks.rados.rados.0.smithi151.stdout:3801: expect (ObjNum 1393 snap 468 seq_num 1393) 2024-09-13T14:36:53.881 INFO:tasks.rados.rados.0.smithi151.stdout:3803: finishing rollback tid 0 to smithi15186934-3 2024-09-13T14:36:53.881 INFO:tasks.rados.rados.0.smithi151.stdout:3800: finishing rollback tid 0 to smithi15186934-50 2024-09-13T14:36:53.881 INFO:tasks.rados.rados.0.smithi151.stdout:3804: expect (ObjNum 1410 snap 474 seq_num 1410) 2024-09-13T14:36:54.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:53 smithi184 ceph-mon[92527]: osdmap e1022: 8 total, 8 up, 8 in 2024-09-13T14:36:54.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:53 smithi151 ceph-mon[108966]: osdmap e1022: 8 total, 8 up, 8 in 2024-09-13T14:36:54.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:53 smithi151 ceph-mon[104450]: osdmap e1022: 8 total, 8 up, 8 in 2024-09-13T14:36:54.193 INFO:tasks.rados.rados.0.smithi151.stdout:3803: finishing rollback tid 1 to smithi15186934-3 2024-09-13T14:36:54.193 INFO:tasks.rados.rados.0.smithi151.stdout:3803: finishing rollback tid 2 to smithi15186934-3 2024-09-13T14:36:54.193 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 1444 (ObjNum 1398 snap 469 seq_num 1398) dirty exists 2024-09-13T14:36:54.193 INFO:tasks.rados.rados.0.smithi151.stdout:3802: expect (ObjNum 1318 snap 449 seq_num 1318) 2024-09-13T14:36:54.353 INFO:tasks.rados.rados.0.smithi151.stdout:3800: finishing rollback tid 1 to smithi15186934-50 2024-09-13T14:36:54.353 INFO:tasks.rados.rados.0.smithi151.stdout:3800: finishing rollback tid 2 to smithi15186934-50 2024-09-13T14:36:54.353 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 50 v 1574 (ObjNum 1103 snap 370 seq_num 1103) dirty exists 2024-09-13T14:36:54.665 INFO:tasks.rados.rados.0.smithi151.stdout:3799: done (6 left) 2024-09-13T14:36:54.665 INFO:tasks.rados.rados.0.smithi151.stdout:3800: done (5 left) 2024-09-13T14:36:54.665 INFO:tasks.rados.rados.0.smithi151.stdout:3801: done (4 left) 2024-09-13T14:36:54.665 INFO:tasks.rados.rados.0.smithi151.stdout:3802: done (3 left) 2024-09-13T14:36:54.665 INFO:tasks.rados.rados.0.smithi151.stdout:3803: done (2 left) 2024-09-13T14:36:54.665 INFO:tasks.rados.rados.0.smithi151.stdout:3804: done (1 left) 2024-09-13T14:36:54.665 INFO:tasks.rados.rados.0.smithi151.stdout:3805: done (0 left) 2024-09-13T14:36:54.665 INFO:tasks.rados.rados.0.smithi151.stdout:3806: snap_create 2024-09-13T14:36:54.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:36:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:36:54] "GET /metrics HTTP/1.1" 200 36508 "" "Prometheus/2.51.0" 2024-09-13T14:36:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:54 smithi151 ceph-mon[104450]: pgmap v1152: 89 pgs: 89 active+clean; 255 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 7.0 MiB/s rd, 2.9 MiB/s wr, 15 op/s 2024-09-13T14:36:54.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:54 smithi151 ceph-mon[104450]: osdmap e1023: 8 total, 8 up, 8 in 2024-09-13T14:36:54.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:54 smithi151 ceph-mon[104450]: osdmap e1024: 8 total, 8 up, 8 in 2024-09-13T14:36:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:54 smithi151 ceph-mon[108966]: pgmap v1152: 89 pgs: 89 active+clean; 255 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 7.0 MiB/s rd, 2.9 MiB/s wr, 15 op/s 2024-09-13T14:36:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:54 smithi151 ceph-mon[108966]: osdmap e1023: 8 total, 8 up, 8 in 2024-09-13T14:36:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:54 smithi151 ceph-mon[108966]: osdmap e1024: 8 total, 8 up, 8 in 2024-09-13T14:36:55.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:54 smithi184 ceph-mon[92527]: pgmap v1152: 89 pgs: 89 active+clean; 255 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 7.0 MiB/s rd, 2.9 MiB/s wr, 15 op/s 2024-09-13T14:36:55.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:54 smithi184 ceph-mon[92527]: osdmap e1023: 8 total, 8 up, 8 in 2024-09-13T14:36:55.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:54 smithi184 ceph-mon[92527]: osdmap e1024: 8 total, 8 up, 8 in 2024-09-13T14:36:55.666 INFO:tasks.rados.rados.0.smithi151.stdout:3806: done (0 left) 2024-09-13T14:36:55.666 INFO:tasks.rados.rados.0.smithi151.stdout:3807: delete oid 32 current snap is 482 2024-09-13T14:36:55.671 INFO:tasks.rados.rados.0.smithi151.stdout:3807: done (0 left) 2024-09-13T14:36:55.671 INFO:tasks.rados.rados.0.smithi151.stdout:3808: write oid 40 current snap is 482 2024-09-13T14:36:55.671 INFO:tasks.rados.rados.0.smithi151.stdout:3808: seq_num 1436 ranges {534852=731928,1814613=320239} 2024-09-13T14:36:55.677 INFO:tasks.rados.rados.0.smithi151.stdout:3808: writing smithi15186934-40 from 534852 to 1266780 tid 1 2024-09-13T14:36:55.682 INFO:tasks.rados.rados.0.smithi151.stdout:3808: writing smithi15186934-40 from 1814613 to 2134852 tid 2 2024-09-13T14:36:55.682 INFO:tasks.rados.rados.0.smithi151.stdout:3809: snap_create 2024-09-13T14:36:55.683 INFO:tasks.rados.rados.0.smithi151.stdout:3808: finishing write tid 1 to smithi15186934-40 2024-09-13T14:36:55.685 INFO:tasks.rados.rados.0.smithi151.stdout:3808: finishing write tid 2 to smithi15186934-40 2024-09-13T14:36:55.686 INFO:tasks.rados.rados.0.smithi151.stdout:3808: finishing write tid 3 to smithi15186934-40 2024-09-13T14:36:55.686 INFO:tasks.rados.rados.0.smithi151.stdout:3808: finishing write tid 4 to smithi15186934-40 2024-09-13T14:36:55.686 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 40 v 1447 (ObjNum 1436 snap 482 seq_num 1436) dirty exists 2024-09-13T14:36:55.686 INFO:tasks.rados.rados.0.smithi151.stdout:3808: left oid 40 (ObjNum 1436 snap 482 seq_num 1436) 2024-09-13T14:36:56.672 INFO:tasks.rados.rados.0.smithi151.stdout:3808: done (1 left) 2024-09-13T14:36:56.672 INFO:tasks.rados.rados.0.smithi151.stdout:3809: done (0 left) 2024-09-13T14:36:56.672 INFO:tasks.rados.rados.0.smithi151.stdout:3810: delete oid 17 current snap is 483 2024-09-13T14:36:56.676 INFO:tasks.rados.rados.0.smithi151.stdout:3810: done (0 left) 2024-09-13T14:36:56.676 INFO:tasks.rados.rados.0.smithi151.stdout:3811: delete oid 17 current snap is 483 2024-09-13T14:36:56.678 INFO:tasks.rados.rados.0.smithi151.stdout:3811: done (0 left) 2024-09-13T14:36:56.678 INFO:tasks.rados.rados.0.smithi151.stdout:3812: read oid 32 snap -1 2024-09-13T14:36:56.678 INFO:tasks.rados.rados.0.smithi151.stdout:3812: expect deleted 2024-09-13T14:36:56.678 INFO:tasks.rados.rados.0.smithi151.stdout:3813: rollback oid 27 current snap is 483 2024-09-13T14:36:56.679 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 27 to 473 2024-09-13T14:36:56.679 INFO:tasks.rados.rados.0.smithi151.stdout:3814: write oid 28 current snap is 483 2024-09-13T14:36:56.679 INFO:tasks.rados.rados.0.smithi151.stdout:3814: seq_num 1439 ranges {585273=674028,1990059=195214} 2024-09-13T14:36:56.685 INFO:tasks.rados.rados.0.smithi151.stdout:3814: writing smithi15186934-28 from 585273 to 1259301 tid 1 2024-09-13T14:36:56.691 INFO:tasks.rados.rados.0.smithi151.stdout:3814: writing smithi15186934-28 from 1990059 to 2185273 tid 2 2024-09-13T14:36:56.691 INFO:tasks.rados.rados.0.smithi151.stdout:3815: read oid 38 snap -1 2024-09-13T14:36:56.691 INFO:tasks.rados.rados.0.smithi151.stdout:3815: expect (ObjNum 1261 snap 428 seq_num 1261) 2024-09-13T14:36:56.691 INFO:tasks.rados.rados.0.smithi151.stdout:3816: snap_remove snap 481 2024-09-13T14:36:56.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:56 smithi151 ceph-mon[104450]: pgmap v1155: 89 pgs: 89 active+clean; 251 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-09-13T14:36:56.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:56 smithi151 ceph-mon[104450]: osdmap e1025: 8 total, 8 up, 8 in 2024-09-13T14:36:56.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:56 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:36:56.947 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:56 smithi151 ceph-mon[108966]: pgmap v1155: 89 pgs: 89 active+clean; 251 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-09-13T14:36:56.947 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:56 smithi151 ceph-mon[108966]: osdmap e1025: 8 total, 8 up, 8 in 2024-09-13T14:36:56.947 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:56 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:36:56.948 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:56 smithi184 ceph-mon[92527]: pgmap v1155: 89 pgs: 89 active+clean; 251 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-09-13T14:36:56.948 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:56 smithi184 ceph-mon[92527]: osdmap e1025: 8 total, 8 up, 8 in 2024-09-13T14:36:56.948 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:56 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:36:57.676 INFO:tasks.rados.rados.0.smithi151.stdout:3816: done (4 left) 2024-09-13T14:36:57.677 INFO:tasks.rados.rados.0.smithi151.stdout:3817: read oid 34 snap -1 2024-09-13T14:36:57.677 INFO:tasks.rados.rados.0.smithi151.stdout:3817: expect (ObjNum 1425 snap 478 seq_num 1425) 2024-09-13T14:36:57.677 INFO:tasks.rados.rados.0.smithi151.stdout:3818: read oid 44 snap 473 2024-09-13T14:36:57.677 INFO:tasks.rados.rados.0.smithi151.stdout:3818: expect (ObjNum 1342 snap 454 seq_num 1342) 2024-09-13T14:36:57.677 INFO:tasks.rados.rados.0.smithi151.stdout:3819: write oid 17 current snap is 483 2024-09-13T14:36:57.677 INFO:tasks.rados.rados.0.smithi151.stdout:3819: seq_num 1440 ranges {602080=749277,2048541=153539} 2024-09-13T14:36:57.684 INFO:tasks.rados.rados.0.smithi151.stdout:3819: writing smithi15186934-17 from 602080 to 1351357 tid 1 2024-09-13T14:36:57.689 INFO:tasks.rados.rados.0.smithi151.stdout:3819: writing smithi15186934-17 from 2048541 to 2202080 tid 2 2024-09-13T14:36:57.689 INFO:tasks.rados.rados.0.smithi151.stdout:3820: write oid 41 current snap is 483 2024-09-13T14:36:57.689 INFO:tasks.rados.rados.0.smithi151.stdout:3820: seq_num 1441 ranges {618887=424526,1590670=604158} 2024-09-13T14:36:57.694 INFO:tasks.rados.rados.0.smithi151.stdout:3820: writing smithi15186934-41 from 618887 to 1043413 tid 1 2024-09-13T14:36:57.702 INFO:tasks.rados.rados.0.smithi151.stdout:3820: writing smithi15186934-41 from 1590670 to 2194828 tid 2 2024-09-13T14:36:57.702 INFO:tasks.rados.rados.0.smithi151.stdout:3821: delete oid 14 current snap is 483 2024-09-13T14:36:57.702 INFO:tasks.rados.rados.0.smithi151.stdout:3813: finishing rollback tid 0 to smithi15186934-27 2024-09-13T14:36:57.702 INFO:tasks.rados.rados.0.smithi151.stdout:3813: finishing rollback tid 1 to smithi15186934-27 2024-09-13T14:36:57.702 INFO:tasks.rados.rados.0.smithi151.stdout:3813: finishing rollback tid 2 to smithi15186934-27 2024-09-13T14:36:57.702 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 27 v 1146 (ObjNum 1406 snap 471 seq_num 1406) dirty exists 2024-09-13T14:36:57.702 INFO:tasks.rados.rados.0.smithi151.stdout:3814: finishing write tid 1 to smithi15186934-28 2024-09-13T14:36:57.703 INFO:tasks.rados.rados.0.smithi151.stdout:3814: finishing write tid 2 to smithi15186934-28 2024-09-13T14:36:57.703 INFO:tasks.rados.rados.0.smithi151.stdout:3814: finishing write tid 3 to smithi15186934-28 2024-09-13T14:36:57.703 INFO:tasks.rados.rados.0.smithi151.stdout:3814: finishing write tid 4 to smithi15186934-28 2024-09-13T14:36:57.703 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 28 v 1421 (ObjNum 1439 snap 483 seq_num 1439) dirty exists 2024-09-13T14:36:57.703 INFO:tasks.rados.rados.0.smithi151.stdout:3814: left oid 28 (ObjNum 1439 snap 483 seq_num 1439) 2024-09-13T14:36:57.703 INFO:tasks.rados.rados.0.smithi151.stdout:3815: expect (ObjNum 1261 snap 428 seq_num 1261) 2024-09-13T14:36:57.841 INFO:tasks.rados.rados.0.smithi151.stdout:3818: expect (ObjNum 1342 snap 454 seq_num 1342) 2024-09-13T14:36:57.967 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:57 smithi151 ceph-mon[104450]: osdmap e1026: 8 total, 8 up, 8 in 2024-09-13T14:36:57.967 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:57 smithi151 ceph-mon[108966]: osdmap e1026: 8 total, 8 up, 8 in 2024-09-13T14:36:57.983 INFO:tasks.rados.rados.0.smithi151.stdout:3817: expect (ObjNum 1425 snap 478 seq_num 1425) 2024-09-13T14:36:58.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:57 smithi184 ceph-mon[92527]: osdmap e1026: 8 total, 8 up, 8 in 2024-09-13T14:36:58.116 INFO:tasks.rados.rados.0.smithi151.stdout:3819: finishing write tid 1 to smithi15186934-17 2024-09-13T14:36:58.116 INFO:tasks.rados.rados.0.smithi151.stdout:3819: finishing write tid 2 to smithi15186934-17 2024-09-13T14:36:58.116 INFO:tasks.rados.rados.0.smithi151.stdout:3819: finishing write tid 3 to smithi15186934-17 2024-09-13T14:36:58.116 INFO:tasks.rados.rados.0.smithi151.stdout:3812: done (8 left) 2024-09-13T14:36:58.116 INFO:tasks.rados.rados.0.smithi151.stdout:3813: done (7 left) 2024-09-13T14:36:58.116 INFO:tasks.rados.rados.0.smithi151.stdout:3814: done (6 left) 2024-09-13T14:36:58.117 INFO:tasks.rados.rados.0.smithi151.stdout:3815: done (5 left) 2024-09-13T14:36:58.117 INFO:tasks.rados.rados.0.smithi151.stdout:3817: done (4 left) 2024-09-13T14:36:58.117 INFO:tasks.rados.rados.0.smithi151.stdout:3818: done (3 left) 2024-09-13T14:36:58.117 INFO:tasks.rados.rados.0.smithi151.stdout:3821: done (2 left) 2024-09-13T14:36:58.117 INFO:tasks.rados.rados.0.smithi151.stdout:3822: delete oid 8 current snap is 483 2024-09-13T14:36:58.117 INFO:tasks.rados.rados.0.smithi151.stdout:3819: finishing write tid 4 to smithi15186934-17 2024-09-13T14:36:58.117 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 17 v 2168 (ObjNum 1440 snap 483 seq_num 1440) dirty exists 2024-09-13T14:36:58.117 INFO:tasks.rados.rados.0.smithi151.stdout:3819: left oid 17 (ObjNum 1440 snap 483 seq_num 1440) 2024-09-13T14:36:58.117 INFO:tasks.rados.rados.0.smithi151.stdout:3820: finishing write tid 1 to smithi15186934-41 2024-09-13T14:36:58.117 INFO:tasks.rados.rados.0.smithi151.stdout:3820: finishing write tid 2 to smithi15186934-41 2024-09-13T14:36:58.117 INFO:tasks.rados.rados.0.smithi151.stdout:3820: finishing write tid 3 to smithi15186934-41 2024-09-13T14:36:58.117 INFO:tasks.rados.rados.0.smithi151.stdout:3820: finishing write tid 4 to smithi15186934-41 2024-09-13T14:36:58.117 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 340 (ObjNum 1441 snap 483 seq_num 1441) dirty exists 2024-09-13T14:36:58.118 INFO:tasks.rados.rados.0.smithi151.stdout:3820: left oid 41 (ObjNum 1441 snap 483 seq_num 1441) 2024-09-13T14:36:58.118 INFO:tasks.rados.rados.0.smithi151.stdout:3819: done (2 left) 2024-09-13T14:36:58.118 INFO:tasks.rados.rados.0.smithi151.stdout:3820: done (1 left) 2024-09-13T14:36:58.118 INFO:tasks.rados.rados.0.smithi151.stdout:3822: done (0 left) 2024-09-13T14:36:58.118 INFO:tasks.rados.rados.0.smithi151.stdout:3823: snap_create 2024-09-13T14:36:58.682 INFO:tasks.rados.rados.0.smithi151.stdout:3823: done (0 left) 2024-09-13T14:36:58.682 INFO:tasks.rados.rados.0.smithi151.stdout:3824: snap_remove snap 476 2024-09-13T14:36:58.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:58 smithi184 ceph-mon[92527]: pgmap v1158: 89 pgs: 89 active+clean; 251 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 2.3 MiB/s wr, 23 op/s 2024-09-13T14:36:58.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:58 smithi184 ceph-mon[92527]: osdmap e1027: 8 total, 8 up, 8 in 2024-09-13T14:36:59.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:58 smithi151 ceph-mon[104450]: pgmap v1158: 89 pgs: 89 active+clean; 251 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 2.3 MiB/s wr, 23 op/s 2024-09-13T14:36:59.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:58 smithi151 ceph-mon[104450]: osdmap e1027: 8 total, 8 up, 8 in 2024-09-13T14:36:59.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:58 smithi151 ceph-mon[108966]: pgmap v1158: 89 pgs: 89 active+clean; 251 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 2.3 MiB/s wr, 23 op/s 2024-09-13T14:36:59.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:58 smithi151 ceph-mon[108966]: osdmap e1027: 8 total, 8 up, 8 in 2024-09-13T14:36:59.666 INFO:tasks.rados.rados.0.smithi151.stdout:3824: done (0 left) 2024-09-13T14:36:59.666 INFO:tasks.rados.rados.0.smithi151.stdout:3825: rollback oid 34 current snap is 484 2024-09-13T14:36:59.666 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 34 to 475 2024-09-13T14:36:59.666 INFO:tasks.rados.rados.0.smithi151.stdout:3826: delete oid 9 current snap is 484 2024-09-13T14:36:59.674 INFO:tasks.rados.rados.0.smithi151.stdout:3826: done (1 left) 2024-09-13T14:36:59.674 INFO:tasks.rados.rados.0.smithi151.stdout:3827: write oid 27 current snap is 484 2024-09-13T14:36:59.674 INFO:tasks.rados.rados.0.smithi151.stdout:3827: seq_num 1445 ranges {686115=441875,1824598=461517} 2024-09-13T14:36:59.681 INFO:tasks.rados.rados.0.smithi151.stdout:3827: writing smithi15186934-27 from 686115 to 1127990 tid 1 2024-09-13T14:36:59.687 INFO:tasks.rados.rados.0.smithi151.stdout:3827: writing smithi15186934-27 from 1824598 to 2286115 tid 2 2024-09-13T14:36:59.688 INFO:tasks.rados.rados.0.smithi151.stdout:3828: delete oid 44 current snap is 484 2024-09-13T14:36:59.688 INFO:tasks.rados.rados.0.smithi151.stdout:3825: finishing rollback tid 0 to smithi15186934-34 2024-09-13T14:36:59.688 INFO:tasks.rados.rados.0.smithi151.stdout:3825: finishing rollback tid 1 to smithi15186934-34 2024-09-13T14:36:59.688 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 1450 (ObjNum 1755290848 snap 32718 seq_num 304) dirty dne 2024-09-13T14:36:59.688 INFO:tasks.rados.rados.0.smithi151.stdout:3827: finishing write tid 1 to smithi15186934-27 2024-09-13T14:36:59.690 INFO:tasks.rados.rados.0.smithi151.stdout:3825: done (2 left) 2024-09-13T14:36:59.690 INFO:tasks.rados.rados.0.smithi151.stdout:3828: done (1 left) 2024-09-13T14:36:59.691 INFO:tasks.rados.rados.0.smithi151.stdout:3829: delete oid 3 current snap is 484 2024-09-13T14:36:59.691 INFO:tasks.rados.rados.0.smithi151.stdout:3827: finishing write tid 2 to smithi15186934-27 2024-09-13T14:36:59.691 INFO:tasks.rados.rados.0.smithi151.stdout:3827: finishing write tid 3 to smithi15186934-27 2024-09-13T14:36:59.691 INFO:tasks.rados.rados.0.smithi151.stdout:3827: finishing write tid 4 to smithi15186934-27 2024-09-13T14:36:59.691 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 27 v 1154 (ObjNum 1445 snap 484 seq_num 1445) dirty exists 2024-09-13T14:36:59.692 INFO:tasks.rados.rados.0.smithi151.stdout:3827: left oid 27 (ObjNum 1445 snap 484 seq_num 1445) 2024-09-13T14:36:59.693 INFO:tasks.rados.rados.0.smithi151.stdout:3827: done (1 left) 2024-09-13T14:36:59.693 INFO:tasks.rados.rados.0.smithi151.stdout:3829: done (0 left) 2024-09-13T14:36:59.693 INFO:tasks.rados.rados.0.smithi151.stdout:3830: read oid 3 snap -1 2024-09-13T14:36:59.693 INFO:tasks.rados.rados.0.smithi151.stdout:3830: expect deleted 2024-09-13T14:36:59.693 INFO:tasks.rados.rados.0.smithi151.stdout:3831: read oid 8 snap 479 2024-09-13T14:36:59.693 INFO:tasks.rados.rados.0.smithi151.stdout:3831: expect (ObjNum 1400 snap 470 seq_num 1400) 2024-09-13T14:36:59.693 INFO:tasks.rados.rados.0.smithi151.stdout:3832: read oid 33 snap -1 2024-09-13T14:36:59.693 INFO:tasks.rados.rados.0.smithi151.stdout:3832: expect (ObjNum 1058 snap 357 seq_num 1058) 2024-09-13T14:36:59.693 INFO:tasks.rados.rados.0.smithi151.stdout:3833: write oid 19 current snap is 484 2024-09-13T14:36:59.693 INFO:tasks.rados.rados.0.smithi151.stdout:3833: seq_num 1448 ranges {736536=667622,2000044=336492} 2024-09-13T14:36:59.700 INFO:tasks.rados.rados.0.smithi151.stdout:3833: writing smithi15186934-19 from 736536 to 1404158 tid 1 2024-09-13T14:36:59.706 INFO:tasks.rados.rados.0.smithi151.stdout:3833: writing smithi15186934-19 from 2000044 to 2336536 tid 2 2024-09-13T14:36:59.706 INFO:tasks.rados.rados.0.smithi151.stdout:3834: snap_create 2024-09-13T14:36:59.706 INFO:tasks.rados.rados.0.smithi151.stdout:3832: expect (ObjNum 1058 snap 357 seq_num 1058) 2024-09-13T14:36:59.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:59 smithi151 ceph-mon[104450]: osdmap e1028: 8 total, 8 up, 8 in 2024-09-13T14:36:59.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:36:59 smithi151 ceph-mon[104450]: osdmap e1029: 8 total, 8 up, 8 in 2024-09-13T14:36:59.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:59 smithi151 ceph-mon[108966]: osdmap e1028: 8 total, 8 up, 8 in 2024-09-13T14:36:59.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:36:59 smithi151 ceph-mon[108966]: osdmap e1029: 8 total, 8 up, 8 in 2024-09-13T14:36:59.885 INFO:tasks.rados.rados.0.smithi151.stdout:3831: expect (ObjNum 1400 snap 470 seq_num 1400) 2024-09-13T14:37:00.005 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:59 smithi184 ceph-mon[92527]: osdmap e1028: 8 total, 8 up, 8 in 2024-09-13T14:37:00.005 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:36:59 smithi184 ceph-mon[92527]: osdmap e1029: 8 total, 8 up, 8 in 2024-09-13T14:37:00.054 INFO:tasks.rados.rados.0.smithi151.stdout:3833: finishing write tid 1 to smithi15186934-19 2024-09-13T14:37:00.054 INFO:tasks.rados.rados.0.smithi151.stdout:3833: finishing write tid 2 to smithi15186934-19 2024-09-13T14:37:00.055 INFO:tasks.rados.rados.0.smithi151.stdout:3833: finishing write tid 3 to smithi15186934-19 2024-09-13T14:37:00.055 INFO:tasks.rados.rados.0.smithi151.stdout:3833: finishing write tid 4 to smithi15186934-19 2024-09-13T14:37:00.055 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 19 v 855 (ObjNum 1448 snap 484 seq_num 1448) dirty exists 2024-09-13T14:37:00.055 INFO:tasks.rados.rados.0.smithi151.stdout:3833: left oid 19 (ObjNum 1448 snap 484 seq_num 1448) 2024-09-13T14:37:00.354 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:37:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:37:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:37:00.691 INFO:tasks.rados.rados.0.smithi151.stdout:3830: done (4 left) 2024-09-13T14:37:00.691 INFO:tasks.rados.rados.0.smithi151.stdout:3831: done (3 left) 2024-09-13T14:37:00.691 INFO:tasks.rados.rados.0.smithi151.stdout:3832: done (2 left) 2024-09-13T14:37:00.691 INFO:tasks.rados.rados.0.smithi151.stdout:3833: done (1 left) 2024-09-13T14:37:00.691 INFO:tasks.rados.rados.0.smithi151.stdout:3834: done (0 left) 2024-09-13T14:37:00.691 INFO:tasks.rados.rados.0.smithi151.stdout:3835: write oid 25 current snap is 485 2024-09-13T14:37:00.691 INFO:tasks.rados.rados.0.smithi151.stdout:3835: seq_num 1449 ranges {753343=742871,1942173=411170} 2024-09-13T14:37:00.698 INFO:tasks.rados.rados.0.smithi151.stdout:3835: writing smithi15186934-25 from 753343 to 1496214 tid 1 2024-09-13T14:37:00.704 INFO:tasks.rados.rados.0.smithi151.stdout:3835: writing smithi15186934-25 from 1942173 to 2353343 tid 2 2024-09-13T14:37:00.704 INFO:tasks.rados.rados.0.smithi151.stdout:3836: read oid 43 snap -1 2024-09-13T14:37:00.704 INFO:tasks.rados.rados.0.smithi151.stdout:3836: expect deleted 2024-09-13T14:37:00.704 INFO:tasks.rados.rados.0.smithi151.stdout:3837: snap_remove snap 484 2024-09-13T14:37:01.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:00 smithi184 ceph-mon[92527]: pgmap v1161: 89 pgs: 89 active+clean; 251 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 7.8 MiB/s rd, 2.1 MiB/s wr, 21 op/s 2024-09-13T14:37:01.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:00 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:37:01.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:00 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:37:01.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:00 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:37:01.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:00 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:37:01.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:00 smithi184 ceph-mon[92527]: osdmap e1030: 8 total, 8 up, 8 in 2024-09-13T14:37:01.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:00 smithi151 ceph-mon[104450]: pgmap v1161: 89 pgs: 89 active+clean; 251 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 7.8 MiB/s rd, 2.1 MiB/s wr, 21 op/s 2024-09-13T14:37:01.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:00 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:37:01.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:00 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:37:01.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:00 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:37:01.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:00 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:37:01.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:00 smithi151 ceph-mon[104450]: osdmap e1030: 8 total, 8 up, 8 in 2024-09-13T14:37:01.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:00 smithi151 ceph-mon[108966]: pgmap v1161: 89 pgs: 89 active+clean; 251 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 7.8 MiB/s rd, 2.1 MiB/s wr, 21 op/s 2024-09-13T14:37:01.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:00 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:37:01.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:00 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:37:01.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:00 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:37:01.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:00 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:37:01.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:00 smithi151 ceph-mon[108966]: osdmap e1030: 8 total, 8 up, 8 in 2024-09-13T14:37:01.695 INFO:tasks.rados.rados.0.smithi151.stdout:3837: done (2 left) 2024-09-13T14:37:01.695 INFO:tasks.rados.rados.0.smithi151.stdout:3838: read oid 15 snap -1 2024-09-13T14:37:01.695 INFO:tasks.rados.rados.0.smithi151.stdout:3838: expect (ObjNum 1310 snap 446 seq_num 1310) 2024-09-13T14:37:01.695 INFO:tasks.rados.rados.0.smithi151.stdout:3839: read oid 11 snap -1 2024-09-13T14:37:01.695 INFO:tasks.rados.rados.0.smithi151.stdout:3839: expect (ObjNum 1430 snap 480 seq_num 1430) 2024-09-13T14:37:01.695 INFO:tasks.rados.rados.0.smithi151.stdout:3840: delete oid 1 current snap is 485 2024-09-13T14:37:01.695 INFO:tasks.rados.rados.0.smithi151.stdout:3835: finishing write tid 1 to smithi15186934-25 2024-09-13T14:37:01.695 INFO:tasks.rados.rados.0.smithi151.stdout:3835: finishing write tid 2 to smithi15186934-25 2024-09-13T14:37:01.695 INFO:tasks.rados.rados.0.smithi151.stdout:3835: finishing write tid 3 to smithi15186934-25 2024-09-13T14:37:01.695 INFO:tasks.rados.rados.0.smithi151.stdout:3835: finishing write tid 4 to smithi15186934-25 2024-09-13T14:37:01.695 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 1868 (ObjNum 1449 snap 485 seq_num 1449) dirty exists 2024-09-13T14:37:01.696 INFO:tasks.rados.rados.0.smithi151.stdout:3835: left oid 25 (ObjNum 1449 snap 485 seq_num 1449) 2024-09-13T14:37:01.698 INFO:tasks.rados.rados.0.smithi151.stdout:3838: expect (ObjNum 1310 snap 446 seq_num 1310) 2024-09-13T14:37:01.765 INFO:tasks.rados.rados.0.smithi151.stdout:3839: expect (ObjNum 1430 snap 480 seq_num 1430) 2024-09-13T14:37:01.859 INFO:tasks.rados.rados.0.smithi151.stdout:3835: done (4 left) 2024-09-13T14:37:01.859 INFO:tasks.rados.rados.0.smithi151.stdout:3836: done (3 left) 2024-09-13T14:37:01.860 INFO:tasks.rados.rados.0.smithi151.stdout:3838: done (2 left) 2024-09-13T14:37:01.860 INFO:tasks.rados.rados.0.smithi151.stdout:3839: done (1 left) 2024-09-13T14:37:01.860 INFO:tasks.rados.rados.0.smithi151.stdout:3840: done (0 left) 2024-09-13T14:37:01.860 INFO:tasks.rados.rados.0.smithi151.stdout:3841: write oid 38 current snap is 485 2024-09-13T14:37:01.860 INFO:tasks.rados.rados.0.smithi151.stdout:3841: seq_num 1451 ranges {786957=493369,2059137=327820} 2024-09-13T14:37:01.866 INFO:tasks.rados.rados.0.smithi151.stdout:3841: writing smithi15186934-38 from 786957 to 1280326 tid 1 2024-09-13T14:37:01.872 INFO:tasks.rados.rados.0.smithi151.stdout:3841: writing smithi15186934-38 from 2059137 to 2386957 tid 2 2024-09-13T14:37:01.872 INFO:tasks.rados.rados.0.smithi151.stdout:3842: snap_remove snap 469 2024-09-13T14:37:02.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:01 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:37:02.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:01 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:37:02.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:01 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:37:02.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:01 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:02.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:01 smithi184 ceph-mon[92527]: osdmap e1031: 8 total, 8 up, 8 in 2024-09-13T14:37:02.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:37:02.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:37:02.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:37:02.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:02.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:01 smithi151 ceph-mon[104450]: osdmap e1031: 8 total, 8 up, 8 in 2024-09-13T14:37:02.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:01 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:37:02.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:01 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:37:02.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:01 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:37:02.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:01 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:02.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:01 smithi151 ceph-mon[108966]: osdmap e1031: 8 total, 8 up, 8 in 2024-09-13T14:37:02.697 INFO:tasks.rados.rados.0.smithi151.stdout:3842: done (1 left) 2024-09-13T14:37:02.697 INFO:tasks.rados.rados.0.smithi151.stdout:3843: read oid 44 snap 475 2024-09-13T14:37:02.697 INFO:tasks.rados.rados.0.smithi151.stdout:3843: expect (ObjNum 1342 snap 454 seq_num 1342) 2024-09-13T14:37:02.697 INFO:tasks.rados.rados.0.smithi151.stdout:3844: snap_create 2024-09-13T14:37:02.697 INFO:tasks.rados.rados.0.smithi151.stdout:3841: finishing write tid 1 to smithi15186934-38 2024-09-13T14:37:02.697 INFO:tasks.rados.rados.0.smithi151.stdout:3841: finishing write tid 2 to smithi15186934-38 2024-09-13T14:37:02.697 INFO:tasks.rados.rados.0.smithi151.stdout:3841: finishing write tid 3 to smithi15186934-38 2024-09-13T14:37:02.697 INFO:tasks.rados.rados.0.smithi151.stdout:3841: finishing write tid 4 to smithi15186934-38 2024-09-13T14:37:02.697 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 38 v 1582 (ObjNum 1451 snap 485 seq_num 1451) dirty exists 2024-09-13T14:37:02.697 INFO:tasks.rados.rados.0.smithi151.stdout:3841: left oid 38 (ObjNum 1451 snap 485 seq_num 1451) 2024-09-13T14:37:02.703 INFO:tasks.rados.rados.0.smithi151.stdout:3843: expect (ObjNum 1342 snap 454 seq_num 1342) 2024-09-13T14:37:03.260 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:03 smithi151 ceph-mon[104450]: pgmap v1164: 89 pgs: 89 active+clean; 254 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 469 KiB/s wr, 11 op/s 2024-09-13T14:37:03.260 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:03 smithi151 ceph-mon[104450]: osdmap e1032: 8 total, 8 up, 8 in 2024-09-13T14:37:03.260 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:03 smithi151 ceph-mon[108966]: pgmap v1164: 89 pgs: 89 active+clean; 254 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 469 KiB/s wr, 11 op/s 2024-09-13T14:37:03.260 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:03 smithi151 ceph-mon[108966]: osdmap e1032: 8 total, 8 up, 8 in 2024-09-13T14:37:03.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:03 smithi184 ceph-mon[92527]: pgmap v1164: 89 pgs: 89 active+clean; 254 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 469 KiB/s wr, 11 op/s 2024-09-13T14:37:03.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:03 smithi184 ceph-mon[92527]: osdmap e1032: 8 total, 8 up, 8 in 2024-09-13T14:37:03.697 INFO:tasks.rados.rados.0.smithi151.stdout:3841: done (2 left) 2024-09-13T14:37:03.697 INFO:tasks.rados.rados.0.smithi151.stdout:3843: done (1 left) 2024-09-13T14:37:03.697 INFO:tasks.rados.rados.0.smithi151.stdout:3844: done (0 left) 2024-09-13T14:37:03.697 INFO:tasks.rados.rados.0.smithi151.stdout:3845: read oid 18 snap -1 2024-09-13T14:37:03.697 INFO:tasks.rados.rados.0.smithi151.stdout:3845: expect deleted 2024-09-13T14:37:03.697 INFO:tasks.rados.rados.0.smithi151.stdout:3846: snap_create 2024-09-13T14:37:04.667 INFO:tasks.rados.rados.0.smithi151.stdout:3845: done (1 left) 2024-09-13T14:37:04.667 INFO:tasks.rados.rados.0.smithi151.stdout:3846: done (0 left) 2024-09-13T14:37:04.667 INFO:tasks.rados.rados.0.smithi151.stdout:3847: write oid 20 current snap is 487 2024-09-13T14:37:04.667 INFO:tasks.rados.rados.0.smithi151.stdout:3847: seq_num 1452 ranges {403764=568618,1717619=686145} 2024-09-13T14:37:04.672 INFO:tasks.rados.rados.0.smithi151.stdout:3847: writing smithi15186934-20 from 403764 to 972382 tid 1 2024-09-13T14:37:04.680 INFO:tasks.rados.rados.0.smithi151.stdout:3847: writing smithi15186934-20 from 1717619 to 2403764 tid 2 2024-09-13T14:37:04.680 INFO:tasks.rados.rados.0.smithi151.stdout:3848: snap_create 2024-09-13T14:37:04.680 INFO:tasks.rados.rados.0.smithi151.stdout:3847: finishing write tid 1 to smithi15186934-20 2024-09-13T14:37:04.684 INFO:tasks.rados.rados.0.smithi151.stdout:3847: finishing write tid 2 to smithi15186934-20 2024-09-13T14:37:04.684 INFO:tasks.rados.rados.0.smithi151.stdout:3847: finishing write tid 3 to smithi15186934-20 2024-09-13T14:37:04.684 INFO:tasks.rados.rados.0.smithi151.stdout:3847: finishing write tid 4 to smithi15186934-20 2024-09-13T14:37:04.684 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 1160 (ObjNum 1452 snap 487 seq_num 1452) dirty exists 2024-09-13T14:37:04.685 INFO:tasks.rados.rados.0.smithi151.stdout:3847: left oid 20 (ObjNum 1452 snap 487 seq_num 1452) 2024-09-13T14:37:04.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:04 smithi151 ceph-mon[104450]: pgmap v1167: 89 pgs: 89 active+clean; 261 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 1.6 MiB/s wr, 17 op/s 2024-09-13T14:37:04.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:04 smithi151 ceph-mon[104450]: osdmap e1033: 8 total, 8 up, 8 in 2024-09-13T14:37:04.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:04 smithi151 ceph-mon[104450]: osdmap e1034: 8 total, 8 up, 8 in 2024-09-13T14:37:04.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:04 smithi151 ceph-mon[108966]: pgmap v1167: 89 pgs: 89 active+clean; 261 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 1.6 MiB/s wr, 17 op/s 2024-09-13T14:37:04.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:04 smithi151 ceph-mon[108966]: osdmap e1033: 8 total, 8 up, 8 in 2024-09-13T14:37:04.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:04 smithi151 ceph-mon[108966]: osdmap e1034: 8 total, 8 up, 8 in 2024-09-13T14:37:04.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:37:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:37:04] "GET /metrics HTTP/1.1" 200 36508 "" "Prometheus/2.51.0" 2024-09-13T14:37:05.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:04 smithi184 ceph-mon[92527]: pgmap v1167: 89 pgs: 89 active+clean; 261 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 1.6 MiB/s wr, 17 op/s 2024-09-13T14:37:05.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:04 smithi184 ceph-mon[92527]: osdmap e1033: 8 total, 8 up, 8 in 2024-09-13T14:37:05.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:04 smithi184 ceph-mon[92527]: osdmap e1034: 8 total, 8 up, 8 in 2024-09-13T14:37:05.668 INFO:tasks.rados.rados.0.smithi151.stdout:3847: done (1 left) 2024-09-13T14:37:05.668 INFO:tasks.rados.rados.0.smithi151.stdout:3848: done (0 left) 2024-09-13T14:37:05.668 INFO:tasks.rados.rados.0.smithi151.stdout:3849: write oid 43 current snap is 488 2024-09-13T14:37:05.668 INFO:tasks.rados.rados.0.smithi151.stdout:3849: seq_num 1453 ranges {420571=760220,1776101=626172} 2024-09-13T14:37:05.674 INFO:tasks.rados.rados.0.smithi151.stdout:3849: writing smithi15186934-43 from 420571 to 1180791 tid 1 2024-09-13T14:37:05.682 INFO:tasks.rados.rados.0.smithi151.stdout:3849: writing smithi15186934-43 from 1776101 to 2402273 tid 2 2024-09-13T14:37:05.682 INFO:tasks.rados.rados.0.smithi151.stdout:3850: rollback oid 16 current snap is 488 2024-09-13T14:37:05.682 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 16 to 473 2024-09-13T14:37:05.682 INFO:tasks.rados.rados.0.smithi151.stdout:3849: finishing write tid 1 to smithi15186934-43 2024-09-13T14:37:05.682 INFO:tasks.rados.rados.0.smithi151.stdout:3851: rollback oid 31 current snap is 488 2024-09-13T14:37:05.683 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 31 to 475 2024-09-13T14:37:05.683 INFO:tasks.rados.rados.0.smithi151.stdout:3852: read oid 4 snap -1 2024-09-13T14:37:05.683 INFO:tasks.rados.rados.0.smithi151.stdout:3852: expect (ObjNum 1427 snap 479 seq_num 1427) 2024-09-13T14:37:05.683 INFO:tasks.rados.rados.0.smithi151.stdout:3853: write oid 29 current snap is 488 2024-09-13T14:37:05.683 INFO:tasks.rados.rados.0.smithi151.stdout:3853: seq_num 1454 ranges {437378=435469,1434583=769830} 2024-09-13T14:37:05.687 INFO:tasks.rados.rados.0.smithi151.stdout:3853: writing smithi15186934-29 from 437378 to 872847 tid 1 2024-09-13T14:37:05.695 INFO:tasks.rados.rados.0.smithi151.stdout:3853: writing smithi15186934-29 from 1434583 to 2204413 tid 2 2024-09-13T14:37:05.696 INFO:tasks.rados.rados.0.smithi151.stdout:3854: write oid 44 current snap is 488 2024-09-13T14:37:05.696 INFO:tasks.rados.rados.0.smithi151.stdout:3850: finishing rollback tid 0 to smithi15186934-16 2024-09-13T14:37:05.696 INFO:tasks.rados.rados.0.smithi151.stdout:3850: finishing rollback tid 1 to smithi15186934-16 2024-09-13T14:37:05.696 INFO:tasks.rados.rados.0.smithi151.stdout:3854: seq_num 1455 ranges {454185=510718,1493065=629841} 2024-09-13T14:37:05.701 INFO:tasks.rados.rados.0.smithi151.stdout:3854: writing smithi15186934-44 from 454185 to 964903 tid 1 2024-09-13T14:37:05.708 INFO:tasks.rados.rados.0.smithi151.stdout:3854: writing smithi15186934-44 from 1493065 to 2122906 tid 2 2024-09-13T14:37:05.708 INFO:tasks.rados.rados.0.smithi151.stdout:3855: read oid 33 snap -1 2024-09-13T14:37:05.708 INFO:tasks.rados.rados.0.smithi151.stdout:3855: expect (ObjNum 1058 snap 357 seq_num 1058) 2024-09-13T14:37:05.708 INFO:tasks.rados.rados.0.smithi151.stdout:3856: snap_create 2024-09-13T14:37:05.708 INFO:tasks.rados.rados.0.smithi151.stdout:3849: finishing write tid 2 to smithi15186934-43 2024-09-13T14:37:05.708 INFO:tasks.rados.rados.0.smithi151.stdout:3850: finishing rollback tid 2 to smithi15186934-16 2024-09-13T14:37:05.708 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 16 v 1164 (ObjNum 1378 snap 465 seq_num 1378) dirty exists 2024-09-13T14:37:05.709 INFO:tasks.rados.rados.0.smithi151.stdout:3849: finishing write tid 3 to smithi15186934-43 2024-09-13T14:37:05.709 INFO:tasks.rados.rados.0.smithi151.stdout:3849: finishing write tid 4 to smithi15186934-43 2024-09-13T14:37:05.709 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 43 v 2177 (ObjNum 1453 snap 488 seq_num 1453) dirty exists 2024-09-13T14:37:05.709 INFO:tasks.rados.rados.0.smithi151.stdout:3849: left oid 43 (ObjNum 1453 snap 488 seq_num 1453) 2024-09-13T14:37:05.709 INFO:tasks.rados.rados.0.smithi151.stdout:3851: finishing rollback tid 0 to smithi15186934-31 2024-09-13T14:37:05.709 INFO:tasks.rados.rados.0.smithi151.stdout:3851: finishing rollback tid 1 to smithi15186934-31 2024-09-13T14:37:05.709 INFO:tasks.rados.rados.0.smithi151.stdout:3851: finishing rollback tid 2 to smithi15186934-31 2024-09-13T14:37:05.709 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 31 v 1431 (ObjNum 1397 snap 468 seq_num 1397) dirty exists 2024-09-13T14:37:05.709 INFO:tasks.rados.rados.0.smithi151.stdout:3853: finishing write tid 1 to smithi15186934-29 2024-09-13T14:37:05.709 INFO:tasks.rados.rados.0.smithi151.stdout:3852: expect (ObjNum 1427 snap 479 seq_num 1427) 2024-09-13T14:37:05.889 INFO:tasks.rados.rados.0.smithi151.stdout:3853: finishing write tid 2 to smithi15186934-29 2024-09-13T14:37:05.889 INFO:tasks.rados.rados.0.smithi151.stdout:3853: finishing write tid 3 to smithi15186934-29 2024-09-13T14:37:05.889 INFO:tasks.rados.rados.0.smithi151.stdout:3853: finishing write tid 4 to smithi15186934-29 2024-09-13T14:37:05.889 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 29 v 1062 (ObjNum 1454 snap 488 seq_num 1454) dirty exists 2024-09-13T14:37:05.890 INFO:tasks.rados.rados.0.smithi151.stdout:3853: left oid 29 (ObjNum 1454 snap 488 seq_num 1454) 2024-09-13T14:37:05.890 INFO:tasks.rados.rados.0.smithi151.stdout:3854: finishing write tid 1 to smithi15186934-44 2024-09-13T14:37:05.890 INFO:tasks.rados.rados.0.smithi151.stdout:3854: finishing write tid 2 to smithi15186934-44 2024-09-13T14:37:05.890 INFO:tasks.rados.rados.0.smithi151.stdout:3854: finishing write tid 3 to smithi15186934-44 2024-09-13T14:37:05.890 INFO:tasks.rados.rados.0.smithi151.stdout:3854: finishing write tid 4 to smithi15186934-44 2024-09-13T14:37:05.890 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 1292 (ObjNum 1455 snap 488 seq_num 1455) dirty exists 2024-09-13T14:37:05.890 INFO:tasks.rados.rados.0.smithi151.stdout:3854: left oid 44 (ObjNum 1455 snap 488 seq_num 1455) 2024-09-13T14:37:05.890 INFO:tasks.rados.rados.0.smithi151.stdout:3855: expect (ObjNum 1058 snap 357 seq_num 1058) 2024-09-13T14:37:06.708 INFO:tasks.rados.rados.0.smithi151.stdout:3849: done (7 left) 2024-09-13T14:37:06.708 INFO:tasks.rados.rados.0.smithi151.stdout:3850: done (6 left) 2024-09-13T14:37:06.708 INFO:tasks.rados.rados.0.smithi151.stdout:3851: done (5 left) 2024-09-13T14:37:06.708 INFO:tasks.rados.rados.0.smithi151.stdout:3852: done (4 left) 2024-09-13T14:37:06.708 INFO:tasks.rados.rados.0.smithi151.stdout:3853: done (3 left) 2024-09-13T14:37:06.708 INFO:tasks.rados.rados.0.smithi151.stdout:3854: done (2 left) 2024-09-13T14:37:06.708 INFO:tasks.rados.rados.0.smithi151.stdout:3855: done (1 left) 2024-09-13T14:37:06.708 INFO:tasks.rados.rados.0.smithi151.stdout:3856: done (0 left) 2024-09-13T14:37:06.708 INFO:tasks.rados.rados.0.smithi151.stdout:3857: write oid 44 current snap is 489 2024-09-13T14:37:06.708 INFO:tasks.rados.rados.0.smithi151.stdout:3857: seq_num 1456 ranges {470992=585967,1551547=773499} 2024-09-13T14:37:06.713 INFO:tasks.rados.rados.0.smithi151.stdout:3857: writing smithi15186934-44 from 470992 to 1056959 tid 1 2024-09-13T14:37:06.721 INFO:tasks.rados.rados.0.smithi151.stdout:3857: writing smithi15186934-44 from 1551547 to 2325046 tid 2 2024-09-13T14:37:06.721 INFO:tasks.rados.rados.0.smithi151.stdout:3857: finishing write tid 1 to smithi15186934-44 2024-09-13T14:37:06.721 INFO:tasks.rados.rados.0.smithi151.stdout:3858: read oid 10 snap 487 2024-09-13T14:37:06.721 INFO:tasks.rados.rados.0.smithi151.stdout:3858: expect (ObjNum 1306 snap 445 seq_num 1306) 2024-09-13T14:37:06.721 INFO:tasks.rados.rados.0.smithi151.stdout:3859: write oid 41 current snap is 489 2024-09-13T14:37:06.721 INFO:tasks.rados.rados.0.smithi151.stdout:3859: seq_num 1457 ranges {487799=661216,1893676=594123} 2024-09-13T14:37:06.727 INFO:tasks.rados.rados.0.smithi151.stdout:3859: writing smithi15186934-41 from 487799 to 1149015 tid 1 2024-09-13T14:37:06.735 INFO:tasks.rados.rados.0.smithi151.stdout:3859: writing smithi15186934-41 from 1893676 to 2487799 tid 2 2024-09-13T14:37:06.735 INFO:tasks.rados.rados.0.smithi151.stdout:3860: delete oid 37 current snap is 489 2024-09-13T14:37:06.735 INFO:tasks.rados.rados.0.smithi151.stdout:3857: finishing write tid 2 to smithi15186934-44 2024-09-13T14:37:06.735 INFO:tasks.rados.rados.0.smithi151.stdout:3857: finishing write tid 3 to smithi15186934-44 2024-09-13T14:37:06.735 INFO:tasks.rados.rados.0.smithi151.stdout:3857: finishing write tid 4 to smithi15186934-44 2024-09-13T14:37:06.735 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 1296 (ObjNum 1456 snap 489 seq_num 1456) dirty exists 2024-09-13T14:37:06.735 INFO:tasks.rados.rados.0.smithi151.stdout:3857: left oid 44 (ObjNum 1456 snap 489 seq_num 1456) 2024-09-13T14:37:06.735 INFO:tasks.rados.rados.0.smithi151.stdout:3858: expect (ObjNum 1306 snap 445 seq_num 1306) 2024-09-13T14:37:07.020 INFO:tasks.rados.rados.0.smithi151.stdout:3859: finishing write tid 1 to smithi15186934-41 2024-09-13T14:37:07.020 INFO:tasks.rados.rados.0.smithi151.stdout:3859: finishing write tid 2 to smithi15186934-41 2024-09-13T14:37:07.020 INFO:tasks.rados.rados.0.smithi151.stdout:3859: finishing write tid 3 to smithi15186934-41 2024-09-13T14:37:07.021 INFO:tasks.rados.rados.0.smithi151.stdout:3859: finishing write tid 4 to smithi15186934-41 2024-09-13T14:37:07.021 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 346 (ObjNum 1457 snap 489 seq_num 1457) dirty exists 2024-09-13T14:37:07.021 INFO:tasks.rados.rados.0.smithi151.stdout:3859: left oid 41 (ObjNum 1457 snap 489 seq_num 1457) 2024-09-13T14:37:07.021 INFO:tasks.rados.rados.0.smithi151.stdout:3857: done (3 left) 2024-09-13T14:37:07.021 INFO:tasks.rados.rados.0.smithi151.stdout:3858: done (2 left) 2024-09-13T14:37:07.021 INFO:tasks.rados.rados.0.smithi151.stdout:3859: done (1 left) 2024-09-13T14:37:07.021 INFO:tasks.rados.rados.0.smithi151.stdout:3860: done (0 left) 2024-09-13T14:37:07.021 INFO:tasks.rados.rados.0.smithi151.stdout:3861: delete oid 18 current snap is 489 2024-09-13T14:37:07.022 INFO:tasks.rados.rados.0.smithi151.stdout:3861: done (0 left) 2024-09-13T14:37:07.022 INFO:tasks.rados.rados.0.smithi151.stdout:3862: delete oid 47 current snap is 489 2024-09-13T14:37:07.025 INFO:tasks.rados.rados.0.smithi151.stdout:3862: done (0 left) 2024-09-13T14:37:07.025 INFO:tasks.rados.rados.0.smithi151.stdout:3863: snap_remove snap 480 2024-09-13T14:37:07.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:06 smithi184 ceph-mon[92527]: pgmap v1170: 89 pgs: 89 active+clean; 259 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.2 MiB/s rd, 1.4 MiB/s wr, 11 op/s 2024-09-13T14:37:07.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:06 smithi184 ceph-mon[92527]: osdmap e1035: 8 total, 8 up, 8 in 2024-09-13T14:37:07.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:06 smithi151 ceph-mon[104450]: pgmap v1170: 89 pgs: 89 active+clean; 259 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.2 MiB/s rd, 1.4 MiB/s wr, 11 op/s 2024-09-13T14:37:07.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:06 smithi151 ceph-mon[104450]: osdmap e1035: 8 total, 8 up, 8 in 2024-09-13T14:37:07.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:06 smithi151 ceph-mon[108966]: pgmap v1170: 89 pgs: 89 active+clean; 259 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6.2 MiB/s rd, 1.4 MiB/s wr, 11 op/s 2024-09-13T14:37:07.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:06 smithi151 ceph-mon[108966]: osdmap e1035: 8 total, 8 up, 8 in 2024-09-13T14:37:07.714 INFO:tasks.rados.rados.0.smithi151.stdout:3863: done (0 left) 2024-09-13T14:37:07.714 INFO:tasks.rados.rados.0.smithi151.stdout:3864: write oid 30 current snap is 489 2024-09-13T14:37:07.714 INFO:tasks.rados.rados.0.smithi151.stdout:3864: seq_num 1461 ranges {555027=678565,1727604=640848} 2024-09-13T14:37:07.721 INFO:tasks.rados.rados.0.smithi151.stdout:3864: writing smithi15186934-30 from 555027 to 1233592 tid 1 2024-09-13T14:37:07.728 INFO:tasks.rados.rados.0.smithi151.stdout:3864: writing smithi15186934-30 from 1727604 to 2368452 tid 2 2024-09-13T14:37:07.728 INFO:tasks.rados.rados.0.smithi151.stdout:3865: snap_remove snap 488 2024-09-13T14:37:08.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:07 smithi184 ceph-mon[92527]: osdmap e1036: 8 total, 8 up, 8 in 2024-09-13T14:37:08.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:07 smithi151 ceph-mon[104450]: osdmap e1036: 8 total, 8 up, 8 in 2024-09-13T14:37:08.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:07 smithi151 ceph-mon[108966]: osdmap e1036: 8 total, 8 up, 8 in 2024-09-13T14:37:08.720 INFO:tasks.rados.rados.0.smithi151.stdout:3865: done (1 left) 2024-09-13T14:37:08.721 INFO:tasks.rados.rados.0.smithi151.stdout:3866: rollback oid 43 current snap is 489 2024-09-13T14:37:08.721 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 43 to 473 2024-09-13T14:37:08.721 INFO:tasks.rados.rados.0.smithi151.stdout:3864: finishing write tid 1 to smithi15186934-30 2024-09-13T14:37:08.721 INFO:tasks.rados.rados.0.smithi151.stdout:3864: finishing write tid 2 to smithi15186934-30 2024-09-13T14:37:08.721 INFO:tasks.rados.rados.0.smithi151.stdout:3864: finishing write tid 3 to smithi15186934-30 2024-09-13T14:37:08.721 INFO:tasks.rados.rados.0.smithi151.stdout:3864: finishing write tid 4 to smithi15186934-30 2024-09-13T14:37:08.721 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 30 v 1464 (ObjNum 1461 snap 489 seq_num 1461) dirty exists 2024-09-13T14:37:08.721 INFO:tasks.rados.rados.0.smithi151.stdout:3864: left oid 30 (ObjNum 1461 snap 489 seq_num 1461) 2024-09-13T14:37:08.721 INFO:tasks.rados.rados.0.smithi151.stdout:3864: done (1 left) 2024-09-13T14:37:08.721 INFO:tasks.rados.rados.0.smithi151.stdout:3867: snap_create 2024-09-13T14:37:08.732 INFO:tasks.rados.rados.0.smithi151.stdout:3866: finishing rollback tid 0 to smithi15186934-43 2024-09-13T14:37:08.732 INFO:tasks.rados.rados.0.smithi151.stdout:3866: finishing rollback tid 1 to smithi15186934-43 2024-09-13T14:37:08.732 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 43 v 2181 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-13T14:37:09.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:08 smithi184 ceph-mon[92527]: pgmap v1173: 89 pgs: 89 active+clean; 258 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 952 KiB/s wr, 15 op/s 2024-09-13T14:37:09.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:08 smithi184 ceph-mon[92527]: osdmap e1037: 8 total, 8 up, 8 in 2024-09-13T14:37:09.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:08 smithi151 ceph-mon[104450]: pgmap v1173: 89 pgs: 89 active+clean; 258 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 952 KiB/s wr, 15 op/s 2024-09-13T14:37:09.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:08 smithi151 ceph-mon[104450]: osdmap e1037: 8 total, 8 up, 8 in 2024-09-13T14:37:09.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:08 smithi151 ceph-mon[108966]: pgmap v1173: 89 pgs: 89 active+clean; 258 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 952 KiB/s wr, 15 op/s 2024-09-13T14:37:09.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:08 smithi151 ceph-mon[108966]: osdmap e1037: 8 total, 8 up, 8 in 2024-09-13T14:37:09.669 INFO:tasks.rados.rados.0.smithi151.stdout:3866: done (1 left) 2024-09-13T14:37:09.669 INFO:tasks.rados.rados.0.smithi151.stdout:3867: done (0 left) 2024-09-13T14:37:09.669 INFO:tasks.rados.rados.0.smithi151.stdout:3868: snap_create 2024-09-13T14:37:10.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:09 smithi184 ceph-mon[92527]: osdmap e1038: 8 total, 8 up, 8 in 2024-09-13T14:37:10.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:09 smithi184 ceph-mon[92527]: osdmap e1039: 8 total, 8 up, 8 in 2024-09-13T14:37:10.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:09 smithi151 ceph-mon[108966]: osdmap e1038: 8 total, 8 up, 8 in 2024-09-13T14:37:10.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:09 smithi151 ceph-mon[108966]: osdmap e1039: 8 total, 8 up, 8 in 2024-09-13T14:37:10.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:09 smithi151 ceph-mon[104450]: osdmap e1038: 8 total, 8 up, 8 in 2024-09-13T14:37:10.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:09 smithi151 ceph-mon[104450]: osdmap e1039: 8 total, 8 up, 8 in 2024-09-13T14:37:10.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:37:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:37:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:37:10.670 INFO:tasks.rados.rados.0.smithi151.stdout:3868: done (0 left) 2024-09-13T14:37:10.670 INFO:tasks.rados.rados.0.smithi151.stdout:3869: write oid 3 current snap is 491 2024-09-13T14:37:10.670 INFO:tasks.rados.rados.0.smithi151.stdout:3869: seq_num 1462 ranges {571834=753814,1786086=784506} 2024-09-13T14:37:10.677 INFO:tasks.rados.rados.0.smithi151.stdout:3869: writing smithi15186934-3 from 571834 to 1325648 tid 1 2024-09-13T14:37:10.685 INFO:tasks.rados.rados.0.smithi151.stdout:3869: writing smithi15186934-3 from 1786086 to 2570592 tid 2 2024-09-13T14:37:10.685 INFO:tasks.rados.rados.0.smithi151.stdout:3870: snap_remove snap 485 2024-09-13T14:37:11.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:10 smithi184 ceph-mon[92527]: pgmap v1176: 89 pgs: 89 active+clean; 258 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 952 KiB/s wr, 16 op/s 2024-09-13T14:37:11.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:10 smithi184 ceph-mon[92527]: osdmap e1040: 8 total, 8 up, 8 in 2024-09-13T14:37:11.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:10 smithi151 ceph-mon[104450]: pgmap v1176: 89 pgs: 89 active+clean; 258 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 952 KiB/s wr, 16 op/s 2024-09-13T14:37:11.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:10 smithi151 ceph-mon[104450]: osdmap e1040: 8 total, 8 up, 8 in 2024-09-13T14:37:11.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:10 smithi151 ceph-mon[108966]: pgmap v1176: 89 pgs: 89 active+clean; 258 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 952 KiB/s wr, 16 op/s 2024-09-13T14:37:11.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:10 smithi151 ceph-mon[108966]: osdmap e1040: 8 total, 8 up, 8 in 2024-09-13T14:37:11.674 INFO:tasks.rados.rados.0.smithi151.stdout:3870: done (1 left) 2024-09-13T14:37:11.674 INFO:tasks.rados.rados.0.smithi151.stdout:3871: snap_remove snap 413 2024-09-13T14:37:12.676 INFO:tasks.rados.rados.0.smithi151.stdout:3871: done (1 left) 2024-09-13T14:37:12.676 INFO:tasks.rados.rados.0.smithi151.stdout:3872: delete oid 45 current snap is 491 2024-09-13T14:37:12.677 INFO:tasks.rados.rados.0.smithi151.stdout:3869: finishing write tid 1 to smithi15186934-3 2024-09-13T14:37:12.677 INFO:tasks.rados.rados.0.smithi151.stdout:3869: finishing write tid 2 to smithi15186934-3 2024-09-13T14:37:12.677 INFO:tasks.rados.rados.0.smithi151.stdout:3869: finishing write tid 3 to smithi15186934-3 2024-09-13T14:37:12.677 INFO:tasks.rados.rados.0.smithi151.stdout:3869: finishing write tid 4 to smithi15186934-3 2024-09-13T14:37:12.677 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 1469 (ObjNum 1462 snap 491 seq_num 1462) dirty exists 2024-09-13T14:37:12.677 INFO:tasks.rados.rados.0.smithi151.stdout:3869: left oid 3 (ObjNum 1462 snap 491 seq_num 1462) 2024-09-13T14:37:12.689 INFO:tasks.rados.rados.0.smithi151.stdout:3869: done (1 left) 2024-09-13T14:37:12.689 INFO:tasks.rados.rados.0.smithi151.stdout:3872: done (0 left) 2024-09-13T14:37:12.689 INFO:tasks.rados.rados.0.smithi151.stdout:3873: rollback oid 39 current snap is 491 2024-09-13T14:37:12.689 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 39 to 473 2024-09-13T14:37:12.689 INFO:tasks.rados.rados.0.smithi151.stdout:3874: write oid 50 current snap is 491 2024-09-13T14:37:12.689 INFO:tasks.rados.rados.0.smithi151.stdout:3874: seq_num 1464 ranges {605448=504312,1903050=702398} 2024-09-13T14:37:12.696 INFO:tasks.rados.rados.0.smithi151.stdout:3874: writing smithi15186934-50 from 605448 to 1109760 tid 1 2024-09-13T14:37:12.705 INFO:tasks.rados.rados.0.smithi151.stdout:3874: writing smithi15186934-50 from 1903050 to 2605448 tid 2 2024-09-13T14:37:12.705 INFO:tasks.rados.rados.0.smithi151.stdout:3875: write oid 37 current snap is 491 2024-09-13T14:37:12.705 INFO:tasks.rados.rados.0.smithi151.stdout:3873: finishing rollback tid 0 to smithi15186934-39 2024-09-13T14:37:12.705 INFO:tasks.rados.rados.0.smithi151.stdout:3873: finishing rollback tid 1 to smithi15186934-39 2024-09-13T14:37:12.705 INFO:tasks.rados.rados.0.smithi151.stdout:3873: finishing rollback tid 2 to smithi15186934-39 2024-09-13T14:37:12.705 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 39 v 1598 (ObjNum 1324 snap 450 seq_num 1324) dirty exists 2024-09-13T14:37:12.705 INFO:tasks.rados.rados.0.smithi151.stdout:3874: finishing write tid 1 to smithi15186934-50 2024-09-13T14:37:12.705 INFO:tasks.rados.rados.0.smithi151.stdout:3875: seq_num 1465 ranges {622255=579561,1845179=531833} 2024-09-13T14:37:12.711 INFO:tasks.rados.rados.0.smithi151.stdout:3875: writing smithi15186934-37 from 622255 to 1201816 tid 1 2024-09-13T14:37:12.717 INFO:tasks.rados.rados.0.smithi151.stdout:3875: writing smithi15186934-37 from 1845179 to 2377012 tid 2 2024-09-13T14:37:12.718 INFO:tasks.rados.rados.0.smithi151.stdout:3873: done (2 left) 2024-09-13T14:37:12.718 INFO:tasks.rados.rados.0.smithi151.stdout:3876: snap_remove snap 483 2024-09-13T14:37:13.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:12 smithi184 ceph-mon[92527]: pgmap v1179: 89 pgs: 89 active+clean; 264 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 7.1 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:37:13.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:12 smithi184 ceph-mon[92527]: osdmap e1041: 8 total, 8 up, 8 in 2024-09-13T14:37:13.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:12 smithi151 ceph-mon[104450]: pgmap v1179: 89 pgs: 89 active+clean; 264 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 7.1 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:37:13.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:12 smithi151 ceph-mon[104450]: osdmap e1041: 8 total, 8 up, 8 in 2024-09-13T14:37:13.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:12 smithi151 ceph-mon[108966]: pgmap v1179: 89 pgs: 89 active+clean; 264 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 7.1 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:37:13.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:12 smithi151 ceph-mon[108966]: osdmap e1041: 8 total, 8 up, 8 in 2024-09-13T14:37:13.693 INFO:tasks.rados.rados.0.smithi151.stdout:3876: done (2 left) 2024-09-13T14:37:13.694 INFO:tasks.rados.rados.0.smithi151.stdout:3877: snap_create 2024-09-13T14:37:13.694 INFO:tasks.rados.rados.0.smithi151.stdout:3874: finishing write tid 2 to smithi15186934-50 2024-09-13T14:37:13.694 INFO:tasks.rados.rados.0.smithi151.stdout:3874: finishing write tid 3 to smithi15186934-50 2024-09-13T14:37:13.694 INFO:tasks.rados.rados.0.smithi151.stdout:3874: finishing write tid 4 to smithi15186934-50 2024-09-13T14:37:13.694 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 50 v 1604 (ObjNum 1464 snap 491 seq_num 1464) dirty exists 2024-09-13T14:37:13.694 INFO:tasks.rados.rados.0.smithi151.stdout:3874: left oid 50 (ObjNum 1464 snap 491 seq_num 1464) 2024-09-13T14:37:13.694 INFO:tasks.rados.rados.0.smithi151.stdout:3875: finishing write tid 1 to smithi15186934-37 2024-09-13T14:37:13.694 INFO:tasks.rados.rados.0.smithi151.stdout:3875: finishing write tid 2 to smithi15186934-37 2024-09-13T14:37:13.694 INFO:tasks.rados.rados.0.smithi151.stdout:3875: finishing write tid 3 to smithi15186934-37 2024-09-13T14:37:13.694 INFO:tasks.rados.rados.0.smithi151.stdout:3875: finishing write tid 4 to smithi15186934-37 2024-09-13T14:37:13.694 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 37 v 1762 (ObjNum 1465 snap 491 seq_num 1465) dirty exists 2024-09-13T14:37:13.694 INFO:tasks.rados.rados.0.smithi151.stdout:3875: left oid 37 (ObjNum 1465 snap 491 seq_num 1465) 2024-09-13T14:37:13.996 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:13 smithi151 ceph-mon[104450]: osdmap e1042: 8 total, 8 up, 8 in 2024-09-13T14:37:13.996 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:13 smithi151 ceph-mon[108966]: osdmap e1042: 8 total, 8 up, 8 in 2024-09-13T14:37:14.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:13 smithi184 ceph-mon[92527]: osdmap e1042: 8 total, 8 up, 8 in 2024-09-13T14:37:14.674 INFO:tasks.rados.rados.0.smithi151.stdout:3874: done (2 left) 2024-09-13T14:37:14.675 INFO:tasks.rados.rados.0.smithi151.stdout:3875: done (1 left) 2024-09-13T14:37:14.675 INFO:tasks.rados.rados.0.smithi151.stdout:3877: done (0 left) 2024-09-13T14:37:14.675 INFO:tasks.rados.rados.0.smithi151.stdout:3878: snap_create 2024-09-13T14:37:14.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:14 smithi151 ceph-mon[108966]: pgmap v1182: 89 pgs: 89 active+clean; 262 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 1.2 MiB/s wr, 5 op/s 2024-09-13T14:37:14.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:14 smithi151 ceph-mon[108966]: osdmap e1043: 8 total, 8 up, 8 in 2024-09-13T14:37:14.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:14 smithi151 ceph-mon[108966]: osdmap e1044: 8 total, 8 up, 8 in 2024-09-13T14:37:14.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:14 smithi151 ceph-mon[104450]: pgmap v1182: 89 pgs: 89 active+clean; 262 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 1.2 MiB/s wr, 5 op/s 2024-09-13T14:37:14.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:14 smithi151 ceph-mon[104450]: osdmap e1043: 8 total, 8 up, 8 in 2024-09-13T14:37:14.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:14 smithi151 ceph-mon[104450]: osdmap e1044: 8 total, 8 up, 8 in 2024-09-13T14:37:14.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:37:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:37:14] "GET /metrics HTTP/1.1" 200 36508 "" "Prometheus/2.51.0" 2024-09-13T14:37:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:14 smithi184 ceph-mon[92527]: pgmap v1182: 89 pgs: 89 active+clean; 262 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 1.2 MiB/s wr, 5 op/s 2024-09-13T14:37:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:14 smithi184 ceph-mon[92527]: osdmap e1043: 8 total, 8 up, 8 in 2024-09-13T14:37:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:14 smithi184 ceph-mon[92527]: osdmap e1044: 8 total, 8 up, 8 in 2024-09-13T14:37:15.671 INFO:tasks.rados.rados.0.smithi151.stdout:3878: done (0 left) 2024-09-13T14:37:15.671 INFO:tasks.rados.rados.0.smithi151.stdout:3879: snap_create 2024-09-13T14:37:16.675 INFO:tasks.rados.rados.0.smithi151.stdout:3879: done (0 left) 2024-09-13T14:37:16.676 INFO:tasks.rados.rados.0.smithi151.stdout:3880: write oid 5 current snap is 494 2024-09-13T14:37:16.676 INFO:tasks.rados.rados.0.smithi151.stdout:3880: seq_num 1466 ranges {639062=654810,1903661=735401} 2024-09-13T14:37:16.682 INFO:tasks.rados.rados.0.smithi151.stdout:3880: writing smithi15186934-5 from 639062 to 1293872 tid 1 2024-09-13T14:37:16.690 INFO:tasks.rados.rados.0.smithi151.stdout:3880: writing smithi15186934-5 from 1903661 to 2639062 tid 2 2024-09-13T14:37:16.690 INFO:tasks.rados.rados.0.smithi151.stdout:3881: write oid 16 current snap is 494 2024-09-13T14:37:16.690 INFO:tasks.rados.rados.0.smithi151.stdout:3881: seq_num 1467 ranges {655869=730059,1962143=535502} 2024-09-13T14:37:16.697 INFO:tasks.rados.rados.0.smithi151.stdout:3881: writing smithi15186934-16 from 655869 to 1385928 tid 1 2024-09-13T14:37:16.704 INFO:tasks.rados.rados.0.smithi151.stdout:3881: writing smithi15186934-16 from 1962143 to 2497645 tid 2 2024-09-13T14:37:16.704 INFO:tasks.rados.rados.0.smithi151.stdout:3882: write oid 46 current snap is 494 2024-09-13T14:37:16.704 INFO:tasks.rados.rados.0.smithi151.stdout:3880: finishing write tid 1 to smithi15186934-5 2024-09-13T14:37:16.704 INFO:tasks.rados.rados.0.smithi151.stdout:3880: finishing write tid 2 to smithi15186934-5 2024-09-13T14:37:16.704 INFO:tasks.rados.rados.0.smithi151.stdout:3882: seq_num 1468 ranges {672676=521661,1736978=795513} 2024-09-13T14:37:16.710 INFO:tasks.rados.rados.0.smithi151.stdout:3882: writing smithi15186934-46 from 672676 to 1194337 tid 1 2024-09-13T14:37:16.718 INFO:tasks.rados.rados.0.smithi151.stdout:3882: writing smithi15186934-46 from 1736978 to 2532491 tid 2 2024-09-13T14:37:16.718 INFO:tasks.rados.rados.0.smithi151.stdout:3883: read oid 38 snap -1 2024-09-13T14:37:16.718 INFO:tasks.rados.rados.0.smithi151.stdout:3883: expect (ObjNum 1451 snap 485 seq_num 1451) 2024-09-13T14:37:16.718 INFO:tasks.rados.rados.0.smithi151.stdout:3884: delete oid 48 current snap is 494 2024-09-13T14:37:16.718 INFO:tasks.rados.rados.0.smithi151.stdout:3880: finishing write tid 3 to smithi15186934-5 2024-09-13T14:37:16.718 INFO:tasks.rados.rados.0.smithi151.stdout:3880: finishing write tid 4 to smithi15186934-5 2024-09-13T14:37:16.718 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 1565 (ObjNum 1466 snap 494 seq_num 1466) dirty exists 2024-09-13T14:37:16.718 INFO:tasks.rados.rados.0.smithi151.stdout:3880: left oid 5 (ObjNum 1466 snap 494 seq_num 1466) 2024-09-13T14:37:16.719 INFO:tasks.rados.rados.0.smithi151.stdout:3881: finishing write tid 1 to smithi15186934-16 2024-09-13T14:37:16.719 INFO:tasks.rados.rados.0.smithi151.stdout:3881: finishing write tid 2 to smithi15186934-16 2024-09-13T14:37:16.719 INFO:tasks.rados.rados.0.smithi151.stdout:3881: finishing write tid 3 to smithi15186934-16 2024-09-13T14:37:16.719 INFO:tasks.rados.rados.0.smithi151.stdout:3881: finishing write tid 4 to smithi15186934-16 2024-09-13T14:37:16.719 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 16 v 1188 (ObjNum 1467 snap 494 seq_num 1467) dirty exists 2024-09-13T14:37:16.719 INFO:tasks.rados.rados.0.smithi151.stdout:3881: left oid 16 (ObjNum 1467 snap 494 seq_num 1467) 2024-09-13T14:37:16.719 INFO:tasks.rados.rados.0.smithi151.stdout:3882: finishing write tid 1 to smithi15186934-46 2024-09-13T14:37:16.721 INFO:tasks.rados.rados.0.smithi151.stdout:3880: done (4 left) 2024-09-13T14:37:16.721 INFO:tasks.rados.rados.0.smithi151.stdout:3881: done (3 left) 2024-09-13T14:37:16.721 INFO:tasks.rados.rados.0.smithi151.stdout:3884: done (2 left) 2024-09-13T14:37:16.721 INFO:tasks.rados.rados.0.smithi151.stdout:3885: read oid 3 snap -1 2024-09-13T14:37:16.721 INFO:tasks.rados.rados.0.smithi151.stdout:3885: expect (ObjNum 1462 snap 491 seq_num 1462) 2024-09-13T14:37:16.721 INFO:tasks.rados.rados.0.smithi151.stdout:3886: write oid 15 current snap is 494 2024-09-13T14:37:16.721 INFO:tasks.rados.rados.0.smithi151.stdout:3886: seq_num 1470 ranges {706290=672159,2137589=568701} 2024-09-13T14:37:16.728 INFO:tasks.rados.rados.0.smithi151.stdout:3886: writing smithi15186934-15 from 706290 to 1378449 tid 1 2024-09-13T14:37:16.736 INFO:tasks.rados.rados.0.smithi151.stdout:3886: writing smithi15186934-15 from 2137589 to 2706290 tid 2 2024-09-13T14:37:16.736 INFO:tasks.rados.rados.0.smithi151.stdout:3887: delete oid 5 current snap is 494 2024-09-13T14:37:16.736 INFO:tasks.rados.rados.0.smithi151.stdout:3882: finishing write tid 2 to smithi15186934-46 2024-09-13T14:37:16.736 INFO:tasks.rados.rados.0.smithi151.stdout:3882: finishing write tid 3 to smithi15186934-46 2024-09-13T14:37:16.736 INFO:tasks.rados.rados.0.smithi151.stdout:3882: finishing write tid 4 to smithi15186934-46 2024-09-13T14:37:16.736 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 46 v 1362 (ObjNum 1468 snap 494 seq_num 1468) dirty exists 2024-09-13T14:37:16.736 INFO:tasks.rados.rados.0.smithi151.stdout:3882: left oid 46 (ObjNum 1468 snap 494 seq_num 1468) 2024-09-13T14:37:16.737 INFO:tasks.rados.rados.0.smithi151.stdout:3883: expect (ObjNum 1451 snap 485 seq_num 1451) 2024-09-13T14:37:16.870 INFO:tasks.rados.rados.0.smithi151.stdout:3885: expect (ObjNum 1462 snap 491 seq_num 1462) 2024-09-13T14:37:16.949 INFO:tasks.rados.rados.0.smithi151.stdout:3886: finishing write tid 1 to smithi15186934-15 2024-09-13T14:37:16.949 INFO:tasks.rados.rados.0.smithi151.stdout:3886: finishing write tid 2 to smithi15186934-15 2024-09-13T14:37:16.949 INFO:tasks.rados.rados.0.smithi151.stdout:3886: finishing write tid 3 to smithi15186934-15 2024-09-13T14:37:16.949 INFO:tasks.rados.rados.0.smithi151.stdout:3886: finishing write tid 4 to smithi15186934-15 2024-09-13T14:37:16.949 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 15 v 1447 (ObjNum 1470 snap 494 seq_num 1470) dirty exists 2024-09-13T14:37:16.949 INFO:tasks.rados.rados.0.smithi151.stdout:3886: left oid 15 (ObjNum 1470 snap 494 seq_num 1470) 2024-09-13T14:37:16.949 INFO:tasks.rados.rados.0.smithi151.stdout:3882: done (4 left) 2024-09-13T14:37:16.949 INFO:tasks.rados.rados.0.smithi151.stdout:3883: done (3 left) 2024-09-13T14:37:16.949 INFO:tasks.rados.rados.0.smithi151.stdout:3885: done (2 left) 2024-09-13T14:37:16.949 INFO:tasks.rados.rados.0.smithi151.stdout:3886: done (1 left) 2024-09-13T14:37:16.950 INFO:tasks.rados.rados.0.smithi151.stdout:3887: done (0 left) 2024-09-13T14:37:16.950 INFO:tasks.rados.rados.0.smithi151.stdout:3888: write oid 47 current snap is 494 2024-09-13T14:37:16.950 INFO:tasks.rados.rados.0.smithi151.stdout:3888: seq_num 1472 ranges {739904=422657,1854553=402851} 2024-09-13T14:37:16.955 INFO:tasks.rados.rados.0.smithi151.stdout:3888: writing smithi15186934-47 from 739904 to 1162561 tid 1 2024-09-13T14:37:16.961 INFO:tasks.rados.rados.0.smithi151.stdout:3888: writing smithi15186934-47 from 1854553 to 2257404 tid 2 2024-09-13T14:37:16.961 INFO:tasks.rados.rados.0.smithi151.stdout:3889: read oid 49 snap -1 2024-09-13T14:37:16.961 INFO:tasks.rados.rados.0.smithi151.stdout:3889: expect deleted 2024-09-13T14:37:16.961 INFO:tasks.rados.rados.0.smithi151.stdout:3890: write oid 23 current snap is 494 2024-09-13T14:37:16.961 INFO:tasks.rados.rados.0.smithi151.stdout:3888: finishing write tid 1 to smithi15186934-47 2024-09-13T14:37:16.961 INFO:tasks.rados.rados.0.smithi151.stdout:3890: seq_num 1473 ranges {756711=497906,1913035=546509} 2024-09-13T14:37:16.968 INFO:tasks.rados.rados.0.smithi151.stdout:3890: writing smithi15186934-23 from 756711 to 1254617 tid 1 2024-09-13T14:37:16.974 INFO:tasks.rados.rados.0.smithi151.stdout:3890: writing smithi15186934-23 from 1913035 to 2459544 tid 2 2024-09-13T14:37:16.974 INFO:tasks.rados.rados.0.smithi151.stdout:3891: write oid 44 current snap is 494 2024-09-13T14:37:16.975 INFO:tasks.rados.rados.0.smithi151.stdout:3891: seq_num 1474 ranges {773518=573155,1855164=406520} 2024-09-13T14:37:16.982 INFO:tasks.rados.rados.0.smithi151.stdout:3891: writing smithi15186934-44 from 773518 to 1346673 tid 1 2024-09-13T14:37:16.987 INFO:tasks.rados.rados.0.smithi151.stdout:3891: writing smithi15186934-44 from 1855164 to 2261684 tid 2 2024-09-13T14:37:16.987 INFO:tasks.rados.rados.0.smithi151.stdout:3892: snap_create 2024-09-13T14:37:16.987 INFO:tasks.rados.rados.0.smithi151.stdout:3888: finishing write tid 2 to smithi15186934-47 2024-09-13T14:37:16.987 INFO:tasks.rados.rados.0.smithi151.stdout:3888: finishing write tid 3 to smithi15186934-47 2024-09-13T14:37:16.987 INFO:tasks.rados.rados.0.smithi151.stdout:3888: finishing write tid 4 to smithi15186934-47 2024-09-13T14:37:16.988 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 1769 (ObjNum 1472 snap 494 seq_num 1472) dirty exists 2024-09-13T14:37:16.988 INFO:tasks.rados.rados.0.smithi151.stdout:3888: left oid 47 (ObjNum 1472 snap 494 seq_num 1472) 2024-09-13T14:37:16.988 INFO:tasks.rados.rados.0.smithi151.stdout:3890: finishing write tid 1 to smithi15186934-23 2024-09-13T14:37:16.988 INFO:tasks.rados.rados.0.smithi151.stdout:3890: finishing write tid 2 to smithi15186934-23 2024-09-13T14:37:16.988 INFO:tasks.rados.rados.0.smithi151.stdout:3890: finishing write tid 3 to smithi15186934-23 2024-09-13T14:37:16.988 INFO:tasks.rados.rados.0.smithi151.stdout:3890: finishing write tid 4 to smithi15186934-23 2024-09-13T14:37:16.988 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 23 v 1773 (ObjNum 1473 snap 494 seq_num 1473) dirty exists 2024-09-13T14:37:16.988 INFO:tasks.rados.rados.0.smithi151.stdout:3890: left oid 23 (ObjNum 1473 snap 494 seq_num 1473) 2024-09-13T14:37:16.988 INFO:tasks.rados.rados.0.smithi151.stdout:3891: finishing write tid 1 to smithi15186934-44 2024-09-13T14:37:16.991 INFO:tasks.rados.rados.0.smithi151.stdout:3891: finishing write tid 2 to smithi15186934-44 2024-09-13T14:37:16.991 INFO:tasks.rados.rados.0.smithi151.stdout:3891: finishing write tid 3 to smithi15186934-44 2024-09-13T14:37:16.991 INFO:tasks.rados.rados.0.smithi151.stdout:3891: finishing write tid 4 to smithi15186934-44 2024-09-13T14:37:16.991 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 1318 (ObjNum 1474 snap 494 seq_num 1474) dirty exists 2024-09-13T14:37:16.991 INFO:tasks.rados.rados.0.smithi151.stdout:3891: left oid 44 (ObjNum 1474 snap 494 seq_num 1474) 2024-09-13T14:37:17.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:16 smithi184 ceph-mon[92527]: pgmap v1185: 89 pgs: 89 active+clean; 242 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:37:17.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:16 smithi184 ceph-mon[92527]: osdmap e1045: 8 total, 8 up, 8 in 2024-09-13T14:37:17.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:17.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:16 smithi151 ceph-mon[108966]: pgmap v1185: 89 pgs: 89 active+clean; 242 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:37:17.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:16 smithi151 ceph-mon[108966]: osdmap e1045: 8 total, 8 up, 8 in 2024-09-13T14:37:17.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:17.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:16 smithi151 ceph-mon[104450]: pgmap v1185: 89 pgs: 89 active+clean; 242 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:37:17.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:16 smithi151 ceph-mon[104450]: osdmap e1045: 8 total, 8 up, 8 in 2024-09-13T14:37:17.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:17.690 INFO:tasks.rados.rados.0.smithi151.stdout:3888: done (4 left) 2024-09-13T14:37:17.690 INFO:tasks.rados.rados.0.smithi151.stdout:3889: done (3 left) 2024-09-13T14:37:17.690 INFO:tasks.rados.rados.0.smithi151.stdout:3890: done (2 left) 2024-09-13T14:37:17.691 INFO:tasks.rados.rados.0.smithi151.stdout:3891: done (1 left) 2024-09-13T14:37:17.691 INFO:tasks.rados.rados.0.smithi151.stdout:3892: done (0 left) 2024-09-13T14:37:17.691 INFO:tasks.rados.rados.0.smithi151.stdout:3893: read oid 50 snap -1 2024-09-13T14:37:17.691 INFO:tasks.rados.rados.0.smithi151.stdout:3893: expect (ObjNum 1464 snap 491 seq_num 1464) 2024-09-13T14:37:17.691 INFO:tasks.rados.rados.0.smithi151.stdout:3894: write oid 13 current snap is 495 2024-09-13T14:37:17.691 INFO:tasks.rados.rados.0.smithi151.stdout:3894: seq_num 1475 ranges {790325=764757,2029999=550178} 2024-09-13T14:37:17.698 INFO:tasks.rados.rados.0.smithi151.stdout:3894: writing smithi15186934-13 from 790325 to 1555082 tid 1 2024-09-13T14:37:17.704 INFO:tasks.rados.rados.0.smithi151.stdout:3894: writing smithi15186934-13 from 2029999 to 2580177 tid 2 2024-09-13T14:37:17.705 INFO:tasks.rados.rados.0.smithi151.stdout:3895: read oid 30 snap 486 2024-09-13T14:37:17.705 INFO:tasks.rados.rados.0.smithi151.stdout:3895: expect (ObjNum 1419 snap 477 seq_num 1419) 2024-09-13T14:37:17.705 INFO:tasks.rados.rados.0.smithi151.stdout:3896: read oid 5 snap -1 2024-09-13T14:37:17.705 INFO:tasks.rados.rados.0.smithi151.stdout:3896: expect deleted 2024-09-13T14:37:17.705 INFO:tasks.rados.rados.0.smithi151.stdout:3897: delete oid 12 current snap is 495 2024-09-13T14:37:17.705 INFO:tasks.rados.rados.0.smithi151.stdout:3893: expect (ObjNum 1464 snap 491 seq_num 1464) 2024-09-13T14:37:17.907 INFO:tasks.rados.rados.0.smithi151.stdout:3894: finishing write tid 1 to smithi15186934-13 2024-09-13T14:37:17.907 INFO:tasks.rados.rados.0.smithi151.stdout:3894: finishing write tid 2 to smithi15186934-13 2024-09-13T14:37:17.907 INFO:tasks.rados.rados.0.smithi151.stdout:3894: finishing write tid 3 to smithi15186934-13 2024-09-13T14:37:17.907 INFO:tasks.rados.rados.0.smithi151.stdout:3894: finishing write tid 4 to smithi15186934-13 2024-09-13T14:37:17.907 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 13 v 1569 (ObjNum 1475 snap 495 seq_num 1475) dirty exists 2024-09-13T14:37:17.907 INFO:tasks.rados.rados.0.smithi151.stdout:3894: left oid 13 (ObjNum 1475 snap 495 seq_num 1475) 2024-09-13T14:37:17.907 INFO:tasks.rados.rados.0.smithi151.stdout:3893: done (4 left) 2024-09-13T14:37:17.907 INFO:tasks.rados.rados.0.smithi151.stdout:3894: done (3 left) 2024-09-13T14:37:17.907 INFO:tasks.rados.rados.0.smithi151.stdout:3896: done (2 left) 2024-09-13T14:37:17.908 INFO:tasks.rados.rados.0.smithi151.stdout:3897: done (1 left) 2024-09-13T14:37:17.908 INFO:tasks.rados.rados.0.smithi151.stdout:3898: snap_create 2024-09-13T14:37:17.908 INFO:tasks.rados.rados.0.smithi151.stdout:3895: expect (ObjNum 1419 snap 477 seq_num 1419) 2024-09-13T14:37:18.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:17 smithi184 ceph-mon[92527]: osdmap e1046: 8 total, 8 up, 8 in 2024-09-13T14:37:18.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:17 smithi151 ceph-mon[104450]: osdmap e1046: 8 total, 8 up, 8 in 2024-09-13T14:37:18.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:17 smithi151 ceph-mon[108966]: osdmap e1046: 8 total, 8 up, 8 in 2024-09-13T14:37:18.693 INFO:tasks.rados.rados.0.smithi151.stdout:3895: done (1 left) 2024-09-13T14:37:18.693 INFO:tasks.rados.rados.0.smithi151.stdout:3898: done (0 left) 2024-09-13T14:37:18.693 INFO:tasks.rados.rados.0.smithi151.stdout:3899: read oid 3 snap -1 2024-09-13T14:37:18.693 INFO:tasks.rados.rados.0.smithi151.stdout:3899: expect (ObjNum 1462 snap 491 seq_num 1462) 2024-09-13T14:37:18.693 INFO:tasks.rados.rados.0.smithi151.stdout:3900: delete oid 34 current snap is 496 2024-09-13T14:37:18.699 INFO:tasks.rados.rados.0.smithi151.stdout:3899: expect (ObjNum 1462 snap 491 seq_num 1462) 2024-09-13T14:37:18.779 INFO:tasks.rados.rados.0.smithi151.stdout:3899: done (1 left) 2024-09-13T14:37:18.779 INFO:tasks.rados.rados.0.smithi151.stdout:3900: done (0 left) 2024-09-13T14:37:18.779 INFO:tasks.rados.rados.0.smithi151.stdout:3901: rollback oid 14 current snap is 496 2024-09-13T14:37:18.779 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 14 to 495 2024-09-13T14:37:18.779 INFO:tasks.rados.rados.0.smithi151.stdout:3902: write oid 26 current snap is 496 2024-09-13T14:37:18.779 INFO:tasks.rados.rados.0.smithi151.stdout:3902: seq_num 1478 ranges {440746=590504,1689092=697505} 2024-09-13T14:37:18.784 INFO:tasks.rados.rados.0.smithi151.stdout:3902: writing smithi15186934-26 from 440746 to 1031250 tid 1 2024-09-13T14:37:18.792 INFO:tasks.rados.rados.0.smithi151.stdout:3902: writing smithi15186934-26 from 1689092 to 2386597 tid 2 2024-09-13T14:37:18.792 INFO:tasks.rados.rados.0.smithi151.stdout:3903: read oid 38 snap -1 2024-09-13T14:37:18.792 INFO:tasks.rados.rados.0.smithi151.stdout:3903: expect (ObjNum 1451 snap 485 seq_num 1451) 2024-09-13T14:37:18.792 INFO:tasks.rados.rados.0.smithi151.stdout:3904: delete oid 21 current snap is 496 2024-09-13T14:37:18.792 INFO:tasks.rados.rados.0.smithi151.stdout:3901: finishing rollback tid 1 to smithi15186934-14 2024-09-13T14:37:18.792 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 14 v 0 (ObjNum 161 snap 0 seq_num 3680956784) dirty dne 2024-09-13T14:37:18.792 INFO:tasks.rados.rados.0.smithi151.stdout:3902: finishing write tid 1 to smithi15186934-26 2024-09-13T14:37:18.795 INFO:tasks.rados.rados.0.smithi151.stdout:3902: finishing write tid 2 to smithi15186934-26 2024-09-13T14:37:18.796 INFO:tasks.rados.rados.0.smithi151.stdout:3902: finishing write tid 3 to smithi15186934-26 2024-09-13T14:37:18.796 INFO:tasks.rados.rados.0.smithi151.stdout:3902: finishing write tid 4 to smithi15186934-26 2024-09-13T14:37:18.796 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 1366 (ObjNum 1478 snap 496 seq_num 1478) dirty exists 2024-09-13T14:37:18.796 INFO:tasks.rados.rados.0.smithi151.stdout:3902: left oid 26 (ObjNum 1478 snap 496 seq_num 1478) 2024-09-13T14:37:18.800 INFO:tasks.rados.rados.0.smithi151.stdout:3903: expect (ObjNum 1451 snap 485 seq_num 1451) 2024-09-13T14:37:18.933 INFO:tasks.rados.rados.0.smithi151.stdout:3901: done (3 left) 2024-09-13T14:37:18.933 INFO:tasks.rados.rados.0.smithi151.stdout:3902: done (2 left) 2024-09-13T14:37:18.933 INFO:tasks.rados.rados.0.smithi151.stdout:3903: done (1 left) 2024-09-13T14:37:18.933 INFO:tasks.rados.rados.0.smithi151.stdout:3904: done (0 left) 2024-09-13T14:37:18.934 INFO:tasks.rados.rados.0.smithi151.stdout:3905: write oid 42 current snap is 496 2024-09-13T14:37:18.934 INFO:tasks.rados.rados.0.smithi151.stdout:3905: seq_num 1480 ranges {474360=741002,1806056=701174} 2024-09-13T14:37:18.939 INFO:tasks.rados.rados.0.smithi151.stdout:3905: writing smithi15186934-42 from 474360 to 1215362 tid 1 2024-09-13T14:37:18.947 INFO:tasks.rados.rados.0.smithi151.stdout:3905: writing smithi15186934-42 from 1806056 to 2507230 tid 2 2024-09-13T14:37:18.947 INFO:tasks.rados.rados.0.smithi151.stdout:3906: delete oid 29 current snap is 496 2024-09-13T14:37:18.947 INFO:tasks.rados.rados.0.smithi151.stdout:3905: finishing write tid 1 to smithi15186934-42 2024-09-13T14:37:18.949 INFO:tasks.rados.rados.0.smithi151.stdout:3906: done (1 left) 2024-09-13T14:37:18.949 INFO:tasks.rados.rados.0.smithi151.stdout:3907: write oid 27 current snap is 496 2024-09-13T14:37:18.949 INFO:tasks.rados.rados.0.smithi151.stdout:3907: seq_num 1482 ranges {507974=491500,1406667=704843,2548287=359687} 2024-09-13T14:37:18.954 INFO:tasks.rados.rados.0.smithi151.stdout:3907: writing smithi15186934-27 from 507974 to 999474 tid 1 2024-09-13T14:37:18.960 INFO:tasks.rados.rados.0.smithi151.stdout:3907: writing smithi15186934-27 from 1406667 to 2111510 tid 2 2024-09-13T14:37:18.965 INFO:tasks.rados.rados.0.smithi151.stdout:3907: writing smithi15186934-27 from 2548287 to 2907974 tid 3 2024-09-13T14:37:18.966 INFO:tasks.rados.rados.0.smithi151.stdout:3908: read oid 38 snap -1 2024-09-13T14:37:18.966 INFO:tasks.rados.rados.0.smithi151.stdout:3908: expect (ObjNum 1451 snap 485 seq_num 1451) 2024-09-13T14:37:18.966 INFO:tasks.rados.rados.0.smithi151.stdout:3909: write oid 24 current snap is 496 2024-09-13T14:37:18.966 INFO:tasks.rados.rados.0.smithi151.stdout:3905: finishing write tid 2 to smithi15186934-42 2024-09-13T14:37:18.966 INFO:tasks.rados.rados.0.smithi151.stdout:3905: finishing write tid 3 to smithi15186934-42 2024-09-13T14:37:18.966 INFO:tasks.rados.rados.0.smithi151.stdout:3905: finishing write tid 4 to smithi15186934-42 2024-09-13T14:37:18.966 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 42 v 1450 (ObjNum 1480 snap 496 seq_num 1480) dirty exists 2024-09-13T14:37:18.966 INFO:tasks.rados.rados.0.smithi151.stdout:3905: left oid 42 (ObjNum 1480 snap 496 seq_num 1480) 2024-09-13T14:37:18.966 INFO:tasks.rados.rados.0.smithi151.stdout:3907: finishing write tid 1 to smithi15186934-27 2024-09-13T14:37:18.966 INFO:tasks.rados.rados.0.smithi151.stdout:3909: seq_num 1483 ranges {524781=683102,1981502=564854} 2024-09-13T14:37:18.971 INFO:tasks.rados.rados.0.smithi151.stdout:3909: writing smithi15186934-24 from 524781 to 1207883 tid 1 2024-09-13T14:37:18.980 INFO:tasks.rados.rados.0.smithi151.stdout:3909: writing smithi15186934-24 from 1981502 to 2546356 tid 2 2024-09-13T14:37:18.980 INFO:tasks.rados.rados.0.smithi151.stdout:3905: done (3 left) 2024-09-13T14:37:18.980 INFO:tasks.rados.rados.0.smithi151.stdout:3910: delete oid 31 current snap is 496 2024-09-13T14:37:18.980 INFO:tasks.rados.rados.0.smithi151.stdout:3907: finishing write tid 2 to smithi15186934-27 2024-09-13T14:37:18.980 INFO:tasks.rados.rados.0.smithi151.stdout:3907: finishing write tid 3 to smithi15186934-27 2024-09-13T14:37:18.980 INFO:tasks.rados.rados.0.smithi151.stdout:3907: finishing write tid 4 to smithi15186934-27 2024-09-13T14:37:18.980 INFO:tasks.rados.rados.0.smithi151.stdout:3907: finishing write tid 5 to smithi15186934-27 2024-09-13T14:37:18.980 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 27 v 1193 (ObjNum 1482 snap 496 seq_num 1482) dirty exists 2024-09-13T14:37:18.980 INFO:tasks.rados.rados.0.smithi151.stdout:3907: left oid 27 (ObjNum 1482 snap 496 seq_num 1482) 2024-09-13T14:37:18.981 INFO:tasks.rados.rados.0.smithi151.stdout:3908: expect (ObjNum 1451 snap 485 seq_num 1451) 2024-09-13T14:37:19.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:18 smithi184 ceph-mon[92527]: pgmap v1188: 89 pgs: 89 active+clean; 191 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 10 op/s 2024-09-13T14:37:19.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:18 smithi184 ceph-mon[92527]: osdmap e1047: 8 total, 8 up, 8 in 2024-09-13T14:37:19.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:18 smithi151 ceph-mon[104450]: pgmap v1188: 89 pgs: 89 active+clean; 191 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 10 op/s 2024-09-13T14:37:19.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:18 smithi151 ceph-mon[104450]: osdmap e1047: 8 total, 8 up, 8 in 2024-09-13T14:37:19.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:18 smithi151 ceph-mon[108966]: pgmap v1188: 89 pgs: 89 active+clean; 191 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 10 op/s 2024-09-13T14:37:19.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:18 smithi151 ceph-mon[108966]: osdmap e1047: 8 total, 8 up, 8 in 2024-09-13T14:37:19.113 INFO:tasks.rados.rados.0.smithi151.stdout:3909: finishing write tid 1 to smithi15186934-24 2024-09-13T14:37:19.113 INFO:tasks.rados.rados.0.smithi151.stdout:3909: finishing write tid 2 to smithi15186934-24 2024-09-13T14:37:19.113 INFO:tasks.rados.rados.0.smithi151.stdout:3909: finishing write tid 3 to smithi15186934-24 2024-09-13T14:37:19.113 INFO:tasks.rados.rados.0.smithi151.stdout:3909: finishing write tid 4 to smithi15186934-24 2024-09-13T14:37:19.114 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 24 v 2189 (ObjNum 1483 snap 496 seq_num 1483) dirty exists 2024-09-13T14:37:19.114 INFO:tasks.rados.rados.0.smithi151.stdout:3909: left oid 24 (ObjNum 1483 snap 496 seq_num 1483) 2024-09-13T14:37:19.114 INFO:tasks.rados.rados.0.smithi151.stdout:3907: done (3 left) 2024-09-13T14:37:19.114 INFO:tasks.rados.rados.0.smithi151.stdout:3908: done (2 left) 2024-09-13T14:37:19.114 INFO:tasks.rados.rados.0.smithi151.stdout:3909: done (1 left) 2024-09-13T14:37:19.114 INFO:tasks.rados.rados.0.smithi151.stdout:3910: done (0 left) 2024-09-13T14:37:19.114 INFO:tasks.rados.rados.0.smithi151.stdout:3911: snap_create 2024-09-13T14:37:19.672 INFO:tasks.rados.rados.0.smithi151.stdout:3911: done (0 left) 2024-09-13T14:37:19.672 INFO:tasks.rados.rados.0.smithi151.stdout:3912: read oid 41 snap 493 2024-09-13T14:37:19.672 INFO:tasks.rados.rados.0.smithi151.stdout:3912: expect (ObjNum 1457 snap 489 seq_num 1457) 2024-09-13T14:37:19.672 INFO:tasks.rados.rados.0.smithi151.stdout:3913: write oid 25 current snap is 497 2024-09-13T14:37:19.672 INFO:tasks.rados.rados.0.smithi151.stdout:3913: seq_num 1485 ranges {558395=433600,1698466=568523,2863262=95133} 2024-09-13T14:37:19.677 INFO:tasks.rados.rados.0.smithi151.stdout:3913: writing smithi15186934-25 from 558395 to 991995 tid 1 2024-09-13T14:37:19.684 INFO:tasks.rados.rados.0.smithi151.stdout:3913: writing smithi15186934-25 from 1698466 to 2266989 tid 2 2024-09-13T14:37:19.688 INFO:tasks.rados.rados.0.smithi151.stdout:3913: writing smithi15186934-25 from 2863262 to 2958395 tid 3 2024-09-13T14:37:19.688 INFO:tasks.rados.rados.0.smithi151.stdout:3914: snap_remove snap 494 2024-09-13T14:37:20.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:19 smithi184 ceph-mon[92527]: osdmap e1048: 8 total, 8 up, 8 in 2024-09-13T14:37:20.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:19 smithi184 ceph-mon[92527]: osdmap e1049: 8 total, 8 up, 8 in 2024-09-13T14:37:20.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:19 smithi151 ceph-mon[104450]: osdmap e1048: 8 total, 8 up, 8 in 2024-09-13T14:37:20.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:19 smithi151 ceph-mon[104450]: osdmap e1049: 8 total, 8 up, 8 in 2024-09-13T14:37:20.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:19 smithi151 ceph-mon[108966]: osdmap e1048: 8 total, 8 up, 8 in 2024-09-13T14:37:20.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:19 smithi151 ceph-mon[108966]: osdmap e1049: 8 total, 8 up, 8 in 2024-09-13T14:37:20.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:37:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:37:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:37:20.673 INFO:tasks.rados.rados.0.smithi151.stdout:3914: done (2 left) 2024-09-13T14:37:20.673 INFO:tasks.rados.rados.0.smithi151.stdout:3915: write oid 2 current snap is 497 2024-09-13T14:37:20.673 INFO:tasks.rados.rados.0.smithi151.stdout:3915: seq_num 1486 ranges {575202=508849,1640595=712181} 2024-09-13T14:37:20.679 INFO:tasks.rados.rados.0.smithi151.stdout:3915: writing smithi15186934-2 from 575202 to 1084051 tid 1 2024-09-13T14:37:20.687 INFO:tasks.rados.rados.0.smithi151.stdout:3915: writing smithi15186934-2 from 1640595 to 2352776 tid 2 2024-09-13T14:37:20.687 INFO:tasks.rados.rados.0.smithi151.stdout:3916: read oid 39 snap -1 2024-09-13T14:37:20.687 INFO:tasks.rados.rados.0.smithi151.stdout:3916: expect (ObjNum 1324 snap 450 seq_num 1324) 2024-09-13T14:37:20.687 INFO:tasks.rados.rados.0.smithi151.stdout:3917: snap_remove snap 475 2024-09-13T14:37:21.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:20 smithi184 ceph-mon[92527]: pgmap v1191: 89 pgs: 89 active+clean; 197 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.9 MiB/s wr, 15 op/s 2024-09-13T14:37:21.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:20 smithi184 ceph-mon[92527]: osdmap e1050: 8 total, 8 up, 8 in 2024-09-13T14:37:21.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:20 smithi151 ceph-mon[104450]: pgmap v1191: 89 pgs: 89 active+clean; 197 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.9 MiB/s wr, 15 op/s 2024-09-13T14:37:21.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:20 smithi151 ceph-mon[104450]: osdmap e1050: 8 total, 8 up, 8 in 2024-09-13T14:37:21.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:20 smithi151 ceph-mon[108966]: pgmap v1191: 89 pgs: 89 active+clean; 197 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.9 MiB/s wr, 15 op/s 2024-09-13T14:37:21.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:20 smithi151 ceph-mon[108966]: osdmap e1050: 8 total, 8 up, 8 in 2024-09-13T14:37:21.675 INFO:tasks.rados.rados.0.smithi151.stdout:3917: done (4 left) 2024-09-13T14:37:21.676 INFO:tasks.rados.rados.0.smithi151.stdout:3918: read oid 3 snap 482 2024-09-13T14:37:21.676 INFO:tasks.rados.rados.0.smithi151.stdout:3918: expect (ObjNum 1398 snap 469 seq_num 1398) 2024-09-13T14:37:21.676 INFO:tasks.rados.rados.0.smithi151.stdout:3919: read oid 46 snap -1 2024-09-13T14:37:21.676 INFO:tasks.rados.rados.0.smithi151.stdout:3919: expect (ObjNum 1468 snap 494 seq_num 1468) 2024-09-13T14:37:21.676 INFO:tasks.rados.rados.0.smithi151.stdout:3920: write oid 18 current snap is 497 2024-09-13T14:37:21.676 INFO:tasks.rados.rados.0.smithi151.stdout:3920: seq_num 1487 ranges {592009=584098,1699077=572192,2801755=190254} 2024-09-13T14:37:21.682 INFO:tasks.rados.rados.0.smithi151.stdout:3920: writing smithi15186934-18 from 592009 to 1176107 tid 1 2024-09-13T14:37:21.690 INFO:tasks.rados.rados.0.smithi151.stdout:3920: writing smithi15186934-18 from 1699077 to 2271269 tid 2 2024-09-13T14:37:21.694 INFO:tasks.rados.rados.0.smithi151.stdout:3920: writing smithi15186934-18 from 2801755 to 2992009 tid 3 2024-09-13T14:37:21.695 INFO:tasks.rados.rados.0.smithi151.stdout:3921: delete oid 38 current snap is 497 2024-09-13T14:37:21.695 INFO:tasks.rados.rados.0.smithi151.stdout:3912: expect (ObjNum 1457 snap 489 seq_num 1457) 2024-09-13T14:37:21.804 INFO:tasks.rados.rados.0.smithi151.stdout:3913: finishing write tid 1 to smithi15186934-25 2024-09-13T14:37:21.804 INFO:tasks.rados.rados.0.smithi151.stdout:3913: finishing write tid 2 to smithi15186934-25 2024-09-13T14:37:21.804 INFO:tasks.rados.rados.0.smithi151.stdout:3913: finishing write tid 3 to smithi15186934-25 2024-09-13T14:37:21.804 INFO:tasks.rados.rados.0.smithi151.stdout:3913: finishing write tid 4 to smithi15186934-25 2024-09-13T14:37:21.804 INFO:tasks.rados.rados.0.smithi151.stdout:3913: finishing write tid 5 to smithi15186934-25 2024-09-13T14:37:21.804 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 25 v 1894 (ObjNum 1485 snap 497 seq_num 1485) dirty exists 2024-09-13T14:37:21.804 INFO:tasks.rados.rados.0.smithi151.stdout:3913: left oid 25 (ObjNum 1485 snap 497 seq_num 1485) 2024-09-13T14:37:21.804 INFO:tasks.rados.rados.0.smithi151.stdout:3915: finishing write tid 1 to smithi15186934-2 2024-09-13T14:37:21.804 INFO:tasks.rados.rados.0.smithi151.stdout:3912: done (7 left) 2024-09-13T14:37:21.804 INFO:tasks.rados.rados.0.smithi151.stdout:3913: done (6 left) 2024-09-13T14:37:21.804 INFO:tasks.rados.rados.0.smithi151.stdout:3921: done (5 left) 2024-09-13T14:37:21.805 INFO:tasks.rados.rados.0.smithi151.stdout:3922: rollback oid 8 current snap is 497 2024-09-13T14:37:21.805 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 8 to 491 2024-09-13T14:37:21.805 INFO:tasks.rados.rados.0.smithi151.stdout:3915: finishing write tid 2 to smithi15186934-2 2024-09-13T14:37:21.805 INFO:tasks.rados.rados.0.smithi151.stdout:3915: finishing write tid 3 to smithi15186934-2 2024-09-13T14:37:21.805 INFO:tasks.rados.rados.0.smithi151.stdout:3915: finishing write tid 4 to smithi15186934-2 2024-09-13T14:37:21.805 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 2 v 1573 (ObjNum 1486 snap 497 seq_num 1486) dirty exists 2024-09-13T14:37:21.805 INFO:tasks.rados.rados.0.smithi151.stdout:3915: left oid 2 (ObjNum 1486 snap 497 seq_num 1486) 2024-09-13T14:37:21.805 INFO:tasks.rados.rados.0.smithi151.stdout:3916: expect (ObjNum 1324 snap 450 seq_num 1324) 2024-09-13T14:37:21.952 INFO:tasks.rados.rados.0.smithi151.stdout:3918: expect (ObjNum 1398 snap 469 seq_num 1398) 2024-09-13T14:37:22.110 INFO:tasks.rados.rados.0.smithi151.stdout:3919: expect (ObjNum 1468 snap 494 seq_num 1468) 2024-09-13T14:37:22.197 INFO:tasks.rados.rados.0.smithi151.stdout:3920: finishing write tid 1 to smithi15186934-18 2024-09-13T14:37:22.197 INFO:tasks.rados.rados.0.smithi151.stdout:3920: finishing write tid 2 to smithi15186934-18 2024-09-13T14:37:22.197 INFO:tasks.rados.rados.0.smithi151.stdout:3920: finishing write tid 3 to smithi15186934-18 2024-09-13T14:37:22.197 INFO:tasks.rados.rados.0.smithi151.stdout:3920: finishing write tid 4 to smithi15186934-18 2024-09-13T14:37:22.197 INFO:tasks.rados.rados.0.smithi151.stdout:3920: finishing write tid 5 to smithi15186934-18 2024-09-13T14:37:22.197 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 18 v 1075 (ObjNum 1487 snap 497 seq_num 1487) dirty exists 2024-09-13T14:37:22.197 INFO:tasks.rados.rados.0.smithi151.stdout:3920: left oid 18 (ObjNum 1487 snap 497 seq_num 1487) 2024-09-13T14:37:22.197 INFO:tasks.rados.rados.0.smithi151.stdout:3922: finishing rollback tid 1 to smithi15186934-8 2024-09-13T14:37:22.197 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 8 v 1275 (ObjNum 129 snap 0 seq_num 3093088760) dirty dne 2024-09-13T14:37:22.197 INFO:tasks.rados.rados.0.smithi151.stdout:3915: done (5 left) 2024-09-13T14:37:22.197 INFO:tasks.rados.rados.0.smithi151.stdout:3916: done (4 left) 2024-09-13T14:37:22.198 INFO:tasks.rados.rados.0.smithi151.stdout:3918: done (3 left) 2024-09-13T14:37:22.198 INFO:tasks.rados.rados.0.smithi151.stdout:3919: done (2 left) 2024-09-13T14:37:22.198 INFO:tasks.rados.rados.0.smithi151.stdout:3920: done (1 left) 2024-09-13T14:37:22.198 INFO:tasks.rados.rados.0.smithi151.stdout:3922: done (0 left) 2024-09-13T14:37:22.198 INFO:tasks.rados.rados.0.smithi151.stdout:3923: read oid 13 snap -1 2024-09-13T14:37:22.198 INFO:tasks.rados.rados.0.smithi151.stdout:3923: expect (ObjNum 1475 snap 495 seq_num 1475) 2024-09-13T14:37:22.198 INFO:tasks.rados.rados.0.smithi151.stdout:3924: snap_remove snap 490 2024-09-13T14:37:22.708 INFO:tasks.rados.rados.0.smithi151.stdout:3923: expect (ObjNum 1475 snap 495 seq_num 1475) 2024-09-13T14:37:22.801 INFO:tasks.rados.rados.0.smithi151.stdout:3923: done (1 left) 2024-09-13T14:37:22.801 INFO:tasks.rados.rados.0.smithi151.stdout:3924: done (0 left) 2024-09-13T14:37:22.801 INFO:tasks.rados.rados.0.smithi151.stdout:3925: snap_remove snap 489 2024-09-13T14:37:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:22 smithi184 ceph-mon[92527]: pgmap v1194: 89 pgs: 89 active+clean; 198 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 1.5 MiB/s wr, 13 op/s 2024-09-13T14:37:23.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:22 smithi184 ceph-mon[92527]: osdmap e1051: 8 total, 8 up, 8 in 2024-09-13T14:37:23.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:22 smithi151 ceph-mon[104450]: pgmap v1194: 89 pgs: 89 active+clean; 198 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 1.5 MiB/s wr, 13 op/s 2024-09-13T14:37:23.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:22 smithi151 ceph-mon[104450]: osdmap e1051: 8 total, 8 up, 8 in 2024-09-13T14:37:23.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:22 smithi151 ceph-mon[108966]: pgmap v1194: 89 pgs: 89 active+clean; 198 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 1.5 MiB/s wr, 13 op/s 2024-09-13T14:37:23.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:22 smithi151 ceph-mon[108966]: osdmap e1051: 8 total, 8 up, 8 in 2024-09-13T14:37:23.719 INFO:tasks.rados.rados.0.smithi151.stdout:3925: done (0 left) 2024-09-13T14:37:23.719 INFO:tasks.rados.rados.0.smithi151.stdout:3926: snap_remove snap 486 2024-09-13T14:37:24.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:23 smithi184 ceph-mon[92527]: osdmap e1052: 8 total, 8 up, 8 in 2024-09-13T14:37:24.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:23 smithi151 ceph-mon[104450]: osdmap e1052: 8 total, 8 up, 8 in 2024-09-13T14:37:24.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:23 smithi151 ceph-mon[108966]: osdmap e1052: 8 total, 8 up, 8 in 2024-09-13T14:37:24.679 INFO:tasks.rados.rados.0.smithi151.stdout:3926: done (0 left) 2024-09-13T14:37:24.679 INFO:tasks.rados.rados.0.smithi151.stdout:3927: rollback oid 21 current snap is 497 2024-09-13T14:37:24.679 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 21 to 493 2024-09-13T14:37:24.679 INFO:tasks.rados.rados.0.smithi151.stdout:3928: snap_remove snap 473 2024-09-13T14:37:24.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:24 smithi151 ceph-mon[108966]: pgmap v1197: 89 pgs: 89 active+clean; 203 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 1.2 MiB/s wr, 22 op/s 2024-09-13T14:37:24.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:24 smithi151 ceph-mon[108966]: osdmap e1053: 8 total, 8 up, 8 in 2024-09-13T14:37:24.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:24 smithi151 ceph-mon[108966]: osdmap e1054: 8 total, 8 up, 8 in 2024-09-13T14:37:24.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:24 smithi151 ceph-mon[104450]: pgmap v1197: 89 pgs: 89 active+clean; 203 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 1.2 MiB/s wr, 22 op/s 2024-09-13T14:37:24.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:24 smithi151 ceph-mon[104450]: osdmap e1053: 8 total, 8 up, 8 in 2024-09-13T14:37:24.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:24 smithi151 ceph-mon[104450]: osdmap e1054: 8 total, 8 up, 8 in 2024-09-13T14:37:24.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:37:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:37:24] "GET /metrics HTTP/1.1" 200 36508 "" "Prometheus/2.51.0" 2024-09-13T14:37:25.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:24 smithi184 ceph-mon[92527]: pgmap v1197: 89 pgs: 89 active+clean; 203 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 1.2 MiB/s wr, 22 op/s 2024-09-13T14:37:25.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:24 smithi184 ceph-mon[92527]: osdmap e1053: 8 total, 8 up, 8 in 2024-09-13T14:37:25.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:24 smithi184 ceph-mon[92527]: osdmap e1054: 8 total, 8 up, 8 in 2024-09-13T14:37:25.675 INFO:tasks.rados.rados.0.smithi151.stdout:3928: done (1 left) 2024-09-13T14:37:25.675 INFO:tasks.rados.rados.0.smithi151.stdout:3927: finishing rollback tid 1 to smithi15186934-21 2024-09-13T14:37:25.675 INFO:tasks.rados.rados.0.smithi151.stdout:3927: finishing rollback tid 2 to smithi15186934-21 2024-09-13T14:37:25.675 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 21 v 795 (ObjNum 1318 snap 449 seq_num 1318) dirty exists 2024-09-13T14:37:25.676 INFO:tasks.rados.rados.0.smithi151.stdout:3929: read oid 50 snap 493 2024-09-13T14:37:25.676 INFO:tasks.rados.rados.0.smithi151.stdout:3929: expect (ObjNum 1464 snap 491 seq_num 1464) 2024-09-13T14:37:25.676 INFO:tasks.rados.rados.0.smithi151.stdout:3927: done (1 left) 2024-09-13T14:37:25.676 INFO:tasks.rados.rados.0.smithi151.stdout:3930: snap_create 2024-09-13T14:37:25.690 INFO:tasks.rados.rados.0.smithi151.stdout:3929: expect (ObjNum 1464 snap 491 seq_num 1464) 2024-09-13T14:37:26.677 INFO:tasks.rados.rados.0.smithi151.stdout:3929: done (1 left) 2024-09-13T14:37:26.677 INFO:tasks.rados.rados.0.smithi151.stdout:3930: done (0 left) 2024-09-13T14:37:26.678 INFO:tasks.rados.rados.0.smithi151.stdout:3931: snap_remove snap 497 2024-09-13T14:37:27.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:26 smithi184 ceph-mon[92527]: pgmap v1200: 89 pgs: 89 active+clean; 206 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 1.9 MiB/s wr, 22 op/s 2024-09-13T14:37:27.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:26 smithi184 ceph-mon[92527]: osdmap e1055: 8 total, 8 up, 8 in 2024-09-13T14:37:27.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:26 smithi151 ceph-mon[104450]: pgmap v1200: 89 pgs: 89 active+clean; 206 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 1.9 MiB/s wr, 22 op/s 2024-09-13T14:37:27.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:26 smithi151 ceph-mon[104450]: osdmap e1055: 8 total, 8 up, 8 in 2024-09-13T14:37:27.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:26 smithi151 ceph-mon[108966]: pgmap v1200: 89 pgs: 89 active+clean; 206 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 1.9 MiB/s wr, 22 op/s 2024-09-13T14:37:27.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:26 smithi151 ceph-mon[108966]: osdmap e1055: 8 total, 8 up, 8 in 2024-09-13T14:37:27.686 INFO:tasks.rados.rados.0.smithi151.stdout:3931: done (0 left) 2024-09-13T14:37:27.686 INFO:tasks.rados.rados.0.smithi151.stdout:3932: write oid 9 current snap is 498 2024-09-13T14:37:27.687 INFO:tasks.rados.rados.0.smithi151.stdout:3932: seq_num 1489 ranges {625623=734596,1816041=459508,2740248=285375} 2024-09-13T14:37:27.694 INFO:tasks.rados.rados.0.smithi151.stdout:3932: writing smithi15186934-9 from 625623 to 1360219 tid 1 2024-09-13T14:37:27.699 INFO:tasks.rados.rados.0.smithi151.stdout:3932: writing smithi15186934-9 from 1816041 to 2275549 tid 2 2024-09-13T14:37:27.704 INFO:tasks.rados.rados.0.smithi151.stdout:3932: writing smithi15186934-9 from 2740248 to 3025623 tid 3 2024-09-13T14:37:27.704 INFO:tasks.rados.rados.0.smithi151.stdout:3933: rollback oid 5 current snap is 498 2024-09-13T14:37:27.704 INFO:tasks.rados.rados.0.smithi151.stdout:3932: finishing write tid 1 to smithi15186934-9 2024-09-13T14:37:27.704 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 5 to 492 2024-09-13T14:37:27.704 INFO:tasks.rados.rados.0.smithi151.stdout:3932: finishing write tid 2 to smithi15186934-9 2024-09-13T14:37:27.704 INFO:tasks.rados.rados.0.smithi151.stdout:3934: delete oid 39 current snap is 498 2024-09-13T14:37:27.708 INFO:tasks.rados.rados.0.smithi151.stdout:3934: done (2 left) 2024-09-13T14:37:27.708 INFO:tasks.rados.rados.0.smithi151.stdout:3935: snap_remove snap 496 2024-09-13T14:37:28.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:27 smithi184 ceph-mon[92527]: osdmap e1056: 8 total, 8 up, 8 in 2024-09-13T14:37:28.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:27 smithi151 ceph-mon[104450]: osdmap e1056: 8 total, 8 up, 8 in 2024-09-13T14:37:28.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:27 smithi151 ceph-mon[108966]: osdmap e1056: 8 total, 8 up, 8 in 2024-09-13T14:37:28.704 INFO:tasks.rados.rados.0.smithi151.stdout:3935: done (2 left) 2024-09-13T14:37:28.704 INFO:tasks.rados.rados.0.smithi151.stdout:3936: write oid 50 current snap is 498 2024-09-13T14:37:28.704 INFO:tasks.rados.rados.0.smithi151.stdout:3932: finishing write tid 3 to smithi15186934-9 2024-09-13T14:37:28.704 INFO:tasks.rados.rados.0.smithi151.stdout:3932: finishing write tid 4 to smithi15186934-9 2024-09-13T14:37:28.705 INFO:tasks.rados.rados.0.smithi151.stdout:3932: finishing write tid 5 to smithi15186934-9 2024-09-13T14:37:28.705 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 9 v 1615 (ObjNum 1489 snap 498 seq_num 1489) dirty exists 2024-09-13T14:37:28.705 INFO:tasks.rados.rados.0.smithi151.stdout:3932: left oid 9 (ObjNum 1489 snap 498 seq_num 1489) 2024-09-13T14:37:28.705 INFO:tasks.rados.rados.0.smithi151.stdout:3933: finishing rollback tid 1 to smithi15186934-5 2024-09-13T14:37:28.705 INFO:tasks.rados.rados.0.smithi151.stdout:3933: finishing rollback tid 2 to smithi15186934-5 2024-09-13T14:37:28.705 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 5 v 1595 (ObjNum 1370 snap 463 seq_num 1370) dirty exists 2024-09-13T14:37:28.705 INFO:tasks.rados.rados.0.smithi151.stdout:3936: seq_num 1491 ranges {659237=601447,1933005=463177} 2024-09-13T14:37:28.711 INFO:tasks.rados.rados.0.smithi151.stdout:3936: writing smithi15186934-50 from 659237 to 1260684 tid 1 2024-09-13T14:37:28.718 INFO:tasks.rados.rados.0.smithi151.stdout:3936: writing smithi15186934-50 from 1933005 to 2396182 tid 2 2024-09-13T14:37:28.718 INFO:tasks.rados.rados.0.smithi151.stdout:3932: done (2 left) 2024-09-13T14:37:28.718 INFO:tasks.rados.rados.0.smithi151.stdout:3933: done (1 left) 2024-09-13T14:37:28.718 INFO:tasks.rados.rados.0.smithi151.stdout:3937: snap_create 2024-09-13T14:37:28.718 INFO:tasks.rados.rados.0.smithi151.stdout:3936: finishing write tid 1 to smithi15186934-50 2024-09-13T14:37:28.721 INFO:tasks.rados.rados.0.smithi151.stdout:3936: finishing write tid 2 to smithi15186934-50 2024-09-13T14:37:28.722 INFO:tasks.rados.rados.0.smithi151.stdout:3936: finishing write tid 3 to smithi15186934-50 2024-09-13T14:37:28.722 INFO:tasks.rados.rados.0.smithi151.stdout:3936: finishing write tid 4 to smithi15186934-50 2024-09-13T14:37:28.722 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 50 v 1652 (ObjNum 1491 snap 498 seq_num 1491) dirty exists 2024-09-13T14:37:28.722 INFO:tasks.rados.rados.0.smithi151.stdout:3936: left oid 50 (ObjNum 1491 snap 498 seq_num 1491) 2024-09-13T14:37:29.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:28 smithi184 ceph-mon[92527]: pgmap v1203: 89 pgs: 89 active+clean; 208 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.3 MiB/s wr, 14 op/s 2024-09-13T14:37:29.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:28 smithi184 ceph-mon[92527]: osdmap e1057: 8 total, 8 up, 8 in 2024-09-13T14:37:29.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:28 smithi151 ceph-mon[104450]: pgmap v1203: 89 pgs: 89 active+clean; 208 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.3 MiB/s wr, 14 op/s 2024-09-13T14:37:29.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:28 smithi151 ceph-mon[104450]: osdmap e1057: 8 total, 8 up, 8 in 2024-09-13T14:37:29.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:28 smithi151 ceph-mon[108966]: pgmap v1203: 89 pgs: 89 active+clean; 208 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.3 MiB/s wr, 14 op/s 2024-09-13T14:37:29.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:28 smithi151 ceph-mon[108966]: osdmap e1057: 8 total, 8 up, 8 in 2024-09-13T14:37:29.677 INFO:tasks.rados.rados.0.smithi151.stdout:3936: done (1 left) 2024-09-13T14:37:29.677 INFO:tasks.rados.rados.0.smithi151.stdout:3937: done (0 left) 2024-09-13T14:37:29.678 INFO:tasks.rados.rados.0.smithi151.stdout:3938: read oid 13 snap -1 2024-09-13T14:37:29.678 INFO:tasks.rados.rados.0.smithi151.stdout:3938: expect (ObjNum 1475 snap 495 seq_num 1475) 2024-09-13T14:37:29.678 INFO:tasks.rados.rados.0.smithi151.stdout:3939: write oid 45 current snap is 499 2024-09-13T14:37:29.678 INFO:tasks.rados.rados.0.smithi151.stdout:3939: seq_num 1492 ranges {676044=676696,1991487=723188} 2024-09-13T14:37:29.685 INFO:tasks.rados.rados.0.smithi151.stdout:3939: writing smithi15186934-45 from 676044 to 1352740 tid 1 2024-09-13T14:37:29.692 INFO:tasks.rados.rados.0.smithi151.stdout:3939: writing smithi15186934-45 from 1991487 to 2714675 tid 2 2024-09-13T14:37:29.693 INFO:tasks.rados.rados.0.smithi151.stdout:3940: snap_remove snap 487 2024-09-13T14:37:30.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:29 smithi184 ceph-mon[92527]: osdmap e1058: 8 total, 8 up, 8 in 2024-09-13T14:37:30.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:29 smithi184 ceph-mon[92527]: osdmap e1059: 8 total, 8 up, 8 in 2024-09-13T14:37:30.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:29 smithi151 ceph-mon[104450]: osdmap e1058: 8 total, 8 up, 8 in 2024-09-13T14:37:30.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:29 smithi151 ceph-mon[104450]: osdmap e1059: 8 total, 8 up, 8 in 2024-09-13T14:37:30.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:29 smithi151 ceph-mon[108966]: osdmap e1058: 8 total, 8 up, 8 in 2024-09-13T14:37:30.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:29 smithi151 ceph-mon[108966]: osdmap e1059: 8 total, 8 up, 8 in 2024-09-13T14:37:30.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:37:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:37:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:37:30.676 INFO:tasks.rados.rados.0.smithi151.stdout:3940: done (2 left) 2024-09-13T14:37:30.676 INFO:tasks.rados.rados.0.smithi151.stdout:3941: write oid 22 current snap is 499 2024-09-13T14:37:30.677 INFO:tasks.rados.rados.0.smithi151.stdout:3941: seq_num 1493 ranges {692851=751945,2049969=466846} 2024-09-13T14:37:30.685 INFO:tasks.rados.rados.0.smithi151.stdout:3941: writing smithi15186934-22 from 692851 to 1444796 tid 1 2024-09-13T14:37:30.693 INFO:tasks.rados.rados.0.smithi151.stdout:3941: writing smithi15186934-22 from 2049969 to 2516815 tid 2 2024-09-13T14:37:30.693 INFO:tasks.rados.rados.0.smithi151.stdout:3942: rollback oid 30 current snap is 499 2024-09-13T14:37:30.693 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 30 to 479 2024-09-13T14:37:30.693 INFO:tasks.rados.rados.0.smithi151.stdout:3938: expect (ObjNum 1475 snap 495 seq_num 1475) 2024-09-13T14:37:30.786 INFO:tasks.rados.rados.0.smithi151.stdout:3939: finishing write tid 1 to smithi15186934-45 2024-09-13T14:37:30.786 INFO:tasks.rados.rados.0.smithi151.stdout:3939: finishing write tid 2 to smithi15186934-45 2024-09-13T14:37:30.786 INFO:tasks.rados.rados.0.smithi151.stdout:3939: finishing write tid 3 to smithi15186934-45 2024-09-13T14:37:30.786 INFO:tasks.rados.rados.0.smithi151.stdout:3939: finishing write tid 4 to smithi15186934-45 2024-09-13T14:37:30.786 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 45 v 1921 (ObjNum 1492 snap 499 seq_num 1492) dirty exists 2024-09-13T14:37:30.786 INFO:tasks.rados.rados.0.smithi151.stdout:3939: left oid 45 (ObjNum 1492 snap 499 seq_num 1492) 2024-09-13T14:37:30.786 INFO:tasks.rados.rados.0.smithi151.stdout:3941: finishing write tid 1 to smithi15186934-22 2024-09-13T14:37:30.786 INFO:tasks.rados.rados.0.smithi151.stdout:3941: finishing write tid 2 to smithi15186934-22 2024-09-13T14:37:30.787 INFO:tasks.rados.rados.0.smithi151.stdout:3941: finishing write tid 3 to smithi15186934-22 2024-09-13T14:37:30.787 INFO:tasks.rados.rados.0.smithi151.stdout:3938: done (3 left) 2024-09-13T14:37:30.787 INFO:tasks.rados.rados.0.smithi151.stdout:3939: done (2 left) 2024-09-13T14:37:30.787 INFO:tasks.rados.rados.0.smithi151.stdout:3943: snap_create 2024-09-13T14:37:30.787 INFO:tasks.rados.rados.0.smithi151.stdout:3941: finishing write tid 4 to smithi15186934-22 2024-09-13T14:37:30.787 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 22 v 807 (ObjNum 1493 snap 499 seq_num 1493) dirty exists 2024-09-13T14:37:30.787 INFO:tasks.rados.rados.0.smithi151.stdout:3941: left oid 22 (ObjNum 1493 snap 499 seq_num 1493) 2024-09-13T14:37:30.787 INFO:tasks.rados.rados.0.smithi151.stdout:3942: finishing rollback tid 0 to smithi15186934-30 2024-09-13T14:37:30.787 INFO:tasks.rados.rados.0.smithi151.stdout:3942: finishing rollback tid 1 to smithi15186934-30 2024-09-13T14:37:30.787 INFO:tasks.rados.rados.0.smithi151.stdout:3942: finishing rollback tid 2 to smithi15186934-30 2024-09-13T14:37:30.787 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 30 v 1494 (ObjNum 1419 snap 477 seq_num 1419) dirty exists 2024-09-13T14:37:31.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:30 smithi184 ceph-mon[92527]: pgmap v1206: 89 pgs: 89 active+clean; 211 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 6.9 MiB/s rd, 329 KiB/s wr, 6 op/s 2024-09-13T14:37:31.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:30 smithi184 ceph-mon[92527]: osdmap e1060: 8 total, 8 up, 8 in 2024-09-13T14:37:31.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:30 smithi151 ceph-mon[104450]: pgmap v1206: 89 pgs: 89 active+clean; 211 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 6.9 MiB/s rd, 329 KiB/s wr, 6 op/s 2024-09-13T14:37:31.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:30 smithi151 ceph-mon[104450]: osdmap e1060: 8 total, 8 up, 8 in 2024-09-13T14:37:31.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:30 smithi151 ceph-mon[108966]: pgmap v1206: 89 pgs: 89 active+clean; 211 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 6.9 MiB/s rd, 329 KiB/s wr, 6 op/s 2024-09-13T14:37:31.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:30 smithi151 ceph-mon[108966]: osdmap e1060: 8 total, 8 up, 8 in 2024-09-13T14:37:31.718 INFO:tasks.rados.rados.0.smithi151.stdout:3941: done (2 left) 2024-09-13T14:37:31.719 INFO:tasks.rados.rados.0.smithi151.stdout:3942: done (1 left) 2024-09-13T14:37:31.719 INFO:tasks.rados.rados.0.smithi151.stdout:3943: done (0 left) 2024-09-13T14:37:31.719 INFO:tasks.rados.rados.0.smithi151.stdout:3944: write oid 20 current snap is 500 2024-09-13T14:37:31.719 INFO:tasks.rados.rados.0.smithi151.stdout:3944: seq_num 1494 ranges {709658=427194,1592098=726857,2761010=348648} 2024-09-13T14:37:31.724 INFO:tasks.rados.rados.0.smithi151.stdout:3944: writing smithi15186934-20 from 709658 to 1136852 tid 1 2024-09-13T14:37:31.731 INFO:tasks.rados.rados.0.smithi151.stdout:3944: writing smithi15186934-20 from 1592098 to 2318955 tid 2 2024-09-13T14:37:31.735 INFO:tasks.rados.rados.0.smithi151.stdout:3944: writing smithi15186934-20 from 2761010 to 3109658 tid 3 2024-09-13T14:37:31.735 INFO:tasks.rados.rados.0.smithi151.stdout:3945: write oid 34 current snap is 500 2024-09-13T14:37:31.735 INFO:tasks.rados.rados.0.smithi151.stdout:3945: seq_num 1495 ranges {726465=502443,1650580=470515,2788433=338032} 2024-09-13T14:37:31.741 INFO:tasks.rados.rados.0.smithi151.stdout:3945: writing smithi15186934-34 from 726465 to 1228908 tid 1 2024-09-13T14:37:31.747 INFO:tasks.rados.rados.0.smithi151.stdout:3945: writing smithi15186934-34 from 1650580 to 2121095 tid 2 2024-09-13T14:37:31.753 INFO:tasks.rados.rados.0.smithi151.stdout:3945: writing smithi15186934-34 from 2788433 to 3126465 tid 3 2024-09-13T14:37:31.753 INFO:tasks.rados.rados.0.smithi151.stdout:3946: read oid 46 snap -1 2024-09-13T14:37:31.753 INFO:tasks.rados.rados.0.smithi151.stdout:3946: expect (ObjNum 1468 snap 494 seq_num 1468) 2024-09-13T14:37:31.753 INFO:tasks.rados.rados.0.smithi151.stdout:3947: snap_create 2024-09-13T14:37:31.753 INFO:tasks.rados.rados.0.smithi151.stdout:3944: finishing write tid 1 to smithi15186934-20 2024-09-13T14:37:31.754 INFO:tasks.rados.rados.0.smithi151.stdout:3944: finishing write tid 2 to smithi15186934-20 2024-09-13T14:37:31.754 INFO:tasks.rados.rados.0.smithi151.stdout:3944: finishing write tid 3 to smithi15186934-20 2024-09-13T14:37:31.754 INFO:tasks.rados.rados.0.smithi151.stdout:3944: finishing write tid 4 to smithi15186934-20 2024-09-13T14:37:31.754 INFO:tasks.rados.rados.0.smithi151.stdout:3944: finishing write tid 5 to smithi15186934-20 2024-09-13T14:37:31.754 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 20 v 1230 (ObjNum 1494 snap 500 seq_num 1494) dirty exists 2024-09-13T14:37:31.754 INFO:tasks.rados.rados.0.smithi151.stdout:3944: left oid 20 (ObjNum 1494 snap 500 seq_num 1494) 2024-09-13T14:37:31.754 INFO:tasks.rados.rados.0.smithi151.stdout:3945: finishing write tid 1 to smithi15186934-34 2024-09-13T14:37:31.754 INFO:tasks.rados.rados.0.smithi151.stdout:3945: finishing write tid 2 to smithi15186934-34 2024-09-13T14:37:31.756 INFO:tasks.rados.rados.0.smithi151.stdout:3945: finishing write tid 3 to smithi15186934-34 2024-09-13T14:37:31.756 INFO:tasks.rados.rados.0.smithi151.stdout:3946: expect (ObjNum 1468 snap 494 seq_num 1468) 2024-09-13T14:37:31.842 INFO:tasks.rados.rados.0.smithi151.stdout:3945: finishing write tid 4 to smithi15186934-34 2024-09-13T14:37:31.843 INFO:tasks.rados.rados.0.smithi151.stdout:3945: finishing write tid 5 to smithi15186934-34 2024-09-13T14:37:31.843 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 34 v 1498 (ObjNum 1495 snap 500 seq_num 1495) dirty exists 2024-09-13T14:37:31.843 INFO:tasks.rados.rados.0.smithi151.stdout:3945: left oid 34 (ObjNum 1495 snap 500 seq_num 1495) 2024-09-13T14:37:32.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:32.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:32.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:32.730 INFO:tasks.rados.rados.0.smithi151.stdout:3944: done (3 left) 2024-09-13T14:37:32.730 INFO:tasks.rados.rados.0.smithi151.stdout:3945: done (2 left) 2024-09-13T14:37:32.730 INFO:tasks.rados.rados.0.smithi151.stdout:3946: done (1 left) 2024-09-13T14:37:32.730 INFO:tasks.rados.rados.0.smithi151.stdout:3947: done (0 left) 2024-09-13T14:37:32.730 INFO:tasks.rados.rados.0.smithi151.stdout:3948: snap_create 2024-09-13T14:37:33.010 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:32 smithi151 ceph-mon[104450]: pgmap v1209: 89 pgs: 89 active+clean; 213 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 361 KiB/s wr, 1 op/s 2024-09-13T14:37:33.010 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:32 smithi151 ceph-mon[104450]: osdmap e1061: 8 total, 8 up, 8 in 2024-09-13T14:37:33.010 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:32 smithi151 ceph-mon[108966]: pgmap v1209: 89 pgs: 89 active+clean; 213 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 361 KiB/s wr, 1 op/s 2024-09-13T14:37:33.010 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:32 smithi151 ceph-mon[108966]: osdmap e1061: 8 total, 8 up, 8 in 2024-09-13T14:37:33.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:32 smithi184 ceph-mon[92527]: pgmap v1209: 89 pgs: 89 active+clean; 213 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 361 KiB/s wr, 1 op/s 2024-09-13T14:37:33.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:32 smithi184 ceph-mon[92527]: osdmap e1061: 8 total, 8 up, 8 in 2024-09-13T14:37:33.738 INFO:tasks.rados.rados.0.smithi151.stdout:3948: done (0 left) 2024-09-13T14:37:33.739 INFO:tasks.rados.rados.0.smithi151.stdout:3949: snap_remove snap 479 2024-09-13T14:37:34.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:33 smithi184 ceph-mon[92527]: osdmap e1062: 8 total, 8 up, 8 in 2024-09-13T14:37:34.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:33 smithi151 ceph-mon[104450]: osdmap e1062: 8 total, 8 up, 8 in 2024-09-13T14:37:34.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:33 smithi151 ceph-mon[108966]: osdmap e1062: 8 total, 8 up, 8 in 2024-09-13T14:37:34.678 INFO:tasks.rados.rados.0.smithi151.stdout:3949: done (0 left) 2024-09-13T14:37:34.678 INFO:tasks.rados.rados.0.smithi151.stdout:3950: write oid 6 current snap is 502 2024-09-13T14:37:34.678 INFO:tasks.rados.rados.0.smithi151.stdout:3950: seq_num 1496 ranges {743272=577692,2109062=730526} 2024-09-13T14:37:34.686 INFO:tasks.rados.rados.0.smithi151.stdout:3950: writing smithi15186934-6 from 743272 to 1320964 tid 1 2024-09-13T14:37:34.696 INFO:tasks.rados.rados.0.smithi151.stdout:3950: writing smithi15186934-6 from 2109062 to 2839588 tid 2 2024-09-13T14:37:34.696 INFO:tasks.rados.rados.0.smithi151.stdout:3951: write oid 2 current snap is 502 2024-09-13T14:37:34.696 INFO:tasks.rados.rados.0.smithi151.stdout:3950: finishing write tid 1 to smithi15186934-6 2024-09-13T14:37:34.696 INFO:tasks.rados.rados.0.smithi151.stdout:3951: seq_num 1497 ranges {760079=652941,2167544=474184} 2024-09-13T14:37:34.704 INFO:tasks.rados.rados.0.smithi151.stdout:3951: writing smithi15186934-2 from 760079 to 1413020 tid 1 2024-09-13T14:37:34.710 INFO:tasks.rados.rados.0.smithi151.stdout:3951: writing smithi15186934-2 from 2167544 to 2641728 tid 2 2024-09-13T14:37:34.710 INFO:tasks.rados.rados.0.smithi151.stdout:3952: snap_remove snap 492 2024-09-13T14:37:34.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:34 smithi151 ceph-mon[108966]: pgmap v1212: 89 pgs: 89 active+clean; 220 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.6 MiB/s wr, 6 op/s 2024-09-13T14:37:34.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:34 smithi151 ceph-mon[108966]: osdmap e1063: 8 total, 8 up, 8 in 2024-09-13T14:37:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:34 smithi151 ceph-mon[108966]: osdmap e1064: 8 total, 8 up, 8 in 2024-09-13T14:37:34.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:34 smithi151 ceph-mon[104450]: pgmap v1212: 89 pgs: 89 active+clean; 220 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.6 MiB/s wr, 6 op/s 2024-09-13T14:37:34.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:34 smithi151 ceph-mon[104450]: osdmap e1063: 8 total, 8 up, 8 in 2024-09-13T14:37:34.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:34 smithi151 ceph-mon[104450]: osdmap e1064: 8 total, 8 up, 8 in 2024-09-13T14:37:34.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:37:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:37:34] "GET /metrics HTTP/1.1" 200 36510 "" "Prometheus/2.51.0" 2024-09-13T14:37:35.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:34 smithi184 ceph-mon[92527]: pgmap v1212: 89 pgs: 89 active+clean; 220 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.6 MiB/s wr, 6 op/s 2024-09-13T14:37:35.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:34 smithi184 ceph-mon[92527]: osdmap e1063: 8 total, 8 up, 8 in 2024-09-13T14:37:35.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:34 smithi184 ceph-mon[92527]: osdmap e1064: 8 total, 8 up, 8 in 2024-09-13T14:37:35.681 INFO:tasks.rados.rados.0.smithi151.stdout:3950: finishing write tid 2 to smithi15186934-6 2024-09-13T14:37:35.682 INFO:tasks.rados.rados.0.smithi151.stdout:3950: finishing write tid 3 to smithi15186934-6 2024-09-13T14:37:35.682 INFO:tasks.rados.rados.0.smithi151.stdout:3950: finishing write tid 4 to smithi15186934-6 2024-09-13T14:37:35.682 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 6 v 871 (ObjNum 1496 snap 502 seq_num 1496) dirty exists 2024-09-13T14:37:35.682 INFO:tasks.rados.rados.0.smithi151.stdout:3950: left oid 6 (ObjNum 1496 snap 502 seq_num 1496) 2024-09-13T14:37:35.682 INFO:tasks.rados.rados.0.smithi151.stdout:3951: finishing write tid 1 to smithi15186934-2 2024-09-13T14:37:35.682 INFO:tasks.rados.rados.0.smithi151.stdout:3951: finishing write tid 2 to smithi15186934-2 2024-09-13T14:37:35.682 INFO:tasks.rados.rados.0.smithi151.stdout:3951: finishing write tid 3 to smithi15186934-2 2024-09-13T14:37:35.682 INFO:tasks.rados.rados.0.smithi151.stdout:3951: finishing write tid 4 to smithi15186934-2 2024-09-13T14:37:35.682 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 2 v 1609 (ObjNum 1497 snap 502 seq_num 1497) dirty exists 2024-09-13T14:37:35.682 INFO:tasks.rados.rados.0.smithi151.stdout:3951: left oid 2 (ObjNum 1497 snap 502 seq_num 1497) 2024-09-13T14:37:35.682 INFO:tasks.rados.rados.0.smithi151.stdout:3950: done (2 left) 2024-09-13T14:37:35.682 INFO:tasks.rados.rados.0.smithi151.stdout:3951: done (1 left) 2024-09-13T14:37:35.682 INFO:tasks.rados.rados.0.smithi151.stdout:3952: done (0 left) 2024-09-13T14:37:35.683 INFO:tasks.rados.rados.0.smithi151.stdout:3953: snap_remove snap 501 2024-09-13T14:37:36.682 INFO:tasks.rados.rados.0.smithi151.stdout:3953: done (0 left) 2024-09-13T14:37:36.682 INFO:tasks.rados.rados.0.smithi151.stdout:3954: snap_remove snap 499 2024-09-13T14:37:37.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:36 smithi184 ceph-mon[92527]: pgmap v1215: 89 pgs: 89 active+clean; 220 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 3.9 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:37:37.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:36 smithi184 ceph-mon[92527]: osdmap e1065: 8 total, 8 up, 8 in 2024-09-13T14:37:37.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:36 smithi151 ceph-mon[108966]: pgmap v1215: 89 pgs: 89 active+clean; 220 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 3.9 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:37:37.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:36 smithi151 ceph-mon[108966]: osdmap e1065: 8 total, 8 up, 8 in 2024-09-13T14:37:37.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:36 smithi151 ceph-mon[104450]: pgmap v1215: 89 pgs: 89 active+clean; 220 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 3.9 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-13T14:37:37.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:36 smithi151 ceph-mon[104450]: osdmap e1065: 8 total, 8 up, 8 in 2024-09-13T14:37:37.685 INFO:tasks.rados.rados.0.smithi151.stdout:3954: done (0 left) 2024-09-13T14:37:37.685 INFO:tasks.rados.rados.0.smithi151.stdout:3955: delete oid 30 current snap is 502 2024-09-13T14:37:37.694 INFO:tasks.rados.rados.0.smithi151.stdout:3955: done (0 left) 2024-09-13T14:37:37.694 INFO:tasks.rados.rados.0.smithi151.stdout:3956: write oid 40 current snap is 502 2024-09-13T14:37:37.694 INFO:tasks.rados.rados.0.smithi151.stdout:3956: seq_num 1499 ranges {793693=519792,1884508=477853,2898125=295568} 2024-09-13T14:37:37.701 INFO:tasks.rados.rados.0.smithi151.stdout:3956: writing smithi15186934-40 from 793693 to 1313485 tid 1 2024-09-13T14:37:37.706 INFO:tasks.rados.rados.0.smithi151.stdout:3956: writing smithi15186934-40 from 1884508 to 2362361 tid 2 2024-09-13T14:37:37.712 INFO:tasks.rados.rados.0.smithi151.stdout:3956: writing smithi15186934-40 from 2898125 to 3193693 tid 3 2024-09-13T14:37:37.712 INFO:tasks.rados.rados.0.smithi151.stdout:3957: read oid 20 snap -1 2024-09-13T14:37:37.712 INFO:tasks.rados.rados.0.smithi151.stdout:3957: expect (ObjNum 1494 snap 500 seq_num 1494) 2024-09-13T14:37:37.712 INFO:tasks.rados.rados.0.smithi151.stdout:3958: read oid 23 snap -1 2024-09-13T14:37:37.712 INFO:tasks.rados.rados.0.smithi151.stdout:3958: expect (ObjNum 1473 snap 494 seq_num 1473) 2024-09-13T14:37:37.712 INFO:tasks.rados.rados.0.smithi151.stdout:3959: read oid 8 snap -1 2024-09-13T14:37:37.712 INFO:tasks.rados.rados.0.smithi151.stdout:3959: expect deleted 2024-09-13T14:37:37.712 INFO:tasks.rados.rados.0.smithi151.stdout:3960: read oid 24 snap -1 2024-09-13T14:37:37.712 INFO:tasks.rados.rados.0.smithi151.stdout:3960: expect (ObjNum 1483 snap 496 seq_num 1483) 2024-09-13T14:37:37.712 INFO:tasks.rados.rados.0.smithi151.stdout:3961: snap_remove snap 495 2024-09-13T14:37:38.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:37 smithi184 ceph-mon[92527]: osdmap e1066: 8 total, 8 up, 8 in 2024-09-13T14:37:38.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:37 smithi151 ceph-mon[104450]: osdmap e1066: 8 total, 8 up, 8 in 2024-09-13T14:37:38.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:37 smithi151 ceph-mon[108966]: osdmap e1066: 8 total, 8 up, 8 in 2024-09-13T14:37:38.708 INFO:tasks.rados.rados.0.smithi151.stdout:3961: done (5 left) 2024-09-13T14:37:38.708 INFO:tasks.rados.rados.0.smithi151.stdout:3962: read oid 2 snap -1 2024-09-13T14:37:38.708 INFO:tasks.rados.rados.0.smithi151.stdout:3962: expect (ObjNum 1497 snap 502 seq_num 1497) 2024-09-13T14:37:38.708 INFO:tasks.rados.rados.0.smithi151.stdout:3963: delete oid 12 current snap is 502 2024-09-13T14:37:38.708 INFO:tasks.rados.rados.0.smithi151.stdout:3956: finishing write tid 1 to smithi15186934-40 2024-09-13T14:37:38.709 INFO:tasks.rados.rados.0.smithi151.stdout:3956: finishing write tid 2 to smithi15186934-40 2024-09-13T14:37:38.709 INFO:tasks.rados.rados.0.smithi151.stdout:3956: finishing write tid 3 to smithi15186934-40 2024-09-13T14:37:38.709 INFO:tasks.rados.rados.0.smithi151.stdout:3956: finishing write tid 4 to smithi15186934-40 2024-09-13T14:37:38.709 INFO:tasks.rados.rados.0.smithi151.stdout:3956: finishing write tid 5 to smithi15186934-40 2024-09-13T14:37:38.709 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 40 v 1513 (ObjNum 1499 snap 502 seq_num 1499) dirty exists 2024-09-13T14:37:38.709 INFO:tasks.rados.rados.0.smithi151.stdout:3956: left oid 40 (ObjNum 1499 snap 502 seq_num 1499) 2024-09-13T14:37:38.709 INFO:tasks.rados.rados.0.smithi151.stdout:3960: expect (ObjNum 1483 snap 496 seq_num 1483) 2024-09-13T14:37:38.836 INFO:tasks.rados.rados.0.smithi151.stdout:3958: expect (ObjNum 1473 snap 494 seq_num 1473) 2024-09-13T14:37:38.958 INFO:tasks.rados.rados.0.smithi151.stdout:3957: expect (ObjNum 1494 snap 500 seq_num 1494) 2024-09-13T14:37:39.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:38 smithi184 ceph-mon[92527]: pgmap v1218: 89 pgs: 89 active+clean; 220 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 3.9 MiB/s rd, 640 KiB/s wr, 7 op/s 2024-09-13T14:37:39.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:38 smithi184 ceph-mon[92527]: osdmap e1067: 8 total, 8 up, 8 in 2024-09-13T14:37:39.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:38 smithi151 ceph-mon[108966]: pgmap v1218: 89 pgs: 89 active+clean; 220 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 3.9 MiB/s rd, 640 KiB/s wr, 7 op/s 2024-09-13T14:37:39.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:38 smithi151 ceph-mon[108966]: osdmap e1067: 8 total, 8 up, 8 in 2024-09-13T14:37:39.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:38 smithi151 ceph-mon[104450]: pgmap v1218: 89 pgs: 89 active+clean; 220 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 3.9 MiB/s rd, 640 KiB/s wr, 7 op/s 2024-09-13T14:37:39.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:38 smithi151 ceph-mon[104450]: osdmap e1067: 8 total, 8 up, 8 in 2024-09-13T14:37:39.128 INFO:tasks.rados.rados.0.smithi151.stdout:3962: expect (ObjNum 1497 snap 502 seq_num 1497) 2024-09-13T14:37:39.312 INFO:tasks.rados.rados.0.smithi151.stdout:3956: done (6 left) 2024-09-13T14:37:39.312 INFO:tasks.rados.rados.0.smithi151.stdout:3957: done (5 left) 2024-09-13T14:37:39.313 INFO:tasks.rados.rados.0.smithi151.stdout:3958: done (4 left) 2024-09-13T14:37:39.313 INFO:tasks.rados.rados.0.smithi151.stdout:3959: done (3 left) 2024-09-13T14:37:39.313 INFO:tasks.rados.rados.0.smithi151.stdout:3960: done (2 left) 2024-09-13T14:37:39.313 INFO:tasks.rados.rados.0.smithi151.stdout:3962: done (1 left) 2024-09-13T14:37:39.313 INFO:tasks.rados.rados.0.smithi151.stdout:3963: done (0 left) 2024-09-13T14:37:39.313 INFO:tasks.rados.rados.0.smithi151.stdout:3964: rollback oid 43 current snap is 502 2024-09-13T14:37:39.313 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 43 to 493 2024-09-13T14:37:39.313 INFO:tasks.rados.rados.0.smithi151.stdout:3965: delete oid 7 current snap is 502 2024-09-13T14:37:39.316 INFO:tasks.rados.rados.0.smithi151.stdout:3964: finishing rollback tid 1 to smithi15186934-43 2024-09-13T14:37:39.316 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 43 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-13T14:37:39.316 INFO:tasks.rados.rados.0.smithi151.stdout:3964: done (1 left) 2024-09-13T14:37:39.317 INFO:tasks.rados.rados.0.smithi151.stdout:3965: done (0 left) 2024-09-13T14:37:39.317 INFO:tasks.rados.rados.0.smithi151.stdout:3966: read oid 44 snap -1 2024-09-13T14:37:39.317 INFO:tasks.rados.rados.0.smithi151.stdout:3966: expect (ObjNum 1474 snap 494 seq_num 1474) 2024-09-13T14:37:39.317 INFO:tasks.rados.rados.0.smithi151.stdout:3967: write oid 30 current snap is 502 2024-09-13T14:37:39.317 INFO:tasks.rados.rados.0.smithi151.stdout:3967: seq_num 1502 ranges {444114=745539,1943601=625180} 2024-09-13T14:37:39.323 INFO:tasks.rados.rados.0.smithi151.stdout:3967: writing smithi15186934-30 from 444114 to 1189653 tid 1 2024-09-13T14:37:39.331 INFO:tasks.rados.rados.0.smithi151.stdout:3967: writing smithi15186934-30 from 1943601 to 2568781 tid 2 2024-09-13T14:37:39.331 INFO:tasks.rados.rados.0.smithi151.stdout:3968: read oid 18 snap -1 2024-09-13T14:37:39.331 INFO:tasks.rados.rados.0.smithi151.stdout:3968: expect (ObjNum 1487 snap 497 seq_num 1487) 2024-09-13T14:37:39.331 INFO:tasks.rados.rados.0.smithi151.stdout:3969: snap_create 2024-09-13T14:37:39.331 INFO:tasks.rados.rados.0.smithi151.stdout:3966: expect (ObjNum 1474 snap 494 seq_num 1474) 2024-09-13T14:37:39.491 INFO:tasks.rados.rados.0.smithi151.stdout:3967: finishing write tid 1 to smithi15186934-30 2024-09-13T14:37:39.491 INFO:tasks.rados.rados.0.smithi151.stdout:3967: finishing write tid 2 to smithi15186934-30 2024-09-13T14:37:39.492 INFO:tasks.rados.rados.0.smithi151.stdout:3967: finishing write tid 3 to smithi15186934-30 2024-09-13T14:37:39.492 INFO:tasks.rados.rados.0.smithi151.stdout:3967: finishing write tid 4 to smithi15186934-30 2024-09-13T14:37:39.492 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 30 v 1520 (ObjNum 1502 snap 502 seq_num 1502) dirty exists 2024-09-13T14:37:39.492 INFO:tasks.rados.rados.0.smithi151.stdout:3967: left oid 30 (ObjNum 1502 snap 502 seq_num 1502) 2024-09-13T14:37:39.492 INFO:tasks.rados.rados.0.smithi151.stdout:3968: expect (ObjNum 1487 snap 497 seq_num 1487) 2024-09-13T14:37:39.682 INFO:tasks.rados.rados.0.smithi151.stdout:3966: done (3 left) 2024-09-13T14:37:39.682 INFO:tasks.rados.rados.0.smithi151.stdout:3967: done (2 left) 2024-09-13T14:37:39.682 INFO:tasks.rados.rados.0.smithi151.stdout:3968: done (1 left) 2024-09-13T14:37:39.682 INFO:tasks.rados.rados.0.smithi151.stdout:3969: done (0 left) 2024-09-13T14:37:39.682 INFO:tasks.rados.rados.0.smithi151.stdout:3970: write oid 43 current snap is 503 2024-09-13T14:37:39.682 INFO:tasks.rados.rados.0.smithi151.stdout:3970: seq_num 1503 ranges {460921=420788,1602083=485191,2491464=621953} 2024-09-13T14:37:39.686 INFO:tasks.rados.rados.0.smithi151.stdout:3970: writing smithi15186934-43 from 460921 to 881709 tid 1 2024-09-13T14:37:39.693 INFO:tasks.rados.rados.0.smithi151.stdout:3970: writing smithi15186934-43 from 1602083 to 2087274 tid 2 2024-09-13T14:37:39.699 INFO:tasks.rados.rados.0.smithi151.stdout:3970: writing smithi15186934-43 from 2491464 to 3113417 tid 3 2024-09-13T14:37:39.699 INFO:tasks.rados.rados.0.smithi151.stdout:3971: write oid 18 current snap is 503 2024-09-13T14:37:39.699 INFO:tasks.rados.rados.0.smithi151.stdout:3971: seq_num 1504 ranges {477728=496037,1660565=628849,2918887=358841} 2024-09-13T14:37:39.705 INFO:tasks.rados.rados.0.smithi151.stdout:3971: writing smithi15186934-18 from 477728 to 973765 tid 1 2024-09-13T14:37:39.713 INFO:tasks.rados.rados.0.smithi151.stdout:3971: writing smithi15186934-18 from 1660565 to 2289414 tid 2 2024-09-13T14:37:39.718 INFO:tasks.rados.rados.0.smithi151.stdout:3971: writing smithi15186934-18 from 2918887 to 3277728 tid 3 2024-09-13T14:37:39.718 INFO:tasks.rados.rados.0.smithi151.stdout:3972: write oid 3 current snap is 503 2024-09-13T14:37:39.719 INFO:tasks.rados.rados.0.smithi151.stdout:3970: finishing write tid 1 to smithi15186934-43 2024-09-13T14:37:39.719 INFO:tasks.rados.rados.0.smithi151.stdout:3970: finishing write tid 2 to smithi15186934-43 2024-09-13T14:37:39.719 INFO:tasks.rados.rados.0.smithi151.stdout:3970: finishing write tid 3 to smithi15186934-43 2024-09-13T14:37:39.719 INFO:tasks.rados.rados.0.smithi151.stdout:3972: seq_num 1505 ranges {494535=571286,1719047=488860,2946310=348225} 2024-09-13T14:37:39.724 INFO:tasks.rados.rados.0.smithi151.stdout:3972: writing smithi15186934-3 from 494535 to 1065821 tid 1 2024-09-13T14:37:39.730 INFO:tasks.rados.rados.0.smithi151.stdout:3972: writing smithi15186934-3 from 1719047 to 2207907 tid 2 2024-09-13T14:37:39.737 INFO:tasks.rados.rados.0.smithi151.stdout:3972: writing smithi15186934-3 from 2946310 to 3294535 tid 3 2024-09-13T14:37:39.737 INFO:tasks.rados.rados.0.smithi151.stdout:3973: read oid 8 snap 502 2024-09-13T14:37:39.737 INFO:tasks.rados.rados.0.smithi151.stdout:3973: expect deleted 2024-09-13T14:37:39.737 INFO:tasks.rados.rados.0.smithi151.stdout:3970: finishing write tid 4 to smithi15186934-43 2024-09-13T14:37:39.737 INFO:tasks.rados.rados.0.smithi151.stdout:3970: finishing write tid 5 to smithi15186934-43 2024-09-13T14:37:39.737 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 43 v 2218 (ObjNum 1503 snap 503 seq_num 1503) dirty exists 2024-09-13T14:37:39.737 INFO:tasks.rados.rados.0.smithi151.stdout:3970: left oid 43 (ObjNum 1503 snap 503 seq_num 1503) 2024-09-13T14:37:39.737 INFO:tasks.rados.rados.0.smithi151.stdout:3971: finishing write tid 1 to smithi15186934-18 2024-09-13T14:37:39.737 INFO:tasks.rados.rados.0.smithi151.stdout:3971: finishing write tid 2 to smithi15186934-18 2024-09-13T14:37:39.738 INFO:tasks.rados.rados.0.smithi151.stdout:3974: read oid 9 snap 491 2024-09-13T14:37:39.738 INFO:tasks.rados.rados.0.smithi151.stdout:3974: expect deleted 2024-09-13T14:37:39.738 INFO:tasks.rados.rados.0.smithi151.stdout:3970: done (4 left) 2024-09-13T14:37:39.738 INFO:tasks.rados.rados.0.smithi151.stdout:3975: snap_remove snap 482 2024-09-13T14:37:40.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:39 smithi184 ceph-mon[92527]: osdmap e1068: 8 total, 8 up, 8 in 2024-09-13T14:37:40.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:39 smithi184 ceph-mon[92527]: osdmap e1069: 8 total, 8 up, 8 in 2024-09-13T14:37:40.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:39 smithi151 ceph-mon[104450]: osdmap e1068: 8 total, 8 up, 8 in 2024-09-13T14:37:40.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:39 smithi151 ceph-mon[104450]: osdmap e1069: 8 total, 8 up, 8 in 2024-09-13T14:37:40.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:39 smithi151 ceph-mon[108966]: osdmap e1068: 8 total, 8 up, 8 in 2024-09-13T14:37:40.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:39 smithi151 ceph-mon[108966]: osdmap e1069: 8 total, 8 up, 8 in 2024-09-13T14:37:40.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:37:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:37:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:37:40.715 INFO:tasks.rados.rados.0.smithi151.stdout:3975: done (4 left) 2024-09-13T14:37:40.715 INFO:tasks.rados.rados.0.smithi151.stdout:3976: snap_create 2024-09-13T14:37:40.715 INFO:tasks.rados.rados.0.smithi151.stdout:3971: finishing write tid 3 to smithi15186934-18 2024-09-13T14:37:40.715 INFO:tasks.rados.rados.0.smithi151.stdout:3971: finishing write tid 4 to smithi15186934-18 2024-09-13T14:37:40.715 INFO:tasks.rados.rados.0.smithi151.stdout:3971: finishing write tid 5 to smithi15186934-18 2024-09-13T14:37:40.715 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 18 v 1096 (ObjNum 1504 snap 503 seq_num 1504) dirty exists 2024-09-13T14:37:40.715 INFO:tasks.rados.rados.0.smithi151.stdout:3971: left oid 18 (ObjNum 1504 snap 503 seq_num 1504) 2024-09-13T14:37:40.715 INFO:tasks.rados.rados.0.smithi151.stdout:3972: finishing write tid 1 to smithi15186934-3 2024-09-13T14:37:40.716 INFO:tasks.rados.rados.0.smithi151.stdout:3972: finishing write tid 2 to smithi15186934-3 2024-09-13T14:37:40.716 INFO:tasks.rados.rados.0.smithi151.stdout:3972: finishing write tid 3 to smithi15186934-3 2024-09-13T14:37:40.716 INFO:tasks.rados.rados.0.smithi151.stdout:3972: finishing write tid 4 to smithi15186934-3 2024-09-13T14:37:40.716 INFO:tasks.rados.rados.0.smithi151.stdout:3972: finishing write tid 5 to smithi15186934-3 2024-09-13T14:37:40.716 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 3 v 1525 (ObjNum 1505 snap 503 seq_num 1505) dirty exists 2024-09-13T14:37:40.716 INFO:tasks.rados.rados.0.smithi151.stdout:3972: left oid 3 (ObjNum 1505 snap 503 seq_num 1505) 2024-09-13T14:37:41.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:40 smithi184 ceph-mon[92527]: pgmap v1221: 89 pgs: 89 active+clean; 222 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 915 KiB/s wr, 6 op/s 2024-09-13T14:37:41.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:40 smithi151 ceph-mon[104450]: pgmap v1221: 89 pgs: 89 active+clean; 222 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 915 KiB/s wr, 6 op/s 2024-09-13T14:37:41.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:40 smithi151 ceph-mon[108966]: pgmap v1221: 89 pgs: 89 active+clean; 222 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 2.4 MiB/s rd, 915 KiB/s wr, 6 op/s 2024-09-13T14:37:41.722 INFO:tasks.rados.rados.0.smithi151.stdout:3971: done (4 left) 2024-09-13T14:37:41.722 INFO:tasks.rados.rados.0.smithi151.stdout:3972: done (3 left) 2024-09-13T14:37:41.722 INFO:tasks.rados.rados.0.smithi151.stdout:3973: done (2 left) 2024-09-13T14:37:41.722 INFO:tasks.rados.rados.0.smithi151.stdout:3974: done (1 left) 2024-09-13T14:37:41.722 INFO:tasks.rados.rados.0.smithi151.stdout:3976: done (0 left) 2024-09-13T14:37:41.722 INFO:tasks.rados.rados.0.smithi151.stdout:3977: write oid 44 current snap is 504 2024-09-13T14:37:41.722 INFO:tasks.rados.rados.0.smithi151.stdout:3977: seq_num 1506 ranges {511342=762888,1777529=632518,2973733=337609} 2024-09-13T14:37:41.728 INFO:tasks.rados.rados.0.smithi151.stdout:3977: writing smithi15186934-44 from 511342 to 1274230 tid 1 2024-09-13T14:37:41.735 INFO:tasks.rados.rados.0.smithi151.stdout:3977: writing smithi15186934-44 from 1777529 to 2410047 tid 2 2024-09-13T14:37:41.740 INFO:tasks.rados.rados.0.smithi151.stdout:3977: writing smithi15186934-44 from 2973733 to 3311342 tid 3 2024-09-13T14:37:41.740 INFO:tasks.rados.rados.0.smithi151.stdout:3978: rollback oid 33 current snap is 504 2024-09-13T14:37:41.740 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 33 to 470 2024-09-13T14:37:41.740 INFO:tasks.rados.rados.0.smithi151.stdout:3977: finishing write tid 1 to smithi15186934-44 2024-09-13T14:37:41.740 INFO:tasks.rados.rados.0.smithi151.stdout:3977: finishing write tid 2 to smithi15186934-44 2024-09-13T14:37:41.740 INFO:tasks.rados.rados.0.smithi151.stdout:3979: snap_remove snap 500 2024-09-13T14:37:42.005 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:41 smithi184 ceph-mon[92527]: osdmap e1070: 8 total, 8 up, 8 in 2024-09-13T14:37:42.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:41 smithi151 ceph-mon[104450]: osdmap e1070: 8 total, 8 up, 8 in 2024-09-13T14:37:42.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:41 smithi151 ceph-mon[108966]: osdmap e1070: 8 total, 8 up, 8 in 2024-09-13T14:37:42.728 INFO:tasks.rados.rados.0.smithi151.stdout:3979: done (2 left) 2024-09-13T14:37:42.728 INFO:tasks.rados.rados.0.smithi151.stdout:3980: read oid 5 snap -1 2024-09-13T14:37:42.728 INFO:tasks.rados.rados.0.smithi151.stdout:3980: expect (ObjNum 1370 snap 463 seq_num 1370) 2024-09-13T14:37:42.728 INFO:tasks.rados.rados.0.smithi151.stdout:3981: snap_create 2024-09-13T14:37:42.728 INFO:tasks.rados.rados.0.smithi151.stdout:3977: finishing write tid 3 to smithi15186934-44 2024-09-13T14:37:42.728 INFO:tasks.rados.rados.0.smithi151.stdout:3977: finishing write tid 4 to smithi15186934-44 2024-09-13T14:37:42.728 INFO:tasks.rados.rados.0.smithi151.stdout:3977: finishing write tid 5 to smithi15186934-44 2024-09-13T14:37:42.728 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 44 v 1350 (ObjNum 1506 snap 504 seq_num 1506) dirty exists 2024-09-13T14:37:42.728 INFO:tasks.rados.rados.0.smithi151.stdout:3977: left oid 44 (ObjNum 1506 snap 504 seq_num 1506) 2024-09-13T14:37:42.729 INFO:tasks.rados.rados.0.smithi151.stdout:3978: finishing rollback tid 0 to smithi15186934-33 2024-09-13T14:37:42.729 INFO:tasks.rados.rados.0.smithi151.stdout:3978: finishing rollback tid 1 to smithi15186934-33 2024-09-13T14:37:42.729 INFO:tasks.rados.rados.0.smithi151.stdout:3978: finishing rollback tid 2 to smithi15186934-33 2024-09-13T14:37:42.729 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 33 v 1627 (ObjNum 1058 snap 357 seq_num 1058) dirty exists 2024-09-13T14:37:42.736 INFO:tasks.rados.rados.0.smithi151.stdout:3980: expect (ObjNum 1370 snap 463 seq_num 1370) 2024-09-13T14:37:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:42 smithi184 ceph-mon[92527]: pgmap v1224: 89 pgs: 89 active+clean; 230 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 6.6 MiB/s rd, 1.6 MiB/s wr, 13 op/s 2024-09-13T14:37:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:42 smithi184 ceph-mon[92527]: osdmap e1071: 8 total, 8 up, 8 in 2024-09-13T14:37:43.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:42 smithi151 ceph-mon[104450]: pgmap v1224: 89 pgs: 89 active+clean; 230 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 6.6 MiB/s rd, 1.6 MiB/s wr, 13 op/s 2024-09-13T14:37:43.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:42 smithi151 ceph-mon[104450]: osdmap e1071: 8 total, 8 up, 8 in 2024-09-13T14:37:43.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:42 smithi151 ceph-mon[108966]: pgmap v1224: 89 pgs: 89 active+clean; 230 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 6.6 MiB/s rd, 1.6 MiB/s wr, 13 op/s 2024-09-13T14:37:43.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:42 smithi151 ceph-mon[108966]: osdmap e1071: 8 total, 8 up, 8 in 2024-09-13T14:37:43.737 INFO:tasks.rados.rados.0.smithi151.stdout:3977: done (3 left) 2024-09-13T14:37:43.737 INFO:tasks.rados.rados.0.smithi151.stdout:3978: done (2 left) 2024-09-13T14:37:43.737 INFO:tasks.rados.rados.0.smithi151.stdout:3980: done (1 left) 2024-09-13T14:37:43.737 INFO:tasks.rados.rados.0.smithi151.stdout:3981: done (0 left) 2024-09-13T14:37:43.737 INFO:tasks.rados.rados.0.smithi151.stdout:3982: write oid 29 current snap is 505 2024-09-13T14:37:43.738 INFO:tasks.rados.rados.0.smithi151.stdout:3982: seq_num 1507 ranges {528149=438137,1436011=492529,2601156=667041} 2024-09-13T14:37:43.742 INFO:tasks.rados.rados.0.smithi151.stdout:3982: writing smithi15186934-29 from 528149 to 966286 tid 1 2024-09-13T14:37:43.748 INFO:tasks.rados.rados.0.smithi151.stdout:3982: writing smithi15186934-29 from 1436011 to 1928540 tid 2 2024-09-13T14:37:43.755 INFO:tasks.rados.rados.0.smithi151.stdout:3982: writing smithi15186934-29 from 2601156 to 3268197 tid 3 2024-09-13T14:37:43.755 INFO:tasks.rados.rados.0.smithi151.stdout:3983: snap_create 2024-09-13T14:37:43.755 INFO:tasks.rados.rados.0.smithi151.stdout:3982: finishing write tid 1 to smithi15186934-29 2024-09-13T14:37:43.756 INFO:tasks.rados.rados.0.smithi151.stdout:3982: finishing write tid 2 to smithi15186934-29 2024-09-13T14:37:43.759 INFO:tasks.rados.rados.0.smithi151.stdout:3982: finishing write tid 3 to smithi15186934-29 2024-09-13T14:37:43.759 INFO:tasks.rados.rados.0.smithi151.stdout:3982: finishing write tid 4 to smithi15186934-29 2024-09-13T14:37:43.759 INFO:tasks.rados.rados.0.smithi151.stdout:3982: finishing write tid 5 to smithi15186934-29 2024-09-13T14:37:43.759 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 29 v 1102 (ObjNum 1507 snap 505 seq_num 1507) dirty exists 2024-09-13T14:37:43.759 INFO:tasks.rados.rados.0.smithi151.stdout:3982: left oid 29 (ObjNum 1507 snap 505 seq_num 1507) 2024-09-13T14:37:44.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:43 smithi184 ceph-mon[92527]: osdmap e1072: 8 total, 8 up, 8 in 2024-09-13T14:37:44.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:43 smithi151 ceph-mon[104450]: osdmap e1072: 8 total, 8 up, 8 in 2024-09-13T14:37:44.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:43 smithi151 ceph-mon[108966]: osdmap e1072: 8 total, 8 up, 8 in 2024-09-13T14:37:44.684 INFO:tasks.rados.rados.0.smithi151.stdout:3982: done (1 left) 2024-09-13T14:37:44.684 INFO:tasks.rados.rados.0.smithi151.stdout:3983: done (0 left) 2024-09-13T14:37:44.684 INFO:tasks.rados.rados.0.smithi151.stdout:3984: snap_create 2024-09-13T14:37:44.736 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:37:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:37:44] "GET /metrics HTTP/1.1" 200 36510 "" "Prometheus/2.51.0" 2024-09-13T14:37:44.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:44 smithi151 ceph-mon[104450]: pgmap v1227: 89 pgs: 89 active+clean; 224 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 1.7 MiB/s wr, 18 op/s 2024-09-13T14:37:44.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:44 smithi151 ceph-mon[104450]: osdmap e1073: 8 total, 8 up, 8 in 2024-09-13T14:37:44.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:44 smithi151 ceph-mon[104450]: osdmap e1074: 8 total, 8 up, 8 in 2024-09-13T14:37:44.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:44 smithi151 ceph-mon[108966]: pgmap v1227: 89 pgs: 89 active+clean; 224 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 1.7 MiB/s wr, 18 op/s 2024-09-13T14:37:44.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:44 smithi151 ceph-mon[108966]: osdmap e1073: 8 total, 8 up, 8 in 2024-09-13T14:37:44.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:44 smithi151 ceph-mon[108966]: osdmap e1074: 8 total, 8 up, 8 in 2024-09-13T14:37:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:44 smithi184 ceph-mon[92527]: pgmap v1227: 89 pgs: 89 active+clean; 224 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 1.7 MiB/s wr, 18 op/s 2024-09-13T14:37:45.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:44 smithi184 ceph-mon[92527]: osdmap e1073: 8 total, 8 up, 8 in 2024-09-13T14:37:45.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:44 smithi184 ceph-mon[92527]: osdmap e1074: 8 total, 8 up, 8 in 2024-09-13T14:37:45.682 INFO:tasks.rados.rados.0.smithi151.stdout:3984: done (0 left) 2024-09-13T14:37:45.683 INFO:tasks.rados.rados.0.smithi151.stdout:3985: read oid 8 snap -1 2024-09-13T14:37:45.683 INFO:tasks.rados.rados.0.smithi151.stdout:3985: expect deleted 2024-09-13T14:37:45.683 INFO:tasks.rados.rados.0.smithi151.stdout:3986: read oid 38 snap -1 2024-09-13T14:37:45.683 INFO:tasks.rados.rados.0.smithi151.stdout:3986: expect deleted 2024-09-13T14:37:45.683 INFO:tasks.rados.rados.0.smithi151.stdout:3987: rollback oid 26 current snap is 507 2024-09-13T14:37:45.683 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 26 to 504 2024-09-13T14:37:45.684 INFO:tasks.rados.rados.0.smithi151.stdout:3988: snap_create 2024-09-13T14:37:45.687 INFO:tasks.rados.rados.0.smithi151.stdout:3987: finishing rollback tid 0 to smithi15186934-26 2024-09-13T14:37:45.688 INFO:tasks.rados.rados.0.smithi151.stdout:3987: finishing rollback tid 1 to smithi15186934-26 2024-09-13T14:37:45.688 INFO:tasks.rados.rados.0.smithi151.stdout:3987: finishing rollback tid 2 to smithi15186934-26 2024-09-13T14:37:45.688 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 1394 (ObjNum 1478 snap 496 seq_num 1478) dirty exists 2024-09-13T14:37:46.684 INFO:tasks.rados.rados.0.smithi151.stdout:3985: done (3 left) 2024-09-13T14:37:46.684 INFO:tasks.rados.rados.0.smithi151.stdout:3986: done (2 left) 2024-09-13T14:37:46.684 INFO:tasks.rados.rados.0.smithi151.stdout:3987: done (1 left) 2024-09-13T14:37:46.684 INFO:tasks.rados.rados.0.smithi151.stdout:3988: done (0 left) 2024-09-13T14:37:46.684 INFO:tasks.rados.rados.0.smithi151.stdout:3989: read oid 35 snap -1 2024-09-13T14:37:46.684 INFO:tasks.rados.rados.0.smithi151.stdout:3989: expect (ObjNum 1431 snap 480 seq_num 1431) 2024-09-13T14:37:46.684 INFO:tasks.rados.rados.0.smithi151.stdout:3990: read oid 28 snap -1 2024-09-13T14:37:46.684 INFO:tasks.rados.rados.0.smithi151.stdout:3990: expect (ObjNum 1439 snap 483 seq_num 1439) 2024-09-13T14:37:46.684 INFO:tasks.rados.rados.0.smithi151.stdout:3991: read oid 8 snap 491 2024-09-13T14:37:46.685 INFO:tasks.rados.rados.0.smithi151.stdout:3991: expect deleted 2024-09-13T14:37:46.685 INFO:tasks.rados.rados.0.smithi151.stdout:3992: write oid 36 current snap is 508 2024-09-13T14:37:46.685 INFO:tasks.rados.rados.0.smithi151.stdout:3992: seq_num 1508 ranges {544956=513386,1494493=636187,2628579=594666} 2024-09-13T14:37:46.690 INFO:tasks.rados.rados.0.smithi151.stdout:3992: writing smithi15186934-36 from 544956 to 1058342 tid 1 2024-09-13T14:37:46.696 INFO:tasks.rados.rados.0.smithi151.stdout:3992: writing smithi15186934-36 from 1494493 to 2130680 tid 2 2024-09-13T14:37:46.702 INFO:tasks.rados.rados.0.smithi151.stdout:3992: writing smithi15186934-36 from 2628579 to 3223245 tid 3 2024-09-13T14:37:46.703 INFO:tasks.rados.rados.0.smithi151.stdout:3993: write oid 21 current snap is 508 2024-09-13T14:37:46.703 INFO:tasks.rados.rados.0.smithi151.stdout:3993: seq_num 1509 ranges {561763=588635,1552975=496198,2656002=405938} 2024-09-13T14:37:46.709 INFO:tasks.rados.rados.0.smithi151.stdout:3993: writing smithi15186934-21 from 561763 to 1150398 tid 1 2024-09-13T14:37:46.713 INFO:tasks.rados.rados.0.smithi151.stdout:3993: writing smithi15186934-21 from 1552975 to 2049173 tid 2 2024-09-13T14:37:46.719 INFO:tasks.rados.rados.0.smithi151.stdout:3993: writing smithi15186934-21 from 2656002 to 3061940 tid 3 2024-09-13T14:37:46.719 INFO:tasks.rados.rados.0.smithi151.stdout:3994: rollback oid 41 current snap is 508 2024-09-13T14:37:46.719 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 41 to 502 2024-09-13T14:37:46.719 INFO:tasks.rados.rados.0.smithi151.stdout:3990: expect (ObjNum 1439 snap 483 seq_num 1439) 2024-09-13T14:37:46.821 INFO:tasks.rados.rados.0.smithi151.stdout:3989: expect (ObjNum 1431 snap 480 seq_num 1431) 2024-09-13T14:37:46.884 INFO:tasks.rados.rados.0.smithi151.stdout:3992: finishing write tid 1 to smithi15186934-36 2024-09-13T14:37:46.884 INFO:tasks.rados.rados.0.smithi151.stdout:3992: finishing write tid 2 to smithi15186934-36 2024-09-13T14:37:46.885 INFO:tasks.rados.rados.0.smithi151.stdout:3992: finishing write tid 3 to smithi15186934-36 2024-09-13T14:37:46.885 INFO:tasks.rados.rados.0.smithi151.stdout:3992: finishing write tid 4 to smithi15186934-36 2024-09-13T14:37:46.885 INFO:tasks.rados.rados.0.smithi151.stdout:3992: finishing write tid 5 to smithi15186934-36 2024-09-13T14:37:46.885 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 36 v 865 (ObjNum 1508 snap 508 seq_num 1508) dirty exists 2024-09-13T14:37:46.885 INFO:tasks.rados.rados.0.smithi151.stdout:3992: left oid 36 (ObjNum 1508 snap 508 seq_num 1508) 2024-09-13T14:37:46.885 INFO:tasks.rados.rados.0.smithi151.stdout:3993: finishing write tid 1 to smithi15186934-21 2024-09-13T14:37:46.885 INFO:tasks.rados.rados.0.smithi151.stdout:3993: finishing write tid 2 to smithi15186934-21 2024-09-13T14:37:46.885 INFO:tasks.rados.rados.0.smithi151.stdout:3989: done (5 left) 2024-09-13T14:37:46.885 INFO:tasks.rados.rados.0.smithi151.stdout:3990: done (4 left) 2024-09-13T14:37:46.885 INFO:tasks.rados.rados.0.smithi151.stdout:3991: done (3 left) 2024-09-13T14:37:46.885 INFO:tasks.rados.rados.0.smithi151.stdout:3992: done (2 left) 2024-09-13T14:37:46.885 INFO:tasks.rados.rados.0.smithi151.stdout:3995: write oid 47 current snap is 508 2024-09-13T14:37:46.885 INFO:tasks.rados.rados.0.smithi151.stdout:3993: finishing write tid 3 to smithi15186934-21 2024-09-13T14:37:46.886 INFO:tasks.rados.rados.0.smithi151.stdout:3993: finishing write tid 4 to smithi15186934-21 2024-09-13T14:37:46.886 INFO:tasks.rados.rados.0.smithi151.stdout:3995: seq_num 1510 ranges {578570=663884,1895104=639856,2967072=411498} 2024-09-13T14:37:46.890 INFO:tasks.rados.rados.0.smithi151.stdout:3995: writing smithi15186934-47 from 578570 to 1242454 tid 1 2024-09-13T14:37:46.897 INFO:tasks.rados.rados.0.smithi151.stdout:3995: writing smithi15186934-47 from 1895104 to 2534960 tid 2 2024-09-13T14:37:46.903 INFO:tasks.rados.rados.0.smithi151.stdout:3995: writing smithi15186934-47 from 2967072 to 3378570 tid 3 2024-09-13T14:37:46.903 INFO:tasks.rados.rados.0.smithi151.stdout:3996: rollback oid 26 current snap is 508 2024-09-13T14:37:46.903 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 26 to 491 2024-09-13T14:37:46.903 INFO:tasks.rados.rados.0.smithi151.stdout:3993: finishing write tid 5 to smithi15186934-21 2024-09-13T14:37:46.903 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 21 v 828 (ObjNum 1509 snap 508 seq_num 1509) dirty exists 2024-09-13T14:37:46.903 INFO:tasks.rados.rados.0.smithi151.stdout:3993: left oid 21 (ObjNum 1509 snap 508 seq_num 1509) 2024-09-13T14:37:46.903 INFO:tasks.rados.rados.0.smithi151.stdout:3994: finishing rollback tid 0 to smithi15186934-41 2024-09-13T14:37:46.903 INFO:tasks.rados.rados.0.smithi151.stdout:3993: done (3 left) 2024-09-13T14:37:46.903 INFO:tasks.rados.rados.0.smithi151.stdout:3997: rollback oid 1 current snap is 508 2024-09-13T14:37:46.903 INFO:tasks.rados.rados.0.smithi151.stdout:rollback oid 1 to 506 2024-09-13T14:37:46.904 INFO:tasks.rados.rados.0.smithi151.stdout:3994: finishing rollback tid 1 to smithi15186934-41 2024-09-13T14:37:46.904 INFO:tasks.rados.rados.0.smithi151.stdout:3994: finishing rollback tid 2 to smithi15186934-41 2024-09-13T14:37:46.904 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 41 v 362 (ObjNum 1457 snap 489 seq_num 1457) dirty exists 2024-09-13T14:37:46.904 INFO:tasks.rados.rados.0.smithi151.stdout:3995: finishing write tid 1 to smithi15186934-47 2024-09-13T14:37:46.904 INFO:tasks.rados.rados.0.smithi151.stdout:3995: finishing write tid 2 to smithi15186934-47 2024-09-13T14:37:46.904 INFO:tasks.rados.rados.0.smithi151.stdout:3994: done (3 left) 2024-09-13T14:37:46.904 INFO:tasks.rados.rados.0.smithi151.stdout:3998: delete oid 24 current snap is 508 2024-09-13T14:37:46.904 INFO:tasks.rados.rados.0.smithi151.stdout:3997: finishing rollback tid 1 to smithi15186934-1 2024-09-13T14:37:46.904 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 1 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-13T14:37:46.906 INFO:tasks.rados.rados.0.smithi151.stdout:3997: done (3 left) 2024-09-13T14:37:46.906 INFO:tasks.rados.rados.0.smithi151.stdout:3998: done (2 left) 2024-09-13T14:37:46.906 INFO:tasks.rados.rados.0.smithi151.stdout:3999: delete oid 37 current snap is 508 2024-09-13T14:37:46.906 INFO:tasks.rados.rados.0.smithi151.stdout:3996: finishing rollback tid 0 to smithi15186934-26 2024-09-13T14:37:46.907 INFO:tasks.rados.rados.0.smithi151.stdout:3996: finishing rollback tid 1 to smithi15186934-26 2024-09-13T14:37:46.907 INFO:tasks.rados.rados.0.smithi151.stdout:3996: finishing rollback tid 2 to smithi15186934-26 2024-09-13T14:37:46.907 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 26 v 1398 (ObjNum 1424 snap 478 seq_num 1424) dirty exists 2024-09-13T14:37:46.907 INFO:tasks.rados.rados.0.smithi151.stdout:3995: finishing write tid 3 to smithi15186934-47 2024-09-13T14:37:46.907 INFO:tasks.rados.rados.0.smithi151.stdout:3995: finishing write tid 4 to smithi15186934-47 2024-09-13T14:37:46.907 INFO:tasks.rados.rados.0.smithi151.stdout:3995: finishing write tid 5 to smithi15186934-47 2024-09-13T14:37:46.907 INFO:tasks.rados.rados.0.smithi151.stdout:update_object_version oid 47 v 1820 (ObjNum 1510 snap 508 seq_num 1510) dirty exists 2024-09-13T14:37:46.908 INFO:tasks.rados.rados.0.smithi151.stdout:3995: left oid 47 (ObjNum 1510 snap 508 seq_num 1510) 2024-09-13T14:37:46.908 INFO:tasks.rados.rados.0.smithi151.stdout:3995: done (2 left) 2024-09-13T14:37:46.908 INFO:tasks.rados.rados.0.smithi151.stdout:3996: done (1 left) 2024-09-13T14:37:46.908 INFO:tasks.rados.rados.0.smithi151.stdout:3999: done (0 left) 2024-09-13T14:37:46.910 INFO:tasks.rados.rados.0.smithi151.stderr:0 errors. 2024-09-13T14:37:46.911 INFO:tasks.rados.rados.0.smithi151.stderr: 2024-09-13T14:37:46.913 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T14:37:46.977 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:46 smithi151 ceph-mon[104450]: pgmap v1230: 89 pgs: 89 active+clean; 228 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 373 KiB/s wr, 11 op/s 2024-09-13T14:37:46.977 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:46 smithi151 ceph-mon[104450]: osdmap e1075: 8 total, 8 up, 8 in 2024-09-13T14:37:46.977 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:46.977 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:46 smithi151 ceph-mon[108966]: pgmap v1230: 89 pgs: 89 active+clean; 228 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 373 KiB/s wr, 11 op/s 2024-09-13T14:37:46.977 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:46 smithi151 ceph-mon[108966]: osdmap e1075: 8 total, 8 up, 8 in 2024-09-13T14:37:46.977 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:47.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:46 smithi184 ceph-mon[92527]: pgmap v1230: 89 pgs: 89 active+clean; 228 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 373 KiB/s wr, 11 op/s 2024-09-13T14:37:47.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:46 smithi184 ceph-mon[92527]: osdmap e1075: 8 total, 8 up, 8 in 2024-09-13T14:37:47.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:47.201 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:37:47.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:47 smithi151 ceph-mon[104450]: osdmap e1076: 8 total, 8 up, 8 in 2024-09-13T14:37:47.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:47 smithi151 ceph-mon[108966]: osdmap e1076: 8 total, 8 up, 8 in 2024-09-13T14:37:48.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:47 smithi184 ceph-mon[92527]: osdmap e1076: 8 total, 8 up, 8 in 2024-09-13T14:37:48.795 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:48 smithi151 ceph-mon[104450]: pgmap v1233: 89 pgs: 89 active+clean; 221 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2024-09-13T14:37:48.795 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:48 smithi151 ceph-mon[108966]: pgmap v1233: 89 pgs: 89 active+clean; 221 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2024-09-13T14:37:49.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:48 smithi184 ceph-mon[92527]: pgmap v1233: 89 pgs: 89 active+clean; 221 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2024-09-13T14:37:49.180 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:37:49.181 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":1076,"fsid":"2401fc50-71da-11ef-bceb-c7b262605968","created":"2024-09-13T14:12:36.168205+0000","modified":"2024-09-13T14:37:46.677453+0000","last_up_change":"2024-09-13T14:30:14.547371+0000","last_in_change":"2024-09-13T14:17:24.771724+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:15:41.780748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:18:09.030957+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":56,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T14:19:24.791384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T14:19:25.101452+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_0","create_time":"2024-09-13T14:20:40.871483+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"1076","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":593,"snap_epoch":1076,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0ad945ee-af7d-4f26-a1e9-18ed963194ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":511,"up_from":524,"up_thru":624,"down_at":512,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2137494296},{"type":"v1","addr":"172.21.15.151:6803","nonce":2137494296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2137494296},{"type":"v1","addr":"172.21.15.151:6805","nonce":2137494296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":2137494296},{"type":"v1","addr":"172.21.15.151:6809","nonce":2137494296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2137494296},{"type":"v1","addr":"172.21.15.151:6807","nonce":2137494296}]},"public_addr":"172.21.15.151:6803/2137494296","cluster_addr":"172.21.15.151:6805/2137494296","heartbeat_back_addr":"172.21.15.151:6809/2137494296","heartbeat_front_addr":"172.21.15.151:6807/2137494296","state":["exists","up"]},{"osd":1,"uuid":"09f5d705-684f-4e86-80a2-ec2292af09d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":544,"up_from":557,"up_thru":624,"down_at":545,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":3884881052},{"type":"v1","addr":"172.21.15.151:6811","nonce":3884881052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3884881052},{"type":"v1","addr":"172.21.15.151:6813","nonce":3884881052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3884881052},{"type":"v1","addr":"172.21.15.151:6817","nonce":3884881052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":3884881052},{"type":"v1","addr":"172.21.15.151:6815","nonce":3884881052}]},"public_addr":"172.21.15.151:6811/3884881052","cluster_addr":"172.21.15.151:6813/3884881052","heartbeat_back_addr":"172.21.15.151:6817/3884881052","heartbeat_front_addr":"172.21.15.151:6815/3884881052","state":["exists","up"]},{"osd":2,"uuid":"6a63f8af-ff69-46bb-830f-03ea89fc78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":578,"up_from":591,"up_thru":624,"down_at":579,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":322435630},{"type":"v1","addr":"172.21.15.151:6819","nonce":322435630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":322435630},{"type":"v1","addr":"172.21.15.151:6821","nonce":322435630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":322435630},{"type":"v1","addr":"172.21.15.151:6825","nonce":322435630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":322435630},{"type":"v1","addr":"172.21.15.151:6823","nonce":322435630}]},"public_addr":"172.21.15.151:6819/322435630","cluster_addr":"172.21.15.151:6821/322435630","heartbeat_back_addr":"172.21.15.151:6825/322435630","heartbeat_front_addr":"172.21.15.151:6823/322435630","state":["exists","up"]},{"osd":3,"uuid":"770e8334-3b3a-46bc-9782-964b2a696660","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":611,"up_from":624,"up_thru":624,"down_at":612,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2997668861},{"type":"v1","addr":"172.21.15.151:6827","nonce":2997668861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":2997668861},{"type":"v1","addr":"172.21.15.151:6829","nonce":2997668861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":2997668861},{"type":"v1","addr":"172.21.15.151:6833","nonce":2997668861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":2997668861},{"type":"v1","addr":"172.21.15.151:6831","nonce":2997668861}]},"public_addr":"172.21.15.151:6827/2997668861","cluster_addr":"172.21.15.151:6829/2997668861","heartbeat_back_addr":"172.21.15.151:6833/2997668861","heartbeat_front_addr":"172.21.15.151:6831/2997668861","state":["exists","up"]},{"osd":4,"uuid":"75e258c0-1145-4aee-86aa-db2b63082cc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":624,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6801","nonce":3275553935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6803","nonce":3275553935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6807","nonce":3275553935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6805","nonce":3275553935}]},"public_addr":"172.21.15.184:6801/3275553935","cluster_addr":"172.21.15.184:6803/3275553935","heartbeat_back_addr":"172.21.15.184:6807/3275553935","heartbeat_front_addr":"172.21.15.184:6805/3275553935","state":["exists","up"]},{"osd":5,"uuid":"f56a5366-9d22-4960-933d-340aa280fbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":624,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6809","nonce":2446419621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6811","nonce":2446419621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6815","nonce":2446419621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6813","nonce":2446419621}]},"public_addr":"172.21.15.184:6809/2446419621","cluster_addr":"172.21.15.184:6811/2446419621","heartbeat_back_addr":"172.21.15.184:6815/2446419621","heartbeat_front_addr":"172.21.15.184:6813/2446419621","state":["exists","up"]},{"osd":6,"uuid":"e9826568-a728-45ab-8180-9777fe31f93a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":612,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6817","nonce":2214837203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6819","nonce":2214837203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6823","nonce":2214837203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6821","nonce":2214837203}]},"public_addr":"172.21.15.184:6817/2214837203","cluster_addr":"172.21.15.184:6819/2214837203","heartbeat_back_addr":"172.21.15.184:6823/2214837203","heartbeat_front_addr":"172.21.15.184:6821/2214837203","state":["exists","up"]},{"osd":7,"uuid":"d76869fe-5121-4467-8b56-1aadef0eaf65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":624,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6825","nonce":3378735997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6827","nonce":3378735997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6831","nonce":3378735997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6829","nonce":3378735997}]},"public_addr":"172.21.15.184:6825/3378735997","cluster_addr":"172.21.15.184:6827/3378735997","heartbeat_back_addr":"172.21.15.184:6831/3378735997","heartbeat_front_addr":"172.21.15.184:6829/3378735997","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:14:49.835715+0000","dead_epoch":511},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:15:14.682882+0000","dead_epoch":544},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:15:39.423890+0000","dead_epoch":578},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:16:04.527460+0000","dead_epoch":611},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:16:27.283661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:16:51.893868+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:16.152592+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:39.763533+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.151:0/514781427":"2024-09-14T14:22:15.389006+0000","172.21.15.151:0/1783031268":"2024-09-14T14:24:01.073069+0000","172.21.15.151:0/1351186656":"2024-09-14T14:18:18.344788+0000","172.21.15.151:0/1775071943":"2024-09-14T14:24:01.073069+0000","172.21.15.151:0/1409232061":"2024-09-14T14:24:01.073069+0000","172.21.15.151:6800/4025698662":"2024-09-14T14:22:15.389006+0000","172.21.15.151:6801/42840":"2024-09-14T14:24:01.073069+0000","172.21.15.151:0/624042464":"2024-09-14T14:13:04.761383+0000","172.21.15.151:0/2201280183":"2024-09-14T14:22:15.389006+0000","172.21.15.151:0/1427922964":"2024-09-14T14:13:27.478847+0000","172.21.15.184:0/1999058007":"2024-09-14T14:23:20.772174+0000","172.21.15.184:0/3047962606":"2024-09-14T14:23:20.772174+0000","172.21.15.151:6801/3383065173":"2024-09-14T14:13:04.761383+0000","172.21.15.184:6834/205203765":"2024-09-14T14:23:20.772174+0000","172.21.15.151:0/841256485":"2024-09-14T14:24:01.073069+0000","172.21.15.151:0/4115173791":"2024-09-14T14:18:18.344788+0000","172.21.15.151:6800/3383065173":"2024-09-14T14:13:04.761383+0000","172.21.15.151:0/663506338":"2024-09-14T14:24:01.073069+0000","172.21.15.151:0/3689756337":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/2889050140":"2024-09-14T14:18:18.344788+0000","172.21.15.151:6801/4025698662":"2024-09-14T14:22:15.389006+0000","172.21.15.151:0/2800235450":"2024-09-14T14:18:18.344788+0000","172.21.15.184:0/3370242044":"2024-09-14T14:23:20.772174+0000","172.21.15.151:6800/2335297718":"2024-09-14T14:18:18.344788+0000","172.21.15.151:0/1517910843":"2024-09-14T14:24:01.073069+0000","172.21.15.151:0/1182681341":"2024-09-14T14:13:04.761383+0000","172.21.15.151:6800/1057983082":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/1530349337":"2024-09-14T14:22:15.389006+0000","172.21.15.151:6801/1057983082":"2024-09-14T14:13:27.478847+0000","172.21.15.184:0/3929228938":"2024-09-14T14:23:20.772174+0000","172.21.15.151:0/2666829207":"2024-09-14T14:13:27.478847+0000","172.21.15.151:6801/2335297718":"2024-09-14T14:18:18.344788+0000","172.21.15.184:6835/205203765":"2024-09-14T14:23:20.772174+0000","172.21.15.184:0/3003411098":"2024-09-14T14:23:20.772174+0000","172.21.15.151:6800/42840":"2024-09-14T14:24:01.073069+0000","172.21.15.151:0/1488764319":"2024-09-14T14:22:15.389006+0000","172.21.15.184:0/2995983270":"2024-09-14T14:23:20.772174+0000","172.21.15.151:0/3946777511":"2024-09-14T14:13:04.761383+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":568,"length":1},{"begin":581,"length":1},{"begin":586,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T14:37:49.773 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:49 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/820906630' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T14:37:49.773 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:49 smithi151 ceph-mon[104450]: osdmap e1077: 8 total, 8 up, 8 in 2024-09-13T14:37:49.774 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:49 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/820906630' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T14:37:49.774 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:49 smithi151 ceph-mon[108966]: osdmap e1077: 8 total, 8 up, 8 in 2024-09-13T14:37:49.797 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:37:50.082 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:37:50.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:49 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/820906630' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T14:37:50.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:49 smithi184 ceph-mon[92527]: osdmap e1077: 8 total, 8 up, 8 in 2024-09-13T14:37:50.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:37:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:37:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:37:50.879 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:50 smithi151 ceph-mon[104450]: pgmap v1234: 89 pgs: 89 active+clean; 223 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 4.5 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2024-09-13T14:37:50.880 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:50 smithi151 ceph-mon[108966]: pgmap v1234: 89 pgs: 89 active+clean; 223 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 4.5 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2024-09-13T14:37:51.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:50 smithi184 ceph-mon[92527]: pgmap v1234: 89 pgs: 89 active+clean; 223 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 4.5 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2024-09-13T14:37:51.995 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:37:51.997 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T14:37:52.636 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1236,"stamp":"2024-09-13T14:37:51.281511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":235991040,"num_objects":132,"num_object_clones":61,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132,"num_whiteouts":8,"num_read":7059,"num_read_kb":7291983,"num_write":5817,"num_write_kb":1351545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":42679117,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22162,"ondisk_log_size":22162,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1829404,"kb_used_data":468956,"kb_used_omap":149,"kb_used_meta":1360170,"kb_avail":748131812,"statfs":{"total":767960285184,"available":766086975488,"internally_reserved":0,"allocated":480210944,"data_stored":535433972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":153401,"internal_metadata":1392814279},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3024979,"num_objects":1,"num_object_clones":2,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":12485,"num_write":33,"num_write_kb":8093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001614"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":1455,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.682403+0000","last_change":"2024-09-13T14:30:15.579251+0000","last_active":"2024-09-13T14:37:46.682403+0000","last_peered":"2024-09-13T14:37:46.682403+0000","last_clean":"2024-09-13T14:37:46.682403+0000","last_became_active":"2024-09-13T14:30:15.578890+0000","last_became_peered":"2024-09-13T14:30:15.578890+0000","last_unstale":"2024-09-13T14:37:46.682403+0000","last_undegraded":"2024-09-13T14:37:46.682403+0000","last_fullsized":"2024-09-13T14:37:46.682403+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:51:28.380523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":1032,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.679340+0000","last_change":"2024-09-13T14:29:42.541093+0000","last_active":"2024-09-13T14:37:45.679340+0000","last_peered":"2024-09-13T14:37:45.679340+0000","last_clean":"2024-09-13T14:37:45.679340+0000","last_became_active":"2024-09-13T14:29:42.540671+0000","last_became_peered":"2024-09-13T14:29:42.540671+0000","last_unstale":"2024-09-13T14:37:45.679340+0000","last_undegraded":"2024-09-13T14:37:45.679340+0000","last_fullsized":"2024-09-13T14:37:45.679340+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:45.722376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1018,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.683956+0000","last_change":"2024-09-13T14:30:15.578414+0000","last_active":"2024-09-13T14:37:45.683956+0000","last_peered":"2024-09-13T14:37:45.683956+0000","last_clean":"2024-09-13T14:37:45.683956+0000","last_became_active":"2024-09-13T14:30:15.577997+0000","last_became_peered":"2024-09-13T14:30:15.577997+0000","last_unstale":"2024-09-13T14:37:45.683956+0000","last_undegraded":"2024-09-13T14:37:45.683956+0000","last_fullsized":"2024-09-13T14:37:45.683956+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:09.290677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1549,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.685842+0000","last_change":"2024-09-13T14:30:15.585508+0000","last_active":"2024-09-13T14:37:46.685842+0000","last_peered":"2024-09-13T14:37:46.685842+0000","last_clean":"2024-09-13T14:37:46.685842+0000","last_became_active":"2024-09-13T14:30:15.585206+0000","last_became_peered":"2024-09-13T14:30:15.585206+0000","last_unstale":"2024-09-13T14:37:46.685842+0000","last_undegraded":"2024-09-13T14:37:46.685842+0000","last_fullsized":"2024-09-13T14:37:46.685842+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:28:39.584397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1516,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.686161+0000","last_change":"2024-09-13T14:30:15.581992+0000","last_active":"2024-09-13T14:37:46.686161+0000","last_peered":"2024-09-13T14:37:46.686161+0000","last_clean":"2024-09-13T14:37:46.686161+0000","last_became_active":"2024-09-13T14:30:15.581692+0000","last_became_peered":"2024-09-13T14:30:15.581692+0000","last_unstale":"2024-09-13T14:37:46.686161+0000","last_undegraded":"2024-09-13T14:37:46.686161+0000","last_fullsized":"2024-09-13T14:37:46.686161+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:59.930184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1013,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.679596+0000","last_change":"2024-09-13T14:19:25.873088+0000","last_active":"2024-09-13T14:37:45.679596+0000","last_peered":"2024-09-13T14:37:45.679596+0000","last_clean":"2024-09-13T14:37:45.679596+0000","last_became_active":"2024-09-13T14:19:25.872781+0000","last_became_peered":"2024-09-13T14:19:25.872781+0000","last_unstale":"2024-09-13T14:37:45.679596+0000","last_undegraded":"2024-09-13T14:37:45.679596+0000","last_fullsized":"2024-09-13T14:37:45.679596+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:05.607382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1455,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.683164+0000","last_change":"2024-09-13T14:30:15.579629+0000","last_active":"2024-09-13T14:37:46.683164+0000","last_peered":"2024-09-13T14:37:46.683164+0000","last_clean":"2024-09-13T14:37:46.683164+0000","last_became_active":"2024-09-13T14:30:15.577913+0000","last_became_peered":"2024-09-13T14:30:15.577913+0000","last_unstale":"2024-09-13T14:37:46.683164+0000","last_undegraded":"2024-09-13T14:37:46.683164+0000","last_fullsized":"2024-09-13T14:37:46.683164+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:45:28.463082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1021,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.416820+0000","last_change":"2024-09-13T14:28:35.603724+0000","last_active":"2024-09-13T14:37:46.416820+0000","last_peered":"2024-09-13T14:37:46.416820+0000","last_clean":"2024-09-13T14:37:46.416820+0000","last_became_active":"2024-09-13T14:28:35.603316+0000","last_became_peered":"2024-09-13T14:28:35.603316+0000","last_unstale":"2024-09-13T14:37:46.416820+0000","last_undegraded":"2024-09-13T14:37:46.416820+0000","last_fullsized":"2024-09-13T14:37:46.416820+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:47.668223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"1070'1254","reported_seq":4370,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.679109+0000","last_change":"2024-09-13T14:37:42.726299+0000","last_active":"2024-09-13T14:37:45.679109+0000","last_peered":"2024-09-13T14:37:45.679109+0000","last_clean":"2024-09-13T14:37:45.679109+0000","last_became_active":"2024-09-13T14:20:41.940713+0000","last_became_peered":"2024-09-13T14:20:41.940713+0000","last_unstale":"2024-09-13T14:37:45.679109+0000","last_undegraded":"2024-09-13T14:37:45.679109+0000","last_fullsized":"2024-09-13T14:37:45.679109+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_clean_scrub_stamp":"2024-09-13T14:20:40.921654+0000","objects_scrubbed":0,"log_size":1254,"log_dups_size":0,"ondisk_log_size":1254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:10.078207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057324399999999999,"stat_sum":{"num_bytes":16716202,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":318,"num_read_kb":346171,"num_write":318,"num_write_kb":80376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"238","length":"1"},{"start":"245","length":"1"},{"start":"24a","length":"1"}]},{"pgid":"3.f","version":"63'1","reported_seq":1031,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.679694+0000","last_change":"2024-09-13T14:28:35.608607+0000","last_active":"2024-09-13T14:37:45.679694+0000","last_peered":"2024-09-13T14:37:45.679694+0000","last_clean":"2024-09-13T14:37:45.679694+0000","last_became_active":"2024-09-13T14:28:35.608254+0000","last_became_peered":"2024-09-13T14:28:35.608254+0000","last_unstale":"2024-09-13T14:37:45.679694+0000","last_undegraded":"2024-09-13T14:37:45.679694+0000","last_fullsized":"2024-09-13T14:37:45.679694+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:24.019173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1001,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.683615+0000","last_change":"2024-09-13T14:19:26.887839+0000","last_active":"2024-09-13T14:37:45.683615+0000","last_peered":"2024-09-13T14:37:45.683615+0000","last_clean":"2024-09-13T14:37:45.683615+0000","last_became_active":"2024-09-13T14:19:26.887498+0000","last_became_peered":"2024-09-13T14:19:26.887498+0000","last_unstale":"2024-09-13T14:37:45.683615+0000","last_undegraded":"2024-09-13T14:37:45.683615+0000","last_fullsized":"2024-09-13T14:37:45.683615+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:24.475263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"1072'1539","reported_seq":5568,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:50.053548+0000","last_change":"2024-09-13T14:37:43.329375+0000","last_active":"2024-09-13T14:37:50.053548+0000","last_peered":"2024-09-13T14:37:50.053548+0000","last_clean":"2024-09-13T14:37:50.053548+0000","last_became_active":"2024-09-13T14:29:42.543892+0000","last_became_peered":"2024-09-13T14:29:42.543892+0000","last_unstale":"2024-09-13T14:37:50.053548+0000","last_undegraded":"2024-09-13T14:37:50.053548+0000","last_fullsized":"2024-09-13T14:37:50.053548+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_clean_scrub_stamp":"2024-09-13T14:20:40.921654+0000","objects_scrubbed":0,"log_size":1539,"log_dups_size":0,"ondisk_log_size":1539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:47:30.779748+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0058537449999999996,"stat_sum":{"num_bytes":20740302,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":532,"num_read_kb":648731,"num_write":404,"num_write_kb":96192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":10188213,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":1032,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.679687+0000","last_change":"2024-09-13T14:29:08.909370+0000","last_active":"2024-09-13T14:37:45.679687+0000","last_peered":"2024-09-13T14:37:45.679687+0000","last_clean":"2024-09-13T14:37:45.679687+0000","last_became_active":"2024-09-13T14:29:08.908983+0000","last_became_peered":"2024-09-13T14:29:08.908983+0000","last_unstale":"2024-09-13T14:37:45.679687+0000","last_undegraded":"2024-09-13T14:37:45.679687+0000","last_fullsized":"2024-09-13T14:37:45.679687+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:44:43.939022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1033,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.686634+0000","last_change":"2024-09-13T14:30:15.580757+0000","last_active":"2024-09-13T14:37:45.686634+0000","last_peered":"2024-09-13T14:37:45.686634+0000","last_clean":"2024-09-13T14:37:45.686634+0000","last_became_active":"2024-09-13T14:30:15.580401+0000","last_became_peered":"2024-09-13T14:30:15.580401+0000","last_unstale":"2024-09-13T14:37:45.686634+0000","last_undegraded":"2024-09-13T14:37:45.686634+0000","last_fullsized":"2024-09-13T14:37:45.686634+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:22:29.402576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"1073'1102","reported_seq":5134,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:50.054378+0000","last_change":"2024-09-13T14:37:43.326917+0000","last_active":"2024-09-13T14:37:50.054378+0000","last_peered":"2024-09-13T14:37:50.054378+0000","last_clean":"2024-09-13T14:37:50.054378+0000","last_became_active":"2024-09-13T14:29:42.543258+0000","last_became_peered":"2024-09-13T14:29:42.543258+0000","last_unstale":"2024-09-13T14:37:50.054378+0000","last_undegraded":"2024-09-13T14:37:50.054378+0000","last_fullsized":"2024-09-13T14:37:50.054378+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_clean_scrub_stamp":"2024-09-13T14:20:40.921654+0000","objects_scrubbed":0,"log_size":1102,"log_dups_size":0,"ondisk_log_size":1102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:07:25.965235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003029473,"stat_sum":{"num_bytes":14776003,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":357,"num_read_kb":343897,"num_write":309,"num_write_kb":73157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1767536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":1014,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.679642+0000","last_change":"2024-09-13T14:19:25.871006+0000","last_active":"2024-09-13T14:37:45.679642+0000","last_peered":"2024-09-13T14:37:45.679642+0000","last_clean":"2024-09-13T14:37:45.679642+0000","last_became_active":"2024-09-13T14:19:25.870713+0000","last_became_peered":"2024-09-13T14:19:25.870713+0000","last_unstale":"2024-09-13T14:37:45.679642+0000","last_undegraded":"2024-09-13T14:37:45.679642+0000","last_fullsized":"2024-09-13T14:37:45.679642+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:34.372124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1012,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.416598+0000","last_change":"2024-09-13T14:29:08.914488+0000","last_active":"2024-09-13T14:37:46.416598+0000","last_peered":"2024-09-13T14:37:46.416598+0000","last_clean":"2024-09-13T14:37:46.416598+0000","last_became_active":"2024-09-13T14:29:08.913220+0000","last_became_peered":"2024-09-13T14:29:08.913220+0000","last_unstale":"2024-09-13T14:37:46.416598+0000","last_undegraded":"2024-09-13T14:37:46.416598+0000","last_fullsized":"2024-09-13T14:37:46.416598+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:08.316150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"1076'865","reported_seq":4893,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:50.053711+0000","last_change":"2024-09-13T14:37:43.324269+0000","last_active":"2024-09-13T14:37:50.053711+0000","last_peered":"2024-09-13T14:37:50.053711+0000","last_clean":"2024-09-13T14:37:50.053711+0000","last_became_active":"2024-09-13T14:30:15.577676+0000","last_became_peered":"2024-09-13T14:30:15.577676+0000","last_unstale":"2024-09-13T14:37:50.053711+0000","last_undegraded":"2024-09-13T14:37:50.053711+0000","last_fullsized":"2024-09-13T14:37:50.053711+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_clean_scrub_stamp":"2024-09-13T14:20:40.921654+0000","objects_scrubbed":0,"log_size":865,"log_dups_size":0,"ondisk_log_size":865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:09:35.050124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062069200000000005,"stat_sum":{"num_bytes":10960618,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":246,"num_read_kb":276230,"num_write":233,"num_write_kb":57185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1230356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1019,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.683910+0000","last_change":"2024-09-13T14:30:15.576953+0000","last_active":"2024-09-13T14:37:45.683910+0000","last_peered":"2024-09-13T14:37:45.683910+0000","last_clean":"2024-09-13T14:37:45.683910+0000","last_became_active":"2024-09-13T14:30:15.576614+0000","last_became_peered":"2024-09-13T14:30:15.576614+0000","last_unstale":"2024-09-13T14:37:45.683910+0000","last_undegraded":"2024-09-13T14:37:45.683910+0000","last_fullsized":"2024-09-13T14:37:45.683910+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:33:57.915263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1517,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.685706+0000","last_change":"2024-09-13T14:29:08.918531+0000","last_active":"2024-09-13T14:37:46.685706+0000","last_peered":"2024-09-13T14:37:46.685706+0000","last_clean":"2024-09-13T14:37:46.685706+0000","last_became_active":"2024-09-13T14:29:08.918264+0000","last_became_peered":"2024-09-13T14:29:08.918264+0000","last_unstale":"2024-09-13T14:37:46.685706+0000","last_undegraded":"2024-09-13T14:37:46.685706+0000","last_fullsized":"2024-09-13T14:37:46.685706+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:04.387551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"1072'1352","reported_seq":5329,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:50.054572+0000","last_change":"2024-09-13T14:37:43.329069+0000","last_active":"2024-09-13T14:37:50.054572+0000","last_peered":"2024-09-13T14:37:50.054572+0000","last_clean":"2024-09-13T14:37:50.054572+0000","last_became_active":"2024-09-13T14:29:42.537076+0000","last_became_peered":"2024-09-13T14:29:42.537076+0000","last_unstale":"2024-09-13T14:37:50.054572+0000","last_undegraded":"2024-09-13T14:37:50.054572+0000","last_fullsized":"2024-09-13T14:37:50.054572+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_clean_scrub_stamp":"2024-09-13T14:20:40.921654+0000","objects_scrubbed":0,"log_size":1352,"log_dups_size":0,"ondisk_log_size":1352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:05:34.288845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050671429999999996,"stat_sum":{"num_bytes":11771217,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":366,"num_read_kb":371283,"num_write":373,"num_write_kb":92127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2452127,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":1466,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.682294+0000","last_change":"2024-09-13T14:30:15.581475+0000","last_active":"2024-09-13T14:37:46.682294+0000","last_peered":"2024-09-13T14:37:46.682294+0000","last_clean":"2024-09-13T14:37:46.682294+0000","last_became_active":"2024-09-13T14:30:15.581148+0000","last_became_peered":"2024-09-13T14:30:15.581148+0000","last_unstale":"2024-09-13T14:37:46.682294+0000","last_undegraded":"2024-09-13T14:37:46.682294+0000","last_fullsized":"2024-09-13T14:37:46.682294+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:34.741839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1016,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.686573+0000","last_change":"2024-09-13T14:30:15.580092+0000","last_active":"2024-09-13T14:37:45.686573+0000","last_peered":"2024-09-13T14:37:45.686573+0000","last_clean":"2024-09-13T14:37:45.686573+0000","last_became_active":"2024-09-13T14:30:15.579540+0000","last_became_peered":"2024-09-13T14:30:15.579540+0000","last_unstale":"2024-09-13T14:37:45.686573+0000","last_undegraded":"2024-09-13T14:37:45.686573+0000","last_fullsized":"2024-09-13T14:37:45.686573+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:54.416259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"1072'885","reported_seq":4977,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.905200+0000","last_change":"2024-09-13T14:37:42.973190+0000","last_active":"2024-09-13T14:37:46.905200+0000","last_peered":"2024-09-13T14:37:46.905200+0000","last_clean":"2024-09-13T14:37:46.905200+0000","last_became_active":"2024-09-13T14:29:08.915600+0000","last_became_peered":"2024-09-13T14:29:08.915600+0000","last_unstale":"2024-09-13T14:37:46.905200+0000","last_undegraded":"2024-09-13T14:37:46.905200+0000","last_fullsized":"2024-09-13T14:37:46.905200+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_clean_scrub_stamp":"2024-09-13T14:20:40.921654+0000","objects_scrubbed":0,"log_size":885,"log_dups_size":0,"ondisk_log_size":885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:13.969878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020722560000000002,"stat_sum":{"num_bytes":8313091,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":303,"num_read_kb":370354,"num_write":207,"num_write_kb":50369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3470690,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"238","length":"1"},{"start":"245","length":"1"},{"start":"24a","length":"1"}]},{"pgid":"3.a","version":"0'0","reported_seq":1013,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.416748+0000","last_change":"2024-09-13T14:29:08.915311+0000","last_active":"2024-09-13T14:37:46.416748+0000","last_peered":"2024-09-13T14:37:46.416748+0000","last_clean":"2024-09-13T14:37:46.416748+0000","last_became_active":"2024-09-13T14:29:08.914923+0000","last_became_peered":"2024-09-13T14:29:08.914923+0000","last_unstale":"2024-09-13T14:37:46.416748+0000","last_undegraded":"2024-09-13T14:37:46.416748+0000","last_fullsized":"2024-09-13T14:37:46.416748+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:26.311233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1033,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.686894+0000","last_change":"2024-09-13T14:29:42.543672+0000","last_active":"2024-09-13T14:37:45.686894+0000","last_peered":"2024-09-13T14:37:45.686894+0000","last_clean":"2024-09-13T14:37:45.686894+0000","last_became_active":"2024-09-13T14:29:42.543329+0000","last_became_peered":"2024-09-13T14:29:42.543329+0000","last_unstale":"2024-09-13T14:37:45.686894+0000","last_undegraded":"2024-09-13T14:37:45.686894+0000","last_fullsized":"2024-09-13T14:37:45.686894+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:09:09.752737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"1076'828","reported_seq":3999,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.722851+0000","last_change":"2024-09-13T14:37:42.738474+0000","last_active":"2024-09-13T14:37:46.722851+0000","last_peered":"2024-09-13T14:37:46.722851+0000","last_clean":"2024-09-13T14:37:46.722851+0000","last_became_active":"2024-09-13T14:20:41.936154+0000","last_became_peered":"2024-09-13T14:20:41.936154+0000","last_unstale":"2024-09-13T14:37:46.722851+0000","last_undegraded":"2024-09-13T14:37:46.722851+0000","last_fullsized":"2024-09-13T14:37:46.722851+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_clean_scrub_stamp":"2024-09-13T14:20:40.921654+0000","objects_scrubbed":0,"log_size":828,"log_dups_size":0,"ondisk_log_size":828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:44.772210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001230851,"stat_sum":{"num_bytes":12622149,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":253,"num_read_kb":265380,"num_write":219,"num_write_kb":57220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"238","length":"1"},{"start":"245","length":"1"},{"start":"24a","length":"1"}]},{"pgid":"3.9","version":"0'0","reported_seq":1017,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.686742+0000","last_change":"2024-09-13T14:29:42.545815+0000","last_active":"2024-09-13T14:37:45.686742+0000","last_peered":"2024-09-13T14:37:45.686742+0000","last_clean":"2024-09-13T14:37:45.686742+0000","last_became_active":"2024-09-13T14:29:42.545508+0000","last_became_peered":"2024-09-13T14:29:42.545508+0000","last_unstale":"2024-09-13T14:37:45.686742+0000","last_undegraded":"2024-09-13T14:37:45.686742+0000","last_fullsized":"2024-09-13T14:37:45.686742+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:39.678381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1016,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.687068+0000","last_change":"2024-09-13T14:29:08.916416+0000","last_active":"2024-09-13T14:37:45.687068+0000","last_peered":"2024-09-13T14:37:45.687068+0000","last_clean":"2024-09-13T14:37:45.687068+0000","last_became_active":"2024-09-13T14:29:08.916106+0000","last_became_peered":"2024-09-13T14:29:08.916106+0000","last_unstale":"2024-09-13T14:37:45.687068+0000","last_undegraded":"2024-09-13T14:37:45.687068+0000","last_fullsized":"2024-09-13T14:37:45.687068+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:25.508423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"1076'362","reported_seq":3563,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.723316+0000","last_change":"2024-09-13T14:37:42.750255+0000","last_active":"2024-09-13T14:37:46.723316+0000","last_peered":"2024-09-13T14:37:46.723316+0000","last_clean":"2024-09-13T14:37:46.723316+0000","last_became_active":"2024-09-13T14:28:35.604614+0000","last_became_peered":"2024-09-13T14:28:35.604614+0000","last_unstale":"2024-09-13T14:37:46.723316+0000","last_undegraded":"2024-09-13T14:37:46.723316+0000","last_fullsized":"2024-09-13T14:37:46.723316+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_clean_scrub_stamp":"2024-09-13T14:20:40.921654+0000","objects_scrubbed":0,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:51.576235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032454100000000002,"stat_sum":{"num_bytes":5395740,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":124,"num_read_kb":119803,"num_write":101,"num_write_kb":23623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"238","length":"1"},{"start":"245","length":"1"},{"start":"24a","length":"1"}]},{"pgid":"3.8","version":"0'0","reported_seq":1456,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.682806+0000","last_change":"2024-09-13T14:30:15.577338+0000","last_active":"2024-09-13T14:37:46.682806+0000","last_peered":"2024-09-13T14:37:46.682806+0000","last_clean":"2024-09-13T14:37:46.682806+0000","last_became_active":"2024-09-13T14:30:15.577039+0000","last_became_peered":"2024-09-13T14:30:15.577039+0000","last_unstale":"2024-09-13T14:37:46.682806+0000","last_undegraded":"2024-09-13T14:37:46.682806+0000","last_fullsized":"2024-09-13T14:37:46.682806+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:43.246020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1494,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.905064+0000","last_change":"2024-09-13T14:30:15.580234+0000","last_active":"2024-09-13T14:37:46.905064+0000","last_peered":"2024-09-13T14:37:46.905064+0000","last_clean":"2024-09-13T14:37:46.905064+0000","last_became_active":"2024-09-13T14:30:15.579921+0000","last_became_peered":"2024-09-13T14:30:15.579921+0000","last_unstale":"2024-09-13T14:37:46.905064+0000","last_undegraded":"2024-09-13T14:37:46.905064+0000","last_fullsized":"2024-09-13T14:37:46.905064+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:42:39.428403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"1076'1822","reported_seq":4902,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.907343+0000","last_change":"2024-09-13T14:37:42.726394+0000","last_active":"2024-09-13T14:37:46.907343+0000","last_peered":"2024-09-13T14:37:46.907343+0000","last_clean":"2024-09-13T14:37:46.907343+0000","last_became_active":"2024-09-13T14:29:42.537577+0000","last_became_peered":"2024-09-13T14:29:42.537577+0000","last_unstale":"2024-09-13T14:37:46.907343+0000","last_undegraded":"2024-09-13T14:37:46.907343+0000","last_fullsized":"2024-09-13T14:37:46.907343+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_clean_scrub_stamp":"2024-09-13T14:20:40.921654+0000","objects_scrubbed":0,"log_size":1822,"log_dups_size":0,"ondisk_log_size":1822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:53:19.755944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052957999999999998,"stat_sum":{"num_bytes":17879506,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":533,"num_read_kb":583913,"num_write":454,"num_write_kb":109129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"238","length":"1"},{"start":"245","length":"1"},{"start":"24a","length":"1"}]},{"pgid":"1.0","version":"261'100","reported_seq":1146,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.679305+0000","last_change":"2024-09-13T14:28:35.604004+0000","last_active":"2024-09-13T14:37:45.679305+0000","last_peered":"2024-09-13T14:37:45.679305+0000","last_clean":"2024-09-13T14:37:45.679305+0000","last_became_active":"2024-09-13T14:28:35.603626+0000","last_became_peered":"2024-09-13T14:28:35.603626+0000","last_unstale":"2024-09-13T14:37:45.679305+0000","last_undegraded":"2024-09-13T14:37:45.679305+0000","last_fullsized":"2024-09-13T14:37:45.679305+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:15:42.069756+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:39:52.205001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":1032,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.683937+0000","last_change":"2024-09-13T14:29:42.539455+0000","last_active":"2024-09-13T14:37:45.683937+0000","last_peered":"2024-09-13T14:37:45.683937+0000","last_clean":"2024-09-13T14:37:45.683937+0000","last_became_active":"2024-09-13T14:29:42.539068+0000","last_became_peered":"2024-09-13T14:29:42.539068+0000","last_unstale":"2024-09-13T14:37:45.683937+0000","last_undegraded":"2024-09-13T14:37:45.683937+0000","last_fullsized":"2024-09-13T14:37:45.683937+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:24:06.048855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034405999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1433,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.682023+0000","last_change":"2024-09-13T14:30:15.574951+0000","last_active":"2024-09-13T14:37:46.682023+0000","last_peered":"2024-09-13T14:37:46.682023+0000","last_clean":"2024-09-13T14:37:46.682023+0000","last_became_active":"2024-09-13T14:30:15.574578+0000","last_became_peered":"2024-09-13T14:30:15.574578+0000","last_unstale":"2024-09-13T14:37:46.682023+0000","last_undegraded":"2024-09-13T14:37:46.682023+0000","last_fullsized":"2024-09-13T14:37:46.682023+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:20.700591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":995,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.417001+0000","last_change":"2024-09-13T14:19:27.379312+0000","last_active":"2024-09-13T14:37:46.417001+0000","last_peered":"2024-09-13T14:37:46.417001+0000","last_clean":"2024-09-13T14:37:46.417001+0000","last_became_active":"2024-09-13T14:19:27.378947+0000","last_became_peered":"2024-09-13T14:19:27.378947+0000","last_unstale":"2024-09-13T14:37:46.417001+0000","last_undegraded":"2024-09-13T14:37:46.417001+0000","last_fullsized":"2024-09-13T14:37:46.417001+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:54:20.706578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"1070'1682","reported_seq":4818,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.684172+0000","last_change":"2024-09-13T14:37:42.737927+0000","last_active":"2024-09-13T14:37:45.684172+0000","last_peered":"2024-09-13T14:37:45.684172+0000","last_clean":"2024-09-13T14:37:45.684172+0000","last_became_active":"2024-09-13T14:29:08.923205+0000","last_became_peered":"2024-09-13T14:29:08.923205+0000","last_unstale":"2024-09-13T14:37:45.684172+0000","last_undegraded":"2024-09-13T14:37:45.684172+0000","last_fullsized":"2024-09-13T14:37:45.684172+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_clean_scrub_stamp":"2024-09-13T14:20:40.921654+0000","objects_scrubbed":0,"log_size":1682,"log_dups_size":0,"ondisk_log_size":1682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:58.915512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080929299999999999,"stat_sum":{"num_bytes":12733093,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":519,"num_read_kb":592161,"num_write":394,"num_write_kb":90675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9326996,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"238","length":"1"},{"start":"245","length":"1"},{"start":"24a","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":1059,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.679211+0000","last_change":"2024-09-13T14:29:08.913445+0000","last_active":"2024-09-13T14:37:45.679211+0000","last_peered":"2024-09-13T14:37:45.679211+0000","last_clean":"2024-09-13T14:37:45.679211+0000","last_became_active":"2024-09-13T14:29:08.913153+0000","last_became_peered":"2024-09-13T14:29:08.913153+0000","last_unstale":"2024-09-13T14:37:45.679211+0000","last_undegraded":"2024-09-13T14:37:45.679211+0000","last_fullsized":"2024-09-13T14:37:45.679211+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:55:34.761217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1550,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.685573+0000","last_change":"2024-09-13T14:30:15.586856+0000","last_active":"2024-09-13T14:37:46.685573+0000","last_peered":"2024-09-13T14:37:46.685573+0000","last_clean":"2024-09-13T14:37:46.685573+0000","last_became_active":"2024-09-13T14:30:15.586543+0000","last_became_peered":"2024-09-13T14:30:15.586543+0000","last_unstale":"2024-09-13T14:37:46.685573+0000","last_undegraded":"2024-09-13T14:37:46.685573+0000","last_fullsized":"2024-09-13T14:37:46.685573+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:57.072216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1528,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.685909+0000","last_change":"2024-09-13T14:29:42.541652+0000","last_active":"2024-09-13T14:37:46.685909+0000","last_peered":"2024-09-13T14:37:46.685909+0000","last_clean":"2024-09-13T14:37:46.685909+0000","last_became_active":"2024-09-13T14:29:42.541341+0000","last_became_peered":"2024-09-13T14:29:42.541341+0000","last_unstale":"2024-09-13T14:37:46.685909+0000","last_undegraded":"2024-09-13T14:37:46.685909+0000","last_fullsized":"2024-09-13T14:37:46.685909+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:52.954478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"1070'1619","reported_seq":5553,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:50.054236+0000","last_change":"2024-09-13T14:37:43.329460+0000","last_active":"2024-09-13T14:37:50.054236+0000","last_peered":"2024-09-13T14:37:50.054236+0000","last_clean":"2024-09-13T14:37:50.054236+0000","last_became_active":"2024-09-13T14:29:42.543180+0000","last_became_peered":"2024-09-13T14:29:42.543180+0000","last_unstale":"2024-09-13T14:37:50.054236+0000","last_undegraded":"2024-09-13T14:37:50.054236+0000","last_fullsized":"2024-09-13T14:37:50.054236+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_clean_scrub_stamp":"2024-09-13T14:20:40.921654+0000","objects_scrubbed":0,"log_size":1619,"log_dups_size":0,"ondisk_log_size":1619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:28:35.696141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0053468989999999996,"stat_sum":{"num_bytes":6303318,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":466,"num_read_kb":534633,"num_write":376,"num_write_kb":87524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1703899,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1521,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:50.053650+0000","last_change":"2024-09-13T14:30:15.579354+0000","last_active":"2024-09-13T14:37:50.053650+0000","last_peered":"2024-09-13T14:37:50.053650+0000","last_clean":"2024-09-13T14:37:50.053650+0000","last_became_active":"2024-09-13T14:30:15.579068+0000","last_became_peered":"2024-09-13T14:30:15.579068+0000","last_unstale":"2024-09-13T14:37:50.053650+0000","last_undegraded":"2024-09-13T14:37:50.053650+0000","last_fullsized":"2024-09-13T14:37:50.053650+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:47:00.887687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1494,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.904852+0000","last_change":"2024-09-13T14:29:42.539284+0000","last_active":"2024-09-13T14:37:46.904852+0000","last_peered":"2024-09-13T14:37:46.904852+0000","last_clean":"2024-09-13T14:37:46.904852+0000","last_became_active":"2024-09-13T14:29:42.538970+0000","last_became_peered":"2024-09-13T14:29:42.538970+0000","last_unstale":"2024-09-13T14:37:46.904852+0000","last_undegraded":"2024-09-13T14:37:46.904852+0000","last_fullsized":"2024-09-13T14:37:46.904852+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:04.562806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1512,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.904855+0000","last_change":"2024-09-13T14:29:08.918645+0000","last_active":"2024-09-13T14:37:46.904855+0000","last_peered":"2024-09-13T14:37:46.904855+0000","last_clean":"2024-09-13T14:37:46.904855+0000","last_became_active":"2024-09-13T14:29:08.915894+0000","last_became_peered":"2024-09-13T14:29:08.915894+0000","last_unstale":"2024-09-13T14:37:46.904855+0000","last_undegraded":"2024-09-13T14:37:46.904855+0000","last_fullsized":"2024-09-13T14:37:46.904855+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:26.651547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"1070'1502","reported_seq":5560,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.690484+0000","last_change":"2024-09-13T14:37:42.730218+0000","last_active":"2024-09-13T14:37:46.690484+0000","last_peered":"2024-09-13T14:37:46.690484+0000","last_clean":"2024-09-13T14:37:46.690484+0000","last_became_active":"2024-09-13T14:29:08.917848+0000","last_became_peered":"2024-09-13T14:29:08.917848+0000","last_unstale":"2024-09-13T14:37:46.690484+0000","last_undegraded":"2024-09-13T14:37:46.690484+0000","last_fullsized":"2024-09-13T14:37:46.690484+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_clean_scrub_stamp":"2024-09-13T14:20:40.921654+0000","objects_scrubbed":0,"log_size":1502,"log_dups_size":0,"ondisk_log_size":1502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:07:04.214281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059875300000000005,"stat_sum":{"num_bytes":16787066,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":395,"num_read_kb":381654,"num_write":361,"num_write_kb":86426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"238","length":"1"},{"start":"245","length":"1"},{"start":"24a","length":"1"}]},{"pgid":"2.2","version":"56'2","reported_seq":1033,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.683722+0000","last_change":"2024-09-13T14:29:08.913956+0000","last_active":"2024-09-13T14:37:45.683722+0000","last_peered":"2024-09-13T14:37:45.683722+0000","last_clean":"2024-09-13T14:37:45.683722+0000","last_became_active":"2024-09-13T14:29:08.913081+0000","last_became_peered":"2024-09-13T14:29:08.913081+0000","last_unstale":"2024-09-13T14:37:45.683722+0000","last_undegraded":"2024-09-13T14:37:45.683722+0000","last_fullsized":"2024-09-13T14:37:45.683722+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:28.876391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1017,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.687001+0000","last_change":"2024-09-13T14:28:35.604404+0000","last_active":"2024-09-13T14:37:45.687001+0000","last_peered":"2024-09-13T14:37:45.687001+0000","last_clean":"2024-09-13T14:37:45.687001+0000","last_became_active":"2024-09-13T14:28:35.604102+0000","last_became_peered":"2024-09-13T14:28:35.604102+0000","last_unstale":"2024-09-13T14:37:45.687001+0000","last_undegraded":"2024-09-13T14:37:45.687001+0000","last_fullsized":"2024-09-13T14:37:45.687001+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:25.546010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1506,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.904633+0000","last_change":"2024-09-13T14:30:15.582454+0000","last_active":"2024-09-13T14:37:46.904633+0000","last_peered":"2024-09-13T14:37:46.904633+0000","last_clean":"2024-09-13T14:37:46.904633+0000","last_became_active":"2024-09-13T14:30:15.582143+0000","last_became_peered":"2024-09-13T14:30:15.582143+0000","last_unstale":"2024-09-13T14:37:46.904633+0000","last_undegraded":"2024-09-13T14:37:46.904633+0000","last_fullsized":"2024-09-13T14:37:46.904633+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:21:20.076270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"1072'1631","reported_seq":5705,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.687562+0000","last_change":"2024-09-13T14:37:42.739189+0000","last_active":"2024-09-13T14:37:46.687562+0000","last_peered":"2024-09-13T14:37:46.687562+0000","last_clean":"2024-09-13T14:37:46.687562+0000","last_became_active":"2024-09-13T14:28:35.610542+0000","last_became_peered":"2024-09-13T14:28:35.610542+0000","last_unstale":"2024-09-13T14:37:46.687562+0000","last_undegraded":"2024-09-13T14:37:46.687562+0000","last_fullsized":"2024-09-13T14:37:46.687562+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_clean_scrub_stamp":"2024-09-13T14:20:40.921654+0000","objects_scrubbed":0,"log_size":1631,"log_dups_size":0,"ondisk_log_size":1631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:50:21.795729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.009741184,"stat_sum":{"num_bytes":26300176,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":551,"num_read_kb":595202,"num_write":428,"num_write_kb":98173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3688887,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"238","length":"1"},{"start":"245","length":"1"},{"start":"24a","length":"1"}]},{"pgid":"3.5","version":"0'0","reported_seq":1036,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.684071+0000","last_change":"2024-09-13T14:30:15.578388+0000","last_active":"2024-09-13T14:37:45.684071+0000","last_peered":"2024-09-13T14:37:45.684071+0000","last_clean":"2024-09-13T14:37:45.684071+0000","last_became_active":"2024-09-13T14:30:15.578026+0000","last_became_peered":"2024-09-13T14:30:15.578026+0000","last_unstale":"2024-09-13T14:37:45.684071+0000","last_undegraded":"2024-09-13T14:37:45.684071+0000","last_fullsized":"2024-09-13T14:37:45.684071+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:16:07.663218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1526,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.904845+0000","last_change":"2024-09-13T14:29:08.916449+0000","last_active":"2024-09-13T14:37:46.904845+0000","last_peered":"2024-09-13T14:37:46.904845+0000","last_clean":"2024-09-13T14:37:46.904845+0000","last_became_active":"2024-09-13T14:29:08.914257+0000","last_became_peered":"2024-09-13T14:29:08.914257+0000","last_unstale":"2024-09-13T14:37:46.904845+0000","last_undegraded":"2024-09-13T14:37:46.904845+0000","last_fullsized":"2024-09-13T14:37:46.904845+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:10:32.449958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1484,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.905326+0000","last_change":"2024-09-13T14:29:08.913030+0000","last_active":"2024-09-13T14:37:46.905326+0000","last_peered":"2024-09-13T14:37:46.905326+0000","last_clean":"2024-09-13T14:37:46.905326+0000","last_became_active":"2024-09-13T14:29:08.912671+0000","last_became_peered":"2024-09-13T14:29:08.912671+0000","last_unstale":"2024-09-13T14:37:46.905326+0000","last_undegraded":"2024-09-13T14:37:46.905326+0000","last_fullsized":"2024-09-13T14:37:46.905326+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:30:41.107429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"1070'1935","reported_seq":5049,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.416505+0000","last_change":"2024-09-13T14:37:42.737780+0000","last_active":"2024-09-13T14:37:46.416505+0000","last_peered":"2024-09-13T14:37:46.416505+0000","last_clean":"2024-09-13T14:37:46.416505+0000","last_became_active":"2024-09-13T14:28:35.608180+0000","last_became_peered":"2024-09-13T14:28:35.608180+0000","last_unstale":"2024-09-13T14:37:46.416505+0000","last_undegraded":"2024-09-13T14:37:46.416505+0000","last_fullsized":"2024-09-13T14:37:46.416505+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_clean_scrub_stamp":"2024-09-13T14:20:40.921654+0000","objects_scrubbed":0,"log_size":1935,"log_dups_size":0,"ondisk_log_size":1935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:10.135774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070052700000000005,"stat_sum":{"num_bytes":16819226,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":686,"num_read_kb":718102,"num_write":495,"num_write_kb":123390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1633531,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"238","length":"1"},{"start":"245","length":"1"},{"start":"24a","length":"1"}]},{"pgid":"3.4","version":"0'0","reported_seq":1494,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.904747+0000","last_change":"2024-09-13T14:29:42.540294+0000","last_active":"2024-09-13T14:37:46.904747+0000","last_peered":"2024-09-13T14:37:46.904747+0000","last_clean":"2024-09-13T14:37:46.904747+0000","last_became_active":"2024-09-13T14:29:42.539930+0000","last_became_peered":"2024-09-13T14:29:42.539930+0000","last_unstale":"2024-09-13T14:37:46.904747+0000","last_undegraded":"2024-09-13T14:37:46.904747+0000","last_fullsized":"2024-09-13T14:37:46.904747+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:42.999879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1042,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.679633+0000","last_change":"2024-09-13T14:28:35.603663+0000","last_active":"2024-09-13T14:37:45.679633+0000","last_peered":"2024-09-13T14:37:45.679633+0000","last_clean":"2024-09-13T14:37:45.679633+0000","last_became_active":"2024-09-13T14:28:35.603358+0000","last_became_peered":"2024-09-13T14:28:35.603358+0000","last_unstale":"2024-09-13T14:37:45.679633+0000","last_undegraded":"2024-09-13T14:37:45.679633+0000","last_fullsized":"2024-09-13T14:37:45.679633+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:24:05.842796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1506,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.686069+0000","last_change":"2024-09-13T14:28:35.611508+0000","last_active":"2024-09-13T14:37:46.686069+0000","last_peered":"2024-09-13T14:37:46.686069+0000","last_clean":"2024-09-13T14:37:46.686069+0000","last_became_active":"2024-09-13T14:28:35.610106+0000","last_became_peered":"2024-09-13T14:28:35.610106+0000","last_unstale":"2024-09-13T14:37:46.686069+0000","last_undegraded":"2024-09-13T14:37:46.686069+0000","last_fullsized":"2024-09-13T14:37:46.686069+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:11.966958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"1076'2228","reported_seq":5301,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.690553+0000","last_change":"2024-09-13T14:37:42.752138+0000","last_active":"2024-09-13T14:37:46.690553+0000","last_peered":"2024-09-13T14:37:46.690553+0000","last_clean":"2024-09-13T14:37:46.690553+0000","last_became_active":"2024-09-13T14:30:15.580584+0000","last_became_peered":"2024-09-13T14:30:15.580584+0000","last_unstale":"2024-09-13T14:37:46.690553+0000","last_undegraded":"2024-09-13T14:37:46.690553+0000","last_fullsized":"2024-09-13T14:37:46.690553+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_clean_scrub_stamp":"2024-09-13T14:20:40.921654+0000","objects_scrubbed":0,"log_size":2228,"log_dups_size":0,"ondisk_log_size":2228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:43.644497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002746479,"stat_sum":{"num_bytes":19014968,"num_objects":9,"num_object_clones":4,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":659,"num_read_kb":690274,"num_write":583,"num_write_kb":141127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4306064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"238","length":"1"},{"start":"245","length":"1"},{"start":"24a","length":"1"}]},{"pgid":"3.7","version":"63'2","reported_seq":1466,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.683066+0000","last_change":"2024-09-13T14:30:15.581728+0000","last_active":"2024-09-13T14:37:46.683066+0000","last_peered":"2024-09-13T14:37:46.683066+0000","last_clean":"2024-09-13T14:37:46.683066+0000","last_became_active":"2024-09-13T14:30:15.581303+0000","last_became_peered":"2024-09-13T14:30:15.581303+0000","last_unstale":"2024-09-13T14:37:46.683066+0000","last_undegraded":"2024-09-13T14:37:46.683066+0000","last_fullsized":"2024-09-13T14:37:46.683066+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:10.027991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1494,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.904937+0000","last_change":"2024-09-13T14:29:08.917504+0000","last_active":"2024-09-13T14:37:46.904937+0000","last_peered":"2024-09-13T14:37:46.904937+0000","last_clean":"2024-09-13T14:37:46.904937+0000","last_became_active":"2024-09-13T14:29:08.917217+0000","last_became_peered":"2024-09-13T14:29:08.917217+0000","last_unstale":"2024-09-13T14:37:46.904937+0000","last_undegraded":"2024-09-13T14:37:46.904937+0000","last_fullsized":"2024-09-13T14:37:46.904937+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:36.651514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1463,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.682401+0000","last_change":"2024-09-13T14:30:15.583561+0000","last_active":"2024-09-13T14:37:46.682401+0000","last_peered":"2024-09-13T14:37:46.682401+0000","last_clean":"2024-09-13T14:37:46.682401+0000","last_became_active":"2024-09-13T14:30:15.581464+0000","last_became_peered":"2024-09-13T14:30:15.581464+0000","last_unstale":"2024-09-13T14:37:46.682401+0000","last_undegraded":"2024-09-13T14:37:46.682401+0000","last_fullsized":"2024-09-13T14:37:46.682401+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:55:19.762648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"1076'1398","reported_seq":5342,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.906613+0000","last_change":"2024-09-13T14:37:42.726658+0000","last_active":"2024-09-13T14:37:46.906613+0000","last_peered":"2024-09-13T14:37:46.906613+0000","last_clean":"2024-09-13T14:37:46.906613+0000","last_became_active":"2024-09-13T14:30:15.577857+0000","last_became_peered":"2024-09-13T14:30:15.577857+0000","last_unstale":"2024-09-13T14:37:46.906613+0000","last_undegraded":"2024-09-13T14:37:46.906613+0000","last_fullsized":"2024-09-13T14:37:46.906613+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:20:40.921654+0000","last_clean_scrub_stamp":"2024-09-13T14:20:40.921654+0000","objects_scrubbed":0,"log_size":1398,"log_dups_size":0,"ondisk_log_size":1398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:19:37.205383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089470900000000004,"stat_sum":{"num_bytes":18262448,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":445,"num_read_kb":453436,"num_write":350,"num_write_kb":83193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2320450,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"238","length":"1"},{"start":"245","length":"1"},{"start":"24a","length":"1"}]},{"pgid":"3.6","version":"63'1","reported_seq":1520,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.686293+0000","last_change":"2024-09-13T14:29:08.916436+0000","last_active":"2024-09-13T14:37:46.686293+0000","last_peered":"2024-09-13T14:37:46.686293+0000","last_clean":"2024-09-13T14:37:46.686293+0000","last_became_active":"2024-09-13T14:29:08.916126+0000","last_became_peered":"2024-09-13T14:29:08.916126+0000","last_unstale":"2024-09-13T14:37:46.686293+0000","last_undegraded":"2024-09-13T14:37:46.686293+0000","last_fullsized":"2024-09-13T14:37:46.686293+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:00.324006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1026,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.417062+0000","last_change":"2024-09-13T14:29:42.538860+0000","last_active":"2024-09-13T14:37:46.417062+0000","last_peered":"2024-09-13T14:37:46.417062+0000","last_clean":"2024-09-13T14:37:46.417062+0000","last_became_active":"2024-09-13T14:29:42.538608+0000","last_became_peered":"2024-09-13T14:29:42.538608+0000","last_unstale":"2024-09-13T14:37:46.417062+0000","last_undegraded":"2024-09-13T14:37:46.417062+0000","last_fullsized":"2024-09-13T14:37:46.417062+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:20.070703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":999,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.686459+0000","last_change":"2024-09-13T14:19:26.131609+0000","last_active":"2024-09-13T14:37:45.686459+0000","last_peered":"2024-09-13T14:37:45.686459+0000","last_clean":"2024-09-13T14:37:45.686459+0000","last_became_active":"2024-09-13T14:19:26.131029+0000","last_became_peered":"2024-09-13T14:19:26.131029+0000","last_unstale":"2024-09-13T14:37:45.686459+0000","last_undegraded":"2024-09-13T14:37:45.686459+0000","last_fullsized":"2024-09-13T14:37:45.686459+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:57.933696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1016,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.686822+0000","last_change":"2024-09-13T14:29:08.911835+0000","last_active":"2024-09-13T14:37:45.686822+0000","last_peered":"2024-09-13T14:37:45.686822+0000","last_clean":"2024-09-13T14:37:45.686822+0000","last_became_active":"2024-09-13T14:29:08.911542+0000","last_became_peered":"2024-09-13T14:29:08.911542+0000","last_unstale":"2024-09-13T14:37:45.686822+0000","last_undegraded":"2024-09-13T14:37:45.686822+0000","last_fullsized":"2024-09-13T14:37:45.686822+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:30.820674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":1022,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.686689+0000","last_change":"2024-09-13T14:19:25.870605+0000","last_active":"2024-09-13T14:37:45.686689+0000","last_peered":"2024-09-13T14:37:45.686689+0000","last_clean":"2024-09-13T14:37:45.686689+0000","last_became_active":"2024-09-13T14:19:25.870282+0000","last_became_peered":"2024-09-13T14:19:25.870282+0000","last_unstale":"2024-09-13T14:37:45.686689+0000","last_undegraded":"2024-09-13T14:37:45.686689+0000","last_fullsized":"2024-09-13T14:37:45.686689+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:52.514915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1494,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.905182+0000","last_change":"2024-09-13T14:29:42.537787+0000","last_active":"2024-09-13T14:37:46.905182+0000","last_peered":"2024-09-13T14:37:46.905182+0000","last_clean":"2024-09-13T14:37:46.905182+0000","last_became_active":"2024-09-13T14:29:42.537499+0000","last_became_peered":"2024-09-13T14:29:42.537499+0000","last_unstale":"2024-09-13T14:37:46.905182+0000","last_undegraded":"2024-09-13T14:37:46.905182+0000","last_fullsized":"2024-09-13T14:37:46.905182+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:15:30.508656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":1033,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.679431+0000","last_change":"2024-09-13T14:30:15.579260+0000","last_active":"2024-09-13T14:37:45.679431+0000","last_peered":"2024-09-13T14:37:45.679431+0000","last_clean":"2024-09-13T14:37:45.679431+0000","last_became_active":"2024-09-13T14:30:15.578897+0000","last_became_peered":"2024-09-13T14:30:15.578897+0000","last_unstale":"2024-09-13T14:37:45.679431+0000","last_undegraded":"2024-09-13T14:37:45.679431+0000","last_fullsized":"2024-09-13T14:37:45.679431+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:56.737849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1019,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.684080+0000","last_change":"2024-09-13T14:29:08.910821+0000","last_active":"2024-09-13T14:37:45.684080+0000","last_peered":"2024-09-13T14:37:45.684080+0000","last_clean":"2024-09-13T14:37:45.684080+0000","last_became_active":"2024-09-13T14:29:08.910509+0000","last_became_peered":"2024-09-13T14:29:08.910509+0000","last_unstale":"2024-09-13T14:37:45.684080+0000","last_undegraded":"2024-09-13T14:37:45.684080+0000","last_fullsized":"2024-09-13T14:37:45.684080+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:59.495053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1434,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.681778+0000","last_change":"2024-09-13T14:30:15.576921+0000","last_active":"2024-09-13T14:37:46.681778+0000","last_peered":"2024-09-13T14:37:46.681778+0000","last_clean":"2024-09-13T14:37:46.681778+0000","last_became_active":"2024-09-13T14:30:15.576607+0000","last_became_peered":"2024-09-13T14:30:15.576607+0000","last_unstale":"2024-09-13T14:37:46.681778+0000","last_undegraded":"2024-09-13T14:37:46.681778+0000","last_fullsized":"2024-09-13T14:37:46.681778+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:35.162460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1549,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.686217+0000","last_change":"2024-09-13T14:30:15.585712+0000","last_active":"2024-09-13T14:37:46.686217+0000","last_peered":"2024-09-13T14:37:46.686217+0000","last_clean":"2024-09-13T14:37:46.686217+0000","last_became_active":"2024-09-13T14:30:15.585147+0000","last_became_peered":"2024-09-13T14:30:15.585147+0000","last_unstale":"2024-09-13T14:37:46.686217+0000","last_undegraded":"2024-09-13T14:37:46.686217+0000","last_fullsized":"2024-09-13T14:37:46.686217+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:40:33.859429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1457,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.681704+0000","last_change":"2024-09-13T14:30:15.576306+0000","last_active":"2024-09-13T14:37:46.681704+0000","last_peered":"2024-09-13T14:37:46.681704+0000","last_clean":"2024-09-13T14:37:46.681704+0000","last_became_active":"2024-09-13T14:30:15.575931+0000","last_became_peered":"2024-09-13T14:30:15.575931+0000","last_unstale":"2024-09-13T14:37:46.681704+0000","last_undegraded":"2024-09-13T14:37:46.681704+0000","last_fullsized":"2024-09-13T14:37:46.681704+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:54.078309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1012,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.417023+0000","last_change":"2024-09-13T14:29:08.913672+0000","last_active":"2024-09-13T14:37:46.417023+0000","last_peered":"2024-09-13T14:37:46.417023+0000","last_clean":"2024-09-13T14:37:46.417023+0000","last_became_active":"2024-09-13T14:29:08.913360+0000","last_became_peered":"2024-09-13T14:29:08.913360+0000","last_unstale":"2024-09-13T14:37:46.417023+0000","last_undegraded":"2024-09-13T14:37:46.417023+0000","last_fullsized":"2024-09-13T14:37:46.417023+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:45.688163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1473,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.682333+0000","last_change":"2024-09-13T14:30:15.583872+0000","last_active":"2024-09-13T14:37:46.682333+0000","last_peered":"2024-09-13T14:37:46.682333+0000","last_clean":"2024-09-13T14:37:46.682333+0000","last_became_active":"2024-09-13T14:30:15.581869+0000","last_became_peered":"2024-09-13T14:30:15.581869+0000","last_unstale":"2024-09-13T14:37:46.682333+0000","last_undegraded":"2024-09-13T14:37:46.682333+0000","last_fullsized":"2024-09-13T14:37:46.682333+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:10:37.393352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1517,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.686012+0000","last_change":"2024-09-13T14:30:15.586018+0000","last_active":"2024-09-13T14:37:46.686012+0000","last_peered":"2024-09-13T14:37:46.686012+0000","last_clean":"2024-09-13T14:37:46.686012+0000","last_became_active":"2024-09-13T14:30:15.585616+0000","last_became_peered":"2024-09-13T14:30:15.585616+0000","last_unstale":"2024-09-13T14:37:46.686012+0000","last_undegraded":"2024-09-13T14:37:46.686012+0000","last_fullsized":"2024-09-13T14:37:46.686012+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:46.836406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1493,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.904513+0000","last_change":"2024-09-13T14:30:15.579398+0000","last_active":"2024-09-13T14:37:46.904513+0000","last_peered":"2024-09-13T14:37:46.904513+0000","last_clean":"2024-09-13T14:37:46.904513+0000","last_became_active":"2024-09-13T14:30:15.579062+0000","last_became_peered":"2024-09-13T14:30:15.579062+0000","last_unstale":"2024-09-13T14:37:46.904513+0000","last_undegraded":"2024-09-13T14:37:46.904513+0000","last_fullsized":"2024-09-13T14:37:46.904513+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:06:39.532417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1455,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.681987+0000","last_change":"2024-09-13T14:30:15.576778+0000","last_active":"2024-09-13T14:37:46.681987+0000","last_peered":"2024-09-13T14:37:46.681987+0000","last_clean":"2024-09-13T14:37:46.681987+0000","last_became_active":"2024-09-13T14:30:15.576434+0000","last_became_peered":"2024-09-13T14:30:15.576434+0000","last_unstale":"2024-09-13T14:37:46.681987+0000","last_undegraded":"2024-09-13T14:37:46.681987+0000","last_fullsized":"2024-09-13T14:37:46.681987+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:54:06.503790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":1035,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.686640+0000","last_change":"2024-09-13T14:29:42.537809+0000","last_active":"2024-09-13T14:37:45.686640+0000","last_peered":"2024-09-13T14:37:45.686640+0000","last_clean":"2024-09-13T14:37:45.686640+0000","last_became_active":"2024-09-13T14:29:42.537482+0000","last_became_peered":"2024-09-13T14:29:42.537482+0000","last_unstale":"2024-09-13T14:37:45.686640+0000","last_undegraded":"2024-09-13T14:37:45.686640+0000","last_fullsized":"2024-09-13T14:37:45.686640+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:19.631210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1498,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:50.054339+0000","last_change":"2024-09-13T14:30:15.578675+0000","last_active":"2024-09-13T14:37:50.054339+0000","last_peered":"2024-09-13T14:37:50.054339+0000","last_clean":"2024-09-13T14:37:50.054339+0000","last_became_active":"2024-09-13T14:30:15.578362+0000","last_became_peered":"2024-09-13T14:30:15.578362+0000","last_unstale":"2024-09-13T14:37:50.054339+0000","last_undegraded":"2024-09-13T14:37:50.054339+0000","last_fullsized":"2024-09-13T14:37:50.054339+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:30:47.445865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":1508,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.686287+0000","last_change":"2024-09-13T14:28:35.612469+0000","last_active":"2024-09-13T14:37:46.686287+0000","last_peered":"2024-09-13T14:37:46.686287+0000","last_clean":"2024-09-13T14:37:46.686287+0000","last_became_active":"2024-09-13T14:28:35.612093+0000","last_became_peered":"2024-09-13T14:28:35.612093+0000","last_unstale":"2024-09-13T14:37:46.686287+0000","last_undegraded":"2024-09-13T14:37:46.686287+0000","last_fullsized":"2024-09-13T14:37:46.686287+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:51.022663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1026,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.686801+0000","last_change":"2024-09-13T14:30:15.580568+0000","last_active":"2024-09-13T14:37:45.686801+0000","last_peered":"2024-09-13T14:37:45.686801+0000","last_clean":"2024-09-13T14:37:45.686801+0000","last_became_active":"2024-09-13T14:30:15.580186+0000","last_became_peered":"2024-09-13T14:30:15.580186+0000","last_unstale":"2024-09-13T14:37:45.686801+0000","last_undegraded":"2024-09-13T14:37:45.686801+0000","last_fullsized":"2024-09-13T14:37:45.686801+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:16:00.781680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":1021,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.683858+0000","last_change":"2024-09-13T14:29:08.912819+0000","last_active":"2024-09-13T14:37:45.683858+0000","last_peered":"2024-09-13T14:37:45.683858+0000","last_clean":"2024-09-13T14:37:45.683858+0000","last_became_active":"2024-09-13T14:29:08.912464+0000","last_became_peered":"2024-09-13T14:29:08.912464+0000","last_unstale":"2024-09-13T14:37:45.683858+0000","last_undegraded":"2024-09-13T14:37:45.683858+0000","last_fullsized":"2024-09-13T14:37:45.683858+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:09.753940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1033,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.686617+0000","last_change":"2024-09-13T14:30:15.580011+0000","last_active":"2024-09-13T14:37:45.686617+0000","last_peered":"2024-09-13T14:37:45.686617+0000","last_clean":"2024-09-13T14:37:45.686617+0000","last_became_active":"2024-09-13T14:30:15.579610+0000","last_became_peered":"2024-09-13T14:30:15.579610+0000","last_unstale":"2024-09-13T14:37:45.686617+0000","last_undegraded":"2024-09-13T14:37:45.686617+0000","last_fullsized":"2024-09-13T14:37:45.686617+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:11:02.733771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":1004,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.684000+0000","last_change":"2024-09-13T14:19:25.873514+0000","last_active":"2024-09-13T14:37:45.684000+0000","last_peered":"2024-09-13T14:37:45.684000+0000","last_clean":"2024-09-13T14:37:45.684000+0000","last_became_active":"2024-09-13T14:19:25.873170+0000","last_became_peered":"2024-09-13T14:19:25.873170+0000","last_unstale":"2024-09-13T14:37:45.684000+0000","last_undegraded":"2024-09-13T14:37:45.684000+0000","last_fullsized":"2024-09-13T14:37:45.684000+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:48.298530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1464,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.682259+0000","last_change":"2024-09-13T14:30:15.583672+0000","last_active":"2024-09-13T14:37:46.682259+0000","last_peered":"2024-09-13T14:37:46.682259+0000","last_clean":"2024-09-13T14:37:46.682259+0000","last_became_active":"2024-09-13T14:30:15.583160+0000","last_became_peered":"2024-09-13T14:30:15.583160+0000","last_unstale":"2024-09-13T14:37:46.682259+0000","last_undegraded":"2024-09-13T14:37:46.682259+0000","last_fullsized":"2024-09-13T14:37:46.682259+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:54.303579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":1449,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.682780+0000","last_change":"2024-09-13T14:30:15.576496+0000","last_active":"2024-09-13T14:37:46.682780+0000","last_peered":"2024-09-13T14:37:46.682780+0000","last_clean":"2024-09-13T14:37:46.682780+0000","last_became_active":"2024-09-13T14:30:15.576137+0000","last_became_peered":"2024-09-13T14:30:15.576137+0000","last_unstale":"2024-09-13T14:37:46.682780+0000","last_undegraded":"2024-09-13T14:37:46.682780+0000","last_fullsized":"2024-09-13T14:37:46.682780+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:24.316047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1026,"reported_epoch":1075,"state":"active+clean","last_fresh":"2024-09-13T14:37:45.686253+0000","last_change":"2024-09-13T14:29:42.539907+0000","last_active":"2024-09-13T14:37:45.686253+0000","last_peered":"2024-09-13T14:37:45.686253+0000","last_clean":"2024-09-13T14:37:45.686253+0000","last_became_active":"2024-09-13T14:29:42.539583+0000","last_became_peered":"2024-09-13T14:29:42.539583+0000","last_unstale":"2024-09-13T14:37:45.686253+0000","last_undegraded":"2024-09-13T14:37:45.686253+0000","last_fullsized":"2024-09-13T14:37:45.686253+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:02:04.628447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":1523,"reported_epoch":1076,"state":"active+clean","last_fresh":"2024-09-13T14:37:46.686373+0000","last_change":"2024-09-13T14:29:42.541263+0000","last_active":"2024-09-13T14:37:46.686373+0000","last_peered":"2024-09-13T14:37:46.686373+0000","last_clean":"2024-09-13T14:37:46.686373+0000","last_became_active":"2024-09-13T14:29:42.540873+0000","last_became_peered":"2024-09-13T14:29:42.540873+0000","last_unstale":"2024-09-13T14:37:46.686373+0000","last_undegraded":"2024-09-13T14:37:46.686373+0000","last_fullsized":"2024-09-13T14:37:46.686373+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:20.611206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":235395123,"num_objects":106,"num_object_clones":61,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106,"num_whiteouts":8,"num_read":6753,"num_read_kb":7291224,"num_write":5605,"num_write_kb":1349886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":42088749,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":406499328,"data_stored":494129427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22004,"ondisk_log_size":22004,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430452,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":355620,"kb_used_data":54100,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93389532,"statfs":{"total":95995035648,"available":95630880768,"internally_reserved":0,"allocated":55398400,"data_stored":56923144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626681,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":355440,"kb_used_data":45604,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93389712,"statfs":{"total":95995035648,"available":95631065088,"internally_reserved":0,"allocated":46698496,"data_stored":55150609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822910,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":372708,"kb_used_data":75348,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93372444,"statfs":{"total":95995035648,"available":95613382656,"internally_reserved":0,"allocated":77156352,"data_stored":91096417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019138,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":376100,"kb_used_data":70420,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93369052,"statfs":{"total":95995035648,"available":95609909248,"internally_reserved":0,"allocated":72110080,"data_stored":79683057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059592796,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":73104,"kb_used_data":37052,"kb_used_omap":26,"kb_used_meta":36005,"kb_avail":93672048,"statfs":{"total":95995035648,"available":95920177152,"internally_reserved":0,"allocated":37941248,"data_stored":39808954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":36869280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":591,"seq":2538325672035,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":84968,"kb_used_data":57052,"kb_used_omap":41,"kb_used_meta":27862,"kb_avail":93660184,"statfs":{"total":95995035648,"available":95908028416,"internally_reserved":0,"allocated":58421248,"data_stored":58596175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42651,"internal_metadata":28531045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":557,"seq":2392296783977,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":82772,"kb_used_data":54912,"kb_used_omap":32,"kb_used_meta":27807,"kb_avail":93662380,"statfs":{"total":95995035648,"available":95910277120,"internally_reserved":0,"allocated":56229888,"data_stored":63890481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33752,"internal_metadata":28474408},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":524,"seq":2250562863216,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":128692,"kb_used_data":74468,"kb_used_omap":41,"kb_used_meta":54166,"kb_avail":93616460,"statfs":{"total":95995035648,"available":95863255040,"internally_reserved":0,"allocated":76255232,"data_stored":90285135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42694,"internal_metadata":55466298},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":66076672,"data_stored":83209369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":46661632,"data_stored":59837087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":48824320,"data_stored":52102359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28360704,"data_stored":33324082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":65777664,"data_stored":81195830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":67567616,"data_stored":84610996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":38031360,"data_stored":50006650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45199360,"data_stored":49843054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:37:52.640 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-13T14:37:52.641 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-13T14:37:52.883 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:37:53.003 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:52 smithi151 ceph-mon[104450]: pgmap v1236: 89 pgs: 89 active+clean; 225 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.3 MiB/s wr, 9 op/s 2024-09-13T14:37:53.003 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:52 smithi151 ceph-mon[104450]: from='client.44954 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:37:53.003 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:52 smithi151 ceph-mon[108966]: pgmap v1236: 89 pgs: 89 active+clean; 225 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.3 MiB/s wr, 9 op/s 2024-09-13T14:37:53.003 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:52 smithi151 ceph-mon[108966]: from='client.44954 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:37:53.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:52 smithi184 ceph-mon[92527]: pgmap v1236: 89 pgs: 89 active+clean; 225 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.3 MiB/s wr, 9 op/s 2024-09-13T14:37:53.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:52 smithi184 ceph-mon[92527]: from='client.44954 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:37:54.711 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_0' removed 2024-09-13T14:37:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:54 smithi151 ceph-mon[104450]: pgmap v1237: 89 pgs: 89 active+clean; 225 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 636 KiB/s wr, 5 op/s 2024-09-13T14:37:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:54 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1389799712' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:37:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:54 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:37:54.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:37:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:37:54] "GET /metrics HTTP/1.1" 200 36511 "" "Prometheus/2.51.0" 2024-09-13T14:37:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:54 smithi151 ceph-mon[108966]: pgmap v1237: 89 pgs: 89 active+clean; 225 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 636 KiB/s wr, 5 op/s 2024-09-13T14:37:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:54 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1389799712' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:37:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:54 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:37:55.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:54 smithi184 ceph-mon[92527]: pgmap v1237: 89 pgs: 89 active+clean; 225 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 636 KiB/s wr, 5 op/s 2024-09-13T14:37:55.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:54 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1389799712' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:37:55.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:54 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:37:55.308 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-13T14:37:55.308 INFO:teuthology.task.print:**** done rados/snaps-few-objects 5-workload 2024-09-13T14:37:55.308 DEBUG:teuthology.parallel:result is None 2024-09-13T14:37:55.308 INFO:teuthology.run_tasks:Running task print... 2024-09-13T14:37:55.318 INFO:teuthology.task.print:**** done upgrading first half of cluster **** 2024-09-13T14:37:55.318 INFO:teuthology.run_tasks:Running task ceph.healthy... 2024-09-13T14:37:55.347 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-13T14:37:55.347 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-13T14:37:55.347 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-13T14:37:55.594 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:37:55.978 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:55 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:37:55.979 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:55 smithi151 ceph-mon[108966]: osdmap e1078: 8 total, 8 up, 8 in 2024-09-13T14:37:55.979 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:55 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:37:55.979 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:55 smithi151 ceph-mon[104450]: osdmap e1078: 8 total, 8 up, 8 in 2024-09-13T14:37:56.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:55 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:37:56.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:55 smithi184 ceph-mon[92527]: osdmap e1078: 8 total, 8 up, 8 in 2024-09-13T14:37:56.991 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:56 smithi151 ceph-mon[104450]: pgmap v1239: 73 pgs: 73 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:37:56.992 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:56 smithi151 ceph-mon[108966]: pgmap v1239: 73 pgs: 73 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:37:57.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:56 smithi184 ceph-mon[92527]: pgmap v1239: 73 pgs: 73 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:37:57.591 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:37:57.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:57 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3443413766' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-13T14:37:57.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:57 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3443413766' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-13T14:37:58.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:57 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3443413766' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-13T14:37:58.255 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":38,"flags":0,"active_gid":25952,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":974639016},{"type":"v1","addr":"172.21.15.151:6801","nonce":974639016}]},"active_addr":"172.21.15.151:6801/974639016","active_change":"2024-09-13T14:24:01.073288+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":25591,"name":"x","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.151:8443/","prometheus":"http://172.21.15.151:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":259,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":2410398088}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":2687218015}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":3463177543}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":575778090}]}]} 2024-09-13T14:37:58.259 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-13T14:37:58.259 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-13T14:37:58.260 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T14:37:58.501 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:37:58.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:37:58 smithi151 ceph-mon[104450]: pgmap v1240: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:37:58.950 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:37:58 smithi151 ceph-mon[108966]: pgmap v1240: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:37:59.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:37:58 smithi184 ceph-mon[92527]: pgmap v1240: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:00.363 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:38:00.363 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":1078,"fsid":"2401fc50-71da-11ef-bceb-c7b262605968","created":"2024-09-13T14:12:36.168205+0000","modified":"2024-09-13T14:37:54.701230+0000","last_up_change":"2024-09-13T14:30:14.547371+0000","last_in_change":"2024-09-13T14:17:24.771724+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:15:41.780748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:18:09.030957+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":56,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T14:19:24.791384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T14:19:25.101452+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0ad945ee-af7d-4f26-a1e9-18ed963194ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":511,"up_from":524,"up_thru":624,"down_at":512,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2137494296},{"type":"v1","addr":"172.21.15.151:6803","nonce":2137494296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2137494296},{"type":"v1","addr":"172.21.15.151:6805","nonce":2137494296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":2137494296},{"type":"v1","addr":"172.21.15.151:6809","nonce":2137494296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2137494296},{"type":"v1","addr":"172.21.15.151:6807","nonce":2137494296}]},"public_addr":"172.21.15.151:6803/2137494296","cluster_addr":"172.21.15.151:6805/2137494296","heartbeat_back_addr":"172.21.15.151:6809/2137494296","heartbeat_front_addr":"172.21.15.151:6807/2137494296","state":["exists","up"]},{"osd":1,"uuid":"09f5d705-684f-4e86-80a2-ec2292af09d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":544,"up_from":557,"up_thru":624,"down_at":545,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":3884881052},{"type":"v1","addr":"172.21.15.151:6811","nonce":3884881052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3884881052},{"type":"v1","addr":"172.21.15.151:6813","nonce":3884881052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3884881052},{"type":"v1","addr":"172.21.15.151:6817","nonce":3884881052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":3884881052},{"type":"v1","addr":"172.21.15.151:6815","nonce":3884881052}]},"public_addr":"172.21.15.151:6811/3884881052","cluster_addr":"172.21.15.151:6813/3884881052","heartbeat_back_addr":"172.21.15.151:6817/3884881052","heartbeat_front_addr":"172.21.15.151:6815/3884881052","state":["exists","up"]},{"osd":2,"uuid":"6a63f8af-ff69-46bb-830f-03ea89fc78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":578,"up_from":591,"up_thru":624,"down_at":579,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":322435630},{"type":"v1","addr":"172.21.15.151:6819","nonce":322435630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":322435630},{"type":"v1","addr":"172.21.15.151:6821","nonce":322435630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":322435630},{"type":"v1","addr":"172.21.15.151:6825","nonce":322435630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":322435630},{"type":"v1","addr":"172.21.15.151:6823","nonce":322435630}]},"public_addr":"172.21.15.151:6819/322435630","cluster_addr":"172.21.15.151:6821/322435630","heartbeat_back_addr":"172.21.15.151:6825/322435630","heartbeat_front_addr":"172.21.15.151:6823/322435630","state":["exists","up"]},{"osd":3,"uuid":"770e8334-3b3a-46bc-9782-964b2a696660","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":611,"up_from":624,"up_thru":624,"down_at":612,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2997668861},{"type":"v1","addr":"172.21.15.151:6827","nonce":2997668861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":2997668861},{"type":"v1","addr":"172.21.15.151:6829","nonce":2997668861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":2997668861},{"type":"v1","addr":"172.21.15.151:6833","nonce":2997668861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":2997668861},{"type":"v1","addr":"172.21.15.151:6831","nonce":2997668861}]},"public_addr":"172.21.15.151:6827/2997668861","cluster_addr":"172.21.15.151:6829/2997668861","heartbeat_back_addr":"172.21.15.151:6833/2997668861","heartbeat_front_addr":"172.21.15.151:6831/2997668861","state":["exists","up"]},{"osd":4,"uuid":"75e258c0-1145-4aee-86aa-db2b63082cc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":624,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6801","nonce":3275553935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6803","nonce":3275553935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6807","nonce":3275553935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6805","nonce":3275553935}]},"public_addr":"172.21.15.184:6801/3275553935","cluster_addr":"172.21.15.184:6803/3275553935","heartbeat_back_addr":"172.21.15.184:6807/3275553935","heartbeat_front_addr":"172.21.15.184:6805/3275553935","state":["exists","up"]},{"osd":5,"uuid":"f56a5366-9d22-4960-933d-340aa280fbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":624,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6809","nonce":2446419621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6811","nonce":2446419621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6815","nonce":2446419621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6813","nonce":2446419621}]},"public_addr":"172.21.15.184:6809/2446419621","cluster_addr":"172.21.15.184:6811/2446419621","heartbeat_back_addr":"172.21.15.184:6815/2446419621","heartbeat_front_addr":"172.21.15.184:6813/2446419621","state":["exists","up"]},{"osd":6,"uuid":"e9826568-a728-45ab-8180-9777fe31f93a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":612,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6817","nonce":2214837203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6819","nonce":2214837203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6823","nonce":2214837203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6821","nonce":2214837203}]},"public_addr":"172.21.15.184:6817/2214837203","cluster_addr":"172.21.15.184:6819/2214837203","heartbeat_back_addr":"172.21.15.184:6823/2214837203","heartbeat_front_addr":"172.21.15.184:6821/2214837203","state":["exists","up"]},{"osd":7,"uuid":"d76869fe-5121-4467-8b56-1aadef0eaf65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":624,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6825","nonce":3378735997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6827","nonce":3378735997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6831","nonce":3378735997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6829","nonce":3378735997}]},"public_addr":"172.21.15.184:6825/3378735997","cluster_addr":"172.21.15.184:6827/3378735997","heartbeat_back_addr":"172.21.15.184:6831/3378735997","heartbeat_front_addr":"172.21.15.184:6829/3378735997","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:14:49.835715+0000","dead_epoch":511},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:15:14.682882+0000","dead_epoch":544},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:15:39.423890+0000","dead_epoch":578},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:16:04.527460+0000","dead_epoch":611},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:16:27.283661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:16:51.893868+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:16.152592+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:39.763533+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.151:0/1783031268":"2024-09-14T14:24:01.073069+0000","172.21.15.151:0/1775071943":"2024-09-14T14:24:01.073069+0000","172.21.15.151:6801/42840":"2024-09-14T14:24:01.073069+0000","172.21.15.151:0/841256485":"2024-09-14T14:24:01.073069+0000","172.21.15.151:0/1182681341":"2024-09-14T14:13:04.761383+0000","172.21.15.151:6800/1057983082":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/2889050140":"2024-09-14T14:18:18.344788+0000","172.21.15.151:6801/4025698662":"2024-09-14T14:22:15.389006+0000","172.21.15.151:0/2800235450":"2024-09-14T14:18:18.344788+0000","172.21.15.184:0/3929228938":"2024-09-14T14:23:20.772174+0000","172.21.15.151:0/1409232061":"2024-09-14T14:24:01.073069+0000","172.21.15.151:6800/4025698662":"2024-09-14T14:22:15.389006+0000","172.21.15.184:0/1999058007":"2024-09-14T14:23:20.772174+0000","172.21.15.151:6801/1057983082":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/514781427":"2024-09-14T14:22:15.389006+0000","172.21.15.151:0/663506338":"2024-09-14T14:24:01.073069+0000","172.21.15.151:6800/3383065173":"2024-09-14T14:13:04.761383+0000","172.21.15.184:6834/205203765":"2024-09-14T14:23:20.772174+0000","172.21.15.151:0/1530349337":"2024-09-14T14:22:15.389006+0000","172.21.15.151:0/1427922964":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/2666829207":"2024-09-14T14:13:27.478847+0000","172.21.15.151:6801/2335297718":"2024-09-14T14:18:18.344788+0000","172.21.15.151:0/1351186656":"2024-09-14T14:18:18.344788+0000","172.21.15.184:6835/205203765":"2024-09-14T14:23:20.772174+0000","172.21.15.184:0/3003411098":"2024-09-14T14:23:20.772174+0000","172.21.15.151:0/4115173791":"2024-09-14T14:18:18.344788+0000","172.21.15.151:0/3689756337":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/1517910843":"2024-09-14T14:24:01.073069+0000","172.21.15.184:0/3370242044":"2024-09-14T14:23:20.772174+0000","172.21.15.151:6800/2335297718":"2024-09-14T14:18:18.344788+0000","172.21.15.151:6800/42840":"2024-09-14T14:24:01.073069+0000","172.21.15.151:0/1488764319":"2024-09-14T14:22:15.389006+0000","172.21.15.184:0/2995983270":"2024-09-14T14:23:20.772174+0000","172.21.15.151:0/3946777511":"2024-09-14T14:13:04.761383+0000","172.21.15.151:0/624042464":"2024-09-14T14:13:04.761383+0000","172.21.15.151:0/2201280183":"2024-09-14T14:22:15.389006+0000","172.21.15.151:6801/3383065173":"2024-09-14T14:13:04.761383+0000","172.21.15.184:0/3047962606":"2024-09-14T14:23:20.772174+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T14:38:00.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:38:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:38:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:38:00.962 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:00 smithi151 ceph-mon[104450]: pgmap v1241: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:00.962 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:00 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3657415366' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T14:38:00.962 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:00 smithi151 ceph-mon[108966]: pgmap v1241: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:00.963 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:00 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3657415366' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T14:38:00.988 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-13T14:38:00.989 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T14:38:01.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:00 smithi184 ceph-mon[92527]: pgmap v1241: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:01.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:00 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3657415366' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T14:38:01.250 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:02.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:01 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:38:02.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:01 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:02.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:38:02.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:02.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:01 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:38:02.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:01 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:02.680 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:38:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=sqlstore.transactions t=2024-09-13T14:38:02.397022985Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-13T14:38:02.844 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:02 smithi151 ceph-mon[104450]: pgmap v1242: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:02.844 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:02 smithi151 ceph-mon[108966]: pgmap v1242: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:03.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:02 smithi184 ceph-mon[92527]: pgmap v1242: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:03.489 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:38:03.489 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":1078,"fsid":"2401fc50-71da-11ef-bceb-c7b262605968","created":"2024-09-13T14:12:36.168205+0000","modified":"2024-09-13T14:37:54.701230+0000","last_up_change":"2024-09-13T14:30:14.547371+0000","last_in_change":"2024-09-13T14:17:24.771724+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:15:41.780748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:18:09.030957+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":56,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T14:19:24.791384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T14:19:25.101452+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0ad945ee-af7d-4f26-a1e9-18ed963194ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":511,"up_from":524,"up_thru":624,"down_at":512,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2137494296},{"type":"v1","addr":"172.21.15.151:6803","nonce":2137494296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2137494296},{"type":"v1","addr":"172.21.15.151:6805","nonce":2137494296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":2137494296},{"type":"v1","addr":"172.21.15.151:6809","nonce":2137494296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2137494296},{"type":"v1","addr":"172.21.15.151:6807","nonce":2137494296}]},"public_addr":"172.21.15.151:6803/2137494296","cluster_addr":"172.21.15.151:6805/2137494296","heartbeat_back_addr":"172.21.15.151:6809/2137494296","heartbeat_front_addr":"172.21.15.151:6807/2137494296","state":["exists","up"]},{"osd":1,"uuid":"09f5d705-684f-4e86-80a2-ec2292af09d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":544,"up_from":557,"up_thru":624,"down_at":545,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":3884881052},{"type":"v1","addr":"172.21.15.151:6811","nonce":3884881052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3884881052},{"type":"v1","addr":"172.21.15.151:6813","nonce":3884881052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3884881052},{"type":"v1","addr":"172.21.15.151:6817","nonce":3884881052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":3884881052},{"type":"v1","addr":"172.21.15.151:6815","nonce":3884881052}]},"public_addr":"172.21.15.151:6811/3884881052","cluster_addr":"172.21.15.151:6813/3884881052","heartbeat_back_addr":"172.21.15.151:6817/3884881052","heartbeat_front_addr":"172.21.15.151:6815/3884881052","state":["exists","up"]},{"osd":2,"uuid":"6a63f8af-ff69-46bb-830f-03ea89fc78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":578,"up_from":591,"up_thru":624,"down_at":579,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":322435630},{"type":"v1","addr":"172.21.15.151:6819","nonce":322435630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":322435630},{"type":"v1","addr":"172.21.15.151:6821","nonce":322435630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":322435630},{"type":"v1","addr":"172.21.15.151:6825","nonce":322435630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":322435630},{"type":"v1","addr":"172.21.15.151:6823","nonce":322435630}]},"public_addr":"172.21.15.151:6819/322435630","cluster_addr":"172.21.15.151:6821/322435630","heartbeat_back_addr":"172.21.15.151:6825/322435630","heartbeat_front_addr":"172.21.15.151:6823/322435630","state":["exists","up"]},{"osd":3,"uuid":"770e8334-3b3a-46bc-9782-964b2a696660","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":611,"up_from":624,"up_thru":624,"down_at":612,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2997668861},{"type":"v1","addr":"172.21.15.151:6827","nonce":2997668861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":2997668861},{"type":"v1","addr":"172.21.15.151:6829","nonce":2997668861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":2997668861},{"type":"v1","addr":"172.21.15.151:6833","nonce":2997668861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":2997668861},{"type":"v1","addr":"172.21.15.151:6831","nonce":2997668861}]},"public_addr":"172.21.15.151:6827/2997668861","cluster_addr":"172.21.15.151:6829/2997668861","heartbeat_back_addr":"172.21.15.151:6833/2997668861","heartbeat_front_addr":"172.21.15.151:6831/2997668861","state":["exists","up"]},{"osd":4,"uuid":"75e258c0-1145-4aee-86aa-db2b63082cc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":624,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6801","nonce":3275553935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6803","nonce":3275553935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6807","nonce":3275553935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6805","nonce":3275553935}]},"public_addr":"172.21.15.184:6801/3275553935","cluster_addr":"172.21.15.184:6803/3275553935","heartbeat_back_addr":"172.21.15.184:6807/3275553935","heartbeat_front_addr":"172.21.15.184:6805/3275553935","state":["exists","up"]},{"osd":5,"uuid":"f56a5366-9d22-4960-933d-340aa280fbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":624,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6809","nonce":2446419621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6811","nonce":2446419621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6815","nonce":2446419621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6813","nonce":2446419621}]},"public_addr":"172.21.15.184:6809/2446419621","cluster_addr":"172.21.15.184:6811/2446419621","heartbeat_back_addr":"172.21.15.184:6815/2446419621","heartbeat_front_addr":"172.21.15.184:6813/2446419621","state":["exists","up"]},{"osd":6,"uuid":"e9826568-a728-45ab-8180-9777fe31f93a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":612,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6817","nonce":2214837203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6819","nonce":2214837203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6823","nonce":2214837203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6821","nonce":2214837203}]},"public_addr":"172.21.15.184:6817/2214837203","cluster_addr":"172.21.15.184:6819/2214837203","heartbeat_back_addr":"172.21.15.184:6823/2214837203","heartbeat_front_addr":"172.21.15.184:6821/2214837203","state":["exists","up"]},{"osd":7,"uuid":"d76869fe-5121-4467-8b56-1aadef0eaf65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":624,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6825","nonce":3378735997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6827","nonce":3378735997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6831","nonce":3378735997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6829","nonce":3378735997}]},"public_addr":"172.21.15.184:6825/3378735997","cluster_addr":"172.21.15.184:6827/3378735997","heartbeat_back_addr":"172.21.15.184:6831/3378735997","heartbeat_front_addr":"172.21.15.184:6829/3378735997","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:14:49.835715+0000","dead_epoch":511},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:15:14.682882+0000","dead_epoch":544},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:15:39.423890+0000","dead_epoch":578},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:16:04.527460+0000","dead_epoch":611},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:16:27.283661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:16:51.893868+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:16.152592+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:39.763533+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.151:0/1783031268":"2024-09-14T14:24:01.073069+0000","172.21.15.151:0/1775071943":"2024-09-14T14:24:01.073069+0000","172.21.15.151:6801/42840":"2024-09-14T14:24:01.073069+0000","172.21.15.151:0/841256485":"2024-09-14T14:24:01.073069+0000","172.21.15.151:0/1182681341":"2024-09-14T14:13:04.761383+0000","172.21.15.151:6800/1057983082":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/2889050140":"2024-09-14T14:18:18.344788+0000","172.21.15.151:6801/4025698662":"2024-09-14T14:22:15.389006+0000","172.21.15.151:0/2800235450":"2024-09-14T14:18:18.344788+0000","172.21.15.184:0/3929228938":"2024-09-14T14:23:20.772174+0000","172.21.15.151:0/1409232061":"2024-09-14T14:24:01.073069+0000","172.21.15.151:6800/4025698662":"2024-09-14T14:22:15.389006+0000","172.21.15.184:0/1999058007":"2024-09-14T14:23:20.772174+0000","172.21.15.151:6801/1057983082":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/514781427":"2024-09-14T14:22:15.389006+0000","172.21.15.151:0/663506338":"2024-09-14T14:24:01.073069+0000","172.21.15.151:6800/3383065173":"2024-09-14T14:13:04.761383+0000","172.21.15.184:6834/205203765":"2024-09-14T14:23:20.772174+0000","172.21.15.151:0/1530349337":"2024-09-14T14:22:15.389006+0000","172.21.15.151:0/1427922964":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/2666829207":"2024-09-14T14:13:27.478847+0000","172.21.15.151:6801/2335297718":"2024-09-14T14:18:18.344788+0000","172.21.15.151:0/1351186656":"2024-09-14T14:18:18.344788+0000","172.21.15.184:6835/205203765":"2024-09-14T14:23:20.772174+0000","172.21.15.184:0/3003411098":"2024-09-14T14:23:20.772174+0000","172.21.15.151:0/4115173791":"2024-09-14T14:18:18.344788+0000","172.21.15.151:0/3689756337":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/1517910843":"2024-09-14T14:24:01.073069+0000","172.21.15.184:0/3370242044":"2024-09-14T14:23:20.772174+0000","172.21.15.151:6800/2335297718":"2024-09-14T14:18:18.344788+0000","172.21.15.151:6800/42840":"2024-09-14T14:24:01.073069+0000","172.21.15.151:0/1488764319":"2024-09-14T14:22:15.389006+0000","172.21.15.184:0/2995983270":"2024-09-14T14:23:20.772174+0000","172.21.15.151:0/3946777511":"2024-09-14T14:13:04.761383+0000","172.21.15.151:0/624042464":"2024-09-14T14:13:04.761383+0000","172.21.15.151:0/2201280183":"2024-09-14T14:22:15.389006+0000","172.21.15.151:6801/3383065173":"2024-09-14T14:13:04.761383+0000","172.21.15.184:0/3047962606":"2024-09-14T14:23:20.772174+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T14:38:04.084 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:03 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/918358449' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T14:38:04.084 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:03 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/918358449' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T14:38:04.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:03 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/918358449' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T14:38:04.391 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-13T14:38:04.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-13T14:38:04.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-13T14:38:04.393 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-13T14:38:04.393 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-13T14:38:04.393 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-13T14:38:04.394 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-13T14:38:04.394 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-13T14:38:04.743 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:38:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:38:04] "GET /metrics HTTP/1.1" 200 34154 "" "Prometheus/2.51.0" 2024-09-13T14:38:04.744 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:04 smithi151 ceph-mon[104450]: pgmap v1243: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:04.875 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:04.973 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:04.974 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:04.974 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:04.976 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:04.977 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:04.977 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:05.054 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:04 smithi151 ceph-mon[108966]: pgmap v1243: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:05.097 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:04 smithi184 ceph-mon[92527]: pgmap v1243: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:06.110 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:06.585 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:06 smithi151 ceph-mon[108966]: pgmap v1244: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:06.585 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:06 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:38:06.585 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:06 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:38:06.586 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:06 smithi151 ceph-mon[104450]: pgmap v1244: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:06.586 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:06 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:38:06.586 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:06 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:38:06.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:06 smithi184 ceph-mon[92527]: pgmap v1244: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:06.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:06 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:38:06.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:06 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:38:08.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:08 smithi151 ceph-mon[104450]: pgmap v1245: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:08.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:08 smithi151 ceph-mon[108966]: pgmap v1245: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:08.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:08 smithi184 ceph-mon[92527]: pgmap v1245: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:10.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:38:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:38:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:38:10.720 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:10 smithi151 ceph-mon[108966]: pgmap v1246: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:10.721 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:10 smithi151 ceph-mon[104450]: pgmap v1246: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:11.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:10 smithi184 ceph-mon[92527]: pgmap v1246: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:12.807 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:12 smithi151 ceph-mon[104450]: pgmap v1247: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:12.807 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:12 smithi151 ceph-mon[108966]: pgmap v1247: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:13.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:12 smithi184 ceph-mon[92527]: pgmap v1247: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:13.124 INFO:teuthology.orchestra.run.smithi151.stdout:2250562863221 2024-09-13T14:38:13.124 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-13T14:38:14.691 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:38:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:38:14] "GET /metrics HTTP/1.1" 200 34154 "" "Prometheus/2.51.0" 2024-09-13T14:38:14.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:14 smithi151 ceph-mon[104450]: pgmap v1248: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:14.950 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:14 smithi151 ceph-mon[108966]: pgmap v1248: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:14 smithi184 ceph-mon[92527]: pgmap v1248: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:16.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:16 smithi151 ceph-mon[104450]: pgmap v1249: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:16.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:16.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:16 smithi151 ceph-mon[108966]: pgmap v1249: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:16.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:17.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:16 smithi184 ceph-mon[92527]: pgmap v1249: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:17.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:18.394 INFO:teuthology.orchestra.run.smithi151.stdout:154618822916 2024-09-13T14:38:18.395 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-13T14:38:19.006 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:18 smithi151 ceph-mon[104450]: pgmap v1250: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:19.006 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:18 smithi151 ceph-mon[108966]: pgmap v1250: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:19.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:18 smithi184 ceph-mon[92527]: pgmap v1250: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:19.396 INFO:teuthology.orchestra.run.smithi151.stdout:2680059592802 2024-09-13T14:38:19.396 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-13T14:38:20.254 INFO:teuthology.orchestra.run.smithi151.stdout:2538325672041 2024-09-13T14:38:20.255 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-13T14:38:20.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:38:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:38:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:38:20.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:20 smithi151 ceph-mon[104450]: pgmap v1251: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:20.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:20 smithi151 ceph-mon[108966]: pgmap v1251: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:21.009 INFO:teuthology.orchestra.run.smithi151.stdout:2392296783984 2024-09-13T14:38:21.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-13T14:38:21.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:20 smithi184 ceph-mon[92527]: pgmap v1251: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:21.766 INFO:teuthology.orchestra.run.smithi151.stdout:128849019145 2024-09-13T14:38:21.766 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-13T14:38:22.540 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:22.540 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:22.984 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:22 smithi151 ceph-mon[104450]: pgmap v1252: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:22.985 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:22 smithi151 ceph-mon[108966]: pgmap v1252: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:23.012 INFO:teuthology.orchestra.run.smithi151.stdout:180388626687 2024-09-13T14:38:23.013 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-13T14:38:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:22 smithi184 ceph-mon[92527]: pgmap v1252: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:24.657 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:38:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:38:24] "GET /metrics HTTP/1.1" 200 34149 "" "Prometheus/2.51.0" 2024-09-13T14:38:24.682 INFO:teuthology.orchestra.run.smithi151.stdout:206158430460 2024-09-13T14:38:24.683 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-13T14:38:25.085 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:24 smithi151 ceph-mon[104450]: pgmap v1253: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:25.086 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:24 smithi151 ceph-mon[108966]: pgmap v1253: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:25.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:24 smithi184 ceph-mon[92527]: pgmap v1253: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:25.930 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:25.934 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:27.228 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:26 smithi151 ceph-mon[104450]: pgmap v1254: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:27.228 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:26 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:38:27.229 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:26 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:38:27.229 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:26 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:38:27.229 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:26 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:38:27.229 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:26 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:38:27.229 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:26 smithi151 ceph-mon[108966]: pgmap v1254: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:27.229 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:26 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:38:27.229 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:26 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:38:27.229 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:26 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:38:27.229 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:26 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:38:27.230 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:26 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:38:27.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:26 smithi184 ceph-mon[92527]: pgmap v1254: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:27.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:26 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:38:27.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:26 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:38:27.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:26 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:38:27.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:26 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:38:27.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:26 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:38:27.397 INFO:teuthology.orchestra.run.smithi151.stdout:2538325672042 2024-09-13T14:38:28.269 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:27 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/670951849' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-13T14:38:28.269 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:27 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/670951849' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-13T14:38:28.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:27 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/670951849' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-13T14:38:29.038 INFO:teuthology.orchestra.run.smithi151.stdout:2680059592804 2024-09-13T14:38:29.148 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:28 smithi151 ceph-mon[104450]: pgmap v1255: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:29.148 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:28 smithi151 ceph-mon[108966]: pgmap v1255: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:29.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:28 smithi184 ceph-mon[92527]: pgmap v1255: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:29.733 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2680059592802 got 2680059592804 for osd.3 2024-09-13T14:38:29.734 DEBUG:teuthology.parallel:result is None 2024-09-13T14:38:30.347 INFO:teuthology.orchestra.run.smithi151.stdout:154618822919 2024-09-13T14:38:30.348 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:29 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2594010704' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-13T14:38:30.348 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:29 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2594010704' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-13T14:38:30.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:29 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2594010704' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-13T14:38:30.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:38:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:38:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:38:30.413 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:30.413 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:30.413 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:30.414 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:31.042 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2538325672041 got 2538325672042 for osd.2 2024-09-13T14:38:31.042 DEBUG:teuthology.parallel:result is None 2024-09-13T14:38:31.295 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:30 smithi151 ceph-mon[104450]: pgmap v1256: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:31.295 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:30 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3560831526' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-13T14:38:31.296 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:30 smithi151 ceph-mon[108966]: pgmap v1256: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:31.296 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:30 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3560831526' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-13T14:38:31.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:30 smithi184 ceph-mon[92527]: pgmap v1256: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:31.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:30 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3560831526' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-13T14:38:32.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:32.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:32.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:32.424 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822916 got 154618822919 for osd.5 2024-09-13T14:38:32.425 DEBUG:teuthology.parallel:result is None 2024-09-13T14:38:33.045 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:33 smithi151 ceph-mon[104450]: pgmap v1257: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:33.045 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:33 smithi151 ceph-mon[108966]: pgmap v1257: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:33.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:33 smithi184 ceph-mon[92527]: pgmap v1257: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:34.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:38:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:38:34] "GET /metrics HTTP/1.1" 200 34149 "" "Prometheus/2.51.0" 2024-09-13T14:38:35.122 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:35 smithi151 ceph-mon[104450]: pgmap v1258: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:35.122 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:35 smithi151 ceph-mon[108966]: pgmap v1258: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:35.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:35 smithi184 ceph-mon[92527]: pgmap v1258: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:35.548 INFO:teuthology.orchestra.run.smithi151.stdout:180388626691 2024-09-13T14:38:36.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:36 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1236638511' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-13T14:38:36.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:36 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1236638511' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-13T14:38:36.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:36 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1236638511' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-13T14:38:36.971 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626687 got 180388626691 for osd.6 2024-09-13T14:38:36.971 DEBUG:teuthology.parallel:result is None 2024-09-13T14:38:37.253 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:37 smithi151 ceph-mon[104450]: pgmap v1259: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:37.253 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:37 smithi151 ceph-mon[108966]: pgmap v1259: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:37.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:37 smithi184 ceph-mon[92527]: pgmap v1259: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:38.014 INFO:teuthology.orchestra.run.smithi151.stdout:206158430462 2024-09-13T14:38:38.131 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:38 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2985664397' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-13T14:38:38.132 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:38 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2985664397' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-13T14:38:38.223 INFO:teuthology.orchestra.run.smithi151.stdout:2392296783988 2024-09-13T14:38:38.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:38 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2985664397' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-13T14:38:38.838 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2392296783984 got 2392296783988 for osd.1 2024-09-13T14:38:38.838 DEBUG:teuthology.parallel:result is None 2024-09-13T14:38:39.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:39 smithi151 ceph-mon[108966]: pgmap v1260: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:39.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:39 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2724709829' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-13T14:38:39.104 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:39 smithi151 ceph-mon[104450]: pgmap v1260: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:39.104 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:39 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2724709829' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-13T14:38:39.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:39 smithi184 ceph-mon[92527]: pgmap v1260: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:39.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:39 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2724709829' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-13T14:38:39.693 INFO:teuthology.orchestra.run.smithi151.stdout:2250562863226 2024-09-13T14:38:39.946 INFO:teuthology.orchestra.run.smithi151.stdout:128849019149 2024-09-13T14:38:40.264 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:40 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1163079705' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-13T14:38:40.264 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:40 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/309020626' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-13T14:38:40.264 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:40 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1163079705' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-13T14:38:40.264 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:40 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/309020626' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-13T14:38:40.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:40 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1163079705' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-13T14:38:40.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:40 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/309020626' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-13T14:38:40.354 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:38:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:38:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:38:40.580 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849019145 got 128849019149 for osd.4 2024-09-13T14:38:40.580 DEBUG:teuthology.parallel:result is None 2024-09-13T14:38:41.000 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430460 got 206158430462 for osd.7 2024-09-13T14:38:41.000 DEBUG:teuthology.parallel:result is None 2024-09-13T14:38:41.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:41 smithi184 ceph-mon[92527]: pgmap v1261: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:41.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:41 smithi151 ceph-mon[104450]: pgmap v1261: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:41.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:41 smithi151 ceph-mon[108966]: pgmap v1261: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:41.465 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2250562863221 got 2250562863226 for osd.0 2024-09-13T14:38:41.465 DEBUG:teuthology.parallel:result is None 2024-09-13T14:38:41.466 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-13T14:38:41.466 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:38:41.710 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:43.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:43 smithi151 ceph-mon[104450]: pgmap v1262: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:43.198 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:43 smithi151 ceph-mon[108966]: pgmap v1262: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:43.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:43 smithi184 ceph-mon[92527]: pgmap v1262: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:43.600 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:38:43.601 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T14:38:44.191 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1263,"stamp":"2024-09-13T14:38:43.293305+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":212,"num_write_kb":1659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1437300,"kb_used_data":76888,"kb_used_omap":149,"kb_used_meta":1360170,"kb_avail":748523916,"statfs":{"total":767960285184,"available":766488489984,"internally_reserved":0,"allocated":78733312,"data_stored":53799839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":153401,"internal_metadata":1392814279},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002569"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":1459,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016656+0000","last_change":"2024-09-13T14:30:15.579251+0000","last_active":"2024-09-13T14:37:57.016656+0000","last_peered":"2024-09-13T14:37:57.016656+0000","last_clean":"2024-09-13T14:37:57.016656+0000","last_became_active":"2024-09-13T14:30:15.578890+0000","last_became_peered":"2024-09-13T14:30:15.578890+0000","last_unstale":"2024-09-13T14:37:57.016656+0000","last_undegraded":"2024-09-13T14:37:57.016656+0000","last_fullsized":"2024-09-13T14:37:57.016656+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:51:28.380523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":1034,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679660+0000","last_change":"2024-09-13T14:29:42.541093+0000","last_active":"2024-09-13T14:37:49.679660+0000","last_peered":"2024-09-13T14:37:49.679660+0000","last_clean":"2024-09-13T14:37:49.679660+0000","last_became_active":"2024-09-13T14:29:42.540671+0000","last_became_peered":"2024-09-13T14:29:42.540671+0000","last_unstale":"2024-09-13T14:37:49.679660+0000","last_undegraded":"2024-09-13T14:37:49.679660+0000","last_fullsized":"2024-09-13T14:37:49.679660+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:45.722376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1020,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679822+0000","last_change":"2024-09-13T14:30:15.578414+0000","last_active":"2024-09-13T14:37:49.679822+0000","last_peered":"2024-09-13T14:37:49.679822+0000","last_clean":"2024-09-13T14:37:49.679822+0000","last_became_active":"2024-09-13T14:30:15.577997+0000","last_became_peered":"2024-09-13T14:30:15.577997+0000","last_unstale":"2024-09-13T14:37:49.679822+0000","last_undegraded":"2024-09-13T14:37:49.679822+0000","last_fullsized":"2024-09-13T14:37:49.679822+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:09.290677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1554,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.377071+0000","last_change":"2024-09-13T14:30:15.585508+0000","last_active":"2024-09-13T14:37:55.377071+0000","last_peered":"2024-09-13T14:37:55.377071+0000","last_clean":"2024-09-13T14:37:55.377071+0000","last_became_active":"2024-09-13T14:30:15.585206+0000","last_became_peered":"2024-09-13T14:30:15.585206+0000","last_unstale":"2024-09-13T14:37:55.377071+0000","last_undegraded":"2024-09-13T14:37:55.377071+0000","last_fullsized":"2024-09-13T14:37:55.377071+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:28:39.584397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1521,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.378348+0000","last_change":"2024-09-13T14:30:15.581992+0000","last_active":"2024-09-13T14:37:55.378348+0000","last_peered":"2024-09-13T14:37:55.378348+0000","last_clean":"2024-09-13T14:37:55.378348+0000","last_became_active":"2024-09-13T14:30:15.581692+0000","last_became_peered":"2024-09-13T14:30:15.581692+0000","last_unstale":"2024-09-13T14:37:55.378348+0000","last_undegraded":"2024-09-13T14:37:55.378348+0000","last_fullsized":"2024-09-13T14:37:55.378348+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:59.930184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1015,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.678847+0000","last_change":"2024-09-13T14:19:25.873088+0000","last_active":"2024-09-13T14:37:49.678847+0000","last_peered":"2024-09-13T14:37:49.678847+0000","last_clean":"2024-09-13T14:37:49.678847+0000","last_became_active":"2024-09-13T14:19:25.872781+0000","last_became_peered":"2024-09-13T14:19:25.872781+0000","last_unstale":"2024-09-13T14:37:49.678847+0000","last_undegraded":"2024-09-13T14:37:49.678847+0000","last_fullsized":"2024-09-13T14:37:49.678847+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:05.607382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1459,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.017200+0000","last_change":"2024-09-13T14:30:15.579629+0000","last_active":"2024-09-13T14:37:57.017200+0000","last_peered":"2024-09-13T14:37:57.017200+0000","last_clean":"2024-09-13T14:37:57.017200+0000","last_became_active":"2024-09-13T14:30:15.577913+0000","last_became_peered":"2024-09-13T14:30:15.577913+0000","last_unstale":"2024-09-13T14:37:57.017200+0000","last_undegraded":"2024-09-13T14:37:57.017200+0000","last_fullsized":"2024-09-13T14:37:57.017200+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:45:28.463082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1023,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:50.053938+0000","last_change":"2024-09-13T14:28:35.603724+0000","last_active":"2024-09-13T14:37:50.053938+0000","last_peered":"2024-09-13T14:37:50.053938+0000","last_clean":"2024-09-13T14:37:50.053938+0000","last_became_active":"2024-09-13T14:28:35.603316+0000","last_became_peered":"2024-09-13T14:28:35.603316+0000","last_unstale":"2024-09-13T14:37:50.053938+0000","last_undegraded":"2024-09-13T14:37:50.053938+0000","last_fullsized":"2024-09-13T14:37:50.053938+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:47.668223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":1033,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679017+0000","last_change":"2024-09-13T14:28:35.608607+0000","last_active":"2024-09-13T14:37:49.679017+0000","last_peered":"2024-09-13T14:37:49.679017+0000","last_clean":"2024-09-13T14:37:49.679017+0000","last_became_active":"2024-09-13T14:28:35.608254+0000","last_became_peered":"2024-09-13T14:28:35.608254+0000","last_unstale":"2024-09-13T14:37:49.679017+0000","last_undegraded":"2024-09-13T14:37:49.679017+0000","last_fullsized":"2024-09-13T14:37:49.679017+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:24.019173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1003,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679067+0000","last_change":"2024-09-13T14:19:26.887839+0000","last_active":"2024-09-13T14:37:49.679067+0000","last_peered":"2024-09-13T14:37:49.679067+0000","last_clean":"2024-09-13T14:37:49.679067+0000","last_became_active":"2024-09-13T14:19:26.887498+0000","last_became_peered":"2024-09-13T14:19:26.887498+0000","last_unstale":"2024-09-13T14:37:49.679067+0000","last_undegraded":"2024-09-13T14:37:49.679067+0000","last_fullsized":"2024-09-13T14:37:49.679067+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:24.475263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":1034,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679078+0000","last_change":"2024-09-13T14:29:08.909370+0000","last_active":"2024-09-13T14:37:49.679078+0000","last_peered":"2024-09-13T14:37:49.679078+0000","last_clean":"2024-09-13T14:37:49.679078+0000","last_became_active":"2024-09-13T14:29:08.908983+0000","last_became_peered":"2024-09-13T14:29:08.908983+0000","last_unstale":"2024-09-13T14:37:49.679078+0000","last_undegraded":"2024-09-13T14:37:49.679078+0000","last_fullsized":"2024-09-13T14:37:49.679078+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:44:43.939022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1035,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750462+0000","last_change":"2024-09-13T14:30:15.580757+0000","last_active":"2024-09-13T14:37:49.750462+0000","last_peered":"2024-09-13T14:37:49.750462+0000","last_clean":"2024-09-13T14:37:49.750462+0000","last_became_active":"2024-09-13T14:30:15.580401+0000","last_became_peered":"2024-09-13T14:30:15.580401+0000","last_unstale":"2024-09-13T14:37:49.750462+0000","last_undegraded":"2024-09-13T14:37:49.750462+0000","last_fullsized":"2024-09-13T14:37:49.750462+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:22:29.402576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":1016,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.678896+0000","last_change":"2024-09-13T14:19:25.871006+0000","last_active":"2024-09-13T14:37:49.678896+0000","last_peered":"2024-09-13T14:37:49.678896+0000","last_clean":"2024-09-13T14:37:49.678896+0000","last_became_active":"2024-09-13T14:19:25.870713+0000","last_became_peered":"2024-09-13T14:19:25.870713+0000","last_unstale":"2024-09-13T14:37:49.678896+0000","last_undegraded":"2024-09-13T14:37:49.678896+0000","last_fullsized":"2024-09-13T14:37:49.678896+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:34.372124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1014,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:50.054245+0000","last_change":"2024-09-13T14:29:08.914488+0000","last_active":"2024-09-13T14:37:50.054245+0000","last_peered":"2024-09-13T14:37:50.054245+0000","last_clean":"2024-09-13T14:37:50.054245+0000","last_became_active":"2024-09-13T14:29:08.913220+0000","last_became_peered":"2024-09-13T14:29:08.913220+0000","last_unstale":"2024-09-13T14:37:50.054245+0000","last_undegraded":"2024-09-13T14:37:50.054245+0000","last_fullsized":"2024-09-13T14:37:50.054245+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:08.316150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1021,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679699+0000","last_change":"2024-09-13T14:30:15.576953+0000","last_active":"2024-09-13T14:37:49.679699+0000","last_peered":"2024-09-13T14:37:49.679699+0000","last_clean":"2024-09-13T14:37:49.679699+0000","last_became_active":"2024-09-13T14:30:15.576614+0000","last_became_peered":"2024-09-13T14:30:15.576614+0000","last_unstale":"2024-09-13T14:37:49.679699+0000","last_undegraded":"2024-09-13T14:37:49.679699+0000","last_fullsized":"2024-09-13T14:37:49.679699+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:33:57.915263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1521,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.377210+0000","last_change":"2024-09-13T14:29:08.918531+0000","last_active":"2024-09-13T14:37:55.377210+0000","last_peered":"2024-09-13T14:37:55.377210+0000","last_clean":"2024-09-13T14:37:55.377210+0000","last_became_active":"2024-09-13T14:29:08.918264+0000","last_became_peered":"2024-09-13T14:29:08.918264+0000","last_unstale":"2024-09-13T14:37:55.377210+0000","last_undegraded":"2024-09-13T14:37:55.377210+0000","last_fullsized":"2024-09-13T14:37:55.377210+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:04.387551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":1470,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016218+0000","last_change":"2024-09-13T14:30:15.581475+0000","last_active":"2024-09-13T14:37:57.016218+0000","last_peered":"2024-09-13T14:37:57.016218+0000","last_clean":"2024-09-13T14:37:57.016218+0000","last_became_active":"2024-09-13T14:30:15.581148+0000","last_became_peered":"2024-09-13T14:30:15.581148+0000","last_unstale":"2024-09-13T14:37:57.016218+0000","last_undegraded":"2024-09-13T14:37:57.016218+0000","last_fullsized":"2024-09-13T14:37:57.016218+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:34.741839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1018,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750498+0000","last_change":"2024-09-13T14:30:15.580092+0000","last_active":"2024-09-13T14:37:49.750498+0000","last_peered":"2024-09-13T14:37:49.750498+0000","last_clean":"2024-09-13T14:37:49.750498+0000","last_became_active":"2024-09-13T14:30:15.579540+0000","last_became_peered":"2024-09-13T14:30:15.579540+0000","last_unstale":"2024-09-13T14:37:49.750498+0000","last_undegraded":"2024-09-13T14:37:49.750498+0000","last_fullsized":"2024-09-13T14:37:49.750498+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:54.416259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1015,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:50.054523+0000","last_change":"2024-09-13T14:29:08.915311+0000","last_active":"2024-09-13T14:37:50.054523+0000","last_peered":"2024-09-13T14:37:50.054523+0000","last_clean":"2024-09-13T14:37:50.054523+0000","last_became_active":"2024-09-13T14:29:08.914923+0000","last_became_peered":"2024-09-13T14:29:08.914923+0000","last_unstale":"2024-09-13T14:37:50.054523+0000","last_undegraded":"2024-09-13T14:37:50.054523+0000","last_fullsized":"2024-09-13T14:37:50.054523+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:26.311233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1035,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750768+0000","last_change":"2024-09-13T14:29:42.543672+0000","last_active":"2024-09-13T14:37:49.750768+0000","last_peered":"2024-09-13T14:37:49.750768+0000","last_clean":"2024-09-13T14:37:49.750768+0000","last_became_active":"2024-09-13T14:29:42.543329+0000","last_became_peered":"2024-09-13T14:29:42.543329+0000","last_unstale":"2024-09-13T14:37:49.750768+0000","last_undegraded":"2024-09-13T14:37:49.750768+0000","last_fullsized":"2024-09-13T14:37:49.750768+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:09:09.752737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1019,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750578+0000","last_change":"2024-09-13T14:29:42.545815+0000","last_active":"2024-09-13T14:37:49.750578+0000","last_peered":"2024-09-13T14:37:49.750578+0000","last_clean":"2024-09-13T14:37:49.750578+0000","last_became_active":"2024-09-13T14:29:42.545508+0000","last_became_peered":"2024-09-13T14:29:42.545508+0000","last_unstale":"2024-09-13T14:37:49.750578+0000","last_undegraded":"2024-09-13T14:37:49.750578+0000","last_fullsized":"2024-09-13T14:37:49.750578+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:39.678381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1018,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750764+0000","last_change":"2024-09-13T14:29:08.916416+0000","last_active":"2024-09-13T14:37:49.750764+0000","last_peered":"2024-09-13T14:37:49.750764+0000","last_clean":"2024-09-13T14:37:49.750764+0000","last_became_active":"2024-09-13T14:29:08.916106+0000","last_became_peered":"2024-09-13T14:29:08.916106+0000","last_unstale":"2024-09-13T14:37:49.750764+0000","last_undegraded":"2024-09-13T14:37:49.750764+0000","last_fullsized":"2024-09-13T14:37:49.750764+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:25.508423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1460,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016619+0000","last_change":"2024-09-13T14:30:15.577338+0000","last_active":"2024-09-13T14:37:57.016619+0000","last_peered":"2024-09-13T14:37:57.016619+0000","last_clean":"2024-09-13T14:37:57.016619+0000","last_became_active":"2024-09-13T14:30:15.577039+0000","last_became_peered":"2024-09-13T14:30:15.577039+0000","last_unstale":"2024-09-13T14:37:57.016619+0000","last_undegraded":"2024-09-13T14:37:57.016619+0000","last_fullsized":"2024-09-13T14:37:57.016619+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:43.246020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1499,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.874264+0000","last_change":"2024-09-13T14:30:15.580234+0000","last_active":"2024-09-13T14:37:54.874264+0000","last_peered":"2024-09-13T14:37:54.874264+0000","last_clean":"2024-09-13T14:37:54.874264+0000","last_became_active":"2024-09-13T14:30:15.579921+0000","last_became_peered":"2024-09-13T14:30:15.579921+0000","last_unstale":"2024-09-13T14:37:54.874264+0000","last_undegraded":"2024-09-13T14:37:54.874264+0000","last_fullsized":"2024-09-13T14:37:54.874264+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:42:39.428403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"261'100","reported_seq":1148,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.678592+0000","last_change":"2024-09-13T14:28:35.604004+0000","last_active":"2024-09-13T14:37:49.678592+0000","last_peered":"2024-09-13T14:37:49.678592+0000","last_clean":"2024-09-13T14:37:49.678592+0000","last_became_active":"2024-09-13T14:28:35.603626+0000","last_became_peered":"2024-09-13T14:28:35.603626+0000","last_unstale":"2024-09-13T14:37:49.678592+0000","last_undegraded":"2024-09-13T14:37:49.678592+0000","last_fullsized":"2024-09-13T14:37:49.678592+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:15:42.069756+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:39:52.205001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":1034,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679415+0000","last_change":"2024-09-13T14:29:42.539455+0000","last_active":"2024-09-13T14:37:49.679415+0000","last_peered":"2024-09-13T14:37:49.679415+0000","last_clean":"2024-09-13T14:37:49.679415+0000","last_became_active":"2024-09-13T14:29:42.539068+0000","last_became_peered":"2024-09-13T14:29:42.539068+0000","last_unstale":"2024-09-13T14:37:49.679415+0000","last_undegraded":"2024-09-13T14:37:49.679415+0000","last_fullsized":"2024-09-13T14:37:49.679415+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:24:06.048855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034405999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1438,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016376+0000","last_change":"2024-09-13T14:30:15.574951+0000","last_active":"2024-09-13T14:37:57.016376+0000","last_peered":"2024-09-13T14:37:57.016376+0000","last_clean":"2024-09-13T14:37:57.016376+0000","last_became_active":"2024-09-13T14:30:15.574578+0000","last_became_peered":"2024-09-13T14:30:15.574578+0000","last_unstale":"2024-09-13T14:37:57.016376+0000","last_undegraded":"2024-09-13T14:37:57.016376+0000","last_fullsized":"2024-09-13T14:37:57.016376+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:20.700591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":997,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:50.054150+0000","last_change":"2024-09-13T14:19:27.379312+0000","last_active":"2024-09-13T14:37:50.054150+0000","last_peered":"2024-09-13T14:37:50.054150+0000","last_clean":"2024-09-13T14:37:50.054150+0000","last_became_active":"2024-09-13T14:19:27.378947+0000","last_became_peered":"2024-09-13T14:19:27.378947+0000","last_unstale":"2024-09-13T14:37:50.054150+0000","last_undegraded":"2024-09-13T14:37:50.054150+0000","last_fullsized":"2024-09-13T14:37:50.054150+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:54:20.706578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1061,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.678480+0000","last_change":"2024-09-13T14:29:08.913445+0000","last_active":"2024-09-13T14:37:49.678480+0000","last_peered":"2024-09-13T14:37:49.678480+0000","last_clean":"2024-09-13T14:37:49.678480+0000","last_became_active":"2024-09-13T14:29:08.913153+0000","last_became_peered":"2024-09-13T14:29:08.913153+0000","last_unstale":"2024-09-13T14:37:49.678480+0000","last_undegraded":"2024-09-13T14:37:49.678480+0000","last_fullsized":"2024-09-13T14:37:49.678480+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:55:34.761217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1554,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.376969+0000","last_change":"2024-09-13T14:30:15.586856+0000","last_active":"2024-09-13T14:37:55.376969+0000","last_peered":"2024-09-13T14:37:55.376969+0000","last_clean":"2024-09-13T14:37:55.376969+0000","last_became_active":"2024-09-13T14:30:15.586543+0000","last_became_peered":"2024-09-13T14:30:15.586543+0000","last_unstale":"2024-09-13T14:37:55.376969+0000","last_undegraded":"2024-09-13T14:37:55.376969+0000","last_fullsized":"2024-09-13T14:37:55.376969+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:57.072216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1532,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.377110+0000","last_change":"2024-09-13T14:29:42.541652+0000","last_active":"2024-09-13T14:37:55.377110+0000","last_peered":"2024-09-13T14:37:55.377110+0000","last_clean":"2024-09-13T14:37:55.377110+0000","last_became_active":"2024-09-13T14:29:42.541341+0000","last_became_peered":"2024-09-13T14:29:42.541341+0000","last_unstale":"2024-09-13T14:37:55.377110+0000","last_undegraded":"2024-09-13T14:37:55.377110+0000","last_fullsized":"2024-09-13T14:37:55.377110+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:52.954478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1524,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016159+0000","last_change":"2024-09-13T14:30:15.579354+0000","last_active":"2024-09-13T14:37:57.016159+0000","last_peered":"2024-09-13T14:37:57.016159+0000","last_clean":"2024-09-13T14:37:57.016159+0000","last_became_active":"2024-09-13T14:30:15.579068+0000","last_became_peered":"2024-09-13T14:30:15.579068+0000","last_unstale":"2024-09-13T14:37:57.016159+0000","last_undegraded":"2024-09-13T14:37:57.016159+0000","last_fullsized":"2024-09-13T14:37:57.016159+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:47:00.887687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1498,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.874298+0000","last_change":"2024-09-13T14:29:42.539284+0000","last_active":"2024-09-13T14:37:54.874298+0000","last_peered":"2024-09-13T14:37:54.874298+0000","last_clean":"2024-09-13T14:37:54.874298+0000","last_became_active":"2024-09-13T14:29:42.538970+0000","last_became_peered":"2024-09-13T14:29:42.538970+0000","last_unstale":"2024-09-13T14:37:54.874298+0000","last_undegraded":"2024-09-13T14:37:54.874298+0000","last_fullsized":"2024-09-13T14:37:54.874298+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:04.562806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1516,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.874028+0000","last_change":"2024-09-13T14:29:08.918645+0000","last_active":"2024-09-13T14:37:54.874028+0000","last_peered":"2024-09-13T14:37:54.874028+0000","last_clean":"2024-09-13T14:37:54.874028+0000","last_became_active":"2024-09-13T14:29:08.915894+0000","last_became_peered":"2024-09-13T14:29:08.915894+0000","last_unstale":"2024-09-13T14:37:54.874028+0000","last_undegraded":"2024-09-13T14:37:54.874028+0000","last_fullsized":"2024-09-13T14:37:54.874028+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:26.651547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":1035,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679417+0000","last_change":"2024-09-13T14:29:08.913956+0000","last_active":"2024-09-13T14:37:49.679417+0000","last_peered":"2024-09-13T14:37:49.679417+0000","last_clean":"2024-09-13T14:37:49.679417+0000","last_became_active":"2024-09-13T14:29:08.913081+0000","last_became_peered":"2024-09-13T14:29:08.913081+0000","last_unstale":"2024-09-13T14:37:49.679417+0000","last_undegraded":"2024-09-13T14:37:49.679417+0000","last_fullsized":"2024-09-13T14:37:49.679417+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:28.876391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1019,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750738+0000","last_change":"2024-09-13T14:28:35.604404+0000","last_active":"2024-09-13T14:37:49.750738+0000","last_peered":"2024-09-13T14:37:49.750738+0000","last_clean":"2024-09-13T14:37:49.750738+0000","last_became_active":"2024-09-13T14:28:35.604102+0000","last_became_peered":"2024-09-13T14:28:35.604102+0000","last_unstale":"2024-09-13T14:37:49.750738+0000","last_undegraded":"2024-09-13T14:37:49.750738+0000","last_fullsized":"2024-09-13T14:37:49.750738+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:25.546010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1510,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.873748+0000","last_change":"2024-09-13T14:30:15.582454+0000","last_active":"2024-09-13T14:37:54.873748+0000","last_peered":"2024-09-13T14:37:54.873748+0000","last_clean":"2024-09-13T14:37:54.873748+0000","last_became_active":"2024-09-13T14:30:15.582143+0000","last_became_peered":"2024-09-13T14:30:15.582143+0000","last_unstale":"2024-09-13T14:37:54.873748+0000","last_undegraded":"2024-09-13T14:37:54.873748+0000","last_fullsized":"2024-09-13T14:37:54.873748+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:21:20.076270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1038,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679925+0000","last_change":"2024-09-13T14:30:15.578388+0000","last_active":"2024-09-13T14:37:49.679925+0000","last_peered":"2024-09-13T14:37:49.679925+0000","last_clean":"2024-09-13T14:37:49.679925+0000","last_became_active":"2024-09-13T14:30:15.578026+0000","last_became_peered":"2024-09-13T14:30:15.578026+0000","last_unstale":"2024-09-13T14:37:49.679925+0000","last_undegraded":"2024-09-13T14:37:49.679925+0000","last_fullsized":"2024-09-13T14:37:49.679925+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:16:07.663218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1530,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.874032+0000","last_change":"2024-09-13T14:29:08.916449+0000","last_active":"2024-09-13T14:37:54.874032+0000","last_peered":"2024-09-13T14:37:54.874032+0000","last_clean":"2024-09-13T14:37:54.874032+0000","last_became_active":"2024-09-13T14:29:08.914257+0000","last_became_peered":"2024-09-13T14:29:08.914257+0000","last_unstale":"2024-09-13T14:37:54.874032+0000","last_undegraded":"2024-09-13T14:37:54.874032+0000","last_fullsized":"2024-09-13T14:37:54.874032+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:10:32.449958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1488,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.873865+0000","last_change":"2024-09-13T14:29:08.913030+0000","last_active":"2024-09-13T14:37:54.873865+0000","last_peered":"2024-09-13T14:37:54.873865+0000","last_clean":"2024-09-13T14:37:54.873865+0000","last_became_active":"2024-09-13T14:29:08.912671+0000","last_became_peered":"2024-09-13T14:29:08.912671+0000","last_unstale":"2024-09-13T14:37:54.873865+0000","last_undegraded":"2024-09-13T14:37:54.873865+0000","last_fullsized":"2024-09-13T14:37:54.873865+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:30:41.107429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1498,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.874171+0000","last_change":"2024-09-13T14:29:42.540294+0000","last_active":"2024-09-13T14:37:54.874171+0000","last_peered":"2024-09-13T14:37:54.874171+0000","last_clean":"2024-09-13T14:37:54.874171+0000","last_became_active":"2024-09-13T14:29:42.539930+0000","last_became_peered":"2024-09-13T14:29:42.539930+0000","last_unstale":"2024-09-13T14:37:54.874171+0000","last_undegraded":"2024-09-13T14:37:54.874171+0000","last_fullsized":"2024-09-13T14:37:54.874171+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:42.999879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1044,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.678990+0000","last_change":"2024-09-13T14:28:35.603663+0000","last_active":"2024-09-13T14:37:49.678990+0000","last_peered":"2024-09-13T14:37:49.678990+0000","last_clean":"2024-09-13T14:37:49.678990+0000","last_became_active":"2024-09-13T14:28:35.603358+0000","last_became_peered":"2024-09-13T14:28:35.603358+0000","last_unstale":"2024-09-13T14:37:49.678990+0000","last_undegraded":"2024-09-13T14:37:49.678990+0000","last_fullsized":"2024-09-13T14:37:49.678990+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:24:05.842796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1510,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.378225+0000","last_change":"2024-09-13T14:28:35.611508+0000","last_active":"2024-09-13T14:37:55.378225+0000","last_peered":"2024-09-13T14:37:55.378225+0000","last_clean":"2024-09-13T14:37:55.378225+0000","last_became_active":"2024-09-13T14:28:35.610106+0000","last_became_peered":"2024-09-13T14:28:35.610106+0000","last_unstale":"2024-09-13T14:37:55.378225+0000","last_undegraded":"2024-09-13T14:37:55.378225+0000","last_fullsized":"2024-09-13T14:37:55.378225+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:11.966958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":1470,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.017208+0000","last_change":"2024-09-13T14:30:15.581728+0000","last_active":"2024-09-13T14:37:57.017208+0000","last_peered":"2024-09-13T14:37:57.017208+0000","last_clean":"2024-09-13T14:37:57.017208+0000","last_became_active":"2024-09-13T14:30:15.581303+0000","last_became_peered":"2024-09-13T14:30:15.581303+0000","last_unstale":"2024-09-13T14:37:57.017208+0000","last_undegraded":"2024-09-13T14:37:57.017208+0000","last_fullsized":"2024-09-13T14:37:57.017208+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:10.027991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1498,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.874089+0000","last_change":"2024-09-13T14:29:08.917504+0000","last_active":"2024-09-13T14:37:54.874089+0000","last_peered":"2024-09-13T14:37:54.874089+0000","last_clean":"2024-09-13T14:37:54.874089+0000","last_became_active":"2024-09-13T14:29:08.917217+0000","last_became_peered":"2024-09-13T14:29:08.917217+0000","last_unstale":"2024-09-13T14:37:54.874089+0000","last_undegraded":"2024-09-13T14:37:54.874089+0000","last_fullsized":"2024-09-13T14:37:54.874089+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:36.651514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1468,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016021+0000","last_change":"2024-09-13T14:30:15.583561+0000","last_active":"2024-09-13T14:37:57.016021+0000","last_peered":"2024-09-13T14:37:57.016021+0000","last_clean":"2024-09-13T14:37:57.016021+0000","last_became_active":"2024-09-13T14:30:15.581464+0000","last_became_peered":"2024-09-13T14:30:15.581464+0000","last_unstale":"2024-09-13T14:37:57.016021+0000","last_undegraded":"2024-09-13T14:37:57.016021+0000","last_fullsized":"2024-09-13T14:37:57.016021+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:55:19.762648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":1524,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.378393+0000","last_change":"2024-09-13T14:29:08.916436+0000","last_active":"2024-09-13T14:37:55.378393+0000","last_peered":"2024-09-13T14:37:55.378393+0000","last_clean":"2024-09-13T14:37:55.378393+0000","last_became_active":"2024-09-13T14:29:08.916126+0000","last_became_peered":"2024-09-13T14:29:08.916126+0000","last_unstale":"2024-09-13T14:37:55.378393+0000","last_undegraded":"2024-09-13T14:37:55.378393+0000","last_fullsized":"2024-09-13T14:37:55.378393+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:00.324006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1028,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:50.054381+0000","last_change":"2024-09-13T14:29:42.538860+0000","last_active":"2024-09-13T14:37:50.054381+0000","last_peered":"2024-09-13T14:37:50.054381+0000","last_clean":"2024-09-13T14:37:50.054381+0000","last_became_active":"2024-09-13T14:29:42.538608+0000","last_became_peered":"2024-09-13T14:29:42.538608+0000","last_unstale":"2024-09-13T14:37:50.054381+0000","last_undegraded":"2024-09-13T14:37:50.054381+0000","last_fullsized":"2024-09-13T14:37:50.054381+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:20.070703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1001,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750320+0000","last_change":"2024-09-13T14:19:26.131609+0000","last_active":"2024-09-13T14:37:49.750320+0000","last_peered":"2024-09-13T14:37:49.750320+0000","last_clean":"2024-09-13T14:37:49.750320+0000","last_became_active":"2024-09-13T14:19:26.131029+0000","last_became_peered":"2024-09-13T14:19:26.131029+0000","last_unstale":"2024-09-13T14:37:49.750320+0000","last_undegraded":"2024-09-13T14:37:49.750320+0000","last_fullsized":"2024-09-13T14:37:49.750320+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:57.933696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1018,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750644+0000","last_change":"2024-09-13T14:29:08.911835+0000","last_active":"2024-09-13T14:37:49.750644+0000","last_peered":"2024-09-13T14:37:49.750644+0000","last_clean":"2024-09-13T14:37:49.750644+0000","last_became_active":"2024-09-13T14:29:08.911542+0000","last_became_peered":"2024-09-13T14:29:08.911542+0000","last_unstale":"2024-09-13T14:37:49.750644+0000","last_undegraded":"2024-09-13T14:37:49.750644+0000","last_fullsized":"2024-09-13T14:37:49.750644+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:30.820674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":1024,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750608+0000","last_change":"2024-09-13T14:19:25.870605+0000","last_active":"2024-09-13T14:37:49.750608+0000","last_peered":"2024-09-13T14:37:49.750608+0000","last_clean":"2024-09-13T14:37:49.750608+0000","last_became_active":"2024-09-13T14:19:25.870282+0000","last_became_peered":"2024-09-13T14:19:25.870282+0000","last_unstale":"2024-09-13T14:37:49.750608+0000","last_undegraded":"2024-09-13T14:37:49.750608+0000","last_fullsized":"2024-09-13T14:37:49.750608+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:52.514915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1499,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.873806+0000","last_change":"2024-09-13T14:29:42.537787+0000","last_active":"2024-09-13T14:37:54.873806+0000","last_peered":"2024-09-13T14:37:54.873806+0000","last_clean":"2024-09-13T14:37:54.873806+0000","last_became_active":"2024-09-13T14:29:42.537499+0000","last_became_peered":"2024-09-13T14:29:42.537499+0000","last_unstale":"2024-09-13T14:37:54.873806+0000","last_undegraded":"2024-09-13T14:37:54.873806+0000","last_fullsized":"2024-09-13T14:37:54.873806+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:15:30.508656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":1035,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.678750+0000","last_change":"2024-09-13T14:30:15.579260+0000","last_active":"2024-09-13T14:37:49.678750+0000","last_peered":"2024-09-13T14:37:49.678750+0000","last_clean":"2024-09-13T14:37:49.678750+0000","last_became_active":"2024-09-13T14:30:15.578897+0000","last_became_peered":"2024-09-13T14:30:15.578897+0000","last_unstale":"2024-09-13T14:37:49.678750+0000","last_undegraded":"2024-09-13T14:37:49.678750+0000","last_fullsized":"2024-09-13T14:37:49.678750+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:56.737849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1021,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.680020+0000","last_change":"2024-09-13T14:29:08.910821+0000","last_active":"2024-09-13T14:37:49.680020+0000","last_peered":"2024-09-13T14:37:49.680020+0000","last_clean":"2024-09-13T14:37:49.680020+0000","last_became_active":"2024-09-13T14:29:08.910509+0000","last_became_peered":"2024-09-13T14:29:08.910509+0000","last_unstale":"2024-09-13T14:37:49.680020+0000","last_undegraded":"2024-09-13T14:37:49.680020+0000","last_fullsized":"2024-09-13T14:37:49.680020+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:59.495053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1438,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.015920+0000","last_change":"2024-09-13T14:30:15.576921+0000","last_active":"2024-09-13T14:37:57.015920+0000","last_peered":"2024-09-13T14:37:57.015920+0000","last_clean":"2024-09-13T14:37:57.015920+0000","last_became_active":"2024-09-13T14:30:15.576607+0000","last_became_peered":"2024-09-13T14:30:15.576607+0000","last_unstale":"2024-09-13T14:37:57.015920+0000","last_undegraded":"2024-09-13T14:37:57.015920+0000","last_fullsized":"2024-09-13T14:37:57.015920+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:35.162460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1554,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.379052+0000","last_change":"2024-09-13T14:30:15.585712+0000","last_active":"2024-09-13T14:37:55.379052+0000","last_peered":"2024-09-13T14:37:55.379052+0000","last_clean":"2024-09-13T14:37:55.379052+0000","last_became_active":"2024-09-13T14:30:15.585147+0000","last_became_peered":"2024-09-13T14:30:15.585147+0000","last_unstale":"2024-09-13T14:37:55.379052+0000","last_undegraded":"2024-09-13T14:37:55.379052+0000","last_fullsized":"2024-09-13T14:37:55.379052+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:40:33.859429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1461,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.015890+0000","last_change":"2024-09-13T14:30:15.576306+0000","last_active":"2024-09-13T14:37:57.015890+0000","last_peered":"2024-09-13T14:37:57.015890+0000","last_clean":"2024-09-13T14:37:57.015890+0000","last_became_active":"2024-09-13T14:30:15.575931+0000","last_became_peered":"2024-09-13T14:30:15.575931+0000","last_unstale":"2024-09-13T14:37:57.015890+0000","last_undegraded":"2024-09-13T14:37:57.015890+0000","last_fullsized":"2024-09-13T14:37:57.015890+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:54.078309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1014,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:50.054296+0000","last_change":"2024-09-13T14:29:08.913672+0000","last_active":"2024-09-13T14:37:50.054296+0000","last_peered":"2024-09-13T14:37:50.054296+0000","last_clean":"2024-09-13T14:37:50.054296+0000","last_became_active":"2024-09-13T14:29:08.913360+0000","last_became_peered":"2024-09-13T14:29:08.913360+0000","last_unstale":"2024-09-13T14:37:50.054296+0000","last_undegraded":"2024-09-13T14:37:50.054296+0000","last_fullsized":"2024-09-13T14:37:50.054296+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:45.688163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1477,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016529+0000","last_change":"2024-09-13T14:30:15.583872+0000","last_active":"2024-09-13T14:37:57.016529+0000","last_peered":"2024-09-13T14:37:57.016529+0000","last_clean":"2024-09-13T14:37:57.016529+0000","last_became_active":"2024-09-13T14:30:15.581869+0000","last_became_peered":"2024-09-13T14:30:15.581869+0000","last_unstale":"2024-09-13T14:37:57.016529+0000","last_undegraded":"2024-09-13T14:37:57.016529+0000","last_fullsized":"2024-09-13T14:37:57.016529+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:10:37.393352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1522,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.378219+0000","last_change":"2024-09-13T14:30:15.586018+0000","last_active":"2024-09-13T14:37:55.378219+0000","last_peered":"2024-09-13T14:37:55.378219+0000","last_clean":"2024-09-13T14:37:55.378219+0000","last_became_active":"2024-09-13T14:30:15.585616+0000","last_became_peered":"2024-09-13T14:30:15.585616+0000","last_unstale":"2024-09-13T14:37:55.378219+0000","last_undegraded":"2024-09-13T14:37:55.378219+0000","last_fullsized":"2024-09-13T14:37:55.378219+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:46.836406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1497,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.873668+0000","last_change":"2024-09-13T14:30:15.579398+0000","last_active":"2024-09-13T14:37:54.873668+0000","last_peered":"2024-09-13T14:37:54.873668+0000","last_clean":"2024-09-13T14:37:54.873668+0000","last_became_active":"2024-09-13T14:30:15.579062+0000","last_became_peered":"2024-09-13T14:30:15.579062+0000","last_unstale":"2024-09-13T14:37:54.873668+0000","last_undegraded":"2024-09-13T14:37:54.873668+0000","last_fullsized":"2024-09-13T14:37:54.873668+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:06:39.532417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1459,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016735+0000","last_change":"2024-09-13T14:30:15.576778+0000","last_active":"2024-09-13T14:37:57.016735+0000","last_peered":"2024-09-13T14:37:57.016735+0000","last_clean":"2024-09-13T14:37:57.016735+0000","last_became_active":"2024-09-13T14:30:15.576434+0000","last_became_peered":"2024-09-13T14:30:15.576434+0000","last_unstale":"2024-09-13T14:37:57.016735+0000","last_undegraded":"2024-09-13T14:37:57.016735+0000","last_fullsized":"2024-09-13T14:37:57.016735+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:54:06.503790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":1037,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750337+0000","last_change":"2024-09-13T14:29:42.537809+0000","last_active":"2024-09-13T14:37:49.750337+0000","last_peered":"2024-09-13T14:37:49.750337+0000","last_clean":"2024-09-13T14:37:49.750337+0000","last_became_active":"2024-09-13T14:29:42.537482+0000","last_became_peered":"2024-09-13T14:29:42.537482+0000","last_unstale":"2024-09-13T14:37:49.750337+0000","last_undegraded":"2024-09-13T14:37:49.750337+0000","last_fullsized":"2024-09-13T14:37:49.750337+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:19.631210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1500,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016353+0000","last_change":"2024-09-13T14:30:15.578675+0000","last_active":"2024-09-13T14:37:57.016353+0000","last_peered":"2024-09-13T14:37:57.016353+0000","last_clean":"2024-09-13T14:37:57.016353+0000","last_became_active":"2024-09-13T14:30:15.578362+0000","last_became_peered":"2024-09-13T14:30:15.578362+0000","last_unstale":"2024-09-13T14:37:57.016353+0000","last_undegraded":"2024-09-13T14:37:57.016353+0000","last_fullsized":"2024-09-13T14:37:57.016353+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:30:47.445865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":1513,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.378569+0000","last_change":"2024-09-13T14:28:35.612469+0000","last_active":"2024-09-13T14:37:55.378569+0000","last_peered":"2024-09-13T14:37:55.378569+0000","last_clean":"2024-09-13T14:37:55.378569+0000","last_became_active":"2024-09-13T14:28:35.612093+0000","last_became_peered":"2024-09-13T14:28:35.612093+0000","last_unstale":"2024-09-13T14:37:55.378569+0000","last_undegraded":"2024-09-13T14:37:55.378569+0000","last_fullsized":"2024-09-13T14:37:55.378569+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:51.022663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1028,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750501+0000","last_change":"2024-09-13T14:30:15.580568+0000","last_active":"2024-09-13T14:37:49.750501+0000","last_peered":"2024-09-13T14:37:49.750501+0000","last_clean":"2024-09-13T14:37:49.750501+0000","last_became_active":"2024-09-13T14:30:15.580186+0000","last_became_peered":"2024-09-13T14:30:15.580186+0000","last_unstale":"2024-09-13T14:37:49.750501+0000","last_undegraded":"2024-09-13T14:37:49.750501+0000","last_fullsized":"2024-09-13T14:37:49.750501+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:16:00.781680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":1023,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679624+0000","last_change":"2024-09-13T14:29:08.912819+0000","last_active":"2024-09-13T14:37:49.679624+0000","last_peered":"2024-09-13T14:37:49.679624+0000","last_clean":"2024-09-13T14:37:49.679624+0000","last_became_active":"2024-09-13T14:29:08.912464+0000","last_became_peered":"2024-09-13T14:29:08.912464+0000","last_unstale":"2024-09-13T14:37:49.679624+0000","last_undegraded":"2024-09-13T14:37:49.679624+0000","last_fullsized":"2024-09-13T14:37:49.679624+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:09.753940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1035,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750210+0000","last_change":"2024-09-13T14:30:15.580011+0000","last_active":"2024-09-13T14:37:49.750210+0000","last_peered":"2024-09-13T14:37:49.750210+0000","last_clean":"2024-09-13T14:37:49.750210+0000","last_became_active":"2024-09-13T14:30:15.579610+0000","last_became_peered":"2024-09-13T14:30:15.579610+0000","last_unstale":"2024-09-13T14:37:49.750210+0000","last_undegraded":"2024-09-13T14:37:49.750210+0000","last_fullsized":"2024-09-13T14:37:49.750210+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:11:02.733771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":1006,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679901+0000","last_change":"2024-09-13T14:19:25.873514+0000","last_active":"2024-09-13T14:37:49.679901+0000","last_peered":"2024-09-13T14:37:49.679901+0000","last_clean":"2024-09-13T14:37:49.679901+0000","last_became_active":"2024-09-13T14:19:25.873170+0000","last_became_peered":"2024-09-13T14:19:25.873170+0000","last_unstale":"2024-09-13T14:37:49.679901+0000","last_undegraded":"2024-09-13T14:37:49.679901+0000","last_fullsized":"2024-09-13T14:37:49.679901+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:48.298530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1468,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016312+0000","last_change":"2024-09-13T14:30:15.583672+0000","last_active":"2024-09-13T14:37:57.016312+0000","last_peered":"2024-09-13T14:37:57.016312+0000","last_clean":"2024-09-13T14:37:57.016312+0000","last_became_active":"2024-09-13T14:30:15.583160+0000","last_became_peered":"2024-09-13T14:30:15.583160+0000","last_unstale":"2024-09-13T14:37:57.016312+0000","last_undegraded":"2024-09-13T14:37:57.016312+0000","last_fullsized":"2024-09-13T14:37:57.016312+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:54.303579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":1454,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.017187+0000","last_change":"2024-09-13T14:30:15.576496+0000","last_active":"2024-09-13T14:37:57.017187+0000","last_peered":"2024-09-13T14:37:57.017187+0000","last_clean":"2024-09-13T14:37:57.017187+0000","last_became_active":"2024-09-13T14:30:15.576137+0000","last_became_peered":"2024-09-13T14:30:15.576137+0000","last_unstale":"2024-09-13T14:37:57.017187+0000","last_undegraded":"2024-09-13T14:37:57.017187+0000","last_fullsized":"2024-09-13T14:37:57.017187+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:24.316047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1028,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750042+0000","last_change":"2024-09-13T14:29:42.539907+0000","last_active":"2024-09-13T14:37:49.750042+0000","last_peered":"2024-09-13T14:37:49.750042+0000","last_clean":"2024-09-13T14:37:49.750042+0000","last_became_active":"2024-09-13T14:29:42.539583+0000","last_became_peered":"2024-09-13T14:29:42.539583+0000","last_unstale":"2024-09-13T14:37:49.750042+0000","last_undegraded":"2024-09-13T14:37:49.750042+0000","last_fullsized":"2024-09-13T14:37:49.750042+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:02:04.628447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":1527,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.379125+0000","last_change":"2024-09-13T14:29:42.541263+0000","last_active":"2024-09-13T14:37:55.379125+0000","last_peered":"2024-09-13T14:37:55.379125+0000","last_clean":"2024-09-13T14:37:55.379125+0000","last_became_active":"2024-09-13T14:29:42.540873+0000","last_became_peered":"2024-09-13T14:29:42.540873+0000","last_unstale":"2024-09-13T14:37:55.379125+0000","last_undegraded":"2024-09-13T14:37:55.379125+0000","last_fullsized":"2024-09-13T14:37:55.379125+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:20.611206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430464,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311504,"kb_used_data":9988,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93433648,"statfs":{"total":95995035648,"available":95676055552,"internally_reserved":0,"allocated":10227712,"data_stored":7096822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626693,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":319816,"kb_used_data":9980,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93425336,"statfs":{"total":95995035648,"available":95667544064,"internally_reserved":0,"allocated":10219520,"data_stored":7096703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822922,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306748,"kb_used_data":9392,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93438404,"statfs":{"total":95995035648,"available":95680925696,"internally_reserved":0,"allocated":9617408,"data_stored":6502153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019150,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":315080,"kb_used_data":9404,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93430072,"statfs":{"total":95995035648,"available":95672393728,"internally_reserved":0,"allocated":9629696,"data_stored":6506384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059592807,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":45428,"kb_used_data":9384,"kb_used_omap":26,"kb_used_meta":36005,"kb_avail":93699724,"statfs":{"total":95995035648,"available":95948517376,"internally_reserved":0,"allocated":9609216,"data_stored":6501604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":36869280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":591,"seq":2538325672046,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37304,"kb_used_data":9388,"kb_used_omap":41,"kb_used_meta":27862,"kb_avail":93707848,"statfs":{"total":95995035648,"available":95956836352,"internally_reserved":0,"allocated":9613312,"data_stored":6502126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42651,"internal_metadata":28531045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":557,"seq":2392296783989,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37232,"kb_used_data":9384,"kb_used_omap":32,"kb_used_meta":27807,"kb_avail":93707920,"statfs":{"total":95995035648,"available":95956910080,"internally_reserved":0,"allocated":9609216,"data_stored":6501549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33752,"internal_metadata":28474408},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":524,"seq":2250562863227,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64188,"kb_used_data":9968,"kb_used_omap":41,"kb_used_meta":54166,"kb_avail":93680964,"statfs":{"total":95995035648,"available":95929307136,"internally_reserved":0,"allocated":10207232,"data_stored":7092498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42694,"internal_metadata":55466298},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:38:44.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:38:44.444 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:44.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:38:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:38:44] "GET /metrics HTTP/1.1" 200 34149 "" "Prometheus/2.51.0" 2024-09-13T14:38:45.340 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:45 smithi151 ceph-mon[104450]: pgmap v1263: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:45.341 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:45 smithi151 ceph-mon[104450]: from='client.45080 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:38:45.341 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:45 smithi151 ceph-mon[108966]: pgmap v1263: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:45.341 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:45 smithi151 ceph-mon[108966]: from='client.45080 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:38:45.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:45 smithi184 ceph-mon[92527]: pgmap v1263: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:45.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:45 smithi184 ceph-mon[92527]: from='client.45080 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:38:46.326 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:38:46.326 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T14:38:46.967 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1264,"stamp":"2024-09-13T14:38:45.293769+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":212,"num_write_kb":1659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1437300,"kb_used_data":76888,"kb_used_omap":149,"kb_used_meta":1360170,"kb_avail":748523916,"statfs":{"total":767960285184,"available":766488489984,"internally_reserved":0,"allocated":78733312,"data_stored":53799839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":153401,"internal_metadata":1392814279},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002710"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":1459,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016656+0000","last_change":"2024-09-13T14:30:15.579251+0000","last_active":"2024-09-13T14:37:57.016656+0000","last_peered":"2024-09-13T14:37:57.016656+0000","last_clean":"2024-09-13T14:37:57.016656+0000","last_became_active":"2024-09-13T14:30:15.578890+0000","last_became_peered":"2024-09-13T14:30:15.578890+0000","last_unstale":"2024-09-13T14:37:57.016656+0000","last_undegraded":"2024-09-13T14:37:57.016656+0000","last_fullsized":"2024-09-13T14:37:57.016656+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:51:28.380523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":1034,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679660+0000","last_change":"2024-09-13T14:29:42.541093+0000","last_active":"2024-09-13T14:37:49.679660+0000","last_peered":"2024-09-13T14:37:49.679660+0000","last_clean":"2024-09-13T14:37:49.679660+0000","last_became_active":"2024-09-13T14:29:42.540671+0000","last_became_peered":"2024-09-13T14:29:42.540671+0000","last_unstale":"2024-09-13T14:37:49.679660+0000","last_undegraded":"2024-09-13T14:37:49.679660+0000","last_fullsized":"2024-09-13T14:37:49.679660+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:45.722376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1020,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679822+0000","last_change":"2024-09-13T14:30:15.578414+0000","last_active":"2024-09-13T14:37:49.679822+0000","last_peered":"2024-09-13T14:37:49.679822+0000","last_clean":"2024-09-13T14:37:49.679822+0000","last_became_active":"2024-09-13T14:30:15.577997+0000","last_became_peered":"2024-09-13T14:30:15.577997+0000","last_unstale":"2024-09-13T14:37:49.679822+0000","last_undegraded":"2024-09-13T14:37:49.679822+0000","last_fullsized":"2024-09-13T14:37:49.679822+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:09.290677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1554,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.377071+0000","last_change":"2024-09-13T14:30:15.585508+0000","last_active":"2024-09-13T14:37:55.377071+0000","last_peered":"2024-09-13T14:37:55.377071+0000","last_clean":"2024-09-13T14:37:55.377071+0000","last_became_active":"2024-09-13T14:30:15.585206+0000","last_became_peered":"2024-09-13T14:30:15.585206+0000","last_unstale":"2024-09-13T14:37:55.377071+0000","last_undegraded":"2024-09-13T14:37:55.377071+0000","last_fullsized":"2024-09-13T14:37:55.377071+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:28:39.584397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1521,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.378348+0000","last_change":"2024-09-13T14:30:15.581992+0000","last_active":"2024-09-13T14:37:55.378348+0000","last_peered":"2024-09-13T14:37:55.378348+0000","last_clean":"2024-09-13T14:37:55.378348+0000","last_became_active":"2024-09-13T14:30:15.581692+0000","last_became_peered":"2024-09-13T14:30:15.581692+0000","last_unstale":"2024-09-13T14:37:55.378348+0000","last_undegraded":"2024-09-13T14:37:55.378348+0000","last_fullsized":"2024-09-13T14:37:55.378348+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:59.930184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1015,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.678847+0000","last_change":"2024-09-13T14:19:25.873088+0000","last_active":"2024-09-13T14:37:49.678847+0000","last_peered":"2024-09-13T14:37:49.678847+0000","last_clean":"2024-09-13T14:37:49.678847+0000","last_became_active":"2024-09-13T14:19:25.872781+0000","last_became_peered":"2024-09-13T14:19:25.872781+0000","last_unstale":"2024-09-13T14:37:49.678847+0000","last_undegraded":"2024-09-13T14:37:49.678847+0000","last_fullsized":"2024-09-13T14:37:49.678847+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:05.607382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1459,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.017200+0000","last_change":"2024-09-13T14:30:15.579629+0000","last_active":"2024-09-13T14:37:57.017200+0000","last_peered":"2024-09-13T14:37:57.017200+0000","last_clean":"2024-09-13T14:37:57.017200+0000","last_became_active":"2024-09-13T14:30:15.577913+0000","last_became_peered":"2024-09-13T14:30:15.577913+0000","last_unstale":"2024-09-13T14:37:57.017200+0000","last_undegraded":"2024-09-13T14:37:57.017200+0000","last_fullsized":"2024-09-13T14:37:57.017200+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:45:28.463082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1023,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:50.053938+0000","last_change":"2024-09-13T14:28:35.603724+0000","last_active":"2024-09-13T14:37:50.053938+0000","last_peered":"2024-09-13T14:37:50.053938+0000","last_clean":"2024-09-13T14:37:50.053938+0000","last_became_active":"2024-09-13T14:28:35.603316+0000","last_became_peered":"2024-09-13T14:28:35.603316+0000","last_unstale":"2024-09-13T14:37:50.053938+0000","last_undegraded":"2024-09-13T14:37:50.053938+0000","last_fullsized":"2024-09-13T14:37:50.053938+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:47.668223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":1033,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679017+0000","last_change":"2024-09-13T14:28:35.608607+0000","last_active":"2024-09-13T14:37:49.679017+0000","last_peered":"2024-09-13T14:37:49.679017+0000","last_clean":"2024-09-13T14:37:49.679017+0000","last_became_active":"2024-09-13T14:28:35.608254+0000","last_became_peered":"2024-09-13T14:28:35.608254+0000","last_unstale":"2024-09-13T14:37:49.679017+0000","last_undegraded":"2024-09-13T14:37:49.679017+0000","last_fullsized":"2024-09-13T14:37:49.679017+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:24.019173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1003,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679067+0000","last_change":"2024-09-13T14:19:26.887839+0000","last_active":"2024-09-13T14:37:49.679067+0000","last_peered":"2024-09-13T14:37:49.679067+0000","last_clean":"2024-09-13T14:37:49.679067+0000","last_became_active":"2024-09-13T14:19:26.887498+0000","last_became_peered":"2024-09-13T14:19:26.887498+0000","last_unstale":"2024-09-13T14:37:49.679067+0000","last_undegraded":"2024-09-13T14:37:49.679067+0000","last_fullsized":"2024-09-13T14:37:49.679067+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:24.475263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":1034,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679078+0000","last_change":"2024-09-13T14:29:08.909370+0000","last_active":"2024-09-13T14:37:49.679078+0000","last_peered":"2024-09-13T14:37:49.679078+0000","last_clean":"2024-09-13T14:37:49.679078+0000","last_became_active":"2024-09-13T14:29:08.908983+0000","last_became_peered":"2024-09-13T14:29:08.908983+0000","last_unstale":"2024-09-13T14:37:49.679078+0000","last_undegraded":"2024-09-13T14:37:49.679078+0000","last_fullsized":"2024-09-13T14:37:49.679078+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:44:43.939022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1035,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750462+0000","last_change":"2024-09-13T14:30:15.580757+0000","last_active":"2024-09-13T14:37:49.750462+0000","last_peered":"2024-09-13T14:37:49.750462+0000","last_clean":"2024-09-13T14:37:49.750462+0000","last_became_active":"2024-09-13T14:30:15.580401+0000","last_became_peered":"2024-09-13T14:30:15.580401+0000","last_unstale":"2024-09-13T14:37:49.750462+0000","last_undegraded":"2024-09-13T14:37:49.750462+0000","last_fullsized":"2024-09-13T14:37:49.750462+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:22:29.402576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":1016,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.678896+0000","last_change":"2024-09-13T14:19:25.871006+0000","last_active":"2024-09-13T14:37:49.678896+0000","last_peered":"2024-09-13T14:37:49.678896+0000","last_clean":"2024-09-13T14:37:49.678896+0000","last_became_active":"2024-09-13T14:19:25.870713+0000","last_became_peered":"2024-09-13T14:19:25.870713+0000","last_unstale":"2024-09-13T14:37:49.678896+0000","last_undegraded":"2024-09-13T14:37:49.678896+0000","last_fullsized":"2024-09-13T14:37:49.678896+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:34.372124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1014,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:50.054245+0000","last_change":"2024-09-13T14:29:08.914488+0000","last_active":"2024-09-13T14:37:50.054245+0000","last_peered":"2024-09-13T14:37:50.054245+0000","last_clean":"2024-09-13T14:37:50.054245+0000","last_became_active":"2024-09-13T14:29:08.913220+0000","last_became_peered":"2024-09-13T14:29:08.913220+0000","last_unstale":"2024-09-13T14:37:50.054245+0000","last_undegraded":"2024-09-13T14:37:50.054245+0000","last_fullsized":"2024-09-13T14:37:50.054245+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:08.316150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1021,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679699+0000","last_change":"2024-09-13T14:30:15.576953+0000","last_active":"2024-09-13T14:37:49.679699+0000","last_peered":"2024-09-13T14:37:49.679699+0000","last_clean":"2024-09-13T14:37:49.679699+0000","last_became_active":"2024-09-13T14:30:15.576614+0000","last_became_peered":"2024-09-13T14:30:15.576614+0000","last_unstale":"2024-09-13T14:37:49.679699+0000","last_undegraded":"2024-09-13T14:37:49.679699+0000","last_fullsized":"2024-09-13T14:37:49.679699+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:33:57.915263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1521,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.377210+0000","last_change":"2024-09-13T14:29:08.918531+0000","last_active":"2024-09-13T14:37:55.377210+0000","last_peered":"2024-09-13T14:37:55.377210+0000","last_clean":"2024-09-13T14:37:55.377210+0000","last_became_active":"2024-09-13T14:29:08.918264+0000","last_became_peered":"2024-09-13T14:29:08.918264+0000","last_unstale":"2024-09-13T14:37:55.377210+0000","last_undegraded":"2024-09-13T14:37:55.377210+0000","last_fullsized":"2024-09-13T14:37:55.377210+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:04.387551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":1470,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016218+0000","last_change":"2024-09-13T14:30:15.581475+0000","last_active":"2024-09-13T14:37:57.016218+0000","last_peered":"2024-09-13T14:37:57.016218+0000","last_clean":"2024-09-13T14:37:57.016218+0000","last_became_active":"2024-09-13T14:30:15.581148+0000","last_became_peered":"2024-09-13T14:30:15.581148+0000","last_unstale":"2024-09-13T14:37:57.016218+0000","last_undegraded":"2024-09-13T14:37:57.016218+0000","last_fullsized":"2024-09-13T14:37:57.016218+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:34.741839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1018,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750498+0000","last_change":"2024-09-13T14:30:15.580092+0000","last_active":"2024-09-13T14:37:49.750498+0000","last_peered":"2024-09-13T14:37:49.750498+0000","last_clean":"2024-09-13T14:37:49.750498+0000","last_became_active":"2024-09-13T14:30:15.579540+0000","last_became_peered":"2024-09-13T14:30:15.579540+0000","last_unstale":"2024-09-13T14:37:49.750498+0000","last_undegraded":"2024-09-13T14:37:49.750498+0000","last_fullsized":"2024-09-13T14:37:49.750498+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:54.416259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1015,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:50.054523+0000","last_change":"2024-09-13T14:29:08.915311+0000","last_active":"2024-09-13T14:37:50.054523+0000","last_peered":"2024-09-13T14:37:50.054523+0000","last_clean":"2024-09-13T14:37:50.054523+0000","last_became_active":"2024-09-13T14:29:08.914923+0000","last_became_peered":"2024-09-13T14:29:08.914923+0000","last_unstale":"2024-09-13T14:37:50.054523+0000","last_undegraded":"2024-09-13T14:37:50.054523+0000","last_fullsized":"2024-09-13T14:37:50.054523+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:26.311233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1035,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750768+0000","last_change":"2024-09-13T14:29:42.543672+0000","last_active":"2024-09-13T14:37:49.750768+0000","last_peered":"2024-09-13T14:37:49.750768+0000","last_clean":"2024-09-13T14:37:49.750768+0000","last_became_active":"2024-09-13T14:29:42.543329+0000","last_became_peered":"2024-09-13T14:29:42.543329+0000","last_unstale":"2024-09-13T14:37:49.750768+0000","last_undegraded":"2024-09-13T14:37:49.750768+0000","last_fullsized":"2024-09-13T14:37:49.750768+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:09:09.752737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1019,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750578+0000","last_change":"2024-09-13T14:29:42.545815+0000","last_active":"2024-09-13T14:37:49.750578+0000","last_peered":"2024-09-13T14:37:49.750578+0000","last_clean":"2024-09-13T14:37:49.750578+0000","last_became_active":"2024-09-13T14:29:42.545508+0000","last_became_peered":"2024-09-13T14:29:42.545508+0000","last_unstale":"2024-09-13T14:37:49.750578+0000","last_undegraded":"2024-09-13T14:37:49.750578+0000","last_fullsized":"2024-09-13T14:37:49.750578+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:39.678381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1018,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750764+0000","last_change":"2024-09-13T14:29:08.916416+0000","last_active":"2024-09-13T14:37:49.750764+0000","last_peered":"2024-09-13T14:37:49.750764+0000","last_clean":"2024-09-13T14:37:49.750764+0000","last_became_active":"2024-09-13T14:29:08.916106+0000","last_became_peered":"2024-09-13T14:29:08.916106+0000","last_unstale":"2024-09-13T14:37:49.750764+0000","last_undegraded":"2024-09-13T14:37:49.750764+0000","last_fullsized":"2024-09-13T14:37:49.750764+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:25.508423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1460,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016619+0000","last_change":"2024-09-13T14:30:15.577338+0000","last_active":"2024-09-13T14:37:57.016619+0000","last_peered":"2024-09-13T14:37:57.016619+0000","last_clean":"2024-09-13T14:37:57.016619+0000","last_became_active":"2024-09-13T14:30:15.577039+0000","last_became_peered":"2024-09-13T14:30:15.577039+0000","last_unstale":"2024-09-13T14:37:57.016619+0000","last_undegraded":"2024-09-13T14:37:57.016619+0000","last_fullsized":"2024-09-13T14:37:57.016619+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:43.246020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1499,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.874264+0000","last_change":"2024-09-13T14:30:15.580234+0000","last_active":"2024-09-13T14:37:54.874264+0000","last_peered":"2024-09-13T14:37:54.874264+0000","last_clean":"2024-09-13T14:37:54.874264+0000","last_became_active":"2024-09-13T14:30:15.579921+0000","last_became_peered":"2024-09-13T14:30:15.579921+0000","last_unstale":"2024-09-13T14:37:54.874264+0000","last_undegraded":"2024-09-13T14:37:54.874264+0000","last_fullsized":"2024-09-13T14:37:54.874264+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:42:39.428403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"261'100","reported_seq":1148,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.678592+0000","last_change":"2024-09-13T14:28:35.604004+0000","last_active":"2024-09-13T14:37:49.678592+0000","last_peered":"2024-09-13T14:37:49.678592+0000","last_clean":"2024-09-13T14:37:49.678592+0000","last_became_active":"2024-09-13T14:28:35.603626+0000","last_became_peered":"2024-09-13T14:28:35.603626+0000","last_unstale":"2024-09-13T14:37:49.678592+0000","last_undegraded":"2024-09-13T14:37:49.678592+0000","last_fullsized":"2024-09-13T14:37:49.678592+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:15:42.069756+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:39:52.205001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":1034,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679415+0000","last_change":"2024-09-13T14:29:42.539455+0000","last_active":"2024-09-13T14:37:49.679415+0000","last_peered":"2024-09-13T14:37:49.679415+0000","last_clean":"2024-09-13T14:37:49.679415+0000","last_became_active":"2024-09-13T14:29:42.539068+0000","last_became_peered":"2024-09-13T14:29:42.539068+0000","last_unstale":"2024-09-13T14:37:49.679415+0000","last_undegraded":"2024-09-13T14:37:49.679415+0000","last_fullsized":"2024-09-13T14:37:49.679415+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:24:06.048855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034405999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1438,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016376+0000","last_change":"2024-09-13T14:30:15.574951+0000","last_active":"2024-09-13T14:37:57.016376+0000","last_peered":"2024-09-13T14:37:57.016376+0000","last_clean":"2024-09-13T14:37:57.016376+0000","last_became_active":"2024-09-13T14:30:15.574578+0000","last_became_peered":"2024-09-13T14:30:15.574578+0000","last_unstale":"2024-09-13T14:37:57.016376+0000","last_undegraded":"2024-09-13T14:37:57.016376+0000","last_fullsized":"2024-09-13T14:37:57.016376+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:20.700591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":997,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:50.054150+0000","last_change":"2024-09-13T14:19:27.379312+0000","last_active":"2024-09-13T14:37:50.054150+0000","last_peered":"2024-09-13T14:37:50.054150+0000","last_clean":"2024-09-13T14:37:50.054150+0000","last_became_active":"2024-09-13T14:19:27.378947+0000","last_became_peered":"2024-09-13T14:19:27.378947+0000","last_unstale":"2024-09-13T14:37:50.054150+0000","last_undegraded":"2024-09-13T14:37:50.054150+0000","last_fullsized":"2024-09-13T14:37:50.054150+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:54:20.706578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1061,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.678480+0000","last_change":"2024-09-13T14:29:08.913445+0000","last_active":"2024-09-13T14:37:49.678480+0000","last_peered":"2024-09-13T14:37:49.678480+0000","last_clean":"2024-09-13T14:37:49.678480+0000","last_became_active":"2024-09-13T14:29:08.913153+0000","last_became_peered":"2024-09-13T14:29:08.913153+0000","last_unstale":"2024-09-13T14:37:49.678480+0000","last_undegraded":"2024-09-13T14:37:49.678480+0000","last_fullsized":"2024-09-13T14:37:49.678480+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:55:34.761217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1554,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.376969+0000","last_change":"2024-09-13T14:30:15.586856+0000","last_active":"2024-09-13T14:37:55.376969+0000","last_peered":"2024-09-13T14:37:55.376969+0000","last_clean":"2024-09-13T14:37:55.376969+0000","last_became_active":"2024-09-13T14:30:15.586543+0000","last_became_peered":"2024-09-13T14:30:15.586543+0000","last_unstale":"2024-09-13T14:37:55.376969+0000","last_undegraded":"2024-09-13T14:37:55.376969+0000","last_fullsized":"2024-09-13T14:37:55.376969+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:57.072216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1532,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.377110+0000","last_change":"2024-09-13T14:29:42.541652+0000","last_active":"2024-09-13T14:37:55.377110+0000","last_peered":"2024-09-13T14:37:55.377110+0000","last_clean":"2024-09-13T14:37:55.377110+0000","last_became_active":"2024-09-13T14:29:42.541341+0000","last_became_peered":"2024-09-13T14:29:42.541341+0000","last_unstale":"2024-09-13T14:37:55.377110+0000","last_undegraded":"2024-09-13T14:37:55.377110+0000","last_fullsized":"2024-09-13T14:37:55.377110+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:52.954478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1524,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016159+0000","last_change":"2024-09-13T14:30:15.579354+0000","last_active":"2024-09-13T14:37:57.016159+0000","last_peered":"2024-09-13T14:37:57.016159+0000","last_clean":"2024-09-13T14:37:57.016159+0000","last_became_active":"2024-09-13T14:30:15.579068+0000","last_became_peered":"2024-09-13T14:30:15.579068+0000","last_unstale":"2024-09-13T14:37:57.016159+0000","last_undegraded":"2024-09-13T14:37:57.016159+0000","last_fullsized":"2024-09-13T14:37:57.016159+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:47:00.887687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1498,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.874298+0000","last_change":"2024-09-13T14:29:42.539284+0000","last_active":"2024-09-13T14:37:54.874298+0000","last_peered":"2024-09-13T14:37:54.874298+0000","last_clean":"2024-09-13T14:37:54.874298+0000","last_became_active":"2024-09-13T14:29:42.538970+0000","last_became_peered":"2024-09-13T14:29:42.538970+0000","last_unstale":"2024-09-13T14:37:54.874298+0000","last_undegraded":"2024-09-13T14:37:54.874298+0000","last_fullsized":"2024-09-13T14:37:54.874298+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:04.562806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1516,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.874028+0000","last_change":"2024-09-13T14:29:08.918645+0000","last_active":"2024-09-13T14:37:54.874028+0000","last_peered":"2024-09-13T14:37:54.874028+0000","last_clean":"2024-09-13T14:37:54.874028+0000","last_became_active":"2024-09-13T14:29:08.915894+0000","last_became_peered":"2024-09-13T14:29:08.915894+0000","last_unstale":"2024-09-13T14:37:54.874028+0000","last_undegraded":"2024-09-13T14:37:54.874028+0000","last_fullsized":"2024-09-13T14:37:54.874028+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:26.651547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":1035,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679417+0000","last_change":"2024-09-13T14:29:08.913956+0000","last_active":"2024-09-13T14:37:49.679417+0000","last_peered":"2024-09-13T14:37:49.679417+0000","last_clean":"2024-09-13T14:37:49.679417+0000","last_became_active":"2024-09-13T14:29:08.913081+0000","last_became_peered":"2024-09-13T14:29:08.913081+0000","last_unstale":"2024-09-13T14:37:49.679417+0000","last_undegraded":"2024-09-13T14:37:49.679417+0000","last_fullsized":"2024-09-13T14:37:49.679417+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:28.876391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1019,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750738+0000","last_change":"2024-09-13T14:28:35.604404+0000","last_active":"2024-09-13T14:37:49.750738+0000","last_peered":"2024-09-13T14:37:49.750738+0000","last_clean":"2024-09-13T14:37:49.750738+0000","last_became_active":"2024-09-13T14:28:35.604102+0000","last_became_peered":"2024-09-13T14:28:35.604102+0000","last_unstale":"2024-09-13T14:37:49.750738+0000","last_undegraded":"2024-09-13T14:37:49.750738+0000","last_fullsized":"2024-09-13T14:37:49.750738+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:25.546010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1510,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.873748+0000","last_change":"2024-09-13T14:30:15.582454+0000","last_active":"2024-09-13T14:37:54.873748+0000","last_peered":"2024-09-13T14:37:54.873748+0000","last_clean":"2024-09-13T14:37:54.873748+0000","last_became_active":"2024-09-13T14:30:15.582143+0000","last_became_peered":"2024-09-13T14:30:15.582143+0000","last_unstale":"2024-09-13T14:37:54.873748+0000","last_undegraded":"2024-09-13T14:37:54.873748+0000","last_fullsized":"2024-09-13T14:37:54.873748+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:21:20.076270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1038,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679925+0000","last_change":"2024-09-13T14:30:15.578388+0000","last_active":"2024-09-13T14:37:49.679925+0000","last_peered":"2024-09-13T14:37:49.679925+0000","last_clean":"2024-09-13T14:37:49.679925+0000","last_became_active":"2024-09-13T14:30:15.578026+0000","last_became_peered":"2024-09-13T14:30:15.578026+0000","last_unstale":"2024-09-13T14:37:49.679925+0000","last_undegraded":"2024-09-13T14:37:49.679925+0000","last_fullsized":"2024-09-13T14:37:49.679925+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:16:07.663218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1530,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.874032+0000","last_change":"2024-09-13T14:29:08.916449+0000","last_active":"2024-09-13T14:37:54.874032+0000","last_peered":"2024-09-13T14:37:54.874032+0000","last_clean":"2024-09-13T14:37:54.874032+0000","last_became_active":"2024-09-13T14:29:08.914257+0000","last_became_peered":"2024-09-13T14:29:08.914257+0000","last_unstale":"2024-09-13T14:37:54.874032+0000","last_undegraded":"2024-09-13T14:37:54.874032+0000","last_fullsized":"2024-09-13T14:37:54.874032+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:10:32.449958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1488,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.873865+0000","last_change":"2024-09-13T14:29:08.913030+0000","last_active":"2024-09-13T14:37:54.873865+0000","last_peered":"2024-09-13T14:37:54.873865+0000","last_clean":"2024-09-13T14:37:54.873865+0000","last_became_active":"2024-09-13T14:29:08.912671+0000","last_became_peered":"2024-09-13T14:29:08.912671+0000","last_unstale":"2024-09-13T14:37:54.873865+0000","last_undegraded":"2024-09-13T14:37:54.873865+0000","last_fullsized":"2024-09-13T14:37:54.873865+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:30:41.107429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1498,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.874171+0000","last_change":"2024-09-13T14:29:42.540294+0000","last_active":"2024-09-13T14:37:54.874171+0000","last_peered":"2024-09-13T14:37:54.874171+0000","last_clean":"2024-09-13T14:37:54.874171+0000","last_became_active":"2024-09-13T14:29:42.539930+0000","last_became_peered":"2024-09-13T14:29:42.539930+0000","last_unstale":"2024-09-13T14:37:54.874171+0000","last_undegraded":"2024-09-13T14:37:54.874171+0000","last_fullsized":"2024-09-13T14:37:54.874171+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:42.999879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1044,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.678990+0000","last_change":"2024-09-13T14:28:35.603663+0000","last_active":"2024-09-13T14:37:49.678990+0000","last_peered":"2024-09-13T14:37:49.678990+0000","last_clean":"2024-09-13T14:37:49.678990+0000","last_became_active":"2024-09-13T14:28:35.603358+0000","last_became_peered":"2024-09-13T14:28:35.603358+0000","last_unstale":"2024-09-13T14:37:49.678990+0000","last_undegraded":"2024-09-13T14:37:49.678990+0000","last_fullsized":"2024-09-13T14:37:49.678990+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:24:05.842796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1510,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.378225+0000","last_change":"2024-09-13T14:28:35.611508+0000","last_active":"2024-09-13T14:37:55.378225+0000","last_peered":"2024-09-13T14:37:55.378225+0000","last_clean":"2024-09-13T14:37:55.378225+0000","last_became_active":"2024-09-13T14:28:35.610106+0000","last_became_peered":"2024-09-13T14:28:35.610106+0000","last_unstale":"2024-09-13T14:37:55.378225+0000","last_undegraded":"2024-09-13T14:37:55.378225+0000","last_fullsized":"2024-09-13T14:37:55.378225+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:11.966958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":1470,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.017208+0000","last_change":"2024-09-13T14:30:15.581728+0000","last_active":"2024-09-13T14:37:57.017208+0000","last_peered":"2024-09-13T14:37:57.017208+0000","last_clean":"2024-09-13T14:37:57.017208+0000","last_became_active":"2024-09-13T14:30:15.581303+0000","last_became_peered":"2024-09-13T14:30:15.581303+0000","last_unstale":"2024-09-13T14:37:57.017208+0000","last_undegraded":"2024-09-13T14:37:57.017208+0000","last_fullsized":"2024-09-13T14:37:57.017208+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:10.027991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1498,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.874089+0000","last_change":"2024-09-13T14:29:08.917504+0000","last_active":"2024-09-13T14:37:54.874089+0000","last_peered":"2024-09-13T14:37:54.874089+0000","last_clean":"2024-09-13T14:37:54.874089+0000","last_became_active":"2024-09-13T14:29:08.917217+0000","last_became_peered":"2024-09-13T14:29:08.917217+0000","last_unstale":"2024-09-13T14:37:54.874089+0000","last_undegraded":"2024-09-13T14:37:54.874089+0000","last_fullsized":"2024-09-13T14:37:54.874089+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:36.651514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1468,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016021+0000","last_change":"2024-09-13T14:30:15.583561+0000","last_active":"2024-09-13T14:37:57.016021+0000","last_peered":"2024-09-13T14:37:57.016021+0000","last_clean":"2024-09-13T14:37:57.016021+0000","last_became_active":"2024-09-13T14:30:15.581464+0000","last_became_peered":"2024-09-13T14:30:15.581464+0000","last_unstale":"2024-09-13T14:37:57.016021+0000","last_undegraded":"2024-09-13T14:37:57.016021+0000","last_fullsized":"2024-09-13T14:37:57.016021+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:55:19.762648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":1524,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.378393+0000","last_change":"2024-09-13T14:29:08.916436+0000","last_active":"2024-09-13T14:37:55.378393+0000","last_peered":"2024-09-13T14:37:55.378393+0000","last_clean":"2024-09-13T14:37:55.378393+0000","last_became_active":"2024-09-13T14:29:08.916126+0000","last_became_peered":"2024-09-13T14:29:08.916126+0000","last_unstale":"2024-09-13T14:37:55.378393+0000","last_undegraded":"2024-09-13T14:37:55.378393+0000","last_fullsized":"2024-09-13T14:37:55.378393+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:00.324006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1028,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:50.054381+0000","last_change":"2024-09-13T14:29:42.538860+0000","last_active":"2024-09-13T14:37:50.054381+0000","last_peered":"2024-09-13T14:37:50.054381+0000","last_clean":"2024-09-13T14:37:50.054381+0000","last_became_active":"2024-09-13T14:29:42.538608+0000","last_became_peered":"2024-09-13T14:29:42.538608+0000","last_unstale":"2024-09-13T14:37:50.054381+0000","last_undegraded":"2024-09-13T14:37:50.054381+0000","last_fullsized":"2024-09-13T14:37:50.054381+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:20.070703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1001,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750320+0000","last_change":"2024-09-13T14:19:26.131609+0000","last_active":"2024-09-13T14:37:49.750320+0000","last_peered":"2024-09-13T14:37:49.750320+0000","last_clean":"2024-09-13T14:37:49.750320+0000","last_became_active":"2024-09-13T14:19:26.131029+0000","last_became_peered":"2024-09-13T14:19:26.131029+0000","last_unstale":"2024-09-13T14:37:49.750320+0000","last_undegraded":"2024-09-13T14:37:49.750320+0000","last_fullsized":"2024-09-13T14:37:49.750320+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:57.933696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1018,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750644+0000","last_change":"2024-09-13T14:29:08.911835+0000","last_active":"2024-09-13T14:37:49.750644+0000","last_peered":"2024-09-13T14:37:49.750644+0000","last_clean":"2024-09-13T14:37:49.750644+0000","last_became_active":"2024-09-13T14:29:08.911542+0000","last_became_peered":"2024-09-13T14:29:08.911542+0000","last_unstale":"2024-09-13T14:37:49.750644+0000","last_undegraded":"2024-09-13T14:37:49.750644+0000","last_fullsized":"2024-09-13T14:37:49.750644+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:30.820674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":1024,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750608+0000","last_change":"2024-09-13T14:19:25.870605+0000","last_active":"2024-09-13T14:37:49.750608+0000","last_peered":"2024-09-13T14:37:49.750608+0000","last_clean":"2024-09-13T14:37:49.750608+0000","last_became_active":"2024-09-13T14:19:25.870282+0000","last_became_peered":"2024-09-13T14:19:25.870282+0000","last_unstale":"2024-09-13T14:37:49.750608+0000","last_undegraded":"2024-09-13T14:37:49.750608+0000","last_fullsized":"2024-09-13T14:37:49.750608+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:52.514915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1499,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.873806+0000","last_change":"2024-09-13T14:29:42.537787+0000","last_active":"2024-09-13T14:37:54.873806+0000","last_peered":"2024-09-13T14:37:54.873806+0000","last_clean":"2024-09-13T14:37:54.873806+0000","last_became_active":"2024-09-13T14:29:42.537499+0000","last_became_peered":"2024-09-13T14:29:42.537499+0000","last_unstale":"2024-09-13T14:37:54.873806+0000","last_undegraded":"2024-09-13T14:37:54.873806+0000","last_fullsized":"2024-09-13T14:37:54.873806+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:15:30.508656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":1035,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.678750+0000","last_change":"2024-09-13T14:30:15.579260+0000","last_active":"2024-09-13T14:37:49.678750+0000","last_peered":"2024-09-13T14:37:49.678750+0000","last_clean":"2024-09-13T14:37:49.678750+0000","last_became_active":"2024-09-13T14:30:15.578897+0000","last_became_peered":"2024-09-13T14:30:15.578897+0000","last_unstale":"2024-09-13T14:37:49.678750+0000","last_undegraded":"2024-09-13T14:37:49.678750+0000","last_fullsized":"2024-09-13T14:37:49.678750+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:56.737849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1021,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.680020+0000","last_change":"2024-09-13T14:29:08.910821+0000","last_active":"2024-09-13T14:37:49.680020+0000","last_peered":"2024-09-13T14:37:49.680020+0000","last_clean":"2024-09-13T14:37:49.680020+0000","last_became_active":"2024-09-13T14:29:08.910509+0000","last_became_peered":"2024-09-13T14:29:08.910509+0000","last_unstale":"2024-09-13T14:37:49.680020+0000","last_undegraded":"2024-09-13T14:37:49.680020+0000","last_fullsized":"2024-09-13T14:37:49.680020+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:59.495053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1438,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.015920+0000","last_change":"2024-09-13T14:30:15.576921+0000","last_active":"2024-09-13T14:37:57.015920+0000","last_peered":"2024-09-13T14:37:57.015920+0000","last_clean":"2024-09-13T14:37:57.015920+0000","last_became_active":"2024-09-13T14:30:15.576607+0000","last_became_peered":"2024-09-13T14:30:15.576607+0000","last_unstale":"2024-09-13T14:37:57.015920+0000","last_undegraded":"2024-09-13T14:37:57.015920+0000","last_fullsized":"2024-09-13T14:37:57.015920+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:35.162460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1554,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.379052+0000","last_change":"2024-09-13T14:30:15.585712+0000","last_active":"2024-09-13T14:37:55.379052+0000","last_peered":"2024-09-13T14:37:55.379052+0000","last_clean":"2024-09-13T14:37:55.379052+0000","last_became_active":"2024-09-13T14:30:15.585147+0000","last_became_peered":"2024-09-13T14:30:15.585147+0000","last_unstale":"2024-09-13T14:37:55.379052+0000","last_undegraded":"2024-09-13T14:37:55.379052+0000","last_fullsized":"2024-09-13T14:37:55.379052+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:40:33.859429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1461,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.015890+0000","last_change":"2024-09-13T14:30:15.576306+0000","last_active":"2024-09-13T14:37:57.015890+0000","last_peered":"2024-09-13T14:37:57.015890+0000","last_clean":"2024-09-13T14:37:57.015890+0000","last_became_active":"2024-09-13T14:30:15.575931+0000","last_became_peered":"2024-09-13T14:30:15.575931+0000","last_unstale":"2024-09-13T14:37:57.015890+0000","last_undegraded":"2024-09-13T14:37:57.015890+0000","last_fullsized":"2024-09-13T14:37:57.015890+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:54.078309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1014,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:50.054296+0000","last_change":"2024-09-13T14:29:08.913672+0000","last_active":"2024-09-13T14:37:50.054296+0000","last_peered":"2024-09-13T14:37:50.054296+0000","last_clean":"2024-09-13T14:37:50.054296+0000","last_became_active":"2024-09-13T14:29:08.913360+0000","last_became_peered":"2024-09-13T14:29:08.913360+0000","last_unstale":"2024-09-13T14:37:50.054296+0000","last_undegraded":"2024-09-13T14:37:50.054296+0000","last_fullsized":"2024-09-13T14:37:50.054296+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:45.688163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1477,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016529+0000","last_change":"2024-09-13T14:30:15.583872+0000","last_active":"2024-09-13T14:37:57.016529+0000","last_peered":"2024-09-13T14:37:57.016529+0000","last_clean":"2024-09-13T14:37:57.016529+0000","last_became_active":"2024-09-13T14:30:15.581869+0000","last_became_peered":"2024-09-13T14:30:15.581869+0000","last_unstale":"2024-09-13T14:37:57.016529+0000","last_undegraded":"2024-09-13T14:37:57.016529+0000","last_fullsized":"2024-09-13T14:37:57.016529+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:10:37.393352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1522,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.378219+0000","last_change":"2024-09-13T14:30:15.586018+0000","last_active":"2024-09-13T14:37:55.378219+0000","last_peered":"2024-09-13T14:37:55.378219+0000","last_clean":"2024-09-13T14:37:55.378219+0000","last_became_active":"2024-09-13T14:30:15.585616+0000","last_became_peered":"2024-09-13T14:30:15.585616+0000","last_unstale":"2024-09-13T14:37:55.378219+0000","last_undegraded":"2024-09-13T14:37:55.378219+0000","last_fullsized":"2024-09-13T14:37:55.378219+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:46.836406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1497,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.873668+0000","last_change":"2024-09-13T14:30:15.579398+0000","last_active":"2024-09-13T14:37:54.873668+0000","last_peered":"2024-09-13T14:37:54.873668+0000","last_clean":"2024-09-13T14:37:54.873668+0000","last_became_active":"2024-09-13T14:30:15.579062+0000","last_became_peered":"2024-09-13T14:30:15.579062+0000","last_unstale":"2024-09-13T14:37:54.873668+0000","last_undegraded":"2024-09-13T14:37:54.873668+0000","last_fullsized":"2024-09-13T14:37:54.873668+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:06:39.532417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1459,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016735+0000","last_change":"2024-09-13T14:30:15.576778+0000","last_active":"2024-09-13T14:37:57.016735+0000","last_peered":"2024-09-13T14:37:57.016735+0000","last_clean":"2024-09-13T14:37:57.016735+0000","last_became_active":"2024-09-13T14:30:15.576434+0000","last_became_peered":"2024-09-13T14:30:15.576434+0000","last_unstale":"2024-09-13T14:37:57.016735+0000","last_undegraded":"2024-09-13T14:37:57.016735+0000","last_fullsized":"2024-09-13T14:37:57.016735+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:54:06.503790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":1037,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750337+0000","last_change":"2024-09-13T14:29:42.537809+0000","last_active":"2024-09-13T14:37:49.750337+0000","last_peered":"2024-09-13T14:37:49.750337+0000","last_clean":"2024-09-13T14:37:49.750337+0000","last_became_active":"2024-09-13T14:29:42.537482+0000","last_became_peered":"2024-09-13T14:29:42.537482+0000","last_unstale":"2024-09-13T14:37:49.750337+0000","last_undegraded":"2024-09-13T14:37:49.750337+0000","last_fullsized":"2024-09-13T14:37:49.750337+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:19.631210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1500,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016353+0000","last_change":"2024-09-13T14:30:15.578675+0000","last_active":"2024-09-13T14:37:57.016353+0000","last_peered":"2024-09-13T14:37:57.016353+0000","last_clean":"2024-09-13T14:37:57.016353+0000","last_became_active":"2024-09-13T14:30:15.578362+0000","last_became_peered":"2024-09-13T14:30:15.578362+0000","last_unstale":"2024-09-13T14:37:57.016353+0000","last_undegraded":"2024-09-13T14:37:57.016353+0000","last_fullsized":"2024-09-13T14:37:57.016353+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:30:47.445865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":1513,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.378569+0000","last_change":"2024-09-13T14:28:35.612469+0000","last_active":"2024-09-13T14:37:55.378569+0000","last_peered":"2024-09-13T14:37:55.378569+0000","last_clean":"2024-09-13T14:37:55.378569+0000","last_became_active":"2024-09-13T14:28:35.612093+0000","last_became_peered":"2024-09-13T14:28:35.612093+0000","last_unstale":"2024-09-13T14:37:55.378569+0000","last_undegraded":"2024-09-13T14:37:55.378569+0000","last_fullsized":"2024-09-13T14:37:55.378569+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:51.022663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1028,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750501+0000","last_change":"2024-09-13T14:30:15.580568+0000","last_active":"2024-09-13T14:37:49.750501+0000","last_peered":"2024-09-13T14:37:49.750501+0000","last_clean":"2024-09-13T14:37:49.750501+0000","last_became_active":"2024-09-13T14:30:15.580186+0000","last_became_peered":"2024-09-13T14:30:15.580186+0000","last_unstale":"2024-09-13T14:37:49.750501+0000","last_undegraded":"2024-09-13T14:37:49.750501+0000","last_fullsized":"2024-09-13T14:37:49.750501+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:16:00.781680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":1023,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679624+0000","last_change":"2024-09-13T14:29:08.912819+0000","last_active":"2024-09-13T14:37:49.679624+0000","last_peered":"2024-09-13T14:37:49.679624+0000","last_clean":"2024-09-13T14:37:49.679624+0000","last_became_active":"2024-09-13T14:29:08.912464+0000","last_became_peered":"2024-09-13T14:29:08.912464+0000","last_unstale":"2024-09-13T14:37:49.679624+0000","last_undegraded":"2024-09-13T14:37:49.679624+0000","last_fullsized":"2024-09-13T14:37:49.679624+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:09.753940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1035,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750210+0000","last_change":"2024-09-13T14:30:15.580011+0000","last_active":"2024-09-13T14:37:49.750210+0000","last_peered":"2024-09-13T14:37:49.750210+0000","last_clean":"2024-09-13T14:37:49.750210+0000","last_became_active":"2024-09-13T14:30:15.579610+0000","last_became_peered":"2024-09-13T14:30:15.579610+0000","last_unstale":"2024-09-13T14:37:49.750210+0000","last_undegraded":"2024-09-13T14:37:49.750210+0000","last_fullsized":"2024-09-13T14:37:49.750210+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:11:02.733771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":1006,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679901+0000","last_change":"2024-09-13T14:19:25.873514+0000","last_active":"2024-09-13T14:37:49.679901+0000","last_peered":"2024-09-13T14:37:49.679901+0000","last_clean":"2024-09-13T14:37:49.679901+0000","last_became_active":"2024-09-13T14:19:25.873170+0000","last_became_peered":"2024-09-13T14:19:25.873170+0000","last_unstale":"2024-09-13T14:37:49.679901+0000","last_undegraded":"2024-09-13T14:37:49.679901+0000","last_fullsized":"2024-09-13T14:37:49.679901+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:48.298530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1468,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016312+0000","last_change":"2024-09-13T14:30:15.583672+0000","last_active":"2024-09-13T14:37:57.016312+0000","last_peered":"2024-09-13T14:37:57.016312+0000","last_clean":"2024-09-13T14:37:57.016312+0000","last_became_active":"2024-09-13T14:30:15.583160+0000","last_became_peered":"2024-09-13T14:30:15.583160+0000","last_unstale":"2024-09-13T14:37:57.016312+0000","last_undegraded":"2024-09-13T14:37:57.016312+0000","last_fullsized":"2024-09-13T14:37:57.016312+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:54.303579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":1454,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.017187+0000","last_change":"2024-09-13T14:30:15.576496+0000","last_active":"2024-09-13T14:37:57.017187+0000","last_peered":"2024-09-13T14:37:57.017187+0000","last_clean":"2024-09-13T14:37:57.017187+0000","last_became_active":"2024-09-13T14:30:15.576137+0000","last_became_peered":"2024-09-13T14:30:15.576137+0000","last_unstale":"2024-09-13T14:37:57.017187+0000","last_undegraded":"2024-09-13T14:37:57.017187+0000","last_fullsized":"2024-09-13T14:37:57.017187+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:24.316047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1028,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750042+0000","last_change":"2024-09-13T14:29:42.539907+0000","last_active":"2024-09-13T14:37:49.750042+0000","last_peered":"2024-09-13T14:37:49.750042+0000","last_clean":"2024-09-13T14:37:49.750042+0000","last_became_active":"2024-09-13T14:29:42.539583+0000","last_became_peered":"2024-09-13T14:29:42.539583+0000","last_unstale":"2024-09-13T14:37:49.750042+0000","last_undegraded":"2024-09-13T14:37:49.750042+0000","last_fullsized":"2024-09-13T14:37:49.750042+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:02:04.628447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":1527,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.379125+0000","last_change":"2024-09-13T14:29:42.541263+0000","last_active":"2024-09-13T14:37:55.379125+0000","last_peered":"2024-09-13T14:37:55.379125+0000","last_clean":"2024-09-13T14:37:55.379125+0000","last_became_active":"2024-09-13T14:29:42.540873+0000","last_became_peered":"2024-09-13T14:29:42.540873+0000","last_unstale":"2024-09-13T14:37:55.379125+0000","last_undegraded":"2024-09-13T14:37:55.379125+0000","last_fullsized":"2024-09-13T14:37:55.379125+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:20.611206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430464,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311504,"kb_used_data":9988,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93433648,"statfs":{"total":95995035648,"available":95676055552,"internally_reserved":0,"allocated":10227712,"data_stored":7096822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626693,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":319816,"kb_used_data":9980,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93425336,"statfs":{"total":95995035648,"available":95667544064,"internally_reserved":0,"allocated":10219520,"data_stored":7096703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822922,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306748,"kb_used_data":9392,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93438404,"statfs":{"total":95995035648,"available":95680925696,"internally_reserved":0,"allocated":9617408,"data_stored":6502153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019150,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":315080,"kb_used_data":9404,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93430072,"statfs":{"total":95995035648,"available":95672393728,"internally_reserved":0,"allocated":9629696,"data_stored":6506384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059592808,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":45428,"kb_used_data":9384,"kb_used_omap":26,"kb_used_meta":36005,"kb_avail":93699724,"statfs":{"total":95995035648,"available":95948517376,"internally_reserved":0,"allocated":9609216,"data_stored":6501604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":36869280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":591,"seq":2538325672046,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37304,"kb_used_data":9388,"kb_used_omap":41,"kb_used_meta":27862,"kb_avail":93707848,"statfs":{"total":95995035648,"available":95956836352,"internally_reserved":0,"allocated":9613312,"data_stored":6502126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42651,"internal_metadata":28531045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":557,"seq":2392296783989,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37232,"kb_used_data":9384,"kb_used_omap":32,"kb_used_meta":27807,"kb_avail":93707920,"statfs":{"total":95995035648,"available":95956910080,"internally_reserved":0,"allocated":9609216,"data_stored":6501549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33752,"internal_metadata":28474408},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":524,"seq":2250562863228,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64188,"kb_used_data":9968,"kb_used_omap":41,"kb_used_meta":54166,"kb_avail":93680964,"statfs":{"total":95995035648,"available":95929307136,"internally_reserved":0,"allocated":10207232,"data_stored":7092498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42694,"internal_metadata":55466298},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:38:46.971 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-13T14:38:46.971 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-13T14:38:46.971 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-13T14:38:46.971 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-13T14:38:47.217 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:47.338 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:47 smithi151 ceph-mon[104450]: pgmap v1264: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:47.338 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:47 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:47.340 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:47 smithi151 ceph-mon[108966]: pgmap v1264: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:47.340 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:47 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:47.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:47 smithi184 ceph-mon[92527]: pgmap v1264: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:47.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:47 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:48.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:48 smithi184 ceph-mon[92527]: from='client.35202 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:38:48.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:48 smithi151 ceph-mon[104450]: from='client.35202 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:38:48.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:48 smithi151 ceph-mon[108966]: from='client.35202 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:38:49.142 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:38:49.142 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T14:38:49.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:49 smithi184 ceph-mon[92527]: pgmap v1265: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:49.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:49 smithi151 ceph-mon[108966]: pgmap v1265: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:49.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:49 smithi151 ceph-mon[104450]: pgmap v1265: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:49.737 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-13T14:38:49.737 INFO:teuthology.run_tasks:Running task print... 2024-09-13T14:38:49.747 INFO:teuthology.task.print:**** applying stress + thrashing to mixed-version cluster **** 2024-09-13T14:38:49.747 INFO:teuthology.run_tasks:Running task parallel... 2024-09-13T14:38:49.781 INFO:teuthology.task.parallel:starting parallel... 2024-09-13T14:38:49.781 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-13T14:38:49.781 INFO:teuthology.task.sequential:In sequential, running task thrashosds... 2024-09-13T14:38:49.784 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_force_recovery': 0, 'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_thrash_pg_upmap_items': 0, 'disable_objectstore_tool_tests': True, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-13T14:38:49.784 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-aclamk-cbt-combined'}, 'ceph': {'conf': {'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'create_rbd_pool': True, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'OSD_DOWN', 'mons down', 'mon down', 'MON_DOWN', 'out of quorum', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSDMAP_FLAGS', 'PG_AVAILABILITY'], 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-aclamk-cbt-combined', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}} 2024-09-13T14:38:49.784 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_force_recovery': 0, 'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_thrash_pg_upmap_items': 0, 'disable_objectstore_tool_tests': True, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-13T14:38:49.785 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-13T14:38:49.785 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-13T14:38:49.785 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:38:50.030 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:50.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:50 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2136737363' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-13T14:38:50.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:38:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:38:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:38:50.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:50 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2136737363' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-13T14:38:50.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:50 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2136737363' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-13T14:38:51.178 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:51 smithi151 ceph-mon[104450]: pgmap v1266: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:51.178 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:51 smithi151 ceph-mon[108966]: pgmap v1266: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:51.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:51 smithi184 ceph-mon[92527]: pgmap v1266: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:51.876 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:38:51.877 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T14:38:52.597 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1267,"stamp":"2024-09-13T14:38:51.295424+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":212,"num_write_kb":1659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1437300,"kb_used_data":76888,"kb_used_omap":149,"kb_used_meta":1360170,"kb_avail":748523916,"statfs":{"total":767960285184,"available":766488489984,"internally_reserved":0,"allocated":78733312,"data_stored":53799839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":153401,"internal_metadata":1392814279},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002906"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":1459,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016656+0000","last_change":"2024-09-13T14:30:15.579251+0000","last_active":"2024-09-13T14:37:57.016656+0000","last_peered":"2024-09-13T14:37:57.016656+0000","last_clean":"2024-09-13T14:37:57.016656+0000","last_became_active":"2024-09-13T14:30:15.578890+0000","last_became_peered":"2024-09-13T14:30:15.578890+0000","last_unstale":"2024-09-13T14:37:57.016656+0000","last_undegraded":"2024-09-13T14:37:57.016656+0000","last_fullsized":"2024-09-13T14:37:57.016656+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:51:28.380523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":1034,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679660+0000","last_change":"2024-09-13T14:29:42.541093+0000","last_active":"2024-09-13T14:37:49.679660+0000","last_peered":"2024-09-13T14:37:49.679660+0000","last_clean":"2024-09-13T14:37:49.679660+0000","last_became_active":"2024-09-13T14:29:42.540671+0000","last_became_peered":"2024-09-13T14:29:42.540671+0000","last_unstale":"2024-09-13T14:37:49.679660+0000","last_undegraded":"2024-09-13T14:37:49.679660+0000","last_fullsized":"2024-09-13T14:37:49.679660+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:45.722376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1020,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679822+0000","last_change":"2024-09-13T14:30:15.578414+0000","last_active":"2024-09-13T14:37:49.679822+0000","last_peered":"2024-09-13T14:37:49.679822+0000","last_clean":"2024-09-13T14:37:49.679822+0000","last_became_active":"2024-09-13T14:30:15.577997+0000","last_became_peered":"2024-09-13T14:30:15.577997+0000","last_unstale":"2024-09-13T14:37:49.679822+0000","last_undegraded":"2024-09-13T14:37:49.679822+0000","last_fullsized":"2024-09-13T14:37:49.679822+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:09.290677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1554,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.377071+0000","last_change":"2024-09-13T14:30:15.585508+0000","last_active":"2024-09-13T14:37:55.377071+0000","last_peered":"2024-09-13T14:37:55.377071+0000","last_clean":"2024-09-13T14:37:55.377071+0000","last_became_active":"2024-09-13T14:30:15.585206+0000","last_became_peered":"2024-09-13T14:30:15.585206+0000","last_unstale":"2024-09-13T14:37:55.377071+0000","last_undegraded":"2024-09-13T14:37:55.377071+0000","last_fullsized":"2024-09-13T14:37:55.377071+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:28:39.584397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1521,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.378348+0000","last_change":"2024-09-13T14:30:15.581992+0000","last_active":"2024-09-13T14:37:55.378348+0000","last_peered":"2024-09-13T14:37:55.378348+0000","last_clean":"2024-09-13T14:37:55.378348+0000","last_became_active":"2024-09-13T14:30:15.581692+0000","last_became_peered":"2024-09-13T14:30:15.581692+0000","last_unstale":"2024-09-13T14:37:55.378348+0000","last_undegraded":"2024-09-13T14:37:55.378348+0000","last_fullsized":"2024-09-13T14:37:55.378348+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:59.930184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1015,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.678847+0000","last_change":"2024-09-13T14:19:25.873088+0000","last_active":"2024-09-13T14:37:49.678847+0000","last_peered":"2024-09-13T14:37:49.678847+0000","last_clean":"2024-09-13T14:37:49.678847+0000","last_became_active":"2024-09-13T14:19:25.872781+0000","last_became_peered":"2024-09-13T14:19:25.872781+0000","last_unstale":"2024-09-13T14:37:49.678847+0000","last_undegraded":"2024-09-13T14:37:49.678847+0000","last_fullsized":"2024-09-13T14:37:49.678847+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:05.607382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1459,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.017200+0000","last_change":"2024-09-13T14:30:15.579629+0000","last_active":"2024-09-13T14:37:57.017200+0000","last_peered":"2024-09-13T14:37:57.017200+0000","last_clean":"2024-09-13T14:37:57.017200+0000","last_became_active":"2024-09-13T14:30:15.577913+0000","last_became_peered":"2024-09-13T14:30:15.577913+0000","last_unstale":"2024-09-13T14:37:57.017200+0000","last_undegraded":"2024-09-13T14:37:57.017200+0000","last_fullsized":"2024-09-13T14:37:57.017200+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:45:28.463082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1023,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:50.053938+0000","last_change":"2024-09-13T14:28:35.603724+0000","last_active":"2024-09-13T14:37:50.053938+0000","last_peered":"2024-09-13T14:37:50.053938+0000","last_clean":"2024-09-13T14:37:50.053938+0000","last_became_active":"2024-09-13T14:28:35.603316+0000","last_became_peered":"2024-09-13T14:28:35.603316+0000","last_unstale":"2024-09-13T14:37:50.053938+0000","last_undegraded":"2024-09-13T14:37:50.053938+0000","last_fullsized":"2024-09-13T14:37:50.053938+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:47.668223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":1033,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679017+0000","last_change":"2024-09-13T14:28:35.608607+0000","last_active":"2024-09-13T14:37:49.679017+0000","last_peered":"2024-09-13T14:37:49.679017+0000","last_clean":"2024-09-13T14:37:49.679017+0000","last_became_active":"2024-09-13T14:28:35.608254+0000","last_became_peered":"2024-09-13T14:28:35.608254+0000","last_unstale":"2024-09-13T14:37:49.679017+0000","last_undegraded":"2024-09-13T14:37:49.679017+0000","last_fullsized":"2024-09-13T14:37:49.679017+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:24.019173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1003,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679067+0000","last_change":"2024-09-13T14:19:26.887839+0000","last_active":"2024-09-13T14:37:49.679067+0000","last_peered":"2024-09-13T14:37:49.679067+0000","last_clean":"2024-09-13T14:37:49.679067+0000","last_became_active":"2024-09-13T14:19:26.887498+0000","last_became_peered":"2024-09-13T14:19:26.887498+0000","last_unstale":"2024-09-13T14:37:49.679067+0000","last_undegraded":"2024-09-13T14:37:49.679067+0000","last_fullsized":"2024-09-13T14:37:49.679067+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:24.475263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":1034,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679078+0000","last_change":"2024-09-13T14:29:08.909370+0000","last_active":"2024-09-13T14:37:49.679078+0000","last_peered":"2024-09-13T14:37:49.679078+0000","last_clean":"2024-09-13T14:37:49.679078+0000","last_became_active":"2024-09-13T14:29:08.908983+0000","last_became_peered":"2024-09-13T14:29:08.908983+0000","last_unstale":"2024-09-13T14:37:49.679078+0000","last_undegraded":"2024-09-13T14:37:49.679078+0000","last_fullsized":"2024-09-13T14:37:49.679078+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:44:43.939022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1035,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750462+0000","last_change":"2024-09-13T14:30:15.580757+0000","last_active":"2024-09-13T14:37:49.750462+0000","last_peered":"2024-09-13T14:37:49.750462+0000","last_clean":"2024-09-13T14:37:49.750462+0000","last_became_active":"2024-09-13T14:30:15.580401+0000","last_became_peered":"2024-09-13T14:30:15.580401+0000","last_unstale":"2024-09-13T14:37:49.750462+0000","last_undegraded":"2024-09-13T14:37:49.750462+0000","last_fullsized":"2024-09-13T14:37:49.750462+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:22:29.402576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":1016,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.678896+0000","last_change":"2024-09-13T14:19:25.871006+0000","last_active":"2024-09-13T14:37:49.678896+0000","last_peered":"2024-09-13T14:37:49.678896+0000","last_clean":"2024-09-13T14:37:49.678896+0000","last_became_active":"2024-09-13T14:19:25.870713+0000","last_became_peered":"2024-09-13T14:19:25.870713+0000","last_unstale":"2024-09-13T14:37:49.678896+0000","last_undegraded":"2024-09-13T14:37:49.678896+0000","last_fullsized":"2024-09-13T14:37:49.678896+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:34.372124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1014,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:50.054245+0000","last_change":"2024-09-13T14:29:08.914488+0000","last_active":"2024-09-13T14:37:50.054245+0000","last_peered":"2024-09-13T14:37:50.054245+0000","last_clean":"2024-09-13T14:37:50.054245+0000","last_became_active":"2024-09-13T14:29:08.913220+0000","last_became_peered":"2024-09-13T14:29:08.913220+0000","last_unstale":"2024-09-13T14:37:50.054245+0000","last_undegraded":"2024-09-13T14:37:50.054245+0000","last_fullsized":"2024-09-13T14:37:50.054245+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:08.316150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1021,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679699+0000","last_change":"2024-09-13T14:30:15.576953+0000","last_active":"2024-09-13T14:37:49.679699+0000","last_peered":"2024-09-13T14:37:49.679699+0000","last_clean":"2024-09-13T14:37:49.679699+0000","last_became_active":"2024-09-13T14:30:15.576614+0000","last_became_peered":"2024-09-13T14:30:15.576614+0000","last_unstale":"2024-09-13T14:37:49.679699+0000","last_undegraded":"2024-09-13T14:37:49.679699+0000","last_fullsized":"2024-09-13T14:37:49.679699+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:33:57.915263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1521,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.377210+0000","last_change":"2024-09-13T14:29:08.918531+0000","last_active":"2024-09-13T14:37:55.377210+0000","last_peered":"2024-09-13T14:37:55.377210+0000","last_clean":"2024-09-13T14:37:55.377210+0000","last_became_active":"2024-09-13T14:29:08.918264+0000","last_became_peered":"2024-09-13T14:29:08.918264+0000","last_unstale":"2024-09-13T14:37:55.377210+0000","last_undegraded":"2024-09-13T14:37:55.377210+0000","last_fullsized":"2024-09-13T14:37:55.377210+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:04.387551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":1470,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016218+0000","last_change":"2024-09-13T14:30:15.581475+0000","last_active":"2024-09-13T14:37:57.016218+0000","last_peered":"2024-09-13T14:37:57.016218+0000","last_clean":"2024-09-13T14:37:57.016218+0000","last_became_active":"2024-09-13T14:30:15.581148+0000","last_became_peered":"2024-09-13T14:30:15.581148+0000","last_unstale":"2024-09-13T14:37:57.016218+0000","last_undegraded":"2024-09-13T14:37:57.016218+0000","last_fullsized":"2024-09-13T14:37:57.016218+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:34.741839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1018,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750498+0000","last_change":"2024-09-13T14:30:15.580092+0000","last_active":"2024-09-13T14:37:49.750498+0000","last_peered":"2024-09-13T14:37:49.750498+0000","last_clean":"2024-09-13T14:37:49.750498+0000","last_became_active":"2024-09-13T14:30:15.579540+0000","last_became_peered":"2024-09-13T14:30:15.579540+0000","last_unstale":"2024-09-13T14:37:49.750498+0000","last_undegraded":"2024-09-13T14:37:49.750498+0000","last_fullsized":"2024-09-13T14:37:49.750498+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:54.416259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1015,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:50.054523+0000","last_change":"2024-09-13T14:29:08.915311+0000","last_active":"2024-09-13T14:37:50.054523+0000","last_peered":"2024-09-13T14:37:50.054523+0000","last_clean":"2024-09-13T14:37:50.054523+0000","last_became_active":"2024-09-13T14:29:08.914923+0000","last_became_peered":"2024-09-13T14:29:08.914923+0000","last_unstale":"2024-09-13T14:37:50.054523+0000","last_undegraded":"2024-09-13T14:37:50.054523+0000","last_fullsized":"2024-09-13T14:37:50.054523+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:26.311233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1035,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750768+0000","last_change":"2024-09-13T14:29:42.543672+0000","last_active":"2024-09-13T14:37:49.750768+0000","last_peered":"2024-09-13T14:37:49.750768+0000","last_clean":"2024-09-13T14:37:49.750768+0000","last_became_active":"2024-09-13T14:29:42.543329+0000","last_became_peered":"2024-09-13T14:29:42.543329+0000","last_unstale":"2024-09-13T14:37:49.750768+0000","last_undegraded":"2024-09-13T14:37:49.750768+0000","last_fullsized":"2024-09-13T14:37:49.750768+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:09:09.752737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1019,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750578+0000","last_change":"2024-09-13T14:29:42.545815+0000","last_active":"2024-09-13T14:37:49.750578+0000","last_peered":"2024-09-13T14:37:49.750578+0000","last_clean":"2024-09-13T14:37:49.750578+0000","last_became_active":"2024-09-13T14:29:42.545508+0000","last_became_peered":"2024-09-13T14:29:42.545508+0000","last_unstale":"2024-09-13T14:37:49.750578+0000","last_undegraded":"2024-09-13T14:37:49.750578+0000","last_fullsized":"2024-09-13T14:37:49.750578+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:39.678381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1018,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750764+0000","last_change":"2024-09-13T14:29:08.916416+0000","last_active":"2024-09-13T14:37:49.750764+0000","last_peered":"2024-09-13T14:37:49.750764+0000","last_clean":"2024-09-13T14:37:49.750764+0000","last_became_active":"2024-09-13T14:29:08.916106+0000","last_became_peered":"2024-09-13T14:29:08.916106+0000","last_unstale":"2024-09-13T14:37:49.750764+0000","last_undegraded":"2024-09-13T14:37:49.750764+0000","last_fullsized":"2024-09-13T14:37:49.750764+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:25.508423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1460,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016619+0000","last_change":"2024-09-13T14:30:15.577338+0000","last_active":"2024-09-13T14:37:57.016619+0000","last_peered":"2024-09-13T14:37:57.016619+0000","last_clean":"2024-09-13T14:37:57.016619+0000","last_became_active":"2024-09-13T14:30:15.577039+0000","last_became_peered":"2024-09-13T14:30:15.577039+0000","last_unstale":"2024-09-13T14:37:57.016619+0000","last_undegraded":"2024-09-13T14:37:57.016619+0000","last_fullsized":"2024-09-13T14:37:57.016619+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:43.246020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1499,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.874264+0000","last_change":"2024-09-13T14:30:15.580234+0000","last_active":"2024-09-13T14:37:54.874264+0000","last_peered":"2024-09-13T14:37:54.874264+0000","last_clean":"2024-09-13T14:37:54.874264+0000","last_became_active":"2024-09-13T14:30:15.579921+0000","last_became_peered":"2024-09-13T14:30:15.579921+0000","last_unstale":"2024-09-13T14:37:54.874264+0000","last_undegraded":"2024-09-13T14:37:54.874264+0000","last_fullsized":"2024-09-13T14:37:54.874264+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:42:39.428403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"261'100","reported_seq":1148,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.678592+0000","last_change":"2024-09-13T14:28:35.604004+0000","last_active":"2024-09-13T14:37:49.678592+0000","last_peered":"2024-09-13T14:37:49.678592+0000","last_clean":"2024-09-13T14:37:49.678592+0000","last_became_active":"2024-09-13T14:28:35.603626+0000","last_became_peered":"2024-09-13T14:28:35.603626+0000","last_unstale":"2024-09-13T14:37:49.678592+0000","last_undegraded":"2024-09-13T14:37:49.678592+0000","last_fullsized":"2024-09-13T14:37:49.678592+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:15:42.069756+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:39:52.205001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":1034,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679415+0000","last_change":"2024-09-13T14:29:42.539455+0000","last_active":"2024-09-13T14:37:49.679415+0000","last_peered":"2024-09-13T14:37:49.679415+0000","last_clean":"2024-09-13T14:37:49.679415+0000","last_became_active":"2024-09-13T14:29:42.539068+0000","last_became_peered":"2024-09-13T14:29:42.539068+0000","last_unstale":"2024-09-13T14:37:49.679415+0000","last_undegraded":"2024-09-13T14:37:49.679415+0000","last_fullsized":"2024-09-13T14:37:49.679415+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:24:06.048855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034405999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1438,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016376+0000","last_change":"2024-09-13T14:30:15.574951+0000","last_active":"2024-09-13T14:37:57.016376+0000","last_peered":"2024-09-13T14:37:57.016376+0000","last_clean":"2024-09-13T14:37:57.016376+0000","last_became_active":"2024-09-13T14:30:15.574578+0000","last_became_peered":"2024-09-13T14:30:15.574578+0000","last_unstale":"2024-09-13T14:37:57.016376+0000","last_undegraded":"2024-09-13T14:37:57.016376+0000","last_fullsized":"2024-09-13T14:37:57.016376+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:20.700591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":997,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:50.054150+0000","last_change":"2024-09-13T14:19:27.379312+0000","last_active":"2024-09-13T14:37:50.054150+0000","last_peered":"2024-09-13T14:37:50.054150+0000","last_clean":"2024-09-13T14:37:50.054150+0000","last_became_active":"2024-09-13T14:19:27.378947+0000","last_became_peered":"2024-09-13T14:19:27.378947+0000","last_unstale":"2024-09-13T14:37:50.054150+0000","last_undegraded":"2024-09-13T14:37:50.054150+0000","last_fullsized":"2024-09-13T14:37:50.054150+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:54:20.706578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1061,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.678480+0000","last_change":"2024-09-13T14:29:08.913445+0000","last_active":"2024-09-13T14:37:49.678480+0000","last_peered":"2024-09-13T14:37:49.678480+0000","last_clean":"2024-09-13T14:37:49.678480+0000","last_became_active":"2024-09-13T14:29:08.913153+0000","last_became_peered":"2024-09-13T14:29:08.913153+0000","last_unstale":"2024-09-13T14:37:49.678480+0000","last_undegraded":"2024-09-13T14:37:49.678480+0000","last_fullsized":"2024-09-13T14:37:49.678480+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:55:34.761217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1554,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.376969+0000","last_change":"2024-09-13T14:30:15.586856+0000","last_active":"2024-09-13T14:37:55.376969+0000","last_peered":"2024-09-13T14:37:55.376969+0000","last_clean":"2024-09-13T14:37:55.376969+0000","last_became_active":"2024-09-13T14:30:15.586543+0000","last_became_peered":"2024-09-13T14:30:15.586543+0000","last_unstale":"2024-09-13T14:37:55.376969+0000","last_undegraded":"2024-09-13T14:37:55.376969+0000","last_fullsized":"2024-09-13T14:37:55.376969+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:57.072216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1532,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.377110+0000","last_change":"2024-09-13T14:29:42.541652+0000","last_active":"2024-09-13T14:37:55.377110+0000","last_peered":"2024-09-13T14:37:55.377110+0000","last_clean":"2024-09-13T14:37:55.377110+0000","last_became_active":"2024-09-13T14:29:42.541341+0000","last_became_peered":"2024-09-13T14:29:42.541341+0000","last_unstale":"2024-09-13T14:37:55.377110+0000","last_undegraded":"2024-09-13T14:37:55.377110+0000","last_fullsized":"2024-09-13T14:37:55.377110+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:52.954478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1524,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016159+0000","last_change":"2024-09-13T14:30:15.579354+0000","last_active":"2024-09-13T14:37:57.016159+0000","last_peered":"2024-09-13T14:37:57.016159+0000","last_clean":"2024-09-13T14:37:57.016159+0000","last_became_active":"2024-09-13T14:30:15.579068+0000","last_became_peered":"2024-09-13T14:30:15.579068+0000","last_unstale":"2024-09-13T14:37:57.016159+0000","last_undegraded":"2024-09-13T14:37:57.016159+0000","last_fullsized":"2024-09-13T14:37:57.016159+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:47:00.887687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1498,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.874298+0000","last_change":"2024-09-13T14:29:42.539284+0000","last_active":"2024-09-13T14:37:54.874298+0000","last_peered":"2024-09-13T14:37:54.874298+0000","last_clean":"2024-09-13T14:37:54.874298+0000","last_became_active":"2024-09-13T14:29:42.538970+0000","last_became_peered":"2024-09-13T14:29:42.538970+0000","last_unstale":"2024-09-13T14:37:54.874298+0000","last_undegraded":"2024-09-13T14:37:54.874298+0000","last_fullsized":"2024-09-13T14:37:54.874298+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:04.562806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1516,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.874028+0000","last_change":"2024-09-13T14:29:08.918645+0000","last_active":"2024-09-13T14:37:54.874028+0000","last_peered":"2024-09-13T14:37:54.874028+0000","last_clean":"2024-09-13T14:37:54.874028+0000","last_became_active":"2024-09-13T14:29:08.915894+0000","last_became_peered":"2024-09-13T14:29:08.915894+0000","last_unstale":"2024-09-13T14:37:54.874028+0000","last_undegraded":"2024-09-13T14:37:54.874028+0000","last_fullsized":"2024-09-13T14:37:54.874028+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:26.651547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":1035,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679417+0000","last_change":"2024-09-13T14:29:08.913956+0000","last_active":"2024-09-13T14:37:49.679417+0000","last_peered":"2024-09-13T14:37:49.679417+0000","last_clean":"2024-09-13T14:37:49.679417+0000","last_became_active":"2024-09-13T14:29:08.913081+0000","last_became_peered":"2024-09-13T14:29:08.913081+0000","last_unstale":"2024-09-13T14:37:49.679417+0000","last_undegraded":"2024-09-13T14:37:49.679417+0000","last_fullsized":"2024-09-13T14:37:49.679417+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:28.876391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1019,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750738+0000","last_change":"2024-09-13T14:28:35.604404+0000","last_active":"2024-09-13T14:37:49.750738+0000","last_peered":"2024-09-13T14:37:49.750738+0000","last_clean":"2024-09-13T14:37:49.750738+0000","last_became_active":"2024-09-13T14:28:35.604102+0000","last_became_peered":"2024-09-13T14:28:35.604102+0000","last_unstale":"2024-09-13T14:37:49.750738+0000","last_undegraded":"2024-09-13T14:37:49.750738+0000","last_fullsized":"2024-09-13T14:37:49.750738+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:25.546010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1510,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.873748+0000","last_change":"2024-09-13T14:30:15.582454+0000","last_active":"2024-09-13T14:37:54.873748+0000","last_peered":"2024-09-13T14:37:54.873748+0000","last_clean":"2024-09-13T14:37:54.873748+0000","last_became_active":"2024-09-13T14:30:15.582143+0000","last_became_peered":"2024-09-13T14:30:15.582143+0000","last_unstale":"2024-09-13T14:37:54.873748+0000","last_undegraded":"2024-09-13T14:37:54.873748+0000","last_fullsized":"2024-09-13T14:37:54.873748+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:21:20.076270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1038,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679925+0000","last_change":"2024-09-13T14:30:15.578388+0000","last_active":"2024-09-13T14:37:49.679925+0000","last_peered":"2024-09-13T14:37:49.679925+0000","last_clean":"2024-09-13T14:37:49.679925+0000","last_became_active":"2024-09-13T14:30:15.578026+0000","last_became_peered":"2024-09-13T14:30:15.578026+0000","last_unstale":"2024-09-13T14:37:49.679925+0000","last_undegraded":"2024-09-13T14:37:49.679925+0000","last_fullsized":"2024-09-13T14:37:49.679925+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:16:07.663218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1530,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.874032+0000","last_change":"2024-09-13T14:29:08.916449+0000","last_active":"2024-09-13T14:37:54.874032+0000","last_peered":"2024-09-13T14:37:54.874032+0000","last_clean":"2024-09-13T14:37:54.874032+0000","last_became_active":"2024-09-13T14:29:08.914257+0000","last_became_peered":"2024-09-13T14:29:08.914257+0000","last_unstale":"2024-09-13T14:37:54.874032+0000","last_undegraded":"2024-09-13T14:37:54.874032+0000","last_fullsized":"2024-09-13T14:37:54.874032+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:10:32.449958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1488,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.873865+0000","last_change":"2024-09-13T14:29:08.913030+0000","last_active":"2024-09-13T14:37:54.873865+0000","last_peered":"2024-09-13T14:37:54.873865+0000","last_clean":"2024-09-13T14:37:54.873865+0000","last_became_active":"2024-09-13T14:29:08.912671+0000","last_became_peered":"2024-09-13T14:29:08.912671+0000","last_unstale":"2024-09-13T14:37:54.873865+0000","last_undegraded":"2024-09-13T14:37:54.873865+0000","last_fullsized":"2024-09-13T14:37:54.873865+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:30:41.107429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1498,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.874171+0000","last_change":"2024-09-13T14:29:42.540294+0000","last_active":"2024-09-13T14:37:54.874171+0000","last_peered":"2024-09-13T14:37:54.874171+0000","last_clean":"2024-09-13T14:37:54.874171+0000","last_became_active":"2024-09-13T14:29:42.539930+0000","last_became_peered":"2024-09-13T14:29:42.539930+0000","last_unstale":"2024-09-13T14:37:54.874171+0000","last_undegraded":"2024-09-13T14:37:54.874171+0000","last_fullsized":"2024-09-13T14:37:54.874171+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:42.999879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1044,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.678990+0000","last_change":"2024-09-13T14:28:35.603663+0000","last_active":"2024-09-13T14:37:49.678990+0000","last_peered":"2024-09-13T14:37:49.678990+0000","last_clean":"2024-09-13T14:37:49.678990+0000","last_became_active":"2024-09-13T14:28:35.603358+0000","last_became_peered":"2024-09-13T14:28:35.603358+0000","last_unstale":"2024-09-13T14:37:49.678990+0000","last_undegraded":"2024-09-13T14:37:49.678990+0000","last_fullsized":"2024-09-13T14:37:49.678990+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:24:05.842796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1510,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.378225+0000","last_change":"2024-09-13T14:28:35.611508+0000","last_active":"2024-09-13T14:37:55.378225+0000","last_peered":"2024-09-13T14:37:55.378225+0000","last_clean":"2024-09-13T14:37:55.378225+0000","last_became_active":"2024-09-13T14:28:35.610106+0000","last_became_peered":"2024-09-13T14:28:35.610106+0000","last_unstale":"2024-09-13T14:37:55.378225+0000","last_undegraded":"2024-09-13T14:37:55.378225+0000","last_fullsized":"2024-09-13T14:37:55.378225+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:11.966958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":1470,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.017208+0000","last_change":"2024-09-13T14:30:15.581728+0000","last_active":"2024-09-13T14:37:57.017208+0000","last_peered":"2024-09-13T14:37:57.017208+0000","last_clean":"2024-09-13T14:37:57.017208+0000","last_became_active":"2024-09-13T14:30:15.581303+0000","last_became_peered":"2024-09-13T14:30:15.581303+0000","last_unstale":"2024-09-13T14:37:57.017208+0000","last_undegraded":"2024-09-13T14:37:57.017208+0000","last_fullsized":"2024-09-13T14:37:57.017208+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:10.027991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1498,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.874089+0000","last_change":"2024-09-13T14:29:08.917504+0000","last_active":"2024-09-13T14:37:54.874089+0000","last_peered":"2024-09-13T14:37:54.874089+0000","last_clean":"2024-09-13T14:37:54.874089+0000","last_became_active":"2024-09-13T14:29:08.917217+0000","last_became_peered":"2024-09-13T14:29:08.917217+0000","last_unstale":"2024-09-13T14:37:54.874089+0000","last_undegraded":"2024-09-13T14:37:54.874089+0000","last_fullsized":"2024-09-13T14:37:54.874089+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:36.651514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1468,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016021+0000","last_change":"2024-09-13T14:30:15.583561+0000","last_active":"2024-09-13T14:37:57.016021+0000","last_peered":"2024-09-13T14:37:57.016021+0000","last_clean":"2024-09-13T14:37:57.016021+0000","last_became_active":"2024-09-13T14:30:15.581464+0000","last_became_peered":"2024-09-13T14:30:15.581464+0000","last_unstale":"2024-09-13T14:37:57.016021+0000","last_undegraded":"2024-09-13T14:37:57.016021+0000","last_fullsized":"2024-09-13T14:37:57.016021+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:55:19.762648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":1524,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.378393+0000","last_change":"2024-09-13T14:29:08.916436+0000","last_active":"2024-09-13T14:37:55.378393+0000","last_peered":"2024-09-13T14:37:55.378393+0000","last_clean":"2024-09-13T14:37:55.378393+0000","last_became_active":"2024-09-13T14:29:08.916126+0000","last_became_peered":"2024-09-13T14:29:08.916126+0000","last_unstale":"2024-09-13T14:37:55.378393+0000","last_undegraded":"2024-09-13T14:37:55.378393+0000","last_fullsized":"2024-09-13T14:37:55.378393+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:00.324006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1028,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:50.054381+0000","last_change":"2024-09-13T14:29:42.538860+0000","last_active":"2024-09-13T14:37:50.054381+0000","last_peered":"2024-09-13T14:37:50.054381+0000","last_clean":"2024-09-13T14:37:50.054381+0000","last_became_active":"2024-09-13T14:29:42.538608+0000","last_became_peered":"2024-09-13T14:29:42.538608+0000","last_unstale":"2024-09-13T14:37:50.054381+0000","last_undegraded":"2024-09-13T14:37:50.054381+0000","last_fullsized":"2024-09-13T14:37:50.054381+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:20.070703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1001,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750320+0000","last_change":"2024-09-13T14:19:26.131609+0000","last_active":"2024-09-13T14:37:49.750320+0000","last_peered":"2024-09-13T14:37:49.750320+0000","last_clean":"2024-09-13T14:37:49.750320+0000","last_became_active":"2024-09-13T14:19:26.131029+0000","last_became_peered":"2024-09-13T14:19:26.131029+0000","last_unstale":"2024-09-13T14:37:49.750320+0000","last_undegraded":"2024-09-13T14:37:49.750320+0000","last_fullsized":"2024-09-13T14:37:49.750320+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:57.933696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1018,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750644+0000","last_change":"2024-09-13T14:29:08.911835+0000","last_active":"2024-09-13T14:37:49.750644+0000","last_peered":"2024-09-13T14:37:49.750644+0000","last_clean":"2024-09-13T14:37:49.750644+0000","last_became_active":"2024-09-13T14:29:08.911542+0000","last_became_peered":"2024-09-13T14:29:08.911542+0000","last_unstale":"2024-09-13T14:37:49.750644+0000","last_undegraded":"2024-09-13T14:37:49.750644+0000","last_fullsized":"2024-09-13T14:37:49.750644+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:30.820674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":1024,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750608+0000","last_change":"2024-09-13T14:19:25.870605+0000","last_active":"2024-09-13T14:37:49.750608+0000","last_peered":"2024-09-13T14:37:49.750608+0000","last_clean":"2024-09-13T14:37:49.750608+0000","last_became_active":"2024-09-13T14:19:25.870282+0000","last_became_peered":"2024-09-13T14:19:25.870282+0000","last_unstale":"2024-09-13T14:37:49.750608+0000","last_undegraded":"2024-09-13T14:37:49.750608+0000","last_fullsized":"2024-09-13T14:37:49.750608+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:52.514915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1499,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.873806+0000","last_change":"2024-09-13T14:29:42.537787+0000","last_active":"2024-09-13T14:37:54.873806+0000","last_peered":"2024-09-13T14:37:54.873806+0000","last_clean":"2024-09-13T14:37:54.873806+0000","last_became_active":"2024-09-13T14:29:42.537499+0000","last_became_peered":"2024-09-13T14:29:42.537499+0000","last_unstale":"2024-09-13T14:37:54.873806+0000","last_undegraded":"2024-09-13T14:37:54.873806+0000","last_fullsized":"2024-09-13T14:37:54.873806+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:15:30.508656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":1035,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.678750+0000","last_change":"2024-09-13T14:30:15.579260+0000","last_active":"2024-09-13T14:37:49.678750+0000","last_peered":"2024-09-13T14:37:49.678750+0000","last_clean":"2024-09-13T14:37:49.678750+0000","last_became_active":"2024-09-13T14:30:15.578897+0000","last_became_peered":"2024-09-13T14:30:15.578897+0000","last_unstale":"2024-09-13T14:37:49.678750+0000","last_undegraded":"2024-09-13T14:37:49.678750+0000","last_fullsized":"2024-09-13T14:37:49.678750+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:56.737849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1021,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.680020+0000","last_change":"2024-09-13T14:29:08.910821+0000","last_active":"2024-09-13T14:37:49.680020+0000","last_peered":"2024-09-13T14:37:49.680020+0000","last_clean":"2024-09-13T14:37:49.680020+0000","last_became_active":"2024-09-13T14:29:08.910509+0000","last_became_peered":"2024-09-13T14:29:08.910509+0000","last_unstale":"2024-09-13T14:37:49.680020+0000","last_undegraded":"2024-09-13T14:37:49.680020+0000","last_fullsized":"2024-09-13T14:37:49.680020+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:59.495053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1438,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.015920+0000","last_change":"2024-09-13T14:30:15.576921+0000","last_active":"2024-09-13T14:37:57.015920+0000","last_peered":"2024-09-13T14:37:57.015920+0000","last_clean":"2024-09-13T14:37:57.015920+0000","last_became_active":"2024-09-13T14:30:15.576607+0000","last_became_peered":"2024-09-13T14:30:15.576607+0000","last_unstale":"2024-09-13T14:37:57.015920+0000","last_undegraded":"2024-09-13T14:37:57.015920+0000","last_fullsized":"2024-09-13T14:37:57.015920+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:35.162460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1554,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.379052+0000","last_change":"2024-09-13T14:30:15.585712+0000","last_active":"2024-09-13T14:37:55.379052+0000","last_peered":"2024-09-13T14:37:55.379052+0000","last_clean":"2024-09-13T14:37:55.379052+0000","last_became_active":"2024-09-13T14:30:15.585147+0000","last_became_peered":"2024-09-13T14:30:15.585147+0000","last_unstale":"2024-09-13T14:37:55.379052+0000","last_undegraded":"2024-09-13T14:37:55.379052+0000","last_fullsized":"2024-09-13T14:37:55.379052+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:40:33.859429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1461,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.015890+0000","last_change":"2024-09-13T14:30:15.576306+0000","last_active":"2024-09-13T14:37:57.015890+0000","last_peered":"2024-09-13T14:37:57.015890+0000","last_clean":"2024-09-13T14:37:57.015890+0000","last_became_active":"2024-09-13T14:30:15.575931+0000","last_became_peered":"2024-09-13T14:30:15.575931+0000","last_unstale":"2024-09-13T14:37:57.015890+0000","last_undegraded":"2024-09-13T14:37:57.015890+0000","last_fullsized":"2024-09-13T14:37:57.015890+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:54.078309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1014,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:50.054296+0000","last_change":"2024-09-13T14:29:08.913672+0000","last_active":"2024-09-13T14:37:50.054296+0000","last_peered":"2024-09-13T14:37:50.054296+0000","last_clean":"2024-09-13T14:37:50.054296+0000","last_became_active":"2024-09-13T14:29:08.913360+0000","last_became_peered":"2024-09-13T14:29:08.913360+0000","last_unstale":"2024-09-13T14:37:50.054296+0000","last_undegraded":"2024-09-13T14:37:50.054296+0000","last_fullsized":"2024-09-13T14:37:50.054296+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:45.688163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1477,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016529+0000","last_change":"2024-09-13T14:30:15.583872+0000","last_active":"2024-09-13T14:37:57.016529+0000","last_peered":"2024-09-13T14:37:57.016529+0000","last_clean":"2024-09-13T14:37:57.016529+0000","last_became_active":"2024-09-13T14:30:15.581869+0000","last_became_peered":"2024-09-13T14:30:15.581869+0000","last_unstale":"2024-09-13T14:37:57.016529+0000","last_undegraded":"2024-09-13T14:37:57.016529+0000","last_fullsized":"2024-09-13T14:37:57.016529+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:10:37.393352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1522,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.378219+0000","last_change":"2024-09-13T14:30:15.586018+0000","last_active":"2024-09-13T14:37:55.378219+0000","last_peered":"2024-09-13T14:37:55.378219+0000","last_clean":"2024-09-13T14:37:55.378219+0000","last_became_active":"2024-09-13T14:30:15.585616+0000","last_became_peered":"2024-09-13T14:30:15.585616+0000","last_unstale":"2024-09-13T14:37:55.378219+0000","last_undegraded":"2024-09-13T14:37:55.378219+0000","last_fullsized":"2024-09-13T14:37:55.378219+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:46.836406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1498,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.873746+0000","last_change":"2024-09-13T14:30:15.579398+0000","last_active":"2024-09-13T14:37:54.873746+0000","last_peered":"2024-09-13T14:37:54.873746+0000","last_clean":"2024-09-13T14:37:54.873746+0000","last_became_active":"2024-09-13T14:30:15.579062+0000","last_became_peered":"2024-09-13T14:30:15.579062+0000","last_unstale":"2024-09-13T14:37:54.873746+0000","last_undegraded":"2024-09-13T14:37:54.873746+0000","last_fullsized":"2024-09-13T14:37:54.873746+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:06:39.532417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1459,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016735+0000","last_change":"2024-09-13T14:30:15.576778+0000","last_active":"2024-09-13T14:37:57.016735+0000","last_peered":"2024-09-13T14:37:57.016735+0000","last_clean":"2024-09-13T14:37:57.016735+0000","last_became_active":"2024-09-13T14:30:15.576434+0000","last_became_peered":"2024-09-13T14:30:15.576434+0000","last_unstale":"2024-09-13T14:37:57.016735+0000","last_undegraded":"2024-09-13T14:37:57.016735+0000","last_fullsized":"2024-09-13T14:37:57.016735+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:54:06.503790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":1037,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750337+0000","last_change":"2024-09-13T14:29:42.537809+0000","last_active":"2024-09-13T14:37:49.750337+0000","last_peered":"2024-09-13T14:37:49.750337+0000","last_clean":"2024-09-13T14:37:49.750337+0000","last_became_active":"2024-09-13T14:29:42.537482+0000","last_became_peered":"2024-09-13T14:29:42.537482+0000","last_unstale":"2024-09-13T14:37:49.750337+0000","last_undegraded":"2024-09-13T14:37:49.750337+0000","last_fullsized":"2024-09-13T14:37:49.750337+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:19.631210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1500,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016353+0000","last_change":"2024-09-13T14:30:15.578675+0000","last_active":"2024-09-13T14:37:57.016353+0000","last_peered":"2024-09-13T14:37:57.016353+0000","last_clean":"2024-09-13T14:37:57.016353+0000","last_became_active":"2024-09-13T14:30:15.578362+0000","last_became_peered":"2024-09-13T14:30:15.578362+0000","last_unstale":"2024-09-13T14:37:57.016353+0000","last_undegraded":"2024-09-13T14:37:57.016353+0000","last_fullsized":"2024-09-13T14:37:57.016353+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:30:47.445865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":1513,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.378569+0000","last_change":"2024-09-13T14:28:35.612469+0000","last_active":"2024-09-13T14:37:55.378569+0000","last_peered":"2024-09-13T14:37:55.378569+0000","last_clean":"2024-09-13T14:37:55.378569+0000","last_became_active":"2024-09-13T14:28:35.612093+0000","last_became_peered":"2024-09-13T14:28:35.612093+0000","last_unstale":"2024-09-13T14:37:55.378569+0000","last_undegraded":"2024-09-13T14:37:55.378569+0000","last_fullsized":"2024-09-13T14:37:55.378569+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:51.022663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1028,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750501+0000","last_change":"2024-09-13T14:30:15.580568+0000","last_active":"2024-09-13T14:37:49.750501+0000","last_peered":"2024-09-13T14:37:49.750501+0000","last_clean":"2024-09-13T14:37:49.750501+0000","last_became_active":"2024-09-13T14:30:15.580186+0000","last_became_peered":"2024-09-13T14:30:15.580186+0000","last_unstale":"2024-09-13T14:37:49.750501+0000","last_undegraded":"2024-09-13T14:37:49.750501+0000","last_fullsized":"2024-09-13T14:37:49.750501+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:16:00.781680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":1023,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679624+0000","last_change":"2024-09-13T14:29:08.912819+0000","last_active":"2024-09-13T14:37:49.679624+0000","last_peered":"2024-09-13T14:37:49.679624+0000","last_clean":"2024-09-13T14:37:49.679624+0000","last_became_active":"2024-09-13T14:29:08.912464+0000","last_became_peered":"2024-09-13T14:29:08.912464+0000","last_unstale":"2024-09-13T14:37:49.679624+0000","last_undegraded":"2024-09-13T14:37:49.679624+0000","last_fullsized":"2024-09-13T14:37:49.679624+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:09.753940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1035,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750210+0000","last_change":"2024-09-13T14:30:15.580011+0000","last_active":"2024-09-13T14:37:49.750210+0000","last_peered":"2024-09-13T14:37:49.750210+0000","last_clean":"2024-09-13T14:37:49.750210+0000","last_became_active":"2024-09-13T14:30:15.579610+0000","last_became_peered":"2024-09-13T14:30:15.579610+0000","last_unstale":"2024-09-13T14:37:49.750210+0000","last_undegraded":"2024-09-13T14:37:49.750210+0000","last_fullsized":"2024-09-13T14:37:49.750210+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:11:02.733771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":1006,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679901+0000","last_change":"2024-09-13T14:19:25.873514+0000","last_active":"2024-09-13T14:37:49.679901+0000","last_peered":"2024-09-13T14:37:49.679901+0000","last_clean":"2024-09-13T14:37:49.679901+0000","last_became_active":"2024-09-13T14:19:25.873170+0000","last_became_peered":"2024-09-13T14:19:25.873170+0000","last_unstale":"2024-09-13T14:37:49.679901+0000","last_undegraded":"2024-09-13T14:37:49.679901+0000","last_fullsized":"2024-09-13T14:37:49.679901+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:48.298530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1468,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016312+0000","last_change":"2024-09-13T14:30:15.583672+0000","last_active":"2024-09-13T14:37:57.016312+0000","last_peered":"2024-09-13T14:37:57.016312+0000","last_clean":"2024-09-13T14:37:57.016312+0000","last_became_active":"2024-09-13T14:30:15.583160+0000","last_became_peered":"2024-09-13T14:30:15.583160+0000","last_unstale":"2024-09-13T14:37:57.016312+0000","last_undegraded":"2024-09-13T14:37:57.016312+0000","last_fullsized":"2024-09-13T14:37:57.016312+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:54.303579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":1454,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.017187+0000","last_change":"2024-09-13T14:30:15.576496+0000","last_active":"2024-09-13T14:37:57.017187+0000","last_peered":"2024-09-13T14:37:57.017187+0000","last_clean":"2024-09-13T14:37:57.017187+0000","last_became_active":"2024-09-13T14:30:15.576137+0000","last_became_peered":"2024-09-13T14:30:15.576137+0000","last_unstale":"2024-09-13T14:37:57.017187+0000","last_undegraded":"2024-09-13T14:37:57.017187+0000","last_fullsized":"2024-09-13T14:37:57.017187+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:24.316047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1028,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750042+0000","last_change":"2024-09-13T14:29:42.539907+0000","last_active":"2024-09-13T14:37:49.750042+0000","last_peered":"2024-09-13T14:37:49.750042+0000","last_clean":"2024-09-13T14:37:49.750042+0000","last_became_active":"2024-09-13T14:29:42.539583+0000","last_became_peered":"2024-09-13T14:29:42.539583+0000","last_unstale":"2024-09-13T14:37:49.750042+0000","last_undegraded":"2024-09-13T14:37:49.750042+0000","last_fullsized":"2024-09-13T14:37:49.750042+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:02:04.628447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":1527,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.379125+0000","last_change":"2024-09-13T14:29:42.541263+0000","last_active":"2024-09-13T14:37:55.379125+0000","last_peered":"2024-09-13T14:37:55.379125+0000","last_clean":"2024-09-13T14:37:55.379125+0000","last_became_active":"2024-09-13T14:29:42.540873+0000","last_became_peered":"2024-09-13T14:29:42.540873+0000","last_unstale":"2024-09-13T14:37:55.379125+0000","last_undegraded":"2024-09-13T14:37:55.379125+0000","last_fullsized":"2024-09-13T14:37:55.379125+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:20.611206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430465,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311504,"kb_used_data":9988,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93433648,"statfs":{"total":95995035648,"available":95676055552,"internally_reserved":0,"allocated":10227712,"data_stored":7096822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626694,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":319816,"kb_used_data":9980,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93425336,"statfs":{"total":95995035648,"available":95667544064,"internally_reserved":0,"allocated":10219520,"data_stored":7096703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822923,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306748,"kb_used_data":9392,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93438404,"statfs":{"total":95995035648,"available":95680925696,"internally_reserved":0,"allocated":9617408,"data_stored":6502153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019151,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":315080,"kb_used_data":9404,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93430072,"statfs":{"total":95995035648,"available":95672393728,"internally_reserved":0,"allocated":9629696,"data_stored":6506384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059592809,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":45428,"kb_used_data":9384,"kb_used_omap":26,"kb_used_meta":36005,"kb_avail":93699724,"statfs":{"total":95995035648,"available":95948517376,"internally_reserved":0,"allocated":9609216,"data_stored":6501604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":36869280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":591,"seq":2538325672048,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37304,"kb_used_data":9388,"kb_used_omap":41,"kb_used_meta":27862,"kb_avail":93707848,"statfs":{"total":95995035648,"available":95956836352,"internally_reserved":0,"allocated":9613312,"data_stored":6502126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42651,"internal_metadata":28531045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":557,"seq":2392296783990,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37232,"kb_used_data":9384,"kb_used_omap":32,"kb_used_meta":27807,"kb_avail":93707920,"statfs":{"total":95995035648,"available":95956910080,"internally_reserved":0,"allocated":9609216,"data_stored":6501549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33752,"internal_metadata":28474408},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":524,"seq":2250562863229,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64188,"kb_used_data":9968,"kb_used_omap":41,"kb_used_meta":54166,"kb_avail":93680964,"statfs":{"total":95995035648,"available":95929307136,"internally_reserved":0,"allocated":10207232,"data_stored":7092498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42694,"internal_metadata":55466298},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:38:52.601 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:38:52.847 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:53.219 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:53 smithi151 ceph-mon[104450]: pgmap v1267: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:53.219 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:53 smithi151 ceph-mon[104450]: from='client.45098 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:38:53.219 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:53 smithi151 ceph-mon[108966]: pgmap v1267: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:53.219 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:53 smithi151 ceph-mon[108966]: from='client.45098 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:38:53.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:53 smithi184 ceph-mon[92527]: pgmap v1267: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:53.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:53 smithi184 ceph-mon[92527]: from='client.45098 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:38:54.677 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:38:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:38:54] "GET /metrics HTTP/1.1" 200 34145 "" "Prometheus/2.51.0" 2024-09-13T14:38:54.835 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:38:54.835 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T14:38:55.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:55 smithi184 ceph-mon[92527]: pgmap v1268: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:55.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:55 smithi151 ceph-mon[104450]: pgmap v1268: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:55.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:55 smithi151 ceph-mon[108966]: pgmap v1268: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:55.541 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1268,"stamp":"2024-09-13T14:38:53.295869+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":212,"num_write_kb":1659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1437300,"kb_used_data":76888,"kb_used_omap":149,"kb_used_meta":1360170,"kb_avail":748523916,"statfs":{"total":767960285184,"available":766488489984,"internally_reserved":0,"allocated":78733312,"data_stored":53799839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":153401,"internal_metadata":1392814279},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002871"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":1459,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016656+0000","last_change":"2024-09-13T14:30:15.579251+0000","last_active":"2024-09-13T14:37:57.016656+0000","last_peered":"2024-09-13T14:37:57.016656+0000","last_clean":"2024-09-13T14:37:57.016656+0000","last_became_active":"2024-09-13T14:30:15.578890+0000","last_became_peered":"2024-09-13T14:30:15.578890+0000","last_unstale":"2024-09-13T14:37:57.016656+0000","last_undegraded":"2024-09-13T14:37:57.016656+0000","last_fullsized":"2024-09-13T14:37:57.016656+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:51:28.380523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":1034,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679660+0000","last_change":"2024-09-13T14:29:42.541093+0000","last_active":"2024-09-13T14:37:49.679660+0000","last_peered":"2024-09-13T14:37:49.679660+0000","last_clean":"2024-09-13T14:37:49.679660+0000","last_became_active":"2024-09-13T14:29:42.540671+0000","last_became_peered":"2024-09-13T14:29:42.540671+0000","last_unstale":"2024-09-13T14:37:49.679660+0000","last_undegraded":"2024-09-13T14:37:49.679660+0000","last_fullsized":"2024-09-13T14:37:49.679660+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:45.722376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1020,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679822+0000","last_change":"2024-09-13T14:30:15.578414+0000","last_active":"2024-09-13T14:37:49.679822+0000","last_peered":"2024-09-13T14:37:49.679822+0000","last_clean":"2024-09-13T14:37:49.679822+0000","last_became_active":"2024-09-13T14:30:15.577997+0000","last_became_peered":"2024-09-13T14:30:15.577997+0000","last_unstale":"2024-09-13T14:37:49.679822+0000","last_undegraded":"2024-09-13T14:37:49.679822+0000","last_fullsized":"2024-09-13T14:37:49.679822+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:09.290677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1554,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.377071+0000","last_change":"2024-09-13T14:30:15.585508+0000","last_active":"2024-09-13T14:37:55.377071+0000","last_peered":"2024-09-13T14:37:55.377071+0000","last_clean":"2024-09-13T14:37:55.377071+0000","last_became_active":"2024-09-13T14:30:15.585206+0000","last_became_peered":"2024-09-13T14:30:15.585206+0000","last_unstale":"2024-09-13T14:37:55.377071+0000","last_undegraded":"2024-09-13T14:37:55.377071+0000","last_fullsized":"2024-09-13T14:37:55.377071+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:28:39.584397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1521,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.378348+0000","last_change":"2024-09-13T14:30:15.581992+0000","last_active":"2024-09-13T14:37:55.378348+0000","last_peered":"2024-09-13T14:37:55.378348+0000","last_clean":"2024-09-13T14:37:55.378348+0000","last_became_active":"2024-09-13T14:30:15.581692+0000","last_became_peered":"2024-09-13T14:30:15.581692+0000","last_unstale":"2024-09-13T14:37:55.378348+0000","last_undegraded":"2024-09-13T14:37:55.378348+0000","last_fullsized":"2024-09-13T14:37:55.378348+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:59.930184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1015,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.678847+0000","last_change":"2024-09-13T14:19:25.873088+0000","last_active":"2024-09-13T14:37:49.678847+0000","last_peered":"2024-09-13T14:37:49.678847+0000","last_clean":"2024-09-13T14:37:49.678847+0000","last_became_active":"2024-09-13T14:19:25.872781+0000","last_became_peered":"2024-09-13T14:19:25.872781+0000","last_unstale":"2024-09-13T14:37:49.678847+0000","last_undegraded":"2024-09-13T14:37:49.678847+0000","last_fullsized":"2024-09-13T14:37:49.678847+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:05.607382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1459,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.017200+0000","last_change":"2024-09-13T14:30:15.579629+0000","last_active":"2024-09-13T14:37:57.017200+0000","last_peered":"2024-09-13T14:37:57.017200+0000","last_clean":"2024-09-13T14:37:57.017200+0000","last_became_active":"2024-09-13T14:30:15.577913+0000","last_became_peered":"2024-09-13T14:30:15.577913+0000","last_unstale":"2024-09-13T14:37:57.017200+0000","last_undegraded":"2024-09-13T14:37:57.017200+0000","last_fullsized":"2024-09-13T14:37:57.017200+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:45:28.463082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1023,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:50.053938+0000","last_change":"2024-09-13T14:28:35.603724+0000","last_active":"2024-09-13T14:37:50.053938+0000","last_peered":"2024-09-13T14:37:50.053938+0000","last_clean":"2024-09-13T14:37:50.053938+0000","last_became_active":"2024-09-13T14:28:35.603316+0000","last_became_peered":"2024-09-13T14:28:35.603316+0000","last_unstale":"2024-09-13T14:37:50.053938+0000","last_undegraded":"2024-09-13T14:37:50.053938+0000","last_fullsized":"2024-09-13T14:37:50.053938+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:47.668223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":1033,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679017+0000","last_change":"2024-09-13T14:28:35.608607+0000","last_active":"2024-09-13T14:37:49.679017+0000","last_peered":"2024-09-13T14:37:49.679017+0000","last_clean":"2024-09-13T14:37:49.679017+0000","last_became_active":"2024-09-13T14:28:35.608254+0000","last_became_peered":"2024-09-13T14:28:35.608254+0000","last_unstale":"2024-09-13T14:37:49.679017+0000","last_undegraded":"2024-09-13T14:37:49.679017+0000","last_fullsized":"2024-09-13T14:37:49.679017+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:24.019173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1003,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679067+0000","last_change":"2024-09-13T14:19:26.887839+0000","last_active":"2024-09-13T14:37:49.679067+0000","last_peered":"2024-09-13T14:37:49.679067+0000","last_clean":"2024-09-13T14:37:49.679067+0000","last_became_active":"2024-09-13T14:19:26.887498+0000","last_became_peered":"2024-09-13T14:19:26.887498+0000","last_unstale":"2024-09-13T14:37:49.679067+0000","last_undegraded":"2024-09-13T14:37:49.679067+0000","last_fullsized":"2024-09-13T14:37:49.679067+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:24.475263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":1034,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679078+0000","last_change":"2024-09-13T14:29:08.909370+0000","last_active":"2024-09-13T14:37:49.679078+0000","last_peered":"2024-09-13T14:37:49.679078+0000","last_clean":"2024-09-13T14:37:49.679078+0000","last_became_active":"2024-09-13T14:29:08.908983+0000","last_became_peered":"2024-09-13T14:29:08.908983+0000","last_unstale":"2024-09-13T14:37:49.679078+0000","last_undegraded":"2024-09-13T14:37:49.679078+0000","last_fullsized":"2024-09-13T14:37:49.679078+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:44:43.939022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1035,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750462+0000","last_change":"2024-09-13T14:30:15.580757+0000","last_active":"2024-09-13T14:37:49.750462+0000","last_peered":"2024-09-13T14:37:49.750462+0000","last_clean":"2024-09-13T14:37:49.750462+0000","last_became_active":"2024-09-13T14:30:15.580401+0000","last_became_peered":"2024-09-13T14:30:15.580401+0000","last_unstale":"2024-09-13T14:37:49.750462+0000","last_undegraded":"2024-09-13T14:37:49.750462+0000","last_fullsized":"2024-09-13T14:37:49.750462+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:22:29.402576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":1016,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.678896+0000","last_change":"2024-09-13T14:19:25.871006+0000","last_active":"2024-09-13T14:37:49.678896+0000","last_peered":"2024-09-13T14:37:49.678896+0000","last_clean":"2024-09-13T14:37:49.678896+0000","last_became_active":"2024-09-13T14:19:25.870713+0000","last_became_peered":"2024-09-13T14:19:25.870713+0000","last_unstale":"2024-09-13T14:37:49.678896+0000","last_undegraded":"2024-09-13T14:37:49.678896+0000","last_fullsized":"2024-09-13T14:37:49.678896+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:34.372124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1014,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:50.054245+0000","last_change":"2024-09-13T14:29:08.914488+0000","last_active":"2024-09-13T14:37:50.054245+0000","last_peered":"2024-09-13T14:37:50.054245+0000","last_clean":"2024-09-13T14:37:50.054245+0000","last_became_active":"2024-09-13T14:29:08.913220+0000","last_became_peered":"2024-09-13T14:29:08.913220+0000","last_unstale":"2024-09-13T14:37:50.054245+0000","last_undegraded":"2024-09-13T14:37:50.054245+0000","last_fullsized":"2024-09-13T14:37:50.054245+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:08.316150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1021,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679699+0000","last_change":"2024-09-13T14:30:15.576953+0000","last_active":"2024-09-13T14:37:49.679699+0000","last_peered":"2024-09-13T14:37:49.679699+0000","last_clean":"2024-09-13T14:37:49.679699+0000","last_became_active":"2024-09-13T14:30:15.576614+0000","last_became_peered":"2024-09-13T14:30:15.576614+0000","last_unstale":"2024-09-13T14:37:49.679699+0000","last_undegraded":"2024-09-13T14:37:49.679699+0000","last_fullsized":"2024-09-13T14:37:49.679699+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:33:57.915263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1521,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.377210+0000","last_change":"2024-09-13T14:29:08.918531+0000","last_active":"2024-09-13T14:37:55.377210+0000","last_peered":"2024-09-13T14:37:55.377210+0000","last_clean":"2024-09-13T14:37:55.377210+0000","last_became_active":"2024-09-13T14:29:08.918264+0000","last_became_peered":"2024-09-13T14:29:08.918264+0000","last_unstale":"2024-09-13T14:37:55.377210+0000","last_undegraded":"2024-09-13T14:37:55.377210+0000","last_fullsized":"2024-09-13T14:37:55.377210+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:04.387551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":1470,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016218+0000","last_change":"2024-09-13T14:30:15.581475+0000","last_active":"2024-09-13T14:37:57.016218+0000","last_peered":"2024-09-13T14:37:57.016218+0000","last_clean":"2024-09-13T14:37:57.016218+0000","last_became_active":"2024-09-13T14:30:15.581148+0000","last_became_peered":"2024-09-13T14:30:15.581148+0000","last_unstale":"2024-09-13T14:37:57.016218+0000","last_undegraded":"2024-09-13T14:37:57.016218+0000","last_fullsized":"2024-09-13T14:37:57.016218+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:34.741839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1018,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750498+0000","last_change":"2024-09-13T14:30:15.580092+0000","last_active":"2024-09-13T14:37:49.750498+0000","last_peered":"2024-09-13T14:37:49.750498+0000","last_clean":"2024-09-13T14:37:49.750498+0000","last_became_active":"2024-09-13T14:30:15.579540+0000","last_became_peered":"2024-09-13T14:30:15.579540+0000","last_unstale":"2024-09-13T14:37:49.750498+0000","last_undegraded":"2024-09-13T14:37:49.750498+0000","last_fullsized":"2024-09-13T14:37:49.750498+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:54.416259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1015,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:50.054523+0000","last_change":"2024-09-13T14:29:08.915311+0000","last_active":"2024-09-13T14:37:50.054523+0000","last_peered":"2024-09-13T14:37:50.054523+0000","last_clean":"2024-09-13T14:37:50.054523+0000","last_became_active":"2024-09-13T14:29:08.914923+0000","last_became_peered":"2024-09-13T14:29:08.914923+0000","last_unstale":"2024-09-13T14:37:50.054523+0000","last_undegraded":"2024-09-13T14:37:50.054523+0000","last_fullsized":"2024-09-13T14:37:50.054523+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:26.311233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1035,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750768+0000","last_change":"2024-09-13T14:29:42.543672+0000","last_active":"2024-09-13T14:37:49.750768+0000","last_peered":"2024-09-13T14:37:49.750768+0000","last_clean":"2024-09-13T14:37:49.750768+0000","last_became_active":"2024-09-13T14:29:42.543329+0000","last_became_peered":"2024-09-13T14:29:42.543329+0000","last_unstale":"2024-09-13T14:37:49.750768+0000","last_undegraded":"2024-09-13T14:37:49.750768+0000","last_fullsized":"2024-09-13T14:37:49.750768+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:09:09.752737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1019,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750578+0000","last_change":"2024-09-13T14:29:42.545815+0000","last_active":"2024-09-13T14:37:49.750578+0000","last_peered":"2024-09-13T14:37:49.750578+0000","last_clean":"2024-09-13T14:37:49.750578+0000","last_became_active":"2024-09-13T14:29:42.545508+0000","last_became_peered":"2024-09-13T14:29:42.545508+0000","last_unstale":"2024-09-13T14:37:49.750578+0000","last_undegraded":"2024-09-13T14:37:49.750578+0000","last_fullsized":"2024-09-13T14:37:49.750578+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:39.678381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1018,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750764+0000","last_change":"2024-09-13T14:29:08.916416+0000","last_active":"2024-09-13T14:37:49.750764+0000","last_peered":"2024-09-13T14:37:49.750764+0000","last_clean":"2024-09-13T14:37:49.750764+0000","last_became_active":"2024-09-13T14:29:08.916106+0000","last_became_peered":"2024-09-13T14:29:08.916106+0000","last_unstale":"2024-09-13T14:37:49.750764+0000","last_undegraded":"2024-09-13T14:37:49.750764+0000","last_fullsized":"2024-09-13T14:37:49.750764+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:25.508423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1460,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016619+0000","last_change":"2024-09-13T14:30:15.577338+0000","last_active":"2024-09-13T14:37:57.016619+0000","last_peered":"2024-09-13T14:37:57.016619+0000","last_clean":"2024-09-13T14:37:57.016619+0000","last_became_active":"2024-09-13T14:30:15.577039+0000","last_became_peered":"2024-09-13T14:30:15.577039+0000","last_unstale":"2024-09-13T14:37:57.016619+0000","last_undegraded":"2024-09-13T14:37:57.016619+0000","last_fullsized":"2024-09-13T14:37:57.016619+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:43.246020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1499,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.874264+0000","last_change":"2024-09-13T14:30:15.580234+0000","last_active":"2024-09-13T14:37:54.874264+0000","last_peered":"2024-09-13T14:37:54.874264+0000","last_clean":"2024-09-13T14:37:54.874264+0000","last_became_active":"2024-09-13T14:30:15.579921+0000","last_became_peered":"2024-09-13T14:30:15.579921+0000","last_unstale":"2024-09-13T14:37:54.874264+0000","last_undegraded":"2024-09-13T14:37:54.874264+0000","last_fullsized":"2024-09-13T14:37:54.874264+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:42:39.428403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"261'100","reported_seq":1148,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.678592+0000","last_change":"2024-09-13T14:28:35.604004+0000","last_active":"2024-09-13T14:37:49.678592+0000","last_peered":"2024-09-13T14:37:49.678592+0000","last_clean":"2024-09-13T14:37:49.678592+0000","last_became_active":"2024-09-13T14:28:35.603626+0000","last_became_peered":"2024-09-13T14:28:35.603626+0000","last_unstale":"2024-09-13T14:37:49.678592+0000","last_undegraded":"2024-09-13T14:37:49.678592+0000","last_fullsized":"2024-09-13T14:37:49.678592+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:15:42.069756+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:39:52.205001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":1034,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679415+0000","last_change":"2024-09-13T14:29:42.539455+0000","last_active":"2024-09-13T14:37:49.679415+0000","last_peered":"2024-09-13T14:37:49.679415+0000","last_clean":"2024-09-13T14:37:49.679415+0000","last_became_active":"2024-09-13T14:29:42.539068+0000","last_became_peered":"2024-09-13T14:29:42.539068+0000","last_unstale":"2024-09-13T14:37:49.679415+0000","last_undegraded":"2024-09-13T14:37:49.679415+0000","last_fullsized":"2024-09-13T14:37:49.679415+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:24:06.048855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034405999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1438,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016376+0000","last_change":"2024-09-13T14:30:15.574951+0000","last_active":"2024-09-13T14:37:57.016376+0000","last_peered":"2024-09-13T14:37:57.016376+0000","last_clean":"2024-09-13T14:37:57.016376+0000","last_became_active":"2024-09-13T14:30:15.574578+0000","last_became_peered":"2024-09-13T14:30:15.574578+0000","last_unstale":"2024-09-13T14:37:57.016376+0000","last_undegraded":"2024-09-13T14:37:57.016376+0000","last_fullsized":"2024-09-13T14:37:57.016376+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:20.700591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":997,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:50.054150+0000","last_change":"2024-09-13T14:19:27.379312+0000","last_active":"2024-09-13T14:37:50.054150+0000","last_peered":"2024-09-13T14:37:50.054150+0000","last_clean":"2024-09-13T14:37:50.054150+0000","last_became_active":"2024-09-13T14:19:27.378947+0000","last_became_peered":"2024-09-13T14:19:27.378947+0000","last_unstale":"2024-09-13T14:37:50.054150+0000","last_undegraded":"2024-09-13T14:37:50.054150+0000","last_fullsized":"2024-09-13T14:37:50.054150+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:54:20.706578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1061,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.678480+0000","last_change":"2024-09-13T14:29:08.913445+0000","last_active":"2024-09-13T14:37:49.678480+0000","last_peered":"2024-09-13T14:37:49.678480+0000","last_clean":"2024-09-13T14:37:49.678480+0000","last_became_active":"2024-09-13T14:29:08.913153+0000","last_became_peered":"2024-09-13T14:29:08.913153+0000","last_unstale":"2024-09-13T14:37:49.678480+0000","last_undegraded":"2024-09-13T14:37:49.678480+0000","last_fullsized":"2024-09-13T14:37:49.678480+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:55:34.761217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1554,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.376969+0000","last_change":"2024-09-13T14:30:15.586856+0000","last_active":"2024-09-13T14:37:55.376969+0000","last_peered":"2024-09-13T14:37:55.376969+0000","last_clean":"2024-09-13T14:37:55.376969+0000","last_became_active":"2024-09-13T14:30:15.586543+0000","last_became_peered":"2024-09-13T14:30:15.586543+0000","last_unstale":"2024-09-13T14:37:55.376969+0000","last_undegraded":"2024-09-13T14:37:55.376969+0000","last_fullsized":"2024-09-13T14:37:55.376969+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:57.072216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1532,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.377110+0000","last_change":"2024-09-13T14:29:42.541652+0000","last_active":"2024-09-13T14:37:55.377110+0000","last_peered":"2024-09-13T14:37:55.377110+0000","last_clean":"2024-09-13T14:37:55.377110+0000","last_became_active":"2024-09-13T14:29:42.541341+0000","last_became_peered":"2024-09-13T14:29:42.541341+0000","last_unstale":"2024-09-13T14:37:55.377110+0000","last_undegraded":"2024-09-13T14:37:55.377110+0000","last_fullsized":"2024-09-13T14:37:55.377110+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:52.954478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1524,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016159+0000","last_change":"2024-09-13T14:30:15.579354+0000","last_active":"2024-09-13T14:37:57.016159+0000","last_peered":"2024-09-13T14:37:57.016159+0000","last_clean":"2024-09-13T14:37:57.016159+0000","last_became_active":"2024-09-13T14:30:15.579068+0000","last_became_peered":"2024-09-13T14:30:15.579068+0000","last_unstale":"2024-09-13T14:37:57.016159+0000","last_undegraded":"2024-09-13T14:37:57.016159+0000","last_fullsized":"2024-09-13T14:37:57.016159+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:47:00.887687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1498,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.874298+0000","last_change":"2024-09-13T14:29:42.539284+0000","last_active":"2024-09-13T14:37:54.874298+0000","last_peered":"2024-09-13T14:37:54.874298+0000","last_clean":"2024-09-13T14:37:54.874298+0000","last_became_active":"2024-09-13T14:29:42.538970+0000","last_became_peered":"2024-09-13T14:29:42.538970+0000","last_unstale":"2024-09-13T14:37:54.874298+0000","last_undegraded":"2024-09-13T14:37:54.874298+0000","last_fullsized":"2024-09-13T14:37:54.874298+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:04.562806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1516,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.874028+0000","last_change":"2024-09-13T14:29:08.918645+0000","last_active":"2024-09-13T14:37:54.874028+0000","last_peered":"2024-09-13T14:37:54.874028+0000","last_clean":"2024-09-13T14:37:54.874028+0000","last_became_active":"2024-09-13T14:29:08.915894+0000","last_became_peered":"2024-09-13T14:29:08.915894+0000","last_unstale":"2024-09-13T14:37:54.874028+0000","last_undegraded":"2024-09-13T14:37:54.874028+0000","last_fullsized":"2024-09-13T14:37:54.874028+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:26.651547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":1035,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679417+0000","last_change":"2024-09-13T14:29:08.913956+0000","last_active":"2024-09-13T14:37:49.679417+0000","last_peered":"2024-09-13T14:37:49.679417+0000","last_clean":"2024-09-13T14:37:49.679417+0000","last_became_active":"2024-09-13T14:29:08.913081+0000","last_became_peered":"2024-09-13T14:29:08.913081+0000","last_unstale":"2024-09-13T14:37:49.679417+0000","last_undegraded":"2024-09-13T14:37:49.679417+0000","last_fullsized":"2024-09-13T14:37:49.679417+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:28.876391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1019,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750738+0000","last_change":"2024-09-13T14:28:35.604404+0000","last_active":"2024-09-13T14:37:49.750738+0000","last_peered":"2024-09-13T14:37:49.750738+0000","last_clean":"2024-09-13T14:37:49.750738+0000","last_became_active":"2024-09-13T14:28:35.604102+0000","last_became_peered":"2024-09-13T14:28:35.604102+0000","last_unstale":"2024-09-13T14:37:49.750738+0000","last_undegraded":"2024-09-13T14:37:49.750738+0000","last_fullsized":"2024-09-13T14:37:49.750738+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:25.546010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1510,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.873748+0000","last_change":"2024-09-13T14:30:15.582454+0000","last_active":"2024-09-13T14:37:54.873748+0000","last_peered":"2024-09-13T14:37:54.873748+0000","last_clean":"2024-09-13T14:37:54.873748+0000","last_became_active":"2024-09-13T14:30:15.582143+0000","last_became_peered":"2024-09-13T14:30:15.582143+0000","last_unstale":"2024-09-13T14:37:54.873748+0000","last_undegraded":"2024-09-13T14:37:54.873748+0000","last_fullsized":"2024-09-13T14:37:54.873748+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:21:20.076270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1038,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679925+0000","last_change":"2024-09-13T14:30:15.578388+0000","last_active":"2024-09-13T14:37:49.679925+0000","last_peered":"2024-09-13T14:37:49.679925+0000","last_clean":"2024-09-13T14:37:49.679925+0000","last_became_active":"2024-09-13T14:30:15.578026+0000","last_became_peered":"2024-09-13T14:30:15.578026+0000","last_unstale":"2024-09-13T14:37:49.679925+0000","last_undegraded":"2024-09-13T14:37:49.679925+0000","last_fullsized":"2024-09-13T14:37:49.679925+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:16:07.663218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1530,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.874032+0000","last_change":"2024-09-13T14:29:08.916449+0000","last_active":"2024-09-13T14:37:54.874032+0000","last_peered":"2024-09-13T14:37:54.874032+0000","last_clean":"2024-09-13T14:37:54.874032+0000","last_became_active":"2024-09-13T14:29:08.914257+0000","last_became_peered":"2024-09-13T14:29:08.914257+0000","last_unstale":"2024-09-13T14:37:54.874032+0000","last_undegraded":"2024-09-13T14:37:54.874032+0000","last_fullsized":"2024-09-13T14:37:54.874032+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:10:32.449958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1488,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.873865+0000","last_change":"2024-09-13T14:29:08.913030+0000","last_active":"2024-09-13T14:37:54.873865+0000","last_peered":"2024-09-13T14:37:54.873865+0000","last_clean":"2024-09-13T14:37:54.873865+0000","last_became_active":"2024-09-13T14:29:08.912671+0000","last_became_peered":"2024-09-13T14:29:08.912671+0000","last_unstale":"2024-09-13T14:37:54.873865+0000","last_undegraded":"2024-09-13T14:37:54.873865+0000","last_fullsized":"2024-09-13T14:37:54.873865+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:30:41.107429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1498,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.874171+0000","last_change":"2024-09-13T14:29:42.540294+0000","last_active":"2024-09-13T14:37:54.874171+0000","last_peered":"2024-09-13T14:37:54.874171+0000","last_clean":"2024-09-13T14:37:54.874171+0000","last_became_active":"2024-09-13T14:29:42.539930+0000","last_became_peered":"2024-09-13T14:29:42.539930+0000","last_unstale":"2024-09-13T14:37:54.874171+0000","last_undegraded":"2024-09-13T14:37:54.874171+0000","last_fullsized":"2024-09-13T14:37:54.874171+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:42.999879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1044,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.678990+0000","last_change":"2024-09-13T14:28:35.603663+0000","last_active":"2024-09-13T14:37:49.678990+0000","last_peered":"2024-09-13T14:37:49.678990+0000","last_clean":"2024-09-13T14:37:49.678990+0000","last_became_active":"2024-09-13T14:28:35.603358+0000","last_became_peered":"2024-09-13T14:28:35.603358+0000","last_unstale":"2024-09-13T14:37:49.678990+0000","last_undegraded":"2024-09-13T14:37:49.678990+0000","last_fullsized":"2024-09-13T14:37:49.678990+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:24:05.842796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1510,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.378225+0000","last_change":"2024-09-13T14:28:35.611508+0000","last_active":"2024-09-13T14:37:55.378225+0000","last_peered":"2024-09-13T14:37:55.378225+0000","last_clean":"2024-09-13T14:37:55.378225+0000","last_became_active":"2024-09-13T14:28:35.610106+0000","last_became_peered":"2024-09-13T14:28:35.610106+0000","last_unstale":"2024-09-13T14:37:55.378225+0000","last_undegraded":"2024-09-13T14:37:55.378225+0000","last_fullsized":"2024-09-13T14:37:55.378225+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:11.966958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":1470,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.017208+0000","last_change":"2024-09-13T14:30:15.581728+0000","last_active":"2024-09-13T14:37:57.017208+0000","last_peered":"2024-09-13T14:37:57.017208+0000","last_clean":"2024-09-13T14:37:57.017208+0000","last_became_active":"2024-09-13T14:30:15.581303+0000","last_became_peered":"2024-09-13T14:30:15.581303+0000","last_unstale":"2024-09-13T14:37:57.017208+0000","last_undegraded":"2024-09-13T14:37:57.017208+0000","last_fullsized":"2024-09-13T14:37:57.017208+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:10.027991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1498,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.874089+0000","last_change":"2024-09-13T14:29:08.917504+0000","last_active":"2024-09-13T14:37:54.874089+0000","last_peered":"2024-09-13T14:37:54.874089+0000","last_clean":"2024-09-13T14:37:54.874089+0000","last_became_active":"2024-09-13T14:29:08.917217+0000","last_became_peered":"2024-09-13T14:29:08.917217+0000","last_unstale":"2024-09-13T14:37:54.874089+0000","last_undegraded":"2024-09-13T14:37:54.874089+0000","last_fullsized":"2024-09-13T14:37:54.874089+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:36.651514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1468,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016021+0000","last_change":"2024-09-13T14:30:15.583561+0000","last_active":"2024-09-13T14:37:57.016021+0000","last_peered":"2024-09-13T14:37:57.016021+0000","last_clean":"2024-09-13T14:37:57.016021+0000","last_became_active":"2024-09-13T14:30:15.581464+0000","last_became_peered":"2024-09-13T14:30:15.581464+0000","last_unstale":"2024-09-13T14:37:57.016021+0000","last_undegraded":"2024-09-13T14:37:57.016021+0000","last_fullsized":"2024-09-13T14:37:57.016021+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:55:19.762648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":1524,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.378393+0000","last_change":"2024-09-13T14:29:08.916436+0000","last_active":"2024-09-13T14:37:55.378393+0000","last_peered":"2024-09-13T14:37:55.378393+0000","last_clean":"2024-09-13T14:37:55.378393+0000","last_became_active":"2024-09-13T14:29:08.916126+0000","last_became_peered":"2024-09-13T14:29:08.916126+0000","last_unstale":"2024-09-13T14:37:55.378393+0000","last_undegraded":"2024-09-13T14:37:55.378393+0000","last_fullsized":"2024-09-13T14:37:55.378393+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:00.324006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1028,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:50.054381+0000","last_change":"2024-09-13T14:29:42.538860+0000","last_active":"2024-09-13T14:37:50.054381+0000","last_peered":"2024-09-13T14:37:50.054381+0000","last_clean":"2024-09-13T14:37:50.054381+0000","last_became_active":"2024-09-13T14:29:42.538608+0000","last_became_peered":"2024-09-13T14:29:42.538608+0000","last_unstale":"2024-09-13T14:37:50.054381+0000","last_undegraded":"2024-09-13T14:37:50.054381+0000","last_fullsized":"2024-09-13T14:37:50.054381+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:20.070703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1001,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750320+0000","last_change":"2024-09-13T14:19:26.131609+0000","last_active":"2024-09-13T14:37:49.750320+0000","last_peered":"2024-09-13T14:37:49.750320+0000","last_clean":"2024-09-13T14:37:49.750320+0000","last_became_active":"2024-09-13T14:19:26.131029+0000","last_became_peered":"2024-09-13T14:19:26.131029+0000","last_unstale":"2024-09-13T14:37:49.750320+0000","last_undegraded":"2024-09-13T14:37:49.750320+0000","last_fullsized":"2024-09-13T14:37:49.750320+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:57.933696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1018,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750644+0000","last_change":"2024-09-13T14:29:08.911835+0000","last_active":"2024-09-13T14:37:49.750644+0000","last_peered":"2024-09-13T14:37:49.750644+0000","last_clean":"2024-09-13T14:37:49.750644+0000","last_became_active":"2024-09-13T14:29:08.911542+0000","last_became_peered":"2024-09-13T14:29:08.911542+0000","last_unstale":"2024-09-13T14:37:49.750644+0000","last_undegraded":"2024-09-13T14:37:49.750644+0000","last_fullsized":"2024-09-13T14:37:49.750644+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:30.820674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":1024,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750608+0000","last_change":"2024-09-13T14:19:25.870605+0000","last_active":"2024-09-13T14:37:49.750608+0000","last_peered":"2024-09-13T14:37:49.750608+0000","last_clean":"2024-09-13T14:37:49.750608+0000","last_became_active":"2024-09-13T14:19:25.870282+0000","last_became_peered":"2024-09-13T14:19:25.870282+0000","last_unstale":"2024-09-13T14:37:49.750608+0000","last_undegraded":"2024-09-13T14:37:49.750608+0000","last_fullsized":"2024-09-13T14:37:49.750608+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:52.514915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1499,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.873806+0000","last_change":"2024-09-13T14:29:42.537787+0000","last_active":"2024-09-13T14:37:54.873806+0000","last_peered":"2024-09-13T14:37:54.873806+0000","last_clean":"2024-09-13T14:37:54.873806+0000","last_became_active":"2024-09-13T14:29:42.537499+0000","last_became_peered":"2024-09-13T14:29:42.537499+0000","last_unstale":"2024-09-13T14:37:54.873806+0000","last_undegraded":"2024-09-13T14:37:54.873806+0000","last_fullsized":"2024-09-13T14:37:54.873806+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:15:30.508656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":1035,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.678750+0000","last_change":"2024-09-13T14:30:15.579260+0000","last_active":"2024-09-13T14:37:49.678750+0000","last_peered":"2024-09-13T14:37:49.678750+0000","last_clean":"2024-09-13T14:37:49.678750+0000","last_became_active":"2024-09-13T14:30:15.578897+0000","last_became_peered":"2024-09-13T14:30:15.578897+0000","last_unstale":"2024-09-13T14:37:49.678750+0000","last_undegraded":"2024-09-13T14:37:49.678750+0000","last_fullsized":"2024-09-13T14:37:49.678750+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:56.737849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1021,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.680020+0000","last_change":"2024-09-13T14:29:08.910821+0000","last_active":"2024-09-13T14:37:49.680020+0000","last_peered":"2024-09-13T14:37:49.680020+0000","last_clean":"2024-09-13T14:37:49.680020+0000","last_became_active":"2024-09-13T14:29:08.910509+0000","last_became_peered":"2024-09-13T14:29:08.910509+0000","last_unstale":"2024-09-13T14:37:49.680020+0000","last_undegraded":"2024-09-13T14:37:49.680020+0000","last_fullsized":"2024-09-13T14:37:49.680020+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:59.495053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1438,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.015920+0000","last_change":"2024-09-13T14:30:15.576921+0000","last_active":"2024-09-13T14:37:57.015920+0000","last_peered":"2024-09-13T14:37:57.015920+0000","last_clean":"2024-09-13T14:37:57.015920+0000","last_became_active":"2024-09-13T14:30:15.576607+0000","last_became_peered":"2024-09-13T14:30:15.576607+0000","last_unstale":"2024-09-13T14:37:57.015920+0000","last_undegraded":"2024-09-13T14:37:57.015920+0000","last_fullsized":"2024-09-13T14:37:57.015920+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:35.162460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1554,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.379052+0000","last_change":"2024-09-13T14:30:15.585712+0000","last_active":"2024-09-13T14:37:55.379052+0000","last_peered":"2024-09-13T14:37:55.379052+0000","last_clean":"2024-09-13T14:37:55.379052+0000","last_became_active":"2024-09-13T14:30:15.585147+0000","last_became_peered":"2024-09-13T14:30:15.585147+0000","last_unstale":"2024-09-13T14:37:55.379052+0000","last_undegraded":"2024-09-13T14:37:55.379052+0000","last_fullsized":"2024-09-13T14:37:55.379052+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:40:33.859429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1461,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.015890+0000","last_change":"2024-09-13T14:30:15.576306+0000","last_active":"2024-09-13T14:37:57.015890+0000","last_peered":"2024-09-13T14:37:57.015890+0000","last_clean":"2024-09-13T14:37:57.015890+0000","last_became_active":"2024-09-13T14:30:15.575931+0000","last_became_peered":"2024-09-13T14:30:15.575931+0000","last_unstale":"2024-09-13T14:37:57.015890+0000","last_undegraded":"2024-09-13T14:37:57.015890+0000","last_fullsized":"2024-09-13T14:37:57.015890+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:54.078309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1014,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:50.054296+0000","last_change":"2024-09-13T14:29:08.913672+0000","last_active":"2024-09-13T14:37:50.054296+0000","last_peered":"2024-09-13T14:37:50.054296+0000","last_clean":"2024-09-13T14:37:50.054296+0000","last_became_active":"2024-09-13T14:29:08.913360+0000","last_became_peered":"2024-09-13T14:29:08.913360+0000","last_unstale":"2024-09-13T14:37:50.054296+0000","last_undegraded":"2024-09-13T14:37:50.054296+0000","last_fullsized":"2024-09-13T14:37:50.054296+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:45.688163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1477,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016529+0000","last_change":"2024-09-13T14:30:15.583872+0000","last_active":"2024-09-13T14:37:57.016529+0000","last_peered":"2024-09-13T14:37:57.016529+0000","last_clean":"2024-09-13T14:37:57.016529+0000","last_became_active":"2024-09-13T14:30:15.581869+0000","last_became_peered":"2024-09-13T14:30:15.581869+0000","last_unstale":"2024-09-13T14:37:57.016529+0000","last_undegraded":"2024-09-13T14:37:57.016529+0000","last_fullsized":"2024-09-13T14:37:57.016529+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:10:37.393352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1522,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.378219+0000","last_change":"2024-09-13T14:30:15.586018+0000","last_active":"2024-09-13T14:37:55.378219+0000","last_peered":"2024-09-13T14:37:55.378219+0000","last_clean":"2024-09-13T14:37:55.378219+0000","last_became_active":"2024-09-13T14:30:15.585616+0000","last_became_peered":"2024-09-13T14:30:15.585616+0000","last_unstale":"2024-09-13T14:37:55.378219+0000","last_undegraded":"2024-09-13T14:37:55.378219+0000","last_fullsized":"2024-09-13T14:37:55.378219+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:46.836406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1498,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:54.873746+0000","last_change":"2024-09-13T14:30:15.579398+0000","last_active":"2024-09-13T14:37:54.873746+0000","last_peered":"2024-09-13T14:37:54.873746+0000","last_clean":"2024-09-13T14:37:54.873746+0000","last_became_active":"2024-09-13T14:30:15.579062+0000","last_became_peered":"2024-09-13T14:30:15.579062+0000","last_unstale":"2024-09-13T14:37:54.873746+0000","last_undegraded":"2024-09-13T14:37:54.873746+0000","last_fullsized":"2024-09-13T14:37:54.873746+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:06:39.532417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1459,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016735+0000","last_change":"2024-09-13T14:30:15.576778+0000","last_active":"2024-09-13T14:37:57.016735+0000","last_peered":"2024-09-13T14:37:57.016735+0000","last_clean":"2024-09-13T14:37:57.016735+0000","last_became_active":"2024-09-13T14:30:15.576434+0000","last_became_peered":"2024-09-13T14:30:15.576434+0000","last_unstale":"2024-09-13T14:37:57.016735+0000","last_undegraded":"2024-09-13T14:37:57.016735+0000","last_fullsized":"2024-09-13T14:37:57.016735+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:54:06.503790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":1037,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750337+0000","last_change":"2024-09-13T14:29:42.537809+0000","last_active":"2024-09-13T14:37:49.750337+0000","last_peered":"2024-09-13T14:37:49.750337+0000","last_clean":"2024-09-13T14:37:49.750337+0000","last_became_active":"2024-09-13T14:29:42.537482+0000","last_became_peered":"2024-09-13T14:29:42.537482+0000","last_unstale":"2024-09-13T14:37:49.750337+0000","last_undegraded":"2024-09-13T14:37:49.750337+0000","last_fullsized":"2024-09-13T14:37:49.750337+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:19.631210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1500,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016353+0000","last_change":"2024-09-13T14:30:15.578675+0000","last_active":"2024-09-13T14:37:57.016353+0000","last_peered":"2024-09-13T14:37:57.016353+0000","last_clean":"2024-09-13T14:37:57.016353+0000","last_became_active":"2024-09-13T14:30:15.578362+0000","last_became_peered":"2024-09-13T14:30:15.578362+0000","last_unstale":"2024-09-13T14:37:57.016353+0000","last_undegraded":"2024-09-13T14:37:57.016353+0000","last_fullsized":"2024-09-13T14:37:57.016353+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:30:47.445865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":1513,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.378569+0000","last_change":"2024-09-13T14:28:35.612469+0000","last_active":"2024-09-13T14:37:55.378569+0000","last_peered":"2024-09-13T14:37:55.378569+0000","last_clean":"2024-09-13T14:37:55.378569+0000","last_became_active":"2024-09-13T14:28:35.612093+0000","last_became_peered":"2024-09-13T14:28:35.612093+0000","last_unstale":"2024-09-13T14:37:55.378569+0000","last_undegraded":"2024-09-13T14:37:55.378569+0000","last_fullsized":"2024-09-13T14:37:55.378569+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:51.022663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1028,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750501+0000","last_change":"2024-09-13T14:30:15.580568+0000","last_active":"2024-09-13T14:37:49.750501+0000","last_peered":"2024-09-13T14:37:49.750501+0000","last_clean":"2024-09-13T14:37:49.750501+0000","last_became_active":"2024-09-13T14:30:15.580186+0000","last_became_peered":"2024-09-13T14:30:15.580186+0000","last_unstale":"2024-09-13T14:37:49.750501+0000","last_undegraded":"2024-09-13T14:37:49.750501+0000","last_fullsized":"2024-09-13T14:37:49.750501+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:16:00.781680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":1023,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679624+0000","last_change":"2024-09-13T14:29:08.912819+0000","last_active":"2024-09-13T14:37:49.679624+0000","last_peered":"2024-09-13T14:37:49.679624+0000","last_clean":"2024-09-13T14:37:49.679624+0000","last_became_active":"2024-09-13T14:29:08.912464+0000","last_became_peered":"2024-09-13T14:29:08.912464+0000","last_unstale":"2024-09-13T14:37:49.679624+0000","last_undegraded":"2024-09-13T14:37:49.679624+0000","last_fullsized":"2024-09-13T14:37:49.679624+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:09.753940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1035,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750210+0000","last_change":"2024-09-13T14:30:15.580011+0000","last_active":"2024-09-13T14:37:49.750210+0000","last_peered":"2024-09-13T14:37:49.750210+0000","last_clean":"2024-09-13T14:37:49.750210+0000","last_became_active":"2024-09-13T14:30:15.579610+0000","last_became_peered":"2024-09-13T14:30:15.579610+0000","last_unstale":"2024-09-13T14:37:49.750210+0000","last_undegraded":"2024-09-13T14:37:49.750210+0000","last_fullsized":"2024-09-13T14:37:49.750210+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:11:02.733771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":1006,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.679901+0000","last_change":"2024-09-13T14:19:25.873514+0000","last_active":"2024-09-13T14:37:49.679901+0000","last_peered":"2024-09-13T14:37:49.679901+0000","last_clean":"2024-09-13T14:37:49.679901+0000","last_became_active":"2024-09-13T14:19:25.873170+0000","last_became_peered":"2024-09-13T14:19:25.873170+0000","last_unstale":"2024-09-13T14:37:49.679901+0000","last_undegraded":"2024-09-13T14:37:49.679901+0000","last_fullsized":"2024-09-13T14:37:49.679901+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:48.298530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1468,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.016312+0000","last_change":"2024-09-13T14:30:15.583672+0000","last_active":"2024-09-13T14:37:57.016312+0000","last_peered":"2024-09-13T14:37:57.016312+0000","last_clean":"2024-09-13T14:37:57.016312+0000","last_became_active":"2024-09-13T14:30:15.583160+0000","last_became_peered":"2024-09-13T14:30:15.583160+0000","last_unstale":"2024-09-13T14:37:57.016312+0000","last_undegraded":"2024-09-13T14:37:57.016312+0000","last_fullsized":"2024-09-13T14:37:57.016312+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:54.303579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":1454,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:57.017187+0000","last_change":"2024-09-13T14:30:15.576496+0000","last_active":"2024-09-13T14:37:57.017187+0000","last_peered":"2024-09-13T14:37:57.017187+0000","last_clean":"2024-09-13T14:37:57.017187+0000","last_became_active":"2024-09-13T14:30:15.576137+0000","last_became_peered":"2024-09-13T14:30:15.576137+0000","last_unstale":"2024-09-13T14:37:57.017187+0000","last_undegraded":"2024-09-13T14:37:57.017187+0000","last_fullsized":"2024-09-13T14:37:57.017187+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:24.316047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1028,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-13T14:37:49.750042+0000","last_change":"2024-09-13T14:29:42.539907+0000","last_active":"2024-09-13T14:37:49.750042+0000","last_peered":"2024-09-13T14:37:49.750042+0000","last_clean":"2024-09-13T14:37:49.750042+0000","last_became_active":"2024-09-13T14:29:42.539583+0000","last_became_peered":"2024-09-13T14:29:42.539583+0000","last_unstale":"2024-09-13T14:37:49.750042+0000","last_undegraded":"2024-09-13T14:37:49.750042+0000","last_fullsized":"2024-09-13T14:37:49.750042+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:02:04.628447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":1527,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-13T14:37:55.379125+0000","last_change":"2024-09-13T14:29:42.541263+0000","last_active":"2024-09-13T14:37:55.379125+0000","last_peered":"2024-09-13T14:37:55.379125+0000","last_clean":"2024-09-13T14:37:55.379125+0000","last_became_active":"2024-09-13T14:29:42.540873+0000","last_became_peered":"2024-09-13T14:29:42.540873+0000","last_unstale":"2024-09-13T14:37:55.379125+0000","last_undegraded":"2024-09-13T14:37:55.379125+0000","last_fullsized":"2024-09-13T14:37:55.379125+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:20.611206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430466,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311504,"kb_used_data":9988,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93433648,"statfs":{"total":95995035648,"available":95676055552,"internally_reserved":0,"allocated":10227712,"data_stored":7096822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626695,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":319816,"kb_used_data":9980,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93425336,"statfs":{"total":95995035648,"available":95667544064,"internally_reserved":0,"allocated":10219520,"data_stored":7096703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822924,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306748,"kb_used_data":9392,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93438404,"statfs":{"total":95995035648,"available":95680925696,"internally_reserved":0,"allocated":9617408,"data_stored":6502153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019152,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":315080,"kb_used_data":9404,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93430072,"statfs":{"total":95995035648,"available":95672393728,"internally_reserved":0,"allocated":9629696,"data_stored":6506384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059592809,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":45428,"kb_used_data":9384,"kb_used_omap":26,"kb_used_meta":36005,"kb_avail":93699724,"statfs":{"total":95995035648,"available":95948517376,"internally_reserved":0,"allocated":9609216,"data_stored":6501604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":36869280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":591,"seq":2538325672048,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37304,"kb_used_data":9388,"kb_used_omap":41,"kb_used_meta":27862,"kb_avail":93707848,"statfs":{"total":95995035648,"available":95956836352,"internally_reserved":0,"allocated":9613312,"data_stored":6502126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42651,"internal_metadata":28531045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":557,"seq":2392296783991,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37232,"kb_used_data":9384,"kb_used_omap":32,"kb_used_meta":27807,"kb_avail":93707920,"statfs":{"total":95995035648,"available":95956910080,"internally_reserved":0,"allocated":9609216,"data_stored":6501549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33752,"internal_metadata":28474408},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":524,"seq":2250562863229,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64188,"kb_used_data":9968,"kb_used_omap":41,"kb_used_meta":54166,"kb_avail":93680964,"statfs":{"total":95995035648,"available":95929307136,"internally_reserved":0,"allocated":10207232,"data_stored":7092498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42694,"internal_metadata":55466298},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:38:55.545 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-13T14:38:55.545 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd dump 2024-09-13T14:38:55.792 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:56.161 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:56 smithi151 ceph-mon[104450]: from='client.35220 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:38:56.161 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:56 smithi151 ceph-mon[108966]: from='client.35220 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:38:56.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:56 smithi184 ceph-mon[92527]: from='client.35220 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:38:57.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:57 smithi184 ceph-mon[92527]: pgmap v1269: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:57.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:57 smithi151 ceph-mon[104450]: pgmap v1269: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:57.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:57 smithi151 ceph-mon[108966]: pgmap v1269: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:57.459 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 1078 2024-09-13T14:38:57.460 INFO:teuthology.orchestra.run.smithi151.stdout:fsid 2401fc50-71da-11ef-bceb-c7b262605968 2024-09-13T14:38:57.460 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-09-13T14:12:36.168205+0000 2024-09-13T14:38:57.460 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-09-13T14:37:54.701230+0000 2024-09-13T14:38:57.460 INFO:teuthology.orchestra.run.smithi151.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-13T14:38:57.460 INFO:teuthology.orchestra.run.smithi151.stdout:crush_version 22 2024-09-13T14:38:57.460 INFO:teuthology.orchestra.run.smithi151.stdout:full_ratio 0.95 2024-09-13T14:38:57.460 INFO:teuthology.orchestra.run.smithi151.stdout:backfillfull_ratio 0.9 2024-09-13T14:38:57.460 INFO:teuthology.orchestra.run.smithi151.stdout:nearfull_ratio 0.85 2024-09-13T14:38:57.460 INFO:teuthology.orchestra.run.smithi151.stdout:require_min_compat_client luminous 2024-09-13T14:38:57.460 INFO:teuthology.orchestra.run.smithi151.stdout:min_compat_client luminous 2024-09-13T14:38:57.460 INFO:teuthology.orchestra.run.smithi151.stdout:require_osd_release quincy 2024-09-13T14:38:57.461 INFO:teuthology.orchestra.run.smithi151.stdout:stretch_mode_enabled false 2024-09-13T14:38:57.461 INFO:teuthology.orchestra.run.smithi151.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-09-13T14:38:57.461 INFO:teuthology.orchestra.run.smithi151.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 56 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.00 2024-09-13T14:38:57.461 INFO:teuthology.orchestra.run.smithi151.stdout:pool 3 'cephfs.foo.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 62 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.50 2024-09-13T14:38:57.461 INFO:teuthology.orchestra.run.smithi151.stdout:pool 4 'cephfs.foo.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 63 flags hashpspool,bulk stripe_width 0 application cephfs read_balance_score 2.25 2024-09-13T14:38:57.461 INFO:teuthology.orchestra.run.smithi151.stdout:max_osd 8 2024-09-13T14:38:57.461 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 up in weight 1 up_from 524 up_thru 624 down_at 512 last_clean_interval [8,511) [v2:172.21.15.151:6802/2137494296,v1:172.21.15.151:6803/2137494296] [v2:172.21.15.151:6804/2137494296,v1:172.21.15.151:6805/2137494296] exists,up 0ad945ee-af7d-4f26-a1e9-18ed963194ad 2024-09-13T14:38:57.461 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 up in weight 1 up_from 557 up_thru 624 down_at 545 last_clean_interval [13,544) [v2:172.21.15.151:6810/3884881052,v1:172.21.15.151:6811/3884881052] [v2:172.21.15.151:6812/3884881052,v1:172.21.15.151:6813/3884881052] exists,up 09f5d705-684f-4e86-80a2-ec2292af09d3 2024-09-13T14:38:57.461 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 up in weight 1 up_from 591 up_thru 624 down_at 579 last_clean_interval [18,578) [v2:172.21.15.151:6818/322435630,v1:172.21.15.151:6819/322435630] [v2:172.21.15.151:6820/322435630,v1:172.21.15.151:6821/322435630] exists,up 6a63f8af-ff69-46bb-830f-03ea89fc78e3 2024-09-13T14:38:57.461 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 up in weight 1 up_from 624 up_thru 624 down_at 612 last_clean_interval [25,611) [v2:172.21.15.151:6826/2997668861,v1:172.21.15.151:6827/2997668861] [v2:172.21.15.151:6828/2997668861,v1:172.21.15.151:6829/2997668861] exists,up 770e8334-3b3a-46bc-9782-964b2a696660 2024-09-13T14:38:57.461 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 up in weight 1 up_from 30 up_thru 624 down_at 0 last_clean_interval [0,0) [v2:172.21.15.184:6800/3275553935,v1:172.21.15.184:6801/3275553935] [v2:172.21.15.184:6802/3275553935,v1:172.21.15.184:6803/3275553935] exists,up 75e258c0-1145-4aee-86aa-db2b63082cc8 2024-09-13T14:38:57.461 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 up in weight 1 up_from 36 up_thru 624 down_at 0 last_clean_interval [0,0) [v2:172.21.15.184:6808/2446419621,v1:172.21.15.184:6809/2446419621] [v2:172.21.15.184:6810/2446419621,v1:172.21.15.184:6811/2446419621] exists,up f56a5366-9d22-4960-933d-340aa280fbd1 2024-09-13T14:38:57.461 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6 up in weight 1 up_from 42 up_thru 612 down_at 0 last_clean_interval [0,0) [v2:172.21.15.184:6816/2214837203,v1:172.21.15.184:6817/2214837203] [v2:172.21.15.184:6818/2214837203,v1:172.21.15.184:6819/2214837203] exists,up e9826568-a728-45ab-8180-9777fe31f93a 2024-09-13T14:38:57.462 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7 up in weight 1 up_from 48 up_thru 624 down_at 0 last_clean_interval [0,0) [v2:172.21.15.184:6824/3378735997,v1:172.21.15.184:6825/3378735997] [v2:172.21.15.184:6826/3378735997,v1:172.21.15.184:6827/3378735997] exists,up d76869fe-5121-4467-8b56-1aadef0eaf65 2024-09-13T14:38:57.462 INFO:teuthology.orchestra.run.smithi151.stdout:pg_upmap_items 4.10 [1,2] 2024-09-13T14:38:57.462 INFO:teuthology.orchestra.run.smithi151.stdout:pg_upmap_items 4.18 [1,2] 2024-09-13T14:38:57.462 INFO:teuthology.orchestra.run.smithi151.stdout:pg_upmap_items 4.1b [1,5] 2024-09-13T14:38:57.462 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/1783031268 expires 2024-09-14T14:24:01.073069+0000 2024-09-13T14:38:57.462 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/1775071943 expires 2024-09-14T14:24:01.073069+0000 2024-09-13T14:38:57.462 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:6801/42840 expires 2024-09-14T14:24:01.073069+0000 2024-09-13T14:38:57.462 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/841256485 expires 2024-09-14T14:24:01.073069+0000 2024-09-13T14:38:57.462 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/1182681341 expires 2024-09-14T14:13:04.761383+0000 2024-09-13T14:38:57.462 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:6800/1057983082 expires 2024-09-14T14:13:27.478847+0000 2024-09-13T14:38:57.462 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/2889050140 expires 2024-09-14T14:18:18.344788+0000 2024-09-13T14:38:57.463 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:6801/4025698662 expires 2024-09-14T14:22:15.389006+0000 2024-09-13T14:38:57.463 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/2800235450 expires 2024-09-14T14:18:18.344788+0000 2024-09-13T14:38:57.463 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.184:0/3929228938 expires 2024-09-14T14:23:20.772174+0000 2024-09-13T14:38:57.463 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/1409232061 expires 2024-09-14T14:24:01.073069+0000 2024-09-13T14:38:57.463 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:6800/4025698662 expires 2024-09-14T14:22:15.389006+0000 2024-09-13T14:38:57.463 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.184:0/1999058007 expires 2024-09-14T14:23:20.772174+0000 2024-09-13T14:38:57.463 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:6801/1057983082 expires 2024-09-14T14:13:27.478847+0000 2024-09-13T14:38:57.463 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/514781427 expires 2024-09-14T14:22:15.389006+0000 2024-09-13T14:38:57.464 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/663506338 expires 2024-09-14T14:24:01.073069+0000 2024-09-13T14:38:57.464 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:6800/3383065173 expires 2024-09-14T14:13:04.761383+0000 2024-09-13T14:38:57.464 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.184:6834/205203765 expires 2024-09-14T14:23:20.772174+0000 2024-09-13T14:38:57.464 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/1530349337 expires 2024-09-14T14:22:15.389006+0000 2024-09-13T14:38:57.464 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/1427922964 expires 2024-09-14T14:13:27.478847+0000 2024-09-13T14:38:57.464 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/2666829207 expires 2024-09-14T14:13:27.478847+0000 2024-09-13T14:38:57.464 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:6801/2335297718 expires 2024-09-14T14:18:18.344788+0000 2024-09-13T14:38:57.464 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/1351186656 expires 2024-09-14T14:18:18.344788+0000 2024-09-13T14:38:57.464 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.184:6835/205203765 expires 2024-09-14T14:23:20.772174+0000 2024-09-13T14:38:57.464 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.184:0/3003411098 expires 2024-09-14T14:23:20.772174+0000 2024-09-13T14:38:57.464 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/4115173791 expires 2024-09-14T14:18:18.344788+0000 2024-09-13T14:38:57.464 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/3689756337 expires 2024-09-14T14:13:27.478847+0000 2024-09-13T14:38:57.464 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/1517910843 expires 2024-09-14T14:24:01.073069+0000 2024-09-13T14:38:57.465 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.184:0/3370242044 expires 2024-09-14T14:23:20.772174+0000 2024-09-13T14:38:57.465 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:6800/2335297718 expires 2024-09-14T14:18:18.344788+0000 2024-09-13T14:38:57.465 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:6800/42840 expires 2024-09-14T14:24:01.073069+0000 2024-09-13T14:38:57.465 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/1488764319 expires 2024-09-14T14:22:15.389006+0000 2024-09-13T14:38:57.465 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.184:0/2995983270 expires 2024-09-14T14:23:20.772174+0000 2024-09-13T14:38:57.465 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/3946777511 expires 2024-09-14T14:13:04.761383+0000 2024-09-13T14:38:57.465 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/624042464 expires 2024-09-14T14:13:04.761383+0000 2024-09-13T14:38:57.465 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/2201280183 expires 2024-09-14T14:22:15.389006+0000 2024-09-13T14:38:57.465 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:6801/3383065173 expires 2024-09-14T14:13:04.761383+0000 2024-09-13T14:38:57.466 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.184:0/3047962606 expires 2024-09-14T14:23:20.772174+0000 2024-09-13T14:38:58.126 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 524 up_thru 624 down_at 512 last_clean_interval [8,511) [v2:172.21.15.151:6802/2137494296,v1:172.21.15.151:6803/2137494296] [v2:172.21.15.151:6804/2137494296,v1:172.21.15.151:6805/2137494296] exists,up 0ad945ee-af7d-4f26-a1e9-18ed963194ad', 'osd.1 up in weight 1 up_from 557 up_thru 624 down_at 545 last_clean_interval [13,544) [v2:172.21.15.151:6810/3884881052,v1:172.21.15.151:6811/3884881052] [v2:172.21.15.151:6812/3884881052,v1:172.21.15.151:6813/3884881052] exists,up 09f5d705-684f-4e86-80a2-ec2292af09d3', 'osd.2 up in weight 1 up_from 591 up_thru 624 down_at 579 last_clean_interval [18,578) [v2:172.21.15.151:6818/322435630,v1:172.21.15.151:6819/322435630] [v2:172.21.15.151:6820/322435630,v1:172.21.15.151:6821/322435630] exists,up 6a63f8af-ff69-46bb-830f-03ea89fc78e3', 'osd.3 up in weight 1 up_from 624 up_thru 624 down_at 612 last_clean_interval [25,611) [v2:172.21.15.151:6826/2997668861,v1:172.21.15.151:6827/2997668861] [v2:172.21.15.151:6828/2997668861,v1:172.21.15.151:6829/2997668861] exists,up 770e8334-3b3a-46bc-9782-964b2a696660', 'osd.4 up in weight 1 up_from 30 up_thru 624 down_at 0 last_clean_interval [0,0) [v2:172.21.15.184:6800/3275553935,v1:172.21.15.184:6801/3275553935] [v2:172.21.15.184:6802/3275553935,v1:172.21.15.184:6803/3275553935] exists,up 75e258c0-1145-4aee-86aa-db2b63082cc8', 'osd.5 up in weight 1 up_from 36 up_thru 624 down_at 0 last_clean_interval [0,0) [v2:172.21.15.184:6808/2446419621,v1:172.21.15.184:6809/2446419621] [v2:172.21.15.184:6810/2446419621,v1:172.21.15.184:6811/2446419621] exists,up f56a5366-9d22-4960-933d-340aa280fbd1', 'osd.6 up in weight 1 up_from 42 up_thru 612 down_at 0 last_clean_interval [0,0) [v2:172.21.15.184:6816/2214837203,v1:172.21.15.184:6817/2214837203] [v2:172.21.15.184:6818/2214837203,v1:172.21.15.184:6819/2214837203] exists,up e9826568-a728-45ab-8180-9777fe31f93a', 'osd.7 up in weight 1 up_from 48 up_thru 624 down_at 0 last_clean_interval [0,0) [v2:172.21.15.184:6824/3378735997,v1:172.21.15.184:6825/3378735997] [v2:172.21.15.184:6826/3378735997,v1:172.21.15.184:6827/3378735997] exists,up d76869fe-5121-4467-8b56-1aadef0eaf65'] 2024-09-13T14:38:58.126 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon mon.a config show 2024-09-13T14:38:58.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:58 smithi184 ceph-mon[92527]: pgmap v1270: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:58.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:38:58 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/4259990924' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-13T14:38:58.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:58 smithi151 ceph-mon[104450]: pgmap v1270: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:58.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:38:58 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/4259990924' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-13T14:38:58.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:58 smithi151 ceph-mon[108966]: pgmap v1270: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:38:58.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:38:58 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/4259990924' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-13T14:38:58.371 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:00.107 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:39:00.107 INFO:teuthology.orchestra.run.smithi151.stdout: "name": "mon.a", 2024-09-13T14:39:00.107 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster": "ceph", 2024-09-13T14:39:00.107 INFO:teuthology.orchestra.run.smithi151.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-13T14:39:00.107 INFO:teuthology.orchestra.run.smithi151.stdout: "admin_socket_mode": "", 2024-09-13T14:39:00.108 INFO:teuthology.orchestra.run.smithi151.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-09-13T14:39:00.108 INFO:teuthology.orchestra.run.smithi151.stdout: "auth_client_required": "cephx, none", 2024-09-13T14:39:00.108 INFO:teuthology.orchestra.run.smithi151.stdout: "auth_cluster_required": "cephx", 2024-09-13T14:39:00.108 INFO:teuthology.orchestra.run.smithi151.stdout: "auth_debug": "true", 2024-09-13T14:39:00.108 INFO:teuthology.orchestra.run.smithi151.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-13T14:39:00.108 INFO:teuthology.orchestra.run.smithi151.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-13T14:39:00.108 INFO:teuthology.orchestra.run.smithi151.stdout: "auth_service_required": "cephx", 2024-09-13T14:39:00.108 INFO:teuthology.orchestra.run.smithi151.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-13T14:39:00.108 INFO:teuthology.orchestra.run.smithi151.stdout: "auth_supported": "", 2024-09-13T14:39:00.108 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_aio": "true", 2024-09-13T14:39:00.108 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-13T14:39:00.108 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_aio_poll_ms": "250", 2024-09-13T14:39:00.108 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_aio_reap_max": "16", 2024-09-13T14:39:00.109 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_async_discard_threads": "0", 2024-09-13T14:39:00.109 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_block_size": "4096", 2024-09-13T14:39:00.109 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_debug_aio": "false", 2024-09-13T14:39:00.109 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-13T14:39:00.109 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-13T14:39:00.109 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_debug_inflight_ios": "false", 2024-09-13T14:39:00.109 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_enable_discard": "false", 2024-09-13T14:39:00.109 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_flock_retry": "3", 2024-09-13T14:39:00.109 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-13T14:39:00.109 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_inject_crash": "0", 2024-09-13T14:39:00.109 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-13T14:39:00.109 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_ioring": "false", 2024-09-13T14:39:00.110 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_ioring_hipri": "false", 2024-09-13T14:39:00.110 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-13T14:39:00.110 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-13T14:39:00.110 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-13T14:39:00.110 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-13T14:39:00.110 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-13T14:39:00.110 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-13T14:39:00.110 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_type": "", 2024-09-13T14:39:00.110 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_alloc_size": "1048576", 2024-09-13T14:39:00.110 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_allocator": "hybrid", 2024-09-13T14:39:00.110 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_buffered_io": "true", 2024-09-13T14:39:00.110 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_check_for_zeros": "false", 2024-09-13T14:39:00.111 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-13T14:39:00.111 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-13T14:39:00.111 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_compact_log_sync": "false", 2024-09-13T14:39:00.111 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-13T14:39:00.111 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-13T14:39:00.111 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-13T14:39:00.111 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-13T14:39:00.111 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_max_log_runway": "4194304", 2024-09-13T14:39:00.111 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_max_prefetch": "1048576", 2024-09-13T14:39:00.111 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_min_flush_size": "524288", 2024-09-13T14:39:00.111 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_min_log_runway": "1048576", 2024-09-13T14:39:00.111 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_replay_recovery": "false", 2024-09-13T14:39:00.111 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-13T14:39:00.112 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-13T14:39:00.112 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_sync_write": "false", 2024-09-13T14:39:00.112 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-13T14:39:00.112 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-13T14:39:00.112 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-13T14:39:00.112 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_allocation_from_file": "true", 2024-09-13T14:39:00.112 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_allocator": "hybrid", 2024-09-13T14:39:00.112 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_async_db_compaction": "true", 2024-09-13T14:39:00.112 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-13T14:39:00.112 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-13T14:39:00.112 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-13T14:39:00.112 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-13T14:39:00.113 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_bdev_label_multi": "true", 2024-09-13T14:39:00.113 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-13T14:39:00.113 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-13T14:39:00.113 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-13T14:39:00.113 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-13T14:39:00.113 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-13T14:39:00.113 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_block_create": "true", 2024-09-13T14:39:00.113 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_block_db_create": "false", 2024-09-13T14:39:00.113 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_block_db_path": "", 2024-09-13T14:39:00.113 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_block_db_size": "0", 2024-09-13T14:39:00.113 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_block_path": "", 2024-09-13T14:39:00.113 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_block_preallocate_file": "false", 2024-09-13T14:39:00.113 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_block_size": "107374182400", 2024-09-13T14:39:00.113 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_block_wal_create": "false", 2024-09-13T14:39:00.114 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_block_wal_path": "", 2024-09-13T14:39:00.114 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_block_wal_size": "100663296", 2024-09-13T14:39:00.114 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_bluefs": "true", 2024-09-13T14:39:00.114 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-13T14:39:00.114 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-13T14:39:00.114 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-13T14:39:00.114 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-09-13T14:39:00.114 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-13T14:39:00.114 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T14:39:00.114 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T14:39:00.114 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-13T14:39:00.114 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T14:39:00.114 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_autotune": "true", 2024-09-13T14:39:00.115 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-13T14:39:00.115 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-13T14:39:00.115 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-13T14:39:00.115 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-13T14:39:00.115 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_size": "0", 2024-09-13T14:39:00.115 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-13T14:39:00.115 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-13T14:39:00.115 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-13T14:39:00.115 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-13T14:39:00.115 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_type": "2q", 2024-09-13T14:39:00.115 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-13T14:39:00.115 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_clone_cow": "true", 2024-09-13T14:39:00.115 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-13T14:39:00.115 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-13T14:39:00.116 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-13T14:39:00.116 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-13T14:39:00.116 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-13T14:39:00.116 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-13T14:39:00.116 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-13T14:39:00.116 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_compression_mode": "none", 2024-09-13T14:39:00.116 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-13T14:39:00.116 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_csum_type": "crc32c", 2024-09-13T14:39:00.116 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-09-13T14:39:00.116 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-13T14:39:00.116 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_freelist": "false", 2024-09-13T14:39:00.116 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-13T14:39:00.116 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-13T14:39:00.117 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-13T14:39:00.117 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-13T14:39:00.117 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-13T14:39:00.117 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-13T14:39:00.117 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-13T14:39:00.117 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-13T14:39:00.117 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-13T14:39:00.117 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-13T14:39:00.117 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-13T14:39:00.117 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-13T14:39:00.117 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_small_allocations": "0", 2024-09-13T14:39:00.117 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-13T14:39:00.118 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_default_buffered_read": "true", 2024-09-13T14:39:00.118 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_default_buffered_write": "false", 2024-09-13T14:39:00.118 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-13T14:39:00.118 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-13T14:39:00.118 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-13T14:39:00.118 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-13T14:39:00.118 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-13T14:39:00.118 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-13T14:39:00.118 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-13T14:39:00.118 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-13T14:39:00.118 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-13T14:39:00.118 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fail_eio": "false", 2024-09-13T14:39:00.118 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-13T14:39:00.118 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-13T14:39:00.119 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-13T14:39:00.119 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-13T14:39:00.119 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-13T14:39:00.119 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-13T14:39:00.119 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_on_mount": "false", 2024-09-13T14:39:00.119 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-13T14:39:00.119 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_on_umount": "false", 2024-09-13T14:39:00.119 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-13T14:39:00.119 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-13T14:39:00.119 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-13T14:39:00.119 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-13T14:39:00.119 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-13T14:39:00.119 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-13T14:39:00.120 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-13T14:39:00.120 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-13T14:39:00.120 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_ignore_data_csum": "false", 2024-09-13T14:39:00.120 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-13T14:39:00.120 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-13T14:39:00.120 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-13T14:39:00.120 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-13T14:39:00.120 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_log_op_age": "5.000000", 2024-09-13T14:39:00.120 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_max_alloc_size": "0", 2024-09-13T14:39:00.121 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_max_blob_size": "0", 2024-09-13T14:39:00.121 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-13T14:39:00.121 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-13T14:39:00.121 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-13T14:39:00.121 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_max_deferred_txc": "32", 2024-09-13T14:39:00.121 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_min_alloc_size": "0", 2024-09-13T14:39:00.121 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-13T14:39:00.121 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-13T14:39:00.121 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_nid_prealloc": "1024", 2024-09-13T14:39:00.121 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-13T14:39:00.121 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-13T14:39:00.121 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-13T14:39:00.121 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-13T14:39:00.122 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_retry_disk_reads": "3", 2024-09-13T14:39:00.122 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_rocksdb_cf": "true", 2024-09-13T14:39:00.122 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-13T14:39:00.122 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-13T14:39:00.122 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-13T14:39:00.122 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-13T14:39:00.122 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-13T14:39:00.122 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-13T14:39:00.122 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-13T14:39:00.122 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-13T14:39:00.122 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_spdk_mem": "512", 2024-09-13T14:39:00.122 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-13T14:39:00.123 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-13T14:39:00.123 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-13T14:39:00.123 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-13T14:39:00.123 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-13T14:39:00.123 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-13T14:39:00.123 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-13T14:39:00.123 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_tracing": "false", 2024-09-13T14:39:00.123 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-13T14:39:00.123 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-13T14:39:00.123 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-13T14:39:00.123 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-13T14:39:00.123 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-13T14:39:00.123 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-13T14:39:00.124 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-13T14:39:00.124 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-13T14:39:00.124 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-13T14:39:00.124 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_write_v2": "false", 2024-09-13T14:39:00.124 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_write_v2_random": "false", 2024-09-13T14:39:00.124 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_zero_block_detection": "false", 2024-09-13T14:39:00.124 INFO:teuthology.orchestra.run.smithi151.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-13T14:39:00.124 INFO:teuthology.orchestra.run.smithi151.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-13T14:39:00.124 INFO:teuthology.orchestra.run.smithi151.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-13T14:39:00.124 INFO:teuthology.orchestra.run.smithi151.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-13T14:39:00.124 INFO:teuthology.orchestra.run.smithi151.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-13T14:39:00.124 INFO:teuthology.orchestra.run.smithi151.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-13T14:39:00.124 INFO:teuthology.orchestra.run.smithi151.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-13T14:39:00.125 INFO:teuthology.orchestra.run.smithi151.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-13T14:39:00.125 INFO:teuthology.orchestra.run.smithi151.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-13T14:39:00.125 INFO:teuthology.orchestra.run.smithi151.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-13T14:39:00.125 INFO:teuthology.orchestra.run.smithi151.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-13T14:39:00.125 INFO:teuthology.orchestra.run.smithi151.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-13T14:39:00.125 INFO:teuthology.orchestra.run.smithi151.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-13T14:39:00.125 INFO:teuthology.orchestra.run.smithi151.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-13T14:39:00.125 INFO:teuthology.orchestra.run.smithi151.stdout: "cephx_cluster_require_signatures": "false", 2024-09-13T14:39:00.125 INFO:teuthology.orchestra.run.smithi151.stdout: "cephx_cluster_require_version": "2", 2024-09-13T14:39:00.125 INFO:teuthology.orchestra.run.smithi151.stdout: "cephx_require_signatures": "false", 2024-09-13T14:39:00.125 INFO:teuthology.orchestra.run.smithi151.stdout: "cephx_require_version": "2", 2024-09-13T14:39:00.125 INFO:teuthology.orchestra.run.smithi151.stdout: "cephx_service_require_signatures": "false", 2024-09-13T14:39:00.126 INFO:teuthology.orchestra.run.smithi151.stdout: "cephx_service_require_version": "2", 2024-09-13T14:39:00.126 INFO:teuthology.orchestra.run.smithi151.stdout: "cephx_sign_messages": "true", 2024-09-13T14:39:00.126 INFO:teuthology.orchestra.run.smithi151.stdout: "chdir": "", 2024-09-13T14:39:00.126 INFO:teuthology.orchestra.run.smithi151.stdout: "client_acl_type": "", 2024-09-13T14:39:00.126 INFO:teuthology.orchestra.run.smithi151.stdout: "client_asio_thread_count": "2", 2024-09-13T14:39:00.126 INFO:teuthology.orchestra.run.smithi151.stdout: "client_cache_mid": "0.750000", 2024-09-13T14:39:00.126 INFO:teuthology.orchestra.run.smithi151.stdout: "client_cache_size": "16384", 2024-09-13T14:39:00.126 INFO:teuthology.orchestra.run.smithi151.stdout: "client_caps_release_delay": "5", 2024-09-13T14:39:00.126 INFO:teuthology.orchestra.run.smithi151.stdout: "client_check_pool_perm": "true", 2024-09-13T14:39:00.126 INFO:teuthology.orchestra.run.smithi151.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-13T14:39:00.126 INFO:teuthology.orchestra.run.smithi151.stdout: "client_debug_force_sync_read": "false", 2024-09-13T14:39:00.126 INFO:teuthology.orchestra.run.smithi151.stdout: "client_debug_getattr_caps": "false", 2024-09-13T14:39:00.126 INFO:teuthology.orchestra.run.smithi151.stdout: "client_debug_inject_features": "", 2024-09-13T14:39:00.127 INFO:teuthology.orchestra.run.smithi151.stdout: "client_debug_inject_tick_delay": "0", 2024-09-13T14:39:00.127 INFO:teuthology.orchestra.run.smithi151.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-13T14:39:00.127 INFO:teuthology.orchestra.run.smithi151.stdout: "client_die_on_failed_remount": "false", 2024-09-13T14:39:00.127 INFO:teuthology.orchestra.run.smithi151.stdout: "client_dirsize_rbytes": "true", 2024-09-13T14:39:00.127 INFO:teuthology.orchestra.run.smithi151.stdout: "client_force_lazyio": "false", 2024-09-13T14:39:00.127 INFO:teuthology.orchestra.run.smithi151.stdout: "client_fs": "", 2024-09-13T14:39:00.129 INFO:teuthology.orchestra.run.smithi151.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-13T14:39:00.129 INFO:teuthology.orchestra.run.smithi151.stdout: "client_inject_release_failure": "false", 2024-09-13T14:39:00.129 INFO:teuthology.orchestra.run.smithi151.stdout: "client_max_inline_size": "4096", 2024-09-13T14:39:00.129 INFO:teuthology.orchestra.run.smithi151.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-13T14:39:00.130 INFO:teuthology.orchestra.run.smithi151.stdout: "client_mds_namespace": "", 2024-09-13T14:39:00.130 INFO:teuthology.orchestra.run.smithi151.stdout: "client_metadata": "", 2024-09-13T14:39:00.130 INFO:teuthology.orchestra.run.smithi151.stdout: "client_mount_gid": "-1", 2024-09-13T14:39:00.130 INFO:teuthology.orchestra.run.smithi151.stdout: "client_mount_timeout": "300", 2024-09-13T14:39:00.130 INFO:teuthology.orchestra.run.smithi151.stdout: "client_mount_uid": "-1", 2024-09-13T14:39:00.130 INFO:teuthology.orchestra.run.smithi151.stdout: "client_mountpoint": "/", 2024-09-13T14:39:00.130 INFO:teuthology.orchestra.run.smithi151.stdout: "client_notify_timeout": "10", 2024-09-13T14:39:00.130 INFO:teuthology.orchestra.run.smithi151.stdout: "client_oc": "true", 2024-09-13T14:39:00.130 INFO:teuthology.orchestra.run.smithi151.stdout: "client_oc_max_dirty": "104857600", 2024-09-13T14:39:00.130 INFO:teuthology.orchestra.run.smithi151.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-13T14:39:00.130 INFO:teuthology.orchestra.run.smithi151.stdout: "client_oc_max_objects": "1000", 2024-09-13T14:39:00.130 INFO:teuthology.orchestra.run.smithi151.stdout: "client_oc_size": "209715200", 2024-09-13T14:39:00.130 INFO:teuthology.orchestra.run.smithi151.stdout: "client_oc_target_dirty": "8388608", 2024-09-13T14:39:00.130 INFO:teuthology.orchestra.run.smithi151.stdout: "client_permissions": "true", 2024-09-13T14:39:00.131 INFO:teuthology.orchestra.run.smithi151.stdout: "client_quota": "true", 2024-09-13T14:39:00.131 INFO:teuthology.orchestra.run.smithi151.stdout: "client_quota_df": "true", 2024-09-13T14:39:00.131 INFO:teuthology.orchestra.run.smithi151.stdout: "client_readahead_max_bytes": "0", 2024-09-13T14:39:00.131 INFO:teuthology.orchestra.run.smithi151.stdout: "client_readahead_max_periods": "4", 2024-09-13T14:39:00.131 INFO:teuthology.orchestra.run.smithi151.stdout: "client_readahead_min": "131072", 2024-09-13T14:39:00.131 INFO:teuthology.orchestra.run.smithi151.stdout: "client_reconnect_stale": "false", 2024-09-13T14:39:00.131 INFO:teuthology.orchestra.run.smithi151.stdout: "client_shutdown_timeout": "30", 2024-09-13T14:39:00.131 INFO:teuthology.orchestra.run.smithi151.stdout: "client_snapdir": ".snap", 2024-09-13T14:39:00.131 INFO:teuthology.orchestra.run.smithi151.stdout: "client_tick_interval": "1", 2024-09-13T14:39:00.131 INFO:teuthology.orchestra.run.smithi151.stdout: "client_trace": "", 2024-09-13T14:39:00.131 INFO:teuthology.orchestra.run.smithi151.stdout: "client_try_dentry_invalidate": "false", 2024-09-13T14:39:00.131 INFO:teuthology.orchestra.run.smithi151.stdout: "client_use_faked_inos": "false", 2024-09-13T14:39:00.131 INFO:teuthology.orchestra.run.smithi151.stdout: "client_use_random_mds": "false", 2024-09-13T14:39:00.132 INFO:teuthology.orchestra.run.smithi151.stdout: "clog_to_graylog": "false", 2024-09-13T14:39:00.132 INFO:teuthology.orchestra.run.smithi151.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-13T14:39:00.132 INFO:teuthology.orchestra.run.smithi151.stdout: "clog_to_graylog_port": "12201", 2024-09-13T14:39:00.132 INFO:teuthology.orchestra.run.smithi151.stdout: "clog_to_monitors": "default=true", 2024-09-13T14:39:00.132 INFO:teuthology.orchestra.run.smithi151.stdout: "clog_to_syslog": "false", 2024-09-13T14:39:00.132 INFO:teuthology.orchestra.run.smithi151.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-13T14:39:00.132 INFO:teuthology.orchestra.run.smithi151.stdout: "clog_to_syslog_level": "info", 2024-09-13T14:39:00.132 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "-", 2024-09-13T14:39:00.132 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_network": "", 2024-09-13T14:39:00.132 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_network_interface": "", 2024-09-13T14:39:00.132 INFO:teuthology.orchestra.run.smithi151.stdout: "compressor_zlib_isal": "false", 2024-09-13T14:39:00.132 INFO:teuthology.orchestra.run.smithi151.stdout: "compressor_zlib_level": "5", 2024-09-13T14:39:00.133 INFO:teuthology.orchestra.run.smithi151.stdout: "compressor_zlib_winsize": "-15", 2024-09-13T14:39:00.133 INFO:teuthology.orchestra.run.smithi151.stdout: "compressor_zstd_level": "1", 2024-09-13T14:39:00.133 INFO:teuthology.orchestra.run.smithi151.stdout: "container_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:39:00.133 INFO:teuthology.orchestra.run.smithi151.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-13T14:39:00.133 INFO:teuthology.orchestra.run.smithi151.stdout: "crimson_alien_op_num_threads": "6", 2024-09-13T14:39:00.133 INFO:teuthology.orchestra.run.smithi151.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-13T14:39:00.133 INFO:teuthology.orchestra.run.smithi151.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-13T14:39:00.133 INFO:teuthology.orchestra.run.smithi151.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-13T14:39:00.133 INFO:teuthology.orchestra.run.smithi151.stdout: "crimson_osd_stat_interval": "0", 2024-09-13T14:39:00.133 INFO:teuthology.orchestra.run.smithi151.stdout: "crimson_seastar_cpu_cores": "", 2024-09-13T14:39:00.133 INFO:teuthology.orchestra.run.smithi151.stdout: "crimson_seastar_num_threads": "0", 2024-09-13T14:39:00.133 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_location": "", 2024-09-13T14:39:00.134 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_location_hook": "", 2024-09-13T14:39:00.134 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_location_hook_timeout": "10", 2024-09-13T14:39:00.134 INFO:teuthology.orchestra.run.smithi151.stdout: "daemonize": "false", 2024-09-13T14:39:00.134 INFO:teuthology.orchestra.run.smithi151.stdout: "daos_pool": "tank", 2024-09-13T14:39:00.134 INFO:teuthology.orchestra.run.smithi151.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-13T14:39:00.134 INFO:teuthology.orchestra.run.smithi151.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-13T14:39:00.134 INFO:teuthology.orchestra.run.smithi151.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-13T14:39:00.134 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_alienstore": "0/5", 2024-09-13T14:39:00.134 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_allow_any_pool_priority": "false", 2024-09-13T14:39:00.134 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_asok": "1/5", 2024-09-13T14:39:00.134 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_asok_assert_abort": "false", 2024-09-13T14:39:00.134 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_asserts_on_shutdown": "true", 2024-09-13T14:39:00.134 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_auth": "1/5", 2024-09-13T14:39:00.135 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_bdev": "1/3", 2024-09-13T14:39:00.135 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_bluefs": "1/5", 2024-09-13T14:39:00.135 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_bluestore": "1/5", 2024-09-13T14:39:00.135 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_buffer": "0/1", 2024-09-13T14:39:00.135 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_ceph_dedup": "0/5", 2024-09-13T14:39:00.135 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_ceph_exporter": "1/5", 2024-09-13T14:39:00.135 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_cephfs_mirror": "0/5", 2024-09-13T14:39:00.135 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_cephsqlite": "0/5", 2024-09-13T14:39:00.135 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_client": "0/5", 2024-09-13T14:39:00.135 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_compressor": "1/5", 2024-09-13T14:39:00.135 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_context": "0/1", 2024-09-13T14:39:00.135 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_crimson_interrupt": "0/5", 2024-09-13T14:39:00.135 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_crush": "1/1", 2024-09-13T14:39:00.135 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_crypto": "1/5", 2024-09-13T14:39:00.136 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_cyanstore": "0/5", 2024-09-13T14:39:00.136 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_deliberately_leak_memory": "false", 2024-09-13T14:39:00.136 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_disable_randomized_ping": "false", 2024-09-13T14:39:00.136 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_dpdk": "1/5", 2024-09-13T14:39:00.136 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_eventtrace": "1/5", 2024-09-13T14:39:00.136 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_filer": "0/1", 2024-09-13T14:39:00.136 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_filestore": "1/3", 2024-09-13T14:39:00.136 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_finisher": "1/1", 2024-09-13T14:39:00.136 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_fuse": "1/5", 2024-09-13T14:39:00.136 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_heartbeat_testing_span": "0", 2024-09-13T14:39:00.136 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_heartbeatmap": "1/5", 2024-09-13T14:39:00.136 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-13T14:39:00.136 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_javaclient": "1/5", 2024-09-13T14:39:00.137 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_journal": "1/3", 2024-09-13T14:39:00.137 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_journaler": "0/5", 2024-09-13T14:39:00.137 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_kstore": "1/5", 2024-09-13T14:39:00.137 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_lockdep": "0/1", 2024-09-13T14:39:00.137 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mclock": "1/5", 2024-09-13T14:39:00.137 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mds": "1/5", 2024-09-13T14:39:00.137 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mds_balancer": "1/5", 2024-09-13T14:39:00.137 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mds_locker": "1/5", 2024-09-13T14:39:00.137 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mds_log": "1/5", 2024-09-13T14:39:00.137 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mds_log_expire": "1/5", 2024-09-13T14:39:00.137 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mds_migrator": "1/5", 2024-09-13T14:39:00.137 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mds_quiesce": "3/5", 2024-09-13T14:39:00.138 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_memstore": "1/5", 2024-09-13T14:39:00.138 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mgr": "2/5", 2024-09-13T14:39:00.138 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mgrc": "1/5", 2024-09-13T14:39:00.138 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mon": "20/20", 2024-09-13T14:39:00.138 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_monc": "0/10", 2024-09-13T14:39:00.138 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_ms": "1/1", 2024-09-13T14:39:00.138 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_none": "0/5", 2024-09-13T14:39:00.138 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_objclass": "0/5", 2024-09-13T14:39:00.138 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_objectcacher": "0/5", 2024-09-13T14:39:00.138 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_objecter": "0/1", 2024-09-13T14:39:00.138 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_optracker": "0/5", 2024-09-13T14:39:00.138 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_osd": "1/5", 2024-09-13T14:39:00.138 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_paxos": "20/20", 2024-09-13T14:39:00.139 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_perfcounter": "1/5", 2024-09-13T14:39:00.139 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_prioritycache": "1/5", 2024-09-13T14:39:00.139 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rados": "0/5", 2024-09-13T14:39:00.139 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rbd": "0/5", 2024-09-13T14:39:00.139 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rbd_mirror": "0/5", 2024-09-13T14:39:00.139 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rbd_pwl": "0/5", 2024-09-13T14:39:00.139 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rbd_replay": "0/5", 2024-09-13T14:39:00.139 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_refs": "0/0", 2024-09-13T14:39:00.139 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_reserver": "1/1", 2024-09-13T14:39:00.140 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rgw": "1/5", 2024-09-13T14:39:00.140 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rgw_access": "1/5", 2024-09-13T14:39:00.140 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rgw_datacache": "1/5", 2024-09-13T14:39:00.140 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rgw_dbstore": "1/5", 2024-09-13T14:39:00.140 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rgw_flight": "1/5", 2024-09-13T14:39:00.140 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-13T14:39:00.140 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rgw_notification": "1/5", 2024-09-13T14:39:00.140 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rgw_sync": "1/5", 2024-09-13T14:39:00.140 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rocksdb": "4/5", 2024-09-13T14:39:00.140 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore": "0/5", 2024-09-13T14:39:00.140 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_backref": "0/5", 2024-09-13T14:39:00.140 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_cache": "0/5", 2024-09-13T14:39:00.141 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_cleaner": "0/5", 2024-09-13T14:39:00.141 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_device": "0/5", 2024-09-13T14:39:00.141 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_epm": "0/5", 2024-09-13T14:39:00.141 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-13T14:39:00.141 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_journal": "0/5", 2024-09-13T14:39:00.141 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_lba": "0/5", 2024-09-13T14:39:00.141 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_odata": "0/5", 2024-09-13T14:39:00.141 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_omap": "0/5", 2024-09-13T14:39:00.141 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_onode": "0/5", 2024-09-13T14:39:00.141 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_t": "0/5", 2024-09-13T14:39:00.141 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_tm": "0/5", 2024-09-13T14:39:00.141 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_striper": "0/1", 2024-09-13T14:39:00.141 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_test": "0/5", 2024-09-13T14:39:00.142 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_throttle": "1/1", 2024-09-13T14:39:00.142 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_timer": "0/1", 2024-09-13T14:39:00.142 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_tp": "0/5", 2024-09-13T14:39:00.142 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_trace": "1/5", 2024-09-13T14:39:00.142 INFO:teuthology.orchestra.run.smithi151.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-13T14:39:00.142 INFO:teuthology.orchestra.run.smithi151.stdout: "device_failure_prediction_mode": "none", 2024-09-13T14:39:00.142 INFO:teuthology.orchestra.run.smithi151.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-13T14:39:00.142 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-13T14:39:00.142 INFO:teuthology.orchestra.run.smithi151.stdout: "err_to_graylog": "false", 2024-09-13T14:39:00.142 INFO:teuthology.orchestra.run.smithi151.stdout: "err_to_journald": "false", 2024-09-13T14:39:00.142 INFO:teuthology.orchestra.run.smithi151.stdout: "err_to_stderr": "true", 2024-09-13T14:39:00.142 INFO:teuthology.orchestra.run.smithi151.stdout: "err_to_syslog": "false", 2024-09-13T14:39:00.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event_tracing": "false", 2024-09-13T14:39:00.143 INFO:teuthology.orchestra.run.smithi151.stdout: "exporter_addr": "0.0.0.0", 2024-09-13T14:39:00.143 INFO:teuthology.orchestra.run.smithi151.stdout: "exporter_cert_file": "", 2024-09-13T14:39:00.143 INFO:teuthology.orchestra.run.smithi151.stdout: "exporter_http_port": "9926", 2024-09-13T14:39:00.143 INFO:teuthology.orchestra.run.smithi151.stdout: "exporter_key_file": "", 2024-09-13T14:39:00.143 INFO:teuthology.orchestra.run.smithi151.stdout: "exporter_prio_limit": "5", 2024-09-13T14:39:00.143 INFO:teuthology.orchestra.run.smithi151.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-13T14:39:00.143 INFO:teuthology.orchestra.run.smithi151.stdout: "exporter_sort_metrics": "true", 2024-09-13T14:39:00.143 INFO:teuthology.orchestra.run.smithi151.stdout: "exporter_stats_period": "5", 2024-09-13T14:39:00.143 INFO:teuthology.orchestra.run.smithi151.stdout: "fake_statfs_for_testing": "0", 2024-09-13T14:39:00.143 INFO:teuthology.orchestra.run.smithi151.stdout: "fatal_signal_handlers": "true", 2024-09-13T14:39:00.143 INFO:teuthology.orchestra.run.smithi151.stdout: "filer_max_purge_ops": "10", 2024-09-13T14:39:00.143 INFO:teuthology.orchestra.run.smithi151.stdout: "filer_max_truncate_ops": "128", 2024-09-13T14:39:00.144 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_apply_finisher_threads": "1", 2024-09-13T14:39:00.144 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_blackhole": "false", 2024-09-13T14:39:00.144 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_btrfs_clone_range": "true", 2024-09-13T14:39:00.144 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_btrfs_snap": "true", 2024-09-13T14:39:00.144 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_caller_concurrency": "10", 2024-09-13T14:39:00.144 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_collect_device_partition_information": "true", 2024-09-13T14:39:00.144 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_commit_timeout": "600.000000", 2024-09-13T14:39:00.144 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_debug_inject_read_err": "false", 2024-09-13T14:39:00.144 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_debug_omap_check": "false", 2024-09-13T14:39:00.144 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_debug_verify_split": "false", 2024-09-13T14:39:00.144 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_dump_file": "", 2024-09-13T14:39:00.144 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-13T14:39:00.144 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-13T14:39:00.145 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_fadvise": "true", 2024-09-13T14:39:00.145 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_fail_eio": "true", 2024-09-13T14:39:00.145 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_fd_cache_shards": "16", 2024-09-13T14:39:00.145 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_fd_cache_size": "128", 2024-09-13T14:39:00.145 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_fiemap": "false", 2024-09-13T14:39:00.145 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_fiemap_threshold": "4096", 2024-09-13T14:39:00.145 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-13T14:39:00.145 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-13T14:39:00.145 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_inject_stall": "0", 2024-09-13T14:39:00.145 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_journal_parallel": "false", 2024-09-13T14:39:00.145 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_journal_trailing": "false", 2024-09-13T14:39:00.145 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_journal_writeahead": "false", 2024-09-13T14:39:00.145 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_kill_at": "0", 2024-09-13T14:39:00.146 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-13T14:39:00.146 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-13T14:39:00.146 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-13T14:39:00.146 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-13T14:39:00.146 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-13T14:39:00.146 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_inline_xattrs": "0", 2024-09-13T14:39:00.146 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-13T14:39:00.146 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-13T14:39:00.146 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-13T14:39:00.146 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-13T14:39:00.146 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_xattr_value_size": "0", 2024-09-13T14:39:00.146 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-13T14:39:00.146 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-13T14:39:00.147 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-13T14:39:00.147 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_merge_threshold": "-10", 2024-09-13T14:39:00.147 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-13T14:39:00.147 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_odsync_write": "false", 2024-09-13T14:39:00.147 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_omap_backend": "rocksdb", 2024-09-13T14:39:00.147 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_omap_backend_path": "", 2024-09-13T14:39:00.147 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-13T14:39:00.147 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-13T14:39:00.147 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-13T14:39:00.147 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_op_thread_timeout": "60", 2024-09-13T14:39:00.147 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_op_threads": "2", 2024-09-13T14:39:00.147 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_punch_hole": "false", 2024-09-13T14:39:00.147 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-13T14:39:00.148 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-13T14:39:00.148 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-13T14:39:00.148 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-13T14:39:00.148 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-13T14:39:00.148 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-13T14:39:00.148 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-13T14:39:00.148 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-13T14:39:00.148 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-13T14:39:00.148 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_queue_max_ops": "50", 2024-09-13T14:39:00.148 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-13T14:39:00.148 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_seek_data_hole": "false", 2024-09-13T14:39:00.148 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_sloppy_crc": "false", 2024-09-13T14:39:00.148 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-13T14:39:00.149 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_splice": "false", 2024-09-13T14:39:00.149 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_split_multiple": "2", 2024-09-13T14:39:00.149 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_split_rand_factor": "20", 2024-09-13T14:39:00.149 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_update_to": "1000", 2024-09-13T14:39:00.149 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-13T14:39:00.149 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-13T14:39:00.149 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-13T14:39:00.149 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-13T14:39:00.149 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-13T14:39:00.149 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-13T14:39:00.149 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_enable": "true", 2024-09-13T14:39:00.149 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-13T14:39:00.149 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-13T14:39:00.150 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-13T14:39:00.150 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-13T14:39:00.150 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-13T14:39:00.150 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-13T14:39:00.150 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_xfs_extsize": "false", 2024-09-13T14:39:00.150 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_zfs_snap": "false", 2024-09-13T14:39:00.150 INFO:teuthology.orchestra.run.smithi151.stdout: "fio_dir": "/tmp/fio", 2024-09-13T14:39:00.150 INFO:teuthology.orchestra.run.smithi151.stdout: "fsid": "2401fc50-71da-11ef-bceb-c7b262605968", 2024-09-13T14:39:00.150 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_allow_other": "true", 2024-09-13T14:39:00.150 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_atomic_o_trunc": "true", 2024-09-13T14:39:00.150 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_big_writes": "true", 2024-09-13T14:39:00.150 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_debug": "false", 2024-09-13T14:39:00.150 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_default_permissions": "false", 2024-09-13T14:39:00.151 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_disable_pagecache": "false", 2024-09-13T14:39:00.151 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_max_write": "0", 2024-09-13T14:39:00.151 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_multithreaded": "true", 2024-09-13T14:39:00.151 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_require_active_mds": "true", 2024-09-13T14:39:00.151 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_set_user_groups": "true", 2024-09-13T14:39:00.151 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_splice_move": "true", 2024-09-13T14:39:00.151 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_splice_read": "true", 2024-09-13T14:39:00.151 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_splice_write": "true", 2024-09-13T14:39:00.151 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-13T14:39:00.151 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_use_invalidate_cb": "true", 2024-09-13T14:39:00.151 INFO:teuthology.orchestra.run.smithi151.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-13T14:39:00.151 INFO:teuthology.orchestra.run.smithi151.stdout: "gss_target_name": "ceph", 2024-09-13T14:39:00.151 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_file": "", 2024-09-13T14:39:00.152 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_inject_failure": "0", 2024-09-13T14:39:00.152 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_interval": "5", 2024-09-13T14:39:00.152 INFO:teuthology.orchestra.run.smithi151.stdout: "host": "smithi151", 2024-09-13T14:39:00.152 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-13T14:39:00.152 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-13T14:39:00.152 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-13T14:39:00.152 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-13T14:39:00.152 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-13T14:39:00.152 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-13T14:39:00.152 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-13T14:39:00.152 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-13T14:39:00.152 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-13T14:39:00.152 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-13T14:39:00.152 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-13T14:39:00.153 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-13T14:39:00.153 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-13T14:39:00.153 INFO:teuthology.orchestra.run.smithi151.stdout: "inject_early_sigterm": "false", 2024-09-13T14:39:00.153 INFO:teuthology.orchestra.run.smithi151.stdout: "jaeger_agent_port": "6799", 2024-09-13T14:39:00.153 INFO:teuthology.orchestra.run.smithi151.stdout: "jaeger_tracing_enable": "false", 2024-09-13T14:39:00.153 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_aio": "true", 2024-09-13T14:39:00.153 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_align_min_size": "65536", 2024-09-13T14:39:00.153 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_block_align": "true", 2024-09-13T14:39:00.153 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_block_size": "4096", 2024-09-13T14:39:00.153 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_dio": "true", 2024-09-13T14:39:00.153 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_discard": "false", 2024-09-13T14:39:00.153 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_force_aio": "false", 2024-09-13T14:39:00.153 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_ignore_corruption": "false", 2024-09-13T14:39:00.154 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_max_write_bytes": "10485760", 2024-09-13T14:39:00.154 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_max_write_entries": "100", 2024-09-13T14:39:00.154 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_replay_from": "0", 2024-09-13T14:39:00.154 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-13T14:39:00.154 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-13T14:39:00.154 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-13T14:39:00.154 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-13T14:39:00.154 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_write_header_frequency": "0", 2024-09-13T14:39:00.154 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_zero_on_create": "false", 2024-09-13T14:39:00.154 INFO:teuthology.orchestra.run.smithi151.stdout: "journaler_prefetch_periods": "10", 2024-09-13T14:39:00.154 INFO:teuthology.orchestra.run.smithi151.stdout: "journaler_prezero_periods": "5", 2024-09-13T14:39:00.154 INFO:teuthology.orchestra.run.smithi151.stdout: "journaler_write_head_interval": "15", 2024-09-13T14:39:00.154 INFO:teuthology.orchestra.run.smithi151.stdout: "key": "", 2024-09-13T14:39:00.154 INFO:teuthology.orchestra.run.smithi151.stdout: "keyfile": "", 2024-09-13T14:39:00.155 INFO:teuthology.orchestra.run.smithi151.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-13T14:39:00.155 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_backend": "rocksdb", 2024-09-13T14:39:00.155 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_default_stripe_size": "65536", 2024-09-13T14:39:00.155 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_fsck_on_mount": "false", 2024-09-13T14:39:00.155 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-13T14:39:00.155 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_max_bytes": "67108864", 2024-09-13T14:39:00.155 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_max_ops": "512", 2024-09-13T14:39:00.155 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_nid_prealloc": "1024", 2024-09-13T14:39:00.155 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_onode_map_size": "1024", 2024-09-13T14:39:00.155 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-13T14:39:00.155 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_sync_submit_transaction": "false", 2024-09-13T14:39:00.155 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_sync_transaction": "false", 2024-09-13T14:39:00.155 INFO:teuthology.orchestra.run.smithi151.stdout: "librados_thread_count": "2", 2024-09-13T14:39:00.156 INFO:teuthology.orchestra.run.smithi151.stdout: "lockdep": "false", 2024-09-13T14:39:00.156 INFO:teuthology.orchestra.run.smithi151.stdout: "lockdep_force_backtrace": "false", 2024-09-13T14:39:00.156 INFO:teuthology.orchestra.run.smithi151.stdout: "log_coarse_timestamps": "true", 2024-09-13T14:39:00.156 INFO:teuthology.orchestra.run.smithi151.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-13T14:39:00.156 INFO:teuthology.orchestra.run.smithi151.stdout: "log_flush_on_exit": "false", 2024-09-13T14:39:00.156 INFO:teuthology.orchestra.run.smithi151.stdout: "log_graylog_host": "127.0.0.1", 2024-09-13T14:39:00.156 INFO:teuthology.orchestra.run.smithi151.stdout: "log_graylog_port": "12201", 2024-09-13T14:39:00.156 INFO:teuthology.orchestra.run.smithi151.stdout: "log_max_new": "1000", 2024-09-13T14:39:00.156 INFO:teuthology.orchestra.run.smithi151.stdout: "log_max_recent": "10000", 2024-09-13T14:39:00.156 INFO:teuthology.orchestra.run.smithi151.stdout: "log_stderr_prefix": "", 2024-09-13T14:39:00.156 INFO:teuthology.orchestra.run.smithi151.stdout: "log_stop_at_utilization": "0.970000", 2024-09-13T14:39:00.156 INFO:teuthology.orchestra.run.smithi151.stdout: "log_to_file": "true", 2024-09-13T14:39:00.156 INFO:teuthology.orchestra.run.smithi151.stdout: "log_to_graylog": "false", 2024-09-13T14:39:00.156 INFO:teuthology.orchestra.run.smithi151.stdout: "log_to_journald": "false", 2024-09-13T14:39:00.157 INFO:teuthology.orchestra.run.smithi151.stdout: "log_to_stderr": "false", 2024-09-13T14:39:00.157 INFO:teuthology.orchestra.run.smithi151.stdout: "log_to_syslog": "false", 2024-09-13T14:39:00.157 INFO:teuthology.orchestra.run.smithi151.stdout: "max_rotating_auth_attempts": "10", 2024-09-13T14:39:00.157 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-13T14:39:00.157 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_action_on_write_error": "1", 2024-09-13T14:39:00.157 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_alternate_name_max": "8192", 2024-09-13T14:39:00.157 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_asio_thread_count": "2", 2024-09-13T14:39:00.157 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_export_pin": "true", 2024-09-13T14:39:00.157 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_fragment_dirs": "true", 2024-09-13T14:39:00.157 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-13T14:39:00.157 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_fragment_interval": "5", 2024-09-13T14:39:00.157 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-13T14:39:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-13T14:39:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_interval": "10", 2024-09-13T14:39:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_max": "-1", 2024-09-13T14:39:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_max_until": "-1", 2024-09-13T14:39:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_merge_size": "50", 2024-09-13T14:39:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_midchunk": "0.300000", 2024-09-13T14:39:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-13T14:39:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_min_start": "0.200000", 2024-09-13T14:39:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_minchunk": "0.001000", 2024-09-13T14:39:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_mode": "0", 2024-09-13T14:39:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_need_max": "1.200000", 2024-09-13T14:39:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_need_min": "0.800000", 2024-09-13T14:39:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_overload_epochs": "2", 2024-09-13T14:39:00.159 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-13T14:39:00.159 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-13T14:39:00.159 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_split_bits": "3", 2024-09-13T14:39:00.159 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-13T14:39:00.159 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_split_size": "10000", 2024-09-13T14:39:00.159 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-13T14:39:00.159 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_target_decay": "10.000000", 2024-09-13T14:39:00.159 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-13T14:39:00.159 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_beacon_grace": "15.000000", 2024-09-13T14:39:00.159 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_beacon_interval": "4.000000", 2024-09-13T14:39:00.159 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-13T14:39:00.159 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-13T14:39:00.159 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_mid": "0.700000", 2024-09-13T14:39:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-13T14:39:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_quiesce_delay": "0", 2024-09-13T14:39:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-13T14:39:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-13T14:39:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-13T14:39:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_release_free_interval": "10", 2024-09-13T14:39:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_reservation": "0.050000", 2024-09-13T14:39:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-13T14:39:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_trim_interval": "1", 2024-09-13T14:39:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_trim_threshold": "262144", 2024-09-13T14:39:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-13T14:39:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-13T14:39:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-13T14:39:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_client_prealloc_inos": "1000", 2024-09-13T14:39:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-13T14:39:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_connect_bootstrapping": "false", 2024-09-13T14:39:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_damage_table_max_entries": "10000", 2024-09-13T14:39:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-13T14:39:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_debug_auth_pins": "false", 2024-09-13T14:39:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_debug_frag": "false", 2024-09-13T14:39:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_debug_scatterstat": "false", 2024-09-13T14:39:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_debug_subtrees": "false", 2024-09-13T14:39:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_decay_halflife": "5.000000", 2024-09-13T14:39:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_default_dir_hash": "2", 2024-09-13T14:39:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_defer_session_stale": "true", 2024-09-13T14:39:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_deny_all_reconnect": "false", 2024-09-13T14:39:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_dir_keys_per_op": "16384", 2024-09-13T14:39:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_dir_max_commit_size": "10", 2024-09-13T14:39:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_dir_max_entries": "0", 2024-09-13T14:39:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_dir_prefetch": "true", 2024-09-13T14:39:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-13T14:39:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-13T14:39:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_dump_cache_on_map": "false", 2024-09-13T14:39:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-13T14:39:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-13T14:39:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_early_reply": "true", 2024-09-13T14:39:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_enable_op_tracker": "true", 2024-09-13T14:39:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_enforce_unique_name": "true", 2024-09-13T14:39:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-13T14:39:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-13T14:39:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_export_ephemeral_random": "true", 2024-09-13T14:39:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-13T14:39:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-13T14:39:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-13T14:39:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-13T14:39:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-13T14:39:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-13T14:39:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-13T14:39:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-13T14:39:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_health_summarize_threshold": "10", 2024-09-13T14:39:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-13T14:39:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-13T14:39:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_inject_health_dummy": "false", 2024-09-13T14:39:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-13T14:39:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_inject_migrator_session_race": "false", 2024-09-13T14:39:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-13T14:39:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-13T14:39:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-13T14:39:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_join_fs": "", 2024-09-13T14:39:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_journal_format": "1", 2024-09-13T14:39:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-13T14:39:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_create_at": "0", 2024-09-13T14:39:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_dirfrag_at": "0", 2024-09-13T14:39:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_export_at": "0", 2024-09-13T14:39:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_import_at": "0", 2024-09-13T14:39:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_journal_at": "0", 2024-09-13T14:39:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_journal_expire_at": "0", 2024-09-13T14:39:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_journal_replay_at": "0", 2024-09-13T14:39:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_link_at": "0", 2024-09-13T14:39:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_mdstable_at": "0", 2024-09-13T14:39:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_openc_at": "0", 2024-09-13T14:39:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_rename_at": "0", 2024-09-13T14:39:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_shutdown_at": "0", 2024-09-13T14:39:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_event_large_threshold": "524288", 2024-09-13T14:39:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_events_per_segment": "1024", 2024-09-13T14:39:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-13T14:39:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_max_events": "-1", 2024-09-13T14:39:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_max_segments": "128", 2024-09-13T14:39:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_pause": "false", 2024-09-13T14:39:00.166 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_segment_size": "0", 2024-09-13T14:39:00.166 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-13T14:39:00.166 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-13T14:39:00.166 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-13T14:39:00.166 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_trim_threshold": "128", 2024-09-13T14:39:00.166 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-13T14:39:00.166 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_warn_factor": "2.000000", 2024-09-13T14:39:00.166 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_max_caps_per_client": "1048576", 2024-09-13T14:39:00.166 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_max_completed_flushes": "100000", 2024-09-13T14:39:00.166 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_max_completed_requests": "100000", 2024-09-13T14:39:00.166 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_max_export_size": "20971520", 2024-09-13T14:39:00.166 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_max_file_recover": "32", 2024-09-13T14:39:00.166 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_max_purge_files": "64", 2024-09-13T14:39:00.166 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_max_purge_ops": "8192", 2024-09-13T14:39:00.167 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-13T14:39:00.167 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-13T14:39:00.167 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_max_snaps_per_dir": "100", 2024-09-13T14:39:00.167 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_metrics_update_interval": "2", 2024-09-13T14:39:00.167 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_min_caps_per_client": "100", 2024-09-13T14:39:00.167 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_min_caps_working_set": "10000", 2024-09-13T14:39:00.167 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-13T14:39:00.167 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_numa_node": "-1", 2024-09-13T14:39:00.167 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-13T14:39:00.167 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_op_complaint_time": "30.000000", 2024-09-13T14:39:00.167 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_op_history_duration": "600", 2024-09-13T14:39:00.167 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_op_history_size": "20", 2024-09-13T14:39:00.167 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_op_history_slow_op_size": "20", 2024-09-13T14:39:00.167 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-13T14:39:00.168 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_op_log_threshold": "5", 2024-09-13T14:39:00.168 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_ping_grace": "15", 2024-09-13T14:39:00.168 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_ping_interval": "5", 2024-09-13T14:39:00.168 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-13T14:39:00.168 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-13T14:39:00.168 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_recall_max_caps": "30000", 2024-09-13T14:39:00.168 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-13T14:39:00.168 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-13T14:39:00.168 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-13T14:39:00.168 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_recall_warning_threshold": "262144", 2024-09-13T14:39:00.168 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-13T14:39:00.168 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_replay_interval": "1.000000", 2024-09-13T14:39:00.168 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-13T14:39:00.169 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-13T14:39:00.169 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_root_ino_gid": "0", 2024-09-13T14:39:00.169 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_root_ino_uid": "0", 2024-09-13T14:39:00.169 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-13T14:39:00.169 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-09-13T14:39:00.169 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-09-13T14:39:00.169 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-13T14:39:00.169 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-13T14:39:00.169 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-13T14:39:00.169 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-13T14:39:00.169 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-13T14:39:00.169 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-13T14:39:00.169 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-13T14:39:00.169 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-13T14:39:00.170 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-13T14:39:00.170 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_shutdown_check": "0", 2024-09-13T14:39:00.170 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_skip_ino": "0", 2024-09-13T14:39:00.170 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-13T14:39:00.170 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_snap_max_uid": "4294967294", 2024-09-13T14:39:00.170 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_snap_min_uid": "0", 2024-09-13T14:39:00.170 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_snap_rstat": "false", 2024-09-13T14:39:00.170 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_standby_replay_damaged": "false", 2024-09-13T14:39:00.170 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_symlink_recovery": "true", 2024-09-13T14:39:00.170 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-13T14:39:00.170 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_thrash_exports": "0", 2024-09-13T14:39:00.170 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_thrash_fragments": "0", 2024-09-13T14:39:00.171 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_tick_interval": "5.000000", 2024-09-13T14:39:00.171 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_valgrind_exit": "false", 2024-09-13T14:39:00.171 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_verify_backtrace": "1", 2024-09-13T14:39:00.171 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_verify_scatter": "false", 2024-09-13T14:39:00.171 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-13T14:39:00.171 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_wipe_sessions": "false", 2024-09-13T14:39:00.171 INFO:teuthology.orchestra.run.smithi151.stdout: "mempool_debug": "false", 2024-09-13T14:39:00.171 INFO:teuthology.orchestra.run.smithi151.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-13T14:39:00.171 INFO:teuthology.orchestra.run.smithi151.stdout: "memstore_device_bytes": "1073741824", 2024-09-13T14:39:00.171 INFO:teuthology.orchestra.run.smithi151.stdout: "memstore_page_set": "false", 2024-09-13T14:39:00.171 INFO:teuthology.orchestra.run.smithi151.stdout: "memstore_page_size": "65536", 2024-09-13T14:39:00.171 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_client_bytes": "134217728", 2024-09-13T14:39:00.171 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_client_messages": "512", 2024-09-13T14:39:00.171 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-13T14:39:00.172 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-13T14:39:00.172 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-13T14:39:00.172 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-13T14:39:00.172 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_disabled_modules": "", 2024-09-13T14:39:00.172 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_enable_op_tracker": "true", 2024-09-13T14:39:00.172 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-13T14:39:00.172 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_max_pg_creating": "1024", 2024-09-13T14:39:00.172 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_max_pg_num_change": "128", 2024-09-13T14:39:00.172 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_mds_bytes": "134217728", 2024-09-13T14:39:00.172 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_mds_messages": "128", 2024-09-13T14:39:00.172 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-13T14:39:00.172 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_mon_bytes": "134217728", 2024-09-13T14:39:00.172 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_mon_messages": "128", 2024-09-13T14:39:00.172 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-13T14:39:00.173 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-13T14:39:00.173 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_op_history_duration": "600", 2024-09-13T14:39:00.173 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_op_history_size": "20", 2024-09-13T14:39:00.173 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-13T14:39:00.173 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-13T14:39:00.173 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_op_log_threshold": "5", 2024-09-13T14:39:00.173 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_osd_bytes": "536870912", 2024-09-13T14:39:00.173 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_osd_messages": "8192", 2024-09-13T14:39:00.173 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_pool": "true", 2024-09-13T14:39:00.173 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-13T14:39:00.173 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_standby_modules": "true", 2024-09-13T14:39:00.173 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_stats_period": "5", 2024-09-13T14:39:00.173 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_stats_threshold": "5", 2024-09-13T14:39:00.174 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_tick_period": "2", 2024-09-13T14:39:00.174 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-13T14:39:00.174 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-13T14:39:00.174 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_allow_pool_delete": "true", 2024-09-13T14:39:00.174 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_allow_pool_size_one": "false", 2024-09-13T14:39:00.174 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_auth_validate_all_caps": "true", 2024-09-13T14:39:00.174 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-13T14:39:00.174 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-13T14:39:00.174 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_bytes": "104857600", 2024-09-13T14:39:00.174 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_directed_command_retry": "2", 2024-09-13T14:39:00.174 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-13T14:39:00.174 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-13T14:39:00.174 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-13T14:39:00.175 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-13T14:39:00.175 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_hunt_parallel": "3", 2024-09-13T14:39:00.175 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_log_interval": "1.000000", 2024-09-13T14:39:00.175 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-13T14:39:00.175 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_ping_interval": "10.000000", 2024-09-13T14:39:00.175 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-13T14:39:00.175 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_target_rank": "-1", 2024-09-13T14:39:00.175 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-13T14:39:00.175 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-13T14:39:00.175 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-13T14:39:00.175 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cluster_log_level": "debug", 2024-09-13T14:39:00.175 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cluster_log_to_file": "true", 2024-09-13T14:39:00.175 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-13T14:39:00.175 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-13T14:39:00.176 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-13T14:39:00.176 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cluster_log_to_journald": "true", 2024-09-13T14:39:00.176 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-13T14:39:00.176 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-13T14:39:00.176 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-13T14:39:00.176 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_compact_on_bootstrap": "false", 2024-09-13T14:39:00.176 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_compact_on_start": "false", 2024-09-13T14:39:00.176 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_compact_on_trim": "true", 2024-09-13T14:39:00.176 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-13T14:39:00.176 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-13T14:39:00.176 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-13T14:39:00.176 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cpu_threads": "4", 2024-09-13T14:39:00.176 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_crush_min_required_version": "hammer", 2024-09-13T14:39:00.177 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_daemon_bytes": "419430400", 2024-09-13T14:39:00.177 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-13T14:39:00.177 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_data_avail_crit": "5", 2024-09-13T14:39:00.177 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_data_avail_warn": "5", 2024-09-13T14:39:00.177 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_data_size_warn": "16106127360", 2024-09-13T14:39:00.177 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-13T14:39:00.177 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-13T14:39:00.177 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_dump_json": "false", 2024-09-13T14:39:00.177 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-13T14:39:00.177 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_dump_transactions": "false", 2024-09-13T14:39:00.177 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_extra_checks": "false", 2024-09-13T14:39:00.177 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-13T14:39:00.177 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-13T14:39:00.177 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_no_require_reef": "false", 2024-09-13T14:39:00.178 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_no_require_squid": "false", 2024-09-13T14:39:00.178 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-13T14:39:00.178 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-13T14:39:00.178 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-13T14:39:00.178 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_down_mkfs_grace": "60", 2024-09-13T14:39:00.178 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_down_uptime_grace": "60", 2024-09-13T14:39:00.178 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_election_default_strategy": "3", 2024-09-13T14:39:00.178 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_election_timeout": "5.000000", 2024-09-13T14:39:00.178 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-13T14:39:00.178 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_elector_ping_divisor": "2", 2024-09-13T14:39:00.178 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-13T14:39:00.178 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_enable_op_tracker": "true", 2024-09-13T14:39:00.178 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_fake_pool_delete": "false", 2024-09-13T14:39:00.179 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_force_quorum_join": "false", 2024-09-13T14:39:00.179 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-13T14:39:00.179 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_globalid_prealloc": "10000", 2024-09-13T14:39:00.179 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_health_detail_to_clog": "true", 2024-09-13T14:39:00.179 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_health_log_update_period": "5", 2024-09-13T14:39:00.179 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_health_max_detail": "50", 2024-09-13T14:39:00.179 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_health_to_clog": "true", 2024-09-13T14:39:00.179 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_health_to_clog_interval": "600", 2024-09-13T14:39:00.179 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-13T14:39:00.179 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_host": "[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0]", 2024-09-13T14:39:00.179 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_host_override": "", 2024-09-13T14:39:00.179 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_initial_members": "", 2024-09-13T14:39:00.179 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-13T14:39:00.180 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-13T14:39:00.180 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-13T14:39:00.180 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-13T14:39:00.180 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-13T14:39:00.180 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_lease": "5.000000", 2024-09-13T14:39:00.180 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-13T14:39:00.180 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-13T14:39:00.180 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_log_full_interval": "50", 2024-09-13T14:39:00.180 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_log_max": "10000", 2024-09-13T14:39:00.180 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_log_max_summary": "50", 2024-09-13T14:39:00.180 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-13T14:39:00.180 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_max_log_epochs": "500", 2024-09-13T14:39:00.181 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-13T14:39:00.181 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-13T14:39:00.181 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_max_nvmeof_epochs": "500", 2024-09-13T14:39:00.181 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_max_osd": "10000", 2024-09-13T14:39:00.181 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_max_pg_per_osd": "10000", 2024-09-13T14:39:00.181 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_max_pool_pg_num": "65536", 2024-09-13T14:39:00.181 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-13T14:39:00.181 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-13T14:39:00.181 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_mds_force_trim_to": "0", 2024-09-13T14:39:00.181 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_mds_skip_sanity": "true", 2024-09-13T14:39:00.181 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_memory_autotune": "true", 2024-09-13T14:39:00.181 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_memory_target": "2147483648", 2024-09-13T14:39:00.182 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_mgr_beacon_grace": "30", 2024-09-13T14:39:00.182 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-13T14:39:00.182 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_mgr_digest_period": "5", 2024-09-13T14:39:00.182 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_mgr_inactive_grace": "60", 2024-09-13T14:39:00.182 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-13T14:39:00.182 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-13T14:39:00.182 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_min_osdmap_epochs": "500", 2024-09-13T14:39:00.182 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-09-13T14:39:00.182 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-09-13T14:39:00.182 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_op_complaint_time": "30", 2024-09-13T14:39:00.182 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_op_history_duration": "600", 2024-09-13T14:39:00.182 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_op_history_size": "20", 2024-09-13T14:39:00.182 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_op_history_slow_op_size": "20", 2024-09-13T14:39:00.182 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-13T14:39:00.183 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_op_log_threshold": "5", 2024-09-13T14:39:00.183 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-13T14:39:00.183 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-13T14:39:00.183 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-13T14:39:00.183 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_auto_mark_in": "false", 2024-09-13T14:39:00.183 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-13T14:39:00.183 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-13T14:39:00.183 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-13T14:39:00.183 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_cache_size": "500", 2024-09-13T14:39:00.183 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-13T14:39:00.183 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-13T14:39:00.183 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-13T14:39:00.183 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_down_out_interval": "600", 2024-09-13T14:39:00.183 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-13T14:39:00.184 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-13T14:39:00.184 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_force_trim_to": "0", 2024-09-13T14:39:00.184 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-13T14:39:00.184 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-13T14:39:00.184 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-13T14:39:00.184 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-13T14:39:00.184 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-13T14:39:00.184 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-13T14:39:00.184 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-13T14:39:00.184 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_min_down_reporters": "2", 2024-09-13T14:39:00.184 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-13T14:39:00.184 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-13T14:39:00.184 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-13T14:39:00.185 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-13T14:39:00.185 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-13T14:39:00.185 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-13T14:39:00.185 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_report_timeout": "900", 2024-09-13T14:39:00.185 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-13T14:39:00.185 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-13T14:39:00.185 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-13T14:39:00.185 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-13T14:39:00.185 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-13T14:39:00.185 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-13T14:39:00.185 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-13T14:39:00.185 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-13T14:39:00.185 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-13T14:39:00.186 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_pg_stuck_threshold": "60", 2024-09-13T14:39:00.186 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-13T14:39:00.186 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-13T14:39:00.186 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-13T14:39:00.186 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-13T14:39:00.186 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-13T14:39:00.186 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-13T14:39:00.186 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_probe_timeout": "2.000000", 2024-09-13T14:39:00.186 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_reweight_max_change": "0.050000", 2024-09-13T14:39:00.186 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_reweight_max_osds": "4", 2024-09-13T14:39:00.186 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-13T14:39:00.186 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-13T14:39:00.186 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-13T14:39:00.186 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-13T14:39:00.187 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-13T14:39:00.187 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_scrub_interval": "86400", 2024-09-13T14:39:00.187 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_scrub_max_keys": "100", 2024-09-13T14:39:00.187 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_scrub_timeout": "300", 2024-09-13T14:39:00.187 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_session_timeout": "300", 2024-09-13T14:39:00.187 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_smart_report_timeout": "5", 2024-09-13T14:39:00.187 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_stat_smooth_intervals": "6", 2024-09-13T14:39:00.187 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-13T14:39:00.187 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_stretch_pool_min_size": "2", 2024-09-13T14:39:00.187 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_stretch_pool_size": "4", 2024-09-13T14:39:00.187 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-13T14:39:00.187 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-13T14:39:00.187 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_sync_debug": "false", 2024-09-13T14:39:00.188 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-13T14:39:00.188 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-13T14:39:00.188 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_sync_provider_kill_at": "0", 2024-09-13T14:39:00.188 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_sync_requester_kill_at": "0", 2024-09-13T14:39:00.188 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_sync_timeout": "60.000000", 2024-09-13T14:39:00.188 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_target_pg_per_osd": "100", 2024-09-13T14:39:00.188 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_tick_interval": "5", 2024-09-13T14:39:00.188 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_timecheck_interval": "300.000000", 2024-09-13T14:39:00.188 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-13T14:39:00.188 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_use_min_delay_socket": "false", 2024-09-13T14:39:00.188 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_older_version_delay": "604800", 2024-09-13T14:39:00.188 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-13T14:39:00.188 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-13T14:39:00.189 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-13T14:39:00.189 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_filestore_osds": "true", 2024-09-13T14:39:00.189 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-13T14:39:00.189 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-13T14:39:00.189 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-13T14:39:00.189 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_misplaced": "false", 2024-09-13T14:39:00.189 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-13T14:39:00.189 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_older_version": "true", 2024-09-13T14:39:00.189 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-13T14:39:00.189 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-13T14:39:00.189 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-09-13T14:39:00.189 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-13T14:39:00.189 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-13T14:39:00.189 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-13T14:39:00.190 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-13T14:39:00.190 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-13T14:39:00.190 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-13T14:39:00.190 INFO:teuthology.orchestra.run.smithi151.stdout: "monmap": "", 2024-09-13T14:39:00.190 INFO:teuthology.orchestra.run.smithi151.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-13T14:39:00.190 INFO:teuthology.orchestra.run.smithi151.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-13T14:39:00.190 INFO:teuthology.orchestra.run.smithi151.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-13T14:39:00.190 INFO:teuthology.orchestra.run.smithi151.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-13T14:39:00.190 INFO:teuthology.orchestra.run.smithi151.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-13T14:39:00.190 INFO:teuthology.orchestra.run.smithi151.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-13T14:39:00.190 INFO:teuthology.orchestra.run.smithi151.stdout: "motr_tracing_enabled": "false", 2024-09-13T14:39:00.190 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_op_threads": "3", 2024-09-13T14:39:00.190 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-13T14:39:00.191 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_cm": "false", 2024-09-13T14:39:00.191 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_device_name": "", 2024-09-13T14:39:00.191 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_dscp": "96", 2024-09-13T14:39:00.191 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-13T14:39:00.191 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-13T14:39:00.191 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_local_gid": "", 2024-09-13T14:39:00.191 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-13T14:39:00.191 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_port_num": "1", 2024-09-13T14:39:00.191 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-13T14:39:00.191 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-13T14:39:00.191 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-13T14:39:00.191 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-13T14:39:00.191 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_sl": "3", 2024-09-13T14:39:00.191 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_support_srq": "true", 2024-09-13T14:39:00.192 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_type": "ib", 2024-09-13T14:39:00.192 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_reap_threshold": "5", 2024-09-13T14:39:00.192 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_bind_before_connect": "false", 2024-09-13T14:39:00.192 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_bind_ipv4": "true", 2024-09-13T14:39:00.192 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_bind_ipv6": "false", 2024-09-13T14:39:00.192 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_bind_msgr1": "true", 2024-09-13T14:39:00.192 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_bind_msgr2": "true", 2024-09-13T14:39:00.192 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_bind_port_max": "7568", 2024-09-13T14:39:00.192 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_bind_port_min": "6800", 2024-09-13T14:39:00.192 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-13T14:39:00.192 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_bind_retry_count": "3", 2024-09-13T14:39:00.192 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_bind_retry_delay": "5", 2024-09-13T14:39:00.193 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_blackhole_client": "false", 2024-09-13T14:39:00.193 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_blackhole_mds": "false", 2024-09-13T14:39:00.193 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_blackhole_mgr": "false", 2024-09-13T14:39:00.193 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_blackhole_mon": "false", 2024-09-13T14:39:00.193 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_blackhole_osd": "false", 2024-09-13T14:39:00.193 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_client_mode": "crc secure", 2024-09-13T14:39:00.193 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-09-13T14:39:00.193 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_cluster_mode": "crc secure", 2024-09-13T14:39:00.193 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_cluster_type": "", 2024-09-13T14:39:00.193 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_compress_secure": "false", 2024-09-13T14:39:00.193 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_connection_idle_timeout": "900", 2024-09-13T14:39:00.193 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_connection_ready_timeout": "10", 2024-09-13T14:39:00.193 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_crc_data": "true", 2024-09-13T14:39:00.193 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_crc_header": "true", 2024-09-13T14:39:00.194 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_die_on_bad_msg": "false", 2024-09-13T14:39:00.194 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_die_on_bug": "true", 2024-09-13T14:39:00.194 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_die_on_old_message": "true", 2024-09-13T14:39:00.194 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_die_on_skipped_message": "false", 2024-09-13T14:39:00.194 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-13T14:39:00.194 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-13T14:39:00.194 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_coremask": "0xF", 2024-09-13T14:39:00.194 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-13T14:39:00.194 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-13T14:39:00.194 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_enable_tso": "true", 2024-09-13T14:39:00.194 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-13T14:39:00.194 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-13T14:39:00.194 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_hugepages": "", 2024-09-13T14:39:00.194 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-13T14:39:00.195 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-13T14:39:00.195 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_lro": "true", 2024-09-13T14:39:00.195 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_memory_channel": "4", 2024-09-13T14:39:00.195 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-13T14:39:00.195 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_pmd": "", 2024-09-13T14:39:00.195 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_port_id": "0", 2024-09-13T14:39:00.195 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-13T14:39:00.195 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-13T14:39:00.195 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dump_on_send": "false", 2024-09-13T14:39:00.195 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_initial_backoff": "0.200000", 2024-09-13T14:39:00.195 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_inject_delay_max": "1.000000", 2024-09-13T14:39:00.195 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-13T14:39:00.195 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_inject_delay_type": "", 2024-09-13T14:39:00.196 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-13T14:39:00.196 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_inject_network_congestion": "0", 2024-09-13T14:39:00.196 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_inject_socket_failures": "0", 2024-09-13T14:39:00.196 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_learn_addr_from_peer": "true", 2024-09-13T14:39:00.196 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_max_accept_failures": "4", 2024-09-13T14:39:00.196 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_max_backoff": "15.000000", 2024-09-13T14:39:00.196 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_mon_client_mode": "secure crc", 2024-09-13T14:39:00.196 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-13T14:39:00.196 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_mon_service_mode": "secure crc", 2024-09-13T14:39:00.196 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_osd_compress_min_size": "1024", 2024-09-13T14:39:00.196 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_osd_compress_mode": "none", 2024-09-13T14:39:00.196 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-13T14:39:00.197 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-13T14:39:00.197 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_pq_min_cost": "65536", 2024-09-13T14:39:00.197 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_public_type": "", 2024-09-13T14:39:00.197 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_service_mode": "crc secure", 2024-09-13T14:39:00.197 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_tcp_listen_backlog": "512", 2024-09-13T14:39:00.197 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_tcp_nodelay": "true", 2024-09-13T14:39:00.197 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-13T14:39:00.197 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_tcp_rcvbuf": "0", 2024-09-13T14:39:00.197 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_time_events_min_wait_interval": "1000", 2024-09-13T14:39:00.197 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_type": "async+posix", 2024-09-13T14:39:00.197 INFO:teuthology.orchestra.run.smithi151.stdout: "no_config_file": "false", 2024-09-13T14:39:00.197 INFO:teuthology.orchestra.run.smithi151.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-09-13T14:39:00.197 INFO:teuthology.orchestra.run.smithi151.stdout: "nvmeof_mon_client_tick_period": "2", 2024-09-13T14:39:00.197 INFO:teuthology.orchestra.run.smithi151.stdout: "objecter_completion_locks_per_session": "32", 2024-09-13T14:39:00.198 INFO:teuthology.orchestra.run.smithi151.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-13T14:39:00.198 INFO:teuthology.orchestra.run.smithi151.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-13T14:39:00.198 INFO:teuthology.orchestra.run.smithi151.stdout: "objecter_inflight_ops": "1024", 2024-09-13T14:39:00.198 INFO:teuthology.orchestra.run.smithi151.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-13T14:39:00.198 INFO:teuthology.orchestra.run.smithi151.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-13T14:39:00.198 INFO:teuthology.orchestra.run.smithi151.stdout: "objecter_tick_interval": "5.000000", 2024-09-13T14:39:00.198 INFO:teuthology.orchestra.run.smithi151.stdout: "objecter_timeout": "10.000000", 2024-09-13T14:39:00.198 INFO:teuthology.orchestra.run.smithi151.stdout: "objectstore_blackhole": "false", 2024-09-13T14:39:00.198 INFO:teuthology.orchestra.run.smithi151.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-13T14:39:00.198 INFO:teuthology.orchestra.run.smithi151.stdout: "openssl_engine_opts": "", 2024-09-13T14:39:00.198 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_agent_delay_time": "5.000000", 2024-09-13T14:39:00.198 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_agent_hist_halflife": "1000", 2024-09-13T14:39:00.199 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_agent_max_low_ops": "2", 2024-09-13T14:39:00.199 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_agent_max_ops": "4", 2024-09-13T14:39:00.199 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-13T14:39:00.199 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-13T14:39:00.199 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_agent_slop": "0.020000", 2024-09-13T14:39:00.199 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-13T14:39:00.199 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-13T14:39:00.199 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_asio_thread_count": "2", 2024-09-13T14:39:00.199 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_async_recovery_min_cost": "100", 2024-09-13T14:39:00.199 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-13T14:39:00.199 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-13T14:39:00.199 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_backfill_scan_max": "512", 2024-09-13T14:39:00.199 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_backfill_scan_min": "64", 2024-09-13T14:39:00.200 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_backoff_on_degraded": "false", 2024-09-13T14:39:00.200 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_backoff_on_peering": "false", 2024-09-13T14:39:00.200 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_backoff_on_unfound": "true", 2024-09-13T14:39:00.200 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_beacon_report_interval": "300", 2024-09-13T14:39:00.200 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_bench_duration": "30", 2024-09-13T14:39:00.200 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-13T14:39:00.200 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_bench_max_block_size": "67108864", 2024-09-13T14:39:00.200 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-13T14:39:00.200 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_blkin_trace_all": "false", 2024-09-13T14:39:00.200 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-13T14:39:00.201 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-13T14:39:00.202 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-13T14:39:00.202 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-13T14:39:00.202 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_check_for_log_corruption": "false", 2024-09-13T14:39:00.202 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-13T14:39:00.202 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-13T14:39:00.202 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-13T14:39:00.202 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-13T14:39:00.202 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_class_update_on_start": "true", 2024-09-13T14:39:00.202 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_client_message_cap": "256", 2024-09-13T14:39:00.202 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_client_message_size_cap": "524288000", 2024-09-13T14:39:00.202 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_client_op_priority": "63", 2024-09-13T14:39:00.202 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_client_watch_timeout": "30", 2024-09-13T14:39:00.203 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_command_max_records": "256", 2024-09-13T14:39:00.203 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_compact_on_start": "false", 2024-09-13T14:39:00.203 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-13T14:39:00.203 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_crush_chooseleaf_type": "1", 2024-09-13T14:39:00.203 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-13T14:39:00.203 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_crush_update_on_start": "true", 2024-09-13T14:39:00.203 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_crush_update_weight_set": "true", 2024-09-13T14:39:00.203 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-13T14:39:00.203 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-13T14:39:00.203 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-13T14:39:00.203 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-13T14:39:00.203 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-13T14:39:00.203 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_feed_pullee": "-1", 2024-09-13T14:39:00.203 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-13T14:39:00.204 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-13T14:39:00.204 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-13T14:39:00.204 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_misdirected_ops": "false", 2024-09-13T14:39:00.204 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_no_acting_change": "false", 2024-09-13T14:39:00.204 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_no_purge_strays": "false", 2024-09-13T14:39:00.204 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_op_order": "false", 2024-09-13T14:39:00.204 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-13T14:39:00.204 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-13T14:39:00.204 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-13T14:39:00.204 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-13T14:39:00.204 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_shutdown": "false", 2024-09-13T14:39:00.204 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-13T14:39:00.205 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-13T14:39:00.205 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_trim_objects": "false", 2024-09-13T14:39:00.205 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-13T14:39:00.205 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-13T14:39:00.205 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_verify_snaps": "false", 2024-09-13T14:39:00.205 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-13T14:39:00.205 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-13T14:39:00.205 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_deep_scrub_keys": "1024", 2024-09-13T14:39:00.205 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-13T14:39:00.205 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-13T14:39:00.205 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-13T14:39:00.205 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_deep_scrub_stride": "524288", 2024-09-13T14:39:00.205 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-13T14:39:00.206 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_default_data_pool_replay_window": "45", 2024-09-13T14:39:00.206 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_default_notify_timeout": "30", 2024-09-13T14:39:00.206 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_delete_sleep": "0.000000", 2024-09-13T14:39:00.206 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-13T14:39:00.206 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-13T14:39:00.206 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-13T14:39:00.206 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_discard_disconnected_ops": "true", 2024-09-13T14:39:00.206 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_ec_partial_reads": "true", 2024-09-13T14:39:00.206 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_enable_op_tracker": "true", 2024-09-13T14:39:00.206 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-13T14:39:00.206 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-13T14:39:00.206 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-13T14:39:00.206 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-13T14:39:00.207 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_fast_info": "true", 2024-09-13T14:39:00.207 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_fast_shutdown": "true", 2024-09-13T14:39:00.207 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-13T14:39:00.207 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-13T14:39:00.207 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-13T14:39:00.207 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-13T14:39:00.207 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-13T14:39:00.207 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_function_tracing": "false", 2024-09-13T14:39:00.207 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_heartbeat_grace": "20", 2024-09-13T14:39:00.207 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_heartbeat_interval": "6", 2024-09-13T14:39:00.208 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-13T14:39:00.208 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_heartbeat_min_peers": "10", 2024-09-13T14:39:00.208 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_heartbeat_min_size": "2000", 2024-09-13T14:39:00.208 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_heartbeat_stale": "600", 2024-09-13T14:39:00.208 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-13T14:39:00.208 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_hit_set_max_size": "100000", 2024-09-13T14:39:00.208 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_hit_set_min_size": "1000", 2024-09-13T14:39:00.208 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-13T14:39:00.208 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-13T14:39:00.208 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-13T14:39:00.208 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-13T14:39:00.208 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-13T14:39:00.208 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-13T14:39:00.209 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_journal_size": "5120", 2024-09-13T14:39:00.209 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_kill_backfill_at": "0", 2024-09-13T14:39:00.209 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-13T14:39:00.209 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_map_cache_size": "50", 2024-09-13T14:39:00.209 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_map_dedup": "true", 2024-09-13T14:39:00.209 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_map_message_max": "40", 2024-09-13T14:39:00.209 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-13T14:39:00.209 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_map_share_max_epochs": "40", 2024-09-13T14:39:00.209 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_attr_name_len": "100", 2024-09-13T14:39:00.209 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_attr_size": "0", 2024-09-13T14:39:00.209 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_backfills": "1", 2024-09-13T14:39:00.209 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_markdown_count": "5", 2024-09-13T14:39:00.209 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_markdown_period": "600", 2024-09-13T14:39:00.210 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_object_name_len": "2048", 2024-09-13T14:39:00.210 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_object_namespace_len": "256", 2024-09-13T14:39:00.210 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_object_size": "134217728", 2024-09-13T14:39:00.210 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-13T14:39:00.210 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-13T14:39:00.210 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_pg_blocked_by": "16", 2024-09-13T14:39:00.210 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_pg_log_entries": "10000", 2024-09-13T14:39:00.210 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-13T14:39:00.210 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_pgls": "1024", 2024-09-13T14:39:00.210 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_push_cost": "8388608", 2024-09-13T14:39:00.210 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_push_objects": "10", 2024-09-13T14:39:00.210 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_scrubs": "3", 2024-09-13T14:39:00.210 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-13T14:39:00.210 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_trimming_pgs": "2", 2024-09-13T14:39:00.211 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_write_op_reply_len": "64", 2024-09-13T14:39:00.211 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_write_size": "90", 2024-09-13T14:39:00.211 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-13T14:39:00.211 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-13T14:39:00.211 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-13T14:39:00.211 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-13T14:39:00.211 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-13T14:39:00.211 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-13T14:39:00.211 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-13T14:39:00.211 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-13T14:39:00.211 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_profile": "balanced", 2024-09-13T14:39:00.211 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-13T14:39:00.211 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-13T14:39:00.211 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-13T14:39:00.212 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-13T14:39:00.212 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-13T14:39:00.212 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-13T14:39:00.212 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-13T14:39:00.212 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-13T14:39:00.212 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-13T14:39:00.212 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-13T14:39:00.212 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-13T14:39:00.212 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_memory_base": "805306368", 2024-09-13T14:39:00.212 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_memory_cache_min": "134217728", 2024-09-13T14:39:00.212 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-13T14:39:00.212 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-13T14:39:00.212 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_memory_target": "4294967296", 2024-09-13T14:39:00.213 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_memory_target_autotune": "false", 2024-09-13T14:39:00.213 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-13T14:39:00.213 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_min_pg_log_entries": "250", 2024-09-13T14:39:00.213 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_min_recovery_priority": "0", 2024-09-13T14:39:00.213 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-13T14:39:00.213 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-13T14:39:00.213 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mon_report_interval": "5", 2024-09-13T14:39:00.213 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-13T14:39:00.213 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-13T14:39:00.213 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_num_cache_shards": "32", 2024-09-13T14:39:00.213 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_num_op_tracker_shard": "32", 2024-09-13T14:39:00.213 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_numa_auto_affinity": "true", 2024-09-13T14:39:00.213 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_numa_node": "-1", 2024-09-13T14:39:00.213 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_numa_prefer_iface": "true", 2024-09-13T14:39:00.214 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-13T14:39:00.214 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_objecter_finishers": "1", 2024-09-13T14:39:00.214 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_objectstore": "bluestore", 2024-09-13T14:39:00.214 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_objectstore_fuse": "false", 2024-09-13T14:39:00.214 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_objectstore_ideal_list_max": "64", 2024-09-13T14:39:00.214 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_objectstore_tracing": "false", 2024-09-13T14:39:00.214 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_complaint_time": "30.000000", 2024-09-13T14:39:00.214 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_history_duration": "600", 2024-09-13T14:39:00.214 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_history_size": "20", 2024-09-13T14:39:00.214 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_history_slow_op_size": "20", 2024-09-13T14:39:00.214 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-13T14:39:00.214 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_log_threshold": "5", 2024-09-13T14:39:00.214 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_num_shards": "0", 2024-09-13T14:39:00.214 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_num_shards_hdd": "5", 2024-09-13T14:39:00.215 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_num_shards_ssd": "8", 2024-09-13T14:39:00.215 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-13T14:39:00.215 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-13T14:39:00.215 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-13T14:39:00.215 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-13T14:39:00.215 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_pq_min_cost": "65536", 2024-09-13T14:39:00.215 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-13T14:39:00.215 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_queue_cut_off": "high", 2024-09-13T14:39:00.215 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-13T14:39:00.215 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_thread_timeout": "15", 2024-09-13T14:39:00.215 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_open_classes_on_start": "true", 2024-09-13T14:39:00.215 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_os_flags": "0", 2024-09-13T14:39:00.215 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_peering_op_priority": "255", 2024-09-13T14:39:00.216 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_delete_cost": "1048576", 2024-09-13T14:39:00.216 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_delete_priority": "5", 2024-09-13T14:39:00.216 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-13T14:39:00.216 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-13T14:39:00.216 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-13T14:39:00.216 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_log_trim_max": "10000", 2024-09-13T14:39:00.216 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_log_trim_min": "100", 2024-09-13T14:39:00.216 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-13T14:39:00.216 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-13T14:39:00.216 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-13T14:39:00.216 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-13T14:39:00.216 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-13T14:39:00.216 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-13T14:39:00.217 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-13T14:39:00.217 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-13T14:39:00.217 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-13T14:39:00.217 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-13T14:39:00.217 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_crimson": "false", 2024-09-13T14:39:00.217 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-13T14:39:00.217 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-13T14:39:00.217 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-13T14:39:00.217 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-13T14:39:00.217 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-13T14:39:00.217 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-13T14:39:00.217 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-13T14:39:00.217 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-13T14:39:00.217 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_flags": "0", 2024-09-13T14:39:00.218 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-13T14:39:00.218 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_min_size": "0", 2024-09-13T14:39:00.218 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-13T14:39:00.218 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_pg_num": "32", 2024-09-13T14:39:00.218 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_pgp_num": "0", 2024-09-13T14:39:00.218 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-13T14:39:00.218 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_read_ratio": "70", 2024-09-13T14:39:00.218 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_size": "3", 2024-09-13T14:39:00.218 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_type": "replicated", 2024-09-13T14:39:00.218 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-13T14:39:00.218 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-13T14:39:00.218 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_push_per_object_cost": "1000", 2024-09-13T14:39:00.218 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-13T14:39:00.219 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recover_clone_overlap": "true", 2024-09-13T14:39:00.219 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-13T14:39:00.219 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_cost": "20971520", 2024-09-13T14:39:00.219 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-13T14:39:00.219 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_max_active": "0", 2024-09-13T14:39:00.219 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-13T14:39:00.219 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-13T14:39:00.219 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-13T14:39:00.219 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-13T14:39:00.219 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_max_single_start": "1", 2024-09-13T14:39:00.219 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_op_priority": "3", 2024-09-13T14:39:00.219 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-13T14:39:00.219 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_priority": "5", 2024-09-13T14:39:00.219 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-13T14:39:00.220 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_sleep": "0.000000", 2024-09-13T14:39:00.220 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-13T14:39:00.220 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-13T14:39:00.220 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-13T14:39:00.220 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_repair_during_recovery": "false", 2024-09-13T14:39:00.220 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_requested_scrub_priority": "120", 2024-09-13T14:39:00.220 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-13T14:39:00.220 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-13T14:39:00.220 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_auto_repair": "false", 2024-09-13T14:39:00.220 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-13T14:39:00.220 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-13T14:39:00.220 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_begin_hour": "0", 2024-09-13T14:39:00.220 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_begin_week_day": "0", 2024-09-13T14:39:00.221 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_chunk_max": "25", 2024-09-13T14:39:00.221 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_chunk_min": "5", 2024-09-13T14:39:00.221 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_cost": "52428800", 2024-09-13T14:39:00.221 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-13T14:39:00.221 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_during_recovery": "false", 2024-09-13T14:39:00.221 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_end_hour": "0", 2024-09-13T14:39:00.221 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_end_week_day": "0", 2024-09-13T14:39:00.221 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_event_cost": "4096", 2024-09-13T14:39:00.221 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-13T14:39:00.221 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-13T14:39:00.221 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_invalid_stats": "true", 2024-09-13T14:39:00.221 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-13T14:39:00.221 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-13T14:39:00.222 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_max_preemptions": "5", 2024-09-13T14:39:00.222 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-13T14:39:00.222 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_priority": "5", 2024-09-13T14:39:00.222 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_sleep": "0.000000", 2024-09-13T14:39:00.222 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-13T14:39:00.222 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-13T14:39:00.222 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-13T14:39:00.222 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-09-13T14:39:00.222 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_skip_data_digest": "false", 2024-09-13T14:39:00.222 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_smart_report_timeout": "5", 2024-09-13T14:39:00.222 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_snap_trim_cost": "1048576", 2024-09-13T14:39:00.222 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_snap_trim_priority": "5", 2024-09-13T14:39:00.222 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-13T14:39:00.223 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-13T14:39:00.223 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-13T14:39:00.223 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-13T14:39:00.223 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-13T14:39:00.223 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-13T14:39:00.223 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-13T14:39:00.223 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_target_transaction_size": "30", 2024-09-13T14:39:00.223 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-13T14:39:00.223 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-13T14:39:00.223 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-13T14:39:00.223 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-13T14:39:00.223 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-13T14:39:00.223 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-13T14:39:00.224 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-13T14:39:00.224 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-13T14:39:00.224 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-13T14:39:00.224 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-13T14:39:00.224 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tracing": "false", 2024-09-13T14:39:00.224 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_use_stale_snap": "false", 2024-09-13T14:39:00.224 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-13T14:39:00.224 INFO:teuthology.orchestra.run.smithi151.stdout: "osdc_blkin_trace_all": "false", 2024-09-13T14:39:00.224 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_kill_at": "0", 2024-09-13T14:39:00.224 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_max_join_drift": "10", 2024-09-13T14:39:00.224 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_min": "500", 2024-09-13T14:39:00.224 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_min_wait": "0.050000", 2024-09-13T14:39:00.224 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_propose_interval": "1.000000", 2024-09-13T14:39:00.224 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_service_trim_max": "500", 2024-09-13T14:39:00.225 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-13T14:39:00.225 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_service_trim_min": "250", 2024-09-13T14:39:00.225 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_stash_full_interval": "25", 2024-09-13T14:39:00.225 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_trim_max": "500", 2024-09-13T14:39:00.225 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_trim_min": "250", 2024-09-13T14:39:00.225 INFO:teuthology.orchestra.run.smithi151.stdout: "perf": "true", 2024-09-13T14:39:00.225 INFO:teuthology.orchestra.run.smithi151.stdout: "pid_file": "", 2024-09-13T14:39:00.225 INFO:teuthology.orchestra.run.smithi151.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-13T14:39:00.225 INFO:teuthology.orchestra.run.smithi151.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-13T14:39:00.225 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "-", 2024-09-13T14:39:00.225 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrv": "", 2024-09-13T14:39:00.225 INFO:teuthology.orchestra.run.smithi151.stdout: "public_bind_addr": "-", 2024-09-13T14:39:00.225 INFO:teuthology.orchestra.run.smithi151.stdout: "public_network": "172.21.0.0/20", 2024-09-13T14:39:00.226 INFO:teuthology.orchestra.run.smithi151.stdout: "public_network_interface": "", 2024-09-13T14:39:00.226 INFO:teuthology.orchestra.run.smithi151.stdout: "qat_compressor_busy_polling": "false", 2024-09-13T14:39:00.226 INFO:teuthology.orchestra.run.smithi151.stdout: "qat_compressor_enabled": "false", 2024-09-13T14:39:00.226 INFO:teuthology.orchestra.run.smithi151.stdout: "qat_compressor_session_max_number": "256", 2024-09-13T14:39:00.226 INFO:teuthology.orchestra.run.smithi151.stdout: "rados_mon_op_timeout": "0", 2024-09-13T14:39:00.226 INFO:teuthology.orchestra.run.smithi151.stdout: "rados_osd_op_timeout": "0", 2024-09-13T14:39:00.226 INFO:teuthology.orchestra.run.smithi151.stdout: "rados_tracing": "false", 2024-09-13T14:39:00.226 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_atime_update_interval": "60", 2024-09-13T14:39:00.226 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-13T14:39:00.226 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_balance_parent_reads": "false", 2024-09-13T14:39:00.226 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_balance_snap_reads": "false", 2024-09-13T14:39:00.226 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_blkin_trace_all": "false", 2024-09-13T14:39:00.226 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-13T14:39:00.226 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-13T14:39:00.227 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_cache": "true", 2024-09-13T14:39:00.227 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-13T14:39:00.227 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-13T14:39:00.227 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-13T14:39:00.227 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-13T14:39:00.227 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_cache_policy": "writearound", 2024-09-13T14:39:00.227 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_cache_size": "33554432", 2024-09-13T14:39:00.227 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-13T14:39:00.227 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-13T14:39:00.227 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_clone_copy_on_read": "false", 2024-09-13T14:39:00.227 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_compression_hint": "none", 2024-09-13T14:39:00.227 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_concurrent_management_ops": "10", 2024-09-13T14:39:00.228 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-13T14:39:00.228 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_default_clone_format": "auto", 2024-09-13T14:39:00.228 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_default_data_pool": "", 2024-09-13T14:39:00.228 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_default_features": "61", 2024-09-13T14:39:00.228 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_default_format": "2", 2024-09-13T14:39:00.228 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_default_map_options": "", 2024-09-13T14:39:00.228 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_default_order": "22", 2024-09-13T14:39:00.228 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_default_pool": "rbd", 2024-09-13T14:39:00.228 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-13T14:39:00.228 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_default_stripe_count": "0", 2024-09-13T14:39:00.228 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_default_stripe_unit": "0", 2024-09-13T14:39:00.228 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-13T14:39:00.228 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-13T14:39:00.228 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-13T14:39:00.229 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_enable_alloc_hint": "true", 2024-09-13T14:39:00.229 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-13T14:39:00.229 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_io_scheduler": "simple", 2024-09-13T14:39:00.229 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-13T14:39:00.229 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-13T14:39:00.229 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-13T14:39:00.229 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-13T14:39:00.229 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-13T14:39:00.229 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-13T14:39:00.229 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-13T14:39:00.229 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-13T14:39:00.229 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-13T14:39:00.229 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_order": "24", 2024-09-13T14:39:00.230 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_pool": "", 2024-09-13T14:39:00.230 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_splay_width": "4", 2024-09-13T14:39:00.230 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_localize_parent_reads": "false", 2024-09-13T14:39:00.230 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_localize_snap_reads": "false", 2024-09-13T14:39:00.230 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-13T14:39:00.230 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-13T14:39:00.230 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-13T14:39:00.230 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-13T14:39:00.230 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-13T14:39:00.230 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-13T14:39:00.230 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-13T14:39:00.230 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-13T14:39:00.230 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-13T14:39:00.231 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-13T14:39:00.231 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-13T14:39:00.231 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-13T14:39:00.231 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-13T14:39:00.231 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-13T14:39:00.231 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-13T14:39:00.231 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-13T14:39:00.231 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-13T14:39:00.231 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-13T14:39:00.231 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-13T14:39:00.231 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-13T14:39:00.231 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-13T14:39:00.231 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-13T14:39:00.232 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-13T14:39:00.232 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-13T14:39:00.232 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-13T14:39:00.232 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-13T14:39:00.232 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-13T14:39:00.232 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-13T14:39:00.232 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-13T14:39:00.232 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-13T14:39:00.232 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-13T14:39:00.232 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mtime_update_interval": "60", 2024-09-13T14:39:00.232 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_non_blocking_aio": "true", 2024-09-13T14:39:00.232 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_op_thread_timeout": "60", 2024-09-13T14:39:00.232 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_op_threads": "1", 2024-09-13T14:39:00.233 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_parent_cache_enabled": "false", 2024-09-13T14:39:00.233 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-13T14:39:00.233 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-13T14:39:00.233 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-13T14:39:00.233 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_plugins": "", 2024-09-13T14:39:00.233 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_bps_burst": "0", 2024-09-13T14:39:00.233 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-13T14:39:00.233 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_bps_limit": "0", 2024-09-13T14:39:00.233 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_exclude_ops": "0", 2024-09-13T14:39:00.233 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_iops_burst": "0", 2024-09-13T14:39:00.233 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-13T14:39:00.233 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_iops_limit": "0", 2024-09-13T14:39:00.233 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-13T14:39:00.233 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-13T14:39:00.234 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-13T14:39:00.234 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-13T14:39:00.234 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-13T14:39:00.234 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-13T14:39:00.234 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-13T14:39:00.234 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-13T14:39:00.234 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-13T14:39:00.234 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-13T14:39:00.234 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-13T14:39:00.234 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-13T14:39:00.234 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-13T14:39:00.234 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-13T14:39:00.234 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_read_from_replica_policy": "default", 2024-09-13T14:39:00.234 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-13T14:39:00.235 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-13T14:39:00.235 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-13T14:39:00.235 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-13T14:39:00.235 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_skip_partial_discard": "true", 2024-09-13T14:39:00.235 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-13T14:39:00.235 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_tracing": "false", 2024-09-13T14:39:00.235 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_validate_names": "true", 2024-09-13T14:39:00.235 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_validate_pool": "true", 2024-09-13T14:39:00.235 INFO:teuthology.orchestra.run.smithi151.stdout: "restapi_base_url": "", 2024-09-13T14:39:00.235 INFO:teuthology.orchestra.run.smithi151.stdout: "restapi_log_level": "", 2024-09-13T14:39:00.235 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-13T14:39:00.235 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-13T14:39:00.235 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_acl_grants_max_num": "100", 2024-09-13T14:39:00.236 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_admin_entry": "admin", 2024-09-13T14:39:00.236 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-13T14:39:00.236 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_asio_assert_yielding": "false", 2024-09-13T14:39:00.236 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_backend_store": "rados", 2024-09-13T14:39:00.236 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_barbican_url": "", 2024-09-13T14:39:00.236 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_beast_enable_async": "true", 2024-09-13T14:39:00.236 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_bucket_counters_cache": "false", 2024-09-13T14:39:00.236 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-13T14:39:00.236 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-13T14:39:00.236 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-13T14:39:00.236 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-13T14:39:00.236 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-13T14:39:00.237 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-13T14:39:00.237 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-13T14:39:00.237 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-13T14:39:00.237 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_cache_enabled": "true", 2024-09-13T14:39:00.237 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_cache_expiry_interval": "900", 2024-09-13T14:39:00.237 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_cache_lru_size": "25000", 2024-09-13T14:39:00.237 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_config_store": "rados", 2024-09-13T14:39:00.237 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_content_length_compat": "false", 2024-09-13T14:39:00.237 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_copy_obj_progress": "true", 2024-09-13T14:39:00.237 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-13T14:39:00.237 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_cors_rules_max_num": "100", 2024-09-13T14:39:00.237 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_cross_domain_policy": "", 2024-09-13T14:39:00.237 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-13T14:39:00.238 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_kmip_addr": "", 2024-09-13T14:39:00.238 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-13T14:39:00.238 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-13T14:39:00.238 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-13T14:39:00.238 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-13T14:39:00.238 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_kmip_password": "", 2024-09-13T14:39:00.238 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-13T14:39:00.238 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_kmip_username": "", 2024-09-13T14:39:00.238 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_require_ssl": "true", 2024-09-13T14:39:00.238 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-13T14:39:00.238 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-13T14:39:00.238 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-13T14:39:00.238 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-13T14:39:00.239 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-13T14:39:00.239 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-13T14:39:00.239 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-13T14:39:00.239 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-13T14:39:00.239 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-13T14:39:00.239 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-13T14:39:00.239 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-13T14:39:00.239 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-13T14:39:00.239 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-13T14:39:00.239 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-13T14:39:00.239 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-13T14:39:00.239 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_vault_addr": "", 2024-09-13T14:39:00.239 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_vault_auth": "token", 2024-09-13T14:39:00.239 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_vault_namespace": "", 2024-09-13T14:39:00.240 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_vault_prefix": "", 2024-09-13T14:39:00.240 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-13T14:39:00.240 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-13T14:39:00.240 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-13T14:39:00.240 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-13T14:39:00.240 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_vault_token_file": "", 2024-09-13T14:39:00.240 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-13T14:39:00.240 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_curl_buffersize": "524288", 2024-09-13T14:39:00.240 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-13T14:39:00.241 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_curl_low_speed_time": "300", 2024-09-13T14:39:00.241 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-13T14:39:00.241 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-13T14:39:00.241 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-13T14:39:00.241 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-13T14:39:00.241 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-13T14:39:00.241 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-13T14:39:00.241 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-13T14:39:00.241 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-13T14:39:00.241 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-13T14:39:00.241 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-13T14:39:00.241 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-09-13T14:39:00.241 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-09-13T14:39:00.241 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-09-13T14:39:00.242 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-09-13T14:39:00.242 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-09-13T14:39:00.242 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d4n_l1_fadvise": "4", 2024-09-13T14:39:00.242 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-09-13T14:39:00.242 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-09-13T14:39:00.242 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-13T14:39:00.242 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_data_log_changes_size": "1000", 2024-09-13T14:39:00.242 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_data_log_num_shards": "128", 2024-09-13T14:39:00.242 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-13T14:39:00.242 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_data_log_window": "30", 2024-09-13T14:39:00.242 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-13T14:39:00.242 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-13T14:39:00.242 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-13T14:39:00.243 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-13T14:39:00.243 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-13T14:39:00.243 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-13T14:39:00.243 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-13T14:39:00.243 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-13T14:39:00.243 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-13T14:39:00.243 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-13T14:39:00.243 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-13T14:39:00.243 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-13T14:39:00.243 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-13T14:39:00.243 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_disable_s3select": "false", 2024-09-13T14:39:00.243 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-13T14:39:00.243 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-13T14:39:00.243 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-13T14:39:00.244 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-13T14:39:00.244 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-13T14:39:00.244 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-13T14:39:00.244 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-13T14:39:00.244 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-13T14:39:00.244 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-13T14:39:00.244 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-13T14:39:00.244 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-13T14:39:00.244 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-13T14:39:00.244 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dns_name": "", 2024-09-13T14:39:00.244 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dns_s3website_name": "", 2024-09-13T14:39:00.244 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dynamic_resharding": "true", 2024-09-13T14:39:00.244 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-09-13T14:39:00.245 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-09-13T14:39:00.245 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-13T14:39:00.245 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_enable_gc_threads": "true", 2024-09-13T14:39:00.245 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_enable_lc_threads": "true", 2024-09-13T14:39:00.245 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_enable_mdsearch": "true", 2024-09-13T14:39:00.245 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_enable_ops_log": "false", 2024-09-13T14:39:00.245 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_enable_quota_threads": "true", 2024-09-13T14:39:00.245 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_enable_static_website": "false", 2024-09-13T14:39:00.245 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_enable_usage_log": "false", 2024-09-13T14:39:00.245 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_enforce_swift_acls": "true", 2024-09-13T14:39:00.245 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_exit_timeout_secs": "120", 2024-09-13T14:39:00.245 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_expose_bucket": "false", 2024-09-13T14:39:00.245 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_extended_http_attrs": "", 2024-09-13T14:39:00.246 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_filter": "none", 2024-09-13T14:39:00.246 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-13T14:39:00.246 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_frontends": "beast port=7480", 2024-09-13T14:39:00.246 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-13T14:39:00.246 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_gc_max_deferred": "50", 2024-09-13T14:39:00.246 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-13T14:39:00.246 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_gc_max_objs": "32", 2024-09-13T14:39:00.246 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-13T14:39:00.246 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-13T14:39:00.246 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-13T14:39:00.246 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-13T14:39:00.246 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_gc_processor_period": "3600", 2024-09-13T14:39:00.246 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-13T14:39:00.247 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-13T14:39:00.247 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-13T14:39:00.247 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-13T14:39:00.247 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_init_timeout": "300", 2024-09-13T14:39:00.247 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-13T14:39:00.247 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-13T14:39:00.247 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_kafka_connection_idle": "300", 2024-09-13T14:39:00.247 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-13T14:39:00.247 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-13T14:39:00.247 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-13T14:39:00.247 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-13T14:39:00.247 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-13T14:39:00.247 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_admin_domain": "", 2024-09-13T14:39:00.248 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_admin_password": "", 2024-09-13T14:39:00.248 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_admin_password_path": "", 2024-09-13T14:39:00.248 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_admin_project": "", 2024-09-13T14:39:00.248 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_admin_tenant": "", 2024-09-13T14:39:00.248 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_admin_token": "", 2024-09-13T14:39:00.248 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_admin_token_path": "", 2024-09-13T14:39:00.248 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_admin_user": "", 2024-09-13T14:39:00.248 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_barbican_domain": "", 2024-09-13T14:39:00.248 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_barbican_password": "", 2024-09-13T14:39:00.248 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_barbican_project": "", 2024-09-13T14:39:00.248 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-13T14:39:00.248 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_barbican_user": "", 2024-09-13T14:39:00.249 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-13T14:39:00.249 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-13T14:39:00.249 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-13T14:39:00.249 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-13T14:39:00.249 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-13T14:39:00.249 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_url": "", 2024-09-13T14:39:00.249 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-13T14:39:00.249 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_lc_debug_interval": "-1", 2024-09-13T14:39:00.249 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_lc_lock_max_time": "90", 2024-09-13T14:39:00.249 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_lc_max_objs": "32", 2024-09-13T14:39:00.249 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_lc_max_rules": "1000", 2024-09-13T14:39:00.249 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_lc_max_worker": "3", 2024-09-13T14:39:00.249 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-13T14:39:00.249 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_lc_thread_delay": "0", 2024-09-13T14:39:00.250 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-13T14:39:00.250 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ldap_dnattr": "uid", 2024-09-13T14:39:00.250 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-13T14:39:00.250 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ldap_searchfilter": "", 2024-09-13T14:39:00.250 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-13T14:39:00.250 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ldap_uri": "", 2024-09-13T14:39:00.250 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_lfuda_sync_frequency": "60", 2024-09-13T14:39:00.250 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-13T14:39:00.250 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-13T14:39:00.250 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-13T14:39:00.250 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_log_http_headers": "", 2024-09-13T14:39:00.250 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-13T14:39:00.250 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-13T14:39:00.251 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_log_object_name_utc": "false", 2024-09-13T14:39:00.251 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-13T14:39:00.251 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-13T14:39:00.251 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_attr_name_len": "0", 2024-09-13T14:39:00.251 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_attr_size": "0", 2024-09-13T14:39:00.251 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-13T14:39:00.251 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_chunk_size": "4194304", 2024-09-13T14:39:00.251 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-13T14:39:00.251 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-13T14:39:00.251 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-13T14:39:00.251 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_listing_results": "1000", 2024-09-13T14:39:00.251 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_notify_retries": "10", 2024-09-13T14:39:00.251 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-13T14:39:00.251 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_put_param_size": "1048576", 2024-09-13T14:39:00.252 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_put_size": "5368709120", 2024-09-13T14:39:00.252 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_slo_entries": "1000", 2024-09-13T14:39:00.252 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_md_log_max_shards": "64", 2024-09-13T14:39:00.252 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-13T14:39:00.252 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-13T14:39:00.252 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-13T14:39:00.252 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-13T14:39:00.252 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_mp_lock_max_time": "600", 2024-09-13T14:39:00.252 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-13T14:39:00.252 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-13T14:39:00.252 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-13T14:39:00.252 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-13T14:39:00.252 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-13T14:39:00.253 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-13T14:39:00.253 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-13T14:39:00.253 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-13T14:39:00.253 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_max_gc": "300", 2024-09-13T14:39:00.253 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-13T14:39:00.253 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-13T14:39:00.253 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-13T14:39:00.253 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-13T14:39:00.253 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-13T14:39:00.253 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-13T14:39:00.253 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-13T14:39:00.253 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_num_async_rados_threads": "32", 2024-09-13T14:39:00.253 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_num_control_oids": "8", 2024-09-13T14:39:00.253 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_numa_node": "-1", 2024-09-13T14:39:00.254 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-13T14:39:00.254 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-13T14:39:00.254 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_objexp_chunk_size": "100", 2024-09-13T14:39:00.254 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_objexp_gc_interval": "600", 2024-09-13T14:39:00.254 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-13T14:39:00.254 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-13T14:39:00.254 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-13T14:39:00.254 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_op_thread_timeout": "600", 2024-09-13T14:39:00.254 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_op_tracing": "false", 2024-09-13T14:39:00.254 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_opa_token": "", 2024-09-13T14:39:00.254 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_opa_url": "", 2024-09-13T14:39:00.254 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_opa_verify_ssl": "true", 2024-09-13T14:39:00.254 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-13T14:39:00.254 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-13T14:39:00.255 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ops_log_rados": "false", 2024-09-13T14:39:00.255 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ops_log_socket_path": "", 2024-09-13T14:39:00.255 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-13T14:39:00.255 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-13T14:39:00.255 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-13T14:39:00.255 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_period_push_interval": "2.000000", 2024-09-13T14:39:00.255 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-13T14:39:00.255 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-13T14:39:00.255 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-13T14:39:00.255 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-13T14:39:00.255 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_posix_cache_lanes": "3", 2024-09-13T14:39:00.255 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-13T14:39:00.255 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-13T14:39:00.256 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_posix_cache_partitions": "3", 2024-09-13T14:39:00.256 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-13T14:39:00.256 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_print_continue": "true", 2024-09-13T14:39:00.256 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-13T14:39:00.256 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-13T14:39:00.256 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-13T14:39:00.256 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-13T14:39:00.256 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-13T14:39:00.256 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_rados_tracing": "false", 2024-09-13T14:39:00.256 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_realm": "", 2024-09-13T14:39:00.256 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_realm_id": "", 2024-09-13T14:39:00.256 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-13T14:39:00.256 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_region": "", 2024-09-13T14:39:00.257 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-13T14:39:00.257 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-13T14:39:00.257 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-13T14:39:00.257 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-13T14:39:00.257 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_request_uri": "", 2024-09-13T14:39:00.257 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_reshard_batch_size": "64", 2024-09-13T14:39:00.257 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-13T14:39:00.257 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_reshard_debug_interval": "-1", 2024-09-13T14:39:00.257 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_reshard_max_aio": "128", 2024-09-13T14:39:00.257 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_reshard_num_logs": "16", 2024-09-13T14:39:00.257 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_reshard_thread_interval": "600", 2024-09-13T14:39:00.257 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_resolve_cname": "false", 2024-09-13T14:39:00.257 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-13T14:39:00.257 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_run_sync_thread": "true", 2024-09-13T14:39:00.258 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-13T14:39:00.258 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-13T14:39:00.258 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-13T14:39:00.258 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-13T14:39:00.258 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-13T14:39:00.258 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-13T14:39:00.258 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-13T14:39:00.258 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-13T14:39:00.258 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-13T14:39:00.258 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_scheduler_type": "throttler", 2024-09-13T14:39:00.258 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_script_uri": "", 2024-09-13T14:39:00.258 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_service_provider_name": "", 2024-09-13T14:39:00.258 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-13T14:39:00.258 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sts_client_id": "", 2024-09-13T14:39:00.259 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sts_client_secret": "", 2024-09-13T14:39:00.259 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sts_entry": "sts", 2024-09-13T14:39:00.259 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sts_key": "sts", 2024-09-13T14:39:00.259 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-13T14:39:00.259 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sts_min_session_duration": "900", 2024-09-13T14:39:00.259 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sts_token_introspection_url": "", 2024-09-13T14:39:00.259 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_account_in_url": "false", 2024-09-13T14:39:00.259 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_auth_entry": "auth", 2024-09-13T14:39:00.259 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_auth_url": "", 2024-09-13T14:39:00.259 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_custom_header": "", 2024-09-13T14:39:00.259 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-13T14:39:00.259 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_need_stats": "true", 2024-09-13T14:39:00.259 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_tenant_name": "", 2024-09-13T14:39:00.259 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_token_expiration": "86400", 2024-09-13T14:39:00.260 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_url": "", 2024-09-13T14:39:00.260 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_url_prefix": "swift", 2024-09-13T14:39:00.260 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-13T14:39:00.260 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-13T14:39:00.260 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-13T14:39:00.260 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_lease_period": "120", 2024-09-13T14:39:00.260 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-13T14:39:00.260 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-13T14:39:00.260 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-13T14:39:00.260 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-13T14:39:00.260 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-13T14:39:00.261 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-13T14:39:00.261 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-13T14:39:00.261 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-13T14:39:00.261 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-13T14:39:00.261 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_thread_pool_size": "512", 2024-09-13T14:39:00.261 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-13T14:39:00.261 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-13T14:39:00.261 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-13T14:39:00.261 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-13T14:39:00.261 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_torrent_comment": "", 2024-09-13T14:39:00.261 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_torrent_createby": "", 2024-09-13T14:39:00.261 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_torrent_encoding": "", 2024-09-13T14:39:00.261 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_torrent_flag": "false", 2024-09-13T14:39:00.261 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-13T14:39:00.262 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_torrent_origin": "", 2024-09-13T14:39:00.262 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-13T14:39:00.262 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_torrent_tracker": "", 2024-09-13T14:39:00.262 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_trust_forwarded_https": "false", 2024-09-13T14:39:00.262 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-13T14:39:00.262 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-13T14:39:00.262 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_usage_max_shards": "32", 2024-09-13T14:39:00.262 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_usage_max_user_shards": "1", 2024-09-13T14:39:00.262 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_use_opa_authz": "false", 2024-09-13T14:39:00.262 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_counters_cache": "false", 2024-09-13T14:39:00.262 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-13T14:39:00.262 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-13T14:39:00.262 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-13T14:39:00.262 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_max_buckets": "1000", 2024-09-13T14:39:00.263 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_policies_max_num": "100", 2024-09-13T14:39:00.263 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-13T14:39:00.263 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-13T14:39:00.263 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-13T14:39:00.263 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-13T14:39:00.263 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_unique_email": "true", 2024-09-13T14:39:00.263 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_verify_ssl": "true", 2024-09-13T14:39:00.263 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-13T14:39:00.263 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_zone": "", 2024-09-13T14:39:00.263 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_zone_id": "", 2024-09-13T14:39:00.263 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-13T14:39:00.263 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_zonegroup": "", 2024-09-13T14:39:00.264 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_zonegroup_id": "", 2024-09-13T14:39:00.264 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-13T14:39:00.264 INFO:teuthology.orchestra.run.smithi151.stdout: "rgwlc_auto_session_clear": "true", 2024-09-13T14:39:00.264 INFO:teuthology.orchestra.run.smithi151.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-13T14:39:00.264 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_block_size": "4096", 2024-09-13T14:39:00.264 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-13T14:39:00.264 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-13T14:39:00.264 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-13T14:39:00.264 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-13T14:39:00.264 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-13T14:39:00.264 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_cache_size": "536870912", 2024-09-13T14:39:00.264 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-13T14:39:00.265 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-13T14:39:00.265 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-13T14:39:00.265 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-13T14:39:00.265 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-13T14:39:00.265 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-13T14:39:00.265 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-13T14:39:00.265 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-13T14:39:00.265 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_index_type": "binary_search", 2024-09-13T14:39:00.265 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-13T14:39:00.265 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-13T14:39:00.265 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_partition_filters": "false", 2024-09-13T14:39:00.265 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_perf": "false", 2024-09-13T14:39:00.265 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-13T14:39:00.266 INFO:teuthology.orchestra.run.smithi151.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-13T14:39:00.266 INFO:teuthology.orchestra.run.smithi151.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-13T14:39:00.266 INFO:teuthology.orchestra.run.smithi151.stdout: "run_dir": "/var/run/ceph", 2024-09-13T14:39:00.266 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_block_create": "true", 2024-09-13T14:39:00.266 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_cache_lru_size": "67108864", 2024-09-13T14:39:00.266 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-13T14:39:00.266 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-13T14:39:00.266 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_default_max_object_size": "16777216", 2024-09-13T14:39:00.266 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-13T14:39:00.266 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_device_size": "53687091200", 2024-09-13T14:39:00.266 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-13T14:39:00.266 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_full_integrity_check": "false", 2024-09-13T14:39:00.266 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_journal_batch_capacity": "16", 2024-09-13T14:39:00.266 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-13T14:39:00.266 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-13T14:39:00.267 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-13T14:39:00.267 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_main_device_type": "SSD", 2024-09-13T14:39:00.267 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-13T14:39:00.267 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-13T14:39:00.267 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-13T14:39:00.267 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-13T14:39:00.267 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-13T14:39:00.267 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-13T14:39:00.267 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_segment_size": "67108864", 2024-09-13T14:39:00.267 INFO:teuthology.orchestra.run.smithi151.stdout: "set_keepcaps": "false", 2024-09-13T14:39:00.267 INFO:teuthology.orchestra.run.smithi151.stdout: "setgroup": "ceph", 2024-09-13T14:39:00.267 INFO:teuthology.orchestra.run.smithi151.stdout: "setuser": "ceph", 2024-09-13T14:39:00.267 INFO:teuthology.orchestra.run.smithi151.stdout: "setuser_match_path": "", 2024-09-13T14:39:00.268 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-13T14:39:00.268 INFO:teuthology.orchestra.run.smithi151.stdout: "thp": "false", 2024-09-13T14:39:00.268 INFO:teuthology.orchestra.run.smithi151.stdout: "threadpool_default_timeout": "60", 2024-09-13T14:39:00.268 INFO:teuthology.orchestra.run.smithi151.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-13T14:39:00.268 INFO:teuthology.orchestra.run.smithi151.stdout: "throttler_perf_counter": "true", 2024-09-13T14:39:00.268 INFO:teuthology.orchestra.run.smithi151.stdout: "tmp_dir": "/tmp", 2024-09-13T14:39:00.268 INFO:teuthology.orchestra.run.smithi151.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2024-09-13T14:39:00.268 INFO:teuthology.orchestra.run.smithi151.stdout: "uadk_compressor_enabled": "false", 2024-09-13T14:39:00.268 INFO:teuthology.orchestra.run.smithi151.stdout: "uadk_wd_sync_ctx_num": "2" 2024-09-13T14:39:00.268 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:39:00.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:00 smithi184 ceph-mon[92527]: pgmap v1271: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:00.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:39:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:39:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:39:00.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:00 smithi151 ceph-mon[104450]: pgmap v1271: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:00.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:00 smithi151 ceph-mon[108966]: pgmap v1271: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:01.090 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-13T14:39:01.337 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:02.550 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:02 smithi151 ceph-mon[104450]: pgmap v1272: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:02.551 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:02.551 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:02 smithi151 ceph-mon[108966]: pgmap v1272: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:02.551 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:02.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:02 smithi184 ceph-mon[92527]: pgmap v1272: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:02.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:03.046 INFO:teuthology.orchestra.run.smithi151.stdout:mon.a: {} 2024-09-13T14:39:03.046 INFO:teuthology.orchestra.run.smithi151.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T14:39:03.056 INFO:teuthology.orchestra.run.smithi151.stdout:mon.b: {} 2024-09-13T14:39:03.056 INFO:teuthology.orchestra.run.smithi151.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T14:39:03.064 INFO:teuthology.orchestra.run.smithi151.stdout:mon.c: {} 2024-09-13T14:39:03.065 INFO:teuthology.orchestra.run.smithi151.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T14:39:03.727 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:03 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2514769002' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T14:39:03.727 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:03 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2514769002' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T14:39:03.752 INFO:tasks.thrashosds.thrasher:Unable to test ceph-objectstore-tool, not available on all OSD nodes 2024-09-13T14:39:03.752 INFO:teuthology.task.sequential:In sequential, running task full_sequential... 2024-09-13T14:39:03.752 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T14:39:03.754 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T14:39:03.754 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-09-13T14:39:03.754 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_1 16 2024-09-13T14:39:03.754 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-13T14:39:03.755 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:39:03.755 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:39:03.755 INFO:tasks.thrashosds.thrasher:Reweighting osd 3 to 0.1741878036956055 2024-09-13T14:39:03.755 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 3 0.1741878036956055 2024-09-13T14:39:03.755 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-13T14:39:03.756 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:03.756 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-13T14:39:03.756 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:39:03.757 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-13T14:39:03.757 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:39:03.757 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-13T14:39:03.757 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:39:03.787 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:03.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:03.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:03.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:03 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2514769002' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T14:39:03.890 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:03.917 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:03.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:03.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:04.019 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:04.045 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:04.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:04.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:04.050 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:04.053 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:04.058 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:04.147 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:04.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:04.170 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:04.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:04.271 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:04.297 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:04.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:04.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:04.400 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:04.421 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:04.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:04.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:04.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:04 smithi151 ceph-mon[104450]: pgmap v1273: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:04.440 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:04 smithi151 ceph-mon[108966]: pgmap v1273: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:04.523 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:04.549 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:04.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:04.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:04.651 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:04.673 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:04.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:04.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:04.774 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:04.784 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:39:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:39:04] "GET /metrics HTTP/1.1" 200 34153 "" "Prometheus/2.51.0" 2024-09-13T14:39:04.796 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:04.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:04.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:04.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:04 smithi184 ceph-mon[92527]: pgmap v1273: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:04.898 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:04.919 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:04.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:04.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:05.021 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:05.047 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:05.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:05.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:05.149 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:05.170 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:05.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:05.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:05.272 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:05.299 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:05.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:05.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:05.404 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:05.429 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:05.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:05.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:05.531 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:05.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:05.556 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:05.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:05.632 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:05.657 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:05.679 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:05.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:05.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:05.781 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:05.806 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:05.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:05.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:05.908 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:05.934 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:05.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:05.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:06.036 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:06.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:06.059 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:06.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:06.160 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:06.186 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:06.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:06.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:06.288 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:06.309 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:06.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:06.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:06.411 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:06.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:06.435 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:06.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:06.536 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:06.562 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:06.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:06.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:06.664 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:06.687 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:06.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:06.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:06.694 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:06.705 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:06 smithi151 ceph-mon[104450]: pgmap v1274: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:06.705 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:06 smithi151 ceph-mon[108966]: pgmap v1274: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:06.789 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:06.814 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:06.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:06.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:06.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:06 smithi184 ceph-mon[92527]: pgmap v1274: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:06.916 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:06.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:06.938 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:06.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:07.041 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:07.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:07.063 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:07.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:07.164 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:07.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:07.187 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:07.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:07.288 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:07.310 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:07.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:07.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:07.412 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:07.438 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:07.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:07.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:07.540 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:07.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:07.565 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:07.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:07.666 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:07.689 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:07.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:07.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:07.791 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:07.816 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:07.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:07.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:07.918 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:07.941 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:07.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:07.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:08.042 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:08.064 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:08.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:08.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:08.166 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:08.189 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:08.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:08.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:08.290 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:08.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:08.313 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:08.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:08.414 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:08.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:08 smithi151 ceph-mon[104450]: pgmap v1275: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:08.428 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:08 smithi151 ceph-mon[108966]: pgmap v1275: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:08.440 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:08.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:08.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:08.542 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:08.567 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:08.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:08.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:08.669 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:08.697 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:08.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:08.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:08.788 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:39:08.788 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-13T14:39:08.788 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-13T14:39:08.788 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:39:08.799 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:08.821 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:08.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:08.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:08.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:08 smithi184 ceph-mon[92527]: pgmap v1275: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:08.923 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:08.949 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:08.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:08.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:09.051 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:09.076 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:09.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:09.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:09.178 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:09.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:09.201 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:09.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:09.302 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:09.327 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:09.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:09.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:09.429 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:09.452 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:09.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:09.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:09.461 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T14:39:09.555 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:09.579 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:09.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:09.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:09.681 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:09.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:09.703 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:09.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:09.804 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:09.827 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:09.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:09.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:09.929 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:09.955 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:09.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:09.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:10.057 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:10.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:10.083 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:10.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:10.184 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:10.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:10.207 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:10.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:10.308 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:10.337 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:10.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:10.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:10.379 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.3 to 0.174188 (2c97) 2024-09-13T14:39:10.396 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:10 smithi151 ceph-mon[108966]: pgmap v1276: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:10.396 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:10 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2712323249' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 0.1741878036956055} : dispatch 2024-09-13T14:39:10.396 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:10 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 0.1741878036956055} : dispatch 2024-09-13T14:39:10.396 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:10 smithi151 ceph-mon[104450]: pgmap v1276: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:10.396 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:10 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2712323249' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 0.1741878036956055} : dispatch 2024-09-13T14:39:10.397 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:10 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 0.1741878036956055} : dispatch 2024-09-13T14:39:10.439 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:10.462 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:10.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:10.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:10.564 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:10.586 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:10.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:10.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:10.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:10 smithi184 ceph-mon[92527]: pgmap v1276: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:10.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:10 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2712323249' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 0.1741878036956055} : dispatch 2024-09-13T14:39:10.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:10 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 0.1741878036956055} : dispatch 2024-09-13T14:39:10.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:39:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:39:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:39:10.688 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:10.709 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:10.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:10.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:10.811 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:10.835 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:10.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:10.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:10.937 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:10.960 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:10.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:10.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:11.063 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:11.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:11.086 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:11.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:11.188 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:11.211 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:11.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:11.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:11.313 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:11.336 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:11.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:11.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:11.439 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:11.461 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:11.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:11.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:11.538 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:11 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.1741878036956055}]': finished 2024-09-13T14:39:11.538 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:11 smithi151 ceph-mon[104450]: osdmap e1079: 8 total, 8 up, 8 in 2024-09-13T14:39:11.538 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:11 smithi151 ceph-mon[104450]: osdmap e1080: 8 total, 8 up, 8 in 2024-09-13T14:39:11.538 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:11 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.1741878036956055}]': finished 2024-09-13T14:39:11.538 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:11 smithi151 ceph-mon[108966]: osdmap e1079: 8 total, 8 up, 8 in 2024-09-13T14:39:11.539 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:11 smithi151 ceph-mon[108966]: osdmap e1080: 8 total, 8 up, 8 in 2024-09-13T14:39:11.563 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:11.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:11.586 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:11.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:11.637 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:39:11.638 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:39:11.653 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:39:11.653 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:39:11.671 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:39:11.671 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:39:11.687 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:11.688 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:39:11.688 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:39:11.705 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:39:11.705 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:39:11.709 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:11.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:11.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:11.720 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:39:11.720 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:39:11.735 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:39:11.735 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:39:11.751 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:39:11.751 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:39:11.811 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:11.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:11.834 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:11.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:11.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:11 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.1741878036956055}]': finished 2024-09-13T14:39:11.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:11 smithi184 ceph-mon[92527]: osdmap e1079: 8 total, 8 up, 8 in 2024-09-13T14:39:11.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:11 smithi184 ceph-mon[92527]: osdmap e1080: 8 total, 8 up, 8 in 2024-09-13T14:39:11.935 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:11.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:11.958 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:11.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:12.059 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:12.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:12.083 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:12.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:12.184 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:12.206 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:12.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:12.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:12.308 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:12.329 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:12.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:12.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:12.408 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-13T14:39:12.432 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:12.454 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:12.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:12.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:12.556 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:12.578 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:12.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:12.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:12.679 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:12.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:12 smithi151 ceph-mon[104450]: pgmap v1278: 73 pgs: 4 peering, 69 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:12.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:12 smithi151 ceph-mon[104450]: Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-13T14:39:12.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:12 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3187840644' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:39:12.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:12 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3875631606' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:39:12.699 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:12 smithi151 ceph-mon[108966]: pgmap v1278: 73 pgs: 4 peering, 69 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:12.699 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:12 smithi151 ceph-mon[108966]: Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-13T14:39:12.699 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:12 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3187840644' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:39:12.699 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:12 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3875631606' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:39:12.701 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:12.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:12.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:12.803 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:12.824 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:12.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:12.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:12.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:12 smithi184 ceph-mon[92527]: pgmap v1278: 73 pgs: 4 peering, 69 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:12.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:12 smithi184 ceph-mon[92527]: Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-13T14:39:12.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:12 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3187840644' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:39:12.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:12 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3875631606' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:39:12.926 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:12.948 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:12.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:12.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:13.050 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:13.071 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:13.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:13.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:13.173 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:13.196 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:13.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:13.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:13.298 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:13.320 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:13.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:13.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:13.422 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:13.445 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:13.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:13.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:13.546 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:13.561 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:13.568 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:13.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:13.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:13.651 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:13 smithi151 ceph-mon[108966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:39:13.651 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:13 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3875631606' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:39:13.651 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:13 smithi151 ceph-mon[108966]: osdmap e1081: 8 total, 8 up, 8 in 2024-09-13T14:39:13.652 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:13 smithi151 ceph-mon[104450]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:39:13.652 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:13 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3875631606' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:39:13.652 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:13 smithi151 ceph-mon[104450]: osdmap e1081: 8 total, 8 up, 8 in 2024-09-13T14:39:13.670 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:13.691 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:13.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:13.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:13.755 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:13 smithi184 ceph-mon[92527]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:39:13.755 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:13 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3875631606' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:39:13.755 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:13 smithi184 ceph-mon[92527]: osdmap e1081: 8 total, 8 up, 8 in 2024-09-13T14:39:13.793 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:13.815 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:13.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:13.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:13.917 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:13.938 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:13.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:13.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:14.041 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:14.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:14.065 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:14.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:14.166 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:14.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:14.190 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:14.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:14.291 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:14.312 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:14.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:14.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:14.414 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:14.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:14.437 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:14.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:14.538 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:14.560 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:14.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:14.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:14.580 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:14 smithi151 ceph-mon[104450]: pgmap v1281: 73 pgs: 4 peering, 69 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:14.580 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:14 smithi151 ceph-mon[108966]: pgmap v1281: 73 pgs: 4 peering, 69 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:14.662 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:14.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:14.685 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:14.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:14.786 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:14.808 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:14.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:14.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:14.833 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:39:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:39:14] "GET /metrics HTTP/1.1" 200 34153 "" "Prometheus/2.51.0" 2024-09-13T14:39:14.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:14 smithi184 ceph-mon[92527]: pgmap v1281: 73 pgs: 4 peering, 69 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:14.909 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:14.931 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:14.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:14.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:15.032 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:15.054 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:15.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:15.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:15.061 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:39:15.156 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:15.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:15.178 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:15.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:15.278 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:15.300 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:15.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:15.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:15.402 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:15.420 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_1' created 2024-09-13T14:39:15.427 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:15.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:15.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:15.529 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:15.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:15.552 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:15.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:15.653 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:15.675 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:15.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:15.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:15.778 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:15.799 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:15.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:15.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:15.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:15 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/232158379' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-13T14:39:15.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:15 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-13T14:39:15.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:15 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/232158379' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-13T14:39:15.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:15 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-13T14:39:15.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:15 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/232158379' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-13T14:39:15.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:15 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-13T14:39:15.902 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:15.923 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:15.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:15.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:16.007 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:39:16.026 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:16.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:16.076 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:16.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:16.177 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:16.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:16.201 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:16.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:16.297 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-09-13T14:39:16.302 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:16.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:16.325 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:16.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:16.425 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:16.452 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:16.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:16.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:16.554 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:16.577 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:16.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:16.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:16.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:16 smithi151 ceph-mon[104450]: pgmap v1282: 73 pgs: 4 peering, 69 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-13T14:39:16.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:16 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-13T14:39:16.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:16 smithi151 ceph-mon[104450]: osdmap e1082: 8 total, 8 up, 8 in 2024-09-13T14:39:16.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:39:16.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:16.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:16 smithi151 ceph-mon[108966]: pgmap v1282: 73 pgs: 4 peering, 69 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-13T14:39:16.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:16 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-13T14:39:16.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:16 smithi151 ceph-mon[108966]: osdmap e1082: 8 total, 8 up, 8 in 2024-09-13T14:39:16.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:39:16.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:16.679 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:16.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:16.703 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:16.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:16.804 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:16.826 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:16.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:16.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:16.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:16 smithi184 ceph-mon[92527]: pgmap v1282: 73 pgs: 4 peering, 69 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-13T14:39:16.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:16 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-13T14:39:16.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:16 smithi184 ceph-mon[92527]: osdmap e1082: 8 total, 8 up, 8 in 2024-09-13T14:39:16.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:39:16.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:16.882 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:39:16.883 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:39:16.928 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:16.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:16.954 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:16.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:17.055 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:17.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:17.078 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:17.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:17.180 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:17.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:17.201 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:17.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:17.302 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:17.323 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:17.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:17.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:17.425 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:17.446 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:17.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:17.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:17.480 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:17 smithi151 ceph-mon[104450]: osdmap e1083: 8 total, 8 up, 8 in 2024-09-13T14:39:17.481 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:17 smithi151 ceph-mon[108966]: osdmap e1083: 8 total, 8 up, 8 in 2024-09-13T14:39:17.502 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:17.506 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T14:39:17.545 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:17.548 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:17.549 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:17.572 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:17.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:17.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:17.674 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:17.695 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:17.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:17.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:17.797 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:17.818 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:17.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:17.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:17.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:17 smithi184 ceph-mon[92527]: osdmap e1083: 8 total, 8 up, 8 in 2024-09-13T14:39:17.920 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:17.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:17.942 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:17.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:18.043 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:18.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:18.066 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:18.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:18.167 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:18.189 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:18.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:18.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:18.292 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:18.314 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:18.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:18.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:18.416 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:18.437 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:18.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:18.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:18.494 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:39:18.494 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:39:18.495 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-13T14:39:18.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.0 'trim stale osdmaps' 2024-09-13T14:39:18.539 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:18.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:18.562 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:18.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:18.664 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:18.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:18.689 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:18.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:18.786 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:18 smithi151 ceph-mon[104450]: pgmap v1285: 89 pgs: 1 creating+peering, 3 unknown, 85 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-13T14:39:18.786 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:18 smithi151 ceph-mon[104450]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:39:18.786 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:18 smithi151 ceph-mon[104450]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 4 pgs peering) 2024-09-13T14:39:18.786 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:18 smithi151 ceph-mon[104450]: osdmap e1084: 8 total, 8 up, 8 in 2024-09-13T14:39:18.786 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:18 smithi151 ceph-mon[108966]: pgmap v1285: 89 pgs: 1 creating+peering, 3 unknown, 85 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-13T14:39:18.786 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:18 smithi151 ceph-mon[108966]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:39:18.786 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:18 smithi151 ceph-mon[108966]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 4 pgs peering) 2024-09-13T14:39:18.786 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:18 smithi151 ceph-mon[108966]: osdmap e1084: 8 total, 8 up, 8 in 2024-09-13T14:39:18.791 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:18.812 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:18.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:18.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:18.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:18 smithi184 ceph-mon[92527]: pgmap v1285: 89 pgs: 1 creating+peering, 3 unknown, 85 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-13T14:39:18.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:18 smithi184 ceph-mon[92527]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:39:18.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:18 smithi184 ceph-mon[92527]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 4 pgs peering) 2024-09-13T14:39:18.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:18 smithi184 ceph-mon[92527]: osdmap e1084: 8 total, 8 up, 8 in 2024-09-13T14:39:18.914 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:18.936 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:18.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:18.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:19.038 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:19.060 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:19.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:19.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:19.162 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:19.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:19.184 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:19.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:19.286 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:19.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:19.309 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:19.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:19.411 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:19.432 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:19.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:19.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:19.534 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:19.557 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:19.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:19.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:19.659 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:19.681 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:19.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:19.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:19.783 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:19.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:19.806 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:19.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:19.910 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:19.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:19.932 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:19.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:20.032 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:20.054 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:20.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:20.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:20.157 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:20.175 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:39:20.176 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:39:20.179 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:20.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:20.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:20.192 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:39:20.193 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:39:20.208 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:39:20.209 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:39:20.225 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:39:20.225 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:39:20.240 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:39:20.241 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:39:20.256 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:39:20.256 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:39:20.271 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:39:20.271 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:39:20.281 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:20.287 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:39:20.287 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:39:20.303 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:20.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:20.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:20.405 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:20.426 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:20.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:20.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:20.528 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:20.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:20.551 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:20.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:20.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:39:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:39:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:39:20.652 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:20.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:20.675 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:20.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:20.776 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:20.798 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:20.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:20.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:20.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:20 smithi151 ceph-mon[104450]: pgmap v1287: 89 pgs: 1 creating+peering, 88 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-13T14:39:20.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:20 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1690437372' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:39:20.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:20 smithi151 ceph-mon[108966]: pgmap v1287: 89 pgs: 1 creating+peering, 88 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-13T14:39:20.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:20 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1690437372' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:39:20.900 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:20.923 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:20.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:20.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:21.030 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:21.051 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:21.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:21.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:21.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:20 smithi184 ceph-mon[92527]: pgmap v1287: 89 pgs: 1 creating+peering, 88 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-13T14:39:21.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:20 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1690437372' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:39:21.153 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:21.175 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:21.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:21.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:21.277 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:21.300 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:21.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:21.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:21.402 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:21.424 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:21.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:21.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:21.526 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:21.547 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:21.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:21.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:21.649 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:21.670 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:21.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:21.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:21.772 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:21.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:21.794 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:21.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:21.895 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:21.921 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:21.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:21.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:22.023 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:22.035 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:22.050 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:22.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:22.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:22.056 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:22.152 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:22.178 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:22.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:22.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:22.280 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:22.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:22.303 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:22.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:22.404 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:22.427 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:22.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:22.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:22.528 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:22.554 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:22.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:22.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:22.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:22 smithi151 ceph-mon[108966]: pgmap v1288: 89 pgs: 89 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:22.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:22 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:39:22.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:22 smithi151 ceph-mon[104450]: pgmap v1288: 89 pgs: 89 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:22.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:22 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:39:22.656 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:22.681 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:22.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:22.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:22.783 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:22.804 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:22.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:22.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:22.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:22 smithi184 ceph-mon[92527]: pgmap v1288: 89 pgs: 89 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:22.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:22 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:39:22.906 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:22.932 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:22.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:22.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:23.034 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:23.056 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:23.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:23.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:23.158 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:23.181 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:23.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:23.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:23.284 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:23.305 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:23.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:23.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:23.407 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:23.433 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:23.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:23.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:23.515 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-13T14:39:23.535 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:23.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:23.557 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:23.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:23.599 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:23 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1102580902' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:39:23.600 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:23 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:39:23.600 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:23 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1102580902' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:39:23.600 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:23 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:39:23.658 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:23.681 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:23.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:23.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:23.783 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:23.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:23.810 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:23.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:23.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:23 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1102580902' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:39:23.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:23 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:39:23.911 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:23.933 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:23.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:23.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:24.035 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:24.047 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:39:24.060 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:24.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:24.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:24.162 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:24.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:24.188 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:24.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:24.289 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:24.315 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:24.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:24.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:24.417 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:24.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:24.441 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:24.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:24.542 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:24.568 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:24.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:24.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:24.574 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:24 smithi151 ceph-mon[104450]: pgmap v1289: 89 pgs: 89 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:24.574 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:24 smithi151 ceph-mon[104450]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:39:24.574 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:24 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:39:24.574 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:24 smithi151 ceph-mon[104450]: osdmap e1085: 8 total, 8 up, 8 in 2024-09-13T14:39:24.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:24 smithi151 ceph-mon[108966]: pgmap v1289: 89 pgs: 89 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:24.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:24 smithi151 ceph-mon[108966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:39:24.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:24 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:39:24.574 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:24 smithi151 ceph-mon[108966]: osdmap e1085: 8 total, 8 up, 8 in 2024-09-13T14:39:24.670 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:24.692 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:24.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:24.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:24.794 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:24.820 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:24.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:24.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:24.824 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:39:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:39:24] "GET /metrics HTTP/1.1" 200 36592 "" "Prometheus/2.51.0" 2024-09-13T14:39:24.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:24 smithi184 ceph-mon[92527]: pgmap v1289: 89 pgs: 89 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:24.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:24 smithi184 ceph-mon[92527]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:39:24.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:24 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:39:24.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:24 smithi184 ceph-mon[92527]: osdmap e1085: 8 total, 8 up, 8 in 2024-09-13T14:39:24.922 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:24.948 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:24.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:24.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:25.050 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:25.071 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:25.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:25.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:25.173 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:25.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:25.195 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:25.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:25.296 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:25.322 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:25.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:25.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:25.424 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:25.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:25.450 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:25.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:25.513 INFO:teuthology.orchestra.run.smithi151.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-09-13T14:39:25.551 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:25.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:25.574 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:25.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:25.675 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:25.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:25.702 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:25.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:25.803 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:25.829 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:25.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:25.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:25.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:25 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/696078941' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:39:25.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:25 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/696078941' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:39:25.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:25 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/696078941' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:39:25.930 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:25.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:25.956 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:25.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:26.057 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:26.082 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:26.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:26.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:26.183 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:26.208 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:26.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:26.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:26.311 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:26.321 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:39:26.335 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:26.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:26.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:26.437 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:26.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:26.460 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:26.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:26.561 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:26.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:26.585 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:26.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:26.686 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:26.708 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:26.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:26.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:26.810 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:26.832 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:26.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:26.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:26.835 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:26 smithi151 ceph-mon[104450]: pgmap v1291: 89 pgs: 89 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:26.835 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:26 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/696078941' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:39:26.835 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:26 smithi151 ceph-mon[104450]: osdmap e1086: 8 total, 8 up, 8 in 2024-09-13T14:39:26.835 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:26 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:39:26.835 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:26 smithi151 ceph-mon[108966]: pgmap v1291: 89 pgs: 89 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:26.835 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:26 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/696078941' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:39:26.836 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:26 smithi151 ceph-mon[108966]: osdmap e1086: 8 total, 8 up, 8 in 2024-09-13T14:39:26.836 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:26 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:39:26.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:26 smithi184 ceph-mon[92527]: pgmap v1291: 89 pgs: 89 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:26.932 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:26 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/696078941' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:39:26.932 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:26 smithi184 ceph-mon[92527]: osdmap e1086: 8 total, 8 up, 8 in 2024-09-13T14:39:26.932 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:26 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:39:26.933 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:39:26.934 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-13T14:39:26.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-13T14:39:26.934 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-13T14:39:26.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.934 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.14820.0:6015 5.5 5:a01dbb68:::smithi15186934-15:head [write 2112507~509299] snapc 18c=[18c,18b,18a,189,187,186,185,181,180,17f,17c,178,174,172,16a] ondisk+write+known_if_redirected+supports_pool_eio e677)", 2024-09-13T14:39:26.934 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:31:07.603424+0000", 2024-09-13T14:39:26.934 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 499.22975106199999, 2024-09-13T14:39:26.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014404716999999999, 2024-09-13T14:39:26.934 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:39:26.934 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:39:26.934 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:39:26.934 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:39:26.934 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.14820", 2024-09-13T14:39:26.935 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1096909476", 2024-09-13T14:39:26.935 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 6015 2024-09-13T14:39:26.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.935 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:39:26.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:39:26.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.603424+0000", 2024-09-13T14:39:26.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:26.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:39:26.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.603424+0000", 2024-09-13T14:39:26.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:26.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:39:26.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.603426+0000", 2024-09-13T14:39:26.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2069999999999998e-06 2024-09-13T14:39:26.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:39:26.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.603565+0000", 2024-09-13T14:39:26.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013856 2024-09-13T14:39:26.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:39:26.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.603588+0000", 2024-09-13T14:39:26.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3071e-05 2024-09-13T14:39:26.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:39:26.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.603597+0000", 2024-09-13T14:39:26.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4220000000000006e-06 2024-09-13T14:39:26.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:39:26.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.603673+0000", 2024-09-13T14:39:26.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5710999999999994e-05 2024-09-13T14:39:26.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:39:26.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.603762+0000", 2024-09-13T14:39:26.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8943999999999994e-05 2024-09-13T14:39:26.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:39:26.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.603879+0000", 2024-09-13T14:39:26.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011704799999999999 2024-09-13T14:39:26.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:39:26.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.605738+0000", 2024-09-13T14:39:26.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018593150000000001 2024-09-13T14:39:26.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:26.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.605987+0000", 2024-09-13T14:39:26.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024825200000000002 2024-09-13T14:39:26.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:26.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.617686+0000", 2024-09-13T14:39:26.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011699393000000001 2024-09-13T14:39:26.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:39:26.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.617759+0000", 2024-09-13T14:39:26.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2637000000000003e-05 2024-09-13T14:39:26.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:39:26.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.617829+0000", 2024-09-13T14:39:26.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0156999999999999e-05 2024-09-13T14:39:26.940 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:26.940 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:39:26.940 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:26.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.940 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.14820.0:6016 5.5 5:a01dbb68:::smithi15186934-15:head [setxattr _header (58) in=7b,truncate 2790160] snapc 18c=[18c,18b,18a,189,187,186,185,181,180,17f,17c,178,174,172,16a] ondisk+write+known_if_redirected+supports_pool_eio e677)", 2024-09-13T14:39:26.940 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:31:07.603622+0000", 2024-09-13T14:39:26.941 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 499.22955336199999, 2024-09-13T14:39:26.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01440486, 2024-09-13T14:39:26.941 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:39:26.941 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:39:26.941 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:39:26.941 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:39:26.941 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.14820", 2024-09-13T14:39:26.941 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1096909476", 2024-09-13T14:39:26.941 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 6016 2024-09-13T14:39:26.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.941 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:39:26.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:39:26.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.603622+0000", 2024-09-13T14:39:26.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:26.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:39:26.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.603622+0000", 2024-09-13T14:39:26.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:26.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:39:26.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.603622+0000", 2024-09-13T14:39:26.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4700000000000001e-07 2024-09-13T14:39:26.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:39:26.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.603626+0000", 2024-09-13T14:39:26.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0319999999999997e-06 2024-09-13T14:39:26.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:39:26.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.603635+0000", 2024-09-13T14:39:26.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2679999999999994e-06 2024-09-13T14:39:26.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:39:26.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.603641+0000", 2024-09-13T14:39:26.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8309999999999997e-06 2024-09-13T14:39:26.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:39:26.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.604898+0000", 2024-09-13T14:39:26.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012571150000000001 2024-09-13T14:39:26.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:39:26.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.604975+0000", 2024-09-13T14:39:26.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7119999999999993e-05 2024-09-13T14:39:26.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:39:26.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.605124+0000", 2024-09-13T14:39:26.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000148825 2024-09-13T14:39:26.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:39:26.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.605931+0000", 2024-09-13T14:39:26.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00080780200000000004 2024-09-13T14:39:26.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:26.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.606088+0000", 2024-09-13T14:39:26.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015652999999999999 2024-09-13T14:39:26.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:26.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.617885+0000", 2024-09-13T14:39:26.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011797512 2024-09-13T14:39:26.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:39:26.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.617956+0000", 2024-09-13T14:39:26.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0463000000000006e-05 2024-09-13T14:39:26.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:39:26.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.618027+0000", 2024-09-13T14:39:26.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0715000000000003e-05 2024-09-13T14:39:26.946 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:26.946 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:39:26.947 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:26.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.947 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.14820.0:6017 5.5 5:a01dbb68:::smithi15186934-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e677)", 2024-09-13T14:39:26.947 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:31:07.603659+0000", 2024-09-13T14:39:26.947 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 499.22951664599998, 2024-09-13T14:39:26.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014609051, 2024-09-13T14:39:26.947 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:39:26.947 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:39:26.947 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-13T14:39:26.947 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:39:26.947 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.14820", 2024-09-13T14:39:26.947 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1096909476", 2024-09-13T14:39:26.948 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 6017 2024-09-13T14:39:26.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.948 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:39:26.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:39:26.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.603659+0000", 2024-09-13T14:39:26.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:26.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:39:26.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.603659+0000", 2024-09-13T14:39:26.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:26.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:39:26.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.603659+0000", 2024-09-13T14:39:26.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9200000000000002e-07 2024-09-13T14:39:26.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:39:26.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.603661+0000", 2024-09-13T14:39:26.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.108e-06 2024-09-13T14:39:26.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:39:26.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.603666+0000", 2024-09-13T14:39:26.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8999999999999997e-06 2024-09-13T14:39:26.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:39:26.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.603669+0000", 2024-09-13T14:39:26.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.428e-06 2024-09-13T14:39:26.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:39:26.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.605778+0000", 2024-09-13T14:39:26.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021082459999999998 2024-09-13T14:39:26.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:39:26.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.605873+0000", 2024-09-13T14:39:26.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5872999999999995e-05 2024-09-13T14:39:26.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:39:26.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.618064+0000", 2024-09-13T14:39:26.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012190141999999999 2024-09-13T14:39:26.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:39:26.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.618164+0000", 2024-09-13T14:39:26.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000100318 2024-09-13T14:39:26.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:39:26.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:07.618268+0000", 2024-09-13T14:39:26.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103644 2024-09-13T14:39:26.952 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:26.952 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:39:26.952 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:26.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.952 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.14820.0:6062 5.3 5:c82f77fb:::smithi15186934-33:head [write 507809~642811] snapc 18e=[18e,18d,18c,18b,18a,189,187,186,185,181,180,17f,17c,178,174,172,16a] ondisk+write+known_if_redirected+supports_pool_eio e679)", 2024-09-13T14:39:26.952 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:31:09.577174+0000", 2024-09-13T14:39:26.952 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 497.25600128600001, 2024-09-13T14:39:26.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014294691999999999, 2024-09-13T14:39:26.953 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:39:26.953 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:39:26.953 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:39:26.953 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:39:26.953 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.14820", 2024-09-13T14:39:26.953 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1096909476", 2024-09-13T14:39:26.953 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 6062 2024-09-13T14:39:26.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.953 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:39:26.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:39:26.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:09.577174+0000", 2024-09-13T14:39:26.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:26.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:39:26.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:09.577174+0000", 2024-09-13T14:39:26.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:26.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:39:26.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:09.577176+0000", 2024-09-13T14:39:26.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6060000000000001e-06 2024-09-13T14:39:26.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:39:26.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:09.577507+0000", 2024-09-13T14:39:26.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000330761 2024-09-13T14:39:26.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:39:26.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:09.577533+0000", 2024-09-13T14:39:26.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6081e-05 2024-09-13T14:39:26.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:39:26.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:09.577544+0000", 2024-09-13T14:39:26.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0971999999999999e-05 2024-09-13T14:39:26.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:39:26.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:09.577613+0000", 2024-09-13T14:39:26.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8318999999999994e-05 2024-09-13T14:39:26.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:39:26.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:09.577692+0000", 2024-09-13T14:39:26.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9495999999999999e-05 2024-09-13T14:39:26.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:39:26.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:09.577815+0000", 2024-09-13T14:39:26.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012273300000000001 2024-09-13T14:39:26.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:39:26.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:09.579635+0000", 2024-09-13T14:39:26.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001820648 2024-09-13T14:39:26.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:26.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:09.591350+0000", 2024-09-13T14:39:26.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011714252 2024-09-13T14:39:26.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:26.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:09.591389+0000", 2024-09-13T14:39:26.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9604999999999998e-05 2024-09-13T14:39:26.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:39:26.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:09.591430+0000", 2024-09-13T14:39:26.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1100000000000003e-05 2024-09-13T14:39:26.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:39:26.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:31:09.591469+0000", 2024-09-13T14:39:26.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8118999999999999e-05 2024-09-13T14:39:26.958 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:26.958 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:39:26.959 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:26.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.959 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.14820.0:7558 5.5 5:aa4201b8:::smithi15186934-28:head [write 689384~664426] snapc 1c9=[1c9,1c8,1c7,1c5,1c2,1c1,1bd,1bc,1ba,1b1] ondisk+write+known_if_redirected+supports_pool_eio e804)", 2024-09-13T14:39:26.959 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:33:14.632741+0000", 2024-09-13T14:39:26.959 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 372.20043380800001, 2024-09-13T14:39:26.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.024649117000000002, 2024-09-13T14:39:26.959 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:39:26.959 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:39:26.959 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:39:26.959 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:39:26.959 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.14820", 2024-09-13T14:39:26.959 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1096909476", 2024-09-13T14:39:26.959 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 7558 2024-09-13T14:39:26.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.960 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:39:26.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:39:26.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.632741+0000", 2024-09-13T14:39:26.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:26.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:39:26.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.632741+0000", 2024-09-13T14:39:26.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:26.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:39:26.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.632743+0000", 2024-09-13T14:39:26.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1210000000000001e-06 2024-09-13T14:39:26.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:39:26.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.632964+0000", 2024-09-13T14:39:26.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00022052899999999999 2024-09-13T14:39:26.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:39:26.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.632988+0000", 2024-09-13T14:39:26.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4032999999999998e-05 2024-09-13T14:39:26.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:39:26.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.632998+0000", 2024-09-13T14:39:26.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0113e-05 2024-09-13T14:39:26.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:39:26.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.633066+0000", 2024-09-13T14:39:26.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8273000000000002e-05 2024-09-13T14:39:26.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:39:26.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.633152+0000", 2024-09-13T14:39:26.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5703000000000003e-05 2024-09-13T14:39:26.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:39:26.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.633294+0000", 2024-09-13T14:39:26.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000142261 2024-09-13T14:39:26.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:26.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.645702+0000", 2024-09-13T14:39:26.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012407581000000001 2024-09-13T14:39:26.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:26.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.657008+0000", 2024-09-13T14:39:26.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011305625 2024-09-13T14:39:26.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:39:26.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.657292+0000", 2024-09-13T14:39:26.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028482399999999998 2024-09-13T14:39:26.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:39:26.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.657334+0000", 2024-09-13T14:39:26.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1842999999999999e-05 2024-09-13T14:39:26.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:39:26.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.657390+0000", 2024-09-13T14:39:26.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6211000000000001e-05 2024-09-13T14:39:26.966 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:26.966 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:39:26.966 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:26.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.966 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.14820.0:7559 5.5 5:aa4201b8:::smithi15186934-28:head [write 1971506~687726] snapc 1c9=[1c9,1c8,1c7,1c5,1c2,1c1,1bd,1bc,1ba,1b1] ondisk+write+known_if_redirected+supports_pool_eio e804)", 2024-09-13T14:39:26.966 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:33:14.639914+0000", 2024-09-13T14:39:26.966 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 372.19326095899999, 2024-09-13T14:39:26.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019461755000000001, 2024-09-13T14:39:26.967 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:39:26.967 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:39:26.967 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:39:26.967 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:39:26.967 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.14820", 2024-09-13T14:39:26.967 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1096909476", 2024-09-13T14:39:26.967 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 7559 2024-09-13T14:39:26.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.967 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:39:26.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:39:26.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.639914+0000", 2024-09-13T14:39:26.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:26.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:39:26.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.639914+0000", 2024-09-13T14:39:26.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:26.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:39:26.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.639917+0000", 2024-09-13T14:39:26.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.328e-06 2024-09-13T14:39:26.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:39:26.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.640307+0000", 2024-09-13T14:39:26.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039002699999999998 2024-09-13T14:39:26.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:39:26.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.640322+0000", 2024-09-13T14:39:26.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5777000000000002e-05 2024-09-13T14:39:26.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:39:26.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.640330+0000", 2024-09-13T14:39:26.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.79e-06 2024-09-13T14:39:26.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:39:26.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.640387+0000", 2024-09-13T14:39:26.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6431000000000002e-05 2024-09-13T14:39:26.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:39:26.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.640465+0000", 2024-09-13T14:39:26.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8339999999999999e-05 2024-09-13T14:39:26.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:39:26.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.640592+0000", 2024-09-13T14:39:26.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012736600000000001 2024-09-13T14:39:26.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:26.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.645743+0000", 2024-09-13T14:39:26.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0051504230000000003 2024-09-13T14:39:26.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:39:26.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.659178+0000", 2024-09-13T14:39:26.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013435641999999999 2024-09-13T14:39:26.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:26.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.659302+0000", 2024-09-13T14:39:26.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012412 2024-09-13T14:39:26.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:39:26.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.659339+0000", 2024-09-13T14:39:26.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6100000000000003e-05 2024-09-13T14:39:26.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:39:26.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.659376+0000", 2024-09-13T14:39:26.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7410999999999998e-05 2024-09-13T14:39:26.976 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:26.976 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:39:26.976 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:26.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.977 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.14820.0:7560 5.5 5:aa4201b8:::smithi15186934-28:head [setxattr _header (58) in=7b,truncate 2689384] snapc 1c9=[1c9,1c8,1c7,1c5,1c2,1c1,1bd,1bc,1ba,1b1] ondisk+write+known_if_redirected+supports_pool_eio e804)", 2024-09-13T14:39:26.977 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:33:14.640351+0000", 2024-09-13T14:39:26.977 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 372.19282444599997, 2024-09-13T14:39:26.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019127379999999999, 2024-09-13T14:39:26.977 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:39:26.977 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:39:26.977 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:39:26.978 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:39:26.978 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.14820", 2024-09-13T14:39:26.978 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1096909476", 2024-09-13T14:39:26.978 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 7560 2024-09-13T14:39:26.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.978 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:39:26.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:39:26.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.640351+0000", 2024-09-13T14:39:26.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:26.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:39:26.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.640351+0000", 2024-09-13T14:39:26.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:26.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:39:26.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.640351+0000", 2024-09-13T14:39:26.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5300000000000004e-07 2024-09-13T14:39:26.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:39:26.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.640354+0000", 2024-09-13T14:39:26.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.508e-06 2024-09-13T14:39:26.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:39:26.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.640359+0000", 2024-09-13T14:39:26.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2909999999999998e-06 2024-09-13T14:39:26.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:39:26.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.640363+0000", 2024-09-13T14:39:26.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2880000000000003e-06 2024-09-13T14:39:26.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:39:26.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.641758+0000", 2024-09-13T14:39:26.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001394539 2024-09-13T14:39:26.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:39:26.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.641902+0000", 2024-09-13T14:39:26.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000144454 2024-09-13T14:39:26.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:39:26.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.642079+0000", 2024-09-13T14:39:26.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000177147 2024-09-13T14:39:26.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:26.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.645772+0000", 2024-09-13T14:39:26.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0036922779999999998 2024-09-13T14:39:26.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:39:26.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.659185+0000", 2024-09-13T14:39:26.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01341351 2024-09-13T14:39:26.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:26.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.659412+0000", 2024-09-13T14:39:26.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00022659199999999999 2024-09-13T14:39:26.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:39:26.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.659444+0000", 2024-09-13T14:39:26.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2601000000000002e-05 2024-09-13T14:39:26.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:39:26.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.659478+0000", 2024-09-13T14:39:26.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3618999999999998e-05 2024-09-13T14:39:26.990 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:26.990 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:39:26.990 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:26.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.990 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.14820.0:7561 5.5 5:aa4201b8:::smithi15186934-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e804)", 2024-09-13T14:39:26.990 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:33:14.640369+0000", 2024-09-13T14:39:26.990 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 372.19280648500001, 2024-09-13T14:39:26.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019252169, 2024-09-13T14:39:26.990 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:39:26.990 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:39:26.991 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-13T14:39:26.991 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:39:26.991 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.14820", 2024-09-13T14:39:26.991 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1096909476", 2024-09-13T14:39:26.991 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 7561 2024-09-13T14:39:26.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.991 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:39:26.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:39:26.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.640369+0000", 2024-09-13T14:39:26.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:26.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:39:26.992 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.640369+0000", 2024-09-13T14:39:26.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:26.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.992 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:39:26.992 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.640369+0000", 2024-09-13T14:39:26.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.53e-07 2024-09-13T14:39:26.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.992 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:39:26.992 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.640370+0000", 2024-09-13T14:39:26.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2780000000000001e-06 2024-09-13T14:39:26.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:39:26.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.640374+0000", 2024-09-13T14:39:26.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9500000000000003e-06 2024-09-13T14:39:26.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:39:26.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.640377+0000", 2024-09-13T14:39:26.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8789999999999999e-06 2024-09-13T14:39:26.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:39:26.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.642534+0000", 2024-09-13T14:39:26.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002156734 2024-09-13T14:39:26.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:39:26.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.642622+0000", 2024-09-13T14:39:26.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7811000000000003e-05 2024-09-13T14:39:26.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:39:26.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.659510+0000", 2024-09-13T14:39:26.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016888185 2024-09-13T14:39:26.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:39:26.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.659557+0000", 2024-09-13T14:39:26.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7284999999999999e-05 2024-09-13T14:39:26.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:39:26.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:14.659621+0000", 2024-09-13T14:39:26.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3794000000000006e-05 2024-09-13T14:39:26.995 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:26.995 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:39:26.995 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:26.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.995 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.14820.0:7959 5.5 5:a01dbb68:::smithi15186934-15:1c8 [read 0~2689351,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e845)", 2024-09-13T14:39:26.995 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:33:55.613430+0000", 2024-09-13T14:39:26.996 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 331.21974498600002, 2024-09-13T14:39:26.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018143115000000001, 2024-09-13T14:39:26.996 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:39:26.996 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:39:26.996 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-13T14:39:26.996 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:39:26.996 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.14820", 2024-09-13T14:39:26.996 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1096909476", 2024-09-13T14:39:26.996 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 7959 2024-09-13T14:39:26.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.996 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:39:26.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:39:26.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.613430+0000", 2024-09-13T14:39:26.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:26.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:39:26.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.613430+0000", 2024-09-13T14:39:26.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:26.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:39:26.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.613432+0000", 2024-09-13T14:39:26.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0590000000000001e-06 2024-09-13T14:39:26.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:39:26.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.613438+0000", 2024-09-13T14:39:26.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.229e-06 2024-09-13T14:39:26.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:39:26.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.613451+0000", 2024-09-13T14:39:26.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2617e-05 2024-09-13T14:39:26.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:39:26.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.613458+0000", 2024-09-13T14:39:26.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8560000000000001e-06 2024-09-13T14:39:26.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:39:26.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.613564+0000", 2024-09-13T14:39:26.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105992 2024-09-13T14:39:26.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:26.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-13T14:39:26.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.613571+0000", 2024-09-13T14:39:26.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7349999999999999e-06 2024-09-13T14:39:26.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:26.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:39:27.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.623678+0000", 2024-09-13T14:39:27.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010107656 2024-09-13T14:39:27.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:39:27.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.623786+0000", 2024-09-13T14:39:27.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000107345 2024-09-13T14:39:27.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:39:27.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.631573+0000", 2024-09-13T14:39:27.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0077876259999999997 2024-09-13T14:39:27.000 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:27.001 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:39:27.001 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:27.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.001 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.14820.0:7960 5.5 5:a01dbb68:::smithi15186934-15:1c8 [sparse-read 0~1212761] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e845)", 2024-09-13T14:39:27.001 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:33:55.613486+0000", 2024-09-13T14:39:27.001 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 331.21968922100001, 2024-09-13T14:39:27.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018372139999999999, 2024-09-13T14:39:27.002 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:39:27.002 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:39:27.002 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-13T14:39:27.002 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:39:27.002 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.14820", 2024-09-13T14:39:27.002 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1096909476", 2024-09-13T14:39:27.002 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 7960 2024-09-13T14:39:27.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.002 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:39:27.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:39:27.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.613486+0000", 2024-09-13T14:39:27.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:27.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:39:27.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.613486+0000", 2024-09-13T14:39:27.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:27.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:39:27.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.613486+0000", 2024-09-13T14:39:27.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7100000000000002e-07 2024-09-13T14:39:27.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:39:27.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.613490+0000", 2024-09-13T14:39:27.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.078e-06 2024-09-13T14:39:27.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:39:27.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.613497+0000", 2024-09-13T14:39:27.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0960000000000003e-06 2024-09-13T14:39:27.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.005 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:39:27.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.613502+0000", 2024-09-13T14:39:27.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2449999999999998e-06 2024-09-13T14:39:27.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.005 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:39:27.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.614089+0000", 2024-09-13T14:39:27.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00058659599999999999 2024-09-13T14:39:27.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting_for_map not empty", 2024-09-13T14:39:27.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.614094+0000", 2024-09-13T14:39:27.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6520000000000003e-06 2024-09-13T14:39:27.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:39:27.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.631639+0000", 2024-09-13T14:39:27.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017544649999999998 2024-09-13T14:39:27.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:39:27.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.631733+0000", 2024-09-13T14:39:27.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4469000000000006e-05 2024-09-13T14:39:27.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:39:27.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.631858+0000", 2024-09-13T14:39:27.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000124783 2024-09-13T14:39:27.007 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:27.007 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:39:27.007 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:27.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.008 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.14820.0:7961 5.5 5:a01dbb68:::smithi15186934-15:1c8 [read 0~2689351,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e845)", 2024-09-13T14:39:27.008 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:33:55.613510+0000", 2024-09-13T14:39:27.008 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 331.21966519300003, 2024-09-13T14:39:27.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019780890999999998, 2024-09-13T14:39:27.008 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:39:27.008 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:39:27.008 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-13T14:39:27.008 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:39:27.008 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.14820", 2024-09-13T14:39:27.008 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1096909476", 2024-09-13T14:39:27.009 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 7961 2024-09-13T14:39:27.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.009 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:39:27.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:39:27.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.613510+0000", 2024-09-13T14:39:27.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:27.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:39:27.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.613510+0000", 2024-09-13T14:39:27.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:27.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:39:27.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.613510+0000", 2024-09-13T14:39:27.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1600000000000002e-07 2024-09-13T14:39:27.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:39:27.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.613514+0000", 2024-09-13T14:39:27.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.641e-06 2024-09-13T14:39:27.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:39:27.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.613519+0000", 2024-09-13T14:39:27.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5230000000000003e-06 2024-09-13T14:39:27.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:39:27.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.613524+0000", 2024-09-13T14:39:27.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7770000000000002e-06 2024-09-13T14:39:27.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:39:27.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.614135+0000", 2024-09-13T14:39:27.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00061043899999999995 2024-09-13T14:39:27.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting_for_map not empty", 2024-09-13T14:39:27.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.614141+0000", 2024-09-13T14:39:27.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0209999999999996e-06 2024-09-13T14:39:27.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:39:27.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.631892+0000", 2024-09-13T14:39:27.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017751589000000002 2024-09-13T14:39:27.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:39:27.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.631994+0000", 2024-09-13T14:39:27.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101618 2024-09-13T14:39:27.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:39:27.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:33:55.633291+0000", 2024-09-13T14:39:27.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001296967 2024-09-13T14:39:27.014 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:27.014 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:39:27.014 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:27.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.014 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.6.0:961 5.2 e916/524 5:480436d5:::smithi15186934-45:1ca v 916'1615, mlcod=912'1611)", 2024-09-13T14:39:27.014 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:35:06.727325+0000", 2024-09-13T14:39:27.014 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 260.10585063899998, 2024-09-13T14:39:27.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013349471, 2024-09-13T14:39:27.015 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:39:27.015 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:39:27.015 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:39:27.015 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:39:27.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:39:27.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:35:06.727325+0000", 2024-09-13T14:39:27.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:27.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:39:27.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:35:06.727325+0000", 2024-09-13T14:39:27.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:27.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:39:27.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:35:06.727325+0000", 2024-09-13T14:39:27.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3999999999999997e-07 2024-09-13T14:39:27.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:39:27.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:35:06.727330+0000", 2024-09-13T14:39:27.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8420000000000004e-06 2024-09-13T14:39:27.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:39:27.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:35:06.727337+0000", 2024-09-13T14:39:27.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8040000000000002e-06 2024-09-13T14:39:27.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:39:27.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:35:06.727345+0000", 2024-09-13T14:39:27.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3490000000000003e-06 2024-09-13T14:39:27.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:39:27.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:35:06.727408+0000", 2024-09-13T14:39:27.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3125000000000003e-05 2024-09-13T14:39:27.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:39:27.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:35:06.727429+0000", 2024-09-13T14:39:27.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1263e-05 2024-09-13T14:39:27.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:39:27.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:35:06.740633+0000", 2024-09-13T14:39:27.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013204169 2024-09-13T14:39:27.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:39:27.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:35:06.740674+0000", 2024-09-13T14:39:27.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0578999999999999e-05 2024-09-13T14:39:27.019 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:27.019 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:39:27.019 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:27.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.019 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.2.0:1264 5.8 e916/591 5:104c5935:::smithi15186934-40:1de v 916'1310, mlcod=914'1306)", 2024-09-13T14:39:27.019 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:35:06.727705+0000", 2024-09-13T14:39:27.019 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 260.10547015899999, 2024-09-13T14:39:27.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013630494, 2024-09-13T14:39:27.020 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:39:27.020 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:39:27.020 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:39:27.020 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:39:27.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:39:27.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:35:06.727705+0000", 2024-09-13T14:39:27.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:27.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:39:27.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:35:06.727705+0000", 2024-09-13T14:39:27.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:27.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:39:27.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:35:06.727705+0000", 2024-09-13T14:39:27.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4600000000000001e-07 2024-09-13T14:39:27.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:39:27.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:35:06.727711+0000", 2024-09-13T14:39:27.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5799999999999999e-06 2024-09-13T14:39:27.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:39:27.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:35:06.727718+0000", 2024-09-13T14:39:27.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2189999999999998e-06 2024-09-13T14:39:27.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:39:27.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:35:06.727725+0000", 2024-09-13T14:39:27.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8399999999999997e-06 2024-09-13T14:39:27.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:39:27.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:35:06.727900+0000", 2024-09-13T14:39:27.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000175472 2024-09-13T14:39:27.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:39:27.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:35:06.727921+0000", 2024-09-13T14:39:27.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1120000000000001e-05 2024-09-13T14:39:27.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:39:27.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:35:06.741294+0000", 2024-09-13T14:39:27.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013372702 2024-09-13T14:39:27.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:39:27.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:35:06.741335+0000", 2024-09-13T14:39:27.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1315e-05 2024-09-13T14:39:27.024 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:27.024 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:39:27.024 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:27.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.025 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.5.0:1252 5.7 e916/557 5:e96e9b50:::smithi15186934-50:1c3 v 916'1420, mlcod=912'1416)", 2024-09-13T14:39:27.025 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:35:06.730470+0000", 2024-09-13T14:39:27.025 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 260.10270533200003, 2024-09-13T14:39:27.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013962967, 2024-09-13T14:39:27.025 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:39:27.025 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:39:27.025 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:39:27.025 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:39:27.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:39:27.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:35:06.730470+0000", 2024-09-13T14:39:27.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:27.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:39:27.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:35:06.730470+0000", 2024-09-13T14:39:27.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:27.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:39:27.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:35:06.730470+0000", 2024-09-13T14:39:27.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2800000000000003e-07 2024-09-13T14:39:27.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:39:27.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:35:06.730475+0000", 2024-09-13T14:39:27.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7840000000000003e-06 2024-09-13T14:39:27.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:39:27.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:35:06.730482+0000", 2024-09-13T14:39:27.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2150000000000004e-06 2024-09-13T14:39:27.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:39:27.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:35:06.730487+0000", 2024-09-13T14:39:27.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1030000000000001e-06 2024-09-13T14:39:27.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:39:27.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:35:06.731186+0000", 2024-09-13T14:39:27.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00069907200000000002 2024-09-13T14:39:27.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:39:27.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:35:06.731210+0000", 2024-09-13T14:39:27.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3218999999999999e-05 2024-09-13T14:39:27.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:39:27.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:35:06.744400+0000", 2024-09-13T14:39:27.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013190161000000001 2024-09-13T14:39:27.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:39:27.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:35:06.744433+0000", 2024-09-13T14:39:27.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3084999999999999e-05 2024-09-13T14:39:27.029 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:27.029 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:39:27.030 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:27.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.030 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.14820.0:9377 5.3 5:c82f77fb:::smithi15186934-33:head [append 0~0] snapc 229=[229,228,227,226,225,223,222,21f,21c,218,215,202,1f3] ondisk+write+known_if_redirected+supports_pool_eio e993)", 2024-09-13T14:39:27.030 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:36:23.730175+0000", 2024-09-13T14:39:27.030 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 183.103000108, 2024-09-13T14:39:27.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013261821, 2024-09-13T14:39:27.030 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:39:27.030 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:39:27.030 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:39:27.031 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:39:27.031 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.14820", 2024-09-13T14:39:27.031 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1096909476", 2024-09-13T14:39:27.031 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 9377 2024-09-13T14:39:27.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.031 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:39:27.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:39:27.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.730175+0000", 2024-09-13T14:39:27.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:27.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:39:27.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.730175+0000", 2024-09-13T14:39:27.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:27.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:39:27.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.730177+0000", 2024-09-13T14:39:27.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0700000000000001e-06 2024-09-13T14:39:27.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:39:27.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.730183+0000", 2024-09-13T14:39:27.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7350000000000001e-06 2024-09-13T14:39:27.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:39:27.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.730196+0000", 2024-09-13T14:39:27.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2843999999999999e-05 2024-09-13T14:39:27.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:39:27.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.730205+0000", 2024-09-13T14:39:27.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.081e-06 2024-09-13T14:39:27.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:39:27.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.730286+0000", 2024-09-13T14:39:27.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1531000000000003e-05 2024-09-13T14:39:27.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-13T14:39:27.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.730294+0000", 2024-09-13T14:39:27.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.661e-06 2024-09-13T14:39:27.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:39:27.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.732750+0000", 2024-09-13T14:39:27.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002455741 2024-09-13T14:39:27.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:39:27.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.732864+0000", 2024-09-13T14:39:27.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114541 2024-09-13T14:39:27.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:39:27.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.733076+0000", 2024-09-13T14:39:27.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00021175999999999999 2024-09-13T14:39:27.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:27.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.740533+0000", 2024-09-13T14:39:27.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0074565339999999999 2024-09-13T14:39:27.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:27.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.740573+0000", 2024-09-13T14:39:27.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0515000000000001e-05 2024-09-13T14:39:27.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:39:27.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.743335+0000", 2024-09-13T14:39:27.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027620420000000001 2024-09-13T14:39:27.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:39:27.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.743382+0000", 2024-09-13T14:39:27.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6513e-05 2024-09-13T14:39:27.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:39:27.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.743437+0000", 2024-09-13T14:39:27.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5253000000000001e-05 2024-09-13T14:39:27.037 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:27.037 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:39:27.037 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:27.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.038 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.14820.0:9378 5.3 5:c82f77fb:::smithi15186934-33:head [rollback 1f3] snapc 229=[229,228,227,226,225,223,222,21f,21c,218,215,202,1f3] ondisk+write+known_if_redirected+supports_pool_eio e993)", 2024-09-13T14:39:27.038 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:36:23.730236+0000", 2024-09-13T14:39:27.038 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 183.10293917499999, 2024-09-13T14:39:27.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013280449, 2024-09-13T14:39:27.038 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:39:27.038 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:39:27.038 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:39:27.038 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:39:27.038 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.14820", 2024-09-13T14:39:27.038 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1096909476", 2024-09-13T14:39:27.038 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 9378 2024-09-13T14:39:27.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.039 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:39:27.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:39:27.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.730236+0000", 2024-09-13T14:39:27.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:27.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:39:27.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.730236+0000", 2024-09-13T14:39:27.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:27.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:39:27.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.730237+0000", 2024-09-13T14:39:27.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7700000000000004e-07 2024-09-13T14:39:27.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:39:27.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.730240+0000", 2024-09-13T14:39:27.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4910000000000001e-06 2024-09-13T14:39:27.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:39:27.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.730249+0000", 2024-09-13T14:39:27.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1619999999999997e-06 2024-09-13T14:39:27.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:39:27.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.730257+0000", 2024-09-13T14:39:27.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3520000000000001e-06 2024-09-13T14:39:27.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:39:27.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.730359+0000", 2024-09-13T14:39:27.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010236199999999999 2024-09-13T14:39:27.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting_for_map not empty", 2024-09-13T14:39:27.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.730365+0000", 2024-09-13T14:39:27.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0460000000000002e-06 2024-09-13T14:39:27.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:39:27.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.736058+0000", 2024-09-13T14:39:27.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0056925220000000002 2024-09-13T14:39:27.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:39:27.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.736187+0000", 2024-09-13T14:39:27.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012932699999999999 2024-09-13T14:39:27.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:39:27.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.737754+0000", 2024-09-13T14:39:27.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015668850000000001 2024-09-13T14:39:27.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:27.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.740616+0000", 2024-09-13T14:39:27.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028617260000000002 2024-09-13T14:39:27.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:27.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.740654+0000", 2024-09-13T14:39:27.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8682e-05 2024-09-13T14:39:27.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:39:27.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.743442+0000", 2024-09-13T14:39:27.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027882699999999998 2024-09-13T14:39:27.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:39:27.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.743476+0000", 2024-09-13T14:39:27.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3801999999999999e-05 2024-09-13T14:39:27.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:39:27.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.743516+0000", 2024-09-13T14:39:27.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0145000000000003e-05 2024-09-13T14:39:27.045 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:27.045 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:39:27.045 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:27.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.046 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.14820.0:9379 5.3 5:c82f77fb:::smithi15186934-33:head [append 0~0] snapc 229=[229,228,227,226,225,223,222,21f,21c,218,215,202,1f3] ondisk+write+known_if_redirected+supports_pool_eio e993)", 2024-09-13T14:39:27.046 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:36:23.730269+0000", 2024-09-13T14:39:27.046 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 183.10290651400001, 2024-09-13T14:39:27.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013308107, 2024-09-13T14:39:27.046 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:39:27.046 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:39:27.046 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:39:27.046 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:39:27.046 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.14820", 2024-09-13T14:39:27.046 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1096909476", 2024-09-13T14:39:27.047 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 9379 2024-09-13T14:39:27.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.047 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:39:27.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:39:27.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.730269+0000", 2024-09-13T14:39:27.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:27.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:39:27.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.730269+0000", 2024-09-13T14:39:27.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:27.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:39:27.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.730269+0000", 2024-09-13T14:39:27.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0100000000000005e-07 2024-09-13T14:39:27.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:39:27.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.730272+0000", 2024-09-13T14:39:27.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1049999999999999e-06 2024-09-13T14:39:27.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:39:27.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.730279+0000", 2024-09-13T14:39:27.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.092e-06 2024-09-13T14:39:27.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:39:27.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.730288+0000", 2024-09-13T14:39:27.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3790000000000006e-06 2024-09-13T14:39:27.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.050 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:39:27.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.730388+0000", 2024-09-13T14:39:27.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010008700000000001 2024-09-13T14:39:27.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.050 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting_for_map not empty", 2024-09-13T14:39:27.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.730392+0000", 2024-09-13T14:39:27.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6450000000000004e-06 2024-09-13T14:39:27.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.050 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:39:27.051 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.739118+0000", 2024-09-13T14:39:27.051 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0087250590000000003 2024-09-13T14:39:27.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.051 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:39:27.051 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.739232+0000", 2024-09-13T14:39:27.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114838 2024-09-13T14:39:27.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.052 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:39:27.052 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.739428+0000", 2024-09-13T14:39:27.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019531 2024-09-13T14:39:27.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.052 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:27.052 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.740693+0000", 2024-09-13T14:39:27.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012649429999999999 2024-09-13T14:39:27.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.053 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:27.053 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.742722+0000", 2024-09-13T14:39:27.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020289409999999998 2024-09-13T14:39:27.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.053 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:39:27.053 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.743520+0000", 2024-09-13T14:39:27.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00079894500000000002 2024-09-13T14:39:27.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.053 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:39:27.053 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.743549+0000", 2024-09-13T14:39:27.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8909999999999999e-05 2024-09-13T14:39:27.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:39:27.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:36:23.743577+0000", 2024-09-13T14:39:27.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7351999999999998e-05 2024-09-13T14:39:27.054 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:27.054 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:39:27.054 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:27.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.055 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(3.b 1080/1079 [PushReplyOp(3:d0630e4c:::mds0_sessionmap:head)])", 2024-09-13T14:39:27.055 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:39:11.402906+0000", 2024-09-13T14:39:27.055 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.430269444, 2024-09-13T14:39:27.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000378122, 2024-09-13T14:39:27.055 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:39:27.055 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:39:27.055 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-13T14:39:27.055 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:39:27.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:39:27.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:11.402906+0000", 2024-09-13T14:39:27.056 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:27.056 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:39:27.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:11.402906+0000", 2024-09-13T14:39:27.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:39:27.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:39:27.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:11.402906+0000", 2024-09-13T14:39:27.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9299999999999999e-07 2024-09-13T14:39:27.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:39:27.058 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:11.402909+0000", 2024-09-13T14:39:27.058 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4709999999999999e-06 2024-09-13T14:39:27.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.058 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:39:27.058 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:11.402917+0000", 2024-09-13T14:39:27.058 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2019999999999998e-06 2024-09-13T14:39:27.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.059 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:39:27.059 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:11.402925+0000", 2024-09-13T14:39:27.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3529999999999995e-06 2024-09-13T14:39:27.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.059 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:39:27.059 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:11.402999+0000", 2024-09-13T14:39:27.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3492999999999996e-05 2024-09-13T14:39:27.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:39:27.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:39:27.059 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:39:27.059 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:11.403284+0000", 2024-09-13T14:39:27.060 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028530999999999999 2024-09-13T14:39:27.060 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:27.060 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:39:27.060 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:27.060 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:39:27.060 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:39:27.060 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:39:27.060 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:27.084 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:27.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:27.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:27.186 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:27.208 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:27.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:27.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:27.217 DEBUG:teuthology.orchestra.run.smithi151:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-09-13T14:39:27.222 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T14:39:27.222 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T14:39:27.226 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:27.274 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:hints = 1 2024-09-13T14:39:27.274 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T14:39:27.274 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Object prefix: benchmark_data_smithi151_155836 2024-09-13T14:39:27.274 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:39:27.274 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 0 0 0 0 0 0 - 0 2024-09-13T14:39:27.311 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:27.347 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:27.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:27.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:27.449 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:27.486 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:27.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:27.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:27.588 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:27.628 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:27.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:27.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:27.704 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:27 smithi151 ceph-mon[104450]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:39:27.704 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:27 smithi151 ceph-mon[108966]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:39:27.732 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:27.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T14:39:27.772 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:27.775 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:27.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:27.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:27.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:27 smithi184 ceph-mon[92527]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:39:27.877 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:27.912 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:27.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:27.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:28.014 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:28.062 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:28.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:28.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:28.164 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:28.204 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:28.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:28.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:28.274 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 1 16 3922 3906 244.112 244.125 0.00332967 0.00407888 2024-09-13T14:39:28.306 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:28.342 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:28.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:28.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:28.444 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:28.482 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:28.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:28.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:28.584 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:28.622 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:28.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:28.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:28.724 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:28.760 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:28.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:28.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:28.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:28 smithi184 ceph-mon[92527]: pgmap v1293: 89 pgs: 89 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:28.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:28 smithi151 ceph-mon[104450]: pgmap v1293: 89 pgs: 89 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:28.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:28 smithi151 ceph-mon[108966]: pgmap v1293: 89 pgs: 89 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-13T14:39:28.862 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:28.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:28.906 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:28.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:29.007 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:29.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:29.038 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:29.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:29.140 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:29.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:29.176 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:29.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:29.274 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 2 16 7537 7521 235.016 225.938 0.00544583 0.0042439 2024-09-13T14:39:29.277 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:29.309 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:29.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:29.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:29.415 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:29.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:29.456 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:29.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:29.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:29.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:29.595 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:29.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:29.696 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:29.734 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:29.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:29.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:29.836 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:29.865 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:29.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:29.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:29.967 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:30.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:30.005 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:30.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:30.107 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:30.142 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:30.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:30.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:30.230 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-13T14:39:30.230 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-13T14:39:30.230 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-13T14:39:30.230 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-13T14:39:30.230 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-13T14:39:30.230 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-13T14:39:30.230 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-13T14:39:30.230 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-13T14:39:30.230 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-13T14:39:30.231 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-13T14:39:30.231 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-13T14:39:30.231 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-13T14:39:30.244 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:30.274 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 3 16 10873 10857 226.171 208.5 0.00394771 0.00441521 2024-09-13T14:39:30.279 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:30.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:30.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:30.381 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:30.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:30.417 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:30.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:30.518 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:30.566 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:30.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:30.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:30.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:39:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:39:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:39:30.669 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:30.700 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:30.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:30.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:30.802 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:30.855 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:30.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:30.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:30.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:30 smithi151 ceph-mon[104450]: pgmap v1294: 89 pgs: 89 active+clean; 64 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s wr, 126 op/s 2024-09-13T14:39:30.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:30 smithi151 ceph-mon[108966]: pgmap v1294: 89 pgs: 89 active+clean; 64 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s wr, 126 op/s 2024-09-13T14:39:30.910 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T14:39:30.910 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T14:39:30.911 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.1 'trim stale osdmaps' 2024-09-13T14:39:30.957 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:30.996 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:30.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:30.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:31.098 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:31.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:30 smithi184 ceph-mon[92527]: pgmap v1294: 89 pgs: 89 active+clean; 64 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s wr, 126 op/s 2024-09-13T14:39:31.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:31.136 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:31.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:31.237 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:31.270 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:31.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:31.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:31.275 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 4 16 13947 13931 217.655 192.125 0.00399214 0.00458762 2024-09-13T14:39:31.372 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:31.427 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:31.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:31.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:31.530 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:31.577 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:31.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:31.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:31.679 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:31.718 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:31.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:31.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:31.820 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:31.856 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:31.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:31.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:31.958 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:31.992 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:31.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:31.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:32.094 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:32.128 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:32.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:32.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:32.230 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:32.266 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:32.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:32.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:32.275 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 5 16 16922 16906 211.281 185.938 0.00453737 0.00472755 2024-09-13T14:39:32.367 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:32.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:32.405 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:32.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:32.473 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:32.506 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:32.558 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:32.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:32.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:32.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:32 smithi184 ceph-mon[92527]: pgmap v1295: 89 pgs: 89 active+clean; 113 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s wr, 223 op/s 2024-09-13T14:39:32.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:39:32.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:32.660 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:32.702 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:32.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:32.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:32.708 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:32 smithi151 ceph-mon[104450]: pgmap v1295: 89 pgs: 89 active+clean; 113 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s wr, 223 op/s 2024-09-13T14:39:32.708 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:39:32.708 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:32.708 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:32 smithi151 ceph-mon[108966]: pgmap v1295: 89 pgs: 89 active+clean; 113 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s wr, 223 op/s 2024-09-13T14:39:32.709 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:39:32.709 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:32.804 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:32.839 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:32.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:32.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:32.941 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:32.974 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:32.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:32.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:33.076 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:33.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:33.119 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:33.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:33.221 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:33.255 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:33.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:33.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:33.275 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 6 15 19569 19554 203.65 165.5 0.00508953 0.00490442 2024-09-13T14:39:33.357 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:33.390 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:33.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:33.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:33.492 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:33.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:33.515 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:33.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:33.616 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:33.628 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-13T14:39:33.656 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:33.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:33.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:33.758 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:33.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:33.798 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:33.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:33.869 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:33 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:39:33.869 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:33 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:39:33.869 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:33 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/953454581' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:39:33.869 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:33 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:39:33.869 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:33 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:39:33.869 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:33 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/953454581' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:39:33.900 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:33.935 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:33.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:33.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:34.037 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:34.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:34.075 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:34.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:34.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:33 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:39:34.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:33 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:39:34.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:33 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/953454581' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:39:34.176 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:34.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:34.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:34.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:34.243 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:39:34.243 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:39:34.243 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:39:34.244 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:39:34.244 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:39:34.244 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:39:34.244 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:39:34.244 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:39:34.244 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:39:34.244 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:39:34.244 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:39:34.244 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:39:34.245 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:39:34.245 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:39:34.245 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:39:34.245 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:39:34.275 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 7 16 22261 22245 198.582 168.188 0.005424 0.00503097 2024-09-13T14:39:34.313 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:34.352 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:34.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:34.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:34.454 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:34.488 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:34.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:34.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:34.591 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:34.622 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:34.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:34.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:34.724 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:34.755 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:34.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:34.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:34.857 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:34.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:34 smithi151 ceph-mon[104450]: pgmap v1296: 89 pgs: 89 active+clean; 885 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-13T14:39:34.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:34 smithi151 ceph-mon[104450]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:39:34.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:34 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/953454581' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:39:34.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:34 smithi151 ceph-mon[104450]: osdmap e1087: 8 total, 8 up, 8 in 2024-09-13T14:39:34.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:34 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/684176705' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:39:34.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:39:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:39:34] "GET /metrics HTTP/1.1" 200 36647 "" "Prometheus/2.51.0" 2024-09-13T14:39:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:34 smithi151 ceph-mon[108966]: pgmap v1296: 89 pgs: 89 active+clean; 885 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-13T14:39:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:34 smithi151 ceph-mon[108966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:39:34.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:34 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/953454581' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:39:34.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:34 smithi151 ceph-mon[108966]: osdmap e1087: 8 total, 8 up, 8 in 2024-09-13T14:39:34.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:34 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/684176705' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:39:34.933 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:34.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:34.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:35.035 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:35.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:35.071 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:35.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:35.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:34 smithi184 ceph-mon[92527]: pgmap v1296: 89 pgs: 89 active+clean; 885 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-13T14:39:35.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:34 smithi184 ceph-mon[92527]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:39:35.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:34 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/953454581' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:39:35.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:34 smithi184 ceph-mon[92527]: osdmap e1087: 8 total, 8 up, 8 in 2024-09-13T14:39:35.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:34 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/684176705' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:39:35.172 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:35.209 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:35.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:35.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:35.275 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 8 16 24839 24823 193.898 161.125 0.00416573 0.00515178 2024-09-13T14:39:35.311 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:35.348 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:35.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:35.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:35.450 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:35.488 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:35.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:35.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:35.590 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:35.632 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:35.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:35.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:35.734 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:35.768 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:35.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:35.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:35.869 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:35.891 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:35.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:35.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:35.993 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:36.029 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:36.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:36.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:36.131 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:36.159 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:36.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:36.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:36.261 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:36.275 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 9 16 27236 27220 188.999 149.812 0.00690547 0.00528635 2024-09-13T14:39:36.296 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:36.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:36.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:36.398 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:36.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:36.434 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:36.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:36.535 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:36.572 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:36.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:36.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:36.674 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:36.710 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:36.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:36.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:36.812 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:36.846 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:36.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:36.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:36.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:36 smithi151 ceph-mon[104450]: pgmap v1298: 89 pgs: 89 active+clean; 1.0 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-09-13T14:39:36.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:36 smithi151 ceph-mon[108966]: pgmap v1298: 89 pgs: 89 active+clean; 1.0 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-09-13T14:39:36.948 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:36.985 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:36.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:36.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:37.087 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:37.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:36 smithi184 ceph-mon[92527]: pgmap v1298: 89 pgs: 89 active+clean; 1.0 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-09-13T14:39:37.120 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:37.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:37.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:37.222 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:37.257 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:37.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:37.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:37.275 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 10 16 29523 29507 184.392 142.938 0.00432303 0.00541709 2024-09-13T14:39:37.359 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:37.395 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:37.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:37.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:37.497 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:37.532 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:37.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:37.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:37.635 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:37.668 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:37.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:37.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:37.771 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:37.812 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:37.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:37.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:37.915 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:37.949 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:37.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:37.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:38.051 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:38.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:38.089 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:38.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:38.190 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:38.227 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:38.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:38.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:38.275 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 11 16 31983 31967 181.605 153.75 0.00605086 0.00550105 2024-09-13T14:39:38.333 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:38.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:38.355 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:38.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:38.457 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:38.493 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:38.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:38.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:38.595 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:38.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:38.638 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:38.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:38.673 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:38.740 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:38.779 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:38.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:38.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:38.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:38 smithi151 ceph-mon[108966]: pgmap v1299: 89 pgs: 89 active+clean; 1.7 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 171 MiB/s wr, 2.74k op/s 2024-09-13T14:39:38.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:38 smithi151 ceph-mon[104450]: pgmap v1299: 89 pgs: 89 active+clean; 1.7 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 171 MiB/s wr, 2.74k op/s 2024-09-13T14:39:38.882 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:38.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:38.916 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:38.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:39.017 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:39.051 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:39.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:39.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:39.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:38 smithi184 ceph-mon[92527]: pgmap v1299: 89 pgs: 89 active+clean; 1.7 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 171 MiB/s wr, 2.74k op/s 2024-09-13T14:39:39.153 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:39.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:39.175 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:39.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:39.276 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 12 16 34165 34149 177.836 136.375 0.0083425 0.00561797 2024-09-13T14:39:39.276 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:39.296 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:39:39.311 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:39.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:39.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:39.413 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:39.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:39.448 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:39.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:39.549 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:39.591 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:39.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:39.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:39.694 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:39.729 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:39.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:39.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:39.830 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:39.860 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:39.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:39.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:39.965 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:40.002 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:40.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:40.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:40.103 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:40.137 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:40.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:40.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:40.239 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:40.276 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 13 16 36349 36333 174.655 136.5 0.00355304 0.00572036 2024-09-13T14:39:40.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:40.278 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:40.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:40.380 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:40.411 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:40.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:40.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:40.513 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:40.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:40.551 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:40.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:40.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:39:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:39:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:39:40.644 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:39:40.652 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:40.740 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:40.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:40.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:40.810 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-13T14:39:40.810 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-13T14:39:40.810 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-13T14:39:40.810 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-13T14:39:40.810 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-13T14:39:40.811 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-13T14:39:40.811 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-13T14:39:40.811 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-13T14:39:40.811 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-13T14:39:40.811 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-13T14:39:40.811 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-13T14:39:40.811 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-13T14:39:40.843 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:40.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:40 smithi151 ceph-mon[104450]: pgmap v1300: 89 pgs: 89 active+clean; 1.8 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 179 MiB/s wr, 2.87k op/s 2024-09-13T14:39:40.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:40 smithi151 ceph-mon[108966]: pgmap v1300: 89 pgs: 89 active+clean; 1.8 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 179 MiB/s wr, 2.87k op/s 2024-09-13T14:39:40.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:40.892 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:40.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:40.993 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:41.029 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:41.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:41.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:41.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:40 smithi184 ceph-mon[92527]: pgmap v1300: 89 pgs: 89 active+clean; 1.8 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 179 MiB/s wr, 2.87k op/s 2024-09-13T14:39:41.132 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:41.164 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:41.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:41.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:41.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:41.276 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 14 16 38460 38444 171.603 131.938 0.00535036 0.00582235 2024-09-13T14:39:41.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:41.306 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:41.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:41.407 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:41.444 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:41.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:41.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:41.545 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:41.582 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:41.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:41.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:41.684 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:41.720 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:41.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:41.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:41.821 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:41.855 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:41.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:41.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:41.957 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:41.993 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:41.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:41.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:42.095 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:42.130 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:42.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:42.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:42.232 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:42.268 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:42.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:42.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:42.276 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 15 16 40661 40645 169.333 137.562 0.0104824 0.00590117 2024-09-13T14:39:42.370 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:42.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:42.410 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:42.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:42.425 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T14:39:42.425 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T14:39:42.425 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.2 'trim stale osdmaps' 2024-09-13T14:39:42.512 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:42.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:42.553 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:42.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:42.654 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:42.692 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:42.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:42.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:42.766 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:42.794 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:42.839 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:42.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:42.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:42.911 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:42 smithi151 ceph-mon[104450]: pgmap v1301: 89 pgs: 89 active+clean; 1.9 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 179 MiB/s wr, 2.86k op/s 2024-09-13T14:39:42.911 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:42 smithi151 ceph-mon[108966]: pgmap v1301: 89 pgs: 89 active+clean; 1.9 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 179 MiB/s wr, 2.86k op/s 2024-09-13T14:39:42.941 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:42.971 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:42.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:42.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:43.029 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:43.073 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:43.083 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:39:43.084 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:39:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:42 smithi184 ceph-mon[92527]: pgmap v1301: 89 pgs: 89 active+clean; 1.9 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 179 MiB/s wr, 2.86k op/s 2024-09-13T14:39:43.119 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:43.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:43.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:43.221 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:43.255 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:43.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:43.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:43.276 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 16 16 42639 42623 166.476 123.625 0.00522044 0.00600176 2024-09-13T14:39:43.356 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:43.390 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:43.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:43.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:43.492 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:43.527 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:43.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:43.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:43.629 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:43.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:43.663 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:43.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:43.764 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:43.798 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:43.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:43.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:43.900 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:43.939 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:43.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:43.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:44.041 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:44.079 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:44.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:44.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:44.181 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:44.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:44.217 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:44.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:44.276 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 17 16 44650 44634 164.076 125.688 0.0049147 0.00608947 2024-09-13T14:39:44.319 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:44.350 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:44.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:44.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:44.452 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:44.489 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:44.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:44.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:44.591 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:44.629 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:44.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:44.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:44.731 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:44.774 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:44.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:44.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:44.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:44 smithi151 ceph-mon[104450]: pgmap v1302: 89 pgs: 89 active+clean; 2.4 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 153 MiB/s wr, 2.46k op/s 2024-09-13T14:39:44.860 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:39:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:39:44] "GET /metrics HTTP/1.1" 200 36647 "" "Prometheus/2.51.0" 2024-09-13T14:39:44.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:44 smithi151 ceph-mon[108966]: pgmap v1302: 89 pgs: 89 active+clean; 2.4 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 153 MiB/s wr, 2.46k op/s 2024-09-13T14:39:44.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:44.905 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:44 smithi184 ceph-mon[92527]: pgmap v1302: 89 pgs: 89 active+clean; 2.4 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 153 MiB/s wr, 2.46k op/s 2024-09-13T14:39:44.908 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:44.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:44.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:45.010 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:45.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:45.050 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:45.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:45.151 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:45.173 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T14:39:45.184 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:45.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:45.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:45.276 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 18 16 46553 46537 161.568 118.938 0.00687406 0.00618468 2024-09-13T14:39:45.287 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:45.323 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:45.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:45.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:45.425 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:45.467 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:45.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:45.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:45.569 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:45.613 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:45.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:45.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:45.715 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:45.755 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:45.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:45.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:45.857 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:45.910 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:45.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:45.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:46.012 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:46.051 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:46.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:46.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:46.153 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:46.193 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:46.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:46.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:46.277 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 19 16 48202 48186 158.488 103.062 0.00823381 0.00630483 2024-09-13T14:39:46.295 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:46.323 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:46.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:46.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:46.425 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:46.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:46.470 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:46.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:46.571 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:46.609 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:46.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:46.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:46.660 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-13T14:39:46.711 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:46.739 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:46.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:46.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:46.841 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:46 smithi151 ceph-mon[104450]: pgmap v1303: 89 pgs: 89 active+clean; 2.5 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-09-13T14:39:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:46 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2983590831' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:39:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:46 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:39:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:46.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:46 smithi151 ceph-mon[108966]: pgmap v1303: 89 pgs: 89 active+clean; 2.5 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-09-13T14:39:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:46 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2983590831' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:39:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:46 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:39:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:46.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:46.876 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:46.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:46.977 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:47.006 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:47.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:47.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:47.016 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:46 smithi184 ceph-mon[92527]: pgmap v1303: 89 pgs: 89 active+clean; 2.5 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-09-13T14:39:47.016 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:46 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2983590831' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:39:47.016 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:46 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:39:47.016 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:47.109 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:47.144 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:47.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:47.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:47.246 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:47.277 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:2024-09-13T14:39:47.276708+0000 min lat: 0.00105737 max lat: 0.0874668 avg lat: 0.00640815 2024-09-13T14:39:47.277 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:39:47.277 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 20 16 49924 49908 155.945 107.625 0.00866713 0.00640815 2024-09-13T14:39:47.284 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:47.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:47.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:47.386 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:47.419 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:47.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:47.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:47.521 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:47.553 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:47.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:47.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:47.655 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:47.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:47.697 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:47.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:47.798 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:47.831 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:47.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:47.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:47.933 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:47.972 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:47.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:47.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:48.074 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:48.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:47 smithi184 ceph-mon[92527]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:39:48.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:47 smithi184 ceph-mon[92527]: Cluster is now healthy 2024-09-13T14:39:48.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:47 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:39:48.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:47 smithi184 ceph-mon[92527]: osdmap e1088: 8 total, 8 up, 8 in 2024-09-13T14:39:48.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:47 smithi151 ceph-mon[108966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:39:48.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:47 smithi151 ceph-mon[108966]: Cluster is now healthy 2024-09-13T14:39:48.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:47 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:39:48.105 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:47 smithi151 ceph-mon[108966]: osdmap e1088: 8 total, 8 up, 8 in 2024-09-13T14:39:48.106 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:47 smithi151 ceph-mon[104450]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:39:48.106 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:47 smithi151 ceph-mon[104450]: Cluster is now healthy 2024-09-13T14:39:48.106 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:47 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:39:48.106 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:47 smithi151 ceph-mon[104450]: osdmap e1088: 8 total, 8 up, 8 in 2024-09-13T14:39:48.109 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:48.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:48.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:48.211 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:48.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:48.249 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:48.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:48.276 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 21 16 51688 51672 153.768 110.25 0.00833997 0.00649895 2024-09-13T14:39:48.350 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:48.390 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:48.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:48.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:48.492 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:48.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:48.543 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:48.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:48.644 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:48.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:48.680 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:48.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:48.781 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:48.813 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:48.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:48.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:48.915 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:48.956 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:48.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:48.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:49.058 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:49.095 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:49.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:49.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:49.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:48 smithi184 ceph-mon[92527]: pgmap v1305: 89 pgs: 89 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 131 MiB/s wr, 2.09k op/s 2024-09-13T14:39:49.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:48 smithi151 ceph-mon[104450]: pgmap v1305: 89 pgs: 89 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 131 MiB/s wr, 2.09k op/s 2024-09-13T14:39:49.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:48 smithi151 ceph-mon[108966]: pgmap v1305: 89 pgs: 89 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 131 MiB/s wr, 2.09k op/s 2024-09-13T14:39:49.198 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:49.232 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:49.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:49.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:49.276 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 22 16 53455 53439 151.798 110.438 0.00705383 0.00658312 2024-09-13T14:39:49.334 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:49.367 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:49.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:49.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:49.469 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:49.504 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:49.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:49.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:49.606 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:49.646 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:49.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:49.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:49.748 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:49.783 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:49.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:49.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:49.884 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:49.921 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:49.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:49.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:49.985 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:50.023 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:50.063 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:50.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:50.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:50.069 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:39:50.069 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:39:50.103 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:39:50.103 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:39:50.136 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:39:50.136 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:39:50.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:50.166 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:39:50.166 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:39:50.197 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:39:50.197 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:39:50.199 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:50.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:50.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:50.228 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:39:50.228 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:39:50.255 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:39:50.255 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:39:50.276 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 23 16 55196 55180 149.929 108.812 0.00857189 0.00666473 2024-09-13T14:39:50.284 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:39:50.284 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:39:50.301 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:50.335 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:50.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:50.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:50.437 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:50.480 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:50.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:50.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:50.582 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:50.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:39:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:39:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:39:50.617 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:50.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:50.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:50.719 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:50.754 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:50.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:50.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:50.857 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:50.881 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:50.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:50.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:50.983 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:51.015 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:51.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:51.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:51.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:50 smithi184 ceph-mon[92527]: pgmap v1306: 89 pgs: 89 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-09-13T14:39:51.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:50 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/385998682' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:39:51.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:50 smithi151 ceph-mon[104450]: pgmap v1306: 89 pgs: 89 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-09-13T14:39:51.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:50 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/385998682' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:39:51.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:50 smithi151 ceph-mon[108966]: pgmap v1306: 89 pgs: 89 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-09-13T14:39:51.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:50 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/385998682' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:39:51.118 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:51.150 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:51.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:51.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:51.253 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:51.277 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 24 16 56819 56803 147.908 101.438 0.00578167 0.00675555 2024-09-13T14:39:51.283 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:51.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:51.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:51.385 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:51.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:51.423 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:51.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:51.525 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:51.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:51.558 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:51.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:51.659 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:51.692 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:51.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:51.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:51.765 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:39:51.794 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:51.830 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:51.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:51.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:51.932 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:51.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:51.975 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:51.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:52.076 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:52.120 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:52.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:52.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:52.222 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:52.254 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:52.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:52.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:52.277 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 25 16 58451 58435 146.072 102 0.0104824 0.00684123 2024-09-13T14:39:52.356 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:52.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:52.393 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:52.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:52.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:52.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:52.526 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:52.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:52.628 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:52.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:52.667 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:52.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:52.768 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:52.803 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:52.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:52.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:52.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:52 smithi151 ceph-mon[104450]: pgmap v1307: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-09-13T14:39:52.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:52 smithi151 ceph-mon[108966]: pgmap v1307: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-09-13T14:39:52.905 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:52.939 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:52.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:52.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:53.041 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:53.081 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:53.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:53.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:53.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:52 smithi184 ceph-mon[92527]: pgmap v1307: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-09-13T14:39:53.184 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:53.214 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:53.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:53.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:53.277 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 26 16 60128 60112 144.484 104.812 0.00756415 0.00691621 2024-09-13T14:39:53.317 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:53.325 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:53.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:53.369 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:53.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:53.470 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:53.505 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:53.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:53.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:53.606 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:53.637 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:53.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:53.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:53.739 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:53.776 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:53.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:53.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:53.878 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:53.907 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:53.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:53.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:54.009 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:54.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:54.045 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:54.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:54.089 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:54.146 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:54.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:54.186 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:54.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:54.277 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 27 16 61726 61710 142.832 99.875 0.0138142 0.00699687 2024-09-13T14:39:54.287 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:54.324 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:54.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:54.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:54.426 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:54.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:54.460 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:54.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:54.561 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:54.596 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:54.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:54.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:54.608 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:39:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:39:54] "GET /metrics HTTP/1.1" 200 36655 "" "Prometheus/2.51.0" 2024-09-13T14:39:54.698 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:54.729 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:54.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:54.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:54.831 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:54.867 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:54.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:54.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:54.968 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:54.997 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:54.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:54.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:55.100 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:55.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:54 smithi184 ceph-mon[92527]: pgmap v1308: 89 pgs: 89 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-09-13T14:39:55.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:54 smithi151 ceph-mon[104450]: pgmap v1308: 89 pgs: 89 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-09-13T14:39:55.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:54 smithi151 ceph-mon[108966]: pgmap v1308: 89 pgs: 89 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-09-13T14:39:55.135 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:55.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:55.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:55.237 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:55.272 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:55.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:55.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:55.277 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 28 16 63313 63297 141.273 99.1875 0.0147203 0.00707383 2024-09-13T14:39:55.368 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:39:55.374 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:55.464 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:55.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:55.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:55.566 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:55.590 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:55.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:55.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:55.692 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:55.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:55.727 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:55.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:55.828 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:55.858 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:55.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:55.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:55.960 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:55.996 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:55.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:55.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:56.098 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:56.133 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:56.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:56.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:56.235 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:56.269 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:56.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:56.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:56.277 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 29 16 64682 64666 139.352 85.5625 0.0037162 0.00717074 2024-09-13T14:39:56.371 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:56.410 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:56.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:56.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:56.512 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:56.549 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:56.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:56.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:56.651 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:56.676 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:56.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:56.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:56.778 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:56.820 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:56.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:56.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:56.922 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:56.957 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:56.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:56.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:57.059 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:57.095 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:57.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:57.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:57.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:56 smithi184 ceph-mon[92527]: pgmap v1309: 89 pgs: 89 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-09-13T14:39:57.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:56 smithi151 ceph-mon[104450]: pgmap v1309: 89 pgs: 89 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-09-13T14:39:57.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:56 smithi151 ceph-mon[108966]: pgmap v1309: 89 pgs: 89 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-09-13T14:39:57.197 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:57.213 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:39:57.214 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-13T14:39:57.214 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-13T14:39:57.214 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-13T14:39:57.214 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:39:57.227 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:57.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:57.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:57.277 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 30 16 66094 66078 137.648 88.25 0.00692866 0.00725985 2024-09-13T14:39:57.332 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:57.367 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:57.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:57.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:57.469 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:57.504 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:57.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:57.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:57.606 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:57.643 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:57.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:57.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:57.745 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:57.775 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:57.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:57.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:57.877 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:57.908 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T14:39:57.910 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:57.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:57.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:58.012 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:58.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:58.056 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:58.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:58.157 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:58.193 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:58.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:58.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:58.277 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 31 16 67634 67618 136.312 96.25 0.0037706 0.00733105 2024-09-13T14:39:58.295 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:58.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:58.326 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:58.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:58.427 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:58.471 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:58.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:58.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:58.573 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:58.605 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:58.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:58.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:58.706 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:58.745 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:58.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:58.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:58.846 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:58.848 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:39:58 smithi151 ceph-mon[104450]: pgmap v1310: 89 pgs: 89 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 137 MiB/s wr, 2.20k op/s 2024-09-13T14:39:58.849 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:39:58 smithi151 ceph-mon[108966]: pgmap v1310: 89 pgs: 89 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 137 MiB/s wr, 2.20k op/s 2024-09-13T14:39:58.881 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:58.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:58.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:58.983 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:59.025 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:59.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:59.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:59.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:39:58 smithi184 ceph-mon[92527]: pgmap v1310: 89 pgs: 89 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 137 MiB/s wr, 2.20k op/s 2024-09-13T14:39:59.126 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:59.156 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:59.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:59.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:59.258 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:59.277 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 32 16 69105 69089 134.925 91.9375 0.0167832 0.00740634 2024-09-13T14:39:59.299 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:59.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:59.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:59.318 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-13T14:39:59.319 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-13T14:39:59.319 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-13T14:39:59.319 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-13T14:39:59.319 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-13T14:39:59.319 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-13T14:39:59.319 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-13T14:39:59.319 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-13T14:39:59.319 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-13T14:39:59.319 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-13T14:39:59.319 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-13T14:39:59.319 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-13T14:39:59.401 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:59.439 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:59.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:59.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:59.541 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:59.577 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:59.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:59.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:59.678 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:59.714 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:59.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:59.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:59.816 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:59.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:59.853 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:39:59.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:59.954 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:59.989 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:39:59.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:59.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:00.052 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T14:40:00.053 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T14:40:00.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-09-13T14:40:00.091 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:00.128 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:00.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:00.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:00.230 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:00.267 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:00.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:00.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:00.281 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 33 16 70557 70541 133.587 90.75 0.0106983 0.00748179 2024-09-13T14:40:00.368 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:00.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:00.407 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:00.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:00.508 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:00.549 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:00.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:00.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:00.605 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:40:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:40:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:40:00.651 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:00.686 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:00.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:00.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:00.788 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:00.824 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:00.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:00.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:00.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:00 smithi151 ceph-mon[104450]: pgmap v1311: 89 pgs: 89 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-13T14:40:00.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:00 smithi151 ceph-mon[104450]: overall HEALTH_OK 2024-09-13T14:40:00.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:00 smithi151 ceph-mon[108966]: pgmap v1311: 89 pgs: 89 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-13T14:40:00.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:00 smithi151 ceph-mon[108966]: overall HEALTH_OK 2024-09-13T14:40:00.926 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:00.965 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:00.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:00.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:01.067 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:01.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:00 smithi184 ceph-mon[92527]: pgmap v1311: 89 pgs: 89 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-13T14:40:01.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:00 smithi184 ceph-mon[92527]: overall HEALTH_OK 2024-09-13T14:40:01.106 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:01.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:01.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:01.208 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:01.263 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:01.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:01.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:01.278 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 34 16 71934 71918 132.189 86.0625 0.0171278 0.00756021 2024-09-13T14:40:01.343 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:01.365 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:01.391 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:01.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:01.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:01.445 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:01.494 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:01.532 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:01.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:01.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:01.634 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:01.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:01.672 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:01.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:01.682 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:01.773 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:01.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:01.810 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:01.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:01.911 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:02.011 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:02.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:02.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:02.071 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-13T14:40:02.113 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:02.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:02.167 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:02.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:02.268 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:02.278 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 35 16 73211 73195 130.692 79.8125 0.0132611 0.00764678 2024-09-13T14:40:02.301 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:02.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:02.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:02.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:40:02.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:40:02.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 2]} : dispatch 2024-09-13T14:40:02.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 0]} : dispatch 2024-09-13T14:40:02.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]} : dispatch 2024-09-13T14:40:02.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:40:02.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:02.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:02 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2869551506' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:40:02.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:40:02.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:40:02.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:40:02.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:40:02.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:40:02.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 2]} : dispatch 2024-09-13T14:40:02.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 0]} : dispatch 2024-09-13T14:40:02.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]} : dispatch 2024-09-13T14:40:02.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:40:02.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:02.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:02 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2869551506' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:40:02.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:40:02.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:40:02.361 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:40:02.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:40:02.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:40:02.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 2]} : dispatch 2024-09-13T14:40:02.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 0]} : dispatch 2024-09-13T14:40:02.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]} : dispatch 2024-09-13T14:40:02.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:40:02.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:02.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:02 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2869551506' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:40:02.362 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:40:02.363 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:40:02.363 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:40:02.403 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:02.435 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:02.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:02.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:02.537 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:02.570 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:02.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:02.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:02.671 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:02.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:02.710 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:02.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:02.811 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:02.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:02.850 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:02.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:02.951 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:02.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:02.987 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:02.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:03.088 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:03.118 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:03.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:03.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:03.220 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:03.255 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:03.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:03.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:03.278 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 36 16 74604 74588 129.48 87.0625 0.00947155 0.00771874 2024-09-13T14:40:03.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:03 smithi184 ceph-mon[92527]: pgmap v1312: 89 pgs: 89 active+clean; 4.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-13T14:40:03.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:03 smithi184 ceph-mon[92527]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:40:03.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:03 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 2]}]': finished 2024-09-13T14:40:03.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:03 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 0]}]': finished 2024-09-13T14:40:03.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:03 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-09-13T14:40:03.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:03 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2869551506' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:40:03.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:03 smithi184 ceph-mon[92527]: osdmap e1089: 8 total, 8 up, 8 in 2024-09-13T14:40:03.357 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:03.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:03 smithi151 ceph-mon[104450]: pgmap v1312: 89 pgs: 89 active+clean; 4.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-13T14:40:03.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:03 smithi151 ceph-mon[104450]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:40:03.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:03 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 2]}]': finished 2024-09-13T14:40:03.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:03 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 0]}]': finished 2024-09-13T14:40:03.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:03 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-09-13T14:40:03.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:03 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2869551506' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:40:03.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:03 smithi151 ceph-mon[104450]: osdmap e1089: 8 total, 8 up, 8 in 2024-09-13T14:40:03.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:03 smithi151 ceph-mon[108966]: pgmap v1312: 89 pgs: 89 active+clean; 4.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-13T14:40:03.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:03 smithi151 ceph-mon[108966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:40:03.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:03 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 2]}]': finished 2024-09-13T14:40:03.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:03 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 0]}]': finished 2024-09-13T14:40:03.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:03 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-09-13T14:40:03.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:03 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2869551506' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:40:03.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:03 smithi151 ceph-mon[108966]: osdmap e1089: 8 total, 8 up, 8 in 2024-09-13T14:40:03.389 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:03.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:03.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:03.491 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:03.530 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:03.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:03.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:03.633 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:03.662 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:03.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:03.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:03.764 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:03.805 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:03.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:03.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:03.908 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:03.937 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:03.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:03.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:04.039 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:04.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:04.077 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:04.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:04.179 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:04.219 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:04.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:04.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:04.278 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 37 16 75987 75971 128.316 86.4375 0.00815963 0.00778825 2024-09-13T14:40:04.322 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:04.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:04 smithi184 ceph-mon[92527]: osdmap e1090: 8 total, 8 up, 8 in 2024-09-13T14:40:04.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:04 smithi151 ceph-mon[104450]: osdmap e1090: 8 total, 8 up, 8 in 2024-09-13T14:40:04.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:04 smithi151 ceph-mon[108966]: osdmap e1090: 8 total, 8 up, 8 in 2024-09-13T14:40:04.360 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:04.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:04.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:04.462 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:04.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:04.500 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:04.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:04.601 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:04.641 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:04.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:04.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:04.718 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:40:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:40:04] "GET /metrics HTTP/1.1" 200 36658 "" "Prometheus/2.51.0" 2024-09-13T14:40:04.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:04.778 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:04.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:04.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:04.880 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:04.922 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:04.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:04.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:05.024 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:05.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:05.063 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:05.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:05.164 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:05.200 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:05.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:05.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:05.278 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 38 16 77324 77308 127.138 83.5625 0.0110878 0.00786114 2024-09-13T14:40:05.302 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:05.338 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:05.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:05.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:05.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:05 smithi184 ceph-mon[92527]: pgmap v1315: 89 pgs: 1 peering, 88 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 105 MiB/s wr, 1.69k op/s 2024-09-13T14:40:05.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:05 smithi151 ceph-mon[104450]: pgmap v1315: 89 pgs: 1 peering, 88 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 105 MiB/s wr, 1.69k op/s 2024-09-13T14:40:05.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:05 smithi151 ceph-mon[108966]: pgmap v1315: 89 pgs: 1 peering, 88 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 105 MiB/s wr, 1.69k op/s 2024-09-13T14:40:05.440 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:05.472 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:05.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:05.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:05.574 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:05.613 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:05.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:05.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:05.715 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:05.749 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:05.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:05.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:05.776 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:40:05.851 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:05.882 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:05.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:05.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:05.984 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:06.016 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:06.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:06.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:06.118 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:06.151 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:06.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:06.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:06.252 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:06.278 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 39 16 78617 78601 125.95 80.8125 0.0196901 0.00793452 2024-09-13T14:40:06.285 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:06.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:06.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:06.349 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:06 smithi151 ceph-mon[104450]: pgmap v1316: 89 pgs: 1 peering, 88 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 0 B/s, 0 objects/s recovering 2024-09-13T14:40:06.349 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:06 smithi151 ceph-mon[108966]: pgmap v1316: 89 pgs: 1 peering, 88 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 0 B/s, 0 objects/s recovering 2024-09-13T14:40:06.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:06 smithi184 ceph-mon[92527]: pgmap v1316: 89 pgs: 1 peering, 88 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 0 B/s, 0 objects/s recovering 2024-09-13T14:40:06.387 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:06.427 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:06.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:06.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:06.529 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:06.565 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:06.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:06.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:06.662 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-13T14:40:06.663 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-13T14:40:06.663 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-13T14:40:06.663 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-13T14:40:06.663 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-13T14:40:06.663 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-13T14:40:06.663 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-13T14:40:06.663 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-13T14:40:06.663 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-13T14:40:06.663 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-13T14:40:06.663 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-13T14:40:06.663 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-13T14:40:06.667 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:06.707 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:06.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:06.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:06.809 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:06.841 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:06.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:06.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:06.943 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:06.979 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:06.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:06.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:07.080 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:07.114 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:07.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:07.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:07.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:07.247 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:07.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:07.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:07.278 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:2024-09-13T14:40:07.278490+0000 min lat: 0.00105737 max lat: 0.106166 avg lat: 0.00800364 2024-09-13T14:40:07.278 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:40:07.278 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 40 16 79937 79921 124.864 82.5 0.00426618 0.00800364 2024-09-13T14:40:07.349 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:07.381 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:07.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:07.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:07.483 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:07.522 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:07.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:07.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:07.624 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:07.656 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:07.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:07.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:07.758 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:07.764 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T14:40:07.764 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T14:40:07.765 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-09-13T14:40:07.792 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:07.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:07.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:07.894 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:07.931 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:07.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:07.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:08.032 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:08.083 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:08.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:08.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:08.128 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:08.170 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:40:08.170 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:40:08.185 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:08.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:08.224 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:08.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:08.238 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:40:08.239 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:40:08.239 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:40:08.239 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:40:08.239 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:40:08.239 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:40:08.239 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:40:08.240 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:40:08.240 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:40:08.240 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:40:08.260 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:40:08.261 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:40:08.278 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 41 16 81195 81179 123.736 78.625 0.0135 0.00807711 2024-09-13T14:40:08.294 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:40:08.294 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:40:08.325 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:08.326 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:40:08.326 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:40:08.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:08.360 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:08.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:08.461 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:08.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:08.492 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:08.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:08.593 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:08.625 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:08.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:08.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:08.727 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:08.760 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:08.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:08.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:08.802 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:08 smithi151 ceph-mon[104450]: pgmap v1317: 89 pgs: 89 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 99 MiB/s wr, 1.58k op/s; 0 B/s, 0 objects/s recovering 2024-09-13T14:40:08.802 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:08 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/891724409' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:40:08.802 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:08 smithi151 ceph-mon[108966]: pgmap v1317: 89 pgs: 89 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 99 MiB/s wr, 1.58k op/s; 0 B/s, 0 objects/s recovering 2024-09-13T14:40:08.802 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:08 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/891724409' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:40:08.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:08 smithi184 ceph-mon[92527]: pgmap v1317: 89 pgs: 89 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 99 MiB/s wr, 1.58k op/s; 0 B/s, 0 objects/s recovering 2024-09-13T14:40:08.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:08 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/891724409' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:40:08.862 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:08.908 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:08.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:08.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:09.010 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:09.050 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:09.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:09.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:09.152 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:09.188 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:09.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:09.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:09.278 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 42 16 82458 82442 122.669 78.9375 0.0197567 0.00814674 2024-09-13T14:40:09.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:09.324 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:09.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:09.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:09.426 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:09.458 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:09.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:09.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:09.560 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:09.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:09.594 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:09.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:09.695 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:09.728 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:09.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:09.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:09.830 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:09.876 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:09.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:09.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:09.978 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:09.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:10.000 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:10.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:10.101 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:10.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:10.140 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:10.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:10.241 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:10.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:10.273 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:10.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:10.278 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 43 16 83647 83631 121.545 74.3125 0.00526794 0.008223 2024-09-13T14:40:10.374 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:10.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:10.405 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:10.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:10.507 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:10.543 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:10.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:10.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:10.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:10 smithi184 ceph-mon[92527]: pgmap v1318: 89 pgs: 89 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 104 MiB/s wr, 1.67k op/s; 0 B/s, 0 objects/s recovering 2024-09-13T14:40:10.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:40:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:40:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:40:10.645 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:10.674 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:10 smithi151 ceph-mon[104450]: pgmap v1318: 89 pgs: 89 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 104 MiB/s wr, 1.67k op/s; 0 B/s, 0 objects/s recovering 2024-09-13T14:40:10.674 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:10 smithi151 ceph-mon[108966]: pgmap v1318: 89 pgs: 89 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 104 MiB/s wr, 1.67k op/s; 0 B/s, 0 objects/s recovering 2024-09-13T14:40:10.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:10.676 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:10.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:10.777 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:10.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:10.815 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:10.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:10.916 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:10.947 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:10.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:10.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:11.049 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:11.082 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:11.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:11.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:11.184 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:11.219 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:11.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:11.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:11.278 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 44 16 84947 84931 120.629 81.25 0.00532079 0.00828482 2024-09-13T14:40:11.320 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:11.354 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:11.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:11.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:11.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:11.487 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:11.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:11.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:11.589 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:11.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:11.625 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:11.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:11.726 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:11.765 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:11.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:11.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:11.867 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:11.899 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:11.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:11.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:12.001 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:12.031 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:12.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:12.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:12.133 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:12.169 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:12.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:12.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:12.174 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T14:40:12.271 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:12.278 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 45 16 86221 86205 119.717 79.625 0.0157551 0.00834778 2024-09-13T14:40:12.307 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:12.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:12.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:12.385 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:12.410 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:12.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:12.466 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:12.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:12.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:12.576 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:40:12.596 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:12.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:12.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:12.667 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:12 smithi151 ceph-mon[104450]: pgmap v1319: 89 pgs: 89 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s; 0 B/s, 0 objects/s recovering 2024-09-13T14:40:12.667 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:12 smithi151 ceph-mon[108966]: pgmap v1319: 89 pgs: 89 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s; 0 B/s, 0 objects/s recovering 2024-09-13T14:40:12.699 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:12.734 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:12.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:12.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:12.836 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:12.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:12 smithi184 ceph-mon[92527]: pgmap v1319: 89 pgs: 89 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s; 0 B/s, 0 objects/s recovering 2024-09-13T14:40:12.869 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:12.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:12.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:12.971 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:13.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:13.004 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:13.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:13.105 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:13.137 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:13.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:13.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:13.239 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:13.271 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:13.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:13.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:13.278 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 46 16 87438 87422 118.768 76.0625 0.0140148 0.00841463 2024-09-13T14:40:13.373 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:13.415 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:13.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:13.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:13.517 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:13.552 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:13.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:13.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:13.655 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:13.690 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:13.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:13.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:13.792 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:13.828 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:13.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:13.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:13.930 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:13.964 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:13.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:13.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:14.066 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:14.097 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:14.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:14.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:14.199 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:14.240 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:14.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:14.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:14.279 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 47 16 88678 88662 117.89 77.5 0.00607517 0.00847818 2024-09-13T14:40:14.341 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:14.378 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:14.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:14.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:14.407 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:14.426 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-13T14:40:14.480 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:14.511 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:14.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:14.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:14.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:14 smithi151 ceph-mon[104450]: pgmap v1320: 89 pgs: 89 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 80 MiB/s wr, 1.27k op/s; 0 B/s, 0 objects/s recovering 2024-09-13T14:40:14.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:14 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2354710396' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:40:14.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:14 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:40:14.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:14 smithi151 ceph-mon[108966]: pgmap v1320: 89 pgs: 89 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 80 MiB/s wr, 1.27k op/s; 0 B/s, 0 objects/s recovering 2024-09-13T14:40:14.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:14 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2354710396' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:40:14.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:14 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:40:14.609 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:40:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:40:14] "GET /metrics HTTP/1.1" 200 36658 "" "Prometheus/2.51.0" 2024-09-13T14:40:14.613 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:14.641 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:14.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:14.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:14.655 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:14 smithi184 ceph-mon[92527]: pgmap v1320: 89 pgs: 89 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 80 MiB/s wr, 1.27k op/s; 0 B/s, 0 objects/s recovering 2024-09-13T14:40:14.655 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:14 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2354710396' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:40:14.655 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:14 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:40:14.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:14.772 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:14.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:14.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:14.874 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:14.921 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:14.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:14.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:15.023 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:15.054 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:15.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:15.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:15.132 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:15.156 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:15.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:15.190 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:15.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:15.279 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 48 16 89770 89754 116.856 68.25 0.0288824 0.00855266 2024-09-13T14:40:15.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:15.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:15.320 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:15.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:15.421 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:15.469 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:15.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:15.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:15.571 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:15.606 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:15.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:15.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:15.708 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:15.712 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:15 smithi151 ceph-mon[104450]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:40:15.712 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:15 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:40:15.712 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:15 smithi151 ceph-mon[104450]: osdmap e1091: 8 total, 8 up, 8 in 2024-09-13T14:40:15.712 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:15 smithi151 ceph-mon[108966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:40:15.713 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:15 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:40:15.713 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:15 smithi151 ceph-mon[108966]: osdmap e1091: 8 total, 8 up, 8 in 2024-09-13T14:40:15.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:15.752 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:15.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:15.853 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:15.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:15 smithi184 ceph-mon[92527]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:40:15.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:15 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:40:15.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:15 smithi184 ceph-mon[92527]: osdmap e1091: 8 total, 8 up, 8 in 2024-09-13T14:40:15.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:15.892 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:15.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:15.993 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:16.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:16.031 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:16.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:16.132 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:16.170 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:16.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:16.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:16.272 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:16.278 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 49 16 90780 90764 115.759 63.125 0.0153997 0.00863309 2024-09-13T14:40:16.300 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:16.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:16.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:16.401 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:16.445 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:16.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:16.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:16.547 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:16.584 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:16.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:16.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:16.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:16 smithi151 ceph-mon[104450]: pgmap v1322: 89 pgs: 89 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-13T14:40:16.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:40:16.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:16.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:16 smithi151 ceph-mon[108966]: pgmap v1322: 89 pgs: 89 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-13T14:40:16.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:40:16.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:16.686 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:16.714 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:16.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:16.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:16.816 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:16.851 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:16.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:16.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:16.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:16 smithi184 ceph-mon[92527]: pgmap v1322: 89 pgs: 89 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-13T14:40:16.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:40:16.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:16.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:16.981 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:16.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:16.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:17.083 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:17.124 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:17.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:17.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:17.154 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:40:17.180 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-13T14:40:17.180 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-13T14:40:17.180 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-13T14:40:17.180 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-13T14:40:17.180 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-13T14:40:17.180 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-13T14:40:17.181 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-13T14:40:17.181 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-13T14:40:17.181 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-13T14:40:17.181 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-13T14:40:17.181 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-13T14:40:17.182 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-13T14:40:17.225 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:17.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:17.260 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:17.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:17.279 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 50 16 91943 91927 114.898 72.6875 0.0140484 0.00869875 2024-09-13T14:40:17.360 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:17.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:17.407 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:17.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:17.508 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:17.554 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:17.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:17.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:17.656 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:17.699 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:17.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:17.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:17.802 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:17.841 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:17.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:17.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:17.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:40:17.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:40:17.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:40:17.943 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:17.977 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:17.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:17.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:18.079 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:18.106 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:18.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:18.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:18.208 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:18.243 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:18.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:18.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:18.279 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 51 16 93081 93065 114.039 71.125 0.010781 0.00876437 2024-09-13T14:40:18.345 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:18.378 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:18.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:18.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:18.480 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:18.510 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:18.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:18.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:18.612 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:18.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:18.647 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:18.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:18.749 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:18.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:18.789 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:18.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:18.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:18 smithi184 ceph-mon[92527]: pgmap v1323: 89 pgs: 89 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-13T14:40:18.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:18 smithi151 ceph-mon[104450]: pgmap v1323: 89 pgs: 89 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-13T14:40:18.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:18 smithi151 ceph-mon[108966]: pgmap v1323: 89 pgs: 89 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-13T14:40:18.890 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:18.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:18.917 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:18.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:19.018 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:19.056 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:19.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:19.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:19.158 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:19.193 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:19.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:19.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:19.280 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 52 16 94224 94208 113.22 71.4375 0.00728981 0.00882818 2024-09-13T14:40:19.295 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:19.322 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:19.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:19.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:19.424 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:19.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:19.462 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:19.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:19.563 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:19.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:19.597 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:19.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:19.699 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:19.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:19.729 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:19.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:19.830 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:19.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:19.861 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:19.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:19.963 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:20.003 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:20.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:20.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:20.105 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:20.138 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:20.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:20.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:20.240 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:20.275 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:20.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:20.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:20.279 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 53 16 95409 95393 112.481 74.0625 0.0178311 0.0088861 2024-09-13T14:40:20.377 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:20.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:20.407 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:20.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:20.508 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:20.539 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:40:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:40:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:40:20.551 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:20.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:20.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:20.653 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:20.687 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:20.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:20.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:20.790 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:20.815 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:20.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:20.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:20.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:20 smithi184 ceph-mon[92527]: pgmap v1324: 89 pgs: 89 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-13T14:40:20.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:20 smithi151 ceph-mon[108966]: pgmap v1324: 89 pgs: 89 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-13T14:40:20.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:20 smithi151 ceph-mon[104450]: pgmap v1324: 89 pgs: 89 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-13T14:40:20.917 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:20.957 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:20.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:20.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:21.059 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:21.091 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:21.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:21.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:21.193 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:21.223 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:21.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:21.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:21.279 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 54 16 96536 96520 111.702 70.4375 0.00948191 0.00894761 2024-09-13T14:40:21.325 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:21.362 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:21.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:21.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:21.430 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T14:40:21.431 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T14:40:21.431 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-09-13T14:40:21.464 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:21.500 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:21.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:21.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:21.602 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:21.685 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:21.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:21.686 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:21.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:21.787 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:21.829 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:21.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:21.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:21.931 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:21.966 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:21.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:21.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:22.068 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:22.109 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:22.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:22.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:22.212 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:22.244 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:22.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:22.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:22.279 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 55 16 97659 97643 110.947 70.1875 0.0108534 0.0090081 2024-09-13T14:40:22.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:22.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:22.390 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:22.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:22.491 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:22.501 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:40:22.501 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:40:22.534 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:22.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:22.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:22.636 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:22.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:22.673 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:22.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:22.774 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:22.807 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:22.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:22.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:22.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:22 smithi184 ceph-mon[92527]: pgmap v1325: 89 pgs: 89 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-13T14:40:22.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:22 smithi151 ceph-mon[104450]: pgmap v1325: 89 pgs: 89 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-13T14:40:22.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:22 smithi151 ceph-mon[108966]: pgmap v1325: 89 pgs: 89 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-13T14:40:22.911 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:22.945 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:22.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:22.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:23.046 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:23.081 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:23.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:23.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:23.183 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:23.219 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:23.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:23.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:23.279 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 56 16 98786 98770 110.224 70.4375 0.00860048 0.00906776 2024-09-13T14:40:23.321 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:23.365 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:23.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:23.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:23.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:23.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:23.498 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:23.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:23.599 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:23.637 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:23.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:23.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:23.656 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:40:23.657 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:40:23.683 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:40:23.683 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:40:23.707 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:40:23.707 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:40:23.727 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:40:23.728 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:40:23.739 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:23.764 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:40:23.764 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:40:23.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:23.773 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:23.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:23.792 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:40:23.792 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:40:23.821 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:40:23.821 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:40:23.851 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:40:23.851 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:40:23.874 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:23.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:23.904 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:23.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:24.005 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:24.033 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:24.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:24.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:24.134 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:24.167 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:24.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:24.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:24.269 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:24.279 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 57 16 99901 99885 109.513 69.6875 0.00651671 0.00912655 2024-09-13T14:40:24.304 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:24.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:24.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:24.340 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T14:40:24.406 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:24.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:24.439 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:24.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:24.540 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:24.550 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:24 smithi151 ceph-mon[104450]: pgmap v1326: 89 pgs: 89 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-13T14:40:24.550 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:24 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/567240765' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:40:24.581 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:24.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:24.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:24.625 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:24.683 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:24.711 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:24.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:24.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:24.813 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:24.831 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:24 smithi151 ceph-mon[108966]: pgmap v1326: 89 pgs: 89 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-13T14:40:24.831 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:24 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/567240765' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:40:24.832 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:40:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:40:24] "GET /metrics HTTP/1.1" 200 36663 "" "Prometheus/2.51.0" 2024-09-13T14:40:24.849 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:24.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:24.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:24.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:24 smithi184 ceph-mon[92527]: pgmap v1326: 89 pgs: 89 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-13T14:40:24.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:24 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/567240765' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:40:24.951 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:24.990 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:24.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:24.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:25.092 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:25.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:25.130 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:25.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:25.231 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:25.269 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:25.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:25.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:25.279 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 58 16 100991 100975 108.799 68.125 0.0156824 0.00918675 2024-09-13T14:40:25.371 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:25.399 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:25.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:25.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:25.501 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:25.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:25.543 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:25.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:25.644 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:25.676 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:25.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:25.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:25.778 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:25.808 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:25.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:25.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:25.910 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:25.942 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:25.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:25.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:26.044 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:26.080 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:26.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:26.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:26.182 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:26.218 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:26.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:26.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:26.280 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 59 16 102060 102044 108.087 66.8125 0.0133591 0.00924694 2024-09-13T14:40:26.319 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:26.351 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:26.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:26.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:26.453 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:26.493 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:26.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:26.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:26.595 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:26.627 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:26.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:26.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:26.729 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:26.758 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:26.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:26.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:26.764 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:26 smithi151 ceph-mon[108966]: pgmap v1327: 89 pgs: 89 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-13T14:40:26.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:26 smithi151 ceph-mon[104450]: pgmap v1327: 89 pgs: 89 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-13T14:40:26.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:26 smithi184 ceph-mon[92527]: pgmap v1327: 89 pgs: 89 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-13T14:40:26.861 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:26.892 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:26.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:26.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:26.994 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:27.038 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:27.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:27.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:27.140 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:27.178 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:27.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:27.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:27.280 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:27.281 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:2024-09-13T14:40:27.280239+0000 min lat: 0.00105737 max lat: 0.122976 avg lat: 0.00930786 2024-09-13T14:40:27.281 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:40:27.281 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 60 16 103109 103093 107.378 65.5625 0.00776921 0.00930786 2024-09-13T14:40:27.316 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:27.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:27.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:27.418 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:27.460 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:27.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:27.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:27.508 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:27.563 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:27.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:27.598 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:27.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:27.699 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:27.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:27.733 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:27.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:27.801 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-13T14:40:27.802 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-13T14:40:27.802 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-13T14:40:27.802 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-13T14:40:27.802 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-13T14:40:27.802 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-13T14:40:27.802 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-13T14:40:27.802 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-13T14:40:27.802 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-13T14:40:27.802 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-13T14:40:27.802 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-13T14:40:27.802 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-13T14:40:27.826 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:40:27.835 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:27.911 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:27.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:27.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:28.013 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:28.044 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:28.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:28.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:28.146 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:28.179 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:28.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:28.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:28.281 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:28.282 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 61 16 104167 104151 106.701 66.125 0.0160614 0.00936717 2024-09-13T14:40:28.322 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:28.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:28.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:28.424 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:28.464 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:28.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:28.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:28.566 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:28.573 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-13T14:40:28.595 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:28 smithi151 ceph-mon[108966]: pgmap v1328: 89 pgs: 89 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-13T14:40:28.595 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:28 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/440895696' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:40:28.595 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:28 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:40:28.596 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:28 smithi151 ceph-mon[104450]: pgmap v1328: 89 pgs: 89 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-13T14:40:28.596 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:28 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/440895696' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:40:28.596 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:28 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:40:28.602 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:28.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:28.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:28.702 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T14:40:28.702 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T14:40:28.703 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.6 'trim stale osdmaps' 2024-09-13T14:40:28.703 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:28.734 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:28.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:28.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:28.835 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:28.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:28 smithi184 ceph-mon[92527]: pgmap v1328: 89 pgs: 89 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-13T14:40:28.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:28 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/440895696' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:40:28.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:28 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:40:28.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:28.869 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:28.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:28.970 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:29.011 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:29.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:29.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:29.113 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:29.146 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:29.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:29.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:29.248 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:29.281 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 62 16 105249 105233 106.07 67.625 0.0260034 0.0094228 2024-09-13T14:40:29.285 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:29.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:29.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:29.387 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:29.428 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:29.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:29.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:29.534 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:29.564 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:29.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:29.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:29.666 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:29.703 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:29.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:29.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:29.805 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:29.830 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:29.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:29.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:29.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:29 smithi184 ceph-mon[92527]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:40:29.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:29 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:40:29.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:29 smithi184 ceph-mon[92527]: osdmap e1092: 8 total, 8 up, 8 in 2024-09-13T14:40:29.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:29 smithi151 ceph-mon[104450]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:40:29.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:29 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:40:29.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:29 smithi151 ceph-mon[104450]: osdmap e1092: 8 total, 8 up, 8 in 2024-09-13T14:40:29.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:29 smithi151 ceph-mon[108966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:40:29.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:29 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:40:29.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:29 smithi151 ceph-mon[108966]: osdmap e1092: 8 total, 8 up, 8 in 2024-09-13T14:40:29.932 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:29.964 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:29.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:29.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:30.062 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:30.066 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:30.114 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:30.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:30.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:30.215 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:30.252 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:30.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:30.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:30.281 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 63 16 106290 106274 105.419 65.0625 0.0122394 0.00948137 2024-09-13T14:40:30.354 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:30.385 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:30.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:30.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:30.487 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:30.530 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:30.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:30.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:30.572 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:40:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:40:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:40:30.632 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:30.666 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:30.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:30.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:30.768 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:30.808 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:30.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:30.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:30.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:30 smithi184 ceph-mon[92527]: pgmap v1330: 89 pgs: 89 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T14:40:30.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:30 smithi151 ceph-mon[104450]: pgmap v1330: 89 pgs: 89 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T14:40:30.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:30 smithi151 ceph-mon[108966]: pgmap v1330: 89 pgs: 89 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T14:40:30.910 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:30.945 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:30.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:30.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:31.047 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:31.088 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:31.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:31.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:31.190 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:31.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:31.227 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:31.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:31.281 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 64 16 107326 107310 104.784 64.75 0.0279147 0.00953831 2024-09-13T14:40:31.329 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:31.364 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:31.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:31.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:31.466 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:31.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:31.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:31.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:31.595 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:31.627 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:31.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:31.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:31.681 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:31.730 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:31.771 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:31.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:31.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:31.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:31.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:31.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:31.874 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:31.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:31.909 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:31.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:31.949 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:40:31.949 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-13T14:40:31.950 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-13T14:40:31.950 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-13T14:40:31.950 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:40:32.010 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:32.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:32.040 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:32.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:32.046 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:40:32.141 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:32.181 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:32.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:32.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:32.281 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 65 16 108388 108372 104.193 66.375 0.019698 0.00959291 2024-09-13T14:40:32.283 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:32.315 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:32.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:32.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:32.418 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:32.451 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:32.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:32.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:32.553 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:32.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:32.600 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:32.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:32.701 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:32.736 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:32.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:32.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:32.837 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:32.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:32 smithi184 ceph-mon[92527]: pgmap v1331: 89 pgs: 89 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T14:40:32.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:32 smithi151 ceph-mon[104450]: pgmap v1331: 89 pgs: 89 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T14:40:32.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:32 smithi151 ceph-mon[108966]: pgmap v1331: 89 pgs: 89 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T14:40:32.870 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:32.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:32.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:32.972 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:33.006 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:33.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:33.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:33.108 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:33.159 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:33.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:33.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:33.261 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:33.281 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 66 16 109417 109401 103.588 64.3125 0.0240745 0.00964878 2024-09-13T14:40:33.304 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:33.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:33.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:33.407 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:33.442 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:33.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:33.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:33.544 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:33.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:33.577 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:33.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:33.678 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:33.706 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:33.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:33.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:33.809 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:33.855 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:33.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:33.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:33.957 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:34.001 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:34.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:34.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:34.103 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:34.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:34.139 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:34.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:34.240 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:34.281 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:34.281 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 67 16 110429 110413 102.986 63.25 0.0193645 0.00970461 2024-09-13T14:40:34.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:34.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:34.382 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:34.414 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:34.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:34.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:34.516 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:34.559 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:34.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:34.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:34.662 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:34.706 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:34.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:34.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:34.717 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:34.754 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T14:40:34.808 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:34.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:34 smithi151 ceph-mon[104450]: pgmap v1332: 89 pgs: 89 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-13T14:40:34.841 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:34 smithi151 ceph-mon[108966]: pgmap v1332: 89 pgs: 89 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-13T14:40:34.841 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:40:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:40:34] "GET /metrics HTTP/1.1" 200 36661 "" "Prometheus/2.51.0" 2024-09-13T14:40:34.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:34.845 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:34.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:34.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:34 smithi184 ceph-mon[92527]: pgmap v1332: 89 pgs: 89 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-13T14:40:34.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:35.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:35.017 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:35.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:35.118 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:35.161 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:35.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:35.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:35.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:35.282 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 68 16 111401 111385 102.364 60.75 0.0200001 0.00976383 2024-09-13T14:40:35.307 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:35.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:35.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:35.409 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:35.414 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-13T14:40:35.414 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-13T14:40:35.414 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-13T14:40:35.414 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-13T14:40:35.414 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-13T14:40:35.414 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-13T14:40:35.414 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-13T14:40:35.414 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-13T14:40:35.415 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-13T14:40:35.415 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-13T14:40:35.415 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-13T14:40:35.415 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-13T14:40:35.445 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:35.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:35.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:35.546 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:35.584 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:35.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:35.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:35.686 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:35.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:35.722 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:35.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:35.823 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:35.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:35.857 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:35.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:35.958 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:36.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:36.000 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:36.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:36.101 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:36.134 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:36.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:36.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:36.237 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:36.275 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:36.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:36.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:36.282 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 69 16 112371 112355 101.759 60.625 0.0224845 0.00982212 2024-09-13T14:40:36.377 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:36.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:36.412 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:36.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:36.513 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:36.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:36.554 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:36.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:36.656 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:36.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:36.693 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:36.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:36.794 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:36.824 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:36.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:36.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:36.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:36 smithi184 ceph-mon[92527]: pgmap v1333: 89 pgs: 89 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-13T14:40:36.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:36 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2695391834' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:40:36.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:36 smithi151 ceph-mon[104450]: pgmap v1333: 89 pgs: 89 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-13T14:40:36.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:36 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2695391834' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:40:36.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:36 smithi151 ceph-mon[108966]: pgmap v1333: 89 pgs: 89 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-13T14:40:36.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:36 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2695391834' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:40:36.912 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:40:36.913 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:40:36.913 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:40:36.913 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:40:36.913 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:40:36.914 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:40:36.914 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:40:36.914 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:40:36.914 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:40:36.914 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:40:36.915 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:40:36.915 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:40:36.915 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:40:36.915 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:40:36.915 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:40:36.915 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:40:36.926 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:36.954 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:36.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:36.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:37.056 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:37.100 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:37.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:37.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:37.202 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:37.236 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:37.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:37.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:37.282 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 70 16 113387 113371 101.212 63.5 0.0142054 0.00987571 2024-09-13T14:40:37.322 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T14:40:37.323 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T14:40:37.323 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-09-13T14:40:37.338 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:37.391 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:37.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:37.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:37.493 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:37.530 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:37.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:37.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:37.632 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:37.671 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:37.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:37.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:37.773 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:37.808 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:37.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:37.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:37.910 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:37.946 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:37.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:37.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:38.048 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:38.085 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:38.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:38.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:38.187 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:38.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:38.230 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:38.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:38.282 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 71 16 114386 114370 100.666 62.4375 0.0104184 0.00992935 2024-09-13T14:40:38.330 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:38.331 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:38.377 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:38.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:38.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:38.380 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:38.478 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:38.516 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:38.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:38.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:38.618 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:38.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:38.659 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:38.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:38.760 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:38.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:38.793 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:38.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:38.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:38 smithi184 ceph-mon[92527]: pgmap v1334: 89 pgs: 89 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-13T14:40:38.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:38 smithi151 ceph-mon[104450]: pgmap v1334: 89 pgs: 89 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-13T14:40:38.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:38 smithi151 ceph-mon[108966]: pgmap v1334: 89 pgs: 89 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-13T14:40:38.893 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:38.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:38.930 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:38.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:39.031 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:39.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:39.067 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:39.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:39.168 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:39.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:39.204 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:39.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:39.282 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 72 16 115382 115366 100.132 62.25 0.0175495 0.0099815 2024-09-13T14:40:39.305 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:39.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:39.338 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:39.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:39.439 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:39.485 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:39.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:39.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:39.587 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:39.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:39.626 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:39.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:39.727 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:39.763 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:39.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:39.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:39.826 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:40:39.865 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:39.899 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:39.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:39.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:40.000 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:40.034 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:40.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:40.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:40.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:40.166 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:40.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:40.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:40.268 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:40.282 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 73 16 116373 116357 99.6092 61.9375 0.0167465 0.0100345 2024-09-13T14:40:40.310 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:40.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:40.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:40.412 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:40.441 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:40.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:40.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:40.544 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:40.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:40.584 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:40.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:40.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:40:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:40:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:40:40.614 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-13T14:40:40.685 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:40.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:40.723 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:40.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:40.824 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:40.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:40.859 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:40.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:40.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:40 smithi151 ceph-mon[104450]: pgmap v1335: 89 pgs: 89 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 65 MiB/s wr, 1.05k op/s 2024-09-13T14:40:40.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:40 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2687314189' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:40:40.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:40 smithi151 ceph-mon[108966]: pgmap v1335: 89 pgs: 89 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 65 MiB/s wr, 1.05k op/s 2024-09-13T14:40:40.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:40 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2687314189' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:40:40.960 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:40.989 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:40.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:40.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:41.091 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:41.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:40 smithi184 ceph-mon[92527]: pgmap v1335: 89 pgs: 89 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 65 MiB/s wr, 1.05k op/s 2024-09-13T14:40:41.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:40 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2687314189' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:40:41.143 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:41.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:41.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:41.245 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:41.277 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:41.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:41.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:41.282 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 74 16 117353 117337 99.0908 61.25 0.0184095 0.0100869 2024-09-13T14:40:41.380 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:41.417 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:41.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:41.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:41.519 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:41.550 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:41.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:41.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:41.652 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:41.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:41.689 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:41.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:41.790 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:41.818 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:41.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:41.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:41.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:41 smithi151 ceph-mon[104450]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:40:41.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:41 smithi151 ceph-mon[104450]: Cluster is now healthy 2024-09-13T14:40:41.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:41 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2687314189' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:40:41.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:41 smithi151 ceph-mon[104450]: osdmap e1093: 8 total, 8 up, 8 in 2024-09-13T14:40:41.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:41 smithi151 ceph-mon[108966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:40:41.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:41 smithi151 ceph-mon[108966]: Cluster is now healthy 2024-09-13T14:40:41.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:41 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2687314189' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:40:41.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:41 smithi151 ceph-mon[108966]: osdmap e1093: 8 total, 8 up, 8 in 2024-09-13T14:40:41.921 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:41.964 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:41.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:41.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:42.066 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:42.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:42.101 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:42.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:42.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:41 smithi184 ceph-mon[92527]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:40:42.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:41 smithi184 ceph-mon[92527]: Cluster is now healthy 2024-09-13T14:40:42.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:41 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2687314189' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:40:42.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:41 smithi184 ceph-mon[92527]: osdmap e1093: 8 total, 8 up, 8 in 2024-09-13T14:40:42.202 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:42.237 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:42.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:42.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:42.283 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 75 16 118333 118317 98.5862 61.25 0.0205389 0.010138 2024-09-13T14:40:42.339 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:42.374 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:42.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:42.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:42.476 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:42.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:42.513 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:42.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:42.614 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:42.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:42.649 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:42.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:42.690 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:40:42.690 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:40:42.750 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:42.789 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:42.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:42.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:42.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:42 smithi151 ceph-mon[104450]: pgmap v1337: 89 pgs: 89 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-13T14:40:42.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:42 smithi151 ceph-mon[108966]: pgmap v1337: 89 pgs: 89 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-13T14:40:42.891 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:42.924 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:42.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:42.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:43.029 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:43.065 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:43.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:43.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:42 smithi184 ceph-mon[92527]: pgmap v1337: 89 pgs: 89 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-13T14:40:43.168 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:43.194 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:43.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:43.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:43.283 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 76 16 119316 119300 98.0974 61.4375 0.0124214 0.0101889 2024-09-13T14:40:43.296 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:43.328 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:43.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:43.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:43.416 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T14:40:43.430 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:43.495 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:43.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:43.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:43.597 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:43.630 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:43.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:43.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:43.732 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:43.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:43.773 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:43.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:43.874 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:43.902 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:43.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:43.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:44.004 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:44.029 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:44.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:44.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:44.131 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:44.174 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:44.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:44.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:44.276 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:44.283 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 77 16 120289 120273 97.6131 60.8125 0.0183441 0.0102402 2024-09-13T14:40:44.302 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:44.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:44.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:44.387 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:44.404 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:44.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:44.464 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:44.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:44.565 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:44.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:44.596 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:44.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:44.608 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:40:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:40:44] "GET /metrics HTTP/1.1" 200 36661 "" "Prometheus/2.51.0" 2024-09-13T14:40:44.697 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:44.731 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:44.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:44.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:44.785 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:44.833 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:44.868 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:44.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:44.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:44.905 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:44 smithi184 ceph-mon[92527]: pgmap v1338: 89 pgs: 89 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-13T14:40:44.913 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:44 smithi151 ceph-mon[104450]: pgmap v1338: 89 pgs: 89 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-13T14:40:44.913 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:44 smithi151 ceph-mon[108966]: pgmap v1338: 89 pgs: 89 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-13T14:40:44.971 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:45.000 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:45.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:45.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:45.102 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:45.141 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:45.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:45.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:45.243 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:45.283 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 78 16 121263 121247 97.142 60.875 0.00885317 0.0102893 2024-09-13T14:40:45.286 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:45.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:45.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:45.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:45.417 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:45.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:45.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:45.519 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:45.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:45.552 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:45.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:45.654 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:45.684 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:45.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:45.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:45.786 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:45.817 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:45.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:45.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:45.863 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:40:45.919 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:45.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:45.966 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:45.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:46.067 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:46.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:46.101 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:46.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:46.202 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:46.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:46.238 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:46.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:46.283 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 79 16 122208 122192 96.66 59.0625 0.0223769 0.0103412 2024-09-13T14:40:46.339 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:46.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:46.379 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:46.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:46.480 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:46.523 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:46.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:46.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:46.625 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:46.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:46.654 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:46.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:46.755 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:46.786 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:46.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:46.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:46.888 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:46.915 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:46.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:46.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:47.017 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:47.023 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:46 smithi151 ceph-mon[108966]: pgmap v1339: 89 pgs: 89 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 62 MiB/s wr, 996 op/s 2024-09-13T14:40:47.023 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:40:47.023 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:47.024 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:46 smithi151 ceph-mon[104450]: pgmap v1339: 89 pgs: 89 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 62 MiB/s wr, 996 op/s 2024-09-13T14:40:47.024 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:40:47.024 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:47.046 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:47.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:47.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:47.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:46 smithi184 ceph-mon[92527]: pgmap v1339: 89 pgs: 89 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 62 MiB/s wr, 996 op/s 2024-09-13T14:40:47.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:40:47.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:47.148 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:47.174 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:47.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:47.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:47.276 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:47.283 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:2024-09-13T14:40:47.283643+0000 min lat: 0.00105737 max lat: 0.122976 avg lat: 0.0103914 2024-09-13T14:40:47.283 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:40:47.283 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 80 15 123151 123136 96.1889 59 0.00922539 0.0103914 2024-09-13T14:40:47.306 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:47.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:47.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:47.408 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:47.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:47.441 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:47.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:47.542 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:47.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:47.579 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:47.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:47.680 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:47.721 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:47.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:47.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:47.823 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:47.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:47.855 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:47.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:47.957 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:47.989 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:47.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:47.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:48.091 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:48.116 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:48.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:48.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:48.218 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:48.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:48.254 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:48.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:48.283 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 81 16 124086 124070 95.722 58.375 0.00829633 0.0104421 2024-09-13T14:40:48.355 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:48.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:48.396 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:48.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:48.497 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:48.543 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:48.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:48.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:48.645 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:48.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:48.684 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:48.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:48.780 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-13T14:40:48.781 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-13T14:40:48.781 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-13T14:40:48.781 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-13T14:40:48.781 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-13T14:40:48.781 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-13T14:40:48.781 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-13T14:40:48.781 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-13T14:40:48.781 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-13T14:40:48.781 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-13T14:40:48.781 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-13T14:40:48.781 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-13T14:40:48.785 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:48.819 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:48.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:48.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:48.921 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:48.951 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:48.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:48.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:49.053 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:49.089 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:49.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:49.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:49.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:48 smithi184 ceph-mon[92527]: pgmap v1340: 89 pgs: 89 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 61 MiB/s wr, 982 op/s 2024-09-13T14:40:49.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:48 smithi151 ceph-mon[104450]: pgmap v1340: 89 pgs: 89 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 61 MiB/s wr, 982 op/s 2024-09-13T14:40:49.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:48 smithi151 ceph-mon[108966]: pgmap v1340: 89 pgs: 89 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 61 MiB/s wr, 982 op/s 2024-09-13T14:40:49.192 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:49.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:49.226 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:49.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:49.283 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 82 16 125034 125018 95.2772 59.25 0.0208002 0.0104912 2024-09-13T14:40:49.327 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:49.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:49.359 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:49.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:49.460 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:49.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:49.495 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:49.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:49.596 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:49.636 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:49.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:49.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:49.738 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:49.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:49.773 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:49.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:49.876 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:49.906 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:49.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:49.916 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:49.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:50.017 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:50.049 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:50.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:50.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:50.151 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:50.188 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:50.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:50.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:50.284 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 83 16 125981 125965 94.8423 59.1875 0.00580772 0.0105385 2024-09-13T14:40:50.290 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:50.311 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:50.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:50.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:50.412 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:50.448 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:50.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:50.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:50.549 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:50.594 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:50.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:50.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:50.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:40:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:40:50.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:40:50.613 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T14:40:50.613 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T14:40:50.695 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:50.725 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:50.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:50.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:50.827 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:50.847 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:50 smithi151 ceph-mon[104450]: pgmap v1341: 89 pgs: 89 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 61 MiB/s wr, 978 op/s 2024-09-13T14:40:50.847 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:50 smithi151 ceph-mon[108966]: pgmap v1341: 89 pgs: 89 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 61 MiB/s wr, 978 op/s 2024-09-13T14:40:50.854 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:50.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:50.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:50.958 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:50.989 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:50.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:50.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:50.999 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:40:50.999 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:40:51.091 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:51.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:50 smithi184 ceph-mon[92527]: pgmap v1341: 89 pgs: 89 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 61 MiB/s wr, 978 op/s 2024-09-13T14:40:51.129 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:51.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:51.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:51.232 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:51.269 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:51.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:51.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:51.283 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 84 16 126885 126869 94.3858 56.5 0.0171153 0.0105899 2024-09-13T14:40:51.372 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:51.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:51.410 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:51.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:51.512 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:51.545 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:51.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:51.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:51.647 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:51.682 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:51.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:51.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:51.783 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:51.826 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:51.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:51.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:51.928 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:51.955 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:51.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:51.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:52.057 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:52.084 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:52.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:52.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:52.187 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:52.220 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:52.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:52.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:52.284 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 85 16 127837 127821 93.9754 59.5 0.016741 0.0106364 2024-09-13T14:40:52.323 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:52.354 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:52.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:52.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:52.457 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:52.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:52.491 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:52.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:52.593 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:52.623 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:52.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:52.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:52.725 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:52.755 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:52.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:52.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:52.851 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:52 smithi151 ceph-mon[104450]: pgmap v1342: 89 pgs: 89 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 59 MiB/s wr, 938 op/s 2024-09-13T14:40:52.851 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:52 smithi151 ceph-mon[108966]: pgmap v1342: 89 pgs: 89 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 59 MiB/s wr, 938 op/s 2024-09-13T14:40:52.857 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:52.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:52.890 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:52.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:52.991 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:53.015 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:53.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:53.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:53.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:52 smithi184 ceph-mon[92527]: pgmap v1342: 89 pgs: 89 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 59 MiB/s wr, 938 op/s 2024-09-13T14:40:53.117 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:53.146 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:53.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:53.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:53.215 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T14:40:53.248 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:53.285 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 86 16 128782 128766 93.5685 59.0625 0.015125 0.0106819 2024-09-13T14:40:53.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:53.293 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:53.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:53.395 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:53.438 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:53.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:53.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:53.540 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:53.582 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:53.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:53.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:53.684 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:53.715 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:53.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:53.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:53.817 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:53.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:53.856 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:53.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:53.917 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:40:53.917 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:40:53.948 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:40:53.949 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:40:53.957 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:53.975 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:40:53.975 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:40:53.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:53.994 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:53.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:53.997 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:40:53.998 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:40:54.029 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:40:54.030 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:40:54.074 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:40:54.075 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:40:54.095 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:54.103 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:40:54.103 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:40:54.127 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:54.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:54.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:54.134 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:40:54.134 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:40:54.229 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:54.267 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:54.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:54.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:54.284 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 87 16 129714 129698 93.1625 58.25 0.0216667 0.0107286 2024-09-13T14:40:54.369 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:54.400 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:54.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:54.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:54.503 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:54.544 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:54.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:54.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:54.646 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:54.681 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:54.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:54.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:54.783 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:54.818 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:54.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:54.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:54 smithi151 ceph-mon[104450]: pgmap v1343: 89 pgs: 89 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-13T14:40:54.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:54 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1115910966' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:40:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:54 smithi151 ceph-mon[108966]: pgmap v1343: 89 pgs: 89 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-13T14:40:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:54 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1115910966' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:40:54.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:40:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:40:54] "GET /metrics HTTP/1.1" 200 36659 "" "Prometheus/2.51.0" 2024-09-13T14:40:54.920 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:54.962 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:54.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:54.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:55.064 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:55.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:55.102 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:55.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:55.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:54 smithi184 ceph-mon[92527]: pgmap v1343: 89 pgs: 89 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-13T14:40:55.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:54 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1115910966' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:40:55.119 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:55.204 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:55.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:55.244 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:55.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:55.285 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 88 16 130622 130606 92.7487 56.75 0.00847688 0.0107768 2024-09-13T14:40:55.345 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:55.385 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:55.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:55.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:55.486 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:55.524 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:55.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:55.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:55.614 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:40:55.614 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:40:55.615 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-13T14:40:55.615 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:40:55.626 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:55.661 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-13T14:40:55.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:55.671 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:55.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:55.772 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:55.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:55.805 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:55.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:55.906 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:55.944 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:55.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:55.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:56.046 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:56.075 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:56.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:56.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:56.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:55 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1929126532' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:40:56.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:55 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1929126532' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:40:56.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:55 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1929126532' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:40:56.177 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:56.212 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:56.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:56.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:56.285 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 89 16 131544 131528 92.354 57.625 0.00861662 0.010823 2024-09-13T14:40:56.314 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:56.352 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:56.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:56.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:56.454 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:56.493 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:56.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:56.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:56.595 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:56.626 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:56.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:56.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:56.728 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:56.758 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:56.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:56.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:56.835 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:56 smithi151 ceph-mon[104450]: pgmap v1344: 89 pgs: 89 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 54 MiB/s wr, 869 op/s 2024-09-13T14:40:56.836 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:56 smithi151 ceph-mon[104450]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:40:56.836 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:56 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1929126532' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:40:56.836 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:56 smithi151 ceph-mon[104450]: osdmap e1094: 8 total, 8 up, 8 in 2024-09-13T14:40:56.836 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:56 smithi151 ceph-mon[108966]: pgmap v1344: 89 pgs: 89 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 54 MiB/s wr, 869 op/s 2024-09-13T14:40:56.836 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:56 smithi151 ceph-mon[108966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:40:56.836 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:56 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1929126532' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:40:56.836 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:56 smithi151 ceph-mon[108966]: osdmap e1094: 8 total, 8 up, 8 in 2024-09-13T14:40:56.860 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:56.890 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:56.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:56.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:56.992 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:57.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:57.036 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:57.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:57.100 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:40:57.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:56 smithi184 ceph-mon[92527]: pgmap v1344: 89 pgs: 89 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 54 MiB/s wr, 869 op/s 2024-09-13T14:40:57.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:56 smithi184 ceph-mon[92527]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:40:57.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:56 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1929126532' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:40:57.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:56 smithi184 ceph-mon[92527]: osdmap e1094: 8 total, 8 up, 8 in 2024-09-13T14:40:57.137 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:57.173 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:57.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:57.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:57.275 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:57.285 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Total time run: 90.0087 2024-09-13T14:40:57.285 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Total writes made: 132455 2024-09-13T14:40:57.285 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Write size: 65536 2024-09-13T14:40:57.285 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Object size: 65536 2024-09-13T14:40:57.285 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Bandwidth (MB/sec): 91.9738 2024-09-13T14:40:57.285 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Stddev Bandwidth: 40.9896 2024-09-13T14:40:57.285 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Max bandwidth (MB/sec): 244.125 2024-09-13T14:40:57.286 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Min bandwidth (MB/sec): 56.5 2024-09-13T14:40:57.286 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Average IOPS: 1471 2024-09-13T14:40:57.286 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Stddev IOPS: 655.833 2024-09-13T14:40:57.286 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Max IOPS: 3906 2024-09-13T14:40:57.286 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Min IOPS: 904 2024-09-13T14:40:57.286 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Average Latency(s): 0.0108686 2024-09-13T14:40:57.286 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Stddev Latency(s): 0.00698198 2024-09-13T14:40:57.286 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Max latency(s): 0.122976 2024-09-13T14:40:57.286 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Min latency(s): 0.00105737 2024-09-13T14:40:57.290 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T14:40:57.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:57.304 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:57.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:57.405 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:57.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:57.440 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:57.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:57.541 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:57.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:57.572 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:57.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:57.673 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:57.710 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:57.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:57.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:57.812 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:57.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:57.855 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:57.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:57.956 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:57.958 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:40:57.958 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-13T14:40:57.958 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-13T14:40:57.958 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-13T14:40:57.958 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:40:57.992 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:57.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:57.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:58.094 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:58.128 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:58.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:58.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:58.230 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:58.257 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:58.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:58.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:58.359 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:58.393 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:58.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:58.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:58.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:58.529 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:58.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:58.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:58.631 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:58.669 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T14:40:58.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:58.673 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:58.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:58.719 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:58.774 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:58.813 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:58.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:58.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:58.878 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:40:58 smithi151 ceph-mon[104450]: pgmap v1346: 89 pgs: 89 active+clean; 8.0 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 102 B/s rd, 58 MiB/s wr, 931 op/s 2024-09-13T14:40:58.879 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:40:58 smithi151 ceph-mon[108966]: pgmap v1346: 89 pgs: 89 active+clean; 8.0 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 102 B/s rd, 58 MiB/s wr, 931 op/s 2024-09-13T14:40:58.914 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:58.955 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:58.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:58.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:59.057 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:59.095 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:59.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:59.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:59.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:40:58 smithi184 ceph-mon[92527]: pgmap v1346: 89 pgs: 89 active+clean; 8.0 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 102 B/s rd, 58 MiB/s wr, 931 op/s 2024-09-13T14:40:59.186 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:40:59.197 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:59.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:59.277 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:40:59.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:59.378 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:59.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:59.417 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:59.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:59.519 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:59.546 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:59.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:59.555 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:59.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:59.656 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:59.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:59.704 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:59.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:59.805 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:59.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:59.842 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:59.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:59.943 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:59.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:59.986 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:40:59.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:00.062 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:00.087 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:00.146 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:00.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:00.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:00.248 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:00.275 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:00.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:00.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:00.377 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:00.410 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:00.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:00.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:00.512 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:00.544 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:00.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:00.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:00.589 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:00.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:41:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:41:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:41:00.646 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:00.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:00.685 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:00.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:00.786 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:00.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:00.824 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:00.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:00.925 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:00.962 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:00 smithi151 ceph-mon[104450]: pgmap v1347: 89 pgs: 89 active+clean; 8.1 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 102 B/s rd, 56 MiB/s wr, 930 op/s 2024-09-13T14:41:00.962 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:00 smithi151 ceph-mon[108966]: pgmap v1347: 89 pgs: 89 active+clean; 8.1 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 102 B/s rd, 56 MiB/s wr, 930 op/s 2024-09-13T14:41:00.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:00.965 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:00.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:01.066 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:01.103 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:01.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:00 smithi184 ceph-mon[92527]: pgmap v1347: 89 pgs: 89 active+clean; 8.1 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 102 B/s rd, 56 MiB/s wr, 930 op/s 2024-09-13T14:41:01.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:01.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:01.146 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:41:01.148 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T14:41:01.205 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:01.245 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:01.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:01.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:01.347 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:01.389 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:01.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:01.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:01.491 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:01.526 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:01.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:01.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:01.628 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:01.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:01.661 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:01.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:01.762 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:01.803 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:01.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:01.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:01.905 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:01.934 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:01.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:01.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:02.036 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:02.064 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:02.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:02.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:02.166 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:02.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:02.201 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:02.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:02.302 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:02.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:02.348 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:02.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:02.401 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:02 smithi151 ceph-mon[108966]: from='client.35394 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:02.401 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:02 smithi151 ceph-mon[108966]: pgmap v1348: 89 pgs: 89 active+clean; 8.1 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 102 B/s rd, 55 MiB/s wr, 932 op/s 2024-09-13T14:41:02.401 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:41:02.401 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:02.401 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:41:02.402 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:02 smithi151 ceph-mon[104450]: from='client.35394 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:02.402 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:02 smithi151 ceph-mon[104450]: pgmap v1348: 89 pgs: 89 active+clean; 8.1 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 102 B/s rd, 55 MiB/s wr, 932 op/s 2024-09-13T14:41:02.402 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:41:02.402 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:02.402 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:41:02.449 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:02.482 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:02.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:02.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:02.584 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:02.611 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:02.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:02.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:02.713 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:02.741 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:02.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:02.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:02.842 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:02.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:02 smithi184 ceph-mon[92527]: from='client.35394 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:02.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:02 smithi184 ceph-mon[92527]: pgmap v1348: 89 pgs: 89 active+clean; 8.1 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 102 B/s rd, 55 MiB/s wr, 932 op/s 2024-09-13T14:41:02.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:41:02.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:02.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:41:02.880 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:02.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:02.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:02.982 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:03.037 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:03.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:03.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:03.139 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:03.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:03.177 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:03.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:03.278 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:03.312 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:03.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:03.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:03.414 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:03.445 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:03.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:03.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:03.546 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:03.571 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:03.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:03.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:03.674 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:03.701 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:03.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:03.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:03.803 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:03.848 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:03.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:03.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:03.950 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:03.959 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1347,"stamp":"2024-09-13T14:40:59.336019+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8655345637,"num_objects":132088,"num_object_clones":0,"num_object_copies":396264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132088,"num_whiteouts":0,"num_read":307,"num_read_kb":760,"num_write":132804,"num_write_kb":8470524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126450,"ondisk_log_size":126450,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":26965788,"kb_used_data":25402704,"kb_used_omap":149,"kb_used_meta":1562858,"kb_avail":722995428,"statfs":{"total":767960285184,"available":740347318272,"internally_reserved":0,"allocated":26012368896,"data_stored":25986934130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":153401,"internal_metadata":1600366791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,2,9,7,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":574947352,"num_objects":8774,"num_object_clones":0,"num_object_copies":26322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8774,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9304,"num_write_kb":578433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002996"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":1492,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:55.670057+0000","last_change":"2024-09-13T14:30:15.579251+0000","last_active":"2024-09-13T14:40:55.670057+0000","last_peered":"2024-09-13T14:40:55.670057+0000","last_clean":"2024-09-13T14:40:55.670057+0000","last_became_active":"2024-09-13T14:30:15.578890+0000","last_became_peered":"2024-09-13T14:30:15.578890+0000","last_unstale":"2024-09-13T14:40:55.670057+0000","last_undegraded":"2024-09-13T14:40:55.670057+0000","last_fullsized":"2024-09-13T14:40:55.670057+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:51:28.380523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":1050,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.623300+0000","last_change":"2024-09-13T14:29:42.541093+0000","last_active":"2024-09-13T14:40:40.623300+0000","last_peered":"2024-09-13T14:40:40.623300+0000","last_clean":"2024-09-13T14:40:40.623300+0000","last_became_active":"2024-09-13T14:29:42.540671+0000","last_became_peered":"2024-09-13T14:29:42.540671+0000","last_unstale":"2024-09-13T14:40:40.623300+0000","last_undegraded":"2024-09-13T14:40:40.623300+0000","last_fullsized":"2024-09-13T14:40:40.623300+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:45.722376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1046,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.635895+0000","last_change":"2024-09-13T14:39:11.379707+0000","last_active":"2024-09-13T14:40:40.635895+0000","last_peered":"2024-09-13T14:40:40.635895+0000","last_clean":"2024-09-13T14:40:40.635895+0000","last_became_active":"2024-09-13T14:39:11.379255+0000","last_became_peered":"2024-09-13T14:39:11.379255+0000","last_unstale":"2024-09-13T14:40:40.635895+0000","last_undegraded":"2024-09-13T14:40:40.635895+0000","last_fullsized":"2024-09-13T14:40:40.635895+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:48.543270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1594,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:55.679445+0000","last_change":"2024-09-13T14:39:11.392755+0000","last_active":"2024-09-13T14:40:55.679445+0000","last_peered":"2024-09-13T14:40:55.679445+0000","last_clean":"2024-09-13T14:40:55.679445+0000","last_became_active":"2024-09-13T14:39:11.392423+0000","last_became_peered":"2024-09-13T14:39:11.392423+0000","last_unstale":"2024-09-13T14:40:55.679445+0000","last_undegraded":"2024-09-13T14:40:55.679445+0000","last_fullsized":"2024-09-13T14:40:55.679445+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:18:16.726858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1561,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:55.679766+0000","last_change":"2024-09-13T14:39:11.394501+0000","last_active":"2024-09-13T14:40:55.679766+0000","last_peered":"2024-09-13T14:40:55.679766+0000","last_clean":"2024-09-13T14:40:55.679766+0000","last_became_active":"2024-09-13T14:39:11.394134+0000","last_became_peered":"2024-09-13T14:39:11.394134+0000","last_unstale":"2024-09-13T14:40:55.679766+0000","last_undegraded":"2024-09-13T14:40:55.679766+0000","last_fullsized":"2024-09-13T14:40:55.679766+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:40:09.359562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1031,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.622110+0000","last_change":"2024-09-13T14:19:25.873088+0000","last_active":"2024-09-13T14:40:40.622110+0000","last_peered":"2024-09-13T14:40:40.622110+0000","last_clean":"2024-09-13T14:40:40.622110+0000","last_became_active":"2024-09-13T14:19:25.872781+0000","last_became_peered":"2024-09-13T14:19:25.872781+0000","last_unstale":"2024-09-13T14:40:40.622110+0000","last_undegraded":"2024-09-13T14:40:40.622110+0000","last_fullsized":"2024-09-13T14:40:40.622110+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:05.607382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":583,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.624580+0000","last_change":"2024-09-13T14:39:11.384869+0000","last_active":"2024-09-13T14:40:40.624580+0000","last_peered":"2024-09-13T14:40:40.624580+0000","last_clean":"2024-09-13T14:40:40.624580+0000","last_became_active":"2024-09-13T14:39:11.384575+0000","last_became_peered":"2024-09-13T14:39:11.384575+0000","last_unstale":"2024-09-13T14:40:40.624580+0000","last_undegraded":"2024-09-13T14:40:40.624580+0000","last_fullsized":"2024-09-13T14:40:40.624580+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:45.765316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1039,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.633641+0000","last_change":"2024-09-13T14:28:35.603724+0000","last_active":"2024-09-13T14:40:40.633641+0000","last_peered":"2024-09-13T14:40:40.633641+0000","last_clean":"2024-09-13T14:40:40.633641+0000","last_became_active":"2024-09-13T14:28:35.603316+0000","last_became_peered":"2024-09-13T14:28:35.603316+0000","last_unstale":"2024-09-13T14:40:40.633641+0000","last_undegraded":"2024-09-13T14:40:40.633641+0000","last_fullsized":"2024-09-13T14:40:40.633641+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:47.668223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"1094'8389","reported_seq":8409,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:57.256681+0000","last_change":"2024-09-13T14:39:16.430434+0000","last_active":"2024-09-13T14:40:57.256681+0000","last_peered":"2024-09-13T14:40:57.256681+0000","last_clean":"2024-09-13T14:40:57.256681+0000","last_became_active":"2024-09-13T14:39:16.430177+0000","last_became_peered":"2024-09-13T14:39:16.430177+0000","last_unstale":"2024-09-13T14:40:57.256681+0000","last_undegraded":"2024-09-13T14:40:57.256681+0000","last_fullsized":"2024-09-13T14:40:57.256681+0000","mapping_epoch":1082,"log_start":"0'0","ondisk_log_start":"0'0","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":8389,"log_dups_size":0,"ondisk_log_size":8389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:44.232119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":549781504,"num_objects":8389,"num_object_clones":0,"num_object_copies":25167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8389,"num_write_kb":536896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":1049,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.624737+0000","last_change":"2024-09-13T14:28:35.608607+0000","last_active":"2024-09-13T14:40:40.624737+0000","last_peered":"2024-09-13T14:40:40.624737+0000","last_clean":"2024-09-13T14:40:40.624737+0000","last_became_active":"2024-09-13T14:28:35.608254+0000","last_became_peered":"2024-09-13T14:28:35.608254+0000","last_unstale":"2024-09-13T14:40:40.624737+0000","last_undegraded":"2024-09-13T14:40:40.624737+0000","last_fullsized":"2024-09-13T14:40:40.624737+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:24.019173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1019,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.635711+0000","last_change":"2024-09-13T14:19:26.887839+0000","last_active":"2024-09-13T14:40:40.635711+0000","last_peered":"2024-09-13T14:40:40.635711+0000","last_clean":"2024-09-13T14:40:40.635711+0000","last_became_active":"2024-09-13T14:19:26.887498+0000","last_became_peered":"2024-09-13T14:19:26.887498+0000","last_unstale":"2024-09-13T14:40:40.635711+0000","last_undegraded":"2024-09-13T14:40:40.635711+0000","last_fullsized":"2024-09-13T14:40:40.635711+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:24.475263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"1094'8413","reported_seq":8448,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:58.651655+0000","last_change":"2024-09-13T14:39:16.434529+0000","last_active":"2024-09-13T14:40:58.651655+0000","last_peered":"2024-09-13T14:40:58.651655+0000","last_clean":"2024-09-13T14:40:58.651655+0000","last_became_active":"2024-09-13T14:39:16.434199+0000","last_became_peered":"2024-09-13T14:39:16.434199+0000","last_unstale":"2024-09-13T14:40:58.651655+0000","last_undegraded":"2024-09-13T14:40:58.651655+0000","last_fullsized":"2024-09-13T14:40:58.651655+0000","mapping_epoch":1082,"log_start":"0'0","ondisk_log_start":"0'0","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":8413,"log_dups_size":0,"ondisk_log_size":8413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:18:22.588627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540344320,"num_objects":8245,"num_object_clones":0,"num_object_copies":24735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8413,"num_write_kb":533056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":1050,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.623549+0000","last_change":"2024-09-13T14:29:08.909370+0000","last_active":"2024-09-13T14:40:40.623549+0000","last_peered":"2024-09-13T14:40:40.623549+0000","last_clean":"2024-09-13T14:40:40.623549+0000","last_became_active":"2024-09-13T14:29:08.908983+0000","last_became_peered":"2024-09-13T14:29:08.908983+0000","last_unstale":"2024-09-13T14:40:40.623549+0000","last_undegraded":"2024-09-13T14:40:40.623549+0000","last_fullsized":"2024-09-13T14:40:40.623549+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:44:43.939022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1061,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.623621+0000","last_change":"2024-09-13T14:39:11.383270+0000","last_active":"2024-09-13T14:40:40.623621+0000","last_peered":"2024-09-13T14:40:40.623621+0000","last_clean":"2024-09-13T14:40:40.623621+0000","last_became_active":"2024-09-13T14:39:11.382968+0000","last_became_peered":"2024-09-13T14:39:11.382968+0000","last_unstale":"2024-09-13T14:40:40.623621+0000","last_undegraded":"2024-09-13T14:40:40.623621+0000","last_fullsized":"2024-09-13T14:40:40.623621+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:11.344101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"1094'8435","reported_seq":8455,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:57.235589+0000","last_change":"2024-09-13T14:39:16.428829+0000","last_active":"2024-09-13T14:40:57.235589+0000","last_peered":"2024-09-13T14:40:57.235589+0000","last_clean":"2024-09-13T14:40:57.235589+0000","last_became_active":"2024-09-13T14:39:16.428558+0000","last_became_peered":"2024-09-13T14:39:16.428558+0000","last_unstale":"2024-09-13T14:40:57.235589+0000","last_undegraded":"2024-09-13T14:40:57.235589+0000","last_fullsized":"2024-09-13T14:40:57.235589+0000","mapping_epoch":1082,"log_start":"1086'300","ondisk_log_start":"1086'300","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":8135,"log_dups_size":0,"ondisk_log_size":8135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:04:19.571418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":552796160,"num_objects":8435,"num_object_clones":0,"num_object_copies":25305,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8435,"num_write_kb":539840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":1032,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.623521+0000","last_change":"2024-09-13T14:19:25.871006+0000","last_active":"2024-09-13T14:40:40.623521+0000","last_peered":"2024-09-13T14:40:40.623521+0000","last_clean":"2024-09-13T14:40:40.623521+0000","last_became_active":"2024-09-13T14:19:25.870713+0000","last_became_peered":"2024-09-13T14:19:25.870713+0000","last_unstale":"2024-09-13T14:40:40.623521+0000","last_undegraded":"2024-09-13T14:40:40.623521+0000","last_fullsized":"2024-09-13T14:40:40.623521+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:34.372124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1030,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.635076+0000","last_change":"2024-09-13T14:29:08.914488+0000","last_active":"2024-09-13T14:40:40.635076+0000","last_peered":"2024-09-13T14:40:40.635076+0000","last_clean":"2024-09-13T14:40:40.635076+0000","last_became_active":"2024-09-13T14:29:08.913220+0000","last_became_peered":"2024-09-13T14:29:08.913220+0000","last_unstale":"2024-09-13T14:40:40.635076+0000","last_undegraded":"2024-09-13T14:40:40.635076+0000","last_fullsized":"2024-09-13T14:40:40.635076+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:08.316150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"1094'8335","reported_seq":8370,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:58.795351+0000","last_change":"2024-09-13T14:39:16.428268+0000","last_active":"2024-09-13T14:40:58.795351+0000","last_peered":"2024-09-13T14:40:58.795351+0000","last_clean":"2024-09-13T14:40:58.795351+0000","last_became_active":"2024-09-13T14:39:16.428016+0000","last_became_peered":"2024-09-13T14:39:16.428016+0000","last_unstale":"2024-09-13T14:40:58.795351+0000","last_undegraded":"2024-09-13T14:40:58.795351+0000","last_fullsized":"2024-09-13T14:40:58.795351+0000","mapping_epoch":1082,"log_start":"0'0","ondisk_log_start":"0'0","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":8335,"log_dups_size":0,"ondisk_log_size":8335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:28:16.657831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536281088,"num_objects":8183,"num_object_clones":0,"num_object_copies":24549,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8335,"num_write_kb":528576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1047,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.637661+0000","last_change":"2024-09-13T14:39:11.384164+0000","last_active":"2024-09-13T14:40:40.637661+0000","last_peered":"2024-09-13T14:40:40.637661+0000","last_clean":"2024-09-13T14:40:40.637661+0000","last_became_active":"2024-09-13T14:39:11.383667+0000","last_became_peered":"2024-09-13T14:39:11.383667+0000","last_unstale":"2024-09-13T14:40:40.637661+0000","last_undegraded":"2024-09-13T14:40:40.637661+0000","last_fullsized":"2024-09-13T14:40:40.637661+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:16:06.909588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1554,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:55.678878+0000","last_change":"2024-09-13T14:29:08.918531+0000","last_active":"2024-09-13T14:40:55.678878+0000","last_peered":"2024-09-13T14:40:55.678878+0000","last_clean":"2024-09-13T14:40:55.678878+0000","last_became_active":"2024-09-13T14:29:08.918264+0000","last_became_peered":"2024-09-13T14:29:08.918264+0000","last_unstale":"2024-09-13T14:40:55.678878+0000","last_undegraded":"2024-09-13T14:40:55.678878+0000","last_fullsized":"2024-09-13T14:40:55.678878+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:04.387551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"1094'8267","reported_seq":8287,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:57.210996+0000","last_change":"2024-09-13T14:39:16.428126+0000","last_active":"2024-09-13T14:40:57.210996+0000","last_peered":"2024-09-13T14:40:57.210996+0000","last_clean":"2024-09-13T14:40:57.210996+0000","last_became_active":"2024-09-13T14:39:16.427658+0000","last_became_peered":"2024-09-13T14:39:16.427658+0000","last_unstale":"2024-09-13T14:40:57.210996+0000","last_undegraded":"2024-09-13T14:40:57.210996+0000","last_fullsized":"2024-09-13T14:40:57.210996+0000","mapping_epoch":1082,"log_start":"1086'700","ondisk_log_start":"1086'700","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":7567,"log_dups_size":0,"ondisk_log_size":7567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:43:55.866559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":541786112,"num_objects":8267,"num_object_clones":0,"num_object_copies":24801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8267,"num_write_kb":529088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":85,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:55.679118+0000","last_change":"2024-09-13T14:39:11.403654+0000","last_active":"2024-09-13T14:40:55.679118+0000","last_peered":"2024-09-13T14:40:55.679118+0000","last_clean":"2024-09-13T14:40:55.679118+0000","last_became_active":"2024-09-13T14:39:11.392399+0000","last_became_peered":"2024-09-13T14:39:11.392399+0000","last_unstale":"2024-09-13T14:40:55.679118+0000","last_undegraded":"2024-09-13T14:40:55.679118+0000","last_fullsized":"2024-09-13T14:40:55.679118+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:45.818821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1044,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.624249+0000","last_change":"2024-09-13T14:39:11.383401+0000","last_active":"2024-09-13T14:40:40.624249+0000","last_peered":"2024-09-13T14:40:40.624249+0000","last_clean":"2024-09-13T14:40:40.624249+0000","last_became_active":"2024-09-13T14:39:11.383096+0000","last_became_peered":"2024-09-13T14:39:11.383096+0000","last_unstale":"2024-09-13T14:40:40.624249+0000","last_undegraded":"2024-09-13T14:40:40.624249+0000","last_fullsized":"2024-09-13T14:40:40.624249+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:09.333669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"1093'8098","reported_seq":8131,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:55.610511+0000","last_change":"2024-09-13T14:39:16.433163+0000","last_active":"2024-09-13T14:40:55.610511+0000","last_peered":"2024-09-13T14:40:55.610511+0000","last_clean":"2024-09-13T14:40:55.610511+0000","last_became_active":"2024-09-13T14:39:16.432783+0000","last_became_peered":"2024-09-13T14:39:16.432783+0000","last_unstale":"2024-09-13T14:40:55.610511+0000","last_undegraded":"2024-09-13T14:40:55.610511+0000","last_fullsized":"2024-09-13T14:40:55.610511+0000","mapping_epoch":1082,"log_start":"0'0","ondisk_log_start":"0'0","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":8098,"log_dups_size":0,"ondisk_log_size":8098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:43:29.182915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":530710528,"num_objects":8098,"num_object_clones":0,"num_object_copies":24294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8098,"num_write_kb":518272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1031,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.635186+0000","last_change":"2024-09-13T14:29:08.915311+0000","last_active":"2024-09-13T14:40:40.635186+0000","last_peered":"2024-09-13T14:40:40.635186+0000","last_clean":"2024-09-13T14:40:40.635186+0000","last_became_active":"2024-09-13T14:29:08.914923+0000","last_became_peered":"2024-09-13T14:29:08.914923+0000","last_unstale":"2024-09-13T14:40:40.635186+0000","last_undegraded":"2024-09-13T14:40:40.635186+0000","last_fullsized":"2024-09-13T14:40:40.635186+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:26.311233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1061,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.624625+0000","last_change":"2024-09-13T14:40:03.076276+0000","last_active":"2024-09-13T14:40:40.624625+0000","last_peered":"2024-09-13T14:40:40.624625+0000","last_clean":"2024-09-13T14:40:40.624625+0000","last_became_active":"2024-09-13T14:40:03.075637+0000","last_became_peered":"2024-09-13T14:40:03.075637+0000","last_unstale":"2024-09-13T14:40:40.624625+0000","last_undegraded":"2024-09-13T14:40:40.624625+0000","last_fullsized":"2024-09-13T14:40:40.624625+0000","mapping_epoch":1089,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1090,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:34.472805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"1094'8294","reported_seq":8314,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:57.182382+0000","last_change":"2024-09-13T14:39:16.428687+0000","last_active":"2024-09-13T14:40:57.182382+0000","last_peered":"2024-09-13T14:40:57.182382+0000","last_clean":"2024-09-13T14:40:57.182382+0000","last_became_active":"2024-09-13T14:39:16.428382+0000","last_became_peered":"2024-09-13T14:39:16.428382+0000","last_unstale":"2024-09-13T14:40:57.182382+0000","last_undegraded":"2024-09-13T14:40:57.182382+0000","last_fullsized":"2024-09-13T14:40:57.182382+0000","mapping_epoch":1082,"log_start":"1087'1300","ondisk_log_start":"1087'1300","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6994,"log_dups_size":0,"ondisk_log_size":6994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:33:44.732564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":543555584,"num_objects":8294,"num_object_clones":0,"num_object_copies":24882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8294,"num_write_kb":530816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1035,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.623688+0000","last_change":"2024-09-13T14:29:42.545815+0000","last_active":"2024-09-13T14:40:40.623688+0000","last_peered":"2024-09-13T14:40:40.623688+0000","last_clean":"2024-09-13T14:40:40.623688+0000","last_became_active":"2024-09-13T14:29:42.545508+0000","last_became_peered":"2024-09-13T14:29:42.545508+0000","last_unstale":"2024-09-13T14:40:40.623688+0000","last_undegraded":"2024-09-13T14:40:40.623688+0000","last_fullsized":"2024-09-13T14:40:40.623688+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:39.678381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1034,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.626908+0000","last_change":"2024-09-13T14:29:08.916416+0000","last_active":"2024-09-13T14:40:40.626908+0000","last_peered":"2024-09-13T14:40:40.626908+0000","last_clean":"2024-09-13T14:40:40.626908+0000","last_became_active":"2024-09-13T14:29:08.916106+0000","last_became_peered":"2024-09-13T14:29:08.916106+0000","last_unstale":"2024-09-13T14:40:40.626908+0000","last_undegraded":"2024-09-13T14:40:40.626908+0000","last_fullsized":"2024-09-13T14:40:40.626908+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:25.508423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"1094'8255","reported_seq":8276,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:57.289648+0000","last_change":"2024-09-13T14:39:16.430380+0000","last_active":"2024-09-13T14:40:57.289648+0000","last_peered":"2024-09-13T14:40:57.289648+0000","last_clean":"2024-09-13T14:40:57.289648+0000","last_became_active":"2024-09-13T14:39:16.430088+0000","last_became_peered":"2024-09-13T14:39:16.430088+0000","last_unstale":"2024-09-13T14:40:57.289648+0000","last_undegraded":"2024-09-13T14:40:57.289648+0000","last_fullsized":"2024-09-13T14:40:57.289648+0000","mapping_epoch":1082,"log_start":"0'0","ondisk_log_start":"0'0","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":8255,"log_dups_size":0,"ondisk_log_size":8255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:32:40.284089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540934168,"num_objects":8255,"num_object_clones":0,"num_object_copies":24765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8255,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8255,"num_write_kb":528257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":598,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:55.666997+0000","last_change":"2024-09-13T14:39:11.384140+0000","last_active":"2024-09-13T14:40:55.666997+0000","last_peered":"2024-09-13T14:40:55.666997+0000","last_clean":"2024-09-13T14:40:55.666997+0000","last_became_active":"2024-09-13T14:39:11.383829+0000","last_became_peered":"2024-09-13T14:39:11.383829+0000","last_unstale":"2024-09-13T14:40:55.666997+0000","last_undegraded":"2024-09-13T14:40:55.666997+0000","last_fullsized":"2024-09-13T14:40:55.666997+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:20:04.914959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1532,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:55.666204+0000","last_change":"2024-09-13T14:30:15.580234+0000","last_active":"2024-09-13T14:40:55.666204+0000","last_peered":"2024-09-13T14:40:55.666204+0000","last_clean":"2024-09-13T14:40:55.666204+0000","last_became_active":"2024-09-13T14:30:15.579921+0000","last_became_peered":"2024-09-13T14:30:15.579921+0000","last_unstale":"2024-09-13T14:40:55.666204+0000","last_undegraded":"2024-09-13T14:40:55.666204+0000","last_fullsized":"2024-09-13T14:40:55.666204+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:42:39.428403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"1094'8351","reported_seq":8370,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:57.272102+0000","last_change":"2024-09-13T14:39:16.431907+0000","last_active":"2024-09-13T14:40:57.272102+0000","last_peered":"2024-09-13T14:40:57.272102+0000","last_clean":"2024-09-13T14:40:57.272102+0000","last_became_active":"2024-09-13T14:39:16.431660+0000","last_became_peered":"2024-09-13T14:39:16.431660+0000","last_unstale":"2024-09-13T14:40:57.272102+0000","last_undegraded":"2024-09-13T14:40:57.272102+0000","last_fullsized":"2024-09-13T14:40:57.272102+0000","mapping_epoch":1082,"log_start":"0'0","ondisk_log_start":"0'0","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":8351,"log_dups_size":0,"ondisk_log_size":8351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:18.941482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":547028992,"num_objects":8347,"num_object_clones":0,"num_object_copies":25041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8351,"num_write_kb":534336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"261'100","reported_seq":1164,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.622291+0000","last_change":"2024-09-13T14:28:35.604004+0000","last_active":"2024-09-13T14:40:40.622291+0000","last_peered":"2024-09-13T14:40:40.622291+0000","last_clean":"2024-09-13T14:40:40.622291+0000","last_became_active":"2024-09-13T14:28:35.603626+0000","last_became_peered":"2024-09-13T14:28:35.603626+0000","last_unstale":"2024-09-13T14:40:40.622291+0000","last_undegraded":"2024-09-13T14:40:40.622291+0000","last_fullsized":"2024-09-13T14:40:40.622291+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:15:42.069756+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:39:52.205001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":1050,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.635913+0000","last_change":"2024-09-13T14:29:42.539455+0000","last_active":"2024-09-13T14:40:40.635913+0000","last_peered":"2024-09-13T14:40:40.635913+0000","last_clean":"2024-09-13T14:40:40.635913+0000","last_became_active":"2024-09-13T14:29:42.539068+0000","last_became_peered":"2024-09-13T14:29:42.539068+0000","last_unstale":"2024-09-13T14:40:40.635913+0000","last_undegraded":"2024-09-13T14:40:40.635913+0000","last_fullsized":"2024-09-13T14:40:40.635913+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:24:06.048855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034405999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":561,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.635983+0000","last_change":"2024-09-13T14:39:11.382555+0000","last_active":"2024-09-13T14:40:40.635983+0000","last_peered":"2024-09-13T14:40:40.635983+0000","last_clean":"2024-09-13T14:40:40.635983+0000","last_became_active":"2024-09-13T14:39:11.382247+0000","last_became_peered":"2024-09-13T14:39:11.382247+0000","last_unstale":"2024-09-13T14:40:40.635983+0000","last_undegraded":"2024-09-13T14:40:40.635983+0000","last_fullsized":"2024-09-13T14:40:40.635983+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:09:48.016987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1013,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.634745+0000","last_change":"2024-09-13T14:19:27.379312+0000","last_active":"2024-09-13T14:40:40.634745+0000","last_peered":"2024-09-13T14:40:40.634745+0000","last_clean":"2024-09-13T14:40:40.634745+0000","last_became_active":"2024-09-13T14:19:27.378947+0000","last_became_peered":"2024-09-13T14:19:27.378947+0000","last_unstale":"2024-09-13T14:40:40.634745+0000","last_undegraded":"2024-09-13T14:40:40.634745+0000","last_fullsized":"2024-09-13T14:40:40.634745+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:54:20.706578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"1094'8408","reported_seq":8443,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:56.931179+0000","last_change":"2024-09-13T14:39:16.430331+0000","last_active":"2024-09-13T14:40:56.931179+0000","last_peered":"2024-09-13T14:40:56.931179+0000","last_clean":"2024-09-13T14:40:56.931179+0000","last_became_active":"2024-09-13T14:39:16.429979+0000","last_became_peered":"2024-09-13T14:39:16.429979+0000","last_unstale":"2024-09-13T14:40:56.931179+0000","last_undegraded":"2024-09-13T14:40:56.931179+0000","last_fullsized":"2024-09-13T14:40:56.931179+0000","mapping_epoch":1082,"log_start":"1087'1400","ondisk_log_start":"1087'1400","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":7008,"log_dups_size":0,"ondisk_log_size":7008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:03.966261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":551026688,"num_objects":8408,"num_object_clones":0,"num_object_copies":25224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8408,"num_write_kb":538112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1077,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.622134+0000","last_change":"2024-09-13T14:29:08.913445+0000","last_active":"2024-09-13T14:40:40.622134+0000","last_peered":"2024-09-13T14:40:40.622134+0000","last_clean":"2024-09-13T14:40:40.622134+0000","last_became_active":"2024-09-13T14:29:08.913153+0000","last_became_peered":"2024-09-13T14:29:08.913153+0000","last_unstale":"2024-09-13T14:40:40.622134+0000","last_undegraded":"2024-09-13T14:40:40.622134+0000","last_fullsized":"2024-09-13T14:40:40.622134+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:55:34.761217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1594,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:55.678881+0000","last_change":"2024-09-13T14:39:11.390055+0000","last_active":"2024-09-13T14:40:55.678881+0000","last_peered":"2024-09-13T14:40:55.678881+0000","last_clean":"2024-09-13T14:40:55.678881+0000","last_became_active":"2024-09-13T14:39:11.389697+0000","last_became_peered":"2024-09-13T14:39:11.389697+0000","last_unstale":"2024-09-13T14:40:55.678881+0000","last_undegraded":"2024-09-13T14:40:55.678881+0000","last_fullsized":"2024-09-13T14:40:55.678881+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:50.975739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1565,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:55.678874+0000","last_change":"2024-09-13T14:29:42.541652+0000","last_active":"2024-09-13T14:40:55.678874+0000","last_peered":"2024-09-13T14:40:55.678874+0000","last_clean":"2024-09-13T14:40:55.678874+0000","last_became_active":"2024-09-13T14:29:42.541341+0000","last_became_peered":"2024-09-13T14:29:42.541341+0000","last_unstale":"2024-09-13T14:40:55.678874+0000","last_undegraded":"2024-09-13T14:40:55.678874+0000","last_fullsized":"2024-09-13T14:40:55.678874+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:52.954478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"1094'8125","reported_seq":8145,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:57.256607+0000","last_change":"2024-09-13T14:39:16.428528+0000","last_active":"2024-09-13T14:40:57.256607+0000","last_peered":"2024-09-13T14:40:57.256607+0000","last_clean":"2024-09-13T14:40:57.256607+0000","last_became_active":"2024-09-13T14:39:16.428195+0000","last_became_peered":"2024-09-13T14:39:16.428195+0000","last_unstale":"2024-09-13T14:40:57.256607+0000","last_undegraded":"2024-09-13T14:40:57.256607+0000","last_fullsized":"2024-09-13T14:40:57.256607+0000","mapping_epoch":1082,"log_start":"0'0","ondisk_log_start":"0'0","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":8125,"log_dups_size":0,"ondisk_log_size":8125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:54:28.451285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":532480000,"num_objects":8125,"num_object_clones":0,"num_object_copies":24375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8125,"num_write_kb":520000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1562,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.622586+0000","last_change":"2024-09-13T14:39:11.397601+0000","last_active":"2024-09-13T14:40:40.622586+0000","last_peered":"2024-09-13T14:40:40.622586+0000","last_clean":"2024-09-13T14:40:40.622586+0000","last_became_active":"2024-09-13T14:39:11.397290+0000","last_became_peered":"2024-09-13T14:39:11.397290+0000","last_unstale":"2024-09-13T14:40:40.622586+0000","last_undegraded":"2024-09-13T14:40:40.622586+0000","last_fullsized":"2024-09-13T14:40:40.622586+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:44:33.189136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1531,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:55.667032+0000","last_change":"2024-09-13T14:29:42.539284+0000","last_active":"2024-09-13T14:40:55.667032+0000","last_peered":"2024-09-13T14:40:55.667032+0000","last_clean":"2024-09-13T14:40:55.667032+0000","last_became_active":"2024-09-13T14:29:42.538970+0000","last_became_peered":"2024-09-13T14:29:42.538970+0000","last_unstale":"2024-09-13T14:40:55.667032+0000","last_undegraded":"2024-09-13T14:40:55.667032+0000","last_fullsized":"2024-09-13T14:40:55.667032+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:04.562806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1549,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:55.666015+0000","last_change":"2024-09-13T14:29:08.918645+0000","last_active":"2024-09-13T14:40:55.666015+0000","last_peered":"2024-09-13T14:40:55.666015+0000","last_clean":"2024-09-13T14:40:55.666015+0000","last_became_active":"2024-09-13T14:29:08.915894+0000","last_became_peered":"2024-09-13T14:29:08.915894+0000","last_unstale":"2024-09-13T14:40:55.666015+0000","last_undegraded":"2024-09-13T14:40:55.666015+0000","last_fullsized":"2024-09-13T14:40:55.666015+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:26.651547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"1094'8241","reported_seq":8261,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:57.218979+0000","last_change":"2024-09-13T14:39:16.427959+0000","last_active":"2024-09-13T14:40:57.218979+0000","last_peered":"2024-09-13T14:40:57.218979+0000","last_clean":"2024-09-13T14:40:57.218979+0000","last_became_active":"2024-09-13T14:39:16.427270+0000","last_became_peered":"2024-09-13T14:39:16.427270+0000","last_unstale":"2024-09-13T14:40:57.218979+0000","last_undegraded":"2024-09-13T14:40:57.218979+0000","last_fullsized":"2024-09-13T14:40:57.218979+0000","mapping_epoch":1082,"log_start":"1086'700","ondisk_log_start":"1086'700","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":7541,"log_dups_size":0,"ondisk_log_size":7541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:51.791556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540082176,"num_objects":8241,"num_object_clones":0,"num_object_copies":24723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8241,"num_write_kb":527424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":1051,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.636093+0000","last_change":"2024-09-13T14:29:08.913956+0000","last_active":"2024-09-13T14:40:40.636093+0000","last_peered":"2024-09-13T14:40:40.636093+0000","last_clean":"2024-09-13T14:40:40.636093+0000","last_became_active":"2024-09-13T14:29:08.913081+0000","last_became_peered":"2024-09-13T14:29:08.913081+0000","last_unstale":"2024-09-13T14:40:40.636093+0000","last_undegraded":"2024-09-13T14:40:40.636093+0000","last_fullsized":"2024-09-13T14:40:40.636093+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:28.876391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1035,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.624111+0000","last_change":"2024-09-13T14:28:35.604404+0000","last_active":"2024-09-13T14:40:40.624111+0000","last_peered":"2024-09-13T14:40:40.624111+0000","last_clean":"2024-09-13T14:40:40.624111+0000","last_became_active":"2024-09-13T14:28:35.604102+0000","last_became_peered":"2024-09-13T14:28:35.604102+0000","last_unstale":"2024-09-13T14:40:40.624111+0000","last_undegraded":"2024-09-13T14:40:40.624111+0000","last_fullsized":"2024-09-13T14:40:40.624111+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:25.546010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1550,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:55.666208+0000","last_change":"2024-09-13T14:39:11.383213+0000","last_active":"2024-09-13T14:40:55.666208+0000","last_peered":"2024-09-13T14:40:55.666208+0000","last_clean":"2024-09-13T14:40:55.666208+0000","last_became_active":"2024-09-13T14:39:11.382911+0000","last_became_peered":"2024-09-13T14:39:11.382911+0000","last_unstale":"2024-09-13T14:40:55.666208+0000","last_undegraded":"2024-09-13T14:40:55.666208+0000","last_fullsized":"2024-09-13T14:40:55.666208+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:10:21.064787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"1094'8429","reported_seq":8464,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:58.770943+0000","last_change":"2024-09-13T14:39:16.429234+0000","last_active":"2024-09-13T14:40:58.770943+0000","last_peered":"2024-09-13T14:40:58.770943+0000","last_clean":"2024-09-13T14:40:58.770943+0000","last_became_active":"2024-09-13T14:39:16.428954+0000","last_became_peered":"2024-09-13T14:39:16.428954+0000","last_unstale":"2024-09-13T14:40:58.770943+0000","last_undegraded":"2024-09-13T14:40:58.770943+0000","last_fullsized":"2024-09-13T14:40:58.770943+0000","mapping_epoch":1082,"log_start":"0'0","ondisk_log_start":"0'0","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":8429,"log_dups_size":0,"ondisk_log_size":8429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:48.562878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538902528,"num_objects":8223,"num_object_clones":0,"num_object_copies":24669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8429,"num_write_kb":532864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1064,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.642229+0000","last_change":"2024-09-13T14:39:11.383045+0000","last_active":"2024-09-13T14:40:40.642229+0000","last_peered":"2024-09-13T14:40:40.642229+0000","last_clean":"2024-09-13T14:40:40.642229+0000","last_became_active":"2024-09-13T14:39:11.382718+0000","last_became_peered":"2024-09-13T14:39:11.382718+0000","last_unstale":"2024-09-13T14:40:40.642229+0000","last_undegraded":"2024-09-13T14:40:40.642229+0000","last_fullsized":"2024-09-13T14:40:40.642229+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:21.812296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1563,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:55.666377+0000","last_change":"2024-09-13T14:29:08.916449+0000","last_active":"2024-09-13T14:40:55.666377+0000","last_peered":"2024-09-13T14:40:55.666377+0000","last_clean":"2024-09-13T14:40:55.666377+0000","last_became_active":"2024-09-13T14:29:08.914257+0000","last_became_peered":"2024-09-13T14:29:08.914257+0000","last_unstale":"2024-09-13T14:40:55.666377+0000","last_undegraded":"2024-09-13T14:40:55.666377+0000","last_fullsized":"2024-09-13T14:40:55.666377+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:10:32.449958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1521,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:55.665519+0000","last_change":"2024-09-13T14:29:08.913030+0000","last_active":"2024-09-13T14:40:55.665519+0000","last_peered":"2024-09-13T14:40:55.665519+0000","last_clean":"2024-09-13T14:40:55.665519+0000","last_became_active":"2024-09-13T14:29:08.912671+0000","last_became_peered":"2024-09-13T14:29:08.912671+0000","last_unstale":"2024-09-13T14:40:55.665519+0000","last_undegraded":"2024-09-13T14:40:55.665519+0000","last_fullsized":"2024-09-13T14:40:55.665519+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:30:41.107429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"1094'8143","reported_seq":8178,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:56.851545+0000","last_change":"2024-09-13T14:39:16.429574+0000","last_active":"2024-09-13T14:40:56.851545+0000","last_peered":"2024-09-13T14:40:56.851545+0000","last_clean":"2024-09-13T14:40:56.851545+0000","last_became_active":"2024-09-13T14:39:16.429293+0000","last_became_peered":"2024-09-13T14:39:16.429293+0000","last_unstale":"2024-09-13T14:40:56.851545+0000","last_undegraded":"2024-09-13T14:40:56.851545+0000","last_fullsized":"2024-09-13T14:40:56.851545+0000","mapping_epoch":1082,"log_start":"1086'1100","ondisk_log_start":"1086'1100","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":7043,"log_dups_size":0,"ondisk_log_size":7043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:37:21.404916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":533659648,"num_objects":8143,"num_object_clones":0,"num_object_copies":24429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8143,"num_write_kb":521152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1531,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:55.666290+0000","last_change":"2024-09-13T14:29:42.540294+0000","last_active":"2024-09-13T14:40:55.666290+0000","last_peered":"2024-09-13T14:40:55.666290+0000","last_clean":"2024-09-13T14:40:55.666290+0000","last_became_active":"2024-09-13T14:29:42.539930+0000","last_became_peered":"2024-09-13T14:29:42.539930+0000","last_unstale":"2024-09-13T14:40:55.666290+0000","last_undegraded":"2024-09-13T14:40:55.666290+0000","last_fullsized":"2024-09-13T14:40:55.666290+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:42.999879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1060,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.623599+0000","last_change":"2024-09-13T14:28:35.603663+0000","last_active":"2024-09-13T14:40:40.623599+0000","last_peered":"2024-09-13T14:40:40.623599+0000","last_clean":"2024-09-13T14:40:40.623599+0000","last_became_active":"2024-09-13T14:28:35.603358+0000","last_became_peered":"2024-09-13T14:28:35.603358+0000","last_unstale":"2024-09-13T14:40:40.623599+0000","last_undegraded":"2024-09-13T14:40:40.623599+0000","last_fullsized":"2024-09-13T14:40:40.623599+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:24:05.842796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1543,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:55.678879+0000","last_change":"2024-09-13T14:28:35.611508+0000","last_active":"2024-09-13T14:40:55.678879+0000","last_peered":"2024-09-13T14:40:55.678879+0000","last_clean":"2024-09-13T14:40:55.678879+0000","last_became_active":"2024-09-13T14:28:35.610106+0000","last_became_peered":"2024-09-13T14:28:35.610106+0000","last_unstale":"2024-09-13T14:40:55.678879+0000","last_undegraded":"2024-09-13T14:40:55.678879+0000","last_fullsized":"2024-09-13T14:40:55.678879+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:11.966958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"1094'8306","reported_seq":8326,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:57.218037+0000","last_change":"2024-09-13T14:39:16.429539+0000","last_active":"2024-09-13T14:40:57.218037+0000","last_peered":"2024-09-13T14:40:57.218037+0000","last_clean":"2024-09-13T14:40:57.218037+0000","last_became_active":"2024-09-13T14:39:16.426462+0000","last_became_peered":"2024-09-13T14:39:16.426462+0000","last_unstale":"2024-09-13T14:40:57.218037+0000","last_undegraded":"2024-09-13T14:40:57.218037+0000","last_fullsized":"2024-09-13T14:40:57.218037+0000","mapping_epoch":1082,"log_start":"1086'800","ondisk_log_start":"1086'800","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":7506,"log_dups_size":0,"ondisk_log_size":7506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:58:57.425171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":544342016,"num_objects":8306,"num_object_clones":0,"num_object_copies":24918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8306,"num_write_kb":531584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":1503,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:55.672380+0000","last_change":"2024-09-13T14:30:15.581728+0000","last_active":"2024-09-13T14:40:55.672380+0000","last_peered":"2024-09-13T14:40:55.672380+0000","last_clean":"2024-09-13T14:40:55.672380+0000","last_became_active":"2024-09-13T14:30:15.581303+0000","last_became_peered":"2024-09-13T14:30:15.581303+0000","last_unstale":"2024-09-13T14:40:55.672380+0000","last_undegraded":"2024-09-13T14:40:55.672380+0000","last_fullsized":"2024-09-13T14:40:55.672380+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:10.027991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1531,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:55.666475+0000","last_change":"2024-09-13T14:29:08.917504+0000","last_active":"2024-09-13T14:40:55.666475+0000","last_peered":"2024-09-13T14:40:55.666475+0000","last_clean":"2024-09-13T14:40:55.666475+0000","last_became_active":"2024-09-13T14:29:08.917217+0000","last_became_peered":"2024-09-13T14:29:08.917217+0000","last_unstale":"2024-09-13T14:40:55.666475+0000","last_undegraded":"2024-09-13T14:40:55.666475+0000","last_fullsized":"2024-09-13T14:40:55.666475+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:36.651514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":591,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.621931+0000","last_change":"2024-09-13T14:39:11.393557+0000","last_active":"2024-09-13T14:40:40.621931+0000","last_peered":"2024-09-13T14:40:40.621931+0000","last_clean":"2024-09-13T14:40:40.621931+0000","last_became_active":"2024-09-13T14:39:11.393230+0000","last_became_peered":"2024-09-13T14:39:11.393230+0000","last_unstale":"2024-09-13T14:40:40.621931+0000","last_undegraded":"2024-09-13T14:40:40.621931+0000","last_fullsized":"2024-09-13T14:40:40.621931+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:25.924635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"1094'8103","reported_seq":8123,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:57.278460+0000","last_change":"2024-09-13T14:39:16.427785+0000","last_active":"2024-09-13T14:40:57.278460+0000","last_peered":"2024-09-13T14:40:57.278460+0000","last_clean":"2024-09-13T14:40:57.278460+0000","last_became_active":"2024-09-13T14:39:16.427520+0000","last_became_peered":"2024-09-13T14:39:16.427520+0000","last_unstale":"2024-09-13T14:40:57.278460+0000","last_undegraded":"2024-09-13T14:40:57.278460+0000","last_fullsized":"2024-09-13T14:40:57.278460+0000","mapping_epoch":1082,"log_start":"0'0","ondisk_log_start":"0'0","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":8103,"log_dups_size":0,"ondisk_log_size":8103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:27.697724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":531038208,"num_objects":8103,"num_object_clones":0,"num_object_copies":24309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8103,"num_write_kb":518592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":1568,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:55.679859+0000","last_change":"2024-09-13T14:40:03.076955+0000","last_active":"2024-09-13T14:40:55.679859+0000","last_peered":"2024-09-13T14:40:55.679859+0000","last_clean":"2024-09-13T14:40:55.679859+0000","last_became_active":"2024-09-13T14:40:03.074488+0000","last_became_peered":"2024-09-13T14:40:03.074488+0000","last_unstale":"2024-09-13T14:40:55.679859+0000","last_undegraded":"2024-09-13T14:40:55.679859+0000","last_fullsized":"2024-09-13T14:40:55.679859+0000","mapping_epoch":1089,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1090,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:21:13.763905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1044,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.632489+0000","last_change":"2024-09-13T14:29:42.538860+0000","last_active":"2024-09-13T14:40:40.632489+0000","last_peered":"2024-09-13T14:40:40.632489+0000","last_clean":"2024-09-13T14:40:40.632489+0000","last_became_active":"2024-09-13T14:29:42.538608+0000","last_became_peered":"2024-09-13T14:29:42.538608+0000","last_unstale":"2024-09-13T14:40:40.632489+0000","last_undegraded":"2024-09-13T14:40:40.632489+0000","last_fullsized":"2024-09-13T14:40:40.632489+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:20.070703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1017,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.623487+0000","last_change":"2024-09-13T14:19:26.131609+0000","last_active":"2024-09-13T14:40:40.623487+0000","last_peered":"2024-09-13T14:40:40.623487+0000","last_clean":"2024-09-13T14:40:40.623487+0000","last_became_active":"2024-09-13T14:19:26.131029+0000","last_became_peered":"2024-09-13T14:19:26.131029+0000","last_unstale":"2024-09-13T14:40:40.623487+0000","last_undegraded":"2024-09-13T14:40:40.623487+0000","last_fullsized":"2024-09-13T14:40:40.623487+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:57.933696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1034,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.624044+0000","last_change":"2024-09-13T14:29:08.911835+0000","last_active":"2024-09-13T14:40:40.624044+0000","last_peered":"2024-09-13T14:40:40.624044+0000","last_clean":"2024-09-13T14:40:40.624044+0000","last_became_active":"2024-09-13T14:29:08.911542+0000","last_became_peered":"2024-09-13T14:29:08.911542+0000","last_unstale":"2024-09-13T14:40:40.624044+0000","last_undegraded":"2024-09-13T14:40:40.624044+0000","last_fullsized":"2024-09-13T14:40:40.624044+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:30.820674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":1040,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.624324+0000","last_change":"2024-09-13T14:19:25.870605+0000","last_active":"2024-09-13T14:40:40.624324+0000","last_peered":"2024-09-13T14:40:40.624324+0000","last_clean":"2024-09-13T14:40:40.624324+0000","last_became_active":"2024-09-13T14:19:25.870282+0000","last_became_peered":"2024-09-13T14:19:25.870282+0000","last_unstale":"2024-09-13T14:40:40.624324+0000","last_undegraded":"2024-09-13T14:40:40.624324+0000","last_fullsized":"2024-09-13T14:40:40.624324+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:52.514915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1532,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:55.665636+0000","last_change":"2024-09-13T14:29:42.537787+0000","last_active":"2024-09-13T14:40:55.665636+0000","last_peered":"2024-09-13T14:40:55.665636+0000","last_clean":"2024-09-13T14:40:55.665636+0000","last_became_active":"2024-09-13T14:29:42.537499+0000","last_became_peered":"2024-09-13T14:29:42.537499+0000","last_unstale":"2024-09-13T14:40:55.665636+0000","last_undegraded":"2024-09-13T14:40:55.665636+0000","last_fullsized":"2024-09-13T14:40:55.665636+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:15:30.508656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":1051,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.623357+0000","last_change":"2024-09-13T14:30:15.579260+0000","last_active":"2024-09-13T14:40:40.623357+0000","last_peered":"2024-09-13T14:40:40.623357+0000","last_clean":"2024-09-13T14:40:40.623357+0000","last_became_active":"2024-09-13T14:30:15.578897+0000","last_became_peered":"2024-09-13T14:30:15.578897+0000","last_unstale":"2024-09-13T14:40:40.623357+0000","last_undegraded":"2024-09-13T14:40:40.623357+0000","last_fullsized":"2024-09-13T14:40:40.623357+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:56.737849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1037,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.635812+0000","last_change":"2024-09-13T14:29:08.910821+0000","last_active":"2024-09-13T14:40:40.635812+0000","last_peered":"2024-09-13T14:40:40.635812+0000","last_clean":"2024-09-13T14:40:40.635812+0000","last_became_active":"2024-09-13T14:29:08.910509+0000","last_became_peered":"2024-09-13T14:29:08.910509+0000","last_unstale":"2024-09-13T14:40:40.635812+0000","last_undegraded":"2024-09-13T14:40:40.635812+0000","last_fullsized":"2024-09-13T14:40:40.635812+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:59.495053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1471,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:55.669776+0000","last_change":"2024-09-13T14:30:15.576921+0000","last_active":"2024-09-13T14:40:55.669776+0000","last_peered":"2024-09-13T14:40:55.669776+0000","last_clean":"2024-09-13T14:40:55.669776+0000","last_became_active":"2024-09-13T14:30:15.576607+0000","last_became_peered":"2024-09-13T14:30:15.576607+0000","last_unstale":"2024-09-13T14:40:55.669776+0000","last_undegraded":"2024-09-13T14:40:55.669776+0000","last_fullsized":"2024-09-13T14:40:55.669776+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:35.162460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1594,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:55.679364+0000","last_change":"2024-09-13T14:39:11.386005+0000","last_active":"2024-09-13T14:40:55.679364+0000","last_peered":"2024-09-13T14:40:55.679364+0000","last_clean":"2024-09-13T14:40:55.679364+0000","last_became_active":"2024-09-13T14:39:11.385691+0000","last_became_peered":"2024-09-13T14:39:11.385691+0000","last_unstale":"2024-09-13T14:40:55.679364+0000","last_undegraded":"2024-09-13T14:40:55.679364+0000","last_fullsized":"2024-09-13T14:40:55.679364+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:11:03.371484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":613,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.621341+0000","last_change":"2024-09-13T14:39:11.388166+0000","last_active":"2024-09-13T14:40:40.621341+0000","last_peered":"2024-09-13T14:40:40.621341+0000","last_clean":"2024-09-13T14:40:40.621341+0000","last_became_active":"2024-09-13T14:39:11.387829+0000","last_became_peered":"2024-09-13T14:39:11.387829+0000","last_unstale":"2024-09-13T14:40:40.621341+0000","last_undegraded":"2024-09-13T14:40:40.621341+0000","last_fullsized":"2024-09-13T14:40:40.621341+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:22:39.070060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1030,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.632413+0000","last_change":"2024-09-13T14:29:08.913672+0000","last_active":"2024-09-13T14:40:40.632413+0000","last_peered":"2024-09-13T14:40:40.632413+0000","last_clean":"2024-09-13T14:40:40.632413+0000","last_became_active":"2024-09-13T14:29:08.913360+0000","last_became_peered":"2024-09-13T14:29:08.913360+0000","last_unstale":"2024-09-13T14:40:40.632413+0000","last_undegraded":"2024-09-13T14:40:40.632413+0000","last_fullsized":"2024-09-13T14:40:40.632413+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:45.688163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":615,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:55.678727+0000","last_change":"2024-09-13T14:39:11.393642+0000","last_active":"2024-09-13T14:40:55.678727+0000","last_peered":"2024-09-13T14:40:55.678727+0000","last_clean":"2024-09-13T14:40:55.678727+0000","last_became_active":"2024-09-13T14:39:11.393326+0000","last_became_peered":"2024-09-13T14:39:11.393326+0000","last_unstale":"2024-09-13T14:40:55.678727+0000","last_undegraded":"2024-09-13T14:40:55.678727+0000","last_fullsized":"2024-09-13T14:40:55.678727+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:46:57.801236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1562,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:55.679670+0000","last_change":"2024-09-13T14:39:11.385662+0000","last_active":"2024-09-13T14:40:55.679670+0000","last_peered":"2024-09-13T14:40:55.679670+0000","last_clean":"2024-09-13T14:40:55.679670+0000","last_became_active":"2024-09-13T14:39:11.385303+0000","last_became_peered":"2024-09-13T14:39:11.385303+0000","last_unstale":"2024-09-13T14:40:55.679670+0000","last_undegraded":"2024-09-13T14:40:55.679670+0000","last_fullsized":"2024-09-13T14:40:55.679670+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:10:32.266845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1530,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:55.665368+0000","last_change":"2024-09-13T14:30:15.579398+0000","last_active":"2024-09-13T14:40:55.665368+0000","last_peered":"2024-09-13T14:40:55.665368+0000","last_clean":"2024-09-13T14:40:55.665368+0000","last_became_active":"2024-09-13T14:30:15.579062+0000","last_became_peered":"2024-09-13T14:30:15.579062+0000","last_unstale":"2024-09-13T14:40:55.665368+0000","last_undegraded":"2024-09-13T14:40:55.665368+0000","last_fullsized":"2024-09-13T14:40:55.665368+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:06:39.532417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":597,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:55.666318+0000","last_change":"2024-09-13T14:39:11.382364+0000","last_active":"2024-09-13T14:40:55.666318+0000","last_peered":"2024-09-13T14:40:55.666318+0000","last_clean":"2024-09-13T14:40:55.666318+0000","last_became_active":"2024-09-13T14:39:11.382036+0000","last_became_peered":"2024-09-13T14:39:11.382036+0000","last_unstale":"2024-09-13T14:40:55.666318+0000","last_undegraded":"2024-09-13T14:40:55.666318+0000","last_fullsized":"2024-09-13T14:40:55.666318+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:52:58.525228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":1053,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.626719+0000","last_change":"2024-09-13T14:29:42.537809+0000","last_active":"2024-09-13T14:40:40.626719+0000","last_peered":"2024-09-13T14:40:40.626719+0000","last_clean":"2024-09-13T14:40:40.626719+0000","last_became_active":"2024-09-13T14:29:42.537482+0000","last_became_peered":"2024-09-13T14:29:42.537482+0000","last_unstale":"2024-09-13T14:40:40.626719+0000","last_undegraded":"2024-09-13T14:40:40.626719+0000","last_fullsized":"2024-09-13T14:40:40.626719+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:19.631210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1545,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.619968+0000","last_change":"2024-09-13T14:40:03.089504+0000","last_active":"2024-09-13T14:40:40.619968+0000","last_peered":"2024-09-13T14:40:40.619968+0000","last_clean":"2024-09-13T14:40:40.619968+0000","last_became_active":"2024-09-13T14:40:03.089109+0000","last_became_peered":"2024-09-13T14:40:03.089109+0000","last_unstale":"2024-09-13T14:40:40.619968+0000","last_undegraded":"2024-09-13T14:40:40.619968+0000","last_fullsized":"2024-09-13T14:40:40.619968+0000","mapping_epoch":1089,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1090,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:36.259353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":1545,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:55.678989+0000","last_change":"2024-09-13T14:28:35.612469+0000","last_active":"2024-09-13T14:40:55.678989+0000","last_peered":"2024-09-13T14:40:55.678989+0000","last_clean":"2024-09-13T14:40:55.678989+0000","last_became_active":"2024-09-13T14:28:35.612093+0000","last_became_peered":"2024-09-13T14:28:35.612093+0000","last_unstale":"2024-09-13T14:40:55.678989+0000","last_undegraded":"2024-09-13T14:40:55.678989+0000","last_fullsized":"2024-09-13T14:40:55.678989+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:51.022663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1044,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.623980+0000","last_change":"2024-09-13T14:30:15.580568+0000","last_active":"2024-09-13T14:40:40.623980+0000","last_peered":"2024-09-13T14:40:40.623980+0000","last_clean":"2024-09-13T14:40:40.623980+0000","last_became_active":"2024-09-13T14:30:15.580186+0000","last_became_peered":"2024-09-13T14:30:15.580186+0000","last_unstale":"2024-09-13T14:40:40.623980+0000","last_undegraded":"2024-09-13T14:40:40.623980+0000","last_fullsized":"2024-09-13T14:40:40.623980+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:16:00.781680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":1039,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.637522+0000","last_change":"2024-09-13T14:29:08.912819+0000","last_active":"2024-09-13T14:40:40.637522+0000","last_peered":"2024-09-13T14:40:40.637522+0000","last_clean":"2024-09-13T14:40:40.637522+0000","last_became_active":"2024-09-13T14:29:08.912464+0000","last_became_peered":"2024-09-13T14:29:08.912464+0000","last_unstale":"2024-09-13T14:40:40.637522+0000","last_undegraded":"2024-09-13T14:40:40.637522+0000","last_fullsized":"2024-09-13T14:40:40.637522+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:09.753940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1051,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.626577+0000","last_change":"2024-09-13T14:30:15.580011+0000","last_active":"2024-09-13T14:40:40.626577+0000","last_peered":"2024-09-13T14:40:40.626577+0000","last_clean":"2024-09-13T14:40:40.626577+0000","last_became_active":"2024-09-13T14:30:15.579610+0000","last_became_peered":"2024-09-13T14:30:15.579610+0000","last_unstale":"2024-09-13T14:40:40.626577+0000","last_undegraded":"2024-09-13T14:40:40.626577+0000","last_fullsized":"2024-09-13T14:40:40.626577+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:11:02.733771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":1022,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.637807+0000","last_change":"2024-09-13T14:19:25.873514+0000","last_active":"2024-09-13T14:40:40.637807+0000","last_peered":"2024-09-13T14:40:40.637807+0000","last_clean":"2024-09-13T14:40:40.637807+0000","last_became_active":"2024-09-13T14:19:25.873170+0000","last_became_peered":"2024-09-13T14:19:25.873170+0000","last_unstale":"2024-09-13T14:40:40.637807+0000","last_undegraded":"2024-09-13T14:40:40.637807+0000","last_fullsized":"2024-09-13T14:40:40.637807+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:48.298530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":620,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.622493+0000","last_change":"2024-09-13T14:39:11.396732+0000","last_active":"2024-09-13T14:40:40.622493+0000","last_peered":"2024-09-13T14:40:40.622493+0000","last_clean":"2024-09-13T14:40:40.622493+0000","last_became_active":"2024-09-13T14:39:11.396322+0000","last_became_peered":"2024-09-13T14:39:11.396322+0000","last_unstale":"2024-09-13T14:40:40.622493+0000","last_undegraded":"2024-09-13T14:40:40.622493+0000","last_fullsized":"2024-09-13T14:40:40.622493+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:54:47.496762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":46,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.632155+0000","last_change":"2024-09-13T14:39:11.386728+0000","last_active":"2024-09-13T14:40:40.632155+0000","last_peered":"2024-09-13T14:40:40.632155+0000","last_clean":"2024-09-13T14:40:40.632155+0000","last_became_active":"2024-09-13T14:39:11.386372+0000","last_became_peered":"2024-09-13T14:39:11.386372+0000","last_unstale":"2024-09-13T14:40:40.632155+0000","last_undegraded":"2024-09-13T14:40:40.632155+0000","last_fullsized":"2024-09-13T14:40:40.632155+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:17:38.290895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1044,"reported_epoch":1093,"state":"active+clean","last_fresh":"2024-09-13T14:40:40.623311+0000","last_change":"2024-09-13T14:29:42.539907+0000","last_active":"2024-09-13T14:40:40.623311+0000","last_peered":"2024-09-13T14:40:40.623311+0000","last_clean":"2024-09-13T14:40:40.623311+0000","last_became_active":"2024-09-13T14:29:42.539583+0000","last_became_peered":"2024-09-13T14:29:42.539583+0000","last_unstale":"2024-09-13T14:40:40.623311+0000","last_undegraded":"2024-09-13T14:40:40.623311+0000","last_fullsized":"2024-09-13T14:40:40.623311+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:02:04.628447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":1560,"reported_epoch":1094,"state":"active+clean","last_fresh":"2024-09-13T14:40:55.679517+0000","last_change":"2024-09-13T14:29:42.541263+0000","last_active":"2024-09-13T14:40:55.679517+0000","last_peered":"2024-09-13T14:40:55.679517+0000","last_clean":"2024-09-13T14:40:55.679517+0000","last_became_active":"2024-09-13T14:29:42.540873+0000","last_became_peered":"2024-09-13T14:29:42.540873+0000","last_unstale":"2024-09-13T14:40:55.679517+0000","last_undegraded":"2024-09-13T14:40:55.679517+0000","last_fullsized":"2024-09-13T14:40:55.679517+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:20.611206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":8654749720,"num_objects":132062,"num_object_clones":0,"num_object_copies":396186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132062,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132592,"num_write_kb":8468865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":25946824704,"data_stored":25946816560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126292,"ondisk_log_size":126292,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430491,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4005840,"kb_used_data":3708484,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":89739312,"statfs":{"total":95995035648,"available":91893055488,"internally_reserved":0,"allocated":3797487616,"data_stored":3794294086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626720,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5596804,"kb_used_data":5299452,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":88148348,"statfs":{"total":95995035648,"available":90263908352,"internally_reserved":0,"allocated":5426638848,"data_stored":5423453391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822949,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3521788,"kb_used_data":3211952,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":90223364,"statfs":{"total":95995035648,"available":92388724736,"internally_reserved":0,"allocated":3289038848,"data_stored":3285860953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019177,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2952968,"kb_used_data":2647292,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":90792184,"statfs":{"total":95995035648,"available":92971196416,"internally_reserved":0,"allocated":2710827008,"data_stored":2707641056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059592835,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1655536,"kb_used_data":1583400,"kb_used_omap":26,"kb_used_meta":72101,"kb_avail":92089616,"statfs":{"total":95995035648,"available":94299766784,"internally_reserved":0,"allocated":1621401600,"data_stored":1618231348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":73831584},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":591,"seq":2538325672073,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4274792,"kb_used_data":4170528,"kb_used_omap":41,"kb_used_meta":104214,"kb_avail":89470360,"statfs":{"total":95995035648,"available":91617648640,"internally_reserved":0,"allocated":4270620672,"data_stored":4267450745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42651,"internal_metadata":106715493},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":557,"seq":2392296784016,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2743088,"kb_used_data":2652776,"kb_used_omap":32,"kb_used_meta":90271,"kb_avail":91002064,"statfs":{"total":95995035648,"available":93186113536,"internally_reserved":0,"allocated":2716442624,"data_stored":2713272317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33752,"internal_metadata":92437544},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":524,"seq":2250562863255,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2214972,"kb_used_data":2128820,"kb_used_omap":41,"kb_used_meta":86102,"kb_avail":91530180,"statfs":{"total":95995035648,"available":93726904320,"internally_reserved":0,"allocated":2179911680,"data_stored":2176730234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42694,"internal_metadata":88168762},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2153385984,"data_stored":2153381912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2710306816,"data_stored":2710306816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4279631872,"data_stored":4279631872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1610088448,"data_stored":1610088448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2709716992,"data_stored":2709716992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3279163392,"data_stored":3279159320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5417664512,"data_stored":5417664512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3786866688,"data_stored":3786866688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:41:03.964 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-13T14:41:03.965 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-09-13T14:41:03.985 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:03.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:03.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:04.087 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:04.127 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:04.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:04.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:04.229 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:04.268 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:04.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:04.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:04.370 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:04.411 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:04.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:04.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:04.513 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:04.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:04.560 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:04.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:04.643 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:04 smithi151 ceph-mon[104450]: pgmap v1349: 89 pgs: 89 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 102 B/s rd, 34 MiB/s wr, 938 op/s 2024-09-13T14:41:04.643 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:04 smithi151 ceph-mon[108966]: pgmap v1349: 89 pgs: 89 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 102 B/s rd, 34 MiB/s wr, 938 op/s 2024-09-13T14:41:04.643 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:41:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:41:04] "GET /metrics HTTP/1.1" 200 36664 "" "Prometheus/2.51.0" 2024-09-13T14:41:04.662 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:04.716 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:04.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:04.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:04.818 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:04.850 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:04.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:04.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:04.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:04 smithi184 ceph-mon[92527]: pgmap v1349: 89 pgs: 89 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 102 B/s rd, 34 MiB/s wr, 938 op/s 2024-09-13T14:41:04.951 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:05.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:05.003 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:05.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:05.104 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:05.141 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:05.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:05.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:05.243 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:05.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:05.285 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:05.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:05.386 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:05.423 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:05.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:05.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:05.525 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:05.560 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:05.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:05.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:05.640 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:41:05.640 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-13T14:41:05.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-13T14:41:05.641 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-13T14:41:05.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.641 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:9299 6.b 6:d767f9b5:::benchmark_data_smithi151_155836_object9298:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1086)", 2024-09-13T14:41:05.641 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:39:29.790869+0000", 2024-09-13T14:41:05.641 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 95.846037924000001, 2024-09-13T14:41:05.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.040077798999999997, 2024-09-13T14:41:05.641 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:41:05.641 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:41:05.641 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:05.641 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:41:05.642 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:41:05.642 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:05.642 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 9299 2024-09-13T14:41:05.642 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.642 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:41:05.642 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.642 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:41:05.642 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:29.790869+0000", 2024-09-13T14:41:05.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.642 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.642 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.642 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:41:05.642 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:29.790869+0000", 2024-09-13T14:41:05.643 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.643 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.643 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.643 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:41:05.643 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:29.790870+0000", 2024-09-13T14:41:05.643 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.474e-06 2024-09-13T14:41:05.643 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.643 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.643 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:41:05.643 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:29.790896+0000", 2024-09-13T14:41:05.643 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5882999999999999e-05 2024-09-13T14:41:05.643 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.643 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.644 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:41:05.644 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:29.790906+0000", 2024-09-13T14:41:05.644 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7880000000000001e-06 2024-09-13T14:41:05.644 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.644 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.644 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:41:05.644 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:29.790912+0000", 2024-09-13T14:41:05.644 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6520000000000003e-06 2024-09-13T14:41:05.644 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.644 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.644 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:41:05.644 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:29.790959+0000", 2024-09-13T14:41:05.644 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7848e-05 2024-09-13T14:41:05.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.645 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.645 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:41:05.645 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:29.791069+0000", 2024-09-13T14:41:05.645 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109603 2024-09-13T14:41:05.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.645 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.645 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:41:05.645 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:29.791152+0000", 2024-09-13T14:41:05.645 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2785000000000001e-05 2024-09-13T14:41:05.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.645 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.645 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:41:05.646 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:29.791698+0000", 2024-09-13T14:41:05.646 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00054640199999999998 2024-09-13T14:41:05.646 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.646 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.646 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.646 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:29.793169+0000", 2024-09-13T14:41:05.646 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0014711139999999999 2024-09-13T14:41:05.646 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.646 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.647 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.647 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:29.830677+0000", 2024-09-13T14:41:05.647 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.037508125000000003 2024-09-13T14:41:05.647 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.647 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.647 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:41:05.647 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:29.830896+0000", 2024-09-13T14:41:05.647 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00021834600000000001 2024-09-13T14:41:05.647 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.647 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.647 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:41:05.647 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:29.830947+0000", 2024-09-13T14:41:05.647 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0779000000000003e-05 2024-09-13T14:41:05.648 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.648 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:41:05.648 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.648 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.648 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.648 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:56428 6.b 6:d7fb96e1:::benchmark_data_smithi151_155836_object56427:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1088)", 2024-09-13T14:41:05.648 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:39:51.004213+0000", 2024-09-13T14:41:05.648 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 74.632693345999996, 2024-09-13T14:41:05.648 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.043097837999999999, 2024-09-13T14:41:05.648 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:41:05.648 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:41:05.648 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:05.649 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:41:05.649 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:41:05.649 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:05.649 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 56428 2024-09-13T14:41:05.649 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.649 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:41:05.649 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.649 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:41:05.649 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.004213+0000", 2024-09-13T14:41:05.649 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.649 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.649 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.649 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:41:05.649 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.004213+0000", 2024-09-13T14:41:05.650 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.650 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.650 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:41:05.650 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.004215+0000", 2024-09-13T14:41:05.650 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.531e-06 2024-09-13T14:41:05.650 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.650 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:41:05.650 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.004237+0000", 2024-09-13T14:41:05.650 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.213e-05 2024-09-13T14:41:05.650 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:41:05.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.004251+0000", 2024-09-13T14:41:05.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4178999999999999e-05 2024-09-13T14:41:05.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.651 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:41:05.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.004259+0000", 2024-09-13T14:41:05.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3100000000000001e-06 2024-09-13T14:41:05.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.651 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:41:05.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.004317+0000", 2024-09-13T14:41:05.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7068999999999999e-05 2024-09-13T14:41:05.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.652 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:41:05.652 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.004452+0000", 2024-09-13T14:41:05.652 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000135854 2024-09-13T14:41:05.652 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.652 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:41:05.652 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.004580+0000", 2024-09-13T14:41:05.652 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012719900000000001 2024-09-13T14:41:05.652 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.652 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:41:05.652 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.006860+0000", 2024-09-13T14:41:05.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002280565 2024-09-13T14:41:05.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.653 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.653 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.011764+0000", 2024-09-13T14:41:05.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0049038450000000004 2024-09-13T14:41:05.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.653 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.653 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.047207+0000", 2024-09-13T14:41:05.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.035442555000000001 2024-09-13T14:41:05.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:41:05.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.047265+0000", 2024-09-13T14:41:05.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8755000000000002e-05 2024-09-13T14:41:05.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:41:05.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.047311+0000", 2024-09-13T14:41:05.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5846000000000001e-05 2024-09-13T14:41:05.654 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.654 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:41:05.654 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.655 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:65146 6.b 6:da598ffa:::benchmark_data_smithi151_155836_object65145:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1088)", 2024-09-13T14:41:05.655 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:39:56.584402+0000", 2024-09-13T14:41:05.655 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 69.052504404999993, 2024-09-13T14:41:05.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051793005000000003, 2024-09-13T14:41:05.655 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:41:05.655 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:41:05.655 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:05.655 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:41:05.655 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:41:05.655 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:05.655 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 65146 2024-09-13T14:41:05.655 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.655 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:41:05.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.656 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:41:05.656 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.584402+0000", 2024-09-13T14:41:05.656 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.656 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.656 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.656 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:41:05.656 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.584402+0000", 2024-09-13T14:41:05.656 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.656 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.656 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.656 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:41:05.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.584404+0000", 2024-09-13T14:41:05.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.404e-06 2024-09-13T14:41:05.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:41:05.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.584425+0000", 2024-09-13T14:41:05.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0856000000000002e-05 2024-09-13T14:41:05.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:41:05.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.584436+0000", 2024-09-13T14:41:05.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1204e-05 2024-09-13T14:41:05.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:41:05.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.584441+0000", 2024-09-13T14:41:05.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.699e-06 2024-09-13T14:41:05.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:41:05.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.584496+0000", 2024-09-13T14:41:05.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4144000000000002e-05 2024-09-13T14:41:05.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:41:05.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.584634+0000", 2024-09-13T14:41:05.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013802399999999999 2024-09-13T14:41:05.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:41:05.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.584770+0000", 2024-09-13T14:41:05.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013645500000000001 2024-09-13T14:41:05.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:41:05.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.586509+0000", 2024-09-13T14:41:05.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017391749999999999 2024-09-13T14:41:05.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.587446+0000", 2024-09-13T14:41:05.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00093641299999999998 2024-09-13T14:41:05.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.635989+0000", 2024-09-13T14:41:05.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048543676000000001 2024-09-13T14:41:05.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.661 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:41:05.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.636150+0000", 2024-09-13T14:41:05.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000160352 2024-09-13T14:41:05.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.661 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:41:05.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.636195+0000", 2024-09-13T14:41:05.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5602999999999997e-05 2024-09-13T14:41:05.661 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.661 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:41:05.661 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.662 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:72857 6.b 6:d272439f:::benchmark_data_smithi151_155836_object72856:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1088)", 2024-09-13T14:41:05.662 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:40:02.015771+0000", 2024-09-13T14:41:05.662 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 63.621135523, 2024-09-13T14:41:05.662 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.039151938999999997, 2024-09-13T14:41:05.662 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:41:05.662 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:41:05.662 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:05.662 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:41:05.662 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:41:05.662 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:05.662 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 72857 2024-09-13T14:41:05.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.663 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:41:05.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:41:05.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:02.015771+0000", 2024-09-13T14:41:05.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:41:05.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:02.015771+0000", 2024-09-13T14:41:05.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:41:05.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:02.015773+0000", 2024-09-13T14:41:05.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8679999999999999e-06 2024-09-13T14:41:05.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:41:05.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:02.015801+0000", 2024-09-13T14:41:05.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8238e-05 2024-09-13T14:41:05.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:41:05.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:02.015822+0000", 2024-09-13T14:41:05.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0808000000000001e-05 2024-09-13T14:41:05.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:41:05.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:02.015831+0000", 2024-09-13T14:41:05.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2539999999999998e-06 2024-09-13T14:41:05.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:41:05.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:02.028334+0000", 2024-09-13T14:41:05.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012502545 2024-09-13T14:41:05.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:41:05.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:02.028520+0000", 2024-09-13T14:41:05.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018637699999999999 2024-09-13T14:41:05.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.666 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:41:05.666 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:02.028987+0000", 2024-09-13T14:41:05.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000466985 2024-09-13T14:41:05.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.666 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.666 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:02.036014+0000", 2024-09-13T14:41:05.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0070262950000000001 2024-09-13T14:41:05.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.666 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:41:05.666 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:02.041985+0000", 2024-09-13T14:41:05.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0059717659999999999 2024-09-13T14:41:05.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:02.053719+0000", 2024-09-13T14:41:05.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011733664 2024-09-13T14:41:05.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:41:05.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:02.054633+0000", 2024-09-13T14:41:05.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00091384300000000001 2024-09-13T14:41:05.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:41:05.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:02.054923+0000", 2024-09-13T14:41:05.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000290296 2024-09-13T14:41:05.668 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.668 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:41:05.668 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.668 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:109563 6.b 6:d7beef57:::benchmark_data_smithi151_155836_object109562:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1092)", 2024-09-13T14:41:05.668 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:40:33.420925+0000", 2024-09-13T14:41:05.668 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 32.215981204000002, 2024-09-13T14:41:05.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050238774, 2024-09-13T14:41:05.668 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:41:05.668 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:41:05.669 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:05.669 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:41:05.669 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:41:05.669 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:05.669 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 109563 2024-09-13T14:41:05.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.669 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:41:05.669 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.669 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:41:05.669 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.420925+0000", 2024-09-13T14:41:05.669 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.669 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.670 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:41:05.670 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.420925+0000", 2024-09-13T14:41:05.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.670 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:05.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:41:05.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.420927+0000", 2024-09-13T14:41:05.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9199999999999998e-06 2024-09-13T14:41:05.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.672 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:41:05.672 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.420952+0000", 2024-09-13T14:41:05.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4555000000000001e-05 2024-09-13T14:41:05.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.672 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:41:05.672 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.420975+0000", 2024-09-13T14:41:05.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2674999999999999e-05 2024-09-13T14:41:05.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.673 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:41:05.673 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.420984+0000", 2024-09-13T14:41:05.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2730000000000008e-06 2024-09-13T14:41:05.673 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.673 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.673 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:41:05.673 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.421044+0000", 2024-09-13T14:41:05.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0291000000000003e-05 2024-09-13T14:41:05.673 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.673 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.673 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:41:05.673 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.421166+0000", 2024-09-13T14:41:05.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012221500000000001 2024-09-13T14:41:05.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.674 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:41:05.674 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.421287+0000", 2024-09-13T14:41:05.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012094700000000001 2024-09-13T14:41:05.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.674 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:41:05.674 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.424715+0000", 2024-09-13T14:41:05.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034279860000000001 2024-09-13T14:41:05.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.674 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.675 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.429770+0000", 2024-09-13T14:41:05.675 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0050546219999999999 2024-09-13T14:41:05.675 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.675 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.675 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.675 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.471025+0000", 2024-09-13T14:41:05.675 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041254636999999997 2024-09-13T14:41:05.675 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.675 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.675 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:41:05.675 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.471109+0000", 2024-09-13T14:41:05.675 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4530999999999994e-05 2024-09-13T14:41:05.675 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.675 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.676 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:41:05.676 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.471164+0000", 2024-09-13T14:41:05.676 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5121999999999998e-05 2024-09-13T14:41:05.676 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.676 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:41:05.676 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.676 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.676 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:109567 6.b 6:d1383ac2:::benchmark_data_smithi151_155836_object109566:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1092)", 2024-09-13T14:41:05.676 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:40:33.425433+0000", 2024-09-13T14:41:05.676 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 32.211474033999998, 2024-09-13T14:41:05.676 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.045888750999999998, 2024-09-13T14:41:05.677 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:41:05.677 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:41:05.677 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:05.677 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:41:05.677 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:41:05.677 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:05.677 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 109567 2024-09-13T14:41:05.677 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.677 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:41:05.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:41:05.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.425433+0000", 2024-09-13T14:41:05.678 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:41:05.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.425433+0000", 2024-09-13T14:41:05.678 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:41:05.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.425434+0000", 2024-09-13T14:41:05.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6449999999999999e-06 2024-09-13T14:41:05.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.679 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:41:05.679 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.425457+0000", 2024-09-13T14:41:05.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2663999999999998e-05 2024-09-13T14:41:05.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.679 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:41:05.679 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.425472+0000", 2024-09-13T14:41:05.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5211000000000001e-05 2024-09-13T14:41:05.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:41:05.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.425481+0000", 2024-09-13T14:41:05.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2390000000000008e-06 2024-09-13T14:41:05.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:41:05.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.425534+0000", 2024-09-13T14:41:05.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2429999999999998e-05 2024-09-13T14:41:05.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:41:05.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.425684+0000", 2024-09-13T14:41:05.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014991200000000001 2024-09-13T14:41:05.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.681 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:41:05.681 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.425845+0000", 2024-09-13T14:41:05.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000161383 2024-09-13T14:41:05.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.681 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:41:05.681 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.429812+0000", 2024-09-13T14:41:05.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039672279999999997 2024-09-13T14:41:05.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.682 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.682 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.429997+0000", 2024-09-13T14:41:05.682 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018453999999999999 2024-09-13T14:41:05.682 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.682 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.682 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.682 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.471225+0000", 2024-09-13T14:41:05.682 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041227660999999999 2024-09-13T14:41:05.682 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.682 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.682 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:41:05.682 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.471283+0000", 2024-09-13T14:41:05.682 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8094000000000003e-05 2024-09-13T14:41:05.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.683 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:41:05.683 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.471321+0000", 2024-09-13T14:41:05.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8744e-05 2024-09-13T14:41:05.683 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.683 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:41:05.683 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.683 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:109574 6.b 6:daba8bda:::benchmark_data_smithi151_155836_object109573:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1092)", 2024-09-13T14:41:05.683 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:40:33.432079+0000", 2024-09-13T14:41:05.683 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 32.204827760000001, 2024-09-13T14:41:05.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.039442896999999998, 2024-09-13T14:41:05.684 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:41:05.684 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:41:05.684 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:05.684 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:41:05.684 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:41:05.684 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:05.684 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 109574 2024-09-13T14:41:05.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.684 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:41:05.684 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.684 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:41:05.684 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.432079+0000", 2024-09-13T14:41:05.684 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:41:05.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.432079+0000", 2024-09-13T14:41:05.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:41:05.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.432081+0000", 2024-09-13T14:41:05.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1119999999999999e-06 2024-09-13T14:41:05.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:41:05.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.432109+0000", 2024-09-13T14:41:05.686 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7637999999999999e-05 2024-09-13T14:41:05.686 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.686 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:41:05.686 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.432130+0000", 2024-09-13T14:41:05.686 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.101e-05 2024-09-13T14:41:05.686 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.686 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:41:05.686 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.432141+0000", 2024-09-13T14:41:05.686 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1634000000000001e-05 2024-09-13T14:41:05.686 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.687 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:41:05.687 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.432208+0000", 2024-09-13T14:41:05.687 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6754999999999994e-05 2024-09-13T14:41:05.687 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.687 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.687 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:41:05.687 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.432359+0000", 2024-09-13T14:41:05.687 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015126199999999999 2024-09-13T14:41:05.687 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.687 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.687 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:41:05.687 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.432525+0000", 2024-09-13T14:41:05.687 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016583 2024-09-13T14:41:05.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.688 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.688 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.688 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.436687+0000", 2024-09-13T14:41:05.688 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0041619509999999997 2024-09-13T14:41:05.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.688 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.688 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:41:05.688 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.436700+0000", 2024-09-13T14:41:05.688 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2639e-05 2024-09-13T14:41:05.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.688 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.688 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.471385+0000", 2024-09-13T14:41:05.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.034685584999999998 2024-09-13T14:41:05.689 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.689 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:41:05.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.471482+0000", 2024-09-13T14:41:05.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6390000000000004e-05 2024-09-13T14:41:05.689 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.689 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:41:05.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:33.471522+0000", 2024-09-13T14:41:05.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0091e-05 2024-09-13T14:41:05.689 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.689 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:41:05.690 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.690 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:110847 6.b 6:d575556f:::benchmark_data_smithi151_155836_object110846:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1092)", 2024-09-13T14:41:05.690 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:40:34.691459+0000", 2024-09-13T14:41:05.690 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 30.945447624, 2024-09-13T14:41:05.690 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.047066504000000002, 2024-09-13T14:41:05.690 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:41:05.690 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:41:05.690 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:05.690 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:41:05.690 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:41:05.690 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:05.691 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 110847 2024-09-13T14:41:05.691 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.691 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:41:05.691 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.691 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:41:05.691 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:34.691459+0000", 2024-09-13T14:41:05.691 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.691 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.691 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.691 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:41:05.691 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:34.691459+0000", 2024-09-13T14:41:05.691 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.691 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.692 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:41:05.692 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:34.691460+0000", 2024-09-13T14:41:05.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0079999999999999e-06 2024-09-13T14:41:05.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.692 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:41:05.692 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:34.691480+0000", 2024-09-13T14:41:05.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9632000000000001e-05 2024-09-13T14:41:05.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.692 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:41:05.692 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:34.691495+0000", 2024-09-13T14:41:05.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5566999999999999e-05 2024-09-13T14:41:05.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.693 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:41:05.693 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:34.691502+0000", 2024-09-13T14:41:05.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1450000000000002e-06 2024-09-13T14:41:05.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.693 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:41:05.693 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:34.717858+0000", 2024-09-13T14:41:05.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.026355771 2024-09-13T14:41:05.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.694 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:41:05.694 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:34.718013+0000", 2024-09-13T14:41:05.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015440300000000001 2024-09-13T14:41:05.694 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.694 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:41:05.694 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:34.718191+0000", 2024-09-13T14:41:05.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000178908 2024-09-13T14:41:05.694 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.694 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.694 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:34.726440+0000", 2024-09-13T14:41:05.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0082488990000000005 2024-09-13T14:41:05.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.695 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.695 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:41:05.695 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:34.727198+0000", 2024-09-13T14:41:05.695 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00075758200000000003 2024-09-13T14:41:05.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.695 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.695 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.695 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:34.738355+0000", 2024-09-13T14:41:05.695 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011157482999999999 2024-09-13T14:41:05.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.695 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.695 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:41:05.696 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:34.738472+0000", 2024-09-13T14:41:05.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011653 2024-09-13T14:41:05.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.696 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:41:05.696 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:34.738526+0000", 2024-09-13T14:41:05.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3575999999999997e-05 2024-09-13T14:41:05.696 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.696 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:41:05.696 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.696 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:129955 6.b 6:d3b13238:::benchmark_data_smithi151_155836_object129954:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1093)", 2024-09-13T14:41:05.697 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:40:54.535957+0000", 2024-09-13T14:41:05.697 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 11.100949527999999, 2024-09-13T14:41:05.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.040222465999999998, 2024-09-13T14:41:05.697 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:41:05.697 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:41:05.697 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:05.697 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:41:05.697 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:41:05.697 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:05.697 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 129955 2024-09-13T14:41:05.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.697 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:41:05.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:41:05.698 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:54.535957+0000", 2024-09-13T14:41:05.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.698 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.698 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:41:05.698 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:54.535957+0000", 2024-09-13T14:41:05.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.698 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.698 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:41:05.698 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:54.535958+0000", 2024-09-13T14:41:05.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9299999999999996e-07 2024-09-13T14:41:05.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.699 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:41:05.699 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:54.535978+0000", 2024-09-13T14:41:05.699 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0077000000000001e-05 2024-09-13T14:41:05.699 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.699 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:41:05.699 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:54.535997+0000", 2024-09-13T14:41:05.699 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8464000000000001e-05 2024-09-13T14:41:05.699 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.699 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:41:05.699 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:54.536003+0000", 2024-09-13T14:41:05.700 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.764e-06 2024-09-13T14:41:05.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.700 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.700 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:41:05.700 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:54.545939+0000", 2024-09-13T14:41:05.700 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0099354530000000003 2024-09-13T14:41:05.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.700 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.700 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:41:05.700 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:54.546086+0000", 2024-09-13T14:41:05.700 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000146787 2024-09-13T14:41:05.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.700 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.701 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:41:05.701 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:54.546231+0000", 2024-09-13T14:41:05.701 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014592300000000001 2024-09-13T14:41:05.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.701 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:41:05.701 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:54.558664+0000", 2024-09-13T14:41:05.701 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012432812 2024-09-13T14:41:05.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.701 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.701 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:54.561304+0000", 2024-09-13T14:41:05.701 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026399190000000001 2024-09-13T14:41:05.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.702 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.702 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:54.576025+0000", 2024-09-13T14:41:05.702 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014721052 2024-09-13T14:41:05.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.702 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:41:05.702 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:54.576118+0000", 2024-09-13T14:41:05.702 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2878000000000001e-05 2024-09-13T14:41:05.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.702 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:41:05.702 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:54.576180+0000", 2024-09-13T14:41:05.703 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1444000000000003e-05 2024-09-13T14:41:05.703 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.703 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:41:05.703 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.703 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.703 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.703 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:133196 6.b 6:d95abe7c:::benchmark_data_smithi151_155836_object738:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1094)", 2024-09-13T14:41:05.703 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:40:58.049132+0000", 2024-09-13T14:41:05.703 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.5877747979999999, 2024-09-13T14:41:05.703 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048800015000000002, 2024-09-13T14:41:05.703 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:41:05.703 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:41:05.703 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:05.704 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:41:05.704 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:41:05.704 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:05.704 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 133196 2024-09-13T14:41:05.704 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.704 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:41:05.704 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.704 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:41:05.704 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.049132+0000", 2024-09-13T14:41:05.704 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.704 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.704 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.704 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:41:05.705 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.049132+0000", 2024-09-13T14:41:05.705 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.705 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.705 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:41:05.705 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.049133+0000", 2024-09-13T14:41:05.705 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8000000000000004e-07 2024-09-13T14:41:05.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.705 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.705 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:41:05.705 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.049137+0000", 2024-09-13T14:41:05.705 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.856e-06 2024-09-13T14:41:05.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.706 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.706 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:41:05.706 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.049152+0000", 2024-09-13T14:41:05.706 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5043e-05 2024-09-13T14:41:05.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.706 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.706 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:41:05.706 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.049159+0000", 2024-09-13T14:41:05.706 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2790000000000001e-06 2024-09-13T14:41:05.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.706 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.706 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:41:05.707 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.051851+0000", 2024-09-13T14:41:05.707 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026925270000000001 2024-09-13T14:41:05.707 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.707 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.707 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:41:05.707 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.051976+0000", 2024-09-13T14:41:05.707 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000124413 2024-09-13T14:41:05.707 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.707 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.707 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:41:05.707 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.052139+0000", 2024-09-13T14:41:05.707 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016277500000000001 2024-09-13T14:41:05.707 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.707 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.708 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:41:05.708 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.055267+0000", 2024-09-13T14:41:05.708 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031283719999999999 2024-09-13T14:41:05.708 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.708 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.708 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.708 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.055480+0000", 2024-09-13T14:41:05.708 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00021278900000000001 2024-09-13T14:41:05.708 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.708 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.708 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.708 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.097814+0000", 2024-09-13T14:41:05.708 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.042333920999999997 2024-09-13T14:41:05.709 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.709 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:41:05.709 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.097882+0000", 2024-09-13T14:41:05.709 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8386999999999994e-05 2024-09-13T14:41:05.709 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.709 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:41:05.709 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.097932+0000", 2024-09-13T14:41:05.709 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9772999999999999e-05 2024-09-13T14:41:05.709 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.709 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:41:05.709 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.710 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.710 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:133685 6.b 6:d81d023f:::benchmark_data_smithi151_155836_object1227:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1094)", 2024-09-13T14:41:05.710 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:40:58.549851+0000", 2024-09-13T14:41:05.710 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.087056112, 2024-09-13T14:41:05.710 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038321256999999997, 2024-09-13T14:41:05.710 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:41:05.710 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:41:05.710 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:05.710 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:41:05.710 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:41:05.710 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:05.710 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 133685 2024-09-13T14:41:05.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.711 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:41:05.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.711 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:41:05.711 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.549851+0000", 2024-09-13T14:41:05.711 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.711 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:41:05.711 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.549851+0000", 2024-09-13T14:41:05.711 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.711 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:41:05.712 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.549852+0000", 2024-09-13T14:41:05.712 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9360000000000002e-06 2024-09-13T14:41:05.712 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.712 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.712 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:41:05.712 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.549859+0000", 2024-09-13T14:41:05.712 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0430000000000004e-06 2024-09-13T14:41:05.712 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.712 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.712 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:41:05.712 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.549872+0000", 2024-09-13T14:41:05.712 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3777e-05 2024-09-13T14:41:05.713 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.713 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.713 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:41:05.713 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.549880+0000", 2024-09-13T14:41:05.713 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7109999999999995e-06 2024-09-13T14:41:05.713 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.713 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.713 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:41:05.713 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.549934+0000", 2024-09-13T14:41:05.713 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3964e-05 2024-09-13T14:41:05.713 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.713 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.713 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:41:05.714 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.550038+0000", 2024-09-13T14:41:05.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000104134 2024-09-13T14:41:05.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.714 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:41:05.714 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.550106+0000", 2024-09-13T14:41:05.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8109999999999997e-05 2024-09-13T14:41:05.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.714 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:41:05.714 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.554184+0000", 2024-09-13T14:41:05.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0040775589999999997 2024-09-13T14:41:05.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.715 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.554365+0000", 2024-09-13T14:41:05.716 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018129600000000001 2024-09-13T14:41:05.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.716 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.716 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.588065+0000", 2024-09-13T14:41:05.716 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.033700103000000002 2024-09-13T14:41:05.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.717 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:41:05.717 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.588121+0000", 2024-09-13T14:41:05.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6165000000000002e-05 2024-09-13T14:41:05.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.717 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:41:05.717 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.588172+0000", 2024-09-13T14:41:05.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0458999999999999e-05 2024-09-13T14:41:05.717 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.717 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:41:05.717 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.718 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.718 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:133892 6.b 6:db20dd85:::benchmark_data_smithi151_155836_object1434:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1094)", 2024-09-13T14:41:05.718 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:40:58.769936+0000", 2024-09-13T14:41:05.718 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 6.8669701669999998, 2024-09-13T14:41:05.718 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.039007721000000002, 2024-09-13T14:41:05.718 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:41:05.718 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:41:05.718 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:05.718 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:41:05.718 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:41:05.718 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:05.718 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 133892 2024-09-13T14:41:05.718 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.719 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:41:05.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:41:05.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.769936+0000", 2024-09-13T14:41:05.719 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:41:05.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.769936+0000", 2024-09-13T14:41:05.719 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:41:05.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.769937+0000", 2024-09-13T14:41:05.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3599999999999999e-07 2024-09-13T14:41:05.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:41:05.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.769939+0000", 2024-09-13T14:41:05.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5050000000000002e-06 2024-09-13T14:41:05.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:41:05.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.769945+0000", 2024-09-13T14:41:05.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9490000000000001e-06 2024-09-13T14:41:05.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.721 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:41:05.721 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.769950+0000", 2024-09-13T14:41:05.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5900000000000001e-06 2024-09-13T14:41:05.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.721 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:41:05.721 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.780978+0000", 2024-09-13T14:41:05.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011028171 2024-09-13T14:41:05.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.721 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:41:05.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.781302+0000", 2024-09-13T14:41:05.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00032345900000000001 2024-09-13T14:41:05.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:41:05.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.781693+0000", 2024-09-13T14:41:05.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039166899999999998 2024-09-13T14:41:05.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.794144+0000", 2024-09-13T14:41:05.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012450348999999999 2024-09-13T14:41:05.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:41:05.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.794283+0000", 2024-09-13T14:41:05.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000138906 2024-09-13T14:41:05.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.808548+0000", 2024-09-13T14:41:05.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014265332 2024-09-13T14:41:05.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:41:05.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.808781+0000", 2024-09-13T14:41:05.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000233632 2024-09-13T14:41:05.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.724 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:41:05.724 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:58.808944+0000", 2024-09-13T14:41:05.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016272300000000001 2024-09-13T14:41:05.724 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.724 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:41:05.724 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.724 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:135219 6.b 6:d06c03c7:::benchmark_data_smithi151_155836_object2761:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1094)", 2024-09-13T14:41:05.725 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:41:00.184956+0000", 2024-09-13T14:41:05.725 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.4519508569999999, 2024-09-13T14:41:05.725 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.040986881000000003, 2024-09-13T14:41:05.725 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:41:05.725 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:41:05.725 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:05.725 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:41:05.726 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:41:05.726 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:05.726 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 135219 2024-09-13T14:41:05.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.726 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:41:05.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:41:05.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.184956+0000", 2024-09-13T14:41:05.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:41:05.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.184956+0000", 2024-09-13T14:41:05.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:41:05.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.184957+0000", 2024-09-13T14:41:05.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.361e-06 2024-09-13T14:41:05.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:41:05.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.184962+0000", 2024-09-13T14:41:05.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.724e-06 2024-09-13T14:41:05.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:41:05.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.184978+0000", 2024-09-13T14:41:05.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5994e-05 2024-09-13T14:41:05.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:41:05.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.184985+0000", 2024-09-13T14:41:05.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3309999999999998e-06 2024-09-13T14:41:05.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:41:05.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.188064+0000", 2024-09-13T14:41:05.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030790129999999998 2024-09-13T14:41:05.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:41:05.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.188186+0000", 2024-09-13T14:41:05.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000122087 2024-09-13T14:41:05.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:41:05.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.188300+0000", 2024-09-13T14:41:05.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000113534 2024-09-13T14:41:05.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:41:05.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.201234+0000", 2024-09-13T14:41:05.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012934599 2024-09-13T14:41:05.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.201836+0000", 2024-09-13T14:41:05.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00060119000000000004 2024-09-13T14:41:05.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.225787+0000", 2024-09-13T14:41:05.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.023951801000000002 2024-09-13T14:41:05.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:41:05.731 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.225893+0000", 2024-09-13T14:41:05.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000106028 2024-09-13T14:41:05.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:41:05.731 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.225943+0000", 2024-09-13T14:41:05.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9218999999999998e-05 2024-09-13T14:41:05.731 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.731 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:41:05.731 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.731 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:135220 6.b 6:d852a3de:::benchmark_data_smithi151_155836_object2762:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1094)", 2024-09-13T14:41:05.731 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:41:00.186733+0000", 2024-09-13T14:41:05.732 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.450173339, 2024-09-13T14:41:05.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.039337314999999998, 2024-09-13T14:41:05.732 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:41:05.732 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:41:05.732 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:05.732 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:41:05.732 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:41:05.732 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:05.732 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 135220 2024-09-13T14:41:05.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.732 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:41:05.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:41:05.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.186733+0000", 2024-09-13T14:41:05.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:41:05.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.186733+0000", 2024-09-13T14:41:05.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:41:05.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.186734+0000", 2024-09-13T14:41:05.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1170000000000001e-06 2024-09-13T14:41:05.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:41:05.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.186739+0000", 2024-09-13T14:41:05.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9049999999999996e-06 2024-09-13T14:41:05.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:41:05.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.186755+0000", 2024-09-13T14:41:05.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5705999999999998e-05 2024-09-13T14:41:05.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:41:05.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.186762+0000", 2024-09-13T14:41:05.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1029999999999996e-06 2024-09-13T14:41:05.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:41:05.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.191923+0000", 2024-09-13T14:41:05.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0051607529999999997 2024-09-13T14:41:05.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:41:05.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.192033+0000", 2024-09-13T14:41:05.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000110338 2024-09-13T14:41:05.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.736 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:41:05.736 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.192295+0000", 2024-09-13T14:41:05.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00026174400000000001 2024-09-13T14:41:05.736 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.736 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.736 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:41:05.736 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.201547+0000", 2024-09-13T14:41:05.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0092520580000000005 2024-09-13T14:41:05.736 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.736 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.736 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.736 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.206243+0000", 2024-09-13T14:41:05.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0046959330000000002 2024-09-13T14:41:05.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.737 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.737 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.225987+0000", 2024-09-13T14:41:05.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019743765 2024-09-13T14:41:05.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.737 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:41:05.737 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.226039+0000", 2024-09-13T14:41:05.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2364000000000002e-05 2024-09-13T14:41:05.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:41:05.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.226071+0000", 2024-09-13T14:41:05.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1529000000000003e-05 2024-09-13T14:41:05.738 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.738 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:41:05.738 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.738 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:135525 6.b 6:dbec0ee2:::benchmark_data_smithi151_155836_object3067:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1094)", 2024-09-13T14:41:05.738 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:41:00.498258+0000", 2024-09-13T14:41:05.738 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.1386486710000003, 2024-09-13T14:41:05.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041387097999999997, 2024-09-13T14:41:05.738 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:41:05.739 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:41:05.739 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:05.739 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:41:05.739 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:41:05.739 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:05.739 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 135525 2024-09-13T14:41:05.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.739 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:41:05.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:41:05.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.498258+0000", 2024-09-13T14:41:05.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:41:05.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.498258+0000", 2024-09-13T14:41:05.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:41:05.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.498259+0000", 2024-09-13T14:41:05.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6899999999999997e-07 2024-09-13T14:41:05.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:41:05.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.498261+0000", 2024-09-13T14:41:05.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.678e-06 2024-09-13T14:41:05.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:41:05.741 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.498268+0000", 2024-09-13T14:41:05.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1409999999999999e-06 2024-09-13T14:41:05.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:41:05.741 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.498274+0000", 2024-09-13T14:41:05.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1020000000000002e-06 2024-09-13T14:41:05.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:41:05.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.509108+0000", 2024-09-13T14:41:05.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010833894 2024-09-13T14:41:05.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:41:05.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.509212+0000", 2024-09-13T14:41:05.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010357900000000001 2024-09-13T14:41:05.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:41:05.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.509321+0000", 2024-09-13T14:41:05.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108842 2024-09-13T14:41:05.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.743 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:41:05.743 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.516831+0000", 2024-09-13T14:41:05.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0075106080000000002 2024-09-13T14:41:05.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.743 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.743 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.517107+0000", 2024-09-13T14:41:05.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027590500000000001 2024-09-13T14:41:05.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.743 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.743 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.539551+0000", 2024-09-13T14:41:05.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.022443556 2024-09-13T14:41:05.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:41:05.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.539604+0000", 2024-09-13T14:41:05.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3015e-05 2024-09-13T14:41:05.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:41:05.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:00.539645+0000", 2024-09-13T14:41:05.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1208999999999998e-05 2024-09-13T14:41:05.744 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.744 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:41:05.744 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.745 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:136439 6.b 6:dddf9ffc:::benchmark_data_smithi151_155836_object3981:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1094)", 2024-09-13T14:41:05.745 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:41:01.491934+0000", 2024-09-13T14:41:05.745 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.1449724440000004, 2024-09-13T14:41:05.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.040502604999999997, 2024-09-13T14:41:05.745 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:41:05.745 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:41:05.745 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:05.745 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:41:05.745 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:41:05.745 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:05.745 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 136439 2024-09-13T14:41:05.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.746 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:41:05.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.746 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:41:05.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.491934+0000", 2024-09-13T14:41:05.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.746 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:41:05.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.491934+0000", 2024-09-13T14:41:05.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.746 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:41:05.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.491935+0000", 2024-09-13T14:41:05.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1800000000000005e-07 2024-09-13T14:41:05.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:41:05.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.491939+0000", 2024-09-13T14:41:05.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9299999999999996e-06 2024-09-13T14:41:05.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:41:05.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.491961+0000", 2024-09-13T14:41:05.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2347000000000001e-05 2024-09-13T14:41:05.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:41:05.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.491968+0000", 2024-09-13T14:41:05.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8109999999999997e-06 2024-09-13T14:41:05.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:41:05.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.496597+0000", 2024-09-13T14:41:05.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0046287180000000004 2024-09-13T14:41:05.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:41:05.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.496721+0000", 2024-09-13T14:41:05.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012387300000000001 2024-09-13T14:41:05.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:41:05.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.496854+0000", 2024-09-13T14:41:05.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013316700000000001 2024-09-13T14:41:05.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:41:05.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.509406+0000", 2024-09-13T14:41:05.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012552368 2024-09-13T14:41:05.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.750 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.509443+0000", 2024-09-13T14:41:05.750 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6560000000000002e-05 2024-09-13T14:41:05.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.750 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.750 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.532329+0000", 2024-09-13T14:41:05.750 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.022886037000000001 2024-09-13T14:41:05.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.750 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:41:05.750 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.532388+0000", 2024-09-13T14:41:05.750 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9181999999999997e-05 2024-09-13T14:41:05.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:41:05.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.532437+0000", 2024-09-13T14:41:05.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8794000000000001e-05 2024-09-13T14:41:05.751 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.751 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:41:05.751 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.751 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:138906 6.b 6:d849a64d:::benchmark_data_smithi151_155836_object6448:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1094)", 2024-09-13T14:41:05.751 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:41:04.319728+0000", 2024-09-13T14:41:05.751 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.3171782730000001, 2024-09-13T14:41:05.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041311128000000003, 2024-09-13T14:41:05.752 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:41:05.752 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:41:05.752 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:05.752 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:41:05.752 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:41:05.752 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:05.752 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 138906 2024-09-13T14:41:05.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.752 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:41:05.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.752 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:41:05.752 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:04.319728+0000", 2024-09-13T14:41:05.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:41:05.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:04.319728+0000", 2024-09-13T14:41:05.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:41:05.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:04.319730+0000", 2024-09-13T14:41:05.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5570000000000001e-06 2024-09-13T14:41:05.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:41:05.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:04.319737+0000", 2024-09-13T14:41:05.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7229999999999998e-06 2024-09-13T14:41:05.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:41:05.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:04.319754+0000", 2024-09-13T14:41:05.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7414e-05 2024-09-13T14:41:05.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:41:05.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:04.319762+0000", 2024-09-13T14:41:05.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0919999999999998e-06 2024-09-13T14:41:05.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:41:05.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:04.326005+0000", 2024-09-13T14:41:05.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0062425759999999997 2024-09-13T14:41:05.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:41:05.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:04.326282+0000", 2024-09-13T14:41:05.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027695699999999998 2024-09-13T14:41:05.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:41:05.756 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:04.326593+0000", 2024-09-13T14:41:05.756 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00031081300000000001 2024-09-13T14:41:05.756 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.756 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.756 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:41:05.756 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:04.331502+0000", 2024-09-13T14:41:05.756 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0049090389999999996 2024-09-13T14:41:05.756 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.756 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.756 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.756 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:04.336537+0000", 2024-09-13T14:41:05.756 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.005035299 2024-09-13T14:41:05.756 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:04.360937+0000", 2024-09-13T14:41:05.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.024400607000000001 2024-09-13T14:41:05.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:41:05.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:04.360997+0000", 2024-09-13T14:41:05.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9598000000000001e-05 2024-09-13T14:41:05.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:41:05.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:04.361040+0000", 2024-09-13T14:41:05.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2453000000000002e-05 2024-09-13T14:41:05.758 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.758 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:41:05.758 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.758 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:139589 6.b 6:dc476c7c:::benchmark_data_smithi151_155836_object7131:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1094)", 2024-09-13T14:41:05.758 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:41:05.133643+0000", 2024-09-13T14:41:05.758 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.50326385200000001, 2024-09-13T14:41:05.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054202012000000001, 2024-09-13T14:41:05.758 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:41:05.758 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:41:05.758 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:05.759 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:41:05.759 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:41:05.759 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:05.759 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 139589 2024-09-13T14:41:05.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.759 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:41:05.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:41:05.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.133643+0000", 2024-09-13T14:41:05.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:41:05.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.133643+0000", 2024-09-13T14:41:05.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:41:05.760 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.133644+0000", 2024-09-13T14:41:05.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5009999999999999e-06 2024-09-13T14:41:05.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:41:05.760 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.133650+0000", 2024-09-13T14:41:05.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2220000000000001e-06 2024-09-13T14:41:05.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.761 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:41:05.761 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.133664+0000", 2024-09-13T14:41:05.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4258e-05 2024-09-13T14:41:05.761 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.761 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:41:05.761 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.133672+0000", 2024-09-13T14:41:05.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.807e-06 2024-09-13T14:41:05.761 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.761 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:41:05.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.133729+0000", 2024-09-13T14:41:05.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7504999999999999e-05 2024-09-13T14:41:05.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:41:05.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.134083+0000", 2024-09-13T14:41:05.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00035409999999999999 2024-09-13T14:41:05.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:41:05.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.134293+0000", 2024-09-13T14:41:05.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00020942199999999999 2024-09-13T14:41:05.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.763 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.763 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.140482+0000", 2024-09-13T14:41:05.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0061896219999999997 2024-09-13T14:41:05.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.763 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:41:05.763 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.140549+0000", 2024-09-13T14:41:05.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6324999999999999e-05 2024-09-13T14:41:05.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.763 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.763 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.187722+0000", 2024-09-13T14:41:05.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.047173723000000001 2024-09-13T14:41:05.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:41:05.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.187775+0000", 2024-09-13T14:41:05.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3148e-05 2024-09-13T14:41:05.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:41:05.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.187845+0000", 2024-09-13T14:41:05.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9379000000000001e-05 2024-09-13T14:41:05.764 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.764 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:41:05.764 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.765 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:139597 6.b 6:dbe20ba9:::benchmark_data_smithi151_155836_object7139:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1094)", 2024-09-13T14:41:05.765 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:41:05.142602+0000", 2024-09-13T14:41:05.765 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.49430497299999998, 2024-09-13T14:41:05.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.045390755999999997, 2024-09-13T14:41:05.765 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:41:05.765 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:41:05.765 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:05.765 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:41:05.765 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:41:05.765 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:05.765 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 139597 2024-09-13T14:41:05.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.766 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:41:05.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:41:05.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.142602+0000", 2024-09-13T14:41:05.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:41:05.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.142602+0000", 2024-09-13T14:41:05.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:41:05.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.142603+0000", 2024-09-13T14:41:05.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.719e-06 2024-09-13T14:41:05.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:41:05.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.142609+0000", 2024-09-13T14:41:05.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6080000000000003e-06 2024-09-13T14:41:05.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:41:05.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.142625+0000", 2024-09-13T14:41:05.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6314e-05 2024-09-13T14:41:05.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:41:05.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.142634+0000", 2024-09-13T14:41:05.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3420000000000008e-06 2024-09-13T14:41:05.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:41:05.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.142685+0000", 2024-09-13T14:41:05.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1604999999999998e-05 2024-09-13T14:41:05.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.769 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:41:05.769 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.142947+0000", 2024-09-13T14:41:05.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00026126200000000001 2024-09-13T14:41:05.769 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.769 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:41:05.769 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.143091+0000", 2024-09-13T14:41:05.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014411099999999999 2024-09-13T14:41:05.769 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.769 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.769 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.152966+0000", 2024-09-13T14:41:05.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0098755140000000002 2024-09-13T14:41:05.770 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.770 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:41:05.770 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.152979+0000", 2024-09-13T14:41:05.770 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2917e-05 2024-09-13T14:41:05.770 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.770 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.770 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.187915+0000", 2024-09-13T14:41:05.770 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.034936063000000003 2024-09-13T14:41:05.770 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.770 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:41:05.770 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.187961+0000", 2024-09-13T14:41:05.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5760999999999998e-05 2024-09-13T14:41:05.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:41:05.771 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.187992+0000", 2024-09-13T14:41:05.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.154e-05 2024-09-13T14:41:05.771 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.771 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:41:05.771 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.771 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:139602 6.b 6:d42cef03:::benchmark_data_smithi151_155836_object7144:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1094)", 2024-09-13T14:41:05.771 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:41:05.146325+0000", 2024-09-13T14:41:05.772 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.49058197399999998, 2024-09-13T14:41:05.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041810917000000003, 2024-09-13T14:41:05.772 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:41:05.772 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:41:05.772 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:05.772 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:41:05.772 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:41:05.772 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:05.772 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 139602 2024-09-13T14:41:05.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.772 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:41:05.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.772 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:41:05.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.146325+0000", 2024-09-13T14:41:05.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.773 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.773 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:41:05.773 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.146325+0000", 2024-09-13T14:41:05.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:41:05.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.773 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.773 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:41:05.773 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.146326+0000", 2024-09-13T14:41:05.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6190000000000001e-06 2024-09-13T14:41:05.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.773 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:41:05.774 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.146333+0000", 2024-09-13T14:41:05.774 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4459999999999998e-06 2024-09-13T14:41:05.774 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.774 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:41:05.774 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.146352+0000", 2024-09-13T14:41:05.774 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9703000000000001e-05 2024-09-13T14:41:05.774 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.774 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:41:05.774 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.146361+0000", 2024-09-13T14:41:05.775 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8410000000000006e-06 2024-09-13T14:41:05.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:41:05.775 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.147587+0000", 2024-09-13T14:41:05.775 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012257419999999999 2024-09-13T14:41:05.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:41:05.775 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.147887+0000", 2024-09-13T14:41:05.775 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00030021799999999999 2024-09-13T14:41:05.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.776 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:41:05.776 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.148232+0000", 2024-09-13T14:41:05.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00034436100000000001 2024-09-13T14:41:05.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.776 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.776 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:41:05.776 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.152999+0000", 2024-09-13T14:41:05.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0047672490000000003 2024-09-13T14:41:05.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.776 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.776 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.776 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.163893+0000", 2024-09-13T14:41:05.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010893943 2024-09-13T14:41:05.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.777 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:05.777 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.188061+0000", 2024-09-13T14:41:05.777 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.024168571999999999 2024-09-13T14:41:05.777 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.777 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:41:05.777 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.188105+0000", 2024-09-13T14:41:05.777 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3415999999999999e-05 2024-09-13T14:41:05.777 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:41:05.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:41:05.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:41:05.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:05.188136+0000", 2024-09-13T14:41:05.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0806999999999998e-05 2024-09-13T14:41:05.778 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.778 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:41:05.778 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.778 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:41:05.778 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:41:05.778 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:41:05.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:05.780 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:05.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:05.881 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:05.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:05.931 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:05.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:06.032 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:06.064 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:06.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:06.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:06.166 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:06.216 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:06.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:06.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:06.256 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:41:06.256 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:41:06.256 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:41:06.256 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:41:06.256 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:41:06.257 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:41:06.257 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:41:06.257 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:41:06.257 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:41:06.257 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:41:06.257 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:41:06.257 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:41:06.257 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:41:06.257 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:41:06.257 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:41:06.258 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:41:06.317 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:06.359 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:06.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:06.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:06.461 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:06.495 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:06.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:06.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:06.543 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:06 smithi184 ceph-mon[92527]: pgmap v1350: 89 pgs: 89 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 102 B/s rd, 28 MiB/s wr, 929 op/s 2024-09-13T14:41:06.543 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:06 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1206665194' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:06.597 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:06.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:06 smithi151 ceph-mon[104450]: pgmap v1350: 89 pgs: 89 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 102 B/s rd, 28 MiB/s wr, 929 op/s 2024-09-13T14:41:06.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:06 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1206665194' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:06.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:06 smithi151 ceph-mon[108966]: pgmap v1350: 89 pgs: 89 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 102 B/s rd, 28 MiB/s wr, 929 op/s 2024-09-13T14:41:06.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:06 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1206665194' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:06.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:06.651 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:06.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:06.752 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:06.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:06.785 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:06.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:06.886 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:06.950 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:06.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:06.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:07.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:07.089 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:07.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:07.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:07.191 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:07.214 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:07.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:07.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:07.316 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:07.352 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:07.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:07.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:07.454 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:07.488 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:07.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:07.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:07.590 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:07.626 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:07.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:07.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:07.728 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:07.758 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:07.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:07.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:07.860 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:07.890 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:07.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:07.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:07.992 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:08.019 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:08.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:08.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:08.121 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:08.148 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:08.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:08.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:08.191 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T14:41:08.250 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:08.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:08.302 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:08.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:08.403 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:08.429 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-13T14:41:08.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:08.473 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:08.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:08.575 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:08.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:08 smithi184 ceph-mon[92527]: pgmap v1351: 89 pgs: 89 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 87 B/s rd, 24 MiB/s wr, 1.03k op/s 2024-09-13T14:41:08.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:08 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/4167541419' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:08.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:08 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:08.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:08 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:41:08.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:08 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:41:08.649 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:41:08.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:08.650 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:08.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:08.751 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:08.790 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:08.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:08.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:08.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:08 smithi151 ceph-mon[104450]: pgmap v1351: 89 pgs: 89 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 87 B/s rd, 24 MiB/s wr, 1.03k op/s 2024-09-13T14:41:08.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:08 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/4167541419' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:08.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:08 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:08.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:08 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:41:08.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:08 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:41:08.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:08 smithi151 ceph-mon[108966]: pgmap v1351: 89 pgs: 89 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 87 B/s rd, 24 MiB/s wr, 1.03k op/s 2024-09-13T14:41:08.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:08 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/4167541419' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:08.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:08 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:08.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:08 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:41:08.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:08 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:41:08.892 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:08.933 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:08.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:08.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:09.035 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:09.069 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:09.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:09.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:09.171 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:09.212 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:09.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:09.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:09.314 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:09.343 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:41:09.352 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:09.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:09.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:09.454 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:09.493 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:09.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:09.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:09.588 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:09 smithi151 ceph-mon[104450]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:09.588 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:09 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:41:09.588 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:09 smithi151 ceph-mon[104450]: osdmap e1095: 8 total, 8 up, 8 in 2024-09-13T14:41:09.588 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:09 smithi151 ceph-mon[108966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:09.589 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:09 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:41:09.589 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:09 smithi151 ceph-mon[108966]: osdmap e1095: 8 total, 8 up, 8 in 2024-09-13T14:41:09.595 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:09.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:09.628 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:09.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:09.729 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:09.768 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:09.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:09.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:09.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:09 smithi184 ceph-mon[92527]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:09.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:09 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:41:09.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:09 smithi184 ceph-mon[92527]: osdmap e1095: 8 total, 8 up, 8 in 2024-09-13T14:41:09.870 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:09.904 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:09.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:09.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:10.005 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:10.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:10.039 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:10.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:10.140 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:10.165 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:10.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:10.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:10.232 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:10.267 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:10.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:10.322 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:10.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:10.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:41:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:41:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:41:10.423 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:10.460 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:10.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:10.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:10.535 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:10 smithi151 ceph-mon[104450]: pgmap v1353: 89 pgs: 89 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 825 KiB/s wr, 897 op/s 2024-09-13T14:41:10.535 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:10 smithi151 ceph-mon[108966]: pgmap v1353: 89 pgs: 89 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 825 KiB/s wr, 897 op/s 2024-09-13T14:41:10.562 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:10.596 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:10.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:10.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:10.698 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:10.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:10.735 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:10.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:10.744 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:10 smithi184 ceph-mon[92527]: pgmap v1353: 89 pgs: 89 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 825 KiB/s wr, 897 op/s 2024-09-13T14:41:10.836 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:10.862 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:10.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:10.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:10.964 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:10.994 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:10.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:10.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:11.092 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:11.096 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:11.144 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:11.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:11.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:11.199 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:41:11.199 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:41:11.246 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:11.282 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:11.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:11.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:11.384 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:11.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:11.425 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:11.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:11.526 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:11.573 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:11.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:11.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:11.674 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:11.707 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:11.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:11.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:11.808 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:11.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:11.856 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:11.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:11.957 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:11.984 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:11.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:11.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:12.086 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:12.120 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:12.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:12.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:12.223 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:12.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:12.257 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:12.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:12.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:41:12.358 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:12.387 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T14:41:12.402 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:12.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:12.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:12.504 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:12.540 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:12.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:12.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:12.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:12 smithi184 ceph-mon[92527]: pgmap v1354: 89 pgs: 89 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 262 KiB/s wr, 894 op/s 2024-09-13T14:41:12.642 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:12.667 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:12.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:12.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:12.769 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:12.828 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:12.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:12.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:12.852 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:41:12.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:12 smithi151 ceph-mon[104450]: pgmap v1354: 89 pgs: 89 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 262 KiB/s wr, 894 op/s 2024-09-13T14:41:12.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:12 smithi151 ceph-mon[108966]: pgmap v1354: 89 pgs: 89 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 262 KiB/s wr, 894 op/s 2024-09-13T14:41:12.931 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:12.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:12.959 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:12.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:13.060 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:13.088 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:13.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:13.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:13.190 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:13.231 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:13.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:13.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:13.333 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:13.369 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:13.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:13.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:13.471 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:13.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:13.510 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:13.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:13.611 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:13.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:13.652 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:13.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:13.753 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:13.792 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:13.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:13.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:13.894 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:13.896 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:13.933 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:13.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:13.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:14.035 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:14.064 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:14.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:14.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:14.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:14.195 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:14.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:14.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:14.297 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:14.326 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:14.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:14.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:14.427 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:14.458 INFO:teuthology.orchestra.run.smithi151.stderr:set pool 1 pg_num to 11 2024-09-13T14:41:14.462 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:14.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:14.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:14.564 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:14.604 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:14.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:14.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:14.706 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:14.737 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:14 smithi151 ceph-mon[104450]: pgmap v1355: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 868 op/s 2024-09-13T14:41:14.737 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:14 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2521162603' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-13T14:41:14.737 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:14 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-13T14:41:14.738 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:41:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:41:14] "GET /metrics HTTP/1.1" 200 36664 "" "Prometheus/2.51.0" 2024-09-13T14:41:14.738 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:14 smithi151 ceph-mon[108966]: pgmap v1355: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 868 op/s 2024-09-13T14:41:14.738 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:14 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2521162603' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-13T14:41:14.738 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:14 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-13T14:41:14.752 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:14.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:14.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:14.809 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:14 smithi184 ceph-mon[92527]: pgmap v1355: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 868 op/s 2024-09-13T14:41:14.809 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:14 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2521162603' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-13T14:41:14.809 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:14 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-13T14:41:14.855 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:14.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:14.900 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:14.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:15.001 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:15.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:15.029 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:15.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:15.130 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:15.175 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:15.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:15.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:15.280 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:15.308 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:15.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:15.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:15.322 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:41:15.322 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:41:15.410 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:15.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:15.441 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:15.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:15.542 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:15.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:15.590 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:15.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:15.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:15 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-13T14:41:15.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:15 smithi184 ceph-mon[92527]: osdmap e1096: 8 total, 8 up, 8 in 2024-09-13T14:41:15.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:15 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-13T14:41:15.691 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:15.697 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:41:15.698 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:41:15.716 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:15 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-13T14:41:15.716 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:15 smithi151 ceph-mon[104450]: osdmap e1096: 8 total, 8 up, 8 in 2024-09-13T14:41:15.716 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:15 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-13T14:41:15.716 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:15 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-13T14:41:15.716 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:15 smithi151 ceph-mon[108966]: osdmap e1096: 8 total, 8 up, 8 in 2024-09-13T14:41:15.717 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:15 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-13T14:41:15.717 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:41:15.717 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:41:15.727 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:15.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:15.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:15.738 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:41:15.739 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:41:15.762 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:41:15.762 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:41:15.792 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:41:15.792 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:41:15.827 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:41:15.827 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:41:15.828 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:15.859 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:15.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:15.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:15.860 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:41:15.861 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:41:15.891 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:41:15.891 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:41:15.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:15.996 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:15.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:15.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:16.075 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T14:41:16.098 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:16.179 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:16.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:16.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:16.281 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:16.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:16.316 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:16.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:16.418 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:16.481 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:16.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:16.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:16.535 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:41:16.583 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:16.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:16 smithi151 ceph-mon[104450]: pgmap v1357: 89 pgs: 89 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 751 op/s 2024-09-13T14:41:16.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-13T14:41:16.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:16 smithi151 ceph-mon[104450]: osdmap e1097: 8 total, 8 up, 8 in 2024-09-13T14:41:16.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:16 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1667043361' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:16.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:16.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:16 smithi151 ceph-mon[108966]: pgmap v1357: 89 pgs: 89 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 751 op/s 2024-09-13T14:41:16.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-13T14:41:16.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:16 smithi151 ceph-mon[108966]: osdmap e1097: 8 total, 8 up, 8 in 2024-09-13T14:41:16.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:16 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1667043361' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:16.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:16.612 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:16.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:16.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:16.714 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:16.728 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:16 smithi184 ceph-mon[92527]: pgmap v1357: 89 pgs: 89 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 751 op/s 2024-09-13T14:41:16.728 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-13T14:41:16.728 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:16 smithi184 ceph-mon[92527]: osdmap e1097: 8 total, 8 up, 8 in 2024-09-13T14:41:16.728 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:16 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1667043361' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:16.728 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:16.755 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:16.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:16.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:16.857 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:16.907 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:16.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:16.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:17.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:17.042 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:17.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:17.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:17.145 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:17.176 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:17.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:17.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:17.278 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:17.321 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:17.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:17.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:17.423 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:17.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:17.457 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:17.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:17.558 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:17.592 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:17.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:17.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:17.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:17 smithi151 ceph-mon[104450]: osdmap e1098: 8 total, 8 up, 8 in 2024-09-13T14:41:17.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:17 smithi151 ceph-mon[108966]: osdmap e1098: 8 total, 8 up, 8 in 2024-09-13T14:41:17.694 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:17.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:17.731 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:17.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:17.780 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:17 smithi184 ceph-mon[92527]: osdmap e1098: 8 total, 8 up, 8 in 2024-09-13T14:41:17.832 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:17.860 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:17.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:17.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:17.962 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:17.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:17.999 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:18.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:18.100 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:18.156 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:18.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:18.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:18.258 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:18.296 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:18.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:18.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:18.398 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:18.440 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:18.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:18.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:18.542 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:18.570 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:18.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:18.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:18.672 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:18.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:18.702 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:18.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:18.742 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:18 smithi184 ceph-mon[92527]: pgmap v1360: 99 pgs: 10 unknown, 89 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-09-13T14:41:18.791 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:41:18.792 INFO:teuthology.orchestra.run.smithi184.stdout: "size": 20, 2024-09-13T14:41:18.792 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 600, 2024-09-13T14:41:18.792 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [ 2024-09-13T14:41:18.792 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.792 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:56433 6.2 6:4bf1136f:::benchmark_data_smithi151_155836_object56432:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1088)", 2024-09-13T14:41:18.792 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:39:51.007613+0000", 2024-09-13T14:41:18.792 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 87.780757197, 2024-09-13T14:41:18.792 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.064793999000000005, 2024-09-13T14:41:18.792 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:18.792 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:18.792 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:18.793 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:18.793 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:18.793 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 56433 2024-09-13T14:41:18.793 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.793 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:18.793 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.793 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:18.793 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:51.007613+0000", 2024-09-13T14:41:18.793 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.793 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.793 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.793 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:18.793 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:51.007613+0000", 2024-09-13T14:41:18.794 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.794 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.794 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.794 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:18.794 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:51.007612+0000", 2024-09-13T14:41:18.794 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:41:18.794 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.794 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.794 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:18.794 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:51.007661+0000", 2024-09-13T14:41:18.794 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9452e-05 2024-09-13T14:41:18.794 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.794 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.795 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:18.795 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:51.007683+0000", 2024-09-13T14:41:18.795 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.2078000000000001e-05 2024-09-13T14:41:18.795 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.795 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.795 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:18.795 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:51.007689+0000", 2024-09-13T14:41:18.795 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4110000000000004e-06 2024-09-13T14:41:18.795 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.795 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.795 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:18.795 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:51.007882+0000", 2024-09-13T14:41:18.795 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000192991 2024-09-13T14:41:18.796 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.796 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.796 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:18.796 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:51.007999+0000", 2024-09-13T14:41:18.796 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000117409 2024-09-13T14:41:18.796 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.796 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.796 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:41:18.796 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:51.008137+0000", 2024-09-13T14:41:18.796 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013816699999999999 2024-09-13T14:41:18.796 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.796 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.796 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:18.797 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:51.012265+0000", 2024-09-13T14:41:18.797 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0041271520000000003 2024-09-13T14:41:18.797 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.797 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.797 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.797 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:51.014224+0000", 2024-09-13T14:41:18.797 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0019593010000000001 2024-09-13T14:41:18.797 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.797 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.797 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.797 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:51.072321+0000", 2024-09-13T14:41:18.797 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.058096684000000003 2024-09-13T14:41:18.797 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.797 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.798 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:18.798 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:51.072372+0000", 2024-09-13T14:41:18.798 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.181e-05 2024-09-13T14:41:18.798 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.798 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.798 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:18.798 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:51.072407+0000", 2024-09-13T14:41:18.798 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.4622000000000002e-05 2024-09-13T14:41:18.798 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.798 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:18.798 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.798 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.798 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.799 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:56435 6.2 6:459af026:::benchmark_data_smithi151_155836_object56434:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1088)", 2024-09-13T14:41:18.799 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:39:51.007722+0000", 2024-09-13T14:41:18.799 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 87.780648169000003, 2024-09-13T14:41:18.799 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.068166751999999997, 2024-09-13T14:41:18.799 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:18.799 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:18.799 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:18.799 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:18.799 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:18.799 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 56435 2024-09-13T14:41:18.799 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.799 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:18.799 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.799 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:18.800 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:51.007722+0000", 2024-09-13T14:41:18.800 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.800 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.800 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.800 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:18.800 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:51.007722+0000", 2024-09-13T14:41:18.800 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.800 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.800 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.800 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:18.800 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:51.007722+0000", 2024-09-13T14:41:18.800 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:41:18.800 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.801 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.801 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:18.801 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:51.007757+0000", 2024-09-13T14:41:18.801 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.5762000000000001e-05 2024-09-13T14:41:18.801 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.801 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.801 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:18.801 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:51.007766+0000", 2024-09-13T14:41:18.801 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.8729999999999996e-06 2024-09-13T14:41:18.801 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.801 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.801 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:18.802 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:51.007771+0000", 2024-09-13T14:41:18.802 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.6149999999999999e-06 2024-09-13T14:41:18.802 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.802 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.802 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:18.802 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:51.010084+0000", 2024-09-13T14:41:18.802 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0023128039999999999 2024-09-13T14:41:18.802 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.802 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.802 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:18.802 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:51.010253+0000", 2024-09-13T14:41:18.802 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000169483 2024-09-13T14:41:18.802 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.802 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.803 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:41:18.803 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:51.010371+0000", 2024-09-13T14:41:18.803 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000118335 2024-09-13T14:41:18.803 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.803 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.803 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:18.803 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:51.012282+0000", 2024-09-13T14:41:18.803 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001910213 2024-09-13T14:41:18.803 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.803 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.803 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.803 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:51.014360+0000", 2024-09-13T14:41:18.803 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0020783379999999999 2024-09-13T14:41:18.803 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.804 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.804 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.804 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:51.075778+0000", 2024-09-13T14:41:18.804 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.061418490999999999 2024-09-13T14:41:18.804 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.804 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.804 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:18.804 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:51.075831+0000", 2024-09-13T14:41:18.804 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.2225999999999998e-05 2024-09-13T14:41:18.804 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.804 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.804 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:18.804 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:51.075889+0000", 2024-09-13T14:41:18.804 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8134000000000003e-05 2024-09-13T14:41:18.805 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.805 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:18.805 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.805 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.805 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.805 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:65139 6.e 6:7f9acf57:::benchmark_data_smithi151_155836_object65138:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1088)", 2024-09-13T14:41:18.805 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:39:56.576045+0000", 2024-09-13T14:41:18.805 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 82.212325243999999, 2024-09-13T14:41:18.805 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.066525982999999997, 2024-09-13T14:41:18.805 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:18.806 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:18.806 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:18.806 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:18.806 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:18.807 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:18.807 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 65139 2024-09-13T14:41:18.807 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.807 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:18.808 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.808 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:18.808 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.576045+0000", 2024-09-13T14:41:18.808 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.808 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.808 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.808 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:18.808 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.576045+0000", 2024-09-13T14:41:18.808 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.808 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.808 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.808 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:18.808 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.576044+0000", 2024-09-13T14:41:18.809 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:41:18.809 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.809 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.809 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:18.809 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.576099+0000", 2024-09-13T14:41:18.809 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4194999999999999e-05 2024-09-13T14:41:18.809 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.809 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.809 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:18.809 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.576112+0000", 2024-09-13T14:41:18.809 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3084999999999999e-05 2024-09-13T14:41:18.809 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.809 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.809 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:18.810 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.576117+0000", 2024-09-13T14:41:18.810 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.87e-06 2024-09-13T14:41:18.810 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.810 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.810 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:18.810 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.581303+0000", 2024-09-13T14:41:18.810 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0051867969999999999 2024-09-13T14:41:18.810 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.810 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.810 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:18.810 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.581428+0000", 2024-09-13T14:41:18.810 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012504899999999999 2024-09-13T14:41:18.810 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.810 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.811 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:41:18.811 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.581569+0000", 2024-09-13T14:41:18.811 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014032500000000001 2024-09-13T14:41:18.811 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.811 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.811 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:18.811 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.585393+0000", 2024-09-13T14:41:18.811 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.003824685 2024-09-13T14:41:18.811 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.811 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.811 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.811 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.638011+0000", 2024-09-13T14:41:18.811 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.052617180999999999 2024-09-13T14:41:18.811 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.812 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.812 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.812 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.642502+0000", 2024-09-13T14:41:18.812 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0044914009999999999 2024-09-13T14:41:18.812 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.812 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.812 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:18.812 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.642541+0000", 2024-09-13T14:41:18.812 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8838000000000003e-05 2024-09-13T14:41:18.812 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.812 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.812 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:18.812 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.642571+0000", 2024-09-13T14:41:18.813 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.0130000000000001e-05 2024-09-13T14:41:18.813 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.813 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:18.813 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.813 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.813 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.813 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:65143 6.2 6:47dd78ac:::benchmark_data_smithi151_155836_object65142:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1088)", 2024-09-13T14:41:18.813 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:39:56.583746+0000", 2024-09-13T14:41:18.813 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 82.204623819999995, 2024-09-13T14:41:18.813 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.051947650999999997, 2024-09-13T14:41:18.813 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:18.813 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:18.813 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:18.814 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:18.814 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:18.814 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 65143 2024-09-13T14:41:18.814 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.814 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:18.814 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.814 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:18.814 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.583746+0000", 2024-09-13T14:41:18.814 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.814 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.814 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.814 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:18.814 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.583746+0000", 2024-09-13T14:41:18.814 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.815 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.815 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.815 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:18.815 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.583745+0000", 2024-09-13T14:41:18.815 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:18.815 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.815 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.815 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:18.815 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.583802+0000", 2024-09-13T14:41:18.815 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7355000000000002e-05 2024-09-13T14:41:18.815 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.815 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.815 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:18.815 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.583814+0000", 2024-09-13T14:41:18.816 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1246999999999999e-05 2024-09-13T14:41:18.816 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.816 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.816 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:18.816 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.583819+0000", 2024-09-13T14:41:18.816 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5820000000000001e-06 2024-09-13T14:41:18.816 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.816 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.816 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:18.816 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.583876+0000", 2024-09-13T14:41:18.816 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7082e-05 2024-09-13T14:41:18.816 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.816 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.817 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:18.817 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.583998+0000", 2024-09-13T14:41:18.817 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000121956 2024-09-13T14:41:18.817 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.817 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.817 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:41:18.817 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.584120+0000", 2024-09-13T14:41:18.817 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000121625 2024-09-13T14:41:18.817 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.817 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.817 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:18.817 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.589510+0000", 2024-09-13T14:41:18.817 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0053903969999999999 2024-09-13T14:41:18.817 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.818 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.818 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.818 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.589686+0000", 2024-09-13T14:41:18.818 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00017597500000000001 2024-09-13T14:41:18.818 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.818 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.818 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.818 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.635609+0000", 2024-09-13T14:41:18.818 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.045923065999999998 2024-09-13T14:41:18.818 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.818 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.818 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:18.818 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.635657+0000", 2024-09-13T14:41:18.818 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8043000000000001e-05 2024-09-13T14:41:18.819 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.819 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.819 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:18.819 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.635694+0000", 2024-09-13T14:41:18.819 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.6696000000000002e-05 2024-09-13T14:41:18.819 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.819 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:18.819 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.819 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.819 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.819 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:65149 6.2 6:4a0b78de:::benchmark_data_smithi151_155836_object65148:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1088)", 2024-09-13T14:41:18.819 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:39:56.586044+0000", 2024-09-13T14:41:18.819 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 82.202326001000003, 2024-09-13T14:41:18.819 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.053899141999999997, 2024-09-13T14:41:18.820 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:18.820 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:18.820 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:18.820 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:18.820 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:18.820 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 65149 2024-09-13T14:41:18.820 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.820 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:18.820 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.820 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:18.820 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.586044+0000", 2024-09-13T14:41:18.820 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.820 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.821 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.821 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:18.821 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.586044+0000", 2024-09-13T14:41:18.821 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.821 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.821 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.821 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:18.821 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.586043+0000", 2024-09-13T14:41:18.821 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:18.821 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.821 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.821 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:18.821 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.586090+0000", 2024-09-13T14:41:18.822 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7289000000000001e-05 2024-09-13T14:41:18.822 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.822 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.822 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:18.822 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.586104+0000", 2024-09-13T14:41:18.822 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3515e-05 2024-09-13T14:41:18.822 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.822 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.822 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:18.822 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.586114+0000", 2024-09-13T14:41:18.822 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.078e-05 2024-09-13T14:41:18.822 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.822 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.822 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:18.823 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.586990+0000", 2024-09-13T14:41:18.823 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00087508699999999998 2024-09-13T14:41:18.823 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.823 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.823 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:18.823 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.587102+0000", 2024-09-13T14:41:18.823 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000112156 2024-09-13T14:41:18.823 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.823 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.823 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:41:18.823 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.587228+0000", 2024-09-13T14:41:18.823 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012585799999999999 2024-09-13T14:41:18.823 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.823 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.824 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:18.824 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.589607+0000", 2024-09-13T14:41:18.824 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0023796020000000002 2024-09-13T14:41:18.824 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.824 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.824 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.824 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.589973+0000", 2024-09-13T14:41:18.824 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00036596399999999998 2024-09-13T14:41:18.824 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.824 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.824 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.824 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.639815+0000", 2024-09-13T14:41:18.824 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.049841367999999997 2024-09-13T14:41:18.824 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.825 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.825 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:18.825 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.639907+0000", 2024-09-13T14:41:18.825 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.2492999999999997e-05 2024-09-13T14:41:18.825 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.825 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.825 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:18.825 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.639943+0000", 2024-09-13T14:41:18.825 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.6337999999999997e-05 2024-09-13T14:41:18.825 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.825 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:18.825 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.825 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.826 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.826 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:65150 6.e 6:7eb31fd4:::benchmark_data_smithi151_155836_object65149:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1088)", 2024-09-13T14:41:18.826 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:39:56.586235+0000", 2024-09-13T14:41:18.826 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 82.202134885999996, 2024-09-13T14:41:18.826 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.056481661000000002, 2024-09-13T14:41:18.826 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:18.826 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:18.826 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:18.826 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:18.826 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:18.826 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 65150 2024-09-13T14:41:18.826 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.826 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:18.826 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.827 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:18.827 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.586235+0000", 2024-09-13T14:41:18.827 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.827 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.827 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.827 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:18.827 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.586235+0000", 2024-09-13T14:41:18.827 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.827 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.827 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.827 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:18.827 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.586235+0000", 2024-09-13T14:41:18.827 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:41:18.828 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.828 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.828 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:18.828 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.586301+0000", 2024-09-13T14:41:18.828 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.6509999999999998e-05 2024-09-13T14:41:18.828 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.828 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.828 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:18.828 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.586315+0000", 2024-09-13T14:41:18.828 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3614999999999999e-05 2024-09-13T14:41:18.828 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.828 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.829 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:18.829 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.586320+0000", 2024-09-13T14:41:18.829 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9250000000000003e-06 2024-09-13T14:41:18.829 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.829 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.829 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:18.829 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.586416+0000", 2024-09-13T14:41:18.829 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.6751999999999998e-05 2024-09-13T14:41:18.829 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.829 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.829 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:18.829 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.586588+0000", 2024-09-13T14:41:18.829 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00017155499999999999 2024-09-13T14:41:18.829 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.830 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.830 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:41:18.830 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.586755+0000", 2024-09-13T14:41:18.830 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00016661399999999999 2024-09-13T14:41:18.830 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.830 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.830 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:18.830 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.588824+0000", 2024-09-13T14:41:18.830 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0020691870000000001 2024-09-13T14:41:18.830 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.830 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.830 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.830 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.642602+0000", 2024-09-13T14:41:18.830 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.053777996000000002 2024-09-13T14:41:18.831 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.831 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.831 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.831 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.642649+0000", 2024-09-13T14:41:18.831 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7527000000000001e-05 2024-09-13T14:41:18.831 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.831 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.831 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:18.831 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.642686+0000", 2024-09-13T14:41:18.831 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.6613000000000003e-05 2024-09-13T14:41:18.831 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.831 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.831 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:18.831 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.642717+0000", 2024-09-13T14:41:18.832 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.1000000000000001e-05 2024-09-13T14:41:18.832 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.832 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:18.832 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.832 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.832 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.832 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:65155 6.e 6:76545be5:::benchmark_data_smithi151_155836_object65154:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1088)", 2024-09-13T14:41:18.832 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:39:56.594210+0000", 2024-09-13T14:41:18.832 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 82.194159733000006, 2024-09-13T14:41:18.832 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.04932843, 2024-09-13T14:41:18.832 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:18.832 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:18.832 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:18.832 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:18.833 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:18.833 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 65155 2024-09-13T14:41:18.833 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.833 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:18.833 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.833 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:18.833 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.594210+0000", 2024-09-13T14:41:18.833 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.833 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.833 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.833 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:18.833 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.594210+0000", 2024-09-13T14:41:18.833 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.833 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.834 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.834 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:18.834 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.594210+0000", 2024-09-13T14:41:18.834 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:41:18.834 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.834 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.834 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:18.834 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.594260+0000", 2024-09-13T14:41:18.834 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0682e-05 2024-09-13T14:41:18.834 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.834 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.834 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:18.834 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.594269+0000", 2024-09-13T14:41:18.834 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.6170000000000007e-06 2024-09-13T14:41:18.835 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.835 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.835 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:18.835 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.594274+0000", 2024-09-13T14:41:18.836 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9409999999999998e-06 2024-09-13T14:41:18.837 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.837 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.837 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:18.837 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.594304+0000", 2024-09-13T14:41:18.837 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.9859999999999999e-05 2024-09-13T14:41:18.837 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.837 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.837 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:18.837 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.594391+0000", 2024-09-13T14:41:18.837 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.7094000000000003e-05 2024-09-13T14:41:18.837 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.837 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.837 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:41:18.838 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.594472+0000", 2024-09-13T14:41:18.838 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0568000000000005e-05 2024-09-13T14:41:18.838 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.838 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.838 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:18.838 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.596679+0000", 2024-09-13T14:41:18.838 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0022078010000000001 2024-09-13T14:41:18.838 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.838 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.838 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.838 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.642745+0000", 2024-09-13T14:41:18.838 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.046066001000000002 2024-09-13T14:41:18.838 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.839 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.839 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.839 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.643459+0000", 2024-09-13T14:41:18.839 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00071395599999999997 2024-09-13T14:41:18.839 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.839 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.839 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:18.839 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.643507+0000", 2024-09-13T14:41:18.839 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8034999999999997e-05 2024-09-13T14:41:18.839 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.839 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.839 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:18.839 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.643539+0000", 2024-09-13T14:41:18.840 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.1569999999999998e-05 2024-09-13T14:41:18.840 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.840 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:18.840 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.840 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.840 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.840 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:65179 6.e 6:781d61f4:::benchmark_data_smithi151_155836_object65178:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1088)", 2024-09-13T14:41:18.840 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:39:56.644859+0000", 2024-09-13T14:41:18.840 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 82.143511336000003, 2024-09-13T14:41:18.840 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.049237138999999999, 2024-09-13T14:41:18.840 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:18.840 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:18.840 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:18.841 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:18.841 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:18.841 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 65179 2024-09-13T14:41:18.841 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.841 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:18.841 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.841 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:18.841 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.644859+0000", 2024-09-13T14:41:18.841 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.841 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.841 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.841 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:18.841 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.644859+0000", 2024-09-13T14:41:18.842 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.842 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.842 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.842 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:18.842 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.644858+0000", 2024-09-13T14:41:18.842 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:18.842 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.842 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.842 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:18.842 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.644900+0000", 2024-09-13T14:41:18.842 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.2191999999999999e-05 2024-09-13T14:41:18.842 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.842 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.843 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:18.843 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.644916+0000", 2024-09-13T14:41:18.843 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6206e-05 2024-09-13T14:41:18.843 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.843 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.843 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:18.843 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.644921+0000", 2024-09-13T14:41:18.843 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4079999999999997e-06 2024-09-13T14:41:18.843 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.843 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.843 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:18.843 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.644974+0000", 2024-09-13T14:41:18.843 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.2374999999999999e-05 2024-09-13T14:41:18.844 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.844 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.844 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:18.844 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.645265+0000", 2024-09-13T14:41:18.844 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00029171200000000002 2024-09-13T14:41:18.844 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.844 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.844 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:41:18.844 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.645627+0000", 2024-09-13T14:41:18.844 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00036164500000000001 2024-09-13T14:41:18.844 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.844 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.844 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.845 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.648381+0000", 2024-09-13T14:41:18.845 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0027542819999999998 2024-09-13T14:41:18.845 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.845 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.845 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:18.845 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.649251+0000", 2024-09-13T14:41:18.845 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00086994999999999998 2024-09-13T14:41:18.845 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.845 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.845 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.845 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.693974+0000", 2024-09-13T14:41:18.845 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.044722774 2024-09-13T14:41:18.846 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.846 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.846 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:18.846 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.694050+0000", 2024-09-13T14:41:18.846 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.5712999999999998e-05 2024-09-13T14:41:18.846 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.846 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.846 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:18.846 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:56.694096+0000", 2024-09-13T14:41:18.846 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.6205000000000002e-05 2024-09-13T14:41:18.846 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.846 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:18.846 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.846 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.847 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.847 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:105798 6.2 6:44c36013:::benchmark_data_smithi151_155836_object105797:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1092)", 2024-09-13T14:41:18.847 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:40:29.799041+0000", 2024-09-13T14:41:18.847 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 48.989328903999997, 2024-09-13T14:41:18.847 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.065233383000000006, 2024-09-13T14:41:18.847 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:18.847 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:18.847 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:18.847 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:18.847 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:18.847 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 105798 2024-09-13T14:41:18.847 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.847 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:18.848 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.848 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:18.848 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:29.799041+0000", 2024-09-13T14:41:18.848 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.848 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.848 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.848 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:18.848 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:29.799041+0000", 2024-09-13T14:41:18.848 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.848 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.848 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.848 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:18.848 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:29.799040+0000", 2024-09-13T14:41:18.849 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:18.849 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.849 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.849 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:18.849 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:29.799080+0000", 2024-09-13T14:41:18.849 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9657999999999999e-05 2024-09-13T14:41:18.849 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.849 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.849 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:18.849 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:29.799094+0000", 2024-09-13T14:41:18.849 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4721e-05 2024-09-13T14:41:18.849 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.849 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.850 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:18.850 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:29.799100+0000", 2024-09-13T14:41:18.850 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6899999999999997e-06 2024-09-13T14:41:18.850 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.850 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.850 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:18.850 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:29.799139+0000", 2024-09-13T14:41:18.850 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8878999999999998e-05 2024-09-13T14:41:18.850 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.850 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.850 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:18.850 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:29.799282+0000", 2024-09-13T14:41:18.850 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000143127 2024-09-13T14:41:18.851 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.851 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.851 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:41:18.851 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:29.799414+0000", 2024-09-13T14:41:18.851 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013220699999999999 2024-09-13T14:41:18.851 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.851 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.851 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:18.851 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:29.804886+0000", 2024-09-13T14:41:18.851 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0054721029999999999 2024-09-13T14:41:18.851 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.851 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.851 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.852 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:29.806070+0000", 2024-09-13T14:41:18.852 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0011832819999999999 2024-09-13T14:41:18.852 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.852 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.852 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.852 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:29.864182+0000", 2024-09-13T14:41:18.852 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.058112320000000002 2024-09-13T14:41:18.852 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.852 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.852 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:18.852 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:29.864235+0000", 2024-09-13T14:41:18.852 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.2677999999999999e-05 2024-09-13T14:41:18.852 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.853 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.853 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:18.853 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:29.864275+0000", 2024-09-13T14:41:18.853 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0022999999999999e-05 2024-09-13T14:41:18.853 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.853 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:18.853 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.853 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.853 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.853 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:105801 6.e 6:7216ae30:::benchmark_data_smithi151_155836_object105800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1092)", 2024-09-13T14:41:18.853 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:40:29.805350+0000", 2024-09-13T14:41:18.853 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 48.983020113999999, 2024-09-13T14:41:18.853 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.04869764, 2024-09-13T14:41:18.854 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:18.854 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:18.854 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:18.854 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:18.854 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:18.854 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 105801 2024-09-13T14:41:18.854 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.854 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:18.854 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.854 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:18.854 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:29.805350+0000", 2024-09-13T14:41:18.854 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.854 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.854 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.855 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:18.855 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:29.805350+0000", 2024-09-13T14:41:18.855 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.855 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.855 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.855 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:18.855 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:29.805349+0000", 2024-09-13T14:41:18.855 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:18.855 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.855 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.855 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:18.855 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:29.805388+0000", 2024-09-13T14:41:18.855 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8865999999999997e-05 2024-09-13T14:41:18.856 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.856 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.856 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:18.856 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:29.805401+0000", 2024-09-13T14:41:18.856 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3032e-05 2024-09-13T14:41:18.856 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.856 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.856 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:18.856 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:29.805407+0000", 2024-09-13T14:41:18.856 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.9830000000000001e-06 2024-09-13T14:41:18.856 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.856 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.857 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:18.857 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:29.810167+0000", 2024-09-13T14:41:18.857 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0047606860000000001 2024-09-13T14:41:18.857 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.857 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.857 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:18.857 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:29.810491+0000", 2024-09-13T14:41:18.857 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00032356300000000002 2024-09-13T14:41:18.857 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.857 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.857 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:41:18.857 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:29.810939+0000", 2024-09-13T14:41:18.857 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000448035 2024-09-13T14:41:18.858 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.858 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.858 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:18.858 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:29.814732+0000", 2024-09-13T14:41:18.858 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0037928749999999998 2024-09-13T14:41:18.858 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.858 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.858 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.859 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:29.815300+0000", 2024-09-13T14:41:18.859 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000568163 2024-09-13T14:41:18.859 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.859 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.859 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.859 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:29.853989+0000", 2024-09-13T14:41:18.859 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.038688860999999998 2024-09-13T14:41:18.859 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.859 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.859 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:18.860 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:29.854023+0000", 2024-09-13T14:41:18.860 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.4471000000000003e-05 2024-09-13T14:41:18.860 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.860 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.860 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:18.860 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:29.854048+0000", 2024-09-13T14:41:18.860 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.4386999999999999e-05 2024-09-13T14:41:18.860 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.861 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:18.861 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.861 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.861 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.861 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:131537 6.6 6:6f9bd1b5:::benchmark_data_smithi151_155836_object131536:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1094)", 2024-09-13T14:41:18.861 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:40:56.278884+0000", 2024-09-13T14:41:18.861 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 22.509486262999999, 2024-09-13T14:41:18.861 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.050405632999999998, 2024-09-13T14:41:18.861 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:18.861 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:18.862 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:18.862 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:18.862 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:18.862 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 131537 2024-09-13T14:41:18.862 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.862 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:18.862 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.862 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:18.862 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.278884+0000", 2024-09-13T14:41:18.862 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.862 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.862 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.862 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:18.862 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.278884+0000", 2024-09-13T14:41:18.863 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.863 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.863 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:18.863 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.278883+0000", 2024-09-13T14:41:18.863 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:41:18.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.863 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.863 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:18.863 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.278919+0000", 2024-09-13T14:41:18.863 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.6004999999999998e-05 2024-09-13T14:41:18.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.863 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.864 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:18.864 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.278929+0000", 2024-09-13T14:41:18.864 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.2089999999999994e-06 2024-09-13T14:41:18.864 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.864 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.864 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:18.864 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.278933+0000", 2024-09-13T14:41:18.864 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.2300000000000002e-06 2024-09-13T14:41:18.864 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.864 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.864 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:18.864 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.295749+0000", 2024-09-13T14:41:18.864 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.016816289000000002 2024-09-13T14:41:18.865 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.865 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.865 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:18.865 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.295868+0000", 2024-09-13T14:41:18.865 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000118906 2024-09-13T14:41:18.865 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.865 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.865 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:41:18.865 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.296023+0000", 2024-09-13T14:41:18.865 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00015477199999999999 2024-09-13T14:41:18.865 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.865 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.865 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:18.865 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.309167+0000", 2024-09-13T14:41:18.866 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.013143704000000001 2024-09-13T14:41:18.866 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.866 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.866 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.866 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.314611+0000", 2024-09-13T14:41:18.866 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0054448750000000001 2024-09-13T14:41:18.866 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.866 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.866 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.866 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.329009+0000", 2024-09-13T14:41:18.866 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.014397801 2024-09-13T14:41:18.866 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.867 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.867 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:18.867 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.329176+0000", 2024-09-13T14:41:18.867 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00016682700000000001 2024-09-13T14:41:18.867 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.867 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.867 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:18.867 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.329290+0000", 2024-09-13T14:41:18.868 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000113504 2024-09-13T14:41:18.868 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.868 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:18.868 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.868 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.868 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.868 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:131541 6.6 6:6cf21b41:::benchmark_data_smithi151_155836_object131540:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1094)", 2024-09-13T14:41:18.868 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:40:56.280048+0000", 2024-09-13T14:41:18.868 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 22.508322574000001, 2024-09-13T14:41:18.868 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.049593529999999997, 2024-09-13T14:41:18.868 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:18.868 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:18.869 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:18.869 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:18.869 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:18.869 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 131541 2024-09-13T14:41:18.869 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.869 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:18.869 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.869 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:18.869 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.280048+0000", 2024-09-13T14:41:18.869 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.869 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.869 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.869 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:18.870 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.280048+0000", 2024-09-13T14:41:18.870 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.870 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.870 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.870 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:18.870 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.280047+0000", 2024-09-13T14:41:18.870 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:41:18.870 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.870 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.870 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:18.870 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.280111+0000", 2024-09-13T14:41:18.870 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.4561999999999996e-05 2024-09-13T14:41:18.870 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.871 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.871 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:18.871 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.280125+0000", 2024-09-13T14:41:18.871 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3977e-05 2024-09-13T14:41:18.871 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.871 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.871 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:18.871 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.280130+0000", 2024-09-13T14:41:18.871 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.356e-06 2024-09-13T14:41:18.871 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.871 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.871 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:18.871 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.304885+0000", 2024-09-13T14:41:18.871 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.024754594000000001 2024-09-13T14:41:18.872 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.872 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.872 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:18.872 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.305182+0000", 2024-09-13T14:41:18.872 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00029710899999999999 2024-09-13T14:41:18.872 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.872 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.872 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:41:18.872 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.305332+0000", 2024-09-13T14:41:18.872 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000149424 2024-09-13T14:41:18.872 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.872 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.872 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:18.873 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.328744+0000", 2024-09-13T14:41:18.873 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.023412513999999999 2024-09-13T14:41:18.873 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.873 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.873 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.873 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.329514+0000", 2024-09-13T14:41:18.873 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00077008100000000004 2024-09-13T14:41:18.873 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.873 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.873 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.873 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.329543+0000", 2024-09-13T14:41:18.873 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.8515000000000001e-05 2024-09-13T14:41:18.873 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.874 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.874 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:18.874 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.329609+0000", 2024-09-13T14:41:18.874 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.6544000000000005e-05 2024-09-13T14:41:18.874 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.874 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.874 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:18.874 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.329641+0000", 2024-09-13T14:41:18.874 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.1887000000000001e-05 2024-09-13T14:41:18.874 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.874 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:18.874 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.874 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.874 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.875 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:131551 6.6 6:6ac9868c:::benchmark_data_smithi151_155836_object131550:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1094)", 2024-09-13T14:41:18.875 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:40:56.291597+0000", 2024-09-13T14:41:18.875 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 22.496772870000001, 2024-09-13T14:41:18.875 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.057138440999999998, 2024-09-13T14:41:18.875 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:18.875 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:18.875 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:18.875 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:18.875 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:18.875 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 131551 2024-09-13T14:41:18.875 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.875 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:18.875 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.875 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:18.876 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.291597+0000", 2024-09-13T14:41:18.876 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.876 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.876 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.876 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:18.876 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.291597+0000", 2024-09-13T14:41:18.876 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.876 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.876 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.876 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:18.876 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.291596+0000", 2024-09-13T14:41:18.876 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:18.876 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.877 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.877 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:18.877 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.291680+0000", 2024-09-13T14:41:18.877 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.3873000000000001e-05 2024-09-13T14:41:18.877 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.877 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.877 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:18.877 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.291696+0000", 2024-09-13T14:41:18.877 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6104999999999999e-05 2024-09-13T14:41:18.877 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.877 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.877 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:18.880 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.291701+0000", 2024-09-13T14:41:18.881 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5110000000000003e-06 2024-09-13T14:41:18.881 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.881 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.881 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:18.881 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.315105+0000", 2024-09-13T14:41:18.881 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.023403360000000002 2024-09-13T14:41:18.881 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.881 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.881 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:18.881 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.320924+0000", 2024-09-13T14:41:18.881 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0058195779999999997 2024-09-13T14:41:18.881 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.881 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.882 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:41:18.882 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.321076+0000", 2024-09-13T14:41:18.882 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000151481 2024-09-13T14:41:18.882 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.882 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.882 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:18.882 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.329680+0000", 2024-09-13T14:41:18.882 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0086038079999999992 2024-09-13T14:41:18.882 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.882 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.882 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.882 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.348555+0000", 2024-09-13T14:41:18.882 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.018875376999999999 2024-09-13T14:41:18.882 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.883 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.883 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.883 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.348595+0000", 2024-09-13T14:41:18.883 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9551999999999997e-05 2024-09-13T14:41:18.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.883 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.883 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:18.883 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.348689+0000", 2024-09-13T14:41:18.883 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.4191999999999995e-05 2024-09-13T14:41:18.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.883 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.883 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:18.883 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:40:56.348736+0000", 2024-09-13T14:41:18.883 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.6962000000000002e-05 2024-09-13T14:41:18.884 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.884 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:18.884 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.884 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.884 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:18.885 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:18.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:18.885 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:18 smithi151 ceph-mon[104450]: pgmap v1360: 99 pgs: 10 unknown, 89 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-09-13T14:41:18.886 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:18 smithi151 ceph-mon[108966]: pgmap v1360: 99 pgs: 10 unknown, 89 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-09-13T14:41:18.886 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:136375 6.2 6:48ee3226:::benchmark_data_smithi151_155836_object3917:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1094)", 2024-09-13T14:41:18.886 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:01.422959+0000", 2024-09-13T14:41:18.887 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 17.365411538, 2024-09-13T14:41:18.887 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.049271897000000002, 2024-09-13T14:41:18.887 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:18.887 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:18.887 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:18.887 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:18.887 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:18.887 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 136375 2024-09-13T14:41:18.887 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.887 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:18.887 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.887 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:18.887 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:01.422959+0000", 2024-09-13T14:41:18.888 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.888 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.888 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.888 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:18.888 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:01.422959+0000", 2024-09-13T14:41:18.888 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.888 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.888 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.888 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:18.888 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:01.422958+0000", 2024-09-13T14:41:18.888 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:41:18.888 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.888 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.889 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:18.889 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:01.422961+0000", 2024-09-13T14:41:18.889 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.57e-06 2024-09-13T14:41:18.889 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.889 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.889 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:18.889 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:01.422966+0000", 2024-09-13T14:41:18.889 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6489999999999996e-06 2024-09-13T14:41:18.889 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.889 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.889 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:18.889 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:01.422970+0000", 2024-09-13T14:41:18.890 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3900000000000002e-06 2024-09-13T14:41:18.890 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.890 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.890 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:18.890 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:01.432562+0000", 2024-09-13T14:41:18.890 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0095919449999999993 2024-09-13T14:41:18.890 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.890 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.890 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:18.891 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:01.432823+0000", 2024-09-13T14:41:18.891 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00026078199999999998 2024-09-13T14:41:18.891 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.891 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.891 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:41:18.891 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:01.433152+0000", 2024-09-13T14:41:18.891 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00032974700000000003 2024-09-13T14:41:18.891 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.891 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.891 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.892 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:01.441203+0000", 2024-09-13T14:41:18.892 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0080510879999999996 2024-09-13T14:41:18.892 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.892 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.892 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:18.892 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:01.441662+0000", 2024-09-13T14:41:18.892 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00045854899999999999 2024-09-13T14:41:18.892 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.892 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.892 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.892 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:01.472153+0000", 2024-09-13T14:41:18.893 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.030491193999999999 2024-09-13T14:41:18.893 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.893 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.893 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:18.893 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:01.472197+0000", 2024-09-13T14:41:18.893 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3477999999999999e-05 2024-09-13T14:41:18.893 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.893 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.893 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:18.893 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:01.472231+0000", 2024-09-13T14:41:18.893 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3966e-05 2024-09-13T14:41:18.893 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.893 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:18.894 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.894 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.894 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.894 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:137064 6.2 6:4879ba93:::benchmark_data_smithi151_155836_object4606:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1094)", 2024-09-13T14:41:18.894 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:02.143167+0000", 2024-09-13T14:41:18.894 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 16.645202811000001, 2024-09-13T14:41:18.894 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.050471578000000003, 2024-09-13T14:41:18.894 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:18.894 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:18.894 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:18.894 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:18.894 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:18.894 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 137064 2024-09-13T14:41:18.894 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.895 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:18.895 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.895 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:18.895 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:02.143167+0000", 2024-09-13T14:41:18.895 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.895 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.895 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.895 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:18.895 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:02.143167+0000", 2024-09-13T14:41:18.895 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.895 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.895 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.895 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:18.895 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:02.143166+0000", 2024-09-13T14:41:18.896 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:18.896 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.896 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.896 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:18.896 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:02.143177+0000", 2024-09-13T14:41:18.896 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.116e-05 2024-09-13T14:41:18.896 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.896 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.896 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:18.896 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:02.143193+0000", 2024-09-13T14:41:18.896 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.607e-05 2024-09-13T14:41:18.896 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.896 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.896 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:18.897 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:02.143200+0000", 2024-09-13T14:41:18.897 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.5479999999999998e-06 2024-09-13T14:41:18.897 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.897 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.897 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:18.897 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:02.145233+0000", 2024-09-13T14:41:18.897 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002033684 2024-09-13T14:41:18.897 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.897 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.897 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:18.897 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:02.145333+0000", 2024-09-13T14:41:18.897 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.9575999999999997e-05 2024-09-13T14:41:18.897 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.898 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.898 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:41:18.898 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:02.145439+0000", 2024-09-13T14:41:18.898 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000105948 2024-09-13T14:41:18.898 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.898 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.898 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.898 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:02.156319+0000", 2024-09-13T14:41:18.898 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.010880465000000001 2024-09-13T14:41:18.898 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.898 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.898 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:18.898 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:02.156614+0000", 2024-09-13T14:41:18.898 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00029469099999999997 2024-09-13T14:41:18.899 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.899 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.899 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.899 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:02.193371+0000", 2024-09-13T14:41:18.899 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.036756664000000001 2024-09-13T14:41:18.899 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.899 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.899 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:18.899 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:02.193529+0000", 2024-09-13T14:41:18.899 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00015861599999999999 2024-09-13T14:41:18.899 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.899 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.899 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:18.899 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:02.193639+0000", 2024-09-13T14:41:18.899 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00010972999999999999 2024-09-13T14:41:18.900 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.900 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:18.900 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.900 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.900 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.900 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:137066 6.2 6:4b3b894b:::benchmark_data_smithi151_155836_object4608:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1094)", 2024-09-13T14:41:18.900 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:02.144151+0000", 2024-09-13T14:41:18.900 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 16.644219483000001, 2024-09-13T14:41:18.900 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.049872448999999999, 2024-09-13T14:41:18.900 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:18.900 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:18.900 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:18.900 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:18.900 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:18.901 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 137066 2024-09-13T14:41:18.901 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.901 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:18.901 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.901 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:18.901 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:02.144151+0000", 2024-09-13T14:41:18.901 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.901 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.901 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.901 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:18.901 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:02.144151+0000", 2024-09-13T14:41:18.901 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.902 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.902 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.902 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:18.902 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:02.144150+0000", 2024-09-13T14:41:18.902 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:41:18.902 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.902 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.902 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:18.902 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:02.144153+0000", 2024-09-13T14:41:18.902 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.289e-06 2024-09-13T14:41:18.902 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.902 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.902 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:18.902 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:02.144159+0000", 2024-09-13T14:41:18.903 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.4819999999999999e-06 2024-09-13T14:41:18.903 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.903 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.903 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:18.903 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:02.144163+0000", 2024-09-13T14:41:18.903 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.58e-06 2024-09-13T14:41:18.903 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.903 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.903 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:18.903 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:02.151281+0000", 2024-09-13T14:41:18.903 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0071187580000000002 2024-09-13T14:41:18.903 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.903 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.903 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:18.904 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:02.151543+0000", 2024-09-13T14:41:18.904 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000261895 2024-09-13T14:41:18.904 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.904 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.904 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:41:18.904 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:02.151998+0000", 2024-09-13T14:41:18.904 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000454432 2024-09-13T14:41:18.904 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.904 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.904 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:18.904 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:02.156630+0000", 2024-09-13T14:41:18.904 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0046325460000000004 2024-09-13T14:41:18.904 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.905 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.905 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.905 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:02.166119+0000", 2024-09-13T14:41:18.905 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0094888520000000007 2024-09-13T14:41:18.905 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.905 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.905 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.905 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:02.193769+0000", 2024-09-13T14:41:18.905 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.027649880000000002 2024-09-13T14:41:18.905 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.905 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.905 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:18.905 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:02.193902+0000", 2024-09-13T14:41:18.905 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013337099999999999 2024-09-13T14:41:18.906 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.906 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.906 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:18.906 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:02.194023+0000", 2024-09-13T14:41:18.906 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012077499999999999 2024-09-13T14:41:18.906 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.906 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:18.906 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.906 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.906 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.906 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:149002 6.6 6:6da4e9ac:::benchmark_data_smithi151_155836_object16544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1097)", 2024-09-13T14:41:18.906 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:16.062929+0000", 2024-09-13T14:41:18.906 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 2.7254411009999999, 2024-09-13T14:41:18.907 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.048961047000000001, 2024-09-13T14:41:18.907 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:18.907 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:18.907 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:18.907 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:18.907 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:18.907 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 149002 2024-09-13T14:41:18.907 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.907 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:18.907 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.907 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:18.907 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.062929+0000", 2024-09-13T14:41:18.907 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.907 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.908 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.908 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:18.908 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.062929+0000", 2024-09-13T14:41:18.908 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.908 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.908 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.908 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:18.908 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.062927+0000", 2024-09-13T14:41:18.908 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:41:18.908 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.908 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.908 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:18.908 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.062935+0000", 2024-09-13T14:41:18.908 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.7179999999999996e-06 2024-09-13T14:41:18.909 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.909 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.909 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:18.909 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.062947+0000", 2024-09-13T14:41:18.909 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.151e-05 2024-09-13T14:41:18.909 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.909 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.909 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:18.909 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.062952+0000", 2024-09-13T14:41:18.909 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6189999999999999e-06 2024-09-13T14:41:18.909 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.909 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.909 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:18.910 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.077919+0000", 2024-09-13T14:41:18.910 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.014966754000000001 2024-09-13T14:41:18.910 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.910 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.910 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:18.910 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.078031+0000", 2024-09-13T14:41:18.910 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00011212 2024-09-13T14:41:18.910 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.910 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.910 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:41:18.910 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.078137+0000", 2024-09-13T14:41:18.910 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00010633600000000001 2024-09-13T14:41:18.910 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.910 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.911 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:18.911 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.083838+0000", 2024-09-13T14:41:18.911 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0057004519999999999 2024-09-13T14:41:18.911 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.911 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.911 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.911 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.106538+0000", 2024-09-13T14:41:18.911 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.022699888000000001 2024-09-13T14:41:18.911 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.911 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.911 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.911 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.111530+0000", 2024-09-13T14:41:18.911 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.004992359 2024-09-13T14:41:18.911 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.912 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.912 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:18.912 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.111841+0000", 2024-09-13T14:41:18.912 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00031129299999999998 2024-09-13T14:41:18.912 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.912 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.912 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:18.912 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.111890+0000", 2024-09-13T14:41:18.912 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8798999999999998e-05 2024-09-13T14:41:18.912 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.912 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:18.912 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.912 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.913 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.913 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:149156 6.e 6:78c10eea:::benchmark_data_smithi151_155836_object16698:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1097)", 2024-09-13T14:41:18.913 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:16.269750+0000", 2024-09-13T14:41:18.913 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 2.5186200899999998, 2024-09-13T14:41:18.913 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.052995338000000003, 2024-09-13T14:41:18.913 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:18.913 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:18.913 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:18.913 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:18.913 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:18.913 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 149156 2024-09-13T14:41:18.913 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.913 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:18.913 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.914 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:18.914 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.269750+0000", 2024-09-13T14:41:18.914 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.914 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.914 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.914 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:18.914 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.269750+0000", 2024-09-13T14:41:18.914 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.914 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.914 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.914 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:18.914 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.269749+0000", 2024-09-13T14:41:18.914 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:18.914 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.915 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.915 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:18.915 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.269756+0000", 2024-09-13T14:41:18.915 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.9809999999999997e-06 2024-09-13T14:41:18.915 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.915 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.915 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:18.915 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.269773+0000", 2024-09-13T14:41:18.915 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6775000000000001e-05 2024-09-13T14:41:18.915 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.915 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.915 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:18.915 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.269779+0000", 2024-09-13T14:41:18.915 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.4899999999999997e-06 2024-09-13T14:41:18.916 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.916 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.916 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:18.916 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.293278+0000", 2024-09-13T14:41:18.916 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.023499276999999999 2024-09-13T14:41:18.916 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.916 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.916 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:18.916 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.293527+0000", 2024-09-13T14:41:18.916 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00024901699999999998 2024-09-13T14:41:18.916 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.916 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.916 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:41:18.916 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.293848+0000", 2024-09-13T14:41:18.917 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00032016899999999998 2024-09-13T14:41:18.917 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.917 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.917 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.917 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.308108+0000", 2024-09-13T14:41:18.917 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.014260752999999999 2024-09-13T14:41:18.917 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.917 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.917 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.917 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.308146+0000", 2024-09-13T14:41:18.917 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8198999999999998e-05 2024-09-13T14:41:18.917 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.917 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.918 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:18.918 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.322652+0000", 2024-09-13T14:41:18.918 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.014505037 2024-09-13T14:41:18.918 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.918 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.918 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:18.918 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.322703+0000", 2024-09-13T14:41:18.918 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1573999999999998e-05 2024-09-13T14:41:18.918 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.918 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.918 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:18.918 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.322745+0000", 2024-09-13T14:41:18.918 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.2347e-05 2024-09-13T14:41:18.918 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.918 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:18.919 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.919 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.919 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.919 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:149329 6.2 6:4220539c:::benchmark_data_smithi151_155836_object16871:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1098)", 2024-09-13T14:41:18.919 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:16.524169+0000", 2024-09-13T14:41:18.919 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 2.2642011430000002, 2024-09-13T14:41:18.919 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.063812353000000002, 2024-09-13T14:41:18.919 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:18.919 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:18.919 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:18.919 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:18.919 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:18.919 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 149329 2024-09-13T14:41:18.919 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.920 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:18.920 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.920 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:18.920 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.524169+0000", 2024-09-13T14:41:18.920 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.920 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.920 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:18.920 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.524169+0000", 2024-09-13T14:41:18.920 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.920 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.920 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:18.920 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.524167+0000", 2024-09-13T14:41:18.920 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:18.921 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.921 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.921 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:18.921 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.524179+0000", 2024-09-13T14:41:18.921 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1336e-05 2024-09-13T14:41:18.921 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.921 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.921 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:18.921 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.524196+0000", 2024-09-13T14:41:18.921 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.7442e-05 2024-09-13T14:41:18.921 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.921 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.921 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:18.921 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.524202+0000", 2024-09-13T14:41:18.921 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.0100000000000001e-06 2024-09-13T14:41:18.922 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.922 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.922 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:18.922 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.524253+0000", 2024-09-13T14:41:18.922 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0355999999999997e-05 2024-09-13T14:41:18.922 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.922 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.922 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:18.922 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.524370+0000", 2024-09-13T14:41:18.922 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000117031 2024-09-13T14:41:18.922 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.922 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.922 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:41:18.922 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.524507+0000", 2024-09-13T14:41:18.923 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013768400000000001 2024-09-13T14:41:18.923 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.923 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.923 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.923 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.531385+0000", 2024-09-13T14:41:18.923 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0068773269999999999 2024-09-13T14:41:18.923 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.923 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.923 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:18.923 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.536775+0000", 2024-09-13T14:41:18.923 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0053908029999999996 2024-09-13T14:41:18.923 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.923 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.924 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.924 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.587816+0000", 2024-09-13T14:41:18.924 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.051040248000000003 2024-09-13T14:41:18.924 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.924 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.924 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:18.924 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.587940+0000", 2024-09-13T14:41:18.924 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012431599999999999 2024-09-13T14:41:18.924 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.924 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.924 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:18.924 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.587981+0000", 2024-09-13T14:41:18.924 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1412000000000003e-05 2024-09-13T14:41:18.925 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.925 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:18.925 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.925 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.925 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.925 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:149330 6.2 6:4bde714e:::benchmark_data_smithi151_155836_object16872:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1098)", 2024-09-13T14:41:18.925 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:16.529286+0000", 2024-09-13T14:41:18.925 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 2.2590837280000002, 2024-09-13T14:41:18.925 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.058946710999999999, 2024-09-13T14:41:18.925 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:18.925 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:18.925 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:18.925 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:18.926 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:18.926 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 149330 2024-09-13T14:41:18.926 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.926 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:18.926 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.926 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:18.926 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.529286+0000", 2024-09-13T14:41:18.926 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.926 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.926 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.926 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:18.926 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.529286+0000", 2024-09-13T14:41:18.926 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:18.926 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.927 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.927 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:18.927 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.529285+0000", 2024-09-13T14:41:18.927 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:18.927 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.927 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.927 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:18.927 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.529293+0000", 2024-09-13T14:41:18.927 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.8699999999999992e-06 2024-09-13T14:41:18.927 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.927 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.927 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:18.927 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.529313+0000", 2024-09-13T14:41:18.927 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.0344999999999999e-05 2024-09-13T14:41:18.928 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.928 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.928 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:18.928 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.529330+0000", 2024-09-13T14:41:18.928 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6923e-05 2024-09-13T14:41:18.928 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.928 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.928 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:18.928 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.531406+0000", 2024-09-13T14:41:18.928 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0020758869999999998 2024-09-13T14:41:18.928 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.928 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.928 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:18.928 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.531503+0000", 2024-09-13T14:41:18.929 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.7353000000000001e-05 2024-09-13T14:41:18.929 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.929 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.929 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:41:18.929 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.531603+0000", 2024-09-13T14:41:18.929 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.9338000000000003e-05 2024-09-13T14:41:18.929 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.929 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.929 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.929 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.536824+0000", 2024-09-13T14:41:18.929 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0052209919999999998 2024-09-13T14:41:18.929 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.929 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.930 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:18.930 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.539383+0000", 2024-09-13T14:41:18.930 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002559431 2024-09-13T14:41:18.930 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.930 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.930 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:18.930 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.588021+0000", 2024-09-13T14:41:18.930 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.048637745000000003 2024-09-13T14:41:18.930 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.930 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.930 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:18.930 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.588199+0000", 2024-09-13T14:41:18.930 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00017853500000000001 2024-09-13T14:41:18.930 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:18.931 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:18.931 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:18.931 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:16.588233+0000", 2024-09-13T14:41:18.931 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3861e-05 2024-09-13T14:41:18.931 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.931 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:18.931 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.931 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:18.931 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:18.931 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:41:18.986 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:19.026 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:19.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:19.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:19.128 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:19.170 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:19.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:19.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:19.272 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:19.315 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:19.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:19.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:19.417 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:19.445 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:19.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:19.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:19.547 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:19.588 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:19.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:19.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:19.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:19 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-13T14:41:19.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:19 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-13T14:41:19.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:19 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-13T14:41:19.689 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:19.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:19.723 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:19.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:19.824 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:19.862 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:19.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:19.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:19.964 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:19.992 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:19.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:19.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:20.094 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:20.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:20.132 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:20.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:20.233 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:20.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:20.273 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:20.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:20.374 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:20.407 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:20.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:20.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:20.425 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T14:41:20.472 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:41:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:41:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:41:20.509 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:20.568 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:20.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:20.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:20.583 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:20 smithi151 ceph-mon[108966]: pgmap v1361: 99 pgs: 99 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 844 op/s 2024-09-13T14:41:20.584 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:20 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-13T14:41:20.584 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:20 smithi151 ceph-mon[108966]: osdmap e1099: 8 total, 8 up, 8 in 2024-09-13T14:41:20.584 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:20 smithi151 ceph-mon[104450]: pgmap v1361: 99 pgs: 99 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 844 op/s 2024-09-13T14:41:20.584 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:20 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-13T14:41:20.585 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:20 smithi151 ceph-mon[104450]: osdmap e1099: 8 total, 8 up, 8 in 2024-09-13T14:41:20.670 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:20.715 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:20.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:20.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:20.816 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:20.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:20 smithi184 ceph-mon[92527]: pgmap v1361: 99 pgs: 99 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 844 op/s 2024-09-13T14:41:20.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:20 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-13T14:41:20.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:20 smithi184 ceph-mon[92527]: osdmap e1099: 8 total, 8 up, 8 in 2024-09-13T14:41:20.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:20.862 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:20.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:20.882 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:41:20.963 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:21.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:21.003 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:21.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:21.104 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:21.147 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:21.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:21.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:21.250 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:21.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:21.292 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:21.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:21.394 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:21.427 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:21.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:21.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:21.500 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-13T14:41:21.530 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:21.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:21.570 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:21.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:21.671 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:21.706 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:21.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:21.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:21.745 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:21 smithi151 ceph-mon[104450]: osdmap e1100: 8 total, 8 up, 8 in 2024-09-13T14:41:21.745 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:21 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/599840558' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:41:21.745 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:21 smithi151 ceph-mon[104450]: 1.3 scrub starts 2024-09-13T14:41:21.745 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:21 smithi151 ceph-mon[108966]: osdmap e1100: 8 total, 8 up, 8 in 2024-09-13T14:41:21.745 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:21 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/599840558' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:41:21.745 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:21 smithi151 ceph-mon[108966]: 1.3 scrub starts 2024-09-13T14:41:21.808 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:21.852 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:21.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:21.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:21.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:21 smithi184 ceph-mon[92527]: osdmap e1100: 8 total, 8 up, 8 in 2024-09-13T14:41:21.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:21 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/599840558' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:41:21.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:21 smithi184 ceph-mon[92527]: 1.3 scrub starts 2024-09-13T14:41:21.954 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:21.989 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:21.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:21.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:21.998 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:41:21.998 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:41:21.998 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:41:21.998 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd out 1 2024-09-13T14:41:22.092 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:22.116 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:41:22.145 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:22.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:22.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:22.247 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:22.305 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:22.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:22.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:22.407 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:22.454 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:22.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:22.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:22.555 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:22.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:22.630 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:22.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:22.666 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:22.673 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:22.731 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:22.757 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:22.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:22.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:22.762 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:22 smithi184 ceph-mon[92527]: pgmap v1364: 99 pgs: 99 active+clean; 7.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 760 op/s 2024-09-13T14:41:22.763 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:22 smithi184 ceph-mon[92527]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:22.763 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:22 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/599840558' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:41:22.763 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:22 smithi184 ceph-mon[92527]: osdmap e1101: 8 total, 8 up, 8 in 2024-09-13T14:41:22.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:22 smithi151 ceph-mon[104450]: pgmap v1364: 99 pgs: 99 active+clean; 7.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 760 op/s 2024-09-13T14:41:22.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:22 smithi151 ceph-mon[104450]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:22.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:22 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/599840558' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:41:22.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:22 smithi151 ceph-mon[104450]: osdmap e1101: 8 total, 8 up, 8 in 2024-09-13T14:41:22.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:22 smithi151 ceph-mon[108966]: pgmap v1364: 99 pgs: 99 active+clean; 7.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 760 op/s 2024-09-13T14:41:22.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:22 smithi151 ceph-mon[108966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:22.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:22 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/599840558' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:41:22.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:22 smithi151 ceph-mon[108966]: osdmap e1101: 8 total, 8 up, 8 in 2024-09-13T14:41:22.859 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:22.904 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:22.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:22.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:22.916 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:41:22.916 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [ 2024-09-13T14:41:22.916 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.916 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:155271 6.7 6:ee63bd47:::benchmark_data_smithi151_155836_object22813:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1101)", 2024-09-13T14:41:22.916 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:22.901949+0000", 2024-09-13T14:41:22.916 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.013025950999999999, 2024-09-13T14:41:22.917 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.013057821000000001, 2024-09-13T14:41:22.917 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:22.917 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:41:22.917 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:22.917 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:22.917 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:22.917 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 155271 2024-09-13T14:41:22.917 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:22.917 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:22.917 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.917 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:22.917 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:22.901949+0000", 2024-09-13T14:41:22.917 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:22.918 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:22.918 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.918 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:22.918 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:22.901949+0000", 2024-09-13T14:41:22.918 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:22.918 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:22.918 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.918 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:22.918 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:22.901948+0000", 2024-09-13T14:41:22.918 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:22.918 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:22.918 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.918 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:22.919 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:22.901954+0000", 2024-09-13T14:41:22.919 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3620000000000002e-06 2024-09-13T14:41:22.919 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:22.919 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.919 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:22.919 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:22.901971+0000", 2024-09-13T14:41:22.919 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6622000000000001e-05 2024-09-13T14:41:22.919 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:22.919 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.919 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:22.919 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:22.901977+0000", 2024-09-13T14:41:22.919 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.6250000000000001e-06 2024-09-13T14:41:22.919 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:22.920 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.920 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:22.920 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:22.902105+0000", 2024-09-13T14:41:22.920 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012732600000000001 2024-09-13T14:41:22.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:22.920 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.920 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:22.920 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:22.902358+0000", 2024-09-13T14:41:22.920 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00025285599999999998 2024-09-13T14:41:22.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:22.920 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.920 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:41:22.920 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:22.902722+0000", 2024-09-13T14:41:22.921 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000364154 2024-09-13T14:41:22.921 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:22.921 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.921 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:22.921 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:22.914517+0000", 2024-09-13T14:41:22.921 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.011794804000000001 2024-09-13T14:41:22.921 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:22.921 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.921 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:22.921 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:22.914557+0000", 2024-09-13T14:41:22.921 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0314000000000001e-05 2024-09-13T14:41:22.921 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:22.921 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:22.922 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:22.922 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:22.922 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.922 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:155274 6.7 6:ed059533:::benchmark_data_smithi151_155836_object22816:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1101)", 2024-09-13T14:41:22.922 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:22.906049+0000", 2024-09-13T14:41:22.922 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.0089260840000000008, 2024-09-13T14:41:22.922 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0090545130000000001, 2024-09-13T14:41:22.922 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:22.922 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:41:22.922 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:22.922 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:22.922 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:22.922 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 155274 2024-09-13T14:41:22.923 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:22.923 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:22.923 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.923 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:22.923 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:22.906049+0000", 2024-09-13T14:41:22.923 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:22.923 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:22.923 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.923 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:22.923 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:22.906049+0000", 2024-09-13T14:41:22.923 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:22.923 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:22.923 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.923 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:22.924 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:22.906048+0000", 2024-09-13T14:41:22.924 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:22.924 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:22.924 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.924 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:22.924 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:22.906057+0000", 2024-09-13T14:41:22.924 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.918e-06 2024-09-13T14:41:22.924 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:22.924 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.924 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:22.924 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:22.906073+0000", 2024-09-13T14:41:22.924 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6008e-05 2024-09-13T14:41:22.924 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:22.925 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.925 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:22.925 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:22.906079+0000", 2024-09-13T14:41:22.925 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7689999999999999e-06 2024-09-13T14:41:22.925 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:22.925 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.925 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:22.925 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:22.907228+0000", 2024-09-13T14:41:22.925 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0011488080000000001 2024-09-13T14:41:22.925 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:22.925 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.925 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:22.925 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:22.907346+0000", 2024-09-13T14:41:22.926 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000117732 2024-09-13T14:41:22.926 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:22.926 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.926 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:41:22.926 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:22.907457+0000", 2024-09-13T14:41:22.926 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000110874 2024-09-13T14:41:22.926 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:22.926 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.926 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:22.926 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:22.914583+0000", 2024-09-13T14:41:22.926 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0071266589999999996 2024-09-13T14:41:22.926 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:22.926 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.926 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:22.927 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:22.914716+0000", 2024-09-13T14:41:22.927 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000132958 2024-09-13T14:41:22.927 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:22.927 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:22.927 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:22.927 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:22.927 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.927 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35289.0:155276 6.0 e1101/1082 6:08547456:::benchmark_data_smithi151_155836_object22818:head v 1101'9765, mlcod=1101'9763)", 2024-09-13T14:41:22.927 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:22.913144+0000", 2024-09-13T14:41:22.927 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.001831446, 2024-09-13T14:41:22.927 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0020472289999999998, 2024-09-13T14:41:22.927 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:22.927 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:41:22.928 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:22.928 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.928 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:22.928 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:22.913144+0000", 2024-09-13T14:41:22.928 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:22.928 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:22.928 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.928 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:22.928 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:22.913144+0000", 2024-09-13T14:41:22.928 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:22.928 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:22.928 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.928 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:22.929 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:22.913143+0000", 2024-09-13T14:41:22.929 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:41:22.929 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:22.929 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.929 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:22.929 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:22.913153+0000", 2024-09-13T14:41:22.929 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.9469999999999998e-06 2024-09-13T14:41:22.929 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:22.930 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.930 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:22.930 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:22.913168+0000", 2024-09-13T14:41:22.930 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4826e-05 2024-09-13T14:41:22.930 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:22.930 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.930 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:22.930 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:22.913173+0000", 2024-09-13T14:41:22.930 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.2630000000000003e-06 2024-09-13T14:41:22.930 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:22.930 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.930 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:22.931 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:22.913277+0000", 2024-09-13T14:41:22.931 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000103788 2024-09-13T14:41:22.931 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:22.931 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:22.931 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:22.931 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:22.913296+0000", 2024-09-13T14:41:22.931 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.9422000000000001e-05 2024-09-13T14:41:22.931 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:22.931 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:22.931 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:22.931 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:22.931 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-13T14:41:22.931 INFO:teuthology.orchestra.run.smithi184.stdout: "num_ops": 3 2024-09-13T14:41:22.932 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:41:23.006 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:23.038 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:23.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:23.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:23.140 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:23.179 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:23.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:23.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:23.281 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:23.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:23.319 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:23.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:23.420 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:23.459 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:23.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:23.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:23.561 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:23.590 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:23.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:23.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:23.692 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:23.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:23.723 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:23.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:23.824 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:23.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:23 smithi184 ceph-mon[92527]: 1.4 scrub starts 2024-09-13T14:41:23.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:23 smithi184 ceph-mon[92527]: 1.4 scrub ok 2024-09-13T14:41:23.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:23 smithi151 ceph-mon[104450]: 1.4 scrub starts 2024-09-13T14:41:23.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:23 smithi151 ceph-mon[104450]: 1.4 scrub ok 2024-09-13T14:41:23.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:23 smithi151 ceph-mon[108966]: 1.4 scrub starts 2024-09-13T14:41:23.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:23 smithi151 ceph-mon[108966]: 1.4 scrub ok 2024-09-13T14:41:23.860 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:23.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:23.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:23.962 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:23.997 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T14:41:24.015 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:24.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:24.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:24.117 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:24.145 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:24.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:24.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:24.247 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:24.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:24.290 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:24.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:24.391 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:24.419 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:24.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:24.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:24.453 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:41:24.522 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:24.544 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:41:24.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:24.558 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:24.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:24.659 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:24.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:24.698 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:24.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:24.780 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:24 smithi184 ceph-mon[92527]: 1.5 scrub starts 2024-09-13T14:41:24.780 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:24 smithi184 ceph-mon[92527]: 1.5 scrub ok 2024-09-13T14:41:24.781 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:24 smithi184 ceph-mon[92527]: 1.0 scrub starts 2024-09-13T14:41:24.781 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:24 smithi184 ceph-mon[92527]: 1.0 scrub ok 2024-09-13T14:41:24.781 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:24 smithi184 ceph-mon[92527]: pgmap v1366: 99 pgs: 99 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 892 op/s; 96 KiB/s, 0 objects/s recovering 2024-09-13T14:41:24.781 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:24 smithi184 ceph-mon[92527]: 1.1 scrub starts 2024-09-13T14:41:24.781 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:24 smithi184 ceph-mon[92527]: 1.1 scrub ok 2024-09-13T14:41:24.786 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:24 smithi151 ceph-mon[104450]: 1.5 scrub starts 2024-09-13T14:41:24.786 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:24 smithi151 ceph-mon[104450]: 1.5 scrub ok 2024-09-13T14:41:24.786 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:24 smithi151 ceph-mon[104450]: 1.0 scrub starts 2024-09-13T14:41:24.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:24 smithi151 ceph-mon[104450]: 1.0 scrub ok 2024-09-13T14:41:24.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:24 smithi151 ceph-mon[104450]: pgmap v1366: 99 pgs: 99 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 892 op/s; 96 KiB/s, 0 objects/s recovering 2024-09-13T14:41:24.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:24 smithi151 ceph-mon[104450]: 1.1 scrub starts 2024-09-13T14:41:24.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:24 smithi151 ceph-mon[104450]: 1.1 scrub ok 2024-09-13T14:41:24.787 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:41:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:41:24] "GET /metrics HTTP/1.1" 200 36669 "" "Prometheus/2.51.0" 2024-09-13T14:41:24.787 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:24 smithi151 ceph-mon[108966]: 1.5 scrub starts 2024-09-13T14:41:24.787 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:24 smithi151 ceph-mon[108966]: 1.5 scrub ok 2024-09-13T14:41:24.787 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:24 smithi151 ceph-mon[108966]: 1.0 scrub starts 2024-09-13T14:41:24.788 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:24 smithi151 ceph-mon[108966]: 1.0 scrub ok 2024-09-13T14:41:24.788 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:24 smithi151 ceph-mon[108966]: pgmap v1366: 99 pgs: 99 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 892 op/s; 96 KiB/s, 0 objects/s recovering 2024-09-13T14:41:24.788 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:24 smithi151 ceph-mon[108966]: 1.1 scrub starts 2024-09-13T14:41:24.788 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:24 smithi151 ceph-mon[108966]: 1.1 scrub ok 2024-09-13T14:41:24.799 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:24.832 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:24.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:24.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:24.938 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:24.969 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:24.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:24.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:25.071 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:25.105 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:25.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:25.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:25.207 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:25.245 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:25.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:25.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:25.347 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:25.378 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:25.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:25.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:25.480 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:25.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:25.524 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:25.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:25.625 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:25.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:25.671 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:25.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:25.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:25 smithi151 ceph-mon[104450]: 1.2 scrub starts 2024-09-13T14:41:25.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:25 smithi151 ceph-mon[104450]: 1.2 scrub ok 2024-09-13T14:41:25.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:25 smithi151 ceph-mon[104450]: 1.6 scrub starts 2024-09-13T14:41:25.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:25 smithi151 ceph-mon[104450]: 1.6 scrub ok 2024-09-13T14:41:25.700 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:25 smithi151 ceph-mon[108966]: 1.2 scrub starts 2024-09-13T14:41:25.700 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:25 smithi151 ceph-mon[108966]: 1.2 scrub ok 2024-09-13T14:41:25.700 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:25 smithi151 ceph-mon[108966]: 1.6 scrub starts 2024-09-13T14:41:25.700 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:25 smithi151 ceph-mon[108966]: 1.6 scrub ok 2024-09-13T14:41:25.751 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:25 smithi184 ceph-mon[92527]: 1.2 scrub starts 2024-09-13T14:41:25.751 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:25 smithi184 ceph-mon[92527]: 1.2 scrub ok 2024-09-13T14:41:25.751 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:25 smithi184 ceph-mon[92527]: 1.6 scrub starts 2024-09-13T14:41:25.751 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:25 smithi184 ceph-mon[92527]: 1.6 scrub ok 2024-09-13T14:41:25.772 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:25.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:25.807 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:25.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:25.909 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:25.943 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:25.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:25.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:26.045 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:26.080 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:26.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:26.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:26.182 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:26.209 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:26.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:26.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:26.311 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:26.353 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:26.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:26.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:26.454 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:26.485 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:41:26.485 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:41:26.495 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:26.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:26.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:26.523 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:41:26.524 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:41:26.560 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:41:26.560 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:41:26.561 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:26 smithi151 ceph-mon[108966]: pgmap v1367: 99 pgs: 99 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 782 op/s; 96 KiB/s, 0 objects/s recovering 2024-09-13T14:41:26.561 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:26 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3601622021' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:26.561 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:26 smithi151 ceph-mon[104450]: pgmap v1367: 99 pgs: 99 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 782 op/s; 96 KiB/s, 0 objects/s recovering 2024-09-13T14:41:26.561 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:26 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3601622021' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:26.586 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:41:26.586 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:41:26.597 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:26.621 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:41:26.621 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:41:26.647 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:26.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:26.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:26.650 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:41:26.651 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:41:26.684 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:41:26.684 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:41:26.687 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:41:26.687 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:41:26.714 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:41:26.714 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:41:26.749 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:26.785 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:26.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:26.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:26.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:26 smithi184 ceph-mon[92527]: pgmap v1367: 99 pgs: 99 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 782 op/s; 96 KiB/s, 0 objects/s recovering 2024-09-13T14:41:26.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:26 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3601622021' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:26.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:26 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a[104427]: 2024-09-13T14:41:26.584+0000 7fe9da14f640 -1 mon.a@0(leader).osd e1101 definitely_dead 0 2024-09-13T14:41:26.887 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:26.913 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:26.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:26.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:27.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:27.066 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:27.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:27.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:27.169 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:27.205 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:27.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:27.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:27.307 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:27.334 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:27.335 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:27.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:27.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:27.434 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T14:41:27.436 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:27.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:27.478 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:27.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:27.568 INFO:teuthology.orchestra.run.smithi151.stderr:marked out osd.1. 2024-09-13T14:41:27.579 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:27.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:27.622 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:27.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:27.723 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:27.745 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:27.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:27.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:27.749 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:41:27.781 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:27 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:41:27.781 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:27 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2175929981' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-13T14:41:27.781 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:27 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-13T14:41:27.781 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:27 smithi151 ceph-mon[104450]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-13T14:41:27.782 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:27 smithi151 ceph-mon[104450]: 1.3 scrub starts 2024-09-13T14:41:27.782 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:27 smithi151 ceph-mon[104450]: 1.3 scrub ok 2024-09-13T14:41:27.782 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:27 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:41:27.782 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:27 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:41:27.782 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:27 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:41:27.782 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:27 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2175929981' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-13T14:41:27.782 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:27 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-13T14:41:27.782 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:27 smithi151 ceph-mon[108966]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-13T14:41:27.782 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:27 smithi151 ceph-mon[108966]: 1.3 scrub starts 2024-09-13T14:41:27.782 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:27 smithi151 ceph-mon[108966]: 1.3 scrub ok 2024-09-13T14:41:27.782 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:27 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:41:27.783 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:27 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:41:27.847 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:27.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:27 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:41:27.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:27 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2175929981' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-13T14:41:27.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:27 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-13T14:41:27.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:27 smithi184 ceph-mon[92527]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-13T14:41:27.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:27 smithi184 ceph-mon[92527]: 1.3 scrub starts 2024-09-13T14:41:27.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:27 smithi184 ceph-mon[92527]: 1.3 scrub ok 2024-09-13T14:41:27.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:27 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:41:27.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:27 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:41:27.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:27.871 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:27.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:27.972 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:27.993 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:27.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:27.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:28.095 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:28.116 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:28.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:28.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:28.218 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:28.239 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:28.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:28.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:28.341 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:28.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:28.363 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:28.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:28.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:28.485 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:28.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:28.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:28.587 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:28.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:28 smithi151 ceph-mon[104450]: pgmap v1368: 99 pgs: 99 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 829 op/s; 60 KiB/s, 0 objects/s recovering 2024-09-13T14:41:28.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:28 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-13T14:41:28.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:28 smithi151 ceph-mon[104450]: osdmap e1102: 8 total, 8 up, 7 in 2024-09-13T14:41:28.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:28 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:41:28.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:28 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:41:28.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:28 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:41:28.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:28 smithi151 ceph-mon[108966]: pgmap v1368: 99 pgs: 99 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 829 op/s; 60 KiB/s, 0 objects/s recovering 2024-09-13T14:41:28.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:28 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-13T14:41:28.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:28 smithi151 ceph-mon[108966]: osdmap e1102: 8 total, 8 up, 7 in 2024-09-13T14:41:28.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:28 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:41:28.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:28 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:41:28.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:28 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:41:28.613 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:28.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:28.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:28.715 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:28.735 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:28.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:28.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:28.837 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:28.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:28 smithi184 ceph-mon[92527]: pgmap v1368: 99 pgs: 99 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 829 op/s; 60 KiB/s, 0 objects/s recovering 2024-09-13T14:41:28.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:28 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-13T14:41:28.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:28 smithi184 ceph-mon[92527]: osdmap e1102: 8 total, 8 up, 7 in 2024-09-13T14:41:28.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:28 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:41:28.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:28 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:41:28.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:28 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:41:28.859 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:28.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:28.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:28.961 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:28.988 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:28.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:28.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:29.089 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:29.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:29.112 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:29.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:29.213 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:29.234 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:29.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:29.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:29.335 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:29.362 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:29.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:29.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:29.464 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:29.490 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:29.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:29.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:29.592 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:29.617 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:29.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:29.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:29.719 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:29.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:29.768 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:29.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:29.805 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:41:29.829 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:29 smithi151 ceph-mon[104450]: 1.9 scrub starts 2024-09-13T14:41:29.829 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:29 smithi151 ceph-mon[104450]: 1.9 scrub ok 2024-09-13T14:41:29.829 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:29 smithi151 ceph-mon[104450]: osdmap e1103: 8 total, 8 up, 7 in 2024-09-13T14:41:29.829 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:29 smithi151 ceph-mon[108966]: 1.9 scrub starts 2024-09-13T14:41:29.829 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:29 smithi151 ceph-mon[108966]: 1.9 scrub ok 2024-09-13T14:41:29.830 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:29 smithi151 ceph-mon[108966]: osdmap e1103: 8 total, 8 up, 7 in 2024-09-13T14:41:29.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:29 smithi184 ceph-mon[92527]: 1.9 scrub starts 2024-09-13T14:41:29.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:29 smithi184 ceph-mon[92527]: 1.9 scrub ok 2024-09-13T14:41:29.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:29 smithi184 ceph-mon[92527]: osdmap e1103: 8 total, 8 up, 7 in 2024-09-13T14:41:29.870 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:29.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:29.896 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:29.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:29.997 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:30.026 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:30.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:30.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:30.128 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:30.152 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:30.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:30.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:30.179 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:41:30.179 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:41:30.254 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:30.301 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:30.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:30.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:30.403 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:30.442 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:30.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:30.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:30.544 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:30.559 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:41:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:41:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:41:30.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:30.582 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:30.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:30.605 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:30.683 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:30.720 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:30.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:30.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:30.734 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:30 smithi151 ceph-mon[104450]: pgmap v1371: 99 pgs: 1 remapped+peering, 98 active+clean; 6.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 1.14k op/s; 0 B/s, 0 objects/s recovering 2024-09-13T14:41:30.734 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:30 smithi151 ceph-mon[104450]: 6.e starting backfill to osd.5 from (0'0,0'0] MAX to 1101'9938 2024-09-13T14:41:30.734 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:30 smithi151 ceph-mon[104450]: osdmap e1104: 8 total, 8 up, 7 in 2024-09-13T14:41:30.734 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:30 smithi151 ceph-mon[104450]: 6.1 starting backfill to osd.7 from (0'0,0'0] MAX to 1101'9855 2024-09-13T14:41:30.735 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:30 smithi151 ceph-mon[104450]: 6.d starting backfill to osd.2 from (0'0,0'0] MAX to 1101'9972 2024-09-13T14:41:30.735 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:30 smithi151 ceph-mon[104450]: 6.4 starting backfill to osd.4 from (0'0,0'0] MAX to 1101'10221 2024-09-13T14:41:30.735 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:30 smithi151 ceph-mon[104450]: 1.a scrub starts 2024-09-13T14:41:30.735 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:30 smithi151 ceph-mon[104450]: 1.a scrub ok 2024-09-13T14:41:30.735 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:30 smithi151 ceph-mon[108966]: pgmap v1371: 99 pgs: 1 remapped+peering, 98 active+clean; 6.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 1.14k op/s; 0 B/s, 0 objects/s recovering 2024-09-13T14:41:30.735 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:30 smithi151 ceph-mon[108966]: 6.e starting backfill to osd.5 from (0'0,0'0] MAX to 1101'9938 2024-09-13T14:41:30.735 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:30 smithi151 ceph-mon[108966]: osdmap e1104: 8 total, 8 up, 7 in 2024-09-13T14:41:30.735 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:30 smithi151 ceph-mon[108966]: 6.1 starting backfill to osd.7 from (0'0,0'0] MAX to 1101'9855 2024-09-13T14:41:30.735 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:30 smithi151 ceph-mon[108966]: 6.d starting backfill to osd.2 from (0'0,0'0] MAX to 1101'9972 2024-09-13T14:41:30.735 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:30 smithi151 ceph-mon[108966]: 6.4 starting backfill to osd.4 from (0'0,0'0] MAX to 1101'10221 2024-09-13T14:41:30.735 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:30 smithi151 ceph-mon[108966]: 1.a scrub starts 2024-09-13T14:41:30.736 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:30 smithi151 ceph-mon[108966]: 1.a scrub ok 2024-09-13T14:41:30.822 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:30.846 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:30.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:30.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:30.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:30 smithi184 ceph-mon[92527]: pgmap v1371: 99 pgs: 1 remapped+peering, 98 active+clean; 6.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 1.14k op/s; 0 B/s, 0 objects/s recovering 2024-09-13T14:41:30.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:30 smithi184 ceph-mon[92527]: 6.e starting backfill to osd.5 from (0'0,0'0] MAX to 1101'9938 2024-09-13T14:41:30.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:30 smithi184 ceph-mon[92527]: osdmap e1104: 8 total, 8 up, 7 in 2024-09-13T14:41:30.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:30 smithi184 ceph-mon[92527]: 6.1 starting backfill to osd.7 from (0'0,0'0] MAX to 1101'9855 2024-09-13T14:41:30.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:30 smithi184 ceph-mon[92527]: 6.d starting backfill to osd.2 from (0'0,0'0] MAX to 1101'9972 2024-09-13T14:41:30.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:30 smithi184 ceph-mon[92527]: 6.4 starting backfill to osd.4 from (0'0,0'0] MAX to 1101'10221 2024-09-13T14:41:30.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:30 smithi184 ceph-mon[92527]: 1.a scrub starts 2024-09-13T14:41:30.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:30 smithi184 ceph-mon[92527]: 1.a scrub ok 2024-09-13T14:41:30.936 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T14:41:30.947 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:31.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:31.030 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:31.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:31.131 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:31.165 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:31.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:31.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:31.267 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:31.293 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:31.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:31.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:31.383 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:41:31.395 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:31.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:31.451 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:31.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:31.552 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:31.576 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:31.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:31.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:31.596 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-13T14:41:31.677 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:31.709 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:31.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:31.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:31.811 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:31.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:31.839 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:31.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:31.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:31 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/866334605' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:31.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:31 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:31.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:31.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:31 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/866334605' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:31.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:31 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:31.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:31.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:31 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/866334605' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:31.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:31 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:31.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:31.941 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:31.975 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:31.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:31.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:32.077 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:32.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:32.108 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:32.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:32.209 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:32.235 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:32.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:32.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:32.337 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:32.359 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:32.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:32.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:32.461 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:32.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:32.505 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:32.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:32.606 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:32.637 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:32.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:32.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:32.739 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:32.763 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:32.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:32.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:32.852 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:32 smithi151 ceph-mon[104450]: pgmap v1373: 99 pgs: 1 remapped+peering, 98 active+clean; 6.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 778 op/s 2024-09-13T14:41:32.852 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:32 smithi151 ceph-mon[104450]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:41:32.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:32 smithi151 ceph-mon[104450]: Cluster is now healthy 2024-09-13T14:41:32.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:32 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:41:32.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:32 smithi151 ceph-mon[104450]: osdmap e1105: 8 total, 8 up, 7 in 2024-09-13T14:41:32.853 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:32 smithi151 ceph-mon[108966]: pgmap v1373: 99 pgs: 1 remapped+peering, 98 active+clean; 6.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 778 op/s 2024-09-13T14:41:32.853 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:32 smithi151 ceph-mon[108966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:41:32.853 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:32 smithi151 ceph-mon[108966]: Cluster is now healthy 2024-09-13T14:41:32.853 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:32 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:41:32.853 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:32 smithi151 ceph-mon[108966]: osdmap e1105: 8 total, 8 up, 7 in 2024-09-13T14:41:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:32 smithi184 ceph-mon[92527]: pgmap v1373: 99 pgs: 1 remapped+peering, 98 active+clean; 6.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 778 op/s 2024-09-13T14:41:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:32 smithi184 ceph-mon[92527]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:41:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:32 smithi184 ceph-mon[92527]: Cluster is now healthy 2024-09-13T14:41:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:32 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:41:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:32 smithi184 ceph-mon[92527]: osdmap e1105: 8 total, 8 up, 7 in 2024-09-13T14:41:32.866 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:32.900 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:32.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:32.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:33.002 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:33.039 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:33.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:33.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:33.140 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:33.141 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:41:33.142 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:41:33.175 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:33.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:33.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:33.177 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:41:33.177 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:41:33.207 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:41:33.207 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:41:33.235 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:41:33.235 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:41:33.264 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:41:33.264 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:41:33.277 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:33.288 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:41:33.288 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:41:33.305 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:33.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:33.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:33.315 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:41:33.315 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:41:33.348 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:41:33.348 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:41:33.407 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:33.432 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:33.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:33.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:33.534 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:33.564 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:33.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:33.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:33.665 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:33.666 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:41:33.667 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:41:33.667 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_1 2024-09-13T14:41:33.667 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:41:33.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:33 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3883437700' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:33.697 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:33 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3883437700' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:33.720 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:33.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:33.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:33.822 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:33.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:33.867 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:33.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:33.885 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:33 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3883437700' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:33.969 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:34.007 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:34.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:34.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:34.109 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:34.147 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:34.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:34.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:34.208 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:34.248 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:34.280 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:34.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:34.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:34.358 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:41:34.359 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [ 2024-09-13T14:41:34.359 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.359 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35289.0:163320 6.b e1105/1103 6:dc363c65:::benchmark_data_smithi151_155836_object30862:head v 1105'10287, mlcod=1087'1957)", 2024-09-13T14:41:34.359 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:34.339031+0000", 2024-09-13T14:41:34.359 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.014873541000000001, 2024-09-13T14:41:34.359 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.014902128000000001, 2024-09-13T14:41:34.360 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:34.360 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:41:34.360 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:34.360 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.360 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:34.360 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.339031+0000", 2024-09-13T14:41:34.360 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:34.360 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.360 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.361 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:34.361 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.339031+0000", 2024-09-13T14:41:34.361 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:34.361 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.362 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.362 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:34.362 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.339030+0000", 2024-09-13T14:41:34.362 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:41:34.362 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.362 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.362 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:34.362 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.339036+0000", 2024-09-13T14:41:34.362 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.0870000000000003e-06 2024-09-13T14:41:34.362 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.362 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.339050+0000", 2024-09-13T14:41:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3862999999999999e-05 2024-09-13T14:41:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.339055+0000", 2024-09-13T14:41:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3510000000000001e-06 2024-09-13T14:41:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.343861+0000", 2024-09-13T14:41:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0048052879999999996 2024-09-13T14:41:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.343890+0000", 2024-09-13T14:41:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.9770000000000001e-05 2024-09-13T14:41:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:163322 6.1 6:8586e939:::benchmark_data_smithi151_155836_object30864:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1105)", 2024-09-13T14:41:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:34.343297+0000", 2024-09-13T14:41:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.010607488999999999, 2024-09-13T14:41:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.010706512, 2024-09-13T14:41:34.365 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:34.365 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:41:34.365 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:34.365 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:34.365 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:34.365 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 163322 2024-09-13T14:41:34.365 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.365 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:34.366 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.366 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:34.366 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.343297+0000", 2024-09-13T14:41:34.366 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:34.366 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.366 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.366 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:34.366 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.343297+0000", 2024-09-13T14:41:34.366 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:34.366 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.366 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.366 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:34.367 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.343295+0000", 2024-09-13T14:41:34.367 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:34.367 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.367 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.367 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:34.367 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.343301+0000", 2024-09-13T14:41:34.367 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1489999999999999e-06 2024-09-13T14:41:34.367 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.367 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.368 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:34.368 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.343314+0000", 2024-09-13T14:41:34.368 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2901000000000001e-05 2024-09-13T14:41:34.368 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.368 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.368 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:34.368 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.343320+0000", 2024-09-13T14:41:34.368 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1800000000000001e-06 2024-09-13T14:41:34.368 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.369 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.369 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:34.369 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.343365+0000", 2024-09-13T14:41:34.369 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4199000000000002e-05 2024-09-13T14:41:34.369 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.370 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.370 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:34.370 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.343477+0000", 2024-09-13T14:41:34.370 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000112812 2024-09-13T14:41:34.370 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.370 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.371 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2,7", 2024-09-13T14:41:34.371 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.343606+0000", 2024-09-13T14:41:34.371 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012860399999999999 2024-09-13T14:41:34.371 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.371 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.371 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:34.371 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.347002+0000", 2024-09-13T14:41:34.372 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.003395716 2024-09-13T14:41:34.372 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.372 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.372 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:34.372 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.348693+0000", 2024-09-13T14:41:34.372 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0016913430000000001 2024-09-13T14:41:34.372 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.372 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.373 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:34.373 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.350833+0000", 2024-09-13T14:41:34.373 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0021401129999999999 2024-09-13T14:41:34.373 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:34.373 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:34.373 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:34.373 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.373 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.373 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35289.0:163314 6.8 e1105/1082 6:1cd5de26:::benchmark_data_smithi151_155836_object30856:head v 1105'10334, mlcod=1105'10330)", 2024-09-13T14:41:34.374 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:34.345938+0000", 2024-09-13T14:41:34.374 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.007966473, 2024-09-13T14:41:34.374 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.008159075, 2024-09-13T14:41:34.374 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:34.374 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:41:34.374 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:34.374 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.374 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:34.374 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.345938+0000", 2024-09-13T14:41:34.374 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:34.374 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.374 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.375 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:34.375 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.345938+0000", 2024-09-13T14:41:34.375 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:34.375 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.375 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.375 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:34.375 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.345937+0000", 2024-09-13T14:41:34.375 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:41:34.375 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.375 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.375 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:34.375 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.345944+0000", 2024-09-13T14:41:34.375 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.5340000000000002e-06 2024-09-13T14:41:34.376 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.376 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.376 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:34.376 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.345957+0000", 2024-09-13T14:41:34.376 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2838e-05 2024-09-13T14:41:34.376 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.376 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.376 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:34.376 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.345963+0000", 2024-09-13T14:41:34.376 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.0170000000000002e-06 2024-09-13T14:41:34.376 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.376 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.376 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:34.377 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.346527+0000", 2024-09-13T14:41:34.377 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00056331700000000003 2024-09-13T14:41:34.377 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.377 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.377 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:34.377 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.346544+0000", 2024-09-13T14:41:34.377 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.7099999999999999e-05 2024-09-13T14:41:34.377 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:34.377 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:34.377 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:34.377 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.377 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.377 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35289.0:163317 6.8 e1105/1082 6:14bf6ed0:::benchmark_data_smithi151_155836_object30859:head v 1105'10335, mlcod=1105'10330)", 2024-09-13T14:41:34.378 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:34.345981+0000", 2024-09-13T14:41:34.378 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.0079235269999999997, 2024-09-13T14:41:34.378 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0081688980000000008, 2024-09-13T14:41:34.378 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:34.378 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:41:34.378 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:34.378 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.378 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:34.378 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.345981+0000", 2024-09-13T14:41:34.378 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:34.379 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.379 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.379 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:34.379 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.345981+0000", 2024-09-13T14:41:34.379 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:34.379 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.379 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.379 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:34.379 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.345980+0000", 2024-09-13T14:41:34.380 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:41:34.380 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.380 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.380 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:34.380 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.345984+0000", 2024-09-13T14:41:34.380 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.2710000000000001e-06 2024-09-13T14:41:34.380 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.380 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.380 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:34.380 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.345989+0000", 2024-09-13T14:41:34.380 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4310000000000002e-06 2024-09-13T14:41:34.380 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.380 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.380 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:34.381 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.345992+0000", 2024-09-13T14:41:34.381 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3000000000000002e-06 2024-09-13T14:41:34.381 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.381 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.381 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:34.381 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.353721+0000", 2024-09-13T14:41:34.381 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0077283430000000004 2024-09-13T14:41:34.381 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.381 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.381 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:34.381 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.353765+0000", 2024-09-13T14:41:34.381 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4141999999999999e-05 2024-09-13T14:41:34.381 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:34.382 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:34.382 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:34.382 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.382 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.382 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35289.0:163323 6.3 e1105/1082)", 2024-09-13T14:41:34.382 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:34.348732+0000", 2024-09-13T14:41:34.382 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.005171859, 2024-09-13T14:41:34.382 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0054684039999999996, 2024-09-13T14:41:34.382 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:34.382 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "queued for pg", 2024-09-13T14:41:34.382 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:34.383 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.383 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:34.383 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.348732+0000", 2024-09-13T14:41:34.383 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:34.383 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.383 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.383 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:34.383 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.348732+0000", 2024-09-13T14:41:34.383 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:34.383 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.383 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.383 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:34.383 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.348732+0000", 2024-09-13T14:41:34.383 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:41:34.384 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.384 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.384 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:34.384 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.348738+0000", 2024-09-13T14:41:34.384 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.0870000000000003e-06 2024-09-13T14:41:34.384 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.384 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.384 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:34.384 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.348755+0000", 2024-09-13T14:41:34.384 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.7436e-05 2024-09-13T14:41:34.384 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.384 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.385 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:34.385 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.348760+0000", 2024-09-13T14:41:34.385 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7299999999999996e-06 2024-09-13T14:41:34.385 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:34.385 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:34.385 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:34.385 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.385 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.385 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35289.0:163324 6.a e1105/1082 6:5cefbf08:::benchmark_data_smithi151_155836_object30866:head v 1105'10379, mlcod=1105'10377)", 2024-09-13T14:41:34.385 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:34.351546+0000", 2024-09-13T14:41:34.385 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.0023584259999999998, 2024-09-13T14:41:34.385 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0027133130000000002, 2024-09-13T14:41:34.385 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:34.386 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:41:34.386 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:34.386 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.386 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:34.386 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.351546+0000", 2024-09-13T14:41:34.386 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:34.386 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.386 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.386 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:34.386 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.351546+0000", 2024-09-13T14:41:34.386 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:34.386 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.386 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.386 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:34.387 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.351545+0000", 2024-09-13T14:41:34.387 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:41:34.387 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.387 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.387 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:34.387 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.351553+0000", 2024-09-13T14:41:34.387 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.8220000000000007e-06 2024-09-13T14:41:34.387 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.387 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.387 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:34.387 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.351567+0000", 2024-09-13T14:41:34.387 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4491000000000001e-05 2024-09-13T14:41:34.387 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.387 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.388 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:34.388 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.351573+0000", 2024-09-13T14:41:34.388 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6740000000000002e-06 2024-09-13T14:41:34.388 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.388 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.388 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:34.388 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.351655+0000", 2024-09-13T14:41:34.388 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.1990000000000006e-05 2024-09-13T14:41:34.388 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.388 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.388 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:34.388 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.351678+0000", 2024-09-13T14:41:34.389 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.3612e-05 2024-09-13T14:41:34.389 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:34.389 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:34.389 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:34.389 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.389 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.389 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop_reply(client.35289.0:163322 6.1 e1105/1103 ondisk, result = 0)", 2024-09-13T14:41:34.389 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:34.354218+0000", 2024-09-13T14:41:34.389 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 4294967295.9996862, 2024-09-13T14:41:34.389 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.5545999999999996e-05, 2024-09-13T14:41:34.389 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:34.389 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:41:34.389 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:34.390 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.390 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:34.390 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.354218+0000", 2024-09-13T14:41:34.390 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:34.390 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.390 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.390 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:34.390 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.354218+0000", 2024-09-13T14:41:34.390 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:34.390 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.390 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.390 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:34.390 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.354217+0000", 2024-09-13T14:41:34.390 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:41:34.391 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.391 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.391 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:34.391 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.354222+0000", 2024-09-13T14:41:34.391 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5719999999999996e-06 2024-09-13T14:41:34.391 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.391 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.391 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:34.391 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.354235+0000", 2024-09-13T14:41:34.391 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3461e-05 2024-09-13T14:41:34.391 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.391 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.392 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:34.392 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.354241+0000", 2024-09-13T14:41:34.392 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4700000000000001e-06 2024-09-13T14:41:34.392 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.392 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.392 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:34.392 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.354294+0000", 2024-09-13T14:41:34.392 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3545999999999999e-05 2024-09-13T14:41:34.392 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.392 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.392 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:34.393 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.354301+0000", 2024-09-13T14:41:34.393 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.8290000000000002e-06 2024-09-13T14:41:34.393 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:34.393 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:34.393 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:34.393 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.393 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.393 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35289.0:163329 6.b e1105/1103)", 2024-09-13T14:41:34.393 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:34.354236+0000", 2024-09-13T14:41:34.393 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 4294967295.9996681, 2024-09-13T14:41:34.394 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012590999999999999, 2024-09-13T14:41:34.394 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:34.394 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "queued for pg", 2024-09-13T14:41:34.394 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:34.394 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.394 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:34.394 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.354236+0000", 2024-09-13T14:41:34.395 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:34.395 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.395 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.395 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:34.395 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.354236+0000", 2024-09-13T14:41:34.395 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:34.395 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.395 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.395 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:34.396 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.354235+0000", 2024-09-13T14:41:34.396 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:41:34.396 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.396 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.396 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:34.396 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.354242+0000", 2024-09-13T14:41:34.396 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.5440000000000004e-06 2024-09-13T14:41:34.396 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.396 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.396 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:34.396 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.354251+0000", 2024-09-13T14:41:34.396 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.3689999999999996e-06 2024-09-13T14:41:34.397 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:34.397 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:34.397 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:34.397 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:34.354256+0000", 2024-09-13T14:41:34.397 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8350000000000003e-06 2024-09-13T14:41:34.397 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:34.397 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:34.397 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:34.397 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:34.397 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-13T14:41:34.397 INFO:teuthology.orchestra.run.smithi184.stdout: "num_ops": 8 2024-09-13T14:41:34.397 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:41:34.398 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:34.439 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:34.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:34.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:34.541 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:34.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:34.582 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:34.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:34.640 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:41:34.684 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:34.727 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:34.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:34.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:34.829 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:34.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:34 smithi151 ceph-mon[104450]: pgmap v1375: 99 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 remapped+peering, 94 active+clean; 6.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 436 op/s; 25051/310830 objects misplaced (8.059%); 7.4 MiB/s, 119 objects/s recovering 2024-09-13T14:41:34.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:41:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:41:34] "GET /metrics HTTP/1.1" 200 36666 "" "Prometheus/2.51.0" 2024-09-13T14:41:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:34 smithi151 ceph-mon[108966]: pgmap v1375: 99 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 remapped+peering, 94 active+clean; 6.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 436 op/s; 25051/310830 objects misplaced (8.059%); 7.4 MiB/s, 119 objects/s recovering 2024-09-13T14:41:34.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:34.862 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:34.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:34.964 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:35.000 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:35.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:35.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:35.102 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:35.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:34 smithi184 ceph-mon[92527]: pgmap v1375: 99 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 remapped+peering, 94 active+clean; 6.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 436 op/s; 25051/310830 objects misplaced (8.059%); 7.4 MiB/s, 119 objects/s recovering 2024-09-13T14:41:35.135 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:35.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:35.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:35.178 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T14:41:35.238 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:35.308 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:35.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:35.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:35.410 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:35.436 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:35.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:35.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:35.538 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:35.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:35.695 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:35.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:35.696 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:41:35.797 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:35.831 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:35.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:35.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:35.840 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:35.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:35.962 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:35.962 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:35 smithi184 ceph-mon[92527]: Health check failed: Degraded data redundancy: 141/309132 objects degraded (0.046%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:41:35.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:35.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:35.970 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:35 smithi151 ceph-mon[104450]: Health check failed: Degraded data redundancy: 141/309132 objects degraded (0.046%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:41:35.970 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:35 smithi151 ceph-mon[108966]: Health check failed: Degraded data redundancy: 141/309132 objects degraded (0.046%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:41:36.064 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:36.099 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:36.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:36.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:36.121 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:41:36.202 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:36.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:36.238 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:36.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:36.339 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:36.373 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:36.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:36.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:36.475 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:36.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:36.516 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:36.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:36.617 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:36.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:36.653 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:36.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:36.755 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:36.783 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:36.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:36.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:36.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:36 smithi151 ceph-mon[104450]: pgmap v1376: 99 pgs: 2 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 456 op/s; 141/309132 objects degraded (0.046%); 31449/309132 objects misplaced (10.173%); 6.8 MiB/s, 109 objects/s recovering 2024-09-13T14:41:36.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:36 smithi151 ceph-mon[108966]: pgmap v1376: 99 pgs: 2 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 456 op/s; 141/309132 objects degraded (0.046%); 31449/309132 objects misplaced (10.173%); 6.8 MiB/s, 109 objects/s recovering 2024-09-13T14:41:36.885 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:36.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:36.930 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:36.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:37.011 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:36 smithi184 ceph-mon[92527]: pgmap v1376: 99 pgs: 2 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 456 op/s; 141/309132 objects degraded (0.046%); 31449/309132 objects misplaced (10.173%); 6.8 MiB/s, 109 objects/s recovering 2024-09-13T14:41:37.031 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:37.086 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:37.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:37.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:37.188 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:37.231 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:37.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:37.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:37.326 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:41:37.328 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T14:41:37.332 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:37.360 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:37.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:37.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:37.462 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:37.506 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:37.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:37.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:37.608 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:37.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:37.650 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:37.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:37.676 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:41:37.676 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [], 2024-09-13T14:41:37.676 INFO:teuthology.orchestra.run.smithi184.stdout: "complaint_time": 30, 2024-09-13T14:41:37.676 INFO:teuthology.orchestra.run.smithi184.stdout: "num_blocked_ops": 0 2024-09-13T14:41:37.677 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:41:37.751 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:37.782 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:37.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:37.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:37.884 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:37.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:37.922 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:37.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:38.023 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:38.060 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:38.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:38.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:38.162 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:38.185 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:38.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:38.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:38.287 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:38.311 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:38.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:38.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:38.368 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T14:41:38.414 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:38.470 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:38.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:38.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:38.572 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:38.608 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:38.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:38.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:38.664 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1376,"stamp":"2024-09-13T14:41:35.346719+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6751881911,"num_objects":103044,"num_object_clones":0,"num_object_copies":309132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":141,"num_objects_misplaced":31449,"num_objects_unfound":0,"num_objects_dirty":103044,"num_whiteouts":0,"num_read":80,"num_read_kb":205,"num_write":161996,"num_write_kb":8477596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":745,"num_bytes_recovered":48723749,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106953,"ondisk_log_size":106953,"up":297,"acting":296,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":292,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":19470740,"kb_used_data":17928384,"kb_used_omap":116,"kb_used_meta":1541963,"kb_avail":636745324,"statfs":{"total":671965249536,"available":652027211776,"internally_reserved":0,"allocated":18358665216,"data_stored":18336113535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":119649,"internal_metadata":1578970271},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-203411877,"num_objects":-3103,"num_object_clones":0,"num_object_copies":-9309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":141,"num_objects_misplaced":31449,"num_objects_unfound":0,"num_objects_dirty":-3103,"num_whiteouts":0,"num_read":-27,"num_read_kb":-69,"num_write":3091,"num_write_kb":-148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":742,"num_bytes_recovered":48246382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.776449"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":1521,"reported_epoch":1105,"state":"active+clean","last_fresh":"2024-09-13T14:41:31.605612+0000","last_change":"2024-09-13T14:41:28.592397+0000","last_active":"2024-09-13T14:41:31.605612+0000","last_peered":"2024-09-13T14:41:31.605612+0000","last_clean":"2024-09-13T14:41:31.605612+0000","last_became_active":"2024-09-13T14:41:28.592068+0000","last_became_peered":"2024-09-13T14:41:28.592068+0000","last_unstale":"2024-09-13T14:41:31.605612+0000","last_undegraded":"2024-09-13T14:41:31.605612+0000","last_fullsized":"2024-09-13T14:41:31.605612+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:08:03.576401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":1061,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.577569+0000","last_change":"2024-09-13T14:29:42.541093+0000","last_active":"2024-09-13T14:41:29.577569+0000","last_peered":"2024-09-13T14:41:29.577569+0000","last_clean":"2024-09-13T14:41:29.577569+0000","last_became_active":"2024-09-13T14:29:42.540671+0000","last_became_peered":"2024-09-13T14:29:42.540671+0000","last_unstale":"2024-09-13T14:41:29.577569+0000","last_undegraded":"2024-09-13T14:41:29.577569+0000","last_fullsized":"2024-09-13T14:41:29.577569+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:45.722376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1057,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.591506+0000","last_change":"2024-09-13T14:39:11.379707+0000","last_active":"2024-09-13T14:41:29.591506+0000","last_peered":"2024-09-13T14:41:29.591506+0000","last_clean":"2024-09-13T14:41:29.591506+0000","last_became_active":"2024-09-13T14:39:11.379255+0000","last_became_peered":"2024-09-13T14:39:11.379255+0000","last_unstale":"2024-09-13T14:41:29.591506+0000","last_undegraded":"2024-09-13T14:41:29.591506+0000","last_fullsized":"2024-09-13T14:41:29.591506+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:48.543270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1623,"reported_epoch":1105,"state":"active+clean","last_fresh":"2024-09-13T14:41:31.612598+0000","last_change":"2024-09-13T14:41:28.585446+0000","last_active":"2024-09-13T14:41:31.612598+0000","last_peered":"2024-09-13T14:41:31.612598+0000","last_clean":"2024-09-13T14:41:31.612598+0000","last_became_active":"2024-09-13T14:41:28.584679+0000","last_became_peered":"2024-09-13T14:41:28.584679+0000","last_unstale":"2024-09-13T14:41:31.612598+0000","last_undegraded":"2024-09-13T14:41:31.612598+0000","last_fullsized":"2024-09-13T14:41:31.612598+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:19:12.592060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1583,"reported_epoch":1105,"state":"active+clean","last_fresh":"2024-09-13T14:41:31.611119+0000","last_change":"2024-09-13T14:39:11.394501+0000","last_active":"2024-09-13T14:41:31.611119+0000","last_peered":"2024-09-13T14:41:31.611119+0000","last_clean":"2024-09-13T14:41:31.611119+0000","last_became_active":"2024-09-13T14:39:11.394134+0000","last_became_peered":"2024-09-13T14:39:11.394134+0000","last_unstale":"2024-09-13T14:41:31.611119+0000","last_undegraded":"2024-09-13T14:41:31.611119+0000","last_fullsized":"2024-09-13T14:41:31.611119+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:40:09.359562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1042,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.577236+0000","last_change":"2024-09-13T14:19:25.873088+0000","last_active":"2024-09-13T14:41:29.577236+0000","last_peered":"2024-09-13T14:41:29.577236+0000","last_clean":"2024-09-13T14:41:29.577236+0000","last_became_active":"2024-09-13T14:19:25.872781+0000","last_became_peered":"2024-09-13T14:19:25.872781+0000","last_unstale":"2024-09-13T14:41:29.577236+0000","last_undegraded":"2024-09-13T14:41:29.577236+0000","last_fullsized":"2024-09-13T14:41:29.577236+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:05.607382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":604,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.577759+0000","last_change":"2024-09-13T14:41:28.594485+0000","last_active":"2024-09-13T14:41:29.577759+0000","last_peered":"2024-09-13T14:41:29.577759+0000","last_clean":"2024-09-13T14:41:29.577759+0000","last_became_active":"2024-09-13T14:41:28.594050+0000","last_became_peered":"2024-09-13T14:41:28.594050+0000","last_unstale":"2024-09-13T14:41:29.577759+0000","last_undegraded":"2024-09-13T14:41:29.577759+0000","last_fullsized":"2024-09-13T14:41:29.577759+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:03.838081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1050,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.584012+0000","last_change":"2024-09-13T14:28:35.603724+0000","last_active":"2024-09-13T14:41:29.584012+0000","last_peered":"2024-09-13T14:41:29.584012+0000","last_clean":"2024-09-13T14:41:29.584012+0000","last_became_active":"2024-09-13T14:28:35.603316+0000","last_became_peered":"2024-09-13T14:28:35.603316+0000","last_unstale":"2024-09-13T14:41:29.584012+0000","last_undegraded":"2024-09-13T14:41:29.584012+0000","last_fullsized":"2024-09-13T14:41:29.584012+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:47.668223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"1105'10278","reported_seq":10309,"reported_epoch":1105,"state":"active+clean","last_fresh":"2024-09-13T14:41:32.302045+0000","last_change":"2024-09-13T14:39:16.430434+0000","last_active":"2024-09-13T14:41:32.302045+0000","last_peered":"2024-09-13T14:41:32.302045+0000","last_clean":"2024-09-13T14:41:32.302045+0000","last_became_active":"2024-09-13T14:39:16.430177+0000","last_became_peered":"2024-09-13T14:39:16.430177+0000","last_unstale":"2024-09-13T14:41:32.302045+0000","last_undegraded":"2024-09-13T14:41:32.302045+0000","last_fullsized":"2024-09-13T14:41:32.302045+0000","mapping_epoch":1082,"log_start":"1088'3588","ondisk_log_start":"1088'3588","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6690,"log_dups_size":0,"ondisk_log_size":6690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:44.232119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425984000,"num_objects":6500,"num_object_clones":0,"num_object_copies":19500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10278,"num_write_kb":536896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":1060,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.577950+0000","last_change":"2024-09-13T14:28:35.608607+0000","last_active":"2024-09-13T14:41:29.577950+0000","last_peered":"2024-09-13T14:41:29.577950+0000","last_clean":"2024-09-13T14:41:29.577950+0000","last_became_active":"2024-09-13T14:28:35.608254+0000","last_became_peered":"2024-09-13T14:28:35.608254+0000","last_unstale":"2024-09-13T14:41:29.577950+0000","last_undegraded":"2024-09-13T14:41:29.577950+0000","last_fullsized":"2024-09-13T14:41:29.577950+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:24.019173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1030,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.583487+0000","last_change":"2024-09-13T14:19:26.887839+0000","last_active":"2024-09-13T14:41:29.583487+0000","last_peered":"2024-09-13T14:41:29.583487+0000","last_clean":"2024-09-13T14:41:29.583487+0000","last_became_active":"2024-09-13T14:19:26.887498+0000","last_became_peered":"2024-09-13T14:19:26.887498+0000","last_unstale":"2024-09-13T14:41:29.583487+0000","last_undegraded":"2024-09-13T14:41:29.583487+0000","last_fullsized":"2024-09-13T14:41:29.583487+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:24.475263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"1105'10260","reported_seq":10416,"reported_epoch":1105,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:41:33.652813+0000","last_change":"2024-09-13T14:41:30.181642+0000","last_active":"2024-09-13T14:41:33.652813+0000","last_peered":"2024-09-13T14:41:33.652813+0000","last_clean":"2024-09-13T14:41:27.568458+0000","last_became_active":"2024-09-13T14:41:29.966413+0000","last_became_peered":"2024-09-13T14:41:29.966413+0000","last_unstale":"2024-09-13T14:41:33.652813+0000","last_undegraded":"2024-09-13T14:41:29.616193+0000","last_fullsized":"2024-09-13T14:41:29.578900+0000","mapping_epoch":1103,"log_start":"1086'200","ondisk_log_start":"1086'200","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419299328,"num_objects":6398,"num_object_clones":0,"num_object_copies":19194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":141,"num_objects_misplaced":6398,"num_objects_unfound":0,"num_objects_dirty":6398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10260,"num_write_kb":533056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":3211264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":6398},{"shards":"3,7","objects":-141}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":1075,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.577898+0000","last_change":"2024-09-13T14:41:28.598356+0000","last_active":"2024-09-13T14:41:29.577898+0000","last_peered":"2024-09-13T14:41:29.577898+0000","last_clean":"2024-09-13T14:41:29.577898+0000","last_became_active":"2024-09-13T14:41:28.588185+0000","last_became_peered":"2024-09-13T14:41:28.588185+0000","last_unstale":"2024-09-13T14:41:29.577898+0000","last_undegraded":"2024-09-13T14:41:29.577898+0000","last_fullsized":"2024-09-13T14:41:29.577898+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:54.340118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1082,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.579623+0000","last_change":"2024-09-13T14:41:28.594910+0000","last_active":"2024-09-13T14:41:29.579623+0000","last_peered":"2024-09-13T14:41:29.579623+0000","last_clean":"2024-09-13T14:41:29.579623+0000","last_became_active":"2024-09-13T14:41:28.593806+0000","last_became_peered":"2024-09-13T14:41:28.593806+0000","last_unstale":"2024-09-13T14:41:29.579623+0000","last_undegraded":"2024-09-13T14:41:29.579623+0000","last_fullsized":"2024-09-13T14:41:29.579623+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:45:14.253295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"1105'10240","reported_seq":10270,"reported_epoch":1105,"state":"active+clean","last_fresh":"2024-09-13T14:41:32.294903+0000","last_change":"2024-09-13T14:39:16.428829+0000","last_active":"2024-09-13T14:41:32.294903+0000","last_peered":"2024-09-13T14:41:32.294903+0000","last_clean":"2024-09-13T14:41:32.294903+0000","last_became_active":"2024-09-13T14:39:16.428558+0000","last_became_peered":"2024-09-13T14:39:16.428558+0000","last_unstale":"2024-09-13T14:41:32.294903+0000","last_undegraded":"2024-09-13T14:41:32.294903+0000","last_fullsized":"2024-09-13T14:41:32.294903+0000","mapping_epoch":1082,"log_start":"1088'4341","ondisk_log_start":"1088'4341","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":5899,"log_dups_size":0,"ondisk_log_size":5899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:04:19.571418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434503680,"num_objects":6630,"num_object_clones":0,"num_object_copies":19890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10240,"num_write_kb":539840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":1043,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.577860+0000","last_change":"2024-09-13T14:19:25.871006+0000","last_active":"2024-09-13T14:41:29.577860+0000","last_peered":"2024-09-13T14:41:29.577860+0000","last_clean":"2024-09-13T14:41:29.577860+0000","last_became_active":"2024-09-13T14:19:25.870713+0000","last_became_peered":"2024-09-13T14:19:25.870713+0000","last_unstale":"2024-09-13T14:41:29.577860+0000","last_undegraded":"2024-09-13T14:41:29.577860+0000","last_fullsized":"2024-09-13T14:41:29.577860+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:34.372124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1051,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.584252+0000","last_change":"2024-09-13T14:41:28.586613+0000","last_active":"2024-09-13T14:41:29.584252+0000","last_peered":"2024-09-13T14:41:29.584252+0000","last_clean":"2024-09-13T14:41:29.584252+0000","last_became_active":"2024-09-13T14:41:28.586345+0000","last_became_peered":"2024-09-13T14:41:28.586345+0000","last_unstale":"2024-09-13T14:41:29.584252+0000","last_undegraded":"2024-09-13T14:41:29.584252+0000","last_fullsized":"2024-09-13T14:41:29.584252+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:37.756544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"1105'10153","reported_seq":10209,"reported_epoch":1105,"state":"active+clean","last_fresh":"2024-09-13T14:41:33.830841+0000","last_change":"2024-09-13T14:39:16.428268+0000","last_active":"2024-09-13T14:41:33.830841+0000","last_peered":"2024-09-13T14:41:33.830841+0000","last_clean":"2024-09-13T14:41:33.830841+0000","last_became_active":"2024-09-13T14:39:16.428016+0000","last_became_peered":"2024-09-13T14:39:16.428016+0000","last_unstale":"2024-09-13T14:41:33.830841+0000","last_undegraded":"2024-09-13T14:41:33.830841+0000","last_fullsized":"2024-09-13T14:41:33.830841+0000","mapping_epoch":1082,"log_start":"1088'3482","ondisk_log_start":"1088'3482","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6671,"log_dups_size":0,"ondisk_log_size":6671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:28:16.657831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417136640,"num_objects":6365,"num_object_clones":0,"num_object_copies":19095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10153,"num_write_kb":528576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1068,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.591770+0000","last_change":"2024-09-13T14:41:28.594298+0000","last_active":"2024-09-13T14:41:29.591770+0000","last_peered":"2024-09-13T14:41:29.591770+0000","last_clean":"2024-09-13T14:41:29.591770+0000","last_became_active":"2024-09-13T14:41:28.589751+0000","last_became_peered":"2024-09-13T14:41:28.589751+0000","last_unstale":"2024-09-13T14:41:29.591770+0000","last_undegraded":"2024-09-13T14:41:29.591770+0000","last_fullsized":"2024-09-13T14:41:29.591770+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:15:07.568716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1583,"reported_epoch":1105,"state":"active+clean","last_fresh":"2024-09-13T14:41:31.610394+0000","last_change":"2024-09-13T14:41:28.587900+0000","last_active":"2024-09-13T14:41:31.610394+0000","last_peered":"2024-09-13T14:41:31.610394+0000","last_clean":"2024-09-13T14:41:31.610394+0000","last_became_active":"2024-09-13T14:41:28.587563+0000","last_became_peered":"2024-09-13T14:41:28.587563+0000","last_unstale":"2024-09-13T14:41:31.610394+0000","last_undegraded":"2024-09-13T14:41:31.610394+0000","last_fullsized":"2024-09-13T14:41:31.610394+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:19:15.384955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"261'100","reported_seq":1205,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.577156+0000","last_change":"2024-09-13T14:41:28.268071+0000","last_active":"2024-09-13T14:41:29.577156+0000","last_peered":"2024-09-13T14:41:29.577156+0000","last_clean":"2024-09-13T14:41:29.577156+0000","last_became_active":"2024-09-13T14:41:20.510300+0000","last_became_peered":"2024-09-13T14:41:20.510300+0000","last_unstale":"2024-09-13T14:41:29.577156+0000","last_undegraded":"2024-09-13T14:41:29.577156+0000","last_fullsized":"2024-09-13T14:41:29.577156+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1100,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:28.268005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:28.268005+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:10.225185+0000","scrub_duration":0.0015413810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"1105'10058","reported_seq":10691,"reported_epoch":1105,"state":"active+remapped+backfilling","last_fresh":"2024-09-13T14:41:32.285771+0000","last_change":"2024-09-13T14:41:29.722228+0000","last_active":"2024-09-13T14:41:32.285771+0000","last_peered":"2024-09-13T14:41:32.285771+0000","last_clean":"2024-09-13T14:41:27.545589+0000","last_became_active":"2024-09-13T14:41:29.700570+0000","last_became_peered":"2024-09-13T14:41:29.700570+0000","last_unstale":"2024-09-13T14:41:32.285771+0000","last_undegraded":"2024-09-13T14:41:32.285771+0000","last_fullsized":"2024-09-13T14:41:32.285771+0000","mapping_epoch":1103,"log_start":"1088'3517","ondisk_log_start":"1088'3517","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6541,"log_dups_size":0,"ondisk_log_size":6541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:00:18.887770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":424542208,"num_objects":6478,"num_object_clones":0,"num_object_copies":19434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6187,"num_objects_unfound":0,"num_objects_dirty":6478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10058,"num_write_kb":529152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":295,"num_bytes_recovered":19333120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,1],"avail_no_missing":["4","1","2"],"object_location_counts":[{"shards":"1,2,4","objects":6478}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":119,"reported_epoch":1105,"state":"active+clean","last_fresh":"2024-09-13T14:41:31.610673+0000","last_change":"2024-09-13T14:41:28.600760+0000","last_active":"2024-09-13T14:41:31.610673+0000","last_peered":"2024-09-13T14:41:31.610673+0000","last_clean":"2024-09-13T14:41:31.610673+0000","last_became_active":"2024-09-13T14:41:28.586678+0000","last_became_peered":"2024-09-13T14:41:28.586678+0000","last_unstale":"2024-09-13T14:41:31.610673+0000","last_undegraded":"2024-09-13T14:41:31.610673+0000","last_fullsized":"2024-09-13T14:41:31.610673+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:44:53.564449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1065,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.579837+0000","last_change":"2024-09-13T14:41:28.598046+0000","last_active":"2024-09-13T14:41:29.579837+0000","last_peered":"2024-09-13T14:41:29.579837+0000","last_clean":"2024-09-13T14:41:29.579837+0000","last_became_active":"2024-09-13T14:41:28.597748+0000","last_became_peered":"2024-09-13T14:41:28.597748+0000","last_unstale":"2024-09-13T14:41:29.579837+0000","last_undegraded":"2024-09-13T14:41:29.579837+0000","last_fullsized":"2024-09-13T14:41:29.579837+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:52.920256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"261'100","reported_seq":20,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.583412+0000","last_change":"2024-09-13T14:41:20.562986+0000","last_active":"2024-09-13T14:41:29.583412+0000","last_peered":"2024-09-13T14:41:29.583412+0000","last_clean":"2024-09-13T14:41:29.583412+0000","last_became_active":"2024-09-13T14:41:20.529244+0000","last_became_peered":"2024-09-13T14:41:20.529244+0000","last_unstale":"2024-09-13T14:41:29.583412+0000","last_undegraded":"2024-09-13T14:41:29.583412+0000","last_fullsized":"2024-09-13T14:41:29.583412+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1100,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:15:42.069756+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":316358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"1104'9911","reported_seq":9967,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:30.688803+0000","last_change":"2024-09-13T14:39:16.433163+0000","last_active":"2024-09-13T14:41:30.688803+0000","last_peered":"2024-09-13T14:41:30.688803+0000","last_clean":"2024-09-13T14:41:30.688803+0000","last_became_active":"2024-09-13T14:39:16.432783+0000","last_became_peered":"2024-09-13T14:39:16.432783+0000","last_unstale":"2024-09-13T14:41:30.688803+0000","last_undegraded":"2024-09-13T14:41:30.688803+0000","last_fullsized":"2024-09-13T14:41:30.688803+0000","mapping_epoch":1082,"log_start":"1087'2554","ondisk_log_start":"1087'2554","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":7357,"log_dups_size":0,"ondisk_log_size":7357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:43:29.182915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423428096,"num_objects":6461,"num_object_clones":0,"num_object_copies":19383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9911,"num_write_kb":523904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1052,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.584405+0000","last_change":"2024-09-13T14:41:28.591027+0000","last_active":"2024-09-13T14:41:29.584405+0000","last_peered":"2024-09-13T14:41:29.584405+0000","last_clean":"2024-09-13T14:41:29.584405+0000","last_became_active":"2024-09-13T14:41:28.590742+0000","last_became_peered":"2024-09-13T14:41:28.590742+0000","last_unstale":"2024-09-13T14:41:29.584405+0000","last_undegraded":"2024-09-13T14:41:29.584405+0000","last_fullsized":"2024-09-13T14:41:29.584405+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:01.865182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1082,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.580170+0000","last_change":"2024-09-13T14:41:28.588833+0000","last_active":"2024-09-13T14:41:29.580170+0000","last_peered":"2024-09-13T14:41:29.580170+0000","last_clean":"2024-09-13T14:41:29.580170+0000","last_became_active":"2024-09-13T14:41:28.588490+0000","last_became_peered":"2024-09-13T14:41:28.588490+0000","last_unstale":"2024-09-13T14:41:29.580170+0000","last_undegraded":"2024-09-13T14:41:29.580170+0000","last_fullsized":"2024-09-13T14:41:29.580170+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:23.440459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"1105'10090","reported_seq":10121,"reported_epoch":1105,"state":"active+clean","last_fresh":"2024-09-13T14:41:32.304030+0000","last_change":"2024-09-13T14:39:16.428687+0000","last_active":"2024-09-13T14:41:32.304030+0000","last_peered":"2024-09-13T14:41:32.304030+0000","last_clean":"2024-09-13T14:41:32.304030+0000","last_became_active":"2024-09-13T14:39:16.428382+0000","last_became_peered":"2024-09-13T14:39:16.428382+0000","last_unstale":"2024-09-13T14:41:32.304030+0000","last_undegraded":"2024-09-13T14:41:32.304030+0000","last_fullsized":"2024-09-13T14:41:32.304030+0000","mapping_epoch":1082,"log_start":"1090'4638","ondisk_log_start":"1090'4638","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":5452,"log_dups_size":0,"ondisk_log_size":5452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:33:44.732564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425852928,"num_objects":6498,"num_object_clones":0,"num_object_copies":19494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10090,"num_write_kb":530816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1046,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.579999+0000","last_change":"2024-09-13T14:29:42.545815+0000","last_active":"2024-09-13T14:41:29.579999+0000","last_peered":"2024-09-13T14:41:29.579999+0000","last_clean":"2024-09-13T14:41:29.579999+0000","last_became_active":"2024-09-13T14:29:42.545508+0000","last_became_peered":"2024-09-13T14:29:42.545508+0000","last_unstale":"2024-09-13T14:41:29.579999+0000","last_undegraded":"2024-09-13T14:41:29.579999+0000","last_fullsized":"2024-09-13T14:41:29.579999+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:39.678381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1055,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.580355+0000","last_change":"2024-09-13T14:41:28.595541+0000","last_active":"2024-09-13T14:41:29.580355+0000","last_peered":"2024-09-13T14:41:29.580355+0000","last_clean":"2024-09-13T14:41:29.580355+0000","last_became_active":"2024-09-13T14:41:28.593863+0000","last_became_peered":"2024-09-13T14:41:28.593863+0000","last_unstale":"2024-09-13T14:41:29.580355+0000","last_undegraded":"2024-09-13T14:41:29.580355+0000","last_fullsized":"2024-09-13T14:41:29.580355+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:06.730588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"261'100","reported_seq":25,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.624312+0000","last_change":"2024-09-13T14:41:29.624236+0000","last_active":"2024-09-13T14:41:29.624312+0000","last_peered":"2024-09-13T14:41:29.624312+0000","last_clean":"2024-09-13T14:41:29.624312+0000","last_became_active":"2024-09-13T14:41:20.530755+0000","last_became_peered":"2024-09-13T14:41:20.530755+0000","last_unstale":"2024-09-13T14:41:29.624312+0000","last_undegraded":"2024-09-13T14:41:29.624312+0000","last_fullsized":"2024-09-13T14:41:29.624312+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1100,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:29.624129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:29.624129+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:16.408396+0000","scrub_duration":0.0076481020000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"1105'10095","reported_seq":10138,"reported_epoch":1105,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:41:32.363100+0000","last_change":"2024-09-13T14:41:29.774197+0000","last_active":"2024-09-13T14:41:32.363100+0000","last_peered":"2024-09-13T14:41:32.363100+0000","last_clean":"2024-09-13T14:41:27.579452+0000","last_became_active":"2024-09-13T14:41:29.706889+0000","last_became_peered":"2024-09-13T14:41:29.706889+0000","last_unstale":"2024-09-13T14:41:32.363100+0000","last_undegraded":"2024-09-13T14:41:32.363100+0000","last_fullsized":"2024-09-13T14:41:32.363100+0000","mapping_epoch":1103,"log_start":"1088'3406","ondisk_log_start":"1088'3406","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6689,"log_dups_size":0,"ondisk_log_size":6689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:38:40.997498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420347928,"num_objects":6415,"num_object_clones":0,"num_object_copies":19245,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6415,"num_objects_unfound":0,"num_objects_dirty":6415,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10095,"num_write_kb":528257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,1],"avail_no_missing":["5","0","1"],"object_location_counts":[{"shards":"0,1,5","objects":6415}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":574,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.577124+0000","last_change":"2024-09-13T14:41:28.587940+0000","last_active":"2024-09-13T14:41:29.577124+0000","last_peered":"2024-09-13T14:41:29.577124+0000","last_clean":"2024-09-13T14:41:29.577124+0000","last_became_active":"2024-09-13T14:41:28.587643+0000","last_became_peered":"2024-09-13T14:41:28.587643+0000","last_unstale":"2024-09-13T14:41:29.577124+0000","last_undegraded":"2024-09-13T14:41:29.577124+0000","last_fullsized":"2024-09-13T14:41:29.577124+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:09:23.651717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":613,"reported_epoch":1105,"state":"active+clean","last_fresh":"2024-09-13T14:41:31.605813+0000","last_change":"2024-09-13T14:41:28.592799+0000","last_active":"2024-09-13T14:41:31.605813+0000","last_peered":"2024-09-13T14:41:31.605813+0000","last_clean":"2024-09-13T14:41:31.605813+0000","last_became_active":"2024-09-13T14:41:28.592345+0000","last_became_peered":"2024-09-13T14:41:28.592345+0000","last_unstale":"2024-09-13T14:41:31.605813+0000","last_undegraded":"2024-09-13T14:41:31.605813+0000","last_fullsized":"2024-09-13T14:41:31.605813+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:17.651032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"1105'10220","reported_seq":10251,"reported_epoch":1105,"state":"active+clean","last_fresh":"2024-09-13T14:41:32.285618+0000","last_change":"2024-09-13T14:39:16.431907+0000","last_active":"2024-09-13T14:41:32.285618+0000","last_peered":"2024-09-13T14:41:32.285618+0000","last_clean":"2024-09-13T14:41:32.285618+0000","last_became_active":"2024-09-13T14:39:16.431660+0000","last_became_peered":"2024-09-13T14:39:16.431660+0000","last_unstale":"2024-09-13T14:41:32.285618+0000","last_undegraded":"2024-09-13T14:41:32.285618+0000","last_fullsized":"2024-09-13T14:41:32.285618+0000","mapping_epoch":1082,"log_start":"1088'3515","ondisk_log_start":"1088'3515","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6705,"log_dups_size":0,"ondisk_log_size":6705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:18.941482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":424542208,"num_objects":6478,"num_object_clones":0,"num_object_copies":19434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10220,"num_write_kb":534336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"261'100","reported_seq":1196,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.577354+0000","last_change":"2024-09-13T14:41:23.257356+0000","last_active":"2024-09-13T14:41:29.577354+0000","last_peered":"2024-09-13T14:41:29.577354+0000","last_clean":"2024-09-13T14:41:29.577354+0000","last_became_active":"2024-09-13T14:41:16.493791+0000","last_became_peered":"2024-09-13T14:41:16.493791+0000","last_unstale":"2024-09-13T14:41:29.577354+0000","last_undegraded":"2024-09-13T14:41:29.577354+0000","last_fullsized":"2024-09-13T14:41:29.577354+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1098,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:23.257279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:23.257279+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:04:29.628179+0000","scrub_duration":0.010350875000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":1061,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.583741+0000","last_change":"2024-09-13T14:29:42.539455+0000","last_active":"2024-09-13T14:41:29.583741+0000","last_peered":"2024-09-13T14:41:29.583741+0000","last_clean":"2024-09-13T14:41:29.583741+0000","last_became_active":"2024-09-13T14:29:42.539068+0000","last_became_peered":"2024-09-13T14:29:42.539068+0000","last_unstale":"2024-09-13T14:41:29.583741+0000","last_undegraded":"2024-09-13T14:41:29.583741+0000","last_fullsized":"2024-09-13T14:41:29.583741+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:24:06.048855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034405999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":572,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.583861+0000","last_change":"2024-09-13T14:39:11.382555+0000","last_active":"2024-09-13T14:41:29.583861+0000","last_peered":"2024-09-13T14:41:29.583861+0000","last_clean":"2024-09-13T14:41:29.583861+0000","last_became_active":"2024-09-13T14:39:11.382247+0000","last_became_peered":"2024-09-13T14:39:11.382247+0000","last_unstale":"2024-09-13T14:41:29.583861+0000","last_undegraded":"2024-09-13T14:41:29.583861+0000","last_fullsized":"2024-09-13T14:41:29.583861+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:09:48.016987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1024,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.587076+0000","last_change":"2024-09-13T14:19:27.379312+0000","last_active":"2024-09-13T14:41:29.587076+0000","last_peered":"2024-09-13T14:41:29.587076+0000","last_clean":"2024-09-13T14:41:29.587076+0000","last_became_active":"2024-09-13T14:19:27.378947+0000","last_became_peered":"2024-09-13T14:19:27.378947+0000","last_unstale":"2024-09-13T14:41:29.587076+0000","last_undegraded":"2024-09-13T14:41:29.587076+0000","last_fullsized":"2024-09-13T14:41:29.587076+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:54:20.706578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"261'100","reported_seq":334,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.584220+0000","last_change":"2024-09-13T14:41:28.592476+0000","last_active":"2024-09-13T14:41:29.584220+0000","last_peered":"2024-09-13T14:41:29.584220+0000","last_clean":"2024-09-13T14:41:29.584220+0000","last_became_active":"2024-09-13T14:41:28.592111+0000","last_became_peered":"2024-09-13T14:41:28.592111+0000","last_unstale":"2024-09-13T14:41:29.584220+0000","last_undegraded":"2024-09-13T14:41:29.584220+0000","last_fullsized":"2024-09-13T14:41:29.584220+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:26.747497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:26.747497+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":68,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"1105'10356","reported_seq":11057,"reported_epoch":1105,"state":"active+remapped+backfilling","last_fresh":"2024-09-13T14:41:32.308448+0000","last_change":"2024-09-13T14:41:29.733124+0000","last_active":"2024-09-13T14:41:32.308448+0000","last_peered":"2024-09-13T14:41:32.308448+0000","last_clean":"2024-09-13T14:41:27.548643+0000","last_became_active":"2024-09-13T14:41:29.686059+0000","last_became_peered":"2024-09-13T14:41:29.686059+0000","last_unstale":"2024-09-13T14:41:32.308448+0000","last_undegraded":"2024-09-13T14:41:32.308448+0000","last_fullsized":"2024-09-13T14:41:32.308448+0000","mapping_epoch":1103,"log_start":"1088'3655","ondisk_log_start":"1088'3655","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6701,"log_dups_size":0,"ondisk_log_size":6701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:21:24.399451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426115072,"num_objects":6502,"num_object_clones":0,"num_object_copies":19506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6184,"num_objects_unfound":0,"num_objects_dirty":6502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10356,"num_write_kb":539456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":319,"num_bytes_recovered":20905984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,1],"avail_no_missing":["5","1","6"],"object_location_counts":[{"shards":"1,5,6","objects":6502}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1098,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.577100+0000","last_change":"2024-09-13T14:41:28.587183+0000","last_active":"2024-09-13T14:41:29.577100+0000","last_peered":"2024-09-13T14:41:29.577100+0000","last_clean":"2024-09-13T14:41:29.577100+0000","last_became_active":"2024-09-13T14:41:28.586268+0000","last_became_peered":"2024-09-13T14:41:28.586268+0000","last_unstale":"2024-09-13T14:41:29.577100+0000","last_undegraded":"2024-09-13T14:41:29.577100+0000","last_fullsized":"2024-09-13T14:41:29.577100+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:34.203731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1617,"reported_epoch":1105,"state":"active+clean","last_fresh":"2024-09-13T14:41:31.610490+0000","last_change":"2024-09-13T14:39:11.390055+0000","last_active":"2024-09-13T14:41:31.610490+0000","last_peered":"2024-09-13T14:41:31.610490+0000","last_clean":"2024-09-13T14:41:31.610490+0000","last_became_active":"2024-09-13T14:39:11.389697+0000","last_became_peered":"2024-09-13T14:39:11.389697+0000","last_unstale":"2024-09-13T14:41:31.610490+0000","last_undegraded":"2024-09-13T14:41:31.610490+0000","last_fullsized":"2024-09-13T14:41:31.610490+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:50.975739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1595,"reported_epoch":1105,"state":"active+clean","last_fresh":"2024-09-13T14:41:31.610902+0000","last_change":"2024-09-13T14:41:28.584816+0000","last_active":"2024-09-13T14:41:31.610902+0000","last_peered":"2024-09-13T14:41:31.610902+0000","last_clean":"2024-09-13T14:41:31.610902+0000","last_became_active":"2024-09-13T14:41:28.583332+0000","last_became_peered":"2024-09-13T14:41:28.583332+0000","last_unstale":"2024-09-13T14:41:31.610902+0000","last_undegraded":"2024-09-13T14:41:31.610902+0000","last_fullsized":"2024-09-13T14:41:31.610902+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:20.608574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"261'100","reported_seq":1205,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.577187+0000","last_change":"2024-09-13T14:41:24.245745+0000","last_active":"2024-09-13T14:41:29.577187+0000","last_peered":"2024-09-13T14:41:29.577187+0000","last_clean":"2024-09-13T14:41:29.577187+0000","last_became_active":"2024-09-13T14:41:20.515333+0000","last_became_peered":"2024-09-13T14:41:20.515333+0000","last_unstale":"2024-09-13T14:41:29.577187+0000","last_undegraded":"2024-09-13T14:41:29.577187+0000","last_fullsized":"2024-09-13T14:41:29.577187+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1100,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:24.245668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:24.245668+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:57:50.398918+0000","scrub_duration":0.0034439399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"1105'9926","reported_seq":9957,"reported_epoch":1105,"state":"active+clean","last_fresh":"2024-09-13T14:41:32.314890+0000","last_change":"2024-09-13T14:39:16.428528+0000","last_active":"2024-09-13T14:41:32.314890+0000","last_peered":"2024-09-13T14:41:32.314890+0000","last_clean":"2024-09-13T14:41:32.314890+0000","last_became_active":"2024-09-13T14:39:16.428195+0000","last_became_peered":"2024-09-13T14:39:16.428195+0000","last_unstale":"2024-09-13T14:41:32.314890+0000","last_undegraded":"2024-09-13T14:41:32.314890+0000","last_fullsized":"2024-09-13T14:41:32.314890+0000","mapping_epoch":1082,"log_start":"1088'4023","ondisk_log_start":"1088'4023","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":5903,"log_dups_size":0,"ondisk_log_size":5903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:54:28.451285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414449664,"num_objects":6324,"num_object_clones":0,"num_object_copies":18972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9926,"num_write_kb":520000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1591,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.598921+0000","last_change":"2024-09-13T14:41:28.585065+0000","last_active":"2024-09-13T14:41:29.598921+0000","last_peered":"2024-09-13T14:41:29.598921+0000","last_clean":"2024-09-13T14:41:29.598921+0000","last_became_active":"2024-09-13T14:41:28.584754+0000","last_became_peered":"2024-09-13T14:41:28.584754+0000","last_unstale":"2024-09-13T14:41:29.598921+0000","last_undegraded":"2024-09-13T14:41:29.598921+0000","last_fullsized":"2024-09-13T14:41:29.598921+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:20:35.484829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":545,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.598450+0000","last_change":"2024-09-13T14:41:28.590194+0000","last_active":"2024-09-13T14:41:29.598450+0000","last_peered":"2024-09-13T14:41:29.598450+0000","last_clean":"2024-09-13T14:41:29.598450+0000","last_became_active":"2024-09-13T14:41:28.589774+0000","last_became_peered":"2024-09-13T14:41:28.589774+0000","last_unstale":"2024-09-13T14:41:29.598450+0000","last_undegraded":"2024-09-13T14:41:29.598450+0000","last_fullsized":"2024-09-13T14:41:29.598450+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:36.382030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":502,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.585152+0000","last_change":"2024-09-13T14:41:28.597828+0000","last_active":"2024-09-13T14:41:29.585152+0000","last_peered":"2024-09-13T14:41:29.585152+0000","last_clean":"2024-09-13T14:41:29.585152+0000","last_became_active":"2024-09-13T14:41:28.597540+0000","last_became_peered":"2024-09-13T14:41:28.597540+0000","last_unstale":"2024-09-13T14:41:29.585152+0000","last_undegraded":"2024-09-13T14:41:29.585152+0000","last_fullsized":"2024-09-13T14:41:29.585152+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:49:04.992589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"261'100","reported_seq":25,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.583372+0000","last_change":"2024-09-13T14:41:23.561816+0000","last_active":"2024-09-13T14:41:29.583372+0000","last_peered":"2024-09-13T14:41:29.583372+0000","last_clean":"2024-09-13T14:41:29.583372+0000","last_became_active":"2024-09-13T14:41:20.509730+0000","last_became_peered":"2024-09-13T14:41:20.509730+0000","last_unstale":"2024-09-13T14:41:29.583372+0000","last_undegraded":"2024-09-13T14:41:29.583372+0000","last_fullsized":"2024-09-13T14:41:29.583372+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1100,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:23.561736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:23.561736+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:00.213136+0000","scrub_duration":0.0091806160000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"1105'10082","reported_seq":10112,"reported_epoch":1105,"state":"active+clean","last_fresh":"2024-09-13T14:41:32.271935+0000","last_change":"2024-09-13T14:39:16.427959+0000","last_active":"2024-09-13T14:41:32.271935+0000","last_peered":"2024-09-13T14:41:32.271935+0000","last_clean":"2024-09-13T14:41:32.271935+0000","last_became_active":"2024-09-13T14:39:16.427270+0000","last_became_peered":"2024-09-13T14:39:16.427270+0000","last_unstale":"2024-09-13T14:41:32.271935+0000","last_undegraded":"2024-09-13T14:41:32.271935+0000","last_fullsized":"2024-09-13T14:41:32.271935+0000","mapping_epoch":1082,"log_start":"1088'3534","ondisk_log_start":"1088'3534","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6548,"log_dups_size":0,"ondisk_log_size":6548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:51.791556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419430400,"num_objects":6400,"num_object_clones":0,"num_object_copies":19200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10082,"num_write_kb":527424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":1079,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.583984+0000","last_change":"2024-09-13T14:41:28.604178+0000","last_active":"2024-09-13T14:41:29.583984+0000","last_peered":"2024-09-13T14:41:29.583984+0000","last_clean":"2024-09-13T14:41:29.583984+0000","last_became_active":"2024-09-13T14:41:28.587558+0000","last_became_peered":"2024-09-13T14:41:28.587558+0000","last_unstale":"2024-09-13T14:41:29.583984+0000","last_undegraded":"2024-09-13T14:41:29.583984+0000","last_fullsized":"2024-09-13T14:41:29.583984+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:04:42.789023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1046,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.579996+0000","last_change":"2024-09-13T14:28:35.604404+0000","last_active":"2024-09-13T14:41:29.579996+0000","last_peered":"2024-09-13T14:41:29.579996+0000","last_clean":"2024-09-13T14:41:29.579996+0000","last_became_active":"2024-09-13T14:28:35.604102+0000","last_became_peered":"2024-09-13T14:28:35.604102+0000","last_unstale":"2024-09-13T14:41:29.579996+0000","last_undegraded":"2024-09-13T14:41:29.579996+0000","last_fullsized":"2024-09-13T14:41:29.579996+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:25.546010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1526,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.598552+0000","last_change":"2024-09-13T14:41:28.597520+0000","last_active":"2024-09-13T14:41:29.598552+0000","last_peered":"2024-09-13T14:41:29.598552+0000","last_clean":"2024-09-13T14:41:29.598552+0000","last_became_active":"2024-09-13T14:41:28.597166+0000","last_became_peered":"2024-09-13T14:41:28.597166+0000","last_unstale":"2024-09-13T14:41:29.598552+0000","last_undegraded":"2024-09-13T14:41:29.598552+0000","last_fullsized":"2024-09-13T14:41:29.598552+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:18:57.954799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"261'100","reported_seq":337,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.587236+0000","last_change":"2024-09-13T14:41:20.544283+0000","last_active":"2024-09-13T14:41:29.587236+0000","last_peered":"2024-09-13T14:41:29.587236+0000","last_clean":"2024-09-13T14:41:29.587236+0000","last_became_active":"2024-09-13T14:41:20.543630+0000","last_became_peered":"2024-09-13T14:41:20.543630+0000","last_unstale":"2024-09-13T14:41:29.587236+0000","last_undegraded":"2024-09-13T14:41:29.587236+0000","last_fullsized":"2024-09-13T14:41:29.587236+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1100,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:15:42.069756+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"1105'10253","reported_seq":10311,"reported_epoch":1105,"state":"active+clean","last_fresh":"2024-09-13T14:41:33.814576+0000","last_change":"2024-09-13T14:39:16.429234+0000","last_active":"2024-09-13T14:41:33.814576+0000","last_peered":"2024-09-13T14:41:33.814576+0000","last_clean":"2024-09-13T14:41:33.814576+0000","last_became_active":"2024-09-13T14:39:16.428954+0000","last_became_peered":"2024-09-13T14:39:16.428954+0000","last_unstale":"2024-09-13T14:41:33.814576+0000","last_undegraded":"2024-09-13T14:41:33.814576+0000","last_fullsized":"2024-09-13T14:41:33.814576+0000","mapping_epoch":1082,"log_start":"1088'3516","ondisk_log_start":"1088'3516","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6737,"log_dups_size":0,"ondisk_log_size":6737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:48.562878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419364864,"num_objects":6399,"num_object_clones":0,"num_object_copies":19197,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10253,"num_write_kb":532864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1075,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.591668+0000","last_change":"2024-09-13T14:39:11.383045+0000","last_active":"2024-09-13T14:41:29.591668+0000","last_peered":"2024-09-13T14:41:29.591668+0000","last_clean":"2024-09-13T14:41:29.591668+0000","last_became_active":"2024-09-13T14:39:11.382718+0000","last_became_peered":"2024-09-13T14:39:11.382718+0000","last_unstale":"2024-09-13T14:41:29.591668+0000","last_undegraded":"2024-09-13T14:41:29.591668+0000","last_fullsized":"2024-09-13T14:41:29.591668+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:21.812296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":528,"reported_epoch":1105,"state":"active+clean","last_fresh":"2024-09-13T14:41:31.610603+0000","last_change":"2024-09-13T14:41:28.585118+0000","last_active":"2024-09-13T14:41:31.610603+0000","last_peered":"2024-09-13T14:41:31.610603+0000","last_clean":"2024-09-13T14:41:31.610603+0000","last_became_active":"2024-09-13T14:41:28.584804+0000","last_became_peered":"2024-09-13T14:41:28.584804+0000","last_unstale":"2024-09-13T14:41:31.610603+0000","last_undegraded":"2024-09-13T14:41:31.610603+0000","last_fullsized":"2024-09-13T14:41:31.610603+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:10:27.164531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":468,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.583905+0000","last_change":"2024-09-13T14:41:28.588613+0000","last_active":"2024-09-13T14:41:29.583905+0000","last_peered":"2024-09-13T14:41:29.583905+0000","last_clean":"2024-09-13T14:41:29.583905+0000","last_became_active":"2024-09-13T14:41:28.587982+0000","last_became_peered":"2024-09-13T14:41:28.587982+0000","last_unstale":"2024-09-13T14:41:29.583905+0000","last_undegraded":"2024-09-13T14:41:29.583905+0000","last_fullsized":"2024-09-13T14:41:29.583905+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:51:16.814445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"261'100","reported_seq":347,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.587230+0000","last_change":"2024-09-13T14:41:25.289063+0000","last_active":"2024-09-13T14:41:29.587230+0000","last_peered":"2024-09-13T14:41:29.587230+0000","last_clean":"2024-09-13T14:41:29.587230+0000","last_became_active":"2024-09-13T14:41:20.526418+0000","last_became_peered":"2024-09-13T14:41:20.526418+0000","last_unstale":"2024-09-13T14:41:29.587230+0000","last_undegraded":"2024-09-13T14:41:29.587230+0000","last_fullsized":"2024-09-13T14:41:29.587230+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1100,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:25.288979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:25.288979+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:55:24.275123+0000","scrub_duration":0.0040379329999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"1105'9977","reported_seq":10193,"reported_epoch":1105,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:41:32.316266+0000","last_change":"2024-09-13T14:41:30.169424+0000","last_active":"2024-09-13T14:41:32.316266+0000","last_peered":"2024-09-13T14:41:32.316266+0000","last_clean":"2024-09-13T14:41:27.528697+0000","last_became_active":"2024-09-13T14:41:29.701483+0000","last_became_peered":"2024-09-13T14:41:29.701483+0000","last_unstale":"2024-09-13T14:41:32.316266+0000","last_undegraded":"2024-09-13T14:41:32.316266+0000","last_fullsized":"2024-09-13T14:41:32.316266+0000","mapping_epoch":1103,"log_start":"1089'4598","ondisk_log_start":"1089'4598","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":5379,"log_dups_size":0,"ondisk_log_size":5379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:11:34.235418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415301632,"num_objects":6337,"num_object_clones":0,"num_object_copies":19011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6265,"num_objects_unfound":0,"num_objects_dirty":6337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9977,"num_write_kb":522048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":4849664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,1],"avail_no_missing":["6","1","2"],"object_location_counts":[{"shards":"1,2,6","objects":6337}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":545,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.598395+0000","last_change":"2024-09-13T14:41:28.593821+0000","last_active":"2024-09-13T14:41:29.598395+0000","last_peered":"2024-09-13T14:41:29.598395+0000","last_clean":"2024-09-13T14:41:29.598395+0000","last_became_active":"2024-09-13T14:41:28.593183+0000","last_became_peered":"2024-09-13T14:41:28.593183+0000","last_unstale":"2024-09-13T14:41:29.598395+0000","last_undegraded":"2024-09-13T14:41:29.598395+0000","last_fullsized":"2024-09-13T14:41:29.598395+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:54.526707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1071,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.577853+0000","last_change":"2024-09-13T14:28:35.603663+0000","last_active":"2024-09-13T14:41:29.577853+0000","last_peered":"2024-09-13T14:41:29.577853+0000","last_clean":"2024-09-13T14:41:29.577853+0000","last_became_active":"2024-09-13T14:28:35.603358+0000","last_became_peered":"2024-09-13T14:28:35.603358+0000","last_unstale":"2024-09-13T14:41:29.577853+0000","last_undegraded":"2024-09-13T14:41:29.577853+0000","last_fullsized":"2024-09-13T14:41:29.577853+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:24:05.842796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1566,"reported_epoch":1105,"state":"active+clean","last_fresh":"2024-09-13T14:41:31.610483+0000","last_change":"2024-09-13T14:28:35.611508+0000","last_active":"2024-09-13T14:41:31.610483+0000","last_peered":"2024-09-13T14:41:31.610483+0000","last_clean":"2024-09-13T14:41:31.610483+0000","last_became_active":"2024-09-13T14:28:35.610106+0000","last_became_peered":"2024-09-13T14:28:35.610106+0000","last_unstale":"2024-09-13T14:41:31.610483+0000","last_undegraded":"2024-09-13T14:41:31.610483+0000","last_fullsized":"2024-09-13T14:41:31.610483+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:11.966958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"261'100","reported_seq":25,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.580067+0000","last_change":"2024-09-13T14:41:22.893041+0000","last_active":"2024-09-13T14:41:29.580067+0000","last_peered":"2024-09-13T14:41:29.580067+0000","last_clean":"2024-09-13T14:41:29.580067+0000","last_became_active":"2024-09-13T14:41:20.530028+0000","last_became_peered":"2024-09-13T14:41:20.530028+0000","last_unstale":"2024-09-13T14:41:29.580067+0000","last_undegraded":"2024-09-13T14:41:29.580067+0000","last_fullsized":"2024-09-13T14:41:29.580067+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1100,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:22.892963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:22.892963+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:32:52.347814+0000","scrub_duration":0.017451785000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"1105'10151","reported_seq":10182,"reported_epoch":1105,"state":"active+clean","last_fresh":"2024-09-13T14:41:32.221108+0000","last_change":"2024-09-13T14:39:16.429539+0000","last_active":"2024-09-13T14:41:32.221108+0000","last_peered":"2024-09-13T14:41:32.221108+0000","last_clean":"2024-09-13T14:41:32.221108+0000","last_became_active":"2024-09-13T14:39:16.426462+0000","last_became_peered":"2024-09-13T14:39:16.426462+0000","last_unstale":"2024-09-13T14:41:32.221108+0000","last_undegraded":"2024-09-13T14:41:32.221108+0000","last_fullsized":"2024-09-13T14:41:32.221108+0000","mapping_epoch":1082,"log_start":"1088'3600","ondisk_log_start":"1088'3600","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6551,"log_dups_size":0,"ondisk_log_size":6551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:58:57.425171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":424083456,"num_objects":6471,"num_object_clones":0,"num_object_copies":19413,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10151,"num_write_kb":531904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":1526,"reported_epoch":1105,"state":"active+clean","last_fresh":"2024-09-13T14:41:31.605916+0000","last_change":"2024-09-13T14:30:15.581728+0000","last_active":"2024-09-13T14:41:31.605916+0000","last_peered":"2024-09-13T14:41:31.605916+0000","last_clean":"2024-09-13T14:41:31.605916+0000","last_became_active":"2024-09-13T14:30:15.581303+0000","last_became_peered":"2024-09-13T14:30:15.581303+0000","last_unstale":"2024-09-13T14:41:31.605916+0000","last_undegraded":"2024-09-13T14:41:31.605916+0000","last_fullsized":"2024-09-13T14:41:31.605916+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:10.027991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":478,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.587157+0000","last_change":"2024-09-13T14:41:28.592508+0000","last_active":"2024-09-13T14:41:29.587157+0000","last_peered":"2024-09-13T14:41:29.587157+0000","last_clean":"2024-09-13T14:41:29.587157+0000","last_became_active":"2024-09-13T14:41:28.590027+0000","last_became_peered":"2024-09-13T14:41:28.590027+0000","last_unstale":"2024-09-13T14:41:29.587157+0000","last_undegraded":"2024-09-13T14:41:29.587157+0000","last_fullsized":"2024-09-13T14:41:29.587157+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:30:39.262649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":612,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.577003+0000","last_change":"2024-09-13T14:41:28.587360+0000","last_active":"2024-09-13T14:41:29.577003+0000","last_peered":"2024-09-13T14:41:29.577003+0000","last_clean":"2024-09-13T14:41:29.577003+0000","last_became_active":"2024-09-13T14:41:28.586655+0000","last_became_peered":"2024-09-13T14:41:28.586655+0000","last_unstale":"2024-09-13T14:41:29.577003+0000","last_undegraded":"2024-09-13T14:41:29.577003+0000","last_fullsized":"2024-09-13T14:41:29.577003+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:21:51.249946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"261'100","reported_seq":29,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.591177+0000","last_change":"2024-09-13T14:41:22.580296+0000","last_active":"2024-09-13T14:41:29.591177+0000","last_peered":"2024-09-13T14:41:29.591177+0000","last_clean":"2024-09-13T14:41:29.591177+0000","last_became_active":"2024-09-13T14:41:20.530363+0000","last_became_peered":"2024-09-13T14:41:20.530363+0000","last_unstale":"2024-09-13T14:41:29.591177+0000","last_undegraded":"2024-09-13T14:41:29.591177+0000","last_fullsized":"2024-09-13T14:41:29.591177+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1100,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:22.580245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:22.580245+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:03.428657+0000","scrub_duration":0.001821837,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"1105'9843","reported_seq":9874,"reported_epoch":1105,"state":"active+clean","last_fresh":"2024-09-13T14:41:32.358713+0000","last_change":"2024-09-13T14:39:16.427785+0000","last_active":"2024-09-13T14:41:32.358713+0000","last_peered":"2024-09-13T14:41:32.358713+0000","last_clean":"2024-09-13T14:41:32.358713+0000","last_became_active":"2024-09-13T14:39:16.427520+0000","last_became_peered":"2024-09-13T14:39:16.427520+0000","last_unstale":"2024-09-13T14:41:32.358713+0000","last_undegraded":"2024-09-13T14:41:32.358713+0000","last_fullsized":"2024-09-13T14:41:32.358713+0000","mapping_epoch":1082,"log_start":"1088'3931","ondisk_log_start":"1088'3931","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":5912,"log_dups_size":0,"ondisk_log_size":5912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:27.697724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417005568,"num_objects":6363,"num_object_clones":0,"num_object_copies":19089,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9843,"num_write_kb":518592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":1602,"reported_epoch":1105,"state":"active+clean","last_fresh":"2024-09-13T14:41:31.611179+0000","last_change":"2024-09-13T14:41:28.598014+0000","last_active":"2024-09-13T14:41:31.611179+0000","last_peered":"2024-09-13T14:41:31.611179+0000","last_clean":"2024-09-13T14:41:31.611179+0000","last_became_active":"2024-09-13T14:41:28.585268+0000","last_became_peered":"2024-09-13T14:41:28.585268+0000","last_unstale":"2024-09-13T14:41:31.611179+0000","last_undegraded":"2024-09-13T14:41:31.611179+0000","last_fullsized":"2024-09-13T14:41:31.611179+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:40.047685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1055,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.587510+0000","last_change":"2024-09-13T14:29:42.538860+0000","last_active":"2024-09-13T14:41:29.587510+0000","last_peered":"2024-09-13T14:41:29.587510+0000","last_clean":"2024-09-13T14:41:29.587510+0000","last_became_active":"2024-09-13T14:29:42.538608+0000","last_became_peered":"2024-09-13T14:29:42.538608+0000","last_unstale":"2024-09-13T14:41:29.587510+0000","last_undegraded":"2024-09-13T14:41:29.587510+0000","last_fullsized":"2024-09-13T14:41:29.587510+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:20.070703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1028,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.579548+0000","last_change":"2024-09-13T14:19:26.131609+0000","last_active":"2024-09-13T14:41:29.579548+0000","last_peered":"2024-09-13T14:41:29.579548+0000","last_clean":"2024-09-13T14:41:29.579548+0000","last_became_active":"2024-09-13T14:19:26.131029+0000","last_became_peered":"2024-09-13T14:19:26.131029+0000","last_unstale":"2024-09-13T14:41:29.579548+0000","last_undegraded":"2024-09-13T14:41:29.579548+0000","last_fullsized":"2024-09-13T14:41:29.579548+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:57.933696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1055,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.579933+0000","last_change":"2024-09-13T14:41:28.593455+0000","last_active":"2024-09-13T14:41:29.579933+0000","last_peered":"2024-09-13T14:41:29.579933+0000","last_clean":"2024-09-13T14:41:29.579933+0000","last_became_active":"2024-09-13T14:41:28.593139+0000","last_became_peered":"2024-09-13T14:41:28.593139+0000","last_unstale":"2024-09-13T14:41:29.579933+0000","last_undegraded":"2024-09-13T14:41:29.579933+0000","last_fullsized":"2024-09-13T14:41:29.579933+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:37.427352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":1051,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.579920+0000","last_change":"2024-09-13T14:19:25.870605+0000","last_active":"2024-09-13T14:41:29.579920+0000","last_peered":"2024-09-13T14:41:29.579920+0000","last_clean":"2024-09-13T14:41:29.579920+0000","last_became_active":"2024-09-13T14:19:25.870282+0000","last_became_peered":"2024-09-13T14:19:25.870282+0000","last_unstale":"2024-09-13T14:41:29.579920+0000","last_undegraded":"2024-09-13T14:41:29.579920+0000","last_fullsized":"2024-09-13T14:41:29.579920+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:52.514915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":546,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.584329+0000","last_change":"2024-09-13T14:41:28.593526+0000","last_active":"2024-09-13T14:41:29.584329+0000","last_peered":"2024-09-13T14:41:29.584329+0000","last_clean":"2024-09-13T14:41:29.584329+0000","last_became_active":"2024-09-13T14:41:28.593217+0000","last_became_peered":"2024-09-13T14:41:28.593217+0000","last_unstale":"2024-09-13T14:41:29.584329+0000","last_undegraded":"2024-09-13T14:41:29.584329+0000","last_fullsized":"2024-09-13T14:41:29.584329+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:57:01.502925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":1062,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.577640+0000","last_change":"2024-09-13T14:30:15.579260+0000","last_active":"2024-09-13T14:41:29.577640+0000","last_peered":"2024-09-13T14:41:29.577640+0000","last_clean":"2024-09-13T14:41:29.577640+0000","last_became_active":"2024-09-13T14:30:15.578897+0000","last_became_peered":"2024-09-13T14:30:15.578897+0000","last_unstale":"2024-09-13T14:41:29.577640+0000","last_undegraded":"2024-09-13T14:41:29.577640+0000","last_fullsized":"2024-09-13T14:41:29.577640+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:56.737849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1058,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.584667+0000","last_change":"2024-09-13T14:41:28.591640+0000","last_active":"2024-09-13T14:41:29.584667+0000","last_peered":"2024-09-13T14:41:29.584667+0000","last_clean":"2024-09-13T14:41:29.584667+0000","last_became_active":"2024-09-13T14:41:28.591354+0000","last_became_peered":"2024-09-13T14:41:28.591354+0000","last_unstale":"2024-09-13T14:41:29.584667+0000","last_undegraded":"2024-09-13T14:41:29.584667+0000","last_fullsized":"2024-09-13T14:41:29.584667+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:53:38.555537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1493,"reported_epoch":1105,"state":"active+clean","last_fresh":"2024-09-13T14:41:31.605642+0000","last_change":"2024-09-13T14:30:15.576921+0000","last_active":"2024-09-13T14:41:31.605642+0000","last_peered":"2024-09-13T14:41:31.605642+0000","last_clean":"2024-09-13T14:41:31.605642+0000","last_became_active":"2024-09-13T14:30:15.576607+0000","last_became_peered":"2024-09-13T14:30:15.576607+0000","last_unstale":"2024-09-13T14:41:31.605642+0000","last_undegraded":"2024-09-13T14:41:31.605642+0000","last_fullsized":"2024-09-13T14:41:31.605642+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:35.162460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1616,"reported_epoch":1105,"state":"active+clean","last_fresh":"2024-09-13T14:41:31.610913+0000","last_change":"2024-09-13T14:39:11.386005+0000","last_active":"2024-09-13T14:41:31.610913+0000","last_peered":"2024-09-13T14:41:31.610913+0000","last_clean":"2024-09-13T14:41:31.610913+0000","last_became_active":"2024-09-13T14:39:11.385691+0000","last_became_peered":"2024-09-13T14:39:11.385691+0000","last_unstale":"2024-09-13T14:41:31.610913+0000","last_undegraded":"2024-09-13T14:41:31.610913+0000","last_fullsized":"2024-09-13T14:41:31.610913+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:11:03.371484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":592,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.583732+0000","last_change":"2024-09-13T14:41:28.592154+0000","last_active":"2024-09-13T14:41:29.583732+0000","last_peered":"2024-09-13T14:41:29.583732+0000","last_clean":"2024-09-13T14:41:29.583732+0000","last_became_active":"2024-09-13T14:41:28.590786+0000","last_became_peered":"2024-09-13T14:41:28.590786+0000","last_unstale":"2024-09-13T14:41:29.583732+0000","last_undegraded":"2024-09-13T14:41:29.583732+0000","last_fullsized":"2024-09-13T14:41:29.583732+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:49:59.861513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1051,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.587436+0000","last_change":"2024-09-13T14:41:28.594659+0000","last_active":"2024-09-13T14:41:29.587436+0000","last_peered":"2024-09-13T14:41:29.587436+0000","last_clean":"2024-09-13T14:41:29.587436+0000","last_became_active":"2024-09-13T14:41:28.588903+0000","last_became_peered":"2024-09-13T14:41:28.588903+0000","last_unstale":"2024-09-13T14:41:29.587436+0000","last_undegraded":"2024-09-13T14:41:29.587436+0000","last_fullsized":"2024-09-13T14:41:29.587436+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:45:51.783371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":644,"reported_epoch":1105,"state":"active+clean","last_fresh":"2024-09-13T14:41:31.610127+0000","last_change":"2024-09-13T14:41:28.585670+0000","last_active":"2024-09-13T14:41:31.610127+0000","last_peered":"2024-09-13T14:41:31.610127+0000","last_clean":"2024-09-13T14:41:31.610127+0000","last_became_active":"2024-09-13T14:41:28.585173+0000","last_became_peered":"2024-09-13T14:41:28.585173+0000","last_unstale":"2024-09-13T14:41:31.610127+0000","last_undegraded":"2024-09-13T14:41:31.610127+0000","last_fullsized":"2024-09-13T14:41:31.610127+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:26:08.019099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1591,"reported_epoch":1105,"state":"active+clean","last_fresh":"2024-09-13T14:41:31.611026+0000","last_change":"2024-09-13T14:41:28.583750+0000","last_active":"2024-09-13T14:41:31.611026+0000","last_peered":"2024-09-13T14:41:31.611026+0000","last_clean":"2024-09-13T14:41:31.611026+0000","last_became_active":"2024-09-13T14:41:28.583293+0000","last_became_peered":"2024-09-13T14:41:28.583293+0000","last_unstale":"2024-09-13T14:41:31.611026+0000","last_undegraded":"2024-09-13T14:41:31.611026+0000","last_fullsized":"2024-09-13T14:41:31.611026+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:41:05.647080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":612,"reported_epoch":1105,"state":"active+clean","last_fresh":"2024-09-13T14:41:31.605730+0000","last_change":"2024-09-13T14:41:28.590716+0000","last_active":"2024-09-13T14:41:31.605730+0000","last_peered":"2024-09-13T14:41:31.605730+0000","last_clean":"2024-09-13T14:41:31.605730+0000","last_became_active":"2024-09-13T14:41:28.590372+0000","last_became_peered":"2024-09-13T14:41:28.590372+0000","last_unstale":"2024-09-13T14:41:31.605730+0000","last_undegraded":"2024-09-13T14:41:31.605730+0000","last_fullsized":"2024-09-13T14:41:31.605730+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:39.348905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":573,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.579973+0000","last_change":"2024-09-13T14:41:28.593532+0000","last_active":"2024-09-13T14:41:29.579973+0000","last_peered":"2024-09-13T14:41:29.579973+0000","last_clean":"2024-09-13T14:41:29.579973+0000","last_became_active":"2024-09-13T14:41:28.593235+0000","last_became_peered":"2024-09-13T14:41:28.593235+0000","last_unstale":"2024-09-13T14:41:29.579973+0000","last_undegraded":"2024-09-13T14:41:29.579973+0000","last_fullsized":"2024-09-13T14:41:29.579973+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:30:22.224415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":1074,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.580039+0000","last_change":"2024-09-13T14:41:28.590173+0000","last_active":"2024-09-13T14:41:29.580039+0000","last_peered":"2024-09-13T14:41:29.580039+0000","last_clean":"2024-09-13T14:41:29.580039+0000","last_became_active":"2024-09-13T14:41:28.589347+0000","last_became_peered":"2024-09-13T14:41:28.589347+0000","last_unstale":"2024-09-13T14:41:29.580039+0000","last_undegraded":"2024-09-13T14:41:29.580039+0000","last_fullsized":"2024-09-13T14:41:29.580039+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:19.873187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1574,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.598450+0000","last_change":"2024-09-13T14:41:28.584508+0000","last_active":"2024-09-13T14:41:29.598450+0000","last_peered":"2024-09-13T14:41:29.598450+0000","last_clean":"2024-09-13T14:41:29.598450+0000","last_became_active":"2024-09-13T14:41:28.584187+0000","last_became_peered":"2024-09-13T14:41:28.584187+0000","last_unstale":"2024-09-13T14:41:29.598450+0000","last_undegraded":"2024-09-13T14:41:29.598450+0000","last_fullsized":"2024-09-13T14:41:29.598450+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:21.364817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":1568,"reported_epoch":1105,"state":"active+clean","last_fresh":"2024-09-13T14:41:31.610572+0000","last_change":"2024-09-13T14:28:35.612469+0000","last_active":"2024-09-13T14:41:31.610572+0000","last_peered":"2024-09-13T14:41:31.610572+0000","last_clean":"2024-09-13T14:41:31.610572+0000","last_became_active":"2024-09-13T14:28:35.612093+0000","last_became_peered":"2024-09-13T14:28:35.612093+0000","last_unstale":"2024-09-13T14:41:31.610572+0000","last_undegraded":"2024-09-13T14:41:31.610572+0000","last_fullsized":"2024-09-13T14:41:31.610572+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:51.022663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1065,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.579823+0000","last_change":"2024-09-13T14:41:28.588664+0000","last_active":"2024-09-13T14:41:29.579823+0000","last_peered":"2024-09-13T14:41:29.579823+0000","last_clean":"2024-09-13T14:41:29.579823+0000","last_became_active":"2024-09-13T14:41:28.588283+0000","last_became_peered":"2024-09-13T14:41:28.588283+0000","last_unstale":"2024-09-13T14:41:29.579823+0000","last_undegraded":"2024-09-13T14:41:29.579823+0000","last_fullsized":"2024-09-13T14:41:29.579823+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:10:02.721554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":1064,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.591339+0000","last_change":"2024-09-13T14:41:28.603145+0000","last_active":"2024-09-13T14:41:29.591339+0000","last_peered":"2024-09-13T14:41:29.591339+0000","last_clean":"2024-09-13T14:41:29.591339+0000","last_became_active":"2024-09-13T14:41:28.590945+0000","last_became_peered":"2024-09-13T14:41:28.590945+0000","last_unstale":"2024-09-13T14:41:29.591339+0000","last_undegraded":"2024-09-13T14:41:29.591339+0000","last_fullsized":"2024-09-13T14:41:29.591339+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:01:21.700348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1062,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.579997+0000","last_change":"2024-09-13T14:30:15.580011+0000","last_active":"2024-09-13T14:41:29.579997+0000","last_peered":"2024-09-13T14:41:29.579997+0000","last_clean":"2024-09-13T14:41:29.579997+0000","last_became_active":"2024-09-13T14:30:15.579610+0000","last_became_peered":"2024-09-13T14:30:15.579610+0000","last_unstale":"2024-09-13T14:41:29.579997+0000","last_undegraded":"2024-09-13T14:41:29.579997+0000","last_fullsized":"2024-09-13T14:41:29.579997+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:11:02.733771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":1033,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.591590+0000","last_change":"2024-09-13T14:19:25.873514+0000","last_active":"2024-09-13T14:41:29.591590+0000","last_peered":"2024-09-13T14:41:29.591590+0000","last_clean":"2024-09-13T14:41:29.591590+0000","last_became_active":"2024-09-13T14:19:25.873170+0000","last_became_peered":"2024-09-13T14:19:25.873170+0000","last_unstale":"2024-09-13T14:41:29.591590+0000","last_undegraded":"2024-09-13T14:41:29.591590+0000","last_fullsized":"2024-09-13T14:41:29.591590+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:48.298530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":642,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.598916+0000","last_change":"2024-09-13T14:39:11.396732+0000","last_active":"2024-09-13T14:41:29.598916+0000","last_peered":"2024-09-13T14:41:29.598916+0000","last_clean":"2024-09-13T14:41:29.598916+0000","last_became_active":"2024-09-13T14:39:11.396322+0000","last_became_peered":"2024-09-13T14:39:11.396322+0000","last_unstale":"2024-09-13T14:41:29.598916+0000","last_undegraded":"2024-09-13T14:41:29.598916+0000","last_fullsized":"2024-09-13T14:41:29.598916+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:54:47.496762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":67,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.587033+0000","last_change":"2024-09-13T14:41:28.586563+0000","last_active":"2024-09-13T14:41:29.587033+0000","last_peered":"2024-09-13T14:41:29.587033+0000","last_clean":"2024-09-13T14:41:29.587033+0000","last_became_active":"2024-09-13T14:41:28.586152+0000","last_became_peered":"2024-09-13T14:41:28.586152+0000","last_unstale":"2024-09-13T14:41:29.587033+0000","last_undegraded":"2024-09-13T14:41:29.587033+0000","last_fullsized":"2024-09-13T14:41:29.587033+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:28.618035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1055,"reported_epoch":1104,"state":"active+clean","last_fresh":"2024-09-13T14:41:29.579479+0000","last_change":"2024-09-13T14:29:42.539907+0000","last_active":"2024-09-13T14:41:29.579479+0000","last_peered":"2024-09-13T14:41:29.579479+0000","last_clean":"2024-09-13T14:41:29.579479+0000","last_became_active":"2024-09-13T14:29:42.539583+0000","last_became_peered":"2024-09-13T14:29:42.539583+0000","last_unstale":"2024-09-13T14:41:29.579479+0000","last_undegraded":"2024-09-13T14:41:29.579479+0000","last_fullsized":"2024-09-13T14:41:29.579479+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:02:04.628447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":1583,"reported_epoch":1105,"state":"active+clean","last_fresh":"2024-09-13T14:41:31.611008+0000","last_change":"2024-09-13T14:29:42.541263+0000","last_active":"2024-09-13T14:41:31.611008+0000","last_peered":"2024-09-13T14:41:31.611008+0000","last_clean":"2024-09-13T14:41:31.611008+0000","last_became_active":"2024-09-13T14:29:42.540873+0000","last_became_peered":"2024-09-13T14:29:42.540873+0000","last_unstale":"2024-09-13T14:41:31.611008+0000","last_undegraded":"2024-09-13T14:41:31.611008+0000","last_fullsized":"2024-09-13T14:41:31.611008+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:20.611206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":6751387672,"num_objects":103019,"num_object_clones":0,"num_object_copies":309057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":141,"num_objects_misplaced":31449,"num_objects_unfound":0,"num_objects_dirty":103019,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161893,"num_write_kb":8477121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":737,"num_bytes_recovered":48300032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20345008128,"data_stored":20344995912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105795,"ondisk_log_size":105795,"up":48,"acting":47,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":488690,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":79,"num_read_kb":204,"num_write":40,"num_write_kb":444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":423698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1100,"ondisk_log_size":1100,"up":33,"acting":33,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430498,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3216460,"kb_used_data":2906628,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":90528692,"statfs":{"total":95995035648,"available":92701380608,"internally_reserved":0,"allocated":2976387072,"data_stored":2973156674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626727,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4444240,"kb_used_data":4134336,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":89300912,"statfs":{"total":95995035648,"available":91444133888,"internally_reserved":0,"allocated":4233560064,"data_stored":4230334187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822956,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2828412,"kb_used_data":2526768,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":90916740,"statfs":{"total":95995035648,"available":93098741760,"internally_reserved":0,"allocated":2587410432,"data_stored":2584188376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019184,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2405256,"kb_used_data":2095424,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91339896,"statfs":{"total":95995035648,"available":93532053504,"internally_reserved":0,"allocated":2145714176,"data_stored":2142487804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059592842,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1317432,"kb_used_data":1241132,"kb_used_omap":26,"kb_used_meta":76261,"kb_avail":92427720,"statfs":{"total":95995035648,"available":94645985280,"internally_reserved":0,"allocated":1270919168,"data_stored":1267708496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":78091424},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":591,"seq":2538325672080,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3516284,"kb_used_data":3367592,"kb_used_omap":41,"kb_used_meta":148630,"kb_avail":90228868,"statfs":{"total":95995035648,"available":92394360832,"internally_reserved":0,"allocated":3448414208,"data_stored":3445200293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42651,"internal_metadata":152197477},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":2392296784023,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":524,"seq":2250562863262,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1742656,"kb_used_data":1656504,"kb_used_omap":41,"kb_used_meta":86102,"kb_avail":92002496,"statfs":{"total":95995035648,"available":94210555904,"internally_reserved":0,"allocated":1696260096,"data_stored":1693037705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42694,"internal_metadata":88168762},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1673465856,"data_stored":1673461784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2122649600,"data_stored":2122645528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3421306880,"data_stored":3421306880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1249640448,"data_stored":1249640448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2130771968,"data_stored":2130771968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2565672960,"data_stored":2565668888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4219600896,"data_stored":4219600896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2961899520,"data_stored":2961899520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:41:38.669 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-13T14:41:38.670 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_1 pg_num 26 2024-09-13T14:41:38.710 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:38.737 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:38.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:38.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:38.797 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:41:38.839 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:38.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:38.900 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:38.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:38.940 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:38 smithi184 ceph-mon[92527]: from='client.45332 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:38.941 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:38 smithi184 ceph-mon[92527]: pgmap v1377: 99 pgs: 2 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.2 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 422 op/s; 141/306186 objects degraded (0.046%); 30591/306186 objects misplaced (9.991%); 11 MiB/s, 173 objects/s recovering 2024-09-13T14:41:38.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:38 smithi151 ceph-mon[104450]: from='client.45332 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:38.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:38 smithi151 ceph-mon[104450]: pgmap v1377: 99 pgs: 2 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.2 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 422 op/s; 141/306186 objects degraded (0.046%); 30591/306186 objects misplaced (9.991%); 11 MiB/s, 173 objects/s recovering 2024-09-13T14:41:38.941 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:38 smithi151 ceph-mon[108966]: from='client.45332 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:38.942 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:38 smithi151 ceph-mon[108966]: pgmap v1377: 99 pgs: 2 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.2 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 422 op/s; 141/306186 objects degraded (0.046%); 30591/306186 objects misplaced (9.991%); 11 MiB/s, 173 objects/s recovering 2024-09-13T14:41:39.001 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:39.034 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:39.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:39.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:39.063 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:39.073 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:39.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:39.166 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:39.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:39.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:39.268 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:39.298 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:39.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:39.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:39.400 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:39.428 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:39.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:39.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:39.531 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:39.575 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:39.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:39.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:39.677 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:39.683 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-13T14:41:39.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:39.721 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:39.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:39.822 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:39.847 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:39.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:39.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:39.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:39 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2141409054' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:41:39.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:39 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:41:39.896 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:39 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2141409054' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:41:39.896 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:39 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:41:39.897 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:39 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2141409054' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:41:39.897 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:39 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:41:39.949 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:39.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:39.984 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:39.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:40.085 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:40.131 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:40.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:40.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:40.234 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:40.259 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:40.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:40.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:40.361 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:40.389 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:40.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:40.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:40.477 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:41:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:41:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:41:40.491 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:40.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:40.519 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:40.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:40.620 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:40.646 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:40.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:40.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:40.748 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:40.774 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:40.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:40.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:40.781 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:40 smithi184 ceph-mon[92527]: pgmap v1378: 99 pgs: 2 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 641 op/s; 335/297501 objects degraded (0.113%); 28966/297501 objects misplaced (9.736%); 14 MiB/s, 220 objects/s recovering 2024-09-13T14:41:40.781 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:40 smithi184 ceph-mon[92527]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:40.781 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:40 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:41:40.781 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:40 smithi184 ceph-mon[92527]: osdmap e1106: 8 total, 8 up, 7 in 2024-09-13T14:41:40.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:40.904 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:40.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:40.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:41.007 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:41.034 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:41.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:41.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:41.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:40 smithi151 ceph-mon[104450]: pgmap v1378: 99 pgs: 2 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 641 op/s; 335/297501 objects degraded (0.113%); 28966/297501 objects misplaced (9.736%); 14 MiB/s, 220 objects/s recovering 2024-09-13T14:41:41.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:40 smithi151 ceph-mon[104450]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:41.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:40 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:41:41.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:40 smithi151 ceph-mon[104450]: osdmap e1106: 8 total, 8 up, 7 in 2024-09-13T14:41:41.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:40 smithi151 ceph-mon[108966]: pgmap v1378: 99 pgs: 2 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 641 op/s; 335/297501 objects degraded (0.113%); 28966/297501 objects misplaced (9.736%); 14 MiB/s, 220 objects/s recovering 2024-09-13T14:41:41.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:40 smithi151 ceph-mon[108966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:41.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:40 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:41:41.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:40 smithi151 ceph-mon[108966]: osdmap e1106: 8 total, 8 up, 7 in 2024-09-13T14:41:41.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:41.164 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:41.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:41.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:41.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:41.293 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:41.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:41.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:41.396 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:41.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:41.431 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:41.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:41.532 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:41.558 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:41.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:41.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:41.660 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:41.689 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:41.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:41.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:41.792 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:41.817 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:41.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:41.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:41.919 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:41.945 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:41.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:41.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:42.043 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:41 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 335/296772 objects degraded (0.113%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:41:42.044 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:41 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 335/296772 objects degraded (0.113%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:41:42.047 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:42.077 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:42.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:42.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:42.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:41 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 335/296772 objects degraded (0.113%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:41:42.178 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:42.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:42.213 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:42.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:42.314 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:42.343 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:42.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:42.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:42.445 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:42.485 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:42.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:42.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:42.587 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:42.611 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:42.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:42.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:42.713 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:42.741 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:42.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:42.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:42.843 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:42.867 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:42.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:42.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:42.924 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:42 smithi184 ceph-mon[92527]: pgmap v1380: 99 pgs: 2 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 651 op/s; 335/296772 objects degraded (0.113%); 28966/296772 objects misplaced (9.760%); 14 MiB/s, 220 objects/s recovering 2024-09-13T14:41:42.961 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:42 smithi151 ceph-mon[104450]: pgmap v1380: 99 pgs: 2 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 651 op/s; 335/296772 objects degraded (0.113%); 28966/296772 objects misplaced (9.760%); 14 MiB/s, 220 objects/s recovering 2024-09-13T14:41:42.961 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:42 smithi151 ceph-mon[108966]: pgmap v1380: 99 pgs: 2 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 651 op/s; 335/296772 objects degraded (0.113%); 28966/296772 objects misplaced (9.760%); 14 MiB/s, 220 objects/s recovering 2024-09-13T14:41:42.969 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:43.001 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:43.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:43.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:43.103 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:43.127 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:43.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:43.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:43.229 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:43.267 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:43.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:43.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:43.369 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:43.399 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:43.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:43.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:43.501 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:43.544 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:43.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:43.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:43.641 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:41:43.641 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:41:43.646 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:43.663 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:41:43.664 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:41:43.682 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:41:43.682 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:41:43.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:43.695 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:43.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:43.703 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:41:43.703 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:41:43.733 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:41:43.734 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:41:43.766 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:41:43.767 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:41:43.767 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:43 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/4228042015' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:43.767 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:43 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/4228042015' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:43.796 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:43.805 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:41:43.805 INFO:teuthology.orchestra.run.smithi184.stdout: "size": 20, 2024-09-13T14:41:43.805 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 600, 2024-09-13T14:41:43.805 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [ 2024-09-13T14:41:43.805 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.805 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:63706 6.c 6:395e08a0:::benchmark_data_smithi151_155836_object63705:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1088)", 2024-09-13T14:41:43.805 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:39:55.559837+0000", 2024-09-13T14:41:43.806 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 108.241790481, 2024-09-13T14:41:43.806 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.096658093, 2024-09-13T14:41:43.806 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:43.806 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:43.806 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:43.806 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:43.806 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:43.806 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 63706 2024-09-13T14:41:43.806 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.806 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:43.806 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.806 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:43.806 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.559837+0000", 2024-09-13T14:41:43.807 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.807 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.807 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.809 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:43.809 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.559837+0000", 2024-09-13T14:41:43.809 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.809 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.809 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.809 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:43.809 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.559835+0000", 2024-09-13T14:41:43.809 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:43.809 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.809 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.809 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:43.809 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.559885+0000", 2024-09-13T14:41:43.810 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9701999999999999e-05 2024-09-13T14:41:43.810 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.810 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.810 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:43.810 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.559898+0000", 2024-09-13T14:41:43.810 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.27e-05 2024-09-13T14:41:43.810 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.810 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.810 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:43.810 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.559903+0000", 2024-09-13T14:41:43.810 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5219999999999998e-06 2024-09-13T14:41:43.810 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.810 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.810 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:43.811 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.559957+0000", 2024-09-13T14:41:43.811 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3665999999999997e-05 2024-09-13T14:41:43.811 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.811 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.811 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:43.811 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.560064+0000", 2024-09-13T14:41:43.811 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000107279 2024-09-13T14:41:43.811 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.811 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.811 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 5,7", 2024-09-13T14:41:43.811 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.560251+0000", 2024-09-13T14:41:43.811 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00018706899999999999 2024-09-13T14:41:43.811 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.812 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.812 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:43.812 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.564371+0000", 2024-09-13T14:41:43.812 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0041198340000000002 2024-09-13T14:41:43.812 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.812 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.812 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:43.812 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.565408+0000", 2024-09-13T14:41:43.812 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001036421 2024-09-13T14:41:43.812 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.812 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.812 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:43.812 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.656365+0000", 2024-09-13T14:41:43.813 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.090957283999999999 2024-09-13T14:41:43.813 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.813 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.813 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:43.813 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.656456+0000", 2024-09-13T14:41:43.813 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.1090999999999999e-05 2024-09-13T14:41:43.813 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.813 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.813 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:43.813 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.656495+0000", 2024-09-13T14:41:43.813 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9116999999999998e-05 2024-09-13T14:41:43.813 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.813 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:43.813 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.814 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.814 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.814 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35289.0:162669 6.b e1105/1103 6:d2cfb8c7:::benchmark_data_smithi151_155836_object30211:head v 1105'10250, mlcod=1087'1918)", 2024-09-13T14:41:43.814 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:33.514641+0000", 2024-09-13T14:41:43.814 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 10.286986098, 2024-09-13T14:41:43.814 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.096299269000000007, 2024-09-13T14:41:43.814 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:43.814 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:43.814 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:43.814 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.814 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:43.814 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.514641+0000", 2024-09-13T14:41:43.814 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.814 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.815 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.815 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:43.815 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.514641+0000", 2024-09-13T14:41:43.815 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.815 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.815 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.815 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:43.815 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.514640+0000", 2024-09-13T14:41:43.815 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:43.815 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.815 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.815 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:43.815 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.514646+0000", 2024-09-13T14:41:43.816 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.2230000000000004e-06 2024-09-13T14:41:43.816 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.816 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.816 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:43.816 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.514665+0000", 2024-09-13T14:41:43.816 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.8552000000000001e-05 2024-09-13T14:41:43.816 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.816 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.816 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:43.816 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.514671+0000", 2024-09-13T14:41:43.816 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1680000000000001e-06 2024-09-13T14:41:43.816 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.816 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.816 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:43.817 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.532293+0000", 2024-09-13T14:41:43.817 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.017622386 2024-09-13T14:41:43.817 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.817 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.817 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:43.817 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.532323+0000", 2024-09-13T14:41:43.817 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.9569000000000001e-05 2024-09-13T14:41:43.817 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.817 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.817 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:43.817 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.610901+0000", 2024-09-13T14:41:43.817 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.078577548999999997 2024-09-13T14:41:43.817 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.818 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.818 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:43.818 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.610941+0000", 2024-09-13T14:41:43.818 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0116e-05 2024-09-13T14:41:43.818 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.818 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:43.818 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.818 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.818 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.818 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35289.0:162675 6.b e1105/1103 6:d4281e38:::benchmark_data_smithi151_155836_object30217:head v 1105'10251, mlcod=1087'1918)", 2024-09-13T14:41:43.818 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:33.522893+0000", 2024-09-13T14:41:43.818 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 10.278734907, 2024-09-13T14:41:43.818 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.095205729000000003, 2024-09-13T14:41:43.818 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:43.819 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:43.819 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:43.819 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.819 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:43.819 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.522893+0000", 2024-09-13T14:41:43.819 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.819 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.819 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.819 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:43.819 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.522893+0000", 2024-09-13T14:41:43.819 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.819 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.819 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.820 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:43.820 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.522892+0000", 2024-09-13T14:41:43.820 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:41:43.820 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.820 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.820 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:43.820 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.522900+0000", 2024-09-13T14:41:43.820 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.1610000000000003e-06 2024-09-13T14:41:43.820 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.820 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.820 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:43.820 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.522914+0000", 2024-09-13T14:41:43.820 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4537e-05 2024-09-13T14:41:43.821 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.821 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.821 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:43.821 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.522920+0000", 2024-09-13T14:41:43.821 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.0929999999999999e-06 2024-09-13T14:41:43.821 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.821 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.821 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:43.822 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:41:43.822 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:41:43.822 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.545308+0000", 2024-09-13T14:41:43.823 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.022387576999999999 2024-09-13T14:41:43.823 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.823 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.823 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:43.823 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.545402+0000", 2024-09-13T14:41:43.823 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.3749e-05 2024-09-13T14:41:43.823 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.823 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.823 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:43.823 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.614615+0000", 2024-09-13T14:41:43.823 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.069213151000000001 2024-09-13T14:41:43.823 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.823 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.824 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:43.824 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.618098+0000", 2024-09-13T14:41:43.824 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0034834829999999999 2024-09-13T14:41:43.824 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.824 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:43.824 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.824 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.824 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.824 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35289.0:162889 6.b e1105/1103 6:d68e3120:::benchmark_data_smithi151_155836_object30431:head v 1105'10268, mlcod=1087'1932)", 2024-09-13T14:41:43.824 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:33.816349+0000", 2024-09-13T14:41:43.824 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 9.9852783370000004, 2024-09-13T14:41:43.824 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.097149745999999995, 2024-09-13T14:41:43.824 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:43.825 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:43.825 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:43.825 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.825 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:43.825 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.816349+0000", 2024-09-13T14:41:43.825 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.825 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.825 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.825 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:43.825 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.816349+0000", 2024-09-13T14:41:43.825 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.825 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.825 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.825 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:43.826 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.816348+0000", 2024-09-13T14:41:43.826 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:41:43.826 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.826 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.826 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:43.826 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.816355+0000", 2024-09-13T14:41:43.826 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.0559999999999999e-06 2024-09-13T14:41:43.826 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.826 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.826 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:43.826 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.816371+0000", 2024-09-13T14:41:43.826 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.5661000000000001e-05 2024-09-13T14:41:43.826 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.826 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.827 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:43.827 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.816377+0000", 2024-09-13T14:41:43.827 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5280000000000003e-06 2024-09-13T14:41:43.827 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.827 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.827 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:43.827 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.825339+0000", 2024-09-13T14:41:43.827 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0089628169999999997 2024-09-13T14:41:43.827 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.827 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.827 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:43.827 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.825408+0000", 2024-09-13T14:41:43.827 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.8015999999999994e-05 2024-09-13T14:41:43.828 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.828 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.828 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:43.828 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.913459+0000", 2024-09-13T14:41:43.828 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.088051931999999999 2024-09-13T14:41:43.828 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.828 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.828 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:43.828 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.913499+0000", 2024-09-13T14:41:43.828 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9434000000000002e-05 2024-09-13T14:41:43.828 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.828 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:43.828 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.828 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.829 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.829 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "MOSDPGPush(6.b 1105/1103 [PushOp(6:da5d77d1:::benchmark_data_smithi151_155836_object32099:head, version: 1087'2031, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:da5d77d1:::benchmark_data_smithi151_155836_object32099:head@1087'2031, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:41:43.829 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:35.917377+0000", 2024-09-13T14:41:43.829 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 7.8842508059999998, 2024-09-13T14:41:43.829 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.11572030699999999, 2024-09-13T14:41:43.829 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:43.829 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:41:43.829 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:43.829 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.829 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:43.829 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.917377+0000", 2024-09-13T14:41:43.829 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.829 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.830 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.830 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:43.830 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.917377+0000", 2024-09-13T14:41:43.830 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.830 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.830 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.830 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:43.830 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.917376+0000", 2024-09-13T14:41:43.830 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:41:43.830 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.830 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.830 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:43.830 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.917426+0000", 2024-09-13T14:41:43.830 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0241999999999998e-05 2024-09-13T14:41:43.831 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.831 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.831 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:43.831 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.917446+0000", 2024-09-13T14:41:43.831 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.0012000000000001e-05 2024-09-13T14:41:43.831 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.831 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.831 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:43.831 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.917455+0000", 2024-09-13T14:41:43.831 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.0869999999999996e-06 2024-09-13T14:41:43.831 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.831 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.831 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:43.831 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.028211+0000", 2024-09-13T14:41:43.832 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.110755874 2024-09-13T14:41:43.832 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.832 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.832 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:43.832 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.028222+0000", 2024-09-13T14:41:43.832 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0728e-05 2024-09-13T14:41:43.832 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.832 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.833 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:43.833 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.033097+0000", 2024-09-13T14:41:43.833 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0048750740000000001 2024-09-13T14:41:43.833 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.833 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:43.833 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.833 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.833 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.833 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35289.0:164844 6.b e1105/1103 6:d64eed8f:::benchmark_data_smithi151_155836_object32386:head v 1105'10382, mlcod=1087'2044)", 2024-09-13T14:41:43.833 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:36.303049+0000", 2024-09-13T14:41:43.833 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 7.4985786860000001, 2024-09-13T14:41:43.833 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.094911832000000002, 2024-09-13T14:41:43.833 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:43.833 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:43.834 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:43.834 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.834 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:43.834 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.303049+0000", 2024-09-13T14:41:43.834 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.834 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.834 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.834 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:43.834 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.303049+0000", 2024-09-13T14:41:43.834 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.835 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.835 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.835 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:43.835 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.303048+0000", 2024-09-13T14:41:43.835 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:41:43.835 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.835 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.835 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:43.835 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.303056+0000", 2024-09-13T14:41:43.835 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.4090000000000003e-06 2024-09-13T14:41:43.835 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.835 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.835 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:43.835 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.303072+0000", 2024-09-13T14:41:43.836 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6022e-05 2024-09-13T14:41:43.836 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.836 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.836 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:43.836 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.303079+0000", 2024-09-13T14:41:43.836 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.4289999999999998e-06 2024-09-13T14:41:43.836 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.836 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.836 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:43.836 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.321567+0000", 2024-09-13T14:41:43.836 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.018488600000000001 2024-09-13T14:41:43.836 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.836 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.837 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:43.837 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.321590+0000", 2024-09-13T14:41:43.837 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.2643999999999999e-05 2024-09-13T14:41:43.837 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.837 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.837 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:43.837 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.397883+0000", 2024-09-13T14:41:43.837 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.076292813000000001 2024-09-13T14:41:43.837 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.837 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.837 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:43.837 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.397961+0000", 2024-09-13T14:41:43.837 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.7929000000000006e-05 2024-09-13T14:41:43.838 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.838 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:43.838 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.838 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.838 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.838 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35289.0:164855 6.b e1105/1103 6:d8fddcfc:::benchmark_data_smithi151_155836_object32397:head v 1105'10383, mlcod=1087'2044)", 2024-09-13T14:41:43.838 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:36.317865+0000", 2024-09-13T14:41:43.838 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 7.4837629149999998, 2024-09-13T14:41:43.838 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.106768057, 2024-09-13T14:41:43.838 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:43.838 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:43.838 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:43.838 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.838 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:43.839 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.317865+0000", 2024-09-13T14:41:43.839 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.839 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.839 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.839 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:43.840 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.317865+0000", 2024-09-13T14:41:43.840 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.840 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.840 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.840 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:43.840 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.317864+0000", 2024-09-13T14:41:43.840 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:41:43.840 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.840 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.840 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:43.840 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.317889+0000", 2024-09-13T14:41:43.841 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.4672999999999999e-05 2024-09-13T14:41:43.841 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.841 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.841 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:43.841 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.317903+0000", 2024-09-13T14:41:43.841 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4204e-05 2024-09-13T14:41:43.841 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.841 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.841 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:43.841 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.317908+0000", 2024-09-13T14:41:43.841 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3750000000000002e-06 2024-09-13T14:41:43.841 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.841 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.841 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:43.842 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.335999+0000", 2024-09-13T14:41:43.842 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.018090563 2024-09-13T14:41:43.842 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.842 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.842 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:43.842 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.336024+0000", 2024-09-13T14:41:43.842 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.5303000000000002e-05 2024-09-13T14:41:43.842 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.842 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.842 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:43.842 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.424592+0000", 2024-09-13T14:41:43.842 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.088568092000000001 2024-09-13T14:41:43.842 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.842 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.843 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:43.843 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.424633+0000", 2024-09-13T14:41:43.843 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0462e-05 2024-09-13T14:41:43.843 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.843 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:43.843 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.843 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.843 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.843 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35289.0:164864 6.b e1105/1103 6:d4d854eb:::benchmark_data_smithi151_155836_object32406:head v 1105'10384, mlcod=1087'2044)", 2024-09-13T14:41:43.843 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:36.323170+0000", 2024-09-13T14:41:43.843 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 7.4784572899999997, 2024-09-13T14:41:43.843 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.10148460199999999, 2024-09-13T14:41:43.843 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:43.844 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:43.844 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:43.844 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.844 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:43.844 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.323170+0000", 2024-09-13T14:41:43.844 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.844 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.844 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.844 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:43.844 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.323170+0000", 2024-09-13T14:41:43.844 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.844 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.844 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.844 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:43.845 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.323169+0000", 2024-09-13T14:41:43.845 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:41:43.845 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.845 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.845 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:43.845 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.323175+0000", 2024-09-13T14:41:43.845 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.463e-06 2024-09-13T14:41:43.845 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.845 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.845 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:43.845 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.323189+0000", 2024-09-13T14:41:43.845 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3957999999999999e-05 2024-09-13T14:41:43.845 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.846 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.846 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:43.846 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.323194+0000", 2024-09-13T14:41:43.846 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3020000000000002e-06 2024-09-13T14:41:43.846 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.846 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.846 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:43.846 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.349139+0000", 2024-09-13T14:41:43.846 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.025944361999999999 2024-09-13T14:41:43.846 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.846 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.846 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:43.846 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.349160+0000", 2024-09-13T14:41:43.846 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.1713999999999999e-05 2024-09-13T14:41:43.847 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.847 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.847 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:43.847 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.424636+0000", 2024-09-13T14:41:43.847 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.075475940000000005 2024-09-13T14:41:43.847 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.847 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.847 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:43.847 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.424655+0000", 2024-09-13T14:41:43.847 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.8553999999999999e-05 2024-09-13T14:41:43.847 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.847 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:43.847 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.847 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.848 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.848 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35289.0:164867 6.b e1105/1103 6:dbac94af:::benchmark_data_smithi151_155836_object32409:head v 1105'10385, mlcod=1087'2044)", 2024-09-13T14:41:43.848 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:36.327841+0000", 2024-09-13T14:41:43.848 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 7.4737862440000002, 2024-09-13T14:41:43.848 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.096833733000000005, 2024-09-13T14:41:43.848 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:43.848 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:43.848 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:43.848 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.848 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:43.848 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.327841+0000", 2024-09-13T14:41:43.848 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.848 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.848 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.849 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:43.849 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.327841+0000", 2024-09-13T14:41:43.849 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.849 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.849 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.849 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:43.849 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.327840+0000", 2024-09-13T14:41:43.849 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:43.849 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.849 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.849 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:43.849 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.327847+0000", 2024-09-13T14:41:43.849 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.3989999999999998e-06 2024-09-13T14:41:43.849 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.850 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.850 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:43.850 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.327859+0000", 2024-09-13T14:41:43.850 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2119e-05 2024-09-13T14:41:43.850 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.850 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.850 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:43.850 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.327865+0000", 2024-09-13T14:41:43.850 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7019999999999998e-06 2024-09-13T14:41:43.850 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.850 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.850 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:43.850 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.361240+0000", 2024-09-13T14:41:43.851 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.033374446000000002 2024-09-13T14:41:43.851 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.851 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.851 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:43.851 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.361263+0000", 2024-09-13T14:41:43.851 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.3879999999999998e-05 2024-09-13T14:41:43.851 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.851 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.851 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:43.851 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.424657+0000", 2024-09-13T14:41:43.851 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.063393566999999998 2024-09-13T14:41:43.851 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.851 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.851 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:43.852 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.424675+0000", 2024-09-13T14:41:43.852 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.7977000000000001e-05 2024-09-13T14:41:43.852 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.852 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:43.852 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.852 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.852 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.852 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35289.0:165062 6.3 e1105/1082 6:ce8e0480:::benchmark_data_smithi151_155836_object32604:head v 1105'10042, mlcod=1105'10037)", 2024-09-13T14:41:43.852 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:36.600865+0000", 2024-09-13T14:41:43.852 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 7.2007628380000002, 2024-09-13T14:41:43.852 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.114697643, 2024-09-13T14:41:43.852 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:43.852 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:43.853 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:43.853 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.853 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:43.853 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.600865+0000", 2024-09-13T14:41:43.853 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.853 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.853 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.853 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:43.853 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.600865+0000", 2024-09-13T14:41:43.853 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.853 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.853 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.853 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:43.853 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.600864+0000", 2024-09-13T14:41:43.854 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:41:43.854 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.854 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.854 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:43.854 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.600877+0000", 2024-09-13T14:41:43.854 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2799000000000001e-05 2024-09-13T14:41:43.854 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.854 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.854 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:43.854 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.600984+0000", 2024-09-13T14:41:43.854 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000107799 2024-09-13T14:41:43.854 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.854 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.855 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:43.855 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.600991+0000", 2024-09-13T14:41:43.855 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.2580000000000001e-06 2024-09-13T14:41:43.855 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.855 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.855 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:43.855 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.625549+0000", 2024-09-13T14:41:43.855 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.024558217 2024-09-13T14:41:43.855 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.855 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.855 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:43.855 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.625566+0000", 2024-09-13T14:41:43.855 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.7129000000000002e-05 2024-09-13T14:41:43.855 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.856 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.856 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:43.856 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.715524+0000", 2024-09-13T14:41:43.856 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.089958163999999993 2024-09-13T14:41:43.856 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.856 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.856 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:43.856 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.715562+0000", 2024-09-13T14:41:43.856 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8025000000000003e-05 2024-09-13T14:41:43.856 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.856 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:43.856 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.856 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.857 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.857 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35289.0:165068 6.b e1105/1103 6:d06194e9:::benchmark_data_smithi151_155836_object32610:head v 1105'10402, mlcod=1087'2061)", 2024-09-13T14:41:43.857 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:36.608085+0000", 2024-09-13T14:41:43.857 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 7.193542205, 2024-09-13T14:41:43.857 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.118391099, 2024-09-13T14:41:43.857 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:43.857 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:43.857 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:43.857 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.857 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:43.857 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.608085+0000", 2024-09-13T14:41:43.857 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.857 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.857 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.858 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:43.858 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.608085+0000", 2024-09-13T14:41:43.858 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.858 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.858 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.858 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:43.858 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.608084+0000", 2024-09-13T14:41:43.858 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:41:43.858 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.858 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.858 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:43.858 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.608092+0000", 2024-09-13T14:41:43.858 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.542e-06 2024-09-13T14:41:43.858 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.859 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.859 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:43.859 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.608107+0000", 2024-09-13T14:41:43.859 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.5075000000000001e-05 2024-09-13T14:41:43.859 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.859 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.859 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:43.859 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.608113+0000", 2024-09-13T14:41:43.859 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.127e-06 2024-09-13T14:41:43.859 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.859 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.859 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:43.859 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.625463+0000", 2024-09-13T14:41:43.859 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.017349694999999998 2024-09-13T14:41:43.859 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.860 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.860 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:43.860 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.625826+0000", 2024-09-13T14:41:43.860 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00036309800000000001 2024-09-13T14:41:43.860 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.860 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.860 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:43.860 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.726399+0000", 2024-09-13T14:41:43.860 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.100573492 2024-09-13T14:41:43.860 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.860 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.860 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:43.860 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.726476+0000", 2024-09-13T14:41:43.860 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.7053999999999998e-05 2024-09-13T14:41:43.861 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.861 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:43.861 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.861 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.861 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.861 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35289.0:165071 6.3 e1105/1082 6:cccdb842:::benchmark_data_smithi151_155836_object32613:head v 1105'10043, mlcod=1105'10039)", 2024-09-13T14:41:43.861 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:36.609711+0000", 2024-09-13T14:41:43.861 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 7.1919164120000003, 2024-09-13T14:41:43.861 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.10587777299999999, 2024-09-13T14:41:43.861 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:43.861 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:43.861 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:43.861 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.862 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:43.862 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.609711+0000", 2024-09-13T14:41:43.862 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.862 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.862 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.862 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:43.862 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.609711+0000", 2024-09-13T14:41:43.862 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.862 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.862 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.862 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:43.862 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.609710+0000", 2024-09-13T14:41:43.862 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:41:43.862 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.862 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.863 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:43.863 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.609717+0000", 2024-09-13T14:41:43.863 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.6139999999999997e-06 2024-09-13T14:41:43.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.863 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.863 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:43.863 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.609730+0000", 2024-09-13T14:41:43.863 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2548000000000001e-05 2024-09-13T14:41:43.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.863 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.864 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:43.864 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.609735+0000", 2024-09-13T14:41:43.864 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3090000000000003e-06 2024-09-13T14:41:43.864 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.864 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.864 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:43.864 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.628635+0000", 2024-09-13T14:41:43.864 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.018899796 2024-09-13T14:41:43.864 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.864 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.864 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:43.864 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.628685+0000", 2024-09-13T14:41:43.864 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0512e-05 2024-09-13T14:41:43.865 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.865 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.865 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:43.865 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.715566+0000", 2024-09-13T14:41:43.865 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.086880392000000001 2024-09-13T14:41:43.865 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.865 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.865 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:43.865 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.715589+0000", 2024-09-13T14:41:43.865 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.3243000000000001e-05 2024-09-13T14:41:43.865 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.865 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:43.865 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.865 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.866 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.866 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35289.0:165069 6.b e1105/1103 6:d2c398a2:::benchmark_data_smithi151_155836_object32611:head v 1105'10403, mlcod=1087'2061)", 2024-09-13T14:41:43.867 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:36.614248+0000", 2024-09-13T14:41:43.867 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 7.1873797130000003, 2024-09-13T14:41:43.867 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.11228423, 2024-09-13T14:41:43.867 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:43.867 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:43.867 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:43.867 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.867 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:43.868 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.614248+0000", 2024-09-13T14:41:43.868 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.868 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.868 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.868 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:43.868 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.614248+0000", 2024-09-13T14:41:43.868 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.868 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.868 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.868 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:43.868 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.614247+0000", 2024-09-13T14:41:43.868 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:41:43.868 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.869 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.869 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:43.869 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.614254+0000", 2024-09-13T14:41:43.869 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.9550000000000003e-06 2024-09-13T14:41:43.869 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.869 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.869 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:43.869 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.614277+0000", 2024-09-13T14:41:43.869 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.281e-05 2024-09-13T14:41:43.869 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.869 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.869 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:43.869 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.614283+0000", 2024-09-13T14:41:43.869 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.0680000000000002e-06 2024-09-13T14:41:43.870 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.870 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.870 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:43.870 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.641788+0000", 2024-09-13T14:41:43.870 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.027504809000000002 2024-09-13T14:41:43.870 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.870 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.870 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:43.870 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.641809+0000", 2024-09-13T14:41:43.870 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.1509999999999999e-05 2024-09-13T14:41:43.870 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.870 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.870 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:43.870 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.726482+0000", 2024-09-13T14:41:43.871 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.084673359000000004 2024-09-13T14:41:43.871 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.871 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.871 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:43.871 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.726532+0000", 2024-09-13T14:41:43.871 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9552999999999998e-05 2024-09-13T14:41:43.871 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.871 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:43.871 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.871 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.871 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.871 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35289.0:165077 6.b e1105/1103 6:de07d0d6:::benchmark_data_smithi151_155836_object32619:head v 1105'10404, mlcod=1087'2061)", 2024-09-13T14:41:43.871 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:36.616926+0000", 2024-09-13T14:41:43.871 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 7.1847014099999997, 2024-09-13T14:41:43.872 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.10965305, 2024-09-13T14:41:43.872 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:43.872 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:43.872 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:43.872 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.872 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:43.872 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.616926+0000", 2024-09-13T14:41:43.872 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.872 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.872 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.872 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:43.872 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.616926+0000", 2024-09-13T14:41:43.872 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.873 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.873 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.873 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:43.873 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.616925+0000", 2024-09-13T14:41:43.873 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:41:43.873 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.873 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.873 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:43.873 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.616933+0000", 2024-09-13T14:41:43.873 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.3170000000000004e-06 2024-09-13T14:41:43.874 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.874 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.874 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:43.874 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.616945+0000", 2024-09-13T14:41:43.874 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1912999999999999e-05 2024-09-13T14:41:43.874 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.874 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.874 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:43.874 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.616949+0000", 2024-09-13T14:41:43.874 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7759999999999997e-06 2024-09-13T14:41:43.874 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.874 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.874 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:43.874 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.655261+0000", 2024-09-13T14:41:43.875 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.038311256000000002 2024-09-13T14:41:43.875 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.875 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.875 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:43.875 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.655284+0000", 2024-09-13T14:41:43.875 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.3666e-05 2024-09-13T14:41:43.875 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.875 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.875 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:43.875 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.726536+0000", 2024-09-13T14:41:43.875 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.071252186999999995 2024-09-13T14:41:43.875 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.875 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.875 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:43.876 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.726579+0000", 2024-09-13T14:41:43.876 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.2648000000000002e-05 2024-09-13T14:41:43.876 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.876 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:43.876 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.876 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.876 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.876 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35289.0:164983 6.b e1105/1103 6:da1931a4:::benchmark_data_smithi151_155836_object32525:head v 1105'10406, mlcod=1087'2070)", 2024-09-13T14:41:43.876 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:36.645059+0000", 2024-09-13T14:41:43.876 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 7.1565686130000001, 2024-09-13T14:41:43.876 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.095231185999999995, 2024-09-13T14:41:43.876 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:43.876 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:43.876 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:43.877 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.877 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:43.877 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.645059+0000", 2024-09-13T14:41:43.877 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.877 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.877 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.877 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:43.877 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.645059+0000", 2024-09-13T14:41:43.877 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.877 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.877 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.877 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:43.877 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.645058+0000", 2024-09-13T14:41:43.877 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:41:43.878 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.878 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.878 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:43.878 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.645070+0000", 2024-09-13T14:41:43.878 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2101e-05 2024-09-13T14:41:43.878 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.878 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.878 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:43.878 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.645081+0000", 2024-09-13T14:41:43.878 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1141999999999999e-05 2024-09-13T14:41:43.878 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.878 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.878 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:43.878 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.645087+0000", 2024-09-13T14:41:43.879 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7100000000000004e-06 2024-09-13T14:41:43.879 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.879 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.879 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:43.879 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.677911+0000", 2024-09-13T14:41:43.879 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.032824187999999997 2024-09-13T14:41:43.879 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.879 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.879 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:43.879 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.678009+0000", 2024-09-13T14:41:43.879 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.7677999999999996e-05 2024-09-13T14:41:43.879 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.879 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.879 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:43.880 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.740194+0000", 2024-09-13T14:41:43.880 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.062185098000000001 2024-09-13T14:41:43.880 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.880 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.880 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:43.880 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.740290+0000", 2024-09-13T14:41:43.880 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.6409999999999993e-05 2024-09-13T14:41:43.880 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.880 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:43.880 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.880 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.880 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.880 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35289.0:165550 6.b e1105/1103 6:d536a3cd:::benchmark_data_smithi151_155836_object33092:head v 1105'10439, mlcod=1087'2104)", 2024-09-13T14:41:43.880 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:37.278479+0000", 2024-09-13T14:41:43.881 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 6.5231480670000002, 2024-09-13T14:41:43.881 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.099543891999999995, 2024-09-13T14:41:43.881 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:43.881 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:43.881 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:43.881 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.881 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:43.881 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.278479+0000", 2024-09-13T14:41:43.881 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.881 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.881 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.881 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:43.881 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.278479+0000", 2024-09-13T14:41:43.882 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.882 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.882 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.882 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:43.882 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.278479+0000", 2024-09-13T14:41:43.882 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:41:43.882 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.882 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.882 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:43.882 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.278484+0000", 2024-09-13T14:41:43.882 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1930000000000001e-06 2024-09-13T14:41:43.882 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.882 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.882 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:43.883 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.278491+0000", 2024-09-13T14:41:43.883 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.3839999999999999e-06 2024-09-13T14:41:43.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.883 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.883 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:43.883 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.278496+0000", 2024-09-13T14:41:43.883 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.023e-06 2024-09-13T14:41:43.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.883 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.883 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:43.883 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.280297+0000", 2024-09-13T14:41:43.883 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001800673 2024-09-13T14:41:43.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.883 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.884 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:43.884 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.280320+0000", 2024-09-13T14:41:43.884 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.2943e-05 2024-09-13T14:41:43.884 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.884 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.884 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:43.884 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.377940+0000", 2024-09-13T14:41:43.884 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.097619528999999997 2024-09-13T14:41:43.884 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.884 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.884 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:43.884 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.378023+0000", 2024-09-13T14:41:43.884 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.3769000000000004e-05 2024-09-13T14:41:43.884 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.885 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:43.885 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.885 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.885 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.885 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35289.0:165594 6.b e1105/1103 6:dbb30615:::benchmark_data_smithi151_155836_object33136:head v 1105'10440, mlcod=1087'2104)", 2024-09-13T14:41:43.885 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:37.283780+0000", 2024-09-13T14:41:43.885 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 6.5178476869999997, 2024-09-13T14:41:43.885 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.094306214999999999, 2024-09-13T14:41:43.885 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:43.885 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:43.885 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:43.885 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.885 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:43.885 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.283780+0000", 2024-09-13T14:41:43.886 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.886 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.886 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.886 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:43.886 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.283780+0000", 2024-09-13T14:41:43.886 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.886 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.886 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.886 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:43.886 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.283779+0000", 2024-09-13T14:41:43.886 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:41:43.886 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.886 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.886 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:43.887 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.283784+0000", 2024-09-13T14:41:43.887 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.33e-06 2024-09-13T14:41:43.887 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.887 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.887 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:43.887 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.283791+0000", 2024-09-13T14:41:43.887 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.2269999999999996e-06 2024-09-13T14:41:43.887 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.887 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.887 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:43.887 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.283795+0000", 2024-09-13T14:41:43.887 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0199999999999996e-06 2024-09-13T14:41:43.887 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.887 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.888 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:43.888 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.292302+0000", 2024-09-13T14:41:43.888 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0085071350000000007 2024-09-13T14:41:43.888 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.888 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.888 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:43.888 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.292324+0000", 2024-09-13T14:41:43.888 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.2402999999999999e-05 2024-09-13T14:41:43.888 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.888 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.888 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:43.888 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.378027+0000", 2024-09-13T14:41:43.888 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.085702865000000003 2024-09-13T14:41:43.888 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.889 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.889 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:43.889 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.378086+0000", 2024-09-13T14:41:43.889 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8650000000000003e-05 2024-09-13T14:41:43.889 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.889 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:43.889 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.889 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.889 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.889 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "MOSDPGPush(6.b 1105/1103 [PushOp(6:dd9c17a0:::benchmark_data_smithi151_155836_object33785:head, version: 1087'2157, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:dd9c17a0:::benchmark_data_smithi151_155836_object33785:head@1087'2157, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:41:43.889 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:38.100986+0000", 2024-09-13T14:41:43.889 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 5.7006417540000003, 2024-09-13T14:41:43.889 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.27630356299999997, 2024-09-13T14:41:43.889 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:43.890 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:41:43.890 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:43.890 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.890 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:43.890 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.100986+0000", 2024-09-13T14:41:43.890 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.890 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.890 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.890 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:43.890 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.100986+0000", 2024-09-13T14:41:43.890 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.890 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.890 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.890 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:43.891 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.100985+0000", 2024-09-13T14:41:43.891 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:41:43.891 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.891 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.891 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:43.891 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.101020+0000", 2024-09-13T14:41:43.891 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.4418000000000001e-05 2024-09-13T14:41:43.891 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.891 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.891 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:43.891 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.101033+0000", 2024-09-13T14:41:43.891 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2979000000000001e-05 2024-09-13T14:41:43.891 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.891 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.892 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:43.892 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.101040+0000", 2024-09-13T14:41:43.892 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.6699999999999994e-06 2024-09-13T14:41:43.892 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.892 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.892 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:43.892 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.372854+0000", 2024-09-13T14:41:43.892 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.27181395800000002 2024-09-13T14:41:43.892 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.892 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.892 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:43.892 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.372935+0000", 2024-09-13T14:41:43.892 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0539999999999998e-05 2024-09-13T14:41:43.892 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.893 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.893 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:43.893 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.377289+0000", 2024-09-13T14:41:43.893 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0043543879999999998 2024-09-13T14:41:43.893 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.893 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:43.893 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.893 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.893 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.893 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "MOSDPGPush(6.b 1105/1103 [PushOp(6:d3097778:::benchmark_data_smithi151_155836_object34146:head, version: 1087'2185, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:d3097778:::benchmark_data_smithi151_155836_object34146:head@1087'2185, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:41:43.893 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:38.592578+0000", 2024-09-13T14:41:43.893 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 5.2090493059999998, 2024-09-13T14:41:43.893 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.24278056200000001, 2024-09-13T14:41:43.894 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:43.894 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:41:43.894 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:43.894 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.894 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:43.894 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.592578+0000", 2024-09-13T14:41:43.894 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.894 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.894 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.894 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:43.894 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.592578+0000", 2024-09-13T14:41:43.894 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.894 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.894 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.894 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:43.895 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.592578+0000", 2024-09-13T14:41:43.895 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:41:43.895 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.895 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.895 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:43.895 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.592625+0000", 2024-09-13T14:41:43.895 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7812999999999998e-05 2024-09-13T14:41:43.895 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.895 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.895 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:43.895 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.592639+0000", 2024-09-13T14:41:43.895 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3390999999999999e-05 2024-09-13T14:41:43.895 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.895 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.896 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:43.896 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.592648+0000", 2024-09-13T14:41:43.896 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.0450000000000006e-06 2024-09-13T14:41:43.896 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.896 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.896 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:43.896 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.829466+0000", 2024-09-13T14:41:43.896 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.23681851300000001 2024-09-13T14:41:43.896 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.896 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.897 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:43.897 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.829473+0000", 2024-09-13T14:41:43.897 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7469999999999999e-06 2024-09-13T14:41:43.897 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.897 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.897 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:43.897 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.835359+0000", 2024-09-13T14:41:43.897 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0058855770000000003 2024-09-13T14:41:43.897 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.897 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:43.897 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.897 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.897 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.897 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "MOSDPGPush(6.b 1106/1103 [PushOp(6:da02d985:::benchmark_data_smithi151_155836_object37651:head, version: 1087'2399, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:da02d985:::benchmark_data_smithi151_155836_object37651:head@1087'2399, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:41:43.898 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:43.168061+0000", 2024-09-13T14:41:43.898 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.63356665899999998, 2024-09-13T14:41:43.898 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.124861429, 2024-09-13T14:41:43.898 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:43.898 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:41:43.898 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:43.898 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.898 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:43.898 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:43.168061+0000", 2024-09-13T14:41:43.898 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.898 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.898 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.898 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:43.898 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:43.168061+0000", 2024-09-13T14:41:43.899 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:43.899 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.899 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.899 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:43.899 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:43.168060+0000", 2024-09-13T14:41:43.899 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:41:43.899 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.899 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.899 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:43.899 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:43.168108+0000", 2024-09-13T14:41:43.899 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7682999999999997e-05 2024-09-13T14:41:43.899 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.899 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.899 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:43.900 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:43.168125+0000", 2024-09-13T14:41:43.900 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.7207e-05 2024-09-13T14:41:43.900 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.900 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.900 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:43.900 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:43.168134+0000", 2024-09-13T14:41:43.900 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.2240000000000001e-06 2024-09-13T14:41:43.900 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.900 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.900 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:43.900 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:43.288556+0000", 2024-09-13T14:41:43.900 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.120421825 2024-09-13T14:41:43.900 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.901 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.901 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:43.901 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:43.288563+0000", 2024-09-13T14:41:43.901 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.8639999999999998e-06 2024-09-13T14:41:43.901 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:43.901 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:43.901 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:43.901 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:43.292922+0000", 2024-09-13T14:41:43.901 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0043593759999999999 2024-09-13T14:41:43.901 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.901 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:43.901 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.901 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:43.901 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:43.902 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:41:43.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:43.902 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:43.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:43.903 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:41:43.903 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:41:44.004 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:44.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:44.048 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:44.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:44.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:43 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/4228042015' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:44.150 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:44.174 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:44.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:44.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:44.276 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:44.301 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:44.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:44.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:44.403 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:44.426 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:44.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:44.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:44.528 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:44.573 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:44.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:44.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:44.675 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:44.701 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:44.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:44.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:44.749 INFO:teuthology.orchestra.run.smithi151.stderr:set pool 6 pg_num to 26 2024-09-13T14:41:44.801 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T14:41:44.803 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:44.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:41:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:41:44] "GET /metrics HTTP/1.1" 200 36666 "" "Prometheus/2.51.0" 2024-09-13T14:41:44.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:44 smithi151 ceph-mon[108966]: pgmap v1381: 99 pgs: 2 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 755 op/s; 335/288171 objects degraded (0.116%); 27172/288171 objects misplaced (9.429%); 15 MiB/s, 246 objects/s recovering 2024-09-13T14:41:44.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:44 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/499920381' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T14:41:44.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:44 smithi151 ceph-mon[104450]: pgmap v1381: 99 pgs: 2 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 755 op/s; 335/288171 objects degraded (0.116%); 27172/288171 objects misplaced (9.429%); 15 MiB/s, 246 objects/s recovering 2024-09-13T14:41:44.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:44 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/499920381' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T14:41:44.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:44.891 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:44.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:44.992 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:45.000 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:41:45.018 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:45.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:45.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:44 smithi184 ceph-mon[92527]: pgmap v1381: 99 pgs: 2 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 755 op/s; 335/288171 objects degraded (0.116%); 27172/288171 objects misplaced (9.429%); 15 MiB/s, 246 objects/s recovering 2024-09-13T14:41:45.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:44 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/499920381' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T14:41:45.120 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:45.152 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:45.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:45.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:45.253 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:41:45.254 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:45.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:45.315 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:45.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:45.416 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:45.455 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:45.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:45.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:45.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:45.582 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:45.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:45.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:45.684 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:45.740 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:45.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:45.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:45.841 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:45.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:45.885 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:45.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:45.986 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:46.016 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:45 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/499920381' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-09-13T14:41:46.016 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:45 smithi184 ceph-mon[92527]: osdmap e1107: 8 total, 8 up, 7 in 2024-09-13T14:41:46.016 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:45 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-13T14:41:46.040 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:46.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:46.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:46.089 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:45 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/499920381' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-09-13T14:41:46.089 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:45 smithi151 ceph-mon[104450]: osdmap e1107: 8 total, 8 up, 7 in 2024-09-13T14:41:46.089 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:45 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-13T14:41:46.089 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:45 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/499920381' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-09-13T14:41:46.089 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:45 smithi151 ceph-mon[108966]: osdmap e1107: 8 total, 8 up, 7 in 2024-09-13T14:41:46.089 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:45 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-13T14:41:46.142 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:46.171 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:46.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:46.189 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:46.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:46.290 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:46.313 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:46.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:46.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:46.415 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:46.437 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:46.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:46.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:46.539 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:46.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:41:46.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:46.562 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:46.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:46.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:46.687 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:46.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:46.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:46.789 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:46.839 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:46.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:46.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:46.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:46 smithi151 ceph-mon[108966]: pgmap v1383: 99 pgs: 2 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 843 op/s; 477/285948 objects degraded (0.167%); 26940/285948 objects misplaced (9.421%); 14 MiB/s, 231 objects/s recovering 2024-09-13T14:41:46.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-13T14:41:46.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:46 smithi151 ceph-mon[108966]: osdmap e1108: 8 total, 8 up, 7 in 2024-09-13T14:41:46.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:41:46.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:46 smithi151 ceph-mon[104450]: pgmap v1383: 99 pgs: 2 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 843 op/s; 477/285948 objects degraded (0.167%); 26940/285948 objects misplaced (9.421%); 14 MiB/s, 231 objects/s recovering 2024-09-13T14:41:46.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-13T14:41:46.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:46 smithi151 ceph-mon[104450]: osdmap e1108: 8 total, 8 up, 7 in 2024-09-13T14:41:46.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:41:46.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:46.941 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:46.962 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:46.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:46.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:47.021 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:46 smithi184 ceph-mon[92527]: pgmap v1383: 99 pgs: 2 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 843 op/s; 477/285948 objects degraded (0.167%); 26940/285948 objects misplaced (9.421%); 14 MiB/s, 231 objects/s recovering 2024-09-13T14:41:47.021 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-13T14:41:47.021 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:46 smithi184 ceph-mon[92527]: osdmap e1108: 8 total, 8 up, 7 in 2024-09-13T14:41:47.021 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:41:47.021 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:47.064 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:47.091 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:47.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:47.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:47.193 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:47.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:47.222 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:47.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:47.323 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:47.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:47.346 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:47.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:47.441 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:41:47.441 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:41:47.447 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:47.469 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:47.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:47.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:47.571 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:47.593 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:47.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:47.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:47.695 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:47.716 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:47.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:47.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:47.818 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:47.842 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:47.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:47.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:47.944 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:47.966 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:47.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:47.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:48.068 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:48.090 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:48.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:48.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:48.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:47 smithi184 ceph-mon[92527]: osdmap e1109: 8 total, 8 up, 7 in 2024-09-13T14:41:48.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:47 smithi184 ceph-mon[92527]: 6.1 starting backfill to osd.7 from (1090'5198,1107'10579] MIN to 1107'10580 2024-09-13T14:41:48.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:47 smithi184 ceph-mon[92527]: 6.4 continuing backfill to osd.4 from (1088'4255,1107'10970] MIN to 1107'10970 2024-09-13T14:41:48.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:47 smithi184 ceph-mon[92527]: 6.0 scrub starts 2024-09-13T14:41:48.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:47 smithi151 ceph-mon[104450]: osdmap e1109: 8 total, 8 up, 7 in 2024-09-13T14:41:48.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:47 smithi151 ceph-mon[104450]: 6.1 starting backfill to osd.7 from (1090'5198,1107'10579] MIN to 1107'10580 2024-09-13T14:41:48.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:47 smithi151 ceph-mon[104450]: 6.4 continuing backfill to osd.4 from (1088'4255,1107'10970] MIN to 1107'10970 2024-09-13T14:41:48.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:47 smithi151 ceph-mon[104450]: 6.0 scrub starts 2024-09-13T14:41:48.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:47 smithi151 ceph-mon[108966]: osdmap e1109: 8 total, 8 up, 7 in 2024-09-13T14:41:48.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:47 smithi151 ceph-mon[108966]: 6.1 starting backfill to osd.7 from (1090'5198,1107'10579] MIN to 1107'10580 2024-09-13T14:41:48.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:47 smithi151 ceph-mon[108966]: 6.4 continuing backfill to osd.4 from (1088'4255,1107'10970] MIN to 1107'10970 2024-09-13T14:41:48.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:47 smithi151 ceph-mon[108966]: 6.0 scrub starts 2024-09-13T14:41:48.157 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T14:41:48.193 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:48.222 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:48.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:48.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:48.324 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:48.348 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:48.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:48.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:48.450 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:48.473 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:48.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:48.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:48.481 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:41:48.575 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:48.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:48.599 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:48.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:48.700 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:48.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:48.723 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:48.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:48.824 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:48.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:48.852 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:48.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:48.926 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:48 smithi184 ceph-mon[92527]: pgmap v1386: 109 pgs: 1 active+recovery_wait, 8 unknown, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 95 active+clean; 5.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 733 op/s; 477/283566 objects degraded (0.168%); 26088/283566 objects misplaced (9.200%); 19 MiB/s, 299 objects/s recovering 2024-09-13T14:41:48.926 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:48 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 477/283566 objects degraded (0.168%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:41:48.926 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:48 smithi184 ceph-mon[92527]: osdmap e1110: 8 total, 8 up, 7 in 2024-09-13T14:41:48.926 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:48 smithi184 ceph-mon[92527]: 6.10 scrub starts 2024-09-13T14:41:48.954 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:48.984 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:48.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:48.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:49.086 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:49.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:48 smithi151 ceph-mon[104450]: pgmap v1386: 109 pgs: 1 active+recovery_wait, 8 unknown, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 95 active+clean; 5.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 733 op/s; 477/283566 objects degraded (0.168%); 26088/283566 objects misplaced (9.200%); 19 MiB/s, 299 objects/s recovering 2024-09-13T14:41:49.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:48 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 477/283566 objects degraded (0.168%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:41:49.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:48 smithi151 ceph-mon[104450]: osdmap e1110: 8 total, 8 up, 7 in 2024-09-13T14:41:49.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:48 smithi151 ceph-mon[104450]: 6.10 scrub starts 2024-09-13T14:41:49.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:48 smithi151 ceph-mon[108966]: pgmap v1386: 109 pgs: 1 active+recovery_wait, 8 unknown, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 95 active+clean; 5.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 733 op/s; 477/283566 objects degraded (0.168%); 26088/283566 objects misplaced (9.200%); 19 MiB/s, 299 objects/s recovering 2024-09-13T14:41:49.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:48 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 477/283566 objects degraded (0.168%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:41:49.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:48 smithi151 ceph-mon[108966]: osdmap e1110: 8 total, 8 up, 7 in 2024-09-13T14:41:49.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:48 smithi151 ceph-mon[108966]: 6.10 scrub starts 2024-09-13T14:41:49.115 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:49.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:49.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:49.217 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:49.253 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:49.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:49.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:49.356 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:49.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:49.391 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:49.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:49.492 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:49.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:49.520 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:49.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:49.621 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:49.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:49.650 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:49.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:49.751 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:49.793 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:49.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:49.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:49.895 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:49.922 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:49.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:49.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:50.024 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:50.059 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:50.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:50.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:50.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:49 smithi184 ceph-mon[92527]: osdmap e1111: 8 total, 8 up, 7 in 2024-09-13T14:41:50.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:49 smithi184 ceph-mon[92527]: 6.14 continuing backfill to osd.4 from (1088'4255,1107'10970] MIN to 1107'10970 2024-09-13T14:41:50.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:49 smithi184 ceph-mon[92527]: 6.11 starting backfill to osd.7 from (1090'5198,1107'10579] MIN to 1107'10580 2024-09-13T14:41:50.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:49 smithi184 ceph-mon[92527]: 6.19 scrub starts 2024-09-13T14:41:50.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:49 smithi151 ceph-mon[104450]: osdmap e1111: 8 total, 8 up, 7 in 2024-09-13T14:41:50.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:49 smithi151 ceph-mon[104450]: 6.14 continuing backfill to osd.4 from (1088'4255,1107'10970] MIN to 1107'10970 2024-09-13T14:41:50.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:49 smithi151 ceph-mon[104450]: 6.11 starting backfill to osd.7 from (1090'5198,1107'10579] MIN to 1107'10580 2024-09-13T14:41:50.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:49 smithi151 ceph-mon[104450]: 6.19 scrub starts 2024-09-13T14:41:50.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:49 smithi151 ceph-mon[108966]: osdmap e1111: 8 total, 8 up, 7 in 2024-09-13T14:41:50.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:49 smithi151 ceph-mon[108966]: 6.14 continuing backfill to osd.4 from (1088'4255,1107'10970] MIN to 1107'10970 2024-09-13T14:41:50.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:49 smithi151 ceph-mon[108966]: 6.11 starting backfill to osd.7 from (1090'5198,1107'10579] MIN to 1107'10580 2024-09-13T14:41:50.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:49 smithi151 ceph-mon[108966]: 6.19 scrub starts 2024-09-13T14:41:50.161 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:50.194 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:50.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:50.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:50.296 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:50.322 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:50.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:50.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:50.385 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:41:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:41:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:41:50.424 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:50.443 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:41:50.443 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:41:50.465 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:50.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:50.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:50.567 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:50.606 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:50.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:50.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:50.708 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:50.737 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:50.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:50.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:50.839 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:50.882 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:50.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:50.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:50.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:51.012 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:51.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:51.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:51.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:50 smithi184 ceph-mon[92527]: pgmap v1389: 109 pgs: 2 peering, 1 active+remapped+backfilling, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 100 active+clean; 5.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 664 op/s; 556/277974 objects degraded (0.200%); 18853/277974 objects misplaced (6.782%); 24 MiB/s, 387 objects/s recovering 2024-09-13T14:41:51.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:50 smithi151 ceph-mon[104450]: pgmap v1389: 109 pgs: 2 peering, 1 active+remapped+backfilling, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 100 active+clean; 5.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 664 op/s; 556/277974 objects degraded (0.200%); 18853/277974 objects misplaced (6.782%); 24 MiB/s, 387 objects/s recovering 2024-09-13T14:41:51.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:50 smithi151 ceph-mon[108966]: pgmap v1389: 109 pgs: 2 peering, 1 active+remapped+backfilling, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 100 active+clean; 5.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 664 op/s; 556/277974 objects degraded (0.200%); 18853/277974 objects misplaced (6.782%); 24 MiB/s, 387 objects/s recovering 2024-09-13T14:41:51.114 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:51.118 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:41:51.118 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:41:51.118 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-13T14:41:51.118 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:41:51.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:51.158 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:51.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:51.243 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:51.258 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:51.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:51.290 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:51.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:51.391 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:51.421 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:51.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:51.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:51.524 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:51.558 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:51.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:51.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:51.660 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:51.709 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T14:41:51.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:51.712 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:51.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:51.813 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:51.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:51.855 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:51.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:51.956 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:51.982 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:51.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:51.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:52.084 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:52.119 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:52.120 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:41:52.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:52.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:52.221 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:52.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:52.245 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:52.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:52.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:52.373 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:52.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:52.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:52.476 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:52.497 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:52.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:52.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:52.544 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:52.599 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:52.625 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:52.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:52.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:52.730 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:52.754 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:52.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:52.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:52.820 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-13T14:41:52.856 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:52.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:52.886 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:52.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:52.984 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:52 smithi151 ceph-mon[104450]: pgmap v1390: 109 pgs: 2 peering, 1 active+remapped+backfilling, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 100 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 457 op/s; 556/277527 objects degraded (0.200%); 18853/277527 objects misplaced (6.793%); 17 MiB/s, 277 objects/s recovering 2024-09-13T14:41:52.984 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:52 smithi151 ceph-mon[104450]: osdmap e1112: 8 total, 8 up, 7 in 2024-09-13T14:41:52.984 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:52 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3597337483' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:52.984 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:52 smithi151 ceph-mon[108966]: pgmap v1390: 109 pgs: 2 peering, 1 active+remapped+backfilling, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 100 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 457 op/s; 556/277527 objects degraded (0.200%); 18853/277527 objects misplaced (6.793%); 17 MiB/s, 277 objects/s recovering 2024-09-13T14:41:52.985 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:52 smithi151 ceph-mon[108966]: osdmap e1112: 8 total, 8 up, 7 in 2024-09-13T14:41:52.985 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:52 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3597337483' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:52.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:53.023 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:53.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:53.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:53.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:52 smithi184 ceph-mon[92527]: pgmap v1390: 109 pgs: 2 peering, 1 active+remapped+backfilling, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 100 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 457 op/s; 556/277527 objects degraded (0.200%); 18853/277527 objects misplaced (6.793%); 17 MiB/s, 277 objects/s recovering 2024-09-13T14:41:53.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:52 smithi184 ceph-mon[92527]: osdmap e1112: 8 total, 8 up, 7 in 2024-09-13T14:41:53.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:52 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3597337483' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:53.125 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:53.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:53.153 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:53.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:53.254 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:53.281 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:53.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:53.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:53.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:53.407 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:53.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:53.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:53.509 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:53.534 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:53.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:53.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:53.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:53.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:53.663 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:53.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:53.764 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:53.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:53.793 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:53.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:53.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:53 smithi151 ceph-mon[104450]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:53.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:53 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3597337483' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:41:53.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:53 smithi151 ceph-mon[104450]: osdmap e1113: 8 total, 8 up, 7 in 2024-09-13T14:41:53.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:53 smithi151 ceph-mon[104450]: 6.0 scrub starts 2024-09-13T14:41:53.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:53 smithi151 ceph-mon[108966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:53.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:53 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3597337483' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:41:53.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:53 smithi151 ceph-mon[108966]: osdmap e1113: 8 total, 8 up, 7 in 2024-09-13T14:41:53.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:53 smithi151 ceph-mon[108966]: 6.0 scrub starts 2024-09-13T14:41:53.895 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:53.923 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:53.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:53.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:54.024 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:54.050 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:54.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:54.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:54.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:53 smithi184 ceph-mon[92527]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:54.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:53 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3597337483' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:41:54.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:53 smithi184 ceph-mon[92527]: osdmap e1113: 8 total, 8 up, 7 in 2024-09-13T14:41:54.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:53 smithi184 ceph-mon[92527]: 6.0 scrub starts 2024-09-13T14:41:54.152 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:54.187 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:54.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:54.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:54.290 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:54.315 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:54.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:54.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:54.420 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:54.444 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:54.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:54.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:54.546 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:54.575 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:54.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:54.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:54.599 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:41:54.599 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:41:54.619 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:41:54.619 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:41:54.643 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:41:54.644 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:41:54.645 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:41:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:41:54] "GET /metrics HTTP/1.1" 200 36683 "" "Prometheus/2.51.0" 2024-09-13T14:41:54.665 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:41:54.665 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:41:54.677 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:54.695 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:41:54.696 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:41:54.710 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:54.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:54.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:54.736 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:41:54.736 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:41:54.761 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:41:54.761 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:41:54.785 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:41:54.785 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:41:54.813 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:54.844 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:54.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:54.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:54.897 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:54 smithi184 ceph-mon[92527]: pgmap v1393: 109 pgs: 1 active+recovery_wait+remapped, 1 peering, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 100 active+clean; 5.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 593 op/s; 556/269130 objects degraded (0.207%); 21991/269130 objects misplaced (8.171%); 15 MiB/s, 241 objects/s recovering 2024-09-13T14:41:54.897 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:54 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 556/269130 objects degraded (0.207%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:41:54.897 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:54 smithi184 ceph-mon[92527]: 6.10 scrub starts 2024-09-13T14:41:54.898 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:54 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2036402094' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:54.945 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:41:54.945 INFO:teuthology.orchestra.run.smithi184.stdout: "size": 20, 2024-09-13T14:41:54.945 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 600, 2024-09-13T14:41:54.946 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [ 2024-09-13T14:41:54.946 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.946 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:139590 6.3 6:c9d826a6:::benchmark_data_smithi151_155836_object7132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1094)", 2024-09-13T14:41:54.946 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:05.134021+0000", 2024-09-13T14:41:54.946 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 49.806305090000002, 2024-09-13T14:41:54.946 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.094063110000000005, 2024-09-13T14:41:54.946 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:54.946 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:54.946 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:54.946 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:54.946 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:54.946 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 139590 2024-09-13T14:41:54.947 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.947 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:54.947 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.947 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:54.947 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:05.134021+0000", 2024-09-13T14:41:54.947 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:54.947 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.947 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.947 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:54.947 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:05.134021+0000", 2024-09-13T14:41:54.947 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:54.947 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.948 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.948 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:54.948 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:05.134020+0000", 2024-09-13T14:41:54.948 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:41:54.948 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.948 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.948 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:54.948 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:05.134030+0000", 2024-09-13T14:41:54.948 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.4809999999999995e-06 2024-09-13T14:41:54.948 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.949 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.949 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:54.949 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:05.134041+0000", 2024-09-13T14:41:54.949 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0682e-05 2024-09-13T14:41:54.949 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.949 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.949 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:54.949 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:05.134047+0000", 2024-09-13T14:41:54.949 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.438e-06 2024-09-13T14:41:54.949 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.949 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.949 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:54.950 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:05.197061+0000", 2024-09-13T14:41:54.950 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.063014068000000006 2024-09-13T14:41:54.950 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.950 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.950 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:54.950 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:05.197156+0000", 2024-09-13T14:41:54.950 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.5210000000000005e-05 2024-09-13T14:41:54.950 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.950 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.950 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:41:54.950 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:05.197254+0000", 2024-09-13T14:41:54.950 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.7244999999999995e-05 2024-09-13T14:41:54.950 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.951 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.951 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:54.951 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:05.205437+0000", 2024-09-13T14:41:54.951 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0081835710000000006 2024-09-13T14:41:54.951 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.951 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.951 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:54.951 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:05.209782+0000", 2024-09-13T14:41:54.952 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0043448970000000003 2024-09-13T14:41:54.952 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.952 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.952 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:54.952 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:05.228002+0000", 2024-09-13T14:41:54.952 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.018219985000000001 2024-09-13T14:41:54.952 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.952 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.952 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:54.952 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:05.228046+0000", 2024-09-13T14:41:54.952 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4073000000000003e-05 2024-09-13T14:41:54.952 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.952 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.953 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:54.953 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:05.228085+0000", 2024-09-13T14:41:54.953 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8386999999999997e-05 2024-09-13T14:41:54.953 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:54.953 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:54.953 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:54.953 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.953 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.953 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:162688 6.3 6:cb6a1afe:::benchmark_data_smithi151_155836_object30230:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1105)", 2024-09-13T14:41:54.953 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:33.539299+0000", 2024-09-13T14:41:54.953 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 21.401027849999998, 2024-09-13T14:41:54.953 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.083987104000000007, 2024-09-13T14:41:54.953 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:54.954 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:54.954 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:54.954 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:54.954 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:54.954 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 162688 2024-09-13T14:41:54.954 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.954 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:54.954 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.954 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:54.954 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.539299+0000", 2024-09-13T14:41:54.954 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:54.954 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.954 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.955 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:54.955 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.539299+0000", 2024-09-13T14:41:54.955 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:54.955 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.955 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.955 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:54.955 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.539297+0000", 2024-09-13T14:41:54.955 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:54.955 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.955 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.955 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:54.956 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.539304+0000", 2024-09-13T14:41:54.956 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.391e-06 2024-09-13T14:41:54.956 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.956 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.956 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:54.956 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.539320+0000", 2024-09-13T14:41:54.956 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.5858999999999998e-05 2024-09-13T14:41:54.956 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.956 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.956 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:54.956 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.539327+0000", 2024-09-13T14:41:54.957 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.4350000000000002e-06 2024-09-13T14:41:54.957 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.957 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.957 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:54.957 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.539375+0000", 2024-09-13T14:41:54.957 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8522000000000001e-05 2024-09-13T14:41:54.957 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.957 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.957 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:54.957 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.539485+0000", 2024-09-13T14:41:54.957 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00011014399999999999 2024-09-13T14:41:54.958 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.958 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.958 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:41:54.958 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.539613+0000", 2024-09-13T14:41:54.958 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000127887 2024-09-13T14:41:54.958 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.958 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.958 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:54.958 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.546244+0000", 2024-09-13T14:41:54.958 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0066306189999999999 2024-09-13T14:41:54.958 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.958 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.958 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:54.959 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.552234+0000", 2024-09-13T14:41:54.959 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0059899669999999997 2024-09-13T14:41:54.959 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.959 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.959 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:54.959 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.623144+0000", 2024-09-13T14:41:54.959 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.070910528 2024-09-13T14:41:54.959 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.960 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.960 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:54.960 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.623238+0000", 2024-09-13T14:41:54.960 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.3851000000000007e-05 2024-09-13T14:41:54.960 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.960 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.960 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:54.960 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:33.623286+0000", 2024-09-13T14:41:54.960 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7537000000000002e-05 2024-09-13T14:41:54.960 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:54.961 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:54.961 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:54.961 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.961 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.961 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:164008 6.3 6:c705e8a2:::benchmark_data_smithi151_155836_object31550:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1105)", 2024-09-13T14:41:54.961 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:35.175185+0000", 2024-09-13T14:41:54.961 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 19.765141641, 2024-09-13T14:41:54.962 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.082319325999999998, 2024-09-13T14:41:54.962 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:54.962 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:54.962 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:54.962 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:54.962 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:54.962 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 164008 2024-09-13T14:41:54.963 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.963 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:54.963 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.963 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:54.963 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.175185+0000", 2024-09-13T14:41:54.963 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:54.963 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.964 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.964 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:54.964 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.175185+0000", 2024-09-13T14:41:54.964 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:54.964 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.964 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.964 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:54.964 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.175183+0000", 2024-09-13T14:41:54.964 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:54.964 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.965 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.965 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:54.965 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.175191+0000", 2024-09-13T14:41:54.965 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.7079999999999997e-06 2024-09-13T14:41:54.965 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.965 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.965 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:54.965 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.176134+0000", 2024-09-13T14:41:54.965 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00094342299999999998 2024-09-13T14:41:54.965 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.965 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.965 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:54.965 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.176144+0000", 2024-09-13T14:41:54.966 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.499e-06 2024-09-13T14:41:54.966 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.966 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.966 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:54.966 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.184678+0000", 2024-09-13T14:41:54.966 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.008533723 2024-09-13T14:41:54.966 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.966 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.966 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:54.966 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.185264+0000", 2024-09-13T14:41:54.967 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00058679200000000004 2024-09-13T14:41:54.967 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.967 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.967 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:41:54.967 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.185404+0000", 2024-09-13T14:41:54.967 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013951 2024-09-13T14:41:54.967 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.967 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.967 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:54.967 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.192197+0000", 2024-09-13T14:41:54.967 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0067927789999999997 2024-09-13T14:41:54.967 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.967 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.968 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:54.968 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.193215+0000", 2024-09-13T14:41:54.968 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001018424 2024-09-13T14:41:54.968 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.968 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.968 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:54.968 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.257237+0000", 2024-09-13T14:41:54.968 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.064021585000000006 2024-09-13T14:41:54.968 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.968 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.968 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:54.968 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.257380+0000", 2024-09-13T14:41:54.969 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000143523 2024-09-13T14:41:54.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.969 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.969 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:54.969 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.257504+0000", 2024-09-13T14:41:54.969 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012399000000000001 2024-09-13T14:41:54.969 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:54.969 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:54.969 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:54.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.969 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.970 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:164538 6.3 6:c366f206:::benchmark_data_smithi151_155836_object32080:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1105)", 2024-09-13T14:41:54.970 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:35.892411+0000", 2024-09-13T14:41:54.970 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 19.047916004000001, 2024-09-13T14:41:54.970 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.080446633000000003, 2024-09-13T14:41:54.970 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:54.970 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:54.970 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:54.970 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:54.970 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:54.970 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 164538 2024-09-13T14:41:54.970 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.970 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:54.970 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.970 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:54.971 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.892411+0000", 2024-09-13T14:41:54.971 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:54.971 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.971 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.971 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:54.971 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.892411+0000", 2024-09-13T14:41:54.971 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:54.971 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.971 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.971 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:54.971 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.892409+0000", 2024-09-13T14:41:54.971 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:41:54.971 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.972 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.972 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:54.972 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.892418+0000", 2024-09-13T14:41:54.972 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.081e-06 2024-09-13T14:41:54.972 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.972 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.972 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:54.972 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.892437+0000", 2024-09-13T14:41:54.972 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.8814999999999999e-05 2024-09-13T14:41:54.973 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.973 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.973 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:54.973 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.892444+0000", 2024-09-13T14:41:54.973 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.8709999999999997e-06 2024-09-13T14:41:54.973 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.973 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.973 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:54.973 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.892570+0000", 2024-09-13T14:41:54.973 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012602100000000001 2024-09-13T14:41:54.974 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.974 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.974 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:54.974 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.892851+0000", 2024-09-13T14:41:54.974 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00028031099999999999 2024-09-13T14:41:54.974 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.974 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.974 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:41:54.974 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.893186+0000", 2024-09-13T14:41:54.974 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00033516399999999999 2024-09-13T14:41:54.974 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.974 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.974 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:54.974 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.901229+0000", 2024-09-13T14:41:54.975 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0080433109999999992 2024-09-13T14:41:54.975 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.975 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.975 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:54.975 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.901698+0000", 2024-09-13T14:41:54.975 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00046840299999999999 2024-09-13T14:41:54.975 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.975 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.975 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:54.975 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.972575+0000", 2024-09-13T14:41:54.975 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.070877672000000003 2024-09-13T14:41:54.975 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.975 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.976 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:54.976 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.972742+0000", 2024-09-13T14:41:54.976 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00016672099999999999 2024-09-13T14:41:54.976 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.976 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.976 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:54.976 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:35.972857+0000", 2024-09-13T14:41:54.976 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00011513100000000001 2024-09-13T14:41:54.976 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:54.977 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:54.977 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:54.977 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.977 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.977 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:164669 6.3 6:c3a15dfe:::benchmark_data_smithi151_155836_object32211:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1105)", 2024-09-13T14:41:54.977 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:36.062789+0000", 2024-09-13T14:41:54.977 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 18.877537203999999, 2024-09-13T14:41:54.977 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.090380663, 2024-09-13T14:41:54.977 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:54.977 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:54.977 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:54.978 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:54.978 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:54.978 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 164669 2024-09-13T14:41:54.978 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.978 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:54.978 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.978 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:54.978 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.062789+0000", 2024-09-13T14:41:54.978 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:54.978 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.978 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.979 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:54.979 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.062789+0000", 2024-09-13T14:41:54.979 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:54.979 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.979 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.979 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:54.979 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.062788+0000", 2024-09-13T14:41:54.979 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:41:54.979 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.979 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.979 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:54.980 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.062797+0000", 2024-09-13T14:41:54.980 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.3840000000000003e-06 2024-09-13T14:41:54.980 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.980 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:54.981 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.982 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:54.982 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.062813+0000", 2024-09-13T14:41:54.982 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6379999999999999e-05 2024-09-13T14:41:54.982 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.982 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.982 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:54.982 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.062819+0000", 2024-09-13T14:41:54.982 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8250000000000001e-06 2024-09-13T14:41:54.982 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.982 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.982 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:54.982 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.062957+0000", 2024-09-13T14:41:54.982 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013797200000000001 2024-09-13T14:41:54.983 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.983 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.983 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:54.983 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.063166+0000", 2024-09-13T14:41:54.983 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00020847199999999999 2024-09-13T14:41:54.983 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.983 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.983 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:41:54.983 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.063316+0000", 2024-09-13T14:41:54.983 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00015000699999999999 2024-09-13T14:41:54.983 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.983 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.983 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:54.983 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.070467+0000", 2024-09-13T14:41:54.984 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0071513610000000002 2024-09-13T14:41:54.984 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.984 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.984 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:54.984 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.077121+0000", 2024-09-13T14:41:54.984 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0066544280000000004 2024-09-13T14:41:54.984 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.984 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.984 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:54.984 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.153047+0000", 2024-09-13T14:41:54.984 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.075925961 2024-09-13T14:41:54.985 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.985 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.985 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:54.985 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.153130+0000", 2024-09-13T14:41:54.985 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.3015000000000004e-05 2024-09-13T14:41:54.985 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.985 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.985 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:54.985 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.153170+0000", 2024-09-13T14:41:54.985 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9601000000000003e-05 2024-09-13T14:41:54.986 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:54.986 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:54.986 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:54.986 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.986 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.986 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:164670 6.3 6:c51a8182:::benchmark_data_smithi151_155836_object32212:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1105)", 2024-09-13T14:41:54.987 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:36.063482+0000", 2024-09-13T14:41:54.987 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 18.876844260999999, 2024-09-13T14:41:54.987 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.089833642000000005, 2024-09-13T14:41:54.987 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:54.987 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:54.987 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:54.988 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:54.988 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:54.988 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 164670 2024-09-13T14:41:54.988 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.988 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:54.988 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.989 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:54.989 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.063482+0000", 2024-09-13T14:41:54.989 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:54.989 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.989 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.989 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:54.990 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.063482+0000", 2024-09-13T14:41:54.990 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:54.990 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.990 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.990 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:54.990 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.063481+0000", 2024-09-13T14:41:54.991 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:41:54.991 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.991 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.991 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:54.991 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.063486+0000", 2024-09-13T14:41:54.991 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0100000000000003e-06 2024-09-13T14:41:54.992 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.992 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.992 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:54.992 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.063499+0000", 2024-09-13T14:41:54.992 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2619000000000001e-05 2024-09-13T14:41:54.992 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.992 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.993 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:54.993 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.063505+0000", 2024-09-13T14:41:54.993 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1990000000000003e-06 2024-09-13T14:41:54.993 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.993 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.994 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:54.994 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.070564+0000", 2024-09-13T14:41:54.994 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00705873 2024-09-13T14:41:54.994 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.995 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.995 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:54.995 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.070838+0000", 2024-09-13T14:41:54.995 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00027366899999999998 2024-09-13T14:41:54.995 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.995 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.996 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:41:54.996 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.071185+0000", 2024-09-13T14:41:54.996 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00034749100000000001 2024-09-13T14:41:54.996 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.996 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.996 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:54.997 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.077224+0000", 2024-09-13T14:41:54.997 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0060385830000000001 2024-09-13T14:41:54.997 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.997 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.997 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:54.997 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.078333+0000", 2024-09-13T14:41:54.998 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0011092820000000001 2024-09-13T14:41:54.998 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.998 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.998 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:54.998 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.153206+0000", 2024-09-13T14:41:54.998 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.074872981000000005 2024-09-13T14:41:54.999 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:54.999 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:54.999 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:54.999 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.153282+0000", 2024-09-13T14:41:54.999 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.5866999999999997e-05 2024-09-13T14:41:54.999 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.000 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.000 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:55.000 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.153316+0000", 2024-09-13T14:41:55.000 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.4004999999999997e-05 2024-09-13T14:41:55.000 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:55.001 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:55.001 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:55.001 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.001 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.001 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:164863 6.3 6:ca1c7e5a:::benchmark_data_smithi151_155836_object32405:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1105)", 2024-09-13T14:41:55.001 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:36.313714+0000", 2024-09-13T14:41:55.002 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 18.626612118000001, 2024-09-13T14:41:55.002 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.096759577999999999, 2024-09-13T14:41:55.002 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:55.002 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:55.002 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:55.002 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:55.003 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:55.003 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 164863 2024-09-13T14:41:55.003 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.003 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:55.003 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.004 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:55.004 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.313714+0000", 2024-09-13T14:41:55.004 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:55.004 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.004 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.004 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:55.004 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.313714+0000", 2024-09-13T14:41:55.005 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:55.005 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.005 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.005 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:55.005 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.313713+0000", 2024-09-13T14:41:55.005 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:55.005 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.005 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.005 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:55.005 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.313719+0000", 2024-09-13T14:41:55.005 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.2369999999999997e-06 2024-09-13T14:41:55.005 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.005 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.006 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:55.006 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.313732+0000", 2024-09-13T14:41:55.006 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2849e-05 2024-09-13T14:41:55.006 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.006 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.006 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:55.006 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.313738+0000", 2024-09-13T14:41:55.006 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4199999999999998e-06 2024-09-13T14:41:55.006 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.006 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.007 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:55.007 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.316117+0000", 2024-09-13T14:41:55.007 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002379033 2024-09-13T14:41:55.007 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.007 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.007 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:55.007 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.316410+0000", 2024-09-13T14:41:55.007 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00029381399999999999 2024-09-13T14:41:55.007 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.007 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.007 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:41:55.007 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.316767+0000", 2024-09-13T14:41:55.007 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000356046 2024-09-13T14:41:55.008 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.008 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.008 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:55.008 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.321903+0000", 2024-09-13T14:41:55.008 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0051365580000000003 2024-09-13T14:41:55.008 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.008 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.008 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:55.008 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.339523+0000", 2024-09-13T14:41:55.008 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.017619487999999999 2024-09-13T14:41:55.008 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.008 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.008 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:55.008 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.410375+0000", 2024-09-13T14:41:55.009 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.070852209999999999 2024-09-13T14:41:55.009 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.009 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.009 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:55.009 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.410432+0000", 2024-09-13T14:41:55.009 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7420000000000003e-05 2024-09-13T14:41:55.009 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.009 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.009 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:55.009 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.410474+0000", 2024-09-13T14:41:55.009 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1789000000000003e-05 2024-09-13T14:41:55.009 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:55.009 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:55.010 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:55.010 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.010 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.010 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:164870 6.3 6:cdd71d82:::benchmark_data_smithi151_155836_object32412:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1105)", 2024-09-13T14:41:55.010 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:36.328335+0000", 2024-09-13T14:41:55.010 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 18.611991053000001, 2024-09-13T14:41:55.010 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.096614990999999997, 2024-09-13T14:41:55.010 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:55.010 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:55.010 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:55.010 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:55.010 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:55.010 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 164870 2024-09-13T14:41:55.010 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.011 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:55.011 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.011 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:55.011 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.328335+0000", 2024-09-13T14:41:55.011 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:55.011 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.011 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.011 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:55.011 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.328335+0000", 2024-09-13T14:41:55.011 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:55.011 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.011 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.011 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:55.011 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.328334+0000", 2024-09-13T14:41:55.012 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:55.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.012 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.012 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:55.012 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.328345+0000", 2024-09-13T14:41:55.012 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1223000000000001e-05 2024-09-13T14:41:55.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.012 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.012 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:55.012 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.328361+0000", 2024-09-13T14:41:55.012 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6303999999999998e-05 2024-09-13T14:41:55.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.012 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.013 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:55.013 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.328368+0000", 2024-09-13T14:41:55.013 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.579e-06 2024-09-13T14:41:55.013 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.013 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.013 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:55.013 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.333801+0000", 2024-09-13T14:41:55.013 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0054335690000000001 2024-09-13T14:41:55.013 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.013 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.013 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:55.013 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.334165+0000", 2024-09-13T14:41:55.013 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00036308 2024-09-13T14:41:55.013 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.014 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.014 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:41:55.014 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.334568+0000", 2024-09-13T14:41:55.014 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00040392200000000003 2024-09-13T14:41:55.014 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.014 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.014 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:55.014 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.339557+0000", 2024-09-13T14:41:55.014 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0049882349999999997 2024-09-13T14:41:55.014 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.014 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.015 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:55.015 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.344164+0000", 2024-09-13T14:41:55.015 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0046071810000000001 2024-09-13T14:41:55.015 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.015 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.015 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:55.015 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.424845+0000", 2024-09-13T14:41:55.015 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.080680689 2024-09-13T14:41:55.015 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.015 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.015 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:55.015 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.424911+0000", 2024-09-13T14:41:55.015 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.6173999999999994e-05 2024-09-13T14:41:55.016 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.016 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.016 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:55.016 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.424950+0000", 2024-09-13T14:41:55.016 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9691000000000003e-05 2024-09-13T14:41:55.016 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:55.016 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:55.016 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:55.016 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.016 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.016 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:164871 6.3 6:ceb29989:::benchmark_data_smithi151_155836_object32413:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1105)", 2024-09-13T14:41:55.016 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:36.329376+0000", 2024-09-13T14:41:55.016 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 18.610950239000001, 2024-09-13T14:41:55.017 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.095708499000000002, 2024-09-13T14:41:55.017 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:55.017 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:55.017 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:55.017 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:55.017 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:55.017 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 164871 2024-09-13T14:41:55.017 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.017 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:55.017 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.017 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:55.017 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.329376+0000", 2024-09-13T14:41:55.017 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:55.018 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.018 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.018 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:55.018 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.329376+0000", 2024-09-13T14:41:55.018 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:55.018 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.018 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.018 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:55.018 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.329375+0000", 2024-09-13T14:41:55.018 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:41:55.018 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.018 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.018 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:55.018 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.329382+0000", 2024-09-13T14:41:55.019 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3990000000000001e-06 2024-09-13T14:41:55.019 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.019 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.019 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:55.019 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.329397+0000", 2024-09-13T14:41:55.019 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.5415000000000001e-05 2024-09-13T14:41:55.019 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.019 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.019 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:55.019 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.329403+0000", 2024-09-13T14:41:55.019 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.0920000000000003e-06 2024-09-13T14:41:55.019 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.019 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.019 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:55.020 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.339582+0000", 2024-09-13T14:41:55.020 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.010179008999999999 2024-09-13T14:41:55.020 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.020 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.020 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:55.020 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.339704+0000", 2024-09-13T14:41:55.020 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012169 2024-09-13T14:41:55.020 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.020 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.020 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:41:55.020 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.339903+0000", 2024-09-13T14:41:55.020 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00019976099999999999 2024-09-13T14:41:55.020 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.020 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.021 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:55.021 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.344274+0000", 2024-09-13T14:41:55.021 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.004370133 2024-09-13T14:41:55.021 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.021 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.021 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:55.021 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.344670+0000", 2024-09-13T14:41:55.021 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00039620200000000001 2024-09-13T14:41:55.021 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.021 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.021 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:55.021 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.425000+0000", 2024-09-13T14:41:55.021 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.080330547000000002 2024-09-13T14:41:55.022 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.022 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.022 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:55.022 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.425049+0000", 2024-09-13T14:41:55.022 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.863e-05 2024-09-13T14:41:55.022 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.022 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.022 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:55.022 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.425085+0000", 2024-09-13T14:41:55.022 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.5794000000000003e-05 2024-09-13T14:41:55.022 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:55.022 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:55.022 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:55.022 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.023 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.023 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:164979 6.3 6:cc0c9a3a:::benchmark_data_smithi151_155836_object32521:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1105)", 2024-09-13T14:41:55.023 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:36.486494+0000", 2024-09-13T14:41:55.023 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 18.453832078000001, 2024-09-13T14:41:55.023 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.083579839000000003, 2024-09-13T14:41:55.023 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:55.023 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:55.023 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:55.023 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:55.023 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:55.024 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 164979 2024-09-13T14:41:55.024 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.024 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:55.024 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.024 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:55.024 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.486494+0000", 2024-09-13T14:41:55.024 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:55.024 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.024 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.024 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:55.024 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.486494+0000", 2024-09-13T14:41:55.024 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:55.024 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.025 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.025 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:55.025 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.486493+0000", 2024-09-13T14:41:55.025 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:41:55.025 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.025 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.025 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:55.025 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.486500+0000", 2024-09-13T14:41:55.025 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.6159999999999996e-06 2024-09-13T14:41:55.025 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.025 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.025 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:55.025 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.486516+0000", 2024-09-13T14:41:55.025 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6137000000000002e-05 2024-09-13T14:41:55.026 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.026 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.026 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:55.026 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.486523+0000", 2024-09-13T14:41:55.026 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.5919999999999997e-06 2024-09-13T14:41:55.026 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.026 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.026 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:55.026 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.493797+0000", 2024-09-13T14:41:55.026 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0072735689999999997 2024-09-13T14:41:55.026 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.026 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.026 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:55.026 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.493935+0000", 2024-09-13T14:41:55.027 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000138035 2024-09-13T14:41:55.027 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.027 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.027 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:41:55.027 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.494074+0000", 2024-09-13T14:41:55.027 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000139188 2024-09-13T14:41:55.027 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.027 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.027 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:55.027 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.497363+0000", 2024-09-13T14:41:55.027 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0032893760000000001 2024-09-13T14:41:55.027 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.027 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.027 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:55.028 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.499773+0000", 2024-09-13T14:41:55.028 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0024098959999999999 2024-09-13T14:41:55.028 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.028 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.028 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:55.028 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.569953+0000", 2024-09-13T14:41:55.028 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.070179615000000001 2024-09-13T14:41:55.028 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.028 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.028 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:55.028 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.570035+0000", 2024-09-13T14:41:55.028 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.2607000000000004e-05 2024-09-13T14:41:55.028 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.028 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.029 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:55.029 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.570074+0000", 2024-09-13T14:41:55.029 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8996999999999999e-05 2024-09-13T14:41:55.029 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:55.029 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:55.029 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:55.029 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.029 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.029 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:165062 6.3 6:ce8e0480:::benchmark_data_smithi151_155836_object32604:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1105)", 2024-09-13T14:41:55.029 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:36.598745+0000", 2024-09-13T14:41:55.029 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 18.341581579, 2024-09-13T14:41:55.029 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.117145496, 2024-09-13T14:41:55.029 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:55.029 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:55.030 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:55.030 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:55.030 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:55.030 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 165062 2024-09-13T14:41:55.030 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.030 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:55.030 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.030 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:55.030 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.598745+0000", 2024-09-13T14:41:55.030 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:55.030 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.030 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.030 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:55.031 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.598745+0000", 2024-09-13T14:41:55.031 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:55.031 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.031 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.031 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:55.031 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.598744+0000", 2024-09-13T14:41:55.031 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:41:55.031 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.031 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.031 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:55.031 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.598748+0000", 2024-09-13T14:41:55.031 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9689999999999996e-06 2024-09-13T14:41:55.031 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.031 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.032 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:55.032 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.598756+0000", 2024-09-13T14:41:55.032 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.7160000000000003e-06 2024-09-13T14:41:55.032 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.032 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.032 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:55.032 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.598760+0000", 2024-09-13T14:41:55.032 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8449999999999996e-06 2024-09-13T14:41:55.032 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.032 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.032 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:55.032 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.600128+0000", 2024-09-13T14:41:55.032 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001368317 2024-09-13T14:41:55.032 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.033 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.033 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:55.033 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.600237+0000", 2024-09-13T14:41:55.033 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000108993 2024-09-13T14:41:55.033 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.033 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.033 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:41:55.033 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.600339+0000", 2024-09-13T14:41:55.033 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000102292 2024-09-13T14:41:55.033 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.033 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.033 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:55.033 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.604286+0000", 2024-09-13T14:41:55.033 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0039471799999999998 2024-09-13T14:41:55.034 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.034 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.034 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:55.034 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.605760+0000", 2024-09-13T14:41:55.034 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0014739239999999999 2024-09-13T14:41:55.034 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.034 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.034 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:55.034 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.715758+0000", 2024-09-13T14:41:55.034 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.109997735 2024-09-13T14:41:55.034 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.034 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.034 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:55.034 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.715845+0000", 2024-09-13T14:41:55.035 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.6551999999999994e-05 2024-09-13T14:41:55.035 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.035 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.035 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:55.035 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.715890+0000", 2024-09-13T14:41:55.035 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5841999999999999e-05 2024-09-13T14:41:55.035 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:55.035 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:55.035 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:55.035 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.035 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.035 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:165071 6.3 6:cccdb842:::benchmark_data_smithi151_155836_object32613:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1105)", 2024-09-13T14:41:55.035 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:36.609340+0000", 2024-09-13T14:41:55.036 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 18.330986544000002, 2024-09-13T14:41:55.036 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.106653145, 2024-09-13T14:41:55.036 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:55.036 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:55.036 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:55.036 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:55.036 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:55.036 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 165071 2024-09-13T14:41:55.036 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.036 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:55.036 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.036 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:55.036 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.609340+0000", 2024-09-13T14:41:55.036 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:55.037 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.037 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.037 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:55.037 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.609340+0000", 2024-09-13T14:41:55.037 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:55.037 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.037 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.037 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:55.037 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.609339+0000", 2024-09-13T14:41:55.037 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:41:55.037 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.037 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.037 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:55.037 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.609344+0000", 2024-09-13T14:41:55.038 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9590000000000003e-06 2024-09-13T14:41:55.038 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.038 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.038 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:55.038 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.609353+0000", 2024-09-13T14:41:55.038 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.1400000000000006e-06 2024-09-13T14:41:55.038 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.038 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.038 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:55.038 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.609369+0000", 2024-09-13T14:41:55.038 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6144999999999999e-05 2024-09-13T14:41:55.038 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.038 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.039 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:55.039 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.609405+0000", 2024-09-13T14:41:55.039 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.5408999999999999e-05 2024-09-13T14:41:55.039 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.039 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.039 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:55.039 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.609519+0000", 2024-09-13T14:41:55.039 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000113863 2024-09-13T14:41:55.039 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.039 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.039 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:41:55.039 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.609634+0000", 2024-09-13T14:41:55.039 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000115359 2024-09-13T14:41:55.039 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.040 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.040 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:55.040 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.613663+0000", 2024-09-13T14:41:55.040 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0040295979999999997 2024-09-13T14:41:55.040 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.040 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.040 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:55.040 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.618823+0000", 2024-09-13T14:41:55.040 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0051598039999999996 2024-09-13T14:41:55.040 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.040 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.040 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:55.040 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.715929+0000", 2024-09-13T14:41:55.040 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.097105271000000007 2024-09-13T14:41:55.041 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.041 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.041 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:55.041 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.715963+0000", 2024-09-13T14:41:55.041 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.4581e-05 2024-09-13T14:41:55.041 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.041 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.041 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:55.041 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.715993+0000", 2024-09-13T14:41:55.041 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.9980000000000001e-05 2024-09-13T14:41:55.041 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:55.041 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:55.041 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:55.042 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.042 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.042 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:165093 6.3 6:ce3baf40:::benchmark_data_smithi151_155836_object32635:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1105)", 2024-09-13T14:41:55.042 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:36.640761+0000", 2024-09-13T14:41:55.042 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 18.299565052999998, 2024-09-13T14:41:55.042 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.086200842999999999, 2024-09-13T14:41:55.042 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:55.042 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:55.042 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:55.042 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:55.043 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:55.043 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 165093 2024-09-13T14:41:55.043 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.043 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:55.043 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.043 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:55.043 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.640761+0000", 2024-09-13T14:41:55.043 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:55.043 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.043 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.044 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:55.044 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.640761+0000", 2024-09-13T14:41:55.044 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:55.044 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.044 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.044 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:55.044 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.640760+0000", 2024-09-13T14:41:55.044 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:41:55.044 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.044 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.044 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:55.044 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.640768+0000", 2024-09-13T14:41:55.044 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.7369999999999995e-06 2024-09-13T14:41:55.045 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.045 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.045 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:55.045 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.640786+0000", 2024-09-13T14:41:55.045 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.7765999999999999e-05 2024-09-13T14:41:55.045 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.045 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.045 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:55.045 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.640793+0000", 2024-09-13T14:41:55.045 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3940000000000001e-06 2024-09-13T14:41:55.045 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.045 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.045 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:55.045 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.640981+0000", 2024-09-13T14:41:55.046 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000188896 2024-09-13T14:41:55.046 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.046 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.046 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:55.046 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.644210+0000", 2024-09-13T14:41:55.046 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0032286429999999998 2024-09-13T14:41:55.046 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.046 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.046 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:41:55.046 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.647528+0000", 2024-09-13T14:41:55.046 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0033180869999999999 2024-09-13T14:41:55.046 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.046 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.046 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:55.046 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.654839+0000", 2024-09-13T14:41:55.047 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0073105440000000004 2024-09-13T14:41:55.047 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.047 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.047 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:55.047 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.655089+0000", 2024-09-13T14:41:55.047 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00024992599999999998 2024-09-13T14:41:55.047 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.047 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.047 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:55.047 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.726845+0000", 2024-09-13T14:41:55.047 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.071756698999999993 2024-09-13T14:41:55.047 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.047 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.047 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:55.048 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.726926+0000", 2024-09-13T14:41:55.048 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0227000000000003e-05 2024-09-13T14:41:55.048 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.048 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.048 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:55.048 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.726962+0000", 2024-09-13T14:41:55.048 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.6677999999999997e-05 2024-09-13T14:41:55.048 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:55.048 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:55.048 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:55.048 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.048 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.048 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:165644 6.3 6:c856d2a3:::benchmark_data_smithi151_155836_object33186:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1105)", 2024-09-13T14:41:55.049 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:37.331797+0000", 2024-09-13T14:41:55.049 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 17.608529558000001, 2024-09-13T14:41:55.049 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.079988781999999994, 2024-09-13T14:41:55.049 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:55.049 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:55.049 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:55.049 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:55.049 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:55.049 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 165644 2024-09-13T14:41:55.049 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.049 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:55.049 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.049 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:55.049 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.331797+0000", 2024-09-13T14:41:55.049 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:55.051 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.051 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.051 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:55.051 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.331797+0000", 2024-09-13T14:41:55.051 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:55.051 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.051 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.051 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:55.051 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.331795+0000", 2024-09-13T14:41:55.051 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:41:55.051 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.051 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.051 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:55.052 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.331803+0000", 2024-09-13T14:41:55.052 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.1850000000000004e-06 2024-09-13T14:41:55.052 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.052 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.052 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:55.052 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.331816+0000", 2024-09-13T14:41:55.052 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.289e-05 2024-09-13T14:41:55.052 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.052 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.052 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:55.052 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.331822+0000", 2024-09-13T14:41:55.052 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8989999999999998e-06 2024-09-13T14:41:55.052 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.053 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.053 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:55.053 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.331861+0000", 2024-09-13T14:41:55.053 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8958999999999997e-05 2024-09-13T14:41:55.053 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.053 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.053 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:55.053 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.331964+0000", 2024-09-13T14:41:55.053 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000102521 2024-09-13T14:41:55.053 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.053 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.053 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:41:55.053 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.332061+0000", 2024-09-13T14:41:55.053 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.7312000000000006e-05 2024-09-13T14:41:55.054 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.054 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.054 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:55.054 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.337728+0000", 2024-09-13T14:41:55.054 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0056668029999999998 2024-09-13T14:41:55.054 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.054 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.054 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:55.054 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.337735+0000", 2024-09-13T14:41:55.054 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.8889999999999996e-06 2024-09-13T14:41:55.054 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.054 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.054 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:55.054 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.411699+0000", 2024-09-13T14:41:55.055 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.073963980999999998 2024-09-13T14:41:55.055 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.055 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.055 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:55.055 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.411750+0000", 2024-09-13T14:41:55.055 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1069999999999997e-05 2024-09-13T14:41:55.055 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.055 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.055 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:55.055 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:37.411786+0000", 2024-09-13T14:41:55.055 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.6028999999999997e-05 2024-09-13T14:41:55.055 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:55.055 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:55.055 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:55.056 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.056 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.056 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:166239 6.3 6:c3d704a2:::benchmark_data_smithi151_155836_object33781:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1105)", 2024-09-13T14:41:55.056 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:38.082940+0000", 2024-09-13T14:41:55.056 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 16.857386721000001, 2024-09-13T14:41:55.056 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.093598293999999999, 2024-09-13T14:41:55.056 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:55.056 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:55.056 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:55.056 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:55.056 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:55.056 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 166239 2024-09-13T14:41:55.056 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.056 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:55.057 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.057 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:55.057 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.082940+0000", 2024-09-13T14:41:55.057 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:55.057 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.057 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.057 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:55.057 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.082940+0000", 2024-09-13T14:41:55.057 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:55.057 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.057 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.057 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:55.057 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.082939+0000", 2024-09-13T14:41:55.057 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:41:55.058 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.058 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.058 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:55.058 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.082946+0000", 2024-09-13T14:41:55.058 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.8129999999999998e-06 2024-09-13T14:41:55.058 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.058 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.058 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:55.058 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.082971+0000", 2024-09-13T14:41:55.058 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.5219999999999999e-05 2024-09-13T14:41:55.058 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.058 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.058 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:55.058 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.082977+0000", 2024-09-13T14:41:55.059 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8839999999999999e-06 2024-09-13T14:41:55.059 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.059 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.059 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:55.059 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.102334+0000", 2024-09-13T14:41:55.059 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.019357464000000001 2024-09-13T14:41:55.059 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.059 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.059 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:55.059 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.102514+0000", 2024-09-13T14:41:55.060 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00017948399999999999 2024-09-13T14:41:55.060 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.060 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.060 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:41:55.060 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.102680+0000", 2024-09-13T14:41:55.060 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000165704 2024-09-13T14:41:55.060 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.060 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.060 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:55.060 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.106192+0000", 2024-09-13T14:41:55.060 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.003512743 2024-09-13T14:41:55.060 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.060 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.060 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:55.061 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.106232+0000", 2024-09-13T14:41:55.061 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.96e-05 2024-09-13T14:41:55.061 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.061 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.061 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:55.061 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.176368+0000", 2024-09-13T14:41:55.061 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.070136292000000003 2024-09-13T14:41:55.061 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.061 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.061 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:55.061 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.176484+0000", 2024-09-13T14:41:55.061 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000115945 2024-09-13T14:41:55.061 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.061 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.062 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:55.062 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.176538+0000", 2024-09-13T14:41:55.062 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3962999999999998e-05 2024-09-13T14:41:55.062 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:55.062 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:55.062 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:55.062 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.062 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.062 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:166397 6.3 6:cb1bba90:::benchmark_data_smithi151_155836_object33939:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1105)", 2024-09-13T14:41:55.062 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:38.290956+0000", 2024-09-13T14:41:55.062 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 16.649370939000001, 2024-09-13T14:41:55.062 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.082366227, 2024-09-13T14:41:55.062 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:55.063 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:55.063 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:55.063 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:55.063 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:55.063 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 166397 2024-09-13T14:41:55.063 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.063 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:55.063 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.063 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:55.063 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.290956+0000", 2024-09-13T14:41:55.063 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:55.063 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.063 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.063 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:55.064 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.290956+0000", 2024-09-13T14:41:55.064 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:55.064 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.064 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.064 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:55.064 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.290954+0000", 2024-09-13T14:41:55.064 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:55.064 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.064 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.064 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:55.064 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.290963+0000", 2024-09-13T14:41:55.064 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.7450000000000001e-06 2024-09-13T14:41:55.064 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.064 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.065 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:55.065 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.290976+0000", 2024-09-13T14:41:55.065 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3212e-05 2024-09-13T14:41:55.065 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.065 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.065 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:55.065 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.290982+0000", 2024-09-13T14:41:55.065 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.2319999999999998e-06 2024-09-13T14:41:55.065 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.065 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.065 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:55.065 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.291030+0000", 2024-09-13T14:41:55.065 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7451999999999999e-05 2024-09-13T14:41:55.066 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.066 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.066 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:55.066 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.291163+0000", 2024-09-13T14:41:55.066 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013287800000000001 2024-09-13T14:41:55.066 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.066 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.066 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:41:55.066 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.291299+0000", 2024-09-13T14:41:55.066 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013624399999999999 2024-09-13T14:41:55.066 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.066 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.066 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:55.067 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.295377+0000", 2024-09-13T14:41:55.067 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0040783979999999996 2024-09-13T14:41:55.067 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.067 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.067 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:55.067 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.295833+0000", 2024-09-13T14:41:55.067 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00045544200000000002 2024-09-13T14:41:55.067 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.067 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.067 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:55.067 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.373148+0000", 2024-09-13T14:41:55.067 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.077315134999999993 2024-09-13T14:41:55.067 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.067 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.068 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:55.068 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.373272+0000", 2024-09-13T14:41:55.068 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012383500000000001 2024-09-13T14:41:55.068 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.068 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.068 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:55.068 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.373322+0000", 2024-09-13T14:41:55.068 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0207999999999998e-05 2024-09-13T14:41:55.068 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:55.068 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:55.068 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:55.068 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.068 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.069 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:166717 6.3 6:c4f800c5:::benchmark_data_smithi151_155836_object34259:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1105)", 2024-09-13T14:41:55.069 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:38.753486+0000", 2024-09-13T14:41:55.069 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 16.186840949, 2024-09-13T14:41:55.069 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.087312492000000005, 2024-09-13T14:41:55.069 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:55.069 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:55.069 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:55.069 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:55.069 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:55.069 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 166717 2024-09-13T14:41:55.069 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.069 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:55.069 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.069 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:55.070 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.753486+0000", 2024-09-13T14:41:55.070 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:55.070 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.070 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.070 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:55.070 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.753486+0000", 2024-09-13T14:41:55.070 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:55.070 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.070 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.070 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:55.070 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.753483+0000", 2024-09-13T14:41:55.070 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999976 2024-09-13T14:41:55.070 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.070 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.071 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:55.071 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.753494+0000", 2024-09-13T14:41:55.071 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0679999999999999e-05 2024-09-13T14:41:55.071 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.071 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.071 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:55.071 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.753515+0000", 2024-09-13T14:41:55.071 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.1471000000000002e-05 2024-09-13T14:41:55.071 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.071 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.071 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:55.071 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.753524+0000", 2024-09-13T14:41:55.071 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.3180000000000007e-06 2024-09-13T14:41:55.071 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.072 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.072 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:55.072 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.753571+0000", 2024-09-13T14:41:55.072 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7216000000000003e-05 2024-09-13T14:41:55.072 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.072 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.072 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:55.072 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.753689+0000", 2024-09-13T14:41:55.072 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000118047 2024-09-13T14:41:55.072 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.072 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.073 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:41:55.073 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.753802+0000", 2024-09-13T14:41:55.073 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000113565 2024-09-13T14:41:55.073 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.073 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.073 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:55.073 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.759333+0000", 2024-09-13T14:41:55.073 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00553105 2024-09-13T14:41:55.073 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.073 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.074 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:55.074 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.767740+0000", 2024-09-13T14:41:55.074 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0084066339999999996 2024-09-13T14:41:55.074 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.074 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.074 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:55.074 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.840716+0000", 2024-09-13T14:41:55.074 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.072976035999999994 2024-09-13T14:41:55.074 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.074 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.074 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:55.074 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.840764+0000", 2024-09-13T14:41:55.074 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7794999999999999e-05 2024-09-13T14:41:55.074 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.075 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.075 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:55.075 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.840798+0000", 2024-09-13T14:41:55.075 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.4161e-05 2024-09-13T14:41:55.075 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:55.075 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:55.075 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:55.075 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.075 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.075 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:169381 6.3 6:ca2bb9af:::benchmark_data_smithi151_155836_object36923:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1106)", 2024-09-13T14:41:55.075 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:42.190244+0000", 2024-09-13T14:41:55.075 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 12.750082062000001, 2024-09-13T14:41:55.075 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.106670681, 2024-09-13T14:41:55.076 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:55.076 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:55.076 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:55.076 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:55.076 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:55.076 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 169381 2024-09-13T14:41:55.076 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.076 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:55.076 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.076 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:55.076 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.190244+0000", 2024-09-13T14:41:55.076 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:55.076 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.076 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.077 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:55.077 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.190244+0000", 2024-09-13T14:41:55.077 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:55.077 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.077 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.077 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:55.077 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.190243+0000", 2024-09-13T14:41:55.077 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:55.077 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.077 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.077 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:55.077 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.190249+0000", 2024-09-13T14:41:55.077 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.319e-06 2024-09-13T14:41:55.078 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.078 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.078 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:55.078 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.190262+0000", 2024-09-13T14:41:55.078 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2497000000000001e-05 2024-09-13T14:41:55.078 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.078 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.078 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:55.080 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.190268+0000", 2024-09-13T14:41:55.080 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1999999999999999e-06 2024-09-13T14:41:55.080 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.080 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.080 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:55.080 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.193255+0000", 2024-09-13T14:41:55.080 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0029874430000000002 2024-09-13T14:41:55.080 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.080 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.081 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:55.081 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.193366+0000", 2024-09-13T14:41:55.081 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000110747 2024-09-13T14:41:55.081 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.081 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.081 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:41:55.081 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.193478+0000", 2024-09-13T14:41:55.081 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000112116 2024-09-13T14:41:55.081 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.081 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.081 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:55.081 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.198215+0000", 2024-09-13T14:41:55.081 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0047369270000000002 2024-09-13T14:41:55.082 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.082 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.082 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:55.082 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.198527+0000", 2024-09-13T14:41:55.082 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00031238499999999998 2024-09-13T14:41:55.082 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.082 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.082 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:55.082 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.296714+0000", 2024-09-13T14:41:55.082 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.098186133999999994 2024-09-13T14:41:55.082 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.082 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.082 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:55.082 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.296823+0000", 2024-09-13T14:41:55.083 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00010893 2024-09-13T14:41:55.083 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.083 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.083 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:55.083 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.296915+0000", 2024-09-13T14:41:55.083 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.2604999999999999e-05 2024-09-13T14:41:55.083 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:55.083 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:55.083 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:55.083 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.083 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.083 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:169395 6.3 6:c57b1f00:::benchmark_data_smithi151_155836_object36937:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1106)", 2024-09-13T14:41:55.083 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:42.208374+0000", 2024-09-13T14:41:55.083 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 12.731952492, 2024-09-13T14:41:55.084 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.089034355999999995, 2024-09-13T14:41:55.084 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:55.084 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:55.084 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:55.084 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:55.084 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:55.084 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 169395 2024-09-13T14:41:55.084 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.084 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:55.084 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.084 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:55.084 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.208374+0000", 2024-09-13T14:41:55.084 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:55.085 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.085 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.085 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:55.085 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.208374+0000", 2024-09-13T14:41:55.085 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:55.085 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.085 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.085 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:55.085 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.208373+0000", 2024-09-13T14:41:55.085 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:55.085 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.085 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.085 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:55.085 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.208378+0000", 2024-09-13T14:41:55.086 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6540000000000004e-06 2024-09-13T14:41:55.086 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.086 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.086 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:55.086 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.208387+0000", 2024-09-13T14:41:55.086 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.1940000000000004e-06 2024-09-13T14:41:55.086 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.086 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.086 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:55.086 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.208392+0000", 2024-09-13T14:41:55.086 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1030000000000004e-06 2024-09-13T14:41:55.086 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.086 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.086 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:55.087 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.208428+0000", 2024-09-13T14:41:55.087 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.6869000000000002e-05 2024-09-13T14:41:55.087 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.087 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.087 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:55.087 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.208504+0000", 2024-09-13T14:41:55.087 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.6070999999999997e-05 2024-09-13T14:41:55.087 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.087 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.087 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:41:55.087 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.208621+0000", 2024-09-13T14:41:55.087 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000116648 2024-09-13T14:41:55.087 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.087 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.088 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:55.088 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.211281+0000", 2024-09-13T14:41:55.088 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0026602660000000001 2024-09-13T14:41:55.088 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.088 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.088 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:55.088 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.211629+0000", 2024-09-13T14:41:55.088 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00034780100000000001 2024-09-13T14:41:55.088 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.088 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.088 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:55.088 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.296952+0000", 2024-09-13T14:41:55.088 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.085323087000000006 2024-09-13T14:41:55.089 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.089 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.089 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:55.089 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.296986+0000", 2024-09-13T14:41:55.089 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.4035000000000002e-05 2024-09-13T14:41:55.089 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.089 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.089 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:55.089 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.297408+0000", 2024-09-13T14:41:55.089 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00042208099999999999 2024-09-13T14:41:55.089 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:55.089 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:55.089 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:55.089 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.090 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.090 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:169402 6.5 6:a99d32f5:::benchmark_data_smithi151_155836_object36944:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1106)", 2024-09-13T14:41:55.090 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:42.218431+0000", 2024-09-13T14:41:55.090 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 12.721895741000001, 2024-09-13T14:41:55.090 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.083828418000000002, 2024-09-13T14:41:55.090 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:41:55.090 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:55.090 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:41:55.090 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:41:55.090 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:41:55.090 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 169402 2024-09-13T14:41:55.090 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.090 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:41:55.090 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.091 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:41:55.091 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.218431+0000", 2024-09-13T14:41:55.091 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:55.091 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.091 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.091 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:41:55.091 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.218431+0000", 2024-09-13T14:41:55.091 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:41:55.091 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.091 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.091 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:41:55.091 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.218429+0000", 2024-09-13T14:41:55.091 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:55.091 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.092 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.092 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:41:55.092 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.218438+0000", 2024-09-13T14:41:55.092 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.7269999999999996e-06 2024-09-13T14:41:55.092 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.092 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.092 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:41:55.092 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.292122+0000", 2024-09-13T14:41:55.092 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.073684082999999997 2024-09-13T14:41:55.092 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.092 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.092 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:41:55.092 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.292139+0000", 2024-09-13T14:41:55.092 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.7136999999999999e-05 2024-09-13T14:41:55.093 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.093 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.093 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:41:55.093 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.292261+0000", 2024-09-13T14:41:55.093 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000122046 2024-09-13T14:41:55.093 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.093 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.093 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:41:55.093 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.292347+0000", 2024-09-13T14:41:55.093 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.5599999999999994e-05 2024-09-13T14:41:55.093 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.093 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.093 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:41:55.094 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.292496+0000", 2024-09-13T14:41:55.094 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000149037 2024-09-13T14:41:55.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.094 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.094 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:55.094 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.298833+0000", 2024-09-13T14:41:55.094 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0063366760000000003 2024-09-13T14:41:55.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.094 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.094 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:55.094 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.298864+0000", 2024-09-13T14:41:55.094 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.1364e-05 2024-09-13T14:41:55.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.094 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.095 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:41:55.095 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.302178+0000", 2024-09-13T14:41:55.095 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0033135999999999999 2024-09-13T14:41:55.095 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.095 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.095 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:41:55.095 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.302224+0000", 2024-09-13T14:41:55.095 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.6582999999999998e-05 2024-09-13T14:41:55.095 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:41:55.095 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:41:55.095 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:41:55.095 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:42.302259+0000", 2024-09-13T14:41:55.095 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.4993999999999997e-05 2024-09-13T14:41:55.096 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:55.096 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:55.096 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:55.096 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:41:55.096 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:41:55.096 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:41:55.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:55.098 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:55.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:55.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:54 smithi151 ceph-mon[104450]: pgmap v1393: 109 pgs: 1 active+recovery_wait+remapped, 1 peering, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 100 active+clean; 5.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 593 op/s; 556/269130 objects degraded (0.207%); 21991/269130 objects misplaced (8.171%); 15 MiB/s, 241 objects/s recovering 2024-09-13T14:41:55.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:54 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 556/269130 objects degraded (0.207%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:41:55.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:54 smithi151 ceph-mon[104450]: 6.10 scrub starts 2024-09-13T14:41:55.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:54 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2036402094' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:55.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:54 smithi151 ceph-mon[108966]: pgmap v1393: 109 pgs: 1 active+recovery_wait+remapped, 1 peering, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 100 active+clean; 5.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 593 op/s; 556/269130 objects degraded (0.207%); 21991/269130 objects misplaced (8.171%); 15 MiB/s, 241 objects/s recovering 2024-09-13T14:41:55.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:54 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 556/269130 objects degraded (0.207%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:41:55.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:54 smithi151 ceph-mon[108966]: 6.10 scrub starts 2024-09-13T14:41:55.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:54 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2036402094' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:55.198 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:55.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:55.236 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:55.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:55.336 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:55.368 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:55.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:55.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:55.470 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:55.513 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:55.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:55.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:55.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:55.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:55.644 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:55.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:55.745 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:55.785 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:55.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:55.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:55.887 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:55.911 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:55.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:55.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:55.976 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:41:56.013 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:56.039 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:56.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:56.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:56.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:55 smithi184 ceph-mon[92527]: 6.19 scrub starts 2024-09-13T14:41:56.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:55 smithi151 ceph-mon[104450]: 6.19 scrub starts 2024-09-13T14:41:56.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:55 smithi151 ceph-mon[108966]: 6.19 scrub starts 2024-09-13T14:41:56.141 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:56.165 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:56.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:56.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:56.248 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:41:56.268 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:56.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:56.310 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:56.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:56.411 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:56.445 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:56.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:56.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:56.547 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:56.577 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:56.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:56.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:56.679 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:56.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:56.707 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:56.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:56.808 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:56.838 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:56.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:56.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:56.941 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:56.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:56.977 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:56.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:57.074 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:56 smithi151 ceph-mon[104450]: pgmap v1394: 109 pgs: 1 active+recovery_wait+remapped, 1 peering, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 100 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 618 op/s; 675/266490 objects degraded (0.253%); 21798/266490 objects misplaced (8.180%); 12 MiB/s, 191 objects/s recovering 2024-09-13T14:41:57.075 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:56 smithi151 ceph-mon[108966]: pgmap v1394: 109 pgs: 1 active+recovery_wait+remapped, 1 peering, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 100 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 618 op/s; 675/266490 objects degraded (0.253%); 21798/266490 objects misplaced (8.180%); 12 MiB/s, 191 objects/s recovering 2024-09-13T14:41:57.078 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:57.078 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:57.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:56 smithi184 ceph-mon[92527]: pgmap v1394: 109 pgs: 1 active+recovery_wait+remapped, 1 peering, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 100 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 618 op/s; 675/266490 objects degraded (0.253%); 21798/266490 objects misplaced (8.180%); 12 MiB/s, 191 objects/s recovering 2024-09-13T14:41:57.114 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:57.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:57.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:57.216 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:57.254 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:57.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:57.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:57.356 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:57.391 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:57.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:57.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:57.493 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:57.532 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:57.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:57.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:57.634 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:57.673 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:41:57.675 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T14:41:57.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:57.676 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:57.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:57.777 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:57.816 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:57.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:57.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:57.917 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:57.952 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:57.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:57.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:58.054 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:58.080 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:58.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:58.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:58.182 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:58.213 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:58.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:58.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:58.315 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:58.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:58.356 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:58.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:58.457 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:58.492 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:58.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:58.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:58.594 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:58.619 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:58.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:58.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:58.721 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:58.746 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:58.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:58.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:58.794 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:41:58.826 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:58.833 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1395,"stamp":"2024-09-13T14:41:57.352278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5742103223,"num_objects":87636,"num_object_clones":0,"num_object_copies":262908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":675,"num_objects_misplaced":21605,"num_objects_unfound":0,"num_objects_dirty":87636,"num_whiteouts":0,"num_read":80,"num_read_kb":205,"num_write":177404,"num_write_kb":8477596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6055,"num_bytes_recovered":396457765,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":170716,"ondisk_log_size":170716,"up":327,"acting":326,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":327,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":17117712,"kb_used_data":15564360,"kb_used_omap":116,"kb_used_meta":1553099,"kb_avail":639098352,"statfs":{"total":671965249536,"available":654436712448,"internally_reserved":0,"allocated":15937904640,"data_stored":15915068070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":119649,"internal_metadata":1590373535},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-329121792,"num_objects":-5022,"num_object_clones":0,"num_object_copies":-15066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":119,"num_objects_misplaced":2752,"num_objects_unfound":0,"num_objects_dirty":-5022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5022,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1255,"num_bytes_recovered":82182144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001989"},"pg_stats":[{"pgid":"6.16","version":"1113'10984","reported_seq":11033,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:57.275448+0000","last_change":"2024-09-13T14:41:46.905438+0000","last_active":"2024-09-13T14:41:57.275448+0000","last_peered":"2024-09-13T14:41:57.275448+0000","last_clean":"2024-09-13T14:41:57.275448+0000","last_became_active":"2024-09-13T14:41:46.886923+0000","last_became_peered":"2024-09-13T14:41:46.886923+0000","last_unstale":"2024-09-13T14:41:57.275448+0000","last_undegraded":"2024-09-13T14:41:57.275448+0000","last_fullsized":"2024-09-13T14:41:57.275448+0000","mapping_epoch":1108,"log_start":"1088'4134","ondisk_log_start":"1088'4134","created":1108,"last_epoch_clean":1109,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6850,"log_dups_size":0,"ondisk_log_size":6850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171999232,"num_objects":2625,"num_object_clones":0,"num_object_copies":7875,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5617,"num_write_kb":263712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1538,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.821682+0000","last_change":"2024-09-13T14:41:28.592397+0000","last_active":"2024-09-13T14:41:52.821682+0000","last_peered":"2024-09-13T14:41:52.821682+0000","last_clean":"2024-09-13T14:41:52.821682+0000","last_became_active":"2024-09-13T14:41:28.592068+0000","last_became_peered":"2024-09-13T14:41:28.592068+0000","last_unstale":"2024-09-13T14:41:52.821682+0000","last_undegraded":"2024-09-13T14:41:52.821682+0000","last_fullsized":"2024-09-13T14:41:52.821682+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:08:03.576401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":1068,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.784764+0000","last_change":"2024-09-13T14:29:42.541093+0000","last_active":"2024-09-13T14:41:48.784764+0000","last_peered":"2024-09-13T14:41:48.784764+0000","last_clean":"2024-09-13T14:41:48.784764+0000","last_became_active":"2024-09-13T14:29:42.540671+0000","last_became_peered":"2024-09-13T14:29:42.540671+0000","last_unstale":"2024-09-13T14:41:48.784764+0000","last_undegraded":"2024-09-13T14:41:48.784764+0000","last_fullsized":"2024-09-13T14:41:48.784764+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:45.722376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"1112'11004","reported_seq":11051,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.414731+0000","last_change":"2024-09-13T14:41:46.872175+0000","last_active":"2024-09-13T14:41:52.414731+0000","last_peered":"2024-09-13T14:41:52.414731+0000","last_clean":"2024-09-13T14:41:52.414731+0000","last_became_active":"2024-09-13T14:41:46.843042+0000","last_became_peered":"2024-09-13T14:41:46.843042+0000","last_unstale":"2024-09-13T14:41:52.414731+0000","last_undegraded":"2024-09-13T14:41:52.414731+0000","last_fullsized":"2024-09-13T14:41:52.414731+0000","mapping_epoch":1108,"log_start":"1088'4115","ondisk_log_start":"1088'4115","created":1108,"last_epoch_clean":1109,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6889,"log_dups_size":0,"ondisk_log_size":6889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182452224,"num_objects":2784,"num_object_clones":0,"num_object_copies":8352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5565,"num_write_kb":267168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1064,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.786983+0000","last_change":"2024-09-13T14:39:11.379707+0000","last_active":"2024-09-13T14:41:48.786983+0000","last_peered":"2024-09-13T14:41:48.786983+0000","last_clean":"2024-09-13T14:41:48.786983+0000","last_became_active":"2024-09-13T14:39:11.379255+0000","last_became_peered":"2024-09-13T14:39:11.379255+0000","last_unstale":"2024-09-13T14:41:48.786983+0000","last_undegraded":"2024-09-13T14:41:48.786983+0000","last_fullsized":"2024-09-13T14:41:48.786983+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:48.543270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1639,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.827164+0000","last_change":"2024-09-13T14:41:28.585446+0000","last_active":"2024-09-13T14:41:52.827164+0000","last_peered":"2024-09-13T14:41:52.827164+0000","last_clean":"2024-09-13T14:41:52.827164+0000","last_became_active":"2024-09-13T14:41:28.584679+0000","last_became_peered":"2024-09-13T14:41:28.584679+0000","last_unstale":"2024-09-13T14:41:52.827164+0000","last_undegraded":"2024-09-13T14:41:52.827164+0000","last_fullsized":"2024-09-13T14:41:52.827164+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:19:12.592060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"1112'11094","reported_seq":14961,"reported_epoch":1112,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:41:52.361910+0000","last_change":"2024-09-13T14:41:48.816604+0000","last_active":"2024-09-13T14:41:52.361910+0000","last_peered":"2024-09-13T14:41:52.361910+0000","last_clean":"2024-09-13T14:41:27.548643+0000","last_became_active":"2024-09-13T14:41:48.792144+0000","last_became_peered":"2024-09-13T14:41:48.792144+0000","last_unstale":"2024-09-13T14:41:52.361910+0000","last_undegraded":"2024-09-13T14:41:52.361910+0000","last_fullsized":"2024-09-13T14:41:52.361910+0000","mapping_epoch":1110,"log_start":"1088'4255","ondisk_log_start":"1088'4255","created":1108,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6839,"log_dups_size":0,"ondisk_log_size":6839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184811520,"num_objects":2820,"num_object_clones":0,"num_object_copies":8460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1929,"num_objects_unfound":0,"num_objects_dirty":2820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5609,"num_write_kb":269728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":947,"num_bytes_recovered":62029824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,1],"avail_no_missing":["5","1","6"],"object_location_counts":[{"shards":"1,5,6","objects":2820}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1599,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.827765+0000","last_change":"2024-09-13T14:39:11.394501+0000","last_active":"2024-09-13T14:41:52.827765+0000","last_peered":"2024-09-13T14:41:52.827765+0000","last_clean":"2024-09-13T14:41:52.827765+0000","last_became_active":"2024-09-13T14:39:11.394134+0000","last_became_peered":"2024-09-13T14:39:11.394134+0000","last_unstale":"2024-09-13T14:41:52.827765+0000","last_undegraded":"2024-09-13T14:41:52.827765+0000","last_fullsized":"2024-09-13T14:41:52.827765+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:40:09.359562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1049,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.784104+0000","last_change":"2024-09-13T14:19:25.873088+0000","last_active":"2024-09-13T14:41:48.784104+0000","last_peered":"2024-09-13T14:41:48.784104+0000","last_clean":"2024-09-13T14:41:48.784104+0000","last_became_active":"2024-09-13T14:19:25.872781+0000","last_became_peered":"2024-09-13T14:19:25.872781+0000","last_unstale":"2024-09-13T14:41:48.784104+0000","last_undegraded":"2024-09-13T14:41:48.784104+0000","last_fullsized":"2024-09-13T14:41:48.784104+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:05.607382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"1112'10643","reported_seq":10691,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.272359+0000","last_change":"2024-09-13T14:41:46.893140+0000","last_active":"2024-09-13T14:41:52.272359+0000","last_peered":"2024-09-13T14:41:52.272359+0000","last_clean":"2024-09-13T14:41:52.272359+0000","last_became_active":"2024-09-13T14:41:46.874728+0000","last_became_peered":"2024-09-13T14:41:46.874728+0000","last_unstale":"2024-09-13T14:41:52.272359+0000","last_undegraded":"2024-09-13T14:41:52.272359+0000","last_fullsized":"2024-09-13T14:41:52.272359+0000","mapping_epoch":1108,"log_start":"1090'4623","ondisk_log_start":"1090'4623","created":1108,"last_epoch_clean":1109,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6020,"log_dups_size":0,"ondisk_log_size":6020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180027392,"num_objects":2747,"num_object_clones":0,"num_object_copies":8241,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5378,"num_write_kb":260000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":611,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.784163+0000","last_change":"2024-09-13T14:41:28.594485+0000","last_active":"2024-09-13T14:41:48.784163+0000","last_peered":"2024-09-13T14:41:48.784163+0000","last_clean":"2024-09-13T14:41:48.784163+0000","last_became_active":"2024-09-13T14:41:28.594050+0000","last_became_peered":"2024-09-13T14:41:28.594050+0000","last_unstale":"2024-09-13T14:41:48.784163+0000","last_undegraded":"2024-09-13T14:41:48.784163+0000","last_fullsized":"2024-09-13T14:41:48.784163+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:03.838081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1057,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.786004+0000","last_change":"2024-09-13T14:28:35.603724+0000","last_active":"2024-09-13T14:41:48.786004+0000","last_peered":"2024-09-13T14:41:48.786004+0000","last_clean":"2024-09-13T14:41:48.786004+0000","last_became_active":"2024-09-13T14:28:35.603316+0000","last_became_peered":"2024-09-13T14:28:35.603316+0000","last_unstale":"2024-09-13T14:41:48.786004+0000","last_undegraded":"2024-09-13T14:41:48.786004+0000","last_fullsized":"2024-09-13T14:41:48.786004+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:47.668223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"1112'11233","reported_seq":11271,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.475659+0000","last_change":"2024-09-13T14:39:16.430434+0000","last_active":"2024-09-13T14:41:52.475659+0000","last_peered":"2024-09-13T14:41:52.475659+0000","last_clean":"2024-09-13T14:41:52.475659+0000","last_became_active":"2024-09-13T14:39:16.430177+0000","last_became_peered":"2024-09-13T14:39:16.430177+0000","last_unstale":"2024-09-13T14:41:52.475659+0000","last_undegraded":"2024-09-13T14:41:52.475659+0000","last_fullsized":"2024-09-13T14:41:52.475659+0000","mapping_epoch":1082,"log_start":"1090'4915","ondisk_log_start":"1090'4915","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6318,"log_dups_size":0,"ondisk_log_size":6318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:44.232119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363397120,"num_objects":5545,"num_object_clones":0,"num_object_copies":16635,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11233,"num_write_kb":536896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":1067,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.784273+0000","last_change":"2024-09-13T14:28:35.608607+0000","last_active":"2024-09-13T14:41:48.784273+0000","last_peered":"2024-09-13T14:41:48.784273+0000","last_clean":"2024-09-13T14:41:48.784273+0000","last_became_active":"2024-09-13T14:28:35.608254+0000","last_became_peered":"2024-09-13T14:28:35.608254+0000","last_unstale":"2024-09-13T14:41:48.784273+0000","last_undegraded":"2024-09-13T14:41:48.784273+0000","last_fullsized":"2024-09-13T14:41:48.784273+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:24.019173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1037,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.787498+0000","last_change":"2024-09-13T14:19:26.887839+0000","last_active":"2024-09-13T14:41:48.787498+0000","last_peered":"2024-09-13T14:41:48.787498+0000","last_clean":"2024-09-13T14:41:48.787498+0000","last_became_active":"2024-09-13T14:19:26.887498+0000","last_became_peered":"2024-09-13T14:19:26.887498+0000","last_unstale":"2024-09-13T14:41:48.787498+0000","last_undegraded":"2024-09-13T14:41:48.787498+0000","last_fullsized":"2024-09-13T14:41:48.787498+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:24.475263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"1113'11182","reported_seq":12136,"reported_epoch":1113,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:41:53.673057+0000","last_change":"2024-09-13T14:41:30.181642+0000","last_active":"2024-09-13T14:41:53.673057+0000","last_peered":"2024-09-13T14:41:53.673057+0000","last_clean":"2024-09-13T14:41:27.568458+0000","last_became_active":"2024-09-13T14:41:29.966413+0000","last_became_peered":"2024-09-13T14:41:29.966413+0000","last_unstale":"2024-09-13T14:41:53.673057+0000","last_undegraded":"2024-09-13T14:41:29.616193+0000","last_fullsized":"2024-09-13T14:41:29.578900+0000","mapping_epoch":1103,"log_start":"1086'1100","ondisk_log_start":"1086'1100","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358875136,"num_objects":5476,"num_object_clones":0,"num_object_copies":16428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":675,"num_objects_misplaced":5475,"num_objects_unfound":0,"num_objects_dirty":5476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11182,"num_write_kb":533056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":438,"num_bytes_recovered":28704768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":5475},{"shards":"3,7","objects":-674}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":1082,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.784509+0000","last_change":"2024-09-13T14:41:28.598356+0000","last_active":"2024-09-13T14:41:48.784509+0000","last_peered":"2024-09-13T14:41:48.784509+0000","last_clean":"2024-09-13T14:41:48.784509+0000","last_became_active":"2024-09-13T14:41:28.588185+0000","last_became_peered":"2024-09-13T14:41:28.588185+0000","last_unstale":"2024-09-13T14:41:48.784509+0000","last_undegraded":"2024-09-13T14:41:48.784509+0000","last_fullsized":"2024-09-13T14:41:48.784509+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:54.340118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1090,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-09-13T14:41:51.827252+0000","last_change":"2024-09-13T14:41:28.594910+0000","last_active":"2024-09-13T14:41:51.827252+0000","last_peered":"2024-09-13T14:41:51.827252+0000","last_clean":"2024-09-13T14:41:51.827252+0000","last_became_active":"2024-09-13T14:41:28.593806+0000","last_became_peered":"2024-09-13T14:41:28.593806+0000","last_unstale":"2024-09-13T14:41:51.827252+0000","last_undegraded":"2024-09-13T14:41:51.827252+0000","last_fullsized":"2024-09-13T14:41:51.827252+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:45:14.253295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"1112'11046","reported_seq":11095,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.382308+0000","last_change":"2024-09-13T14:41:46.803069+0000","last_active":"2024-09-13T14:41:52.382308+0000","last_peered":"2024-09-13T14:41:52.382308+0000","last_clean":"2024-09-13T14:41:52.382308+0000","last_became_active":"2024-09-13T14:41:46.783681+0000","last_became_peered":"2024-09-13T14:41:46.783681+0000","last_unstale":"2024-09-13T14:41:52.382308+0000","last_undegraded":"2024-09-13T14:41:52.382308+0000","last_fullsized":"2024-09-13T14:41:52.382308+0000","mapping_epoch":1082,"log_start":"1090'4941","ondisk_log_start":"1090'4941","created":1082,"last_epoch_clean":1109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6105,"log_dups_size":0,"ondisk_log_size":6105,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186712064,"num_objects":2849,"num_object_clones":0,"num_object_copies":8547,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5586,"num_write_kb":269920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":1050,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.784281+0000","last_change":"2024-09-13T14:19:25.871006+0000","last_active":"2024-09-13T14:41:48.784281+0000","last_peered":"2024-09-13T14:41:48.784281+0000","last_clean":"2024-09-13T14:41:48.784281+0000","last_became_active":"2024-09-13T14:19:25.870713+0000","last_became_peered":"2024-09-13T14:19:25.870713+0000","last_unstale":"2024-09-13T14:41:48.784281+0000","last_undegraded":"2024-09-13T14:41:48.784281+0000","last_fullsized":"2024-09-13T14:41:48.784281+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:34.372124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1058,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.786586+0000","last_change":"2024-09-13T14:41:28.586613+0000","last_active":"2024-09-13T14:41:48.786586+0000","last_peered":"2024-09-13T14:41:48.786586+0000","last_clean":"2024-09-13T14:41:48.786586+0000","last_became_active":"2024-09-13T14:41:28.586345+0000","last_became_peered":"2024-09-13T14:41:28.586345+0000","last_unstale":"2024-09-13T14:41:48.786586+0000","last_undegraded":"2024-09-13T14:41:48.786586+0000","last_fullsized":"2024-09-13T14:41:48.786586+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:37.756544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"1113'10878","reported_seq":10959,"reported_epoch":1113,"state":"active+recovery_wait","last_fresh":"2024-09-13T14:41:53.786442+0000","last_change":"2024-09-13T14:41:46.918979+0000","last_active":"2024-09-13T14:41:53.786442+0000","last_peered":"2024-09-13T14:41:53.786442+0000","last_clean":"2024-09-13T14:41:45.766051+0000","last_became_active":"2024-09-13T14:41:46.868938+0000","last_became_peered":"2024-09-13T14:41:46.868938+0000","last_unstale":"2024-09-13T14:41:53.786442+0000","last_undegraded":"2024-09-13T14:41:53.786442+0000","last_fullsized":"2024-09-13T14:41:53.786442+0000","mapping_epoch":1082,"log_start":"1088'3982","ondisk_log_start":"1088'3982","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6896,"log_dups_size":0,"ondisk_log_size":6896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179699712,"num_objects":2742,"num_object_clones":0,"num_object_copies":8226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5517,"num_write_kb":264288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":["0","2","7"],"object_location_counts":[{"shards":"0,2,7","objects":2742}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1075,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.790555+0000","last_change":"2024-09-13T14:41:28.594298+0000","last_active":"2024-09-13T14:41:48.790555+0000","last_peered":"2024-09-13T14:41:48.790555+0000","last_clean":"2024-09-13T14:41:48.790555+0000","last_became_active":"2024-09-13T14:41:28.589751+0000","last_became_peered":"2024-09-13T14:41:28.589751+0000","last_unstale":"2024-09-13T14:41:48.790555+0000","last_undegraded":"2024-09-13T14:41:48.790555+0000","last_fullsized":"2024-09-13T14:41:48.790555+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:15:07.568716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1599,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.826819+0000","last_change":"2024-09-13T14:41:28.587900+0000","last_active":"2024-09-13T14:41:52.826819+0000","last_peered":"2024-09-13T14:41:52.826819+0000","last_clean":"2024-09-13T14:41:52.826819+0000","last_became_active":"2024-09-13T14:41:28.587563+0000","last_became_peered":"2024-09-13T14:41:28.587563+0000","last_unstale":"2024-09-13T14:41:52.826819+0000","last_undegraded":"2024-09-13T14:41:52.826819+0000","last_fullsized":"2024-09-13T14:41:52.826819+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:19:15.384955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"261'100","reported_seq":1212,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.783678+0000","last_change":"2024-09-13T14:41:28.268071+0000","last_active":"2024-09-13T14:41:48.783678+0000","last_peered":"2024-09-13T14:41:48.783678+0000","last_clean":"2024-09-13T14:41:48.783678+0000","last_became_active":"2024-09-13T14:41:20.510300+0000","last_became_peered":"2024-09-13T14:41:20.510300+0000","last_unstale":"2024-09-13T14:41:48.783678+0000","last_undegraded":"2024-09-13T14:41:48.783678+0000","last_fullsized":"2024-09-13T14:41:48.783678+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1100,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:28.268005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:28.268005+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:10.225185+0000","scrub_duration":0.0015413810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"1113'11209","reported_seq":15557,"reported_epoch":1113,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:41:57.311869+0000","last_change":"2024-09-13T14:41:46.903219+0000","last_active":"2024-09-13T14:41:57.311869+0000","last_peered":"2024-09-13T14:41:57.311869+0000","last_clean":"2024-09-13T14:41:27.545589+0000","last_became_active":"2024-09-13T14:41:29.700570+0000","last_became_peered":"2024-09-13T14:41:29.700570+0000","last_unstale":"2024-09-13T14:41:57.311869+0000","last_undegraded":"2024-09-13T14:41:57.311869+0000","last_fullsized":"2024-09-13T14:41:57.311869+0000","mapping_epoch":1103,"log_start":"1090'5162","ondisk_log_start":"1090'5162","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6047,"log_dups_size":0,"ondisk_log_size":6047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:00:18.887770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349110272,"num_objects":5327,"num_object_clones":0,"num_object_copies":15981,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3499,"num_objects_unfound":0,"num_objects_dirty":5327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11209,"num_write_kb":529152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2146,"num_bytes_recovered":140640256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,1],"avail_no_missing":["4","1","2"],"object_location_counts":[{"shards":"1,2,4","objects":5327}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":135,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.827109+0000","last_change":"2024-09-13T14:41:28.600760+0000","last_active":"2024-09-13T14:41:52.827109+0000","last_peered":"2024-09-13T14:41:52.827109+0000","last_clean":"2024-09-13T14:41:52.827109+0000","last_became_active":"2024-09-13T14:41:28.586678+0000","last_became_peered":"2024-09-13T14:41:28.586678+0000","last_unstale":"2024-09-13T14:41:52.827109+0000","last_undegraded":"2024-09-13T14:41:52.827109+0000","last_fullsized":"2024-09-13T14:41:52.827109+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:44:53.564449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1073,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-09-13T14:41:51.828320+0000","last_change":"2024-09-13T14:41:28.598046+0000","last_active":"2024-09-13T14:41:51.828320+0000","last_peered":"2024-09-13T14:41:51.828320+0000","last_clean":"2024-09-13T14:41:51.828320+0000","last_became_active":"2024-09-13T14:41:28.597748+0000","last_became_peered":"2024-09-13T14:41:28.597748+0000","last_unstale":"2024-09-13T14:41:51.828320+0000","last_undegraded":"2024-09-13T14:41:51.828320+0000","last_fullsized":"2024-09-13T14:41:51.828320+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:52.920256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"261'100","reported_seq":27,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.786889+0000","last_change":"2024-09-13T14:41:20.562986+0000","last_active":"2024-09-13T14:41:48.786889+0000","last_peered":"2024-09-13T14:41:48.786889+0000","last_clean":"2024-09-13T14:41:48.786889+0000","last_became_active":"2024-09-13T14:41:20.529244+0000","last_became_peered":"2024-09-13T14:41:20.529244+0000","last_unstale":"2024-09-13T14:41:48.786889+0000","last_undegraded":"2024-09-13T14:41:48.786889+0000","last_fullsized":"2024-09-13T14:41:48.786889+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1100,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:15:42.069756+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":316358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"1113'11099","reported_seq":11175,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:55.696602+0000","last_change":"2024-09-13T14:39:16.433163+0000","last_active":"2024-09-13T14:41:55.696602+0000","last_peered":"2024-09-13T14:41:55.696602+0000","last_clean":"2024-09-13T14:41:55.696602+0000","last_became_active":"2024-09-13T14:39:16.432783+0000","last_became_peered":"2024-09-13T14:39:16.432783+0000","last_unstale":"2024-09-13T14:41:55.696602+0000","last_undegraded":"2024-09-13T14:41:55.696602+0000","last_fullsized":"2024-09-13T14:41:55.696602+0000","mapping_epoch":1082,"log_start":"1090'4669","ondisk_log_start":"1090'4669","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6430,"log_dups_size":0,"ondisk_log_size":6430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:35:05.068064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345571328,"num_objects":5273,"num_object_clones":0,"num_object_copies":15819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11099,"num_write_kb":523904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1059,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.786740+0000","last_change":"2024-09-13T14:41:28.591027+0000","last_active":"2024-09-13T14:41:48.786740+0000","last_peered":"2024-09-13T14:41:48.786740+0000","last_clean":"2024-09-13T14:41:48.786740+0000","last_became_active":"2024-09-13T14:41:28.590742+0000","last_became_peered":"2024-09-13T14:41:28.590742+0000","last_unstale":"2024-09-13T14:41:48.786740+0000","last_undegraded":"2024-09-13T14:41:48.786740+0000","last_fullsized":"2024-09-13T14:41:48.786740+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:01.865182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1090,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-09-13T14:41:51.827712+0000","last_change":"2024-09-13T14:41:28.588833+0000","last_active":"2024-09-13T14:41:51.827712+0000","last_peered":"2024-09-13T14:41:51.827712+0000","last_clean":"2024-09-13T14:41:51.827712+0000","last_became_active":"2024-09-13T14:41:28.588490+0000","last_became_peered":"2024-09-13T14:41:28.588490+0000","last_unstale":"2024-09-13T14:41:51.827712+0000","last_undegraded":"2024-09-13T14:41:51.827712+0000","last_fullsized":"2024-09-13T14:41:51.827712+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:23.440459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"1112'10971","reported_seq":11009,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.368042+0000","last_change":"2024-09-13T14:39:16.428687+0000","last_active":"2024-09-13T14:41:52.368042+0000","last_peered":"2024-09-13T14:41:52.368042+0000","last_clean":"2024-09-13T14:41:52.368042+0000","last_became_active":"2024-09-13T14:39:16.428382+0000","last_became_peered":"2024-09-13T14:39:16.428382+0000","last_unstale":"2024-09-13T14:41:52.368042+0000","last_undegraded":"2024-09-13T14:41:52.368042+0000","last_fullsized":"2024-09-13T14:41:52.368042+0000","mapping_epoch":1082,"log_start":"1091'6158","ondisk_log_start":"1091'6158","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":4813,"log_dups_size":0,"ondisk_log_size":4813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:33:44.732564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368115712,"num_objects":5617,"num_object_clones":0,"num_object_copies":16851,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10971,"num_write_kb":530816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1054,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-09-13T14:41:51.827336+0000","last_change":"2024-09-13T14:29:42.545815+0000","last_active":"2024-09-13T14:41:51.827336+0000","last_peered":"2024-09-13T14:41:51.827336+0000","last_clean":"2024-09-13T14:41:51.827336+0000","last_became_active":"2024-09-13T14:29:42.545508+0000","last_became_peered":"2024-09-13T14:29:42.545508+0000","last_unstale":"2024-09-13T14:41:51.827336+0000","last_undegraded":"2024-09-13T14:41:51.827336+0000","last_fullsized":"2024-09-13T14:41:51.827336+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:39.678381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1063,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-09-13T14:41:51.828315+0000","last_change":"2024-09-13T14:41:28.595541+0000","last_active":"2024-09-13T14:41:51.828315+0000","last_peered":"2024-09-13T14:41:51.828315+0000","last_clean":"2024-09-13T14:41:51.828315+0000","last_became_active":"2024-09-13T14:41:28.593863+0000","last_became_peered":"2024-09-13T14:41:28.593863+0000","last_unstale":"2024-09-13T14:41:51.828315+0000","last_undegraded":"2024-09-13T14:41:51.828315+0000","last_fullsized":"2024-09-13T14:41:51.828315+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:06.730588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"261'100","reported_seq":32,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.787900+0000","last_change":"2024-09-13T14:41:29.624236+0000","last_active":"2024-09-13T14:41:48.787900+0000","last_peered":"2024-09-13T14:41:48.787900+0000","last_clean":"2024-09-13T14:41:48.787900+0000","last_became_active":"2024-09-13T14:41:20.530755+0000","last_became_peered":"2024-09-13T14:41:20.530755+0000","last_unstale":"2024-09-13T14:41:48.787900+0000","last_undegraded":"2024-09-13T14:41:48.787900+0000","last_fullsized":"2024-09-13T14:41:48.787900+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1100,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:29.624129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:29.624129+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:16.408396+0000","scrub_duration":0.0076481020000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"1112'10978","reported_seq":11659,"reported_epoch":1112,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:41:52.472426+0000","last_change":"2024-09-13T14:41:46.889575+0000","last_active":"2024-09-13T14:41:52.472426+0000","last_peered":"2024-09-13T14:41:52.472426+0000","last_clean":"2024-09-13T14:41:27.579452+0000","last_became_active":"2024-09-13T14:41:29.706889+0000","last_became_peered":"2024-09-13T14:41:29.706889+0000","last_unstale":"2024-09-13T14:41:52.472426+0000","last_undegraded":"2024-09-13T14:41:52.472426+0000","last_fullsized":"2024-09-13T14:41:52.472426+0000","mapping_epoch":1103,"log_start":"1090'4672","ondisk_log_start":"1090'4672","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6306,"log_dups_size":0,"ondisk_log_size":6306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:38:40.997498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362479640,"num_objects":5532,"num_object_clones":0,"num_object_copies":16596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5233,"num_objects_unfound":0,"num_objects_dirty":5532,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10978,"num_write_kb":528257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":312,"num_bytes_recovered":20447232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,1],"avail_no_missing":["5","0","1"],"object_location_counts":[{"shards":"0,1,5","objects":5532}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":581,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.783669+0000","last_change":"2024-09-13T14:41:28.587940+0000","last_active":"2024-09-13T14:41:48.783669+0000","last_peered":"2024-09-13T14:41:48.783669+0000","last_clean":"2024-09-13T14:41:48.783669+0000","last_became_active":"2024-09-13T14:41:28.587643+0000","last_became_peered":"2024-09-13T14:41:28.587643+0000","last_unstale":"2024-09-13T14:41:48.783669+0000","last_undegraded":"2024-09-13T14:41:48.783669+0000","last_fullsized":"2024-09-13T14:41:48.783669+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:09:23.651717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":629,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.821831+0000","last_change":"2024-09-13T14:41:28.592799+0000","last_active":"2024-09-13T14:41:52.821831+0000","last_peered":"2024-09-13T14:41:52.821831+0000","last_clean":"2024-09-13T14:41:52.821831+0000","last_became_active":"2024-09-13T14:41:28.592345+0000","last_became_peered":"2024-09-13T14:41:28.592345+0000","last_unstale":"2024-09-13T14:41:52.821831+0000","last_undegraded":"2024-09-13T14:41:52.821831+0000","last_fullsized":"2024-09-13T14:41:52.821831+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:17.651032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"1112'11014","reported_seq":11062,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.544462+0000","last_change":"2024-09-13T14:41:46.836890+0000","last_active":"2024-09-13T14:41:52.544462+0000","last_peered":"2024-09-13T14:41:52.544462+0000","last_clean":"2024-09-13T14:41:52.544462+0000","last_became_active":"2024-09-13T14:41:46.813650+0000","last_became_peered":"2024-09-13T14:41:46.813650+0000","last_unstale":"2024-09-13T14:41:52.544462+0000","last_undegraded":"2024-09-13T14:41:52.544462+0000","last_fullsized":"2024-09-13T14:41:52.544462+0000","mapping_epoch":1082,"log_start":"1088'4115","ondisk_log_start":"1088'4115","created":1082,"last_epoch_clean":1109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6899,"log_dups_size":0,"ondisk_log_size":6899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181796864,"num_objects":2774,"num_object_clones":0,"num_object_copies":8322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5575,"num_write_kb":267168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"261'100","reported_seq":1203,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.783966+0000","last_change":"2024-09-13T14:41:23.257356+0000","last_active":"2024-09-13T14:41:48.783966+0000","last_peered":"2024-09-13T14:41:48.783966+0000","last_clean":"2024-09-13T14:41:48.783966+0000","last_became_active":"2024-09-13T14:41:16.493791+0000","last_became_peered":"2024-09-13T14:41:16.493791+0000","last_unstale":"2024-09-13T14:41:48.783966+0000","last_undegraded":"2024-09-13T14:41:48.783966+0000","last_fullsized":"2024-09-13T14:41:48.783966+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1098,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:23.257279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:23.257279+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:04:29.628179+0000","scrub_duration":0.010350875000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":1068,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.788544+0000","last_change":"2024-09-13T14:29:42.539455+0000","last_active":"2024-09-13T14:41:48.788544+0000","last_peered":"2024-09-13T14:41:48.788544+0000","last_clean":"2024-09-13T14:41:48.788544+0000","last_became_active":"2024-09-13T14:29:42.539068+0000","last_became_peered":"2024-09-13T14:29:42.539068+0000","last_unstale":"2024-09-13T14:41:48.788544+0000","last_undegraded":"2024-09-13T14:41:48.788544+0000","last_fullsized":"2024-09-13T14:41:48.788544+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:24:06.048855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034405999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":579,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.788107+0000","last_change":"2024-09-13T14:39:11.382555+0000","last_active":"2024-09-13T14:41:48.788107+0000","last_peered":"2024-09-13T14:41:48.788107+0000","last_clean":"2024-09-13T14:41:48.788107+0000","last_became_active":"2024-09-13T14:39:11.382247+0000","last_became_peered":"2024-09-13T14:39:11.382247+0000","last_unstale":"2024-09-13T14:41:48.788107+0000","last_undegraded":"2024-09-13T14:41:48.788107+0000","last_fullsized":"2024-09-13T14:41:48.788107+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:09:48.016987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1031,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.787024+0000","last_change":"2024-09-13T14:19:27.379312+0000","last_active":"2024-09-13T14:41:48.787024+0000","last_peered":"2024-09-13T14:41:48.787024+0000","last_clean":"2024-09-13T14:41:48.787024+0000","last_became_active":"2024-09-13T14:19:27.378947+0000","last_became_peered":"2024-09-13T14:19:27.378947+0000","last_unstale":"2024-09-13T14:41:48.787024+0000","last_undegraded":"2024-09-13T14:41:48.787024+0000","last_fullsized":"2024-09-13T14:41:48.787024+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:54:20.706578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"261'100","reported_seq":341,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.786353+0000","last_change":"2024-09-13T14:41:28.592476+0000","last_active":"2024-09-13T14:41:48.786353+0000","last_peered":"2024-09-13T14:41:48.786353+0000","last_clean":"2024-09-13T14:41:48.786353+0000","last_became_active":"2024-09-13T14:41:28.592111+0000","last_became_peered":"2024-09-13T14:41:28.592111+0000","last_unstale":"2024-09-13T14:41:48.786353+0000","last_undegraded":"2024-09-13T14:41:48.786353+0000","last_fullsized":"2024-09-13T14:41:48.786353+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:26.747497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:26.747497+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":68,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"1111'11066","reported_seq":17298,"reported_epoch":1112,"state":"peering","last_fresh":"2024-09-13T14:41:51.821810+0000","last_change":"2024-09-13T14:41:51.822268+0000","last_active":"2024-09-13T14:41:51.821781+0000","last_peered":"2024-09-13T14:41:51.758739+0000","last_clean":"2024-09-13T14:41:51.758739+0000","last_became_active":"2024-09-13T14:41:46.818474+0000","last_became_peered":"2024-09-13T14:41:46.818474+0000","last_unstale":"2024-09-13T14:41:51.821810+0000","last_undegraded":"2024-09-13T14:41:51.821810+0000","last_fullsized":"2024-09-13T14:41:51.821810+0000","mapping_epoch":1112,"log_start":"1088'4255","ondisk_log_start":"1088'4255","created":1082,"last_epoch_clean":1111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6811,"log_dups_size":0,"ondisk_log_size":6811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186646528,"num_objects":2848,"num_object_clones":0,"num_object_copies":8544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5581,"num_write_kb":269728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2126,"num_bytes_recovered":139362304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4,6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1105,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.783729+0000","last_change":"2024-09-13T14:41:28.587183+0000","last_active":"2024-09-13T14:41:48.783729+0000","last_peered":"2024-09-13T14:41:48.783729+0000","last_clean":"2024-09-13T14:41:48.783729+0000","last_became_active":"2024-09-13T14:41:28.586268+0000","last_became_peered":"2024-09-13T14:41:28.586268+0000","last_unstale":"2024-09-13T14:41:48.783729+0000","last_undegraded":"2024-09-13T14:41:48.783729+0000","last_fullsized":"2024-09-13T14:41:48.783729+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:34.203731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1633,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.827309+0000","last_change":"2024-09-13T14:39:11.390055+0000","last_active":"2024-09-13T14:41:52.827309+0000","last_peered":"2024-09-13T14:41:52.827309+0000","last_clean":"2024-09-13T14:41:52.827309+0000","last_became_active":"2024-09-13T14:39:11.389697+0000","last_became_peered":"2024-09-13T14:39:11.389697+0000","last_unstale":"2024-09-13T14:41:52.827309+0000","last_undegraded":"2024-09-13T14:41:52.827309+0000","last_fullsized":"2024-09-13T14:41:52.827309+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:50.975739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1611,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.827626+0000","last_change":"2024-09-13T14:41:28.584816+0000","last_active":"2024-09-13T14:41:52.827626+0000","last_peered":"2024-09-13T14:41:52.827626+0000","last_clean":"2024-09-13T14:41:52.827626+0000","last_became_active":"2024-09-13T14:41:28.583332+0000","last_became_peered":"2024-09-13T14:41:28.583332+0000","last_unstale":"2024-09-13T14:41:52.827626+0000","last_undegraded":"2024-09-13T14:41:52.827626+0000","last_fullsized":"2024-09-13T14:41:52.827626+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:20.608574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"261'100","reported_seq":1212,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.783699+0000","last_change":"2024-09-13T14:41:24.245745+0000","last_active":"2024-09-13T14:41:48.783699+0000","last_peered":"2024-09-13T14:41:48.783699+0000","last_clean":"2024-09-13T14:41:48.783699+0000","last_became_active":"2024-09-13T14:41:20.515333+0000","last_became_peered":"2024-09-13T14:41:20.515333+0000","last_unstale":"2024-09-13T14:41:48.783699+0000","last_undegraded":"2024-09-13T14:41:48.783699+0000","last_fullsized":"2024-09-13T14:41:48.783699+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1100,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:24.245668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:24.245668+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:57:50.398918+0000","scrub_duration":0.0034439399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"1112'10648","reported_seq":10697,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.323072+0000","last_change":"2024-09-13T14:41:46.859677+0000","last_active":"2024-09-13T14:41:52.323072+0000","last_peered":"2024-09-13T14:41:52.323072+0000","last_clean":"2024-09-13T14:41:52.323072+0000","last_became_active":"2024-09-13T14:41:46.803818+0000","last_became_peered":"2024-09-13T14:41:46.803818+0000","last_unstale":"2024-09-13T14:41:52.323072+0000","last_undegraded":"2024-09-13T14:41:52.323072+0000","last_fullsized":"2024-09-13T14:41:52.323072+0000","mapping_epoch":1082,"log_start":"1090'4623","ondisk_log_start":"1090'4623","created":1082,"last_epoch_clean":1109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6025,"log_dups_size":0,"ondisk_log_size":6025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179699712,"num_objects":2742,"num_object_clones":0,"num_object_copies":8226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5383,"num_write_kb":260000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1609,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.832587+0000","last_change":"2024-09-13T14:41:28.585065+0000","last_active":"2024-09-13T14:41:52.832587+0000","last_peered":"2024-09-13T14:41:52.832587+0000","last_clean":"2024-09-13T14:41:52.832587+0000","last_became_active":"2024-09-13T14:41:28.584754+0000","last_became_peered":"2024-09-13T14:41:28.584754+0000","last_unstale":"2024-09-13T14:41:52.832587+0000","last_undegraded":"2024-09-13T14:41:52.832587+0000","last_fullsized":"2024-09-13T14:41:52.832587+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:20:35.484829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":563,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.832058+0000","last_change":"2024-09-13T14:41:28.590194+0000","last_active":"2024-09-13T14:41:52.832058+0000","last_peered":"2024-09-13T14:41:52.832058+0000","last_clean":"2024-09-13T14:41:52.832058+0000","last_became_active":"2024-09-13T14:41:28.589774+0000","last_became_peered":"2024-09-13T14:41:28.589774+0000","last_unstale":"2024-09-13T14:41:52.832058+0000","last_undegraded":"2024-09-13T14:41:52.832058+0000","last_fullsized":"2024-09-13T14:41:52.832058+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:36.382030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":509,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.789010+0000","last_change":"2024-09-13T14:41:28.597828+0000","last_active":"2024-09-13T14:41:48.789010+0000","last_peered":"2024-09-13T14:41:48.789010+0000","last_clean":"2024-09-13T14:41:48.789010+0000","last_became_active":"2024-09-13T14:41:28.597540+0000","last_became_peered":"2024-09-13T14:41:28.597540+0000","last_unstale":"2024-09-13T14:41:48.789010+0000","last_undegraded":"2024-09-13T14:41:48.789010+0000","last_fullsized":"2024-09-13T14:41:48.789010+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:49:04.992589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"261'100","reported_seq":32,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.787651+0000","last_change":"2024-09-13T14:41:23.561816+0000","last_active":"2024-09-13T14:41:48.787651+0000","last_peered":"2024-09-13T14:41:48.787651+0000","last_clean":"2024-09-13T14:41:48.787651+0000","last_became_active":"2024-09-13T14:41:20.509730+0000","last_became_peered":"2024-09-13T14:41:20.509730+0000","last_unstale":"2024-09-13T14:41:48.787651+0000","last_undegraded":"2024-09-13T14:41:48.787651+0000","last_fullsized":"2024-09-13T14:41:48.787651+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1100,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:23.561736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:23.561736+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:00.213136+0000","scrub_duration":0.0091806160000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"1113'11000","reported_seq":11050,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:57.289353+0000","last_change":"2024-09-13T14:41:46.873839+0000","last_active":"2024-09-13T14:41:57.289353+0000","last_peered":"2024-09-13T14:41:57.289353+0000","last_clean":"2024-09-13T14:41:57.289353+0000","last_became_active":"2024-09-13T14:41:46.861555+0000","last_became_peered":"2024-09-13T14:41:46.861555+0000","last_unstale":"2024-09-13T14:41:57.289353+0000","last_undegraded":"2024-09-13T14:41:57.289353+0000","last_fullsized":"2024-09-13T14:41:57.289353+0000","mapping_epoch":1082,"log_start":"1088'4134","ondisk_log_start":"1088'4134","created":1082,"last_epoch_clean":1109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6866,"log_dups_size":0,"ondisk_log_size":6866,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170950656,"num_objects":2608,"num_object_clones":0,"num_object_copies":7824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5632,"num_write_kb":263712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":1086,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.788254+0000","last_change":"2024-09-13T14:41:28.604178+0000","last_active":"2024-09-13T14:41:48.788254+0000","last_peered":"2024-09-13T14:41:48.788254+0000","last_clean":"2024-09-13T14:41:48.788254+0000","last_became_active":"2024-09-13T14:41:28.587558+0000","last_became_peered":"2024-09-13T14:41:28.587558+0000","last_unstale":"2024-09-13T14:41:48.788254+0000","last_undegraded":"2024-09-13T14:41:48.788254+0000","last_fullsized":"2024-09-13T14:41:48.788254+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:04:42.789023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1054,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-09-13T14:41:51.827181+0000","last_change":"2024-09-13T14:28:35.604404+0000","last_active":"2024-09-13T14:41:51.827181+0000","last_peered":"2024-09-13T14:41:51.827181+0000","last_clean":"2024-09-13T14:41:51.827181+0000","last_became_active":"2024-09-13T14:28:35.604102+0000","last_became_peered":"2024-09-13T14:28:35.604102+0000","last_unstale":"2024-09-13T14:41:51.827181+0000","last_undegraded":"2024-09-13T14:41:51.827181+0000","last_fullsized":"2024-09-13T14:41:51.827181+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:25.546010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1544,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.835439+0000","last_change":"2024-09-13T14:41:28.597520+0000","last_active":"2024-09-13T14:41:52.835439+0000","last_peered":"2024-09-13T14:41:52.835439+0000","last_clean":"2024-09-13T14:41:52.835439+0000","last_became_active":"2024-09-13T14:41:28.597166+0000","last_became_peered":"2024-09-13T14:41:28.597166+0000","last_unstale":"2024-09-13T14:41:52.835439+0000","last_undegraded":"2024-09-13T14:41:52.835439+0000","last_fullsized":"2024-09-13T14:41:52.835439+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:18:57.954799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"261'100","reported_seq":344,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.787171+0000","last_change":"2024-09-13T14:41:20.544283+0000","last_active":"2024-09-13T14:41:48.787171+0000","last_peered":"2024-09-13T14:41:48.787171+0000","last_clean":"2024-09-13T14:41:48.787171+0000","last_became_active":"2024-09-13T14:41:20.543630+0000","last_became_peered":"2024-09-13T14:41:20.543630+0000","last_unstale":"2024-09-13T14:41:48.787171+0000","last_undegraded":"2024-09-13T14:41:48.787171+0000","last_fullsized":"2024-09-13T14:41:48.787171+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1100,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:15:42.069756+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"1113'11050","reported_seq":11138,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.808281+0000","last_change":"2024-09-13T14:41:52.921963+0000","last_active":"2024-09-13T14:41:53.808281+0000","last_peered":"2024-09-13T14:41:53.808281+0000","last_clean":"2024-09-13T14:41:53.808281+0000","last_became_active":"2024-09-13T14:41:46.802094+0000","last_became_peered":"2024-09-13T14:41:46.802094+0000","last_unstale":"2024-09-13T14:41:53.808281+0000","last_undegraded":"2024-09-13T14:41:53.808281+0000","last_fullsized":"2024-09-13T14:41:53.808281+0000","mapping_epoch":1082,"log_start":"1088'4216","ondisk_log_start":"1088'4216","created":1082,"last_epoch_clean":1109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6834,"log_dups_size":0,"ondisk_log_size":6834,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T14:41:57.915547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178126848,"num_objects":2718,"num_object_clones":0,"num_object_copies":8154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5608,"num_write_kb":266432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1082,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.787138+0000","last_change":"2024-09-13T14:39:11.383045+0000","last_active":"2024-09-13T14:41:48.787138+0000","last_peered":"2024-09-13T14:41:48.787138+0000","last_clean":"2024-09-13T14:41:48.787138+0000","last_became_active":"2024-09-13T14:39:11.382718+0000","last_became_peered":"2024-09-13T14:39:11.382718+0000","last_unstale":"2024-09-13T14:41:48.787138+0000","last_undegraded":"2024-09-13T14:41:48.787138+0000","last_fullsized":"2024-09-13T14:41:48.787138+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:21.812296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":544,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.827109+0000","last_change":"2024-09-13T14:41:28.585118+0000","last_active":"2024-09-13T14:41:52.827109+0000","last_peered":"2024-09-13T14:41:52.827109+0000","last_clean":"2024-09-13T14:41:52.827109+0000","last_became_active":"2024-09-13T14:41:28.584804+0000","last_became_peered":"2024-09-13T14:41:28.584804+0000","last_unstale":"2024-09-13T14:41:52.827109+0000","last_undegraded":"2024-09-13T14:41:52.827109+0000","last_fullsized":"2024-09-13T14:41:52.827109+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:10:27.164531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":475,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.788172+0000","last_change":"2024-09-13T14:41:28.588613+0000","last_active":"2024-09-13T14:41:48.788172+0000","last_peered":"2024-09-13T14:41:48.788172+0000","last_clean":"2024-09-13T14:41:48.788172+0000","last_became_active":"2024-09-13T14:41:28.587982+0000","last_became_peered":"2024-09-13T14:41:28.587982+0000","last_unstale":"2024-09-13T14:41:48.788172+0000","last_undegraded":"2024-09-13T14:41:48.788172+0000","last_fullsized":"2024-09-13T14:41:48.788172+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:51:16.814445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"261'100","reported_seq":354,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.787022+0000","last_change":"2024-09-13T14:41:25.289063+0000","last_active":"2024-09-13T14:41:48.787022+0000","last_peered":"2024-09-13T14:41:48.787022+0000","last_clean":"2024-09-13T14:41:48.787022+0000","last_became_active":"2024-09-13T14:41:20.526418+0000","last_became_peered":"2024-09-13T14:41:20.526418+0000","last_unstale":"2024-09-13T14:41:48.787022+0000","last_undegraded":"2024-09-13T14:41:48.787022+0000","last_fullsized":"2024-09-13T14:41:48.787022+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1100,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:25.288979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:25.288979+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:55:24.275123+0000","scrub_duration":0.0040379329999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"1112'10724","reported_seq":10957,"reported_epoch":1112,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:41:52.244831+0000","last_change":"2024-09-13T14:41:46.912020+0000","last_active":"2024-09-13T14:41:52.244831+0000","last_peered":"2024-09-13T14:41:52.244831+0000","last_clean":"2024-09-13T14:41:27.528697+0000","last_became_active":"2024-09-13T14:41:46.899012+0000","last_became_peered":"2024-09-13T14:41:46.899012+0000","last_unstale":"2024-09-13T14:41:52.244831+0000","last_undegraded":"2024-09-13T14:41:52.244831+0000","last_fullsized":"2024-09-13T14:41:52.244831+0000","mapping_epoch":1103,"log_start":"1090'5198","ondisk_log_start":"1090'5198","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":5526,"log_dups_size":0,"ondisk_log_size":5526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178454528,"num_objects":2723,"num_object_clones":0,"num_object_copies":8169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2723,"num_objects_unfound":0,"num_objects_dirty":2723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5434,"num_write_kb":261024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":2424832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,1],"avail_no_missing":["6","1","2"],"object_location_counts":[{"shards":"1,2,6","objects":2723}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":563,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.835228+0000","last_change":"2024-09-13T14:41:28.593821+0000","last_active":"2024-09-13T14:41:52.835228+0000","last_peered":"2024-09-13T14:41:52.835228+0000","last_clean":"2024-09-13T14:41:52.835228+0000","last_became_active":"2024-09-13T14:41:28.593183+0000","last_became_peered":"2024-09-13T14:41:28.593183+0000","last_unstale":"2024-09-13T14:41:52.835228+0000","last_undegraded":"2024-09-13T14:41:52.835228+0000","last_fullsized":"2024-09-13T14:41:52.835228+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:54.526707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1078,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.784568+0000","last_change":"2024-09-13T14:28:35.603663+0000","last_active":"2024-09-13T14:41:48.784568+0000","last_peered":"2024-09-13T14:41:48.784568+0000","last_clean":"2024-09-13T14:41:48.784568+0000","last_became_active":"2024-09-13T14:28:35.603358+0000","last_became_peered":"2024-09-13T14:28:35.603358+0000","last_unstale":"2024-09-13T14:41:48.784568+0000","last_undegraded":"2024-09-13T14:41:48.784568+0000","last_fullsized":"2024-09-13T14:41:48.784568+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:24:05.842796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1582,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.826913+0000","last_change":"2024-09-13T14:28:35.611508+0000","last_active":"2024-09-13T14:41:52.826913+0000","last_peered":"2024-09-13T14:41:52.826913+0000","last_clean":"2024-09-13T14:41:52.826913+0000","last_became_active":"2024-09-13T14:28:35.610106+0000","last_became_peered":"2024-09-13T14:28:35.610106+0000","last_unstale":"2024-09-13T14:41:52.826913+0000","last_undegraded":"2024-09-13T14:41:52.826913+0000","last_fullsized":"2024-09-13T14:41:52.826913+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:11.966958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"261'100","reported_seq":33,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-09-13T14:41:51.827529+0000","last_change":"2024-09-13T14:41:22.893041+0000","last_active":"2024-09-13T14:41:51.827529+0000","last_peered":"2024-09-13T14:41:51.827529+0000","last_clean":"2024-09-13T14:41:51.827529+0000","last_became_active":"2024-09-13T14:41:20.530028+0000","last_became_peered":"2024-09-13T14:41:20.530028+0000","last_unstale":"2024-09-13T14:41:51.827529+0000","last_undegraded":"2024-09-13T14:41:51.827529+0000","last_fullsized":"2024-09-13T14:41:51.827529+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1100,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:22.892963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:22.892963+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:32:52.347814+0000","scrub_duration":0.017451785000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"1113'11114","reported_seq":11162,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:57.280667+0000","last_change":"2024-09-13T14:41:46.841786+0000","last_active":"2024-09-13T14:41:57.280667+0000","last_peered":"2024-09-13T14:41:57.280667+0000","last_clean":"2024-09-13T14:41:57.280667+0000","last_became_active":"2024-09-13T14:41:46.809092+0000","last_became_peered":"2024-09-13T14:41:46.809092+0000","last_unstale":"2024-09-13T14:41:57.280667+0000","last_undegraded":"2024-09-13T14:41:57.280667+0000","last_fullsized":"2024-09-13T14:41:57.280667+0000","mapping_epoch":1082,"log_start":"1088'4300","ondisk_log_start":"1088'4300","created":1082,"last_epoch_clean":1109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6814,"log_dups_size":0,"ondisk_log_size":6814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171769856,"num_objects":2621,"num_object_clones":0,"num_object_copies":7863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5690,"num_write_kb":265952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":1542,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.821922+0000","last_change":"2024-09-13T14:30:15.581728+0000","last_active":"2024-09-13T14:41:52.821922+0000","last_peered":"2024-09-13T14:41:52.821922+0000","last_clean":"2024-09-13T14:41:52.821922+0000","last_became_active":"2024-09-13T14:30:15.581303+0000","last_became_peered":"2024-09-13T14:30:15.581303+0000","last_unstale":"2024-09-13T14:41:52.821922+0000","last_undegraded":"2024-09-13T14:41:52.821922+0000","last_fullsized":"2024-09-13T14:41:52.821922+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:10.027991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":485,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.787175+0000","last_change":"2024-09-13T14:41:28.592508+0000","last_active":"2024-09-13T14:41:48.787175+0000","last_peered":"2024-09-13T14:41:48.787175+0000","last_clean":"2024-09-13T14:41:48.787175+0000","last_became_active":"2024-09-13T14:41:28.590027+0000","last_became_peered":"2024-09-13T14:41:28.590027+0000","last_unstale":"2024-09-13T14:41:48.787175+0000","last_undegraded":"2024-09-13T14:41:48.787175+0000","last_fullsized":"2024-09-13T14:41:48.787175+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:30:39.262649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":619,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.783549+0000","last_change":"2024-09-13T14:41:28.587360+0000","last_active":"2024-09-13T14:41:48.783549+0000","last_peered":"2024-09-13T14:41:48.783549+0000","last_clean":"2024-09-13T14:41:48.783549+0000","last_became_active":"2024-09-13T14:41:28.586655+0000","last_became_peered":"2024-09-13T14:41:28.586655+0000","last_unstale":"2024-09-13T14:41:48.783549+0000","last_undegraded":"2024-09-13T14:41:48.783549+0000","last_fullsized":"2024-09-13T14:41:48.783549+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:21:51.249946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"261'100","reported_seq":36,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.789242+0000","last_change":"2024-09-13T14:41:22.580296+0000","last_active":"2024-09-13T14:41:48.789242+0000","last_peered":"2024-09-13T14:41:48.789242+0000","last_clean":"2024-09-13T14:41:48.789242+0000","last_became_active":"2024-09-13T14:41:20.530363+0000","last_became_peered":"2024-09-13T14:41:20.530363+0000","last_unstale":"2024-09-13T14:41:48.789242+0000","last_undegraded":"2024-09-13T14:41:48.789242+0000","last_fullsized":"2024-09-13T14:41:48.789242+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1100,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:22.580245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:22.580245+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:03.428657+0000","scrub_duration":0.001821837,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"1112'10586","reported_seq":10634,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.357009+0000","last_change":"2024-09-13T14:41:46.856515+0000","last_active":"2024-09-13T14:41:52.357009+0000","last_peered":"2024-09-13T14:41:52.357009+0000","last_clean":"2024-09-13T14:41:52.357009+0000","last_became_active":"2024-09-13T14:41:46.805366+0000","last_became_peered":"2024-09-13T14:41:46.805366+0000","last_unstale":"2024-09-13T14:41:52.357009+0000","last_undegraded":"2024-09-13T14:41:52.357009+0000","last_fullsized":"2024-09-13T14:41:52.357009+0000","mapping_epoch":1082,"log_start":"1089'4531","ondisk_log_start":"1089'4531","created":1082,"last_epoch_clean":1109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6055,"log_dups_size":0,"ondisk_log_size":6055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179896320,"num_objects":2745,"num_object_clones":0,"num_object_copies":8235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5358,"num_write_kb":259296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":1618,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.827804+0000","last_change":"2024-09-13T14:41:28.598014+0000","last_active":"2024-09-13T14:41:52.827804+0000","last_peered":"2024-09-13T14:41:52.827804+0000","last_clean":"2024-09-13T14:41:52.827804+0000","last_became_active":"2024-09-13T14:41:28.585268+0000","last_became_peered":"2024-09-13T14:41:28.585268+0000","last_unstale":"2024-09-13T14:41:52.827804+0000","last_undegraded":"2024-09-13T14:41:52.827804+0000","last_fullsized":"2024-09-13T14:41:52.827804+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:40.047685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1062,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.787307+0000","last_change":"2024-09-13T14:29:42.538860+0000","last_active":"2024-09-13T14:41:48.787307+0000","last_peered":"2024-09-13T14:41:48.787307+0000","last_clean":"2024-09-13T14:41:48.787307+0000","last_became_active":"2024-09-13T14:29:42.538608+0000","last_became_peered":"2024-09-13T14:29:42.538608+0000","last_unstale":"2024-09-13T14:41:48.787307+0000","last_undegraded":"2024-09-13T14:41:48.787307+0000","last_fullsized":"2024-09-13T14:41:48.787307+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:20.070703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1036,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-09-13T14:41:51.827108+0000","last_change":"2024-09-13T14:19:26.131609+0000","last_active":"2024-09-13T14:41:51.827108+0000","last_peered":"2024-09-13T14:41:51.827108+0000","last_clean":"2024-09-13T14:41:51.827108+0000","last_became_active":"2024-09-13T14:19:26.131029+0000","last_became_peered":"2024-09-13T14:19:26.131029+0000","last_unstale":"2024-09-13T14:41:51.827108+0000","last_undegraded":"2024-09-13T14:41:51.827108+0000","last_fullsized":"2024-09-13T14:41:51.827108+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:57.933696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"1112'10701","reported_seq":10936,"reported_epoch":1112,"state":"active+recovery_wait+remapped","last_fresh":"2024-09-13T14:41:52.208385+0000","last_change":"2024-09-13T14:41:48.859115+0000","last_active":"2024-09-13T14:41:52.208385+0000","last_peered":"2024-09-13T14:41:52.208385+0000","last_clean":"2024-09-13T14:41:27.528697+0000","last_became_active":"2024-09-13T14:41:48.834530+0000","last_became_peered":"2024-09-13T14:41:48.834530+0000","last_unstale":"2024-09-13T14:41:52.208385+0000","last_undegraded":"2024-09-13T14:41:52.208385+0000","last_fullsized":"2024-09-13T14:41:52.208385+0000","mapping_epoch":1110,"log_start":"1090'5198","ondisk_log_start":"1090'5198","created":1108,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":5503,"log_dups_size":0,"ondisk_log_size":5503,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179961856,"num_objects":2746,"num_object_clones":0,"num_object_copies":8238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2746,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5411,"num_write_kb":261024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":2424832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,1],"avail_no_missing":["6","1","2"],"object_location_counts":[{"shards":"1,2,6","objects":2746}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1063,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-09-13T14:41:51.827103+0000","last_change":"2024-09-13T14:41:28.593455+0000","last_active":"2024-09-13T14:41:51.827103+0000","last_peered":"2024-09-13T14:41:51.827103+0000","last_clean":"2024-09-13T14:41:51.827103+0000","last_became_active":"2024-09-13T14:41:28.593139+0000","last_became_peered":"2024-09-13T14:41:28.593139+0000","last_unstale":"2024-09-13T14:41:51.827103+0000","last_undegraded":"2024-09-13T14:41:51.827103+0000","last_fullsized":"2024-09-13T14:41:51.827103+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:37.427352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":1059,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-09-13T14:41:51.828405+0000","last_change":"2024-09-13T14:19:25.870605+0000","last_active":"2024-09-13T14:41:51.828405+0000","last_peered":"2024-09-13T14:41:51.828405+0000","last_clean":"2024-09-13T14:41:51.828405+0000","last_became_active":"2024-09-13T14:19:25.870282+0000","last_became_peered":"2024-09-13T14:19:25.870282+0000","last_unstale":"2024-09-13T14:41:51.828405+0000","last_undegraded":"2024-09-13T14:41:51.828405+0000","last_fullsized":"2024-09-13T14:41:51.828405+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:52.514915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"1113'11044","reported_seq":11133,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.804382+0000","last_change":"2024-09-13T14:41:48.309229+0000","last_active":"2024-09-13T14:41:53.804382+0000","last_peered":"2024-09-13T14:41:53.804382+0000","last_clean":"2024-09-13T14:41:53.804382+0000","last_became_active":"2024-09-13T14:41:46.825752+0000","last_became_peered":"2024-09-13T14:41:46.825752+0000","last_unstale":"2024-09-13T14:41:53.804382+0000","last_undegraded":"2024-09-13T14:41:53.804382+0000","last_fullsized":"2024-09-13T14:41:53.804382+0000","mapping_epoch":1108,"log_start":"1088'4216","ondisk_log_start":"1088'4216","created":1108,"last_epoch_clean":1109,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6828,"log_dups_size":0,"ondisk_log_size":6828,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178520064,"num_objects":2724,"num_object_clones":0,"num_object_copies":8172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5602,"num_write_kb":266432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":553,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.786663+0000","last_change":"2024-09-13T14:41:28.593526+0000","last_active":"2024-09-13T14:41:48.786663+0000","last_peered":"2024-09-13T14:41:48.786663+0000","last_clean":"2024-09-13T14:41:48.786663+0000","last_became_active":"2024-09-13T14:41:28.593217+0000","last_became_peered":"2024-09-13T14:41:28.593217+0000","last_unstale":"2024-09-13T14:41:48.786663+0000","last_undegraded":"2024-09-13T14:41:48.786663+0000","last_fullsized":"2024-09-13T14:41:48.786663+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:57:01.502925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":1069,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.784197+0000","last_change":"2024-09-13T14:30:15.579260+0000","last_active":"2024-09-13T14:41:48.784197+0000","last_peered":"2024-09-13T14:41:48.784197+0000","last_clean":"2024-09-13T14:41:48.784197+0000","last_became_active":"2024-09-13T14:30:15.578897+0000","last_became_peered":"2024-09-13T14:30:15.578897+0000","last_unstale":"2024-09-13T14:41:48.784197+0000","last_undegraded":"2024-09-13T14:41:48.784197+0000","last_fullsized":"2024-09-13T14:41:48.784197+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:56.737849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"1112'10586","reported_seq":10633,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.403817+0000","last_change":"2024-09-13T14:41:46.905783+0000","last_active":"2024-09-13T14:41:52.403817+0000","last_peered":"2024-09-13T14:41:52.403817+0000","last_clean":"2024-09-13T14:41:52.403817+0000","last_became_active":"2024-09-13T14:41:46.887125+0000","last_became_peered":"2024-09-13T14:41:46.887125+0000","last_unstale":"2024-09-13T14:41:52.403817+0000","last_undegraded":"2024-09-13T14:41:52.403817+0000","last_fullsized":"2024-09-13T14:41:52.403817+0000","mapping_epoch":1108,"log_start":"1089'4531","ondisk_log_start":"1089'4531","created":1108,"last_epoch_clean":1109,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6055,"log_dups_size":0,"ondisk_log_size":6055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179896320,"num_objects":2745,"num_object_clones":0,"num_object_copies":8235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5358,"num_write_kb":259296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1065,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.787140+0000","last_change":"2024-09-13T14:41:28.591640+0000","last_active":"2024-09-13T14:41:48.787140+0000","last_peered":"2024-09-13T14:41:48.787140+0000","last_clean":"2024-09-13T14:41:48.787140+0000","last_became_active":"2024-09-13T14:41:28.591354+0000","last_became_peered":"2024-09-13T14:41:28.591354+0000","last_unstale":"2024-09-13T14:41:48.787140+0000","last_undegraded":"2024-09-13T14:41:48.787140+0000","last_fullsized":"2024-09-13T14:41:48.787140+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:53:38.555537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1510,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.821563+0000","last_change":"2024-09-13T14:30:15.576921+0000","last_active":"2024-09-13T14:41:52.821563+0000","last_peered":"2024-09-13T14:41:52.821563+0000","last_clean":"2024-09-13T14:41:52.821563+0000","last_became_active":"2024-09-13T14:30:15.576607+0000","last_became_peered":"2024-09-13T14:30:15.576607+0000","last_unstale":"2024-09-13T14:41:52.821563+0000","last_undegraded":"2024-09-13T14:41:52.821563+0000","last_fullsized":"2024-09-13T14:41:52.821563+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:35.162460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"1113'11078","reported_seq":11125,"reported_epoch":1113,"state":"active+recovery_wait","last_fresh":"2024-09-13T14:41:57.204242+0000","last_change":"2024-09-13T14:41:46.917041+0000","last_active":"2024-09-13T14:41:57.204242+0000","last_peered":"2024-09-13T14:41:57.204242+0000","last_clean":"2024-09-13T14:41:45.747357+0000","last_became_active":"2024-09-13T14:41:46.879995+0000","last_became_peered":"2024-09-13T14:41:46.879995+0000","last_unstale":"2024-09-13T14:41:57.204242+0000","last_undegraded":"2024-09-13T14:41:57.204242+0000","last_fullsized":"2024-09-13T14:41:57.204242+0000","mapping_epoch":1108,"log_start":"1088'4300","ondisk_log_start":"1088'4300","created":1108,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6778,"log_dups_size":0,"ondisk_log_size":6778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174129152,"num_objects":2657,"num_object_clones":0,"num_object_copies":7971,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5654,"num_write_kb":265952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":["4","2","6"],"object_location_counts":[{"shards":"2,4,6","objects":2657}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1632,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.827908+0000","last_change":"2024-09-13T14:39:11.386005+0000","last_active":"2024-09-13T14:41:52.827908+0000","last_peered":"2024-09-13T14:41:52.827908+0000","last_clean":"2024-09-13T14:41:52.827908+0000","last_became_active":"2024-09-13T14:39:11.385691+0000","last_became_peered":"2024-09-13T14:39:11.385691+0000","last_unstale":"2024-09-13T14:41:52.827908+0000","last_undegraded":"2024-09-13T14:41:52.827908+0000","last_fullsized":"2024-09-13T14:41:52.827908+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:11:03.371484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":599,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.785758+0000","last_change":"2024-09-13T14:41:28.592154+0000","last_active":"2024-09-13T14:41:48.785758+0000","last_peered":"2024-09-13T14:41:48.785758+0000","last_clean":"2024-09-13T14:41:48.785758+0000","last_became_active":"2024-09-13T14:41:28.590786+0000","last_became_peered":"2024-09-13T14:41:28.590786+0000","last_unstale":"2024-09-13T14:41:48.785758+0000","last_undegraded":"2024-09-13T14:41:48.785758+0000","last_fullsized":"2024-09-13T14:41:48.785758+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:49:59.861513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1058,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.787306+0000","last_change":"2024-09-13T14:41:28.594659+0000","last_active":"2024-09-13T14:41:48.787306+0000","last_peered":"2024-09-13T14:41:48.787306+0000","last_clean":"2024-09-13T14:41:48.787306+0000","last_became_active":"2024-09-13T14:41:28.588903+0000","last_became_peered":"2024-09-13T14:41:28.588903+0000","last_unstale":"2024-09-13T14:41:48.787306+0000","last_undegraded":"2024-09-13T14:41:48.787306+0000","last_fullsized":"2024-09-13T14:41:48.787306+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:45:51.783371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":661,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.826738+0000","last_change":"2024-09-13T14:41:28.585670+0000","last_active":"2024-09-13T14:41:52.826738+0000","last_peered":"2024-09-13T14:41:52.826738+0000","last_clean":"2024-09-13T14:41:52.826738+0000","last_became_active":"2024-09-13T14:41:28.585173+0000","last_became_peered":"2024-09-13T14:41:28.585173+0000","last_unstale":"2024-09-13T14:41:52.826738+0000","last_undegraded":"2024-09-13T14:41:52.826738+0000","last_fullsized":"2024-09-13T14:41:52.826738+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:26:08.019099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1607,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.827713+0000","last_change":"2024-09-13T14:41:28.583750+0000","last_active":"2024-09-13T14:41:52.827713+0000","last_peered":"2024-09-13T14:41:52.827713+0000","last_clean":"2024-09-13T14:41:52.827713+0000","last_became_active":"2024-09-13T14:41:28.583293+0000","last_became_peered":"2024-09-13T14:41:28.583293+0000","last_unstale":"2024-09-13T14:41:52.827713+0000","last_undegraded":"2024-09-13T14:41:52.827713+0000","last_fullsized":"2024-09-13T14:41:52.827713+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:41:05.647080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":628,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.821537+0000","last_change":"2024-09-13T14:41:28.590716+0000","last_active":"2024-09-13T14:41:52.821537+0000","last_peered":"2024-09-13T14:41:52.821537+0000","last_clean":"2024-09-13T14:41:52.821537+0000","last_became_active":"2024-09-13T14:41:28.590372+0000","last_became_peered":"2024-09-13T14:41:28.590372+0000","last_unstale":"2024-09-13T14:41:52.821537+0000","last_undegraded":"2024-09-13T14:41:52.821537+0000","last_fullsized":"2024-09-13T14:41:52.821537+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:39.348905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":581,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-09-13T14:41:51.827395+0000","last_change":"2024-09-13T14:41:28.593532+0000","last_active":"2024-09-13T14:41:51.827395+0000","last_peered":"2024-09-13T14:41:51.827395+0000","last_clean":"2024-09-13T14:41:51.827395+0000","last_became_active":"2024-09-13T14:41:28.593235+0000","last_became_peered":"2024-09-13T14:41:28.593235+0000","last_unstale":"2024-09-13T14:41:51.827395+0000","last_undegraded":"2024-09-13T14:41:51.827395+0000","last_fullsized":"2024-09-13T14:41:51.827395+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:30:22.224415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":1082,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-09-13T14:41:51.827287+0000","last_change":"2024-09-13T14:41:28.590173+0000","last_active":"2024-09-13T14:41:51.827287+0000","last_peered":"2024-09-13T14:41:51.827287+0000","last_clean":"2024-09-13T14:41:51.827287+0000","last_became_active":"2024-09-13T14:41:28.589347+0000","last_became_peered":"2024-09-13T14:41:28.589347+0000","last_unstale":"2024-09-13T14:41:51.827287+0000","last_undegraded":"2024-09-13T14:41:51.827287+0000","last_fullsized":"2024-09-13T14:41:51.827287+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:19.873187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1592,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.835339+0000","last_change":"2024-09-13T14:41:28.584508+0000","last_active":"2024-09-13T14:41:52.835339+0000","last_peered":"2024-09-13T14:41:52.835339+0000","last_clean":"2024-09-13T14:41:52.835339+0000","last_became_active":"2024-09-13T14:41:28.584187+0000","last_became_peered":"2024-09-13T14:41:28.584187+0000","last_unstale":"2024-09-13T14:41:52.835339+0000","last_undegraded":"2024-09-13T14:41:52.835339+0000","last_fullsized":"2024-09-13T14:41:52.835339+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:21.364817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":1584,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.827006+0000","last_change":"2024-09-13T14:28:35.612469+0000","last_active":"2024-09-13T14:41:52.827006+0000","last_peered":"2024-09-13T14:41:52.827006+0000","last_clean":"2024-09-13T14:41:52.827006+0000","last_became_active":"2024-09-13T14:28:35.612093+0000","last_became_peered":"2024-09-13T14:28:35.612093+0000","last_unstale":"2024-09-13T14:41:52.827006+0000","last_undegraded":"2024-09-13T14:41:52.827006+0000","last_fullsized":"2024-09-13T14:41:52.827006+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:51.022663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"1113'10852","reported_seq":10937,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.794821+0000","last_change":"2024-09-13T14:41:48.978508+0000","last_active":"2024-09-13T14:41:53.794821+0000","last_peered":"2024-09-13T14:41:53.794821+0000","last_clean":"2024-09-13T14:41:53.794821+0000","last_became_active":"2024-09-13T14:41:46.908994+0000","last_became_peered":"2024-09-13T14:41:46.908994+0000","last_unstale":"2024-09-13T14:41:53.794821+0000","last_undegraded":"2024-09-13T14:41:53.794821+0000","last_fullsized":"2024-09-13T14:41:53.794821+0000","mapping_epoch":1108,"log_start":"1088'3982","ondisk_log_start":"1088'3982","created":1108,"last_epoch_clean":1109,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6870,"log_dups_size":0,"ondisk_log_size":6870,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T14:41:53.893112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181403648,"num_objects":2768,"num_object_clones":0,"num_object_copies":8304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5491,"num_write_kb":264288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1073,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-09-13T14:41:51.827102+0000","last_change":"2024-09-13T14:41:28.588664+0000","last_active":"2024-09-13T14:41:51.827102+0000","last_peered":"2024-09-13T14:41:51.827102+0000","last_clean":"2024-09-13T14:41:51.827102+0000","last_became_active":"2024-09-13T14:41:28.588283+0000","last_became_peered":"2024-09-13T14:41:28.588283+0000","last_unstale":"2024-09-13T14:41:51.827102+0000","last_undegraded":"2024-09-13T14:41:51.827102+0000","last_fullsized":"2024-09-13T14:41:51.827102+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:10:02.721554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":1071,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.789357+0000","last_change":"2024-09-13T14:41:28.603145+0000","last_active":"2024-09-13T14:41:48.789357+0000","last_peered":"2024-09-13T14:41:48.789357+0000","last_clean":"2024-09-13T14:41:48.789357+0000","last_became_active":"2024-09-13T14:41:28.590945+0000","last_became_peered":"2024-09-13T14:41:28.590945+0000","last_unstale":"2024-09-13T14:41:48.789357+0000","last_undegraded":"2024-09-13T14:41:48.789357+0000","last_fullsized":"2024-09-13T14:41:48.789357+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:01:21.700348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"1112'11040","reported_seq":11088,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.209338+0000","last_change":"2024-09-13T14:41:46.900621+0000","last_active":"2024-09-13T14:41:52.209338+0000","last_peered":"2024-09-13T14:41:52.209338+0000","last_clean":"2024-09-13T14:41:52.209338+0000","last_became_active":"2024-09-13T14:41:46.868571+0000","last_became_peered":"2024-09-13T14:41:46.868571+0000","last_unstale":"2024-09-13T14:41:52.209338+0000","last_undegraded":"2024-09-13T14:41:52.209338+0000","last_fullsized":"2024-09-13T14:41:52.209338+0000","mapping_epoch":1108,"log_start":"1090'4941","ondisk_log_start":"1090'4941","created":1108,"last_epoch_clean":1109,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6099,"log_dups_size":0,"ondisk_log_size":6099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187105280,"num_objects":2855,"num_object_clones":0,"num_object_copies":8565,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5580,"num_write_kb":269920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1070,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-09-13T14:41:51.827193+0000","last_change":"2024-09-13T14:30:15.580011+0000","last_active":"2024-09-13T14:41:51.827193+0000","last_peered":"2024-09-13T14:41:51.827193+0000","last_clean":"2024-09-13T14:41:51.827193+0000","last_became_active":"2024-09-13T14:30:15.579610+0000","last_became_peered":"2024-09-13T14:30:15.579610+0000","last_unstale":"2024-09-13T14:41:51.827193+0000","last_undegraded":"2024-09-13T14:41:51.827193+0000","last_fullsized":"2024-09-13T14:41:51.827193+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:11:02.733771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":1040,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.787064+0000","last_change":"2024-09-13T14:19:25.873514+0000","last_active":"2024-09-13T14:41:48.787064+0000","last_peered":"2024-09-13T14:41:48.787064+0000","last_clean":"2024-09-13T14:41:48.787064+0000","last_became_active":"2024-09-13T14:19:25.873170+0000","last_became_peered":"2024-09-13T14:19:25.873170+0000","last_unstale":"2024-09-13T14:41:48.787064+0000","last_undegraded":"2024-09-13T14:41:48.787064+0000","last_fullsized":"2024-09-13T14:41:48.787064+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:48.298530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":661,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.832496+0000","last_change":"2024-09-13T14:39:11.396732+0000","last_active":"2024-09-13T14:41:52.832496+0000","last_peered":"2024-09-13T14:41:52.832496+0000","last_clean":"2024-09-13T14:41:52.832496+0000","last_became_active":"2024-09-13T14:39:11.396322+0000","last_became_peered":"2024-09-13T14:39:11.396322+0000","last_unstale":"2024-09-13T14:41:52.832496+0000","last_undegraded":"2024-09-13T14:41:52.832496+0000","last_fullsized":"2024-09-13T14:41:52.832496+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:54:47.496762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":74,"reported_epoch":1111,"state":"active+clean","last_fresh":"2024-09-13T14:41:48.787107+0000","last_change":"2024-09-13T14:41:28.586563+0000","last_active":"2024-09-13T14:41:48.787107+0000","last_peered":"2024-09-13T14:41:48.787107+0000","last_clean":"2024-09-13T14:41:48.787107+0000","last_became_active":"2024-09-13T14:41:28.586152+0000","last_became_peered":"2024-09-13T14:41:28.586152+0000","last_unstale":"2024-09-13T14:41:48.787107+0000","last_undegraded":"2024-09-13T14:41:48.787107+0000","last_fullsized":"2024-09-13T14:41:48.787107+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:28.618035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1063,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-09-13T14:41:51.826658+0000","last_change":"2024-09-13T14:29:42.539907+0000","last_active":"2024-09-13T14:41:51.826658+0000","last_peered":"2024-09-13T14:41:51.826658+0000","last_clean":"2024-09-13T14:41:51.826658+0000","last_became_active":"2024-09-13T14:29:42.539583+0000","last_became_peered":"2024-09-13T14:29:42.539583+0000","last_unstale":"2024-09-13T14:41:51.826658+0000","last_undegraded":"2024-09-13T14:41:51.826658+0000","last_fullsized":"2024-09-13T14:41:51.826658+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:02:04.628447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":1599,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.827914+0000","last_change":"2024-09-13T14:29:42.541263+0000","last_active":"2024-09-13T14:41:52.827914+0000","last_peered":"2024-09-13T14:41:52.827914+0000","last_clean":"2024-09-13T14:41:52.827914+0000","last_became_active":"2024-09-13T14:29:42.540873+0000","last_became_peered":"2024-09-13T14:29:42.540873+0000","last_unstale":"2024-09-13T14:41:52.827914+0000","last_undegraded":"2024-09-13T14:41:52.827914+0000","last_fullsized":"2024-09-13T14:41:52.827914+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:20.611206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":26,"stat_sum":{"num_bytes":5741608984,"num_objects":87611,"num_object_clones":0,"num_object_copies":262833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":675,"num_objects_misplaced":21605,"num_objects_unfound":0,"num_objects_dirty":87611,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177301,"num_write_kb":8477121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6047,"num_bytes_recovered":396034048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17568706560,"data_stored":17568694344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":169558,"ondisk_log_size":169558,"up":78,"acting":77,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":488690,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":79,"num_read_kb":204,"num_write":40,"num_write_kb":444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":423698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1100,"ondisk_log_size":1100,"up":33,"acting":33,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430502,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2835548,"kb_used_data":2534036,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":90909604,"statfs":{"total":95995035648,"available":93091434496,"internally_reserved":0,"allocated":2594852864,"data_stored":2591591934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626731,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3882028,"kb_used_data":3572192,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":89863124,"statfs":{"total":95995035648,"available":92019838976,"internally_reserved":0,"allocated":3657924608,"data_stored":3654660282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822960,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2591124,"kb_used_data":2289612,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91154028,"statfs":{"total":95995035648,"available":93341724672,"internally_reserved":0,"allocated":2344562688,"data_stored":2341306217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019189,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2182456,"kb_used_data":1876720,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91562696,"statfs":{"total":95995035648,"available":93760200704,"internally_reserved":0,"allocated":1921761280,"data_stored":1918488521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059592846,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1142872,"kb_used_data":1070732,"kb_used_omap":26,"kb_used_meta":72101,"kb_avail":92602280,"statfs":{"total":95995035648,"available":94824734720,"internally_reserved":0,"allocated":1096429568,"data_stored":1093180447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":73831584},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":591,"seq":2538325672085,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2957356,"kb_used_data":2794788,"kb_used_omap":41,"kb_used_meta":162518,"kb_avail":90787796,"statfs":{"total":95995035648,"available":92966703104,"internally_reserved":0,"allocated":2861862912,"data_stored":2858598727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42651,"internal_metadata":166418789},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":2392296784028,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":524,"seq":2250562863266,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1526328,"kb_used_data":1426280,"kb_used_omap":41,"kb_used_meta":99990,"kb_avail":92218824,"statfs":{"total":95995035648,"available":94432075776,"internally_reserved":0,"allocated":1460510720,"data_stored":1457241942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42694,"internal_metadata":102390074},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1435308032,"data_stored":1435303960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1817120768,"data_stored":1817116696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2825650176,"data_stored":2825650176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1075642368,"data_stored":1075642368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1898643456,"data_stored":1898643456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2318602240,"data_stored":2318598168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3641114624,"data_stored":3641114624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2556624896,"data_stored":2556624896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:41:58.838 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-13T14:41:58.838 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 21 2024-09-13T14:41:58.849 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:58.877 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:58.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:58.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:58.979 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:59.016 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:59.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:59.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:59.079 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:58 smithi151 ceph-mon[104450]: pgmap v1395: 109 pgs: 1 active+recovery_wait+remapped, 1 peering, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 100 active+clean; 5.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 627 op/s; 675/262908 objects degraded (0.257%); 21605/262908 objects misplaced (8.218%); 9.8 MiB/s, 156 objects/s recovering 2024-09-13T14:41:59.079 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:58 smithi151 ceph-mon[104450]: from='client.45362 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:59.079 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:58 smithi151 ceph-mon[104450]: 6.0 scrub starts 2024-09-13T14:41:59.080 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:58 smithi151 ceph-mon[108966]: pgmap v1395: 109 pgs: 1 active+recovery_wait+remapped, 1 peering, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 100 active+clean; 5.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 627 op/s; 675/262908 objects degraded (0.257%); 21605/262908 objects misplaced (8.218%); 9.8 MiB/s, 156 objects/s recovering 2024-09-13T14:41:59.080 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:58 smithi151 ceph-mon[108966]: from='client.45362 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:59.080 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:58 smithi151 ceph-mon[108966]: 6.0 scrub starts 2024-09-13T14:41:59.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:58 smithi184 ceph-mon[92527]: pgmap v1395: 109 pgs: 1 active+recovery_wait+remapped, 1 peering, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 100 active+clean; 5.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 627 op/s; 675/262908 objects degraded (0.257%); 21605/262908 objects misplaced (8.218%); 9.8 MiB/s, 156 objects/s recovering 2024-09-13T14:41:59.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:58 smithi184 ceph-mon[92527]: from='client.45362 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:59.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:58 smithi184 ceph-mon[92527]: 6.0 scrub starts 2024-09-13T14:41:59.118 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:59.164 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:59.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:59.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:59.266 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:59.298 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:59.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:59.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:59.400 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:59.425 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:59.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:59.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:59.527 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:59.566 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:41:59.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:59.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:59.667 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:59.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:59.710 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:59.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:59.811 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:59.837 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:59.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:59.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:59.939 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:59.963 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:41:59.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:59.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:00.066 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:00.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:59 smithi184 ceph-mon[92527]: 6.10 scrub starts 2024-09-13T14:42:00.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:41:59 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 675/262908 objects degraded (0.257%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:42:00.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:59 smithi151 ceph-mon[104450]: 6.10 scrub starts 2024-09-13T14:42:00.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:41:59 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 675/262908 objects degraded (0.257%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:42:00.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:59 smithi151 ceph-mon[108966]: 6.10 scrub starts 2024-09-13T14:42:00.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:41:59 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 675/262908 objects degraded (0.257%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:42:00.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:00.109 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:00.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:00.210 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:00.243 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:00.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:00.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:00.344 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:00.374 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:00.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:00.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:00.475 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:00.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:00.511 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:00.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:00.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:42:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:42:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:42:00.613 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:00.649 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:00.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:00.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:00.751 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:00.786 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:00.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:00.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:00.888 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:00.912 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:00.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:00.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:01.014 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:01.046 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:01.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:01.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:01.084 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:00 smithi151 ceph-mon[104450]: pgmap v1396: 109 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.10k op/s; 969/250989 objects degraded (0.386%); 20481/250989 objects misplaced (8.160%); 10 MiB/s, 162 objects/s recovering 2024-09-13T14:42:01.084 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:00 smithi151 ceph-mon[104450]: 6.19 scrub starts 2024-09-13T14:42:01.084 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:00 smithi151 ceph-mon[108966]: pgmap v1396: 109 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.10k op/s; 969/250989 objects degraded (0.386%); 20481/250989 objects misplaced (8.160%); 10 MiB/s, 162 objects/s recovering 2024-09-13T14:42:01.085 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:00 smithi151 ceph-mon[108966]: 6.19 scrub starts 2024-09-13T14:42:01.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:00 smithi184 ceph-mon[92527]: pgmap v1396: 109 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.10k op/s; 969/250989 objects degraded (0.386%); 20481/250989 objects misplaced (8.160%); 10 MiB/s, 162 objects/s recovering 2024-09-13T14:42:01.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:00 smithi184 ceph-mon[92527]: 6.19 scrub starts 2024-09-13T14:42:01.148 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:01.175 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:01.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:01.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:01.277 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:01.305 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:01.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:01.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:01.407 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:01.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:01.442 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:01.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:01.543 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:01.555 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:42:01.556 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-13T14:42:01.556 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-13T14:42:01.556 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:42:01.579 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:01.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:01.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:01.681 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:01.709 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:01.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:01.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:01.811 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:01.845 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:01.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:01.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:01.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:01 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/450150645' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:42:01.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:01 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:42:01.854 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:01.862 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-13T14:42:01.947 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:01.988 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:01.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:01.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:02.090 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:02.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:01 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/450150645' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:42:02.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:01 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:42:02.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:01 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:02.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:01 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/450150645' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:42:02.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:01 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:42:02.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:01 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:02.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:02.131 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:02.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:02.232 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:02.262 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:02.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:02.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:02.303 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T14:42:02.365 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:02.419 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:02.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:02.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:02.521 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:02.553 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:02.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:02.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:02.655 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:02.687 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:02.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:02.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:02.789 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:02.824 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:02.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:02.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:02.833 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:02.926 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:02.954 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:02.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:02.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:03.056 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:03.083 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:03.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:03.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:03.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:02 smithi184 ceph-mon[92527]: pgmap v1397: 109 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 961 op/s; 969/250008 objects degraded (0.388%); 20481/250008 objects misplaced (8.192%); 8.5 MiB/s, 136 objects/s recovering 2024-09-13T14:42:03.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:02 smithi184 ceph-mon[92527]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:42:03.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:02 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:42:03.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:02 smithi184 ceph-mon[92527]: osdmap e1114: 8 total, 8 up, 7 in 2024-09-13T14:42:03.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:02 smithi151 ceph-mon[104450]: pgmap v1397: 109 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 961 op/s; 969/250008 objects degraded (0.388%); 20481/250008 objects misplaced (8.192%); 8.5 MiB/s, 136 objects/s recovering 2024-09-13T14:42:03.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:02 smithi151 ceph-mon[104450]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:42:03.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:02 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:42:03.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:02 smithi151 ceph-mon[104450]: osdmap e1114: 8 total, 8 up, 7 in 2024-09-13T14:42:03.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:02 smithi151 ceph-mon[108966]: pgmap v1397: 109 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 961 op/s; 969/250008 objects degraded (0.388%); 20481/250008 objects misplaced (8.192%); 8.5 MiB/s, 136 objects/s recovering 2024-09-13T14:42:03.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:02 smithi151 ceph-mon[108966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:42:03.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:02 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:42:03.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:02 smithi151 ceph-mon[108966]: osdmap e1114: 8 total, 8 up, 7 in 2024-09-13T14:42:03.184 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:03.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:03.216 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:03.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:03.317 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:03.357 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:03.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:03.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:03.399 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:03.433 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:03.459 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:03.491 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:03.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:03.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:03.593 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:03.625 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:03.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:03.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:03.727 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:03.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:03.762 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:03.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:03.863 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:03.897 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:03.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:03.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:03.999 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:04.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:04.035 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:04.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:04.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:04.168 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:04.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:04.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:04.270 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:04.317 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:04.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:04.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:04.420 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:04.445 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:04.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:04.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:04.547 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:04.573 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:04.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:04.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:04.676 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:04.708 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:04.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:04.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:04.810 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:04.846 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:04.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:04.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:04.856 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:42:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:42:04] "GET /metrics HTTP/1.1" 200 36688 "" "Prometheus/2.51.0" 2024-09-13T14:42:04.949 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:04.988 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:04.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:04.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:05.090 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:05.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:04 smithi151 ceph-mon[104450]: pgmap v1399: 109 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s; 969/237393 objects degraded (0.408%); 19392/237393 objects misplaced (8.169%); 780 KiB/s, 12 objects/s recovering 2024-09-13T14:42:05.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:04 smithi151 ceph-mon[104450]: 6.0 scrub starts 2024-09-13T14:42:05.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:04 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 969/237393 objects degraded (0.408%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:42:05.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:04 smithi151 ceph-mon[108966]: pgmap v1399: 109 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s; 969/237393 objects degraded (0.408%); 19392/237393 objects misplaced (8.169%); 780 KiB/s, 12 objects/s recovering 2024-09-13T14:42:05.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:04 smithi151 ceph-mon[108966]: 6.0 scrub starts 2024-09-13T14:42:05.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:04 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 969/237393 objects degraded (0.408%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:42:05.119 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:05.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:05.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:05.221 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:05.263 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:05.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:05.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:05.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:04 smithi184 ceph-mon[92527]: pgmap v1399: 109 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s; 969/237393 objects degraded (0.408%); 19392/237393 objects misplaced (8.169%); 780 KiB/s, 12 objects/s recovering 2024-09-13T14:42:05.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:04 smithi184 ceph-mon[92527]: 6.0 scrub starts 2024-09-13T14:42:05.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:04 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 969/237393 objects degraded (0.408%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:42:05.362 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:42:05.365 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:05.399 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:05.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:05.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:05.501 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:05.523 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:05.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:05.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:05.625 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:05.663 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:05.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:05.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:05.765 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:05.796 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:05.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:05.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:05.898 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:05.925 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:05.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:05.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:06.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:06.055 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:06.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:06.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:06.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:05 smithi151 ceph-mon[104450]: 6.0 scrub ok 2024-09-13T14:42:06.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:05 smithi151 ceph-mon[104450]: 6.10 scrub starts 2024-09-13T14:42:06.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:05 smithi151 ceph-mon[108966]: 6.0 scrub ok 2024-09-13T14:42:06.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:05 smithi151 ceph-mon[108966]: 6.10 scrub starts 2024-09-13T14:42:06.156 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:06.186 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:06.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:06.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:06.288 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:06.325 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:06.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:06.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:06.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:05 smithi184 ceph-mon[92527]: 6.0 scrub ok 2024-09-13T14:42:06.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:05 smithi184 ceph-mon[92527]: 6.10 scrub starts 2024-09-13T14:42:06.427 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:06.458 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:06.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:06.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:06.560 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:06.797 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:06.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:06.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:06.899 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:06.936 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:06.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:06.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:06.997 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:06 smithi151 ceph-mon[104450]: 6.10 scrub ok 2024-09-13T14:42:06.997 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:06 smithi151 ceph-mon[104450]: pgmap v1400: 109 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.07k op/s; 1232/234390 objects degraded (0.526%); 19075/234390 objects misplaced (8.138%); 652 KiB/s, 10 objects/s recovering 2024-09-13T14:42:06.997 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:06 smithi151 ceph-mon[104450]: 6.19 scrub starts 2024-09-13T14:42:06.997 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:06 smithi151 ceph-mon[108966]: 6.10 scrub ok 2024-09-13T14:42:06.997 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:06 smithi151 ceph-mon[108966]: pgmap v1400: 109 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.07k op/s; 1232/234390 objects degraded (0.526%); 19075/234390 objects misplaced (8.138%); 652 KiB/s, 10 objects/s recovering 2024-09-13T14:42:06.997 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:06 smithi151 ceph-mon[108966]: 6.19 scrub starts 2024-09-13T14:42:07.038 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:07.066 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:07.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:07.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:07.168 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:07.195 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:07.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:07.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:07.298 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:07.336 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:07.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:07.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:07.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:06 smithi184 ceph-mon[92527]: 6.10 scrub ok 2024-09-13T14:42:07.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:06 smithi184 ceph-mon[92527]: pgmap v1400: 109 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.07k op/s; 1232/234390 objects degraded (0.526%); 19075/234390 objects misplaced (8.138%); 652 KiB/s, 10 objects/s recovering 2024-09-13T14:42:07.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:06 smithi184 ceph-mon[92527]: 6.19 scrub starts 2024-09-13T14:42:07.438 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:07.464 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:07.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:07.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:07.566 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:07.599 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:07.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:07.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:07.701 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:07.731 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:07.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:07.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:07.833 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:07.862 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:07.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:07.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:07.963 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:07.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:07.996 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:07.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:08.097 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:08.104 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:07 smithi151 ceph-mon[108966]: 6.19 scrub ok 2024-09-13T14:42:08.105 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:07 smithi151 ceph-mon[104450]: 6.19 scrub ok 2024-09-13T14:42:08.139 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:08.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:08.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:08.241 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:08.279 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:08.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:08.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:08.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:07 smithi184 ceph-mon[92527]: 6.19 scrub ok 2024-09-13T14:42:08.381 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:08.408 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:08.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:08.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:08.510 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:08.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:08.543 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:08.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:08.644 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:08.671 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:08.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:08.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:08.773 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:08.803 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:08.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:08.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:08.887 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:42:08.887 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-13T14:42:08.887 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-13T14:42:08.887 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-13T14:42:08.888 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:42:08.905 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:08.930 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:08.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:08.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:09.032 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:09.065 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:09.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:09.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:09.166 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:09.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:09.195 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:09.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:09.296 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:09.344 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:09.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:09.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:09.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:08 smithi184 ceph-mon[92527]: pgmap v1401: 109 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.08k op/s; 1232/230379 objects degraded (0.535%); 18858/230379 objects misplaced (8.186%); 652 KiB/s, 10 objects/s recovering 2024-09-13T14:42:09.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:08 smithi151 ceph-mon[104450]: pgmap v1401: 109 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.08k op/s; 1232/230379 objects degraded (0.535%); 18858/230379 objects misplaced (8.186%); 652 KiB/s, 10 objects/s recovering 2024-09-13T14:42:09.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:08 smithi151 ceph-mon[108966]: pgmap v1401: 109 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.08k op/s; 1232/230379 objects degraded (0.535%); 18858/230379 objects misplaced (8.186%); 652 KiB/s, 10 objects/s recovering 2024-09-13T14:42:09.447 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:09.484 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:09.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:09.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:09.586 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:09.615 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:09.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:09.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:09.717 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:09.748 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:09.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:09.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:09.850 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:09.878 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:09.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:09.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:09.914 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:42:09.915 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:42:09.943 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:42:09.943 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:42:09.969 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:42:09.969 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:42:09.981 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:09.996 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:42:09.997 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:42:09.997 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:09 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 1232/230379 objects degraded (0.535%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:42:09.997 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:09 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 1232/230379 objects degraded (0.535%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:42:10.014 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:10.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:10.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:10.023 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:42:10.023 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:42:10.053 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:42:10.053 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:42:10.081 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:42:10.082 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:42:10.103 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:42:10.103 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:42:10.116 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:10.149 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:10.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:10.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:10.213 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:09 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 1232/230379 objects degraded (0.535%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:42:10.250 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:10.273 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:10.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:10.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:10.375 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:10.403 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:10.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:10.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:10.505 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:10.541 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:10.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:10.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:10.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:42:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:42:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:42:10.643 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:10.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:10.683 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:10.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:10.784 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:10.823 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:10.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:10.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:10.925 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:10.950 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:10.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:10.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:11.052 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:11.077 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:11.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:11.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:11.158 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:10 smithi151 ceph-mon[104450]: pgmap v1402: 109 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 4.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1533/218283 objects degraded (0.702%); 17715/218283 objects misplaced (8.116%); 633 KiB/s, 9 objects/s recovering 2024-09-13T14:42:11.158 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:10 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/4279341402' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:11.158 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:10 smithi151 ceph-mon[108966]: pgmap v1402: 109 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 4.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1533/218283 objects degraded (0.702%); 17715/218283 objects misplaced (8.116%); 633 KiB/s, 9 objects/s recovering 2024-09-13T14:42:11.158 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:10 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/4279341402' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:11.179 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:11.205 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:11.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:11.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:11.307 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:11.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:11.341 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:11.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:11.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:10 smithi184 ceph-mon[92527]: pgmap v1402: 109 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 4.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1533/218283 objects degraded (0.702%); 17715/218283 objects misplaced (8.116%); 633 KiB/s, 9 objects/s recovering 2024-09-13T14:42:11.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:10 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/4279341402' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:11.442 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:11.469 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:11.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:11.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:11.494 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:11.525 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T14:42:11.571 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:11.650 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:11.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:11.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:11.751 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:11.791 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:11.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:11.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:11.893 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:11.896 INFO:teuthology.orchestra.run.smithi151.stderr:set pool 1 pg_num to 21 2024-09-13T14:42:11.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:11.931 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:11.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:12.032 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:12.063 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:12.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:12.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:12.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:12.189 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:12.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:12.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:12.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:12.318 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:12.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:12.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:12.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:11 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/906061068' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-13T14:42:12.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:11 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/906061068' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-13T14:42:12.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:11 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/906061068' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-13T14:42:12.419 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:12.457 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:12.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:12.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:12.558 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:12.592 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:12.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:12.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:12.694 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:12.717 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:12.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:12.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:12.819 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:12.847 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:12.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:12.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:12.949 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:12.983 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:12.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:12.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:13.060 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:42:13.085 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:13.143 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:13.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:13.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:13.245 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:13.246 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:12 smithi151 ceph-mon[104450]: pgmap v1403: 109 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 4.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1533/217284 objects degraded (0.706%); 17715/217284 objects misplaced (8.153%); 633 KiB/s, 9 objects/s recovering 2024-09-13T14:42:13.246 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:12 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/906061068' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-13T14:42:13.246 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:12 smithi151 ceph-mon[104450]: osdmap e1115: 8 total, 8 up, 7 in 2024-09-13T14:42:13.247 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:12 smithi151 ceph-mon[108966]: pgmap v1403: 109 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 4.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1533/217284 objects degraded (0.706%); 17715/217284 objects misplaced (8.153%); 633 KiB/s, 9 objects/s recovering 2024-09-13T14:42:13.247 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:12 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/906061068' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-13T14:42:13.247 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:12 smithi151 ceph-mon[108966]: osdmap e1115: 8 total, 8 up, 7 in 2024-09-13T14:42:13.288 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:13.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:13.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:13.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:12 smithi184 ceph-mon[92527]: pgmap v1403: 109 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 4.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1533/217284 objects degraded (0.706%); 17715/217284 objects misplaced (8.153%); 633 KiB/s, 9 objects/s recovering 2024-09-13T14:42:13.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:12 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/906061068' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-13T14:42:13.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:12 smithi184 ceph-mon[92527]: osdmap e1115: 8 total, 8 up, 7 in 2024-09-13T14:42:13.390 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:13.428 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:13.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:13.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:13.530 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:13.574 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:13.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:13.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:13.676 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:13.715 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:13.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:13.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:13.816 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:13.859 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:13.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:13.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:13.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:14.016 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:14.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:14.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:14.118 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:14.149 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:14.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:14.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:14.175 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:13 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-13T14:42:14.175 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:13 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-13T14:42:14.251 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:14.254 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:13 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-13T14:42:14.289 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:14.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:14.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:14.391 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:14.419 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:14.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:14.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:14.521 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:14.548 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:14.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:14.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:14.650 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:14.676 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:14.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:14.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:14.778 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:14.812 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:14.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:14.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:14.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:42:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:42:14] "GET /metrics HTTP/1.1" 200 36688 "" "Prometheus/2.51.0" 2024-09-13T14:42:14.914 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:14.950 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:14.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:14.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:15.052 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:15.084 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:15.088 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:15.089 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:15.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:15.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:15.191 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:15.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:14 smithi151 ceph-mon[104450]: pgmap v1405: 109 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 4.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1533/204993 objects degraded (0.748%); 16657/204993 objects misplaced (8.126%); 633 KiB/s, 9 objects/s recovering 2024-09-13T14:42:15.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:14 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-13T14:42:15.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:14 smithi151 ceph-mon[104450]: osdmap e1116: 8 total, 8 up, 7 in 2024-09-13T14:42:15.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:14 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 1533/204993 objects degraded (0.748%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:42:15.197 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:14 smithi151 ceph-mon[108966]: pgmap v1405: 109 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 4.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1533/204993 objects degraded (0.748%); 16657/204993 objects misplaced (8.126%); 633 KiB/s, 9 objects/s recovering 2024-09-13T14:42:15.197 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:14 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-13T14:42:15.197 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:14 smithi151 ceph-mon[108966]: osdmap e1116: 8 total, 8 up, 7 in 2024-09-13T14:42:15.197 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:14 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 1533/204993 objects degraded (0.748%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:42:15.229 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:15.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:15.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:15.331 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:15.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:14 smithi184 ceph-mon[92527]: pgmap v1405: 109 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 4.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1533/204993 objects degraded (0.748%); 16657/204993 objects misplaced (8.126%); 633 KiB/s, 9 objects/s recovering 2024-09-13T14:42:15.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:14 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-13T14:42:15.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:14 smithi184 ceph-mon[92527]: osdmap e1116: 8 total, 8 up, 7 in 2024-09-13T14:42:15.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:14 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 1533/204993 objects degraded (0.748%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:42:15.361 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:15.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:15.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:15.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:15.488 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:15.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:15.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:15.590 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:15.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:15.632 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:15.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:15.733 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:15.775 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:15.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:15.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:15.877 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:15.906 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:15.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:15.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:16.008 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:16.034 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:15 smithi151 ceph-mon[104450]: osdmap e1117: 8 total, 8 up, 7 in 2024-09-13T14:42:16.034 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:15 smithi151 ceph-mon[108966]: osdmap e1117: 8 total, 8 up, 7 in 2024-09-13T14:42:16.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:16.036 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:16.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:16.137 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:16.168 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:16.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:16.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:16.185 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:15 smithi184 ceph-mon[92527]: osdmap e1117: 8 total, 8 up, 7 in 2024-09-13T14:42:16.270 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:16.312 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:16.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:16.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:16.414 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:16.445 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:16.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:16.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:16.547 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:16.572 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:16.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:16.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:16.674 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:16.712 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:16.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:16.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:16.814 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:16.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:16.842 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:16.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:16.940 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-13T14:42:16.943 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:16.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:16.987 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:16.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:17.088 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:17.115 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:17.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:17.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:17.169 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:16 smithi151 ceph-mon[104450]: pgmap v1408: 119 pgs: 10 unknown, 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 4.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 903 op/s; 1791/202014 objects degraded (0.887%); 16340/202014 objects misplaced (8.089%); 629 KiB/s, 9 objects/s recovering 2024-09-13T14:42:17.169 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:16 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1362748753' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:42:17.169 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:17.169 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:16 smithi151 ceph-mon[108966]: pgmap v1408: 119 pgs: 10 unknown, 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 4.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 903 op/s; 1791/202014 objects degraded (0.887%); 16340/202014 objects misplaced (8.089%); 629 KiB/s, 9 objects/s recovering 2024-09-13T14:42:17.170 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:16 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1362748753' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:42:17.170 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:17.216 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:17.251 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:17.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:17.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:17.353 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:17.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:16 smithi184 ceph-mon[92527]: pgmap v1408: 119 pgs: 10 unknown, 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 4.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 903 op/s; 1791/202014 objects degraded (0.887%); 16340/202014 objects misplaced (8.089%); 629 KiB/s, 9 objects/s recovering 2024-09-13T14:42:17.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:16 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1362748753' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:42:17.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:17.380 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:17.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:17.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:17.482 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:17.505 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:17.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:17.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:17.607 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:17.637 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:17.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:17.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:17.739 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:17.777 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:17.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:17.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:17.879 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:17.905 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:17.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:17.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:18.007 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:18.043 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:18.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:18.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:18.145 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:18.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:18.172 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:18.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:18.273 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:18.302 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:18.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:18.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:18.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:17 smithi184 ceph-mon[92527]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:42:18.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:17 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1362748753' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:42:18.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:17 smithi184 ceph-mon[92527]: osdmap e1118: 8 total, 8 up, 7 in 2024-09-13T14:42:18.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:17 smithi151 ceph-mon[104450]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:42:18.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:17 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1362748753' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:42:18.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:17 smithi151 ceph-mon[104450]: osdmap e1118: 8 total, 8 up, 7 in 2024-09-13T14:42:18.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:17 smithi151 ceph-mon[108966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:42:18.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:17 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1362748753' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:42:18.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:17 smithi151 ceph-mon[108966]: osdmap e1118: 8 total, 8 up, 7 in 2024-09-13T14:42:18.404 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:18.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:18.439 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:18.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:18.540 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:18.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:18.570 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:18.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:18.671 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:18.711 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:18.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:18.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:18.812 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:18.852 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:18.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:18.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:18.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:18.977 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:18.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:18.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:19.079 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:19.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:18 smithi151 ceph-mon[104450]: pgmap v1410: 119 pgs: 9 unknown, 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 102 active+clean; 4.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 1.17k op/s; 1791/198147 objects degraded (0.904%); 16117/198147 objects misplaced (8.134%); 691 KiB/s, 10 objects/s recovering 2024-09-13T14:42:19.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:18 smithi151 ceph-mon[108966]: pgmap v1410: 119 pgs: 9 unknown, 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 102 active+clean; 4.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 1.17k op/s; 1791/198147 objects degraded (0.904%); 16117/198147 objects misplaced (8.134%); 691 KiB/s, 10 objects/s recovering 2024-09-13T14:42:19.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:19.112 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:19.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:19.213 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:19.247 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:19.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:19.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:19.349 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:19.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:18 smithi184 ceph-mon[92527]: pgmap v1410: 119 pgs: 9 unknown, 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 102 active+clean; 4.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 1.17k op/s; 1791/198147 objects degraded (0.904%); 16117/198147 objects misplaced (8.134%); 691 KiB/s, 10 objects/s recovering 2024-09-13T14:42:19.390 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:19.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:19.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:19.461 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:42:19.462 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:42:19.462 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-13T14:42:19.462 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:42:19.491 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:19.534 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:19.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:19.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:19.636 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:19.674 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:19.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:19.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:19.775 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:19.800 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:19.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:19.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:19.902 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:19.926 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:19.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:19.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:20.028 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:20.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:20.053 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:20.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:20.154 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:20.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:20.191 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:20.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:20.213 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:19 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 1791/198147 objects degraded (0.904%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:42:20.292 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:20.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:20.324 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:20.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:20.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:19 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 1791/198147 objects degraded (0.904%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:42:20.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:19 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 1791/198147 objects degraded (0.904%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:42:20.426 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:20.447 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:20.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:20.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:20.549 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:20.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:20.585 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:20.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:20.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:42:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:42:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:42:20.687 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:20.727 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:20.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:20.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:20.829 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:20.860 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:20.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:20.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:20.962 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:20.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:20.994 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:20.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:21.095 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:21.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:20 smithi151 ceph-mon[104450]: pgmap v1411: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 3.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s; 2045/186540 objects degraded (1.096%); 14998/186540 objects misplaced (8.040%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:42:21.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:20 smithi151 ceph-mon[108966]: pgmap v1411: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 3.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s; 2045/186540 objects degraded (1.096%); 14998/186540 objects misplaced (8.040%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:42:21.135 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:21.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:21.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:21.237 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:21.265 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:21.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:21.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:21.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:20 smithi184 ceph-mon[92527]: pgmap v1411: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 3.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s; 2045/186540 objects degraded (1.096%); 14998/186540 objects misplaced (8.040%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:42:21.366 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:21.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:21.406 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:21.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:21.507 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:21.536 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:21.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:21.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:21.638 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:21.676 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:21.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:21.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:21.778 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:21.811 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:21.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:21.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:21.913 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:21.946 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:21.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:21.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:22.048 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:22.076 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:22.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:22.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:22.139 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:42:22.139 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:42:22.166 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:42:22.166 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:42:22.179 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:22.191 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:42:22.191 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:42:22.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:22.217 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:22.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:22.221 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:42:22.221 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:42:22.243 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:42:22.243 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:42:22.265 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:42:22.266 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:42:22.302 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:42:22.302 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:42:22.317 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:22.320 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:42:22.320 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:42:22.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:22.364 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:22.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:22.451 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:42:22.451 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-13T14:42:22.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-13T14:42:22.451 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-13T14:42:22.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.452 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.35289.0:20058 6.a e1086/1082 6:5e2dffa5:::benchmark_data_smithi151_155836_object20057:head v 1086'1339, mlcod=1086'1336)", 2024-09-13T14:42:22.452 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:39:33.450339+0000", 2024-09-13T14:42:22.452 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 168.99538417599999, 2024-09-13T14:42:22.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.063391816000000004, 2024-09-13T14:42:22.452 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:22.452 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:22.453 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:22.453 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:22.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:22.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.450339+0000", 2024-09-13T14:42:22.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:22.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.450339+0000", 2024-09-13T14:42:22.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:22.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.450340+0000", 2024-09-13T14:42:22.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8700000000000001e-07 2024-09-13T14:42:22.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:22.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.450366+0000", 2024-09-13T14:42:22.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6443000000000001e-05 2024-09-13T14:42:22.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:22.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.450375+0000", 2024-09-13T14:42:22.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5550000000000003e-06 2024-09-13T14:42:22.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:22.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.450380+0000", 2024-09-13T14:42:22.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7530000000000001e-06 2024-09-13T14:42:22.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:22.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.450433+0000", 2024-09-13T14:42:22.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3397999999999999e-05 2024-09-13T14:42:22.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:22.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.450446+0000", 2024-09-13T14:42:22.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2743e-05 2024-09-13T14:42:22.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:22.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.451428+0000", 2024-09-13T14:42:22.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00098184899999999992 2024-09-13T14:42:22.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:22.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.513731+0000", 2024-09-13T14:42:22.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.062303688000000003 2024-09-13T14:42:22.461 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.461 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:22.461 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.463 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:20063 6.0 6:0dfbae9d:::benchmark_data_smithi151_155836_object20062:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1086)", 2024-09-13T14:42:22.463 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:39:33.451380+0000", 2024-09-13T14:42:22.463 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 168.99434364000001, 2024-09-13T14:42:22.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.070814190999999999, 2024-09-13T14:42:22.463 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:22.464 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:22.464 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:22.464 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:22.464 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:22.464 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:22.465 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 20063 2024-09-13T14:42:22.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.465 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:22.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:22.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.451380+0000", 2024-09-13T14:42:22.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:22.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.451380+0000", 2024-09-13T14:42:22.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:22.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.451382+0000", 2024-09-13T14:42:22.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.548e-06 2024-09-13T14:42:22.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:22.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.451410+0000", 2024-09-13T14:42:22.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8408e-05 2024-09-13T14:42:22.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:22.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.451423+0000", 2024-09-13T14:42:22.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2982999999999999e-05 2024-09-13T14:42:22.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:22.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.513686+0000", 2024-09-13T14:42:22.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.062263458000000001 2024-09-13T14:42:22.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:22.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.513827+0000", 2024-09-13T14:42:22.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014098100000000001 2024-09-13T14:42:22.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:22.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.513958+0000", 2024-09-13T14:42:22.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013056100000000001 2024-09-13T14:42:22.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:42:22.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.514086+0000", 2024-09-13T14:42:22.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012783599999999999 2024-09-13T14:42:22.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.516351+0000", 2024-09-13T14:42:22.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0022654749999999999 2024-09-13T14:42:22.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:22.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.516362+0000", 2024-09-13T14:42:22.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0783e-05 2024-09-13T14:42:22.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.522099+0000", 2024-09-13T14:42:22.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0057370620000000002 2024-09-13T14:42:22.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:22.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.522154+0000", 2024-09-13T14:42:22.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4472000000000003e-05 2024-09-13T14:42:22.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:22.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.522194+0000", 2024-09-13T14:42:22.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0624000000000003e-05 2024-09-13T14:42:22.471 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.471 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:22.471 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.471 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.35289.0:20069 6.d e1086/1082 6:b320b20e:::benchmark_data_smithi151_155836_object20068:head v 1086'1240, mlcod=1086'1238)", 2024-09-13T14:42:22.471 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:39:33.452533+0000", 2024-09-13T14:42:22.471 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 168.993191134, 2024-09-13T14:42:22.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.062941201000000002, 2024-09-13T14:42:22.471 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:22.471 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:22.472 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:22.472 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:22.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:22.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.452533+0000", 2024-09-13T14:42:22.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:22.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.452533+0000", 2024-09-13T14:42:22.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:22.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.452534+0000", 2024-09-13T14:42:22.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0610000000000001e-06 2024-09-13T14:42:22.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:22.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.452657+0000", 2024-09-13T14:42:22.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012295000000000001 2024-09-13T14:42:22.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:22.474 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.513694+0000", 2024-09-13T14:42:22.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.061037462000000001 2024-09-13T14:42:22.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.475 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:22.475 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.513705+0000", 2024-09-13T14:42:22.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1344e-05 2024-09-13T14:42:22.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:22.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.513760+0000", 2024-09-13T14:42:22.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4925e-05 2024-09-13T14:42:22.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:22.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.513782+0000", 2024-09-13T14:42:22.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1384000000000001e-05 2024-09-13T14:42:22.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:22.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.515325+0000", 2024-09-13T14:42:22.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001543862 2024-09-13T14:42:22.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:22.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:33.515474+0000", 2024-09-13T14:42:22.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014821300000000001 2024-09-13T14:42:22.478 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.478 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:22.478 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.479 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:56441 6.9 6:93dafce0:::benchmark_data_smithi151_155836_object56440:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1088)", 2024-09-13T14:42:22.479 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:39:51.012950+0000", 2024-09-13T14:42:22.479 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 151.43277359000001, 2024-09-13T14:42:22.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.062829593000000003, 2024-09-13T14:42:22.479 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:22.480 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:22.480 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:22.480 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:22.480 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:22.480 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:22.480 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 56441 2024-09-13T14:42:22.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.481 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:22.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:22.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.012950+0000", 2024-09-13T14:42:22.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:22.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.012950+0000", 2024-09-13T14:42:22.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:22.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.012951+0000", 2024-09-13T14:42:22.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.569999999999999e-07 2024-09-13T14:42:22.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:22.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.012984+0000", 2024-09-13T14:42:22.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2691000000000003e-05 2024-09-13T14:42:22.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:22.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.012996+0000", 2024-09-13T14:42:22.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2479e-05 2024-09-13T14:42:22.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:22.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.013001+0000", 2024-09-13T14:42:22.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0069999999999997e-06 2024-09-13T14:42:22.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:22.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.013040+0000", 2024-09-13T14:42:22.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9189e-05 2024-09-13T14:42:22.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:22.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.013131+0000", 2024-09-13T14:42:22.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0444000000000003e-05 2024-09-13T14:42:22.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,7", 2024-09-13T14:42:22.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.013218+0000", 2024-09-13T14:42:22.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7085999999999999e-05 2024-09-13T14:42:22.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:22.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.015837+0000", 2024-09-13T14:42:22.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002619138 2024-09-13T14:42:22.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.015881+0000", 2024-09-13T14:42:22.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4409000000000001e-05 2024-09-13T14:42:22.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.075661+0000", 2024-09-13T14:42:22.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.059779653000000002 2024-09-13T14:42:22.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:22.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.075724+0000", 2024-09-13T14:42:22.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2446000000000005e-05 2024-09-13T14:42:22.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:22.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.075780+0000", 2024-09-13T14:42:22.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6094000000000002e-05 2024-09-13T14:42:22.491 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.491 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:22.491 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.491 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:56443 6.9 6:97dfd562:::benchmark_data_smithi151_155836_object56442:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1088)", 2024-09-13T14:42:22.491 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:39:51.013788+0000", 2024-09-13T14:42:22.492 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 151.431935971, 2024-09-13T14:42:22.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.062173067999999998, 2024-09-13T14:42:22.492 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:22.492 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:22.492 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:22.492 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:22.492 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:22.492 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:22.492 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 56443 2024-09-13T14:42:22.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.492 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:22.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:22.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.013788+0000", 2024-09-13T14:42:22.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:22.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.013788+0000", 2024-09-13T14:42:22.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:22.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.013788+0000", 2024-09-13T14:42:22.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6599999999999996e-07 2024-09-13T14:42:22.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:22.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.013813+0000", 2024-09-13T14:42:22.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5109e-05 2024-09-13T14:42:22.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:22.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.013821+0000", 2024-09-13T14:42:22.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5089999999999996e-06 2024-09-13T14:42:22.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:22.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.013825+0000", 2024-09-13T14:42:22.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4700000000000004e-06 2024-09-13T14:42:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.014709+0000", 2024-09-13T14:42:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00088385999999999998 2024-09-13T14:42:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.014783+0000", 2024-09-13T14:42:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3405999999999995e-05 2024-09-13T14:42:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,7", 2024-09-13T14:42:22.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.014988+0000", 2024-09-13T14:42:22.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000205155 2024-09-13T14:42:22.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:22.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.015968+0000", 2024-09-13T14:42:22.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00098038599999999997 2024-09-13T14:42:22.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.016872+0000", 2024-09-13T14:42:22.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00090380600000000005 2024-09-13T14:42:22.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.075843+0000", 2024-09-13T14:42:22.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.058971303000000003 2024-09-13T14:42:22.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:22.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.075915+0000", 2024-09-13T14:42:22.497 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:22.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1898999999999997e-05 2024-09-13T14:42:22.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:22.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:51.075961+0000", 2024-09-13T14:42:22.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5498999999999999e-05 2024-09-13T14:42:22.500 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.500 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:22.500 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.500 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:63700 6.0 6:0866df32:::benchmark_data_smithi151_155836_object63699:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1088)", 2024-09-13T14:42:22.500 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:39:55.554341+0000", 2024-09-13T14:42:22.500 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 146.891383015, 2024-09-13T14:42:22.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.10001168000000001, 2024-09-13T14:42:22.500 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:22.500 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:22.501 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:22.501 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:22.501 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:22.501 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:22.501 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 63700 2024-09-13T14:42:22.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.501 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:22.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:22.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.554341+0000", 2024-09-13T14:42:22.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:22.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.554341+0000", 2024-09-13T14:42:22.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:22.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.554342+0000", 2024-09-13T14:42:22.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.048e-06 2024-09-13T14:42:22.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:22.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.554364+0000", 2024-09-13T14:42:22.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2413999999999999e-05 2024-09-13T14:42:22.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:22.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.554380+0000", 2024-09-13T14:42:22.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6132000000000001e-05 2024-09-13T14:42:22.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:22.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.554388+0000", 2024-09-13T14:42:22.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8760000000000005e-06 2024-09-13T14:42:22.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:22.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.555805+0000", 2024-09-13T14:42:22.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001416582 2024-09-13T14:42:22.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:22.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.555936+0000", 2024-09-13T14:42:22.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013179199999999999 2024-09-13T14:42:22.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:42:22.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.556075+0000", 2024-09-13T14:42:22.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000138107 2024-09-13T14:42:22.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.505 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:22.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.559244+0000", 2024-09-13T14:42:22.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003169832 2024-09-13T14:42:22.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.505 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.563587+0000", 2024-09-13T14:42:22.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0043428010000000003 2024-09-13T14:42:22.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.654214+0000", 2024-09-13T14:42:22.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.090626994000000002 2024-09-13T14:42:22.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:22.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.654300+0000", 2024-09-13T14:42:22.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5384999999999996e-05 2024-09-13T14:42:22.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:22.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.654352+0000", 2024-09-13T14:42:22.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2716999999999997e-05 2024-09-13T14:42:22.506 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.507 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:22.507 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.507 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:63708 6.0 6:0c0735af:::benchmark_data_smithi151_155836_object63707:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1088)", 2024-09-13T14:42:22.507 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:39:55.560757+0000", 2024-09-13T14:42:22.507 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 146.88496685800001, 2024-09-13T14:42:22.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.097823137000000004, 2024-09-13T14:42:22.507 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:22.507 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:22.507 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:22.507 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:22.507 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:22.508 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:22.508 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 63708 2024-09-13T14:42:22.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.508 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:22.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:22.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.560757+0000", 2024-09-13T14:42:22.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:22.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.560757+0000", 2024-09-13T14:42:22.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:22.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.560759+0000", 2024-09-13T14:42:22.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.75e-06 2024-09-13T14:42:22.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:22.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.560783+0000", 2024-09-13T14:42:22.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4533999999999999e-05 2024-09-13T14:42:22.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:22.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.560808+0000", 2024-09-13T14:42:22.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5256e-05 2024-09-13T14:42:22.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:22.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.560816+0000", 2024-09-13T14:42:22.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5700000000000004e-06 2024-09-13T14:42:22.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:22.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.560876+0000", 2024-09-13T14:42:22.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0568e-05 2024-09-13T14:42:22.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:22.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.561066+0000", 2024-09-13T14:42:22.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018988699999999999 2024-09-13T14:42:22.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:42:22.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.561204+0000", 2024-09-13T14:42:22.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000137472 2024-09-13T14:42:22.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:22.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.563600+0000", 2024-09-13T14:42:22.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0023959580000000001 2024-09-13T14:42:22.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.566109+0000", 2024-09-13T14:42:22.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0025091419999999998 2024-09-13T14:42:22.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.658465+0000", 2024-09-13T14:42:22.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.092355698999999999 2024-09-13T14:42:22.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:22.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.658525+0000", 2024-09-13T14:42:22.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0532000000000003e-05 2024-09-13T14:42:22.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:22.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.658580+0000", 2024-09-13T14:42:22.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4768999999999997e-05 2024-09-13T14:42:22.513 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.513 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:22.513 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.514 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:63718 6.0 6:0bb97f89:::benchmark_data_smithi151_155836_object63717:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1088)", 2024-09-13T14:42:22.514 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:39:55.567043+0000", 2024-09-13T14:42:22.514 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 146.87868074799999, 2024-09-13T14:42:22.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.091663517, 2024-09-13T14:42:22.514 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:22.514 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:22.514 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:22.514 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:22.514 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:22.514 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:22.515 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 63718 2024-09-13T14:42:22.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.515 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:22.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:22.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.567043+0000", 2024-09-13T14:42:22.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:22.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.567043+0000", 2024-09-13T14:42:22.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:22.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.567044+0000", 2024-09-13T14:42:22.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5379999999999999e-06 2024-09-13T14:42:22.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:22.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.567071+0000", 2024-09-13T14:42:22.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6180999999999999e-05 2024-09-13T14:42:22.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:22.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.567085+0000", 2024-09-13T14:42:22.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4127999999999999e-05 2024-09-13T14:42:22.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:22.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.567092+0000", 2024-09-13T14:42:22.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9310000000000002e-06 2024-09-13T14:42:22.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:22.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.567148+0000", 2024-09-13T14:42:22.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6326999999999998e-05 2024-09-13T14:42:22.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:22.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.567282+0000", 2024-09-13T14:42:22.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000134084 2024-09-13T14:42:22.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:42:22.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.567372+0000", 2024-09-13T14:42:22.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9996999999999999e-05 2024-09-13T14:42:22.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:22.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.569617+0000", 2024-09-13T14:42:22.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0022450310000000002 2024-09-13T14:42:22.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.570825+0000", 2024-09-13T14:42:22.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012079599999999999 2024-09-13T14:42:22.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.658618+0000", 2024-09-13T14:42:22.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.087793379000000005 2024-09-13T14:42:22.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:22.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.658669+0000", 2024-09-13T14:42:22.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0392999999999997e-05 2024-09-13T14:42:22.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.520 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:22.520 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.658706+0000", 2024-09-13T14:42:22.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7567999999999997e-05 2024-09-13T14:42:22.520 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.520 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:22.520 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.520 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:105787 6.0 6:021d7619:::benchmark_data_smithi151_155836_object105786:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1092)", 2024-09-13T14:42:22.520 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:40:29.790930+0000", 2024-09-13T14:42:22.521 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 112.654793974, 2024-09-13T14:42:22.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.064371177000000002, 2024-09-13T14:42:22.521 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:22.521 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:22.521 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:22.521 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:22.521 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:22.521 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:22.521 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 105787 2024-09-13T14:42:22.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.521 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:22.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:22.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:29.790930+0000", 2024-09-13T14:42:22.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:22.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:29.790930+0000", 2024-09-13T14:42:22.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:22.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:29.790931+0000", 2024-09-13T14:42:22.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2410000000000001e-06 2024-09-13T14:42:22.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:22.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:29.790980+0000", 2024-09-13T14:42:22.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8785999999999997e-05 2024-09-13T14:42:22.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:22.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:29.791064+0000", 2024-09-13T14:42:22.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4474000000000005e-05 2024-09-13T14:42:22.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:22.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:29.791073+0000", 2024-09-13T14:42:22.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0170000000000002e-06 2024-09-13T14:42:22.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:22.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:29.791205+0000", 2024-09-13T14:42:22.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013184899999999999 2024-09-13T14:42:22.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:22.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:29.791396+0000", 2024-09-13T14:42:22.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019127 2024-09-13T14:42:22.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:42:22.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:29.791544+0000", 2024-09-13T14:42:22.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014810499999999999 2024-09-13T14:42:22.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:29.798751+0000", 2024-09-13T14:42:22.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0072070750000000003 2024-09-13T14:42:22.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:22.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:29.798775+0000", 2024-09-13T14:42:22.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3176999999999998e-05 2024-09-13T14:42:22.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.526 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:29.855205+0000", 2024-09-13T14:42:22.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.056430274000000002 2024-09-13T14:42:22.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.526 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:22.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:29.855258+0000", 2024-09-13T14:42:22.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3020999999999999e-05 2024-09-13T14:42:22.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.526 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:22.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:29.855301+0000", 2024-09-13T14:42:22.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2888e-05 2024-09-13T14:42:22.527 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.527 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:22.527 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.527 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:151161 6.0 6:0f74eb8a:::benchmark_data_smithi151_155836_object18703:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1098)", 2024-09-13T14:42:22.527 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:41:18.491558+0000", 2024-09-13T14:42:22.527 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 63.954165314000001, 2024-09-13T14:42:22.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.065308802999999999, 2024-09-13T14:42:22.527 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:22.527 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:22.527 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:22.527 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:22.528 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:22.528 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:22.528 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 151161 2024-09-13T14:42:22.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.528 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:22.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:22.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.491558+0000", 2024-09-13T14:42:22.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:22.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.491558+0000", 2024-09-13T14:42:22.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:22.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.491559+0000", 2024-09-13T14:42:22.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.145e-06 2024-09-13T14:42:22.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:22.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.491564+0000", 2024-09-13T14:42:22.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.425e-06 2024-09-13T14:42:22.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:22.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.491574+0000", 2024-09-13T14:42:22.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0247e-05 2024-09-13T14:42:22.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:22.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.491579+0000", 2024-09-13T14:42:22.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8180000000000003e-06 2024-09-13T14:42:22.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:22.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.491635+0000", 2024-09-13T14:42:22.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5831000000000001e-05 2024-09-13T14:42:22.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:22.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.491715+0000", 2024-09-13T14:42:22.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9772999999999997e-05 2024-09-13T14:42:22.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:42:22.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.491782+0000", 2024-09-13T14:42:22.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7003999999999997e-05 2024-09-13T14:42:22.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.532 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:22.532 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.495588+0000", 2024-09-13T14:42:22.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0038060260000000001 2024-09-13T14:42:22.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.532 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.532 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.496183+0000", 2024-09-13T14:42:22.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00059497499999999997 2024-09-13T14:42:22.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.532 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.556763+0000", 2024-09-13T14:42:22.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.06058032 2024-09-13T14:42:22.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:22.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.556821+0000", 2024-09-13T14:42:22.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8362999999999997e-05 2024-09-13T14:42:22.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:22.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.556867+0000", 2024-09-13T14:42:22.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5875999999999999e-05 2024-09-13T14:42:22.533 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.534 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:22.534 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.534 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:151166 6.9 6:9107791e:::benchmark_data_smithi151_155836_object18708:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1098)", 2024-09-13T14:42:22.534 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:41:18.498529+0000", 2024-09-13T14:42:22.534 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 63.947194228999997, 2024-09-13T14:42:22.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.061555895999999999, 2024-09-13T14:42:22.534 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:22.534 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:22.534 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:22.534 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:22.534 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:22.535 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:22.535 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 151166 2024-09-13T14:42:22.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.535 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:22.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:22.535 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.498529+0000", 2024-09-13T14:42:22.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:22.535 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.498529+0000", 2024-09-13T14:42:22.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:22.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.498531+0000", 2024-09-13T14:42:22.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.35e-06 2024-09-13T14:42:22.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:22.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.498534+0000", 2024-09-13T14:42:22.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7050000000000001e-06 2024-09-13T14:42:22.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:22.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.498544+0000", 2024-09-13T14:42:22.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3300000000000005e-06 2024-09-13T14:42:22.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:22.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.498549+0000", 2024-09-13T14:42:22.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1039999999999998e-06 2024-09-13T14:42:22.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:22.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.498591+0000", 2024-09-13T14:42:22.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2571999999999998e-05 2024-09-13T14:42:22.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.538 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:22.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.498672+0000", 2024-09-13T14:42:22.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0097000000000002e-05 2024-09-13T14:42:22.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.538 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,7", 2024-09-13T14:42:22.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.498738+0000", 2024-09-13T14:42:22.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6623000000000002e-05 2024-09-13T14:42:22.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.538 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:22.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.559839+0000", 2024-09-13T14:42:22.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.061100598999999998 2024-09-13T14:42:22.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.539 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.559937+0000", 2024-09-13T14:42:22.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8518999999999996e-05 2024-09-13T14:42:22.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.539 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.559988+0000", 2024-09-13T14:42:22.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0578000000000003e-05 2024-09-13T14:42:22.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:22.539 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.560040+0000", 2024-09-13T14:42:22.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2117000000000003e-05 2024-09-13T14:42:22.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.540 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:22.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.560085+0000", 2024-09-13T14:42:22.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5302000000000001e-05 2024-09-13T14:42:22.540 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.540 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:22.540 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.540 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:151171 6.9 6:92227b7f:::benchmark_data_smithi151_155836_object18713:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1098)", 2024-09-13T14:42:22.540 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:41:18.499638+0000", 2024-09-13T14:42:22.541 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 63.946085818999997, 2024-09-13T14:42:22.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.087989601000000001, 2024-09-13T14:42:22.541 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:22.541 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:22.541 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:22.541 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:22.541 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:22.541 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:22.541 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 151171 2024-09-13T14:42:22.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.541 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:22.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:22.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.499638+0000", 2024-09-13T14:42:22.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:22.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.499638+0000", 2024-09-13T14:42:22.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:22.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.499639+0000", 2024-09-13T14:42:22.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8299999999999995e-07 2024-09-13T14:42:22.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:22.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.499643+0000", 2024-09-13T14:42:22.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0380000000000001e-06 2024-09-13T14:42:22.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:22.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.499653+0000", 2024-09-13T14:42:22.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0740000000000001e-05 2024-09-13T14:42:22.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:22.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.499660+0000", 2024-09-13T14:42:22.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0119999999999999e-06 2024-09-13T14:42:22.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.544 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:22.544 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.555841+0000", 2024-09-13T14:42:22.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.056180289000000001 2024-09-13T14:42:22.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.544 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:22.544 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.555979+0000", 2024-09-13T14:42:22.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013826600000000001 2024-09-13T14:42:22.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,7", 2024-09-13T14:42:22.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.556110+0000", 2024-09-13T14:42:22.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000131039 2024-09-13T14:42:22.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:22.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.559948+0000", 2024-09-13T14:42:22.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0038383969999999999 2024-09-13T14:42:22.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.567861+0000", 2024-09-13T14:42:22.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0079127309999999992 2024-09-13T14:42:22.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.587529+0000", 2024-09-13T14:42:22.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019667694999999999 2024-09-13T14:42:22.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:22.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.587586+0000", 2024-09-13T14:42:22.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7349000000000003e-05 2024-09-13T14:42:22.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:22.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.587627+0000", 2024-09-13T14:42:22.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1162000000000003e-05 2024-09-13T14:42:22.547 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.547 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:22.547 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.547 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:151172 6.0 6:029e18bc:::benchmark_data_smithi151_155836_object18714:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1098)", 2024-09-13T14:42:22.547 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:41:18.499920+0000", 2024-09-13T14:42:22.547 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 63.945803789999999, 2024-09-13T14:42:22.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.070157033999999993, 2024-09-13T14:42:22.547 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:22.547 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:22.547 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:22.548 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:22.548 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:22.548 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:22.548 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 151172 2024-09-13T14:42:22.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.548 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:22.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:22.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.499920+0000", 2024-09-13T14:42:22.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:22.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.499920+0000", 2024-09-13T14:42:22.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:22.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.499921+0000", 2024-09-13T14:42:22.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2800000000000005e-07 2024-09-13T14:42:22.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:22.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.499926+0000", 2024-09-13T14:42:22.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9779999999999996e-06 2024-09-13T14:42:22.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:22.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.499937+0000", 2024-09-13T14:42:22.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1005000000000001e-05 2024-09-13T14:42:22.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:22.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.499943+0000", 2024-09-13T14:42:22.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6939999999999998e-06 2024-09-13T14:42:22.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:22.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.499994+0000", 2024-09-13T14:42:22.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0763999999999997e-05 2024-09-13T14:42:22.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:22.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.500096+0000", 2024-09-13T14:42:22.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010220700000000001 2024-09-13T14:42:22.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:42:22.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.552875+0000", 2024-09-13T14:42:22.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052778609999999997 2024-09-13T14:42:22.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.556906+0000", 2024-09-13T14:42:22.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0040313349999999996 2024-09-13T14:42:22.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:22.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.569895+0000", 2024-09-13T14:42:22.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012988665 2024-09-13T14:42:22.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.569988+0000", 2024-09-13T14:42:22.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2906999999999997e-05 2024-09-13T14:42:22.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.553 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:22.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.570036+0000", 2024-09-13T14:42:22.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7732000000000003e-05 2024-09-13T14:42:22.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.553 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:22.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:18.570077+0000", 2024-09-13T14:42:22.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1208999999999998e-05 2024-09-13T14:42:22.553 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.553 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:22.553 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.554 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:168006 6.0 6:0666a339:::benchmark_data_smithi151_155836_object35548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1106)", 2024-09-13T14:42:22.554 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:41:40.439427+0000", 2024-09-13T14:42:22.554 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 42.006296790999997, 2024-09-13T14:42:22.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.069235413999999995, 2024-09-13T14:42:22.554 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:22.554 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:22.554 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:22.554 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:22.554 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:22.554 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:22.554 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 168006 2024-09-13T14:42:22.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.555 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:22.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:22.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:40.439427+0000", 2024-09-13T14:42:22.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:22.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:40.439427+0000", 2024-09-13T14:42:22.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:22.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:40.439428+0000", 2024-09-13T14:42:22.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6499999999999999e-07 2024-09-13T14:42:22.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:22.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:40.439431+0000", 2024-09-13T14:42:22.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3830000000000001e-06 2024-09-13T14:42:22.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:22.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:40.439441+0000", 2024-09-13T14:42:22.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6509999999999996e-06 2024-09-13T14:42:22.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.557 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:22.557 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:40.439447+0000", 2024-09-13T14:42:22.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7750000000000004e-06 2024-09-13T14:42:22.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.557 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:22.557 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:40.439489+0000", 2024-09-13T14:42:22.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2070000000000002e-05 2024-09-13T14:42:22.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:22.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:40.439591+0000", 2024-09-13T14:42:22.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101987 2024-09-13T14:42:22.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:42:22.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:40.439693+0000", 2024-09-13T14:42:22.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101724 2024-09-13T14:42:22.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:22.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:40.442154+0000", 2024-09-13T14:42:22.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0024612739999999998 2024-09-13T14:42:22.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:40.446962+0000", 2024-09-13T14:42:22.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0048079280000000004 2024-09-13T14:42:22.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:40.508552+0000", 2024-09-13T14:42:22.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.061589629999999999 2024-09-13T14:42:22.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:22.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:40.508616+0000", 2024-09-13T14:42:22.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4430999999999993e-05 2024-09-13T14:42:22.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:22.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:40.508662+0000", 2024-09-13T14:42:22.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5896000000000002e-05 2024-09-13T14:42:22.561 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.561 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:22.561 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.561 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:168008 6.0 6:0df57809:::benchmark_data_smithi151_155836_object35550:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1106)", 2024-09-13T14:42:22.561 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:41:40.443461+0000", 2024-09-13T14:42:22.561 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 42.002262838999997, 2024-09-13T14:42:22.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.065302674000000005, 2024-09-13T14:42:22.561 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:22.561 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:22.561 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:22.561 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:22.562 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:22.562 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:22.562 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 168008 2024-09-13T14:42:22.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.562 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:22.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:22.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:40.443461+0000", 2024-09-13T14:42:22.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:22.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:40.443461+0000", 2024-09-13T14:42:22.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:22.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:40.443462+0000", 2024-09-13T14:42:22.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8000000000000004e-07 2024-09-13T14:42:22.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:22.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:40.443466+0000", 2024-09-13T14:42:22.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8700000000000002e-06 2024-09-13T14:42:22.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:22.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:40.443476+0000", 2024-09-13T14:42:22.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0052e-05 2024-09-13T14:42:22.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:22.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:40.443481+0000", 2024-09-13T14:42:22.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.75e-06 2024-09-13T14:42:22.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:22.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:40.443526+0000", 2024-09-13T14:42:22.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5102999999999998e-05 2024-09-13T14:42:22.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:22.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:40.443611+0000", 2024-09-13T14:42:22.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4658000000000002e-05 2024-09-13T14:42:22.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:42:22.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:40.443681+0000", 2024-09-13T14:42:22.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9407999999999997e-05 2024-09-13T14:42:22.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:22.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:40.446805+0000", 2024-09-13T14:42:22.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031246870000000001 2024-09-13T14:42:22.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.566 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:40.447081+0000", 2024-09-13T14:42:22.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027625799999999998 2024-09-13T14:42:22.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.566 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:40.508696+0000", 2024-09-13T14:42:22.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.061614056 2024-09-13T14:42:22.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.567 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:22.567 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:40.508734+0000", 2024-09-13T14:42:22.567 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8343000000000002e-05 2024-09-13T14:42:22.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.567 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:22.567 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:40.508763+0000", 2024-09-13T14:42:22.567 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9609000000000001e-05 2024-09-13T14:42:22.567 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.567 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:22.567 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.568 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:168708 6.0 6:0ccbcbda:::benchmark_data_smithi151_155836_object36250:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1106)", 2024-09-13T14:42:22.568 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:41:41.297909+0000", 2024-09-13T14:42:22.568 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 41.147814392999997, 2024-09-13T14:42:22.568 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.10607417600000001, 2024-09-13T14:42:22.568 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:22.568 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:22.568 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:22.568 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:22.568 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:22.568 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:22.568 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 168708 2024-09-13T14:42:22.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.568 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:22.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:22.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:41.297909+0000", 2024-09-13T14:42:22.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:22.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:41.297909+0000", 2024-09-13T14:42:22.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:22.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:41.297911+0000", 2024-09-13T14:42:22.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.46e-06 2024-09-13T14:42:22.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:22.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:41.297917+0000", 2024-09-13T14:42:22.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8359999999999997e-06 2024-09-13T14:42:22.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:22.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:41.297931+0000", 2024-09-13T14:42:22.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4630000000000001e-05 2024-09-13T14:42:22.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:22.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:41.297938+0000", 2024-09-13T14:42:22.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4450000000000001e-06 2024-09-13T14:42:22.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:22.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:41.297996+0000", 2024-09-13T14:42:22.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8474000000000003e-05 2024-09-13T14:42:22.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:22.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:41.298129+0000", 2024-09-13T14:42:22.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013291 2024-09-13T14:42:22.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.572 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:42:22.572 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:41.298259+0000", 2024-09-13T14:42:22.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013005800000000001 2024-09-13T14:42:22.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.572 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:22.572 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:41.303397+0000", 2024-09-13T14:42:22.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0051376429999999999 2024-09-13T14:42:22.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.573 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.573 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:41.304026+0000", 2024-09-13T14:42:22.573 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00062940399999999997 2024-09-13T14:42:22.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.573 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.573 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:41.403904+0000", 2024-09-13T14:42:22.573 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.099878156999999995 2024-09-13T14:42:22.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:22.574 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:41.403944+0000", 2024-09-13T14:42:22.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9820000000000002e-05 2024-09-13T14:42:22.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:22.574 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:41.403983+0000", 2024-09-13T14:42:22.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9338999999999997e-05 2024-09-13T14:42:22.574 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.574 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:22.574 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.575 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:168709 6.0 6:0945acc1:::benchmark_data_smithi151_155836_object36251:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1106)", 2024-09-13T14:42:22.575 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:41:41.299683+0000", 2024-09-13T14:42:22.575 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 41.146040913999997, 2024-09-13T14:42:22.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.104414584, 2024-09-13T14:42:22.575 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:22.575 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:22.575 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:22.575 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:22.575 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:22.575 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:22.575 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 168709 2024-09-13T14:42:22.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.575 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:22.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:22.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:41.299683+0000", 2024-09-13T14:42:22.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:22.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:41.299683+0000", 2024-09-13T14:42:22.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:22.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:41.299683+0000", 2024-09-13T14:42:22.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6000000000000003e-07 2024-09-13T14:42:22.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.577 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:22.577 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:41.299687+0000", 2024-09-13T14:42:22.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8099999999999999e-06 2024-09-13T14:42:22.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.577 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:22.577 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:41.299698+0000", 2024-09-13T14:42:22.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0699e-05 2024-09-13T14:42:22.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.578 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:22.578 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:41.299703+0000", 2024-09-13T14:42:22.578 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4009999999999996e-06 2024-09-13T14:42:22.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.578 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:22.578 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:41.300820+0000", 2024-09-13T14:42:22.578 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011166080000000001 2024-09-13T14:42:22.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.578 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:22.578 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:41.300928+0000", 2024-09-13T14:42:22.578 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010805 2024-09-13T14:42:22.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:42:22.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:41.301000+0000", 2024-09-13T14:42:22.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1808000000000001e-05 2024-09-13T14:42:22.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:22.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:41.303536+0000", 2024-09-13T14:42:22.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0025363730000000002 2024-09-13T14:42:22.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:41.304136+0000", 2024-09-13T14:42:22.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00059938799999999996 2024-09-13T14:42:22.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:41.404029+0000", 2024-09-13T14:42:22.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.099892951999999993 2024-09-13T14:42:22.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:22.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:41.404064+0000", 2024-09-13T14:42:22.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5358999999999998e-05 2024-09-13T14:42:22.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.581 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:22.581 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:41.404097+0000", 2024-09-13T14:42:22.581 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3376e-05 2024-09-13T14:42:22.581 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.581 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:22.581 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.581 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.581 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:169394 6.0 6:02d8c8b4:::benchmark_data_smithi151_155836_object36936:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1106)", 2024-09-13T14:42:22.581 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:41:42.206040+0000", 2024-09-13T14:42:22.581 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.239683581999998, 2024-09-13T14:42:22.581 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.064796333999999997, 2024-09-13T14:42:22.581 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:22.582 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:22.582 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:22.582 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:22.582 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:22.582 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:22.582 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 169394 2024-09-13T14:42:22.582 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.582 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:22.582 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.582 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:22.582 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:42.206040+0000", 2024-09-13T14:42:22.582 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.582 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:22.583 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:42.206040+0000", 2024-09-13T14:42:22.583 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:22.583 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:42.206041+0000", 2024-09-13T14:42:22.583 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3650000000000001e-06 2024-09-13T14:42:22.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:22.583 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:42.206047+0000", 2024-09-13T14:42:22.584 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9460000000000003e-06 2024-09-13T14:42:22.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.584 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:22.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:42.206063+0000", 2024-09-13T14:42:22.584 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5162e-05 2024-09-13T14:42:22.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.584 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:22.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:42.206068+0000", 2024-09-13T14:42:22.584 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8459999999999996e-06 2024-09-13T14:42:22.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.585 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:22.585 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:42.206128+0000", 2024-09-13T14:42:22.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9502000000000001e-05 2024-09-13T14:42:22.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.585 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:22.585 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:42.206267+0000", 2024-09-13T14:42:22.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013899800000000001 2024-09-13T14:42:22.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.585 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:42:22.585 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:42.206402+0000", 2024-09-13T14:42:22.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013543499999999999 2024-09-13T14:42:22.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.586 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.586 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:42.209694+0000", 2024-09-13T14:42:22.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003291876 2024-09-13T14:42:22.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.586 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:22.586 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:42.209788+0000", 2024-09-13T14:42:22.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4149999999999998e-05 2024-09-13T14:42:22.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.586 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.586 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:42.270749+0000", 2024-09-13T14:42:22.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.060960266999999999 2024-09-13T14:42:22.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.587 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:22.587 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:42.270784+0000", 2024-09-13T14:42:22.587 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5281999999999999e-05 2024-09-13T14:42:22.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.587 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:22.587 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:42.270836+0000", 2024-09-13T14:42:22.587 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2505e-05 2024-09-13T14:42:22.587 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.588 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:22.588 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.588 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:186150 6.9 6:9514c1fa:::benchmark_data_smithi151_155836_object53692:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1114)", 2024-09-13T14:42:22.588 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:42:02.009662+0000", 2024-09-13T14:42:22.588 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 20.436061168999998, 2024-09-13T14:42:22.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.073312319000000001, 2024-09-13T14:42:22.588 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:22.588 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:22.588 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:22.588 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:22.588 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:22.589 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:22.589 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 186150 2024-09-13T14:42:22.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.589 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:22.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:22.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.009662+0000", 2024-09-13T14:42:22.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:22.590 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.009662+0000", 2024-09-13T14:42:22.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.590 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:22.590 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.009664+0000", 2024-09-13T14:42:22.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9740000000000001e-06 2024-09-13T14:42:22.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.590 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:22.590 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.009671+0000", 2024-09-13T14:42:22.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8480000000000003e-06 2024-09-13T14:42:22.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.591 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:22.591 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.009690+0000", 2024-09-13T14:42:22.591 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8923000000000002e-05 2024-09-13T14:42:22.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.591 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:22.591 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.009700+0000", 2024-09-13T14:42:22.591 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6439999999999995e-06 2024-09-13T14:42:22.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.591 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:22.591 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.009766+0000", 2024-09-13T14:42:22.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6210000000000005e-05 2024-09-13T14:42:22.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:22.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.009904+0000", 2024-09-13T14:42:22.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000137577 2024-09-13T14:42:22.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,7", 2024-09-13T14:42:22.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.010233+0000", 2024-09-13T14:42:22.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000329353 2024-09-13T14:42:22.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:22.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.012303+0000", 2024-09-13T14:42:22.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020698919999999998 2024-09-13T14:42:22.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.012771+0000", 2024-09-13T14:42:22.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046795499999999999 2024-09-13T14:42:22.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.082800+0000", 2024-09-13T14:42:22.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.070029394999999994 2024-09-13T14:42:22.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:22.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.082874+0000", 2024-09-13T14:42:22.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3658999999999994e-05 2024-09-13T14:42:22.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:22.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.082975+0000", 2024-09-13T14:42:22.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000100889 2024-09-13T14:42:22.594 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.594 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:22.594 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.595 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:186158 6.19 6:9e779b5e:::benchmark_data_smithi151_155836_object53700:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1114)", 2024-09-13T14:42:22.595 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:42:02.016343+0000", 2024-09-13T14:42:22.595 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 20.429380872999999, 2024-09-13T14:42:22.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.066638163, 2024-09-13T14:42:22.595 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:22.595 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:22.595 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:22.595 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:22.595 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:22.595 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:22.595 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 186158 2024-09-13T14:42:22.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.595 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:22.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:22.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.016343+0000", 2024-09-13T14:42:22.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:22.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.016343+0000", 2024-09-13T14:42:22.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:22.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:22.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.016344+0000", 2024-09-13T14:42:22.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.54e-07 2024-09-13T14:42:22.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:22.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.016348+0000", 2024-09-13T14:42:22.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2529999999999998e-06 2024-09-13T14:42:22.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:22.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.016358+0000", 2024-09-13T14:42:22.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0365e-05 2024-09-13T14:42:22.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:22.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.016367+0000", 2024-09-13T14:42:22.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9080000000000001e-06 2024-09-13T14:42:22.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:22.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.016421+0000", 2024-09-13T14:42:22.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3528e-05 2024-09-13T14:42:22.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:22.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.016531+0000", 2024-09-13T14:42:22.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000110691 2024-09-13T14:42:22.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,7", 2024-09-13T14:42:22.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.016844+0000", 2024-09-13T14:42:22.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00031241599999999998 2024-09-13T14:42:22.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.018759+0000", 2024-09-13T14:42:22.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019154179999999999 2024-09-13T14:42:22.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:22.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.018905+0000", 2024-09-13T14:42:22.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000146071 2024-09-13T14:42:22.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.600 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:22.600 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.082819+0000", 2024-09-13T14:42:22.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.063913397999999996 2024-09-13T14:42:22.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.600 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:22.600 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.082945+0000", 2024-09-13T14:42:22.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126489 2024-09-13T14:42:22.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:22.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:22.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:22.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.082981+0000", 2024-09-13T14:42:22.601 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5772000000000002e-05 2024-09-13T14:42:22.601 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.601 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:22.601 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.601 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:22.601 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:22.601 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:42:22.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:22.633 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:22.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:22.734 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:22.760 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:22.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:22.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:22.861 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:22.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:22.899 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:22.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:23.000 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:23.039 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:23.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:23.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:23.141 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:23.178 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:23.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:23.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:23.280 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:23.310 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:23.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:23.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:23.343 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:42:23.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:22 smithi184 ceph-mon[92527]: pgmap v1412: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 3.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 739 op/s; 2045/185478 objects degraded (1.103%); 14998/185478 objects misplaced (8.086%); 524 KiB/s, 8 objects/s recovering 2024-09-13T14:42:23.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:22 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3638445653' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:23.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:22 smithi151 ceph-mon[108966]: pgmap v1412: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 3.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 739 op/s; 2045/185478 objects degraded (1.103%); 14998/185478 objects misplaced (8.086%); 524 KiB/s, 8 objects/s recovering 2024-09-13T14:42:23.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:22 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3638445653' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:23.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:22 smithi151 ceph-mon[104450]: pgmap v1412: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 3.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 739 op/s; 2045/185478 objects degraded (1.103%); 14998/185478 objects misplaced (8.086%); 524 KiB/s, 8 objects/s recovering 2024-09-13T14:42:23.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:22 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3638445653' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:23.412 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:23.448 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:23.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:23.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:23.550 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:23.585 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:23.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:23.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:23.687 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:23.711 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T14:42:23.722 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:23.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:23.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:23.823 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:23.851 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:23.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:23.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:23.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:23.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:23.984 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:23.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:24.085 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:24.115 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:24.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:24.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:24.217 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:24.255 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:24.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:24.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:24.324 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:24.339 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:24.342 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:24.356 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:24.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:24.392 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:24.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:24.493 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:24.519 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:24.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:24.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:24.621 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:24.653 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:24.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:24.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:24.755 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:24.767 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:42:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:42:24] "GET /metrics HTTP/1.1" 200 36684 "" "Prometheus/2.51.0" 2024-09-13T14:42:24.781 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:24.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:24.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:24.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:24.920 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:24.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:24.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:25.022 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:25.050 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:25.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:25.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:25.089 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:24 smithi151 ceph-mon[108966]: pgmap v1413: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 3.5 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.12k op/s; 2045/173556 objects degraded (1.178%); 13995/173556 objects misplaced (8.064%); 462 KiB/s, 7 objects/s recovering 2024-09-13T14:42:25.089 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:24 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 2045/173556 objects degraded (1.178%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:42:25.090 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:24 smithi151 ceph-mon[104450]: pgmap v1413: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 3.5 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.12k op/s; 2045/173556 objects degraded (1.178%); 13995/173556 objects misplaced (8.064%); 462 KiB/s, 7 objects/s recovering 2024-09-13T14:42:25.090 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:24 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 2045/173556 objects degraded (1.178%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:42:25.152 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:25.181 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:25.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:25.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:25.283 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:25.316 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:25.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:25.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:25.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:24 smithi184 ceph-mon[92527]: pgmap v1413: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 3.5 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.12k op/s; 2045/173556 objects degraded (1.178%); 13995/173556 objects misplaced (8.064%); 462 KiB/s, 7 objects/s recovering 2024-09-13T14:42:25.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:24 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 2045/173556 objects degraded (1.178%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:42:25.418 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:25.440 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:25.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:25.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:25.542 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:25.575 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:25.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:25.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:25.677 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:25.707 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:25.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:25.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:25.809 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:25.834 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:25.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:25.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:25.936 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:25.963 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:25.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:25.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:26.065 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:26.091 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:26.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:26.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:26.193 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:26.220 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:26.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:26.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:26.251 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:42:26.322 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:26.354 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:26.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:26.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:26.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:26.487 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:26.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:26.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:26.589 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:26.613 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:26.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:26.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:26.715 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:26.740 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:26.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:26.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:26.842 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:26.881 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:26.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:26.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:26.983 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:27.011 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:27.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:27.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:27.113 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:27.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:27.146 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:27.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:27.247 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:27.282 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:27.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:27.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:27.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:26 smithi184 ceph-mon[92527]: pgmap v1414: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 3.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.04k op/s; 2321/170856 objects degraded (1.358%); 13666/170856 objects misplaced (7.999%); 729 KiB/s, 11 objects/s recovering 2024-09-13T14:42:27.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:26 smithi151 ceph-mon[104450]: pgmap v1414: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 3.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.04k op/s; 2321/170856 objects degraded (1.358%); 13666/170856 objects misplaced (7.999%); 729 KiB/s, 11 objects/s recovering 2024-09-13T14:42:27.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:26 smithi151 ceph-mon[108966]: pgmap v1414: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 3.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.04k op/s; 2321/170856 objects degraded (1.358%); 13666/170856 objects misplaced (7.999%); 729 KiB/s, 11 objects/s recovering 2024-09-13T14:42:27.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:27.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:27.419 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:27.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:27.520 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:27.545 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:27.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:27.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:27.647 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:27.672 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:27.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:27.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:27.773 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:27.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:27.810 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:27.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:27.911 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:27.944 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:27.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:27.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:28.045 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:28.070 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:28.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:28.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:28.099 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:27 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:42:28.100 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:27 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:42:28.172 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:28.195 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:28.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:28.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:28.297 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:28.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:28.352 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:28.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:28.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:27 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:42:28.453 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:28.483 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:28.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:28.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:28.585 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:28.610 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:28.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:28.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:28.712 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:28.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:28.739 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:28.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:28.840 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:28.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:28.867 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:28.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:28.968 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:29.003 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:29.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:29.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:29.106 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:29.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:28 smithi151 ceph-mon[104450]: pgmap v1415: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.09k op/s; 2321/166908 objects degraded (1.391%); 13430/166908 objects misplaced (8.046%); 700 KiB/s, 10 objects/s recovering 2024-09-13T14:42:29.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:28 smithi151 ceph-mon[108966]: pgmap v1415: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.09k op/s; 2321/166908 objects degraded (1.391%); 13430/166908 objects misplaced (8.046%); 700 KiB/s, 10 objects/s recovering 2024-09-13T14:42:29.141 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:29.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:29.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:29.173 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:28 smithi184 ceph-mon[92527]: pgmap v1415: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.09k op/s; 2321/166908 objects degraded (1.391%); 13430/166908 objects misplaced (8.046%); 700 KiB/s, 10 objects/s recovering 2024-09-13T14:42:29.243 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:29.278 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:29.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:29.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:29.379 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:29.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:29.411 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:29.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:29.512 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:29.553 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:29.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:29.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:29.563 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:42:29.565 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T14:42:29.654 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:29.686 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:29.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:29.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:29.788 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:29.812 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:29.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:29.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:29.914 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:29.941 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:29.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:29.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:30.042 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:30.074 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:30.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:30.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:30.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:29 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 2321/166908 objects degraded (1.391%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:42:30.177 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:30.208 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:30.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:30.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:30.310 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:30.335 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:30.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:30.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:30.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:29 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 2321/166908 objects degraded (1.391%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:42:30.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:29 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 2321/166908 objects degraded (1.391%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:42:30.437 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:30.485 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:30.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:30.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:30.587 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:30.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:42:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:42:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:42:30.613 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:30.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:30.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:30.715 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:30.758 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:30.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:30.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:30.860 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:30.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:30.886 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:30.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:30.987 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:31.031 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:31.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:31.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:31.132 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:31.168 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:31.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:31.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:31.270 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:31.283 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:30 smithi151 ceph-mon[108966]: pgmap v1416: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 3.2 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.19k op/s; 2588/155142 objects degraded (1.668%); 12301/155142 objects misplaced (7.929%); 874 KiB/s, 13 objects/s recovering 2024-09-13T14:42:31.283 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:30 smithi151 ceph-mon[108966]: from='client.35520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:42:31.283 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:30 smithi151 ceph-mon[104450]: pgmap v1416: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 3.2 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.19k op/s; 2588/155142 objects degraded (1.668%); 12301/155142 objects misplaced (7.929%); 874 KiB/s, 13 objects/s recovering 2024-09-13T14:42:31.284 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:30 smithi151 ceph-mon[104450]: from='client.35520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:42:31.305 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:31.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:31.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:31.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:30 smithi184 ceph-mon[92527]: pgmap v1416: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 3.2 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.19k op/s; 2588/155142 objects degraded (1.668%); 12301/155142 objects misplaced (7.929%); 874 KiB/s, 13 objects/s recovering 2024-09-13T14:42:31.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:30 smithi184 ceph-mon[92527]: from='client.35520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:42:31.407 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:31.439 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:31.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:31.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:31.541 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:31.568 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:31.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:31.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:31.670 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:31.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:31.698 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:31.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:31.799 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:31.826 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:31.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:31.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:31.928 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:31.967 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:31.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:31.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:32.068 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:32.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:32.106 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:32.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:32.207 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:32.232 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:32.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:32.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:32.335 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:32.362 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:32.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:32.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:32.406 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-13T14:42:32.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:32.493 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:32.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:32.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:32.595 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:32.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:32 smithi184 ceph-mon[92527]: pgmap v1417: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 3.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 894 op/s; 2588/154320 objects degraded (1.677%); 12301/154320 objects misplaced (7.971%); 549 KiB/s, 8 objects/s recovering 2024-09-13T14:42:32.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:42:32.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:32.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:32 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3399793430' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:42:32.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:32 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:42:32.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:32.645 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:32.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:32.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:32.774 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:32.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:32.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:32.782 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:32 smithi151 ceph-mon[104450]: pgmap v1417: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 3.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 894 op/s; 2588/154320 objects degraded (1.677%); 12301/154320 objects misplaced (7.971%); 549 KiB/s, 8 objects/s recovering 2024-09-13T14:42:32.782 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:42:32.783 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:32.783 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:32 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3399793430' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:42:32.783 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:32 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:42:32.783 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:32 smithi151 ceph-mon[108966]: pgmap v1417: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 3.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 894 op/s; 2588/154320 objects degraded (1.677%); 12301/154320 objects misplaced (7.971%); 549 KiB/s, 8 objects/s recovering 2024-09-13T14:42:32.783 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:42:32.783 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:32.783 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:32 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3399793430' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:42:32.783 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:32 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:42:32.876 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:32.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:32.911 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:32.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:33.012 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:33.040 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:33.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:33.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:33.142 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:33.193 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:33.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:33.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:33.295 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:33.332 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:33.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:33.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:33.434 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:33.464 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:33.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:33.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:33.566 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:33.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:33.601 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:33.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:33.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:33 smithi184 ceph-mon[92527]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:42:33.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:33 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:42:33.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:33 smithi184 ceph-mon[92527]: osdmap e1119: 8 total, 8 up, 7 in 2024-09-13T14:42:33.702 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:33.703 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:33 smithi151 ceph-mon[104450]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:42:33.703 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:33 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:42:33.703 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:33 smithi151 ceph-mon[104450]: osdmap e1119: 8 total, 8 up, 7 in 2024-09-13T14:42:33.703 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:33 smithi151 ceph-mon[108966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:42:33.703 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:33 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:42:33.703 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:33 smithi151 ceph-mon[108966]: osdmap e1119: 8 total, 8 up, 7 in 2024-09-13T14:42:33.712 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:33.726 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:42:33.727 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-13T14:42:33.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:33.727 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.35289.0:218966 6.e e1119/1103 6:7e93b665:::benchmark_data_smithi151_155836_object86508:head v 1119'13645, mlcod=1119'13642)", 2024-09-13T14:42:33.727 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:42:33.723740+0000", 2024-09-13T14:42:33.727 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.00074525500000000001, 2024-09-13T14:42:33.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00077365099999999998, 2024-09-13T14:42:33.727 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:33.727 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:33.727 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-13T14:42:33.727 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:33.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:33.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:33.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:33.723740+0000", 2024-09-13T14:42:33.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:33.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:33.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:33.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:33.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:33.723740+0000", 2024-09-13T14:42:33.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:33.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:33.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:33.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:33.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:33.723741+0000", 2024-09-13T14:42:33.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.313e-06 2024-09-13T14:42:33.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:33.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:33.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:33.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:33.723753+0000", 2024-09-13T14:42:33.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2092999999999999e-05 2024-09-13T14:42:33.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:33.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:33.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:33.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:33.723771+0000", 2024-09-13T14:42:33.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7473e-05 2024-09-13T14:42:33.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:33.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:33.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:33.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:33.723781+0000", 2024-09-13T14:42:33.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0154e-05 2024-09-13T14:42:33.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:33.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:33.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:33.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:33.723873+0000", 2024-09-13T14:42:33.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2229000000000001e-05 2024-09-13T14:42:33.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:33.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:33.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:33.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:33.723900+0000", 2024-09-13T14:42:33.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6812e-05 2024-09-13T14:42:33.730 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:33.730 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:33.731 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:33.731 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:33.731 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-13T14:42:33.731 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 1 2024-09-13T14:42:33.731 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:42:33.745 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:33.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:33.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:33.843 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1416,"stamp":"2024-09-13T14:42:29.365147+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3387919031,"num_objects":51714,"num_object_clones":0,"num_object_copies":155142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2588,"num_objects_misplaced":12301,"num_objects_unfound":0,"num_objects_dirty":51714,"num_whiteouts":0,"num_read":80,"num_read_kb":205,"num_write":195382,"num_write_kb":7680444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6424,"num_bytes_recovered":420706085,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":191457,"ondisk_log_size":191457,"up":357,"acting":356,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":357,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":10856144,"kb_used_data":9296856,"kb_used_omap":116,"kb_used_meta":1558667,"kb_avail":645359920,"statfs":{"total":671965249536,"available":660848558080,"internally_reserved":0,"allocated":9519980544,"data_stored":9496823738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":119649,"internal_metadata":1596075167},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,5,5,5,3,1,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-939458560,"num_objects":-14335,"num_object_clones":0,"num_object_copies":-43005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":797,"num_objects_misplaced":-3816,"num_objects_unfound":0,"num_objects_dirty":-14335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14335,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":10747904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004010"},"pg_stats":[{"pgid":"1.11","version":"261'100","reported_seq":47,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.947758+0000","last_change":"2024-09-13T14:42:14.947277+0000","last_active":"2024-09-13T14:42:14.947758+0000","last_peered":"2024-09-13T14:42:14.947758+0000","last_clean":"2024-09-13T14:42:14.947758+0000","last_became_active":"2024-09-13T14:42:14.946932+0000","last_became_peered":"2024-09-13T14:42:14.946932+0000","last_unstale":"2024-09-13T14:42:14.947758+0000","last_undegraded":"2024-09-13T14:42:14.947758+0000","last_fullsized":"2024-09-13T14:42:14.947758+0000","mapping_epoch":1116,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:23.561736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:23.561736+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0091806160000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"1118'11971","reported_seq":12025,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:27.156011+0000","last_change":"2024-09-13T14:41:46.905438+0000","last_active":"2024-09-13T14:42:27.156011+0000","last_peered":"2024-09-13T14:42:27.156011+0000","last_clean":"2024-09-13T14:42:27.156011+0000","last_became_active":"2024-09-13T14:41:46.886923+0000","last_became_peered":"2024-09-13T14:41:46.886923+0000","last_unstale":"2024-09-13T14:42:27.156011+0000","last_undegraded":"2024-09-13T14:42:27.156011+0000","last_fullsized":"2024-09-13T14:42:27.156011+0000","mapping_epoch":1108,"log_start":"1088'4134","ondisk_log_start":"1088'4134","created":1108,"last_epoch_clean":1109,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":7837,"log_dups_size":0,"ondisk_log_size":7837,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107315200,"num_objects":1638,"num_object_clones":0,"num_object_copies":4914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6604,"num_write_kb":263712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1548,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:16.958096+0000","last_change":"2024-09-13T14:41:28.592397+0000","last_active":"2024-09-13T14:42:16.958096+0000","last_peered":"2024-09-13T14:42:16.958096+0000","last_clean":"2024-09-13T14:42:16.958096+0000","last_became_active":"2024-09-13T14:41:28.592068+0000","last_became_peered":"2024-09-13T14:41:28.592068+0000","last_unstale":"2024-09-13T14:42:16.958096+0000","last_undegraded":"2024-09-13T14:42:16.958096+0000","last_fullsized":"2024-09-13T14:42:16.958096+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:08:03.576401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":1074,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.935529+0000","last_change":"2024-09-13T14:29:42.541093+0000","last_active":"2024-09-13T14:42:14.935529+0000","last_peered":"2024-09-13T14:42:14.935529+0000","last_clean":"2024-09-13T14:42:14.935529+0000","last_became_active":"2024-09-13T14:29:42.540671+0000","last_became_peered":"2024-09-13T14:29:42.540671+0000","last_unstale":"2024-09-13T14:42:14.935529+0000","last_undegraded":"2024-09-13T14:42:14.935529+0000","last_fullsized":"2024-09-13T14:42:14.935529+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:45.722376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"261'100","reported_seq":1218,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.939393+0000","last_change":"2024-09-13T14:42:14.933662+0000","last_active":"2024-09-13T14:42:14.939393+0000","last_peered":"2024-09-13T14:42:14.939393+0000","last_clean":"2024-09-13T14:42:14.939393+0000","last_became_active":"2024-09-13T14:42:14.933341+0000","last_became_peered":"2024-09-13T14:42:14.933341+0000","last_unstale":"2024-09-13T14:42:14.939393+0000","last_undegraded":"2024-09-13T14:42:14.939393+0000","last_fullsized":"2024-09-13T14:42:14.939393+0000","mapping_epoch":1116,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:23.257279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:23.257279+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010350875000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"1118'12158","reported_seq":12211,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:27.498211+0000","last_change":"2024-09-13T14:41:46.872175+0000","last_active":"2024-09-13T14:42:27.498211+0000","last_peered":"2024-09-13T14:42:27.498211+0000","last_clean":"2024-09-13T14:42:27.498211+0000","last_became_active":"2024-09-13T14:41:46.843042+0000","last_became_peered":"2024-09-13T14:41:46.843042+0000","last_unstale":"2024-09-13T14:42:27.498211+0000","last_undegraded":"2024-09-13T14:42:27.498211+0000","last_fullsized":"2024-09-13T14:42:27.498211+0000","mapping_epoch":1108,"log_start":"1088'4115","ondisk_log_start":"1088'4115","created":1108,"last_epoch_clean":1109,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":8043,"log_dups_size":0,"ondisk_log_size":8043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106823680,"num_objects":1630,"num_object_clones":0,"num_object_copies":4890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6719,"num_write_kb":267168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1070,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.930188+0000","last_change":"2024-09-13T14:39:11.379707+0000","last_active":"2024-09-13T14:42:14.930188+0000","last_peered":"2024-09-13T14:42:14.930188+0000","last_clean":"2024-09-13T14:42:14.930188+0000","last_became_active":"2024-09-13T14:39:11.379255+0000","last_became_peered":"2024-09-13T14:39:11.379255+0000","last_unstale":"2024-09-13T14:42:14.930188+0000","last_undegraded":"2024-09-13T14:42:14.930188+0000","last_fullsized":"2024-09-13T14:42:14.930188+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:48.543270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1649,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:16.942972+0000","last_change":"2024-09-13T14:41:28.585446+0000","last_active":"2024-09-13T14:42:16.942972+0000","last_peered":"2024-09-13T14:42:16.942972+0000","last_clean":"2024-09-13T14:42:16.942972+0000","last_became_active":"2024-09-13T14:41:28.584679+0000","last_became_peered":"2024-09-13T14:41:28.584679+0000","last_unstale":"2024-09-13T14:42:16.942972+0000","last_undegraded":"2024-09-13T14:42:16.942972+0000","last_fullsized":"2024-09-13T14:42:16.942972+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:19:12.592060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"261'100","reported_seq":356,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.981498+0000","last_change":"2024-09-13T14:42:14.980553+0000","last_active":"2024-09-13T14:42:14.981498+0000","last_peered":"2024-09-13T14:42:14.981498+0000","last_clean":"2024-09-13T14:42:14.981498+0000","last_became_active":"2024-09-13T14:42:14.980267+0000","last_became_peered":"2024-09-13T14:42:14.980267+0000","last_unstale":"2024-09-13T14:42:14.981498+0000","last_undegraded":"2024-09-13T14:42:14.981498+0000","last_fullsized":"2024-09-13T14:42:14.981498+0000","mapping_epoch":1116,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:26.747497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:26.747497+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":23,"num_write":5,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":17890,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"1118'12261","reported_seq":16134,"reported_epoch":1118,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:42:27.544219+0000","last_change":"2024-09-13T14:41:48.816604+0000","last_active":"2024-09-13T14:42:27.544219+0000","last_peered":"2024-09-13T14:42:27.544219+0000","last_clean":"2024-09-13T14:41:27.548643+0000","last_became_active":"2024-09-13T14:41:48.792144+0000","last_became_peered":"2024-09-13T14:41:48.792144+0000","last_unstale":"2024-09-13T14:42:27.544219+0000","last_undegraded":"2024-09-13T14:42:27.544219+0000","last_fullsized":"2024-09-13T14:42:27.544219+0000","mapping_epoch":1110,"log_start":"1088'4255","ondisk_log_start":"1088'4255","created":1108,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":8006,"log_dups_size":0,"ondisk_log_size":8006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108331008,"num_objects":1653,"num_object_clones":0,"num_object_copies":4959,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":762,"num_objects_unfound":0,"num_objects_dirty":1653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6776,"num_write_kb":269728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":947,"num_bytes_recovered":62029824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,1],"avail_no_missing":["5","1","6"],"object_location_counts":[{"shards":"1,5,6","objects":1653}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1609,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:16.942870+0000","last_change":"2024-09-13T14:39:11.394501+0000","last_active":"2024-09-13T14:42:16.942870+0000","last_peered":"2024-09-13T14:42:16.942870+0000","last_clean":"2024-09-13T14:42:16.942870+0000","last_became_active":"2024-09-13T14:39:11.394134+0000","last_became_peered":"2024-09-13T14:39:11.394134+0000","last_unstale":"2024-09-13T14:42:16.942870+0000","last_undegraded":"2024-09-13T14:42:16.942870+0000","last_fullsized":"2024-09-13T14:42:16.942870+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:40:09.359562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1055,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.922069+0000","last_change":"2024-09-13T14:19:25.873088+0000","last_active":"2024-09-13T14:42:14.922069+0000","last_peered":"2024-09-13T14:42:14.922069+0000","last_clean":"2024-09-13T14:42:14.922069+0000","last_became_active":"2024-09-13T14:19:25.872781+0000","last_became_peered":"2024-09-13T14:19:25.872781+0000","last_unstale":"2024-09-13T14:42:14.922069+0000","last_undegraded":"2024-09-13T14:42:14.922069+0000","last_fullsized":"2024-09-13T14:42:14.922069+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:05.607382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"261'100","reported_seq":1227,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.939356+0000","last_change":"2024-09-13T14:42:14.934277+0000","last_active":"2024-09-13T14:42:14.939356+0000","last_peered":"2024-09-13T14:42:14.939356+0000","last_clean":"2024-09-13T14:42:14.939356+0000","last_became_active":"2024-09-13T14:42:14.933957+0000","last_became_peered":"2024-09-13T14:42:14.933957+0000","last_unstale":"2024-09-13T14:42:14.939356+0000","last_undegraded":"2024-09-13T14:42:14.939356+0000","last_fullsized":"2024-09-13T14:42:14.939356+0000","mapping_epoch":1116,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:24.245668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:24.245668+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034439399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"1118'11854","reported_seq":11908,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:27.345927+0000","last_change":"2024-09-13T14:41:46.893140+0000","last_active":"2024-09-13T14:42:27.345927+0000","last_peered":"2024-09-13T14:42:27.345927+0000","last_clean":"2024-09-13T14:42:27.345927+0000","last_became_active":"2024-09-13T14:41:46.874728+0000","last_became_peered":"2024-09-13T14:41:46.874728+0000","last_unstale":"2024-09-13T14:42:27.345927+0000","last_undegraded":"2024-09-13T14:42:27.345927+0000","last_fullsized":"2024-09-13T14:42:27.345927+0000","mapping_epoch":1108,"log_start":"1090'4623","ondisk_log_start":"1090'4623","created":1108,"last_epoch_clean":1109,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":7231,"log_dups_size":0,"ondisk_log_size":7231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":1536,"num_object_clones":0,"num_object_copies":4608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6589,"num_write_kb":260000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":617,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.920331+0000","last_change":"2024-09-13T14:41:28.594485+0000","last_active":"2024-09-13T14:42:14.920331+0000","last_peered":"2024-09-13T14:42:14.920331+0000","last_clean":"2024-09-13T14:42:14.920331+0000","last_became_active":"2024-09-13T14:41:28.594050+0000","last_became_peered":"2024-09-13T14:41:28.594050+0000","last_unstale":"2024-09-13T14:42:14.920331+0000","last_undegraded":"2024-09-13T14:42:14.920331+0000","last_fullsized":"2024-09-13T14:42:14.920331+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:03.838081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1063,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.922886+0000","last_change":"2024-09-13T14:28:35.603724+0000","last_active":"2024-09-13T14:42:14.922886+0000","last_peered":"2024-09-13T14:42:14.922886+0000","last_clean":"2024-09-13T14:42:14.922886+0000","last_became_active":"2024-09-13T14:28:35.603316+0000","last_became_peered":"2024-09-13T14:28:35.603316+0000","last_unstale":"2024-09-13T14:42:14.922886+0000","last_undegraded":"2024-09-13T14:42:14.922886+0000","last_fullsized":"2024-09-13T14:42:14.922886+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:47.668223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"261'100","reported_seq":47,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.939763+0000","last_change":"2024-09-13T14:42:14.933145+0000","last_active":"2024-09-13T14:42:14.939763+0000","last_peered":"2024-09-13T14:42:14.939763+0000","last_clean":"2024-09-13T14:42:14.939763+0000","last_became_active":"2024-09-13T14:42:14.932848+0000","last_became_peered":"2024-09-13T14:42:14.932848+0000","last_unstale":"2024-09-13T14:42:14.939763+0000","last_undegraded":"2024-09-13T14:42:14.939763+0000","last_fullsized":"2024-09-13T14:42:14.939763+0000","mapping_epoch":1116,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:22.892963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:22.892963+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017451785000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"1118'13543","reported_seq":13586,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:27.589582+0000","last_change":"2024-09-13T14:39:16.430434+0000","last_active":"2024-09-13T14:42:27.589582+0000","last_peered":"2024-09-13T14:42:27.589582+0000","last_clean":"2024-09-13T14:42:27.589582+0000","last_became_active":"2024-09-13T14:39:16.430177+0000","last_became_peered":"2024-09-13T14:39:16.430177+0000","last_unstale":"2024-09-13T14:42:27.589582+0000","last_undegraded":"2024-09-13T14:42:27.589582+0000","last_fullsized":"2024-09-13T14:42:27.589582+0000","mapping_epoch":1082,"log_start":"1093'7908","ondisk_log_start":"1093'7908","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":5635,"log_dups_size":0,"ondisk_log_size":5635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:44.232119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212008960,"num_objects":3235,"num_object_clones":0,"num_object_copies":9705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13543,"num_write_kb":536896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":1073,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.920496+0000","last_change":"2024-09-13T14:28:35.608607+0000","last_active":"2024-09-13T14:42:14.920496+0000","last_peered":"2024-09-13T14:42:14.920496+0000","last_clean":"2024-09-13T14:42:14.920496+0000","last_became_active":"2024-09-13T14:28:35.608254+0000","last_became_peered":"2024-09-13T14:28:35.608254+0000","last_unstale":"2024-09-13T14:42:14.920496+0000","last_undegraded":"2024-09-13T14:42:14.920496+0000","last_fullsized":"2024-09-13T14:42:14.920496+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:24.019173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1043,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.930226+0000","last_change":"2024-09-13T14:19:26.887839+0000","last_active":"2024-09-13T14:42:14.930226+0000","last_peered":"2024-09-13T14:42:14.930226+0000","last_clean":"2024-09-13T14:42:14.930226+0000","last_became_active":"2024-09-13T14:19:26.887498+0000","last_became_peered":"2024-09-13T14:19:26.887498+0000","last_unstale":"2024-09-13T14:42:14.930226+0000","last_undegraded":"2024-09-13T14:42:14.930226+0000","last_fullsized":"2024-09-13T14:42:14.930226+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:24.475263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"261'100","reported_seq":52,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.960762+0000","last_change":"2024-09-13T14:42:14.957697+0000","last_active":"2024-09-13T14:42:14.960762+0000","last_peered":"2024-09-13T14:42:14.960762+0000","last_clean":"2024-09-13T14:42:14.960762+0000","last_became_active":"2024-09-13T14:42:14.957387+0000","last_became_peered":"2024-09-13T14:42:14.957387+0000","last_unstale":"2024-09-13T14:42:14.960762+0000","last_undegraded":"2024-09-13T14:42:14.960762+0000","last_fullsized":"2024-09-13T14:42:14.960762+0000","mapping_epoch":1116,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:22.580245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:22.580245+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001821837,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"1118'13466","reported_seq":15166,"reported_epoch":1118,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:42:28.701247+0000","last_change":"2024-09-13T14:41:30.181642+0000","last_active":"2024-09-13T14:42:28.701247+0000","last_peered":"2024-09-13T14:42:28.701247+0000","last_clean":"2024-09-13T14:41:27.568458+0000","last_became_active":"2024-09-13T14:41:29.966413+0000","last_became_peered":"2024-09-13T14:41:29.966413+0000","last_unstale":"2024-09-13T14:42:28.701247+0000","last_undegraded":"2024-09-13T14:41:29.616193+0000","last_fullsized":"2024-09-13T14:41:29.578900+0000","mapping_epoch":1103,"log_start":"1088'3400","ondisk_log_start":"1088'3400","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209190912,"num_objects":3192,"num_object_clones":0,"num_object_copies":9576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2588,"num_objects_misplaced":3192,"num_objects_unfound":0,"num_objects_dirty":3192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13466,"num_write_kb":533056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":808,"num_bytes_recovered":52953088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":3192},{"shards":"3,7","objects":-2588}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":1088,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.921105+0000","last_change":"2024-09-13T14:41:28.598356+0000","last_active":"2024-09-13T14:42:14.921105+0000","last_peered":"2024-09-13T14:42:14.921105+0000","last_clean":"2024-09-13T14:42:14.921105+0000","last_became_active":"2024-09-13T14:41:28.588185+0000","last_became_peered":"2024-09-13T14:41:28.588185+0000","last_unstale":"2024-09-13T14:42:14.921105+0000","last_undegraded":"2024-09-13T14:42:14.921105+0000","last_fullsized":"2024-09-13T14:42:14.921105+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:54.340118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1095,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.925245+0000","last_change":"2024-09-13T14:41:28.594910+0000","last_active":"2024-09-13T14:42:14.925245+0000","last_peered":"2024-09-13T14:42:14.925245+0000","last_clean":"2024-09-13T14:42:14.925245+0000","last_became_active":"2024-09-13T14:41:28.593806+0000","last_became_peered":"2024-09-13T14:41:28.593806+0000","last_unstale":"2024-09-13T14:42:14.925245+0000","last_undegraded":"2024-09-13T14:42:14.925245+0000","last_fullsized":"2024-09-13T14:42:14.925245+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:45:14.253295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"261'100","reported_seq":359,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.952360+0000","last_change":"2024-09-13T14:42:14.950679+0000","last_active":"2024-09-13T14:42:14.952360+0000","last_peered":"2024-09-13T14:42:14.952360+0000","last_clean":"2024-09-13T14:42:14.952360+0000","last_became_active":"2024-09-13T14:42:14.950174+0000","last_became_peered":"2024-09-13T14:42:14.950174+0000","last_unstale":"2024-09-13T14:42:14.952360+0000","last_undegraded":"2024-09-13T14:42:14.952360+0000","last_fullsized":"2024-09-13T14:42:14.952360+0000","mapping_epoch":1116,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:15:42.069756+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"1118'12228","reported_seq":12283,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:27.435944+0000","last_change":"2024-09-13T14:41:46.803069+0000","last_active":"2024-09-13T14:42:27.435944+0000","last_peered":"2024-09-13T14:42:27.435944+0000","last_clean":"2024-09-13T14:42:27.435944+0000","last_became_active":"2024-09-13T14:41:46.783681+0000","last_became_peered":"2024-09-13T14:41:46.783681+0000","last_unstale":"2024-09-13T14:42:27.435944+0000","last_undegraded":"2024-09-13T14:42:27.435944+0000","last_fullsized":"2024-09-13T14:42:27.435944+0000","mapping_epoch":1082,"log_start":"1090'4941","ondisk_log_start":"1090'4941","created":1082,"last_epoch_clean":1109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":7287,"log_dups_size":0,"ondisk_log_size":7287,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109248512,"num_objects":1667,"num_object_clones":0,"num_object_copies":5001,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6768,"num_write_kb":269920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":1056,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.920531+0000","last_change":"2024-09-13T14:19:25.871006+0000","last_active":"2024-09-13T14:42:14.920531+0000","last_peered":"2024-09-13T14:42:14.920531+0000","last_clean":"2024-09-13T14:42:14.920531+0000","last_became_active":"2024-09-13T14:19:25.870713+0000","last_became_peered":"2024-09-13T14:19:25.870713+0000","last_unstale":"2024-09-13T14:42:14.920531+0000","last_undegraded":"2024-09-13T14:42:14.920531+0000","last_fullsized":"2024-09-13T14:42:14.920531+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:34.372124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1064,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.934033+0000","last_change":"2024-09-13T14:41:28.586613+0000","last_active":"2024-09-13T14:42:14.934033+0000","last_peered":"2024-09-13T14:42:14.934033+0000","last_clean":"2024-09-13T14:42:14.934033+0000","last_became_active":"2024-09-13T14:41:28.586345+0000","last_became_peered":"2024-09-13T14:41:28.586345+0000","last_unstale":"2024-09-13T14:42:14.934033+0000","last_undegraded":"2024-09-13T14:42:14.934033+0000","last_fullsized":"2024-09-13T14:42:14.934033+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:37.756544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"261'100","reported_seq":369,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.941370+0000","last_change":"2024-09-13T14:42:14.941081+0000","last_active":"2024-09-13T14:42:14.941370+0000","last_peered":"2024-09-13T14:42:14.941370+0000","last_clean":"2024-09-13T14:42:14.941370+0000","last_became_active":"2024-09-13T14:42:14.940763+0000","last_became_peered":"2024-09-13T14:42:14.940763+0000","last_unstale":"2024-09-13T14:42:14.941370+0000","last_undegraded":"2024-09-13T14:42:14.941370+0000","last_fullsized":"2024-09-13T14:42:14.941370+0000","mapping_epoch":1116,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:25.288979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:25.288979+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040379329999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"1118'12040","reported_seq":12131,"reported_epoch":1118,"state":"active+recovery_wait","last_fresh":"2024-09-13T14:42:28.815796+0000","last_change":"2024-09-13T14:41:46.918979+0000","last_active":"2024-09-13T14:42:28.815796+0000","last_peered":"2024-09-13T14:42:28.815796+0000","last_clean":"2024-09-13T14:41:45.766051+0000","last_became_active":"2024-09-13T14:41:46.868938+0000","last_became_peered":"2024-09-13T14:41:46.868938+0000","last_unstale":"2024-09-13T14:42:28.815796+0000","last_undegraded":"2024-09-13T14:42:28.815796+0000","last_fullsized":"2024-09-13T14:42:28.815796+0000","mapping_epoch":1082,"log_start":"1088'3982","ondisk_log_start":"1088'3982","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":8058,"log_dups_size":0,"ondisk_log_size":8058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103546880,"num_objects":1580,"num_object_clones":0,"num_object_copies":4740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6679,"num_write_kb":264288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":["0","2","7"],"object_location_counts":[{"shards":"0,2,7","objects":1580}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1081,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.930981+0000","last_change":"2024-09-13T14:41:28.594298+0000","last_active":"2024-09-13T14:42:14.930981+0000","last_peered":"2024-09-13T14:42:14.930981+0000","last_clean":"2024-09-13T14:42:14.930981+0000","last_became_active":"2024-09-13T14:41:28.589751+0000","last_became_peered":"2024-09-13T14:41:28.589751+0000","last_unstale":"2024-09-13T14:42:14.930981+0000","last_undegraded":"2024-09-13T14:42:14.930981+0000","last_fullsized":"2024-09-13T14:42:14.930981+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:15:07.568716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1610,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:16.942718+0000","last_change":"2024-09-13T14:41:28.587900+0000","last_active":"2024-09-13T14:42:16.942718+0000","last_peered":"2024-09-13T14:42:16.942718+0000","last_clean":"2024-09-13T14:42:16.942718+0000","last_became_active":"2024-09-13T14:41:28.587563+0000","last_became_peered":"2024-09-13T14:41:28.587563+0000","last_unstale":"2024-09-13T14:42:16.942718+0000","last_undegraded":"2024-09-13T14:42:16.942718+0000","last_fullsized":"2024-09-13T14:42:16.942718+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:19:15.384955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"261'100","reported_seq":1218,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.922181+0000","last_change":"2024-09-13T14:41:28.268071+0000","last_active":"2024-09-13T14:42:14.922181+0000","last_peered":"2024-09-13T14:42:14.922181+0000","last_clean":"2024-09-13T14:42:14.922181+0000","last_became_active":"2024-09-13T14:41:20.510300+0000","last_became_peered":"2024-09-13T14:41:20.510300+0000","last_unstale":"2024-09-13T14:42:14.922181+0000","last_undegraded":"2024-09-13T14:42:14.922181+0000","last_fullsized":"2024-09-13T14:42:14.922181+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1100,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:28.268005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:28.268005+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:10.225185+0000","scrub_duration":0.0015413810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"1118'13272","reported_seq":17625,"reported_epoch":1118,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:42:27.338851+0000","last_change":"2024-09-13T14:41:46.903219+0000","last_active":"2024-09-13T14:42:27.338851+0000","last_peered":"2024-09-13T14:42:27.338851+0000","last_clean":"2024-09-13T14:41:27.545589+0000","last_became_active":"2024-09-13T14:41:29.700570+0000","last_became_peered":"2024-09-13T14:41:29.700570+0000","last_unstale":"2024-09-13T14:42:27.338851+0000","last_undegraded":"2024-09-13T14:42:27.338851+0000","last_fullsized":"2024-09-13T14:42:27.338851+0000","mapping_epoch":1103,"log_start":"1093'7776","ondisk_log_start":"1093'7776","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":5496,"log_dups_size":0,"ondisk_log_size":5496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:00:18.887770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213909504,"num_objects":3264,"num_object_clones":0,"num_object_copies":9792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2137,"num_objects_unfound":0,"num_objects_dirty":3264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13272,"num_write_kb":529152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2146,"num_bytes_recovered":140640256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,1],"avail_no_missing":["4","1","2"],"object_location_counts":[{"shards":"1,2,4","objects":3264}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":145,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:16.943002+0000","last_change":"2024-09-13T14:41:28.600760+0000","last_active":"2024-09-13T14:42:16.943002+0000","last_peered":"2024-09-13T14:42:16.943002+0000","last_clean":"2024-09-13T14:42:16.943002+0000","last_became_active":"2024-09-13T14:41:28.586678+0000","last_became_peered":"2024-09-13T14:41:28.586678+0000","last_unstale":"2024-09-13T14:42:16.943002+0000","last_undegraded":"2024-09-13T14:42:16.943002+0000","last_fullsized":"2024-09-13T14:42:16.943002+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:44:53.564449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1078,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.926120+0000","last_change":"2024-09-13T14:41:28.598046+0000","last_active":"2024-09-13T14:42:14.926120+0000","last_peered":"2024-09-13T14:42:14.926120+0000","last_clean":"2024-09-13T14:42:14.926120+0000","last_became_active":"2024-09-13T14:41:28.597748+0000","last_became_peered":"2024-09-13T14:41:28.597748+0000","last_unstale":"2024-09-13T14:42:14.926120+0000","last_undegraded":"2024-09-13T14:42:14.926120+0000","last_fullsized":"2024-09-13T14:42:14.926120+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:52.920256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"261'100","reported_seq":33,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.930046+0000","last_change":"2024-09-13T14:41:20.562986+0000","last_active":"2024-09-13T14:42:14.930046+0000","last_peered":"2024-09-13T14:42:14.930046+0000","last_clean":"2024-09-13T14:42:14.930046+0000","last_became_active":"2024-09-13T14:41:20.529244+0000","last_became_peered":"2024-09-13T14:41:20.529244+0000","last_unstale":"2024-09-13T14:42:14.930046+0000","last_undegraded":"2024-09-13T14:42:14.930046+0000","last_fullsized":"2024-09-13T14:42:14.930046+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1100,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:15:42.069756+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":316358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"1118'13106","reported_seq":13192,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:25.687758+0000","last_change":"2024-09-13T14:39:16.433163+0000","last_active":"2024-09-13T14:42:25.687758+0000","last_peered":"2024-09-13T14:42:25.687758+0000","last_clean":"2024-09-13T14:42:25.687758+0000","last_became_active":"2024-09-13T14:39:16.432783+0000","last_became_peered":"2024-09-13T14:39:16.432783+0000","last_unstale":"2024-09-13T14:42:25.687758+0000","last_undegraded":"2024-09-13T14:42:25.687758+0000","last_fullsized":"2024-09-13T14:42:25.687758+0000","mapping_epoch":1082,"log_start":"1092'6899","ondisk_log_start":"1092'6899","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6207,"log_dups_size":0,"ondisk_log_size":6207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:35:05.068064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214040576,"num_objects":3266,"num_object_clones":0,"num_object_copies":9798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13106,"num_write_kb":523904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1065,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.934182+0000","last_change":"2024-09-13T14:41:28.591027+0000","last_active":"2024-09-13T14:42:14.934182+0000","last_peered":"2024-09-13T14:42:14.934182+0000","last_clean":"2024-09-13T14:42:14.934182+0000","last_became_active":"2024-09-13T14:41:28.590742+0000","last_became_peered":"2024-09-13T14:41:28.590742+0000","last_unstale":"2024-09-13T14:42:14.934182+0000","last_undegraded":"2024-09-13T14:42:14.934182+0000","last_fullsized":"2024-09-13T14:42:14.934182+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:01.865182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1095,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.926364+0000","last_change":"2024-09-13T14:41:28.588833+0000","last_active":"2024-09-13T14:42:14.926364+0000","last_peered":"2024-09-13T14:42:14.926364+0000","last_clean":"2024-09-13T14:42:14.926364+0000","last_became_active":"2024-09-13T14:41:28.588490+0000","last_became_peered":"2024-09-13T14:41:28.588490+0000","last_unstale":"2024-09-13T14:42:14.926364+0000","last_undegraded":"2024-09-13T14:42:14.926364+0000","last_fullsized":"2024-09-13T14:42:14.926364+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:23.440459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"261'100","reported_seq":356,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:15.000194+0000","last_change":"2024-09-13T14:42:14.987318+0000","last_active":"2024-09-13T14:42:15.000194+0000","last_peered":"2024-09-13T14:42:15.000194+0000","last_clean":"2024-09-13T14:42:15.000194+0000","last_became_active":"2024-09-13T14:42:14.986883+0000","last_became_peered":"2024-09-13T14:42:14.986883+0000","last_unstale":"2024-09-13T14:42:15.000194+0000","last_undegraded":"2024-09-13T14:42:15.000194+0000","last_fullsized":"2024-09-13T14:42:15.000194+0000","mapping_epoch":1116,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:26.747497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:26.747497+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":23,"num_write":5,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":17890,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"1118'13330","reported_seq":13374,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:27.496105+0000","last_change":"2024-09-13T14:39:16.428687+0000","last_active":"2024-09-13T14:42:27.496105+0000","last_peered":"2024-09-13T14:42:27.496105+0000","last_clean":"2024-09-13T14:42:27.496105+0000","last_became_active":"2024-09-13T14:39:16.428382+0000","last_became_peered":"2024-09-13T14:39:16.428382+0000","last_unstale":"2024-09-13T14:42:27.496105+0000","last_undegraded":"2024-09-13T14:42:27.496105+0000","last_fullsized":"2024-09-13T14:42:27.496105+0000","mapping_epoch":1082,"log_start":"1095'9096","ondisk_log_start":"1095'9096","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":4234,"log_dups_size":0,"ondisk_log_size":4234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:33:44.732564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213516288,"num_objects":3258,"num_object_clones":0,"num_object_copies":9774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13330,"num_write_kb":530816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1059,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.925317+0000","last_change":"2024-09-13T14:29:42.545815+0000","last_active":"2024-09-13T14:42:14.925317+0000","last_peered":"2024-09-13T14:42:14.925317+0000","last_clean":"2024-09-13T14:42:14.925317+0000","last_became_active":"2024-09-13T14:29:42.545508+0000","last_became_peered":"2024-09-13T14:29:42.545508+0000","last_unstale":"2024-09-13T14:42:14.925317+0000","last_undegraded":"2024-09-13T14:42:14.925317+0000","last_fullsized":"2024-09-13T14:42:14.925317+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:39.678381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1068,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.932453+0000","last_change":"2024-09-13T14:41:28.595541+0000","last_active":"2024-09-13T14:42:14.932453+0000","last_peered":"2024-09-13T14:42:14.932453+0000","last_clean":"2024-09-13T14:42:14.932453+0000","last_became_active":"2024-09-13T14:41:28.593863+0000","last_became_peered":"2024-09-13T14:41:28.593863+0000","last_unstale":"2024-09-13T14:42:14.932453+0000","last_undegraded":"2024-09-13T14:42:14.932453+0000","last_fullsized":"2024-09-13T14:42:14.932453+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:06.730588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"261'100","reported_seq":38,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.930203+0000","last_change":"2024-09-13T14:41:29.624236+0000","last_active":"2024-09-13T14:42:14.930203+0000","last_peered":"2024-09-13T14:42:14.930203+0000","last_clean":"2024-09-13T14:42:14.930203+0000","last_became_active":"2024-09-13T14:41:20.530755+0000","last_became_peered":"2024-09-13T14:41:20.530755+0000","last_unstale":"2024-09-13T14:42:14.930203+0000","last_undegraded":"2024-09-13T14:42:14.930203+0000","last_fullsized":"2024-09-13T14:42:14.930203+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1100,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:29.624129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:29.624129+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:16.408396+0000","scrub_duration":0.0076481020000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"1118'13300","reported_seq":13987,"reported_epoch":1118,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:42:27.568859+0000","last_change":"2024-09-13T14:41:46.889575+0000","last_active":"2024-09-13T14:42:27.568859+0000","last_peered":"2024-09-13T14:42:27.568859+0000","last_clean":"2024-09-13T14:41:27.579452+0000","last_became_active":"2024-09-13T14:41:29.706889+0000","last_became_peered":"2024-09-13T14:41:29.706889+0000","last_unstale":"2024-09-13T14:42:27.568859+0000","last_undegraded":"2024-09-13T14:42:27.568859+0000","last_fullsized":"2024-09-13T14:42:27.568859+0000","mapping_epoch":1103,"log_start":"1093'7744","ondisk_log_start":"1093'7744","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":5556,"log_dups_size":0,"ondisk_log_size":5556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:38:40.997498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210305048,"num_objects":3210,"num_object_clones":0,"num_object_copies":9630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3034,"num_objects_unfound":0,"num_objects_dirty":3210,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13300,"num_write_kb":528257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":312,"num_bytes_recovered":20447232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,1],"avail_no_missing":["5","0","1"],"object_location_counts":[{"shards":"0,1,5","objects":3210}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":587,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.918667+0000","last_change":"2024-09-13T14:41:28.587940+0000","last_active":"2024-09-13T14:42:14.918667+0000","last_peered":"2024-09-13T14:42:14.918667+0000","last_clean":"2024-09-13T14:42:14.918667+0000","last_became_active":"2024-09-13T14:41:28.587643+0000","last_became_peered":"2024-09-13T14:41:28.587643+0000","last_unstale":"2024-09-13T14:42:14.918667+0000","last_undegraded":"2024-09-13T14:42:14.918667+0000","last_fullsized":"2024-09-13T14:42:14.918667+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:09:23.651717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":639,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:16.959394+0000","last_change":"2024-09-13T14:41:28.592799+0000","last_active":"2024-09-13T14:42:16.959394+0000","last_peered":"2024-09-13T14:42:16.959394+0000","last_clean":"2024-09-13T14:42:16.959394+0000","last_became_active":"2024-09-13T14:41:28.592345+0000","last_became_peered":"2024-09-13T14:41:28.592345+0000","last_unstale":"2024-09-13T14:42:16.959394+0000","last_undegraded":"2024-09-13T14:42:16.959394+0000","last_fullsized":"2024-09-13T14:42:16.959394+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:17.651032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"1118'12160","reported_seq":12214,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:27.504094+0000","last_change":"2024-09-13T14:41:46.836890+0000","last_active":"2024-09-13T14:42:27.504094+0000","last_peered":"2024-09-13T14:42:27.504094+0000","last_clean":"2024-09-13T14:42:27.504094+0000","last_became_active":"2024-09-13T14:41:46.813650+0000","last_became_peered":"2024-09-13T14:41:46.813650+0000","last_unstale":"2024-09-13T14:42:27.504094+0000","last_undegraded":"2024-09-13T14:42:27.504094+0000","last_fullsized":"2024-09-13T14:42:27.504094+0000","mapping_epoch":1082,"log_start":"1088'4115","ondisk_log_start":"1088'4115","created":1082,"last_epoch_clean":1109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":8045,"log_dups_size":0,"ondisk_log_size":8045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106692608,"num_objects":1628,"num_object_clones":0,"num_object_copies":4884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6721,"num_write_kb":267168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"261'100","reported_seq":1219,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.929248+0000","last_change":"2024-09-13T14:42:14.928093+0000","last_active":"2024-09-13T14:42:14.929248+0000","last_peered":"2024-09-13T14:42:14.929248+0000","last_clean":"2024-09-13T14:42:14.929248+0000","last_became_active":"2024-09-13T14:42:14.927771+0000","last_became_peered":"2024-09-13T14:42:14.927771+0000","last_unstale":"2024-09-13T14:42:14.929248+0000","last_undegraded":"2024-09-13T14:42:14.929248+0000","last_fullsized":"2024-09-13T14:42:14.929248+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:23.257279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:23.257279+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010350875000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":1074,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.929728+0000","last_change":"2024-09-13T14:29:42.539455+0000","last_active":"2024-09-13T14:42:14.929728+0000","last_peered":"2024-09-13T14:42:14.929728+0000","last_clean":"2024-09-13T14:42:14.929728+0000","last_became_active":"2024-09-13T14:29:42.539068+0000","last_became_peered":"2024-09-13T14:29:42.539068+0000","last_unstale":"2024-09-13T14:42:14.929728+0000","last_undegraded":"2024-09-13T14:42:14.929728+0000","last_fullsized":"2024-09-13T14:42:14.929728+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:24:06.048855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034405999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":585,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.930409+0000","last_change":"2024-09-13T14:39:11.382555+0000","last_active":"2024-09-13T14:42:14.930409+0000","last_peered":"2024-09-13T14:42:14.930409+0000","last_clean":"2024-09-13T14:42:14.930409+0000","last_became_active":"2024-09-13T14:39:11.382247+0000","last_became_peered":"2024-09-13T14:39:11.382247+0000","last_unstale":"2024-09-13T14:42:14.930409+0000","last_undegraded":"2024-09-13T14:42:14.930409+0000","last_fullsized":"2024-09-13T14:42:14.930409+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:09:48.016987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1037,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.932665+0000","last_change":"2024-09-13T14:19:27.379312+0000","last_active":"2024-09-13T14:42:14.932665+0000","last_peered":"2024-09-13T14:42:14.932665+0000","last_clean":"2024-09-13T14:42:14.932665+0000","last_became_active":"2024-09-13T14:19:27.378947+0000","last_became_peered":"2024-09-13T14:19:27.378947+0000","last_unstale":"2024-09-13T14:42:14.932665+0000","last_undegraded":"2024-09-13T14:42:14.932665+0000","last_fullsized":"2024-09-13T14:42:14.932665+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:54:20.706578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"261'100","reported_seq":357,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.980026+0000","last_change":"2024-09-13T14:42:14.974324+0000","last_active":"2024-09-13T14:42:14.980026+0000","last_peered":"2024-09-13T14:42:14.980026+0000","last_clean":"2024-09-13T14:42:14.980026+0000","last_became_active":"2024-09-13T14:42:14.973976+0000","last_became_peered":"2024-09-13T14:42:14.973976+0000","last_unstale":"2024-09-13T14:42:14.980026+0000","last_undegraded":"2024-09-13T14:42:14.980026+0000","last_fullsized":"2024-09-13T14:42:14.980026+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:26.747497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:26.747497+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":22,"num_write":4,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":17890,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"1118'12259","reported_seq":18505,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:27.521725+0000","last_change":"2024-09-13T14:41:52.861670+0000","last_active":"2024-09-13T14:42:27.521725+0000","last_peered":"2024-09-13T14:42:27.521725+0000","last_clean":"2024-09-13T14:42:27.521725+0000","last_became_active":"2024-09-13T14:41:52.829864+0000","last_became_peered":"2024-09-13T14:41:52.829864+0000","last_unstale":"2024-09-13T14:42:27.521725+0000","last_undegraded":"2024-09-13T14:42:27.521725+0000","last_fullsized":"2024-09-13T14:42:27.521725+0000","mapping_epoch":1112,"log_start":"1088'4255","ondisk_log_start":"1088'4255","created":1082,"last_epoch_clean":1113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":8004,"log_dups_size":0,"ondisk_log_size":8004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108462080,"num_objects":1655,"num_object_clones":0,"num_object_copies":4965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6774,"num_write_kb":269728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2126,"num_bytes_recovered":139362304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1111,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.918735+0000","last_change":"2024-09-13T14:41:28.587183+0000","last_active":"2024-09-13T14:42:14.918735+0000","last_peered":"2024-09-13T14:42:14.918735+0000","last_clean":"2024-09-13T14:42:14.918735+0000","last_became_active":"2024-09-13T14:41:28.586268+0000","last_became_peered":"2024-09-13T14:41:28.586268+0000","last_unstale":"2024-09-13T14:42:14.918735+0000","last_undegraded":"2024-09-13T14:42:14.918735+0000","last_fullsized":"2024-09-13T14:42:14.918735+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:34.203731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1643,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:16.942542+0000","last_change":"2024-09-13T14:39:11.390055+0000","last_active":"2024-09-13T14:42:16.942542+0000","last_peered":"2024-09-13T14:42:16.942542+0000","last_clean":"2024-09-13T14:42:16.942542+0000","last_became_active":"2024-09-13T14:39:11.389697+0000","last_became_peered":"2024-09-13T14:39:11.389697+0000","last_unstale":"2024-09-13T14:42:16.942542+0000","last_undegraded":"2024-09-13T14:42:16.942542+0000","last_fullsized":"2024-09-13T14:42:16.942542+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:50.975739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1621,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:16.942627+0000","last_change":"2024-09-13T14:41:28.584816+0000","last_active":"2024-09-13T14:42:16.942627+0000","last_peered":"2024-09-13T14:42:16.942627+0000","last_clean":"2024-09-13T14:42:16.942627+0000","last_became_active":"2024-09-13T14:41:28.583332+0000","last_became_peered":"2024-09-13T14:41:28.583332+0000","last_unstale":"2024-09-13T14:42:16.942627+0000","last_undegraded":"2024-09-13T14:42:16.942627+0000","last_fullsized":"2024-09-13T14:42:16.942627+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:20.608574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"261'100","reported_seq":1228,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.932466+0000","last_change":"2024-09-13T14:42:14.929239+0000","last_active":"2024-09-13T14:42:14.932466+0000","last_peered":"2024-09-13T14:42:14.932466+0000","last_clean":"2024-09-13T14:42:14.932466+0000","last_became_active":"2024-09-13T14:42:14.928923+0000","last_became_peered":"2024-09-13T14:42:14.928923+0000","last_unstale":"2024-09-13T14:42:14.932466+0000","last_undegraded":"2024-09-13T14:42:14.932466+0000","last_fullsized":"2024-09-13T14:42:14.932466+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:24.245668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:24.245668+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034439399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"1118'11755","reported_seq":11810,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:27.364456+0000","last_change":"2024-09-13T14:41:46.859677+0000","last_active":"2024-09-13T14:42:27.364456+0000","last_peered":"2024-09-13T14:42:27.364456+0000","last_clean":"2024-09-13T14:42:27.364456+0000","last_became_active":"2024-09-13T14:41:46.803818+0000","last_became_peered":"2024-09-13T14:41:46.803818+0000","last_unstale":"2024-09-13T14:42:27.364456+0000","last_undegraded":"2024-09-13T14:42:27.364456+0000","last_fullsized":"2024-09-13T14:42:27.364456+0000","mapping_epoch":1082,"log_start":"1090'4623","ondisk_log_start":"1090'4623","created":1082,"last_epoch_clean":1109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":7132,"log_dups_size":0,"ondisk_log_size":7132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107151360,"num_objects":1635,"num_object_clones":0,"num_object_copies":4905,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6490,"num_write_kb":260000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1620,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:16.951486+0000","last_change":"2024-09-13T14:41:28.585065+0000","last_active":"2024-09-13T14:42:16.951486+0000","last_peered":"2024-09-13T14:42:16.951486+0000","last_clean":"2024-09-13T14:42:16.951486+0000","last_became_active":"2024-09-13T14:41:28.584754+0000","last_became_peered":"2024-09-13T14:41:28.584754+0000","last_unstale":"2024-09-13T14:42:16.951486+0000","last_undegraded":"2024-09-13T14:42:16.951486+0000","last_fullsized":"2024-09-13T14:42:16.951486+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:20:35.484829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":573,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:16.948228+0000","last_change":"2024-09-13T14:41:28.590194+0000","last_active":"2024-09-13T14:42:16.948228+0000","last_peered":"2024-09-13T14:42:16.948228+0000","last_clean":"2024-09-13T14:42:16.948228+0000","last_became_active":"2024-09-13T14:41:28.589774+0000","last_became_peered":"2024-09-13T14:41:28.589774+0000","last_unstale":"2024-09-13T14:42:16.948228+0000","last_undegraded":"2024-09-13T14:42:16.948228+0000","last_fullsized":"2024-09-13T14:42:16.948228+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:36.382030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":515,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.940632+0000","last_change":"2024-09-13T14:41:28.597828+0000","last_active":"2024-09-13T14:42:14.940632+0000","last_peered":"2024-09-13T14:42:14.940632+0000","last_clean":"2024-09-13T14:42:14.940632+0000","last_became_active":"2024-09-13T14:41:28.597540+0000","last_became_peered":"2024-09-13T14:41:28.597540+0000","last_unstale":"2024-09-13T14:42:14.940632+0000","last_undegraded":"2024-09-13T14:42:14.940632+0000","last_fullsized":"2024-09-13T14:42:14.940632+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:49:04.992589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"261'100","reported_seq":48,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.947739+0000","last_change":"2024-09-13T14:42:14.947213+0000","last_active":"2024-09-13T14:42:14.947739+0000","last_peered":"2024-09-13T14:42:14.947739+0000","last_clean":"2024-09-13T14:42:14.947739+0000","last_became_active":"2024-09-13T14:42:14.946591+0000","last_became_peered":"2024-09-13T14:42:14.946591+0000","last_unstale":"2024-09-13T14:42:14.947739+0000","last_undegraded":"2024-09-13T14:42:14.947739+0000","last_fullsized":"2024-09-13T14:42:14.947739+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:23.561736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:23.561736+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0091806160000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"1118'11981","reported_seq":12036,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:27.332124+0000","last_change":"2024-09-13T14:41:46.873839+0000","last_active":"2024-09-13T14:42:27.332124+0000","last_peered":"2024-09-13T14:42:27.332124+0000","last_clean":"2024-09-13T14:42:27.332124+0000","last_became_active":"2024-09-13T14:41:46.861555+0000","last_became_peered":"2024-09-13T14:41:46.861555+0000","last_unstale":"2024-09-13T14:42:27.332124+0000","last_undegraded":"2024-09-13T14:42:27.332124+0000","last_fullsized":"2024-09-13T14:42:27.332124+0000","mapping_epoch":1082,"log_start":"1088'4134","ondisk_log_start":"1088'4134","created":1082,"last_epoch_clean":1109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":7847,"log_dups_size":0,"ondisk_log_size":7847,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106659840,"num_objects":1627,"num_object_clones":0,"num_object_copies":4881,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6613,"num_write_kb":263712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":1092,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.930523+0000","last_change":"2024-09-13T14:41:28.604178+0000","last_active":"2024-09-13T14:42:14.930523+0000","last_peered":"2024-09-13T14:42:14.930523+0000","last_clean":"2024-09-13T14:42:14.930523+0000","last_became_active":"2024-09-13T14:41:28.587558+0000","last_became_peered":"2024-09-13T14:41:28.587558+0000","last_unstale":"2024-09-13T14:42:14.930523+0000","last_undegraded":"2024-09-13T14:42:14.930523+0000","last_fullsized":"2024-09-13T14:42:14.930523+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:04:42.789023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1059,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.925479+0000","last_change":"2024-09-13T14:28:35.604404+0000","last_active":"2024-09-13T14:42:14.925479+0000","last_peered":"2024-09-13T14:42:14.925479+0000","last_clean":"2024-09-13T14:42:14.925479+0000","last_became_active":"2024-09-13T14:28:35.604102+0000","last_became_peered":"2024-09-13T14:28:35.604102+0000","last_unstale":"2024-09-13T14:42:14.925479+0000","last_undegraded":"2024-09-13T14:42:14.925479+0000","last_fullsized":"2024-09-13T14:42:14.925479+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:25.546010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1554,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:16.948692+0000","last_change":"2024-09-13T14:41:28.597520+0000","last_active":"2024-09-13T14:42:16.948692+0000","last_peered":"2024-09-13T14:42:16.948692+0000","last_clean":"2024-09-13T14:42:16.948692+0000","last_became_active":"2024-09-13T14:41:28.597166+0000","last_became_peered":"2024-09-13T14:41:28.597166+0000","last_unstale":"2024-09-13T14:42:16.948692+0000","last_undegraded":"2024-09-13T14:42:16.948692+0000","last_fullsized":"2024-09-13T14:42:16.948692+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:18:57.954799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"261'100","reported_seq":360,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.952338+0000","last_change":"2024-09-13T14:42:14.948814+0000","last_active":"2024-09-13T14:42:14.952338+0000","last_peered":"2024-09-13T14:42:14.952338+0000","last_clean":"2024-09-13T14:42:14.952338+0000","last_became_active":"2024-09-13T14:42:14.948392+0000","last_became_peered":"2024-09-13T14:42:14.948392+0000","last_unstale":"2024-09-13T14:42:14.952338+0000","last_undegraded":"2024-09-13T14:42:14.952338+0000","last_fullsized":"2024-09-13T14:42:14.952338+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:15:42.069756+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":34,"num_write":6,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"1118'12188","reported_seq":12322,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:28.705594+0000","last_change":"2024-09-13T14:42:04.496730+0000","last_active":"2024-09-13T14:42:28.705594+0000","last_peered":"2024-09-13T14:42:28.705594+0000","last_clean":"2024-09-13T14:42:28.705594+0000","last_became_active":"2024-09-13T14:41:46.802094+0000","last_became_peered":"2024-09-13T14:41:46.802094+0000","last_unstale":"2024-09-13T14:42:28.705594+0000","last_undegraded":"2024-09-13T14:42:28.705594+0000","last_fullsized":"2024-09-13T14:42:28.705594+0000","mapping_epoch":1082,"log_start":"1088'4216","ondisk_log_start":"1088'4216","created":1082,"last_epoch_clean":1109,"parent":"0.0","parent_split_bits":0,"last_scrub":"1114'11415","last_scrub_stamp":"2024-09-13T14:42:04.496642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:42:04.496642+0000","objects_scrubbed":2429,"log_size":7972,"log_dups_size":0,"ondisk_log_size":7972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:33.734025+0000","scrub_duration":552,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108199936,"num_objects":1651,"num_object_clones":0,"num_object_copies":4953,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":778,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1088,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.930353+0000","last_change":"2024-09-13T14:39:11.383045+0000","last_active":"2024-09-13T14:42:14.930353+0000","last_peered":"2024-09-13T14:42:14.930353+0000","last_clean":"2024-09-13T14:42:14.930353+0000","last_became_active":"2024-09-13T14:39:11.382718+0000","last_became_peered":"2024-09-13T14:39:11.382718+0000","last_unstale":"2024-09-13T14:42:14.930353+0000","last_undegraded":"2024-09-13T14:42:14.930353+0000","last_fullsized":"2024-09-13T14:42:14.930353+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:21.812296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":554,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:16.942392+0000","last_change":"2024-09-13T14:41:28.585118+0000","last_active":"2024-09-13T14:42:16.942392+0000","last_peered":"2024-09-13T14:42:16.942392+0000","last_clean":"2024-09-13T14:42:16.942392+0000","last_became_active":"2024-09-13T14:41:28.584804+0000","last_became_peered":"2024-09-13T14:41:28.584804+0000","last_unstale":"2024-09-13T14:42:16.942392+0000","last_undegraded":"2024-09-13T14:42:16.942392+0000","last_fullsized":"2024-09-13T14:42:16.942392+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:10:27.164531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":481,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.930481+0000","last_change":"2024-09-13T14:41:28.588613+0000","last_active":"2024-09-13T14:42:14.930481+0000","last_peered":"2024-09-13T14:42:14.930481+0000","last_clean":"2024-09-13T14:42:14.930481+0000","last_became_active":"2024-09-13T14:41:28.587982+0000","last_became_peered":"2024-09-13T14:41:28.587982+0000","last_unstale":"2024-09-13T14:42:14.930481+0000","last_undegraded":"2024-09-13T14:42:14.930481+0000","last_fullsized":"2024-09-13T14:42:14.930481+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:51:16.814445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"261'100","reported_seq":370,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.941265+0000","last_change":"2024-09-13T14:42:14.941034+0000","last_active":"2024-09-13T14:42:14.941265+0000","last_peered":"2024-09-13T14:42:14.941265+0000","last_clean":"2024-09-13T14:42:14.941265+0000","last_became_active":"2024-09-13T14:42:14.940703+0000","last_became_peered":"2024-09-13T14:42:14.940703+0000","last_unstale":"2024-09-13T14:42:14.941265+0000","last_undegraded":"2024-09-13T14:42:14.941265+0000","last_fullsized":"2024-09-13T14:42:14.941265+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:25.288979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:25.288979+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040379329999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"1118'11918","reported_seq":12157,"reported_epoch":1118,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:42:27.492461+0000","last_change":"2024-09-13T14:41:46.912020+0000","last_active":"2024-09-13T14:42:27.492461+0000","last_peered":"2024-09-13T14:42:27.492461+0000","last_clean":"2024-09-13T14:41:27.528697+0000","last_became_active":"2024-09-13T14:41:46.899012+0000","last_became_peered":"2024-09-13T14:41:46.899012+0000","last_unstale":"2024-09-13T14:42:27.492461+0000","last_undegraded":"2024-09-13T14:42:27.492461+0000","last_fullsized":"2024-09-13T14:42:27.492461+0000","mapping_epoch":1103,"log_start":"1090'5198","ondisk_log_start":"1090'5198","created":1082,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6720,"log_dups_size":0,"ondisk_log_size":6720,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100204544,"num_objects":1529,"num_object_clones":0,"num_object_copies":4587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1529,"num_objects_unfound":0,"num_objects_dirty":1529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6628,"num_write_kb":261024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":2424832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,1],"avail_no_missing":["6","1","2"],"object_location_counts":[{"shards":"1,2,6","objects":1529}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":573,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:16.948383+0000","last_change":"2024-09-13T14:41:28.593821+0000","last_active":"2024-09-13T14:42:16.948383+0000","last_peered":"2024-09-13T14:42:16.948383+0000","last_clean":"2024-09-13T14:42:16.948383+0000","last_became_active":"2024-09-13T14:41:28.593183+0000","last_became_peered":"2024-09-13T14:41:28.593183+0000","last_unstale":"2024-09-13T14:42:16.948383+0000","last_undegraded":"2024-09-13T14:42:16.948383+0000","last_fullsized":"2024-09-13T14:42:16.948383+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:54.526707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1084,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.920609+0000","last_change":"2024-09-13T14:28:35.603663+0000","last_active":"2024-09-13T14:42:14.920609+0000","last_peered":"2024-09-13T14:42:14.920609+0000","last_clean":"2024-09-13T14:42:14.920609+0000","last_became_active":"2024-09-13T14:28:35.603358+0000","last_became_peered":"2024-09-13T14:28:35.603358+0000","last_unstale":"2024-09-13T14:42:14.920609+0000","last_undegraded":"2024-09-13T14:42:14.920609+0000","last_fullsized":"2024-09-13T14:42:14.920609+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:24:05.842796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1592,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:16.942810+0000","last_change":"2024-09-13T14:28:35.611508+0000","last_active":"2024-09-13T14:42:16.942810+0000","last_peered":"2024-09-13T14:42:16.942810+0000","last_clean":"2024-09-13T14:42:16.942810+0000","last_became_active":"2024-09-13T14:28:35.610106+0000","last_became_peered":"2024-09-13T14:28:35.610106+0000","last_unstale":"2024-09-13T14:42:16.942810+0000","last_undegraded":"2024-09-13T14:42:16.942810+0000","last_fullsized":"2024-09-13T14:42:16.942810+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:11.966958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"261'100","reported_seq":48,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.939743+0000","last_change":"2024-09-13T14:42:14.932815+0000","last_active":"2024-09-13T14:42:14.939743+0000","last_peered":"2024-09-13T14:42:14.939743+0000","last_clean":"2024-09-13T14:42:14.939743+0000","last_became_active":"2024-09-13T14:42:14.932516+0000","last_became_peered":"2024-09-13T14:42:14.932516+0000","last_unstale":"2024-09-13T14:42:14.939743+0000","last_undegraded":"2024-09-13T14:42:14.939743+0000","last_fullsized":"2024-09-13T14:42:14.939743+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:22.892963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:22.892963+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017451785000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"1118'12077","reported_seq":12130,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:27.291381+0000","last_change":"2024-09-13T14:41:46.841786+0000","last_active":"2024-09-13T14:42:27.291381+0000","last_peered":"2024-09-13T14:42:27.291381+0000","last_clean":"2024-09-13T14:42:27.291381+0000","last_became_active":"2024-09-13T14:41:46.809092+0000","last_became_peered":"2024-09-13T14:41:46.809092+0000","last_unstale":"2024-09-13T14:42:27.291381+0000","last_undegraded":"2024-09-13T14:42:27.291381+0000","last_fullsized":"2024-09-13T14:42:27.291381+0000","mapping_epoch":1082,"log_start":"1088'4300","ondisk_log_start":"1088'4300","created":1082,"last_epoch_clean":1109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":7777,"log_dups_size":0,"ondisk_log_size":7777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108658688,"num_objects":1658,"num_object_clones":0,"num_object_copies":4974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6653,"num_write_kb":265952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":1552,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:16.959533+0000","last_change":"2024-09-13T14:30:15.581728+0000","last_active":"2024-09-13T14:42:16.959533+0000","last_peered":"2024-09-13T14:42:16.959533+0000","last_clean":"2024-09-13T14:42:16.959533+0000","last_became_active":"2024-09-13T14:30:15.581303+0000","last_became_peered":"2024-09-13T14:30:15.581303+0000","last_unstale":"2024-09-13T14:42:16.959533+0000","last_undegraded":"2024-09-13T14:42:16.959533+0000","last_fullsized":"2024-09-13T14:42:16.959533+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:10.027991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":491,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.933085+0000","last_change":"2024-09-13T14:41:28.592508+0000","last_active":"2024-09-13T14:42:14.933085+0000","last_peered":"2024-09-13T14:42:14.933085+0000","last_clean":"2024-09-13T14:42:14.933085+0000","last_became_active":"2024-09-13T14:41:28.590027+0000","last_became_peered":"2024-09-13T14:41:28.590027+0000","last_unstale":"2024-09-13T14:42:14.933085+0000","last_undegraded":"2024-09-13T14:42:14.933085+0000","last_fullsized":"2024-09-13T14:42:14.933085+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:30:39.262649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":625,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.918526+0000","last_change":"2024-09-13T14:41:28.587360+0000","last_active":"2024-09-13T14:42:14.918526+0000","last_peered":"2024-09-13T14:42:14.918526+0000","last_clean":"2024-09-13T14:42:14.918526+0000","last_became_active":"2024-09-13T14:41:28.586655+0000","last_became_peered":"2024-09-13T14:41:28.586655+0000","last_unstale":"2024-09-13T14:42:14.918526+0000","last_undegraded":"2024-09-13T14:42:14.918526+0000","last_fullsized":"2024-09-13T14:42:14.918526+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:21:51.249946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"261'100","reported_seq":52,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.948653+0000","last_change":"2024-09-13T14:42:14.948141+0000","last_active":"2024-09-13T14:42:14.948653+0000","last_peered":"2024-09-13T14:42:14.948653+0000","last_clean":"2024-09-13T14:42:14.948653+0000","last_became_active":"2024-09-13T14:42:14.947797+0000","last_became_peered":"2024-09-13T14:42:14.947797+0000","last_unstale":"2024-09-13T14:42:14.948653+0000","last_undegraded":"2024-09-13T14:42:14.948653+0000","last_fullsized":"2024-09-13T14:42:14.948653+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:22.580245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:22.580245+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001821837,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109226,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"1118'11788","reported_seq":11842,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:27.381774+0000","last_change":"2024-09-13T14:41:46.856515+0000","last_active":"2024-09-13T14:42:27.381774+0000","last_peered":"2024-09-13T14:42:27.381774+0000","last_clean":"2024-09-13T14:42:27.381774+0000","last_became_active":"2024-09-13T14:41:46.805366+0000","last_became_peered":"2024-09-13T14:41:46.805366+0000","last_unstale":"2024-09-13T14:42:27.381774+0000","last_undegraded":"2024-09-13T14:42:27.381774+0000","last_fullsized":"2024-09-13T14:42:27.381774+0000","mapping_epoch":1082,"log_start":"1089'4531","ondisk_log_start":"1089'4531","created":1082,"last_epoch_clean":1109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":7257,"log_dups_size":0,"ondisk_log_size":7257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101122048,"num_objects":1543,"num_object_clones":0,"num_object_copies":4629,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6560,"num_write_kb":259296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":1628,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:16.942955+0000","last_change":"2024-09-13T14:41:28.598014+0000","last_active":"2024-09-13T14:42:16.942955+0000","last_peered":"2024-09-13T14:42:16.942955+0000","last_clean":"2024-09-13T14:42:16.942955+0000","last_became_active":"2024-09-13T14:41:28.585268+0000","last_became_peered":"2024-09-13T14:41:28.585268+0000","last_unstale":"2024-09-13T14:42:16.942955+0000","last_undegraded":"2024-09-13T14:42:16.942955+0000","last_fullsized":"2024-09-13T14:42:16.942955+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:40.047685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1068,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.935091+0000","last_change":"2024-09-13T14:29:42.538860+0000","last_active":"2024-09-13T14:42:14.935091+0000","last_peered":"2024-09-13T14:42:14.935091+0000","last_clean":"2024-09-13T14:42:14.935091+0000","last_became_active":"2024-09-13T14:29:42.538608+0000","last_became_peered":"2024-09-13T14:29:42.538608+0000","last_unstale":"2024-09-13T14:42:14.935091+0000","last_undegraded":"2024-09-13T14:42:14.935091+0000","last_fullsized":"2024-09-13T14:42:14.935091+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:20.070703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1041,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.925124+0000","last_change":"2024-09-13T14:19:26.131609+0000","last_active":"2024-09-13T14:42:14.925124+0000","last_peered":"2024-09-13T14:42:14.925124+0000","last_clean":"2024-09-13T14:42:14.925124+0000","last_became_active":"2024-09-13T14:19:26.131029+0000","last_became_peered":"2024-09-13T14:19:26.131029+0000","last_unstale":"2024-09-13T14:42:14.925124+0000","last_undegraded":"2024-09-13T14:42:14.925124+0000","last_fullsized":"2024-09-13T14:42:14.925124+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:57.933696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"1118'11800","reported_seq":12041,"reported_epoch":1118,"state":"active+recovery_wait+remapped","last_fresh":"2024-09-13T14:42:27.431806+0000","last_change":"2024-09-13T14:41:48.859115+0000","last_active":"2024-09-13T14:42:27.431806+0000","last_peered":"2024-09-13T14:42:27.431806+0000","last_clean":"2024-09-13T14:41:27.528697+0000","last_became_active":"2024-09-13T14:41:48.834530+0000","last_became_peered":"2024-09-13T14:41:48.834530+0000","last_unstale":"2024-09-13T14:42:27.431806+0000","last_undegraded":"2024-09-13T14:42:27.431806+0000","last_fullsized":"2024-09-13T14:42:27.431806+0000","mapping_epoch":1110,"log_start":"1090'5198","ondisk_log_start":"1090'5198","created":1108,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":6602,"log_dups_size":0,"ondisk_log_size":6602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107937792,"num_objects":1647,"num_object_clones":0,"num_object_copies":4941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1647,"num_objects_unfound":0,"num_objects_dirty":1647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6510,"num_write_kb":261024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":2424832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,1],"avail_no_missing":["6","1","2"],"object_location_counts":[{"shards":"1,2,6","objects":1647}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1068,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.925404+0000","last_change":"2024-09-13T14:41:28.593455+0000","last_active":"2024-09-13T14:42:14.925404+0000","last_peered":"2024-09-13T14:42:14.925404+0000","last_clean":"2024-09-13T14:42:14.925404+0000","last_became_active":"2024-09-13T14:41:28.593139+0000","last_became_peered":"2024-09-13T14:41:28.593139+0000","last_unstale":"2024-09-13T14:42:14.925404+0000","last_undegraded":"2024-09-13T14:42:14.925404+0000","last_fullsized":"2024-09-13T14:42:14.925404+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:37.427352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":1064,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.926196+0000","last_change":"2024-09-13T14:19:25.870605+0000","last_active":"2024-09-13T14:42:14.926196+0000","last_peered":"2024-09-13T14:42:14.926196+0000","last_clean":"2024-09-13T14:42:14.926196+0000","last_became_active":"2024-09-13T14:19:25.870282+0000","last_became_peered":"2024-09-13T14:19:25.870282+0000","last_unstale":"2024-09-13T14:42:14.926196+0000","last_undegraded":"2024-09-13T14:42:14.926196+0000","last_fullsized":"2024-09-13T14:42:14.926196+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:52.514915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"1118'12163","reported_seq":12298,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:28.695542+0000","last_change":"2024-09-13T14:42:05.296915+0000","last_active":"2024-09-13T14:42:28.695542+0000","last_peered":"2024-09-13T14:42:28.695542+0000","last_clean":"2024-09-13T14:42:28.695542+0000","last_became_active":"2024-09-13T14:41:46.825752+0000","last_became_peered":"2024-09-13T14:41:46.825752+0000","last_unstale":"2024-09-13T14:42:28.695542+0000","last_undegraded":"2024-09-13T14:42:28.695542+0000","last_fullsized":"2024-09-13T14:42:28.695542+0000","mapping_epoch":1108,"log_start":"1088'4216","ondisk_log_start":"1088'4216","created":1108,"last_epoch_clean":1109,"parent":"0.0","parent_split_bits":5,"last_scrub":"1114'11474","last_scrub_stamp":"2024-09-13T14:42:05.296689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:42:05.296689+0000","objects_scrubbed":2229,"log_size":7947,"log_dups_size":0,"ondisk_log_size":7947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:03.947606+0000","scrub_duration":387,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100532224,"num_objects":1534,"num_object_clones":0,"num_object_copies":4602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":695,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":559,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.934106+0000","last_change":"2024-09-13T14:41:28.593526+0000","last_active":"2024-09-13T14:42:14.934106+0000","last_peered":"2024-09-13T14:42:14.934106+0000","last_clean":"2024-09-13T14:42:14.934106+0000","last_became_active":"2024-09-13T14:41:28.593217+0000","last_became_peered":"2024-09-13T14:41:28.593217+0000","last_unstale":"2024-09-13T14:42:14.934106+0000","last_undegraded":"2024-09-13T14:42:14.934106+0000","last_fullsized":"2024-09-13T14:42:14.934106+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:57:01.502925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":1075,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.920366+0000","last_change":"2024-09-13T14:30:15.579260+0000","last_active":"2024-09-13T14:42:14.920366+0000","last_peered":"2024-09-13T14:42:14.920366+0000","last_clean":"2024-09-13T14:42:14.920366+0000","last_became_active":"2024-09-13T14:30:15.578897+0000","last_became_peered":"2024-09-13T14:30:15.578897+0000","last_unstale":"2024-09-13T14:42:14.920366+0000","last_undegraded":"2024-09-13T14:42:14.920366+0000","last_fullsized":"2024-09-13T14:42:14.920366+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:56.737849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"261'100","reported_seq":51,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.960734+0000","last_change":"2024-09-13T14:42:14.954582+0000","last_active":"2024-09-13T14:42:14.960734+0000","last_peered":"2024-09-13T14:42:14.960734+0000","last_clean":"2024-09-13T14:42:14.960734+0000","last_became_active":"2024-09-13T14:42:14.954301+0000","last_became_peered":"2024-09-13T14:42:14.954301+0000","last_unstale":"2024-09-13T14:42:14.960734+0000","last_undegraded":"2024-09-13T14:42:14.960734+0000","last_fullsized":"2024-09-13T14:42:14.960734+0000","mapping_epoch":1116,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:22.580245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:22.580245+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001821837,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.13","version":"1118'11738","reported_seq":11791,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:27.352477+0000","last_change":"2024-09-13T14:41:46.905783+0000","last_active":"2024-09-13T14:42:27.352477+0000","last_peered":"2024-09-13T14:42:27.352477+0000","last_clean":"2024-09-13T14:42:27.352477+0000","last_became_active":"2024-09-13T14:41:46.887125+0000","last_became_peered":"2024-09-13T14:41:46.887125+0000","last_unstale":"2024-09-13T14:42:27.352477+0000","last_undegraded":"2024-09-13T14:42:27.352477+0000","last_fullsized":"2024-09-13T14:42:27.352477+0000","mapping_epoch":1108,"log_start":"1089'4531","ondisk_log_start":"1089'4531","created":1108,"last_epoch_clean":1109,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":7207,"log_dups_size":0,"ondisk_log_size":7207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104398848,"num_objects":1593,"num_object_clones":0,"num_object_copies":4779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6510,"num_write_kb":259296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1071,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.930126+0000","last_change":"2024-09-13T14:41:28.591640+0000","last_active":"2024-09-13T14:42:14.930126+0000","last_peered":"2024-09-13T14:42:14.930126+0000","last_clean":"2024-09-13T14:42:14.930126+0000","last_became_active":"2024-09-13T14:41:28.591354+0000","last_became_peered":"2024-09-13T14:41:28.591354+0000","last_unstale":"2024-09-13T14:42:14.930126+0000","last_undegraded":"2024-09-13T14:42:14.930126+0000","last_fullsized":"2024-09-13T14:42:14.930126+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:53:38.555537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1520,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:16.958188+0000","last_change":"2024-09-13T14:30:15.576921+0000","last_active":"2024-09-13T14:42:16.958188+0000","last_peered":"2024-09-13T14:42:16.958188+0000","last_clean":"2024-09-13T14:42:16.958188+0000","last_became_active":"2024-09-13T14:30:15.576607+0000","last_became_peered":"2024-09-13T14:30:15.576607+0000","last_unstale":"2024-09-13T14:42:16.958188+0000","last_undegraded":"2024-09-13T14:42:16.958188+0000","last_fullsized":"2024-09-13T14:42:16.958188+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:35.162460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"1118'12107","reported_seq":12159,"reported_epoch":1118,"state":"active+recovery_wait","last_fresh":"2024-09-13T14:42:27.311360+0000","last_change":"2024-09-13T14:41:46.917041+0000","last_active":"2024-09-13T14:42:27.311360+0000","last_peered":"2024-09-13T14:42:27.311360+0000","last_clean":"2024-09-13T14:41:45.747357+0000","last_became_active":"2024-09-13T14:41:46.879995+0000","last_became_peered":"2024-09-13T14:41:46.879995+0000","last_unstale":"2024-09-13T14:42:27.311360+0000","last_undegraded":"2024-09-13T14:42:27.311360+0000","last_fullsized":"2024-09-13T14:42:27.311360+0000","mapping_epoch":1108,"log_start":"1088'4300","ondisk_log_start":"1088'4300","created":1108,"last_epoch_clean":1083,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":7807,"log_dups_size":0,"ondisk_log_size":7807,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106692608,"num_objects":1628,"num_object_clones":0,"num_object_copies":4884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6683,"num_write_kb":265952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":["4","2","6"],"object_location_counts":[{"shards":"2,4,6","objects":1628}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1642,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:16.943200+0000","last_change":"2024-09-13T14:39:11.386005+0000","last_active":"2024-09-13T14:42:16.943200+0000","last_peered":"2024-09-13T14:42:16.943200+0000","last_clean":"2024-09-13T14:42:16.943200+0000","last_became_active":"2024-09-13T14:39:11.385691+0000","last_became_peered":"2024-09-13T14:39:11.385691+0000","last_unstale":"2024-09-13T14:42:16.943200+0000","last_undegraded":"2024-09-13T14:42:16.943200+0000","last_fullsized":"2024-09-13T14:42:16.943200+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:11:03.371484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":605,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.922748+0000","last_change":"2024-09-13T14:41:28.592154+0000","last_active":"2024-09-13T14:42:14.922748+0000","last_peered":"2024-09-13T14:42:14.922748+0000","last_clean":"2024-09-13T14:42:14.922748+0000","last_became_active":"2024-09-13T14:41:28.590786+0000","last_became_peered":"2024-09-13T14:41:28.590786+0000","last_unstale":"2024-09-13T14:42:14.922748+0000","last_undegraded":"2024-09-13T14:42:14.922748+0000","last_fullsized":"2024-09-13T14:42:14.922748+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:49:59.861513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1064,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.935018+0000","last_change":"2024-09-13T14:41:28.594659+0000","last_active":"2024-09-13T14:42:14.935018+0000","last_peered":"2024-09-13T14:42:14.935018+0000","last_clean":"2024-09-13T14:42:14.935018+0000","last_became_active":"2024-09-13T14:41:28.588903+0000","last_became_peered":"2024-09-13T14:41:28.588903+0000","last_unstale":"2024-09-13T14:42:14.935018+0000","last_undegraded":"2024-09-13T14:42:14.935018+0000","last_fullsized":"2024-09-13T14:42:14.935018+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:45:51.783371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":671,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:16.942151+0000","last_change":"2024-09-13T14:41:28.585670+0000","last_active":"2024-09-13T14:42:16.942151+0000","last_peered":"2024-09-13T14:42:16.942151+0000","last_clean":"2024-09-13T14:42:16.942151+0000","last_became_active":"2024-09-13T14:41:28.585173+0000","last_became_peered":"2024-09-13T14:41:28.585173+0000","last_unstale":"2024-09-13T14:42:16.942151+0000","last_undegraded":"2024-09-13T14:42:16.942151+0000","last_fullsized":"2024-09-13T14:42:16.942151+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:26:08.019099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1617,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:16.942774+0000","last_change":"2024-09-13T14:41:28.583750+0000","last_active":"2024-09-13T14:42:16.942774+0000","last_peered":"2024-09-13T14:42:16.942774+0000","last_clean":"2024-09-13T14:42:16.942774+0000","last_became_active":"2024-09-13T14:41:28.583293+0000","last_became_peered":"2024-09-13T14:41:28.583293+0000","last_unstale":"2024-09-13T14:42:16.942774+0000","last_undegraded":"2024-09-13T14:42:16.942774+0000","last_fullsized":"2024-09-13T14:42:16.942774+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:41:05.647080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":638,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:16.958281+0000","last_change":"2024-09-13T14:41:28.590716+0000","last_active":"2024-09-13T14:42:16.958281+0000","last_peered":"2024-09-13T14:42:16.958281+0000","last_clean":"2024-09-13T14:42:16.958281+0000","last_became_active":"2024-09-13T14:41:28.590372+0000","last_became_peered":"2024-09-13T14:41:28.590372+0000","last_unstale":"2024-09-13T14:42:16.958281+0000","last_undegraded":"2024-09-13T14:42:16.958281+0000","last_fullsized":"2024-09-13T14:42:16.958281+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:39.348905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":586,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.925756+0000","last_change":"2024-09-13T14:41:28.593532+0000","last_active":"2024-09-13T14:42:14.925756+0000","last_peered":"2024-09-13T14:42:14.925756+0000","last_clean":"2024-09-13T14:42:14.925756+0000","last_became_active":"2024-09-13T14:41:28.593235+0000","last_became_peered":"2024-09-13T14:41:28.593235+0000","last_unstale":"2024-09-13T14:42:14.925756+0000","last_undegraded":"2024-09-13T14:42:14.925756+0000","last_fullsized":"2024-09-13T14:42:14.925756+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:30:22.224415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":1087,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.926096+0000","last_change":"2024-09-13T14:41:28.590173+0000","last_active":"2024-09-13T14:42:14.926096+0000","last_peered":"2024-09-13T14:42:14.926096+0000","last_clean":"2024-09-13T14:42:14.926096+0000","last_became_active":"2024-09-13T14:41:28.589347+0000","last_became_peered":"2024-09-13T14:41:28.589347+0000","last_unstale":"2024-09-13T14:42:14.926096+0000","last_undegraded":"2024-09-13T14:42:14.926096+0000","last_fullsized":"2024-09-13T14:42:14.926096+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:19.873187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1603,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:16.948602+0000","last_change":"2024-09-13T14:41:28.584508+0000","last_active":"2024-09-13T14:42:16.948602+0000","last_peered":"2024-09-13T14:42:16.948602+0000","last_clean":"2024-09-13T14:42:16.948602+0000","last_became_active":"2024-09-13T14:41:28.584187+0000","last_became_peered":"2024-09-13T14:41:28.584187+0000","last_unstale":"2024-09-13T14:42:16.948602+0000","last_undegraded":"2024-09-13T14:42:16.948602+0000","last_fullsized":"2024-09-13T14:42:16.948602+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:21.364817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":1594,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:16.942897+0000","last_change":"2024-09-13T14:28:35.612469+0000","last_active":"2024-09-13T14:42:16.942897+0000","last_peered":"2024-09-13T14:42:16.942897+0000","last_clean":"2024-09-13T14:42:16.942897+0000","last_became_active":"2024-09-13T14:28:35.612093+0000","last_became_peered":"2024-09-13T14:28:35.612093+0000","last_unstale":"2024-09-13T14:42:16.942897+0000","last_undegraded":"2024-09-13T14:42:16.942897+0000","last_fullsized":"2024-09-13T14:42:16.942897+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:51.022663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"1118'11998","reported_seq":12130,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:28.610517+0000","last_change":"2024-09-13T14:42:06.208302+0000","last_active":"2024-09-13T14:42:28.610517+0000","last_peered":"2024-09-13T14:42:28.610517+0000","last_clean":"2024-09-13T14:42:28.610517+0000","last_became_active":"2024-09-13T14:41:46.908994+0000","last_became_peered":"2024-09-13T14:41:46.908994+0000","last_unstale":"2024-09-13T14:42:28.610517+0000","last_undegraded":"2024-09-13T14:42:28.610517+0000","last_fullsized":"2024-09-13T14:42:28.610517+0000","mapping_epoch":1108,"log_start":"1088'3982","ondisk_log_start":"1088'3982","created":1108,"last_epoch_clean":1109,"parent":"0.0","parent_split_bits":5,"last_scrub":"1114'11283","last_scrub_stamp":"2024-09-13T14:42:06.208222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:42:06.208222+0000","objects_scrubbed":2309,"log_size":8016,"log_dups_size":0,"ondisk_log_size":8016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:42:05.492222+0000","scrub_duration":337,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104071168,"num_objects":1588,"num_object_clones":0,"num_object_copies":4764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":721,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1078,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.925273+0000","last_change":"2024-09-13T14:41:28.588664+0000","last_active":"2024-09-13T14:42:14.925273+0000","last_peered":"2024-09-13T14:42:14.925273+0000","last_clean":"2024-09-13T14:42:14.925273+0000","last_became_active":"2024-09-13T14:41:28.588283+0000","last_became_peered":"2024-09-13T14:41:28.588283+0000","last_unstale":"2024-09-13T14:42:14.925273+0000","last_undegraded":"2024-09-13T14:42:14.925273+0000","last_fullsized":"2024-09-13T14:42:14.925273+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:10:02.721554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":1077,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.930791+0000","last_change":"2024-09-13T14:41:28.603145+0000","last_active":"2024-09-13T14:42:14.930791+0000","last_peered":"2024-09-13T14:42:14.930791+0000","last_clean":"2024-09-13T14:42:14.930791+0000","last_became_active":"2024-09-13T14:41:28.590945+0000","last_became_peered":"2024-09-13T14:41:28.590945+0000","last_unstale":"2024-09-13T14:42:14.930791+0000","last_undegraded":"2024-09-13T14:42:14.930791+0000","last_fullsized":"2024-09-13T14:42:14.930791+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:01:21.700348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"1118'12251","reported_seq":12305,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:27.369843+0000","last_change":"2024-09-13T14:41:46.900621+0000","last_active":"2024-09-13T14:42:27.369843+0000","last_peered":"2024-09-13T14:42:27.369843+0000","last_clean":"2024-09-13T14:42:27.369843+0000","last_became_active":"2024-09-13T14:41:46.868571+0000","last_became_peered":"2024-09-13T14:41:46.868571+0000","last_unstale":"2024-09-13T14:42:27.369843+0000","last_undegraded":"2024-09-13T14:42:27.369843+0000","last_fullsized":"2024-09-13T14:42:27.369843+0000","mapping_epoch":1108,"log_start":"1090'4941","ondisk_log_start":"1090'4941","created":1108,"last_epoch_clean":1109,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:39:15.410230+0000","last_clean_scrub_stamp":"2024-09-13T14:39:15.410230+0000","objects_scrubbed":0,"log_size":7310,"log_dups_size":0,"ondisk_log_size":7310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107741184,"num_objects":1644,"num_object_clones":0,"num_object_copies":4932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6791,"num_write_kb":269920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1075,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.925988+0000","last_change":"2024-09-13T14:30:15.580011+0000","last_active":"2024-09-13T14:42:14.925988+0000","last_peered":"2024-09-13T14:42:14.925988+0000","last_clean":"2024-09-13T14:42:14.925988+0000","last_became_active":"2024-09-13T14:30:15.579610+0000","last_became_peered":"2024-09-13T14:30:15.579610+0000","last_unstale":"2024-09-13T14:42:14.925988+0000","last_undegraded":"2024-09-13T14:42:14.925988+0000","last_fullsized":"2024-09-13T14:42:14.925988+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:11:02.733771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":1046,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.930276+0000","last_change":"2024-09-13T14:19:25.873514+0000","last_active":"2024-09-13T14:42:14.930276+0000","last_peered":"2024-09-13T14:42:14.930276+0000","last_clean":"2024-09-13T14:42:14.930276+0000","last_became_active":"2024-09-13T14:19:25.873170+0000","last_became_peered":"2024-09-13T14:19:25.873170+0000","last_unstale":"2024-09-13T14:42:14.930276+0000","last_undegraded":"2024-09-13T14:42:14.930276+0000","last_fullsized":"2024-09-13T14:42:14.930276+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:48.298530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":671,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:16.951442+0000","last_change":"2024-09-13T14:39:11.396732+0000","last_active":"2024-09-13T14:42:16.951442+0000","last_peered":"2024-09-13T14:42:16.951442+0000","last_clean":"2024-09-13T14:42:16.951442+0000","last_became_active":"2024-09-13T14:39:11.396322+0000","last_became_peered":"2024-09-13T14:39:11.396322+0000","last_unstale":"2024-09-13T14:42:16.951442+0000","last_undegraded":"2024-09-13T14:42:16.951442+0000","last_fullsized":"2024-09-13T14:42:16.951442+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:54:47.496762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":80,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.932958+0000","last_change":"2024-09-13T14:41:28.586563+0000","last_active":"2024-09-13T14:42:14.932958+0000","last_peered":"2024-09-13T14:42:14.932958+0000","last_clean":"2024-09-13T14:42:14.932958+0000","last_became_active":"2024-09-13T14:41:28.586152+0000","last_became_peered":"2024-09-13T14:41:28.586152+0000","last_unstale":"2024-09-13T14:42:14.932958+0000","last_undegraded":"2024-09-13T14:42:14.932958+0000","last_fullsized":"2024-09-13T14:42:14.932958+0000","mapping_epoch":1102,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:28.618035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1068,"reported_epoch":1117,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.924920+0000","last_change":"2024-09-13T14:29:42.539907+0000","last_active":"2024-09-13T14:42:14.924920+0000","last_peered":"2024-09-13T14:42:14.924920+0000","last_clean":"2024-09-13T14:42:14.924920+0000","last_became_active":"2024-09-13T14:29:42.539583+0000","last_became_peered":"2024-09-13T14:29:42.539583+0000","last_unstale":"2024-09-13T14:42:14.924920+0000","last_undegraded":"2024-09-13T14:42:14.924920+0000","last_fullsized":"2024-09-13T14:42:14.924920+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:02:04.628447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":1609,"reported_epoch":1118,"state":"active+clean","last_fresh":"2024-09-13T14:42:16.943399+0000","last_change":"2024-09-13T14:29:42.541263+0000","last_active":"2024-09-13T14:42:16.943399+0000","last_peered":"2024-09-13T14:42:16.943399+0000","last_clean":"2024-09-13T14:42:16.943399+0000","last_became_active":"2024-09-13T14:29:42.540873+0000","last_became_peered":"2024-09-13T14:29:42.540873+0000","last_unstale":"2024-09-13T14:42:16.943399+0000","last_undegraded":"2024-09-13T14:42:16.943399+0000","last_fullsized":"2024-09-13T14:42:16.943399+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:20.611206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":26,"stat_sum":{"num_bytes":3387424792,"num_objects":51689,"num_object_clones":0,"num_object_copies":155067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2588,"num_objects_misplaced":12301,"num_objects_unfound":0,"num_objects_dirty":51689,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":195279,"num_write_kb":7679969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6416,"num_bytes_recovered":420282368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10653085696,"data_stored":10653073480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":189299,"ondisk_log_size":189299,"up":78,"acting":77,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":488690,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":79,"num_read_kb":204,"num_write":40,"num_write_kb":444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":423698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430509,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1801996,"kb_used_data":1496324,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91943156,"statfs":{"total":95995035648,"available":94149791744,"internally_reserved":0,"allocated":1532235776,"data_stored":1528920041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626738,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2381640,"kb_used_data":2079872,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91363512,"statfs":{"total":95995035648,"available":93556236288,"internally_reserved":0,"allocated":2129788928,"data_stored":2126477714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1,1,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822967,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1638916,"kb_used_data":1341436,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":92106236,"statfs":{"total":95995035648,"available":94316785664,"internally_reserved":0,"allocated":1373630464,"data_stored":1370319188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019195,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1472776,"kb_used_data":1171264,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":92272376,"statfs":{"total":95995035648,"available":94486913024,"internally_reserved":0,"allocated":1199374336,"data_stored":1196062627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,2,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059592853,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":687924,"kb_used_data":619948,"kb_used_omap":26,"kb_used_meta":67941,"kb_avail":93057228,"statfs":{"total":95995035648,"available":95290601472,"internally_reserved":0,"allocated":634826752,"data_stored":631530743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":69571744},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":591,"seq":2538325672091,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1946364,"kb_used_data":1765684,"kb_used_omap":41,"kb_used_meta":180566,"kb_avail":91798788,"statfs":{"total":95995035648,"available":94001958912,"internally_reserved":0,"allocated":1808060416,"data_stored":1804757281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42651,"internal_metadata":184899941},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":2392296784034,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":524,"seq":2250562863273,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":926528,"kb_used_data":822328,"kb_used_omap":41,"kb_used_meta":104150,"kb_avail":92818624,"statfs":{"total":95995035648,"available":95046270976,"internally_reserved":0,"allocated":842063872,"data_stored":838756144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42694,"internal_metadata":106649914},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":831328256,"data_stored":831324184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1297616896,"data_stored":1297612824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1768357888,"data_stored":1768357888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":622526464,"data_stored":622526464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1175257088,"data_stored":1175257088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1352470528,"data_stored":1352466456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2112815104,"data_stored":2112815104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1492713472,"data_stored":1492713472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:42:33.849 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-13T14:42:33.849 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 31 2024-09-13T14:42:33.850 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:33.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:33.892 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:33.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:33.993 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:34.020 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:34.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:34.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:34.122 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:34.165 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:34.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:34.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:34.267 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:34.297 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:34.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:34.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:34.398 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:34.433 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:34.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:34.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:34.535 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:34.557 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:34.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:34.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:34.659 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:34.681 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:34 smithi151 ceph-mon[104450]: pgmap v1419: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 2.9 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 988 op/s; 2588/143877 objects degraded (1.799%); 11428/143877 objects misplaced (7.943%); 659 KiB/s, 10 objects/s recovering 2024-09-13T14:42:34.681 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:34 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:42:34.681 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:34 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:42:34.681 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:34 smithi151 ceph-mon[108966]: pgmap v1419: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 2.9 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 988 op/s; 2588/143877 objects degraded (1.799%); 11428/143877 objects misplaced (7.943%); 659 KiB/s, 10 objects/s recovering 2024-09-13T14:42:34.681 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:34 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:42:34.681 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:34 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:42:34.682 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:42:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:42:34] "GET /metrics HTTP/1.1" 200 36683 "" "Prometheus/2.51.0" 2024-09-13T14:42:34.684 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:34.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:34.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:34.786 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:34.817 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:34.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:34.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:34.919 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:34.953 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:34.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:34.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:35.056 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:35.086 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:35.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:35.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:35.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:34 smithi184 ceph-mon[92527]: pgmap v1419: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 2.9 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 988 op/s; 2588/143877 objects degraded (1.799%); 11428/143877 objects misplaced (7.943%); 659 KiB/s, 10 objects/s recovering 2024-09-13T14:42:35.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:34 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:42:35.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:34 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:42:35.188 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:35.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:35.229 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:35.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:35.270 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:42:35.318 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T14:42:35.330 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:35.428 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:35.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:35.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:35.530 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:35.564 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:35.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:35.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:35.666 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:35.699 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:35.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:35.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:35.801 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:35.844 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:35.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:35.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:35.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:35 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 2588/143877 objects degraded (1.799%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:42:35.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:35 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 2588/143877 objects degraded (1.799%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:42:35.946 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:35.981 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:35.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:35.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:36.083 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:36.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:35 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 2588/143877 objects degraded (1.799%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:42:36.119 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:36.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:36.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:36.223 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:36.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:36.254 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:36.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:36.355 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:36.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:36.386 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:36.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:36.487 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:36.513 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:36.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:36.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:36.615 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:36.640 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:36.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:36.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:36.741 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:36.774 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:36.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:36.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:36.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:36.901 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:36.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:36.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:37.003 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:37.041 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:37.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:37.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:37.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:36 smithi184 ceph-mon[92527]: pgmap v1420: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 2.9 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 989 op/s; 2832/141144 objects degraded (2.006%); 11125/141144 objects misplaced (7.882%); 697 KiB/s, 10 objects/s recovering 2024-09-13T14:42:37.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:36 smithi151 ceph-mon[104450]: pgmap v1420: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 2.9 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 989 op/s; 2832/141144 objects degraded (2.006%); 11125/141144 objects misplaced (7.882%); 697 KiB/s, 10 objects/s recovering 2024-09-13T14:42:37.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:36 smithi151 ceph-mon[108966]: pgmap v1420: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 2.9 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 989 op/s; 2832/141144 objects degraded (2.006%); 11125/141144 objects misplaced (7.882%); 697 KiB/s, 10 objects/s recovering 2024-09-13T14:42:37.143 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:37.175 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:37.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:37.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:37.277 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:37.306 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:37.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:37.308 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:37.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:37.409 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:37.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:37.442 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:37.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:37.543 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:37.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:37.577 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:37.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:37.678 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:37.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:37.719 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:37.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:37.821 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:38.225 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:38.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:38.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:38.327 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:38.363 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:38.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:38.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:38.465 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:38.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:38.493 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:38.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:38.594 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:38.623 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:38.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:38.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:38.725 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:38.766 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:38.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:38.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:38.822 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:38 smithi151 ceph-mon[104450]: pgmap v1421: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 2.8 GiB data, 9.3 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 977 op/s; 2832/137571 objects degraded (2.059%); 10908/137571 objects misplaced (7.929%); 697 KiB/s, 10 objects/s recovering 2024-09-13T14:42:38.822 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:38 smithi151 ceph-mon[108966]: pgmap v1421: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 2.8 GiB data, 9.3 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 977 op/s; 2832/137571 objects degraded (2.059%); 10908/137571 objects misplaced (7.929%); 697 KiB/s, 10 objects/s recovering 2024-09-13T14:42:38.868 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:38.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:38.906 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:38.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:39.007 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:39.041 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:39.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:39.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:39.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:38 smithi184 ceph-mon[92527]: pgmap v1421: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 2.8 GiB data, 9.3 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 977 op/s; 2832/137571 objects degraded (2.059%); 10908/137571 objects misplaced (7.929%); 697 KiB/s, 10 objects/s recovering 2024-09-13T14:42:39.142 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:39.172 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:39.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:39.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:39.274 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:39.301 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:39.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:39.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:39.403 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:39.439 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:39.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:39.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:39.513 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:42:39.513 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:42:39.513 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:42:39.513 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:42:39.516 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:42:39.516 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:42:39.540 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:42:39.540 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:42:39.541 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:39.565 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:42:39.565 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:42:39.568 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:39.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:39.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:39.591 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:42:39.591 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:42:39.618 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:42:39.618 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:42:39.644 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:42:39.644 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:42:39.669 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:39.711 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:39.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:39.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:39.715 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:39 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/552040317' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:39.716 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:39 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/552040317' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:39.813 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:39.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:39.849 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:39.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:39.950 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:39.964 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:39.964 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:40.009 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:40.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:40.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:40.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:39 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/552040317' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:40.111 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:40.147 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:40.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:40.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:40.249 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:40.278 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:40.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:40.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:40.380 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:40.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:40.413 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:40.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:40.514 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:40.549 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:40.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:40.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:40.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:42:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:42:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:42:40.651 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:40.662 INFO:teuthology.orchestra.run.smithi151.stderr:set pool 1 pg_num to 31 2024-09-13T14:42:40.683 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:40.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:40.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:40.786 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:40.809 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:40 smithi151 ceph-mon[104450]: pgmap v1422: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 2.6 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 965 op/s; 3078/126168 objects degraded (2.440%); 9822/126168 objects misplaced (7.785%); 755 KiB/s, 11 objects/s recovering 2024-09-13T14:42:40.809 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:40 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 3078/126168 objects degraded (2.440%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:42:40.809 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:40 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/781701786' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-09-13T14:42:40.809 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:40 smithi151 ceph-mon[108966]: pgmap v1422: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 2.6 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 965 op/s; 3078/126168 objects degraded (2.440%); 9822/126168 objects misplaced (7.785%); 755 KiB/s, 11 objects/s recovering 2024-09-13T14:42:40.809 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:40 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 3078/126168 objects degraded (2.440%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:42:40.809 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:40 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/781701786' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-09-13T14:42:40.814 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:40.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:40.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:40.916 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:40.942 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:40.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:40.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:41.044 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:41.069 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:41.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:41.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:41.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:40 smithi184 ceph-mon[92527]: pgmap v1422: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 2.6 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 965 op/s; 3078/126168 objects degraded (2.440%); 9822/126168 objects misplaced (7.785%); 755 KiB/s, 11 objects/s recovering 2024-09-13T14:42:41.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:40 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 3078/126168 objects degraded (2.440%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:42:41.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:40 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/781701786' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-09-13T14:42:41.170 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:41.203 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:41.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:41.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:41.305 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:41.339 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:41.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:41.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:41.441 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:41.486 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:41.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:41.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:41.588 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:41.618 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:41.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:41.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:41.720 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:41.756 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:41.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:41.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:41.858 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:41.896 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:41.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:41.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:41.998 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:42.030 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:42.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:42.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:42.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:41 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/781701786' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-09-13T14:42:42.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:41 smithi184 ceph-mon[92527]: osdmap e1120: 8 total, 8 up, 7 in 2024-09-13T14:42:42.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:41 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-09-13T14:42:42.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:41 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/781701786' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-09-13T14:42:42.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:41 smithi151 ceph-mon[104450]: osdmap e1120: 8 total, 8 up, 7 in 2024-09-13T14:42:42.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:41 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-09-13T14:42:42.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:41 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/781701786' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-09-13T14:42:42.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:41 smithi151 ceph-mon[108966]: osdmap e1120: 8 total, 8 up, 7 in 2024-09-13T14:42:42.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:41 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-09-13T14:42:42.132 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:42.161 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:42.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:42.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:42.263 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:42.293 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:42.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:42.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:42.395 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:42.423 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:42.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:42.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:42.525 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:42.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:42.564 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:42.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:42.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:42:42.665 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:42.722 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:42.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:42.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:42.824 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:42.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:42 smithi151 ceph-mon[104450]: pgmap v1424: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 2.5 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 982 op/s; 3078/125232 objects degraded (2.458%); 9822/125232 objects misplaced (7.843%); 842 KiB/s, 13 objects/s recovering 2024-09-13T14:42:42.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:42 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-13T14:42:42.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:42 smithi151 ceph-mon[104450]: osdmap e1121: 8 total, 8 up, 7 in 2024-09-13T14:42:42.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:42 smithi151 ceph-mon[108966]: pgmap v1424: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 2.5 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 982 op/s; 3078/125232 objects degraded (2.458%); 9822/125232 objects misplaced (7.843%); 842 KiB/s, 13 objects/s recovering 2024-09-13T14:42:42.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:42 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-13T14:42:42.861 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:42 smithi151 ceph-mon[108966]: osdmap e1121: 8 total, 8 up, 7 in 2024-09-13T14:42:42.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:42.862 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:42.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:42.963 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:43.030 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:43.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:43.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:42 smithi184 ceph-mon[92527]: pgmap v1424: 119 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 111 active+clean; 2.5 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 982 op/s; 3078/125232 objects degraded (2.458%); 9822/125232 objects misplaced (7.843%); 842 KiB/s, 13 objects/s recovering 2024-09-13T14:42:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:42 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-13T14:42:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:42 smithi184 ceph-mon[92527]: osdmap e1121: 8 total, 8 up, 7 in 2024-09-13T14:42:43.132 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:43.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:43.174 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:43.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:43.275 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:43.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:43.306 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:43.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:43.407 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:43.434 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:43.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:43.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:43.536 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:43.563 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:43.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:43.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:43.665 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:43.689 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:43.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:43.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:43.792 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:43.817 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:43.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:43.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:43.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:43 smithi151 ceph-mon[104450]: osdmap e1122: 8 total, 8 up, 7 in 2024-09-13T14:42:43.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:43 smithi151 ceph-mon[108966]: osdmap e1122: 8 total, 8 up, 7 in 2024-09-13T14:42:43.919 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:43.952 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:43.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:43.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:44.004 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:43 smithi184 ceph-mon[92527]: osdmap e1122: 8 total, 8 up, 7 in 2024-09-13T14:42:44.054 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:44.081 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:44.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:44.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:44.185 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:44.206 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:44.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:44.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:44.308 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:44.338 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:42:44.338 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:42:44.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:44.343 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:44.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:44.444 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:44.471 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:44.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:44.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:44.573 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:44.592 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:42:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:42:44] "GET /metrics HTTP/1.1" 200 36683 "" "Prometheus/2.51.0" 2024-09-13T14:42:44.616 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:44.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:44.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:44.718 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:44.756 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:44.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:44.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:44.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:44 smithi151 ceph-mon[104450]: pgmap v1427: 129 pgs: 1 active+recovery_wait+remapped, 10 peering, 10 unknown, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 2.3 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 1.33k op/s; 3078/113619 objects degraded (2.709%); 8812/113619 objects misplaced (7.756%); 600 KiB/s, 9 objects/s recovering 2024-09-13T14:42:44.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:44 smithi151 ceph-mon[108966]: pgmap v1427: 129 pgs: 1 active+recovery_wait+remapped, 10 peering, 10 unknown, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 2.3 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 1.33k op/s; 3078/113619 objects degraded (2.709%); 8812/113619 objects misplaced (7.756%); 600 KiB/s, 9 objects/s recovering 2024-09-13T14:42:44.859 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:44.897 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:44.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:44.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:44.999 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:45.035 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:45.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:45.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:45.071 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T14:42:45.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:44 smithi184 ceph-mon[92527]: pgmap v1427: 129 pgs: 1 active+recovery_wait+remapped, 10 peering, 10 unknown, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 2.3 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 1.33k op/s; 3078/113619 objects degraded (2.709%); 8812/113619 objects misplaced (7.756%); 600 KiB/s, 9 objects/s recovering 2024-09-13T14:42:45.138 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:45.176 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:45.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:45.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:45.278 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:45.312 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:45.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:45.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:45.414 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:45.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:45.451 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:45.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:45.499 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:45.553 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:45.584 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:45.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:45.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:45.687 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:45.722 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:45.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:45.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:45.825 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:45.862 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:45.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:45.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:45.962 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:45 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 3078/113619 objects degraded (2.709%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:42:45.964 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:45.987 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:45.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:45.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:46.089 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:46.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:45 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 3078/113619 objects degraded (2.709%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:42:46.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:45 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 3078/113619 objects degraded (2.709%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:42:46.122 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:46.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:46.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:46.224 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:46.259 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:46.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:46.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:46.361 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:46.389 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:46.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:46.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:46.491 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:46.524 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:46.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:46.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:46.626 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:46.652 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:46.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:46.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:46.725 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-13T14:42:46.755 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:46.794 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:46.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:46.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:46.895 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:46.927 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:46.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:46.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:47.029 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:47.058 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:47.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:47.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:47.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:46 smithi184 ceph-mon[92527]: pgmap v1428: 129 pgs: 1 active+recovery_wait+remapped, 10 peering, 10 unknown, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 2.2 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 861 op/s; 3328/110619 objects degraded (3.009%); 8496/110619 objects misplaced (7.680%); 674 KiB/s, 10 objects/s recovering 2024-09-13T14:42:47.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:46 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1314409566' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:42:47.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:46 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:42:47.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:42:47.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:47.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:46 smithi151 ceph-mon[108966]: pgmap v1428: 129 pgs: 1 active+recovery_wait+remapped, 10 peering, 10 unknown, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 2.2 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 861 op/s; 3328/110619 objects degraded (3.009%); 8496/110619 objects misplaced (7.680%); 674 KiB/s, 10 objects/s recovering 2024-09-13T14:42:47.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:46 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1314409566' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:42:47.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:46 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:42:47.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:42:47.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:47.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:46 smithi151 ceph-mon[104450]: pgmap v1428: 129 pgs: 1 active+recovery_wait+remapped, 10 peering, 10 unknown, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 2.2 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 861 op/s; 3328/110619 objects degraded (3.009%); 8496/110619 objects misplaced (7.680%); 674 KiB/s, 10 objects/s recovering 2024-09-13T14:42:47.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:46 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1314409566' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:42:47.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:46 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:42:47.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:42:47.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:47.160 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:47.194 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:47.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:47.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:47.296 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:47.322 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:47.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:47.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:47.424 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:47.449 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:47.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:47.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:47.551 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:47.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:47.584 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:47.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:47.685 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:47.716 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:47.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:47.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:47.818 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:47.850 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:47.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:47.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:47.952 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:47.986 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:47.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:47.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:48.088 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:48.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:47 smithi184 ceph-mon[92527]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:42:48.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:47 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:42:48.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:47 smithi184 ceph-mon[92527]: osdmap e1123: 8 total, 8 up, 7 in 2024-09-13T14:42:48.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:47 smithi151 ceph-mon[104450]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:42:48.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:47 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:42:48.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:47 smithi151 ceph-mon[104450]: osdmap e1123: 8 total, 8 up, 7 in 2024-09-13T14:42:48.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:47 smithi151 ceph-mon[108966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:42:48.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:47 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:42:48.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:47 smithi151 ceph-mon[108966]: osdmap e1123: 8 total, 8 up, 7 in 2024-09-13T14:42:48.110 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:48.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:48.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:48.212 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:48.242 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:48.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:48.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:48.344 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:48.370 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:48.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:48.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:48.472 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:48.509 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:48.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:48.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:48.594 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:42:48.594 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:42:48.594 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-09-13T14:42:48.594 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd in 1 2024-09-13T14:42:48.610 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:48.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:48.652 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:48.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:48.753 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:48.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:48.786 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:48.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:48.879 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:48 smithi151 ceph-mon[104450]: pgmap v1430: 129 pgs: 1 active+recovery_wait+remapped, 10 peering, 10 unknown, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 2.2 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 858 op/s; 3328/109728 objects degraded (3.033%); 8496/109728 objects misplaced (7.743%); 674 KiB/s, 10 objects/s recovering 2024-09-13T14:42:48.879 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:48 smithi151 ceph-mon[108966]: pgmap v1430: 129 pgs: 1 active+recovery_wait+remapped, 10 peering, 10 unknown, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 2.2 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 858 op/s; 3328/109728 objects degraded (3.033%); 8496/109728 objects misplaced (7.743%); 674 KiB/s, 10 objects/s recovering 2024-09-13T14:42:48.887 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:48.918 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:48.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:48.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:49.021 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:49.055 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:49.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:49.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:49.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:48 smithi184 ceph-mon[92527]: pgmap v1430: 129 pgs: 1 active+recovery_wait+remapped, 10 peering, 10 unknown, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 101 active+clean; 2.2 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 858 op/s; 3328/109728 objects degraded (3.033%); 8496/109728 objects misplaced (7.743%); 674 KiB/s, 10 objects/s recovering 2024-09-13T14:42:49.115 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:49.156 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:49.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:49.200 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:49.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:49.301 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:49.332 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:49.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:49.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:49.434 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:49.471 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:49.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:49.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:49.573 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:49.601 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:49.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:49.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:49.703 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:49.742 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:49.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:49.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:49.760 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:49 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3901141303' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:49.761 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:49 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3901141303' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:49.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:49.869 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:49.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:49.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:49.875 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:42:49.876 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:42:49.876 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:42:49.876 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:42:49.876 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:42:49.876 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:42:49.876 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:42:49.876 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:42:49.877 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:42:49.877 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:42:49.877 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:42:49.877 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:42:49.877 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:42:49.877 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:42:49.878 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:42:49.878 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:42:49.971 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:49.995 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:49.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:49.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:50.097 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:50.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:49 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3901141303' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:50.143 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:50.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:50.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:50.245 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:50.280 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:50.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:50.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:50.382 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:50.407 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:50.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:50.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:50.509 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:50.537 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:50.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:50.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:50.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:42:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:42:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:42:50.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:50.663 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:50.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:50.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:50.765 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:50.801 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:50.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:50.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:50.903 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:50.936 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:50.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:50.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:50.978 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:50 smithi151 ceph-mon[108966]: pgmap v1431: 129 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 121 active+clean; 2.0 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.24k op/s; 3561/96558 objects degraded (3.688%); 7285/96558 objects misplaced (7.545%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:42:50.978 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:50 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 3561/96558 objects degraded (3.688%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:42:50.978 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:50 smithi151 ceph-mon[104450]: pgmap v1431: 129 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 121 active+clean; 2.0 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.24k op/s; 3561/96558 objects degraded (3.688%); 7285/96558 objects misplaced (7.545%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:42:50.978 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:50 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 3561/96558 objects degraded (3.688%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:42:51.038 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:51.072 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:51.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:51.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:51.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:50 smithi184 ceph-mon[92527]: pgmap v1431: 129 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 121 active+clean; 2.0 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.24k op/s; 3561/96558 objects degraded (3.688%); 7285/96558 objects misplaced (7.545%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:42:51.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:50 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 3561/96558 objects degraded (3.688%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:42:51.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:51.206 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:51.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:51.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:51.308 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:51.335 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:51.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:51.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:51.437 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:51.467 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:51.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:51.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:51.569 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:51.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:51.606 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:51.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:51.707 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:51.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:51.729 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:51.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:51.762 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:42:51.831 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:51.859 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:51.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:51.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:51.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:51.990 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:51.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:51.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:52.092 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:52.119 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:52.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:52.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:52.221 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:52.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:52.248 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:52.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:52.349 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:52.374 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:52.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:52.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:52.476 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:52.514 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:52.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:52.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:52.616 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:52.647 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:52.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:52.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:52.749 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:52.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:52.792 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:52.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:52.893 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:52.920 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:52.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:52.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:53.022 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:53.050 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:53.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:53.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:53.051 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:52 smithi151 ceph-mon[108966]: pgmap v1432: 129 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 121 active+clean; 1.9 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 7.2 KiB/s rd, 16 KiB/s wr, 695 op/s; 3561/95592 objects degraded (3.725%); 7285/95592 objects misplaced (7.621%); 941 KiB/s, 14 objects/s recovering 2024-09-13T14:42:53.052 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:52 smithi151 ceph-mon[104450]: pgmap v1432: 129 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 121 active+clean; 1.9 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 7.2 KiB/s rd, 16 KiB/s wr, 695 op/s; 3561/95592 objects degraded (3.725%); 7285/95592 objects misplaced (7.621%); 941 KiB/s, 14 objects/s recovering 2024-09-13T14:42:53.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:52 smithi184 ceph-mon[92527]: pgmap v1432: 129 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 121 active+clean; 1.9 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 7.2 KiB/s rd, 16 KiB/s wr, 695 op/s; 3561/95592 objects degraded (3.725%); 7285/95592 objects misplaced (7.621%); 941 KiB/s, 14 objects/s recovering 2024-09-13T14:42:53.152 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:53.187 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:53.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:53.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:53.289 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:53.327 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:53.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:53.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:53.429 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:53.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:53.471 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:53.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:53.572 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:53.597 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:53.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:53.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:53.699 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:53.731 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:53.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:53.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:53.833 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:53.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:53.870 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:53.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:53.971 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:53.998 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:53.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:53.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:54.100 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:54.132 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:54.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:54.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:54.234 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:54.273 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:54.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:54.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:54.375 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:54.400 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:54.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:54.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:54.405 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:42:54.405 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-13T14:42:54.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-13T14:42:54.405 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-13T14:42:54.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.405 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:63699 6.4 6:28895231:::benchmark_data_smithi151_155836_object63698:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1088)", 2024-09-13T14:42:54.405 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:39:55.554206+0000", 2024-09-13T14:42:54.406 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 178.10466607500001, 2024-09-13T14:42:54.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.10026875, 2024-09-13T14:42:54.406 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:54.406 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:54.406 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:54.406 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:54.406 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:54.406 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:54.406 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 63699 2024-09-13T14:42:54.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.406 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:54.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.406 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:54.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.554206+0000", 2024-09-13T14:42:54.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:54.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.554206+0000", 2024-09-13T14:42:54.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:54.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.554207+0000", 2024-09-13T14:42:54.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.792e-06 2024-09-13T14:42:54.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:54.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.554234+0000", 2024-09-13T14:42:54.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6231999999999999e-05 2024-09-13T14:42:54.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:54.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.554250+0000", 2024-09-13T14:42:54.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6667000000000002e-05 2024-09-13T14:42:54.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:54.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.554259+0000", 2024-09-13T14:42:54.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3059999999999998e-06 2024-09-13T14:42:54.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:54.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.554315+0000", 2024-09-13T14:42:54.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5921000000000002e-05 2024-09-13T14:42:54.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:54.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.554487+0000", 2024-09-13T14:42:54.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000172769 2024-09-13T14:42:54.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:42:54.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.554656+0000", 2024-09-13T14:42:54.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000169075 2024-09-13T14:42:54.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:54.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.557043+0000", 2024-09-13T14:42:54.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002386113 2024-09-13T14:42:54.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:54.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.566192+0000", 2024-09-13T14:42:54.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.009149941 2024-09-13T14:42:54.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:54.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.654365+0000", 2024-09-13T14:42:54.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.088173006999999998 2024-09-13T14:42:54.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:54.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.654422+0000", 2024-09-13T14:42:54.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6276999999999997e-05 2024-09-13T14:42:54.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:54.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.654474+0000", 2024-09-13T14:42:54.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2649999999999999e-05 2024-09-13T14:42:54.419 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.419 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:54.419 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.420 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:63707 6.4 6:2668e18c:::benchmark_data_smithi151_155836_object63706:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1088)", 2024-09-13T14:42:54.420 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:39:55.560346+0000", 2024-09-13T14:42:54.420 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 178.09852548200001, 2024-09-13T14:42:54.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.106071423, 2024-09-13T14:42:54.420 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:54.420 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:54.420 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:54.420 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:54.420 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:54.420 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:54.420 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 63707 2024-09-13T14:42:54.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.421 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:54.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:54.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.560346+0000", 2024-09-13T14:42:54.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:54.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.560346+0000", 2024-09-13T14:42:54.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:54.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.560348+0000", 2024-09-13T14:42:54.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.046e-06 2024-09-13T14:42:54.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:54.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.560382+0000", 2024-09-13T14:42:54.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3420000000000002e-05 2024-09-13T14:42:54.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:54.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.560401+0000", 2024-09-13T14:42:54.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9358e-05 2024-09-13T14:42:54.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:54.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.560411+0000", 2024-09-13T14:42:54.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5619999999999993e-06 2024-09-13T14:42:54.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:54.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.560491+0000", 2024-09-13T14:42:54.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0130999999999996e-05 2024-09-13T14:42:54.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:54.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.560659+0000", 2024-09-13T14:42:54.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000168305 2024-09-13T14:42:54.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:42:54.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.560833+0000", 2024-09-13T14:42:54.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000174022 2024-09-13T14:42:54.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:54.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.562873+0000", 2024-09-13T14:42:54.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020398780000000002 2024-09-13T14:42:54.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:54.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.568252+0000", 2024-09-13T14:42:54.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0053794589999999996 2024-09-13T14:42:54.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:54.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.666303+0000", 2024-09-13T14:42:54.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.098050494000000002 2024-09-13T14:42:54.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:54.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.666364+0000", 2024-09-13T14:42:54.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1535e-05 2024-09-13T14:42:54.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.426 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:54.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.666418+0000", 2024-09-13T14:42:54.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3213e-05 2024-09-13T14:42:54.426 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.426 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:54.426 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.426 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:63715 6.4 6:2d79d07d:::benchmark_data_smithi151_155836_object63714:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1088)", 2024-09-13T14:42:54.426 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:39:55.563619+0000", 2024-09-13T14:42:54.426 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 178.095252808, 2024-09-13T14:42:54.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.104432595, 2024-09-13T14:42:54.426 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:54.427 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:54.427 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:54.427 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:54.427 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:54.427 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:54.427 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 63715 2024-09-13T14:42:54.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.427 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:54.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.427 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:54.427 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.563619+0000", 2024-09-13T14:42:54.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.428 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:54.428 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.563619+0000", 2024-09-13T14:42:54.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.428 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:54.428 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.563621+0000", 2024-09-13T14:42:54.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8300000000000001e-06 2024-09-13T14:42:54.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:54.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.563648+0000", 2024-09-13T14:42:54.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7759e-05 2024-09-13T14:42:54.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:54.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.563664+0000", 2024-09-13T14:42:54.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5648999999999999e-05 2024-09-13T14:42:54.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:54.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.563673+0000", 2024-09-13T14:42:54.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0690000000000008e-06 2024-09-13T14:42:54.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:54.431 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.563729+0000", 2024-09-13T14:42:54.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6156999999999998e-05 2024-09-13T14:42:54.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:54.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.563878+0000", 2024-09-13T14:42:54.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014882900000000001 2024-09-13T14:42:54.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:42:54.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.564035+0000", 2024-09-13T14:42:54.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000157188 2024-09-13T14:42:54.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:54.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.568198+0000", 2024-09-13T14:42:54.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0041622839999999996 2024-09-13T14:42:54.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:54.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.571237+0000", 2024-09-13T14:42:54.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003039295 2024-09-13T14:42:54.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:54.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.667963+0000", 2024-09-13T14:42:54.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.096725820000000004 2024-09-13T14:42:54.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.435 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:54.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.668013+0000", 2024-09-13T14:42:54.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0489999999999999e-05 2024-09-13T14:42:54.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.436 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:54.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.668051+0000", 2024-09-13T14:42:54.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8225000000000001e-05 2024-09-13T14:42:54.436 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.436 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:54.437 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.437 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:63717 6.4 6:257b9bcd:::benchmark_data_smithi151_155836_object63716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1088)", 2024-09-13T14:42:54.437 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:39:55.565577+0000", 2024-09-13T14:42:54.437 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 178.09329423700001, 2024-09-13T14:42:54.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.102593225, 2024-09-13T14:42:54.437 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:54.437 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:54.437 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:54.437 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:54.437 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:54.438 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:54.438 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 63717 2024-09-13T14:42:54.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.438 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:54.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:54.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.565577+0000", 2024-09-13T14:42:54.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:54.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.565577+0000", 2024-09-13T14:42:54.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:54.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.565579+0000", 2024-09-13T14:42:54.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.702e-06 2024-09-13T14:42:54.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:54.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.565603+0000", 2024-09-13T14:42:54.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4136999999999999e-05 2024-09-13T14:42:54.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:54.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.565623+0000", 2024-09-13T14:42:54.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9337000000000001e-05 2024-09-13T14:42:54.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:54.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.565631+0000", 2024-09-13T14:42:54.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3629999999999994e-06 2024-09-13T14:42:54.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:54.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.566218+0000", 2024-09-13T14:42:54.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00058655899999999995 2024-09-13T14:42:54.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:54.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.566323+0000", 2024-09-13T14:42:54.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105506 2024-09-13T14:42:54.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:42:54.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.566438+0000", 2024-09-13T14:42:54.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114453 2024-09-13T14:42:54.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:54.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.571280+0000", 2024-09-13T14:42:54.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0048429390000000001 2024-09-13T14:42:54.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:54.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.571380+0000", 2024-09-13T14:42:54.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9130999999999997e-05 2024-09-13T14:42:54.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:54.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.668087+0000", 2024-09-13T14:42:54.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.096707357999999993 2024-09-13T14:42:54.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:54.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.668135+0000", 2024-09-13T14:42:54.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8520000000000003e-05 2024-09-13T14:42:54.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:54.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.668171+0000", 2024-09-13T14:42:54.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5219999999999998e-05 2024-09-13T14:42:54.443 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.444 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:54.444 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.444 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:63719 6.4 6:27bfbc92:::benchmark_data_smithi151_155836_object63718:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1088)", 2024-09-13T14:42:54.444 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:39:55.567063+0000", 2024-09-13T14:42:54.444 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 178.09180859200001, 2024-09-13T14:42:54.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.103515671, 2024-09-13T14:42:54.444 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:54.444 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:54.444 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:54.444 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:54.444 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:54.445 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:54.445 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 63719 2024-09-13T14:42:54.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.445 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:54.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:54.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.567063+0000", 2024-09-13T14:42:54.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:54.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.567063+0000", 2024-09-13T14:42:54.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:54.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.567064+0000", 2024-09-13T14:42:54.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.189e-06 2024-09-13T14:42:54.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:54.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.567115+0000", 2024-09-13T14:42:54.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0234e-05 2024-09-13T14:42:54.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:54.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.567133+0000", 2024-09-13T14:42:54.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8474999999999998e-05 2024-09-13T14:42:54.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:54.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.567142+0000", 2024-09-13T14:42:54.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9020000000000005e-06 2024-09-13T14:42:54.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:54.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.568285+0000", 2024-09-13T14:42:54.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001143177 2024-09-13T14:42:54.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:54.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.568419+0000", 2024-09-13T14:42:54.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013382699999999999 2024-09-13T14:42:54.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:42:54.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.568556+0000", 2024-09-13T14:42:54.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013754699999999999 2024-09-13T14:42:54.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:54.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.571367+0000", 2024-09-13T14:42:54.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002810758 2024-09-13T14:42:54.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:54.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.571400+0000", 2024-09-13T14:42:54.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2394000000000002e-05 2024-09-13T14:42:54.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:54.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.670405+0000", 2024-09-13T14:42:54.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.099005211999999995 2024-09-13T14:42:54.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:54.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.670533+0000", 2024-09-13T14:42:54.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012807000000000001 2024-09-13T14:42:54.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:54.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.670579+0000", 2024-09-13T14:42:54.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5886000000000001e-05 2024-09-13T14:42:54.450 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.450 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:54.450 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.451 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:63726 6.4 6:2643d4fa:::benchmark_data_smithi151_155836_object63725:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1088)", 2024-09-13T14:42:54.451 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:39:55.575189+0000", 2024-09-13T14:42:54.451 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 178.08368308499999, 2024-09-13T14:42:54.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.098080497000000003, 2024-09-13T14:42:54.451 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:54.451 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:54.451 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:54.451 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:54.451 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:54.451 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:54.451 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 63726 2024-09-13T14:42:54.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.452 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:54.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:54.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.575189+0000", 2024-09-13T14:42:54.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:54.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.575189+0000", 2024-09-13T14:42:54.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:54.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.575191+0000", 2024-09-13T14:42:54.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.013e-06 2024-09-13T14:42:54.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:54.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.575218+0000", 2024-09-13T14:42:54.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7826000000000001e-05 2024-09-13T14:42:54.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:54.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.575235+0000", 2024-09-13T14:42:54.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6461999999999999e-05 2024-09-13T14:42:54.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:54.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.575243+0000", 2024-09-13T14:42:54.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3569999999999998e-06 2024-09-13T14:42:54.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:54.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.575301+0000", 2024-09-13T14:42:54.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8041000000000002e-05 2024-09-13T14:42:54.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:54.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.575451+0000", 2024-09-13T14:42:54.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000149371 2024-09-13T14:42:54.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:42:54.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.575604+0000", 2024-09-13T14:42:54.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015333600000000001 2024-09-13T14:42:54.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:54.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.577647+0000", 2024-09-13T14:42:54.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002042889 2024-09-13T14:42:54.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:54.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.577673+0000", 2024-09-13T14:42:54.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5925e-05 2024-09-13T14:42:54.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:54.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.673154+0000", 2024-09-13T14:42:54.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.095481607999999996 2024-09-13T14:42:54.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:54.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.673230+0000", 2024-09-13T14:42:54.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5166000000000005e-05 2024-09-13T14:42:54.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:54.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.673269+0000", 2024-09-13T14:42:54.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9502999999999998e-05 2024-09-13T14:42:54.457 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.457 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:54.457 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.457 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:63731 6.4 6:2b903c7a:::benchmark_data_smithi151_155836_object63730:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1088)", 2024-09-13T14:42:54.457 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:39:55.582419+0000", 2024-09-13T14:42:54.457 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 178.076452346, 2024-09-13T14:42:54.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.093284669000000001, 2024-09-13T14:42:54.458 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:54.458 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:54.458 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:54.458 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:54.458 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:54.458 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:54.458 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 63731 2024-09-13T14:42:54.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.458 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:54.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:54.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.582419+0000", 2024-09-13T14:42:54.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:54.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.582419+0000", 2024-09-13T14:42:54.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:54.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.582421+0000", 2024-09-13T14:42:54.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8029999999999999e-06 2024-09-13T14:42:54.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:54.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.582443+0000", 2024-09-13T14:42:54.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1790999999999999e-05 2024-09-13T14:42:54.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:54.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.582460+0000", 2024-09-13T14:42:54.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6614999999999999e-05 2024-09-13T14:42:54.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:54.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.582466+0000", 2024-09-13T14:42:54.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4389999999999997e-06 2024-09-13T14:42:54.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:54.461 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.582519+0000", 2024-09-13T14:42:54.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3174999999999998e-05 2024-09-13T14:42:54.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:54.461 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.582623+0000", 2024-09-13T14:42:54.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103818 2024-09-13T14:42:54.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:42:54.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.582718+0000", 2024-09-13T14:42:54.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5178000000000003e-05 2024-09-13T14:42:54.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:54.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.584449+0000", 2024-09-13T14:42:54.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001730875 2024-09-13T14:42:54.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:54.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.584542+0000", 2024-09-13T14:42:54.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2955999999999996e-05 2024-09-13T14:42:54.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:54.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.675603+0000", 2024-09-13T14:42:54.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.091061469000000006 2024-09-13T14:42:54.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:54.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.675661+0000", 2024-09-13T14:42:54.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7361999999999997e-05 2024-09-13T14:42:54.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:54.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:55.675704+0000", 2024-09-13T14:42:54.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3188000000000001e-05 2024-09-13T14:42:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:65140 6.4 6:250f28d4:::benchmark_data_smithi151_155836_object65139:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1088)", 2024-09-13T14:42:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:39:56.576330+0000", 2024-09-13T14:42:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 177.08254170500001, 2024-09-13T14:42:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.055446729, 2024-09-13T14:42:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 65140 2024-09-13T14:42:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.576330+0000", 2024-09-13T14:42:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.576330+0000", 2024-09-13T14:42:54.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:54.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.576331+0000", 2024-09-13T14:42:54.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3799999999999996e-07 2024-09-13T14:42:54.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:54.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.576350+0000", 2024-09-13T14:42:54.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9301e-05 2024-09-13T14:42:54.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:54.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.576365+0000", 2024-09-13T14:42:54.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5128e-05 2024-09-13T14:42:54.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:54.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.576372+0000", 2024-09-13T14:42:54.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.922e-06 2024-09-13T14:42:54.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:54.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.576430+0000", 2024-09-13T14:42:54.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7374000000000003e-05 2024-09-13T14:42:54.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:54.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.576540+0000", 2024-09-13T14:42:54.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000110702 2024-09-13T14:42:54.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:42:54.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.576658+0000", 2024-09-13T14:42:54.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011777099999999999 2024-09-13T14:42:54.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:54.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.578448+0000", 2024-09-13T14:42:54.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017901600000000001 2024-09-13T14:42:54.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:54.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.583546+0000", 2024-09-13T14:42:54.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0050982299999999996 2024-09-13T14:42:54.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:54.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.583716+0000", 2024-09-13T14:42:54.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016953899999999999 2024-09-13T14:42:54.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:54.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.631727+0000", 2024-09-13T14:42:54.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048010875000000001 2024-09-13T14:42:54.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:54.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.631777+0000", 2024-09-13T14:42:54.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9889000000000003e-05 2024-09-13T14:42:54.470 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.470 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:54.470 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.471 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.35289.0:65142 6.d e1088/1082 6:b077a29e:::benchmark_data_smithi151_155836_object65141:head v 1088'4052, mlcod=1088'4050)", 2024-09-13T14:42:54.471 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:39:56.583195+0000", 2024-09-13T14:42:54.471 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 177.07567705899999, 2024-09-13T14:42:54.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048543203, 2024-09-13T14:42:54.471 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:54.471 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:54.471 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:54.471 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:54.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:54.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.583195+0000", 2024-09-13T14:42:54.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:54.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.583195+0000", 2024-09-13T14:42:54.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:54.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.583196+0000", 2024-09-13T14:42:54.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5269999999999999e-06 2024-09-13T14:42:54.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:54.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.583254+0000", 2024-09-13T14:42:54.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7745999999999999e-05 2024-09-13T14:42:54.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:54.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.583281+0000", 2024-09-13T14:42:54.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.686e-05 2024-09-13T14:42:54.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:54.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.583316+0000", 2024-09-13T14:42:54.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4755999999999997e-05 2024-09-13T14:42:54.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:54.474 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.583449+0000", 2024-09-13T14:42:54.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000133287 2024-09-13T14:42:54.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:54.474 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.583466+0000", 2024-09-13T14:42:54.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.702e-05 2024-09-13T14:42:54.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:54.474 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.631682+0000", 2024-09-13T14:42:54.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.04821607 2024-09-13T14:42:54.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.475 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:54.475 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.631738+0000", 2024-09-13T14:42:54.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5937000000000003e-05 2024-09-13T14:42:54.475 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.475 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:54.475 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.475 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.35289.0:65139 6.e e1088/1082 6:7f9acf57:::benchmark_data_smithi151_155836_object65138:head v 1088'3980, mlcod=1088'3976)", 2024-09-13T14:42:54.475 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:39:56.583516+0000", 2024-09-13T14:42:54.475 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 177.07535526699999, 2024-09-13T14:42:54.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.056258381000000003, 2024-09-13T14:42:54.476 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:54.476 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:54.476 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:54.476 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:54.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:54.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.583516+0000", 2024-09-13T14:42:54.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:54.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.583516+0000", 2024-09-13T14:42:54.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:54.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.583517+0000", 2024-09-13T14:42:54.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2899999999999999e-07 2024-09-13T14:42:54.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:54.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.583561+0000", 2024-09-13T14:42:54.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4372000000000001e-05 2024-09-13T14:42:54.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:54.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.583569+0000", 2024-09-13T14:42:54.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2199999999999992e-06 2024-09-13T14:42:54.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:54.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.583575+0000", 2024-09-13T14:42:54.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3299999999999998e-06 2024-09-13T14:42:54.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:54.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.583606+0000", 2024-09-13T14:42:54.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1016999999999998e-05 2024-09-13T14:42:54.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:54.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.583617+0000", 2024-09-13T14:42:54.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0892e-05 2024-09-13T14:42:54.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:54.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.639726+0000", 2024-09-13T14:42:54.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.056109489999999998 2024-09-13T14:42:54.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:54.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.639775+0000", 2024-09-13T14:42:54.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8631000000000002e-05 2024-09-13T14:42:54.480 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.480 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:54.480 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.480 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.35289.0:65140 6.4 e1088/1082 ondisk, result = 0)", 2024-09-13T14:42:54.480 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:39:56.583644+0000", 2024-09-13T14:42:54.480 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 177.07522729900001, 2024-09-13T14:42:54.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048139921000000002, 2024-09-13T14:42:54.480 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:54.481 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:54.481 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-13T14:42:54.481 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:54.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:54.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.583644+0000", 2024-09-13T14:42:54.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:54.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.583644+0000", 2024-09-13T14:42:54.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:54.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.583645+0000", 2024-09-13T14:42:54.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8899999999999999e-07 2024-09-13T14:42:54.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:54.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.583649+0000", 2024-09-13T14:42:54.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6619999999999998e-06 2024-09-13T14:42:54.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:54.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.583656+0000", 2024-09-13T14:42:54.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6190000000000002e-06 2024-09-13T14:42:54.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:54.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.583662+0000", 2024-09-13T14:42:54.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8749999999999997e-06 2024-09-13T14:42:54.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:54.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.583708+0000", 2024-09-13T14:42:54.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6026999999999998e-05 2024-09-13T14:42:54.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:54.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.583713+0000", 2024-09-13T14:42:54.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.668e-06 2024-09-13T14:42:54.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:54.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.631784+0000", 2024-09-13T14:42:54.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048071480999999999 2024-09-13T14:42:54.484 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.485 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:54.485 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.485 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.35289.0:65146 6.b e1088/1082 6:da598ffa:::benchmark_data_smithi151_155836_object65145:head v 1088'4119, mlcod=1088'4117)", 2024-09-13T14:42:54.485 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:39:56.584866+0000", 2024-09-13T14:42:54.485 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 177.07400598199999, 2024-09-13T14:42:54.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050869966000000003, 2024-09-13T14:42:54.485 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:54.485 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:54.485 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:54.485 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:54.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:54.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.584866+0000", 2024-09-13T14:42:54.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:54.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.584866+0000", 2024-09-13T14:42:54.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:54.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.584866+0000", 2024-09-13T14:42:54.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5099999999999999e-07 2024-09-13T14:42:54.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:54.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.584890+0000", 2024-09-13T14:42:54.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3467000000000001e-05 2024-09-13T14:42:54.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:54.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.633304+0000", 2024-09-13T14:42:54.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048413954000000002 2024-09-13T14:42:54.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:54.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.633315+0000", 2024-09-13T14:42:54.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0958000000000001e-05 2024-09-13T14:42:54.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:54.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.633378+0000", 2024-09-13T14:42:54.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3487999999999999e-05 2024-09-13T14:42:54.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:54.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.633408+0000", 2024-09-13T14:42:54.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9708000000000001e-05 2024-09-13T14:42:54.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.635695+0000", 2024-09-13T14:42:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0022866570000000001 2024-09-13T14:42:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.635736+0000", 2024-09-13T14:42:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0982999999999997e-05 2024-09-13T14:42:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.490 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.35289.0:65150 6.e e1088/1082 6:7eb31fd4:::benchmark_data_smithi151_155836_object65149:head v 1088'3981, mlcod=1088'3978)", 2024-09-13T14:42:54.490 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:39:56.586846+0000", 2024-09-13T14:42:54.490 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 177.07202610499999, 2024-09-13T14:42:54.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052976648000000001, 2024-09-13T14:42:54.490 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:54.490 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:54.490 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:54.490 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:54.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:54.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.586846+0000", 2024-09-13T14:42:54.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:54.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.586846+0000", 2024-09-13T14:42:54.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:54.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.586846+0000", 2024-09-13T14:42:54.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4799999999999998e-07 2024-09-13T14:42:54.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:54.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.586894+0000", 2024-09-13T14:42:54.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.727e-05 2024-09-13T14:42:54.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:54.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.631666+0000", 2024-09-13T14:42:54.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.044772609999999997 2024-09-13T14:42:54.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:54.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.631677+0000", 2024-09-13T14:42:54.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0747000000000001e-05 2024-09-13T14:42:54.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:54.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.635463+0000", 2024-09-13T14:42:54.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037861119999999999 2024-09-13T14:42:54.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:54.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.635484+0000", 2024-09-13T14:42:54.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1202000000000001e-05 2024-09-13T14:42:54.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:54.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.639783+0000", 2024-09-13T14:42:54.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0042984729999999997 2024-09-13T14:42:54.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:54.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.639822+0000", 2024-09-13T14:42:54.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9585999999999997e-05 2024-09-13T14:42:54.494 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.494 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:54.494 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.494 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:65180 6.1 6:80660476:::benchmark_data_smithi151_155836_object65179:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1088)", 2024-09-13T14:42:54.495 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:39:56.643660+0000", 2024-09-13T14:42:54.495 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 177.01521213300001, 2024-09-13T14:42:54.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048827176, 2024-09-13T14:42:54.495 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:54.495 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:54.495 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:54.495 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:54.495 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:54.495 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:54.495 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 65180 2024-09-13T14:42:54.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.495 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:54.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:54.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.643660+0000", 2024-09-13T14:42:54.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:54.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.643660+0000", 2024-09-13T14:42:54.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:54.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.643661+0000", 2024-09-13T14:42:54.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.375e-06 2024-09-13T14:42:54.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:54.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.643683+0000", 2024-09-13T14:42:54.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1858999999999999e-05 2024-09-13T14:42:54.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:54.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.643697+0000", 2024-09-13T14:42:54.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4385e-05 2024-09-13T14:42:54.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:54.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.643705+0000", 2024-09-13T14:42:54.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1829999999999994e-06 2024-09-13T14:42:54.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:54.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.643779+0000", 2024-09-13T14:42:54.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3273000000000001e-05 2024-09-13T14:42:54.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:54.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.643925+0000", 2024-09-13T14:42:54.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014602199999999999 2024-09-13T14:42:54.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:42:54.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.644101+0000", 2024-09-13T14:42:54.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000176544 2024-09-13T14:42:54.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:54.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.648668+0000", 2024-09-13T14:42:54.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0045667290000000003 2024-09-13T14:42:54.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:54.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.651535+0000", 2024-09-13T14:42:54.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028668500000000002 2024-09-13T14:42:54.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:54.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.692390+0000", 2024-09-13T14:42:54.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.040855252000000002 2024-09-13T14:42:54.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:54.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.692443+0000", 2024-09-13T14:42:54.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3035000000000003e-05 2024-09-13T14:42:54.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:54.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:39:56.692487+0000", 2024-09-13T14:42:54.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3668999999999998e-05 2024-09-13T14:42:54.501 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.501 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:54.501 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.501 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:100063 6.4 6:20b2b3e6:::benchmark_data_smithi151_155836_object100062:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1091)", 2024-09-13T14:42:54.501 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:40:24.439439+0000", 2024-09-13T14:42:54.501 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 149.219432353, 2024-09-13T14:42:54.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049622230000000003, 2024-09-13T14:42:54.501 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:54.501 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:54.501 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:54.501 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:54.502 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:54.502 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:54.502 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 100063 2024-09-13T14:42:54.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.502 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:54.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:54.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:24.439439+0000", 2024-09-13T14:42:54.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:54.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:24.439439+0000", 2024-09-13T14:42:54.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:54.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:24.439440+0000", 2024-09-13T14:42:54.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0999999999999999e-07 2024-09-13T14:42:54.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:54.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:24.439460+0000", 2024-09-13T14:42:54.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9737000000000001e-05 2024-09-13T14:42:54.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:54.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:24.439466+0000", 2024-09-13T14:42:54.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6030000000000001e-06 2024-09-13T14:42:54.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:54.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:24.439471+0000", 2024-09-13T14:42:54.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9679999999999997e-06 2024-09-13T14:42:54.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:54.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:24.442802+0000", 2024-09-13T14:42:54.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003331274 2024-09-13T14:42:54.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:54.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:24.442921+0000", 2024-09-13T14:42:54.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118682 2024-09-13T14:42:54.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.505 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:42:54.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:24.443050+0000", 2024-09-13T14:42:54.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012854500000000001 2024-09-13T14:42:54.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:54.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:24.450537+0000", 2024-09-13T14:42:54.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0074872949999999997 2024-09-13T14:42:54.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:54.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:24.459023+0000", 2024-09-13T14:42:54.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.008485724 2024-09-13T14:42:54.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:54.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:24.488900+0000", 2024-09-13T14:42:54.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.029877007000000001 2024-09-13T14:42:54.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:54.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:24.489012+0000", 2024-09-13T14:42:54.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011211 2024-09-13T14:42:54.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:54.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:24.489062+0000", 2024-09-13T14:42:54.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9874999999999999e-05 2024-09-13T14:42:54.510 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.510 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:54.510 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.510 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:105794 6.1 6:8aa8d5d3:::benchmark_data_smithi151_155836_object105793:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1092)", 2024-09-13T14:42:54.510 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:40:29.794929+0000", 2024-09-13T14:42:54.510 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 143.863942259, 2024-09-13T14:42:54.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.058581611999999998, 2024-09-13T14:42:54.510 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:54.511 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:54.511 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:54.511 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:54.511 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:54.511 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:54.511 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 105794 2024-09-13T14:42:54.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.511 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:54.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:54.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:29.794929+0000", 2024-09-13T14:42:54.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:54.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:29.794929+0000", 2024-09-13T14:42:54.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:54.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:29.794931+0000", 2024-09-13T14:42:54.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.314e-06 2024-09-13T14:42:54.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:54.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:29.794949+0000", 2024-09-13T14:42:54.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8714000000000001e-05 2024-09-13T14:42:54.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:54.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:29.794959+0000", 2024-09-13T14:42:54.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8749999999999995e-06 2024-09-13T14:42:54.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:54.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:29.794965+0000", 2024-09-13T14:42:54.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3199999999999999e-06 2024-09-13T14:42:54.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:54.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:29.795095+0000", 2024-09-13T14:42:54.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000130789 2024-09-13T14:42:54.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:54.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:29.795342+0000", 2024-09-13T14:42:54.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024629200000000001 2024-09-13T14:42:54.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:42:54.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:29.795464+0000", 2024-09-13T14:42:54.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000122593 2024-09-13T14:42:54.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:54.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:29.798877+0000", 2024-09-13T14:42:54.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034124070000000001 2024-09-13T14:42:54.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:54.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:29.804461+0000", 2024-09-13T14:42:54.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0055846059999999998 2024-09-13T14:42:54.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:54.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:29.853411+0000", 2024-09-13T14:42:54.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048949419000000001 2024-09-13T14:42:54.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:54.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:29.853464+0000", 2024-09-13T14:42:54.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3535000000000002e-05 2024-09-13T14:42:54.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:54.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:29.853511+0000", 2024-09-13T14:42:54.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6748e-05 2024-09-13T14:42:54.516 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.517 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:54.517 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.517 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:111456 6.1 6:86382108:::benchmark_data_smithi151_155836_object111455:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1092)", 2024-09-13T14:42:54.517 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:40:35.341528+0000", 2024-09-13T14:42:54.517 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 138.31734388000001, 2024-09-13T14:42:54.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048322704000000001, 2024-09-13T14:42:54.517 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:54.517 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:54.517 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:54.517 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:54.517 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:54.518 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:54.518 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 111456 2024-09-13T14:42:54.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.518 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:54.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:54.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:35.341528+0000", 2024-09-13T14:42:54.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:54.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:35.341528+0000", 2024-09-13T14:42:54.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:54.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:35.341529+0000", 2024-09-13T14:42:54.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7000000000000004e-07 2024-09-13T14:42:54.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:54.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:35.341551+0000", 2024-09-13T14:42:54.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2140000000000001e-05 2024-09-13T14:42:54.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.520 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:54.520 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:35.341559+0000", 2024-09-13T14:42:54.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8420000000000005e-06 2024-09-13T14:42:54.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.520 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:54.520 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:35.341565+0000", 2024-09-13T14:42:54.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2659999999999998e-06 2024-09-13T14:42:54.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:54.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:35.347353+0000", 2024-09-13T14:42:54.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0057885439999999996 2024-09-13T14:42:54.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:54.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:35.347500+0000", 2024-09-13T14:42:54.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014652900000000001 2024-09-13T14:42:54.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:42:54.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:35.347659+0000", 2024-09-13T14:42:54.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015884100000000001 2024-09-13T14:42:54.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:54.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:35.369021+0000", 2024-09-13T14:42:54.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021362055000000001 2024-09-13T14:42:54.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:54.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:35.369064+0000", 2024-09-13T14:42:54.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2879000000000001e-05 2024-09-13T14:42:54.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:54.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:35.389738+0000", 2024-09-13T14:42:54.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020674719000000001 2024-09-13T14:42:54.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:54.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:35.389801+0000", 2024-09-13T14:42:54.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2261999999999994e-05 2024-09-13T14:42:54.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:54.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:40:35.389851+0000", 2024-09-13T14:42:54.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9857000000000001e-05 2024-09-13T14:42:54.523 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.523 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:54.523 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.524 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:136289 6.4 6:2e5cc4e1:::benchmark_data_smithi151_155836_object3831:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1094)", 2024-09-13T14:42:54.524 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:41:01.310089+0000", 2024-09-13T14:42:54.524 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 112.34878270900001, 2024-09-13T14:42:54.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053792802000000001, 2024-09-13T14:42:54.524 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:54.524 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:54.524 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:54.524 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:54.524 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:54.524 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:54.524 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 136289 2024-09-13T14:42:54.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.525 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:54.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:54.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.310089+0000", 2024-09-13T14:42:54.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:54.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.310089+0000", 2024-09-13T14:42:54.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.526 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:54.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.310140+0000", 2024-09-13T14:42:54.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1031e-05 2024-09-13T14:42:54.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.526 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:54.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.310146+0000", 2024-09-13T14:42:54.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3380000000000001e-06 2024-09-13T14:42:54.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.526 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:54.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.310160+0000", 2024-09-13T14:42:54.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3482999999999999e-05 2024-09-13T14:42:54.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:54.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.310168+0000", 2024-09-13T14:42:54.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8879999999999997e-06 2024-09-13T14:42:54.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:54.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.310987+0000", 2024-09-13T14:42:54.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00081918599999999996 2024-09-13T14:42:54.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:54.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.311090+0000", 2024-09-13T14:42:54.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010330199999999999 2024-09-13T14:42:54.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:42:54.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.311155+0000", 2024-09-13T14:42:54.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5240999999999994e-05 2024-09-13T14:42:54.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:54.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.317276+0000", 2024-09-13T14:42:54.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.006120281 2024-09-13T14:42:54.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:54.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.321679+0000", 2024-09-13T14:42:54.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0044031720000000003 2024-09-13T14:42:54.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:54.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.363778+0000", 2024-09-13T14:42:54.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.042099090999999998 2024-09-13T14:42:54.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:54.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.363830+0000", 2024-09-13T14:42:54.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2439999999999999e-05 2024-09-13T14:42:54.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:54.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.363882+0000", 2024-09-13T14:42:54.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1348999999999999e-05 2024-09-13T14:42:54.530 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.530 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:54.530 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.530 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:136292 6.4 6:2ba80663:::benchmark_data_smithi151_155836_object3834:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1094)", 2024-09-13T14:42:54.530 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:41:01.311041+0000", 2024-09-13T14:42:54.530 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 112.347830785, 2024-09-13T14:42:54.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052957463000000003, 2024-09-13T14:42:54.531 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:54.531 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:54.531 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:54.531 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:54.531 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:54.531 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:54.531 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 136292 2024-09-13T14:42:54.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.531 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:54.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:54.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.311041+0000", 2024-09-13T14:42:54.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.532 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:54.532 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.311041+0000", 2024-09-13T14:42:54.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.532 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:54.532 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.311042+0000", 2024-09-13T14:42:54.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5000000000000002e-07 2024-09-13T14:42:54.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:54.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.311046+0000", 2024-09-13T14:42:54.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.357e-06 2024-09-13T14:42:54.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:54.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.311056+0000", 2024-09-13T14:42:54.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0214000000000001e-05 2024-09-13T14:42:54.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:54.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.311063+0000", 2024-09-13T14:42:54.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.635e-06 2024-09-13T14:42:54.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.534 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:54.534 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.314070+0000", 2024-09-13T14:42:54.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003007217 2024-09-13T14:42:54.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.534 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:54.534 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.314269+0000", 2024-09-13T14:42:54.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019851100000000001 2024-09-13T14:42:54.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.534 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:42:54.534 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.314405+0000", 2024-09-13T14:42:54.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013620200000000001 2024-09-13T14:42:54.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:54.535 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.317284+0000", 2024-09-13T14:42:54.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002879548 2024-09-13T14:42:54.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:54.535 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.321701+0000", 2024-09-13T14:42:54.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0044167030000000001 2024-09-13T14:42:54.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:54.535 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.363920+0000", 2024-09-13T14:42:54.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.042219117 2024-09-13T14:42:54.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:54.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.363958+0000", 2024-09-13T14:42:54.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7389999999999999e-05 2024-09-13T14:42:54.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:54.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:01.363998+0000", 2024-09-13T14:42:54.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0818999999999997e-05 2024-09-13T14:42:54.536 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.536 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:54.536 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.537 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:137319 6.4 6:2dcc428f:::benchmark_data_smithi151_155836_object4861:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1094)", 2024-09-13T14:42:54.537 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:41:02.471897+0000", 2024-09-13T14:42:54.537 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 111.18697478, 2024-09-13T14:42:54.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048985543999999999, 2024-09-13T14:42:54.537 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:42:54.537 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:42:54.537 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:42:54.537 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:42:54.537 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:42:54.537 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:42:54.537 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 137319 2024-09-13T14:42:54.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.538 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:42:54.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.538 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:42:54.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:02.471897+0000", 2024-09-13T14:42:54.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.538 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:42:54.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:02.471897+0000", 2024-09-13T14:42:54.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:42:54.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:42:54.539 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:02.471898+0000", 2024-09-13T14:42:54.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.516e-06 2024-09-13T14:42:54.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:42:54.539 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:02.471904+0000", 2024-09-13T14:42:54.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6790000000000002e-06 2024-09-13T14:42:54.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:42:54.539 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:02.471919+0000", 2024-09-13T14:42:54.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4773000000000001e-05 2024-09-13T14:42:54.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.540 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:42:54.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:02.471927+0000", 2024-09-13T14:42:54.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7379999999999994e-06 2024-09-13T14:42:54.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.540 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:42:54.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:02.471981+0000", 2024-09-13T14:42:54.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4749e-05 2024-09-13T14:42:54.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.540 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:42:54.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:02.472095+0000", 2024-09-13T14:42:54.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011343700000000001 2024-09-13T14:42:54.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:42:54.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:02.472208+0000", 2024-09-13T14:42:54.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000112821 2024-09-13T14:42:54.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:42:54.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:02.475570+0000", 2024-09-13T14:42:54.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033623920000000001 2024-09-13T14:42:54.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:54.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:02.478992+0000", 2024-09-13T14:42:54.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034224220000000001 2024-09-13T14:42:54.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:42:54.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:02.520812+0000", 2024-09-13T14:42:54.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041819262000000003 2024-09-13T14:42:54.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:42:54.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:02.520849+0000", 2024-09-13T14:42:54.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7503000000000003e-05 2024-09-13T14:42:54.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:42:54.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:42:54.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:42:54.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:02.520882+0000", 2024-09-13T14:42:54.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3251999999999999e-05 2024-09-13T14:42:54.543 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.543 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:54.543 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.543 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:42:54.543 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:42:54.543 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:42:54.544 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:54.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:54.583 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:54.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:54.684 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:54.727 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:54.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:54.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:54.737 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:42:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:42:54] "GET /metrics HTTP/1.1" 200 36683 "" "Prometheus/2.51.0" 2024-09-13T14:42:54.829 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:54.866 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:54.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:54.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:54.968 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:54.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:54.996 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:54.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:55.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:42:55.079 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:54 smithi151 ceph-mon[104450]: pgmap v1433: 129 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 121 active+clean; 1.7 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 6.3 KiB/s rd, 14 KiB/s wr, 992 op/s; 3561/83949 objects degraded (4.242%); 6329/83949 objects misplaced (7.539%); 817 KiB/s, 12 objects/s recovering 2024-09-13T14:42:55.079 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:54 smithi151 ceph-mon[108966]: pgmap v1433: 129 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 121 active+clean; 1.7 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 6.3 KiB/s rd, 14 KiB/s wr, 992 op/s; 3561/83949 objects degraded (4.242%); 6329/83949 objects misplaced (7.539%); 817 KiB/s, 12 objects/s recovering 2024-09-13T14:42:55.096 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:55.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:54 smithi184 ceph-mon[92527]: pgmap v1433: 129 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 121 active+clean; 1.7 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 6.3 KiB/s rd, 14 KiB/s wr, 992 op/s; 3561/83949 objects degraded (4.242%); 6329/83949 objects misplaced (7.539%); 817 KiB/s, 12 objects/s recovering 2024-09-13T14:42:55.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:55.127 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:55.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:55.228 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:55.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:55.268 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:55.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:55.369 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:55.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:55.406 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:55.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:55.507 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:55.546 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:55.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:55.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:55.648 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:55.680 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:55.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:55.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:55.782 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:55.813 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:55.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:55.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:55.915 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:55.954 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:55.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:55.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:56.057 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:56.070 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:55 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 3561/83949 objects degraded (4.242%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:42:56.103 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:56.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:56.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:56.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:55 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 3561/83949 objects degraded (4.242%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:42:56.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:55 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 3561/83949 objects degraded (4.242%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:42:56.205 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:56.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:56.234 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:56.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:56.335 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:56.360 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:56.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:56.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:56.462 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:56.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:56.501 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:56.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:56.604 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:56.642 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:56.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:56.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:56.744 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:56.776 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:56.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:56.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:56.878 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:56.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:56.910 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:56.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:57.011 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:57.047 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:57.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:57.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:57.061 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:56 smithi151 ceph-mon[104450]: pgmap v1434: 129 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 121 active+clean; 1.6 GiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 6.3 KiB/s rd, 14 KiB/s wr, 986 op/s; 3834/81123 objects degraded (4.726%); 6007/81123 objects misplaced (7.405%); 709 KiB/s, 10 objects/s recovering 2024-09-13T14:42:57.062 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:56 smithi151 ceph-mon[108966]: pgmap v1434: 129 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 121 active+clean; 1.6 GiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 6.3 KiB/s rd, 14 KiB/s wr, 986 op/s; 3834/81123 objects degraded (4.726%); 6007/81123 objects misplaced (7.405%); 709 KiB/s, 10 objects/s recovering 2024-09-13T14:42:57.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:56 smithi184 ceph-mon[92527]: pgmap v1434: 129 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 121 active+clean; 1.6 GiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 6.3 KiB/s rd, 14 KiB/s wr, 986 op/s; 3834/81123 objects degraded (4.726%); 6007/81123 objects misplaced (7.405%); 709 KiB/s, 10 objects/s recovering 2024-09-13T14:42:57.150 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:57.185 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:57.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:57.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:57.287 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:57.320 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:57.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:57.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:57.422 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:57.451 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:57.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:57.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:57.553 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:57.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:57.581 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:57.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:57.682 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:57.712 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:57.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:57.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:57.813 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:57.837 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:57.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:57.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:57.939 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:57.968 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:57.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:57.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:58.070 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:58.098 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:58.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:58.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:58.200 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:58.229 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:58.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:58.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:58.331 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:58.366 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:58.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:58.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:58.468 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:58.515 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:58.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:58.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:58.617 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:58.652 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:58.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:58.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:58.754 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:58.781 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:58.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:58.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:58.853 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T14:42:58.884 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:58.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:58.922 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:58.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:59.023 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:42:58 smithi151 ceph-mon[104450]: pgmap v1435: 129 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 121 active+clean; 1.6 GiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 5.9 KiB/s rd, 13 KiB/s wr, 929 op/s; 3834/80133 objects degraded (4.785%); 6007/80133 objects misplaced (7.496%); 665 KiB/s, 10 objects/s recovering 2024-09-13T14:42:59.024 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:42:58 smithi151 ceph-mon[108966]: pgmap v1435: 129 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 121 active+clean; 1.6 GiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 5.9 KiB/s rd, 13 KiB/s wr, 929 op/s; 3834/80133 objects degraded (4.785%); 6007/80133 objects misplaced (7.496%); 665 KiB/s, 10 objects/s recovering 2024-09-13T14:42:59.024 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:59.056 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:59.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:59.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:59.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:42:58 smithi184 ceph-mon[92527]: pgmap v1435: 129 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 121 active+clean; 1.6 GiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 5.9 KiB/s rd, 13 KiB/s wr, 929 op/s; 3834/80133 objects degraded (4.785%); 6007/80133 objects misplaced (7.496%); 665 KiB/s, 10 objects/s recovering 2024-09-13T14:42:59.158 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:59.188 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:59.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:59.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:59.290 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:59.313 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:59.327 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:59.328 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:59.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:59.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:59.431 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:59.475 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:59.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:59.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:59.576 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:59.605 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:59.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:59.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:59.706 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:59.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:59.740 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:42:59.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:59.842 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:59.877 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:42:59.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:59.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:59.979 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:00.012 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:00.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:00.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:00.114 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:00.144 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:00.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:00.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:00.246 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:00.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:00.277 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:00.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:00.378 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:00.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:00.410 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:00.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:00.511 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:00.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:00.549 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:00.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:00.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:43:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:43:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:43:00.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:00 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:43:00.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:00 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:43:00.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:00 smithi184 ceph-mon[92527]: pgmap v1436: 129 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 121 active+clean; 1.3 GiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 5.2 KiB/s rd, 11 KiB/s wr, 1.22k op/s; 4104/66000 objects degraded (6.218%); 4876/66000 objects misplaced (7.388%); 862 KiB/s, 13 objects/s recovering 2024-09-13T14:43:00.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:00 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 3834/80133 objects degraded (4.785%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:43:00.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:00 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:43:00.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:00 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:43:00.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:00 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:43:00.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:00 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:43:00.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:00 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:43:00.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:00 smithi151 ceph-mon[104450]: pgmap v1436: 129 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 121 active+clean; 1.3 GiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 5.2 KiB/s rd, 11 KiB/s wr, 1.22k op/s; 4104/66000 objects degraded (6.218%); 4876/66000 objects misplaced (7.388%); 862 KiB/s, 13 objects/s recovering 2024-09-13T14:43:00.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:00 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 3834/80133 objects degraded (4.785%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:43:00.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:00 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:43:00.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:00 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:43:00.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:00 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:43:00.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:00 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:43:00.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:00 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:43:00.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:00 smithi151 ceph-mon[108966]: pgmap v1436: 129 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 121 active+clean; 1.3 GiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 5.2 KiB/s rd, 11 KiB/s wr, 1.22k op/s; 4104/66000 objects degraded (6.218%); 4876/66000 objects misplaced (7.388%); 862 KiB/s, 13 objects/s recovering 2024-09-13T14:43:00.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:00 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 3834/80133 objects degraded (4.785%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:43:00.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:00 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:43:00.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:00 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:43:00.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:00 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:43:00.651 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:00.686 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:00.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:00.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:00.788 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:00.812 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:00.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:00.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:00.914 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:00.941 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:00.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:00.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:01.043 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:01.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:01.078 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:01.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:01.179 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:01.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:01.219 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:01.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:01.320 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:01.349 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:01.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:01.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:01.451 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:01.484 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:01.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:01.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:01.586 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:01.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:01.613 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:01.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:01.714 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:01.743 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:01.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:01.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:01.845 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:01.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:01 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:01.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:01.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:01 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:01.885 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:01.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:01.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:01.986 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:02.012 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:02.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:02.089 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:02.090 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:02.114 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:02.146 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:02.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:02.248 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:02.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.271 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:02.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:02.372 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:02.413 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:02.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:02.515 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:02.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.540 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:02.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:02.641 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:02.668 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:02.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:02.770 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:02.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.807 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:02.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:02.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:02 smithi184 ceph-mon[92527]: pgmap v1437: 129 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 121 active+clean; 1.3 GiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 875 op/s; 4104/65034 objects degraded (6.311%); 4876/65034 objects misplaced (7.498%); 533 KiB/s, 8 objects/s recovering 2024-09-13T14:43:02.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:02 smithi151 ceph-mon[104450]: pgmap v1437: 129 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 121 active+clean; 1.3 GiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 875 op/s; 4104/65034 objects degraded (6.311%); 4876/65034 objects misplaced (7.498%); 533 KiB/s, 8 objects/s recovering 2024-09-13T14:43:02.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:02 smithi151 ceph-mon[108966]: pgmap v1437: 129 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 121 active+clean; 1.3 GiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 875 op/s; 4104/65034 objects degraded (6.311%); 4876/65034 objects misplaced (7.498%); 533 KiB/s, 8 objects/s recovering 2024-09-13T14:43:02.908 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:02.937 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:02.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:03.039 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:03.065 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:03.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:03.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:03.166 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:03.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:03.200 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:03.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:03.301 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:03.332 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:03.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:03.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:03.434 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:03.466 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:03.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:03.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:03.568 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:03.603 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:43:03 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=cleanup t=2024-09-13T14:43:03.302284727Z level=info msg="Completed cleanup jobs" duration=12.538005ms 2024-09-13T14:43:03.606 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:03.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:03.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:03.708 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:03.748 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:03.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:03.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:03.850 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:03.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:03.882 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:03.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:03.983 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:04.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:04.018 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:04.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:04.119 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:04.157 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:04.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:04.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:04.259 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:04.291 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:04.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:04.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:04.393 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:04.418 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:04.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:04.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:04.520 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:04.547 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:04.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:04.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:04.648 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:04.688 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:04.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:04.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:04.736 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:43:04.736 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:43:04.736 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:04 smithi151 ceph-mon[104450]: pgmap v1438: 129 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 121 active+clean; 1.1 GiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 1.16k op/s; 4104/53784 objects degraded (7.631%); 4068/53784 objects misplaced (7.564%); 533 KiB/s, 8 objects/s recovering 2024-09-13T14:43:04.740 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:04 smithi151 ceph-mon[108966]: pgmap v1438: 129 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 121 active+clean; 1.1 GiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 1.16k op/s; 4104/53784 objects degraded (7.631%); 4068/53784 objects misplaced (7.564%); 533 KiB/s, 8 objects/s recovering 2024-09-13T14:43:04.740 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:43:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:43:04] "GET /metrics HTTP/1.1" 200 36679 "" "Prometheus/2.51.0" 2024-09-13T14:43:04.759 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:43:04.760 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:43:04.785 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:43:04.785 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:43:04.789 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:04.811 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:43:04.811 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:43:04.817 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:04.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:04.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:04.840 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:43:04.840 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:43:04.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:04 smithi184 ceph-mon[92527]: pgmap v1438: 129 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 121 active+clean; 1.1 GiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 1.16k op/s; 4104/53784 objects degraded (7.631%); 4068/53784 objects misplaced (7.564%); 533 KiB/s, 8 objects/s recovering 2024-09-13T14:43:04.867 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:43:04.867 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:43:04.893 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:43:04.893 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:43:04.920 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:04.923 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:43:04.923 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:43:04.954 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:04.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:04.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:05.056 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:05.084 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:05.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:05.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:05.186 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:05.211 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:05.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:05.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:05.312 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:05.344 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:05.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:05.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:05.445 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:05.476 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:05.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:05.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:05.578 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:05.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:05.616 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:05.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:05.716 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:05.717 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:05 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1912475474' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:05.718 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:05 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 4104/53784 objects degraded (7.631%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:43:05.718 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:05 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1912475474' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:05.718 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:05 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 4104/53784 objects degraded (7.631%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:43:05.749 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:05.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:05.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:05.851 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:05.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:05 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1912475474' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:05.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:05 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 4104/53784 objects degraded (7.631%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:43:05.880 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:05.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:05.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:05.982 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:06.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:06.014 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:06.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:06.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:05 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a[104427]: 2024-09-13T14:43:05.881+0000 7fe9da14f640 -1 mon.a@0(leader).osd e1123 definitely_dead 0 2024-09-13T14:43:06.114 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:06.141 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:06.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:06.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:06.244 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:06.278 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:06.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:06.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:06.380 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:06.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:06.407 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:06.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:06.483 INFO:teuthology.orchestra.run.smithi151.stderr:marked in osd.1. 2024-09-13T14:43:06.508 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:06.542 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:06.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:06.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:06.644 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:06.665 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:06.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:06.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:06.732 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:06 smithi151 ceph-mon[104450]: pgmap v1439: 129 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 121 active+clean; 1.0 GiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 918 op/s; 4378/50862 objects degraded (8.608%); 3741/50862 objects misplaced (7.355%); 816 KiB/s, 12 objects/s recovering 2024-09-13T14:43:06.732 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:06 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/4253108856' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-13T14:43:06.732 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:06 smithi151 ceph-mon[108966]: pgmap v1439: 129 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 121 active+clean; 1.0 GiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 918 op/s; 4378/50862 objects degraded (8.608%); 3741/50862 objects misplaced (7.355%); 816 KiB/s, 12 objects/s recovering 2024-09-13T14:43:06.732 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:06 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/4253108856' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-13T14:43:06.766 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:06.788 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:06.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:06.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:06.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:06 smithi184 ceph-mon[92527]: pgmap v1439: 129 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 121 active+clean; 1.0 GiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 918 op/s; 4378/50862 objects degraded (8.608%); 3741/50862 objects misplaced (7.355%); 816 KiB/s, 12 objects/s recovering 2024-09-13T14:43:06.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:06 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/4253108856' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-13T14:43:06.890 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:06.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:06.912 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:06.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:07.014 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:07.035 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:07.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:07.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:07.137 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:07.161 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:07.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:07.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:07.263 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:07.288 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:07.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:07.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:07.395 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:07.398 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-09-13T14:43:07.459 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:07.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:07.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:07.561 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:07.583 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:07.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:07.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:07.685 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:07.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:07.708 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:07.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:07.810 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:07.811 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:07 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/4253108856' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-13T14:43:07.812 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:07 smithi151 ceph-mon[104450]: osdmap e1124: 8 total, 8 up, 8 in 2024-09-13T14:43:07.812 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:07 smithi151 ceph-mon[104450]: 6.9 scrub starts 2024-09-13T14:43:07.812 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:07 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/4253108856' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-13T14:43:07.812 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:07 smithi151 ceph-mon[108966]: osdmap e1124: 8 total, 8 up, 8 in 2024-09-13T14:43:07.812 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:07 smithi151 ceph-mon[108966]: 6.9 scrub starts 2024-09-13T14:43:07.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:07.833 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:07.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:07.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:07 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/4253108856' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-13T14:43:07.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:07 smithi184 ceph-mon[92527]: osdmap e1124: 8 total, 8 up, 8 in 2024-09-13T14:43:07.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:07 smithi184 ceph-mon[92527]: 6.9 scrub starts 2024-09-13T14:43:07.934 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:07.956 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:07.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:07.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:08.058 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:08.080 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:08.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:08.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:08.182 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:08.204 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:08.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:08.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:08.228 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:43:08.306 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:08.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:08.330 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:08.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:08.432 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:08.455 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:08.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:08.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:08.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:08.558 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:08 smithi151 ceph-mon[108966]: pgmap v1441: 129 pgs: 18 peering, 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 103 active+clean; 1.2 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 773 op/s; 4378/56907 objects degraded (7.693%); 3741/56907 objects misplaced (6.574%); 665 KiB/s, 10 objects/s recovering 2024-09-13T14:43:08.558 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:08 smithi151 ceph-mon[108966]: Health check failed: Reduced data availability: 12 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-09-13T14:43:08.558 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:08 smithi151 ceph-mon[108966]: osdmap e1125: 8 total, 8 up, 8 in 2024-09-13T14:43:08.558 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:08 smithi151 ceph-mon[108966]: 1.3 scrub starts 2024-09-13T14:43:08.559 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:08 smithi151 ceph-mon[104450]: pgmap v1441: 129 pgs: 18 peering, 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 103 active+clean; 1.2 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 773 op/s; 4378/56907 objects degraded (7.693%); 3741/56907 objects misplaced (6.574%); 665 KiB/s, 10 objects/s recovering 2024-09-13T14:43:08.559 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:08 smithi151 ceph-mon[104450]: Health check failed: Reduced data availability: 12 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-09-13T14:43:08.559 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:08 smithi151 ceph-mon[104450]: osdmap e1125: 8 total, 8 up, 8 in 2024-09-13T14:43:08.559 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:08 smithi151 ceph-mon[104450]: 1.3 scrub starts 2024-09-13T14:43:08.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:08.609 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:08.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:08.710 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:08.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:08.804 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:08.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:08.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:08 smithi184 ceph-mon[92527]: pgmap v1441: 129 pgs: 18 peering, 1 active+recovery_wait+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 103 active+clean; 1.2 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 773 op/s; 4378/56907 objects degraded (7.693%); 3741/56907 objects misplaced (6.574%); 665 KiB/s, 10 objects/s recovering 2024-09-13T14:43:08.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:08 smithi184 ceph-mon[92527]: Health check failed: Reduced data availability: 12 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-09-13T14:43:08.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:08 smithi184 ceph-mon[92527]: osdmap e1125: 8 total, 8 up, 8 in 2024-09-13T14:43:08.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:08 smithi184 ceph-mon[92527]: 1.3 scrub starts 2024-09-13T14:43:08.903 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:43:08.903 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:43:08.905 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:08.917 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:08.935 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:08.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:08.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:09.037 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:09.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:09.070 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:09.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:09.171 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:09.209 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:09.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:09.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:09.311 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:09.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:09.343 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:09.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:09.444 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:09.482 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:09.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:09.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:09.584 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:09.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:09 smithi151 ceph-mon[104450]: osdmap e1126: 8 total, 8 up, 8 in 2024-09-13T14:43:09.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:09 smithi151 ceph-mon[104450]: 6.4 starting backfill to osd.1 from (1088'4255,1111'11066] MIN to 1123'13516 2024-09-13T14:43:09.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:09 smithi151 ceph-mon[104450]: 1.b scrub starts 2024-09-13T14:43:09.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:09 smithi151 ceph-mon[108966]: osdmap e1126: 8 total, 8 up, 8 in 2024-09-13T14:43:09.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:09 smithi151 ceph-mon[108966]: 6.4 starting backfill to osd.1 from (1088'4255,1111'11066] MIN to 1123'13516 2024-09-13T14:43:09.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:09 smithi151 ceph-mon[108966]: 1.b scrub starts 2024-09-13T14:43:09.614 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:09.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:09.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:09.717 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:09.749 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:09.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:09.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:09.850 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:09.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:09 smithi184 ceph-mon[92527]: osdmap e1126: 8 total, 8 up, 8 in 2024-09-13T14:43:09.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:09 smithi184 ceph-mon[92527]: 6.4 starting backfill to osd.1 from (1088'4255,1111'11066] MIN to 1123'13516 2024-09-13T14:43:09.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:09 smithi184 ceph-mon[92527]: 1.b scrub starts 2024-09-13T14:43:09.885 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:09.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:09.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:09.987 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:10.021 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:10.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:10.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:10.123 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:10.163 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:10.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:10.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:10.265 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:10.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:10.306 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:10.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:10.407 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:10.440 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:10.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:10.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:10.502 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:43:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:43:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:43:10.542 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:10.575 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:10.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:10.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:10.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:10 smithi151 ceph-mon[104450]: pgmap v1444: 129 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 31 peering, 1 active+recovery_wait+remapped, 2 active+remapped+backfill_wait, 93 active+clean; 994 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 335 op/s; 1040/47745 objects misplaced (2.178%); 842 KiB/s, 30 objects/s recovering 2024-09-13T14:43:10.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:10 smithi151 ceph-mon[104450]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4378/56907 objects degraded (7.693%), 1 pg degraded, 1 pg undersized) 2024-09-13T14:43:10.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:10 smithi151 ceph-mon[104450]: 1.13 scrub starts 2024-09-13T14:43:10.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:10 smithi151 ceph-mon[108966]: pgmap v1444: 129 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 31 peering, 1 active+recovery_wait+remapped, 2 active+remapped+backfill_wait, 93 active+clean; 994 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 335 op/s; 1040/47745 objects misplaced (2.178%); 842 KiB/s, 30 objects/s recovering 2024-09-13T14:43:10.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:10 smithi151 ceph-mon[108966]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4378/56907 objects degraded (7.693%), 1 pg degraded, 1 pg undersized) 2024-09-13T14:43:10.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:10 smithi151 ceph-mon[108966]: 1.13 scrub starts 2024-09-13T14:43:10.677 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:10.707 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:10.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:10.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:10.809 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:10.841 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:10.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:10.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:10.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:10 smithi184 ceph-mon[92527]: pgmap v1444: 129 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 31 peering, 1 active+recovery_wait+remapped, 2 active+remapped+backfill_wait, 93 active+clean; 994 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 335 op/s; 1040/47745 objects misplaced (2.178%); 842 KiB/s, 30 objects/s recovering 2024-09-13T14:43:10.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:10 smithi184 ceph-mon[92527]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4378/56907 objects degraded (7.693%), 1 pg degraded, 1 pg undersized) 2024-09-13T14:43:10.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:10 smithi184 ceph-mon[92527]: 1.13 scrub starts 2024-09-13T14:43:10.944 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:10.975 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:10.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:10.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:11.077 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:11.112 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:11.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:11.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:11.214 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:11.246 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:11.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:11.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:11.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:11.382 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:11.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:11.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:11.432 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T14:43:11.484 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:11.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:11.525 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:11.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:11.626 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:11.656 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:11.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:11.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:11.747 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:11 smithi151 ceph-mon[104450]: 1.1b scrub starts 2024-09-13T14:43:11.748 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:11 smithi151 ceph-mon[108966]: 1.1b scrub starts 2024-09-13T14:43:11.758 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:11.788 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:11.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:11.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:11.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:11 smithi184 ceph-mon[92527]: 1.1b scrub starts 2024-09-13T14:43:11.890 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:11.927 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:11.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:11.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:12.029 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:12.062 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:12.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:12.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:12.164 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:12.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:12.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:12.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:12.291 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:12.291 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:43:12.292 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:43:12.311 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:12.319 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:43:12.319 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:43:12.350 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:12.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:12.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:12.351 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:43:12.352 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:43:12.386 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:43:12.386 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:43:12.399 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:43:12.400 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:43:12.400 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-13T14:43:12.400 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 0.288180 2024-09-13T14:43:12.400 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 6 0.2881800167017552 2024-09-13T14:43:12.417 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:43:12.417 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:43:12.449 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:43:12.449 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:43:12.452 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:12.483 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:43:12.483 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:43:12.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:12.495 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:12.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:12.524 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:43:12.525 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:43:12.539 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-13T14:43:12.596 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:12.627 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:12.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:12.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:12.730 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:12.761 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:12.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:12.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:12.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:12 smithi184 ceph-mon[92527]: pgmap v1445: 129 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 31 peering, 1 active+recovery_wait+remapped, 2 active+remapped+backfill_wait, 93 active+clean; 981 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 206 op/s; 1040/47136 objects misplaced (2.206%); 277 KiB/s, 21 objects/s recovering 2024-09-13T14:43:12.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:12 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2501112037' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:43:12.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:12 smithi184 ceph-mon[92527]: 6.14 scrub starts 2024-09-13T14:43:12.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:12 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3183813847' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:12.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:12 smithi151 ceph-mon[104450]: pgmap v1445: 129 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 31 peering, 1 active+recovery_wait+remapped, 2 active+remapped+backfill_wait, 93 active+clean; 981 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 206 op/s; 1040/47136 objects misplaced (2.206%); 277 KiB/s, 21 objects/s recovering 2024-09-13T14:43:12.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:12 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2501112037' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:43:12.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:12 smithi151 ceph-mon[104450]: 6.14 scrub starts 2024-09-13T14:43:12.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:12 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3183813847' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:12.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:12 smithi151 ceph-mon[108966]: pgmap v1445: 129 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 31 peering, 1 active+recovery_wait+remapped, 2 active+remapped+backfill_wait, 93 active+clean; 981 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 206 op/s; 1040/47136 objects misplaced (2.206%); 277 KiB/s, 21 objects/s recovering 2024-09-13T14:43:12.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:12 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2501112037' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:43:12.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:12 smithi151 ceph-mon[108966]: 6.14 scrub starts 2024-09-13T14:43:12.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:12 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3183813847' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:12.863 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:12.906 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:12.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:12.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:13.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:13.042 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:13.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:13.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:13.144 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:13.174 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:13.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:13.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:13.276 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:13.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:13.303 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:13.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:13.404 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:13.439 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:13.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:13.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:13.541 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:13.580 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:13.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:13.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:13.682 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:13.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:13.710 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:13.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:13.811 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:13.839 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:13.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:13.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:13.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:13 smithi184 ceph-mon[92527]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:43:13.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:13 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2501112037' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:43:13.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:13 smithi184 ceph-mon[92527]: osdmap e1127: 8 total, 8 up, 8 in 2024-09-13T14:43:13.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:13 smithi184 ceph-mon[92527]: 6.9 scrub starts 2024-09-13T14:43:13.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:13 smithi184 ceph-mon[92527]: 6.9 scrub ok 2024-09-13T14:43:13.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:13 smithi184 ceph-mon[92527]: 6.1 scrub starts 2024-09-13T14:43:13.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:13 smithi184 ceph-mon[92527]: 6.1 scrub ok 2024-09-13T14:43:13.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:13 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-09-13T14:43:13.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:13 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-13T14:43:13.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:13 smithi184 ceph-mon[92527]: 1.f scrub starts 2024-09-13T14:43:13.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:13 smithi151 ceph-mon[104450]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:43:13.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:13 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2501112037' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:43:13.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:13 smithi151 ceph-mon[104450]: osdmap e1127: 8 total, 8 up, 8 in 2024-09-13T14:43:13.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:13 smithi151 ceph-mon[104450]: 6.9 scrub starts 2024-09-13T14:43:13.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:13 smithi151 ceph-mon[104450]: 6.9 scrub ok 2024-09-13T14:43:13.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:13 smithi151 ceph-mon[104450]: 6.1 scrub starts 2024-09-13T14:43:13.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:13 smithi151 ceph-mon[104450]: 6.1 scrub ok 2024-09-13T14:43:13.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:13 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-09-13T14:43:13.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:13 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-13T14:43:13.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:13 smithi151 ceph-mon[104450]: 1.f scrub starts 2024-09-13T14:43:13.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:13 smithi151 ceph-mon[108966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:43:13.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:13 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2501112037' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:43:13.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:13 smithi151 ceph-mon[108966]: osdmap e1127: 8 total, 8 up, 8 in 2024-09-13T14:43:13.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:13 smithi151 ceph-mon[108966]: 6.9 scrub starts 2024-09-13T14:43:13.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:13 smithi151 ceph-mon[108966]: 6.9 scrub ok 2024-09-13T14:43:13.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:13 smithi151 ceph-mon[108966]: 6.1 scrub starts 2024-09-13T14:43:13.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:13 smithi151 ceph-mon[108966]: 6.1 scrub ok 2024-09-13T14:43:13.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:13 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-09-13T14:43:13.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:13 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-13T14:43:13.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:13 smithi151 ceph-mon[108966]: 1.f scrub starts 2024-09-13T14:43:13.942 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:13.963 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:13.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:13.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:14.065 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:14.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:14.094 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:14.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:14.196 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:14.219 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:14.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:14.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:14.322 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:14.343 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:14.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:14.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:14.445 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:14.466 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:14.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:14.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:14.568 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:14.590 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:14.590 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:14 smithi151 ceph-mon[104450]: pgmap v1447: 129 pgs: 3 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 124 active+clean; 586 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.59k op/s; 246/28200 objects misplaced (0.872%); 277 KiB/s, 21 objects/s recovering 2024-09-13T14:43:14.590 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:14 smithi151 ceph-mon[104450]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 12 pgs inactive, 13 pgs peering) 2024-09-13T14:43:14.591 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:14 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-13T14:43:14.591 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:14 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-13T14:43:14.591 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:14 smithi151 ceph-mon[104450]: osdmap e1128: 8 total, 8 up, 8 in 2024-09-13T14:43:14.591 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:14 smithi151 ceph-mon[104450]: 6.11 scrub starts 2024-09-13T14:43:14.591 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:14 smithi151 ceph-mon[104450]: 6.11 scrub ok 2024-09-13T14:43:14.591 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:14 smithi151 ceph-mon[108966]: pgmap v1447: 129 pgs: 3 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 124 active+clean; 586 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.59k op/s; 246/28200 objects misplaced (0.872%); 277 KiB/s, 21 objects/s recovering 2024-09-13T14:43:14.591 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:14 smithi151 ceph-mon[108966]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 12 pgs inactive, 13 pgs peering) 2024-09-13T14:43:14.592 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:14 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-13T14:43:14.592 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:14 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-13T14:43:14.592 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:14 smithi151 ceph-mon[108966]: osdmap e1128: 8 total, 8 up, 8 in 2024-09-13T14:43:14.592 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:14 smithi151 ceph-mon[108966]: 6.11 scrub starts 2024-09-13T14:43:14.592 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:14 smithi151 ceph-mon[108966]: 6.11 scrub ok 2024-09-13T14:43:14.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:14.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:14.693 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:14.716 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:14.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:14.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:14.775 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:14 smithi184 ceph-mon[92527]: pgmap v1447: 129 pgs: 3 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 124 active+clean; 586 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.59k op/s; 246/28200 objects misplaced (0.872%); 277 KiB/s, 21 objects/s recovering 2024-09-13T14:43:14.775 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:14 smithi184 ceph-mon[92527]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 12 pgs inactive, 13 pgs peering) 2024-09-13T14:43:14.775 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:14 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-13T14:43:14.775 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:14 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-13T14:43:14.775 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:14 smithi184 ceph-mon[92527]: osdmap e1128: 8 total, 8 up, 8 in 2024-09-13T14:43:14.776 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:14 smithi184 ceph-mon[92527]: 6.11 scrub starts 2024-09-13T14:43:14.776 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:14 smithi184 ceph-mon[92527]: 6.11 scrub ok 2024-09-13T14:43:14.818 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:14.839 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:14.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:14.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:14.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:43:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:43:14] "GET /metrics HTTP/1.1" 200 36679 "" "Prometheus/2.51.0" 2024-09-13T14:43:14.941 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:14.965 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:14.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:14.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:15.067 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:15.090 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:15.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:15.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:15.192 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:15.213 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:15.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:15.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:15.315 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:15.339 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:15.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:15.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:15.407 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:15.441 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:15.463 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:15.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:15.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:15.565 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:15.586 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:15.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:15.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:15.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:15 smithi151 ceph-mon[104450]: osdmap e1129: 8 total, 8 up, 8 in 2024-09-13T14:43:15.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:15 smithi151 ceph-mon[104450]: 1.11 scrub starts 2024-09-13T14:43:15.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:15 smithi151 ceph-mon[104450]: 1.11 scrub ok 2024-09-13T14:43:15.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:15 smithi151 ceph-mon[104450]: 1.14 scrub starts 2024-09-13T14:43:15.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:15 smithi151 ceph-mon[104450]: 1.14 scrub ok 2024-09-13T14:43:15.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:15 smithi151 ceph-mon[104450]: 6.14 scrub starts 2024-09-13T14:43:15.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:15 smithi151 ceph-mon[104450]: 6.14 scrub ok 2024-09-13T14:43:15.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:15 smithi151 ceph-mon[108966]: osdmap e1129: 8 total, 8 up, 8 in 2024-09-13T14:43:15.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:15 smithi151 ceph-mon[108966]: 1.11 scrub starts 2024-09-13T14:43:15.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:15 smithi151 ceph-mon[108966]: 1.11 scrub ok 2024-09-13T14:43:15.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:15 smithi151 ceph-mon[108966]: 1.14 scrub starts 2024-09-13T14:43:15.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:15 smithi151 ceph-mon[108966]: 1.14 scrub ok 2024-09-13T14:43:15.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:15 smithi151 ceph-mon[108966]: 6.14 scrub starts 2024-09-13T14:43:15.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:15 smithi151 ceph-mon[108966]: 6.14 scrub ok 2024-09-13T14:43:15.688 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:15.711 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:15.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:15.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:15.813 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:15.848 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:15.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:15.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:15.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:15 smithi184 ceph-mon[92527]: osdmap e1129: 8 total, 8 up, 8 in 2024-09-13T14:43:15.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:15 smithi184 ceph-mon[92527]: 1.11 scrub starts 2024-09-13T14:43:15.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:15 smithi184 ceph-mon[92527]: 1.11 scrub ok 2024-09-13T14:43:15.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:15 smithi184 ceph-mon[92527]: 1.14 scrub starts 2024-09-13T14:43:15.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:15 smithi184 ceph-mon[92527]: 1.14 scrub ok 2024-09-13T14:43:15.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:15 smithi184 ceph-mon[92527]: 6.14 scrub starts 2024-09-13T14:43:15.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:15 smithi184 ceph-mon[92527]: 6.14 scrub ok 2024-09-13T14:43:15.950 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:15.985 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:15.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:15.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:16.087 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:16.118 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:16.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:16.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:16.220 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:16.251 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:16.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:16.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:16.353 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:16.389 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:16.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:16.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:16.454 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:43:16.490 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:16.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:16.545 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:16.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:16.646 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:16.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:16.674 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:16.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:16.735 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:43:16.735 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-13T14:43:16.735 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-13T14:43:16.735 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-13T14:43:16.735 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:43:16.748 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:16 smithi151 ceph-mon[104450]: pgmap v1450: 129 pgs: 1 active+recovering+undersized+remapped, 4 peering, 3 active+clean+scrubbing, 1 active+remapped+backfill_wait, 120 active+clean; 519 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 4 op/s; 246/24978 objects misplaced (0.985%); 8.7 KiB/s, 12 objects/s recovering 2024-09-13T14:43:16.749 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:16 smithi151 ceph-mon[104450]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-13T14:43:16.749 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:16 smithi151 ceph-mon[104450]: osdmap e1130: 8 total, 8 up, 8 in 2024-09-13T14:43:16.749 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:16 smithi151 ceph-mon[104450]: 6.10 starting backfill to osd.1 from (0'0,0'0] MAX to 1127'13470 2024-09-13T14:43:16.749 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:16 smithi151 ceph-mon[104450]: 1.19 scrub starts 2024-09-13T14:43:16.749 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:16 smithi151 ceph-mon[104450]: 1.19 scrub ok 2024-09-13T14:43:16.749 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:16 smithi151 ceph-mon[104450]: 1.3 scrub starts 2024-09-13T14:43:16.749 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:16 smithi151 ceph-mon[104450]: 1.3 scrub ok 2024-09-13T14:43:16.749 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:43:16.749 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:16.749 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:16 smithi151 ceph-mon[104450]: 1.7 scrub starts 2024-09-13T14:43:16.753 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:16 smithi151 ceph-mon[108966]: pgmap v1450: 129 pgs: 1 active+recovering+undersized+remapped, 4 peering, 3 active+clean+scrubbing, 1 active+remapped+backfill_wait, 120 active+clean; 519 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 4 op/s; 246/24978 objects misplaced (0.985%); 8.7 KiB/s, 12 objects/s recovering 2024-09-13T14:43:16.754 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:16 smithi151 ceph-mon[108966]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-13T14:43:16.754 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:16 smithi151 ceph-mon[108966]: osdmap e1130: 8 total, 8 up, 8 in 2024-09-13T14:43:16.754 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:16 smithi151 ceph-mon[108966]: 6.10 starting backfill to osd.1 from (0'0,0'0] MAX to 1127'13470 2024-09-13T14:43:16.754 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:16 smithi151 ceph-mon[108966]: 1.19 scrub starts 2024-09-13T14:43:16.754 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:16 smithi151 ceph-mon[108966]: 1.19 scrub ok 2024-09-13T14:43:16.754 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:16 smithi151 ceph-mon[108966]: 1.3 scrub starts 2024-09-13T14:43:16.754 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:16 smithi151 ceph-mon[108966]: 1.3 scrub ok 2024-09-13T14:43:16.754 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:43:16.754 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:16.754 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:16 smithi151 ceph-mon[108966]: 1.7 scrub starts 2024-09-13T14:43:16.775 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:16.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:16.820 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:16.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:16.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:16 smithi184 ceph-mon[92527]: pgmap v1450: 129 pgs: 1 active+recovering+undersized+remapped, 4 peering, 3 active+clean+scrubbing, 1 active+remapped+backfill_wait, 120 active+clean; 519 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 4 op/s; 246/24978 objects misplaced (0.985%); 8.7 KiB/s, 12 objects/s recovering 2024-09-13T14:43:16.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:16 smithi184 ceph-mon[92527]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-13T14:43:16.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:16 smithi184 ceph-mon[92527]: osdmap e1130: 8 total, 8 up, 8 in 2024-09-13T14:43:16.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:16 smithi184 ceph-mon[92527]: 6.10 starting backfill to osd.1 from (0'0,0'0] MAX to 1127'13470 2024-09-13T14:43:16.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:16 smithi184 ceph-mon[92527]: 1.19 scrub starts 2024-09-13T14:43:16.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:16 smithi184 ceph-mon[92527]: 1.19 scrub ok 2024-09-13T14:43:16.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:16 smithi184 ceph-mon[92527]: 1.3 scrub starts 2024-09-13T14:43:16.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:16 smithi184 ceph-mon[92527]: 1.3 scrub ok 2024-09-13T14:43:16.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:43:16.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:16.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:16 smithi184 ceph-mon[92527]: 1.7 scrub starts 2024-09-13T14:43:16.888 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:43:16.921 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:16.960 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:16.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:16.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:17.062 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:17.091 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:17.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:17.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:17.193 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:17.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:17.230 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:17.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:17.331 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:17.369 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:17.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:17.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:17.471 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:17.500 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:17.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:17.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:17.554 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T14:43:17.602 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:17.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:17.633 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:17.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:17.734 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:17.772 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:17.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:17.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:17.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:17 smithi184 ceph-mon[92527]: 1.c scrub starts 2024-09-13T14:43:17.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:17 smithi184 ceph-mon[92527]: 1.c scrub ok 2024-09-13T14:43:17.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:17 smithi184 ceph-mon[92527]: 1.16 scrub starts 2024-09-13T14:43:17.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:17 smithi184 ceph-mon[92527]: 1.16 scrub ok 2024-09-13T14:43:17.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:17 smithi151 ceph-mon[108966]: 1.c scrub starts 2024-09-13T14:43:17.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:17 smithi151 ceph-mon[108966]: 1.c scrub ok 2024-09-13T14:43:17.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:17 smithi151 ceph-mon[108966]: 1.16 scrub starts 2024-09-13T14:43:17.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:17 smithi151 ceph-mon[108966]: 1.16 scrub ok 2024-09-13T14:43:17.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:17 smithi151 ceph-mon[104450]: 1.c scrub starts 2024-09-13T14:43:17.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:17 smithi151 ceph-mon[104450]: 1.c scrub ok 2024-09-13T14:43:17.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:17 smithi151 ceph-mon[104450]: 1.16 scrub starts 2024-09-13T14:43:17.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:17 smithi151 ceph-mon[104450]: 1.16 scrub ok 2024-09-13T14:43:17.873 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:17.902 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:17.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:17.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:18.004 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:18.042 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:18.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:18.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:18.145 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:18.170 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:18.173 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:18.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:18.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:18.274 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:18.308 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:18.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:18.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:18.405 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:18.405 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:18.410 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:18.459 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:18.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:18.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:18.561 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:18.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:18.597 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:18.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:18.698 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:18.731 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:18.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:18.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:18.833 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:18.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:18 smithi184 ceph-mon[92527]: 6.2 scrub starts 2024-09-13T14:43:18.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:18 smithi184 ceph-mon[92527]: 6.2 scrub ok 2024-09-13T14:43:18.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:18 smithi184 ceph-mon[92527]: pgmap v1452: 129 pgs: 1 active+recovering+undersized+remapped, 4 peering, 1 active+clean+scrubbing, 1 active+remapped+backfill_wait, 122 active+clean; 480 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 5 op/s; 246/23118 objects misplaced (1.064%) 2024-09-13T14:43:18.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:18 smithi184 ceph-mon[92527]: 1.1d scrub starts 2024-09-13T14:43:18.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:18 smithi184 ceph-mon[92527]: 1.1d scrub ok 2024-09-13T14:43:18.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:18 smithi151 ceph-mon[104450]: 6.2 scrub starts 2024-09-13T14:43:18.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:18 smithi151 ceph-mon[104450]: 6.2 scrub ok 2024-09-13T14:43:18.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:18 smithi151 ceph-mon[104450]: pgmap v1452: 129 pgs: 1 active+recovering+undersized+remapped, 4 peering, 1 active+clean+scrubbing, 1 active+remapped+backfill_wait, 122 active+clean; 480 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 5 op/s; 246/23118 objects misplaced (1.064%) 2024-09-13T14:43:18.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:18 smithi151 ceph-mon[104450]: 1.1d scrub starts 2024-09-13T14:43:18.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:18 smithi151 ceph-mon[104450]: 1.1d scrub ok 2024-09-13T14:43:18.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:18 smithi151 ceph-mon[108966]: 6.2 scrub starts 2024-09-13T14:43:18.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:18 smithi151 ceph-mon[108966]: 6.2 scrub ok 2024-09-13T14:43:18.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:18 smithi151 ceph-mon[108966]: pgmap v1452: 129 pgs: 1 active+recovering+undersized+remapped, 4 peering, 1 active+clean+scrubbing, 1 active+remapped+backfill_wait, 122 active+clean; 480 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 5 op/s; 246/23118 objects misplaced (1.064%) 2024-09-13T14:43:18.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:18 smithi151 ceph-mon[108966]: 1.1d scrub starts 2024-09-13T14:43:18.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:18 smithi151 ceph-mon[108966]: 1.1d scrub ok 2024-09-13T14:43:18.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:18.860 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:18.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:18.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:19.002 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:19.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:19.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:19.104 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:19.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:19.144 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:19.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:19.246 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:19.279 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:19.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:19.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:19.381 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:19.418 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:19.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:19.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:19.520 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:19.548 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:19.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:19.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:19.650 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:19.687 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:19.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:19.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:19.790 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:19.821 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:19.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:19.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:19.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:19 smithi184 ceph-mon[92527]: 1.b scrub starts 2024-09-13T14:43:19.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:19 smithi184 ceph-mon[92527]: 1.b scrub ok 2024-09-13T14:43:19.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:19 smithi184 ceph-mon[92527]: 1.6 scrub starts 2024-09-13T14:43:19.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:19 smithi184 ceph-mon[92527]: 1.6 scrub ok 2024-09-13T14:43:19.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:19 smithi151 ceph-mon[104450]: 1.b scrub starts 2024-09-13T14:43:19.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:19 smithi151 ceph-mon[104450]: 1.b scrub ok 2024-09-13T14:43:19.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:19 smithi151 ceph-mon[104450]: 1.6 scrub starts 2024-09-13T14:43:19.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:19 smithi151 ceph-mon[104450]: 1.6 scrub ok 2024-09-13T14:43:19.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:19 smithi151 ceph-mon[108966]: 1.b scrub starts 2024-09-13T14:43:19.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:19 smithi151 ceph-mon[108966]: 1.b scrub ok 2024-09-13T14:43:19.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:19 smithi151 ceph-mon[108966]: 1.6 scrub starts 2024-09-13T14:43:19.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:19 smithi151 ceph-mon[108966]: 1.6 scrub ok 2024-09-13T14:43:19.923 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:19.955 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:19.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:19.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:20.057 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:20.093 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:20.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:20.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:20.194 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:20.219 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:20.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:20.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:20.321 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:20.358 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:20.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:20.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:20.460 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:20.495 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:20.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:20.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:20.575 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:43:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:43:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:43:20.597 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:20.635 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:20.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:20.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:20.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:20.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:20.771 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:20.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:20.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:20 smithi184 ceph-mon[92527]: 6.6 scrub starts 2024-09-13T14:43:20.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:20 smithi184 ceph-mon[92527]: pgmap v1453: 129 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 125 active+clean; 279 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 10 op/s; 1/13437 objects degraded (0.007%); 201/13437 objects misplaced (1.496%) 2024-09-13T14:43:20.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:20 smithi184 ceph-mon[92527]: Health check failed: Degraded data redundancy: 1/13437 objects degraded (0.007%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:43:20.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:20 smithi184 ceph-mon[92527]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-13T14:43:20.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:20 smithi184 ceph-mon[92527]: 1.13 scrub starts 2024-09-13T14:43:20.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:20 smithi184 ceph-mon[92527]: 1.13 scrub ok 2024-09-13T14:43:20.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:20 smithi184 ceph-mon[92527]: 1.1e scrub starts 2024-09-13T14:43:20.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:20 smithi151 ceph-mon[104450]: 6.6 scrub starts 2024-09-13T14:43:20.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:20 smithi151 ceph-mon[104450]: pgmap v1453: 129 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 125 active+clean; 279 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 10 op/s; 1/13437 objects degraded (0.007%); 201/13437 objects misplaced (1.496%) 2024-09-13T14:43:20.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:20 smithi151 ceph-mon[104450]: Health check failed: Degraded data redundancy: 1/13437 objects degraded (0.007%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:43:20.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:20 smithi151 ceph-mon[104450]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-13T14:43:20.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:20 smithi151 ceph-mon[104450]: 1.13 scrub starts 2024-09-13T14:43:20.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:20 smithi151 ceph-mon[104450]: 1.13 scrub ok 2024-09-13T14:43:20.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:20 smithi151 ceph-mon[104450]: 1.1e scrub starts 2024-09-13T14:43:20.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:20 smithi151 ceph-mon[108966]: 6.6 scrub starts 2024-09-13T14:43:20.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:20 smithi151 ceph-mon[108966]: pgmap v1453: 129 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 125 active+clean; 279 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 10 op/s; 1/13437 objects degraded (0.007%); 201/13437 objects misplaced (1.496%) 2024-09-13T14:43:20.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:20 smithi151 ceph-mon[108966]: Health check failed: Degraded data redundancy: 1/13437 objects degraded (0.007%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:43:20.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:20 smithi151 ceph-mon[108966]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-13T14:43:20.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:20 smithi151 ceph-mon[108966]: 1.13 scrub starts 2024-09-13T14:43:20.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:20 smithi151 ceph-mon[108966]: 1.13 scrub ok 2024-09-13T14:43:20.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:20 smithi151 ceph-mon[108966]: 1.1e scrub starts 2024-09-13T14:43:20.873 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:20.903 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:20.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:20.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:21.005 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:21.028 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:21.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:21.130 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:21.161 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:21.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:21.263 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:21.298 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:21.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:21.400 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:21.435 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:21.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:21.536 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:21.571 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:21.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:21.673 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:21.674 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Removed 132455 objects 2024-09-13T14:43:21.674 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Clean up completed and total clean up time :144.383 2024-09-13T14:43:21.696 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:21.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:21.798 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:21.820 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:21.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:21.857 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:21 smithi184 ceph-mon[92527]: 1.7 scrub starts 2024-09-13T14:43:21.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:21 smithi151 ceph-mon[104450]: 1.7 scrub starts 2024-09-13T14:43:21.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:21 smithi151 ceph-mon[108966]: 1.7 scrub starts 2024-09-13T14:43:21.923 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:21.944 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:21.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:22.046 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:22.069 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:22.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:22.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:22.171 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:22.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:22.198 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:22.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:22.299 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:22.325 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:22.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:22.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:22.427 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:22.453 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:22.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:22.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:22.555 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:22.580 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:22.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:22.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:22.683 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:22.709 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:22.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:22.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:22.814 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:22.840 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:22.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:22.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:22.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:22 smithi184 ceph-mon[92527]: pgmap v1454: 129 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 125 active+clean; 258 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 20 KiB/s rd, 8 op/s; 1/12435 objects degraded (0.008%); 201/12435 objects misplaced (1.616%) 2024-09-13T14:43:22.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:22 smithi151 ceph-mon[104450]: pgmap v1454: 129 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 125 active+clean; 258 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 20 KiB/s rd, 8 op/s; 1/12435 objects degraded (0.008%); 201/12435 objects misplaced (1.616%) 2024-09-13T14:43:22.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:22 smithi151 ceph-mon[108966]: pgmap v1454: 129 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 125 active+clean; 258 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 20 KiB/s rd, 8 op/s; 1/12435 objects degraded (0.008%); 201/12435 objects misplaced (1.616%) 2024-09-13T14:43:22.942 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:22.967 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:22.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:22.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:23.069 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:23.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:23.092 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:23.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:23.193 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:23.218 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:23.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:23.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:23.320 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:23.342 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:23.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:23.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:23.444 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:23.466 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:23.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:23.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:23.568 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:23.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:23.590 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:23.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:23.691 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:23.717 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:23.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:23.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:23.819 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:23.842 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:23.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:23.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:23.845 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:23 smithi151 ceph-mon[104450]: 6.12 scrub starts 2024-09-13T14:43:23.845 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:23 smithi151 ceph-mon[104450]: 6.12 scrub ok 2024-09-13T14:43:23.846 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:23 smithi151 ceph-mon[108966]: 6.12 scrub starts 2024-09-13T14:43:23.846 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:23 smithi151 ceph-mon[108966]: 6.12 scrub ok 2024-09-13T14:43:23.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:23 smithi184 ceph-mon[92527]: 6.12 scrub starts 2024-09-13T14:43:23.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:23 smithi184 ceph-mon[92527]: 6.12 scrub ok 2024-09-13T14:43:23.944 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:23.968 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:23.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:23.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:24.071 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:24.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:24.096 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:24.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:24.197 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:24.222 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:24.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:24.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:24.324 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:24.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:24.352 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:24.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:24.452 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:24.477 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:24.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:24.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:24.579 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:24.601 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:24.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:24.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:24.703 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:24.724 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:24.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:24.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:24.826 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:24.852 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:24.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:24.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:24.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:24 smithi184 ceph-mon[92527]: pgmap v1455: 129 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 125 active+clean; 53 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 1 op/s; 1/2598 objects degraded (0.038%); 92/2598 objects misplaced (3.541%) 2024-09-13T14:43:24.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:24 smithi151 ceph-mon[104450]: pgmap v1455: 129 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 125 active+clean; 53 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 1 op/s; 1/2598 objects degraded (0.038%); 92/2598 objects misplaced (3.541%) 2024-09-13T14:43:24.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:24 smithi151 ceph-mon[108966]: pgmap v1455: 129 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 125 active+clean; 53 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 1 op/s; 1/2598 objects degraded (0.038%); 92/2598 objects misplaced (3.541%) 2024-09-13T14:43:24.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:43:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:43:24] "GET /metrics HTTP/1.1" 200 36669 "" "Prometheus/2.51.0" 2024-09-13T14:43:24.954 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:24.980 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:24.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:24.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:25.082 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:25.107 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:25.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:25.174 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-09-13T14:43:25.174 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-09-13T14:43:25.209 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:25.233 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:25.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:25.335 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:25.361 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:25.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:25.463 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:25.488 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:25.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:25.590 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:25.613 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:25.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:25.715 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:25.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.737 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:25.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:25.838 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:25.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:25 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 1/2598 objects degraded (0.038%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:43:25.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:25 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 1/2598 objects degraded (0.038%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:43:25.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:25 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 1/2598 objects degraded (0.038%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:43:25.861 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:25.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:25.962 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:25.988 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:25.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:26.090 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:26.112 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:26.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:26.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:26.214 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:26.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:26.236 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:26.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:26.337 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:26.360 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:26.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:26.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:26.462 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:26.483 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:26.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:26.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:26.585 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:26.607 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:26.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:26.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:26.709 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:26.731 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:26.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:26.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:26.832 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:26.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:26 smithi184 ceph-mon[92527]: pgmap v1456: 129 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 125 active+clean; 14 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 1 op/s; 1/738 objects degraded (0.136%) 2024-09-13T14:43:26.854 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:26.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:26.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:26.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:26 smithi151 ceph-mon[104450]: pgmap v1456: 129 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 125 active+clean; 14 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 1 op/s; 1/738 objects degraded (0.136%) 2024-09-13T14:43:26.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:26 smithi151 ceph-mon[108966]: pgmap v1456: 129 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 125 active+clean; 14 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 1 op/s; 1/738 objects degraded (0.136%) 2024-09-13T14:43:26.956 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:26.978 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:26.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:26.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:27.080 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:27.101 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:27.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:27.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:27.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.228 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:27.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:27.329 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:27.351 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:27.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:27.454 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:27.475 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:27.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:27.577 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:27.598 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:27.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:27.700 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:27.722 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:27.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:27.823 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:27.845 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:27.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:27.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:27.969 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:27.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:28.071 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:28.092 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:28.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:28.194 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:28.220 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:28.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:28.322 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:28.344 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:28.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:28.446 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:28.469 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:28.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:28.571 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:28.592 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:28.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:28.694 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:28.716 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:28.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:28.817 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:28.843 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:28.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:28.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:28 smithi151 ceph-mon[104450]: 6.16 scrub starts 2024-09-13T14:43:28.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:28 smithi151 ceph-mon[104450]: pgmap v1457: 129 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 125 active+clean; 11 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 1 op/s; 1/561 objects degraded (0.178%) 2024-09-13T14:43:28.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:28 smithi151 ceph-mon[108966]: 6.16 scrub starts 2024-09-13T14:43:28.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:28 smithi151 ceph-mon[108966]: pgmap v1457: 129 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 125 active+clean; 11 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 1 op/s; 1/561 objects degraded (0.178%) 2024-09-13T14:43:28.945 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:28.970 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:28.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:29.071 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:29.093 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:29.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:29.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:29.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:28 smithi184 ceph-mon[92527]: 6.16 scrub starts 2024-09-13T14:43:29.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:28 smithi184 ceph-mon[92527]: pgmap v1457: 129 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 125 active+clean; 11 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 1 op/s; 1/561 objects degraded (0.178%) 2024-09-13T14:43:29.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:29.216 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:29.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:29.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:29.318 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:29.343 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:29.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:29.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:29.444 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:29.469 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:29.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:29.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:29.571 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:29.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:29.606 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:29.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:29.707 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:29.747 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:29.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:29.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:29.848 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:29.874 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:29.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:29.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:29.976 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:29.998 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:29.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:29.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:30.100 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:30.122 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:30.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:30.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:30.224 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:30.245 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:30.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:30.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:30.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:30.370 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:30.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:30.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:30.472 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:30.497 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:30.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:30.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:30.599 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:30.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:43:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:43:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:43:30.621 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:30.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:30.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:30.723 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:30.749 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:30.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:30.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:30.852 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:30.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:30 smithi151 ceph-mon[104450]: pgmap v1458: 129 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 125 active+clean; 11 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 1 op/s; 1/561 objects degraded (0.178%); 2.9 KiB/s, 83 objects/s recovering 2024-09-13T14:43:30.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:30 smithi151 ceph-mon[104450]: 1.1e scrub starts 2024-09-13T14:43:30.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:30 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 1/561 objects degraded (0.178%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:43:30.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:30 smithi151 ceph-mon[108966]: pgmap v1458: 129 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 125 active+clean; 11 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 1 op/s; 1/561 objects degraded (0.178%); 2.9 KiB/s, 83 objects/s recovering 2024-09-13T14:43:30.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:30 smithi151 ceph-mon[108966]: 1.1e scrub starts 2024-09-13T14:43:30.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:30 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 1/561 objects degraded (0.178%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:43:30.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:30.876 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:30.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:30.977 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:31.001 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:31.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:31.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:31.102 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:31.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:30 smithi184 ceph-mon[92527]: pgmap v1458: 129 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 125 active+clean; 11 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 1 op/s; 1/561 objects degraded (0.178%); 2.9 KiB/s, 83 objects/s recovering 2024-09-13T14:43:31.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:30 smithi184 ceph-mon[92527]: 1.1e scrub starts 2024-09-13T14:43:31.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:30 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 1/561 objects degraded (0.178%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:43:31.124 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:31.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:31.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:31.226 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:31.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:31.248 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:31.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:31.349 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:31.375 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:31.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:31.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:31.477 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:31.499 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:31.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:31.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:31.601 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:31.618 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.6 primary-affinity to 0.28818 (8188862) 2024-09-13T14:43:31.627 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:31.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:31.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:31.729 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:31.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:31.754 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:31.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:31.855 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:31.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:31 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2942590219' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 0.2881800167017552} : dispatch 2024-09-13T14:43:31.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:43:31.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:31.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:31 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2942590219' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 0.2881800167017552} : dispatch 2024-09-13T14:43:31.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:43:31.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:31.877 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:31.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:31.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:31.978 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:32.000 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:32.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:32.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:32.102 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:32.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:31 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2942590219' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 0.2881800167017552} : dispatch 2024-09-13T14:43:32.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:43:32.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:32.124 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:32.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:32.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:32.226 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:32.248 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:32.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:32.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:32.350 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:32.375 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:32.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:32.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:32.477 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:32.503 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:32.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:32.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:32.604 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:32.635 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:32.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:32.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:32.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:32.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:32.760 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:32.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:32.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:32 smithi151 ceph-mon[104450]: 1.5 scrub starts 2024-09-13T14:43:32.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:32 smithi151 ceph-mon[104450]: 1.5 scrub ok 2024-09-13T14:43:32.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:32 smithi151 ceph-mon[104450]: pgmap v1459: 129 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 125 active+clean; 11 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1/561 objects degraded (0.178%); 0 B/s, 63 objects/s recovering 2024-09-13T14:43:32.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:32 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2942590219' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.2881800167017552}]': finished 2024-09-13T14:43:32.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:32 smithi151 ceph-mon[104450]: osdmap e1131: 8 total, 8 up, 8 in 2024-09-13T14:43:32.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:32 smithi151 ceph-mon[108966]: 1.5 scrub starts 2024-09-13T14:43:32.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:32 smithi151 ceph-mon[108966]: 1.5 scrub ok 2024-09-13T14:43:32.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:32 smithi151 ceph-mon[108966]: pgmap v1459: 129 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 125 active+clean; 11 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1/561 objects degraded (0.178%); 0 B/s, 63 objects/s recovering 2024-09-13T14:43:32.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:32 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2942590219' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.2881800167017552}]': finished 2024-09-13T14:43:32.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:32 smithi151 ceph-mon[108966]: osdmap e1131: 8 total, 8 up, 8 in 2024-09-13T14:43:32.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:32.884 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:32.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:32.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:32.986 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:33.009 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:33.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:33.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:32 smithi184 ceph-mon[92527]: 1.5 scrub starts 2024-09-13T14:43:33.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:32 smithi184 ceph-mon[92527]: 1.5 scrub ok 2024-09-13T14:43:33.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:32 smithi184 ceph-mon[92527]: pgmap v1459: 129 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 125 active+clean; 11 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1/561 objects degraded (0.178%); 0 B/s, 63 objects/s recovering 2024-09-13T14:43:33.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:32 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2942590219' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.2881800167017552}]': finished 2024-09-13T14:43:33.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:32 smithi184 ceph-mon[92527]: osdmap e1131: 8 total, 8 up, 8 in 2024-09-13T14:43:33.111 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:33.133 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:33.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:33.235 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:33.261 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:33.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:33.363 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:33.387 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:33.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:33.489 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:33.513 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:33.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:33.614 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:33.638 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:33.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:33.740 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:33.763 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:33.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:33.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:33 smithi151 ceph-mon[104450]: osdmap e1132: 8 total, 8 up, 8 in 2024-09-13T14:43:33.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:33 smithi151 ceph-mon[108966]: osdmap e1132: 8 total, 8 up, 8 in 2024-09-13T14:43:33.866 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:33.889 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:33.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:33.991 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:34.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:34.016 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:34.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:34.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:33 smithi184 ceph-mon[92527]: osdmap e1132: 8 total, 8 up, 8 in 2024-09-13T14:43:34.117 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:34.141 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:34.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:34.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:34.243 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:34.267 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:34.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:34.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:34.369 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:34.380 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:34.392 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:34.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:34.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:34.494 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:34.517 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:34.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:34.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:34.619 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:34.641 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:34.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:34.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:34.743 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:34.766 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:34.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:34.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:34.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:34 smithi151 ceph-mon[104450]: 1.e scrub starts 2024-09-13T14:43:34.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:34 smithi151 ceph-mon[104450]: pgmap v1462: 129 pgs: 2 active, 4 peering, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 119 active+clean; 11 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 7 op/s; 1/561 objects degraded (0.178%); 0 B/s, 62 objects/s recovering 2024-09-13T14:43:34.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:34 smithi151 ceph-mon[104450]: 1.6 scrub starts 2024-09-13T14:43:34.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:34 smithi151 ceph-mon[104450]: 1.6 scrub ok 2024-09-13T14:43:34.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:34 smithi151 ceph-mon[104450]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-13T14:43:34.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:34 smithi151 ceph-mon[104450]: 1.1e scrub starts 2024-09-13T14:43:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:34 smithi151 ceph-mon[108966]: 1.e scrub starts 2024-09-13T14:43:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:34 smithi151 ceph-mon[108966]: pgmap v1462: 129 pgs: 2 active, 4 peering, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 119 active+clean; 11 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 7 op/s; 1/561 objects degraded (0.178%); 0 B/s, 62 objects/s recovering 2024-09-13T14:43:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:34 smithi151 ceph-mon[108966]: 1.6 scrub starts 2024-09-13T14:43:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:34 smithi151 ceph-mon[108966]: 1.6 scrub ok 2024-09-13T14:43:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:34 smithi151 ceph-mon[108966]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-13T14:43:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:34 smithi151 ceph-mon[108966]: 1.1e scrub starts 2024-09-13T14:43:34.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:43:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:43:34] "GET /metrics HTTP/1.1" 200 36659 "" "Prometheus/2.51.0" 2024-09-13T14:43:34.869 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:34.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:34.893 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:34.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:34.994 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:35.016 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:35.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:35.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:35.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:34 smithi184 ceph-mon[92527]: 1.e scrub starts 2024-09-13T14:43:35.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:34 smithi184 ceph-mon[92527]: pgmap v1462: 129 pgs: 2 active, 4 peering, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 119 active+clean; 11 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 7 op/s; 1/561 objects degraded (0.178%); 0 B/s, 62 objects/s recovering 2024-09-13T14:43:35.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:34 smithi184 ceph-mon[92527]: 1.6 scrub starts 2024-09-13T14:43:35.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:34 smithi184 ceph-mon[92527]: 1.6 scrub ok 2024-09-13T14:43:35.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:34 smithi184 ceph-mon[92527]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-13T14:43:35.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:34 smithi184 ceph-mon[92527]: 1.1e scrub starts 2024-09-13T14:43:35.117 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:35.141 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:35.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:35.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:35.242 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:35.266 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:35.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:35.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:35.368 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:35.392 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:35.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:35.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:35.493 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:35.518 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:35.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:35.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:35.620 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:35.647 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:35.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:35.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:35.749 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:35.775 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:35.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:35.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:35.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:35.900 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:35.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:35.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:36.002 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:36.028 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:36.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:36.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:36.130 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:36.155 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:36.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:36.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:36.257 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:36.281 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:36.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:36.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:36.382 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:36.406 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:36.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:36.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:36.508 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:36.531 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:36.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:36.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:36.584 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:43:36.584 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:43:36.599 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:43:36.599 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:43:36.616 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:43:36.616 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:43:36.633 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:36.637 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-13T14:43:36.643 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:43:36.643 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:43:36.660 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:36.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:36.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:36.664 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:43:36.665 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:43:36.683 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:43:36.683 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:43:36.704 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:43:36.704 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:43:36.721 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:43:36.721 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:43:36.722 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:36 smithi151 ceph-mon[104450]: pgmap v1463: 129 pgs: 1 active+recovering+undersized+remapped, 2 active, 4 peering, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 119 active+clean; 11 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 1/561 objects degraded (0.178%); 0 B/s, 135 objects/s recovering 2024-09-13T14:43:36.722 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:36 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/698939356' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:36.722 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:36 smithi151 ceph-mon[104450]: 1.1e scrub starts 2024-09-13T14:43:36.722 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:36 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3444468721' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:36.722 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:36 smithi151 ceph-mon[108966]: pgmap v1463: 129 pgs: 1 active+recovering+undersized+remapped, 2 active, 4 peering, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 119 active+clean; 11 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 1/561 objects degraded (0.178%); 0 B/s, 135 objects/s recovering 2024-09-13T14:43:36.722 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:36 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/698939356' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:36.722 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:36 smithi151 ceph-mon[108966]: 1.1e scrub starts 2024-09-13T14:43:36.722 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:36 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3444468721' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:36.762 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:36.787 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:36.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:36.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:36.820 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:43:36.821 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:43:36.889 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:36.915 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:36.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:36.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:37.017 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:37.039 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:37.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:37.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:37.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:36 smithi184 ceph-mon[92527]: pgmap v1463: 129 pgs: 1 active+recovering+undersized+remapped, 2 active, 4 peering, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 119 active+clean; 11 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 1/561 objects degraded (0.178%); 0 B/s, 135 objects/s recovering 2024-09-13T14:43:37.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:36 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/698939356' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:37.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:36 smithi184 ceph-mon[92527]: 1.1e scrub starts 2024-09-13T14:43:37.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:36 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3444468721' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:37.141 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:37.163 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:37.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:37.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:37.265 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:37.290 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:37.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:37.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:37.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:37.414 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:37.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:37.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:37.516 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:37.539 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:37.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:37.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:37.641 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:37.663 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:37.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:37.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:37.765 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:37.791 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:37.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:37.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:37.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:37 smithi151 ceph-mon[104450]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:43:37.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:37 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/698939356' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:43:37.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:37 smithi151 ceph-mon[104450]: osdmap e1133: 8 total, 8 up, 8 in 2024-09-13T14:43:37.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:37 smithi151 ceph-mon[108966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:43:37.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:37 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/698939356' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:43:37.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:37 smithi151 ceph-mon[108966]: osdmap e1133: 8 total, 8 up, 8 in 2024-09-13T14:43:37.892 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:37.917 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:37.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:37.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:38.019 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:38.042 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:38.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:38.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:38.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:37 smithi184 ceph-mon[92527]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:43:38.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:37 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/698939356' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:43:38.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:37 smithi184 ceph-mon[92527]: osdmap e1133: 8 total, 8 up, 8 in 2024-09-13T14:43:38.144 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:38.170 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:38.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:38.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:38.272 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:38.297 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:38.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:38.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:38.399 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:38.421 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:38.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:38.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:38.523 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:38.546 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:38.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:38.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:38.648 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:38.671 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:38.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:38.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:38.774 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:38.797 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:38.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:38.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:38.899 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:38.924 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:38.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:38.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:38.982 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:43:38.982 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:43:39.026 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:39.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:39.052 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:39.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:39.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:38 smithi184 ceph-mon[92527]: pgmap v1465: 129 pgs: 1 active+recovering+undersized+remapped, 2 active, 4 peering, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 119 active+clean; 11 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 1/561 objects degraded (0.178%); 0 B/s, 97 objects/s recovering 2024-09-13T14:43:39.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:38 smithi151 ceph-mon[104450]: pgmap v1465: 129 pgs: 1 active+recovering+undersized+remapped, 2 active, 4 peering, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 119 active+clean; 11 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 1/561 objects degraded (0.178%); 0 B/s, 97 objects/s recovering 2024-09-13T14:43:39.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:38 smithi151 ceph-mon[108966]: pgmap v1465: 129 pgs: 1 active+recovering+undersized+remapped, 2 active, 4 peering, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 119 active+clean; 11 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 1/561 objects degraded (0.178%); 0 B/s, 97 objects/s recovering 2024-09-13T14:43:39.149 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T14:43:39.153 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:39.213 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:39.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:39.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:39.315 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:39.341 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:39.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:39.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:39.442 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:39.469 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:39.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:39.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:39.571 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:39.596 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:39.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:39.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:39.698 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:39.721 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:39.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:39.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:39.823 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:39.845 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:39.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:39.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:39.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:39.969 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:40.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:40.030 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:39 smithi151 ceph-mon[104450]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-09-13T14:43:40.030 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:39 smithi151 ceph-mon[108966]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-09-13T14:43:40.060 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:43:40.103 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:40.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:39 smithi184 ceph-mon[92527]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-09-13T14:43:40.128 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:40.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:40.230 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:40.255 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:40.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:40.284 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:40.358 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:40.383 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:40.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:40.485 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:40.511 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:40.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:40.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:43:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:43:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:43:40.613 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:40.635 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:40.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:40.699 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_1' removed 2024-09-13T14:43:40.738 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:40.761 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:40.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:40.863 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:40.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.886 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:40.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:40.987 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:41.014 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:41.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:41.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:41.050 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:40 smithi151 ceph-mon[104450]: pgmap v1466: 129 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 125 active+clean; 11 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 1/561 objects degraded (0.178%); 0 B/s, 191 objects/s recovering 2024-09-13T14:43:41.050 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:40 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2559887612' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:43:41.050 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:40 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:43:41.051 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:40 smithi151 ceph-mon[108966]: pgmap v1466: 129 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 125 active+clean; 11 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 1/561 objects degraded (0.178%); 0 B/s, 191 objects/s recovering 2024-09-13T14:43:41.051 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:40 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2559887612' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:43:41.051 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:40 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:43:41.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:40 smithi184 ceph-mon[92527]: pgmap v1466: 129 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 125 active+clean; 11 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 1/561 objects degraded (0.178%); 0 B/s, 191 objects/s recovering 2024-09-13T14:43:41.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:40 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2559887612' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:43:41.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:40 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:43:41.116 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:41.137 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:41.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:41.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:41.239 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:41.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:41.265 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:41.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:41.281 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:41.365 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:41.387 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:41.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:41.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:41.489 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:41.511 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:41.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:41.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:41.612 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:41.633 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:41.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:41.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:41.696 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:43:41.735 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:41.756 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:41.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:41.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:41.858 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:41.880 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:41.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:41.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:41.982 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:42.003 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:42.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:42.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:42.091 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:41 smithi151 ceph-mon[104450]: 1.1e scrub starts 2024-09-13T14:43:42.092 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:41 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:43:42.092 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:41 smithi151 ceph-mon[104450]: osdmap e1134: 8 total, 8 up, 8 in 2024-09-13T14:43:42.092 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:41 smithi151 ceph-mon[108966]: 1.1e scrub starts 2024-09-13T14:43:42.092 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:41 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:43:42.092 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:41 smithi151 ceph-mon[108966]: osdmap e1134: 8 total, 8 up, 8 in 2024-09-13T14:43:42.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:41 smithi184 ceph-mon[92527]: 1.1e scrub starts 2024-09-13T14:43:42.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:41 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:43:42.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:41 smithi184 ceph-mon[92527]: osdmap e1134: 8 total, 8 up, 8 in 2024-09-13T14:43:42.105 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:42.117 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T14:43:42.121 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T14:43:42.122 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-09-13T14:43:42.122 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_2 16 2024-09-13T14:43:42.129 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:42.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:42.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:42.232 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:42.254 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:42.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:42.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:42.356 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:42.380 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:42.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:42.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:42.482 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:42.504 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:42.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:42.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:42.586 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:43:42.587 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:43:42.606 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:42.630 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:42.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:42.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:42.732 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:42.754 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:42.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:42.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:42.856 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:42.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:42.878 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:42.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:42.965 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:42 smithi151 ceph-mon[108966]: pgmap v1468: 103 pgs: 1 active+recovery_wait+degraded, 102 active+clean; 672 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1/78 objects degraded (1.282%) 2024-09-13T14:43:42.966 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:42 smithi151 ceph-mon[108966]: 1.1e deep-scrub starts 2024-09-13T14:43:42.966 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:42 smithi151 ceph-mon[108966]: 1.1e deep-scrub ok 2024-09-13T14:43:42.966 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:42 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 1/78 objects degraded (1.282%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:43:42.966 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:42 smithi151 ceph-mon[108966]: 1.1a scrub starts 2024-09-13T14:43:42.966 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:42 smithi151 ceph-mon[108966]: 1.1a scrub ok 2024-09-13T14:43:42.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:42 smithi151 ceph-mon[104450]: pgmap v1468: 103 pgs: 1 active+recovery_wait+degraded, 102 active+clean; 672 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1/78 objects degraded (1.282%) 2024-09-13T14:43:42.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:42 smithi151 ceph-mon[104450]: 1.1e deep-scrub starts 2024-09-13T14:43:42.967 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:42 smithi151 ceph-mon[104450]: 1.1e deep-scrub ok 2024-09-13T14:43:42.967 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:42 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 1/78 objects degraded (1.282%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:43:42.967 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:42 smithi151 ceph-mon[104450]: 1.1a scrub starts 2024-09-13T14:43:42.967 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:42 smithi151 ceph-mon[104450]: 1.1a scrub ok 2024-09-13T14:43:42.979 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:43.000 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:43.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:43.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:43.102 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:43.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:42 smithi184 ceph-mon[92527]: pgmap v1468: 103 pgs: 1 active+recovery_wait+degraded, 102 active+clean; 672 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1/78 objects degraded (1.282%) 2024-09-13T14:43:43.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:42 smithi184 ceph-mon[92527]: 1.1e deep-scrub starts 2024-09-13T14:43:43.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:42 smithi184 ceph-mon[92527]: 1.1e deep-scrub ok 2024-09-13T14:43:43.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:42 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 1/78 objects degraded (1.282%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:43:43.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:42 smithi184 ceph-mon[92527]: 1.1a scrub starts 2024-09-13T14:43:43.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:42 smithi184 ceph-mon[92527]: 1.1a scrub ok 2024-09-13T14:43:43.123 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:43.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:43.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:43.225 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:43.251 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:43.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:43.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:43.353 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:43.379 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:43.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:43.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:43.481 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:43.506 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:43.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:43.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:43.608 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:43.633 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:43.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:43.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:43.735 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:43.761 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:43.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:43.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:43.863 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:43.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:43.887 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:43.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:43.988 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:44.005 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:43 smithi184 ceph-mon[92527]: 1.2 scrub starts 2024-09-13T14:43:44.005 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:43 smithi184 ceph-mon[92527]: 1.2 scrub ok 2024-09-13T14:43:44.005 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:43 smithi184 ceph-mon[92527]: 1.e scrub starts 2024-09-13T14:43:44.005 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:43 smithi184 ceph-mon[92527]: 1.e scrub ok 2024-09-13T14:43:44.010 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:44.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:44.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:44.089 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:43 smithi151 ceph-mon[108966]: 1.2 scrub starts 2024-09-13T14:43:44.089 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:43 smithi151 ceph-mon[108966]: 1.2 scrub ok 2024-09-13T14:43:44.089 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:43 smithi151 ceph-mon[108966]: 1.e scrub starts 2024-09-13T14:43:44.089 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:43 smithi151 ceph-mon[108966]: 1.e scrub ok 2024-09-13T14:43:44.090 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:43 smithi151 ceph-mon[104450]: 1.2 scrub starts 2024-09-13T14:43:44.090 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:43 smithi151 ceph-mon[104450]: 1.2 scrub ok 2024-09-13T14:43:44.090 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:43 smithi151 ceph-mon[104450]: 1.e scrub starts 2024-09-13T14:43:44.090 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:43 smithi151 ceph-mon[104450]: 1.e scrub ok 2024-09-13T14:43:44.113 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:44.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:44.141 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:44.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:44.160 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:44.160 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:44.242 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:44.267 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:44.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:44.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:44.369 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:44.394 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:44.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:44.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:44.495 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:44.520 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:44.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:44.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:44.623 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:44.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:44.646 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:44.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:44.712 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:43:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:43:44] "GET /metrics HTTP/1.1" 200 36659 "" "Prometheus/2.51.0" 2024-09-13T14:43:44.747 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:44.768 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:44.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:44.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:44.870 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:44.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:44.897 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:44.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:44.945 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T14:43:44.967 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:44 smithi151 ceph-mon[104450]: pgmap v1469: 103 pgs: 103 active+clean; 646 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:43:44.967 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:44 smithi151 ceph-mon[104450]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/78 objects degraded (1.282%), 1 pg degraded) 2024-09-13T14:43:44.967 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:44 smithi151 ceph-mon[104450]: Cluster is now healthy 2024-09-13T14:43:44.968 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:44 smithi151 ceph-mon[108966]: pgmap v1469: 103 pgs: 103 active+clean; 646 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:43:44.968 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:44 smithi151 ceph-mon[108966]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/78 objects degraded (1.282%), 1 pg degraded) 2024-09-13T14:43:44.968 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:44 smithi151 ceph-mon[108966]: Cluster is now healthy 2024-09-13T14:43:44.998 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:45.019 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:45.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:45.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:44 smithi184 ceph-mon[92527]: pgmap v1469: 103 pgs: 103 active+clean; 646 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:43:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:44 smithi184 ceph-mon[92527]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/78 objects degraded (1.282%), 1 pg degraded) 2024-09-13T14:43:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:44 smithi184 ceph-mon[92527]: Cluster is now healthy 2024-09-13T14:43:45.121 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:45.143 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:45.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:45.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:45.245 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:45.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:45.267 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:45.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:45.368 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:45.389 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:45.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:45.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:45.494 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:45.516 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:45.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:45.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:45.618 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:45.643 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:45.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:45.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:45.745 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:45.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:45.767 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:45.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:45.772 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-13T14:43:45.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:45 smithi151 ceph-mon[104450]: 1.1 scrub starts 2024-09-13T14:43:45.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:45 smithi151 ceph-mon[104450]: 1.1 scrub ok 2024-09-13T14:43:45.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:45 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/45288618' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:43:45.853 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:45 smithi151 ceph-mon[108966]: 1.1 scrub starts 2024-09-13T14:43:45.853 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:45 smithi151 ceph-mon[108966]: 1.1 scrub ok 2024-09-13T14:43:45.853 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:45 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/45288618' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:43:45.868 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:45.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:45.894 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:45.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:45.995 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:46.017 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:46.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:46.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:46.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:45 smithi184 ceph-mon[92527]: 1.1 scrub starts 2024-09-13T14:43:46.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:45 smithi184 ceph-mon[92527]: 1.1 scrub ok 2024-09-13T14:43:46.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:45 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/45288618' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:43:46.119 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:46.145 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:46.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:46.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:46.247 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:46.273 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:46.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:46.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:46.375 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:46.397 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:46.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:46.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:46.499 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:46.522 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:46.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:46.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:46.624 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:46.650 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:46.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:46.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:46.752 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:46.774 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:46.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:46.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:46 smithi151 ceph-mon[104450]: pgmap v1470: 103 pgs: 103 active+clean; 646 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:43:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:46 smithi151 ceph-mon[104450]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:43:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:46 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/45288618' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:43:46.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:46 smithi151 ceph-mon[104450]: osdmap e1135: 8 total, 8 up, 8 in 2024-09-13T14:43:46.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:43:46.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:46 smithi151 ceph-mon[108966]: pgmap v1470: 103 pgs: 103 active+clean; 646 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:43:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:46 smithi151 ceph-mon[108966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:43:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:46 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/45288618' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:43:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:46 smithi151 ceph-mon[108966]: osdmap e1135: 8 total, 8 up, 8 in 2024-09-13T14:43:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:43:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:46.876 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:46.902 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:46.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:46.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:47.004 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:47.025 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:47.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:47.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:47.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:46 smithi184 ceph-mon[92527]: pgmap v1470: 103 pgs: 103 active+clean; 646 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:43:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:46 smithi184 ceph-mon[92527]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:43:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:46 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/45288618' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:43:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:46 smithi184 ceph-mon[92527]: osdmap e1135: 8 total, 8 up, 8 in 2024-09-13T14:43:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:43:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:47.128 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:47.150 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:47.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:47.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:47.253 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:47.278 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:47.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:47.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:47.379 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:47.404 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:47.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:47.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:47.506 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:47.532 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:47.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:47.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:47.634 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:47.657 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:47.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:47.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:47.759 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:47.783 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:47.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:47.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:47.885 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:47.886 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:47.910 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:47.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:47.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:48.012 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:48.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:48.034 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:48.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:48.135 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:48.159 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:48.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:48.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:48.261 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:48.282 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:48.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:48.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:48.384 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:48.406 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:48.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:48.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:48.507 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:48.532 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:48.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:48.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:48.634 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:48.657 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:48.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:48.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:48.759 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:48.783 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:48.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:48.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:48.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:48 smithi151 ceph-mon[104450]: pgmap v1472: 103 pgs: 103 active+clean; 646 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:43:48.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:48 smithi151 ceph-mon[108966]: pgmap v1472: 103 pgs: 103 active+clean; 646 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:43:48.885 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:48.906 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:48.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:48.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:49.009 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:49.035 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:49.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:49.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:49.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:48 smithi184 ceph-mon[92527]: pgmap v1472: 103 pgs: 103 active+clean; 646 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:43:49.137 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:49.158 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:49.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:49.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:49.260 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:49.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:49.283 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:49.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:49.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:49.406 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:49.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:49.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:49.508 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:49.535 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:49.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:49.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:49.539 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:43:49.611 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:43:49.612 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:43:49.629 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:43:49.629 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:43:49.636 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:49.645 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:43:49.645 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:43:49.661 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:49.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:49.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:49.673 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:43:49.674 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:43:49.689 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:43:49.690 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:43:49.705 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:43:49.706 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:43:49.721 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:43:49.721 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:43:49.738 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:43:49.738 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:43:49.763 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:49.786 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_2' created 2024-09-13T14:43:49.788 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:49.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:49.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:49.892 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:49.914 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:49.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:49.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:50.016 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:50.037 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:50.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:50.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:50.048 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:49 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/650010750' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-13T14:43:50.048 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:49 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-13T14:43:50.049 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:49 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2126036545' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:50.049 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:49 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/650010750' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-13T14:43:50.049 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:49 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-13T14:43:50.049 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:49 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2126036545' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:50.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:49 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/650010750' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-13T14:43:50.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:49 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-13T14:43:50.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:49 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2126036545' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:50.139 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:50.165 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:50.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:50.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:50.267 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:50.295 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:50.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:50.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:50.397 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:50.423 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:50.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:50.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:50.525 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:50.547 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:50.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:50.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:50.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:43:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:43:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:43:50.649 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:50.674 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:50.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:50.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:50.776 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:50.807 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:50.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:50.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:50.909 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:50.930 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:50.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:50.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:51.032 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:51.057 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:51.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:51.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:51.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:50 smithi184 ceph-mon[92527]: pgmap v1473: 103 pgs: 103 active+clean; 633 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s, 0 objects/s recovering 2024-09-13T14:43:51.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:50 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-13T14:43:51.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:50 smithi184 ceph-mon[92527]: osdmap e1136: 8 total, 8 up, 8 in 2024-09-13T14:43:51.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:50 smithi151 ceph-mon[104450]: pgmap v1473: 103 pgs: 103 active+clean; 633 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s, 0 objects/s recovering 2024-09-13T14:43:51.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:50 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-13T14:43:51.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:50 smithi151 ceph-mon[104450]: osdmap e1136: 8 total, 8 up, 8 in 2024-09-13T14:43:51.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:50 smithi151 ceph-mon[108966]: pgmap v1473: 103 pgs: 103 active+clean; 633 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s, 0 objects/s recovering 2024-09-13T14:43:51.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:50 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-13T14:43:51.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:50 smithi151 ceph-mon[108966]: osdmap e1136: 8 total, 8 up, 8 in 2024-09-13T14:43:51.159 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:51.184 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:51.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:51.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:51.286 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:51.311 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:51.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:51.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:51.413 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:51.439 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:51.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:51.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:51.541 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:51.564 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:51.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:51.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:51.666 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:51.688 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:51.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:51.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:51.790 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:51.815 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:51.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:51.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:51.917 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:51.938 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:51.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:51.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:52.040 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:52.061 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:51 smithi151 ceph-mon[104450]: osdmap e1137: 8 total, 8 up, 8 in 2024-09-13T14:43:52.061 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:51 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:43:52.061 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:51 smithi151 ceph-mon[108966]: osdmap e1137: 8 total, 8 up, 8 in 2024-09-13T14:43:52.061 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:51 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:43:52.062 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:52.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:52.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:52.083 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-09-13T14:43:52.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:51 smithi184 ceph-mon[92527]: osdmap e1137: 8 total, 8 up, 8 in 2024-09-13T14:43:52.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:51 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:43:52.164 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:52.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:52.189 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:52.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:52.290 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:52.312 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:52.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:52.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:52.362 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:43:52.362 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-13T14:43:52.362 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-13T14:43:52.362 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-13T14:43:52.362 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:43:52.414 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:52.440 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:52.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:52.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:52.542 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:52.563 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:52.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:52.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:52.648 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:52.665 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:52.686 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:52.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:52.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:52.788 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:52.814 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:52.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:52.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:52.916 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:52.941 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:52.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:52.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:53.033 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:52 smithi151 ceph-mon[104450]: pgmap v1476: 119 pgs: 4 creating+peering, 12 unknown, 103 active+clean; 633 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:43:53.034 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:52 smithi151 ceph-mon[104450]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:43:53.034 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:52 smithi151 ceph-mon[104450]: osdmap e1138: 8 total, 8 up, 8 in 2024-09-13T14:43:53.034 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:52 smithi151 ceph-mon[108966]: pgmap v1476: 119 pgs: 4 creating+peering, 12 unknown, 103 active+clean; 633 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:43:53.034 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:52 smithi151 ceph-mon[108966]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:43:53.034 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:52 smithi151 ceph-mon[108966]: osdmap e1138: 8 total, 8 up, 8 in 2024-09-13T14:43:53.043 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:53.066 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T14:43:53.067 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:53.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:53.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:53.072 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:53.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:52 smithi184 ceph-mon[92527]: pgmap v1476: 119 pgs: 4 creating+peering, 12 unknown, 103 active+clean; 633 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:43:53.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:52 smithi184 ceph-mon[92527]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:43:53.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:52 smithi184 ceph-mon[92527]: osdmap e1138: 8 total, 8 up, 8 in 2024-09-13T14:43:53.169 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:53.191 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:53.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:53.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:53.292 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:53.318 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:53.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:53.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:53.407 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:43:53.420 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:53.472 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:53.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:53.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:53.574 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:53.599 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:53.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:53.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:53.702 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:53.724 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:53.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:53.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:53.827 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:53.852 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:53.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:53.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:53.954 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:53.979 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:53.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:53.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:54.081 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:54.102 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:54.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:54.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:54.204 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:54.229 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:54.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:54.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:54.331 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:54.353 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:54.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:54.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:54.455 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:54.480 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:54.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:54.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:54.582 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:54.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:54.606 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:54.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:54.707 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:54.733 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:54.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:54.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:54.835 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:54.857 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:54.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:54.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:54 smithi151 ceph-mon[104450]: pgmap v1478: 119 pgs: 4 creating+peering, 5 unknown, 110 active+clean; 633 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:43:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:54 smithi151 ceph-mon[108966]: pgmap v1478: 119 pgs: 4 creating+peering, 5 unknown, 110 active+clean; 633 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:43:54.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:43:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:43:54] "GET /metrics HTTP/1.1" 200 34313 "" "Prometheus/2.51.0" 2024-09-13T14:43:54.959 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:54.984 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:54.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:54.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:55.086 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:55.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:54 smithi184 ceph-mon[92527]: pgmap v1478: 119 pgs: 4 creating+peering, 5 unknown, 110 active+clean; 633 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:43:55.109 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:55.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:55.127 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:55.211 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:55.236 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:55.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:55.338 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:55.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.361 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:55.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:55.462 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:55.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.485 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:55.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:55.586 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:55.590 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:55.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.615 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:55.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:55.716 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:55.741 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:55.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:55.843 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:55.865 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:55.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:55.966 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:55.992 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:55.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:56.094 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:56.115 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:56.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:56.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:56.217 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:56.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:56.239 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:56.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:56.341 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:56.366 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:56.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:56.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:56.468 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:56.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:56.491 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:56.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:56.593 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:56.618 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:56.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:56.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:56.720 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:56.746 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:56.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:56.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:56.848 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:56.869 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:56.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:56.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:56.971 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:56.997 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:56.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:56.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:57.099 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:57.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:56 smithi184 ceph-mon[92527]: pgmap v1479: 119 pgs: 4 creating+peering, 115 active+clean; 633 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:43:57.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:56 smithi151 ceph-mon[104450]: pgmap v1479: 119 pgs: 4 creating+peering, 115 active+clean; 633 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:43:57.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:56 smithi151 ceph-mon[108966]: pgmap v1479: 119 pgs: 4 creating+peering, 115 active+clean; 633 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:43:57.124 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:57.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:57.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:57.226 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:57.251 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:57.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:57.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:57.353 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:57.377 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:57.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:57.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:57.479 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:57.503 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:57.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:57.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:57.605 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:57.631 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:57.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:57.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:57.733 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:57.758 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:57.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:57.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:57.860 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:57.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:57.884 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:57.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:57.985 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:58.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:58.011 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:58.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:58.112 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:58.136 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:58.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:58.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:58.238 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:58.261 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:58.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:58.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:58.362 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:58.384 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:58.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:58.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:58.485 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:58.507 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:58.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:58.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:58.609 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:58.635 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:58.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:58.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:58.737 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:58.762 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:58.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:58.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:58.806 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-13T14:43:58.810 INFO:teuthology.orchestra.run.smithi151.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-09-13T14:43:58.864 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:58.889 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:58.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:58.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:58.987 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:58 smithi151 ceph-mon[104450]: pgmap v1480: 119 pgs: 119 active+clean; 633 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:43:58.987 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:58 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2126565526' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:58.987 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:58 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/507345268' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:43:58.987 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:58 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:43:58.987 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:58 smithi151 ceph-mon[108966]: pgmap v1480: 119 pgs: 119 active+clean; 633 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:43:58.987 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:58 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2126565526' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:58.987 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:58 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/507345268' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:43:58.987 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:58 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:43:58.991 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:59.012 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:59.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:59.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:59.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:58 smithi184 ceph-mon[92527]: pgmap v1480: 119 pgs: 119 active+clean; 633 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:43:59.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:58 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2126565526' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:59.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:58 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/507345268' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:43:59.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:58 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:43:59.114 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:59.141 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:59.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:59.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:59.243 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:59.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:59.266 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:59.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:59.367 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:59.388 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:59.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:59.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:59.490 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:59.512 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:59.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:59.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:59.613 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:59.635 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:43:59.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:59.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:59.737 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:59.762 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:59.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:59.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:59.864 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:59.889 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:43:59.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:59.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:59.991 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:00.013 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:00.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:00.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:00.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:59 smithi184 ceph-mon[92527]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:00.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:59 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2126565526' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:44:00.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:59 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:44:00.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:43:59 smithi184 ceph-mon[92527]: osdmap e1139: 8 total, 8 up, 8 in 2024-09-13T14:44:00.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:59 smithi151 ceph-mon[104450]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:00.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:59 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2126565526' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:44:00.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:59 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:44:00.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:43:59 smithi151 ceph-mon[104450]: osdmap e1139: 8 total, 8 up, 8 in 2024-09-13T14:44:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:59 smithi151 ceph-mon[108966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:59 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2126565526' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:44:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:59 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:44:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:43:59 smithi151 ceph-mon[108966]: osdmap e1139: 8 total, 8 up, 8 in 2024-09-13T14:44:00.114 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:00.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:00.136 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:00.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:00.237 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:00.261 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:00.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:00.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:00.364 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:00.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:00.385 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:00.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:00.486 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:00.508 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:00.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:00.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:00.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:44:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:44:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:44:00.610 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:00.630 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:00.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:00.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:00.731 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:00.753 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:00.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:00.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:00.855 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:00.876 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:00.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:00.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:00.978 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:00.999 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:01.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:01.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:01.069 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:00 smithi151 ceph-mon[104450]: pgmap v1482: 119 pgs: 119 active+clean; 633 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:44:01.070 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:00 smithi151 ceph-mon[104450]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:44:01.070 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:00 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:44:01.070 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:00 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:44:01.070 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:00 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:44:01.070 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:00 smithi151 ceph-mon[108966]: pgmap v1482: 119 pgs: 119 active+clean; 633 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:44:01.070 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:00 smithi151 ceph-mon[108966]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:44:01.070 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:00 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:44:01.070 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:00 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:44:01.070 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:00 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:44:01.076 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:00 smithi184 ceph-mon[92527]: pgmap v1482: 119 pgs: 119 active+clean; 633 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:44:01.076 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:00 smithi184 ceph-mon[92527]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:44:01.076 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:00 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:44:01.077 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:00 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:44:01.077 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:00 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:44:01.101 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:01.123 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:01.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:01.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:01.225 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:01.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:01.248 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:01.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:01.268 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:44:01.268 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-13T14:44:01.268 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-13T14:44:01.268 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-13T14:44:01.268 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.268 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:166604 6.b 6:d3097778:::benchmark_data_smithi151_155836_object34146:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1105)", 2024-09-13T14:44:01.269 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:41:38.581664+0000", 2024-09-13T14:44:01.269 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 142.683804146, 2024-09-13T14:44:01.269 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35588444600000002, 2024-09-13T14:44:01.269 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:44:01.269 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:44:01.269 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:01.269 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:44:01.269 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:44:01.269 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:44:01.269 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 166604 2024-09-13T14:44:01.269 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.269 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:44:01.269 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.270 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:44:01.270 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:38.581664+0000", 2024-09-13T14:44:01.270 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.270 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.270 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.270 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:44:01.270 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:38.581664+0000", 2024-09-13T14:44:01.270 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.270 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.270 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.270 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:44:01.270 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:38.581665+0000", 2024-09-13T14:44:01.270 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6819999999999999e-06 2024-09-13T14:44:01.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:44:01.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:38.581672+0000", 2024-09-13T14:44:01.271 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1980000000000004e-06 2024-09-13T14:44:01.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:44:01.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:38.581689+0000", 2024-09-13T14:44:01.271 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7085e-05 2024-09-13T14:44:01.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:44:01.272 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:38.581699+0000", 2024-09-13T14:44:01.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4700000000000008e-06 2024-09-13T14:44:01.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.272 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.272 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.272 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:38.588906+0000", 2024-09-13T14:44:01.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0072066739999999997 2024-09-13T14:44:01.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.272 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.272 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:44:01.272 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:38.588978+0000", 2024-09-13T14:44:01.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2180000000000003e-05 2024-09-13T14:44:01.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.273 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.273 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:38.865559+0000", 2024-09-13T14:44:01.273 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27658111000000002 2024-09-13T14:44:01.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.273 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:44:01.273 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:38.865662+0000", 2024-09-13T14:44:01.273 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102598 2024-09-13T14:44:01.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.273 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:44:01.274 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:38.865814+0000", 2024-09-13T14:44:01.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000152409 2024-09-13T14:44:01.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.274 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.274 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:44:01.274 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:38.886619+0000", 2024-09-13T14:44:01.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020804663000000001 2024-09-13T14:44:01.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.274 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.274 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.274 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:38.886679+0000", 2024-09-13T14:44:01.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0661000000000001e-05 2024-09-13T14:44:01.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.274 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.275 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.275 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:38.934424+0000", 2024-09-13T14:44:01.275 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.047745105000000003 2024-09-13T14:44:01.275 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.275 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.275 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:44:01.275 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:38.934496+0000", 2024-09-13T14:44:01.275 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1155999999999994e-05 2024-09-13T14:44:01.275 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.275 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.275 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:44:01.275 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:38.937548+0000", 2024-09-13T14:44:01.275 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030524549999999999 2024-09-13T14:44:01.276 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.276 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:44:01.276 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.276 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.276 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.276 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:183098 6.b 6:d09715c7:::benchmark_data_smithi151_155836_object50640:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1113)", 2024-09-13T14:44:01.276 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:41:59.194610+0000", 2024-09-13T14:44:01.276 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 122.070857496, 2024-09-13T14:44:01.276 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.36153331399999999, 2024-09-13T14:44:01.276 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:44:01.276 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:44:01.276 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:01.276 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:44:01.276 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:44:01.277 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:44:01.277 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 183098 2024-09-13T14:44:01.277 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.277 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:44:01.277 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.277 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:44:01.277 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.194610+0000", 2024-09-13T14:44:01.277 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.277 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.277 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.277 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:44:01.277 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.194610+0000", 2024-09-13T14:44:01.277 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.278 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.278 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:44:01.278 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.194612+0000", 2024-09-13T14:44:01.278 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.502e-06 2024-09-13T14:44:01.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.278 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.278 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:44:01.278 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.194616+0000", 2024-09-13T14:44:01.278 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1060000000000002e-06 2024-09-13T14:44:01.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.278 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.278 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:44:01.278 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.194632+0000", 2024-09-13T14:44:01.279 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6325e-05 2024-09-13T14:44:01.279 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.279 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.279 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:44:01.279 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.194639+0000", 2024-09-13T14:44:01.279 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2690000000000002e-06 2024-09-13T14:44:01.279 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.279 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.279 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.279 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.200186+0000", 2024-09-13T14:44:01.280 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0055466229999999997 2024-09-13T14:44:01.280 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.280 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.280 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:44:01.280 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.200308+0000", 2024-09-13T14:44:01.280 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012204 2024-09-13T14:44:01.280 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.280 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.280 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.280 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.520305+0000", 2024-09-13T14:44:01.280 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31999744699999999 2024-09-13T14:44:01.280 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.280 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.281 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:44:01.281 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.520393+0000", 2024-09-13T14:44:01.281 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7542999999999999e-05 2024-09-13T14:44:01.281 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.281 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.281 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:44:01.281 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.520507+0000", 2024-09-13T14:44:01.281 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011389 2024-09-13T14:44:01.281 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.281 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.281 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.282 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.538812+0000", 2024-09-13T14:44:01.282 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018305578999999999 2024-09-13T14:44:01.282 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.282 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:44:01.282 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.538823+0000", 2024-09-13T14:44:01.282 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0817e-05 2024-09-13T14:44:01.282 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.282 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.282 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.553071+0000", 2024-09-13T14:44:01.282 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014247261000000001 2024-09-13T14:44:01.282 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.283 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.284 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:44:01.284 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.553145+0000", 2024-09-13T14:44:01.284 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4408999999999999e-05 2024-09-13T14:44:01.284 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.284 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.285 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:44:01.285 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.556143+0000", 2024-09-13T14:44:01.285 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002998503 2024-09-13T14:44:01.285 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.285 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:44:01.286 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.286 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.286 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.286 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:183616 6.b 6:d73bec53:::benchmark_data_smithi151_155836_object51158:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1113)", 2024-09-13T14:44:01.286 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:41:59.678101+0000", 2024-09-13T14:44:01.286 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 121.58736688400001, 2024-09-13T14:44:01.286 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.39668080999999999, 2024-09-13T14:44:01.286 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:44:01.286 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:44:01.286 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:01.286 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:44:01.286 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:44:01.286 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:44:01.286 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 183616 2024-09-13T14:44:01.287 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.287 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:44:01.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.287 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:44:01.287 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.678101+0000", 2024-09-13T14:44:01.287 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.287 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.287 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:44:01.287 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.678101+0000", 2024-09-13T14:44:01.287 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.287 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.288 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:44:01.288 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.678103+0000", 2024-09-13T14:44:01.288 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1009999999999999e-06 2024-09-13T14:44:01.288 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.288 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.288 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:44:01.288 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.678109+0000", 2024-09-13T14:44:01.288 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.93e-06 2024-09-13T14:44:01.288 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.288 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.288 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:44:01.288 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.678127+0000", 2024-09-13T14:44:01.288 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8002000000000001e-05 2024-09-13T14:44:01.289 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.289 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.289 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:44:01.289 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.678135+0000", 2024-09-13T14:44:01.289 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4100000000000008e-06 2024-09-13T14:44:01.289 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.289 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.289 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.289 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.699979+0000", 2024-09-13T14:44:01.289 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021843619000000002 2024-09-13T14:44:01.289 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.289 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.289 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:44:01.290 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.700100+0000", 2024-09-13T14:44:01.290 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000120896 2024-09-13T14:44:01.290 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.290 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.290 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.290 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:00.051897+0000", 2024-09-13T14:44:01.290 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35179707900000001 2024-09-13T14:44:01.290 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.290 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.291 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:44:01.291 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:00.052009+0000", 2024-09-13T14:44:01.291 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011215 2024-09-13T14:44:01.291 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.291 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.291 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:44:01.291 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:00.052168+0000", 2024-09-13T14:44:01.291 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015894600000000001 2024-09-13T14:44:01.291 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.291 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.291 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.291 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:00.071742+0000", 2024-09-13T14:44:01.291 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019573958999999998 2024-09-13T14:44:01.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.292 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.292 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:44:01.292 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:00.071755+0000", 2024-09-13T14:44:01.292 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3291e-05 2024-09-13T14:44:01.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.292 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.292 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.292 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:00.071825+0000", 2024-09-13T14:44:01.292 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9987999999999995e-05 2024-09-13T14:44:01.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.292 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.292 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:44:01.293 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:00.071894+0000", 2024-09-13T14:44:01.293 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9153999999999996e-05 2024-09-13T14:44:01.293 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.293 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.293 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:44:01.293 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:00.074782+0000", 2024-09-13T14:44:01.293 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028872849999999999 2024-09-13T14:44:01.293 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.293 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:44:01.293 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.293 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.293 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.293 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:187151 6.b 6:decc40c7:::benchmark_data_smithi151_155836_object54693:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1114)", 2024-09-13T14:44:01.294 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:42:02.893043+0000", 2024-09-13T14:44:01.294 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 118.372424372, 2024-09-13T14:44:01.294 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.469180977, 2024-09-13T14:44:01.294 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:44:01.294 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:44:01.294 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:01.294 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:44:01.294 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:44:01.294 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:44:01.294 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 187151 2024-09-13T14:44:01.294 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.294 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:44:01.294 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.294 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:44:01.295 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.893043+0000", 2024-09-13T14:44:01.295 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.295 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.295 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.295 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:44:01.295 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.893043+0000", 2024-09-13T14:44:01.295 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.295 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.295 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.295 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:44:01.295 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.893045+0000", 2024-09-13T14:44:01.296 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8300000000000001e-06 2024-09-13T14:44:01.296 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.296 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.296 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:44:01.296 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.893052+0000", 2024-09-13T14:44:01.296 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8399999999999997e-06 2024-09-13T14:44:01.296 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.296 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.296 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:44:01.296 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.893072+0000", 2024-09-13T14:44:01.296 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9879000000000001e-05 2024-09-13T14:44:01.296 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.296 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.297 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:44:01.297 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.893081+0000", 2024-09-13T14:44:01.297 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3130000000000004e-06 2024-09-13T14:44:01.297 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.297 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.297 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.297 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.906416+0000", 2024-09-13T14:44:01.297 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013334917999999999 2024-09-13T14:44:01.297 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.297 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.297 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:44:01.297 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.906486+0000", 2024-09-13T14:44:01.297 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0264000000000004e-05 2024-09-13T14:44:01.297 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.298 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.298 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.298 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.320086+0000", 2024-09-13T14:44:01.298 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.41359949600000001 2024-09-13T14:44:01.298 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.298 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.298 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:44:01.298 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.320218+0000", 2024-09-13T14:44:01.298 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013243900000000001 2024-09-13T14:44:01.298 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.298 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.299 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:44:01.299 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.320403+0000", 2024-09-13T14:44:01.299 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018463400000000001 2024-09-13T14:44:01.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.299 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.299 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.299 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.352889+0000", 2024-09-13T14:44:01.299 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.03248579 2024-09-13T14:44:01.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.299 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.299 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.299 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.352929+0000", 2024-09-13T14:44:01.299 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0676000000000002e-05 2024-09-13T14:44:01.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.300 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.300 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:44:01.300 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.358639+0000", 2024-09-13T14:44:01.300 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0057095870000000003 2024-09-13T14:44:01.300 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.300 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.300 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:44:01.301 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.358704+0000", 2024-09-13T14:44:01.301 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4659000000000006e-05 2024-09-13T14:44:01.301 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.301 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.301 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:44:01.301 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.362224+0000", 2024-09-13T14:44:01.301 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003520652 2024-09-13T14:44:01.302 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.302 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:44:01.302 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.302 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.302 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.302 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:187745 6.b 6:dee52bbe:::benchmark_data_smithi151_155836_object55287:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1114)", 2024-09-13T14:44:01.303 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:42:03.457662+0000", 2024-09-13T14:44:01.303 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 117.807805179, 2024-09-13T14:44:01.303 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35777732699999998, 2024-09-13T14:44:01.303 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:44:01.303 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:44:01.303 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:01.304 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:44:01.304 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:44:01.304 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:44:01.304 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 187745 2024-09-13T14:44:01.304 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.305 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:44:01.305 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.305 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:44:01.305 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.457662+0000", 2024-09-13T14:44:01.305 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.305 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.306 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:44:01.306 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.457662+0000", 2024-09-13T14:44:01.306 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.306 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.307 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.307 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:44:01.307 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.457664+0000", 2024-09-13T14:44:01.307 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8160000000000001e-06 2024-09-13T14:44:01.307 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.307 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.307 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:44:01.308 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.457671+0000", 2024-09-13T14:44:01.308 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9650000000000002e-06 2024-09-13T14:44:01.308 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.308 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.308 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:44:01.309 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.457689+0000", 2024-09-13T14:44:01.309 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7645000000000001e-05 2024-09-13T14:44:01.309 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.309 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.309 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:44:01.309 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.457697+0000", 2024-09-13T14:44:01.310 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7400000000000004e-06 2024-09-13T14:44:01.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.310 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.310 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.477562+0000", 2024-09-13T14:44:01.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019864936 2024-09-13T14:44:01.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:44:01.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.477749+0000", 2024-09-13T14:44:01.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018749500000000001 2024-09-13T14:44:01.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.312 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.312 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.743789+0000", 2024-09-13T14:44:01.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26604012500000002 2024-09-13T14:44:01.313 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.313 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:44:01.313 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.743902+0000", 2024-09-13T14:44:01.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000113171 2024-09-13T14:44:01.313 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.314 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:44:01.314 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.744085+0000", 2024-09-13T14:44:01.314 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018276699999999999 2024-09-13T14:44:01.314 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.314 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.765964+0000", 2024-09-13T14:44:01.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021879182 2024-09-13T14:44:01.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:44:01.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.766141+0000", 2024-09-13T14:44:01.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017679800000000001 2024-09-13T14:44:01.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.813067+0000", 2024-09-13T14:44:01.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.046926027000000002 2024-09-13T14:44:01.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.317 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:44:01.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.813121+0000", 2024-09-13T14:44:01.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4138999999999998e-05 2024-09-13T14:44:01.318 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.318 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.318 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:44:01.318 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.815440+0000", 2024-09-13T14:44:01.318 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002318521 2024-09-13T14:44:01.319 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.319 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:44:01.319 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.320 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:188826 6.b 6:d0c74c68:::benchmark_data_smithi151_155836_object56368:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1114)", 2024-09-13T14:44:01.320 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:42:04.417169+0000", 2024-09-13T14:44:01.321 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 116.848298411, 2024-09-13T14:44:01.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38574094599999997, 2024-09-13T14:44:01.321 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:44:01.321 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:44:01.321 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:01.322 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:44:01.322 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:44:01.322 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:44:01.322 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 188826 2024-09-13T14:44:01.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.322 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:44:01.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:44:01.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.417169+0000", 2024-09-13T14:44:01.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.324 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:44:01.324 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.417169+0000", 2024-09-13T14:44:01.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.325 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:44:01.325 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.417170+0000", 2024-09-13T14:44:01.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8299999999999995e-07 2024-09-13T14:44:01.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.325 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:44:01.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.417174+0000", 2024-09-13T14:44:01.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2560000000000004e-06 2024-09-13T14:44:01.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:44:01.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.417183+0000", 2024-09-13T14:44:01.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5089999999999993e-06 2024-09-13T14:44:01.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:44:01.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.417190+0000", 2024-09-13T14:44:01.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9990000000000002e-06 2024-09-13T14:44:01.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.433627+0000", 2024-09-13T14:44:01.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016436563000000001 2024-09-13T14:44:01.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:44:01.329 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.433755+0000", 2024-09-13T14:44:01.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012852999999999999 2024-09-13T14:44:01.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.737039+0000", 2024-09-13T14:44:01.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.30328408000000001 2024-09-13T14:44:01.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.331 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:44:01.331 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.737150+0000", 2024-09-13T14:44:01.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011105899999999999 2024-09-13T14:44:01.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:44:01.332 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.737326+0000", 2024-09-13T14:44:01.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017530099999999999 2024-09-13T14:44:01.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.333 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:44:01.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.756971+0000", 2024-09-13T14:44:01.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019645875 2024-09-13T14:44:01.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.757196+0000", 2024-09-13T14:44:01.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000224293 2024-09-13T14:44:01.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.338 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.800579+0000", 2024-09-13T14:44:01.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.043383781000000003 2024-09-13T14:44:01.339 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.339 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.339 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:44:01.339 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.800647+0000", 2024-09-13T14:44:01.339 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7751999999999998e-05 2024-09-13T14:44:01.339 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.340 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:44:01.340 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.802910+0000", 2024-09-13T14:44:01.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002262965 2024-09-13T14:44:01.340 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.341 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:44:01.341 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.341 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:190363 6.b 6:d5a279ab:::benchmark_data_smithi151_155836_object57905:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1114)", 2024-09-13T14:44:01.341 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:42:05.910014+0000", 2024-09-13T14:44:01.342 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 115.355454045, 2024-09-13T14:44:01.342 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.64606729100000004, 2024-09-13T14:44:01.342 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:44:01.342 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:44:01.342 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:01.342 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:44:01.343 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:44:01.343 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:44:01.343 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 190363 2024-09-13T14:44:01.343 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.343 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:44:01.343 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.344 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:44:01.344 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:05.910014+0000", 2024-09-13T14:44:01.344 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.345 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:44:01.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:05.910014+0000", 2024-09-13T14:44:01.345 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.345 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.345 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.346 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:44:01.346 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:05.910015+0000", 2024-09-13T14:44:01.346 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.091e-06 2024-09-13T14:44:01.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.346 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:44:01.347 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:05.910019+0000", 2024-09-13T14:44:01.347 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9779999999999999e-06 2024-09-13T14:44:01.347 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.347 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.347 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:44:01.347 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:05.910031+0000", 2024-09-13T14:44:01.348 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.183e-05 2024-09-13T14:44:01.348 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.348 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.348 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:44:01.348 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:05.910036+0000", 2024-09-13T14:44:01.349 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4609999999999999e-06 2024-09-13T14:44:01.349 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.349 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.349 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.349 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:05.938331+0000", 2024-09-13T14:44:01.349 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.028295247999999999 2024-09-13T14:44:01.350 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.350 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.350 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:44:01.350 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:05.938412+0000", 2024-09-13T14:44:01.350 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1163999999999997e-05 2024-09-13T14:44:01.351 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.351 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.351 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.351 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:06.533060+0000", 2024-09-13T14:44:01.351 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.59464760900000002 2024-09-13T14:44:01.351 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:44:01.352 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:06.533158+0000", 2024-09-13T14:44:01.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8400000000000007e-05 2024-09-13T14:44:01.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.353 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:44:01.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:06.533330+0000", 2024-09-13T14:44:01.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000171516 2024-09-13T14:44:01.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.354 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.354 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:06.553204+0000", 2024-09-13T14:44:01.354 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019874315 2024-09-13T14:44:01.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.355 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.355 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:06.553264+0000", 2024-09-13T14:44:01.355 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9621999999999999e-05 2024-09-13T14:44:01.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.356 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:44:01.356 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:06.553280+0000", 2024-09-13T14:44:01.356 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6354999999999998e-05 2024-09-13T14:44:01.356 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.356 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.356 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:44:01.357 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:06.553333+0000", 2024-09-13T14:44:01.357 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2459e-05 2024-09-13T14:44:01.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.357 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:44:01.357 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:06.556081+0000", 2024-09-13T14:44:01.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002748243 2024-09-13T14:44:01.358 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.358 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:44:01.358 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.358 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.359 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:192127 6.b 6:dbb5192a:::benchmark_data_smithi151_155836_object59669:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1114)", 2024-09-13T14:44:01.359 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:42:07.565556+0000", 2024-09-13T14:44:01.359 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 113.69991157, 2024-09-13T14:44:01.359 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.391339781, 2024-09-13T14:44:01.359 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:44:01.360 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:44:01.360 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:01.360 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:44:01.360 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:44:01.360 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:44:01.360 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 192127 2024-09-13T14:44:01.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.361 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:44:01.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.361 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:44:01.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.565556+0000", 2024-09-13T14:44:01.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:44:01.362 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.565556+0000", 2024-09-13T14:44:01.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:44:01.362 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.565558+0000", 2024-09-13T14:44:01.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6449999999999999e-06 2024-09-13T14:44:01.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.363 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:44:01.363 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.565563+0000", 2024-09-13T14:44:01.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9180000000000002e-06 2024-09-13T14:44:01.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.363 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:44:01.363 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.565575+0000", 2024-09-13T14:44:01.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2731000000000001e-05 2024-09-13T14:44:01.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.363 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:44:01.363 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.565584+0000", 2024-09-13T14:44:01.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8950000000000004e-06 2024-09-13T14:44:01.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.364 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.582767+0000", 2024-09-13T14:44:01.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017182317999999999 2024-09-13T14:44:01.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:44:01.364 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.582896+0000", 2024-09-13T14:44:01.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000129111 2024-09-13T14:44:01.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.915205+0000", 2024-09-13T14:44:01.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33230937599999999 2024-09-13T14:44:01.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:44:01.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.915313+0000", 2024-09-13T14:44:01.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108392 2024-09-13T14:44:01.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:44:01.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.915478+0000", 2024-09-13T14:44:01.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016415700000000001 2024-09-13T14:44:01.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:44:01.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.936883+0000", 2024-09-13T14:44:01.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021405061 2024-09-13T14:44:01.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.936918+0000", 2024-09-13T14:44:01.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5267e-05 2024-09-13T14:44:01.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.954585+0000", 2024-09-13T14:44:01.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017667157999999999 2024-09-13T14:44:01.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:44:01.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.954625+0000", 2024-09-13T14:44:01.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9453000000000003e-05 2024-09-13T14:44:01.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:44:01.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.956896+0000", 2024-09-13T14:44:01.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0022712990000000001 2024-09-13T14:44:01.367 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.367 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:44:01.368 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.368 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:207761 6.b 6:d0beb7b7:::benchmark_data_smithi151_155836_object75303:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1118)", 2024-09-13T14:44:01.368 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:42:22.393179+0000", 2024-09-13T14:44:01.368 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 98.872288225999995, 2024-09-13T14:44:01.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.584697884, 2024-09-13T14:44:01.368 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:44:01.368 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:44:01.368 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:01.368 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:44:01.368 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:44:01.368 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:44:01.368 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 207761 2024-09-13T14:44:01.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.369 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:44:01.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.369 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:44:01.369 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.393179+0000", 2024-09-13T14:44:01.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.369 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:44:01.369 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.393179+0000", 2024-09-13T14:44:01.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.369 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:44:01.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.393181+0000", 2024-09-13T14:44:01.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6959999999999999e-06 2024-09-13T14:44:01.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:44:01.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.393188+0000", 2024-09-13T14:44:01.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8229999999999997e-06 2024-09-13T14:44:01.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:44:01.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.393204+0000", 2024-09-13T14:44:01.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5905000000000001e-05 2024-09-13T14:44:01.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:44:01.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.393214+0000", 2024-09-13T14:44:01.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0278000000000001e-05 2024-09-13T14:44:01.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.409428+0000", 2024-09-13T14:44:01.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016213676 2024-09-13T14:44:01.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:44:01.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.409553+0000", 2024-09-13T14:44:01.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000124731 2024-09-13T14:44:01.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.372 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.904787+0000", 2024-09-13T14:44:01.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.49523448199999998 2024-09-13T14:44:01.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:44:01.372 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.904875+0000", 2024-09-13T14:44:01.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7701000000000006e-05 2024-09-13T14:44:01.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:44:01.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.904973+0000", 2024-09-13T14:44:01.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8047000000000005e-05 2024-09-13T14:44:01.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:44:01.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.926815+0000", 2024-09-13T14:44:01.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021841843 2024-09-13T14:44:01.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.926877+0000", 2024-09-13T14:44:01.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2391000000000006e-05 2024-09-13T14:44:01.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.975715+0000", 2024-09-13T14:44:01.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048838006000000003 2024-09-13T14:44:01.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:44:01.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.975805+0000", 2024-09-13T14:44:01.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9659000000000003e-05 2024-09-13T14:44:01.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:44:01.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.977877+0000", 2024-09-13T14:44:01.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002072646 2024-09-13T14:44:01.374 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.375 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:44:01.375 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.375 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:209001 6.b 6:d18fd3e1:::benchmark_data_smithi151_155836_object76543:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1118)", 2024-09-13T14:44:01.375 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:42:23.590318+0000", 2024-09-13T14:44:01.375 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 97.675149828000002, 2024-09-13T14:44:01.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.54397842500000004, 2024-09-13T14:44:01.375 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:44:01.375 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:44:01.375 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:01.375 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:44:01.375 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:44:01.375 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:44:01.376 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 209001 2024-09-13T14:44:01.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.376 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:44:01.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:44:01.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:23.590318+0000", 2024-09-13T14:44:01.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:44:01.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:23.590318+0000", 2024-09-13T14:44:01.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:44:01.377 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:23.590320+0000", 2024-09-13T14:44:01.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.928e-06 2024-09-13T14:44:01.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:44:01.377 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:23.590326+0000", 2024-09-13T14:44:01.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9970000000000004e-06 2024-09-13T14:44:01.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:44:01.377 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:23.590343+0000", 2024-09-13T14:44:01.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6920000000000001e-05 2024-09-13T14:44:01.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:44:01.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:23.590351+0000", 2024-09-13T14:44:01.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.816e-06 2024-09-13T14:44:01.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:23.608401+0000", 2024-09-13T14:44:01.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01804944 2024-09-13T14:44:01.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:44:01.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:23.608508+0000", 2024-09-13T14:44:01.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000106989 2024-09-13T14:44:01.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:24.109068+0000", 2024-09-13T14:44:01.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.50055971099999996 2024-09-13T14:44:01.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:44:01.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:24.109189+0000", 2024-09-13T14:44:01.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012140499999999999 2024-09-13T14:44:01.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:44:01.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:24.109365+0000", 2024-09-13T14:44:01.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017573400000000001 2024-09-13T14:44:01.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:24.125803+0000", 2024-09-13T14:44:01.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016438090999999998 2024-09-13T14:44:01.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:44:01.380 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:01.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:24.125983+0000", 2024-09-13T14:44:01.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018028799999999999 2024-09-13T14:44:01.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:24.132081+0000", 2024-09-13T14:44:01.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0060981000000000004 2024-09-13T14:44:01.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:44:01.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:24.132141+0000", 2024-09-13T14:44:01.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9317000000000001e-05 2024-09-13T14:44:01.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:44:01.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:24.134296+0000", 2024-09-13T14:44:01.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021556890000000001 2024-09-13T14:44:01.384 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.384 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:44:01.384 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.384 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:210473 6.b 6:dd7eb092:::benchmark_data_smithi151_155836_object78015:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1118)", 2024-09-13T14:44:01.384 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:42:24.990992+0000", 2024-09-13T14:44:01.384 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 96.274475314, 2024-09-13T14:44:01.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37692983699999999, 2024-09-13T14:44:01.384 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:44:01.384 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:44:01.384 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:01.384 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:44:01.384 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:44:01.385 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:44:01.385 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 210473 2024-09-13T14:44:01.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.385 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:44:01.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:44:01.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:24.990992+0000", 2024-09-13T14:44:01.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:44:01.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:24.990992+0000", 2024-09-13T14:44:01.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:44:01.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:24.990994+0000", 2024-09-13T14:44:01.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.55e-06 2024-09-13T14:44:01.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:44:01.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:24.991000+0000", 2024-09-13T14:44:01.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7270000000000004e-06 2024-09-13T14:44:01.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:44:01.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:24.991014+0000", 2024-09-13T14:44:01.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4643e-05 2024-09-13T14:44:01.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:44:01.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:24.991023+0000", 2024-09-13T14:44:01.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3990000000000004e-06 2024-09-13T14:44:01.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:25.004370+0000", 2024-09-13T14:44:01.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01334752 2024-09-13T14:44:01.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:44:01.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:25.004463+0000", 2024-09-13T14:44:01.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2398000000000005e-05 2024-09-13T14:44:01.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:25.319879+0000", 2024-09-13T14:44:01.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31541669500000002 2024-09-13T14:44:01.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:44:01.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:25.319979+0000", 2024-09-13T14:44:01.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9910999999999999e-05 2024-09-13T14:44:01.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:44:01.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:25.320133+0000", 2024-09-13T14:44:01.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015411 2024-09-13T14:44:01.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:25.343091+0000", 2024-09-13T14:44:01.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.022957898000000001 2024-09-13T14:44:01.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:44:01.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:25.343298+0000", 2024-09-13T14:44:01.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00020694600000000001 2024-09-13T14:44:01.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:25.365334+0000", 2024-09-13T14:44:01.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.022036152 2024-09-13T14:44:01.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:44:01.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:25.365385+0000", 2024-09-13T14:44:01.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0351e-05 2024-09-13T14:44:01.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.391 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:44:01.391 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:25.367922+0000", 2024-09-13T14:44:01.391 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0025375369999999999 2024-09-13T14:44:01.391 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.391 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:44:01.391 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.391 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:213990 6.b 6:d3efb36d:::benchmark_data_smithi151_155836_object81532:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1118)", 2024-09-13T14:44:01.391 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:42:28.482119+0000", 2024-09-13T14:44:01.391 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 92.783348172000004, 2024-09-13T14:44:01.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.43243266600000002, 2024-09-13T14:44:01.392 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:44:01.392 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:44:01.392 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:01.392 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:44:01.392 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:44:01.392 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:44:01.392 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 213990 2024-09-13T14:44:01.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.392 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:44:01.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.392 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:44:01.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.482119+0000", 2024-09-13T14:44:01.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:44:01.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.482119+0000", 2024-09-13T14:44:01.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:44:01.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.482121+0000", 2024-09-13T14:44:01.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.725e-06 2024-09-13T14:44:01.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:44:01.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.482128+0000", 2024-09-13T14:44:01.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3799999999999999e-06 2024-09-13T14:44:01.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:44:01.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.482152+0000", 2024-09-13T14:44:01.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4417e-05 2024-09-13T14:44:01.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:44:01.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.482161+0000", 2024-09-13T14:44:01.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2809999999999997e-06 2024-09-13T14:44:01.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.395 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.498739+0000", 2024-09-13T14:44:01.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016577273999999999 2024-09-13T14:44:01.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:44:01.395 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.498872+0000", 2024-09-13T14:44:01.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000133153 2024-09-13T14:44:01.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.395 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.833103+0000", 2024-09-13T14:44:01.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.334231428 2024-09-13T14:44:01.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.396 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:44:01.396 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.833172+0000", 2024-09-13T14:44:01.396 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9220000000000005e-05 2024-09-13T14:44:01.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.396 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:44:01.396 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.833291+0000", 2024-09-13T14:44:01.396 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118146 2024-09-13T14:44:01.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.396 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:44:01.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.843364+0000", 2024-09-13T14:44:01.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010073716 2024-09-13T14:44:01.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.871140+0000", 2024-09-13T14:44:01.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.027776085999999998 2024-09-13T14:44:01.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.912981+0000", 2024-09-13T14:44:01.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041840255999999999 2024-09-13T14:44:01.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:44:01.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.913056+0000", 2024-09-13T14:44:01.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5321000000000006e-05 2024-09-13T14:44:01.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:44:01.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.914552+0000", 2024-09-13T14:44:01.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0014962630000000001 2024-09-13T14:44:01.398 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.398 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:44:01.398 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.399 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:234885 6.b 6:df1bf793:::benchmark_data_smithi151_155836_object102427:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1123)", 2024-09-13T14:44:01.399 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:42:49.694861+0000", 2024-09-13T14:44:01.399 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 71.570606233999996, 2024-09-13T14:44:01.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37100666999999998, 2024-09-13T14:44:01.399 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:44:01.399 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:44:01.399 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:01.399 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:44:01.399 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:44:01.399 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:44:01.399 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 234885 2024-09-13T14:44:01.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.400 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:44:01.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:44:01.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:49.694861+0000", 2024-09-13T14:44:01.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:44:01.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:49.694861+0000", 2024-09-13T14:44:01.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:44:01.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:49.694863+0000", 2024-09-13T14:44:01.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3629999999999999e-06 2024-09-13T14:44:01.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:44:01.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:49.694868+0000", 2024-09-13T14:44:01.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.57e-06 2024-09-13T14:44:01.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:44:01.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:49.694880+0000", 2024-09-13T14:44:01.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1952e-05 2024-09-13T14:44:01.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:44:01.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:49.694886+0000", 2024-09-13T14:44:01.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8259999999999998e-06 2024-09-13T14:44:01.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:49.694941+0000", 2024-09-13T14:44:01.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5299e-05 2024-09-13T14:44:01.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:44:01.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:49.695070+0000", 2024-09-13T14:44:01.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012866099999999999 2024-09-13T14:44:01.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:49.991850+0000", 2024-09-13T14:44:01.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29677958799999998 2024-09-13T14:44:01.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:44:01.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:49.991929+0000", 2024-09-13T14:44:01.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9664000000000002e-05 2024-09-13T14:44:01.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:44:01.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:49.992047+0000", 2024-09-13T14:44:01.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117774 2024-09-13T14:44:01.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.006509+0000", 2024-09-13T14:44:01.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014462226 2024-09-13T14:44:01.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:44:01.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.006524+0000", 2024-09-13T14:44:01.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4389000000000001e-05 2024-09-13T14:44:01.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.063605+0000", 2024-09-13T14:44:01.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.057081112000000003 2024-09-13T14:44:01.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:44:01.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.063648+0000", 2024-09-13T14:44:01.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3260999999999998e-05 2024-09-13T14:44:01.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:44:01.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.065868+0000", 2024-09-13T14:44:01.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0022199849999999998 2024-09-13T14:44:01.405 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.405 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:44:01.405 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.406 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:235375 6.b 6:d73ddd5f:::benchmark_data_smithi151_155836_object102917:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1123)", 2024-09-13T14:44:01.406 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:42:50.193168+0000", 2024-09-13T14:44:01.406 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 71.072299540000003, 2024-09-13T14:44:01.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.51423233800000001, 2024-09-13T14:44:01.406 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:44:01.406 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:44:01.406 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:01.406 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:44:01.406 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:44:01.406 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:44:01.406 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 235375 2024-09-13T14:44:01.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.407 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:44:01.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:44:01.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.193168+0000", 2024-09-13T14:44:01.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:44:01.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.193168+0000", 2024-09-13T14:44:01.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:44:01.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.193170+0000", 2024-09-13T14:44:01.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5740000000000001e-06 2024-09-13T14:44:01.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:44:01.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.193175+0000", 2024-09-13T14:44:01.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6439999999999997e-06 2024-09-13T14:44:01.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:44:01.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.193188+0000", 2024-09-13T14:44:01.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2225999999999999e-05 2024-09-13T14:44:01.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:44:01.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.193196+0000", 2024-09-13T14:44:01.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6500000000000002e-06 2024-09-13T14:44:01.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.213195+0000", 2024-09-13T14:44:01.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019998616 2024-09-13T14:44:01.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:44:01.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.213331+0000", 2024-09-13T14:44:01.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013614000000000001 2024-09-13T14:44:01.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.410 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.658507+0000", 2024-09-13T14:44:01.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.44517631200000002 2024-09-13T14:44:01.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.410 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:44:01.410 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.658619+0000", 2024-09-13T14:44:01.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011189899999999999 2024-09-13T14:44:01.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.410 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:44:01.410 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.658779+0000", 2024-09-13T14:44:01.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015954900000000001 2024-09-13T14:44:01.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.677188+0000", 2024-09-13T14:44:01.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018409132000000002 2024-09-13T14:44:01.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:44:01.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.677210+0000", 2024-09-13T14:44:01.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1832999999999999e-05 2024-09-13T14:44:01.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.705544+0000", 2024-09-13T14:44:01.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.028333816000000001 2024-09-13T14:44:01.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:44:01.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.705601+0000", 2024-09-13T14:44:01.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7379000000000001e-05 2024-09-13T14:44:01.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:44:01.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.707400+0000", 2024-09-13T14:44:01.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017995680000000001 2024-09-13T14:44:01.412 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.412 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:44:01.412 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.413 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:236769 6.b 6:d4580e47:::benchmark_data_smithi151_155836_object104311:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1123)", 2024-09-13T14:44:01.413 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:42:51.576789+0000", 2024-09-13T14:44:01.413 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 69.688678316999997, 2024-09-13T14:44:01.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.48812771999999999, 2024-09-13T14:44:01.413 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:44:01.413 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:44:01.413 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:01.413 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:44:01.413 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:44:01.413 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:44:01.413 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 236769 2024-09-13T14:44:01.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.414 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:44:01.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:44:01.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:51.576789+0000", 2024-09-13T14:44:01.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:44:01.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:51.576789+0000", 2024-09-13T14:44:01.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:44:01.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:51.576798+0000", 2024-09-13T14:44:01.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0340000000000002e-06 2024-09-13T14:44:01.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:44:01.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:51.576803+0000", 2024-09-13T14:44:01.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.656e-06 2024-09-13T14:44:01.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:44:01.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:51.576816+0000", 2024-09-13T14:44:01.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3448999999999999e-05 2024-09-13T14:44:01.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:44:01.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:51.576823+0000", 2024-09-13T14:44:01.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8140000000000003e-06 2024-09-13T14:44:01.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:51.585868+0000", 2024-09-13T14:44:01.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0090445979999999992 2024-09-13T14:44:01.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:44:01.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:51.585933+0000", 2024-09-13T14:44:01.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5246000000000005e-05 2024-09-13T14:44:01.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:52.033369+0000", 2024-09-13T14:44:01.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.44743548599999999 2024-09-13T14:44:01.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:44:01.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:52.033475+0000", 2024-09-13T14:44:01.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000106031 2024-09-13T14:44:01.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:44:01.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:52.033634+0000", 2024-09-13T14:44:01.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000159474 2024-09-13T14:44:01.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:52.049326+0000", 2024-09-13T14:44:01.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015692053000000001 2024-09-13T14:44:01.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:44:01.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:52.049360+0000", 2024-09-13T14:44:01.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3855e-05 2024-09-13T14:44:01.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:52.063272+0000", 2024-09-13T14:44:01.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013912203 2024-09-13T14:44:01.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:44:01.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:52.063313+0000", 2024-09-13T14:44:01.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0306999999999999e-05 2024-09-13T14:44:01.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:44:01.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:52.064917+0000", 2024-09-13T14:44:01.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001604514 2024-09-13T14:44:01.420 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.420 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:44:01.420 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.420 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:240044 6.b 6:ddd6b124:::benchmark_data_smithi151_155836_object107586:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1123)", 2024-09-13T14:44:01.420 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:42:54.772464+0000", 2024-09-13T14:44:01.420 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 66.493003967999996, 2024-09-13T14:44:01.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.78119298299999995, 2024-09-13T14:44:01.421 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:44:01.421 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:44:01.421 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:01.421 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:44:01.421 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:44:01.421 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:44:01.421 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 240044 2024-09-13T14:44:01.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.421 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:44:01.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:44:01.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:54.772464+0000", 2024-09-13T14:44:01.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:44:01.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:54.772464+0000", 2024-09-13T14:44:01.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:44:01.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:54.772465+0000", 2024-09-13T14:44:01.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3030000000000001e-06 2024-09-13T14:44:01.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:44:01.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:54.772470+0000", 2024-09-13T14:44:01.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0969999999999996e-06 2024-09-13T14:44:01.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:44:01.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:54.772494+0000", 2024-09-13T14:44:01.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4165999999999999e-05 2024-09-13T14:44:01.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:44:01.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:54.772503+0000", 2024-09-13T14:44:01.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2310000000000002e-06 2024-09-13T14:44:01.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:54.785159+0000", 2024-09-13T14:44:01.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012655614000000001 2024-09-13T14:44:01.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:44:01.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:54.785278+0000", 2024-09-13T14:44:01.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011937700000000001 2024-09-13T14:44:01.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:55.494601+0000", 2024-09-13T14:44:01.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.70932297600000005 2024-09-13T14:44:01.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:44:01.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:55.494702+0000", 2024-09-13T14:44:01.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000100374 2024-09-13T14:44:01.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:44:01.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:55.494970+0000", 2024-09-13T14:44:01.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00026814299999999998 2024-09-13T14:44:01.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:55.510832+0000", 2024-09-13T14:44:01.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015861896 2024-09-13T14:44:01.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.426 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:44:01.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:55.511008+0000", 2024-09-13T14:44:01.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017613199999999999 2024-09-13T14:44:01.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.426 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:55.551768+0000", 2024-09-13T14:44:01.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.040759660000000003 2024-09-13T14:44:01.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.426 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:44:01.427 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:55.551814+0000", 2024-09-13T14:44:01.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6601999999999999e-05 2024-09-13T14:44:01.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.427 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:44:01.427 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:55.553657+0000", 2024-09-13T14:44:01.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018424120000000001 2024-09-13T14:44:01.427 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.427 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:44:01.427 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.427 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:242839 6.b 6:d98e4068:::benchmark_data_smithi151_155836_object110381:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1123)", 2024-09-13T14:44:01.427 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:42:57.578423+0000", 2024-09-13T14:44:01.428 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 63.687044696999997, 2024-09-13T14:44:01.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33881746099999999, 2024-09-13T14:44:01.428 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:44:01.428 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:44:01.428 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:01.428 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:44:01.428 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:44:01.428 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:44:01.428 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 242839 2024-09-13T14:44:01.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.428 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:44:01.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.428 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:44:01.428 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:57.578423+0000", 2024-09-13T14:44:01.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:44:01.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:57.578423+0000", 2024-09-13T14:44:01.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:44:01.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:57.578425+0000", 2024-09-13T14:44:01.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6619999999999999e-06 2024-09-13T14:44:01.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:44:01.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:57.578430+0000", 2024-09-13T14:44:01.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6529999999999999e-06 2024-09-13T14:44:01.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:44:01.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:57.578444+0000", 2024-09-13T14:44:01.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3875e-05 2024-09-13T14:44:01.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:44:01.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:57.578452+0000", 2024-09-13T14:44:01.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7689999999999996e-06 2024-09-13T14:44:01.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.431 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:57.578507+0000", 2024-09-13T14:44:01.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5118999999999999e-05 2024-09-13T14:44:01.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:44:01.431 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:57.578614+0000", 2024-09-13T14:44:01.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000107384 2024-09-13T14:44:01.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.431 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:57.856226+0000", 2024-09-13T14:44:01.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27761200200000002 2024-09-13T14:44:01.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:44:01.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:57.856323+0000", 2024-09-13T14:44:01.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6570000000000005e-05 2024-09-13T14:44:01.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:44:01.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:57.856419+0000", 2024-09-13T14:44:01.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6291000000000004e-05 2024-09-13T14:44:01.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:44:01.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:57.873625+0000", 2024-09-13T14:44:01.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017206064 2024-09-13T14:44:01.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:57.873666+0000", 2024-09-13T14:44:01.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0868000000000002e-05 2024-09-13T14:44:01.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:57.915575+0000", 2024-09-13T14:44:01.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041909277000000002 2024-09-13T14:44:01.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:44:01.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:57.915621+0000", 2024-09-13T14:44:01.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5476999999999998e-05 2024-09-13T14:44:01.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:44:01.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:57.917240+0000", 2024-09-13T14:44:01.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016194499999999999 2024-09-13T14:44:01.434 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.434 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:44:01.434 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.434 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:243296 6.b 6:da27bd7d:::benchmark_data_smithi151_155836_object110838:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1123)", 2024-09-13T14:44:01.435 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:42:58.035785+0000", 2024-09-13T14:44:01.435 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 63.229683061999999, 2024-09-13T14:44:01.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.42214216399999999, 2024-09-13T14:44:01.435 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:44:01.435 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:44:01.435 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:01.435 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:44:01.435 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:44:01.435 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:44:01.435 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 243296 2024-09-13T14:44:01.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.435 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:44:01.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.435 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:44:01.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.035785+0000", 2024-09-13T14:44:01.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.436 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:44:01.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.035785+0000", 2024-09-13T14:44:01.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.436 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:44:01.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.035786+0000", 2024-09-13T14:44:01.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.863e-06 2024-09-13T14:44:01.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:44:01.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.035802+0000", 2024-09-13T14:44:01.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5777000000000002e-05 2024-09-13T14:44:01.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:44:01.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.035820+0000", 2024-09-13T14:44:01.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8002000000000001e-05 2024-09-13T14:44:01.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:44:01.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.035829+0000", 2024-09-13T14:44:01.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7199999999999995e-06 2024-09-13T14:44:01.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.035902+0000", 2024-09-13T14:44:01.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3250000000000005e-05 2024-09-13T14:44:01.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:44:01.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.036031+0000", 2024-09-13T14:44:01.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012864800000000001 2024-09-13T14:44:01.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.414050+0000", 2024-09-13T14:44:01.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37801879799999999 2024-09-13T14:44:01.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:44:01.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.414136+0000", 2024-09-13T14:44:01.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6286999999999996e-05 2024-09-13T14:44:01.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:44:01.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.414281+0000", 2024-09-13T14:44:01.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014502100000000001 2024-09-13T14:44:01.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.430721+0000", 2024-09-13T14:44:01.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016440132 2024-09-13T14:44:01.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:44:01.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.430746+0000", 2024-09-13T14:44:01.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5298000000000001e-05 2024-09-13T14:44:01.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.456137+0000", 2024-09-13T14:44:01.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.025390942 2024-09-13T14:44:01.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:44:01.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.456208+0000", 2024-09-13T14:44:01.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0693999999999998e-05 2024-09-13T14:44:01.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:44:01.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.457927+0000", 2024-09-13T14:44:01.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017187319999999999 2024-09-13T14:44:01.441 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.441 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:44:01.441 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.442 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:246042 6.b 6:d9559628:::benchmark_data_smithi151_155836_object113584:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1123)", 2024-09-13T14:44:01.442 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:00.744657+0000", 2024-09-13T14:44:01.442 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.520810265000001, 2024-09-13T14:44:01.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.44821939, 2024-09-13T14:44:01.442 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:44:01.442 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:44:01.442 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:01.442 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:44:01.442 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:44:01.442 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:44:01.442 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 246042 2024-09-13T14:44:01.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.442 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:44:01.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:44:01.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:00.744657+0000", 2024-09-13T14:44:01.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:44:01.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:00.744657+0000", 2024-09-13T14:44:01.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:44:01.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:00.744659+0000", 2024-09-13T14:44:01.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7099999999999999e-06 2024-09-13T14:44:01.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:44:01.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:00.744667+0000", 2024-09-13T14:44:01.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6059999999999997e-06 2024-09-13T14:44:01.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:44:01.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:00.744685+0000", 2024-09-13T14:44:01.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8692999999999999e-05 2024-09-13T14:44:01.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:44:01.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:00.744694+0000", 2024-09-13T14:44:01.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0450000000000006e-06 2024-09-13T14:44:01.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:00.751063+0000", 2024-09-13T14:44:01.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.006368851 2024-09-13T14:44:01.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:44:01.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:00.751190+0000", 2024-09-13T14:44:01.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126785 2024-09-13T14:44:01.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.161878+0000", 2024-09-13T14:44:01.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.41068771199999998 2024-09-13T14:44:01.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:44:01.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.161945+0000", 2024-09-13T14:44:01.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7687000000000004e-05 2024-09-13T14:44:01.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:44:01.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.162037+0000", 2024-09-13T14:44:01.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1244999999999998e-05 2024-09-13T14:44:01.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.176824+0000", 2024-09-13T14:44:01.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014787719 2024-09-13T14:44:01.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:44:01.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.176841+0000", 2024-09-13T14:44:01.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6333000000000001e-05 2024-09-13T14:44:01.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.191064+0000", 2024-09-13T14:44:01.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014222894999999999 2024-09-13T14:44:01.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:44:01.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.191140+0000", 2024-09-13T14:44:01.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6179000000000004e-05 2024-09-13T14:44:01.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:44:01.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.192877+0000", 2024-09-13T14:44:01.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00173693 2024-09-13T14:44:01.448 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.448 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:44:01.448 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.449 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:247199 6.b 6:da84a270:::benchmark_data_smithi151_155836_object114741:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1123)", 2024-09-13T14:44:01.449 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:01.891288+0000", 2024-09-13T14:44:01.449 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 59.374179376999997, 2024-09-13T14:44:01.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.47403641499999999, 2024-09-13T14:44:01.449 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:44:01.449 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:44:01.449 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:01.449 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:44:01.449 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:44:01.449 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:44:01.449 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 247199 2024-09-13T14:44:01.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.449 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:44:01.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:44:01.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.891288+0000", 2024-09-13T14:44:01.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:44:01.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.891288+0000", 2024-09-13T14:44:01.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:44:01.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:44:01.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.891290+0000", 2024-09-13T14:44:01.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4649999999999999e-06 2024-09-13T14:44:01.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:44:01.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.891295+0000", 2024-09-13T14:44:01.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5709999999999997e-06 2024-09-13T14:44:01.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:44:01.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.891309+0000", 2024-09-13T14:44:01.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3893000000000001e-05 2024-09-13T14:44:01.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:44:01.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.891316+0000", 2024-09-13T14:44:01.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8909999999999998e-06 2024-09-13T14:44:01.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.901646+0000", 2024-09-13T14:44:01.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010329589 2024-09-13T14:44:01.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:44:01.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.901712+0000", 2024-09-13T14:44:01.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6644999999999996e-05 2024-09-13T14:44:01.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:44:01.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:02.327995+0000", 2024-09-13T14:44:01.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.426282457 2024-09-13T14:44:01.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:44:01.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:02.328102+0000", 2024-09-13T14:44:01.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010686 2024-09-13T14:44:01.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:44:01.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:02.328263+0000", 2024-09-13T14:44:01.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000161686 2024-09-13T14:44:01.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:02.346868+0000", 2024-09-13T14:44:01.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018605047999999999 2024-09-13T14:44:01.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:44:01.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:02.346878+0000", 2024-09-13T14:44:01.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6339999999999996e-06 2024-09-13T14:44:01.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:01.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:02.363734+0000", 2024-09-13T14:44:01.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016856428 2024-09-13T14:44:01.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:44:01.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:02.363810+0000", 2024-09-13T14:44:01.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5617000000000005e-05 2024-09-13T14:44:01.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:44:01.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:44:01.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:44:01.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:02.365325+0000", 2024-09-13T14:44:01.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001514631 2024-09-13T14:44:01.455 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.455 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:44:01.455 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.455 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:44:01.455 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:44:01.455 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:44:01.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:01.457 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:01.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:01.558 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:01.579 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:01.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:01.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:01.669 DEBUG:teuthology.orchestra.run.smithi151:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-09-13T14:44:01.670 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-13T14:44:01.670 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:44:01.672 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T14:44:01.672 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T14:44:01.681 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:01.702 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:01.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:01.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:01.718 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:hints = 1 2024-09-13T14:44:01.719 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T14:44:01.719 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Object prefix: benchmark_data_smithi151_190906 2024-09-13T14:44:01.719 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:44:01.719 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 0 0 0 0 0 0 - 0 2024-09-13T14:44:01.803 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:01.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:44:01.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:01.847 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:01.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:01.948 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:01.967 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:01 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:01.968 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:01 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:44:01.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:01.996 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:01.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:02.097 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:02.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:02.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:44:02.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:01 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:02.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:01 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:44:02.129 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:02.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:02.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:02.230 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:02.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:02.266 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:02.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:02.367 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:02.403 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:02.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:02.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:02.505 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:02.544 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:02.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:02.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:02.646 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:02.681 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:02.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:02.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:02.720 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 1 16 3305 3289 205.551 205.562 0.00109766 0.00463524 2024-09-13T14:44:02.784 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:02.823 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:02.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:02.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:02.925 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:02.957 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:02.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:02.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:03.059 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:03.079 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:03.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:03.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:03.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:02 smithi184 ceph-mon[92527]: pgmap v1483: 119 pgs: 119 active+clean; 633 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:44:03.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]} : dispatch 2024-09-13T14:44:03.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-13T14:44:03.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]} : dispatch 2024-09-13T14:44:03.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [1, 5]} : dispatch 2024-09-13T14:44:03.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]} : dispatch 2024-09-13T14:44:03.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:02 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/179427722' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:03.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:02 smithi151 ceph-mon[104450]: pgmap v1483: 119 pgs: 119 active+clean; 633 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:44:03.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]} : dispatch 2024-09-13T14:44:03.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-13T14:44:03.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]} : dispatch 2024-09-13T14:44:03.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [1, 5]} : dispatch 2024-09-13T14:44:03.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]} : dispatch 2024-09-13T14:44:03.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:02 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/179427722' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:03.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:02 smithi151 ceph-mon[108966]: pgmap v1483: 119 pgs: 119 active+clean; 633 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-13T14:44:03.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]} : dispatch 2024-09-13T14:44:03.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-13T14:44:03.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]} : dispatch 2024-09-13T14:44:03.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [1, 5]} : dispatch 2024-09-13T14:44:03.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]} : dispatch 2024-09-13T14:44:03.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:02 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/179427722' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:03.182 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:03.222 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:03.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:03.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:03.324 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:03.360 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:03.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:03.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:03.462 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:03.499 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:03.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:03.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:03.601 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:03.637 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:03.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:03.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:03.719 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 2 16 6504 6488 202.737 199.938 0.00385378 0.00492389 2024-09-13T14:44:03.739 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:03.776 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:03.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:03.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:03.878 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:03.904 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:44:03.904 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:44:03.904 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:44:03.904 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:44:03.904 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:44:03.904 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:44:03.904 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:44:03.905 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:44:03.905 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:44:03.905 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:44:03.905 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:44:03.905 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:44:03.905 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:44:03.906 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:44:03.906 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:44:03.906 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:44:03.913 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:03.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:03.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:03.922 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:03 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-09-13T14:44:03.922 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:03 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-13T14:44:03.922 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:03 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-09-13T14:44:03.922 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:03 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [1, 5]}]': finished 2024-09-13T14:44:03.922 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:03 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-09-13T14:44:03.922 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:03 smithi184 ceph-mon[92527]: osdmap e1140: 8 total, 8 up, 8 in 2024-09-13T14:44:03.922 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:03 smithi184 ceph-mon[92527]: osdmap e1141: 8 total, 8 up, 8 in 2024-09-13T14:44:04.015 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:04.048 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:04.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:04.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:04.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:03 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-09-13T14:44:04.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:03 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-13T14:44:04.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:03 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-09-13T14:44:04.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:03 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [1, 5]}]': finished 2024-09-13T14:44:04.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:03 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-09-13T14:44:04.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:03 smithi151 ceph-mon[104450]: osdmap e1140: 8 total, 8 up, 8 in 2024-09-13T14:44:04.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:03 smithi151 ceph-mon[104450]: osdmap e1141: 8 total, 8 up, 8 in 2024-09-13T14:44:04.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:03 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-09-13T14:44:04.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:03 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-13T14:44:04.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:03 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-09-13T14:44:04.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:03 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [1, 5]}]': finished 2024-09-13T14:44:04.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:03 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-09-13T14:44:04.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:03 smithi151 ceph-mon[108966]: osdmap e1140: 8 total, 8 up, 8 in 2024-09-13T14:44:04.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:03 smithi151 ceph-mon[108966]: osdmap e1141: 8 total, 8 up, 8 in 2024-09-13T14:44:04.150 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:04.188 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:04.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:04.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:04.290 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:04.330 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:04.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:04.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:04.432 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:04.463 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:04.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:04.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:04.565 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:04.594 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:04.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:04.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:04.696 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:04.720 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 3 16 9751 9735 202.718 202.938 0.00735532 0.00492304 2024-09-13T14:44:04.732 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:04.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:04.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:04.834 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:04.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:44:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:44:04] "GET /metrics HTTP/1.1" 200 36595 "" "Prometheus/2.51.0" 2024-09-13T14:44:04.871 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:04.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:04.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:04.909 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:04 smithi184 ceph-mon[92527]: pgmap v1485: 119 pgs: 119 active+clean; 60 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 7.5 MiB/s wr, 119 op/s 2024-09-13T14:44:04.973 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:05.003 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:05.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:05.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:05.105 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:05.134 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:05.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:05.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:05.236 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:05.277 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:05.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:05.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:05.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:04 smithi151 ceph-mon[104450]: pgmap v1485: 119 pgs: 119 active+clean; 60 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 7.5 MiB/s wr, 119 op/s 2024-09-13T14:44:05.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:04 smithi151 ceph-mon[108966]: pgmap v1485: 119 pgs: 119 active+clean; 60 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 7.5 MiB/s wr, 119 op/s 2024-09-13T14:44:05.378 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:05.411 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:05.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:05.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:05.513 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:05.549 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:05.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:05.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:05.651 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:05.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:05.688 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:05.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:05.720 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 4 16 12444 12428 194.116 168.312 0.00324429 0.00514495 2024-09-13T14:44:05.789 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:05.827 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:05.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:05.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:05.929 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:05.961 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:05.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:05.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:06.063 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:06.102 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:06.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:06.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:06.204 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:06.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:06.246 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:06.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:06.347 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:06.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:06 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:44:06.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:06 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:44:06.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:06 smithi151 ceph-mon[104450]: pgmap v1487: 119 pgs: 119 active+clean; 197 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 30 MiB/s wr, 476 op/s 2024-09-13T14:44:06.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:06 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:44:06.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:06 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:44:06.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:06 smithi151 ceph-mon[108966]: pgmap v1487: 119 pgs: 119 active+clean; 197 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 30 MiB/s wr, 476 op/s 2024-09-13T14:44:06.381 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:06.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:06.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:06.484 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:06.515 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:06.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:06.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:06.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:06 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:44:06.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:06 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:44:06.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:06 smithi184 ceph-mon[92527]: pgmap v1487: 119 pgs: 119 active+clean; 197 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 30 MiB/s wr, 476 op/s 2024-09-13T14:44:06.617 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:06.649 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:06.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:06.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:06.720 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 5 16 15006 14990 187.316 160.125 0.00903048 0.00533226 2024-09-13T14:44:06.751 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:06.784 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:06.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:06.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:06.886 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:06.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:06.918 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:06.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:07.019 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:07.054 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:07.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:07.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:07.156 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:07.187 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:07.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:07.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:07.289 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:07.324 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:07.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:07.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:07.426 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:07.447 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:07.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:07.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:07.549 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:07.586 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:07.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:07.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:07.688 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:07.708 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:07.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:07.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:07.720 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 6 16 17315 17299 180.147 144.312 0.00486103 0.00551184 2024-09-13T14:44:07.810 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:07.841 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:07.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:07.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:07.942 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:07.975 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:07.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:07.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:08.077 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:08.113 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:08.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:08.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:08.215 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:08.247 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:08.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:08.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:08.349 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:08.374 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:08.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:08.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:08.476 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:08.508 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:08.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:08.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:08.610 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:08.631 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:08.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:08.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:08.721 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 7 16 19466 19450 173.617 134.438 0.00439099 0.00575445 2024-09-13T14:44:08.732 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:08.769 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:08.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:08.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:08.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:08 smithi184 ceph-mon[92527]: pgmap v1488: 119 pgs: 119 active+clean; 563 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-13T14:44:08.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:08 smithi151 ceph-mon[108966]: pgmap v1488: 119 pgs: 119 active+clean; 563 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-13T14:44:08.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:08 smithi151 ceph-mon[104450]: pgmap v1488: 119 pgs: 119 active+clean; 563 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-13T14:44:08.872 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:08.903 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:08.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:08.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:09.005 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:09.026 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:09.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:09.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:09.128 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:09.161 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:09.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:09.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:09.263 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:09.295 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:09.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:09.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:09.397 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:09.418 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:09.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:09.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:09.520 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:09.558 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:09.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:09.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:09.660 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:09.690 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:09.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:09.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:09.721 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 8 16 21693 21677 169.311 139.188 0.00242922 0.0058831 2024-09-13T14:44:09.793 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:09.826 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:09.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:09.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:09.928 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:09.964 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:09.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:09.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:10.066 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:10.087 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:10.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:10.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:10.189 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:10.219 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:10.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:10.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:10.321 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:10.342 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:10.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:10.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:10.444 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:10.466 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:10.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:10.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:10.567 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:10.599 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:10.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:10.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:10.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:10 smithi184 ceph-mon[92527]: pgmap v1489: 119 pgs: 119 active+clean; 1.0 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-09-13T14:44:10.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:44:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:44:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:44:10.701 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:10.721 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 9 16 23467 23451 162.818 110.875 0.00163408 0.00612961 2024-09-13T14:44:10.726 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:10.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:10.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:10.827 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:10.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:10 smithi151 ceph-mon[104450]: pgmap v1489: 119 pgs: 119 active+clean; 1.0 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-09-13T14:44:10.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:10 smithi151 ceph-mon[108966]: pgmap v1489: 119 pgs: 119 active+clean; 1.0 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-09-13T14:44:10.863 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:10.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:10.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:10.964 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:11.047 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:11.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:11.149 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:11.181 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:11.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:11.283 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:11.313 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:11.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:11.415 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:11.445 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:11.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:11.547 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:11.577 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:11.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:11.679 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:11.713 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:11.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:11.721 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 10 16 25730 25714 160.679 141.438 0.00534106 0.00621788 2024-09-13T14:44:11.816 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:11.837 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:11.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:11.939 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:11.973 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:11.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:12.075 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:12.112 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:12.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:12.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:12.213 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:12.237 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:12.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:12.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:12.338 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:12.375 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:12.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:12.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:12.477 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:12.508 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:12.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:12.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:12.610 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:12.644 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:12.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:12.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:12.721 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 11 16 27914 27898 158.48 136.5 0.0033842 0.00630498 2024-09-13T14:44:12.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:12.784 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:12.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:12.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:12.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:12 smithi184 ceph-mon[92527]: pgmap v1490: 119 pgs: 119 active+clean; 1.2 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 135 MiB/s wr, 2.16k op/s 2024-09-13T14:44:12.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:12 smithi151 ceph-mon[104450]: pgmap v1490: 119 pgs: 119 active+clean; 1.2 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 135 MiB/s wr, 2.16k op/s 2024-09-13T14:44:12.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:12 smithi151 ceph-mon[108966]: pgmap v1490: 119 pgs: 119 active+clean; 1.2 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 135 MiB/s wr, 2.16k op/s 2024-09-13T14:44:12.885 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:12.917 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:12.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:12.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:13.018 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:13.039 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:13.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:13.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:13.141 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:13.171 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:13.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:13.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:13.273 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:13.303 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:13.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:13.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:13.404 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:13.433 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:13.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:13.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:13.535 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:13.572 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:13.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:13.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:13.674 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:13.705 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:13.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:13.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:13.721 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 12 16 30094 30078 156.627 136.25 0.00839662 0.0063788 2024-09-13T14:44:13.809 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:13.830 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:13.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:13.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:13.932 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:13.963 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:13.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:13.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:14.065 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:14.097 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:14.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:14.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:14.199 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:14.222 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:14.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:14.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:14.324 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:14.356 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:14.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:14.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:14.458 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:14.486 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:14.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:14.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:14.588 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:14.609 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:14.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:14.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:14.710 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:14.721 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 13 16 31494 31478 151.309 87.5 0.00252458 0.006588 2024-09-13T14:44:14.731 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:14.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:14.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:14.833 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:14.850 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:14 smithi184 ceph-mon[92527]: pgmap v1491: 119 pgs: 119 active+clean; 1.5 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-09-13T14:44:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:14 smithi151 ceph-mon[104450]: pgmap v1491: 119 pgs: 119 active+clean; 1.5 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-09-13T14:44:14.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:44:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:44:14] "GET /metrics HTTP/1.1" 200 36595 "" "Prometheus/2.51.0" 2024-09-13T14:44:14.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:14 smithi151 ceph-mon[108966]: pgmap v1491: 119 pgs: 119 active+clean; 1.5 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-09-13T14:44:14.866 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:14.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:14.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:14.968 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:15.005 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:15.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:15.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:15.107 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:15.142 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:15.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:15.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:15.244 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:15.279 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:15.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:15.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:15.381 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:15.403 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:15.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:15.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:15.504 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:15.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:15.543 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:15.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:15.644 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:15.679 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:15.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:15.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:15.721 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 14 16 33180 33164 148.027 105.375 0.00523324 0.00674215 2024-09-13T14:44:15.781 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:15.802 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:15.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:15.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:15.903 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:15.940 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:15.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:15.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:16.042 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:16.063 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:16.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:16.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:16.165 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:16.200 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:16.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:16.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:16.303 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:16.324 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:16.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:16.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:16.425 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:16.460 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:16.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:16.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:16.561 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:16.604 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:16.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:16.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:16.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:16 smithi151 ceph-mon[104450]: pgmap v1492: 119 pgs: 119 active+clean; 1.7 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-09-13T14:44:16.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:16.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:16 smithi151 ceph-mon[108966]: pgmap v1492: 119 pgs: 119 active+clean; 1.7 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-09-13T14:44:16.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:16.707 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:16.721 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 15 16 34665 34649 144.346 92.8125 0.0397118 0.00691407 2024-09-13T14:44:16.737 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:16.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:16.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:16.839 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:16.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:16 smithi184 ceph-mon[92527]: pgmap v1492: 119 pgs: 119 active+clean; 1.7 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-09-13T14:44:16.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:16.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:16.874 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:16.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:16.975 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:17.006 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:17.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:17.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:17.108 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:17.136 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:17.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:17.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:17.238 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:17.269 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:17.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:17.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:17.371 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:17.392 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:17.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:17.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:17.494 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:17.526 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:17.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:17.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:17.627 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:17.661 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:17.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:17.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:17.722 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 16 16 36538 36522 142.64 117.062 0.00691098 0.00699925 2024-09-13T14:44:17.762 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:17.788 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:17.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:17.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:17.890 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:17.927 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:17.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:17.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:18.029 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:18.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:18.057 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:18.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:18.158 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:18.191 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:18.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:18.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:18.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:18.328 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:18.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:18.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:18.430 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:18.462 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:18.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:18.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:18.564 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:18.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:18.585 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:18.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:18.686 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:18.719 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:18.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:18.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:18.721 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 17 15 38260 38245 140.583 107.688 0.00363683 0.00710864 2024-09-13T14:44:18.821 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:18.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:18 smithi184 ceph-mon[92527]: pgmap v1493: 119 pgs: 119 active+clean; 1.9 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 149 MiB/s wr, 2.38k op/s 2024-09-13T14:44:18.855 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:18.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:18.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:18.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:18 smithi151 ceph-mon[104450]: pgmap v1493: 119 pgs: 119 active+clean; 1.9 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 149 MiB/s wr, 2.38k op/s 2024-09-13T14:44:18.863 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:18 smithi151 ceph-mon[108966]: pgmap v1493: 119 pgs: 119 active+clean; 1.9 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 149 MiB/s wr, 2.38k op/s 2024-09-13T14:44:18.957 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:18.996 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:18.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:18.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:19.098 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:19.135 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:19.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:19.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:19.237 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:19.262 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:19.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:19.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:19.364 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:19.396 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:19.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:19.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:19.498 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:19.532 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:19.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:19.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:19.634 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:19.659 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:19.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:19.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:19.722 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 18 16 40035 40019 138.932 110.875 0.00879188 0.00719273 2024-09-13T14:44:19.760 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:19.794 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:19.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:19.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:19.896 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:19.918 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:19.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:19.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:20.020 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:20.052 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:20.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:20.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:20.154 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:20.188 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:20.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:20.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:20.289 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:20.326 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:20.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:20.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:20.428 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:20.458 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:20.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:20.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:20.479 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:44:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:44:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:44:20.561 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:20.597 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T14:44:20.606 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:20.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:20.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:20.708 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:20.722 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 19 16 41773 41757 137.337 108.625 0.055741 0.00725849 2024-09-13T14:44:20.733 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:20.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:20.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:20.834 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:20.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:20 smithi184 ceph-mon[92527]: pgmap v1494: 119 pgs: 119 active+clean; 2.2 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-09-13T14:44:20.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:20 smithi151 ceph-mon[104450]: pgmap v1494: 119 pgs: 119 active+clean; 2.2 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-09-13T14:44:20.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:20 smithi151 ceph-mon[108966]: pgmap v1494: 119 pgs: 119 active+clean; 2.2 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-09-13T14:44:20.872 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:20.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:20.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:20.967 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:44:20.974 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:21.035 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:21.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:21.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:21.098 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:21.112 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:21.137 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:21.165 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:21.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:21.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:21.267 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:21.306 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:21.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:21.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:21.407 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:21.451 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:21.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:21.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:21.553 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:21.600 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:21.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:21.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:21.702 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:21.722 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:2024-09-13T14:44:21.722714+0000 min lat: 0.00109766 max lat: 0.193167 avg lat: 0.00734517 2024-09-13T14:44:21.722 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:44:21.722 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 20 16 43551 43535 136.023 111.125 0.0151002 0.00734517 2024-09-13T14:44:21.749 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:21.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:21.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:21.851 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:21.881 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:21.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:21.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:21.983 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:22.015 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:22.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:22.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:22.117 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:22.153 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:22.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:22.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:22.255 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:22.288 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:22.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:22.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:22.390 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:22.426 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:22.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:22.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:22.528 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:22.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:22.562 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:22.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:22.579 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:22 smithi184 ceph-mon[92527]: pgmap v1495: 119 pgs: 119 active+clean; 2.3 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-09-13T14:44:22.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:22 smithi151 ceph-mon[104450]: pgmap v1495: 119 pgs: 119 active+clean; 2.3 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-09-13T14:44:22.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:22 smithi151 ceph-mon[108966]: pgmap v1495: 119 pgs: 119 active+clean; 2.3 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-09-13T14:44:22.663 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:22.694 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:22.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:22.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:22.722 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 21 16 45481 45465 135.289 120.625 0.00685053 0.00738668 2024-09-13T14:44:22.796 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:22.832 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:22.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:22.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:22.891 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:44:22.891 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:44:22.934 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:22.963 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:22.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:22.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:23.033 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:44:23.065 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:23.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:23.129 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:23.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:23.230 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:23.273 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:23.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:23.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:23.375 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:23.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:23.407 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:23.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:23.507 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:23.540 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:23.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:23.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:23.575 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T14:44:23.642 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:23.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:23.694 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:23.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:23.723 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 22 16 47219 47203 134.077 108.625 0.0096603 0.00745342 2024-09-13T14:44:23.795 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:23.830 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:23.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:23.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:23.932 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:23.972 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:23.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:23.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:24.010 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:44:24.074 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:24.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:24.107 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:24.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:24.208 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:24.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:24.249 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:24.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:24.350 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:24.389 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:24.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:24.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:24.491 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:24.535 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:24.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:24.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:24.637 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:24.675 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:24.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:24.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:24.723 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 23 16 48934 48918 132.907 107.188 0.00953079 0.00751849 2024-09-13T14:44:24.777 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:24.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:24.818 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:24.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:24.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:24 smithi184 ceph-mon[92527]: pgmap v1496: 119 pgs: 119 active+clean; 2.6 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-09-13T14:44:24.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:24 smithi151 ceph-mon[104450]: pgmap v1496: 119 pgs: 119 active+clean; 2.6 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-09-13T14:44:24.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:44:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:44:24] "GET /metrics HTTP/1.1" 200 36644 "" "Prometheus/2.51.0" 2024-09-13T14:44:24.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:24 smithi151 ceph-mon[108966]: pgmap v1496: 119 pgs: 119 active+clean; 2.6 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-09-13T14:44:24.919 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:24.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:24.962 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:24.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:25.063 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:25.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:25.111 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:25.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:25.211 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:25.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:25.257 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:25.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:25.358 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:25.398 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:25.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:25.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:25.500 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:25.508 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-13T14:44:25.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:25.562 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:25.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:25.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:25 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3779468366' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:44:25.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:25 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3779468366' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:44:25.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:25 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3779468366' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:44:25.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:25.704 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:25.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:25.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:25.723 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 24 16 50495 50479 131.434 97.5625 0.00574848 0.0076029 2024-09-13T14:44:25.806 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:25.839 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:25.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:25.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:25.941 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:25.976 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:25.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:25.983 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:25.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:26.062 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:44:26.062 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:44:26.085 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:26.124 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:26.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:26.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:26.148 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:44:26.149 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T14:44:26.227 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:26.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:26.272 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:26.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:26.373 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:26.410 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:26.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:26.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:26.512 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:26.546 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:26.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:26.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:26.649 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:26.681 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:26.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:26.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:26.723 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 25 16 52106 52090 130.204 100.688 0.00837561 0.00767529 2024-09-13T14:44:26.783 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:26.794 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:26 smithi184 ceph-mon[92527]: pgmap v1497: 119 pgs: 119 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-09-13T14:44:26.794 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:26 smithi184 ceph-mon[92527]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:26.794 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:26 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3779468366' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:44:26.795 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:26 smithi184 ceph-mon[92527]: osdmap e1142: 8 total, 8 up, 8 in 2024-09-13T14:44:26.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:26.820 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:26.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:26.839 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T14:44:26.857 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:26 smithi151 ceph-mon[104450]: pgmap v1497: 119 pgs: 119 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-09-13T14:44:26.857 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:26 smithi151 ceph-mon[104450]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:26.857 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:26 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3779468366' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:44:26.857 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:26 smithi151 ceph-mon[104450]: osdmap e1142: 8 total, 8 up, 8 in 2024-09-13T14:44:26.857 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:26 smithi151 ceph-mon[108966]: pgmap v1497: 119 pgs: 119 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-09-13T14:44:26.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:26 smithi151 ceph-mon[108966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:26.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:26 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3779468366' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:44:26.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:26 smithi151 ceph-mon[108966]: osdmap e1142: 8 total, 8 up, 8 in 2024-09-13T14:44:26.922 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:26.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:26.955 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:26.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:27.056 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:27.087 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:27.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:27.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:27.189 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:27.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:27.230 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:27.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:27.276 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:44:27.331 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:27.365 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:27.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:27.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:27.468 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:27.502 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:27.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:27.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:27.600 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1498,"stamp":"2024-09-13T14:44:25.511895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2942231550,"num_objects":44911,"num_object_clones":0,"num_object_copies":134733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44911,"num_whiteouts":0,"num_read":104,"num_read_kb":262,"num_write":45001,"num_write_kb":2873239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":731149,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48043,"ondisk_log_size":48043,"up":357,"acting":357,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":358,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":10559804,"kb_used_data":8603384,"kb_used_omap":149,"kb_used_meta":1956202,"kb_avail":739401412,"statfs":{"total":767960285184,"available":757147045888,"internally_reserved":0,"allocated":8809865216,"data_stored":8782502127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":153401,"internal_metadata":2003151047},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1161625600,"num_objects":17725,"num_object_clones":0,"num_object_copies":53175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17725,"num_write_kb":1134400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.109309"},"pg_stats":[{"pgid":"1.1d","version":"261'100","reported_seq":44,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.848919+0000","last_change":"2024-09-13T14:43:17.988745+0000","last_active":"2024-09-13T14:44:03.848919+0000","last_peered":"2024-09-13T14:44:03.848919+0000","last_clean":"2024-09-13T14:44:03.848919+0000","last_became_active":"2024-09-13T14:43:14.557060+0000","last_became_peered":"2024-09-13T14:43:14.557060+0000","last_unstale":"2024-09-13T14:44:03.848919+0000","last_undegraded":"2024-09-13T14:44:03.848919+0000","last_fullsized":"2024-09-13T14:44:03.848919+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:17.988345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:17.988345+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:49:51.195679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":1656,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.850536+0000","last_change":"2024-09-13T14:29:42.541263+0000","last_active":"2024-09-13T14:44:03.850536+0000","last_peered":"2024-09-13T14:44:03.850536+0000","last_clean":"2024-09-13T14:44:03.850536+0000","last_became_active":"2024-09-13T14:29:42.540873+0000","last_became_peered":"2024-09-13T14:29:42.540873+0000","last_unstale":"2024-09-13T14:44:03.850536+0000","last_undegraded":"2024-09-13T14:44:03.850536+0000","last_fullsized":"2024-09-13T14:44:03.850536+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:20.611206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1101,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.845023+0000","last_change":"2024-09-13T14:43:07.516715+0000","last_active":"2024-09-13T14:44:02.845023+0000","last_peered":"2024-09-13T14:44:02.845023+0000","last_clean":"2024-09-13T14:44:02.845023+0000","last_became_active":"2024-09-13T14:43:07.516403+0000","last_became_peered":"2024-09-13T14:43:07.516403+0000","last_unstale":"2024-09-13T14:44:02.845023+0000","last_undegraded":"2024-09-13T14:44:02.845023+0000","last_fullsized":"2024-09-13T14:44:02.845023+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:56.469690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":31,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.846288+0000","last_change":"2024-09-13T14:43:32.634574+0000","last_active":"2024-09-13T14:44:03.846288+0000","last_peered":"2024-09-13T14:44:03.846288+0000","last_clean":"2024-09-13T14:44:03.846288+0000","last_became_active":"2024-09-13T14:43:32.634170+0000","last_became_peered":"2024-09-13T14:43:32.634170+0000","last_unstale":"2024-09-13T14:44:03.846288+0000","last_undegraded":"2024-09-13T14:44:03.846288+0000","last_fullsized":"2024-09-13T14:44:03.846288+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:19.416210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":718,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.846760+0000","last_change":"2024-09-13T14:39:11.396732+0000","last_active":"2024-09-13T14:44:03.846760+0000","last_peered":"2024-09-13T14:44:03.846760+0000","last_clean":"2024-09-13T14:44:03.846760+0000","last_became_active":"2024-09-13T14:39:11.396322+0000","last_became_peered":"2024-09-13T14:39:11.396322+0000","last_unstale":"2024-09-13T14:44:03.846760+0000","last_undegraded":"2024-09-13T14:44:03.846760+0000","last_fullsized":"2024-09-13T14:44:03.846760+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:54:47.496762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1c","version":"261'100","reported_seq":1202,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.839323+0000","last_change":"2024-09-13T14:43:14.564132+0000","last_active":"2024-09-13T14:44:02.839323+0000","last_peered":"2024-09-13T14:44:02.839323+0000","last_clean":"2024-09-13T14:44:02.839323+0000","last_became_active":"2024-09-13T14:43:14.563717+0000","last_became_peered":"2024-09-13T14:43:14.563717+0000","last_unstale":"2024-09-13T14:44:02.839323+0000","last_undegraded":"2024-09-13T14:44:02.839323+0000","last_fullsized":"2024-09-13T14:44:02.839323+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:22.580245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:22.580245+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8945,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":12,"num_write":3,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":8945,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":1069,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.844036+0000","last_change":"2024-09-13T14:19:25.873514+0000","last_active":"2024-09-13T14:44:02.844036+0000","last_peered":"2024-09-13T14:44:02.844036+0000","last_clean":"2024-09-13T14:44:02.844036+0000","last_became_active":"2024-09-13T14:19:25.873170+0000","last_became_peered":"2024-09-13T14:19:25.873170+0000","last_unstale":"2024-09-13T14:44:02.844036+0000","last_undegraded":"2024-09-13T14:44:02.844036+0000","last_fullsized":"2024-09-13T14:44:02.844036+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:48.298530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1098,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.845527+0000","last_change":"2024-09-13T14:30:15.580011+0000","last_active":"2024-09-13T14:44:02.845527+0000","last_peered":"2024-09-13T14:44:02.845527+0000","last_clean":"2024-09-13T14:44:02.845527+0000","last_became_active":"2024-09-13T14:30:15.579610+0000","last_became_peered":"2024-09-13T14:30:15.579610+0000","last_unstale":"2024-09-13T14:44:02.845527+0000","last_undegraded":"2024-09-13T14:44:02.845527+0000","last_fullsized":"2024-09-13T14:44:02.845527+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:11:02.733771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"261'100","reported_seq":436,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.847941+0000","last_change":"2024-09-13T14:43:41.569860+0000","last_active":"2024-09-13T14:44:02.847941+0000","last_peered":"2024-09-13T14:44:02.847941+0000","last_clean":"2024-09-13T14:44:02.847941+0000","last_became_active":"2024-09-13T14:43:14.564014+0000","last_became_peered":"2024-09-13T14:43:14.564014+0000","last_unstale":"2024-09-13T14:44:02.847941+0000","last_undegraded":"2024-09-13T14:44:02.847941+0000","last_fullsized":"2024-09-13T14:44:02.847941+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:41.569795+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:43:41.569795+0000","last_clean_scrub_stamp":"2024-09-13T14:43:41.569795+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:26.336048+0000","scrub_duration":0.001861749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":1114,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.843180+0000","last_change":"2024-09-13T14:43:07.536116+0000","last_active":"2024-09-13T14:44:02.843180+0000","last_peered":"2024-09-13T14:44:02.843180+0000","last_clean":"2024-09-13T14:44:02.843180+0000","last_became_active":"2024-09-13T14:43:07.524112+0000","last_became_peered":"2024-09-13T14:43:07.524112+0000","last_unstale":"2024-09-13T14:44:02.843180+0000","last_undegraded":"2024-09-13T14:44:02.843180+0000","last_fullsized":"2024-09-13T14:44:02.843180+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:36.947060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1111,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.846341+0000","last_change":"2024-09-13T14:43:07.530236+0000","last_active":"2024-09-13T14:44:02.846341+0000","last_peered":"2024-09-13T14:44:02.846341+0000","last_clean":"2024-09-13T14:44:02.846341+0000","last_became_active":"2024-09-13T14:43:07.529967+0000","last_became_peered":"2024-09-13T14:43:07.529967+0000","last_unstale":"2024-09-13T14:44:02.846341+0000","last_undegraded":"2024-09-13T14:44:02.846341+0000","last_fullsized":"2024-09-13T14:44:02.846341+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:27.276550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":1641,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.848645+0000","last_change":"2024-09-13T14:28:35.612469+0000","last_active":"2024-09-13T14:44:03.848645+0000","last_peered":"2024-09-13T14:44:03.848645+0000","last_clean":"2024-09-13T14:44:03.848645+0000","last_became_active":"2024-09-13T14:28:35.612093+0000","last_became_peered":"2024-09-13T14:28:35.612093+0000","last_unstale":"2024-09-13T14:44:03.848645+0000","last_undegraded":"2024-09-13T14:44:03.848645+0000","last_fullsized":"2024-09-13T14:44:03.848645+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:51.022663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1656,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.845749+0000","last_change":"2024-09-13T14:43:07.524473+0000","last_active":"2024-09-13T14:44:03.845749+0000","last_peered":"2024-09-13T14:44:03.845749+0000","last_clean":"2024-09-13T14:44:03.845749+0000","last_became_active":"2024-09-13T14:43:07.524162+0000","last_became_peered":"2024-09-13T14:43:07.524162+0000","last_unstale":"2024-09-13T14:44:03.845749+0000","last_undegraded":"2024-09-13T14:44:03.845749+0000","last_fullsized":"2024-09-13T14:44:03.845749+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:44:13.536401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"261'100","reported_seq":44,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.849444+0000","last_change":"2024-09-13T14:43:15.614393+0000","last_active":"2024-09-13T14:44:03.849444+0000","last_peered":"2024-09-13T14:44:03.849444+0000","last_clean":"2024-09-13T14:44:03.849444+0000","last_became_active":"2024-09-13T14:43:14.561088+0000","last_became_peered":"2024-09-13T14:43:14.561088+0000","last_unstale":"2024-09-13T14:44:03.849444+0000","last_undegraded":"2024-09-13T14:44:03.849444+0000","last_fullsized":"2024-09-13T14:44:03.849444+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.614340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.614340+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:10:32.000080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":1120,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.845614+0000","last_change":"2024-09-13T14:43:07.522950+0000","last_active":"2024-09-13T14:44:02.845614+0000","last_peered":"2024-09-13T14:44:02.845614+0000","last_clean":"2024-09-13T14:44:02.845614+0000","last_became_active":"2024-09-13T14:43:07.522566+0000","last_became_peered":"2024-09-13T14:43:07.522566+0000","last_unstale":"2024-09-13T14:44:02.845614+0000","last_undegraded":"2024-09-13T14:44:02.845614+0000","last_fullsized":"2024-09-13T14:44:02.845614+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:03.201478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":44,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.849131+0000","last_change":"2024-09-13T14:43:07.514011+0000","last_active":"2024-09-13T14:44:03.849131+0000","last_peered":"2024-09-13T14:44:03.849131+0000","last_clean":"2024-09-13T14:44:03.849131+0000","last_became_active":"2024-09-13T14:43:07.512756+0000","last_became_peered":"2024-09-13T14:43:07.512756+0000","last_unstale":"2024-09-13T14:44:03.849131+0000","last_undegraded":"2024-09-13T14:44:03.849131+0000","last_fullsized":"2024-09-13T14:44:03.849131+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:26.489194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"261'100","reported_seq":80,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.843023+0000","last_change":"2024-09-13T14:43:40.769005+0000","last_active":"2024-09-13T14:44:02.843023+0000","last_peered":"2024-09-13T14:44:02.843023+0000","last_clean":"2024-09-13T14:44:02.843023+0000","last_became_active":"2024-09-13T14:43:14.558502+0000","last_became_peered":"2024-09-13T14:43:14.558502+0000","last_unstale":"2024-09-13T14:44:02.843023+0000","last_undegraded":"2024-09-13T14:44:02.843023+0000","last_fullsized":"2024-09-13T14:44:02.843023+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1134,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:15:42.069756+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":420867,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":44,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.848045+0000","last_change":"2024-09-13T14:43:07.516845+0000","last_active":"2024-09-13T14:44:03.848045+0000","last_peered":"2024-09-13T14:44:03.848045+0000","last_clean":"2024-09-13T14:44:03.848045+0000","last_became_active":"2024-09-13T14:43:07.514974+0000","last_became_peered":"2024-09-13T14:43:07.514974+0000","last_unstale":"2024-09-13T14:44:03.848045+0000","last_undegraded":"2024-09-13T14:44:03.848045+0000","last_fullsized":"2024-09-13T14:44:03.848045+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:55.419483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1671,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.848710+0000","last_change":"2024-09-13T14:43:07.518461+0000","last_active":"2024-09-13T14:44:03.848710+0000","last_peered":"2024-09-13T14:44:03.848710+0000","last_clean":"2024-09-13T14:44:03.848710+0000","last_became_active":"2024-09-13T14:43:07.518161+0000","last_became_peered":"2024-09-13T14:43:07.518161+0000","last_unstale":"2024-09-13T14:44:03.848710+0000","last_undegraded":"2024-09-13T14:44:03.848710+0000","last_fullsized":"2024-09-13T14:44:03.848710+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:07.277688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1b","version":"261'100","reported_seq":22,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.843204+0000","last_change":"2024-09-13T14:43:14.557092+0000","last_active":"2024-09-13T14:44:02.843204+0000","last_peered":"2024-09-13T14:44:02.843204+0000","last_clean":"2024-09-13T14:44:02.843204+0000","last_became_active":"2024-09-13T14:43:14.556712+0000","last_became_peered":"2024-09-13T14:43:14.556712+0000","last_unstale":"2024-09-13T14:44:02.843204+0000","last_undegraded":"2024-09-13T14:44:02.843204+0000","last_fullsized":"2024-09-13T14:44:02.843204+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:26.747497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:26.747497+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8945,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":12,"num_write":3,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":8945,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":725,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.847771+0000","last_change":"2024-09-13T14:43:07.516804+0000","last_active":"2024-09-13T14:44:03.847771+0000","last_peered":"2024-09-13T14:44:03.847771+0000","last_clean":"2024-09-13T14:44:03.847771+0000","last_became_active":"2024-09-13T14:43:07.515841+0000","last_became_peered":"2024-09-13T14:43:07.515841+0000","last_unstale":"2024-09-13T14:44:03.847771+0000","last_undegraded":"2024-09-13T14:44:03.847771+0000","last_fullsized":"2024-09-13T14:44:03.847771+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:48:50.077012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1094,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.843366+0000","last_change":"2024-09-13T14:43:32.637751+0000","last_active":"2024-09-13T14:44:02.843366+0000","last_peered":"2024-09-13T14:44:02.843366+0000","last_clean":"2024-09-13T14:44:02.843366+0000","last_became_active":"2024-09-13T14:43:32.636901+0000","last_became_peered":"2024-09-13T14:43:32.636901+0000","last_unstale":"2024-09-13T14:44:02.843366+0000","last_undegraded":"2024-09-13T14:44:02.843366+0000","last_fullsized":"2024-09-13T14:44:02.843366+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:21.942419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1a","version":"261'100","reported_seq":32,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.845313+0000","last_change":"2024-09-13T14:43:41.910041+0000","last_active":"2024-09-13T14:44:02.845313+0000","last_peered":"2024-09-13T14:44:02.845313+0000","last_clean":"2024-09-13T14:44:02.845313+0000","last_became_active":"2024-09-13T14:43:14.555083+0000","last_became_peered":"2024-09-13T14:43:14.555083+0000","last_unstale":"2024-09-13T14:44:02.845313+0000","last_undegraded":"2024-09-13T14:44:02.845313+0000","last_fullsized":"2024-09-13T14:44:02.845313+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:41.909988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:41.909988+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:42:19.954838+0000","scrub_duration":0.0012891619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":44,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.847164+0000","last_change":"2024-09-13T14:43:07.519854+0000","last_active":"2024-09-13T14:44:03.847164+0000","last_peered":"2024-09-13T14:44:03.847164+0000","last_clean":"2024-09-13T14:44:03.847164+0000","last_became_active":"2024-09-13T14:43:07.519546+0000","last_became_peered":"2024-09-13T14:43:07.519546+0000","last_unstale":"2024-09-13T14:44:03.847164+0000","last_undegraded":"2024-09-13T14:44:03.847164+0000","last_fullsized":"2024-09-13T14:44:03.847164+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:14.770086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1689,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.850452+0000","last_change":"2024-09-13T14:39:11.386005+0000","last_active":"2024-09-13T14:44:03.850452+0000","last_peered":"2024-09-13T14:44:03.850452+0000","last_clean":"2024-09-13T14:44:03.850452+0000","last_became_active":"2024-09-13T14:39:11.385691+0000","last_became_peered":"2024-09-13T14:39:11.385691+0000","last_unstale":"2024-09-13T14:44:03.850452+0000","last_undegraded":"2024-09-13T14:44:03.850452+0000","last_fullsized":"2024-09-13T14:44:03.850452+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:11:03.371484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"261'100","reported_seq":22,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.839857+0000","last_change":"2024-09-13T14:43:14.558305+0000","last_active":"2024-09-13T14:44:02.839857+0000","last_peered":"2024-09-13T14:44:02.839857+0000","last_clean":"2024-09-13T14:44:02.839857+0000","last_became_active":"2024-09-13T14:43:14.557883+0000","last_became_peered":"2024-09-13T14:43:14.557883+0000","last_unstale":"2024-09-13T14:44:02.839857+0000","last_undegraded":"2024-09-13T14:44:02.839857+0000","last_fullsized":"2024-09-13T14:44:02.839857+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:22.892963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:22.892963+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":17,"num_write":4,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":13418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1567,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:04.054401+0000","last_change":"2024-09-13T14:30:15.576921+0000","last_active":"2024-09-13T14:44:04.054401+0000","last_peered":"2024-09-13T14:44:04.054401+0000","last_clean":"2024-09-13T14:44:04.054401+0000","last_became_active":"2024-09-13T14:30:15.576607+0000","last_became_peered":"2024-09-13T14:30:15.576607+0000","last_unstale":"2024-09-13T14:44:04.054401+0000","last_undegraded":"2024-09-13T14:44:04.054401+0000","last_fullsized":"2024-09-13T14:44:04.054401+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:35.162460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1104,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.842940+0000","last_change":"2024-09-13T14:43:07.518801+0000","last_active":"2024-09-13T14:44:02.842940+0000","last_peered":"2024-09-13T14:44:02.842940+0000","last_clean":"2024-09-13T14:44:02.842940+0000","last_became_active":"2024-09-13T14:43:07.517610+0000","last_became_peered":"2024-09-13T14:43:07.517610+0000","last_unstale":"2024-09-13T14:44:02.842940+0000","last_undegraded":"2024-09-13T14:44:02.842940+0000","last_fullsized":"2024-09-13T14:44:02.842940+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:51.632254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"261'100","reported_seq":44,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.845365+0000","last_change":"2024-09-13T14:43:15.037565+0000","last_active":"2024-09-13T14:44:03.845365+0000","last_peered":"2024-09-13T14:44:03.845365+0000","last_clean":"2024-09-13T14:44:03.845365+0000","last_became_active":"2024-09-13T14:43:14.554183+0000","last_became_peered":"2024-09-13T14:43:14.554183+0000","last_unstale":"2024-09-13T14:44:03.845365+0000","last_undegraded":"2024-09-13T14:44:03.845365+0000","last_fullsized":"2024-09-13T14:44:03.845365+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.037517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.037517+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:01:35.137195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":1098,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.839714+0000","last_change":"2024-09-13T14:30:15.579260+0000","last_active":"2024-09-13T14:44:02.839714+0000","last_peered":"2024-09-13T14:44:02.839714+0000","last_clean":"2024-09-13T14:44:02.839714+0000","last_became_active":"2024-09-13T14:30:15.578897+0000","last_became_peered":"2024-09-13T14:30:15.578897+0000","last_unstale":"2024-09-13T14:44:02.839714+0000","last_undegraded":"2024-09-13T14:44:02.839714+0000","last_fullsized":"2024-09-13T14:44:02.839714+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:56.737849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":44,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.852708+0000","last_change":"2024-09-13T14:43:07.521576+0000","last_active":"2024-09-13T14:44:03.852708+0000","last_peered":"2024-09-13T14:44:03.852708+0000","last_clean":"2024-09-13T14:44:03.852708+0000","last_became_active":"2024-09-13T14:43:07.520478+0000","last_became_peered":"2024-09-13T14:43:07.520478+0000","last_unstale":"2024-09-13T14:44:03.852708+0000","last_undegraded":"2024-09-13T14:44:03.852708+0000","last_fullsized":"2024-09-13T14:44:03.852708+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:53.410135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"261'100","reported_seq":348,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.843145+0000","last_change":"2024-09-13T14:43:14.560650+0000","last_active":"2024-09-13T14:44:02.843145+0000","last_peered":"2024-09-13T14:44:02.843145+0000","last_clean":"2024-09-13T14:44:02.843145+0000","last_became_active":"2024-09-13T14:43:14.560297+0000","last_became_peered":"2024-09-13T14:43:14.560297+0000","last_unstale":"2024-09-13T14:44:02.843145+0000","last_undegraded":"2024-09-13T14:44:02.843145+0000","last_fullsized":"2024-09-13T14:44:02.843145+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:15:42.069756+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":17,"num_write":3,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":13418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":1087,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.846271+0000","last_change":"2024-09-13T14:19:25.870605+0000","last_active":"2024-09-13T14:44:02.846271+0000","last_peered":"2024-09-13T14:44:02.846271+0000","last_clean":"2024-09-13T14:44:02.846271+0000","last_became_active":"2024-09-13T14:19:25.870282+0000","last_became_peered":"2024-09-13T14:19:25.870282+0000","last_unstale":"2024-09-13T14:44:02.846271+0000","last_undegraded":"2024-09-13T14:44:02.846271+0000","last_fullsized":"2024-09-13T14:44:02.846271+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:52.514915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1101,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.846166+0000","last_change":"2024-09-13T14:43:07.526476+0000","last_active":"2024-09-13T14:44:02.846166+0000","last_peered":"2024-09-13T14:44:02.846166+0000","last_clean":"2024-09-13T14:44:02.846166+0000","last_became_active":"2024-09-13T14:43:07.525802+0000","last_became_peered":"2024-09-13T14:43:07.525802+0000","last_unstale":"2024-09-13T14:44:02.846166+0000","last_undegraded":"2024-09-13T14:44:02.846166+0000","last_fullsized":"2024-09-13T14:44:02.846166+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:23.154726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"261'100","reported_seq":44,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.849067+0000","last_change":"2024-09-13T14:43:16.955257+0000","last_active":"2024-09-13T14:44:03.849067+0000","last_peered":"2024-09-13T14:44:03.849067+0000","last_clean":"2024-09-13T14:44:03.849067+0000","last_became_active":"2024-09-13T14:43:14.555885+0000","last_became_peered":"2024-09-13T14:43:14.555885+0000","last_unstale":"2024-09-13T14:44:03.849067+0000","last_undegraded":"2024-09-13T14:44:03.849067+0000","last_fullsized":"2024-09-13T14:44:03.849067+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.955197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.955197+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:36.334338+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"1141'2734","reported_seq":2754,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:22.114668+0000","last_change":"2024-09-13T14:43:50.792917+0000","last_active":"2024-09-13T14:44:22.114668+0000","last_peered":"2024-09-13T14:44:22.114668+0000","last_clean":"2024-09-13T14:44:22.114668+0000","last_became_active":"2024-09-13T14:43:50.792608+0000","last_became_peered":"2024-09-13T14:43:50.792608+0000","last_unstale":"2024-09-13T14:44:22.114668+0000","last_undegraded":"2024-09-13T14:44:22.114668+0000","last_fullsized":"2024-09-13T14:44:22.114668+0000","mapping_epoch":1136,"log_start":"0'0","ondisk_log_start":"0'0","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":2734,"log_dups_size":0,"ondisk_log_size":2734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:06.251060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179175424,"num_objects":2734,"num_object_clones":0,"num_object_copies":8202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2734,"num_write_kb":174976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1064,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.845233+0000","last_change":"2024-09-13T14:19:26.131609+0000","last_active":"2024-09-13T14:44:02.845233+0000","last_peered":"2024-09-13T14:44:02.845233+0000","last_clean":"2024-09-13T14:44:02.845233+0000","last_became_active":"2024-09-13T14:19:26.131029+0000","last_became_peered":"2024-09-13T14:19:26.131029+0000","last_unstale":"2024-09-13T14:44:02.845233+0000","last_undegraded":"2024-09-13T14:44:02.845233+0000","last_fullsized":"2024-09-13T14:44:02.845233+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:57.933696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":566,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.839540+0000","last_change":"2024-09-13T14:43:32.635070+0000","last_active":"2024-09-13T14:44:02.839540+0000","last_peered":"2024-09-13T14:44:02.839540+0000","last_clean":"2024-09-13T14:44:02.839540+0000","last_became_active":"2024-09-13T14:43:32.634734+0000","last_became_peered":"2024-09-13T14:43:32.634734+0000","last_unstale":"2024-09-13T14:44:02.839540+0000","last_undegraded":"2024-09-13T14:44:02.839540+0000","last_fullsized":"2024-09-13T14:44:02.839540+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:33.107010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":1686,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.848908+0000","last_change":"2024-09-13T14:43:07.537759+0000","last_active":"2024-09-13T14:44:03.848908+0000","last_peered":"2024-09-13T14:44:03.848908+0000","last_clean":"2024-09-13T14:44:03.848908+0000","last_became_active":"2024-09-13T14:43:07.520303+0000","last_became_peered":"2024-09-13T14:43:07.520303+0000","last_unstale":"2024-09-13T14:44:03.848908+0000","last_undegraded":"2024-09-13T14:44:03.848908+0000","last_fullsized":"2024-09-13T14:44:03.848908+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:32.772570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"261'100","reported_seq":75,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.842944+0000","last_change":"2024-09-13T14:42:14.948141+0000","last_active":"2024-09-13T14:44:02.842944+0000","last_peered":"2024-09-13T14:44:02.842944+0000","last_clean":"2024-09-13T14:44:02.842944+0000","last_became_active":"2024-09-13T14:42:14.947797+0000","last_became_peered":"2024-09-13T14:42:14.947797+0000","last_unstale":"2024-09-13T14:44:02.842944+0000","last_undegraded":"2024-09-13T14:44:02.842944+0000","last_fullsized":"2024-09-13T14:44:02.842944+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:22.580245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:22.580245+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001821837,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109226,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"1141'2564","reported_seq":2585,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:20.787083+0000","last_change":"2024-09-13T14:43:51.536874+0000","last_active":"2024-09-13T14:44:20.787083+0000","last_peered":"2024-09-13T14:44:20.787083+0000","last_clean":"2024-09-13T14:44:20.787083+0000","last_became_active":"2024-09-13T14:43:51.536457+0000","last_became_peered":"2024-09-13T14:43:51.536457+0000","last_unstale":"2024-09-13T14:44:20.787083+0000","last_undegraded":"2024-09-13T14:44:20.787083+0000","last_fullsized":"2024-09-13T14:44:20.787083+0000","mapping_epoch":1136,"log_start":"0'0","ondisk_log_start":"0'0","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":2564,"log_dups_size":0,"ondisk_log_size":2564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:38:10.101672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168034304,"num_objects":2564,"num_object_clones":0,"num_object_copies":7692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2564,"num_write_kb":164096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":658,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.838442+0000","last_change":"2024-09-13T14:43:07.505178+0000","last_active":"2024-09-13T14:44:02.838442+0000","last_peered":"2024-09-13T14:44:02.838442+0000","last_clean":"2024-09-13T14:44:02.838442+0000","last_became_active":"2024-09-13T14:43:07.504308+0000","last_became_peered":"2024-09-13T14:43:07.504308+0000","last_unstale":"2024-09-13T14:44:02.838442+0000","last_undegraded":"2024-09-13T14:44:02.838442+0000","last_fullsized":"2024-09-13T14:44:02.838442+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:48:11.778576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":44,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.849574+0000","last_change":"2024-09-13T14:43:07.522730+0000","last_active":"2024-09-13T14:44:03.849574+0000","last_peered":"2024-09-13T14:44:03.849574+0000","last_clean":"2024-09-13T14:44:03.849574+0000","last_became_active":"2024-09-13T14:43:07.517420+0000","last_became_peered":"2024-09-13T14:43:07.517420+0000","last_unstale":"2024-09-13T14:44:03.849574+0000","last_undegraded":"2024-09-13T14:44:03.849574+0000","last_fullsized":"2024-09-13T14:44:03.849574+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:21:30.265293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":1599,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:04.055000+0000","last_change":"2024-09-13T14:30:15.581728+0000","last_active":"2024-09-13T14:44:04.055000+0000","last_peered":"2024-09-13T14:44:04.055000+0000","last_clean":"2024-09-13T14:44:04.055000+0000","last_became_active":"2024-09-13T14:30:15.581303+0000","last_became_peered":"2024-09-13T14:30:15.581303+0000","last_unstale":"2024-09-13T14:44:04.055000+0000","last_undegraded":"2024-09-13T14:44:04.055000+0000","last_fullsized":"2024-09-13T14:44:04.055000+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:10.027991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"261'100","reported_seq":91,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.845812+0000","last_change":"2024-09-13T14:43:31.112173+0000","last_active":"2024-09-13T14:44:02.845812+0000","last_peered":"2024-09-13T14:44:02.845812+0000","last_clean":"2024-09-13T14:44:02.845812+0000","last_became_active":"2024-09-13T14:42:42.709263+0000","last_became_peered":"2024-09-13T14:42:42.709263+0000","last_unstale":"2024-09-13T14:44:02.845812+0000","last_undegraded":"2024-09-13T14:44:02.845812+0000","last_fullsized":"2024-09-13T14:44:02.845812+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:31.112111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:31.112111+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:02.707372+0000","scrub_duration":0.0015759260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"1141'2845","reported_seq":2858,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:22.638239+0000","last_change":"2024-09-13T14:43:50.798770+0000","last_active":"2024-09-13T14:44:22.638239+0000","last_peered":"2024-09-13T14:44:22.638239+0000","last_clean":"2024-09-13T14:44:22.638239+0000","last_became_active":"2024-09-13T14:43:50.798485+0000","last_became_peered":"2024-09-13T14:43:50.798485+0000","last_unstale":"2024-09-13T14:44:22.638239+0000","last_undegraded":"2024-09-13T14:44:22.638239+0000","last_fullsized":"2024-09-13T14:44:22.638239+0000","mapping_epoch":1136,"log_start":"0'0","ondisk_log_start":"0'0","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":2845,"log_dups_size":0,"ondisk_log_size":2845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:17:46.879168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186449920,"num_objects":2845,"num_object_clones":0,"num_object_copies":8535,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2845,"num_write_kb":182080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1639,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.848534+0000","last_change":"2024-09-13T14:28:35.611508+0000","last_active":"2024-09-13T14:44:03.848534+0000","last_peered":"2024-09-13T14:44:03.848534+0000","last_clean":"2024-09-13T14:44:03.848534+0000","last_became_active":"2024-09-13T14:28:35.610106+0000","last_became_peered":"2024-09-13T14:28:35.610106+0000","last_unstale":"2024-09-13T14:44:03.848534+0000","last_undegraded":"2024-09-13T14:44:03.848534+0000","last_fullsized":"2024-09-13T14:44:03.848534+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:11.966958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1107,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.839867+0000","last_change":"2024-09-13T14:28:35.603663+0000","last_active":"2024-09-13T14:44:02.839867+0000","last_peered":"2024-09-13T14:44:02.839867+0000","last_clean":"2024-09-13T14:44:02.839867+0000","last_became_active":"2024-09-13T14:28:35.603358+0000","last_became_peered":"2024-09-13T14:28:35.603358+0000","last_unstale":"2024-09-13T14:44:02.839867+0000","last_undegraded":"2024-09-13T14:44:02.839867+0000","last_fullsized":"2024-09-13T14:44:02.839867+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:24:05.842796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":44,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.847966+0000","last_change":"2024-09-13T14:43:07.522750+0000","last_active":"2024-09-13T14:44:03.847966+0000","last_peered":"2024-09-13T14:44:03.847966+0000","last_clean":"2024-09-13T14:44:03.847966+0000","last_became_active":"2024-09-13T14:43:07.517267+0000","last_became_peered":"2024-09-13T14:43:07.517267+0000","last_unstale":"2024-09-13T14:44:03.847966+0000","last_undegraded":"2024-09-13T14:44:03.847966+0000","last_fullsized":"2024-09-13T14:44:03.847966+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:40.217748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"261'100","reported_seq":358,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.848538+0000","last_change":"2024-09-13T14:43:33.574489+0000","last_active":"2024-09-13T14:44:03.848538+0000","last_peered":"2024-09-13T14:44:03.848538+0000","last_clean":"2024-09-13T14:44:03.848538+0000","last_became_active":"2024-09-13T14:43:32.634789+0000","last_became_peered":"2024-09-13T14:43:32.634789+0000","last_unstale":"2024-09-13T14:44:03.848538+0000","last_undegraded":"2024-09-13T14:44:03.848538+0000","last_fullsized":"2024-09-13T14:44:03.848538+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:33.574443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:33.574443+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:49.621231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"1141'2875","reported_seq":2888,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:22.382210+0000","last_change":"2024-09-13T14:43:51.534631+0000","last_active":"2024-09-13T14:44:22.382210+0000","last_peered":"2024-09-13T14:44:22.382210+0000","last_clean":"2024-09-13T14:44:22.382210+0000","last_became_active":"2024-09-13T14:43:51.534232+0000","last_became_peered":"2024-09-13T14:43:51.534232+0000","last_unstale":"2024-09-13T14:44:22.382210+0000","last_undegraded":"2024-09-13T14:44:22.382210+0000","last_fullsized":"2024-09-13T14:44:22.382210+0000","mapping_epoch":1136,"log_start":"0'0","ondisk_log_start":"0'0","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":2875,"log_dups_size":0,"ondisk_log_size":2875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:08:20.221222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188416000,"num_objects":2875,"num_object_clones":0,"num_object_copies":8625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2875,"num_write_kb":184000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":44,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.852572+0000","last_change":"2024-09-13T14:43:07.520049+0000","last_active":"2024-09-13T14:44:03.852572+0000","last_peered":"2024-09-13T14:44:03.852572+0000","last_clean":"2024-09-13T14:44:03.852572+0000","last_became_active":"2024-09-13T14:43:07.514646+0000","last_became_peered":"2024-09-13T14:43:07.514646+0000","last_unstale":"2024-09-13T14:44:03.852572+0000","last_undegraded":"2024-09-13T14:44:03.852572+0000","last_fullsized":"2024-09-13T14:44:03.852572+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:10:35.200778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":44,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.847872+0000","last_change":"2024-09-13T14:43:07.525666+0000","last_active":"2024-09-13T14:44:03.847872+0000","last_peered":"2024-09-13T14:44:03.847872+0000","last_clean":"2024-09-13T14:44:03.847872+0000","last_became_active":"2024-09-13T14:43:07.524064+0000","last_became_peered":"2024-09-13T14:43:07.524064+0000","last_unstale":"2024-09-13T14:44:03.847872+0000","last_undegraded":"2024-09-13T14:44:03.847872+0000","last_fullsized":"2024-09-13T14:44:03.847872+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:10.270333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1111,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.844079+0000","last_change":"2024-09-13T14:39:11.383045+0000","last_active":"2024-09-13T14:44:02.844079+0000","last_peered":"2024-09-13T14:44:02.844079+0000","last_clean":"2024-09-13T14:44:02.844079+0000","last_became_active":"2024-09-13T14:39:11.382718+0000","last_became_peered":"2024-09-13T14:39:11.382718+0000","last_unstale":"2024-09-13T14:44:02.844079+0000","last_undegraded":"2024-09-13T14:44:02.844079+0000","last_fullsized":"2024-09-13T14:44:02.844079+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:21.812296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"261'100","reported_seq":1199,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.838911+0000","last_change":"2024-09-13T14:43:32.635099+0000","last_active":"2024-09-13T14:44:02.838911+0000","last_peered":"2024-09-13T14:44:02.838911+0000","last_clean":"2024-09-13T14:44:02.838911+0000","last_became_active":"2024-09-13T14:43:32.634755+0000","last_became_peered":"2024-09-13T14:43:32.634755+0000","last_unstale":"2024-09-13T14:44:02.838911+0000","last_undegraded":"2024-09-13T14:44:02.838911+0000","last_fullsized":"2024-09-13T14:44:02.838911+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:15:42.069756+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13417,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":17,"num_write":3,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":13417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"1141'2995","reported_seq":3016,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:23.941050+0000","last_change":"2024-09-13T14:43:51.536695+0000","last_active":"2024-09-13T14:44:23.941050+0000","last_peered":"2024-09-13T14:44:23.941050+0000","last_clean":"2024-09-13T14:44:23.941050+0000","last_became_active":"2024-09-13T14:43:51.536393+0000","last_became_peered":"2024-09-13T14:43:51.536393+0000","last_unstale":"2024-09-13T14:44:23.941050+0000","last_undegraded":"2024-09-13T14:44:23.941050+0000","last_fullsized":"2024-09-13T14:44:23.941050+0000","mapping_epoch":1136,"log_start":"0'0","ondisk_log_start":"0'0","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":2995,"log_dups_size":0,"ondisk_log_size":2995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:32:34.644191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196280320,"num_objects":2995,"num_object_clones":0,"num_object_copies":8985,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2995,"num_write_kb":191680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.851615+0000","last_change":"2024-09-13T14:44:03.851615+0000","last_active":"2024-09-13T14:44:03.851615+0000","last_peered":"2024-09-13T14:44:03.851615+0000","last_clean":"2024-09-13T14:44:03.851615+0000","last_became_active":"2024-09-13T14:44:03.851363+0000","last_became_peered":"2024-09-13T14:44:03.851363+0000","last_unstale":"2024-09-13T14:44:03.851615+0000","last_undegraded":"2024-09-13T14:44:03.851615+0000","last_fullsized":"2024-09-13T14:44:03.851615+0000","mapping_epoch":1140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:14:06.692121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1082,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.846315+0000","last_change":"2024-09-13T14:28:35.604404+0000","last_active":"2024-09-13T14:44:02.846315+0000","last_peered":"2024-09-13T14:44:02.846315+0000","last_clean":"2024-09-13T14:44:02.846315+0000","last_became_active":"2024-09-13T14:28:35.604102+0000","last_became_peered":"2024-09-13T14:28:35.604102+0000","last_unstale":"2024-09-13T14:44:02.846315+0000","last_undegraded":"2024-09-13T14:44:02.846315+0000","last_fullsized":"2024-09-13T14:44:02.846315+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:25.546010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":1132,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.842971+0000","last_change":"2024-09-13T14:43:07.538919+0000","last_active":"2024-09-13T14:44:02.842971+0000","last_peered":"2024-09-13T14:44:02.842971+0000","last_clean":"2024-09-13T14:44:02.842971+0000","last_became_active":"2024-09-13T14:43:07.513109+0000","last_became_peered":"2024-09-13T14:43:07.513109+0000","last_unstale":"2024-09-13T14:44:02.842971+0000","last_undegraded":"2024-09-13T14:44:02.842971+0000","last_fullsized":"2024-09-13T14:44:02.842971+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:27.771720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"261'100","reported_seq":81,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.842694+0000","last_change":"2024-09-13T14:43:44.720333+0000","last_active":"2024-09-13T14:44:02.842694+0000","last_peered":"2024-09-13T14:44:02.842694+0000","last_clean":"2024-09-13T14:44:02.842694+0000","last_became_active":"2024-09-13T14:42:14.946591+0000","last_became_peered":"2024-09-13T14:42:14.946591+0000","last_unstale":"2024-09-13T14:44:02.842694+0000","last_undegraded":"2024-09-13T14:44:02.842694+0000","last_fullsized":"2024-09-13T14:44:02.842694+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:44.720258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:44.720258+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:33.281357+0000","scrub_duration":0.0018417920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"1141'2962","reported_seq":2983,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:23.941117+0000","last_change":"2024-09-13T14:43:50.794091+0000","last_active":"2024-09-13T14:44:23.941117+0000","last_peered":"2024-09-13T14:44:23.941117+0000","last_clean":"2024-09-13T14:44:23.941117+0000","last_became_active":"2024-09-13T14:43:50.793741+0000","last_became_peered":"2024-09-13T14:43:50.793741+0000","last_unstale":"2024-09-13T14:44:23.941117+0000","last_undegraded":"2024-09-13T14:44:23.941117+0000","last_fullsized":"2024-09-13T14:44:23.941117+0000","mapping_epoch":1136,"log_start":"0'0","ondisk_log_start":"0'0","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":2962,"log_dups_size":0,"ondisk_log_size":2962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:15.846494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194117632,"num_objects":2962,"num_object_clones":0,"num_object_copies":8886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2962,"num_write_kb":189568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.859349+0000","last_change":"2024-09-13T14:44:03.858211+0000","last_active":"2024-09-13T14:44:03.859349+0000","last_peered":"2024-09-13T14:44:03.859349+0000","last_clean":"2024-09-13T14:44:03.859349+0000","last_became_active":"2024-09-13T14:44:03.857928+0000","last_became_peered":"2024-09-13T14:44:03.857928+0000","last_unstale":"2024-09-13T14:44:03.859349+0000","last_undegraded":"2024-09-13T14:44:03.859349+0000","last_fullsized":"2024-09-13T14:44:03.859349+0000","mapping_epoch":1140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:28.428693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":44,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.846901+0000","last_change":"2024-09-13T14:43:07.516243+0000","last_active":"2024-09-13T14:44:03.846901+0000","last_peered":"2024-09-13T14:44:03.846901+0000","last_clean":"2024-09-13T14:44:03.846901+0000","last_became_active":"2024-09-13T14:43:07.515863+0000","last_became_peered":"2024-09-13T14:43:07.515863+0000","last_unstale":"2024-09-13T14:44:03.846901+0000","last_undegraded":"2024-09-13T14:44:03.846901+0000","last_fullsized":"2024-09-13T14:44:03.846901+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:08.085298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1674,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.846900+0000","last_change":"2024-09-13T14:43:07.513602+0000","last_active":"2024-09-13T14:44:03.846900+0000","last_peered":"2024-09-13T14:44:03.846900+0000","last_clean":"2024-09-13T14:44:03.846900+0000","last_became_active":"2024-09-13T14:43:07.513239+0000","last_became_peered":"2024-09-13T14:43:07.513239+0000","last_unstale":"2024-09-13T14:44:03.846900+0000","last_undegraded":"2024-09-13T14:44:03.846900+0000","last_fullsized":"2024-09-13T14:44:03.846900+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:42.345882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"261'100","reported_seq":1261,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.838778+0000","last_change":"2024-09-13T14:43:42.577312+0000","last_active":"2024-09-13T14:44:02.838778+0000","last_peered":"2024-09-13T14:44:02.838778+0000","last_clean":"2024-09-13T14:44:02.838778+0000","last_became_active":"2024-09-13T14:42:14.928923+0000","last_became_peered":"2024-09-13T14:42:14.928923+0000","last_unstale":"2024-09-13T14:44:02.838778+0000","last_undegraded":"2024-09-13T14:44:02.838778+0000","last_fullsized":"2024-09-13T14:44:02.838778+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:42.577243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:42.577243+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:01.234250+0000","scrub_duration":0.001367839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"1141'2841","reported_seq":2854,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:22.492889+0000","last_change":"2024-09-13T14:43:50.793810+0000","last_active":"2024-09-13T14:44:22.492889+0000","last_peered":"2024-09-13T14:44:22.492889+0000","last_clean":"2024-09-13T14:44:22.492889+0000","last_became_active":"2024-09-13T14:43:50.793506+0000","last_became_peered":"2024-09-13T14:43:50.793506+0000","last_unstale":"2024-09-13T14:44:22.492889+0000","last_undegraded":"2024-09-13T14:44:22.492889+0000","last_fullsized":"2024-09-13T14:44:22.492889+0000","mapping_epoch":1136,"log_start":"0'0","ondisk_log_start":"0'0","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":2841,"log_dups_size":0,"ondisk_log_size":2841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:50:32.061762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186187776,"num_objects":2841,"num_object_clones":0,"num_object_copies":8523,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2841,"num_write_kb":181824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1675,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.848629+0000","last_change":"2024-09-13T14:43:07.520212+0000","last_active":"2024-09-13T14:44:03.848629+0000","last_peered":"2024-09-13T14:44:03.848629+0000","last_clean":"2024-09-13T14:44:03.848629+0000","last_became_active":"2024-09-13T14:43:07.519831+0000","last_became_peered":"2024-09-13T14:43:07.519831+0000","last_unstale":"2024-09-13T14:44:03.848629+0000","last_undegraded":"2024-09-13T14:44:03.848629+0000","last_fullsized":"2024-09-13T14:44:03.848629+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:20.990736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1690,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.849599+0000","last_change":"2024-09-13T14:39:11.390055+0000","last_active":"2024-09-13T14:44:03.849599+0000","last_peered":"2024-09-13T14:44:03.849599+0000","last_clean":"2024-09-13T14:44:03.849599+0000","last_became_active":"2024-09-13T14:39:11.389697+0000","last_became_peered":"2024-09-13T14:39:11.389697+0000","last_unstale":"2024-09-13T14:44:03.849599+0000","last_undegraded":"2024-09-13T14:44:03.849599+0000","last_fullsized":"2024-09-13T14:44:03.849599+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:50.975739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1144,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.838619+0000","last_change":"2024-09-13T14:43:07.516140+0000","last_active":"2024-09-13T14:44:02.838619+0000","last_peered":"2024-09-13T14:44:02.838619+0000","last_clean":"2024-09-13T14:44:02.838619+0000","last_became_active":"2024-09-13T14:43:07.515481+0000","last_became_peered":"2024-09-13T14:43:07.515481+0000","last_unstale":"2024-09-13T14:44:02.838619+0000","last_undegraded":"2024-09-13T14:44:02.838619+0000","last_fullsized":"2024-09-13T14:44:02.838619+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:43.706214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"261'100","reported_seq":56,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.850120+0000","last_change":"2024-09-13T14:43:16.008346+0000","last_active":"2024-09-13T14:44:03.850120+0000","last_peered":"2024-09-13T14:44:03.850120+0000","last_clean":"2024-09-13T14:44:03.850120+0000","last_became_active":"2024-09-13T14:43:07.523121+0000","last_became_peered":"2024-09-13T14:43:07.523121+0000","last_unstale":"2024-09-13T14:44:03.850120+0000","last_undegraded":"2024-09-13T14:44:03.850120+0000","last_fullsized":"2024-09-13T14:44:03.850120+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.008264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.008264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:58:35.564470+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"1141'2914","reported_seq":2935,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:23.905612+0000","last_change":"2024-09-13T14:43:50.793299+0000","last_active":"2024-09-13T14:44:23.905612+0000","last_peered":"2024-09-13T14:44:23.905612+0000","last_clean":"2024-09-13T14:44:23.905612+0000","last_became_active":"2024-09-13T14:43:50.792776+0000","last_became_peered":"2024-09-13T14:43:50.792776+0000","last_unstale":"2024-09-13T14:44:23.905612+0000","last_undegraded":"2024-09-13T14:44:23.905612+0000","last_fullsized":"2024-09-13T14:44:23.905612+0000","mapping_epoch":1136,"log_start":"0'0","ondisk_log_start":"0'0","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":2914,"log_dups_size":0,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:08:05.139788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190971904,"num_objects":2914,"num_object_clones":0,"num_object_copies":8742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":186496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":23,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.845732+0000","last_change":"2024-09-13T14:43:32.636990+0000","last_active":"2024-09-13T14:44:02.845732+0000","last_peered":"2024-09-13T14:44:02.845732+0000","last_clean":"2024-09-13T14:44:02.845732+0000","last_became_active":"2024-09-13T14:43:32.636589+0000","last_became_peered":"2024-09-13T14:43:32.636589+0000","last_unstale":"2024-09-13T14:44:02.845732+0000","last_undegraded":"2024-09-13T14:44:02.845732+0000","last_fullsized":"2024-09-13T14:44:02.845732+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:36:25.129769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":608,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.842760+0000","last_change":"2024-09-13T14:39:11.382555+0000","last_active":"2024-09-13T14:44:02.842760+0000","last_peered":"2024-09-13T14:44:02.842760+0000","last_clean":"2024-09-13T14:44:02.842760+0000","last_became_active":"2024-09-13T14:39:11.382247+0000","last_became_peered":"2024-09-13T14:39:11.382247+0000","last_unstale":"2024-09-13T14:44:02.842760+0000","last_undegraded":"2024-09-13T14:44:02.842760+0000","last_fullsized":"2024-09-13T14:44:02.842760+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:09:48.016987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":1097,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.843320+0000","last_change":"2024-09-13T14:29:42.539455+0000","last_active":"2024-09-13T14:44:02.843320+0000","last_peered":"2024-09-13T14:44:02.843320+0000","last_clean":"2024-09-13T14:44:02.843320+0000","last_became_active":"2024-09-13T14:29:42.539068+0000","last_became_peered":"2024-09-13T14:29:42.539068+0000","last_unstale":"2024-09-13T14:44:02.843320+0000","last_undegraded":"2024-09-13T14:44:02.843320+0000","last_fullsized":"2024-09-13T14:44:02.843320+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:24:06.048855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034405999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"261'100","reported_seq":1242,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.838712+0000","last_change":"2024-09-13T14:42:14.928093+0000","last_active":"2024-09-13T14:44:02.838712+0000","last_peered":"2024-09-13T14:44:02.838712+0000","last_clean":"2024-09-13T14:44:02.838712+0000","last_became_active":"2024-09-13T14:42:14.927771+0000","last_became_peered":"2024-09-13T14:42:14.927771+0000","last_unstale":"2024-09-13T14:44:02.838712+0000","last_undegraded":"2024-09-13T14:44:02.838712+0000","last_fullsized":"2024-09-13T14:44:02.838712+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:23.257279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:23.257279+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010350875000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"1141'2783","reported_seq":2793,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:22.500095+0000","last_change":"2024-09-13T14:43:51.534910+0000","last_active":"2024-09-13T14:44:22.500095+0000","last_peered":"2024-09-13T14:44:22.500095+0000","last_clean":"2024-09-13T14:44:22.500095+0000","last_became_active":"2024-09-13T14:43:51.534597+0000","last_became_peered":"2024-09-13T14:43:51.534597+0000","last_unstale":"2024-09-13T14:44:22.500095+0000","last_undegraded":"2024-09-13T14:44:22.500095+0000","last_fullsized":"2024-09-13T14:44:22.500095+0000","mapping_epoch":1136,"log_start":"0'0","ondisk_log_start":"0'0","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":2783,"log_dups_size":0,"ondisk_log_size":2783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:46.270886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182386688,"num_objects":2783,"num_object_clones":0,"num_object_copies":8349,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2783,"num_write_kb":178112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":44,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.847713+0000","last_change":"2024-09-13T14:43:07.520176+0000","last_active":"2024-09-13T14:44:03.847713+0000","last_peered":"2024-09-13T14:44:03.847713+0000","last_clean":"2024-09-13T14:44:03.847713+0000","last_became_active":"2024-09-13T14:43:07.516748+0000","last_became_peered":"2024-09-13T14:43:07.516748+0000","last_unstale":"2024-09-13T14:44:03.847713+0000","last_undegraded":"2024-09-13T14:44:03.847713+0000","last_fullsized":"2024-09-13T14:44:03.847713+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:52:04.392821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":44,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.847010+0000","last_change":"2024-09-13T14:43:07.513888+0000","last_active":"2024-09-13T14:44:03.847010+0000","last_peered":"2024-09-13T14:44:03.847010+0000","last_clean":"2024-09-13T14:44:03.847010+0000","last_became_active":"2024-09-13T14:43:07.512719+0000","last_became_peered":"2024-09-13T14:43:07.512719+0000","last_unstale":"2024-09-13T14:44:03.847010+0000","last_undegraded":"2024-09-13T14:44:03.847010+0000","last_fullsized":"2024-09-13T14:44:03.847010+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:40.669613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"261'100","reported_seq":71,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.842704+0000","last_change":"2024-09-13T14:42:42.708207+0000","last_active":"2024-09-13T14:44:02.842704+0000","last_peered":"2024-09-13T14:44:02.842704+0000","last_clean":"2024-09-13T14:44:02.842704+0000","last_became_active":"2024-09-13T14:42:42.707803+0000","last_became_peered":"2024-09-13T14:42:42.707803+0000","last_unstale":"2024-09-13T14:44:02.842704+0000","last_undegraded":"2024-09-13T14:44:02.842704+0000","last_fullsized":"2024-09-13T14:44:02.842704+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:29.624129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:29.624129+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076481020000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"1141'2781","reported_seq":2802,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:22.105126+0000","last_change":"2024-09-13T14:43:51.534705+0000","last_active":"2024-09-13T14:44:22.105126+0000","last_peered":"2024-09-13T14:44:22.105126+0000","last_clean":"2024-09-13T14:44:22.105126+0000","last_became_active":"2024-09-13T14:43:51.534381+0000","last_became_peered":"2024-09-13T14:43:51.534381+0000","last_unstale":"2024-09-13T14:44:22.105126+0000","last_undegraded":"2024-09-13T14:44:22.105126+0000","last_fullsized":"2024-09-13T14:44:22.105126+0000","mapping_epoch":1136,"log_start":"0'0","ondisk_log_start":"0'0","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":2781,"log_dups_size":0,"ondisk_log_size":2781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:37:22.385742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182255616,"num_objects":2781,"num_object_clones":0,"num_object_copies":8343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2781,"num_write_kb":177984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1111,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.855577+0000","last_change":"2024-09-13T14:44:03.855577+0000","last_active":"2024-09-13T14:44:03.855577+0000","last_peered":"2024-09-13T14:44:03.855577+0000","last_clean":"2024-09-13T14:44:03.855577+0000","last_became_active":"2024-09-13T14:44:03.855267+0000","last_became_peered":"2024-09-13T14:44:03.855267+0000","last_unstale":"2024-09-13T14:44:03.855577+0000","last_undegraded":"2024-09-13T14:44:03.855577+0000","last_fullsized":"2024-09-13T14:44:03.855577+0000","mapping_epoch":1140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:46.148814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1082,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.845371+0000","last_change":"2024-09-13T14:29:42.545815+0000","last_active":"2024-09-13T14:44:02.845371+0000","last_peered":"2024-09-13T14:44:02.845371+0000","last_clean":"2024-09-13T14:44:02.845371+0000","last_became_active":"2024-09-13T14:29:42.545508+0000","last_became_peered":"2024-09-13T14:29:42.545508+0000","last_unstale":"2024-09-13T14:44:02.845371+0000","last_undegraded":"2024-09-13T14:44:02.845371+0000","last_fullsized":"2024-09-13T14:44:02.845371+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:39.678381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"261'100","reported_seq":62,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.849928+0000","last_change":"2024-09-13T14:43:18.988482+0000","last_active":"2024-09-13T14:44:03.849928+0000","last_peered":"2024-09-13T14:44:03.849928+0000","last_clean":"2024-09-13T14:44:03.849928+0000","last_became_active":"2024-09-13T14:43:14.556754+0000","last_became_peered":"2024-09-13T14:43:14.556754+0000","last_unstale":"2024-09-13T14:44:03.849928+0000","last_undegraded":"2024-09-13T14:44:03.849928+0000","last_fullsized":"2024-09-13T14:44:03.849928+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:18.988405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:18.988405+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:45:01.726326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"1141'2624","reported_seq":2645,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:20.772091+0000","last_change":"2024-09-13T14:43:50.796182+0000","last_active":"2024-09-13T14:44:20.772091+0000","last_peered":"2024-09-13T14:44:20.772091+0000","last_clean":"2024-09-13T14:44:20.772091+0000","last_became_active":"2024-09-13T14:43:50.795875+0000","last_became_peered":"2024-09-13T14:43:50.795875+0000","last_unstale":"2024-09-13T14:44:20.772091+0000","last_undegraded":"2024-09-13T14:44:20.772091+0000","last_fullsized":"2024-09-13T14:44:20.772091+0000","mapping_epoch":1136,"log_start":"0'0","ondisk_log_start":"0'0","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":2624,"log_dups_size":0,"ondisk_log_size":2624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:56:34.308870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":2624,"num_object_clones":0,"num_object_copies":7872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2624,"num_write_kb":167936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1128,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.846007+0000","last_change":"2024-09-13T14:43:07.515418+0000","last_active":"2024-09-13T14:44:02.846007+0000","last_peered":"2024-09-13T14:44:02.846007+0000","last_clean":"2024-09-13T14:44:02.846007+0000","last_became_active":"2024-09-13T14:43:07.515113+0000","last_became_peered":"2024-09-13T14:43:07.515113+0000","last_unstale":"2024-09-13T14:44:02.846007+0000","last_undegraded":"2024-09-13T14:44:02.846007+0000","last_fullsized":"2024-09-13T14:44:02.846007+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:10:25.810604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1095,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.845498+0000","last_change":"2024-09-13T14:43:32.636334+0000","last_active":"2024-09-13T14:44:02.845498+0000","last_peered":"2024-09-13T14:44:02.845498+0000","last_clean":"2024-09-13T14:44:02.845498+0000","last_became_active":"2024-09-13T14:43:32.635736+0000","last_became_peered":"2024-09-13T14:43:32.635736+0000","last_unstale":"2024-09-13T14:44:02.845498+0000","last_undegraded":"2024-09-13T14:44:02.845498+0000","last_fullsized":"2024-09-13T14:44:02.845498+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:49.076014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"261'100","reported_seq":66,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.842755+0000","last_change":"2024-09-13T14:42:42.699673+0000","last_active":"2024-09-13T14:44:02.842755+0000","last_peered":"2024-09-13T14:44:02.842755+0000","last_clean":"2024-09-13T14:44:02.842755+0000","last_became_active":"2024-09-13T14:42:42.699317+0000","last_became_peered":"2024-09-13T14:42:42.699317+0000","last_unstale":"2024-09-13T14:44:02.842755+0000","last_undegraded":"2024-09-13T14:44:02.842755+0000","last_fullsized":"2024-09-13T14:44:02.842755+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:15:42.069756+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":34,"num_write":6,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":158179,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"1141'2966","reported_seq":2987,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:23.937560+0000","last_change":"2024-09-13T14:43:50.796641+0000","last_active":"2024-09-13T14:44:23.937560+0000","last_peered":"2024-09-13T14:44:23.937560+0000","last_clean":"2024-09-13T14:44:23.937560+0000","last_became_active":"2024-09-13T14:43:50.796319+0000","last_became_peered":"2024-09-13T14:43:50.796319+0000","last_unstale":"2024-09-13T14:44:23.937560+0000","last_undegraded":"2024-09-13T14:44:23.937560+0000","last_fullsized":"2024-09-13T14:44:23.937560+0000","mapping_epoch":1136,"log_start":"0'0","ondisk_log_start":"0'0","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":2966,"log_dups_size":0,"ondisk_log_size":2966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:40.667128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194379776,"num_objects":2966,"num_object_clones":0,"num_object_copies":8898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2966,"num_write_kb":189824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1121,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.850677+0000","last_change":"2024-09-13T14:44:03.850677+0000","last_active":"2024-09-13T14:44:03.850677+0000","last_peered":"2024-09-13T14:44:03.850677+0000","last_clean":"2024-09-13T14:44:03.850677+0000","last_became_active":"2024-09-13T14:44:03.850330+0000","last_became_peered":"2024-09-13T14:44:03.850330+0000","last_unstale":"2024-09-13T14:44:03.850677+0000","last_undegraded":"2024-09-13T14:44:03.850677+0000","last_fullsized":"2024-09-13T14:44:03.850677+0000","mapping_epoch":1140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:17.962873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":203,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.848762+0000","last_change":"2024-09-13T14:43:07.532034+0000","last_active":"2024-09-13T14:44:03.848762+0000","last_peered":"2024-09-13T14:44:03.848762+0000","last_clean":"2024-09-13T14:44:03.848762+0000","last_became_active":"2024-09-13T14:43:07.515187+0000","last_became_peered":"2024-09-13T14:43:07.515187+0000","last_unstale":"2024-09-13T14:44:03.848762+0000","last_undegraded":"2024-09-13T14:44:03.848762+0000","last_fullsized":"2024-09-13T14:44:03.848762+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:08.970491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"261'100","reported_seq":1251,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.838556+0000","last_change":"2024-09-13T14:42:42.702433+0000","last_active":"2024-09-13T14:44:02.838556+0000","last_peered":"2024-09-13T14:44:02.838556+0000","last_clean":"2024-09-13T14:44:02.838556+0000","last_became_active":"2024-09-13T14:42:42.702124+0000","last_became_peered":"2024-09-13T14:42:42.702124+0000","last_unstale":"2024-09-13T14:44:02.838556+0000","last_undegraded":"2024-09-13T14:44:02.838556+0000","last_fullsized":"2024-09-13T14:44:02.838556+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:28.268005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:28.268005+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015413810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"1141'2636","reported_seq":2657,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:20.792105+0000","last_change":"2024-09-13T14:43:50.795029+0000","last_active":"2024-09-13T14:44:20.792105+0000","last_peered":"2024-09-13T14:44:20.792105+0000","last_clean":"2024-09-13T14:44:20.792105+0000","last_became_active":"2024-09-13T14:43:50.794715+0000","last_became_peered":"2024-09-13T14:43:50.794715+0000","last_unstale":"2024-09-13T14:44:20.792105+0000","last_undegraded":"2024-09-13T14:44:20.792105+0000","last_fullsized":"2024-09-13T14:44:20.792105+0000","mapping_epoch":1136,"log_start":"0'0","ondisk_log_start":"0'0","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":2636,"log_dups_size":0,"ondisk_log_size":2636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:21.257279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172752896,"num_objects":2636,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2636,"num_write_kb":168704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1664,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.848493+0000","last_change":"2024-09-13T14:43:07.515572+0000","last_active":"2024-09-13T14:44:03.848493+0000","last_peered":"2024-09-13T14:44:03.848493+0000","last_clean":"2024-09-13T14:44:03.848493+0000","last_became_active":"2024-09-13T14:43:07.515231+0000","last_became_peered":"2024-09-13T14:43:07.515231+0000","last_unstale":"2024-09-13T14:44:03.848493+0000","last_undegraded":"2024-09-13T14:44:03.848493+0000","last_fullsized":"2024-09-13T14:44:03.848493+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:57.966380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1114,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.843082+0000","last_change":"2024-09-13T14:43:07.524335+0000","last_active":"2024-09-13T14:44:02.843082+0000","last_peered":"2024-09-13T14:44:02.843082+0000","last_clean":"2024-09-13T14:44:02.843082+0000","last_became_active":"2024-09-13T14:43:07.523993+0000","last_became_peered":"2024-09-13T14:43:07.523993+0000","last_unstale":"2024-09-13T14:44:02.843082+0000","last_undegraded":"2024-09-13T14:44:02.843082+0000","last_fullsized":"2024-09-13T14:44:02.843082+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:08:32.728123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"261'100","reported_seq":361,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.846255+0000","last_change":"2024-09-13T14:43:42.881584+0000","last_active":"2024-09-13T14:44:02.846255+0000","last_peered":"2024-09-13T14:44:02.846255+0000","last_clean":"2024-09-13T14:44:02.846255+0000","last_became_active":"2024-09-13T14:43:14.562161+0000","last_became_peered":"2024-09-13T14:43:14.562161+0000","last_unstale":"2024-09-13T14:44:02.846255+0000","last_undegraded":"2024-09-13T14:44:02.846255+0000","last_fullsized":"2024-09-13T14:44:02.846255+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:42.881520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:42.881520+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:06:50.353442+0000","scrub_duration":0.0013804310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"1141'2928","reported_seq":2949,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:23.932636+0000","last_change":"2024-09-13T14:43:50.793879+0000","last_active":"2024-09-13T14:44:23.932636+0000","last_peered":"2024-09-13T14:44:23.932636+0000","last_clean":"2024-09-13T14:44:23.932636+0000","last_became_active":"2024-09-13T14:43:50.793571+0000","last_became_peered":"2024-09-13T14:43:50.793571+0000","last_unstale":"2024-09-13T14:44:23.932636+0000","last_undegraded":"2024-09-13T14:44:23.932636+0000","last_fullsized":"2024-09-13T14:44:23.932636+0000","mapping_epoch":1136,"log_start":"0'0","ondisk_log_start":"0'0","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":2928,"log_dups_size":0,"ondisk_log_size":2928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:22.880360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191889408,"num_objects":2928,"num_object_clones":0,"num_object_copies":8784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2928,"num_write_kb":187392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1097,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.847474+0000","last_change":"2024-09-13T14:43:07.518120+0000","last_active":"2024-09-13T14:44:02.847474+0000","last_peered":"2024-09-13T14:44:02.847474+0000","last_clean":"2024-09-13T14:44:02.847474+0000","last_became_active":"2024-09-13T14:43:07.517361+0000","last_became_peered":"2024-09-13T14:43:07.517361+0000","last_unstale":"2024-09-13T14:44:02.847474+0000","last_undegraded":"2024-09-13T14:44:02.847474+0000","last_fullsized":"2024-09-13T14:44:02.847474+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:39.366473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":1079,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.839793+0000","last_change":"2024-09-13T14:19:25.871006+0000","last_active":"2024-09-13T14:44:02.839793+0000","last_peered":"2024-09-13T14:44:02.839793+0000","last_clean":"2024-09-13T14:44:02.839793+0000","last_became_active":"2024-09-13T14:19:25.870713+0000","last_became_peered":"2024-09-13T14:19:25.870713+0000","last_unstale":"2024-09-13T14:44:02.839793+0000","last_undegraded":"2024-09-13T14:44:02.839793+0000","last_fullsized":"2024-09-13T14:44:02.839793+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:34.372124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"261'100","reported_seq":345,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.843286+0000","last_change":"2024-09-13T14:43:32.637553+0000","last_active":"2024-09-13T14:44:02.843286+0000","last_peered":"2024-09-13T14:44:02.843286+0000","last_clean":"2024-09-13T14:44:02.843286+0000","last_became_active":"2024-09-13T14:43:32.636545+0000","last_became_peered":"2024-09-13T14:43:32.636545+0000","last_unstale":"2024-09-13T14:44:02.843286+0000","last_undegraded":"2024-09-13T14:44:02.843286+0000","last_fullsized":"2024-09-13T14:44:02.843286+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:15:42.069756+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"1141'2634","reported_seq":2655,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:20.787211+0000","last_change":"2024-09-13T14:43:50.793453+0000","last_active":"2024-09-13T14:44:20.787211+0000","last_peered":"2024-09-13T14:44:20.787211+0000","last_clean":"2024-09-13T14:44:20.787211+0000","last_became_active":"2024-09-13T14:43:50.793179+0000","last_became_peered":"2024-09-13T14:43:50.793179+0000","last_unstale":"2024-09-13T14:44:20.787211+0000","last_undegraded":"2024-09-13T14:44:20.787211+0000","last_fullsized":"2024-09-13T14:44:20.787211+0000","mapping_epoch":1136,"log_start":"0'0","ondisk_log_start":"0'0","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":2634,"log_dups_size":0,"ondisk_log_size":2634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:12:51.150099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172621824,"num_objects":2634,"num_object_clones":0,"num_object_copies":7902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2634,"num_write_kb":168576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1138,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.851271+0000","last_change":"2024-09-13T14:44:03.851271+0000","last_active":"2024-09-13T14:44:03.851271+0000","last_peered":"2024-09-13T14:44:03.851271+0000","last_clean":"2024-09-13T14:44:03.851271+0000","last_became_active":"2024-09-13T14:44:03.850962+0000","last_became_peered":"2024-09-13T14:44:03.850962+0000","last_unstale":"2024-09-13T14:44:03.851271+0000","last_undegraded":"2024-09-13T14:44:03.851271+0000","last_fullsized":"2024-09-13T14:44:03.851271+0000","mapping_epoch":1140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:43.434802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":1125,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.839277+0000","last_change":"2024-09-13T14:43:07.533955+0000","last_active":"2024-09-13T14:44:02.839277+0000","last_peered":"2024-09-13T14:44:02.839277+0000","last_clean":"2024-09-13T14:44:02.839277+0000","last_became_active":"2024-09-13T14:43:07.516816+0000","last_became_peered":"2024-09-13T14:43:07.516816+0000","last_unstale":"2024-09-13T14:44:02.839277+0000","last_undegraded":"2024-09-13T14:44:02.839277+0000","last_fullsized":"2024-09-13T14:44:02.839277+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:26.232184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"261'100","reported_seq":49,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.847850+0000","last_change":"2024-09-13T14:43:16.597650+0000","last_active":"2024-09-13T14:44:03.847850+0000","last_peered":"2024-09-13T14:44:03.847850+0000","last_clean":"2024-09-13T14:44:03.847850+0000","last_became_active":"2024-09-13T14:43:14.563104+0000","last_became_peered":"2024-09-13T14:43:14.563104+0000","last_unstale":"2024-09-13T14:44:03.847850+0000","last_undegraded":"2024-09-13T14:44:03.847850+0000","last_fullsized":"2024-09-13T14:44:03.847850+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.597568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.597568+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:20.426709+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"1141'2803","reported_seq":2823,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:22.118872+0000","last_change":"2024-09-13T14:43:51.537117+0000","last_active":"2024-09-13T14:44:22.118872+0000","last_peered":"2024-09-13T14:44:22.118872+0000","last_clean":"2024-09-13T14:44:22.118872+0000","last_became_active":"2024-09-13T14:43:51.536532+0000","last_became_peered":"2024-09-13T14:43:51.536532+0000","last_unstale":"2024-09-13T14:44:22.118872+0000","last_undegraded":"2024-09-13T14:44:22.118872+0000","last_fullsized":"2024-09-13T14:44:22.118872+0000","mapping_epoch":1136,"log_start":"0'0","ondisk_log_start":"0'0","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":2803,"log_dups_size":0,"ondisk_log_size":2803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:41.713211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183697408,"num_objects":2803,"num_object_clones":0,"num_object_copies":8409,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2803,"num_write_kb":179392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1066,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.843112+0000","last_change":"2024-09-13T14:19:26.887839+0000","last_active":"2024-09-13T14:44:02.843112+0000","last_peered":"2024-09-13T14:44:02.843112+0000","last_clean":"2024-09-13T14:44:02.843112+0000","last_became_active":"2024-09-13T14:19:26.887498+0000","last_became_peered":"2024-09-13T14:19:26.887498+0000","last_unstale":"2024-09-13T14:44:02.843112+0000","last_undegraded":"2024-09-13T14:44:02.843112+0000","last_fullsized":"2024-09-13T14:44:02.843112+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:24.475263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":1096,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.839427+0000","last_change":"2024-09-13T14:28:35.608607+0000","last_active":"2024-09-13T14:44:02.839427+0000","last_peered":"2024-09-13T14:44:02.839427+0000","last_clean":"2024-09-13T14:44:02.839427+0000","last_became_active":"2024-09-13T14:28:35.608254+0000","last_became_peered":"2024-09-13T14:28:35.608254+0000","last_unstale":"2024-09-13T14:44:02.839427+0000","last_undegraded":"2024-09-13T14:44:02.839427+0000","last_fullsized":"2024-09-13T14:44:02.839427+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:24.019173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"261'100","reported_seq":22,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.839642+0000","last_change":"2024-09-13T14:43:14.557765+0000","last_active":"2024-09-13T14:44:02.839642+0000","last_peered":"2024-09-13T14:44:02.839642+0000","last_clean":"2024-09-13T14:44:02.839642+0000","last_became_active":"2024-09-13T14:43:14.557376+0000","last_became_peered":"2024-09-13T14:43:14.557376+0000","last_unstale":"2024-09-13T14:44:02.839642+0000","last_undegraded":"2024-09-13T14:44:02.839642+0000","last_fullsized":"2024-09-13T14:44:02.839642+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:22.892963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:22.892963+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13417,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":17,"num_write":3,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":13417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1086,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.846957+0000","last_change":"2024-09-13T14:28:35.603724+0000","last_active":"2024-09-13T14:44:02.846957+0000","last_peered":"2024-09-13T14:44:02.846957+0000","last_clean":"2024-09-13T14:44:02.846957+0000","last_became_active":"2024-09-13T14:28:35.603316+0000","last_became_peered":"2024-09-13T14:28:35.603316+0000","last_unstale":"2024-09-13T14:44:02.846957+0000","last_undegraded":"2024-09-13T14:44:02.846957+0000","last_fullsized":"2024-09-13T14:44:02.846957+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:47.668223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":650,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.839337+0000","last_change":"2024-09-13T14:43:07.515517+0000","last_active":"2024-09-13T14:44:02.839337+0000","last_peered":"2024-09-13T14:44:02.839337+0000","last_clean":"2024-09-13T14:44:02.839337+0000","last_became_active":"2024-09-13T14:43:07.515121+0000","last_became_peered":"2024-09-13T14:43:07.515121+0000","last_unstale":"2024-09-13T14:44:02.839337+0000","last_undegraded":"2024-09-13T14:44:02.839337+0000","last_fullsized":"2024-09-13T14:44:02.839337+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:30:41.713771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"261'100","reported_seq":22,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.842798+0000","last_change":"2024-09-13T14:43:14.563346+0000","last_active":"2024-09-13T14:44:02.842798+0000","last_peered":"2024-09-13T14:44:02.842798+0000","last_clean":"2024-09-13T14:44:02.842798+0000","last_became_active":"2024-09-13T14:43:14.562922+0000","last_became_peered":"2024-09-13T14:43:14.562922+0000","last_unstale":"2024-09-13T14:44:02.842798+0000","last_undegraded":"2024-09-13T14:44:02.842798+0000","last_fullsized":"2024-09-13T14:44:02.842798+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:24.245668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:24.245668+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1078,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.838672+0000","last_change":"2024-09-13T14:19:25.873088+0000","last_active":"2024-09-13T14:44:02.838672+0000","last_peered":"2024-09-13T14:44:02.838672+0000","last_clean":"2024-09-13T14:44:02.838672+0000","last_became_active":"2024-09-13T14:19:25.872781+0000","last_became_peered":"2024-09-13T14:19:25.872781+0000","last_unstale":"2024-09-13T14:44:02.838672+0000","last_undegraded":"2024-09-13T14:44:02.838672+0000","last_fullsized":"2024-09-13T14:44:02.838672+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:05.607382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1656,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.848819+0000","last_change":"2024-09-13T14:39:11.394501+0000","last_active":"2024-09-13T14:44:03.848819+0000","last_peered":"2024-09-13T14:44:03.848819+0000","last_clean":"2024-09-13T14:44:03.848819+0000","last_became_active":"2024-09-13T14:39:11.394134+0000","last_became_peered":"2024-09-13T14:39:11.394134+0000","last_unstale":"2024-09-13T14:44:03.848819+0000","last_undegraded":"2024-09-13T14:44:03.848819+0000","last_fullsized":"2024-09-13T14:44:03.848819+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:40:09.359562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"261'100","reported_seq":62,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.850043+0000","last_change":"2024-09-13T14:43:19.978531+0000","last_active":"2024-09-13T14:44:03.850043+0000","last_peered":"2024-09-13T14:44:03.850043+0000","last_clean":"2024-09-13T14:44:03.850043+0000","last_became_active":"2024-09-13T14:43:14.558124+0000","last_became_peered":"2024-09-13T14:43:14.558124+0000","last_unstale":"2024-09-13T14:44:03.850043+0000","last_undegraded":"2024-09-13T14:44:03.850043+0000","last_fullsized":"2024-09-13T14:44:03.850043+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:19.978464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:19.978464+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:40:55.465228+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1703,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.848246+0000","last_change":"2024-09-13T14:43:07.523135+0000","last_active":"2024-09-13T14:44:03.848246+0000","last_peered":"2024-09-13T14:44:03.848246+0000","last_clean":"2024-09-13T14:44:03.848246+0000","last_became_active":"2024-09-13T14:43:07.522594+0000","last_became_peered":"2024-09-13T14:43:07.522594+0000","last_unstale":"2024-09-13T14:44:03.848246+0000","last_undegraded":"2024-09-13T14:44:03.848246+0000","last_fullsized":"2024-09-13T14:44:03.848246+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:41.704718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1093,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.843046+0000","last_change":"2024-09-13T14:39:11.379707+0000","last_active":"2024-09-13T14:44:02.843046+0000","last_peered":"2024-09-13T14:44:02.843046+0000","last_clean":"2024-09-13T14:44:02.843046+0000","last_became_active":"2024-09-13T14:39:11.379255+0000","last_became_peered":"2024-09-13T14:39:11.379255+0000","last_unstale":"2024-09-13T14:44:02.843046+0000","last_undegraded":"2024-09-13T14:44:02.843046+0000","last_fullsized":"2024-09-13T14:44:02.843046+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:48.543270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"261'100","reported_seq":22,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.842595+0000","last_change":"2024-09-13T14:43:14.564667+0000","last_active":"2024-09-13T14:44:02.842595+0000","last_peered":"2024-09-13T14:44:02.842595+0000","last_clean":"2024-09-13T14:44:02.842595+0000","last_became_active":"2024-09-13T14:43:14.563451+0000","last_became_peered":"2024-09-13T14:43:14.563451+0000","last_unstale":"2024-09-13T14:44:02.842595+0000","last_undegraded":"2024-09-13T14:44:02.842595+0000","last_fullsized":"2024-09-13T14:44:02.842595+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:41:23.257279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:41:23.257279+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":1097,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.838866+0000","last_change":"2024-09-13T14:29:42.541093+0000","last_active":"2024-09-13T14:44:02.838866+0000","last_peered":"2024-09-13T14:44:02.838866+0000","last_clean":"2024-09-13T14:44:02.838866+0000","last_became_active":"2024-09-13T14:29:42.540671+0000","last_became_peered":"2024-09-13T14:29:42.540671+0000","last_unstale":"2024-09-13T14:44:02.838866+0000","last_undegraded":"2024-09-13T14:44:02.838866+0000","last_fullsized":"2024-09-13T14:44:02.838866+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:45.722376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1602,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:04.054838+0000","last_change":"2024-09-13T14:43:07.518679+0000","last_active":"2024-09-13T14:44:04.054838+0000","last_peered":"2024-09-13T14:44:04.054838+0000","last_clean":"2024-09-13T14:44:04.054838+0000","last_became_active":"2024-09-13T14:43:07.518333+0000","last_became_peered":"2024-09-13T14:43:07.518333+0000","last_unstale":"2024-09-13T14:44:04.054838+0000","last_undegraded":"2024-09-13T14:44:04.054838+0000","last_fullsized":"2024-09-13T14:44:04.054838+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:20.616274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"261'100","reported_seq":364,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-09-13T14:44:03.849725+0000","last_change":"2024-09-13T14:43:14.654650+0000","last_active":"2024-09-13T14:44:03.849725+0000","last_peered":"2024-09-13T14:44:03.849725+0000","last_clean":"2024-09-13T14:44:03.849725+0000","last_became_active":"2024-09-13T14:43:14.556924+0000","last_became_peered":"2024-09-13T14:43:14.556924+0000","last_unstale":"2024-09-13T14:44:03.849725+0000","last_undegraded":"2024-09-13T14:44:03.849725+0000","last_fullsized":"2024-09-13T14:44:03.849725+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:14.654603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:14.654603+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:13:08.446767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2941583360,"num_objects":44885,"num_object_clones":0,"num_object_copies":134655,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44885,"num_write_kb":2872640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8821276672,"data_stored":8821276672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44885,"ondisk_log_size":44885,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":642641,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":104,"num_read_kb":262,"num_write":53,"num_write_kb":568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":731111,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430532,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1371072,"kb_used_data":1073720,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":92374080,"statfs":{"total":95995035648,"available":94591057920,"internally_reserved":0,"allocated":1099489280,"data_stored":1096054688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626761,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1374068,"kb_used_data":1068396,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92371084,"statfs":{"total":95995035648,"available":94587990016,"internally_reserved":0,"allocated":1094037504,"data_stored":1090614537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822990,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1183408,"kb_used_data":881896,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":92561744,"statfs":{"total":95995035648,"available":94783225856,"internally_reserved":0,"allocated":903061504,"data_stored":899638995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019218,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1376760,"kb_used_data":1071088,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92368392,"statfs":{"total":95995035648,"available":94585233408,"internally_reserved":0,"allocated":1096794112,"data_stored":1093367098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059592876,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":100392,"kb_used_data":10204,"kb_used_omap":26,"kb_used_meta":90149,"kb_avail":93644760,"statfs":{"total":95995035648,"available":95892234240,"internally_reserved":0,"allocated":10448896,"data_stored":7037582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":92312736},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":591,"seq":2538325672114,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1613548,"kb_used_data":1314468,"kb_used_omap":41,"kb_used_meta":299030,"kb_avail":92131604,"statfs":{"total":95995035648,"available":94342762496,"internally_reserved":0,"allocated":1346015232,"data_stored":1342596792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42651,"internal_metadata":306207077},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":557,"seq":2392296784057,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1888932,"kb_used_data":1686172,"kb_used_omap":32,"kb_used_meta":202719,"kb_avail":91856220,"statfs":{"total":95995035648,"available":94060769280,"internally_reserved":0,"allocated":1726640128,"data_stored":1723228759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33752,"internal_metadata":207584296},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":524,"seq":2250562863296,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1651624,"kb_used_data":1497440,"kb_used_omap":41,"kb_used_meta":154134,"kb_avail":92093528,"statfs":{"total":95995035648,"available":94303772672,"internally_reserved":0,"allocated":1533378560,"data_stored":1529963676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42694,"internal_metadata":157833530},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1556348928,"data_stored":1556348928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1639120896,"data_stored":1639120896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1371340800,"data_stored":1371340800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1102315520,"data_stored":1102315520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":928251904,"data_stored":928251904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1121648640,"data_stored":1121648640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1102249984,"data_stored":1102249984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:44:27.606 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-13T14:44:27.606 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 41 2024-09-13T14:44:27.606 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:27.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:27.648 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:27.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:27.723 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 26 15 53763 53748 129.182 103.625 0.0104339 0.00773561 2024-09-13T14:44:27.730 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:27 smithi184 ceph-mon[92527]: from='client.45512 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:44:27.749 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:27.783 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:27.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:27.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:27.799 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:27 smithi151 ceph-mon[104450]: from='client.45512 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:44:27.799 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:27 smithi151 ceph-mon[108966]: from='client.45512 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:44:27.885 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:27.912 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:27.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:27.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:28.014 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:28.064 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:28.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:28.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:28.166 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:28.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:28.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:28.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:28.311 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:28.346 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:28.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:28.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:28.448 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:28.481 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:28.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:28.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:28.583 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:28.603 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:44:28.622 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:28.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:28.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:28.723 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 27 16 55370 55354 128.114 100.375 0.0105704 0.00780013 2024-09-13T14:44:28.723 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:28.729 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:28.764 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:28 smithi184 ceph-mon[92527]: pgmap v1499: 119 pgs: 119 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 110 MiB/s wr, 1.75k op/s 2024-09-13T14:44:28.808 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:28.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:28.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:28.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:28 smithi151 ceph-mon[104450]: pgmap v1499: 119 pgs: 119 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 110 MiB/s wr, 1.75k op/s 2024-09-13T14:44:28.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:28 smithi151 ceph-mon[108966]: pgmap v1499: 119 pgs: 119 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 110 MiB/s wr, 1.75k op/s 2024-09-13T14:44:28.910 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:28.941 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:28.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:28.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:29.042 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:29.095 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:29.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:29.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:29.197 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:29.231 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:29.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:29.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:29.243 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:44:29.243 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:44:29.333 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:29.363 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:29.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:29.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:29.465 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:29.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:29.496 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:29.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:29.597 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:29.613 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:44:29.613 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:44:29.625 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:29.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:29.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:29.650 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:44:29.650 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:44:29.684 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:44:29.684 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:44:29.716 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:44:29.717 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:44:29.717 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:29 smithi151 ceph-mon[104450]: 1.0 scrub starts 2024-09-13T14:44:29.717 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:29 smithi151 ceph-mon[104450]: 1.0 scrub ok 2024-09-13T14:44:29.717 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:29 smithi151 ceph-mon[108966]: 1.0 scrub starts 2024-09-13T14:44:29.717 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:29 smithi151 ceph-mon[108966]: 1.0 scrub ok 2024-09-13T14:44:29.723 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 28 16 56851 56835 126.844 92.5625 0.00756627 0.00787719 2024-09-13T14:44:29.726 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:29.748 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:44:29.748 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:44:29.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:29.778 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:29.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:29.781 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:44:29.782 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:44:29.812 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:44:29.812 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:44:29.846 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:44:29.846 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:44:29.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:29 smithi184 ceph-mon[92527]: 1.0 scrub starts 2024-09-13T14:44:29.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:29 smithi184 ceph-mon[92527]: 1.0 scrub ok 2024-09-13T14:44:29.880 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:29.908 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:29.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:29.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:29.986 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T14:44:30.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:30.056 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:30.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:30.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:30.158 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:30.193 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:30.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:30.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:30.295 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:30.333 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:30.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:30.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:30.370 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:44:30.435 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:30.471 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:30.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:30.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:30.500 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:44:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:44:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:44:30.573 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:30.607 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:30.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:30.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:30.709 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:30.723 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 29 16 58385 58369 125.775 95.875 0.0079791 0.00794431 2024-09-13T14:44:30.748 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:30.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:30.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:30.831 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:30 smithi184 ceph-mon[92527]: 1.9 scrub starts 2024-09-13T14:44:30.831 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:30 smithi184 ceph-mon[92527]: 1.9 scrub ok 2024-09-13T14:44:30.831 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:30 smithi184 ceph-mon[92527]: pgmap v1500: 119 pgs: 119 active+clean; 3.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-09-13T14:44:30.831 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:30 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/674536745' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:30.850 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:30.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:30 smithi151 ceph-mon[104450]: 1.9 scrub starts 2024-09-13T14:44:30.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:30 smithi151 ceph-mon[104450]: 1.9 scrub ok 2024-09-13T14:44:30.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:30 smithi151 ceph-mon[104450]: pgmap v1500: 119 pgs: 119 active+clean; 3.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-09-13T14:44:30.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:30 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/674536745' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:30.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:30 smithi151 ceph-mon[108966]: 1.9 scrub starts 2024-09-13T14:44:30.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:30 smithi151 ceph-mon[108966]: 1.9 scrub ok 2024-09-13T14:44:30.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:30 smithi151 ceph-mon[108966]: pgmap v1500: 119 pgs: 119 active+clean; 3.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-09-13T14:44:30.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:30 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/674536745' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:30.882 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:30.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:30.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:30.984 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:31.020 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:31.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:31.122 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:31.156 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:31.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:31.258 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:31.291 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:31.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:31.393 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:31.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.415 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:31.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:31.516 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:31.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.556 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:31.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:31.657 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:31.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.693 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:31.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:31.723 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 30 16 59823 59807 124.578 89.875 0.00742548 0.00802235 2024-09-13T14:44:31.794 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:31.815 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:31.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:31.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:31.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:31.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:31.918 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:31.952 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:31.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:32.054 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:32.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:32.076 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:32.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:32.177 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:32.210 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:32.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:32.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:32.312 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:32.337 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:32.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:32.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:32.439 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:32.462 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:32.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:32.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:32.563 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:32.595 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:44:32.596 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [ 2024-09-13T14:44:32.596 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:32.596 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:60726 7.0 7:05ab024e:::benchmark_data_smithi151_190906_object60725:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1142)", 2024-09-13T14:44:32.596 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:44:32.588903+0000", 2024-09-13T14:44:32.596 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.004838597, 2024-09-13T14:44:32.596 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0048751350000000001, 2024-09-13T14:44:32.596 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:44:32.596 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:44:32.596 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:44:32.596 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:44:32.596 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:44:32.597 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 60726 2024-09-13T14:44:32.597 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:32.597 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:44:32.597 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:32.597 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:44:32.597 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.588903+0000", 2024-09-13T14:44:32.597 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:32.597 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:32.597 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:32.597 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:44:32.597 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.588903+0000", 2024-09-13T14:44:32.597 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:32.597 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:32.598 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:32.598 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:44:32.598 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.588901+0000", 2024-09-13T14:44:32.598 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999976 2024-09-13T14:44:32.598 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:32.598 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:32.598 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:44:32.598 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.588960+0000", 2024-09-13T14:44:32.598 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8310999999999998e-05 2024-09-13T14:44:32.598 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:32.598 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:32.598 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:44:32.599 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.588973+0000", 2024-09-13T14:44:32.599 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3762e-05 2024-09-13T14:44:32.599 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:32.599 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:32.599 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:44:32.599 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.588979+0000", 2024-09-13T14:44:32.599 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.0440000000000001e-06 2024-09-13T14:44:32.599 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:32.599 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:32.599 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:44:32.599 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.589023+0000", 2024-09-13T14:44:32.599 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3260999999999998e-05 2024-09-13T14:44:32.599 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:32.599 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:32.600 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:44:32.600 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.589165+0000", 2024-09-13T14:44:32.600 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000142309 2024-09-13T14:44:32.600 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:32.600 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:32.600 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:44:32.600 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.589300+0000", 2024-09-13T14:44:32.600 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000135065 2024-09-13T14:44:32.600 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:32.600 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:32.601 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:44:32.601 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.591724+0000", 2024-09-13T14:44:32.601 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0024236629999999999 2024-09-13T14:44:32.601 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:32.601 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:44:32.601 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:32.601 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:32.601 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:32.601 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.45494.0:60729 7.a e1142/1136 7:5a1295f7:::benchmark_data_smithi151_190906_object60728:head v 1142'3766, mlcod=1142'3764)", 2024-09-13T14:44:32.601 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:44:32.591807+0000", 2024-09-13T14:44:32.601 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.0019351329999999999, 2024-09-13T14:44:32.601 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0020927110000000001, 2024-09-13T14:44:32.601 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:44:32.602 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:44:32.602 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:44:32.602 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:32.602 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:44:32.602 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.591807+0000", 2024-09-13T14:44:32.602 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:32.602 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:32.602 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:32.602 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:44:32.603 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.591807+0000", 2024-09-13T14:44:32.603 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:32.603 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:32.603 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:32.603 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:44:32.603 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.591806+0000", 2024-09-13T14:44:32.603 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:44:32.603 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:32.603 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:32.603 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:44:32.603 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.591902+0000", 2024-09-13T14:44:32.603 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.5884999999999994e-05 2024-09-13T14:44:32.603 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:32.604 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:32.604 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:44:32.604 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.591916+0000", 2024-09-13T14:44:32.604 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4099e-05 2024-09-13T14:44:32.604 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:32.604 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:32.604 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:44:32.604 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.591922+0000", 2024-09-13T14:44:32.604 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7209999999999999e-06 2024-09-13T14:44:32.605 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:32.605 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:32.605 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:44:32.605 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.591971+0000", 2024-09-13T14:44:32.605 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9367999999999999e-05 2024-09-13T14:44:32.605 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:32.605 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:32.605 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:44:32.605 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.591992+0000", 2024-09-13T14:44:32.605 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.1399e-05 2024-09-13T14:44:32.605 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:32.606 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:44:32.606 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:32.606 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:32.606 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:32.606 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.45494.0:60730 7.e e1142/1136 7:7b330308:::benchmark_data_smithi151_190906_object60729:head v 1142'3773, mlcod=1142'3769)", 2024-09-13T14:44:32.606 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:44:32.593282+0000", 2024-09-13T14:44:32.606 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.00046021400000000002, 2024-09-13T14:44:32.606 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00067995999999999996, 2024-09-13T14:44:32.606 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:44:32.607 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:44:32.607 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:44:32.607 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:32.607 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:44:32.607 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.593282+0000", 2024-09-13T14:44:32.607 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:32.607 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:32.607 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:32.607 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:44:32.607 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.593282+0000", 2024-09-13T14:44:32.607 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:32.607 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:32.607 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:32.607 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:44:32.608 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.593281+0000", 2024-09-13T14:44:32.608 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:44:32.608 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:32.608 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:32.608 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:44:32.608 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.593333+0000", 2024-09-13T14:44:32.608 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.2055999999999998e-05 2024-09-13T14:44:32.608 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:32.608 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:32.608 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:44:32.608 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.593347+0000", 2024-09-13T14:44:32.608 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4129e-05 2024-09-13T14:44:32.608 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:32.609 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:32.609 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:44:32.609 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.593356+0000", 2024-09-13T14:44:32.609 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.9900000000000003e-06 2024-09-13T14:44:32.609 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:32.609 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:32.609 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:44:32.609 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.593389+0000", 2024-09-13T14:44:32.609 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.2929999999999998e-05 2024-09-13T14:44:32.609 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:32.609 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:32.609 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:44:32.609 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.593408+0000", 2024-09-13T14:44:32.609 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.912e-05 2024-09-13T14:44:32.610 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:32.610 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:44:32.610 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:32.610 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:32.610 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-13T14:44:32.610 INFO:teuthology.orchestra.run.smithi184.stdout: "num_ops": 3 2024-09-13T14:44:32.610 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:44:32.610 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:32 smithi184 ceph-mon[92527]: 1.d scrub starts 2024-09-13T14:44:32.610 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:32 smithi184 ceph-mon[92527]: 1.d scrub ok 2024-09-13T14:44:32.611 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:32 smithi184 ceph-mon[92527]: pgmap v1501: 119 pgs: 119 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-09-13T14:44:32.611 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:32 smithi184 ceph-mon[92527]: 1.4 scrub starts 2024-09-13T14:44:32.611 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:32 smithi184 ceph-mon[92527]: 1.4 scrub ok 2024-09-13T14:44:32.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:32.611 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:32.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:32.712 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:32.723 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 31 16 60937 60921 122.806 69.625 0.0106835 0.00813776 2024-09-13T14:44:32.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:32.742 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:32.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:32.843 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:32.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:32 smithi151 ceph-mon[104450]: 1.d scrub starts 2024-09-13T14:44:32.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:32 smithi151 ceph-mon[104450]: 1.d scrub ok 2024-09-13T14:44:32.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:32 smithi151 ceph-mon[104450]: pgmap v1501: 119 pgs: 119 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-09-13T14:44:32.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:32 smithi151 ceph-mon[104450]: 1.4 scrub starts 2024-09-13T14:44:32.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:32 smithi151 ceph-mon[104450]: 1.4 scrub ok 2024-09-13T14:44:32.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:32 smithi151 ceph-mon[108966]: 1.d scrub starts 2024-09-13T14:44:32.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:32 smithi151 ceph-mon[108966]: 1.d scrub ok 2024-09-13T14:44:32.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:32 smithi151 ceph-mon[108966]: pgmap v1501: 119 pgs: 119 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-09-13T14:44:32.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:32 smithi151 ceph-mon[108966]: 1.4 scrub starts 2024-09-13T14:44:32.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:32 smithi151 ceph-mon[108966]: 1.4 scrub ok 2024-09-13T14:44:32.864 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:32.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:32.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:32.966 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:33.000 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:33.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:33.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:33.102 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:33.126 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:33.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:33.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:33.228 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:33.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:33.257 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:33.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:33.358 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:33.398 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:33.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:33.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:33.500 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:33.527 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:33.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:33.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:33.628 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:33.630 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T14:44:33.649 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:33.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:33.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:33.679 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:33 smithi151 ceph-mon[104450]: 1.8 scrub starts 2024-09-13T14:44:33.679 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:33 smithi151 ceph-mon[108966]: 1.8 scrub starts 2024-09-13T14:44:33.724 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 32 16 61895 61879 120.839 59.875 0.00325265 0.00821855 2024-09-13T14:44:33.751 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:33.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:33.779 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:33.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:33.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:33 smithi184 ceph-mon[92527]: 1.8 scrub starts 2024-09-13T14:44:33.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:33 smithi184 ceph-mon[92527]: 1.8 scrub ok 2024-09-13T14:44:33.880 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:33.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:33.914 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:33.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:33.936 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:44:34.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:34.018 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:33 smithi151 ceph-mon[104450]: 1.8 scrub ok 2024-09-13T14:44:34.018 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:33 smithi151 ceph-mon[108966]: 1.8 scrub ok 2024-09-13T14:44:34.052 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:34.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:34.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:34.154 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:34.189 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:34.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:34.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:34.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:34.330 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:34.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:34.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:34.432 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:34.461 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:34.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:34.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:34.563 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:34.598 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:34.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:34.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:34.696 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:34 smithi184 ceph-mon[92527]: pgmap v1502: 119 pgs: 119 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 102 MiB/s wr, 1.62k op/s 2024-09-13T14:44:34.699 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:34.723 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:34.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:34.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:34.725 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 33 16 63027 63011 119.321 70.75 0.0100723 0.00835305 2024-09-13T14:44:34.825 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:34.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:34 smithi151 ceph-mon[104450]: pgmap v1502: 119 pgs: 119 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 102 MiB/s wr, 1.62k op/s 2024-09-13T14:44:34.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:44:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:44:34] "GET /metrics HTTP/1.1" 200 36655 "" "Prometheus/2.51.0" 2024-09-13T14:44:34.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:34 smithi151 ceph-mon[108966]: pgmap v1502: 119 pgs: 119 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 102 MiB/s wr, 1.62k op/s 2024-09-13T14:44:34.860 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:34.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:34.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:34.963 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:34.984 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:34.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:34.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:35.087 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:35.115 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:35.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:35.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:35.217 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:35.250 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:35.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:35.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:35.352 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:35.384 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:35.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:35.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:35.486 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:35.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:35.532 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:35.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:35.633 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:35.665 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:35.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:35.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:35.724 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 34 16 64186 64170 117.942 72.4375 0.0201287 0.00847396 2024-09-13T14:44:35.767 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:35.795 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:35.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:35.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:35.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:35 smithi184 ceph-mon[92527]: 1.a scrub starts 2024-09-13T14:44:35.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:35 smithi184 ceph-mon[92527]: 1.a scrub ok 2024-09-13T14:44:35.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:35 smithi151 ceph-mon[104450]: 1.a scrub starts 2024-09-13T14:44:35.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:35 smithi151 ceph-mon[104450]: 1.a scrub ok 2024-09-13T14:44:35.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:35 smithi151 ceph-mon[108966]: 1.a scrub starts 2024-09-13T14:44:35.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:35 smithi151 ceph-mon[108966]: 1.a scrub ok 2024-09-13T14:44:35.898 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:35.934 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:35.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:35.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:36.036 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:36.074 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:36.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:36.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:36.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:36.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:36.197 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:36.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:36.298 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:36.330 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:36.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:36.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:36.432 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:36.453 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:36.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:36.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:36.496 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:44:36.497 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [], 2024-09-13T14:44:36.497 INFO:teuthology.orchestra.run.smithi184.stdout: "complaint_time": 30, 2024-09-13T14:44:36.497 INFO:teuthology.orchestra.run.smithi184.stdout: "num_blocked_ops": 0 2024-09-13T14:44:36.497 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:44:36.555 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:36.584 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:36.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:36.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:36.686 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:36.709 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:36.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:36.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:36.724 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 35 16 65373 65357 116.692 74.1875 0.00615265 0.008563 2024-09-13T14:44:36.811 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:36.840 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:36.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:36.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:36.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:36 smithi184 ceph-mon[92527]: pgmap v1503: 119 pgs: 119 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-09-13T14:44:36.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:36 smithi151 ceph-mon[104450]: pgmap v1503: 119 pgs: 119 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-09-13T14:44:36.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:36 smithi151 ceph-mon[108966]: pgmap v1503: 119 pgs: 119 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-09-13T14:44:36.942 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:36.964 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:36.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:36.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:37.066 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:37.087 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:37.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:37.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:37.189 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:37.223 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:37.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:37.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:37.325 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:37.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:37.366 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:37.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:37.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:37.499 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:37.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:37.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:37.601 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:37.635 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:37.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:37.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:37.668 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T14:44:37.724 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 36 16 66571 66555 115.53 74.875 0.0749572 0.00865092 2024-09-13T14:44:37.737 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:37.775 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:37.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:37.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:37.877 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:37.917 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:37.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:37.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:38.001 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:44:38.018 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:38.039 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:38.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:38.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:38.141 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:38.171 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:38.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:38.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:38.273 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:38.305 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:38.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:38.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:38.407 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:38.440 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:38.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:38.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:38.542 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:38.581 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:38.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:38.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:38.683 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:38.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:38.705 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:38.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:38.724 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 37 16 67692 67676 114.301 70.0625 0.0897219 0.00872951 2024-09-13T14:44:38.806 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:38.842 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:38.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:38.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:38.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:38 smithi184 ceph-mon[92527]: pgmap v1504: 119 pgs: 119 active+clean; 3.9 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-13T14:44:38.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:38 smithi151 ceph-mon[104450]: pgmap v1504: 119 pgs: 119 active+clean; 3.9 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-13T14:44:38.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:38 smithi151 ceph-mon[108966]: pgmap v1504: 119 pgs: 119 active+clean; 3.9 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-13T14:44:38.944 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:38.977 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:38.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:38.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:39.079 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:39.153 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:39.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:39.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:39.255 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:39.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:39.306 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:39.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:39.408 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:39.435 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:39.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:39.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:39.537 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:39.579 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:39.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:39.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:39.681 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:39.707 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:39 smithi151 ceph-mon[104450]: 1.15 scrub starts 2024-09-13T14:44:39.707 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:39 smithi151 ceph-mon[104450]: 1.15 scrub ok 2024-09-13T14:44:39.707 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:39 smithi151 ceph-mon[108966]: 1.15 scrub starts 2024-09-13T14:44:39.707 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:39 smithi151 ceph-mon[108966]: 1.15 scrub ok 2024-09-13T14:44:39.720 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:39.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:39.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:39.726 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 38 16 69095 69079 113.595 87.6875 0.00789807 0.00879804 2024-09-13T14:44:39.822 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:39.849 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:39.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:39.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:39.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:39 smithi184 ceph-mon[92527]: 1.15 scrub starts 2024-09-13T14:44:39.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:39 smithi184 ceph-mon[92527]: 1.15 scrub ok 2024-09-13T14:44:39.952 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:39.978 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:39.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:39.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:40.080 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:40.112 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:40.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:40.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:40.177 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:40.214 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:40.245 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:40.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:40.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:40.347 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:40.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:40.377 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:40.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:40.481 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:40.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:40.524 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:40.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:40.574 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: specified pg_num 41 < pg_num_max 32 2024-09-13T14:44:40.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:44:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:44:40.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:44:40.626 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:40.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:40.669 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:40.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:40.726 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 39 16 70373 70357 112.73 79.875 0.0137731 0.00886641 2024-09-13T14:44:40.770 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:40.816 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:40.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:40.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:40.918 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:40.957 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:40 smithi184 ceph-mon[92527]: pgmap v1505: 119 pgs: 119 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-13T14:44:40.957 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:40 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1328117771' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"} : dispatch 2024-09-13T14:44:40.957 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:40 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"} : dispatch 2024-09-13T14:44:40.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:40.960 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:40.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:41.061 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:41.079 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:44:41.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:41.099 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:41.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:41.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:40 smithi151 ceph-mon[104450]: pgmap v1505: 119 pgs: 119 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-13T14:44:41.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:40 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1328117771' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"} : dispatch 2024-09-13T14:44:41.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:40 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"} : dispatch 2024-09-13T14:44:41.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:40 smithi151 ceph-mon[108966]: pgmap v1505: 119 pgs: 119 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-13T14:44:41.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:40 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1328117771' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"} : dispatch 2024-09-13T14:44:41.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:40 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"} : dispatch 2024-09-13T14:44:41.200 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:41.232 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:41.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:41.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:41.334 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:41.368 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:41.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:41.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:41.469 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:41.516 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:41.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:41.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:41.618 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:41.635 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:44:41.636 INFO:teuthology.orchestra.run.smithi184.stdout: "size": 20, 2024-09-13T14:44:41.636 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 600, 2024-09-13T14:44:41.636 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [ 2024-09-13T14:44:41.636 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.636 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:63696 6.d 6:b8303ab0:::benchmark_data_smithi151_155836_object63695:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1088)", 2024-09-13T14:44:41.636 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:39:55.553817+0000", 2024-09-13T14:44:41.636 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 286.078361239, 2024-09-13T14:44:41.636 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.100030725, 2024-09-13T14:44:41.636 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:44:41.636 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:41.636 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:44:41.637 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:44:41.637 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:44:41.637 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 63696 2024-09-13T14:44:41.637 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.637 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:44:41.637 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.637 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:44:41.637 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.553817+0000", 2024-09-13T14:44:41.637 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.637 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.637 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.637 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:44:41.637 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.553817+0000", 2024-09-13T14:44:41.637 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.638 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.638 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.638 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:44:41.638 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.553809+0000", 2024-09-13T14:44:41.638 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999919 2024-09-13T14:44:41.638 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.638 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.638 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:44:41.638 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.553850+0000", 2024-09-13T14:44:41.638 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0754000000000003e-05 2024-09-13T14:44:41.638 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.638 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.638 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:44:41.639 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.553864+0000", 2024-09-13T14:44:41.639 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4015e-05 2024-09-13T14:44:41.639 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.639 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.639 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:44:41.639 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.553870+0000", 2024-09-13T14:44:41.639 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.711e-06 2024-09-13T14:44:41.639 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.639 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.639 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:44:41.639 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.553947+0000", 2024-09-13T14:44:41.639 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.7608999999999995e-05 2024-09-13T14:44:41.639 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.639 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.640 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:44:41.640 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.554093+0000", 2024-09-13T14:44:41.640 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014612699999999999 2024-09-13T14:44:41.640 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.640 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.640 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:44:41.640 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.554225+0000", 2024-09-13T14:44:41.640 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000132115 2024-09-13T14:44:41.640 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.640 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.641 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:44:41.641 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.556573+0000", 2024-09-13T14:44:41.641 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0023471070000000002 2024-09-13T14:44:41.641 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.641 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.641 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:41.641 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.559652+0000", 2024-09-13T14:44:41.641 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.003079591 2024-09-13T14:44:41.641 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.641 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.641 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:41.641 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.560749+0000", 2024-09-13T14:44:41.641 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001097156 2024-09-13T14:44:41.642 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.642 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.642 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:44:41.642 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.653812+0000", 2024-09-13T14:44:41.642 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.093062244000000002 2024-09-13T14:44:41.642 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.642 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.642 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:44:41.642 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.653848+0000", 2024-09-13T14:44:41.642 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.6637000000000002e-05 2024-09-13T14:44:41.642 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.642 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:44:41.642 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.642 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.643 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.643 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:63701 6.7 6:e60a1a3a:::benchmark_data_smithi151_155836_object63700:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1088)", 2024-09-13T14:44:41.643 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:39:55.554728+0000", 2024-09-13T14:44:41.643 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 286.077450489, 2024-09-13T14:44:41.643 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.103798987, 2024-09-13T14:44:41.643 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:44:41.643 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:41.643 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:44:41.643 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:44:41.643 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:44:41.643 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 63701 2024-09-13T14:44:41.643 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.643 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:44:41.644 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.644 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:44:41.644 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.554728+0000", 2024-09-13T14:44:41.644 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.644 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.644 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.644 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:44:41.644 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.554728+0000", 2024-09-13T14:44:41.644 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.644 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.644 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.644 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:44:41.644 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.554727+0000", 2024-09-13T14:44:41.644 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:44:41.645 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.645 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.645 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:44:41.645 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.554780+0000", 2024-09-13T14:44:41.645 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3207000000000001e-05 2024-09-13T14:44:41.645 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.645 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.645 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:44:41.645 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.554791+0000", 2024-09-13T14:44:41.645 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0203e-05 2024-09-13T14:44:41.645 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.645 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.645 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:44:41.646 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.554798+0000", 2024-09-13T14:44:41.646 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.4900000000000003e-06 2024-09-13T14:44:41.646 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.646 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.646 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:44:41.646 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.554851+0000", 2024-09-13T14:44:41.646 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.329e-05 2024-09-13T14:44:41.646 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.646 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.646 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:44:41.646 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.554972+0000", 2024-09-13T14:44:41.646 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000120913 2024-09-13T14:44:41.646 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.646 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.647 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:44:41.647 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.555100+0000", 2024-09-13T14:44:41.647 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012752099999999999 2024-09-13T14:44:41.647 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.647 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.647 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:41.647 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.559350+0000", 2024-09-13T14:44:41.647 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0042505939999999999 2024-09-13T14:44:41.647 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.647 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.647 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:44:41.647 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.560710+0000", 2024-09-13T14:44:41.647 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001359114 2024-09-13T14:44:41.648 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.648 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.648 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:41.648 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.658396+0000", 2024-09-13T14:44:41.648 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.097686440999999999 2024-09-13T14:44:41.648 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.648 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.648 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:44:41.648 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.658483+0000", 2024-09-13T14:44:41.648 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.7247e-05 2024-09-13T14:44:41.648 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.648 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.648 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:44:41.649 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.658527+0000", 2024-09-13T14:44:41.649 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3946000000000002e-05 2024-09-13T14:44:41.649 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.649 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:44:41.649 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.649 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.649 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.649 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35289.0:63699 6.4 e1088/1082 6:28895231:::benchmark_data_smithi151_155836_object63698:head v 1088'4016, mlcod=1088'4013)", 2024-09-13T14:44:41.649 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:39:55.554792+0000", 2024-09-13T14:44:41.649 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 286.077387079, 2024-09-13T14:44:41.649 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.099114134000000007, 2024-09-13T14:44:41.649 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:44:41.649 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:41.649 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:44:41.650 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.650 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:44:41.650 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.554792+0000", 2024-09-13T14:44:41.650 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.650 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.650 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.651 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:44:41.651 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.554792+0000", 2024-09-13T14:44:41.651 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.651 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.651 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.651 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:44:41.651 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.554791+0000", 2024-09-13T14:44:41.651 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:44:41.651 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.651 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.651 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:44:41.651 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.554865+0000", 2024-09-13T14:44:41.652 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.4043999999999998e-05 2024-09-13T14:44:41.652 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.652 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.652 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:44:41.652 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.554880+0000", 2024-09-13T14:44:41.652 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.451e-05 2024-09-13T14:44:41.652 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.652 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.652 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:44:41.652 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.554885+0000", 2024-09-13T14:44:41.652 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3900000000000001e-06 2024-09-13T14:44:41.652 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.653 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.653 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:44:41.653 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.555017+0000", 2024-09-13T14:44:41.653 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013219499999999999 2024-09-13T14:44:41.653 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.653 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.653 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:44:41.653 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.555099+0000", 2024-09-13T14:44:41.653 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.2146999999999998e-05 2024-09-13T14:44:41.653 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.653 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.653 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:44:41.653 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.653879+0000", 2024-09-13T14:44:41.653 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.098779994999999995 2024-09-13T14:44:41.654 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.654 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.654 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:44:41.654 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.653906+0000", 2024-09-13T14:44:41.654 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.6526e-05 2024-09-13T14:44:41.654 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.654 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:44:41.654 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.654 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.654 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.654 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35289.0:63700 6.0 e1088/1082 6:0866df32:::benchmark_data_smithi151_155836_object63699:head v 1088'4087, mlcod=1088'4084)", 2024-09-13T14:44:41.654 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:39:55.557111+0000", 2024-09-13T14:44:41.654 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 286.075067811, 2024-09-13T14:44:41.655 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.096669219000000001, 2024-09-13T14:44:41.655 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:44:41.655 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:41.655 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:44:41.655 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.655 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:44:41.655 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.557111+0000", 2024-09-13T14:44:41.655 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.655 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.655 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.655 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:44:41.655 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.557111+0000", 2024-09-13T14:44:41.655 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.655 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.656 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.656 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:44:41.656 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.557110+0000", 2024-09-13T14:44:41.656 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:44:41.656 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.656 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.656 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:44:41.656 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.557142+0000", 2024-09-13T14:44:41.656 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.1532999999999998e-05 2024-09-13T14:44:41.656 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.656 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.656 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:44:41.656 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.557153+0000", 2024-09-13T14:44:41.657 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1586000000000001e-05 2024-09-13T14:44:41.657 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.657 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.657 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:44:41.657 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.557159+0000", 2024-09-13T14:44:41.657 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4399999999999996e-06 2024-09-13T14:44:41.657 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.657 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.657 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:44:41.657 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.557292+0000", 2024-09-13T14:44:41.657 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013345699999999999 2024-09-13T14:44:41.657 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.657 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.657 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:44:41.658 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.557338+0000", 2024-09-13T14:44:41.658 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5831000000000002e-05 2024-09-13T14:44:41.658 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.658 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.658 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:44:41.658 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.560748+0000", 2024-09-13T14:44:41.658 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.003409816 2024-09-13T14:44:41.658 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.658 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.658 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:44:41.658 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.653780+0000", 2024-09-13T14:44:41.658 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.093032411999999995 2024-09-13T14:44:41.658 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.658 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:44:41.659 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.659 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.659 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.659 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35289.0:63706 6.c e1088/1082 6:395e08a0:::benchmark_data_smithi151_155836_object63705:head v 1088'3943, mlcod=1088'3940)", 2024-09-13T14:44:41.659 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:39:55.560440+0000", 2024-09-13T14:44:41.659 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 286.07173849100002, 2024-09-13T14:44:41.659 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.094317200000000004, 2024-09-13T14:44:41.659 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:44:41.659 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:41.659 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:44:41.659 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.659 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:44:41.659 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.560440+0000", 2024-09-13T14:44:41.659 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.660 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.660 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.660 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:44:41.660 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.560440+0000", 2024-09-13T14:44:41.660 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.660 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.660 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.660 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:44:41.660 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.560439+0000", 2024-09-13T14:44:41.660 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:44:41.660 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.660 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.660 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:44:41.661 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.560463+0000", 2024-09-13T14:44:41.661 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.3305000000000001e-05 2024-09-13T14:44:41.661 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.661 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.661 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:44:41.661 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.560474+0000", 2024-09-13T14:44:41.661 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.095e-05 2024-09-13T14:44:41.661 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.661 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.661 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:44:41.661 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.560479+0000", 2024-09-13T14:44:41.661 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1460000000000004e-06 2024-09-13T14:44:41.661 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.661 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.662 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:44:41.662 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.560538+0000", 2024-09-13T14:44:41.662 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.9098000000000002e-05 2024-09-13T14:44:41.662 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.662 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.662 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:44:41.662 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.560555+0000", 2024-09-13T14:44:41.662 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.7515000000000001e-05 2024-09-13T14:44:41.662 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.662 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.662 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:44:41.662 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.654721+0000", 2024-09-13T14:44:41.662 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.094165993000000003 2024-09-13T14:44:41.662 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.663 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.663 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:44:41.663 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.654757+0000", 2024-09-13T14:44:41.663 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.6140999999999998e-05 2024-09-13T14:44:41.663 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.663 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:44:41.663 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.663 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.663 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.663 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35289.0:63707 6.4 e1088/1082 6:2668e18c:::benchmark_data_smithi151_155836_object63706:head v 1088'4017, mlcod=1088'4014)", 2024-09-13T14:44:41.663 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:39:55.561027+0000", 2024-09-13T14:44:41.663 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 286.07115205500003, 2024-09-13T14:44:41.663 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.104750317, 2024-09-13T14:44:41.663 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:44:41.664 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:41.664 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:44:41.664 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.664 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:44:41.664 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.561027+0000", 2024-09-13T14:44:41.664 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.664 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.664 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.664 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:44:41.664 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.561027+0000", 2024-09-13T14:44:41.664 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.664 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.664 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.664 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:44:41.665 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.561022+0000", 2024-09-13T14:44:41.665 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999952 2024-09-13T14:44:41.665 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.665 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.665 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:44:41.665 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.561097+0000", 2024-09-13T14:44:41.665 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.4924000000000003e-05 2024-09-13T14:44:41.665 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.665 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.665 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:44:41.665 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.653753+0000", 2024-09-13T14:44:41.665 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.092655536999999996 2024-09-13T14:44:41.666 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.666 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.666 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:44:41.666 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.653763+0000", 2024-09-13T14:44:41.666 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0757000000000001e-05 2024-09-13T14:44:41.666 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.666 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.666 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:44:41.666 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.653942+0000", 2024-09-13T14:44:41.666 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00017865000000000001 2024-09-13T14:44:41.666 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.666 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.666 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:44:41.666 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.653959+0000", 2024-09-13T14:44:41.667 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.7435000000000002e-05 2024-09-13T14:44:41.667 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.667 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.667 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:44:41.667 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.665720+0000", 2024-09-13T14:44:41.667 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.011760115999999999 2024-09-13T14:44:41.667 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.667 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.667 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:44:41.667 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.665777+0000", 2024-09-13T14:44:41.667 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7448000000000003e-05 2024-09-13T14:44:41.667 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.667 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:44:41.668 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.668 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.668 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.668 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35289.0:63708 6.0 e1088/1082 6:0c0735af:::benchmark_data_smithi151_155836_object63707:head v 1088'4088, mlcod=1088'4084)", 2024-09-13T14:44:41.668 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:39:55.561371+0000", 2024-09-13T14:44:41.668 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 286.07080793, 2024-09-13T14:44:41.668 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.096990879000000002, 2024-09-13T14:44:41.668 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:44:41.668 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:41.668 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:44:41.668 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.668 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:44:41.668 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.561371+0000", 2024-09-13T14:44:41.669 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.669 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.669 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.669 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:44:41.669 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.561371+0000", 2024-09-13T14:44:41.669 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.669 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.669 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.669 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:44:41.669 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.561370+0000", 2024-09-13T14:44:41.669 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:44:41.669 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.669 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.669 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:44:41.670 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.561412+0000", 2024-09-13T14:44:41.670 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1776000000000001e-05 2024-09-13T14:44:41.670 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.670 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.670 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:44:41.670 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.653770+0000", 2024-09-13T14:44:41.670 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.092357894999999995 2024-09-13T14:44:41.670 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.670 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.670 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:44:41.670 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.653779+0000", 2024-09-13T14:44:41.670 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.0729999999999994e-06 2024-09-13T14:44:41.670 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:44:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.653816+0000", 2024-09-13T14:44:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.6788000000000001e-05 2024-09-13T14:44:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:44:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.653836+0000", 2024-09-13T14:44:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.0145000000000001e-05 2024-09-13T14:44:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:44:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.658280+0000", 2024-09-13T14:44:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0044442220000000003 2024-09-13T14:44:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:44:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.658362+0000", 2024-09-13T14:44:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.1701999999999998e-05 2024-09-13T14:44:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:44:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:63709 6.7 6:e60e3a7e:::benchmark_data_smithi151_155836_object63708:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1088)", 2024-09-13T14:44:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:39:55.561779+0000", 2024-09-13T14:44:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 286.07039982399999, 2024-09-13T14:44:41.673 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.096919175999999996, 2024-09-13T14:44:41.673 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:44:41.673 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:41.673 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:44:41.673 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:44:41.673 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:44:41.673 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 63709 2024-09-13T14:44:41.673 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.673 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:44:41.673 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.674 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:44:41.674 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.561779+0000", 2024-09-13T14:44:41.674 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.674 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.674 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.674 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:44:41.674 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.561779+0000", 2024-09-13T14:44:41.674 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.674 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.674 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.674 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:44:41.674 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.561777+0000", 2024-09-13T14:44:41.674 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:44:41.674 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.675 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.675 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:44:41.675 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.561822+0000", 2024-09-13T14:44:41.675 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5358000000000002e-05 2024-09-13T14:44:41.675 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.675 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.675 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:44:41.675 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.653739+0000", 2024-09-13T14:44:41.675 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.091916838000000001 2024-09-13T14:44:41.675 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.675 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.675 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:44:41.675 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.653754+0000", 2024-09-13T14:44:41.675 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4585e-05 2024-09-13T14:44:41.676 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.676 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.676 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:44:41.676 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.653814+0000", 2024-09-13T14:44:41.676 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.9972000000000001e-05 2024-09-13T14:44:41.676 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.676 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.676 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:44:41.676 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.653962+0000", 2024-09-13T14:44:41.676 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0001483 2024-09-13T14:44:41.676 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.676 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.677 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:44:41.677 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.654059+0000", 2024-09-13T14:44:41.677 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.6717000000000003e-05 2024-09-13T14:44:41.677 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.677 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.677 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:44:41.677 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.658545+0000", 2024-09-13T14:44:41.677 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0044865210000000003 2024-09-13T14:44:41.677 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.677 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.677 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:41.677 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.658577+0000", 2024-09-13T14:44:41.677 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.1501999999999998e-05 2024-09-13T14:44:41.677 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.678 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.678 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:41.678 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.658622+0000", 2024-09-13T14:44:41.678 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4981000000000002e-05 2024-09-13T14:44:41.678 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.678 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.678 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:44:41.678 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.658665+0000", 2024-09-13T14:44:41.678 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.2626999999999997e-05 2024-09-13T14:44:41.678 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.678 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.678 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:44:41.678 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:39:55.658698+0000", 2024-09-13T14:44:41.678 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3519000000000002e-05 2024-09-13T14:44:41.679 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.679 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:44:41.679 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.679 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.679 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.679 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35289.0:168701 6.e e1106/1103 6:777e91d5:::benchmark_data_smithi151_155836_object36243:head v 1106'10444, mlcod=1106'10441)", 2024-09-13T14:44:41.679 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:41.291594+0000", 2024-09-13T14:44:41.679 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 180.340584425, 2024-09-13T14:44:41.679 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.102177539, 2024-09-13T14:44:41.679 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:44:41.679 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:41.679 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:44:41.679 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.680 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:44:41.680 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.291594+0000", 2024-09-13T14:44:41.680 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.680 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.680 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.680 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:44:41.680 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.291594+0000", 2024-09-13T14:44:41.680 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.680 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.680 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.680 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:44:41.680 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.291594+0000", 2024-09-13T14:44:41.680 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:44:41.681 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.681 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.681 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:44:41.681 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.291599+0000", 2024-09-13T14:44:41.681 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3870000000000003e-06 2024-09-13T14:44:41.681 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.681 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.681 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:44:41.681 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.291609+0000", 2024-09-13T14:44:41.681 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.38e-06 2024-09-13T14:44:41.681 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.681 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.681 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:44:41.682 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.291613+0000", 2024-09-13T14:44:41.682 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.977e-06 2024-09-13T14:44:41.682 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.682 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.682 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:44:41.682 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.291671+0000", 2024-09-13T14:44:41.682 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7175000000000001e-05 2024-09-13T14:44:41.682 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.682 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.682 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:44:41.682 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.291711+0000", 2024-09-13T14:44:41.682 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0333000000000002e-05 2024-09-13T14:44:41.682 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.683 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.683 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:44:41.683 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.393746+0000", 2024-09-13T14:44:41.683 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.102035468 2024-09-13T14:44:41.683 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.683 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.683 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:44:41.683 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.393772+0000", 2024-09-13T14:44:41.683 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.5358999999999999e-05 2024-09-13T14:44:41.683 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.683 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:44:41.683 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.683 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.683 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.684 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:168703 6.d 6:ba3d7371:::benchmark_data_smithi151_155836_object36245:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1106)", 2024-09-13T14:44:41.684 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:41.293491+0000", 2024-09-13T14:44:41.684 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 180.338688039, 2024-09-13T14:44:41.684 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.112706482, 2024-09-13T14:44:41.684 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:44:41.684 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:41.684 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:44:41.684 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:44:41.684 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:44:41.684 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 168703 2024-09-13T14:44:41.684 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.684 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:44:41.684 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.685 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:44:41.685 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.293491+0000", 2024-09-13T14:44:41.685 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.685 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.685 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.685 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:44:41.685 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.293491+0000", 2024-09-13T14:44:41.685 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.685 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.685 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.685 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:44:41.685 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.293490+0000", 2024-09-13T14:44:41.685 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:44:41.685 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.686 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.686 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:44:41.686 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.293495+0000", 2024-09-13T14:44:41.686 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9609999999999996e-06 2024-09-13T14:44:41.686 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.686 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.686 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:44:41.686 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.293501+0000", 2024-09-13T14:44:41.686 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.8050000000000001e-06 2024-09-13T14:44:41.686 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.686 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.686 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:44:41.686 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.293505+0000", 2024-09-13T14:44:41.686 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.546e-06 2024-09-13T14:44:41.686 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.686 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.687 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:44:41.687 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.293538+0000", 2024-09-13T14:44:41.687 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.2848999999999997e-05 2024-09-13T14:44:41.687 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.687 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.687 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:44:41.687 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.294788+0000", 2024-09-13T14:44:41.687 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0012499189999999999 2024-09-13T14:44:41.687 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.687 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.687 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,1,2", 2024-09-13T14:44:41.687 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.294916+0000", 2024-09-13T14:44:41.687 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000128121 2024-09-13T14:44:41.687 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.688 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.688 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:41.688 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.394228+0000", 2024-09-13T14:44:41.688 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.099312127 2024-09-13T14:44:41.688 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.688 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.688 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:41.688 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.394260+0000", 2024-09-13T14:44:41.688 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.1872999999999998e-05 2024-09-13T14:44:41.688 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.688 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.688 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:41.688 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.394294+0000", 2024-09-13T14:44:41.688 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3739999999999999e-05 2024-09-13T14:44:41.689 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.689 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.689 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:44:41.689 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.406039+0000", 2024-09-13T14:44:41.689 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.011744967 2024-09-13T14:44:41.689 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.689 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.689 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:44:41.689 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.406088+0000", 2024-09-13T14:44:41.689 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9598000000000002e-05 2024-09-13T14:44:41.689 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.689 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.689 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:44:41.689 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.406197+0000", 2024-09-13T14:44:41.690 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000109005 2024-09-13T14:44:41.690 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.690 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:44:41.690 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.690 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.690 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.690 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:168706 6.7 6:ead8e1a5:::benchmark_data_smithi151_155836_object36248:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1106)", 2024-09-13T14:44:41.690 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:41.295842+0000", 2024-09-13T14:44:41.690 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 180.33633633400001, 2024-09-13T14:44:41.690 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.111450719, 2024-09-13T14:44:41.690 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:44:41.690 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:41.690 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:44:41.690 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:44:41.690 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:44:41.691 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 168706 2024-09-13T14:44:41.691 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.691 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:44:41.691 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.691 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:44:41.691 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.295842+0000", 2024-09-13T14:44:41.691 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.691 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.691 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.691 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:44:41.691 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.295842+0000", 2024-09-13T14:44:41.691 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.691 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.691 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.692 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:44:41.692 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.295841+0000", 2024-09-13T14:44:41.692 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:44:41.692 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.692 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.692 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:44:41.692 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.295851+0000", 2024-09-13T14:44:41.692 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.5510000000000005e-06 2024-09-13T14:44:41.692 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.692 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.692 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:44:41.692 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.391217+0000", 2024-09-13T14:44:41.693 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.095366485000000001 2024-09-13T14:44:41.693 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.693 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.693 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:44:41.693 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.391233+0000", 2024-09-13T14:44:41.693 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.5559000000000001e-05 2024-09-13T14:44:41.693 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.693 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.693 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:44:41.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:41.694 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:41.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:41.695 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.391287+0000", 2024-09-13T14:44:41.695 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4583000000000002e-05 2024-09-13T14:44:41.695 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.695 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.696 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:44:41.696 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.391400+0000", 2024-09-13T14:44:41.696 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000112866 2024-09-13T14:44:41.696 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.696 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.696 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:44:41.696 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.391470+0000", 2024-09-13T14:44:41.696 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.0211999999999998e-05 2024-09-13T14:44:41.696 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.696 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.696 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:41.696 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.406108+0000", 2024-09-13T14:44:41.696 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.014638165999999999 2024-09-13T14:44:41.697 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.697 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.697 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:44:41.697 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.406178+0000", 2024-09-13T14:44:41.697 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.9170000000000004e-05 2024-09-13T14:44:41.697 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.697 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.697 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:41.697 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.407225+0000", 2024-09-13T14:44:41.697 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001047426 2024-09-13T14:44:41.697 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.697 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.697 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:44:41.697 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.407262+0000", 2024-09-13T14:44:41.698 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.6599e-05 2024-09-13T14:44:41.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.698 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.698 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:44:41.698 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.407293+0000", 2024-09-13T14:44:41.698 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.1402000000000002e-05 2024-09-13T14:44:41.698 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.698 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:44:41.698 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.698 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.698 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop_reply(client.35289.0:168703 6.d e1106/1103 ondisk, result = 0)", 2024-09-13T14:44:41.698 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:41.298006+0000", 2024-09-13T14:44:41.699 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 180.33417249999999, 2024-09-13T14:44:41.699 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.096223864000000006, 2024-09-13T14:44:41.699 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:44:41.699 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:44:41.699 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:44:41.699 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.699 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:44:41.699 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.298006+0000", 2024-09-13T14:44:41.699 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.699 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.699 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.699 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:44:41.699 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.298006+0000", 2024-09-13T14:44:41.700 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.700 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.700 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.700 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:44:41.700 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.298006+0000", 2024-09-13T14:44:41.700 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:44:41.700 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.700 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.700 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:44:41.700 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.298012+0000", 2024-09-13T14:44:41.700 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.0889999999999996e-06 2024-09-13T14:44:41.700 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.700 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.701 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:44:41.701 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.391219+0000", 2024-09-13T14:44:41.701 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.093207180000000001 2024-09-13T14:44:41.701 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.701 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.701 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:44:41.701 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.391231+0000", 2024-09-13T14:44:41.701 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2082999999999999e-05 2024-09-13T14:44:41.701 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.701 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.701 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:44:41.701 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.394214+0000", 2024-09-13T14:44:41.701 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0029830989999999999 2024-09-13T14:44:41.701 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.702 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.702 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:44:41.702 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.394221+0000", 2024-09-13T14:44:41.702 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.4719999999999998e-06 2024-09-13T14:44:41.702 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.702 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.702 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:44:41.702 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:41.394230+0000", 2024-09-13T14:44:41.702 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.6389999999999998e-06 2024-09-13T14:44:41.702 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.702 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:44:41.702 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.703 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.703 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.703 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:213528 6.4 6:211c997c:::benchmark_data_smithi151_155836_object81070:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1118)", 2024-09-13T14:44:41.703 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:42:27.920170+0000", 2024-09-13T14:44:41.703 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 133.71200850599999, 2024-09-13T14:44:41.703 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.093724460999999995, 2024-09-13T14:44:41.703 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:44:41.703 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:41.703 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:44:41.703 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:44:41.703 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:44:41.703 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 213528 2024-09-13T14:44:41.703 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.704 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:44:41.704 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.704 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:44:41.704 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:27.920170+0000", 2024-09-13T14:44:41.704 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.704 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.704 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.704 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:44:41.704 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:27.920170+0000", 2024-09-13T14:44:41.704 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.704 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.704 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.705 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:44:41.705 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:27.920169+0000", 2024-09-13T14:44:41.705 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:44:41.705 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.705 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.705 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:44:41.705 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:27.920175+0000", 2024-09-13T14:44:41.705 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7320000000000001e-06 2024-09-13T14:44:41.705 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.705 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.705 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:44:41.706 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:27.920188+0000", 2024-09-13T14:44:41.706 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2051e-05 2024-09-13T14:44:41.706 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.706 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.706 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:44:41.706 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:27.920193+0000", 2024-09-13T14:44:41.706 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4580000000000001e-06 2024-09-13T14:44:41.706 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.706 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.706 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:44:41.706 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:27.923793+0000", 2024-09-13T14:44:41.706 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0035997300000000002 2024-09-13T14:44:41.706 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.707 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.707 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:44:41.707 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:27.923911+0000", 2024-09-13T14:44:41.707 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00011877799999999999 2024-09-13T14:44:41.707 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.707 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.707 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:44:41.707 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:27.924327+0000", 2024-09-13T14:44:41.707 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00041556500000000001 2024-09-13T14:44:41.707 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.707 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.707 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:44:41.707 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:27.927280+0000", 2024-09-13T14:44:41.707 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002952849 2024-09-13T14:44:41.708 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.708 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.708 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:41.708 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:27.927648+0000", 2024-09-13T14:44:41.708 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00036769199999999998 2024-09-13T14:44:41.708 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.708 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.708 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:41.708 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.013750+0000", 2024-09-13T14:44:41.708 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.086102445999999999 2024-09-13T14:44:41.708 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.708 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.709 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:44:41.709 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.013847+0000", 2024-09-13T14:44:41.709 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.7079999999999999e-05 2024-09-13T14:44:41.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.709 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.709 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:44:41.709 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.013895+0000", 2024-09-13T14:44:41.709 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7559999999999999e-05 2024-09-13T14:44:41.709 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.709 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:44:41.709 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.709 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.710 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:59828 7.0 7:07a57938:::benchmark_data_smithi151_190906_object59827:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1142)", 2024-09-13T14:44:41.710 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:44:31.726568+0000", 2024-09-13T14:44:41.710 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 9.9056111579999993, 2024-09-13T14:44:41.710 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.106949052, 2024-09-13T14:44:41.710 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:44:41.710 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:41.710 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:44:41.710 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:44:41.710 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:44:41.710 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 59828 2024-09-13T14:44:41.710 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.710 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:44:41.710 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.711 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:44:41.711 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:31.726568+0000", 2024-09-13T14:44:41.711 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.711 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.711 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.711 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:44:41.711 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:31.726568+0000", 2024-09-13T14:44:41.711 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.711 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.711 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.711 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:44:41.711 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:31.726566+0000", 2024-09-13T14:44:41.711 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:44:41.711 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.712 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.712 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:44:41.712 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:31.726629+0000", 2024-09-13T14:44:41.712 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3115999999999997e-05 2024-09-13T14:44:41.712 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.712 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.712 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:44:41.712 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:31.726642+0000", 2024-09-13T14:44:41.712 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3241000000000001e-05 2024-09-13T14:44:41.712 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.712 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.712 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:44:41.712 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:31.726649+0000", 2024-09-13T14:44:41.713 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.6789999999999999e-06 2024-09-13T14:44:41.713 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.713 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.713 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:44:41.713 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:31.726693+0000", 2024-09-13T14:44:41.713 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3810000000000002e-05 2024-09-13T14:44:41.713 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.713 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.713 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:44:41.713 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:31.726828+0000", 2024-09-13T14:44:41.713 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013518100000000001 2024-09-13T14:44:41.713 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.713 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.714 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:44:41.714 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:31.726988+0000", 2024-09-13T14:44:41.714 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00015995399999999999 2024-09-13T14:44:41.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.714 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.714 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:44:41.714 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:31.728940+0000", 2024-09-13T14:44:41.714 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0019522529999999999 2024-09-13T14:44:41.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.714 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.714 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:41.714 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:31.729834+0000", 2024-09-13T14:44:41.714 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00089389199999999997 2024-09-13T14:44:41.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.715 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.715 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:41.715 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:31.833417+0000", 2024-09-13T14:44:41.715 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.10358323899999999 2024-09-13T14:44:41.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.715 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.715 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:44:41.715 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:31.833479+0000", 2024-09-13T14:44:41.715 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1743999999999997e-05 2024-09-13T14:44:41.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.715 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.715 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:44:41.715 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:31.833517+0000", 2024-09-13T14:44:41.716 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.7540000000000003e-05 2024-09-13T14:44:41.716 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.716 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:44:41.716 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.716 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.716 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.716 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:60538 7.0 7:01747ade:::benchmark_data_smithi151_190906_object60537:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1142)", 2024-09-13T14:44:41.716 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:44:32.324096+0000", 2024-09-13T14:44:41.716 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 9.3080822649999995, 2024-09-13T14:44:41.716 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.15480245200000001, 2024-09-13T14:44:41.716 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:44:41.716 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:41.717 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:44:41.717 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:44:41.717 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:44:41.717 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 60538 2024-09-13T14:44:41.717 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.717 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:44:41.717 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.717 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:44:41.717 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.324096+0000", 2024-09-13T14:44:41.717 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.717 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.717 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.717 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:44:41.718 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.324096+0000", 2024-09-13T14:44:41.718 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.718 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.718 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.718 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:44:41.718 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.324094+0000", 2024-09-13T14:44:41.718 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:44:41.718 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.718 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.718 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:44:41.718 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.324133+0000", 2024-09-13T14:44:41.718 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9070999999999999e-05 2024-09-13T14:44:41.718 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.718 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.719 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:44:41.719 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.324156+0000", 2024-09-13T14:44:41.719 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.2197000000000001e-05 2024-09-13T14:44:41.719 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.719 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.719 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:44:41.719 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.324164+0000", 2024-09-13T14:44:41.719 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.1799999999999996e-06 2024-09-13T14:44:41.719 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.719 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.719 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:44:41.719 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.324219+0000", 2024-09-13T14:44:41.719 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5053000000000003e-05 2024-09-13T14:44:41.720 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.720 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.720 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:44:41.720 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.324351+0000", 2024-09-13T14:44:41.720 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013169700000000001 2024-09-13T14:44:41.720 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.720 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.720 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:44:41.720 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.324501+0000", 2024-09-13T14:44:41.720 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000149925 2024-09-13T14:44:41.720 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.720 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.720 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:41.721 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.326686+0000", 2024-09-13T14:44:41.721 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00218534 2024-09-13T14:44:41.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.721 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.721 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:44:41.721 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.326835+0000", 2024-09-13T14:44:41.721 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014893400000000001 2024-09-13T14:44:41.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.721 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.721 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:41.721 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.478806+0000", 2024-09-13T14:44:41.721 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.15197136999999999 2024-09-13T14:44:41.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.722 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.722 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:44:41.722 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.478856+0000", 2024-09-13T14:44:41.722 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9458e-05 2024-09-13T14:44:41.722 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.722 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.722 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:44:41.722 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.478899+0000", 2024-09-13T14:44:41.722 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3276999999999999e-05 2024-09-13T14:44:41.722 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.722 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:44:41.722 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.722 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.722 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.723 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:61028 7.0 7:0497349d:::benchmark_data_smithi151_190906_object61027:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1142)", 2024-09-13T14:44:41.723 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:44:32.778762+0000", 2024-09-13T14:44:41.723 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 8.8534172049999995, 2024-09-13T14:44:41.723 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.118253609, 2024-09-13T14:44:41.723 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:44:41.723 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:41.723 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:44:41.723 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:44:41.723 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:44:41.723 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 61028 2024-09-13T14:44:41.723 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.723 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:44:41.723 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.724 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:44:41.724 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.778762+0000", 2024-09-13T14:44:41.724 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.724 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.724 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.724 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:44:41.724 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.778762+0000", 2024-09-13T14:44:41.724 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.724 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.724 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.724 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:44:41.724 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.778760+0000", 2024-09-13T14:44:41.725 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:44:41.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.725 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.725 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:44:41.725 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.778818+0000", 2024-09-13T14:44:41.725 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8289999999999999e-05 2024-09-13T14:44:41.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.725 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.725 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:44:41.725 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.778830+0000", 2024-09-13T14:44:41.725 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1751e-05 2024-09-13T14:44:41.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.726 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.726 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:44:41.726 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.778836+0000", 2024-09-13T14:44:41.726 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8710000000000002e-06 2024-09-13T14:44:41.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.726 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.726 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:44:41.726 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.778862+0000", 2024-09-13T14:44:41.726 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.5870000000000001e-05 2024-09-13T14:44:41.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.726 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.726 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:44:41.726 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.779427+0000", 2024-09-13T14:44:41.726 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00056503800000000002 2024-09-13T14:44:41.727 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.727 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.727 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:44:41.727 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.779971+0000", 2024-09-13T14:44:41.727 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00054429200000000004 2024-09-13T14:44:41.727 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.727 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.727 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:44:41.727 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.782642+0000", 2024-09-13T14:44:41.727 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0026708069999999999 2024-09-13T14:44:41.727 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.727 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.728 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:41.728 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.785371+0000", 2024-09-13T14:44:41.728 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0027290230000000001 2024-09-13T14:44:41.728 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.728 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.728 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:41.728 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.896861+0000", 2024-09-13T14:44:41.728 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.11149044900000001 2024-09-13T14:44:41.728 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.728 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.728 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:44:41.728 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.896967+0000", 2024-09-13T14:44:41.728 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00010556 2024-09-13T14:44:41.728 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.729 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.729 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:44:41.729 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.897015+0000", 2024-09-13T14:44:41.729 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8199999999999999e-05 2024-09-13T14:44:41.729 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.729 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:44:41.729 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.729 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.729 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.729 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:61044 7.0 7:0c17ec76:::benchmark_data_smithi151_190906_object61043:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1142)", 2024-09-13T14:44:41.729 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:44:32.787807+0000", 2024-09-13T14:44:41.729 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 8.8443712919999999, 2024-09-13T14:44:41.729 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.111674227, 2024-09-13T14:44:41.730 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:44:41.730 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:41.730 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:44:41.730 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:44:41.730 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:44:41.730 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 61044 2024-09-13T14:44:41.730 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.730 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:44:41.730 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.730 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:44:41.730 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.787807+0000", 2024-09-13T14:44:41.730 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.730 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.730 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.731 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:44:41.731 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.787807+0000", 2024-09-13T14:44:41.731 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.731 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.731 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.731 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:44:41.731 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.787806+0000", 2024-09-13T14:44:41.731 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:44:41.731 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.731 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.731 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:44:41.731 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.787953+0000", 2024-09-13T14:44:41.731 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000146969 2024-09-13T14:44:41.731 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.732 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.732 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:44:41.732 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.787969+0000", 2024-09-13T14:44:41.732 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6358000000000002e-05 2024-09-13T14:44:41.732 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.732 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.732 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:44:41.732 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.787975+0000", 2024-09-13T14:44:41.732 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3620000000000002e-06 2024-09-13T14:44:41.732 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.732 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.732 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:44:41.732 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.788021+0000", 2024-09-13T14:44:41.733 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5575999999999999e-05 2024-09-13T14:44:41.733 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.733 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.733 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:44:41.733 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.788166+0000", 2024-09-13T14:44:41.733 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014536599999999999 2024-09-13T14:44:41.733 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.733 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.733 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:44:41.733 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.788324+0000", 2024-09-13T14:44:41.733 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00015798200000000001 2024-09-13T14:44:41.733 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.733 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.734 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:44:41.734 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.790421+0000", 2024-09-13T14:44:41.734 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0020972040000000001 2024-09-13T14:44:41.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.734 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.734 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:41.734 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.790961+0000", 2024-09-13T14:44:41.734 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00053915899999999999 2024-09-13T14:44:41.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.734 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.734 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:41.734 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.899356+0000", 2024-09-13T14:44:41.734 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.108395504 2024-09-13T14:44:41.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.734 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.735 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:44:41.735 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.899443+0000", 2024-09-13T14:44:41.735 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.7049e-05 2024-09-13T14:44:41.735 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.735 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.735 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:44:41.735 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.899482+0000", 2024-09-13T14:44:41.735 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8507999999999998e-05 2024-09-13T14:44:41.735 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.735 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:44:41.735 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.735 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.735 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.736 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:63288 7.0 7:0f2db799:::benchmark_data_smithi151_190906_object63287:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1142)", 2024-09-13T14:44:41.736 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:44:34.959333+0000", 2024-09-13T14:44:41.736 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 6.672845208, 2024-09-13T14:44:41.736 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.13504291399999999, 2024-09-13T14:44:41.736 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:44:41.736 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:41.736 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:44:41.736 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:44:41.736 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:44:41.736 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 63288 2024-09-13T14:44:41.736 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.736 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:44:41.736 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.737 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:44:41.737 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.959333+0000", 2024-09-13T14:44:41.737 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.737 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.737 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.737 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:44:41.737 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.959333+0000", 2024-09-13T14:44:41.737 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.737 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.737 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.737 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:44:41.737 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.959332+0000", 2024-09-13T14:44:41.737 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:44:41.738 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.738 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.738 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:44:41.738 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.959426+0000", 2024-09-13T14:44:41.738 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.3410000000000002e-05 2024-09-13T14:44:41.738 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.738 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.738 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:44:41.738 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.959440+0000", 2024-09-13T14:44:41.738 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4334e-05 2024-09-13T14:44:41.738 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.738 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.738 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:44:41.739 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.959446+0000", 2024-09-13T14:44:41.739 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.0000000000000002e-06 2024-09-13T14:44:41.739 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.739 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.739 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:44:41.739 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.959569+0000", 2024-09-13T14:44:41.739 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012257500000000001 2024-09-13T14:44:41.739 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.739 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.739 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:44:41.739 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.960373+0000", 2024-09-13T14:44:41.739 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00080422200000000005 2024-09-13T14:44:41.739 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.739 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.740 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:44:41.740 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.960509+0000", 2024-09-13T14:44:41.740 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013623199999999999 2024-09-13T14:44:41.740 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.740 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.740 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:44:41.740 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.963086+0000", 2024-09-13T14:44:41.740 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0025767390000000002 2024-09-13T14:44:41.740 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.741 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.741 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:41.741 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.964835+0000", 2024-09-13T14:44:41.741 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0017488950000000001 2024-09-13T14:44:41.741 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.741 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.741 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:41.741 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:35.094270+0000", 2024-09-13T14:44:41.741 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.129435884 2024-09-13T14:44:41.741 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.741 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.741 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:44:41.741 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:35.094330+0000", 2024-09-13T14:44:41.742 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.9240000000000002e-05 2024-09-13T14:44:41.742 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.742 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.742 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:44:41.742 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:35.094376+0000", 2024-09-13T14:44:41.742 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.668e-05 2024-09-13T14:44:41.742 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.742 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:44:41.742 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.742 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.742 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.742 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:63295 7.0 7:05e71afe:::benchmark_data_smithi151_190906_object63294:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1142)", 2024-09-13T14:44:41.742 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:44:34.964725+0000", 2024-09-13T14:44:41.742 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 6.6674539340000001, 2024-09-13T14:44:41.743 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.139807191, 2024-09-13T14:44:41.743 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:44:41.743 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:41.743 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:44:41.743 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:44:41.743 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:44:41.743 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 63295 2024-09-13T14:44:41.743 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.743 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:44:41.743 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.743 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:44:41.743 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.964725+0000", 2024-09-13T14:44:41.743 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.743 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.744 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.744 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:44:41.744 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.964725+0000", 2024-09-13T14:44:41.744 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.744 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.744 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.744 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:44:41.744 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.964699+0000", 2024-09-13T14:44:41.744 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999738 2024-09-13T14:44:41.744 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.744 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.744 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:44:41.744 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.964786+0000", 2024-09-13T14:44:41.745 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.7260000000000001e-05 2024-09-13T14:44:41.745 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.745 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.745 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:44:41.745 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.964798+0000", 2024-09-13T14:44:41.745 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2480000000000001e-05 2024-09-13T14:44:41.745 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.745 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.745 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:44:41.745 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.964804+0000", 2024-09-13T14:44:41.745 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3499999999999996e-06 2024-09-13T14:44:41.745 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.745 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.745 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:44:41.746 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.964938+0000", 2024-09-13T14:44:41.746 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000134793 2024-09-13T14:44:41.746 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.746 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.746 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:44:41.746 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.965079+0000", 2024-09-13T14:44:41.746 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014024999999999999 2024-09-13T14:44:41.746 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.746 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.746 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:44:41.746 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.965234+0000", 2024-09-13T14:44:41.746 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00015550300000000001 2024-09-13T14:44:41.746 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.746 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.747 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:44:41.747 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.967109+0000", 2024-09-13T14:44:41.747 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001874749 2024-09-13T14:44:41.747 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.747 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.747 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:41.747 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:35.094419+0000", 2024-09-13T14:44:41.747 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.127310437 2024-09-13T14:44:41.747 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.747 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.747 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:41.747 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:35.104436+0000", 2024-09-13T14:44:41.747 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.010016166 2024-09-13T14:44:41.747 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.748 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.748 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:44:41.748 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:35.104491+0000", 2024-09-13T14:44:41.748 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5787e-05 2024-09-13T14:44:41.748 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.748 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.748 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:44:41.748 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:35.104532+0000", 2024-09-13T14:44:41.748 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0618999999999999e-05 2024-09-13T14:44:41.748 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.748 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:44:41.748 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.748 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.749 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.749 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:65049 7.0 7:03a43fd6:::benchmark_data_smithi151_190906_object65048:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1142)", 2024-09-13T14:44:41.749 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:44:36.352061+0000", 2024-09-13T14:44:41.749 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 5.2801174550000001, 2024-09-13T14:44:41.749 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.168600316, 2024-09-13T14:44:41.749 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:44:41.749 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:41.749 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:44:41.749 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:44:41.749 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:44:41.749 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 65049 2024-09-13T14:44:41.749 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.749 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:44:41.750 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.750 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:44:41.750 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:36.352061+0000", 2024-09-13T14:44:41.750 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.750 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.750 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.750 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:44:41.750 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:36.352061+0000", 2024-09-13T14:44:41.750 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:41.750 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.750 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.750 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:44:41.750 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:36.352059+0000", 2024-09-13T14:44:41.750 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999976 2024-09-13T14:44:41.751 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.751 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.751 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:44:41.751 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:36.352098+0000", 2024-09-13T14:44:41.751 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8430000000000003e-05 2024-09-13T14:44:41.751 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.751 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.751 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:44:41.751 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:36.352123+0000", 2024-09-13T14:44:41.751 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.5265e-05 2024-09-13T14:44:41.751 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.751 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.751 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:44:41.751 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:36.352130+0000", 2024-09-13T14:44:41.752 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.6179999999999997e-06 2024-09-13T14:44:41.752 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.752 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.752 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:44:41.752 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:36.352178+0000", 2024-09-13T14:44:41.752 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7389000000000003e-05 2024-09-13T14:44:41.752 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.752 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.752 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:44:41.752 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:36.352331+0000", 2024-09-13T14:44:41.752 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000153041 2024-09-13T14:44:41.752 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.752 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.752 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:44:41.753 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:36.352480+0000", 2024-09-13T14:44:41.753 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014935999999999999 2024-09-13T14:44:41.753 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.753 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.753 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:44:41.753 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:36.356134+0000", 2024-09-13T14:44:41.753 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.003653482 2024-09-13T14:44:41.753 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.753 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.753 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:41.753 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:36.356306+0000", 2024-09-13T14:44:41.753 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000171841 2024-09-13T14:44:41.753 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.754 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.754 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:41.754 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:36.520559+0000", 2024-09-13T14:44:41.754 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.16425332600000001 2024-09-13T14:44:41.754 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.754 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.754 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:44:41.754 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:36.520614+0000", 2024-09-13T14:44:41.754 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4685999999999998e-05 2024-09-13T14:44:41.754 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:41.754 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:41.754 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:44:41.754 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:36.520662+0000", 2024-09-13T14:44:41.754 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8026999999999999e-05 2024-09-13T14:44:41.755 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.755 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:44:41.755 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.755 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:41.755 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:44:41.755 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:44:41.755 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:2024-09-13T14:44:41.726636+0000 min lat: 0.00109766 max lat: 0.257165 avg lat: 0.0089196 2024-09-13T14:44:41.755 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:44:41.756 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 40 16 71743 71727 112.053 85.625 0.0191795 0.0089196 2024-09-13T14:44:41.796 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:41.826 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:41.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:41.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:41.927 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:41.959 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:42.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:42.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:42.327 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:42.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:42.353 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:42.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:42.454 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:42.495 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:42.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:42.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:42.597 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:42.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:42.637 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:42.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:42.726 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 41 16 73183 73167 111.514 90 0.015351 0.00896268 2024-09-13T14:44:42.738 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:42.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:42.774 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:42.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:42.875 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:42.912 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:42.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:42.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:42.961 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T14:44:43.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:43.071 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:43.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:43.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:42 smithi184 ceph-mon[92527]: pgmap v1506: 119 pgs: 119 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-09-13T14:44:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:42 smithi184 ceph-mon[92527]: 1.18 scrub starts 2024-09-13T14:44:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:42 smithi184 ceph-mon[92527]: 1.18 scrub ok 2024-09-13T14:44:43.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:42 smithi151 ceph-mon[104450]: pgmap v1506: 119 pgs: 119 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-09-13T14:44:43.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:42 smithi151 ceph-mon[104450]: 1.18 scrub starts 2024-09-13T14:44:43.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:42 smithi151 ceph-mon[104450]: 1.18 scrub ok 2024-09-13T14:44:43.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:42 smithi151 ceph-mon[108966]: pgmap v1506: 119 pgs: 119 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-09-13T14:44:43.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:42 smithi151 ceph-mon[108966]: 1.18 scrub starts 2024-09-13T14:44:43.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:42 smithi151 ceph-mon[108966]: 1.18 scrub ok 2024-09-13T14:44:43.173 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:43.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:43.226 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:43.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:43.327 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:43.363 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:43.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:43.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:43.444 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T14:44:43.465 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:43.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:43.527 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:43.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:43.607 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:43.628 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:43.657 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:43.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:43.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:43.726 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 42 16 74488 74472 110.801 81.5625 0.00994805 0.00902014 2024-09-13T14:44:43.759 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:43.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:43.800 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:43.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:43.810 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:43 smithi184 ceph-mon[92527]: 1.10 scrub starts 2024-09-13T14:44:43.810 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:43 smithi184 ceph-mon[92527]: 1.10 scrub ok 2024-09-13T14:44:43.833 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:44:43.901 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:43.930 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:43.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:43.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:44.032 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:44.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:44.061 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:44.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:44.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:43 smithi151 ceph-mon[104450]: 1.10 scrub starts 2024-09-13T14:44:44.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:43 smithi151 ceph-mon[104450]: 1.10 scrub ok 2024-09-13T14:44:44.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:43 smithi151 ceph-mon[108966]: 1.10 scrub starts 2024-09-13T14:44:44.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:43 smithi151 ceph-mon[108966]: 1.10 scrub ok 2024-09-13T14:44:44.161 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:44.199 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:44.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:44.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:44.301 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:44.339 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:44.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:44.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:44.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:44.481 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:44.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:44.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:44.583 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:44.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:44.615 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:44.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:44.716 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:44.726 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 43 16 75687 75671 109.967 74.9375 0.0341456 0.00908818 2024-09-13T14:44:44.745 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:44.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:44.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:44.809 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:44:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:44:44] "GET /metrics HTTP/1.1" 200 36655 "" "Prometheus/2.51.0" 2024-09-13T14:44:44.847 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:44.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:44.885 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:44.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:44.986 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:45.024 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:45.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:45.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:45.085 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:44 smithi151 ceph-mon[104450]: pgmap v1507: 119 pgs: 119 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-13T14:44:45.085 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:44 smithi151 ceph-mon[108966]: pgmap v1507: 119 pgs: 119 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-13T14:44:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:44 smithi184 ceph-mon[92527]: pgmap v1507: 119 pgs: 119 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-13T14:44:45.127 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:45.166 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:45.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:45.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:45.268 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:45.307 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:45.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:45.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:45.408 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:45.443 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:45.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:45.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:45.545 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:45.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:45.584 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:45.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:45.685 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:45.727 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 44 16 76962 76946 109.279 79.6875 0.0153337 0.0091463 2024-09-13T14:44:45.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:45.730 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:45.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:45.822 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-13T14:44:45.831 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:45.833 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:44:45.833 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [ 2024-09-13T14:44:45.833 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:45.833 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.45494.0:77081 7.7 e1142/1136 7:ea00dffd:::benchmark_data_smithi151_190906_object77080:head v 1142'4781, mlcod=1142'4778)", 2024-09-13T14:44:45.833 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:44:45.831169+0000", 2024-09-13T14:44:45.833 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.000336645, 2024-09-13T14:44:45.833 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000360343, 2024-09-13T14:44:45.833 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:44:45.833 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:44:45.833 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:44:45.833 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:45.834 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:44:45.834 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:45.831169+0000", 2024-09-13T14:44:45.834 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:45.834 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:45.834 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:45.834 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:44:45.834 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:45.831169+0000", 2024-09-13T14:44:45.834 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:44:45.834 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:45.834 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:45.834 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:44:45.834 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:45.831168+0000", 2024-09-13T14:44:45.834 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:44:45.834 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:45.834 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:45.835 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:44:45.835 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:45.831199+0000", 2024-09-13T14:44:45.835 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.0834999999999999e-05 2024-09-13T14:44:45.835 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:45.835 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:45.835 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:44:45.835 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:45.831215+0000", 2024-09-13T14:44:45.835 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.5902000000000001e-05 2024-09-13T14:44:45.835 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:45.835 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:45.835 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:44:45.835 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:45.831221+0000", 2024-09-13T14:44:45.835 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3749999999999999e-06 2024-09-13T14:44:45.835 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:45.836 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:45.836 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:44:45.836 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:45.831408+0000", 2024-09-13T14:44:45.836 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000186159 2024-09-13T14:44:45.836 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:44:45.836 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:44:45.836 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:44:45.836 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:45.831457+0000", 2024-09-13T14:44:45.836 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8918000000000002e-05 2024-09-13T14:44:45.836 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:45.836 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:44:45.836 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:45.836 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:44:45.837 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-13T14:44:45.837 INFO:teuthology.orchestra.run.smithi184.stdout: "num_ops": 1 2024-09-13T14:44:45.837 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:44:45.867 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:45.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:45.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:45.968 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:46.001 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:46.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:46.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:46.050 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:45 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3568310657' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:46.104 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:46.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:45 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3568310657' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:46.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:45 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3568310657' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:46.139 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:46.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:46.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:46.241 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:46.281 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:46.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:46.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:46.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:46.417 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:46.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:46.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:46.519 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:46.556 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:46.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:46.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:46.658 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:46.690 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:46.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:46.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:46.727 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 45 16 78275 78259 108.674 82.0625 0.0107521 0.00919745 2024-09-13T14:44:46.790 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T14:44:46.792 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:46.820 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:46.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:46.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:46.922 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:46.955 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:46.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:46.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:47.057 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:47.093 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:47.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:47.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:47.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:46 smithi184 ceph-mon[92527]: pgmap v1508: 119 pgs: 119 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-13T14:44:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:46 smithi184 ceph-mon[92527]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:44:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:46 smithi184 ceph-mon[92527]: Cluster is now healthy 2024-09-13T14:44:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:46 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3568310657' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:44:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:46 smithi184 ceph-mon[92527]: osdmap e1143: 8 total, 8 up, 8 in 2024-09-13T14:44:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:44:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:47.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:46 smithi151 ceph-mon[104450]: pgmap v1508: 119 pgs: 119 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-13T14:44:47.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:46 smithi151 ceph-mon[104450]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:44:47.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:46 smithi151 ceph-mon[104450]: Cluster is now healthy 2024-09-13T14:44:47.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:46 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3568310657' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:44:47.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:46 smithi151 ceph-mon[104450]: osdmap e1143: 8 total, 8 up, 8 in 2024-09-13T14:44:47.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:44:47.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:47.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:46 smithi151 ceph-mon[108966]: pgmap v1508: 119 pgs: 119 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-13T14:44:47.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:46 smithi151 ceph-mon[108966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:44:47.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:46 smithi151 ceph-mon[108966]: Cluster is now healthy 2024-09-13T14:44:47.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:46 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3568310657' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:44:47.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:46 smithi151 ceph-mon[108966]: osdmap e1143: 8 total, 8 up, 8 in 2024-09-13T14:44:47.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:44:47.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:47.196 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:47.197 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:44:47.249 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:47.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:47.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:47.351 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:47.375 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:47.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:47.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:47.477 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:47.504 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:44:47.505 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:44:47.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:47.525 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:47.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:47.539 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:44:47.539 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:44:47.570 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:44:47.570 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:44:47.602 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:44:47.603 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:44:47.626 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:47.632 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:44:47.633 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:44:47.667 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:44:47.667 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:44:47.672 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:47.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:47.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:47.701 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:44:47.701 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:44:47.728 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 46 16 79446 79430 107.901 73.1875 0.0178385 0.00926268 2024-09-13T14:44:47.732 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:44:47.732 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:44:47.774 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:47.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:47.840 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:47.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:47.941 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:47.962 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:44:47.963 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T14:44:47.981 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:47.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:47.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:48.084 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:48.127 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:48.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:48.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:48.210 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:48 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:44:48.211 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:48 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:44:48.211 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:48 smithi184 ceph-mon[92527]: pgmap v1510: 119 pgs: 119 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-13T14:44:48.211 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:48 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3362430262' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:48.211 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:48 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:44:48.211 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:48 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:44:48.211 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:48 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:44:48.228 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:48.260 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:48.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:48.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:48.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:48 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:44:48.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:48 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:44:48.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:48 smithi151 ceph-mon[104450]: pgmap v1510: 119 pgs: 119 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-13T14:44:48.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:48 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3362430262' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:48.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:48 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:44:48.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:48 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:44:48.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:48 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:44:48.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:48 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:44:48.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:48 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:44:48.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:48 smithi151 ceph-mon[108966]: pgmap v1510: 119 pgs: 119 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-13T14:44:48.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:48 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3362430262' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:48.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:48 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:44:48.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:48 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:44:48.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:48 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:44:48.362 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:48.403 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:48.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:48.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:48.506 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:48.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:48.537 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:48.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:48.638 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:48.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:48.672 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:48.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:48.728 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 47 16 80631 80615 107.181 74.0625 0.00916383 0.00932476 2024-09-13T14:44:48.774 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:48.779 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:48.819 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:48.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:48.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:48.826 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:44:48.921 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:48.951 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:48.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:48.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:49.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:49.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:49.101 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:49.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:49.202 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:49.236 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:49.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:49.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:49.272 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:44:49.272 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:44:49.338 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:49.366 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:49.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:49.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:49.468 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:49.508 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:49.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:49.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:49.610 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:49.646 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:49.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:49.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:49.729 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 48 16 81839 81823 106.518 75.5 0.00831841 0.00938323 2024-09-13T14:44:49.748 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:49.789 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:49.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:49.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:49.891 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:49.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:49.931 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:49.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:49.957 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T14:44:50.031 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:50.065 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:50.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:50.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:50.166 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:50.197 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:50.201 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:50.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:50.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:50.303 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:50.325 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:44:50.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:50.343 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:50.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:50.445 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:50.473 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:44:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:44:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:44:50.478 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:50.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:50.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:50.581 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:50.611 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:50.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:50.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:50.616 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:44:50.712 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:50.730 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 49 16 83065 83049 105.906 76.625 0.0149747 0.00943695 2024-09-13T14:44:50.756 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:50.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:50.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:50.774 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:50 smithi184 ceph-mon[92527]: pgmap v1511: 119 pgs: 119 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-13T14:44:50.774 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:50 smithi184 ceph-mon[92527]: 1.17 scrub starts 2024-09-13T14:44:50.774 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:50 smithi184 ceph-mon[92527]: 1.17 scrub ok 2024-09-13T14:44:50.807 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:50 smithi151 ceph-mon[104450]: pgmap v1511: 119 pgs: 119 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-13T14:44:50.808 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:50 smithi151 ceph-mon[104450]: 1.17 scrub starts 2024-09-13T14:44:50.808 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:50 smithi151 ceph-mon[104450]: 1.17 scrub ok 2024-09-13T14:44:50.808 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:50 smithi151 ceph-mon[108966]: pgmap v1511: 119 pgs: 119 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-13T14:44:50.808 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:50 smithi151 ceph-mon[108966]: 1.17 scrub starts 2024-09-13T14:44:50.808 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:50 smithi151 ceph-mon[108966]: 1.17 scrub ok 2024-09-13T14:44:50.858 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:50.885 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:50.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:50.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:50.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:51.034 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:51.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:51.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:51.136 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:51.174 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:51.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:51.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:51.275 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:51.310 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:51.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:51.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:51.412 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:51.456 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:51.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:51.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:51.558 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:51.587 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:51.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:51.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:51.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:51 smithi184 ceph-mon[92527]: 1.12 scrub starts 2024-09-13T14:44:51.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:51 smithi184 ceph-mon[92527]: 1.12 scrub ok 2024-09-13T14:44:51.674 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:51 smithi151 ceph-mon[108966]: 1.12 scrub starts 2024-09-13T14:44:51.674 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:51 smithi151 ceph-mon[108966]: 1.12 scrub ok 2024-09-13T14:44:51.674 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:51 smithi151 ceph-mon[104450]: 1.12 scrub starts 2024-09-13T14:44:51.674 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:51 smithi151 ceph-mon[104450]: 1.12 scrub ok 2024-09-13T14:44:51.689 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:51.724 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:51.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:51.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:51.730 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 50 16 84267 84251 105.29 75.125 0.0099709 0.00949296 2024-09-13T14:44:51.826 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:51.859 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:51.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:51.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:51.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:51.997 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:51.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:51.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:52.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:52.138 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:52.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:52.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:52.240 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:52.292 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:52.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:52.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:52.296 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:44:52.296 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:44:52.394 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:52.427 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:52.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:52.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:52.501 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T14:44:52.528 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:52.570 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:52.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:52.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:52.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:52 smithi184 ceph-mon[92527]: pgmap v1512: 119 pgs: 119 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-13T14:44:52.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:52 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1737417100' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:44:52.672 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:52.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:52.710 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:52.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:52.730 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 51 16 85470 85454 104.7 75.1875 0.0110468 0.00954621 2024-09-13T14:44:52.811 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:52.842 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:52.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:52.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:52.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:52 smithi151 ceph-mon[104450]: pgmap v1512: 119 pgs: 119 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-13T14:44:52.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:52 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1737417100' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:44:52.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:52 smithi151 ceph-mon[108966]: pgmap v1512: 119 pgs: 119 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-13T14:44:52.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:52 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1737417100' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:44:52.944 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:52.973 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:52.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:52.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:53.075 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:53.102 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:53.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:53.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:53.150 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T14:44:53.204 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:53.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:53.242 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:53.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:53.342 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:53.379 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:53.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:53.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:53.403 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T14:44:53.481 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:53.518 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:53.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:53.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:53.544 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:44:53.621 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:53.625 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:53 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1737417100' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:44:53.625 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:53 smithi151 ceph-mon[104450]: osdmap e1144: 8 total, 8 up, 8 in 2024-09-13T14:44:53.625 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:53 smithi151 ceph-mon[104450]: 1.1b scrub starts 2024-09-13T14:44:53.625 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:53 smithi151 ceph-mon[104450]: 1.1b scrub ok 2024-09-13T14:44:53.625 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:53 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1737417100' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:44:53.625 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:53 smithi151 ceph-mon[108966]: osdmap e1144: 8 total, 8 up, 8 in 2024-09-13T14:44:53.625 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:53 smithi151 ceph-mon[108966]: 1.1b scrub starts 2024-09-13T14:44:53.625 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:53 smithi151 ceph-mon[108966]: 1.1b scrub ok 2024-09-13T14:44:53.648 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:53.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:53.677 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:53.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:53.731 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 52 16 86644 86628 104.095 73.375 0.0162674 0.00960095 2024-09-13T14:44:53.778 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:53.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:53.829 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:53.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:53.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:53 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1737417100' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:44:53.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:53 smithi184 ceph-mon[92527]: osdmap e1144: 8 total, 8 up, 8 in 2024-09-13T14:44:53.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:53 smithi184 ceph-mon[92527]: 1.1b scrub starts 2024-09-13T14:44:53.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:53 smithi184 ceph-mon[92527]: 1.1b scrub ok 2024-09-13T14:44:53.930 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:53.960 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:53.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:53.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:54.061 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:54.086 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:54.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:54.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:54.188 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:54.217 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:54.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:54.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:54.319 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:54.355 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:54.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:54.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:54.457 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:54.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:54.502 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:54.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:54.533 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-13T14:44:54.603 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:54.634 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:54.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:54.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:54.732 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 53 16 87689 87673 103.363 65.3125 0.0145514 0.00966996 2024-09-13T14:44:54.736 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:54.767 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:54.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:54.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:54.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:54 smithi184 ceph-mon[92527]: pgmap v1514: 119 pgs: 119 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-13T14:44:54.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:54 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3373524706' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:44:54.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:54 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:44:54.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:54 smithi151 ceph-mon[104450]: pgmap v1514: 119 pgs: 119 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-13T14:44:54.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:54 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3373524706' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:44:54.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:54 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:44:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:54 smithi151 ceph-mon[108966]: pgmap v1514: 119 pgs: 119 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-13T14:44:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:54 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3373524706' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:44:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:54 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:44:54.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:44:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:44:54] "GET /metrics HTTP/1.1" 200 36659 "" "Prometheus/2.51.0" 2024-09-13T14:44:54.869 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:54.898 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:54.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:54.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:55.000 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:55.034 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:55.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:55.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:55.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:55.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:55.189 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:55.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:55.290 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:55.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:55.333 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:55.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:55.434 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:55.465 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:55.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:55.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:55.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:55.604 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:55.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:55.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:55.706 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:55.728 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:55.732 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 54 16 88925 88909 102.879 77.25 0.0130077 0.0097142 2024-09-13T14:44:55.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:55.772 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:55.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:55.797 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:55 smithi184 ceph-mon[92527]: 1.1c deep-scrub starts 2024-09-13T14:44:55.797 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:55 smithi184 ceph-mon[92527]: 1.1c deep-scrub ok 2024-09-13T14:44:55.797 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:55 smithi184 ceph-mon[92527]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:55.798 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:55 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:44:55.798 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:55 smithi184 ceph-mon[92527]: osdmap e1145: 8 total, 8 up, 8 in 2024-09-13T14:44:55.844 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:55 smithi151 ceph-mon[104450]: 1.1c deep-scrub starts 2024-09-13T14:44:55.844 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:55 smithi151 ceph-mon[104450]: 1.1c deep-scrub ok 2024-09-13T14:44:55.844 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:55 smithi151 ceph-mon[104450]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:55.845 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:55 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:44:55.845 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:55 smithi151 ceph-mon[104450]: osdmap e1145: 8 total, 8 up, 8 in 2024-09-13T14:44:55.845 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:55 smithi151 ceph-mon[108966]: 1.1c deep-scrub starts 2024-09-13T14:44:55.846 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:55 smithi151 ceph-mon[108966]: 1.1c deep-scrub ok 2024-09-13T14:44:55.846 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:55 smithi151 ceph-mon[108966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:55.846 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:55 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:44:55.846 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:55 smithi151 ceph-mon[108966]: osdmap e1145: 8 total, 8 up, 8 in 2024-09-13T14:44:55.874 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:55.911 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:55.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:55.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:55.947 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:44:55.947 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:44:56.013 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:56.049 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:56.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:56.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:56.151 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:56.186 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:56.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:56.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:56.287 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:56.316 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:56.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:56.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:56.418 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:56.446 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:56.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:56.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:56.548 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:56.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:56.577 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:56.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:56.678 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:56.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:56.711 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:56.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:56.725 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T14:44:56.732 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 55 16 90144 90128 102.394 76.1875 0.0219602 0.00976156 2024-09-13T14:44:56.799 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:56 smithi184 ceph-mon[92527]: pgmap v1516: 119 pgs: 119 active+clean; 5.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-13T14:44:56.812 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:56.833 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:56 smithi151 ceph-mon[104450]: pgmap v1516: 119 pgs: 119 active+clean; 5.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-13T14:44:56.834 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:56 smithi151 ceph-mon[108966]: pgmap v1516: 119 pgs: 119 active+clean; 5.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-13T14:44:56.855 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:56.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:56.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:56.957 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:56.999 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:57.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:57.101 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:57.147 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:57.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:57.169 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:44:57.249 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:57.279 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:57.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:57.361 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:44:57.361 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:44:57.381 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:57.395 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:44:57.395 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:44:57.410 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:57.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:57.440 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:44:57.440 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:44:57.472 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:44:57.472 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:44:57.505 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:44:57.506 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:44:57.512 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:57.537 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:44:57.538 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:44:57.550 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:57.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:57.568 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:44:57.569 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:44:57.596 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:44:57.596 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:44:57.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:57 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2757319711' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:57.652 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:57.683 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:57.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:57.732 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 56 16 91299 91283 101.855 72.1875 0.00649319 0.00981303 2024-09-13T14:44:57.784 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:57.821 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:57.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:57.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:57 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2757319711' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:57.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:57 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2757319711' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:57.923 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:57.960 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:57.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:58.062 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:58.093 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:58.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:58.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:58.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:58.232 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:44:58.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:58.234 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:58.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:58.335 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:58.366 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:58.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:58.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:58.469 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:58.498 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:58.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:58.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:58.600 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:58.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:58.636 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:58.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:58.732 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 57 16 92456 92440 101.336 72.3125 0.01108 0.0098632 2024-09-13T14:44:58.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:58.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:58.793 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:58.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:58.822 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:44:58 smithi184 ceph-mon[92527]: pgmap v1517: 119 pgs: 119 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-13T14:44:58.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:44:58 smithi151 ceph-mon[104450]: pgmap v1517: 119 pgs: 119 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-13T14:44:58.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:44:58 smithi151 ceph-mon[108966]: pgmap v1517: 119 pgs: 119 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-13T14:44:58.895 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:58.928 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:58.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:58.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:59.029 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:59.060 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:59.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:59.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:59.162 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:59.208 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:59.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:59.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:59.254 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:44:59.254 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [], 2024-09-13T14:44:59.254 INFO:teuthology.orchestra.run.smithi184.stdout: "complaint_time": 30, 2024-09-13T14:44:59.254 INFO:teuthology.orchestra.run.smithi184.stdout: "num_blocked_ops": 0 2024-09-13T14:44:59.254 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:44:59.310 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:59.346 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:59.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:59.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:59.448 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:59.485 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:59.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:59.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:59.586 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:59.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:59.623 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:59.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:59.724 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:59.733 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 58 16 93644 93628 100.867 74.25 0.00764555 0.00990825 2024-09-13T14:44:59.762 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:44:59.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:59.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:59.864 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:59.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:59.897 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:44:59.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:59.979 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T14:44:59.999 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:00.044 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:00.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:00.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:00.146 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:00.180 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:00.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:00.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:00.282 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:00.313 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:00.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:00.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:00.330 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:45:00.415 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:00.451 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:00.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:00.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:00.490 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:45:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:45:00.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:45:00.553 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:00.589 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:00.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:00.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:00.691 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:00.726 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:00.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:00.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:00.733 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 59 16 94583 94567 100.152 58.6875 0.0118073 0.00998031 2024-09-13T14:45:00.815 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:00 smithi151 ceph-mon[104450]: pgmap v1518: 119 pgs: 119 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 82 MiB/s wr, 1.30k op/s 2024-09-13T14:45:00.815 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:00 smithi151 ceph-mon[108966]: pgmap v1518: 119 pgs: 119 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 82 MiB/s wr, 1.30k op/s 2024-09-13T14:45:00.827 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:00.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:00 smithi184 ceph-mon[92527]: pgmap v1518: 119 pgs: 119 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 82 MiB/s wr, 1.30k op/s 2024-09-13T14:45:00.856 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:00.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:00.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:00.958 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:00.979 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:00.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:00.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:01.081 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:01.113 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:01.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:01.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:01.215 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:01.244 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:01.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:01.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:01.346 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:01.374 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:01.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:01.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:01.476 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:01.498 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:01.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:01.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:01.520 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:45:01.600 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:01.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:01.652 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:01.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:01.734 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:2024-09-13T14:45:01.733825+0000 min lat: 0.00109766 max lat: 0.257165 avg lat: 0.0100536 2024-09-13T14:45:01.734 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:45:01.734 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 60 16 95481 95465 99.4184 56.125 0.0187568 0.0100536 2024-09-13T14:45:01.753 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:01.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:01.776 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:01.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:01.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:01 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3242461851' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:45:01.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:01 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:45:01.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:01 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:45:01.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:01 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:01.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:01 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3242461851' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:45:01.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:01 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:45:01.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:45:01.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:01.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:01 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3242461851' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:45:01.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:01 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:45:01.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:01 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:45:01.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:01 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:01.877 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:01.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:01.928 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:01.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:01.997 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T14:45:02.029 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:02.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:02.064 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:02.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:02.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:02.202 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:02.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:02.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:02.304 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:02.334 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:02.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:02.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:02.436 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:02.478 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:02.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:02.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:02.580 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:02.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:02 smithi184 ceph-mon[92527]: pgmap v1519: 119 pgs: 119 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-13T14:45:02.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:02 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T14:45:02.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:02 smithi184 ceph-mon[92527]: osdmap e1146: 8 total, 8 up, 8 in 2024-09-13T14:45:02.610 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:02.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:02.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:02.712 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:02.734 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 61 16 96436 96420 98.7671 59.6875 0.0142043 0.0101194 2024-09-13T14:45:02.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:02.758 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:02.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:02.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:02 smithi151 ceph-mon[108966]: pgmap v1519: 119 pgs: 119 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-13T14:45:02.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:02 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T14:45:02.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:02 smithi151 ceph-mon[108966]: osdmap e1146: 8 total, 8 up, 8 in 2024-09-13T14:45:02.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:02 smithi151 ceph-mon[104450]: pgmap v1519: 119 pgs: 119 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-13T14:45:02.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:02 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T14:45:02.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:02 smithi151 ceph-mon[104450]: osdmap e1146: 8 total, 8 up, 8 in 2024-09-13T14:45:02.859 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:02.880 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:02.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:02.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:02.982 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:03.022 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:03.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:03.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:03.124 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:03.158 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:03.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:03.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:03.260 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:03.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:03.302 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:03.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:03.403 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:03.439 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:03.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:03.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:03.541 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:03.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:03.566 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:03.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:03.652 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:45:03.652 INFO:teuthology.orchestra.run.smithi184.stdout: "size": 20, 2024-09-13T14:45:03.652 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 600, 2024-09-13T14:45:03.653 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [ 2024-09-13T14:45:03.653 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.653 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:165062 6.3 6:ce8e0480:::benchmark_data_smithi151_155836_object32604:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1105)", 2024-09-13T14:45:03.653 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:36.598745+0000", 2024-09-13T14:45:03.653 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 207.050210959, 2024-09-13T14:45:03.653 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.117145496, 2024-09-13T14:45:03.653 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:45:03.653 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:03.653 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:45:03.653 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:45:03.653 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:45:03.653 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 165062 2024-09-13T14:45:03.654 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.654 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:45:03.654 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.654 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:45:03.654 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.598745+0000", 2024-09-13T14:45:03.654 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.654 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.654 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.654 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:45:03.654 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.598745+0000", 2024-09-13T14:45:03.654 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.654 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.654 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.654 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:45:03.655 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.598744+0000", 2024-09-13T14:45:03.655 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:45:03.655 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.655 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.655 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:45:03.655 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.598748+0000", 2024-09-13T14:45:03.655 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9689999999999996e-06 2024-09-13T14:45:03.655 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.655 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.655 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:45:03.655 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.598756+0000", 2024-09-13T14:45:03.655 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.7160000000000003e-06 2024-09-13T14:45:03.655 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.656 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.656 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:45:03.656 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.598760+0000", 2024-09-13T14:45:03.656 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8449999999999996e-06 2024-09-13T14:45:03.656 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.656 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.656 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:45:03.656 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.600128+0000", 2024-09-13T14:45:03.656 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001368317 2024-09-13T14:45:03.656 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.656 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.656 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:45:03.656 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.600237+0000", 2024-09-13T14:45:03.657 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000108993 2024-09-13T14:45:03.657 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.657 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.657 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:45:03.657 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.600339+0000", 2024-09-13T14:45:03.657 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000102292 2024-09-13T14:45:03.657 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.657 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.657 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:45:03.657 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.604286+0000", 2024-09-13T14:45:03.657 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0039471799999999998 2024-09-13T14:45:03.657 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.657 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.658 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.658 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.605760+0000", 2024-09-13T14:45:03.658 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0014739239999999999 2024-09-13T14:45:03.658 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.658 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.658 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.658 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.715758+0000", 2024-09-13T14:45:03.658 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.109997735 2024-09-13T14:45:03.658 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.658 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.658 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:45:03.658 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.715845+0000", 2024-09-13T14:45:03.658 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.6551999999999994e-05 2024-09-13T14:45:03.658 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.659 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.659 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:45:03.659 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:36.715890+0000", 2024-09-13T14:45:03.659 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5841999999999999e-05 2024-09-13T14:45:03.659 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.659 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:45:03.659 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.659 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.659 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.659 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:189886 6.13 6:cd49390e:::benchmark_data_smithi151_155836_object57428:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1114)", 2024-09-13T14:45:03.659 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:42:05.424200+0000", 2024-09-13T14:45:03.659 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 178.22475568999999, 2024-09-13T14:45:03.659 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.126102626, 2024-09-13T14:45:03.659 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:45:03.660 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:03.660 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:45:03.660 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:45:03.660 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:45:03.660 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 189886 2024-09-13T14:45:03.660 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.660 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:45:03.660 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.660 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:45:03.660 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.424200+0000", 2024-09-13T14:45:03.660 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.660 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.660 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.661 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:45:03.661 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.424200+0000", 2024-09-13T14:45:03.661 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.661 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.661 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.661 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:45:03.661 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.424199+0000", 2024-09-13T14:45:03.661 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:03.661 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.661 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.661 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:45:03.661 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.424205+0000", 2024-09-13T14:45:03.661 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1659999999999999e-06 2024-09-13T14:45:03.662 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.662 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.662 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:45:03.662 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.424218+0000", 2024-09-13T14:45:03.662 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3304e-05 2024-09-13T14:45:03.662 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.662 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.662 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:45:03.662 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.424234+0000", 2024-09-13T14:45:03.662 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.5987000000000001e-05 2024-09-13T14:45:03.662 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.662 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.662 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:45:03.663 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.424286+0000", 2024-09-13T14:45:03.663 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.2182000000000003e-05 2024-09-13T14:45:03.663 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.663 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.663 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:45:03.663 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.424396+0000", 2024-09-13T14:45:03.663 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00010926599999999999 2024-09-13T14:45:03.663 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.663 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.663 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:45:03.663 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.424677+0000", 2024-09-13T14:45:03.663 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00028157399999999998 2024-09-13T14:45:03.663 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.664 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.664 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:45:03.664 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.426533+0000", 2024-09-13T14:45:03.664 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00185528 2024-09-13T14:45:03.664 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.664 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.664 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.664 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.427531+0000", 2024-09-13T14:45:03.664 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0009981320000000001 2024-09-13T14:45:03.664 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.664 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.664 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.664 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.550174+0000", 2024-09-13T14:45:03.665 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.122643101 2024-09-13T14:45:03.665 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.665 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.665 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:45:03.665 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.550261+0000", 2024-09-13T14:45:03.665 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.6792999999999994e-05 2024-09-13T14:45:03.665 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.665 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.665 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:45:03.665 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.550303+0000", 2024-09-13T14:45:03.665 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.2230000000000001e-05 2024-09-13T14:45:03.665 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.665 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:45:03.665 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.666 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.666 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.666 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:189918 6.3 6:c2ecd01e:::benchmark_data_smithi151_155836_object57460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1114)", 2024-09-13T14:45:03.666 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:42:05.455481+0000", 2024-09-13T14:45:03.666 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 178.193474638, 2024-09-13T14:45:03.666 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.11657774799999999, 2024-09-13T14:45:03.666 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:45:03.666 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:03.666 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:45:03.666 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:45:03.666 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:45:03.666 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 189918 2024-09-13T14:45:03.666 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.666 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:45:03.666 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.667 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:45:03.667 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:03.668 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.455481+0000", 2024-09-13T14:45:03.668 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.668 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.668 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.668 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:45:03.668 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.455481+0000", 2024-09-13T14:45:03.668 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.668 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.669 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.669 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:45:03.669 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.455480+0000", 2024-09-13T14:45:03.669 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:45:03.669 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.669 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.669 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:45:03.669 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.455489+0000", 2024-09-13T14:45:03.669 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.6850000000000007e-06 2024-09-13T14:45:03.669 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.669 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.669 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:45:03.669 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.455501+0000", 2024-09-13T14:45:03.669 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2386e-05 2024-09-13T14:45:03.670 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.670 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.670 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:45:03.670 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.455507+0000", 2024-09-13T14:45:03.670 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8409999999999996e-06 2024-09-13T14:45:03.670 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.670 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.670 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:45:03.670 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.455548+0000", 2024-09-13T14:45:03.670 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0354e-05 2024-09-13T14:45:03.670 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.670 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.670 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:45:03.670 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.455655+0000", 2024-09-13T14:45:03.671 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000107867 2024-09-13T14:45:03.671 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.671 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.671 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:45:03.671 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.455995+0000", 2024-09-13T14:45:03.671 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00033996999999999999 2024-09-13T14:45:03.671 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.671 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.671 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:45:03.671 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.457808+0000", 2024-09-13T14:45:03.671 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0018126780000000001 2024-09-13T14:45:03.671 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.671 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.671 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.672 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.458927+0000", 2024-09-13T14:45:03.672 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001118602 2024-09-13T14:45:03.672 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.672 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.672 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.672 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.571936+0000", 2024-09-13T14:45:03.672 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.11300884999999999 2024-09-13T14:45:03.672 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.672 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.672 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:45:03.672 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.572019+0000", 2024-09-13T14:45:03.672 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.3447000000000003e-05 2024-09-13T14:45:03.672 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.672 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.673 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:45:03.673 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.572059+0000", 2024-09-13T14:45:03.673 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0005000000000001e-05 2024-09-13T14:45:03.673 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.673 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:45:03.673 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.673 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.673 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.673 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:208102 6.3 6:c7f972fa:::benchmark_data_smithi151_155836_object75644:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1118)", 2024-09-13T14:45:03.673 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:42:22.751530+0000", 2024-09-13T14:45:03.673 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 160.89742569000001, 2024-09-13T14:45:03.673 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.110000746, 2024-09-13T14:45:03.673 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:45:03.673 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:03.674 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:45:03.674 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:45:03.674 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:45:03.674 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 208102 2024-09-13T14:45:03.674 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.674 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:45:03.674 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.674 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:45:03.674 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:22.751530+0000", 2024-09-13T14:45:03.674 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.674 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.674 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.674 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:45:03.675 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:22.751530+0000", 2024-09-13T14:45:03.675 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.675 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.675 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.675 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:45:03.675 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:22.751529+0000", 2024-09-13T14:45:03.675 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:45:03.675 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.675 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.675 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:45:03.675 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:22.751535+0000", 2024-09-13T14:45:03.675 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6949999999999997e-06 2024-09-13T14:45:03.675 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.676 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.676 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:45:03.676 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:22.751546+0000", 2024-09-13T14:45:03.676 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1365e-05 2024-09-13T14:45:03.676 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.676 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.676 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:45:03.676 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:22.751551+0000", 2024-09-13T14:45:03.676 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9989999999999999e-06 2024-09-13T14:45:03.676 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.676 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.676 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:45:03.676 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:22.751589+0000", 2024-09-13T14:45:03.677 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8108999999999997e-05 2024-09-13T14:45:03.677 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.677 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.677 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:45:03.677 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:22.751695+0000", 2024-09-13T14:45:03.677 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00010569300000000001 2024-09-13T14:45:03.677 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.677 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.677 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:45:03.677 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:22.752125+0000", 2024-09-13T14:45:03.677 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00042977800000000002 2024-09-13T14:45:03.677 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.677 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.677 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:45:03.678 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:22.753852+0000", 2024-09-13T14:45:03.678 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0017275929999999999 2024-09-13T14:45:03.678 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.678 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.678 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.678 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:22.754663+0000", 2024-09-13T14:45:03.678 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000810712 2024-09-13T14:45:03.678 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.678 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.678 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.678 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:22.861454+0000", 2024-09-13T14:45:03.678 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.10679121799999999 2024-09-13T14:45:03.678 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.679 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.679 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:45:03.679 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:22.861497+0000", 2024-09-13T14:45:03.679 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3007999999999999e-05 2024-09-13T14:45:03.679 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.679 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.679 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:45:03.679 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:22.861531+0000", 2024-09-13T14:45:03.679 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3717000000000003e-05 2024-09-13T14:45:03.679 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.679 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:45:03.679 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.679 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.680 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.680 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:213987 6.13 6:ca37c951:::benchmark_data_smithi151_155836_object81529:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1118)", 2024-09-13T14:45:03.680 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:42:28.478062+0000", 2024-09-13T14:45:03.680 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 155.170893783, 2024-09-13T14:45:03.680 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.126784759, 2024-09-13T14:45:03.680 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:45:03.680 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:03.680 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:45:03.680 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:45:03.680 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:45:03.680 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 213987 2024-09-13T14:45:03.680 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.680 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:45:03.680 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.681 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:45:03.681 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.478062+0000", 2024-09-13T14:45:03.681 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.681 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.681 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.681 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:45:03.681 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.478062+0000", 2024-09-13T14:45:03.681 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.681 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.681 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.681 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:45:03.681 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.478061+0000", 2024-09-13T14:45:03.681 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:03.681 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.682 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.682 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:45:03.682 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.478067+0000", 2024-09-13T14:45:03.682 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.0419999999999999e-06 2024-09-13T14:45:03.682 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.682 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.682 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:45:03.682 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.478078+0000", 2024-09-13T14:45:03.682 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1608e-05 2024-09-13T14:45:03.682 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.682 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.682 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:45:03.682 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.478084+0000", 2024-09-13T14:45:03.682 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7010000000000001e-06 2024-09-13T14:45:03.683 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.683 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.683 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:45:03.683 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.486802+0000", 2024-09-13T14:45:03.683 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0087175770000000007 2024-09-13T14:45:03.683 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.683 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.683 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:45:03.683 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.486958+0000", 2024-09-13T14:45:03.683 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00015602699999999999 2024-09-13T14:45:03.683 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.683 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.683 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:45:03.684 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.487284+0000", 2024-09-13T14:45:03.684 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00032676800000000001 2024-09-13T14:45:03.684 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.684 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.684 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:45:03.684 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.489316+0000", 2024-09-13T14:45:03.684 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0020320780000000001 2024-09-13T14:45:03.684 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.684 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.684 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.684 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.489720+0000", 2024-09-13T14:45:03.684 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00040378099999999998 2024-09-13T14:45:03.684 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.684 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.685 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.685 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.604738+0000", 2024-09-13T14:45:03.685 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.115018 2024-09-13T14:45:03.685 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.685 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.685 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:45:03.685 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.604810+0000", 2024-09-13T14:45:03.685 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.2101999999999995e-05 2024-09-13T14:45:03.685 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.685 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.685 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:45:03.685 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.604847+0000", 2024-09-13T14:45:03.685 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.6489999999999998e-05 2024-09-13T14:45:03.686 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.686 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:45:03.686 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.686 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.686 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.686 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:214043 6.13 6:c9f434f8:::benchmark_data_smithi151_155836_object81585:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1118)", 2024-09-13T14:45:03.686 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:42:28.538332+0000", 2024-09-13T14:45:03.686 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 155.110623498, 2024-09-13T14:45:03.686 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.10999534900000001, 2024-09-13T14:45:03.686 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:45:03.687 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:03.687 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:45:03.687 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:45:03.687 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:45:03.687 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 214043 2024-09-13T14:45:03.687 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.687 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:45:03.687 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.687 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:45:03.687 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.538332+0000", 2024-09-13T14:45:03.687 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.687 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.687 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.688 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:45:03.688 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.538332+0000", 2024-09-13T14:45:03.688 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.688 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.688 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.688 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:45:03.688 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.538331+0000", 2024-09-13T14:45:03.688 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:03.688 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.688 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.688 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:45:03.688 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.538341+0000", 2024-09-13T14:45:03.689 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0541e-05 2024-09-13T14:45:03.689 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.689 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.689 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:45:03.689 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.538357+0000", 2024-09-13T14:45:03.689 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.5934e-05 2024-09-13T14:45:03.689 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.689 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.689 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:45:03.689 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.538364+0000", 2024-09-13T14:45:03.689 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.4969999999999998e-06 2024-09-13T14:45:03.689 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.689 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.690 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:45:03.690 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.538444+0000", 2024-09-13T14:45:03.690 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0432000000000005e-05 2024-09-13T14:45:03.690 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.690 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.690 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:45:03.690 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.538586+0000", 2024-09-13T14:45:03.690 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000141524 2024-09-13T14:45:03.690 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.690 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.690 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:45:03.690 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.538993+0000", 2024-09-13T14:45:03.690 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00040718200000000002 2024-09-13T14:45:03.690 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.691 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.691 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.691 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.540996+0000", 2024-09-13T14:45:03.691 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0020025590000000001 2024-09-13T14:45:03.691 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.691 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.691 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:45:03.691 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.541011+0000", 2024-09-13T14:45:03.691 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.5529e-05 2024-09-13T14:45:03.691 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.691 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.691 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.691 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.648215+0000", 2024-09-13T14:45:03.691 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.107203852 2024-09-13T14:45:03.692 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.692 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.692 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:45:03.692 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.648290+0000", 2024-09-13T14:45:03.692 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.5258000000000003e-05 2024-09-13T14:45:03.692 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.692 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.692 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:45:03.692 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.648328+0000", 2024-09-13T14:45:03.692 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.7574000000000003e-05 2024-09-13T14:45:03.692 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.692 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:45:03.692 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.693 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.693 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.693 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:242879 6.3 6:c772ce9b:::benchmark_data_smithi151_155836_object110421:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1123)", 2024-09-13T14:45:03.693 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:42:57.618644+0000", 2024-09-13T14:45:03.693 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 126.03031233999999, 2024-09-13T14:45:03.693 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.11800498800000001, 2024-09-13T14:45:03.693 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:45:03.693 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:03.693 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:45:03.693 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:45:03.693 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:45:03.693 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 242879 2024-09-13T14:45:03.693 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.694 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:45:03.694 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.694 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:45:03.694 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:57.618644+0000", 2024-09-13T14:45:03.694 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.694 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.694 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.694 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:45:03.694 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:57.618644+0000", 2024-09-13T14:45:03.694 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.694 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.694 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.694 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:45:03.694 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:57.618642+0000", 2024-09-13T14:45:03.695 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:03.695 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.695 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.695 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:45:03.695 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:57.618650+0000", 2024-09-13T14:45:03.695 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.5730000000000008e-06 2024-09-13T14:45:03.695 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.695 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.695 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:45:03.695 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:57.618666+0000", 2024-09-13T14:45:03.695 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.5965e-05 2024-09-13T14:45:03.695 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.695 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.695 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:45:03.695 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:57.618672+0000", 2024-09-13T14:45:03.696 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8259999999999998e-06 2024-09-13T14:45:03.696 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.696 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.696 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:45:03.696 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:57.634524+0000", 2024-09-13T14:45:03.696 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.015852158000000002 2024-09-13T14:45:03.696 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.696 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.696 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:45:03.696 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:57.634644+0000", 2024-09-13T14:45:03.696 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000119972 2024-09-13T14:45:03.696 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.696 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.696 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:45:03.697 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:57.635072+0000", 2024-09-13T14:45:03.697 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00042752000000000003 2024-09-13T14:45:03.697 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.697 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.697 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.697 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:57.637538+0000", 2024-09-13T14:45:03.697 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0024664600000000002 2024-09-13T14:45:03.697 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.697 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.697 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:45:03.697 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:57.637681+0000", 2024-09-13T14:45:03.697 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014278899999999999 2024-09-13T14:45:03.697 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.698 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.698 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.698 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:57.736524+0000", 2024-09-13T14:45:03.698 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.098842907999999993 2024-09-13T14:45:03.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.698 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.698 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:45:03.698 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:57.736609+0000", 2024-09-13T14:45:03.698 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.4599999999999996e-05 2024-09-13T14:45:03.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.698 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.698 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:45:03.698 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:57.736649+0000", 2024-09-13T14:45:03.699 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9867000000000003e-05 2024-09-13T14:45:03.699 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.699 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:45:03.699 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.699 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.699 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.699 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35289.0:244969 6.3 6:c1e60316:::benchmark_data_smithi151_155836_object112511:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1123)", 2024-09-13T14:45:03.699 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:42:59.654201+0000", 2024-09-13T14:45:03.699 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 123.994755041, 2024-09-13T14:45:03.699 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.108848796, 2024-09-13T14:45:03.699 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:45:03.699 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:03.699 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:45:03.700 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35289", 2024-09-13T14:45:03.700 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:45:03.700 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 244969 2024-09-13T14:45:03.700 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.700 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:45:03.700 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.700 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:45:03.700 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:59.654201+0000", 2024-09-13T14:45:03.700 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.700 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.700 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.700 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:45:03.700 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:59.654201+0000", 2024-09-13T14:45:03.701 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.701 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.701 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.701 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:45:03.701 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:59.654200+0000", 2024-09-13T14:45:03.701 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:45:03.701 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.701 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.701 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:45:03.701 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:59.654205+0000", 2024-09-13T14:45:03.701 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6760000000000004e-06 2024-09-13T14:45:03.701 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.701 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.701 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:45:03.701 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:59.654216+0000", 2024-09-13T14:45:03.702 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0941000000000001e-05 2024-09-13T14:45:03.702 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.702 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.702 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:45:03.702 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:59.654222+0000", 2024-09-13T14:45:03.702 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3480000000000003e-06 2024-09-13T14:45:03.702 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.702 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.702 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:45:03.702 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:59.665064+0000", 2024-09-13T14:45:03.702 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.010842770999999999 2024-09-13T14:45:03.702 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.702 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.702 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:45:03.703 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:59.665284+0000", 2024-09-13T14:45:03.703 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00021976400000000001 2024-09-13T14:45:03.703 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.703 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.703 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:45:03.703 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:59.665721+0000", 2024-09-13T14:45:03.703 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00043711299999999999 2024-09-13T14:45:03.703 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.703 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.703 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:45:03.703 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:59.669774+0000", 2024-09-13T14:45:03.703 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.004053053 2024-09-13T14:45:03.703 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.703 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.704 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.704 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:59.669856+0000", 2024-09-13T14:45:03.704 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.1662000000000005e-05 2024-09-13T14:45:03.704 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.704 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.704 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.704 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:59.762935+0000", 2024-09-13T14:45:03.704 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.093078737999999994 2024-09-13T14:45:03.704 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.704 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.704 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:45:03.704 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:59.763008+0000", 2024-09-13T14:45:03.704 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.3619000000000001e-05 2024-09-13T14:45:03.705 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.705 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.705 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:45:03.705 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:59.763050+0000", 2024-09-13T14:45:03.705 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1276999999999998e-05 2024-09-13T14:45:03.705 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.705 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:45:03.705 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.705 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.705 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.705 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:60535 7.5 7:ab646982:::benchmark_data_smithi151_190906_object60534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1142)", 2024-09-13T14:45:03.705 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:44:32.321485+0000", 2024-09-13T14:45:03.705 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 31.327470460000001, 2024-09-13T14:45:03.706 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.16541595000000001, 2024-09-13T14:45:03.706 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:45:03.706 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:03.706 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:45:03.706 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:45:03.706 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:45:03.706 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 60535 2024-09-13T14:45:03.706 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.706 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:45:03.706 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.706 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:45:03.706 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.321485+0000", 2024-09-13T14:45:03.706 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.706 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.706 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.707 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:45:03.707 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.321485+0000", 2024-09-13T14:45:03.707 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.707 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.707 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.707 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:45:03.707 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.321484+0000", 2024-09-13T14:45:03.707 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:45:03.707 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.707 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.707 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:45:03.707 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.322401+0000", 2024-09-13T14:45:03.707 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00091768700000000004 2024-09-13T14:45:03.707 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.708 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.708 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:45:03.708 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.322422+0000", 2024-09-13T14:45:03.708 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.1018000000000001e-05 2024-09-13T14:45:03.708 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.708 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.708 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:45:03.708 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.322429+0000", 2024-09-13T14:45:03.708 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3169999999999998e-06 2024-09-13T14:45:03.708 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.708 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.708 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:45:03.708 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.322920+0000", 2024-09-13T14:45:03.708 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00049131800000000003 2024-09-13T14:45:03.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.709 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.709 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:45:03.709 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.323083+0000", 2024-09-13T14:45:03.709 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00016288699999999999 2024-09-13T14:45:03.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.709 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.709 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:03.709 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.323276+0000", 2024-09-13T14:45:03.709 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00019269999999999999 2024-09-13T14:45:03.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.709 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.709 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:45:03.709 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.325333+0000", 2024-09-13T14:45:03.710 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002057686 2024-09-13T14:45:03.710 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.710 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.710 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.710 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.483240+0000", 2024-09-13T14:45:03.710 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.15790709 2024-09-13T14:45:03.710 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.710 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.711 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.711 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.486772+0000", 2024-09-13T14:45:03.711 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0035317740000000001 2024-09-13T14:45:03.711 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.711 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.711 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:45:03.711 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.486856+0000", 2024-09-13T14:45:03.712 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.4178000000000006e-05 2024-09-13T14:45:03.712 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.712 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.712 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:45:03.712 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.486901+0000", 2024-09-13T14:45:03.712 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5104e-05 2024-09-13T14:45:03.712 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.712 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:45:03.712 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.712 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.712 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.712 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:60537 7.c 7:3d857777:::benchmark_data_smithi151_190906_object60536:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1142)", 2024-09-13T14:45:03.713 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:44:32.323865+0000", 2024-09-13T14:45:03.713 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 31.325090735, 2024-09-13T14:45:03.713 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.156707083, 2024-09-13T14:45:03.713 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:45:03.713 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:03.713 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:45:03.713 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:45:03.713 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:45:03.713 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 60537 2024-09-13T14:45:03.713 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.713 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:45:03.713 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.713 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:45:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.323865+0000", 2024-09-13T14:45:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:45:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.323865+0000", 2024-09-13T14:45:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:45:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.323859+0000", 2024-09-13T14:45:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999938 2024-09-13T14:45:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.715 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.715 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:45:03.715 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.323914+0000", 2024-09-13T14:45:03.715 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4979000000000003e-05 2024-09-13T14:45:03.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.715 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.715 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:45:03.715 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.323927+0000", 2024-09-13T14:45:03.715 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3139000000000001e-05 2024-09-13T14:45:03.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.715 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.715 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:45:03.715 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.323934+0000", 2024-09-13T14:45:03.716 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7479999999999996e-06 2024-09-13T14:45:03.716 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.716 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.716 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:45:03.716 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.323973+0000", 2024-09-13T14:45:03.716 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9527000000000003e-05 2024-09-13T14:45:03.716 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.716 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.716 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:45:03.716 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.324064+0000", 2024-09-13T14:45:03.716 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.1040999999999998e-05 2024-09-13T14:45:03.716 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.717 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.717 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:45:03.717 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.324166+0000", 2024-09-13T14:45:03.717 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00010198 2024-09-13T14:45:03.717 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.717 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.717 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:45:03.717 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.326541+0000", 2024-09-13T14:45:03.717 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0023743530000000001 2024-09-13T14:45:03.717 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.717 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.717 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.717 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.326736+0000", 2024-09-13T14:45:03.717 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00019539899999999999 2024-09-13T14:45:03.718 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.718 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.718 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.718 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.480442+0000", 2024-09-13T14:45:03.718 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.153705644 2024-09-13T14:45:03.718 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.718 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.718 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:45:03.718 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.480491+0000", 2024-09-13T14:45:03.718 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9375000000000001e-05 2024-09-13T14:45:03.718 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.718 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.718 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:45:03.718 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.480572+0000", 2024-09-13T14:45:03.719 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.1205999999999995e-05 2024-09-13T14:45:03.719 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.719 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:45:03.719 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.719 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.719 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.719 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:61895 7.5 7:a46f2851:::benchmark_data_smithi151_190906_object61894:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1142)", 2024-09-13T14:45:03.719 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:44:33.523769+0000", 2024-09-13T14:45:03.719 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 30.125187176000001, 2024-09-13T14:45:03.719 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.24447361400000001, 2024-09-13T14:45:03.719 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:45:03.719 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:03.719 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:45:03.719 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:45:03.720 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:45:03.720 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 61895 2024-09-13T14:45:03.720 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.720 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:45:03.720 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.720 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:45:03.720 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.523769+0000", 2024-09-13T14:45:03.720 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.720 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.720 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.720 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:45:03.720 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.523769+0000", 2024-09-13T14:45:03.720 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.721 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.721 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:45:03.721 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.523765+0000", 2024-09-13T14:45:03.721 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999962 2024-09-13T14:45:03.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.721 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.721 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:45:03.721 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.523823+0000", 2024-09-13T14:45:03.721 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8334000000000001e-05 2024-09-13T14:45:03.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.721 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.721 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:45:03.721 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.523839+0000", 2024-09-13T14:45:03.722 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.5883999999999999e-05 2024-09-13T14:45:03.722 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.722 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.722 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:45:03.722 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.523845+0000", 2024-09-13T14:45:03.722 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.2890000000000003e-06 2024-09-13T14:45:03.722 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.722 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.722 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:45:03.722 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.523918+0000", 2024-09-13T14:45:03.722 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.3069000000000001e-05 2024-09-13T14:45:03.722 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.722 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.722 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:45:03.723 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.524056+0000", 2024-09-13T14:45:03.723 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000138007 2024-09-13T14:45:03.723 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.723 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.723 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:03.723 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.524210+0000", 2024-09-13T14:45:03.723 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00015325000000000001 2024-09-13T14:45:03.723 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.723 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.723 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:45:03.723 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.526052+0000", 2024-09-13T14:45:03.723 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001842764 2024-09-13T14:45:03.724 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.724 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.724 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.724 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.764788+0000", 2024-09-13T14:45:03.724 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.238735528 2024-09-13T14:45:03.724 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.724 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.724 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.724 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.768139+0000", 2024-09-13T14:45:03.724 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.003350829 2024-09-13T14:45:03.724 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.724 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.724 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:45:03.724 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.768197+0000", 2024-09-13T14:45:03.725 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7899999999999998e-05 2024-09-13T14:45:03.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.725 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.725 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:45:03.725 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.768242+0000", 2024-09-13T14:45:03.725 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5809999999999997e-05 2024-09-13T14:45:03.725 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.725 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:45:03.725 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.725 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.725 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:63024 7.5 7:a28b3da7:::benchmark_data_smithi151_190906_object63023:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1142)", 2024-09-13T14:45:03.725 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:44:34.633955+0000", 2024-09-13T14:45:03.725 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 29.015000511, 2024-09-13T14:45:03.726 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.14815014100000001, 2024-09-13T14:45:03.726 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:45:03.726 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:03.726 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:45:03.726 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:45:03.726 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:45:03.726 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 63024 2024-09-13T14:45:03.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.726 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:45:03.726 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.726 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:45:03.726 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.633955+0000", 2024-09-13T14:45:03.726 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.726 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.727 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:45:03.727 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.633955+0000", 2024-09-13T14:45:03.727 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.727 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.727 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.727 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:45:03.727 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.633954+0000", 2024-09-13T14:45:03.727 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:03.727 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.727 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.727 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:45:03.727 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.634020+0000", 2024-09-13T14:45:03.727 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.6606000000000006e-05 2024-09-13T14:45:03.728 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.728 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.728 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:45:03.728 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.634033+0000", 2024-09-13T14:45:03.728 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2574e-05 2024-09-13T14:45:03.728 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.728 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.728 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:45:03.728 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.634039+0000", 2024-09-13T14:45:03.728 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5219999999999998e-06 2024-09-13T14:45:03.728 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.728 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.728 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:45:03.728 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.634085+0000", 2024-09-13T14:45:03.729 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.6412000000000002e-05 2024-09-13T14:45:03.729 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.729 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.729 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:45:03.729 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.634226+0000", 2024-09-13T14:45:03.729 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014136199999999999 2024-09-13T14:45:03.729 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.729 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.729 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:03.729 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.634370+0000", 2024-09-13T14:45:03.729 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000143607 2024-09-13T14:45:03.729 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.729 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.730 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:45:03.730 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.636483+0000", 2024-09-13T14:45:03.730 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0021130279999999999 2024-09-13T14:45:03.730 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.730 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.730 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.730 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.753310+0000", 2024-09-13T14:45:03.730 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.116827134 2024-09-13T14:45:03.730 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.730 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.730 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.730 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.781847+0000", 2024-09-13T14:45:03.730 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.028536736 2024-09-13T14:45:03.731 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.731 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.731 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:45:03.731 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.782052+0000", 2024-09-13T14:45:03.731 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00020504 2024-09-13T14:45:03.731 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.731 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.731 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:45:03.731 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.782106+0000", 2024-09-13T14:45:03.731 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3667e-05 2024-09-13T14:45:03.731 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.731 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:45:03.731 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.731 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.732 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.732 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:66880 7.5 7:aba8be0f:::benchmark_data_smithi151_190906_object66879:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1142)", 2024-09-13T14:45:03.732 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:44:37.928179+0000", 2024-09-13T14:45:03.732 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 25.720777285, 2024-09-13T14:45:03.732 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.21699406800000001, 2024-09-13T14:45:03.732 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:45:03.732 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:03.732 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:45:03.732 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:45:03.732 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:45:03.732 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 66880 2024-09-13T14:45:03.732 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.732 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:45:03.732 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.733 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:45:03.733 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:37.928179+0000", 2024-09-13T14:45:03.733 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.733 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.733 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.733 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:45:03.733 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:37.928179+0000", 2024-09-13T14:45:03.733 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.733 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.733 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.733 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:45:03.733 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:37.928178+0000", 2024-09-13T14:45:03.733 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:45:03.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.734 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.734 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:45:03.734 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:37.928211+0000", 2024-09-13T14:45:03.734 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.2928000000000001e-05 2024-09-13T14:45:03.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.734 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.734 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:45:03.734 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:37.928221+0000", 2024-09-13T14:45:03.734 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0215e-05 2024-09-13T14:45:03.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.734 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.734 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:45:03.734 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:37.928227+0000", 2024-09-13T14:45:03.734 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.0859999999999998e-06 2024-09-13T14:45:03.735 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.735 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.735 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:45:03.735 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:37.930791+0000", 2024-09-13T14:45:03.735 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002564181 2024-09-13T14:45:03.735 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.735 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.735 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:45:03.735 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:37.930931+0000", 2024-09-13T14:45:03.735 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000140187 2024-09-13T14:45:03.735 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.735 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.735 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:03.735 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:37.931071+0000", 2024-09-13T14:45:03.736 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013932800000000001 2024-09-13T14:45:03.736 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.736 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.736 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:45:03.736 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:37.933828+0000", 2024-09-13T14:45:03.736 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0027570680000000001 2024-09-13T14:45:03.736 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.736 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.736 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.736 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:37.936581+0000", 2024-09-13T14:45:03.736 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002753279 2024-09-13T14:45:03.736 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.736 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.737 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.737 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:38.145102+0000", 2024-09-13T14:45:03.737 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.20852106500000001 2024-09-13T14:45:03.737 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.737 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.737 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:45:03.737 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:38.145145+0000", 2024-09-13T14:45:03.737 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3263999999999998e-05 2024-09-13T14:45:03.737 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.737 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.737 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:45:03.737 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:38.145173+0000", 2024-09-13T14:45:03.737 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.7389000000000002e-05 2024-09-13T14:45:03.737 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.737 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:45:03.738 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.738 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.738 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.738 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:93867 7.c 7:333b2c00:::benchmark_data_smithi151_190906_object93866:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1145)", 2024-09-13T14:45:03.738 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:44:59.912033+0000", 2024-09-13T14:45:03.738 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 3.736923354, 2024-09-13T14:45:03.738 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.16074064299999999, 2024-09-13T14:45:03.738 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:45:03.738 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:03.738 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:45:03.738 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:45:03.738 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:45:03.738 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 93867 2024-09-13T14:45:03.738 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.739 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:45:03.739 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.739 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:45:03.739 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.912033+0000", 2024-09-13T14:45:03.739 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.739 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.739 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.739 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:45:03.739 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.912033+0000", 2024-09-13T14:45:03.739 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.739 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.739 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.740 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:45:03.740 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.912025+0000", 2024-09-13T14:45:03.740 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999919 2024-09-13T14:45:03.740 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.740 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.740 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:45:03.740 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.912073+0000", 2024-09-13T14:45:03.740 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8429e-05 2024-09-13T14:45:03.740 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.740 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.740 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:45:03.740 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.912092+0000", 2024-09-13T14:45:03.740 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.8536999999999999e-05 2024-09-13T14:45:03.741 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.741 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.741 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:45:03.741 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.912098+0000", 2024-09-13T14:45:03.741 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.0190000000000003e-06 2024-09-13T14:45:03.741 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.741 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.741 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:45:03.741 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.912139+0000", 2024-09-13T14:45:03.741 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1306000000000001e-05 2024-09-13T14:45:03.741 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.741 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.741 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:45:03.741 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.912280+0000", 2024-09-13T14:45:03.742 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000140753 2024-09-13T14:45:03.742 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.742 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.742 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:45:03.742 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.912440+0000", 2024-09-13T14:45:03.742 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000160755 2024-09-13T14:45:03.742 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.742 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.742 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:45:03.742 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.919007+0000", 2024-09-13T14:45:03.742 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.006566934 2024-09-13T14:45:03.742 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.742 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.742 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.743 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.919285+0000", 2024-09-13T14:45:03.743 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00027790399999999999 2024-09-13T14:45:03.743 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.743 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.743 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.743 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.072630+0000", 2024-09-13T14:45:03.743 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.15334468400000001 2024-09-13T14:45:03.743 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.743 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.743 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:45:03.743 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.072731+0000", 2024-09-13T14:45:03.743 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00010129200000000001 2024-09-13T14:45:03.743 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.744 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.744 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:45:03.744 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.072773+0000", 2024-09-13T14:45:03.744 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.2006999999999999e-05 2024-09-13T14:45:03.744 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.744 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:45:03.744 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.744 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.744 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.744 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:93871 7.c 7:3f478944:::benchmark_data_smithi151_190906_object93870:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1145)", 2024-09-13T14:45:03.744 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:44:59.914328+0000", 2024-09-13T14:45:03.744 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 3.7346279099999999, 2024-09-13T14:45:03.744 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.15858155500000001, 2024-09-13T14:45:03.744 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:45:03.745 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:03.745 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:45:03.745 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:45:03.745 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:45:03.745 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 93871 2024-09-13T14:45:03.745 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.745 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:45:03.745 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.745 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:45:03.745 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.914328+0000", 2024-09-13T14:45:03.745 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.745 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.745 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.745 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:45:03.746 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.914328+0000", 2024-09-13T14:45:03.746 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.746 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.746 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.746 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:45:03.746 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.914327+0000", 2024-09-13T14:45:03.746 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:45:03.746 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.746 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.746 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:45:03.746 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.914382+0000", 2024-09-13T14:45:03.746 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4993e-05 2024-09-13T14:45:03.746 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.747 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.747 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:45:03.747 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.914397+0000", 2024-09-13T14:45:03.747 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.5413999999999998e-05 2024-09-13T14:45:03.747 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.747 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.747 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:45:03.747 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.914403+0000", 2024-09-13T14:45:03.747 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.0889999999999996e-06 2024-09-13T14:45:03.747 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.747 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.747 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:45:03.747 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.915554+0000", 2024-09-13T14:45:03.747 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0011510260000000001 2024-09-13T14:45:03.748 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.748 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.748 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:45:03.748 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.915687+0000", 2024-09-13T14:45:03.748 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000132879 2024-09-13T14:45:03.748 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.748 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.748 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:45:03.748 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.915834+0000", 2024-09-13T14:45:03.748 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000147248 2024-09-13T14:45:03.748 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.748 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.748 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:45:03.748 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.919317+0000", 2024-09-13T14:45:03.749 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0034822799999999999 2024-09-13T14:45:03.749 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.749 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.749 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.749 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.919609+0000", 2024-09-13T14:45:03.749 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00029270099999999997 2024-09-13T14:45:03.749 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.749 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.749 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.749 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.072807+0000", 2024-09-13T14:45:03.749 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.15319735400000001 2024-09-13T14:45:03.749 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.749 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.749 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:45:03.750 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.072879+0000", 2024-09-13T14:45:03.750 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.1826e-05 2024-09-13T14:45:03.750 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.750 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.750 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:45:03.750 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.072910+0000", 2024-09-13T14:45:03.750 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.0920999999999998e-05 2024-09-13T14:45:03.750 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.750 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:45:03.750 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.750 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.750 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.750 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:93879 7.5 7:a47db476:::benchmark_data_smithi151_190906_object93878:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1145)", 2024-09-13T14:45:03.750 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:44:59.919741+0000", 2024-09-13T14:45:03.751 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 3.7292151929999999, 2024-09-13T14:45:03.751 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.15622032599999999, 2024-09-13T14:45:03.751 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:45:03.751 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:03.751 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:45:03.751 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:45:03.751 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:45:03.751 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 93879 2024-09-13T14:45:03.751 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.751 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:45:03.751 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.751 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:45:03.751 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.919741+0000", 2024-09-13T14:45:03.751 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.752 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.752 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.752 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:45:03.752 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.919741+0000", 2024-09-13T14:45:03.752 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.752 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.752 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.752 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:45:03.752 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.919739+0000", 2024-09-13T14:45:03.752 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:45:03.752 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.752 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.752 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:45:03.753 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 62 16 97308 97292 98.0532 54.5 0.0220021 0.0101931 2024-09-13T14:45:03.754 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.919769+0000", 2024-09-13T14:45:03.754 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.0221000000000001e-05 2024-09-13T14:45:03.754 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.754 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.754 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:45:03.754 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.919784+0000", 2024-09-13T14:45:03.754 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.5308999999999999e-05 2024-09-13T14:45:03.755 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.755 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.755 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:45:03.755 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.919790+0000", 2024-09-13T14:45:03.755 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6899999999999997e-06 2024-09-13T14:45:03.755 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.755 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.755 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:45:03.755 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.919830+0000", 2024-09-13T14:45:03.755 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9993999999999997e-05 2024-09-13T14:45:03.755 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.755 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.755 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:45:03.756 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.919964+0000", 2024-09-13T14:45:03.756 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013362700000000001 2024-09-13T14:45:03.756 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.756 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.756 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:03.756 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.920098+0000", 2024-09-13T14:45:03.756 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013436899999999999 2024-09-13T14:45:03.756 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.756 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.756 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:45:03.756 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.923018+0000", 2024-09-13T14:45:03.756 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0029198119999999999 2024-09-13T14:45:03.756 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.757 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.757 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.757 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.074680+0000", 2024-09-13T14:45:03.757 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.15166197100000001 2024-09-13T14:45:03.757 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.757 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.757 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.757 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.075859+0000", 2024-09-13T14:45:03.757 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0011794310000000001 2024-09-13T14:45:03.757 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.757 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.757 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:45:03.757 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.075919+0000", 2024-09-13T14:45:03.757 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.9697999999999997e-05 2024-09-13T14:45:03.758 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.758 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.758 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:45:03.758 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.075961+0000", 2024-09-13T14:45:03.758 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.2005000000000002e-05 2024-09-13T14:45:03.758 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.758 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:45:03.758 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.758 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.758 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.758 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:94769 7.5 7:a3d7bc9b:::benchmark_data_smithi151_190906_object94768:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1145)", 2024-09-13T14:45:03.758 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:45:00.889866+0000", 2024-09-13T14:45:03.758 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 2.7590901699999999, 2024-09-13T14:45:03.758 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.106925627, 2024-09-13T14:45:03.759 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:45:03.759 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:03.759 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:45:03.759 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:45:03.759 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:45:03.759 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 94769 2024-09-13T14:45:03.759 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.759 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:45:03.759 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.759 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:45:03.759 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.889866+0000", 2024-09-13T14:45:03.759 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.759 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.759 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.760 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:45:03.760 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.889866+0000", 2024-09-13T14:45:03.760 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.760 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.760 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.760 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:45:03.760 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.889864+0000", 2024-09-13T14:45:03.760 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:03.760 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.760 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.760 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:45:03.760 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.889912+0000", 2024-09-13T14:45:03.760 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7555999999999997e-05 2024-09-13T14:45:03.761 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.761 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.761 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:45:03.761 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.889925+0000", 2024-09-13T14:45:03.761 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3341e-05 2024-09-13T14:45:03.761 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.761 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.761 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:45:03.761 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.889931+0000", 2024-09-13T14:45:03.761 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.2039999999999996e-06 2024-09-13T14:45:03.761 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.761 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.761 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:45:03.762 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.889972+0000", 2024-09-13T14:45:03.762 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1616999999999998e-05 2024-09-13T14:45:03.762 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.762 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.762 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:45:03.762 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.890106+0000", 2024-09-13T14:45:03.762 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000133903 2024-09-13T14:45:03.762 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.762 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.762 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:03.762 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.890220+0000", 2024-09-13T14:45:03.762 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00011393200000000001 2024-09-13T14:45:03.762 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.762 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.763 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:45:03.763 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.892667+0000", 2024-09-13T14:45:03.763 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0024468350000000001 2024-09-13T14:45:03.763 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.763 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.763 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.763 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.893641+0000", 2024-09-13T14:45:03.763 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00097453800000000005 2024-09-13T14:45:03.763 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.763 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.763 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.763 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.996686+0000", 2024-09-13T14:45:03.763 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.10304476999999999 2024-09-13T14:45:03.763 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.764 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.764 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:45:03.764 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.996745+0000", 2024-09-13T14:45:03.764 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.9008999999999997e-05 2024-09-13T14:45:03.764 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.764 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.764 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:45:03.764 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.996791+0000", 2024-09-13T14:45:03.764 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.6208999999999998e-05 2024-09-13T14:45:03.764 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.764 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:45:03.764 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.764 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.764 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.765 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:96473 7.c 7:3ff07018:::benchmark_data_smithi151_190906_object96472:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1146)", 2024-09-13T14:45:03.765 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:45:02.766467+0000", 2024-09-13T14:45:03.765 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.88248933200000002, 2024-09-13T14:45:03.765 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.133263459, 2024-09-13T14:45:03.765 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:45:03.765 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:03.765 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:45:03.765 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:45:03.765 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:45:03.765 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 96473 2024-09-13T14:45:03.765 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.765 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:45:03.765 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.765 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:45:03.766 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.766467+0000", 2024-09-13T14:45:03.766 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.766 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.766 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.766 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:45:03.766 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.766467+0000", 2024-09-13T14:45:03.766 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.766 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.766 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.766 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:45:03.766 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.766465+0000", 2024-09-13T14:45:03.766 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:45:03.766 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.766 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.767 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:45:03.767 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.766528+0000", 2024-09-13T14:45:03.767 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3390000000000001e-05 2024-09-13T14:45:03.767 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.767 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.767 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:45:03.767 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.766540+0000", 2024-09-13T14:45:03.767 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2187999999999999e-05 2024-09-13T14:45:03.767 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.767 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.767 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:45:03.767 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.766546+0000", 2024-09-13T14:45:03.767 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7080000000000002e-06 2024-09-13T14:45:03.767 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.768 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.768 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:45:03.768 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.766693+0000", 2024-09-13T14:45:03.768 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014641900000000001 2024-09-13T14:45:03.768 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.768 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.768 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:45:03.768 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.766840+0000", 2024-09-13T14:45:03.768 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000147687 2024-09-13T14:45:03.768 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.768 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.768 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:45:03.768 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.766979+0000", 2024-09-13T14:45:03.769 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013896399999999999 2024-09-13T14:45:03.769 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.769 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.769 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:45:03.769 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.776889+0000", 2024-09-13T14:45:03.769 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0099098559999999999 2024-09-13T14:45:03.769 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.769 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.769 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.769 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.776917+0000", 2024-09-13T14:45:03.769 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.7616000000000001e-05 2024-09-13T14:45:03.769 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.769 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.769 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.769 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.899643+0000", 2024-09-13T14:45:03.770 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.122726135 2024-09-13T14:45:03.770 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.770 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.770 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:45:03.770 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.899693+0000", 2024-09-13T14:45:03.770 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0290999999999997e-05 2024-09-13T14:45:03.770 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.770 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.770 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:45:03.770 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.899730+0000", 2024-09-13T14:45:03.770 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.6897000000000003e-05 2024-09-13T14:45:03.770 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.770 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:45:03.770 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.771 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.771 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.771 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:96476 7.c 7:3981dbac:::benchmark_data_smithi151_190906_object96475:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1146)", 2024-09-13T14:45:03.771 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:45:02.766730+0000", 2024-09-13T14:45:03.771 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.88222570700000003, 2024-09-13T14:45:03.771 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.13698331499999999, 2024-09-13T14:45:03.771 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:45:03.771 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:03.771 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:45:03.771 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:45:03.771 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:45:03.771 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 96476 2024-09-13T14:45:03.771 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.771 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:45:03.772 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.772 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:45:03.772 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.766730+0000", 2024-09-13T14:45:03.772 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.772 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.772 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.772 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:45:03.772 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.766730+0000", 2024-09-13T14:45:03.772 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.772 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.772 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.772 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:45:03.772 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.766729+0000", 2024-09-13T14:45:03.773 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:45:03.773 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.773 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.773 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:45:03.773 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.766801+0000", 2024-09-13T14:45:03.773 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.1561000000000001e-05 2024-09-13T14:45:03.773 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.773 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.773 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:45:03.773 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.766810+0000", 2024-09-13T14:45:03.773 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.6740000000000003e-06 2024-09-13T14:45:03.773 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.773 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.773 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:45:03.774 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.766815+0000", 2024-09-13T14:45:03.774 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0529999999999997e-06 2024-09-13T14:45:03.774 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.774 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.774 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:45:03.774 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.770601+0000", 2024-09-13T14:45:03.774 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.003786588 2024-09-13T14:45:03.774 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.774 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.774 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:45:03.774 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.770766+0000", 2024-09-13T14:45:03.774 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000164578 2024-09-13T14:45:03.774 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.774 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.775 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:45:03.775 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.770950+0000", 2024-09-13T14:45:03.775 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000183876 2024-09-13T14:45:03.775 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.775 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.775 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:45:03.775 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.776893+0000", 2024-09-13T14:45:03.775 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0059437750000000001 2024-09-13T14:45:03.775 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.775 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.775 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.775 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.777748+0000", 2024-09-13T14:45:03.775 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00085424199999999996 2024-09-13T14:45:03.776 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.776 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.776 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.776 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.903618+0000", 2024-09-13T14:45:03.776 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.125870642 2024-09-13T14:45:03.776 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.776 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.776 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:45:03.776 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.903670+0000", 2024-09-13T14:45:03.776 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1505000000000002e-05 2024-09-13T14:45:03.776 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.776 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.776 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:45:03.776 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.903714+0000", 2024-09-13T14:45:03.777 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3686999999999997e-05 2024-09-13T14:45:03.777 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.777 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:45:03.777 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.777 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.777 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.777 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:96480 7.c 7:3a65f711:::benchmark_data_smithi151_190906_object96479:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1146)", 2024-09-13T14:45:03.777 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:45:02.768414+0000", 2024-09-13T14:45:03.777 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.88054232700000001, 2024-09-13T14:45:03.777 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.135415806, 2024-09-13T14:45:03.777 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:45:03.777 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:03.777 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:45:03.777 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:45:03.778 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:45:03.778 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 96480 2024-09-13T14:45:03.778 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.778 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:45:03.778 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.778 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:45:03.778 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.768414+0000", 2024-09-13T14:45:03.778 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.778 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.778 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.778 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:45:03.778 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.768414+0000", 2024-09-13T14:45:03.778 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:45:03.778 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.779 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.779 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:45:03.779 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.768413+0000", 2024-09-13T14:45:03.779 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:45:03.779 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.779 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.779 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:45:03.779 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.768472+0000", 2024-09-13T14:45:03.779 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.9524000000000001e-05 2024-09-13T14:45:03.779 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.779 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.779 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:45:03.779 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.768482+0000", 2024-09-13T14:45:03.779 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.3749999999999992e-06 2024-09-13T14:45:03.780 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.780 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.780 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:45:03.780 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.768486+0000", 2024-09-13T14:45:03.780 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8160000000000001e-06 2024-09-13T14:45:03.780 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.780 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.780 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:45:03.780 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.774065+0000", 2024-09-13T14:45:03.780 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0055783509999999996 2024-09-13T14:45:03.780 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.780 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.780 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:45:03.780 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.774169+0000", 2024-09-13T14:45:03.781 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00010398199999999999 2024-09-13T14:45:03.781 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.781 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.781 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:45:03.781 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.774287+0000", 2024-09-13T14:45:03.781 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000118515 2024-09-13T14:45:03.781 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.781 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.781 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:45:03.781 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.777057+0000", 2024-09-13T14:45:03.781 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0027700770000000001 2024-09-13T14:45:03.781 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.781 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.782 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.782 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.778312+0000", 2024-09-13T14:45:03.782 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0012551319999999999 2024-09-13T14:45:03.782 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.782 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.782 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:03.782 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.903753+0000", 2024-09-13T14:45:03.782 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.12544032499999999 2024-09-13T14:45:03.782 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.782 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.782 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:45:03.782 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.903799+0000", 2024-09-13T14:45:03.782 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.6254000000000001e-05 2024-09-13T14:45:03.782 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:45:03.783 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:45:03.783 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:45:03.783 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:02.903829+0000", 2024-09-13T14:45:03.783 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.0409e-05 2024-09-13T14:45:03.783 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.783 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:45:03.783 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.783 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:45:03.783 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:45:03.783 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:45:03.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:03.784 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:03.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:03.885 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:03.916 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:03.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:03.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:04.018 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:04.051 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:04.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:04.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:04.153 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:04.174 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:04.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:04.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:04.276 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:04.312 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:04.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:04.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:04.414 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:04.435 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:04.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:04.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:04.537 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:04.572 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:04.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:04.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:04.674 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:04.720 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:04.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:04.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:04.734 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 63 16 98181 98165 97.3629 54.5625 0.0172795 0.010266 2024-09-13T14:45:04.749 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:45:04.822 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:04.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:04 smithi184 ceph-mon[92527]: pgmap v1521: 119 pgs: 119 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-13T14:45:04.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:04 smithi151 ceph-mon[108966]: pgmap v1521: 119 pgs: 119 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-13T14:45:04.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:45:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:45:04] "GET /metrics HTTP/1.1" 200 36662 "" "Prometheus/2.51.0" 2024-09-13T14:45:04.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:04 smithi151 ceph-mon[104450]: pgmap v1521: 119 pgs: 119 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-13T14:45:04.874 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:04.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:04.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:04.976 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:05.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:05.024 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:05.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:05.125 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:05.153 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:05.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:05.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:05.254 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:05.287 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:05.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:05.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:05.389 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:05.416 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:05.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:05.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:05.518 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:05.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:05.552 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:05.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:05.653 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:05.684 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:05.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:05.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:05.734 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 64 16 98980 98964 96.6219 49.9375 0.00939094 0.010345 2024-09-13T14:45:05.786 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:05.809 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:05.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:05.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:05.911 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:05.949 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:05.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:05.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:06.051 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:06.085 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:06.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:06.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:06.187 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:06.213 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:06.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:06.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:06.315 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:06.347 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:06.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:06.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:06.449 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:06.473 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:06.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:06.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:06.574 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:06.614 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:06.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:06.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:06.716 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:06.734 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 65 16 99923 99907 96.042 58.9375 0.0215708 0.0104073 2024-09-13T14:45:06.750 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:06.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:06.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:06.852 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:06.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:06 smithi184 ceph-mon[92527]: pgmap v1522: 119 pgs: 119 active+clean; 5.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-13T14:45:06.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:06 smithi151 ceph-mon[108966]: pgmap v1522: 119 pgs: 119 active+clean; 5.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-13T14:45:06.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:06 smithi151 ceph-mon[104450]: pgmap v1522: 119 pgs: 119 active+clean; 5.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-13T14:45:06.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:06.885 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:06.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:06.987 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:07.008 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:07.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:07.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:07.110 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:07.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:07.144 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:07.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:07.245 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:07.281 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:07.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:07.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:07.382 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:07.416 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:07.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:07.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:07.518 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:07.545 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:07.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:07.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:07.647 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:07.684 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:07.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:07.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:07.734 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 66 16 100957 100941 95.5659 64.625 0.0135541 0.0104595 2024-09-13T14:45:07.786 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:07.813 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:07.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:07.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:07.915 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:07.940 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:07.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:07.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:07.944 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T14:45:08.005 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:08.012 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:08.042 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:08.062 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:08.066 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:08.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:08.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:08.168 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:08.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:08.198 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:08.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:08.299 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:08.352 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:08.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:08.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:08.454 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:08.484 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:08.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:08.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:08.586 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:08.641 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:08.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:08.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:08.677 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:08 smithi151 ceph-mon[104450]: pgmap v1523: 119 pgs: 119 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T14:45:08.678 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:08 smithi151 ceph-mon[108966]: pgmap v1523: 119 pgs: 119 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T14:45:08.734 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 67 16 101953 101937 95.0686 62.25 0.0138078 0.0105135 2024-09-13T14:45:08.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:08.783 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:08.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:08.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:08.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:08 smithi184 ceph-mon[92527]: pgmap v1523: 119 pgs: 119 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T14:45:08.885 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:08.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:08.919 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:08.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:09.020 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:09.055 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:09.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:09.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:09.157 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:09.196 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:09.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:09.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:09.299 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:09.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:09.331 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:09.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:09.433 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:09.464 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:09.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:09.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:09.566 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:09.598 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:09.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:09.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:09.699 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:09.734 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 68 16 103097 103081 94.722 71.5 0.0168731 0.0105521 2024-09-13T14:45:09.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:09.736 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:09.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:09.837 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:09.880 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:09.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:09.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:09.982 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:10.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:10.013 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:10.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:10.114 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:10.142 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:10.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:10.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:10.244 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:10.273 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:10.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:10.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:10.375 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:10.408 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:10.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:10.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:10.510 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:10.546 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:10.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:10.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:10.586 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:45:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:45:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:45:10.648 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:10.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:10.681 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:10.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:10.734 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 69 16 104262 104246 94.4044 72.8125 0.00634286 0.0105876 2024-09-13T14:45:10.782 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:10.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:10.817 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:10.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:10.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:10 smithi184 ceph-mon[92527]: pgmap v1524: 119 pgs: 119 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 61 MiB/s wr, 973 op/s 2024-09-13T14:45:10.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:10 smithi151 ceph-mon[104450]: pgmap v1524: 119 pgs: 119 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 61 MiB/s wr, 973 op/s 2024-09-13T14:45:10.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:10 smithi151 ceph-mon[108966]: pgmap v1524: 119 pgs: 119 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 61 MiB/s wr, 973 op/s 2024-09-13T14:45:10.918 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:10.951 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:10.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:10.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:11.052 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:11.080 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:11.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:11.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:11.182 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:11.221 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:11.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:11.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:11.322 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:11.368 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:11.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:11.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:11.469 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:11.517 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:11.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:11.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:11.619 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:11.657 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:11.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:11.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:11.735 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 70 16 105382 105366 94.0557 70 0.0200104 0.0106271 2024-09-13T14:45:11.760 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:11.791 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:11.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:11.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:11.893 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:11.921 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:11.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:11.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:12.023 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:12.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:12.068 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:12.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:12.170 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:12.198 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:12.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:12.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:12.300 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:12.335 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:12.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:12.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:12.437 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:12.477 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:12.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:12.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:12.578 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:12.615 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:12.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:12.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:12.717 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:12.740 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 71 16 106536 106520 93.7468 72.125 0.0169448 0.0106625 2024-09-13T14:45:12.774 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:12.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:12.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:12.817 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:12 smithi151 ceph-mon[108966]: pgmap v1525: 119 pgs: 119 active+clean; 6.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 60 MiB/s wr, 966 op/s 2024-09-13T14:45:12.820 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:12 smithi151 ceph-mon[104450]: pgmap v1525: 119 pgs: 119 active+clean; 6.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 60 MiB/s wr, 966 op/s 2024-09-13T14:45:12.829 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:12.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:12 smithi184 ceph-mon[92527]: pgmap v1525: 119 pgs: 119 active+clean; 6.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 60 MiB/s wr, 966 op/s 2024-09-13T14:45:12.876 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:12.904 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:12.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:12.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:13.006 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:13.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:13.046 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:13.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:13.147 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:13.182 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:13.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:13.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:13.284 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:13.317 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:13.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:13.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:13.420 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:13.463 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:13.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:13.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:13.565 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:13.594 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:13.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:13.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:13.696 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:13.724 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:13.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:13.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:13.735 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 72 16 107588 107572 93.3579 65.75 0.0291172 0.0107057 2024-09-13T14:45:13.807 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:45:13.807 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:45:13.807 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:45:13.807 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:45:13.808 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:45:13.808 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:45:13.808 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:45:13.808 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:45:13.815 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:45:13.815 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:45:13.826 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:13.846 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:45:13.846 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:45:13.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:13.856 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:13.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:13.877 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:45:13.878 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:45:13.907 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:45:13.908 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:45:13.957 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:13.990 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:13.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:13.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:14.092 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:14.122 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:14.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:14.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:14.224 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:14.252 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:14.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:14.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:14.354 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:14.388 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:14.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:14.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:14.490 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:14.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:14.527 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:14.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:14.622 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-13T14:45:14.628 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:14.654 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:14.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:14.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:14.735 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 73 16 108603 108587 92.948 63.4375 0.00818666 0.0107541 2024-09-13T14:45:14.755 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:14.787 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:14.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:14.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:14.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:14 smithi184 ceph-mon[92527]: pgmap v1526: 119 pgs: 119 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T14:45:14.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:14 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2408911724' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:14.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:14 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1059679992' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:14 smithi151 ceph-mon[104450]: pgmap v1526: 119 pgs: 119 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T14:45:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:14 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2408911724' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:14 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1059679992' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:14 smithi151 ceph-mon[108966]: pgmap v1526: 119 pgs: 119 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T14:45:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:14 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2408911724' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:14 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1059679992' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:14.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:45:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:45:14] "GET /metrics HTTP/1.1" 200 36662 "" "Prometheus/2.51.0" 2024-09-13T14:45:14.888 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:14.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:14.927 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:14.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:15.028 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:15.062 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:15.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:15.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:15.164 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:15.196 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:15.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:15.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:15.264 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:45:15.264 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:45:15.297 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:15.335 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:15.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:15.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:15.436 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:15.473 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:15.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:15.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:15.574 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:15.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:15.613 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:15.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:15.714 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:15.735 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 74 16 109684 109668 92.6042 67.5625 0.00667069 0.0107933 2024-09-13T14:45:15.752 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:15.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:15.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:15.778 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:15 smithi151 ceph-mon[108966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:15.778 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:15 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1059679992' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:45:15.778 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:15 smithi151 ceph-mon[108966]: osdmap e1147: 8 total, 8 up, 8 in 2024-09-13T14:45:15.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:15 smithi151 ceph-mon[104450]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:15.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:15 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1059679992' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:45:15.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:15 smithi151 ceph-mon[104450]: osdmap e1147: 8 total, 8 up, 8 in 2024-09-13T14:45:15.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:15 smithi184 ceph-mon[92527]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:15.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:15 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1059679992' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:45:15.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:15 smithi184 ceph-mon[92527]: osdmap e1147: 8 total, 8 up, 8 in 2024-09-13T14:45:15.854 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:15.889 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:15.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:15.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:15.991 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:16.027 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:16.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:16.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:16.129 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:16.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:16.163 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:16.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:16.264 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:16.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:16.299 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:16.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:16.400 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:16.435 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:16.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:16.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:16.537 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:16.576 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:16.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:16.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:16.678 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:16.708 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:16.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:16.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:16.735 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 75 16 110764 110748 92.2694 67.5 0.0100831 0.0108332 2024-09-13T14:45:16.810 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:16.851 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:16.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:16.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:16.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:16 smithi184 ceph-mon[92527]: pgmap v1528: 119 pgs: 119 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T14:45:16.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:16.878 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:16 smithi151 ceph-mon[108966]: pgmap v1528: 119 pgs: 119 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T14:45:16.878 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:16.878 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:16 smithi151 ceph-mon[104450]: pgmap v1528: 119 pgs: 119 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T14:45:16.878 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:16.953 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:16.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:16.982 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:16.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:16.990 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:45:17.083 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:17.111 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:17.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:17.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:17.212 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:17.240 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:17.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:17.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:17.342 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:17.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:17.372 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:17.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:17.473 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:17.503 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:17.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:17.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:17.605 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:17.636 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:17.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:17.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:17.736 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 76 16 111789 111773 91.8982 64.0625 0.0382804 0.0108719 2024-09-13T14:45:17.737 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:17.762 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:17.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:17.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:17.864 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:17.896 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:17.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:17.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:17.998 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:18.030 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:18.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:18.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:18.132 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:18.164 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:18.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:18.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:18.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:18.286 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T14:45:18.307 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:18.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:18.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:18.409 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:18.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:18.456 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:18.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:18.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:18.604 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:18.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:18.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:18.706 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:18.728 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:45:18.739 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 77 16 112821 112805 91.5384 64.5 0.0185252 0.010919 2024-09-13T14:45:18.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:18.750 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:18.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:18.852 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:18.880 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:18.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:18.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:18.912 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:18 smithi151 ceph-mon[104450]: pgmap v1529: 119 pgs: 119 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-13T14:45:18.912 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:18 smithi151 ceph-mon[108966]: pgmap v1529: 119 pgs: 119 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-13T14:45:18.982 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:18.998 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:18.999 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:19.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:19.062 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:19.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:19.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:18 smithi184 ceph-mon[92527]: pgmap v1529: 119 pgs: 119 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-13T14:45:19.163 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:19.190 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:19.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:19.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:19.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:19.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:19.327 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:19.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:19.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:19.471 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:19.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:19.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:19.573 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:19.599 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:19.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:19.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:19.633 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T14:45:19.701 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:19.722 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:19.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:19.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:19.739 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 78 16 113676 113660 91.0499 53.4375 0.00786116 0.0109667 2024-09-13T14:45:19.755 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:19 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/334890671' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:45:19.756 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:19 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:45:19.756 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:19 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/334890671' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:45:19.756 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:19 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:45:19.824 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:19.846 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:19.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:19.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:19.948 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:19.970 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:19.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:19.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:20.072 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:20.093 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:20.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:20.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:20.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:19 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/334890671' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:45:20.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:19 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:45:20.196 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:20.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:20.220 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:20.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:20.321 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:20.343 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:20.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:20.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:20.444 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:20.468 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:20.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:20.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:20.570 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:20.592 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:20.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:20.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:20.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:45:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:45:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:45:20.694 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:20.715 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:20.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:20.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:20.739 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 79 16 113676 113660 89.8976 0 - 0.0109667 2024-09-13T14:45:20.817 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:20.819 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:20 smithi151 ceph-mon[104450]: pgmap v1530: 119 pgs: 119 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-13T14:45:20.819 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:20 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:45:20.819 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:20 smithi151 ceph-mon[104450]: osdmap e1148: 8 total, 8 up, 8 in 2024-09-13T14:45:20.819 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:20 smithi151 ceph-mon[104450]: osdmap e1149: 8 total, 8 up, 8 in 2024-09-13T14:45:20.820 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:20 smithi151 ceph-mon[108966]: pgmap v1530: 119 pgs: 119 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-13T14:45:20.820 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:20 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:45:20.820 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:20 smithi151 ceph-mon[108966]: osdmap e1148: 8 total, 8 up, 8 in 2024-09-13T14:45:20.820 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:20 smithi151 ceph-mon[108966]: osdmap e1149: 8 total, 8 up, 8 in 2024-09-13T14:45:20.838 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:20.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:20.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:20.940 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:20.962 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:20.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:20.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:21.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:21.097 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:21.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:21.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:21.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:20 smithi184 ceph-mon[92527]: pgmap v1530: 119 pgs: 119 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-13T14:45:21.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:20 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:45:21.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:20 smithi184 ceph-mon[92527]: osdmap e1148: 8 total, 8 up, 8 in 2024-09-13T14:45:21.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:20 smithi184 ceph-mon[92527]: osdmap e1149: 8 total, 8 up, 8 in 2024-09-13T14:45:21.199 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:21.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:21.225 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:21.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:21.326 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:21.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:21.350 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:21.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:21.451 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:21.475 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:21.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:21.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:21.577 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:21.602 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:21.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:21.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:21.704 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:21.727 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:21.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:21.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:21.740 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:2024-09-13T14:45:21.739773+0000 min lat: 0.00109766 max lat: 1.54218 avg lat: 0.0112131 2024-09-13T14:45:21.740 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:45:21.740 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 80 16 114133 114117 89.131 14.2812 0.018715 0.0112131 2024-09-13T14:45:21.742 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-13T14:45:21.829 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:21.884 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:21.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:21.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:21.963 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:21 smithi151 ceph-mon[104450]: 1.1e scrub starts 2024-09-13T14:45:21.963 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:21 smithi151 ceph-mon[104450]: osdmap e1150: 8 total, 8 up, 8 in 2024-09-13T14:45:21.964 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:21 smithi151 ceph-mon[104450]: 7.c starting backfill to osd.3 from (0'0,0'0] MAX to 1147'7023 2024-09-13T14:45:21.964 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:21 smithi151 ceph-mon[104450]: 7.f starting backfill to osd.3 from (0'0,0'0] MAX to 1147'7150 2024-09-13T14:45:21.964 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:21 smithi151 ceph-mon[104450]: 7.6 starting backfill to osd.3 from (0'0,0'0] MAX to 1147'7083 2024-09-13T14:45:21.964 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:21 smithi151 ceph-mon[104450]: 7.2 starting backfill to osd.3 from (0'0,0'0] MAX to 1147'7158 2024-09-13T14:45:21.964 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:21 smithi151 ceph-mon[104450]: 7.1 starting backfill to osd.3 from (0'0,0'0] MAX to 1147'7131 2024-09-13T14:45:21.964 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:21 smithi151 ceph-mon[104450]: 7.0 starting backfill to osd.3 from (0'0,0'0] MAX to 1147'7193 2024-09-13T14:45:21.964 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:21 smithi151 ceph-mon[104450]: 7.7 starting backfill to osd.3 from (0'0,0'0] MAX to 1147'7026 2024-09-13T14:45:21.964 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:21 smithi151 ceph-mon[108966]: 1.1e scrub starts 2024-09-13T14:45:21.964 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:21 smithi151 ceph-mon[108966]: osdmap e1150: 8 total, 8 up, 8 in 2024-09-13T14:45:21.964 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:21 smithi151 ceph-mon[108966]: 7.c starting backfill to osd.3 from (0'0,0'0] MAX to 1147'7023 2024-09-13T14:45:21.964 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:21 smithi151 ceph-mon[108966]: 7.f starting backfill to osd.3 from (0'0,0'0] MAX to 1147'7150 2024-09-13T14:45:21.965 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:21 smithi151 ceph-mon[108966]: 7.6 starting backfill to osd.3 from (0'0,0'0] MAX to 1147'7083 2024-09-13T14:45:21.965 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:21 smithi151 ceph-mon[108966]: 7.2 starting backfill to osd.3 from (0'0,0'0] MAX to 1147'7158 2024-09-13T14:45:21.965 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:21 smithi151 ceph-mon[108966]: 7.1 starting backfill to osd.3 from (0'0,0'0] MAX to 1147'7131 2024-09-13T14:45:21.965 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:21 smithi151 ceph-mon[108966]: 7.0 starting backfill to osd.3 from (0'0,0'0] MAX to 1147'7193 2024-09-13T14:45:21.965 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:21 smithi151 ceph-mon[108966]: 7.7 starting backfill to osd.3 from (0'0,0'0] MAX to 1147'7026 2024-09-13T14:45:21.986 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:22.009 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:22.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:22.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:22.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:21 smithi184 ceph-mon[92527]: 1.1e scrub starts 2024-09-13T14:45:22.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:21 smithi184 ceph-mon[92527]: osdmap e1150: 8 total, 8 up, 8 in 2024-09-13T14:45:22.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:21 smithi184 ceph-mon[92527]: 7.c starting backfill to osd.3 from (0'0,0'0] MAX to 1147'7023 2024-09-13T14:45:22.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:21 smithi184 ceph-mon[92527]: 7.f starting backfill to osd.3 from (0'0,0'0] MAX to 1147'7150 2024-09-13T14:45:22.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:21 smithi184 ceph-mon[92527]: 7.6 starting backfill to osd.3 from (0'0,0'0] MAX to 1147'7083 2024-09-13T14:45:22.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:21 smithi184 ceph-mon[92527]: 7.2 starting backfill to osd.3 from (0'0,0'0] MAX to 1147'7158 2024-09-13T14:45:22.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:21 smithi184 ceph-mon[92527]: 7.1 starting backfill to osd.3 from (0'0,0'0] MAX to 1147'7131 2024-09-13T14:45:22.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:21 smithi184 ceph-mon[92527]: 7.0 starting backfill to osd.3 from (0'0,0'0] MAX to 1147'7193 2024-09-13T14:45:22.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:21 smithi184 ceph-mon[92527]: 7.7 starting backfill to osd.3 from (0'0,0'0] MAX to 1147'7026 2024-09-13T14:45:22.111 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:22.136 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:22.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:22.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:22.238 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:22.273 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:22.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:22.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:22.375 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:22.417 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:22.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:22.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:22.519 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:22.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:22.569 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:22.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:22.670 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:22.696 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:22.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:22.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:22.740 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 81 16 114796 114780 88.5419 41.4375 0.0264533 0.011288 2024-09-13T14:45:22.798 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:22.836 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:22.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:22.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:22.938 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:22.962 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:22.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:22.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:23.010 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:22 smithi151 ceph-mon[104450]: pgmap v1534: 119 pgs: 2 remapped+peering, 117 active+clean; 6.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T14:45:23.010 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:22 smithi151 ceph-mon[104450]: 1.7 scrub starts 2024-09-13T14:45:23.011 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:22 smithi151 ceph-mon[108966]: pgmap v1534: 119 pgs: 2 remapped+peering, 117 active+clean; 6.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T14:45:23.011 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:22 smithi151 ceph-mon[108966]: 1.7 scrub starts 2024-09-13T14:45:23.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:23.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:23.093 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:23.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:22 smithi184 ceph-mon[92527]: pgmap v1534: 119 pgs: 2 remapped+peering, 117 active+clean; 6.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T14:45:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:22 smithi184 ceph-mon[92527]: 1.7 scrub starts 2024-09-13T14:45:23.194 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:23.216 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:23.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:23.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:23.317 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:23.343 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:23.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:23.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:23.445 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:23.468 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:23.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:23.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:23.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:23.626 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:23.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:23.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:23.631 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:45:23.631 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:45:23.661 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:45:23.661 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:45:23.698 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:45:23.698 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:45:23.727 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:23.731 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:45:23.731 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:45:23.740 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 82 16 115407 115391 87.9279 38.1875 0.0180027 0.0113673 2024-09-13T14:45:23.751 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:23.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:23.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:23.759 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:45:23.760 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:45:23.796 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:45:23.796 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:45:23.834 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:45:23.834 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:45:23.853 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:23.874 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:45:23.874 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:45:23.875 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:23.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:23.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:23.916 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:23 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/950145431' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:23.917 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:23 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/950145431' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:23.977 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:24.007 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:45:24.007 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-13T14:45:24.007 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-13T14:45:24.007 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-13T14:45:24.008 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:45:24.027 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:24.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:24.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:24.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:23 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/950145431' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:24.129 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:24.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:24.168 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:24.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:24.269 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:24.294 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:24.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:24.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:24.397 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:24.440 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:24.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:24.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:24.544 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:24.587 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:24.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:24.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:24.689 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:24.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:24.728 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:24.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:24.741 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 83 16 116068 116052 87.3663 41.3125 0.00519157 0.0114394 2024-09-13T14:45:24.829 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:24.831 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:45:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:45:24] "GET /metrics HTTP/1.1" 200 36659 "" "Prometheus/2.51.0" 2024-09-13T14:45:24.852 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:24.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:24.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:24.954 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:24.976 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:24.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:24.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:24.985 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:25.023 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T14:45:25.078 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:25.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:24 smithi184 ceph-mon[92527]: pgmap v1535: 119 pgs: 1 active+recovery_wait+remapped, 3 active+remapped+backfill_wait, 2 remapped+peering, 113 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 55 MiB/s wr, 885 op/s; 28698/341754 objects misplaced (8.397%); 320 KiB/s, 4 objects/s recovering 2024-09-13T14:45:25.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:24 smithi151 ceph-mon[104450]: pgmap v1535: 119 pgs: 1 active+recovery_wait+remapped, 3 active+remapped+backfill_wait, 2 remapped+peering, 113 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 55 MiB/s wr, 885 op/s; 28698/341754 objects misplaced (8.397%); 320 KiB/s, 4 objects/s recovering 2024-09-13T14:45:25.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:24 smithi151 ceph-mon[108966]: pgmap v1535: 119 pgs: 1 active+recovery_wait+remapped, 3 active+remapped+backfill_wait, 2 remapped+peering, 113 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 55 MiB/s wr, 885 op/s; 28698/341754 objects misplaced (8.397%); 320 KiB/s, 4 objects/s recovering 2024-09-13T14:45:25.151 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:25.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:25.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:25.253 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:25.275 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:25.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:25.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:25.377 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:25.429 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:25.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:25.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:25.503 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:25.530 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:25.556 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:25.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:25.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:25.658 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:25.707 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:25.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:25.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:25.740 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 84 16 116701 116685 86.7973 39.5625 0.0270383 0.0115141 2024-09-13T14:45:25.809 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:25.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:25.836 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:25.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:25.937 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:25.972 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:25.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:25.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:26.074 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:26.122 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:26.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:26.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:26.225 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:26.248 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:26.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:26.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:26.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:26.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:26.390 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:26.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:26.491 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:26.515 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:26.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:26.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:26.617 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:26.655 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:26.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:26.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:26.740 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 85 16 117333 117317 86.2409 39.5 0.0153496 0.0115893 2024-09-13T14:45:26.758 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:26.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:26.802 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:26.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:26.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:26 smithi184 ceph-mon[92527]: pgmap v1536: 119 pgs: 1 active+recovery_wait+remapped, 6 active+remapped+backfill_wait, 2 remapped+peering, 110 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 2.8 KiB/s rd, 32 MiB/s wr, 521 op/s; 50335/344124 objects misplaced (14.627%); 375 KiB/s, 5 objects/s recovering 2024-09-13T14:45:26.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:26 smithi184 ceph-mon[92527]: 1.1e scrub starts 2024-09-13T14:45:26.904 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:26.948 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:26.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:26.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:27.050 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:27.074 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:27.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:27.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:27.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:26 smithi151 ceph-mon[104450]: pgmap v1536: 119 pgs: 1 active+recovery_wait+remapped, 6 active+remapped+backfill_wait, 2 remapped+peering, 110 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 2.8 KiB/s rd, 32 MiB/s wr, 521 op/s; 50335/344124 objects misplaced (14.627%); 375 KiB/s, 5 objects/s recovering 2024-09-13T14:45:27.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:26 smithi151 ceph-mon[104450]: 1.1e scrub starts 2024-09-13T14:45:27.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:26 smithi151 ceph-mon[108966]: pgmap v1536: 119 pgs: 1 active+recovery_wait+remapped, 6 active+remapped+backfill_wait, 2 remapped+peering, 110 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 2.8 KiB/s rd, 32 MiB/s wr, 521 op/s; 50335/344124 objects misplaced (14.627%); 375 KiB/s, 5 objects/s recovering 2024-09-13T14:45:27.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:26 smithi151 ceph-mon[108966]: 1.1e scrub starts 2024-09-13T14:45:27.176 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:27.198 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:27.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:27.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:27.300 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:27.348 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:27.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:27.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:27.450 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:27.476 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:27.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:27.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:27.530 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:45:27.578 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:27.601 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:27.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:27.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:27.703 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:27.741 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 86 16 117993 117977 85.7175 41.25 0.0656852 0.0116602 2024-09-13T14:45:27.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:27.756 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:27.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:27.857 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:27.916 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:27.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:27.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:28.018 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:28.070 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:28.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:28.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:28.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:27 smithi184 ceph-mon[92527]: 1.7 scrub starts 2024-09-13T14:45:28.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:27 smithi151 ceph-mon[104450]: 1.7 scrub starts 2024-09-13T14:45:28.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:27 smithi151 ceph-mon[108966]: 1.7 scrub starts 2024-09-13T14:45:28.172 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:28.195 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:28.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:28.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:28.297 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:28.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:28.322 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:28.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:28.385 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:28.422 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:28.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:28.469 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:28.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:28.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:28.619 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:28.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:28.620 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:28.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:28.721 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:28.741 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 87 16 118606 118590 85.1723 38.3125 0.143888 0.0117326 2024-09-13T14:45:28.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:28.746 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:28.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:28.847 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:28.857 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:28 smithi151 ceph-mon[104450]: pgmap v1537: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 2.2 KiB/s rd, 38 MiB/s wr, 602 op/s; 64848/348831 objects misplaced (18.590%); 683 KiB/s, 10 objects/s recovering 2024-09-13T14:45:28.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:28.887 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:28.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:28.988 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:29.025 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:29.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:29.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:29.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:28 smithi184 ceph-mon[92527]: pgmap v1537: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 2.2 KiB/s rd, 38 MiB/s wr, 602 op/s; 64848/348831 objects misplaced (18.590%); 683 KiB/s, 10 objects/s recovering 2024-09-13T14:45:29.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:28 smithi151 ceph-mon[108966]: pgmap v1537: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 2.2 KiB/s rd, 38 MiB/s wr, 602 op/s; 64848/348831 objects misplaced (18.590%); 683 KiB/s, 10 objects/s recovering 2024-09-13T14:45:29.127 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:29.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:29.150 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:29.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:29.252 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:29.273 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:29.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:29.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:29.375 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:29.412 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:29.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:29.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:29.514 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:29.552 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:29.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:29.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:29.654 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:29.700 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:29.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:29.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:29.741 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 88 16 119268 119252 84.6746 41.375 0.0807643 0.0118041 2024-09-13T14:45:29.802 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:29.828 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:29.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:29.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:29.930 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:29.953 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:29.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:29.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:30.055 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:30.101 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:30.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:30.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:30.203 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:30.226 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:30.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:30.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:30.327 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:30.372 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:30.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:30.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:30.474 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:30.518 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:30.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:30.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:30.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:45:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:45:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:45:30.620 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:30.643 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:30.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:30.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:30.741 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 89 16 119908 119892 84.1727 40 0.0169676 0.0118747 2024-09-13T14:45:30.746 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:30.769 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:30.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:30.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:30.871 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:30.894 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:30.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:30.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:30.901 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:30 smithi151 ceph-mon[104450]: pgmap v1538: 119 pgs: 2 active+clean+scrubbing, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 7 active+remapped+backfill_wait, 108 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.8 KiB/s rd, 41 MiB/s wr, 653 op/s; 65855/353637 objects misplaced (18.622%); 552 KiB/s, 8 objects/s recovering 2024-09-13T14:45:30.901 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:30 smithi151 ceph-mon[108966]: pgmap v1538: 119 pgs: 2 active+clean+scrubbing, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 7 active+remapped+backfill_wait, 108 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.8 KiB/s rd, 41 MiB/s wr, 653 op/s; 65855/353637 objects misplaced (18.622%); 552 KiB/s, 8 objects/s recovering 2024-09-13T14:45:30.996 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:31.020 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:31.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:31.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:31.122 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:31.146 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:31.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:31.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:31.247 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:31.273 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:31.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:31.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:31.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:30 smithi184 ceph-mon[92527]: pgmap v1538: 119 pgs: 2 active+clean+scrubbing, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 7 active+remapped+backfill_wait, 108 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.8 KiB/s rd, 41 MiB/s wr, 653 op/s; 65855/353637 objects misplaced (18.622%); 552 KiB/s, 8 objects/s recovering 2024-09-13T14:45:31.375 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:31.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:31.431 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:31.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:31.532 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:31.572 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:31.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:31.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:31.674 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:31.726 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:31.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:31.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:31.741 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 90 6 120458 120452 83.6264 35 0.0161075 0.0119505 2024-09-13T14:45:31.828 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:31.849 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:31.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:31.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:31.861 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T14:45:31.861 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-13T14:45:31.951 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:31.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:31.974 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:31.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:32.074 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:32.097 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:32.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:32.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:32.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:31 smithi151 ceph-mon[108966]: 1.1e scrub starts 2024-09-13T14:45:32.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:31 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/116502264' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:45:32.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:31 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:45:32.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:32.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:45:32.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:31 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1671278503' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:45:32.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:31 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:45:32.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:31 smithi151 ceph-mon[104450]: 1.1e scrub starts 2024-09-13T14:45:32.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:31 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/116502264' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:45:32.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:31 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:45:32.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:32.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:45:32.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:31 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1671278503' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:45:32.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:31 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:45:32.199 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:32.221 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:32.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:32.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:32.324 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:32.346 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:32.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:32.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:32.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:31 smithi184 ceph-mon[92527]: 1.1e scrub starts 2024-09-13T14:45:32.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:31 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/116502264' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:45:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:31 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:45:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:45:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:31 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1671278503' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:45:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:31 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:45:32.447 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:32.469 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:32.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:32.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:32.571 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:32.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:32.595 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:32.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:32.696 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:32.718 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:32.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:32.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:32.741 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Total time run: 90.0355 2024-09-13T14:45:32.742 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Total writes made: 120458 2024-09-13T14:45:32.742 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Write size: 65536 2024-09-13T14:45:32.742 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Object size: 65536 2024-09-13T14:45:32.742 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Bandwidth (MB/sec): 83.6184 2024-09-13T14:45:32.742 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Stddev Bandwidth: 37.7558 2024-09-13T14:45:32.742 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Max bandwidth (MB/sec): 205.562 2024-09-13T14:45:32.742 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Min bandwidth (MB/sec): 0 2024-09-13T14:45:32.742 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Average IOPS: 1337 2024-09-13T14:45:32.742 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Stddev IOPS: 604.103 2024-09-13T14:45:32.742 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Max IOPS: 3289 2024-09-13T14:45:32.742 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Min IOPS: 0 2024-09-13T14:45:32.742 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Average Latency(s): 0.0119525 2024-09-13T14:45:32.742 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Stddev Latency(s): 0.0208238 2024-09-13T14:45:32.743 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Max latency(s): 1.54218 2024-09-13T14:45:32.743 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Min latency(s): 0.00109766 2024-09-13T14:45:32.748 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T14:45:32.821 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:32.845 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:32.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:32.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:32.948 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:32.993 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:32.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:32.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:33.095 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:33.121 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:33.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:33.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:33.223 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:33.246 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:33.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:33.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:33.348 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:33.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:32 smithi184 ceph-mon[92527]: pgmap v1539: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 1 active+recovery_wait+remapped, 7 active+remapped+backfill_wait, 108 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.6 KiB/s rd, 36 MiB/s wr, 584 op/s; 65994/356022 objects misplaced (18.536%); 825 KiB/s, 12 objects/s recovering 2024-09-13T14:45:33.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:32 smithi184 ceph-mon[92527]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:33.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:32 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:45:33.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:32 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:45:33.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:32 smithi184 ceph-mon[92527]: osdmap e1151: 8 total, 8 up, 8 in 2024-09-13T14:45:33.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:32 smithi184 ceph-mon[92527]: 1.7 scrub starts 2024-09-13T14:45:33.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:32 smithi151 ceph-mon[104450]: pgmap v1539: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 1 active+recovery_wait+remapped, 7 active+remapped+backfill_wait, 108 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.6 KiB/s rd, 36 MiB/s wr, 584 op/s; 65994/356022 objects misplaced (18.536%); 825 KiB/s, 12 objects/s recovering 2024-09-13T14:45:33.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:32 smithi151 ceph-mon[104450]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:33.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:32 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:45:33.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:32 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:45:33.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:32 smithi151 ceph-mon[104450]: osdmap e1151: 8 total, 8 up, 8 in 2024-09-13T14:45:33.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:32 smithi151 ceph-mon[104450]: 1.7 scrub starts 2024-09-13T14:45:33.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:32 smithi151 ceph-mon[108966]: pgmap v1539: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 1 active+recovery_wait+remapped, 7 active+remapped+backfill_wait, 108 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.6 KiB/s rd, 36 MiB/s wr, 584 op/s; 65994/356022 objects misplaced (18.536%); 825 KiB/s, 12 objects/s recovering 2024-09-13T14:45:33.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:32 smithi151 ceph-mon[108966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:33.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:32 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:45:33.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:32 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:45:33.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:32 smithi151 ceph-mon[108966]: osdmap e1151: 8 total, 8 up, 8 in 2024-09-13T14:45:33.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:32 smithi151 ceph-mon[108966]: 1.7 scrub starts 2024-09-13T14:45:33.373 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:33.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:33.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:33.475 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:33.501 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:33.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:33.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:33.603 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:33.627 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:33.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:33.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:33.729 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:33.754 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:33.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:33.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:33.856 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:33.879 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:33.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:33.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:33.981 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:34.018 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:34.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:34.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:34.120 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:34.146 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:34.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:34.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:34.248 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:34.271 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:34.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:34.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:34.373 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:34.394 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:34.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:34.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:34.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:34.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:34.541 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:34.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:34.642 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:34.664 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:34.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:34.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:34.766 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:34.791 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:34.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:34.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:34.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:45:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:45:34] "GET /metrics HTTP/1.1" 200 36669 "" "Prometheus/2.51.0" 2024-09-13T14:45:34.893 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:34.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:34.940 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:34.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:35.041 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:35.062 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:35.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:35.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:35.164 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:35.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:35.189 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:35.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:35.210 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:34 smithi151 ceph-mon[104450]: pgmap v1541: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 7 active+remapped+backfill_wait, 108 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.7 KiB/s rd, 37 MiB/s wr, 595 op/s; 66488/359592 objects misplaced (18.490%); 686 KiB/s, 10 objects/s recovering 2024-09-13T14:45:35.210 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:34 smithi151 ceph-mon[108966]: pgmap v1541: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 7 active+remapped+backfill_wait, 108 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.7 KiB/s rd, 37 MiB/s wr, 595 op/s; 66488/359592 objects misplaced (18.490%); 686 KiB/s, 10 objects/s recovering 2024-09-13T14:45:35.290 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:35.330 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:35.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:35.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:35.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:34 smithi184 ceph-mon[92527]: pgmap v1541: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 7 active+remapped+backfill_wait, 108 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.7 KiB/s rd, 37 MiB/s wr, 595 op/s; 66488/359592 objects misplaced (18.490%); 686 KiB/s, 10 objects/s recovering 2024-09-13T14:45:35.432 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:35.455 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:35.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:35.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:35.557 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:35.582 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:35.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:35.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:35.683 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:35.721 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:35.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:35.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:35.822 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:35.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:35.860 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:35.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:35.961 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:35.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:35.985 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:35.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:36.086 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:36.137 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:36.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:36.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:36.239 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:36.261 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:36.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:36.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:36.363 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:36.387 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:36.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:36.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:36.478 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T14:45:36.488 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:36.510 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:36.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:36.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:36.612 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:36.636 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:36.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:36.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:36.738 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:36.808 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:36.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:36.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:36.910 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:36.934 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:36.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:36.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:37.036 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:37.059 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:37.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:37.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:37.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:36 smithi151 ceph-mon[104450]: pgmap v1542: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 35 MiB/s wr, 590 op/s; 66578/360048 objects misplaced (18.491%); 652 KiB/s, 10 objects/s recovering 2024-09-13T14:45:37.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:36 smithi151 ceph-mon[104450]: 1.1e scrub starts 2024-09-13T14:45:37.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:36 smithi151 ceph-mon[104450]: 1.1e scrub ok 2024-09-13T14:45:37.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:36 smithi151 ceph-mon[108966]: pgmap v1542: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 35 MiB/s wr, 590 op/s; 66578/360048 objects misplaced (18.491%); 652 KiB/s, 10 objects/s recovering 2024-09-13T14:45:37.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:36 smithi151 ceph-mon[108966]: 1.1e scrub starts 2024-09-13T14:45:37.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:36 smithi151 ceph-mon[108966]: 1.1e scrub ok 2024-09-13T14:45:37.161 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:37.184 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:37.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:37.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:37.286 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:37.334 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:37.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:37.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:37.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:36 smithi184 ceph-mon[92527]: pgmap v1542: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 35 MiB/s wr, 590 op/s; 66578/360048 objects misplaced (18.491%); 652 KiB/s, 10 objects/s recovering 2024-09-13T14:45:37.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:36 smithi184 ceph-mon[92527]: 1.1e scrub starts 2024-09-13T14:45:37.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:36 smithi184 ceph-mon[92527]: 1.1e scrub ok 2024-09-13T14:45:37.437 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:37.478 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:37.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:37.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:37.580 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:37.617 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:37.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:37.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:37.719 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:37.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:37.758 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:37.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:37.859 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:37.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:37.884 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:37.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:37.985 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:38.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:38.023 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:38.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:38.097 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:37 smithi151 ceph-mon[104450]: 1.7 scrub starts 2024-09-13T14:45:38.097 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:37 smithi151 ceph-mon[104450]: 1.7 scrub ok 2024-09-13T14:45:38.097 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:37 smithi151 ceph-mon[108966]: 1.7 scrub starts 2024-09-13T14:45:38.098 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:37 smithi151 ceph-mon[108966]: 1.7 scrub ok 2024-09-13T14:45:38.124 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:38.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:38.150 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:38.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:38.251 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:38.290 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:45:38.290 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:45:38.308 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:38.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:38.332 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:38.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:38.340 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:45:38.340 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:45:38.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:37 smithi184 ceph-mon[92527]: 1.7 scrub starts 2024-09-13T14:45:38.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:37 smithi184 ceph-mon[92527]: 1.7 scrub ok 2024-09-13T14:45:38.378 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:45:38.378 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:45:38.411 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:45:38.411 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:45:38.434 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:38.438 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:45:38.439 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:45:38.472 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:45:38.472 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:45:38.486 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:38.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:38.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:38.500 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:45:38.500 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:45:38.534 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:45:38.534 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:45:38.588 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:38.613 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:38.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:38.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:38.715 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:38.759 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:38.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:38.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:38.861 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:38.885 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:38.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:38.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:38.916 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:38 smithi151 ceph-mon[104450]: pgmap v1543: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 102 B/s rd, 26 MiB/s wr, 572 op/s; 66200/356892 objects misplaced (18.549%); 998 KiB/s, 15 objects/s recovering 2024-09-13T14:45:38.916 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:38 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1279281237' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:38.916 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:38 smithi151 ceph-mon[108966]: pgmap v1543: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 102 B/s rd, 26 MiB/s wr, 572 op/s; 66200/356892 objects misplaced (18.549%); 998 KiB/s, 15 objects/s recovering 2024-09-13T14:45:38.916 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:38 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1279281237' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:38.987 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:39.011 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:39.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:39.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:39.112 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:39.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:39.140 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:39.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:39.242 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:39.264 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:39.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:39.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:39.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:38 smithi184 ceph-mon[92527]: pgmap v1543: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 102 B/s rd, 26 MiB/s wr, 572 op/s; 66200/356892 objects misplaced (18.549%); 998 KiB/s, 15 objects/s recovering 2024-09-13T14:45:39.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:38 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1279281237' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:39.359 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:45:39.366 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:39.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:39.475 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:39.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:39.576 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:39.599 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:39.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:39.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:39.701 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:39.727 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:39.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:39.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:39.829 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:39.872 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:39.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:39.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:39.974 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:40.012 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:40.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:40.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:40.114 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:40.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:40.156 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:40.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:40.257 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:40.280 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:40.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:40.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:40.382 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:40.405 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:40.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:40.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:40.506 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:40.542 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:40.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:40.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:40.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:45:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:45:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:45:40.644 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:40.692 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:40.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:40.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:40.794 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:40.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:40.816 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:40.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:40.917 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:40.953 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:40.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:40.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:41.055 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:41.078 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:41.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:41.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:41.180 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:41.203 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:41.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:41.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:41.285 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:40 smithi151 ceph-mon[104450]: pgmap v1544: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 16 MiB/s wr, 563 op/s; 65257/352347 objects misplaced (18.521%); 995 KiB/s, 15 objects/s recovering 2024-09-13T14:45:41.285 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:40 smithi151 ceph-mon[108966]: pgmap v1544: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 16 MiB/s wr, 563 op/s; 65257/352347 objects misplaced (18.521%); 995 KiB/s, 15 objects/s recovering 2024-09-13T14:45:41.305 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:41.352 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:41.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:41.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:41.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:40 smithi184 ceph-mon[92527]: pgmap v1544: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 16 MiB/s wr, 563 op/s; 65257/352347 objects misplaced (18.521%); 995 KiB/s, 15 objects/s recovering 2024-09-13T14:45:41.454 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:41.478 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:41.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:41.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:41.580 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:41.621 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:41.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:41.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:41.723 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:41.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:41.764 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:41.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:41.865 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:41.888 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:41.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:41.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:41.990 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:42.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:42.037 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:42.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:42.054 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:45:42.054 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:45:42.139 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:42.166 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:42.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:42.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:42.268 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:42.316 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:42.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:42.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:42.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:41 smithi184 ceph-mon[92527]: Health check failed: Degraded data redundancy: 60/350115 objects degraded (0.017%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:45:42.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:41 smithi151 ceph-mon[104450]: Health check failed: Degraded data redundancy: 60/350115 objects degraded (0.017%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:45:42.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:41 smithi151 ceph-mon[108966]: Health check failed: Degraded data redundancy: 60/350115 objects degraded (0.017%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:45:42.418 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:42.441 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:42.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:42.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:42.542 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:42.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:42.571 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:42.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:42.672 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:42.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:42.715 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:42.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:42.816 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:42.856 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:42.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:42.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:42.956 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T14:45:42.958 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:43.022 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:43.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:43.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:43.124 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:43.163 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:43.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:43.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:43.265 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:43.289 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:43.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:43.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:43.326 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:45:43.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:42 smithi184 ceph-mon[92527]: pgmap v1545: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 11 MiB/s wr, 558 op/s; 60/350115 objects degraded (0.017%); 64963/350115 objects misplaced (18.555%); 1008 KiB/s, 15 objects/s recovering 2024-09-13T14:45:43.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:42 smithi151 ceph-mon[104450]: pgmap v1545: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 11 MiB/s wr, 558 op/s; 60/350115 objects degraded (0.017%); 64963/350115 objects misplaced (18.555%); 1008 KiB/s, 15 objects/s recovering 2024-09-13T14:45:43.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:42 smithi151 ceph-mon[108966]: pgmap v1545: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 11 MiB/s wr, 558 op/s; 60/350115 objects degraded (0.017%); 64963/350115 objects misplaced (18.555%); 1008 KiB/s, 15 objects/s recovering 2024-09-13T14:45:43.391 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:43.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:43.482 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:43.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:43.582 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:43.600 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:43.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:43.632 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:43.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:43.733 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:43.757 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:43.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:43.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:43.859 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:43.882 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:43.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:43.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:43.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:44.020 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:44.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:44.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:44.122 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:44.165 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:44.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:44.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:44.266 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:44.289 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:44.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:44.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:44.391 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:44.432 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:44.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:44.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:44.533 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:44.558 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:44.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:44.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:44.660 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:44.687 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:44.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:44.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:44.789 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:44.811 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:44.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:44.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:44.846 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:45:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:45:44] "GET /metrics HTTP/1.1" 200 36669 "" "Prometheus/2.51.0" 2024-09-13T14:45:44.855 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:44.913 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:44.966 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T14:45:44.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:44.994 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:44.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:45.095 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:45.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:44 smithi151 ceph-mon[104450]: pgmap v1546: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 9.8 MiB/s wr, 606 op/s; 60/345801 objects degraded (0.017%); 64335/345801 objects misplaced (18.605%); 871 KiB/s, 13 objects/s recovering 2024-09-13T14:45:45.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:44 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3283267575' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:45:45.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:44 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:45:45.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:44 smithi151 ceph-mon[108966]: pgmap v1546: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 9.8 MiB/s wr, 606 op/s; 60/345801 objects degraded (0.017%); 64335/345801 objects misplaced (18.605%); 871 KiB/s, 13 objects/s recovering 2024-09-13T14:45:45.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:44 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3283267575' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:45:45.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:44 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:45:45.136 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:45.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:45.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:45.237 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:45.277 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:45.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:45.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:45.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:44 smithi184 ceph-mon[92527]: pgmap v1546: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 9.8 MiB/s wr, 606 op/s; 60/345801 objects degraded (0.017%); 64335/345801 objects misplaced (18.605%); 871 KiB/s, 13 objects/s recovering 2024-09-13T14:45:45.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:44 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3283267575' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:45:45.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:44 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:45:45.379 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:45.415 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:45.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:45.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:45.517 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:45.543 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:45.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:45.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:45.645 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:45.672 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:45.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:45.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:45.776 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:45.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:45.838 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:45.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:45.939 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:45.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:45.988 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:45.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:46.089 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:46.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:46.132 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:46.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:46.153 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:46 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:45:46.153 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:46 smithi151 ceph-mon[104450]: osdmap e1152: 8 total, 8 up, 8 in 2024-09-13T14:45:46.154 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:46 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:45:46.154 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:46 smithi151 ceph-mon[108966]: osdmap e1152: 8 total, 8 up, 8 in 2024-09-13T14:45:46.233 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:46.267 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:46.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:46.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:46.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:46 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:45:46.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:46 smithi184 ceph-mon[92527]: osdmap e1152: 8 total, 8 up, 8 in 2024-09-13T14:45:46.369 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:46.393 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:46.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:46.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:46.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:46.543 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:46.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:46.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:46.644 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:46.695 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:46.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:46.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:46.797 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:46.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T14:45:46.863 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:46.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:46.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:46.964 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:46.986 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:46.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:46.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:47.074 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:47 smithi151 ceph-mon[104450]: pgmap v1548: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.0 MiB/s wr, 602 op/s; 60/342945 objects degraded (0.017%); 63754/342945 objects misplaced (18.590%); 1008 KiB/s, 15 objects/s recovering 2024-09-13T14:45:47.074 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:47 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:45:47.074 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:47 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:47.075 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:47 smithi151 ceph-mon[108966]: pgmap v1548: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.0 MiB/s wr, 602 op/s; 60/342945 objects degraded (0.017%); 63754/342945 objects misplaced (18.590%); 1008 KiB/s, 15 objects/s recovering 2024-09-13T14:45:47.075 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:47 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:45:47.075 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:47 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:47.088 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:47.110 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:47.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:47.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:47.212 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:47.236 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:47.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:47.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:47.338 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:47.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:47 smithi184 ceph-mon[92527]: pgmap v1548: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.0 MiB/s wr, 602 op/s; 60/342945 objects degraded (0.017%); 63754/342945 objects misplaced (18.590%); 1008 KiB/s, 15 objects/s recovering 2024-09-13T14:45:47.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:47 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:45:47.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:47 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:47.363 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:47.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:47.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:47.465 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:47.502 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:47.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:47.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:47.590 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:47.603 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:47.626 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:47.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:47.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:47.727 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:47.776 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:47.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:47.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:47.878 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:47.920 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:47.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:47.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:48.012 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-13T14:45:48.022 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:48.064 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:48.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:48.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:48.166 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:48.209 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:48.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:48.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:48.312 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:48.357 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:48 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2520367483' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:48.357 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:48 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:48.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:48 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2520367483' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:48.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:48 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:48.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:48 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2520367483' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:48.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:48 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:48.360 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:48.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:48.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:48.462 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:48.485 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:48.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:48.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:48.587 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:48.624 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:48.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:48.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:48.726 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:48.775 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:48.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:48.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:48.877 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:48.916 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:48.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:48.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:49.018 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:49.042 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:49.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:49.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:49.144 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:49.174 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:49.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:49.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:49.276 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:49.316 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:49.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:49.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:49.351 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:49 smithi184 ceph-mon[92527]: pgmap v1549: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 626 op/s; 176/338067 objects degraded (0.052%); 63175/338067 objects misplaced (18.687%); 784 KiB/s, 12 objects/s recovering 2024-09-13T14:45:49.352 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:49 smithi184 ceph-mon[92527]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:45:49.352 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:49 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 176/338067 objects degraded (0.052%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:45:49.352 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:49 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:45:49.352 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:49 smithi184 ceph-mon[92527]: osdmap e1153: 8 total, 8 up, 8 in 2024-09-13T14:45:49.352 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:49 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:45:49.352 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:49 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:45:49.352 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:49 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:45:49.355 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:49 smithi151 ceph-mon[104450]: pgmap v1549: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 626 op/s; 176/338067 objects degraded (0.052%); 63175/338067 objects misplaced (18.687%); 784 KiB/s, 12 objects/s recovering 2024-09-13T14:45:49.355 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:49 smithi151 ceph-mon[104450]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:45:49.355 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:49 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 176/338067 objects degraded (0.052%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:45:49.356 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:49 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:45:49.356 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:49 smithi151 ceph-mon[104450]: osdmap e1153: 8 total, 8 up, 8 in 2024-09-13T14:45:49.356 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:49 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:45:49.356 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:49 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:45:49.356 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:49 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:45:49.356 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:49 smithi151 ceph-mon[108966]: pgmap v1549: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 626 op/s; 176/338067 objects degraded (0.052%); 63175/338067 objects misplaced (18.687%); 784 KiB/s, 12 objects/s recovering 2024-09-13T14:45:49.356 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:49 smithi151 ceph-mon[108966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:45:49.356 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:49 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 176/338067 objects degraded (0.052%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:45:49.356 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:49 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:45:49.356 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:49 smithi151 ceph-mon[108966]: osdmap e1153: 8 total, 8 up, 8 in 2024-09-13T14:45:49.356 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:49 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:45:49.357 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:49 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:45:49.357 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:49 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:45:49.418 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:49.443 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:49.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:49.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:49.545 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:49.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:49.591 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:49.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:49.693 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:49.734 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:49.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:49.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:49.836 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:49.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:49.896 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:49.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:49.997 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:50.038 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:50.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:50.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:50.140 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:50.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:50.193 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:50.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:50.294 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:50.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:50.335 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:50.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:50.436 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:50.460 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:50.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:50.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:50.515 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:45:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:45:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:45:50.561 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:50.602 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:50.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:50.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:50.704 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:50.745 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:50.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:50.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:50.847 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:50.898 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:50.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:50.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:51.000 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:51.023 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:51.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:51.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:51.107 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:51 smithi151 ceph-mon[104450]: pgmap v1551: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 176/333501 objects degraded (0.053%); 62210/333501 objects misplaced (18.654%); 536 KiB/s, 8 objects/s recovering 2024-09-13T14:45:51.107 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:51 smithi151 ceph-mon[108966]: pgmap v1551: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 176/333501 objects degraded (0.053%); 62210/333501 objects misplaced (18.654%); 536 KiB/s, 8 objects/s recovering 2024-09-13T14:45:51.117 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:51.124 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:51.145 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:51.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:51.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:51.155 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:45:51.155 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:45:51.247 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:51.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:51.296 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:51.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:51.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:51 smithi184 ceph-mon[92527]: pgmap v1551: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 176/333501 objects degraded (0.053%); 62210/333501 objects misplaced (18.654%); 536 KiB/s, 8 objects/s recovering 2024-09-13T14:45:51.397 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:51.449 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:51.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:51.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:51.551 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:51.605 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:51.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:51.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:51.707 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:51.734 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:51.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:51.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:51.826 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:45:51.836 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:51.858 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:51.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:51.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:51.940 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:45:51.941 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:45:51.959 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:51.986 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:45:51.987 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:45:52.020 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:45:52.021 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:45:52.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:52.024 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:52.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:52.061 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:45:52.062 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:45:52.098 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:45:52.099 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:45:52.125 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:52.140 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:45:52.140 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:45:52.183 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:45:52.183 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:45:52.188 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:52.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:52.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:52.234 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:45:52.235 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:45:52.235 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:52 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3907635332' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:52.235 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:52 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3907635332' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:52.291 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:52.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:52.317 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:52.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:52.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:52 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3907635332' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:52.419 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:52.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:52.471 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:52.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:52.573 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:52.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:52.616 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:52.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:52.717 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:52.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:52.745 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:52.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:52.847 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:52.879 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:52.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:52.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:52.981 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:53.004 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:53.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:53.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:53.106 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:53.142 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:53.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:53.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:53.244 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:53.288 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:53.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:53.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:53.324 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:53 smithi184 ceph-mon[92527]: pgmap v1552: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 610 op/s; 298/331140 objects degraded (0.090%); 61824/331140 objects misplaced (18.670%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:45:53.346 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:53 smithi151 ceph-mon[108966]: pgmap v1552: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 610 op/s; 298/331140 objects degraded (0.090%); 61824/331140 objects misplaced (18.670%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:45:53.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:53 smithi151 ceph-mon[104450]: pgmap v1552: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 610 op/s; 298/331140 objects degraded (0.090%); 61824/331140 objects misplaced (18.670%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:45:53.390 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:53.413 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:53.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:53.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:53.514 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:53.537 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:53.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:53.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:53.618 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T14:45:53.639 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:53.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:53.664 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:53.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:53.765 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:53.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:53.824 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:53.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:53.925 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:53.986 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:53.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:53.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:54.088 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:54.151 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:54.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:54.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:54.252 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:54.276 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:54.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:54.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:54.378 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:54.402 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:54.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:54.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:54.504 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:54.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:54.553 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:54.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:54.653 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:54.701 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:54.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:54.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:54.804 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:54 smithi151 ceph-mon[104450]: pgmap v1553: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 629 op/s; 298/326934 objects degraded (0.091%); 61194/326934 objects misplaced (18.718%); 973 KiB/s, 15 objects/s recovering 2024-09-13T14:45:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:54 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:45:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:54 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:45:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:54 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 298/326934 objects degraded (0.091%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:45:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:54 smithi151 ceph-mon[108966]: pgmap v1553: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 629 op/s; 298/326934 objects degraded (0.091%); 61194/326934 objects misplaced (18.718%); 973 KiB/s, 15 objects/s recovering 2024-09-13T14:45:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:54 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:45:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:54 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:45:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:54 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 298/326934 objects degraded (0.091%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:45:54.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:45:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:45:54] "GET /metrics HTTP/1.1" 200 36662 "" "Prometheus/2.51.0" 2024-09-13T14:45:54.867 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:54.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:54.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:54.941 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:54.969 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:55.025 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:55.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:55.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:55.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:54 smithi184 ceph-mon[92527]: pgmap v1553: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 629 op/s; 298/326934 objects degraded (0.091%); 61194/326934 objects misplaced (18.718%); 973 KiB/s, 15 objects/s recovering 2024-09-13T14:45:55.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:54 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:45:55.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:54 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:45:55.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:54 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 298/326934 objects degraded (0.091%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:45:55.126 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:55.148 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:55.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:55.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:55.249 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:55.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:55.306 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:55.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:55.407 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:55.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:55.453 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:55.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:55.554 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:55.578 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:55.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:55.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:55.633 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T14:45:55.680 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:55.703 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:55.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:55.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:55.805 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:55.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:55.828 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:55.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:55.929 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:55.951 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:55.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:55.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:56.052 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:56.068 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:55 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2996645872' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:45:56.080 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:56.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:56.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:56.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:55 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2996645872' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:45:56.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:55 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2996645872' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:45:56.182 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:56.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:56.225 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:56.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:56.326 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:56.363 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:56.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:56.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:56.465 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:56.506 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:56.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:56.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:56.608 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:56.646 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:56.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:56.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:56.748 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:56.772 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:56.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:56.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:56.788 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:45:56.873 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:56.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:56 smithi151 ceph-mon[104450]: pgmap v1554: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 630 op/s; 298/324021 objects degraded (0.092%); 60616/324021 objects misplaced (18.707%); 825 KiB/s, 12 objects/s recovering 2024-09-13T14:45:56.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:56 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2996645872' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:45:56.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:56 smithi151 ceph-mon[104450]: osdmap e1154: 8 total, 8 up, 8 in 2024-09-13T14:45:56.944 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:56 smithi151 ceph-mon[108966]: pgmap v1554: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 630 op/s; 298/324021 objects degraded (0.092%); 60616/324021 objects misplaced (18.707%); 825 KiB/s, 12 objects/s recovering 2024-09-13T14:45:56.944 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:56 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2996645872' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:45:56.944 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:56 smithi151 ceph-mon[108966]: osdmap e1154: 8 total, 8 up, 8 in 2024-09-13T14:45:56.946 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:56.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:56.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:57.048 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:57.071 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:57.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:57.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:57.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:56 smithi184 ceph-mon[92527]: pgmap v1554: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 630 op/s; 298/324021 objects degraded (0.092%); 60616/324021 objects misplaced (18.707%); 825 KiB/s, 12 objects/s recovering 2024-09-13T14:45:57.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:56 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2996645872' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:45:57.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:56 smithi184 ceph-mon[92527]: osdmap e1154: 8 total, 8 up, 8 in 2024-09-13T14:45:57.173 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:57.195 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:57.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:57.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:57.296 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:57.340 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:57.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:57.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:57.397 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T14:45:57.441 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:57.465 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:57.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:57.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:57.567 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:57.590 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:57.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:57.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:57.691 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:57.735 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:57.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:57.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:57.836 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:57.860 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:57.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:57.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:57.962 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:57.987 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:57.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:57.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:58.089 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:58.113 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:58.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:58.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:58.215 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:58.238 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:58.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:58.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:58.340 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:58.363 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:58.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:58.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:58.465 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:58.487 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:58.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:58.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:58.589 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:58.618 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:58.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:58.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:58.636 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:58.720 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:58.742 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:58.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:58.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:58.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:58.886 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:58.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:58.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:58.967 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:58 smithi151 ceph-mon[104450]: pgmap v1556: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 595 op/s; 432/319440 objects degraded (0.135%); 60043/319440 objects misplaced (18.796%); 814 KiB/s, 12 objects/s recovering 2024-09-13T14:45:58.967 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:58 smithi151 ceph-mon[108966]: pgmap v1556: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 595 op/s; 432/319440 objects degraded (0.135%); 60043/319440 objects misplaced (18.796%); 814 KiB/s, 12 objects/s recovering 2024-09-13T14:45:58.991 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:59.013 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:59.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:59.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:59.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:58 smithi184 ceph-mon[92527]: pgmap v1556: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 595 op/s; 432/319440 objects degraded (0.135%); 60043/319440 objects misplaced (18.796%); 814 KiB/s, 12 objects/s recovering 2024-09-13T14:45:59.115 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:59.142 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:59.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:59.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:59.244 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:59.306 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:59.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:59.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:59.408 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:59.433 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:59.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:59.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:59.534 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:59.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:59.560 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:45:59.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:59.661 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:59.706 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:59.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:59.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:59.807 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:59.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:59.855 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:45:59.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:59.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:45:59 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 432/314727 objects degraded (0.137%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:45:59.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:45:59 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 432/314727 objects degraded (0.137%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:45:59.956 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:00.003 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:00.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:00.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:00.092 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:00.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:45:59 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 432/314727 objects degraded (0.137%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:46:00.105 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:00.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:00.161 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:00.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:00.262 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:00.287 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:00.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:00.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:00.388 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:00.412 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:00.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:00.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:00.513 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:00.539 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:00.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:00.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:00.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:46:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:46:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:46:00.641 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:00.670 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:00.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:00.702 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:00.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:00.705 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-13T14:46:00.802 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:00.862 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:00.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:00.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:00.932 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:00 smithi151 ceph-mon[104450]: pgmap v1557: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 625 op/s; 432/314727 objects degraded (0.137%); 59052/314727 objects misplaced (18.763%); 768 KiB/s, 11 objects/s recovering 2024-09-13T14:46:00.932 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:00 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3028458260' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:00.932 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:00 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:00.932 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:00 smithi151 ceph-mon[108966]: pgmap v1557: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 625 op/s; 432/314727 objects degraded (0.137%); 59052/314727 objects misplaced (18.763%); 768 KiB/s, 11 objects/s recovering 2024-09-13T14:46:00.932 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:00 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3028458260' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:00.932 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:00 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:00.965 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:01.026 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:01.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:01.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:01.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:00 smithi184 ceph-mon[92527]: pgmap v1557: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 625 op/s; 432/314727 objects degraded (0.137%); 59052/314727 objects misplaced (18.763%); 768 KiB/s, 11 objects/s recovering 2024-09-13T14:46:01.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:00 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3028458260' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:01.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:00 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:01.129 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:01.172 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:01.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:01.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:01.274 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:01.302 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:01.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:01.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:01.404 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:01.450 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:01.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:01.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:01.552 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:01.594 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:01.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:01.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:01.695 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:01.736 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:01.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:01.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:01.838 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:01.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:01.883 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:01.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:01.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:02.017 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:02.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:02.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:02.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:01 smithi184 ceph-mon[92527]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:02.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:01 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:46:02.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:01 smithi184 ceph-mon[92527]: osdmap e1155: 8 total, 8 up, 8 in 2024-09-13T14:46:02.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:01 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:02.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:01 smithi151 ceph-mon[104450]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:02.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:01 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:46:02.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:01 smithi151 ceph-mon[104450]: osdmap e1155: 8 total, 8 up, 8 in 2024-09-13T14:46:02.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:02.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:01 smithi151 ceph-mon[108966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:02.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:01 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:46:02.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:01 smithi151 ceph-mon[108966]: osdmap e1155: 8 total, 8 up, 8 in 2024-09-13T14:46:02.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:01 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:02.119 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:02.142 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:02.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:02.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:02.243 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:02.265 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:02.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:02.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:02.367 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:02.390 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:02.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:02.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:02.491 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:02.512 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:02.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:02.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:02.614 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:02.638 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:02.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:02.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:02.740 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:02.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:02.785 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:02.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:02.886 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:02.910 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:02.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:02.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:03.012 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:03.033 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:03.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:03.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:03.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:02 smithi184 ceph-mon[92527]: pgmap v1559: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 610 op/s; 577/312285 objects degraded (0.185%); 58620/312285 objects misplaced (18.771%); 920 KiB/s, 14 objects/s recovering 2024-09-13T14:46:03.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:02 smithi151 ceph-mon[104450]: pgmap v1559: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 610 op/s; 577/312285 objects degraded (0.185%); 58620/312285 objects misplaced (18.771%); 920 KiB/s, 14 objects/s recovering 2024-09-13T14:46:03.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:02 smithi151 ceph-mon[108966]: pgmap v1559: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 610 op/s; 577/312285 objects degraded (0.185%); 58620/312285 objects misplaced (18.771%); 920 KiB/s, 14 objects/s recovering 2024-09-13T14:46:03.136 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:03.160 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:03.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:03.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:03.262 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:03.286 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:03.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:03.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:03.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:03.427 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:03.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:03.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:03.529 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:03.574 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:03.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:03.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:03.675 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:03.700 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:03.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:03.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:03.802 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:03.823 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:03.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:03.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:03.924 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:03.962 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:03.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:03.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:04.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:04.086 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:04.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:04.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:04.188 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:04.212 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:04.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:04.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:04.314 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:04.356 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:04.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:04.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:04.458 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:04.481 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:04.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:04.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:04.583 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:04.620 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:04.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:04.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:04.722 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:04.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:04.770 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:04.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:04.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:04 smithi151 ceph-mon[104450]: pgmap v1560: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 656 op/s; 577/308271 objects degraded (0.187%); 58036/308271 objects misplaced (18.826%); 920 KiB/s, 14 objects/s recovering 2024-09-13T14:46:04.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:46:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:46:04] "GET /metrics HTTP/1.1" 200 36662 "" "Prometheus/2.51.0" 2024-09-13T14:46:04.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:04 smithi151 ceph-mon[108966]: pgmap v1560: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 656 op/s; 577/308271 objects degraded (0.187%); 58036/308271 objects misplaced (18.826%); 920 KiB/s, 14 objects/s recovering 2024-09-13T14:46:04.871 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:04.910 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:04.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:04.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:05.012 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:05.035 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:05.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:05.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:05.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:04 smithi184 ceph-mon[92527]: pgmap v1560: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 656 op/s; 577/308271 objects degraded (0.187%); 58036/308271 objects misplaced (18.826%); 920 KiB/s, 14 objects/s recovering 2024-09-13T14:46:05.137 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:05.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:05.179 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:05.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:05.281 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:05.303 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:05.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:05.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:05.405 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:05.425 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:05.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:05.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:05.527 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:05.570 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:05.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:05.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:05.672 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:05.720 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:05.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:05.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:05.822 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:05.863 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:05.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:05.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:05.965 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:05.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:05.991 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:05.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:06.092 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:06.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:05 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 577/308271 objects degraded (0.187%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:46:06.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:05 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 577/308271 objects degraded (0.187%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:46:06.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:05 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 577/308271 objects degraded (0.187%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:46:06.115 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:06.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:06.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:06.217 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:06.244 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:06.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:06.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:06.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:06.393 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:06.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:06.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:06.495 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:06.519 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:06.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:06.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:06.621 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:06.648 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:06.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:06.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:06.749 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:06.790 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:06.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:06.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:06.891 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:06.913 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:06.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:06.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:07.015 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:07.039 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:07.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:07.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:07.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:06 smithi184 ceph-mon[92527]: pgmap v1561: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 631 op/s; 577/305418 objects degraded (0.189%); 57466/305418 objects misplaced (18.816%); 750 KiB/s, 11 objects/s recovering 2024-09-13T14:46:07.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:06 smithi151 ceph-mon[104450]: pgmap v1561: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 631 op/s; 577/305418 objects degraded (0.189%); 57466/305418 objects misplaced (18.816%); 750 KiB/s, 11 objects/s recovering 2024-09-13T14:46:07.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:06 smithi151 ceph-mon[108966]: pgmap v1561: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 631 op/s; 577/305418 objects degraded (0.189%); 57466/305418 objects misplaced (18.816%); 750 KiB/s, 11 objects/s recovering 2024-09-13T14:46:07.141 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:07.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:07.169 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:07.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:07.271 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:07.294 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:07.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:07.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:07.395 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:07.422 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:07.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:07.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:07.524 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:07.560 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:07.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:07.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:07.662 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:07.687 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:07.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:07.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:07.789 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:07.817 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:07.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:07.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:07.918 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:07.958 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:07.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:07.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:08.060 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:08.081 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:08.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:08.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:08.183 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:08.208 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:08.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:08.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:08.310 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:08.334 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:08.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:08.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:08.436 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:08.462 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:08.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:08.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:08.564 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:08.587 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:08.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:08.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:08.689 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:08.711 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:08.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:08.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:08.813 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:08.851 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:08.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:08.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:08.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:08.997 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:08.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:08.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:09.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:09.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:08 smithi184 ceph-mon[92527]: pgmap v1562: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 542 op/s; 699/303153 objects degraded (0.231%); 57104/303153 objects misplaced (18.837%); 723 KiB/s, 11 objects/s recovering 2024-09-13T14:46:09.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:08 smithi151 ceph-mon[104450]: pgmap v1562: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 542 op/s; 699/303153 objects degraded (0.231%); 57104/303153 objects misplaced (18.837%); 723 KiB/s, 11 objects/s recovering 2024-09-13T14:46:09.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:08 smithi151 ceph-mon[108966]: pgmap v1562: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 542 op/s; 699/303153 objects degraded (0.231%); 57104/303153 objects misplaced (18.837%); 723 KiB/s, 11 objects/s recovering 2024-09-13T14:46:09.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:09.145 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:09.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:09.246 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:09.270 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:09.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:09.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:09.372 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:09.411 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:09.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:09.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:09.512 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:09.536 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:09.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:09.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:09.638 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:09.685 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:09.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:09.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:09.787 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:09.809 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:09.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:09.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:09.911 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:09.934 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:09.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:09.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:10.036 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:10.058 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:10.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:10.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:10.160 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:10.182 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:10.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:10.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:10.283 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:10.327 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:10.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:10.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:10.429 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:10.468 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:10.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:10.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:10.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:10.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:46:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:46:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:46:10.604 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:10.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:10.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:10.708 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:10.760 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:10.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:10.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:10.864 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:10.886 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:10.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:10.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:10.988 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:11.013 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:11.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:11.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:11.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:10 smithi184 ceph-mon[92527]: pgmap v1563: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 597 op/s; 699/296790 objects degraded (0.236%); 56073/296790 objects misplaced (18.893%); 723 KiB/s, 11 objects/s recovering 2024-09-13T14:46:11.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:10 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 699/296790 objects degraded (0.236%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:46:11.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:10 smithi151 ceph-mon[104450]: pgmap v1563: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 597 op/s; 699/296790 objects degraded (0.236%); 56073/296790 objects misplaced (18.893%); 723 KiB/s, 11 objects/s recovering 2024-09-13T14:46:11.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:10 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 699/296790 objects degraded (0.236%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:46:11.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:10 smithi151 ceph-mon[108966]: pgmap v1563: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 597 op/s; 699/296790 objects degraded (0.236%); 56073/296790 objects misplaced (18.893%); 723 KiB/s, 11 objects/s recovering 2024-09-13T14:46:11.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:10 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 699/296790 objects degraded (0.236%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:46:11.115 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:11.160 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:11.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:11.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:11.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:11.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:11.302 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:11.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:11.403 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:11.427 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:11.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:11.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:11.529 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:11.536 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:46:11.536 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-13T14:46:11.536 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-13T14:46:11.536 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-13T14:46:11.536 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:46:11.569 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:11.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:11.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:11.671 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:11.693 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:11.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:11.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:11.795 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:11.819 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:11.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:11.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:11.920 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:11.944 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:11.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:11.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:12.046 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:12.069 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:46:12.083 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:12.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:12.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:12.185 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:12.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:12.208 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:12.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:12.309 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:12.332 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:12.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:12.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:12.434 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:12.498 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:12.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:12.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:12.579 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T14:46:12.603 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:12.628 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:12.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:12.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:12.730 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:12.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:12.754 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:12.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:12.855 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:12.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:12.904 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:12.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:13.005 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:13.038 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:12 smithi151 ceph-mon[104450]: pgmap v1564: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 630 op/s; 845/294399 objects degraded (0.287%); 55691/294399 objects misplaced (18.917%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:46:13.038 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:12 smithi151 ceph-mon[108966]: pgmap v1564: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 630 op/s; 845/294399 objects degraded (0.287%); 55691/294399 objects misplaced (18.917%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:46:13.061 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:13.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:13.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:13.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:12 smithi184 ceph-mon[92527]: pgmap v1564: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 630 op/s; 845/294399 objects degraded (0.287%); 55691/294399 objects misplaced (18.917%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:46:13.163 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:13.201 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:13.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:13.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:13.303 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:13.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:13.353 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:13.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:13.454 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:13.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:13.504 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:13.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:13.605 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:13.627 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:13.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:13.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:13.729 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:13.777 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:13.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:13.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:13.879 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:13.903 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:13.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:13.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:14.005 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:14.054 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:14.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:14.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:14.156 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:14.200 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:14.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:14.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:14.302 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:14.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:14.346 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:14.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:14.447 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:14.491 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:14.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:14.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:14.593 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:14.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:14.642 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:14.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:14.743 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:14.765 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:14.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:14.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:14.786 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:46:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:46:14] "GET /metrics HTTP/1.1" 200 36662 "" "Prometheus/2.51.0" 2024-09-13T14:46:14.868 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:14.892 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:14.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:14.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:14.994 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:15.019 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:15.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:15.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:14 smithi184 ceph-mon[92527]: pgmap v1565: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 611 op/s; 845/290265 objects degraded (0.291%); 55128/290265 objects misplaced (18.992%); 624 KiB/s, 9 objects/s recovering 2024-09-13T14:46:15.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:14 smithi151 ceph-mon[104450]: pgmap v1565: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 611 op/s; 845/290265 objects degraded (0.291%); 55128/290265 objects misplaced (18.992%); 624 KiB/s, 9 objects/s recovering 2024-09-13T14:46:15.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:14 smithi151 ceph-mon[108966]: pgmap v1565: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 611 op/s; 845/290265 objects degraded (0.291%); 55128/290265 objects misplaced (18.992%); 624 KiB/s, 9 objects/s recovering 2024-09-13T14:46:15.121 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:15.145 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:15.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:15.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:15.247 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:15.271 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:15.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:15.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:15.373 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:15.397 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:15.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:15.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:15.499 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:15.521 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:15.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:15.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:15.623 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:15.646 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:15.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:15.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:15.748 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:15.786 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:15.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:15.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:15.888 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:15.913 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:15.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:15.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:16.016 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:16.021 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:15 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 845/290265 objects degraded (0.291%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:46:16.022 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:15 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 845/290265 objects degraded (0.291%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:46:16.042 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:16.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:16.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:16.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:15 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 845/290265 objects degraded (0.291%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:46:16.144 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:16.183 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:16.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:16.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:16.285 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:16.310 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:16.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:16.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:16.412 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:16.437 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:16.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:16.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:16.539 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:16.589 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:16.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:16.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:16.691 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:16.696 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:46:16.696 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:46:16.715 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:16.727 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:46:16.728 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:46:16.751 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:16.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:16.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:16.765 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:46:16.765 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:46:16.830 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:46:16.830 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:46:16.853 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:16.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:16 smithi184 ceph-mon[92527]: pgmap v1566: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 581 op/s; 845/287313 objects degraded (0.294%); 54539/287313 objects misplaced (18.982%); 624 KiB/s, 9 objects/s recovering 2024-09-13T14:46:16.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:16.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:16 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3172082082' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:16.862 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:46:16.862 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:46:16.893 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:46:16.893 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:46:16.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:16.911 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:16.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:16.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:16 smithi151 ceph-mon[104450]: pgmap v1566: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 581 op/s; 845/287313 objects degraded (0.294%); 54539/287313 objects misplaced (18.982%); 624 KiB/s, 9 objects/s recovering 2024-09-13T14:46:16.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:16.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:16 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3172082082' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:16.939 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:46:16.939 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:46:16.939 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:16 smithi151 ceph-mon[108966]: pgmap v1566: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 581 op/s; 845/287313 objects degraded (0.294%); 54539/287313 objects misplaced (18.982%); 624 KiB/s, 9 objects/s recovering 2024-09-13T14:46:16.940 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:16.940 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:16 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3172082082' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:16.971 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:46:16.971 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:46:17.012 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:17.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:17.059 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:17.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:17.160 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:17.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:17.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:17.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:17.311 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:17.367 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:17.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:17.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:17.469 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:17.500 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:17.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:17.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:17.602 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:17.650 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:17.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:17.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:17.752 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:17.790 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:17.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:17.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:17.833 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T14:46:17.892 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:17.917 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:17.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:17.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:18.018 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:18.043 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:18.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:18.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:18.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:17 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1470193135' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:46:18.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:17 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:46:18.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:17 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1470193135' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:46:18.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:17 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:46:18.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:17 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1470193135' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:46:18.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:17 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:46:18.145 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:18.169 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:18.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:18.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:18.271 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:18.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:18.296 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:18.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:18.397 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:18.443 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:18.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:18.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:18.545 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:18.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:18.583 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:18.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:18.684 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:18.729 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:18.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:18.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:18.815 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T14:46:18.831 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:18.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:18.905 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:18.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:19.006 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:19.029 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:19.029 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:18 smithi151 ceph-mon[104450]: pgmap v1567: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 568 op/s; 986/284943 objects degraded (0.346%); 54155/284943 objects misplaced (19.006%); 954 KiB/s, 14 objects/s recovering 2024-09-13T14:46:19.029 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:18 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T14:46:19.029 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:18 smithi151 ceph-mon[104450]: osdmap e1156: 8 total, 8 up, 8 in 2024-09-13T14:46:19.030 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:18 smithi151 ceph-mon[108966]: pgmap v1567: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 568 op/s; 986/284943 objects degraded (0.346%); 54155/284943 objects misplaced (19.006%); 954 KiB/s, 14 objects/s recovering 2024-09-13T14:46:19.030 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:18 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T14:46:19.030 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:18 smithi151 ceph-mon[108966]: osdmap e1156: 8 total, 8 up, 8 in 2024-09-13T14:46:19.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:19.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:19.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:18 smithi184 ceph-mon[92527]: pgmap v1567: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 568 op/s; 986/284943 objects degraded (0.346%); 54155/284943 objects misplaced (19.006%); 954 KiB/s, 14 objects/s recovering 2024-09-13T14:46:19.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:18 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T14:46:19.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:18 smithi184 ceph-mon[92527]: osdmap e1156: 8 total, 8 up, 8 in 2024-09-13T14:46:19.131 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:19.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:19.190 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:19.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:19.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:19.346 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:19.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:19.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:19.366 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:19.449 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:19.497 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:19.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:19.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:19.599 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:19.623 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:19.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:19.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:19.725 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:19.747 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:19.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:19.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:19.850 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:19.873 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:19.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:19.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:19.976 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:20.000 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:20.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:20.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:20.102 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:20.117 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:46:20.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:20.139 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:20.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:20.240 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:20.301 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:20.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:20.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:20.403 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:20.465 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:20.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:20.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:20.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:20.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:20 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:46:20.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:20 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:46:20.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:20 smithi184 ceph-mon[92527]: pgmap v1569: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 630 op/s; 986/277866 objects degraded (0.355%); 53017/277866 objects misplaced (19.080%); 787 KiB/s, 12 objects/s recovering 2024-09-13T14:46:20.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:20 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 986/284943 objects degraded (0.346%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:46:20.605 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:46:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:46:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:46:20.611 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:20 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:46:20.611 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:20 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:46:20.611 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:20 smithi151 ceph-mon[104450]: pgmap v1569: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 630 op/s; 986/277866 objects degraded (0.355%); 53017/277866 objects misplaced (19.080%); 787 KiB/s, 12 objects/s recovering 2024-09-13T14:46:20.611 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:20 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 986/284943 objects degraded (0.346%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:46:20.611 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:20 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:46:20.611 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:20 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:46:20.611 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:20 smithi151 ceph-mon[108966]: pgmap v1569: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 630 op/s; 986/277866 objects degraded (0.355%); 53017/277866 objects misplaced (19.080%); 787 KiB/s, 12 objects/s recovering 2024-09-13T14:46:20.611 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:20 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 986/284943 objects degraded (0.346%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:46:20.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:20.636 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:20.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:20.737 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:20.761 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:20.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:20.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:20.864 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:20.911 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:20.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:20.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:21.012 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:21.036 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:21.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:21.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:21.138 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:21.161 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:21.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:21.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:21.263 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:21.285 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:21.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:21.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:21.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:21 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:46:21.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:21 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:46:21.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:21 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:46:21.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:21 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:46:21.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:21 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:46:21.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:21 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:46:21.387 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:21.427 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:21.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:21.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:21.529 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:21.563 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:21.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:21.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:21.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:21 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:46:21.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:21 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:46:21.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:21 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:46:21.665 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:21.689 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:21.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:21.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:21.757 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:21.791 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:21.814 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:21.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:21.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:21.916 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:21.971 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:21.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:21.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:22.073 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:22.095 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:22.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:22.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:22.197 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:22.220 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:22.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:22.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:22.321 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:22.344 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:22.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:22.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:22.447 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:22.469 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:22.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:22.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:22.571 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:22.572 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:22 smithi151 ceph-mon[104450]: pgmap v1570: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 628 op/s; 1105/275541 objects degraded (0.401%); 52635/275541 objects misplaced (19.102%); 800 KiB/s, 12 objects/s recovering 2024-09-13T14:46:22.573 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:22 smithi151 ceph-mon[108966]: pgmap v1570: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 628 op/s; 1105/275541 objects degraded (0.401%); 52635/275541 objects misplaced (19.102%); 800 KiB/s, 12 objects/s recovering 2024-09-13T14:46:22.598 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:22.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:22 smithi184 ceph-mon[92527]: pgmap v1570: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 628 op/s; 1105/275541 objects degraded (0.401%); 52635/275541 objects misplaced (19.102%); 800 KiB/s, 12 objects/s recovering 2024-09-13T14:46:22.624 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:22.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:22.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:22.729 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:22.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:22.778 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:22.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:22.879 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:22.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:22.922 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:22.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:23.023 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:23.059 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:23.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:23.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:23.160 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:23.209 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:23.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:23.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:23.311 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:23.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:23.360 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:23.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:23.461 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:23.504 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:23.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:23.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:23.606 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:23.629 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:23.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:23.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:23.731 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:23.756 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:23.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:23.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:23.857 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:23.915 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:23.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:23.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:24.017 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:24.068 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:24.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:24.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:24.170 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:24.222 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:24.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:24.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:24.324 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:24.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:24.366 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:24.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:24.459 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:46:24.459 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:46:24.468 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:24.515 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:24.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:24.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:24.577 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:24 smithi151 ceph-mon[108966]: pgmap v1571: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 629 op/s; 1105/271365 objects degraded (0.407%); 52028/271365 objects misplaced (19.173%); 800 KiB/s, 12 objects/s recovering 2024-09-13T14:46:24.577 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:24 smithi151 ceph-mon[104450]: pgmap v1571: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 629 op/s; 1105/271365 objects degraded (0.407%); 52028/271365 objects misplaced (19.173%); 800 KiB/s, 12 objects/s recovering 2024-09-13T14:46:24.618 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:24.641 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:24.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:24.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:24.743 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:24.765 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:24.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:24.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:24.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:24 smithi184 ceph-mon[92527]: pgmap v1571: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 629 op/s; 1105/271365 objects degraded (0.407%); 52028/271365 objects misplaced (19.173%); 800 KiB/s, 12 objects/s recovering 2024-09-13T14:46:24.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:46:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:46:24] "GET /metrics HTTP/1.1" 200 36664 "" "Prometheus/2.51.0" 2024-09-13T14:46:24.868 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:24.892 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:24.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:24.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:24.994 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:25.038 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:25.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:25.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:25.140 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:25.165 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:25.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:25.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:25.267 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:25.312 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:25.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:25.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:25.414 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:25.440 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:25.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:25.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:25.542 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:25.566 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:25.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:25.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:25.593 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T14:46:25.667 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:25.692 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:25.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:25.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:25.794 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:25.817 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:25.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:25.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:25.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:25 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 1105/271365 objects degraded (0.407%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:46:25.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:25 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 1105/271365 objects degraded (0.407%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:46:25.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:25 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 1105/271365 objects degraded (0.407%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:46:25.919 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:25.942 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:25.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:25.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:26.044 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:26.066 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:26.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:26.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:26.168 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:26.192 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:26.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:26.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:26.294 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:26.319 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:26.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:26.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:26.421 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:26.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:26.463 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:26.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:26.564 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:26.588 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:26.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:26.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:26.690 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:26.713 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:26.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:26.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:26.745 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:26 smithi151 ceph-mon[104450]: pgmap v1572: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 630 op/s; 1105/268392 objects degraded (0.412%); 51421/268392 objects misplaced (19.159%); 800 KiB/s, 12 objects/s recovering 2024-09-13T14:46:26.745 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:26 smithi151 ceph-mon[108966]: pgmap v1572: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 630 op/s; 1105/268392 objects degraded (0.412%); 51421/268392 objects misplaced (19.159%); 800 KiB/s, 12 objects/s recovering 2024-09-13T14:46:26.815 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:26.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:26.837 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:26.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:26.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:26 smithi184 ceph-mon[92527]: pgmap v1572: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 630 op/s; 1105/268392 objects degraded (0.412%); 51421/268392 objects misplaced (19.159%); 800 KiB/s, 12 objects/s recovering 2024-09-13T14:46:26.939 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:26.986 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:26.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:26.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:27.088 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:27.110 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:27.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:27.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:27.212 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:27.259 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:27.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:27.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:27.361 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:27.399 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:27.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:27.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:27.501 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:27.511 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:27.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:27.544 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:27.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:27.553 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-13T14:46:27.562 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T14:46:27.636 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:46:27.636 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:46:27.637 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:27 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/622362798' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:46:27.637 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:27 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:46:27.637 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:27 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/425028417' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:27.637 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:27 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/622362798' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:46:27.637 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:27 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:46:27.637 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:27 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/425028417' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:27.645 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:27.669 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:27.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:27.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:27.675 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:46:27.676 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:46:27.771 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:27.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:27.794 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:27.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:27.847 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:46:27.848 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:46:27.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:27 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/622362798' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:46:27.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:27 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:46:27.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:27 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/425028417' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:27.889 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:46:27.889 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:46:27.896 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:27.920 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:27.921 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:46:27.921 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:46:27.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:27.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:27.955 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:46:27.955 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:46:27.984 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:46:27.984 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:46:28.013 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:46:28.014 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:46:28.023 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:28.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:28.066 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:28.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:28.168 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:28.191 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:28.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:28.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:28.293 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:28.349 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:28.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:28.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:28.451 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:28.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:28.484 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:28.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:28.585 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:28.626 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:28.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:28.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:28.728 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:28.752 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:28.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:28.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:28.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:28 smithi184 ceph-mon[92527]: pgmap v1573: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 630 op/s; 1234/266007 objects degraded (0.464%); 51020/266007 objects misplaced (19.180%); 818 KiB/s, 12 objects/s recovering 2024-09-13T14:46:28.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:28 smithi184 ceph-mon[92527]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:28.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:28 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2400608029' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:28.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:28 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T14:46:28.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:28 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/425028417' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:46:28.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:28 smithi184 ceph-mon[92527]: osdmap e1157: 8 total, 8 up, 8 in 2024-09-13T14:46:28.854 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:28.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:28 smithi151 ceph-mon[104450]: pgmap v1573: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 630 op/s; 1234/266007 objects degraded (0.464%); 51020/266007 objects misplaced (19.180%); 818 KiB/s, 12 objects/s recovering 2024-09-13T14:46:28.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:28 smithi151 ceph-mon[104450]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:28.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:28 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2400608029' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:28.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:28 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T14:46:28.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:28 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/425028417' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:46:28.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:28 smithi151 ceph-mon[104450]: osdmap e1157: 8 total, 8 up, 8 in 2024-09-13T14:46:28.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:28 smithi151 ceph-mon[108966]: pgmap v1573: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 630 op/s; 1234/266007 objects degraded (0.464%); 51020/266007 objects misplaced (19.180%); 818 KiB/s, 12 objects/s recovering 2024-09-13T14:46:28.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:28 smithi151 ceph-mon[108966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:28.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:28 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2400608029' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:28.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:28 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T14:46:28.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:28 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/425028417' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:46:28.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:28 smithi151 ceph-mon[108966]: osdmap e1157: 8 total, 8 up, 8 in 2024-09-13T14:46:28.877 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:28.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:28.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:28.979 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:29.004 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:29.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:29.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:29.106 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:29.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:29.147 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:29.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:29.248 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:29.283 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:29.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:29.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:29.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:29.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:29.427 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:29.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:29.528 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:29.550 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:29.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:29.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:29.652 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:29.698 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:29.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:29.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:29.800 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:29.844 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:29.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:29.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:29.946 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:29.981 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:29.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:29.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:30.083 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:30.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:30.121 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:30.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:30.222 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:30.245 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:30.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:30.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:30.347 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:30.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:30.389 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:30.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:30.481 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T14:46:30.481 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:46:30.491 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:30.542 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:46:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:46:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:46:30.567 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:30.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:30.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:30.669 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:30.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:30.692 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:30.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:30.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:30 smithi151 ceph-mon[104450]: pgmap v1575: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 622 op/s; 1234/259188 objects degraded (0.476%); 49898/259188 objects misplaced (19.252%); 818 KiB/s, 12 objects/s recovering 2024-09-13T14:46:30.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:30 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 1234/259188 objects degraded (0.476%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:46:30.778 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:30 smithi151 ceph-mon[108966]: pgmap v1575: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 622 op/s; 1234/259188 objects degraded (0.476%); 49898/259188 objects misplaced (19.252%); 818 KiB/s, 12 objects/s recovering 2024-09-13T14:46:30.778 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:30 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 1234/259188 objects degraded (0.476%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:46:30.793 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:30.816 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:30.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:30.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:30.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:30 smithi184 ceph-mon[92527]: pgmap v1575: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 622 op/s; 1234/259188 objects degraded (0.476%); 49898/259188 objects misplaced (19.252%); 818 KiB/s, 12 objects/s recovering 2024-09-13T14:46:30.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:30 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 1234/259188 objects degraded (0.476%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:46:30.918 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:30.964 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:30.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:30.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:31.066 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:31.108 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:31.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:31.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:31.211 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:31.249 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:31.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:31.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:31.351 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:31.377 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:31.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:31.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:31.479 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:31.518 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:31.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:31.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:31.620 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:31.657 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:31.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:31.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:31.760 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:31.797 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:31.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:31.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:31.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:31.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:31.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:31.900 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:31.923 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:31.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:31.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:32.025 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:32.049 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:32.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:32.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:32.150 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:32.194 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:32.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:32.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:32.299 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:32.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:32.325 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:32.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:32.425 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:32.450 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:32.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:32.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:32.552 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:32.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:32.577 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:32.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:32.678 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:32.702 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:32.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:32.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:32.804 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:32.831 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:32.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:32.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:32.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:32 smithi184 ceph-mon[92527]: pgmap v1576: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 617 op/s; 1329/257001 objects degraded (0.517%); 49558/257001 objects misplaced (19.283%); 870 KiB/s, 13 objects/s recovering 2024-09-13T14:46:32.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:32 smithi151 ceph-mon[104450]: pgmap v1576: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 617 op/s; 1329/257001 objects degraded (0.517%); 49558/257001 objects misplaced (19.283%); 870 KiB/s, 13 objects/s recovering 2024-09-13T14:46:32.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:32 smithi151 ceph-mon[108966]: pgmap v1576: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 617 op/s; 1329/257001 objects degraded (0.517%); 49558/257001 objects misplaced (19.283%); 870 KiB/s, 13 objects/s recovering 2024-09-13T14:46:32.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:32.974 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:32.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:32.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:33.076 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:33.101 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:33.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:33.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:33.154 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:46:33.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:33.271 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:33.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:33.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:33.376 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:33.399 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:33.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:33.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:33.501 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:33.525 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:33.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:33.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:33.626 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:33.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:33.649 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:33.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:33.750 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:33.787 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:33.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:33.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:33.889 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:33.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:33.942 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:33.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:34.043 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:34.067 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:34.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:34.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:34.168 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:34.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:34.243 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:34.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:34.249 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:34.344 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:34.392 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:34.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:34.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:34.412 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:46:34.494 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:34.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:34.531 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:34.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:34.591 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:34 smithi151 ceph-mon[104450]: pgmap v1577: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 619 op/s; 1329/252762 objects degraded (0.526%); 48979/252762 objects misplaced (19.378%); 870 KiB/s, 13 objects/s recovering 2024-09-13T14:46:34.592 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:34 smithi151 ceph-mon[108966]: pgmap v1577: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 619 op/s; 1329/252762 objects degraded (0.526%); 48979/252762 objects misplaced (19.378%); 870 KiB/s, 13 objects/s recovering 2024-09-13T14:46:34.632 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:34.703 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:34.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:34.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:34.806 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:34.829 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:34.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:34.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:34.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:34 smithi184 ceph-mon[92527]: pgmap v1577: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 619 op/s; 1329/252762 objects degraded (0.526%); 48979/252762 objects misplaced (19.378%); 870 KiB/s, 13 objects/s recovering 2024-09-13T14:46:34.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:46:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:46:34] "GET /metrics HTTP/1.1" 200 36667 "" "Prometheus/2.51.0" 2024-09-13T14:46:34.931 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:34.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:34.977 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:34.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:35.078 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:35.100 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:35.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:35.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:35.127 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:46:35.127 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-13T14:46:35.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.127 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(7.3 1157/1149 [PushReplyOp(7:ca756b87:::benchmark_data_smithi151_190906_object38524:head)])", 2024-09-13T14:46:35.127 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:46:35.098573+0000", 2024-09-13T14:46:35.128 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.025347367999999999, 2024-09-13T14:46:35.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.025372485, 2024-09-13T14:46:35.128 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:46:35.128 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:46:35.128 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-13T14:46:35.128 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:46:35.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:46:35.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.098573+0000", 2024-09-13T14:46:35.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:46:35.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:46:35.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.098573+0000", 2024-09-13T14:46:35.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:46:35.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:46:35.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.098573+0000", 2024-09-13T14:46:35.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1199999999999998e-07 2024-09-13T14:46:35.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:46:35.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.098577+0000", 2024-09-13T14:46:35.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0369999999999996e-06 2024-09-13T14:46:35.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:46:35.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.098594+0000", 2024-09-13T14:46:35.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6378000000000001e-05 2024-09-13T14:46:35.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:46:35.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.098600+0000", 2024-09-13T14:46:35.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4239999999999998e-06 2024-09-13T14:46:35.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:46:35.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.115587+0000", 2024-09-13T14:46:35.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016987108000000001 2024-09-13T14:46:35.131 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:46:35.131 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:46:35.131 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:46:35.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.131 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:158985 7.3 7.e1d6ae53 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1157)", 2024-09-13T14:46:35.131 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:46:35.105892+0000", 2024-09-13T14:46:35.131 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.018028377000000002, 2024-09-13T14:46:35.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018103126000000001, 2024-09-13T14:46:35.132 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:46:35.132 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:46:35.132 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "queued for pg", 2024-09-13T14:46:35.132 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:46:35.132 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:46:35.132 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:46:35.132 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 158985 2024-09-13T14:46:35.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.132 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:46:35.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:46:35.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.105892+0000", 2024-09-13T14:46:35.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:46:35.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:46:35.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.105892+0000", 2024-09-13T14:46:35.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:46:35.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:46:35.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.105894+0000", 2024-09-13T14:46:35.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5689999999999999e-06 2024-09-13T14:46:35.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:46:35.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.105907+0000", 2024-09-13T14:46:35.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3827e-05 2024-09-13T14:46:35.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:46:35.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.106037+0000", 2024-09-13T14:46:35.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000129932 2024-09-13T14:46:35.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:46:35.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.106047+0000", 2024-09-13T14:46:35.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6830000000000003e-06 2024-09-13T14:46:35.134 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:46:35.135 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:46:35.135 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:46:35.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.135 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:158987 7.e 7:7ca16e22:::benchmark_data_smithi151_190906_object38526:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1157)", 2024-09-13T14:46:35.135 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:46:35.110831+0000", 2024-09-13T14:46:35.135 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.013089847999999999, 2024-09-13T14:46:35.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013215787999999999, 2024-09-13T14:46:35.135 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:46:35.135 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:46:35.135 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:46:35.135 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:46:35.135 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:46:35.135 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:46:35.136 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 158987 2024-09-13T14:46:35.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.136 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:46:35.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:46:35.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.110831+0000", 2024-09-13T14:46:35.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:46:35.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:46:35.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.110831+0000", 2024-09-13T14:46:35.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:46:35.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:46:35.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.110833+0000", 2024-09-13T14:46:35.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0360000000000001e-06 2024-09-13T14:46:35.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:46:35.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.110841+0000", 2024-09-13T14:46:35.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.994e-06 2024-09-13T14:46:35.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:46:35.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.110860+0000", 2024-09-13T14:46:35.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9347999999999998e-05 2024-09-13T14:46:35.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:46:35.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.110872+0000", 2024-09-13T14:46:35.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2386e-05 2024-09-13T14:46:35.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:46:35.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.113904+0000", 2024-09-13T14:46:35.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003032154 2024-09-13T14:46:35.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:46:35.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.114041+0000", 2024-09-13T14:46:35.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013633499999999999 2024-09-13T14:46:35.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:46:35.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.114167+0000", 2024-09-13T14:46:35.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012616799999999999 2024-09-13T14:46:35.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:35.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.122079+0000", 2024-09-13T14:46:35.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0079119059999999998 2024-09-13T14:46:35.139 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:46:35.139 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:46:35.139 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:46:35.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.140 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.45494.0:158988 7.0 e1157/1149 7:0a92d186:::benchmark_data_smithi151_190906_object38527:head v 1157'10042, mlcod=1157'10039)", 2024-09-13T14:46:35.140 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:46:35.114256+0000", 2024-09-13T14:46:35.140 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0096646040000000003, 2024-09-13T14:46:35.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0098543209999999992, 2024-09-13T14:46:35.140 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:46:35.140 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:46:35.140 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-13T14:46:35.140 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:46:35.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:46:35.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.114256+0000", 2024-09-13T14:46:35.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:46:35.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:46:35.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.114256+0000", 2024-09-13T14:46:35.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:46:35.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:46:35.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.114256+0000", 2024-09-13T14:46:35.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4799999999999998e-07 2024-09-13T14:46:35.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:46:35.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.114263+0000", 2024-09-13T14:46:35.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4180000000000002e-06 2024-09-13T14:46:35.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:46:35.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.114271+0000", 2024-09-13T14:46:35.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1680000000000004e-06 2024-09-13T14:46:35.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:46:35.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.114277+0000", 2024-09-13T14:46:35.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4049999999999997e-06 2024-09-13T14:46:35.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:46:35.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.114351+0000", 2024-09-13T14:46:35.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3756000000000003e-05 2024-09-13T14:46:35.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:46:35.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.114373+0000", 2024-09-13T14:46:35.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1658000000000002e-05 2024-09-13T14:46:35.144 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:46:35.145 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:46:35.145 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:46:35.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.145 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:158992 7.c 7:350cab1e:::benchmark_data_smithi151_190906_object38531:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1157)", 2024-09-13T14:46:35.145 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:46:35.122586+0000", 2024-09-13T14:46:35.145 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.001334301, 2024-09-13T14:46:35.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00157193, 2024-09-13T14:46:35.145 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:46:35.145 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:46:35.145 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:46:35.145 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:46:35.145 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:46:35.146 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:46:35.146 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 158992 2024-09-13T14:46:35.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.146 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:46:35.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:46:35.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.122586+0000", 2024-09-13T14:46:35.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:46:35.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:46:35.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.122586+0000", 2024-09-13T14:46:35.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:46:35.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:46:35.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.122588+0000", 2024-09-13T14:46:35.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.716e-06 2024-09-13T14:46:35.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:46:35.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.122593+0000", 2024-09-13T14:46:35.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4809999999999997e-06 2024-09-13T14:46:35.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:46:35.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.122605+0000", 2024-09-13T14:46:35.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2113000000000001e-05 2024-09-13T14:46:35.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:46:35.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.122613+0000", 2024-09-13T14:46:35.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9440000000000005e-06 2024-09-13T14:46:35.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:46:35.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.122670+0000", 2024-09-13T14:46:35.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7058000000000002e-05 2024-09-13T14:46:35.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:46:35.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.122819+0000", 2024-09-13T14:46:35.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000148169 2024-09-13T14:46:35.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,6,7", 2024-09-13T14:46:35.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.122948+0000", 2024-09-13T14:46:35.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012950099999999999 2024-09-13T14:46:35.149 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:46:35.149 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:46:35.149 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:46:35.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.149 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.45494.0:158991 7.5 e1157/1136 7:a3836ba5:::benchmark_data_smithi151_190906_object38530:head v 1157'9942, mlcod=1157'9940)", 2024-09-13T14:46:35.150 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:46:35.122988+0000", 2024-09-13T14:46:35.150 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.00093274099999999995, 2024-09-13T14:46:35.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012267529999999999, 2024-09-13T14:46:35.150 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:46:35.150 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:46:35.150 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-13T14:46:35.150 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:46:35.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:46:35.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.122988+0000", 2024-09-13T14:46:35.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:46:35.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:46:35.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.122988+0000", 2024-09-13T14:46:35.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:46:35.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:46:35.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.122989+0000", 2024-09-13T14:46:35.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.003e-06 2024-09-13T14:46:35.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:46:35.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.122996+0000", 2024-09-13T14:46:35.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.888e-06 2024-09-13T14:46:35.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:46:35.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.123006+0000", 2024-09-13T14:46:35.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.057e-05 2024-09-13T14:46:35.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:46:35.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.123013+0000", 2024-09-13T14:46:35.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.737e-06 2024-09-13T14:46:35.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:46:35.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.123082+0000", 2024-09-13T14:46:35.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9387000000000005e-05 2024-09-13T14:46:35.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:46:35.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:46:35.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:46:35.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:35.123104+0000", 2024-09-13T14:46:35.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1461e-05 2024-09-13T14:46:35.153 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:46:35.153 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:46:35.153 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:46:35.153 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:46:35.154 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-13T14:46:35.154 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 6 2024-09-13T14:46:35.154 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:46:35.201 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:35.239 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:35.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:35.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:35.341 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:35.361 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:35.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:35.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:35.463 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:35.487 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:35.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:35.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:35.589 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:35.612 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:35.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:35.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:35.714 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:35.737 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:35.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:35.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:35.839 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:35.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:35 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 1329/252762 objects degraded (0.526%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:46:35.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:35 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 1329/252762 objects degraded (0.526%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:46:35.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:35 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 1329/252762 objects degraded (0.526%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:46:35.861 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:35.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:35.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:35.963 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:36.003 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:36.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:36.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:36.029 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:36.105 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:36.127 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:36.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:36.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:36.229 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:36.279 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:36.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:36.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:36.381 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:36.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:36.436 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:36.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:36.537 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:36.578 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:36.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:36.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:36.607 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T14:46:36.680 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:36.704 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:36.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:36.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:36.806 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:36.829 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:36.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:36.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:36.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:36 smithi184 ceph-mon[92527]: pgmap v1578: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 623 op/s; 1329/249684 objects degraded (0.532%); 48368/249684 objects misplaced (19.372%); 870 KiB/s, 13 objects/s recovering 2024-09-13T14:46:36.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:36 smithi151 ceph-mon[104450]: pgmap v1578: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 623 op/s; 1329/249684 objects degraded (0.532%); 48368/249684 objects misplaced (19.372%); 870 KiB/s, 13 objects/s recovering 2024-09-13T14:46:36.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:36 smithi151 ceph-mon[108966]: pgmap v1578: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 623 op/s; 1329/249684 objects degraded (0.532%); 48368/249684 objects misplaced (19.372%); 870 KiB/s, 13 objects/s recovering 2024-09-13T14:46:36.931 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:36.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:36.985 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:36.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:37.085 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:37.111 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:37.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:37.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:37.213 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:37.238 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:37.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:37.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:37.340 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:37.362 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:37.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:37.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:37.464 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:37.489 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:37.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:37.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:37.591 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:37.618 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:37.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:37.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:37.720 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:37.755 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:37.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:37.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:37.857 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:37.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:37.881 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:37.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:37.982 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:38.028 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:38.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:38.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:38.129 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:38.169 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:38.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:38.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:38.271 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:38.317 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:38.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:38.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:38.419 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:38.444 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:38.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:38.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:38.546 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:38.571 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:38.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:38.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:38.673 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:38.696 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:38.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:38.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:38.798 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:38.841 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:38.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:38.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:38.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:38 smithi184 ceph-mon[92527]: pgmap v1579: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 613 op/s; 1441/247596 objects degraded (0.582%); 48017/247596 objects misplaced (19.393%); 896 KiB/s, 13 objects/s recovering 2024-09-13T14:46:38.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:38 smithi151 ceph-mon[104450]: pgmap v1579: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 613 op/s; 1441/247596 objects degraded (0.582%); 48017/247596 objects misplaced (19.393%); 896 KiB/s, 13 objects/s recovering 2024-09-13T14:46:38.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:38 smithi151 ceph-mon[108966]: pgmap v1579: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 613 op/s; 1441/247596 objects degraded (0.582%); 48017/247596 objects misplaced (19.393%); 896 KiB/s, 13 objects/s recovering 2024-09-13T14:46:38.943 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:38.968 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:38.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:38.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:39.070 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:39.111 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:39.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:39.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:39.212 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:39.285 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:39.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:39.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:39.387 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:39.414 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:39.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:39.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:39.516 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:39.537 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:39.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:39.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:39.639 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:39.662 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:39.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:39.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:39.764 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:39.810 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:39.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:39.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:39.912 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:39.949 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:39.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:39.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:40.051 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:40.092 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:40.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:40.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:40.193 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:40.219 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:40.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:40.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:40.321 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:40.357 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:40.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:40.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:40.459 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:40.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:40.487 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:40.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:40.588 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:40.593 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:46:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:46:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:46:40.612 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:40.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:40.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:40.714 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:40.723 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:40 smithi151 ceph-mon[104450]: pgmap v1580: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 616 op/s; 1441/241098 objects degraded (0.598%); 46862/241098 objects misplaced (19.437%); 752 KiB/s, 11 objects/s recovering 2024-09-13T14:46:40.723 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:40 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 1441/241098 objects degraded (0.598%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:46:40.725 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:40 smithi151 ceph-mon[108966]: pgmap v1580: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 616 op/s; 1441/241098 objects degraded (0.598%); 46862/241098 objects misplaced (19.437%); 752 KiB/s, 11 objects/s recovering 2024-09-13T14:46:40.725 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:40 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 1441/241098 objects degraded (0.598%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:46:40.768 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:40.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:40.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:40.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:40 smithi184 ceph-mon[92527]: pgmap v1580: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 616 op/s; 1441/241098 objects degraded (0.598%); 46862/241098 objects misplaced (19.437%); 752 KiB/s, 11 objects/s recovering 2024-09-13T14:46:40.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:40 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 1441/241098 objects degraded (0.598%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:46:40.870 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:40.902 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:40.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:40.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:40.970 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:41.004 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:41.028 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:41.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:41.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:41.130 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:41.171 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:41.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:41.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:41.273 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:41.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:41.320 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:41.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:41.421 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:41.445 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:46:41.446 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:46:41.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:41.447 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:41.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:41.490 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:46:41.490 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:46:41.531 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:46:41.531 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:46:41.548 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:41.566 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:46:41.566 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:46:41.586 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:41.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:41.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:41.601 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:46:41.601 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:46:41.630 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:46:41.630 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:46:41.670 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:46:41.670 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:46:41.688 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:41.702 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:46:41.702 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:46:41.734 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:41.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:41.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:41.834 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:41 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/215639251' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:41.835 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:41 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/215639251' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:41.836 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:41.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:41.873 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:41.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:41.974 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:41.997 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:41.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:41.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:42.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:42.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:41 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/215639251' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:42.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:42.138 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:42.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:42.239 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:42.278 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:42.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:42.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:42.379 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:42.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:42.430 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:42.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:42.531 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:42.556 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:42.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:42.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:42.657 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:42.681 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:42.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:42.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:42.783 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:42.804 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:42.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:42.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:42.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:42 smithi151 ceph-mon[104450]: pgmap v1581: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 569 op/s; 1589/238680 objects degraded (0.666%); 46459/238680 objects misplaced (19.465%); 1.1 MiB/s, 16 objects/s recovering 2024-09-13T14:46:42.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:42 smithi151 ceph-mon[108966]: pgmap v1581: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 569 op/s; 1589/238680 objects degraded (0.666%); 46459/238680 objects misplaced (19.465%); 1.1 MiB/s, 16 objects/s recovering 2024-09-13T14:46:42.906 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:42.933 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:42.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:42.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:43.035 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:43.076 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:43.083 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:43.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:43.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:42 smithi184 ceph-mon[92527]: pgmap v1581: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 569 op/s; 1589/238680 objects degraded (0.666%); 46459/238680 objects misplaced (19.465%); 1.1 MiB/s, 16 objects/s recovering 2024-09-13T14:46:43.186 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:43.209 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:43.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:43.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:43.310 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:43.335 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:43.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:43.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:43.437 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:43.460 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:43.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:43.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:43.562 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:43.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:43.607 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:43.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:43.708 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:43.733 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:43.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:43.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:43.835 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:43.861 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:43.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:43.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:43.963 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:43.986 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:43.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:43.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:44.088 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:44.127 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:44.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:44.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:44.229 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:44.254 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:44.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:44.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:44.356 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:44.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:44.399 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:44.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:44.500 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:44.540 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:44.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:44.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:44.642 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:44.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:44.686 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:44.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:44.787 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:44.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:44.849 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:44.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:44.850 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:44 smithi151 ceph-mon[104450]: pgmap v1582: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 625 op/s; 1589/234459 objects degraded (0.678%); 45856/234459 objects misplaced (19.558%); 698 KiB/s, 10 objects/s recovering 2024-09-13T14:46:44.850 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:44 smithi151 ceph-mon[108966]: pgmap v1582: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 625 op/s; 1589/234459 objects degraded (0.678%); 45856/234459 objects misplaced (19.558%); 698 KiB/s, 10 objects/s recovering 2024-09-13T14:46:44.851 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:46:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:46:44] "GET /metrics HTTP/1.1" 200 36667 "" "Prometheus/2.51.0" 2024-09-13T14:46:44.901 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:46:44.950 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:45.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:45.008 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:45.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:44 smithi184 ceph-mon[92527]: pgmap v1582: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 625 op/s; 1589/234459 objects degraded (0.678%); 45856/234459 objects misplaced (19.558%); 698 KiB/s, 10 objects/s recovering 2024-09-13T14:46:45.109 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:45.133 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:45.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:45.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:45.234 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:45.291 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:46:45.292 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T14:46:45.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:45.297 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:45.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:45.398 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:45.456 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:45.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:45.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:45.557 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:45.579 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:45.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:45.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:45.681 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:45.719 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:45.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:45.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:45.783 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:45 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 1589/234459 objects degraded (0.678%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:46:45.783 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:45 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 1589/234459 objects degraded (0.678%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:46:45.821 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:45.861 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:45.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:45.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:45.963 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:45.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:45.985 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:45.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:46.005 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:45 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 1589/234459 objects degraded (0.678%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:46:46.085 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:46.129 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:46.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:46.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:46.231 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:46.255 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:46.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:46.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:46.357 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:46.378 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:46.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:46.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:46.480 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:46.503 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:46.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:46.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:46.605 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:46.630 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:46.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:46.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:46.732 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:46.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:46.757 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:46.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:46.858 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:46.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:46 smithi151 ceph-mon[104450]: from='client.45614 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:46:46.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:46 smithi151 ceph-mon[104450]: pgmap v1583: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 587 op/s; 1589/231600 objects degraded (0.686%); 45289/231600 objects misplaced (19.555%); 698 KiB/s, 10 objects/s recovering 2024-09-13T14:46:46.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:46 smithi151 ceph-mon[108966]: from='client.45614 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:46:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:46 smithi151 ceph-mon[108966]: pgmap v1583: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 587 op/s; 1589/231600 objects degraded (0.686%); 45289/231600 objects misplaced (19.555%); 698 KiB/s, 10 objects/s recovering 2024-09-13T14:46:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:46.901 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:46.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:46.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:47.003 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:47.024 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:47.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:47.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:47.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:46 smithi184 ceph-mon[92527]: from='client.45614 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:46:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:46 smithi184 ceph-mon[92527]: pgmap v1583: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 587 op/s; 1589/231600 objects degraded (0.686%); 45289/231600 objects misplaced (19.555%); 698 KiB/s, 10 objects/s recovering 2024-09-13T14:46:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:47.125 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:47.148 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:47.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:47.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:47.250 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:47.282 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:47.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:47.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:47.384 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:47.406 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:47.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:47.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:47.508 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:47.529 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:47.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:47.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:47.630 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:47.654 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:47.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:47.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:47.756 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:47.803 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:47.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:47.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:47.905 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:47.927 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:47.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:47.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:48.028 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:48.076 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:48.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:48.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:48.178 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:48.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:48.214 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:48.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:48.315 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:48.338 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:48.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:48.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:48.440 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:48.462 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:48.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:48.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:48.470 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1582,"stamp":"2024-09-13T14:46:43.468184+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5120688312,"num_objects":78153,"num_object_clones":0,"num_object_copies":234459,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1589,"num_objects_misplaced":45856,"num_objects_unfound":0,"num_objects_dirty":78153,"num_whiteouts":0,"num_read":14,"num_read_kb":35,"num_write":162861,"num_write_kb":7709418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1057,"num_bytes_recovered":68708601,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115272,"ondisk_log_size":115272,"up":357,"acting":356,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":366,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":17631460,"kb_used_data":15272868,"kb_used_omap":149,"kb_used_meta":2358314,"kb_avail":732329756,"statfs":{"total":767960285184,"available":749905670144,"internally_reserved":0,"allocated":15639416832,"data_stored":15611281567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":153401,"internal_metadata":2414913735},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,3,4,6,3,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-492437504,"num_objects":-7514,"num_object_clones":0,"num_object_copies":-22542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":260,"num_objects_misplaced":-3702,"num_objects_unfound":0,"num_objects_dirty":-7514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7514,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":8585216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005431"},"pg_stats":[{"pgid":"1.1d","version":"261'100","reported_seq":84,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.575270+0000","last_change":"2024-09-13T14:45:19.807356+0000","last_active":"2024-09-13T14:46:27.575270+0000","last_peered":"2024-09-13T14:46:27.575270+0000","last_clean":"2024-09-13T14:46:27.575270+0000","last_became_active":"2024-09-13T14:45:19.807081+0000","last_became_peered":"2024-09-13T14:45:19.807081+0000","last_unstale":"2024-09-13T14:46:27.575270+0000","last_undegraded":"2024-09-13T14:46:27.575270+0000","last_fullsized":"2024-09-13T14:46:27.575270+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:17.988345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:17.988345+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:41:05.216335+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":1689,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.569844+0000","last_change":"2024-09-13T14:29:42.541263+0000","last_active":"2024-09-13T14:46:27.569844+0000","last_peered":"2024-09-13T14:46:27.569844+0000","last_clean":"2024-09-13T14:46:27.569844+0000","last_became_active":"2024-09-13T14:29:42.540873+0000","last_became_peered":"2024-09-13T14:29:42.540873+0000","last_unstale":"2024-09-13T14:46:27.569844+0000","last_undegraded":"2024-09-13T14:46:27.569844+0000","last_fullsized":"2024-09-13T14:46:27.569844+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:20.611206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1117,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.857069+0000","last_change":"2024-09-13T14:43:07.516715+0000","last_active":"2024-09-13T14:46:17.857069+0000","last_peered":"2024-09-13T14:46:17.857069+0000","last_clean":"2024-09-13T14:46:17.857069+0000","last_became_active":"2024-09-13T14:43:07.516403+0000","last_became_peered":"2024-09-13T14:43:07.516403+0000","last_unstale":"2024-09-13T14:46:17.857069+0000","last_undegraded":"2024-09-13T14:46:17.857069+0000","last_fullsized":"2024-09-13T14:46:17.857069+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:56.469690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":28,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.576151+0000","last_change":"2024-09-13T14:45:19.810203+0000","last_active":"2024-09-13T14:46:27.576151+0000","last_peered":"2024-09-13T14:46:27.576151+0000","last_clean":"2024-09-13T14:46:27.576151+0000","last_became_active":"2024-09-13T14:45:19.809693+0000","last_became_peered":"2024-09-13T14:45:19.809693+0000","last_unstale":"2024-09-13T14:46:27.576151+0000","last_undegraded":"2024-09-13T14:46:27.576151+0000","last_fullsized":"2024-09-13T14:46:27.576151+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:20.219673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":28,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.576941+0000","last_change":"2024-09-13T14:45:19.807991+0000","last_active":"2024-09-13T14:46:27.576941+0000","last_peered":"2024-09-13T14:46:27.576941+0000","last_clean":"2024-09-13T14:46:27.576941+0000","last_became_active":"2024-09-13T14:45:19.807662+0000","last_became_peered":"2024-09-13T14:45:19.807662+0000","last_unstale":"2024-09-13T14:46:27.576941+0000","last_undegraded":"2024-09-13T14:46:27.576941+0000","last_fullsized":"2024-09-13T14:46:27.576941+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:29.306108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"261'100","reported_seq":1228,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.829855+0000","last_change":"2024-09-13T14:44:54.259562+0000","last_active":"2024-09-13T14:46:17.829855+0000","last_peered":"2024-09-13T14:46:17.829855+0000","last_clean":"2024-09-13T14:46:17.829855+0000","last_became_active":"2024-09-13T14:43:14.563717+0000","last_became_peered":"2024-09-13T14:43:14.563717+0000","last_unstale":"2024-09-13T14:46:17.829855+0000","last_undegraded":"2024-09-13T14:46:17.829855+0000","last_fullsized":"2024-09-13T14:46:17.829855+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_clean_scrub_stamp":"2024-09-13T14:44:54.259482+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:43:52.729109+0000","scrub_duration":0.015265569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":1085,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.849868+0000","last_change":"2024-09-13T14:19:25.873514+0000","last_active":"2024-09-13T14:46:17.849868+0000","last_peered":"2024-09-13T14:46:17.849868+0000","last_clean":"2024-09-13T14:46:17.849868+0000","last_became_active":"2024-09-13T14:19:25.873170+0000","last_became_peered":"2024-09-13T14:19:25.873170+0000","last_unstale":"2024-09-13T14:46:17.849868+0000","last_undegraded":"2024-09-13T14:46:17.849868+0000","last_fullsized":"2024-09-13T14:46:17.849868+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:48.298530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1114,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.856217+0000","last_change":"2024-09-13T14:30:15.580011+0000","last_active":"2024-09-13T14:46:17.856217+0000","last_peered":"2024-09-13T14:46:17.856217+0000","last_clean":"2024-09-13T14:46:17.856217+0000","last_became_active":"2024-09-13T14:30:15.579610+0000","last_became_peered":"2024-09-13T14:30:15.579610+0000","last_unstale":"2024-09-13T14:46:17.856217+0000","last_undegraded":"2024-09-13T14:46:17.856217+0000","last_fullsized":"2024-09-13T14:46:17.856217+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:11:02.733771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"261'100","reported_seq":45,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.576315+0000","last_change":"2024-09-13T14:45:35.947332+0000","last_active":"2024-09-13T14:46:27.576315+0000","last_peered":"2024-09-13T14:46:27.576315+0000","last_clean":"2024-09-13T14:46:27.576315+0000","last_became_active":"2024-09-13T14:45:19.810775+0000","last_became_peered":"2024-09-13T14:45:19.810775+0000","last_unstale":"2024-09-13T14:46:27.576315+0000","last_undegraded":"2024-09-13T14:46:27.576315+0000","last_fullsized":"2024-09-13T14:46:27.576315+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:35.947265+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:43:41.569795+0000","last_clean_scrub_stamp":"2024-09-13T14:45:35.947265+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:00.606370+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":1130,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.848298+0000","last_change":"2024-09-13T14:43:07.536116+0000","last_active":"2024-09-13T14:46:17.848298+0000","last_peered":"2024-09-13T14:46:17.848298+0000","last_clean":"2024-09-13T14:46:17.848298+0000","last_became_active":"2024-09-13T14:43:07.524112+0000","last_became_peered":"2024-09-13T14:43:07.524112+0000","last_unstale":"2024-09-13T14:46:17.848298+0000","last_undegraded":"2024-09-13T14:46:17.848298+0000","last_fullsized":"2024-09-13T14:46:17.848298+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:36.947060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1127,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.856446+0000","last_change":"2024-09-13T14:43:07.530236+0000","last_active":"2024-09-13T14:46:17.856446+0000","last_peered":"2024-09-13T14:46:17.856446+0000","last_clean":"2024-09-13T14:46:17.856446+0000","last_became_active":"2024-09-13T14:43:07.529967+0000","last_became_peered":"2024-09-13T14:43:07.529967+0000","last_unstale":"2024-09-13T14:46:17.856446+0000","last_undegraded":"2024-09-13T14:46:17.856446+0000","last_fullsized":"2024-09-13T14:46:17.856446+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:27.276550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":1674,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.564239+0000","last_change":"2024-09-13T14:28:35.612469+0000","last_active":"2024-09-13T14:46:27.564239+0000","last_peered":"2024-09-13T14:46:27.564239+0000","last_clean":"2024-09-13T14:46:27.564239+0000","last_became_active":"2024-09-13T14:28:35.612093+0000","last_became_peered":"2024-09-13T14:28:35.612093+0000","last_unstale":"2024-09-13T14:46:27.564239+0000","last_undegraded":"2024-09-13T14:46:27.564239+0000","last_fullsized":"2024-09-13T14:46:27.564239+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:51.022663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1696,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.568207+0000","last_change":"2024-09-13T14:45:19.808811+0000","last_active":"2024-09-13T14:46:27.568207+0000","last_peered":"2024-09-13T14:46:27.568207+0000","last_clean":"2024-09-13T14:46:27.568207+0000","last_became_active":"2024-09-13T14:45:19.808471+0000","last_became_peered":"2024-09-13T14:45:19.808471+0000","last_unstale":"2024-09-13T14:46:27.568207+0000","last_undegraded":"2024-09-13T14:46:27.568207+0000","last_fullsized":"2024-09-13T14:46:27.568207+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:53.638161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"261'100","reported_seq":28,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.577040+0000","last_change":"2024-09-13T14:45:19.813412+0000","last_active":"2024-09-13T14:46:27.577040+0000","last_peered":"2024-09-13T14:46:27.577040+0000","last_clean":"2024-09-13T14:46:27.577040+0000","last_became_active":"2024-09-13T14:45:19.812585+0000","last_became_peered":"2024-09-13T14:45:19.812585+0000","last_unstale":"2024-09-13T14:46:27.577040+0000","last_undegraded":"2024-09-13T14:46:27.577040+0000","last_fullsized":"2024-09-13T14:46:27.577040+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.614340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.614340+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:32:07.434109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":1136,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.856282+0000","last_change":"2024-09-13T14:43:07.522950+0000","last_active":"2024-09-13T14:46:17.856282+0000","last_peered":"2024-09-13T14:46:17.856282+0000","last_clean":"2024-09-13T14:46:17.856282+0000","last_became_active":"2024-09-13T14:43:07.522566+0000","last_became_peered":"2024-09-13T14:43:07.522566+0000","last_unstale":"2024-09-13T14:46:17.856282+0000","last_undegraded":"2024-09-13T14:46:17.856282+0000","last_fullsized":"2024-09-13T14:46:17.856282+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:03.201478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":28,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.576071+0000","last_change":"2024-09-13T14:45:19.807231+0000","last_active":"2024-09-13T14:46:27.576071+0000","last_peered":"2024-09-13T14:46:27.576071+0000","last_clean":"2024-09-13T14:46:27.576071+0000","last_became_active":"2024-09-13T14:45:19.806936+0000","last_became_peered":"2024-09-13T14:45:19.806936+0000","last_unstale":"2024-09-13T14:46:27.576071+0000","last_undegraded":"2024-09-13T14:46:27.576071+0000","last_fullsized":"2024-09-13T14:46:27.576071+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:04:26.723153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.18","version":"261'100","reported_seq":106,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.847757+0000","last_change":"2024-09-13T14:44:41.685537+0000","last_active":"2024-09-13T14:46:17.847757+0000","last_peered":"2024-09-13T14:46:17.847757+0000","last_clean":"2024-09-13T14:46:17.847757+0000","last_became_active":"2024-09-13T14:43:14.558502+0000","last_became_peered":"2024-09-13T14:43:14.558502+0000","last_unstale":"2024-09-13T14:46:17.847757+0000","last_undegraded":"2024-09-13T14:46:17.847757+0000","last_fullsized":"2024-09-13T14:46:17.847757+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1134,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:41.685124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:41.685124+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:37:58.412497+0000","scrub_duration":0.0075205970000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":77,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.571116+0000","last_change":"2024-09-13T14:43:07.516845+0000","last_active":"2024-09-13T14:46:27.571116+0000","last_peered":"2024-09-13T14:46:27.571116+0000","last_clean":"2024-09-13T14:46:27.571116+0000","last_became_active":"2024-09-13T14:43:07.514974+0000","last_became_peered":"2024-09-13T14:43:07.514974+0000","last_unstale":"2024-09-13T14:46:27.571116+0000","last_undegraded":"2024-09-13T14:46:27.571116+0000","last_fullsized":"2024-09-13T14:46:27.571116+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:55.419483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1712,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.573569+0000","last_change":"2024-09-13T14:45:19.815341+0000","last_active":"2024-09-13T14:46:27.573569+0000","last_peered":"2024-09-13T14:46:27.573569+0000","last_clean":"2024-09-13T14:46:27.573569+0000","last_became_active":"2024-09-13T14:45:19.810262+0000","last_became_peered":"2024-09-13T14:45:19.810262+0000","last_unstale":"2024-09-13T14:46:27.573569+0000","last_undegraded":"2024-09-13T14:46:27.573569+0000","last_fullsized":"2024-09-13T14:46:27.573569+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:39.847653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1b","version":"261'100","reported_seq":48,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.849643+0000","last_change":"2024-09-13T14:44:52.803330+0000","last_active":"2024-09-13T14:46:17.849643+0000","last_peered":"2024-09-13T14:46:17.849643+0000","last_clean":"2024-09-13T14:46:17.849643+0000","last_became_active":"2024-09-13T14:43:14.556712+0000","last_became_peered":"2024-09-13T14:43:14.556712+0000","last_unstale":"2024-09-13T14:46:17.849643+0000","last_undegraded":"2024-09-13T14:46:17.849643+0000","last_fullsized":"2024-09-13T14:46:17.849643+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:52.803242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:52.803242+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:06.658301+0000","scrub_duration":0.0019769660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":28,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.575909+0000","last_change":"2024-09-13T14:45:19.806623+0000","last_active":"2024-09-13T14:46:27.575909+0000","last_peered":"2024-09-13T14:46:27.575909+0000","last_clean":"2024-09-13T14:46:27.575909+0000","last_became_active":"2024-09-13T14:45:19.806219+0000","last_became_peered":"2024-09-13T14:45:19.806219+0000","last_unstale":"2024-09-13T14:46:27.575909+0000","last_undegraded":"2024-09-13T14:46:27.575909+0000","last_fullsized":"2024-09-13T14:46:27.575909+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:34:41.601201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1110,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.848410+0000","last_change":"2024-09-13T14:43:32.637751+0000","last_active":"2024-09-13T14:46:17.848410+0000","last_peered":"2024-09-13T14:46:17.848410+0000","last_clean":"2024-09-13T14:46:17.848410+0000","last_became_active":"2024-09-13T14:43:32.636901+0000","last_became_peered":"2024-09-13T14:43:32.636901+0000","last_unstale":"2024-09-13T14:46:17.848410+0000","last_undegraded":"2024-09-13T14:46:17.848410+0000","last_fullsized":"2024-09-13T14:46:17.848410+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:21.942419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1a","version":"261'100","reported_seq":48,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.856102+0000","last_change":"2024-09-13T14:43:41.910041+0000","last_active":"2024-09-13T14:46:17.856102+0000","last_peered":"2024-09-13T14:46:17.856102+0000","last_clean":"2024-09-13T14:46:17.856102+0000","last_became_active":"2024-09-13T14:43:14.555083+0000","last_became_peered":"2024-09-13T14:43:14.555083+0000","last_unstale":"2024-09-13T14:46:17.856102+0000","last_undegraded":"2024-09-13T14:46:17.856102+0000","last_fullsized":"2024-09-13T14:46:17.856102+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:41.909988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:41.909988+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:42:19.954838+0000","scrub_duration":0.0012891619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":28,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.576086+0000","last_change":"2024-09-13T14:45:19.812071+0000","last_active":"2024-09-13T14:46:27.576086+0000","last_peered":"2024-09-13T14:46:27.576086+0000","last_clean":"2024-09-13T14:46:27.576086+0000","last_became_active":"2024-09-13T14:45:19.811336+0000","last_became_peered":"2024-09-13T14:45:19.811336+0000","last_unstale":"2024-09-13T14:46:27.576086+0000","last_undegraded":"2024-09-13T14:46:27.576086+0000","last_fullsized":"2024-09-13T14:46:27.576086+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:40:29.814608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1729,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.569730+0000","last_change":"2024-09-13T14:45:19.810478+0000","last_active":"2024-09-13T14:46:27.569730+0000","last_peered":"2024-09-13T14:46:27.569730+0000","last_clean":"2024-09-13T14:46:27.569730+0000","last_became_active":"2024-09-13T14:45:19.808278+0000","last_became_peered":"2024-09-13T14:45:19.808278+0000","last_unstale":"2024-09-13T14:46:27.569730+0000","last_undegraded":"2024-09-13T14:46:27.569730+0000","last_fullsized":"2024-09-13T14:46:27.569730+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:12.211089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"261'100","reported_seq":48,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.829928+0000","last_change":"2024-09-13T14:44:38.214961+0000","last_active":"2024-09-13T14:46:17.829928+0000","last_peered":"2024-09-13T14:46:17.829928+0000","last_clean":"2024-09-13T14:46:17.829928+0000","last_became_active":"2024-09-13T14:43:14.557883+0000","last_became_peered":"2024-09-13T14:43:14.557883+0000","last_unstale":"2024-09-13T14:46:17.829928+0000","last_undegraded":"2024-09-13T14:46:17.829928+0000","last_fullsized":"2024-09-13T14:46:17.829928+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:38.214865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:38.214865+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:49:36.013783+0000","scrub_duration":0.0016389639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1600,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.577217+0000","last_change":"2024-09-13T14:30:15.576921+0000","last_active":"2024-09-13T14:46:27.577217+0000","last_peered":"2024-09-13T14:46:27.577217+0000","last_clean":"2024-09-13T14:46:27.577217+0000","last_became_active":"2024-09-13T14:30:15.576607+0000","last_became_peered":"2024-09-13T14:30:15.576607+0000","last_unstale":"2024-09-13T14:46:27.577217+0000","last_undegraded":"2024-09-13T14:46:27.577217+0000","last_fullsized":"2024-09-13T14:46:27.577217+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:35.162460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1120,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.847922+0000","last_change":"2024-09-13T14:43:07.518801+0000","last_active":"2024-09-13T14:46:17.847922+0000","last_peered":"2024-09-13T14:46:17.847922+0000","last_clean":"2024-09-13T14:46:17.847922+0000","last_became_active":"2024-09-13T14:43:07.517610+0000","last_became_peered":"2024-09-13T14:43:07.517610+0000","last_unstale":"2024-09-13T14:46:17.847922+0000","last_undegraded":"2024-09-13T14:46:17.847922+0000","last_fullsized":"2024-09-13T14:46:17.847922+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:51.632254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"261'100","reported_seq":77,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.567816+0000","last_change":"2024-09-13T14:43:15.037565+0000","last_active":"2024-09-13T14:46:27.567816+0000","last_peered":"2024-09-13T14:46:27.567816+0000","last_clean":"2024-09-13T14:46:27.567816+0000","last_became_active":"2024-09-13T14:43:14.554183+0000","last_became_peered":"2024-09-13T14:43:14.554183+0000","last_unstale":"2024-09-13T14:46:27.567816+0000","last_undegraded":"2024-09-13T14:46:27.567816+0000","last_fullsized":"2024-09-13T14:46:27.567816+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.037517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.037517+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:01:35.137195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":1114,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.829808+0000","last_change":"2024-09-13T14:30:15.579260+0000","last_active":"2024-09-13T14:46:17.829808+0000","last_peered":"2024-09-13T14:46:17.829808+0000","last_clean":"2024-09-13T14:46:17.829808+0000","last_became_active":"2024-09-13T14:30:15.578897+0000","last_became_peered":"2024-09-13T14:30:15.578897+0000","last_unstale":"2024-09-13T14:46:17.829808+0000","last_undegraded":"2024-09-13T14:46:17.829808+0000","last_fullsized":"2024-09-13T14:46:17.829808+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:56.737849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":77,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.590694+0000","last_change":"2024-09-13T14:43:07.521576+0000","last_active":"2024-09-13T14:46:27.590694+0000","last_peered":"2024-09-13T14:46:27.590694+0000","last_clean":"2024-09-13T14:46:27.590694+0000","last_became_active":"2024-09-13T14:43:07.520478+0000","last_became_peered":"2024-09-13T14:43:07.520478+0000","last_unstale":"2024-09-13T14:46:27.590694+0000","last_undegraded":"2024-09-13T14:46:27.590694+0000","last_fullsized":"2024-09-13T14:46:27.590694+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:53.410135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"261'100","reported_seq":374,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.848272+0000","last_change":"2024-09-13T14:44:49.775941+0000","last_active":"2024-09-13T14:46:17.848272+0000","last_peered":"2024-09-13T14:46:17.848272+0000","last_clean":"2024-09-13T14:46:17.848272+0000","last_became_active":"2024-09-13T14:43:14.560297+0000","last_became_peered":"2024-09-13T14:43:14.560297+0000","last_unstale":"2024-09-13T14:46:17.848272+0000","last_undegraded":"2024-09-13T14:46:17.848272+0000","last_fullsized":"2024-09-13T14:46:17.848272+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:49.775844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:49.775844+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:15:53.138661+0000","scrub_duration":0.012350274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":1103,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.856438+0000","last_change":"2024-09-13T14:19:25.870605+0000","last_active":"2024-09-13T14:46:17.856438+0000","last_peered":"2024-09-13T14:46:17.856438+0000","last_clean":"2024-09-13T14:46:17.856438+0000","last_became_active":"2024-09-13T14:19:25.870282+0000","last_became_peered":"2024-09-13T14:19:25.870282+0000","last_unstale":"2024-09-13T14:46:17.856438+0000","last_undegraded":"2024-09-13T14:46:17.856438+0000","last_fullsized":"2024-09-13T14:46:17.856438+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:52.514915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1117,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.856753+0000","last_change":"2024-09-13T14:43:07.526476+0000","last_active":"2024-09-13T14:46:17.856753+0000","last_peered":"2024-09-13T14:46:17.856753+0000","last_clean":"2024-09-13T14:46:17.856753+0000","last_became_active":"2024-09-13T14:43:07.525802+0000","last_became_peered":"2024-09-13T14:43:07.525802+0000","last_unstale":"2024-09-13T14:46:17.856753+0000","last_undegraded":"2024-09-13T14:46:17.856753+0000","last_fullsized":"2024-09-13T14:46:17.856753+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:23.154726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"261'100","reported_seq":84,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.574191+0000","last_change":"2024-09-13T14:45:19.807461+0000","last_active":"2024-09-13T14:46:27.574191+0000","last_peered":"2024-09-13T14:46:27.574191+0000","last_clean":"2024-09-13T14:46:27.574191+0000","last_became_active":"2024-09-13T14:45:19.807117+0000","last_became_peered":"2024-09-13T14:45:19.807117+0000","last_unstale":"2024-09-13T14:46:27.574191+0000","last_undegraded":"2024-09-13T14:46:27.574191+0000","last_fullsized":"2024-09-13T14:46:27.574191+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.955197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.955197+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:03.531555+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"1157'10263","reported_seq":10323,"reported_epoch":1157,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:46:42.568155+0000","last_change":"2024-09-13T14:45:21.161658+0000","last_active":"2024-09-13T14:46:42.568155+0000","last_peered":"2024-09-13T14:46:42.568155+0000","last_clean":"2024-09-13T14:45:19.622135+0000","last_became_active":"2024-09-13T14:45:21.090442+0000","last_became_peered":"2024-09-13T14:45:21.090442+0000","last_unstale":"2024-09-13T14:46:42.568155+0000","last_undegraded":"2024-09-13T14:46:42.568155+0000","last_fullsized":"2024-09-13T14:46:42.568155+0000","mapping_epoch":1149,"log_start":"1142'4200","ondisk_log_start":"1142'4200","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6063,"log_dups_size":0,"ondisk_log_size":6063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323944448,"num_objects":4943,"num_object_clones":0,"num_object_copies":14829,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4943,"num_objects_unfound":0,"num_objects_dirty":4943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10263,"num_write_kb":486592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[1,2,7],"avail_no_missing":["1","2","7"],"object_location_counts":[{"shards":"1,2,7","objects":4943}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1080,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.856217+0000","last_change":"2024-09-13T14:19:26.131609+0000","last_active":"2024-09-13T14:46:17.856217+0000","last_peered":"2024-09-13T14:46:17.856217+0000","last_clean":"2024-09-13T14:46:17.856217+0000","last_became_active":"2024-09-13T14:19:26.131029+0000","last_became_peered":"2024-09-13T14:19:26.131029+0000","last_unstale":"2024-09-13T14:46:17.856217+0000","last_undegraded":"2024-09-13T14:46:17.856217+0000","last_fullsized":"2024-09-13T14:46:17.856217+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:57.933696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":582,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.830092+0000","last_change":"2024-09-13T14:43:32.635070+0000","last_active":"2024-09-13T14:46:17.830092+0000","last_peered":"2024-09-13T14:46:17.830092+0000","last_clean":"2024-09-13T14:46:17.830092+0000","last_became_active":"2024-09-13T14:43:32.634734+0000","last_became_peered":"2024-09-13T14:43:32.634734+0000","last_unstale":"2024-09-13T14:46:17.830092+0000","last_undegraded":"2024-09-13T14:46:17.830092+0000","last_fullsized":"2024-09-13T14:46:17.830092+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:33.107010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":1719,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.572437+0000","last_change":"2024-09-13T14:43:07.537759+0000","last_active":"2024-09-13T14:46:27.572437+0000","last_peered":"2024-09-13T14:46:27.572437+0000","last_clean":"2024-09-13T14:46:27.572437+0000","last_became_active":"2024-09-13T14:43:07.520303+0000","last_became_peered":"2024-09-13T14:43:07.520303+0000","last_unstale":"2024-09-13T14:46:27.572437+0000","last_undegraded":"2024-09-13T14:46:27.572437+0000","last_fullsized":"2024-09-13T14:46:27.572437+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:32.772570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"261'100","reported_seq":28,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.576751+0000","last_change":"2024-09-13T14:45:19.813527+0000","last_active":"2024-09-13T14:46:27.576751+0000","last_peered":"2024-09-13T14:46:27.576751+0000","last_clean":"2024-09-13T14:46:27.576751+0000","last_became_active":"2024-09-13T14:45:19.812834+0000","last_became_peered":"2024-09-13T14:45:19.812834+0000","last_unstale":"2024-09-13T14:46:27.576751+0000","last_undegraded":"2024-09-13T14:46:27.576751+0000","last_fullsized":"2024-09-13T14:46:27.576751+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.710192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.710192+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:08:55.152703+0000","scrub_duration":0.0019250910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"1157'10140","reported_seq":12222,"reported_epoch":1157,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:46:41.321634+0000","last_change":"2024-09-13T14:45:38.479094+0000","last_active":"2024-09-13T14:46:41.321634+0000","last_peered":"2024-09-13T14:46:41.321634+0000","last_clean":"2024-09-13T14:45:19.642017+0000","last_became_active":"2024-09-13T14:45:21.111093+0000","last_became_peered":"2024-09-13T14:45:21.111093+0000","last_unstale":"2024-09-13T14:46:41.321634+0000","last_undegraded":"2024-09-13T14:45:38.458706+0000","last_fullsized":"2024-09-13T14:45:20.800066+0000","mapping_epoch":1149,"log_start":"1141'1800","ondisk_log_start":"1141'1800","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":8340,"log_dups_size":0,"ondisk_log_size":8340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323485696,"num_objects":4936,"num_object_clones":0,"num_object_copies":14808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1589,"num_objects_misplaced":4493,"num_objects_unfound":0,"num_objects_dirty":4936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10140,"num_write_kb":482432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1013,"num_bytes_recovered":66387968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[2,4],"avail_no_missing":["2","4","6"],"object_location_counts":[{"shards":"2,4","objects":-1146},{"shards":"2,4,6","objects":4493}],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":28,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.576171+0000","last_change":"2024-09-13T14:45:19.813025+0000","last_active":"2024-09-13T14:46:27.576171+0000","last_peered":"2024-09-13T14:46:27.576171+0000","last_clean":"2024-09-13T14:46:27.576171+0000","last_became_active":"2024-09-13T14:45:19.812714+0000","last_became_peered":"2024-09-13T14:45:19.812714+0000","last_unstale":"2024-09-13T14:46:27.576171+0000","last_undegraded":"2024-09-13T14:46:27.576171+0000","last_fullsized":"2024-09-13T14:46:27.576171+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:40.671012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":77,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.574461+0000","last_change":"2024-09-13T14:43:07.522730+0000","last_active":"2024-09-13T14:46:27.574461+0000","last_peered":"2024-09-13T14:46:27.574461+0000","last_clean":"2024-09-13T14:46:27.574461+0000","last_became_active":"2024-09-13T14:43:07.517420+0000","last_became_peered":"2024-09-13T14:43:07.517420+0000","last_unstale":"2024-09-13T14:46:27.574461+0000","last_undegraded":"2024-09-13T14:46:27.574461+0000","last_fullsized":"2024-09-13T14:46:27.574461+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:21:30.265293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":1632,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.593605+0000","last_change":"2024-09-13T14:30:15.581728+0000","last_active":"2024-09-13T14:46:27.593605+0000","last_peered":"2024-09-13T14:46:27.593605+0000","last_clean":"2024-09-13T14:46:27.593605+0000","last_became_active":"2024-09-13T14:30:15.581303+0000","last_became_peered":"2024-09-13T14:30:15.581303+0000","last_unstale":"2024-09-13T14:46:27.593605+0000","last_undegraded":"2024-09-13T14:46:27.593605+0000","last_fullsized":"2024-09-13T14:46:27.593605+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:10.027991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"261'100","reported_seq":107,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.856610+0000","last_change":"2024-09-13T14:43:31.112173+0000","last_active":"2024-09-13T14:46:17.856610+0000","last_peered":"2024-09-13T14:46:17.856610+0000","last_clean":"2024-09-13T14:46:17.856610+0000","last_became_active":"2024-09-13T14:42:42.709263+0000","last_became_peered":"2024-09-13T14:42:42.709263+0000","last_unstale":"2024-09-13T14:46:17.856610+0000","last_undegraded":"2024-09-13T14:46:17.856610+0000","last_fullsized":"2024-09-13T14:46:17.856610+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:31.112111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:31.112111+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:02.707372+0000","scrub_duration":0.0015759260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"1157'10347","reported_seq":10388,"reported_epoch":1157,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:46:42.631141+0000","last_change":"2024-09-13T14:45:21.011501+0000","last_active":"2024-09-13T14:46:42.631141+0000","last_peered":"2024-09-13T14:46:42.631141+0000","last_clean":"2024-09-13T14:45:19.622974+0000","last_became_active":"2024-09-13T14:45:21.006075+0000","last_became_peered":"2024-09-13T14:45:21.006075+0000","last_unstale":"2024-09-13T14:46:42.631141+0000","last_undegraded":"2024-09-13T14:46:42.631141+0000","last_fullsized":"2024-09-13T14:46:42.631141+0000","mapping_epoch":1149,"log_start":"1141'2600","ondisk_log_start":"1141'2600","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":7747,"log_dups_size":0,"ondisk_log_size":7747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:30.090601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321847296,"num_objects":4911,"num_object_clones":0,"num_object_copies":14733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4911,"num_objects_unfound":0,"num_objects_dirty":4911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10347,"num_write_kb":488256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,2],"avail_no_missing":["5","1","2"],"object_location_counts":[{"shards":"1,2,5","objects":4911}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1672,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.564134+0000","last_change":"2024-09-13T14:28:35.611508+0000","last_active":"2024-09-13T14:46:27.564134+0000","last_peered":"2024-09-13T14:46:27.564134+0000","last_clean":"2024-09-13T14:46:27.564134+0000","last_became_active":"2024-09-13T14:28:35.610106+0000","last_became_peered":"2024-09-13T14:28:35.610106+0000","last_unstale":"2024-09-13T14:46:27.564134+0000","last_undegraded":"2024-09-13T14:46:27.564134+0000","last_fullsized":"2024-09-13T14:46:27.564134+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:11.966958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1123,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.829994+0000","last_change":"2024-09-13T14:28:35.603663+0000","last_active":"2024-09-13T14:46:17.829994+0000","last_peered":"2024-09-13T14:46:17.829994+0000","last_clean":"2024-09-13T14:46:17.829994+0000","last_became_active":"2024-09-13T14:28:35.603358+0000","last_became_peered":"2024-09-13T14:28:35.603358+0000","last_unstale":"2024-09-13T14:46:17.829994+0000","last_undegraded":"2024-09-13T14:46:17.829994+0000","last_fullsized":"2024-09-13T14:46:17.829994+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:24:05.842796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":77,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.563824+0000","last_change":"2024-09-13T14:43:07.522750+0000","last_active":"2024-09-13T14:46:27.563824+0000","last_peered":"2024-09-13T14:46:27.563824+0000","last_clean":"2024-09-13T14:46:27.563824+0000","last_became_active":"2024-09-13T14:43:07.517267+0000","last_became_peered":"2024-09-13T14:43:07.517267+0000","last_unstale":"2024-09-13T14:46:27.563824+0000","last_undegraded":"2024-09-13T14:46:27.563824+0000","last_fullsized":"2024-09-13T14:46:27.563824+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:40.217748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"261'100","reported_seq":398,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.573373+0000","last_change":"2024-09-13T14:45:19.815050+0000","last_active":"2024-09-13T14:46:27.573373+0000","last_peered":"2024-09-13T14:46:27.573373+0000","last_clean":"2024-09-13T14:46:27.573373+0000","last_became_active":"2024-09-13T14:45:19.809732+0000","last_became_peered":"2024-09-13T14:45:19.809732+0000","last_unstale":"2024-09-13T14:46:27.573373+0000","last_undegraded":"2024-09-13T14:46:27.573373+0000","last_fullsized":"2024-09-13T14:46:27.573373+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:33.574443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:33.574443+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:51:40.261036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"1157'10316","reported_seq":10415,"reported_epoch":1157,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:46:42.445043+0000","last_change":"2024-09-13T14:45:21.136109+0000","last_active":"2024-09-13T14:46:42.445043+0000","last_peered":"2024-09-13T14:46:42.445043+0000","last_clean":"2024-09-13T14:45:19.607954+0000","last_became_active":"2024-09-13T14:45:21.004300+0000","last_became_peered":"2024-09-13T14:45:21.004300+0000","last_unstale":"2024-09-13T14:46:42.445043+0000","last_undegraded":"2024-09-13T14:46:42.445043+0000","last_fullsized":"2024-09-13T14:46:42.445043+0000","mapping_epoch":1149,"log_start":"1142'4100","ondisk_log_start":"1142'4100","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6216,"log_dups_size":0,"ondisk_log_size":6216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:04:23.351582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314048512,"num_objects":4792,"num_object_clones":0,"num_object_copies":14376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4770,"num_objects_unfound":0,"num_objects_dirty":4792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10316,"num_write_kb":483456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":1835008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,6,1],"avail_no_missing":["4","1","6"],"object_location_counts":[{"shards":"1,4,6","objects":4792}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":77,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.590540+0000","last_change":"2024-09-13T14:43:07.520049+0000","last_active":"2024-09-13T14:46:27.590540+0000","last_peered":"2024-09-13T14:46:27.590540+0000","last_clean":"2024-09-13T14:46:27.590540+0000","last_became_active":"2024-09-13T14:43:07.514646+0000","last_became_peered":"2024-09-13T14:43:07.514646+0000","last_unstale":"2024-09-13T14:46:27.590540+0000","last_undegraded":"2024-09-13T14:46:27.590540+0000","last_fullsized":"2024-09-13T14:46:27.590540+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:10:35.200778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":77,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.560364+0000","last_change":"2024-09-13T14:43:07.525666+0000","last_active":"2024-09-13T14:46:27.560364+0000","last_peered":"2024-09-13T14:46:27.560364+0000","last_clean":"2024-09-13T14:46:27.560364+0000","last_became_active":"2024-09-13T14:43:07.524064+0000","last_became_peered":"2024-09-13T14:43:07.524064+0000","last_unstale":"2024-09-13T14:46:27.560364+0000","last_undegraded":"2024-09-13T14:46:27.560364+0000","last_fullsized":"2024-09-13T14:46:27.560364+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:10.270333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1137,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.848794+0000","last_change":"2024-09-13T14:45:19.804177+0000","last_active":"2024-09-13T14:46:17.848794+0000","last_peered":"2024-09-13T14:46:17.848794+0000","last_clean":"2024-09-13T14:46:17.848794+0000","last_became_active":"2024-09-13T14:45:19.803890+0000","last_became_peered":"2024-09-13T14:45:19.803890+0000","last_unstale":"2024-09-13T14:46:17.848794+0000","last_undegraded":"2024-09-13T14:46:17.848794+0000","last_fullsized":"2024-09-13T14:46:17.848794+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:25.140789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"261'100","reported_seq":45,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.593428+0000","last_change":"2024-09-13T14:45:36.965992+0000","last_active":"2024-09-13T14:46:27.593428+0000","last_peered":"2024-09-13T14:46:27.593428+0000","last_clean":"2024-09-13T14:46:27.593428+0000","last_became_active":"2024-09-13T14:45:19.811238+0000","last_became_peered":"2024-09-13T14:45:19.811238+0000","last_unstale":"2024-09-13T14:46:27.593428+0000","last_undegraded":"2024-09-13T14:46:27.593428+0000","last_fullsized":"2024-09-13T14:46:27.593428+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:36.965775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:45:36.965775+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:36:01.623251+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"1157'10007","reported_seq":10064,"reported_epoch":1157,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:46:39.364227+0000","last_change":"2024-09-13T14:45:21.036181+0000","last_active":"2024-09-13T14:46:39.364227+0000","last_peered":"2024-09-13T14:46:39.364227+0000","last_clean":"2024-09-13T14:45:19.587756+0000","last_became_active":"2024-09-13T14:45:21.029769+0000","last_became_peered":"2024-09-13T14:45:21.029769+0000","last_unstale":"2024-09-13T14:46:39.364227+0000","last_undegraded":"2024-09-13T14:46:39.364227+0000","last_fullsized":"2024-09-13T14:46:39.364227+0000","mapping_epoch":1149,"log_start":"1141'3100","ondisk_log_start":"1141'3100","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6907,"log_dups_size":0,"ondisk_log_size":6907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313982976,"num_objects":4791,"num_object_clones":0,"num_object_copies":14373,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4791,"num_objects_unfound":0,"num_objects_dirty":4791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10007,"num_write_kb":473536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,1],"avail_no_missing":["0","1","6"],"object_location_counts":[{"shards":"0,1,6","objects":4791}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":36,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.848115+0000","last_change":"2024-09-13T14:45:19.806318+0000","last_active":"2024-09-13T14:46:17.848115+0000","last_peered":"2024-09-13T14:46:17.848115+0000","last_clean":"2024-09-13T14:46:17.848115+0000","last_became_active":"2024-09-13T14:45:19.806074+0000","last_became_peered":"2024-09-13T14:45:19.806074+0000","last_unstale":"2024-09-13T14:46:17.848115+0000","last_undegraded":"2024-09-13T14:46:17.848115+0000","last_fullsized":"2024-09-13T14:46:17.848115+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:21.679797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1098,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.856594+0000","last_change":"2024-09-13T14:28:35.604404+0000","last_active":"2024-09-13T14:46:17.856594+0000","last_peered":"2024-09-13T14:46:17.856594+0000","last_clean":"2024-09-13T14:46:17.856594+0000","last_became_active":"2024-09-13T14:28:35.604102+0000","last_became_peered":"2024-09-13T14:28:35.604102+0000","last_unstale":"2024-09-13T14:46:17.856594+0000","last_undegraded":"2024-09-13T14:46:17.856594+0000","last_fullsized":"2024-09-13T14:46:17.856594+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:25.546010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":1148,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.848168+0000","last_change":"2024-09-13T14:43:07.538919+0000","last_active":"2024-09-13T14:46:17.848168+0000","last_peered":"2024-09-13T14:46:17.848168+0000","last_clean":"2024-09-13T14:46:17.848168+0000","last_became_active":"2024-09-13T14:43:07.513109+0000","last_became_peered":"2024-09-13T14:43:07.513109+0000","last_unstale":"2024-09-13T14:46:17.848168+0000","last_undegraded":"2024-09-13T14:46:17.848168+0000","last_fullsized":"2024-09-13T14:46:17.848168+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:27.771720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"261'100","reported_seq":97,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.847558+0000","last_change":"2024-09-13T14:43:44.720333+0000","last_active":"2024-09-13T14:46:17.847558+0000","last_peered":"2024-09-13T14:46:17.847558+0000","last_clean":"2024-09-13T14:46:17.847558+0000","last_became_active":"2024-09-13T14:42:14.946591+0000","last_became_peered":"2024-09-13T14:42:14.946591+0000","last_unstale":"2024-09-13T14:46:17.847558+0000","last_undegraded":"2024-09-13T14:46:17.847558+0000","last_fullsized":"2024-09-13T14:46:17.847558+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:44.720258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:44.720258+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:33.281357+0000","scrub_duration":0.0018417920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"1157'10103","reported_seq":10157,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:39.286070+0000","last_change":"2024-09-13T14:43:50.794091+0000","last_active":"2024-09-13T14:46:39.286070+0000","last_peered":"2024-09-13T14:46:39.286070+0000","last_clean":"2024-09-13T14:46:39.286070+0000","last_became_active":"2024-09-13T14:43:50.793741+0000","last_became_peered":"2024-09-13T14:43:50.793741+0000","last_unstale":"2024-09-13T14:46:39.286070+0000","last_undegraded":"2024-09-13T14:46:39.286070+0000","last_fullsized":"2024-09-13T14:46:39.286070+0000","mapping_epoch":1136,"log_start":"1142'3200","ondisk_log_start":"1142'3200","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6903,"log_dups_size":0,"ondisk_log_size":6903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:15.846494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326828032,"num_objects":4987,"num_object_clones":0,"num_object_copies":14961,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10103,"num_write_kb":482880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":45,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.568263+0000","last_change":"2024-09-13T14:44:03.858211+0000","last_active":"2024-09-13T14:46:27.568263+0000","last_peered":"2024-09-13T14:46:27.568263+0000","last_clean":"2024-09-13T14:46:27.568263+0000","last_became_active":"2024-09-13T14:44:03.857928+0000","last_became_peered":"2024-09-13T14:44:03.857928+0000","last_unstale":"2024-09-13T14:46:27.568263+0000","last_undegraded":"2024-09-13T14:46:27.568263+0000","last_fullsized":"2024-09-13T14:46:27.568263+0000","mapping_epoch":1140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:28.428693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":77,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.576026+0000","last_change":"2024-09-13T14:43:07.516243+0000","last_active":"2024-09-13T14:46:27.576026+0000","last_peered":"2024-09-13T14:46:27.576026+0000","last_clean":"2024-09-13T14:46:27.576026+0000","last_became_active":"2024-09-13T14:43:07.515863+0000","last_became_peered":"2024-09-13T14:43:07.515863+0000","last_unstale":"2024-09-13T14:46:27.576026+0000","last_undegraded":"2024-09-13T14:46:27.576026+0000","last_fullsized":"2024-09-13T14:46:27.576026+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:08.085298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1714,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.567865+0000","last_change":"2024-09-13T14:45:19.812713+0000","last_active":"2024-09-13T14:46:27.567865+0000","last_peered":"2024-09-13T14:46:27.567865+0000","last_clean":"2024-09-13T14:46:27.567865+0000","last_became_active":"2024-09-13T14:45:19.812203+0000","last_became_peered":"2024-09-13T14:45:19.812203+0000","last_unstale":"2024-09-13T14:46:27.567865+0000","last_undegraded":"2024-09-13T14:46:27.567865+0000","last_fullsized":"2024-09-13T14:46:27.567865+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:30:46.149347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"261'100","reported_seq":1277,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.829553+0000","last_change":"2024-09-13T14:43:42.577312+0000","last_active":"2024-09-13T14:46:17.829553+0000","last_peered":"2024-09-13T14:46:17.829553+0000","last_clean":"2024-09-13T14:46:17.829553+0000","last_became_active":"2024-09-13T14:42:14.928923+0000","last_became_peered":"2024-09-13T14:42:14.928923+0000","last_unstale":"2024-09-13T14:46:17.829553+0000","last_undegraded":"2024-09-13T14:46:17.829553+0000","last_fullsized":"2024-09-13T14:46:17.829553+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:42.577243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:42.577243+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:01.234250+0000","scrub_duration":0.001367839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"1157'10235","reported_seq":10263,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:42.532032+0000","last_change":"2024-09-13T14:43:50.793810+0000","last_active":"2024-09-13T14:46:42.532032+0000","last_peered":"2024-09-13T14:46:42.532032+0000","last_clean":"2024-09-13T14:46:42.532032+0000","last_became_active":"2024-09-13T14:43:50.793506+0000","last_became_peered":"2024-09-13T14:43:50.793506+0000","last_unstale":"2024-09-13T14:46:42.532032+0000","last_undegraded":"2024-09-13T14:46:42.532032+0000","last_fullsized":"2024-09-13T14:46:42.532032+0000","mapping_epoch":1136,"log_start":"1142'4100","ondisk_log_start":"1142'4100","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6135,"log_dups_size":0,"ondisk_log_size":6135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:50:32.061762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314114048,"num_objects":4793,"num_object_clones":0,"num_object_copies":14379,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10235,"num_write_kb":480896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1708,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.572491+0000","last_change":"2024-09-13T14:43:07.520212+0000","last_active":"2024-09-13T14:46:27.572491+0000","last_peered":"2024-09-13T14:46:27.572491+0000","last_clean":"2024-09-13T14:46:27.572491+0000","last_became_active":"2024-09-13T14:43:07.519831+0000","last_became_peered":"2024-09-13T14:43:07.519831+0000","last_unstale":"2024-09-13T14:46:27.572491+0000","last_undegraded":"2024-09-13T14:46:27.572491+0000","last_fullsized":"2024-09-13T14:46:27.572491+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:20.990736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1730,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.564740+0000","last_change":"2024-09-13T14:45:19.809505+0000","last_active":"2024-09-13T14:46:27.564740+0000","last_peered":"2024-09-13T14:46:27.564740+0000","last_clean":"2024-09-13T14:46:27.564740+0000","last_became_active":"2024-09-13T14:45:19.809181+0000","last_became_peered":"2024-09-13T14:45:19.809181+0000","last_unstale":"2024-09-13T14:46:27.564740+0000","last_undegraded":"2024-09-13T14:46:27.564740+0000","last_fullsized":"2024-09-13T14:46:27.564740+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:24.239188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1160,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.829664+0000","last_change":"2024-09-13T14:43:07.516140+0000","last_active":"2024-09-13T14:46:17.829664+0000","last_peered":"2024-09-13T14:46:17.829664+0000","last_clean":"2024-09-13T14:46:17.829664+0000","last_became_active":"2024-09-13T14:43:07.515481+0000","last_became_peered":"2024-09-13T14:43:07.515481+0000","last_unstale":"2024-09-13T14:46:17.829664+0000","last_undegraded":"2024-09-13T14:46:17.829664+0000","last_fullsized":"2024-09-13T14:46:17.829664+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:43.706214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"261'100","reported_seq":89,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.575757+0000","last_change":"2024-09-13T14:43:16.008346+0000","last_active":"2024-09-13T14:46:27.575757+0000","last_peered":"2024-09-13T14:46:27.575757+0000","last_clean":"2024-09-13T14:46:27.575757+0000","last_became_active":"2024-09-13T14:43:07.523121+0000","last_became_peered":"2024-09-13T14:43:07.523121+0000","last_unstale":"2024-09-13T14:46:27.575757+0000","last_undegraded":"2024-09-13T14:46:27.575757+0000","last_fullsized":"2024-09-13T14:46:27.575757+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.008264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.008264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:58:35.564470+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"1157'10043","reported_seq":10101,"reported_epoch":1157,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:46:39.346928+0000","last_change":"2024-09-13T14:45:21.035848+0000","last_active":"2024-09-13T14:46:39.346928+0000","last_peered":"2024-09-13T14:46:39.346928+0000","last_clean":"2024-09-13T14:45:19.637667+0000","last_became_active":"2024-09-13T14:45:21.025308+0000","last_became_peered":"2024-09-13T14:45:21.025308+0000","last_unstale":"2024-09-13T14:46:39.346928+0000","last_undegraded":"2024-09-13T14:46:39.346928+0000","last_fullsized":"2024-09-13T14:46:39.346928+0000","mapping_epoch":1149,"log_start":"1142'3200","ondisk_log_start":"1142'3200","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6843,"log_dups_size":0,"ondisk_log_size":6843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327352320,"num_objects":4995,"num_object_clones":0,"num_object_copies":14985,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4995,"num_objects_unfound":0,"num_objects_dirty":4995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10043,"num_write_kb":481216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,5],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":4995}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":39,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.856578+0000","last_change":"2024-09-13T14:43:32.636990+0000","last_active":"2024-09-13T14:46:17.856578+0000","last_peered":"2024-09-13T14:46:17.856578+0000","last_clean":"2024-09-13T14:46:17.856578+0000","last_became_active":"2024-09-13T14:43:32.636589+0000","last_became_peered":"2024-09-13T14:43:32.636589+0000","last_unstale":"2024-09-13T14:46:17.856578+0000","last_undegraded":"2024-09-13T14:46:17.856578+0000","last_fullsized":"2024-09-13T14:46:17.856578+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:36:25.129769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":28,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.584928+0000","last_change":"2024-09-13T14:45:19.810888+0000","last_active":"2024-09-13T14:46:27.584928+0000","last_peered":"2024-09-13T14:46:27.584928+0000","last_clean":"2024-09-13T14:46:27.584928+0000","last_became_active":"2024-09-13T14:45:19.810486+0000","last_became_peered":"2024-09-13T14:45:19.810486+0000","last_unstale":"2024-09-13T14:46:27.584928+0000","last_undegraded":"2024-09-13T14:46:27.584928+0000","last_fullsized":"2024-09-13T14:46:27.584928+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:03.886942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":1113,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.849746+0000","last_change":"2024-09-13T14:29:42.539455+0000","last_active":"2024-09-13T14:46:17.849746+0000","last_peered":"2024-09-13T14:46:17.849746+0000","last_clean":"2024-09-13T14:46:17.849746+0000","last_became_active":"2024-09-13T14:29:42.539068+0000","last_became_peered":"2024-09-13T14:29:42.539068+0000","last_unstale":"2024-09-13T14:46:17.849746+0000","last_undegraded":"2024-09-13T14:46:17.849746+0000","last_fullsized":"2024-09-13T14:46:17.849746+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:24:06.048855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034405999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"261'100","reported_seq":1268,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.829710+0000","last_change":"2024-09-13T14:44:28.186029+0000","last_active":"2024-09-13T14:46:17.829710+0000","last_peered":"2024-09-13T14:46:17.829710+0000","last_clean":"2024-09-13T14:46:17.829710+0000","last_became_active":"2024-09-13T14:42:14.927771+0000","last_became_peered":"2024-09-13T14:42:14.927771+0000","last_unstale":"2024-09-13T14:46:17.829710+0000","last_undegraded":"2024-09-13T14:46:17.829710+0000","last_fullsized":"2024-09-13T14:46:17.829710+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:28.185949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:28.185949+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:38.363961+0000","scrub_duration":0.0018071129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"1157'10083","reported_seq":10126,"reported_epoch":1157,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:46:41.258945+0000","last_change":"2024-09-13T14:45:21.030922+0000","last_active":"2024-09-13T14:46:41.258945+0000","last_peered":"2024-09-13T14:46:41.258945+0000","last_clean":"2024-09-13T14:45:19.630214+0000","last_became_active":"2024-09-13T14:45:21.021807+0000","last_became_peered":"2024-09-13T14:45:21.021807+0000","last_unstale":"2024-09-13T14:46:41.258945+0000","last_undegraded":"2024-09-13T14:46:41.258945+0000","last_fullsized":"2024-09-13T14:46:41.258945+0000","mapping_epoch":1149,"log_start":"1141'1700","ondisk_log_start":"1141'1700","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":8383,"log_dups_size":0,"ondisk_log_size":8383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318570496,"num_objects":4861,"num_object_clones":0,"num_object_copies":14583,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4861,"num_objects_unfound":0,"num_objects_dirty":4861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10083,"num_write_kb":478208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[2,6,7],"avail_no_missing":["2","6","7"],"object_location_counts":[{"shards":"2,6,7","objects":4861}],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":77,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.570325+0000","last_change":"2024-09-13T14:43:07.520176+0000","last_active":"2024-09-13T14:46:27.570325+0000","last_peered":"2024-09-13T14:46:27.570325+0000","last_clean":"2024-09-13T14:46:27.570325+0000","last_became_active":"2024-09-13T14:43:07.516748+0000","last_became_peered":"2024-09-13T14:43:07.516748+0000","last_unstale":"2024-09-13T14:46:27.570325+0000","last_undegraded":"2024-09-13T14:46:27.570325+0000","last_fullsized":"2024-09-13T14:46:27.570325+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:52:04.392821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":28,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.576256+0000","last_change":"2024-09-13T14:45:19.812353+0000","last_active":"2024-09-13T14:46:27.576256+0000","last_peered":"2024-09-13T14:46:27.576256+0000","last_clean":"2024-09-13T14:46:27.576256+0000","last_became_active":"2024-09-13T14:45:19.811743+0000","last_became_peered":"2024-09-13T14:45:19.811743+0000","last_unstale":"2024-09-13T14:46:27.576256+0000","last_undegraded":"2024-09-13T14:46:27.576256+0000","last_fullsized":"2024-09-13T14:46:27.576256+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:43.728170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"261'100","reported_seq":97,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.847830+0000","last_change":"2024-09-13T14:44:34.814157+0000","last_active":"2024-09-13T14:46:17.847830+0000","last_peered":"2024-09-13T14:46:17.847830+0000","last_clean":"2024-09-13T14:46:17.847830+0000","last_became_active":"2024-09-13T14:42:42.707803+0000","last_became_peered":"2024-09-13T14:42:42.707803+0000","last_unstale":"2024-09-13T14:46:17.847830+0000","last_undegraded":"2024-09-13T14:46:17.847830+0000","last_fullsized":"2024-09-13T14:46:17.847830+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:34.814070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:34.814070+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:38.408561+0000","scrub_duration":0.01686414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"1157'10257","reported_seq":10312,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:42.547942+0000","last_change":"2024-09-13T14:43:51.534705+0000","last_active":"2024-09-13T14:46:42.547942+0000","last_peered":"2024-09-13T14:46:42.547942+0000","last_clean":"2024-09-13T14:46:42.547942+0000","last_became_active":"2024-09-13T14:43:51.534381+0000","last_became_peered":"2024-09-13T14:43:51.534381+0000","last_unstale":"2024-09-13T14:46:42.547942+0000","last_undegraded":"2024-09-13T14:46:42.547942+0000","last_fullsized":"2024-09-13T14:46:42.547942+0000","mapping_epoch":1136,"log_start":"1142'4200","ondisk_log_start":"1142'4200","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6057,"log_dups_size":0,"ondisk_log_size":6057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:37:22.385742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318898200,"num_objects":4867,"num_object_clones":0,"num_object_copies":14601,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4867,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10257,"num_write_kb":483905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1127,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.856688+0000","last_change":"2024-09-13T14:44:03.855577+0000","last_active":"2024-09-13T14:46:17.856688+0000","last_peered":"2024-09-13T14:46:17.856688+0000","last_clean":"2024-09-13T14:46:17.856688+0000","last_became_active":"2024-09-13T14:44:03.855267+0000","last_became_peered":"2024-09-13T14:44:03.855267+0000","last_unstale":"2024-09-13T14:46:17.856688+0000","last_undegraded":"2024-09-13T14:46:17.856688+0000","last_fullsized":"2024-09-13T14:46:17.856688+0000","mapping_epoch":1140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:46.148814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1098,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.856524+0000","last_change":"2024-09-13T14:29:42.545815+0000","last_active":"2024-09-13T14:46:17.856524+0000","last_peered":"2024-09-13T14:46:17.856524+0000","last_clean":"2024-09-13T14:46:17.856524+0000","last_became_active":"2024-09-13T14:29:42.545508+0000","last_became_peered":"2024-09-13T14:29:42.545508+0000","last_unstale":"2024-09-13T14:46:17.856524+0000","last_undegraded":"2024-09-13T14:46:17.856524+0000","last_fullsized":"2024-09-13T14:46:17.856524+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:39.678381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"261'100","reported_seq":95,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.576146+0000","last_change":"2024-09-13T14:43:18.988482+0000","last_active":"2024-09-13T14:46:27.576146+0000","last_peered":"2024-09-13T14:46:27.576146+0000","last_clean":"2024-09-13T14:46:27.576146+0000","last_became_active":"2024-09-13T14:43:14.556754+0000","last_became_peered":"2024-09-13T14:43:14.556754+0000","last_unstale":"2024-09-13T14:46:27.576146+0000","last_undegraded":"2024-09-13T14:46:27.576146+0000","last_fullsized":"2024-09-13T14:46:27.576146+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:18.988405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:18.988405+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:45:01.726326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"1157'10036","reported_seq":10090,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:41.281785+0000","last_change":"2024-09-13T14:43:50.796182+0000","last_active":"2024-09-13T14:46:41.281785+0000","last_peered":"2024-09-13T14:46:41.281785+0000","last_clean":"2024-09-13T14:46:41.281785+0000","last_became_active":"2024-09-13T14:43:50.795875+0000","last_became_peered":"2024-09-13T14:43:50.795875+0000","last_unstale":"2024-09-13T14:46:41.281785+0000","last_undegraded":"2024-09-13T14:46:41.281785+0000","last_fullsized":"2024-09-13T14:46:41.281785+0000","mapping_epoch":1136,"log_start":"1141'1700","ondisk_log_start":"1141'1700","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":8336,"log_dups_size":0,"ondisk_log_size":8336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:56:34.308870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310378496,"num_objects":4736,"num_object_clones":0,"num_object_copies":14208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10036,"num_write_kb":472704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1144,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.856705+0000","last_change":"2024-09-13T14:43:07.515418+0000","last_active":"2024-09-13T14:46:17.856705+0000","last_peered":"2024-09-13T14:46:17.856705+0000","last_clean":"2024-09-13T14:46:17.856705+0000","last_became_active":"2024-09-13T14:43:07.515113+0000","last_became_peered":"2024-09-13T14:43:07.515113+0000","last_unstale":"2024-09-13T14:46:17.856705+0000","last_undegraded":"2024-09-13T14:46:17.856705+0000","last_fullsized":"2024-09-13T14:46:17.856705+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:10:25.810604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1111,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.856198+0000","last_change":"2024-09-13T14:43:32.636334+0000","last_active":"2024-09-13T14:46:17.856198+0000","last_peered":"2024-09-13T14:46:17.856198+0000","last_clean":"2024-09-13T14:46:17.856198+0000","last_became_active":"2024-09-13T14:43:32.635736+0000","last_became_peered":"2024-09-13T14:43:32.635736+0000","last_unstale":"2024-09-13T14:46:17.856198+0000","last_undegraded":"2024-09-13T14:46:17.856198+0000","last_fullsized":"2024-09-13T14:46:17.856198+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:49.076014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"261'100","reported_seq":102,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.847638+0000","last_change":"2024-09-13T14:45:19.805634+0000","last_active":"2024-09-13T14:46:17.847638+0000","last_peered":"2024-09-13T14:46:17.847638+0000","last_clean":"2024-09-13T14:46:17.847638+0000","last_became_active":"2024-09-13T14:45:19.805046+0000","last_became_peered":"2024-09-13T14:45:19.805046+0000","last_unstale":"2024-09-13T14:46:17.847638+0000","last_undegraded":"2024-09-13T14:46:17.847638+0000","last_fullsized":"2024-09-13T14:46:17.847638+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:32.752408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:32.752408+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:06:53.140239+0000","scrub_duration":0.0057731559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"1157'10161","reported_seq":10220,"reported_epoch":1157,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:46:39.324644+0000","last_change":"2024-09-13T14:45:21.029184+0000","last_active":"2024-09-13T14:46:39.324644+0000","last_peered":"2024-09-13T14:46:39.324644+0000","last_clean":"2024-09-13T14:45:19.633080+0000","last_became_active":"2024-09-13T14:45:21.019809+0000","last_became_peered":"2024-09-13T14:45:21.019809+0000","last_unstale":"2024-09-13T14:46:39.324644+0000","last_undegraded":"2024-09-13T14:46:39.324644+0000","last_fullsized":"2024-09-13T14:46:39.324644+0000","mapping_epoch":1149,"log_start":"1142'3300","ondisk_log_start":"1142'3300","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6861,"log_dups_size":0,"ondisk_log_size":6861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328663040,"num_objects":5015,"num_object_clones":0,"num_object_copies":15045,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5015,"num_objects_unfound":0,"num_objects_dirty":5015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10161,"num_write_kb":485632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,7,4],"avail_no_missing":["0","4","7"],"object_location_counts":[{"shards":"0,4,7","objects":5015}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1147,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.856403+0000","last_change":"2024-09-13T14:45:19.804441+0000","last_active":"2024-09-13T14:46:17.856403+0000","last_peered":"2024-09-13T14:46:17.856403+0000","last_clean":"2024-09-13T14:46:17.856403+0000","last_became_active":"2024-09-13T14:45:19.804039+0000","last_became_peered":"2024-09-13T14:45:19.804039+0000","last_unstale":"2024-09-13T14:46:17.856403+0000","last_undegraded":"2024-09-13T14:46:17.856403+0000","last_fullsized":"2024-09-13T14:46:17.856403+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:58:00.322042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":32,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.580388+0000","last_change":"2024-09-13T14:45:19.816666+0000","last_active":"2024-09-13T14:46:27.580388+0000","last_peered":"2024-09-13T14:46:27.580388+0000","last_clean":"2024-09-13T14:46:27.580388+0000","last_became_active":"2024-09-13T14:45:19.807224+0000","last_became_peered":"2024-09-13T14:45:19.807224+0000","last_unstale":"2024-09-13T14:46:27.580388+0000","last_undegraded":"2024-09-13T14:46:27.580388+0000","last_fullsized":"2024-09-13T14:46:27.580388+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:15:20.171682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"261'100","reported_seq":1287,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.829457+0000","last_change":"2024-09-13T14:45:19.798553+0000","last_active":"2024-09-13T14:46:17.829457+0000","last_peered":"2024-09-13T14:46:17.829457+0000","last_clean":"2024-09-13T14:46:17.829457+0000","last_became_active":"2024-09-13T14:45:19.798189+0000","last_became_peered":"2024-09-13T14:45:19.798189+0000","last_unstale":"2024-09-13T14:46:17.829457+0000","last_undegraded":"2024-09-13T14:46:17.829457+0000","last_fullsized":"2024-09-13T14:46:17.829457+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:29.169325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:29.169325+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:10.412666+0000","scrub_duration":0.0019358890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"1157'10221","reported_seq":10275,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:41.250965+0000","last_change":"2024-09-13T14:43:50.795029+0000","last_active":"2024-09-13T14:46:41.250965+0000","last_peered":"2024-09-13T14:46:41.250965+0000","last_clean":"2024-09-13T14:46:41.250965+0000","last_became_active":"2024-09-13T14:43:50.794715+0000","last_became_peered":"2024-09-13T14:43:50.794715+0000","last_unstale":"2024-09-13T14:46:41.250965+0000","last_undegraded":"2024-09-13T14:46:41.250965+0000","last_fullsized":"2024-09-13T14:46:41.250965+0000","mapping_epoch":1136,"log_start":"1141'1800","ondisk_log_start":"1141'1800","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":8421,"log_dups_size":0,"ondisk_log_size":8421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:21.257279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320274432,"num_objects":4887,"num_object_clones":0,"num_object_copies":14661,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10221,"num_write_kb":483456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1697,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.563766+0000","last_change":"2024-09-13T14:43:07.515572+0000","last_active":"2024-09-13T14:46:27.563766+0000","last_peered":"2024-09-13T14:46:27.563766+0000","last_clean":"2024-09-13T14:46:27.563766+0000","last_became_active":"2024-09-13T14:43:07.515231+0000","last_became_peered":"2024-09-13T14:43:07.515231+0000","last_unstale":"2024-09-13T14:46:27.563766+0000","last_undegraded":"2024-09-13T14:46:27.563766+0000","last_fullsized":"2024-09-13T14:46:27.563766+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:57.966380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1140,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.848369+0000","last_change":"2024-09-13T14:45:19.804982+0000","last_active":"2024-09-13T14:46:17.848369+0000","last_peered":"2024-09-13T14:46:17.848369+0000","last_clean":"2024-09-13T14:46:17.848369+0000","last_became_active":"2024-09-13T14:45:19.804663+0000","last_became_peered":"2024-09-13T14:45:19.804663+0000","last_unstale":"2024-09-13T14:46:17.848369+0000","last_undegraded":"2024-09-13T14:46:17.848369+0000","last_fullsized":"2024-09-13T14:46:17.848369+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:43.211479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"261'100","reported_seq":377,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.856642+0000","last_change":"2024-09-13T14:43:42.881584+0000","last_active":"2024-09-13T14:46:17.856642+0000","last_peered":"2024-09-13T14:46:17.856642+0000","last_clean":"2024-09-13T14:46:17.856642+0000","last_became_active":"2024-09-13T14:43:14.562161+0000","last_became_peered":"2024-09-13T14:43:14.562161+0000","last_unstale":"2024-09-13T14:46:17.856642+0000","last_undegraded":"2024-09-13T14:46:17.856642+0000","last_fullsized":"2024-09-13T14:46:17.856642+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:42.881520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:42.881520+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:06:50.353442+0000","scrub_duration":0.0013804310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"1157'10154","reported_seq":10208,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:39.284991+0000","last_change":"2024-09-13T14:43:50.793879+0000","last_active":"2024-09-13T14:46:39.284991+0000","last_peered":"2024-09-13T14:46:39.284991+0000","last_clean":"2024-09-13T14:46:39.284991+0000","last_became_active":"2024-09-13T14:43:50.793571+0000","last_became_peered":"2024-09-13T14:43:50.793571+0000","last_unstale":"2024-09-13T14:46:39.284991+0000","last_undegraded":"2024-09-13T14:46:39.284991+0000","last_fullsized":"2024-09-13T14:46:39.284991+0000","mapping_epoch":1136,"log_start":"1142'3300","ondisk_log_start":"1142'3300","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6854,"log_dups_size":0,"ondisk_log_size":6854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:22.880360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331350016,"num_objects":5056,"num_object_clones":0,"num_object_copies":15168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10154,"num_write_kb":486720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1113,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.848893+0000","last_change":"2024-09-13T14:43:07.518120+0000","last_active":"2024-09-13T14:46:17.848893+0000","last_peered":"2024-09-13T14:46:17.848893+0000","last_clean":"2024-09-13T14:46:17.848893+0000","last_became_active":"2024-09-13T14:43:07.517361+0000","last_became_peered":"2024-09-13T14:43:07.517361+0000","last_unstale":"2024-09-13T14:46:17.848893+0000","last_undegraded":"2024-09-13T14:46:17.848893+0000","last_fullsized":"2024-09-13T14:46:17.848893+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:39.366473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":1095,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.829905+0000","last_change":"2024-09-13T14:19:25.871006+0000","last_active":"2024-09-13T14:46:17.829905+0000","last_peered":"2024-09-13T14:46:17.829905+0000","last_clean":"2024-09-13T14:46:17.829905+0000","last_became_active":"2024-09-13T14:19:25.870713+0000","last_became_peered":"2024-09-13T14:19:25.870713+0000","last_unstale":"2024-09-13T14:46:17.829905+0000","last_undegraded":"2024-09-13T14:46:17.829905+0000","last_fullsized":"2024-09-13T14:46:17.829905+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:34.372124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"261'100","reported_seq":361,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.848345+0000","last_change":"2024-09-13T14:43:32.637553+0000","last_active":"2024-09-13T14:46:17.848345+0000","last_peered":"2024-09-13T14:46:17.848345+0000","last_clean":"2024-09-13T14:46:17.848345+0000","last_became_active":"2024-09-13T14:43:32.636545+0000","last_became_peered":"2024-09-13T14:43:32.636545+0000","last_unstale":"2024-09-13T14:46:17.848345+0000","last_undegraded":"2024-09-13T14:46:17.848345+0000","last_fullsized":"2024-09-13T14:46:17.848345+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:15:42.069756+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"1157'10202","reported_seq":10256,"reported_epoch":1157,"state":"active+recovery_wait+remapped","last_fresh":"2024-09-13T14:46:42.488621+0000","last_change":"2024-09-13T14:45:21.130410+0000","last_active":"2024-09-13T14:46:42.488621+0000","last_peered":"2024-09-13T14:46:42.488621+0000","last_clean":"2024-09-13T14:45:19.587556+0000","last_became_active":"2024-09-13T14:45:21.100167+0000","last_became_peered":"2024-09-13T14:45:21.100167+0000","last_unstale":"2024-09-13T14:46:42.488621+0000","last_undegraded":"2024-09-13T14:46:42.488621+0000","last_fullsized":"2024-09-13T14:46:42.488621+0000","mapping_epoch":1149,"log_start":"1142'4177","ondisk_log_start":"1142'4177","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6025,"log_dups_size":0,"ondisk_log_size":6025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315490304,"num_objects":4814,"num_object_clones":0,"num_object_copies":14442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7077,"num_objects_unfound":0,"num_objects_dirty":4814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10202,"num_write_kb":480512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[1,2,5],"avail_no_missing":["1","2","5"],"object_location_counts":[{"shards":"1,2,5","objects":-2263}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1164,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.856371+0000","last_change":"2024-09-13T14:45:19.804868+0000","last_active":"2024-09-13T14:46:17.856371+0000","last_peered":"2024-09-13T14:46:17.856371+0000","last_clean":"2024-09-13T14:46:17.856371+0000","last_became_active":"2024-09-13T14:45:19.804520+0000","last_became_peered":"2024-09-13T14:45:19.804520+0000","last_unstale":"2024-09-13T14:46:17.856371+0000","last_undegraded":"2024-09-13T14:46:17.856371+0000","last_fullsized":"2024-09-13T14:46:17.856371+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:17:53.924629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":1141,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.830023+0000","last_change":"2024-09-13T14:43:07.533955+0000","last_active":"2024-09-13T14:46:17.830023+0000","last_peered":"2024-09-13T14:46:17.830023+0000","last_clean":"2024-09-13T14:46:17.830023+0000","last_became_active":"2024-09-13T14:43:07.516816+0000","last_became_peered":"2024-09-13T14:43:07.516816+0000","last_unstale":"2024-09-13T14:46:17.830023+0000","last_undegraded":"2024-09-13T14:46:17.830023+0000","last_fullsized":"2024-09-13T14:46:17.830023+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:26.232184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"261'100","reported_seq":32,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.576515+0000","last_change":"2024-09-13T14:45:19.821649+0000","last_active":"2024-09-13T14:46:27.576515+0000","last_peered":"2024-09-13T14:46:27.576515+0000","last_clean":"2024-09-13T14:46:27.576515+0000","last_became_active":"2024-09-13T14:45:19.812346+0000","last_became_peered":"2024-09-13T14:45:19.812346+0000","last_unstale":"2024-09-13T14:46:27.576515+0000","last_undegraded":"2024-09-13T14:46:27.576515+0000","last_fullsized":"2024-09-13T14:46:27.576515+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.597568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.597568+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:45.218892+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"1157'10223","reported_seq":10277,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:42.553208+0000","last_change":"2024-09-13T14:43:51.537117+0000","last_active":"2024-09-13T14:46:42.553208+0000","last_peered":"2024-09-13T14:46:42.553208+0000","last_clean":"2024-09-13T14:46:42.553208+0000","last_became_active":"2024-09-13T14:43:51.536532+0000","last_became_peered":"2024-09-13T14:43:51.536532+0000","last_unstale":"2024-09-13T14:46:42.553208+0000","last_undegraded":"2024-09-13T14:46:42.553208+0000","last_fullsized":"2024-09-13T14:46:42.553208+0000","mapping_epoch":1136,"log_start":"1142'4200","ondisk_log_start":"1142'4200","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6023,"log_dups_size":0,"ondisk_log_size":6023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:41.713211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310837248,"num_objects":4743,"num_object_clones":0,"num_object_copies":14229,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10223,"num_write_kb":478912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1082,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.847832+0000","last_change":"2024-09-13T14:19:26.887839+0000","last_active":"2024-09-13T14:46:17.847832+0000","last_peered":"2024-09-13T14:46:17.847832+0000","last_clean":"2024-09-13T14:46:17.847832+0000","last_became_active":"2024-09-13T14:19:26.887498+0000","last_became_peered":"2024-09-13T14:19:26.887498+0000","last_unstale":"2024-09-13T14:46:17.847832+0000","last_undegraded":"2024-09-13T14:46:17.847832+0000","last_fullsized":"2024-09-13T14:46:17.847832+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:24.475263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":1112,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.830096+0000","last_change":"2024-09-13T14:28:35.608607+0000","last_active":"2024-09-13T14:46:17.830096+0000","last_peered":"2024-09-13T14:46:17.830096+0000","last_clean":"2024-09-13T14:46:17.830096+0000","last_became_active":"2024-09-13T14:28:35.608254+0000","last_became_peered":"2024-09-13T14:28:35.608254+0000","last_unstale":"2024-09-13T14:46:17.830096+0000","last_undegraded":"2024-09-13T14:46:17.830096+0000","last_fullsized":"2024-09-13T14:46:17.830096+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:24.019173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"261'100","reported_seq":48,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.829705+0000","last_change":"2024-09-13T14:44:31.191431+0000","last_active":"2024-09-13T14:46:17.829705+0000","last_peered":"2024-09-13T14:46:17.829705+0000","last_clean":"2024-09-13T14:46:17.829705+0000","last_became_active":"2024-09-13T14:43:14.557376+0000","last_became_peered":"2024-09-13T14:43:14.557376+0000","last_unstale":"2024-09-13T14:46:17.829705+0000","last_undegraded":"2024-09-13T14:46:17.829705+0000","last_fullsized":"2024-09-13T14:46:17.829705+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.191326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.191326+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:15:27.352698+0000","scrub_duration":0.0025317360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1102,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.847662+0000","last_change":"2024-09-13T14:28:35.603724+0000","last_active":"2024-09-13T14:46:17.847662+0000","last_peered":"2024-09-13T14:46:17.847662+0000","last_clean":"2024-09-13T14:46:17.847662+0000","last_became_active":"2024-09-13T14:28:35.603316+0000","last_became_peered":"2024-09-13T14:28:35.603316+0000","last_unstale":"2024-09-13T14:46:17.847662+0000","last_undegraded":"2024-09-13T14:46:17.847662+0000","last_fullsized":"2024-09-13T14:46:17.847662+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:47.668223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":28,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.597467+0000","last_change":"2024-09-13T14:45:19.811466+0000","last_active":"2024-09-13T14:46:27.597467+0000","last_peered":"2024-09-13T14:46:27.597467+0000","last_clean":"2024-09-13T14:46:27.597467+0000","last_became_active":"2024-09-13T14:45:19.810617+0000","last_became_peered":"2024-09-13T14:45:19.810617+0000","last_unstale":"2024-09-13T14:46:27.597467+0000","last_undegraded":"2024-09-13T14:46:27.597467+0000","last_fullsized":"2024-09-13T14:46:27.597467+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:16.246669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"261'100","reported_seq":48,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.847997+0000","last_change":"2024-09-13T14:44:50.805306+0000","last_active":"2024-09-13T14:46:17.847997+0000","last_peered":"2024-09-13T14:46:17.847997+0000","last_clean":"2024-09-13T14:46:17.847997+0000","last_became_active":"2024-09-13T14:43:14.562922+0000","last_became_peered":"2024-09-13T14:43:14.562922+0000","last_unstale":"2024-09-13T14:46:17.847997+0000","last_undegraded":"2024-09-13T14:46:17.847997+0000","last_fullsized":"2024-09-13T14:46:17.847997+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:50.805210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:50.805210+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:28:38.529599+0000","scrub_duration":0.0042900009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1094,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.829551+0000","last_change":"2024-09-13T14:19:25.873088+0000","last_active":"2024-09-13T14:46:17.829551+0000","last_peered":"2024-09-13T14:46:17.829551+0000","last_clean":"2024-09-13T14:46:17.829551+0000","last_became_active":"2024-09-13T14:19:25.872781+0000","last_became_peered":"2024-09-13T14:19:25.872781+0000","last_unstale":"2024-09-13T14:46:17.829551+0000","last_undegraded":"2024-09-13T14:46:17.829551+0000","last_fullsized":"2024-09-13T14:46:17.829551+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:05.607382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1696,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.573615+0000","last_change":"2024-09-13T14:45:19.813876+0000","last_active":"2024-09-13T14:46:27.573615+0000","last_peered":"2024-09-13T14:46:27.573615+0000","last_clean":"2024-09-13T14:46:27.573615+0000","last_became_active":"2024-09-13T14:45:19.809136+0000","last_became_peered":"2024-09-13T14:45:19.809136+0000","last_unstale":"2024-09-13T14:46:27.573615+0000","last_undegraded":"2024-09-13T14:46:27.573615+0000","last_fullsized":"2024-09-13T14:46:27.573615+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:50.013528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"261'100","reported_seq":102,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.576285+0000","last_change":"2024-09-13T14:45:19.810654+0000","last_active":"2024-09-13T14:46:27.576285+0000","last_peered":"2024-09-13T14:46:27.576285+0000","last_clean":"2024-09-13T14:46:27.576285+0000","last_became_active":"2024-09-13T14:45:19.810066+0000","last_became_peered":"2024-09-13T14:45:19.810066+0000","last_unstale":"2024-09-13T14:46:27.576285+0000","last_undegraded":"2024-09-13T14:46:27.576285+0000","last_fullsized":"2024-09-13T14:46:27.576285+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:19.978464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:19.978464+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:33.639332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1743,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.563911+0000","last_change":"2024-09-13T14:45:19.809166+0000","last_active":"2024-09-13T14:46:27.563911+0000","last_peered":"2024-09-13T14:46:27.563911+0000","last_clean":"2024-09-13T14:46:27.563911+0000","last_became_active":"2024-09-13T14:45:19.808631+0000","last_became_peered":"2024-09-13T14:45:19.808631+0000","last_unstale":"2024-09-13T14:46:27.563911+0000","last_undegraded":"2024-09-13T14:46:27.563911+0000","last_fullsized":"2024-09-13T14:46:27.563911+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:38.822245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1119,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.848512+0000","last_change":"2024-09-13T14:45:19.804229+0000","last_active":"2024-09-13T14:46:17.848512+0000","last_peered":"2024-09-13T14:46:17.848512+0000","last_clean":"2024-09-13T14:46:17.848512+0000","last_became_active":"2024-09-13T14:45:19.803929+0000","last_became_peered":"2024-09-13T14:45:19.803929+0000","last_unstale":"2024-09-13T14:46:17.848512+0000","last_undegraded":"2024-09-13T14:46:17.848512+0000","last_fullsized":"2024-09-13T14:46:17.848512+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:33:53.675568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"261'100","reported_seq":48,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.847557+0000","last_change":"2024-09-13T14:44:42.654506+0000","last_active":"2024-09-13T14:46:17.847557+0000","last_peered":"2024-09-13T14:46:17.847557+0000","last_clean":"2024-09-13T14:46:17.847557+0000","last_became_active":"2024-09-13T14:43:14.563451+0000","last_became_peered":"2024-09-13T14:43:14.563451+0000","last_unstale":"2024-09-13T14:46:17.847557+0000","last_undegraded":"2024-09-13T14:46:17.847557+0000","last_fullsized":"2024-09-13T14:46:17.847557+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:42.654455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:42.654455+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:57:32.251597+0000","scrub_duration":0.001494097,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":1113,"reported_epoch":1156,"state":"active+clean","last_fresh":"2024-09-13T14:46:17.829671+0000","last_change":"2024-09-13T14:29:42.541093+0000","last_active":"2024-09-13T14:46:17.829671+0000","last_peered":"2024-09-13T14:46:17.829671+0000","last_clean":"2024-09-13T14:46:17.829671+0000","last_became_active":"2024-09-13T14:29:42.540671+0000","last_became_peered":"2024-09-13T14:29:42.540671+0000","last_unstale":"2024-09-13T14:46:17.829671+0000","last_undegraded":"2024-09-13T14:46:17.829671+0000","last_fullsized":"2024-09-13T14:46:17.829671+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:45.722376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1635,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.576944+0000","last_change":"2024-09-13T14:43:07.518679+0000","last_active":"2024-09-13T14:46:27.576944+0000","last_peered":"2024-09-13T14:46:27.576944+0000","last_clean":"2024-09-13T14:46:27.576944+0000","last_became_active":"2024-09-13T14:43:07.518333+0000","last_became_peered":"2024-09-13T14:43:07.518333+0000","last_unstale":"2024-09-13T14:46:27.576944+0000","last_undegraded":"2024-09-13T14:46:27.576944+0000","last_fullsized":"2024-09-13T14:46:27.576944+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:20.616274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"261'100","reported_seq":404,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.564846+0000","last_change":"2024-09-13T14:45:19.809268+0000","last_active":"2024-09-13T14:46:27.564846+0000","last_peered":"2024-09-13T14:46:27.564846+0000","last_clean":"2024-09-13T14:46:27.564846+0000","last_became_active":"2024-09-13T14:45:19.808772+0000","last_became_peered":"2024-09-13T14:45:19.808772+0000","last_unstale":"2024-09-13T14:46:27.564846+0000","last_undegraded":"2024-09-13T14:46:27.564846+0000","last_fullsized":"2024-09-13T14:46:27.564846+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:14.654603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:14.654603+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:40.964553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":5120065560,"num_objects":78127,"num_object_clones":0,"num_object_copies":234381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1589,"num_objects_misplaced":45856,"num_objects_unfound":0,"num_objects_dirty":78127,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162791,"num_write_kb":7709313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1043,"num_bytes_recovered":68354048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15492198400,"data_stored":15492186184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112114,"ondisk_log_size":112114,"up":48,"acting":47,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":354515,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430560,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2189860,"kb_used_data":1884188,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91555292,"statfs":{"total":95995035648,"available":93752619008,"internally_reserved":0,"allocated":1929408512,"data_stored":1925878925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626789,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2296092,"kb_used_data":1990420,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91449060,"statfs":{"total":95995035648,"available":93643837440,"internally_reserved":0,"allocated":2038190080,"data_stored":2034668046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823018,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1849492,"kb_used_data":1547980,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91895660,"statfs":{"total":95995035648,"available":94101155840,"internally_reserved":0,"allocated":1585131520,"data_stored":1581614016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019246,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2178972,"kb_used_data":1881620,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":91566180,"statfs":{"total":95995035648,"available":93763768320,"internally_reserved":0,"allocated":1926778880,"data_stored":1923256871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059592903,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":212552,"kb_used_data":68224,"kb_used_omap":26,"kb_used_meta":144293,"kb_avail":93532600,"statfs":{"total":95995035648,"available":95777382400,"internally_reserved":0,"allocated":69861376,"data_stored":66355067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":147756192},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":591,"seq":2538325672142,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2892560,"kb_used_data":2529608,"kb_used_omap":41,"kb_used_meta":362902,"kb_avail":90852592,"statfs":{"total":95995035648,"available":93033054208,"internally_reserved":0,"allocated":2590318592,"data_stored":2586805157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42651,"internal_metadata":371612005},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,0,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":557,"seq":2392296784085,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3157836,"kb_used_data":2814660,"kb_used_omap":32,"kb_used_meta":343135,"kb_avail":90587316,"statfs":{"total":95995035648,"available":92761411584,"internally_reserved":0,"allocated":2882211840,"data_stored":2878701404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33752,"internal_metadata":351370280},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":524,"seq":2250562863323,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2854096,"kb_used_data":2556168,"kb_used_omap":41,"kb_used_meta":297814,"kb_avail":90891056,"statfs":{"total":95995035648,"available":93072441344,"internally_reserved":0,"allocated":2617516032,"data_stored":2614002081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42694,"internal_metadata":304961850},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2592477184,"data_stored":2592473112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2854424576,"data_stored":2854420504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2560163840,"data_stored":2560163840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":58851328,"data_stored":58851328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1915879424,"data_stored":1915879424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1573847040,"data_stored":1573847040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2026442752,"data_stored":2026438680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1910112256,"data_stored":1910112256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:46:48.475 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:46:48.563 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:48.585 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:48.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:48.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:48.641 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-13T14:46:48.687 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:48.721 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:48.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:48.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:48.825 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:48.850 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:48.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:48.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:48.852 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:48 smithi151 ceph-mon[104450]: pgmap v1584: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 565 op/s; 1714/229299 objects degraded (0.747%); 44904/229299 objects misplaced (19.583%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:46:48.852 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:48 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1237377250' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:46:48.852 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:48 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:46:48.853 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:48 smithi151 ceph-mon[108966]: pgmap v1584: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 565 op/s; 1714/229299 objects degraded (0.747%); 44904/229299 objects misplaced (19.583%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:46:48.853 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:48 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1237377250' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:46:48.853 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:48 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:46:48.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:48.987 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:48.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:48.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:49.089 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:49.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:48 smithi184 ceph-mon[92527]: pgmap v1584: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 565 op/s; 1714/229299 objects degraded (0.747%); 44904/229299 objects misplaced (19.583%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:46:49.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:48 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1237377250' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:46:49.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:48 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:46:49.140 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:49.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:49.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:49.242 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:49.267 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:49.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:49.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:49.369 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:49.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:49.412 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:49.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:49.513 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:49.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:49.571 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:49.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:49.672 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:49.694 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:49.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:49.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:49.795 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:49.820 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:49.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:49.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:49.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:49 smithi151 ceph-mon[104450]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:49.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:49 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:46:49.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:49 smithi151 ceph-mon[104450]: osdmap e1158: 8 total, 8 up, 8 in 2024-09-13T14:46:49.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:49 smithi151 ceph-mon[108966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:49.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:49 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:46:49.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:49 smithi151 ceph-mon[108966]: osdmap e1158: 8 total, 8 up, 8 in 2024-09-13T14:46:49.922 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:49.944 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:49.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:49.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:50.046 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:50.072 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:50.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:50.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:50.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:49 smithi184 ceph-mon[92527]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:50.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:49 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:46:50.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:49 smithi184 ceph-mon[92527]: osdmap e1158: 8 total, 8 up, 8 in 2024-09-13T14:46:50.173 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:50.215 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:50.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:50.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:50.317 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:50.357 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:50.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:50.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:50.459 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:50.493 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:50.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:50.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:50.595 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:50.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:46:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:46:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:46:50.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:50.635 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:50.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:50.670 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:50.736 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:50.758 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:50.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:50.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:50.859 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:50.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:50.910 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:50.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:50.954 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:50 smithi151 ceph-mon[104450]: pgmap v1586: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 613 op/s; 1714/222687 objects degraded (0.770%); 43790/222687 objects misplaced (19.664%); 832 KiB/s, 12 objects/s recovering 2024-09-13T14:46:50.955 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:50 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 1714/222687 objects degraded (0.770%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:46:50.955 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:50 smithi151 ceph-mon[108966]: pgmap v1586: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 613 op/s; 1714/222687 objects degraded (0.770%); 43790/222687 objects misplaced (19.664%); 832 KiB/s, 12 objects/s recovering 2024-09-13T14:46:50.955 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:50 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 1714/222687 objects degraded (0.770%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:46:51.011 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:51.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:51.065 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:51.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:51.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:50 smithi184 ceph-mon[92527]: pgmap v1586: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 613 op/s; 1714/222687 objects degraded (0.770%); 43790/222687 objects misplaced (19.664%); 832 KiB/s, 12 objects/s recovering 2024-09-13T14:46:51.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:50 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 1714/222687 objects degraded (0.770%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:46:51.166 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:51.189 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:51.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:51.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:51.290 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:51.330 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:51.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:51.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:51.432 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:51.456 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:51.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:51.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:51.558 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:51.582 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:51.582 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:51.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:51.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:51.621 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:46:51.621 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:46:51.684 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:51.706 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:51.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:51.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:51.807 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:51.830 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:51.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:51.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:51.932 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:51.957 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:51.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:51.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:52.058 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:52.101 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:52.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:52.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:52.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:52.249 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:52.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:52.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:52.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:52.395 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:52.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:52.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:52.497 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:52.521 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:52.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:52.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:52.623 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:52.666 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:52.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:52.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:52.724 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:46:52.769 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:52.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:52.809 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:52.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:52.911 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:52.934 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:52.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:52.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:53.036 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:53.060 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:53.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:53.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:53.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:52 smithi184 ceph-mon[92527]: pgmap v1587: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 611 op/s; 1842/220329 objects degraded (0.836%); 43406/220329 objects misplaced (19.701%); 851 KiB/s, 13 objects/s recovering 2024-09-13T14:46:53.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:52 smithi151 ceph-mon[104450]: pgmap v1587: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 611 op/s; 1842/220329 objects degraded (0.836%); 43406/220329 objects misplaced (19.701%); 851 KiB/s, 13 objects/s recovering 2024-09-13T14:46:53.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:52 smithi151 ceph-mon[108966]: pgmap v1587: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 611 op/s; 1842/220329 objects degraded (0.836%); 43406/220329 objects misplaced (19.701%); 851 KiB/s, 13 objects/s recovering 2024-09-13T14:46:53.162 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:53.186 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:53.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:53.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:53.288 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:53.327 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:53.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:53.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:53.429 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:53.466 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:53.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:53.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:53.568 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:53.591 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:53.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:53.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:53.693 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:53.718 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:53.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:53.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:53.820 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:53.866 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:53.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:53.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:53.968 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:53.992 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:53.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:53.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:54.081 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T14:46:54.094 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:54.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:54.182 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:54.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:54.284 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:54.308 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:54.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:54.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:54.409 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:54.462 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:54.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:54.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:54.564 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:54.588 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:54.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:54.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:54.608 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:46:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:46:54] "GET /metrics HTTP/1.1" 200 36661 "" "Prometheus/2.51.0" 2024-09-13T14:46:54.690 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:54.711 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:54.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:54.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:54.813 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:54.837 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:54.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:54.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:54.938 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:54.963 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:54.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:54.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:55.021 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:54 smithi151 ceph-mon[104450]: pgmap v1588: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 607 op/s; 1842/216210 objects degraded (0.852%); 42833/216210 objects misplaced (19.811%); 851 KiB/s, 13 objects/s recovering 2024-09-13T14:46:55.021 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:54 smithi151 ceph-mon[108966]: pgmap v1588: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 607 op/s; 1842/216210 objects degraded (0.852%); 42833/216210 objects misplaced (19.811%); 851 KiB/s, 13 objects/s recovering 2024-09-13T14:46:55.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:55.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:54 smithi184 ceph-mon[92527]: pgmap v1588: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 607 op/s; 1842/216210 objects degraded (0.852%); 42833/216210 objects misplaced (19.811%); 851 KiB/s, 13 objects/s recovering 2024-09-13T14:46:55.114 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:55.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:55.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:55.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:55.250 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:55.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:55.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:55.352 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:55.374 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:55.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:55.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:55.476 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:55.513 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:55.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:55.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:55.615 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:55.639 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:55.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:55.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:55.741 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:55.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:55.776 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:55.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:55.877 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:55.902 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:55.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:55.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:56.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:56.034 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:56.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:56.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:56.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:55 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 1842/216210 objects degraded (0.852%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:46:56.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:55 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 1842/216210 objects degraded (0.852%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:46:56.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:55 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 1842/216210 objects degraded (0.852%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:46:56.137 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:56.170 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:56.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:56.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:56.272 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:56.296 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:56.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:56.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:56.398 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:56.438 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:56.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:56.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:56.540 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:56.574 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:56.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:56.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:56.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:56.697 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:56.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:56.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:56.799 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:56.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:56.832 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:56.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:56.933 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:56.957 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:56.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:56.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:57.059 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:57.066 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:56 smithi151 ceph-mon[104450]: pgmap v1589: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.3 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 610 op/s; 1842/213264 objects degraded (0.864%); 42276/213264 objects misplaced (19.823%); 851 KiB/s, 13 objects/s recovering 2024-09-13T14:46:57.066 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:56 smithi151 ceph-mon[108966]: pgmap v1589: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.3 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 610 op/s; 1842/213264 objects degraded (0.864%); 42276/213264 objects misplaced (19.823%); 851 KiB/s, 13 objects/s recovering 2024-09-13T14:46:57.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:56 smithi184 ceph-mon[92527]: pgmap v1589: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.3 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 610 op/s; 1842/213264 objects degraded (0.864%); 42276/213264 objects misplaced (19.823%); 851 KiB/s, 13 objects/s recovering 2024-09-13T14:46:57.105 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:57.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:57.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:57.134 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:46:57.134 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:46:57.134 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:46:57.134 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:46:57.145 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:46:57.145 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:46:57.194 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:46:57.194 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:46:57.207 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:57.227 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:46:57.227 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:46:57.251 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:57.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:57.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:57.262 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:46:57.263 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:46:57.293 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:46:57.293 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:46:57.324 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:46:57.324 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:46:57.353 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:57.394 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:57.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:57.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:57.496 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:57.520 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:57.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:57.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:57.622 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:57.666 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:57.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:57.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:57.767 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:57.793 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:57.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:57.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:57.895 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:57.920 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:57.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:57.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:57.992 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:57 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/4224702994' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:57.992 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:57 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/4224702994' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:58.022 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:58.044 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:58.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:58.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:58.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:57 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/4224702994' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:58.146 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:58.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:58.209 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:58.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:58.310 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:58.349 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:58.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:58.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:58.451 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:58.475 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:58.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:58.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:58.577 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:58.622 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:58.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:58.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:58.696 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:58.700 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:58.724 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:58.746 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:58.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:58.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:58.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:46:58 smithi151 ceph-mon[108966]: pgmap v1590: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.3 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 610 op/s; 1971/210963 objects degraded (0.934%); 41889/210963 objects misplaced (19.856%); 883 KiB/s, 13 objects/s recovering 2024-09-13T14:46:58.840 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:46:58 smithi151 ceph-mon[104450]: pgmap v1590: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.3 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 610 op/s; 1971/210963 objects degraded (0.934%); 41889/210963 objects misplaced (19.856%); 883 KiB/s, 13 objects/s recovering 2024-09-13T14:46:58.848 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:58.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:58.889 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:58.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:58.990 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:59.014 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:59.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:59.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:46:58 smithi184 ceph-mon[92527]: pgmap v1590: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.3 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 610 op/s; 1971/210963 objects degraded (0.934%); 41889/210963 objects misplaced (19.856%); 883 KiB/s, 13 objects/s recovering 2024-09-13T14:46:59.116 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:59.163 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:59.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:59.265 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:59.288 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:59.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:59.390 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:59.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.434 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:59.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:59.535 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:59.579 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:59.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:59.680 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:59.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.723 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:59.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:59.824 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:59.848 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:46:59.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:59.913 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:46:59.917 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T14:46:59.950 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:59.992 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:46:59.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:00.094 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:00.117 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:00.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:00.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:00.219 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:00.242 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:00.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:00.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:00.344 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:00.376 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:00.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:00.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:00.478 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:00.513 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:00.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:00.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:00.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:47:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:47:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:47:00.615 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:00.634 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:47:00.639 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:00.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:00.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:00.741 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:00.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:00.813 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:00.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:00.914 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:00.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:00.970 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:00.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:01.071 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:01.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:00 smithi184 ceph-mon[92527]: pgmap v1591: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 661 op/s; 1971/203775 objects degraded (0.967%); 40716/203775 objects misplaced (19.981%); 814 KiB/s, 12 objects/s recovering 2024-09-13T14:47:01.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:00 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 1971/203775 objects degraded (0.967%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:01.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:00 smithi184 ceph-mon[92527]: from='client.55117 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:47:01.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:00 smithi151 ceph-mon[108966]: pgmap v1591: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 661 op/s; 1971/203775 objects degraded (0.967%); 40716/203775 objects misplaced (19.981%); 814 KiB/s, 12 objects/s recovering 2024-09-13T14:47:01.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:00 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 1971/203775 objects degraded (0.967%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:01.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:00 smithi151 ceph-mon[108966]: from='client.55117 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:47:01.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:00 smithi151 ceph-mon[104450]: pgmap v1591: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 661 op/s; 1971/203775 objects degraded (0.967%); 40716/203775 objects misplaced (19.981%); 814 KiB/s, 12 objects/s recovering 2024-09-13T14:47:01.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:00 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 1971/203775 objects degraded (0.967%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:01.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:00 smithi151 ceph-mon[104450]: from='client.55117 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:47:01.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:01.118 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:01.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:01.219 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:01.230 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1591,"stamp":"2024-09-13T14:46:59.476944+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4450386104,"num_objects":67925,"num_object_clones":0,"num_object_copies":203775,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1971,"num_objects_misplaced":40716,"num_objects_unfound":0,"num_objects_dirty":67925,"num_whiteouts":0,"num_read":14,"num_read_kb":35,"num_write":173089,"num_write_kb":7709418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1263,"num_bytes_recovered":82209017,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115500,"ondisk_log_size":115500,"up":357,"acting":356,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":366,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15834788,"kb_used_data":13446308,"kb_used_omap":149,"kb_used_meta":2388138,"kb_avail":734126428,"statfs":{"total":767960285184,"available":751745462272,"internally_reserved":0,"allocated":13769019392,"data_stored":13740822535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":153401,"internal_metadata":2445453511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-470155264,"num_objects":-7174,"num_object_clones":0,"num_object_copies":-21522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":257,"num_objects_misplaced":-3599,"num_objects_unfound":0,"num_objects_dirty":-7174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7174,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":9043968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.843979"},"pg_stats":[{"pgid":"1.1d","version":"261'100","reported_seq":86,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.650851+0000","last_change":"2024-09-13T14:45:19.807356+0000","last_active":"2024-09-13T14:46:48.650851+0000","last_peered":"2024-09-13T14:46:48.650851+0000","last_clean":"2024-09-13T14:46:48.650851+0000","last_became_active":"2024-09-13T14:45:19.807081+0000","last_became_peered":"2024-09-13T14:45:19.807081+0000","last_unstale":"2024-09-13T14:46:48.650851+0000","last_undegraded":"2024-09-13T14:46:48.650851+0000","last_fullsized":"2024-09-13T14:46:48.650851+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:17.988345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:17.988345+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:41:05.216335+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":1691,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.660616+0000","last_change":"2024-09-13T14:29:42.541263+0000","last_active":"2024-09-13T14:46:48.660616+0000","last_peered":"2024-09-13T14:46:48.660616+0000","last_clean":"2024-09-13T14:46:48.660616+0000","last_became_active":"2024-09-13T14:29:42.540873+0000","last_became_peered":"2024-09-13T14:29:42.540873+0000","last_unstale":"2024-09-13T14:46:48.660616+0000","last_undegraded":"2024-09-13T14:46:48.660616+0000","last_fullsized":"2024-09-13T14:46:48.660616+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:20.611206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1118,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.572678+0000","last_change":"2024-09-13T14:43:07.516715+0000","last_active":"2024-09-13T14:46:27.572678+0000","last_peered":"2024-09-13T14:46:27.572678+0000","last_clean":"2024-09-13T14:46:27.572678+0000","last_became_active":"2024-09-13T14:43:07.516403+0000","last_became_peered":"2024-09-13T14:43:07.516403+0000","last_unstale":"2024-09-13T14:46:27.572678+0000","last_undegraded":"2024-09-13T14:46:27.572678+0000","last_fullsized":"2024-09-13T14:46:27.572678+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:56.469690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":30,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.668224+0000","last_change":"2024-09-13T14:45:19.810203+0000","last_active":"2024-09-13T14:46:48.668224+0000","last_peered":"2024-09-13T14:46:48.668224+0000","last_clean":"2024-09-13T14:46:48.668224+0000","last_became_active":"2024-09-13T14:45:19.809693+0000","last_became_peered":"2024-09-13T14:45:19.809693+0000","last_unstale":"2024-09-13T14:46:48.668224+0000","last_undegraded":"2024-09-13T14:46:48.668224+0000","last_fullsized":"2024-09-13T14:46:48.668224+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:20.219673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":30,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.669107+0000","last_change":"2024-09-13T14:45:19.807991+0000","last_active":"2024-09-13T14:46:48.669107+0000","last_peered":"2024-09-13T14:46:48.669107+0000","last_clean":"2024-09-13T14:46:48.669107+0000","last_became_active":"2024-09-13T14:45:19.807662+0000","last_became_peered":"2024-09-13T14:45:19.807662+0000","last_unstale":"2024-09-13T14:46:48.669107+0000","last_undegraded":"2024-09-13T14:46:48.669107+0000","last_fullsized":"2024-09-13T14:46:48.669107+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:29.306108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"261'100","reported_seq":1229,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.554051+0000","last_change":"2024-09-13T14:44:54.259562+0000","last_active":"2024-09-13T14:46:27.554051+0000","last_peered":"2024-09-13T14:46:27.554051+0000","last_clean":"2024-09-13T14:46:27.554051+0000","last_became_active":"2024-09-13T14:43:14.563717+0000","last_became_peered":"2024-09-13T14:43:14.563717+0000","last_unstale":"2024-09-13T14:46:27.554051+0000","last_undegraded":"2024-09-13T14:46:27.554051+0000","last_fullsized":"2024-09-13T14:46:27.554051+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_clean_scrub_stamp":"2024-09-13T14:44:54.259482+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:43:52.729109+0000","scrub_duration":0.015265569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":1086,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.577449+0000","last_change":"2024-09-13T14:19:25.873514+0000","last_active":"2024-09-13T14:46:27.577449+0000","last_peered":"2024-09-13T14:46:27.577449+0000","last_clean":"2024-09-13T14:46:27.577449+0000","last_became_active":"2024-09-13T14:19:25.873170+0000","last_became_peered":"2024-09-13T14:19:25.873170+0000","last_unstale":"2024-09-13T14:46:27.577449+0000","last_undegraded":"2024-09-13T14:46:27.577449+0000","last_fullsized":"2024-09-13T14:46:27.577449+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:48.298530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1115,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.572964+0000","last_change":"2024-09-13T14:30:15.580011+0000","last_active":"2024-09-13T14:46:27.572964+0000","last_peered":"2024-09-13T14:46:27.572964+0000","last_clean":"2024-09-13T14:46:27.572964+0000","last_became_active":"2024-09-13T14:30:15.579610+0000","last_became_peered":"2024-09-13T14:30:15.579610+0000","last_unstale":"2024-09-13T14:46:27.572964+0000","last_undegraded":"2024-09-13T14:46:27.572964+0000","last_fullsized":"2024-09-13T14:46:27.572964+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:11:02.733771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"261'100","reported_seq":47,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.668415+0000","last_change":"2024-09-13T14:45:35.947332+0000","last_active":"2024-09-13T14:46:48.668415+0000","last_peered":"2024-09-13T14:46:48.668415+0000","last_clean":"2024-09-13T14:46:48.668415+0000","last_became_active":"2024-09-13T14:45:19.810775+0000","last_became_peered":"2024-09-13T14:45:19.810775+0000","last_unstale":"2024-09-13T14:46:48.668415+0000","last_undegraded":"2024-09-13T14:46:48.668415+0000","last_fullsized":"2024-09-13T14:46:48.668415+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:35.947265+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:43:41.569795+0000","last_clean_scrub_stamp":"2024-09-13T14:45:35.947265+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:00.606370+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":1131,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.577306+0000","last_change":"2024-09-13T14:43:07.536116+0000","last_active":"2024-09-13T14:46:27.577306+0000","last_peered":"2024-09-13T14:46:27.577306+0000","last_clean":"2024-09-13T14:46:27.577306+0000","last_became_active":"2024-09-13T14:43:07.524112+0000","last_became_peered":"2024-09-13T14:43:07.524112+0000","last_unstale":"2024-09-13T14:46:27.577306+0000","last_undegraded":"2024-09-13T14:46:27.577306+0000","last_fullsized":"2024-09-13T14:46:27.577306+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:36.947060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1128,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.573212+0000","last_change":"2024-09-13T14:43:07.530236+0000","last_active":"2024-09-13T14:46:27.573212+0000","last_peered":"2024-09-13T14:46:27.573212+0000","last_clean":"2024-09-13T14:46:27.573212+0000","last_became_active":"2024-09-13T14:43:07.529967+0000","last_became_peered":"2024-09-13T14:43:07.529967+0000","last_unstale":"2024-09-13T14:46:27.573212+0000","last_undegraded":"2024-09-13T14:46:27.573212+0000","last_fullsized":"2024-09-13T14:46:27.573212+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:27.276550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":1676,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.648610+0000","last_change":"2024-09-13T14:28:35.612469+0000","last_active":"2024-09-13T14:46:48.648610+0000","last_peered":"2024-09-13T14:46:48.648610+0000","last_clean":"2024-09-13T14:46:48.648610+0000","last_became_active":"2024-09-13T14:28:35.612093+0000","last_became_peered":"2024-09-13T14:28:35.612093+0000","last_unstale":"2024-09-13T14:46:48.648610+0000","last_undegraded":"2024-09-13T14:46:48.648610+0000","last_fullsized":"2024-09-13T14:46:48.648610+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:51.022663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1698,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.641743+0000","last_change":"2024-09-13T14:45:19.808811+0000","last_active":"2024-09-13T14:46:48.641743+0000","last_peered":"2024-09-13T14:46:48.641743+0000","last_clean":"2024-09-13T14:46:48.641743+0000","last_became_active":"2024-09-13T14:45:19.808471+0000","last_became_peered":"2024-09-13T14:45:19.808471+0000","last_unstale":"2024-09-13T14:46:48.641743+0000","last_undegraded":"2024-09-13T14:46:48.641743+0000","last_fullsized":"2024-09-13T14:46:48.641743+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:53.638161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"261'100","reported_seq":30,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.669195+0000","last_change":"2024-09-13T14:45:19.813412+0000","last_active":"2024-09-13T14:46:48.669195+0000","last_peered":"2024-09-13T14:46:48.669195+0000","last_clean":"2024-09-13T14:46:48.669195+0000","last_became_active":"2024-09-13T14:45:19.812585+0000","last_became_peered":"2024-09-13T14:45:19.812585+0000","last_unstale":"2024-09-13T14:46:48.669195+0000","last_undegraded":"2024-09-13T14:46:48.669195+0000","last_fullsized":"2024-09-13T14:46:48.669195+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.614340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.614340+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:32:07.434109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":1137,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.572994+0000","last_change":"2024-09-13T14:43:07.522950+0000","last_active":"2024-09-13T14:46:27.572994+0000","last_peered":"2024-09-13T14:46:27.572994+0000","last_clean":"2024-09-13T14:46:27.572994+0000","last_became_active":"2024-09-13T14:43:07.522566+0000","last_became_peered":"2024-09-13T14:43:07.522566+0000","last_unstale":"2024-09-13T14:46:27.572994+0000","last_undegraded":"2024-09-13T14:46:27.572994+0000","last_fullsized":"2024-09-13T14:46:27.572994+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:03.201478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":30,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.666738+0000","last_change":"2024-09-13T14:45:19.807231+0000","last_active":"2024-09-13T14:46:48.666738+0000","last_peered":"2024-09-13T14:46:48.666738+0000","last_clean":"2024-09-13T14:46:48.666738+0000","last_became_active":"2024-09-13T14:45:19.806936+0000","last_became_peered":"2024-09-13T14:45:19.806936+0000","last_unstale":"2024-09-13T14:46:48.666738+0000","last_undegraded":"2024-09-13T14:46:48.666738+0000","last_fullsized":"2024-09-13T14:46:48.666738+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:04:26.723153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.18","version":"261'100","reported_seq":107,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.577593+0000","last_change":"2024-09-13T14:44:41.685537+0000","last_active":"2024-09-13T14:46:27.577593+0000","last_peered":"2024-09-13T14:46:27.577593+0000","last_clean":"2024-09-13T14:46:27.577593+0000","last_became_active":"2024-09-13T14:43:14.558502+0000","last_became_peered":"2024-09-13T14:43:14.558502+0000","last_unstale":"2024-09-13T14:46:27.577593+0000","last_undegraded":"2024-09-13T14:46:27.577593+0000","last_fullsized":"2024-09-13T14:46:27.577593+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1134,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:41.685124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:41.685124+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:37:58.412497+0000","scrub_duration":0.0075205970000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":79,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.655316+0000","last_change":"2024-09-13T14:43:07.516845+0000","last_active":"2024-09-13T14:46:48.655316+0000","last_peered":"2024-09-13T14:46:48.655316+0000","last_clean":"2024-09-13T14:46:48.655316+0000","last_became_active":"2024-09-13T14:43:07.514974+0000","last_became_peered":"2024-09-13T14:43:07.514974+0000","last_unstale":"2024-09-13T14:46:48.655316+0000","last_undegraded":"2024-09-13T14:46:48.655316+0000","last_fullsized":"2024-09-13T14:46:48.655316+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:55.419483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1714,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.650197+0000","last_change":"2024-09-13T14:45:19.815341+0000","last_active":"2024-09-13T14:46:48.650197+0000","last_peered":"2024-09-13T14:46:48.650197+0000","last_clean":"2024-09-13T14:46:48.650197+0000","last_became_active":"2024-09-13T14:45:19.810262+0000","last_became_peered":"2024-09-13T14:45:19.810262+0000","last_unstale":"2024-09-13T14:46:48.650197+0000","last_undegraded":"2024-09-13T14:46:48.650197+0000","last_fullsized":"2024-09-13T14:46:48.650197+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:39.847653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1b","version":"261'100","reported_seq":49,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.579304+0000","last_change":"2024-09-13T14:44:52.803330+0000","last_active":"2024-09-13T14:46:27.579304+0000","last_peered":"2024-09-13T14:46:27.579304+0000","last_clean":"2024-09-13T14:46:27.579304+0000","last_became_active":"2024-09-13T14:43:14.556712+0000","last_became_peered":"2024-09-13T14:43:14.556712+0000","last_unstale":"2024-09-13T14:46:27.579304+0000","last_undegraded":"2024-09-13T14:46:27.579304+0000","last_fullsized":"2024-09-13T14:46:27.579304+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:52.803242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:52.803242+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:06.658301+0000","scrub_duration":0.0019769660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":30,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.663366+0000","last_change":"2024-09-13T14:45:19.806623+0000","last_active":"2024-09-13T14:46:48.663366+0000","last_peered":"2024-09-13T14:46:48.663366+0000","last_clean":"2024-09-13T14:46:48.663366+0000","last_became_active":"2024-09-13T14:45:19.806219+0000","last_became_peered":"2024-09-13T14:45:19.806219+0000","last_unstale":"2024-09-13T14:46:48.663366+0000","last_undegraded":"2024-09-13T14:46:48.663366+0000","last_fullsized":"2024-09-13T14:46:48.663366+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:34:41.601201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1111,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.577862+0000","last_change":"2024-09-13T14:43:32.637751+0000","last_active":"2024-09-13T14:46:27.577862+0000","last_peered":"2024-09-13T14:46:27.577862+0000","last_clean":"2024-09-13T14:46:27.577862+0000","last_became_active":"2024-09-13T14:43:32.636901+0000","last_became_peered":"2024-09-13T14:43:32.636901+0000","last_unstale":"2024-09-13T14:46:27.577862+0000","last_undegraded":"2024-09-13T14:46:27.577862+0000","last_fullsized":"2024-09-13T14:46:27.577862+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:21.942419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1a","version":"261'100","reported_seq":49,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.572900+0000","last_change":"2024-09-13T14:43:41.910041+0000","last_active":"2024-09-13T14:46:27.572900+0000","last_peered":"2024-09-13T14:46:27.572900+0000","last_clean":"2024-09-13T14:46:27.572900+0000","last_became_active":"2024-09-13T14:43:14.555083+0000","last_became_peered":"2024-09-13T14:43:14.555083+0000","last_unstale":"2024-09-13T14:46:27.572900+0000","last_undegraded":"2024-09-13T14:46:27.572900+0000","last_fullsized":"2024-09-13T14:46:27.572900+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:41.909988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:41.909988+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:42:19.954838+0000","scrub_duration":0.0012891619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":30,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.663577+0000","last_change":"2024-09-13T14:45:19.812071+0000","last_active":"2024-09-13T14:46:48.663577+0000","last_peered":"2024-09-13T14:46:48.663577+0000","last_clean":"2024-09-13T14:46:48.663577+0000","last_became_active":"2024-09-13T14:45:19.811336+0000","last_became_peered":"2024-09-13T14:45:19.811336+0000","last_unstale":"2024-09-13T14:46:48.663577+0000","last_undegraded":"2024-09-13T14:46:48.663577+0000","last_fullsized":"2024-09-13T14:46:48.663577+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:40:29.814608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1731,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.660504+0000","last_change":"2024-09-13T14:45:19.810478+0000","last_active":"2024-09-13T14:46:48.660504+0000","last_peered":"2024-09-13T14:46:48.660504+0000","last_clean":"2024-09-13T14:46:48.660504+0000","last_became_active":"2024-09-13T14:45:19.808278+0000","last_became_peered":"2024-09-13T14:45:19.808278+0000","last_unstale":"2024-09-13T14:46:48.660504+0000","last_undegraded":"2024-09-13T14:46:48.660504+0000","last_fullsized":"2024-09-13T14:46:48.660504+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:12.211089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"261'100","reported_seq":49,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.553710+0000","last_change":"2024-09-13T14:44:38.214961+0000","last_active":"2024-09-13T14:46:27.553710+0000","last_peered":"2024-09-13T14:46:27.553710+0000","last_clean":"2024-09-13T14:46:27.553710+0000","last_became_active":"2024-09-13T14:43:14.557883+0000","last_became_peered":"2024-09-13T14:43:14.557883+0000","last_unstale":"2024-09-13T14:46:27.553710+0000","last_undegraded":"2024-09-13T14:46:27.553710+0000","last_fullsized":"2024-09-13T14:46:27.553710+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:38.214865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:38.214865+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:49:36.013783+0000","scrub_duration":0.0016389639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1602,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.669345+0000","last_change":"2024-09-13T14:30:15.576921+0000","last_active":"2024-09-13T14:46:48.669345+0000","last_peered":"2024-09-13T14:46:48.669345+0000","last_clean":"2024-09-13T14:46:48.669345+0000","last_became_active":"2024-09-13T14:30:15.576607+0000","last_became_peered":"2024-09-13T14:30:15.576607+0000","last_unstale":"2024-09-13T14:46:48.669345+0000","last_undegraded":"2024-09-13T14:46:48.669345+0000","last_fullsized":"2024-09-13T14:46:48.669345+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:35.162460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1121,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.577681+0000","last_change":"2024-09-13T14:43:07.518801+0000","last_active":"2024-09-13T14:46:27.577681+0000","last_peered":"2024-09-13T14:46:27.577681+0000","last_clean":"2024-09-13T14:46:27.577681+0000","last_became_active":"2024-09-13T14:43:07.517610+0000","last_became_peered":"2024-09-13T14:43:07.517610+0000","last_unstale":"2024-09-13T14:46:27.577681+0000","last_undegraded":"2024-09-13T14:46:27.577681+0000","last_fullsized":"2024-09-13T14:46:27.577681+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:51.632254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"261'100","reported_seq":79,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.641307+0000","last_change":"2024-09-13T14:43:15.037565+0000","last_active":"2024-09-13T14:46:48.641307+0000","last_peered":"2024-09-13T14:46:48.641307+0000","last_clean":"2024-09-13T14:46:48.641307+0000","last_became_active":"2024-09-13T14:43:14.554183+0000","last_became_peered":"2024-09-13T14:43:14.554183+0000","last_unstale":"2024-09-13T14:46:48.641307+0000","last_undegraded":"2024-09-13T14:46:48.641307+0000","last_fullsized":"2024-09-13T14:46:48.641307+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.037517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.037517+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:01:35.137195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":1115,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.553525+0000","last_change":"2024-09-13T14:30:15.579260+0000","last_active":"2024-09-13T14:46:27.553525+0000","last_peered":"2024-09-13T14:46:27.553525+0000","last_clean":"2024-09-13T14:46:27.553525+0000","last_became_active":"2024-09-13T14:30:15.578897+0000","last_became_peered":"2024-09-13T14:30:15.578897+0000","last_unstale":"2024-09-13T14:46:27.553525+0000","last_undegraded":"2024-09-13T14:46:27.553525+0000","last_fullsized":"2024-09-13T14:46:27.553525+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:56.737849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":79,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.662350+0000","last_change":"2024-09-13T14:43:07.521576+0000","last_active":"2024-09-13T14:46:48.662350+0000","last_peered":"2024-09-13T14:46:48.662350+0000","last_clean":"2024-09-13T14:46:48.662350+0000","last_became_active":"2024-09-13T14:43:07.520478+0000","last_became_peered":"2024-09-13T14:43:07.520478+0000","last_unstale":"2024-09-13T14:46:48.662350+0000","last_undegraded":"2024-09-13T14:46:48.662350+0000","last_fullsized":"2024-09-13T14:46:48.662350+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:53.410135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"261'100","reported_seq":375,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.577698+0000","last_change":"2024-09-13T14:44:49.775941+0000","last_active":"2024-09-13T14:46:27.577698+0000","last_peered":"2024-09-13T14:46:27.577698+0000","last_clean":"2024-09-13T14:46:27.577698+0000","last_became_active":"2024-09-13T14:43:14.560297+0000","last_became_peered":"2024-09-13T14:43:14.560297+0000","last_unstale":"2024-09-13T14:46:27.577698+0000","last_undegraded":"2024-09-13T14:46:27.577698+0000","last_fullsized":"2024-09-13T14:46:27.577698+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:49.775844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:49.775844+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:15:53.138661+0000","scrub_duration":0.012350274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":1104,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.574620+0000","last_change":"2024-09-13T14:19:25.870605+0000","last_active":"2024-09-13T14:46:27.574620+0000","last_peered":"2024-09-13T14:46:27.574620+0000","last_clean":"2024-09-13T14:46:27.574620+0000","last_became_active":"2024-09-13T14:19:25.870282+0000","last_became_peered":"2024-09-13T14:19:25.870282+0000","last_unstale":"2024-09-13T14:46:27.574620+0000","last_undegraded":"2024-09-13T14:46:27.574620+0000","last_fullsized":"2024-09-13T14:46:27.574620+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:52.514915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1118,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.573267+0000","last_change":"2024-09-13T14:43:07.526476+0000","last_active":"2024-09-13T14:46:27.573267+0000","last_peered":"2024-09-13T14:46:27.573267+0000","last_clean":"2024-09-13T14:46:27.573267+0000","last_became_active":"2024-09-13T14:43:07.525802+0000","last_became_peered":"2024-09-13T14:43:07.525802+0000","last_unstale":"2024-09-13T14:46:27.573267+0000","last_undegraded":"2024-09-13T14:46:27.573267+0000","last_fullsized":"2024-09-13T14:46:27.573267+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:23.154726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"261'100","reported_seq":86,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.650672+0000","last_change":"2024-09-13T14:45:19.807461+0000","last_active":"2024-09-13T14:46:48.650672+0000","last_peered":"2024-09-13T14:46:48.650672+0000","last_clean":"2024-09-13T14:46:48.650672+0000","last_became_active":"2024-09-13T14:45:19.807117+0000","last_became_peered":"2024-09-13T14:45:19.807117+0000","last_unstale":"2024-09-13T14:46:48.650672+0000","last_undegraded":"2024-09-13T14:46:48.650672+0000","last_fullsized":"2024-09-13T14:46:48.650672+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.955197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.955197+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:03.531555+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"1158'10870","reported_seq":10932,"reported_epoch":1158,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:46:57.572967+0000","last_change":"2024-09-13T14:45:21.161658+0000","last_active":"2024-09-13T14:46:57.572967+0000","last_peered":"2024-09-13T14:46:57.572967+0000","last_clean":"2024-09-13T14:45:19.622135+0000","last_became_active":"2024-09-13T14:45:21.090442+0000","last_became_peered":"2024-09-13T14:45:21.090442+0000","last_unstale":"2024-09-13T14:46:57.572967+0000","last_undegraded":"2024-09-13T14:46:57.572967+0000","last_fullsized":"2024-09-13T14:46:57.572967+0000","mapping_epoch":1149,"log_start":"1142'4800","ondisk_log_start":"1142'4800","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6070,"log_dups_size":0,"ondisk_log_size":6070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284164096,"num_objects":4336,"num_object_clones":0,"num_object_copies":13008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4336,"num_objects_unfound":0,"num_objects_dirty":4336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10870,"num_write_kb":486592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[1,2,7],"avail_no_missing":["1","2","7"],"object_location_counts":[{"shards":"1,2,7","objects":4336}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1081,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.573039+0000","last_change":"2024-09-13T14:19:26.131609+0000","last_active":"2024-09-13T14:46:27.573039+0000","last_peered":"2024-09-13T14:46:27.573039+0000","last_clean":"2024-09-13T14:46:27.573039+0000","last_became_active":"2024-09-13T14:19:26.131029+0000","last_became_peered":"2024-09-13T14:19:26.131029+0000","last_unstale":"2024-09-13T14:46:27.573039+0000","last_undegraded":"2024-09-13T14:46:27.573039+0000","last_fullsized":"2024-09-13T14:46:27.573039+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:57.933696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":583,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.554731+0000","last_change":"2024-09-13T14:43:32.635070+0000","last_active":"2024-09-13T14:46:27.554731+0000","last_peered":"2024-09-13T14:46:27.554731+0000","last_clean":"2024-09-13T14:46:27.554731+0000","last_became_active":"2024-09-13T14:43:32.634734+0000","last_became_peered":"2024-09-13T14:43:32.634734+0000","last_unstale":"2024-09-13T14:46:27.554731+0000","last_undegraded":"2024-09-13T14:46:27.554731+0000","last_fullsized":"2024-09-13T14:46:27.554731+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:33.107010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":1721,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.650367+0000","last_change":"2024-09-13T14:43:07.537759+0000","last_active":"2024-09-13T14:46:48.650367+0000","last_peered":"2024-09-13T14:46:48.650367+0000","last_clean":"2024-09-13T14:46:48.650367+0000","last_became_active":"2024-09-13T14:43:07.520303+0000","last_became_peered":"2024-09-13T14:43:07.520303+0000","last_unstale":"2024-09-13T14:46:48.650367+0000","last_undegraded":"2024-09-13T14:46:48.650367+0000","last_fullsized":"2024-09-13T14:46:48.650367+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:32.772570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"261'100","reported_seq":30,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.667435+0000","last_change":"2024-09-13T14:45:19.813527+0000","last_active":"2024-09-13T14:46:48.667435+0000","last_peered":"2024-09-13T14:46:48.667435+0000","last_clean":"2024-09-13T14:46:48.667435+0000","last_became_active":"2024-09-13T14:45:19.812834+0000","last_became_peered":"2024-09-13T14:45:19.812834+0000","last_unstale":"2024-09-13T14:46:48.667435+0000","last_undegraded":"2024-09-13T14:46:48.667435+0000","last_fullsized":"2024-09-13T14:46:48.667435+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.710192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.710192+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:08:55.152703+0000","scrub_duration":0.0019250910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"1158'10728","reported_seq":13220,"reported_epoch":1158,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:46:56.409130+0000","last_change":"2024-09-13T14:45:38.479094+0000","last_active":"2024-09-13T14:46:56.409130+0000","last_peered":"2024-09-13T14:46:56.409130+0000","last_clean":"2024-09-13T14:45:19.642017+0000","last_became_active":"2024-09-13T14:45:21.111093+0000","last_became_peered":"2024-09-13T14:45:21.111093+0000","last_unstale":"2024-09-13T14:46:56.409130+0000","last_undegraded":"2024-09-13T14:45:38.458706+0000","last_fullsized":"2024-09-13T14:45:20.800066+0000","mapping_epoch":1149,"log_start":"1141'2300","ondisk_log_start":"1141'2300","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":8428,"log_dups_size":0,"ondisk_log_size":8428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284950528,"num_objects":4348,"num_object_clones":0,"num_object_copies":13044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1971,"num_objects_misplaced":3905,"num_objects_unfound":0,"num_objects_dirty":4348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10728,"num_write_kb":482432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1219,"num_bytes_recovered":79888384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[2,4],"avail_no_missing":["2","4","6"],"object_location_counts":[{"shards":"2,4","objects":-1528},{"shards":"2,4,6","objects":3905}],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":30,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.663654+0000","last_change":"2024-09-13T14:45:19.813025+0000","last_active":"2024-09-13T14:46:48.663654+0000","last_peered":"2024-09-13T14:46:48.663654+0000","last_clean":"2024-09-13T14:46:48.663654+0000","last_became_active":"2024-09-13T14:45:19.812714+0000","last_became_peered":"2024-09-13T14:45:19.812714+0000","last_unstale":"2024-09-13T14:46:48.663654+0000","last_undegraded":"2024-09-13T14:46:48.663654+0000","last_fullsized":"2024-09-13T14:46:48.663654+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:40.671012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":79,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.650944+0000","last_change":"2024-09-13T14:43:07.522730+0000","last_active":"2024-09-13T14:46:48.650944+0000","last_peered":"2024-09-13T14:46:48.650944+0000","last_clean":"2024-09-13T14:46:48.650944+0000","last_became_active":"2024-09-13T14:43:07.517420+0000","last_became_peered":"2024-09-13T14:43:07.517420+0000","last_unstale":"2024-09-13T14:46:48.650944+0000","last_undegraded":"2024-09-13T14:46:48.650944+0000","last_fullsized":"2024-09-13T14:46:48.650944+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:21:30.265293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":1634,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.671166+0000","last_change":"2024-09-13T14:30:15.581728+0000","last_active":"2024-09-13T14:46:48.671166+0000","last_peered":"2024-09-13T14:46:48.671166+0000","last_clean":"2024-09-13T14:46:48.671166+0000","last_became_active":"2024-09-13T14:30:15.581303+0000","last_became_peered":"2024-09-13T14:30:15.581303+0000","last_unstale":"2024-09-13T14:46:48.671166+0000","last_undegraded":"2024-09-13T14:46:48.671166+0000","last_fullsized":"2024-09-13T14:46:48.671166+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:10.027991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"261'100","reported_seq":108,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.573358+0000","last_change":"2024-09-13T14:43:31.112173+0000","last_active":"2024-09-13T14:46:27.573358+0000","last_peered":"2024-09-13T14:46:27.573358+0000","last_clean":"2024-09-13T14:46:27.573358+0000","last_became_active":"2024-09-13T14:42:42.709263+0000","last_became_peered":"2024-09-13T14:42:42.709263+0000","last_unstale":"2024-09-13T14:46:27.573358+0000","last_undegraded":"2024-09-13T14:46:27.573358+0000","last_fullsized":"2024-09-13T14:46:27.573358+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:31.112111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:31.112111+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:02.707372+0000","scrub_duration":0.0015759260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"1158'10945","reported_seq":10987,"reported_epoch":1158,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:46:57.648029+0000","last_change":"2024-09-13T14:45:21.011501+0000","last_active":"2024-09-13T14:46:57.648029+0000","last_peered":"2024-09-13T14:46:57.648029+0000","last_clean":"2024-09-13T14:45:19.622974+0000","last_became_active":"2024-09-13T14:45:21.006075+0000","last_became_peered":"2024-09-13T14:45:21.006075+0000","last_unstale":"2024-09-13T14:46:57.648029+0000","last_undegraded":"2024-09-13T14:46:57.648029+0000","last_fullsized":"2024-09-13T14:46:57.648029+0000","mapping_epoch":1149,"log_start":"1142'3200","ondisk_log_start":"1142'3200","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":7745,"log_dups_size":0,"ondisk_log_size":7745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:30.090601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282656768,"num_objects":4313,"num_object_clones":0,"num_object_copies":12939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4313,"num_objects_unfound":0,"num_objects_dirty":4313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10945,"num_write_kb":488256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,2],"avail_no_missing":["5","1","2"],"object_location_counts":[{"shards":"1,2,5","objects":4313}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1674,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.648536+0000","last_change":"2024-09-13T14:28:35.611508+0000","last_active":"2024-09-13T14:46:48.648536+0000","last_peered":"2024-09-13T14:46:48.648536+0000","last_clean":"2024-09-13T14:46:48.648536+0000","last_became_active":"2024-09-13T14:28:35.610106+0000","last_became_peered":"2024-09-13T14:28:35.610106+0000","last_unstale":"2024-09-13T14:46:48.648536+0000","last_undegraded":"2024-09-13T14:46:48.648536+0000","last_fullsized":"2024-09-13T14:46:48.648536+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:11.966958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1124,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.553774+0000","last_change":"2024-09-13T14:28:35.603663+0000","last_active":"2024-09-13T14:46:27.553774+0000","last_peered":"2024-09-13T14:46:27.553774+0000","last_clean":"2024-09-13T14:46:27.553774+0000","last_became_active":"2024-09-13T14:28:35.603358+0000","last_became_peered":"2024-09-13T14:28:35.603358+0000","last_unstale":"2024-09-13T14:46:27.553774+0000","last_undegraded":"2024-09-13T14:46:27.553774+0000","last_fullsized":"2024-09-13T14:46:27.553774+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:24:05.842796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":79,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.655150+0000","last_change":"2024-09-13T14:43:07.522750+0000","last_active":"2024-09-13T14:46:48.655150+0000","last_peered":"2024-09-13T14:46:48.655150+0000","last_clean":"2024-09-13T14:46:48.655150+0000","last_became_active":"2024-09-13T14:43:07.517267+0000","last_became_peered":"2024-09-13T14:43:07.517267+0000","last_unstale":"2024-09-13T14:46:48.655150+0000","last_undegraded":"2024-09-13T14:46:48.655150+0000","last_fullsized":"2024-09-13T14:46:48.655150+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:40.217748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"261'100","reported_seq":400,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.650022+0000","last_change":"2024-09-13T14:45:19.815050+0000","last_active":"2024-09-13T14:46:48.650022+0000","last_peered":"2024-09-13T14:46:48.650022+0000","last_clean":"2024-09-13T14:46:48.650022+0000","last_became_active":"2024-09-13T14:45:19.809732+0000","last_became_peered":"2024-09-13T14:45:19.809732+0000","last_unstale":"2024-09-13T14:46:48.650022+0000","last_undegraded":"2024-09-13T14:46:48.650022+0000","last_fullsized":"2024-09-13T14:46:48.650022+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:33.574443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:33.574443+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:51:40.261036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"1158'10883","reported_seq":10983,"reported_epoch":1158,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:46:57.397077+0000","last_change":"2024-09-13T14:45:21.136109+0000","last_active":"2024-09-13T14:46:57.397077+0000","last_peered":"2024-09-13T14:46:57.397077+0000","last_clean":"2024-09-13T14:45:19.607954+0000","last_became_active":"2024-09-13T14:45:21.004300+0000","last_became_peered":"2024-09-13T14:45:21.004300+0000","last_unstale":"2024-09-13T14:46:57.397077+0000","last_undegraded":"2024-09-13T14:46:57.397077+0000","last_fullsized":"2024-09-13T14:46:57.397077+0000","mapping_epoch":1149,"log_start":"1142'4700","ondisk_log_start":"1142'4700","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6183,"log_dups_size":0,"ondisk_log_size":6183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:04:23.351582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276889600,"num_objects":4225,"num_object_clones":0,"num_object_copies":12675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4205,"num_objects_unfound":0,"num_objects_dirty":4225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10883,"num_write_kb":483456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":1835008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,6,1],"avail_no_missing":["4","1","6"],"object_location_counts":[{"shards":"1,4,6","objects":4225}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":79,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.662187+0000","last_change":"2024-09-13T14:43:07.520049+0000","last_active":"2024-09-13T14:46:48.662187+0000","last_peered":"2024-09-13T14:46:48.662187+0000","last_clean":"2024-09-13T14:46:48.662187+0000","last_became_active":"2024-09-13T14:43:07.514646+0000","last_became_peered":"2024-09-13T14:43:07.514646+0000","last_unstale":"2024-09-13T14:46:48.662187+0000","last_undegraded":"2024-09-13T14:46:48.662187+0000","last_fullsized":"2024-09-13T14:46:48.662187+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:10:35.200778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":79,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.655062+0000","last_change":"2024-09-13T14:43:07.525666+0000","last_active":"2024-09-13T14:46:48.655062+0000","last_peered":"2024-09-13T14:46:48.655062+0000","last_clean":"2024-09-13T14:46:48.655062+0000","last_became_active":"2024-09-13T14:43:07.524064+0000","last_became_peered":"2024-09-13T14:43:07.524064+0000","last_unstale":"2024-09-13T14:46:48.655062+0000","last_undegraded":"2024-09-13T14:46:48.655062+0000","last_fullsized":"2024-09-13T14:46:48.655062+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:10.270333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1138,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.577488+0000","last_change":"2024-09-13T14:45:19.804177+0000","last_active":"2024-09-13T14:46:27.577488+0000","last_peered":"2024-09-13T14:46:27.577488+0000","last_clean":"2024-09-13T14:46:27.577488+0000","last_became_active":"2024-09-13T14:45:19.803890+0000","last_became_peered":"2024-09-13T14:45:19.803890+0000","last_unstale":"2024-09-13T14:46:27.577488+0000","last_undegraded":"2024-09-13T14:46:27.577488+0000","last_fullsized":"2024-09-13T14:46:27.577488+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:25.140789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"261'100","reported_seq":47,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.671092+0000","last_change":"2024-09-13T14:45:36.965992+0000","last_active":"2024-09-13T14:46:48.671092+0000","last_peered":"2024-09-13T14:46:48.671092+0000","last_clean":"2024-09-13T14:46:48.671092+0000","last_became_active":"2024-09-13T14:45:19.811238+0000","last_became_peered":"2024-09-13T14:45:19.811238+0000","last_unstale":"2024-09-13T14:46:48.671092+0000","last_undegraded":"2024-09-13T14:46:48.671092+0000","last_fullsized":"2024-09-13T14:46:48.671092+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:36.965775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:45:36.965775+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:36:01.623251+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"1158'10733","reported_seq":10792,"reported_epoch":1158,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:46:59.284414+0000","last_change":"2024-09-13T14:45:21.036181+0000","last_active":"2024-09-13T14:46:59.284414+0000","last_peered":"2024-09-13T14:46:59.284414+0000","last_clean":"2024-09-13T14:45:19.587756+0000","last_became_active":"2024-09-13T14:45:21.029769+0000","last_became_peered":"2024-09-13T14:45:21.029769+0000","last_unstale":"2024-09-13T14:46:59.284414+0000","last_undegraded":"2024-09-13T14:46:59.284414+0000","last_fullsized":"2024-09-13T14:46:59.284414+0000","mapping_epoch":1149,"log_start":"1142'3900","ondisk_log_start":"1142'3900","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6833,"log_dups_size":0,"ondisk_log_size":6833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266403840,"num_objects":4065,"num_object_clones":0,"num_object_copies":12195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4065,"num_objects_unfound":0,"num_objects_dirty":4065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10733,"num_write_kb":473536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,1],"avail_no_missing":["0","1","6"],"object_location_counts":[{"shards":"0,1,6","objects":4065}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":37,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.577130+0000","last_change":"2024-09-13T14:45:19.806318+0000","last_active":"2024-09-13T14:46:27.577130+0000","last_peered":"2024-09-13T14:46:27.577130+0000","last_clean":"2024-09-13T14:46:27.577130+0000","last_became_active":"2024-09-13T14:45:19.806074+0000","last_became_peered":"2024-09-13T14:45:19.806074+0000","last_unstale":"2024-09-13T14:46:27.577130+0000","last_undegraded":"2024-09-13T14:46:27.577130+0000","last_fullsized":"2024-09-13T14:46:27.577130+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:21.679797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1099,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.573319+0000","last_change":"2024-09-13T14:28:35.604404+0000","last_active":"2024-09-13T14:46:27.573319+0000","last_peered":"2024-09-13T14:46:27.573319+0000","last_clean":"2024-09-13T14:46:27.573319+0000","last_became_active":"2024-09-13T14:28:35.604102+0000","last_became_peered":"2024-09-13T14:28:35.604102+0000","last_unstale":"2024-09-13T14:46:27.573319+0000","last_undegraded":"2024-09-13T14:46:27.573319+0000","last_fullsized":"2024-09-13T14:46:27.573319+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:25.546010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":1149,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.577352+0000","last_change":"2024-09-13T14:43:07.538919+0000","last_active":"2024-09-13T14:46:27.577352+0000","last_peered":"2024-09-13T14:46:27.577352+0000","last_clean":"2024-09-13T14:46:27.577352+0000","last_became_active":"2024-09-13T14:43:07.513109+0000","last_became_peered":"2024-09-13T14:43:07.513109+0000","last_unstale":"2024-09-13T14:46:27.577352+0000","last_undegraded":"2024-09-13T14:46:27.577352+0000","last_fullsized":"2024-09-13T14:46:27.577352+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:27.771720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"261'100","reported_seq":98,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.576830+0000","last_change":"2024-09-13T14:43:44.720333+0000","last_active":"2024-09-13T14:46:27.576830+0000","last_peered":"2024-09-13T14:46:27.576830+0000","last_clean":"2024-09-13T14:46:27.576830+0000","last_became_active":"2024-09-13T14:42:14.946591+0000","last_became_peered":"2024-09-13T14:42:14.946591+0000","last_unstale":"2024-09-13T14:46:27.576830+0000","last_undegraded":"2024-09-13T14:46:27.576830+0000","last_fullsized":"2024-09-13T14:46:27.576830+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:44.720258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:44.720258+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:33.281357+0000","scrub_duration":0.0018417920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"1158'10893","reported_seq":10949,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.401649+0000","last_change":"2024-09-13T14:43:50.794091+0000","last_active":"2024-09-13T14:46:59.401649+0000","last_peered":"2024-09-13T14:46:59.401649+0000","last_clean":"2024-09-13T14:46:59.401649+0000","last_became_active":"2024-09-13T14:43:50.793741+0000","last_became_peered":"2024-09-13T14:43:50.793741+0000","last_unstale":"2024-09-13T14:46:59.401649+0000","last_undegraded":"2024-09-13T14:46:59.401649+0000","last_fullsized":"2024-09-13T14:46:59.401649+0000","mapping_epoch":1136,"log_start":"1142'4000","ondisk_log_start":"1142'4000","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6893,"log_dups_size":0,"ondisk_log_size":6893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:15.846494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275054592,"num_objects":4197,"num_object_clones":0,"num_object_copies":12591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10893,"num_write_kb":482880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":47,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.642776+0000","last_change":"2024-09-13T14:44:03.858211+0000","last_active":"2024-09-13T14:46:48.642776+0000","last_peered":"2024-09-13T14:46:48.642776+0000","last_clean":"2024-09-13T14:46:48.642776+0000","last_became_active":"2024-09-13T14:44:03.857928+0000","last_became_peered":"2024-09-13T14:44:03.857928+0000","last_unstale":"2024-09-13T14:46:48.642776+0000","last_undegraded":"2024-09-13T14:46:48.642776+0000","last_fullsized":"2024-09-13T14:46:48.642776+0000","mapping_epoch":1140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:28.428693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":79,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.651271+0000","last_change":"2024-09-13T14:43:07.516243+0000","last_active":"2024-09-13T14:46:48.651271+0000","last_peered":"2024-09-13T14:46:48.651271+0000","last_clean":"2024-09-13T14:46:48.651271+0000","last_became_active":"2024-09-13T14:43:07.515863+0000","last_became_peered":"2024-09-13T14:43:07.515863+0000","last_unstale":"2024-09-13T14:46:48.651271+0000","last_undegraded":"2024-09-13T14:46:48.651271+0000","last_fullsized":"2024-09-13T14:46:48.651271+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:08.085298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1716,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.637461+0000","last_change":"2024-09-13T14:45:19.812713+0000","last_active":"2024-09-13T14:46:48.637461+0000","last_peered":"2024-09-13T14:46:48.637461+0000","last_clean":"2024-09-13T14:46:48.637461+0000","last_became_active":"2024-09-13T14:45:19.812203+0000","last_became_peered":"2024-09-13T14:45:19.812203+0000","last_unstale":"2024-09-13T14:46:48.637461+0000","last_undegraded":"2024-09-13T14:46:48.637461+0000","last_fullsized":"2024-09-13T14:46:48.637461+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:30:46.149347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"261'100","reported_seq":1278,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.553188+0000","last_change":"2024-09-13T14:43:42.577312+0000","last_active":"2024-09-13T14:46:27.553188+0000","last_peered":"2024-09-13T14:46:27.553188+0000","last_clean":"2024-09-13T14:46:27.553188+0000","last_became_active":"2024-09-13T14:42:14.928923+0000","last_became_peered":"2024-09-13T14:42:14.928923+0000","last_unstale":"2024-09-13T14:46:27.553188+0000","last_undegraded":"2024-09-13T14:46:27.553188+0000","last_fullsized":"2024-09-13T14:46:27.553188+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:42.577243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:42.577243+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:01.234250+0000","scrub_duration":0.001367839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"1158'10804","reported_seq":10834,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:57.475819+0000","last_change":"2024-09-13T14:43:50.793810+0000","last_active":"2024-09-13T14:46:57.475819+0000","last_peered":"2024-09-13T14:46:57.475819+0000","last_clean":"2024-09-13T14:46:57.475819+0000","last_became_active":"2024-09-13T14:43:50.793506+0000","last_became_peered":"2024-09-13T14:43:50.793506+0000","last_unstale":"2024-09-13T14:46:57.475819+0000","last_undegraded":"2024-09-13T14:46:57.475819+0000","last_fullsized":"2024-09-13T14:46:57.475819+0000","mapping_epoch":1136,"log_start":"1142'4600","ondisk_log_start":"1142'4600","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6204,"log_dups_size":0,"ondisk_log_size":6204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:50:32.061762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276824064,"num_objects":4224,"num_object_clones":0,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10804,"num_write_kb":480896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1710,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.650118+0000","last_change":"2024-09-13T14:43:07.520212+0000","last_active":"2024-09-13T14:46:48.650118+0000","last_peered":"2024-09-13T14:46:48.650118+0000","last_clean":"2024-09-13T14:46:48.650118+0000","last_became_active":"2024-09-13T14:43:07.519831+0000","last_became_peered":"2024-09-13T14:43:07.519831+0000","last_unstale":"2024-09-13T14:46:48.650118+0000","last_undegraded":"2024-09-13T14:46:48.650118+0000","last_fullsized":"2024-09-13T14:46:48.650118+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:20.990736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1732,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.645453+0000","last_change":"2024-09-13T14:45:19.809505+0000","last_active":"2024-09-13T14:46:48.645453+0000","last_peered":"2024-09-13T14:46:48.645453+0000","last_clean":"2024-09-13T14:46:48.645453+0000","last_became_active":"2024-09-13T14:45:19.809181+0000","last_became_peered":"2024-09-13T14:45:19.809181+0000","last_unstale":"2024-09-13T14:46:48.645453+0000","last_undegraded":"2024-09-13T14:46:48.645453+0000","last_fullsized":"2024-09-13T14:46:48.645453+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:24.239188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1161,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.553301+0000","last_change":"2024-09-13T14:43:07.516140+0000","last_active":"2024-09-13T14:46:27.553301+0000","last_peered":"2024-09-13T14:46:27.553301+0000","last_clean":"2024-09-13T14:46:27.553301+0000","last_became_active":"2024-09-13T14:43:07.515481+0000","last_became_peered":"2024-09-13T14:43:07.515481+0000","last_unstale":"2024-09-13T14:46:27.553301+0000","last_undegraded":"2024-09-13T14:46:27.553301+0000","last_fullsized":"2024-09-13T14:46:27.553301+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:43.706214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"261'100","reported_seq":91,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.651365+0000","last_change":"2024-09-13T14:43:16.008346+0000","last_active":"2024-09-13T14:46:48.651365+0000","last_peered":"2024-09-13T14:46:48.651365+0000","last_clean":"2024-09-13T14:46:48.651365+0000","last_became_active":"2024-09-13T14:43:07.523121+0000","last_became_peered":"2024-09-13T14:43:07.523121+0000","last_unstale":"2024-09-13T14:46:48.651365+0000","last_undegraded":"2024-09-13T14:46:48.651365+0000","last_fullsized":"2024-09-13T14:46:48.651365+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.008264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.008264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:58:35.564470+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"1158'10778","reported_seq":10838,"reported_epoch":1158,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:46:59.363834+0000","last_change":"2024-09-13T14:45:21.035848+0000","last_active":"2024-09-13T14:46:59.363834+0000","last_peered":"2024-09-13T14:46:59.363834+0000","last_clean":"2024-09-13T14:45:19.637667+0000","last_became_active":"2024-09-13T14:45:21.025308+0000","last_became_peered":"2024-09-13T14:45:21.025308+0000","last_unstale":"2024-09-13T14:46:59.363834+0000","last_undegraded":"2024-09-13T14:46:59.363834+0000","last_fullsized":"2024-09-13T14:46:59.363834+0000","mapping_epoch":1149,"log_start":"1142'3900","ondisk_log_start":"1142'3900","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6878,"log_dups_size":0,"ondisk_log_size":6878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279183360,"num_objects":4260,"num_object_clones":0,"num_object_copies":12780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4260,"num_objects_unfound":0,"num_objects_dirty":4260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10778,"num_write_kb":481216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,5],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":4260}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":40,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.573304+0000","last_change":"2024-09-13T14:43:32.636990+0000","last_active":"2024-09-13T14:46:27.573304+0000","last_peered":"2024-09-13T14:46:27.573304+0000","last_clean":"2024-09-13T14:46:27.573304+0000","last_became_active":"2024-09-13T14:43:32.636589+0000","last_became_peered":"2024-09-13T14:43:32.636589+0000","last_unstale":"2024-09-13T14:46:27.573304+0000","last_undegraded":"2024-09-13T14:46:27.573304+0000","last_fullsized":"2024-09-13T14:46:27.573304+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:36:25.129769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":30,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.666549+0000","last_change":"2024-09-13T14:45:19.810888+0000","last_active":"2024-09-13T14:46:48.666549+0000","last_peered":"2024-09-13T14:46:48.666549+0000","last_clean":"2024-09-13T14:46:48.666549+0000","last_became_active":"2024-09-13T14:45:19.810486+0000","last_became_peered":"2024-09-13T14:45:19.810486+0000","last_unstale":"2024-09-13T14:46:48.666549+0000","last_undegraded":"2024-09-13T14:46:48.666549+0000","last_fullsized":"2024-09-13T14:46:48.666549+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:03.886942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":1114,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.579352+0000","last_change":"2024-09-13T14:29:42.539455+0000","last_active":"2024-09-13T14:46:27.579352+0000","last_peered":"2024-09-13T14:46:27.579352+0000","last_clean":"2024-09-13T14:46:27.579352+0000","last_became_active":"2024-09-13T14:29:42.539068+0000","last_became_peered":"2024-09-13T14:29:42.539068+0000","last_unstale":"2024-09-13T14:46:27.579352+0000","last_undegraded":"2024-09-13T14:46:27.579352+0000","last_fullsized":"2024-09-13T14:46:27.579352+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:24:06.048855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034405999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"261'100","reported_seq":1269,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.553379+0000","last_change":"2024-09-13T14:44:28.186029+0000","last_active":"2024-09-13T14:46:27.553379+0000","last_peered":"2024-09-13T14:46:27.553379+0000","last_clean":"2024-09-13T14:46:27.553379+0000","last_became_active":"2024-09-13T14:42:14.927771+0000","last_became_peered":"2024-09-13T14:42:14.927771+0000","last_unstale":"2024-09-13T14:46:27.553379+0000","last_undegraded":"2024-09-13T14:46:27.553379+0000","last_fullsized":"2024-09-13T14:46:27.553379+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:28.185949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:28.185949+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:38.363961+0000","scrub_duration":0.0018071129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"1158'10651","reported_seq":10696,"reported_epoch":1158,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:46:56.380162+0000","last_change":"2024-09-13T14:45:21.030922+0000","last_active":"2024-09-13T14:46:56.380162+0000","last_peered":"2024-09-13T14:46:56.380162+0000","last_clean":"2024-09-13T14:45:19.630214+0000","last_became_active":"2024-09-13T14:45:21.021807+0000","last_became_peered":"2024-09-13T14:45:21.021807+0000","last_unstale":"2024-09-13T14:46:56.380162+0000","last_undegraded":"2024-09-13T14:46:56.380162+0000","last_fullsized":"2024-09-13T14:46:56.380162+0000","mapping_epoch":1149,"log_start":"1141'2300","ondisk_log_start":"1141'2300","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":8351,"log_dups_size":0,"ondisk_log_size":8351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281346048,"num_objects":4293,"num_object_clones":0,"num_object_copies":12879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4293,"num_objects_unfound":0,"num_objects_dirty":4293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10651,"num_write_kb":478208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[2,6,7],"avail_no_missing":["2","6","7"],"object_location_counts":[{"shards":"2,6,7","objects":4293}],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":79,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.658588+0000","last_change":"2024-09-13T14:43:07.520176+0000","last_active":"2024-09-13T14:46:48.658588+0000","last_peered":"2024-09-13T14:46:48.658588+0000","last_clean":"2024-09-13T14:46:48.658588+0000","last_became_active":"2024-09-13T14:43:07.516748+0000","last_became_peered":"2024-09-13T14:43:07.516748+0000","last_unstale":"2024-09-13T14:46:48.658588+0000","last_undegraded":"2024-09-13T14:46:48.658588+0000","last_fullsized":"2024-09-13T14:46:48.658588+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:52:04.392821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":30,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.663729+0000","last_change":"2024-09-13T14:45:19.812353+0000","last_active":"2024-09-13T14:46:48.663729+0000","last_peered":"2024-09-13T14:46:48.663729+0000","last_clean":"2024-09-13T14:46:48.663729+0000","last_became_active":"2024-09-13T14:45:19.811743+0000","last_became_peered":"2024-09-13T14:45:19.811743+0000","last_unstale":"2024-09-13T14:46:48.663729+0000","last_undegraded":"2024-09-13T14:46:48.663729+0000","last_fullsized":"2024-09-13T14:46:48.663729+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:43.728170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"261'100","reported_seq":98,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.577036+0000","last_change":"2024-09-13T14:44:34.814157+0000","last_active":"2024-09-13T14:46:27.577036+0000","last_peered":"2024-09-13T14:46:27.577036+0000","last_clean":"2024-09-13T14:46:27.577036+0000","last_became_active":"2024-09-13T14:42:42.707803+0000","last_became_peered":"2024-09-13T14:42:42.707803+0000","last_unstale":"2024-09-13T14:46:27.577036+0000","last_undegraded":"2024-09-13T14:46:27.577036+0000","last_fullsized":"2024-09-13T14:46:27.577036+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:34.814070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:34.814070+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:38.408561+0000","scrub_duration":0.01686414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"1158'10896","reported_seq":10953,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:57.579750+0000","last_change":"2024-09-13T14:43:51.534705+0000","last_active":"2024-09-13T14:46:57.579750+0000","last_peered":"2024-09-13T14:46:57.579750+0000","last_clean":"2024-09-13T14:46:57.579750+0000","last_became_active":"2024-09-13T14:43:51.534381+0000","last_became_peered":"2024-09-13T14:43:51.534381+0000","last_unstale":"2024-09-13T14:46:57.579750+0000","last_undegraded":"2024-09-13T14:46:57.579750+0000","last_fullsized":"2024-09-13T14:46:57.579750+0000","mapping_epoch":1136,"log_start":"1142'4800","ondisk_log_start":"1142'4800","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6096,"log_dups_size":0,"ondisk_log_size":6096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:37:22.385742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277020696,"num_objects":4228,"num_object_clones":0,"num_object_copies":12684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4228,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10896,"num_write_kb":483905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1128,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.573425+0000","last_change":"2024-09-13T14:44:03.855577+0000","last_active":"2024-09-13T14:46:27.573425+0000","last_peered":"2024-09-13T14:46:27.573425+0000","last_clean":"2024-09-13T14:46:27.573425+0000","last_became_active":"2024-09-13T14:44:03.855267+0000","last_became_peered":"2024-09-13T14:44:03.855267+0000","last_unstale":"2024-09-13T14:46:27.573425+0000","last_undegraded":"2024-09-13T14:46:27.573425+0000","last_fullsized":"2024-09-13T14:46:27.573425+0000","mapping_epoch":1140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:46.148814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1099,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.573277+0000","last_change":"2024-09-13T14:29:42.545815+0000","last_active":"2024-09-13T14:46:27.573277+0000","last_peered":"2024-09-13T14:46:27.573277+0000","last_clean":"2024-09-13T14:46:27.573277+0000","last_became_active":"2024-09-13T14:29:42.545508+0000","last_became_peered":"2024-09-13T14:29:42.545508+0000","last_unstale":"2024-09-13T14:46:27.573277+0000","last_undegraded":"2024-09-13T14:46:27.573277+0000","last_fullsized":"2024-09-13T14:46:27.573277+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:39.678381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"261'100","reported_seq":97,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.651095+0000","last_change":"2024-09-13T14:43:18.988482+0000","last_active":"2024-09-13T14:46:48.651095+0000","last_peered":"2024-09-13T14:46:48.651095+0000","last_clean":"2024-09-13T14:46:48.651095+0000","last_became_active":"2024-09-13T14:43:14.556754+0000","last_became_peered":"2024-09-13T14:43:14.556754+0000","last_unstale":"2024-09-13T14:46:48.651095+0000","last_undegraded":"2024-09-13T14:46:48.651095+0000","last_fullsized":"2024-09-13T14:46:48.651095+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:18.988405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:18.988405+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:45:01.726326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"1158'10615","reported_seq":10670,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:56.359976+0000","last_change":"2024-09-13T14:43:50.796182+0000","last_active":"2024-09-13T14:46:56.359976+0000","last_peered":"2024-09-13T14:46:56.359976+0000","last_clean":"2024-09-13T14:46:56.359976+0000","last_became_active":"2024-09-13T14:43:50.795875+0000","last_became_peered":"2024-09-13T14:43:50.795875+0000","last_unstale":"2024-09-13T14:46:56.359976+0000","last_undegraded":"2024-09-13T14:46:56.359976+0000","last_fullsized":"2024-09-13T14:46:56.359976+0000","mapping_epoch":1136,"log_start":"1141'2200","ondisk_log_start":"1141'2200","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":8415,"log_dups_size":0,"ondisk_log_size":8415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:56:34.308870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272433152,"num_objects":4157,"num_object_clones":0,"num_object_copies":12471,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10615,"num_write_kb":472704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1145,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.573447+0000","last_change":"2024-09-13T14:43:07.515418+0000","last_active":"2024-09-13T14:46:27.573447+0000","last_peered":"2024-09-13T14:46:27.573447+0000","last_clean":"2024-09-13T14:46:27.573447+0000","last_became_active":"2024-09-13T14:43:07.515113+0000","last_became_peered":"2024-09-13T14:43:07.515113+0000","last_unstale":"2024-09-13T14:46:27.573447+0000","last_undegraded":"2024-09-13T14:46:27.573447+0000","last_fullsized":"2024-09-13T14:46:27.573447+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:10:25.810604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1112,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.572859+0000","last_change":"2024-09-13T14:43:32.636334+0000","last_active":"2024-09-13T14:46:27.572859+0000","last_peered":"2024-09-13T14:46:27.572859+0000","last_clean":"2024-09-13T14:46:27.572859+0000","last_became_active":"2024-09-13T14:43:32.635736+0000","last_became_peered":"2024-09-13T14:43:32.635736+0000","last_unstale":"2024-09-13T14:46:27.572859+0000","last_undegraded":"2024-09-13T14:46:27.572859+0000","last_fullsized":"2024-09-13T14:46:27.572859+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:49.076014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"261'100","reported_seq":103,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.576943+0000","last_change":"2024-09-13T14:45:19.805634+0000","last_active":"2024-09-13T14:46:27.576943+0000","last_peered":"2024-09-13T14:46:27.576943+0000","last_clean":"2024-09-13T14:46:27.576943+0000","last_became_active":"2024-09-13T14:45:19.805046+0000","last_became_peered":"2024-09-13T14:45:19.805046+0000","last_unstale":"2024-09-13T14:46:27.576943+0000","last_undegraded":"2024-09-13T14:46:27.576943+0000","last_fullsized":"2024-09-13T14:46:27.576943+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:32.752408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:32.752408+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:06:53.140239+0000","scrub_duration":0.0057731559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"1158'10914","reported_seq":10975,"reported_epoch":1158,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:46:59.290991+0000","last_change":"2024-09-13T14:45:21.029184+0000","last_active":"2024-09-13T14:46:59.290991+0000","last_peered":"2024-09-13T14:46:59.290991+0000","last_clean":"2024-09-13T14:45:19.633080+0000","last_became_active":"2024-09-13T14:45:21.019809+0000","last_became_peered":"2024-09-13T14:45:21.019809+0000","last_unstale":"2024-09-13T14:46:59.290991+0000","last_undegraded":"2024-09-13T14:46:59.290991+0000","last_fullsized":"2024-09-13T14:46:59.290991+0000","mapping_epoch":1149,"log_start":"1142'4000","ondisk_log_start":"1142'4000","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6914,"log_dups_size":0,"ondisk_log_size":6914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279314432,"num_objects":4262,"num_object_clones":0,"num_object_copies":12786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4262,"num_objects_unfound":0,"num_objects_dirty":4262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10914,"num_write_kb":485632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,7,4],"avail_no_missing":["0","4","7"],"object_location_counts":[{"shards":"0,4,7","objects":4262}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1148,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.574618+0000","last_change":"2024-09-13T14:45:19.804441+0000","last_active":"2024-09-13T14:46:27.574618+0000","last_peered":"2024-09-13T14:46:27.574618+0000","last_clean":"2024-09-13T14:46:27.574618+0000","last_became_active":"2024-09-13T14:45:19.804039+0000","last_became_peered":"2024-09-13T14:45:19.804039+0000","last_unstale":"2024-09-13T14:46:27.574618+0000","last_undegraded":"2024-09-13T14:46:27.574618+0000","last_fullsized":"2024-09-13T14:46:27.574618+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:58:00.322042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":34,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.667088+0000","last_change":"2024-09-13T14:45:19.816666+0000","last_active":"2024-09-13T14:46:48.667088+0000","last_peered":"2024-09-13T14:46:48.667088+0000","last_clean":"2024-09-13T14:46:48.667088+0000","last_became_active":"2024-09-13T14:45:19.807224+0000","last_became_peered":"2024-09-13T14:45:19.807224+0000","last_unstale":"2024-09-13T14:46:48.667088+0000","last_undegraded":"2024-09-13T14:46:48.667088+0000","last_fullsized":"2024-09-13T14:46:48.667088+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:15:20.171682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"261'100","reported_seq":1288,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.552944+0000","last_change":"2024-09-13T14:45:19.798553+0000","last_active":"2024-09-13T14:46:27.552944+0000","last_peered":"2024-09-13T14:46:27.552944+0000","last_clean":"2024-09-13T14:46:27.552944+0000","last_became_active":"2024-09-13T14:45:19.798189+0000","last_became_peered":"2024-09-13T14:45:19.798189+0000","last_unstale":"2024-09-13T14:46:27.552944+0000","last_undegraded":"2024-09-13T14:46:27.552944+0000","last_fullsized":"2024-09-13T14:46:27.552944+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:29.169325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:29.169325+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:10.412666+0000","scrub_duration":0.0019358890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"1158'10806","reported_seq":10862,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:56.346278+0000","last_change":"2024-09-13T14:43:50.795029+0000","last_active":"2024-09-13T14:46:56.346278+0000","last_peered":"2024-09-13T14:46:56.346278+0000","last_clean":"2024-09-13T14:46:56.346278+0000","last_became_active":"2024-09-13T14:43:50.794715+0000","last_became_peered":"2024-09-13T14:43:50.794715+0000","last_unstale":"2024-09-13T14:46:56.346278+0000","last_undegraded":"2024-09-13T14:46:56.346278+0000","last_fullsized":"2024-09-13T14:46:56.346278+0000","mapping_epoch":1136,"log_start":"1141'2400","ondisk_log_start":"1141'2400","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":8406,"log_dups_size":0,"ondisk_log_size":8406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:21.257279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281935872,"num_objects":4302,"num_object_clones":0,"num_object_copies":12906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10806,"num_write_kb":483456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1699,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.648460+0000","last_change":"2024-09-13T14:43:07.515572+0000","last_active":"2024-09-13T14:46:48.648460+0000","last_peered":"2024-09-13T14:46:48.648460+0000","last_clean":"2024-09-13T14:46:48.648460+0000","last_became_active":"2024-09-13T14:43:07.515231+0000","last_became_peered":"2024-09-13T14:43:07.515231+0000","last_unstale":"2024-09-13T14:46:48.648460+0000","last_undegraded":"2024-09-13T14:46:48.648460+0000","last_fullsized":"2024-09-13T14:46:48.648460+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:57.966380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1141,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.577374+0000","last_change":"2024-09-13T14:45:19.804982+0000","last_active":"2024-09-13T14:46:27.577374+0000","last_peered":"2024-09-13T14:46:27.577374+0000","last_clean":"2024-09-13T14:46:27.577374+0000","last_became_active":"2024-09-13T14:45:19.804663+0000","last_became_peered":"2024-09-13T14:45:19.804663+0000","last_unstale":"2024-09-13T14:46:27.577374+0000","last_undegraded":"2024-09-13T14:46:27.577374+0000","last_fullsized":"2024-09-13T14:46:27.577374+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:43.211479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"261'100","reported_seq":378,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.573325+0000","last_change":"2024-09-13T14:43:42.881584+0000","last_active":"2024-09-13T14:46:27.573325+0000","last_peered":"2024-09-13T14:46:27.573325+0000","last_clean":"2024-09-13T14:46:27.573325+0000","last_became_active":"2024-09-13T14:43:14.562161+0000","last_became_peered":"2024-09-13T14:43:14.562161+0000","last_unstale":"2024-09-13T14:46:27.573325+0000","last_undegraded":"2024-09-13T14:46:27.573325+0000","last_fullsized":"2024-09-13T14:46:27.573325+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:42.881520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:42.881520+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:06:50.353442+0000","scrub_duration":0.0013804310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"1158'10925","reported_seq":10981,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.365864+0000","last_change":"2024-09-13T14:43:50.793879+0000","last_active":"2024-09-13T14:46:59.365864+0000","last_peered":"2024-09-13T14:46:59.365864+0000","last_clean":"2024-09-13T14:46:59.365864+0000","last_became_active":"2024-09-13T14:43:50.793571+0000","last_became_peered":"2024-09-13T14:43:50.793571+0000","last_unstale":"2024-09-13T14:46:59.365864+0000","last_undegraded":"2024-09-13T14:46:59.365864+0000","last_fullsized":"2024-09-13T14:46:59.365864+0000","mapping_epoch":1136,"log_start":"1142'4100","ondisk_log_start":"1142'4100","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6825,"log_dups_size":0,"ondisk_log_size":6825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:22.880360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280821760,"num_objects":4285,"num_object_clones":0,"num_object_copies":12855,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10925,"num_write_kb":486720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1114,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.578743+0000","last_change":"2024-09-13T14:43:07.518120+0000","last_active":"2024-09-13T14:46:27.578743+0000","last_peered":"2024-09-13T14:46:27.578743+0000","last_clean":"2024-09-13T14:46:27.578743+0000","last_became_active":"2024-09-13T14:43:07.517361+0000","last_became_peered":"2024-09-13T14:43:07.517361+0000","last_unstale":"2024-09-13T14:46:27.578743+0000","last_undegraded":"2024-09-13T14:46:27.578743+0000","last_fullsized":"2024-09-13T14:46:27.578743+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:39.366473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":1096,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.553641+0000","last_change":"2024-09-13T14:19:25.871006+0000","last_active":"2024-09-13T14:46:27.553641+0000","last_peered":"2024-09-13T14:46:27.553641+0000","last_clean":"2024-09-13T14:46:27.553641+0000","last_became_active":"2024-09-13T14:19:25.870713+0000","last_became_peered":"2024-09-13T14:19:25.870713+0000","last_unstale":"2024-09-13T14:46:27.553641+0000","last_undegraded":"2024-09-13T14:46:27.553641+0000","last_fullsized":"2024-09-13T14:46:27.553641+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:34.372124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"261'100","reported_seq":362,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.577778+0000","last_change":"2024-09-13T14:43:32.637553+0000","last_active":"2024-09-13T14:46:27.577778+0000","last_peered":"2024-09-13T14:46:27.577778+0000","last_clean":"2024-09-13T14:46:27.577778+0000","last_became_active":"2024-09-13T14:43:32.636545+0000","last_became_peered":"2024-09-13T14:43:32.636545+0000","last_unstale":"2024-09-13T14:46:27.577778+0000","last_undegraded":"2024-09-13T14:46:27.577778+0000","last_fullsized":"2024-09-13T14:46:27.577778+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:15:42.069756+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"1158'10758","reported_seq":10814,"reported_epoch":1158,"state":"active+recovery_wait+remapped","last_fresh":"2024-09-13T14:46:57.572930+0000","last_change":"2024-09-13T14:45:21.130410+0000","last_active":"2024-09-13T14:46:57.572930+0000","last_peered":"2024-09-13T14:46:57.572930+0000","last_clean":"2024-09-13T14:45:19.587556+0000","last_became_active":"2024-09-13T14:45:21.100167+0000","last_became_peered":"2024-09-13T14:45:21.100167+0000","last_unstale":"2024-09-13T14:46:57.572930+0000","last_undegraded":"2024-09-13T14:46:57.572930+0000","last_fullsized":"2024-09-13T14:46:57.572930+0000","mapping_epoch":1149,"log_start":"1142'4677","ondisk_log_start":"1142'4677","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6081,"log_dups_size":0,"ondisk_log_size":6081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279052288,"num_objects":4258,"num_object_clones":0,"num_object_copies":12774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7077,"num_objects_unfound":0,"num_objects_dirty":4258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10758,"num_write_kb":480512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[1,2,5],"avail_no_missing":["1","2","5"],"object_location_counts":[{"shards":"1,2,5","objects":-2819}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1165,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.573174+0000","last_change":"2024-09-13T14:45:19.804868+0000","last_active":"2024-09-13T14:46:27.573174+0000","last_peered":"2024-09-13T14:46:27.573174+0000","last_clean":"2024-09-13T14:46:27.573174+0000","last_became_active":"2024-09-13T14:45:19.804520+0000","last_became_peered":"2024-09-13T14:45:19.804520+0000","last_unstale":"2024-09-13T14:46:27.573174+0000","last_undegraded":"2024-09-13T14:46:27.573174+0000","last_fullsized":"2024-09-13T14:46:27.573174+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:17:53.924629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":1142,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.554001+0000","last_change":"2024-09-13T14:43:07.533955+0000","last_active":"2024-09-13T14:46:27.554001+0000","last_peered":"2024-09-13T14:46:27.554001+0000","last_clean":"2024-09-13T14:46:27.554001+0000","last_became_active":"2024-09-13T14:43:07.516816+0000","last_became_peered":"2024-09-13T14:43:07.516816+0000","last_unstale":"2024-09-13T14:46:27.554001+0000","last_undegraded":"2024-09-13T14:46:27.554001+0000","last_fullsized":"2024-09-13T14:46:27.554001+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:26.232184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"261'100","reported_seq":34,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.667192+0000","last_change":"2024-09-13T14:45:19.821649+0000","last_active":"2024-09-13T14:46:48.667192+0000","last_peered":"2024-09-13T14:46:48.667192+0000","last_clean":"2024-09-13T14:46:48.667192+0000","last_became_active":"2024-09-13T14:45:19.812346+0000","last_became_peered":"2024-09-13T14:45:19.812346+0000","last_unstale":"2024-09-13T14:46:48.667192+0000","last_undegraded":"2024-09-13T14:46:48.667192+0000","last_fullsized":"2024-09-13T14:46:48.667192+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.597568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.597568+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:45.218892+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"1158'10820","reported_seq":10876,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:57.609051+0000","last_change":"2024-09-13T14:43:51.537117+0000","last_active":"2024-09-13T14:46:57.609051+0000","last_peered":"2024-09-13T14:46:57.609051+0000","last_clean":"2024-09-13T14:46:57.609051+0000","last_became_active":"2024-09-13T14:43:51.536532+0000","last_became_peered":"2024-09-13T14:43:51.536532+0000","last_unstale":"2024-09-13T14:46:57.609051+0000","last_undegraded":"2024-09-13T14:46:57.609051+0000","last_fullsized":"2024-09-13T14:46:57.609051+0000","mapping_epoch":1136,"log_start":"1142'4800","ondisk_log_start":"1142'4800","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6020,"log_dups_size":0,"ondisk_log_size":6020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:41.713211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271712256,"num_objects":4146,"num_object_clones":0,"num_object_copies":12438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10820,"num_write_kb":478912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1083,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.577592+0000","last_change":"2024-09-13T14:19:26.887839+0000","last_active":"2024-09-13T14:46:27.577592+0000","last_peered":"2024-09-13T14:46:27.577592+0000","last_clean":"2024-09-13T14:46:27.577592+0000","last_became_active":"2024-09-13T14:19:26.887498+0000","last_became_peered":"2024-09-13T14:19:26.887498+0000","last_unstale":"2024-09-13T14:46:27.577592+0000","last_undegraded":"2024-09-13T14:46:27.577592+0000","last_fullsized":"2024-09-13T14:46:27.577592+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:24.475263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":1113,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.554705+0000","last_change":"2024-09-13T14:28:35.608607+0000","last_active":"2024-09-13T14:46:27.554705+0000","last_peered":"2024-09-13T14:46:27.554705+0000","last_clean":"2024-09-13T14:46:27.554705+0000","last_became_active":"2024-09-13T14:28:35.608254+0000","last_became_peered":"2024-09-13T14:28:35.608254+0000","last_unstale":"2024-09-13T14:46:27.554705+0000","last_undegraded":"2024-09-13T14:46:27.554705+0000","last_fullsized":"2024-09-13T14:46:27.554705+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:24.019173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"261'100","reported_seq":49,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.553408+0000","last_change":"2024-09-13T14:44:31.191431+0000","last_active":"2024-09-13T14:46:27.553408+0000","last_peered":"2024-09-13T14:46:27.553408+0000","last_clean":"2024-09-13T14:46:27.553408+0000","last_became_active":"2024-09-13T14:43:14.557376+0000","last_became_peered":"2024-09-13T14:43:14.557376+0000","last_unstale":"2024-09-13T14:46:27.553408+0000","last_undegraded":"2024-09-13T14:46:27.553408+0000","last_fullsized":"2024-09-13T14:46:27.553408+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.191326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.191326+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:15:27.352698+0000","scrub_duration":0.0025317360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1103,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.578452+0000","last_change":"2024-09-13T14:28:35.603724+0000","last_active":"2024-09-13T14:46:27.578452+0000","last_peered":"2024-09-13T14:46:27.578452+0000","last_clean":"2024-09-13T14:46:27.578452+0000","last_became_active":"2024-09-13T14:28:35.603316+0000","last_became_peered":"2024-09-13T14:28:35.603316+0000","last_unstale":"2024-09-13T14:46:27.578452+0000","last_undegraded":"2024-09-13T14:46:27.578452+0000","last_fullsized":"2024-09-13T14:46:27.578452+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:47.668223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":30,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.671018+0000","last_change":"2024-09-13T14:45:19.811466+0000","last_active":"2024-09-13T14:46:48.671018+0000","last_peered":"2024-09-13T14:46:48.671018+0000","last_clean":"2024-09-13T14:46:48.671018+0000","last_became_active":"2024-09-13T14:45:19.810617+0000","last_became_peered":"2024-09-13T14:45:19.810617+0000","last_unstale":"2024-09-13T14:46:48.671018+0000","last_undegraded":"2024-09-13T14:46:48.671018+0000","last_fullsized":"2024-09-13T14:46:48.671018+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:16.246669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"261'100","reported_seq":49,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.577165+0000","last_change":"2024-09-13T14:44:50.805306+0000","last_active":"2024-09-13T14:46:27.577165+0000","last_peered":"2024-09-13T14:46:27.577165+0000","last_clean":"2024-09-13T14:46:27.577165+0000","last_became_active":"2024-09-13T14:43:14.562922+0000","last_became_peered":"2024-09-13T14:43:14.562922+0000","last_unstale":"2024-09-13T14:46:27.577165+0000","last_undegraded":"2024-09-13T14:46:27.577165+0000","last_fullsized":"2024-09-13T14:46:27.577165+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:50.805210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:50.805210+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:28:38.529599+0000","scrub_duration":0.0042900009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1095,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.553045+0000","last_change":"2024-09-13T14:19:25.873088+0000","last_active":"2024-09-13T14:46:27.553045+0000","last_peered":"2024-09-13T14:46:27.553045+0000","last_clean":"2024-09-13T14:46:27.553045+0000","last_became_active":"2024-09-13T14:19:25.872781+0000","last_became_peered":"2024-09-13T14:19:25.872781+0000","last_unstale":"2024-09-13T14:46:27.553045+0000","last_undegraded":"2024-09-13T14:46:27.553045+0000","last_fullsized":"2024-09-13T14:46:27.553045+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:05.607382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1698,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.650281+0000","last_change":"2024-09-13T14:45:19.813876+0000","last_active":"2024-09-13T14:46:48.650281+0000","last_peered":"2024-09-13T14:46:48.650281+0000","last_clean":"2024-09-13T14:46:48.650281+0000","last_became_active":"2024-09-13T14:45:19.809136+0000","last_became_peered":"2024-09-13T14:45:19.809136+0000","last_unstale":"2024-09-13T14:46:48.650281+0000","last_undegraded":"2024-09-13T14:46:48.650281+0000","last_fullsized":"2024-09-13T14:46:48.650281+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:50.013528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"261'100","reported_seq":104,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.651237+0000","last_change":"2024-09-13T14:45:19.810654+0000","last_active":"2024-09-13T14:46:48.651237+0000","last_peered":"2024-09-13T14:46:48.651237+0000","last_clean":"2024-09-13T14:46:48.651237+0000","last_became_active":"2024-09-13T14:45:19.810066+0000","last_became_peered":"2024-09-13T14:45:19.810066+0000","last_unstale":"2024-09-13T14:46:48.651237+0000","last_undegraded":"2024-09-13T14:46:48.651237+0000","last_fullsized":"2024-09-13T14:46:48.651237+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:19.978464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:19.978464+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:33.639332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1745,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.648070+0000","last_change":"2024-09-13T14:45:19.809166+0000","last_active":"2024-09-13T14:46:48.648070+0000","last_peered":"2024-09-13T14:46:48.648070+0000","last_clean":"2024-09-13T14:46:48.648070+0000","last_became_active":"2024-09-13T14:45:19.808631+0000","last_became_peered":"2024-09-13T14:45:19.808631+0000","last_unstale":"2024-09-13T14:46:48.648070+0000","last_undegraded":"2024-09-13T14:46:48.648070+0000","last_fullsized":"2024-09-13T14:46:48.648070+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:38.822245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1120,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.577416+0000","last_change":"2024-09-13T14:45:19.804229+0000","last_active":"2024-09-13T14:46:27.577416+0000","last_peered":"2024-09-13T14:46:27.577416+0000","last_clean":"2024-09-13T14:46:27.577416+0000","last_became_active":"2024-09-13T14:45:19.803929+0000","last_became_peered":"2024-09-13T14:45:19.803929+0000","last_unstale":"2024-09-13T14:46:27.577416+0000","last_undegraded":"2024-09-13T14:46:27.577416+0000","last_fullsized":"2024-09-13T14:46:27.577416+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:33:53.675568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"261'100","reported_seq":49,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.576851+0000","last_change":"2024-09-13T14:44:42.654506+0000","last_active":"2024-09-13T14:46:27.576851+0000","last_peered":"2024-09-13T14:46:27.576851+0000","last_clean":"2024-09-13T14:46:27.576851+0000","last_became_active":"2024-09-13T14:43:14.563451+0000","last_became_peered":"2024-09-13T14:43:14.563451+0000","last_unstale":"2024-09-13T14:46:27.576851+0000","last_undegraded":"2024-09-13T14:46:27.576851+0000","last_fullsized":"2024-09-13T14:46:27.576851+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:42.654455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:42.654455+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:57:32.251597+0000","scrub_duration":0.001494097,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":1114,"reported_epoch":1157,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.553513+0000","last_change":"2024-09-13T14:29:42.541093+0000","last_active":"2024-09-13T14:46:27.553513+0000","last_peered":"2024-09-13T14:46:27.553513+0000","last_clean":"2024-09-13T14:46:27.553513+0000","last_became_active":"2024-09-13T14:29:42.540671+0000","last_became_peered":"2024-09-13T14:29:42.540671+0000","last_unstale":"2024-09-13T14:46:27.553513+0000","last_undegraded":"2024-09-13T14:46:27.553513+0000","last_fullsized":"2024-09-13T14:46:27.553513+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:45.722376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1637,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.667538+0000","last_change":"2024-09-13T14:43:07.518679+0000","last_active":"2024-09-13T14:46:48.667538+0000","last_peered":"2024-09-13T14:46:48.667538+0000","last_clean":"2024-09-13T14:46:48.667538+0000","last_became_active":"2024-09-13T14:43:07.518333+0000","last_became_peered":"2024-09-13T14:43:07.518333+0000","last_unstale":"2024-09-13T14:46:48.667538+0000","last_undegraded":"2024-09-13T14:46:48.667538+0000","last_fullsized":"2024-09-13T14:46:48.667538+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:20.616274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"261'100","reported_seq":406,"reported_epoch":1158,"state":"active+clean","last_fresh":"2024-09-13T14:46:48.645566+0000","last_change":"2024-09-13T14:45:19.809268+0000","last_active":"2024-09-13T14:46:48.645566+0000","last_peered":"2024-09-13T14:46:48.645566+0000","last_clean":"2024-09-13T14:46:48.645566+0000","last_became_active":"2024-09-13T14:45:19.808772+0000","last_became_peered":"2024-09-13T14:45:19.808772+0000","last_unstale":"2024-09-13T14:46:48.645566+0000","last_undegraded":"2024-09-13T14:46:48.645566+0000","last_fullsized":"2024-09-13T14:46:48.645566+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:14.654603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:14.654603+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:40.964553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":4449763352,"num_objects":67899,"num_object_clones":0,"num_object_copies":203697,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1971,"num_objects_misplaced":40716,"num_objects_unfound":0,"num_objects_dirty":67899,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173019,"num_write_kb":7709313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1249,"num_bytes_recovered":81854464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13609021440,"data_stored":13609009224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112342,"ondisk_log_size":112342,"up":48,"acting":47,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":354515,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430563,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1972212,"kb_used_data":1666540,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91772940,"statfs":{"total":95995035648,"available":93975490560,"internally_reserved":0,"allocated":1706536960,"data_stored":1702999674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626792,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2114540,"kb_used_data":1817188,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":91630612,"statfs":{"total":95995035648,"available":93829746688,"internally_reserved":0,"allocated":1860800512,"data_stored":1857270779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823021,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1662820,"kb_used_data":1365468,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":92082332,"statfs":{"total":95995035648,"available":94292307968,"internally_reserved":0,"allocated":1398239232,"data_stored":1394714029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019249,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1972652,"kb_used_data":1662820,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91772500,"statfs":{"total":95995035648,"available":93975040000,"internally_reserved":0,"allocated":1702727680,"data_stored":1699197972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059592907,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":228248,"kb_used_data":67984,"kb_used_omap":26,"kb_used_meta":160229,"kb_avail":93516904,"statfs":{"total":95995035648,"available":95761309696,"internally_reserved":0,"allocated":69615616,"data_stored":66101608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":164074656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":591,"seq":2538325672145,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2593632,"kb_used_data":2230680,"kb_used_omap":41,"kb_used_meta":362902,"kb_avail":91151520,"statfs":{"total":95995035648,"available":93339156480,"internally_reserved":0,"allocated":2284216320,"data_stored":2280695186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42651,"internal_metadata":371612005},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":557,"seq":2392296784088,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2820828,"kb_used_data":2473492,"kb_used_omap":32,"kb_used_meta":347295,"kb_avail":90924324,"statfs":{"total":95995035648,"available":93106507776,"internally_reserved":0,"allocated":2532855808,"data_stored":2529337673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33752,"internal_metadata":355630120},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":524,"seq":2250562863327,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2469856,"kb_used_data":2162136,"kb_used_omap":41,"kb_used_meta":307542,"kb_avail":91275296,"statfs":{"total":95995035648,"available":93465903104,"internally_reserved":0,"allocated":2214027264,"data_stored":2210505614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42694,"internal_metadata":314923322},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2189430784,"data_stored":2189426712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2506887168,"data_stored":2506883096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2254635008,"data_stored":2254635008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":58654720,"data_stored":58654720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1689845760,"data_stored":1689845760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1385496576,"data_stored":1385496576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1836781568,"data_stored":1836777496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1687289856,"data_stored":1687289856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:47:01.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-13T14:47:01.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:01.294 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:01.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:01.394 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:01.467 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:01.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:01.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:01.569 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:01.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:01.610 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:01.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:01.711 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:01.739 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:01.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:01.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:01.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:01.837 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:01 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:01.841 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:01.897 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:01.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:01.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:02.002 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:02.048 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:02.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:02.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:02.100 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:47:02.100 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-13T14:47:02.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-13T14:47:02.100 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-13T14:47:02.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.100 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:61879 7.8 7:1ec09d57:::benchmark_data_smithi151_190906_object61878:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1142)", 2024-09-13T14:47:02.100 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:44:33.514032+0000", 2024-09-13T14:47:02.100 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 148.321726673, 2024-09-13T14:47:02.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.250095026, 2024-09-13T14:47:02.100 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:02.100 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:02.101 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:02.101 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:02.101 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:47:02.101 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:47:02.101 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 61879 2024-09-13T14:47:02.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.101 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:02.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:02.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.514032+0000", 2024-09-13T14:47:02.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:02.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.514032+0000", 2024-09-13T14:47:02.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:02.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.514034+0000", 2024-09-13T14:47:02.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.55e-06 2024-09-13T14:47:02.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:02.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.514062+0000", 2024-09-13T14:47:02.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8362000000000001e-05 2024-09-13T14:47:02.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:02.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.514076+0000", 2024-09-13T14:47:02.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.349e-05 2024-09-13T14:47:02.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:02.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.514083+0000", 2024-09-13T14:47:02.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2139999999999999e-06 2024-09-13T14:47:02.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:02.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.514855+0000", 2024-09-13T14:47:02.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00077275399999999995 2024-09-13T14:47:02.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:02.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.514968+0000", 2024-09-13T14:47:02.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000112222 2024-09-13T14:47:02.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-13T14:47:02.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.515083+0000", 2024-09-13T14:47:02.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000115077 2024-09-13T14:47:02.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:02.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.518799+0000", 2024-09-13T14:47:02.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037163500000000002 2024-09-13T14:47:02.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:02.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.519124+0000", 2024-09-13T14:47:02.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00032487400000000001 2024-09-13T14:47:02.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:02.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.519160+0000", 2024-09-13T14:47:02.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6489999999999998e-05 2024-09-13T14:47:02.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:02.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.519216+0000", 2024-09-13T14:47:02.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5379e-05 2024-09-13T14:47:02.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:02.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.764127+0000", 2024-09-13T14:47:02.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24491126399999999 2024-09-13T14:47:02.106 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.106 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:02.106 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.107 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:61880 7.3 7:c5b9b7cf:::benchmark_data_smithi151_190906_object61879:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1142)", 2024-09-13T14:47:02.107 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:44:33.514103+0000", 2024-09-13T14:47:02.107 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 148.321655765, 2024-09-13T14:47:02.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25375483100000001, 2024-09-13T14:47:02.107 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:02.107 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:02.107 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:02.107 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:02.107 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:47:02.107 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:47:02.107 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 61880 2024-09-13T14:47:02.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.107 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:02.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:02.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.514103+0000", 2024-09-13T14:47:02.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:02.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.514103+0000", 2024-09-13T14:47:02.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:02.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.514104+0000", 2024-09-13T14:47:02.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9299999999999998e-07 2024-09-13T14:47:02.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:02.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.514127+0000", 2024-09-13T14:47:02.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3777e-05 2024-09-13T14:47:02.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:02.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.514136+0000", 2024-09-13T14:47:02.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4290000000000001e-06 2024-09-13T14:47:02.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:02.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.514142+0000", 2024-09-13T14:47:02.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2480000000000001e-06 2024-09-13T14:47:02.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:02.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.514196+0000", 2024-09-13T14:47:02.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3763e-05 2024-09-13T14:47:02.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:02.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.514403+0000", 2024-09-13T14:47:02.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000207054 2024-09-13T14:47:02.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:47:02.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.514574+0000", 2024-09-13T14:47:02.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017115199999999999 2024-09-13T14:47:02.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:02.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.764651+0000", 2024-09-13T14:47:02.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.250076941 2024-09-13T14:47:02.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:02.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.764697+0000", 2024-09-13T14:47:02.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5819999999999998e-05 2024-09-13T14:47:02.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:02.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.767681+0000", 2024-09-13T14:47:02.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00298387 2024-09-13T14:47:02.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:02.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.767816+0000", 2024-09-13T14:47:02.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000134924 2024-09-13T14:47:02.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:02.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.767858+0000", 2024-09-13T14:47:02.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2259999999999999e-05 2024-09-13T14:47:02.113 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.113 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:02.113 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.113 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.45494.0:61880 7.3 e1142/1136 ondisk, result = 0)", 2024-09-13T14:47:02.113 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:44:33.517171+0000", 2024-09-13T14:47:02.113 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 148.31858819199999, 2024-09-13T14:47:02.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.250691777, 2024-09-13T14:47:02.114 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:02.114 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:02.114 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-13T14:47:02.114 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:02.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:02.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.517171+0000", 2024-09-13T14:47:02.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:02.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.517171+0000", 2024-09-13T14:47:02.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:02.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.517171+0000", 2024-09-13T14:47:02.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3399999999999998e-07 2024-09-13T14:47:02.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:02.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.517177+0000", 2024-09-13T14:47:02.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0900000000000004e-06 2024-09-13T14:47:02.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:02.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.517188+0000", 2024-09-13T14:47:02.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1511999999999999e-05 2024-09-13T14:47:02.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:02.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.517195+0000", 2024-09-13T14:47:02.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2459999999999997e-06 2024-09-13T14:47:02.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:02.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.764638+0000", 2024-09-13T14:47:02.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24744301899999999 2024-09-13T14:47:02.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:02.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.764646+0000", 2024-09-13T14:47:02.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2459999999999997e-06 2024-09-13T14:47:02.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:02.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.767862+0000", 2024-09-13T14:47:02.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0032168299999999999 2024-09-13T14:47:02.117 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.117 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:02.117 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.118 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.45494.0:61880 7.3 e1142/1136 ondisk, result = 0)", 2024-09-13T14:47:02.118 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:44:33.519026+0000", 2024-09-13T14:47:02.118 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 148.316732462, 2024-09-13T14:47:02.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24568285000000001, 2024-09-13T14:47:02.118 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:02.118 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:02.118 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-13T14:47:02.118 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:02.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:02.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.519026+0000", 2024-09-13T14:47:02.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:02.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.519026+0000", 2024-09-13T14:47:02.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:02.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.519027+0000", 2024-09-13T14:47:02.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1299999999999999e-07 2024-09-13T14:47:02.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:02.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.519031+0000", 2024-09-13T14:47:02.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6509999999999999e-06 2024-09-13T14:47:02.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:02.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.519042+0000", 2024-09-13T14:47:02.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1538e-05 2024-09-13T14:47:02.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:02.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.519049+0000", 2024-09-13T14:47:02.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8129999999999998e-06 2024-09-13T14:47:02.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:02.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.764684+0000", 2024-09-13T14:47:02.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.245634617 2024-09-13T14:47:02.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:02.121 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.764691+0000", 2024-09-13T14:47:02.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3409999999999997e-06 2024-09-13T14:47:02.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:02.121 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.764709+0000", 2024-09-13T14:47:02.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8077e-05 2024-09-13T14:47:02.121 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.121 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:02.121 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.122 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.45494.0:61879 7.8 e1142/1136 ondisk, result = 0)", 2024-09-13T14:47:02.122 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:44:33.519101+0000", 2024-09-13T14:47:02.122 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 148.316657779, 2024-09-13T14:47:02.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.245034743, 2024-09-13T14:47:02.122 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:02.122 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:02.122 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-13T14:47:02.122 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:02.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:02.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.519101+0000", 2024-09-13T14:47:02.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:02.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.519101+0000", 2024-09-13T14:47:02.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:02.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.519102+0000", 2024-09-13T14:47:02.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7100000000000002e-07 2024-09-13T14:47:02.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:02.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.519103+0000", 2024-09-13T14:47:02.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5859999999999999e-06 2024-09-13T14:47:02.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:02.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.519109+0000", 2024-09-13T14:47:02.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9830000000000001e-06 2024-09-13T14:47:02.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:02.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.519113+0000", 2024-09-13T14:47:02.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3579999999999996e-06 2024-09-13T14:47:02.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:02.125 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.519150+0000", 2024-09-13T14:47:02.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6293e-05 2024-09-13T14:47:02.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:02.125 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.519155+0000", 2024-09-13T14:47:02.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.045e-06 2024-09-13T14:47:02.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:02.125 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.764136+0000", 2024-09-13T14:47:02.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.244980907 2024-09-13T14:47:02.125 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.126 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:02.126 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.126 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:61888 7.e 7:721fc974:::benchmark_data_smithi151_190906_object61887:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1142)", 2024-09-13T14:47:02.126 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:44:33.519509+0000", 2024-09-13T14:47:02.126 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 148.31625016800001, 2024-09-13T14:47:02.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25468754100000002, 2024-09-13T14:47:02.126 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:02.126 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:02.126 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:02.126 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:02.126 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:47:02.127 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:47:02.127 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 61888 2024-09-13T14:47:02.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.127 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:02.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:02.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.519509+0000", 2024-09-13T14:47:02.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:02.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.519509+0000", 2024-09-13T14:47:02.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:02.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.519510+0000", 2024-09-13T14:47:02.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.68e-06 2024-09-13T14:47:02.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:02.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.519536+0000", 2024-09-13T14:47:02.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5675000000000001e-05 2024-09-13T14:47:02.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:02.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.761529+0000", 2024-09-13T14:47:02.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.241993233 2024-09-13T14:47:02.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:02.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.761542+0000", 2024-09-13T14:47:02.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2424e-05 2024-09-13T14:47:02.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:02.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.761750+0000", 2024-09-13T14:47:02.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000208369 2024-09-13T14:47:02.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:02.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.762066+0000", 2024-09-13T14:47:02.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00031649100000000002 2024-09-13T14:47:02.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:47:02.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.762194+0000", 2024-09-13T14:47:02.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000127584 2024-09-13T14:47:02.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:02.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.767030+0000", 2024-09-13T14:47:02.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0048361630000000001 2024-09-13T14:47:02.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:02.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.767247+0000", 2024-09-13T14:47:02.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00021726699999999999 2024-09-13T14:47:02.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:02.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.774038+0000", 2024-09-13T14:47:02.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0067906870000000001 2024-09-13T14:47:02.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:02.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.774133+0000", 2024-09-13T14:47:02.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4335999999999999e-05 2024-09-13T14:47:02.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:02.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.774196+0000", 2024-09-13T14:47:02.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3632000000000003e-05 2024-09-13T14:47:02.132 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.132 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:02.132 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.132 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.45494.0:61895 7.5 e1142/1136 7:a46f2851:::benchmark_data_smithi151_190906_object61894:head v 1142'3872, mlcod=1142'3870)", 2024-09-13T14:47:02.132 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:44:33.524416+0000", 2024-09-13T14:47:02.132 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 148.31134303600001, 2024-09-13T14:47:02.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24026043599999999, 2024-09-13T14:47:02.133 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:02.133 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:02.133 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:02.133 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:02.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:02.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.524416+0000", 2024-09-13T14:47:02.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:02.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.524416+0000", 2024-09-13T14:47:02.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:02.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.524417+0000", 2024-09-13T14:47:02.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0839999999999999e-06 2024-09-13T14:47:02.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:02.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.524457+0000", 2024-09-13T14:47:02.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0166000000000001e-05 2024-09-13T14:47:02.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:02.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.760832+0000", 2024-09-13T14:47:02.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23637483000000001 2024-09-13T14:47:02.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:02.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.760849+0000", 2024-09-13T14:47:02.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7308000000000001e-05 2024-09-13T14:47:02.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:02.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.760942+0000", 2024-09-13T14:47:02.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3259999999999998e-05 2024-09-13T14:47:02.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:02.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.760969+0000", 2024-09-13T14:47:02.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6721e-05 2024-09-13T14:47:02.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:02.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.764596+0000", 2024-09-13T14:47:02.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0036272980000000002 2024-09-13T14:47:02.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:02.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.764676+0000", 2024-09-13T14:47:02.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9769000000000001e-05 2024-09-13T14:47:02.136 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.137 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:02.137 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.137 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:138690 7.3 7:cd5fade0:::benchmark_data_smithi151_190906_object18229:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1155)", 2024-09-13T14:47:02.137 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:46:01.835114+0000", 2024-09-13T14:47:02.137 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.000644758999996, 2024-09-13T14:47:02.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25913276699999999, 2024-09-13T14:47:02.137 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:02.137 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:02.137 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:02.137 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:02.137 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:47:02.137 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:47:02.138 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 138690 2024-09-13T14:47:02.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.138 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:02.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:02.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:01.835114+0000", 2024-09-13T14:47:02.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:02.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:01.835114+0000", 2024-09-13T14:47:02.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:02.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:01.835116+0000", 2024-09-13T14:47:02.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.79e-06 2024-09-13T14:47:02.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:02.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:01.835123+0000", 2024-09-13T14:47:02.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3010000000000001e-06 2024-09-13T14:47:02.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:02.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:01.835138+0000", 2024-09-13T14:47:02.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4466e-05 2024-09-13T14:47:02.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:02.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:01.835146+0000", 2024-09-13T14:47:02.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5029999999999997e-06 2024-09-13T14:47:02.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:02.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:01.849681+0000", 2024-09-13T14:47:02.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014534811999999999 2024-09-13T14:47:02.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:47:02.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:01.849813+0000", 2024-09-13T14:47:02.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013260600000000001 2024-09-13T14:47:02.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:02.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.054089+0000", 2024-09-13T14:47:02.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.204275818 2024-09-13T14:47:02.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:02.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.054207+0000", 2024-09-13T14:47:02.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117998 2024-09-13T14:47:02.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:47:02.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.054387+0000", 2024-09-13T14:47:02.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017924600000000001 2024-09-13T14:47:02.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:02.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.059406+0000", 2024-09-13T14:47:02.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0050195470000000001 2024-09-13T14:47:02.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:02.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.077812+0000", 2024-09-13T14:47:02.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018406388999999999 2024-09-13T14:47:02.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:02.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.089182+0000", 2024-09-13T14:47:02.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01136962 2024-09-13T14:47:02.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:02.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.089260+0000", 2024-09-13T14:47:02.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8265999999999999e-05 2024-09-13T14:47:02.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:02.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.094247+0000", 2024-09-13T14:47:02.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0049864050000000002 2024-09-13T14:47:02.144 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.144 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:02.144 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.144 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:138848 7.3 7:c24d64b9:::benchmark_data_smithi151_190906_object18387:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1155)", 2024-09-13T14:47:02.144 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:46:02.114087+0000", 2024-09-13T14:47:02.144 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 59.721672126999998, 2024-09-13T14:47:02.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23060863600000001, 2024-09-13T14:47:02.144 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:02.145 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:02.145 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:02.145 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:02.145 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:47:02.145 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:47:02.145 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 138848 2024-09-13T14:47:02.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.145 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:02.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:02.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.114087+0000", 2024-09-13T14:47:02.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:02.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.114087+0000", 2024-09-13T14:47:02.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:02.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.114088+0000", 2024-09-13T14:47:02.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.843e-06 2024-09-13T14:47:02.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:02.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.114095+0000", 2024-09-13T14:47:02.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4060000000000002e-06 2024-09-13T14:47:02.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:02.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.114116+0000", 2024-09-13T14:47:02.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1531000000000001e-05 2024-09-13T14:47:02.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:02.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.114127+0000", 2024-09-13T14:47:02.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0249e-05 2024-09-13T14:47:02.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:02.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.134442+0000", 2024-09-13T14:47:02.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020315100999999999 2024-09-13T14:47:02.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:47:02.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.134563+0000", 2024-09-13T14:47:02.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000121106 2024-09-13T14:47:02.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:02.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.229689+0000", 2024-09-13T14:47:02.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.095126393000000004 2024-09-13T14:47:02.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:02.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.230637+0000", 2024-09-13T14:47:02.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000947776 2024-09-13T14:47:02.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:47:02.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.230803+0000", 2024-09-13T14:47:02.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000165766 2024-09-13T14:47:02.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:02.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.306045+0000", 2024-09-13T14:47:02.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.075242017999999994 2024-09-13T14:47:02.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:02.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.306137+0000", 2024-09-13T14:47:02.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2235e-05 2024-09-13T14:47:02.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:02.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.334291+0000", 2024-09-13T14:47:02.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.028153556999999999 2024-09-13T14:47:02.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:02.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.334353+0000", 2024-09-13T14:47:02.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2051999999999995e-05 2024-09-13T14:47:02.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:02.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.344695+0000", 2024-09-13T14:47:02.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010342603000000001 2024-09-13T14:47:02.151 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.151 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:02.151 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.152 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:139236 7.3 7:cbe4457b:::benchmark_data_smithi151_190906_object18775:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1155)", 2024-09-13T14:47:02.152 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:46:02.783805+0000", 2024-09-13T14:47:02.152 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 59.051953363000003, 2024-09-13T14:47:02.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.30471862300000002, 2024-09-13T14:47:02.152 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:02.152 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:02.152 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:02.152 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:02.152 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:47:02.152 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:47:02.152 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 139236 2024-09-13T14:47:02.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.152 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:02.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:02.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.783805+0000", 2024-09-13T14:47:02.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:02.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.783805+0000", 2024-09-13T14:47:02.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:02.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.783807+0000", 2024-09-13T14:47:02.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.832e-06 2024-09-13T14:47:02.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:02.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.783814+0000", 2024-09-13T14:47:02.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9360000000000002e-06 2024-09-13T14:47:02.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:02.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.783830+0000", 2024-09-13T14:47:02.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6121e-05 2024-09-13T14:47:02.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:02.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.783838+0000", 2024-09-13T14:47:02.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1720000000000007e-06 2024-09-13T14:47:02.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:02.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.809208+0000", 2024-09-13T14:47:02.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.025369532 2024-09-13T14:47:02.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:47:02.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.809372+0000", 2024-09-13T14:47:02.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000163957 2024-09-13T14:47:02.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:02.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.873434+0000", 2024-09-13T14:47:02.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.064061877000000003 2024-09-13T14:47:02.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:02.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.873660+0000", 2024-09-13T14:47:02.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000226476 2024-09-13T14:47:02.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:47:02.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.874058+0000", 2024-09-13T14:47:02.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039782800000000003 2024-09-13T14:47:02.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:02.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.905971+0000", 2024-09-13T14:47:02.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.031912425000000001 2024-09-13T14:47:02.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:02.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.906014+0000", 2024-09-13T14:47:02.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3068000000000002e-05 2024-09-13T14:47:02.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:02.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.938630+0000", 2024-09-13T14:47:02.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.032616457000000001 2024-09-13T14:47:02.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:02.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.938683+0000", 2024-09-13T14:47:02.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2689999999999999e-05 2024-09-13T14:47:02.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:02.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:03.088524+0000", 2024-09-13T14:47:02.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.14984125200000001 2024-09-13T14:47:02.158 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.159 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:02.159 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.159 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:139277 7.3 7:c94ac58f:::benchmark_data_smithi151_190906_object18816:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1155)", 2024-09-13T14:47:02.159 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:46:02.841626+0000", 2024-09-13T14:47:02.159 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 58.994132649000001, 2024-09-13T14:47:02.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.254796672, 2024-09-13T14:47:02.159 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:02.159 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:02.159 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:02.159 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:02.159 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:47:02.159 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:47:02.160 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 139277 2024-09-13T14:47:02.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.160 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:02.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:02.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.841626+0000", 2024-09-13T14:47:02.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:02.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.841626+0000", 2024-09-13T14:47:02.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:02.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.841627+0000", 2024-09-13T14:47:02.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.71e-07 2024-09-13T14:47:02.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:02.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.841631+0000", 2024-09-13T14:47:02.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3810000000000001e-06 2024-09-13T14:47:02.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:02.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.841646+0000", 2024-09-13T14:47:02.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4501000000000001e-05 2024-09-13T14:47:02.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:02.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.841653+0000", 2024-09-13T14:47:02.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4800000000000004e-06 2024-09-13T14:47:02.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:02.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.879710+0000", 2024-09-13T14:47:02.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038056476999999998 2024-09-13T14:47:02.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:02.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.879865+0000", 2024-09-13T14:47:02.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000155086 2024-09-13T14:47:02.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:47:02.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.880026+0000", 2024-09-13T14:47:02.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016150800000000001 2024-09-13T14:47:02.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:02.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.906029+0000", 2024-09-13T14:47:02.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.026002214999999999 2024-09-13T14:47:02.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:02.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.906337+0000", 2024-09-13T14:47:02.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00030863 2024-09-13T14:47:02.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:02.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:03.088605+0000", 2024-09-13T14:47:02.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18226777 2024-09-13T14:47:02.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:02.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:03.088677+0000", 2024-09-13T14:47:02.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1754000000000005e-05 2024-09-13T14:47:02.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:02.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:03.096423+0000", 2024-09-13T14:47:02.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0077459989999999999 2024-09-13T14:47:02.165 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.165 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:02.165 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.165 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(7.3 1155/1149 [PushReplyOp(7:c8bb2d1f:::benchmark_data_smithi151_190906_object18871:head)])", 2024-09-13T14:47:02.165 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:46:02.919133+0000", 2024-09-13T14:47:02.165 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 58.916625811999999, 2024-09-13T14:47:02.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22932897199999999, 2024-09-13T14:47:02.166 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:02.166 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:02.166 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-13T14:47:02.166 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:02.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:02.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.919133+0000", 2024-09-13T14:47:02.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:02.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.919133+0000", 2024-09-13T14:47:02.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:02.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.919134+0000", 2024-09-13T14:47:02.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7199999999999997e-07 2024-09-13T14:47:02.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:02.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.919140+0000", 2024-09-13T14:47:02.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.003e-06 2024-09-13T14:47:02.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:02.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.919151+0000", 2024-09-13T14:47:02.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1246e-05 2024-09-13T14:47:02.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:02.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.919159+0000", 2024-09-13T14:47:02.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4889999999999998e-06 2024-09-13T14:47:02.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:02.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:03.132812+0000", 2024-09-13T14:47:02.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21365309099999999 2024-09-13T14:47:02.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:02.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:03.148462+0000", 2024-09-13T14:47:02.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015650271 2024-09-13T14:47:02.169 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.169 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:02.169 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.169 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:139332 7.3 7:c8bb2d1f:::benchmark_data_smithi151_190906_object18871:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1155)", 2024-09-13T14:47:02.169 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:46:02.931012+0000", 2024-09-13T14:47:02.169 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 58.904746697, 2024-09-13T14:47:02.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27453928100000002, 2024-09-13T14:47:02.170 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:02.170 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:02.170 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:02.170 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:02.170 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:47:02.170 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:47:02.170 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 139332 2024-09-13T14:47:02.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.170 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:02.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:02.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.931012+0000", 2024-09-13T14:47:02.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:02.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.931012+0000", 2024-09-13T14:47:02.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:02.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.931013+0000", 2024-09-13T14:47:02.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2640000000000001e-06 2024-09-13T14:47:02.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:02.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.931018+0000", 2024-09-13T14:47:02.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7260000000000002e-06 2024-09-13T14:47:02.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:02.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.931029+0000", 2024-09-13T14:47:02.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1079e-05 2024-09-13T14:47:02.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:02.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:02.931036+0000", 2024-09-13T14:47:02.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9890000000000003e-06 2024-09-13T14:47:02.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:02.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:03.096757+0000", 2024-09-13T14:47:02.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16572062500000001 2024-09-13T14:47:02.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:47:02.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:03.097886+0000", 2024-09-13T14:47:02.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011294639999999999 2024-09-13T14:47:02.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:02.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:03.148498+0000", 2024-09-13T14:47:02.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050612179 2024-09-13T14:47:02.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:02.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:03.148601+0000", 2024-09-13T14:47:02.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102608 2024-09-13T14:47:02.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:47:02.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:03.148754+0000", 2024-09-13T14:47:02.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015299000000000001 2024-09-13T14:47:02.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:02.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:03.163777+0000", 2024-09-13T14:47:02.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01502303 2024-09-13T14:47:02.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:02.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:03.170801+0000", 2024-09-13T14:47:02.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0070236489999999999 2024-09-13T14:47:02.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:02.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:03.200971+0000", 2024-09-13T14:47:02.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.030169976000000001 2024-09-13T14:47:02.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:02.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:03.201034+0000", 2024-09-13T14:47:02.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3820999999999998e-05 2024-09-13T14:47:02.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:02.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:03.205551+0000", 2024-09-13T14:47:02.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0045168810000000004 2024-09-13T14:47:02.176 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.177 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:02.177 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.177 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:141665 7.3 7:ca41c4fd:::benchmark_data_smithi151_190906_object21204:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1155)", 2024-09-13T14:47:02.177 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:46:07.151964+0000", 2024-09-13T14:47:02.177 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 54.683794360999997, 2024-09-13T14:47:02.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.234282925, 2024-09-13T14:47:02.177 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:02.177 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:02.177 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:02.177 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:02.177 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:47:02.178 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:47:02.178 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 141665 2024-09-13T14:47:02.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.178 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:02.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:02.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:07.151964+0000", 2024-09-13T14:47:02.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:02.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:07.151964+0000", 2024-09-13T14:47:02.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:02.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:07.151965+0000", 2024-09-13T14:47:02.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7199999999999999e-07 2024-09-13T14:47:02.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:02.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:07.151968+0000", 2024-09-13T14:47:02.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5349999999999999e-06 2024-09-13T14:47:02.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:02.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:07.151975+0000", 2024-09-13T14:47:02.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0249999999999997e-06 2024-09-13T14:47:02.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:02.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:07.151980+0000", 2024-09-13T14:47:02.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7350000000000001e-06 2024-09-13T14:47:02.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:02.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:07.162110+0000", 2024-09-13T14:47:02.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010129454 2024-09-13T14:47:02.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:47:02.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:07.162225+0000", 2024-09-13T14:47:02.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000115277 2024-09-13T14:47:02.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:02.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:07.292043+0000", 2024-09-13T14:47:02.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.12981763099999999 2024-09-13T14:47:02.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:02.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:07.292158+0000", 2024-09-13T14:47:02.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011485999999999999 2024-09-13T14:47:02.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:47:02.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:07.292326+0000", 2024-09-13T14:47:02.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016893 2024-09-13T14:47:02.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:02.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:07.319421+0000", 2024-09-13T14:47:02.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.027094927000000001 2024-09-13T14:47:02.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:02.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:07.319445+0000", 2024-09-13T14:47:02.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3920000000000001e-05 2024-09-13T14:47:02.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:02.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:07.378060+0000", 2024-09-13T14:47:02.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.058614982000000003 2024-09-13T14:47:02.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:02.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:07.378306+0000", 2024-09-13T14:47:02.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024604500000000002 2024-09-13T14:47:02.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:02.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:07.386247+0000", 2024-09-13T14:47:02.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0079410320000000006 2024-09-13T14:47:02.184 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.184 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:02.184 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.184 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:162147 7.3 7:c29c871e:::benchmark_data_smithi151_190906_object41686:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1157)", 2024-09-13T14:47:02.184 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:46:40.247128+0000", 2024-09-13T14:47:02.184 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 21.588630650999999, 2024-09-13T14:47:02.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.242284571, 2024-09-13T14:47:02.184 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:02.184 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:02.184 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:02.185 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:02.185 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:47:02.185 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:47:02.185 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 162147 2024-09-13T14:47:02.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.185 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:02.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:02.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.247128+0000", 2024-09-13T14:47:02.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:02.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.247128+0000", 2024-09-13T14:47:02.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:02.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.247129+0000", 2024-09-13T14:47:02.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5600000000000004e-07 2024-09-13T14:47:02.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:02.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.247134+0000", 2024-09-13T14:47:02.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.809e-06 2024-09-13T14:47:02.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:02.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.247144+0000", 2024-09-13T14:47:02.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0403e-05 2024-09-13T14:47:02.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:02.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.247151+0000", 2024-09-13T14:47:02.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7490000000000001e-06 2024-09-13T14:47:02.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:02.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.299222+0000", 2024-09-13T14:47:02.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052070524999999999 2024-09-13T14:47:02.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:47:02.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.299327+0000", 2024-09-13T14:47:02.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105352 2024-09-13T14:47:02.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:02.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.353876+0000", 2024-09-13T14:47:02.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054549225 2024-09-13T14:47:02.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:02.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.353973+0000", 2024-09-13T14:47:02.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6694000000000006e-05 2024-09-13T14:47:02.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:47:02.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.354116+0000", 2024-09-13T14:47:02.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000143177 2024-09-13T14:47:02.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:02.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.372861+0000", 2024-09-13T14:47:02.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018745370000000001 2024-09-13T14:47:02.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:02.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.372886+0000", 2024-09-13T14:47:02.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.51e-05 2024-09-13T14:47:02.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:02.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.484610+0000", 2024-09-13T14:47:02.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.111723181 2024-09-13T14:47:02.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:02.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.484817+0000", 2024-09-13T14:47:02.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00020762299999999999 2024-09-13T14:47:02.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:02.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.489413+0000", 2024-09-13T14:47:02.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0045954070000000001 2024-09-13T14:47:02.191 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.191 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:02.191 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.191 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:162218 7.3 7:c1045da1:::benchmark_data_smithi151_190906_object41757:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1157)", 2024-09-13T14:47:02.191 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:46:40.352652+0000", 2024-09-13T14:47:02.191 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 21.483106363000001, 2024-09-13T14:47:02.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23107936000000001, 2024-09-13T14:47:02.192 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:02.192 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:02.192 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:02.192 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:02.192 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:47:02.192 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:47:02.192 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 162218 2024-09-13T14:47:02.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.192 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:02.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:02.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.352652+0000", 2024-09-13T14:47:02.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:02.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.352652+0000", 2024-09-13T14:47:02.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:02.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.352654+0000", 2024-09-13T14:47:02.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.77e-06 2024-09-13T14:47:02.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:02.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.352660+0000", 2024-09-13T14:47:02.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1369999999999999e-06 2024-09-13T14:47:02.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:02.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.352673+0000", 2024-09-13T14:47:02.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2559e-05 2024-09-13T14:47:02.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:02.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.352680+0000", 2024-09-13T14:47:02.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0539999999999997e-06 2024-09-13T14:47:02.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:02.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.500615+0000", 2024-09-13T14:47:02.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.147935396 2024-09-13T14:47:02.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:02.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.500777+0000", 2024-09-13T14:47:02.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000161538 2024-09-13T14:47:02.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:47:02.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.500950+0000", 2024-09-13T14:47:02.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017353099999999999 2024-09-13T14:47:02.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.196 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:02.196 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.515844+0000", 2024-09-13T14:47:02.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014893692 2024-09-13T14:47:02.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.196 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:02.196 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.515864+0000", 2024-09-13T14:47:02.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9860999999999998e-05 2024-09-13T14:47:02.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.197 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:02.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.579248+0000", 2024-09-13T14:47:02.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.063384519 2024-09-13T14:47:02.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.197 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.197 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:02.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.579324+0000", 2024-09-13T14:47:02.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5087999999999997e-05 2024-09-13T14:47:02.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.197 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.197 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:02.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.583732+0000", 2024-09-13T14:47:02.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0044082150000000001 2024-09-13T14:47:02.197 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.197 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:02.198 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.198 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.198 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:162219 7.3 7:cc0c1a43:::benchmark_data_smithi151_190906_object41758:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1157)", 2024-09-13T14:47:02.198 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:46:40.353166+0000", 2024-09-13T14:47:02.198 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 21.482592562000001, 2024-09-13T14:47:02.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23457676699999999, 2024-09-13T14:47:02.198 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:02.198 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:02.198 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:02.198 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:02.198 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:47:02.198 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:47:02.198 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 162219 2024-09-13T14:47:02.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.199 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:02.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.199 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:02.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.353166+0000", 2024-09-13T14:47:02.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.199 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:02.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.353166+0000", 2024-09-13T14:47:02.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:02.200 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.353167+0000", 2024-09-13T14:47:02.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0200000000000001e-07 2024-09-13T14:47:02.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:02.200 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.353171+0000", 2024-09-13T14:47:02.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8700000000000002e-06 2024-09-13T14:47:02.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:02.200 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.353181+0000", 2024-09-13T14:47:02.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0295000000000001e-05 2024-09-13T14:47:02.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:02.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.353188+0000", 2024-09-13T14:47:02.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4799999999999998e-06 2024-09-13T14:47:02.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:02.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.505514+0000", 2024-09-13T14:47:02.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.15232675900000001 2024-09-13T14:47:02.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:02.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.505675+0000", 2024-09-13T14:47:02.202 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000160421 2024-09-13T14:47:02.202 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.202 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:47:02.202 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.506759+0000", 2024-09-13T14:47:02.202 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010844889999999999 2024-09-13T14:47:02.202 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.202 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:02.202 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.515870+0000", 2024-09-13T14:47:02.202 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0091106179999999991 2024-09-13T14:47:02.202 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.202 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:02.203 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.515941+0000", 2024-09-13T14:47:02.203 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1241000000000005e-05 2024-09-13T14:47:02.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.203 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:02.203 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.583805+0000", 2024-09-13T14:47:02.203 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.067863772000000003 2024-09-13T14:47:02.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.203 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:02.203 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.583865+0000", 2024-09-13T14:47:02.203 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0414000000000001e-05 2024-09-13T14:47:02.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.204 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.204 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:02.204 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.587743+0000", 2024-09-13T14:47:02.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0038777059999999999 2024-09-13T14:47:02.204 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.204 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:02.204 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.204 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.204 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.204 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:162220 7.3 7:c8763aa7:::benchmark_data_smithi151_190906_object41759:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1157)", 2024-09-13T14:47:02.204 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:46:40.353393+0000", 2024-09-13T14:47:02.204 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 21.482365340000001, 2024-09-13T14:47:02.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24034249599999999, 2024-09-13T14:47:02.205 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:02.205 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:02.205 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:02.205 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:02.205 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:47:02.205 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:47:02.205 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 162220 2024-09-13T14:47:02.205 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.205 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:02.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.205 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:02.205 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.353393+0000", 2024-09-13T14:47:02.205 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.205 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.206 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.206 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:02.206 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.353393+0000", 2024-09-13T14:47:02.206 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.206 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.206 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.206 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:02.206 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.353394+0000", 2024-09-13T14:47:02.206 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.06e-07 2024-09-13T14:47:02.206 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.206 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.206 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:02.206 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.353398+0000", 2024-09-13T14:47:02.206 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5470000000000001e-06 2024-09-13T14:47:02.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.207 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:02.207 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.353407+0000", 2024-09-13T14:47:02.207 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1549999999999996e-06 2024-09-13T14:47:02.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.207 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:02.207 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.353413+0000", 2024-09-13T14:47:02.207 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0909999999999998e-06 2024-09-13T14:47:02.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.207 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:02.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.511100+0000", 2024-09-13T14:47:02.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.157687402 2024-09-13T14:47:02.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:02.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.511289+0000", 2024-09-13T14:47:02.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018829200000000001 2024-09-13T14:47:02.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:47:02.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.511433+0000", 2024-09-13T14:47:02.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014485999999999999 2024-09-13T14:47:02.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.209 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:02.209 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.536990+0000", 2024-09-13T14:47:02.209 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.025556399 2024-09-13T14:47:02.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.209 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:02.209 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.537214+0000", 2024-09-13T14:47:02.209 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000224188 2024-09-13T14:47:02.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.209 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:02.209 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.587812+0000", 2024-09-13T14:47:02.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050598320000000002 2024-09-13T14:47:02.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:02.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.587916+0000", 2024-09-13T14:47:02.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103459 2024-09-13T14:47:02.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:02.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.593736+0000", 2024-09-13T14:47:02.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0058201770000000002 2024-09-13T14:47:02.210 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.210 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:02.210 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.211 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.211 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:162224 7.3 7:c0443e81:::benchmark_data_smithi151_190906_object41763:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1157)", 2024-09-13T14:47:02.211 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:46:40.356804+0000", 2024-09-13T14:47:02.211 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 21.478954849000001, 2024-09-13T14:47:02.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24975730199999999, 2024-09-13T14:47:02.211 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:02.211 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:02.211 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:02.211 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:02.211 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:47:02.211 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:47:02.211 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 162224 2024-09-13T14:47:02.211 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.211 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:02.212 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.212 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:02.212 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.356804+0000", 2024-09-13T14:47:02.212 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.212 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.212 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.212 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:02.212 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.356804+0000", 2024-09-13T14:47:02.212 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.212 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.212 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.212 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:02.213 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.356805+0000", 2024-09-13T14:47:02.213 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5600000000000001e-06 2024-09-13T14:47:02.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.213 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.213 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:02.213 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.356811+0000", 2024-09-13T14:47:02.213 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5249999999999996e-06 2024-09-13T14:47:02.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.213 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.213 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:02.213 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.356823+0000", 2024-09-13T14:47:02.213 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1633e-05 2024-09-13T14:47:02.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.213 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:02.214 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.356830+0000", 2024-09-13T14:47:02.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1040000000000001e-06 2024-09-13T14:47:02.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:02.214 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.515979+0000", 2024-09-13T14:47:02.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.159149242 2024-09-13T14:47:02.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:02.214 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.516411+0000", 2024-09-13T14:47:02.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00043246499999999999 2024-09-13T14:47:02.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.215 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.215 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:47:02.215 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.516819+0000", 2024-09-13T14:47:02.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00040799299999999999 2024-09-13T14:47:02.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.215 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.215 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:02.215 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.537146+0000", 2024-09-13T14:47:02.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.02032701 2024-09-13T14:47:02.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.215 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.215 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:02.215 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.537276+0000", 2024-09-13T14:47:02.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012997799999999999 2024-09-13T14:47:02.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.216 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:02.216 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.602255+0000", 2024-09-13T14:47:02.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.064978452000000006 2024-09-13T14:47:02.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.216 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:02.216 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.602304+0000", 2024-09-13T14:47:02.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8832999999999998e-05 2024-09-13T14:47:02.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:02.217 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:40.606561+0000", 2024-09-13T14:47:02.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0042575069999999998 2024-09-13T14:47:02.217 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.217 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:02.217 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.217 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:162611 7.3 7:c4cfa36a:::benchmark_data_smithi151_190906_object42150:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1157)", 2024-09-13T14:47:02.217 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:46:41.037979+0000", 2024-09-13T14:47:02.217 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 20.797780199999998, 2024-09-13T14:47:02.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27731364200000003, 2024-09-13T14:47:02.217 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:02.217 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:02.218 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:02.218 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:02.218 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:47:02.218 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:47:02.218 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 162611 2024-09-13T14:47:02.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.218 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:02.218 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.218 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:02.218 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:41.037979+0000", 2024-09-13T14:47:02.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.218 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.218 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:02.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:41.037979+0000", 2024-09-13T14:47:02.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:02.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:02.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:41.037980+0000", 2024-09-13T14:47:02.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2550000000000001e-06 2024-09-13T14:47:02.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:02.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:41.037984+0000", 2024-09-13T14:47:02.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3549999999999998e-06 2024-09-13T14:47:02.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:02.220 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:41.037992+0000", 2024-09-13T14:47:02.220 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6399999999999997e-06 2024-09-13T14:47:02.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.220 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.220 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:02.220 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:41.037996+0000", 2024-09-13T14:47:02.220 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.639e-06 2024-09-13T14:47:02.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.220 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.220 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:02.220 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:41.040049+0000", 2024-09-13T14:47:02.220 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002052379 2024-09-13T14:47:02.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.221 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:47:02.221 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:41.040175+0000", 2024-09-13T14:47:02.221 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000125967 2024-09-13T14:47:02.221 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.221 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:02.221 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:41.212915+0000", 2024-09-13T14:47:02.221 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17273999400000001 2024-09-13T14:47:02.221 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.221 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:02.221 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:41.213016+0000", 2024-09-13T14:47:02.221 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010132300000000001 2024-09-13T14:47:02.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.222 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:47:02.222 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:41.213170+0000", 2024-09-13T14:47:02.222 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000153857 2024-09-13T14:47:02.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.222 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:02.222 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:41.219291+0000", 2024-09-13T14:47:02.222 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0061206029999999996 2024-09-13T14:47:02.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.222 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:02.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:41.224057+0000", 2024-09-13T14:47:02.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0047661980000000001 2024-09-13T14:47:02.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.223 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:02.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:41.310209+0000", 2024-09-13T14:47:02.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.086152307999999997 2024-09-13T14:47:02.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.223 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:02.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:41.310493+0000", 2024-09-13T14:47:02.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028412400000000002 2024-09-13T14:47:02.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:02.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:02.224 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:02.224 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:46:41.315292+0000", 2024-09-13T14:47:02.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0047990000000000003 2024-09-13T14:47:02.224 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.224 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:02.224 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.224 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:02.224 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:02.224 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:47:02.224 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:02.225 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:01 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:02.270 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:02.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:02.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:02.372 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:02.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:02.414 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:02.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:02.515 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:02.539 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:02.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:02.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:02.641 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:02.690 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:02.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:02.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:02.792 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:02.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:02.850 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:02.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:02.951 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:02.975 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:02.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:02.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:03.077 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:03.100 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:03.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:03.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:03.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:02 smithi184 ceph-mon[92527]: pgmap v1592: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 587 op/s; 2110/201522 objects degraded (1.047%); 40327/201522 objects misplaced (20.011%); 1.1 MiB/s, 16 objects/s recovering 2024-09-13T14:47:03.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:02 smithi151 ceph-mon[104450]: pgmap v1592: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 587 op/s; 2110/201522 objects degraded (1.047%); 40327/201522 objects misplaced (20.011%); 1.1 MiB/s, 16 objects/s recovering 2024-09-13T14:47:03.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:02 smithi151 ceph-mon[108966]: pgmap v1592: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 587 op/s; 2110/201522 objects degraded (1.047%); 40327/201522 objects misplaced (20.011%); 1.1 MiB/s, 16 objects/s recovering 2024-09-13T14:47:03.202 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:03.226 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:03.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:03.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:03.328 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:03.354 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:03.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:03.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:03.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:03.499 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:03.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:03.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:03.600 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:03.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:03.648 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:03.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:03.749 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:03.772 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:03.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:03.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:03.782 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T14:47:03.874 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:03.897 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:03.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:03.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:03.999 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:04.022 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:04.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:04.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:04.124 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:04.150 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:04.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:04.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:04.251 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:04.275 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:04.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:04.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:04.376 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:04.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:04.427 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:04.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:04.528 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:04.598 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:04.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:04.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:04.663 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:04.664 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:04.682 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:04.700 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:04.723 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:04 smithi151 ceph-mon[104450]: pgmap v1593: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 641 op/s; 2110/197232 objects degraded (1.070%); 39704/197232 objects misplaced (20.131%); 736 KiB/s, 11 objects/s recovering 2024-09-13T14:47:04.723 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:47:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:47:04] "GET /metrics HTTP/1.1" 200 36663 "" "Prometheus/2.51.0" 2024-09-13T14:47:04.744 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:04.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:04.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:04.846 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:04.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:04.884 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:04.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:04.985 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:04.991 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:04 smithi151 ceph-mon[108966]: pgmap v1593: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 641 op/s; 2110/197232 objects degraded (1.070%); 39704/197232 objects misplaced (20.131%); 736 KiB/s, 11 objects/s recovering 2024-09-13T14:47:05.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:05.032 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:05.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:05.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:04 smithi184 ceph-mon[92527]: pgmap v1593: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 641 op/s; 2110/197232 objects degraded (1.070%); 39704/197232 objects misplaced (20.131%); 736 KiB/s, 11 objects/s recovering 2024-09-13T14:47:05.133 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:05.156 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:05.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:05.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:05.258 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:05.311 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:05.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:05.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:05.413 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:05.462 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:05.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:05.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:05.564 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:05.612 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:05.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:05.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:05.714 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:05.751 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-13T14:47:05.767 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:05.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:05.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:05.869 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:05.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:05.905 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:05.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:06.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:06.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:06.062 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:06.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:06.070 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:05 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 2110/197232 objects degraded (1.070%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:06.070 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:05 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/976234542' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:06.070 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:05 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:06.071 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:05 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 2110/197232 objects degraded (1.070%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:06.071 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:05 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/976234542' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:06.071 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:05 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:06.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:05 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 2110/197232 objects degraded (1.070%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:06.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:05 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/976234542' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:06.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:05 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:06.163 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:06.189 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:06.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:06.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:06.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:06.355 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:06.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:06.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:06.457 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:06.483 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:06.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:06.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:06.584 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:06.612 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:06.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:06.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:06.714 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:06.738 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:06.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:06.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:06.840 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:06.882 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:06.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:06.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:06.983 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:07.009 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:07.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:07.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:07.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:06 smithi184 ceph-mon[92527]: pgmap v1594: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 608 op/s; 2110/194286 objects degraded (1.086%); 39104/194286 objects misplaced (20.127%); 736 KiB/s, 11 objects/s recovering 2024-09-13T14:47:07.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:06 smithi184 ceph-mon[92527]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:47:07.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:06 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:47:07.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:06 smithi184 ceph-mon[92527]: osdmap e1159: 8 total, 8 up, 8 in 2024-09-13T14:47:07.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:06 smithi151 ceph-mon[104450]: pgmap v1594: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 608 op/s; 2110/194286 objects degraded (1.086%); 39104/194286 objects misplaced (20.127%); 736 KiB/s, 11 objects/s recovering 2024-09-13T14:47:07.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:06 smithi151 ceph-mon[104450]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:47:07.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:06 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:47:07.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:06 smithi151 ceph-mon[104450]: osdmap e1159: 8 total, 8 up, 8 in 2024-09-13T14:47:07.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:06 smithi151 ceph-mon[108966]: pgmap v1594: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 608 op/s; 2110/194286 objects degraded (1.086%); 39104/194286 objects misplaced (20.127%); 736 KiB/s, 11 objects/s recovering 2024-09-13T14:47:07.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:06 smithi151 ceph-mon[108966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:47:07.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:06 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:47:07.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:06 smithi151 ceph-mon[108966]: osdmap e1159: 8 total, 8 up, 8 in 2024-09-13T14:47:07.111 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:07.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:07.154 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:07.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:07.255 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:07.293 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:07.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:07.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:07.395 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:07.435 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:07.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:07.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:07.537 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:07.561 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:07.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:07.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:07.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:07.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:07.699 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:07.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:07.800 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:07.825 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:07.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:07.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:07.927 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:07.973 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:07.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:07.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:08.075 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:08.114 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:08.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:08.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:08.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:08.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:08.268 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:08.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:08.369 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:08.421 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:08.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:08.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:08.523 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:08.545 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:08.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:08.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:08.647 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:08.670 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:08.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:08.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:08.772 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:08.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:08.818 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:08.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:08.919 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:08.942 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:08.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:08.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:09.028 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:08 smithi151 ceph-mon[104450]: pgmap v1596: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 624 op/s; 2218/192213 objects degraded (1.154%); 38750/192213 objects misplaced (20.160%); 876 KiB/s, 13 objects/s recovering 2024-09-13T14:47:09.028 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:08 smithi151 ceph-mon[108966]: pgmap v1596: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 624 op/s; 2218/192213 objects degraded (1.154%); 38750/192213 objects misplaced (20.160%); 876 KiB/s, 13 objects/s recovering 2024-09-13T14:47:09.044 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:09.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:09.085 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:09.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:09.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:08 smithi184 ceph-mon[92527]: pgmap v1596: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 624 op/s; 2218/192213 objects degraded (1.154%); 38750/192213 objects misplaced (20.160%); 876 KiB/s, 13 objects/s recovering 2024-09-13T14:47:09.186 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:09.211 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:09.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:09.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:09.289 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:47:09.312 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:09.336 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:09.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:09.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:09.438 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:09.460 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:09.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:09.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:09.561 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:09.585 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:09.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:09.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:09.686 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:09.723 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:09.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:09.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:09.825 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:09.853 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:09.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:09.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:09.955 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:09.976 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:09.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:09.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:10.078 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:10.113 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:10.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:10.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:10.214 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:10.239 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:10.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:10.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:10.341 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:10.364 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:10.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:10.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:10.466 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:10.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:10.500 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:10.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:10.601 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:10.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:47:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:47:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:47:10.625 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:10.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:10.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:10.727 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:10.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:10.788 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:10.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:10.788 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:10 smithi151 ceph-mon[104450]: pgmap v1597: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 608 op/s; 2218/185523 objects degraded (1.196%); 37664/185523 objects misplaced (20.302%); 876 KiB/s, 13 objects/s recovering 2024-09-13T14:47:10.789 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:10 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 2218/185523 objects degraded (1.196%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:10.789 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:10 smithi151 ceph-mon[108966]: pgmap v1597: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 608 op/s; 2218/185523 objects degraded (1.196%); 37664/185523 objects misplaced (20.302%); 876 KiB/s, 13 objects/s recovering 2024-09-13T14:47:10.789 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:10 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 2218/185523 objects degraded (1.196%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:10.889 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:10.911 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:10.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:10.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:11.013 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:11.036 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:11.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:11.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:11.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:10 smithi184 ceph-mon[92527]: pgmap v1597: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 608 op/s; 2218/185523 objects degraded (1.196%); 37664/185523 objects misplaced (20.302%); 876 KiB/s, 13 objects/s recovering 2024-09-13T14:47:11.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:10 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 2218/185523 objects degraded (1.196%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:11.138 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:11.163 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:11.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:11.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:11.265 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:11.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:11.318 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:11.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:11.419 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:11.471 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:11.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:11.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:11.573 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:11.597 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:11.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:11.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:11.699 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:11.745 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:11.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:11.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:11.841 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:47:11.842 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:47:11.842 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:47:11.842 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:47:11.842 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:47:11.842 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:47:11.842 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:47:11.842 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:47:11.842 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:47:11.842 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:47:11.843 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:47:11.843 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:47:11.843 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:47:11.843 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:47:11.843 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:47:11.843 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:47:11.843 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:11 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/304186800' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:11.843 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:11 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/304186800' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:11.847 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:11.870 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:11.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:11.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:11.972 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:12.007 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:12.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:12.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:12.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:11 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/304186800' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:12.108 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:12.131 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:12.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:12.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:12.233 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:12.259 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:12.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:12.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:12.319 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:47:12.319 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:47:12.360 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:12.385 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:12.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:12.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:12.487 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:12.533 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:12.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:12.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:12.635 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:12.658 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:12.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:12.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:12.761 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:12.785 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:12.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:12.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:12.886 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:12.921 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:12.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:12.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:13.023 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:13.070 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:13.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:13.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:13.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:12 smithi184 ceph-mon[92527]: pgmap v1598: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 532 op/s; 2218/185523 objects degraded (1.196%); 37664/185523 objects misplaced (20.302%); 461 KiB/s, 7 objects/s recovering 2024-09-13T14:47:13.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:12 smithi151 ceph-mon[104450]: pgmap v1598: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 532 op/s; 2218/185523 objects degraded (1.196%); 37664/185523 objects misplaced (20.302%); 461 KiB/s, 7 objects/s recovering 2024-09-13T14:47:13.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:12 smithi151 ceph-mon[108966]: pgmap v1598: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 532 op/s; 2218/185523 objects degraded (1.196%); 37664/185523 objects misplaced (20.302%); 461 KiB/s, 7 objects/s recovering 2024-09-13T14:47:13.172 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:13.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T14:47:13.230 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:13.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:13.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:13.332 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:13.355 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:13.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:13.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:13.457 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:13.479 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:13.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:13.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:13.580 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:13.629 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:13.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:13.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:13.731 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:13.754 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:13.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:13.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:13.856 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:13.879 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:13.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:13.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:13.981 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:14.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:14.019 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:14.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:14.120 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:14.168 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:14.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:14.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:14.269 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:14.291 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:14.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:14.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:14.329 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:14.394 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:14.419 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:14.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:14.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:14.521 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:14.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:14.580 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:14.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:14.587 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:14.681 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:14.707 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:14.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:14.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:14.808 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:14.833 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:14.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:14.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:14 smithi151 ceph-mon[104450]: pgmap v1599: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 602 op/s; 2348/179145 objects degraded (1.311%); 36665/179145 objects misplaced (20.467%); 947 KiB/s, 14 objects/s recovering 2024-09-13T14:47:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:14 smithi151 ceph-mon[108966]: pgmap v1599: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 602 op/s; 2348/179145 objects degraded (1.311%); 36665/179145 objects misplaced (20.467%); 947 KiB/s, 14 objects/s recovering 2024-09-13T14:47:14.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:47:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:47:14] "GET /metrics HTTP/1.1" 200 36663 "" "Prometheus/2.51.0" 2024-09-13T14:47:14.935 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:14.960 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:14.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:14.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:15.062 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:15.086 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:15.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:15.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:14 smithi184 ceph-mon[92527]: pgmap v1599: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 602 op/s; 2348/179145 objects degraded (1.311%); 36665/179145 objects misplaced (20.467%); 947 KiB/s, 14 objects/s recovering 2024-09-13T14:47:15.188 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:15.214 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:15.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:15.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:15.316 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:15.345 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:15.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:15.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:15.447 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:15.471 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:15.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:15.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:15.573 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:15.624 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:15.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:15.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:15.725 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:15.759 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:15.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:15.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:15.861 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:15.883 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:15.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:15.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:15.984 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:16.009 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:16.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:16.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:15 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 2348/179145 objects degraded (1.311%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:16.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:15 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 2348/179145 objects degraded (1.311%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:16.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:15 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 2348/179145 objects degraded (1.311%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:16.111 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:16.144 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:16.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:16.246 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:16.282 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:16.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:16.306 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:47:16.384 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:16.407 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:16.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:16.509 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:16.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.534 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:16.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:16.635 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:16.657 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:16.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:16.759 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:16.782 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:16.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:16.884 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:16.922 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:16.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:17.025 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:17.062 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:17.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:17.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:17.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:16 smithi184 ceph-mon[92527]: pgmap v1600: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 608 op/s; 2348/176019 objects degraded (1.334%); 36044/176019 objects misplaced (20.477%); 947 KiB/s, 14 objects/s recovering 2024-09-13T14:47:17.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:47:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:17.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:16 smithi151 ceph-mon[104450]: pgmap v1600: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 608 op/s; 2348/176019 objects degraded (1.334%); 36044/176019 objects misplaced (20.477%); 947 KiB/s, 14 objects/s recovering 2024-09-13T14:47:17.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:47:17.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:17.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:16 smithi151 ceph-mon[108966]: pgmap v1600: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 608 op/s; 2348/176019 objects degraded (1.334%); 36044/176019 objects misplaced (20.477%); 947 KiB/s, 14 objects/s recovering 2024-09-13T14:47:17.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:47:17.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:17.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:17.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:17.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:17.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:17.312 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:17.351 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:17.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:17.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:17.453 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:17.477 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:17.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:17.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:17.579 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:17.600 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:17.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:17.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:17.702 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:17.736 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:17.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:17.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:17.838 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:17.862 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:17.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:17.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:17.964 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:18.011 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:18.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:18.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:18.113 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:18.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:18.156 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:18.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:18.257 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:18.287 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:18.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:18.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:18.389 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:18.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:18.428 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:18.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:18.529 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:18.566 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:18.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:18.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:18.668 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:18.692 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:18.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:18.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:18.794 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:18.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:18.831 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:18.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:18.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:18 smithi151 ceph-mon[104450]: pgmap v1601: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 584 op/s; 2477/173700 objects degraded (1.426%); 35646/173700 objects misplaced (20.522%); 1.2 MiB/s, 18 objects/s recovering 2024-09-13T14:47:18.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:18 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/4195431840' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:47:18.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:18 smithi151 ceph-mon[108966]: pgmap v1601: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 584 op/s; 2477/173700 objects degraded (1.426%); 35646/173700 objects misplaced (20.522%); 1.2 MiB/s, 18 objects/s recovering 2024-09-13T14:47:18.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:18 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/4195431840' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:47:18.929 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:47:18.929 INFO:teuthology.orchestra.run.smithi151.stdout:{"fsid":"2401fc50-71da-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2477/173700 objects degraded (1.426%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1220,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":1159,"num_osds":8,"num_up_osds":8,"osd_up_since":1726237814,"num_in_osds":8,"osd_in_since":1726238586,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":110},{"state_name":"active+remapped+backfill_wait","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovery_wait+remapped","count":1}],"num_pgs":119,"num_pools":5,"num_objects":57900,"data_bytes":3793387704,"bytes_used":14401245184,"bytes_avail":753559040000,"bytes_total":767960285184,"degraded_objects":2477,"degraded_total":173700,"degraded_ratio":0.014260218767990789,"misplaced_objects":35646,"misplaced_total":173700,"misplaced_ratio":0.20521588946459413,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":1227932,"recovering_keys_per_sec":0,"num_objects_recovered":220,"num_bytes_recovered":14417920,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":584},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi151.rqqklw","status":"up:active","gid":24614}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.151:8443/","prometheus":"http://172.21.15.151:9283/"}},"servicemap":{"epoch":24,"modified":"2024-09-13T14:45:41.438774+0000","services":{}},"progress_events":{"5a616d1f-cba6-436c-9b6b-5733d6f6b5ec":{"message":"Global Recovery Event (0s)\n [............................] ","progress":0,"add_to_ceph_s":true},"b24a386d-2177-4b0f-a83d-e6638be369da":{"message":"Upgrade to 19.3.0-4697-gd58f7306 (4m)\n [============................] (remaining: 6m)","progress":0.44999998807907104,"add_to_ceph_s":true}}} 2024-09-13T14:47:18.933 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:18.954 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:18.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:18.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:19.056 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:19.077 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:19.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:19.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:19.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:18 smithi184 ceph-mon[92527]: pgmap v1601: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 584 op/s; 2477/173700 objects degraded (1.426%); 35646/173700 objects misplaced (20.522%); 1.2 MiB/s, 18 objects/s recovering 2024-09-13T14:47:19.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:18 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/4195431840' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:47:19.180 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:19.228 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:19.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:19.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:19.329 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:19.350 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:19.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:19.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:19.452 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:19.486 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:19.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:19.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:19.588 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:19.615 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:19.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:19.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:19.717 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:19.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:19.741 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:19.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:19.842 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:19.867 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:19.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:19.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:19.969 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:19.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:19.997 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:19.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:20.098 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:20.143 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:20.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:20.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:20.244 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:20.291 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:20.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:20.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:20.393 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:20.415 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:20.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:20.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:20.516 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:20.538 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:20.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:20.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:20.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:47:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:47:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:47:20.640 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:20.664 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:20.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:20.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:20.766 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:20.806 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:20.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:20.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:20.908 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:20.934 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:20.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:20.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:21.036 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:21.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:20 smithi151 ceph-mon[104450]: pgmap v1602: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 697 op/s; 2477/167094 objects degraded (1.482%); 34545/167094 objects misplaced (20.674%); 789 KiB/s, 12 objects/s recovering 2024-09-13T14:47:21.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:20 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 2477/167094 objects degraded (1.482%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:21.058 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:20 smithi151 ceph-mon[108966]: pgmap v1602: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 697 op/s; 2477/167094 objects degraded (1.482%); 34545/167094 objects misplaced (20.674%); 789 KiB/s, 12 objects/s recovering 2024-09-13T14:47:21.058 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:20 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 2477/167094 objects degraded (1.482%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:21.060 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:21.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:21.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:20 smithi184 ceph-mon[92527]: pgmap v1602: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 697 op/s; 2477/167094 objects degraded (1.482%); 34545/167094 objects misplaced (20.674%); 789 KiB/s, 12 objects/s recovering 2024-09-13T14:47:21.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:20 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 2477/167094 objects degraded (1.482%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:21.107 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T14:47:21.107 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:47:21.162 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:21.185 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:21.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:21.287 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:21.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.315 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:21.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:21.416 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:21.442 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:21.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:21.545 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:21.569 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:21.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:21.671 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:21.739 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:21.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:21.841 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:21.870 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:21.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:21.972 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:21.998 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:21.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:22.000 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:21 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:47:22.000 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:21 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:47:22.100 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:22.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:21 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:47:22.141 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:22.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:22.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:22.220 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:22.243 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:22.268 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:22.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:22.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:22.369 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:22.395 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:22.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:22.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:22.497 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:22.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:22.524 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:22.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:22.625 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:22.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:22.658 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:22.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:22.758 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:22.782 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:22.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:22.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:22.884 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:22.930 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:22 smithi151 ceph-mon[104450]: pgmap v1603: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 511 op/s; 2477/167094 objects degraded (1.482%); 34545/167094 objects misplaced (20.674%); 789 KiB/s, 12 objects/s recovering 2024-09-13T14:47:22.930 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:22 smithi151 ceph-mon[108966]: pgmap v1603: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 511 op/s; 2477/167094 objects degraded (1.482%); 34545/167094 objects misplaced (20.674%); 789 KiB/s, 12 objects/s recovering 2024-09-13T14:47:22.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:22.945 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:22.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:23.047 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:23.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:23.073 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:23.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:22 smithi184 ceph-mon[92527]: pgmap v1603: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 511 op/s; 2477/167094 objects degraded (1.482%); 34545/167094 objects misplaced (20.674%); 789 KiB/s, 12 objects/s recovering 2024-09-13T14:47:23.174 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:23.202 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:23.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:23.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:23.304 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:23.330 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:23.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:23.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:23.432 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:23.456 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:23.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:23.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:23.558 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:23.593 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:23.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:23.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:23.696 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:23.735 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:23.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:23.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:23.837 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:23.840 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-13T14:47:23.888 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:23.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:23.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:23.991 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:24.016 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:24.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:24.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:24.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:23 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3416677436' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:47:24.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:23 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:47:24.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:23 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3416677436' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:47:24.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:23 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:47:24.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:23 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3416677436' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:47:24.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:23 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:47:24.118 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:24.160 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:24.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:24.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:24.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:24.321 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:24.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:24.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:24.423 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:24.450 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:24.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:24.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:24.552 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:24.578 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:24.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:24.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:24.680 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:24.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:24.705 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:24.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:24.777 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:47:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:47:24] "GET /metrics HTTP/1.1" 200 36662 "" "Prometheus/2.51.0" 2024-09-13T14:47:24.806 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:24.831 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:24.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:24.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:24.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:24.974 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:24.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:24.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:25.076 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:25.081 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:24 smithi151 ceph-mon[104450]: pgmap v1604: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 685 op/s; 2591/160824 objects degraded (1.611%); 33583/160824 objects misplaced (20.882%); 1.2 MiB/s, 18 objects/s recovering 2024-09-13T14:47:25.081 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:24 smithi151 ceph-mon[104450]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:25.081 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:24 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:47:25.081 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:24 smithi151 ceph-mon[104450]: osdmap e1160: 8 total, 8 up, 8 in 2024-09-13T14:47:25.081 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:24 smithi151 ceph-mon[108966]: pgmap v1604: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 685 op/s; 2591/160824 objects degraded (1.611%); 33583/160824 objects misplaced (20.882%); 1.2 MiB/s, 18 objects/s recovering 2024-09-13T14:47:25.081 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:24 smithi151 ceph-mon[108966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:25.082 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:24 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:47:25.082 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:24 smithi151 ceph-mon[108966]: osdmap e1160: 8 total, 8 up, 8 in 2024-09-13T14:47:25.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:24 smithi184 ceph-mon[92527]: pgmap v1604: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 685 op/s; 2591/160824 objects degraded (1.611%); 33583/160824 objects misplaced (20.882%); 1.2 MiB/s, 18 objects/s recovering 2024-09-13T14:47:25.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:24 smithi184 ceph-mon[92527]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:25.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:24 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:47:25.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:24 smithi184 ceph-mon[92527]: osdmap e1160: 8 total, 8 up, 8 in 2024-09-13T14:47:25.123 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:25.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:25.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:25.225 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:25.249 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:25.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:25.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:25.350 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:25.377 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:25.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:25.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:25.479 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:25.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:25.534 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:25.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:25.635 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:25.673 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:25.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:25.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:25.775 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:25.818 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:25.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:25.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:25.920 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:25.944 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:25.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:25.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:26.046 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:26.085 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:26.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:26.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:26.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:25 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 2591/160824 objects degraded (1.611%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:26.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:25 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1152196329' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:26.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:25 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 2591/160824 objects degraded (1.611%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:26.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:25 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1152196329' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:26.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:25 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 2591/160824 objects degraded (1.611%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:26.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:25 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1152196329' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:26.188 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:26.241 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:26.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:26.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:26.328 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:47:26.329 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:47:26.329 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:47:26.329 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:47:26.329 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:47:26.329 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:47:26.329 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:47:26.329 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:47:26.330 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:47:26.330 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:47:26.330 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:47:26.330 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:47:26.330 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:47:26.330 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:47:26.330 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:47:26.330 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:47:26.343 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:26.385 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:26.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:26.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:26.487 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:26.510 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:26.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:26.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:26.566 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:26.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:26.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:26.667 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:26.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:26.768 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:26.805 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:26.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:26.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:26.906 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:26.962 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:26.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:26.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:27.038 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:26 smithi151 ceph-mon[104450]: pgmap v1606: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 602 op/s; 2591/157944 objects degraded (1.640%); 32996/157944 objects misplaced (20.891%); 941 KiB/s, 14 objects/s recovering 2024-09-13T14:47:27.038 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:26 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:47:27.038 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:26 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:47:27.038 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:26 smithi151 ceph-mon[108966]: pgmap v1606: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 602 op/s; 2591/157944 objects degraded (1.640%); 32996/157944 objects misplaced (20.891%); 941 KiB/s, 14 objects/s recovering 2024-09-13T14:47:27.038 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:26 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:47:27.039 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:26 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:47:27.065 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:27.109 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:27.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:27.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:27.212 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:27.237 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:27.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:27.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:27.338 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:27.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:26 smithi184 ceph-mon[92527]: pgmap v1606: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 602 op/s; 2591/157944 objects degraded (1.640%); 32996/157944 objects misplaced (20.891%); 941 KiB/s, 14 objects/s recovering 2024-09-13T14:47:27.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:26 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:47:27.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:26 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:47:27.382 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:27.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:27.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:27.484 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:27.511 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:27.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:27.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:27.515 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:47:27.515 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-13T14:47:27.515 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-13T14:47:27.515 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-13T14:47:27.515 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:47:27.613 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:27.635 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:27.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:27.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:27.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:27.784 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:27.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:27.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:27.886 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:27.927 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:27.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:27.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:28.029 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:28.051 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:47:28.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:28.055 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:28.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:28.156 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:28.182 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:28.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:28.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:28.270 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T14:47:28.284 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:28.379 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:28.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:28.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:28.481 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:28.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:28.535 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:28.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:28.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:28.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:28.688 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:28.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:28.789 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:28.814 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:28.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:28.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:28.916 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:28.965 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:28.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:28.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:29.067 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:29.089 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:29.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:29.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:29.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:28 smithi151 ceph-mon[104450]: pgmap v1607: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 613 op/s; 2720/155295 objects degraded (1.752%); 32563/155295 objects misplaced (20.968%); 959 KiB/s, 14 objects/s recovering 2024-09-13T14:47:29.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:28 smithi151 ceph-mon[108966]: pgmap v1607: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 613 op/s; 2720/155295 objects degraded (1.752%); 32563/155295 objects misplaced (20.968%); 959 KiB/s, 14 objects/s recovering 2024-09-13T14:47:29.191 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:29.239 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:29.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:29.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:29.263 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:47:29.341 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:29.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:28 smithi184 ceph-mon[92527]: pgmap v1607: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 613 op/s; 2720/155295 objects degraded (1.752%); 32563/155295 objects misplaced (20.968%); 959 KiB/s, 14 objects/s recovering 2024-09-13T14:47:29.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:29.392 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:29.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:29.493 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:29.514 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:29.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:29.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:29.616 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:29.639 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:29.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:29.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:29.741 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:29.780 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:29.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:29.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:29.881 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:29.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:29.904 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:29.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:30.005 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:30.030 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:30.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:30.132 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:30.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.172 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:30.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:30.214 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:29 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 2720/155295 objects degraded (1.752%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:30.272 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:30.320 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:30.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:30.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:29 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 2720/155295 objects degraded (1.752%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:30.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:29 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 2720/155295 objects degraded (1.752%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:30.422 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:30.464 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:30.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:30.566 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:30.591 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:30.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:30.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:47:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:47:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:47:30.693 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:30.716 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:30.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:30.818 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:30.843 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:30.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:30.945 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:30.967 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:30.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:31.069 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:31.091 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:31.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:31.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:31.193 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:31.216 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:31.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:31.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:31.318 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:31.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:30 smithi184 ceph-mon[92527]: pgmap v1608: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 626 op/s; 2720/148287 objects degraded (1.834%); 31390/148287 objects misplaced (21.168%); 959 KiB/s, 14 objects/s recovering 2024-09-13T14:47:31.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:30 smithi151 ceph-mon[104450]: pgmap v1608: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 626 op/s; 2720/148287 objects degraded (1.834%); 31390/148287 objects misplaced (21.168%); 959 KiB/s, 14 objects/s recovering 2024-09-13T14:47:31.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:30 smithi151 ceph-mon[108966]: pgmap v1608: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 626 op/s; 2720/148287 objects degraded (1.834%); 31390/148287 objects misplaced (21.168%); 959 KiB/s, 14 objects/s recovering 2024-09-13T14:47:31.360 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:31.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:31.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:31.462 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:31.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:31.487 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:31.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:31.588 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:31.615 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:31.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:31.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:31.717 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:31.738 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:31.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:31.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:31.839 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:31.863 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:31.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:31.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:31.964 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:31.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:31.989 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:31.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:32.090 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:32.116 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:32.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:32.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:32.218 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:32.243 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:32.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:32.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:32.345 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:32.388 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:32.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:32.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:32.490 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:32.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:32.516 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:32.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:32.617 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:32.640 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:32.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:32.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:32.741 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:32.765 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:32.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:32.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:32.781 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:47:32.781 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:32.782 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:32 smithi151 ceph-mon[104450]: pgmap v1609: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 626 op/s; 2720/148287 objects degraded (1.834%); 31390/148287 objects misplaced (21.168%); 959 KiB/s, 14 objects/s recovering 2024-09-13T14:47:32.782 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:47:32.782 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:32.782 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:32 smithi151 ceph-mon[108966]: pgmap v1609: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 626 op/s; 2720/148287 objects degraded (1.834%); 31390/148287 objects misplaced (21.168%); 959 KiB/s, 14 objects/s recovering 2024-09-13T14:47:32.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:47:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:32 smithi184 ceph-mon[92527]: pgmap v1609: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 626 op/s; 2720/148287 objects degraded (1.834%); 31390/148287 objects misplaced (21.168%); 959 KiB/s, 14 objects/s recovering 2024-09-13T14:47:32.867 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:32.888 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:32.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:32.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:32.990 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:33.024 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:33.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:33.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:33.126 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:33.147 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:33.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:33.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:33.249 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:33.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:33.284 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:33.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:33.385 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:33.414 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:33.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:33.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:33.516 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:33.540 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:33.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:33.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:33.642 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:33.664 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:33.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:33.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:33.766 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:33.790 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:33.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:33.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:33.891 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:33.928 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:33.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:33.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:34.030 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:34.054 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:34.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:34.156 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:34.193 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:34.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:34.295 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:34.340 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:34.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:34.442 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:34.463 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:34.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:34.565 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:34.614 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:34.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:34.716 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:34.738 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:34.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:34.839 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:34.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:34 smithi184 ceph-mon[92527]: pgmap v1610: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 619 op/s; 2817/142215 objects degraded (1.981%); 30453/142215 objects misplaced (21.413%); 991 KiB/s, 15 objects/s recovering 2024-09-13T14:47:34.857 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:34 smithi151 ceph-mon[104450]: pgmap v1610: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 619 op/s; 2817/142215 objects degraded (1.981%); 30453/142215 objects misplaced (21.413%); 991 KiB/s, 15 objects/s recovering 2024-09-13T14:47:34.857 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:34 smithi151 ceph-mon[108966]: pgmap v1610: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 619 op/s; 2817/142215 objects degraded (1.981%); 30453/142215 objects misplaced (21.413%); 991 KiB/s, 15 objects/s recovering 2024-09-13T14:47:34.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:47:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:47:34] "GET /metrics HTTP/1.1" 200 36659 "" "Prometheus/2.51.0" 2024-09-13T14:47:34.860 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:34.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:34.962 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:34.987 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:34.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:35.089 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:35.110 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:35.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:35.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:35.212 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:35.258 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:35.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:35.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:35.360 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:35.385 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:35.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:35.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:35.487 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:35.531 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:35.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:35.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:35.557 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:35 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 2817/142215 objects degraded (1.981%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:35.614 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:35.633 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:35.655 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:35.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:35.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:35.757 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:35.783 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:35.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:35.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:35.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:35 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 2817/142215 objects degraded (1.981%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:35.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:35 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 2817/142215 objects degraded (1.981%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:35.885 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:35.910 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:35.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:35.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:36.011 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:36.034 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:36.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:36.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:36.135 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:36.158 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:36.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:36.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:36.260 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:36.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:36.299 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:36.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:36.400 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:36.424 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:36.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:36.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:36.526 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:36.547 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:36.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:36.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:36.649 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:36.671 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:36.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:36.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:36.772 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:36.793 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:36.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:36.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:36.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:36 smithi184 ceph-mon[92527]: pgmap v1611: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 616 op/s; 2817/139269 objects degraded (2.023%); 29854/139269 objects misplaced (21.436%); 851 KiB/s, 13 objects/s recovering 2024-09-13T14:47:36.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:36 smithi151 ceph-mon[104450]: pgmap v1611: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 616 op/s; 2817/139269 objects degraded (2.023%); 29854/139269 objects misplaced (21.436%); 851 KiB/s, 13 objects/s recovering 2024-09-13T14:47:36.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:36 smithi151 ceph-mon[108966]: pgmap v1611: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 616 op/s; 2817/139269 objects degraded (2.023%); 29854/139269 objects misplaced (21.436%); 851 KiB/s, 13 objects/s recovering 2024-09-13T14:47:36.895 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:36.922 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:36.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:36.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:37.024 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:37.048 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:37.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:37.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:37.096 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:47:37.104 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T14:47:37.150 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:37.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:37.200 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:37.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:37.301 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:37.353 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:37.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:37.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:37.455 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:37.500 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:37.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:37.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:37.602 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:37.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:37.641 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:37.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:37.742 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:37.779 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:37.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:37.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:37.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:37 smithi184 ceph-mon[92527]: from='client.35796 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:47:37.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:37 smithi151 ceph-mon[104450]: from='client.35796 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:47:37.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:37 smithi151 ceph-mon[108966]: from='client.35796 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:47:37.881 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:37.903 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:37.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:37.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:38.005 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:38.029 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:38.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:38.130 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:38.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.174 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:38.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:38.275 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:38.317 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:38.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:38.419 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:38.456 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:38.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:38.486 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:38.499 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1611,"stamp":"2024-09-13T14:47:35.494273+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3041231032,"num_objects":46423,"num_object_clones":0,"num_object_copies":139269,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2817,"num_objects_misplaced":29854,"num_objects_unfound":0,"num_objects_dirty":46423,"num_whiteouts":0,"num_read":14,"num_read_kb":35,"num_write":194591,"num_write_kb":7709418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1778,"num_bytes_recovered":115960057,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115502,"ondisk_log_size":115502,"up":357,"acting":356,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":366,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":11873380,"kb_used_data":9393380,"kb_used_omap":149,"kb_used_meta":2479722,"kb_avail":738087836,"statfs":{"total":767960285184,"available":755801944064,"internally_reserved":0,"allocated":9618821120,"data_stored":9590501079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":153401,"internal_metadata":2539235527},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,10,4,8,5,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-470876160,"num_objects":-7185,"num_object_clones":0,"num_object_copies":-21555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":226,"num_objects_misplaced":-3729,"num_objects_unfound":0,"num_objects_dirty":-7185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7185,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":10158080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.661338"},"pg_stats":[{"pgid":"1.1d","version":"261'100","reported_seq":90,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.847881+0000","last_change":"2024-09-13T14:45:19.807356+0000","last_active":"2024-09-13T14:47:23.847881+0000","last_peered":"2024-09-13T14:47:23.847881+0000","last_clean":"2024-09-13T14:47:23.847881+0000","last_became_active":"2024-09-13T14:45:19.807081+0000","last_became_peered":"2024-09-13T14:45:19.807081+0000","last_unstale":"2024-09-13T14:47:23.847881+0000","last_undegraded":"2024-09-13T14:47:23.847881+0000","last_fullsized":"2024-09-13T14:47:23.847881+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:17.988345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:17.988345+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:41:05.216335+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":1696,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.855134+0000","last_change":"2024-09-13T14:29:42.541263+0000","last_active":"2024-09-13T14:47:23.855134+0000","last_peered":"2024-09-13T14:47:23.855134+0000","last_clean":"2024-09-13T14:47:23.855134+0000","last_became_active":"2024-09-13T14:29:42.540873+0000","last_became_peered":"2024-09-13T14:29:42.540873+0000","last_unstale":"2024-09-13T14:47:23.855134+0000","last_undegraded":"2024-09-13T14:47:23.855134+0000","last_fullsized":"2024-09-13T14:47:23.855134+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:20.611206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1120,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.754330+0000","last_change":"2024-09-13T14:43:07.516715+0000","last_active":"2024-09-13T14:47:05.754330+0000","last_peered":"2024-09-13T14:47:05.754330+0000","last_clean":"2024-09-13T14:47:05.754330+0000","last_became_active":"2024-09-13T14:43:07.516403+0000","last_became_peered":"2024-09-13T14:43:07.516403+0000","last_unstale":"2024-09-13T14:47:05.754330+0000","last_undegraded":"2024-09-13T14:47:05.754330+0000","last_fullsized":"2024-09-13T14:47:05.754330+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:56.469690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":35,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.858362+0000","last_change":"2024-09-13T14:45:19.810203+0000","last_active":"2024-09-13T14:47:23.858362+0000","last_peered":"2024-09-13T14:47:23.858362+0000","last_clean":"2024-09-13T14:47:23.858362+0000","last_became_active":"2024-09-13T14:45:19.809693+0000","last_became_peered":"2024-09-13T14:45:19.809693+0000","last_unstale":"2024-09-13T14:47:23.858362+0000","last_undegraded":"2024-09-13T14:47:23.858362+0000","last_fullsized":"2024-09-13T14:47:23.858362+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:20.219673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":34,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.855099+0000","last_change":"2024-09-13T14:45:19.807991+0000","last_active":"2024-09-13T14:47:23.855099+0000","last_peered":"2024-09-13T14:47:23.855099+0000","last_clean":"2024-09-13T14:47:23.855099+0000","last_became_active":"2024-09-13T14:45:19.807662+0000","last_became_peered":"2024-09-13T14:45:19.807662+0000","last_unstale":"2024-09-13T14:47:23.855099+0000","last_undegraded":"2024-09-13T14:47:23.855099+0000","last_fullsized":"2024-09-13T14:47:23.855099+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:29.306108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"261'100","reported_seq":1231,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.745823+0000","last_change":"2024-09-13T14:44:54.259562+0000","last_active":"2024-09-13T14:47:05.745823+0000","last_peered":"2024-09-13T14:47:05.745823+0000","last_clean":"2024-09-13T14:47:05.745823+0000","last_became_active":"2024-09-13T14:43:14.563717+0000","last_became_peered":"2024-09-13T14:43:14.563717+0000","last_unstale":"2024-09-13T14:47:05.745823+0000","last_undegraded":"2024-09-13T14:47:05.745823+0000","last_fullsized":"2024-09-13T14:47:05.745823+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_clean_scrub_stamp":"2024-09-13T14:44:54.259482+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:43:52.729109+0000","scrub_duration":0.015265569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":1088,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.775700+0000","last_change":"2024-09-13T14:19:25.873514+0000","last_active":"2024-09-13T14:47:05.775700+0000","last_peered":"2024-09-13T14:47:05.775700+0000","last_clean":"2024-09-13T14:47:05.775700+0000","last_became_active":"2024-09-13T14:19:25.873170+0000","last_became_peered":"2024-09-13T14:19:25.873170+0000","last_unstale":"2024-09-13T14:47:05.775700+0000","last_undegraded":"2024-09-13T14:47:05.775700+0000","last_fullsized":"2024-09-13T14:47:05.775700+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:48.298530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1117,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.754519+0000","last_change":"2024-09-13T14:30:15.580011+0000","last_active":"2024-09-13T14:47:05.754519+0000","last_peered":"2024-09-13T14:47:05.754519+0000","last_clean":"2024-09-13T14:47:05.754519+0000","last_became_active":"2024-09-13T14:30:15.579610+0000","last_became_peered":"2024-09-13T14:30:15.579610+0000","last_unstale":"2024-09-13T14:47:05.754519+0000","last_undegraded":"2024-09-13T14:47:05.754519+0000","last_fullsized":"2024-09-13T14:47:05.754519+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:11:02.733771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"261'100","reported_seq":52,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.858503+0000","last_change":"2024-09-13T14:45:35.947332+0000","last_active":"2024-09-13T14:47:23.858503+0000","last_peered":"2024-09-13T14:47:23.858503+0000","last_clean":"2024-09-13T14:47:23.858503+0000","last_became_active":"2024-09-13T14:45:19.810775+0000","last_became_peered":"2024-09-13T14:45:19.810775+0000","last_unstale":"2024-09-13T14:47:23.858503+0000","last_undegraded":"2024-09-13T14:47:23.858503+0000","last_fullsized":"2024-09-13T14:47:23.858503+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:35.947265+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:43:41.569795+0000","last_clean_scrub_stamp":"2024-09-13T14:45:35.947265+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:00.606370+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":1133,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.775598+0000","last_change":"2024-09-13T14:43:07.536116+0000","last_active":"2024-09-13T14:47:05.775598+0000","last_peered":"2024-09-13T14:47:05.775598+0000","last_clean":"2024-09-13T14:47:05.775598+0000","last_became_active":"2024-09-13T14:43:07.524112+0000","last_became_peered":"2024-09-13T14:43:07.524112+0000","last_unstale":"2024-09-13T14:47:05.775598+0000","last_undegraded":"2024-09-13T14:47:05.775598+0000","last_fullsized":"2024-09-13T14:47:05.775598+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:36.947060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1130,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.754705+0000","last_change":"2024-09-13T14:43:07.530236+0000","last_active":"2024-09-13T14:47:05.754705+0000","last_peered":"2024-09-13T14:47:05.754705+0000","last_clean":"2024-09-13T14:47:05.754705+0000","last_became_active":"2024-09-13T14:43:07.529967+0000","last_became_peered":"2024-09-13T14:43:07.529967+0000","last_unstale":"2024-09-13T14:47:05.754705+0000","last_undegraded":"2024-09-13T14:47:05.754705+0000","last_fullsized":"2024-09-13T14:47:05.754705+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:27.276550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":1680,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.852104+0000","last_change":"2024-09-13T14:28:35.612469+0000","last_active":"2024-09-13T14:47:23.852104+0000","last_peered":"2024-09-13T14:47:23.852104+0000","last_clean":"2024-09-13T14:47:23.852104+0000","last_became_active":"2024-09-13T14:28:35.612093+0000","last_became_peered":"2024-09-13T14:28:35.612093+0000","last_unstale":"2024-09-13T14:47:23.852104+0000","last_undegraded":"2024-09-13T14:47:23.852104+0000","last_fullsized":"2024-09-13T14:47:23.852104+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:51.022663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1703,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.858419+0000","last_change":"2024-09-13T14:45:19.808811+0000","last_active":"2024-09-13T14:47:23.858419+0000","last_peered":"2024-09-13T14:47:23.858419+0000","last_clean":"2024-09-13T14:47:23.858419+0000","last_became_active":"2024-09-13T14:45:19.808471+0000","last_became_peered":"2024-09-13T14:45:19.808471+0000","last_unstale":"2024-09-13T14:47:23.858419+0000","last_undegraded":"2024-09-13T14:47:23.858419+0000","last_fullsized":"2024-09-13T14:47:23.858419+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:53.638161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"261'100","reported_seq":34,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.855187+0000","last_change":"2024-09-13T14:45:19.813412+0000","last_active":"2024-09-13T14:47:23.855187+0000","last_peered":"2024-09-13T14:47:23.855187+0000","last_clean":"2024-09-13T14:47:23.855187+0000","last_became_active":"2024-09-13T14:45:19.812585+0000","last_became_peered":"2024-09-13T14:45:19.812585+0000","last_unstale":"2024-09-13T14:47:23.855187+0000","last_undegraded":"2024-09-13T14:47:23.855187+0000","last_fullsized":"2024-09-13T14:47:23.855187+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.614340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.614340+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:32:07.434109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":1139,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.754598+0000","last_change":"2024-09-13T14:43:07.522950+0000","last_active":"2024-09-13T14:47:05.754598+0000","last_peered":"2024-09-13T14:47:05.754598+0000","last_clean":"2024-09-13T14:47:05.754598+0000","last_became_active":"2024-09-13T14:43:07.522566+0000","last_became_peered":"2024-09-13T14:43:07.522566+0000","last_unstale":"2024-09-13T14:47:05.754598+0000","last_undegraded":"2024-09-13T14:47:05.754598+0000","last_fullsized":"2024-09-13T14:47:05.754598+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:03.201478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":34,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.855839+0000","last_change":"2024-09-13T14:45:19.807231+0000","last_active":"2024-09-13T14:47:23.855839+0000","last_peered":"2024-09-13T14:47:23.855839+0000","last_clean":"2024-09-13T14:47:23.855839+0000","last_became_active":"2024-09-13T14:45:19.806936+0000","last_became_peered":"2024-09-13T14:45:19.806936+0000","last_unstale":"2024-09-13T14:47:23.855839+0000","last_undegraded":"2024-09-13T14:47:23.855839+0000","last_fullsized":"2024-09-13T14:47:23.855839+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:04:26.723153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.18","version":"261'100","reported_seq":109,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.775378+0000","last_change":"2024-09-13T14:44:41.685537+0000","last_active":"2024-09-13T14:47:05.775378+0000","last_peered":"2024-09-13T14:47:05.775378+0000","last_clean":"2024-09-13T14:47:05.775378+0000","last_became_active":"2024-09-13T14:43:14.558502+0000","last_became_peered":"2024-09-13T14:43:14.558502+0000","last_unstale":"2024-09-13T14:47:05.775378+0000","last_undegraded":"2024-09-13T14:47:05.775378+0000","last_fullsized":"2024-09-13T14:47:05.775378+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1134,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:41.685124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:41.685124+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:37:58.412497+0000","scrub_duration":0.0075205970000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":83,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.846911+0000","last_change":"2024-09-13T14:43:07.516845+0000","last_active":"2024-09-13T14:47:23.846911+0000","last_peered":"2024-09-13T14:47:23.846911+0000","last_clean":"2024-09-13T14:47:23.846911+0000","last_became_active":"2024-09-13T14:43:07.514974+0000","last_became_peered":"2024-09-13T14:43:07.514974+0000","last_unstale":"2024-09-13T14:47:23.846911+0000","last_undegraded":"2024-09-13T14:47:23.846911+0000","last_fullsized":"2024-09-13T14:47:23.846911+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:55.419483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1718,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.861360+0000","last_change":"2024-09-13T14:45:19.815341+0000","last_active":"2024-09-13T14:47:23.861360+0000","last_peered":"2024-09-13T14:47:23.861360+0000","last_clean":"2024-09-13T14:47:23.861360+0000","last_became_active":"2024-09-13T14:45:19.810262+0000","last_became_peered":"2024-09-13T14:45:19.810262+0000","last_unstale":"2024-09-13T14:47:23.861360+0000","last_undegraded":"2024-09-13T14:47:23.861360+0000","last_fullsized":"2024-09-13T14:47:23.861360+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:39.847653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1b","version":"261'100","reported_seq":51,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.777361+0000","last_change":"2024-09-13T14:44:52.803330+0000","last_active":"2024-09-13T14:47:05.777361+0000","last_peered":"2024-09-13T14:47:05.777361+0000","last_clean":"2024-09-13T14:47:05.777361+0000","last_became_active":"2024-09-13T14:43:14.556712+0000","last_became_peered":"2024-09-13T14:43:14.556712+0000","last_unstale":"2024-09-13T14:47:05.777361+0000","last_undegraded":"2024-09-13T14:47:05.777361+0000","last_fullsized":"2024-09-13T14:47:05.777361+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:52.803242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:52.803242+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:06.658301+0000","scrub_duration":0.0019769660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":34,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.854606+0000","last_change":"2024-09-13T14:45:19.806623+0000","last_active":"2024-09-13T14:47:23.854606+0000","last_peered":"2024-09-13T14:47:23.854606+0000","last_clean":"2024-09-13T14:47:23.854606+0000","last_became_active":"2024-09-13T14:45:19.806219+0000","last_became_peered":"2024-09-13T14:45:19.806219+0000","last_unstale":"2024-09-13T14:47:23.854606+0000","last_undegraded":"2024-09-13T14:47:23.854606+0000","last_fullsized":"2024-09-13T14:47:23.854606+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:34:41.601201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1113,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.775835+0000","last_change":"2024-09-13T14:43:32.637751+0000","last_active":"2024-09-13T14:47:05.775835+0000","last_peered":"2024-09-13T14:47:05.775835+0000","last_clean":"2024-09-13T14:47:05.775835+0000","last_became_active":"2024-09-13T14:43:32.636901+0000","last_became_peered":"2024-09-13T14:43:32.636901+0000","last_unstale":"2024-09-13T14:47:05.775835+0000","last_undegraded":"2024-09-13T14:47:05.775835+0000","last_fullsized":"2024-09-13T14:47:05.775835+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:21.942419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1a","version":"261'100","reported_seq":51,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.754432+0000","last_change":"2024-09-13T14:43:41.910041+0000","last_active":"2024-09-13T14:47:05.754432+0000","last_peered":"2024-09-13T14:47:05.754432+0000","last_clean":"2024-09-13T14:47:05.754432+0000","last_became_active":"2024-09-13T14:43:14.555083+0000","last_became_peered":"2024-09-13T14:43:14.555083+0000","last_unstale":"2024-09-13T14:47:05.754432+0000","last_undegraded":"2024-09-13T14:47:05.754432+0000","last_fullsized":"2024-09-13T14:47:05.754432+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:41.909988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:41.909988+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:42:19.954838+0000","scrub_duration":0.0012891619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":34,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.854766+0000","last_change":"2024-09-13T14:45:19.812071+0000","last_active":"2024-09-13T14:47:23.854766+0000","last_peered":"2024-09-13T14:47:23.854766+0000","last_clean":"2024-09-13T14:47:23.854766+0000","last_became_active":"2024-09-13T14:45:19.811336+0000","last_became_peered":"2024-09-13T14:45:19.811336+0000","last_unstale":"2024-09-13T14:47:23.854766+0000","last_undegraded":"2024-09-13T14:47:23.854766+0000","last_fullsized":"2024-09-13T14:47:23.854766+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:40:29.814608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1735,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.854754+0000","last_change":"2024-09-13T14:45:19.810478+0000","last_active":"2024-09-13T14:47:23.854754+0000","last_peered":"2024-09-13T14:47:23.854754+0000","last_clean":"2024-09-13T14:47:23.854754+0000","last_became_active":"2024-09-13T14:45:19.808278+0000","last_became_peered":"2024-09-13T14:45:19.808278+0000","last_unstale":"2024-09-13T14:47:23.854754+0000","last_undegraded":"2024-09-13T14:47:23.854754+0000","last_fullsized":"2024-09-13T14:47:23.854754+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:12.211089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"261'100","reported_seq":51,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.746075+0000","last_change":"2024-09-13T14:44:38.214961+0000","last_active":"2024-09-13T14:47:05.746075+0000","last_peered":"2024-09-13T14:47:05.746075+0000","last_clean":"2024-09-13T14:47:05.746075+0000","last_became_active":"2024-09-13T14:43:14.557883+0000","last_became_peered":"2024-09-13T14:43:14.557883+0000","last_unstale":"2024-09-13T14:47:05.746075+0000","last_undegraded":"2024-09-13T14:47:05.746075+0000","last_fullsized":"2024-09-13T14:47:05.746075+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:38.214865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:38.214865+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:49:36.013783+0000","scrub_duration":0.0016389639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1606,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.855341+0000","last_change":"2024-09-13T14:30:15.576921+0000","last_active":"2024-09-13T14:47:23.855341+0000","last_peered":"2024-09-13T14:47:23.855341+0000","last_clean":"2024-09-13T14:47:23.855341+0000","last_became_active":"2024-09-13T14:30:15.576607+0000","last_became_peered":"2024-09-13T14:30:15.576607+0000","last_unstale":"2024-09-13T14:47:23.855341+0000","last_undegraded":"2024-09-13T14:47:23.855341+0000","last_fullsized":"2024-09-13T14:47:23.855341+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:35.162460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1123,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.775672+0000","last_change":"2024-09-13T14:43:07.518801+0000","last_active":"2024-09-13T14:47:05.775672+0000","last_peered":"2024-09-13T14:47:05.775672+0000","last_clean":"2024-09-13T14:47:05.775672+0000","last_became_active":"2024-09-13T14:43:07.517610+0000","last_became_peered":"2024-09-13T14:43:07.517610+0000","last_unstale":"2024-09-13T14:47:05.775672+0000","last_undegraded":"2024-09-13T14:47:05.775672+0000","last_fullsized":"2024-09-13T14:47:05.775672+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:51.632254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"261'100","reported_seq":83,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.858049+0000","last_change":"2024-09-13T14:43:15.037565+0000","last_active":"2024-09-13T14:47:23.858049+0000","last_peered":"2024-09-13T14:47:23.858049+0000","last_clean":"2024-09-13T14:47:23.858049+0000","last_became_active":"2024-09-13T14:43:14.554183+0000","last_became_peered":"2024-09-13T14:43:14.554183+0000","last_unstale":"2024-09-13T14:47:23.858049+0000","last_undegraded":"2024-09-13T14:47:23.858049+0000","last_fullsized":"2024-09-13T14:47:23.858049+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.037517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.037517+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:01:35.137195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":1117,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.745906+0000","last_change":"2024-09-13T14:30:15.579260+0000","last_active":"2024-09-13T14:47:05.745906+0000","last_peered":"2024-09-13T14:47:05.745906+0000","last_clean":"2024-09-13T14:47:05.745906+0000","last_became_active":"2024-09-13T14:30:15.578897+0000","last_became_peered":"2024-09-13T14:30:15.578897+0000","last_unstale":"2024-09-13T14:47:05.745906+0000","last_undegraded":"2024-09-13T14:47:05.745906+0000","last_fullsized":"2024-09-13T14:47:05.745906+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:56.737849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":83,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.859302+0000","last_change":"2024-09-13T14:43:07.521576+0000","last_active":"2024-09-13T14:47:23.859302+0000","last_peered":"2024-09-13T14:47:23.859302+0000","last_clean":"2024-09-13T14:47:23.859302+0000","last_became_active":"2024-09-13T14:43:07.520478+0000","last_became_peered":"2024-09-13T14:43:07.520478+0000","last_unstale":"2024-09-13T14:47:23.859302+0000","last_undegraded":"2024-09-13T14:47:23.859302+0000","last_fullsized":"2024-09-13T14:47:23.859302+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:53.410135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"261'100","reported_seq":377,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.775673+0000","last_change":"2024-09-13T14:44:49.775941+0000","last_active":"2024-09-13T14:47:05.775673+0000","last_peered":"2024-09-13T14:47:05.775673+0000","last_clean":"2024-09-13T14:47:05.775673+0000","last_became_active":"2024-09-13T14:43:14.560297+0000","last_became_peered":"2024-09-13T14:43:14.560297+0000","last_unstale":"2024-09-13T14:47:05.775673+0000","last_undegraded":"2024-09-13T14:47:05.775673+0000","last_fullsized":"2024-09-13T14:47:05.775673+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:49.775844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:49.775844+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:15:53.138661+0000","scrub_duration":0.012350274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":1106,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.754906+0000","last_change":"2024-09-13T14:19:25.870605+0000","last_active":"2024-09-13T14:47:05.754906+0000","last_peered":"2024-09-13T14:47:05.754906+0000","last_clean":"2024-09-13T14:47:05.754906+0000","last_became_active":"2024-09-13T14:19:25.870282+0000","last_became_peered":"2024-09-13T14:19:25.870282+0000","last_unstale":"2024-09-13T14:47:05.754906+0000","last_undegraded":"2024-09-13T14:47:05.754906+0000","last_fullsized":"2024-09-13T14:47:05.754906+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:52.514915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1120,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.754740+0000","last_change":"2024-09-13T14:43:07.526476+0000","last_active":"2024-09-13T14:47:05.754740+0000","last_peered":"2024-09-13T14:47:05.754740+0000","last_clean":"2024-09-13T14:47:05.754740+0000","last_became_active":"2024-09-13T14:43:07.525802+0000","last_became_peered":"2024-09-13T14:43:07.525802+0000","last_unstale":"2024-09-13T14:47:05.754740+0000","last_undegraded":"2024-09-13T14:47:05.754740+0000","last_fullsized":"2024-09-13T14:47:05.754740+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:23.154726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"261'100","reported_seq":90,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.846404+0000","last_change":"2024-09-13T14:45:19.807461+0000","last_active":"2024-09-13T14:47:23.846404+0000","last_peered":"2024-09-13T14:47:23.846404+0000","last_clean":"2024-09-13T14:47:23.846404+0000","last_became_active":"2024-09-13T14:45:19.807117+0000","last_became_peered":"2024-09-13T14:45:19.807117+0000","last_unstale":"2024-09-13T14:47:23.846404+0000","last_undegraded":"2024-09-13T14:47:23.846404+0000","last_fullsized":"2024-09-13T14:47:23.846404+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.955197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.955197+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:03.531555+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"1160'12235","reported_seq":12301,"reported_epoch":1160,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:47:32.724019+0000","last_change":"2024-09-13T14:45:21.161658+0000","last_active":"2024-09-13T14:47:32.724019+0000","last_peered":"2024-09-13T14:47:32.724019+0000","last_clean":"2024-09-13T14:45:19.622135+0000","last_became_active":"2024-09-13T14:45:21.090442+0000","last_became_peered":"2024-09-13T14:45:21.090442+0000","last_unstale":"2024-09-13T14:47:32.724019+0000","last_undegraded":"2024-09-13T14:47:32.724019+0000","last_fullsized":"2024-09-13T14:47:32.724019+0000","mapping_epoch":1149,"log_start":"1146'6200","ondisk_log_start":"1146'6200","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6035,"log_dups_size":0,"ondisk_log_size":6035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194707456,"num_objects":2971,"num_object_clones":0,"num_object_copies":8913,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2971,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12235,"num_write_kb":486592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[1,2,7],"avail_no_missing":["1","2","7"],"object_location_counts":[{"shards":"1,2,7","objects":2971}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1083,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.755575+0000","last_change":"2024-09-13T14:19:26.131609+0000","last_active":"2024-09-13T14:47:05.755575+0000","last_peered":"2024-09-13T14:47:05.755575+0000","last_clean":"2024-09-13T14:47:05.755575+0000","last_became_active":"2024-09-13T14:19:26.131029+0000","last_became_peered":"2024-09-13T14:19:26.131029+0000","last_unstale":"2024-09-13T14:47:05.755575+0000","last_undegraded":"2024-09-13T14:47:05.755575+0000","last_fullsized":"2024-09-13T14:47:05.755575+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:57.933696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":585,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.746188+0000","last_change":"2024-09-13T14:43:32.635070+0000","last_active":"2024-09-13T14:47:05.746188+0000","last_peered":"2024-09-13T14:47:05.746188+0000","last_clean":"2024-09-13T14:47:05.746188+0000","last_became_active":"2024-09-13T14:43:32.634734+0000","last_became_peered":"2024-09-13T14:43:32.634734+0000","last_unstale":"2024-09-13T14:47:05.746188+0000","last_undegraded":"2024-09-13T14:47:05.746188+0000","last_fullsized":"2024-09-13T14:47:05.746188+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:33.107010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":1725,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.861504+0000","last_change":"2024-09-13T14:43:07.537759+0000","last_active":"2024-09-13T14:47:23.861504+0000","last_peered":"2024-09-13T14:47:23.861504+0000","last_clean":"2024-09-13T14:47:23.861504+0000","last_became_active":"2024-09-13T14:43:07.520303+0000","last_became_peered":"2024-09-13T14:43:07.520303+0000","last_unstale":"2024-09-13T14:47:23.861504+0000","last_undegraded":"2024-09-13T14:47:23.861504+0000","last_fullsized":"2024-09-13T14:47:23.861504+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:32.772570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"261'100","reported_seq":35,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.858469+0000","last_change":"2024-09-13T14:45:19.813527+0000","last_active":"2024-09-13T14:47:23.858469+0000","last_peered":"2024-09-13T14:47:23.858469+0000","last_clean":"2024-09-13T14:47:23.858469+0000","last_became_active":"2024-09-13T14:45:19.812834+0000","last_became_peered":"2024-09-13T14:45:19.812834+0000","last_unstale":"2024-09-13T14:47:23.858469+0000","last_undegraded":"2024-09-13T14:47:23.858469+0000","last_fullsized":"2024-09-13T14:47:23.858469+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.710192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.710192+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:08:55.152703+0000","scrub_duration":0.0019250910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"1160'12088","reported_seq":15619,"reported_epoch":1160,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:47:31.654994+0000","last_change":"2024-09-13T14:45:38.479094+0000","last_active":"2024-09-13T14:47:31.654994+0000","last_peered":"2024-09-13T14:47:31.654994+0000","last_clean":"2024-09-13T14:45:19.642017+0000","last_became_active":"2024-09-13T14:45:21.111093+0000","last_became_peered":"2024-09-13T14:45:21.111093+0000","last_unstale":"2024-09-13T14:47:31.654994+0000","last_undegraded":"2024-09-13T14:45:38.458706+0000","last_fullsized":"2024-09-13T14:45:20.800066+0000","mapping_epoch":1149,"log_start":"1142'3700","ondisk_log_start":"1142'3700","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":8388,"log_dups_size":0,"ondisk_log_size":8388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195821568,"num_objects":2988,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2817,"num_objects_misplaced":2544,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12088,"num_write_kb":482432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1734,"num_bytes_recovered":113639424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[2,4],"avail_no_missing":["2","4","6"],"object_location_counts":[{"shards":"2,4","objects":-2373},{"shards":"2,4,6","objects":2544}],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":35,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.854886+0000","last_change":"2024-09-13T14:45:19.813025+0000","last_active":"2024-09-13T14:47:23.854886+0000","last_peered":"2024-09-13T14:47:23.854886+0000","last_clean":"2024-09-13T14:47:23.854886+0000","last_became_active":"2024-09-13T14:45:19.812714+0000","last_became_peered":"2024-09-13T14:45:19.812714+0000","last_unstale":"2024-09-13T14:47:23.854886+0000","last_undegraded":"2024-09-13T14:47:23.854886+0000","last_fullsized":"2024-09-13T14:47:23.854886+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:40.671012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":84,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.846707+0000","last_change":"2024-09-13T14:43:07.522730+0000","last_active":"2024-09-13T14:47:23.846707+0000","last_peered":"2024-09-13T14:47:23.846707+0000","last_clean":"2024-09-13T14:47:23.846707+0000","last_became_active":"2024-09-13T14:43:07.517420+0000","last_became_peered":"2024-09-13T14:43:07.517420+0000","last_unstale":"2024-09-13T14:47:23.846707+0000","last_undegraded":"2024-09-13T14:47:23.846707+0000","last_fullsized":"2024-09-13T14:47:23.846707+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:21:30.265293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":1639,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.858918+0000","last_change":"2024-09-13T14:30:15.581728+0000","last_active":"2024-09-13T14:47:23.858918+0000","last_peered":"2024-09-13T14:47:23.858918+0000","last_clean":"2024-09-13T14:47:23.858918+0000","last_became_active":"2024-09-13T14:30:15.581303+0000","last_became_peered":"2024-09-13T14:30:15.581303+0000","last_unstale":"2024-09-13T14:47:23.858918+0000","last_undegraded":"2024-09-13T14:47:23.858918+0000","last_fullsized":"2024-09-13T14:47:23.858918+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:10.027991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"261'100","reported_seq":110,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.755072+0000","last_change":"2024-09-13T14:43:31.112173+0000","last_active":"2024-09-13T14:47:05.755072+0000","last_peered":"2024-09-13T14:47:05.755072+0000","last_clean":"2024-09-13T14:47:05.755072+0000","last_became_active":"2024-09-13T14:42:42.709263+0000","last_became_peered":"2024-09-13T14:42:42.709263+0000","last_unstale":"2024-09-13T14:47:05.755072+0000","last_undegraded":"2024-09-13T14:47:05.755072+0000","last_fullsized":"2024-09-13T14:47:05.755072+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:31.112111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:31.112111+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:02.707372+0000","scrub_duration":0.0015759260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"1160'12287","reported_seq":12330,"reported_epoch":1160,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:47:32.632447+0000","last_change":"2024-09-13T14:45:21.011501+0000","last_active":"2024-09-13T14:47:32.632447+0000","last_peered":"2024-09-13T14:47:32.632447+0000","last_clean":"2024-09-13T14:45:19.622974+0000","last_became_active":"2024-09-13T14:45:21.006075+0000","last_became_peered":"2024-09-13T14:45:21.006075+0000","last_unstale":"2024-09-13T14:47:32.632447+0000","last_undegraded":"2024-09-13T14:47:32.632447+0000","last_fullsized":"2024-09-13T14:47:32.632447+0000","mapping_epoch":1149,"log_start":"1142'4500","ondisk_log_start":"1142'4500","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":7787,"log_dups_size":0,"ondisk_log_size":7787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:30.090601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194707456,"num_objects":2971,"num_object_clones":0,"num_object_copies":8913,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2971,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12287,"num_write_kb":488256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,2],"avail_no_missing":["5","1","2"],"object_location_counts":[{"shards":"1,2,5","objects":2971}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1679,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.852016+0000","last_change":"2024-09-13T14:28:35.611508+0000","last_active":"2024-09-13T14:47:23.852016+0000","last_peered":"2024-09-13T14:47:23.852016+0000","last_clean":"2024-09-13T14:47:23.852016+0000","last_became_active":"2024-09-13T14:28:35.610106+0000","last_became_peered":"2024-09-13T14:28:35.610106+0000","last_unstale":"2024-09-13T14:47:23.852016+0000","last_undegraded":"2024-09-13T14:47:23.852016+0000","last_fullsized":"2024-09-13T14:47:23.852016+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:11.966958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1126,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.746118+0000","last_change":"2024-09-13T14:28:35.603663+0000","last_active":"2024-09-13T14:47:05.746118+0000","last_peered":"2024-09-13T14:47:05.746118+0000","last_clean":"2024-09-13T14:47:05.746118+0000","last_became_active":"2024-09-13T14:28:35.603358+0000","last_became_peered":"2024-09-13T14:28:35.603358+0000","last_unstale":"2024-09-13T14:47:05.746118+0000","last_undegraded":"2024-09-13T14:47:05.746118+0000","last_fullsized":"2024-09-13T14:47:05.746118+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:24:05.842796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":84,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.848164+0000","last_change":"2024-09-13T14:43:07.522750+0000","last_active":"2024-09-13T14:47:23.848164+0000","last_peered":"2024-09-13T14:47:23.848164+0000","last_clean":"2024-09-13T14:47:23.848164+0000","last_became_active":"2024-09-13T14:43:07.517267+0000","last_became_peered":"2024-09-13T14:43:07.517267+0000","last_unstale":"2024-09-13T14:47:23.848164+0000","last_undegraded":"2024-09-13T14:47:23.848164+0000","last_fullsized":"2024-09-13T14:47:23.848164+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:40.217748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"261'100","reported_seq":404,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.861212+0000","last_change":"2024-09-13T14:45:19.815050+0000","last_active":"2024-09-13T14:47:23.861212+0000","last_peered":"2024-09-13T14:47:23.861212+0000","last_clean":"2024-09-13T14:47:23.861212+0000","last_became_active":"2024-09-13T14:45:19.809732+0000","last_became_peered":"2024-09-13T14:45:19.809732+0000","last_unstale":"2024-09-13T14:47:23.861212+0000","last_undegraded":"2024-09-13T14:47:23.861212+0000","last_fullsized":"2024-09-13T14:47:23.861212+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:33.574443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:33.574443+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:51:40.261036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"1160'12221","reported_seq":12323,"reported_epoch":1160,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:47:32.453421+0000","last_change":"2024-09-13T14:45:21.136109+0000","last_active":"2024-09-13T14:47:32.453421+0000","last_peered":"2024-09-13T14:47:32.453421+0000","last_clean":"2024-09-13T14:45:19.607954+0000","last_became_active":"2024-09-13T14:45:21.004300+0000","last_became_peered":"2024-09-13T14:45:21.004300+0000","last_unstale":"2024-09-13T14:47:32.453421+0000","last_undegraded":"2024-09-13T14:47:32.453421+0000","last_fullsized":"2024-09-13T14:47:32.453421+0000","mapping_epoch":1149,"log_start":"1146'6000","ondisk_log_start":"1146'6000","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6221,"log_dups_size":0,"ondisk_log_size":6221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:04:23.351582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189202432,"num_objects":2887,"num_object_clones":0,"num_object_copies":8661,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2871,"num_objects_unfound":0,"num_objects_dirty":2887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12221,"num_write_kb":483456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":1835008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,6,1],"avail_no_missing":["4","1","6"],"object_location_counts":[{"shards":"1,4,6","objects":2887}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":84,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.859143+0000","last_change":"2024-09-13T14:43:07.520049+0000","last_active":"2024-09-13T14:47:23.859143+0000","last_peered":"2024-09-13T14:47:23.859143+0000","last_clean":"2024-09-13T14:47:23.859143+0000","last_became_active":"2024-09-13T14:43:07.514646+0000","last_became_peered":"2024-09-13T14:43:07.514646+0000","last_unstale":"2024-09-13T14:47:23.859143+0000","last_undegraded":"2024-09-13T14:47:23.859143+0000","last_fullsized":"2024-09-13T14:47:23.859143+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:10:35.200778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":84,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.847048+0000","last_change":"2024-09-13T14:43:07.525666+0000","last_active":"2024-09-13T14:47:23.847048+0000","last_peered":"2024-09-13T14:47:23.847048+0000","last_clean":"2024-09-13T14:47:23.847048+0000","last_became_active":"2024-09-13T14:43:07.524064+0000","last_became_peered":"2024-09-13T14:43:07.524064+0000","last_unstale":"2024-09-13T14:47:23.847048+0000","last_undegraded":"2024-09-13T14:47:23.847048+0000","last_fullsized":"2024-09-13T14:47:23.847048+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:10.270333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1140,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.775750+0000","last_change":"2024-09-13T14:45:19.804177+0000","last_active":"2024-09-13T14:47:05.775750+0000","last_peered":"2024-09-13T14:47:05.775750+0000","last_clean":"2024-09-13T14:47:05.775750+0000","last_became_active":"2024-09-13T14:45:19.803890+0000","last_became_peered":"2024-09-13T14:45:19.803890+0000","last_unstale":"2024-09-13T14:47:05.775750+0000","last_undegraded":"2024-09-13T14:47:05.775750+0000","last_fullsized":"2024-09-13T14:47:05.775750+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:25.140789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"261'100","reported_seq":52,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.858751+0000","last_change":"2024-09-13T14:45:36.965992+0000","last_active":"2024-09-13T14:47:23.858751+0000","last_peered":"2024-09-13T14:47:23.858751+0000","last_clean":"2024-09-13T14:47:23.858751+0000","last_became_active":"2024-09-13T14:45:19.811238+0000","last_became_peered":"2024-09-13T14:45:19.811238+0000","last_unstale":"2024-09-13T14:47:23.858751+0000","last_undegraded":"2024-09-13T14:47:23.858751+0000","last_fullsized":"2024-09-13T14:47:23.858751+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:36.965775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:45:36.965775+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:36:01.623251+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"1160'12063","reported_seq":12126,"reported_epoch":1160,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:47:34.438915+0000","last_change":"2024-09-13T14:45:21.036181+0000","last_active":"2024-09-13T14:47:34.438915+0000","last_peered":"2024-09-13T14:47:34.438915+0000","last_clean":"2024-09-13T14:45:19.587756+0000","last_became_active":"2024-09-13T14:45:21.029769+0000","last_became_peered":"2024-09-13T14:45:21.029769+0000","last_unstale":"2024-09-13T14:47:34.438915+0000","last_undegraded":"2024-09-13T14:47:34.438915+0000","last_fullsized":"2024-09-13T14:47:34.438915+0000","mapping_epoch":1149,"log_start":"1143'5200","ondisk_log_start":"1143'5200","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6863,"log_dups_size":0,"ondisk_log_size":6863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179240960,"num_objects":2735,"num_object_clones":0,"num_object_copies":8205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2735,"num_objects_unfound":0,"num_objects_dirty":2735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12063,"num_write_kb":473536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,1],"avail_no_missing":["0","1","6"],"object_location_counts":[{"shards":"0,1,6","objects":2735}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":39,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.775481+0000","last_change":"2024-09-13T14:45:19.806318+0000","last_active":"2024-09-13T14:47:05.775481+0000","last_peered":"2024-09-13T14:47:05.775481+0000","last_clean":"2024-09-13T14:47:05.775481+0000","last_became_active":"2024-09-13T14:45:19.806074+0000","last_became_peered":"2024-09-13T14:45:19.806074+0000","last_unstale":"2024-09-13T14:47:05.775481+0000","last_undegraded":"2024-09-13T14:47:05.775481+0000","last_fullsized":"2024-09-13T14:47:05.775481+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:21.679797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1101,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.755085+0000","last_change":"2024-09-13T14:28:35.604404+0000","last_active":"2024-09-13T14:47:05.755085+0000","last_peered":"2024-09-13T14:47:05.755085+0000","last_clean":"2024-09-13T14:47:05.755085+0000","last_became_active":"2024-09-13T14:28:35.604102+0000","last_became_peered":"2024-09-13T14:28:35.604102+0000","last_unstale":"2024-09-13T14:47:05.755085+0000","last_undegraded":"2024-09-13T14:47:05.755085+0000","last_fullsized":"2024-09-13T14:47:05.755085+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:25.546010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":1151,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.775654+0000","last_change":"2024-09-13T14:43:07.538919+0000","last_active":"2024-09-13T14:47:05.775654+0000","last_peered":"2024-09-13T14:47:05.775654+0000","last_clean":"2024-09-13T14:47:05.775654+0000","last_became_active":"2024-09-13T14:43:07.513109+0000","last_became_peered":"2024-09-13T14:43:07.513109+0000","last_unstale":"2024-09-13T14:47:05.775654+0000","last_undegraded":"2024-09-13T14:47:05.775654+0000","last_fullsized":"2024-09-13T14:47:05.775654+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:27.771720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"261'100","reported_seq":100,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.775245+0000","last_change":"2024-09-13T14:43:44.720333+0000","last_active":"2024-09-13T14:47:05.775245+0000","last_peered":"2024-09-13T14:47:05.775245+0000","last_clean":"2024-09-13T14:47:05.775245+0000","last_became_active":"2024-09-13T14:42:14.946591+0000","last_became_peered":"2024-09-13T14:42:14.946591+0000","last_unstale":"2024-09-13T14:47:05.775245+0000","last_undegraded":"2024-09-13T14:47:05.775245+0000","last_fullsized":"2024-09-13T14:47:05.775245+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:44.720258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:44.720258+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:33.281357+0000","scrub_duration":0.0018417920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"1160'12233","reported_seq":12293,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:34.465422+0000","last_change":"2024-09-13T14:43:50.794091+0000","last_active":"2024-09-13T14:47:34.465422+0000","last_peered":"2024-09-13T14:47:34.465422+0000","last_clean":"2024-09-13T14:47:34.465422+0000","last_became_active":"2024-09-13T14:43:50.793741+0000","last_became_peered":"2024-09-13T14:43:50.793741+0000","last_unstale":"2024-09-13T14:47:34.465422+0000","last_undegraded":"2024-09-13T14:47:34.465422+0000","last_fullsized":"2024-09-13T14:47:34.465422+0000","mapping_epoch":1136,"log_start":"1144'5400","ondisk_log_start":"1144'5400","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6833,"log_dups_size":0,"ondisk_log_size":6833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:15.846494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187236352,"num_objects":2857,"num_object_clones":0,"num_object_copies":8571,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12233,"num_write_kb":482880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":51,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.856765+0000","last_change":"2024-09-13T14:44:03.858211+0000","last_active":"2024-09-13T14:47:23.856765+0000","last_peered":"2024-09-13T14:47:23.856765+0000","last_clean":"2024-09-13T14:47:23.856765+0000","last_became_active":"2024-09-13T14:44:03.857928+0000","last_became_peered":"2024-09-13T14:44:03.857928+0000","last_unstale":"2024-09-13T14:47:23.856765+0000","last_undegraded":"2024-09-13T14:47:23.856765+0000","last_fullsized":"2024-09-13T14:47:23.856765+0000","mapping_epoch":1140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:28.428693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":84,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.846316+0000","last_change":"2024-09-13T14:43:07.516243+0000","last_active":"2024-09-13T14:47:23.846316+0000","last_peered":"2024-09-13T14:47:23.846316+0000","last_clean":"2024-09-13T14:47:23.846316+0000","last_became_active":"2024-09-13T14:43:07.515863+0000","last_became_peered":"2024-09-13T14:43:07.515863+0000","last_unstale":"2024-09-13T14:47:23.846316+0000","last_undegraded":"2024-09-13T14:47:23.846316+0000","last_fullsized":"2024-09-13T14:47:23.846316+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:08.085298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1720,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.856665+0000","last_change":"2024-09-13T14:45:19.812713+0000","last_active":"2024-09-13T14:47:23.856665+0000","last_peered":"2024-09-13T14:47:23.856665+0000","last_clean":"2024-09-13T14:47:23.856665+0000","last_became_active":"2024-09-13T14:45:19.812203+0000","last_became_peered":"2024-09-13T14:45:19.812203+0000","last_unstale":"2024-09-13T14:47:23.856665+0000","last_undegraded":"2024-09-13T14:47:23.856665+0000","last_fullsized":"2024-09-13T14:47:23.856665+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:30:46.149347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"261'100","reported_seq":1280,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.745584+0000","last_change":"2024-09-13T14:43:42.577312+0000","last_active":"2024-09-13T14:47:05.745584+0000","last_peered":"2024-09-13T14:47:05.745584+0000","last_clean":"2024-09-13T14:47:05.745584+0000","last_became_active":"2024-09-13T14:42:14.928923+0000","last_became_peered":"2024-09-13T14:42:14.928923+0000","last_unstale":"2024-09-13T14:47:05.745584+0000","last_undegraded":"2024-09-13T14:47:05.745584+0000","last_fullsized":"2024-09-13T14:47:05.745584+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:42.577243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:42.577243+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:01.234250+0000","scrub_duration":0.001367839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"1160'12077","reported_seq":12109,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:32.528186+0000","last_change":"2024-09-13T14:43:50.793810+0000","last_active":"2024-09-13T14:47:32.528186+0000","last_peered":"2024-09-13T14:47:32.528186+0000","last_clean":"2024-09-13T14:47:32.528186+0000","last_became_active":"2024-09-13T14:43:50.793506+0000","last_became_peered":"2024-09-13T14:43:50.793506+0000","last_unstale":"2024-09-13T14:47:32.528186+0000","last_undegraded":"2024-09-13T14:47:32.528186+0000","last_fullsized":"2024-09-13T14:47:32.528186+0000","mapping_epoch":1136,"log_start":"1145'5900","ondisk_log_start":"1145'5900","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6177,"log_dups_size":0,"ondisk_log_size":6177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:50:32.061762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193396736,"num_objects":2951,"num_object_clones":0,"num_object_copies":8853,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":480896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1714,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.861291+0000","last_change":"2024-09-13T14:43:07.520212+0000","last_active":"2024-09-13T14:47:23.861291+0000","last_peered":"2024-09-13T14:47:23.861291+0000","last_clean":"2024-09-13T14:47:23.861291+0000","last_became_active":"2024-09-13T14:43:07.519831+0000","last_became_peered":"2024-09-13T14:43:07.519831+0000","last_unstale":"2024-09-13T14:47:23.861291+0000","last_undegraded":"2024-09-13T14:47:23.861291+0000","last_fullsized":"2024-09-13T14:47:23.861291+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:20.990736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1737,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.850530+0000","last_change":"2024-09-13T14:45:19.809505+0000","last_active":"2024-09-13T14:47:23.850530+0000","last_peered":"2024-09-13T14:47:23.850530+0000","last_clean":"2024-09-13T14:47:23.850530+0000","last_became_active":"2024-09-13T14:45:19.809181+0000","last_became_peered":"2024-09-13T14:45:19.809181+0000","last_unstale":"2024-09-13T14:47:23.850530+0000","last_undegraded":"2024-09-13T14:47:23.850530+0000","last_fullsized":"2024-09-13T14:47:23.850530+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:24.239188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1163,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.745565+0000","last_change":"2024-09-13T14:43:07.516140+0000","last_active":"2024-09-13T14:47:05.745565+0000","last_peered":"2024-09-13T14:47:05.745565+0000","last_clean":"2024-09-13T14:47:05.745565+0000","last_became_active":"2024-09-13T14:43:07.515481+0000","last_became_peered":"2024-09-13T14:43:07.515481+0000","last_unstale":"2024-09-13T14:47:05.745565+0000","last_undegraded":"2024-09-13T14:47:05.745565+0000","last_fullsized":"2024-09-13T14:47:05.745565+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:43.706214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"261'100","reported_seq":96,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.851557+0000","last_change":"2024-09-13T14:43:16.008346+0000","last_active":"2024-09-13T14:47:23.851557+0000","last_peered":"2024-09-13T14:47:23.851557+0000","last_clean":"2024-09-13T14:47:23.851557+0000","last_became_active":"2024-09-13T14:43:07.523121+0000","last_became_peered":"2024-09-13T14:43:07.523121+0000","last_unstale":"2024-09-13T14:47:23.851557+0000","last_undegraded":"2024-09-13T14:47:23.851557+0000","last_fullsized":"2024-09-13T14:47:23.851557+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.008264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.008264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:58:35.564470+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"1160'12199","reported_seq":12263,"reported_epoch":1160,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:47:34.312133+0000","last_change":"2024-09-13T14:45:21.035848+0000","last_active":"2024-09-13T14:47:34.312133+0000","last_peered":"2024-09-13T14:47:34.312133+0000","last_clean":"2024-09-13T14:45:19.637667+0000","last_became_active":"2024-09-13T14:45:21.025308+0000","last_became_peered":"2024-09-13T14:45:21.025308+0000","last_unstale":"2024-09-13T14:47:34.312133+0000","last_undegraded":"2024-09-13T14:47:34.312133+0000","last_fullsized":"2024-09-13T14:47:34.312133+0000","mapping_epoch":1149,"log_start":"1143'5300","ondisk_log_start":"1143'5300","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6899,"log_dups_size":0,"ondisk_log_size":6899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186056704,"num_objects":2839,"num_object_clones":0,"num_object_copies":8517,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2839,"num_objects_unfound":0,"num_objects_dirty":2839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12199,"num_write_kb":481216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,5],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":2839}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":42,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.755002+0000","last_change":"2024-09-13T14:43:32.636990+0000","last_active":"2024-09-13T14:47:05.755002+0000","last_peered":"2024-09-13T14:47:05.755002+0000","last_clean":"2024-09-13T14:47:05.755002+0000","last_became_active":"2024-09-13T14:43:32.636589+0000","last_became_peered":"2024-09-13T14:43:32.636589+0000","last_unstale":"2024-09-13T14:47:05.755002+0000","last_undegraded":"2024-09-13T14:47:05.755002+0000","last_fullsized":"2024-09-13T14:47:05.755002+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:36:25.129769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":35,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.856931+0000","last_change":"2024-09-13T14:45:19.810888+0000","last_active":"2024-09-13T14:47:23.856931+0000","last_peered":"2024-09-13T14:47:23.856931+0000","last_clean":"2024-09-13T14:47:23.856931+0000","last_became_active":"2024-09-13T14:45:19.810486+0000","last_became_peered":"2024-09-13T14:45:19.810486+0000","last_unstale":"2024-09-13T14:47:23.856931+0000","last_undegraded":"2024-09-13T14:47:23.856931+0000","last_fullsized":"2024-09-13T14:47:23.856931+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:03.886942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":1116,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.777414+0000","last_change":"2024-09-13T14:29:42.539455+0000","last_active":"2024-09-13T14:47:05.777414+0000","last_peered":"2024-09-13T14:47:05.777414+0000","last_clean":"2024-09-13T14:47:05.777414+0000","last_became_active":"2024-09-13T14:29:42.539068+0000","last_became_peered":"2024-09-13T14:29:42.539068+0000","last_unstale":"2024-09-13T14:47:05.777414+0000","last_undegraded":"2024-09-13T14:47:05.777414+0000","last_fullsized":"2024-09-13T14:47:05.777414+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:24:06.048855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034405999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"261'100","reported_seq":1271,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.745718+0000","last_change":"2024-09-13T14:44:28.186029+0000","last_active":"2024-09-13T14:47:05.745718+0000","last_peered":"2024-09-13T14:47:05.745718+0000","last_clean":"2024-09-13T14:47:05.745718+0000","last_became_active":"2024-09-13T14:42:14.927771+0000","last_became_peered":"2024-09-13T14:42:14.927771+0000","last_unstale":"2024-09-13T14:47:05.745718+0000","last_undegraded":"2024-09-13T14:47:05.745718+0000","last_fullsized":"2024-09-13T14:47:05.745718+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:28.185949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:28.185949+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:38.363961+0000","scrub_duration":0.0018071129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"1160'12002","reported_seq":12051,"reported_epoch":1160,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:47:31.625741+0000","last_change":"2024-09-13T14:45:21.030922+0000","last_active":"2024-09-13T14:47:31.625741+0000","last_peered":"2024-09-13T14:47:31.625741+0000","last_clean":"2024-09-13T14:45:19.630214+0000","last_became_active":"2024-09-13T14:45:21.021807+0000","last_became_peered":"2024-09-13T14:45:21.021807+0000","last_unstale":"2024-09-13T14:47:31.625741+0000","last_undegraded":"2024-09-13T14:47:31.625741+0000","last_fullsized":"2024-09-13T14:47:31.625741+0000","mapping_epoch":1149,"log_start":"1142'3600","ondisk_log_start":"1142'3600","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":8402,"log_dups_size":0,"ondisk_log_size":8402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192806912,"num_objects":2942,"num_object_clones":0,"num_object_copies":8826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2942,"num_objects_unfound":0,"num_objects_dirty":2942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12002,"num_write_kb":478208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[2,6,7],"avail_no_missing":["2","6","7"],"object_location_counts":[{"shards":"2,6,7","objects":2942}],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":83,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.846855+0000","last_change":"2024-09-13T14:43:07.520176+0000","last_active":"2024-09-13T14:47:23.846855+0000","last_peered":"2024-09-13T14:47:23.846855+0000","last_clean":"2024-09-13T14:47:23.846855+0000","last_became_active":"2024-09-13T14:43:07.516748+0000","last_became_peered":"2024-09-13T14:43:07.516748+0000","last_unstale":"2024-09-13T14:47:23.846855+0000","last_undegraded":"2024-09-13T14:47:23.846855+0000","last_fullsized":"2024-09-13T14:47:23.846855+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:52:04.392821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":35,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.854966+0000","last_change":"2024-09-13T14:45:19.812353+0000","last_active":"2024-09-13T14:47:23.854966+0000","last_peered":"2024-09-13T14:47:23.854966+0000","last_clean":"2024-09-13T14:47:23.854966+0000","last_became_active":"2024-09-13T14:45:19.811743+0000","last_became_peered":"2024-09-13T14:45:19.811743+0000","last_unstale":"2024-09-13T14:47:23.854966+0000","last_undegraded":"2024-09-13T14:47:23.854966+0000","last_fullsized":"2024-09-13T14:47:23.854966+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:43.728170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"261'100","reported_seq":100,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.775468+0000","last_change":"2024-09-13T14:44:34.814157+0000","last_active":"2024-09-13T14:47:05.775468+0000","last_peered":"2024-09-13T14:47:05.775468+0000","last_clean":"2024-09-13T14:47:05.775468+0000","last_became_active":"2024-09-13T14:42:42.707803+0000","last_became_peered":"2024-09-13T14:42:42.707803+0000","last_unstale":"2024-09-13T14:47:05.775468+0000","last_undegraded":"2024-09-13T14:47:05.775468+0000","last_fullsized":"2024-09-13T14:47:05.775468+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:34.814070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:34.814070+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:38.408561+0000","scrub_duration":0.01686414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"1160'12259","reported_seq":12320,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:32.715135+0000","last_change":"2024-09-13T14:43:51.534705+0000","last_active":"2024-09-13T14:47:32.715135+0000","last_peered":"2024-09-13T14:47:32.715135+0000","last_clean":"2024-09-13T14:47:32.715135+0000","last_became_active":"2024-09-13T14:43:51.534381+0000","last_became_peered":"2024-09-13T14:43:51.534381+0000","last_unstale":"2024-09-13T14:47:32.715135+0000","last_undegraded":"2024-09-13T14:47:32.715135+0000","last_fullsized":"2024-09-13T14:47:32.715135+0000","mapping_epoch":1136,"log_start":"1146'6200","ondisk_log_start":"1146'6200","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6059,"log_dups_size":0,"ondisk_log_size":6059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:37:22.385742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187695128,"num_objects":2865,"num_object_clones":0,"num_object_copies":8595,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12259,"num_write_kb":483905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1130,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.755429+0000","last_change":"2024-09-13T14:44:03.855577+0000","last_active":"2024-09-13T14:47:05.755429+0000","last_peered":"2024-09-13T14:47:05.755429+0000","last_clean":"2024-09-13T14:47:05.755429+0000","last_became_active":"2024-09-13T14:44:03.855267+0000","last_became_peered":"2024-09-13T14:44:03.855267+0000","last_unstale":"2024-09-13T14:47:05.755429+0000","last_undegraded":"2024-09-13T14:47:05.755429+0000","last_fullsized":"2024-09-13T14:47:05.755429+0000","mapping_epoch":1140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:46.148814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1101,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.755647+0000","last_change":"2024-09-13T14:29:42.545815+0000","last_active":"2024-09-13T14:47:05.755647+0000","last_peered":"2024-09-13T14:47:05.755647+0000","last_clean":"2024-09-13T14:47:05.755647+0000","last_became_active":"2024-09-13T14:29:42.545508+0000","last_became_peered":"2024-09-13T14:29:42.545508+0000","last_unstale":"2024-09-13T14:47:05.755647+0000","last_undegraded":"2024-09-13T14:47:05.755647+0000","last_fullsized":"2024-09-13T14:47:05.755647+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:39.678381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"261'100","reported_seq":101,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.851292+0000","last_change":"2024-09-13T14:43:18.988482+0000","last_active":"2024-09-13T14:47:23.851292+0000","last_peered":"2024-09-13T14:47:23.851292+0000","last_clean":"2024-09-13T14:47:23.851292+0000","last_became_active":"2024-09-13T14:43:14.556754+0000","last_became_peered":"2024-09-13T14:43:14.556754+0000","last_unstale":"2024-09-13T14:47:23.851292+0000","last_undegraded":"2024-09-13T14:47:23.851292+0000","last_fullsized":"2024-09-13T14:47:23.851292+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:18.988405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:18.988405+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:45:01.726326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"1160'11950","reported_seq":12010,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:31.548807+0000","last_change":"2024-09-13T14:43:50.796182+0000","last_active":"2024-09-13T14:47:31.548807+0000","last_peered":"2024-09-13T14:47:31.548807+0000","last_clean":"2024-09-13T14:47:31.548807+0000","last_became_active":"2024-09-13T14:43:50.795875+0000","last_became_peered":"2024-09-13T14:43:50.795875+0000","last_unstale":"2024-09-13T14:47:31.548807+0000","last_undegraded":"2024-09-13T14:47:31.548807+0000","last_fullsized":"2024-09-13T14:47:31.548807+0000","mapping_epoch":1136,"log_start":"1142'3600","ondisk_log_start":"1142'3600","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":8350,"log_dups_size":0,"ondisk_log_size":8350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:56:34.308870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184942592,"num_objects":2822,"num_object_clones":0,"num_object_copies":8466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11950,"num_write_kb":472704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1147,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.755257+0000","last_change":"2024-09-13T14:43:07.515418+0000","last_active":"2024-09-13T14:47:05.755257+0000","last_peered":"2024-09-13T14:47:05.755257+0000","last_clean":"2024-09-13T14:47:05.755257+0000","last_became_active":"2024-09-13T14:43:07.515113+0000","last_became_peered":"2024-09-13T14:43:07.515113+0000","last_unstale":"2024-09-13T14:47:05.755257+0000","last_undegraded":"2024-09-13T14:47:05.755257+0000","last_fullsized":"2024-09-13T14:47:05.755257+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:10:25.810604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1114,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.754525+0000","last_change":"2024-09-13T14:43:32.636334+0000","last_active":"2024-09-13T14:47:05.754525+0000","last_peered":"2024-09-13T14:47:05.754525+0000","last_clean":"2024-09-13T14:47:05.754525+0000","last_became_active":"2024-09-13T14:43:32.635736+0000","last_became_peered":"2024-09-13T14:43:32.635736+0000","last_unstale":"2024-09-13T14:47:05.754525+0000","last_undegraded":"2024-09-13T14:47:05.754525+0000","last_fullsized":"2024-09-13T14:47:05.754525+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:49.076014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"261'100","reported_seq":105,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.775290+0000","last_change":"2024-09-13T14:45:19.805634+0000","last_active":"2024-09-13T14:47:05.775290+0000","last_peered":"2024-09-13T14:47:05.775290+0000","last_clean":"2024-09-13T14:47:05.775290+0000","last_became_active":"2024-09-13T14:45:19.805046+0000","last_became_peered":"2024-09-13T14:45:19.805046+0000","last_unstale":"2024-09-13T14:47:05.775290+0000","last_undegraded":"2024-09-13T14:47:05.775290+0000","last_fullsized":"2024-09-13T14:47:05.775290+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:32.752408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:32.752408+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:06:53.140239+0000","scrub_duration":0.0057731559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"1160'12272","reported_seq":12337,"reported_epoch":1160,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:47:34.450874+0000","last_change":"2024-09-13T14:45:21.029184+0000","last_active":"2024-09-13T14:47:34.450874+0000","last_peered":"2024-09-13T14:47:34.450874+0000","last_clean":"2024-09-13T14:45:19.633080+0000","last_became_active":"2024-09-13T14:45:21.019809+0000","last_became_peered":"2024-09-13T14:45:21.019809+0000","last_unstale":"2024-09-13T14:47:34.450874+0000","last_undegraded":"2024-09-13T14:47:34.450874+0000","last_fullsized":"2024-09-13T14:47:34.450874+0000","mapping_epoch":1149,"log_start":"1144'5400","ondisk_log_start":"1144'5400","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6872,"log_dups_size":0,"ondisk_log_size":6872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190316544,"num_objects":2904,"num_object_clones":0,"num_object_copies":8712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2904,"num_objects_unfound":0,"num_objects_dirty":2904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12272,"num_write_kb":485632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,7,4],"avail_no_missing":["0","4","7"],"object_location_counts":[{"shards":"0,4,7","objects":2904}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1150,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.754815+0000","last_change":"2024-09-13T14:45:19.804441+0000","last_active":"2024-09-13T14:47:05.754815+0000","last_peered":"2024-09-13T14:47:05.754815+0000","last_clean":"2024-09-13T14:47:05.754815+0000","last_became_active":"2024-09-13T14:45:19.804039+0000","last_became_peered":"2024-09-13T14:45:19.804039+0000","last_unstale":"2024-09-13T14:47:05.754815+0000","last_undegraded":"2024-09-13T14:47:05.754815+0000","last_fullsized":"2024-09-13T14:47:05.754815+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:58:00.322042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":38,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.857003+0000","last_change":"2024-09-13T14:45:19.816666+0000","last_active":"2024-09-13T14:47:23.857003+0000","last_peered":"2024-09-13T14:47:23.857003+0000","last_clean":"2024-09-13T14:47:23.857003+0000","last_became_active":"2024-09-13T14:45:19.807224+0000","last_became_peered":"2024-09-13T14:45:19.807224+0000","last_unstale":"2024-09-13T14:47:23.857003+0000","last_undegraded":"2024-09-13T14:47:23.857003+0000","last_fullsized":"2024-09-13T14:47:23.857003+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:15:20.171682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"261'100","reported_seq":1290,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.745428+0000","last_change":"2024-09-13T14:45:19.798553+0000","last_active":"2024-09-13T14:47:05.745428+0000","last_peered":"2024-09-13T14:47:05.745428+0000","last_clean":"2024-09-13T14:47:05.745428+0000","last_became_active":"2024-09-13T14:45:19.798189+0000","last_became_peered":"2024-09-13T14:45:19.798189+0000","last_unstale":"2024-09-13T14:47:05.745428+0000","last_undegraded":"2024-09-13T14:47:05.745428+0000","last_fullsized":"2024-09-13T14:47:05.745428+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:29.169325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:29.169325+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:10.412666+0000","scrub_duration":0.0019358890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"1160'12114","reported_seq":12174,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:31.661140+0000","last_change":"2024-09-13T14:43:50.795029+0000","last_active":"2024-09-13T14:47:31.661140+0000","last_peered":"2024-09-13T14:47:31.661140+0000","last_clean":"2024-09-13T14:47:31.661140+0000","last_became_active":"2024-09-13T14:43:50.794715+0000","last_became_peered":"2024-09-13T14:43:50.794715+0000","last_unstale":"2024-09-13T14:47:31.661140+0000","last_undegraded":"2024-09-13T14:47:31.661140+0000","last_fullsized":"2024-09-13T14:47:31.661140+0000","mapping_epoch":1136,"log_start":"1142'3700","ondisk_log_start":"1142'3700","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":8414,"log_dups_size":0,"ondisk_log_size":8414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:21.257279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196214784,"num_objects":2994,"num_object_clones":0,"num_object_copies":8982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12114,"num_write_kb":483456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1703,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.851909+0000","last_change":"2024-09-13T14:43:07.515572+0000","last_active":"2024-09-13T14:47:23.851909+0000","last_peered":"2024-09-13T14:47:23.851909+0000","last_clean":"2024-09-13T14:47:23.851909+0000","last_became_active":"2024-09-13T14:43:07.515231+0000","last_became_peered":"2024-09-13T14:43:07.515231+0000","last_unstale":"2024-09-13T14:47:23.851909+0000","last_undegraded":"2024-09-13T14:47:23.851909+0000","last_fullsized":"2024-09-13T14:47:23.851909+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:57.966380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1143,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.775723+0000","last_change":"2024-09-13T14:45:19.804982+0000","last_active":"2024-09-13T14:47:05.775723+0000","last_peered":"2024-09-13T14:47:05.775723+0000","last_clean":"2024-09-13T14:47:05.775723+0000","last_became_active":"2024-09-13T14:45:19.804663+0000","last_became_peered":"2024-09-13T14:45:19.804663+0000","last_unstale":"2024-09-13T14:47:05.775723+0000","last_undegraded":"2024-09-13T14:47:05.775723+0000","last_fullsized":"2024-09-13T14:47:05.775723+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:43.211479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"261'100","reported_seq":380,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.755338+0000","last_change":"2024-09-13T14:43:42.881584+0000","last_active":"2024-09-13T14:47:05.755338+0000","last_peered":"2024-09-13T14:47:05.755338+0000","last_clean":"2024-09-13T14:47:05.755338+0000","last_became_active":"2024-09-13T14:43:14.562161+0000","last_became_peered":"2024-09-13T14:43:14.562161+0000","last_unstale":"2024-09-13T14:47:05.755338+0000","last_undegraded":"2024-09-13T14:47:05.755338+0000","last_fullsized":"2024-09-13T14:47:05.755338+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:42.881520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:42.881520+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:06:50.353442+0000","scrub_duration":0.0013804310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"1160'12289","reported_seq":12349,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:34.438140+0000","last_change":"2024-09-13T14:43:50.793879+0000","last_active":"2024-09-13T14:47:34.438140+0000","last_peered":"2024-09-13T14:47:34.438140+0000","last_clean":"2024-09-13T14:47:34.438140+0000","last_became_active":"2024-09-13T14:43:50.793571+0000","last_became_peered":"2024-09-13T14:43:50.793571+0000","last_unstale":"2024-09-13T14:47:34.438140+0000","last_undegraded":"2024-09-13T14:47:34.438140+0000","last_fullsized":"2024-09-13T14:47:34.438140+0000","mapping_epoch":1136,"log_start":"1144'5400","ondisk_log_start":"1144'5400","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6889,"log_dups_size":0,"ondisk_log_size":6889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:22.880360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191430656,"num_objects":2921,"num_object_clones":0,"num_object_copies":8763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12289,"num_write_kb":486720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1116,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.757091+0000","last_change":"2024-09-13T14:43:07.518120+0000","last_active":"2024-09-13T14:47:05.757091+0000","last_peered":"2024-09-13T14:47:05.757091+0000","last_clean":"2024-09-13T14:47:05.757091+0000","last_became_active":"2024-09-13T14:43:07.517361+0000","last_became_peered":"2024-09-13T14:43:07.517361+0000","last_unstale":"2024-09-13T14:47:05.757091+0000","last_undegraded":"2024-09-13T14:47:05.757091+0000","last_fullsized":"2024-09-13T14:47:05.757091+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:39.366473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":1098,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.746022+0000","last_change":"2024-09-13T14:19:25.871006+0000","last_active":"2024-09-13T14:47:05.746022+0000","last_peered":"2024-09-13T14:47:05.746022+0000","last_clean":"2024-09-13T14:47:05.746022+0000","last_became_active":"2024-09-13T14:19:25.870713+0000","last_became_peered":"2024-09-13T14:19:25.870713+0000","last_unstale":"2024-09-13T14:47:05.746022+0000","last_undegraded":"2024-09-13T14:47:05.746022+0000","last_fullsized":"2024-09-13T14:47:05.746022+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:34.372124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"261'100","reported_seq":364,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.775829+0000","last_change":"2024-09-13T14:43:32.637553+0000","last_active":"2024-09-13T14:47:05.775829+0000","last_peered":"2024-09-13T14:47:05.775829+0000","last_clean":"2024-09-13T14:47:05.775829+0000","last_became_active":"2024-09-13T14:43:32.636545+0000","last_became_peered":"2024-09-13T14:43:32.636545+0000","last_unstale":"2024-09-13T14:47:05.775829+0000","last_undegraded":"2024-09-13T14:47:05.775829+0000","last_fullsized":"2024-09-13T14:47:05.775829+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:15:42.069756+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"1160'12069","reported_seq":12126,"reported_epoch":1160,"state":"active+recovery_wait+remapped","last_fresh":"2024-09-13T14:47:32.723726+0000","last_change":"2024-09-13T14:45:21.130410+0000","last_active":"2024-09-13T14:47:32.723726+0000","last_peered":"2024-09-13T14:47:32.723726+0000","last_clean":"2024-09-13T14:45:19.587556+0000","last_became_active":"2024-09-13T14:45:21.100167+0000","last_became_peered":"2024-09-13T14:45:21.100167+0000","last_unstale":"2024-09-13T14:47:32.723726+0000","last_undegraded":"2024-09-13T14:47:32.723726+0000","last_fullsized":"2024-09-13T14:47:32.723726+0000","mapping_epoch":1149,"log_start":"1146'5977","ondisk_log_start":"1146'5977","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6092,"log_dups_size":0,"ondisk_log_size":6092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193134592,"num_objects":2947,"num_object_clones":0,"num_object_copies":8841,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7077,"num_objects_unfound":0,"num_objects_dirty":2947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12069,"num_write_kb":480512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[1,2,5],"avail_no_missing":["1","2","5"],"object_location_counts":[{"shards":"1,2,5","objects":-4130}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1167,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.755647+0000","last_change":"2024-09-13T14:45:19.804868+0000","last_active":"2024-09-13T14:47:05.755647+0000","last_peered":"2024-09-13T14:47:05.755647+0000","last_clean":"2024-09-13T14:47:05.755647+0000","last_became_active":"2024-09-13T14:45:19.804520+0000","last_became_peered":"2024-09-13T14:45:19.804520+0000","last_unstale":"2024-09-13T14:47:05.755647+0000","last_undegraded":"2024-09-13T14:47:05.755647+0000","last_fullsized":"2024-09-13T14:47:05.755647+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:17:53.924629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":1144,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.746070+0000","last_change":"2024-09-13T14:43:07.533955+0000","last_active":"2024-09-13T14:47:05.746070+0000","last_peered":"2024-09-13T14:47:05.746070+0000","last_clean":"2024-09-13T14:47:05.746070+0000","last_became_active":"2024-09-13T14:43:07.516816+0000","last_became_peered":"2024-09-13T14:43:07.516816+0000","last_unstale":"2024-09-13T14:47:05.746070+0000","last_undegraded":"2024-09-13T14:47:05.746070+0000","last_fullsized":"2024-09-13T14:47:05.746070+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:26.232184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"261'100","reported_seq":38,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.858221+0000","last_change":"2024-09-13T14:45:19.821649+0000","last_active":"2024-09-13T14:47:23.858221+0000","last_peered":"2024-09-13T14:47:23.858221+0000","last_clean":"2024-09-13T14:47:23.858221+0000","last_became_active":"2024-09-13T14:45:19.812346+0000","last_became_peered":"2024-09-13T14:45:19.812346+0000","last_unstale":"2024-09-13T14:47:23.858221+0000","last_undegraded":"2024-09-13T14:47:23.858221+0000","last_fullsized":"2024-09-13T14:47:23.858221+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.597568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.597568+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:45.218892+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"1160'12163","reported_seq":12223,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:32.716417+0000","last_change":"2024-09-13T14:43:51.537117+0000","last_active":"2024-09-13T14:47:32.716417+0000","last_peered":"2024-09-13T14:47:32.716417+0000","last_clean":"2024-09-13T14:47:32.716417+0000","last_became_active":"2024-09-13T14:43:51.536532+0000","last_became_peered":"2024-09-13T14:43:51.536532+0000","last_unstale":"2024-09-13T14:47:32.716417+0000","last_undegraded":"2024-09-13T14:47:32.716417+0000","last_fullsized":"2024-09-13T14:47:32.716417+0000","mapping_epoch":1136,"log_start":"1146'6100","ondisk_log_start":"1146'6100","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6063,"log_dups_size":0,"ondisk_log_size":6063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:41.713211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183697408,"num_objects":2803,"num_object_clones":0,"num_object_copies":8409,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":478912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1085,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.775433+0000","last_change":"2024-09-13T14:19:26.887839+0000","last_active":"2024-09-13T14:47:05.775433+0000","last_peered":"2024-09-13T14:47:05.775433+0000","last_clean":"2024-09-13T14:47:05.775433+0000","last_became_active":"2024-09-13T14:19:26.887498+0000","last_became_peered":"2024-09-13T14:19:26.887498+0000","last_unstale":"2024-09-13T14:47:05.775433+0000","last_undegraded":"2024-09-13T14:47:05.775433+0000","last_fullsized":"2024-09-13T14:47:05.775433+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:24.475263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":1115,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.746179+0000","last_change":"2024-09-13T14:28:35.608607+0000","last_active":"2024-09-13T14:47:05.746179+0000","last_peered":"2024-09-13T14:47:05.746179+0000","last_clean":"2024-09-13T14:47:05.746179+0000","last_became_active":"2024-09-13T14:28:35.608254+0000","last_became_peered":"2024-09-13T14:28:35.608254+0000","last_unstale":"2024-09-13T14:47:05.746179+0000","last_undegraded":"2024-09-13T14:47:05.746179+0000","last_fullsized":"2024-09-13T14:47:05.746179+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:24.019173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"261'100","reported_seq":51,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.745797+0000","last_change":"2024-09-13T14:44:31.191431+0000","last_active":"2024-09-13T14:47:05.745797+0000","last_peered":"2024-09-13T14:47:05.745797+0000","last_clean":"2024-09-13T14:47:05.745797+0000","last_became_active":"2024-09-13T14:43:14.557376+0000","last_became_peered":"2024-09-13T14:43:14.557376+0000","last_unstale":"2024-09-13T14:47:05.745797+0000","last_undegraded":"2024-09-13T14:47:05.745797+0000","last_fullsized":"2024-09-13T14:47:05.745797+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.191326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.191326+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:15:27.352698+0000","scrub_duration":0.0025317360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1105,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.756991+0000","last_change":"2024-09-13T14:28:35.603724+0000","last_active":"2024-09-13T14:47:05.756991+0000","last_peered":"2024-09-13T14:47:05.756991+0000","last_clean":"2024-09-13T14:47:05.756991+0000","last_became_active":"2024-09-13T14:28:35.603316+0000","last_became_peered":"2024-09-13T14:28:35.603316+0000","last_unstale":"2024-09-13T14:47:05.756991+0000","last_undegraded":"2024-09-13T14:47:05.756991+0000","last_fullsized":"2024-09-13T14:47:05.756991+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:47.668223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":34,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.858611+0000","last_change":"2024-09-13T14:45:19.811466+0000","last_active":"2024-09-13T14:47:23.858611+0000","last_peered":"2024-09-13T14:47:23.858611+0000","last_clean":"2024-09-13T14:47:23.858611+0000","last_became_active":"2024-09-13T14:45:19.810617+0000","last_became_peered":"2024-09-13T14:45:19.810617+0000","last_unstale":"2024-09-13T14:47:23.858611+0000","last_undegraded":"2024-09-13T14:47:23.858611+0000","last_fullsized":"2024-09-13T14:47:23.858611+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:16.246669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"261'100","reported_seq":51,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.775472+0000","last_change":"2024-09-13T14:44:50.805306+0000","last_active":"2024-09-13T14:47:05.775472+0000","last_peered":"2024-09-13T14:47:05.775472+0000","last_clean":"2024-09-13T14:47:05.775472+0000","last_became_active":"2024-09-13T14:43:14.562922+0000","last_became_peered":"2024-09-13T14:43:14.562922+0000","last_unstale":"2024-09-13T14:47:05.775472+0000","last_undegraded":"2024-09-13T14:47:05.775472+0000","last_fullsized":"2024-09-13T14:47:05.775472+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:50.805210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:50.805210+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:28:38.529599+0000","scrub_duration":0.0042900009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1097,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.745518+0000","last_change":"2024-09-13T14:19:25.873088+0000","last_active":"2024-09-13T14:47:05.745518+0000","last_peered":"2024-09-13T14:47:05.745518+0000","last_clean":"2024-09-13T14:47:05.745518+0000","last_became_active":"2024-09-13T14:19:25.872781+0000","last_became_peered":"2024-09-13T14:19:25.872781+0000","last_unstale":"2024-09-13T14:47:05.745518+0000","last_undegraded":"2024-09-13T14:47:05.745518+0000","last_fullsized":"2024-09-13T14:47:05.745518+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:05.607382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1702,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.861437+0000","last_change":"2024-09-13T14:45:19.813876+0000","last_active":"2024-09-13T14:47:23.861437+0000","last_peered":"2024-09-13T14:47:23.861437+0000","last_clean":"2024-09-13T14:47:23.861437+0000","last_became_active":"2024-09-13T14:45:19.809136+0000","last_became_peered":"2024-09-13T14:45:19.809136+0000","last_unstale":"2024-09-13T14:47:23.861437+0000","last_undegraded":"2024-09-13T14:47:23.861437+0000","last_fullsized":"2024-09-13T14:47:23.861437+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:50.013528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"261'100","reported_seq":108,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.851428+0000","last_change":"2024-09-13T14:45:19.810654+0000","last_active":"2024-09-13T14:47:23.851428+0000","last_peered":"2024-09-13T14:47:23.851428+0000","last_clean":"2024-09-13T14:47:23.851428+0000","last_became_active":"2024-09-13T14:45:19.810066+0000","last_became_peered":"2024-09-13T14:45:19.810066+0000","last_unstale":"2024-09-13T14:47:23.851428+0000","last_undegraded":"2024-09-13T14:47:23.851428+0000","last_fullsized":"2024-09-13T14:47:23.851428+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:19.978464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:19.978464+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:33.639332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1750,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.859011+0000","last_change":"2024-09-13T14:45:19.809166+0000","last_active":"2024-09-13T14:47:23.859011+0000","last_peered":"2024-09-13T14:47:23.859011+0000","last_clean":"2024-09-13T14:47:23.859011+0000","last_became_active":"2024-09-13T14:45:19.808631+0000","last_became_peered":"2024-09-13T14:45:19.808631+0000","last_unstale":"2024-09-13T14:47:23.859011+0000","last_undegraded":"2024-09-13T14:47:23.859011+0000","last_fullsized":"2024-09-13T14:47:23.859011+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:38.822245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1122,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.775667+0000","last_change":"2024-09-13T14:45:19.804229+0000","last_active":"2024-09-13T14:47:05.775667+0000","last_peered":"2024-09-13T14:47:05.775667+0000","last_clean":"2024-09-13T14:47:05.775667+0000","last_became_active":"2024-09-13T14:45:19.803929+0000","last_became_peered":"2024-09-13T14:45:19.803929+0000","last_unstale":"2024-09-13T14:47:05.775667+0000","last_undegraded":"2024-09-13T14:47:05.775667+0000","last_fullsized":"2024-09-13T14:47:05.775667+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:33:53.675568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"261'100","reported_seq":51,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.775200+0000","last_change":"2024-09-13T14:44:42.654506+0000","last_active":"2024-09-13T14:47:05.775200+0000","last_peered":"2024-09-13T14:47:05.775200+0000","last_clean":"2024-09-13T14:47:05.775200+0000","last_became_active":"2024-09-13T14:43:14.563451+0000","last_became_peered":"2024-09-13T14:43:14.563451+0000","last_unstale":"2024-09-13T14:47:05.775200+0000","last_undegraded":"2024-09-13T14:47:05.775200+0000","last_fullsized":"2024-09-13T14:47:05.775200+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:42.654455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:42.654455+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:57:32.251597+0000","scrub_duration":0.001494097,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":1116,"reported_epoch":1159,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.745693+0000","last_change":"2024-09-13T14:29:42.541093+0000","last_active":"2024-09-13T14:47:05.745693+0000","last_peered":"2024-09-13T14:47:05.745693+0000","last_clean":"2024-09-13T14:47:05.745693+0000","last_became_active":"2024-09-13T14:29:42.540671+0000","last_became_peered":"2024-09-13T14:29:42.540671+0000","last_unstale":"2024-09-13T14:47:05.745693+0000","last_undegraded":"2024-09-13T14:47:05.745693+0000","last_fullsized":"2024-09-13T14:47:05.745693+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:45.722376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1641,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.858515+0000","last_change":"2024-09-13T14:43:07.518679+0000","last_active":"2024-09-13T14:47:23.858515+0000","last_peered":"2024-09-13T14:47:23.858515+0000","last_clean":"2024-09-13T14:47:23.858515+0000","last_became_active":"2024-09-13T14:43:07.518333+0000","last_became_peered":"2024-09-13T14:43:07.518333+0000","last_unstale":"2024-09-13T14:47:23.858515+0000","last_undegraded":"2024-09-13T14:47:23.858515+0000","last_fullsized":"2024-09-13T14:47:23.858515+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:20.616274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"261'100","reported_seq":410,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.850623+0000","last_change":"2024-09-13T14:45:19.809268+0000","last_active":"2024-09-13T14:47:23.850623+0000","last_peered":"2024-09-13T14:47:23.850623+0000","last_clean":"2024-09-13T14:47:23.850623+0000","last_became_active":"2024-09-13T14:45:19.808772+0000","last_became_peered":"2024-09-13T14:45:19.808772+0000","last_unstale":"2024-09-13T14:47:23.850623+0000","last_undegraded":"2024-09-13T14:47:23.850623+0000","last_fullsized":"2024-09-13T14:47:23.850623+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:14.654603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:14.654603+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:40.964553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3040608280,"num_objects":46397,"num_object_clones":0,"num_object_copies":139191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2817,"num_objects_misplaced":29854,"num_objects_unfound":0,"num_objects_dirty":46397,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":194521,"num_write_kb":7709313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1764,"num_bytes_recovered":115605504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9480450048,"data_stored":9480437832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112344,"ondisk_log_size":112344,"up":48,"acting":47,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":354515,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430570,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1455316,"kb_used_data":1153804,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":92289836,"statfs":{"total":95995035648,"available":94504792064,"internally_reserved":0,"allocated":1181495296,"data_stored":1177942612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626799,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1695244,"kb_used_data":1385412,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":92049908,"statfs":{"total":95995035648,"available":94259105792,"internally_reserved":0,"allocated":1418661888,"data_stored":1415116757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823028,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1240388,"kb_used_data":934716,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92504764,"statfs":{"total":95995035648,"available":94724878336,"internally_reserved":0,"allocated":957149184,"data_stored":953608583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019256,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1456268,"kb_used_data":1146436,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":92288884,"statfs":{"total":95995035648,"available":94503817216,"internally_reserved":0,"allocated":1173950464,"data_stored":1170405358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059592914,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":254456,"kb_used_data":67824,"kb_used_omap":26,"kb_used_meta":186597,"kb_avail":93490696,"statfs":{"total":95995035648,"available":95734472704,"internally_reserved":0,"allocated":69451776,"data_stored":65922370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":191075488},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":591,"seq":2538325672152,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1914752,"kb_used_data":1529528,"kb_used_omap":41,"kb_used_meta":385110,"kb_avail":91830400,"statfs":{"total":95995035648,"available":94034329600,"internally_reserved":0,"allocated":1566236672,"data_stored":1562700140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42651,"internal_metadata":394352997},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,0,0,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":557,"seq":2392296784095,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2072444,"kb_used_data":1702900,"kb_used_omap":32,"kb_used_meta":369503,"kb_avail":91672708,"statfs":{"total":95995035648,"available":93872852992,"internally_reserved":0,"allocated":1743769600,"data_stored":1740236067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33752,"internal_metadata":378371112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":524,"seq":2250562863334,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1784512,"kb_used_data":1472760,"kb_used_omap":41,"kb_used_meta":311702,"kb_avail":91960640,"statfs":{"total":95995035648,"available":94167695360,"internally_reserved":0,"allocated":1508106240,"data_stored":1504569192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42694,"internal_metadata":319183162},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1484591104,"data_stored":1484587032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1713311744,"data_stored":1713307672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1555038208,"data_stored":1555038208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":58392576,"data_stored":58392576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1162870784,"data_stored":1162870784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":945487872,"data_stored":945487872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1399001088,"data_stored":1398997016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1161756672,"data_stored":1161756672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:47:38.559 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:38.585 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:38.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:38.687 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:38.710 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:38.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:38.812 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:38.835 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:38.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:38.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:38 smithi184 ceph-mon[92527]: pgmap v1612: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 573 op/s; 2919/137277 objects degraded (2.126%); 29521/137277 objects misplaced (21.505%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:47:38.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:38 smithi151 ceph-mon[108966]: pgmap v1612: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 573 op/s; 2919/137277 objects degraded (2.126%); 29521/137277 objects misplaced (21.505%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:47:38.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:38 smithi151 ceph-mon[104450]: pgmap v1612: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 573 op/s; 2919/137277 objects degraded (2.126%); 29521/137277 objects misplaced (21.505%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:47:38.937 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:38.964 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:38.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:39.066 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:39.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:39.115 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:39.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:39.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:39.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:39.264 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:39.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:39.365 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:39.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:39.416 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:39.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:39.517 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:39.566 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:39.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:39.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:39.668 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:39.691 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:39.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:39.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:39.792 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:39.815 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:39.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:39.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:39.917 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:39.966 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:39.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:39.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:40.068 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:40.101 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:40.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:40.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:40.203 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:40.224 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:40.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:40.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:40.326 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:40.347 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:40.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:40.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:40.448 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:40.470 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:40.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:40.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:40.572 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:40.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:47:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:47:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:47:40.619 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:40.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:40.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:40.721 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:40.763 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:40.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:40.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:40.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:40 smithi184 ceph-mon[92527]: pgmap v1613: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 596 op/s; 2919/133797 objects degraded (2.182%); 29015/133797 objects misplaced (21.686%); 858 KiB/s, 13 objects/s recovering 2024-09-13T14:47:40.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:40 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 2919/133797 objects degraded (2.182%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:40.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:40 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1376293269' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:40.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:40 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:40.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:40 smithi151 ceph-mon[104450]: pgmap v1613: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 596 op/s; 2919/133797 objects degraded (2.182%); 29015/133797 objects misplaced (21.686%); 858 KiB/s, 13 objects/s recovering 2024-09-13T14:47:40.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:40 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 2919/133797 objects degraded (2.182%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:40.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:40 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1376293269' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:40.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:40 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:40.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:40 smithi151 ceph-mon[108966]: pgmap v1613: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 596 op/s; 2919/133797 objects degraded (2.182%); 29015/133797 objects misplaced (21.686%); 858 KiB/s, 13 objects/s recovering 2024-09-13T14:47:40.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:40 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 2919/133797 objects degraded (2.182%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:40.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:40 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1376293269' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:40.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:40 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:40.865 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:40.886 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:40.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:40.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:40.988 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:41.010 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:41.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:41.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:41.112 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:41.135 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:41.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:41.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:41.237 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:41.260 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:41.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:41.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:41.362 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:41.397 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:41.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:41.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:41.499 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:41.505 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:47:41.521 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:41.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:41.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:41.623 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:41.684 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:41.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:41.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:41.785 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:41.808 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:41.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:41.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:41.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:41 smithi151 ceph-mon[104450]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:41.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:41 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:47:41.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:41 smithi151 ceph-mon[104450]: osdmap e1161: 8 total, 8 up, 8 in 2024-09-13T14:47:41.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:41 smithi151 ceph-mon[108966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:41.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:41 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:47:41.862 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:41 smithi151 ceph-mon[108966]: osdmap e1161: 8 total, 8 up, 8 in 2024-09-13T14:47:41.910 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:41.960 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:41.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:41.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:42.062 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:42.100 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:42.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:42.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:42.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:41 smithi184 ceph-mon[92527]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:42.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:41 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:47:42.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:41 smithi184 ceph-mon[92527]: osdmap e1161: 8 total, 8 up, 8 in 2024-09-13T14:47:42.202 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:42.226 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:42.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:42.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:42.252 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-13T14:47:42.328 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:42.355 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:42.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:42.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:42.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:42.500 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:42.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:42.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:42.602 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:42.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:42.644 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:42.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:42.744 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:42.790 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:42 smithi151 ceph-mon[108966]: pgmap v1615: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 562 op/s; 2919/131397 objects degraded (2.222%); 28529/131397 objects misplaced (21.712%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:47:42.792 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:42 smithi151 ceph-mon[104450]: pgmap v1615: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 562 op/s; 2919/131397 objects degraded (2.222%); 28529/131397 objects misplaced (21.712%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:47:42.803 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:42.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:42.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:42.905 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:42.930 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:42.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:42.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:43.032 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:43.070 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:43.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:43.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:42 smithi184 ceph-mon[92527]: pgmap v1615: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 562 op/s; 2919/131397 objects degraded (2.222%); 28529/131397 objects misplaced (21.712%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:47:43.172 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:43.212 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:43.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:43.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:43.314 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:43.351 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:43.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:43.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:43.452 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:43.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:43.489 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:43.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:43.590 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:43.633 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:43.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:43.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:43.735 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:43.775 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:43.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:43.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:43.877 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:43.916 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:43.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:43.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:44.018 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:44.043 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:44.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:44.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:44.145 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:44.179 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:44.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:44.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:44.280 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:44.322 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:44.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:44.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:44.326 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:44.350 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:47:44.350 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-13T14:47:44.350 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-13T14:47:44.351 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-13T14:47:44.351 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.351 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:183098 6.b 6:d09715c7:::benchmark_data_smithi151_155836_object50640:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1113)", 2024-09-13T14:47:44.351 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:41:59.194610+0000", 2024-09-13T14:47:44.351 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 345.15159687400001, 2024-09-13T14:47:44.351 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.36153331399999999, 2024-09-13T14:47:44.351 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:44.351 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:44.351 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:44.351 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:44.351 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:47:44.351 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:47:44.352 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 183098 2024-09-13T14:47:44.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.352 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:44.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:44.352 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.194610+0000", 2024-09-13T14:47:44.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:44.352 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.194610+0000", 2024-09-13T14:47:44.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.353 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:44.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.194612+0000", 2024-09-13T14:47:44.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.502e-06 2024-09-13T14:47:44.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.353 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:44.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.194616+0000", 2024-09-13T14:47:44.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1060000000000002e-06 2024-09-13T14:47:44.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.353 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:44.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.194632+0000", 2024-09-13T14:47:44.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6325e-05 2024-09-13T14:47:44.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.354 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:44.354 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.194639+0000", 2024-09-13T14:47:44.354 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2690000000000002e-06 2024-09-13T14:47:44.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.354 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.354 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.200186+0000", 2024-09-13T14:47:44.354 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0055466229999999997 2024-09-13T14:47:44.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.354 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:47:44.355 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.200308+0000", 2024-09-13T14:47:44.355 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012204 2024-09-13T14:47:44.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.355 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.355 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.520305+0000", 2024-09-13T14:47:44.355 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31999744699999999 2024-09-13T14:47:44.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.355 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:44.355 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.520393+0000", 2024-09-13T14:47:44.355 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7542999999999999e-05 2024-09-13T14:47:44.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.356 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:47:44.356 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.520507+0000", 2024-09-13T14:47:44.356 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011389 2024-09-13T14:47:44.356 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.356 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.356 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.356 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.538812+0000", 2024-09-13T14:47:44.356 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018305578999999999 2024-09-13T14:47:44.356 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.356 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.356 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:44.356 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.538823+0000", 2024-09-13T14:47:44.356 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0817e-05 2024-09-13T14:47:44.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.357 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.357 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.553071+0000", 2024-09-13T14:47:44.357 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014247261000000001 2024-09-13T14:47:44.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.357 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:44.357 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.553145+0000", 2024-09-13T14:47:44.357 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4408999999999999e-05 2024-09-13T14:47:44.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.357 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:44.358 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.556143+0000", 2024-09-13T14:47:44.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002998503 2024-09-13T14:47:44.358 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.358 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:44.358 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.358 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.358 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:183616 6.b 6:d73bec53:::benchmark_data_smithi151_155836_object51158:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1113)", 2024-09-13T14:47:44.358 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:41:59.678101+0000", 2024-09-13T14:47:44.358 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 344.66810626199998, 2024-09-13T14:47:44.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.39668080999999999, 2024-09-13T14:47:44.358 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:44.358 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:44.358 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:44.359 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:44.359 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:47:44.359 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:47:44.359 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 183616 2024-09-13T14:47:44.359 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.359 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:44.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.359 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:44.359 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.678101+0000", 2024-09-13T14:47:44.359 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.359 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.359 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:44.359 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.678101+0000", 2024-09-13T14:47:44.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.360 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:44.360 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.678103+0000", 2024-09-13T14:47:44.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1009999999999999e-06 2024-09-13T14:47:44.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.360 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:44.360 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.678109+0000", 2024-09-13T14:47:44.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.93e-06 2024-09-13T14:47:44.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.361 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:44.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.678127+0000", 2024-09-13T14:47:44.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8002000000000001e-05 2024-09-13T14:47:44.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.361 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:44.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.678135+0000", 2024-09-13T14:47:44.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4100000000000008e-06 2024-09-13T14:47:44.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.361 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.699979+0000", 2024-09-13T14:47:44.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021843619000000002 2024-09-13T14:47:44.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:47:44.362 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:41:59.700100+0000", 2024-09-13T14:47:44.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000120896 2024-09-13T14:47:44.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.362 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:00.051897+0000", 2024-09-13T14:47:44.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35179707900000001 2024-09-13T14:47:44.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:44.363 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:00.052009+0000", 2024-09-13T14:47:44.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011215 2024-09-13T14:47:44.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.363 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:47:44.363 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:00.052168+0000", 2024-09-13T14:47:44.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015894600000000001 2024-09-13T14:47:44.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.363 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.363 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:00.071742+0000", 2024-09-13T14:47:44.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019573958999999998 2024-09-13T14:47:44.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:44.364 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:00.071755+0000", 2024-09-13T14:47:44.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3291e-05 2024-09-13T14:47:44.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.364 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:00.071825+0000", 2024-09-13T14:47:44.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9987999999999995e-05 2024-09-13T14:47:44.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:44.364 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:00.071894+0000", 2024-09-13T14:47:44.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9153999999999996e-05 2024-09-13T14:47:44.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:44.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:00.074782+0000", 2024-09-13T14:47:44.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028872849999999999 2024-09-13T14:47:44.365 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.365 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:44.365 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.365 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:187151 6.b 6:decc40c7:::benchmark_data_smithi151_155836_object54693:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1114)", 2024-09-13T14:47:44.365 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:42:02.893043+0000", 2024-09-13T14:47:44.366 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 341.45316374999999, 2024-09-13T14:47:44.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.469180977, 2024-09-13T14:47:44.366 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:44.366 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:44.366 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:44.366 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:44.366 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:47:44.366 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:47:44.366 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 187151 2024-09-13T14:47:44.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.366 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:44.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:44.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.893043+0000", 2024-09-13T14:47:44.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:44.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.893043+0000", 2024-09-13T14:47:44.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:44.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.893045+0000", 2024-09-13T14:47:44.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8300000000000001e-06 2024-09-13T14:47:44.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:44.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.893052+0000", 2024-09-13T14:47:44.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8399999999999997e-06 2024-09-13T14:47:44.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:44.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.893072+0000", 2024-09-13T14:47:44.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9879000000000001e-05 2024-09-13T14:47:44.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:44.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.893081+0000", 2024-09-13T14:47:44.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3130000000000004e-06 2024-09-13T14:47:44.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.369 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.369 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.906416+0000", 2024-09-13T14:47:44.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013334917999999999 2024-09-13T14:47:44.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.369 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:47:44.369 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:02.906486+0000", 2024-09-13T14:47:44.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0264000000000004e-05 2024-09-13T14:47:44.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.369 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.369 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.320086+0000", 2024-09-13T14:47:44.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.41359949600000001 2024-09-13T14:47:44.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:44.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.320218+0000", 2024-09-13T14:47:44.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013243900000000001 2024-09-13T14:47:44.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:47:44.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.320403+0000", 2024-09-13T14:47:44.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018463400000000001 2024-09-13T14:47:44.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.352889+0000", 2024-09-13T14:47:44.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.03248579 2024-09-13T14:47:44.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.352929+0000", 2024-09-13T14:47:44.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0676000000000002e-05 2024-09-13T14:47:44.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:44.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.358639+0000", 2024-09-13T14:47:44.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0057095870000000003 2024-09-13T14:47:44.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:44.372 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.358704+0000", 2024-09-13T14:47:44.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4659000000000006e-05 2024-09-13T14:47:44.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:44.372 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:03.362224+0000", 2024-09-13T14:47:44.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003520652 2024-09-13T14:47:44.372 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.372 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:44.372 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.373 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:188826 6.b 6:d0c74c68:::benchmark_data_smithi151_155836_object56368:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1114)", 2024-09-13T14:47:44.373 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:42:04.417169+0000", 2024-09-13T14:47:44.373 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 339.92903778900001, 2024-09-13T14:47:44.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38574094599999997, 2024-09-13T14:47:44.373 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:44.373 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:44.373 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:44.373 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:44.373 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:47:44.373 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:47:44.373 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 188826 2024-09-13T14:47:44.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.374 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:44.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:44.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.417169+0000", 2024-09-13T14:47:44.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:44.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.417169+0000", 2024-09-13T14:47:44.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:44.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.417170+0000", 2024-09-13T14:47:44.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8299999999999995e-07 2024-09-13T14:47:44.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:44.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.417174+0000", 2024-09-13T14:47:44.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2560000000000004e-06 2024-09-13T14:47:44.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:44.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.417183+0000", 2024-09-13T14:47:44.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5089999999999993e-06 2024-09-13T14:47:44.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:44.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.417190+0000", 2024-09-13T14:47:44.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9990000000000002e-06 2024-09-13T14:47:44.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.433627+0000", 2024-09-13T14:47:44.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016436563000000001 2024-09-13T14:47:44.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:47:44.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.433755+0000", 2024-09-13T14:47:44.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012852999999999999 2024-09-13T14:47:44.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.377 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.737039+0000", 2024-09-13T14:47:44.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.30328408000000001 2024-09-13T14:47:44.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:44.377 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.737150+0000", 2024-09-13T14:47:44.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011105899999999999 2024-09-13T14:47:44.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:47:44.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.737326+0000", 2024-09-13T14:47:44.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017530099999999999 2024-09-13T14:47:44.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:44.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.756971+0000", 2024-09-13T14:47:44.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019645875 2024-09-13T14:47:44.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.757196+0000", 2024-09-13T14:47:44.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000224293 2024-09-13T14:47:44.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.800579+0000", 2024-09-13T14:47:44.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.043383781000000003 2024-09-13T14:47:44.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:44.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.800647+0000", 2024-09-13T14:47:44.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7751999999999998e-05 2024-09-13T14:47:44.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:44.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:04.802910+0000", 2024-09-13T14:47:44.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002262965 2024-09-13T14:47:44.381 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.381 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:44.381 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.381 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:190363 6.b 6:d5a279ab:::benchmark_data_smithi151_155836_object57905:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1114)", 2024-09-13T14:47:44.381 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:42:05.910014+0000", 2024-09-13T14:47:44.381 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 338.43619342300002, 2024-09-13T14:47:44.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.64606729100000004, 2024-09-13T14:47:44.381 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:44.381 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:44.382 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:44.382 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:44.382 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:47:44.382 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:47:44.382 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 190363 2024-09-13T14:47:44.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.382 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:44.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:44.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:05.910014+0000", 2024-09-13T14:47:44.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:44.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:05.910014+0000", 2024-09-13T14:47:44.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:44.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:05.910015+0000", 2024-09-13T14:47:44.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.091e-06 2024-09-13T14:47:44.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:44.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:05.910019+0000", 2024-09-13T14:47:44.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9779999999999999e-06 2024-09-13T14:47:44.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:44.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:05.910031+0000", 2024-09-13T14:47:44.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.183e-05 2024-09-13T14:47:44.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:44.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:05.910036+0000", 2024-09-13T14:47:44.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4609999999999999e-06 2024-09-13T14:47:44.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:05.938331+0000", 2024-09-13T14:47:44.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.028295247999999999 2024-09-13T14:47:44.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:47:44.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:05.938412+0000", 2024-09-13T14:47:44.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1163999999999997e-05 2024-09-13T14:47:44.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:06.533060+0000", 2024-09-13T14:47:44.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.59464760900000002 2024-09-13T14:47:44.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:44.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:06.533158+0000", 2024-09-13T14:47:44.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8400000000000007e-05 2024-09-13T14:47:44.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:47:44.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:06.533330+0000", 2024-09-13T14:47:44.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000171516 2024-09-13T14:47:44.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:06.553204+0000", 2024-09-13T14:47:44.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019874315 2024-09-13T14:47:44.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:06.553264+0000", 2024-09-13T14:47:44.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9621999999999999e-05 2024-09-13T14:47:44.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:44.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:06.553280+0000", 2024-09-13T14:47:44.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6354999999999998e-05 2024-09-13T14:47:44.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:44.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:06.553333+0000", 2024-09-13T14:47:44.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2459e-05 2024-09-13T14:47:44.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:44.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:06.556081+0000", 2024-09-13T14:47:44.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002748243 2024-09-13T14:47:44.388 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.388 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:44.388 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.388 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:192127 6.b 6:dbb5192a:::benchmark_data_smithi151_155836_object59669:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1114)", 2024-09-13T14:47:44.388 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:42:07.565556+0000", 2024-09-13T14:47:44.389 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 336.78065094800002, 2024-09-13T14:47:44.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.391339781, 2024-09-13T14:47:44.389 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:44.389 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:44.389 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:44.389 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:44.389 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:47:44.389 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:47:44.389 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 192127 2024-09-13T14:47:44.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.389 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:44.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:44.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.565556+0000", 2024-09-13T14:47:44.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:44.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.565556+0000", 2024-09-13T14:47:44.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:44.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.565558+0000", 2024-09-13T14:47:44.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6449999999999999e-06 2024-09-13T14:47:44.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.391 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:44.391 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.565563+0000", 2024-09-13T14:47:44.391 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9180000000000002e-06 2024-09-13T14:47:44.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.391 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:44.391 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.565575+0000", 2024-09-13T14:47:44.391 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2731000000000001e-05 2024-09-13T14:47:44.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.391 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:44.391 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.565584+0000", 2024-09-13T14:47:44.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8950000000000004e-06 2024-09-13T14:47:44.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.392 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.582767+0000", 2024-09-13T14:47:44.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017182317999999999 2024-09-13T14:47:44.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.392 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:47:44.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.582896+0000", 2024-09-13T14:47:44.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000129111 2024-09-13T14:47:44.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.392 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.915205+0000", 2024-09-13T14:47:44.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33230937599999999 2024-09-13T14:47:44.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:44.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.915313+0000", 2024-09-13T14:47:44.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108392 2024-09-13T14:47:44.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:47:44.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.915478+0000", 2024-09-13T14:47:44.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016415700000000001 2024-09-13T14:47:44.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:44.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.936883+0000", 2024-09-13T14:47:44.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021405061 2024-09-13T14:47:44.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.936918+0000", 2024-09-13T14:47:44.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5267e-05 2024-09-13T14:47:44.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.954585+0000", 2024-09-13T14:47:44.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017667157999999999 2024-09-13T14:47:44.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:44.395 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.954625+0000", 2024-09-13T14:47:44.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9453000000000003e-05 2024-09-13T14:47:44.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:44.395 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:07.956896+0000", 2024-09-13T14:47:44.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0022712990000000001 2024-09-13T14:47:44.395 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.395 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:44.395 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.396 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:207761 6.b 6:d0beb7b7:::benchmark_data_smithi151_155836_object75303:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1118)", 2024-09-13T14:47:44.396 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:42:22.393179+0000", 2024-09-13T14:47:44.396 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 321.953027604, 2024-09-13T14:47:44.396 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.584697884, 2024-09-13T14:47:44.396 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:44.396 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:44.396 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:44.396 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:44.396 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:47:44.396 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:47:44.396 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 207761 2024-09-13T14:47:44.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.396 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:44.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:44.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.393179+0000", 2024-09-13T14:47:44.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:44.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.393179+0000", 2024-09-13T14:47:44.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:44.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.393181+0000", 2024-09-13T14:47:44.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6959999999999999e-06 2024-09-13T14:47:44.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:44.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.393188+0000", 2024-09-13T14:47:44.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8229999999999997e-06 2024-09-13T14:47:44.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:44.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.393204+0000", 2024-09-13T14:47:44.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5905000000000001e-05 2024-09-13T14:47:44.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:44.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.393214+0000", 2024-09-13T14:47:44.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0278000000000001e-05 2024-09-13T14:47:44.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.409428+0000", 2024-09-13T14:47:44.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016213676 2024-09-13T14:47:44.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:47:44.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.409553+0000", 2024-09-13T14:47:44.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000124731 2024-09-13T14:47:44.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.904787+0000", 2024-09-13T14:47:44.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.49523448199999998 2024-09-13T14:47:44.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:44.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.904875+0000", 2024-09-13T14:47:44.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7701000000000006e-05 2024-09-13T14:47:44.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:47:44.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.904973+0000", 2024-09-13T14:47:44.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8047000000000005e-05 2024-09-13T14:47:44.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:44.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.926815+0000", 2024-09-13T14:47:44.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021841843 2024-09-13T14:47:44.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.926877+0000", 2024-09-13T14:47:44.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2391000000000006e-05 2024-09-13T14:47:44.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.975715+0000", 2024-09-13T14:47:44.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048838006000000003 2024-09-13T14:47:44.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:44.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.975805+0000", 2024-09-13T14:47:44.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9659000000000003e-05 2024-09-13T14:47:44.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:44.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:22.977877+0000", 2024-09-13T14:47:44.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002072646 2024-09-13T14:47:44.402 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.402 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:44.402 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.403 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:209001 6.b 6:d18fd3e1:::benchmark_data_smithi151_155836_object76543:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1118)", 2024-09-13T14:47:44.403 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:42:23.590318+0000", 2024-09-13T14:47:44.403 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 320.75588920600001, 2024-09-13T14:47:44.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.54397842500000004, 2024-09-13T14:47:44.403 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:44.403 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:44.403 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:44.403 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:44.403 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:47:44.403 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:47:44.403 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 209001 2024-09-13T14:47:44.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.403 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:44.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:44.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:23.590318+0000", 2024-09-13T14:47:44.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:44.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:23.590318+0000", 2024-09-13T14:47:44.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:44.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:23.590320+0000", 2024-09-13T14:47:44.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.928e-06 2024-09-13T14:47:44.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:44.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:23.590326+0000", 2024-09-13T14:47:44.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9970000000000004e-06 2024-09-13T14:47:44.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:44.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:23.590343+0000", 2024-09-13T14:47:44.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6920000000000001e-05 2024-09-13T14:47:44.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.406 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:44.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:23.590351+0000", 2024-09-13T14:47:44.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.816e-06 2024-09-13T14:47:44.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.406 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:23.608401+0000", 2024-09-13T14:47:44.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01804944 2024-09-13T14:47:44.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.406 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:47:44.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:23.608508+0000", 2024-09-13T14:47:44.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000106989 2024-09-13T14:47:44.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:24.109068+0000", 2024-09-13T14:47:44.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.50055971099999996 2024-09-13T14:47:44.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:44.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:24.109189+0000", 2024-09-13T14:47:44.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012140499999999999 2024-09-13T14:47:44.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:47:44.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:24.109365+0000", 2024-09-13T14:47:44.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017573400000000001 2024-09-13T14:47:44.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:24.125803+0000", 2024-09-13T14:47:44.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016438090999999998 2024-09-13T14:47:44.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:44.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:24.125983+0000", 2024-09-13T14:47:44.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018028799999999999 2024-09-13T14:47:44.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:24.132081+0000", 2024-09-13T14:47:44.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0060981000000000004 2024-09-13T14:47:44.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:44.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:24.132141+0000", 2024-09-13T14:47:44.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9317000000000001e-05 2024-09-13T14:47:44.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:44.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:24.134296+0000", 2024-09-13T14:47:44.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021556890000000001 2024-09-13T14:47:44.409 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.409 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:44.410 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.410 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:210473 6.b 6:dd7eb092:::benchmark_data_smithi151_155836_object78015:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1118)", 2024-09-13T14:47:44.410 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:42:24.990992+0000", 2024-09-13T14:47:44.410 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 319.355214692, 2024-09-13T14:47:44.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37692983699999999, 2024-09-13T14:47:44.410 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:44.410 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:44.410 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:44.410 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:44.410 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:47:44.410 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:47:44.411 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 210473 2024-09-13T14:47:44.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.411 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:44.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:44.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:24.990992+0000", 2024-09-13T14:47:44.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:44.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:24.990992+0000", 2024-09-13T14:47:44.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:44.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:24.990994+0000", 2024-09-13T14:47:44.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.55e-06 2024-09-13T14:47:44.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:44.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:24.991000+0000", 2024-09-13T14:47:44.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7270000000000004e-06 2024-09-13T14:47:44.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:44.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:24.991014+0000", 2024-09-13T14:47:44.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4643e-05 2024-09-13T14:47:44.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.413 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:44.413 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:24.991023+0000", 2024-09-13T14:47:44.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3990000000000004e-06 2024-09-13T14:47:44.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.413 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.413 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:25.004370+0000", 2024-09-13T14:47:44.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01334752 2024-09-13T14:47:44.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.413 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:47:44.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:25.004463+0000", 2024-09-13T14:47:44.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2398000000000005e-05 2024-09-13T14:47:44.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:25.319879+0000", 2024-09-13T14:47:44.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31541669500000002 2024-09-13T14:47:44.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:44.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:25.319979+0000", 2024-09-13T14:47:44.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9910999999999999e-05 2024-09-13T14:47:44.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:47:44.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:25.320133+0000", 2024-09-13T14:47:44.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015411 2024-09-13T14:47:44.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:25.343091+0000", 2024-09-13T14:47:44.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.022957898000000001 2024-09-13T14:47:44.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:44.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:25.343298+0000", 2024-09-13T14:47:44.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00020694600000000001 2024-09-13T14:47:44.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:25.365334+0000", 2024-09-13T14:47:44.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.022036152 2024-09-13T14:47:44.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:44.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:25.365385+0000", 2024-09-13T14:47:44.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0351e-05 2024-09-13T14:47:44.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:44.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:25.367922+0000", 2024-09-13T14:47:44.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0025375369999999999 2024-09-13T14:47:44.417 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.417 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:44.417 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.417 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:213990 6.b 6:d3efb36d:::benchmark_data_smithi151_155836_object81532:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1118)", 2024-09-13T14:47:44.417 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:42:28.482119+0000", 2024-09-13T14:47:44.417 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 315.86408755000002, 2024-09-13T14:47:44.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.43243266600000002, 2024-09-13T14:47:44.417 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:44.418 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:44.418 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:44.418 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:44.418 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:47:44.418 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:47:44.418 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 213990 2024-09-13T14:47:44.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.418 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:44.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:44.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.482119+0000", 2024-09-13T14:47:44.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:44.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.482119+0000", 2024-09-13T14:47:44.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:44.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.482121+0000", 2024-09-13T14:47:44.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.725e-06 2024-09-13T14:47:44.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:44.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.482128+0000", 2024-09-13T14:47:44.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3799999999999999e-06 2024-09-13T14:47:44.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:44.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.482152+0000", 2024-09-13T14:47:44.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4417e-05 2024-09-13T14:47:44.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:44.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.482161+0000", 2024-09-13T14:47:44.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2809999999999997e-06 2024-09-13T14:47:44.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.498739+0000", 2024-09-13T14:47:44.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016577273999999999 2024-09-13T14:47:44.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:47:44.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.498872+0000", 2024-09-13T14:47:44.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000133153 2024-09-13T14:47:44.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.833103+0000", 2024-09-13T14:47:44.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.334231428 2024-09-13T14:47:44.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:44.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.833172+0000", 2024-09-13T14:47:44.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9220000000000005e-05 2024-09-13T14:47:44.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:47:44.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.833291+0000", 2024-09-13T14:47:44.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118146 2024-09-13T14:47:44.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:44.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.843364+0000", 2024-09-13T14:47:44.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010073716 2024-09-13T14:47:44.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.871140+0000", 2024-09-13T14:47:44.423 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:44.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.027776085999999998 2024-09-13T14:47:44.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.912981+0000", 2024-09-13T14:47:44.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041840255999999999 2024-09-13T14:47:44.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:44.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.913056+0000", 2024-09-13T14:47:44.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5321000000000006e-05 2024-09-13T14:47:44.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:44.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:28.914552+0000", 2024-09-13T14:47:44.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0014962630000000001 2024-09-13T14:47:44.425 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.425 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:44.425 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.426 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:234885 6.b 6:df1bf793:::benchmark_data_smithi151_155836_object102427:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1123)", 2024-09-13T14:47:44.426 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:42:49.694861+0000", 2024-09-13T14:47:44.426 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 294.651345612, 2024-09-13T14:47:44.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37100666999999998, 2024-09-13T14:47:44.426 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:44.426 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:44.426 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:44.426 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:44.426 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:47:44.426 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:47:44.426 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 234885 2024-09-13T14:47:44.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.427 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:44.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.427 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:44.427 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:49.694861+0000", 2024-09-13T14:47:44.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.427 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:44.427 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:49.694861+0000", 2024-09-13T14:47:44.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.427 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:44.427 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:49.694863+0000", 2024-09-13T14:47:44.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3629999999999999e-06 2024-09-13T14:47:44.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.428 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:44.428 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:49.694868+0000", 2024-09-13T14:47:44.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.57e-06 2024-09-13T14:47:44.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.428 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:44.428 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:49.694880+0000", 2024-09-13T14:47:44.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1952e-05 2024-09-13T14:47:44.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:44.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:49.694886+0000", 2024-09-13T14:47:44.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8259999999999998e-06 2024-09-13T14:47:44.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:49.694941+0000", 2024-09-13T14:47:44.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5299e-05 2024-09-13T14:47:44.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:47:44.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:49.695070+0000", 2024-09-13T14:47:44.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012866099999999999 2024-09-13T14:47:44.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:49.991850+0000", 2024-09-13T14:47:44.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29677958799999998 2024-09-13T14:47:44.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:44.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:49.991929+0000", 2024-09-13T14:47:44.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9664000000000002e-05 2024-09-13T14:47:44.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:47:44.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:49.992047+0000", 2024-09-13T14:47:44.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117774 2024-09-13T14:47:44.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.431 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.006509+0000", 2024-09-13T14:47:44.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014462226 2024-09-13T14:47:44.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:44.431 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.006524+0000", 2024-09-13T14:47:44.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4389000000000001e-05 2024-09-13T14:47:44.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.431 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.063605+0000", 2024-09-13T14:47:44.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.057081112000000003 2024-09-13T14:47:44.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:44.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.063648+0000", 2024-09-13T14:47:44.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3260999999999998e-05 2024-09-13T14:47:44.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:44.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.065868+0000", 2024-09-13T14:47:44.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0022199849999999998 2024-09-13T14:47:44.432 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.432 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:44.432 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.433 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:235375 6.b 6:d73ddd5f:::benchmark_data_smithi151_155836_object102917:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1123)", 2024-09-13T14:47:44.433 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:42:50.193168+0000", 2024-09-13T14:47:44.433 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 294.15303891799999, 2024-09-13T14:47:44.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.51423233800000001, 2024-09-13T14:47:44.433 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:44.433 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:44.433 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:44.433 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:44.433 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:47:44.433 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:47:44.433 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 235375 2024-09-13T14:47:44.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.434 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:44.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:44.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.193168+0000", 2024-09-13T14:47:44.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:44.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.193168+0000", 2024-09-13T14:47:44.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:44.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.193170+0000", 2024-09-13T14:47:44.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5740000000000001e-06 2024-09-13T14:47:44.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.435 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:44.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.193175+0000", 2024-09-13T14:47:44.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6439999999999997e-06 2024-09-13T14:47:44.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.435 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:44.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.193188+0000", 2024-09-13T14:47:44.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2225999999999999e-05 2024-09-13T14:47:44.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.435 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:44.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.193196+0000", 2024-09-13T14:47:44.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6500000000000002e-06 2024-09-13T14:47:44.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.436 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.213195+0000", 2024-09-13T14:47:44.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019998616 2024-09-13T14:47:44.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.436 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:47:44.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.213331+0000", 2024-09-13T14:47:44.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013614000000000001 2024-09-13T14:47:44.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.658507+0000", 2024-09-13T14:47:44.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.44517631200000002 2024-09-13T14:47:44.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:44.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.658619+0000", 2024-09-13T14:47:44.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011189899999999999 2024-09-13T14:47:44.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:47:44.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.658779+0000", 2024-09-13T14:47:44.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015954900000000001 2024-09-13T14:47:44.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.677188+0000", 2024-09-13T14:47:44.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018409132000000002 2024-09-13T14:47:44.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:44.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.677210+0000", 2024-09-13T14:47:44.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1832999999999999e-05 2024-09-13T14:47:44.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.705544+0000", 2024-09-13T14:47:44.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.028333816000000001 2024-09-13T14:47:44.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:44.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.705601+0000", 2024-09-13T14:47:44.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7379000000000001e-05 2024-09-13T14:47:44.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:44.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:50.707400+0000", 2024-09-13T14:47:44.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017995680000000001 2024-09-13T14:47:44.440 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.440 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:44.440 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.440 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:236769 6.b 6:d4580e47:::benchmark_data_smithi151_155836_object104311:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1123)", 2024-09-13T14:47:44.440 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:42:51.576789+0000", 2024-09-13T14:47:44.440 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 292.76941769500002, 2024-09-13T14:47:44.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.48812771999999999, 2024-09-13T14:47:44.440 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:44.440 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:44.440 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:44.440 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:44.441 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:47:44.441 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:47:44.441 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 236769 2024-09-13T14:47:44.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.441 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:44.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:44.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:51.576789+0000", 2024-09-13T14:47:44.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:44.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:51.576789+0000", 2024-09-13T14:47:44.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:44.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:51.576798+0000", 2024-09-13T14:47:44.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0340000000000002e-06 2024-09-13T14:47:44.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:44.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:51.576803+0000", 2024-09-13T14:47:44.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.656e-06 2024-09-13T14:47:44.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:44.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:51.576816+0000", 2024-09-13T14:47:44.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3448999999999999e-05 2024-09-13T14:47:44.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:44.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:51.576823+0000", 2024-09-13T14:47:44.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8140000000000003e-06 2024-09-13T14:47:44.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:51.585868+0000", 2024-09-13T14:47:44.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0090445979999999992 2024-09-13T14:47:44.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:47:44.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:51.585933+0000", 2024-09-13T14:47:44.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5246000000000005e-05 2024-09-13T14:47:44.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:52.033369+0000", 2024-09-13T14:47:44.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.44743548599999999 2024-09-13T14:47:44.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:44.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:52.033475+0000", 2024-09-13T14:47:44.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000106031 2024-09-13T14:47:44.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:47:44.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:52.033634+0000", 2024-09-13T14:47:44.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000159474 2024-09-13T14:47:44.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:52.049326+0000", 2024-09-13T14:47:44.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015692053000000001 2024-09-13T14:47:44.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:44.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:52.049360+0000", 2024-09-13T14:47:44.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3855e-05 2024-09-13T14:47:44.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:52.063272+0000", 2024-09-13T14:47:44.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013912203 2024-09-13T14:47:44.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:44.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:52.063313+0000", 2024-09-13T14:47:44.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0306999999999999e-05 2024-09-13T14:47:44.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:44.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:52.064917+0000", 2024-09-13T14:47:44.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001604514 2024-09-13T14:47:44.447 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.447 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:44.447 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.447 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:240044 6.b 6:ddd6b124:::benchmark_data_smithi151_155836_object107586:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1123)", 2024-09-13T14:47:44.447 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:42:54.772464+0000", 2024-09-13T14:47:44.447 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 289.57374334600001, 2024-09-13T14:47:44.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.78119298299999995, 2024-09-13T14:47:44.448 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:44.448 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:44.448 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:44.448 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:44.448 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:47:44.448 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:47:44.448 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 240044 2024-09-13T14:47:44.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.448 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:44.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:44.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:54.772464+0000", 2024-09-13T14:47:44.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:44.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:54.772464+0000", 2024-09-13T14:47:44.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:44.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:54.772465+0000", 2024-09-13T14:47:44.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3030000000000001e-06 2024-09-13T14:47:44.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:44.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:54.772470+0000", 2024-09-13T14:47:44.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0969999999999996e-06 2024-09-13T14:47:44.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:44.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:54.772494+0000", 2024-09-13T14:47:44.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4165999999999999e-05 2024-09-13T14:47:44.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:44.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:54.772503+0000", 2024-09-13T14:47:44.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2310000000000002e-06 2024-09-13T14:47:44.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:54.785159+0000", 2024-09-13T14:47:44.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012655614000000001 2024-09-13T14:47:44.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:47:44.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:54.785278+0000", 2024-09-13T14:47:44.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011937700000000001 2024-09-13T14:47:44.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:55.494601+0000", 2024-09-13T14:47:44.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.70932297600000005 2024-09-13T14:47:44.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:44.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:55.494702+0000", 2024-09-13T14:47:44.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000100374 2024-09-13T14:47:44.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:47:44.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:55.494970+0000", 2024-09-13T14:47:44.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00026814299999999998 2024-09-13T14:47:44.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:55.510832+0000", 2024-09-13T14:47:44.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015861896 2024-09-13T14:47:44.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:44.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:55.511008+0000", 2024-09-13T14:47:44.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017613199999999999 2024-09-13T14:47:44.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:55.551768+0000", 2024-09-13T14:47:44.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.040759660000000003 2024-09-13T14:47:44.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:44.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:55.551814+0000", 2024-09-13T14:47:44.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6601999999999999e-05 2024-09-13T14:47:44.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:44.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:55.553657+0000", 2024-09-13T14:47:44.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018424120000000001 2024-09-13T14:47:44.454 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.454 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:44.455 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.455 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:243296 6.b 6:da27bd7d:::benchmark_data_smithi151_155836_object110838:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1123)", 2024-09-13T14:47:44.455 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:42:58.035785+0000", 2024-09-13T14:47:44.455 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 286.31042244000002, 2024-09-13T14:47:44.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.42214216399999999, 2024-09-13T14:47:44.455 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:44.455 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:44.455 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:44.455 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:44.455 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:47:44.455 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:47:44.455 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 243296 2024-09-13T14:47:44.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.456 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:44.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:44.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.035785+0000", 2024-09-13T14:47:44.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:44.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.035785+0000", 2024-09-13T14:47:44.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:44.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.035786+0000", 2024-09-13T14:47:44.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.863e-06 2024-09-13T14:47:44.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:44.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.035802+0000", 2024-09-13T14:47:44.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5777000000000002e-05 2024-09-13T14:47:44.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:44.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.035820+0000", 2024-09-13T14:47:44.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8002000000000001e-05 2024-09-13T14:47:44.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:44.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.035829+0000", 2024-09-13T14:47:44.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7199999999999995e-06 2024-09-13T14:47:44.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.035902+0000", 2024-09-13T14:47:44.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3250000000000005e-05 2024-09-13T14:47:44.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:47:44.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.036031+0000", 2024-09-13T14:47:44.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012864800000000001 2024-09-13T14:47:44.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.414050+0000", 2024-09-13T14:47:44.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37801879799999999 2024-09-13T14:47:44.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:44.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.414136+0000", 2024-09-13T14:47:44.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6286999999999996e-05 2024-09-13T14:47:44.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:47:44.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.414281+0000", 2024-09-13T14:47:44.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014502100000000001 2024-09-13T14:47:44.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.430721+0000", 2024-09-13T14:47:44.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016440132 2024-09-13T14:47:44.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:44.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.430746+0000", 2024-09-13T14:47:44.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5298000000000001e-05 2024-09-13T14:47:44.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.461 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.456137+0000", 2024-09-13T14:47:44.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.025390942 2024-09-13T14:47:44.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:44.461 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.456208+0000", 2024-09-13T14:47:44.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0693999999999998e-05 2024-09-13T14:47:44.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:44.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:42:58.457927+0000", 2024-09-13T14:47:44.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017187319999999999 2024-09-13T14:47:44.462 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.462 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:44.462 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.462 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:246042 6.b 6:d9559628:::benchmark_data_smithi151_155836_object113584:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1123)", 2024-09-13T14:47:44.462 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:00.744657+0000", 2024-09-13T14:47:44.462 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 283.601549643, 2024-09-13T14:47:44.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.44821939, 2024-09-13T14:47:44.462 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:44.462 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:44.462 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:44.463 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:44.463 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:47:44.463 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:47:44.463 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 246042 2024-09-13T14:47:44.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.463 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:44.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:44.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:00.744657+0000", 2024-09-13T14:47:44.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:44.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:00.744657+0000", 2024-09-13T14:47:44.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:44.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:00.744659+0000", 2024-09-13T14:47:44.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7099999999999999e-06 2024-09-13T14:47:44.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:44.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:00.744667+0000", 2024-09-13T14:47:44.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6059999999999997e-06 2024-09-13T14:47:44.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:44.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:00.744685+0000", 2024-09-13T14:47:44.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8692999999999999e-05 2024-09-13T14:47:44.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:44.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:00.744694+0000", 2024-09-13T14:47:44.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0450000000000006e-06 2024-09-13T14:47:44.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:00.751063+0000", 2024-09-13T14:47:44.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.006368851 2024-09-13T14:47:44.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:47:44.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:00.751190+0000", 2024-09-13T14:47:44.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126785 2024-09-13T14:47:44.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.161878+0000", 2024-09-13T14:47:44.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.41068771199999998 2024-09-13T14:47:44.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:44.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.161945+0000", 2024-09-13T14:47:44.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7687000000000004e-05 2024-09-13T14:47:44.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:47:44.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.162037+0000", 2024-09-13T14:47:44.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1244999999999998e-05 2024-09-13T14:47:44.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.176824+0000", 2024-09-13T14:47:44.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014787719 2024-09-13T14:47:44.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:44.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.176841+0000", 2024-09-13T14:47:44.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6333000000000001e-05 2024-09-13T14:47:44.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.191064+0000", 2024-09-13T14:47:44.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014222894999999999 2024-09-13T14:47:44.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:44.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.191140+0000", 2024-09-13T14:47:44.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6179000000000004e-05 2024-09-13T14:47:44.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:44.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.192877+0000", 2024-09-13T14:47:44.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00173693 2024-09-13T14:47:44.470 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.470 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:44.470 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.470 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:247199 6.b 6:da84a270:::benchmark_data_smithi151_155836_object114741:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1123)", 2024-09-13T14:47:44.470 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:01.891288+0000", 2024-09-13T14:47:44.470 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 282.45491875499999, 2024-09-13T14:47:44.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.47403641499999999, 2024-09-13T14:47:44.470 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:44.470 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:44.470 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:44.470 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:47:44.470 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:47:44.471 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:47:44.471 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 247199 2024-09-13T14:47:44.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.471 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:44.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:44.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.891288+0000", 2024-09-13T14:47:44.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:44.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.891288+0000", 2024-09-13T14:47:44.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:44.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.891290+0000", 2024-09-13T14:47:44.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4649999999999999e-06 2024-09-13T14:47:44.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:44.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.891295+0000", 2024-09-13T14:47:44.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5709999999999997e-06 2024-09-13T14:47:44.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:44.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.891309+0000", 2024-09-13T14:47:44.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3893000000000001e-05 2024-09-13T14:47:44.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:44.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.891316+0000", 2024-09-13T14:47:44.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8909999999999998e-06 2024-09-13T14:47:44.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.901646+0000", 2024-09-13T14:47:44.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010329589 2024-09-13T14:47:44.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-13T14:47:44.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:01.901712+0000", 2024-09-13T14:47:44.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6644999999999996e-05 2024-09-13T14:47:44.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.474 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:02.327995+0000", 2024-09-13T14:47:44.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.426282457 2024-09-13T14:47:44.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:44.474 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:02.328102+0000", 2024-09-13T14:47:44.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010686 2024-09-13T14:47:44.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:47:44.474 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:02.328263+0000", 2024-09-13T14:47:44.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000161686 2024-09-13T14:47:44.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.475 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.475 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:02.346868+0000", 2024-09-13T14:47:44.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018605047999999999 2024-09-13T14:47:44.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.475 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:47:44.475 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:02.346878+0000", 2024-09-13T14:47:44.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6339999999999996e-06 2024-09-13T14:47:44.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.475 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:44.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:02.363734+0000", 2024-09-13T14:47:44.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016856428 2024-09-13T14:47:44.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:44.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:02.363810+0000", 2024-09-13T14:47:44.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5617000000000005e-05 2024-09-13T14:47:44.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:44.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:02.365325+0000", 2024-09-13T14:47:44.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001514631 2024-09-13T14:47:44.476 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.476 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:44.477 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.477 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.45494.0:198469 7.a e1160/1149 7:5f7e472e:::benchmark_data_smithi151_190906_object78008:head v 1160'12311, mlcod=0'0)", 2024-09-13T14:47:44.477 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:47:40.136348+0000", 2024-09-13T14:47:44.477 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.2098591670000003, 2024-09-13T14:47:44.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37852921299999998, 2024-09-13T14:47:44.477 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:44.477 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:44.477 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:44.477 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:44.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:44.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.136348+0000", 2024-09-13T14:47:44.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:44.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.136348+0000", 2024-09-13T14:47:44.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:44.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.136349+0000", 2024-09-13T14:47:44.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9399999999999993e-07 2024-09-13T14:47:44.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:44.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.136355+0000", 2024-09-13T14:47:44.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3489999999999997e-06 2024-09-13T14:47:44.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:44.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.136367+0000", 2024-09-13T14:47:44.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1814000000000001e-05 2024-09-13T14:47:44.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:44.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.136375+0000", 2024-09-13T14:47:44.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3760000000000008e-06 2024-09-13T14:47:44.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.140003+0000", 2024-09-13T14:47:44.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003627805 2024-09-13T14:47:44.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:44.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.140055+0000", 2024-09-13T14:47:44.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2086999999999998e-05 2024-09-13T14:47:44.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:44.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.514752+0000", 2024-09-13T14:47:44.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37469675099999999 2024-09-13T14:47:44.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:44.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.514877+0000", 2024-09-13T14:47:44.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012503699999999999 2024-09-13T14:47:44.481 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.481 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:44.481 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.482 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.45494.0:198480 7.a e1160/1149 7:57c00ec3:::benchmark_data_smithi151_190906_object78019:head v 1160'12312, mlcod=0'0)", 2024-09-13T14:47:44.482 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:47:40.147822+0000", 2024-09-13T14:47:44.482 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.1983847579999996, 2024-09-13T14:47:44.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.36709185599999999, 2024-09-13T14:47:44.482 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:44.482 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:44.482 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:44.482 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:44.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:44.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.147822+0000", 2024-09-13T14:47:44.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:44.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.147822+0000", 2024-09-13T14:47:44.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:44.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.147824+0000", 2024-09-13T14:47:44.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4219999999999999e-06 2024-09-13T14:47:44.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:44.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.147833+0000", 2024-09-13T14:47:44.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8899999999999996e-06 2024-09-13T14:47:44.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:44.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.147845+0000", 2024-09-13T14:47:44.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2316e-05 2024-09-13T14:47:44.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:44.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.147853+0000", 2024-09-13T14:47:44.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5369999999999997e-06 2024-09-13T14:47:44.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.151316+0000", 2024-09-13T14:47:44.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034624880000000001 2024-09-13T14:47:44.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:44.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.151337+0000", 2024-09-13T14:47:44.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1322999999999999e-05 2024-09-13T14:47:44.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:44.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.514883+0000", 2024-09-13T14:47:44.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.36354525900000001 2024-09-13T14:47:44.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:44.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.514914+0000", 2024-09-13T14:47:44.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1621000000000001e-05 2024-09-13T14:47:44.486 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.486 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:44.486 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.486 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.45494.0:198489 7.a e1160/1149 7:5f5f3bb7:::benchmark_data_smithi151_190906_object78028:head v 1160'12313, mlcod=0'0)", 2024-09-13T14:47:44.486 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:47:40.151631+0000", 2024-09-13T14:47:44.486 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.1945761179999996, 2024-09-13T14:47:44.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.39077605700000001, 2024-09-13T14:47:44.486 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:47:44.487 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:47:44.487 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:44.487 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:47:44.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:47:44.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.151631+0000", 2024-09-13T14:47:44.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:47:44.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.151631+0000", 2024-09-13T14:47:44.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:47:44.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:47:44.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.151632+0000", 2024-09-13T14:47:44.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2580000000000001e-06 2024-09-13T14:47:44.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:47:44.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.151640+0000", 2024-09-13T14:47:44.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7270000000000007e-06 2024-09-13T14:47:44.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:47:44.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.151652+0000", 2024-09-13T14:47:44.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1874e-05 2024-09-13T14:47:44.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:47:44.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.151659+0000", 2024-09-13T14:47:44.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4710000000000001e-06 2024-09-13T14:47:44.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:47:44.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.504655+0000", 2024-09-13T14:47:44.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35299582000000002 2024-09-13T14:47:44.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:47:44.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.504684+0000", 2024-09-13T14:47:44.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.847e-05 2024-09-13T14:47:44.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:47:44.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.542356+0000", 2024-09-13T14:47:44.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.037672237999999997 2024-09-13T14:47:44.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:47:44.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:47:44.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:47:44.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.542407+0000", 2024-09-13T14:47:44.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1199000000000002e-05 2024-09-13T14:47:44.490 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.490 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:44.490 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.490 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:47:44.491 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:47:44.491 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:47:44.506 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:44.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:44.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:44.608 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:44.609 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:47:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:47:44] "GET /metrics HTTP/1.1" 200 36659 "" "Prometheus/2.51.0" 2024-09-13T14:47:44.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:44.658 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:44.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:44.759 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:44.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:44.816 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:44.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:44.916 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:44.962 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:44.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:44.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:45.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:45.077 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:44 smithi151 ceph-mon[104450]: pgmap v1616: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 556 op/s; 3038/125526 objects degraded (2.420%); 27654/125526 objects misplaced (22.030%); 934 KiB/s, 14 objects/s recovering 2024-09-13T14:47:45.079 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:44 smithi151 ceph-mon[108966]: pgmap v1616: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 556 op/s; 3038/125526 objects degraded (2.420%); 27654/125526 objects misplaced (22.030%); 934 KiB/s, 14 objects/s recovering 2024-09-13T14:47:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:44 smithi184 ceph-mon[92527]: pgmap v1616: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 556 op/s; 3038/125526 objects degraded (2.420%); 27654/125526 objects misplaced (22.030%); 934 KiB/s, 14 objects/s recovering 2024-09-13T14:47:45.107 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:45.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:45.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:45.209 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:45.259 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:45.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:45.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:45.361 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:45.395 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:45.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:45.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:45.497 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:45.520 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:45.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:45.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:45.622 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:45.646 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:45.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:45.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:45.748 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:45.797 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:45.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:45.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:45.901 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:45.907 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:45 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 3038/125526 objects degraded (2.420%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:45.907 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:45 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 3038/125526 objects degraded (2.420%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:45.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:45.942 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:45.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:46.004 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:45 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 3038/125526 objects degraded (2.420%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:46.043 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:46.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:46.070 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:46.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:46.095 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:47:46.170 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:46.196 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:46.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:46.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:46.298 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:46.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:46.348 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:46.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:46.449 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:46.477 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:46.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:46.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:46.576 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T14:47:46.579 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:46.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:46.625 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:46.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:46.726 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:46.754 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:46.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:46.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:46.844 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:46 smithi151 ceph-mon[104450]: pgmap v1617: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 544 op/s; 3038/122925 objects degraded (2.471%); 27137/122925 objects misplaced (22.076%); 934 KiB/s, 14 objects/s recovering 2024-09-13T14:47:46.845 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:46.845 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:46 smithi151 ceph-mon[108966]: pgmap v1617: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 544 op/s; 3038/122925 objects degraded (2.471%); 27137/122925 objects misplaced (22.076%); 934 KiB/s, 14 objects/s recovering 2024-09-13T14:47:46.845 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:46.857 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:46.906 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:47:46.907 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:46.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:46.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:47.009 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:47.035 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:47.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:47.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:47.039 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:46 smithi184 ceph-mon[92527]: pgmap v1617: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 544 op/s; 3038/122925 objects degraded (2.471%); 27137/122925 objects misplaced (22.076%); 934 KiB/s, 14 objects/s recovering 2024-09-13T14:47:47.040 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:47.111 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:47.137 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:47.163 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:47.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:47.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:47.265 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:47.300 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:47.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:47.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:47.402 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:47.438 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:47.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:47.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:47.540 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:47.578 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:47.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:47.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:47.680 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:47.703 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:47.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:47.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:47.805 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:47.827 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:47.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:47.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:47.929 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:47.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:47.993 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:47.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:48.094 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:48.141 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:48.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:48.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:48.243 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:48.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:48.297 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:48.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:48.397 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:48.440 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:48.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:48.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:48.542 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:48.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:48.568 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:48.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:48.669 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:48.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:48.707 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:48.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:48.808 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:48.850 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:48.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:48.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:48.918 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:48 smithi184 ceph-mon[92527]: pgmap v1618: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 557 op/s; 3137/120549 objects degraded (2.602%); 26749/120549 objects misplaced (22.189%); 966 KiB/s, 15 objects/s recovering 2024-09-13T14:47:48.931 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:47:48.931 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [ 2024-09-13T14:47:48.931 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.931 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.45494.0:203700 7.4 e1161/1136 7:21936707:::benchmark_data_smithi151_190906_object83239:head v 1161'12755, mlcod=1161'12752)", 2024-09-13T14:47:48.931 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:47:48.925127+0000", 2024-09-13T14:47:48.932 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.0045662489999999997, 2024-09-13T14:47:48.932 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0045930490000000001, 2024-09-13T14:47:48.932 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:47:48.932 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:47:48.932 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:47:48.932 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.932 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:47:48.932 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.925127+0000", 2024-09-13T14:47:48.932 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:47:48.932 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.932 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.932 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:47:48.932 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.925127+0000", 2024-09-13T14:47:48.933 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:47:48.933 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.933 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.933 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:47:48.933 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.925126+0000", 2024-09-13T14:47:48.933 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:47:48.933 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.933 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.933 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:47:48.933 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.925132+0000", 2024-09-13T14:47:48.933 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5149999999999997e-06 2024-09-13T14:47:48.933 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.933 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.934 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:47:48.934 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.925141+0000", 2024-09-13T14:47:48.934 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.6250000000000002e-06 2024-09-13T14:47:48.934 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.934 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.934 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:47:48.934 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.925145+0000", 2024-09-13T14:47:48.934 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0169999999999998e-06 2024-09-13T14:47:48.934 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.934 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.934 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:47:48.934 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.925184+0000", 2024-09-13T14:47:48.934 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8556000000000002e-05 2024-09-13T14:47:48.935 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.935 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.935 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:47:48.935 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.925204+0000", 2024-09-13T14:47:48.935 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.0140999999999999e-05 2024-09-13T14:47:48.935 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:47:48.935 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:47:48.935 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:47:48.935 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.935 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.935 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.45494.0:203705 7.4 e1161/1136 7:24d7eb05:::benchmark_data_smithi151_190906_object83244:head v 1161'12756, mlcod=1161'12752)", 2024-09-13T14:47:48.935 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:47:48.926490+0000", 2024-09-13T14:47:48.935 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.0032033679999999998, 2024-09-13T14:47:48.935 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0032908009999999999, 2024-09-13T14:47:48.936 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:47:48.936 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:47:48.936 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:47:48.936 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.936 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:47:48.936 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.926490+0000", 2024-09-13T14:47:48.936 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:47:48.936 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.936 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.936 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:47:48.936 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.926490+0000", 2024-09-13T14:47:48.936 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:47:48.937 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.937 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.937 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:47:48.937 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.926489+0000", 2024-09-13T14:47:48.937 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:47:48.937 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.937 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.937 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:47:48.937 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.926493+0000", 2024-09-13T14:47:48.937 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8530000000000002e-06 2024-09-13T14:47:48.937 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.937 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.937 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:47:48.938 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.926500+0000", 2024-09-13T14:47:48.938 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.0740000000000004e-06 2024-09-13T14:47:48.938 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.938 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.938 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:47:48.938 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.926505+0000", 2024-09-13T14:47:48.938 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.6120000000000001e-06 2024-09-13T14:47:48.938 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.938 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.938 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:47:48.938 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.927668+0000", 2024-09-13T14:47:48.938 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001162799 2024-09-13T14:47:48.938 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.938 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.939 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:47:48.939 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.927690+0000", 2024-09-13T14:47:48.939 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.1875999999999999e-05 2024-09-13T14:47:48.939 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:47:48.939 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:47:48.939 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:47:48.939 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.939 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.939 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.45494.0:203696 7.3 e1161/1149 7:c5ff1551:::benchmark_data_smithi151_190906_object83235:head v 1161'12752, mlcod=1143'5207)", 2024-09-13T14:47:48.939 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:47:48.928484+0000", 2024-09-13T14:47:48.939 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.001208988, 2024-09-13T14:47:48.939 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001353183, 2024-09-13T14:47:48.939 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:47:48.940 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:47:48.940 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:47:48.940 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.940 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:47:48.940 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.928484+0000", 2024-09-13T14:47:48.940 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:47:48.940 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.940 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.940 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:47:48.940 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.928484+0000", 2024-09-13T14:47:48.940 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:47:48.940 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.940 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.940 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:47:48.941 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.928484+0000", 2024-09-13T14:47:48.941 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:47:48.941 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.941 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.941 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:47:48.941 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.928491+0000", 2024-09-13T14:47:48.941 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.7500000000000003e-06 2024-09-13T14:47:48.941 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.941 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.941 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:47:48.941 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.928503+0000", 2024-09-13T14:47:48.941 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1972e-05 2024-09-13T14:47:48.941 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.941 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.942 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:47:48.942 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.928510+0000", 2024-09-13T14:47:48.942 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.0950000000000001e-06 2024-09-13T14:47:48.942 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.942 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.942 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:47:48.942 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.928546+0000", 2024-09-13T14:47:48.942 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.6905e-05 2024-09-13T14:47:48.942 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.942 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.942 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:47:48.942 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.928570+0000", 2024-09-13T14:47:48.942 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.3093000000000001e-05 2024-09-13T14:47:48.942 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:47:48.942 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:47:48.943 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:47:48.943 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.943 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.943 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:203709 7.1 7:8fa5e6d4:::benchmark_data_smithi151_190906_object83248:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1161)", 2024-09-13T14:47:48.943 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:47:48.928988+0000", 2024-09-13T14:47:48.943 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.00070487000000000002, 2024-09-13T14:47:48.943 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00091321000000000002, 2024-09-13T14:47:48.943 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:47:48.943 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:47:48.943 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:47:48.943 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:47:48.943 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:47:48.943 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 203709 2024-09-13T14:47:48.943 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.944 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:47:48.944 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.944 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:47:48.944 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.928988+0000", 2024-09-13T14:47:48.944 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:47:48.944 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.944 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.944 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:47:48.944 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.928988+0000", 2024-09-13T14:47:48.944 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:47:48.944 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.944 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.944 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:47:48.944 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.928987+0000", 2024-09-13T14:47:48.945 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:47:48.945 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.945 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.945 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:47:48.945 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.928993+0000", 2024-09-13T14:47:48.945 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.643e-06 2024-09-13T14:47:48.945 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.945 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.945 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:47:48.945 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.929003+0000", 2024-09-13T14:47:48.945 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0407e-05 2024-09-13T14:47:48.945 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.945 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.945 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:47:48.946 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.929007+0000", 2024-09-13T14:47:48.946 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.2470000000000002e-06 2024-09-13T14:47:48.946 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.946 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.946 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:47:48.946 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.929049+0000", 2024-09-13T14:47:48.946 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1270000000000003e-05 2024-09-13T14:47:48.946 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.946 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.946 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:47:48.946 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.929186+0000", 2024-09-13T14:47:48.946 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013694299999999999 2024-09-13T14:47:48.947 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.947 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.947 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:47:48.947 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.929326+0000", 2024-09-13T14:47:48.947 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014025499999999999 2024-09-13T14:47:48.947 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:47:48.947 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:47:48.947 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:47:48.947 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.947 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.947 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.45494.0:203710 7.f e1161/1149 7:f5fba63b:::benchmark_data_smithi151_190906_object83249:head v 1161'12786, mlcod=1161'12784)", 2024-09-13T14:47:48.947 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:47:48.929547+0000", 2024-09-13T14:47:48.947 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.00014576899999999999, 2024-09-13T14:47:48.948 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00042244400000000001, 2024-09-13T14:47:48.948 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:47:48.948 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:47:48.948 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:47:48.948 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.948 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:47:48.948 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.929547+0000", 2024-09-13T14:47:48.948 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:47:48.948 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.948 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.948 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:47:48.948 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.929547+0000", 2024-09-13T14:47:48.948 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:47:48.949 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.949 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.949 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:47:48.949 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.929547+0000", 2024-09-13T14:47:48.949 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:47:48.949 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.949 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.949 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:47:48.949 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.929553+0000", 2024-09-13T14:47:48.949 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.0530000000000003e-06 2024-09-13T14:47:48.949 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.949 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.949 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:47:48.950 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.929565+0000", 2024-09-13T14:47:48.950 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1813e-05 2024-09-13T14:47:48.950 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.950 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.950 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:47:48.950 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.929571+0000", 2024-09-13T14:47:48.950 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.0109999999999997e-06 2024-09-13T14:47:48.950 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.950 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.950 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:47:48.950 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.929614+0000", 2024-09-13T14:47:48.950 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3371999999999997e-05 2024-09-13T14:47:48.950 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:47:48.950 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:47:48.951 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:47:48.951 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:48.929637+0000", 2024-09-13T14:47:48.951 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.3181e-05 2024-09-13T14:47:48.951 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:47:48.951 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:47:48.951 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:47:48.951 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:47:48.951 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-13T14:47:48.951 INFO:teuthology.orchestra.run.smithi184.stdout: "num_ops": 5 2024-09-13T14:47:48.951 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:47:48.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:48.974 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:48.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:48.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:49.076 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:49.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:48 smithi151 ceph-mon[104450]: pgmap v1618: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 557 op/s; 3137/120549 objects degraded (2.602%); 26749/120549 objects misplaced (22.189%); 966 KiB/s, 15 objects/s recovering 2024-09-13T14:47:49.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:48 smithi151 ceph-mon[108966]: pgmap v1618: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 557 op/s; 3137/120549 objects degraded (2.602%); 26749/120549 objects misplaced (22.189%); 966 KiB/s, 15 objects/s recovering 2024-09-13T14:47:49.125 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:49.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:49.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:49.226 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:49.270 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:49.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:49.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:49.372 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:49.421 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:49.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:49.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:49.523 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:49.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:49.551 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:49.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:49.652 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:49.673 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:49.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:49.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:49.775 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:49.819 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:49.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:49.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:49.921 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:49.944 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:49.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:49.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:49.975 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T14:47:50.046 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:50.089 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:50.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:50.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:50.191 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:50.237 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:50.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:50.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:50.317 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:47:50.339 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:50.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:50.375 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:50.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:50.476 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:50.498 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:50.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:50.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:50.585 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:47:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:47:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:47:50.601 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:50.629 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:50.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:50.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:50.730 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:50.771 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:50.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:50.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:50.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:50 smithi184 ceph-mon[92527]: pgmap v1619: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.4 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 583 op/s; 3137/116298 objects degraded (2.697%); 26126/116298 objects misplaced (22.465%); 966 KiB/s, 15 objects/s recovering 2024-09-13T14:47:50.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:50 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 3137/116298 objects degraded (2.697%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:50.873 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:50.897 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:50.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:50.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:50.999 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:51.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:51.041 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:51.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:51.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:50 smithi151 ceph-mon[108966]: pgmap v1619: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.4 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 583 op/s; 3137/116298 objects degraded (2.697%); 26126/116298 objects misplaced (22.465%); 966 KiB/s, 15 objects/s recovering 2024-09-13T14:47:51.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:50 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 3137/116298 objects degraded (2.697%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:51.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:50 smithi151 ceph-mon[104450]: pgmap v1619: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.4 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 583 op/s; 3137/116298 objects degraded (2.697%); 26126/116298 objects misplaced (22.465%); 966 KiB/s, 15 objects/s recovering 2024-09-13T14:47:51.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:50 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 3137/116298 objects degraded (2.697%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:51.142 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:51.167 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:51.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:51.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:51.269 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:51.292 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:51.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:51.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:51.394 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:51.439 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:51.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:51.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:51.541 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:51.598 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:51.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:51.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:51.700 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:51.724 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:51.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:51.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:51.825 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:51.847 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:51.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:51.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:51.949 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:51.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:51.991 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:51.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:52.092 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:52.118 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:52.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:52.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:52.179 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:52.220 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:52.244 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:52.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:52.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:52.257 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:47:52.257 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:47:52.258 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:47:52.258 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:47:52.258 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:47:52.258 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:47:52.258 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:47:52.259 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:47:52.259 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:47:52.259 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:47:52.259 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:47:52.259 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:47:52.259 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:47:52.260 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:47:52.287 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:47:52.287 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:47:52.345 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:52.391 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:52.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:52.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:52.493 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:52.514 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:52.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:52.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:52.615 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:52.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:52.641 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:52.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:52.667 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:47:52.667 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:47:52.742 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:52.766 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:52.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:52.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:52.869 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:52.915 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:52.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:52.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:53.017 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:53.039 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:52 smithi184 ceph-mon[92527]: pgmap v1620: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.3 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 548 op/s; 3137/113445 objects degraded (2.765%); 25566/113445 objects misplaced (22.536%); 885 KiB/s, 13 objects/s recovering 2024-09-13T14:47:53.039 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:52 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3183367398' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:53.053 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:53.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:53.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:53.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:52 smithi151 ceph-mon[108966]: pgmap v1620: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.3 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 548 op/s; 3137/113445 objects degraded (2.765%); 25566/113445 objects misplaced (22.536%); 885 KiB/s, 13 objects/s recovering 2024-09-13T14:47:53.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:52 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3183367398' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:53.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:52 smithi151 ceph-mon[104450]: pgmap v1620: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.3 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 548 op/s; 3137/113445 objects degraded (2.765%); 25566/113445 objects misplaced (22.536%); 885 KiB/s, 13 objects/s recovering 2024-09-13T14:47:53.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:52 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3183367398' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:53.154 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:53.180 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:53.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:53.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:53.282 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:53.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:53.329 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:53.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:53.406 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T14:47:53.430 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:53.476 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:53.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:53.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:53.579 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:53.626 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:53.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:53.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:53.720 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:47:53.729 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:53.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:53.780 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:53.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:53.881 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:53.906 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:53.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:53.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:54.008 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:54.033 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:54.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:54.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:54.135 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:54.158 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:54.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:54.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:54.260 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:54.293 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:54.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:54.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:54.395 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:54.421 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:54.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:54.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:54.523 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:54.555 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:54.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:54.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:54.657 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:54.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:54.698 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:54.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:54.799 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:54.824 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:54.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:54.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:54.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:54 smithi151 ceph-mon[108966]: pgmap v1621: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.2 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 674 op/s; 3287/107121 objects degraded (3.068%); 24597/107121 objects misplaced (22.962%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:47:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:54 smithi151 ceph-mon[104450]: pgmap v1621: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.2 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 674 op/s; 3287/107121 objects degraded (3.068%); 24597/107121 objects misplaced (22.962%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:47:54.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:47:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:47:54] "GET /metrics HTTP/1.1" 200 36663 "" "Prometheus/2.51.0" 2024-09-13T14:47:54.926 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:54.948 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:54.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:54.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:55.050 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:55.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:54 smithi184 ceph-mon[92527]: pgmap v1621: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.2 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 674 op/s; 3287/107121 objects degraded (3.068%); 24597/107121 objects misplaced (22.962%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:47:55.108 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:55.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:55.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:55.210 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:55.234 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:55.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:55.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:55.336 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:55.368 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:55.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:55.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:55.469 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:55.496 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:55.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:55.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:55.598 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:55.645 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:55.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:55.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:55.747 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:55.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:55.803 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:55.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:55.885 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:47:55.886 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:47:55.886 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:55 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 3287/107121 objects degraded (3.068%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:55.904 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:55.931 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:55.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:55.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:55.960 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:55 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 3287/107121 objects degraded (3.068%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:55.961 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:55 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 3287/107121 objects degraded (3.068%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:47:56.032 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:56.056 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:56.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:56.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:56.158 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:56.180 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:56.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:56.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:56.282 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:56.307 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:56.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:56.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:56.409 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:56.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:56.442 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:56.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:56.496 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:47:56.542 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:56.564 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T14:47:56.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:56.586 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:56.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:56.687 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:56.732 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:56.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:56.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:56.834 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:56.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:56 smithi184 ceph-mon[92527]: pgmap v1622: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.1 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 587 op/s; 3287/104367 objects degraded (3.149%); 24055/104367 objects misplaced (23.048%); 885 KiB/s, 13 objects/s recovering 2024-09-13T14:47:56.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:56 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:47:56.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:56 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:47:56.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:56 smithi151 ceph-mon[104450]: pgmap v1622: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.1 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 587 op/s; 3287/104367 objects degraded (3.149%); 24055/104367 objects misplaced (23.048%); 885 KiB/s, 13 objects/s recovering 2024-09-13T14:47:56.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:56 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:47:56.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:56 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:47:56.864 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:56 smithi151 ceph-mon[108966]: pgmap v1622: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.1 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 587 op/s; 3287/104367 objects degraded (3.149%); 24055/104367 objects misplaced (23.048%); 885 KiB/s, 13 objects/s recovering 2024-09-13T14:47:56.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:56 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:47:56.865 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:56 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:47:56.867 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:56.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:56.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:56.868 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:47:56.969 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:56.993 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:56.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:56.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:57.095 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:57.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:57.120 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:57.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:57.221 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:57.259 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:57.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:57.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:57.361 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:57.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:57.403 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:57.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:57.504 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:57.529 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:57.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:57.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:57.631 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:57.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:57.681 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:57.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:57.782 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:57.829 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:57.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:57.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:57.930 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:57.968 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:57.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:57.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:58.070 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:58.072 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:57 smithi184 ceph-mon[92527]: from='client.55150 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:47:58.072 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:57 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:47:58.072 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:57 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:47:58.072 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:57 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:47:58.106 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:58.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:58.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:58.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:57 smithi151 ceph-mon[104450]: from='client.55150 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:47:58.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:57 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:47:58.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:57 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:47:58.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:57 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:47:58.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:57 smithi151 ceph-mon[108966]: from='client.55150 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:47:58.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:57 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:47:58.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:57 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:47:58.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:57 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:47:58.209 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:58.254 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:58.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:58.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:58.356 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:58.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:58.379 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:58.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:58.480 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:58.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:58.509 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:58.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:58.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:58.657 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:58.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:58.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:58.667 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:47:58.667 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:47:58.760 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:58.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:58.789 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:58.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:58.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:47:58 smithi151 ceph-mon[104450]: pgmap v1623: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.1 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 581 op/s; 3390/101970 objects degraded (3.325%); 23662/101970 objects misplaced (23.205%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T14:47:58.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:47:58 smithi151 ceph-mon[108966]: pgmap v1623: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.1 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 581 op/s; 3390/101970 objects degraded (3.325%); 23662/101970 objects misplaced (23.205%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T14:47:58.889 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:58.928 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:58.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:58.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:58.976 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:47:58.983 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T14:47:58.998 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:47:58 smithi184 ceph-mon[92527]: pgmap v1623: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.1 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 581 op/s; 3390/101970 objects degraded (3.325%); 23662/101970 objects misplaced (23.205%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T14:47:59.030 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:59.086 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:59.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:59.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:59.188 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:59.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:59.236 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:59.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:59.338 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:59.361 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T14:47:59.367 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:59.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:59.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:59.470 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:59.509 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:59.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:59.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:59.610 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:59.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:59.663 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:59.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:59.675 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:47:59.725 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1622,"stamp":"2024-09-13T14:47:55.503853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2278785208,"num_objects":34789,"num_object_clones":0,"num_object_copies":104367,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3287,"num_objects_misplaced":24055,"num_objects_unfound":0,"num_objects_dirty":34789,"num_whiteouts":0,"num_read":14,"num_read_kb":35,"num_write":206225,"num_write_kb":7709418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2090,"num_bytes_recovered":136407289,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115336,"ondisk_log_size":115336,"up":357,"acting":356,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":366,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":9675748,"kb_used_data":7199908,"kb_used_omap":149,"kb_used_meta":2475562,"kb_avail":740285468,"statfs":{"total":767960285184,"available":758052319232,"internally_reserved":0,"allocated":7372705792,"data_stored":7344324159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":153401,"internal_metadata":2534975687},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,3,4,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-462225408,"num_objects":-7053,"num_object_clones":0,"num_object_copies":-21159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":249,"num_objects_misplaced":-3599,"num_objects_unfound":0,"num_objects_dirty":-7053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7053,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":10878976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007160"},"pg_stats":[{"pgid":"1.1d","version":"261'100","reported_seq":92,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.610360+0000","last_change":"2024-09-13T14:45:19.807356+0000","last_active":"2024-09-13T14:47:40.610360+0000","last_peered":"2024-09-13T14:47:40.610360+0000","last_clean":"2024-09-13T14:47:40.610360+0000","last_became_active":"2024-09-13T14:45:19.807081+0000","last_became_peered":"2024-09-13T14:45:19.807081+0000","last_unstale":"2024-09-13T14:47:40.610360+0000","last_undegraded":"2024-09-13T14:47:40.610360+0000","last_fullsized":"2024-09-13T14:47:40.610360+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:17.988345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:17.988345+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:41:05.216335+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":1698,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.606616+0000","last_change":"2024-09-13T14:29:42.541263+0000","last_active":"2024-09-13T14:47:40.606616+0000","last_peered":"2024-09-13T14:47:40.606616+0000","last_clean":"2024-09-13T14:47:40.606616+0000","last_became_active":"2024-09-13T14:29:42.540873+0000","last_became_peered":"2024-09-13T14:29:42.540873+0000","last_unstale":"2024-09-13T14:47:40.606616+0000","last_undegraded":"2024-09-13T14:47:40.606616+0000","last_fullsized":"2024-09-13T14:47:40.606616+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:20.611206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1121,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.838057+0000","last_change":"2024-09-13T14:43:07.516715+0000","last_active":"2024-09-13T14:47:23.838057+0000","last_peered":"2024-09-13T14:47:23.838057+0000","last_clean":"2024-09-13T14:47:23.838057+0000","last_became_active":"2024-09-13T14:43:07.516403+0000","last_became_peered":"2024-09-13T14:43:07.516403+0000","last_unstale":"2024-09-13T14:47:23.838057+0000","last_undegraded":"2024-09-13T14:47:23.838057+0000","last_fullsized":"2024-09-13T14:47:23.838057+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:56.469690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":37,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.613417+0000","last_change":"2024-09-13T14:45:19.810203+0000","last_active":"2024-09-13T14:47:40.613417+0000","last_peered":"2024-09-13T14:47:40.613417+0000","last_clean":"2024-09-13T14:47:40.613417+0000","last_became_active":"2024-09-13T14:45:19.809693+0000","last_became_peered":"2024-09-13T14:45:19.809693+0000","last_unstale":"2024-09-13T14:47:40.613417+0000","last_undegraded":"2024-09-13T14:47:40.613417+0000","last_fullsized":"2024-09-13T14:47:40.613417+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:20.219673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":37,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.613822+0000","last_change":"2024-09-13T14:45:19.807991+0000","last_active":"2024-09-13T14:47:40.613822+0000","last_peered":"2024-09-13T14:47:40.613822+0000","last_clean":"2024-09-13T14:47:40.613822+0000","last_became_active":"2024-09-13T14:45:19.807662+0000","last_became_peered":"2024-09-13T14:45:19.807662+0000","last_unstale":"2024-09-13T14:47:40.613822+0000","last_undegraded":"2024-09-13T14:47:40.613822+0000","last_fullsized":"2024-09-13T14:47:40.613822+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:29.306108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"261'100","reported_seq":1232,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.836766+0000","last_change":"2024-09-13T14:44:54.259562+0000","last_active":"2024-09-13T14:47:23.836766+0000","last_peered":"2024-09-13T14:47:23.836766+0000","last_clean":"2024-09-13T14:47:23.836766+0000","last_became_active":"2024-09-13T14:43:14.563717+0000","last_became_peered":"2024-09-13T14:43:14.563717+0000","last_unstale":"2024-09-13T14:47:23.836766+0000","last_undegraded":"2024-09-13T14:47:23.836766+0000","last_fullsized":"2024-09-13T14:47:23.836766+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_clean_scrub_stamp":"2024-09-13T14:44:54.259482+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:43:52.729109+0000","scrub_duration":0.015265569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":1089,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.848332+0000","last_change":"2024-09-13T14:19:25.873514+0000","last_active":"2024-09-13T14:47:23.848332+0000","last_peered":"2024-09-13T14:47:23.848332+0000","last_clean":"2024-09-13T14:47:23.848332+0000","last_became_active":"2024-09-13T14:19:25.873170+0000","last_became_peered":"2024-09-13T14:19:25.873170+0000","last_unstale":"2024-09-13T14:47:23.848332+0000","last_undegraded":"2024-09-13T14:47:23.848332+0000","last_fullsized":"2024-09-13T14:47:23.848332+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:48.298530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1118,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.838247+0000","last_change":"2024-09-13T14:30:15.580011+0000","last_active":"2024-09-13T14:47:23.838247+0000","last_peered":"2024-09-13T14:47:23.838247+0000","last_clean":"2024-09-13T14:47:23.838247+0000","last_became_active":"2024-09-13T14:30:15.579610+0000","last_became_peered":"2024-09-13T14:30:15.579610+0000","last_unstale":"2024-09-13T14:47:23.838247+0000","last_undegraded":"2024-09-13T14:47:23.838247+0000","last_fullsized":"2024-09-13T14:47:23.838247+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:11:02.733771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"261'100","reported_seq":54,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.613548+0000","last_change":"2024-09-13T14:45:35.947332+0000","last_active":"2024-09-13T14:47:40.613548+0000","last_peered":"2024-09-13T14:47:40.613548+0000","last_clean":"2024-09-13T14:47:40.613548+0000","last_became_active":"2024-09-13T14:45:19.810775+0000","last_became_peered":"2024-09-13T14:45:19.810775+0000","last_unstale":"2024-09-13T14:47:40.613548+0000","last_undegraded":"2024-09-13T14:47:40.613548+0000","last_fullsized":"2024-09-13T14:47:40.613548+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:35.947265+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:43:41.569795+0000","last_clean_scrub_stamp":"2024-09-13T14:45:35.947265+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:00.606370+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":1134,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.848164+0000","last_change":"2024-09-13T14:43:07.536116+0000","last_active":"2024-09-13T14:47:23.848164+0000","last_peered":"2024-09-13T14:47:23.848164+0000","last_clean":"2024-09-13T14:47:23.848164+0000","last_became_active":"2024-09-13T14:43:07.524112+0000","last_became_peered":"2024-09-13T14:43:07.524112+0000","last_unstale":"2024-09-13T14:47:23.848164+0000","last_undegraded":"2024-09-13T14:47:23.848164+0000","last_fullsized":"2024-09-13T14:47:23.848164+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:36.947060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1131,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.838517+0000","last_change":"2024-09-13T14:43:07.530236+0000","last_active":"2024-09-13T14:47:23.838517+0000","last_peered":"2024-09-13T14:47:23.838517+0000","last_clean":"2024-09-13T14:47:23.838517+0000","last_became_active":"2024-09-13T14:43:07.529967+0000","last_became_peered":"2024-09-13T14:43:07.529967+0000","last_unstale":"2024-09-13T14:47:23.838517+0000","last_undegraded":"2024-09-13T14:47:23.838517+0000","last_fullsized":"2024-09-13T14:47:23.838517+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:27.276550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":1682,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.599849+0000","last_change":"2024-09-13T14:28:35.612469+0000","last_active":"2024-09-13T14:47:40.599849+0000","last_peered":"2024-09-13T14:47:40.599849+0000","last_clean":"2024-09-13T14:47:40.599849+0000","last_became_active":"2024-09-13T14:28:35.612093+0000","last_became_peered":"2024-09-13T14:28:35.612093+0000","last_unstale":"2024-09-13T14:47:40.599849+0000","last_undegraded":"2024-09-13T14:47:40.599849+0000","last_fullsized":"2024-09-13T14:47:40.599849+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:51.022663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1705,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.616839+0000","last_change":"2024-09-13T14:45:19.808811+0000","last_active":"2024-09-13T14:47:40.616839+0000","last_peered":"2024-09-13T14:47:40.616839+0000","last_clean":"2024-09-13T14:47:40.616839+0000","last_became_active":"2024-09-13T14:45:19.808471+0000","last_became_peered":"2024-09-13T14:45:19.808471+0000","last_unstale":"2024-09-13T14:47:40.616839+0000","last_undegraded":"2024-09-13T14:47:40.616839+0000","last_fullsized":"2024-09-13T14:47:40.616839+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:53.638161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"261'100","reported_seq":37,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.613906+0000","last_change":"2024-09-13T14:45:19.813412+0000","last_active":"2024-09-13T14:47:40.613906+0000","last_peered":"2024-09-13T14:47:40.613906+0000","last_clean":"2024-09-13T14:47:40.613906+0000","last_became_active":"2024-09-13T14:45:19.812585+0000","last_became_peered":"2024-09-13T14:45:19.812585+0000","last_unstale":"2024-09-13T14:47:40.613906+0000","last_undegraded":"2024-09-13T14:47:40.613906+0000","last_fullsized":"2024-09-13T14:47:40.613906+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.614340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.614340+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:32:07.434109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":1140,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.838327+0000","last_change":"2024-09-13T14:43:07.522950+0000","last_active":"2024-09-13T14:47:23.838327+0000","last_peered":"2024-09-13T14:47:23.838327+0000","last_clean":"2024-09-13T14:47:23.838327+0000","last_became_active":"2024-09-13T14:43:07.522566+0000","last_became_peered":"2024-09-13T14:43:07.522566+0000","last_unstale":"2024-09-13T14:47:23.838327+0000","last_undegraded":"2024-09-13T14:47:23.838327+0000","last_fullsized":"2024-09-13T14:47:23.838327+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:03.201478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":37,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.614127+0000","last_change":"2024-09-13T14:45:19.807231+0000","last_active":"2024-09-13T14:47:40.614127+0000","last_peered":"2024-09-13T14:47:40.614127+0000","last_clean":"2024-09-13T14:47:40.614127+0000","last_became_active":"2024-09-13T14:45:19.806936+0000","last_became_peered":"2024-09-13T14:45:19.806936+0000","last_unstale":"2024-09-13T14:47:40.614127+0000","last_undegraded":"2024-09-13T14:47:40.614127+0000","last_fullsized":"2024-09-13T14:47:40.614127+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:04:26.723153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.18","version":"261'100","reported_seq":110,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.848060+0000","last_change":"2024-09-13T14:44:41.685537+0000","last_active":"2024-09-13T14:47:23.848060+0000","last_peered":"2024-09-13T14:47:23.848060+0000","last_clean":"2024-09-13T14:47:23.848060+0000","last_became_active":"2024-09-13T14:43:14.558502+0000","last_became_peered":"2024-09-13T14:43:14.558502+0000","last_unstale":"2024-09-13T14:47:23.848060+0000","last_undegraded":"2024-09-13T14:47:23.848060+0000","last_fullsized":"2024-09-13T14:47:23.848060+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1134,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:41.685124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:41.685124+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:37:58.412497+0000","scrub_duration":0.0075205970000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":85,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.611288+0000","last_change":"2024-09-13T14:43:07.516845+0000","last_active":"2024-09-13T14:47:40.611288+0000","last_peered":"2024-09-13T14:47:40.611288+0000","last_clean":"2024-09-13T14:47:40.611288+0000","last_became_active":"2024-09-13T14:43:07.514974+0000","last_became_peered":"2024-09-13T14:43:07.514974+0000","last_unstale":"2024-09-13T14:47:40.611288+0000","last_undegraded":"2024-09-13T14:47:40.611288+0000","last_fullsized":"2024-09-13T14:47:40.611288+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:55.419483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1720,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.607281+0000","last_change":"2024-09-13T14:45:19.815341+0000","last_active":"2024-09-13T14:47:40.607281+0000","last_peered":"2024-09-13T14:47:40.607281+0000","last_clean":"2024-09-13T14:47:40.607281+0000","last_became_active":"2024-09-13T14:45:19.810262+0000","last_became_peered":"2024-09-13T14:45:19.810262+0000","last_unstale":"2024-09-13T14:47:40.607281+0000","last_undegraded":"2024-09-13T14:47:40.607281+0000","last_fullsized":"2024-09-13T14:47:40.607281+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:39.847653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1b","version":"261'100","reported_seq":52,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.849307+0000","last_change":"2024-09-13T14:44:52.803330+0000","last_active":"2024-09-13T14:47:23.849307+0000","last_peered":"2024-09-13T14:47:23.849307+0000","last_clean":"2024-09-13T14:47:23.849307+0000","last_became_active":"2024-09-13T14:43:14.556712+0000","last_became_peered":"2024-09-13T14:43:14.556712+0000","last_unstale":"2024-09-13T14:47:23.849307+0000","last_undegraded":"2024-09-13T14:47:23.849307+0000","last_fullsized":"2024-09-13T14:47:23.849307+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:52.803242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:52.803242+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:06.658301+0000","scrub_duration":0.0019769660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":37,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.612667+0000","last_change":"2024-09-13T14:45:19.806623+0000","last_active":"2024-09-13T14:47:40.612667+0000","last_peered":"2024-09-13T14:47:40.612667+0000","last_clean":"2024-09-13T14:47:40.612667+0000","last_became_active":"2024-09-13T14:45:19.806219+0000","last_became_peered":"2024-09-13T14:45:19.806219+0000","last_unstale":"2024-09-13T14:47:40.612667+0000","last_undegraded":"2024-09-13T14:47:40.612667+0000","last_fullsized":"2024-09-13T14:47:40.612667+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:34:41.601201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1114,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.848432+0000","last_change":"2024-09-13T14:43:32.637751+0000","last_active":"2024-09-13T14:47:23.848432+0000","last_peered":"2024-09-13T14:47:23.848432+0000","last_clean":"2024-09-13T14:47:23.848432+0000","last_became_active":"2024-09-13T14:43:32.636901+0000","last_became_peered":"2024-09-13T14:43:32.636901+0000","last_unstale":"2024-09-13T14:47:23.848432+0000","last_undegraded":"2024-09-13T14:47:23.848432+0000","last_fullsized":"2024-09-13T14:47:23.848432+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:21.942419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1a","version":"261'100","reported_seq":52,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.838154+0000","last_change":"2024-09-13T14:43:41.910041+0000","last_active":"2024-09-13T14:47:23.838154+0000","last_peered":"2024-09-13T14:47:23.838154+0000","last_clean":"2024-09-13T14:47:23.838154+0000","last_became_active":"2024-09-13T14:43:14.555083+0000","last_became_peered":"2024-09-13T14:43:14.555083+0000","last_unstale":"2024-09-13T14:47:23.838154+0000","last_undegraded":"2024-09-13T14:47:23.838154+0000","last_fullsized":"2024-09-13T14:47:23.838154+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:41.909988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:41.909988+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:42:19.954838+0000","scrub_duration":0.0012891619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":37,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.614318+0000","last_change":"2024-09-13T14:45:19.812071+0000","last_active":"2024-09-13T14:47:40.614318+0000","last_peered":"2024-09-13T14:47:40.614318+0000","last_clean":"2024-09-13T14:47:40.614318+0000","last_became_active":"2024-09-13T14:45:19.811336+0000","last_became_peered":"2024-09-13T14:45:19.811336+0000","last_unstale":"2024-09-13T14:47:40.614318+0000","last_undegraded":"2024-09-13T14:47:40.614318+0000","last_fullsized":"2024-09-13T14:47:40.614318+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:40:29.814608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1738,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.606573+0000","last_change":"2024-09-13T14:45:19.810478+0000","last_active":"2024-09-13T14:47:40.606573+0000","last_peered":"2024-09-13T14:47:40.606573+0000","last_clean":"2024-09-13T14:47:40.606573+0000","last_became_active":"2024-09-13T14:45:19.808278+0000","last_became_peered":"2024-09-13T14:45:19.808278+0000","last_unstale":"2024-09-13T14:47:40.606573+0000","last_undegraded":"2024-09-13T14:47:40.606573+0000","last_fullsized":"2024-09-13T14:47:40.606573+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:12.211089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"261'100","reported_seq":52,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.836746+0000","last_change":"2024-09-13T14:44:38.214961+0000","last_active":"2024-09-13T14:47:23.836746+0000","last_peered":"2024-09-13T14:47:23.836746+0000","last_clean":"2024-09-13T14:47:23.836746+0000","last_became_active":"2024-09-13T14:43:14.557883+0000","last_became_peered":"2024-09-13T14:43:14.557883+0000","last_unstale":"2024-09-13T14:47:23.836746+0000","last_undegraded":"2024-09-13T14:47:23.836746+0000","last_fullsized":"2024-09-13T14:47:23.836746+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:38.214865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:38.214865+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:49:36.013783+0000","scrub_duration":0.0016389639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1608,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.614006+0000","last_change":"2024-09-13T14:30:15.576921+0000","last_active":"2024-09-13T14:47:40.614006+0000","last_peered":"2024-09-13T14:47:40.614006+0000","last_clean":"2024-09-13T14:47:40.614006+0000","last_became_active":"2024-09-13T14:30:15.576607+0000","last_became_peered":"2024-09-13T14:30:15.576607+0000","last_unstale":"2024-09-13T14:47:40.614006+0000","last_undegraded":"2024-09-13T14:47:40.614006+0000","last_fullsized":"2024-09-13T14:47:40.614006+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:35.162460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1124,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.848350+0000","last_change":"2024-09-13T14:43:07.518801+0000","last_active":"2024-09-13T14:47:23.848350+0000","last_peered":"2024-09-13T14:47:23.848350+0000","last_clean":"2024-09-13T14:47:23.848350+0000","last_became_active":"2024-09-13T14:43:07.517610+0000","last_became_peered":"2024-09-13T14:43:07.517610+0000","last_unstale":"2024-09-13T14:47:23.848350+0000","last_undegraded":"2024-09-13T14:47:23.848350+0000","last_fullsized":"2024-09-13T14:47:23.848350+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:51.632254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"261'100","reported_seq":85,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.616390+0000","last_change":"2024-09-13T14:43:15.037565+0000","last_active":"2024-09-13T14:47:40.616390+0000","last_peered":"2024-09-13T14:47:40.616390+0000","last_clean":"2024-09-13T14:47:40.616390+0000","last_became_active":"2024-09-13T14:43:14.554183+0000","last_became_peered":"2024-09-13T14:43:14.554183+0000","last_unstale":"2024-09-13T14:47:40.616390+0000","last_undegraded":"2024-09-13T14:47:40.616390+0000","last_fullsized":"2024-09-13T14:47:40.616390+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.037517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.037517+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:01:35.137195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":1118,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.836560+0000","last_change":"2024-09-13T14:30:15.579260+0000","last_active":"2024-09-13T14:47:23.836560+0000","last_peered":"2024-09-13T14:47:23.836560+0000","last_clean":"2024-09-13T14:47:23.836560+0000","last_became_active":"2024-09-13T14:30:15.578897+0000","last_became_peered":"2024-09-13T14:30:15.578897+0000","last_unstale":"2024-09-13T14:47:23.836560+0000","last_undegraded":"2024-09-13T14:47:23.836560+0000","last_fullsized":"2024-09-13T14:47:23.836560+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:56.737849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":86,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.607036+0000","last_change":"2024-09-13T14:43:07.521576+0000","last_active":"2024-09-13T14:47:40.607036+0000","last_peered":"2024-09-13T14:47:40.607036+0000","last_clean":"2024-09-13T14:47:40.607036+0000","last_became_active":"2024-09-13T14:43:07.520478+0000","last_became_peered":"2024-09-13T14:43:07.520478+0000","last_unstale":"2024-09-13T14:47:40.607036+0000","last_undegraded":"2024-09-13T14:47:40.607036+0000","last_fullsized":"2024-09-13T14:47:40.607036+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:53.410135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"261'100","reported_seq":378,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.848334+0000","last_change":"2024-09-13T14:44:49.775941+0000","last_active":"2024-09-13T14:47:23.848334+0000","last_peered":"2024-09-13T14:47:23.848334+0000","last_clean":"2024-09-13T14:47:23.848334+0000","last_became_active":"2024-09-13T14:43:14.560297+0000","last_became_peered":"2024-09-13T14:43:14.560297+0000","last_unstale":"2024-09-13T14:47:23.848334+0000","last_undegraded":"2024-09-13T14:47:23.848334+0000","last_fullsized":"2024-09-13T14:47:23.848334+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:49.775844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:49.775844+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:15:53.138661+0000","scrub_duration":0.012350274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":1107,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.838627+0000","last_change":"2024-09-13T14:19:25.870605+0000","last_active":"2024-09-13T14:47:23.838627+0000","last_peered":"2024-09-13T14:47:23.838627+0000","last_clean":"2024-09-13T14:47:23.838627+0000","last_became_active":"2024-09-13T14:19:25.870282+0000","last_became_peered":"2024-09-13T14:19:25.870282+0000","last_unstale":"2024-09-13T14:47:23.838627+0000","last_undegraded":"2024-09-13T14:47:23.838627+0000","last_fullsized":"2024-09-13T14:47:23.838627+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:52.514915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1121,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.838523+0000","last_change":"2024-09-13T14:43:07.526476+0000","last_active":"2024-09-13T14:47:23.838523+0000","last_peered":"2024-09-13T14:47:23.838523+0000","last_clean":"2024-09-13T14:47:23.838523+0000","last_became_active":"2024-09-13T14:43:07.525802+0000","last_became_peered":"2024-09-13T14:43:07.525802+0000","last_unstale":"2024-09-13T14:47:23.838523+0000","last_undegraded":"2024-09-13T14:47:23.838523+0000","last_fullsized":"2024-09-13T14:47:23.838523+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:23.154726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"261'100","reported_seq":92,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.606342+0000","last_change":"2024-09-13T14:45:19.807461+0000","last_active":"2024-09-13T14:47:40.606342+0000","last_peered":"2024-09-13T14:47:40.606342+0000","last_clean":"2024-09-13T14:47:40.606342+0000","last_became_active":"2024-09-13T14:45:19.807117+0000","last_became_peered":"2024-09-13T14:45:19.807117+0000","last_unstale":"2024-09-13T14:47:40.606342+0000","last_undegraded":"2024-09-13T14:47:40.606342+0000","last_fullsized":"2024-09-13T14:47:40.606342+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.955197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.955197+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:03.531555+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"1161'12997","reported_seq":13065,"reported_epoch":1161,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:47:52.757838+0000","last_change":"2024-09-13T14:45:21.161658+0000","last_active":"2024-09-13T14:47:52.757838+0000","last_peered":"2024-09-13T14:47:52.757838+0000","last_clean":"2024-09-13T14:45:19.622135+0000","last_became_active":"2024-09-13T14:45:21.090442+0000","last_became_peered":"2024-09-13T14:45:21.090442+0000","last_unstale":"2024-09-13T14:47:52.757838+0000","last_undegraded":"2024-09-13T14:47:52.757838+0000","last_fullsized":"2024-09-13T14:47:52.757838+0000","mapping_epoch":1149,"log_start":"1147'6900","ondisk_log_start":"1147'6900","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6097,"log_dups_size":0,"ondisk_log_size":6097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144769024,"num_objects":2209,"num_object_clones":0,"num_object_copies":6627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2209,"num_objects_unfound":0,"num_objects_dirty":2209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12997,"num_write_kb":486592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[1,2,7],"avail_no_missing":["1","2","7"],"object_location_counts":[{"shards":"1,2,7","objects":2209}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1084,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.838271+0000","last_change":"2024-09-13T14:19:26.131609+0000","last_active":"2024-09-13T14:47:23.838271+0000","last_peered":"2024-09-13T14:47:23.838271+0000","last_clean":"2024-09-13T14:47:23.838271+0000","last_became_active":"2024-09-13T14:19:26.131029+0000","last_became_peered":"2024-09-13T14:19:26.131029+0000","last_unstale":"2024-09-13T14:47:23.838271+0000","last_undegraded":"2024-09-13T14:47:23.838271+0000","last_fullsized":"2024-09-13T14:47:23.838271+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:57.933696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":586,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.836964+0000","last_change":"2024-09-13T14:43:32.635070+0000","last_active":"2024-09-13T14:47:23.836964+0000","last_peered":"2024-09-13T14:47:23.836964+0000","last_clean":"2024-09-13T14:47:23.836964+0000","last_became_active":"2024-09-13T14:43:32.634734+0000","last_became_peered":"2024-09-13T14:43:32.634734+0000","last_unstale":"2024-09-13T14:47:23.836964+0000","last_undegraded":"2024-09-13T14:47:23.836964+0000","last_fullsized":"2024-09-13T14:47:23.836964+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:33.107010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":1727,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.607439+0000","last_change":"2024-09-13T14:43:07.537759+0000","last_active":"2024-09-13T14:47:40.607439+0000","last_peered":"2024-09-13T14:47:40.607439+0000","last_clean":"2024-09-13T14:47:40.607439+0000","last_became_active":"2024-09-13T14:43:07.520303+0000","last_became_peered":"2024-09-13T14:43:07.520303+0000","last_unstale":"2024-09-13T14:47:40.607439+0000","last_undegraded":"2024-09-13T14:47:40.607439+0000","last_fullsized":"2024-09-13T14:47:40.607439+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:32.772570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"261'100","reported_seq":37,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.613394+0000","last_change":"2024-09-13T14:45:19.813527+0000","last_active":"2024-09-13T14:47:40.613394+0000","last_peered":"2024-09-13T14:47:40.613394+0000","last_clean":"2024-09-13T14:47:40.613394+0000","last_became_active":"2024-09-13T14:45:19.812834+0000","last_became_peered":"2024-09-13T14:45:19.812834+0000","last_unstale":"2024-09-13T14:47:40.613394+0000","last_undegraded":"2024-09-13T14:47:40.613394+0000","last_fullsized":"2024-09-13T14:47:40.613394+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.710192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.710192+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:08:55.152703+0000","scrub_duration":0.0019250910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"1161'12870","reported_seq":17027,"reported_epoch":1161,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:47:51.757000+0000","last_change":"2024-09-13T14:45:38.479094+0000","last_active":"2024-09-13T14:47:51.757000+0000","last_peered":"2024-09-13T14:47:51.757000+0000","last_clean":"2024-09-13T14:45:19.642017+0000","last_became_active":"2024-09-13T14:45:21.111093+0000","last_became_peered":"2024-09-13T14:45:21.111093+0000","last_unstale":"2024-09-13T14:47:51.757000+0000","last_undegraded":"2024-09-13T14:45:38.458706+0000","last_fullsized":"2024-09-13T14:45:20.800066+0000","mapping_epoch":1149,"log_start":"1142'4500","ondisk_log_start":"1142'4500","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":8370,"log_dups_size":0,"ondisk_log_size":8370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144572416,"num_objects":2206,"num_object_clones":0,"num_object_copies":6618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3287,"num_objects_misplaced":1762,"num_objects_unfound":0,"num_objects_dirty":2206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12870,"num_write_kb":482432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2046,"num_bytes_recovered":134086656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[2,4],"avail_no_missing":["2","4","6"],"object_location_counts":[{"shards":"2,4","objects":-2843},{"shards":"2,4,6","objects":1762}],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":37,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.614360+0000","last_change":"2024-09-13T14:45:19.813025+0000","last_active":"2024-09-13T14:47:40.614360+0000","last_peered":"2024-09-13T14:47:40.614360+0000","last_clean":"2024-09-13T14:47:40.614360+0000","last_became_active":"2024-09-13T14:45:19.812714+0000","last_became_peered":"2024-09-13T14:45:19.812714+0000","last_unstale":"2024-09-13T14:47:40.614360+0000","last_undegraded":"2024-09-13T14:47:40.614360+0000","last_fullsized":"2024-09-13T14:47:40.614360+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:40.671012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":86,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.618254+0000","last_change":"2024-09-13T14:43:07.522730+0000","last_active":"2024-09-13T14:47:40.618254+0000","last_peered":"2024-09-13T14:47:40.618254+0000","last_clean":"2024-09-13T14:47:40.618254+0000","last_became_active":"2024-09-13T14:43:07.517420+0000","last_became_peered":"2024-09-13T14:43:07.517420+0000","last_unstale":"2024-09-13T14:47:40.618254+0000","last_undegraded":"2024-09-13T14:47:40.618254+0000","last_fullsized":"2024-09-13T14:47:40.618254+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:21:30.265293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":1641,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.616824+0000","last_change":"2024-09-13T14:30:15.581728+0000","last_active":"2024-09-13T14:47:40.616824+0000","last_peered":"2024-09-13T14:47:40.616824+0000","last_clean":"2024-09-13T14:47:40.616824+0000","last_became_active":"2024-09-13T14:30:15.581303+0000","last_became_peered":"2024-09-13T14:30:15.581303+0000","last_unstale":"2024-09-13T14:47:40.616824+0000","last_undegraded":"2024-09-13T14:47:40.616824+0000","last_fullsized":"2024-09-13T14:47:40.616824+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:10.027991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"261'100","reported_seq":111,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.838559+0000","last_change":"2024-09-13T14:43:31.112173+0000","last_active":"2024-09-13T14:47:23.838559+0000","last_peered":"2024-09-13T14:47:23.838559+0000","last_clean":"2024-09-13T14:47:23.838559+0000","last_became_active":"2024-09-13T14:42:42.709263+0000","last_became_peered":"2024-09-13T14:42:42.709263+0000","last_unstale":"2024-09-13T14:47:23.838559+0000","last_undegraded":"2024-09-13T14:47:23.838559+0000","last_fullsized":"2024-09-13T14:47:23.838559+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:31.112111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:31.112111+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:02.707372+0000","scrub_duration":0.0015759260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"1161'13039","reported_seq":13083,"reported_epoch":1161,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:47:52.680500+0000","last_change":"2024-09-13T14:45:21.011501+0000","last_active":"2024-09-13T14:47:52.680500+0000","last_peered":"2024-09-13T14:47:52.680500+0000","last_clean":"2024-09-13T14:45:19.622974+0000","last_became_active":"2024-09-13T14:45:21.006075+0000","last_became_peered":"2024-09-13T14:45:21.006075+0000","last_unstale":"2024-09-13T14:47:52.680500+0000","last_undegraded":"2024-09-13T14:47:52.680500+0000","last_fullsized":"2024-09-13T14:47:52.680500+0000","mapping_epoch":1149,"log_start":"1143'5300","ondisk_log_start":"1143'5300","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":7739,"log_dups_size":0,"ondisk_log_size":7739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:30.090601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145424384,"num_objects":2219,"num_object_clones":0,"num_object_copies":6657,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2219,"num_objects_unfound":0,"num_objects_dirty":2219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13039,"num_write_kb":488256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,2],"avail_no_missing":["5","1","2"],"object_location_counts":[{"shards":"1,2,5","objects":2219}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1681,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.599767+0000","last_change":"2024-09-13T14:28:35.611508+0000","last_active":"2024-09-13T14:47:40.599767+0000","last_peered":"2024-09-13T14:47:40.599767+0000","last_clean":"2024-09-13T14:47:40.599767+0000","last_became_active":"2024-09-13T14:28:35.610106+0000","last_became_peered":"2024-09-13T14:28:35.610106+0000","last_unstale":"2024-09-13T14:47:40.599767+0000","last_undegraded":"2024-09-13T14:47:40.599767+0000","last_fullsized":"2024-09-13T14:47:40.599767+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:11.966958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1127,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.836823+0000","last_change":"2024-09-13T14:28:35.603663+0000","last_active":"2024-09-13T14:47:23.836823+0000","last_peered":"2024-09-13T14:47:23.836823+0000","last_clean":"2024-09-13T14:47:23.836823+0000","last_became_active":"2024-09-13T14:28:35.603358+0000","last_became_peered":"2024-09-13T14:28:35.603358+0000","last_unstale":"2024-09-13T14:47:23.836823+0000","last_undegraded":"2024-09-13T14:47:23.836823+0000","last_fullsized":"2024-09-13T14:47:23.836823+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:24:05.842796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":86,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.609201+0000","last_change":"2024-09-13T14:43:07.522750+0000","last_active":"2024-09-13T14:47:40.609201+0000","last_peered":"2024-09-13T14:47:40.609201+0000","last_clean":"2024-09-13T14:47:40.609201+0000","last_became_active":"2024-09-13T14:43:07.517267+0000","last_became_peered":"2024-09-13T14:43:07.517267+0000","last_unstale":"2024-09-13T14:47:40.609201+0000","last_undegraded":"2024-09-13T14:47:40.609201+0000","last_fullsized":"2024-09-13T14:47:40.609201+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:40.217748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"261'100","reported_seq":407,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.607156+0000","last_change":"2024-09-13T14:45:19.815050+0000","last_active":"2024-09-13T14:47:40.607156+0000","last_peered":"2024-09-13T14:47:40.607156+0000","last_clean":"2024-09-13T14:47:40.607156+0000","last_became_active":"2024-09-13T14:45:19.809732+0000","last_became_peered":"2024-09-13T14:45:19.809732+0000","last_unstale":"2024-09-13T14:47:40.607156+0000","last_undegraded":"2024-09-13T14:47:40.607156+0000","last_fullsized":"2024-09-13T14:47:40.607156+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:33.574443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:33.574443+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:51:40.261036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"1161'12942","reported_seq":13045,"reported_epoch":1161,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:47:52.411981+0000","last_change":"2024-09-13T14:45:21.136109+0000","last_active":"2024-09-13T14:47:52.411981+0000","last_peered":"2024-09-13T14:47:52.411981+0000","last_clean":"2024-09-13T14:45:19.607954+0000","last_became_active":"2024-09-13T14:45:21.004300+0000","last_became_peered":"2024-09-13T14:45:21.004300+0000","last_unstale":"2024-09-13T14:47:52.411981+0000","last_undegraded":"2024-09-13T14:47:52.411981+0000","last_fullsized":"2024-09-13T14:47:52.411981+0000","mapping_epoch":1149,"log_start":"1146'6800","ondisk_log_start":"1146'6800","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6142,"log_dups_size":0,"ondisk_log_size":6142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:04:23.351582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141950976,"num_objects":2166,"num_object_clones":0,"num_object_copies":6498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2154,"num_objects_unfound":0,"num_objects_dirty":2166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12942,"num_write_kb":483456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":1835008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,6,1],"avail_no_missing":["4","1","6"],"object_location_counts":[{"shards":"1,4,6","objects":2166}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":86,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.606893+0000","last_change":"2024-09-13T14:43:07.520049+0000","last_active":"2024-09-13T14:47:40.606893+0000","last_peered":"2024-09-13T14:47:40.606893+0000","last_clean":"2024-09-13T14:47:40.606893+0000","last_became_active":"2024-09-13T14:43:07.514646+0000","last_became_peered":"2024-09-13T14:43:07.514646+0000","last_unstale":"2024-09-13T14:47:40.606893+0000","last_undegraded":"2024-09-13T14:47:40.606893+0000","last_fullsized":"2024-09-13T14:47:40.606893+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:10:35.200778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":86,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.609109+0000","last_change":"2024-09-13T14:43:07.525666+0000","last_active":"2024-09-13T14:47:40.609109+0000","last_peered":"2024-09-13T14:47:40.609109+0000","last_clean":"2024-09-13T14:47:40.609109+0000","last_became_active":"2024-09-13T14:43:07.524064+0000","last_became_peered":"2024-09-13T14:43:07.524064+0000","last_unstale":"2024-09-13T14:47:40.609109+0000","last_undegraded":"2024-09-13T14:47:40.609109+0000","last_fullsized":"2024-09-13T14:47:40.609109+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:10.270333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1141,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.848405+0000","last_change":"2024-09-13T14:45:19.804177+0000","last_active":"2024-09-13T14:47:23.848405+0000","last_peered":"2024-09-13T14:47:23.848405+0000","last_clean":"2024-09-13T14:47:23.848405+0000","last_became_active":"2024-09-13T14:45:19.803890+0000","last_became_peered":"2024-09-13T14:45:19.803890+0000","last_unstale":"2024-09-13T14:47:23.848405+0000","last_undegraded":"2024-09-13T14:47:23.848405+0000","last_fullsized":"2024-09-13T14:47:23.848405+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:25.140789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"261'100","reported_seq":54,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.616657+0000","last_change":"2024-09-13T14:45:36.965992+0000","last_active":"2024-09-13T14:47:40.616657+0000","last_peered":"2024-09-13T14:47:40.616657+0000","last_clean":"2024-09-13T14:47:40.616657+0000","last_became_active":"2024-09-13T14:45:19.811238+0000","last_became_peered":"2024-09-13T14:45:19.811238+0000","last_unstale":"2024-09-13T14:47:40.616657+0000","last_undegraded":"2024-09-13T14:47:40.616657+0000","last_fullsized":"2024-09-13T14:47:40.616657+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:36.965775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:45:36.965775+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:36:01.623251+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"1161'12727","reported_seq":12792,"reported_epoch":1161,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:47:54.457817+0000","last_change":"2024-09-13T14:45:21.036181+0000","last_active":"2024-09-13T14:47:54.457817+0000","last_peered":"2024-09-13T14:47:54.457817+0000","last_clean":"2024-09-13T14:45:19.587756+0000","last_became_active":"2024-09-13T14:45:21.029769+0000","last_became_peered":"2024-09-13T14:45:21.029769+0000","last_unstale":"2024-09-13T14:47:54.457817+0000","last_undegraded":"2024-09-13T14:47:54.457817+0000","last_fullsized":"2024-09-13T14:47:54.457817+0000","mapping_epoch":1149,"log_start":"1146'5900","ondisk_log_start":"1146'5900","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6827,"log_dups_size":0,"ondisk_log_size":6827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135725056,"num_objects":2071,"num_object_clones":0,"num_object_copies":6213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2071,"num_objects_unfound":0,"num_objects_dirty":2071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12727,"num_write_kb":473536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,1],"avail_no_missing":["0","1","6"],"object_location_counts":[{"shards":"0,1,6","objects":2071}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":40,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.848068+0000","last_change":"2024-09-13T14:45:19.806318+0000","last_active":"2024-09-13T14:47:23.848068+0000","last_peered":"2024-09-13T14:47:23.848068+0000","last_clean":"2024-09-13T14:47:23.848068+0000","last_became_active":"2024-09-13T14:45:19.806074+0000","last_became_peered":"2024-09-13T14:45:19.806074+0000","last_unstale":"2024-09-13T14:47:23.848068+0000","last_undegraded":"2024-09-13T14:47:23.848068+0000","last_fullsized":"2024-09-13T14:47:23.848068+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:21.679797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1102,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.838622+0000","last_change":"2024-09-13T14:28:35.604404+0000","last_active":"2024-09-13T14:47:23.838622+0000","last_peered":"2024-09-13T14:47:23.838622+0000","last_clean":"2024-09-13T14:47:23.838622+0000","last_became_active":"2024-09-13T14:28:35.604102+0000","last_became_peered":"2024-09-13T14:28:35.604102+0000","last_unstale":"2024-09-13T14:47:23.838622+0000","last_undegraded":"2024-09-13T14:47:23.838622+0000","last_fullsized":"2024-09-13T14:47:23.838622+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:25.546010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":1152,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.848314+0000","last_change":"2024-09-13T14:43:07.538919+0000","last_active":"2024-09-13T14:47:23.848314+0000","last_peered":"2024-09-13T14:47:23.848314+0000","last_clean":"2024-09-13T14:47:23.848314+0000","last_became_active":"2024-09-13T14:43:07.513109+0000","last_became_peered":"2024-09-13T14:43:07.513109+0000","last_unstale":"2024-09-13T14:47:23.848314+0000","last_undegraded":"2024-09-13T14:47:23.848314+0000","last_fullsized":"2024-09-13T14:47:23.848314+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:27.771720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"261'100","reported_seq":101,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.847934+0000","last_change":"2024-09-13T14:43:44.720333+0000","last_active":"2024-09-13T14:47:23.847934+0000","last_peered":"2024-09-13T14:47:23.847934+0000","last_clean":"2024-09-13T14:47:23.847934+0000","last_became_active":"2024-09-13T14:42:14.946591+0000","last_became_peered":"2024-09-13T14:42:14.946591+0000","last_unstale":"2024-09-13T14:47:23.847934+0000","last_undegraded":"2024-09-13T14:47:23.847934+0000","last_fullsized":"2024-09-13T14:47:23.847934+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:44.720258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:44.720258+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:33.281357+0000","scrub_duration":0.0018417920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"1161'12955","reported_seq":13017,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:54.493841+0000","last_change":"2024-09-13T14:43:50.794091+0000","last_active":"2024-09-13T14:47:54.493841+0000","last_peered":"2024-09-13T14:47:54.493841+0000","last_clean":"2024-09-13T14:47:54.493841+0000","last_became_active":"2024-09-13T14:43:50.793741+0000","last_became_peered":"2024-09-13T14:43:50.793741+0000","last_unstale":"2024-09-13T14:47:54.493841+0000","last_undegraded":"2024-09-13T14:47:54.493841+0000","last_fullsized":"2024-09-13T14:47:54.493841+0000","mapping_epoch":1136,"log_start":"1146'6100","ondisk_log_start":"1146'6100","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6855,"log_dups_size":0,"ondisk_log_size":6855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:15.846494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139919360,"num_objects":2135,"num_object_clones":0,"num_object_copies":6405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12955,"num_write_kb":482880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":53,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.612132+0000","last_change":"2024-09-13T14:44:03.858211+0000","last_active":"2024-09-13T14:47:40.612132+0000","last_peered":"2024-09-13T14:47:40.612132+0000","last_clean":"2024-09-13T14:47:40.612132+0000","last_became_active":"2024-09-13T14:44:03.857928+0000","last_became_peered":"2024-09-13T14:44:03.857928+0000","last_unstale":"2024-09-13T14:47:40.612132+0000","last_undegraded":"2024-09-13T14:47:40.612132+0000","last_fullsized":"2024-09-13T14:47:40.612132+0000","mapping_epoch":1140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:28.428693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":86,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.618601+0000","last_change":"2024-09-13T14:43:07.516243+0000","last_active":"2024-09-13T14:47:40.618601+0000","last_peered":"2024-09-13T14:47:40.618601+0000","last_clean":"2024-09-13T14:47:40.618601+0000","last_became_active":"2024-09-13T14:43:07.515863+0000","last_became_peered":"2024-09-13T14:43:07.515863+0000","last_unstale":"2024-09-13T14:47:40.618601+0000","last_undegraded":"2024-09-13T14:47:40.618601+0000","last_fullsized":"2024-09-13T14:47:40.618601+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:08.085298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1722,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.612042+0000","last_change":"2024-09-13T14:45:19.812713+0000","last_active":"2024-09-13T14:47:40.612042+0000","last_peered":"2024-09-13T14:47:40.612042+0000","last_clean":"2024-09-13T14:47:40.612042+0000","last_became_active":"2024-09-13T14:45:19.812203+0000","last_became_peered":"2024-09-13T14:45:19.812203+0000","last_unstale":"2024-09-13T14:47:40.612042+0000","last_undegraded":"2024-09-13T14:47:40.612042+0000","last_fullsized":"2024-09-13T14:47:40.612042+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:30:46.149347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"261'100","reported_seq":1281,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.836409+0000","last_change":"2024-09-13T14:43:42.577312+0000","last_active":"2024-09-13T14:47:23.836409+0000","last_peered":"2024-09-13T14:47:23.836409+0000","last_clean":"2024-09-13T14:47:23.836409+0000","last_became_active":"2024-09-13T14:42:14.928923+0000","last_became_peered":"2024-09-13T14:42:14.928923+0000","last_unstale":"2024-09-13T14:47:23.836409+0000","last_undegraded":"2024-09-13T14:47:23.836409+0000","last_fullsized":"2024-09-13T14:47:23.836409+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:42.577243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:42.577243+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:01.234250+0000","scrub_duration":0.001367839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"1161'12854","reported_seq":12887,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:52.539265+0000","last_change":"2024-09-13T14:43:50.793810+0000","last_active":"2024-09-13T14:47:52.539265+0000","last_peered":"2024-09-13T14:47:52.539265+0000","last_clean":"2024-09-13T14:47:52.539265+0000","last_became_active":"2024-09-13T14:43:50.793506+0000","last_became_peered":"2024-09-13T14:43:50.793506+0000","last_unstale":"2024-09-13T14:47:52.539265+0000","last_undegraded":"2024-09-13T14:47:52.539265+0000","last_fullsized":"2024-09-13T14:47:52.539265+0000","mapping_epoch":1136,"log_start":"1146'6700","ondisk_log_start":"1146'6700","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6154,"log_dups_size":0,"ondisk_log_size":6154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:50:32.061762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142475264,"num_objects":2174,"num_object_clones":0,"num_object_copies":6522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12854,"num_write_kb":480896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1716,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.607204+0000","last_change":"2024-09-13T14:43:07.520212+0000","last_active":"2024-09-13T14:47:40.607204+0000","last_peered":"2024-09-13T14:47:40.607204+0000","last_clean":"2024-09-13T14:47:40.607204+0000","last_became_active":"2024-09-13T14:43:07.519831+0000","last_became_peered":"2024-09-13T14:43:07.519831+0000","last_unstale":"2024-09-13T14:47:40.607204+0000","last_undegraded":"2024-09-13T14:47:40.607204+0000","last_fullsized":"2024-09-13T14:47:40.607204+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:20.990736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1739,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.607317+0000","last_change":"2024-09-13T14:45:19.809505+0000","last_active":"2024-09-13T14:47:40.607317+0000","last_peered":"2024-09-13T14:47:40.607317+0000","last_clean":"2024-09-13T14:47:40.607317+0000","last_became_active":"2024-09-13T14:45:19.809181+0000","last_became_peered":"2024-09-13T14:45:19.809181+0000","last_unstale":"2024-09-13T14:47:40.607317+0000","last_undegraded":"2024-09-13T14:47:40.607317+0000","last_fullsized":"2024-09-13T14:47:40.607317+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:24.239188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1164,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.836231+0000","last_change":"2024-09-13T14:43:07.516140+0000","last_active":"2024-09-13T14:47:23.836231+0000","last_peered":"2024-09-13T14:47:23.836231+0000","last_clean":"2024-09-13T14:47:23.836231+0000","last_became_active":"2024-09-13T14:43:07.515481+0000","last_became_peered":"2024-09-13T14:43:07.515481+0000","last_unstale":"2024-09-13T14:47:23.836231+0000","last_undegraded":"2024-09-13T14:47:23.836231+0000","last_fullsized":"2024-09-13T14:47:23.836231+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:43.706214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"261'100","reported_seq":98,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.617513+0000","last_change":"2024-09-13T14:43:16.008346+0000","last_active":"2024-09-13T14:47:40.617513+0000","last_peered":"2024-09-13T14:47:40.617513+0000","last_clean":"2024-09-13T14:47:40.617513+0000","last_became_active":"2024-09-13T14:43:07.523121+0000","last_became_peered":"2024-09-13T14:43:07.523121+0000","last_unstale":"2024-09-13T14:47:40.617513+0000","last_undegraded":"2024-09-13T14:47:40.617513+0000","last_fullsized":"2024-09-13T14:47:40.617513+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.008264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.008264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:58:35.564470+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"1161'12913","reported_seq":12979,"reported_epoch":1161,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:47:54.513832+0000","last_change":"2024-09-13T14:45:21.035848+0000","last_active":"2024-09-13T14:47:54.513832+0000","last_peered":"2024-09-13T14:47:54.513832+0000","last_clean":"2024-09-13T14:45:19.637667+0000","last_became_active":"2024-09-13T14:45:21.025308+0000","last_became_peered":"2024-09-13T14:45:21.025308+0000","last_unstale":"2024-09-13T14:47:54.513832+0000","last_undegraded":"2024-09-13T14:47:54.513832+0000","last_fullsized":"2024-09-13T14:47:54.513832+0000","mapping_epoch":1149,"log_start":"1146'6000","ondisk_log_start":"1146'6000","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6913,"log_dups_size":0,"ondisk_log_size":6913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139264000,"num_objects":2125,"num_object_clones":0,"num_object_copies":6375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2125,"num_objects_unfound":0,"num_objects_dirty":2125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12913,"num_write_kb":481216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,5],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":2125}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":43,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.838527+0000","last_change":"2024-09-13T14:43:32.636990+0000","last_active":"2024-09-13T14:47:23.838527+0000","last_peered":"2024-09-13T14:47:23.838527+0000","last_clean":"2024-09-13T14:47:23.838527+0000","last_became_active":"2024-09-13T14:43:32.636589+0000","last_became_peered":"2024-09-13T14:43:32.636589+0000","last_unstale":"2024-09-13T14:47:23.838527+0000","last_undegraded":"2024-09-13T14:47:23.838527+0000","last_fullsized":"2024-09-13T14:47:23.838527+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:36:25.129769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":37,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.612962+0000","last_change":"2024-09-13T14:45:19.810888+0000","last_active":"2024-09-13T14:47:40.612962+0000","last_peered":"2024-09-13T14:47:40.612962+0000","last_clean":"2024-09-13T14:47:40.612962+0000","last_became_active":"2024-09-13T14:45:19.810486+0000","last_became_peered":"2024-09-13T14:45:19.810486+0000","last_unstale":"2024-09-13T14:47:40.612962+0000","last_undegraded":"2024-09-13T14:47:40.612962+0000","last_fullsized":"2024-09-13T14:47:40.612962+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:03.886942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":1117,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.849339+0000","last_change":"2024-09-13T14:29:42.539455+0000","last_active":"2024-09-13T14:47:23.849339+0000","last_peered":"2024-09-13T14:47:23.849339+0000","last_clean":"2024-09-13T14:47:23.849339+0000","last_became_active":"2024-09-13T14:29:42.539068+0000","last_became_peered":"2024-09-13T14:29:42.539068+0000","last_unstale":"2024-09-13T14:47:23.849339+0000","last_undegraded":"2024-09-13T14:47:23.849339+0000","last_fullsized":"2024-09-13T14:47:23.849339+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:24:06.048855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034405999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"261'100","reported_seq":1272,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.836307+0000","last_change":"2024-09-13T14:44:28.186029+0000","last_active":"2024-09-13T14:47:23.836307+0000","last_peered":"2024-09-13T14:47:23.836307+0000","last_clean":"2024-09-13T14:47:23.836307+0000","last_became_active":"2024-09-13T14:42:14.927771+0000","last_became_peered":"2024-09-13T14:42:14.927771+0000","last_unstale":"2024-09-13T14:47:23.836307+0000","last_undegraded":"2024-09-13T14:47:23.836307+0000","last_fullsized":"2024-09-13T14:47:23.836307+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:28.185949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:28.185949+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:38.363961+0000","scrub_duration":0.0018071129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"1161'12683","reported_seq":12735,"reported_epoch":1161,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:47:51.761028+0000","last_change":"2024-09-13T14:45:21.030922+0000","last_active":"2024-09-13T14:47:51.761028+0000","last_peered":"2024-09-13T14:47:51.761028+0000","last_clean":"2024-09-13T14:45:19.630214+0000","last_became_active":"2024-09-13T14:45:21.021807+0000","last_became_peered":"2024-09-13T14:45:21.021807+0000","last_unstale":"2024-09-13T14:47:51.761028+0000","last_undegraded":"2024-09-13T14:47:51.761028+0000","last_fullsized":"2024-09-13T14:47:51.761028+0000","mapping_epoch":1149,"log_start":"1142'4300","ondisk_log_start":"1142'4300","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":8383,"log_dups_size":0,"ondisk_log_size":8383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148176896,"num_objects":2261,"num_object_clones":0,"num_object_copies":6783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2261,"num_objects_unfound":0,"num_objects_dirty":2261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12683,"num_write_kb":478208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[2,6,7],"avail_no_missing":["2","6","7"],"object_location_counts":[{"shards":"2,6,7","objects":2261}],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":86,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.618905+0000","last_change":"2024-09-13T14:43:07.520176+0000","last_active":"2024-09-13T14:47:40.618905+0000","last_peered":"2024-09-13T14:47:40.618905+0000","last_clean":"2024-09-13T14:47:40.618905+0000","last_became_active":"2024-09-13T14:43:07.516748+0000","last_became_peered":"2024-09-13T14:43:07.516748+0000","last_unstale":"2024-09-13T14:47:40.618905+0000","last_undegraded":"2024-09-13T14:47:40.618905+0000","last_fullsized":"2024-09-13T14:47:40.618905+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:52:04.392821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":37,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.614436+0000","last_change":"2024-09-13T14:45:19.812353+0000","last_active":"2024-09-13T14:47:40.614436+0000","last_peered":"2024-09-13T14:47:40.614436+0000","last_clean":"2024-09-13T14:47:40.614436+0000","last_became_active":"2024-09-13T14:45:19.811743+0000","last_became_peered":"2024-09-13T14:45:19.811743+0000","last_unstale":"2024-09-13T14:47:40.614436+0000","last_undegraded":"2024-09-13T14:47:40.614436+0000","last_fullsized":"2024-09-13T14:47:40.614436+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:43.728170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"261'100","reported_seq":101,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.847959+0000","last_change":"2024-09-13T14:44:34.814157+0000","last_active":"2024-09-13T14:47:23.847959+0000","last_peered":"2024-09-13T14:47:23.847959+0000","last_clean":"2024-09-13T14:47:23.847959+0000","last_became_active":"2024-09-13T14:42:42.707803+0000","last_became_peered":"2024-09-13T14:42:42.707803+0000","last_unstale":"2024-09-13T14:47:23.847959+0000","last_undegraded":"2024-09-13T14:47:23.847959+0000","last_fullsized":"2024-09-13T14:47:23.847959+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:34.814070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:34.814070+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:38.408561+0000","scrub_duration":0.01686414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"1161'12992","reported_seq":13055,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:52.845650+0000","last_change":"2024-09-13T14:43:51.534705+0000","last_active":"2024-09-13T14:47:52.845650+0000","last_peered":"2024-09-13T14:47:52.845650+0000","last_clean":"2024-09-13T14:47:52.845650+0000","last_became_active":"2024-09-13T14:43:51.534381+0000","last_became_peered":"2024-09-13T14:43:51.534381+0000","last_unstale":"2024-09-13T14:47:52.845650+0000","last_undegraded":"2024-09-13T14:47:52.845650+0000","last_fullsized":"2024-09-13T14:47:52.845650+0000","mapping_epoch":1136,"log_start":"1147'6900","ondisk_log_start":"1147'6900","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6092,"log_dups_size":0,"ondisk_log_size":6092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:37:22.385742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139657240,"num_objects":2132,"num_object_clones":0,"num_object_copies":6396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2132,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12992,"num_write_kb":483905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1131,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.838789+0000","last_change":"2024-09-13T14:44:03.855577+0000","last_active":"2024-09-13T14:47:23.838789+0000","last_peered":"2024-09-13T14:47:23.838789+0000","last_clean":"2024-09-13T14:47:23.838789+0000","last_became_active":"2024-09-13T14:44:03.855267+0000","last_became_peered":"2024-09-13T14:44:03.855267+0000","last_unstale":"2024-09-13T14:47:23.838789+0000","last_undegraded":"2024-09-13T14:47:23.838789+0000","last_fullsized":"2024-09-13T14:47:23.838789+0000","mapping_epoch":1140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:46.148814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1102,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.838439+0000","last_change":"2024-09-13T14:29:42.545815+0000","last_active":"2024-09-13T14:47:23.838439+0000","last_peered":"2024-09-13T14:47:23.838439+0000","last_clean":"2024-09-13T14:47:23.838439+0000","last_became_active":"2024-09-13T14:29:42.545508+0000","last_became_peered":"2024-09-13T14:29:42.545508+0000","last_unstale":"2024-09-13T14:47:23.838439+0000","last_undegraded":"2024-09-13T14:47:23.838439+0000","last_fullsized":"2024-09-13T14:47:23.838439+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:39.678381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"261'100","reported_seq":104,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.617230+0000","last_change":"2024-09-13T14:43:18.988482+0000","last_active":"2024-09-13T14:47:40.617230+0000","last_peered":"2024-09-13T14:47:40.617230+0000","last_clean":"2024-09-13T14:47:40.617230+0000","last_became_active":"2024-09-13T14:43:14.556754+0000","last_became_peered":"2024-09-13T14:43:14.556754+0000","last_unstale":"2024-09-13T14:47:40.617230+0000","last_undegraded":"2024-09-13T14:47:40.617230+0000","last_fullsized":"2024-09-13T14:47:40.617230+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:18.988405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:18.988405+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:45:01.726326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"1161'12662","reported_seq":12724,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:51.781947+0000","last_change":"2024-09-13T14:43:50.796182+0000","last_active":"2024-09-13T14:47:51.781947+0000","last_peered":"2024-09-13T14:47:51.781947+0000","last_clean":"2024-09-13T14:47:51.781947+0000","last_became_active":"2024-09-13T14:43:50.795875+0000","last_became_peered":"2024-09-13T14:43:50.795875+0000","last_unstale":"2024-09-13T14:47:51.781947+0000","last_undegraded":"2024-09-13T14:47:51.781947+0000","last_fullsized":"2024-09-13T14:47:51.781947+0000","mapping_epoch":1136,"log_start":"1142'4300","ondisk_log_start":"1142'4300","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":8362,"log_dups_size":0,"ondisk_log_size":8362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:56:34.308870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138280960,"num_objects":2110,"num_object_clones":0,"num_object_copies":6330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12662,"num_write_kb":472704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1148,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.838735+0000","last_change":"2024-09-13T14:43:07.515418+0000","last_active":"2024-09-13T14:47:23.838735+0000","last_peered":"2024-09-13T14:47:23.838735+0000","last_clean":"2024-09-13T14:47:23.838735+0000","last_became_active":"2024-09-13T14:43:07.515113+0000","last_became_peered":"2024-09-13T14:43:07.515113+0000","last_unstale":"2024-09-13T14:47:23.838735+0000","last_undegraded":"2024-09-13T14:47:23.838735+0000","last_fullsized":"2024-09-13T14:47:23.838735+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:10:25.810604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1115,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.838219+0000","last_change":"2024-09-13T14:43:32.636334+0000","last_active":"2024-09-13T14:47:23.838219+0000","last_peered":"2024-09-13T14:47:23.838219+0000","last_clean":"2024-09-13T14:47:23.838219+0000","last_became_active":"2024-09-13T14:43:32.635736+0000","last_became_peered":"2024-09-13T14:43:32.635736+0000","last_unstale":"2024-09-13T14:47:23.838219+0000","last_undegraded":"2024-09-13T14:47:23.838219+0000","last_fullsized":"2024-09-13T14:47:23.838219+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:49.076014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"261'100","reported_seq":106,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.847971+0000","last_change":"2024-09-13T14:45:19.805634+0000","last_active":"2024-09-13T14:47:23.847971+0000","last_peered":"2024-09-13T14:47:23.847971+0000","last_clean":"2024-09-13T14:47:23.847971+0000","last_became_active":"2024-09-13T14:45:19.805046+0000","last_became_peered":"2024-09-13T14:45:19.805046+0000","last_unstale":"2024-09-13T14:47:23.847971+0000","last_undegraded":"2024-09-13T14:47:23.847971+0000","last_fullsized":"2024-09-13T14:47:23.847971+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:32.752408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:32.752408+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:06:53.140239+0000","scrub_duration":0.0057731559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"1161'12999","reported_seq":13066,"reported_epoch":1161,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:47:54.508335+0000","last_change":"2024-09-13T14:45:21.029184+0000","last_active":"2024-09-13T14:47:54.508335+0000","last_peered":"2024-09-13T14:47:54.508335+0000","last_clean":"2024-09-13T14:45:19.633080+0000","last_became_active":"2024-09-13T14:45:21.019809+0000","last_became_peered":"2024-09-13T14:45:21.019809+0000","last_unstale":"2024-09-13T14:47:54.508335+0000","last_undegraded":"2024-09-13T14:47:54.508335+0000","last_fullsized":"2024-09-13T14:47:54.508335+0000","mapping_epoch":1149,"log_start":"1146'6100","ondisk_log_start":"1146'6100","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6899,"log_dups_size":0,"ondisk_log_size":6899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142671872,"num_objects":2177,"num_object_clones":0,"num_object_copies":6531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2177,"num_objects_unfound":0,"num_objects_dirty":2177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12999,"num_write_kb":485632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,7,4],"avail_no_missing":["0","4","7"],"object_location_counts":[{"shards":"0,4,7","objects":2177}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1151,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.838515+0000","last_change":"2024-09-13T14:45:19.804441+0000","last_active":"2024-09-13T14:47:23.838515+0000","last_peered":"2024-09-13T14:47:23.838515+0000","last_clean":"2024-09-13T14:47:23.838515+0000","last_became_active":"2024-09-13T14:45:19.804039+0000","last_became_peered":"2024-09-13T14:45:19.804039+0000","last_unstale":"2024-09-13T14:47:23.838515+0000","last_undegraded":"2024-09-13T14:47:23.838515+0000","last_fullsized":"2024-09-13T14:47:23.838515+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:58:00.322042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":41,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.614931+0000","last_change":"2024-09-13T14:45:19.816666+0000","last_active":"2024-09-13T14:47:40.614931+0000","last_peered":"2024-09-13T14:47:40.614931+0000","last_clean":"2024-09-13T14:47:40.614931+0000","last_became_active":"2024-09-13T14:45:19.807224+0000","last_became_peered":"2024-09-13T14:45:19.807224+0000","last_unstale":"2024-09-13T14:47:40.614931+0000","last_undegraded":"2024-09-13T14:47:40.614931+0000","last_fullsized":"2024-09-13T14:47:40.614931+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:15:20.171682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"261'100","reported_seq":1291,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.835981+0000","last_change":"2024-09-13T14:45:19.798553+0000","last_active":"2024-09-13T14:47:23.835981+0000","last_peered":"2024-09-13T14:47:23.835981+0000","last_clean":"2024-09-13T14:47:23.835981+0000","last_became_active":"2024-09-13T14:45:19.798189+0000","last_became_peered":"2024-09-13T14:45:19.798189+0000","last_unstale":"2024-09-13T14:47:23.835981+0000","last_undegraded":"2024-09-13T14:47:23.835981+0000","last_fullsized":"2024-09-13T14:47:23.835981+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:29.169325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:29.169325+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:10.412666+0000","scrub_duration":0.0019358890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"1161'12884","reported_seq":12946,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:51.759468+0000","last_change":"2024-09-13T14:43:50.795029+0000","last_active":"2024-09-13T14:47:51.759468+0000","last_peered":"2024-09-13T14:47:51.759468+0000","last_clean":"2024-09-13T14:47:51.759468+0000","last_became_active":"2024-09-13T14:43:50.794715+0000","last_became_peered":"2024-09-13T14:43:50.794715+0000","last_unstale":"2024-09-13T14:47:51.759468+0000","last_undegraded":"2024-09-13T14:47:51.759468+0000","last_fullsized":"2024-09-13T14:47:51.759468+0000","mapping_epoch":1136,"log_start":"1142'4500","ondisk_log_start":"1142'4500","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":8384,"log_dups_size":0,"ondisk_log_size":8384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:21.257279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145752064,"num_objects":2224,"num_object_clones":0,"num_object_copies":6672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12884,"num_write_kb":483456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1705,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.599684+0000","last_change":"2024-09-13T14:43:07.515572+0000","last_active":"2024-09-13T14:47:40.599684+0000","last_peered":"2024-09-13T14:47:40.599684+0000","last_clean":"2024-09-13T14:47:40.599684+0000","last_became_active":"2024-09-13T14:43:07.515231+0000","last_became_peered":"2024-09-13T14:43:07.515231+0000","last_unstale":"2024-09-13T14:47:40.599684+0000","last_undegraded":"2024-09-13T14:47:40.599684+0000","last_fullsized":"2024-09-13T14:47:40.599684+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:57.966380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1144,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.848214+0000","last_change":"2024-09-13T14:45:19.804982+0000","last_active":"2024-09-13T14:47:23.848214+0000","last_peered":"2024-09-13T14:47:23.848214+0000","last_clean":"2024-09-13T14:47:23.848214+0000","last_became_active":"2024-09-13T14:45:19.804663+0000","last_became_peered":"2024-09-13T14:45:19.804663+0000","last_unstale":"2024-09-13T14:47:23.848214+0000","last_undegraded":"2024-09-13T14:47:23.848214+0000","last_fullsized":"2024-09-13T14:47:23.848214+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:43.211479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"261'100","reported_seq":381,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.838706+0000","last_change":"2024-09-13T14:43:42.881584+0000","last_active":"2024-09-13T14:47:23.838706+0000","last_peered":"2024-09-13T14:47:23.838706+0000","last_clean":"2024-09-13T14:47:23.838706+0000","last_became_active":"2024-09-13T14:43:14.562161+0000","last_became_peered":"2024-09-13T14:43:14.562161+0000","last_unstale":"2024-09-13T14:47:23.838706+0000","last_undegraded":"2024-09-13T14:47:23.838706+0000","last_fullsized":"2024-09-13T14:47:23.838706+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:42.881520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:42.881520+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:06:50.353442+0000","scrub_duration":0.0013804310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"1161'12998","reported_seq":13060,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:54.451227+0000","last_change":"2024-09-13T14:43:50.793879+0000","last_active":"2024-09-13T14:47:54.451227+0000","last_peered":"2024-09-13T14:47:54.451227+0000","last_clean":"2024-09-13T14:47:54.451227+0000","last_became_active":"2024-09-13T14:43:50.793571+0000","last_became_peered":"2024-09-13T14:43:50.793571+0000","last_unstale":"2024-09-13T14:47:54.451227+0000","last_undegraded":"2024-09-13T14:47:54.451227+0000","last_fullsized":"2024-09-13T14:47:54.451227+0000","mapping_epoch":1136,"log_start":"1146'6100","ondisk_log_start":"1146'6100","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6898,"log_dups_size":0,"ondisk_log_size":6898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:22.880360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144965632,"num_objects":2212,"num_object_clones":0,"num_object_copies":6636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12998,"num_write_kb":486720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1117,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.862455+0000","last_change":"2024-09-13T14:43:07.518120+0000","last_active":"2024-09-13T14:47:23.862455+0000","last_peered":"2024-09-13T14:47:23.862455+0000","last_clean":"2024-09-13T14:47:23.862455+0000","last_became_active":"2024-09-13T14:43:07.517361+0000","last_became_peered":"2024-09-13T14:43:07.517361+0000","last_unstale":"2024-09-13T14:47:23.862455+0000","last_undegraded":"2024-09-13T14:47:23.862455+0000","last_fullsized":"2024-09-13T14:47:23.862455+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:39.366473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":1099,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.836898+0000","last_change":"2024-09-13T14:19:25.871006+0000","last_active":"2024-09-13T14:47:23.836898+0000","last_peered":"2024-09-13T14:47:23.836898+0000","last_clean":"2024-09-13T14:47:23.836898+0000","last_became_active":"2024-09-13T14:19:25.870713+0000","last_became_peered":"2024-09-13T14:19:25.870713+0000","last_unstale":"2024-09-13T14:47:23.836898+0000","last_undegraded":"2024-09-13T14:47:23.836898+0000","last_fullsized":"2024-09-13T14:47:23.836898+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:34.372124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"261'100","reported_seq":365,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.848431+0000","last_change":"2024-09-13T14:43:32.637553+0000","last_active":"2024-09-13T14:47:23.848431+0000","last_peered":"2024-09-13T14:47:23.848431+0000","last_clean":"2024-09-13T14:47:23.848431+0000","last_became_active":"2024-09-13T14:43:32.636545+0000","last_became_peered":"2024-09-13T14:43:32.636545+0000","last_unstale":"2024-09-13T14:47:23.848431+0000","last_undegraded":"2024-09-13T14:47:23.848431+0000","last_fullsized":"2024-09-13T14:47:23.848431+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:15:42.069756+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"1161'12783","reported_seq":12843,"reported_epoch":1161,"state":"active+recovery_wait+remapped","last_fresh":"2024-09-13T14:47:52.809888+0000","last_change":"2024-09-13T14:45:21.130410+0000","last_active":"2024-09-13T14:47:52.809888+0000","last_peered":"2024-09-13T14:47:52.809888+0000","last_clean":"2024-09-13T14:45:19.587556+0000","last_became_active":"2024-09-13T14:45:21.100167+0000","last_became_peered":"2024-09-13T14:45:21.100167+0000","last_unstale":"2024-09-13T14:47:52.809888+0000","last_undegraded":"2024-09-13T14:47:52.809888+0000","last_fullsized":"2024-09-13T14:47:52.809888+0000","mapping_epoch":1149,"log_start":"1147'6777","ondisk_log_start":"1147'6777","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6006,"log_dups_size":0,"ondisk_log_size":6006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146341888,"num_objects":2233,"num_object_clones":0,"num_object_copies":6699,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7077,"num_objects_unfound":0,"num_objects_dirty":2233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12783,"num_write_kb":480512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[1,2,5],"avail_no_missing":["1","2","5"],"object_location_counts":[{"shards":"1,2,5","objects":-4844}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1168,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.838358+0000","last_change":"2024-09-13T14:45:19.804868+0000","last_active":"2024-09-13T14:47:23.838358+0000","last_peered":"2024-09-13T14:47:23.838358+0000","last_clean":"2024-09-13T14:47:23.838358+0000","last_became_active":"2024-09-13T14:45:19.804520+0000","last_became_peered":"2024-09-13T14:45:19.804520+0000","last_unstale":"2024-09-13T14:47:23.838358+0000","last_undegraded":"2024-09-13T14:47:23.838358+0000","last_fullsized":"2024-09-13T14:47:23.838358+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:17:53.924629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":1145,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.837028+0000","last_change":"2024-09-13T14:43:07.533955+0000","last_active":"2024-09-13T14:47:23.837028+0000","last_peered":"2024-09-13T14:47:23.837028+0000","last_clean":"2024-09-13T14:47:23.837028+0000","last_became_active":"2024-09-13T14:43:07.516816+0000","last_became_peered":"2024-09-13T14:43:07.516816+0000","last_unstale":"2024-09-13T14:47:23.837028+0000","last_undegraded":"2024-09-13T14:47:23.837028+0000","last_fullsized":"2024-09-13T14:47:23.837028+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:26.232184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"261'100","reported_seq":41,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.613196+0000","last_change":"2024-09-13T14:45:19.821649+0000","last_active":"2024-09-13T14:47:40.613196+0000","last_peered":"2024-09-13T14:47:40.613196+0000","last_clean":"2024-09-13T14:47:40.613196+0000","last_became_active":"2024-09-13T14:45:19.812346+0000","last_became_peered":"2024-09-13T14:45:19.812346+0000","last_unstale":"2024-09-13T14:47:40.613196+0000","last_undegraded":"2024-09-13T14:47:40.613196+0000","last_fullsized":"2024-09-13T14:47:40.613196+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.597568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.597568+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:45.218892+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"1161'12857","reported_seq":12919,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:52.670971+0000","last_change":"2024-09-13T14:43:51.537117+0000","last_active":"2024-09-13T14:47:52.670971+0000","last_peered":"2024-09-13T14:47:52.670971+0000","last_clean":"2024-09-13T14:47:52.670971+0000","last_became_active":"2024-09-13T14:43:51.536532+0000","last_became_peered":"2024-09-13T14:43:51.536532+0000","last_unstale":"2024-09-13T14:47:52.670971+0000","last_undegraded":"2024-09-13T14:47:52.670971+0000","last_fullsized":"2024-09-13T14:47:52.670971+0000","mapping_epoch":1136,"log_start":"1147'6800","ondisk_log_start":"1147'6800","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6057,"log_dups_size":0,"ondisk_log_size":6057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:41.713211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138215424,"num_objects":2109,"num_object_clones":0,"num_object_copies":6327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12857,"num_write_kb":478912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1086,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.848157+0000","last_change":"2024-09-13T14:19:26.887839+0000","last_active":"2024-09-13T14:47:23.848157+0000","last_peered":"2024-09-13T14:47:23.848157+0000","last_clean":"2024-09-13T14:47:23.848157+0000","last_became_active":"2024-09-13T14:19:26.887498+0000","last_became_peered":"2024-09-13T14:19:26.887498+0000","last_unstale":"2024-09-13T14:47:23.848157+0000","last_undegraded":"2024-09-13T14:47:23.848157+0000","last_fullsized":"2024-09-13T14:47:23.848157+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:24.475263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":1116,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.836745+0000","last_change":"2024-09-13T14:28:35.608607+0000","last_active":"2024-09-13T14:47:23.836745+0000","last_peered":"2024-09-13T14:47:23.836745+0000","last_clean":"2024-09-13T14:47:23.836745+0000","last_became_active":"2024-09-13T14:28:35.608254+0000","last_became_peered":"2024-09-13T14:28:35.608254+0000","last_unstale":"2024-09-13T14:47:23.836745+0000","last_undegraded":"2024-09-13T14:47:23.836745+0000","last_fullsized":"2024-09-13T14:47:23.836745+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:24.019173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"261'100","reported_seq":52,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.836412+0000","last_change":"2024-09-13T14:44:31.191431+0000","last_active":"2024-09-13T14:47:23.836412+0000","last_peered":"2024-09-13T14:47:23.836412+0000","last_clean":"2024-09-13T14:47:23.836412+0000","last_became_active":"2024-09-13T14:43:14.557376+0000","last_became_peered":"2024-09-13T14:43:14.557376+0000","last_unstale":"2024-09-13T14:47:23.836412+0000","last_undegraded":"2024-09-13T14:47:23.836412+0000","last_fullsized":"2024-09-13T14:47:23.836412+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.191326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.191326+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:15:27.352698+0000","scrub_duration":0.0025317360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1106,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.862454+0000","last_change":"2024-09-13T14:28:35.603724+0000","last_active":"2024-09-13T14:47:23.862454+0000","last_peered":"2024-09-13T14:47:23.862454+0000","last_clean":"2024-09-13T14:47:23.862454+0000","last_became_active":"2024-09-13T14:28:35.603316+0000","last_became_peered":"2024-09-13T14:28:35.603316+0000","last_unstale":"2024-09-13T14:47:23.862454+0000","last_undegraded":"2024-09-13T14:47:23.862454+0000","last_fullsized":"2024-09-13T14:47:23.862454+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:47.668223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":37,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.616603+0000","last_change":"2024-09-13T14:45:19.811466+0000","last_active":"2024-09-13T14:47:40.616603+0000","last_peered":"2024-09-13T14:47:40.616603+0000","last_clean":"2024-09-13T14:47:40.616603+0000","last_became_active":"2024-09-13T14:45:19.810617+0000","last_became_peered":"2024-09-13T14:45:19.810617+0000","last_unstale":"2024-09-13T14:47:40.616603+0000","last_undegraded":"2024-09-13T14:47:40.616603+0000","last_fullsized":"2024-09-13T14:47:40.616603+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:16.246669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"261'100","reported_seq":52,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.848145+0000","last_change":"2024-09-13T14:44:50.805306+0000","last_active":"2024-09-13T14:47:23.848145+0000","last_peered":"2024-09-13T14:47:23.848145+0000","last_clean":"2024-09-13T14:47:23.848145+0000","last_became_active":"2024-09-13T14:43:14.562922+0000","last_became_peered":"2024-09-13T14:43:14.562922+0000","last_unstale":"2024-09-13T14:47:23.848145+0000","last_undegraded":"2024-09-13T14:47:23.848145+0000","last_fullsized":"2024-09-13T14:47:23.848145+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:50.805210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:50.805210+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:28:38.529599+0000","scrub_duration":0.0042900009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1098,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.836165+0000","last_change":"2024-09-13T14:19:25.873088+0000","last_active":"2024-09-13T14:47:23.836165+0000","last_peered":"2024-09-13T14:47:23.836165+0000","last_clean":"2024-09-13T14:47:23.836165+0000","last_became_active":"2024-09-13T14:19:25.872781+0000","last_became_peered":"2024-09-13T14:19:25.872781+0000","last_unstale":"2024-09-13T14:47:23.836165+0000","last_undegraded":"2024-09-13T14:47:23.836165+0000","last_fullsized":"2024-09-13T14:47:23.836165+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:05.607382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1705,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.607393+0000","last_change":"2024-09-13T14:45:19.813876+0000","last_active":"2024-09-13T14:47:40.607393+0000","last_peered":"2024-09-13T14:47:40.607393+0000","last_clean":"2024-09-13T14:47:40.607393+0000","last_became_active":"2024-09-13T14:45:19.809136+0000","last_became_peered":"2024-09-13T14:45:19.809136+0000","last_unstale":"2024-09-13T14:47:40.607393+0000","last_undegraded":"2024-09-13T14:47:40.607393+0000","last_fullsized":"2024-09-13T14:47:40.607393+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:50.013528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"261'100","reported_seq":111,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.617413+0000","last_change":"2024-09-13T14:45:19.810654+0000","last_active":"2024-09-13T14:47:40.617413+0000","last_peered":"2024-09-13T14:47:40.617413+0000","last_clean":"2024-09-13T14:47:40.617413+0000","last_became_active":"2024-09-13T14:45:19.810066+0000","last_became_peered":"2024-09-13T14:45:19.810066+0000","last_unstale":"2024-09-13T14:47:40.617413+0000","last_undegraded":"2024-09-13T14:47:40.617413+0000","last_fullsized":"2024-09-13T14:47:40.617413+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:19.978464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:19.978464+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:33.639332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1752,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.599256+0000","last_change":"2024-09-13T14:45:19.809166+0000","last_active":"2024-09-13T14:47:40.599256+0000","last_peered":"2024-09-13T14:47:40.599256+0000","last_clean":"2024-09-13T14:47:40.599256+0000","last_became_active":"2024-09-13T14:45:19.808631+0000","last_became_peered":"2024-09-13T14:45:19.808631+0000","last_unstale":"2024-09-13T14:47:40.599256+0000","last_undegraded":"2024-09-13T14:47:40.599256+0000","last_fullsized":"2024-09-13T14:47:40.599256+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:38.822245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1123,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.848375+0000","last_change":"2024-09-13T14:45:19.804229+0000","last_active":"2024-09-13T14:47:23.848375+0000","last_peered":"2024-09-13T14:47:23.848375+0000","last_clean":"2024-09-13T14:47:23.848375+0000","last_became_active":"2024-09-13T14:45:19.803929+0000","last_became_peered":"2024-09-13T14:45:19.803929+0000","last_unstale":"2024-09-13T14:47:23.848375+0000","last_undegraded":"2024-09-13T14:47:23.848375+0000","last_fullsized":"2024-09-13T14:47:23.848375+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:33:53.675568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"261'100","reported_seq":52,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.847885+0000","last_change":"2024-09-13T14:44:42.654506+0000","last_active":"2024-09-13T14:47:23.847885+0000","last_peered":"2024-09-13T14:47:23.847885+0000","last_clean":"2024-09-13T14:47:23.847885+0000","last_became_active":"2024-09-13T14:43:14.563451+0000","last_became_peered":"2024-09-13T14:43:14.563451+0000","last_unstale":"2024-09-13T14:47:23.847885+0000","last_undegraded":"2024-09-13T14:47:23.847885+0000","last_fullsized":"2024-09-13T14:47:23.847885+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:42.654455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:42.654455+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:57:32.251597+0000","scrub_duration":0.001494097,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":1117,"reported_epoch":1160,"state":"active+clean","last_fresh":"2024-09-13T14:47:23.836257+0000","last_change":"2024-09-13T14:29:42.541093+0000","last_active":"2024-09-13T14:47:23.836257+0000","last_peered":"2024-09-13T14:47:23.836257+0000","last_clean":"2024-09-13T14:47:23.836257+0000","last_became_active":"2024-09-13T14:29:42.540671+0000","last_became_peered":"2024-09-13T14:29:42.540671+0000","last_unstale":"2024-09-13T14:47:23.836257+0000","last_undegraded":"2024-09-13T14:47:23.836257+0000","last_fullsized":"2024-09-13T14:47:23.836257+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:45.722376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1643,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.613480+0000","last_change":"2024-09-13T14:43:07.518679+0000","last_active":"2024-09-13T14:47:40.613480+0000","last_peered":"2024-09-13T14:47:40.613480+0000","last_clean":"2024-09-13T14:47:40.613480+0000","last_became_active":"2024-09-13T14:43:07.518333+0000","last_became_peered":"2024-09-13T14:43:07.518333+0000","last_unstale":"2024-09-13T14:47:40.613480+0000","last_undegraded":"2024-09-13T14:47:40.613480+0000","last_fullsized":"2024-09-13T14:47:40.613480+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:20.616274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"261'100","reported_seq":413,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.607447+0000","last_change":"2024-09-13T14:45:19.809268+0000","last_active":"2024-09-13T14:47:40.607447+0000","last_peered":"2024-09-13T14:47:40.607447+0000","last_clean":"2024-09-13T14:47:40.607447+0000","last_became_active":"2024-09-13T14:45:19.808772+0000","last_became_peered":"2024-09-13T14:45:19.808772+0000","last_unstale":"2024-09-13T14:47:40.607447+0000","last_undegraded":"2024-09-13T14:47:40.607447+0000","last_fullsized":"2024-09-13T14:47:40.607447+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:14.654603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:14.654603+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:40.964553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2278162456,"num_objects":34763,"num_object_clones":0,"num_object_copies":104289,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3287,"num_objects_misplaced":24055,"num_objects_unfound":0,"num_objects_dirty":34763,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":206155,"num_write_kb":7709313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2076,"num_bytes_recovered":136052736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7236300800,"data_stored":7236288584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112178,"ondisk_log_size":112178,"up":48,"acting":47,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":354515,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430574,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1168612,"kb_used_data":862940,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92576540,"statfs":{"total":95995035648,"available":94798376960,"internally_reserved":0,"allocated":883650560,"data_stored":880090177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626803,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1463196,"kb_used_data":1153364,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":92281956,"statfs":{"total":95995035648,"available":94496722944,"internally_reserved":0,"allocated":1181044736,"data_stored":1177491906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823032,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1012692,"kb_used_data":707020,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92732460,"statfs":{"total":95995035648,"available":94958039040,"internally_reserved":0,"allocated":723988480,"data_stored":720440180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019260,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1184668,"kb_used_data":874836,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":92560484,"statfs":{"total":95995035648,"available":94781935616,"internally_reserved":0,"allocated":895832064,"data_stored":892279259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059592918,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":245832,"kb_used_data":67456,"kb_used_omap":26,"kb_used_meta":178277,"kb_avail":93499320,"statfs":{"total":95995035648,"available":95743303680,"internally_reserved":0,"allocated":69074944,"data_stored":65537839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":182555808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":591,"seq":2538325672156,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1538576,"kb_used_data":1149256,"kb_used_omap":41,"kb_used_meta":389270,"kb_avail":92206576,"statfs":{"total":95995035648,"available":94419533824,"internally_reserved":0,"allocated":1176838144,"data_stored":1173293913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42651,"internal_metadata":398612837},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":557,"seq":2392296784099,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1647820,"kb_used_data":1286596,"kb_used_omap":32,"kb_used_meta":361183,"kb_avail":92097332,"statfs":{"total":95995035648,"available":94307667968,"internally_reserved":0,"allocated":1317474304,"data_stored":1313933072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33752,"internal_metadata":369851432},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":524,"seq":2250562863338,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1414352,"kb_used_data":1098440,"kb_used_omap":41,"kb_used_meta":315862,"kb_avail":92330800,"statfs":{"total":95995035648,"available":94546739200,"internally_reserved":0,"allocated":1124802560,"data_stored":1121257813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42694,"internal_metadata":323443002},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1104547840,"data_stored":1104543768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1288114176,"data_stored":1288110104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1164050432,"data_stored":1164050432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":58195968,"data_stored":58195968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":872480768,"data_stored":872480768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":710672384,"data_stored":710672384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1169231872,"data_stored":1169227800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":869007360,"data_stored":869007360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:47:59.730 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-13T14:47:59.764 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:59.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:59.813 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:47:59.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:59.914 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:59.939 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:47:59.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:59.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:00.041 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:00.064 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:00.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:00.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:00.166 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:00.216 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:00.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:00.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:00.319 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:00.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:00.375 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:00.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:00.382 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:00.468 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:00.475 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:00.537 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:00.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:00.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:00.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:48:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:48:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:48:00.641 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:00.664 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:00.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:00.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:00.766 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:00.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:00.823 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:00.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:00.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:00 smithi151 ceph-mon[104450]: pgmap v1624: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.0 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 629 op/s; 3390/97872 objects degraded (3.464%); 23074/97872 objects misplaced (23.576%); 874 KiB/s, 13 objects/s recovering 2024-09-13T14:48:00.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:00 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 3390/97872 objects degraded (3.464%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:00.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:00 smithi151 ceph-mon[108966]: pgmap v1624: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.0 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 629 op/s; 3390/97872 objects degraded (3.464%); 23074/97872 objects misplaced (23.576%); 874 KiB/s, 13 objects/s recovering 2024-09-13T14:48:00.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:00 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 3390/97872 objects degraded (3.464%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:00.924 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:00.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:00.969 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:00.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:01.018 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:00 smithi184 ceph-mon[92527]: pgmap v1624: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 2.0 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 629 op/s; 3390/97872 objects degraded (3.464%); 23074/97872 objects misplaced (23.576%); 874 KiB/s, 13 objects/s recovering 2024-09-13T14:48:01.018 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:00 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 3390/97872 objects degraded (3.464%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:01.069 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:01.095 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:01.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:01.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:01.196 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:01.219 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:01.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:01.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:01.321 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:01.345 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:01.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:01.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:01.447 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:01.471 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:01.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:01.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:01.572 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:01.600 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:01.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:01.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:01.636 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:48:01.636 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:48:01.702 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:01.727 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:01.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:01.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:01.750 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-13T14:48:01.829 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:01.868 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:01.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:01.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:01.961 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:01 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1160196739' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:48:01.961 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:01 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:01.969 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:02.003 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:02.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:02.105 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:02.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:01 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1160196739' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:48:02.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:02.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:01 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1160196739' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:48:02.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:01 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:02.130 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:02.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:02.232 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:02.275 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:02.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:02.318 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T14:48:02.377 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:02.415 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:02.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:02.517 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:02.546 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:02.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:02.632 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:48:02.648 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:02.685 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:02.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:02.787 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:02.812 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:02.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:02.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:02 smithi151 ceph-mon[108966]: pgmap v1625: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.9 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 596 op/s; 3390/94824 objects degraded (3.575%); 22481/94824 objects misplaced (23.708%); 874 KiB/s, 13 objects/s recovering 2024-09-13T14:48:02.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:02 smithi151 ceph-mon[108966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:02.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:02 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1160196739' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:48:02.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:02 smithi151 ceph-mon[108966]: osdmap e1162: 8 total, 8 up, 8 in 2024-09-13T14:48:02.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:02 smithi151 ceph-mon[104450]: pgmap v1625: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.9 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 596 op/s; 3390/94824 objects degraded (3.575%); 22481/94824 objects misplaced (23.708%); 874 KiB/s, 13 objects/s recovering 2024-09-13T14:48:02.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:02 smithi151 ceph-mon[104450]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:02.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:02 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1160196739' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:48:02.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:02 smithi151 ceph-mon[104450]: osdmap e1162: 8 total, 8 up, 8 in 2024-09-13T14:48:02.914 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:02.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.947 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:02.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:03.048 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:03.066 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:02 smithi184 ceph-mon[92527]: pgmap v1625: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.9 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 596 op/s; 3390/94824 objects degraded (3.575%); 22481/94824 objects misplaced (23.708%); 874 KiB/s, 13 objects/s recovering 2024-09-13T14:48:03.066 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:02 smithi184 ceph-mon[92527]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:03.066 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:02 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1160196739' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:48:03.066 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:02 smithi184 ceph-mon[92527]: osdmap e1162: 8 total, 8 up, 8 in 2024-09-13T14:48:03.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:03.099 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:03.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:03.200 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:03.224 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:03.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:03.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:03.326 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:03.359 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:03.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:03.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:03.461 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:03.499 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:03.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:03.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:03.601 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:03.637 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:03.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:03.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:03.739 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:03.763 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:03.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:03.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:03.865 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:03.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:03.893 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:03.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:03.994 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:04.046 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:04.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:04.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:04.148 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:04.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:04.189 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:04.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:04.290 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:04.317 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:04.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:04.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:04.419 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:04.431 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:48:04.431 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:48:04.462 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:04.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:04.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:04.564 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:04.598 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:04.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:04.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:04.700 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:04.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:04.737 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:04.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:04.838 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:04.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:04 smithi151 ceph-mon[104450]: pgmap v1627: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.8 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 617 op/s; 3474/88581 objects degraded (3.922%); 21500/88581 objects misplaced (24.272%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:48:04.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:48:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:48:04] "GET /metrics HTTP/1.1" 200 36656 "" "Prometheus/2.51.0" 2024-09-13T14:48:04.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:04 smithi151 ceph-mon[108966]: pgmap v1627: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.8 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 617 op/s; 3474/88581 objects degraded (3.922%); 21500/88581 objects misplaced (24.272%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:48:04.880 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:04.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:04.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:04.982 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:05.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:05.007 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:05.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:05.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:04 smithi184 ceph-mon[92527]: pgmap v1627: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.8 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 617 op/s; 3474/88581 objects degraded (3.922%); 21500/88581 objects misplaced (24.272%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:48:05.108 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:05.143 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:05.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:05.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:05.187 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T14:48:05.246 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:05.278 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:05.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:05.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:05.380 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:05.405 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:05.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:05.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:05.490 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:48:05.507 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:05.534 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:05.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:05.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:05.636 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:05.658 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:48:05.658 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:48:05.660 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:05.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:05.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:05.701 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:48:05.702 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:48:05.738 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:48:05.738 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:48:05.763 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:05.777 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:48:05.777 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:48:05.788 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:05.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:05.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:05.793 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:48:05.793 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:48:05.825 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:48:05.826 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:48:05.853 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:48:05.853 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:48:05.884 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:48:05.885 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:48:05.890 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:05.935 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:05.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:05.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:05.937 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:05 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 3474/88581 objects degraded (3.922%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:05.937 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:05 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1063935185' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:06.037 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:06.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:06.081 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:06.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:06.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:05 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 3474/88581 objects degraded (3.922%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:06.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:05 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1063935185' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:06.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:05 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 3474/88581 objects degraded (3.922%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:06.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:05 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1063935185' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:06.182 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:06.207 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:06.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:06.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:06.309 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:06.330 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:06.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:06.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:06.432 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:06.454 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:06.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:06.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:06.555 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:06.578 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:06.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:06.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:06.680 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:06.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:06.704 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:06.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:06.711 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:48:06.804 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:06.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:06.846 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:06.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:06.947 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:06.971 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:06.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:06.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:07.015 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:06 smithi151 ceph-mon[104450]: pgmap v1628: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 622 op/s; 3474/85671 objects degraded (4.055%); 20921/85671 objects misplaced (24.420%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:48:07.015 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:06 smithi151 ceph-mon[108966]: pgmap v1628: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 622 op/s; 3474/85671 objects degraded (4.055%); 20921/85671 objects misplaced (24.420%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:48:07.065 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:06 smithi184 ceph-mon[92527]: pgmap v1628: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 622 op/s; 3474/85671 objects degraded (4.055%); 20921/85671 objects misplaced (24.420%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:48:07.073 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:07.121 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:07.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:07.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:07.223 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:07.257 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:07.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:07.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:07.338 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:48:07.338 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [], 2024-09-13T14:48:07.338 INFO:teuthology.orchestra.run.smithi184.stdout: "num_ops": 0 2024-09-13T14:48:07.338 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:48:07.361 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:07.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:07.404 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:07.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:07.494 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:48:07.494 INFO:teuthology.orchestra.run.smithi151.stdout:{"fsid":"2401fc50-71da-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3474/85671 objects degraded (4.055%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1268,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":1162,"num_osds":8,"num_up_osds":8,"osd_up_since":1726237814,"num_in_osds":8,"osd_in_since":1726238586,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":110},{"state_name":"active+remapped+backfill_wait","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovery_wait+remapped","count":1}],"num_pgs":119,"num_pools":5,"num_objects":28557,"data_bytes":1870364856,"bytes_used":8744026112,"bytes_avail":759216259072,"bytes_total":767960285184,"degraded_objects":3474,"degraded_total":85671,"degraded_ratio":0.040550477991385649,"misplaced_objects":20921,"misplaced_total":85671,"misplaced_ratio":0.24420165516919379,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":1146419,"recovering_keys_per_sec":0,"num_objects_recovered":175,"num_bytes_recovered":11468800,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":622},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi151.rqqklw","status":"up:active","gid":24614}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.151:8443/","prometheus":"http://172.21.15.151:9283/"}},"servicemap":{"epoch":24,"modified":"2024-09-13T14:45:41.438774+0000","services":{}},"progress_events":{"5a616d1f-cba6-436c-9b6b-5733d6f6b5ec":{"message":"Global Recovery Event (0s)\n [............................] ","progress":0,"add_to_ceph_s":true},"b24a386d-2177-4b0f-a83d-e6638be369da":{"message":"Upgrade to 19.3.0-4697-gd58f7306 (4m)\n [============................] (remaining: 6m)","progress":0.44999998807907104,"add_to_ceph_s":true}}} 2024-09-13T14:48:07.505 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:07.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:07.545 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:07.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:07.647 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:07.669 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:07.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:07.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:07.770 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:07.792 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:07.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:07.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:07.893 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:07.919 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:07.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:07.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:08.013 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:07 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2066866434' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:48:08.021 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:08.042 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T14:48:08.062 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:08.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:08.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:08.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:07 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2066866434' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:48:08.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:07 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2066866434' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:48:08.164 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:08.203 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:08.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:08.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:08.304 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:08.327 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:08.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:08.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:08.352 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:48:08.430 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:08.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:08.474 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:08.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:08.575 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:08.623 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:08.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:08.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:08.728 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:08.766 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:08.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:08.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:08.868 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:08.892 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:08.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:08.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:08.994 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:09.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:09.030 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:09.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:09.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:08 smithi184 ceph-mon[92527]: pgmap v1629: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 615 op/s; 3567/83487 objects degraded (4.273%); 20567/83487 objects misplaced (24.635%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:48:09.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:08 smithi151 ceph-mon[104450]: pgmap v1629: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 615 op/s; 3567/83487 objects degraded (4.273%); 20567/83487 objects misplaced (24.635%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:48:09.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:08 smithi151 ceph-mon[108966]: pgmap v1629: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 615 op/s; 3567/83487 objects degraded (4.273%); 20567/83487 objects misplaced (24.635%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:48:09.131 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:09.195 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:09.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:09.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:09.297 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:09.321 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:09.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:09.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:09.423 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:09.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:09.447 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:09.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:09.548 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:09.576 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:09.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:09.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:09.679 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:09.700 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:09.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:09.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:09.801 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:09.833 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:09.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:09.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:09.935 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:09.960 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:09.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:09.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:10.061 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:10.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:10.109 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:10.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:10.210 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:10.234 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:10.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:10.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:10.294 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:48:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:48:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:48:10.336 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:10.362 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:10.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:10.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:10.410 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:48:10.410 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [], 2024-09-13T14:48:10.410 INFO:teuthology.orchestra.run.smithi184.stdout: "complaint_time": 30, 2024-09-13T14:48:10.410 INFO:teuthology.orchestra.run.smithi184.stdout: "num_blocked_ops": 0 2024-09-13T14:48:10.410 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:48:10.464 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:10.508 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:10.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:10.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:10.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:10.651 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:10.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:10.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:10.712 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T14:48:10.712 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:48:10.753 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:10.793 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:10.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:10.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:10.899 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:10.921 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:10.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:10.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:11.024 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:11.081 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:11.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:11.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:11.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:10 smithi184 ceph-mon[92527]: pgmap v1630: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.6 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 600 op/s; 3567/79851 objects degraded (4.467%); 20026/79851 objects misplaced (25.079%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:48:11.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:10 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 3567/79851 objects degraded (4.467%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:11.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:10 smithi151 ceph-mon[104450]: pgmap v1630: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.6 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 600 op/s; 3567/79851 objects degraded (4.467%); 20026/79851 objects misplaced (25.079%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:48:11.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:10 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 3567/79851 objects degraded (4.467%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:11.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:10 smithi151 ceph-mon[108966]: pgmap v1630: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.6 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 600 op/s; 3567/79851 objects degraded (4.467%); 20026/79851 objects misplaced (25.079%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:48:11.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:10 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 3567/79851 objects degraded (4.467%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:11.183 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:11.208 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:11.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:11.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:11.310 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:11.333 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:11.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:11.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:11.435 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:11.456 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:11.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:11.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:11.476 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:11.512 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:11.527 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T14:48:11.559 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:11.593 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:11.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:11.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:11.694 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:11.718 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:11.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:11.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:11.820 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:11.836 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:48:11.849 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:11.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:11.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:11.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:11.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:12.000 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:12.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:12.101 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:12.141 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:12.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:12.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:12.242 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:12.281 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:12.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:12.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:12.383 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:12.407 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:12.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:12.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:12.510 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:12.532 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:12.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:12.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:12.634 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:12.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:12.680 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:12.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:12.781 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:12.807 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:12.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:12.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:12.902 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:12 smithi151 ceph-mon[104450]: pgmap v1631: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.6 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 585 op/s; 3567/77244 objects degraded (4.618%); 19516/77244 objects misplaced (25.265%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:48:12.902 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:12 smithi151 ceph-mon[108966]: pgmap v1631: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.6 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 585 op/s; 3567/77244 objects degraded (4.618%); 19516/77244 objects misplaced (25.265%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:48:12.909 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:12.956 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:12.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:12.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:13.058 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:13.079 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:13.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:13.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:13.101 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:12 smithi184 ceph-mon[92527]: pgmap v1631: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.6 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 585 op/s; 3567/77244 objects degraded (4.618%); 19516/77244 objects misplaced (25.265%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:48:13.181 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:13.202 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:13.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:13.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:13.305 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:13.326 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:13.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:13.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:13.343 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:48:13.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:13.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:13.507 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:13.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:13.609 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:13.636 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:13.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:13.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:13.687 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:48:13.687 INFO:teuthology.orchestra.run.smithi184.stdout: "size": 20, 2024-09-13T14:48:13.687 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 600, 2024-09-13T14:48:13.687 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [ 2024-09-13T14:48:13.688 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.688 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "MOSDPGPush(6.b 1105/1103 [PushOp(6:dd9c17a0:::benchmark_data_smithi151_155836_object33785:head, version: 1087'2157, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:dd9c17a0:::benchmark_data_smithi151_155836_object33785:head@1087'2157, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:48:13.688 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:38.100986+0000", 2024-09-13T14:48:13.688 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 395.58344248600002, 2024-09-13T14:48:13.688 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.27630356299999997, 2024-09-13T14:48:13.688 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:48:13.688 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:48:13.688 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:48:13.688 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.688 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:48:13.688 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.100986+0000", 2024-09-13T14:48:13.688 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.689 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.689 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.689 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:48:13.689 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.100986+0000", 2024-09-13T14:48:13.689 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.689 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.689 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.689 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:48:13.689 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.100985+0000", 2024-09-13T14:48:13.689 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:48:13.690 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.690 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.690 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:48:13.690 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.101020+0000", 2024-09-13T14:48:13.690 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.4418000000000001e-05 2024-09-13T14:48:13.690 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.690 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.690 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:48:13.690 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.101033+0000", 2024-09-13T14:48:13.690 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2979000000000001e-05 2024-09-13T14:48:13.690 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.690 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.691 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:48:13.691 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.101040+0000", 2024-09-13T14:48:13.691 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.6699999999999994e-06 2024-09-13T14:48:13.691 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.691 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.691 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:48:13.691 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.372854+0000", 2024-09-13T14:48:13.691 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.27181395800000002 2024-09-13T14:48:13.691 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.691 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.691 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:48:13.691 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.372935+0000", 2024-09-13T14:48:13.691 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0539999999999998e-05 2024-09-13T14:48:13.692 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.692 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.692 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:48:13.692 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:38.377289+0000", 2024-09-13T14:48:13.692 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0043543879999999998 2024-09-13T14:48:13.692 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.692 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:48:13.692 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.692 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.692 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.692 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "MOSDPGPush(6.b 1113/1103 [PushOp(6:d09715c7:::benchmark_data_smithi151_155836_object50640:head, version: 1088'3227, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:d09715c7:::benchmark_data_smithi151_155836_object50640:head@1088'3227, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:48:13.692 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:59.200022+0000", 2024-09-13T14:48:13.692 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 374.484405923, 2024-09-13T14:48:13.692 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.31017858599999998, 2024-09-13T14:48:13.693 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:48:13.693 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:48:13.693 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:48:13.693 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.693 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:48:13.693 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:59.200022+0000", 2024-09-13T14:48:13.693 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.693 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.693 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.693 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:48:13.693 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:59.200022+0000", 2024-09-13T14:48:13.693 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.694 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.694 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.694 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:48:13.694 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:59.200021+0000", 2024-09-13T14:48:13.694 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:48:13.694 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.694 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.694 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:48:13.694 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:59.200077+0000", 2024-09-13T14:48:13.694 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5075999999999999e-05 2024-09-13T14:48:13.694 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.694 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.694 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:48:13.694 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:59.200095+0000", 2024-09-13T14:48:13.694 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.8165e-05 2024-09-13T14:48:13.694 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.695 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.695 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:48:13.695 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:59.200105+0000", 2024-09-13T14:48:13.695 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0436e-05 2024-09-13T14:48:13.695 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.695 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.695 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:48:13.695 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:59.506514+0000", 2024-09-13T14:48:13.695 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.30640889599999999 2024-09-13T14:48:13.695 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.695 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.695 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:48:13.695 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:59.506521+0000", 2024-09-13T14:48:13.696 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.3309999999999998e-06 2024-09-13T14:48:13.696 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.696 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.696 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:48:13.696 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:59.510201+0000", 2024-09-13T14:48:13.696 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0036795259999999998 2024-09-13T14:48:13.696 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.696 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:48:13.696 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.696 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.696 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.697 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "MOSDPGPush(6.b 1113/1103 [PushOp(6:d73bec53:::benchmark_data_smithi151_155836_object51158:head, version: 1088'3260, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:d73bec53:::benchmark_data_smithi151_155836_object51158:head@1088'3260, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:48:13.697 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:41:59.700025+0000", 2024-09-13T14:48:13.697 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 373.98440356899999, 2024-09-13T14:48:13.697 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.34375962100000002, 2024-09-13T14:48:13.697 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:48:13.697 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:48:13.697 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:48:13.697 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.697 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:48:13.697 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:59.700025+0000", 2024-09-13T14:48:13.697 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.698 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.698 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:48:13.698 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:59.700025+0000", 2024-09-13T14:48:13.698 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.698 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.698 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:48:13.698 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:59.700024+0000", 2024-09-13T14:48:13.698 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:48:13.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.699 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.699 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:48:13.699 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:59.700081+0000", 2024-09-13T14:48:13.699 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7297999999999999e-05 2024-09-13T14:48:13.699 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.699 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.699 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:48:13.699 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:59.700095+0000", 2024-09-13T14:48:13.699 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3920000000000001e-05 2024-09-13T14:48:13.699 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.699 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.699 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:48:13.700 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:41:59.700103+0000", 2024-09-13T14:48:13.700 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.011e-06 2024-09-13T14:48:13.700 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.700 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.701 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:48:13.701 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:00.039807+0000", 2024-09-13T14:48:13.701 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.33970362700000001 2024-09-13T14:48:13.701 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.701 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.701 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:48:13.701 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:00.039814+0000", 2024-09-13T14:48:13.701 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.1369999999999996e-06 2024-09-13T14:48:13.702 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.702 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.702 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:48:13.702 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:00.043784+0000", 2024-09-13T14:48:13.702 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0039701490000000001 2024-09-13T14:48:13.702 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.702 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:48:13.702 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.702 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.702 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.702 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "MOSDPGPush(6.b 1114/1103 [PushOp(6:decc40c7:::benchmark_data_smithi151_155836_object54693:head, version: 1088'3458, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:decc40c7:::benchmark_data_smithi151_155836_object54693:head@1088'3458, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:48:13.702 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:42:02.907860+0000", 2024-09-13T14:48:13.702 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 370.77656840100002, 2024-09-13T14:48:13.703 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.39222533599999998, 2024-09-13T14:48:13.703 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:48:13.703 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:48:13.703 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:48:13.703 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.703 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:48:13.703 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:02.907860+0000", 2024-09-13T14:48:13.704 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.704 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.704 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.704 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:48:13.704 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:02.907860+0000", 2024-09-13T14:48:13.705 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.705 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.705 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.705 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:48:13.705 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:02.907859+0000", 2024-09-13T14:48:13.705 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:48:13.705 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.705 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.705 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:48:13.706 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:02.907912+0000", 2024-09-13T14:48:13.706 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3063999999999999e-05 2024-09-13T14:48:13.706 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.706 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.706 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:48:13.706 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:02.907930+0000", 2024-09-13T14:48:13.706 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.8445e-05 2024-09-13T14:48:13.706 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.707 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.707 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:48:13.707 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:02.907939+0000", 2024-09-13T14:48:13.707 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.3109999999999995e-06 2024-09-13T14:48:13.707 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.707 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.707 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:48:13.708 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:03.295216+0000", 2024-09-13T14:48:13.708 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.387277025 2024-09-13T14:48:13.708 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.708 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.708 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:48:13.708 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:03.295223+0000", 2024-09-13T14:48:13.708 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.0149999999999998e-06 2024-09-13T14:48:13.708 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.708 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.708 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:48:13.708 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:03.300085+0000", 2024-09-13T14:48:13.708 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0048616930000000003 2024-09-13T14:48:13.708 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.708 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:48:13.709 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.709 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.709 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "MOSDPGPush(6.b 1114/1103 [PushOp(6:d5a279ab:::benchmark_data_smithi151_155836_object57905:head, version: 1088'3666, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:d5a279ab:::benchmark_data_smithi151_155836_object57905:head@1088'3666, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:48:13.709 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:42:05.928618+0000", 2024-09-13T14:48:13.709 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 367.75581015300003, 2024-09-13T14:48:13.709 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.59717940400000002, 2024-09-13T14:48:13.709 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:48:13.709 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:48:13.709 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:48:13.709 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.709 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:48:13.710 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.928618+0000", 2024-09-13T14:48:13.710 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.710 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.710 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.710 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:48:13.710 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.928618+0000", 2024-09-13T14:48:13.711 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.711 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.711 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.711 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:48:13.711 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.928617+0000", 2024-09-13T14:48:13.711 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:48:13.711 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.711 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.711 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:48:13.712 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.928666+0000", 2024-09-13T14:48:13.712 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9001000000000001e-05 2024-09-13T14:48:13.712 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.712 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.712 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:48:13.712 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.928683+0000", 2024-09-13T14:48:13.712 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6838e-05 2024-09-13T14:48:13.712 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.712 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.712 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:48:13.712 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:05.928691+0000", 2024-09-13T14:48:13.712 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.4249999999999998e-06 2024-09-13T14:48:13.712 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.712 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.713 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:48:13.713 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:06.520917+0000", 2024-09-13T14:48:13.713 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.59222553499999997 2024-09-13T14:48:13.713 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.713 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.713 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:48:13.713 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:06.520924+0000", 2024-09-13T14:48:13.713 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.5180000000000001e-06 2024-09-13T14:48:13.713 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.713 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.713 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:48:13.713 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:06.525797+0000", 2024-09-13T14:48:13.713 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0048739029999999997 2024-09-13T14:48:13.714 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.714 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:48:13.714 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.714 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.714 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "MOSDPGPush(6.b 1114/1103 [PushOp(6:dbb5192a:::benchmark_data_smithi151_155836_object59669:head, version: 1088'3783, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:dbb5192a:::benchmark_data_smithi151_155836_object59669:head@1088'3783, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:48:13.714 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:42:07.583326+0000", 2024-09-13T14:48:13.714 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 366.10110265399999, 2024-09-13T14:48:13.714 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.32661004199999999, 2024-09-13T14:48:13.714 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:48:13.714 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:48:13.714 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:48:13.714 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.714 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:48:13.714 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:07.583326+0000", 2024-09-13T14:48:13.715 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.715 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.715 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:48:13.715 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:07.583326+0000", 2024-09-13T14:48:13.715 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.715 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.715 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:48:13.715 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:07.583325+0000", 2024-09-13T14:48:13.715 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:48:13.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.715 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.715 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:48:13.716 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:07.583376+0000", 2024-09-13T14:48:13.716 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0708999999999998e-05 2024-09-13T14:48:13.716 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.716 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.716 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:48:13.716 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:07.583391+0000", 2024-09-13T14:48:13.716 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4911e-05 2024-09-13T14:48:13.716 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.717 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.717 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:48:13.717 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:07.583400+0000", 2024-09-13T14:48:13.717 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.8300000000000002e-06 2024-09-13T14:48:13.717 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.717 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.717 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:48:13.717 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:07.899890+0000", 2024-09-13T14:48:13.718 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.31649057600000002 2024-09-13T14:48:13.718 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.718 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.718 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:48:13.718 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:07.899897+0000", 2024-09-13T14:48:13.718 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.2280000000000001e-06 2024-09-13T14:48:13.718 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.718 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.719 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:48:13.719 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:07.909936+0000", 2024-09-13T14:48:13.719 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.010038134000000001 2024-09-13T14:48:13.719 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.719 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:48:13.719 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.719 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.719 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.719 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "MOSDPGPush(6.b 1114/1103 [PushOp(6:d39020a4:::benchmark_data_smithi151_155836_object61835:head, version: 1088'3909, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:d39020a4:::benchmark_data_smithi151_155836_object61835:head@1088'3909, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:48:13.719 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:42:09.535665+0000", 2024-09-13T14:48:13.719 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 364.14876339400001, 2024-09-13T14:48:13.719 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.39573968500000001, 2024-09-13T14:48:13.719 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:48:13.720 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:48:13.720 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:48:13.720 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.720 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:48:13.720 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:09.535665+0000", 2024-09-13T14:48:13.720 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.720 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.720 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.720 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:48:13.720 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:09.535665+0000", 2024-09-13T14:48:13.720 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.720 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.720 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.721 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:48:13.721 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:09.535661+0000", 2024-09-13T14:48:13.721 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999957 2024-09-13T14:48:13.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.721 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.721 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:48:13.721 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:09.535718+0000", 2024-09-13T14:48:13.721 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7720999999999998e-05 2024-09-13T14:48:13.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.721 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.721 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:48:13.721 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:09.535740+0000", 2024-09-13T14:48:13.721 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.1679e-05 2024-09-13T14:48:13.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.722 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.722 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:48:13.722 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:09.535749+0000", 2024-09-13T14:48:13.722 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.9490000000000002e-06 2024-09-13T14:48:13.722 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.722 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.722 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:48:13.722 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:09.916854+0000", 2024-09-13T14:48:13.722 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.381105472 2024-09-13T14:48:13.722 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.722 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.722 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:48:13.722 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:09.916863+0000", 2024-09-13T14:48:13.722 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.7520000000000002e-06 2024-09-13T14:48:13.723 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.723 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.723 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:48:13.723 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:09.931404+0000", 2024-09-13T14:48:13.723 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.014541383999999999 2024-09-13T14:48:13.723 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.723 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:48:13.723 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.723 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.723 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.723 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "MOSDPGPush(6.b 1118/1103 [PushOp(6:d0beb7b7:::benchmark_data_smithi151_155836_object75303:head, version: 1090'4733, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:d0beb7b7:::benchmark_data_smithi151_155836_object75303:head@1090'4733, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:48:13.724 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:42:22.403607+0000", 2024-09-13T14:48:13.724 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 351.28082120900001, 2024-09-13T14:48:13.724 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.47577770800000002, 2024-09-13T14:48:13.724 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:48:13.724 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:48:13.724 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:48:13.724 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.724 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:48:13.724 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:22.403607+0000", 2024-09-13T14:48:13.724 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.724 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.724 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.724 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:48:13.725 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:22.403607+0000", 2024-09-13T14:48:13.725 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.725 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.725 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:48:13.725 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:22.403606+0000", 2024-09-13T14:48:13.725 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:48:13.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.725 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.725 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:48:13.725 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:22.403649+0000", 2024-09-13T14:48:13.725 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.2503000000000003e-05 2024-09-13T14:48:13.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.726 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.726 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:48:13.726 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:22.403668+0000", 2024-09-13T14:48:13.726 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.9177e-05 2024-09-13T14:48:13.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.726 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.726 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:48:13.726 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:22.403678+0000", 2024-09-13T14:48:13.726 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.8849999999999994e-06 2024-09-13T14:48:13.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.727 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.727 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:48:13.727 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:22.857374+0000", 2024-09-13T14:48:13.727 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.453696404 2024-09-13T14:48:13.727 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.727 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.727 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:48:13.728 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:22.857382+0000", 2024-09-13T14:48:13.728 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.1189999999999997e-06 2024-09-13T14:48:13.728 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.728 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.728 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:48:13.728 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:22.879385+0000", 2024-09-13T14:48:13.729 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.022002346999999998 2024-09-13T14:48:13.729 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.729 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:48:13.729 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.729 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.729 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.730 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "MOSDPGPush(6.b 1118/1103 [PushOp(6:d18fd3e1:::benchmark_data_smithi151_155836_object76543:head, version: 1090'4811, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:d18fd3e1:::benchmark_data_smithi151_155836_object76543:head@1090'4811, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:48:13.730 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:42:23.608609+0000", 2024-09-13T14:48:13.730 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 350.07581958399999, 2024-09-13T14:48:13.730 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.464993616, 2024-09-13T14:48:13.730 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:48:13.730 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:48:13.730 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:48:13.731 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.731 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:48:13.731 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:23.608609+0000", 2024-09-13T14:48:13.731 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.731 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.731 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.731 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:48:13.732 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:23.608609+0000", 2024-09-13T14:48:13.732 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.732 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.732 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.732 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:48:13.732 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:23.608608+0000", 2024-09-13T14:48:13.732 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:48:13.732 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.733 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.733 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:48:13.733 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:23.608652+0000", 2024-09-13T14:48:13.733 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4653000000000001e-05 2024-09-13T14:48:13.733 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.733 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.734 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:48:13.734 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:23.608673+0000", 2024-09-13T14:48:13.734 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.101e-05 2024-09-13T14:48:13.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.734 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.734 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:48:13.735 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:23.608682+0000", 2024-09-13T14:48:13.735 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.0880000000000001e-06 2024-09-13T14:48:13.735 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.735 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.735 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:48:13.735 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:24.067830+0000", 2024-09-13T14:48:13.736 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.45914716999999999 2024-09-13T14:48:13.736 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.736 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.736 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:48:13.736 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:24.067837+0000", 2024-09-13T14:48:13.736 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.6790000000000005e-06 2024-09-13T14:48:13.736 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.736 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.736 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:48:13.736 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:24.073602+0000", 2024-09-13T14:48:13.737 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0057649110000000002 2024-09-13T14:48:13.737 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.737 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:48:13.737 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.737 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.737 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.737 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "MOSDPGPush(6.b 1118/1103 [PushOp(6:dd7eb092:::benchmark_data_smithi151_155836_object78015:head, version: 1090'4902, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:dd7eb092:::benchmark_data_smithi151_155836_object78015:head@1090'4902, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:48:13.737 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:42:25.005572+0000", 2024-09-13T14:48:13.737 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 348.67885639600001, 2024-09-13T14:48:13.737 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.28881849999999998, 2024-09-13T14:48:13.737 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:48:13.737 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:48:13.737 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:48:13.738 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.738 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:48:13.738 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:25.005572+0000", 2024-09-13T14:48:13.738 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.738 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.738 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.738 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:48:13.738 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:25.005572+0000", 2024-09-13T14:48:13.738 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.738 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.738 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.738 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:48:13.738 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:25.005571+0000", 2024-09-13T14:48:13.738 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:48:13.739 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.739 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.739 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:48:13.739 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:25.005617+0000", 2024-09-13T14:48:13.739 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5840000000000002e-05 2024-09-13T14:48:13.739 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.739 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.739 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:48:13.739 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:25.005635+0000", 2024-09-13T14:48:13.739 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.7833999999999999e-05 2024-09-13T14:48:13.739 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.739 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.739 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:48:13.740 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:25.005644+0000", 2024-09-13T14:48:13.740 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.1090000000000004e-06 2024-09-13T14:48:13.740 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.740 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.740 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:48:13.740 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:25.290137+0000", 2024-09-13T14:48:13.740 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.28449300599999999 2024-09-13T14:48:13.740 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.740 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.740 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:48:13.740 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:25.290145+0000", 2024-09-13T14:48:13.740 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.9289999999999998e-06 2024-09-13T14:48:13.741 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.741 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.741 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:48:13.741 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:25.294390+0000", 2024-09-13T14:48:13.741 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.004245202 2024-09-13T14:48:13.741 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.741 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:48:13.741 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.741 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.741 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.741 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "MOSDPGPush(6.b 1118/1103 [PushOp(6:d3efb36d:::benchmark_data_smithi151_155836_object81532:head, version: 1090'5121, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:d3efb36d:::benchmark_data_smithi151_155836_object81532:head@1090'5121, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:48:13.741 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:42:28.496299+0000", 2024-09-13T14:48:13.741 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 345.18812899300002, 2024-09-13T14:48:13.742 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.30702775199999999, 2024-09-13T14:48:13.742 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:48:13.742 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:48:13.742 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:48:13.742 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.742 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:48:13.742 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.496299+0000", 2024-09-13T14:48:13.742 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.742 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.743 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.743 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:48:13.743 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.496299+0000", 2024-09-13T14:48:13.744 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.744 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.744 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.744 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:48:13.744 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.496298+0000", 2024-09-13T14:48:13.745 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:48:13.745 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.745 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.745 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:48:13.745 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.496358+0000", 2024-09-13T14:48:13.745 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.0210000000000001e-05 2024-09-13T14:48:13.745 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.745 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.745 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:48:13.745 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.496377+0000", 2024-09-13T14:48:13.745 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.8890999999999999e-05 2024-09-13T14:48:13.745 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.745 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.746 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:48:13.746 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.496387+0000", 2024-09-13T14:48:13.746 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.2650000000000002e-06 2024-09-13T14:48:13.746 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.746 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.746 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:48:13.746 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.796689+0000", 2024-09-13T14:48:13.746 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.30030266799999999 2024-09-13T14:48:13.746 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.746 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.746 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:48:13.746 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.796697+0000", 2024-09-13T14:48:13.746 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.4059999999999999e-06 2024-09-13T14:48:13.747 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.747 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.747 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:48:13.747 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:28.803327+0000", 2024-09-13T14:48:13.747 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0066302720000000004 2024-09-13T14:48:13.747 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.747 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:48:13.747 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.747 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.747 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.747 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "MOSDPGPush(6.b 1123/1103 [PushOp(6:df1bf793:::benchmark_data_smithi151_155836_object102427:head, version: 1091'6412, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:df1bf793:::benchmark_data_smithi151_155836_object102427:head@1091'6412, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:48:13.747 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:42:49.694424+0000", 2024-09-13T14:48:13.748 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 323.990004494, 2024-09-13T14:48:13.748 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.27876336099999999, 2024-09-13T14:48:13.748 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:48:13.748 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:48:13.748 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:48:13.748 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.748 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:48:13.748 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:49.694424+0000", 2024-09-13T14:48:13.748 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.748 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.748 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.748 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:48:13.748 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:49.694424+0000", 2024-09-13T14:48:13.749 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.749 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.749 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.749 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:48:13.749 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:49.694423+0000", 2024-09-13T14:48:13.749 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:48:13.749 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.749 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.749 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:48:13.749 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:49.694506+0000", 2024-09-13T14:48:13.749 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.3022000000000006e-05 2024-09-13T14:48:13.749 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.749 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.750 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:48:13.750 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:49.694527+0000", 2024-09-13T14:48:13.750 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.0953000000000001e-05 2024-09-13T14:48:13.750 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.750 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.750 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:48:13.750 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:49.694536+0000", 2024-09-13T14:48:13.750 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.1730000000000001e-06 2024-09-13T14:48:13.750 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.750 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.750 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:48:13.750 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:49.928362+0000", 2024-09-13T14:48:13.750 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.233826584 2024-09-13T14:48:13.750 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.751 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.751 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:48:13.751 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:49.928369+0000", 2024-09-13T14:48:13.751 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.9639999999999997e-06 2024-09-13T14:48:13.751 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.751 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.751 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:48:13.751 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:49.973187+0000", 2024-09-13T14:48:13.751 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.044817633000000003 2024-09-13T14:48:13.752 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.752 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:48:13.752 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.752 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.752 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.752 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "MOSDPGPush(6.b 1123/1103 [PushOp(6:d73ddd5f:::benchmark_data_smithi151_155836_object102917:head, version: 1091'6447, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:d73ddd5f:::benchmark_data_smithi151_155836_object102917:head@1091'6447, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:48:13.752 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:42:50.209379+0000", 2024-09-13T14:48:13.752 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 323.47504930000002, 2024-09-13T14:48:13.752 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.42185424199999999, 2024-09-13T14:48:13.752 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:48:13.752 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:48:13.752 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:48:13.753 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.753 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:48:13.753 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:50.209379+0000", 2024-09-13T14:48:13.753 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.753 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.753 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.753 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:48:13.753 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:50.209379+0000", 2024-09-13T14:48:13.753 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.754 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.754 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.754 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:48:13.754 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:50.209369+0000", 2024-09-13T14:48:13.754 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999905 2024-09-13T14:48:13.754 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.754 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.754 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:48:13.754 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:50.209434+0000", 2024-09-13T14:48:13.754 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.4679999999999997e-05 2024-09-13T14:48:13.755 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.755 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.755 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:48:13.755 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:50.209452+0000", 2024-09-13T14:48:13.755 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.749e-05 2024-09-13T14:48:13.755 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.755 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.755 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:48:13.755 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:50.209460+0000", 2024-09-13T14:48:13.755 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.6489999999999997e-06 2024-09-13T14:48:13.755 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.755 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.755 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:48:13.755 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:50.619097+0000", 2024-09-13T14:48:13.756 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.409637207 2024-09-13T14:48:13.756 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.756 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.756 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:48:13.756 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:50.619104+0000", 2024-09-13T14:48:13.756 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.6610000000000003e-06 2024-09-13T14:48:13.756 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.756 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.756 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:48:13.756 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:50.631233+0000", 2024-09-13T14:48:13.756 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.012129035 2024-09-13T14:48:13.756 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.756 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:48:13.757 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.757 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.757 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.757 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "MOSDPGPush(6.b 1123/1103 [PushOp(6:d4580e47:::benchmark_data_smithi151_155836_object104311:head, version: 1091'6538, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:d4580e47:::benchmark_data_smithi151_155836_object104311:head@1091'6538, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:48:13.757 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:42:51.585827+0000", 2024-09-13T14:48:13.757 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 322.09860158599997, 2024-09-13T14:48:13.757 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.40485529399999998, 2024-09-13T14:48:13.757 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:48:13.757 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:48:13.757 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:48:13.757 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.757 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:48:13.757 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:51.585827+0000", 2024-09-13T14:48:13.758 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.758 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.758 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.758 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:48:13.758 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:51.585827+0000", 2024-09-13T14:48:13.758 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.758 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.758 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.758 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:48:13.758 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:51.585826+0000", 2024-09-13T14:48:13.758 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:48:13.758 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.758 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.758 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:48:13.758 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:51.585878+0000", 2024-09-13T14:48:13.759 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1931999999999997e-05 2024-09-13T14:48:13.759 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.759 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.759 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:48:13.759 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:51.585895+0000", 2024-09-13T14:48:13.759 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.7946000000000001e-05 2024-09-13T14:48:13.759 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.759 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.759 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:48:13.759 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:51.585905+0000", 2024-09-13T14:48:13.759 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.3270000000000007e-06 2024-09-13T14:48:13.759 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.759 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.759 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:48:13.760 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:51.982427+0000", 2024-09-13T14:48:13.760 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.39652254599999998 2024-09-13T14:48:13.760 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.760 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.760 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:48:13.760 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:51.982434+0000", 2024-09-13T14:48:13.760 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.5400000000000001e-06 2024-09-13T14:48:13.760 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.760 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.760 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:48:13.760 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:51.990682+0000", 2024-09-13T14:48:13.760 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0082480160000000004 2024-09-13T14:48:13.760 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.760 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:48:13.761 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:13.762 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.762 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.762 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.762 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "MOSDPGPush(6.b 1123/1103 [PushOp(6:ddd6b124:::benchmark_data_smithi151_155836_object107586:head, version: 1092'6737, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:ddd6b124:::benchmark_data_smithi151_155836_object107586:head@1092'6737, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:48:13.762 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:42:54.785140+0000", 2024-09-13T14:48:13.762 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 318.899287759, 2024-09-13T14:48:13.763 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.70343214300000001, 2024-09-13T14:48:13.763 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:48:13.763 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:48:13.763 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:48:13.763 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.763 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:48:13.763 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:54.785140+0000", 2024-09-13T14:48:13.763 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.763 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.763 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.763 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:48:13.763 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:54.785140+0000", 2024-09-13T14:48:13.763 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.763 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.764 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.764 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:48:13.764 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:54.785140+0000", 2024-09-13T14:48:13.764 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:48:13.764 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.764 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.764 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:48:13.764 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:54.785193+0000", 2024-09-13T14:48:13.764 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3386e-05 2024-09-13T14:48:13.764 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.764 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.764 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:48:13.765 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:54.785213+0000", 2024-09-13T14:48:13.765 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.0047999999999998e-05 2024-09-13T14:48:13.765 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.765 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.765 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:48:13.765 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:54.785222+0000", 2024-09-13T14:48:13.765 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.7509999999999997e-06 2024-09-13T14:48:13.765 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.765 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.766 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:48:13.766 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:55.481924+0000", 2024-09-13T14:48:13.766 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.69670191000000004 2024-09-13T14:48:13.766 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.766 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.766 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:48:13.766 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:55.481930+0000", 2024-09-13T14:48:13.766 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7580000000000003e-06 2024-09-13T14:48:13.766 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.766 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.766 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:48:13.766 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:55.488573+0000", 2024-09-13T14:48:13.766 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0066420849999999998 2024-09-13T14:48:13.767 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.767 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:48:13.767 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.767 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.767 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.767 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "MOSDPGPush(6.b 1123/1103 [PushOp(6:d98e4068:::benchmark_data_smithi151_155836_object110381:head, version: 1092'6924, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:d98e4068:::benchmark_data_smithi151_155836_object110381:head@1092'6924, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:48:13.767 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:42:57.570213+0000", 2024-09-13T14:48:13.767 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 316.11421505499999, 2024-09-13T14:48:13.767 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.280807003, 2024-09-13T14:48:13.767 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:48:13.767 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:48:13.767 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:48:13.767 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.768 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:48:13.768 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:57.570213+0000", 2024-09-13T14:48:13.768 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.768 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.768 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.768 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:48:13.768 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:57.570213+0000", 2024-09-13T14:48:13.768 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.768 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.768 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.768 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:48:13.768 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:57.570212+0000", 2024-09-13T14:48:13.768 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:48:13.768 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.768 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.769 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:48:13.769 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:57.570261+0000", 2024-09-13T14:48:13.769 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8412000000000003e-05 2024-09-13T14:48:13.769 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.769 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.769 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:48:13.769 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:57.570279+0000", 2024-09-13T14:48:13.769 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.8376000000000002e-05 2024-09-13T14:48:13.769 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.769 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.769 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:48:13.769 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:57.570289+0000", 2024-09-13T14:48:13.769 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0172e-05 2024-09-13T14:48:13.769 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.770 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.770 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:48:13.770 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:57.846227+0000", 2024-09-13T14:48:13.770 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.27593732900000001 2024-09-13T14:48:13.770 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.770 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.770 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:48:13.770 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:57.846234+0000", 2024-09-13T14:48:13.770 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.2810000000000003e-06 2024-09-13T14:48:13.771 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.771 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.771 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:48:13.771 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:57.851020+0000", 2024-09-13T14:48:13.771 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0047863030000000004 2024-09-13T14:48:13.771 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.772 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:48:13.772 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.772 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.772 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.772 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "MOSDPGPush(6.b 1123/1103 [PushOp(6:da27bd7d:::benchmark_data_smithi151_155836_object110838:head, version: 1092'6956, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:da27bd7d:::benchmark_data_smithi151_155836_object110838:head@1092'6956, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:48:13.772 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:42:58.000080+0000", 2024-09-13T14:48:13.772 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 315.68434790600003, 2024-09-13T14:48:13.772 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.40871167600000002, 2024-09-13T14:48:13.772 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:48:13.772 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:48:13.773 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:48:13.773 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.773 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:48:13.773 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:58.000080+0000", 2024-09-13T14:48:13.773 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.773 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.773 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.773 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:48:13.773 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:58.000080+0000", 2024-09-13T14:48:13.773 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.773 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.773 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.773 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:48:13.773 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:58.000080+0000", 2024-09-13T14:48:13.774 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:48:13.774 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.774 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.774 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:48:13.774 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:58.000147+0000", 2024-09-13T14:48:13.774 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7428999999999994e-05 2024-09-13T14:48:13.774 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.774 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.774 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:48:13.774 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:58.000167+0000", 2024-09-13T14:48:13.774 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.9565e-05 2024-09-13T14:48:13.774 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.774 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.774 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:48:13.775 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:58.000178+0000", 2024-09-13T14:48:13.775 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1073999999999999e-05 2024-09-13T14:48:13.775 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.775 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.775 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:48:13.775 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:58.405979+0000", 2024-09-13T14:48:13.775 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.40580137500000002 2024-09-13T14:48:13.775 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.775 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.775 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:48:13.775 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:58.405984+0000", 2024-09-13T14:48:13.775 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5260000000000004e-06 2024-09-13T14:48:13.775 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.775 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.776 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:48:13.776 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:42:58.408792+0000", 2024-09-13T14:48:13.776 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0028083880000000002 2024-09-13T14:48:13.776 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.776 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:48:13.776 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.776 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.776 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.776 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "MOSDPGPush(6.b 1123/1103 [PushOp(6:d9559628:::benchmark_data_smithi151_155836_object113584:head, version: 1092'7126, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:d9559628:::benchmark_data_smithi151_155836_object113584:head@1092'7126, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:48:13.776 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:43:00.748514+0000", 2024-09-13T14:48:13.776 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 312.93591421500003, 2024-09-13T14:48:13.776 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.38776434300000001, 2024-09-13T14:48:13.776 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:48:13.776 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:48:13.777 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:48:13.777 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.777 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:48:13.777 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:43:00.748514+0000", 2024-09-13T14:48:13.777 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.777 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.777 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.777 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:48:13.777 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:43:00.748514+0000", 2024-09-13T14:48:13.777 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.777 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.777 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.777 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:48:13.777 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:43:00.748513+0000", 2024-09-13T14:48:13.778 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:48:13.778 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.778 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.778 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:48:13.778 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:43:00.748558+0000", 2024-09-13T14:48:13.778 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4784999999999999e-05 2024-09-13T14:48:13.778 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.778 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.778 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:48:13.778 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:43:00.748578+0000", 2024-09-13T14:48:13.778 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.9992000000000001e-05 2024-09-13T14:48:13.778 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.778 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.779 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:48:13.779 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:43:00.748587+0000", 2024-09-13T14:48:13.779 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.7310000000000005e-06 2024-09-13T14:48:13.779 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.779 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.779 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:48:13.779 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:43:01.131962+0000", 2024-09-13T14:48:13.779 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.38337483700000002 2024-09-13T14:48:13.779 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.779 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.779 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:48:13.779 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:43:01.131972+0000", 2024-09-13T14:48:13.779 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.3980000000000005e-06 2024-09-13T14:48:13.779 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.780 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.780 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:48:13.780 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:43:01.136278+0000", 2024-09-13T14:48:13.780 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.004306748 2024-09-13T14:48:13.780 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.780 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:48:13.780 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.780 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.780 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.780 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "MOSDPGPush(6.b 1123/1103 [PushOp(6:da84a270:::benchmark_data_smithi151_155836_object114741:head, version: 1092'7201, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:da84a270:::benchmark_data_smithi151_155836_object114741:head@1092'7201, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:48:13.780 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:43:01.902841+0000", 2024-09-13T14:48:13.780 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 311.78158746899999, 2024-09-13T14:48:13.781 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.38981042999999999, 2024-09-13T14:48:13.781 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:48:13.781 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:48:13.781 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:48:13.781 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.781 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:48:13.781 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:43:01.902841+0000", 2024-09-13T14:48:13.781 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.781 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.781 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.781 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:48:13.781 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:43:01.902841+0000", 2024-09-13T14:48:13.781 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.782 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.782 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.782 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:48:13.782 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:43:01.902840+0000", 2024-09-13T14:48:13.782 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:48:13.782 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.782 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.782 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:48:13.782 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:43:01.902894+0000", 2024-09-13T14:48:13.782 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3609000000000001e-05 2024-09-13T14:48:13.783 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.783 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.783 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:48:13.783 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:43:01.902909+0000", 2024-09-13T14:48:13.783 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.5468000000000002e-05 2024-09-13T14:48:13.783 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.783 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.783 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:48:13.783 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:43:01.902917+0000", 2024-09-13T14:48:13.783 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0139999999999998e-06 2024-09-13T14:48:13.783 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.783 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.783 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:48:13.783 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:43:02.289144+0000", 2024-09-13T14:48:13.783 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.386227245 2024-09-13T14:48:13.784 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.784 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.784 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:48:13.784 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:43:02.289152+0000", 2024-09-13T14:48:13.784 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.2030000000000003e-06 2024-09-13T14:48:13.784 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.784 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.784 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:48:13.784 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:43:02.292651+0000", 2024-09-13T14:48:13.784 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0034994560000000002 2024-09-13T14:48:13.784 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.784 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:48:13.784 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.784 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.785 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.785 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "MOSDPGPush(6.b 1123/1103 [PushOp(6:d86bdb1c:::benchmark_data_smithi151_155836_object115994:head, version: 1092'7291, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:d86bdb1c:::benchmark_data_smithi151_155836_object115994:head@1092'7291, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:48:13.785 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:43:03.209785+0000", 2024-09-13T14:48:13.785 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 310.47464357899997, 2024-09-13T14:48:13.785 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.280035111, 2024-09-13T14:48:13.785 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:48:13.785 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:48:13.785 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:48:13.785 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.785 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:48:13.785 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:43:03.209785+0000", 2024-09-13T14:48:13.785 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.785 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.785 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.786 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:48:13.786 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:43:03.209785+0000", 2024-09-13T14:48:13.786 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:48:13.786 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.786 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.786 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:48:13.786 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:43:03.209784+0000", 2024-09-13T14:48:13.786 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:48:13.786 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.786 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.787 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:48:13.787 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:43:03.209823+0000", 2024-09-13T14:48:13.787 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8940999999999999e-05 2024-09-13T14:48:13.787 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.787 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.787 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:48:13.787 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:43:03.209844+0000", 2024-09-13T14:48:13.787 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.1072000000000001e-05 2024-09-13T14:48:13.787 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.787 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.787 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:48:13.787 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:43:03.209853+0000", 2024-09-13T14:48:13.787 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.1169999999999993e-06 2024-09-13T14:48:13.787 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.788 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.788 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:48:13.788 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:43:03.484970+0000", 2024-09-13T14:48:13.788 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.27511705199999997 2024-09-13T14:48:13.788 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.788 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.788 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:48:13.788 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:43:03.484977+0000", 2024-09-13T14:48:13.788 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.0809999999999996e-06 2024-09-13T14:48:13.788 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:48:13.788 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:48:13.788 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:48:13.788 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:43:03.489820+0000", 2024-09-13T14:48:13.788 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0048425719999999998 2024-09-13T14:48:13.789 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.789 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:48:13.789 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.789 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:48:13.789 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:48:13.789 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:48:13.815 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:13.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:13.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:13.916 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:13.943 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:13.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:13.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:14.045 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:14.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:14.067 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:14.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:14.167 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:14.193 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:14.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:14.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:14.295 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:14.321 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:14.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:14.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:14.423 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:14.471 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:14.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:14.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:14.526 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T14:48:14.573 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:14.615 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:14.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:14.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:14.717 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:14.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:14.754 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:14.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:14.841 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:48:14.855 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:14.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:14 smithi151 ceph-mon[104450]: pgmap v1632: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.4 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 666 op/s; 3609/71295 objects degraded (5.062%); 18646/71295 objects misplaced (26.153%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:48:14.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:48:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:48:14] "GET /metrics HTTP/1.1" 200 36656 "" "Prometheus/2.51.0" 2024-09-13T14:48:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:14 smithi151 ceph-mon[108966]: pgmap v1632: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.4 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 666 op/s; 3609/71295 objects degraded (5.062%); 18646/71295 objects misplaced (26.153%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:48:14.885 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:14.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:14.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:14.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:15.026 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:15.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:15.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:14 smithi184 ceph-mon[92527]: pgmap v1632: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.4 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 666 op/s; 3609/71295 objects degraded (5.062%); 18646/71295 objects misplaced (26.153%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:48:15.128 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:15.164 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:15.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:15.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:15.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:15.300 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:15.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:15.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:15.402 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:15.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:15.445 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:15.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:15.546 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:15.574 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:15.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:15.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:15.676 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:15.700 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:15.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:15.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:15.802 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:15.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:15.826 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:15.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:15.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:15 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 3609/71295 objects degraded (5.062%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:15.927 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:15.951 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:15.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:15.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:16.054 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:16.094 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:16.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:16.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:16.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:15 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 3609/71295 objects degraded (5.062%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:16.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:15 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 3609/71295 objects degraded (5.062%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:16.196 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:16.250 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:16.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:16.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:16.352 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:16.379 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:16.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:16.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:16.481 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:16.507 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:16.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:16.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:16.609 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:16.634 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:16.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:16.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:16.736 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:16.760 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:16.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:16.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:16.862 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:16.884 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:16.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:16.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:16.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:16 smithi151 ceph-mon[104450]: pgmap v1633: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.4 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 554 op/s; 3609/68622 objects degraded (5.259%); 18099/68622 objects misplaced (26.375%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:48:16.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:16.928 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:16 smithi151 ceph-mon[108966]: pgmap v1633: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.4 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 554 op/s; 3609/68622 objects degraded (5.259%); 18099/68622 objects misplaced (26.375%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:48:16.928 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:16.937 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:16 smithi184 ceph-mon[92527]: pgmap v1633: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.4 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 554 op/s; 3609/68622 objects degraded (5.259%); 18099/68622 objects misplaced (26.375%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:48:16.937 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:16.946 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:48:16.946 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [], 2024-09-13T14:48:16.946 INFO:teuthology.orchestra.run.smithi184.stdout: "num_ops": 0 2024-09-13T14:48:16.946 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:48:16.986 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:17.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:17.115 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:17.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:17.216 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:17.242 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:17.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:17.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:17.343 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:17.389 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:17.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:17.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:17.490 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:17.525 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:17.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:17.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:17.627 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:17.682 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:17.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:17.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:17.784 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:17.831 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:17.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:17.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:17.870 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:17.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:17.984 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T14:48:17.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:17.998 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:17.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:18.015 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:48:18.017 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T14:48:18.100 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:18.142 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:18.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:18.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:18.244 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:18.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:18.278 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:18.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:18.316 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:48:18.380 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:18.410 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:18.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:18.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:18.513 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:18.558 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:18.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:18.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:18.660 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:18.711 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:18.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:18.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:18.801 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-13T14:48:18.814 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:18.838 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:18.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:18.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:18.940 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:18.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:18.966 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:18.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:19.011 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:18 smithi151 ceph-mon[104450]: pgmap v1634: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.4 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 533 op/s; 3702/66462 objects degraded (5.570%); 17722/66462 objects misplaced (26.665%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T14:48:19.011 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:18 smithi151 ceph-mon[104450]: from='client.35835 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:48:19.011 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:18 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1449350328' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:19.011 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:18 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:19.012 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:18 smithi151 ceph-mon[108966]: pgmap v1634: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.4 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 533 op/s; 3702/66462 objects degraded (5.570%); 17722/66462 objects misplaced (26.665%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T14:48:19.012 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:18 smithi151 ceph-mon[108966]: from='client.35835 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:48:19.012 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:18 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1449350328' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:19.012 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:18 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:19.066 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:19.089 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:19.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:19.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:19.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:18 smithi184 ceph-mon[92527]: pgmap v1634: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.4 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 533 op/s; 3702/66462 objects degraded (5.570%); 17722/66462 objects misplaced (26.665%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T14:48:19.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:18 smithi184 ceph-mon[92527]: from='client.35835 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:48:19.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:18 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1449350328' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:19.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:18 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:19.191 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:19.223 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:19.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:19.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:19.325 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:19.368 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:19.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:19.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:19.385 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1634,"stamp":"2024-09-13T14:48:17.512116+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1450737848,"num_objects":22154,"num_object_clones":0,"num_object_copies":66462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3702,"num_objects_misplaced":17722,"num_objects_unfound":0,"num_objects_dirty":22154,"num_whiteouts":0,"num_read":14,"num_read_kb":35,"num_write":218860,"num_write_kb":7709418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2546,"num_bytes_recovered":166291705,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115371,"ondisk_log_size":115371,"up":357,"acting":356,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":366,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7414436,"kb_used_data":4874532,"kb_used_omap":149,"kb_used_meta":2539562,"kb_avail":742546780,"statfs":{"total":767960285184,"available":760367902720,"internally_reserved":0,"allocated":4991520768,"data_stored":4963077543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":153401,"internal_metadata":2600511687},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,13,1,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-419627008,"num_objects":-6403,"num_object_clones":0,"num_object_copies":-19209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":228,"num_objects_misplaced":-3199,"num_objects_unfound":0,"num_objects_dirty":-6403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6403,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":281,"num_bytes_recovered":18415616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004247"},"pg_stats":[{"pgid":"1.1d","version":"261'100","reported_seq":94,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.751697+0000","last_change":"2024-09-13T14:45:19.807356+0000","last_active":"2024-09-13T14:48:01.751697+0000","last_peered":"2024-09-13T14:48:01.751697+0000","last_clean":"2024-09-13T14:48:01.751697+0000","last_became_active":"2024-09-13T14:45:19.807081+0000","last_became_peered":"2024-09-13T14:45:19.807081+0000","last_unstale":"2024-09-13T14:48:01.751697+0000","last_undegraded":"2024-09-13T14:48:01.751697+0000","last_fullsized":"2024-09-13T14:48:01.751697+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:17.988345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:17.988345+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:41:05.216335+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":1700,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.766967+0000","last_change":"2024-09-13T14:29:42.541263+0000","last_active":"2024-09-13T14:48:01.766967+0000","last_peered":"2024-09-13T14:48:01.766967+0000","last_clean":"2024-09-13T14:48:01.766967+0000","last_became_active":"2024-09-13T14:29:42.540873+0000","last_became_peered":"2024-09-13T14:29:42.540873+0000","last_unstale":"2024-09-13T14:48:01.766967+0000","last_undegraded":"2024-09-13T14:48:01.766967+0000","last_fullsized":"2024-09-13T14:48:01.766967+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:20.611206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1122,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.601436+0000","last_change":"2024-09-13T14:43:07.516715+0000","last_active":"2024-09-13T14:47:40.601436+0000","last_peered":"2024-09-13T14:47:40.601436+0000","last_clean":"2024-09-13T14:47:40.601436+0000","last_became_active":"2024-09-13T14:43:07.516403+0000","last_became_peered":"2024-09-13T14:43:07.516403+0000","last_unstale":"2024-09-13T14:47:40.601436+0000","last_undegraded":"2024-09-13T14:47:40.601436+0000","last_fullsized":"2024-09-13T14:47:40.601436+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:56.469690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":39,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.744351+0000","last_change":"2024-09-13T14:45:19.810203+0000","last_active":"2024-09-13T14:48:01.744351+0000","last_peered":"2024-09-13T14:48:01.744351+0000","last_clean":"2024-09-13T14:48:01.744351+0000","last_became_active":"2024-09-13T14:45:19.809693+0000","last_became_peered":"2024-09-13T14:45:19.809693+0000","last_unstale":"2024-09-13T14:48:01.744351+0000","last_undegraded":"2024-09-13T14:48:01.744351+0000","last_fullsized":"2024-09-13T14:48:01.744351+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:20.219673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":39,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.743434+0000","last_change":"2024-09-13T14:45:19.807991+0000","last_active":"2024-09-13T14:48:01.743434+0000","last_peered":"2024-09-13T14:48:01.743434+0000","last_clean":"2024-09-13T14:48:01.743434+0000","last_became_active":"2024-09-13T14:45:19.807662+0000","last_became_peered":"2024-09-13T14:45:19.807662+0000","last_unstale":"2024-09-13T14:48:01.743434+0000","last_undegraded":"2024-09-13T14:48:01.743434+0000","last_fullsized":"2024-09-13T14:48:01.743434+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:29.306108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"261'100","reported_seq":1233,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.589964+0000","last_change":"2024-09-13T14:44:54.259562+0000","last_active":"2024-09-13T14:47:40.589964+0000","last_peered":"2024-09-13T14:47:40.589964+0000","last_clean":"2024-09-13T14:47:40.589964+0000","last_became_active":"2024-09-13T14:43:14.563717+0000","last_became_peered":"2024-09-13T14:43:14.563717+0000","last_unstale":"2024-09-13T14:47:40.589964+0000","last_undegraded":"2024-09-13T14:47:40.589964+0000","last_fullsized":"2024-09-13T14:47:40.589964+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_clean_scrub_stamp":"2024-09-13T14:44:54.259482+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:43:52.729109+0000","scrub_duration":0.015265569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":1090,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.617950+0000","last_change":"2024-09-13T14:19:25.873514+0000","last_active":"2024-09-13T14:47:40.617950+0000","last_peered":"2024-09-13T14:47:40.617950+0000","last_clean":"2024-09-13T14:47:40.617950+0000","last_became_active":"2024-09-13T14:19:25.873170+0000","last_became_peered":"2024-09-13T14:19:25.873170+0000","last_unstale":"2024-09-13T14:47:40.617950+0000","last_undegraded":"2024-09-13T14:47:40.617950+0000","last_fullsized":"2024-09-13T14:47:40.617950+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:48.298530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.601591+0000","last_change":"2024-09-13T14:30:15.580011+0000","last_active":"2024-09-13T14:47:40.601591+0000","last_peered":"2024-09-13T14:47:40.601591+0000","last_clean":"2024-09-13T14:47:40.601591+0000","last_became_active":"2024-09-13T14:30:15.579610+0000","last_became_peered":"2024-09-13T14:30:15.579610+0000","last_unstale":"2024-09-13T14:47:40.601591+0000","last_undegraded":"2024-09-13T14:47:40.601591+0000","last_fullsized":"2024-09-13T14:47:40.601591+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:11:02.733771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"261'100","reported_seq":56,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.744476+0000","last_change":"2024-09-13T14:45:35.947332+0000","last_active":"2024-09-13T14:48:01.744476+0000","last_peered":"2024-09-13T14:48:01.744476+0000","last_clean":"2024-09-13T14:48:01.744476+0000","last_became_active":"2024-09-13T14:45:19.810775+0000","last_became_peered":"2024-09-13T14:45:19.810775+0000","last_unstale":"2024-09-13T14:48:01.744476+0000","last_undegraded":"2024-09-13T14:48:01.744476+0000","last_fullsized":"2024-09-13T14:48:01.744476+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:35.947265+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:43:41.569795+0000","last_clean_scrub_stamp":"2024-09-13T14:45:35.947265+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:00.606370+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":1135,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.617997+0000","last_change":"2024-09-13T14:43:07.536116+0000","last_active":"2024-09-13T14:47:40.617997+0000","last_peered":"2024-09-13T14:47:40.617997+0000","last_clean":"2024-09-13T14:47:40.617997+0000","last_became_active":"2024-09-13T14:43:07.524112+0000","last_became_peered":"2024-09-13T14:43:07.524112+0000","last_unstale":"2024-09-13T14:47:40.617997+0000","last_undegraded":"2024-09-13T14:47:40.617997+0000","last_fullsized":"2024-09-13T14:47:40.617997+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:36.947060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1132,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.601992+0000","last_change":"2024-09-13T14:43:07.530236+0000","last_active":"2024-09-13T14:47:40.601992+0000","last_peered":"2024-09-13T14:47:40.601992+0000","last_clean":"2024-09-13T14:47:40.601992+0000","last_became_active":"2024-09-13T14:43:07.529967+0000","last_became_peered":"2024-09-13T14:43:07.529967+0000","last_unstale":"2024-09-13T14:47:40.601992+0000","last_undegraded":"2024-09-13T14:47:40.601992+0000","last_fullsized":"2024-09-13T14:47:40.601992+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:27.276550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":1684,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.755879+0000","last_change":"2024-09-13T14:28:35.612469+0000","last_active":"2024-09-13T14:48:01.755879+0000","last_peered":"2024-09-13T14:48:01.755879+0000","last_clean":"2024-09-13T14:48:01.755879+0000","last_became_active":"2024-09-13T14:28:35.612093+0000","last_became_peered":"2024-09-13T14:28:35.612093+0000","last_unstale":"2024-09-13T14:48:01.755879+0000","last_undegraded":"2024-09-13T14:48:01.755879+0000","last_fullsized":"2024-09-13T14:48:01.755879+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:51.022663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1707,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.760339+0000","last_change":"2024-09-13T14:45:19.808811+0000","last_active":"2024-09-13T14:48:01.760339+0000","last_peered":"2024-09-13T14:48:01.760339+0000","last_clean":"2024-09-13T14:48:01.760339+0000","last_became_active":"2024-09-13T14:45:19.808471+0000","last_became_peered":"2024-09-13T14:45:19.808471+0000","last_unstale":"2024-09-13T14:48:01.760339+0000","last_undegraded":"2024-09-13T14:48:01.760339+0000","last_fullsized":"2024-09-13T14:48:01.760339+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:53.638161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"261'100","reported_seq":39,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.743522+0000","last_change":"2024-09-13T14:45:19.813412+0000","last_active":"2024-09-13T14:48:01.743522+0000","last_peered":"2024-09-13T14:48:01.743522+0000","last_clean":"2024-09-13T14:48:01.743522+0000","last_became_active":"2024-09-13T14:45:19.812585+0000","last_became_peered":"2024-09-13T14:45:19.812585+0000","last_unstale":"2024-09-13T14:48:01.743522+0000","last_undegraded":"2024-09-13T14:48:01.743522+0000","last_fullsized":"2024-09-13T14:48:01.743522+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.614340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.614340+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:32:07.434109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":1141,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.601685+0000","last_change":"2024-09-13T14:43:07.522950+0000","last_active":"2024-09-13T14:47:40.601685+0000","last_peered":"2024-09-13T14:47:40.601685+0000","last_clean":"2024-09-13T14:47:40.601685+0000","last_became_active":"2024-09-13T14:43:07.522566+0000","last_became_peered":"2024-09-13T14:43:07.522566+0000","last_unstale":"2024-09-13T14:47:40.601685+0000","last_undegraded":"2024-09-13T14:47:40.601685+0000","last_fullsized":"2024-09-13T14:47:40.601685+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:03.201478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":39,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.744005+0000","last_change":"2024-09-13T14:45:19.807231+0000","last_active":"2024-09-13T14:48:01.744005+0000","last_peered":"2024-09-13T14:48:01.744005+0000","last_clean":"2024-09-13T14:48:01.744005+0000","last_became_active":"2024-09-13T14:45:19.806936+0000","last_became_peered":"2024-09-13T14:45:19.806936+0000","last_unstale":"2024-09-13T14:48:01.744005+0000","last_undegraded":"2024-09-13T14:48:01.744005+0000","last_fullsized":"2024-09-13T14:48:01.744005+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:04:26.723153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.18","version":"261'100","reported_seq":111,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.618073+0000","last_change":"2024-09-13T14:44:41.685537+0000","last_active":"2024-09-13T14:47:40.618073+0000","last_peered":"2024-09-13T14:47:40.618073+0000","last_clean":"2024-09-13T14:47:40.618073+0000","last_became_active":"2024-09-13T14:43:14.558502+0000","last_became_peered":"2024-09-13T14:43:14.558502+0000","last_unstale":"2024-09-13T14:47:40.618073+0000","last_undegraded":"2024-09-13T14:47:40.618073+0000","last_fullsized":"2024-09-13T14:47:40.618073+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1134,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:41.685124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:41.685124+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:37:58.412497+0000","scrub_duration":0.0075205970000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":88,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.752164+0000","last_change":"2024-09-13T14:43:07.516845+0000","last_active":"2024-09-13T14:48:01.752164+0000","last_peered":"2024-09-13T14:48:01.752164+0000","last_clean":"2024-09-13T14:48:01.752164+0000","last_became_active":"2024-09-13T14:43:07.514974+0000","last_became_peered":"2024-09-13T14:43:07.514974+0000","last_unstale":"2024-09-13T14:48:01.752164+0000","last_undegraded":"2024-09-13T14:48:01.752164+0000","last_fullsized":"2024-09-13T14:48:01.752164+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:55.419483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1722,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.757515+0000","last_change":"2024-09-13T14:45:19.815341+0000","last_active":"2024-09-13T14:48:01.757515+0000","last_peered":"2024-09-13T14:48:01.757515+0000","last_clean":"2024-09-13T14:48:01.757515+0000","last_became_active":"2024-09-13T14:45:19.810262+0000","last_became_peered":"2024-09-13T14:45:19.810262+0000","last_unstale":"2024-09-13T14:48:01.757515+0000","last_undegraded":"2024-09-13T14:48:01.757515+0000","last_fullsized":"2024-09-13T14:48:01.757515+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:39.847653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1b","version":"261'100","reported_seq":53,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.617778+0000","last_change":"2024-09-13T14:44:52.803330+0000","last_active":"2024-09-13T14:47:40.617778+0000","last_peered":"2024-09-13T14:47:40.617778+0000","last_clean":"2024-09-13T14:47:40.617778+0000","last_became_active":"2024-09-13T14:43:14.556712+0000","last_became_peered":"2024-09-13T14:43:14.556712+0000","last_unstale":"2024-09-13T14:47:40.617778+0000","last_undegraded":"2024-09-13T14:47:40.617778+0000","last_fullsized":"2024-09-13T14:47:40.617778+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:52.803242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:52.803242+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:06.658301+0000","scrub_duration":0.0019769660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":39,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.742490+0000","last_change":"2024-09-13T14:45:19.806623+0000","last_active":"2024-09-13T14:48:01.742490+0000","last_peered":"2024-09-13T14:48:01.742490+0000","last_clean":"2024-09-13T14:48:01.742490+0000","last_became_active":"2024-09-13T14:45:19.806219+0000","last_became_peered":"2024-09-13T14:45:19.806219+0000","last_unstale":"2024-09-13T14:48:01.742490+0000","last_undegraded":"2024-09-13T14:48:01.742490+0000","last_fullsized":"2024-09-13T14:48:01.742490+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:34:41.601201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1115,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.618100+0000","last_change":"2024-09-13T14:43:32.637751+0000","last_active":"2024-09-13T14:47:40.618100+0000","last_peered":"2024-09-13T14:47:40.618100+0000","last_clean":"2024-09-13T14:47:40.618100+0000","last_became_active":"2024-09-13T14:43:32.636901+0000","last_became_peered":"2024-09-13T14:43:32.636901+0000","last_unstale":"2024-09-13T14:47:40.618100+0000","last_undegraded":"2024-09-13T14:47:40.618100+0000","last_fullsized":"2024-09-13T14:47:40.618100+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:21.942419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1a","version":"261'100","reported_seq":53,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.601490+0000","last_change":"2024-09-13T14:43:41.910041+0000","last_active":"2024-09-13T14:47:40.601490+0000","last_peered":"2024-09-13T14:47:40.601490+0000","last_clean":"2024-09-13T14:47:40.601490+0000","last_became_active":"2024-09-13T14:43:14.555083+0000","last_became_peered":"2024-09-13T14:43:14.555083+0000","last_unstale":"2024-09-13T14:47:40.601490+0000","last_undegraded":"2024-09-13T14:47:40.601490+0000","last_fullsized":"2024-09-13T14:47:40.601490+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:41.909988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:41.909988+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:42:19.954838+0000","scrub_duration":0.0012891619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":39,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.742944+0000","last_change":"2024-09-13T14:45:19.812071+0000","last_active":"2024-09-13T14:48:01.742944+0000","last_peered":"2024-09-13T14:48:01.742944+0000","last_clean":"2024-09-13T14:48:01.742944+0000","last_became_active":"2024-09-13T14:45:19.811336+0000","last_became_peered":"2024-09-13T14:45:19.811336+0000","last_unstale":"2024-09-13T14:48:01.742944+0000","last_undegraded":"2024-09-13T14:48:01.742944+0000","last_fullsized":"2024-09-13T14:48:01.742944+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:40:29.814608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1740,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.766838+0000","last_change":"2024-09-13T14:45:19.810478+0000","last_active":"2024-09-13T14:48:01.766838+0000","last_peered":"2024-09-13T14:48:01.766838+0000","last_clean":"2024-09-13T14:48:01.766838+0000","last_became_active":"2024-09-13T14:45:19.808278+0000","last_became_peered":"2024-09-13T14:45:19.808278+0000","last_unstale":"2024-09-13T14:48:01.766838+0000","last_undegraded":"2024-09-13T14:48:01.766838+0000","last_fullsized":"2024-09-13T14:48:01.766838+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:12.211089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"261'100","reported_seq":53,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.593664+0000","last_change":"2024-09-13T14:44:38.214961+0000","last_active":"2024-09-13T14:47:40.593664+0000","last_peered":"2024-09-13T14:47:40.593664+0000","last_clean":"2024-09-13T14:47:40.593664+0000","last_became_active":"2024-09-13T14:43:14.557883+0000","last_became_peered":"2024-09-13T14:43:14.557883+0000","last_unstale":"2024-09-13T14:47:40.593664+0000","last_undegraded":"2024-09-13T14:47:40.593664+0000","last_fullsized":"2024-09-13T14:47:40.593664+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:38.214865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:38.214865+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:49:36.013783+0000","scrub_duration":0.0016389639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1610,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.743668+0000","last_change":"2024-09-13T14:30:15.576921+0000","last_active":"2024-09-13T14:48:01.743668+0000","last_peered":"2024-09-13T14:48:01.743668+0000","last_clean":"2024-09-13T14:48:01.743668+0000","last_became_active":"2024-09-13T14:30:15.576607+0000","last_became_peered":"2024-09-13T14:30:15.576607+0000","last_unstale":"2024-09-13T14:48:01.743668+0000","last_undegraded":"2024-09-13T14:48:01.743668+0000","last_fullsized":"2024-09-13T14:48:01.743668+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:35.162460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1125,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.618099+0000","last_change":"2024-09-13T14:43:07.518801+0000","last_active":"2024-09-13T14:47:40.618099+0000","last_peered":"2024-09-13T14:47:40.618099+0000","last_clean":"2024-09-13T14:47:40.618099+0000","last_became_active":"2024-09-13T14:43:07.517610+0000","last_became_peered":"2024-09-13T14:43:07.517610+0000","last_unstale":"2024-09-13T14:47:40.618099+0000","last_undegraded":"2024-09-13T14:47:40.618099+0000","last_fullsized":"2024-09-13T14:47:40.618099+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:51.632254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"261'100","reported_seq":88,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.760283+0000","last_change":"2024-09-13T14:43:15.037565+0000","last_active":"2024-09-13T14:48:01.760283+0000","last_peered":"2024-09-13T14:48:01.760283+0000","last_clean":"2024-09-13T14:48:01.760283+0000","last_became_active":"2024-09-13T14:43:14.554183+0000","last_became_peered":"2024-09-13T14:43:14.554183+0000","last_unstale":"2024-09-13T14:48:01.760283+0000","last_undegraded":"2024-09-13T14:48:01.760283+0000","last_fullsized":"2024-09-13T14:48:01.760283+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.037517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.037517+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:01:35.137195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.593558+0000","last_change":"2024-09-13T14:30:15.579260+0000","last_active":"2024-09-13T14:47:40.593558+0000","last_peered":"2024-09-13T14:47:40.593558+0000","last_clean":"2024-09-13T14:47:40.593558+0000","last_became_active":"2024-09-13T14:30:15.578897+0000","last_became_peered":"2024-09-13T14:30:15.578897+0000","last_unstale":"2024-09-13T14:47:40.593558+0000","last_undegraded":"2024-09-13T14:47:40.593558+0000","last_fullsized":"2024-09-13T14:47:40.593558+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:56.737849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":88,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.759063+0000","last_change":"2024-09-13T14:43:07.521576+0000","last_active":"2024-09-13T14:48:01.759063+0000","last_peered":"2024-09-13T14:48:01.759063+0000","last_clean":"2024-09-13T14:48:01.759063+0000","last_became_active":"2024-09-13T14:43:07.520478+0000","last_became_peered":"2024-09-13T14:43:07.520478+0000","last_unstale":"2024-09-13T14:48:01.759063+0000","last_undegraded":"2024-09-13T14:48:01.759063+0000","last_fullsized":"2024-09-13T14:48:01.759063+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:53.410135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"261'100","reported_seq":379,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.618016+0000","last_change":"2024-09-13T14:44:49.775941+0000","last_active":"2024-09-13T14:47:40.618016+0000","last_peered":"2024-09-13T14:47:40.618016+0000","last_clean":"2024-09-13T14:47:40.618016+0000","last_became_active":"2024-09-13T14:43:14.560297+0000","last_became_peered":"2024-09-13T14:43:14.560297+0000","last_unstale":"2024-09-13T14:47:40.618016+0000","last_undegraded":"2024-09-13T14:47:40.618016+0000","last_fullsized":"2024-09-13T14:47:40.618016+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:49.775844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:49.775844+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:15:53.138661+0000","scrub_duration":0.012350274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":1108,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.601823+0000","last_change":"2024-09-13T14:19:25.870605+0000","last_active":"2024-09-13T14:47:40.601823+0000","last_peered":"2024-09-13T14:47:40.601823+0000","last_clean":"2024-09-13T14:47:40.601823+0000","last_became_active":"2024-09-13T14:19:25.870282+0000","last_became_peered":"2024-09-13T14:19:25.870282+0000","last_unstale":"2024-09-13T14:47:40.601823+0000","last_undegraded":"2024-09-13T14:47:40.601823+0000","last_fullsized":"2024-09-13T14:47:40.601823+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:52.514915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1122,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.602061+0000","last_change":"2024-09-13T14:43:07.526476+0000","last_active":"2024-09-13T14:47:40.602061+0000","last_peered":"2024-09-13T14:47:40.602061+0000","last_clean":"2024-09-13T14:47:40.602061+0000","last_became_active":"2024-09-13T14:43:07.525802+0000","last_became_peered":"2024-09-13T14:43:07.525802+0000","last_unstale":"2024-09-13T14:47:40.602061+0000","last_undegraded":"2024-09-13T14:47:40.602061+0000","last_fullsized":"2024-09-13T14:47:40.602061+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:23.154726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"261'100","reported_seq":95,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.751120+0000","last_change":"2024-09-13T14:45:19.807461+0000","last_active":"2024-09-13T14:48:01.751120+0000","last_peered":"2024-09-13T14:48:01.751120+0000","last_clean":"2024-09-13T14:48:01.751120+0000","last_became_active":"2024-09-13T14:45:19.807117+0000","last_became_peered":"2024-09-13T14:45:19.807117+0000","last_unstale":"2024-09-13T14:48:01.751120+0000","last_undegraded":"2024-09-13T14:48:01.751120+0000","last_fullsized":"2024-09-13T14:48:01.751120+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.955197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.955197+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:03.531555+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"1162'13767","reported_seq":13838,"reported_epoch":1162,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:48:12.795158+0000","last_change":"2024-09-13T14:45:21.161658+0000","last_active":"2024-09-13T14:48:12.795158+0000","last_peered":"2024-09-13T14:48:12.795158+0000","last_clean":"2024-09-13T14:45:19.622135+0000","last_became_active":"2024-09-13T14:45:21.090442+0000","last_became_peered":"2024-09-13T14:45:21.090442+0000","last_unstale":"2024-09-13T14:48:12.795158+0000","last_undegraded":"2024-09-13T14:48:12.795158+0000","last_fullsized":"2024-09-13T14:48:12.795158+0000","mapping_epoch":1149,"log_start":"1151'7700","ondisk_log_start":"1151'7700","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6067,"log_dups_size":0,"ondisk_log_size":6067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94306304,"num_objects":1439,"num_object_clones":0,"num_object_copies":4317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1439,"num_objects_unfound":0,"num_objects_dirty":1439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13767,"num_write_kb":486592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[1,2,7],"avail_no_missing":["1","2","7"],"object_location_counts":[{"shards":"1,2,7","objects":1439}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1085,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.605425+0000","last_change":"2024-09-13T14:19:26.131609+0000","last_active":"2024-09-13T14:47:40.605425+0000","last_peered":"2024-09-13T14:47:40.605425+0000","last_clean":"2024-09-13T14:47:40.605425+0000","last_became_active":"2024-09-13T14:19:26.131029+0000","last_became_peered":"2024-09-13T14:19:26.131029+0000","last_unstale":"2024-09-13T14:47:40.605425+0000","last_undegraded":"2024-09-13T14:47:40.605425+0000","last_fullsized":"2024-09-13T14:47:40.605425+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:57.933696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":587,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.590687+0000","last_change":"2024-09-13T14:43:32.635070+0000","last_active":"2024-09-13T14:47:40.590687+0000","last_peered":"2024-09-13T14:47:40.590687+0000","last_clean":"2024-09-13T14:47:40.590687+0000","last_became_active":"2024-09-13T14:43:32.634734+0000","last_became_peered":"2024-09-13T14:43:32.634734+0000","last_unstale":"2024-09-13T14:47:40.590687+0000","last_undegraded":"2024-09-13T14:47:40.590687+0000","last_fullsized":"2024-09-13T14:47:40.590687+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:33.107010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":1730,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.757737+0000","last_change":"2024-09-13T14:43:07.537759+0000","last_active":"2024-09-13T14:48:01.757737+0000","last_peered":"2024-09-13T14:48:01.757737+0000","last_clean":"2024-09-13T14:48:01.757737+0000","last_became_active":"2024-09-13T14:43:07.520303+0000","last_became_peered":"2024-09-13T14:43:07.520303+0000","last_unstale":"2024-09-13T14:48:01.757737+0000","last_undegraded":"2024-09-13T14:48:01.757737+0000","last_fullsized":"2024-09-13T14:48:01.757737+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:32.772570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"261'100","reported_seq":39,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.746402+0000","last_change":"2024-09-13T14:45:19.813527+0000","last_active":"2024-09-13T14:48:01.746402+0000","last_peered":"2024-09-13T14:48:01.746402+0000","last_clean":"2024-09-13T14:48:01.746402+0000","last_became_active":"2024-09-13T14:45:19.812834+0000","last_became_peered":"2024-09-13T14:45:19.812834+0000","last_unstale":"2024-09-13T14:48:01.746402+0000","last_undegraded":"2024-09-13T14:48:01.746402+0000","last_fullsized":"2024-09-13T14:48:01.746402+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.710192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.710192+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:08:55.152703+0000","scrub_duration":0.0019250910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"1162'13741","reported_seq":18811,"reported_epoch":1162,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:48:16.996373+0000","last_change":"2024-09-13T14:45:38.479094+0000","last_active":"2024-09-13T14:48:16.996373+0000","last_peered":"2024-09-13T14:48:16.996373+0000","last_clean":"2024-09-13T14:45:19.642017+0000","last_became_active":"2024-09-13T14:45:21.111093+0000","last_became_peered":"2024-09-13T14:45:21.111093+0000","last_unstale":"2024-09-13T14:48:16.996373+0000","last_undegraded":"2024-09-13T14:45:38.458706+0000","last_fullsized":"2024-09-13T14:45:20.800066+0000","mapping_epoch":1149,"log_start":"1144'5400","ondisk_log_start":"1144'5400","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":8341,"log_dups_size":0,"ondisk_log_size":8341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87490560,"num_objects":1335,"num_object_clones":0,"num_object_copies":4005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3702,"num_objects_misplaced":891,"num_objects_unfound":0,"num_objects_dirty":1335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13741,"num_write_kb":482432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2502,"num_bytes_recovered":163971072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[2,4],"avail_no_missing":["2","4","6"],"object_location_counts":[{"shards":"2,4","objects":-3258},{"shards":"2,4,6","objects":891}],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":39,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.743027+0000","last_change":"2024-09-13T14:45:19.813025+0000","last_active":"2024-09-13T14:48:01.743027+0000","last_peered":"2024-09-13T14:48:01.743027+0000","last_clean":"2024-09-13T14:48:01.743027+0000","last_became_active":"2024-09-13T14:45:19.812714+0000","last_became_peered":"2024-09-13T14:45:19.812714+0000","last_unstale":"2024-09-13T14:48:01.743027+0000","last_undegraded":"2024-09-13T14:48:01.743027+0000","last_fullsized":"2024-09-13T14:48:01.743027+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:40.671012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":88,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.751318+0000","last_change":"2024-09-13T14:43:07.522730+0000","last_active":"2024-09-13T14:48:01.751318+0000","last_peered":"2024-09-13T14:48:01.751318+0000","last_clean":"2024-09-13T14:48:01.751318+0000","last_became_active":"2024-09-13T14:43:07.517420+0000","last_became_peered":"2024-09-13T14:43:07.517420+0000","last_unstale":"2024-09-13T14:48:01.751318+0000","last_undegraded":"2024-09-13T14:48:01.751318+0000","last_fullsized":"2024-09-13T14:48:01.751318+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:21:30.265293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":1643,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.743593+0000","last_change":"2024-09-13T14:30:15.581728+0000","last_active":"2024-09-13T14:48:01.743593+0000","last_peered":"2024-09-13T14:48:01.743593+0000","last_clean":"2024-09-13T14:48:01.743593+0000","last_became_active":"2024-09-13T14:30:15.581303+0000","last_became_peered":"2024-09-13T14:30:15.581303+0000","last_unstale":"2024-09-13T14:48:01.743593+0000","last_undegraded":"2024-09-13T14:48:01.743593+0000","last_fullsized":"2024-09-13T14:48:01.743593+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:10.027991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"261'100","reported_seq":112,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.602023+0000","last_change":"2024-09-13T14:43:31.112173+0000","last_active":"2024-09-13T14:47:40.602023+0000","last_peered":"2024-09-13T14:47:40.602023+0000","last_clean":"2024-09-13T14:47:40.602023+0000","last_became_active":"2024-09-13T14:42:42.709263+0000","last_became_peered":"2024-09-13T14:42:42.709263+0000","last_unstale":"2024-09-13T14:47:40.602023+0000","last_undegraded":"2024-09-13T14:47:40.602023+0000","last_fullsized":"2024-09-13T14:47:40.602023+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:31.112111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:31.112111+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:02.707372+0000","scrub_duration":0.0015759260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"1162'13834","reported_seq":13880,"reported_epoch":1162,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:48:12.609505+0000","last_change":"2024-09-13T14:45:21.011501+0000","last_active":"2024-09-13T14:48:12.609505+0000","last_peered":"2024-09-13T14:48:12.609505+0000","last_clean":"2024-09-13T14:45:19.622974+0000","last_became_active":"2024-09-13T14:45:21.006075+0000","last_became_peered":"2024-09-13T14:45:21.006075+0000","last_unstale":"2024-09-13T14:48:12.609505+0000","last_undegraded":"2024-09-13T14:48:12.609505+0000","last_fullsized":"2024-09-13T14:48:12.609505+0000","mapping_epoch":1149,"log_start":"1146'6100","ondisk_log_start":"1146'6100","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":7734,"log_dups_size":0,"ondisk_log_size":7734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:30.090601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93323264,"num_objects":1424,"num_object_clones":0,"num_object_copies":4272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1424,"num_objects_unfound":0,"num_objects_dirty":1424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13834,"num_write_kb":488256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,2],"avail_no_missing":["5","1","2"],"object_location_counts":[{"shards":"1,2,5","objects":1424}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1683,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.755809+0000","last_change":"2024-09-13T14:28:35.611508+0000","last_active":"2024-09-13T14:48:01.755809+0000","last_peered":"2024-09-13T14:48:01.755809+0000","last_clean":"2024-09-13T14:48:01.755809+0000","last_became_active":"2024-09-13T14:28:35.610106+0000","last_became_peered":"2024-09-13T14:28:35.610106+0000","last_unstale":"2024-09-13T14:48:01.755809+0000","last_undegraded":"2024-09-13T14:48:01.755809+0000","last_fullsized":"2024-09-13T14:48:01.755809+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:11.966958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1128,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.593682+0000","last_change":"2024-09-13T14:28:35.603663+0000","last_active":"2024-09-13T14:47:40.593682+0000","last_peered":"2024-09-13T14:47:40.593682+0000","last_clean":"2024-09-13T14:47:40.593682+0000","last_became_active":"2024-09-13T14:28:35.603358+0000","last_became_peered":"2024-09-13T14:28:35.603358+0000","last_unstale":"2024-09-13T14:47:40.593682+0000","last_undegraded":"2024-09-13T14:47:40.593682+0000","last_fullsized":"2024-09-13T14:47:40.593682+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:24:05.842796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":88,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.751394+0000","last_change":"2024-09-13T14:43:07.522750+0000","last_active":"2024-09-13T14:48:01.751394+0000","last_peered":"2024-09-13T14:48:01.751394+0000","last_clean":"2024-09-13T14:48:01.751394+0000","last_became_active":"2024-09-13T14:43:07.517267+0000","last_became_peered":"2024-09-13T14:43:07.517267+0000","last_unstale":"2024-09-13T14:48:01.751394+0000","last_undegraded":"2024-09-13T14:48:01.751394+0000","last_fullsized":"2024-09-13T14:48:01.751394+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:40.217748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"261'100","reported_seq":409,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.757317+0000","last_change":"2024-09-13T14:45:19.815050+0000","last_active":"2024-09-13T14:48:01.757317+0000","last_peered":"2024-09-13T14:48:01.757317+0000","last_clean":"2024-09-13T14:48:01.757317+0000","last_became_active":"2024-09-13T14:45:19.809732+0000","last_became_peered":"2024-09-13T14:45:19.809732+0000","last_unstale":"2024-09-13T14:48:01.757317+0000","last_undegraded":"2024-09-13T14:48:01.757317+0000","last_fullsized":"2024-09-13T14:48:01.757317+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:33.574443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:33.574443+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:51:40.261036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"1162'13656","reported_seq":13760,"reported_epoch":1162,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:48:12.501916+0000","last_change":"2024-09-13T14:45:21.136109+0000","last_active":"2024-09-13T14:48:12.501916+0000","last_peered":"2024-09-13T14:48:12.501916+0000","last_clean":"2024-09-13T14:45:19.607954+0000","last_became_active":"2024-09-13T14:45:21.004300+0000","last_became_peered":"2024-09-13T14:45:21.004300+0000","last_unstale":"2024-09-13T14:48:12.501916+0000","last_undegraded":"2024-09-13T14:48:12.501916+0000","last_fullsized":"2024-09-13T14:48:12.501916+0000","mapping_epoch":1149,"log_start":"1150'7500","ondisk_log_start":"1150'7500","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6156,"log_dups_size":0,"ondisk_log_size":6156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:04:23.351582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95158272,"num_objects":1452,"num_object_clones":0,"num_object_copies":4356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1444,"num_objects_unfound":0,"num_objects_dirty":1452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13656,"num_write_kb":483456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":1835008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,6,1],"avail_no_missing":["4","1","6"],"object_location_counts":[{"shards":"1,4,6","objects":1452}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":88,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.758890+0000","last_change":"2024-09-13T14:43:07.520049+0000","last_active":"2024-09-13T14:48:01.758890+0000","last_peered":"2024-09-13T14:48:01.758890+0000","last_clean":"2024-09-13T14:48:01.758890+0000","last_became_active":"2024-09-13T14:43:07.514646+0000","last_became_peered":"2024-09-13T14:43:07.514646+0000","last_unstale":"2024-09-13T14:48:01.758890+0000","last_undegraded":"2024-09-13T14:48:01.758890+0000","last_fullsized":"2024-09-13T14:48:01.758890+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:10:35.200778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":88,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.751313+0000","last_change":"2024-09-13T14:43:07.525666+0000","last_active":"2024-09-13T14:48:01.751313+0000","last_peered":"2024-09-13T14:48:01.751313+0000","last_clean":"2024-09-13T14:48:01.751313+0000","last_became_active":"2024-09-13T14:43:07.524064+0000","last_became_peered":"2024-09-13T14:43:07.524064+0000","last_unstale":"2024-09-13T14:48:01.751313+0000","last_undegraded":"2024-09-13T14:48:01.751313+0000","last_fullsized":"2024-09-13T14:48:01.751313+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:10.270333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1142,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.618006+0000","last_change":"2024-09-13T14:45:19.804177+0000","last_active":"2024-09-13T14:47:40.618006+0000","last_peered":"2024-09-13T14:47:40.618006+0000","last_clean":"2024-09-13T14:47:40.618006+0000","last_became_active":"2024-09-13T14:45:19.803890+0000","last_became_peered":"2024-09-13T14:45:19.803890+0000","last_unstale":"2024-09-13T14:47:40.618006+0000","last_undegraded":"2024-09-13T14:47:40.618006+0000","last_fullsized":"2024-09-13T14:47:40.618006+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:25.140789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"261'100","reported_seq":56,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.743462+0000","last_change":"2024-09-13T14:45:36.965992+0000","last_active":"2024-09-13T14:48:01.743462+0000","last_peered":"2024-09-13T14:48:01.743462+0000","last_clean":"2024-09-13T14:48:01.743462+0000","last_became_active":"2024-09-13T14:45:19.811238+0000","last_became_peered":"2024-09-13T14:45:19.811238+0000","last_unstale":"2024-09-13T14:48:01.743462+0000","last_undegraded":"2024-09-13T14:48:01.743462+0000","last_fullsized":"2024-09-13T14:48:01.743462+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:36.965775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:45:36.965775+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:36:01.623251+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"1162'13470","reported_seq":13537,"reported_epoch":1162,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:48:14.540041+0000","last_change":"2024-09-13T14:45:21.036181+0000","last_active":"2024-09-13T14:48:14.540041+0000","last_peered":"2024-09-13T14:48:14.540041+0000","last_clean":"2024-09-13T14:45:19.587756+0000","last_became_active":"2024-09-13T14:45:21.029769+0000","last_became_peered":"2024-09-13T14:45:21.029769+0000","last_unstale":"2024-09-13T14:48:14.540041+0000","last_undegraded":"2024-09-13T14:48:14.540041+0000","last_fullsized":"2024-09-13T14:48:14.540041+0000","mapping_epoch":1149,"log_start":"1146'6600","ondisk_log_start":"1146'6600","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6870,"log_dups_size":0,"ondisk_log_size":6870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87031808,"num_objects":1328,"num_object_clones":0,"num_object_copies":3984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1328,"num_objects_unfound":0,"num_objects_dirty":1328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13470,"num_write_kb":473536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,1],"avail_no_missing":["0","1","6"],"object_location_counts":[{"shards":"0,1,6","objects":1328}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":41,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.618030+0000","last_change":"2024-09-13T14:45:19.806318+0000","last_active":"2024-09-13T14:47:40.618030+0000","last_peered":"2024-09-13T14:47:40.618030+0000","last_clean":"2024-09-13T14:47:40.618030+0000","last_became_active":"2024-09-13T14:45:19.806074+0000","last_became_peered":"2024-09-13T14:45:19.806074+0000","last_unstale":"2024-09-13T14:47:40.618030+0000","last_undegraded":"2024-09-13T14:47:40.618030+0000","last_fullsized":"2024-09-13T14:47:40.618030+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:21.679797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1103,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.602163+0000","last_change":"2024-09-13T14:28:35.604404+0000","last_active":"2024-09-13T14:47:40.602163+0000","last_peered":"2024-09-13T14:47:40.602163+0000","last_clean":"2024-09-13T14:47:40.602163+0000","last_became_active":"2024-09-13T14:28:35.604102+0000","last_became_peered":"2024-09-13T14:28:35.604102+0000","last_unstale":"2024-09-13T14:47:40.602163+0000","last_undegraded":"2024-09-13T14:47:40.602163+0000","last_fullsized":"2024-09-13T14:47:40.602163+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:25.546010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":1153,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.617830+0000","last_change":"2024-09-13T14:43:07.538919+0000","last_active":"2024-09-13T14:47:40.617830+0000","last_peered":"2024-09-13T14:47:40.617830+0000","last_clean":"2024-09-13T14:47:40.617830+0000","last_became_active":"2024-09-13T14:43:07.513109+0000","last_became_peered":"2024-09-13T14:43:07.513109+0000","last_unstale":"2024-09-13T14:47:40.617830+0000","last_undegraded":"2024-09-13T14:47:40.617830+0000","last_fullsized":"2024-09-13T14:47:40.617830+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:27.771720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"261'100","reported_seq":102,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.617598+0000","last_change":"2024-09-13T14:43:44.720333+0000","last_active":"2024-09-13T14:47:40.617598+0000","last_peered":"2024-09-13T14:47:40.617598+0000","last_clean":"2024-09-13T14:47:40.617598+0000","last_became_active":"2024-09-13T14:42:14.946591+0000","last_became_peered":"2024-09-13T14:42:14.946591+0000","last_unstale":"2024-09-13T14:47:40.617598+0000","last_undegraded":"2024-09-13T14:47:40.617598+0000","last_fullsized":"2024-09-13T14:47:40.617598+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:44.720258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:44.720258+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:33.281357+0000","scrub_duration":0.0018417920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"1162'13714","reported_seq":13779,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:14.560265+0000","last_change":"2024-09-13T14:43:50.794091+0000","last_active":"2024-09-13T14:48:14.560265+0000","last_peered":"2024-09-13T14:48:14.560265+0000","last_clean":"2024-09-13T14:48:14.560265+0000","last_became_active":"2024-09-13T14:43:50.793741+0000","last_became_peered":"2024-09-13T14:43:50.793741+0000","last_unstale":"2024-09-13T14:48:14.560265+0000","last_undegraded":"2024-09-13T14:48:14.560265+0000","last_fullsized":"2024-09-13T14:48:14.560265+0000","mapping_epoch":1136,"log_start":"1147'6800","ondisk_log_start":"1147'6800","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6914,"log_dups_size":0,"ondisk_log_size":6914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:15.846494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90177536,"num_objects":1376,"num_object_clones":0,"num_object_copies":4128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13714,"num_write_kb":482880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":55,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.758957+0000","last_change":"2024-09-13T14:44:03.858211+0000","last_active":"2024-09-13T14:48:01.758957+0000","last_peered":"2024-09-13T14:48:01.758957+0000","last_clean":"2024-09-13T14:48:01.758957+0000","last_became_active":"2024-09-13T14:44:03.857928+0000","last_became_peered":"2024-09-13T14:44:03.857928+0000","last_unstale":"2024-09-13T14:48:01.758957+0000","last_undegraded":"2024-09-13T14:48:01.758957+0000","last_fullsized":"2024-09-13T14:48:01.758957+0000","mapping_epoch":1140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:28.428693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":88,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.750833+0000","last_change":"2024-09-13T14:43:07.516243+0000","last_active":"2024-09-13T14:48:01.750833+0000","last_peered":"2024-09-13T14:48:01.750833+0000","last_clean":"2024-09-13T14:48:01.750833+0000","last_became_active":"2024-09-13T14:43:07.515863+0000","last_became_peered":"2024-09-13T14:43:07.515863+0000","last_unstale":"2024-09-13T14:48:01.750833+0000","last_undegraded":"2024-09-13T14:48:01.750833+0000","last_fullsized":"2024-09-13T14:48:01.750833+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:08.085298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1724,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.758618+0000","last_change":"2024-09-13T14:45:19.812713+0000","last_active":"2024-09-13T14:48:01.758618+0000","last_peered":"2024-09-13T14:48:01.758618+0000","last_clean":"2024-09-13T14:48:01.758618+0000","last_became_active":"2024-09-13T14:45:19.812203+0000","last_became_peered":"2024-09-13T14:45:19.812203+0000","last_unstale":"2024-09-13T14:48:01.758618+0000","last_undegraded":"2024-09-13T14:48:01.758618+0000","last_fullsized":"2024-09-13T14:48:01.758618+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:30:46.149347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"261'100","reported_seq":1282,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.589528+0000","last_change":"2024-09-13T14:43:42.577312+0000","last_active":"2024-09-13T14:47:40.589528+0000","last_peered":"2024-09-13T14:47:40.589528+0000","last_clean":"2024-09-13T14:47:40.589528+0000","last_became_active":"2024-09-13T14:42:14.928923+0000","last_became_peered":"2024-09-13T14:42:14.928923+0000","last_unstale":"2024-09-13T14:47:40.589528+0000","last_undegraded":"2024-09-13T14:47:40.589528+0000","last_fullsized":"2024-09-13T14:47:40.589528+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:42.577243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:42.577243+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:01.234250+0000","scrub_duration":0.001367839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"1162'13601","reported_seq":13635,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:12.479484+0000","last_change":"2024-09-13T14:43:50.793810+0000","last_active":"2024-09-13T14:48:12.479484+0000","last_peered":"2024-09-13T14:48:12.479484+0000","last_clean":"2024-09-13T14:48:12.479484+0000","last_became_active":"2024-09-13T14:43:50.793506+0000","last_became_peered":"2024-09-13T14:43:50.793506+0000","last_unstale":"2024-09-13T14:48:12.479484+0000","last_undegraded":"2024-09-13T14:48:12.479484+0000","last_fullsized":"2024-09-13T14:48:12.479484+0000","mapping_epoch":1136,"log_start":"1150'7400","ondisk_log_start":"1150'7400","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6201,"log_dups_size":0,"ondisk_log_size":6201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:50:32.061762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93519872,"num_objects":1427,"num_object_clones":0,"num_object_copies":4281,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13601,"num_write_kb":480896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1719,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.757461+0000","last_change":"2024-09-13T14:43:07.520212+0000","last_active":"2024-09-13T14:48:01.757461+0000","last_peered":"2024-09-13T14:48:01.757461+0000","last_clean":"2024-09-13T14:48:01.757461+0000","last_became_active":"2024-09-13T14:43:07.519831+0000","last_became_peered":"2024-09-13T14:43:07.519831+0000","last_unstale":"2024-09-13T14:48:01.757461+0000","last_undegraded":"2024-09-13T14:48:01.757461+0000","last_fullsized":"2024-09-13T14:48:01.757461+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:20.990736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1741,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.763942+0000","last_change":"2024-09-13T14:45:19.809505+0000","last_active":"2024-09-13T14:48:01.763942+0000","last_peered":"2024-09-13T14:48:01.763942+0000","last_clean":"2024-09-13T14:48:01.763942+0000","last_became_active":"2024-09-13T14:45:19.809181+0000","last_became_peered":"2024-09-13T14:45:19.809181+0000","last_unstale":"2024-09-13T14:48:01.763942+0000","last_undegraded":"2024-09-13T14:48:01.763942+0000","last_fullsized":"2024-09-13T14:48:01.763942+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:24.239188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1165,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.590209+0000","last_change":"2024-09-13T14:43:07.516140+0000","last_active":"2024-09-13T14:47:40.590209+0000","last_peered":"2024-09-13T14:47:40.590209+0000","last_clean":"2024-09-13T14:47:40.590209+0000","last_became_active":"2024-09-13T14:43:07.515481+0000","last_became_peered":"2024-09-13T14:43:07.515481+0000","last_unstale":"2024-09-13T14:47:40.590209+0000","last_undegraded":"2024-09-13T14:47:40.590209+0000","last_fullsized":"2024-09-13T14:47:40.590209+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:43.706214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"261'100","reported_seq":100,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.751876+0000","last_change":"2024-09-13T14:43:16.008346+0000","last_active":"2024-09-13T14:48:01.751876+0000","last_peered":"2024-09-13T14:48:01.751876+0000","last_clean":"2024-09-13T14:48:01.751876+0000","last_became_active":"2024-09-13T14:43:07.523121+0000","last_became_peered":"2024-09-13T14:43:07.523121+0000","last_unstale":"2024-09-13T14:48:01.751876+0000","last_undegraded":"2024-09-13T14:48:01.751876+0000","last_fullsized":"2024-09-13T14:48:01.751876+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.008264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.008264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:58:35.564470+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"1162'13618","reported_seq":13686,"reported_epoch":1162,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:48:14.573346+0000","last_change":"2024-09-13T14:45:21.035848+0000","last_active":"2024-09-13T14:48:14.573346+0000","last_peered":"2024-09-13T14:48:14.573346+0000","last_clean":"2024-09-13T14:45:19.637667+0000","last_became_active":"2024-09-13T14:45:21.025308+0000","last_became_peered":"2024-09-13T14:45:21.025308+0000","last_unstale":"2024-09-13T14:48:14.573346+0000","last_undegraded":"2024-09-13T14:48:14.573346+0000","last_fullsized":"2024-09-13T14:48:14.573346+0000","mapping_epoch":1149,"log_start":"1146'6700","ondisk_log_start":"1146'6700","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6918,"log_dups_size":0,"ondisk_log_size":6918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93061120,"num_objects":1420,"num_object_clones":0,"num_object_copies":4260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1420,"num_objects_unfound":0,"num_objects_dirty":1420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13618,"num_write_kb":481216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,5],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":1420}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":44,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.601971+0000","last_change":"2024-09-13T14:43:32.636990+0000","last_active":"2024-09-13T14:47:40.601971+0000","last_peered":"2024-09-13T14:47:40.601971+0000","last_clean":"2024-09-13T14:47:40.601971+0000","last_became_active":"2024-09-13T14:43:32.636589+0000","last_became_peered":"2024-09-13T14:43:32.636589+0000","last_unstale":"2024-09-13T14:47:40.601971+0000","last_undegraded":"2024-09-13T14:47:40.601971+0000","last_fullsized":"2024-09-13T14:47:40.601971+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:36:25.129769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":39,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.748681+0000","last_change":"2024-09-13T14:45:19.810888+0000","last_active":"2024-09-13T14:48:01.748681+0000","last_peered":"2024-09-13T14:48:01.748681+0000","last_clean":"2024-09-13T14:48:01.748681+0000","last_became_active":"2024-09-13T14:45:19.810486+0000","last_became_peered":"2024-09-13T14:45:19.810486+0000","last_unstale":"2024-09-13T14:48:01.748681+0000","last_undegraded":"2024-09-13T14:48:01.748681+0000","last_fullsized":"2024-09-13T14:48:01.748681+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:03.886942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":1118,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.617882+0000","last_change":"2024-09-13T14:29:42.539455+0000","last_active":"2024-09-13T14:47:40.617882+0000","last_peered":"2024-09-13T14:47:40.617882+0000","last_clean":"2024-09-13T14:47:40.617882+0000","last_became_active":"2024-09-13T14:29:42.539068+0000","last_became_peered":"2024-09-13T14:29:42.539068+0000","last_unstale":"2024-09-13T14:47:40.617882+0000","last_undegraded":"2024-09-13T14:47:40.617882+0000","last_fullsized":"2024-09-13T14:47:40.617882+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:24:06.048855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034405999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"261'100","reported_seq":1273,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.590280+0000","last_change":"2024-09-13T14:44:28.186029+0000","last_active":"2024-09-13T14:47:40.590280+0000","last_peered":"2024-09-13T14:47:40.590280+0000","last_clean":"2024-09-13T14:47:40.590280+0000","last_became_active":"2024-09-13T14:42:14.927771+0000","last_became_peered":"2024-09-13T14:42:14.927771+0000","last_unstale":"2024-09-13T14:47:40.590280+0000","last_undegraded":"2024-09-13T14:47:40.590280+0000","last_fullsized":"2024-09-13T14:47:40.590280+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:28.185949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:28.185949+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:38.363961+0000","scrub_duration":0.0018071129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"1162'13641","reported_seq":13695,"reported_epoch":1162,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:48:16.967010+0000","last_change":"2024-09-13T14:45:21.030922+0000","last_active":"2024-09-13T14:48:16.967010+0000","last_peered":"2024-09-13T14:48:16.967010+0000","last_clean":"2024-09-13T14:45:19.630214+0000","last_became_active":"2024-09-13T14:45:21.021807+0000","last_became_peered":"2024-09-13T14:45:21.021807+0000","last_unstale":"2024-09-13T14:48:16.967010+0000","last_undegraded":"2024-09-13T14:48:16.967010+0000","last_fullsized":"2024-09-13T14:48:16.967010+0000","mapping_epoch":1149,"log_start":"1144'5300","ondisk_log_start":"1144'5300","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":8341,"log_dups_size":0,"ondisk_log_size":8341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85393408,"num_objects":1303,"num_object_clones":0,"num_object_copies":3909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1303,"num_objects_unfound":0,"num_objects_dirty":1303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13641,"num_write_kb":478208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[2,6,7],"avail_no_missing":["2","6","7"],"object_location_counts":[{"shards":"2,6,7","objects":1303}],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":88,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.752120+0000","last_change":"2024-09-13T14:43:07.520176+0000","last_active":"2024-09-13T14:48:01.752120+0000","last_peered":"2024-09-13T14:48:01.752120+0000","last_clean":"2024-09-13T14:48:01.752120+0000","last_became_active":"2024-09-13T14:43:07.516748+0000","last_became_peered":"2024-09-13T14:43:07.516748+0000","last_unstale":"2024-09-13T14:48:01.752120+0000","last_undegraded":"2024-09-13T14:48:01.752120+0000","last_fullsized":"2024-09-13T14:48:01.752120+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:52:04.392821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":39,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.743111+0000","last_change":"2024-09-13T14:45:19.812353+0000","last_active":"2024-09-13T14:48:01.743111+0000","last_peered":"2024-09-13T14:48:01.743111+0000","last_clean":"2024-09-13T14:48:01.743111+0000","last_became_active":"2024-09-13T14:45:19.811743+0000","last_became_peered":"2024-09-13T14:45:19.811743+0000","last_unstale":"2024-09-13T14:48:01.743111+0000","last_undegraded":"2024-09-13T14:48:01.743111+0000","last_fullsized":"2024-09-13T14:48:01.743111+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:43.728170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"261'100","reported_seq":102,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.617660+0000","last_change":"2024-09-13T14:44:34.814157+0000","last_active":"2024-09-13T14:47:40.617660+0000","last_peered":"2024-09-13T14:47:40.617660+0000","last_clean":"2024-09-13T14:47:40.617660+0000","last_became_active":"2024-09-13T14:42:42.707803+0000","last_became_peered":"2024-09-13T14:42:42.707803+0000","last_unstale":"2024-09-13T14:47:40.617660+0000","last_undegraded":"2024-09-13T14:47:40.617660+0000","last_fullsized":"2024-09-13T14:47:40.617660+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:34.814070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:34.814070+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:38.408561+0000","scrub_duration":0.01686414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"1162'13705","reported_seq":13770,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:12.936495+0000","last_change":"2024-09-13T14:43:51.534705+0000","last_active":"2024-09-13T14:48:12.936495+0000","last_peered":"2024-09-13T14:48:12.936495+0000","last_clean":"2024-09-13T14:48:12.936495+0000","last_became_active":"2024-09-13T14:43:51.534381+0000","last_became_peered":"2024-09-13T14:43:51.534381+0000","last_unstale":"2024-09-13T14:48:12.936495+0000","last_undegraded":"2024-09-13T14:48:12.936495+0000","last_fullsized":"2024-09-13T14:48:12.936495+0000","mapping_epoch":1136,"log_start":"1151'7700","ondisk_log_start":"1151'7700","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6005,"log_dups_size":0,"ondisk_log_size":6005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:37:22.385742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92930072,"num_objects":1419,"num_object_clones":0,"num_object_copies":4257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1419,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13705,"num_write_kb":483905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1132,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.601915+0000","last_change":"2024-09-13T14:44:03.855577+0000","last_active":"2024-09-13T14:47:40.601915+0000","last_peered":"2024-09-13T14:47:40.601915+0000","last_clean":"2024-09-13T14:47:40.601915+0000","last_became_active":"2024-09-13T14:44:03.855267+0000","last_became_peered":"2024-09-13T14:44:03.855267+0000","last_unstale":"2024-09-13T14:47:40.601915+0000","last_undegraded":"2024-09-13T14:47:40.601915+0000","last_fullsized":"2024-09-13T14:47:40.601915+0000","mapping_epoch":1140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:46.148814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1103,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.605562+0000","last_change":"2024-09-13T14:29:42.545815+0000","last_active":"2024-09-13T14:47:40.605562+0000","last_peered":"2024-09-13T14:47:40.605562+0000","last_clean":"2024-09-13T14:47:40.605562+0000","last_became_active":"2024-09-13T14:29:42.545508+0000","last_became_peered":"2024-09-13T14:29:42.545508+0000","last_unstale":"2024-09-13T14:47:40.605562+0000","last_undegraded":"2024-09-13T14:47:40.605562+0000","last_fullsized":"2024-09-13T14:47:40.605562+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:39.678381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"261'100","reported_seq":106,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.751599+0000","last_change":"2024-09-13T14:43:18.988482+0000","last_active":"2024-09-13T14:48:01.751599+0000","last_peered":"2024-09-13T14:48:01.751599+0000","last_clean":"2024-09-13T14:48:01.751599+0000","last_became_active":"2024-09-13T14:43:14.556754+0000","last_became_peered":"2024-09-13T14:43:14.556754+0000","last_unstale":"2024-09-13T14:48:01.751599+0000","last_undegraded":"2024-09-13T14:48:01.751599+0000","last_fullsized":"2024-09-13T14:48:01.751599+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:18.988405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:18.988405+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:45:01.726326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"1162'13554","reported_seq":13618,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:16.946931+0000","last_change":"2024-09-13T14:43:50.796182+0000","last_active":"2024-09-13T14:48:16.946931+0000","last_peered":"2024-09-13T14:48:16.946931+0000","last_clean":"2024-09-13T14:48:16.946931+0000","last_became_active":"2024-09-13T14:43:50.795875+0000","last_became_peered":"2024-09-13T14:43:50.795875+0000","last_unstale":"2024-09-13T14:48:16.946931+0000","last_undegraded":"2024-09-13T14:48:16.946931+0000","last_fullsized":"2024-09-13T14:48:16.946931+0000","mapping_epoch":1136,"log_start":"1143'5200","ondisk_log_start":"1143'5200","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":8354,"log_dups_size":0,"ondisk_log_size":8354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:56:34.308870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79822848,"num_objects":1218,"num_object_clones":0,"num_object_copies":3654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13554,"num_write_kb":472704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1149,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.602101+0000","last_change":"2024-09-13T14:43:07.515418+0000","last_active":"2024-09-13T14:47:40.602101+0000","last_peered":"2024-09-13T14:47:40.602101+0000","last_clean":"2024-09-13T14:47:40.602101+0000","last_became_active":"2024-09-13T14:43:07.515113+0000","last_became_peered":"2024-09-13T14:43:07.515113+0000","last_unstale":"2024-09-13T14:47:40.602101+0000","last_undegraded":"2024-09-13T14:47:40.602101+0000","last_fullsized":"2024-09-13T14:47:40.602101+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:10:25.810604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1116,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.601593+0000","last_change":"2024-09-13T14:43:32.636334+0000","last_active":"2024-09-13T14:47:40.601593+0000","last_peered":"2024-09-13T14:47:40.601593+0000","last_clean":"2024-09-13T14:47:40.601593+0000","last_became_active":"2024-09-13T14:43:32.635736+0000","last_became_peered":"2024-09-13T14:43:32.635736+0000","last_unstale":"2024-09-13T14:47:40.601593+0000","last_undegraded":"2024-09-13T14:47:40.601593+0000","last_fullsized":"2024-09-13T14:47:40.601593+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:49.076014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"261'100","reported_seq":107,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.617652+0000","last_change":"2024-09-13T14:45:19.805634+0000","last_active":"2024-09-13T14:47:40.617652+0000","last_peered":"2024-09-13T14:47:40.617652+0000","last_clean":"2024-09-13T14:47:40.617652+0000","last_became_active":"2024-09-13T14:45:19.805046+0000","last_became_peered":"2024-09-13T14:45:19.805046+0000","last_unstale":"2024-09-13T14:47:40.617652+0000","last_undegraded":"2024-09-13T14:47:40.617652+0000","last_fullsized":"2024-09-13T14:47:40.617652+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:32.752408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:32.752408+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:06:53.140239+0000","scrub_duration":0.0057731559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"1162'13780","reported_seq":13849,"reported_epoch":1162,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:48:14.519069+0000","last_change":"2024-09-13T14:45:21.029184+0000","last_active":"2024-09-13T14:48:14.519069+0000","last_peered":"2024-09-13T14:48:14.519069+0000","last_clean":"2024-09-13T14:45:19.633080+0000","last_became_active":"2024-09-13T14:45:21.019809+0000","last_became_peered":"2024-09-13T14:45:21.019809+0000","last_unstale":"2024-09-13T14:48:14.519069+0000","last_undegraded":"2024-09-13T14:48:14.519069+0000","last_fullsized":"2024-09-13T14:48:14.519069+0000","mapping_epoch":1149,"log_start":"1147'6900","ondisk_log_start":"1147'6900","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6880,"log_dups_size":0,"ondisk_log_size":6880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91488256,"num_objects":1396,"num_object_clones":0,"num_object_copies":4188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1396,"num_objects_unfound":0,"num_objects_dirty":1396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13780,"num_write_kb":485632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,7,4],"avail_no_missing":["0","4","7"],"object_location_counts":[{"shards":"0,4,7","objects":1396}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1152,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.601891+0000","last_change":"2024-09-13T14:45:19.804441+0000","last_active":"2024-09-13T14:47:40.601891+0000","last_peered":"2024-09-13T14:47:40.601891+0000","last_clean":"2024-09-13T14:47:40.601891+0000","last_became_active":"2024-09-13T14:45:19.804039+0000","last_became_peered":"2024-09-13T14:45:19.804039+0000","last_unstale":"2024-09-13T14:47:40.601891+0000","last_undegraded":"2024-09-13T14:47:40.601891+0000","last_fullsized":"2024-09-13T14:47:40.601891+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:58:00.322042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":43,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.745846+0000","last_change":"2024-09-13T14:45:19.816666+0000","last_active":"2024-09-13T14:48:01.745846+0000","last_peered":"2024-09-13T14:48:01.745846+0000","last_clean":"2024-09-13T14:48:01.745846+0000","last_became_active":"2024-09-13T14:45:19.807224+0000","last_became_peered":"2024-09-13T14:45:19.807224+0000","last_unstale":"2024-09-13T14:48:01.745846+0000","last_undegraded":"2024-09-13T14:48:01.745846+0000","last_fullsized":"2024-09-13T14:48:01.745846+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:15:20.171682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"261'100","reported_seq":1292,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.589445+0000","last_change":"2024-09-13T14:45:19.798553+0000","last_active":"2024-09-13T14:47:40.589445+0000","last_peered":"2024-09-13T14:47:40.589445+0000","last_clean":"2024-09-13T14:47:40.589445+0000","last_became_active":"2024-09-13T14:45:19.798189+0000","last_became_peered":"2024-09-13T14:45:19.798189+0000","last_unstale":"2024-09-13T14:47:40.589445+0000","last_undegraded":"2024-09-13T14:47:40.589445+0000","last_fullsized":"2024-09-13T14:47:40.589445+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:29.169325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:29.169325+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:10.412666+0000","scrub_duration":0.0019358890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"1162'13824","reported_seq":13889,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:16.976624+0000","last_change":"2024-09-13T14:43:50.795029+0000","last_active":"2024-09-13T14:48:16.976624+0000","last_peered":"2024-09-13T14:48:16.976624+0000","last_clean":"2024-09-13T14:48:16.976624+0000","last_became_active":"2024-09-13T14:43:50.794715+0000","last_became_peered":"2024-09-13T14:43:50.794715+0000","last_unstale":"2024-09-13T14:48:16.976624+0000","last_undegraded":"2024-09-13T14:48:16.976624+0000","last_fullsized":"2024-09-13T14:48:16.976624+0000","mapping_epoch":1136,"log_start":"1144'5400","ondisk_log_start":"1144'5400","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":8424,"log_dups_size":0,"ondisk_log_size":8424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:21.257279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84148224,"num_objects":1284,"num_object_clones":0,"num_object_copies":3852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13824,"num_write_kb":483456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1707,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.755726+0000","last_change":"2024-09-13T14:43:07.515572+0000","last_active":"2024-09-13T14:48:01.755726+0000","last_peered":"2024-09-13T14:48:01.755726+0000","last_clean":"2024-09-13T14:48:01.755726+0000","last_became_active":"2024-09-13T14:43:07.515231+0000","last_became_peered":"2024-09-13T14:43:07.515231+0000","last_unstale":"2024-09-13T14:48:01.755726+0000","last_undegraded":"2024-09-13T14:48:01.755726+0000","last_fullsized":"2024-09-13T14:48:01.755726+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:57.966380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1145,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.618099+0000","last_change":"2024-09-13T14:45:19.804982+0000","last_active":"2024-09-13T14:47:40.618099+0000","last_peered":"2024-09-13T14:47:40.618099+0000","last_clean":"2024-09-13T14:47:40.618099+0000","last_became_active":"2024-09-13T14:45:19.804663+0000","last_became_peered":"2024-09-13T14:45:19.804663+0000","last_unstale":"2024-09-13T14:47:40.618099+0000","last_undegraded":"2024-09-13T14:47:40.618099+0000","last_fullsized":"2024-09-13T14:47:40.618099+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:43.211479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"261'100","reported_seq":382,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.601809+0000","last_change":"2024-09-13T14:43:42.881584+0000","last_active":"2024-09-13T14:47:40.601809+0000","last_peered":"2024-09-13T14:47:40.601809+0000","last_clean":"2024-09-13T14:47:40.601809+0000","last_became_active":"2024-09-13T14:43:14.562161+0000","last_became_peered":"2024-09-13T14:43:14.562161+0000","last_unstale":"2024-09-13T14:47:40.601809+0000","last_undegraded":"2024-09-13T14:47:40.601809+0000","last_fullsized":"2024-09-13T14:47:40.601809+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:42.881520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:42.881520+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:06:50.353442+0000","scrub_duration":0.0013804310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"1162'13756","reported_seq":13821,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:14.570765+0000","last_change":"2024-09-13T14:43:50.793879+0000","last_active":"2024-09-13T14:48:14.570765+0000","last_peered":"2024-09-13T14:48:14.570765+0000","last_clean":"2024-09-13T14:48:14.570765+0000","last_became_active":"2024-09-13T14:43:50.793571+0000","last_became_peered":"2024-09-13T14:43:50.793571+0000","last_unstale":"2024-09-13T14:48:14.570765+0000","last_undegraded":"2024-09-13T14:48:14.570765+0000","last_fullsized":"2024-09-13T14:48:14.570765+0000","mapping_epoch":1136,"log_start":"1147'6900","ondisk_log_start":"1147'6900","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6856,"log_dups_size":0,"ondisk_log_size":6856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:22.880360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95289344,"num_objects":1454,"num_object_clones":0,"num_object_copies":4362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13756,"num_write_kb":486720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1118,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.591002+0000","last_change":"2024-09-13T14:43:07.518120+0000","last_active":"2024-09-13T14:47:40.591002+0000","last_peered":"2024-09-13T14:47:40.591002+0000","last_clean":"2024-09-13T14:47:40.591002+0000","last_became_active":"2024-09-13T14:43:07.517361+0000","last_became_peered":"2024-09-13T14:43:07.517361+0000","last_unstale":"2024-09-13T14:47:40.591002+0000","last_undegraded":"2024-09-13T14:47:40.591002+0000","last_fullsized":"2024-09-13T14:47:40.591002+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:39.366473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":1100,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.593609+0000","last_change":"2024-09-13T14:19:25.871006+0000","last_active":"2024-09-13T14:47:40.593609+0000","last_peered":"2024-09-13T14:47:40.593609+0000","last_clean":"2024-09-13T14:47:40.593609+0000","last_became_active":"2024-09-13T14:19:25.870713+0000","last_became_peered":"2024-09-13T14:19:25.870713+0000","last_unstale":"2024-09-13T14:47:40.593609+0000","last_undegraded":"2024-09-13T14:47:40.593609+0000","last_fullsized":"2024-09-13T14:47:40.593609+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:34.372124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"261'100","reported_seq":366,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.618023+0000","last_change":"2024-09-13T14:43:32.637553+0000","last_active":"2024-09-13T14:47:40.618023+0000","last_peered":"2024-09-13T14:47:40.618023+0000","last_clean":"2024-09-13T14:47:40.618023+0000","last_became_active":"2024-09-13T14:43:32.636545+0000","last_became_peered":"2024-09-13T14:43:32.636545+0000","last_unstale":"2024-09-13T14:47:40.618023+0000","last_undegraded":"2024-09-13T14:47:40.618023+0000","last_fullsized":"2024-09-13T14:47:40.618023+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:15:42.069756+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"1162'13534","reported_seq":13599,"reported_epoch":1162,"state":"active+recovery_wait+remapped","last_fresh":"2024-09-13T14:48:12.916114+0000","last_change":"2024-09-13T14:45:21.130410+0000","last_active":"2024-09-13T14:48:12.916114+0000","last_peered":"2024-09-13T14:48:12.916114+0000","last_clean":"2024-09-13T14:45:19.587556+0000","last_became_active":"2024-09-13T14:45:21.100167+0000","last_became_peered":"2024-09-13T14:45:21.100167+0000","last_unstale":"2024-09-13T14:48:12.916114+0000","last_undegraded":"2024-09-13T14:48:12.916114+0000","last_fullsized":"2024-09-13T14:48:12.916114+0000","mapping_epoch":1149,"log_start":"1150'7477","ondisk_log_start":"1150'7477","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6057,"log_dups_size":0,"ondisk_log_size":6057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97124352,"num_objects":1482,"num_object_clones":0,"num_object_copies":4446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7077,"num_objects_unfound":0,"num_objects_dirty":1482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13534,"num_write_kb":480512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[1,2,5],"avail_no_missing":["1","2","5"],"object_location_counts":[{"shards":"1,2,5","objects":-5595}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1169,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.605523+0000","last_change":"2024-09-13T14:45:19.804868+0000","last_active":"2024-09-13T14:47:40.605523+0000","last_peered":"2024-09-13T14:47:40.605523+0000","last_clean":"2024-09-13T14:47:40.605523+0000","last_became_active":"2024-09-13T14:45:19.804520+0000","last_became_peered":"2024-09-13T14:45:19.804520+0000","last_unstale":"2024-09-13T14:47:40.605523+0000","last_undegraded":"2024-09-13T14:47:40.605523+0000","last_fullsized":"2024-09-13T14:47:40.605523+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:17:53.924629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":1146,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.590327+0000","last_change":"2024-09-13T14:43:07.533955+0000","last_active":"2024-09-13T14:47:40.590327+0000","last_peered":"2024-09-13T14:47:40.590327+0000","last_clean":"2024-09-13T14:47:40.590327+0000","last_became_active":"2024-09-13T14:43:07.516816+0000","last_became_peered":"2024-09-13T14:43:07.516816+0000","last_unstale":"2024-09-13T14:47:40.590327+0000","last_undegraded":"2024-09-13T14:47:40.590327+0000","last_fullsized":"2024-09-13T14:47:40.590327+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:26.232184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"261'100","reported_seq":43,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.746192+0000","last_change":"2024-09-13T14:45:19.821649+0000","last_active":"2024-09-13T14:48:01.746192+0000","last_peered":"2024-09-13T14:48:01.746192+0000","last_clean":"2024-09-13T14:48:01.746192+0000","last_became_active":"2024-09-13T14:45:19.812346+0000","last_became_peered":"2024-09-13T14:45:19.812346+0000","last_unstale":"2024-09-13T14:48:01.746192+0000","last_undegraded":"2024-09-13T14:48:01.746192+0000","last_fullsized":"2024-09-13T14:48:01.746192+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.597568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.597568+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:45.218892+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"1162'13595","reported_seq":13659,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:12.912158+0000","last_change":"2024-09-13T14:43:51.537117+0000","last_active":"2024-09-13T14:48:12.912158+0000","last_peered":"2024-09-13T14:48:12.912158+0000","last_clean":"2024-09-13T14:48:12.912158+0000","last_became_active":"2024-09-13T14:43:51.536532+0000","last_became_peered":"2024-09-13T14:43:51.536532+0000","last_unstale":"2024-09-13T14:48:12.912158+0000","last_undegraded":"2024-09-13T14:48:12.912158+0000","last_fullsized":"2024-09-13T14:48:12.912158+0000","mapping_epoch":1136,"log_start":"1151'7500","ondisk_log_start":"1151'7500","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6095,"log_dups_size":0,"ondisk_log_size":6095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:41.713211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89849856,"num_objects":1371,"num_object_clones":0,"num_object_copies":4113,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13595,"num_write_kb":478912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1087,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.617925+0000","last_change":"2024-09-13T14:19:26.887839+0000","last_active":"2024-09-13T14:47:40.617925+0000","last_peered":"2024-09-13T14:47:40.617925+0000","last_clean":"2024-09-13T14:47:40.617925+0000","last_became_active":"2024-09-13T14:19:26.887498+0000","last_became_peered":"2024-09-13T14:19:26.887498+0000","last_unstale":"2024-09-13T14:47:40.617925+0000","last_undegraded":"2024-09-13T14:47:40.617925+0000","last_fullsized":"2024-09-13T14:47:40.617925+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:24.475263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":1117,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.590624+0000","last_change":"2024-09-13T14:28:35.608607+0000","last_active":"2024-09-13T14:47:40.590624+0000","last_peered":"2024-09-13T14:47:40.590624+0000","last_clean":"2024-09-13T14:47:40.590624+0000","last_became_active":"2024-09-13T14:28:35.608254+0000","last_became_peered":"2024-09-13T14:28:35.608254+0000","last_unstale":"2024-09-13T14:47:40.590624+0000","last_undegraded":"2024-09-13T14:47:40.590624+0000","last_fullsized":"2024-09-13T14:47:40.590624+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:24.019173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"261'100","reported_seq":53,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.593474+0000","last_change":"2024-09-13T14:44:31.191431+0000","last_active":"2024-09-13T14:47:40.593474+0000","last_peered":"2024-09-13T14:47:40.593474+0000","last_clean":"2024-09-13T14:47:40.593474+0000","last_became_active":"2024-09-13T14:43:14.557376+0000","last_became_peered":"2024-09-13T14:43:14.557376+0000","last_unstale":"2024-09-13T14:47:40.593474+0000","last_undegraded":"2024-09-13T14:47:40.593474+0000","last_fullsized":"2024-09-13T14:47:40.593474+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.191326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.191326+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:15:27.352698+0000","scrub_duration":0.0025317360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1107,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.590907+0000","last_change":"2024-09-13T14:28:35.603724+0000","last_active":"2024-09-13T14:47:40.590907+0000","last_peered":"2024-09-13T14:47:40.590907+0000","last_clean":"2024-09-13T14:47:40.590907+0000","last_became_active":"2024-09-13T14:28:35.603316+0000","last_became_peered":"2024-09-13T14:28:35.603316+0000","last_unstale":"2024-09-13T14:47:40.590907+0000","last_undegraded":"2024-09-13T14:47:40.590907+0000","last_fullsized":"2024-09-13T14:47:40.590907+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:47.668223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":39,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.743380+0000","last_change":"2024-09-13T14:45:19.811466+0000","last_active":"2024-09-13T14:48:01.743380+0000","last_peered":"2024-09-13T14:48:01.743380+0000","last_clean":"2024-09-13T14:48:01.743380+0000","last_became_active":"2024-09-13T14:45:19.810617+0000","last_became_peered":"2024-09-13T14:45:19.810617+0000","last_unstale":"2024-09-13T14:48:01.743380+0000","last_undegraded":"2024-09-13T14:48:01.743380+0000","last_fullsized":"2024-09-13T14:48:01.743380+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:16.246669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"261'100","reported_seq":53,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.617742+0000","last_change":"2024-09-13T14:44:50.805306+0000","last_active":"2024-09-13T14:47:40.617742+0000","last_peered":"2024-09-13T14:47:40.617742+0000","last_clean":"2024-09-13T14:47:40.617742+0000","last_became_active":"2024-09-13T14:43:14.562922+0000","last_became_peered":"2024-09-13T14:43:14.562922+0000","last_unstale":"2024-09-13T14:47:40.617742+0000","last_undegraded":"2024-09-13T14:47:40.617742+0000","last_fullsized":"2024-09-13T14:47:40.617742+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:50.805210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:50.805210+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:28:38.529599+0000","scrub_duration":0.0042900009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1099,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.589525+0000","last_change":"2024-09-13T14:19:25.873088+0000","last_active":"2024-09-13T14:47:40.589525+0000","last_peered":"2024-09-13T14:47:40.589525+0000","last_clean":"2024-09-13T14:47:40.589525+0000","last_became_active":"2024-09-13T14:19:25.872781+0000","last_became_peered":"2024-09-13T14:19:25.872781+0000","last_unstale":"2024-09-13T14:47:40.589525+0000","last_undegraded":"2024-09-13T14:47:40.589525+0000","last_fullsized":"2024-09-13T14:47:40.589525+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:05.607382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1707,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.757605+0000","last_change":"2024-09-13T14:45:19.813876+0000","last_active":"2024-09-13T14:48:01.757605+0000","last_peered":"2024-09-13T14:48:01.757605+0000","last_clean":"2024-09-13T14:48:01.757605+0000","last_became_active":"2024-09-13T14:45:19.809136+0000","last_became_peered":"2024-09-13T14:45:19.809136+0000","last_unstale":"2024-09-13T14:48:01.757605+0000","last_undegraded":"2024-09-13T14:48:01.757605+0000","last_fullsized":"2024-09-13T14:48:01.757605+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:50.013528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"261'100","reported_seq":113,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.751733+0000","last_change":"2024-09-13T14:45:19.810654+0000","last_active":"2024-09-13T14:48:01.751733+0000","last_peered":"2024-09-13T14:48:01.751733+0000","last_clean":"2024-09-13T14:48:01.751733+0000","last_became_active":"2024-09-13T14:45:19.810066+0000","last_became_peered":"2024-09-13T14:45:19.810066+0000","last_unstale":"2024-09-13T14:48:01.751733+0000","last_undegraded":"2024-09-13T14:48:01.751733+0000","last_fullsized":"2024-09-13T14:48:01.751733+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:19.978464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:19.978464+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:33.639332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1754,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.755480+0000","last_change":"2024-09-13T14:45:19.809166+0000","last_active":"2024-09-13T14:48:01.755480+0000","last_peered":"2024-09-13T14:48:01.755480+0000","last_clean":"2024-09-13T14:48:01.755480+0000","last_became_active":"2024-09-13T14:45:19.808631+0000","last_became_peered":"2024-09-13T14:45:19.808631+0000","last_unstale":"2024-09-13T14:48:01.755480+0000","last_undegraded":"2024-09-13T14:48:01.755480+0000","last_fullsized":"2024-09-13T14:48:01.755480+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:38.822245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1124,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.617914+0000","last_change":"2024-09-13T14:45:19.804229+0000","last_active":"2024-09-13T14:47:40.617914+0000","last_peered":"2024-09-13T14:47:40.617914+0000","last_clean":"2024-09-13T14:47:40.617914+0000","last_became_active":"2024-09-13T14:45:19.803929+0000","last_became_peered":"2024-09-13T14:45:19.803929+0000","last_unstale":"2024-09-13T14:47:40.617914+0000","last_undegraded":"2024-09-13T14:47:40.617914+0000","last_fullsized":"2024-09-13T14:47:40.617914+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:33:53.675568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"261'100","reported_seq":53,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.617564+0000","last_change":"2024-09-13T14:44:42.654506+0000","last_active":"2024-09-13T14:47:40.617564+0000","last_peered":"2024-09-13T14:47:40.617564+0000","last_clean":"2024-09-13T14:47:40.617564+0000","last_became_active":"2024-09-13T14:43:14.563451+0000","last_became_peered":"2024-09-13T14:43:14.563451+0000","last_unstale":"2024-09-13T14:47:40.617564+0000","last_undegraded":"2024-09-13T14:47:40.617564+0000","last_fullsized":"2024-09-13T14:47:40.617564+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:42.654455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:42.654455+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:57:32.251597+0000","scrub_duration":0.001494097,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":1118,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.590861+0000","last_change":"2024-09-13T14:29:42.541093+0000","last_active":"2024-09-13T14:47:40.590861+0000","last_peered":"2024-09-13T14:47:40.590861+0000","last_clean":"2024-09-13T14:47:40.590861+0000","last_became_active":"2024-09-13T14:29:42.540671+0000","last_became_peered":"2024-09-13T14:29:42.540671+0000","last_unstale":"2024-09-13T14:47:40.590861+0000","last_undegraded":"2024-09-13T14:47:40.590861+0000","last_fullsized":"2024-09-13T14:47:40.590861+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:45.722376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1645,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.746337+0000","last_change":"2024-09-13T14:43:07.518679+0000","last_active":"2024-09-13T14:48:01.746337+0000","last_peered":"2024-09-13T14:48:01.746337+0000","last_clean":"2024-09-13T14:48:01.746337+0000","last_became_active":"2024-09-13T14:43:07.518333+0000","last_became_peered":"2024-09-13T14:43:07.518333+0000","last_unstale":"2024-09-13T14:48:01.746337+0000","last_undegraded":"2024-09-13T14:48:01.746337+0000","last_fullsized":"2024-09-13T14:48:01.746337+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:20.616274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"261'100","reported_seq":415,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.764049+0000","last_change":"2024-09-13T14:45:19.809268+0000","last_active":"2024-09-13T14:48:01.764049+0000","last_peered":"2024-09-13T14:48:01.764049+0000","last_clean":"2024-09-13T14:48:01.764049+0000","last_became_active":"2024-09-13T14:45:19.808772+0000","last_became_peered":"2024-09-13T14:45:19.808772+0000","last_unstale":"2024-09-13T14:48:01.764049+0000","last_undegraded":"2024-09-13T14:48:01.764049+0000","last_fullsized":"2024-09-13T14:48:01.764049+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:14.654603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:14.654603+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:40.964553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":1450115096,"num_objects":22128,"num_object_clones":0,"num_object_copies":66384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3702,"num_objects_misplaced":17722,"num_objects_unfound":0,"num_objects_dirty":22128,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":218790,"num_write_kb":7709313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2532,"num_bytes_recovered":165937152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4844302336,"data_stored":4844290120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112213,"ondisk_log_size":112213,"up":48,"acting":47,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":354515,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430578,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":876276,"kb_used_data":570604,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92868876,"statfs":{"total":95995035648,"available":95097729024,"internally_reserved":0,"allocated":584298496,"data_stored":580730414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626807,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1226284,"kb_used_data":920612,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92518868,"statfs":{"total":95995035648,"available":94739320832,"internally_reserved":0,"allocated":942706688,"data_stored":939146159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823036,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":781220,"kb_used_data":479708,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":92963932,"statfs":{"total":95995035648,"available":95195066368,"internally_reserved":0,"allocated":491220992,"data_stored":487664993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019264,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":885100,"kb_used_data":587748,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":92860052,"statfs":{"total":95995035648,"available":95088693248,"internally_reserved":0,"allocated":601853952,"data_stored":598293448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059592922,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":271960,"kb_used_data":67280,"kb_used_omap":26,"kb_used_meta":204645,"kb_avail":93473192,"statfs":{"total":95995035648,"available":95716548608,"internally_reserved":0,"allocated":68894720,"data_stored":65349916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":209556640},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":591,"seq":2538325672161,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1076512,"kb_used_data":673176,"kb_used_omap":41,"kb_used_meta":403286,"kb_avail":92668640,"statfs":{"total":95995035648,"available":94892687360,"internally_reserved":0,"allocated":689332224,"data_stored":685780294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42651,"internal_metadata":412965221},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,0,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":557,"seq":2392296784103,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1234076,"kb_used_data":850644,"kb_used_omap":32,"kb_used_meta":383391,"kb_avail":92511076,"statfs":{"total":95995035648,"available":94731341824,"internally_reserved":0,"allocated":871059456,"data_stored":867510525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33752,"internal_metadata":392592424},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":524,"seq":2250562863342,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1063008,"kb_used_data":724760,"kb_used_omap":41,"kb_used_meta":338070,"kb_avail":92682144,"statfs":{"total":95995035648,"available":94906515456,"internally_reserved":0,"allocated":742154240,"data_stored":738601794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42694,"internal_metadata":346183994},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":720506880,"data_stored":720502808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":843386880,"data_stored":843382808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":675610624,"data_stored":675610624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57868288,"data_stored":57868288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":577372160,"data_stored":577372160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":467795968,"data_stored":467795968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":929632256,"data_stored":929628184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":572129280,"data_stored":572129280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:48:19.470 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:19.495 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:19.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:19.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:19.597 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:19.632 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:19.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:19.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:19.733 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:19.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:19.783 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:19.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:19.884 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:19.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:19.929 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:19.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:19.976 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:19 smithi184 ceph-mon[92527]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:48:19.977 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:19 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:48:19.977 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:19 smithi184 ceph-mon[92527]: osdmap e1163: 8 total, 8 up, 8 in 2024-09-13T14:48:19.998 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:19 smithi151 ceph-mon[104450]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:48:19.998 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:19 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:48:19.998 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:19 smithi151 ceph-mon[104450]: osdmap e1163: 8 total, 8 up, 8 in 2024-09-13T14:48:19.999 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:19 smithi151 ceph-mon[108966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:48:19.999 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:19 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:48:19.999 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:19 smithi151 ceph-mon[108966]: osdmap e1163: 8 total, 8 up, 8 in 2024-09-13T14:48:20.030 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:20.063 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:20.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:20.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:20.165 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:20.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:20.194 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:20.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:20.241 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:48:20.241 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [], 2024-09-13T14:48:20.242 INFO:teuthology.orchestra.run.smithi184.stdout: "complaint_time": 30, 2024-09-13T14:48:20.242 INFO:teuthology.orchestra.run.smithi184.stdout: "num_blocked_ops": 0 2024-09-13T14:48:20.242 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:48:20.242 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:48:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:48:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:48:20.296 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:20.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:20.320 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:20.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:20.421 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:20.466 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:20.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:20.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:20.568 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:20.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:20.593 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:20.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:20.694 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:20.717 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:20.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:20.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:20.819 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:20.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:20.843 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:20.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:20.944 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:20.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:20.970 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:20.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:20.972 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:20 smithi151 ceph-mon[104450]: pgmap v1636: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.3 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 571 op/s; 3702/62685 objects degraded (5.906%); 17157/62685 objects misplaced (27.370%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:48:20.972 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:20 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 3702/62685 objects degraded (5.906%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:20.972 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:20 smithi151 ceph-mon[108966]: pgmap v1636: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.3 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 571 op/s; 3702/62685 objects degraded (5.906%); 17157/62685 objects misplaced (27.370%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:48:20.973 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:20 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 3702/62685 objects degraded (5.906%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:21.072 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:21.074 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T14:48:21.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:20 smithi184 ceph-mon[92527]: pgmap v1636: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.3 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 571 op/s; 3702/62685 objects degraded (5.906%); 17157/62685 objects misplaced (27.370%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:48:21.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:20 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 3702/62685 objects degraded (5.906%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:21.131 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:21.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:21.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:21.233 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:21.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:21.279 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:21.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:21.381 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:21.423 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:48:21.427 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:21.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:21.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:21.529 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:21.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:21.555 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:21.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:21.657 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:21.697 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:21.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:21.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:21.745 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:48:21.746 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:48:21.781 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:48:21.782 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:48:21.799 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:21.817 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:48:21.817 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:48:21.825 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:21.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:21.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:21.850 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:48:21.851 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:48:21.883 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:48:21.883 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:48:21.919 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:48:21.919 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:48:21.920 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:21 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/4075869875' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:21.920 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:21 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/4075869875' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:21.927 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:21.952 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:21.953 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:48:21.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:21.954 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:48:21.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:21.983 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:48:21.984 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:48:22.040 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:48:22.055 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:22.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:21 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/4075869875' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:22.135 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:22.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:22.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:22.241 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:22.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:22.282 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:22.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:22.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:22.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:48:22.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:22.438 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:22.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:22.539 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:22.596 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:22.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:22.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:22.698 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:22.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:22.755 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:22.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:22.856 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:22.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:22.904 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:22.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:22.948 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:22.964 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:23.005 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:23.030 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:23.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:23.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:23.091 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:22 smithi151 ceph-mon[104450]: pgmap v1637: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.2 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 578 op/s; 3702/59883 objects degraded (6.182%); 16609/59883 objects misplaced (27.736%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:48:23.091 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:22 smithi151 ceph-mon[108966]: pgmap v1637: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.2 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 578 op/s; 3702/59883 objects degraded (6.182%); 16609/59883 objects misplaced (27.736%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:48:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:22 smithi184 ceph-mon[92527]: pgmap v1637: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.2 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 578 op/s; 3702/59883 objects degraded (6.182%); 16609/59883 objects misplaced (27.736%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:48:23.132 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:23.157 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:23.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:23.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:23.258 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:23.283 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:23.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:23.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:23.385 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:23.406 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:23.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:23.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:23.430 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:48:23.430 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:48:23.508 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:23.559 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:23.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:23.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:23.661 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:23.699 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:23.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:23.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:23.801 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:23.822 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:23.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:23.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:23.924 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:23.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:23.946 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:23.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:24.047 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:24.067 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:24.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:24.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:24.125 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:48:24.170 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:24.193 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:24.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:24.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:24.295 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:24.318 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:24.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:24.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:24.419 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:24.443 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:24.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:24.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:24.545 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:24.570 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:24.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:24.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:24.672 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:24.696 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:24.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:24.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:24.798 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:24.816 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:48:24.832 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:24.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:24.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:24.855 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:48:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:48:24] "GET /metrics HTTP/1.1" 200 36658 "" "Prometheus/2.51.0" 2024-09-13T14:48:24.934 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:24.957 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:24.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:24.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:25.059 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:25.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:25.084 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:25.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:25.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:24 smithi151 ceph-mon[104450]: pgmap v1638: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.1 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 589 op/s; 3805/53589 objects degraded (7.100%); 15657/53589 objects misplaced (29.217%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:48:25.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:24 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 3805/53589 objects degraded (7.100%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:25.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:24 smithi151 ceph-mon[108966]: pgmap v1638: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.1 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 589 op/s; 3805/53589 objects degraded (7.100%); 15657/53589 objects misplaced (29.217%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:48:25.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:24 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 3805/53589 objects degraded (7.100%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:25.184 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:25.211 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:25.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:25.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:25.312 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:25.336 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:25.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:25.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:25.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:24 smithi184 ceph-mon[92527]: pgmap v1638: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.1 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 589 op/s; 3805/53589 objects degraded (7.100%); 15657/53589 objects misplaced (29.217%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:48:25.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:24 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 3805/53589 objects degraded (7.100%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:25.437 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:25.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:25.460 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:25.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:25.561 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:25.586 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:25.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:25.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:25.687 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:25.724 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:25.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:25.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:25.826 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:25.850 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:25.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:25.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:25.952 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:25.976 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:25.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:25.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:26.077 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:26.101 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:26.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:26.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:26.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:26.243 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:26.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:26.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:26.345 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:26.386 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:26.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:26.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:26.487 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:26.516 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:26.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:26.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:26.618 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:26.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:26.643 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:26.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:26.744 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:26.783 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:26.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:26.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:26.885 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:26.907 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:26.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:26.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:27.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:27.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:27.056 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:27.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:27.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:26 smithi151 ceph-mon[108966]: pgmap v1639: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.0 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 595 op/s; 3805/50748 objects degraded (7.498%); 15081/50748 objects misplaced (29.717%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:48:27.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:26 smithi151 ceph-mon[104450]: pgmap v1639: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.0 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 595 op/s; 3805/50748 objects degraded (7.498%); 15081/50748 objects misplaced (29.717%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:48:27.161 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:27.215 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:27.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:27.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:27.317 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:27.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:26 smithi184 ceph-mon[92527]: pgmap v1639: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1.0 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 595 op/s; 3805/50748 objects degraded (7.498%); 15081/50748 objects misplaced (29.717%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:48:27.355 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:27.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:27.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:27.457 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:27.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:27.500 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:27.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:27.602 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:27.627 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:27.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:27.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:27.729 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:27.771 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:27.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:27.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:27.879 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:27.918 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:27.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:27.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:28.020 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:28.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:28.069 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:28.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:28.170 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:28.196 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:28.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:28.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:28.298 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:28.323 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:28.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:28.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:28.426 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:28.468 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:28.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:28.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:28.571 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:28.594 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:28.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:28.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:28.696 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:28.720 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:28.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:28.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:28.821 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:28.868 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:28.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:28.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:28.970 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:28.994 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:28.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:28.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:29.096 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:29.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:28 smithi184 ceph-mon[92527]: pgmap v1640: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1011 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 595 op/s; 3924/48591 objects degraded (8.076%); 14714/48591 objects misplaced (30.281%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:48:29.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:28 smithi151 ceph-mon[104450]: pgmap v1640: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1011 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 595 op/s; 3924/48591 objects degraded (8.076%); 14714/48591 objects misplaced (30.281%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:48:29.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:28 smithi151 ceph-mon[108966]: pgmap v1640: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 1011 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 595 op/s; 3924/48591 objects degraded (8.076%); 14714/48591 objects misplaced (30.281%); 1.1 MiB/s, 17 objects/s recovering 2024-09-13T14:48:29.140 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:29.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:29.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:29.193 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:29.197 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:29.244 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:29.267 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:29.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:29.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:29.369 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:29.394 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:29.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:29.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:29.496 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:29.539 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:29.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:29.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:29.641 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:29.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:29.683 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:29.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:29.784 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:29.808 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:29.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:29.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:29.910 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:29.932 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:29.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:29.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:29.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:29 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 3924/48591 objects degraded (8.076%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:29.939 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:29 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 3924/48591 objects degraded (8.076%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:30.034 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:30.078 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:30.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:30.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:30.180 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:30.202 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:30.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:30.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:30.214 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:29 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 3924/48591 objects degraded (8.076%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:30.304 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:30.345 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:30.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:30.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:30.447 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:30.468 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:30.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:30.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:30.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:30.604 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:30.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:48:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:48:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:48:30.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:30.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:30.706 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:30.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:30.744 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:30.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:30.846 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:30.867 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:30.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:30.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:30.970 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:31.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:31.002 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:31.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:31.104 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:31.126 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:31.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:31.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:31.229 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:31.255 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:31.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:31.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:31.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:30 smithi184 ceph-mon[92527]: pgmap v1641: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 935 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 551 op/s; 3924/44937 objects degraded (8.732%); 14189/44937 objects misplaced (31.575%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:48:31.357 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:31.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:30 smithi151 ceph-mon[104450]: pgmap v1641: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 935 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 551 op/s; 3924/44937 objects degraded (8.732%); 14189/44937 objects misplaced (31.575%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:48:31.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:30 smithi151 ceph-mon[108966]: pgmap v1641: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 935 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 551 op/s; 3924/44937 objects degraded (8.732%); 14189/44937 objects misplaced (31.575%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:48:31.394 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:31.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:31.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:31.440 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:48:31.445 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T14:48:31.497 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:31.519 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:31.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:31.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:31.621 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:31.660 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:31.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:31.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:31.762 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:31.792 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:31.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:31.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:31.894 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:31.927 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:31.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:31.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:32.029 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:32.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:32.054 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:32.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:32.155 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:32.179 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:32.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:32.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:32.281 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:32.290 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:31 smithi151 ceph-mon[104450]: from='client.45701 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:48:32.291 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:48:32.291 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:32.291 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:31 smithi151 ceph-mon[108966]: from='client.45701 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:48:32.291 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:48:32.291 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:32.322 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:32.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:32.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:32.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:31 smithi184 ceph-mon[92527]: from='client.45701 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:48:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:48:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:32.424 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:32.463 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:32.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:32.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:32.565 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:32.605 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:32.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:32.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:32.707 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:32.743 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:32.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:32.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:32.845 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:32.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:32.870 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:32.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:32.971 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:33.013 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:33.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:33.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:33.114 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:33.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:33.139 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:33.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:33.240 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:33.274 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:33.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:33.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:33.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:32 smithi184 ceph-mon[92527]: pgmap v1642: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 873 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 575 op/s; 3924/41961 objects degraded (9.352%); 13606/41961 objects misplaced (32.425%); 954 KiB/s, 14 objects/s recovering 2024-09-13T14:48:33.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:32 smithi151 ceph-mon[104450]: pgmap v1642: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 873 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 575 op/s; 3924/41961 objects degraded (9.352%); 13606/41961 objects misplaced (32.425%); 954 KiB/s, 14 objects/s recovering 2024-09-13T14:48:33.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:32 smithi151 ceph-mon[108966]: pgmap v1642: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 873 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 575 op/s; 3924/41961 objects degraded (9.352%); 13606/41961 objects misplaced (32.425%); 954 KiB/s, 14 objects/s recovering 2024-09-13T14:48:33.376 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:33.418 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:33.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:33.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:33.520 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:33.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:33.563 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:33.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:33.664 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:33.700 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:33.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:33.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:33.802 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:33.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:33.828 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:33.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:33.930 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:33.981 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:33.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:33.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:34.083 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:34.131 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:34.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:34.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:34.233 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:34.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:34.274 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:34.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:34.375 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:34.398 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:34.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:34.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:34.500 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:34.525 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:34.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:34.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:34.627 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:34.650 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:34.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:34.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:34.752 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:34.796 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:34.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:34.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:34.805 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:48:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:48:34] "GET /metrics HTTP/1.1" 200 36661 "" "Prometheus/2.51.0" 2024-09-13T14:48:34.898 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:34.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:34.944 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:34.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:35.045 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:35.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:35.069 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:35.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:35.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:34 smithi151 ceph-mon[104450]: pgmap v1643: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 749 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 663 op/s; 3998/35994 objects degraded (11.107%); 12669/35994 objects misplaced (35.198%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:48:35.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:34 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 3998/35994 objects degraded (11.107%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:35.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:34 smithi151 ceph-mon[108966]: pgmap v1643: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 749 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 663 op/s; 3998/35994 objects degraded (11.107%); 12669/35994 objects misplaced (35.198%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:48:35.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:34 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 3998/35994 objects degraded (11.107%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:35.170 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:35.192 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:35.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:35.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:35.294 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:35.317 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:35.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:35.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:35.348 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:48:35.349 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:48:35.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:34 smithi184 ceph-mon[92527]: pgmap v1643: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 749 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 663 op/s; 3998/35994 objects degraded (11.107%); 12669/35994 objects misplaced (35.198%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:48:35.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:34 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 3998/35994 objects degraded (11.107%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:35.387 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:48:35.387 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:48:35.419 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:35.425 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:48:35.425 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:48:35.466 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:48:35.466 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:48:35.475 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:35.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:35.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:35.497 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:48:35.497 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:48:35.529 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:48:35.529 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:48:35.561 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:48:35.561 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:48:35.578 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:35.596 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:48:35.597 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:48:35.620 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:35.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:35.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:35.722 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:35.766 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:35.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:35.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:35.868 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:35.892 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:35.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:35.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:35.903 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-13T14:48:35.994 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:36.057 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:36.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:36.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:36.159 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:36.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:36.201 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:36.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:36.239 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:35 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/231161223' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:36.240 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:35 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3229279543' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:48:36.240 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:35 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/231161223' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:36.240 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:35 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3229279543' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:48:36.303 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:36.340 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:36.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:36.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:36.356 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:35 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/231161223' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:36.356 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:35 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3229279543' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:48:36.442 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:36.465 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:36.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:36.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:36.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:36.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:36.615 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:36.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:36.716 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:36.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:36.742 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:36.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:36.843 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:36.867 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:36.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:36.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:36.969 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:36.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:36.994 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:36.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:37.095 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:37.142 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:37.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:37.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:37.236 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:36 smithi151 ceph-mon[104450]: pgmap v1644: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 690 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 566 op/s; 3998/33195 objects degraded (12.044%); 12120/33195 objects misplaced (36.512%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:48:37.236 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:36 smithi151 ceph-mon[104450]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:37.236 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:36 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3229279543' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:48:37.236 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:36 smithi151 ceph-mon[104450]: osdmap e1164: 8 total, 8 up, 8 in 2024-09-13T14:48:37.236 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:36 smithi151 ceph-mon[108966]: pgmap v1644: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 690 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 566 op/s; 3998/33195 objects degraded (12.044%); 12120/33195 objects misplaced (36.512%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:48:37.237 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:36 smithi151 ceph-mon[108966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:37.237 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:36 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3229279543' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:48:37.237 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:36 smithi151 ceph-mon[108966]: osdmap e1164: 8 total, 8 up, 8 in 2024-09-13T14:48:37.244 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:37.268 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:37.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:37.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:37.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:36 smithi184 ceph-mon[92527]: pgmap v1644: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 690 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 566 op/s; 3998/33195 objects degraded (12.044%); 12120/33195 objects misplaced (36.512%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:48:37.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:36 smithi184 ceph-mon[92527]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:37.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:36 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3229279543' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:48:37.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:36 smithi184 ceph-mon[92527]: osdmap e1164: 8 total, 8 up, 8 in 2024-09-13T14:48:37.370 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:37.405 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:37.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:37.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:37.507 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:37.531 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:37.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:37.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:37.633 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:37.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:37.658 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:37.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:37.758 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:37.784 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:37.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:37.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:37.886 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:37.908 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:37.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:37.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:37.945 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1641,"stamp":"2024-09-13T14:48:29.518052+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":980517048,"num_objects":14979,"num_object_clones":0,"num_object_copies":44937,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3924,"num_objects_misplaced":14189,"num_objects_unfound":0,"num_objects_dirty":14979,"num_whiteouts":0,"num_read":14,"num_read_kb":35,"num_write":226035,"num_write_kb":7709418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2725,"num_bytes_recovered":178022649,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115346,"ondisk_log_size":115346,"up":357,"acting":356,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":366,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5926244,"kb_used_data":3394788,"kb_used_omap":149,"kb_used_meta":2531242,"kb_avail":744034972,"statfs":{"total":767960285184,"available":761891811328,"internally_reserved":0,"allocated":3476262912,"data_stored":3447758095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":153401,"internal_metadata":2591992007},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-387710976,"num_objects":-5916,"num_object_clones":0,"num_object_copies":-17748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":222,"num_objects_misplaced":-2968,"num_objects_unfound":0,"num_objects_dirty":-5916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5916,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":179,"num_bytes_recovered":11730944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.723944"},"pg_stats":[{"pgid":"1.1d","version":"261'100","reported_seq":97,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.820036+0000","last_change":"2024-09-13T14:45:19.807356+0000","last_active":"2024-09-13T14:48:18.820036+0000","last_peered":"2024-09-13T14:48:18.820036+0000","last_clean":"2024-09-13T14:48:18.820036+0000","last_became_active":"2024-09-13T14:45:19.807081+0000","last_became_peered":"2024-09-13T14:45:19.807081+0000","last_unstale":"2024-09-13T14:48:18.820036+0000","last_undegraded":"2024-09-13T14:48:18.820036+0000","last_fullsized":"2024-09-13T14:48:18.820036+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:17.988345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:17.988345+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:41:05.216335+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":1702,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.827357+0000","last_change":"2024-09-13T14:29:42.541263+0000","last_active":"2024-09-13T14:48:18.827357+0000","last_peered":"2024-09-13T14:48:18.827357+0000","last_clean":"2024-09-13T14:48:18.827357+0000","last_became_active":"2024-09-13T14:29:42.540873+0000","last_became_peered":"2024-09-13T14:29:42.540873+0000","last_unstale":"2024-09-13T14:48:18.827357+0000","last_undegraded":"2024-09-13T14:48:18.827357+0000","last_fullsized":"2024-09-13T14:48:18.827357+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:20.611206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1123,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.755959+0000","last_change":"2024-09-13T14:43:07.516715+0000","last_active":"2024-09-13T14:48:01.755959+0000","last_peered":"2024-09-13T14:48:01.755959+0000","last_clean":"2024-09-13T14:48:01.755959+0000","last_became_active":"2024-09-13T14:43:07.516403+0000","last_became_peered":"2024-09-13T14:43:07.516403+0000","last_unstale":"2024-09-13T14:48:01.755959+0000","last_undegraded":"2024-09-13T14:48:01.755959+0000","last_fullsized":"2024-09-13T14:48:01.755959+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:56.469690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":41,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.804759+0000","last_change":"2024-09-13T14:45:19.810203+0000","last_active":"2024-09-13T14:48:18.804759+0000","last_peered":"2024-09-13T14:48:18.804759+0000","last_clean":"2024-09-13T14:48:18.804759+0000","last_became_active":"2024-09-13T14:45:19.809693+0000","last_became_peered":"2024-09-13T14:45:19.809693+0000","last_unstale":"2024-09-13T14:48:18.804759+0000","last_undegraded":"2024-09-13T14:48:18.804759+0000","last_fullsized":"2024-09-13T14:48:18.804759+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:20.219673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":41,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.810825+0000","last_change":"2024-09-13T14:45:19.807991+0000","last_active":"2024-09-13T14:48:18.810825+0000","last_peered":"2024-09-13T14:48:18.810825+0000","last_clean":"2024-09-13T14:48:18.810825+0000","last_became_active":"2024-09-13T14:45:19.807662+0000","last_became_peered":"2024-09-13T14:45:19.807662+0000","last_unstale":"2024-09-13T14:48:18.810825+0000","last_undegraded":"2024-09-13T14:48:18.810825+0000","last_fullsized":"2024-09-13T14:48:18.810825+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:29.306108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"261'100","reported_seq":1234,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.747973+0000","last_change":"2024-09-13T14:44:54.259562+0000","last_active":"2024-09-13T14:48:01.747973+0000","last_peered":"2024-09-13T14:48:01.747973+0000","last_clean":"2024-09-13T14:48:01.747973+0000","last_became_active":"2024-09-13T14:43:14.563717+0000","last_became_peered":"2024-09-13T14:43:14.563717+0000","last_unstale":"2024-09-13T14:48:01.747973+0000","last_undegraded":"2024-09-13T14:48:01.747973+0000","last_fullsized":"2024-09-13T14:48:01.747973+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_clean_scrub_stamp":"2024-09-13T14:44:54.259482+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:43:52.729109+0000","scrub_duration":0.015265569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":1091,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.759151+0000","last_change":"2024-09-13T14:19:25.873514+0000","last_active":"2024-09-13T14:48:01.759151+0000","last_peered":"2024-09-13T14:48:01.759151+0000","last_clean":"2024-09-13T14:48:01.759151+0000","last_became_active":"2024-09-13T14:19:25.873170+0000","last_became_peered":"2024-09-13T14:19:25.873170+0000","last_unstale":"2024-09-13T14:48:01.759151+0000","last_undegraded":"2024-09-13T14:48:01.759151+0000","last_fullsized":"2024-09-13T14:48:01.759151+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:48.298530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1120,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.756633+0000","last_change":"2024-09-13T14:30:15.580011+0000","last_active":"2024-09-13T14:48:01.756633+0000","last_peered":"2024-09-13T14:48:01.756633+0000","last_clean":"2024-09-13T14:48:01.756633+0000","last_became_active":"2024-09-13T14:30:15.579610+0000","last_became_peered":"2024-09-13T14:30:15.579610+0000","last_unstale":"2024-09-13T14:48:01.756633+0000","last_undegraded":"2024-09-13T14:48:01.756633+0000","last_fullsized":"2024-09-13T14:48:01.756633+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:11:02.733771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"261'100","reported_seq":58,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.805923+0000","last_change":"2024-09-13T14:45:35.947332+0000","last_active":"2024-09-13T14:48:18.805923+0000","last_peered":"2024-09-13T14:48:18.805923+0000","last_clean":"2024-09-13T14:48:18.805923+0000","last_became_active":"2024-09-13T14:45:19.810775+0000","last_became_peered":"2024-09-13T14:45:19.810775+0000","last_unstale":"2024-09-13T14:48:18.805923+0000","last_undegraded":"2024-09-13T14:48:18.805923+0000","last_fullsized":"2024-09-13T14:48:18.805923+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:35.947265+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:43:41.569795+0000","last_clean_scrub_stamp":"2024-09-13T14:45:35.947265+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:00.606370+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":1136,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.758974+0000","last_change":"2024-09-13T14:43:07.536116+0000","last_active":"2024-09-13T14:48:01.758974+0000","last_peered":"2024-09-13T14:48:01.758974+0000","last_clean":"2024-09-13T14:48:01.758974+0000","last_became_active":"2024-09-13T14:43:07.524112+0000","last_became_peered":"2024-09-13T14:43:07.524112+0000","last_unstale":"2024-09-13T14:48:01.758974+0000","last_undegraded":"2024-09-13T14:48:01.758974+0000","last_fullsized":"2024-09-13T14:48:01.758974+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:36.947060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1133,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.756831+0000","last_change":"2024-09-13T14:43:07.530236+0000","last_active":"2024-09-13T14:48:01.756831+0000","last_peered":"2024-09-13T14:48:01.756831+0000","last_clean":"2024-09-13T14:48:01.756831+0000","last_became_active":"2024-09-13T14:43:07.529967+0000","last_became_peered":"2024-09-13T14:43:07.529967+0000","last_unstale":"2024-09-13T14:48:01.756831+0000","last_undegraded":"2024-09-13T14:48:01.756831+0000","last_fullsized":"2024-09-13T14:48:01.756831+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:27.276550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":1686,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.826218+0000","last_change":"2024-09-13T14:28:35.612469+0000","last_active":"2024-09-13T14:48:18.826218+0000","last_peered":"2024-09-13T14:48:18.826218+0000","last_clean":"2024-09-13T14:48:18.826218+0000","last_became_active":"2024-09-13T14:28:35.612093+0000","last_became_peered":"2024-09-13T14:28:35.612093+0000","last_unstale":"2024-09-13T14:48:18.826218+0000","last_undegraded":"2024-09-13T14:48:18.826218+0000","last_fullsized":"2024-09-13T14:48:18.826218+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:51.022663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1709,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.807824+0000","last_change":"2024-09-13T14:45:19.808811+0000","last_active":"2024-09-13T14:48:18.807824+0000","last_peered":"2024-09-13T14:48:18.807824+0000","last_clean":"2024-09-13T14:48:18.807824+0000","last_became_active":"2024-09-13T14:45:19.808471+0000","last_became_peered":"2024-09-13T14:45:19.808471+0000","last_unstale":"2024-09-13T14:48:18.807824+0000","last_undegraded":"2024-09-13T14:48:18.807824+0000","last_fullsized":"2024-09-13T14:48:18.807824+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:53.638161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"261'100","reported_seq":41,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.810914+0000","last_change":"2024-09-13T14:45:19.813412+0000","last_active":"2024-09-13T14:48:18.810914+0000","last_peered":"2024-09-13T14:48:18.810914+0000","last_clean":"2024-09-13T14:48:18.810914+0000","last_became_active":"2024-09-13T14:45:19.812585+0000","last_became_peered":"2024-09-13T14:45:19.812585+0000","last_unstale":"2024-09-13T14:48:18.810914+0000","last_undegraded":"2024-09-13T14:48:18.810914+0000","last_fullsized":"2024-09-13T14:48:18.810914+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.614340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.614340+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:32:07.434109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":1142,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.756766+0000","last_change":"2024-09-13T14:43:07.522950+0000","last_active":"2024-09-13T14:48:01.756766+0000","last_peered":"2024-09-13T14:48:01.756766+0000","last_clean":"2024-09-13T14:48:01.756766+0000","last_became_active":"2024-09-13T14:43:07.522566+0000","last_became_peered":"2024-09-13T14:43:07.522566+0000","last_unstale":"2024-09-13T14:48:01.756766+0000","last_undegraded":"2024-09-13T14:48:01.756766+0000","last_fullsized":"2024-09-13T14:48:01.756766+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:03.201478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":41,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.802918+0000","last_change":"2024-09-13T14:45:19.807231+0000","last_active":"2024-09-13T14:48:18.802918+0000","last_peered":"2024-09-13T14:48:18.802918+0000","last_clean":"2024-09-13T14:48:18.802918+0000","last_became_active":"2024-09-13T14:45:19.806936+0000","last_became_peered":"2024-09-13T14:45:19.806936+0000","last_unstale":"2024-09-13T14:48:18.802918+0000","last_undegraded":"2024-09-13T14:48:18.802918+0000","last_fullsized":"2024-09-13T14:48:18.802918+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:04:26.723153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.18","version":"261'100","reported_seq":112,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.758695+0000","last_change":"2024-09-13T14:44:41.685537+0000","last_active":"2024-09-13T14:48:01.758695+0000","last_peered":"2024-09-13T14:48:01.758695+0000","last_clean":"2024-09-13T14:48:01.758695+0000","last_became_active":"2024-09-13T14:43:14.558502+0000","last_became_peered":"2024-09-13T14:43:14.558502+0000","last_unstale":"2024-09-13T14:48:01.758695+0000","last_undegraded":"2024-09-13T14:48:01.758695+0000","last_fullsized":"2024-09-13T14:48:01.758695+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1134,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:41.685124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:41.685124+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:37:58.412497+0000","scrub_duration":0.0075205970000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":90,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.820972+0000","last_change":"2024-09-13T14:43:07.516845+0000","last_active":"2024-09-13T14:48:18.820972+0000","last_peered":"2024-09-13T14:48:18.820972+0000","last_clean":"2024-09-13T14:48:18.820972+0000","last_became_active":"2024-09-13T14:43:07.514974+0000","last_became_peered":"2024-09-13T14:43:07.514974+0000","last_unstale":"2024-09-13T14:48:18.820972+0000","last_undegraded":"2024-09-13T14:48:18.820972+0000","last_fullsized":"2024-09-13T14:48:18.820972+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:55.419483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1724,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.840637+0000","last_change":"2024-09-13T14:45:19.815341+0000","last_active":"2024-09-13T14:48:18.840637+0000","last_peered":"2024-09-13T14:48:18.840637+0000","last_clean":"2024-09-13T14:48:18.840637+0000","last_became_active":"2024-09-13T14:45:19.810262+0000","last_became_peered":"2024-09-13T14:45:19.810262+0000","last_unstale":"2024-09-13T14:48:18.840637+0000","last_undegraded":"2024-09-13T14:48:18.840637+0000","last_fullsized":"2024-09-13T14:48:18.840637+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:39.847653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1b","version":"261'100","reported_seq":54,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.760126+0000","last_change":"2024-09-13T14:44:52.803330+0000","last_active":"2024-09-13T14:48:01.760126+0000","last_peered":"2024-09-13T14:48:01.760126+0000","last_clean":"2024-09-13T14:48:01.760126+0000","last_became_active":"2024-09-13T14:43:14.556712+0000","last_became_peered":"2024-09-13T14:43:14.556712+0000","last_unstale":"2024-09-13T14:48:01.760126+0000","last_undegraded":"2024-09-13T14:48:01.760126+0000","last_fullsized":"2024-09-13T14:48:01.760126+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:52.803242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:52.803242+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:06.658301+0000","scrub_duration":0.0019769660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":41,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.802362+0000","last_change":"2024-09-13T14:45:19.806623+0000","last_active":"2024-09-13T14:48:18.802362+0000","last_peered":"2024-09-13T14:48:18.802362+0000","last_clean":"2024-09-13T14:48:18.802362+0000","last_became_active":"2024-09-13T14:45:19.806219+0000","last_became_peered":"2024-09-13T14:45:19.806219+0000","last_unstale":"2024-09-13T14:48:18.802362+0000","last_undegraded":"2024-09-13T14:48:18.802362+0000","last_fullsized":"2024-09-13T14:48:18.802362+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:34:41.601201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1116,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.759298+0000","last_change":"2024-09-13T14:43:32.637751+0000","last_active":"2024-09-13T14:48:01.759298+0000","last_peered":"2024-09-13T14:48:01.759298+0000","last_clean":"2024-09-13T14:48:01.759298+0000","last_became_active":"2024-09-13T14:43:32.636901+0000","last_became_peered":"2024-09-13T14:43:32.636901+0000","last_unstale":"2024-09-13T14:48:01.759298+0000","last_undegraded":"2024-09-13T14:48:01.759298+0000","last_fullsized":"2024-09-13T14:48:01.759298+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:21.942419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1a","version":"261'100","reported_seq":54,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.756556+0000","last_change":"2024-09-13T14:43:41.910041+0000","last_active":"2024-09-13T14:48:01.756556+0000","last_peered":"2024-09-13T14:48:01.756556+0000","last_clean":"2024-09-13T14:48:01.756556+0000","last_became_active":"2024-09-13T14:43:14.555083+0000","last_became_peered":"2024-09-13T14:43:14.555083+0000","last_unstale":"2024-09-13T14:48:01.756556+0000","last_undegraded":"2024-09-13T14:48:01.756556+0000","last_fullsized":"2024-09-13T14:48:01.756556+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:41.909988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:41.909988+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:42:19.954838+0000","scrub_duration":0.0012891619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":41,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.802569+0000","last_change":"2024-09-13T14:45:19.812071+0000","last_active":"2024-09-13T14:48:18.802569+0000","last_peered":"2024-09-13T14:48:18.802569+0000","last_clean":"2024-09-13T14:48:18.802569+0000","last_became_active":"2024-09-13T14:45:19.811336+0000","last_became_peered":"2024-09-13T14:45:19.811336+0000","last_unstale":"2024-09-13T14:48:18.802569+0000","last_undegraded":"2024-09-13T14:48:18.802569+0000","last_fullsized":"2024-09-13T14:48:18.802569+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:40:29.814608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1742,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.827332+0000","last_change":"2024-09-13T14:45:19.810478+0000","last_active":"2024-09-13T14:48:18.827332+0000","last_peered":"2024-09-13T14:48:18.827332+0000","last_clean":"2024-09-13T14:48:18.827332+0000","last_became_active":"2024-09-13T14:45:19.808278+0000","last_became_peered":"2024-09-13T14:45:19.808278+0000","last_unstale":"2024-09-13T14:48:18.827332+0000","last_undegraded":"2024-09-13T14:48:18.827332+0000","last_fullsized":"2024-09-13T14:48:18.827332+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:12.211089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"261'100","reported_seq":54,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.748116+0000","last_change":"2024-09-13T14:44:38.214961+0000","last_active":"2024-09-13T14:48:01.748116+0000","last_peered":"2024-09-13T14:48:01.748116+0000","last_clean":"2024-09-13T14:48:01.748116+0000","last_became_active":"2024-09-13T14:43:14.557883+0000","last_became_peered":"2024-09-13T14:43:14.557883+0000","last_unstale":"2024-09-13T14:48:01.748116+0000","last_undegraded":"2024-09-13T14:48:01.748116+0000","last_fullsized":"2024-09-13T14:48:01.748116+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:38.214865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:38.214865+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:49:36.013783+0000","scrub_duration":0.0016389639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1612,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.811049+0000","last_change":"2024-09-13T14:30:15.576921+0000","last_active":"2024-09-13T14:48:18.811049+0000","last_peered":"2024-09-13T14:48:18.811049+0000","last_clean":"2024-09-13T14:48:18.811049+0000","last_became_active":"2024-09-13T14:30:15.576607+0000","last_became_peered":"2024-09-13T14:30:15.576607+0000","last_unstale":"2024-09-13T14:48:18.811049+0000","last_undegraded":"2024-09-13T14:48:18.811049+0000","last_fullsized":"2024-09-13T14:48:18.811049+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:35.162460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1126,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.759378+0000","last_change":"2024-09-13T14:43:07.518801+0000","last_active":"2024-09-13T14:48:01.759378+0000","last_peered":"2024-09-13T14:48:01.759378+0000","last_clean":"2024-09-13T14:48:01.759378+0000","last_became_active":"2024-09-13T14:43:07.517610+0000","last_became_peered":"2024-09-13T14:43:07.517610+0000","last_unstale":"2024-09-13T14:48:01.759378+0000","last_undegraded":"2024-09-13T14:48:01.759378+0000","last_fullsized":"2024-09-13T14:48:01.759378+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:51.632254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"261'100","reported_seq":90,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.807369+0000","last_change":"2024-09-13T14:43:15.037565+0000","last_active":"2024-09-13T14:48:18.807369+0000","last_peered":"2024-09-13T14:48:18.807369+0000","last_clean":"2024-09-13T14:48:18.807369+0000","last_became_active":"2024-09-13T14:43:14.554183+0000","last_became_peered":"2024-09-13T14:43:14.554183+0000","last_unstale":"2024-09-13T14:48:18.807369+0000","last_undegraded":"2024-09-13T14:48:18.807369+0000","last_fullsized":"2024-09-13T14:48:18.807369+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.037517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.037517+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:01:35.137195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":1120,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.748251+0000","last_change":"2024-09-13T14:30:15.579260+0000","last_active":"2024-09-13T14:48:01.748251+0000","last_peered":"2024-09-13T14:48:01.748251+0000","last_clean":"2024-09-13T14:48:01.748251+0000","last_became_active":"2024-09-13T14:30:15.578897+0000","last_became_peered":"2024-09-13T14:30:15.578897+0000","last_unstale":"2024-09-13T14:48:01.748251+0000","last_undegraded":"2024-09-13T14:48:01.748251+0000","last_fullsized":"2024-09-13T14:48:01.748251+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:56.737849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":90,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.824203+0000","last_change":"2024-09-13T14:43:07.521576+0000","last_active":"2024-09-13T14:48:18.824203+0000","last_peered":"2024-09-13T14:48:18.824203+0000","last_clean":"2024-09-13T14:48:18.824203+0000","last_became_active":"2024-09-13T14:43:07.520478+0000","last_became_peered":"2024-09-13T14:43:07.520478+0000","last_unstale":"2024-09-13T14:48:18.824203+0000","last_undegraded":"2024-09-13T14:48:18.824203+0000","last_fullsized":"2024-09-13T14:48:18.824203+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:53.410135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"261'100","reported_seq":380,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.759225+0000","last_change":"2024-09-13T14:44:49.775941+0000","last_active":"2024-09-13T14:48:01.759225+0000","last_peered":"2024-09-13T14:48:01.759225+0000","last_clean":"2024-09-13T14:48:01.759225+0000","last_became_active":"2024-09-13T14:43:14.560297+0000","last_became_peered":"2024-09-13T14:43:14.560297+0000","last_unstale":"2024-09-13T14:48:01.759225+0000","last_undegraded":"2024-09-13T14:48:01.759225+0000","last_fullsized":"2024-09-13T14:48:01.759225+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:49.775844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:49.775844+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:15:53.138661+0000","scrub_duration":0.012350274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":1109,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.756718+0000","last_change":"2024-09-13T14:19:25.870605+0000","last_active":"2024-09-13T14:48:01.756718+0000","last_peered":"2024-09-13T14:48:01.756718+0000","last_clean":"2024-09-13T14:48:01.756718+0000","last_became_active":"2024-09-13T14:19:25.870282+0000","last_became_peered":"2024-09-13T14:19:25.870282+0000","last_unstale":"2024-09-13T14:48:01.756718+0000","last_undegraded":"2024-09-13T14:48:01.756718+0000","last_fullsized":"2024-09-13T14:48:01.756718+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:52.514915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1123,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.756920+0000","last_change":"2024-09-13T14:43:07.526476+0000","last_active":"2024-09-13T14:48:01.756920+0000","last_peered":"2024-09-13T14:48:01.756920+0000","last_clean":"2024-09-13T14:48:01.756920+0000","last_became_active":"2024-09-13T14:43:07.525802+0000","last_became_peered":"2024-09-13T14:43:07.525802+0000","last_unstale":"2024-09-13T14:48:01.756920+0000","last_undegraded":"2024-09-13T14:48:01.756920+0000","last_fullsized":"2024-09-13T14:48:01.756920+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:23.154726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"261'100","reported_seq":97,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.818019+0000","last_change":"2024-09-13T14:45:19.807461+0000","last_active":"2024-09-13T14:48:18.818019+0000","last_peered":"2024-09-13T14:48:18.818019+0000","last_clean":"2024-09-13T14:48:18.818019+0000","last_became_active":"2024-09-13T14:45:19.807117+0000","last_became_peered":"2024-09-13T14:45:19.807117+0000","last_unstale":"2024-09-13T14:48:18.818019+0000","last_undegraded":"2024-09-13T14:48:18.818019+0000","last_fullsized":"2024-09-13T14:48:18.818019+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.955197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.955197+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:03.531555+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"1163'14339","reported_seq":14411,"reported_epoch":1163,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:48:27.968919+0000","last_change":"2024-09-13T14:45:21.161658+0000","last_active":"2024-09-13T14:48:27.968919+0000","last_peered":"2024-09-13T14:48:27.968919+0000","last_clean":"2024-09-13T14:45:19.622135+0000","last_became_active":"2024-09-13T14:45:21.090442+0000","last_became_peered":"2024-09-13T14:45:21.090442+0000","last_unstale":"2024-09-13T14:48:27.968919+0000","last_undegraded":"2024-09-13T14:48:27.968919+0000","last_fullsized":"2024-09-13T14:48:27.968919+0000","mapping_epoch":1149,"log_start":"1153'8300","ondisk_log_start":"1153'8300","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6039,"log_dups_size":0,"ondisk_log_size":6039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56819712,"num_objects":867,"num_object_clones":0,"num_object_copies":2601,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":867,"num_objects_unfound":0,"num_objects_dirty":867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14339,"num_write_kb":486592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[1,2,7],"avail_no_missing":["1","2","7"],"object_location_counts":[{"shards":"1,2,7","objects":867}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1086,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.757104+0000","last_change":"2024-09-13T14:19:26.131609+0000","last_active":"2024-09-13T14:48:01.757104+0000","last_peered":"2024-09-13T14:48:01.757104+0000","last_clean":"2024-09-13T14:48:01.757104+0000","last_became_active":"2024-09-13T14:19:26.131029+0000","last_became_peered":"2024-09-13T14:19:26.131029+0000","last_unstale":"2024-09-13T14:48:01.757104+0000","last_undegraded":"2024-09-13T14:48:01.757104+0000","last_fullsized":"2024-09-13T14:48:01.757104+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:57.933696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":588,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.747981+0000","last_change":"2024-09-13T14:43:32.635070+0000","last_active":"2024-09-13T14:48:01.747981+0000","last_peered":"2024-09-13T14:48:01.747981+0000","last_clean":"2024-09-13T14:48:01.747981+0000","last_became_active":"2024-09-13T14:43:32.634734+0000","last_became_peered":"2024-09-13T14:43:32.634734+0000","last_unstale":"2024-09-13T14:48:01.747981+0000","last_undegraded":"2024-09-13T14:48:01.747981+0000","last_fullsized":"2024-09-13T14:48:01.747981+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:33.107010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":1732,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.840783+0000","last_change":"2024-09-13T14:43:07.537759+0000","last_active":"2024-09-13T14:48:18.840783+0000","last_peered":"2024-09-13T14:48:18.840783+0000","last_clean":"2024-09-13T14:48:18.840783+0000","last_became_active":"2024-09-13T14:43:07.520303+0000","last_became_peered":"2024-09-13T14:43:07.520303+0000","last_unstale":"2024-09-13T14:48:18.840783+0000","last_undegraded":"2024-09-13T14:48:18.840783+0000","last_fullsized":"2024-09-13T14:48:18.840783+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:32.772570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"261'100","reported_seq":41,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.803351+0000","last_change":"2024-09-13T14:45:19.813527+0000","last_active":"2024-09-13T14:48:18.803351+0000","last_peered":"2024-09-13T14:48:18.803351+0000","last_clean":"2024-09-13T14:48:18.803351+0000","last_became_active":"2024-09-13T14:45:19.812834+0000","last_became_peered":"2024-09-13T14:45:19.812834+0000","last_unstale":"2024-09-13T14:48:18.803351+0000","last_undegraded":"2024-09-13T14:48:18.803351+0000","last_fullsized":"2024-09-13T14:48:18.803351+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.710192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.710192+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:08:55.152703+0000","scrub_duration":0.0019250910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"1163'14143","reported_seq":19575,"reported_epoch":1163,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:48:27.019196+0000","last_change":"2024-09-13T14:45:38.479094+0000","last_active":"2024-09-13T14:48:27.019196+0000","last_peered":"2024-09-13T14:48:27.019196+0000","last_clean":"2024-09-13T14:45:19.642017+0000","last_became_active":"2024-09-13T14:45:21.111093+0000","last_became_peered":"2024-09-13T14:45:21.111093+0000","last_unstale":"2024-09-13T14:48:27.019196+0000","last_undegraded":"2024-09-13T14:45:38.458706+0000","last_fullsized":"2024-09-13T14:45:20.800066+0000","mapping_epoch":1149,"log_start":"1145'5800","ondisk_log_start":"1145'5800","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":8343,"log_dups_size":0,"ondisk_log_size":8343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61145088,"num_objects":933,"num_object_clones":0,"num_object_copies":2799,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3924,"num_objects_misplaced":490,"num_objects_unfound":0,"num_objects_dirty":933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14143,"num_write_kb":482432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2681,"num_bytes_recovered":175702016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[2,4],"avail_no_missing":["2","4","6"],"object_location_counts":[{"shards":"2,4","objects":-3481},{"shards":"2,4,6","objects":490}],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":41,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.802623+0000","last_change":"2024-09-13T14:45:19.813025+0000","last_active":"2024-09-13T14:48:18.802623+0000","last_peered":"2024-09-13T14:48:18.802623+0000","last_clean":"2024-09-13T14:48:18.802623+0000","last_became_active":"2024-09-13T14:45:19.812714+0000","last_became_peered":"2024-09-13T14:45:19.812714+0000","last_unstale":"2024-09-13T14:48:18.802623+0000","last_undegraded":"2024-09-13T14:48:18.802623+0000","last_fullsized":"2024-09-13T14:48:18.802623+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:40.671012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":90,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.818281+0000","last_change":"2024-09-13T14:43:07.522730+0000","last_active":"2024-09-13T14:48:18.818281+0000","last_peered":"2024-09-13T14:48:18.818281+0000","last_clean":"2024-09-13T14:48:18.818281+0000","last_became_active":"2024-09-13T14:43:07.517420+0000","last_became_peered":"2024-09-13T14:43:07.517420+0000","last_unstale":"2024-09-13T14:48:18.818281+0000","last_undegraded":"2024-09-13T14:48:18.818281+0000","last_fullsized":"2024-09-13T14:48:18.818281+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:21:30.265293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":1645,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.813825+0000","last_change":"2024-09-13T14:30:15.581728+0000","last_active":"2024-09-13T14:48:18.813825+0000","last_peered":"2024-09-13T14:48:18.813825+0000","last_clean":"2024-09-13T14:48:18.813825+0000","last_became_active":"2024-09-13T14:30:15.581303+0000","last_became_peered":"2024-09-13T14:30:15.581303+0000","last_unstale":"2024-09-13T14:48:18.813825+0000","last_undegraded":"2024-09-13T14:48:18.813825+0000","last_fullsized":"2024-09-13T14:48:18.813825+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:10.027991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"261'100","reported_seq":113,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.757034+0000","last_change":"2024-09-13T14:43:31.112173+0000","last_active":"2024-09-13T14:48:01.757034+0000","last_peered":"2024-09-13T14:48:01.757034+0000","last_clean":"2024-09-13T14:48:01.757034+0000","last_became_active":"2024-09-13T14:42:42.709263+0000","last_became_peered":"2024-09-13T14:42:42.709263+0000","last_unstale":"2024-09-13T14:48:01.757034+0000","last_undegraded":"2024-09-13T14:48:01.757034+0000","last_fullsized":"2024-09-13T14:48:01.757034+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:31.112111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:31.112111+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:02.707372+0000","scrub_duration":0.0015759260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"1163'14373","reported_seq":14420,"reported_epoch":1163,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:48:27.722193+0000","last_change":"2024-09-13T14:45:21.011501+0000","last_active":"2024-09-13T14:48:27.722193+0000","last_peered":"2024-09-13T14:48:27.722193+0000","last_clean":"2024-09-13T14:45:19.622974+0000","last_became_active":"2024-09-13T14:45:21.006075+0000","last_became_peered":"2024-09-13T14:45:21.006075+0000","last_unstale":"2024-09-13T14:48:27.722193+0000","last_undegraded":"2024-09-13T14:48:27.722193+0000","last_fullsized":"2024-09-13T14:48:27.722193+0000","mapping_epoch":1149,"log_start":"1146'6600","ondisk_log_start":"1146'6600","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":7773,"log_dups_size":0,"ondisk_log_size":7773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:30.090601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57999360,"num_objects":885,"num_object_clones":0,"num_object_copies":2655,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":885,"num_objects_unfound":0,"num_objects_dirty":885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14373,"num_write_kb":488256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,2],"avail_no_missing":["5","1","2"],"object_location_counts":[{"shards":"1,2,5","objects":885}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1685,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.826141+0000","last_change":"2024-09-13T14:28:35.611508+0000","last_active":"2024-09-13T14:48:18.826141+0000","last_peered":"2024-09-13T14:48:18.826141+0000","last_clean":"2024-09-13T14:48:18.826141+0000","last_became_active":"2024-09-13T14:28:35.610106+0000","last_became_peered":"2024-09-13T14:28:35.610106+0000","last_unstale":"2024-09-13T14:48:18.826141+0000","last_undegraded":"2024-09-13T14:48:18.826141+0000","last_fullsized":"2024-09-13T14:48:18.826141+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:11.966958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1129,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.748182+0000","last_change":"2024-09-13T14:28:35.603663+0000","last_active":"2024-09-13T14:48:01.748182+0000","last_peered":"2024-09-13T14:48:01.748182+0000","last_clean":"2024-09-13T14:48:01.748182+0000","last_became_active":"2024-09-13T14:28:35.603358+0000","last_became_peered":"2024-09-13T14:28:35.603358+0000","last_unstale":"2024-09-13T14:48:01.748182+0000","last_undegraded":"2024-09-13T14:48:01.748182+0000","last_fullsized":"2024-09-13T14:48:01.748182+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:24:05.842796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":90,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.818563+0000","last_change":"2024-09-13T14:43:07.522750+0000","last_active":"2024-09-13T14:48:18.818563+0000","last_peered":"2024-09-13T14:48:18.818563+0000","last_clean":"2024-09-13T14:48:18.818563+0000","last_became_active":"2024-09-13T14:43:07.517267+0000","last_became_peered":"2024-09-13T14:43:07.517267+0000","last_unstale":"2024-09-13T14:48:18.818563+0000","last_undegraded":"2024-09-13T14:48:18.818563+0000","last_fullsized":"2024-09-13T14:48:18.818563+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:40.217748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"261'100","reported_seq":411,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.840482+0000","last_change":"2024-09-13T14:45:19.815050+0000","last_active":"2024-09-13T14:48:18.840482+0000","last_peered":"2024-09-13T14:48:18.840482+0000","last_clean":"2024-09-13T14:48:18.840482+0000","last_became_active":"2024-09-13T14:45:19.809732+0000","last_became_peered":"2024-09-13T14:45:19.809732+0000","last_unstale":"2024-09-13T14:48:18.840482+0000","last_undegraded":"2024-09-13T14:48:18.840482+0000","last_fullsized":"2024-09-13T14:48:18.840482+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:33.574443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:33.574443+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:51:40.261036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"1163'14205","reported_seq":14310,"reported_epoch":1163,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:48:27.467718+0000","last_change":"2024-09-13T14:45:21.136109+0000","last_active":"2024-09-13T14:48:27.467718+0000","last_peered":"2024-09-13T14:48:27.467718+0000","last_clean":"2024-09-13T14:45:19.607954+0000","last_became_active":"2024-09-13T14:45:21.004300+0000","last_became_peered":"2024-09-13T14:45:21.004300+0000","last_unstale":"2024-09-13T14:48:27.467718+0000","last_undegraded":"2024-09-13T14:48:27.467718+0000","last_fullsized":"2024-09-13T14:48:27.467718+0000","mapping_epoch":1149,"log_start":"1151'8000","ondisk_log_start":"1151'8000","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6205,"log_dups_size":0,"ondisk_log_size":6205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:04:23.351582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59179008,"num_objects":903,"num_object_clones":0,"num_object_copies":2709,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":897,"num_objects_unfound":0,"num_objects_dirty":903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14205,"num_write_kb":483456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":1835008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,6,1],"avail_no_missing":["4","1","6"],"object_location_counts":[{"shards":"1,4,6","objects":903}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":90,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.824081+0000","last_change":"2024-09-13T14:43:07.520049+0000","last_active":"2024-09-13T14:48:18.824081+0000","last_peered":"2024-09-13T14:48:18.824081+0000","last_clean":"2024-09-13T14:48:18.824081+0000","last_became_active":"2024-09-13T14:43:07.514646+0000","last_became_peered":"2024-09-13T14:43:07.514646+0000","last_unstale":"2024-09-13T14:48:18.824081+0000","last_undegraded":"2024-09-13T14:48:18.824081+0000","last_fullsized":"2024-09-13T14:48:18.824081+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:10:35.200778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":90,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.818481+0000","last_change":"2024-09-13T14:43:07.525666+0000","last_active":"2024-09-13T14:48:18.818481+0000","last_peered":"2024-09-13T14:48:18.818481+0000","last_clean":"2024-09-13T14:48:18.818481+0000","last_became_active":"2024-09-13T14:43:07.524064+0000","last_became_peered":"2024-09-13T14:43:07.524064+0000","last_unstale":"2024-09-13T14:48:18.818481+0000","last_undegraded":"2024-09-13T14:48:18.818481+0000","last_fullsized":"2024-09-13T14:48:18.818481+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:10.270333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1143,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.759144+0000","last_change":"2024-09-13T14:45:19.804177+0000","last_active":"2024-09-13T14:48:01.759144+0000","last_peered":"2024-09-13T14:48:01.759144+0000","last_clean":"2024-09-13T14:48:01.759144+0000","last_became_active":"2024-09-13T14:45:19.803890+0000","last_became_peered":"2024-09-13T14:45:19.803890+0000","last_unstale":"2024-09-13T14:48:01.759144+0000","last_undegraded":"2024-09-13T14:48:01.759144+0000","last_fullsized":"2024-09-13T14:48:01.759144+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:25.140789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"261'100","reported_seq":58,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.812898+0000","last_change":"2024-09-13T14:45:36.965992+0000","last_active":"2024-09-13T14:48:18.812898+0000","last_peered":"2024-09-13T14:48:18.812898+0000","last_clean":"2024-09-13T14:48:18.812898+0000","last_became_active":"2024-09-13T14:45:19.811238+0000","last_became_peered":"2024-09-13T14:45:19.811238+0000","last_unstale":"2024-09-13T14:48:18.812898+0000","last_undegraded":"2024-09-13T14:48:18.812898+0000","last_fullsized":"2024-09-13T14:48:18.812898+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:36.965775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:45:36.965775+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:36:01.623251+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"1163'13832","reported_seq":13902,"reported_epoch":1163,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:48:24.523834+0000","last_change":"2024-09-13T14:45:21.036181+0000","last_active":"2024-09-13T14:48:24.523834+0000","last_peered":"2024-09-13T14:48:24.523834+0000","last_clean":"2024-09-13T14:45:19.587756+0000","last_became_active":"2024-09-13T14:45:21.029769+0000","last_became_peered":"2024-09-13T14:45:21.029769+0000","last_unstale":"2024-09-13T14:48:24.523834+0000","last_undegraded":"2024-09-13T14:48:24.523834+0000","last_fullsized":"2024-09-13T14:48:24.523834+0000","mapping_epoch":1149,"log_start":"1147'7000","ondisk_log_start":"1147'7000","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6832,"log_dups_size":0,"ondisk_log_size":6832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63307776,"num_objects":966,"num_object_clones":0,"num_object_copies":2898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":966,"num_objects_unfound":0,"num_objects_dirty":966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13832,"num_write_kb":473536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,1],"avail_no_missing":["0","1","6"],"object_location_counts":[{"shards":"0,1,6","objects":966}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":42,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.758884+0000","last_change":"2024-09-13T14:45:19.806318+0000","last_active":"2024-09-13T14:48:01.758884+0000","last_peered":"2024-09-13T14:48:01.758884+0000","last_clean":"2024-09-13T14:48:01.758884+0000","last_became_active":"2024-09-13T14:45:19.806074+0000","last_became_peered":"2024-09-13T14:45:19.806074+0000","last_unstale":"2024-09-13T14:48:01.758884+0000","last_undegraded":"2024-09-13T14:48:01.758884+0000","last_fullsized":"2024-09-13T14:48:01.758884+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:21.679797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1104,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.757046+0000","last_change":"2024-09-13T14:28:35.604404+0000","last_active":"2024-09-13T14:48:01.757046+0000","last_peered":"2024-09-13T14:48:01.757046+0000","last_clean":"2024-09-13T14:48:01.757046+0000","last_became_active":"2024-09-13T14:28:35.604102+0000","last_became_peered":"2024-09-13T14:28:35.604102+0000","last_unstale":"2024-09-13T14:48:01.757046+0000","last_undegraded":"2024-09-13T14:48:01.757046+0000","last_fullsized":"2024-09-13T14:48:01.757046+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:25.546010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":1154,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.759160+0000","last_change":"2024-09-13T14:43:07.538919+0000","last_active":"2024-09-13T14:48:01.759160+0000","last_peered":"2024-09-13T14:48:01.759160+0000","last_clean":"2024-09-13T14:48:01.759160+0000","last_became_active":"2024-09-13T14:43:07.513109+0000","last_became_peered":"2024-09-13T14:43:07.513109+0000","last_unstale":"2024-09-13T14:48:01.759160+0000","last_undegraded":"2024-09-13T14:48:01.759160+0000","last_fullsized":"2024-09-13T14:48:01.759160+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:27.771720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"261'100","reported_seq":103,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.758735+0000","last_change":"2024-09-13T14:43:44.720333+0000","last_active":"2024-09-13T14:48:01.758735+0000","last_peered":"2024-09-13T14:48:01.758735+0000","last_clean":"2024-09-13T14:48:01.758735+0000","last_became_active":"2024-09-13T14:42:14.946591+0000","last_became_peered":"2024-09-13T14:42:14.946591+0000","last_unstale":"2024-09-13T14:48:01.758735+0000","last_undegraded":"2024-09-13T14:48:01.758735+0000","last_fullsized":"2024-09-13T14:48:01.758735+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:44.720258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:44.720258+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:33.281357+0000","scrub_duration":0.0018417920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"1163'14075","reported_seq":14142,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:24.553410+0000","last_change":"2024-09-13T14:43:50.794091+0000","last_active":"2024-09-13T14:48:24.553410+0000","last_peered":"2024-09-13T14:48:24.553410+0000","last_clean":"2024-09-13T14:48:24.553410+0000","last_became_active":"2024-09-13T14:43:50.793741+0000","last_became_peered":"2024-09-13T14:43:50.793741+0000","last_unstale":"2024-09-13T14:48:24.553410+0000","last_undegraded":"2024-09-13T14:48:24.553410+0000","last_fullsized":"2024-09-13T14:48:24.553410+0000","mapping_epoch":1136,"log_start":"1150'7200","ondisk_log_start":"1150'7200","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6875,"log_dups_size":0,"ondisk_log_size":6875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:15.846494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66519040,"num_objects":1015,"num_object_clones":0,"num_object_copies":3045,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14075,"num_write_kb":482880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":58,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.805514+0000","last_change":"2024-09-13T14:44:03.858211+0000","last_active":"2024-09-13T14:48:18.805514+0000","last_peered":"2024-09-13T14:48:18.805514+0000","last_clean":"2024-09-13T14:48:18.805514+0000","last_became_active":"2024-09-13T14:44:03.857928+0000","last_became_peered":"2024-09-13T14:44:03.857928+0000","last_unstale":"2024-09-13T14:48:18.805514+0000","last_undegraded":"2024-09-13T14:48:18.805514+0000","last_fullsized":"2024-09-13T14:48:18.805514+0000","mapping_epoch":1140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:28.428693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":90,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.819401+0000","last_change":"2024-09-13T14:43:07.516243+0000","last_active":"2024-09-13T14:48:18.819401+0000","last_peered":"2024-09-13T14:48:18.819401+0000","last_clean":"2024-09-13T14:48:18.819401+0000","last_became_active":"2024-09-13T14:43:07.515863+0000","last_became_peered":"2024-09-13T14:43:07.515863+0000","last_unstale":"2024-09-13T14:48:18.819401+0000","last_undegraded":"2024-09-13T14:48:18.819401+0000","last_fullsized":"2024-09-13T14:48:18.819401+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:08.085298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1727,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.805287+0000","last_change":"2024-09-13T14:45:19.812713+0000","last_active":"2024-09-13T14:48:18.805287+0000","last_peered":"2024-09-13T14:48:18.805287+0000","last_clean":"2024-09-13T14:48:18.805287+0000","last_became_active":"2024-09-13T14:45:19.812203+0000","last_became_peered":"2024-09-13T14:45:19.812203+0000","last_unstale":"2024-09-13T14:48:18.805287+0000","last_undegraded":"2024-09-13T14:48:18.805287+0000","last_fullsized":"2024-09-13T14:48:18.805287+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:30:46.149347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"261'100","reported_seq":1283,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.747998+0000","last_change":"2024-09-13T14:43:42.577312+0000","last_active":"2024-09-13T14:48:01.747998+0000","last_peered":"2024-09-13T14:48:01.747998+0000","last_clean":"2024-09-13T14:48:01.747998+0000","last_became_active":"2024-09-13T14:42:14.928923+0000","last_became_peered":"2024-09-13T14:42:14.928923+0000","last_unstale":"2024-09-13T14:48:01.747998+0000","last_undegraded":"2024-09-13T14:48:01.747998+0000","last_fullsized":"2024-09-13T14:48:01.747998+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:42.577243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:42.577243+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:01.234250+0000","scrub_duration":0.001367839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"1163'14131","reported_seq":14166,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:27.566837+0000","last_change":"2024-09-13T14:43:50.793810+0000","last_active":"2024-09-13T14:48:27.566837+0000","last_peered":"2024-09-13T14:48:27.566837+0000","last_clean":"2024-09-13T14:48:27.566837+0000","last_became_active":"2024-09-13T14:43:50.793506+0000","last_became_peered":"2024-09-13T14:43:50.793506+0000","last_unstale":"2024-09-13T14:48:27.566837+0000","last_undegraded":"2024-09-13T14:48:27.566837+0000","last_fullsized":"2024-09-13T14:48:27.566837+0000","mapping_epoch":1136,"log_start":"1151'8000","ondisk_log_start":"1151'8000","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6131,"log_dups_size":0,"ondisk_log_size":6131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:50:32.061762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58785792,"num_objects":897,"num_object_clones":0,"num_object_copies":2691,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14131,"num_write_kb":480896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1721,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.840565+0000","last_change":"2024-09-13T14:43:07.520212+0000","last_active":"2024-09-13T14:48:18.840565+0000","last_peered":"2024-09-13T14:48:18.840565+0000","last_clean":"2024-09-13T14:48:18.840565+0000","last_became_active":"2024-09-13T14:43:07.519831+0000","last_became_peered":"2024-09-13T14:43:07.519831+0000","last_unstale":"2024-09-13T14:48:18.840565+0000","last_undegraded":"2024-09-13T14:48:18.840565+0000","last_fullsized":"2024-09-13T14:48:18.840565+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:20.990736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1743,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.826084+0000","last_change":"2024-09-13T14:45:19.809505+0000","last_active":"2024-09-13T14:48:18.826084+0000","last_peered":"2024-09-13T14:48:18.826084+0000","last_clean":"2024-09-13T14:48:18.826084+0000","last_became_active":"2024-09-13T14:45:19.809181+0000","last_became_peered":"2024-09-13T14:45:19.809181+0000","last_unstale":"2024-09-13T14:48:18.826084+0000","last_undegraded":"2024-09-13T14:48:18.826084+0000","last_fullsized":"2024-09-13T14:48:18.826084+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:24.239188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1166,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.747811+0000","last_change":"2024-09-13T14:43:07.516140+0000","last_active":"2024-09-13T14:48:01.747811+0000","last_peered":"2024-09-13T14:48:01.747811+0000","last_clean":"2024-09-13T14:48:01.747811+0000","last_became_active":"2024-09-13T14:43:07.515481+0000","last_became_peered":"2024-09-13T14:43:07.515481+0000","last_unstale":"2024-09-13T14:48:01.747811+0000","last_undegraded":"2024-09-13T14:48:01.747811+0000","last_fullsized":"2024-09-13T14:48:01.747811+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:43.706214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"261'100","reported_seq":102,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.819127+0000","last_change":"2024-09-13T14:43:16.008346+0000","last_active":"2024-09-13T14:48:18.819127+0000","last_peered":"2024-09-13T14:48:18.819127+0000","last_clean":"2024-09-13T14:48:18.819127+0000","last_became_active":"2024-09-13T14:43:07.523121+0000","last_became_peered":"2024-09-13T14:43:07.523121+0000","last_unstale":"2024-09-13T14:48:18.819127+0000","last_undegraded":"2024-09-13T14:48:18.819127+0000","last_fullsized":"2024-09-13T14:48:18.819127+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.008264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.008264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:58:35.564470+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"1163'14007","reported_seq":14078,"reported_epoch":1163,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:48:24.563203+0000","last_change":"2024-09-13T14:45:21.035848+0000","last_active":"2024-09-13T14:48:24.563203+0000","last_peered":"2024-09-13T14:48:24.563203+0000","last_clean":"2024-09-13T14:45:19.637667+0000","last_became_active":"2024-09-13T14:45:21.025308+0000","last_became_peered":"2024-09-13T14:45:21.025308+0000","last_unstale":"2024-09-13T14:48:24.563203+0000","last_undegraded":"2024-09-13T14:48:24.563203+0000","last_fullsized":"2024-09-13T14:48:24.563203+0000","mapping_epoch":1149,"log_start":"1150'7100","ondisk_log_start":"1150'7100","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6907,"log_dups_size":0,"ondisk_log_size":6907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67567616,"num_objects":1031,"num_object_clones":0,"num_object_copies":3093,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1031,"num_objects_unfound":0,"num_objects_dirty":1031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14007,"num_write_kb":481216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,5],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":1031}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":45,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.757004+0000","last_change":"2024-09-13T14:43:32.636990+0000","last_active":"2024-09-13T14:48:01.757004+0000","last_peered":"2024-09-13T14:48:01.757004+0000","last_clean":"2024-09-13T14:48:01.757004+0000","last_became_active":"2024-09-13T14:43:32.636589+0000","last_became_peered":"2024-09-13T14:43:32.636589+0000","last_unstale":"2024-09-13T14:48:01.757004+0000","last_undegraded":"2024-09-13T14:48:01.757004+0000","last_fullsized":"2024-09-13T14:48:01.757004+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:36:25.129769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":41,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.819822+0000","last_change":"2024-09-13T14:45:19.810888+0000","last_active":"2024-09-13T14:48:18.819822+0000","last_peered":"2024-09-13T14:48:18.819822+0000","last_clean":"2024-09-13T14:48:18.819822+0000","last_became_active":"2024-09-13T14:45:19.810486+0000","last_became_peered":"2024-09-13T14:45:19.810486+0000","last_unstale":"2024-09-13T14:48:18.819822+0000","last_undegraded":"2024-09-13T14:48:18.819822+0000","last_fullsized":"2024-09-13T14:48:18.819822+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:03.886942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":1119,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.760172+0000","last_change":"2024-09-13T14:29:42.539455+0000","last_active":"2024-09-13T14:48:01.760172+0000","last_peered":"2024-09-13T14:48:01.760172+0000","last_clean":"2024-09-13T14:48:01.760172+0000","last_became_active":"2024-09-13T14:29:42.539068+0000","last_became_peered":"2024-09-13T14:29:42.539068+0000","last_unstale":"2024-09-13T14:48:01.760172+0000","last_undegraded":"2024-09-13T14:48:01.760172+0000","last_fullsized":"2024-09-13T14:48:01.760172+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:24:06.048855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034405999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"261'100","reported_seq":1274,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.747870+0000","last_change":"2024-09-13T14:44:28.186029+0000","last_active":"2024-09-13T14:48:01.747870+0000","last_peered":"2024-09-13T14:48:01.747870+0000","last_clean":"2024-09-13T14:48:01.747870+0000","last_became_active":"2024-09-13T14:42:14.927771+0000","last_became_peered":"2024-09-13T14:42:14.927771+0000","last_unstale":"2024-09-13T14:48:01.747870+0000","last_undegraded":"2024-09-13T14:48:01.747870+0000","last_fullsized":"2024-09-13T14:48:01.747870+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:28.185949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:28.185949+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:38.363961+0000","scrub_duration":0.0018071129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"1163'13991","reported_seq":14047,"reported_epoch":1163,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:48:27.041240+0000","last_change":"2024-09-13T14:45:21.030922+0000","last_active":"2024-09-13T14:48:27.041240+0000","last_peered":"2024-09-13T14:48:27.041240+0000","last_clean":"2024-09-13T14:45:19.630214+0000","last_became_active":"2024-09-13T14:45:21.021807+0000","last_became_peered":"2024-09-13T14:45:21.021807+0000","last_unstale":"2024-09-13T14:48:27.041240+0000","last_undegraded":"2024-09-13T14:48:27.041240+0000","last_fullsized":"2024-09-13T14:48:27.041240+0000","mapping_epoch":1149,"log_start":"1145'5600","ondisk_log_start":"1145'5600","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":8391,"log_dups_size":0,"ondisk_log_size":8391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62455808,"num_objects":953,"num_object_clones":0,"num_object_copies":2859,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":953,"num_objects_unfound":0,"num_objects_dirty":953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13991,"num_write_kb":478208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[2,6,7],"avail_no_missing":["2","6","7"],"object_location_counts":[{"shards":"2,6,7","objects":953}],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":90,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.818669+0000","last_change":"2024-09-13T14:43:07.520176+0000","last_active":"2024-09-13T14:48:18.818669+0000","last_peered":"2024-09-13T14:48:18.818669+0000","last_clean":"2024-09-13T14:48:18.818669+0000","last_became_active":"2024-09-13T14:43:07.516748+0000","last_became_peered":"2024-09-13T14:43:07.516748+0000","last_unstale":"2024-09-13T14:48:18.818669+0000","last_undegraded":"2024-09-13T14:48:18.818669+0000","last_fullsized":"2024-09-13T14:48:18.818669+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:52:04.392821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":41,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.802713+0000","last_change":"2024-09-13T14:45:19.812353+0000","last_active":"2024-09-13T14:48:18.802713+0000","last_peered":"2024-09-13T14:48:18.802713+0000","last_clean":"2024-09-13T14:48:18.802713+0000","last_became_active":"2024-09-13T14:45:19.811743+0000","last_became_peered":"2024-09-13T14:45:19.811743+0000","last_unstale":"2024-09-13T14:48:18.802713+0000","last_undegraded":"2024-09-13T14:48:18.802713+0000","last_fullsized":"2024-09-13T14:48:18.802713+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:43.728170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"261'100","reported_seq":103,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.758843+0000","last_change":"2024-09-13T14:44:34.814157+0000","last_active":"2024-09-13T14:48:01.758843+0000","last_peered":"2024-09-13T14:48:01.758843+0000","last_clean":"2024-09-13T14:48:01.758843+0000","last_became_active":"2024-09-13T14:42:42.707803+0000","last_became_peered":"2024-09-13T14:42:42.707803+0000","last_unstale":"2024-09-13T14:48:01.758843+0000","last_undegraded":"2024-09-13T14:48:01.758843+0000","last_fullsized":"2024-09-13T14:48:01.758843+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:34.814070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:34.814070+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:38.408561+0000","scrub_duration":0.01686414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"1163'14253","reported_seq":14320,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:27.940353+0000","last_change":"2024-09-13T14:43:51.534705+0000","last_active":"2024-09-13T14:48:27.940353+0000","last_peered":"2024-09-13T14:48:27.940353+0000","last_clean":"2024-09-13T14:48:27.940353+0000","last_became_active":"2024-09-13T14:43:51.534381+0000","last_became_peered":"2024-09-13T14:43:51.534381+0000","last_unstale":"2024-09-13T14:48:27.940353+0000","last_undegraded":"2024-09-13T14:48:27.940353+0000","last_fullsized":"2024-09-13T14:48:27.940353+0000","mapping_epoch":1136,"log_start":"1153'8200","ondisk_log_start":"1153'8200","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6053,"log_dups_size":0,"ondisk_log_size":6053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:37:22.385742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57016344,"num_objects":871,"num_object_clones":0,"num_object_copies":2613,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":871,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14253,"num_write_kb":483905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1133,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.756849+0000","last_change":"2024-09-13T14:44:03.855577+0000","last_active":"2024-09-13T14:48:01.756849+0000","last_peered":"2024-09-13T14:48:01.756849+0000","last_clean":"2024-09-13T14:48:01.756849+0000","last_became_active":"2024-09-13T14:44:03.855267+0000","last_became_peered":"2024-09-13T14:44:03.855267+0000","last_unstale":"2024-09-13T14:48:01.756849+0000","last_undegraded":"2024-09-13T14:48:01.756849+0000","last_fullsized":"2024-09-13T14:48:01.756849+0000","mapping_epoch":1140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:46.148814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1104,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.757263+0000","last_change":"2024-09-13T14:29:42.545815+0000","last_active":"2024-09-13T14:48:01.757263+0000","last_peered":"2024-09-13T14:48:01.757263+0000","last_clean":"2024-09-13T14:48:01.757263+0000","last_became_active":"2024-09-13T14:29:42.545508+0000","last_became_peered":"2024-09-13T14:29:42.545508+0000","last_unstale":"2024-09-13T14:48:01.757263+0000","last_undegraded":"2024-09-13T14:48:01.757263+0000","last_fullsized":"2024-09-13T14:48:01.757263+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:39.678381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"261'100","reported_seq":108,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.818858+0000","last_change":"2024-09-13T14:43:18.988482+0000","last_active":"2024-09-13T14:48:18.818858+0000","last_peered":"2024-09-13T14:48:18.818858+0000","last_clean":"2024-09-13T14:48:18.818858+0000","last_became_active":"2024-09-13T14:43:14.556754+0000","last_became_peered":"2024-09-13T14:43:14.556754+0000","last_unstale":"2024-09-13T14:48:18.818858+0000","last_undegraded":"2024-09-13T14:48:18.818858+0000","last_fullsized":"2024-09-13T14:48:18.818858+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:18.988405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:18.988405+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:45:01.726326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"1163'13908","reported_seq":13975,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:26.945109+0000","last_change":"2024-09-13T14:43:50.796182+0000","last_active":"2024-09-13T14:48:26.945109+0000","last_peered":"2024-09-13T14:48:26.945109+0000","last_clean":"2024-09-13T14:48:26.945109+0000","last_became_active":"2024-09-13T14:43:50.795875+0000","last_became_peered":"2024-09-13T14:43:50.795875+0000","last_unstale":"2024-09-13T14:48:26.945109+0000","last_undegraded":"2024-09-13T14:48:26.945109+0000","last_fullsized":"2024-09-13T14:48:26.945109+0000","mapping_epoch":1136,"log_start":"1145'5500","ondisk_log_start":"1145'5500","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":8408,"log_dups_size":0,"ondisk_log_size":8408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:56:34.308870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56623104,"num_objects":864,"num_object_clones":0,"num_object_copies":2592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13908,"num_write_kb":472704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1150,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.757151+0000","last_change":"2024-09-13T14:43:07.515418+0000","last_active":"2024-09-13T14:48:01.757151+0000","last_peered":"2024-09-13T14:48:01.757151+0000","last_clean":"2024-09-13T14:48:01.757151+0000","last_became_active":"2024-09-13T14:43:07.515113+0000","last_became_peered":"2024-09-13T14:43:07.515113+0000","last_unstale":"2024-09-13T14:48:01.757151+0000","last_undegraded":"2024-09-13T14:48:01.757151+0000","last_fullsized":"2024-09-13T14:48:01.757151+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:10:25.810604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1117,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.756891+0000","last_change":"2024-09-13T14:43:32.636334+0000","last_active":"2024-09-13T14:48:01.756891+0000","last_peered":"2024-09-13T14:48:01.756891+0000","last_clean":"2024-09-13T14:48:01.756891+0000","last_became_active":"2024-09-13T14:43:32.635736+0000","last_became_peered":"2024-09-13T14:43:32.635736+0000","last_unstale":"2024-09-13T14:48:01.756891+0000","last_undegraded":"2024-09-13T14:48:01.756891+0000","last_fullsized":"2024-09-13T14:48:01.756891+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:49.076014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"261'100","reported_seq":108,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.758612+0000","last_change":"2024-09-13T14:45:19.805634+0000","last_active":"2024-09-13T14:48:01.758612+0000","last_peered":"2024-09-13T14:48:01.758612+0000","last_clean":"2024-09-13T14:48:01.758612+0000","last_became_active":"2024-09-13T14:45:19.805046+0000","last_became_peered":"2024-09-13T14:45:19.805046+0000","last_unstale":"2024-09-13T14:48:01.758612+0000","last_undegraded":"2024-09-13T14:48:01.758612+0000","last_fullsized":"2024-09-13T14:48:01.758612+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:32.752408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:32.752408+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:06:53.140239+0000","scrub_duration":0.0057731559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"1163'14153","reported_seq":14224,"reported_epoch":1163,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:48:24.595111+0000","last_change":"2024-09-13T14:45:21.029184+0000","last_active":"2024-09-13T14:48:24.595111+0000","last_peered":"2024-09-13T14:48:24.595111+0000","last_clean":"2024-09-13T14:45:19.633080+0000","last_became_active":"2024-09-13T14:45:21.019809+0000","last_became_peered":"2024-09-13T14:45:21.019809+0000","last_unstale":"2024-09-13T14:48:24.595111+0000","last_undegraded":"2024-09-13T14:48:24.595111+0000","last_fullsized":"2024-09-13T14:48:24.595111+0000","mapping_epoch":1149,"log_start":"1150'7300","ondisk_log_start":"1150'7300","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6853,"log_dups_size":0,"ondisk_log_size":6853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67043328,"num_objects":1023,"num_object_clones":0,"num_object_copies":3069,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1023,"num_objects_unfound":0,"num_objects_dirty":1023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14153,"num_write_kb":485632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,7,4],"avail_no_missing":["0","4","7"],"object_location_counts":[{"shards":"0,4,7","objects":1023}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1153,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.756633+0000","last_change":"2024-09-13T14:45:19.804441+0000","last_active":"2024-09-13T14:48:01.756633+0000","last_peered":"2024-09-13T14:48:01.756633+0000","last_clean":"2024-09-13T14:48:01.756633+0000","last_became_active":"2024-09-13T14:45:19.804039+0000","last_became_peered":"2024-09-13T14:45:19.804039+0000","last_unstale":"2024-09-13T14:48:01.756633+0000","last_undegraded":"2024-09-13T14:48:01.756633+0000","last_fullsized":"2024-09-13T14:48:01.756633+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:58:00.322042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":45,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.802826+0000","last_change":"2024-09-13T14:45:19.816666+0000","last_active":"2024-09-13T14:48:18.802826+0000","last_peered":"2024-09-13T14:48:18.802826+0000","last_clean":"2024-09-13T14:48:18.802826+0000","last_became_active":"2024-09-13T14:45:19.807224+0000","last_became_peered":"2024-09-13T14:45:19.807224+0000","last_unstale":"2024-09-13T14:48:18.802826+0000","last_undegraded":"2024-09-13T14:48:18.802826+0000","last_fullsized":"2024-09-13T14:48:18.802826+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:15:20.171682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"261'100","reported_seq":1293,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.747532+0000","last_change":"2024-09-13T14:45:19.798553+0000","last_active":"2024-09-13T14:48:01.747532+0000","last_peered":"2024-09-13T14:48:01.747532+0000","last_clean":"2024-09-13T14:48:01.747532+0000","last_became_active":"2024-09-13T14:45:19.798189+0000","last_became_peered":"2024-09-13T14:45:19.798189+0000","last_unstale":"2024-09-13T14:48:01.747532+0000","last_undegraded":"2024-09-13T14:48:01.747532+0000","last_fullsized":"2024-09-13T14:48:01.747532+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:29.169325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:29.169325+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:10.412666+0000","scrub_duration":0.0019358890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"1163'14198","reported_seq":14264,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:27.034281+0000","last_change":"2024-09-13T14:43:50.795029+0000","last_active":"2024-09-13T14:48:27.034281+0000","last_peered":"2024-09-13T14:48:27.034281+0000","last_clean":"2024-09-13T14:48:27.034281+0000","last_became_active":"2024-09-13T14:43:50.794715+0000","last_became_peered":"2024-09-13T14:43:50.794715+0000","last_unstale":"2024-09-13T14:48:27.034281+0000","last_undegraded":"2024-09-13T14:48:27.034281+0000","last_fullsized":"2024-09-13T14:48:27.034281+0000","mapping_epoch":1136,"log_start":"1145'5800","ondisk_log_start":"1145'5800","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":8398,"log_dups_size":0,"ondisk_log_size":8398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:21.257279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59637760,"num_objects":910,"num_object_clones":0,"num_object_copies":2730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14198,"num_write_kb":483456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1709,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.826064+0000","last_change":"2024-09-13T14:43:07.515572+0000","last_active":"2024-09-13T14:48:18.826064+0000","last_peered":"2024-09-13T14:48:18.826064+0000","last_clean":"2024-09-13T14:48:18.826064+0000","last_became_active":"2024-09-13T14:43:07.515231+0000","last_became_peered":"2024-09-13T14:43:07.515231+0000","last_unstale":"2024-09-13T14:48:18.826064+0000","last_undegraded":"2024-09-13T14:48:18.826064+0000","last_fullsized":"2024-09-13T14:48:18.826064+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:57.966380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1146,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.759047+0000","last_change":"2024-09-13T14:45:19.804982+0000","last_active":"2024-09-13T14:48:01.759047+0000","last_peered":"2024-09-13T14:48:01.759047+0000","last_clean":"2024-09-13T14:48:01.759047+0000","last_became_active":"2024-09-13T14:45:19.804663+0000","last_became_peered":"2024-09-13T14:45:19.804663+0000","last_unstale":"2024-09-13T14:48:01.759047+0000","last_undegraded":"2024-09-13T14:48:01.759047+0000","last_fullsized":"2024-09-13T14:48:01.759047+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:43.211479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"261'100","reported_seq":383,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.756762+0000","last_change":"2024-09-13T14:43:42.881584+0000","last_active":"2024-09-13T14:48:01.756762+0000","last_peered":"2024-09-13T14:48:01.756762+0000","last_clean":"2024-09-13T14:48:01.756762+0000","last_became_active":"2024-09-13T14:43:14.562161+0000","last_became_peered":"2024-09-13T14:43:14.562161+0000","last_unstale":"2024-09-13T14:48:01.756762+0000","last_undegraded":"2024-09-13T14:48:01.756762+0000","last_fullsized":"2024-09-13T14:48:01.756762+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:42.881520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:42.881520+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:06:50.353442+0000","scrub_duration":0.0013804310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"1163'14152","reported_seq":14219,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:24.551788+0000","last_change":"2024-09-13T14:43:50.793879+0000","last_active":"2024-09-13T14:48:24.551788+0000","last_peered":"2024-09-13T14:48:24.551788+0000","last_clean":"2024-09-13T14:48:24.551788+0000","last_became_active":"2024-09-13T14:43:50.793571+0000","last_became_peered":"2024-09-13T14:43:50.793571+0000","last_unstale":"2024-09-13T14:48:24.551788+0000","last_undegraded":"2024-09-13T14:48:24.551788+0000","last_fullsized":"2024-09-13T14:48:24.551788+0000","mapping_epoch":1136,"log_start":"1150'7300","ondisk_log_start":"1150'7300","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6852,"log_dups_size":0,"ondisk_log_size":6852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:22.880360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69337088,"num_objects":1058,"num_object_clones":0,"num_object_copies":3174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14152,"num_write_kb":486720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1119,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.756171+0000","last_change":"2024-09-13T14:43:07.518120+0000","last_active":"2024-09-13T14:48:01.756171+0000","last_peered":"2024-09-13T14:48:01.756171+0000","last_clean":"2024-09-13T14:48:01.756171+0000","last_became_active":"2024-09-13T14:43:07.517361+0000","last_became_peered":"2024-09-13T14:43:07.517361+0000","last_unstale":"2024-09-13T14:48:01.756171+0000","last_undegraded":"2024-09-13T14:48:01.756171+0000","last_fullsized":"2024-09-13T14:48:01.756171+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:39.366473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":1101,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.748037+0000","last_change":"2024-09-13T14:19:25.871006+0000","last_active":"2024-09-13T14:48:01.748037+0000","last_peered":"2024-09-13T14:48:01.748037+0000","last_clean":"2024-09-13T14:48:01.748037+0000","last_became_active":"2024-09-13T14:19:25.870713+0000","last_became_peered":"2024-09-13T14:19:25.870713+0000","last_unstale":"2024-09-13T14:48:01.748037+0000","last_undegraded":"2024-09-13T14:48:01.748037+0000","last_fullsized":"2024-09-13T14:48:01.748037+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:34.372124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"261'100","reported_seq":367,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.759251+0000","last_change":"2024-09-13T14:43:32.637553+0000","last_active":"2024-09-13T14:48:01.759251+0000","last_peered":"2024-09-13T14:48:01.759251+0000","last_clean":"2024-09-13T14:48:01.759251+0000","last_became_active":"2024-09-13T14:43:32.636545+0000","last_became_peered":"2024-09-13T14:43:32.636545+0000","last_unstale":"2024-09-13T14:48:01.759251+0000","last_undegraded":"2024-09-13T14:48:01.759251+0000","last_fullsized":"2024-09-13T14:48:01.759251+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:15:42.069756+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"1163'14111","reported_seq":14178,"reported_epoch":1163,"state":"active+recovery_wait+remapped","last_fresh":"2024-09-13T14:48:27.967217+0000","last_change":"2024-09-13T14:45:21.130410+0000","last_active":"2024-09-13T14:48:27.967217+0000","last_peered":"2024-09-13T14:48:27.967217+0000","last_clean":"2024-09-13T14:45:19.587556+0000","last_became_active":"2024-09-13T14:45:21.100167+0000","last_became_peered":"2024-09-13T14:45:21.100167+0000","last_unstale":"2024-09-13T14:48:27.967217+0000","last_undegraded":"2024-09-13T14:48:27.967217+0000","last_fullsized":"2024-09-13T14:48:27.967217+0000","mapping_epoch":1149,"log_start":"1152'8077","ondisk_log_start":"1152'8077","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6034,"log_dups_size":0,"ondisk_log_size":6034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59310080,"num_objects":905,"num_object_clones":0,"num_object_copies":2715,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7077,"num_objects_unfound":0,"num_objects_dirty":905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14111,"num_write_kb":480512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[1,2,5],"avail_no_missing":["1","2","5"],"object_location_counts":[{"shards":"1,2,5","objects":-6172}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1170,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.757201+0000","last_change":"2024-09-13T14:45:19.804868+0000","last_active":"2024-09-13T14:48:01.757201+0000","last_peered":"2024-09-13T14:48:01.757201+0000","last_clean":"2024-09-13T14:48:01.757201+0000","last_became_active":"2024-09-13T14:45:19.804520+0000","last_became_peered":"2024-09-13T14:45:19.804520+0000","last_unstale":"2024-09-13T14:48:01.757201+0000","last_undegraded":"2024-09-13T14:48:01.757201+0000","last_fullsized":"2024-09-13T14:48:01.757201+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:17:53.924629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":1147,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.748640+0000","last_change":"2024-09-13T14:43:07.533955+0000","last_active":"2024-09-13T14:48:01.748640+0000","last_peered":"2024-09-13T14:48:01.748640+0000","last_clean":"2024-09-13T14:48:01.748640+0000","last_became_active":"2024-09-13T14:43:07.516816+0000","last_became_peered":"2024-09-13T14:43:07.516816+0000","last_unstale":"2024-09-13T14:48:01.748640+0000","last_undegraded":"2024-09-13T14:48:01.748640+0000","last_fullsized":"2024-09-13T14:48:01.748640+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:26.232184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"261'100","reported_seq":45,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.803163+0000","last_change":"2024-09-13T14:45:19.821649+0000","last_active":"2024-09-13T14:48:18.803163+0000","last_peered":"2024-09-13T14:48:18.803163+0000","last_clean":"2024-09-13T14:48:18.803163+0000","last_became_active":"2024-09-13T14:45:19.812346+0000","last_became_peered":"2024-09-13T14:45:19.812346+0000","last_unstale":"2024-09-13T14:48:18.803163+0000","last_undegraded":"2024-09-13T14:48:18.803163+0000","last_fullsized":"2024-09-13T14:48:18.803163+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.597568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.597568+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:45.218892+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"1163'14094","reported_seq":14161,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:27.984446+0000","last_change":"2024-09-13T14:43:51.537117+0000","last_active":"2024-09-13T14:48:27.984446+0000","last_peered":"2024-09-13T14:48:27.984446+0000","last_clean":"2024-09-13T14:48:27.984446+0000","last_became_active":"2024-09-13T14:43:51.536532+0000","last_became_peered":"2024-09-13T14:43:51.536532+0000","last_unstale":"2024-09-13T14:48:27.984446+0000","last_undegraded":"2024-09-13T14:48:27.984446+0000","last_fullsized":"2024-09-13T14:48:27.984446+0000","mapping_epoch":1136,"log_start":"1152'8000","ondisk_log_start":"1152'8000","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6094,"log_dups_size":0,"ondisk_log_size":6094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:41.713211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57147392,"num_objects":872,"num_object_clones":0,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14094,"num_write_kb":478912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1088,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.758750+0000","last_change":"2024-09-13T14:19:26.887839+0000","last_active":"2024-09-13T14:48:01.758750+0000","last_peered":"2024-09-13T14:48:01.758750+0000","last_clean":"2024-09-13T14:48:01.758750+0000","last_became_active":"2024-09-13T14:19:26.887498+0000","last_became_peered":"2024-09-13T14:19:26.887498+0000","last_unstale":"2024-09-13T14:48:01.758750+0000","last_undegraded":"2024-09-13T14:48:01.758750+0000","last_fullsized":"2024-09-13T14:48:01.758750+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:24.475263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":1118,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.747961+0000","last_change":"2024-09-13T14:28:35.608607+0000","last_active":"2024-09-13T14:48:01.747961+0000","last_peered":"2024-09-13T14:48:01.747961+0000","last_clean":"2024-09-13T14:48:01.747961+0000","last_became_active":"2024-09-13T14:28:35.608254+0000","last_became_peered":"2024-09-13T14:28:35.608254+0000","last_unstale":"2024-09-13T14:48:01.747961+0000","last_undegraded":"2024-09-13T14:48:01.747961+0000","last_fullsized":"2024-09-13T14:48:01.747961+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:24.019173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"261'100","reported_seq":54,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.747799+0000","last_change":"2024-09-13T14:44:31.191431+0000","last_active":"2024-09-13T14:48:01.747799+0000","last_peered":"2024-09-13T14:48:01.747799+0000","last_clean":"2024-09-13T14:48:01.747799+0000","last_became_active":"2024-09-13T14:43:14.557376+0000","last_became_peered":"2024-09-13T14:43:14.557376+0000","last_unstale":"2024-09-13T14:48:01.747799+0000","last_undegraded":"2024-09-13T14:48:01.747799+0000","last_fullsized":"2024-09-13T14:48:01.747799+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.191326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.191326+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:15:27.352698+0000","scrub_duration":0.0025317360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1108,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.756248+0000","last_change":"2024-09-13T14:28:35.603724+0000","last_active":"2024-09-13T14:48:01.756248+0000","last_peered":"2024-09-13T14:48:01.756248+0000","last_clean":"2024-09-13T14:48:01.756248+0000","last_became_active":"2024-09-13T14:28:35.603316+0000","last_became_peered":"2024-09-13T14:28:35.603316+0000","last_unstale":"2024-09-13T14:48:01.756248+0000","last_undegraded":"2024-09-13T14:48:01.756248+0000","last_fullsized":"2024-09-13T14:48:01.756248+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:47.668223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":41,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.812011+0000","last_change":"2024-09-13T14:45:19.811466+0000","last_active":"2024-09-13T14:48:18.812011+0000","last_peered":"2024-09-13T14:48:18.812011+0000","last_clean":"2024-09-13T14:48:18.812011+0000","last_became_active":"2024-09-13T14:45:19.810617+0000","last_became_peered":"2024-09-13T14:45:19.810617+0000","last_unstale":"2024-09-13T14:48:18.812011+0000","last_undegraded":"2024-09-13T14:48:18.812011+0000","last_fullsized":"2024-09-13T14:48:18.812011+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:16.246669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"261'100","reported_seq":54,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.758976+0000","last_change":"2024-09-13T14:44:50.805306+0000","last_active":"2024-09-13T14:48:01.758976+0000","last_peered":"2024-09-13T14:48:01.758976+0000","last_clean":"2024-09-13T14:48:01.758976+0000","last_became_active":"2024-09-13T14:43:14.562922+0000","last_became_peered":"2024-09-13T14:43:14.562922+0000","last_unstale":"2024-09-13T14:48:01.758976+0000","last_undegraded":"2024-09-13T14:48:01.758976+0000","last_fullsized":"2024-09-13T14:48:01.758976+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:50.805210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:50.805210+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:28:38.529599+0000","scrub_duration":0.0042900009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1100,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.747717+0000","last_change":"2024-09-13T14:19:25.873088+0000","last_active":"2024-09-13T14:48:01.747717+0000","last_peered":"2024-09-13T14:48:01.747717+0000","last_clean":"2024-09-13T14:48:01.747717+0000","last_became_active":"2024-09-13T14:19:25.872781+0000","last_became_peered":"2024-09-13T14:19:25.872781+0000","last_unstale":"2024-09-13T14:48:01.747717+0000","last_undegraded":"2024-09-13T14:48:01.747717+0000","last_fullsized":"2024-09-13T14:48:01.747717+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:05.607382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1709,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.840709+0000","last_change":"2024-09-13T14:45:19.813876+0000","last_active":"2024-09-13T14:48:18.840709+0000","last_peered":"2024-09-13T14:48:18.840709+0000","last_clean":"2024-09-13T14:48:18.840709+0000","last_became_active":"2024-09-13T14:45:19.809136+0000","last_became_peered":"2024-09-13T14:45:19.809136+0000","last_unstale":"2024-09-13T14:48:18.840709+0000","last_undegraded":"2024-09-13T14:48:18.840709+0000","last_fullsized":"2024-09-13T14:48:18.840709+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:50.013528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"261'100","reported_seq":115,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.818989+0000","last_change":"2024-09-13T14:45:19.810654+0000","last_active":"2024-09-13T14:48:18.818989+0000","last_peered":"2024-09-13T14:48:18.818989+0000","last_clean":"2024-09-13T14:48:18.818989+0000","last_became_active":"2024-09-13T14:45:19.810066+0000","last_became_peered":"2024-09-13T14:45:19.810066+0000","last_unstale":"2024-09-13T14:48:18.818989+0000","last_undegraded":"2024-09-13T14:48:18.818989+0000","last_fullsized":"2024-09-13T14:48:18.818989+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:19.978464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:19.978464+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:33.639332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1756,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.825898+0000","last_change":"2024-09-13T14:45:19.809166+0000","last_active":"2024-09-13T14:48:18.825898+0000","last_peered":"2024-09-13T14:48:18.825898+0000","last_clean":"2024-09-13T14:48:18.825898+0000","last_became_active":"2024-09-13T14:45:19.808631+0000","last_became_peered":"2024-09-13T14:45:19.808631+0000","last_unstale":"2024-09-13T14:48:18.825898+0000","last_undegraded":"2024-09-13T14:48:18.825898+0000","last_fullsized":"2024-09-13T14:48:18.825898+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:38.822245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1125,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.759075+0000","last_change":"2024-09-13T14:45:19.804229+0000","last_active":"2024-09-13T14:48:01.759075+0000","last_peered":"2024-09-13T14:48:01.759075+0000","last_clean":"2024-09-13T14:48:01.759075+0000","last_became_active":"2024-09-13T14:45:19.803929+0000","last_became_peered":"2024-09-13T14:45:19.803929+0000","last_unstale":"2024-09-13T14:48:01.759075+0000","last_undegraded":"2024-09-13T14:48:01.759075+0000","last_fullsized":"2024-09-13T14:48:01.759075+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:33:53.675568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"261'100","reported_seq":54,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.758529+0000","last_change":"2024-09-13T14:44:42.654506+0000","last_active":"2024-09-13T14:48:01.758529+0000","last_peered":"2024-09-13T14:48:01.758529+0000","last_clean":"2024-09-13T14:48:01.758529+0000","last_became_active":"2024-09-13T14:43:14.563451+0000","last_became_peered":"2024-09-13T14:43:14.563451+0000","last_unstale":"2024-09-13T14:48:01.758529+0000","last_undegraded":"2024-09-13T14:48:01.758529+0000","last_fullsized":"2024-09-13T14:48:01.758529+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:42.654455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:42.654455+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:57:32.251597+0000","scrub_duration":0.001494097,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":1119,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-13T14:48:01.747713+0000","last_change":"2024-09-13T14:29:42.541093+0000","last_active":"2024-09-13T14:48:01.747713+0000","last_peered":"2024-09-13T14:48:01.747713+0000","last_clean":"2024-09-13T14:48:01.747713+0000","last_became_active":"2024-09-13T14:29:42.540671+0000","last_became_peered":"2024-09-13T14:29:42.540671+0000","last_unstale":"2024-09-13T14:48:01.747713+0000","last_undegraded":"2024-09-13T14:48:01.747713+0000","last_fullsized":"2024-09-13T14:48:01.747713+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:45.722376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1647,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.803434+0000","last_change":"2024-09-13T14:43:07.518679+0000","last_active":"2024-09-13T14:48:18.803434+0000","last_peered":"2024-09-13T14:48:18.803434+0000","last_clean":"2024-09-13T14:48:18.803434+0000","last_became_active":"2024-09-13T14:43:07.518333+0000","last_became_peered":"2024-09-13T14:43:07.518333+0000","last_unstale":"2024-09-13T14:48:18.803434+0000","last_undegraded":"2024-09-13T14:48:18.803434+0000","last_fullsized":"2024-09-13T14:48:18.803434+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:20.616274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"261'100","reported_seq":417,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.825908+0000","last_change":"2024-09-13T14:45:19.809268+0000","last_active":"2024-09-13T14:48:18.825908+0000","last_peered":"2024-09-13T14:48:18.825908+0000","last_clean":"2024-09-13T14:48:18.825908+0000","last_became_active":"2024-09-13T14:45:19.808772+0000","last_became_peered":"2024-09-13T14:45:19.808772+0000","last_unstale":"2024-09-13T14:48:18.825908+0000","last_undegraded":"2024-09-13T14:48:18.825908+0000","last_fullsized":"2024-09-13T14:48:18.825908+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:14.654603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:14.654603+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:40.964553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":979894296,"num_objects":14953,"num_object_clones":0,"num_object_copies":44859,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3924,"num_objects_misplaced":14189,"num_objects_unfound":0,"num_objects_dirty":14953,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":225965,"num_write_kb":7709313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2711,"num_bytes_recovered":177668096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3342282752,"data_stored":3342270536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112188,"ondisk_log_size":112188,"up":48,"acting":47,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":354515,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430581,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":665156,"kb_used_data":359484,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93079996,"statfs":{"total":95995035648,"available":95313915904,"internally_reserved":0,"allocated":368111616,"data_stored":364535835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626810,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1049980,"kb_used_data":748468,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":92695172,"statfs":{"total":95995035648,"available":94919856128,"internally_reserved":0,"allocated":766431232,"data_stored":762863004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823039,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":593780,"kb_used_data":296428,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93151372,"statfs":{"total":95995035648,"available":95387004928,"internally_reserved":0,"allocated":303542272,"data_stored":299978574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019267,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":682492,"kb_used_data":372660,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93062660,"statfs":{"total":95995035648,"available":95296163840,"internally_reserved":0,"allocated":381603840,"data_stored":378035637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059592925,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":271912,"kb_used_data":67232,"kb_used_omap":26,"kb_used_meta":204645,"kb_avail":93473240,"statfs":{"total":95995035648,"available":95716597760,"internally_reserved":0,"allocated":68845568,"data_stored":65293065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":209556640},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":591,"seq":2538325672163,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":881264,"kb_used_data":482088,"kb_used_omap":41,"kb_used_meta":399126,"kb_avail":92863888,"statfs":{"total":95995035648,"available":95092621312,"internally_reserved":0,"allocated":493658112,"data_stored":490098483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42651,"internal_metadata":408705381},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":557,"seq":2392296784106,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":921324,"kb_used_data":537892,"kb_used_omap":32,"kb_used_meta":383391,"kb_avail":92823828,"statfs":{"total":95995035648,"available":95051599872,"internally_reserved":0,"allocated":550801408,"data_stored":547244778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33752,"internal_metadata":392592424},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":524,"seq":2250562863344,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":860336,"kb_used_data":530536,"kb_used_omap":41,"kb_used_meta":329750,"kb_avail":92884816,"statfs":{"total":95995035648,"available":95114051584,"internally_reserved":0,"allocated":543268864,"data_stored":539708719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42694,"internal_metadata":337664314},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":525733888,"data_stored":525729816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":519507968,"data_stored":519503896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":481951744,"data_stored":481951744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57606144,"data_stored":57606144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":357695488,"data_stored":357695488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":290390016,"data_stored":290390016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":754847744,"data_stored":754843672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":354549760,"data_stored":354549760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:48:37.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-13T14:48:38.010 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:38.034 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:38.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:38.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:38.136 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:38.160 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:38.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:38.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:38.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:38.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:38.318 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:38.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:38.419 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:38.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:38.454 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:38.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:38.555 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:38.579 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:38.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:38.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:38.681 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:38.705 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:38.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:38.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:38.807 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:38.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:38.838 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:38.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:38.939 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:38.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:38.961 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:38.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:39.062 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:39.085 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:39.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:39.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:39.104 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:48:39.161 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:38 smithi151 ceph-mon[104450]: pgmap v1646: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 640 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 593 op/s; 4087/30780 objects degraded (13.278%); 11730/30780 objects misplaced (38.109%); 1.4 MiB/s, 21 objects/s recovering 2024-09-13T14:48:39.161 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:38 smithi151 ceph-mon[108966]: pgmap v1646: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 640 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 593 op/s; 4087/30780 objects degraded (13.278%); 11730/30780 objects misplaced (38.109%); 1.4 MiB/s, 21 objects/s recovering 2024-09-13T14:48:39.187 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:39.248 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:39.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:39.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:39.350 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:39.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:38 smithi184 ceph-mon[92527]: pgmap v1646: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 640 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 593 op/s; 4087/30780 objects degraded (13.278%); 11730/30780 objects misplaced (38.109%); 1.4 MiB/s, 21 objects/s recovering 2024-09-13T14:48:39.374 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:39.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:39.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:39.475 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:39.502 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:39.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:39.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:39.603 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:39.629 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:39.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:39.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:39.731 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:39.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:39.770 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:39.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:39.871 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:39.910 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:39.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:39.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:40.012 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:40.033 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:40.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:40.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:40.135 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:40.161 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:40.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:40.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:40.213 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:39 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 4087/30780 objects degraded (13.278%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:40.263 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:40.286 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:40.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:40.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:40.306 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:39 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 4087/30780 objects degraded (13.278%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:40.306 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:39 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 4087/30780 objects degraded (13.278%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:40.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:40.437 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:40.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:40.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:40.538 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:40.593 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:40.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:40.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:40.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:48:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:48:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:48:40.694 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:40.718 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:40.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:40.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:40.820 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:40.863 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:40.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:40.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:40.965 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:41.000 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:41.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:41.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:41.102 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:41.126 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:41.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:41.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:41.227 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:41.252 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:41.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:41.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:41.318 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:40 smithi151 ceph-mon[104450]: pgmap v1647: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 555 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 608 op/s; 4087/26688 objects degraded (15.314%); 11179/26688 objects misplaced (41.888%); 1.4 MiB/s, 21 objects/s recovering 2024-09-13T14:48:41.318 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:40 smithi151 ceph-mon[108966]: pgmap v1647: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 555 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 608 op/s; 4087/26688 objects degraded (15.314%); 11179/26688 objects misplaced (41.888%); 1.4 MiB/s, 21 objects/s recovering 2024-09-13T14:48:41.353 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:41.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:40 smithi184 ceph-mon[92527]: pgmap v1647: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 555 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 608 op/s; 4087/26688 objects degraded (15.314%); 11179/26688 objects misplaced (41.888%); 1.4 MiB/s, 21 objects/s recovering 2024-09-13T14:48:41.376 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:41.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:41.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:41.478 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:41.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:41.515 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:41.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:41.616 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:41.638 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:41.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:41.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:41.740 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:41.767 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:41.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:41.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:41.869 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:41.890 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:41.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:41.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:41.992 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:42.015 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:48:42.015 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-13T14:48:42.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.015 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:234810 7.9 7:98f8b21c:::benchmark_data_smithi151_190906_object114349:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1164)", 2024-09-13T14:48:42.016 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:48:41.307104+0000", 2024-09-13T14:48:42.016 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0093138800000000001, 2024-09-13T14:48:42.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0093450819999999993, 2024-09-13T14:48:42.016 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:48:42.016 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:48:42.016 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:48:42.016 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:48:42.016 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:48:42.016 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:48:42.016 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 234810 2024-09-13T14:48:42.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:48:42.016 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:48:42.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:48:42.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:41.307104+0000", 2024-09-13T14:48:42.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:48:42.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:48:42.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:48:42.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:41.307104+0000", 2024-09-13T14:48:42.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:48:42.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:48:42.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:48:42.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:41.307106+0000", 2024-09-13T14:48:42.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0880000000000002e-06 2024-09-13T14:48:42.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:48:42.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:48:42.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:41.307112+0000", 2024-09-13T14:48:42.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3879999999999997e-06 2024-09-13T14:48:42.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:48:42.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:48:42.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:41.307129+0000", 2024-09-13T14:48:42.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6949e-05 2024-09-13T14:48:42.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:48:42.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:48:42.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:41.307138+0000", 2024-09-13T14:48:42.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4789999999999996e-06 2024-09-13T14:48:42.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:48:42.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:48:42.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:41.307385+0000", 2024-09-13T14:48:42.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024724700000000003 2024-09-13T14:48:42.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:48:42.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:48:42.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:41.307518+0000", 2024-09-13T14:48:42.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000133041 2024-09-13T14:48:42.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:48:42.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:48:42.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:41.307644+0000", 2024-09-13T14:48:42.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012632299999999999 2024-09-13T14:48:42.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:48:42.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:42.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:41.311627+0000", 2024-09-13T14:48:42.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039828140000000003 2024-09-13T14:48:42.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:48:42.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:48:42.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:41.311666+0000", 2024-09-13T14:48:42.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8772000000000001e-05 2024-09-13T14:48:42.021 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:48:42.021 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:48:42.021 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:48:42.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:48:42.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.021 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:234813 7.6 7:6b93fcfa:::benchmark_data_smithi151_190906_object114352:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1164)", 2024-09-13T14:48:42.021 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:48:41.310669+0000", 2024-09-13T14:48:42.021 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.005748581, 2024-09-13T14:48:42.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0058592169999999999, 2024-09-13T14:48:42.021 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:48:42.022 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:48:42.022 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:48:42.022 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:48:42.022 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:48:42.022 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:48:42.022 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 234813 2024-09-13T14:48:42.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:48:42.022 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:48:42.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:48:42.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:41.310669+0000", 2024-09-13T14:48:42.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:48:42.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:48:42.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:48:42.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:41.310669+0000", 2024-09-13T14:48:42.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:48:42.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:48:42.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:48:42.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:41.310671+0000", 2024-09-13T14:48:42.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0310000000000001e-06 2024-09-13T14:48:42.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:48:42.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:48:42.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:41.310676+0000", 2024-09-13T14:48:42.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2410000000000003e-06 2024-09-13T14:48:42.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:48:42.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:48:42.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:41.310691+0000", 2024-09-13T14:48:42.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.522e-05 2024-09-13T14:48:42.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:48:42.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:48:42.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:41.310701+0000", 2024-09-13T14:48:42.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5650000000000007e-06 2024-09-13T14:48:42.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:48:42.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:48:42.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:41.310775+0000", 2024-09-13T14:48:42.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4354e-05 2024-09-13T14:48:42.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:48:42.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:48:42.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:41.310923+0000", 2024-09-13T14:48:42.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014721999999999999 2024-09-13T14:48:42.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:48:42.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,5,7", 2024-09-13T14:48:42.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:41.311062+0000", 2024-09-13T14:48:42.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013946499999999999 2024-09-13T14:48:42.025 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:48:42.025 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:48:42.025 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:48:42.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:48:42.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.026 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:234814 7.4 7:2375f7f5:::benchmark_data_smithi151_190906_object114353:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1164)", 2024-09-13T14:48:42.026 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:48:41.312342+0000", 2024-09-13T14:48:42.026 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0040758419999999997, 2024-09-13T14:48:42.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0042501780000000003, 2024-09-13T14:48:42.026 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:48:42.026 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:48:42.026 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:48:42.026 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:48:42.026 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:48:42.026 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:48:42.026 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 234814 2024-09-13T14:48:42.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:48:42.027 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:48:42.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:48:42.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:41.312342+0000", 2024-09-13T14:48:42.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:48:42.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:48:42.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:48:42.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:41.312342+0000", 2024-09-13T14:48:42.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:48:42.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:48:42.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:48:42.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:41.312343+0000", 2024-09-13T14:48:42.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.217e-06 2024-09-13T14:48:42.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:48:42.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:48:42.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:41.312348+0000", 2024-09-13T14:48:42.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2970000000000003e-06 2024-09-13T14:48:42.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:48:42.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:48:42.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:41.312366+0000", 2024-09-13T14:48:42.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8172000000000002e-05 2024-09-13T14:48:42.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:48:42.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:48:42.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:41.312374+0000", 2024-09-13T14:48:42.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.306e-06 2024-09-13T14:48:42.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:48:42.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:48:42.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:41.313229+0000", 2024-09-13T14:48:42.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00085544500000000003 2024-09-13T14:48:42.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:48:42.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:48:42.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:41.313353+0000", 2024-09-13T14:48:42.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012378500000000001 2024-09-13T14:48:42.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:48:42.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:48:42.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:48:42.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:41.313470+0000", 2024-09-13T14:48:42.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116853 2024-09-13T14:48:42.030 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:48:42.030 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:48:42.030 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:48:42.030 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:48:42.030 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-13T14:48:42.030 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 3 2024-09-13T14:48:42.030 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:48:42.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:42.032 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:42.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:42.133 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:42.176 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:42.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:42.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:42.278 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:42.304 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:42.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:42.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:42.406 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:42.428 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:42.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:42.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:42.530 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:42.555 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:42.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:42.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:42.658 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:42.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:42.683 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:42.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:42.784 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:42.824 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:42.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:42.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:42.925 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:42.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:42.950 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:42.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:43.051 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:43.075 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:43.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:43.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:43.177 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:43.199 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:43.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:43.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:43.264 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:42 smithi151 ceph-mon[104450]: pgmap v1648: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 493 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 607 op/s; 4087/23721 objects degraded (17.229%); 10581/23721 objects misplaced (44.606%); 1.4 MiB/s, 21 objects/s recovering 2024-09-13T14:48:43.264 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:42 smithi151 ceph-mon[108966]: pgmap v1648: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 493 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 607 op/s; 4087/23721 objects degraded (17.229%); 10581/23721 objects misplaced (44.606%); 1.4 MiB/s, 21 objects/s recovering 2024-09-13T14:48:43.300 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:43.323 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:43.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:43.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:43.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:42 smithi184 ceph-mon[92527]: pgmap v1648: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 493 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 607 op/s; 4087/23721 objects degraded (17.229%); 10581/23721 objects misplaced (44.606%); 1.4 MiB/s, 21 objects/s recovering 2024-09-13T14:48:43.426 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:43.451 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:43.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:43.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:43.553 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:43.597 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:43.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:43.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:43.698 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T14:48:43.699 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:43.730 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:43.755 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:43.767 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:43.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:43.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:43.869 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:43.892 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:43.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:43.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:43.994 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:44.016 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:44.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:44.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:44.117 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:44.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:44.179 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:44.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:44.280 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:44.303 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:44.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:44.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:44.405 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:44.449 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:44.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:44.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:44.551 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:44.574 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:44.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:44.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:44.676 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:44.701 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:44.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:44.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:44.803 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:44.827 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:44.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:44.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:44.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:48:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:48:44] "GET /metrics HTTP/1.1" 200 36661 "" "Prometheus/2.51.0" 2024-09-13T14:48:44.930 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:44.977 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:44.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:44.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:45.080 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:45.083 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:48:45.102 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:45.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:45.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:45.205 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:45.228 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:45.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:45.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:45.330 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:45.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:44 smithi184 ceph-mon[92527]: pgmap v1649: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 369 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 606 op/s; 4083/17778 objects degraded (22.967%); 9727/17778 objects misplaced (54.714%); 1.0 MiB/s, 21 objects/s recovering 2024-09-13T14:48:45.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:44 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 4083/17778 objects degraded (22.967%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:45.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:45.354 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:45.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:45.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:44 smithi151 ceph-mon[104450]: pgmap v1649: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 369 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 606 op/s; 4083/17778 objects degraded (22.967%); 9727/17778 objects misplaced (54.714%); 1.0 MiB/s, 21 objects/s recovering 2024-09-13T14:48:45.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:44 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 4083/17778 objects degraded (22.967%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:45.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:44 smithi151 ceph-mon[108966]: pgmap v1649: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 369 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 606 op/s; 4083/17778 objects degraded (22.967%); 9727/17778 objects misplaced (54.714%); 1.0 MiB/s, 21 objects/s recovering 2024-09-13T14:48:45.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:44 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 4083/17778 objects degraded (22.967%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:45.455 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:45.499 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:45.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:45.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:45.601 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:45.624 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:45.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:45.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:45.726 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:45.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:45.796 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:45.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:45.897 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:45.956 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:45.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:45.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:46.058 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:46.080 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:46.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:46.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:46.182 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:46.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:46.209 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:46.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:46.310 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:46.333 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:46.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:46.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:46.436 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:46.460 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:46.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:46.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:46.562 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:46.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:46.586 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:46.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:46.687 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:46.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:46.748 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:46.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:46.849 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:46.886 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:46.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:46.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:46.987 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:47.013 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:47.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:47.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:47.114 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:47.139 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:47.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:47.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:47.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:46 smithi151 ceph-mon[104450]: pgmap v1650: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 311 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 607 op/s; 4083/14958 objects degraded (27.296%); 9185/14958 objects misplaced (61.405%); 1.0 MiB/s, 21 objects/s recovering 2024-09-13T14:48:47.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:48:47.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:47.193 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:46 smithi151 ceph-mon[108966]: pgmap v1650: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 311 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 607 op/s; 4083/14958 objects degraded (27.296%); 9185/14958 objects misplaced (61.405%); 1.0 MiB/s, 21 objects/s recovering 2024-09-13T14:48:47.193 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:48:47.193 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:47.241 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:47.289 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:47.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:47.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:47.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:46 smithi184 ceph-mon[92527]: pgmap v1650: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 311 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 607 op/s; 4083/14958 objects degraded (27.296%); 9185/14958 objects misplaced (61.405%); 1.0 MiB/s, 21 objects/s recovering 2024-09-13T14:48:47.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:48:47.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:47.391 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:47.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:47.420 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:47.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:47.521 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:47.568 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:47.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:47.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:47.670 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:47.716 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:47.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:47.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:47.831 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:47.851 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:47.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:47.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:47.953 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:47.980 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:47.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:47.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:48.082 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:48.107 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:48.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:48.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:48.209 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:48.259 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:48.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:48.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:48.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:48 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/899956177' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:48:48.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:48 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/899956177' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:48:48.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:48 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/899956177' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:48:48.361 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:48.390 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:48.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:48.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:48.492 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:48.516 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:48.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:48.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:48.618 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:48.650 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:48.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:48.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:48.752 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:48.787 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:48.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:48.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:48.831 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:48:48.832 INFO:teuthology.orchestra.run.smithi151.stdout:{"fsid":"2401fc50-71da-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4083/14958 objects degraded (27.296%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1309,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":1164,"num_osds":8,"num_up_osds":8,"osd_up_since":1726237814,"num_in_osds":8,"osd_in_since":1726238586,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":110},{"state_name":"active+remapped+backfill_wait","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovery_wait+remapped","count":1}],"num_pgs":119,"num_pools":5,"num_objects":4986,"data_bytes":325615800,"bytes_used":4259811328,"bytes_avail":763700473856,"bytes_total":767960285184,"degraded_objects":4083,"degraded_total":14958,"degraded_ratio":0.2729643000401123,"misplaced_objects":9185,"misplaced_total":14958,"misplaced_ratio":0.61405268083968445,"recovering_objects_per_sec":21,"recovering_bytes_per_sec":1054704,"recovering_keys_per_sec":0,"num_objects_recovered":216,"num_bytes_recovered":10551296,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":607},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi151.rqqklw","status":"up:active","gid":24614}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.151:8443/","prometheus":"http://172.21.15.151:9283/"}},"servicemap":{"epoch":24,"modified":"2024-09-13T14:45:41.438774+0000","services":{}},"progress_events":{"5a616d1f-cba6-436c-9b6b-5733d6f6b5ec":{"message":"Global Recovery Event (0s)\n [............................] ","progress":0,"add_to_ceph_s":true},"b24a386d-2177-4b0f-a83d-e6638be369da":{"message":"Upgrade to 19.3.0-4697-gd58f7306 (4m)\n [============................] (remaining: 6m)","progress":0.44999998807907104,"add_to_ceph_s":true}}} 2024-09-13T14:48:48.889 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:48.910 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:48.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:48.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:49.012 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:49.050 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:49.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:49.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:49.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:49 smithi151 ceph-mon[104450]: pgmap v1651: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 252 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 602 op/s; 3981/12171 objects degraded (32.709%); 8800/12171 objects misplaced (72.303%); 886 KiB/s, 27 objects/s recovering 2024-09-13T14:48:49.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:49 smithi151 ceph-mon[108966]: pgmap v1651: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 252 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 602 op/s; 3981/12171 objects degraded (32.709%); 8800/12171 objects misplaced (72.303%); 886 KiB/s, 27 objects/s recovering 2024-09-13T14:48:49.152 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:49.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:49.182 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:49.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:49.283 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:49.307 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:49.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:49.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:49.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:49 smithi184 ceph-mon[92527]: pgmap v1651: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 252 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 602 op/s; 3981/12171 objects degraded (32.709%); 8800/12171 objects misplaced (72.303%); 886 KiB/s, 27 objects/s recovering 2024-09-13T14:48:49.409 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:49.446 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:49.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:49.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:49.547 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:49.588 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:49.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:49.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:49.690 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:49.729 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:49.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:49.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:49.831 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:49.869 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:49.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:49.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:49.970 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:50.010 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:50.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:50.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:50.112 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:50.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:50.134 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:50.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:50.235 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:50.258 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:50.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:50.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:50.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:50 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 3981/12171 objects degraded (32.709%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:50.354 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:48:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:48:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:48:50.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:50 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 3981/12171 objects degraded (32.709%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:50.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:50 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 3981/12171 objects degraded (32.709%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:50.361 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:50.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:50.396 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:50.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:50.498 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:50.540 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:50.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:50.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:50.642 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:50.664 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:50.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:50.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:50.766 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:50.786 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:50.802 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:50.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:50.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:50.904 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:50.929 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:50.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:50.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:51.030 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:51.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:51.080 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:51.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:51.111 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:48:51.111 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:48:51.143 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:48:51.143 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:48:51.182 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:51.186 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:51 smithi151 ceph-mon[104450]: pgmap v1652: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 176 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 618 op/s; 3981/8496 objects degraded (46.857%); 8381/8496 objects misplaced (98.646%); 309 KiB/s, 17 objects/s recovering 2024-09-13T14:48:51.186 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:48:51.187 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:48:51.187 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:51 smithi151 ceph-mon[108966]: pgmap v1652: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 176 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 618 op/s; 3981/8496 objects degraded (46.857%); 8381/8496 objects misplaced (98.646%); 309 KiB/s, 17 objects/s recovering 2024-09-13T14:48:51.226 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:48:51.226 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:48:51.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:51.232 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:51.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:51.255 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:48:51.256 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:48:51.291 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:51.291 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:48:51.292 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:48:51.332 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:48:51.332 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:48:51.332 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:51.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:51 smithi184 ceph-mon[92527]: pgmap v1652: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 176 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 618 op/s; 3981/8496 objects degraded (46.857%); 8381/8496 objects misplaced (98.646%); 309 KiB/s, 17 objects/s recovering 2024-09-13T14:48:51.369 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:48:51.370 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:48:51.394 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:51.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:51.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:51.415 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Removed 120458 objects 2024-09-13T14:48:51.415 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Clean up completed and total clean up time :198.667 2024-09-13T14:48:51.496 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:51.518 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:51.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:51.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:51.620 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:51.641 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:51.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:51.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:51.743 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:51.764 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:51.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:51.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:51.866 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:51.887 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:51.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:51.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:51.989 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:52.010 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:52.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:52.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:52.087 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T14:48:52.087 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:48:52.111 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:52.153 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:52.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:52.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:52.255 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:52.277 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:52.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:52.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:52.315 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:52 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3250556277' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:52.315 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:52 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3250556277' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:52.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:52 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3250556277' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:52.379 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:52.402 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:52.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:52.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:52.504 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:52.530 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:52.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:52.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:52.631 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:52.653 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:52.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:52.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:52.755 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:52.780 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:52.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:52.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:52.882 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:52.904 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:52.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:52.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:53.007 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:53.032 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:53.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:53.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:53.134 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:53.156 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:53.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:53.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:53.258 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:53.283 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:53.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:53.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:53.356 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:53 smithi184 ceph-mon[92527]: pgmap v1653: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 117 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 583 op/s; 3981/5655 objects degraded (70.398%); 7814/5655 objects misplaced (138.179%); 309 KiB/s, 17 objects/s recovering 2024-09-13T14:48:53.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:53 smithi151 ceph-mon[104450]: pgmap v1653: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 117 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 583 op/s; 3981/5655 objects degraded (70.398%); 7814/5655 objects misplaced (138.179%); 309 KiB/s, 17 objects/s recovering 2024-09-13T14:48:53.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:53 smithi151 ceph-mon[108966]: pgmap v1653: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 117 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 583 op/s; 3981/5655 objects degraded (70.398%); 7814/5655 objects misplaced (138.179%); 309 KiB/s, 17 objects/s recovering 2024-09-13T14:48:53.385 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:53.410 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:53.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:53.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:53.512 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:53.538 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:53.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:53.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:53.641 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:53.667 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:53.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:53.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:53.768 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:53.793 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:53.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:53.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:53.895 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:53.920 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:53.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:53.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:54.022 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:54.045 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:54.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:54.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:54.148 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:54.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:54.173 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:54.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:54.274 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:54.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:54.301 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:54.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:54.402 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:54.427 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:54.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:54.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:54.529 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:54.536 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-09-13T14:48:54.537 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-09-13T14:48:54.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:54.553 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:54.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:54.654 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:54.677 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:54.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:54.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:54.705 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:48:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:48:54] "GET /metrics HTTP/1.1" 200 36667 "" "Prometheus/2.51.0" 2024-09-13T14:48:54.778 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:54.804 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:54.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:54.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:54.906 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:54.929 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:54.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:54.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:55.031 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:55.057 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:55.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:55.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:55.158 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:55.179 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:55.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:55.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:55.281 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:55.290 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:55 smithi151 ceph-mon[104450]: pgmap v1654: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 19 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 632 op/s; 3809/954 objects degraded (399.266%); 7258/954 objects misplaced (760.797%); 309 KiB/s, 32 objects/s recovering 2024-09-13T14:48:55.290 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:55 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 3809/954 objects degraded (399.266%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:55.291 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:55 smithi151 ceph-mon[108966]: pgmap v1654: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 19 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 632 op/s; 3809/954 objects degraded (399.266%); 7258/954 objects misplaced (760.797%); 309 KiB/s, 32 objects/s recovering 2024-09-13T14:48:55.291 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:55 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 3809/954 objects degraded (399.266%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:55.306 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:55.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:55.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:55.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:55 smithi184 ceph-mon[92527]: pgmap v1654: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 19 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 632 op/s; 3809/954 objects degraded (399.266%); 7258/954 objects misplaced (760.797%); 309 KiB/s, 32 objects/s recovering 2024-09-13T14:48:55.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:55 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 3809/954 objects degraded (399.266%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:48:55.408 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:55.431 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:55.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:55.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:55.533 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:55.558 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:55.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:55.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:55.660 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:55.683 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:55.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:55.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:55.785 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:55.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:55.813 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:55.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:55.818 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:48:55.914 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:55.935 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:55.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:55.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:56.037 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:56.059 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:56.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:56.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:56.161 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:56.183 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:56.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:56.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:56.284 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:56.310 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:56.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:56.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:56.411 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:56.436 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:56.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:56.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:56.538 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:56.562 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:56.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:56.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:56.664 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:56.690 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:56.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:56.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:56.792 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:56.817 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:56.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:56.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:56.920 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:56.945 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:56.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:56.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:57.049 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:57.073 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:57.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:57.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:57.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:57.197 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:57.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:57.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:57.299 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:57.324 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:57.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:57.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:57.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:57 smithi184 ceph-mon[92527]: pgmap v1655: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 608 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 491 op/s; 3809/78 objects degraded (4883.333%); 7077/78 objects misplaced (9073.077%); 0 B/s, 22 objects/s recovering 2024-09-13T14:48:57.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:57 smithi151 ceph-mon[104450]: pgmap v1655: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 608 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 491 op/s; 3809/78 objects degraded (4883.333%); 7077/78 objects misplaced (9073.077%); 0 B/s, 22 objects/s recovering 2024-09-13T14:48:57.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:57 smithi151 ceph-mon[108966]: pgmap v1655: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 608 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 491 op/s; 3809/78 objects degraded (4883.333%); 7077/78 objects misplaced (9073.077%); 0 B/s, 22 objects/s recovering 2024-09-13T14:48:57.427 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:57.451 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:57.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:57.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:57.553 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:57.579 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:57.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:57.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:57.681 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:57.705 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:57.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:57.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:57.807 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:57.830 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:57.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:57.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:57.932 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:57.958 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:57.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:57.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:58.060 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:58.081 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:58.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:58.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:58.183 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:58.207 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:58.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:58.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:58.309 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:58.312 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:48:58.313 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:48:58.313 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:58 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:48:58.313 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:58 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:48:58.330 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:58.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:58.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:58.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:58 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:48:58.432 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:58.454 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:58.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:58.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:58.556 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:58.579 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:58.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:58.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:58.682 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:58.704 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:58.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:58.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:58.807 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:58.828 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:58.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:58.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:58.930 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:58.953 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:58.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:58.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:59.055 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:59.077 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:59.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:59.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:59.183 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:59.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:59.206 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:59.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:59.307 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:59.328 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:59.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:59.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:59.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:48:59 smithi184 ceph-mon[92527]: pgmap v1656: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 608 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 413 op/s; 3239/78 objects degraded (4152.564%); 7077/78 objects misplaced (9073.077%); 0 B/s, 70 objects/s recovering 2024-09-13T14:48:59.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:48:59 smithi151 ceph-mon[104450]: pgmap v1656: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 608 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 413 op/s; 3239/78 objects degraded (4152.564%); 7077/78 objects misplaced (9073.077%); 0 B/s, 70 objects/s recovering 2024-09-13T14:48:59.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:48:59 smithi151 ceph-mon[108966]: pgmap v1656: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 608 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 413 op/s; 3239/78 objects degraded (4152.564%); 7077/78 objects misplaced (9073.077%); 0 B/s, 70 objects/s recovering 2024-09-13T14:48:59.430 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:59.453 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:59.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:59.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:59.555 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:59.577 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:59.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:59.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:59.679 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:59.700 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:48:59.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:59.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:59.802 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:59.824 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:59.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:59.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:59.926 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:59.948 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:48:59.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:59.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:00.050 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:00.072 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:00.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:00.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:00.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:00.198 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:00.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:00.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:00.300 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:00.322 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:00.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:00.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:00.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:00 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 3239/78 objects degraded (4152.564%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:00.354 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:49:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:49:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:49:00.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:00 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 3239/78 objects degraded (4152.564%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:00.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:00 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 3239/78 objects degraded (4152.564%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:00.424 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:00.445 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:00.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:00.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:00.546 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:00.568 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:00.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:00.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:00.670 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:00.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:00.692 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:00.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:00.736 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:00.745 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T14:49:00.752 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:00.793 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:00.815 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:00.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:00.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:00.917 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:00.938 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:00.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:00.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:00.984 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:01.040 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:01.061 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:01.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:01.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:01.163 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:01.184 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:01.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:01.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:01.238 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:01 smithi151 ceph-mon[104450]: pgmap v1657: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 608 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 335 op/s; 3239/78 objects degraded (4152.564%); 7077/78 objects misplaced (9073.077%); 0 B/s, 61 objects/s recovering 2024-09-13T14:49:01.238 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:01 smithi151 ceph-mon[108966]: pgmap v1657: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 608 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 335 op/s; 3239/78 objects degraded (4152.564%); 7077/78 objects misplaced (9073.077%); 0 B/s, 61 objects/s recovering 2024-09-13T14:49:01.286 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:01.310 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:01.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:01.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:01.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:01 smithi184 ceph-mon[92527]: pgmap v1657: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 608 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 335 op/s; 3239/78 objects degraded (4152.564%); 7077/78 objects misplaced (9073.077%); 0 B/s, 61 objects/s recovering 2024-09-13T14:49:01.411 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:01.434 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:01.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:01.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:01.536 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:01.560 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:01.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:01.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:01.662 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:01.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:01.687 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:01.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:01.788 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:01.810 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:01.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:01.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:01.912 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:01.934 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:01.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:01.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:02.037 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:02.059 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:02.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:02.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:02.161 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:02.184 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:02.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:02.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:02.264 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:02.264 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:02 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/4244306159' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:02.264 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:02 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:02.265 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:02.265 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:02 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/4244306159' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:02.265 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:02 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:02.287 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:02.308 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:02.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:02.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:02.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:02.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:02 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/4244306159' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:02.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:02 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:02.409 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:02.431 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:02.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:02.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:02.532 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:02.555 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:02.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:02.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:02.657 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:02.679 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:02.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:02.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:02.781 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:02.803 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:02.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:02.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:02.905 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:02.928 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:02.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:02.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:03.030 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:03.052 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:03.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:03.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:03.154 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:03.178 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:03.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:03.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:03.280 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:03.301 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:03.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:03.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:03.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:03 smithi184 ceph-mon[92527]: pgmap v1658: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 608 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 233 op/s; 3239/78 objects degraded (4152.564%); 7077/78 objects misplaced (9073.077%); 0 B/s, 61 objects/s recovering 2024-09-13T14:49:03.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:03 smithi184 ceph-mon[92527]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:03.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:03 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:49:03.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:03 smithi184 ceph-mon[92527]: osdmap e1165: 8 total, 8 up, 8 in 2024-09-13T14:49:03.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:03 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:49:03.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:03 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:49:03.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:03 smithi151 ceph-mon[104450]: pgmap v1658: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 608 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 233 op/s; 3239/78 objects degraded (4152.564%); 7077/78 objects misplaced (9073.077%); 0 B/s, 61 objects/s recovering 2024-09-13T14:49:03.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:03 smithi151 ceph-mon[104450]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:03.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:03 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:49:03.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:03 smithi151 ceph-mon[104450]: osdmap e1165: 8 total, 8 up, 8 in 2024-09-13T14:49:03.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:03 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:49:03.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:03 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:49:03.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:03 smithi151 ceph-mon[108966]: pgmap v1658: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 608 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 233 op/s; 3239/78 objects degraded (4152.564%); 7077/78 objects misplaced (9073.077%); 0 B/s, 61 objects/s recovering 2024-09-13T14:49:03.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:03 smithi151 ceph-mon[108966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:03.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:03 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:49:03.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:03 smithi151 ceph-mon[108966]: osdmap e1165: 8 total, 8 up, 8 in 2024-09-13T14:49:03.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:03 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:49:03.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:03 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:49:03.404 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:03.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:03.427 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:03.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:03.528 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:03.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:03.551 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:03.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:03.611 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-13T14:49:03.652 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:03.674 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:03.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:03.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:03.776 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:03.799 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:03.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:03.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:03.901 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:03.923 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:03.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:03.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:04.025 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:04.049 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:04.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:04.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:04.151 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:04.174 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:04.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:04.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:04.276 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:04.299 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:04.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:04.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:04.401 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:04.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:04.428 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:04.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:04.529 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:04.555 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:04.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:04.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:04.657 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:04.682 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:04.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:04.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:04.784 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:04.809 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:04.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:04.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:04.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:49:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:49:04] "GET /metrics HTTP/1.1" 200 36647 "" "Prometheus/2.51.0" 2024-09-13T14:49:04.911 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:04.935 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:04.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:04.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:05.037 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:05.062 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:05.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:05.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:05.164 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:05.184 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:05 smithi151 ceph-mon[104450]: pgmap v1660: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 608 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 29 op/s; 2593/78 objects degraded (3324.359%); 7077/78 objects misplaced (9073.077%); 0 B/s, 121 objects/s recovering 2024-09-13T14:49:05.184 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:05 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 2593/78 objects degraded (3324.359%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:05.185 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:05 smithi151 ceph-mon[108966]: pgmap v1660: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 608 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 29 op/s; 2593/78 objects degraded (3324.359%); 7077/78 objects misplaced (9073.077%); 0 B/s, 121 objects/s recovering 2024-09-13T14:49:05.185 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:05 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 2593/78 objects degraded (3324.359%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:05.188 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:05.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:05.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:05.290 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:05.312 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:05.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:05.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:05.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:05 smithi184 ceph-mon[92527]: pgmap v1660: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 608 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 29 op/s; 2593/78 objects degraded (3324.359%); 7077/78 objects misplaced (9073.077%); 0 B/s, 121 objects/s recovering 2024-09-13T14:49:05.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:05 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 2593/78 objects degraded (3324.359%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:05.415 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:05.440 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:05.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:05.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:05.542 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:05.567 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:05.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:05.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:05.669 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:05.693 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:05.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:05.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:05.718 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:49:05.719 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T14:49:05.794 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:05.818 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:05.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:05.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:05.920 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:05.944 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:05.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:05.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:06.046 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:06.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:06.074 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:06.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:06.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:06.197 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:06.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:06.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:06.299 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:06.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:06.325 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:06.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:06.415 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:49:06.427 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:06.453 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:06.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:06.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:06.555 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:06.580 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:06.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:06.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:06.682 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:06.706 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:06.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:06.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:06.808 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:06.830 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:06.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:06.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:06.932 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:06.958 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:06.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:06.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:07.060 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:07.085 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:07.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:07.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:07.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:07 smithi151 ceph-mon[104450]: pgmap v1661: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 608 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2593/78 objects degraded (3324.359%); 7077/78 objects misplaced (9073.077%); 0 B/s, 121 objects/s recovering 2024-09-13T14:49:07.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:07 smithi151 ceph-mon[104450]: from='client.35877 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:49:07.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:07 smithi151 ceph-mon[108966]: pgmap v1661: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 608 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2593/78 objects degraded (3324.359%); 7077/78 objects misplaced (9073.077%); 0 B/s, 121 objects/s recovering 2024-09-13T14:49:07.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:07 smithi151 ceph-mon[108966]: from='client.35877 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:49:07.187 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:07.211 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:07.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:07.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:07.313 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:07.338 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:07.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:07.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:07.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:07 smithi184 ceph-mon[92527]: pgmap v1661: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 608 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2593/78 objects degraded (3324.359%); 7077/78 objects misplaced (9073.077%); 0 B/s, 121 objects/s recovering 2024-09-13T14:49:07.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:07 smithi184 ceph-mon[92527]: from='client.35877 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:49:07.440 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:07.462 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:07.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:07.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:07.564 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:07.590 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:07.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:07.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:07.691 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:07.713 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:07.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:07.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:07.815 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:07.841 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:07.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:07.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:07.872 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1661,"stamp":"2024-09-13T14:49:05.532003+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":622752,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2593,"num_objects_misplaced":7077,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":35,"num_write":240988,"num_write_kb":7709418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4546,"num_bytes_recovered":196110585,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115299,"ondisk_log_size":115299,"up":357,"acting":356,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":366,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3081488,"kb_used_data":540296,"kb_used_omap":149,"kb_used_meta":2540970,"kb_avail":746879728,"statfs":{"total":767960285184,"available":764804841472,"internally_reserved":0,"allocated":553263104,"data_stored":524685814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":153401,"internal_metadata":2601953479},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1216,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1216,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002469"},"pg_stats":[{"pgid":"1.1d","version":"261'100","reported_seq":99,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.913087+0000","last_change":"2024-09-13T14:45:19.807356+0000","last_active":"2024-09-13T14:48:35.913087+0000","last_peered":"2024-09-13T14:48:35.913087+0000","last_clean":"2024-09-13T14:48:35.913087+0000","last_became_active":"2024-09-13T14:45:19.807081+0000","last_became_peered":"2024-09-13T14:45:19.807081+0000","last_unstale":"2024-09-13T14:48:35.913087+0000","last_undegraded":"2024-09-13T14:48:35.913087+0000","last_fullsized":"2024-09-13T14:48:35.913087+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:17.988345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:17.988345+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:41:05.216335+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":1706,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:02.409447+0000","last_change":"2024-09-13T14:29:42.541263+0000","last_active":"2024-09-13T14:49:02.409447+0000","last_peered":"2024-09-13T14:49:02.409447+0000","last_clean":"2024-09-13T14:49:02.409447+0000","last_became_active":"2024-09-13T14:29:42.540873+0000","last_became_peered":"2024-09-13T14:29:42.540873+0000","last_unstale":"2024-09-13T14:49:02.409447+0000","last_undegraded":"2024-09-13T14:49:02.409447+0000","last_fullsized":"2024-09-13T14:49:02.409447+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:20.611206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1125,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.916565+0000","last_change":"2024-09-13T14:43:07.516715+0000","last_active":"2024-09-13T14:48:35.916565+0000","last_peered":"2024-09-13T14:48:35.916565+0000","last_clean":"2024-09-13T14:48:35.916565+0000","last_became_active":"2024-09-13T14:43:07.516403+0000","last_became_peered":"2024-09-13T14:43:07.516403+0000","last_unstale":"2024-09-13T14:48:35.916565+0000","last_undegraded":"2024-09-13T14:48:35.916565+0000","last_fullsized":"2024-09-13T14:48:35.916565+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:56.469690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":45,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:03.002701+0000","last_change":"2024-09-13T14:45:19.810203+0000","last_active":"2024-09-13T14:49:03.002701+0000","last_peered":"2024-09-13T14:49:03.002701+0000","last_clean":"2024-09-13T14:49:03.002701+0000","last_became_active":"2024-09-13T14:45:19.809693+0000","last_became_peered":"2024-09-13T14:45:19.809693+0000","last_unstale":"2024-09-13T14:49:03.002701+0000","last_undegraded":"2024-09-13T14:49:03.002701+0000","last_fullsized":"2024-09-13T14:49:03.002701+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:20.219673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":45,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:03.002275+0000","last_change":"2024-09-13T14:45:19.807991+0000","last_active":"2024-09-13T14:49:03.002275+0000","last_peered":"2024-09-13T14:49:03.002275+0000","last_clean":"2024-09-13T14:49:03.002275+0000","last_became_active":"2024-09-13T14:45:19.807662+0000","last_became_peered":"2024-09-13T14:45:19.807662+0000","last_unstale":"2024-09-13T14:49:03.002275+0000","last_undegraded":"2024-09-13T14:49:03.002275+0000","last_fullsized":"2024-09-13T14:49:03.002275+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:29.306108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"261'100","reported_seq":1236,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.901467+0000","last_change":"2024-09-13T14:44:54.259562+0000","last_active":"2024-09-13T14:48:35.901467+0000","last_peered":"2024-09-13T14:48:35.901467+0000","last_clean":"2024-09-13T14:48:35.901467+0000","last_became_active":"2024-09-13T14:43:14.563717+0000","last_became_peered":"2024-09-13T14:43:14.563717+0000","last_unstale":"2024-09-13T14:48:35.901467+0000","last_undegraded":"2024-09-13T14:48:35.901467+0000","last_fullsized":"2024-09-13T14:48:35.901467+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_clean_scrub_stamp":"2024-09-13T14:44:54.259482+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:43:52.729109+0000","scrub_duration":0.015265569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":1093,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.903447+0000","last_change":"2024-09-13T14:19:25.873514+0000","last_active":"2024-09-13T14:48:35.903447+0000","last_peered":"2024-09-13T14:48:35.903447+0000","last_clean":"2024-09-13T14:48:35.903447+0000","last_became_active":"2024-09-13T14:19:25.873170+0000","last_became_peered":"2024-09-13T14:19:25.873170+0000","last_unstale":"2024-09-13T14:48:35.903447+0000","last_undegraded":"2024-09-13T14:48:35.903447+0000","last_fullsized":"2024-09-13T14:48:35.903447+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:48.298530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1122,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.915586+0000","last_change":"2024-09-13T14:30:15.580011+0000","last_active":"2024-09-13T14:48:35.915586+0000","last_peered":"2024-09-13T14:48:35.915586+0000","last_clean":"2024-09-13T14:48:35.915586+0000","last_became_active":"2024-09-13T14:30:15.579610+0000","last_became_peered":"2024-09-13T14:30:15.579610+0000","last_unstale":"2024-09-13T14:48:35.915586+0000","last_undegraded":"2024-09-13T14:48:35.915586+0000","last_fullsized":"2024-09-13T14:48:35.915586+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:11:02.733771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"261'100","reported_seq":62,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:03.002812+0000","last_change":"2024-09-13T14:45:35.947332+0000","last_active":"2024-09-13T14:49:03.002812+0000","last_peered":"2024-09-13T14:49:03.002812+0000","last_clean":"2024-09-13T14:49:03.002812+0000","last_became_active":"2024-09-13T14:45:19.810775+0000","last_became_peered":"2024-09-13T14:45:19.810775+0000","last_unstale":"2024-09-13T14:49:03.002812+0000","last_undegraded":"2024-09-13T14:49:03.002812+0000","last_fullsized":"2024-09-13T14:49:03.002812+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:35.947265+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:43:41.569795+0000","last_clean_scrub_stamp":"2024-09-13T14:45:35.947265+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:00.606370+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":1138,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.903219+0000","last_change":"2024-09-13T14:43:07.536116+0000","last_active":"2024-09-13T14:48:35.903219+0000","last_peered":"2024-09-13T14:48:35.903219+0000","last_clean":"2024-09-13T14:48:35.903219+0000","last_became_active":"2024-09-13T14:43:07.524112+0000","last_became_peered":"2024-09-13T14:43:07.524112+0000","last_unstale":"2024-09-13T14:48:35.903219+0000","last_undegraded":"2024-09-13T14:48:35.903219+0000","last_fullsized":"2024-09-13T14:48:35.903219+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:36.947060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1135,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.916339+0000","last_change":"2024-09-13T14:43:07.530236+0000","last_active":"2024-09-13T14:48:35.916339+0000","last_peered":"2024-09-13T14:48:35.916339+0000","last_clean":"2024-09-13T14:48:35.916339+0000","last_became_active":"2024-09-13T14:43:07.529967+0000","last_became_peered":"2024-09-13T14:43:07.529967+0000","last_unstale":"2024-09-13T14:48:35.916339+0000","last_undegraded":"2024-09-13T14:48:35.916339+0000","last_fullsized":"2024-09-13T14:48:35.916339+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:27.276550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":1691,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:02.408951+0000","last_change":"2024-09-13T14:28:35.612469+0000","last_active":"2024-09-13T14:49:02.408951+0000","last_peered":"2024-09-13T14:49:02.408951+0000","last_clean":"2024-09-13T14:49:02.408951+0000","last_became_active":"2024-09-13T14:28:35.612093+0000","last_became_peered":"2024-09-13T14:28:35.612093+0000","last_unstale":"2024-09-13T14:49:02.408951+0000","last_undegraded":"2024-09-13T14:49:02.408951+0000","last_fullsized":"2024-09-13T14:49:02.408951+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:51.022663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1711,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.910263+0000","last_change":"2024-09-13T14:45:19.808811+0000","last_active":"2024-09-13T14:48:35.910263+0000","last_peered":"2024-09-13T14:48:35.910263+0000","last_clean":"2024-09-13T14:48:35.910263+0000","last_became_active":"2024-09-13T14:45:19.808471+0000","last_became_peered":"2024-09-13T14:45:19.808471+0000","last_unstale":"2024-09-13T14:48:35.910263+0000","last_undegraded":"2024-09-13T14:48:35.910263+0000","last_fullsized":"2024-09-13T14:48:35.910263+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:53.638161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"261'100","reported_seq":45,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:03.002368+0000","last_change":"2024-09-13T14:45:19.813412+0000","last_active":"2024-09-13T14:49:03.002368+0000","last_peered":"2024-09-13T14:49:03.002368+0000","last_clean":"2024-09-13T14:49:03.002368+0000","last_became_active":"2024-09-13T14:45:19.812585+0000","last_became_peered":"2024-09-13T14:45:19.812585+0000","last_unstale":"2024-09-13T14:49:03.002368+0000","last_undegraded":"2024-09-13T14:49:03.002368+0000","last_fullsized":"2024-09-13T14:49:03.002368+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.614340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.614340+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:32:07.434109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":1144,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.915658+0000","last_change":"2024-09-13T14:43:07.522950+0000","last_active":"2024-09-13T14:48:35.915658+0000","last_peered":"2024-09-13T14:48:35.915658+0000","last_clean":"2024-09-13T14:48:35.915658+0000","last_became_active":"2024-09-13T14:43:07.522566+0000","last_became_peered":"2024-09-13T14:43:07.522566+0000","last_unstale":"2024-09-13T14:48:35.915658+0000","last_undegraded":"2024-09-13T14:48:35.915658+0000","last_fullsized":"2024-09-13T14:48:35.915658+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:03.201478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":45,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:03.002624+0000","last_change":"2024-09-13T14:45:19.807231+0000","last_active":"2024-09-13T14:49:03.002624+0000","last_peered":"2024-09-13T14:49:03.002624+0000","last_clean":"2024-09-13T14:49:03.002624+0000","last_became_active":"2024-09-13T14:45:19.806936+0000","last_became_peered":"2024-09-13T14:45:19.806936+0000","last_unstale":"2024-09-13T14:49:03.002624+0000","last_undegraded":"2024-09-13T14:49:03.002624+0000","last_fullsized":"2024-09-13T14:49:03.002624+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:04:26.723153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.18","version":"261'100","reported_seq":114,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.902695+0000","last_change":"2024-09-13T14:44:41.685537+0000","last_active":"2024-09-13T14:48:35.902695+0000","last_peered":"2024-09-13T14:48:35.902695+0000","last_clean":"2024-09-13T14:48:35.902695+0000","last_became_active":"2024-09-13T14:43:14.558502+0000","last_became_peered":"2024-09-13T14:43:14.558502+0000","last_unstale":"2024-09-13T14:48:35.902695+0000","last_undegraded":"2024-09-13T14:48:35.902695+0000","last_fullsized":"2024-09-13T14:48:35.902695+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1134,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:41.685124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:41.685124+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:37:58.412497+0000","scrub_duration":0.0075205970000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":92,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.912459+0000","last_change":"2024-09-13T14:43:07.516845+0000","last_active":"2024-09-13T14:48:35.912459+0000","last_peered":"2024-09-13T14:48:35.912459+0000","last_clean":"2024-09-13T14:48:35.912459+0000","last_became_active":"2024-09-13T14:43:07.514974+0000","last_became_peered":"2024-09-13T14:43:07.514974+0000","last_unstale":"2024-09-13T14:48:35.912459+0000","last_undegraded":"2024-09-13T14:48:35.912459+0000","last_fullsized":"2024-09-13T14:48:35.912459+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:55.419483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1729,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:02.409219+0000","last_change":"2024-09-13T14:45:19.815341+0000","last_active":"2024-09-13T14:49:02.409219+0000","last_peered":"2024-09-13T14:49:02.409219+0000","last_clean":"2024-09-13T14:49:02.409219+0000","last_became_active":"2024-09-13T14:45:19.810262+0000","last_became_peered":"2024-09-13T14:45:19.810262+0000","last_unstale":"2024-09-13T14:49:02.409219+0000","last_undegraded":"2024-09-13T14:49:02.409219+0000","last_fullsized":"2024-09-13T14:49:02.409219+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:39.847653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1b","version":"261'100","reported_seq":56,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.902827+0000","last_change":"2024-09-13T14:44:52.803330+0000","last_active":"2024-09-13T14:48:35.902827+0000","last_peered":"2024-09-13T14:48:35.902827+0000","last_clean":"2024-09-13T14:48:35.902827+0000","last_became_active":"2024-09-13T14:43:14.556712+0000","last_became_peered":"2024-09-13T14:43:14.556712+0000","last_unstale":"2024-09-13T14:48:35.902827+0000","last_undegraded":"2024-09-13T14:48:35.902827+0000","last_fullsized":"2024-09-13T14:48:35.902827+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:52.803242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:52.803242+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:06.658301+0000","scrub_duration":0.0019769660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":45,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:03.002051+0000","last_change":"2024-09-13T14:45:19.806623+0000","last_active":"2024-09-13T14:49:03.002051+0000","last_peered":"2024-09-13T14:49:03.002051+0000","last_clean":"2024-09-13T14:49:03.002051+0000","last_became_active":"2024-09-13T14:45:19.806219+0000","last_became_peered":"2024-09-13T14:45:19.806219+0000","last_unstale":"2024-09-13T14:49:03.002051+0000","last_undegraded":"2024-09-13T14:49:03.002051+0000","last_fullsized":"2024-09-13T14:49:03.002051+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:34:41.601201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1118,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.903639+0000","last_change":"2024-09-13T14:43:32.637751+0000","last_active":"2024-09-13T14:48:35.903639+0000","last_peered":"2024-09-13T14:48:35.903639+0000","last_clean":"2024-09-13T14:48:35.903639+0000","last_became_active":"2024-09-13T14:43:32.636901+0000","last_became_peered":"2024-09-13T14:43:32.636901+0000","last_unstale":"2024-09-13T14:48:35.903639+0000","last_undegraded":"2024-09-13T14:48:35.903639+0000","last_fullsized":"2024-09-13T14:48:35.903639+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:21.942419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1a","version":"261'100","reported_seq":56,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.915402+0000","last_change":"2024-09-13T14:43:41.910041+0000","last_active":"2024-09-13T14:48:35.915402+0000","last_peered":"2024-09-13T14:48:35.915402+0000","last_clean":"2024-09-13T14:48:35.915402+0000","last_became_active":"2024-09-13T14:43:14.555083+0000","last_became_peered":"2024-09-13T14:43:14.555083+0000","last_unstale":"2024-09-13T14:48:35.915402+0000","last_undegraded":"2024-09-13T14:48:35.915402+0000","last_fullsized":"2024-09-13T14:48:35.915402+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:41.909988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:41.909988+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:42:19.954838+0000","scrub_duration":0.0012891619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":45,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:03.002146+0000","last_change":"2024-09-13T14:45:19.812071+0000","last_active":"2024-09-13T14:49:03.002146+0000","last_peered":"2024-09-13T14:49:03.002146+0000","last_clean":"2024-09-13T14:49:03.002146+0000","last_became_active":"2024-09-13T14:45:19.811336+0000","last_became_peered":"2024-09-13T14:45:19.811336+0000","last_unstale":"2024-09-13T14:49:03.002146+0000","last_undegraded":"2024-09-13T14:49:03.002146+0000","last_fullsized":"2024-09-13T14:49:03.002146+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:40:29.814608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1746,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:02.409442+0000","last_change":"2024-09-13T14:45:19.810478+0000","last_active":"2024-09-13T14:49:02.409442+0000","last_peered":"2024-09-13T14:49:02.409442+0000","last_clean":"2024-09-13T14:49:02.409442+0000","last_became_active":"2024-09-13T14:45:19.808278+0000","last_became_peered":"2024-09-13T14:45:19.808278+0000","last_unstale":"2024-09-13T14:49:02.409442+0000","last_undegraded":"2024-09-13T14:49:02.409442+0000","last_fullsized":"2024-09-13T14:49:02.409442+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:12.211089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"261'100","reported_seq":56,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.901668+0000","last_change":"2024-09-13T14:44:38.214961+0000","last_active":"2024-09-13T14:48:35.901668+0000","last_peered":"2024-09-13T14:48:35.901668+0000","last_clean":"2024-09-13T14:48:35.901668+0000","last_became_active":"2024-09-13T14:43:14.557883+0000","last_became_peered":"2024-09-13T14:43:14.557883+0000","last_unstale":"2024-09-13T14:48:35.901668+0000","last_undegraded":"2024-09-13T14:48:35.901668+0000","last_fullsized":"2024-09-13T14:48:35.901668+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:38.214865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:38.214865+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:49:36.013783+0000","scrub_duration":0.0016389639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1617,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:03.002448+0000","last_change":"2024-09-13T14:30:15.576921+0000","last_active":"2024-09-13T14:49:03.002448+0000","last_peered":"2024-09-13T14:49:03.002448+0000","last_clean":"2024-09-13T14:49:03.002448+0000","last_became_active":"2024-09-13T14:30:15.576607+0000","last_became_peered":"2024-09-13T14:30:15.576607+0000","last_unstale":"2024-09-13T14:49:03.002448+0000","last_undegraded":"2024-09-13T14:49:03.002448+0000","last_fullsized":"2024-09-13T14:49:03.002448+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:35.162460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1128,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.903546+0000","last_change":"2024-09-13T14:43:07.518801+0000","last_active":"2024-09-13T14:48:35.903546+0000","last_peered":"2024-09-13T14:48:35.903546+0000","last_clean":"2024-09-13T14:48:35.903546+0000","last_became_active":"2024-09-13T14:43:07.517610+0000","last_became_peered":"2024-09-13T14:43:07.517610+0000","last_unstale":"2024-09-13T14:48:35.903546+0000","last_undegraded":"2024-09-13T14:48:35.903546+0000","last_fullsized":"2024-09-13T14:48:35.903546+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:51.632254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"261'100","reported_seq":92,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.910039+0000","last_change":"2024-09-13T14:43:15.037565+0000","last_active":"2024-09-13T14:48:35.910039+0000","last_peered":"2024-09-13T14:48:35.910039+0000","last_clean":"2024-09-13T14:48:35.910039+0000","last_became_active":"2024-09-13T14:43:14.554183+0000","last_became_peered":"2024-09-13T14:43:14.554183+0000","last_unstale":"2024-09-13T14:48:35.910039+0000","last_undegraded":"2024-09-13T14:48:35.910039+0000","last_fullsized":"2024-09-13T14:48:35.910039+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.037517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.037517+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:01:35.137195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":1122,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.901529+0000","last_change":"2024-09-13T14:30:15.579260+0000","last_active":"2024-09-13T14:48:35.901529+0000","last_peered":"2024-09-13T14:48:35.901529+0000","last_clean":"2024-09-13T14:48:35.901529+0000","last_became_active":"2024-09-13T14:30:15.578897+0000","last_became_peered":"2024-09-13T14:30:15.578897+0000","last_unstale":"2024-09-13T14:48:35.901529+0000","last_undegraded":"2024-09-13T14:48:35.901529+0000","last_fullsized":"2024-09-13T14:48:35.901529+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:56.737849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":92,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.922577+0000","last_change":"2024-09-13T14:43:07.521576+0000","last_active":"2024-09-13T14:48:35.922577+0000","last_peered":"2024-09-13T14:48:35.922577+0000","last_clean":"2024-09-13T14:48:35.922577+0000","last_became_active":"2024-09-13T14:43:07.520478+0000","last_became_peered":"2024-09-13T14:43:07.520478+0000","last_unstale":"2024-09-13T14:48:35.922577+0000","last_undegraded":"2024-09-13T14:48:35.922577+0000","last_fullsized":"2024-09-13T14:48:35.922577+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:53.410135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"261'100","reported_seq":382,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.903570+0000","last_change":"2024-09-13T14:44:49.775941+0000","last_active":"2024-09-13T14:48:35.903570+0000","last_peered":"2024-09-13T14:48:35.903570+0000","last_clean":"2024-09-13T14:48:35.903570+0000","last_became_active":"2024-09-13T14:43:14.560297+0000","last_became_peered":"2024-09-13T14:43:14.560297+0000","last_unstale":"2024-09-13T14:48:35.903570+0000","last_undegraded":"2024-09-13T14:48:35.903570+0000","last_fullsized":"2024-09-13T14:48:35.903570+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:49.775844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:49.775844+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:15:53.138661+0000","scrub_duration":0.012350274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":1111,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.915971+0000","last_change":"2024-09-13T14:19:25.870605+0000","last_active":"2024-09-13T14:48:35.915971+0000","last_peered":"2024-09-13T14:48:35.915971+0000","last_clean":"2024-09-13T14:48:35.915971+0000","last_became_active":"2024-09-13T14:19:25.870282+0000","last_became_peered":"2024-09-13T14:19:25.870282+0000","last_unstale":"2024-09-13T14:48:35.915971+0000","last_undegraded":"2024-09-13T14:48:35.915971+0000","last_fullsized":"2024-09-13T14:48:35.915971+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:52.514915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1125,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.916398+0000","last_change":"2024-09-13T14:43:07.526476+0000","last_active":"2024-09-13T14:48:35.916398+0000","last_peered":"2024-09-13T14:48:35.916398+0000","last_clean":"2024-09-13T14:48:35.916398+0000","last_became_active":"2024-09-13T14:43:07.525802+0000","last_became_peered":"2024-09-13T14:43:07.525802+0000","last_unstale":"2024-09-13T14:48:35.916398+0000","last_undegraded":"2024-09-13T14:48:35.916398+0000","last_fullsized":"2024-09-13T14:48:35.916398+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:23.154726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"261'100","reported_seq":99,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.912503+0000","last_change":"2024-09-13T14:45:19.807461+0000","last_active":"2024-09-13T14:48:35.912503+0000","last_peered":"2024-09-13T14:48:35.912503+0000","last_clean":"2024-09-13T14:48:35.912503+0000","last_became_active":"2024-09-13T14:45:19.807117+0000","last_became_peered":"2024-09-13T14:45:19.807117+0000","last_unstale":"2024-09-13T14:48:35.912503+0000","last_undegraded":"2024-09-13T14:48:35.912503+0000","last_fullsized":"2024-09-13T14:48:35.912503+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.955197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.955197+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:03.531555+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"1164'15206","reported_seq":15281,"reported_epoch":1164,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:48:51.347472+0000","last_change":"2024-09-13T14:45:21.161658+0000","last_active":"2024-09-13T14:48:51.347472+0000","last_peered":"2024-09-13T14:48:51.347472+0000","last_clean":"2024-09-13T14:45:19.622135+0000","last_became_active":"2024-09-13T14:45:21.090442+0000","last_became_peered":"2024-09-13T14:45:21.090442+0000","last_unstale":"2024-09-13T14:48:51.347472+0000","last_undegraded":"2024-09-13T14:48:51.347472+0000","last_fullsized":"2024-09-13T14:48:51.347472+0000","mapping_epoch":1149,"log_start":"1155'9200","ondisk_log_start":"1155'9200","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6006,"log_dups_size":0,"ondisk_log_size":6006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15206,"num_write_kb":486592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[1,2,7],"avail_no_missing":["1","2","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1088,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.915479+0000","last_change":"2024-09-13T14:19:26.131609+0000","last_active":"2024-09-13T14:48:35.915479+0000","last_peered":"2024-09-13T14:48:35.915479+0000","last_clean":"2024-09-13T14:48:35.915479+0000","last_became_active":"2024-09-13T14:19:26.131029+0000","last_became_peered":"2024-09-13T14:19:26.131029+0000","last_unstale":"2024-09-13T14:48:35.915479+0000","last_undegraded":"2024-09-13T14:48:35.915479+0000","last_fullsized":"2024-09-13T14:48:35.915479+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:57.933696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":590,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.901717+0000","last_change":"2024-09-13T14:43:32.635070+0000","last_active":"2024-09-13T14:48:35.901717+0000","last_peered":"2024-09-13T14:48:35.901717+0000","last_clean":"2024-09-13T14:48:35.901717+0000","last_became_active":"2024-09-13T14:43:32.634734+0000","last_became_peered":"2024-09-13T14:43:32.634734+0000","last_unstale":"2024-09-13T14:48:35.901717+0000","last_undegraded":"2024-09-13T14:48:35.901717+0000","last_fullsized":"2024-09-13T14:48:35.901717+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:33.107010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":1736,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:02.409309+0000","last_change":"2024-09-13T14:43:07.537759+0000","last_active":"2024-09-13T14:49:02.409309+0000","last_peered":"2024-09-13T14:49:02.409309+0000","last_clean":"2024-09-13T14:49:02.409309+0000","last_became_active":"2024-09-13T14:43:07.520303+0000","last_became_peered":"2024-09-13T14:43:07.520303+0000","last_unstale":"2024-09-13T14:49:02.409309+0000","last_undegraded":"2024-09-13T14:49:02.409309+0000","last_fullsized":"2024-09-13T14:49:02.409309+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:32.772570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"261'100","reported_seq":45,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:03.002526+0000","last_change":"2024-09-13T14:45:19.813527+0000","last_active":"2024-09-13T14:49:03.002526+0000","last_peered":"2024-09-13T14:49:03.002526+0000","last_clean":"2024-09-13T14:49:03.002526+0000","last_became_active":"2024-09-13T14:45:19.812834+0000","last_became_peered":"2024-09-13T14:45:19.812834+0000","last_unstale":"2024-09-13T14:49:03.002526+0000","last_undegraded":"2024-09-13T14:49:03.002526+0000","last_fullsized":"2024-09-13T14:49:03.002526+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.710192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.710192+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:08:55.152703+0000","scrub_duration":0.0019250910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"1164'15076","reported_seq":24152,"reported_epoch":1164,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:49:02.113112+0000","last_change":"2024-09-13T14:45:38.479094+0000","last_active":"2024-09-13T14:49:02.113112+0000","last_peered":"2024-09-13T14:49:02.113112+0000","last_clean":"2024-09-13T14:45:19.642017+0000","last_became_active":"2024-09-13T14:45:21.111093+0000","last_became_peered":"2024-09-13T14:45:21.111093+0000","last_unstale":"2024-09-13T14:49:02.113112+0000","last_undegraded":"2024-09-13T14:45:38.458706+0000","last_fullsized":"2024-09-13T14:45:20.800066+0000","mapping_epoch":1149,"log_start":"1146'6700","ondisk_log_start":"1146'6700","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":8376,"log_dups_size":0,"ondisk_log_size":8376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15076,"num_write_kb":482432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4502,"num_bytes_recovered":193789952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[2,4],"avail_no_missing":["2","4","6"],"object_location_counts":[{"shards":"2,4","objects":-2593}],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":45,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:03.002202+0000","last_change":"2024-09-13T14:45:19.813025+0000","last_active":"2024-09-13T14:49:03.002202+0000","last_peered":"2024-09-13T14:49:03.002202+0000","last_clean":"2024-09-13T14:49:03.002202+0000","last_became_active":"2024-09-13T14:45:19.812714+0000","last_became_peered":"2024-09-13T14:45:19.812714+0000","last_unstale":"2024-09-13T14:49:03.002202+0000","last_undegraded":"2024-09-13T14:49:03.002202+0000","last_fullsized":"2024-09-13T14:49:03.002202+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:40.671012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":92,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.912759+0000","last_change":"2024-09-13T14:43:07.522730+0000","last_active":"2024-09-13T14:48:35.912759+0000","last_peered":"2024-09-13T14:48:35.912759+0000","last_clean":"2024-09-13T14:48:35.912759+0000","last_became_active":"2024-09-13T14:43:07.517420+0000","last_became_peered":"2024-09-13T14:43:07.517420+0000","last_unstale":"2024-09-13T14:48:35.912759+0000","last_undegraded":"2024-09-13T14:48:35.912759+0000","last_fullsized":"2024-09-13T14:48:35.912759+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:21:30.265293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":1649,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:03.002906+0000","last_change":"2024-09-13T14:30:15.581728+0000","last_active":"2024-09-13T14:49:03.002906+0000","last_peered":"2024-09-13T14:49:03.002906+0000","last_clean":"2024-09-13T14:49:03.002906+0000","last_became_active":"2024-09-13T14:30:15.581303+0000","last_became_peered":"2024-09-13T14:30:15.581303+0000","last_unstale":"2024-09-13T14:49:03.002906+0000","last_undegraded":"2024-09-13T14:49:03.002906+0000","last_fullsized":"2024-09-13T14:49:03.002906+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:10.027991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"261'100","reported_seq":115,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.916228+0000","last_change":"2024-09-13T14:43:31.112173+0000","last_active":"2024-09-13T14:48:35.916228+0000","last_peered":"2024-09-13T14:48:35.916228+0000","last_clean":"2024-09-13T14:48:35.916228+0000","last_became_active":"2024-09-13T14:42:42.709263+0000","last_became_peered":"2024-09-13T14:42:42.709263+0000","last_unstale":"2024-09-13T14:48:35.916228+0000","last_undegraded":"2024-09-13T14:48:35.916228+0000","last_fullsized":"2024-09-13T14:48:35.916228+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:31.112111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:31.112111+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:02.707372+0000","scrub_duration":0.0015759260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"1164'15258","reported_seq":15307,"reported_epoch":1164,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:48:51.375675+0000","last_change":"2024-09-13T14:45:21.011501+0000","last_active":"2024-09-13T14:48:51.375675+0000","last_peered":"2024-09-13T14:48:51.375675+0000","last_clean":"2024-09-13T14:45:19.622974+0000","last_became_active":"2024-09-13T14:45:21.006075+0000","last_became_peered":"2024-09-13T14:45:21.006075+0000","last_unstale":"2024-09-13T14:48:51.375675+0000","last_undegraded":"2024-09-13T14:48:51.375675+0000","last_fullsized":"2024-09-13T14:48:51.375675+0000","mapping_epoch":1149,"log_start":"1150'7500","ondisk_log_start":"1150'7500","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":7758,"log_dups_size":0,"ondisk_log_size":7758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:30.090601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15258,"num_write_kb":488256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,2],"avail_no_missing":["5","1","2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1689,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:02.408883+0000","last_change":"2024-09-13T14:28:35.611508+0000","last_active":"2024-09-13T14:49:02.408883+0000","last_peered":"2024-09-13T14:49:02.408883+0000","last_clean":"2024-09-13T14:49:02.408883+0000","last_became_active":"2024-09-13T14:28:35.610106+0000","last_became_peered":"2024-09-13T14:28:35.610106+0000","last_unstale":"2024-09-13T14:49:02.408883+0000","last_undegraded":"2024-09-13T14:49:02.408883+0000","last_fullsized":"2024-09-13T14:49:02.408883+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:11.966958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1131,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.901728+0000","last_change":"2024-09-13T14:28:35.603663+0000","last_active":"2024-09-13T14:48:35.901728+0000","last_peered":"2024-09-13T14:48:35.901728+0000","last_clean":"2024-09-13T14:48:35.901728+0000","last_became_active":"2024-09-13T14:28:35.603358+0000","last_became_peered":"2024-09-13T14:28:35.603358+0000","last_unstale":"2024-09-13T14:48:35.901728+0000","last_undegraded":"2024-09-13T14:48:35.901728+0000","last_fullsized":"2024-09-13T14:48:35.901728+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:24:05.842796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":92,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.913085+0000","last_change":"2024-09-13T14:43:07.522750+0000","last_active":"2024-09-13T14:48:35.913085+0000","last_peered":"2024-09-13T14:48:35.913085+0000","last_clean":"2024-09-13T14:48:35.913085+0000","last_became_active":"2024-09-13T14:43:07.517267+0000","last_became_peered":"2024-09-13T14:43:07.517267+0000","last_unstale":"2024-09-13T14:48:35.913085+0000","last_undegraded":"2024-09-13T14:48:35.913085+0000","last_fullsized":"2024-09-13T14:48:35.913085+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:40.217748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"261'100","reported_seq":415,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:02.409113+0000","last_change":"2024-09-13T14:45:19.815050+0000","last_active":"2024-09-13T14:49:02.409113+0000","last_peered":"2024-09-13T14:49:02.409113+0000","last_clean":"2024-09-13T14:49:02.409113+0000","last_became_active":"2024-09-13T14:45:19.809732+0000","last_became_peered":"2024-09-13T14:45:19.809732+0000","last_unstale":"2024-09-13T14:49:02.409113+0000","last_undegraded":"2024-09-13T14:49:02.409113+0000","last_fullsized":"2024-09-13T14:49:02.409113+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:33.574443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:33.574443+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:51:40.261036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"1164'15108","reported_seq":15215,"reported_epoch":1164,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:48:51.380952+0000","last_change":"2024-09-13T14:45:21.136109+0000","last_active":"2024-09-13T14:48:51.380952+0000","last_peered":"2024-09-13T14:48:51.380952+0000","last_clean":"2024-09-13T14:45:19.607954+0000","last_became_active":"2024-09-13T14:45:21.004300+0000","last_became_peered":"2024-09-13T14:45:21.004300+0000","last_unstale":"2024-09-13T14:48:51.380952+0000","last_undegraded":"2024-09-13T14:48:51.380952+0000","last_fullsized":"2024-09-13T14:48:51.380952+0000","mapping_epoch":1149,"log_start":"1155'8900","ondisk_log_start":"1155'8900","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6208,"log_dups_size":0,"ondisk_log_size":6208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:04:23.351582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15108,"num_write_kb":483456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":1835008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,6,1],"avail_no_missing":["4","1","3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":92,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.922444+0000","last_change":"2024-09-13T14:43:07.520049+0000","last_active":"2024-09-13T14:48:35.922444+0000","last_peered":"2024-09-13T14:48:35.922444+0000","last_clean":"2024-09-13T14:48:35.922444+0000","last_became_active":"2024-09-13T14:43:07.514646+0000","last_became_peered":"2024-09-13T14:43:07.514646+0000","last_unstale":"2024-09-13T14:48:35.922444+0000","last_undegraded":"2024-09-13T14:48:35.922444+0000","last_fullsized":"2024-09-13T14:48:35.922444+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:10:35.200778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":92,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.912992+0000","last_change":"2024-09-13T14:43:07.525666+0000","last_active":"2024-09-13T14:48:35.912992+0000","last_peered":"2024-09-13T14:48:35.912992+0000","last_clean":"2024-09-13T14:48:35.912992+0000","last_became_active":"2024-09-13T14:43:07.524064+0000","last_became_peered":"2024-09-13T14:43:07.524064+0000","last_unstale":"2024-09-13T14:48:35.912992+0000","last_undegraded":"2024-09-13T14:48:35.912992+0000","last_fullsized":"2024-09-13T14:48:35.912992+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:10.270333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1145,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.903482+0000","last_change":"2024-09-13T14:45:19.804177+0000","last_active":"2024-09-13T14:48:35.903482+0000","last_peered":"2024-09-13T14:48:35.903482+0000","last_clean":"2024-09-13T14:48:35.903482+0000","last_became_active":"2024-09-13T14:45:19.803890+0000","last_became_peered":"2024-09-13T14:45:19.803890+0000","last_unstale":"2024-09-13T14:48:35.903482+0000","last_undegraded":"2024-09-13T14:48:35.903482+0000","last_fullsized":"2024-09-13T14:48:35.903482+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:25.140789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"261'100","reported_seq":62,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:03.002824+0000","last_change":"2024-09-13T14:45:36.965992+0000","last_active":"2024-09-13T14:49:03.002824+0000","last_peered":"2024-09-13T14:49:03.002824+0000","last_clean":"2024-09-13T14:49:03.002824+0000","last_became_active":"2024-09-13T14:45:19.811238+0000","last_became_peered":"2024-09-13T14:45:19.811238+0000","last_unstale":"2024-09-13T14:49:03.002824+0000","last_undegraded":"2024-09-13T14:49:03.002824+0000","last_fullsized":"2024-09-13T14:49:03.002824+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:36.965775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:45:36.965775+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:36:01.623251+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"1164'14798","reported_seq":14872,"reported_epoch":1165,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:49:02.409062+0000","last_change":"2024-09-13T14:45:21.036181+0000","last_active":"2024-09-13T14:49:02.409062+0000","last_peered":"2024-09-13T14:49:02.409062+0000","last_clean":"2024-09-13T14:45:19.587756+0000","last_became_active":"2024-09-13T14:45:21.029769+0000","last_became_peered":"2024-09-13T14:45:21.029769+0000","last_unstale":"2024-09-13T14:49:02.409062+0000","last_undegraded":"2024-09-13T14:49:02.409062+0000","last_fullsized":"2024-09-13T14:49:02.409062+0000","mapping_epoch":1149,"log_start":"1152'7900","ondisk_log_start":"1152'7900","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6898,"log_dups_size":0,"ondisk_log_size":6898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14798,"num_write_kb":473536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,1],"avail_no_missing":["0","1","3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":44,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.903042+0000","last_change":"2024-09-13T14:45:19.806318+0000","last_active":"2024-09-13T14:48:35.903042+0000","last_peered":"2024-09-13T14:48:35.903042+0000","last_clean":"2024-09-13T14:48:35.903042+0000","last_became_active":"2024-09-13T14:45:19.806074+0000","last_became_peered":"2024-09-13T14:45:19.806074+0000","last_unstale":"2024-09-13T14:48:35.903042+0000","last_undegraded":"2024-09-13T14:48:35.903042+0000","last_fullsized":"2024-09-13T14:48:35.903042+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:21.679797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1106,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.916506+0000","last_change":"2024-09-13T14:28:35.604404+0000","last_active":"2024-09-13T14:48:35.916506+0000","last_peered":"2024-09-13T14:48:35.916506+0000","last_clean":"2024-09-13T14:48:35.916506+0000","last_became_active":"2024-09-13T14:28:35.604102+0000","last_became_peered":"2024-09-13T14:28:35.604102+0000","last_unstale":"2024-09-13T14:48:35.916506+0000","last_undegraded":"2024-09-13T14:48:35.916506+0000","last_fullsized":"2024-09-13T14:48:35.916506+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:25.546010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":1156,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.903070+0000","last_change":"2024-09-13T14:43:07.538919+0000","last_active":"2024-09-13T14:48:35.903070+0000","last_peered":"2024-09-13T14:48:35.903070+0000","last_clean":"2024-09-13T14:48:35.903070+0000","last_became_active":"2024-09-13T14:43:07.513109+0000","last_became_peered":"2024-09-13T14:43:07.513109+0000","last_unstale":"2024-09-13T14:48:35.903070+0000","last_undegraded":"2024-09-13T14:48:35.903070+0000","last_fullsized":"2024-09-13T14:48:35.903070+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:27.771720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"261'100","reported_seq":105,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.903209+0000","last_change":"2024-09-13T14:43:44.720333+0000","last_active":"2024-09-13T14:48:35.903209+0000","last_peered":"2024-09-13T14:48:35.903209+0000","last_clean":"2024-09-13T14:48:35.903209+0000","last_became_active":"2024-09-13T14:42:14.946591+0000","last_became_peered":"2024-09-13T14:42:14.946591+0000","last_unstale":"2024-09-13T14:48:35.903209+0000","last_undegraded":"2024-09-13T14:48:35.903209+0000","last_fullsized":"2024-09-13T14:48:35.903209+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:44.720258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:44.720258+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:33.281357+0000","scrub_duration":0.0018417920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"1164'15090","reported_seq":15161,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:02.408783+0000","last_change":"2024-09-13T14:43:50.794091+0000","last_active":"2024-09-13T14:49:02.408783+0000","last_peered":"2024-09-13T14:49:02.408783+0000","last_clean":"2024-09-13T14:49:02.408783+0000","last_became_active":"2024-09-13T14:43:50.793741+0000","last_became_peered":"2024-09-13T14:43:50.793741+0000","last_unstale":"2024-09-13T14:49:02.408783+0000","last_undegraded":"2024-09-13T14:49:02.408783+0000","last_fullsized":"2024-09-13T14:49:02.408783+0000","mapping_epoch":1136,"log_start":"1153'8200","ondisk_log_start":"1153'8200","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6890,"log_dups_size":0,"ondisk_log_size":6890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:15.846494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15090,"num_write_kb":482880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":60,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.909060+0000","last_change":"2024-09-13T14:44:03.858211+0000","last_active":"2024-09-13T14:48:35.909060+0000","last_peered":"2024-09-13T14:48:35.909060+0000","last_clean":"2024-09-13T14:48:35.909060+0000","last_became_active":"2024-09-13T14:44:03.857928+0000","last_became_peered":"2024-09-13T14:44:03.857928+0000","last_unstale":"2024-09-13T14:48:35.909060+0000","last_undegraded":"2024-09-13T14:48:35.909060+0000","last_fullsized":"2024-09-13T14:48:35.909060+0000","mapping_epoch":1140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:28.428693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":92,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.922031+0000","last_change":"2024-09-13T14:43:07.516243+0000","last_active":"2024-09-13T14:48:35.922031+0000","last_peered":"2024-09-13T14:48:35.922031+0000","last_clean":"2024-09-13T14:48:35.922031+0000","last_became_active":"2024-09-13T14:43:07.515863+0000","last_became_peered":"2024-09-13T14:43:07.515863+0000","last_unstale":"2024-09-13T14:48:35.922031+0000","last_undegraded":"2024-09-13T14:48:35.922031+0000","last_fullsized":"2024-09-13T14:48:35.922031+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:08.085298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1729,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.908560+0000","last_change":"2024-09-13T14:45:19.812713+0000","last_active":"2024-09-13T14:48:35.908560+0000","last_peered":"2024-09-13T14:48:35.908560+0000","last_clean":"2024-09-13T14:48:35.908560+0000","last_became_active":"2024-09-13T14:45:19.812203+0000","last_became_peered":"2024-09-13T14:45:19.812203+0000","last_unstale":"2024-09-13T14:48:35.908560+0000","last_undegraded":"2024-09-13T14:48:35.908560+0000","last_fullsized":"2024-09-13T14:48:35.908560+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:30:46.149347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"261'100","reported_seq":1285,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.901255+0000","last_change":"2024-09-13T14:43:42.577312+0000","last_active":"2024-09-13T14:48:35.901255+0000","last_peered":"2024-09-13T14:48:35.901255+0000","last_clean":"2024-09-13T14:48:35.901255+0000","last_became_active":"2024-09-13T14:42:14.928923+0000","last_became_peered":"2024-09-13T14:42:14.928923+0000","last_unstale":"2024-09-13T14:48:35.901255+0000","last_undegraded":"2024-09-13T14:48:35.901255+0000","last_fullsized":"2024-09-13T14:48:35.901255+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:42.577243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:42.577243+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:01.234250+0000","scrub_duration":0.001367839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"1164'15028","reported_seq":15065,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:51.381684+0000","last_change":"2024-09-13T14:43:50.793810+0000","last_active":"2024-09-13T14:48:51.381684+0000","last_peered":"2024-09-13T14:48:51.381684+0000","last_clean":"2024-09-13T14:48:51.381684+0000","last_became_active":"2024-09-13T14:43:50.793506+0000","last_became_peered":"2024-09-13T14:43:50.793506+0000","last_unstale":"2024-09-13T14:48:51.381684+0000","last_undegraded":"2024-09-13T14:48:51.381684+0000","last_fullsized":"2024-09-13T14:48:51.381684+0000","mapping_epoch":1136,"log_start":"1155'8900","ondisk_log_start":"1155'8900","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6128,"log_dups_size":0,"ondisk_log_size":6128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:50:32.061762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15028,"num_write_kb":480896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1725,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:02.409220+0000","last_change":"2024-09-13T14:43:07.520212+0000","last_active":"2024-09-13T14:49:02.409220+0000","last_peered":"2024-09-13T14:49:02.409220+0000","last_clean":"2024-09-13T14:49:02.409220+0000","last_became_active":"2024-09-13T14:43:07.519831+0000","last_became_peered":"2024-09-13T14:43:07.519831+0000","last_unstale":"2024-09-13T14:49:02.409220+0000","last_undegraded":"2024-09-13T14:49:02.409220+0000","last_fullsized":"2024-09-13T14:49:02.409220+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:20.990736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1747,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:02.408674+0000","last_change":"2024-09-13T14:45:19.809505+0000","last_active":"2024-09-13T14:49:02.408674+0000","last_peered":"2024-09-13T14:49:02.408674+0000","last_clean":"2024-09-13T14:49:02.408674+0000","last_became_active":"2024-09-13T14:45:19.809181+0000","last_became_peered":"2024-09-13T14:45:19.809181+0000","last_unstale":"2024-09-13T14:49:02.408674+0000","last_undegraded":"2024-09-13T14:49:02.408674+0000","last_fullsized":"2024-09-13T14:49:02.408674+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:24.239188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1168,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.901259+0000","last_change":"2024-09-13T14:43:07.516140+0000","last_active":"2024-09-13T14:48:35.901259+0000","last_peered":"2024-09-13T14:48:35.901259+0000","last_clean":"2024-09-13T14:48:35.901259+0000","last_became_active":"2024-09-13T14:43:07.515481+0000","last_became_peered":"2024-09-13T14:43:07.515481+0000","last_unstale":"2024-09-13T14:48:35.901259+0000","last_undegraded":"2024-09-13T14:48:35.901259+0000","last_fullsized":"2024-09-13T14:48:35.901259+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:43.706214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"261'100","reported_seq":104,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.914562+0000","last_change":"2024-09-13T14:43:16.008346+0000","last_active":"2024-09-13T14:48:35.914562+0000","last_peered":"2024-09-13T14:48:35.914562+0000","last_clean":"2024-09-13T14:48:35.914562+0000","last_became_active":"2024-09-13T14:43:07.523121+0000","last_became_peered":"2024-09-13T14:43:07.523121+0000","last_unstale":"2024-09-13T14:48:35.914562+0000","last_undegraded":"2024-09-13T14:48:35.914562+0000","last_fullsized":"2024-09-13T14:48:35.914562+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.008264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.008264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:58:35.564470+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"1164'15038","reported_seq":15113,"reported_epoch":1165,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:49:02.408994+0000","last_change":"2024-09-13T14:45:21.035848+0000","last_active":"2024-09-13T14:49:02.408994+0000","last_peered":"2024-09-13T14:49:02.408994+0000","last_clean":"2024-09-13T14:45:19.637667+0000","last_became_active":"2024-09-13T14:45:21.025308+0000","last_became_peered":"2024-09-13T14:45:21.025308+0000","last_unstale":"2024-09-13T14:49:02.408994+0000","last_undegraded":"2024-09-13T14:49:02.408994+0000","last_fullsized":"2024-09-13T14:49:02.408994+0000","mapping_epoch":1149,"log_start":"1153'8200","ondisk_log_start":"1153'8200","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6838,"log_dups_size":0,"ondisk_log_size":6838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15038,"num_write_kb":481216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,5],"avail_no_missing":["0","3","5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":47,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.916125+0000","last_change":"2024-09-13T14:43:32.636990+0000","last_active":"2024-09-13T14:48:35.916125+0000","last_peered":"2024-09-13T14:48:35.916125+0000","last_clean":"2024-09-13T14:48:35.916125+0000","last_became_active":"2024-09-13T14:43:32.636589+0000","last_became_peered":"2024-09-13T14:43:32.636589+0000","last_unstale":"2024-09-13T14:48:35.916125+0000","last_undegraded":"2024-09-13T14:48:35.916125+0000","last_fullsized":"2024-09-13T14:48:35.916125+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:36:25.129769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":45,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:03.002248+0000","last_change":"2024-09-13T14:45:19.810888+0000","last_active":"2024-09-13T14:49:03.002248+0000","last_peered":"2024-09-13T14:49:03.002248+0000","last_clean":"2024-09-13T14:49:03.002248+0000","last_became_active":"2024-09-13T14:45:19.810486+0000","last_became_peered":"2024-09-13T14:45:19.810486+0000","last_unstale":"2024-09-13T14:49:03.002248+0000","last_undegraded":"2024-09-13T14:49:03.002248+0000","last_fullsized":"2024-09-13T14:49:03.002248+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:03.886942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":1121,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.902967+0000","last_change":"2024-09-13T14:29:42.539455+0000","last_active":"2024-09-13T14:48:35.902967+0000","last_peered":"2024-09-13T14:48:35.902967+0000","last_clean":"2024-09-13T14:48:35.902967+0000","last_became_active":"2024-09-13T14:29:42.539068+0000","last_became_peered":"2024-09-13T14:29:42.539068+0000","last_unstale":"2024-09-13T14:48:35.902967+0000","last_undegraded":"2024-09-13T14:48:35.902967+0000","last_fullsized":"2024-09-13T14:48:35.902967+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:24:06.048855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034405999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"261'100","reported_seq":1276,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.901279+0000","last_change":"2024-09-13T14:44:28.186029+0000","last_active":"2024-09-13T14:48:35.901279+0000","last_peered":"2024-09-13T14:48:35.901279+0000","last_clean":"2024-09-13T14:48:35.901279+0000","last_became_active":"2024-09-13T14:42:14.927771+0000","last_became_peered":"2024-09-13T14:42:14.927771+0000","last_unstale":"2024-09-13T14:48:35.901279+0000","last_undegraded":"2024-09-13T14:48:35.901279+0000","last_fullsized":"2024-09-13T14:48:35.901279+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:28.185949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:28.185949+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:38.363961+0000","scrub_duration":0.0018071129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"1164'14944","reported_seq":15002,"reported_epoch":1164,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:48:51.370439+0000","last_change":"2024-09-13T14:45:21.030922+0000","last_active":"2024-09-13T14:48:51.370439+0000","last_peered":"2024-09-13T14:48:51.370439+0000","last_clean":"2024-09-13T14:45:19.630214+0000","last_became_active":"2024-09-13T14:45:21.021807+0000","last_became_peered":"2024-09-13T14:45:21.021807+0000","last_unstale":"2024-09-13T14:48:51.370439+0000","last_undegraded":"2024-09-13T14:48:51.370439+0000","last_fullsized":"2024-09-13T14:48:51.370439+0000","mapping_epoch":1149,"log_start":"1146'6600","ondisk_log_start":"1146'6600","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":8344,"log_dups_size":0,"ondisk_log_size":8344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14944,"num_write_kb":478208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[2,6,7],"avail_no_missing":["2","3","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":92,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.913555+0000","last_change":"2024-09-13T14:43:07.520176+0000","last_active":"2024-09-13T14:48:35.913555+0000","last_peered":"2024-09-13T14:48:35.913555+0000","last_clean":"2024-09-13T14:48:35.913555+0000","last_became_active":"2024-09-13T14:43:07.516748+0000","last_became_peered":"2024-09-13T14:43:07.516748+0000","last_unstale":"2024-09-13T14:48:35.913555+0000","last_undegraded":"2024-09-13T14:48:35.913555+0000","last_fullsized":"2024-09-13T14:48:35.913555+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:52:04.392821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":45,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:03.002248+0000","last_change":"2024-09-13T14:45:19.812353+0000","last_active":"2024-09-13T14:49:03.002248+0000","last_peered":"2024-09-13T14:49:03.002248+0000","last_clean":"2024-09-13T14:49:03.002248+0000","last_became_active":"2024-09-13T14:45:19.811743+0000","last_became_peered":"2024-09-13T14:45:19.811743+0000","last_unstale":"2024-09-13T14:49:03.002248+0000","last_undegraded":"2024-09-13T14:49:03.002248+0000","last_fullsized":"2024-09-13T14:49:03.002248+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:43.728170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"261'100","reported_seq":105,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.902696+0000","last_change":"2024-09-13T14:44:34.814157+0000","last_active":"2024-09-13T14:48:35.902696+0000","last_peered":"2024-09-13T14:48:35.902696+0000","last_clean":"2024-09-13T14:48:35.902696+0000","last_became_active":"2024-09-13T14:42:42.707803+0000","last_became_peered":"2024-09-13T14:42:42.707803+0000","last_unstale":"2024-09-13T14:48:35.902696+0000","last_undegraded":"2024-09-13T14:48:35.902696+0000","last_fullsized":"2024-09-13T14:48:35.902696+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:34.814070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:34.814070+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:38.408561+0000","scrub_duration":0.01686414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"1164'15124","reported_seq":15194,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:51.354232+0000","last_change":"2024-09-13T14:43:51.534705+0000","last_active":"2024-09-13T14:48:51.354232+0000","last_peered":"2024-09-13T14:48:51.354232+0000","last_clean":"2024-09-13T14:48:51.354232+0000","last_became_active":"2024-09-13T14:43:51.534381+0000","last_became_peered":"2024-09-13T14:43:51.534381+0000","last_unstale":"2024-09-13T14:48:51.354232+0000","last_undegraded":"2024-09-13T14:48:51.354232+0000","last_fullsized":"2024-09-13T14:48:51.354232+0000","mapping_epoch":1136,"log_start":"1155'9100","ondisk_log_start":"1155'9100","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6024,"log_dups_size":0,"ondisk_log_size":6024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:37:22.385742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15124,"num_write_kb":483905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1135,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.916005+0000","last_change":"2024-09-13T14:44:03.855577+0000","last_active":"2024-09-13T14:48:35.916005+0000","last_peered":"2024-09-13T14:48:35.916005+0000","last_clean":"2024-09-13T14:48:35.916005+0000","last_became_active":"2024-09-13T14:44:03.855267+0000","last_became_peered":"2024-09-13T14:44:03.855267+0000","last_unstale":"2024-09-13T14:48:35.916005+0000","last_undegraded":"2024-09-13T14:48:35.916005+0000","last_fullsized":"2024-09-13T14:48:35.916005+0000","mapping_epoch":1140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:46.148814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1106,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.915654+0000","last_change":"2024-09-13T14:29:42.545815+0000","last_active":"2024-09-13T14:48:35.915654+0000","last_peered":"2024-09-13T14:48:35.915654+0000","last_clean":"2024-09-13T14:48:35.915654+0000","last_became_active":"2024-09-13T14:29:42.545508+0000","last_became_peered":"2024-09-13T14:29:42.545508+0000","last_unstale":"2024-09-13T14:48:35.915654+0000","last_undegraded":"2024-09-13T14:48:35.915654+0000","last_fullsized":"2024-09-13T14:48:35.915654+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:39.678381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"261'100","reported_seq":110,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.913033+0000","last_change":"2024-09-13T14:43:18.988482+0000","last_active":"2024-09-13T14:48:35.913033+0000","last_peered":"2024-09-13T14:48:35.913033+0000","last_clean":"2024-09-13T14:48:35.913033+0000","last_became_active":"2024-09-13T14:43:14.556754+0000","last_became_peered":"2024-09-13T14:43:14.556754+0000","last_unstale":"2024-09-13T14:48:35.913033+0000","last_undegraded":"2024-09-13T14:48:35.913033+0000","last_fullsized":"2024-09-13T14:48:35.913033+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:18.988405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:18.988405+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:45:01.726326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"1164'14772","reported_seq":14841,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:51.272277+0000","last_change":"2024-09-13T14:43:50.796182+0000","last_active":"2024-09-13T14:48:51.272277+0000","last_peered":"2024-09-13T14:48:51.272277+0000","last_clean":"2024-09-13T14:48:51.272277+0000","last_became_active":"2024-09-13T14:43:50.795875+0000","last_became_peered":"2024-09-13T14:43:50.795875+0000","last_unstale":"2024-09-13T14:48:51.272277+0000","last_undegraded":"2024-09-13T14:48:51.272277+0000","last_fullsized":"2024-09-13T14:48:51.272277+0000","mapping_epoch":1136,"log_start":"1146'6400","ondisk_log_start":"1146'6400","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":8372,"log_dups_size":0,"ondisk_log_size":8372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:56:34.308870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14772,"num_write_kb":472704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1152,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.916310+0000","last_change":"2024-09-13T14:43:07.515418+0000","last_active":"2024-09-13T14:48:35.916310+0000","last_peered":"2024-09-13T14:48:35.916310+0000","last_clean":"2024-09-13T14:48:35.916310+0000","last_became_active":"2024-09-13T14:43:07.515113+0000","last_became_peered":"2024-09-13T14:43:07.515113+0000","last_unstale":"2024-09-13T14:48:35.916310+0000","last_undegraded":"2024-09-13T14:48:35.916310+0000","last_fullsized":"2024-09-13T14:48:35.916310+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:10:25.810604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1119,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.915524+0000","last_change":"2024-09-13T14:43:32.636334+0000","last_active":"2024-09-13T14:48:35.915524+0000","last_peered":"2024-09-13T14:48:35.915524+0000","last_clean":"2024-09-13T14:48:35.915524+0000","last_became_active":"2024-09-13T14:43:32.635736+0000","last_became_peered":"2024-09-13T14:43:32.635736+0000","last_unstale":"2024-09-13T14:48:35.915524+0000","last_undegraded":"2024-09-13T14:48:35.915524+0000","last_fullsized":"2024-09-13T14:48:35.915524+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:49.076014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"261'100","reported_seq":110,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.902587+0000","last_change":"2024-09-13T14:45:19.805634+0000","last_active":"2024-09-13T14:48:35.902587+0000","last_peered":"2024-09-13T14:48:35.902587+0000","last_clean":"2024-09-13T14:48:35.902587+0000","last_became_active":"2024-09-13T14:45:19.805046+0000","last_became_peered":"2024-09-13T14:45:19.805046+0000","last_unstale":"2024-09-13T14:48:35.902587+0000","last_undegraded":"2024-09-13T14:48:35.902587+0000","last_fullsized":"2024-09-13T14:48:35.902587+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:32.752408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:32.752408+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:06:53.140239+0000","scrub_duration":0.0057731559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"1164'15176","reported_seq":15252,"reported_epoch":1165,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:49:02.409051+0000","last_change":"2024-09-13T14:45:21.029184+0000","last_active":"2024-09-13T14:49:02.409051+0000","last_peered":"2024-09-13T14:49:02.409051+0000","last_clean":"2024-09-13T14:45:19.633080+0000","last_became_active":"2024-09-13T14:45:21.019809+0000","last_became_peered":"2024-09-13T14:45:21.019809+0000","last_unstale":"2024-09-13T14:49:02.409051+0000","last_undegraded":"2024-09-13T14:49:02.409051+0000","last_fullsized":"2024-09-13T14:49:02.409051+0000","mapping_epoch":1149,"log_start":"1153'8300","ondisk_log_start":"1153'8300","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6876,"log_dups_size":0,"ondisk_log_size":6876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15176,"num_write_kb":485632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,7,4],"avail_no_missing":["0","3","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1155,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.915920+0000","last_change":"2024-09-13T14:45:19.804441+0000","last_active":"2024-09-13T14:48:35.915920+0000","last_peered":"2024-09-13T14:48:35.915920+0000","last_clean":"2024-09-13T14:48:35.915920+0000","last_became_active":"2024-09-13T14:45:19.804039+0000","last_became_peered":"2024-09-13T14:45:19.804039+0000","last_unstale":"2024-09-13T14:48:35.915920+0000","last_undegraded":"2024-09-13T14:48:35.915920+0000","last_fullsized":"2024-09-13T14:48:35.915920+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:58:00.322042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":49,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:03.002287+0000","last_change":"2024-09-13T14:45:19.816666+0000","last_active":"2024-09-13T14:49:03.002287+0000","last_peered":"2024-09-13T14:49:03.002287+0000","last_clean":"2024-09-13T14:49:03.002287+0000","last_became_active":"2024-09-13T14:45:19.807224+0000","last_became_peered":"2024-09-13T14:45:19.807224+0000","last_unstale":"2024-09-13T14:49:03.002287+0000","last_undegraded":"2024-09-13T14:49:03.002287+0000","last_fullsized":"2024-09-13T14:49:03.002287+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:15:20.171682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"261'100","reported_seq":1295,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.901028+0000","last_change":"2024-09-13T14:45:19.798553+0000","last_active":"2024-09-13T14:48:35.901028+0000","last_peered":"2024-09-13T14:48:35.901028+0000","last_clean":"2024-09-13T14:48:35.901028+0000","last_became_active":"2024-09-13T14:45:19.798189+0000","last_became_peered":"2024-09-13T14:45:19.798189+0000","last_unstale":"2024-09-13T14:48:35.901028+0000","last_undegraded":"2024-09-13T14:48:35.901028+0000","last_fullsized":"2024-09-13T14:48:35.901028+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:29.169325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:29.169325+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:10.412666+0000","scrub_duration":0.0019358890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"1164'15108","reported_seq":15177,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:51.386802+0000","last_change":"2024-09-13T14:43:50.795029+0000","last_active":"2024-09-13T14:48:51.386802+0000","last_peered":"2024-09-13T14:48:51.386802+0000","last_clean":"2024-09-13T14:48:51.386802+0000","last_became_active":"2024-09-13T14:43:50.794715+0000","last_became_peered":"2024-09-13T14:43:50.794715+0000","last_unstale":"2024-09-13T14:48:51.386802+0000","last_undegraded":"2024-09-13T14:48:51.386802+0000","last_fullsized":"2024-09-13T14:48:51.386802+0000","mapping_epoch":1136,"log_start":"1146'6700","ondisk_log_start":"1146'6700","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":8408,"log_dups_size":0,"ondisk_log_size":8408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:21.257279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15108,"num_write_kb":483456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1714,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:02.408859+0000","last_change":"2024-09-13T14:43:07.515572+0000","last_active":"2024-09-13T14:49:02.408859+0000","last_peered":"2024-09-13T14:49:02.408859+0000","last_clean":"2024-09-13T14:49:02.408859+0000","last_became_active":"2024-09-13T14:43:07.515231+0000","last_became_peered":"2024-09-13T14:43:07.515231+0000","last_unstale":"2024-09-13T14:49:02.408859+0000","last_undegraded":"2024-09-13T14:49:02.408859+0000","last_fullsized":"2024-09-13T14:49:02.408859+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:57.966380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1148,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.903289+0000","last_change":"2024-09-13T14:45:19.804982+0000","last_active":"2024-09-13T14:48:35.903289+0000","last_peered":"2024-09-13T14:48:35.903289+0000","last_clean":"2024-09-13T14:48:35.903289+0000","last_became_active":"2024-09-13T14:45:19.804663+0000","last_became_peered":"2024-09-13T14:45:19.804663+0000","last_unstale":"2024-09-13T14:48:35.903289+0000","last_undegraded":"2024-09-13T14:48:35.903289+0000","last_fullsized":"2024-09-13T14:48:35.903289+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:43.211479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"261'100","reported_seq":385,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.915857+0000","last_change":"2024-09-13T14:43:42.881584+0000","last_active":"2024-09-13T14:48:35.915857+0000","last_peered":"2024-09-13T14:48:35.915857+0000","last_clean":"2024-09-13T14:48:35.915857+0000","last_became_active":"2024-09-13T14:43:14.562161+0000","last_became_peered":"2024-09-13T14:43:14.562161+0000","last_unstale":"2024-09-13T14:48:35.915857+0000","last_undegraded":"2024-09-13T14:48:35.915857+0000","last_fullsized":"2024-09-13T14:48:35.915857+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:42.881520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:42.881520+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:06:50.353442+0000","scrub_duration":0.0013804310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"1164'15210","reported_seq":15281,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:02.408571+0000","last_change":"2024-09-13T14:43:50.793879+0000","last_active":"2024-09-13T14:49:02.408571+0000","last_peered":"2024-09-13T14:49:02.408571+0000","last_clean":"2024-09-13T14:49:02.408571+0000","last_became_active":"2024-09-13T14:43:50.793571+0000","last_became_peered":"2024-09-13T14:43:50.793571+0000","last_unstale":"2024-09-13T14:49:02.408571+0000","last_undegraded":"2024-09-13T14:49:02.408571+0000","last_fullsized":"2024-09-13T14:49:02.408571+0000","mapping_epoch":1136,"log_start":"1153'8300","ondisk_log_start":"1153'8300","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6910,"log_dups_size":0,"ondisk_log_size":6910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:22.880360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15210,"num_write_kb":486720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1120,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.828256+0000","last_change":"2024-09-13T14:43:07.518120+0000","last_active":"2024-09-13T14:48:18.828256+0000","last_peered":"2024-09-13T14:48:18.828256+0000","last_clean":"2024-09-13T14:48:18.828256+0000","last_became_active":"2024-09-13T14:43:07.517361+0000","last_became_peered":"2024-09-13T14:43:07.517361+0000","last_unstale":"2024-09-13T14:48:18.828256+0000","last_undegraded":"2024-09-13T14:48:18.828256+0000","last_fullsized":"2024-09-13T14:48:18.828256+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:39.366473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":1103,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.901646+0000","last_change":"2024-09-13T14:19:25.871006+0000","last_active":"2024-09-13T14:48:35.901646+0000","last_peered":"2024-09-13T14:48:35.901646+0000","last_clean":"2024-09-13T14:48:35.901646+0000","last_became_active":"2024-09-13T14:19:25.870713+0000","last_became_peered":"2024-09-13T14:19:25.870713+0000","last_unstale":"2024-09-13T14:48:35.901646+0000","last_undegraded":"2024-09-13T14:48:35.901646+0000","last_fullsized":"2024-09-13T14:48:35.901646+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:34.372124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"261'100","reported_seq":369,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.903612+0000","last_change":"2024-09-13T14:43:32.637553+0000","last_active":"2024-09-13T14:48:35.903612+0000","last_peered":"2024-09-13T14:48:35.903612+0000","last_clean":"2024-09-13T14:48:35.903612+0000","last_became_active":"2024-09-13T14:43:32.636545+0000","last_became_peered":"2024-09-13T14:43:32.636545+0000","last_unstale":"2024-09-13T14:48:35.903612+0000","last_undegraded":"2024-09-13T14:48:35.903612+0000","last_fullsized":"2024-09-13T14:48:35.903612+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:15:42.069756+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"1164'15016","reported_seq":15085,"reported_epoch":1164,"state":"active+recovery_wait+remapped","last_fresh":"2024-09-13T14:48:51.414323+0000","last_change":"2024-09-13T14:45:21.130410+0000","last_active":"2024-09-13T14:48:51.414323+0000","last_peered":"2024-09-13T14:48:51.414323+0000","last_clean":"2024-09-13T14:45:19.587556+0000","last_became_active":"2024-09-13T14:45:21.100167+0000","last_became_peered":"2024-09-13T14:45:21.100167+0000","last_unstale":"2024-09-13T14:48:51.414323+0000","last_undegraded":"2024-09-13T14:48:51.414323+0000","last_fullsized":"2024-09-13T14:48:51.414323+0000","mapping_epoch":1149,"log_start":"1155'8977","ondisk_log_start":"1155'8977","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6039,"log_dups_size":0,"ondisk_log_size":6039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7077,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15016,"num_write_kb":480512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[1,2,5],"avail_no_missing":["1","2","5"],"object_location_counts":[{"shards":"1,2,5","objects":-7077}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1172,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.915587+0000","last_change":"2024-09-13T14:45:19.804868+0000","last_active":"2024-09-13T14:48:35.915587+0000","last_peered":"2024-09-13T14:48:35.915587+0000","last_clean":"2024-09-13T14:48:35.915587+0000","last_became_active":"2024-09-13T14:45:19.804520+0000","last_became_peered":"2024-09-13T14:45:19.804520+0000","last_unstale":"2024-09-13T14:48:35.915587+0000","last_undegraded":"2024-09-13T14:48:35.915587+0000","last_fullsized":"2024-09-13T14:48:35.915587+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:17:53.924629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":1149,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.901752+0000","last_change":"2024-09-13T14:43:07.533955+0000","last_active":"2024-09-13T14:48:35.901752+0000","last_peered":"2024-09-13T14:48:35.901752+0000","last_clean":"2024-09-13T14:48:35.901752+0000","last_became_active":"2024-09-13T14:43:07.516816+0000","last_became_peered":"2024-09-13T14:43:07.516816+0000","last_unstale":"2024-09-13T14:48:35.901752+0000","last_undegraded":"2024-09-13T14:48:35.901752+0000","last_fullsized":"2024-09-13T14:48:35.901752+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:26.232184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"261'100","reported_seq":49,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:03.002408+0000","last_change":"2024-09-13T14:45:19.821649+0000","last_active":"2024-09-13T14:49:03.002408+0000","last_peered":"2024-09-13T14:49:03.002408+0000","last_clean":"2024-09-13T14:49:03.002408+0000","last_became_active":"2024-09-13T14:45:19.812346+0000","last_became_peered":"2024-09-13T14:45:19.812346+0000","last_unstale":"2024-09-13T14:49:03.002408+0000","last_undegraded":"2024-09-13T14:49:03.002408+0000","last_fullsized":"2024-09-13T14:49:03.002408+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.597568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.597568+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:45.218892+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"1164'14966","reported_seq":15035,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:51.279250+0000","last_change":"2024-09-13T14:43:51.537117+0000","last_active":"2024-09-13T14:48:51.279250+0000","last_peered":"2024-09-13T14:48:51.279250+0000","last_clean":"2024-09-13T14:48:51.279250+0000","last_became_active":"2024-09-13T14:43:51.536532+0000","last_became_peered":"2024-09-13T14:43:51.536532+0000","last_unstale":"2024-09-13T14:48:51.279250+0000","last_undegraded":"2024-09-13T14:48:51.279250+0000","last_fullsized":"2024-09-13T14:48:51.279250+0000","mapping_epoch":1136,"log_start":"1155'8900","ondisk_log_start":"1155'8900","created":1136,"last_epoch_clean":1137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:49.775594+0000","last_clean_scrub_stamp":"2024-09-13T14:43:49.775594+0000","objects_scrubbed":0,"log_size":6066,"log_dups_size":0,"ondisk_log_size":6066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:41.713211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14966,"num_write_kb":478912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1090,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.902769+0000","last_change":"2024-09-13T14:19:26.887839+0000","last_active":"2024-09-13T14:48:35.902769+0000","last_peered":"2024-09-13T14:48:35.902769+0000","last_clean":"2024-09-13T14:48:35.902769+0000","last_became_active":"2024-09-13T14:19:26.887498+0000","last_became_peered":"2024-09-13T14:19:26.887498+0000","last_unstale":"2024-09-13T14:48:35.902769+0000","last_undegraded":"2024-09-13T14:48:35.902769+0000","last_fullsized":"2024-09-13T14:48:35.902769+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:24.475263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":1120,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.901641+0000","last_change":"2024-09-13T14:28:35.608607+0000","last_active":"2024-09-13T14:48:35.901641+0000","last_peered":"2024-09-13T14:48:35.901641+0000","last_clean":"2024-09-13T14:48:35.901641+0000","last_became_active":"2024-09-13T14:28:35.608254+0000","last_became_peered":"2024-09-13T14:28:35.608254+0000","last_unstale":"2024-09-13T14:48:35.901641+0000","last_undegraded":"2024-09-13T14:48:35.901641+0000","last_fullsized":"2024-09-13T14:48:35.901641+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:24.019173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"261'100","reported_seq":56,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.901563+0000","last_change":"2024-09-13T14:44:31.191431+0000","last_active":"2024-09-13T14:48:35.901563+0000","last_peered":"2024-09-13T14:48:35.901563+0000","last_clean":"2024-09-13T14:48:35.901563+0000","last_became_active":"2024-09-13T14:43:14.557376+0000","last_became_peered":"2024-09-13T14:43:14.557376+0000","last_unstale":"2024-09-13T14:48:35.901563+0000","last_undegraded":"2024-09-13T14:48:35.901563+0000","last_fullsized":"2024-09-13T14:48:35.901563+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.191326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.191326+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:15:27.352698+0000","scrub_duration":0.0025317360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1109,"reported_epoch":1163,"state":"active+clean","last_fresh":"2024-09-13T14:48:18.828210+0000","last_change":"2024-09-13T14:28:35.603724+0000","last_active":"2024-09-13T14:48:18.828210+0000","last_peered":"2024-09-13T14:48:18.828210+0000","last_clean":"2024-09-13T14:48:18.828210+0000","last_became_active":"2024-09-13T14:28:35.603316+0000","last_became_peered":"2024-09-13T14:28:35.603316+0000","last_unstale":"2024-09-13T14:48:18.828210+0000","last_undegraded":"2024-09-13T14:48:18.828210+0000","last_fullsized":"2024-09-13T14:48:18.828210+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:47.668223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":45,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:03.002819+0000","last_change":"2024-09-13T14:45:19.811466+0000","last_active":"2024-09-13T14:49:03.002819+0000","last_peered":"2024-09-13T14:49:03.002819+0000","last_clean":"2024-09-13T14:49:03.002819+0000","last_became_active":"2024-09-13T14:45:19.810617+0000","last_became_peered":"2024-09-13T14:45:19.810617+0000","last_unstale":"2024-09-13T14:49:03.002819+0000","last_undegraded":"2024-09-13T14:49:03.002819+0000","last_fullsized":"2024-09-13T14:49:03.002819+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:16.246669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"261'100","reported_seq":56,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.902842+0000","last_change":"2024-09-13T14:44:50.805306+0000","last_active":"2024-09-13T14:48:35.902842+0000","last_peered":"2024-09-13T14:48:35.902842+0000","last_clean":"2024-09-13T14:48:35.902842+0000","last_became_active":"2024-09-13T14:43:14.562922+0000","last_became_peered":"2024-09-13T14:43:14.562922+0000","last_unstale":"2024-09-13T14:48:35.902842+0000","last_undegraded":"2024-09-13T14:48:35.902842+0000","last_fullsized":"2024-09-13T14:48:35.902842+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:50.805210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:50.805210+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:28:38.529599+0000","scrub_duration":0.0042900009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1102,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.901177+0000","last_change":"2024-09-13T14:19:25.873088+0000","last_active":"2024-09-13T14:48:35.901177+0000","last_peered":"2024-09-13T14:48:35.901177+0000","last_clean":"2024-09-13T14:48:35.901177+0000","last_became_active":"2024-09-13T14:19:25.872781+0000","last_became_peered":"2024-09-13T14:19:25.872781+0000","last_unstale":"2024-09-13T14:48:35.901177+0000","last_undegraded":"2024-09-13T14:48:35.901177+0000","last_fullsized":"2024-09-13T14:48:35.901177+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:05.607382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1713,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:02.409271+0000","last_change":"2024-09-13T14:45:19.813876+0000","last_active":"2024-09-13T14:49:02.409271+0000","last_peered":"2024-09-13T14:49:02.409271+0000","last_clean":"2024-09-13T14:49:02.409271+0000","last_became_active":"2024-09-13T14:45:19.809136+0000","last_became_peered":"2024-09-13T14:45:19.809136+0000","last_unstale":"2024-09-13T14:49:02.409271+0000","last_undegraded":"2024-09-13T14:49:02.409271+0000","last_fullsized":"2024-09-13T14:49:02.409271+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:50.013528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"261'100","reported_seq":117,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.913171+0000","last_change":"2024-09-13T14:45:19.810654+0000","last_active":"2024-09-13T14:48:35.913171+0000","last_peered":"2024-09-13T14:48:35.913171+0000","last_clean":"2024-09-13T14:48:35.913171+0000","last_became_active":"2024-09-13T14:45:19.810066+0000","last_became_peered":"2024-09-13T14:45:19.810066+0000","last_unstale":"2024-09-13T14:48:35.913171+0000","last_undegraded":"2024-09-13T14:48:35.913171+0000","last_fullsized":"2024-09-13T14:48:35.913171+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:19.978464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:19.978464+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:33.639332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1760,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:02.408727+0000","last_change":"2024-09-13T14:45:19.809166+0000","last_active":"2024-09-13T14:49:02.408727+0000","last_peered":"2024-09-13T14:49:02.408727+0000","last_clean":"2024-09-13T14:49:02.408727+0000","last_became_active":"2024-09-13T14:45:19.808631+0000","last_became_peered":"2024-09-13T14:45:19.808631+0000","last_unstale":"2024-09-13T14:49:02.408727+0000","last_undegraded":"2024-09-13T14:49:02.408727+0000","last_fullsized":"2024-09-13T14:49:02.408727+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:38.822245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1127,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.903401+0000","last_change":"2024-09-13T14:45:19.804229+0000","last_active":"2024-09-13T14:48:35.903401+0000","last_peered":"2024-09-13T14:48:35.903401+0000","last_clean":"2024-09-13T14:48:35.903401+0000","last_became_active":"2024-09-13T14:45:19.803929+0000","last_became_peered":"2024-09-13T14:45:19.803929+0000","last_unstale":"2024-09-13T14:48:35.903401+0000","last_undegraded":"2024-09-13T14:48:35.903401+0000","last_fullsized":"2024-09-13T14:48:35.903401+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:33:53.675568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"261'100","reported_seq":56,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.902506+0000","last_change":"2024-09-13T14:44:42.654506+0000","last_active":"2024-09-13T14:48:35.902506+0000","last_peered":"2024-09-13T14:48:35.902506+0000","last_clean":"2024-09-13T14:48:35.902506+0000","last_became_active":"2024-09-13T14:43:14.563451+0000","last_became_peered":"2024-09-13T14:43:14.563451+0000","last_unstale":"2024-09-13T14:48:35.902506+0000","last_undegraded":"2024-09-13T14:48:35.902506+0000","last_fullsized":"2024-09-13T14:48:35.902506+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:42.654455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:42.654455+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:57:32.251597+0000","scrub_duration":0.001494097,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":1121,"reported_epoch":1164,"state":"active+clean","last_fresh":"2024-09-13T14:48:35.901148+0000","last_change":"2024-09-13T14:29:42.541093+0000","last_active":"2024-09-13T14:48:35.901148+0000","last_peered":"2024-09-13T14:48:35.901148+0000","last_clean":"2024-09-13T14:48:35.901148+0000","last_became_active":"2024-09-13T14:29:42.540671+0000","last_became_peered":"2024-09-13T14:29:42.540671+0000","last_unstale":"2024-09-13T14:48:35.901148+0000","last_undegraded":"2024-09-13T14:48:35.901148+0000","last_fullsized":"2024-09-13T14:48:35.901148+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:45.722376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1652,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:03.002591+0000","last_change":"2024-09-13T14:43:07.518679+0000","last_active":"2024-09-13T14:49:03.002591+0000","last_peered":"2024-09-13T14:49:03.002591+0000","last_clean":"2024-09-13T14:49:03.002591+0000","last_became_active":"2024-09-13T14:43:07.518333+0000","last_became_peered":"2024-09-13T14:43:07.518333+0000","last_unstale":"2024-09-13T14:49:03.002591+0000","last_undegraded":"2024-09-13T14:49:03.002591+0000","last_fullsized":"2024-09-13T14:49:03.002591+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:20.616274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"261'100","reported_seq":421,"reported_epoch":1165,"state":"active+clean","last_fresh":"2024-09-13T14:49:02.408730+0000","last_change":"2024-09-13T14:45:19.809268+0000","last_active":"2024-09-13T14:49:02.408730+0000","last_peered":"2024-09-13T14:49:02.408730+0000","last_clean":"2024-09-13T14:49:02.408730+0000","last_became_active":"2024-09-13T14:45:19.808772+0000","last_became_peered":"2024-09-13T14:45:19.808772+0000","last_unstale":"2024-09-13T14:49:02.408730+0000","last_undegraded":"2024-09-13T14:49:02.408730+0000","last_fullsized":"2024-09-13T14:49:02.408730+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:14.654603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:14.654603+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:40.964553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2593,"num_objects_misplaced":7077,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":240918,"num_write_kb":7709313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4532,"num_bytes_recovered":195756032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":464977920,"data_stored":464977920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112141,"ondisk_log_size":112141,"up":48,"acting":47,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":354515,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430588,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":320660,"kb_used_data":10828,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93424492,"statfs":{"total":95995035648,"available":95666679808,"internally_reserved":0,"allocated":11087872,"data_stored":7504392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626817,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":766472,"kb_used_data":464960,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":92978680,"statfs":{"total":95995035648,"available":95210168320,"internally_reserved":0,"allocated":476119040,"data_stored":472547185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823046,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312344,"kb_used_data":10828,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93432808,"statfs":{"total":95995035648,"available":95675195392,"internally_reserved":0,"allocated":11087872,"data_stored":7508776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019274,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312076,"kb_used_data":10564,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93433076,"statfs":{"total":95995035648,"available":95675469824,"internally_reserved":0,"allocated":10817536,"data_stored":7241634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059592932,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":225288,"kb_used_data":10880,"kb_used_omap":26,"kb_used_meta":214373,"kb_avail":93519864,"statfs":{"total":95995035648,"available":95764340736,"internally_reserved":0,"allocated":11141120,"data_stored":7573219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":219518112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":591,"seq":2538325672170,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":418304,"kb_used_data":10808,"kb_used_omap":41,"kb_used_meta":407446,"kb_avail":93326848,"statfs":{"total":95995035648,"available":95566692352,"internally_reserved":0,"allocated":11067392,"data_stored":7500064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42651,"internal_metadata":417225061},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":557,"seq":2392296784113,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":389816,"kb_used_data":10544,"kb_used_omap":32,"kb_used_meta":379231,"kb_avail":93355336,"statfs":{"total":95995035648,"available":95595864064,"internally_reserved":0,"allocated":10797056,"data_stored":7236799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33752,"internal_metadata":388332584},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":524,"seq":2250562863352,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":336528,"kb_used_data":10884,"kb_used_omap":41,"kb_used_meta":325590,"kb_avail":93408624,"statfs":{"total":95995035648,"available":95650430976,"internally_reserved":0,"allocated":11145216,"data_stored":7573745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42694,"internal_metadata":333404474},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":464977920,"data_stored":464977920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:49:07.942 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:07.967 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:07.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:07.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:08.069 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:08.074 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_2' removed 2024-09-13T14:49:08.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:08 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3565105086' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:49:08.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:08 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:49:08.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:08 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3565105086' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:49:08.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:08 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:49:08.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:08.110 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:08.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:08.211 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:08.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:08.249 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:08.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:08.350 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:08.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:08 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3565105086' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:49:08.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:08 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:49:08.376 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:08.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:08.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:08.478 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:08.499 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:08.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:08.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:08.601 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:08.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:08.624 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:08.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:08.725 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:08.749 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:08.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:08.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:08.851 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:08.873 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:08.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:08.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:08.974 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:08.995 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:08.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:08.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:09.097 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:09.121 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:09.121 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:09.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:09.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:09.223 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:09.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:09.246 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:09.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:09.347 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:09.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:09 smithi184 ceph-mon[92527]: pgmap v1662: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 608 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1821/78 objects degraded (2334.615%); 7077/78 objects misplaced (9073.077%); 0 B/s, 141 objects/s recovering 2024-09-13T14:49:09.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:09 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:49:09.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:09 smithi184 ceph-mon[92527]: osdmap e1166: 8 total, 8 up, 8 in 2024-09-13T14:49:09.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:09 smithi151 ceph-mon[104450]: pgmap v1662: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 608 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1821/78 objects degraded (2334.615%); 7077/78 objects misplaced (9073.077%); 0 B/s, 141 objects/s recovering 2024-09-13T14:49:09.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:09 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:49:09.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:09 smithi151 ceph-mon[104450]: osdmap e1166: 8 total, 8 up, 8 in 2024-09-13T14:49:09.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:09 smithi151 ceph-mon[108966]: pgmap v1662: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 110 active+clean; 608 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1821/78 objects degraded (2334.615%); 7077/78 objects misplaced (9073.077%); 0 B/s, 141 objects/s recovering 2024-09-13T14:49:09.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:09 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:49:09.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:09 smithi151 ceph-mon[108966]: osdmap e1166: 8 total, 8 up, 8 in 2024-09-13T14:49:09.369 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:09.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:09.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:09.472 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:09.493 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:09.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:09.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:09.595 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:09.600 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:09.617 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:09.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:09.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:09.621 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T14:49:09.622 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T14:49:09.623 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-09-13T14:49:09.623 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_3 16 2024-09-13T14:49:09.719 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:09.740 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:09.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:09.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:09.842 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:09.868 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:09.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:09.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:09.970 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:09.992 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:09.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:09.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:10.093 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:10.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:10 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 1821/78 objects degraded (2334.615%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:10.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:10 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 1821/78 objects degraded (2334.615%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:10.115 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:10.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:10.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:10.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:10.237 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:10.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:10.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:10.339 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:10.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:10 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 1821/78 objects degraded (2334.615%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:10.354 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:49:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:49:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:49:10.362 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:10.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:10.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:10.464 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:10.489 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:10.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:10.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:10.558 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:49:10.559 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:49:10.575 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:49:10.575 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:49:10.591 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:10.592 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:49:10.592 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:49:10.607 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:49:10.608 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:49:10.617 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:10.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:10.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:10.623 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:49:10.624 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:49:10.640 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:49:10.640 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:49:10.657 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:49:10.657 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:49:10.674 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:49:10.674 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:49:10.718 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:10.739 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:10.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:10.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:10.841 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:10.862 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:10.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:10.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:10.879 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:49:10.964 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:10.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:10.992 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:10.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:11.093 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:11.114 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:11.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:11.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:11.216 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:11.241 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:11.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:11.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:11.343 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:11.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:11 smithi184 ceph-mon[92527]: pgmap v1664: 103 pgs: 103 active+clean; 608 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:11.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:11 smithi184 ceph-mon[92527]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1821/78 objects degraded (2334.615%), 1 pg degraded, 1 pg undersized) 2024-09-13T14:49:11.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:11 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1080107593' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:11.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:11 smithi151 ceph-mon[104450]: pgmap v1664: 103 pgs: 103 active+clean; 608 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:11.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:11 smithi151 ceph-mon[104450]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1821/78 objects degraded (2334.615%), 1 pg degraded, 1 pg undersized) 2024-09-13T14:49:11.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:11 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1080107593' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:11.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:11 smithi151 ceph-mon[108966]: pgmap v1664: 103 pgs: 103 active+clean; 608 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:11.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:11 smithi151 ceph-mon[108966]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1821/78 objects degraded (2334.615%), 1 pg degraded, 1 pg undersized) 2024-09-13T14:49:11.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:11 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1080107593' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:11.368 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:11.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:11.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:11.470 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:11.492 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:11.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:11.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:11.594 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:11.619 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:11.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:11.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:11.721 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:11.746 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:11.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:11.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:11.848 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:11.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:11.874 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:11.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:11.975 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:11.997 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:11.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:11.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:12.100 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:12.125 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:12.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:12.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:12.227 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:12.251 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:12.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:12.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:12.353 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:12.377 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:12.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:12.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:12.478 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:12.502 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:12.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:12.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:12.604 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:12.630 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:12.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:12.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:12.732 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:12.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:12.758 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:12.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:12.859 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:12.882 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:12.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:12.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:12.984 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:13.009 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:13.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:13.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:13.110 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:13.136 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:13.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:13.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:13.238 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:13.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:13.262 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:13.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:13.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:13 smithi184 ceph-mon[92527]: pgmap v1665: 103 pgs: 103 active+clean; 608 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:13.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:13 smithi151 ceph-mon[104450]: pgmap v1665: 103 pgs: 103 active+clean; 608 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:13.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:13 smithi151 ceph-mon[108966]: pgmap v1665: 103 pgs: 103 active+clean; 608 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:13.363 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:13.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:13.386 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:13.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:13.487 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:13.513 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:13.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:13.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:13.615 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:13.636 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:13.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:13.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:13.738 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:13.745 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:49:13.745 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-13T14:49:13.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-13T14:49:13.745 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-13T14:49:13.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.745 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:257683 6.10 6:0d4d0d4b:::benchmark_data_smithi151_155836_object125225:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1127)", 2024-09-13T14:49:13.745 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:13.538763+0000", 2024-09-13T14:49:13.745 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 360.20301666199998, 2024-09-13T14:49:13.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1550639309999999, 2024-09-13T14:49:13.746 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:49:13.746 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:49:13.746 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-13T14:49:13.746 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:49:13.746 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:49:13.746 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:49:13.746 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 257683 2024-09-13T14:49:13.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.746 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:49:13.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.746 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:49:13.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.538763+0000", 2024-09-13T14:49:13.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:49:13.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.538763+0000", 2024-09-13T14:49:13.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:49:13.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.538764+0000", 2024-09-13T14:49:13.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8059999999999999e-06 2024-09-13T14:49:13.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:49:13.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.538772+0000", 2024-09-13T14:49:13.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9610000000000005e-06 2024-09-13T14:49:13.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:49:13.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.538799+0000", 2024-09-13T14:49:13.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6308999999999998e-05 2024-09-13T14:49:13.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:49:13.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.538808+0000", 2024-09-13T14:49:13.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5210000000000008e-06 2024-09-13T14:49:13.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.538879+0000", 2024-09-13T14:49:13.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1119999999999997e-05 2024-09-13T14:49:13.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:49:13.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.539045+0000", 2024-09-13T14:49:13.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016596100000000001 2024-09-13T14:49:13.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:49:13.750 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.539664+0000", 2024-09-13T14:49:13.750 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00061883100000000003 2024-09-13T14:49:13.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.750 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:49:13.750 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.542365+0000", 2024-09-13T14:49:13.750 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027009109999999999 2024-09-13T14:49:13.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.750 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.750 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.543024+0000", 2024-09-13T14:49:13.750 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00065861000000000001 2024-09-13T14:49:13.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.549078+0000", 2024-09-13T14:49:13.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0060546599999999999 2024-09-13T14:49:13.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.549089+0000", 2024-09-13T14:49:13.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1021e-05 2024-09-13T14:49:13.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.551828+0000", 2024-09-13T14:49:13.752 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0027385710000001 2024-09-13T14:49:13.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.752 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.752 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.551840+0000", 2024-09-13T14:49:13.752 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2179e-05 2024-09-13T14:49:13.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.752 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.752 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.693758+0000", 2024-09-13T14:49:13.752 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.141917796 2024-09-13T14:49:13.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:49:13.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.693827+0000", 2024-09-13T14:49:13.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8674e-05 2024-09-13T14:49:13.753 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.753 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:49:13.753 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.753 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:257709 6.10 6:0bc2a4a6:::benchmark_data_smithi151_155836_object125251:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-13T14:49:13.753 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:13.582787+0000", 2024-09-13T14:49:13.753 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 360.15899239999999, 2024-09-13T14:49:13.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.119586478, 2024-09-13T14:49:13.754 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:49:13.754 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:49:13.754 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:13.754 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:49:13.754 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:49:13.754 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:49:13.754 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 257709 2024-09-13T14:49:13.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.754 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:49:13.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:49:13.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.582787+0000", 2024-09-13T14:49:13.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:49:13.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.582787+0000", 2024-09-13T14:49:13.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:49:13.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.582789+0000", 2024-09-13T14:49:13.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3700000000000002e-06 2024-09-13T14:49:13.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:49:13.756 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.582801+0000", 2024-09-13T14:49:13.756 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1606e-05 2024-09-13T14:49:13.756 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.756 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.756 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:49:13.756 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.582819+0000", 2024-09-13T14:49:13.756 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8074e-05 2024-09-13T14:49:13.756 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.756 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.756 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:49:13.756 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.582827+0000", 2024-09-13T14:49:13.756 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5660000000000007e-06 2024-09-13T14:49:13.756 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.582890+0000", 2024-09-13T14:49:13.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2322000000000004e-05 2024-09-13T14:49:13.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.582898+0000", 2024-09-13T14:49:13.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2179999999999999e-06 2024-09-13T14:49:13.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.551868+0000", 2024-09-13T14:49:13.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.96897046099999995 2024-09-13T14:49:13.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.551877+0000", 2024-09-13T14:49:13.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2940000000000006e-06 2024-09-13T14:49:13.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.694026+0000", 2024-09-13T14:49:13.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.142149455 2024-09-13T14:49:13.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:49:13.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.694107+0000", 2024-09-13T14:49:13.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0277000000000004e-05 2024-09-13T14:49:13.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:49:13.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.694558+0000", 2024-09-13T14:49:13.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000451929 2024-09-13T14:49:13.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702090+0000", 2024-09-13T14:49:13.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0075311750000000002 2024-09-13T14:49:13.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:49:13.760 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702111+0000", 2024-09-13T14:49:13.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1143999999999999e-05 2024-09-13T14:49:13.761 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.761 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.761 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702162+0000", 2024-09-13T14:49:13.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1102e-05 2024-09-13T14:49:13.761 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.761 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.761 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702208+0000", 2024-09-13T14:49:13.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6020000000000003e-05 2024-09-13T14:49:13.761 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:49:13.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702270+0000", 2024-09-13T14:49:13.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1797999999999993e-05 2024-09-13T14:49:13.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:49:13.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702373+0000", 2024-09-13T14:49:13.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103667 2024-09-13T14:49:13.762 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.762 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:49:13.762 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.763 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:257728 6.10 6:0d2b457a:::benchmark_data_smithi151_155836_object125270:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-13T14:49:13.763 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:13.598628+0000", 2024-09-13T14:49:13.763 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 360.14315113399999, 2024-09-13T14:49:13.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1041190269999999, 2024-09-13T14:49:13.763 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:49:13.763 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:49:13.763 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:13.763 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:49:13.763 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:49:13.763 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:49:13.763 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 257728 2024-09-13T14:49:13.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.763 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:49:13.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:49:13.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.598628+0000", 2024-09-13T14:49:13.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:49:13.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.598628+0000", 2024-09-13T14:49:13.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:49:13.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.598630+0000", 2024-09-13T14:49:13.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.874e-06 2024-09-13T14:49:13.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.765 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:49:13.765 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.598635+0000", 2024-09-13T14:49:13.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2780000000000001e-06 2024-09-13T14:49:13.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.765 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:49:13.765 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.598650+0000", 2024-09-13T14:49:13.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4844999999999999e-05 2024-09-13T14:49:13.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.765 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:49:13.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.598657+0000", 2024-09-13T14:49:13.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1169999999999998e-06 2024-09-13T14:49:13.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.598729+0000", 2024-09-13T14:49:13.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1586000000000002e-05 2024-09-13T14:49:13.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.598735+0000", 2024-09-13T14:49:13.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5300000000000002e-06 2024-09-13T14:49:13.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.551905+0000", 2024-09-13T14:49:13.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.95316925399999997 2024-09-13T14:49:13.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.551912+0000", 2024-09-13T14:49:13.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8590000000000005e-06 2024-09-13T14:49:13.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.697203+0000", 2024-09-13T14:49:13.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.145290602 2024-09-13T14:49:13.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:49:13.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.697330+0000", 2024-09-13T14:49:13.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012713399999999999 2024-09-13T14:49:13.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:49:13.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.697632+0000", 2024-09-13T14:49:13.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00030206900000000002 2024-09-13T14:49:13.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:49:13.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702118+0000", 2024-09-13T14:49:13.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0044853760000000001 2024-09-13T14:49:13.769 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.769 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.769 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702437+0000", 2024-09-13T14:49:13.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000318921 2024-09-13T14:49:13.769 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.769 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.769 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702489+0000", 2024-09-13T14:49:13.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2710000000000002e-05 2024-09-13T14:49:13.769 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.770 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.770 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702635+0000", 2024-09-13T14:49:13.770 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014527400000000001 2024-09-13T14:49:13.770 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.770 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:49:13.770 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702670+0000", 2024-09-13T14:49:13.770 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5197999999999997e-05 2024-09-13T14:49:13.770 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.770 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:49:13.771 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702747+0000", 2024-09-13T14:49:13.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7399999999999998e-05 2024-09-13T14:49:13.771 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.771 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:49:13.771 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.771 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:257772 6.10 6:0ebb2d78:::benchmark_data_smithi151_155836_object125314:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-13T14:49:13.771 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:13.644338+0000", 2024-09-13T14:49:13.771 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 360.09744124399998, 2024-09-13T14:49:13.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0617536259999998, 2024-09-13T14:49:13.771 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:49:13.771 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:49:13.772 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:13.772 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:49:13.772 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:49:13.772 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:49:13.772 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 257772 2024-09-13T14:49:13.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.772 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:49:13.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.772 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:49:13.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.644338+0000", 2024-09-13T14:49:13.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.772 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:49:13.773 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.644338+0000", 2024-09-13T14:49:13.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.773 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.773 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:49:13.773 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.644339+0000", 2024-09-13T14:49:13.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.339e-06 2024-09-13T14:49:13.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.773 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.773 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:49:13.773 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.644345+0000", 2024-09-13T14:49:13.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.31e-06 2024-09-13T14:49:13.774 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.774 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:49:13.774 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.644367+0000", 2024-09-13T14:49:13.774 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2107999999999999e-05 2024-09-13T14:49:13.774 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.774 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:49:13.774 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.644376+0000", 2024-09-13T14:49:13.774 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6930000000000002e-06 2024-09-13T14:49:13.774 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.775 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.644445+0000", 2024-09-13T14:49:13.775 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8768000000000003e-05 2024-09-13T14:49:13.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.775 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.644458+0000", 2024-09-13T14:49:13.775 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3213999999999999e-05 2024-09-13T14:49:13.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.775 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552174+0000", 2024-09-13T14:49:13.775 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.90771541899999997 2024-09-13T14:49:13.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.776 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.776 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.776 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552245+0000", 2024-09-13T14:49:13.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0680999999999996e-05 2024-09-13T14:49:13.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.776 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.776 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.776 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.699789+0000", 2024-09-13T14:49:13.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1475443649999999 2024-09-13T14:49:13.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.776 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.777 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:49:13.777 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.699933+0000", 2024-09-13T14:49:13.777 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000144356 2024-09-13T14:49:13.777 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.777 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:49:13.777 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.700163+0000", 2024-09-13T14:49:13.777 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00022942000000000001 2024-09-13T14:49:13.777 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.777 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:49:13.777 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702386+0000", 2024-09-13T14:49:13.777 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002222973 2024-09-13T14:49:13.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702550+0000", 2024-09-13T14:49:13.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016436399999999999 2024-09-13T14:49:13.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702591+0000", 2024-09-13T14:49:13.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0670000000000002e-05 2024-09-13T14:49:13.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.779 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.705911+0000", 2024-09-13T14:49:13.779 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033206339999999998 2024-09-13T14:49:13.779 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.779 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:49:13.779 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.705977+0000", 2024-09-13T14:49:13.779 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5827000000000005e-05 2024-09-13T14:49:13.779 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.779 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:49:13.779 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.706092+0000", 2024-09-13T14:49:13.779 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114485 2024-09-13T14:49:13.779 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.780 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:49:13.780 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.780 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.780 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.780 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:257796 6.10 6:08274f39:::benchmark_data_smithi151_155836_object125338:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-13T14:49:13.780 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:13.665405+0000", 2024-09-13T14:49:13.780 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 360.07637420200001, 2024-09-13T14:49:13.780 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0473038649999999, 2024-09-13T14:49:13.780 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:49:13.780 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:49:13.780 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:13.780 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:49:13.780 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:49:13.781 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:49:13.781 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 257796 2024-09-13T14:49:13.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.781 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:49:13.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.781 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:49:13.781 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.665405+0000", 2024-09-13T14:49:13.781 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.781 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:49:13.781 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.665405+0000", 2024-09-13T14:49:13.782 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.782 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.782 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:49:13.782 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.665407+0000", 2024-09-13T14:49:13.782 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.137e-06 2024-09-13T14:49:13.782 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.782 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:49:13.782 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.665415+0000", 2024-09-13T14:49:13.782 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5009999999999998e-06 2024-09-13T14:49:13.782 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.783 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:49:13.783 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.665435+0000", 2024-09-13T14:49:13.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9928999999999998e-05 2024-09-13T14:49:13.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.783 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.783 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:49:13.783 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.665443+0000", 2024-09-13T14:49:13.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6770000000000001e-06 2024-09-13T14:49:13.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.783 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.783 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.783 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.665518+0000", 2024-09-13T14:49:13.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4191999999999997e-05 2024-09-13T14:49:13.784 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.784 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.784 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.665529+0000", 2024-09-13T14:49:13.784 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.119e-05 2024-09-13T14:49:13.784 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.784 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.784 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552280+0000", 2024-09-13T14:49:13.784 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.88675172700000005 2024-09-13T14:49:13.784 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.784 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552289+0000", 2024-09-13T14:49:13.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1410000000000005e-06 2024-09-13T14:49:13.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702789+0000", 2024-09-13T14:49:13.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1505001939999999 2024-09-13T14:49:13.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:49:13.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702931+0000", 2024-09-13T14:49:13.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000142201 2024-09-13T14:49:13.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.786 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:49:13.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.703228+0000", 2024-09-13T14:49:13.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00029683299999999998 2024-09-13T14:49:13.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.786 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.712398+0000", 2024-09-13T14:49:13.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0091702940000000007 2024-09-13T14:49:13.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.786 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.712453+0000", 2024-09-13T14:49:13.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4666000000000001e-05 2024-09-13T14:49:13.787 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.787 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.787 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.787 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.712507+0000", 2024-09-13T14:49:13.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4418e-05 2024-09-13T14:49:13.787 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.787 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.787 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:49:13.787 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.712556+0000", 2024-09-13T14:49:13.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8690000000000003e-05 2024-09-13T14:49:13.787 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.787 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.787 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:49:13.788 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.712592+0000", 2024-09-13T14:49:13.788 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6566999999999997e-05 2024-09-13T14:49:13.788 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.788 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.788 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:49:13.788 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.712709+0000", 2024-09-13T14:49:13.788 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116508 2024-09-13T14:49:13.788 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.788 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:49:13.788 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.788 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.788 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.788 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:257805 6.10 6:0dc62792:::benchmark_data_smithi151_155836_object125347:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-13T14:49:13.788 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:13.676320+0000", 2024-09-13T14:49:13.789 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 360.065459127, 2024-09-13T14:49:13.789 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0366515989999998, 2024-09-13T14:49:13.789 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:49:13.789 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:49:13.789 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:13.789 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:49:13.789 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:49:13.789 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:49:13.789 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 257805 2024-09-13T14:49:13.789 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.789 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:49:13.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.789 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:49:13.790 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.676320+0000", 2024-09-13T14:49:13.790 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.790 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:49:13.790 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.676320+0000", 2024-09-13T14:49:13.790 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.790 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:49:13.790 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.676322+0000", 2024-09-13T14:49:13.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.649e-06 2024-09-13T14:49:13.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.791 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:49:13.791 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.676328+0000", 2024-09-13T14:49:13.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1090000000000003e-06 2024-09-13T14:49:13.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.791 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:49:13.791 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.676344+0000", 2024-09-13T14:49:13.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6498999999999999e-05 2024-09-13T14:49:13.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.792 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:49:13.792 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.676351+0000", 2024-09-13T14:49:13.792 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8140000000000003e-06 2024-09-13T14:49:13.792 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.792 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.792 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.792 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.676407+0000", 2024-09-13T14:49:13.792 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6198e-05 2024-09-13T14:49:13.792 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.792 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.792 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.793 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.676415+0000", 2024-09-13T14:49:13.793 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9160000000000001e-06 2024-09-13T14:49:13.793 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.793 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.793 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.793 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552313+0000", 2024-09-13T14:49:13.793 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.87589755899999999 2024-09-13T14:49:13.793 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.793 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.793 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.793 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552320+0000", 2024-09-13T14:49:13.793 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0480000000000001e-06 2024-09-13T14:49:13.793 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.794 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.794 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.794 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.706131+0000", 2024-09-13T14:49:13.794 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.153810649 2024-09-13T14:49:13.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.794 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.794 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:49:13.794 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.706263+0000", 2024-09-13T14:49:13.794 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013223 2024-09-13T14:49:13.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.794 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.794 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:49:13.794 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.706719+0000", 2024-09-13T14:49:13.795 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045607500000000001 2024-09-13T14:49:13.795 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.795 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.795 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.712543+0000", 2024-09-13T14:49:13.795 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0058245789999999999 2024-09-13T14:49:13.795 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.795 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:49:13.795 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.712717+0000", 2024-09-13T14:49:13.795 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000173924 2024-09-13T14:49:13.795 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.796 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.796 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.712753+0000", 2024-09-13T14:49:13.796 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5793000000000001e-05 2024-09-13T14:49:13.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.796 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.796 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.712809+0000", 2024-09-13T14:49:13.796 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5949000000000003e-05 2024-09-13T14:49:13.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.796 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:49:13.796 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.712844+0000", 2024-09-13T14:49:13.796 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4761000000000001e-05 2024-09-13T14:49:13.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.797 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:49:13.797 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.712972+0000", 2024-09-13T14:49:13.797 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000127847 2024-09-13T14:49:13.797 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.797 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:49:13.797 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.797 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:257827 6.10 6:0d3dae4d:::benchmark_data_smithi151_155836_object125369:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-13T14:49:13.797 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:13.700521+0000", 2024-09-13T14:49:13.797 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 360.041257911, 2024-09-13T14:49:13.798 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0188779120000002, 2024-09-13T14:49:13.798 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:49:13.798 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:49:13.798 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:13.798 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:49:13.798 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:49:13.798 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:49:13.798 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 257827 2024-09-13T14:49:13.798 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.798 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:49:13.798 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.798 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:49:13.798 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.700521+0000", 2024-09-13T14:49:13.798 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.799 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.799 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.799 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:49:13.799 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.700521+0000", 2024-09-13T14:49:13.799 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.799 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.799 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.799 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:49:13.799 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.700522+0000", 2024-09-13T14:49:13.799 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1999999999999998e-07 2024-09-13T14:49:13.799 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.799 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.799 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:49:13.800 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.700527+0000", 2024-09-13T14:49:13.800 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3159999999999998e-06 2024-09-13T14:49:13.800 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.800 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.800 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:49:13.800 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.700535+0000", 2024-09-13T14:49:13.800 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1729999999999995e-06 2024-09-13T14:49:13.800 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.800 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.800 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:49:13.800 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.700541+0000", 2024-09-13T14:49:13.800 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6189999999999996e-06 2024-09-13T14:49:13.800 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.801 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.801 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.801 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.700603+0000", 2024-09-13T14:49:13.801 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1200999999999999e-05 2024-09-13T14:49:13.801 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.801 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.801 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.801 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.700611+0000", 2024-09-13T14:49:13.801 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4060000000000005e-06 2024-09-13T14:49:13.801 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.801 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.801 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.801 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552345+0000", 2024-09-13T14:49:13.801 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.85173386600000001 2024-09-13T14:49:13.802 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.802 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.802 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.802 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552352+0000", 2024-09-13T14:49:13.802 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4839999999999998e-06 2024-09-13T14:49:13.802 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.802 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.802 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.802 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.709531+0000", 2024-09-13T14:49:13.802 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.157178778 2024-09-13T14:49:13.802 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.802 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.802 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:49:13.803 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.709647+0000", 2024-09-13T14:49:13.803 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000115513 2024-09-13T14:49:13.803 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.803 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.803 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:49:13.803 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.710047+0000", 2024-09-13T14:49:13.803 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00040017699999999999 2024-09-13T14:49:13.803 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.803 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.803 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:49:13.803 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.712763+0000", 2024-09-13T14:49:13.803 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027165779999999999 2024-09-13T14:49:13.803 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.804 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.804 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.804 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.719122+0000", 2024-09-13T14:49:13.804 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0063590690000000002 2024-09-13T14:49:13.804 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.804 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.804 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.804 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.719177+0000", 2024-09-13T14:49:13.804 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4982000000000003e-05 2024-09-13T14:49:13.804 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.804 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.804 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.805 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.719219+0000", 2024-09-13T14:49:13.805 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1709999999999999e-05 2024-09-13T14:49:13.805 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.805 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.805 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:49:13.805 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.719282+0000", 2024-09-13T14:49:13.805 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3262000000000005e-05 2024-09-13T14:49:13.805 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.805 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.805 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:49:13.805 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.719399+0000", 2024-09-13T14:49:13.805 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116858 2024-09-13T14:49:13.805 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.805 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:49:13.806 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.806 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.806 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.806 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:257880 6.10 6:0d9bf4af:::benchmark_data_smithi151_155836_object125422:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-13T14:49:13.806 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:13.763399+0000", 2024-09-13T14:49:13.806 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 359.97838051500003, 2024-09-13T14:49:13.806 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.963654477, 2024-09-13T14:49:13.806 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:49:13.806 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:49:13.806 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:13.806 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:49:13.806 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:49:13.807 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:49:13.807 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 257880 2024-09-13T14:49:13.807 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.807 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:49:13.807 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.807 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:49:13.807 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.763399+0000", 2024-09-13T14:49:13.807 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.807 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.807 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.807 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:49:13.807 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.763399+0000", 2024-09-13T14:49:13.807 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.808 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.808 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.808 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:49:13.808 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.763400+0000", 2024-09-13T14:49:13.808 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6499999999999998e-07 2024-09-13T14:49:13.808 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.808 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.808 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:49:13.808 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.763404+0000", 2024-09-13T14:49:13.808 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9589999999999997e-06 2024-09-13T14:49:13.808 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.808 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.808 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:49:13.809 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.763415+0000", 2024-09-13T14:49:13.809 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1127e-05 2024-09-13T14:49:13.809 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.809 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.809 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:49:13.809 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.763422+0000", 2024-09-13T14:49:13.809 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4089999999999997e-06 2024-09-13T14:49:13.809 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.809 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.809 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.809 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.763483+0000", 2024-09-13T14:49:13.809 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0893000000000001e-05 2024-09-13T14:49:13.809 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.810 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.810 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.810 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.763491+0000", 2024-09-13T14:49:13.810 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9880000000000004e-06 2024-09-13T14:49:13.810 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.810 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.810 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.810 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552377+0000", 2024-09-13T14:49:13.810 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.78888635399999996 2024-09-13T14:49:13.810 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.810 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.810 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.810 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552385+0000", 2024-09-13T14:49:13.811 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5730000000000002e-06 2024-09-13T14:49:13.811 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.811 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.811 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.811 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.713023+0000", 2024-09-13T14:49:13.811 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.160637929 2024-09-13T14:49:13.811 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.811 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.811 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:49:13.811 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.713145+0000", 2024-09-13T14:49:13.811 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012216000000000001 2024-09-13T14:49:13.811 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.811 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.812 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:49:13.812 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.713535+0000", 2024-09-13T14:49:13.812 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039047000000000002 2024-09-13T14:49:13.812 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.812 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.812 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:49:13.812 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.719136+0000", 2024-09-13T14:49:13.812 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0056008610000000004 2024-09-13T14:49:13.812 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.812 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.812 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.812 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.726475+0000", 2024-09-13T14:49:13.812 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0073387640000000002 2024-09-13T14:49:13.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.813 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.813 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.813 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.726526+0000", 2024-09-13T14:49:13.813 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1286999999999999e-05 2024-09-13T14:49:13.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.813 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.813 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.813 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.726606+0000", 2024-09-13T14:49:13.813 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9840999999999997e-05 2024-09-13T14:49:13.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.813 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.813 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:49:13.813 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.726925+0000", 2024-09-13T14:49:13.814 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00031910899999999998 2024-09-13T14:49:13.814 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.814 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.814 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:49:13.814 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.727053+0000", 2024-09-13T14:49:13.814 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012788799999999999 2024-09-13T14:49:13.814 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.814 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:49:13.814 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.814 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.814 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.814 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:257940 6.10 6:09b2c975:::benchmark_data_smithi151_155836_object125482:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-13T14:49:13.814 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:13.834914+0000", 2024-09-13T14:49:13.815 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 359.90686572300001, 2024-09-13T14:49:13.815 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.892473069, 2024-09-13T14:49:13.815 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:49:13.815 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:49:13.815 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:13.815 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:49:13.815 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:49:13.815 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:49:13.815 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 257940 2024-09-13T14:49:13.815 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.815 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:49:13.815 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.815 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:49:13.815 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.834914+0000", 2024-09-13T14:49:13.816 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.816 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.816 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.816 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:49:13.816 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.834914+0000", 2024-09-13T14:49:13.816 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.816 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.816 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.816 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:49:13.816 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.834915+0000", 2024-09-13T14:49:13.816 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4529999999999999e-06 2024-09-13T14:49:13.816 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.816 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.816 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:49:13.817 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.834920+0000", 2024-09-13T14:49:13.817 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5560000000000001e-06 2024-09-13T14:49:13.817 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.817 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.817 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:49:13.817 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.834946+0000", 2024-09-13T14:49:13.817 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6834000000000001e-05 2024-09-13T14:49:13.817 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.817 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.817 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:49:13.817 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.834954+0000", 2024-09-13T14:49:13.817 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4830000000000002e-06 2024-09-13T14:49:13.817 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.817 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.818 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.818 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.835153+0000", 2024-09-13T14:49:13.818 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019867299999999999 2024-09-13T14:49:13.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.818 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.818 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.818 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.835165+0000", 2024-09-13T14:49:13.818 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.225e-05 2024-09-13T14:49:13.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.818 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.818 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.818 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552411+0000", 2024-09-13T14:49:13.818 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.71724599 2024-09-13T14:49:13.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.819 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:13.820 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:13.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:13.821 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.821 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552418+0000", 2024-09-13T14:49:13.821 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.277e-06 2024-09-13T14:49:13.821 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.821 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.821 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.821 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.716114+0000", 2024-09-13T14:49:13.822 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.163696184 2024-09-13T14:49:13.822 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.822 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.822 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:49:13.822 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.716215+0000", 2024-09-13T14:49:13.822 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101093 2024-09-13T14:49:13.822 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.822 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.822 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:49:13.822 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.716531+0000", 2024-09-13T14:49:13.822 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00031590900000000001 2024-09-13T14:49:13.822 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.823 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.823 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:49:13.823 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.719143+0000", 2024-09-13T14:49:13.823 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002611333 2024-09-13T14:49:13.823 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.823 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.823 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.823 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.726572+0000", 2024-09-13T14:49:13.823 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0074290759999999997 2024-09-13T14:49:13.823 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.823 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.823 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.823 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.727096+0000", 2024-09-13T14:49:13.824 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00052479899999999995 2024-09-13T14:49:13.824 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.824 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.824 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.824 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.727135+0000", 2024-09-13T14:49:13.824 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8365999999999998e-05 2024-09-13T14:49:13.824 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.824 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.824 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:49:13.824 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.727193+0000", 2024-09-13T14:49:13.824 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8394999999999999e-05 2024-09-13T14:49:13.824 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.824 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.825 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:49:13.825 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.727387+0000", 2024-09-13T14:49:13.825 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019339800000000001 2024-09-13T14:49:13.825 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.825 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:49:13.825 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.825 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.825 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.825 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:257961 6.10 6:0bf0500a:::benchmark_data_smithi151_155836_object125503:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-13T14:49:13.825 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:13.855136+0000", 2024-09-13T14:49:13.825 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 359.88664281000001, 2024-09-13T14:49:13.825 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.883789376, 2024-09-13T14:49:13.825 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:49:13.826 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:49:13.826 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:13.826 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:49:13.826 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:49:13.826 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:49:13.826 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 257961 2024-09-13T14:49:13.826 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.826 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:49:13.826 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.826 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:49:13.826 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.855136+0000", 2024-09-13T14:49:13.826 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.826 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.827 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:49:13.827 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.855136+0000", 2024-09-13T14:49:13.827 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.827 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:49:13.827 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.855138+0000", 2024-09-13T14:49:13.827 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1999999999999999e-06 2024-09-13T14:49:13.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.827 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:49:13.827 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.855142+0000", 2024-09-13T14:49:13.827 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3239999999999996e-06 2024-09-13T14:49:13.828 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.828 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.828 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:49:13.828 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.855153+0000", 2024-09-13T14:49:13.828 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1096e-05 2024-09-13T14:49:13.828 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.828 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.828 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:49:13.828 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.855160+0000", 2024-09-13T14:49:13.828 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9789999999999996e-06 2024-09-13T14:49:13.828 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.828 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.828 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.829 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.855220+0000", 2024-09-13T14:49:13.829 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0195000000000002e-05 2024-09-13T14:49:13.829 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.829 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.829 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.829 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.855229+0000", 2024-09-13T14:49:13.829 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7450000000000001e-06 2024-09-13T14:49:13.829 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.830 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.830 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.830 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552444+0000", 2024-09-13T14:49:13.830 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.69721484 2024-09-13T14:49:13.830 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.830 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.830 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.830 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552451+0000", 2024-09-13T14:49:13.830 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5120000000000002e-06 2024-09-13T14:49:13.830 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.830 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.830 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.830 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.719431+0000", 2024-09-13T14:49:13.830 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.166979166 2024-09-13T14:49:13.831 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.831 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.831 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:49:13.831 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.719556+0000", 2024-09-13T14:49:13.831 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012549699999999999 2024-09-13T14:49:13.831 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.831 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.831 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:49:13.831 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.719996+0000", 2024-09-13T14:49:13.831 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00044005699999999999 2024-09-13T14:49:13.831 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.831 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.831 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:49:13.832 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.726489+0000", 2024-09-13T14:49:13.832 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0064931499999999996 2024-09-13T14:49:13.832 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.832 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.832 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.832 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.727436+0000", 2024-09-13T14:49:13.832 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00094664900000000004 2024-09-13T14:49:13.832 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.832 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.832 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.832 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.738590+0000", 2024-09-13T14:49:13.832 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011154410999999999 2024-09-13T14:49:13.832 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.833 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.833 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.833 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.738673+0000", 2024-09-13T14:49:13.833 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2311000000000005e-05 2024-09-13T14:49:13.833 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.833 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.833 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:49:13.833 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.738783+0000", 2024-09-13T14:49:13.833 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000110529 2024-09-13T14:49:13.833 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.833 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.833 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:49:13.833 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.738926+0000", 2024-09-13T14:49:13.833 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014271499999999999 2024-09-13T14:49:13.834 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.834 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:49:13.834 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.834 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.834 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.834 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:258052 6.10 6:08f50a5c:::benchmark_data_smithi151_155836_object125594:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-13T14:49:13.834 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:13.975509+0000", 2024-09-13T14:49:13.834 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 359.76627018900001, 2024-09-13T14:49:13.834 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7637252050000001, 2024-09-13T14:49:13.834 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:49:13.834 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:49:13.834 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:13.834 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:49:13.834 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:49:13.835 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:49:13.835 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 258052 2024-09-13T14:49:13.835 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.835 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:49:13.835 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.835 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:49:13.835 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.975509+0000", 2024-09-13T14:49:13.835 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.835 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.835 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.835 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:49:13.835 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.975509+0000", 2024-09-13T14:49:13.835 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.836 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.836 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.836 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:49:13.836 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.975511+0000", 2024-09-13T14:49:13.836 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.438e-06 2024-09-13T14:49:13.836 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.836 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.836 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:49:13.836 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.975517+0000", 2024-09-13T14:49:13.836 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.55e-06 2024-09-13T14:49:13.836 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.836 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.836 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:49:13.837 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.975532+0000", 2024-09-13T14:49:13.837 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4935e-05 2024-09-13T14:49:13.837 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.837 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.837 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:49:13.837 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.975547+0000", 2024-09-13T14:49:13.837 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4927e-05 2024-09-13T14:49:13.837 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.837 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.837 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.837 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.975614+0000", 2024-09-13T14:49:13.837 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7502000000000005e-05 2024-09-13T14:49:13.837 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.838 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.838 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.838 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.975627+0000", 2024-09-13T14:49:13.838 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2982999999999999e-05 2024-09-13T14:49:13.838 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.838 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.838 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.838 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552478+0000", 2024-09-13T14:49:13.838 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.57685019599999998 2024-09-13T14:49:13.838 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.838 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.838 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.838 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552485+0000", 2024-09-13T14:49:13.838 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2099999999999996e-06 2024-09-13T14:49:13.839 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.839 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.839 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.839 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.722770+0000", 2024-09-13T14:49:13.839 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.170285639 2024-09-13T14:49:13.839 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.839 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.839 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:49:13.839 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.722894+0000", 2024-09-13T14:49:13.839 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000123898 2024-09-13T14:49:13.839 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.839 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.839 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:49:13.839 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.723290+0000", 2024-09-13T14:49:13.840 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039553100000000002 2024-09-13T14:49:13.840 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.840 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.840 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:49:13.840 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.726538+0000", 2024-09-13T14:49:13.840 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0032478350000000001 2024-09-13T14:49:13.840 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.840 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.840 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.840 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.738540+0000", 2024-09-13T14:49:13.840 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012001927000000001 2024-09-13T14:49:13.840 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.840 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.841 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.841 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.738633+0000", 2024-09-13T14:49:13.841 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3621000000000004e-05 2024-09-13T14:49:13.841 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.841 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.841 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.738976+0000", 2024-09-13T14:49:13.841 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00034300999999999998 2024-09-13T14:49:13.841 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.841 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:49:13.841 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.739124+0000", 2024-09-13T14:49:13.841 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014781500000000001 2024-09-13T14:49:13.842 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.842 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:49:13.842 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.739234+0000", 2024-09-13T14:49:13.842 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000110188 2024-09-13T14:49:13.842 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.842 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:49:13.842 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.842 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.842 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:258153 6.10 6:0bb5117f:::benchmark_data_smithi151_155836_object125695:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-13T14:49:13.842 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:14.087371+0000", 2024-09-13T14:49:13.842 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 359.65440838199999, 2024-09-13T14:49:13.843 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.659579189, 2024-09-13T14:49:13.843 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:49:13.843 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:49:13.843 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:13.843 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:49:13.843 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:49:13.843 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:49:13.843 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 258153 2024-09-13T14:49:13.843 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.843 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:49:13.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.843 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:49:13.843 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.087371+0000", 2024-09-13T14:49:13.844 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.844 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.844 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:49:13.844 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.087371+0000", 2024-09-13T14:49:13.844 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.844 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.844 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:49:13.844 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.087372+0000", 2024-09-13T14:49:13.844 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.567e-06 2024-09-13T14:49:13.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.844 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.845 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:49:13.845 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.087378+0000", 2024-09-13T14:49:13.845 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9200000000000001e-06 2024-09-13T14:49:13.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.845 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:49:13.845 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.087391+0000", 2024-09-13T14:49:13.845 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2313000000000001e-05 2024-09-13T14:49:13.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.845 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:49:13.845 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.087396+0000", 2024-09-13T14:49:13.845 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6969999999999998e-06 2024-09-13T14:49:13.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.846 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.846 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.087448+0000", 2024-09-13T14:49:13.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1969000000000003e-05 2024-09-13T14:49:13.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.846 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.846 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.087456+0000", 2024-09-13T14:49:13.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0940000000000008e-06 2024-09-13T14:49:13.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.846 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.847 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552511+0000", 2024-09-13T14:49:13.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.46505427399999999 2024-09-13T14:49:13.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.847 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.847 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552518+0000", 2024-09-13T14:49:13.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6960000000000005e-06 2024-09-13T14:49:13.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.847 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.847 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.727474+0000", 2024-09-13T14:49:13.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.174955296 2024-09-13T14:49:13.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.848 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:49:13.848 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.727607+0000", 2024-09-13T14:49:13.848 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013341899999999999 2024-09-13T14:49:13.848 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.848 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:49:13.848 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.728101+0000", 2024-09-13T14:49:13.848 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00049359700000000002 2024-09-13T14:49:13.848 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.848 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.848 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.739276+0000", 2024-09-13T14:49:13.848 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011175197 2024-09-13T14:49:13.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.849 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.849 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.849 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.739352+0000", 2024-09-13T14:49:13.849 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5711999999999996e-05 2024-09-13T14:49:13.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.849 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.849 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:49:13.849 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.746603+0000", 2024-09-13T14:49:13.849 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0072512039999999998 2024-09-13T14:49:13.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.849 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.849 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.850 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.746671+0000", 2024-09-13T14:49:13.850 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8139000000000006e-05 2024-09-13T14:49:13.850 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.850 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.850 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:49:13.850 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.746734+0000", 2024-09-13T14:49:13.850 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3295999999999998e-05 2024-09-13T14:49:13.850 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.850 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.850 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:49:13.850 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.746950+0000", 2024-09-13T14:49:13.850 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000215799 2024-09-13T14:49:13.850 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.851 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:49:13.851 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.851 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.851 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.851 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:258159 6.10 6:0bd99f59:::benchmark_data_smithi151_155836_object125701:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-13T14:49:13.851 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:14.093771+0000", 2024-09-13T14:49:13.851 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 359.64800834800002, 2024-09-13T14:49:13.851 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6632812779999999, 2024-09-13T14:49:13.851 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:49:13.851 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:49:13.851 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:13.851 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:49:13.851 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:49:13.852 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:49:13.852 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 258159 2024-09-13T14:49:13.852 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.852 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:49:13.852 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.852 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:49:13.852 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.093771+0000", 2024-09-13T14:49:13.852 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.852 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.852 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.852 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:49:13.852 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.093771+0000", 2024-09-13T14:49:13.852 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.852 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.853 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.853 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:49:13.853 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.093771+0000", 2024-09-13T14:49:13.853 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3399999999999999e-07 2024-09-13T14:49:13.853 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.853 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.853 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:49:13.853 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.093774+0000", 2024-09-13T14:49:13.853 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9579999999999999e-06 2024-09-13T14:49:13.853 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.853 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.853 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:49:13.853 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.093789+0000", 2024-09-13T14:49:13.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4372999999999999e-05 2024-09-13T14:49:13.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:49:13.854 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.093799+0000", 2024-09-13T14:49:13.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0061000000000001e-05 2024-09-13T14:49:13.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.854 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.093891+0000", 2024-09-13T14:49:13.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2108000000000006e-05 2024-09-13T14:49:13.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.855 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.855 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.093902+0000", 2024-09-13T14:49:13.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1147e-05 2024-09-13T14:49:13.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.855 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.855 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552544+0000", 2024-09-13T14:49:13.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.45864198499999997 2024-09-13T14:49:13.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.855 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.855 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552552+0000", 2024-09-13T14:49:13.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5809999999999999e-06 2024-09-13T14:49:13.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.856 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.856 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.856 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.731453+0000", 2024-09-13T14:49:13.856 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.178901524 2024-09-13T14:49:13.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.856 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.856 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:49:13.856 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.731594+0000", 2024-09-13T14:49:13.856 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014075000000000001 2024-09-13T14:49:13.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.856 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.856 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:49:13.857 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.731982+0000", 2024-09-13T14:49:13.857 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00038823200000000002 2024-09-13T14:49:13.857 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.857 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.857 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.857 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.739313+0000", 2024-09-13T14:49:13.857 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0073308289999999996 2024-09-13T14:49:13.857 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.857 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.857 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:49:13.857 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.746610+0000", 2024-09-13T14:49:13.857 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0072970980000000001 2024-09-13T14:49:13.857 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.858 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.858 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.858 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.747139+0000", 2024-09-13T14:49:13.858 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00052901100000000002 2024-09-13T14:49:13.858 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.858 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.858 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.858 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.755755+0000", 2024-09-13T14:49:13.858 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0086160409999999996 2024-09-13T14:49:13.858 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.858 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.858 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:49:13.858 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.756934+0000", 2024-09-13T14:49:13.858 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011788790000000001 2024-09-13T14:49:13.859 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.859 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.859 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:49:13.859 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.757052+0000", 2024-09-13T14:49:13.859 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118167 2024-09-13T14:49:13.859 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.859 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:49:13.859 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.859 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.859 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.859 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:258205 6.10 6:0d7fd54b:::benchmark_data_smithi151_155836_object125747:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-13T14:49:13.859 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:14.154924+0000", 2024-09-13T14:49:13.859 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 359.58685496800001, 2024-09-13T14:49:13.860 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.602350766, 2024-09-13T14:49:13.860 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:49:13.860 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:49:13.860 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:13.860 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:49:13.860 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:49:13.860 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:49:13.860 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 258205 2024-09-13T14:49:13.860 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.860 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:49:13.860 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.860 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:49:13.860 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.154924+0000", 2024-09-13T14:49:13.860 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.861 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.861 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:49:13.861 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.154924+0000", 2024-09-13T14:49:13.861 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.861 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.861 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:49:13.861 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.154925+0000", 2024-09-13T14:49:13.861 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.181e-06 2024-09-13T14:49:13.861 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.862 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:49:13.862 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.154930+0000", 2024-09-13T14:49:13.862 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0289999999999999e-06 2024-09-13T14:49:13.862 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.862 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.862 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:49:13.862 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.154941+0000", 2024-09-13T14:49:13.862 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1651000000000001e-05 2024-09-13T14:49:13.862 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.862 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.862 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:49:13.862 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.154948+0000", 2024-09-13T14:49:13.862 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0330000000000003e-06 2024-09-13T14:49:13.862 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.863 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.863 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.155012+0000", 2024-09-13T14:49:13.863 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3864999999999999e-05 2024-09-13T14:49:13.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.863 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.863 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.155025+0000", 2024-09-13T14:49:13.863 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2625e-05 2024-09-13T14:49:13.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.863 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.864 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552577+0000", 2024-09-13T14:49:13.864 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.39755244899999997 2024-09-13T14:49:13.864 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.864 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.864 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.864 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552584+0000", 2024-09-13T14:49:13.864 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3370000000000002e-06 2024-09-13T14:49:13.864 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.864 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.864 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.864 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.734466+0000", 2024-09-13T14:49:13.864 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.181881196 2024-09-13T14:49:13.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.865 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.865 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:49:13.865 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.734592+0000", 2024-09-13T14:49:13.865 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012631099999999999 2024-09-13T14:49:13.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.865 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.865 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:49:13.865 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.734982+0000", 2024-09-13T14:49:13.865 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000390252 2024-09-13T14:49:13.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.866 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:49:13.866 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.747156+0000", 2024-09-13T14:49:13.866 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012174114 2024-09-13T14:49:13.866 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.866 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.866 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.747200+0000", 2024-09-13T14:49:13.866 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3495999999999998e-05 2024-09-13T14:49:13.866 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.866 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.866 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.750743+0000", 2024-09-13T14:49:13.866 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0035429839999999999 2024-09-13T14:49:13.867 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.867 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.867 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.867 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.757115+0000", 2024-09-13T14:49:13.867 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0063716789999999999 2024-09-13T14:49:13.867 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.867 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.867 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:49:13.867 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.757174+0000", 2024-09-13T14:49:13.867 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9864999999999997e-05 2024-09-13T14:49:13.867 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.867 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.867 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:49:13.868 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.757275+0000", 2024-09-13T14:49:13.868 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010069900000000001 2024-09-13T14:49:13.868 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.868 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:49:13.868 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.868 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.868 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.868 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:258258 6.10 6:0ac2d4b7:::benchmark_data_smithi151_155836_object125800:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-13T14:49:13.868 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:14.255036+0000", 2024-09-13T14:49:13.868 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 359.48674283999998, 2024-09-13T14:49:13.868 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.502433766, 2024-09-13T14:49:13.868 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:49:13.868 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:49:13.869 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:13.869 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:49:13.869 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:49:13.869 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:49:13.869 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 258258 2024-09-13T14:49:13.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.869 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:49:13.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.869 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:49:13.869 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.255036+0000", 2024-09-13T14:49:13.869 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.872 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:49:13.872 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.255036+0000", 2024-09-13T14:49:13.872 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.872 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.873 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:49:13.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.255037+0000", 2024-09-13T14:49:13.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7000000000000003e-07 2024-09-13T14:49:13.873 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.873 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.873 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:49:13.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.255042+0000", 2024-09-13T14:49:13.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5210000000000004e-06 2024-09-13T14:49:13.873 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.873 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.873 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:49:13.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.255051+0000", 2024-09-13T14:49:13.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7739999999999993e-06 2024-09-13T14:49:13.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:49:13.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.255056+0000", 2024-09-13T14:49:13.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7960000000000003e-06 2024-09-13T14:49:13.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.255105+0000", 2024-09-13T14:49:13.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9925000000000001e-05 2024-09-13T14:49:13.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.875 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.875 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.255114+0000", 2024-09-13T14:49:13.875 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2900000000000002e-06 2024-09-13T14:49:13.875 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.875 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.875 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552610+0000", 2024-09-13T14:49:13.875 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29749615499999998 2024-09-13T14:49:13.875 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.875 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.875 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552617+0000", 2024-09-13T14:49:13.875 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3560000000000004e-06 2024-09-13T14:49:13.876 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.876 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.876 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.876 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.739380+0000", 2024-09-13T14:49:13.876 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.186762997 2024-09-13T14:49:13.876 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.876 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.876 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:49:13.876 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.739504+0000", 2024-09-13T14:49:13.876 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000123793 2024-09-13T14:49:13.876 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.876 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.876 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:49:13.877 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.740007+0000", 2024-09-13T14:49:13.877 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00050336499999999998 2024-09-13T14:49:13.877 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.877 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.877 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:49:13.877 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.747163+0000", 2024-09-13T14:49:13.877 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0071556759999999997 2024-09-13T14:49:13.877 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.877 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.877 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.877 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.747244+0000", 2024-09-13T14:49:13.877 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0593000000000006e-05 2024-09-13T14:49:13.877 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.878 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.878 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.878 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.750787+0000", 2024-09-13T14:49:13.878 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003543792 2024-09-13T14:49:13.878 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.878 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.878 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.878 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.757323+0000", 2024-09-13T14:49:13.878 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0065354250000000001 2024-09-13T14:49:13.878 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.878 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.878 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:49:13.878 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.757374+0000", 2024-09-13T14:49:13.878 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1622000000000001e-05 2024-09-13T14:49:13.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.879 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:49:13.879 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.757470+0000", 2024-09-13T14:49:13.879 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5716000000000003e-05 2024-09-13T14:49:13.879 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.879 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:49:13.879 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.879 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:258270 6.10 6:0a17d4dd:::benchmark_data_smithi151_155836_object125812:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-13T14:49:13.879 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:14.294755+0000", 2024-09-13T14:49:13.879 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 359.447024537, 2024-09-13T14:49:13.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.46420789, 2024-09-13T14:49:13.880 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:49:13.880 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:49:13.880 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:13.880 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:49:13.880 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:49:13.880 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:49:13.880 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 258270 2024-09-13T14:49:13.880 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.880 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:49:13.880 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.880 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:49:13.881 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.294755+0000", 2024-09-13T14:49:13.881 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.881 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.881 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.881 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:49:13.881 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.294755+0000", 2024-09-13T14:49:13.881 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.881 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.881 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.881 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:49:13.881 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.294757+0000", 2024-09-13T14:49:13.881 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9400000000000001e-06 2024-09-13T14:49:13.881 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.881 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.882 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:49:13.882 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.294764+0000", 2024-09-13T14:49:13.882 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3499999999999999e-06 2024-09-13T14:49:13.882 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.882 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.882 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:13.883 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:49:13.883 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.294782+0000", 2024-09-13T14:49:13.883 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8267999999999999e-05 2024-09-13T14:49:13.883 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.883 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.883 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:49:13.883 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.294798+0000", 2024-09-13T14:49:13.884 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5389000000000001e-05 2024-09-13T14:49:13.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.884 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.884 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.884 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.294856+0000", 2024-09-13T14:49:13.884 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8264000000000003e-05 2024-09-13T14:49:13.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.884 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.884 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.884 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.294865+0000", 2024-09-13T14:49:13.884 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.952e-06 2024-09-13T14:49:13.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.884 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.885 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.885 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552648+0000", 2024-09-13T14:49:13.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25778293299999999 2024-09-13T14:49:13.885 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.885 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.885 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:49:13.885 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552656+0000", 2024-09-13T14:49:13.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6699999999999994e-06 2024-09-13T14:49:13.885 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.885 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.885 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.885 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.742910+0000", 2024-09-13T14:49:13.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.19025483 2024-09-13T14:49:13.885 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.886 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.886 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:49:13.886 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.743051+0000", 2024-09-13T14:49:13.886 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014111899999999999 2024-09-13T14:49:13.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.886 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.886 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:49:13.886 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.743528+0000", 2024-09-13T14:49:13.886 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000476298 2024-09-13T14:49:13.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.886 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.886 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:49:13.886 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.747212+0000", 2024-09-13T14:49:13.887 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0036839500000000001 2024-09-13T14:49:13.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.887 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.887 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.747369+0000", 2024-09-13T14:49:13.887 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015746600000000001 2024-09-13T14:49:13.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.887 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.887 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.757512+0000", 2024-09-13T14:49:13.887 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010142744 2024-09-13T14:49:13.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.758753+0000", 2024-09-13T14:49:13.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012413840000000001 2024-09-13T14:49:13.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:49:13.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.758861+0000", 2024-09-13T14:49:13.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010781900000000001 2024-09-13T14:49:13.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:49:13.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.758963+0000", 2024-09-13T14:49:13.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101514 2024-09-13T14:49:13.889 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.889 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:49:13.889 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.889 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.889 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.889 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:198495 7.6 7:675a1080:::benchmark_data_smithi151_190906_object78034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1160)", 2024-09-13T14:49:13.889 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:47:40.150956+0000", 2024-09-13T14:49:13.889 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 93.590822974999995, 2024-09-13T14:49:13.889 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35556555699999998, 2024-09-13T14:49:13.889 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:49:13.889 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:49:13.889 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:13.889 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:49:13.890 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:49:13.890 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:49:13.890 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 198495 2024-09-13T14:49:13.890 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.890 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:49:13.890 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.890 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:49:13.890 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.150956+0000", 2024-09-13T14:49:13.890 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.890 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.890 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.890 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:49:13.890 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.150956+0000", 2024-09-13T14:49:13.891 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.891 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:49:13.891 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.150958+0000", 2024-09-13T14:49:13.891 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3120000000000001e-06 2024-09-13T14:49:13.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.891 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:49:13.891 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.150961+0000", 2024-09-13T14:49:13.891 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.399e-06 2024-09-13T14:49:13.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.892 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:49:13.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.150971+0000", 2024-09-13T14:49:13.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8449999999999998e-06 2024-09-13T14:49:13.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.892 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:49:13.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.150977+0000", 2024-09-13T14:49:13.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3350000000000003e-06 2024-09-13T14:49:13.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.892 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.152702+0000", 2024-09-13T14:49:13.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017246710000000001 2024-09-13T14:49:13.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.893 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:49:13.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.152853+0000", 2024-09-13T14:49:13.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015092799999999999 2024-09-13T14:49:13.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.893 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,5,7", 2024-09-13T14:49:13.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.153007+0000", 2024-09-13T14:49:13.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015429699999999999 2024-09-13T14:49:13.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.894 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.894 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.159050+0000", 2024-09-13T14:49:13.894 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0060432929999999999 2024-09-13T14:49:13.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.894 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.894 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.159214+0000", 2024-09-13T14:49:13.894 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000163337 2024-09-13T14:49:13.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.894 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:49:13.894 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.160210+0000", 2024-09-13T14:49:13.894 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00099668999999999995 2024-09-13T14:49:13.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.895 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.895 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.506285+0000", 2024-09-13T14:49:13.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34607454900000001 2024-09-13T14:49:13.895 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.895 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:49:13.895 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.506348+0000", 2024-09-13T14:49:13.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3131000000000003e-05 2024-09-13T14:49:13.895 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.895 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:49:13.895 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.506522+0000", 2024-09-13T14:49:13.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017377000000000001 2024-09-13T14:49:13.896 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.896 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:49:13.896 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.896 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:198496 7.f 7:f2f11d94:::benchmark_data_smithi151_190906_object78035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1160)", 2024-09-13T14:49:13.896 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:47:40.152885+0000", 2024-09-13T14:49:13.896 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 93.588894374999995, 2024-09-13T14:49:13.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.36139159500000001, 2024-09-13T14:49:13.896 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:49:13.896 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:49:13.896 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:13.896 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:49:13.896 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:49:13.897 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:49:13.897 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 198496 2024-09-13T14:49:13.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.897 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:49:13.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.897 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:49:13.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.152885+0000", 2024-09-13T14:49:13.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.897 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:49:13.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.152885+0000", 2024-09-13T14:49:13.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:49:13.898 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.152886+0000", 2024-09-13T14:49:13.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.311e-06 2024-09-13T14:49:13.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:49:13.898 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.152892+0000", 2024-09-13T14:49:13.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5829999999999997e-06 2024-09-13T14:49:13.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:49:13.899 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.152904+0000", 2024-09-13T14:49:13.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2289999999999999e-05 2024-09-13T14:49:13.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.899 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:49:13.899 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.152913+0000", 2024-09-13T14:49:13.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.507e-06 2024-09-13T14:49:13.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.899 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.899 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.153014+0000", 2024-09-13T14:49:13.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101382 2024-09-13T14:49:13.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:49:13.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.153144+0000", 2024-09-13T14:49:13.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012953800000000001 2024-09-13T14:49:13.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,4,7", 2024-09-13T14:49:13.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.153285+0000", 2024-09-13T14:49:13.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000141044 2024-09-13T14:49:13.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.164848+0000", 2024-09-13T14:49:13.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011563845999999999 2024-09-13T14:49:13.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.168990+0000", 2024-09-13T14:49:13.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0041417320000000004 2024-09-13T14:49:13.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:49:13.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.169125+0000", 2024-09-13T14:49:13.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013531200000000001 2024-09-13T14:49:13.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.514094+0000", 2024-09-13T14:49:13.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34496871200000001 2024-09-13T14:49:13.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:49:13.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.514153+0000", 2024-09-13T14:49:13.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8579000000000003e-05 2024-09-13T14:49:13.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:49:13.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.514277+0000", 2024-09-13T14:49:13.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012375900000000001 2024-09-13T14:49:13.902 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.902 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:49:13.903 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.903 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:198498 7.f 7:f602de94:::benchmark_data_smithi151_190906_object78037:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1160)", 2024-09-13T14:49:13.903 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:47:40.152936+0000", 2024-09-13T14:49:13.903 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 93.588843466, 2024-09-13T14:49:13.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.36154148600000002, 2024-09-13T14:49:13.903 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:49:13.903 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:49:13.903 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:13.903 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:49:13.903 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:49:13.903 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:49:13.904 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 198498 2024-09-13T14:49:13.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.904 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:49:13.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:49:13.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.152936+0000", 2024-09-13T14:49:13.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:49:13.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.152936+0000", 2024-09-13T14:49:13.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:49:13.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.152936+0000", 2024-09-13T14:49:13.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8100000000000003e-07 2024-09-13T14:49:13.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:49:13.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.152940+0000", 2024-09-13T14:49:13.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1820000000000002e-06 2024-09-13T14:49:13.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:49:13.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.152947+0000", 2024-09-13T14:49:13.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7959999999999996e-06 2024-09-13T14:49:13.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:49:13.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.152953+0000", 2024-09-13T14:49:13.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7939999999999999e-06 2024-09-13T14:49:13.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.160725+0000", 2024-09-13T14:49:13.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0077716790000000001 2024-09-13T14:49:13.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:49:13.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.160853+0000", 2024-09-13T14:49:13.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012778400000000001 2024-09-13T14:49:13.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,4,7", 2024-09-13T14:49:13.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.161209+0000", 2024-09-13T14:49:13.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00035686799999999999 2024-09-13T14:49:13.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.164933+0000", 2024-09-13T14:49:13.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037232319999999999 2024-09-13T14:49:13.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.169033+0000", 2024-09-13T14:49:13.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0041006230000000003 2024-09-13T14:49:13.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:49:13.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.169131+0000", 2024-09-13T14:49:13.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7410000000000004e-05 2024-09-13T14:49:13.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.514322+0000", 2024-09-13T14:49:13.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34519118399999998 2024-09-13T14:49:13.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:49:13.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.514370+0000", 2024-09-13T14:49:13.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7799000000000001e-05 2024-09-13T14:49:13.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:49:13.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.514477+0000", 2024-09-13T14:49:13.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010755399999999999 2024-09-13T14:49:13.909 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.909 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:49:13.910 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.910 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:198501 7.f 7:f83f8ca0:::benchmark_data_smithi151_190906_object78040:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1160)", 2024-09-13T14:49:13.910 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:47:40.156238+0000", 2024-09-13T14:49:13.910 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 93.585541239999998, 2024-09-13T14:49:13.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.36201745699999999, 2024-09-13T14:49:13.910 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:49:13.910 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:49:13.910 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:13.910 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:49:13.910 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:49:13.910 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:49:13.911 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 198501 2024-09-13T14:49:13.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.911 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:49:13.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:49:13.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.156238+0000", 2024-09-13T14:49:13.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:49:13.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.156238+0000", 2024-09-13T14:49:13.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:49:13.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:49:13.912 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.156239+0000", 2024-09-13T14:49:13.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3519999999999999e-06 2024-09-13T14:49:13.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:49:13.912 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.156243+0000", 2024-09-13T14:49:13.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.743e-06 2024-09-13T14:49:13.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:49:13.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.156255+0000", 2024-09-13T14:49:13.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2094e-05 2024-09-13T14:49:13.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:49:13.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.156263+0000", 2024-09-13T14:49:13.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5349999999999999e-06 2024-09-13T14:49:13.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:49:13.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.164974+0000", 2024-09-13T14:49:13.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0087109119999999995 2024-09-13T14:49:13.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:49:13.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.165113+0000", 2024-09-13T14:49:13.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013891700000000001 2024-09-13T14:49:13.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,4,7", 2024-09-13T14:49:13.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.165241+0000", 2024-09-13T14:49:13.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012797000000000001 2024-09-13T14:49:13.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.169071+0000", 2024-09-13T14:49:13.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0038301310000000001 2024-09-13T14:49:13.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.169111+0000", 2024-09-13T14:49:13.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0642999999999997e-05 2024-09-13T14:49:13.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:49:13.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.169136+0000", 2024-09-13T14:49:13.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4247999999999999e-05 2024-09-13T14:49:13.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:13.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.514530+0000", 2024-09-13T14:49:13.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34539413899999999 2024-09-13T14:49:13.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:49:13.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.518094+0000", 2024-09-13T14:49:13.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003564546 2024-09-13T14:49:13.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:49:13.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:49:13.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:49:13.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.518255+0000", 2024-09-13T14:49:13.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016122700000000001 2024-09-13T14:49:13.917 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.917 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:49:13.917 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.917 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:49:13.917 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:49:13.917 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:49:13.921 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:13.942 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:13.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:13.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:14.044 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:14.065 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:14.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:14.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:14.167 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:14.193 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:14.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:14.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:14.295 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:14.318 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:14.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:14.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:14.421 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:14.446 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:14.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:14.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:14.548 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:14.575 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:14.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:14.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:14.677 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:14.702 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:14.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:14.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:14.805 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:14.830 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:14.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:14.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:14.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:49:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:49:14] "GET /metrics HTTP/1.1" 200 36647 "" "Prometheus/2.51.0" 2024-09-13T14:49:14.932 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:14.958 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:14.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:14.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:15.060 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:15.086 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:15.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:15.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:15.108 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:49:15.187 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:15.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:15.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:15.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:15.312 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:15.333 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:15.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:15.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:15.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:15 smithi184 ceph-mon[92527]: pgmap v1666: 103 pgs: 103 active+clean; 608 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:15.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:15 smithi151 ceph-mon[104450]: pgmap v1666: 103 pgs: 103 active+clean; 608 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:15.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:15 smithi151 ceph-mon[108966]: pgmap v1666: 103 pgs: 103 active+clean; 608 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:15.435 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:15.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:15.458 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:15.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:15.559 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:15.584 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:15.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:15.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:15.686 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:15.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:15.713 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:15.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:15.814 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:15.839 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:15.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:15.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:15.941 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:15.966 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:15.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:15.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:16.068 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:16.093 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:16.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:16.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:16.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:16.221 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:16.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:16.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:16.324 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:16.350 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:16.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:16.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:16.452 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:16.477 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:16.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:16.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:16.579 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:16.605 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:16.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:16.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:16.707 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:16.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:16.734 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:16.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:16.835 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:16.859 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:16.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:16.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:16.961 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:16.986 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:16.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:16.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:17.087 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:17.113 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:17.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:17.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:17.215 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:17.239 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:17.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:17.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:17.341 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:17.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:17 smithi184 ceph-mon[92527]: pgmap v1667: 103 pgs: 103 active+clean; 608 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:17.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:49:17.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:17.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:49:17.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:17 smithi151 ceph-mon[104450]: pgmap v1667: 103 pgs: 103 active+clean; 608 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:17.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:49:17.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:17.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:49:17.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:17 smithi151 ceph-mon[108966]: pgmap v1667: 103 pgs: 103 active+clean; 608 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:17.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:49:17.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:17.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:49:17.369 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:17.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:17.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:17.471 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:17.493 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:17.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:17.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:17.595 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:17.617 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:17.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:17.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:17.719 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:17.740 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:17.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:17.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:17.842 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:17.864 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:17.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:17.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:17.965 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:17.987 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:17.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:17.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:18.088 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:18.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:18.110 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:18.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:18.211 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:18.232 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:18.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:18.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:18.334 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:18.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:18.356 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:18.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:18.457 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:18.479 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:18.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:18.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:18.581 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:18.603 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:18.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:18.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:18.704 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T14:49:18.705 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:18.725 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:18.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:18.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:18.810 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:18.827 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:18.853 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:18.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:18.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:18.955 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:18.977 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:18.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:18.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:19.079 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:19.102 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:19.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:19.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:19.204 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:19.227 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:19.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:19.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:19.243 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:19 smithi151 ceph-mon[104450]: pgmap v1668: 103 pgs: 103 active+clean; 608 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:19.244 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:19 smithi151 ceph-mon[108966]: pgmap v1668: 103 pgs: 103 active+clean; 608 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:19.328 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:19.351 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:19.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:19.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:19.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:19 smithi184 ceph-mon[92527]: pgmap v1668: 103 pgs: 103 active+clean; 608 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:19.453 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:19.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:19.476 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:19.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:19.577 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:19.598 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:19.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:19.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:19.701 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:19.725 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:19.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:19.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:19.827 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:19.852 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:19.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:19.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:19.954 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:19.976 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:19.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:19.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:20.078 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:20.102 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:20.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:20.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:20.204 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:20.229 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:20.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:20.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:20.292 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:20.331 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:20.355 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:20.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:20.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:20.457 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:20.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:20.482 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:20.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:20.583 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:20.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:49:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:49:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:49:20.608 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:20.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:20.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:20.710 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:20.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:20.735 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:20.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:20.836 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:20.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:20.860 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:20.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:20.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:20.988 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:20.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:20.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:21.090 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:21.092 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_3' created 2024-09-13T14:49:21.092 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-13T14:49:21.117 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:21.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:21.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:21.219 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:21.244 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:21.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:21.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:21.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:21.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:21 smithi151 ceph-mon[104450]: pgmap v1669: 103 pgs: 103 active+clean; 608 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:21.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:21 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/588505509' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:49:21.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:21 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2229298976' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-13T14:49:21.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:21 smithi151 ceph-mon[108966]: pgmap v1669: 103 pgs: 103 active+clean; 608 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:21.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:21 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/588505509' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:49:21.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:21 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2229298976' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-13T14:49:21.369 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:21.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:21.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:21.471 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:21.497 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:21.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:21.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:21.599 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:21.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:21 smithi184 ceph-mon[92527]: pgmap v1669: 103 pgs: 103 active+clean; 608 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:21.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:21 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/588505509' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:49:21.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:21 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2229298976' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-13T14:49:21.624 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:21.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:21.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:21.726 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:21.749 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:21.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:21.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:21.852 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:21.874 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:21.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:21.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:21.976 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:22.002 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:22.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:22.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:22.103 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:22.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:22.134 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:22.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:22.236 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:22.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:22.262 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:22.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:22.327 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:22 smithi151 ceph-mon[104450]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:22.327 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:22 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/588505509' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:49:22.327 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:22 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2229298976' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-13T14:49:22.328 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:22 smithi151 ceph-mon[104450]: osdmap e1167: 8 total, 8 up, 8 in 2024-09-13T14:49:22.328 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:22 smithi151 ceph-mon[108966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:22.328 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:22 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/588505509' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:49:22.328 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:22 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2229298976' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-13T14:49:22.328 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:22 smithi151 ceph-mon[108966]: osdmap e1167: 8 total, 8 up, 8 in 2024-09-13T14:49:22.363 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:22.389 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:22.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:22.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:22.491 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:22.518 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:22.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:22.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:22.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:22 smithi184 ceph-mon[92527]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:22.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:22 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/588505509' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:49:22.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:22 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2229298976' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-13T14:49:22.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:22 smithi184 ceph-mon[92527]: osdmap e1167: 8 total, 8 up, 8 in 2024-09-13T14:49:22.619 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:22.641 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:22.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:22.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:22.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:22.764 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:22.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:22.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:22.848 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-09-13T14:49:22.866 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:22.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:22.893 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:22.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:22.994 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:23.016 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:23.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:23.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:23.118 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:23.144 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:23.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:23.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:23.246 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:23.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:23.269 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:23.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:23.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:23 smithi151 ceph-mon[104450]: pgmap v1671: 119 pgs: 16 unknown, 103 active+clean; 608 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:23.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:23 smithi151 ceph-mon[104450]: 1.f scrub starts 2024-09-13T14:49:23.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:23 smithi151 ceph-mon[104450]: 1.f scrub ok 2024-09-13T14:49:23.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:23 smithi151 ceph-mon[104450]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:49:23.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:23 smithi151 ceph-mon[104450]: osdmap e1168: 8 total, 8 up, 8 in 2024-09-13T14:49:23.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:23 smithi151 ceph-mon[108966]: pgmap v1671: 119 pgs: 16 unknown, 103 active+clean; 608 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:23.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:23 smithi151 ceph-mon[108966]: 1.f scrub starts 2024-09-13T14:49:23.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:23 smithi151 ceph-mon[108966]: 1.f scrub ok 2024-09-13T14:49:23.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:23 smithi151 ceph-mon[108966]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:49:23.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:23 smithi151 ceph-mon[108966]: osdmap e1168: 8 total, 8 up, 8 in 2024-09-13T14:49:23.371 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:23.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:23.393 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:23.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:23.494 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:23.516 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:23.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:23.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:23.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:23 smithi184 ceph-mon[92527]: pgmap v1671: 119 pgs: 16 unknown, 103 active+clean; 608 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:23.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:23 smithi184 ceph-mon[92527]: 1.f scrub starts 2024-09-13T14:49:23.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:23 smithi184 ceph-mon[92527]: 1.f scrub ok 2024-09-13T14:49:23.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:23 smithi184 ceph-mon[92527]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:49:23.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:23 smithi184 ceph-mon[92527]: osdmap e1168: 8 total, 8 up, 8 in 2024-09-13T14:49:23.617 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:23.643 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:23.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:23.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:23.745 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:23.771 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:23.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:23.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:23.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:23.894 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:23.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:23.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:23.996 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:23.999 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:24.022 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:24.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:24.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:24.123 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:24.150 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:24.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:24.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:24.252 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:24.269 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:24 smithi151 ceph-mon[104450]: osdmap e1169: 8 total, 8 up, 8 in 2024-09-13T14:49:24.270 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:24 smithi151 ceph-mon[108966]: osdmap e1169: 8 total, 8 up, 8 in 2024-09-13T14:49:24.277 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:24.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:24.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:24.378 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:24.403 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:49:24.404 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T14:49:24.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:24.407 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:24.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:24.508 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:24.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:24.532 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:24.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:24.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:24 smithi184 ceph-mon[92527]: osdmap e1169: 8 total, 8 up, 8 in 2024-09-13T14:49:24.633 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:24.658 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:24.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:24.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:24.760 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:24.784 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:24.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:24.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:24.811 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:49:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:49:24] "GET /metrics HTTP/1.1" 200 34298 "" "Prometheus/2.51.0" 2024-09-13T14:49:24.886 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:24.907 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:24.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:24.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:25.009 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:25.031 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:25.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:25.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:25.133 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:25.146 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1674,"stamp":"2024-09-13T14:49:23.536166+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":327718,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3158,"ondisk_log_size":3158,"up":339,"acting":339,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":310,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3099460,"kb_used_data":513808,"kb_used_omap":149,"kb_used_meta":2585386,"kb_avail":746861756,"statfs":{"total":767960285184,"available":764786438144,"internally_reserved":0,"allocated":526139392,"data_stored":497424581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":153401,"internal_metadata":2647435463},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-13,"num_read_kb":-34,"num_write":-7,"num_write_kb":-74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":-26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001232"},"pg_stats":[{"pgid":"1.1d","version":"261'100","reported_seq":107,"reported_epoch":1168,"state":"active+clean","last_fresh":"2024-09-13T14:49:22.115707+0000","last_change":"2024-09-13T14:45:19.807356+0000","last_active":"2024-09-13T14:49:22.115707+0000","last_peered":"2024-09-13T14:49:22.115707+0000","last_clean":"2024-09-13T14:49:22.115707+0000","last_became_active":"2024-09-13T14:45:19.807081+0000","last_became_peered":"2024-09-13T14:45:19.807081+0000","last_unstale":"2024-09-13T14:49:22.115707+0000","last_undegraded":"2024-09-13T14:49:22.115707+0000","last_fullsized":"2024-09-13T14:49:22.115707+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:17.988345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:17.988345+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:41:05.216335+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":1708,"reported_epoch":1166,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.718552+0000","last_change":"2024-09-13T14:29:42.541263+0000","last_active":"2024-09-13T14:49:08.718552+0000","last_peered":"2024-09-13T14:49:08.718552+0000","last_clean":"2024-09-13T14:49:08.718552+0000","last_became_active":"2024-09-13T14:29:42.540873+0000","last_became_peered":"2024-09-13T14:29:42.540873+0000","last_unstale":"2024-09-13T14:49:08.718552+0000","last_undegraded":"2024-09-13T14:49:08.718552+0000","last_fullsized":"2024-09-13T14:49:08.718552+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:20.611206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1128,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.100819+0000","last_change":"2024-09-13T14:43:07.516715+0000","last_active":"2024-09-13T14:49:21.100819+0000","last_peered":"2024-09-13T14:49:21.100819+0000","last_clean":"2024-09-13T14:49:21.100819+0000","last_became_active":"2024-09-13T14:43:07.516403+0000","last_became_peered":"2024-09-13T14:43:07.516403+0000","last_unstale":"2024-09-13T14:49:21.100819+0000","last_undegraded":"2024-09-13T14:49:21.100819+0000","last_fullsized":"2024-09-13T14:49:21.100819+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:56.469690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":47,"reported_epoch":1166,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.203981+0000","last_change":"2024-09-13T14:45:19.810203+0000","last_active":"2024-09-13T14:49:08.203981+0000","last_peered":"2024-09-13T14:49:08.203981+0000","last_clean":"2024-09-13T14:49:08.203981+0000","last_became_active":"2024-09-13T14:45:19.809693+0000","last_became_peered":"2024-09-13T14:45:19.809693+0000","last_unstale":"2024-09-13T14:49:08.203981+0000","last_undegraded":"2024-09-13T14:49:08.203981+0000","last_fullsized":"2024-09-13T14:49:08.203981+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:20.219673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":47,"reported_epoch":1166,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.203375+0000","last_change":"2024-09-13T14:45:19.807991+0000","last_active":"2024-09-13T14:49:08.203375+0000","last_peered":"2024-09-13T14:49:08.203375+0000","last_clean":"2024-09-13T14:49:08.203375+0000","last_became_active":"2024-09-13T14:45:19.807662+0000","last_became_peered":"2024-09-13T14:45:19.807662+0000","last_unstale":"2024-09-13T14:49:08.203375+0000","last_undegraded":"2024-09-13T14:49:08.203375+0000","last_fullsized":"2024-09-13T14:49:08.203375+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:29.306108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"261'100","reported_seq":1239,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.095298+0000","last_change":"2024-09-13T14:44:54.259562+0000","last_active":"2024-09-13T14:49:21.095298+0000","last_peered":"2024-09-13T14:49:21.095298+0000","last_clean":"2024-09-13T14:49:21.095298+0000","last_became_active":"2024-09-13T14:43:14.563717+0000","last_became_peered":"2024-09-13T14:43:14.563717+0000","last_unstale":"2024-09-13T14:49:21.095298+0000","last_undegraded":"2024-09-13T14:49:21.095298+0000","last_fullsized":"2024-09-13T14:49:21.095298+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_clean_scrub_stamp":"2024-09-13T14:44:54.259482+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:43:52.729109+0000","scrub_duration":0.015265569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":1096,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.099046+0000","last_change":"2024-09-13T14:19:25.873514+0000","last_active":"2024-09-13T14:49:21.099046+0000","last_peered":"2024-09-13T14:49:21.099046+0000","last_clean":"2024-09-13T14:49:21.099046+0000","last_became_active":"2024-09-13T14:19:25.873170+0000","last_became_peered":"2024-09-13T14:19:25.873170+0000","last_unstale":"2024-09-13T14:49:21.099046+0000","last_undegraded":"2024-09-13T14:49:21.099046+0000","last_fullsized":"2024-09-13T14:49:21.099046+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:48.298530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1125,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.102052+0000","last_change":"2024-09-13T14:30:15.580011+0000","last_active":"2024-09-13T14:49:21.102052+0000","last_peered":"2024-09-13T14:49:21.102052+0000","last_clean":"2024-09-13T14:49:21.102052+0000","last_became_active":"2024-09-13T14:30:15.579610+0000","last_became_peered":"2024-09-13T14:30:15.579610+0000","last_unstale":"2024-09-13T14:49:21.102052+0000","last_undegraded":"2024-09-13T14:49:21.102052+0000","last_fullsized":"2024-09-13T14:49:21.102052+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:11:02.733771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"261'100","reported_seq":64,"reported_epoch":1166,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.204142+0000","last_change":"2024-09-13T14:45:35.947332+0000","last_active":"2024-09-13T14:49:08.204142+0000","last_peered":"2024-09-13T14:49:08.204142+0000","last_clean":"2024-09-13T14:49:08.204142+0000","last_became_active":"2024-09-13T14:45:19.810775+0000","last_became_peered":"2024-09-13T14:45:19.810775+0000","last_unstale":"2024-09-13T14:49:08.204142+0000","last_undegraded":"2024-09-13T14:49:08.204142+0000","last_fullsized":"2024-09-13T14:49:08.204142+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:35.947265+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:43:41.569795+0000","last_clean_scrub_stamp":"2024-09-13T14:45:35.947265+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:00.606370+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":1141,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.099005+0000","last_change":"2024-09-13T14:43:07.536116+0000","last_active":"2024-09-13T14:49:21.099005+0000","last_peered":"2024-09-13T14:49:21.099005+0000","last_clean":"2024-09-13T14:49:21.099005+0000","last_became_active":"2024-09-13T14:43:07.524112+0000","last_became_peered":"2024-09-13T14:43:07.524112+0000","last_unstale":"2024-09-13T14:49:21.099005+0000","last_undegraded":"2024-09-13T14:49:21.099005+0000","last_fullsized":"2024-09-13T14:49:21.099005+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:36.947060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1138,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.101366+0000","last_change":"2024-09-13T14:43:07.530236+0000","last_active":"2024-09-13T14:49:21.101366+0000","last_peered":"2024-09-13T14:49:21.101366+0000","last_clean":"2024-09-13T14:49:21.101366+0000","last_became_active":"2024-09-13T14:43:07.529967+0000","last_became_peered":"2024-09-13T14:43:07.529967+0000","last_unstale":"2024-09-13T14:49:21.101366+0000","last_undegraded":"2024-09-13T14:49:21.101366+0000","last_fullsized":"2024-09-13T14:49:21.101366+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:27.276550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":1693,"reported_epoch":1166,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.711616+0000","last_change":"2024-09-13T14:28:35.612469+0000","last_active":"2024-09-13T14:49:08.711616+0000","last_peered":"2024-09-13T14:49:08.711616+0000","last_clean":"2024-09-13T14:49:08.711616+0000","last_became_active":"2024-09-13T14:28:35.612093+0000","last_became_peered":"2024-09-13T14:28:35.612093+0000","last_unstale":"2024-09-13T14:49:08.711616+0000","last_undegraded":"2024-09-13T14:49:08.711616+0000","last_fullsized":"2024-09-13T14:49:08.711616+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:51.022663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1719,"reported_epoch":1168,"state":"active+clean","last_fresh":"2024-09-13T14:49:22.109263+0000","last_change":"2024-09-13T14:45:19.808811+0000","last_active":"2024-09-13T14:49:22.109263+0000","last_peered":"2024-09-13T14:49:22.109263+0000","last_clean":"2024-09-13T14:49:22.109263+0000","last_became_active":"2024-09-13T14:45:19.808471+0000","last_became_peered":"2024-09-13T14:45:19.808471+0000","last_unstale":"2024-09-13T14:49:22.109263+0000","last_undegraded":"2024-09-13T14:49:22.109263+0000","last_fullsized":"2024-09-13T14:49:22.109263+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:53.638161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"261'100","reported_seq":47,"reported_epoch":1166,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.203466+0000","last_change":"2024-09-13T14:45:19.813412+0000","last_active":"2024-09-13T14:49:08.203466+0000","last_peered":"2024-09-13T14:49:08.203466+0000","last_clean":"2024-09-13T14:49:08.203466+0000","last_became_active":"2024-09-13T14:45:19.812585+0000","last_became_peered":"2024-09-13T14:45:19.812585+0000","last_unstale":"2024-09-13T14:49:08.203466+0000","last_undegraded":"2024-09-13T14:49:08.203466+0000","last_fullsized":"2024-09-13T14:49:08.203466+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.614340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.614340+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:32:07.434109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":1147,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.102177+0000","last_change":"2024-09-13T14:43:07.522950+0000","last_active":"2024-09-13T14:49:21.102177+0000","last_peered":"2024-09-13T14:49:21.102177+0000","last_clean":"2024-09-13T14:49:21.102177+0000","last_became_active":"2024-09-13T14:43:07.522566+0000","last_became_peered":"2024-09-13T14:43:07.522566+0000","last_unstale":"2024-09-13T14:49:21.102177+0000","last_undegraded":"2024-09-13T14:49:21.102177+0000","last_fullsized":"2024-09-13T14:49:21.102177+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:03.201478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":47,"reported_epoch":1166,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.203885+0000","last_change":"2024-09-13T14:45:19.807231+0000","last_active":"2024-09-13T14:49:08.203885+0000","last_peered":"2024-09-13T14:49:08.203885+0000","last_clean":"2024-09-13T14:49:08.203885+0000","last_became_active":"2024-09-13T14:45:19.806936+0000","last_became_peered":"2024-09-13T14:45:19.806936+0000","last_unstale":"2024-09-13T14:49:08.203885+0000","last_undegraded":"2024-09-13T14:49:08.203885+0000","last_fullsized":"2024-09-13T14:49:08.203885+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:04:26.723153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.18","version":"261'100","reported_seq":117,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.098728+0000","last_change":"2024-09-13T14:44:41.685537+0000","last_active":"2024-09-13T14:49:21.098728+0000","last_peered":"2024-09-13T14:49:21.098728+0000","last_clean":"2024-09-13T14:49:21.098728+0000","last_became_active":"2024-09-13T14:43:14.558502+0000","last_became_peered":"2024-09-13T14:43:14.558502+0000","last_unstale":"2024-09-13T14:49:21.098728+0000","last_undegraded":"2024-09-13T14:49:21.098728+0000","last_fullsized":"2024-09-13T14:49:21.098728+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1134,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:41.685124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:41.685124+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:37:58.412497+0000","scrub_duration":0.0075205970000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":100,"reported_epoch":1168,"state":"active+clean","last_fresh":"2024-09-13T14:49:22.115843+0000","last_change":"2024-09-13T14:43:07.516845+0000","last_active":"2024-09-13T14:49:22.115843+0000","last_peered":"2024-09-13T14:49:22.115843+0000","last_clean":"2024-09-13T14:49:22.115843+0000","last_became_active":"2024-09-13T14:43:07.514974+0000","last_became_peered":"2024-09-13T14:43:07.514974+0000","last_unstale":"2024-09-13T14:49:22.115843+0000","last_undegraded":"2024-09-13T14:49:22.115843+0000","last_fullsized":"2024-09-13T14:49:22.115843+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:55.419483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1731,"reported_epoch":1166,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.712414+0000","last_change":"2024-09-13T14:45:19.815341+0000","last_active":"2024-09-13T14:49:08.712414+0000","last_peered":"2024-09-13T14:49:08.712414+0000","last_clean":"2024-09-13T14:49:08.712414+0000","last_became_active":"2024-09-13T14:45:19.810262+0000","last_became_peered":"2024-09-13T14:45:19.810262+0000","last_unstale":"2024-09-13T14:49:08.712414+0000","last_undegraded":"2024-09-13T14:49:08.712414+0000","last_fullsized":"2024-09-13T14:49:08.712414+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:39.847653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1b","version":"261'100","reported_seq":59,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.098833+0000","last_change":"2024-09-13T14:44:52.803330+0000","last_active":"2024-09-13T14:49:21.098833+0000","last_peered":"2024-09-13T14:49:21.098833+0000","last_clean":"2024-09-13T14:49:21.098833+0000","last_became_active":"2024-09-13T14:43:14.556712+0000","last_became_peered":"2024-09-13T14:43:14.556712+0000","last_unstale":"2024-09-13T14:49:21.098833+0000","last_undegraded":"2024-09-13T14:49:21.098833+0000","last_fullsized":"2024-09-13T14:49:21.098833+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:52.803242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:52.803242+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:06.658301+0000","scrub_duration":0.0019769660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":47,"reported_epoch":1166,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.203146+0000","last_change":"2024-09-13T14:45:19.806623+0000","last_active":"2024-09-13T14:49:08.203146+0000","last_peered":"2024-09-13T14:49:08.203146+0000","last_clean":"2024-09-13T14:49:08.203146+0000","last_became_active":"2024-09-13T14:45:19.806219+0000","last_became_peered":"2024-09-13T14:45:19.806219+0000","last_unstale":"2024-09-13T14:49:08.203146+0000","last_undegraded":"2024-09-13T14:49:08.203146+0000","last_fullsized":"2024-09-13T14:49:08.203146+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:34:41.601201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1121,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.099068+0000","last_change":"2024-09-13T14:43:32.637751+0000","last_active":"2024-09-13T14:49:21.099068+0000","last_peered":"2024-09-13T14:49:21.099068+0000","last_clean":"2024-09-13T14:49:21.099068+0000","last_became_active":"2024-09-13T14:43:32.636901+0000","last_became_peered":"2024-09-13T14:43:32.636901+0000","last_unstale":"2024-09-13T14:49:21.099068+0000","last_undegraded":"2024-09-13T14:49:21.099068+0000","last_fullsized":"2024-09-13T14:49:21.099068+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:21.942419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1a","version":"261'100","reported_seq":59,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.101781+0000","last_change":"2024-09-13T14:43:41.910041+0000","last_active":"2024-09-13T14:49:21.101781+0000","last_peered":"2024-09-13T14:49:21.101781+0000","last_clean":"2024-09-13T14:49:21.101781+0000","last_became_active":"2024-09-13T14:43:14.555083+0000","last_became_peered":"2024-09-13T14:43:14.555083+0000","last_unstale":"2024-09-13T14:49:21.101781+0000","last_undegraded":"2024-09-13T14:49:21.101781+0000","last_fullsized":"2024-09-13T14:49:21.101781+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:41.909988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:41.909988+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:42:19.954838+0000","scrub_duration":0.0012891619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":47,"reported_epoch":1166,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.203256+0000","last_change":"2024-09-13T14:45:19.812071+0000","last_active":"2024-09-13T14:49:08.203256+0000","last_peered":"2024-09-13T14:49:08.203256+0000","last_clean":"2024-09-13T14:49:08.203256+0000","last_became_active":"2024-09-13T14:45:19.811336+0000","last_became_peered":"2024-09-13T14:45:19.811336+0000","last_unstale":"2024-09-13T14:49:08.203256+0000","last_undegraded":"2024-09-13T14:49:08.203256+0000","last_fullsized":"2024-09-13T14:49:08.203256+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:40:29.814608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1748,"reported_epoch":1166,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.718442+0000","last_change":"2024-09-13T14:45:19.810478+0000","last_active":"2024-09-13T14:49:08.718442+0000","last_peered":"2024-09-13T14:49:08.718442+0000","last_clean":"2024-09-13T14:49:08.718442+0000","last_became_active":"2024-09-13T14:45:19.808278+0000","last_became_peered":"2024-09-13T14:45:19.808278+0000","last_unstale":"2024-09-13T14:49:08.718442+0000","last_undegraded":"2024-09-13T14:49:08.718442+0000","last_fullsized":"2024-09-13T14:49:08.718442+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:12.211089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"261'100","reported_seq":59,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.095648+0000","last_change":"2024-09-13T14:44:38.214961+0000","last_active":"2024-09-13T14:49:21.095648+0000","last_peered":"2024-09-13T14:49:21.095648+0000","last_clean":"2024-09-13T14:49:21.095648+0000","last_became_active":"2024-09-13T14:43:14.557883+0000","last_became_peered":"2024-09-13T14:43:14.557883+0000","last_unstale":"2024-09-13T14:49:21.095648+0000","last_undegraded":"2024-09-13T14:49:21.095648+0000","last_fullsized":"2024-09-13T14:49:21.095648+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:38.214865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:38.214865+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:49:36.013783+0000","scrub_duration":0.0016389639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1619,"reported_epoch":1166,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.203624+0000","last_change":"2024-09-13T14:30:15.576921+0000","last_active":"2024-09-13T14:49:08.203624+0000","last_peered":"2024-09-13T14:49:08.203624+0000","last_clean":"2024-09-13T14:49:08.203624+0000","last_became_active":"2024-09-13T14:30:15.576607+0000","last_became_peered":"2024-09-13T14:30:15.576607+0000","last_unstale":"2024-09-13T14:49:08.203624+0000","last_undegraded":"2024-09-13T14:49:08.203624+0000","last_fullsized":"2024-09-13T14:49:08.203624+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:35.162460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1131,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.098876+0000","last_change":"2024-09-13T14:43:07.518801+0000","last_active":"2024-09-13T14:49:21.098876+0000","last_peered":"2024-09-13T14:49:21.098876+0000","last_clean":"2024-09-13T14:49:21.098876+0000","last_became_active":"2024-09-13T14:43:07.517610+0000","last_became_peered":"2024-09-13T14:43:07.517610+0000","last_unstale":"2024-09-13T14:49:21.098876+0000","last_undegraded":"2024-09-13T14:49:21.098876+0000","last_fullsized":"2024-09-13T14:49:21.098876+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:51.632254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"261'100","reported_seq":100,"reported_epoch":1168,"state":"active+clean","last_fresh":"2024-09-13T14:49:22.109096+0000","last_change":"2024-09-13T14:43:15.037565+0000","last_active":"2024-09-13T14:49:22.109096+0000","last_peered":"2024-09-13T14:49:22.109096+0000","last_clean":"2024-09-13T14:49:22.109096+0000","last_became_active":"2024-09-13T14:43:14.554183+0000","last_became_peered":"2024-09-13T14:43:14.554183+0000","last_unstale":"2024-09-13T14:49:22.109096+0000","last_undegraded":"2024-09-13T14:49:22.109096+0000","last_fullsized":"2024-09-13T14:49:22.109096+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.037517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.037517+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:01:35.137195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":1125,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.095516+0000","last_change":"2024-09-13T14:30:15.579260+0000","last_active":"2024-09-13T14:49:21.095516+0000","last_peered":"2024-09-13T14:49:21.095516+0000","last_clean":"2024-09-13T14:49:21.095516+0000","last_became_active":"2024-09-13T14:30:15.578897+0000","last_became_peered":"2024-09-13T14:30:15.578897+0000","last_unstale":"2024-09-13T14:49:21.095516+0000","last_undegraded":"2024-09-13T14:49:21.095516+0000","last_fullsized":"2024-09-13T14:49:21.095516+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:56.737849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":100,"reported_epoch":1168,"state":"active+clean","last_fresh":"2024-09-13T14:49:22.115844+0000","last_change":"2024-09-13T14:43:07.521576+0000","last_active":"2024-09-13T14:49:22.115844+0000","last_peered":"2024-09-13T14:49:22.115844+0000","last_clean":"2024-09-13T14:49:22.115844+0000","last_became_active":"2024-09-13T14:43:07.520478+0000","last_became_peered":"2024-09-13T14:43:07.520478+0000","last_unstale":"2024-09-13T14:49:22.115844+0000","last_undegraded":"2024-09-13T14:49:22.115844+0000","last_fullsized":"2024-09-13T14:49:22.115844+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:53.410135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"261'100","reported_seq":385,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.098888+0000","last_change":"2024-09-13T14:44:49.775941+0000","last_active":"2024-09-13T14:49:21.098888+0000","last_peered":"2024-09-13T14:49:21.098888+0000","last_clean":"2024-09-13T14:49:21.098888+0000","last_became_active":"2024-09-13T14:43:14.560297+0000","last_became_peered":"2024-09-13T14:43:14.560297+0000","last_unstale":"2024-09-13T14:49:21.098888+0000","last_undegraded":"2024-09-13T14:49:21.098888+0000","last_fullsized":"2024-09-13T14:49:21.098888+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:49.775844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:49.775844+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:15:53.138661+0000","scrub_duration":0.012350274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":1114,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.101946+0000","last_change":"2024-09-13T14:19:25.870605+0000","last_active":"2024-09-13T14:49:21.101946+0000","last_peered":"2024-09-13T14:49:21.101946+0000","last_clean":"2024-09-13T14:49:21.101946+0000","last_became_active":"2024-09-13T14:19:25.870282+0000","last_became_peered":"2024-09-13T14:19:25.870282+0000","last_unstale":"2024-09-13T14:49:21.101946+0000","last_undegraded":"2024-09-13T14:49:21.101946+0000","last_fullsized":"2024-09-13T14:49:21.101946+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:52.514915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1128,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.101440+0000","last_change":"2024-09-13T14:43:07.526476+0000","last_active":"2024-09-13T14:49:21.101440+0000","last_peered":"2024-09-13T14:49:21.101440+0000","last_clean":"2024-09-13T14:49:21.101440+0000","last_became_active":"2024-09-13T14:43:07.525802+0000","last_became_peered":"2024-09-13T14:43:07.525802+0000","last_unstale":"2024-09-13T14:49:21.101440+0000","last_undegraded":"2024-09-13T14:49:21.101440+0000","last_fullsized":"2024-09-13T14:49:21.101440+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:23.154726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"261'100","reported_seq":107,"reported_epoch":1168,"state":"active+clean","last_fresh":"2024-09-13T14:49:22.115716+0000","last_change":"2024-09-13T14:45:19.807461+0000","last_active":"2024-09-13T14:49:22.115716+0000","last_peered":"2024-09-13T14:49:22.115716+0000","last_clean":"2024-09-13T14:49:22.115716+0000","last_became_active":"2024-09-13T14:45:19.807117+0000","last_became_peered":"2024-09-13T14:45:19.807117+0000","last_unstale":"2024-09-13T14:49:22.115716+0000","last_undegraded":"2024-09-13T14:49:22.115716+0000","last_fullsized":"2024-09-13T14:49:22.115716+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.955197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.955197+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:03.531555+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":1168,"state":"active+clean","last_fresh":"2024-09-13T14:49:22.120980+0000","last_change":"2024-09-13T14:49:22.120980+0000","last_active":"2024-09-13T14:49:22.120980+0000","last_peered":"2024-09-13T14:49:22.120980+0000","last_clean":"2024-09-13T14:49:22.120980+0000","last_became_active":"2024-09-13T14:49:22.120684+0000","last_became_peered":"2024-09-13T14:49:22.120684+0000","last_unstale":"2024-09-13T14:49:22.120980+0000","last_undegraded":"2024-09-13T14:49:22.120980+0000","last_fullsized":"2024-09-13T14:49:22.120980+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:49:34.732581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1091,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.100634+0000","last_change":"2024-09-13T14:19:26.131609+0000","last_active":"2024-09-13T14:49:21.100634+0000","last_peered":"2024-09-13T14:49:21.100634+0000","last_clean":"2024-09-13T14:49:21.100634+0000","last_became_active":"2024-09-13T14:19:26.131029+0000","last_became_peered":"2024-09-13T14:19:26.131029+0000","last_unstale":"2024-09-13T14:49:21.100634+0000","last_undegraded":"2024-09-13T14:49:21.100634+0000","last_fullsized":"2024-09-13T14:49:21.100634+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:57.933696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":593,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.095631+0000","last_change":"2024-09-13T14:43:32.635070+0000","last_active":"2024-09-13T14:49:21.095631+0000","last_peered":"2024-09-13T14:49:21.095631+0000","last_clean":"2024-09-13T14:49:21.095631+0000","last_became_active":"2024-09-13T14:43:32.634734+0000","last_became_peered":"2024-09-13T14:43:32.634734+0000","last_unstale":"2024-09-13T14:49:21.095631+0000","last_undegraded":"2024-09-13T14:49:21.095631+0000","last_fullsized":"2024-09-13T14:49:21.095631+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:33.107010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":1738,"reported_epoch":1166,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.712567+0000","last_change":"2024-09-13T14:43:07.537759+0000","last_active":"2024-09-13T14:49:08.712567+0000","last_peered":"2024-09-13T14:49:08.712567+0000","last_clean":"2024-09-13T14:49:08.712567+0000","last_became_active":"2024-09-13T14:43:07.520303+0000","last_became_peered":"2024-09-13T14:43:07.520303+0000","last_unstale":"2024-09-13T14:49:08.712567+0000","last_undegraded":"2024-09-13T14:49:08.712567+0000","last_fullsized":"2024-09-13T14:49:08.712567+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:32.772570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"261'100","reported_seq":47,"reported_epoch":1166,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.204554+0000","last_change":"2024-09-13T14:45:19.813527+0000","last_active":"2024-09-13T14:49:08.204554+0000","last_peered":"2024-09-13T14:49:08.204554+0000","last_clean":"2024-09-13T14:49:08.204554+0000","last_became_active":"2024-09-13T14:45:19.812834+0000","last_became_peered":"2024-09-13T14:45:19.812834+0000","last_unstale":"2024-09-13T14:49:08.204554+0000","last_undegraded":"2024-09-13T14:49:08.204554+0000","last_fullsized":"2024-09-13T14:49:08.204554+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.710192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.710192+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:08:55.152703+0000","scrub_duration":0.0019250910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T14:49:21.089133+0000","last_change":"2024-09-13T14:49:21.089133+0000","last_active":"2024-09-13T14:49:21.089133+0000","last_peered":"2024-09-13T14:49:21.089133+0000","last_clean":"2024-09-13T14:49:21.089133+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:49:21.089133+0000","last_undegraded":"2024-09-13T14:49:21.089133+0000","last_fullsized":"2024-09-13T14:49:21.089133+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":47,"reported_epoch":1166,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.203357+0000","last_change":"2024-09-13T14:45:19.813025+0000","last_active":"2024-09-13T14:49:08.203357+0000","last_peered":"2024-09-13T14:49:08.203357+0000","last_clean":"2024-09-13T14:49:08.203357+0000","last_became_active":"2024-09-13T14:45:19.812714+0000","last_became_peered":"2024-09-13T14:45:19.812714+0000","last_unstale":"2024-09-13T14:49:08.203357+0000","last_undegraded":"2024-09-13T14:49:08.203357+0000","last_fullsized":"2024-09-13T14:49:08.203357+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:40.671012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":100,"reported_epoch":1168,"state":"active+clean","last_fresh":"2024-09-13T14:49:22.116286+0000","last_change":"2024-09-13T14:43:07.522730+0000","last_active":"2024-09-13T14:49:22.116286+0000","last_peered":"2024-09-13T14:49:22.116286+0000","last_clean":"2024-09-13T14:49:22.116286+0000","last_became_active":"2024-09-13T14:43:07.517420+0000","last_became_peered":"2024-09-13T14:43:07.517420+0000","last_unstale":"2024-09-13T14:49:22.116286+0000","last_undegraded":"2024-09-13T14:49:22.116286+0000","last_fullsized":"2024-09-13T14:49:22.116286+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:21:30.265293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":1651,"reported_epoch":1166,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.204983+0000","last_change":"2024-09-13T14:30:15.581728+0000","last_active":"2024-09-13T14:49:08.204983+0000","last_peered":"2024-09-13T14:49:08.204983+0000","last_clean":"2024-09-13T14:49:08.204983+0000","last_became_active":"2024-09-13T14:30:15.581303+0000","last_became_peered":"2024-09-13T14:30:15.581303+0000","last_unstale":"2024-09-13T14:49:08.204983+0000","last_undegraded":"2024-09-13T14:49:08.204983+0000","last_fullsized":"2024-09-13T14:49:08.204983+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:10.027991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"261'100","reported_seq":118,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.102030+0000","last_change":"2024-09-13T14:43:31.112173+0000","last_active":"2024-09-13T14:49:21.102030+0000","last_peered":"2024-09-13T14:49:21.102030+0000","last_clean":"2024-09-13T14:49:21.102030+0000","last_became_active":"2024-09-13T14:42:42.709263+0000","last_became_peered":"2024-09-13T14:42:42.709263+0000","last_unstale":"2024-09-13T14:49:21.102030+0000","last_undegraded":"2024-09-13T14:49:21.102030+0000","last_fullsized":"2024-09-13T14:49:21.102030+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:31.112111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:31.112111+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:02.707372+0000","scrub_duration":0.0015759260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T14:49:21.089133+0000","last_change":"2024-09-13T14:49:21.089133+0000","last_active":"2024-09-13T14:49:21.089133+0000","last_peered":"2024-09-13T14:49:21.089133+0000","last_clean":"2024-09-13T14:49:21.089133+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:49:21.089133+0000","last_undegraded":"2024-09-13T14:49:21.089133+0000","last_fullsized":"2024-09-13T14:49:21.089133+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1691,"reported_epoch":1166,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.711540+0000","last_change":"2024-09-13T14:28:35.611508+0000","last_active":"2024-09-13T14:49:08.711540+0000","last_peered":"2024-09-13T14:49:08.711540+0000","last_clean":"2024-09-13T14:49:08.711540+0000","last_became_active":"2024-09-13T14:28:35.610106+0000","last_became_peered":"2024-09-13T14:28:35.610106+0000","last_unstale":"2024-09-13T14:49:08.711540+0000","last_undegraded":"2024-09-13T14:49:08.711540+0000","last_fullsized":"2024-09-13T14:49:08.711540+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:11.966958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1134,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.095649+0000","last_change":"2024-09-13T14:28:35.603663+0000","last_active":"2024-09-13T14:49:21.095649+0000","last_peered":"2024-09-13T14:49:21.095649+0000","last_clean":"2024-09-13T14:49:21.095649+0000","last_became_active":"2024-09-13T14:28:35.603358+0000","last_became_peered":"2024-09-13T14:28:35.603358+0000","last_unstale":"2024-09-13T14:49:21.095649+0000","last_undegraded":"2024-09-13T14:49:21.095649+0000","last_fullsized":"2024-09-13T14:49:21.095649+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:24:05.842796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":101,"reported_epoch":1168,"state":"active+clean","last_fresh":"2024-09-13T14:49:22.115479+0000","last_change":"2024-09-13T14:43:07.522750+0000","last_active":"2024-09-13T14:49:22.115479+0000","last_peered":"2024-09-13T14:49:22.115479+0000","last_clean":"2024-09-13T14:49:22.115479+0000","last_became_active":"2024-09-13T14:43:07.517267+0000","last_became_peered":"2024-09-13T14:43:07.517267+0000","last_unstale":"2024-09-13T14:49:22.115479+0000","last_undegraded":"2024-09-13T14:49:22.115479+0000","last_fullsized":"2024-09-13T14:49:22.115479+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:40.217748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"261'100","reported_seq":417,"reported_epoch":1166,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.712238+0000","last_change":"2024-09-13T14:45:19.815050+0000","last_active":"2024-09-13T14:49:08.712238+0000","last_peered":"2024-09-13T14:49:08.712238+0000","last_clean":"2024-09-13T14:49:08.712238+0000","last_became_active":"2024-09-13T14:45:19.809732+0000","last_became_peered":"2024-09-13T14:45:19.809732+0000","last_unstale":"2024-09-13T14:49:08.712238+0000","last_undegraded":"2024-09-13T14:49:08.712238+0000","last_fullsized":"2024-09-13T14:49:08.712238+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:33.574443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:33.574443+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:51:40.261036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T14:49:21.089133+0000","last_change":"2024-09-13T14:49:21.089133+0000","last_active":"2024-09-13T14:49:21.089133+0000","last_peered":"2024-09-13T14:49:21.089133+0000","last_clean":"2024-09-13T14:49:21.089133+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:49:21.089133+0000","last_undegraded":"2024-09-13T14:49:21.089133+0000","last_fullsized":"2024-09-13T14:49:21.089133+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":101,"reported_epoch":1168,"state":"active+clean","last_fresh":"2024-09-13T14:49:22.115714+0000","last_change":"2024-09-13T14:43:07.520049+0000","last_active":"2024-09-13T14:49:22.115714+0000","last_peered":"2024-09-13T14:49:22.115714+0000","last_clean":"2024-09-13T14:49:22.115714+0000","last_became_active":"2024-09-13T14:43:07.514646+0000","last_became_peered":"2024-09-13T14:43:07.514646+0000","last_unstale":"2024-09-13T14:49:22.115714+0000","last_undegraded":"2024-09-13T14:49:22.115714+0000","last_fullsized":"2024-09-13T14:49:22.115714+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:10:35.200778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":101,"reported_epoch":1168,"state":"active+clean","last_fresh":"2024-09-13T14:49:22.115381+0000","last_change":"2024-09-13T14:43:07.525666+0000","last_active":"2024-09-13T14:49:22.115381+0000","last_peered":"2024-09-13T14:49:22.115381+0000","last_clean":"2024-09-13T14:49:22.115381+0000","last_became_active":"2024-09-13T14:43:07.524064+0000","last_became_peered":"2024-09-13T14:43:07.524064+0000","last_unstale":"2024-09-13T14:49:22.115381+0000","last_undegraded":"2024-09-13T14:49:22.115381+0000","last_fullsized":"2024-09-13T14:49:22.115381+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:10.270333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1148,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.099136+0000","last_change":"2024-09-13T14:45:19.804177+0000","last_active":"2024-09-13T14:49:21.099136+0000","last_peered":"2024-09-13T14:49:21.099136+0000","last_clean":"2024-09-13T14:49:21.099136+0000","last_became_active":"2024-09-13T14:45:19.803890+0000","last_became_peered":"2024-09-13T14:45:19.803890+0000","last_unstale":"2024-09-13T14:49:21.099136+0000","last_undegraded":"2024-09-13T14:49:21.099136+0000","last_fullsized":"2024-09-13T14:49:21.099136+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:25.140789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"261'100","reported_seq":64,"reported_epoch":1166,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.204787+0000","last_change":"2024-09-13T14:45:36.965992+0000","last_active":"2024-09-13T14:49:08.204787+0000","last_peered":"2024-09-13T14:49:08.204787+0000","last_clean":"2024-09-13T14:49:08.204787+0000","last_became_active":"2024-09-13T14:45:19.811238+0000","last_became_peered":"2024-09-13T14:45:19.811238+0000","last_unstale":"2024-09-13T14:49:08.204787+0000","last_undegraded":"2024-09-13T14:49:08.204787+0000","last_fullsized":"2024-09-13T14:49:08.204787+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:36.965775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:45:36.965775+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:36:01.623251+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":1168,"state":"active+clean","last_fresh":"2024-09-13T14:49:22.122277+0000","last_change":"2024-09-13T14:49:22.122277+0000","last_active":"2024-09-13T14:49:22.122277+0000","last_peered":"2024-09-13T14:49:22.122277+0000","last_clean":"2024-09-13T14:49:22.122277+0000","last_became_active":"2024-09-13T14:49:22.121980+0000","last_became_peered":"2024-09-13T14:49:22.121980+0000","last_unstale":"2024-09-13T14:49:22.122277+0000","last_undegraded":"2024-09-13T14:49:22.122277+0000","last_fullsized":"2024-09-13T14:49:22.122277+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:58.275564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":47,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.098836+0000","last_change":"2024-09-13T14:45:19.806318+0000","last_active":"2024-09-13T14:49:21.098836+0000","last_peered":"2024-09-13T14:49:21.098836+0000","last_clean":"2024-09-13T14:49:21.098836+0000","last_became_active":"2024-09-13T14:45:19.806074+0000","last_became_peered":"2024-09-13T14:45:19.806074+0000","last_unstale":"2024-09-13T14:49:21.098836+0000","last_undegraded":"2024-09-13T14:49:21.098836+0000","last_fullsized":"2024-09-13T14:49:21.098836+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:21.679797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1109,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.101570+0000","last_change":"2024-09-13T14:28:35.604404+0000","last_active":"2024-09-13T14:49:21.101570+0000","last_peered":"2024-09-13T14:49:21.101570+0000","last_clean":"2024-09-13T14:49:21.101570+0000","last_became_active":"2024-09-13T14:28:35.604102+0000","last_became_peered":"2024-09-13T14:28:35.604102+0000","last_unstale":"2024-09-13T14:49:21.101570+0000","last_undegraded":"2024-09-13T14:49:21.101570+0000","last_fullsized":"2024-09-13T14:49:21.101570+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:25.546010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":1159,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.098824+0000","last_change":"2024-09-13T14:43:07.538919+0000","last_active":"2024-09-13T14:49:21.098824+0000","last_peered":"2024-09-13T14:49:21.098824+0000","last_clean":"2024-09-13T14:49:21.098824+0000","last_became_active":"2024-09-13T14:43:07.513109+0000","last_became_peered":"2024-09-13T14:43:07.513109+0000","last_unstale":"2024-09-13T14:49:21.098824+0000","last_undegraded":"2024-09-13T14:49:21.098824+0000","last_fullsized":"2024-09-13T14:49:21.098824+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:27.771720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"261'100","reported_seq":108,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.098557+0000","last_change":"2024-09-13T14:43:44.720333+0000","last_active":"2024-09-13T14:49:21.098557+0000","last_peered":"2024-09-13T14:49:21.098557+0000","last_clean":"2024-09-13T14:49:21.098557+0000","last_became_active":"2024-09-13T14:42:14.946591+0000","last_became_peered":"2024-09-13T14:42:14.946591+0000","last_unstale":"2024-09-13T14:49:21.098557+0000","last_undegraded":"2024-09-13T14:49:21.098557+0000","last_fullsized":"2024-09-13T14:49:21.098557+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:44.720258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:44.720258+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:33.281357+0000","scrub_duration":0.0018417920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T14:49:21.089133+0000","last_change":"2024-09-13T14:49:21.089133+0000","last_active":"2024-09-13T14:49:21.089133+0000","last_peered":"2024-09-13T14:49:21.089133+0000","last_clean":"2024-09-13T14:49:21.089133+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:49:21.089133+0000","last_undegraded":"2024-09-13T14:49:21.089133+0000","last_fullsized":"2024-09-13T14:49:21.089133+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":68,"reported_epoch":1168,"state":"active+clean","last_fresh":"2024-09-13T14:49:22.109504+0000","last_change":"2024-09-13T14:44:03.858211+0000","last_active":"2024-09-13T14:49:22.109504+0000","last_peered":"2024-09-13T14:49:22.109504+0000","last_clean":"2024-09-13T14:49:22.109504+0000","last_became_active":"2024-09-13T14:44:03.857928+0000","last_became_peered":"2024-09-13T14:44:03.857928+0000","last_unstale":"2024-09-13T14:49:22.109504+0000","last_undegraded":"2024-09-13T14:49:22.109504+0000","last_fullsized":"2024-09-13T14:49:22.109504+0000","mapping_epoch":1140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:28.428693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":101,"reported_epoch":1168,"state":"active+clean","last_fresh":"2024-09-13T14:49:22.116174+0000","last_change":"2024-09-13T14:43:07.516243+0000","last_active":"2024-09-13T14:49:22.116174+0000","last_peered":"2024-09-13T14:49:22.116174+0000","last_clean":"2024-09-13T14:49:22.116174+0000","last_became_active":"2024-09-13T14:43:07.515863+0000","last_became_peered":"2024-09-13T14:43:07.515863+0000","last_unstale":"2024-09-13T14:49:22.116174+0000","last_undegraded":"2024-09-13T14:49:22.116174+0000","last_fullsized":"2024-09-13T14:49:22.116174+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:08.085298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1737,"reported_epoch":1168,"state":"active+clean","last_fresh":"2024-09-13T14:49:22.108659+0000","last_change":"2024-09-13T14:45:19.812713+0000","last_active":"2024-09-13T14:49:22.108659+0000","last_peered":"2024-09-13T14:49:22.108659+0000","last_clean":"2024-09-13T14:49:22.108659+0000","last_became_active":"2024-09-13T14:45:19.812203+0000","last_became_peered":"2024-09-13T14:45:19.812203+0000","last_unstale":"2024-09-13T14:49:22.108659+0000","last_undegraded":"2024-09-13T14:49:22.108659+0000","last_fullsized":"2024-09-13T14:49:22.108659+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:30:46.149347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"261'100","reported_seq":1288,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.095061+0000","last_change":"2024-09-13T14:43:42.577312+0000","last_active":"2024-09-13T14:49:21.095061+0000","last_peered":"2024-09-13T14:49:21.095061+0000","last_clean":"2024-09-13T14:49:21.095061+0000","last_became_active":"2024-09-13T14:42:14.928923+0000","last_became_peered":"2024-09-13T14:42:14.928923+0000","last_unstale":"2024-09-13T14:49:21.095061+0000","last_undegraded":"2024-09-13T14:49:21.095061+0000","last_fullsized":"2024-09-13T14:49:21.095061+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:42.577243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:42.577243+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:01.234250+0000","scrub_duration":0.001367839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":1168,"state":"active+clean","last_fresh":"2024-09-13T14:49:22.121072+0000","last_change":"2024-09-13T14:49:22.118690+0000","last_active":"2024-09-13T14:49:22.121072+0000","last_peered":"2024-09-13T14:49:22.121072+0000","last_clean":"2024-09-13T14:49:22.121072+0000","last_became_active":"2024-09-13T14:49:22.117901+0000","last_became_peered":"2024-09-13T14:49:22.117901+0000","last_unstale":"2024-09-13T14:49:22.121072+0000","last_undegraded":"2024-09-13T14:49:22.121072+0000","last_fullsized":"2024-09-13T14:49:22.121072+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:39:36.336919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1727,"reported_epoch":1166,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.712316+0000","last_change":"2024-09-13T14:43:07.520212+0000","last_active":"2024-09-13T14:49:08.712316+0000","last_peered":"2024-09-13T14:49:08.712316+0000","last_clean":"2024-09-13T14:49:08.712316+0000","last_became_active":"2024-09-13T14:43:07.519831+0000","last_became_peered":"2024-09-13T14:43:07.519831+0000","last_unstale":"2024-09-13T14:49:08.712316+0000","last_undegraded":"2024-09-13T14:49:08.712316+0000","last_fullsized":"2024-09-13T14:49:08.712316+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:20.990736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1749,"reported_epoch":1166,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.711345+0000","last_change":"2024-09-13T14:45:19.809505+0000","last_active":"2024-09-13T14:49:08.711345+0000","last_peered":"2024-09-13T14:49:08.711345+0000","last_clean":"2024-09-13T14:49:08.711345+0000","last_became_active":"2024-09-13T14:45:19.809181+0000","last_became_peered":"2024-09-13T14:45:19.809181+0000","last_unstale":"2024-09-13T14:49:08.711345+0000","last_undegraded":"2024-09-13T14:49:08.711345+0000","last_fullsized":"2024-09-13T14:49:08.711345+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:24.239188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1171,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.095633+0000","last_change":"2024-09-13T14:43:07.516140+0000","last_active":"2024-09-13T14:49:21.095633+0000","last_peered":"2024-09-13T14:49:21.095633+0000","last_clean":"2024-09-13T14:49:21.095633+0000","last_became_active":"2024-09-13T14:43:07.515481+0000","last_became_peered":"2024-09-13T14:43:07.515481+0000","last_unstale":"2024-09-13T14:49:21.095633+0000","last_undegraded":"2024-09-13T14:49:21.095633+0000","last_fullsized":"2024-09-13T14:49:21.095633+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:43.706214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"261'100","reported_seq":113,"reported_epoch":1168,"state":"active+clean","last_fresh":"2024-09-13T14:49:22.116280+0000","last_change":"2024-09-13T14:43:16.008346+0000","last_active":"2024-09-13T14:49:22.116280+0000","last_peered":"2024-09-13T14:49:22.116280+0000","last_clean":"2024-09-13T14:49:22.116280+0000","last_became_active":"2024-09-13T14:43:07.523121+0000","last_became_peered":"2024-09-13T14:43:07.523121+0000","last_unstale":"2024-09-13T14:49:22.116280+0000","last_undegraded":"2024-09-13T14:49:22.116280+0000","last_fullsized":"2024-09-13T14:49:22.116280+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.008264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.008264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:58:35.564470+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":1168,"state":"active+clean","last_fresh":"2024-09-13T14:49:22.119570+0000","last_change":"2024-09-13T14:49:22.119570+0000","last_active":"2024-09-13T14:49:22.119570+0000","last_peered":"2024-09-13T14:49:22.119570+0000","last_clean":"2024-09-13T14:49:22.119570+0000","last_became_active":"2024-09-13T14:49:22.119224+0000","last_became_peered":"2024-09-13T14:49:22.119224+0000","last_unstale":"2024-09-13T14:49:22.119570+0000","last_undegraded":"2024-09-13T14:49:22.119570+0000","last_fullsized":"2024-09-13T14:49:22.119570+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:43.554416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":50,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.101938+0000","last_change":"2024-09-13T14:43:32.636990+0000","last_active":"2024-09-13T14:49:21.101938+0000","last_peered":"2024-09-13T14:49:21.101938+0000","last_clean":"2024-09-13T14:49:21.101938+0000","last_became_active":"2024-09-13T14:43:32.636589+0000","last_became_peered":"2024-09-13T14:43:32.636589+0000","last_unstale":"2024-09-13T14:49:21.101938+0000","last_undegraded":"2024-09-13T14:49:21.101938+0000","last_fullsized":"2024-09-13T14:49:21.101938+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:36:25.129769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":47,"reported_epoch":1166,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.203754+0000","last_change":"2024-09-13T14:45:19.810888+0000","last_active":"2024-09-13T14:49:08.203754+0000","last_peered":"2024-09-13T14:49:08.203754+0000","last_clean":"2024-09-13T14:49:08.203754+0000","last_became_active":"2024-09-13T14:45:19.810486+0000","last_became_peered":"2024-09-13T14:45:19.810486+0000","last_unstale":"2024-09-13T14:49:08.203754+0000","last_undegraded":"2024-09-13T14:49:08.203754+0000","last_fullsized":"2024-09-13T14:49:08.203754+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:03.886942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":1124,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.099273+0000","last_change":"2024-09-13T14:29:42.539455+0000","last_active":"2024-09-13T14:49:21.099273+0000","last_peered":"2024-09-13T14:49:21.099273+0000","last_clean":"2024-09-13T14:49:21.099273+0000","last_became_active":"2024-09-13T14:29:42.539068+0000","last_became_peered":"2024-09-13T14:29:42.539068+0000","last_unstale":"2024-09-13T14:49:21.099273+0000","last_undegraded":"2024-09-13T14:49:21.099273+0000","last_fullsized":"2024-09-13T14:49:21.099273+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:24:06.048855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034405999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"261'100","reported_seq":1279,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.095196+0000","last_change":"2024-09-13T14:44:28.186029+0000","last_active":"2024-09-13T14:49:21.095196+0000","last_peered":"2024-09-13T14:49:21.095196+0000","last_clean":"2024-09-13T14:49:21.095196+0000","last_became_active":"2024-09-13T14:42:14.927771+0000","last_became_peered":"2024-09-13T14:42:14.927771+0000","last_unstale":"2024-09-13T14:49:21.095196+0000","last_undegraded":"2024-09-13T14:49:21.095196+0000","last_fullsized":"2024-09-13T14:49:21.095196+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:28.185949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:28.185949+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:38.363961+0000","scrub_duration":0.0018071129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":1168,"state":"active+clean","last_fresh":"2024-09-13T14:49:22.121329+0000","last_change":"2024-09-13T14:49:22.119776+0000","last_active":"2024-09-13T14:49:22.121329+0000","last_peered":"2024-09-13T14:49:22.121329+0000","last_clean":"2024-09-13T14:49:22.121329+0000","last_became_active":"2024-09-13T14:49:22.119517+0000","last_became_peered":"2024-09-13T14:49:22.119517+0000","last_unstale":"2024-09-13T14:49:22.121329+0000","last_undegraded":"2024-09-13T14:49:22.121329+0000","last_fullsized":"2024-09-13T14:49:22.121329+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:54:48.856174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":100,"reported_epoch":1168,"state":"active+clean","last_fresh":"2024-09-13T14:49:22.116546+0000","last_change":"2024-09-13T14:43:07.520176+0000","last_active":"2024-09-13T14:49:22.116546+0000","last_peered":"2024-09-13T14:49:22.116546+0000","last_clean":"2024-09-13T14:49:22.116546+0000","last_became_active":"2024-09-13T14:43:07.516748+0000","last_became_peered":"2024-09-13T14:43:07.516748+0000","last_unstale":"2024-09-13T14:49:22.116546+0000","last_undegraded":"2024-09-13T14:49:22.116546+0000","last_fullsized":"2024-09-13T14:49:22.116546+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:52:04.392821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":47,"reported_epoch":1166,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.203441+0000","last_change":"2024-09-13T14:45:19.812353+0000","last_active":"2024-09-13T14:49:08.203441+0000","last_peered":"2024-09-13T14:49:08.203441+0000","last_clean":"2024-09-13T14:49:08.203441+0000","last_became_active":"2024-09-13T14:45:19.811743+0000","last_became_peered":"2024-09-13T14:45:19.811743+0000","last_unstale":"2024-09-13T14:49:08.203441+0000","last_undegraded":"2024-09-13T14:49:08.203441+0000","last_fullsized":"2024-09-13T14:49:08.203441+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:43.728170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"261'100","reported_seq":108,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.099122+0000","last_change":"2024-09-13T14:44:34.814157+0000","last_active":"2024-09-13T14:49:21.099122+0000","last_peered":"2024-09-13T14:49:21.099122+0000","last_clean":"2024-09-13T14:49:21.099122+0000","last_became_active":"2024-09-13T14:42:42.707803+0000","last_became_peered":"2024-09-13T14:42:42.707803+0000","last_unstale":"2024-09-13T14:49:21.099122+0000","last_undegraded":"2024-09-13T14:49:21.099122+0000","last_fullsized":"2024-09-13T14:49:21.099122+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:34.814070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:34.814070+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:38.408561+0000","scrub_duration":0.01686414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":1168,"state":"active+clean","last_fresh":"2024-09-13T14:49:22.119578+0000","last_change":"2024-09-13T14:49:22.119578+0000","last_active":"2024-09-13T14:49:22.119578+0000","last_peered":"2024-09-13T14:49:22.119578+0000","last_clean":"2024-09-13T14:49:22.119578+0000","last_became_active":"2024-09-13T14:49:22.119280+0000","last_became_peered":"2024-09-13T14:49:22.119280+0000","last_unstale":"2024-09-13T14:49:22.119578+0000","last_undegraded":"2024-09-13T14:49:22.119578+0000","last_fullsized":"2024-09-13T14:49:22.119578+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:13:15.669993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1138,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.102130+0000","last_change":"2024-09-13T14:44:03.855577+0000","last_active":"2024-09-13T14:49:21.102130+0000","last_peered":"2024-09-13T14:49:21.102130+0000","last_clean":"2024-09-13T14:49:21.102130+0000","last_became_active":"2024-09-13T14:44:03.855267+0000","last_became_peered":"2024-09-13T14:44:03.855267+0000","last_unstale":"2024-09-13T14:49:21.102130+0000","last_undegraded":"2024-09-13T14:49:21.102130+0000","last_fullsized":"2024-09-13T14:49:21.102130+0000","mapping_epoch":1140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:46.148814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1109,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.100759+0000","last_change":"2024-09-13T14:29:42.545815+0000","last_active":"2024-09-13T14:49:21.100759+0000","last_peered":"2024-09-13T14:49:21.100759+0000","last_clean":"2024-09-13T14:49:21.100759+0000","last_became_active":"2024-09-13T14:29:42.545508+0000","last_became_peered":"2024-09-13T14:29:42.545508+0000","last_unstale":"2024-09-13T14:49:21.100759+0000","last_undegraded":"2024-09-13T14:49:21.100759+0000","last_fullsized":"2024-09-13T14:49:21.100759+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:39.678381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"261'100","reported_seq":118,"reported_epoch":1168,"state":"active+clean","last_fresh":"2024-09-13T14:49:22.116009+0000","last_change":"2024-09-13T14:43:18.988482+0000","last_active":"2024-09-13T14:49:22.116009+0000","last_peered":"2024-09-13T14:49:22.116009+0000","last_clean":"2024-09-13T14:49:22.116009+0000","last_became_active":"2024-09-13T14:43:14.556754+0000","last_became_peered":"2024-09-13T14:43:14.556754+0000","last_unstale":"2024-09-13T14:49:22.116009+0000","last_undegraded":"2024-09-13T14:49:22.116009+0000","last_fullsized":"2024-09-13T14:49:22.116009+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:18.988405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:18.988405+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:45:01.726326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":1168,"state":"active+clean","last_fresh":"2024-09-13T14:49:22.123069+0000","last_change":"2024-09-13T14:49:22.121230+0000","last_active":"2024-09-13T14:49:22.123069+0000","last_peered":"2024-09-13T14:49:22.123069+0000","last_clean":"2024-09-13T14:49:22.123069+0000","last_became_active":"2024-09-13T14:49:22.120836+0000","last_became_peered":"2024-09-13T14:49:22.120836+0000","last_unstale":"2024-09-13T14:49:22.123069+0000","last_undegraded":"2024-09-13T14:49:22.123069+0000","last_fullsized":"2024-09-13T14:49:22.123069+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:08:52.359684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1155,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.102283+0000","last_change":"2024-09-13T14:43:07.515418+0000","last_active":"2024-09-13T14:49:21.102283+0000","last_peered":"2024-09-13T14:49:21.102283+0000","last_clean":"2024-09-13T14:49:21.102283+0000","last_became_active":"2024-09-13T14:43:07.515113+0000","last_became_peered":"2024-09-13T14:43:07.515113+0000","last_unstale":"2024-09-13T14:49:21.102283+0000","last_undegraded":"2024-09-13T14:49:21.102283+0000","last_fullsized":"2024-09-13T14:49:21.102283+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:10:25.810604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1122,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.101963+0000","last_change":"2024-09-13T14:43:32.636334+0000","last_active":"2024-09-13T14:49:21.101963+0000","last_peered":"2024-09-13T14:49:21.101963+0000","last_clean":"2024-09-13T14:49:21.101963+0000","last_became_active":"2024-09-13T14:43:32.635736+0000","last_became_peered":"2024-09-13T14:43:32.635736+0000","last_unstale":"2024-09-13T14:49:21.101963+0000","last_undegraded":"2024-09-13T14:49:21.101963+0000","last_fullsized":"2024-09-13T14:49:21.101963+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:49.076014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"261'100","reported_seq":113,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.098485+0000","last_change":"2024-09-13T14:45:19.805634+0000","last_active":"2024-09-13T14:49:21.098485+0000","last_peered":"2024-09-13T14:49:21.098485+0000","last_clean":"2024-09-13T14:49:21.098485+0000","last_became_active":"2024-09-13T14:45:19.805046+0000","last_became_peered":"2024-09-13T14:45:19.805046+0000","last_unstale":"2024-09-13T14:49:21.098485+0000","last_undegraded":"2024-09-13T14:49:21.098485+0000","last_fullsized":"2024-09-13T14:49:21.098485+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:32.752408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:32.752408+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:06:53.140239+0000","scrub_duration":0.0057731559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T14:49:21.089133+0000","last_change":"2024-09-13T14:49:21.089133+0000","last_active":"2024-09-13T14:49:21.089133+0000","last_peered":"2024-09-13T14:49:21.089133+0000","last_clean":"2024-09-13T14:49:21.089133+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:49:21.089133+0000","last_undegraded":"2024-09-13T14:49:21.089133+0000","last_fullsized":"2024-09-13T14:49:21.089133+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1158,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.101839+0000","last_change":"2024-09-13T14:45:19.804441+0000","last_active":"2024-09-13T14:49:21.101839+0000","last_peered":"2024-09-13T14:49:21.101839+0000","last_clean":"2024-09-13T14:49:21.101839+0000","last_became_active":"2024-09-13T14:45:19.804039+0000","last_became_peered":"2024-09-13T14:45:19.804039+0000","last_unstale":"2024-09-13T14:49:21.101839+0000","last_undegraded":"2024-09-13T14:49:21.101839+0000","last_fullsized":"2024-09-13T14:49:21.101839+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:58:00.322042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":51,"reported_epoch":1166,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.203541+0000","last_change":"2024-09-13T14:45:19.816666+0000","last_active":"2024-09-13T14:49:08.203541+0000","last_peered":"2024-09-13T14:49:08.203541+0000","last_clean":"2024-09-13T14:49:08.203541+0000","last_became_active":"2024-09-13T14:45:19.807224+0000","last_became_peered":"2024-09-13T14:45:19.807224+0000","last_unstale":"2024-09-13T14:49:08.203541+0000","last_undegraded":"2024-09-13T14:49:08.203541+0000","last_fullsized":"2024-09-13T14:49:08.203541+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:15:20.171682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"261'100","reported_seq":1298,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.095003+0000","last_change":"2024-09-13T14:45:19.798553+0000","last_active":"2024-09-13T14:49:21.095003+0000","last_peered":"2024-09-13T14:49:21.095003+0000","last_clean":"2024-09-13T14:49:21.095003+0000","last_became_active":"2024-09-13T14:45:19.798189+0000","last_became_peered":"2024-09-13T14:45:19.798189+0000","last_unstale":"2024-09-13T14:49:21.095003+0000","last_undegraded":"2024-09-13T14:49:21.095003+0000","last_fullsized":"2024-09-13T14:49:21.095003+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:29.169325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:29.169325+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:10.412666+0000","scrub_duration":0.0019358890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":11,"reported_epoch":1168,"state":"active+clean","last_fresh":"2024-09-13T14:49:22.121359+0000","last_change":"2024-09-13T14:49:22.120585+0000","last_active":"2024-09-13T14:49:22.121359+0000","last_peered":"2024-09-13T14:49:22.121359+0000","last_clean":"2024-09-13T14:49:22.121359+0000","last_became_active":"2024-09-13T14:49:22.120301+0000","last_became_peered":"2024-09-13T14:49:22.120301+0000","last_unstale":"2024-09-13T14:49:22.121359+0000","last_undegraded":"2024-09-13T14:49:22.121359+0000","last_fullsized":"2024-09-13T14:49:22.121359+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:50:44.900688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1716,"reported_epoch":1166,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.711430+0000","last_change":"2024-09-13T14:43:07.515572+0000","last_active":"2024-09-13T14:49:08.711430+0000","last_peered":"2024-09-13T14:49:08.711430+0000","last_clean":"2024-09-13T14:49:08.711430+0000","last_became_active":"2024-09-13T14:43:07.515231+0000","last_became_peered":"2024-09-13T14:43:07.515231+0000","last_unstale":"2024-09-13T14:49:08.711430+0000","last_undegraded":"2024-09-13T14:49:08.711430+0000","last_fullsized":"2024-09-13T14:49:08.711430+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:57.966380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1151,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.099051+0000","last_change":"2024-09-13T14:45:19.804982+0000","last_active":"2024-09-13T14:49:21.099051+0000","last_peered":"2024-09-13T14:49:21.099051+0000","last_clean":"2024-09-13T14:49:21.099051+0000","last_became_active":"2024-09-13T14:45:19.804663+0000","last_became_peered":"2024-09-13T14:45:19.804663+0000","last_unstale":"2024-09-13T14:49:21.099051+0000","last_undegraded":"2024-09-13T14:49:21.099051+0000","last_fullsized":"2024-09-13T14:49:21.099051+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:43.211479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"261'100","reported_seq":388,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.102026+0000","last_change":"2024-09-13T14:43:42.881584+0000","last_active":"2024-09-13T14:49:21.102026+0000","last_peered":"2024-09-13T14:49:21.102026+0000","last_clean":"2024-09-13T14:49:21.102026+0000","last_became_active":"2024-09-13T14:43:14.562161+0000","last_became_peered":"2024-09-13T14:43:14.562161+0000","last_unstale":"2024-09-13T14:49:21.102026+0000","last_undegraded":"2024-09-13T14:49:21.102026+0000","last_fullsized":"2024-09-13T14:49:21.102026+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:42.881520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:42.881520+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:06:50.353442+0000","scrub_duration":0.0013804310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T14:49:21.089133+0000","last_change":"2024-09-13T14:49:21.089133+0000","last_active":"2024-09-13T14:49:21.089133+0000","last_peered":"2024-09-13T14:49:21.089133+0000","last_clean":"2024-09-13T14:49:21.089133+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:49:21.089133+0000","last_undegraded":"2024-09-13T14:49:21.089133+0000","last_fullsized":"2024-09-13T14:49:21.089133+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1124,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.100885+0000","last_change":"2024-09-13T14:43:07.518120+0000","last_active":"2024-09-13T14:49:21.100885+0000","last_peered":"2024-09-13T14:49:21.100885+0000","last_clean":"2024-09-13T14:49:21.100885+0000","last_became_active":"2024-09-13T14:43:07.517361+0000","last_became_peered":"2024-09-13T14:43:07.517361+0000","last_unstale":"2024-09-13T14:49:21.100885+0000","last_undegraded":"2024-09-13T14:49:21.100885+0000","last_fullsized":"2024-09-13T14:49:21.100885+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:39.366473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":1106,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.095563+0000","last_change":"2024-09-13T14:19:25.871006+0000","last_active":"2024-09-13T14:49:21.095563+0000","last_peered":"2024-09-13T14:49:21.095563+0000","last_clean":"2024-09-13T14:49:21.095563+0000","last_became_active":"2024-09-13T14:19:25.870713+0000","last_became_peered":"2024-09-13T14:19:25.870713+0000","last_unstale":"2024-09-13T14:49:21.095563+0000","last_undegraded":"2024-09-13T14:49:21.095563+0000","last_fullsized":"2024-09-13T14:49:21.095563+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:34.372124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"261'100","reported_seq":382,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.723059+0000","last_change":"2024-09-13T14:49:21.723011+0000","last_active":"2024-09-13T14:49:21.723059+0000","last_peered":"2024-09-13T14:49:21.723059+0000","last_clean":"2024-09-13T14:49:21.723059+0000","last_became_active":"2024-09-13T14:43:32.636545+0000","last_became_peered":"2024-09-13T14:43:32.636545+0000","last_unstale":"2024-09-13T14:49:21.723059+0000","last_undegraded":"2024-09-13T14:49:21.723059+0000","last_fullsized":"2024-09-13T14:49:21.723059+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:49:21.722939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.722939+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:18:07.462147+0000","scrub_duration":0.0016642790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":1168,"state":"active+clean","last_fresh":"2024-09-13T14:49:22.120623+0000","last_change":"2024-09-13T14:49:22.120623+0000","last_active":"2024-09-13T14:49:22.120623+0000","last_peered":"2024-09-13T14:49:22.120623+0000","last_clean":"2024-09-13T14:49:22.120623+0000","last_became_active":"2024-09-13T14:49:22.120295+0000","last_became_peered":"2024-09-13T14:49:22.120295+0000","last_unstale":"2024-09-13T14:49:22.120623+0000","last_undegraded":"2024-09-13T14:49:22.120623+0000","last_fullsized":"2024-09-13T14:49:22.120623+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.848357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1175,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.100721+0000","last_change":"2024-09-13T14:45:19.804868+0000","last_active":"2024-09-13T14:49:21.100721+0000","last_peered":"2024-09-13T14:49:21.100721+0000","last_clean":"2024-09-13T14:49:21.100721+0000","last_became_active":"2024-09-13T14:45:19.804520+0000","last_became_peered":"2024-09-13T14:45:19.804520+0000","last_unstale":"2024-09-13T14:49:21.100721+0000","last_undegraded":"2024-09-13T14:49:21.100721+0000","last_fullsized":"2024-09-13T14:49:21.100721+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:17:53.924629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":1152,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.095706+0000","last_change":"2024-09-13T14:43:07.533955+0000","last_active":"2024-09-13T14:49:21.095706+0000","last_peered":"2024-09-13T14:49:21.095706+0000","last_clean":"2024-09-13T14:49:21.095706+0000","last_became_active":"2024-09-13T14:43:07.516816+0000","last_became_peered":"2024-09-13T14:43:07.516816+0000","last_unstale":"2024-09-13T14:49:21.095706+0000","last_undegraded":"2024-09-13T14:49:21.095706+0000","last_fullsized":"2024-09-13T14:49:21.095706+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:26.232184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"261'100","reported_seq":51,"reported_epoch":1166,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.204457+0000","last_change":"2024-09-13T14:45:19.821649+0000","last_active":"2024-09-13T14:49:08.204457+0000","last_peered":"2024-09-13T14:49:08.204457+0000","last_clean":"2024-09-13T14:49:08.204457+0000","last_became_active":"2024-09-13T14:45:19.812346+0000","last_became_peered":"2024-09-13T14:45:19.812346+0000","last_unstale":"2024-09-13T14:49:08.204457+0000","last_undegraded":"2024-09-13T14:49:08.204457+0000","last_fullsized":"2024-09-13T14:49:08.204457+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.597568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.597568+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:45.218892+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":1168,"state":"active+clean","last_fresh":"2024-09-13T14:49:22.120890+0000","last_change":"2024-09-13T14:49:22.120890+0000","last_active":"2024-09-13T14:49:22.120890+0000","last_peered":"2024-09-13T14:49:22.120890+0000","last_clean":"2024-09-13T14:49:22.120890+0000","last_became_active":"2024-09-13T14:49:22.120420+0000","last_became_peered":"2024-09-13T14:49:22.120420+0000","last_unstale":"2024-09-13T14:49:22.120890+0000","last_undegraded":"2024-09-13T14:49:22.120890+0000","last_fullsized":"2024-09-13T14:49:22.120890+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:03:49.271123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1093,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.098767+0000","last_change":"2024-09-13T14:19:26.887839+0000","last_active":"2024-09-13T14:49:21.098767+0000","last_peered":"2024-09-13T14:49:21.098767+0000","last_clean":"2024-09-13T14:49:21.098767+0000","last_became_active":"2024-09-13T14:19:26.887498+0000","last_became_peered":"2024-09-13T14:19:26.887498+0000","last_unstale":"2024-09-13T14:49:21.098767+0000","last_undegraded":"2024-09-13T14:49:21.098767+0000","last_fullsized":"2024-09-13T14:49:21.098767+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:24.475263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":1123,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.095681+0000","last_change":"2024-09-13T14:28:35.608607+0000","last_active":"2024-09-13T14:49:21.095681+0000","last_peered":"2024-09-13T14:49:21.095681+0000","last_clean":"2024-09-13T14:49:21.095681+0000","last_became_active":"2024-09-13T14:28:35.608254+0000","last_became_peered":"2024-09-13T14:28:35.608254+0000","last_unstale":"2024-09-13T14:49:21.095681+0000","last_undegraded":"2024-09-13T14:49:21.095681+0000","last_fullsized":"2024-09-13T14:49:21.095681+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:24.019173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"261'100","reported_seq":59,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.095361+0000","last_change":"2024-09-13T14:44:31.191431+0000","last_active":"2024-09-13T14:49:21.095361+0000","last_peered":"2024-09-13T14:49:21.095361+0000","last_clean":"2024-09-13T14:49:21.095361+0000","last_became_active":"2024-09-13T14:43:14.557376+0000","last_became_peered":"2024-09-13T14:43:14.557376+0000","last_unstale":"2024-09-13T14:49:21.095361+0000","last_undegraded":"2024-09-13T14:49:21.095361+0000","last_fullsized":"2024-09-13T14:49:21.095361+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.191326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.191326+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:15:27.352698+0000","scrub_duration":0.0025317360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1113,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.100753+0000","last_change":"2024-09-13T14:28:35.603724+0000","last_active":"2024-09-13T14:49:21.100753+0000","last_peered":"2024-09-13T14:49:21.100753+0000","last_clean":"2024-09-13T14:49:21.100753+0000","last_became_active":"2024-09-13T14:28:35.603316+0000","last_became_peered":"2024-09-13T14:28:35.603316+0000","last_unstale":"2024-09-13T14:49:21.100753+0000","last_undegraded":"2024-09-13T14:49:21.100753+0000","last_fullsized":"2024-09-13T14:49:21.100753+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:47.668223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":47,"reported_epoch":1166,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.204678+0000","last_change":"2024-09-13T14:45:19.811466+0000","last_active":"2024-09-13T14:49:08.204678+0000","last_peered":"2024-09-13T14:49:08.204678+0000","last_clean":"2024-09-13T14:49:08.204678+0000","last_became_active":"2024-09-13T14:45:19.810617+0000","last_became_peered":"2024-09-13T14:45:19.810617+0000","last_unstale":"2024-09-13T14:49:08.204678+0000","last_undegraded":"2024-09-13T14:49:08.204678+0000","last_fullsized":"2024-09-13T14:49:08.204678+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:16.246669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"261'100","reported_seq":59,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.098656+0000","last_change":"2024-09-13T14:44:50.805306+0000","last_active":"2024-09-13T14:49:21.098656+0000","last_peered":"2024-09-13T14:49:21.098656+0000","last_clean":"2024-09-13T14:49:21.098656+0000","last_became_active":"2024-09-13T14:43:14.562922+0000","last_became_peered":"2024-09-13T14:43:14.562922+0000","last_unstale":"2024-09-13T14:49:21.098656+0000","last_undegraded":"2024-09-13T14:49:21.098656+0000","last_fullsized":"2024-09-13T14:49:21.098656+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:50.805210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:50.805210+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:28:38.529599+0000","scrub_duration":0.0042900009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1105,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.095158+0000","last_change":"2024-09-13T14:19:25.873088+0000","last_active":"2024-09-13T14:49:21.095158+0000","last_peered":"2024-09-13T14:49:21.095158+0000","last_clean":"2024-09-13T14:49:21.095158+0000","last_became_active":"2024-09-13T14:19:25.872781+0000","last_became_peered":"2024-09-13T14:19:25.872781+0000","last_unstale":"2024-09-13T14:49:21.095158+0000","last_undegraded":"2024-09-13T14:49:21.095158+0000","last_fullsized":"2024-09-13T14:49:21.095158+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:05.607382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1715,"reported_epoch":1166,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.712489+0000","last_change":"2024-09-13T14:45:19.813876+0000","last_active":"2024-09-13T14:49:08.712489+0000","last_peered":"2024-09-13T14:49:08.712489+0000","last_clean":"2024-09-13T14:49:08.712489+0000","last_became_active":"2024-09-13T14:45:19.809136+0000","last_became_peered":"2024-09-13T14:45:19.809136+0000","last_unstale":"2024-09-13T14:49:08.712489+0000","last_undegraded":"2024-09-13T14:49:08.712489+0000","last_fullsized":"2024-09-13T14:49:08.712489+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:50.013528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"261'100","reported_seq":125,"reported_epoch":1168,"state":"active+clean","last_fresh":"2024-09-13T14:49:22.116171+0000","last_change":"2024-09-13T14:45:19.810654+0000","last_active":"2024-09-13T14:49:22.116171+0000","last_peered":"2024-09-13T14:49:22.116171+0000","last_clean":"2024-09-13T14:49:22.116171+0000","last_became_active":"2024-09-13T14:45:19.810066+0000","last_became_peered":"2024-09-13T14:45:19.810066+0000","last_unstale":"2024-09-13T14:49:22.116171+0000","last_undegraded":"2024-09-13T14:49:22.116171+0000","last_fullsized":"2024-09-13T14:49:22.116171+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:19.978464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:19.978464+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:33.639332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1762,"reported_epoch":1166,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.711352+0000","last_change":"2024-09-13T14:45:19.809166+0000","last_active":"2024-09-13T14:49:08.711352+0000","last_peered":"2024-09-13T14:49:08.711352+0000","last_clean":"2024-09-13T14:49:08.711352+0000","last_became_active":"2024-09-13T14:45:19.808631+0000","last_became_peered":"2024-09-13T14:45:19.808631+0000","last_unstale":"2024-09-13T14:49:08.711352+0000","last_undegraded":"2024-09-13T14:49:08.711352+0000","last_fullsized":"2024-09-13T14:49:08.711352+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:38.822245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1130,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.099034+0000","last_change":"2024-09-13T14:45:19.804229+0000","last_active":"2024-09-13T14:49:21.099034+0000","last_peered":"2024-09-13T14:49:21.099034+0000","last_clean":"2024-09-13T14:49:21.099034+0000","last_became_active":"2024-09-13T14:45:19.803929+0000","last_became_peered":"2024-09-13T14:45:19.803929+0000","last_unstale":"2024-09-13T14:49:21.099034+0000","last_undegraded":"2024-09-13T14:49:21.099034+0000","last_fullsized":"2024-09-13T14:49:21.099034+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:33:53.675568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"261'100","reported_seq":59,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.098483+0000","last_change":"2024-09-13T14:44:42.654506+0000","last_active":"2024-09-13T14:49:21.098483+0000","last_peered":"2024-09-13T14:49:21.098483+0000","last_clean":"2024-09-13T14:49:21.098483+0000","last_became_active":"2024-09-13T14:43:14.563451+0000","last_became_peered":"2024-09-13T14:43:14.563451+0000","last_unstale":"2024-09-13T14:49:21.098483+0000","last_undegraded":"2024-09-13T14:49:21.098483+0000","last_fullsized":"2024-09-13T14:49:21.098483+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:42.654455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:42.654455+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:57:32.251597+0000","scrub_duration":0.001494097,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":1124,"reported_epoch":1167,"state":"active+clean","last_fresh":"2024-09-13T14:49:21.095277+0000","last_change":"2024-09-13T14:29:42.541093+0000","last_active":"2024-09-13T14:49:21.095277+0000","last_peered":"2024-09-13T14:49:21.095277+0000","last_clean":"2024-09-13T14:49:21.095277+0000","last_became_active":"2024-09-13T14:29:42.540671+0000","last_became_peered":"2024-09-13T14:29:42.540671+0000","last_unstale":"2024-09-13T14:49:21.095277+0000","last_undegraded":"2024-09-13T14:49:21.095277+0000","last_fullsized":"2024-09-13T14:49:21.095277+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:45.722376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1654,"reported_epoch":1166,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.204649+0000","last_change":"2024-09-13T14:43:07.518679+0000","last_active":"2024-09-13T14:49:08.204649+0000","last_peered":"2024-09-13T14:49:08.204649+0000","last_clean":"2024-09-13T14:49:08.204649+0000","last_became_active":"2024-09-13T14:43:07.518333+0000","last_became_peered":"2024-09-13T14:43:07.518333+0000","last_unstale":"2024-09-13T14:49:08.204649+0000","last_undegraded":"2024-09-13T14:49:08.204649+0000","last_fullsized":"2024-09-13T14:49:08.204649+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:20.616274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"261'100","reported_seq":423,"reported_epoch":1166,"state":"active+clean","last_fresh":"2024-09-13T14:49:08.711452+0000","last_change":"2024-09-13T14:45:19.809268+0000","last_active":"2024-09-13T14:49:08.711452+0000","last_peered":"2024-09-13T14:49:08.711452+0000","last_clean":"2024-09-13T14:49:08.711452+0000","last_became_active":"2024-09-13T14:45:19.808772+0000","last_became_peered":"2024-09-13T14:45:19.808772+0000","last_unstale":"2024-09-13T14:49:08.711452+0000","last_undegraded":"2024-09-13T14:49:08.711452+0000","last_fullsized":"2024-09-13T14:49:08.711452+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:14.654603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:14.654603+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:40.964553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":30,"acting":30,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430592,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":320708,"kb_used_data":10872,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93424444,"statfs":{"total":95995035648,"available":95666630656,"internally_reserved":0,"allocated":11132928,"data_stored":7530417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626821,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":739720,"kb_used_data":438200,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93005432,"statfs":{"total":95995035648,"available":95237562368,"internally_reserved":0,"allocated":448716800,"data_stored":445122053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823050,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312388,"kb_used_data":10868,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93432764,"statfs":{"total":95995035648,"available":95675150336,"internally_reserved":0,"allocated":11128832,"data_stored":7534543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019278,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312140,"kb_used_data":10620,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93433012,"statfs":{"total":95995035648,"available":95675404288,"internally_reserved":0,"allocated":10874880,"data_stored":7276086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059592935,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":233628,"kb_used_data":10896,"kb_used_omap":26,"kb_used_meta":222693,"kb_avail":93511524,"statfs":{"total":95995035648,"available":95755800576,"internally_reserved":0,"allocated":11157504,"data_stored":7581808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":228037792},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":591,"seq":2538325672174,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":436404,"kb_used_data":10852,"kb_used_omap":41,"kb_used_meta":425494,"kb_avail":93308748,"statfs":{"total":95995035648,"available":95548157952,"internally_reserved":0,"allocated":11112448,"data_stored":7526089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42651,"internal_metadata":435706213},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":557,"seq":2392296784117,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":407928,"kb_used_data":10600,"kb_used_omap":32,"kb_used_meta":397279,"kb_avail":93337224,"statfs":{"total":95995035648,"available":95577317376,"internally_reserved":0,"allocated":10854400,"data_stored":7271251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33752,"internal_metadata":406813736},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":524,"seq":2250562863355,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":336544,"kb_used_data":10900,"kb_used_omap":41,"kb_used_meta":325590,"kb_avail":93408608,"statfs":{"total":95995035648,"available":95650414592,"internally_reserved":0,"allocated":11161600,"data_stored":7582334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42694,"internal_metadata":333404474},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:49:25.152 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-13T14:49:25.156 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:25.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:25.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:25.257 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:25.281 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:25.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:25.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:25.309 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:49:25.383 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:25.385 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:25 smithi151 ceph-mon[104450]: pgmap v1674: 119 pgs: 6 unknown, 113 active+clean; 582 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:25.385 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:25 smithi151 ceph-mon[108966]: pgmap v1674: 119 pgs: 6 unknown, 113 active+clean; 582 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:25.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:25.407 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:25.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:25.416 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:49:25.416 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:49:25.416 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:49:25.416 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:49:25.419 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:49:25.419 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:49:25.436 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:49:25.436 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:49:25.455 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:49:25.455 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:49:25.484 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:49:25.484 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:49:25.506 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:49:25.507 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:49:25.508 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:25.522 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:49:25.523 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:49:25.531 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:25.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:25.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:25.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:25 smithi184 ceph-mon[92527]: pgmap v1674: 119 pgs: 6 unknown, 113 active+clean; 582 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:25.632 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:25.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:25.657 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:25.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:25.758 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:25.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:25.782 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:25.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:25.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:25.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:25.905 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:25.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:26.006 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:26.028 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:26.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:26.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:26.130 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:26.152 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:26.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:26.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:26.253 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:26.278 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:26.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:26.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:26.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:26 smithi151 ceph-mon[108966]: from='client.35907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:49:26.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:26 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/368080031' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:26.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:26 smithi151 ceph-mon[104450]: from='client.35907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:49:26.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:26 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/368080031' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:26.380 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:26.402 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:26.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:26.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:26.504 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:26.526 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:26.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:26.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:26.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:26 smithi184 ceph-mon[92527]: from='client.35907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:49:26.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:26 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/368080031' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:26.627 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:26.653 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:26.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:26.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:26.755 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:26.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:26.778 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:26.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:26.880 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:26.901 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:26.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:26.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:27.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:27.024 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:27.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:27.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:27.126 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:27.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:27.152 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:27.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:27.253 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:27.278 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:27.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:27.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:27.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:27 smithi151 ceph-mon[104450]: pgmap v1675: 119 pgs: 119 active+clean; 582 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:27.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:27 smithi151 ceph-mon[108966]: pgmap v1675: 119 pgs: 119 active+clean; 582 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:27.380 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:27.404 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:27.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:27.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:27.506 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:27.528 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:27.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:27.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:27.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:27 smithi184 ceph-mon[92527]: pgmap v1675: 119 pgs: 119 active+clean; 582 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:27.629 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:27.654 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:27.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:27.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:27.733 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:27.756 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:27.780 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:27.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:27.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:27.882 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:27.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:27.905 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:27.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:28.006 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:28.031 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:28.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:28.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:28.133 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:28.156 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:28.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:28.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:28.258 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:28.280 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:28.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:28.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:28.382 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:28.403 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:28.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:28.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:28.457 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:49:28.458 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:49:28.505 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:28.530 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:28.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:28.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:28.631 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:28.656 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:28.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:28.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:28.758 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:28.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:28.782 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:28.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:28.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:28.905 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:28.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:28.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:29.006 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:29.028 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:29.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:29.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:29.077 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T14:49:29.130 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:29.131 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:49:29.153 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:29.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:29.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:29.216 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:29.216 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:29.255 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:29.278 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:29.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:29.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:29.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:29 smithi151 ceph-mon[104450]: pgmap v1676: 119 pgs: 119 active+clean; 582 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:29.345 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:29 smithi151 ceph-mon[108966]: pgmap v1676: 119 pgs: 119 active+clean; 582 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:29.380 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:29.402 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:29.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:29.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:29.504 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:29.528 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:29.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:29.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:29.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:29 smithi184 ceph-mon[92527]: pgmap v1676: 119 pgs: 119 active+clean; 582 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:29.630 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:29.655 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:29.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:29.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:29.757 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:29.779 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:29.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:29.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:29.881 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:29.906 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:29.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:29.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:30.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:30.030 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:30.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:30.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:30.132 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:30.158 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:30.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:30.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:30.260 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:30.286 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:30.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:30.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:30.387 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:30.409 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:30.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:30.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:30.511 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:30.532 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:30.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:30.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:30.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:49:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:49:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:49:30.634 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:30.659 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:30.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:30.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:30.761 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:30.783 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:30.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:30.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:30.884 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:30.909 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:30.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:30.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:31.011 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:31.033 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:31.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:31.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:31.134 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:31.160 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:31.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:31.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:31.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:31.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:31.285 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:31.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:31.386 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:31.397 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:31 smithi151 ceph-mon[104450]: pgmap v1677: 119 pgs: 119 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:31.398 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:31 smithi151 ceph-mon[108966]: pgmap v1677: 119 pgs: 119 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:31.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:31.411 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:31.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:31.512 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:31.534 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:31.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:31.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:31.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:31 smithi184 ceph-mon[92527]: pgmap v1677: 119 pgs: 119 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:31.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:31.657 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:31.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:31.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:31.759 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:31.780 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:31.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:31.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:31.882 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:31.904 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:31.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:31.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:32.006 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:32.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:32.028 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:32.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:32.129 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:32.150 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:32.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:32.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:32.252 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:32.278 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:32.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:32.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:32.379 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:32.403 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:32.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:32.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:32.505 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:32.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:32.528 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:32.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:32.534 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:49:32.534 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:32.534 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:32 smithi151 ceph-mon[104450]: pgmap v1678: 119 pgs: 119 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:32.534 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:49:32.534 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:49:32.534 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:32.534 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:32 smithi151 ceph-mon[108966]: pgmap v1678: 119 pgs: 119 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:32.534 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:49:32.629 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:32.655 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:32.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:32.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:32.757 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:32.783 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:32.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:32.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:49:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:32 smithi184 ceph-mon[92527]: pgmap v1678: 119 pgs: 119 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:49:32.885 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:32.908 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:32.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:32.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:33.010 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:33.036 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:33.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:33.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:33.138 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:33.159 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:33.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:33.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:33.261 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:33.286 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:33.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:33.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:33.389 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:33.411 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:33.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:33.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:33.513 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:33.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:33.540 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:33.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:33.641 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:33.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:33.666 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:33.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:33.767 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:33.792 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:33.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:33.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:33.894 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:33.919 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:33.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:33.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:34.021 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:34.046 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:34.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:34.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:34.148 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:34.170 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:34.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:34.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:34.271 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:34.296 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:34.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:34.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:34.398 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:34.420 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:34.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:34.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:34.522 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:34.544 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:34.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:34.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:34.601 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-13T14:49:34.635 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:34 smithi151 ceph-mon[104450]: pgmap v1679: 119 pgs: 119 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:34.635 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:34 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1772355538' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:34.636 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:34 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:34.636 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:34 smithi151 ceph-mon[108966]: pgmap v1679: 119 pgs: 119 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:34.636 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:34 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1772355538' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:34.636 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:34 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:34.636 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:49:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:49:34] "GET /metrics HTTP/1.1" 200 36575 "" "Prometheus/2.51.0" 2024-09-13T14:49:34.646 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:34.671 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:34.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:34.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:34.773 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:34.797 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:34.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:34.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:34.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:34 smithi184 ceph-mon[92527]: pgmap v1679: 119 pgs: 119 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:34.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:34 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1772355538' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:34.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:34 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:34.899 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:34.923 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:34.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:34.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:35.025 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:35.049 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:35.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:35.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:35.151 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:35.175 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:35.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:35.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:35.277 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:35.301 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:35.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:35.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:35.403 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:35.426 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:35.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:35.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:35.528 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:35.554 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:35.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:35.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:35.600 INFO:teuthology.orchestra.run.smithi151.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-09-13T14:49:35.655 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:35.680 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:35.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:35.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:35.744 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:35 smithi151 ceph-mon[104450]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:49:35.744 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:35 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:49:35.744 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:35 smithi151 ceph-mon[104450]: osdmap e1170: 8 total, 8 up, 8 in 2024-09-13T14:49:35.744 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:35 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1306495182' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:49:35.745 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:35 smithi151 ceph-mon[108966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:49:35.745 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:35 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:49:35.745 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:35 smithi151 ceph-mon[108966]: osdmap e1170: 8 total, 8 up, 8 in 2024-09-13T14:49:35.745 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:35 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1306495182' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:49:35.750 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:35.751 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:35.782 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:35.805 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:35.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:35.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:35.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:35 smithi184 ceph-mon[92527]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:49:35.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:35 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:49:35.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:35 smithi184 ceph-mon[92527]: osdmap e1170: 8 total, 8 up, 8 in 2024-09-13T14:49:35.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:35 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1306495182' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:49:35.907 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:35.929 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:35.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:35.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:35.988 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:49:35.988 INFO:teuthology.orchestra.run.smithi151.stdout:{"fsid":"2401fc50-71da-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1357,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":1171,"num_osds":8,"num_up_osds":8,"osd_up_since":1726237814,"num_in_osds":8,"osd_in_since":1726238586,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":119}],"num_pgs":119,"num_pools":5,"num_objects":26,"data_bytes":595917,"bytes_used":3154341888,"bytes_avail":764805943296,"bytes_total":767960285184},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi151.rqqklw","status":"up:active","gid":24614}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.151:8443/","prometheus":"http://172.21.15.151:9283/"}},"servicemap":{"epoch":24,"modified":"2024-09-13T14:45:41.438774+0000","services":{}},"progress_events":{"b24a386d-2177-4b0f-a83d-e6638be369da":{"message":"Upgrade to 19.3.0-4697-gd58f7306 (4m)\n [============................] (remaining: 6m)","progress":0.44999998807907104,"add_to_ceph_s":true}}} 2024-09-13T14:49:36.031 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:36.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:36.054 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:36.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:36.154 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:36.181 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:36.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:36.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:36.283 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:36.309 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:36.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:36.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:36.410 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:36.435 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:36.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:36.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:36.537 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:36.560 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:36.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:36.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:36.662 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:36.685 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:36.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:36.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:36.787 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:36.813 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:36.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:36.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:36.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:36 smithi184 ceph-mon[92527]: pgmap v1681: 119 pgs: 119 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:36.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:36 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1306495182' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:49:36.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:36 smithi184 ceph-mon[92527]: osdmap e1171: 8 total, 8 up, 8 in 2024-09-13T14:49:36.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:36 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/769284679' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:49:36.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:36 smithi151 ceph-mon[104450]: pgmap v1681: 119 pgs: 119 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:36.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:36 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1306495182' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:49:36.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:36 smithi151 ceph-mon[104450]: osdmap e1171: 8 total, 8 up, 8 in 2024-09-13T14:49:36.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:36 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/769284679' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:49:36.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:36 smithi151 ceph-mon[108966]: pgmap v1681: 119 pgs: 119 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:36.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:36 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1306495182' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:49:36.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:36 smithi151 ceph-mon[108966]: osdmap e1171: 8 total, 8 up, 8 in 2024-09-13T14:49:36.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:36 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/769284679' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:49:36.915 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:36.937 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:36.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:36.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:37.038 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:37.065 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:37.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:37.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:37.166 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:37.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:37.193 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:37.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:37.293 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:37.318 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:37.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:37.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:37.420 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:37.444 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:37.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:37.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:37.546 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:37.552 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:49:37.570 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:37.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:37.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:37.663 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-13T14:49:37.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:49:37.671 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:37.727 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:37.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:37.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:37.829 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:37.851 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:37.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:37.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:37.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:37 smithi151 ceph-mon[104450]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:49:37.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:37 smithi151 ceph-mon[104450]: Cluster is now healthy 2024-09-13T14:49:37.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:37 smithi151 ceph-mon[108966]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:49:37.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:37 smithi151 ceph-mon[108966]: Cluster is now healthy 2024-09-13T14:49:37.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:37.974 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:37.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:37.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:38.076 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:38.102 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:38.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:38.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:38.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:37 smithi184 ceph-mon[92527]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:49:38.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:37 smithi184 ceph-mon[92527]: Cluster is now healthy 2024-09-13T14:49:38.204 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:38.229 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:38.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:38.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:38.331 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:38.357 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:38.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:38.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:38.459 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:38.484 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:38.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:38.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:38.586 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:38.608 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:38.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:38.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:38.710 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:38.732 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:38.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:38.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:38.834 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:38.855 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:38.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:38.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:38.957 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:38.978 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:38.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:38.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:39.080 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:39.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:38 smithi184 ceph-mon[92527]: pgmap v1683: 119 pgs: 119 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:39.106 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:39.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:39.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:39.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:38 smithi151 ceph-mon[104450]: pgmap v1683: 119 pgs: 119 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:39.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:38 smithi151 ceph-mon[108966]: pgmap v1683: 119 pgs: 119 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:39.150 DEBUG:teuthology.orchestra.run.smithi151:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-09-13T14:49:39.152 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T14:49:39.153 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T14:49:39.208 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:39.215 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:hints = 1 2024-09-13T14:49:39.215 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T14:49:39.215 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Object prefix: benchmark_data_smithi151_229097 2024-09-13T14:49:39.216 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:49:39.216 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 0 0 0 0 0 0 - 0 2024-09-13T14:49:39.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:39.242 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:39.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:39.343 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:39.385 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:39.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:39.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:39.488 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:39.527 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:39.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:39.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:39.630 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:39.654 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:39.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:39.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:39.756 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:39.779 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:39.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:39.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:39.881 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:39.912 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:39.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:39.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:40.014 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:40.047 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:40.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:40.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:40.149 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:40.180 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:40.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:40.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:40.215 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 1 16 3040 3024 188.995 189 0.00433388 0.00527432 2024-09-13T14:49:40.281 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:40.315 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:40.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:40.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:40.416 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:40.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:40.456 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:40.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:40.557 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:40.588 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:40.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:40.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:40.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:49:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:49:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:49:40.690 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:40.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:40 smithi151 ceph-mon[104450]: pgmap v1684: 119 pgs: 119 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:40.697 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:40 smithi151 ceph-mon[108966]: pgmap v1684: 119 pgs: 119 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:40.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:40.729 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:40.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:40.753 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:49:40.754 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:49:40.830 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:40.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:40.865 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:40.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:40.966 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:40.998 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:40.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:40.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:41.100 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:41.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:40 smithi184 ceph-mon[92527]: pgmap v1684: 119 pgs: 119 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-13T14:49:41.130 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:41.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:41.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:41.216 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 2 15 6612 6597 206.114 223.312 0.00592483 0.00484198 2024-09-13T14:49:41.231 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:41.261 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:41.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:41.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:41.298 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:41.363 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:41.394 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:41.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:41.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:41.496 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:41.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:41.537 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:41.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:41.638 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:41.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:41.668 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:41.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:41.770 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:41.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:41.810 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:41.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:41.911 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:41.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:41.949 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:41.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:42.050 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:42.084 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:42.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:42.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:42.135 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T14:49:42.186 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:42.216 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:42.217 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 3 16 9676 9660 201.211 191.438 0.00488216 0.00496279 2024-09-13T14:49:42.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:42.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:42.318 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:42.332 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:42 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:49:42.332 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:42 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:49:42.333 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:42 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:49:42.333 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:42 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:49:42.333 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:42 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:49:42.333 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:42 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:49:42.333 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:42 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:49:42.333 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:42 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:49:42.333 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:42 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:49:42.333 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:42 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:49:42.343 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:42.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:42.346 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:42.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:42.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:42 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:49:42.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:42 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:49:42.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:42 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:49:42.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:42 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:49:42.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:42 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:49:42.438 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:49:42.438 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:49:42.448 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:42.473 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:49:42.473 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:49:42.474 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:42.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:42.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:42.506 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:49:42.506 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:49:42.539 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:49:42.539 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:49:42.569 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:49:42.570 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:49:42.575 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:42.604 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:49:42.604 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:49:42.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:42.618 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:42.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:42.632 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:49:42.632 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:49:42.665 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:49:42.666 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:49:42.719 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:42.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:42.753 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:42.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:42.854 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:42.883 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:42.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:42.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:42.985 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:43.018 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:43.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:43.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:43.120 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:43.148 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:43.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:43.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:43.216 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 4 16 12332 12316 192.405 166 0.00293502 0.00518977 2024-09-13T14:49:43.250 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:43.288 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:43.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:43.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:43.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:43 smithi184 ceph-mon[92527]: pgmap v1685: 119 pgs: 119 active+clean; 34 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 4.2 MiB/s wr, 66 op/s 2024-09-13T14:49:43.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:43 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/19012893' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:43.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:43 smithi151 ceph-mon[104450]: pgmap v1685: 119 pgs: 119 active+clean; 34 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 4.2 MiB/s wr, 66 op/s 2024-09-13T14:49:43.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:43 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/19012893' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:43.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:43 smithi151 ceph-mon[108966]: pgmap v1685: 119 pgs: 119 active+clean; 34 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 4.2 MiB/s wr, 66 op/s 2024-09-13T14:49:43.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:43 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/19012893' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:43.390 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:43.429 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:43.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:43.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:43.531 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:43.565 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:43.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:43.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:43.667 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:43.699 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:43.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:43.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:43.801 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:43.832 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:43.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:43.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:43.934 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:43.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:43.967 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:43.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:44.068 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:44.107 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:44.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:44.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:44.209 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:44.216 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 5 16 15412 15396 192.421 192.5 0.00539449 0.00519031 2024-09-13T14:49:44.241 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:44.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:44.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:44.343 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:44.381 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:44.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:44.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:44.483 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:44.521 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:44.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:44.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:44.624 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:44.660 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:44.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:44.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:44.762 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:44.801 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:44.802 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:49:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:49:44] "GET /metrics HTTP/1.1" 200 36575 "" "Prometheus/2.51.0" 2024-09-13T14:49:44.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:44.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:44.904 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:44.933 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:44.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:44.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:45.035 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:45.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:45.070 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:45.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:45.171 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:45.205 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:45.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:45.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:45.216 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 6 16 18165 18149 189.025 172.062 0.00456949 0.00528381 2024-09-13T14:49:45.307 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:45.339 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:45.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:45.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:45.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:45 smithi184 ceph-mon[92527]: pgmap v1686: 119 pgs: 119 active+clean; 471 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 53 MiB/s wr, 841 op/s 2024-09-13T14:49:45.355 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:45 smithi151 ceph-mon[104450]: pgmap v1686: 119 pgs: 119 active+clean; 471 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 53 MiB/s wr, 841 op/s 2024-09-13T14:49:45.356 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:45 smithi151 ceph-mon[108966]: pgmap v1686: 119 pgs: 119 active+clean; 471 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 53 MiB/s wr, 841 op/s 2024-09-13T14:49:45.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:45.491 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:45.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:45.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:45.592 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:45.635 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:45.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:45.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:45.737 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:45.767 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:45.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:45.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:45.869 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:45.896 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:45.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:45.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:45.998 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:46.031 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:46.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:46.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:46.133 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:49:46.135 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T14:49:46.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:46.171 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:46.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:46.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:46.216 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 7 16 20692 20676 184.583 157.938 0.0037767 0.00541143 2024-09-13T14:49:46.273 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:46.300 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:46.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:46.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:46.402 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:46.441 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:46.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:46.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:46.543 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:46.583 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:46.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:46.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:46.685 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:46.719 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:46.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:46.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:46.821 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:46.861 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:46.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:46.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:46.963 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:46.997 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:46.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:46.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:47.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:47.140 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:47.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:47.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:47.217 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 8 16 23396 23380 182.633 169 0.0129611 0.00547011 2024-09-13T14:49:47.241 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:47.272 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:47.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:47.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:47.336 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:47 smithi151 ceph-mon[104450]: pgmap v1687: 119 pgs: 119 active+clean; 828 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-09-13T14:49:47.337 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:47 smithi151 ceph-mon[104450]: from='client.45809 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:49:47.337 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:47 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:49:47.337 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:47 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:47.337 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:47 smithi151 ceph-mon[108966]: pgmap v1687: 119 pgs: 119 active+clean; 828 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-09-13T14:49:47.337 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:47 smithi151 ceph-mon[108966]: from='client.45809 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:49:47.337 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:47 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:49:47.337 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:47 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:47.343 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:47.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:47 smithi184 ceph-mon[92527]: pgmap v1687: 119 pgs: 119 active+clean; 828 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-09-13T14:49:47.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:47 smithi184 ceph-mon[92527]: from='client.45809 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:49:47.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:47 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:49:47.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:47 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:47.374 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:47.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:47.409 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:47.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:47.510 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:47.557 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:47.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:47.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:47.659 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:47.696 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:47.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:47.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:47.798 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:47.836 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:47.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:47.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:47.938 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:47.976 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:47.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:47.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:48.078 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:48.114 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:48.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:48.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:48.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:48.217 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 9 16 25782 25766 178.908 149.125 0.00687273 0.00558397 2024-09-13T14:49:48.251 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:48.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:48.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:48.300 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:49:48.353 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:48.386 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:48.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:48.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:48.412 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1687,"stamp":"2024-09-13T14:49:45.542969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":868554701,"num_objects":13270,"num_object_clones":0,"num_object_copies":39810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13307,"num_write_kb":847647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":327718,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16402,"ondisk_log_size":16402,"up":357,"acting":357,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":358,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5427320,"kb_used_data":2543608,"kb_used_omap":149,"kb_used_meta":2883498,"kb_avail":744533896,"statfs":{"total":767960285184,"available":762402709504,"internally_reserved":0,"allocated":2604654592,"data_stored":2575828887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":153401,"internal_metadata":2952702151},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,3,0,0,0,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":867958784,"num_objects":13244,"num_object_clones":0,"num_object_copies":39732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13244,"num_write_kb":847616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003683"},"pg_stats":[{"pgid":"1.1d","version":"261'100","reported_seq":113,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.209031+0000","last_change":"2024-09-13T14:45:19.807356+0000","last_active":"2024-09-13T14:49:36.209031+0000","last_peered":"2024-09-13T14:49:36.209031+0000","last_clean":"2024-09-13T14:49:36.209031+0000","last_became_active":"2024-09-13T14:45:19.807081+0000","last_became_peered":"2024-09-13T14:45:19.807081+0000","last_unstale":"2024-09-13T14:49:36.209031+0000","last_undegraded":"2024-09-13T14:49:36.209031+0000","last_fullsized":"2024-09-13T14:49:36.209031+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:17.988345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:17.988345+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:41:05.216335+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":1719,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.132288+0000","last_change":"2024-09-13T14:29:42.541263+0000","last_active":"2024-09-13T14:49:36.132288+0000","last_peered":"2024-09-13T14:49:36.132288+0000","last_clean":"2024-09-13T14:49:36.132288+0000","last_became_active":"2024-09-13T14:29:42.540873+0000","last_became_peered":"2024-09-13T14:29:42.540873+0000","last_unstale":"2024-09-13T14:49:36.132288+0000","last_undegraded":"2024-09-13T14:49:36.132288+0000","last_fullsized":"2024-09-13T14:49:36.132288+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:20.611206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1131,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.598786+0000","last_change":"2024-09-13T14:43:07.516715+0000","last_active":"2024-09-13T14:49:34.598786+0000","last_peered":"2024-09-13T14:49:34.598786+0000","last_clean":"2024-09-13T14:49:34.598786+0000","last_became_active":"2024-09-13T14:43:07.516403+0000","last_became_peered":"2024-09-13T14:43:07.516403+0000","last_unstale":"2024-09-13T14:49:34.598786+0000","last_undegraded":"2024-09-13T14:49:34.598786+0000","last_fullsized":"2024-09-13T14:49:34.598786+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:56.469690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":58,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.602887+0000","last_change":"2024-09-13T14:45:19.810203+0000","last_active":"2024-09-13T14:49:35.602887+0000","last_peered":"2024-09-13T14:49:35.602887+0000","last_clean":"2024-09-13T14:49:35.602887+0000","last_became_active":"2024-09-13T14:45:19.809693+0000","last_became_peered":"2024-09-13T14:45:19.809693+0000","last_unstale":"2024-09-13T14:49:35.602887+0000","last_undegraded":"2024-09-13T14:49:35.602887+0000","last_fullsized":"2024-09-13T14:49:35.602887+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:20.219673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":57,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.602159+0000","last_change":"2024-09-13T14:45:19.807991+0000","last_active":"2024-09-13T14:49:35.602159+0000","last_peered":"2024-09-13T14:49:35.602159+0000","last_clean":"2024-09-13T14:49:35.602159+0000","last_became_active":"2024-09-13T14:45:19.807662+0000","last_became_peered":"2024-09-13T14:45:19.807662+0000","last_unstale":"2024-09-13T14:49:35.602159+0000","last_undegraded":"2024-09-13T14:49:35.602159+0000","last_fullsized":"2024-09-13T14:49:35.602159+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:29.306108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"261'100","reported_seq":1242,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.598480+0000","last_change":"2024-09-13T14:44:54.259562+0000","last_active":"2024-09-13T14:49:34.598480+0000","last_peered":"2024-09-13T14:49:34.598480+0000","last_clean":"2024-09-13T14:49:34.598480+0000","last_became_active":"2024-09-13T14:43:14.563717+0000","last_became_peered":"2024-09-13T14:43:14.563717+0000","last_unstale":"2024-09-13T14:49:34.598480+0000","last_undegraded":"2024-09-13T14:49:34.598480+0000","last_fullsized":"2024-09-13T14:49:34.598480+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_clean_scrub_stamp":"2024-09-13T14:44:54.259482+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:43:52.729109+0000","scrub_duration":0.015265569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":1099,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.744325+0000","last_change":"2024-09-13T14:19:25.873514+0000","last_active":"2024-09-13T14:49:34.744325+0000","last_peered":"2024-09-13T14:49:34.744325+0000","last_clean":"2024-09-13T14:49:34.744325+0000","last_became_active":"2024-09-13T14:19:25.873170+0000","last_became_peered":"2024-09-13T14:19:25.873170+0000","last_unstale":"2024-09-13T14:49:34.744325+0000","last_undegraded":"2024-09-13T14:49:34.744325+0000","last_fullsized":"2024-09-13T14:49:34.744325+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:48.298530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1128,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.599320+0000","last_change":"2024-09-13T14:30:15.580011+0000","last_active":"2024-09-13T14:49:34.599320+0000","last_peered":"2024-09-13T14:49:34.599320+0000","last_clean":"2024-09-13T14:49:34.599320+0000","last_became_active":"2024-09-13T14:30:15.579610+0000","last_became_peered":"2024-09-13T14:30:15.579610+0000","last_unstale":"2024-09-13T14:49:34.599320+0000","last_undegraded":"2024-09-13T14:49:34.599320+0000","last_fullsized":"2024-09-13T14:49:34.599320+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:11:02.733771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"261'100","reported_seq":75,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.603021+0000","last_change":"2024-09-13T14:45:35.947332+0000","last_active":"2024-09-13T14:49:35.603021+0000","last_peered":"2024-09-13T14:49:35.603021+0000","last_clean":"2024-09-13T14:49:35.603021+0000","last_became_active":"2024-09-13T14:45:19.810775+0000","last_became_peered":"2024-09-13T14:45:19.810775+0000","last_unstale":"2024-09-13T14:49:35.603021+0000","last_undegraded":"2024-09-13T14:49:35.603021+0000","last_fullsized":"2024-09-13T14:49:35.603021+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:35.947265+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:43:41.569795+0000","last_clean_scrub_stamp":"2024-09-13T14:45:35.947265+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:00.606370+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":1144,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.744185+0000","last_change":"2024-09-13T14:43:07.536116+0000","last_active":"2024-09-13T14:49:34.744185+0000","last_peered":"2024-09-13T14:49:34.744185+0000","last_clean":"2024-09-13T14:49:34.744185+0000","last_became_active":"2024-09-13T14:43:07.524112+0000","last_became_peered":"2024-09-13T14:43:07.524112+0000","last_unstale":"2024-09-13T14:49:34.744185+0000","last_undegraded":"2024-09-13T14:49:34.744185+0000","last_fullsized":"2024-09-13T14:49:34.744185+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:36.947060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1141,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.599273+0000","last_change":"2024-09-13T14:43:07.530236+0000","last_active":"2024-09-13T14:49:34.599273+0000","last_peered":"2024-09-13T14:49:34.599273+0000","last_clean":"2024-09-13T14:49:34.599273+0000","last_became_active":"2024-09-13T14:43:07.529967+0000","last_became_peered":"2024-09-13T14:43:07.529967+0000","last_unstale":"2024-09-13T14:49:34.599273+0000","last_undegraded":"2024-09-13T14:49:34.599273+0000","last_fullsized":"2024-09-13T14:49:34.599273+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:27.276550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":1703,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.131887+0000","last_change":"2024-09-13T14:28:35.612469+0000","last_active":"2024-09-13T14:49:36.131887+0000","last_peered":"2024-09-13T14:49:36.131887+0000","last_clean":"2024-09-13T14:49:36.131887+0000","last_became_active":"2024-09-13T14:28:35.612093+0000","last_became_peered":"2024-09-13T14:28:35.612093+0000","last_unstale":"2024-09-13T14:49:36.131887+0000","last_undegraded":"2024-09-13T14:49:36.131887+0000","last_fullsized":"2024-09-13T14:49:36.131887+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:51.022663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1726,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.209564+0000","last_change":"2024-09-13T14:45:19.808811+0000","last_active":"2024-09-13T14:49:36.209564+0000","last_peered":"2024-09-13T14:49:36.209564+0000","last_clean":"2024-09-13T14:49:36.209564+0000","last_became_active":"2024-09-13T14:45:19.808471+0000","last_became_peered":"2024-09-13T14:45:19.808471+0000","last_unstale":"2024-09-13T14:49:36.209564+0000","last_undegraded":"2024-09-13T14:49:36.209564+0000","last_fullsized":"2024-09-13T14:49:36.209564+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:53.638161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"261'100","reported_seq":57,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.602256+0000","last_change":"2024-09-13T14:45:19.813412+0000","last_active":"2024-09-13T14:49:35.602256+0000","last_peered":"2024-09-13T14:49:35.602256+0000","last_clean":"2024-09-13T14:49:35.602256+0000","last_became_active":"2024-09-13T14:45:19.812585+0000","last_became_peered":"2024-09-13T14:45:19.812585+0000","last_unstale":"2024-09-13T14:49:35.602256+0000","last_undegraded":"2024-09-13T14:49:35.602256+0000","last_fullsized":"2024-09-13T14:49:35.602256+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.614340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.614340+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:32:07.434109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":1150,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.599423+0000","last_change":"2024-09-13T14:43:07.522950+0000","last_active":"2024-09-13T14:49:34.599423+0000","last_peered":"2024-09-13T14:49:34.599423+0000","last_clean":"2024-09-13T14:49:34.599423+0000","last_became_active":"2024-09-13T14:43:07.522566+0000","last_became_peered":"2024-09-13T14:43:07.522566+0000","last_unstale":"2024-09-13T14:49:34.599423+0000","last_undegraded":"2024-09-13T14:49:34.599423+0000","last_fullsized":"2024-09-13T14:49:34.599423+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:03.201478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":57,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.602788+0000","last_change":"2024-09-13T14:45:19.807231+0000","last_active":"2024-09-13T14:49:35.602788+0000","last_peered":"2024-09-13T14:49:35.602788+0000","last_clean":"2024-09-13T14:49:35.602788+0000","last_became_active":"2024-09-13T14:45:19.806936+0000","last_became_peered":"2024-09-13T14:45:19.806936+0000","last_unstale":"2024-09-13T14:49:35.602788+0000","last_undegraded":"2024-09-13T14:49:35.602788+0000","last_fullsized":"2024-09-13T14:49:35.602788+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:04:26.723153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.18","version":"261'100","reported_seq":120,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.743927+0000","last_change":"2024-09-13T14:44:41.685537+0000","last_active":"2024-09-13T14:49:34.743927+0000","last_peered":"2024-09-13T14:49:34.743927+0000","last_clean":"2024-09-13T14:49:34.743927+0000","last_became_active":"2024-09-13T14:43:14.558502+0000","last_became_peered":"2024-09-13T14:43:14.558502+0000","last_unstale":"2024-09-13T14:49:34.743927+0000","last_undegraded":"2024-09-13T14:49:34.743927+0000","last_fullsized":"2024-09-13T14:49:34.743927+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1134,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:41.685124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:41.685124+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:37:58.412497+0000","scrub_duration":0.0075205970000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":106,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.210457+0000","last_change":"2024-09-13T14:43:07.516845+0000","last_active":"2024-09-13T14:49:36.210457+0000","last_peered":"2024-09-13T14:49:36.210457+0000","last_clean":"2024-09-13T14:49:36.210457+0000","last_became_active":"2024-09-13T14:43:07.514974+0000","last_became_peered":"2024-09-13T14:43:07.514974+0000","last_unstale":"2024-09-13T14:49:36.210457+0000","last_undegraded":"2024-09-13T14:49:36.210457+0000","last_fullsized":"2024-09-13T14:49:36.210457+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:55.419483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1741,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.131528+0000","last_change":"2024-09-13T14:45:19.815341+0000","last_active":"2024-09-13T14:49:36.131528+0000","last_peered":"2024-09-13T14:49:36.131528+0000","last_clean":"2024-09-13T14:49:36.131528+0000","last_became_active":"2024-09-13T14:45:19.810262+0000","last_became_peered":"2024-09-13T14:45:19.810262+0000","last_unstale":"2024-09-13T14:49:36.131528+0000","last_undegraded":"2024-09-13T14:49:36.131528+0000","last_fullsized":"2024-09-13T14:49:36.131528+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:39.847653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1b","version":"261'100","reported_seq":62,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.744161+0000","last_change":"2024-09-13T14:44:52.803330+0000","last_active":"2024-09-13T14:49:34.744161+0000","last_peered":"2024-09-13T14:49:34.744161+0000","last_clean":"2024-09-13T14:49:34.744161+0000","last_became_active":"2024-09-13T14:43:14.556712+0000","last_became_peered":"2024-09-13T14:43:14.556712+0000","last_unstale":"2024-09-13T14:49:34.744161+0000","last_undegraded":"2024-09-13T14:49:34.744161+0000","last_fullsized":"2024-09-13T14:49:34.744161+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:52.803242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:52.803242+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:06.658301+0000","scrub_duration":0.0019769660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":57,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.602085+0000","last_change":"2024-09-13T14:45:19.806623+0000","last_active":"2024-09-13T14:49:35.602085+0000","last_peered":"2024-09-13T14:49:35.602085+0000","last_clean":"2024-09-13T14:49:35.602085+0000","last_became_active":"2024-09-13T14:45:19.806219+0000","last_became_peered":"2024-09-13T14:45:19.806219+0000","last_unstale":"2024-09-13T14:49:35.602085+0000","last_undegraded":"2024-09-13T14:49:35.602085+0000","last_fullsized":"2024-09-13T14:49:35.602085+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:34:41.601201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1124,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.744341+0000","last_change":"2024-09-13T14:43:32.637751+0000","last_active":"2024-09-13T14:49:34.744341+0000","last_peered":"2024-09-13T14:49:34.744341+0000","last_clean":"2024-09-13T14:49:34.744341+0000","last_became_active":"2024-09-13T14:43:32.636901+0000","last_became_peered":"2024-09-13T14:43:32.636901+0000","last_unstale":"2024-09-13T14:49:34.744341+0000","last_undegraded":"2024-09-13T14:49:34.744341+0000","last_fullsized":"2024-09-13T14:49:34.744341+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:21.942419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1a","version":"261'100","reported_seq":62,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.599201+0000","last_change":"2024-09-13T14:43:41.910041+0000","last_active":"2024-09-13T14:49:34.599201+0000","last_peered":"2024-09-13T14:49:34.599201+0000","last_clean":"2024-09-13T14:49:34.599201+0000","last_became_active":"2024-09-13T14:43:14.555083+0000","last_became_peered":"2024-09-13T14:43:14.555083+0000","last_unstale":"2024-09-13T14:49:34.599201+0000","last_undegraded":"2024-09-13T14:49:34.599201+0000","last_fullsized":"2024-09-13T14:49:34.599201+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:41.909988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:41.909988+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:42:19.954838+0000","scrub_duration":0.0012891619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":57,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.602226+0000","last_change":"2024-09-13T14:45:19.812071+0000","last_active":"2024-09-13T14:49:35.602226+0000","last_peered":"2024-09-13T14:49:35.602226+0000","last_clean":"2024-09-13T14:49:35.602226+0000","last_became_active":"2024-09-13T14:45:19.811336+0000","last_became_peered":"2024-09-13T14:45:19.811336+0000","last_unstale":"2024-09-13T14:49:35.602226+0000","last_undegraded":"2024-09-13T14:49:35.602226+0000","last_fullsized":"2024-09-13T14:49:35.602226+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:40:29.814608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1758,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.131544+0000","last_change":"2024-09-13T14:45:19.810478+0000","last_active":"2024-09-13T14:49:36.131544+0000","last_peered":"2024-09-13T14:49:36.131544+0000","last_clean":"2024-09-13T14:49:36.131544+0000","last_became_active":"2024-09-13T14:45:19.808278+0000","last_became_peered":"2024-09-13T14:45:19.808278+0000","last_unstale":"2024-09-13T14:49:36.131544+0000","last_undegraded":"2024-09-13T14:49:36.131544+0000","last_fullsized":"2024-09-13T14:49:36.131544+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:12.211089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"261'100","reported_seq":62,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.599253+0000","last_change":"2024-09-13T14:44:38.214961+0000","last_active":"2024-09-13T14:49:34.599253+0000","last_peered":"2024-09-13T14:49:34.599253+0000","last_clean":"2024-09-13T14:49:34.599253+0000","last_became_active":"2024-09-13T14:43:14.557883+0000","last_became_peered":"2024-09-13T14:43:14.557883+0000","last_unstale":"2024-09-13T14:49:34.599253+0000","last_undegraded":"2024-09-13T14:49:34.599253+0000","last_fullsized":"2024-09-13T14:49:34.599253+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:38.214865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:38.214865+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:49:36.013783+0000","scrub_duration":0.0016389639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1629,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.602425+0000","last_change":"2024-09-13T14:30:15.576921+0000","last_active":"2024-09-13T14:49:35.602425+0000","last_peered":"2024-09-13T14:49:35.602425+0000","last_clean":"2024-09-13T14:49:35.602425+0000","last_became_active":"2024-09-13T14:30:15.576607+0000","last_became_peered":"2024-09-13T14:30:15.576607+0000","last_unstale":"2024-09-13T14:49:35.602425+0000","last_undegraded":"2024-09-13T14:49:35.602425+0000","last_fullsized":"2024-09-13T14:49:35.602425+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:35.162460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1134,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.744226+0000","last_change":"2024-09-13T14:43:07.518801+0000","last_active":"2024-09-13T14:49:34.744226+0000","last_peered":"2024-09-13T14:49:34.744226+0000","last_clean":"2024-09-13T14:49:34.744226+0000","last_became_active":"2024-09-13T14:43:07.517610+0000","last_became_peered":"2024-09-13T14:43:07.517610+0000","last_unstale":"2024-09-13T14:49:34.744226+0000","last_undegraded":"2024-09-13T14:49:34.744226+0000","last_fullsized":"2024-09-13T14:49:34.744226+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:51.632254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"261'100","reported_seq":106,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.209691+0000","last_change":"2024-09-13T14:43:15.037565+0000","last_active":"2024-09-13T14:49:36.209691+0000","last_peered":"2024-09-13T14:49:36.209691+0000","last_clean":"2024-09-13T14:49:36.209691+0000","last_became_active":"2024-09-13T14:43:14.554183+0000","last_became_peered":"2024-09-13T14:43:14.554183+0000","last_unstale":"2024-09-13T14:49:36.209691+0000","last_undegraded":"2024-09-13T14:49:36.209691+0000","last_fullsized":"2024-09-13T14:49:36.209691+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.037517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.037517+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:01:35.137195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":1128,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.599195+0000","last_change":"2024-09-13T14:30:15.579260+0000","last_active":"2024-09-13T14:49:34.599195+0000","last_peered":"2024-09-13T14:49:34.599195+0000","last_clean":"2024-09-13T14:49:34.599195+0000","last_became_active":"2024-09-13T14:30:15.578897+0000","last_became_peered":"2024-09-13T14:30:15.578897+0000","last_unstale":"2024-09-13T14:49:34.599195+0000","last_undegraded":"2024-09-13T14:49:34.599195+0000","last_fullsized":"2024-09-13T14:49:34.599195+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:56.737849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":106,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.210044+0000","last_change":"2024-09-13T14:43:07.521576+0000","last_active":"2024-09-13T14:49:36.210044+0000","last_peered":"2024-09-13T14:49:36.210044+0000","last_clean":"2024-09-13T14:49:36.210044+0000","last_became_active":"2024-09-13T14:43:07.520478+0000","last_became_peered":"2024-09-13T14:43:07.520478+0000","last_unstale":"2024-09-13T14:49:36.210044+0000","last_undegraded":"2024-09-13T14:49:36.210044+0000","last_fullsized":"2024-09-13T14:49:36.210044+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:53.410135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"261'100","reported_seq":388,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.744270+0000","last_change":"2024-09-13T14:44:49.775941+0000","last_active":"2024-09-13T14:49:34.744270+0000","last_peered":"2024-09-13T14:49:34.744270+0000","last_clean":"2024-09-13T14:49:34.744270+0000","last_became_active":"2024-09-13T14:43:14.560297+0000","last_became_peered":"2024-09-13T14:43:14.560297+0000","last_unstale":"2024-09-13T14:49:34.744270+0000","last_undegraded":"2024-09-13T14:49:34.744270+0000","last_fullsized":"2024-09-13T14:49:34.744270+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:49.775844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:49.775844+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:15:53.138661+0000","scrub_duration":0.012350274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":1117,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.599689+0000","last_change":"2024-09-13T14:19:25.870605+0000","last_active":"2024-09-13T14:49:34.599689+0000","last_peered":"2024-09-13T14:49:34.599689+0000","last_clean":"2024-09-13T14:49:34.599689+0000","last_became_active":"2024-09-13T14:19:25.870282+0000","last_became_peered":"2024-09-13T14:19:25.870282+0000","last_unstale":"2024-09-13T14:49:34.599689+0000","last_undegraded":"2024-09-13T14:49:34.599689+0000","last_fullsized":"2024-09-13T14:49:34.599689+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:52.514915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1131,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.599377+0000","last_change":"2024-09-13T14:43:07.526476+0000","last_active":"2024-09-13T14:49:34.599377+0000","last_peered":"2024-09-13T14:49:34.599377+0000","last_clean":"2024-09-13T14:49:34.599377+0000","last_became_active":"2024-09-13T14:43:07.525802+0000","last_became_peered":"2024-09-13T14:43:07.525802+0000","last_unstale":"2024-09-13T14:49:34.599377+0000","last_undegraded":"2024-09-13T14:49:34.599377+0000","last_fullsized":"2024-09-13T14:49:34.599377+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:23.154726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"261'100","reported_seq":113,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.208956+0000","last_change":"2024-09-13T14:45:19.807461+0000","last_active":"2024-09-13T14:49:36.208956+0000","last_peered":"2024-09-13T14:49:36.208956+0000","last_clean":"2024-09-13T14:49:36.208956+0000","last_became_active":"2024-09-13T14:45:19.807117+0000","last_became_peered":"2024-09-13T14:45:19.807117+0000","last_unstale":"2024-09-13T14:49:36.208956+0000","last_undegraded":"2024-09-13T14:49:36.208956+0000","last_fullsized":"2024-09-13T14:49:36.208956+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.955197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.955197+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:03.531555+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.d","version":"1171'694","reported_seq":706,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:42.711737+0000","last_change":"2024-09-13T14:49:22.120980+0000","last_active":"2024-09-13T14:49:42.711737+0000","last_peered":"2024-09-13T14:49:42.711737+0000","last_clean":"2024-09-13T14:49:42.711737+0000","last_became_active":"2024-09-13T14:49:22.120684+0000","last_became_peered":"2024-09-13T14:49:22.120684+0000","last_unstale":"2024-09-13T14:49:42.711737+0000","last_undegraded":"2024-09-13T14:49:42.711737+0000","last_fullsized":"2024-09-13T14:49:42.711737+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":694,"log_dups_size":0,"ondisk_log_size":694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:49:34.732581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45481984,"num_objects":694,"num_object_clones":0,"num_object_copies":2082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":694,"num_write_kb":44416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1094,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.598932+0000","last_change":"2024-09-13T14:19:26.131609+0000","last_active":"2024-09-13T14:49:34.598932+0000","last_peered":"2024-09-13T14:49:34.598932+0000","last_clean":"2024-09-13T14:49:34.598932+0000","last_became_active":"2024-09-13T14:19:26.131029+0000","last_became_peered":"2024-09-13T14:19:26.131029+0000","last_unstale":"2024-09-13T14:49:34.598932+0000","last_undegraded":"2024-09-13T14:49:34.598932+0000","last_fullsized":"2024-09-13T14:49:34.598932+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:57.933696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":596,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.598949+0000","last_change":"2024-09-13T14:43:32.635070+0000","last_active":"2024-09-13T14:49:34.598949+0000","last_peered":"2024-09-13T14:49:34.598949+0000","last_clean":"2024-09-13T14:49:34.598949+0000","last_became_active":"2024-09-13T14:43:32.634734+0000","last_became_peered":"2024-09-13T14:43:32.634734+0000","last_unstale":"2024-09-13T14:49:34.598949+0000","last_undegraded":"2024-09-13T14:49:34.598949+0000","last_fullsized":"2024-09-13T14:49:34.598949+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:33.107010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":1748,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.132374+0000","last_change":"2024-09-13T14:43:07.537759+0000","last_active":"2024-09-13T14:49:36.132374+0000","last_peered":"2024-09-13T14:49:36.132374+0000","last_clean":"2024-09-13T14:49:36.132374+0000","last_became_active":"2024-09-13T14:43:07.520303+0000","last_became_peered":"2024-09-13T14:43:07.520303+0000","last_unstale":"2024-09-13T14:49:36.132374+0000","last_undegraded":"2024-09-13T14:49:36.132374+0000","last_fullsized":"2024-09-13T14:49:36.132374+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:32.772570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"261'100","reported_seq":58,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.602719+0000","last_change":"2024-09-13T14:45:19.813527+0000","last_active":"2024-09-13T14:49:35.602719+0000","last_peered":"2024-09-13T14:49:35.602719+0000","last_clean":"2024-09-13T14:49:35.602719+0000","last_became_active":"2024-09-13T14:45:19.812834+0000","last_became_peered":"2024-09-13T14:45:19.812834+0000","last_unstale":"2024-09-13T14:49:35.602719+0000","last_undegraded":"2024-09-13T14:49:35.602719+0000","last_fullsized":"2024-09-13T14:49:35.602719+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.710192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.710192+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:08:55.152703+0000","scrub_duration":0.0019250910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"1171'1075","reported_seq":1093,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:44.680955+0000","last_change":"2024-09-13T14:49:22.114132+0000","last_active":"2024-09-13T14:49:44.680955+0000","last_peered":"2024-09-13T14:49:44.680955+0000","last_clean":"2024-09-13T14:49:44.680955+0000","last_became_active":"2024-09-13T14:49:22.113820+0000","last_became_peered":"2024-09-13T14:49:22.113820+0000","last_unstale":"2024-09-13T14:49:44.680955+0000","last_undegraded":"2024-09-13T14:49:44.680955+0000","last_fullsized":"2024-09-13T14:49:44.680955+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":1075,"log_dups_size":0,"ondisk_log_size":1075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:34:37.889567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70451200,"num_objects":1075,"num_object_clones":0,"num_object_copies":3225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1075,"num_write_kb":68800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":58,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.602250+0000","last_change":"2024-09-13T14:45:19.813025+0000","last_active":"2024-09-13T14:49:35.602250+0000","last_peered":"2024-09-13T14:49:35.602250+0000","last_clean":"2024-09-13T14:49:35.602250+0000","last_became_active":"2024-09-13T14:45:19.812714+0000","last_became_peered":"2024-09-13T14:45:19.812714+0000","last_unstale":"2024-09-13T14:49:35.602250+0000","last_undegraded":"2024-09-13T14:49:35.602250+0000","last_fullsized":"2024-09-13T14:49:35.602250+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:40.671012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":107,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.210630+0000","last_change":"2024-09-13T14:43:07.522730+0000","last_active":"2024-09-13T14:49:36.210630+0000","last_peered":"2024-09-13T14:49:36.210630+0000","last_clean":"2024-09-13T14:49:36.210630+0000","last_became_active":"2024-09-13T14:43:07.517420+0000","last_became_peered":"2024-09-13T14:43:07.517420+0000","last_unstale":"2024-09-13T14:49:36.210630+0000","last_undegraded":"2024-09-13T14:49:36.210630+0000","last_fullsized":"2024-09-13T14:49:36.210630+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:21:30.265293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":1662,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.603213+0000","last_change":"2024-09-13T14:30:15.581728+0000","last_active":"2024-09-13T14:49:35.603213+0000","last_peered":"2024-09-13T14:49:35.603213+0000","last_clean":"2024-09-13T14:49:35.603213+0000","last_became_active":"2024-09-13T14:30:15.581303+0000","last_became_peered":"2024-09-13T14:30:15.581303+0000","last_unstale":"2024-09-13T14:49:35.603213+0000","last_undegraded":"2024-09-13T14:49:35.603213+0000","last_fullsized":"2024-09-13T14:49:35.603213+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:10.027991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"261'100","reported_seq":121,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.599753+0000","last_change":"2024-09-13T14:43:31.112173+0000","last_active":"2024-09-13T14:49:34.599753+0000","last_peered":"2024-09-13T14:49:34.599753+0000","last_clean":"2024-09-13T14:49:34.599753+0000","last_became_active":"2024-09-13T14:42:42.709263+0000","last_became_peered":"2024-09-13T14:42:42.709263+0000","last_unstale":"2024-09-13T14:49:34.599753+0000","last_undegraded":"2024-09-13T14:49:34.599753+0000","last_fullsized":"2024-09-13T14:49:34.599753+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:31.112111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:31.112111+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:02.707372+0000","scrub_duration":0.0015759260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"1171'1057","reported_seq":1076,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:44.574457+0000","last_change":"2024-09-13T14:49:22.118734+0000","last_active":"2024-09-13T14:49:44.574457+0000","last_peered":"2024-09-13T14:49:44.574457+0000","last_clean":"2024-09-13T14:49:44.574457+0000","last_became_active":"2024-09-13T14:49:22.118438+0000","last_became_peered":"2024-09-13T14:49:22.118438+0000","last_unstale":"2024-09-13T14:49:44.574457+0000","last_undegraded":"2024-09-13T14:49:44.574457+0000","last_fullsized":"2024-09-13T14:49:44.574457+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":1057,"log_dups_size":0,"ondisk_log_size":1057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:27:01.810072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69271552,"num_objects":1057,"num_object_clones":0,"num_object_copies":3171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1057,"num_write_kb":67648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1702,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.131841+0000","last_change":"2024-09-13T14:28:35.611508+0000","last_active":"2024-09-13T14:49:36.131841+0000","last_peered":"2024-09-13T14:49:36.131841+0000","last_clean":"2024-09-13T14:49:36.131841+0000","last_became_active":"2024-09-13T14:28:35.610106+0000","last_became_peered":"2024-09-13T14:28:35.610106+0000","last_unstale":"2024-09-13T14:49:36.131841+0000","last_undegraded":"2024-09-13T14:49:36.131841+0000","last_fullsized":"2024-09-13T14:49:36.131841+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:11.966958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1137,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.599276+0000","last_change":"2024-09-13T14:28:35.603663+0000","last_active":"2024-09-13T14:49:34.599276+0000","last_peered":"2024-09-13T14:49:34.599276+0000","last_clean":"2024-09-13T14:49:34.599276+0000","last_became_active":"2024-09-13T14:28:35.603358+0000","last_became_peered":"2024-09-13T14:28:35.603358+0000","last_unstale":"2024-09-13T14:49:34.599276+0000","last_undegraded":"2024-09-13T14:49:34.599276+0000","last_fullsized":"2024-09-13T14:49:34.599276+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:24:05.842796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":107,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.210297+0000","last_change":"2024-09-13T14:43:07.522750+0000","last_active":"2024-09-13T14:49:36.210297+0000","last_peered":"2024-09-13T14:49:36.210297+0000","last_clean":"2024-09-13T14:49:36.210297+0000","last_became_active":"2024-09-13T14:43:07.517267+0000","last_became_peered":"2024-09-13T14:43:07.517267+0000","last_unstale":"2024-09-13T14:49:36.210297+0000","last_undegraded":"2024-09-13T14:49:36.210297+0000","last_fullsized":"2024-09-13T14:49:36.210297+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:40.217748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"261'100","reported_seq":427,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.132023+0000","last_change":"2024-09-13T14:45:19.815050+0000","last_active":"2024-09-13T14:49:36.132023+0000","last_peered":"2024-09-13T14:49:36.132023+0000","last_clean":"2024-09-13T14:49:36.132023+0000","last_became_active":"2024-09-13T14:45:19.809732+0000","last_became_peered":"2024-09-13T14:45:19.809732+0000","last_unstale":"2024-09-13T14:49:36.132023+0000","last_undegraded":"2024-09-13T14:49:36.132023+0000","last_fullsized":"2024-09-13T14:49:36.132023+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:33.574443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:33.574443+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:51:40.261036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"1171'1020","reported_seq":1039,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:44.675700+0000","last_change":"2024-09-13T14:49:22.122114+0000","last_active":"2024-09-13T14:49:44.675700+0000","last_peered":"2024-09-13T14:49:44.675700+0000","last_clean":"2024-09-13T14:49:44.675700+0000","last_became_active":"2024-09-13T14:49:22.120044+0000","last_became_peered":"2024-09-13T14:49:22.120044+0000","last_unstale":"2024-09-13T14:49:44.675700+0000","last_undegraded":"2024-09-13T14:49:44.675700+0000","last_fullsized":"2024-09-13T14:49:44.675700+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":1020,"log_dups_size":0,"ondisk_log_size":1020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:09:06.965253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66846720,"num_objects":1020,"num_object_clones":0,"num_object_copies":3060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1020,"num_write_kb":65280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":107,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.209900+0000","last_change":"2024-09-13T14:43:07.520049+0000","last_active":"2024-09-13T14:49:36.209900+0000","last_peered":"2024-09-13T14:49:36.209900+0000","last_clean":"2024-09-13T14:49:36.209900+0000","last_became_active":"2024-09-13T14:43:07.514646+0000","last_became_peered":"2024-09-13T14:43:07.514646+0000","last_unstale":"2024-09-13T14:49:36.209900+0000","last_undegraded":"2024-09-13T14:49:36.209900+0000","last_fullsized":"2024-09-13T14:49:36.209900+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:10:35.200778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":107,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.210212+0000","last_change":"2024-09-13T14:43:07.525666+0000","last_active":"2024-09-13T14:49:36.210212+0000","last_peered":"2024-09-13T14:49:36.210212+0000","last_clean":"2024-09-13T14:49:36.210212+0000","last_became_active":"2024-09-13T14:43:07.524064+0000","last_became_peered":"2024-09-13T14:43:07.524064+0000","last_unstale":"2024-09-13T14:49:36.210212+0000","last_undegraded":"2024-09-13T14:49:36.210212+0000","last_fullsized":"2024-09-13T14:49:36.210212+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:10.270333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1151,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.744388+0000","last_change":"2024-09-13T14:45:19.804177+0000","last_active":"2024-09-13T14:49:34.744388+0000","last_peered":"2024-09-13T14:49:34.744388+0000","last_clean":"2024-09-13T14:49:34.744388+0000","last_became_active":"2024-09-13T14:45:19.803890+0000","last_became_peered":"2024-09-13T14:45:19.803890+0000","last_unstale":"2024-09-13T14:49:34.744388+0000","last_undegraded":"2024-09-13T14:49:34.744388+0000","last_fullsized":"2024-09-13T14:49:34.744388+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:25.140789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"261'100","reported_seq":75,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.603197+0000","last_change":"2024-09-13T14:45:36.965992+0000","last_active":"2024-09-13T14:49:35.603197+0000","last_peered":"2024-09-13T14:49:35.603197+0000","last_clean":"2024-09-13T14:49:35.603197+0000","last_became_active":"2024-09-13T14:45:19.811238+0000","last_became_peered":"2024-09-13T14:45:19.811238+0000","last_unstale":"2024-09-13T14:49:35.603197+0000","last_undegraded":"2024-09-13T14:49:35.603197+0000","last_fullsized":"2024-09-13T14:49:35.603197+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:36.965775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:45:36.965775+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:36:01.623251+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"1171'715","reported_seq":727,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:42.716963+0000","last_change":"2024-09-13T14:49:22.122277+0000","last_active":"2024-09-13T14:49:42.716963+0000","last_peered":"2024-09-13T14:49:42.716963+0000","last_clean":"2024-09-13T14:49:42.716963+0000","last_became_active":"2024-09-13T14:49:22.121980+0000","last_became_peered":"2024-09-13T14:49:22.121980+0000","last_unstale":"2024-09-13T14:49:42.716963+0000","last_undegraded":"2024-09-13T14:49:42.716963+0000","last_fullsized":"2024-09-13T14:49:42.716963+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":715,"log_dups_size":0,"ondisk_log_size":715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:58.275564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46858240,"num_objects":715,"num_object_clones":0,"num_object_copies":2145,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":715,"num_write_kb":45760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":50,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.743981+0000","last_change":"2024-09-13T14:45:19.806318+0000","last_active":"2024-09-13T14:49:34.743981+0000","last_peered":"2024-09-13T14:49:34.743981+0000","last_clean":"2024-09-13T14:49:34.743981+0000","last_became_active":"2024-09-13T14:45:19.806074+0000","last_became_peered":"2024-09-13T14:45:19.806074+0000","last_unstale":"2024-09-13T14:49:34.743981+0000","last_undegraded":"2024-09-13T14:49:34.743981+0000","last_fullsized":"2024-09-13T14:49:34.743981+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:21.679797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1112,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.599535+0000","last_change":"2024-09-13T14:28:35.604404+0000","last_active":"2024-09-13T14:49:34.599535+0000","last_peered":"2024-09-13T14:49:34.599535+0000","last_clean":"2024-09-13T14:49:34.599535+0000","last_became_active":"2024-09-13T14:28:35.604102+0000","last_became_peered":"2024-09-13T14:28:35.604102+0000","last_unstale":"2024-09-13T14:49:34.599535+0000","last_undegraded":"2024-09-13T14:49:34.599535+0000","last_fullsized":"2024-09-13T14:49:34.599535+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:25.546010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":1162,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.744034+0000","last_change":"2024-09-13T14:43:07.538919+0000","last_active":"2024-09-13T14:49:34.744034+0000","last_peered":"2024-09-13T14:49:34.744034+0000","last_clean":"2024-09-13T14:49:34.744034+0000","last_became_active":"2024-09-13T14:43:07.513109+0000","last_became_peered":"2024-09-13T14:43:07.513109+0000","last_unstale":"2024-09-13T14:49:34.744034+0000","last_undegraded":"2024-09-13T14:49:34.744034+0000","last_fullsized":"2024-09-13T14:49:34.744034+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:27.771720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"261'100","reported_seq":111,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.743742+0000","last_change":"2024-09-13T14:43:44.720333+0000","last_active":"2024-09-13T14:49:34.743742+0000","last_peered":"2024-09-13T14:49:34.743742+0000","last_clean":"2024-09-13T14:49:34.743742+0000","last_became_active":"2024-09-13T14:42:14.946591+0000","last_became_peered":"2024-09-13T14:42:14.946591+0000","last_unstale":"2024-09-13T14:49:34.743742+0000","last_undegraded":"2024-09-13T14:49:34.743742+0000","last_fullsized":"2024-09-13T14:49:34.743742+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:44.720258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:44.720258+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:33.281357+0000","scrub_duration":0.0018417920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"1171'1069","reported_seq":1088,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:44.568047+0000","last_change":"2024-09-13T14:49:22.119554+0000","last_active":"2024-09-13T14:49:44.568047+0000","last_peered":"2024-09-13T14:49:44.568047+0000","last_clean":"2024-09-13T14:49:44.568047+0000","last_became_active":"2024-09-13T14:49:22.119249+0000","last_became_peered":"2024-09-13T14:49:22.119249+0000","last_unstale":"2024-09-13T14:49:44.568047+0000","last_undegraded":"2024-09-13T14:49:44.568047+0000","last_fullsized":"2024-09-13T14:49:44.568047+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":1069,"log_dups_size":0,"ondisk_log_size":1069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:03:16.735024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70057984,"num_objects":1069,"num_object_clones":0,"num_object_copies":3207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1069,"num_write_kb":68416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":74,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.209551+0000","last_change":"2024-09-13T14:44:03.858211+0000","last_active":"2024-09-13T14:49:36.209551+0000","last_peered":"2024-09-13T14:49:36.209551+0000","last_clean":"2024-09-13T14:49:36.209551+0000","last_became_active":"2024-09-13T14:44:03.857928+0000","last_became_peered":"2024-09-13T14:44:03.857928+0000","last_unstale":"2024-09-13T14:49:36.209551+0000","last_undegraded":"2024-09-13T14:49:36.209551+0000","last_fullsized":"2024-09-13T14:49:36.209551+0000","mapping_epoch":1140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:28.428693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":107,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.210117+0000","last_change":"2024-09-13T14:43:07.516243+0000","last_active":"2024-09-13T14:49:36.210117+0000","last_peered":"2024-09-13T14:49:36.210117+0000","last_clean":"2024-09-13T14:49:36.210117+0000","last_became_active":"2024-09-13T14:43:07.515863+0000","last_became_peered":"2024-09-13T14:43:07.515863+0000","last_unstale":"2024-09-13T14:49:36.210117+0000","last_undegraded":"2024-09-13T14:49:36.210117+0000","last_fullsized":"2024-09-13T14:49:36.210117+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:08.085298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1743,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.208948+0000","last_change":"2024-09-13T14:45:19.812713+0000","last_active":"2024-09-13T14:49:36.208948+0000","last_peered":"2024-09-13T14:49:36.208948+0000","last_clean":"2024-09-13T14:49:36.208948+0000","last_became_active":"2024-09-13T14:45:19.812203+0000","last_became_peered":"2024-09-13T14:45:19.812203+0000","last_unstale":"2024-09-13T14:49:36.208948+0000","last_undegraded":"2024-09-13T14:49:36.208948+0000","last_fullsized":"2024-09-13T14:49:36.208948+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:30:46.149347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"261'100","reported_seq":1291,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.598123+0000","last_change":"2024-09-13T14:43:42.577312+0000","last_active":"2024-09-13T14:49:34.598123+0000","last_peered":"2024-09-13T14:49:34.598123+0000","last_clean":"2024-09-13T14:49:34.598123+0000","last_became_active":"2024-09-13T14:42:14.928923+0000","last_became_peered":"2024-09-13T14:42:14.928923+0000","last_unstale":"2024-09-13T14:49:34.598123+0000","last_undegraded":"2024-09-13T14:49:34.598123+0000","last_fullsized":"2024-09-13T14:49:34.598123+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:42.577243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:42.577243+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:01.234250+0000","scrub_duration":0.001367839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"1171'737","reported_seq":756,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:43.026763+0000","last_change":"2024-09-13T14:49:22.118690+0000","last_active":"2024-09-13T14:49:43.026763+0000","last_peered":"2024-09-13T14:49:43.026763+0000","last_clean":"2024-09-13T14:49:43.026763+0000","last_became_active":"2024-09-13T14:49:22.117901+0000","last_became_peered":"2024-09-13T14:49:22.117901+0000","last_unstale":"2024-09-13T14:49:43.026763+0000","last_undegraded":"2024-09-13T14:49:43.026763+0000","last_fullsized":"2024-09-13T14:49:43.026763+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":737,"log_dups_size":0,"ondisk_log_size":737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:28.098913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48300032,"num_objects":737,"num_object_clones":0,"num_object_copies":2211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":737,"num_write_kb":47168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1737,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.132341+0000","last_change":"2024-09-13T14:43:07.520212+0000","last_active":"2024-09-13T14:49:36.132341+0000","last_peered":"2024-09-13T14:49:36.132341+0000","last_clean":"2024-09-13T14:49:36.132341+0000","last_became_active":"2024-09-13T14:43:07.519831+0000","last_became_peered":"2024-09-13T14:43:07.519831+0000","last_unstale":"2024-09-13T14:49:36.132341+0000","last_undegraded":"2024-09-13T14:49:36.132341+0000","last_fullsized":"2024-09-13T14:49:36.132341+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:20.990736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1760,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.131472+0000","last_change":"2024-09-13T14:45:19.809505+0000","last_active":"2024-09-13T14:49:36.131472+0000","last_peered":"2024-09-13T14:49:36.131472+0000","last_clean":"2024-09-13T14:49:36.131472+0000","last_became_active":"2024-09-13T14:45:19.809181+0000","last_became_peered":"2024-09-13T14:45:19.809181+0000","last_unstale":"2024-09-13T14:49:36.131472+0000","last_undegraded":"2024-09-13T14:49:36.131472+0000","last_fullsized":"2024-09-13T14:49:36.131472+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:24.239188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1174,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.598135+0000","last_change":"2024-09-13T14:43:07.516140+0000","last_active":"2024-09-13T14:49:34.598135+0000","last_peered":"2024-09-13T14:49:34.598135+0000","last_clean":"2024-09-13T14:49:34.598135+0000","last_became_active":"2024-09-13T14:43:07.515481+0000","last_became_peered":"2024-09-13T14:43:07.515481+0000","last_unstale":"2024-09-13T14:49:34.598135+0000","last_undegraded":"2024-09-13T14:49:34.598135+0000","last_fullsized":"2024-09-13T14:49:34.598135+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:43.706214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"261'100","reported_seq":119,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.210506+0000","last_change":"2024-09-13T14:43:16.008346+0000","last_active":"2024-09-13T14:49:36.210506+0000","last_peered":"2024-09-13T14:49:36.210506+0000","last_clean":"2024-09-13T14:49:36.210506+0000","last_became_active":"2024-09-13T14:43:07.523121+0000","last_became_peered":"2024-09-13T14:43:07.523121+0000","last_unstale":"2024-09-13T14:49:36.210506+0000","last_undegraded":"2024-09-13T14:49:36.210506+0000","last_fullsized":"2024-09-13T14:49:36.210506+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.008264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.008264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:58:35.564470+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"1171'686","reported_seq":698,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:42.713358+0000","last_change":"2024-09-13T14:49:22.119570+0000","last_active":"2024-09-13T14:49:42.713358+0000","last_peered":"2024-09-13T14:49:42.713358+0000","last_clean":"2024-09-13T14:49:42.713358+0000","last_became_active":"2024-09-13T14:49:22.119224+0000","last_became_peered":"2024-09-13T14:49:22.119224+0000","last_unstale":"2024-09-13T14:49:42.713358+0000","last_undegraded":"2024-09-13T14:49:42.713358+0000","last_fullsized":"2024-09-13T14:49:42.713358+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":686,"log_dups_size":0,"ondisk_log_size":686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:43.554416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44957696,"num_objects":686,"num_object_clones":0,"num_object_copies":2058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":686,"num_write_kb":43904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":53,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.599719+0000","last_change":"2024-09-13T14:43:32.636990+0000","last_active":"2024-09-13T14:49:34.599719+0000","last_peered":"2024-09-13T14:49:34.599719+0000","last_clean":"2024-09-13T14:49:34.599719+0000","last_became_active":"2024-09-13T14:43:32.636589+0000","last_became_peered":"2024-09-13T14:43:32.636589+0000","last_unstale":"2024-09-13T14:49:34.599719+0000","last_undegraded":"2024-09-13T14:49:34.599719+0000","last_fullsized":"2024-09-13T14:49:34.599719+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:36:25.129769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":58,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.602091+0000","last_change":"2024-09-13T14:45:19.810888+0000","last_active":"2024-09-13T14:49:35.602091+0000","last_peered":"2024-09-13T14:49:35.602091+0000","last_clean":"2024-09-13T14:49:35.602091+0000","last_became_active":"2024-09-13T14:45:19.810486+0000","last_became_peered":"2024-09-13T14:45:19.810486+0000","last_unstale":"2024-09-13T14:49:35.602091+0000","last_undegraded":"2024-09-13T14:49:35.602091+0000","last_fullsized":"2024-09-13T14:49:35.602091+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:03.886942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":1127,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.743649+0000","last_change":"2024-09-13T14:29:42.539455+0000","last_active":"2024-09-13T14:49:34.743649+0000","last_peered":"2024-09-13T14:49:34.743649+0000","last_clean":"2024-09-13T14:49:34.743649+0000","last_became_active":"2024-09-13T14:29:42.539068+0000","last_became_peered":"2024-09-13T14:29:42.539068+0000","last_unstale":"2024-09-13T14:49:34.743649+0000","last_undegraded":"2024-09-13T14:49:34.743649+0000","last_fullsized":"2024-09-13T14:49:34.743649+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:24:06.048855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034405999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"261'100","reported_seq":1282,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.598182+0000","last_change":"2024-09-13T14:44:28.186029+0000","last_active":"2024-09-13T14:49:34.598182+0000","last_peered":"2024-09-13T14:49:34.598182+0000","last_clean":"2024-09-13T14:49:34.598182+0000","last_became_active":"2024-09-13T14:42:14.927771+0000","last_became_peered":"2024-09-13T14:42:14.927771+0000","last_unstale":"2024-09-13T14:49:34.598182+0000","last_undegraded":"2024-09-13T14:49:34.598182+0000","last_fullsized":"2024-09-13T14:49:34.598182+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:28.185949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:28.185949+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:38.363961+0000","scrub_duration":0.0018071129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3","version":"1171'686","reported_seq":705,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:43.028684+0000","last_change":"2024-09-13T14:49:22.119776+0000","last_active":"2024-09-13T14:49:43.028684+0000","last_peered":"2024-09-13T14:49:43.028684+0000","last_clean":"2024-09-13T14:49:43.028684+0000","last_became_active":"2024-09-13T14:49:22.119517+0000","last_became_peered":"2024-09-13T14:49:22.119517+0000","last_unstale":"2024-09-13T14:49:43.028684+0000","last_undegraded":"2024-09-13T14:49:43.028684+0000","last_fullsized":"2024-09-13T14:49:43.028684+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":686,"log_dups_size":0,"ondisk_log_size":686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:53.152882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44957696,"num_objects":686,"num_object_clones":0,"num_object_copies":2058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":686,"num_write_kb":43904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":106,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.210730+0000","last_change":"2024-09-13T14:43:07.520176+0000","last_active":"2024-09-13T14:49:36.210730+0000","last_peered":"2024-09-13T14:49:36.210730+0000","last_clean":"2024-09-13T14:49:36.210730+0000","last_became_active":"2024-09-13T14:43:07.516748+0000","last_became_peered":"2024-09-13T14:43:07.516748+0000","last_unstale":"2024-09-13T14:49:36.210730+0000","last_undegraded":"2024-09-13T14:49:36.210730+0000","last_fullsized":"2024-09-13T14:49:36.210730+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:52:04.392821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":58,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.602353+0000","last_change":"2024-09-13T14:45:19.812353+0000","last_active":"2024-09-13T14:49:35.602353+0000","last_peered":"2024-09-13T14:49:35.602353+0000","last_clean":"2024-09-13T14:49:35.602353+0000","last_became_active":"2024-09-13T14:45:19.811743+0000","last_became_peered":"2024-09-13T14:45:19.811743+0000","last_unstale":"2024-09-13T14:49:35.602353+0000","last_undegraded":"2024-09-13T14:49:35.602353+0000","last_fullsized":"2024-09-13T14:49:35.602353+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:43.728170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"261'100","reported_seq":111,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.743891+0000","last_change":"2024-09-13T14:44:34.814157+0000","last_active":"2024-09-13T14:49:34.743891+0000","last_peered":"2024-09-13T14:49:34.743891+0000","last_clean":"2024-09-13T14:49:34.743891+0000","last_became_active":"2024-09-13T14:42:42.707803+0000","last_became_peered":"2024-09-13T14:42:42.707803+0000","last_unstale":"2024-09-13T14:49:34.743891+0000","last_undegraded":"2024-09-13T14:49:34.743891+0000","last_fullsized":"2024-09-13T14:49:34.743891+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:34.814070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:34.814070+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:38.408561+0000","scrub_duration":0.01686414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"1171'683","reported_seq":695,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:42.520500+0000","last_change":"2024-09-13T14:49:22.119578+0000","last_active":"2024-09-13T14:49:42.520500+0000","last_peered":"2024-09-13T14:49:42.520500+0000","last_clean":"2024-09-13T14:49:42.520500+0000","last_became_active":"2024-09-13T14:49:22.119280+0000","last_became_peered":"2024-09-13T14:49:22.119280+0000","last_unstale":"2024-09-13T14:49:42.520500+0000","last_undegraded":"2024-09-13T14:49:42.520500+0000","last_fullsized":"2024-09-13T14:49:42.520500+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":683,"log_dups_size":0,"ondisk_log_size":683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:13:15.669993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44761088,"num_objects":683,"num_object_clones":0,"num_object_copies":2049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":683,"num_write_kb":43712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1141,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.599844+0000","last_change":"2024-09-13T14:44:03.855577+0000","last_active":"2024-09-13T14:49:34.599844+0000","last_peered":"2024-09-13T14:49:34.599844+0000","last_clean":"2024-09-13T14:49:34.599844+0000","last_became_active":"2024-09-13T14:44:03.855267+0000","last_became_peered":"2024-09-13T14:44:03.855267+0000","last_unstale":"2024-09-13T14:49:34.599844+0000","last_undegraded":"2024-09-13T14:49:34.599844+0000","last_fullsized":"2024-09-13T14:49:34.599844+0000","mapping_epoch":1140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:46.148814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1112,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.599069+0000","last_change":"2024-09-13T14:29:42.545815+0000","last_active":"2024-09-13T14:49:34.599069+0000","last_peered":"2024-09-13T14:49:34.599069+0000","last_clean":"2024-09-13T14:49:34.599069+0000","last_became_active":"2024-09-13T14:29:42.545508+0000","last_became_peered":"2024-09-13T14:29:42.545508+0000","last_unstale":"2024-09-13T14:49:34.599069+0000","last_undegraded":"2024-09-13T14:49:34.599069+0000","last_fullsized":"2024-09-13T14:49:34.599069+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:39.678381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"261'100","reported_seq":124,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.210344+0000","last_change":"2024-09-13T14:43:18.988482+0000","last_active":"2024-09-13T14:49:36.210344+0000","last_peered":"2024-09-13T14:49:36.210344+0000","last_clean":"2024-09-13T14:49:36.210344+0000","last_became_active":"2024-09-13T14:43:14.556754+0000","last_became_peered":"2024-09-13T14:43:14.556754+0000","last_unstale":"2024-09-13T14:49:36.210344+0000","last_undegraded":"2024-09-13T14:49:36.210344+0000","last_fullsized":"2024-09-13T14:49:36.210344+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:18.988405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:18.988405+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:45:01.726326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"1171'731","reported_seq":749,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:43.022030+0000","last_change":"2024-09-13T14:49:22.121230+0000","last_active":"2024-09-13T14:49:43.022030+0000","last_peered":"2024-09-13T14:49:43.022030+0000","last_clean":"2024-09-13T14:49:43.022030+0000","last_became_active":"2024-09-13T14:49:22.120836+0000","last_became_peered":"2024-09-13T14:49:22.120836+0000","last_unstale":"2024-09-13T14:49:43.022030+0000","last_undegraded":"2024-09-13T14:49:43.022030+0000","last_fullsized":"2024-09-13T14:49:43.022030+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":731,"log_dups_size":0,"ondisk_log_size":731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:21.843404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47906816,"num_objects":731,"num_object_clones":0,"num_object_copies":2193,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":731,"num_write_kb":46784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1158,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.599949+0000","last_change":"2024-09-13T14:43:07.515418+0000","last_active":"2024-09-13T14:49:34.599949+0000","last_peered":"2024-09-13T14:49:34.599949+0000","last_clean":"2024-09-13T14:49:34.599949+0000","last_became_active":"2024-09-13T14:43:07.515113+0000","last_became_peered":"2024-09-13T14:43:07.515113+0000","last_unstale":"2024-09-13T14:49:34.599949+0000","last_undegraded":"2024-09-13T14:49:34.599949+0000","last_fullsized":"2024-09-13T14:49:34.599949+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:10:25.810604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1125,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.599329+0000","last_change":"2024-09-13T14:43:32.636334+0000","last_active":"2024-09-13T14:49:34.599329+0000","last_peered":"2024-09-13T14:49:34.599329+0000","last_clean":"2024-09-13T14:49:34.599329+0000","last_became_active":"2024-09-13T14:43:32.635736+0000","last_became_peered":"2024-09-13T14:43:32.635736+0000","last_unstale":"2024-09-13T14:49:34.599329+0000","last_undegraded":"2024-09-13T14:49:34.599329+0000","last_fullsized":"2024-09-13T14:49:34.599329+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:49.076014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"261'100","reported_seq":116,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.743763+0000","last_change":"2024-09-13T14:45:19.805634+0000","last_active":"2024-09-13T14:49:34.743763+0000","last_peered":"2024-09-13T14:49:34.743763+0000","last_clean":"2024-09-13T14:49:34.743763+0000","last_became_active":"2024-09-13T14:45:19.805046+0000","last_became_peered":"2024-09-13T14:45:19.805046+0000","last_unstale":"2024-09-13T14:49:34.743763+0000","last_undegraded":"2024-09-13T14:49:34.743763+0000","last_fullsized":"2024-09-13T14:49:34.743763+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:32.752408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:32.752408+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:06:53.140239+0000","scrub_duration":0.0057731559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.0","version":"1171'989","reported_seq":1008,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:44.568635+0000","last_change":"2024-09-13T14:49:22.118730+0000","last_active":"2024-09-13T14:49:44.568635+0000","last_peered":"2024-09-13T14:49:44.568635+0000","last_clean":"2024-09-13T14:49:44.568635+0000","last_became_active":"2024-09-13T14:49:22.118428+0000","last_became_peered":"2024-09-13T14:49:22.118428+0000","last_unstale":"2024-09-13T14:49:44.568635+0000","last_undegraded":"2024-09-13T14:49:44.568635+0000","last_fullsized":"2024-09-13T14:49:44.568635+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":989,"log_dups_size":0,"ondisk_log_size":989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:54.951267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64815104,"num_objects":989,"num_object_clones":0,"num_object_copies":2967,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":989,"num_write_kb":63296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1161,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.599638+0000","last_change":"2024-09-13T14:45:19.804441+0000","last_active":"2024-09-13T14:49:34.599638+0000","last_peered":"2024-09-13T14:49:34.599638+0000","last_clean":"2024-09-13T14:49:34.599638+0000","last_became_active":"2024-09-13T14:45:19.804039+0000","last_became_peered":"2024-09-13T14:45:19.804039+0000","last_unstale":"2024-09-13T14:49:34.599638+0000","last_undegraded":"2024-09-13T14:49:34.599638+0000","last_fullsized":"2024-09-13T14:49:34.599638+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:58:00.322042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":61,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.602248+0000","last_change":"2024-09-13T14:45:19.816666+0000","last_active":"2024-09-13T14:49:35.602248+0000","last_peered":"2024-09-13T14:49:35.602248+0000","last_clean":"2024-09-13T14:49:35.602248+0000","last_became_active":"2024-09-13T14:45:19.807224+0000","last_became_peered":"2024-09-13T14:45:19.807224+0000","last_unstale":"2024-09-13T14:49:35.602248+0000","last_undegraded":"2024-09-13T14:49:35.602248+0000","last_fullsized":"2024-09-13T14:49:35.602248+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:15:20.171682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"261'100","reported_seq":1301,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.597987+0000","last_change":"2024-09-13T14:45:19.798553+0000","last_active":"2024-09-13T14:49:34.597987+0000","last_peered":"2024-09-13T14:49:34.597987+0000","last_clean":"2024-09-13T14:49:34.597987+0000","last_became_active":"2024-09-13T14:45:19.798189+0000","last_became_peered":"2024-09-13T14:45:19.798189+0000","last_unstale":"2024-09-13T14:49:34.597987+0000","last_undegraded":"2024-09-13T14:49:34.597987+0000","last_fullsized":"2024-09-13T14:49:34.597987+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:29.169325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:29.169325+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:10.412666+0000","scrub_duration":0.0019358890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"1171'681","reported_seq":700,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:43.025352+0000","last_change":"2024-09-13T14:49:22.120585+0000","last_active":"2024-09-13T14:49:43.025352+0000","last_peered":"2024-09-13T14:49:43.025352+0000","last_clean":"2024-09-13T14:49:43.025352+0000","last_became_active":"2024-09-13T14:49:22.120301+0000","last_became_peered":"2024-09-13T14:49:22.120301+0000","last_unstale":"2024-09-13T14:49:43.025352+0000","last_undegraded":"2024-09-13T14:49:43.025352+0000","last_fullsized":"2024-09-13T14:49:43.025352+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":681,"log_dups_size":0,"ondisk_log_size":681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:20:25.183879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44630016,"num_objects":681,"num_object_clones":0,"num_object_copies":2043,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":681,"num_write_kb":43584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1726,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.131788+0000","last_change":"2024-09-13T14:43:07.515572+0000","last_active":"2024-09-13T14:49:36.131788+0000","last_peered":"2024-09-13T14:49:36.131788+0000","last_clean":"2024-09-13T14:49:36.131788+0000","last_became_active":"2024-09-13T14:43:07.515231+0000","last_became_peered":"2024-09-13T14:43:07.515231+0000","last_unstale":"2024-09-13T14:49:36.131788+0000","last_undegraded":"2024-09-13T14:49:36.131788+0000","last_fullsized":"2024-09-13T14:49:36.131788+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:57.966380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1154,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.744150+0000","last_change":"2024-09-13T14:45:19.804982+0000","last_active":"2024-09-13T14:49:34.744150+0000","last_peered":"2024-09-13T14:49:34.744150+0000","last_clean":"2024-09-13T14:49:34.744150+0000","last_became_active":"2024-09-13T14:45:19.804663+0000","last_became_peered":"2024-09-13T14:45:19.804663+0000","last_unstale":"2024-09-13T14:49:34.744150+0000","last_undegraded":"2024-09-13T14:49:34.744150+0000","last_fullsized":"2024-09-13T14:49:34.744150+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:43.211479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"261'100","reported_seq":391,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.599798+0000","last_change":"2024-09-13T14:43:42.881584+0000","last_active":"2024-09-13T14:49:34.599798+0000","last_peered":"2024-09-13T14:49:34.599798+0000","last_clean":"2024-09-13T14:49:34.599798+0000","last_became_active":"2024-09-13T14:43:14.562161+0000","last_became_peered":"2024-09-13T14:43:14.562161+0000","last_unstale":"2024-09-13T14:49:34.599798+0000","last_undegraded":"2024-09-13T14:49:34.599798+0000","last_fullsized":"2024-09-13T14:49:34.599798+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:42.881520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:42.881520+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:06:50.353442+0000","scrub_duration":0.0013804310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"1171'1044","reported_seq":1063,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:44.678607+0000","last_change":"2024-09-13T14:49:22.117570+0000","last_active":"2024-09-13T14:49:44.678607+0000","last_peered":"2024-09-13T14:49:44.678607+0000","last_clean":"2024-09-13T14:49:44.678607+0000","last_became_active":"2024-09-13T14:49:22.117259+0000","last_became_peered":"2024-09-13T14:49:22.117259+0000","last_unstale":"2024-09-13T14:49:44.678607+0000","last_undegraded":"2024-09-13T14:49:44.678607+0000","last_fullsized":"2024-09-13T14:49:44.678607+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":1044,"log_dups_size":0,"ondisk_log_size":1044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:59:23.669062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68419584,"num_objects":1044,"num_object_clones":0,"num_object_copies":3132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":66816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1127,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.235208+0000","last_change":"2024-09-13T14:43:07.518120+0000","last_active":"2024-09-13T14:49:35.235208+0000","last_peered":"2024-09-13T14:49:35.235208+0000","last_clean":"2024-09-13T14:49:35.235208+0000","last_became_active":"2024-09-13T14:43:07.517361+0000","last_became_peered":"2024-09-13T14:43:07.517361+0000","last_unstale":"2024-09-13T14:49:35.235208+0000","last_undegraded":"2024-09-13T14:49:35.235208+0000","last_fullsized":"2024-09-13T14:49:35.235208+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:39.366473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":1109,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.599115+0000","last_change":"2024-09-13T14:19:25.871006+0000","last_active":"2024-09-13T14:49:34.599115+0000","last_peered":"2024-09-13T14:49:34.599115+0000","last_clean":"2024-09-13T14:49:34.599115+0000","last_became_active":"2024-09-13T14:19:25.870713+0000","last_became_peered":"2024-09-13T14:19:25.870713+0000","last_unstale":"2024-09-13T14:49:34.599115+0000","last_undegraded":"2024-09-13T14:49:34.599115+0000","last_fullsized":"2024-09-13T14:49:34.599115+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:34.372124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"261'100","reported_seq":385,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.744247+0000","last_change":"2024-09-13T14:49:21.723011+0000","last_active":"2024-09-13T14:49:34.744247+0000","last_peered":"2024-09-13T14:49:34.744247+0000","last_clean":"2024-09-13T14:49:34.744247+0000","last_became_active":"2024-09-13T14:43:32.636545+0000","last_became_peered":"2024-09-13T14:43:32.636545+0000","last_unstale":"2024-09-13T14:49:34.744247+0000","last_undegraded":"2024-09-13T14:49:34.744247+0000","last_fullsized":"2024-09-13T14:49:34.744247+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:49:21.722939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.722939+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:18:07.462147+0000","scrub_duration":0.0016642790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.5","version":"1171'703","reported_seq":715,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:42.765690+0000","last_change":"2024-09-13T14:49:22.120623+0000","last_active":"2024-09-13T14:49:42.765690+0000","last_peered":"2024-09-13T14:49:42.765690+0000","last_clean":"2024-09-13T14:49:42.765690+0000","last_became_active":"2024-09-13T14:49:22.120295+0000","last_became_peered":"2024-09-13T14:49:22.120295+0000","last_unstale":"2024-09-13T14:49:42.765690+0000","last_undegraded":"2024-09-13T14:49:42.765690+0000","last_fullsized":"2024-09-13T14:49:42.765690+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":703,"log_dups_size":0,"ondisk_log_size":703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.848357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46071808,"num_objects":703,"num_object_clones":0,"num_object_copies":2109,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":703,"num_write_kb":44992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1178,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.599029+0000","last_change":"2024-09-13T14:45:19.804868+0000","last_active":"2024-09-13T14:49:34.599029+0000","last_peered":"2024-09-13T14:49:34.599029+0000","last_clean":"2024-09-13T14:49:34.599029+0000","last_became_active":"2024-09-13T14:45:19.804520+0000","last_became_peered":"2024-09-13T14:45:19.804520+0000","last_unstale":"2024-09-13T14:49:34.599029+0000","last_undegraded":"2024-09-13T14:49:34.599029+0000","last_fullsized":"2024-09-13T14:49:34.599029+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:17:53.924629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":1155,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.598956+0000","last_change":"2024-09-13T14:43:07.533955+0000","last_active":"2024-09-13T14:49:34.598956+0000","last_peered":"2024-09-13T14:49:34.598956+0000","last_clean":"2024-09-13T14:49:34.598956+0000","last_became_active":"2024-09-13T14:43:07.516816+0000","last_became_peered":"2024-09-13T14:43:07.516816+0000","last_unstale":"2024-09-13T14:49:34.598956+0000","last_undegraded":"2024-09-13T14:49:34.598956+0000","last_fullsized":"2024-09-13T14:49:34.598956+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:26.232184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"261'100","reported_seq":61,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.602644+0000","last_change":"2024-09-13T14:45:19.821649+0000","last_active":"2024-09-13T14:49:35.602644+0000","last_peered":"2024-09-13T14:49:35.602644+0000","last_clean":"2024-09-13T14:49:35.602644+0000","last_became_active":"2024-09-13T14:45:19.812346+0000","last_became_peered":"2024-09-13T14:45:19.812346+0000","last_unstale":"2024-09-13T14:49:35.602644+0000","last_undegraded":"2024-09-13T14:49:35.602644+0000","last_fullsized":"2024-09-13T14:49:35.602644+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.597568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.597568+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:45.218892+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"1171'674","reported_seq":686,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:42.731186+0000","last_change":"2024-09-13T14:49:22.120890+0000","last_active":"2024-09-13T14:49:42.731186+0000","last_peered":"2024-09-13T14:49:42.731186+0000","last_clean":"2024-09-13T14:49:42.731186+0000","last_became_active":"2024-09-13T14:49:22.120420+0000","last_became_peered":"2024-09-13T14:49:22.120420+0000","last_unstale":"2024-09-13T14:49:42.731186+0000","last_undegraded":"2024-09-13T14:49:42.731186+0000","last_fullsized":"2024-09-13T14:49:42.731186+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":674,"log_dups_size":0,"ondisk_log_size":674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:03:49.271123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44171264,"num_objects":674,"num_object_clones":0,"num_object_copies":2022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":674,"num_write_kb":43136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1096,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.743985+0000","last_change":"2024-09-13T14:19:26.887839+0000","last_active":"2024-09-13T14:49:34.743985+0000","last_peered":"2024-09-13T14:49:34.743985+0000","last_clean":"2024-09-13T14:49:34.743985+0000","last_became_active":"2024-09-13T14:19:26.887498+0000","last_became_peered":"2024-09-13T14:19:26.887498+0000","last_unstale":"2024-09-13T14:49:34.743985+0000","last_undegraded":"2024-09-13T14:49:34.743985+0000","last_fullsized":"2024-09-13T14:49:34.743985+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:24.475263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":1126,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.599032+0000","last_change":"2024-09-13T14:28:35.608607+0000","last_active":"2024-09-13T14:49:34.599032+0000","last_peered":"2024-09-13T14:49:34.599032+0000","last_clean":"2024-09-13T14:49:34.599032+0000","last_became_active":"2024-09-13T14:28:35.608254+0000","last_became_peered":"2024-09-13T14:28:35.608254+0000","last_unstale":"2024-09-13T14:49:34.599032+0000","last_undegraded":"2024-09-13T14:49:34.599032+0000","last_fullsized":"2024-09-13T14:49:34.599032+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:24.019173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"261'100","reported_seq":62,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.598634+0000","last_change":"2024-09-13T14:44:31.191431+0000","last_active":"2024-09-13T14:49:34.598634+0000","last_peered":"2024-09-13T14:49:34.598634+0000","last_clean":"2024-09-13T14:49:34.598634+0000","last_became_active":"2024-09-13T14:43:14.557376+0000","last_became_peered":"2024-09-13T14:43:14.557376+0000","last_unstale":"2024-09-13T14:49:34.598634+0000","last_undegraded":"2024-09-13T14:49:34.598634+0000","last_fullsized":"2024-09-13T14:49:34.598634+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.191326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.191326+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:15:27.352698+0000","scrub_duration":0.0025317360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1116,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.235227+0000","last_change":"2024-09-13T14:28:35.603724+0000","last_active":"2024-09-13T14:49:35.235227+0000","last_peered":"2024-09-13T14:49:35.235227+0000","last_clean":"2024-09-13T14:49:35.235227+0000","last_became_active":"2024-09-13T14:28:35.603316+0000","last_became_peered":"2024-09-13T14:28:35.603316+0000","last_unstale":"2024-09-13T14:49:35.235227+0000","last_undegraded":"2024-09-13T14:49:35.235227+0000","last_fullsized":"2024-09-13T14:49:35.235227+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:47.668223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":57,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.603063+0000","last_change":"2024-09-13T14:45:19.811466+0000","last_active":"2024-09-13T14:49:35.603063+0000","last_peered":"2024-09-13T14:49:35.603063+0000","last_clean":"2024-09-13T14:49:35.603063+0000","last_became_active":"2024-09-13T14:45:19.810617+0000","last_became_peered":"2024-09-13T14:45:19.810617+0000","last_unstale":"2024-09-13T14:49:35.603063+0000","last_undegraded":"2024-09-13T14:49:35.603063+0000","last_fullsized":"2024-09-13T14:49:35.603063+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:16.246669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"261'100","reported_seq":62,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.743941+0000","last_change":"2024-09-13T14:44:50.805306+0000","last_active":"2024-09-13T14:49:34.743941+0000","last_peered":"2024-09-13T14:49:34.743941+0000","last_clean":"2024-09-13T14:49:34.743941+0000","last_became_active":"2024-09-13T14:43:14.562922+0000","last_became_peered":"2024-09-13T14:43:14.562922+0000","last_unstale":"2024-09-13T14:49:34.743941+0000","last_undegraded":"2024-09-13T14:49:34.743941+0000","last_fullsized":"2024-09-13T14:49:34.743941+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:50.805210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:50.805210+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:28:38.529599+0000","scrub_duration":0.0042900009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1108,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.598166+0000","last_change":"2024-09-13T14:19:25.873088+0000","last_active":"2024-09-13T14:49:34.598166+0000","last_peered":"2024-09-13T14:49:34.598166+0000","last_clean":"2024-09-13T14:49:34.598166+0000","last_became_active":"2024-09-13T14:19:25.872781+0000","last_became_peered":"2024-09-13T14:19:25.872781+0000","last_unstale":"2024-09-13T14:49:34.598166+0000","last_undegraded":"2024-09-13T14:49:34.598166+0000","last_fullsized":"2024-09-13T14:49:34.598166+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:05.607382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1725,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.131386+0000","last_change":"2024-09-13T14:45:19.813876+0000","last_active":"2024-09-13T14:49:36.131386+0000","last_peered":"2024-09-13T14:49:36.131386+0000","last_clean":"2024-09-13T14:49:36.131386+0000","last_became_active":"2024-09-13T14:45:19.809136+0000","last_became_peered":"2024-09-13T14:45:19.809136+0000","last_unstale":"2024-09-13T14:49:36.131386+0000","last_undegraded":"2024-09-13T14:49:36.131386+0000","last_fullsized":"2024-09-13T14:49:36.131386+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:50.013528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"261'100","reported_seq":131,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.208880+0000","last_change":"2024-09-13T14:45:19.810654+0000","last_active":"2024-09-13T14:49:36.208880+0000","last_peered":"2024-09-13T14:49:36.208880+0000","last_clean":"2024-09-13T14:49:36.208880+0000","last_became_active":"2024-09-13T14:45:19.810066+0000","last_became_peered":"2024-09-13T14:45:19.810066+0000","last_unstale":"2024-09-13T14:49:36.208880+0000","last_undegraded":"2024-09-13T14:49:36.208880+0000","last_fullsized":"2024-09-13T14:49:36.208880+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:19.978464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:19.978464+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:33.639332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1773,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.131332+0000","last_change":"2024-09-13T14:45:19.809166+0000","last_active":"2024-09-13T14:49:36.131332+0000","last_peered":"2024-09-13T14:49:36.131332+0000","last_clean":"2024-09-13T14:49:36.131332+0000","last_became_active":"2024-09-13T14:45:19.808631+0000","last_became_peered":"2024-09-13T14:45:19.808631+0000","last_unstale":"2024-09-13T14:49:36.131332+0000","last_undegraded":"2024-09-13T14:49:36.131332+0000","last_fullsized":"2024-09-13T14:49:36.131332+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:38.822245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1133,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.744262+0000","last_change":"2024-09-13T14:45:19.804229+0000","last_active":"2024-09-13T14:49:34.744262+0000","last_peered":"2024-09-13T14:49:34.744262+0000","last_clean":"2024-09-13T14:49:34.744262+0000","last_became_active":"2024-09-13T14:45:19.803929+0000","last_became_peered":"2024-09-13T14:45:19.803929+0000","last_unstale":"2024-09-13T14:49:34.744262+0000","last_undegraded":"2024-09-13T14:49:34.744262+0000","last_fullsized":"2024-09-13T14:49:34.744262+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:33:53.675568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"261'100","reported_seq":62,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.743760+0000","last_change":"2024-09-13T14:44:42.654506+0000","last_active":"2024-09-13T14:49:34.743760+0000","last_peered":"2024-09-13T14:49:34.743760+0000","last_clean":"2024-09-13T14:49:34.743760+0000","last_became_active":"2024-09-13T14:43:14.563451+0000","last_became_peered":"2024-09-13T14:43:14.563451+0000","last_unstale":"2024-09-13T14:49:34.743760+0000","last_undegraded":"2024-09-13T14:49:34.743760+0000","last_fullsized":"2024-09-13T14:49:34.743760+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:42.654455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:42.654455+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:57:32.251597+0000","scrub_duration":0.001494097,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":1127,"reported_epoch":1170,"state":"active+clean","last_fresh":"2024-09-13T14:49:34.598303+0000","last_change":"2024-09-13T14:29:42.541093+0000","last_active":"2024-09-13T14:49:34.598303+0000","last_peered":"2024-09-13T14:49:34.598303+0000","last_clean":"2024-09-13T14:49:34.598303+0000","last_became_active":"2024-09-13T14:29:42.540671+0000","last_became_peered":"2024-09-13T14:29:42.540671+0000","last_unstale":"2024-09-13T14:49:34.598303+0000","last_undegraded":"2024-09-13T14:49:34.598303+0000","last_fullsized":"2024-09-13T14:49:34.598303+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:45.722376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1664,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.602784+0000","last_change":"2024-09-13T14:43:07.518679+0000","last_active":"2024-09-13T14:49:35.602784+0000","last_peered":"2024-09-13T14:49:35.602784+0000","last_clean":"2024-09-13T14:49:35.602784+0000","last_became_active":"2024-09-13T14:43:07.518333+0000","last_became_peered":"2024-09-13T14:43:07.518333+0000","last_unstale":"2024-09-13T14:49:35.602784+0000","last_undegraded":"2024-09-13T14:49:35.602784+0000","last_fullsized":"2024-09-13T14:49:35.602784+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:20.616274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"261'100","reported_seq":433,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.131337+0000","last_change":"2024-09-13T14:45:19.809268+0000","last_active":"2024-09-13T14:49:36.131337+0000","last_peered":"2024-09-13T14:49:36.131337+0000","last_clean":"2024-09-13T14:49:36.131337+0000","last_became_active":"2024-09-13T14:45:19.808772+0000","last_became_peered":"2024-09-13T14:45:19.808772+0000","last_unstale":"2024-09-13T14:49:36.131337+0000","last_undegraded":"2024-09-13T14:49:36.131337+0000","last_fullsized":"2024-09-13T14:49:36.131337+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:14.654603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:14.654603+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:40.964553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":867958784,"num_objects":13244,"num_object_clones":0,"num_object_copies":39732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13244,"num_write_kb":847616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2405498880,"data_stored":2405498880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13244,"ondisk_log_size":13244,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430596,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":479024,"kb_used_data":177512,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93266128,"statfs":{"total":95995035648,"available":95504515072,"internally_reserved":0,"allocated":181772288,"data_stored":178154869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626825,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1193380,"kb_used_data":607644,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":92551772,"statfs":{"total":95995035648,"available":94773014528,"internally_reserved":0,"allocated":622227456,"data_stored":618621662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823054,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":661856,"kb_used_data":360344,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93083296,"statfs":{"total":95995035648,"available":95317295104,"internally_reserved":0,"allocated":368992256,"data_stored":365386920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019282,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":476776,"kb_used_data":166944,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93268376,"statfs":{"total":95995035648,"available":95506817024,"internally_reserved":0,"allocated":170950656,"data_stored":167340815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059592940,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":618388,"kb_used_data":395660,"kb_used_omap":26,"kb_used_meta":222693,"kb_avail":93126764,"statfs":{"total":95995035648,"available":95361806336,"internally_reserved":0,"allocated":405155840,"data_stored":401561699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":228037792},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":591,"seq":2538325672178,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":581852,"kb_used_data":164628,"kb_used_omap":41,"kb_used_meta":417174,"kb_avail":93163300,"statfs":{"total":95995035648,"available":95399219200,"internally_reserved":0,"allocated":168579072,"data_stored":164977805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42651,"internal_metadata":427186533},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":557,"seq":2392296784121,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":693588,"kb_used_data":296268,"kb_used_omap":32,"kb_used_meta":397279,"kb_avail":93051564,"statfs":{"total":95995035648,"available":95284801536,"internally_reserved":0,"allocated":303378432,"data_stored":299784236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33752,"internal_metadata":406813736},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":524,"seq":2250562863360,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":722456,"kb_used_data":374608,"kb_used_omap":41,"kb_used_meta":347798,"kb_avail":93022696,"statfs":{"total":95995035648,"available":95255240704,"internally_reserved":0,"allocated":383598592,"data_stored":380000881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42694,"internal_metadata":356145466},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":411172864,"data_stored":411172864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":333840384,"data_stored":333840384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":227672064,"data_stored":227672064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":407699456,"data_stored":407699456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":216072192,"data_stored":216072192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":361758720,"data_stored":361758720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":222363648,"data_stored":222363648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":224919552,"data_stored":224919552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:49:48.488 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:48.520 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:48.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:48.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:48.622 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:48.653 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:48.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:48.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:48.755 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:48.789 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:48.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:48.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:48.891 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:48.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:48.934 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:48.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:49.035 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:49.058 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:49.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:49.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:49.160 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:49.211 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:49.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:49.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:49.217 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 10 16 27839 27823 173.873 128.562 0.0102723 0.00574578 2024-09-13T14:49:49.313 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:49.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:49.335 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:49.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:49.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:49 smithi184 ceph-mon[92527]: pgmap v1688: 119 pgs: 119 active+clean; 880 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-13T14:49:49.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:49 smithi151 ceph-mon[104450]: pgmap v1688: 119 pgs: 119 active+clean; 880 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-13T14:49:49.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:49 smithi151 ceph-mon[108966]: pgmap v1688: 119 pgs: 119 active+clean; 880 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-13T14:49:49.436 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:49.474 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:49.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:49.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:49.576 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:49.601 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:49.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:49.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:49.703 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:49.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:49.736 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:49.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:49.837 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:49.875 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:49.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:49.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:49.976 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:49.998 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:49.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:49.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:50.100 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:50.132 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:50.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:50.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:50.217 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 11 16 29876 29860 169.64 127.312 0.00696607 0.0058894 2024-09-13T14:49:50.233 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:50.269 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:50.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:50.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:50.371 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:50.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:50.394 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:50.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:50.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:50.537 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:50.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:50.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:50.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:49:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:49:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:49:50.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:50.675 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:50.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:50.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:50.777 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:50.798 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:50.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:50.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:50.900 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:50.921 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:50.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:50.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:51.024 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:51.059 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:51.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:51.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:51.160 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:51.183 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:51.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:51.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:51.217 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 12 16 31331 31315 163.081 90.9375 0.00451353 0.00612037 2024-09-13T14:49:51.285 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:51.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:51.321 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:51.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:51.418 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:49:51.421 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:51.456 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:51 smithi151 ceph-mon[104450]: pgmap v1689: 119 pgs: 119 active+clean; 1.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-09-13T14:49:51.456 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:51 smithi151 ceph-mon[108966]: pgmap v1689: 119 pgs: 119 active+clean; 1.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-09-13T14:49:51.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:51.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:51.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:51.595 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:51.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:51 smithi184 ceph-mon[92527]: pgmap v1689: 119 pgs: 119 active+clean; 1.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-09-13T14:49:51.632 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:51.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:51.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:51.734 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:51.766 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:51.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:51.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:51.868 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:51.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:51.891 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:51.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:51.992 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:52.023 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:52.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:52.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:52.125 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:52.160 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:52.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:52.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:52.217 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 13 16 33129 33113 159.179 112.375 0.00989221 0.00627681 2024-09-13T14:49:52.262 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:52.293 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:52.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:52.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:52.395 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:52.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:52.438 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:52.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:52.539 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:52.569 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:52.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:52.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:52.671 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:52.700 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:52.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:52.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:52.802 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:52.834 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:52.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:52.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:52.936 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:52.967 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:52.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:52.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:53.069 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:53.098 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:53.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:53.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:53.200 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:53.217 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 14 16 35135 35119 156.764 125.375 0.00535496 0.00637291 2024-09-13T14:49:53.231 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:53.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:53.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:53.333 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:53.365 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:53.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:53.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:53.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:53.504 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:53.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:53.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:53.527 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-13T14:49:53.527 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:53 smithi151 ceph-mon[104450]: pgmap v1690: 119 pgs: 119 active+clean; 1.6 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-09-13T14:49:53.527 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:53 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2148769326' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:49:53.528 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:53 smithi151 ceph-mon[108966]: pgmap v1690: 119 pgs: 119 active+clean; 1.6 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-09-13T14:49:53.528 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:53 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2148769326' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:49:53.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:53 smithi184 ceph-mon[92527]: pgmap v1690: 119 pgs: 119 active+clean; 1.6 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-09-13T14:49:53.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:53 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2148769326' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:49:53.606 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:53.645 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:53.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:53.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:53.747 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:53.768 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:53.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:53.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:53.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:53.911 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:53.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:53.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:54.013 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:54.034 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:54.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:54.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:54.135 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:54.173 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:54.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:54.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:54.217 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 15 16 37014 36998 154.142 117.438 0.00668442 0.00648137 2024-09-13T14:49:54.276 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:54.305 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:54.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:54.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:54.407 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:54.436 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:54.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:54.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:54.538 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:54.576 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:54.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:54.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:54.590 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:54 smithi151 ceph-mon[104450]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:54.590 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:54 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2148769326' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:49:54.590 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:54 smithi151 ceph-mon[104450]: osdmap e1172: 8 total, 8 up, 8 in 2024-09-13T14:49:54.591 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:54 smithi151 ceph-mon[108966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:54.591 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:54 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2148769326' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:49:54.591 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:54 smithi151 ceph-mon[108966]: osdmap e1172: 8 total, 8 up, 8 in 2024-09-13T14:49:54.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:54 smithi184 ceph-mon[92527]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:54.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:54 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2148769326' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:49:54.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:54 smithi184 ceph-mon[92527]: osdmap e1172: 8 total, 8 up, 8 in 2024-09-13T14:49:54.679 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:54.707 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:54.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:54.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:54.809 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:54.844 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:54.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:54.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:54.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:49:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:49:54] "GET /metrics HTTP/1.1" 200 36635 "" "Prometheus/2.51.0" 2024-09-13T14:49:54.946 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:54.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:54.977 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:54.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:55.078 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:55.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:55.109 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:55.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:55.210 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:55.217 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 16 16 38786 38770 151.429 110.75 0.00562401 0.00659848 2024-09-13T14:49:55.243 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:55.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:55.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:55.345 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:55.382 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:55.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:55.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:55.484 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:55.519 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:55.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:55.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:55.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:55 smithi184 ceph-mon[92527]: pgmap v1692: 119 pgs: 119 active+clean; 2.0 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 155 MiB/s wr, 2.48k op/s 2024-09-13T14:49:55.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:55 smithi151 ceph-mon[104450]: pgmap v1692: 119 pgs: 119 active+clean; 2.0 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 155 MiB/s wr, 2.48k op/s 2024-09-13T14:49:55.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:55 smithi151 ceph-mon[108966]: pgmap v1692: 119 pgs: 119 active+clean; 2.0 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 155 MiB/s wr, 2.48k op/s 2024-09-13T14:49:55.621 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:55.659 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:55.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:55.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:55.761 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:55.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:55.799 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:55.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:55.900 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:55.931 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:55.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:55.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:56.033 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:56.071 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:56.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:56.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:56.173 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:56.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:56.201 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:56.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:56.217 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 17 16 40368 40352 148.337 98.875 0.00561646 0.00673672 2024-09-13T14:49:56.302 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:56.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:56.334 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:56.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:56.435 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:56.469 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:56.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:56.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:56.571 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:56.602 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:56.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:56.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:56.704 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:56.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:56.740 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:56.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:56.841 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:56.862 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:56.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:56.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:56.964 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:57.004 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:57.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:57.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:57.106 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:57.143 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:57.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:57.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:57.217 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 18 16 42076 42060 146.026 106.75 0.00795363 0.00684253 2024-09-13T14:49:57.245 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:57.274 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:57.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:57.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:57.375 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:57.397 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:57.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:57.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:57.499 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:57.530 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:57.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:57.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:57.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:57 smithi184 ceph-mon[92527]: pgmap v1693: 119 pgs: 119 active+clean; 2.2 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 140 MiB/s wr, 2.24k op/s 2024-09-13T14:49:57.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:57 smithi151 ceph-mon[104450]: pgmap v1693: 119 pgs: 119 active+clean; 2.2 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 140 MiB/s wr, 2.24k op/s 2024-09-13T14:49:57.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:57 smithi151 ceph-mon[108966]: pgmap v1693: 119 pgs: 119 active+clean; 2.2 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 140 MiB/s wr, 2.24k op/s 2024-09-13T14:49:57.632 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:57.662 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:57.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:57.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:57.763 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:57.784 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:57.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:57.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:57.886 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:57.921 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:57.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:57.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:58.023 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:58.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:58.048 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:58.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:58.149 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:58.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:58.187 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:58.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:58.217 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 19 16 43351 43335 142.535 79.6875 0.00384707 0.00701062 2024-09-13T14:49:58.288 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:58.324 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:58.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:58.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:58.426 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:58.447 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:58.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:58.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:58.549 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:58.583 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:58.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:58.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:58.689 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:58.724 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:58.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:58.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:58.826 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:58.857 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:58.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:58.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:58.959 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:58.995 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:49:58.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:58.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:59.097 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:59.129 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:59.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:59.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:59.218 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:2024-09-13T14:49:59.217685+0000 min lat: 0.00100112 max lat: 0.247171 avg lat: 0.00711101 2024-09-13T14:49:59.218 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:49:59.218 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 20 16 44870 44854 140.154 94.9375 0.00199365 0.00711101 2024-09-13T14:49:59.231 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:59.253 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:59.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:59.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:59.355 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:59.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:49:59 smithi151 ceph-mon[104450]: pgmap v1694: 119 pgs: 119 active+clean; 2.2 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-09-13T14:49:59.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:49:59 smithi151 ceph-mon[108966]: pgmap v1694: 119 pgs: 119 active+clean; 2.2 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-09-13T14:49:59.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:59.392 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:59.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:59.493 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:59.529 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:59.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:59.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:59.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:49:59 smithi184 ceph-mon[92527]: pgmap v1694: 119 pgs: 119 active+clean; 2.2 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-09-13T14:49:59.631 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:59.667 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:59.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:59.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:59.768 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:59.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:59.810 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:59.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:59.911 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:59.949 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:49:59.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:59.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:00.053 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:00.083 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:00.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:00.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:00.185 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:00.209 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:00.213 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:00.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:00.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:00.218 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 21 16 46577 46561 138.557 106.688 0.00578683 0.00721155 2024-09-13T14:50:00.315 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:00.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:00.356 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:00.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:00.457 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:00.478 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:00 smithi151 ceph-mon[104450]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-09-13T14:50:00.478 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:00 smithi151 ceph-mon[104450]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-13T14:50:00.479 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:00 smithi151 ceph-mon[108966]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-09-13T14:50:00.479 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:00 smithi151 ceph-mon[108966]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-13T14:50:00.500 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:00.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:00.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:00.602 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:00.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:00 smithi184 ceph-mon[92527]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-09-13T14:50:00.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:00 smithi184 ceph-mon[92527]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-13T14:50:00.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:50:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:50:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:50:00.641 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:00.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:00.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:00.743 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:00.772 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:00.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:00.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:00.874 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:00.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:00.909 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:00.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:01.010 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:01.019 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:50:01.019 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-13T14:50:01.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-13T14:50:01.019 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-13T14:50:01.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.019 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.b e1127,1124 6:daaee32b:::benchmark_data_smithi151_155836_object30047:head,1105'10241)", 2024-09-13T14:50:01.019 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:13.511269+0000", 2024-09-13T14:50:01.019 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 407.50352029099997, 2024-09-13T14:50:01.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25709800100000002, 2024-09-13T14:50:01.019 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:50:01.020 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:50:01.020 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-13T14:50:01.020 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:50:01.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:50:01.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.511269+0000", 2024-09-13T14:50:01.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:50:01.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.511269+0000", 2024-09-13T14:50:01.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:50:01.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.511270+0000", 2024-09-13T14:50:01.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2900000000000003e-07 2024-09-13T14:50:01.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:50:01.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.511276+0000", 2024-09-13T14:50:01.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2999999999999998e-06 2024-09-13T14:50:01.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:50:01.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.511287+0000", 2024-09-13T14:50:01.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0934999999999999e-05 2024-09-13T14:50:01.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:50:01.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.511294+0000", 2024-09-13T14:50:01.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0890000000000002e-06 2024-09-13T14:50:01.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:50:01.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.768304+0000", 2024-09-13T14:50:01.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25701023899999997 2024-09-13T14:50:01.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:50:01.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.768318+0000", 2024-09-13T14:50:01.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.398e-05 2024-09-13T14:50:01.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:50:01.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.768367+0000", 2024-09-13T14:50:01.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8829000000000003e-05 2024-09-13T14:50:01.027 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.027 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:50:01.027 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.028 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:61877 7.d 7:bf2b88b4:::benchmark_data_smithi151_190906_object61876:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1142)", 2024-09-13T14:50:01.028 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:44:33.512449+0000", 2024-09-13T14:50:01.028 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 327.502339936, 2024-09-13T14:50:01.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25874180499999999, 2024-09-13T14:50:01.028 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:50:01.028 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:50:01.028 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:01.028 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:50:01.028 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:50:01.029 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:50:01.029 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 61877 2024-09-13T14:50:01.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.029 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:50:01.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:50:01.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.512449+0000", 2024-09-13T14:50:01.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:50:01.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.512449+0000", 2024-09-13T14:50:01.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:50:01.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.512450+0000", 2024-09-13T14:50:01.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.82e-07 2024-09-13T14:50:01.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:50:01.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.512468+0000", 2024-09-13T14:50:01.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8641e-05 2024-09-13T14:50:01.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:50:01.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.512475+0000", 2024-09-13T14:50:01.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7510000000000002e-06 2024-09-13T14:50:01.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:50:01.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.512480+0000", 2024-09-13T14:50:01.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0950000000000003e-06 2024-09-13T14:50:01.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:50:01.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.517452+0000", 2024-09-13T14:50:01.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0049716869999999998 2024-09-13T14:50:01.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:50:01.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.517613+0000", 2024-09-13T14:50:01.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016100700000000001 2024-09-13T14:50:01.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:50:01.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.517755+0000", 2024-09-13T14:50:01.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014175799999999999 2024-09-13T14:50:01.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.520879+0000", 2024-09-13T14:50:01.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031243199999999999 2024-09-13T14:50:01.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:50:01.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.521303+0000", 2024-09-13T14:50:01.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00042433299999999999 2024-09-13T14:50:01.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.768327+0000", 2024-09-13T14:50:01.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24702404 2024-09-13T14:50:01.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:50:01.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.771144+0000", 2024-09-13T14:50:01.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002816364 2024-09-13T14:50:01.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:50:01.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:44:33.771191+0000", 2024-09-13T14:50:01.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7327000000000003e-05 2024-09-13T14:50:01.034 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.034 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:50:01.034 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.035 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:196311 7.a 7:5cc06252:::benchmark_data_smithi151_190906_object75850:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1160)", 2024-09-13T14:50:01.035 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:47:36.567686+0000", 2024-09-13T14:50:01.035 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 144.44710330000001, 2024-09-13T14:50:01.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28348468199999999, 2024-09-13T14:50:01.035 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:50:01.035 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:50:01.035 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:01.035 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:50:01.035 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:50:01.035 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:50:01.035 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 196311 2024-09-13T14:50:01.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.036 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:50:01.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:50:01.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.567686+0000", 2024-09-13T14:50:01.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:50:01.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.567686+0000", 2024-09-13T14:50:01.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:50:01.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.567687+0000", 2024-09-13T14:50:01.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.305e-06 2024-09-13T14:50:01.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:50:01.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.567692+0000", 2024-09-13T14:50:01.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9200000000000003e-06 2024-09-13T14:50:01.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:50:01.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.567702+0000", 2024-09-13T14:50:01.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0276e-05 2024-09-13T14:50:01.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:50:01.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.567709+0000", 2024-09-13T14:50:01.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.224e-06 2024-09-13T14:50:01.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:50:01.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.610177+0000", 2024-09-13T14:50:01.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.042468738999999998 2024-09-13T14:50:01.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:50:01.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.610534+0000", 2024-09-13T14:50:01.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00035662299999999998 2024-09-13T14:50:01.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,3,5", 2024-09-13T14:50:01.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.611118+0000", 2024-09-13T14:50:01.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00058423899999999996 2024-09-13T14:50:01.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.632034+0000", 2024-09-13T14:50:01.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020915789000000001 2024-09-13T14:50:01.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:50:01.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.632166+0000", 2024-09-13T14:50:01.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013152899999999999 2024-09-13T14:50:01.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.635459+0000", 2024-09-13T14:50:01.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003293645 2024-09-13T14:50:01.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.847655+0000", 2024-09-13T14:50:01.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.212196091 2024-09-13T14:50:01.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:50:01.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.847755+0000", 2024-09-13T14:50:01.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9203000000000005e-05 2024-09-13T14:50:01.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:50:01.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.851171+0000", 2024-09-13T14:50:01.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034160990000000001 2024-09-13T14:50:01.044 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.044 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:50:01.044 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.044 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:196322 7.2 7:44d41967:::benchmark_data_smithi151_190906_object75861:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1160)", 2024-09-13T14:50:01.044 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:47:36.567760+0000", 2024-09-13T14:50:01.044 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 144.447029181, 2024-09-13T14:50:01.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27070144200000001, 2024-09-13T14:50:01.044 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:50:01.044 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:50:01.045 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:01.045 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:50:01.045 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:50:01.045 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:50:01.045 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 196322 2024-09-13T14:50:01.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.045 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:50:01.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:50:01.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.567760+0000", 2024-09-13T14:50:01.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:50:01.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.567760+0000", 2024-09-13T14:50:01.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:50:01.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.567760+0000", 2024-09-13T14:50:01.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3700000000000001e-07 2024-09-13T14:50:01.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:50:01.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.567762+0000", 2024-09-13T14:50:01.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.53e-06 2024-09-13T14:50:01.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:50:01.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.567767+0000", 2024-09-13T14:50:01.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.955e-06 2024-09-13T14:50:01.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:50:01.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.567771+0000", 2024-09-13T14:50:01.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1829999999999996e-06 2024-09-13T14:50:01.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:50:01.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.610019+0000", 2024-09-13T14:50:01.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.042248007999999997 2024-09-13T14:50:01.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:50:01.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.620710+0000", 2024-09-13T14:50:01.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010690837999999999 2024-09-13T14:50:01.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,3,7", 2024-09-13T14:50:01.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.620902+0000", 2024-09-13T14:50:01.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019165999999999999 2024-09-13T14:50:01.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.627177+0000", 2024-09-13T14:50:01.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0062753160000000004 2024-09-13T14:50:01.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:50:01.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.632176+0000", 2024-09-13T14:50:01.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0049988109999999997 2024-09-13T14:50:01.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.050 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.635628+0000", 2024-09-13T14:50:01.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034521740000000001 2024-09-13T14:50:01.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.050 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.838226+0000", 2024-09-13T14:50:01.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.202598054 2024-09-13T14:50:01.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.050 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:50:01.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.838334+0000", 2024-09-13T14:50:01.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000107778 2024-09-13T14:50:01.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.051 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:50:01.051 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.838462+0000", 2024-09-13T14:50:01.051 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012779800000000001 2024-09-13T14:50:01.051 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.051 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:50:01.051 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.051 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:196332 7.a 7:53af4101:::benchmark_data_smithi151_190906_object75871:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1160)", 2024-09-13T14:50:01.051 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:47:36.590819+0000", 2024-09-13T14:50:01.051 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 144.423969959, 2024-09-13T14:50:01.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.263847096, 2024-09-13T14:50:01.052 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:50:01.052 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:50:01.052 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:01.052 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:50:01.052 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:50:01.052 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:50:01.052 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 196332 2024-09-13T14:50:01.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.052 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:50:01.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.052 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:50:01.052 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.590819+0000", 2024-09-13T14:50:01.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.053 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:50:01.053 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.590819+0000", 2024-09-13T14:50:01.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.053 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:50:01.053 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.590821+0000", 2024-09-13T14:50:01.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9089999999999998e-06 2024-09-13T14:50:01.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:50:01.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.590828+0000", 2024-09-13T14:50:01.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.46e-06 2024-09-13T14:50:01.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:50:01.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.590846+0000", 2024-09-13T14:50:01.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8776000000000001e-05 2024-09-13T14:50:01.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:50:01.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.590855+0000", 2024-09-13T14:50:01.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.755e-06 2024-09-13T14:50:01.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:50:01.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.622389+0000", 2024-09-13T14:50:01.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.031533897999999998 2024-09-13T14:50:01.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:50:01.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.622533+0000", 2024-09-13T14:50:01.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014422799999999999 2024-09-13T14:50:01.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.056 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,3,5", 2024-09-13T14:50:01.056 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.622680+0000", 2024-09-13T14:50:01.056 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000146422 2024-09-13T14:50:01.056 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.056 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.056 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.632092+0000", 2024-09-13T14:50:01.056 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0094117899999999997 2024-09-13T14:50:01.056 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.056 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:50:01.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.632184+0000", 2024-09-13T14:50:01.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2275000000000007e-05 2024-09-13T14:50:01.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.640361+0000", 2024-09-13T14:50:01.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0081772219999999996 2024-09-13T14:50:01.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.851271+0000", 2024-09-13T14:50:01.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.210910299 2024-09-13T14:50:01.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.058 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:50:01.058 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.851340+0000", 2024-09-13T14:50:01.058 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8311000000000004e-05 2024-09-13T14:50:01.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.058 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:50:01.058 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:36.854666+0000", 2024-09-13T14:50:01.058 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033267510000000002 2024-09-13T14:50:01.058 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.058 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:50:01.058 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.059 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:198469 7.a 7:5f7e472e:::benchmark_data_smithi151_190906_object78008:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1160)", 2024-09-13T14:50:01.059 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:47:40.111658+0000", 2024-09-13T14:50:01.059 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 140.90313090399999, 2024-09-13T14:50:01.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.41094132900000002, 2024-09-13T14:50:01.059 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:50:01.059 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:50:01.059 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:01.059 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:50:01.059 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:50:01.059 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:50:01.060 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 198469 2024-09-13T14:50:01.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.060 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:50:01.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.060 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:50:01.060 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.111658+0000", 2024-09-13T14:50:01.060 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.060 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:50:01.060 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.111658+0000", 2024-09-13T14:50:01.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:50:01.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.111659+0000", 2024-09-13T14:50:01.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7700000000000005e-07 2024-09-13T14:50:01.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:50:01.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.111661+0000", 2024-09-13T14:50:01.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5610000000000001e-06 2024-09-13T14:50:01.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.062 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:50:01.062 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.111668+0000", 2024-09-13T14:50:01.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0920000000000003e-06 2024-09-13T14:50:01.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.062 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:50:01.062 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.111673+0000", 2024-09-13T14:50:01.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1440000000000002e-06 2024-09-13T14:50:01.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.062 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:50:01.063 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.135179+0000", 2024-09-13T14:50:01.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.023506484000000001 2024-09-13T14:50:01.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.063 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:50:01.063 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.135320+0000", 2024-09-13T14:50:01.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014057800000000001 2024-09-13T14:50:01.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.063 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,3,5", 2024-09-13T14:50:01.063 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.135851+0000", 2024-09-13T14:50:01.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00053118399999999997 2024-09-13T14:50:01.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:50:01.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.154575+0000", 2024-09-13T14:50:01.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018724405999999999 2024-09-13T14:50:01.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.154835+0000", 2024-09-13T14:50:01.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00025918700000000002 2024-09-13T14:50:01.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.065 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.154914+0000", 2024-09-13T14:50:01.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9190999999999994e-05 2024-09-13T14:50:01.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.065 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.518574+0000", 2024-09-13T14:50:01.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.36366067600000002 2024-09-13T14:50:01.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:50:01.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.518629+0000", 2024-09-13T14:50:01.066 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4175999999999998e-05 2024-09-13T14:50:01.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:50:01.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.522600+0000", 2024-09-13T14:50:01.066 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039711729999999997 2024-09-13T14:50:01.066 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.067 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:50:01.067 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.067 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:198480 7.a 7:57c00ec3:::benchmark_data_smithi151_190906_object78019:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1160)", 2024-09-13T14:50:01.067 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:47:40.131626+0000", 2024-09-13T14:50:01.067 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 140.88316288199999, 2024-09-13T14:50:01.067 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.40130970700000002, 2024-09-13T14:50:01.067 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:50:01.067 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:50:01.067 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:01.067 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:50:01.067 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:50:01.068 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:50:01.068 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 198480 2024-09-13T14:50:01.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.068 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:50:01.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.068 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:50:01.068 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.131626+0000", 2024-09-13T14:50:01.068 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.068 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:50:01.068 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.131626+0000", 2024-09-13T14:50:01.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.069 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:50:01.069 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.131628+0000", 2024-09-13T14:50:01.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.593e-06 2024-09-13T14:50:01.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.069 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:50:01.069 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.131632+0000", 2024-09-13T14:50:01.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8550000000000004e-06 2024-09-13T14:50:01.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.069 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:50:01.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.131652+0000", 2024-09-13T14:50:01.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0424e-05 2024-09-13T14:50:01.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:50:01.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.131659+0000", 2024-09-13T14:50:01.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7920000000000004e-06 2024-09-13T14:50:01.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:50:01.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.147257+0000", 2024-09-13T14:50:01.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015598107999999999 2024-09-13T14:50:01.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.071 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:50:01.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.147423+0000", 2024-09-13T14:50:01.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016567400000000001 2024-09-13T14:50:01.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.071 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,3,5", 2024-09-13T14:50:01.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.147599+0000", 2024-09-13T14:50:01.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000175854 2024-09-13T14:50:01.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.071 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.072 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.154874+0000", 2024-09-13T14:50:01.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0072752720000000002 2024-09-13T14:50:01.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.072 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.072 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.159034+0000", 2024-09-13T14:50:01.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0041596790000000003 2024-09-13T14:50:01.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.072 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:50:01.072 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.163345+0000", 2024-09-13T14:50:01.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0043108720000000003 2024-09-13T14:50:01.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.073 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.073 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.523241+0000", 2024-09-13T14:50:01.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35989649699999998 2024-09-13T14:50:01.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.073 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:50:01.073 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.523297+0000", 2024-09-13T14:50:01.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6319000000000001e-05 2024-09-13T14:50:01.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.074 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:50:01.074 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.532936+0000", 2024-09-13T14:50:01.074 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0096387680000000007 2024-09-13T14:50:01.074 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.074 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:50:01.074 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.074 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:198489 7.a 7:5f5f3bb7:::benchmark_data_smithi151_190906_object78028:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1160)", 2024-09-13T14:50:01.074 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:47:40.136221+0000", 2024-09-13T14:50:01.075 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 140.87856818099999, 2024-09-13T14:50:01.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.41120310300000001, 2024-09-13T14:50:01.075 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:50:01.075 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:50:01.075 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:01.075 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:50:01.075 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:50:01.075 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:50:01.075 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 198489 2024-09-13T14:50:01.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.075 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:50:01.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.076 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:50:01.076 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.136221+0000", 2024-09-13T14:50:01.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.076 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:50:01.076 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.136221+0000", 2024-09-13T14:50:01.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.076 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:50:01.076 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.136222+0000", 2024-09-13T14:50:01.077 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8100000000000003e-07 2024-09-13T14:50:01.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.077 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:50:01.077 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.136224+0000", 2024-09-13T14:50:01.077 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1880000000000001e-06 2024-09-13T14:50:01.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.077 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:50:01.077 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.136240+0000", 2024-09-13T14:50:01.077 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6052000000000001e-05 2024-09-13T14:50:01.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.078 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:50:01.078 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.136250+0000", 2024-09-13T14:50:01.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9620000000000005e-06 2024-09-13T14:50:01.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.078 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:50:01.078 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.151025+0000", 2024-09-13T14:50:01.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014775447000000001 2024-09-13T14:50:01.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.078 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:50:01.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.151171+0000", 2024-09-13T14:50:01.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000145792 2024-09-13T14:50:01.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,3,5", 2024-09-13T14:50:01.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.151320+0000", 2024-09-13T14:50:01.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000148856 2024-09-13T14:50:01.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.158981+0000", 2024-09-13T14:50:01.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0076612479999999998 2024-09-13T14:50:01.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.159106+0000", 2024-09-13T14:50:01.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012490900000000001 2024-09-13T14:50:01.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:50:01.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.163352+0000", 2024-09-13T14:50:01.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0042464160000000003 2024-09-13T14:50:01.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.543338+0000", 2024-09-13T14:50:01.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37998550399999997 2024-09-13T14:50:01.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:50:01.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.543406+0000", 2024-09-13T14:50:01.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8402e-05 2024-09-13T14:50:01.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:50:01.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.547424+0000", 2024-09-13T14:50:01.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0040178460000000003 2024-09-13T14:50:01.082 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.082 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:50:01.082 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.082 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:198494 7.a 7:5a35b70b:::benchmark_data_smithi151_190906_object78033:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1160)", 2024-09-13T14:50:01.082 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:47:40.149282+0000", 2024-09-13T14:50:01.082 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 140.86550707699999, 2024-09-13T14:50:01.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.42094174499999998, 2024-09-13T14:50:01.082 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:50:01.082 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:50:01.082 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:01.083 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:50:01.083 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:50:01.083 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:50:01.083 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 198494 2024-09-13T14:50:01.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.083 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:50:01.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:50:01.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.149282+0000", 2024-09-13T14:50:01.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:50:01.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.149282+0000", 2024-09-13T14:50:01.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.084 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:50:01.084 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.149284+0000", 2024-09-13T14:50:01.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5689999999999999e-06 2024-09-13T14:50:01.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.084 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:50:01.084 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.149289+0000", 2024-09-13T14:50:01.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.507e-06 2024-09-13T14:50:01.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.085 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:50:01.085 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.149300+0000", 2024-09-13T14:50:01.085 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0361e-05 2024-09-13T14:50:01.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.085 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:50:01.085 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.149308+0000", 2024-09-13T14:50:01.085 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2500000000000006e-06 2024-09-13T14:50:01.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.085 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:50:01.085 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.155135+0000", 2024-09-13T14:50:01.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0058269400000000001 2024-09-13T14:50:01.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:50:01.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.155272+0000", 2024-09-13T14:50:01.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013690299999999999 2024-09-13T14:50:01.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,3,5", 2024-09-13T14:50:01.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.155416+0000", 2024-09-13T14:50:01.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014426599999999999 2024-09-13T14:50:01.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.159070+0000", 2024-09-13T14:50:01.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:01.087 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:01.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:01.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0036544709999999998 2024-09-13T14:50:01.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:50:01.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.163361+0000", 2024-09-13T14:50:01.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0042903409999999996 2024-09-13T14:50:01.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.165033+0000", 2024-09-13T14:50:01.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001671919 2024-09-13T14:50:01.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.565033+0000", 2024-09-13T14:50:01.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.40000057100000003 2024-09-13T14:50:01.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:50:01.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.565242+0000", 2024-09-13T14:50:01.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000209082 2024-09-13T14:50:01.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:50:01.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.570224+0000", 2024-09-13T14:50:01.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0049815650000000003 2024-09-13T14:50:01.091 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.091 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:50:01.091 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.091 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:198500 7.a 7:5a732806:::benchmark_data_smithi151_190906_object78039:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1160)", 2024-09-13T14:50:01.091 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:47:40.156065+0000", 2024-09-13T14:50:01.091 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 140.858724172, 2024-09-13T14:50:01.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.42361019100000002, 2024-09-13T14:50:01.091 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:50:01.091 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:50:01.091 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:01.092 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:50:01.092 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:50:01.092 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:50:01.092 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 198500 2024-09-13T14:50:01.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.092 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:50:01.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:50:01.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.156065+0000", 2024-09-13T14:50:01.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.093 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:50:01.093 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.156065+0000", 2024-09-13T14:50:01.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.093 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:50:01.093 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.156066+0000", 2024-09-13T14:50:01.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.048e-06 2024-09-13T14:50:01.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.093 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:50:01.093 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.156069+0000", 2024-09-13T14:50:01.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1690000000000001e-06 2024-09-13T14:50:01.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:50:01.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.156078+0000", 2024-09-13T14:50:01.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3299999999999999e-06 2024-09-13T14:50:01.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:50:01.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.156084+0000", 2024-09-13T14:50:01.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9710000000000001e-06 2024-09-13T14:50:01.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:50:01.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.159134+0000", 2024-09-13T14:50:01.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030501980000000001 2024-09-13T14:50:01.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:50:01.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.159409+0000", 2024-09-13T14:50:01.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000275006 2024-09-13T14:50:01.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,3,5", 2024-09-13T14:50:01.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.159804+0000", 2024-09-13T14:50:01.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039560499999999999 2024-09-13T14:50:01.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.163265+0000", 2024-09-13T14:50:01.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034601850000000002 2024-09-13T14:50:01.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:50:01.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.163367+0000", 2024-09-13T14:50:01.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102063 2024-09-13T14:50:01.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.167086+0000", 2024-09-13T14:50:01.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037189979999999998 2024-09-13T14:50:01.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.575691+0000", 2024-09-13T14:50:01.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.40860484499999999 2024-09-13T14:50:01.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:50:01.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.575746+0000", 2024-09-13T14:50:01.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5921999999999998e-05 2024-09-13T14:50:01.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.098 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:50:01.098 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.579675+0000", 2024-09-13T14:50:01.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039288509999999997 2024-09-13T14:50:01.098 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.098 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:50:01.098 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.098 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:198504 7.2 7:4589c3ea:::benchmark_data_smithi151_190906_object78043:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1160)", 2024-09-13T14:50:01.098 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:47:40.160407+0000", 2024-09-13T14:50:01.098 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 140.854382668, 2024-09-13T14:50:01.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.408265775, 2024-09-13T14:50:01.099 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:50:01.099 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:50:01.099 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:01.099 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:50:01.099 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:50:01.099 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:50:01.099 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 198504 2024-09-13T14:50:01.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.099 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:50:01.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:50:01.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.160407+0000", 2024-09-13T14:50:01.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:50:01.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.160407+0000", 2024-09-13T14:50:01.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:50:01.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.160407+0000", 2024-09-13T14:50:01.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3600000000000004e-07 2024-09-13T14:50:01.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:50:01.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.160411+0000", 2024-09-13T14:50:01.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.45e-06 2024-09-13T14:50:01.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:50:01.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.160419+0000", 2024-09-13T14:50:01.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7749999999999998e-06 2024-09-13T14:50:01.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:50:01.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.160424+0000", 2024-09-13T14:50:01.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9479999999999999e-06 2024-09-13T14:50:01.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:50:01.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.163309+0000", 2024-09-13T14:50:01.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028849090000000002 2024-09-13T14:50:01.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:50:01.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.163449+0000", 2024-09-13T14:50:01.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013961999999999999 2024-09-13T14:50:01.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,3,7", 2024-09-13T14:50:01.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.163603+0000", 2024-09-13T14:50:01.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015405400000000001 2024-09-13T14:50:01.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.169896+0000", 2024-09-13T14:50:01.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0062931159999999996 2024-09-13T14:50:01.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:50:01.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.169910+0000", 2024-09-13T14:50:01.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3818000000000001e-05 2024-09-13T14:50:01.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.169957+0000", 2024-09-13T14:50:01.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6989e-05 2024-09-13T14:50:01.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.568470+0000", 2024-09-13T14:50:01.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.39851281799999999 2024-09-13T14:50:01.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:50:01.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.568542+0000", 2024-09-13T14:50:01.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2440000000000004e-05 2024-09-13T14:50:01.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:50:01.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.568672+0000", 2024-09-13T14:50:01.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000130302 2024-09-13T14:50:01.105 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.105 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:50:01.105 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.106 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:198505 7.2 7:4f3aeb81:::benchmark_data_smithi151_190906_object78044:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1160)", 2024-09-13T14:50:01.106 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:47:40.165409+0000", 2024-09-13T14:50:01.106 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 140.84938011099999, 2024-09-13T14:50:01.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.403485971, 2024-09-13T14:50:01.106 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:50:01.106 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:50:01.106 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:01.106 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:50:01.106 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:50:01.107 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:50:01.107 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 198505 2024-09-13T14:50:01.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.107 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:50:01.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:50:01.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.165409+0000", 2024-09-13T14:50:01.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:50:01.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.165409+0000", 2024-09-13T14:50:01.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:50:01.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.165410+0000", 2024-09-13T14:50:01.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.111e-06 2024-09-13T14:50:01.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:50:01.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.165416+0000", 2024-09-13T14:50:01.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4750000000000001e-06 2024-09-13T14:50:01.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:50:01.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.165426+0000", 2024-09-13T14:50:01.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8609999999999993e-06 2024-09-13T14:50:01.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:50:01.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.165431+0000", 2024-09-13T14:50:01.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0150000000000003e-06 2024-09-13T14:50:01.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:50:01.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.167046+0000", 2024-09-13T14:50:01.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001615338 2024-09-13T14:50:01.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:50:01.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.167167+0000", 2024-09-13T14:50:01.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000121043 2024-09-13T14:50:01.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,3,7", 2024-09-13T14:50:01.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.167325+0000", 2024-09-13T14:50:01.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015795499999999999 2024-09-13T14:50:01.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.170039+0000", 2024-09-13T14:50:01.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027137789999999999 2024-09-13T14:50:01.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:50:01.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.170076+0000", 2024-09-13T14:50:01.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.773e-05 2024-09-13T14:50:01.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.170534+0000", 2024-09-13T14:50:01.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045798399999999998 2024-09-13T14:50:01.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.568726+0000", 2024-09-13T14:50:01.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.39819145299999997 2024-09-13T14:50:01.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:50:01.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.568783+0000", 2024-09-13T14:50:01.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6852999999999999e-05 2024-09-13T14:50:01.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:50:01.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:47:40.568895+0000", 2024-09-13T14:50:01.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011237400000000001 2024-09-13T14:50:01.113 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.113 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:50:01.113 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.113 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:214274 7.2 7:475f1bae:::benchmark_data_smithi151_190906_object93813:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1162)", 2024-09-13T14:50:01.113 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:48:06.231536+0000", 2024-09-13T14:50:01.113 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 114.783252924, 2024-09-13T14:50:01.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27906902700000003, 2024-09-13T14:50:01.114 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:50:01.114 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:50:01.114 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:01.114 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:50:01.114 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:50:01.114 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:50:01.114 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 214274 2024-09-13T14:50:01.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.114 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:50:01.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:50:01.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.231536+0000", 2024-09-13T14:50:01.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:50:01.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.231536+0000", 2024-09-13T14:50:01.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:50:01.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.231538+0000", 2024-09-13T14:50:01.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6929999999999999e-06 2024-09-13T14:50:01.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:50:01.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.231544+0000", 2024-09-13T14:50:01.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9490000000000001e-06 2024-09-13T14:50:01.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:50:01.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.231563+0000", 2024-09-13T14:50:01.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9009999999999999e-05 2024-09-13T14:50:01.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:50:01.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.231571+0000", 2024-09-13T14:50:01.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2120000000000003e-06 2024-09-13T14:50:01.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:50:01.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.231734+0000", 2024-09-13T14:50:01.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016258000000000001 2024-09-13T14:50:01.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:50:01.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.231861+0000", 2024-09-13T14:50:01.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012766599999999999 2024-09-13T14:50:01.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,3,7", 2024-09-13T14:50:01.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.231986+0000", 2024-09-13T14:50:01.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012454799999999999 2024-09-13T14:50:01.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:50:01.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.253252+0000", 2024-09-13T14:50:01.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021265724999999999 2024-09-13T14:50:01.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.253283+0000", 2024-09-13T14:50:01.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1455999999999998e-05 2024-09-13T14:50:01.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.257943+0000", 2024-09-13T14:50:01.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0046600139999999996 2024-09-13T14:50:01.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.510426+0000", 2024-09-13T14:50:01.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.252483083 2024-09-13T14:50:01.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:50:01.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.510494+0000", 2024-09-13T14:50:01.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7479999999999998e-05 2024-09-13T14:50:01.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:50:01.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.510605+0000", 2024-09-13T14:50:01.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000111611 2024-09-13T14:50:01.120 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.120 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:50:01.120 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.120 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:214278 7.2 7:44cc2d6f:::benchmark_data_smithi151_190906_object93817:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1162)", 2024-09-13T14:50:01.120 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:48:06.231600+0000", 2024-09-13T14:50:01.121 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 114.783188928, 2024-09-13T14:50:01.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.279202332, 2024-09-13T14:50:01.121 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:50:01.121 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:50:01.121 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:01.121 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:50:01.121 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:50:01.121 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:50:01.121 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 214278 2024-09-13T14:50:01.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.121 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:50:01.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:50:01.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.231600+0000", 2024-09-13T14:50:01.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:50:01.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.231600+0000", 2024-09-13T14:50:01.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:50:01.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.231601+0000", 2024-09-13T14:50:01.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6499999999999999e-07 2024-09-13T14:50:01.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:50:01.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.231603+0000", 2024-09-13T14:50:01.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3420000000000002e-06 2024-09-13T14:50:01.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:50:01.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.231610+0000", 2024-09-13T14:50:01.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1729999999999998e-06 2024-09-13T14:50:01.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:50:01.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.231615+0000", 2024-09-13T14:50:01.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9139999999999999e-06 2024-09-13T14:50:01.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:50:01.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.249742+0000", 2024-09-13T14:50:01.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018126458000000002 2024-09-13T14:50:01.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:50:01.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.249865+0000", 2024-09-13T14:50:01.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012348200000000001 2024-09-13T14:50:01.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,3,7", 2024-09-13T14:50:01.125 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.250002+0000", 2024-09-13T14:50:01.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000136475 2024-09-13T14:50:01.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:50:01.125 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.257957+0000", 2024-09-13T14:50:01.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0079550630000000001 2024-09-13T14:50:01.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.125 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.257988+0000", 2024-09-13T14:50:01.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1715999999999999e-05 2024-09-13T14:50:01.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.263789+0000", 2024-09-13T14:50:01.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0058007170000000004 2024-09-13T14:50:01.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.510648+0000", 2024-09-13T14:50:01.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24685863 2024-09-13T14:50:01.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:50:01.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.510694+0000", 2024-09-13T14:50:01.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5901e-05 2024-09-13T14:50:01.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:50:01.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.510803+0000", 2024-09-13T14:50:01.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108996 2024-09-13T14:50:01.127 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.127 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:50:01.127 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.127 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:214290 7.2 7:4d8c6bfe:::benchmark_data_smithi151_190906_object93829:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1162)", 2024-09-13T14:50:01.128 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:48:06.251532+0000", 2024-09-13T14:50:01.128 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 114.763257146, 2024-09-13T14:50:01.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27791528500000001, 2024-09-13T14:50:01.128 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:50:01.128 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:50:01.128 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:01.128 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:50:01.128 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:50:01.128 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:50:01.128 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 214290 2024-09-13T14:50:01.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.128 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:50:01.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:50:01.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.251532+0000", 2024-09-13T14:50:01.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:50:01.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.251532+0000", 2024-09-13T14:50:01.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:50:01.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.251533+0000", 2024-09-13T14:50:01.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.333e-06 2024-09-13T14:50:01.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:50:01.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.251538+0000", 2024-09-13T14:50:01.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5650000000000003e-06 2024-09-13T14:50:01.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:50:01.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.251550+0000", 2024-09-13T14:50:01.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1620000000000001e-05 2024-09-13T14:50:01.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:50:01.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.251557+0000", 2024-09-13T14:50:01.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0820000000000001e-06 2024-09-13T14:50:01.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:50:01.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.253310+0000", 2024-09-13T14:50:01.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001753405 2024-09-13T14:50:01.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:50:01.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.253428+0000", 2024-09-13T14:50:01.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117848 2024-09-13T14:50:01.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,3,7", 2024-09-13T14:50:01.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.253553+0000", 2024-09-13T14:50:01.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012459400000000001 2024-09-13T14:50:01.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:50:01.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.258000+0000", 2024-09-13T14:50:01.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0044472130000000002 2024-09-13T14:50:01.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.258048+0000", 2024-09-13T14:50:01.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8461999999999998e-05 2024-09-13T14:50:01.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.263832+0000", 2024-09-13T14:50:01.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0057836700000000003 2024-09-13T14:50:01.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.529210+0000", 2024-09-13T14:50:01.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.265377802 2024-09-13T14:50:01.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:50:01.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.529318+0000", 2024-09-13T14:50:01.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108585 2024-09-13T14:50:01.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:50:01.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.529447+0000", 2024-09-13T14:50:01.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012910600000000001 2024-09-13T14:50:01.134 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.134 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:50:01.134 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.134 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:214291 7.a 7:549c24c5:::benchmark_data_smithi151_190906_object93830:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1162)", 2024-09-13T14:50:01.134 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:48:06.251753+0000", 2024-09-13T14:50:01.134 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 114.763036598, 2024-09-13T14:50:01.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.30186339600000001, 2024-09-13T14:50:01.135 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:50:01.135 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:50:01.135 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:01.135 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:50:01.135 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:50:01.135 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:50:01.135 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 214291 2024-09-13T14:50:01.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.135 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:50:01.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:50:01.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.251753+0000", 2024-09-13T14:50:01.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:50:01.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.251753+0000", 2024-09-13T14:50:01.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:50:01.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.251753+0000", 2024-09-13T14:50:01.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0200000000000001e-07 2024-09-13T14:50:01.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:50:01.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.251757+0000", 2024-09-13T14:50:01.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8020000000000002e-06 2024-09-13T14:50:01.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:50:01.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.251768+0000", 2024-09-13T14:50:01.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0616e-05 2024-09-13T14:50:01.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:50:01.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.251774+0000", 2024-09-13T14:50:01.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5509999999999996e-06 2024-09-13T14:50:01.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:50:01.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.264114+0000", 2024-09-13T14:50:01.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012339558 2024-09-13T14:50:01.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:50:01.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.264244+0000", 2024-09-13T14:50:01.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013038500000000001 2024-09-13T14:50:01.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,3,5", 2024-09-13T14:50:01.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.264426+0000", 2024-09-13T14:50:01.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018203800000000001 2024-09-13T14:50:01.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.270759+0000", 2024-09-13T14:50:01.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0063323930000000004 2024-09-13T14:50:01.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:50:01.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.270817+0000", 2024-09-13T14:50:01.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8304000000000003e-05 2024-09-13T14:50:01.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.271092+0000", 2024-09-13T14:50:01.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027502400000000001 2024-09-13T14:50:01.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.550253+0000", 2024-09-13T14:50:01.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.279161255 2024-09-13T14:50:01.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:50:01.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.550306+0000", 2024-09-13T14:50:01.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2463000000000002e-05 2024-09-13T14:50:01.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:50:01.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.553616+0000", 2024-09-13T14:50:01.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033103049999999999 2024-09-13T14:50:01.141 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.141 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:50:01.141 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.141 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:214297 7.2 7:4e55f0c1:::benchmark_data_smithi151_190906_object93836:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1162)", 2024-09-13T14:50:01.141 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:48:06.256475+0000", 2024-09-13T14:50:01.141 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 114.758314328, 2024-09-13T14:50:01.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27326157499999998, 2024-09-13T14:50:01.142 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:50:01.142 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:50:01.142 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:01.142 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:50:01.142 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:50:01.142 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:50:01.142 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 214297 2024-09-13T14:50:01.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.142 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:50:01.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:50:01.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.256475+0000", 2024-09-13T14:50:01.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:50:01.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.256475+0000", 2024-09-13T14:50:01.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:50:01.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.256476+0000", 2024-09-13T14:50:01.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6300000000000004e-07 2024-09-13T14:50:01.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:50:01.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.256480+0000", 2024-09-13T14:50:01.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2509999999999996e-06 2024-09-13T14:50:01.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:50:01.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.256489+0000", 2024-09-13T14:50:01.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3549999999999994e-06 2024-09-13T14:50:01.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:50:01.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.256496+0000", 2024-09-13T14:50:01.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2199999999999997e-06 2024-09-13T14:50:01.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:50:01.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.263859+0000", 2024-09-13T14:50:01.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0073635239999999998 2024-09-13T14:50:01.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:50:01.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.263974+0000", 2024-09-13T14:50:01.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011489699999999999 2024-09-13T14:50:01.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,3,7", 2024-09-13T14:50:01.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.264099+0000", 2024-09-13T14:50:01.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000124459 2024-09-13T14:50:01.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.267340+0000", 2024-09-13T14:50:01.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0032410189999999999 2024-09-13T14:50:01.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:50:01.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.267660+0000", 2024-09-13T14:50:01.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000320701 2024-09-13T14:50:01.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.270804+0000", 2024-09-13T14:50:01.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031441899999999998 2024-09-13T14:50:01.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.529534+0000", 2024-09-13T14:50:01.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25872956800000002 2024-09-13T14:50:01.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:50:01.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.529625+0000", 2024-09-13T14:50:01.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1323999999999995e-05 2024-09-13T14:50:01.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:50:01.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.529737+0000", 2024-09-13T14:50:01.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000111204 2024-09-13T14:50:01.149 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.149 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:50:01.149 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.149 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:214303 7.a 7:5abfeea1:::benchmark_data_smithi151_190906_object93842:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1162)", 2024-09-13T14:50:01.149 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:48:06.267330+0000", 2024-09-13T14:50:01.150 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 114.747459631, 2024-09-13T14:50:01.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31414630300000002, 2024-09-13T14:50:01.150 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:50:01.150 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:50:01.150 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:01.150 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:50:01.150 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:50:01.150 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:50:01.150 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 214303 2024-09-13T14:50:01.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.150 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:50:01.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:50:01.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.267330+0000", 2024-09-13T14:50:01.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:50:01.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.267330+0000", 2024-09-13T14:50:01.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:50:01.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.267331+0000", 2024-09-13T14:50:01.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.26e-06 2024-09-13T14:50:01.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:50:01.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.267336+0000", 2024-09-13T14:50:01.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3480000000000003e-06 2024-09-13T14:50:01.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:50:01.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.267347+0000", 2024-09-13T14:50:01.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0436e-05 2024-09-13T14:50:01.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:50:01.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.267354+0000", 2024-09-13T14:50:01.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6720000000000004e-06 2024-09-13T14:50:01.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:50:01.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.267673+0000", 2024-09-13T14:50:01.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00031868499999999998 2024-09-13T14:50:01.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:50:01.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.267834+0000", 2024-09-13T14:50:01.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016131899999999999 2024-09-13T14:50:01.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,3,5", 2024-09-13T14:50:01.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.267990+0000", 2024-09-13T14:50:01.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000156059 2024-09-13T14:50:01.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.271042+0000", 2024-09-13T14:50:01.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030517589999999998 2024-09-13T14:50:01.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.271132+0000", 2024-09-13T14:50:01.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9551999999999999e-05 2024-09-13T14:50:01.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:50:01.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.271174+0000", 2024-09-13T14:50:01.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2293000000000003e-05 2024-09-13T14:50:01.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.577932+0000", 2024-09-13T14:50:01.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.30675774700000003 2024-09-13T14:50:01.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:50:01.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.577983+0000", 2024-09-13T14:50:01.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0911000000000001e-05 2024-09-13T14:50:01.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:50:01.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:06.581476+0000", 2024-09-13T14:50:01.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003493262 2024-09-13T14:50:01.156 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.157 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:50:01.157 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.157 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:218108 7.a 7:5aafc383:::benchmark_data_smithi151_190906_object97647:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1162)", 2024-09-13T14:50:01.157 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:48:13.120760+0000", 2024-09-13T14:50:01.157 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 107.894029514, 2024-09-13T14:50:01.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26434223600000001, 2024-09-13T14:50:01.157 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:50:01.157 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:50:01.157 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:01.157 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:50:01.158 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:50:01.158 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:50:01.158 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 218108 2024-09-13T14:50:01.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.158 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:50:01.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:50:01.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:13.120760+0000", 2024-09-13T14:50:01.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:50:01.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:13.120760+0000", 2024-09-13T14:50:01.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:50:01.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:13.120761+0000", 2024-09-13T14:50:01.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.13e-06 2024-09-13T14:50:01.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:50:01.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:13.120764+0000", 2024-09-13T14:50:01.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9409999999999999e-06 2024-09-13T14:50:01.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:50:01.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:13.120772+0000", 2024-09-13T14:50:01.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3399999999999998e-06 2024-09-13T14:50:01.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:50:01.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:13.120777+0000", 2024-09-13T14:50:01.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2079999999999999e-06 2024-09-13T14:50:01.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:50:01.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:13.121886+0000", 2024-09-13T14:50:01.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011085050000000001 2024-09-13T14:50:01.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:50:01.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:13.122022+0000", 2024-09-13T14:50:01.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000136286 2024-09-13T14:50:01.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,3,5", 2024-09-13T14:50:01.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:13.122162+0000", 2024-09-13T14:50:01.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000140295 2024-09-13T14:50:01.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:13.128519+0000", 2024-09-13T14:50:01.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0063563200000000004 2024-09-13T14:50:01.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:50:01.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:13.132834+0000", 2024-09-13T14:50:01.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0043156219999999999 2024-09-13T14:50:01.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:13.137539+0000", 2024-09-13T14:50:01.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0047046459999999998 2024-09-13T14:50:01.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:13.381486+0000", 2024-09-13T14:50:01.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.243946737 2024-09-13T14:50:01.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:50:01.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:13.381610+0000", 2024-09-13T14:50:01.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012437200000000001 2024-09-13T14:50:01.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:50:01.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:13.385102+0000", 2024-09-13T14:50:01.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034918340000000001 2024-09-13T14:50:01.164 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.164 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:50:01.164 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.164 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.45494.0:218112 7.a 7:55084175:::benchmark_data_smithi151_190906_object97651:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1162)", 2024-09-13T14:50:01.164 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:48:13.134169+0000", 2024-09-13T14:50:01.164 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 107.880620226, 2024-09-13T14:50:01.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25903401999999998, 2024-09-13T14:50:01.164 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:50:01.164 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:50:01.165 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:01.165 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:50:01.165 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.45494", 2024-09-13T14:50:01.165 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:50:01.165 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 218112 2024-09-13T14:50:01.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.165 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:50:01.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:50:01.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:13.134169+0000", 2024-09-13T14:50:01.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:50:01.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:13.134169+0000", 2024-09-13T14:50:01.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:50:01.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:50:01.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:13.134170+0000", 2024-09-13T14:50:01.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.269e-06 2024-09-13T14:50:01.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:50:01.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:13.134173+0000", 2024-09-13T14:50:01.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.813e-06 2024-09-13T14:50:01.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:50:01.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:13.134181+0000", 2024-09-13T14:50:01.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1960000000000008e-06 2024-09-13T14:50:01.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:50:01.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:13.134189+0000", 2024-09-13T14:50:01.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0879999999999995e-06 2024-09-13T14:50:01.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:50:01.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:13.137573+0000", 2024-09-13T14:50:01.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033837659999999999 2024-09-13T14:50:01.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:50:01.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:13.137741+0000", 2024-09-13T14:50:01.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016826800000000001 2024-09-13T14:50:01.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,3,5", 2024-09-13T14:50:01.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:13.137930+0000", 2024-09-13T14:50:01.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000188791 2024-09-13T14:50:01.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:13.145315+0000", 2024-09-13T14:50:01.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00738515 2024-09-13T14:50:01.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:13.145618+0000", 2024-09-13T14:50:01.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000302133 2024-09-13T14:50:01.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:50:01.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:13.161954+0000", 2024-09-13T14:50:01.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016336849000000001 2024-09-13T14:50:01.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:01.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:13.389992+0000", 2024-09-13T14:50:01.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22803728600000001 2024-09-13T14:50:01.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:50:01.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:13.390043+0000", 2024-09-13T14:50:01.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1539000000000002e-05 2024-09-13T14:50:01.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:50:01.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:50:01.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:50:01.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:48:13.393203+0000", 2024-09-13T14:50:01.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031598720000000002 2024-09-13T14:50:01.171 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.171 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:50:01.171 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.171 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:50:01.171 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:50:01.172 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:50:01.188 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:01.218 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 22 16 48276 48260 137.085 106.188 0.00692124 0.00728877 2024-09-13T14:50:01.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:01.228 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:01.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:01.329 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:01.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:01 smithi151 ceph-mon[104450]: pgmap v1695: 119 pgs: 119 active+clean; 2.5 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 120 MiB/s wr, 1.91k op/s 2024-09-13T14:50:01.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:01 smithi151 ceph-mon[108966]: pgmap v1695: 119 pgs: 119 active+clean; 2.5 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 120 MiB/s wr, 1.91k op/s 2024-09-13T14:50:01.364 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:01.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:01.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:01.465 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:01.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:01.498 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:01.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:01.599 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:01.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:01 smithi184 ceph-mon[92527]: pgmap v1695: 119 pgs: 119 active+clean; 2.5 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 120 MiB/s wr, 1.91k op/s 2024-09-13T14:50:01.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:01.641 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:01.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:01.742 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:01.777 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:01.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:01.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:01.878 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:01.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:01.910 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:01.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:02.011 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:02.041 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:02.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:02.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:02.143 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:02.173 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:50:02.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:02.186 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:02.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:02.218 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 23 16 49934 49918 135.63 103.625 0.00785263 0.00736679 2024-09-13T14:50:02.287 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:02.326 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:02.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:02.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:02.427 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:02.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:02.466 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:02.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:02.567 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:02.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:02.595 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:02.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:02.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:50:02.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:02.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:02 smithi151 ceph-mon[104450]: pgmap v1696: 119 pgs: 119 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-09-13T14:50:02.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:50:02.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:02.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:02 smithi151 ceph-mon[108966]: pgmap v1696: 119 pgs: 119 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-09-13T14:50:02.696 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:02.741 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:02.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:02.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:02.843 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:02.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:50:02.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:02.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:02 smithi184 ceph-mon[92527]: pgmap v1696: 119 pgs: 119 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-09-13T14:50:02.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:02.876 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:02.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:02.977 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:03.017 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:03.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:03.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:03.119 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:03.155 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:03.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:03.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:03.218 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 24 16 51485 51469 134.018 96.9375 0.00446955 0.00745634 2024-09-13T14:50:03.257 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:03.292 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:03.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:03.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:03.347 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:03.394 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:03.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:03.449 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:03.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:03.550 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:03.579 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:03.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:03.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:03.681 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:03.709 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:03.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:03.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:03.811 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:03.836 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:03.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:03.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:03.938 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:03.959 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:03.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:03.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:04.061 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:04.104 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:04.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:04.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:04.205 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:04.220 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 25 15 52819 52804 131.994 83.4375 0.00636643 0.00757089 2024-09-13T14:50:04.242 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:04.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:04.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:04.344 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:04.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:04.383 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:04.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:04.484 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:04.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:04.506 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:04.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:04.607 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:04.649 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:04.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:04.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:04.751 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:04.787 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:04.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:04.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:04.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:04 smithi151 ceph-mon[104450]: pgmap v1697: 119 pgs: 119 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 99 MiB/s wr, 1.59k op/s 2024-09-13T14:50:04.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:04 smithi151 ceph-mon[108966]: pgmap v1697: 119 pgs: 119 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 99 MiB/s wr, 1.59k op/s 2024-09-13T14:50:04.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:50:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:50:04] "GET /metrics HTTP/1.1" 200 36639 "" "Prometheus/2.51.0" 2024-09-13T14:50:04.889 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:04.922 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:04.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:04.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:05.024 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:05.060 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:05.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:05.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:05.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:04 smithi184 ceph-mon[92527]: pgmap v1697: 119 pgs: 119 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 99 MiB/s wr, 1.59k op/s 2024-09-13T14:50:05.162 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:05.198 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:05.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:05.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:05.219 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 26 16 54361 54345 130.622 96.3125 0.0103693 0.00765129 2024-09-13T14:50:05.301 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:05.332 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:05.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:05.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:05.434 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:05.455 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:05.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:05.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:05.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:05.591 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:05.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:05.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:05.693 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:05.729 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:05.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:05.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:05.830 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:05.862 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:05.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:05.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:05.964 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:06.013 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:06.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:06.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:06.114 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:06.147 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:06.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:06.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:06.219 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 27 16 55866 55850 129.267 94.0625 0.0101869 0.00773093 2024-09-13T14:50:06.249 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:06.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:06.283 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:06.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:06.384 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:06.414 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:06.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:06.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:06.516 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:06.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:06.551 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:06.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:06.653 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:06.681 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:06.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:06.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:06.784 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:06.816 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:06.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:06.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:06.919 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:06.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:06.956 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:06.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:07.057 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:07.085 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:07.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:07.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:07.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:06 smithi184 ceph-mon[92527]: pgmap v1698: 119 pgs: 119 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-09-13T14:50:07.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:06 smithi151 ceph-mon[104450]: pgmap v1698: 119 pgs: 119 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-09-13T14:50:07.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:06 smithi151 ceph-mon[108966]: pgmap v1698: 119 pgs: 119 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-09-13T14:50:07.187 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:07.219 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 28 16 57308 57292 127.866 90.125 0.00367676 0.0078143 2024-09-13T14:50:07.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:07.222 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:07.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:07.323 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:07.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:07.355 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:07.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:07.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:07.492 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:07.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:07.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:07.594 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:07.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:07.634 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:07.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:07.735 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:07.767 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:07.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:07.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:07.869 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:07.884 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T14:50:07.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:07.918 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:07.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:08.019 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:08.046 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:08.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:08.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:08.148 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:08.178 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:08.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:08.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:08.220 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 29 16 58653 58637 126.354 84.0625 0.0185462 0.00790836 2024-09-13T14:50:08.281 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:08.313 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:08.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:08.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:08.415 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:08.453 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:08.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:08.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:08.555 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:08.587 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:08.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:08.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:08.689 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:08.721 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:08.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:08.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:08.754 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:08 smithi151 ceph-mon[104450]: pgmap v1699: 119 pgs: 119 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-13T14:50:08.754 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:08 smithi151 ceph-mon[108966]: pgmap v1699: 119 pgs: 119 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-13T14:50:08.824 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:08.874 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:08.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:08.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:08.978 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:09.011 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:09.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:09.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:09.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:08 smithi184 ceph-mon[92527]: pgmap v1699: 119 pgs: 119 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-13T14:50:09.113 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:09.147 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:09.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:09.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:09.220 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 30 16 60113 60097 125.184 91.25 0.0176322 0.00798315 2024-09-13T14:50:09.249 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:09.285 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:09.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:09.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:09.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:09.413 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:09.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:09.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:09.516 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:09.544 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:09.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:09.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:09.648 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:09.678 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:09.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:09.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:09.780 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:09.801 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:09.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:09.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:09.903 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:09.924 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:09.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:09.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:10.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:10.067 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:10.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:10.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:10.169 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:10.204 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:10.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:10.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:10.220 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 31 16 61250 61234 123.438 71.0625 0.00586986 0.00809517 2024-09-13T14:50:10.306 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:10.343 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:10.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:10.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:10.446 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:10.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:10.483 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:10.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:10.584 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:10.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:50:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:50:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:50:10.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:10.624 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:10.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:10.653 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:10.657 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:50:10.657 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:50:10.694 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:50:10.694 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:50:10.725 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:10.728 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:50:10.729 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:50:10.758 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:10.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:10.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:10.764 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:50:10.764 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:50:10.794 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:50:10.795 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:50:10.829 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:50:10.829 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:50:10.858 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:50:10.859 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:50:10.860 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:10.889 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:10.890 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:50:10.890 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:50:10.891 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:10 smithi151 ceph-mon[104450]: pgmap v1700: 119 pgs: 119 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 105 MiB/s wr, 1.69k op/s 2024-09-13T14:50:10.891 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:10 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/299395373' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:10.891 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:10 smithi151 ceph-mon[108966]: pgmap v1700: 119 pgs: 119 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 105 MiB/s wr, 1.69k op/s 2024-09-13T14:50:10.891 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:10 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/299395373' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:10.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:10.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:10.993 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:11.041 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:11.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:11.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:11.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:10 smithi184 ceph-mon[92527]: pgmap v1700: 119 pgs: 119 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 105 MiB/s wr, 1.69k op/s 2024-09-13T14:50:11.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:10 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/299395373' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:11.143 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:11.173 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:11.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:11.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:11.220 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 32 16 62532 62516 122.084 80.125 0.0176748 0.00818536 2024-09-13T14:50:11.274 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:11.313 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:11.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:11.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:11.416 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:11.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:11.453 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:11.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:11.554 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:11.579 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:11.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:11.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:11.681 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:11.709 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:11.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:11.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:11.811 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:11.839 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:11.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:11.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:11.940 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:11.981 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:11.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:11.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:12.084 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:12.104 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:12.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:12.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:12.206 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:12.221 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 33 16 63642 63626 120.485 69.375 0.0129186 0.00829479 2024-09-13T14:50:12.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:12.243 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:12.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:12.344 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:12.374 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:12.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:12.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:12.476 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:12.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:12.511 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:12.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:12.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:12.647 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:12.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:12.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:12.749 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:12.777 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:12.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:12.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:12.879 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:12.913 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:12.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:12.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:13.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:13.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:13.051 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:13.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:13.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:12 smithi184 ceph-mon[92527]: pgmap v1701: 119 pgs: 119 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-09-13T14:50:13.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:12 smithi184 ceph-mon[92527]: from='client.55237 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:50:13.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:12 smithi151 ceph-mon[104450]: pgmap v1701: 119 pgs: 119 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-09-13T14:50:13.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:12 smithi151 ceph-mon[104450]: from='client.55237 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:50:13.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:12 smithi151 ceph-mon[108966]: pgmap v1701: 119 pgs: 119 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-09-13T14:50:13.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:12 smithi151 ceph-mon[108966]: from='client.55237 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:50:13.153 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:13.193 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:13.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:13.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:13.221 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 34 16 65131 65115 119.678 93.0625 0.00595545 0.00835031 2024-09-13T14:50:13.295 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:13.343 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:13.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:13.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:13.445 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:13.473 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:13.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:13.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:13.575 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:13.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:13.605 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:13.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:13.706 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:13.745 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:13.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:13.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:13.850 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:13.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:13.903 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:13.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:14.004 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:14.034 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:14.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:14.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:14.136 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:14.167 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:14.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:14.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:14.221 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 35 16 66533 66517 118.762 87.625 0.00551554 0.00841523 2024-09-13T14:50:14.249 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:50:14.253 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T14:50:14.269 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:14.298 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:14.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:14.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:14.400 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:14.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:14.429 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:14.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:14.530 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:14.561 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:14.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:14.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:14.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:14.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:14.701 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:14.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:14.714 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:14.802 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:14.813 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:14 smithi151 ceph-mon[108966]: pgmap v1702: 119 pgs: 119 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-13T14:50:14.814 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:14 smithi151 ceph-mon[104450]: pgmap v1702: 119 pgs: 119 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-13T14:50:14.814 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:50:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:50:14] "GET /metrics HTTP/1.1" 200 36639 "" "Prometheus/2.51.0" 2024-09-13T14:50:14.830 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:14.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:14.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:14.932 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:14.962 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:14.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:14.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:15.064 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:15.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:15.097 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:15.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:14 smithi184 ceph-mon[92527]: pgmap v1702: 119 pgs: 119 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-13T14:50:15.182 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1701,"stamp":"2024-09-13T14:50:11.551478+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3873642445,"num_objects":59124,"num_object_clones":0,"num_object_copies":177372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59161,"num_write_kb":3782303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":327718,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62256,"ondisk_log_size":62256,"up":357,"acting":357,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":358,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":14509720,"kb_used_data":11519128,"kb_used_omap":149,"kb_used_meta":2990378,"kb_avail":735451496,"statfs":{"total":767960285184,"available":753102331904,"internally_reserved":0,"allocated":11795587072,"data_stored":11766730199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":153401,"internal_metadata":3062147271},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,6,6,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1184890880,"num_objects":18080,"num_object_clones":0,"num_object_copies":54240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18080,"num_write_kb":1157120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004192"},"pg_stats":[{"pgid":"1.1d","version":"261'100","reported_seq":115,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.137520+0000","last_change":"2024-09-13T14:45:19.807356+0000","last_active":"2024-09-13T14:49:53.137520+0000","last_peered":"2024-09-13T14:49:53.137520+0000","last_clean":"2024-09-13T14:49:53.137520+0000","last_became_active":"2024-09-13T14:45:19.807081+0000","last_became_peered":"2024-09-13T14:45:19.807081+0000","last_unstale":"2024-09-13T14:49:53.137520+0000","last_undegraded":"2024-09-13T14:49:53.137520+0000","last_fullsized":"2024-09-13T14:49:53.137520+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:17.988345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:17.988345+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:41:05.216335+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":1721,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.130822+0000","last_change":"2024-09-13T14:29:42.541263+0000","last_active":"2024-09-13T14:49:53.130822+0000","last_peered":"2024-09-13T14:49:53.130822+0000","last_clean":"2024-09-13T14:49:53.130822+0000","last_became_active":"2024-09-13T14:29:42.540873+0000","last_became_peered":"2024-09-13T14:29:42.540873+0000","last_unstale":"2024-09-13T14:49:53.130822+0000","last_undegraded":"2024-09-13T14:49:53.130822+0000","last_fullsized":"2024-09-13T14:49:53.130822+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:20.611206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1132,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.787276+0000","last_change":"2024-09-13T14:43:07.516715+0000","last_active":"2024-09-13T14:49:35.787276+0000","last_peered":"2024-09-13T14:49:35.787276+0000","last_clean":"2024-09-13T14:49:35.787276+0000","last_became_active":"2024-09-13T14:43:07.516403+0000","last_became_peered":"2024-09-13T14:43:07.516403+0000","last_unstale":"2024-09-13T14:49:35.787276+0000","last_undegraded":"2024-09-13T14:49:35.787276+0000","last_fullsized":"2024-09-13T14:49:35.787276+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:56.469690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":60,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.129959+0000","last_change":"2024-09-13T14:45:19.810203+0000","last_active":"2024-09-13T14:49:53.129959+0000","last_peered":"2024-09-13T14:49:53.129959+0000","last_clean":"2024-09-13T14:49:53.129959+0000","last_became_active":"2024-09-13T14:45:19.809693+0000","last_became_peered":"2024-09-13T14:45:19.809693+0000","last_unstale":"2024-09-13T14:49:53.129959+0000","last_undegraded":"2024-09-13T14:49:53.129959+0000","last_fullsized":"2024-09-13T14:49:53.129959+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:20.219673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":59,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.130212+0000","last_change":"2024-09-13T14:45:19.807991+0000","last_active":"2024-09-13T14:49:53.130212+0000","last_peered":"2024-09-13T14:49:53.130212+0000","last_clean":"2024-09-13T14:49:53.130212+0000","last_became_active":"2024-09-13T14:45:19.807662+0000","last_became_peered":"2024-09-13T14:45:19.807662+0000","last_unstale":"2024-09-13T14:49:53.130212+0000","last_undegraded":"2024-09-13T14:49:53.130212+0000","last_fullsized":"2024-09-13T14:49:53.130212+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:29.306108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"261'100","reported_seq":1243,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.603648+0000","last_change":"2024-09-13T14:44:54.259562+0000","last_active":"2024-09-13T14:49:35.603648+0000","last_peered":"2024-09-13T14:49:35.603648+0000","last_clean":"2024-09-13T14:49:35.603648+0000","last_became_active":"2024-09-13T14:43:14.563717+0000","last_became_peered":"2024-09-13T14:43:14.563717+0000","last_unstale":"2024-09-13T14:49:35.603648+0000","last_undegraded":"2024-09-13T14:49:35.603648+0000","last_fullsized":"2024-09-13T14:49:35.603648+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_clean_scrub_stamp":"2024-09-13T14:44:54.259482+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:43:52.729109+0000","scrub_duration":0.015265569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":1100,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.211288+0000","last_change":"2024-09-13T14:19:25.873514+0000","last_active":"2024-09-13T14:49:36.211288+0000","last_peered":"2024-09-13T14:49:36.211288+0000","last_clean":"2024-09-13T14:49:36.211288+0000","last_became_active":"2024-09-13T14:19:25.873170+0000","last_became_peered":"2024-09-13T14:19:25.873170+0000","last_unstale":"2024-09-13T14:49:36.211288+0000","last_undegraded":"2024-09-13T14:49:36.211288+0000","last_fullsized":"2024-09-13T14:49:36.211288+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:48.298530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1129,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.787222+0000","last_change":"2024-09-13T14:30:15.580011+0000","last_active":"2024-09-13T14:49:35.787222+0000","last_peered":"2024-09-13T14:49:35.787222+0000","last_clean":"2024-09-13T14:49:35.787222+0000","last_became_active":"2024-09-13T14:30:15.579610+0000","last_became_peered":"2024-09-13T14:30:15.579610+0000","last_unstale":"2024-09-13T14:49:35.787222+0000","last_undegraded":"2024-09-13T14:49:35.787222+0000","last_fullsized":"2024-09-13T14:49:35.787222+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:11:02.733771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"261'100","reported_seq":77,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.130111+0000","last_change":"2024-09-13T14:45:35.947332+0000","last_active":"2024-09-13T14:49:53.130111+0000","last_peered":"2024-09-13T14:49:53.130111+0000","last_clean":"2024-09-13T14:49:53.130111+0000","last_became_active":"2024-09-13T14:45:19.810775+0000","last_became_peered":"2024-09-13T14:45:19.810775+0000","last_unstale":"2024-09-13T14:49:53.130111+0000","last_undegraded":"2024-09-13T14:49:53.130111+0000","last_fullsized":"2024-09-13T14:49:53.130111+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:35.947265+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:43:41.569795+0000","last_clean_scrub_stamp":"2024-09-13T14:45:35.947265+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:00.606370+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":1145,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.210995+0000","last_change":"2024-09-13T14:43:07.536116+0000","last_active":"2024-09-13T14:49:36.210995+0000","last_peered":"2024-09-13T14:49:36.210995+0000","last_clean":"2024-09-13T14:49:36.210995+0000","last_became_active":"2024-09-13T14:43:07.524112+0000","last_became_peered":"2024-09-13T14:43:07.524112+0000","last_unstale":"2024-09-13T14:49:36.210995+0000","last_undegraded":"2024-09-13T14:49:36.210995+0000","last_fullsized":"2024-09-13T14:49:36.210995+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:36.947060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1142,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.787801+0000","last_change":"2024-09-13T14:43:07.530236+0000","last_active":"2024-09-13T14:49:35.787801+0000","last_peered":"2024-09-13T14:49:35.787801+0000","last_clean":"2024-09-13T14:49:35.787801+0000","last_became_active":"2024-09-13T14:43:07.529967+0000","last_became_peered":"2024-09-13T14:43:07.529967+0000","last_unstale":"2024-09-13T14:49:35.787801+0000","last_undegraded":"2024-09-13T14:49:35.787801+0000","last_fullsized":"2024-09-13T14:49:35.787801+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:27.276550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":1705,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.129034+0000","last_change":"2024-09-13T14:28:35.612469+0000","last_active":"2024-09-13T14:49:53.129034+0000","last_peered":"2024-09-13T14:49:53.129034+0000","last_clean":"2024-09-13T14:49:53.129034+0000","last_became_active":"2024-09-13T14:28:35.612093+0000","last_became_peered":"2024-09-13T14:28:35.612093+0000","last_unstale":"2024-09-13T14:49:53.129034+0000","last_undegraded":"2024-09-13T14:49:53.129034+0000","last_fullsized":"2024-09-13T14:49:53.129034+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:51.022663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1728,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.135865+0000","last_change":"2024-09-13T14:45:19.808811+0000","last_active":"2024-09-13T14:49:53.135865+0000","last_peered":"2024-09-13T14:49:53.135865+0000","last_clean":"2024-09-13T14:49:53.135865+0000","last_became_active":"2024-09-13T14:45:19.808471+0000","last_became_peered":"2024-09-13T14:45:19.808471+0000","last_unstale":"2024-09-13T14:49:53.135865+0000","last_undegraded":"2024-09-13T14:49:53.135865+0000","last_fullsized":"2024-09-13T14:49:53.135865+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:53.638161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"261'100","reported_seq":59,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.130313+0000","last_change":"2024-09-13T14:45:19.813412+0000","last_active":"2024-09-13T14:49:53.130313+0000","last_peered":"2024-09-13T14:49:53.130313+0000","last_clean":"2024-09-13T14:49:53.130313+0000","last_became_active":"2024-09-13T14:45:19.812585+0000","last_became_peered":"2024-09-13T14:45:19.812585+0000","last_unstale":"2024-09-13T14:49:53.130313+0000","last_undegraded":"2024-09-13T14:49:53.130313+0000","last_fullsized":"2024-09-13T14:49:53.130313+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.614340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.614340+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:32:07.434109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":1151,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.787743+0000","last_change":"2024-09-13T14:43:07.522950+0000","last_active":"2024-09-13T14:49:35.787743+0000","last_peered":"2024-09-13T14:49:35.787743+0000","last_clean":"2024-09-13T14:49:35.787743+0000","last_became_active":"2024-09-13T14:43:07.522566+0000","last_became_peered":"2024-09-13T14:43:07.522566+0000","last_unstale":"2024-09-13T14:49:35.787743+0000","last_undegraded":"2024-09-13T14:49:35.787743+0000","last_fullsized":"2024-09-13T14:49:35.787743+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:03.201478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":59,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.131003+0000","last_change":"2024-09-13T14:45:19.807231+0000","last_active":"2024-09-13T14:49:53.131003+0000","last_peered":"2024-09-13T14:49:53.131003+0000","last_clean":"2024-09-13T14:49:53.131003+0000","last_became_active":"2024-09-13T14:45:19.806936+0000","last_became_peered":"2024-09-13T14:45:19.806936+0000","last_unstale":"2024-09-13T14:49:53.131003+0000","last_undegraded":"2024-09-13T14:49:53.131003+0000","last_fullsized":"2024-09-13T14:49:53.131003+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:04:26.723153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.18","version":"261'100","reported_seq":121,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.210286+0000","last_change":"2024-09-13T14:44:41.685537+0000","last_active":"2024-09-13T14:49:36.210286+0000","last_peered":"2024-09-13T14:49:36.210286+0000","last_clean":"2024-09-13T14:49:36.210286+0000","last_became_active":"2024-09-13T14:43:14.558502+0000","last_became_peered":"2024-09-13T14:43:14.558502+0000","last_unstale":"2024-09-13T14:49:36.210286+0000","last_undegraded":"2024-09-13T14:49:36.210286+0000","last_fullsized":"2024-09-13T14:49:36.210286+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1134,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:41.685124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:41.685124+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:37:58.412497+0000","scrub_duration":0.0075205970000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":108,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.137814+0000","last_change":"2024-09-13T14:43:07.516845+0000","last_active":"2024-09-13T14:49:53.137814+0000","last_peered":"2024-09-13T14:49:53.137814+0000","last_clean":"2024-09-13T14:49:53.137814+0000","last_became_active":"2024-09-13T14:43:07.514974+0000","last_became_peered":"2024-09-13T14:43:07.514974+0000","last_unstale":"2024-09-13T14:49:53.137814+0000","last_undegraded":"2024-09-13T14:49:53.137814+0000","last_fullsized":"2024-09-13T14:49:53.137814+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:55.419483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1743,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.131693+0000","last_change":"2024-09-13T14:45:19.815341+0000","last_active":"2024-09-13T14:49:53.131693+0000","last_peered":"2024-09-13T14:49:53.131693+0000","last_clean":"2024-09-13T14:49:53.131693+0000","last_became_active":"2024-09-13T14:45:19.810262+0000","last_became_peered":"2024-09-13T14:45:19.810262+0000","last_unstale":"2024-09-13T14:49:53.131693+0000","last_undegraded":"2024-09-13T14:49:53.131693+0000","last_fullsized":"2024-09-13T14:49:53.131693+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:39.847653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1b","version":"261'100","reported_seq":63,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.211264+0000","last_change":"2024-09-13T14:44:52.803330+0000","last_active":"2024-09-13T14:49:36.211264+0000","last_peered":"2024-09-13T14:49:36.211264+0000","last_clean":"2024-09-13T14:49:36.211264+0000","last_became_active":"2024-09-13T14:43:14.556712+0000","last_became_peered":"2024-09-13T14:43:14.556712+0000","last_unstale":"2024-09-13T14:49:36.211264+0000","last_undegraded":"2024-09-13T14:49:36.211264+0000","last_fullsized":"2024-09-13T14:49:36.211264+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:52.803242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:52.803242+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:06.658301+0000","scrub_duration":0.0019769660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":59,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.130035+0000","last_change":"2024-09-13T14:45:19.806623+0000","last_active":"2024-09-13T14:49:53.130035+0000","last_peered":"2024-09-13T14:49:53.130035+0000","last_clean":"2024-09-13T14:49:53.130035+0000","last_became_active":"2024-09-13T14:45:19.806219+0000","last_became_peered":"2024-09-13T14:45:19.806219+0000","last_unstale":"2024-09-13T14:49:53.130035+0000","last_undegraded":"2024-09-13T14:49:53.130035+0000","last_fullsized":"2024-09-13T14:49:53.130035+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:34:41.601201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1125,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.211050+0000","last_change":"2024-09-13T14:43:32.637751+0000","last_active":"2024-09-13T14:49:36.211050+0000","last_peered":"2024-09-13T14:49:36.211050+0000","last_clean":"2024-09-13T14:49:36.211050+0000","last_became_active":"2024-09-13T14:43:32.636901+0000","last_became_peered":"2024-09-13T14:43:32.636901+0000","last_unstale":"2024-09-13T14:49:36.211050+0000","last_undegraded":"2024-09-13T14:49:36.211050+0000","last_fullsized":"2024-09-13T14:49:36.211050+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:21.942419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1a","version":"261'100","reported_seq":63,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.787555+0000","last_change":"2024-09-13T14:43:41.910041+0000","last_active":"2024-09-13T14:49:35.787555+0000","last_peered":"2024-09-13T14:49:35.787555+0000","last_clean":"2024-09-13T14:49:35.787555+0000","last_became_active":"2024-09-13T14:43:14.555083+0000","last_became_peered":"2024-09-13T14:43:14.555083+0000","last_unstale":"2024-09-13T14:49:35.787555+0000","last_undegraded":"2024-09-13T14:49:35.787555+0000","last_fullsized":"2024-09-13T14:49:35.787555+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:41.909988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:41.909988+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:42:19.954838+0000","scrub_duration":0.0012891619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":60,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.130169+0000","last_change":"2024-09-13T14:45:19.812071+0000","last_active":"2024-09-13T14:49:53.130169+0000","last_peered":"2024-09-13T14:49:53.130169+0000","last_clean":"2024-09-13T14:49:53.130169+0000","last_became_active":"2024-09-13T14:45:19.811336+0000","last_became_peered":"2024-09-13T14:45:19.811336+0000","last_unstale":"2024-09-13T14:49:53.130169+0000","last_undegraded":"2024-09-13T14:49:53.130169+0000","last_fullsized":"2024-09-13T14:49:53.130169+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:40:29.814608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1760,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.130721+0000","last_change":"2024-09-13T14:45:19.810478+0000","last_active":"2024-09-13T14:49:53.130721+0000","last_peered":"2024-09-13T14:49:53.130721+0000","last_clean":"2024-09-13T14:49:53.130721+0000","last_became_active":"2024-09-13T14:45:19.808278+0000","last_became_peered":"2024-09-13T14:45:19.808278+0000","last_unstale":"2024-09-13T14:49:53.130721+0000","last_undegraded":"2024-09-13T14:49:53.130721+0000","last_fullsized":"2024-09-13T14:49:53.130721+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:12.211089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"261'100","reported_seq":63,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.603814+0000","last_change":"2024-09-13T14:44:38.214961+0000","last_active":"2024-09-13T14:49:35.603814+0000","last_peered":"2024-09-13T14:49:35.603814+0000","last_clean":"2024-09-13T14:49:35.603814+0000","last_became_active":"2024-09-13T14:43:14.557883+0000","last_became_peered":"2024-09-13T14:43:14.557883+0000","last_unstale":"2024-09-13T14:49:35.603814+0000","last_undegraded":"2024-09-13T14:49:35.603814+0000","last_fullsized":"2024-09-13T14:49:35.603814+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:38.214865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:38.214865+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:49:36.013783+0000","scrub_duration":0.0016389639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1631,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.131115+0000","last_change":"2024-09-13T14:30:15.576921+0000","last_active":"2024-09-13T14:49:53.131115+0000","last_peered":"2024-09-13T14:49:53.131115+0000","last_clean":"2024-09-13T14:49:53.131115+0000","last_became_active":"2024-09-13T14:30:15.576607+0000","last_became_peered":"2024-09-13T14:30:15.576607+0000","last_unstale":"2024-09-13T14:49:53.131115+0000","last_undegraded":"2024-09-13T14:49:53.131115+0000","last_fullsized":"2024-09-13T14:49:53.131115+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:35.162460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1135,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.210687+0000","last_change":"2024-09-13T14:43:07.518801+0000","last_active":"2024-09-13T14:49:36.210687+0000","last_peered":"2024-09-13T14:49:36.210687+0000","last_clean":"2024-09-13T14:49:36.210687+0000","last_became_active":"2024-09-13T14:43:07.517610+0000","last_became_peered":"2024-09-13T14:43:07.517610+0000","last_unstale":"2024-09-13T14:49:36.210687+0000","last_undegraded":"2024-09-13T14:49:36.210687+0000","last_fullsized":"2024-09-13T14:49:36.210687+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:51.632254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"261'100","reported_seq":108,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.135695+0000","last_change":"2024-09-13T14:43:15.037565+0000","last_active":"2024-09-13T14:49:53.135695+0000","last_peered":"2024-09-13T14:49:53.135695+0000","last_clean":"2024-09-13T14:49:53.135695+0000","last_became_active":"2024-09-13T14:43:14.554183+0000","last_became_peered":"2024-09-13T14:43:14.554183+0000","last_unstale":"2024-09-13T14:49:53.135695+0000","last_undegraded":"2024-09-13T14:49:53.135695+0000","last_fullsized":"2024-09-13T14:49:53.135695+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.037517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.037517+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:01:35.137195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":1129,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.603595+0000","last_change":"2024-09-13T14:30:15.579260+0000","last_active":"2024-09-13T14:49:35.603595+0000","last_peered":"2024-09-13T14:49:35.603595+0000","last_clean":"2024-09-13T14:49:35.603595+0000","last_became_active":"2024-09-13T14:30:15.578897+0000","last_became_peered":"2024-09-13T14:30:15.578897+0000","last_unstale":"2024-09-13T14:49:35.603595+0000","last_undegraded":"2024-09-13T14:49:35.603595+0000","last_fullsized":"2024-09-13T14:49:35.603595+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:56.737849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":109,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.138713+0000","last_change":"2024-09-13T14:43:07.521576+0000","last_active":"2024-09-13T14:49:53.138713+0000","last_peered":"2024-09-13T14:49:53.138713+0000","last_clean":"2024-09-13T14:49:53.138713+0000","last_became_active":"2024-09-13T14:43:07.520478+0000","last_became_peered":"2024-09-13T14:43:07.520478+0000","last_unstale":"2024-09-13T14:49:53.138713+0000","last_undegraded":"2024-09-13T14:49:53.138713+0000","last_fullsized":"2024-09-13T14:49:53.138713+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:53.410135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"261'100","reported_seq":389,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.210814+0000","last_change":"2024-09-13T14:44:49.775941+0000","last_active":"2024-09-13T14:49:36.210814+0000","last_peered":"2024-09-13T14:49:36.210814+0000","last_clean":"2024-09-13T14:49:36.210814+0000","last_became_active":"2024-09-13T14:43:14.560297+0000","last_became_peered":"2024-09-13T14:43:14.560297+0000","last_unstale":"2024-09-13T14:49:36.210814+0000","last_undegraded":"2024-09-13T14:49:36.210814+0000","last_fullsized":"2024-09-13T14:49:36.210814+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:49.775844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:49.775844+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:15:53.138661+0000","scrub_duration":0.012350274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":1118,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.787862+0000","last_change":"2024-09-13T14:19:25.870605+0000","last_active":"2024-09-13T14:49:35.787862+0000","last_peered":"2024-09-13T14:49:35.787862+0000","last_clean":"2024-09-13T14:49:35.787862+0000","last_became_active":"2024-09-13T14:19:25.870282+0000","last_became_peered":"2024-09-13T14:19:25.870282+0000","last_unstale":"2024-09-13T14:49:35.787862+0000","last_undegraded":"2024-09-13T14:49:35.787862+0000","last_fullsized":"2024-09-13T14:49:35.787862+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:52.514915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1132,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.787870+0000","last_change":"2024-09-13T14:43:07.526476+0000","last_active":"2024-09-13T14:49:35.787870+0000","last_peered":"2024-09-13T14:49:35.787870+0000","last_clean":"2024-09-13T14:49:35.787870+0000","last_became_active":"2024-09-13T14:43:07.525802+0000","last_became_peered":"2024-09-13T14:43:07.525802+0000","last_unstale":"2024-09-13T14:49:35.787870+0000","last_undegraded":"2024-09-13T14:49:35.787870+0000","last_fullsized":"2024-09-13T14:49:35.787870+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:23.154726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"261'100","reported_seq":115,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.137630+0000","last_change":"2024-09-13T14:45:19.807461+0000","last_active":"2024-09-13T14:49:53.137630+0000","last_peered":"2024-09-13T14:49:53.137630+0000","last_clean":"2024-09-13T14:49:53.137630+0000","last_became_active":"2024-09-13T14:45:19.807117+0000","last_became_peered":"2024-09-13T14:45:19.807117+0000","last_unstale":"2024-09-13T14:49:53.137630+0000","last_undegraded":"2024-09-13T14:49:53.137630+0000","last_fullsized":"2024-09-13T14:49:53.137630+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.955197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.955197+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:03.531555+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.d","version":"1172'3721","reported_seq":3734,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.702671+0000","last_change":"2024-09-13T14:49:22.120980+0000","last_active":"2024-09-13T14:50:07.702671+0000","last_peered":"2024-09-13T14:50:07.702671+0000","last_clean":"2024-09-13T14:50:07.702671+0000","last_became_active":"2024-09-13T14:49:22.120684+0000","last_became_peered":"2024-09-13T14:49:22.120684+0000","last_unstale":"2024-09-13T14:50:07.702671+0000","last_undegraded":"2024-09-13T14:50:07.702671+0000","last_fullsized":"2024-09-13T14:50:07.702671+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":3721,"log_dups_size":0,"ondisk_log_size":3721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:49:34.732581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243859456,"num_objects":3721,"num_object_clones":0,"num_object_copies":11163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3721,"num_write_kb":238144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1095,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.787380+0000","last_change":"2024-09-13T14:19:26.131609+0000","last_active":"2024-09-13T14:49:35.787380+0000","last_peered":"2024-09-13T14:49:35.787380+0000","last_clean":"2024-09-13T14:49:35.787380+0000","last_became_active":"2024-09-13T14:19:26.131029+0000","last_became_peered":"2024-09-13T14:19:26.131029+0000","last_unstale":"2024-09-13T14:49:35.787380+0000","last_undegraded":"2024-09-13T14:49:35.787380+0000","last_fullsized":"2024-09-13T14:49:35.787380+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:57.933696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":597,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.603726+0000","last_change":"2024-09-13T14:43:32.635070+0000","last_active":"2024-09-13T14:49:35.603726+0000","last_peered":"2024-09-13T14:49:35.603726+0000","last_clean":"2024-09-13T14:49:35.603726+0000","last_became_active":"2024-09-13T14:43:32.634734+0000","last_became_peered":"2024-09-13T14:43:32.634734+0000","last_unstale":"2024-09-13T14:49:35.603726+0000","last_undegraded":"2024-09-13T14:49:35.603726+0000","last_fullsized":"2024-09-13T14:49:35.603726+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:33.107010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":1750,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.131870+0000","last_change":"2024-09-13T14:43:07.537759+0000","last_active":"2024-09-13T14:49:53.131870+0000","last_peered":"2024-09-13T14:49:53.131870+0000","last_clean":"2024-09-13T14:49:53.131870+0000","last_became_active":"2024-09-13T14:43:07.520303+0000","last_became_peered":"2024-09-13T14:43:07.520303+0000","last_unstale":"2024-09-13T14:49:53.131870+0000","last_undegraded":"2024-09-13T14:49:53.131870+0000","last_fullsized":"2024-09-13T14:49:53.131870+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:32.772570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"261'100","reported_seq":60,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.132525+0000","last_change":"2024-09-13T14:45:19.813527+0000","last_active":"2024-09-13T14:49:53.132525+0000","last_peered":"2024-09-13T14:49:53.132525+0000","last_clean":"2024-09-13T14:49:53.132525+0000","last_became_active":"2024-09-13T14:45:19.812834+0000","last_became_peered":"2024-09-13T14:45:19.812834+0000","last_unstale":"2024-09-13T14:49:53.132525+0000","last_undegraded":"2024-09-13T14:49:53.132525+0000","last_fullsized":"2024-09-13T14:49:53.132525+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.710192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.710192+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:08:55.152703+0000","scrub_duration":0.0019250910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"1172'3725","reported_seq":3746,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:50:09.669806+0000","last_change":"2024-09-13T14:49:22.114132+0000","last_active":"2024-09-13T14:50:09.669806+0000","last_peered":"2024-09-13T14:50:09.669806+0000","last_clean":"2024-09-13T14:50:09.669806+0000","last_became_active":"2024-09-13T14:49:22.113820+0000","last_became_peered":"2024-09-13T14:49:22.113820+0000","last_unstale":"2024-09-13T14:50:09.669806+0000","last_undegraded":"2024-09-13T14:50:09.669806+0000","last_fullsized":"2024-09-13T14:50:09.669806+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":3725,"log_dups_size":0,"ondisk_log_size":3725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:34:37.889567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244121600,"num_objects":3725,"num_object_clones":0,"num_object_copies":11175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3725,"num_write_kb":238400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":60,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.130213+0000","last_change":"2024-09-13T14:45:19.813025+0000","last_active":"2024-09-13T14:49:53.130213+0000","last_peered":"2024-09-13T14:49:53.130213+0000","last_clean":"2024-09-13T14:49:53.130213+0000","last_became_active":"2024-09-13T14:45:19.812714+0000","last_became_peered":"2024-09-13T14:45:19.812714+0000","last_unstale":"2024-09-13T14:49:53.130213+0000","last_undegraded":"2024-09-13T14:49:53.130213+0000","last_fullsized":"2024-09-13T14:49:53.130213+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:40.671012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":109,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.137916+0000","last_change":"2024-09-13T14:43:07.522730+0000","last_active":"2024-09-13T14:49:53.137916+0000","last_peered":"2024-09-13T14:49:53.137916+0000","last_clean":"2024-09-13T14:49:53.137916+0000","last_became_active":"2024-09-13T14:43:07.517420+0000","last_became_peered":"2024-09-13T14:43:07.517420+0000","last_unstale":"2024-09-13T14:49:53.137916+0000","last_undegraded":"2024-09-13T14:49:53.137916+0000","last_fullsized":"2024-09-13T14:49:53.137916+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:21:30.265293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":1664,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.134528+0000","last_change":"2024-09-13T14:30:15.581728+0000","last_active":"2024-09-13T14:49:53.134528+0000","last_peered":"2024-09-13T14:49:53.134528+0000","last_clean":"2024-09-13T14:49:53.134528+0000","last_became_active":"2024-09-13T14:30:15.581303+0000","last_became_peered":"2024-09-13T14:30:15.581303+0000","last_unstale":"2024-09-13T14:49:53.134528+0000","last_undegraded":"2024-09-13T14:49:53.134528+0000","last_fullsized":"2024-09-13T14:49:53.134528+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:10.027991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"261'100","reported_seq":122,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.788066+0000","last_change":"2024-09-13T14:43:31.112173+0000","last_active":"2024-09-13T14:49:35.788066+0000","last_peered":"2024-09-13T14:49:35.788066+0000","last_clean":"2024-09-13T14:49:35.788066+0000","last_became_active":"2024-09-13T14:42:42.709263+0000","last_became_peered":"2024-09-13T14:42:42.709263+0000","last_unstale":"2024-09-13T14:49:35.788066+0000","last_undegraded":"2024-09-13T14:49:35.788066+0000","last_fullsized":"2024-09-13T14:49:35.788066+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:31.112111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:31.112111+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:02.707372+0000","scrub_duration":0.0015759260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"1172'3920","reported_seq":3941,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:50:09.586862+0000","last_change":"2024-09-13T14:49:22.118734+0000","last_active":"2024-09-13T14:50:09.586862+0000","last_peered":"2024-09-13T14:50:09.586862+0000","last_clean":"2024-09-13T14:50:09.586862+0000","last_became_active":"2024-09-13T14:49:22.118438+0000","last_became_peered":"2024-09-13T14:49:22.118438+0000","last_unstale":"2024-09-13T14:50:09.586862+0000","last_undegraded":"2024-09-13T14:50:09.586862+0000","last_fullsized":"2024-09-13T14:50:09.586862+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":3920,"log_dups_size":0,"ondisk_log_size":3920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:27:01.810072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256901120,"num_objects":3920,"num_object_clones":0,"num_object_copies":11760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3920,"num_write_kb":250880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1704,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.128964+0000","last_change":"2024-09-13T14:28:35.611508+0000","last_active":"2024-09-13T14:49:53.128964+0000","last_peered":"2024-09-13T14:49:53.128964+0000","last_clean":"2024-09-13T14:49:53.128964+0000","last_became_active":"2024-09-13T14:28:35.610106+0000","last_became_peered":"2024-09-13T14:28:35.610106+0000","last_unstale":"2024-09-13T14:49:53.128964+0000","last_undegraded":"2024-09-13T14:49:53.128964+0000","last_fullsized":"2024-09-13T14:49:53.128964+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:11.966958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1138,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.603821+0000","last_change":"2024-09-13T14:28:35.603663+0000","last_active":"2024-09-13T14:49:35.603821+0000","last_peered":"2024-09-13T14:49:35.603821+0000","last_clean":"2024-09-13T14:49:35.603821+0000","last_became_active":"2024-09-13T14:28:35.603358+0000","last_became_peered":"2024-09-13T14:28:35.603358+0000","last_unstale":"2024-09-13T14:49:35.603821+0000","last_undegraded":"2024-09-13T14:49:35.603821+0000","last_fullsized":"2024-09-13T14:49:35.603821+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:24:05.842796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":109,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.138117+0000","last_change":"2024-09-13T14:43:07.522750+0000","last_active":"2024-09-13T14:49:53.138117+0000","last_peered":"2024-09-13T14:49:53.138117+0000","last_clean":"2024-09-13T14:49:53.138117+0000","last_became_active":"2024-09-13T14:43:07.517267+0000","last_became_peered":"2024-09-13T14:43:07.517267+0000","last_unstale":"2024-09-13T14:49:53.138117+0000","last_undegraded":"2024-09-13T14:49:53.138117+0000","last_fullsized":"2024-09-13T14:49:53.138117+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:40.217748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"261'100","reported_seq":430,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.131571+0000","last_change":"2024-09-13T14:45:19.815050+0000","last_active":"2024-09-13T14:49:53.131571+0000","last_peered":"2024-09-13T14:49:53.131571+0000","last_clean":"2024-09-13T14:49:53.131571+0000","last_became_active":"2024-09-13T14:45:19.809732+0000","last_became_peered":"2024-09-13T14:45:19.809732+0000","last_unstale":"2024-09-13T14:49:53.131571+0000","last_undegraded":"2024-09-13T14:49:53.131571+0000","last_fullsized":"2024-09-13T14:49:53.131571+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:33.574443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:33.574443+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:51:40.261036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"1172'3800","reported_seq":3821,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:50:09.692058+0000","last_change":"2024-09-13T14:49:22.122114+0000","last_active":"2024-09-13T14:50:09.692058+0000","last_peered":"2024-09-13T14:50:09.692058+0000","last_clean":"2024-09-13T14:50:09.692058+0000","last_became_active":"2024-09-13T14:49:22.120044+0000","last_became_peered":"2024-09-13T14:49:22.120044+0000","last_unstale":"2024-09-13T14:50:09.692058+0000","last_undegraded":"2024-09-13T14:50:09.692058+0000","last_fullsized":"2024-09-13T14:50:09.692058+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":3800,"log_dups_size":0,"ondisk_log_size":3800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:09:06.965253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249036800,"num_objects":3800,"num_object_clones":0,"num_object_copies":11400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3800,"num_write_kb":243200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":109,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.138525+0000","last_change":"2024-09-13T14:43:07.520049+0000","last_active":"2024-09-13T14:49:53.138525+0000","last_peered":"2024-09-13T14:49:53.138525+0000","last_clean":"2024-09-13T14:49:53.138525+0000","last_became_active":"2024-09-13T14:43:07.514646+0000","last_became_peered":"2024-09-13T14:43:07.514646+0000","last_unstale":"2024-09-13T14:49:53.138525+0000","last_undegraded":"2024-09-13T14:49:53.138525+0000","last_fullsized":"2024-09-13T14:49:53.138525+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:10:35.200778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":109,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.137844+0000","last_change":"2024-09-13T14:43:07.525666+0000","last_active":"2024-09-13T14:49:53.137844+0000","last_peered":"2024-09-13T14:49:53.137844+0000","last_clean":"2024-09-13T14:49:53.137844+0000","last_became_active":"2024-09-13T14:43:07.524064+0000","last_became_peered":"2024-09-13T14:43:07.524064+0000","last_unstale":"2024-09-13T14:49:53.137844+0000","last_undegraded":"2024-09-13T14:49:53.137844+0000","last_fullsized":"2024-09-13T14:49:53.137844+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:10.270333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1152,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.209446+0000","last_change":"2024-09-13T14:45:19.804177+0000","last_active":"2024-09-13T14:49:36.209446+0000","last_peered":"2024-09-13T14:49:36.209446+0000","last_clean":"2024-09-13T14:49:36.209446+0000","last_became_active":"2024-09-13T14:45:19.803890+0000","last_became_peered":"2024-09-13T14:45:19.803890+0000","last_unstale":"2024-09-13T14:49:36.209446+0000","last_undegraded":"2024-09-13T14:49:36.209446+0000","last_fullsized":"2024-09-13T14:49:36.209446+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:25.140789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"261'100","reported_seq":77,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.134384+0000","last_change":"2024-09-13T14:45:36.965992+0000","last_active":"2024-09-13T14:49:53.134384+0000","last_peered":"2024-09-13T14:49:53.134384+0000","last_clean":"2024-09-13T14:49:53.134384+0000","last_became_active":"2024-09-13T14:45:19.811238+0000","last_became_peered":"2024-09-13T14:45:19.811238+0000","last_unstale":"2024-09-13T14:49:53.134384+0000","last_undegraded":"2024-09-13T14:49:53.134384+0000","last_fullsized":"2024-09-13T14:49:53.134384+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:36.965775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:45:36.965775+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:36:01.623251+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"1172'3659","reported_seq":3672,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.706734+0000","last_change":"2024-09-13T14:49:22.122277+0000","last_active":"2024-09-13T14:50:07.706734+0000","last_peered":"2024-09-13T14:50:07.706734+0000","last_clean":"2024-09-13T14:50:07.706734+0000","last_became_active":"2024-09-13T14:49:22.121980+0000","last_became_peered":"2024-09-13T14:49:22.121980+0000","last_unstale":"2024-09-13T14:50:07.706734+0000","last_undegraded":"2024-09-13T14:50:07.706734+0000","last_fullsized":"2024-09-13T14:50:07.706734+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":3659,"log_dups_size":0,"ondisk_log_size":3659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:58.275564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239796224,"num_objects":3659,"num_object_clones":0,"num_object_copies":10977,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3659,"num_write_kb":234176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":51,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.209656+0000","last_change":"2024-09-13T14:45:19.806318+0000","last_active":"2024-09-13T14:49:36.209656+0000","last_peered":"2024-09-13T14:49:36.209656+0000","last_clean":"2024-09-13T14:49:36.209656+0000","last_became_active":"2024-09-13T14:45:19.806074+0000","last_became_peered":"2024-09-13T14:45:19.806074+0000","last_unstale":"2024-09-13T14:49:36.209656+0000","last_undegraded":"2024-09-13T14:49:36.209656+0000","last_fullsized":"2024-09-13T14:49:36.209656+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:21.679797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1113,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.787970+0000","last_change":"2024-09-13T14:28:35.604404+0000","last_active":"2024-09-13T14:49:35.787970+0000","last_peered":"2024-09-13T14:49:35.787970+0000","last_clean":"2024-09-13T14:49:35.787970+0000","last_became_active":"2024-09-13T14:28:35.604102+0000","last_became_peered":"2024-09-13T14:28:35.604102+0000","last_unstale":"2024-09-13T14:49:35.787970+0000","last_undegraded":"2024-09-13T14:49:35.787970+0000","last_fullsized":"2024-09-13T14:49:35.787970+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:25.546010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":1163,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.210549+0000","last_change":"2024-09-13T14:43:07.538919+0000","last_active":"2024-09-13T14:49:36.210549+0000","last_peered":"2024-09-13T14:49:36.210549+0000","last_clean":"2024-09-13T14:49:36.210549+0000","last_became_active":"2024-09-13T14:43:07.513109+0000","last_became_peered":"2024-09-13T14:43:07.513109+0000","last_unstale":"2024-09-13T14:49:36.210549+0000","last_undegraded":"2024-09-13T14:49:36.210549+0000","last_fullsized":"2024-09-13T14:49:36.210549+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:27.771720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"261'100","reported_seq":112,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.211115+0000","last_change":"2024-09-13T14:43:44.720333+0000","last_active":"2024-09-13T14:49:36.211115+0000","last_peered":"2024-09-13T14:49:36.211115+0000","last_clean":"2024-09-13T14:49:36.211115+0000","last_became_active":"2024-09-13T14:42:14.946591+0000","last_became_peered":"2024-09-13T14:42:14.946591+0000","last_unstale":"2024-09-13T14:49:36.211115+0000","last_undegraded":"2024-09-13T14:49:36.211115+0000","last_fullsized":"2024-09-13T14:49:36.211115+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:44.720258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:44.720258+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:33.281357+0000","scrub_duration":0.0018417920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"1172'3812","reported_seq":3832,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:50:09.587877+0000","last_change":"2024-09-13T14:49:22.119554+0000","last_active":"2024-09-13T14:50:09.587877+0000","last_peered":"2024-09-13T14:50:09.587877+0000","last_clean":"2024-09-13T14:50:09.587877+0000","last_became_active":"2024-09-13T14:49:22.119249+0000","last_became_peered":"2024-09-13T14:49:22.119249+0000","last_unstale":"2024-09-13T14:50:09.587877+0000","last_undegraded":"2024-09-13T14:50:09.587877+0000","last_fullsized":"2024-09-13T14:50:09.587877+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":3812,"log_dups_size":0,"ondisk_log_size":3812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:03:16.735024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249823232,"num_objects":3812,"num_object_clones":0,"num_object_copies":11436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3812,"num_write_kb":243968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":76,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.135030+0000","last_change":"2024-09-13T14:44:03.858211+0000","last_active":"2024-09-13T14:49:53.135030+0000","last_peered":"2024-09-13T14:49:53.135030+0000","last_clean":"2024-09-13T14:49:53.135030+0000","last_became_active":"2024-09-13T14:44:03.857928+0000","last_became_peered":"2024-09-13T14:44:03.857928+0000","last_unstale":"2024-09-13T14:49:53.135030+0000","last_undegraded":"2024-09-13T14:49:53.135030+0000","last_fullsized":"2024-09-13T14:49:53.135030+0000","mapping_epoch":1140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:28.428693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":109,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.137846+0000","last_change":"2024-09-13T14:43:07.516243+0000","last_active":"2024-09-13T14:49:53.137846+0000","last_peered":"2024-09-13T14:49:53.137846+0000","last_clean":"2024-09-13T14:49:53.137846+0000","last_became_active":"2024-09-13T14:43:07.515863+0000","last_became_peered":"2024-09-13T14:43:07.515863+0000","last_unstale":"2024-09-13T14:49:53.137846+0000","last_undegraded":"2024-09-13T14:49:53.137846+0000","last_fullsized":"2024-09-13T14:49:53.137846+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:08.085298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1745,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.135756+0000","last_change":"2024-09-13T14:45:19.812713+0000","last_active":"2024-09-13T14:49:53.135756+0000","last_peered":"2024-09-13T14:49:53.135756+0000","last_clean":"2024-09-13T14:49:53.135756+0000","last_became_active":"2024-09-13T14:45:19.812203+0000","last_became_peered":"2024-09-13T14:45:19.812203+0000","last_unstale":"2024-09-13T14:49:53.135756+0000","last_undegraded":"2024-09-13T14:49:53.135756+0000","last_fullsized":"2024-09-13T14:49:53.135756+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:30:46.149347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"261'100","reported_seq":1292,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.603093+0000","last_change":"2024-09-13T14:43:42.577312+0000","last_active":"2024-09-13T14:49:35.603093+0000","last_peered":"2024-09-13T14:49:35.603093+0000","last_clean":"2024-09-13T14:49:35.603093+0000","last_became_active":"2024-09-13T14:42:14.928923+0000","last_became_peered":"2024-09-13T14:42:14.928923+0000","last_unstale":"2024-09-13T14:49:35.603093+0000","last_undegraded":"2024-09-13T14:49:35.603093+0000","last_fullsized":"2024-09-13T14:49:35.603093+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:42.577243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:42.577243+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:01.234250+0000","scrub_duration":0.001367839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"1172'3749","reported_seq":3770,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:50:08.140167+0000","last_change":"2024-09-13T14:49:22.118690+0000","last_active":"2024-09-13T14:50:08.140167+0000","last_peered":"2024-09-13T14:50:08.140167+0000","last_clean":"2024-09-13T14:50:08.140167+0000","last_became_active":"2024-09-13T14:49:22.117901+0000","last_became_peered":"2024-09-13T14:49:22.117901+0000","last_unstale":"2024-09-13T14:50:08.140167+0000","last_undegraded":"2024-09-13T14:50:08.140167+0000","last_fullsized":"2024-09-13T14:50:08.140167+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":3749,"log_dups_size":0,"ondisk_log_size":3749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:28.098913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245694464,"num_objects":3749,"num_object_clones":0,"num_object_copies":11247,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3749,"num_write_kb":239936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1739,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.131618+0000","last_change":"2024-09-13T14:43:07.520212+0000","last_active":"2024-09-13T14:49:53.131618+0000","last_peered":"2024-09-13T14:49:53.131618+0000","last_clean":"2024-09-13T14:49:53.131618+0000","last_became_active":"2024-09-13T14:43:07.519831+0000","last_became_peered":"2024-09-13T14:43:07.519831+0000","last_unstale":"2024-09-13T14:49:53.131618+0000","last_undegraded":"2024-09-13T14:49:53.131618+0000","last_fullsized":"2024-09-13T14:49:53.131618+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:20.990736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1762,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.129998+0000","last_change":"2024-09-13T14:45:19.809505+0000","last_active":"2024-09-13T14:49:53.129998+0000","last_peered":"2024-09-13T14:49:53.129998+0000","last_clean":"2024-09-13T14:49:53.129998+0000","last_became_active":"2024-09-13T14:45:19.809181+0000","last_became_peered":"2024-09-13T14:45:19.809181+0000","last_unstale":"2024-09-13T14:49:53.129998+0000","last_undegraded":"2024-09-13T14:49:53.129998+0000","last_fullsized":"2024-09-13T14:49:53.129998+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:24.239188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1175,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.603195+0000","last_change":"2024-09-13T14:43:07.516140+0000","last_active":"2024-09-13T14:49:35.603195+0000","last_peered":"2024-09-13T14:49:35.603195+0000","last_clean":"2024-09-13T14:49:35.603195+0000","last_became_active":"2024-09-13T14:43:07.515481+0000","last_became_peered":"2024-09-13T14:43:07.515481+0000","last_unstale":"2024-09-13T14:49:35.603195+0000","last_undegraded":"2024-09-13T14:49:35.603195+0000","last_fullsized":"2024-09-13T14:49:35.603195+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:43.706214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"261'100","reported_seq":121,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.139303+0000","last_change":"2024-09-13T14:43:16.008346+0000","last_active":"2024-09-13T14:49:53.139303+0000","last_peered":"2024-09-13T14:49:53.139303+0000","last_clean":"2024-09-13T14:49:53.139303+0000","last_became_active":"2024-09-13T14:43:07.523121+0000","last_became_peered":"2024-09-13T14:43:07.523121+0000","last_unstale":"2024-09-13T14:49:53.139303+0000","last_undegraded":"2024-09-13T14:49:53.139303+0000","last_fullsized":"2024-09-13T14:49:53.139303+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.008264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.008264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:58:35.564470+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"1172'3536","reported_seq":3549,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.689868+0000","last_change":"2024-09-13T14:49:22.119570+0000","last_active":"2024-09-13T14:50:07.689868+0000","last_peered":"2024-09-13T14:50:07.689868+0000","last_clean":"2024-09-13T14:50:07.689868+0000","last_became_active":"2024-09-13T14:49:22.119224+0000","last_became_peered":"2024-09-13T14:49:22.119224+0000","last_unstale":"2024-09-13T14:50:07.689868+0000","last_undegraded":"2024-09-13T14:50:07.689868+0000","last_fullsized":"2024-09-13T14:50:07.689868+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":3536,"log_dups_size":0,"ondisk_log_size":3536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:43.554416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231735296,"num_objects":3536,"num_object_clones":0,"num_object_copies":10608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3536,"num_write_kb":226304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":54,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.788008+0000","last_change":"2024-09-13T14:43:32.636990+0000","last_active":"2024-09-13T14:49:35.788008+0000","last_peered":"2024-09-13T14:49:35.788008+0000","last_clean":"2024-09-13T14:49:35.788008+0000","last_became_active":"2024-09-13T14:43:32.636589+0000","last_became_peered":"2024-09-13T14:43:32.636589+0000","last_unstale":"2024-09-13T14:49:35.788008+0000","last_undegraded":"2024-09-13T14:49:35.788008+0000","last_fullsized":"2024-09-13T14:49:35.788008+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:36:25.129769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":60,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.129653+0000","last_change":"2024-09-13T14:45:19.810888+0000","last_active":"2024-09-13T14:49:53.129653+0000","last_peered":"2024-09-13T14:49:53.129653+0000","last_clean":"2024-09-13T14:49:53.129653+0000","last_became_active":"2024-09-13T14:45:19.810486+0000","last_became_peered":"2024-09-13T14:45:19.810486+0000","last_unstale":"2024-09-13T14:49:53.129653+0000","last_undegraded":"2024-09-13T14:49:53.129653+0000","last_fullsized":"2024-09-13T14:49:53.129653+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:03.886942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":1128,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.211358+0000","last_change":"2024-09-13T14:29:42.539455+0000","last_active":"2024-09-13T14:49:36.211358+0000","last_peered":"2024-09-13T14:49:36.211358+0000","last_clean":"2024-09-13T14:49:36.211358+0000","last_became_active":"2024-09-13T14:29:42.539068+0000","last_became_peered":"2024-09-13T14:29:42.539068+0000","last_unstale":"2024-09-13T14:49:36.211358+0000","last_undegraded":"2024-09-13T14:49:36.211358+0000","last_fullsized":"2024-09-13T14:49:36.211358+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:24:06.048855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034405999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"261'100","reported_seq":1283,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.603259+0000","last_change":"2024-09-13T14:44:28.186029+0000","last_active":"2024-09-13T14:49:35.603259+0000","last_peered":"2024-09-13T14:49:35.603259+0000","last_clean":"2024-09-13T14:49:35.603259+0000","last_became_active":"2024-09-13T14:42:14.927771+0000","last_became_peered":"2024-09-13T14:42:14.927771+0000","last_unstale":"2024-09-13T14:49:35.603259+0000","last_undegraded":"2024-09-13T14:49:35.603259+0000","last_fullsized":"2024-09-13T14:49:35.603259+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:28.185949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:28.185949+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:38.363961+0000","scrub_duration":0.0018071129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3","version":"1172'3593","reported_seq":3614,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:50:08.083095+0000","last_change":"2024-09-13T14:49:22.119776+0000","last_active":"2024-09-13T14:50:08.083095+0000","last_peered":"2024-09-13T14:50:08.083095+0000","last_clean":"2024-09-13T14:50:08.083095+0000","last_became_active":"2024-09-13T14:49:22.119517+0000","last_became_peered":"2024-09-13T14:49:22.119517+0000","last_unstale":"2024-09-13T14:50:08.083095+0000","last_undegraded":"2024-09-13T14:50:08.083095+0000","last_fullsized":"2024-09-13T14:50:08.083095+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":3593,"log_dups_size":0,"ondisk_log_size":3593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:53.152882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235470848,"num_objects":3593,"num_object_clones":0,"num_object_copies":10779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3593,"num_write_kb":229952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":109,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.138276+0000","last_change":"2024-09-13T14:43:07.520176+0000","last_active":"2024-09-13T14:49:53.138276+0000","last_peered":"2024-09-13T14:49:53.138276+0000","last_clean":"2024-09-13T14:49:53.138276+0000","last_became_active":"2024-09-13T14:43:07.516748+0000","last_became_peered":"2024-09-13T14:43:07.516748+0000","last_unstale":"2024-09-13T14:49:53.138276+0000","last_undegraded":"2024-09-13T14:49:53.138276+0000","last_fullsized":"2024-09-13T14:49:53.138276+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:52:04.392821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":60,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.130375+0000","last_change":"2024-09-13T14:45:19.812353+0000","last_active":"2024-09-13T14:49:53.130375+0000","last_peered":"2024-09-13T14:49:53.130375+0000","last_clean":"2024-09-13T14:49:53.130375+0000","last_became_active":"2024-09-13T14:45:19.811743+0000","last_became_peered":"2024-09-13T14:45:19.811743+0000","last_unstale":"2024-09-13T14:49:53.130375+0000","last_undegraded":"2024-09-13T14:49:53.130375+0000","last_fullsized":"2024-09-13T14:49:53.130375+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:43.728170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"261'100","reported_seq":112,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.210186+0000","last_change":"2024-09-13T14:44:34.814157+0000","last_active":"2024-09-13T14:49:36.210186+0000","last_peered":"2024-09-13T14:49:36.210186+0000","last_clean":"2024-09-13T14:49:36.210186+0000","last_became_active":"2024-09-13T14:42:42.707803+0000","last_became_peered":"2024-09-13T14:42:42.707803+0000","last_unstale":"2024-09-13T14:49:36.210186+0000","last_undegraded":"2024-09-13T14:49:36.210186+0000","last_fullsized":"2024-09-13T14:49:36.210186+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:34.814070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:34.814070+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:38.408561+0000","scrub_duration":0.01686414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"1172'3642","reported_seq":3655,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.497246+0000","last_change":"2024-09-13T14:49:22.119578+0000","last_active":"2024-09-13T14:50:07.497246+0000","last_peered":"2024-09-13T14:50:07.497246+0000","last_clean":"2024-09-13T14:50:07.497246+0000","last_became_active":"2024-09-13T14:49:22.119280+0000","last_became_peered":"2024-09-13T14:49:22.119280+0000","last_unstale":"2024-09-13T14:50:07.497246+0000","last_undegraded":"2024-09-13T14:50:07.497246+0000","last_fullsized":"2024-09-13T14:50:07.497246+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":3642,"log_dups_size":0,"ondisk_log_size":3642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:13:15.669993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238682112,"num_objects":3642,"num_object_clones":0,"num_object_copies":10926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3642,"num_write_kb":233088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1142,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.788181+0000","last_change":"2024-09-13T14:44:03.855577+0000","last_active":"2024-09-13T14:49:35.788181+0000","last_peered":"2024-09-13T14:49:35.788181+0000","last_clean":"2024-09-13T14:49:35.788181+0000","last_became_active":"2024-09-13T14:44:03.855267+0000","last_became_peered":"2024-09-13T14:44:03.855267+0000","last_unstale":"2024-09-13T14:49:35.788181+0000","last_undegraded":"2024-09-13T14:49:35.788181+0000","last_fullsized":"2024-09-13T14:49:35.788181+0000","mapping_epoch":1140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:46.148814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1113,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.787468+0000","last_change":"2024-09-13T14:29:42.545815+0000","last_active":"2024-09-13T14:49:35.787468+0000","last_peered":"2024-09-13T14:49:35.787468+0000","last_clean":"2024-09-13T14:49:35.787468+0000","last_became_active":"2024-09-13T14:29:42.545508+0000","last_became_peered":"2024-09-13T14:29:42.545508+0000","last_unstale":"2024-09-13T14:49:35.787468+0000","last_undegraded":"2024-09-13T14:49:35.787468+0000","last_fullsized":"2024-09-13T14:49:35.787468+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:39.678381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"261'100","reported_seq":127,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.138175+0000","last_change":"2024-09-13T14:43:18.988482+0000","last_active":"2024-09-13T14:49:53.138175+0000","last_peered":"2024-09-13T14:49:53.138175+0000","last_clean":"2024-09-13T14:49:53.138175+0000","last_became_active":"2024-09-13T14:43:14.556754+0000","last_became_peered":"2024-09-13T14:43:14.556754+0000","last_unstale":"2024-09-13T14:49:53.138175+0000","last_undegraded":"2024-09-13T14:49:53.138175+0000","last_fullsized":"2024-09-13T14:49:53.138175+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:18.988405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:18.988405+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:45:01.726326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"1172'3675","reported_seq":3696,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:50:08.132930+0000","last_change":"2024-09-13T14:49:22.121230+0000","last_active":"2024-09-13T14:50:08.132930+0000","last_peered":"2024-09-13T14:50:08.132930+0000","last_clean":"2024-09-13T14:50:08.132930+0000","last_became_active":"2024-09-13T14:49:22.120836+0000","last_became_peered":"2024-09-13T14:49:22.120836+0000","last_unstale":"2024-09-13T14:50:08.132930+0000","last_undegraded":"2024-09-13T14:50:08.132930+0000","last_fullsized":"2024-09-13T14:50:08.132930+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":3675,"log_dups_size":0,"ondisk_log_size":3675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:21.843404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240844800,"num_objects":3675,"num_object_clones":0,"num_object_copies":11025,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3675,"num_write_kb":235200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1159,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.788209+0000","last_change":"2024-09-13T14:43:07.515418+0000","last_active":"2024-09-13T14:49:35.788209+0000","last_peered":"2024-09-13T14:49:35.788209+0000","last_clean":"2024-09-13T14:49:35.788209+0000","last_became_active":"2024-09-13T14:43:07.515113+0000","last_became_peered":"2024-09-13T14:43:07.515113+0000","last_unstale":"2024-09-13T14:49:35.788209+0000","last_undegraded":"2024-09-13T14:49:35.788209+0000","last_fullsized":"2024-09-13T14:49:35.788209+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:10:25.810604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1126,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.787651+0000","last_change":"2024-09-13T14:43:32.636334+0000","last_active":"2024-09-13T14:49:35.787651+0000","last_peered":"2024-09-13T14:49:35.787651+0000","last_clean":"2024-09-13T14:49:35.787651+0000","last_became_active":"2024-09-13T14:43:32.635736+0000","last_became_peered":"2024-09-13T14:43:32.635736+0000","last_unstale":"2024-09-13T14:49:35.787651+0000","last_undegraded":"2024-09-13T14:49:35.787651+0000","last_fullsized":"2024-09-13T14:49:35.787651+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:49.076014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"261'100","reported_seq":117,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.209363+0000","last_change":"2024-09-13T14:45:19.805634+0000","last_active":"2024-09-13T14:49:36.209363+0000","last_peered":"2024-09-13T14:49:36.209363+0000","last_clean":"2024-09-13T14:49:36.209363+0000","last_became_active":"2024-09-13T14:45:19.805046+0000","last_became_peered":"2024-09-13T14:45:19.805046+0000","last_unstale":"2024-09-13T14:49:36.209363+0000","last_undegraded":"2024-09-13T14:49:36.209363+0000","last_fullsized":"2024-09-13T14:49:36.209363+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:32.752408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:32.752408+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:06:53.140239+0000","scrub_duration":0.0057731559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.0","version":"1172'3665","reported_seq":3686,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:50:09.581711+0000","last_change":"2024-09-13T14:49:22.118730+0000","last_active":"2024-09-13T14:50:09.581711+0000","last_peered":"2024-09-13T14:50:09.581711+0000","last_clean":"2024-09-13T14:50:09.581711+0000","last_became_active":"2024-09-13T14:49:22.118428+0000","last_became_peered":"2024-09-13T14:49:22.118428+0000","last_unstale":"2024-09-13T14:50:09.581711+0000","last_undegraded":"2024-09-13T14:50:09.581711+0000","last_fullsized":"2024-09-13T14:50:09.581711+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":3665,"log_dups_size":0,"ondisk_log_size":3665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:54.951267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240189440,"num_objects":3665,"num_object_clones":0,"num_object_copies":10995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3665,"num_write_kb":234560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1162,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.787801+0000","last_change":"2024-09-13T14:45:19.804441+0000","last_active":"2024-09-13T14:49:35.787801+0000","last_peered":"2024-09-13T14:49:35.787801+0000","last_clean":"2024-09-13T14:49:35.787801+0000","last_became_active":"2024-09-13T14:45:19.804039+0000","last_became_peered":"2024-09-13T14:45:19.804039+0000","last_unstale":"2024-09-13T14:49:35.787801+0000","last_undegraded":"2024-09-13T14:49:35.787801+0000","last_fullsized":"2024-09-13T14:49:35.787801+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:58:00.322042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":64,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.129742+0000","last_change":"2024-09-13T14:45:19.816666+0000","last_active":"2024-09-13T14:49:53.129742+0000","last_peered":"2024-09-13T14:49:53.129742+0000","last_clean":"2024-09-13T14:49:53.129742+0000","last_became_active":"2024-09-13T14:45:19.807224+0000","last_became_peered":"2024-09-13T14:45:19.807224+0000","last_unstale":"2024-09-13T14:49:53.129742+0000","last_undegraded":"2024-09-13T14:49:53.129742+0000","last_fullsized":"2024-09-13T14:49:53.129742+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:15:20.171682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"261'100","reported_seq":1302,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.603014+0000","last_change":"2024-09-13T14:45:19.798553+0000","last_active":"2024-09-13T14:49:35.603014+0000","last_peered":"2024-09-13T14:49:35.603014+0000","last_clean":"2024-09-13T14:49:35.603014+0000","last_became_active":"2024-09-13T14:45:19.798189+0000","last_became_peered":"2024-09-13T14:45:19.798189+0000","last_unstale":"2024-09-13T14:49:35.603014+0000","last_undegraded":"2024-09-13T14:49:35.603014+0000","last_fullsized":"2024-09-13T14:49:35.603014+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:29.169325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:29.169325+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:10.412666+0000","scrub_duration":0.0019358890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"1172'3621","reported_seq":3642,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:50:08.132116+0000","last_change":"2024-09-13T14:49:22.120585+0000","last_active":"2024-09-13T14:50:08.132116+0000","last_peered":"2024-09-13T14:50:08.132116+0000","last_clean":"2024-09-13T14:50:08.132116+0000","last_became_active":"2024-09-13T14:49:22.120301+0000","last_became_peered":"2024-09-13T14:49:22.120301+0000","last_unstale":"2024-09-13T14:50:08.132116+0000","last_undegraded":"2024-09-13T14:50:08.132116+0000","last_fullsized":"2024-09-13T14:50:08.132116+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":3621,"log_dups_size":0,"ondisk_log_size":3621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:20:25.183879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237305856,"num_objects":3621,"num_object_clones":0,"num_object_copies":10863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3621,"num_write_kb":231744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1728,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.128881+0000","last_change":"2024-09-13T14:43:07.515572+0000","last_active":"2024-09-13T14:49:53.128881+0000","last_peered":"2024-09-13T14:49:53.128881+0000","last_clean":"2024-09-13T14:49:53.128881+0000","last_became_active":"2024-09-13T14:43:07.515231+0000","last_became_peered":"2024-09-13T14:43:07.515231+0000","last_unstale":"2024-09-13T14:49:53.128881+0000","last_undegraded":"2024-09-13T14:49:53.128881+0000","last_fullsized":"2024-09-13T14:49:53.128881+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:57.966380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1155,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.209527+0000","last_change":"2024-09-13T14:45:19.804982+0000","last_active":"2024-09-13T14:49:36.209527+0000","last_peered":"2024-09-13T14:49:36.209527+0000","last_clean":"2024-09-13T14:49:36.209527+0000","last_became_active":"2024-09-13T14:45:19.804663+0000","last_became_peered":"2024-09-13T14:45:19.804663+0000","last_unstale":"2024-09-13T14:49:36.209527+0000","last_undegraded":"2024-09-13T14:49:36.209527+0000","last_fullsized":"2024-09-13T14:49:36.209527+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:43.211479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"261'100","reported_seq":392,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.788096+0000","last_change":"2024-09-13T14:43:42.881584+0000","last_active":"2024-09-13T14:49:35.788096+0000","last_peered":"2024-09-13T14:49:35.788096+0000","last_clean":"2024-09-13T14:49:35.788096+0000","last_became_active":"2024-09-13T14:43:14.562161+0000","last_became_peered":"2024-09-13T14:43:14.562161+0000","last_unstale":"2024-09-13T14:49:35.788096+0000","last_undegraded":"2024-09-13T14:49:35.788096+0000","last_fullsized":"2024-09-13T14:49:35.788096+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:42.881520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:42.881520+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:06:50.353442+0000","scrub_duration":0.0013804310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"1172'3682","reported_seq":3703,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:50:09.695624+0000","last_change":"2024-09-13T14:49:22.117570+0000","last_active":"2024-09-13T14:50:09.695624+0000","last_peered":"2024-09-13T14:50:09.695624+0000","last_clean":"2024-09-13T14:50:09.695624+0000","last_became_active":"2024-09-13T14:49:22.117259+0000","last_became_peered":"2024-09-13T14:49:22.117259+0000","last_unstale":"2024-09-13T14:50:09.695624+0000","last_undegraded":"2024-09-13T14:50:09.695624+0000","last_fullsized":"2024-09-13T14:50:09.695624+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":3682,"log_dups_size":0,"ondisk_log_size":3682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:59:23.669062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241303552,"num_objects":3682,"num_object_clones":0,"num_object_copies":11046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3682,"num_write_kb":235648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1128,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.210028+0000","last_change":"2024-09-13T14:43:07.518120+0000","last_active":"2024-09-13T14:49:36.210028+0000","last_peered":"2024-09-13T14:49:36.210028+0000","last_clean":"2024-09-13T14:49:36.210028+0000","last_became_active":"2024-09-13T14:43:07.517361+0000","last_became_peered":"2024-09-13T14:43:07.517361+0000","last_unstale":"2024-09-13T14:49:36.210028+0000","last_undegraded":"2024-09-13T14:49:36.210028+0000","last_fullsized":"2024-09-13T14:49:36.210028+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:39.366473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":1110,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.603695+0000","last_change":"2024-09-13T14:19:25.871006+0000","last_active":"2024-09-13T14:49:35.603695+0000","last_peered":"2024-09-13T14:49:35.603695+0000","last_clean":"2024-09-13T14:49:35.603695+0000","last_became_active":"2024-09-13T14:19:25.870713+0000","last_became_peered":"2024-09-13T14:19:25.870713+0000","last_unstale":"2024-09-13T14:49:35.603695+0000","last_undegraded":"2024-09-13T14:49:35.603695+0000","last_fullsized":"2024-09-13T14:49:35.603695+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:34.372124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"261'100","reported_seq":386,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.210946+0000","last_change":"2024-09-13T14:49:21.723011+0000","last_active":"2024-09-13T14:49:36.210946+0000","last_peered":"2024-09-13T14:49:36.210946+0000","last_clean":"2024-09-13T14:49:36.210946+0000","last_became_active":"2024-09-13T14:43:32.636545+0000","last_became_peered":"2024-09-13T14:43:32.636545+0000","last_unstale":"2024-09-13T14:49:36.210946+0000","last_undegraded":"2024-09-13T14:49:36.210946+0000","last_fullsized":"2024-09-13T14:49:36.210946+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:49:21.722939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.722939+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:18:07.462147+0000","scrub_duration":0.0016642790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.5","version":"1172'3697","reported_seq":3710,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.792087+0000","last_change":"2024-09-13T14:49:22.120623+0000","last_active":"2024-09-13T14:50:07.792087+0000","last_peered":"2024-09-13T14:50:07.792087+0000","last_clean":"2024-09-13T14:50:07.792087+0000","last_became_active":"2024-09-13T14:49:22.120295+0000","last_became_peered":"2024-09-13T14:49:22.120295+0000","last_unstale":"2024-09-13T14:50:07.792087+0000","last_undegraded":"2024-09-13T14:50:07.792087+0000","last_fullsized":"2024-09-13T14:50:07.792087+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":3697,"log_dups_size":0,"ondisk_log_size":3697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.848357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242286592,"num_objects":3697,"num_object_clones":0,"num_object_copies":11091,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3697,"num_write_kb":236608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1179,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.787466+0000","last_change":"2024-09-13T14:45:19.804868+0000","last_active":"2024-09-13T14:49:35.787466+0000","last_peered":"2024-09-13T14:49:35.787466+0000","last_clean":"2024-09-13T14:49:35.787466+0000","last_became_active":"2024-09-13T14:45:19.804520+0000","last_became_peered":"2024-09-13T14:45:19.804520+0000","last_unstale":"2024-09-13T14:49:35.787466+0000","last_undegraded":"2024-09-13T14:49:35.787466+0000","last_fullsized":"2024-09-13T14:49:35.787466+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:17:53.924629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":1156,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.603825+0000","last_change":"2024-09-13T14:43:07.533955+0000","last_active":"2024-09-13T14:49:35.603825+0000","last_peered":"2024-09-13T14:49:35.603825+0000","last_clean":"2024-09-13T14:49:35.603825+0000","last_became_active":"2024-09-13T14:43:07.516816+0000","last_became_peered":"2024-09-13T14:43:07.516816+0000","last_unstale":"2024-09-13T14:49:35.603825+0000","last_undegraded":"2024-09-13T14:49:35.603825+0000","last_fullsized":"2024-09-13T14:49:35.603825+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:26.232184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"261'100","reported_seq":64,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.132416+0000","last_change":"2024-09-13T14:45:19.821649+0000","last_active":"2024-09-13T14:49:53.132416+0000","last_peered":"2024-09-13T14:49:53.132416+0000","last_clean":"2024-09-13T14:49:53.132416+0000","last_became_active":"2024-09-13T14:45:19.812346+0000","last_became_peered":"2024-09-13T14:45:19.812346+0000","last_unstale":"2024-09-13T14:49:53.132416+0000","last_undegraded":"2024-09-13T14:49:53.132416+0000","last_fullsized":"2024-09-13T14:49:53.132416+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.597568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.597568+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:45.218892+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"1172'3601","reported_seq":3614,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.711403+0000","last_change":"2024-09-13T14:49:22.120890+0000","last_active":"2024-09-13T14:50:07.711403+0000","last_peered":"2024-09-13T14:50:07.711403+0000","last_clean":"2024-09-13T14:50:07.711403+0000","last_became_active":"2024-09-13T14:49:22.120420+0000","last_became_peered":"2024-09-13T14:49:22.120420+0000","last_unstale":"2024-09-13T14:50:07.711403+0000","last_undegraded":"2024-09-13T14:50:07.711403+0000","last_fullsized":"2024-09-13T14:50:07.711403+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":3601,"log_dups_size":0,"ondisk_log_size":3601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:03:49.271123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235995136,"num_objects":3601,"num_object_clones":0,"num_object_copies":10803,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3601,"num_write_kb":230464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1097,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.210368+0000","last_change":"2024-09-13T14:19:26.887839+0000","last_active":"2024-09-13T14:49:36.210368+0000","last_peered":"2024-09-13T14:49:36.210368+0000","last_clean":"2024-09-13T14:49:36.210368+0000","last_became_active":"2024-09-13T14:19:26.887498+0000","last_became_peered":"2024-09-13T14:19:26.887498+0000","last_unstale":"2024-09-13T14:49:36.210368+0000","last_undegraded":"2024-09-13T14:49:36.210368+0000","last_fullsized":"2024-09-13T14:49:36.210368+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:24.475263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":1127,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.603770+0000","last_change":"2024-09-13T14:28:35.608607+0000","last_active":"2024-09-13T14:49:35.603770+0000","last_peered":"2024-09-13T14:49:35.603770+0000","last_clean":"2024-09-13T14:49:35.603770+0000","last_became_active":"2024-09-13T14:28:35.608254+0000","last_became_peered":"2024-09-13T14:28:35.608254+0000","last_unstale":"2024-09-13T14:49:35.603770+0000","last_undegraded":"2024-09-13T14:49:35.603770+0000","last_fullsized":"2024-09-13T14:49:35.603770+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:24.019173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"261'100","reported_seq":63,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.603523+0000","last_change":"2024-09-13T14:44:31.191431+0000","last_active":"2024-09-13T14:49:35.603523+0000","last_peered":"2024-09-13T14:49:35.603523+0000","last_clean":"2024-09-13T14:49:35.603523+0000","last_became_active":"2024-09-13T14:43:14.557376+0000","last_became_peered":"2024-09-13T14:43:14.557376+0000","last_unstale":"2024-09-13T14:49:35.603523+0000","last_undegraded":"2024-09-13T14:49:35.603523+0000","last_fullsized":"2024-09-13T14:49:35.603523+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.191326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.191326+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:15:27.352698+0000","scrub_duration":0.0025317360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1117,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.210245+0000","last_change":"2024-09-13T14:28:35.603724+0000","last_active":"2024-09-13T14:49:36.210245+0000","last_peered":"2024-09-13T14:49:36.210245+0000","last_clean":"2024-09-13T14:49:36.210245+0000","last_became_active":"2024-09-13T14:28:35.603316+0000","last_became_peered":"2024-09-13T14:28:35.603316+0000","last_unstale":"2024-09-13T14:49:36.210245+0000","last_undegraded":"2024-09-13T14:49:36.210245+0000","last_fullsized":"2024-09-13T14:49:36.210245+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:47.668223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":59,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.134293+0000","last_change":"2024-09-13T14:45:19.811466+0000","last_active":"2024-09-13T14:49:53.134293+0000","last_peered":"2024-09-13T14:49:53.134293+0000","last_clean":"2024-09-13T14:49:53.134293+0000","last_became_active":"2024-09-13T14:45:19.810617+0000","last_became_peered":"2024-09-13T14:45:19.810617+0000","last_unstale":"2024-09-13T14:49:53.134293+0000","last_undegraded":"2024-09-13T14:49:53.134293+0000","last_fullsized":"2024-09-13T14:49:53.134293+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:16.246669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"261'100","reported_seq":63,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.210405+0000","last_change":"2024-09-13T14:44:50.805306+0000","last_active":"2024-09-13T14:49:36.210405+0000","last_peered":"2024-09-13T14:49:36.210405+0000","last_clean":"2024-09-13T14:49:36.210405+0000","last_became_active":"2024-09-13T14:43:14.562922+0000","last_became_peered":"2024-09-13T14:43:14.562922+0000","last_unstale":"2024-09-13T14:49:36.210405+0000","last_undegraded":"2024-09-13T14:49:36.210405+0000","last_fullsized":"2024-09-13T14:49:36.210405+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:50.805210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:50.805210+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:28:38.529599+0000","scrub_duration":0.0042900009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1109,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.603186+0000","last_change":"2024-09-13T14:19:25.873088+0000","last_active":"2024-09-13T14:49:35.603186+0000","last_peered":"2024-09-13T14:49:35.603186+0000","last_clean":"2024-09-13T14:49:35.603186+0000","last_became_active":"2024-09-13T14:19:25.872781+0000","last_became_peered":"2024-09-13T14:19:25.872781+0000","last_unstale":"2024-09-13T14:49:35.603186+0000","last_undegraded":"2024-09-13T14:49:35.603186+0000","last_fullsized":"2024-09-13T14:49:35.603186+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:05.607382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1727,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.131776+0000","last_change":"2024-09-13T14:45:19.813876+0000","last_active":"2024-09-13T14:49:53.131776+0000","last_peered":"2024-09-13T14:49:53.131776+0000","last_clean":"2024-09-13T14:49:53.131776+0000","last_became_active":"2024-09-13T14:45:19.809136+0000","last_became_peered":"2024-09-13T14:45:19.809136+0000","last_unstale":"2024-09-13T14:49:53.131776+0000","last_undegraded":"2024-09-13T14:49:53.131776+0000","last_fullsized":"2024-09-13T14:49:53.131776+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:50.013528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"261'100","reported_seq":134,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.138993+0000","last_change":"2024-09-13T14:45:19.810654+0000","last_active":"2024-09-13T14:49:53.138993+0000","last_peered":"2024-09-13T14:49:53.138993+0000","last_clean":"2024-09-13T14:49:53.138993+0000","last_became_active":"2024-09-13T14:45:19.810066+0000","last_became_peered":"2024-09-13T14:45:19.810066+0000","last_unstale":"2024-09-13T14:49:53.138993+0000","last_undegraded":"2024-09-13T14:49:53.138993+0000","last_fullsized":"2024-09-13T14:49:53.138993+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:19.978464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:19.978464+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:33.639332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1775,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.128813+0000","last_change":"2024-09-13T14:45:19.809166+0000","last_active":"2024-09-13T14:49:53.128813+0000","last_peered":"2024-09-13T14:49:53.128813+0000","last_clean":"2024-09-13T14:49:53.128813+0000","last_became_active":"2024-09-13T14:45:19.808631+0000","last_became_peered":"2024-09-13T14:45:19.808631+0000","last_unstale":"2024-09-13T14:49:53.128813+0000","last_undegraded":"2024-09-13T14:49:53.128813+0000","last_fullsized":"2024-09-13T14:49:53.128813+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:38.822245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1134,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.209551+0000","last_change":"2024-09-13T14:45:19.804229+0000","last_active":"2024-09-13T14:49:36.209551+0000","last_peered":"2024-09-13T14:49:36.209551+0000","last_clean":"2024-09-13T14:49:36.209551+0000","last_became_active":"2024-09-13T14:45:19.803929+0000","last_became_peered":"2024-09-13T14:45:19.803929+0000","last_unstale":"2024-09-13T14:49:36.209551+0000","last_undegraded":"2024-09-13T14:49:36.209551+0000","last_fullsized":"2024-09-13T14:49:36.209551+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:33:53.675568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"261'100","reported_seq":63,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:36.210102+0000","last_change":"2024-09-13T14:44:42.654506+0000","last_active":"2024-09-13T14:49:36.210102+0000","last_peered":"2024-09-13T14:49:36.210102+0000","last_clean":"2024-09-13T14:49:36.210102+0000","last_became_active":"2024-09-13T14:43:14.563451+0000","last_became_peered":"2024-09-13T14:43:14.563451+0000","last_unstale":"2024-09-13T14:49:36.210102+0000","last_undegraded":"2024-09-13T14:49:36.210102+0000","last_fullsized":"2024-09-13T14:49:36.210102+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:42.654455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:42.654455+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:57:32.251597+0000","scrub_duration":0.001494097,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":1128,"reported_epoch":1171,"state":"active+clean","last_fresh":"2024-09-13T14:49:35.603353+0000","last_change":"2024-09-13T14:29:42.541093+0000","last_active":"2024-09-13T14:49:35.603353+0000","last_peered":"2024-09-13T14:49:35.603353+0000","last_clean":"2024-09-13T14:49:35.603353+0000","last_became_active":"2024-09-13T14:29:42.540671+0000","last_became_peered":"2024-09-13T14:29:42.540671+0000","last_unstale":"2024-09-13T14:49:35.603353+0000","last_undegraded":"2024-09-13T14:49:35.603353+0000","last_fullsized":"2024-09-13T14:49:35.603353+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:45.722376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1666,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.134511+0000","last_change":"2024-09-13T14:43:07.518679+0000","last_active":"2024-09-13T14:49:53.134511+0000","last_peered":"2024-09-13T14:49:53.134511+0000","last_clean":"2024-09-13T14:49:53.134511+0000","last_became_active":"2024-09-13T14:43:07.518333+0000","last_became_peered":"2024-09-13T14:43:07.518333+0000","last_unstale":"2024-09-13T14:49:53.134511+0000","last_undegraded":"2024-09-13T14:49:53.134511+0000","last_fullsized":"2024-09-13T14:49:53.134511+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:20.616274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"261'100","reported_seq":435,"reported_epoch":1172,"state":"active+clean","last_fresh":"2024-09-13T14:49:53.130084+0000","last_change":"2024-09-13T14:45:19.809268+0000","last_active":"2024-09-13T14:49:53.130084+0000","last_peered":"2024-09-13T14:49:53.130084+0000","last_clean":"2024-09-13T14:49:53.130084+0000","last_became_active":"2024-09-13T14:45:19.808772+0000","last_became_peered":"2024-09-13T14:45:19.808772+0000","last_unstale":"2024-09-13T14:49:53.130084+0000","last_undegraded":"2024-09-13T14:49:53.130084+0000","last_fullsized":"2024-09-13T14:49:53.130084+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:14.654603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:14.654603+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:40.964553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":3873046528,"num_objects":59098,"num_object_clones":0,"num_object_copies":177294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59098,"num_write_kb":3782272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11515527168,"data_stored":11515527168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59098,"ondisk_log_size":59098,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430601,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1463804,"kb_used_data":1149812,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":92281348,"statfs":{"total":95995035648,"available":94496100352,"internally_reserved":0,"allocated":1177407488,"data_stored":1173786173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626830,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2088688,"kb_used_data":1494632,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":91656464,"statfs":{"total":95995035648,"available":93856219136,"internally_reserved":0,"allocated":1530503168,"data_stored":1526893478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823059,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2155628,"kb_used_data":1849956,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91589524,"statfs":{"total":95995035648,"available":93787672576,"internally_reserved":0,"allocated":1894354944,"data_stored":1890745712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019287,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1454068,"kb_used_data":1152556,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":92291084,"statfs":{"total":95995035648,"available":94506070016,"internally_reserved":0,"allocated":1180217344,"data_stored":1176603607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059592945,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1682208,"kb_used_data":1431704,"kb_used_omap":26,"kb_used_meta":250469,"kb_avail":92062944,"statfs":{"total":95995035648,"available":94272454656,"internally_reserved":0,"allocated":1466064896,"data_stored":1462466859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":256480416},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":591,"seq":2538325672183,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1809448,"kb_used_data":1374176,"kb_used_omap":41,"kb_used_meta":435222,"kb_avail":91935704,"statfs":{"total":95995035648,"available":94142160896,"internally_reserved":0,"allocated":1407156224,"data_stored":1403551061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42651,"internal_metadata":445667685},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":557,"seq":2392296784126,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2039456,"kb_used_data":1624088,"kb_used_omap":32,"kb_used_meta":415327,"kb_avail":91705696,"statfs":{"total":95995035648,"available":93906632704,"internally_reserved":0,"allocated":1663066112,"data_stored":1659468020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33752,"internal_metadata":425294888},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,2,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":524,"seq":2250562863365,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1816420,"kb_used_data":1442204,"kb_used_omap":41,"kb_used_meta":374166,"kb_avail":91928732,"statfs":{"total":95995035648,"available":94135021568,"internally_reserved":0,"allocated":1476816896,"data_stored":1473215289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42694,"internal_metadata":383146298},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1475674112,"data_stored":1475674112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1680736256,"data_stored":1680736256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1423179776,"data_stored":1423179776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1486815232,"data_stored":1486815232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1190068224,"data_stored":1190068224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1897725952,"data_stored":1897725952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1174601728,"data_stored":1174601728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1186725888,"data_stored":1186725888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:50:15.188 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T14:50:15.188 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:50:15.188 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-13T14:50:15.189 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:50:15.189 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:50:15.189 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-09-13T14:50:15.197 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:15.221 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 36 16 67836 67820 117.725 81.4375 0.0119351 0.00848937 2024-09-13T14:50:15.231 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:15.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:15.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:15.333 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:15.360 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:15.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:15.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:15.462 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:15.488 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:15.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:15.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:15.590 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:15.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:15.622 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:15.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:15.676 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-13T14:50:15.723 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:15.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:15.766 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:15.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:15.790 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:15 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3330083160' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:15.790 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:15 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:15.790 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:15 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3330083160' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:15.790 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:15 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:15.867 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:15.901 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:50:15.903 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:15.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:15.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:16.005 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:16.043 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:16.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:16.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:16.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:15 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3330083160' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:16.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:15 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:16.145 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:16.170 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:16.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:16.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:16.221 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 37 16 69055 69039 116.602 76.1875 0.0163527 0.00857114 2024-09-13T14:50:16.272 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:16.307 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:16.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:16.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:16.408 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:16.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:16.441 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:16.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:16.542 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:16.572 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:16.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:16.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:16.674 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:16.711 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:16.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:16.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:16.813 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:16.853 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:16.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:16.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:16.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:16 smithi151 ceph-mon[108966]: pgmap v1703: 119 pgs: 119 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-13T14:50:16.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:16 smithi151 ceph-mon[108966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:16.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:16 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:50:16.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:16 smithi151 ceph-mon[108966]: osdmap e1173: 8 total, 8 up, 8 in 2024-09-13T14:50:16.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:16.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:16 smithi151 ceph-mon[104450]: pgmap v1703: 119 pgs: 119 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-13T14:50:16.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:16 smithi151 ceph-mon[104450]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:16.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:16 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:50:16.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:16 smithi151 ceph-mon[104450]: osdmap e1173: 8 total, 8 up, 8 in 2024-09-13T14:50:16.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:16.955 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:16.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:16.996 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:16.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:17.004 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:16 smithi184 ceph-mon[92527]: pgmap v1703: 119 pgs: 119 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-13T14:50:17.004 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:16 smithi184 ceph-mon[92527]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:17.004 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:16 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:50:17.004 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:16 smithi184 ceph-mon[92527]: osdmap e1173: 8 total, 8 up, 8 in 2024-09-13T14:50:17.004 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:17.097 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:17.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:17.132 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:17.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:17.222 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 38 16 70506 70490 115.92 90.6875 0.0061376 0.008622 2024-09-13T14:50:17.233 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:17.267 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:17.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:17.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:17.369 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:17.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:17.398 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:17.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:17.499 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:17.502 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:17.552 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:17.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:17.559 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:17.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:17.660 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:17.686 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:17.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:17.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:17.788 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:17.818 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:17.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:17.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:17.919 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:17.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:17.955 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:17.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:18.056 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:18.077 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:18.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:18.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:18.179 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:18.219 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:18.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:18.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:18.221 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 39 16 71807 71791 115.032 81.3125 0.0192517 0.00868815 2024-09-13T14:50:18.321 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:18.357 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:18.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:18.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:18.459 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:18.497 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:18.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:18.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:18.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:50:18.599 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:18.645 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:18.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:18.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:18.747 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:18.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:18.769 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:18.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:18.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:18 smithi151 ceph-mon[104450]: pgmap v1705: 119 pgs: 119 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-13T14:50:18.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:18 smithi151 ceph-mon[108966]: pgmap v1705: 119 pgs: 119 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-13T14:50:18.870 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:18.902 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:18.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:18.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:19.005 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:19.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:19.036 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:19.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:19.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:18 smithi184 ceph-mon[92527]: pgmap v1705: 119 pgs: 119 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-13T14:50:19.137 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:19.174 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:19.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:19.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:19.222 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:2024-09-13T14:50:19.221743+0000 min lat: 0.00100112 max lat: 0.247171 avg lat: 0.00875278 2024-09-13T14:50:19.222 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:50:19.222 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 40 16 73080 73064 114.145 79.5625 0.0037758 0.00875278 2024-09-13T14:50:19.276 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:19.305 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:19.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:19.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:19.407 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:19.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:19.438 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:19.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:19.539 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:19.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:19.577 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:19.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:19.678 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:19.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:19.730 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:19.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:19.831 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:19.871 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:19.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:19.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:20.029 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:20.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:20.071 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:20.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:20.172 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:20.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:20.206 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:20.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:20.222 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 41 16 74326 74310 113.26 77.875 0.00732982 0.00882438 2024-09-13T14:50:20.307 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:20.352 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:20.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:20.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:20.454 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:20.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:20.482 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:20.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:20.583 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:20.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:50:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:50:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:50:20.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:20.623 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:20.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:20.724 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:20.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:20.762 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:20.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:20.863 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:20.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:20.888 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:20.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:20.967 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:20 smithi151 ceph-mon[104450]: pgmap v1706: 119 pgs: 119 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-13T14:50:20.967 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:20 smithi151 ceph-mon[108966]: pgmap v1706: 119 pgs: 119 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-13T14:50:20.989 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:21.025 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:21.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:21.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:21.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:20 smithi184 ceph-mon[92527]: pgmap v1706: 119 pgs: 119 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-13T14:50:21.127 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:21.161 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:21.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:21.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:21.222 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 42 16 75586 75570 112.439 78.75 0.0131701 0.00888895 2024-09-13T14:50:21.263 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:21.297 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:21.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:21.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:21.398 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:21.440 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:21.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:21.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:21.450 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:50:21.450 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:50:21.542 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:21.573 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:21.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:21.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:21.675 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:21.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:21.719 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:21.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:21.820 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:21.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:21.857 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:21.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:21.958 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:21.993 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:21.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:21.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:22.095 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:22.131 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:22.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:22.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:22.222 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 43 16 76856 76840 111.67 79.375 0.0139113 0.00894975 2024-09-13T14:50:22.233 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:22.270 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:22.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:22.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:22.372 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:22.400 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:22.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:22.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:22.502 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:22.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:22.528 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:22.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:22.629 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:22.667 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:22.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:22.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:22.769 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:22.804 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:22.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:22.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:22.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:22 smithi151 ceph-mon[104450]: pgmap v1707: 119 pgs: 119 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-13T14:50:22.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:22 smithi151 ceph-mon[108966]: pgmap v1707: 119 pgs: 119 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-13T14:50:22.905 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:22.942 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:22.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:22.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:23.044 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:23.071 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:23.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:23.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:22 smithi184 ceph-mon[92527]: pgmap v1707: 119 pgs: 119 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-13T14:50:23.173 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:23.198 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:23.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:23.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:23.212 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T14:50:23.222 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 44 16 78165 78149 110.991 81.8125 0.0132523 0.00900432 2024-09-13T14:50:23.300 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:23.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:23.348 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:23.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:23.449 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:23.478 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:23.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:23.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:23.580 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:23.606 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:23.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:23.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:23.708 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:23.744 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:23.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:23.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:23.807 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:23.846 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:23.896 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:23.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:23.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:23.998 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:24.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:24.038 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:24.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:24.139 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:24.178 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:24.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:24.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:24.222 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 45 16 79341 79325 110.157 73.5 0.0124064 0.00907278 2024-09-13T14:50:24.281 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:24.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:24.329 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:24.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:24.430 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:24.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:24.463 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:24.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:24.564 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:24.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:24.609 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:24.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:24.710 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:24.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:24.748 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:24.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:24.849 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:24.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:24 smithi151 ceph-mon[104450]: pgmap v1708: 119 pgs: 119 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 82 MiB/s wr, 1.30k op/s 2024-09-13T14:50:24.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:24 smithi151 ceph-mon[108966]: pgmap v1708: 119 pgs: 119 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 82 MiB/s wr, 1.30k op/s 2024-09-13T14:50:24.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:50:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:50:24] "GET /metrics HTTP/1.1" 200 36640 "" "Prometheus/2.51.0" 2024-09-13T14:50:24.877 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:24.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:24.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:24.939 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:50:24.939 INFO:teuthology.orchestra.run.smithi151.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T14:50:24.939 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:50:24.970 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:50:24.970 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:50:24.970 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:50:24.971 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:50:24.979 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:50:24.979 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:50:24.979 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:25.004 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:25.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:25.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:25.022 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:50:25.022 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:50:25.051 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:50:25.051 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:50:25.084 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:50:25.084 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:50:25.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:24 smithi184 ceph-mon[92527]: pgmap v1708: 119 pgs: 119 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 82 MiB/s wr, 1.30k op/s 2024-09-13T14:50:25.107 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:25.120 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:50:25.120 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:50:25.144 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:25.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:25.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:25.152 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:50:25.153 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:50:25.222 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 46 16 80542 80526 109.394 75.0625 0.0124391 0.00913598 2024-09-13T14:50:25.246 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:25.283 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:25.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:25.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:25.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:25.426 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:25.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:25.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:25.528 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:25.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:25.558 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:25.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:25.659 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:25.682 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:25.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:25.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:25.785 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:25.821 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:25.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:25.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:25.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:25 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2461089162' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:25.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:25 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2461089162' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:25.923 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:25.955 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:25.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:25.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:26.057 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:26.094 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:26.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:26.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:26.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:25 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2461089162' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:26.196 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:26.222 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 47 16 81725 81709 108.64 73.9375 0.00375434 0.00919924 2024-09-13T14:50:26.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:26.224 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:26.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:26.325 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:26.355 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:26.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:26.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:26.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:26.496 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:26.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:26.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:26.598 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:26.635 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:26.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:26.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:26.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:26.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:26.780 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:26.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:26.881 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:26.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:26.902 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:26.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:27.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:27.041 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:27.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:27.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:27.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:26 smithi184 ceph-mon[92527]: pgmap v1709: 119 pgs: 119 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-13T14:50:27.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:26 smithi151 ceph-mon[108966]: pgmap v1709: 119 pgs: 119 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-13T14:50:27.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:26 smithi151 ceph-mon[104450]: pgmap v1709: 119 pgs: 119 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-13T14:50:27.143 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:27.176 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:27.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:27.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:27.222 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 48 16 82853 82837 107.845 70.5 0.00532607 0.00926751 2024-09-13T14:50:27.278 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:27.305 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:27.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:27.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:27.407 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:27.442 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:27.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:27.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:27.544 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:27.578 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:27.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:27.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:27.679 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:27.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:27.720 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:27.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:27.821 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:27.855 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:27.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:27.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:27.957 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:27.999 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:28.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:28.102 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:28.135 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:28.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:28.223 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 49 16 84051 84035 107.172 74.875 0.0131346 0.00932538 2024-09-13T14:50:28.237 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:28.273 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:28.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:28.374 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:28.410 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:28.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:28.512 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:28.539 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:28.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:28.641 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:28.682 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:28.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:28.784 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:28.821 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:28.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:28.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:28 smithi151 ceph-mon[104450]: pgmap v1710: 119 pgs: 119 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-13T14:50:28.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:28 smithi151 ceph-mon[108966]: pgmap v1710: 119 pgs: 119 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-13T14:50:28.924 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:28.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.964 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:28.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:29.065 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:29.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:28 smithi184 ceph-mon[92527]: pgmap v1710: 119 pgs: 119 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-13T14:50:29.105 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:29.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:29.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:29.207 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:29.224 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 50 16 85294 85278 106.581 77.6875 0.00568314 0.00937615 2024-09-13T14:50:29.246 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:29.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:29.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:29.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:29.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:29.392 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:29.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:29.428 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:29.494 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:29.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:29.526 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:29.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:29.627 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:29.659 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:29.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:29.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:29.761 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:29.796 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:29.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:29.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:29.898 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:29.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:29.922 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:29.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:30.023 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:30.080 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:30.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:30.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:30.181 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:30.210 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:30.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:30.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:30.223 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 51 16 86481 86465 105.946 74.1875 0.0140284 0.00943299 2024-09-13T14:50:30.312 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:30.347 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:30.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:30.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:30.449 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:30.486 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:30.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:30.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:30.588 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:30.605 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:50:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:50:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:50:30.623 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:30.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:30.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:30.725 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:30.768 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:30.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:30.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:30.870 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:30.905 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:30.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:30.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:30.913 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:50:30.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:30 smithi151 ceph-mon[104450]: pgmap v1711: 119 pgs: 119 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-13T14:50:30.950 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:30 smithi151 ceph-mon[108966]: pgmap v1711: 119 pgs: 119 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-13T14:50:31.007 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:31.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:31.056 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:31.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:31.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:30 smithi184 ceph-mon[92527]: pgmap v1711: 119 pgs: 119 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-13T14:50:31.157 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:31.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:31.218 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:31.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:31.224 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 52 16 87614 87598 105.269 70.8125 0.00803676 0.00949357 2024-09-13T14:50:31.319 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:31.348 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:31.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:31.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:31.450 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:31.490 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:31.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:31.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:31.592 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:31.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:31.630 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:31.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:31.715 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-13T14:50:31.731 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:31.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:31.775 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:31.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:31.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:31.919 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:31.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:31.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:32.021 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:32.044 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:32.044 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:31 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1992760961' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:50:32.044 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:32.044 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:31 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1992760961' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:50:32.047 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:32.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:32.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:32.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:32.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:31 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1992760961' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:50:32.149 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:32.183 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:32.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:32.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:32.224 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 53 16 88713 88697 104.579 68.6875 0.0204919 0.009557 2024-09-13T14:50:32.284 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:32.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:32.324 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:32.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:32.425 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:32.466 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:32.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:32.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:32.568 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:32.598 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:32.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:32.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:32.700 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:32.738 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:32.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:32.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:32.840 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:32.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:32.874 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:32.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:32.975 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:32.999 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:32 smithi151 ceph-mon[104450]: pgmap v1712: 119 pgs: 119 active+clean; 5.2 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-13T14:50:32.999 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:32 smithi151 ceph-mon[104450]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:32.999 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:32 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1992760961' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:50:33.000 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:32 smithi151 ceph-mon[104450]: osdmap e1174: 8 total, 8 up, 8 in 2024-09-13T14:50:33.000 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:32 smithi151 ceph-mon[108966]: pgmap v1712: 119 pgs: 119 active+clean; 5.2 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-13T14:50:33.000 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:32 smithi151 ceph-mon[108966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:33.001 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:32 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1992760961' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:50:33.001 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:32 smithi151 ceph-mon[108966]: osdmap e1174: 8 total, 8 up, 8 in 2024-09-13T14:50:33.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:33.015 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:33.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:33.046 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:50:33.046 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-13T14:50:33.046 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-13T14:50:33.046 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-13T14:50:33.047 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:50:33.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:32 smithi184 ceph-mon[92527]: pgmap v1712: 119 pgs: 119 active+clean; 5.2 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-13T14:50:33.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:32 smithi184 ceph-mon[92527]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:33.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:32 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1992760961' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:50:33.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:32 smithi184 ceph-mon[92527]: osdmap e1174: 8 total, 8 up, 8 in 2024-09-13T14:50:33.117 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:33.158 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:33.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:33.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:33.224 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 54 16 89875 89859 103.987 72.625 0.0188007 0.00961189 2024-09-13T14:50:33.260 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:33.302 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:33.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:33.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:33.404 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:33.438 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:33.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:33.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:33.539 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:33.576 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:33.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:33.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:33.678 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:33.706 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:33.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:33.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:33.808 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:33.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:33.849 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:33.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:33.950 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:33.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:33.982 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:33.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:34.083 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:34.128 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:34.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:34.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:34.224 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 55 16 91016 91000 103.393 71.3125 0.00907424 0.00966648 2024-09-13T14:50:34.230 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:34.270 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:34.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:34.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:34.373 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:34.407 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:50:34.408 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:50:34.408 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-13T14:50:34.408 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:50:34.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:34.410 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:34.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:34.511 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:34.543 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:34.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:34.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:34.645 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:34.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:34.669 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:34.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:34.700 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:50:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:50:34] "GET /metrics HTTP/1.1" 200 36647 "" "Prometheus/2.51.0" 2024-09-13T14:50:34.770 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:34.781 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:50:34.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:34.797 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:34.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:34.899 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:34.926 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:34.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:34.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:35.028 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:35.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:35.050 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:35.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:35.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:34 smithi184 ceph-mon[92527]: pgmap v1714: 119 pgs: 119 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-13T14:50:35.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:34 smithi151 ceph-mon[108966]: pgmap v1714: 119 pgs: 119 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-13T14:50:35.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:34 smithi151 ceph-mon[104450]: pgmap v1714: 119 pgs: 119 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-13T14:50:35.151 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:35.182 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:35.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:35.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:35.224 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 56 16 92003 91987 102.648 61.6875 0.0144613 0.00973752 2024-09-13T14:50:35.284 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:35.306 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:35.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:35.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:35.407 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:35.438 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:35.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:35.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:35.540 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:35.574 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:35.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:35.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:35.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:35.700 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:35.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:35.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:35.802 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:35.829 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:35.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:35.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:35.930 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:35.966 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:35.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:35.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:36.067 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:36.089 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:36.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:36.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:36.191 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:36.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:36.226 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:36.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:36.227 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 57 16 92806 92790 101.725 50.1875 0.0168082 0.00982587 2024-09-13T14:50:36.328 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:36.363 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:36.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:36.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:36.466 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:36.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:36.506 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:36.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:36.607 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:36.621 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:36.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:36.638 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:36.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:36.665 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T14:50:36.739 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:36.784 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:36.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:36.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:36.886 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:36.927 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:36.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:36.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:37.019 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:36 smithi151 ceph-mon[104450]: pgmap v1715: 119 pgs: 119 active+clean; 5.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-13T14:50:37.019 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:36 smithi151 ceph-mon[108966]: pgmap v1715: 119 pgs: 119 active+clean; 5.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-13T14:50:37.029 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:37.055 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:37.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:37.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:37.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:36 smithi184 ceph-mon[92527]: pgmap v1715: 119 pgs: 119 active+clean; 5.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-13T14:50:37.157 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:37.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:37.190 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:37.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:37.226 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 58 16 93820 93804 101.064 63.375 0.00691677 0.0098899 2024-09-13T14:50:37.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:37.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:37.326 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:37.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:37.427 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:37.459 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:37.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:37.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:37.561 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:37.600 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:37.621 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:37.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:37.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:37.723 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:37.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:37.774 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:37.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:37.875 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:37.911 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:37.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:37.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:38.013 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:38.041 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:38.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:38.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:38.142 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:38.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:38.165 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:38.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:38.226 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 59 16 94853 94837 100.446 64.5625 0.0232379 0.0099502 2024-09-13T14:50:38.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:38.305 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:38.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:38.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:38.407 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:38.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:38.440 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:38.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:38.541 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:38.584 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:38.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:38.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:38.686 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:38.706 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:38.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:38.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:38.808 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:38.842 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:38.864 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:38.871 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:38.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:38.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:38.974 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:39.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:39.030 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:39.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:39.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:38 smithi184 ceph-mon[92527]: pgmap v1716: 119 pgs: 119 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-13T14:50:39.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:38 smithi151 ceph-mon[104450]: pgmap v1716: 119 pgs: 119 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-13T14:50:39.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:38 smithi151 ceph-mon[108966]: pgmap v1716: 119 pgs: 119 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-13T14:50:39.131 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:39.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:39.154 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:39.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:39.226 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:2024-09-13T14:50:39.225957+0000 min lat: 0.00100112 max lat: 0.247171 avg lat: 0.0100263 2024-09-13T14:50:39.226 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:50:39.226 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 60 16 95733 95717 99.688 55 0.0238049 0.0100263 2024-09-13T14:50:39.255 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:39.294 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:39.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:39.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:39.396 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:39.429 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:39.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:39.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:39.531 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:39.553 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:39.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:39.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:39.655 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:39.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:39.699 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:39.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:39.800 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:39.830 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:39.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:39.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:39.932 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:39.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:39.967 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:39.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:40.068 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:40.098 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:40.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:40.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:40.200 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:40.226 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 61 15 96676 96661 99.021 59 0.00452615 0.0100938 2024-09-13T14:50:40.230 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:40.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:40.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:40.332 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:40.359 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:40.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:40.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:40.461 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:40.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:40.493 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:40.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:40.594 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:40.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:50:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:50:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:50:40.636 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:40.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:40.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:40.738 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:40.765 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:40.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:40.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:40.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:40 smithi151 ceph-mon[104450]: pgmap v1717: 119 pgs: 119 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T14:50:40.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:40 smithi151 ceph-mon[108966]: pgmap v1717: 119 pgs: 119 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T14:50:40.867 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:40.904 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:40.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:40.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:41.005 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:41.046 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:41.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:41.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:40 smithi184 ceph-mon[92527]: pgmap v1717: 119 pgs: 119 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T14:50:41.147 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:41.177 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:41.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:41.228 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 62 16 97802 97786 98.5552 70.3125 0.0134723 0.0101416 2024-09-13T14:50:41.278 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:41.308 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:41.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:41.410 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:41.443 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:41.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:41.545 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:41.573 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:41.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:41.675 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:41.712 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:41.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:41.814 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:41.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.844 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:41.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:41.945 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:41.986 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:41.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:42.088 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:42.113 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:42.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:42.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:42.215 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:42.229 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 63 16 98849 98833 98.0275 65.4375 0.0109496 0.0101954 2024-09-13T14:50:42.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:42.262 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:42.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:42.363 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:42.406 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:42.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:42.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:42.508 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:42.545 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:42.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:42.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:42.647 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:42.687 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:42.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:42.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:42.789 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:42.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:42.821 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:42.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:42.922 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:42.929 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:42 smithi151 ceph-mon[104450]: pgmap v1718: 119 pgs: 119 active+clean; 5.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-13T14:50:42.929 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:42 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:50:42.930 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:42 smithi151 ceph-mon[108966]: pgmap v1718: 119 pgs: 119 active+clean; 5.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-13T14:50:42.930 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:42 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:50:42.966 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:42.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:42.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:43.068 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:43.101 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:42 smithi184 ceph-mon[92527]: pgmap v1718: 119 pgs: 119 active+clean; 5.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-13T14:50:43.101 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:42 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:50:43.110 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:43.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:43.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:43.212 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:43.229 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 64 16 99794 99778 97.4187 59.0625 0.0106315 0.0102589 2024-09-13T14:50:43.238 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:43.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:43.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:43.257 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:50:43.257 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:50:43.257 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:50:43.257 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:50:43.258 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:50:43.258 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:50:43.258 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:50:43.258 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:50:43.258 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:50:43.258 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:50:43.258 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:50:43.258 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:50:43.259 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:50:43.259 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:50:43.259 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:50:43.259 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:50:43.340 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:43.375 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:43.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:43.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:43.477 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:43.530 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:43.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:43.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:43.632 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:43.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:43.655 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:43.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:43.756 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:43.795 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:43.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:43.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:43.897 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:43.918 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:43.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:43.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:44.020 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:44.033 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:43 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2009626497' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:44.062 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:44.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:44.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:44.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:43 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2009626497' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:44.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:43 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2009626497' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:44.163 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:44.192 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:44.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:44.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:44.229 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 65 16 100757 100741 96.8459 60.1875 0.00879505 0.0103213 2024-09-13T14:50:44.294 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:44.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:44.324 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:44.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:44.425 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:44.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:44.465 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:44.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:44.566 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:44.605 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:44.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:44.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:44.707 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:44.740 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:44.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:44.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:44.841 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:44.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:44 smithi151 ceph-mon[104450]: pgmap v1719: 119 pgs: 119 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-13T14:50:44.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:44 smithi151 ceph-mon[108966]: pgmap v1719: 119 pgs: 119 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-13T14:50:44.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:50:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:50:44] "GET /metrics HTTP/1.1" 200 36647 "" "Prometheus/2.51.0" 2024-09-13T14:50:44.878 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:44.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:44.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:44.980 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:45.008 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:44 smithi184 ceph-mon[92527]: pgmap v1719: 119 pgs: 119 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-13T14:50:45.016 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:45.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:45.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:45.118 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:45.146 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:50:45.148 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T14:50:45.152 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:45.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:45.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:45.229 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 66 16 101748 101732 96.3169 61.9375 0.0220619 0.0103771 2024-09-13T14:50:45.253 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:45.294 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:45.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:45.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:45.396 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:45.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:45.419 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:45.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:45.520 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:45.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:45.550 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:45.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:45.651 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:45.675 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:45.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:45.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:45.777 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:45.808 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:45.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:45.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:45.910 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:45.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:45.946 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:45.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:46.046 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:46.078 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:46.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:46.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:46.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:45 smithi184 ceph-mon[92527]: from='client.55252 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:50:46.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:45 smithi151 ceph-mon[104450]: from='client.55252 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:50:46.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:45 smithi151 ceph-mon[108966]: from='client.55252 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:50:46.180 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:46.222 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:46.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:46.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:46.229 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 67 16 102632 102616 95.704 55.25 0.0217652 0.0104444 2024-09-13T14:50:46.323 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:46.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:46.365 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:46.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:46.466 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:46.499 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:46.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:46.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:46.601 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:46.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:46.642 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:46.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:46.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:46.783 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:46.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:46.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:46.885 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:46.911 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:46.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:46.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:47.008 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:46 smithi151 ceph-mon[108966]: pgmap v1720: 119 pgs: 119 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 63 MiB/s wr, 1k op/s 2024-09-13T14:50:47.008 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:47.008 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:46 smithi151 ceph-mon[104450]: pgmap v1720: 119 pgs: 119 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 63 MiB/s wr, 1k op/s 2024-09-13T14:50:47.008 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:47.013 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:47.061 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:47.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:47.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:46 smithi184 ceph-mon[92527]: pgmap v1720: 119 pgs: 119 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 63 MiB/s wr, 1k op/s 2024-09-13T14:50:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:47.163 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:47.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:47.211 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:47.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:47.230 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 68 16 103683 103667 95.2625 65.6875 0.00603868 0.0104925 2024-09-13T14:50:47.312 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:47.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:47.349 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:47.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:47.450 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:47.483 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1719,"stamp":"2024-09-13T14:50:43.564242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6440818637,"num_objects":98296,"num_object_clones":0,"num_object_copies":294888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98333,"num_write_kb":6289311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":327718,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101228,"ondisk_log_size":101228,"up":357,"acting":357,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":358,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":22264664,"kb_used_data":19172248,"kb_used_omap":149,"kb_used_meta":3092202,"kb_avail":727696552,"statfs":{"total":767960285184,"available":745161269248,"internally_reserved":0,"allocated":19632381952,"data_stored":19603462743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":153401,"internal_metadata":3166415047},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,3,12,5,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":902627328,"num_objects":13773,"num_object_clones":0,"num_object_copies":41319,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13773,"num_write_kb":881472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.842954"},"pg_stats":[{"pgid":"1.1d","version":"261'100","reported_seq":119,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.743916+0000","last_change":"2024-09-13T14:45:19.807356+0000","last_active":"2024-09-13T14:50:31.743916+0000","last_peered":"2024-09-13T14:50:31.743916+0000","last_clean":"2024-09-13T14:50:31.743916+0000","last_became_active":"2024-09-13T14:45:19.807081+0000","last_became_peered":"2024-09-13T14:45:19.807081+0000","last_unstale":"2024-09-13T14:50:31.743916+0000","last_undegraded":"2024-09-13T14:50:31.743916+0000","last_fullsized":"2024-09-13T14:50:31.743916+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:17.988345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:17.988345+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:41:05.216335+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":1725,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.737630+0000","last_change":"2024-09-13T14:29:42.541263+0000","last_active":"2024-09-13T14:50:31.737630+0000","last_peered":"2024-09-13T14:50:31.737630+0000","last_clean":"2024-09-13T14:50:31.737630+0000","last_became_active":"2024-09-13T14:29:42.540873+0000","last_became_peered":"2024-09-13T14:29:42.540873+0000","last_unstale":"2024-09-13T14:50:31.737630+0000","last_undegraded":"2024-09-13T14:50:31.737630+0000","last_fullsized":"2024-09-13T14:50:31.737630+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:20.611206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1134,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.680574+0000","last_change":"2024-09-13T14:43:07.516715+0000","last_active":"2024-09-13T14:50:15.680574+0000","last_peered":"2024-09-13T14:50:15.680574+0000","last_clean":"2024-09-13T14:50:15.680574+0000","last_became_active":"2024-09-13T14:43:07.516403+0000","last_became_peered":"2024-09-13T14:43:07.516403+0000","last_unstale":"2024-09-13T14:50:15.680574+0000","last_undegraded":"2024-09-13T14:50:15.680574+0000","last_fullsized":"2024-09-13T14:50:15.680574+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:56.469690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":64,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.736206+0000","last_change":"2024-09-13T14:45:19.810203+0000","last_active":"2024-09-13T14:50:31.736206+0000","last_peered":"2024-09-13T14:50:31.736206+0000","last_clean":"2024-09-13T14:50:31.736206+0000","last_became_active":"2024-09-13T14:45:19.809693+0000","last_became_peered":"2024-09-13T14:45:19.809693+0000","last_unstale":"2024-09-13T14:50:31.736206+0000","last_undegraded":"2024-09-13T14:50:31.736206+0000","last_fullsized":"2024-09-13T14:50:31.736206+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:20.219673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":64,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.736652+0000","last_change":"2024-09-13T14:45:19.807991+0000","last_active":"2024-09-13T14:50:31.736652+0000","last_peered":"2024-09-13T14:50:31.736652+0000","last_clean":"2024-09-13T14:50:31.736652+0000","last_became_active":"2024-09-13T14:45:19.807662+0000","last_became_peered":"2024-09-13T14:45:19.807662+0000","last_unstale":"2024-09-13T14:50:31.736652+0000","last_undegraded":"2024-09-13T14:50:31.736652+0000","last_fullsized":"2024-09-13T14:50:31.736652+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:29.306108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"261'100","reported_seq":1245,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.677825+0000","last_change":"2024-09-13T14:44:54.259562+0000","last_active":"2024-09-13T14:50:15.677825+0000","last_peered":"2024-09-13T14:50:15.677825+0000","last_clean":"2024-09-13T14:50:15.677825+0000","last_became_active":"2024-09-13T14:43:14.563717+0000","last_became_peered":"2024-09-13T14:43:14.563717+0000","last_unstale":"2024-09-13T14:50:15.677825+0000","last_undegraded":"2024-09-13T14:50:15.677825+0000","last_fullsized":"2024-09-13T14:50:15.677825+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_clean_scrub_stamp":"2024-09-13T14:44:54.259482+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:43:52.729109+0000","scrub_duration":0.015265569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":1102,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.674613+0000","last_change":"2024-09-13T14:19:25.873514+0000","last_active":"2024-09-13T14:50:15.674613+0000","last_peered":"2024-09-13T14:50:15.674613+0000","last_clean":"2024-09-13T14:50:15.674613+0000","last_became_active":"2024-09-13T14:19:25.873170+0000","last_became_peered":"2024-09-13T14:19:25.873170+0000","last_unstale":"2024-09-13T14:50:15.674613+0000","last_undegraded":"2024-09-13T14:50:15.674613+0000","last_fullsized":"2024-09-13T14:50:15.674613+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:48.298530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1131,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.681269+0000","last_change":"2024-09-13T14:30:15.580011+0000","last_active":"2024-09-13T14:50:15.681269+0000","last_peered":"2024-09-13T14:50:15.681269+0000","last_clean":"2024-09-13T14:50:15.681269+0000","last_became_active":"2024-09-13T14:30:15.579610+0000","last_became_peered":"2024-09-13T14:30:15.579610+0000","last_unstale":"2024-09-13T14:50:15.681269+0000","last_undegraded":"2024-09-13T14:50:15.681269+0000","last_fullsized":"2024-09-13T14:50:15.681269+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:11:02.733771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"261'100","reported_seq":81,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.736362+0000","last_change":"2024-09-13T14:45:35.947332+0000","last_active":"2024-09-13T14:50:31.736362+0000","last_peered":"2024-09-13T14:50:31.736362+0000","last_clean":"2024-09-13T14:50:31.736362+0000","last_became_active":"2024-09-13T14:45:19.810775+0000","last_became_peered":"2024-09-13T14:45:19.810775+0000","last_unstale":"2024-09-13T14:50:31.736362+0000","last_undegraded":"2024-09-13T14:50:31.736362+0000","last_fullsized":"2024-09-13T14:50:31.736362+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:35.947265+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:43:41.569795+0000","last_clean_scrub_stamp":"2024-09-13T14:45:35.947265+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:00.606370+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":1147,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.674510+0000","last_change":"2024-09-13T14:43:07.536116+0000","last_active":"2024-09-13T14:50:15.674510+0000","last_peered":"2024-09-13T14:50:15.674510+0000","last_clean":"2024-09-13T14:50:15.674510+0000","last_became_active":"2024-09-13T14:43:07.524112+0000","last_became_peered":"2024-09-13T14:43:07.524112+0000","last_unstale":"2024-09-13T14:50:15.674510+0000","last_undegraded":"2024-09-13T14:50:15.674510+0000","last_fullsized":"2024-09-13T14:50:15.674510+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:36.947060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1144,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.681468+0000","last_change":"2024-09-13T14:43:07.530236+0000","last_active":"2024-09-13T14:50:15.681468+0000","last_peered":"2024-09-13T14:50:15.681468+0000","last_clean":"2024-09-13T14:50:15.681468+0000","last_became_active":"2024-09-13T14:43:07.529967+0000","last_became_peered":"2024-09-13T14:43:07.529967+0000","last_unstale":"2024-09-13T14:50:15.681468+0000","last_undegraded":"2024-09-13T14:50:15.681468+0000","last_fullsized":"2024-09-13T14:50:15.681468+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:27.276550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":1709,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.736882+0000","last_change":"2024-09-13T14:28:35.612469+0000","last_active":"2024-09-13T14:50:31.736882+0000","last_peered":"2024-09-13T14:50:31.736882+0000","last_clean":"2024-09-13T14:50:31.736882+0000","last_became_active":"2024-09-13T14:28:35.612093+0000","last_became_peered":"2024-09-13T14:28:35.612093+0000","last_unstale":"2024-09-13T14:50:31.736882+0000","last_undegraded":"2024-09-13T14:50:31.736882+0000","last_fullsized":"2024-09-13T14:50:31.736882+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:51.022663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1732,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.731273+0000","last_change":"2024-09-13T14:45:19.808811+0000","last_active":"2024-09-13T14:50:31.731273+0000","last_peered":"2024-09-13T14:50:31.731273+0000","last_clean":"2024-09-13T14:50:31.731273+0000","last_became_active":"2024-09-13T14:45:19.808471+0000","last_became_peered":"2024-09-13T14:45:19.808471+0000","last_unstale":"2024-09-13T14:50:31.731273+0000","last_undegraded":"2024-09-13T14:50:31.731273+0000","last_fullsized":"2024-09-13T14:50:31.731273+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:53.638161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"261'100","reported_seq":64,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.736744+0000","last_change":"2024-09-13T14:45:19.813412+0000","last_active":"2024-09-13T14:50:31.736744+0000","last_peered":"2024-09-13T14:50:31.736744+0000","last_clean":"2024-09-13T14:50:31.736744+0000","last_became_active":"2024-09-13T14:45:19.812585+0000","last_became_peered":"2024-09-13T14:45:19.812585+0000","last_unstale":"2024-09-13T14:50:31.736744+0000","last_undegraded":"2024-09-13T14:50:31.736744+0000","last_fullsized":"2024-09-13T14:50:31.736744+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.614340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.614340+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:32:07.434109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":1153,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.681342+0000","last_change":"2024-09-13T14:43:07.522950+0000","last_active":"2024-09-13T14:50:15.681342+0000","last_peered":"2024-09-13T14:50:15.681342+0000","last_clean":"2024-09-13T14:50:15.681342+0000","last_became_active":"2024-09-13T14:43:07.522566+0000","last_became_peered":"2024-09-13T14:43:07.522566+0000","last_unstale":"2024-09-13T14:50:15.681342+0000","last_undegraded":"2024-09-13T14:50:15.681342+0000","last_fullsized":"2024-09-13T14:50:15.681342+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:03.201478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":64,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.735862+0000","last_change":"2024-09-13T14:45:19.807231+0000","last_active":"2024-09-13T14:50:31.735862+0000","last_peered":"2024-09-13T14:50:31.735862+0000","last_clean":"2024-09-13T14:50:31.735862+0000","last_became_active":"2024-09-13T14:45:19.806936+0000","last_became_peered":"2024-09-13T14:45:19.806936+0000","last_unstale":"2024-09-13T14:50:31.735862+0000","last_undegraded":"2024-09-13T14:50:31.735862+0000","last_fullsized":"2024-09-13T14:50:31.735862+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:04:26.723153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.18","version":"261'100","reported_seq":123,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.673729+0000","last_change":"2024-09-13T14:44:41.685537+0000","last_active":"2024-09-13T14:50:15.673729+0000","last_peered":"2024-09-13T14:50:15.673729+0000","last_clean":"2024-09-13T14:50:15.673729+0000","last_became_active":"2024-09-13T14:43:14.558502+0000","last_became_peered":"2024-09-13T14:43:14.558502+0000","last_unstale":"2024-09-13T14:50:15.673729+0000","last_undegraded":"2024-09-13T14:50:15.673729+0000","last_fullsized":"2024-09-13T14:50:15.673729+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1134,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:41.685124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:41.685124+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:37:58.412497+0000","scrub_duration":0.0075205970000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":113,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.754491+0000","last_change":"2024-09-13T14:43:07.516845+0000","last_active":"2024-09-13T14:50:31.754491+0000","last_peered":"2024-09-13T14:50:31.754491+0000","last_clean":"2024-09-13T14:50:31.754491+0000","last_became_active":"2024-09-13T14:43:07.514974+0000","last_became_peered":"2024-09-13T14:43:07.514974+0000","last_unstale":"2024-09-13T14:50:31.754491+0000","last_undegraded":"2024-09-13T14:50:31.754491+0000","last_fullsized":"2024-09-13T14:50:31.754491+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:55.419483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1748,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.736767+0000","last_change":"2024-09-13T14:45:19.815341+0000","last_active":"2024-09-13T14:50:31.736767+0000","last_peered":"2024-09-13T14:50:31.736767+0000","last_clean":"2024-09-13T14:50:31.736767+0000","last_became_active":"2024-09-13T14:45:19.810262+0000","last_became_peered":"2024-09-13T14:45:19.810262+0000","last_unstale":"2024-09-13T14:50:31.736767+0000","last_undegraded":"2024-09-13T14:50:31.736767+0000","last_fullsized":"2024-09-13T14:50:31.736767+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:39.847653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1b","version":"261'100","reported_seq":65,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.672799+0000","last_change":"2024-09-13T14:44:52.803330+0000","last_active":"2024-09-13T14:50:15.672799+0000","last_peered":"2024-09-13T14:50:15.672799+0000","last_clean":"2024-09-13T14:50:15.672799+0000","last_became_active":"2024-09-13T14:43:14.556712+0000","last_became_peered":"2024-09-13T14:43:14.556712+0000","last_unstale":"2024-09-13T14:50:15.672799+0000","last_undegraded":"2024-09-13T14:50:15.672799+0000","last_fullsized":"2024-09-13T14:50:15.672799+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:52.803242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:52.803242+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:06.658301+0000","scrub_duration":0.0019769660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":64,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.738553+0000","last_change":"2024-09-13T14:45:19.806623+0000","last_active":"2024-09-13T14:50:31.738553+0000","last_peered":"2024-09-13T14:50:31.738553+0000","last_clean":"2024-09-13T14:50:31.738553+0000","last_became_active":"2024-09-13T14:45:19.806219+0000","last_became_peered":"2024-09-13T14:45:19.806219+0000","last_unstale":"2024-09-13T14:50:31.738553+0000","last_undegraded":"2024-09-13T14:50:31.738553+0000","last_fullsized":"2024-09-13T14:50:31.738553+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:34:41.601201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1127,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.675263+0000","last_change":"2024-09-13T14:43:32.637751+0000","last_active":"2024-09-13T14:50:15.675263+0000","last_peered":"2024-09-13T14:50:15.675263+0000","last_clean":"2024-09-13T14:50:15.675263+0000","last_became_active":"2024-09-13T14:43:32.636901+0000","last_became_peered":"2024-09-13T14:43:32.636901+0000","last_unstale":"2024-09-13T14:50:15.675263+0000","last_undegraded":"2024-09-13T14:50:15.675263+0000","last_fullsized":"2024-09-13T14:50:15.675263+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:21.942419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1a","version":"261'100","reported_seq":65,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.680709+0000","last_change":"2024-09-13T14:43:41.910041+0000","last_active":"2024-09-13T14:50:15.680709+0000","last_peered":"2024-09-13T14:50:15.680709+0000","last_clean":"2024-09-13T14:50:15.680709+0000","last_became_active":"2024-09-13T14:43:14.555083+0000","last_became_peered":"2024-09-13T14:43:14.555083+0000","last_unstale":"2024-09-13T14:50:15.680709+0000","last_undegraded":"2024-09-13T14:50:15.680709+0000","last_fullsized":"2024-09-13T14:50:15.680709+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:41.909988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:41.909988+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:42:19.954838+0000","scrub_duration":0.0012891619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":64,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.738635+0000","last_change":"2024-09-13T14:45:19.812071+0000","last_active":"2024-09-13T14:50:31.738635+0000","last_peered":"2024-09-13T14:50:31.738635+0000","last_clean":"2024-09-13T14:50:31.738635+0000","last_became_active":"2024-09-13T14:45:19.811336+0000","last_became_peered":"2024-09-13T14:45:19.811336+0000","last_unstale":"2024-09-13T14:50:31.738635+0000","last_undegraded":"2024-09-13T14:50:31.738635+0000","last_fullsized":"2024-09-13T14:50:31.738635+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:40:29.814608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1765,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.737533+0000","last_change":"2024-09-13T14:45:19.810478+0000","last_active":"2024-09-13T14:50:31.737533+0000","last_peered":"2024-09-13T14:50:31.737533+0000","last_clean":"2024-09-13T14:50:31.737533+0000","last_became_active":"2024-09-13T14:45:19.808278+0000","last_became_peered":"2024-09-13T14:45:19.808278+0000","last_unstale":"2024-09-13T14:50:31.737533+0000","last_undegraded":"2024-09-13T14:50:31.737533+0000","last_fullsized":"2024-09-13T14:50:31.737533+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:12.211089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"261'100","reported_seq":65,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.676921+0000","last_change":"2024-09-13T14:44:38.214961+0000","last_active":"2024-09-13T14:50:15.676921+0000","last_peered":"2024-09-13T14:50:15.676921+0000","last_clean":"2024-09-13T14:50:15.676921+0000","last_became_active":"2024-09-13T14:43:14.557883+0000","last_became_peered":"2024-09-13T14:43:14.557883+0000","last_unstale":"2024-09-13T14:50:15.676921+0000","last_undegraded":"2024-09-13T14:50:15.676921+0000","last_fullsized":"2024-09-13T14:50:15.676921+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:38.214865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:38.214865+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:49:36.013783+0000","scrub_duration":0.0016389639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1635,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.736958+0000","last_change":"2024-09-13T14:30:15.576921+0000","last_active":"2024-09-13T14:50:31.736958+0000","last_peered":"2024-09-13T14:50:31.736958+0000","last_clean":"2024-09-13T14:50:31.736958+0000","last_became_active":"2024-09-13T14:30:15.576607+0000","last_became_peered":"2024-09-13T14:30:15.576607+0000","last_unstale":"2024-09-13T14:50:31.736958+0000","last_undegraded":"2024-09-13T14:50:31.736958+0000","last_fullsized":"2024-09-13T14:50:31.736958+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:35.162460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1137,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.675426+0000","last_change":"2024-09-13T14:43:07.518801+0000","last_active":"2024-09-13T14:50:15.675426+0000","last_peered":"2024-09-13T14:50:15.675426+0000","last_clean":"2024-09-13T14:50:15.675426+0000","last_became_active":"2024-09-13T14:43:07.517610+0000","last_became_peered":"2024-09-13T14:43:07.517610+0000","last_unstale":"2024-09-13T14:50:15.675426+0000","last_undegraded":"2024-09-13T14:50:15.675426+0000","last_fullsized":"2024-09-13T14:50:15.675426+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:51.632254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"261'100","reported_seq":113,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.731126+0000","last_change":"2024-09-13T14:43:15.037565+0000","last_active":"2024-09-13T14:50:31.731126+0000","last_peered":"2024-09-13T14:50:31.731126+0000","last_clean":"2024-09-13T14:50:31.731126+0000","last_became_active":"2024-09-13T14:43:14.554183+0000","last_became_peered":"2024-09-13T14:43:14.554183+0000","last_unstale":"2024-09-13T14:50:31.731126+0000","last_undegraded":"2024-09-13T14:50:31.731126+0000","last_fullsized":"2024-09-13T14:50:31.731126+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.037517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.037517+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:01:35.137195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":1131,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.676641+0000","last_change":"2024-09-13T14:30:15.579260+0000","last_active":"2024-09-13T14:50:15.676641+0000","last_peered":"2024-09-13T14:50:15.676641+0000","last_clean":"2024-09-13T14:50:15.676641+0000","last_became_active":"2024-09-13T14:30:15.578897+0000","last_became_peered":"2024-09-13T14:30:15.578897+0000","last_unstale":"2024-09-13T14:50:15.676641+0000","last_undegraded":"2024-09-13T14:50:15.676641+0000","last_fullsized":"2024-09-13T14:50:15.676641+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:56.737849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":113,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.744696+0000","last_change":"2024-09-13T14:43:07.521576+0000","last_active":"2024-09-13T14:50:31.744696+0000","last_peered":"2024-09-13T14:50:31.744696+0000","last_clean":"2024-09-13T14:50:31.744696+0000","last_became_active":"2024-09-13T14:43:07.520478+0000","last_became_peered":"2024-09-13T14:43:07.520478+0000","last_unstale":"2024-09-13T14:50:31.744696+0000","last_undegraded":"2024-09-13T14:50:31.744696+0000","last_fullsized":"2024-09-13T14:50:31.744696+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:53.410135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"261'100","reported_seq":391,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.675106+0000","last_change":"2024-09-13T14:44:49.775941+0000","last_active":"2024-09-13T14:50:15.675106+0000","last_peered":"2024-09-13T14:50:15.675106+0000","last_clean":"2024-09-13T14:50:15.675106+0000","last_became_active":"2024-09-13T14:43:14.560297+0000","last_became_peered":"2024-09-13T14:43:14.560297+0000","last_unstale":"2024-09-13T14:50:15.675106+0000","last_undegraded":"2024-09-13T14:50:15.675106+0000","last_fullsized":"2024-09-13T14:50:15.675106+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:49.775844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:49.775844+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:15:53.138661+0000","scrub_duration":0.012350274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":1120,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.682642+0000","last_change":"2024-09-13T14:19:25.870605+0000","last_active":"2024-09-13T14:50:15.682642+0000","last_peered":"2024-09-13T14:50:15.682642+0000","last_clean":"2024-09-13T14:50:15.682642+0000","last_became_active":"2024-09-13T14:19:25.870282+0000","last_became_peered":"2024-09-13T14:19:25.870282+0000","last_unstale":"2024-09-13T14:50:15.682642+0000","last_undegraded":"2024-09-13T14:50:15.682642+0000","last_fullsized":"2024-09-13T14:50:15.682642+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:52.514915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1134,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.681538+0000","last_change":"2024-09-13T14:43:07.526476+0000","last_active":"2024-09-13T14:50:15.681538+0000","last_peered":"2024-09-13T14:50:15.681538+0000","last_clean":"2024-09-13T14:50:15.681538+0000","last_became_active":"2024-09-13T14:43:07.525802+0000","last_became_peered":"2024-09-13T14:43:07.525802+0000","last_unstale":"2024-09-13T14:50:15.681538+0000","last_undegraded":"2024-09-13T14:50:15.681538+0000","last_fullsized":"2024-09-13T14:50:15.681538+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:23.154726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"261'100","reported_seq":120,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.744573+0000","last_change":"2024-09-13T14:45:19.807461+0000","last_active":"2024-09-13T14:50:31.744573+0000","last_peered":"2024-09-13T14:50:31.744573+0000","last_clean":"2024-09-13T14:50:31.744573+0000","last_became_active":"2024-09-13T14:45:19.807117+0000","last_became_peered":"2024-09-13T14:45:19.807117+0000","last_unstale":"2024-09-13T14:50:31.744573+0000","last_undegraded":"2024-09-13T14:50:31.744573+0000","last_fullsized":"2024-09-13T14:50:31.744573+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.955197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.955197+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:03.531555+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.d","version":"1174'6332","reported_seq":6347,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:42.773338+0000","last_change":"2024-09-13T14:49:22.120980+0000","last_active":"2024-09-13T14:50:42.773338+0000","last_peered":"2024-09-13T14:50:42.773338+0000","last_clean":"2024-09-13T14:50:42.773338+0000","last_became_active":"2024-09-13T14:49:22.120684+0000","last_became_peered":"2024-09-13T14:49:22.120684+0000","last_unstale":"2024-09-13T14:50:42.773338+0000","last_undegraded":"2024-09-13T14:50:42.773338+0000","last_fullsized":"2024-09-13T14:50:42.773338+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":6332,"log_dups_size":0,"ondisk_log_size":6332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:49:34.732581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414973952,"num_objects":6332,"num_object_clones":0,"num_object_copies":18996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6332,"num_write_kb":405248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1097,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.680728+0000","last_change":"2024-09-13T14:19:26.131609+0000","last_active":"2024-09-13T14:50:15.680728+0000","last_peered":"2024-09-13T14:50:15.680728+0000","last_clean":"2024-09-13T14:50:15.680728+0000","last_became_active":"2024-09-13T14:19:26.131029+0000","last_became_peered":"2024-09-13T14:19:26.131029+0000","last_unstale":"2024-09-13T14:50:15.680728+0000","last_undegraded":"2024-09-13T14:50:15.680728+0000","last_fullsized":"2024-09-13T14:50:15.680728+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:57.933696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":599,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.677495+0000","last_change":"2024-09-13T14:43:32.635070+0000","last_active":"2024-09-13T14:50:15.677495+0000","last_peered":"2024-09-13T14:50:15.677495+0000","last_clean":"2024-09-13T14:50:15.677495+0000","last_became_active":"2024-09-13T14:43:32.634734+0000","last_became_peered":"2024-09-13T14:43:32.634734+0000","last_unstale":"2024-09-13T14:50:15.677495+0000","last_undegraded":"2024-09-13T14:50:15.677495+0000","last_fullsized":"2024-09-13T14:50:15.677495+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:33.107010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":1755,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.736894+0000","last_change":"2024-09-13T14:43:07.537759+0000","last_active":"2024-09-13T14:50:31.736894+0000","last_peered":"2024-09-13T14:50:31.736894+0000","last_clean":"2024-09-13T14:50:31.736894+0000","last_became_active":"2024-09-13T14:43:07.520303+0000","last_became_peered":"2024-09-13T14:43:07.520303+0000","last_unstale":"2024-09-13T14:50:31.736894+0000","last_undegraded":"2024-09-13T14:50:31.736894+0000","last_fullsized":"2024-09-13T14:50:31.736894+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:32.772570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"261'100","reported_seq":64,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.737585+0000","last_change":"2024-09-13T14:45:19.813527+0000","last_active":"2024-09-13T14:50:31.737585+0000","last_peered":"2024-09-13T14:50:31.737585+0000","last_clean":"2024-09-13T14:50:31.737585+0000","last_became_active":"2024-09-13T14:45:19.812834+0000","last_became_peered":"2024-09-13T14:45:19.812834+0000","last_unstale":"2024-09-13T14:50:31.737585+0000","last_undegraded":"2024-09-13T14:50:31.737585+0000","last_fullsized":"2024-09-13T14:50:31.737585+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.710192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.710192+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:08:55.152703+0000","scrub_duration":0.0019250910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"1174'5992","reported_seq":6017,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:39.676491+0000","last_change":"2024-09-13T14:49:22.114132+0000","last_active":"2024-09-13T14:50:39.676491+0000","last_peered":"2024-09-13T14:50:39.676491+0000","last_clean":"2024-09-13T14:50:39.676491+0000","last_became_active":"2024-09-13T14:49:22.113820+0000","last_became_peered":"2024-09-13T14:49:22.113820+0000","last_unstale":"2024-09-13T14:50:39.676491+0000","last_undegraded":"2024-09-13T14:50:39.676491+0000","last_fullsized":"2024-09-13T14:50:39.676491+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":5992,"log_dups_size":0,"ondisk_log_size":5992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:34:37.889567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392691712,"num_objects":5992,"num_object_clones":0,"num_object_copies":17976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5992,"num_write_kb":383488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":64,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.738713+0000","last_change":"2024-09-13T14:45:19.813025+0000","last_active":"2024-09-13T14:50:31.738713+0000","last_peered":"2024-09-13T14:50:31.738713+0000","last_clean":"2024-09-13T14:50:31.738713+0000","last_became_active":"2024-09-13T14:45:19.812714+0000","last_became_peered":"2024-09-13T14:45:19.812714+0000","last_unstale":"2024-09-13T14:50:31.738713+0000","last_undegraded":"2024-09-13T14:50:31.738713+0000","last_fullsized":"2024-09-13T14:50:31.738713+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:40.671012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":113,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.744397+0000","last_change":"2024-09-13T14:43:07.522730+0000","last_active":"2024-09-13T14:50:31.744397+0000","last_peered":"2024-09-13T14:50:31.744397+0000","last_clean":"2024-09-13T14:50:31.744397+0000","last_became_active":"2024-09-13T14:43:07.517420+0000","last_became_peered":"2024-09-13T14:43:07.517420+0000","last_unstale":"2024-09-13T14:50:31.744397+0000","last_undegraded":"2024-09-13T14:50:31.744397+0000","last_fullsized":"2024-09-13T14:50:31.744397+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:21:30.265293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":1668,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.739557+0000","last_change":"2024-09-13T14:30:15.581728+0000","last_active":"2024-09-13T14:50:31.739557+0000","last_peered":"2024-09-13T14:50:31.739557+0000","last_clean":"2024-09-13T14:50:31.739557+0000","last_became_active":"2024-09-13T14:30:15.581303+0000","last_became_peered":"2024-09-13T14:30:15.581303+0000","last_unstale":"2024-09-13T14:50:31.739557+0000","last_undegraded":"2024-09-13T14:50:31.739557+0000","last_fullsized":"2024-09-13T14:50:31.739557+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:10.027991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"261'100","reported_seq":124,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.683098+0000","last_change":"2024-09-13T14:43:31.112173+0000","last_active":"2024-09-13T14:50:15.683098+0000","last_peered":"2024-09-13T14:50:15.683098+0000","last_clean":"2024-09-13T14:50:15.683098+0000","last_became_active":"2024-09-13T14:42:42.709263+0000","last_became_peered":"2024-09-13T14:42:42.709263+0000","last_unstale":"2024-09-13T14:50:15.683098+0000","last_undegraded":"2024-09-13T14:50:15.683098+0000","last_fullsized":"2024-09-13T14:50:15.683098+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:31.112111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:31.112111+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:02.707372+0000","scrub_duration":0.0015759260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"1174'6147","reported_seq":6172,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:39.665945+0000","last_change":"2024-09-13T14:49:22.118734+0000","last_active":"2024-09-13T14:50:39.665945+0000","last_peered":"2024-09-13T14:50:39.665945+0000","last_clean":"2024-09-13T14:50:39.665945+0000","last_became_active":"2024-09-13T14:49:22.118438+0000","last_became_peered":"2024-09-13T14:49:22.118438+0000","last_unstale":"2024-09-13T14:50:39.665945+0000","last_undegraded":"2024-09-13T14:50:39.665945+0000","last_fullsized":"2024-09-13T14:50:39.665945+0000","mapping_epoch":1167,"log_start":"1171'100","ondisk_log_start":"1171'100","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":6047,"log_dups_size":0,"ondisk_log_size":6047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:27:01.810072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402849792,"num_objects":6147,"num_object_clones":0,"num_object_copies":18441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6147,"num_write_kb":393408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1708,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.737920+0000","last_change":"2024-09-13T14:28:35.611508+0000","last_active":"2024-09-13T14:50:31.737920+0000","last_peered":"2024-09-13T14:50:31.737920+0000","last_clean":"2024-09-13T14:50:31.737920+0000","last_became_active":"2024-09-13T14:28:35.610106+0000","last_became_peered":"2024-09-13T14:28:35.610106+0000","last_unstale":"2024-09-13T14:50:31.737920+0000","last_undegraded":"2024-09-13T14:50:31.737920+0000","last_fullsized":"2024-09-13T14:50:31.737920+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:11.966958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1140,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.676996+0000","last_change":"2024-09-13T14:28:35.603663+0000","last_active":"2024-09-13T14:50:15.676996+0000","last_peered":"2024-09-13T14:50:15.676996+0000","last_clean":"2024-09-13T14:50:15.676996+0000","last_became_active":"2024-09-13T14:28:35.603358+0000","last_became_peered":"2024-09-13T14:28:35.603358+0000","last_unstale":"2024-09-13T14:50:15.676996+0000","last_undegraded":"2024-09-13T14:50:15.676996+0000","last_fullsized":"2024-09-13T14:50:15.676996+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:24:05.842796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":113,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.744808+0000","last_change":"2024-09-13T14:43:07.522750+0000","last_active":"2024-09-13T14:50:31.744808+0000","last_peered":"2024-09-13T14:50:31.744808+0000","last_clean":"2024-09-13T14:50:31.744808+0000","last_became_active":"2024-09-13T14:43:07.517267+0000","last_became_peered":"2024-09-13T14:43:07.517267+0000","last_unstale":"2024-09-13T14:50:31.744808+0000","last_undegraded":"2024-09-13T14:50:31.744808+0000","last_fullsized":"2024-09-13T14:50:31.744808+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:40.217748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"261'100","reported_seq":434,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.736554+0000","last_change":"2024-09-13T14:45:19.815050+0000","last_active":"2024-09-13T14:50:31.736554+0000","last_peered":"2024-09-13T14:50:31.736554+0000","last_clean":"2024-09-13T14:50:31.736554+0000","last_became_active":"2024-09-13T14:45:19.809732+0000","last_became_peered":"2024-09-13T14:45:19.809732+0000","last_unstale":"2024-09-13T14:50:31.736554+0000","last_undegraded":"2024-09-13T14:50:31.736554+0000","last_fullsized":"2024-09-13T14:50:31.736554+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:33.574443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:33.574443+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:51:40.261036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"1174'6006","reported_seq":6031,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:39.692264+0000","last_change":"2024-09-13T14:49:22.122114+0000","last_active":"2024-09-13T14:50:39.692264+0000","last_peered":"2024-09-13T14:50:39.692264+0000","last_clean":"2024-09-13T14:50:39.692264+0000","last_became_active":"2024-09-13T14:49:22.120044+0000","last_became_peered":"2024-09-13T14:49:22.120044+0000","last_unstale":"2024-09-13T14:50:39.692264+0000","last_undegraded":"2024-09-13T14:50:39.692264+0000","last_fullsized":"2024-09-13T14:50:39.692264+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":6006,"log_dups_size":0,"ondisk_log_size":6006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:09:06.965253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393609216,"num_objects":6006,"num_object_clones":0,"num_object_copies":18018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6006,"num_write_kb":384384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":113,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.744564+0000","last_change":"2024-09-13T14:43:07.520049+0000","last_active":"2024-09-13T14:50:31.744564+0000","last_peered":"2024-09-13T14:50:31.744564+0000","last_clean":"2024-09-13T14:50:31.744564+0000","last_became_active":"2024-09-13T14:43:07.514646+0000","last_became_peered":"2024-09-13T14:43:07.514646+0000","last_unstale":"2024-09-13T14:50:31.744564+0000","last_undegraded":"2024-09-13T14:50:31.744564+0000","last_fullsized":"2024-09-13T14:50:31.744564+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:10:35.200778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":113,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.744696+0000","last_change":"2024-09-13T14:43:07.525666+0000","last_active":"2024-09-13T14:50:31.744696+0000","last_peered":"2024-09-13T14:50:31.744696+0000","last_clean":"2024-09-13T14:50:31.744696+0000","last_became_active":"2024-09-13T14:43:07.524064+0000","last_became_peered":"2024-09-13T14:43:07.524064+0000","last_unstale":"2024-09-13T14:50:31.744696+0000","last_undegraded":"2024-09-13T14:50:31.744696+0000","last_fullsized":"2024-09-13T14:50:31.744696+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:10.270333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1154,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.674685+0000","last_change":"2024-09-13T14:45:19.804177+0000","last_active":"2024-09-13T14:50:15.674685+0000","last_peered":"2024-09-13T14:50:15.674685+0000","last_clean":"2024-09-13T14:50:15.674685+0000","last_became_active":"2024-09-13T14:45:19.803890+0000","last_became_peered":"2024-09-13T14:45:19.803890+0000","last_unstale":"2024-09-13T14:50:15.674685+0000","last_undegraded":"2024-09-13T14:50:15.674685+0000","last_fullsized":"2024-09-13T14:50:15.674685+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:25.140789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"261'100","reported_seq":81,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.739434+0000","last_change":"2024-09-13T14:45:36.965992+0000","last_active":"2024-09-13T14:50:31.739434+0000","last_peered":"2024-09-13T14:50:31.739434+0000","last_clean":"2024-09-13T14:50:31.739434+0000","last_became_active":"2024-09-13T14:45:19.811238+0000","last_became_peered":"2024-09-13T14:45:19.811238+0000","last_unstale":"2024-09-13T14:50:31.739434+0000","last_undegraded":"2024-09-13T14:50:31.739434+0000","last_fullsized":"2024-09-13T14:50:31.739434+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:36.965775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:45:36.965775+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:36:01.623251+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"1174'6243","reported_seq":6258,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:42.760037+0000","last_change":"2024-09-13T14:49:22.122277+0000","last_active":"2024-09-13T14:50:42.760037+0000","last_peered":"2024-09-13T14:50:42.760037+0000","last_clean":"2024-09-13T14:50:42.760037+0000","last_became_active":"2024-09-13T14:49:22.121980+0000","last_became_peered":"2024-09-13T14:49:22.121980+0000","last_unstale":"2024-09-13T14:50:42.760037+0000","last_undegraded":"2024-09-13T14:50:42.760037+0000","last_fullsized":"2024-09-13T14:50:42.760037+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":6243,"log_dups_size":0,"ondisk_log_size":6243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:58.275564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409141248,"num_objects":6243,"num_object_clones":0,"num_object_copies":18729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6243,"num_write_kb":399552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":53,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.674330+0000","last_change":"2024-09-13T14:45:19.806318+0000","last_active":"2024-09-13T14:50:15.674330+0000","last_peered":"2024-09-13T14:50:15.674330+0000","last_clean":"2024-09-13T14:50:15.674330+0000","last_became_active":"2024-09-13T14:45:19.806074+0000","last_became_peered":"2024-09-13T14:45:19.806074+0000","last_unstale":"2024-09-13T14:50:15.674330+0000","last_undegraded":"2024-09-13T14:50:15.674330+0000","last_fullsized":"2024-09-13T14:50:15.674330+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:21.679797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1115,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.681671+0000","last_change":"2024-09-13T14:28:35.604404+0000","last_active":"2024-09-13T14:50:15.681671+0000","last_peered":"2024-09-13T14:50:15.681671+0000","last_clean":"2024-09-13T14:50:15.681671+0000","last_became_active":"2024-09-13T14:28:35.604102+0000","last_became_peered":"2024-09-13T14:28:35.604102+0000","last_unstale":"2024-09-13T14:50:15.681671+0000","last_undegraded":"2024-09-13T14:50:15.681671+0000","last_fullsized":"2024-09-13T14:50:15.681671+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:25.546010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":1165,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.671675+0000","last_change":"2024-09-13T14:43:07.538919+0000","last_active":"2024-09-13T14:50:15.671675+0000","last_peered":"2024-09-13T14:50:15.671675+0000","last_clean":"2024-09-13T14:50:15.671675+0000","last_became_active":"2024-09-13T14:43:07.513109+0000","last_became_peered":"2024-09-13T14:43:07.513109+0000","last_unstale":"2024-09-13T14:50:15.671675+0000","last_undegraded":"2024-09-13T14:50:15.671675+0000","last_fullsized":"2024-09-13T14:50:15.671675+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:27.771720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"261'100","reported_seq":114,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.671166+0000","last_change":"2024-09-13T14:43:44.720333+0000","last_active":"2024-09-13T14:50:15.671166+0000","last_peered":"2024-09-13T14:50:15.671166+0000","last_clean":"2024-09-13T14:50:15.671166+0000","last_became_active":"2024-09-13T14:42:14.946591+0000","last_became_peered":"2024-09-13T14:42:14.946591+0000","last_unstale":"2024-09-13T14:50:15.671166+0000","last_undegraded":"2024-09-13T14:50:15.671166+0000","last_fullsized":"2024-09-13T14:50:15.671166+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:44.720258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:44.720258+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:33.281357+0000","scrub_duration":0.0018417920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"1174'6076","reported_seq":6101,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:39.593785+0000","last_change":"2024-09-13T14:49:22.119554+0000","last_active":"2024-09-13T14:50:39.593785+0000","last_peered":"2024-09-13T14:50:39.593785+0000","last_clean":"2024-09-13T14:50:39.593785+0000","last_became_active":"2024-09-13T14:49:22.119249+0000","last_became_peered":"2024-09-13T14:49:22.119249+0000","last_unstale":"2024-09-13T14:50:39.593785+0000","last_undegraded":"2024-09-13T14:50:39.593785+0000","last_fullsized":"2024-09-13T14:50:39.593785+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":6076,"log_dups_size":0,"ondisk_log_size":6076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:03:16.735024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398196736,"num_objects":6076,"num_object_clones":0,"num_object_copies":18228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6076,"num_write_kb":388864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":81,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.721114+0000","last_change":"2024-09-13T14:44:03.858211+0000","last_active":"2024-09-13T14:50:31.721114+0000","last_peered":"2024-09-13T14:50:31.721114+0000","last_clean":"2024-09-13T14:50:31.721114+0000","last_became_active":"2024-09-13T14:44:03.857928+0000","last_became_peered":"2024-09-13T14:44:03.857928+0000","last_unstale":"2024-09-13T14:50:31.721114+0000","last_undegraded":"2024-09-13T14:50:31.721114+0000","last_fullsized":"2024-09-13T14:50:31.721114+0000","mapping_epoch":1140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:28.428693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":113,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.744578+0000","last_change":"2024-09-13T14:43:07.516243+0000","last_active":"2024-09-13T14:50:31.744578+0000","last_peered":"2024-09-13T14:50:31.744578+0000","last_clean":"2024-09-13T14:50:31.744578+0000","last_became_active":"2024-09-13T14:43:07.515863+0000","last_became_peered":"2024-09-13T14:43:07.515863+0000","last_unstale":"2024-09-13T14:50:31.744578+0000","last_undegraded":"2024-09-13T14:50:31.744578+0000","last_fullsized":"2024-09-13T14:50:31.744578+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:08.085298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1750,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.729877+0000","last_change":"2024-09-13T14:45:19.812713+0000","last_active":"2024-09-13T14:50:31.729877+0000","last_peered":"2024-09-13T14:50:31.729877+0000","last_clean":"2024-09-13T14:50:31.729877+0000","last_became_active":"2024-09-13T14:45:19.812203+0000","last_became_peered":"2024-09-13T14:45:19.812203+0000","last_unstale":"2024-09-13T14:50:31.729877+0000","last_undegraded":"2024-09-13T14:50:31.729877+0000","last_fullsized":"2024-09-13T14:50:31.729877+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:30:46.149347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"261'100","reported_seq":1294,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.676544+0000","last_change":"2024-09-13T14:43:42.577312+0000","last_active":"2024-09-13T14:50:15.676544+0000","last_peered":"2024-09-13T14:50:15.676544+0000","last_clean":"2024-09-13T14:50:15.676544+0000","last_became_active":"2024-09-13T14:42:14.928923+0000","last_became_peered":"2024-09-13T14:42:14.928923+0000","last_unstale":"2024-09-13T14:50:15.676544+0000","last_undegraded":"2024-09-13T14:50:15.676544+0000","last_fullsized":"2024-09-13T14:50:15.676544+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:42.577243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:42.577243+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:01.234250+0000","scrub_duration":0.001367839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"1174'6420","reported_seq":6445,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:43.162524+0000","last_change":"2024-09-13T14:49:22.118690+0000","last_active":"2024-09-13T14:50:43.162524+0000","last_peered":"2024-09-13T14:50:43.162524+0000","last_clean":"2024-09-13T14:50:43.162524+0000","last_became_active":"2024-09-13T14:49:22.117901+0000","last_became_peered":"2024-09-13T14:49:22.117901+0000","last_unstale":"2024-09-13T14:50:43.162524+0000","last_undegraded":"2024-09-13T14:50:43.162524+0000","last_fullsized":"2024-09-13T14:50:43.162524+0000","mapping_epoch":1167,"log_start":"1171'100","ondisk_log_start":"1171'100","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":6320,"log_dups_size":0,"ondisk_log_size":6320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:28.098913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420741120,"num_objects":6420,"num_object_clones":0,"num_object_copies":19260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6420,"num_write_kb":410880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1744,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.736647+0000","last_change":"2024-09-13T14:43:07.520212+0000","last_active":"2024-09-13T14:50:31.736647+0000","last_peered":"2024-09-13T14:50:31.736647+0000","last_clean":"2024-09-13T14:50:31.736647+0000","last_became_active":"2024-09-13T14:43:07.519831+0000","last_became_peered":"2024-09-13T14:43:07.519831+0000","last_unstale":"2024-09-13T14:50:31.736647+0000","last_undegraded":"2024-09-13T14:50:31.736647+0000","last_fullsized":"2024-09-13T14:50:31.736647+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:20.990736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1766,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.745585+0000","last_change":"2024-09-13T14:45:19.809505+0000","last_active":"2024-09-13T14:50:31.745585+0000","last_peered":"2024-09-13T14:50:31.745585+0000","last_clean":"2024-09-13T14:50:31.745585+0000","last_became_active":"2024-09-13T14:45:19.809181+0000","last_became_peered":"2024-09-13T14:45:19.809181+0000","last_unstale":"2024-09-13T14:50:31.745585+0000","last_undegraded":"2024-09-13T14:50:31.745585+0000","last_fullsized":"2024-09-13T14:50:31.745585+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:24.239188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1177,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.677455+0000","last_change":"2024-09-13T14:43:07.516140+0000","last_active":"2024-09-13T14:50:15.677455+0000","last_peered":"2024-09-13T14:50:15.677455+0000","last_clean":"2024-09-13T14:50:15.677455+0000","last_became_active":"2024-09-13T14:43:07.515481+0000","last_became_peered":"2024-09-13T14:43:07.515481+0000","last_unstale":"2024-09-13T14:50:15.677455+0000","last_undegraded":"2024-09-13T14:50:15.677455+0000","last_fullsized":"2024-09-13T14:50:15.677455+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:43.706214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"261'100","reported_seq":125,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.761568+0000","last_change":"2024-09-13T14:43:16.008346+0000","last_active":"2024-09-13T14:50:31.761568+0000","last_peered":"2024-09-13T14:50:31.761568+0000","last_clean":"2024-09-13T14:50:31.761568+0000","last_became_active":"2024-09-13T14:43:07.523121+0000","last_became_peered":"2024-09-13T14:43:07.523121+0000","last_unstale":"2024-09-13T14:50:31.761568+0000","last_undegraded":"2024-09-13T14:50:31.761568+0000","last_fullsized":"2024-09-13T14:50:31.761568+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.008264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.008264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:58:35.564470+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"1174'6129","reported_seq":6144,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:42.750697+0000","last_change":"2024-09-13T14:49:22.119570+0000","last_active":"2024-09-13T14:50:42.750697+0000","last_peered":"2024-09-13T14:50:42.750697+0000","last_clean":"2024-09-13T14:50:42.750697+0000","last_became_active":"2024-09-13T14:49:22.119224+0000","last_became_peered":"2024-09-13T14:49:22.119224+0000","last_unstale":"2024-09-13T14:50:42.750697+0000","last_undegraded":"2024-09-13T14:50:42.750697+0000","last_fullsized":"2024-09-13T14:50:42.750697+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":6129,"log_dups_size":0,"ondisk_log_size":6129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:43.554416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401670144,"num_objects":6129,"num_object_clones":0,"num_object_copies":18387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6129,"num_write_kb":392256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":56,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.683002+0000","last_change":"2024-09-13T14:43:32.636990+0000","last_active":"2024-09-13T14:50:15.683002+0000","last_peered":"2024-09-13T14:50:15.683002+0000","last_clean":"2024-09-13T14:50:15.683002+0000","last_became_active":"2024-09-13T14:43:32.636589+0000","last_became_peered":"2024-09-13T14:43:32.636589+0000","last_unstale":"2024-09-13T14:50:15.683002+0000","last_undegraded":"2024-09-13T14:50:15.683002+0000","last_fullsized":"2024-09-13T14:50:15.683002+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:36:25.129769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":64,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.735451+0000","last_change":"2024-09-13T14:45:19.810888+0000","last_active":"2024-09-13T14:50:31.735451+0000","last_peered":"2024-09-13T14:50:31.735451+0000","last_clean":"2024-09-13T14:50:31.735451+0000","last_became_active":"2024-09-13T14:45:19.810486+0000","last_became_peered":"2024-09-13T14:45:19.810486+0000","last_unstale":"2024-09-13T14:50:31.735451+0000","last_undegraded":"2024-09-13T14:50:31.735451+0000","last_fullsized":"2024-09-13T14:50:31.735451+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:03.886942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":1130,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.673209+0000","last_change":"2024-09-13T14:29:42.539455+0000","last_active":"2024-09-13T14:50:15.673209+0000","last_peered":"2024-09-13T14:50:15.673209+0000","last_clean":"2024-09-13T14:50:15.673209+0000","last_became_active":"2024-09-13T14:29:42.539068+0000","last_became_peered":"2024-09-13T14:29:42.539068+0000","last_unstale":"2024-09-13T14:50:15.673209+0000","last_undegraded":"2024-09-13T14:50:15.673209+0000","last_fullsized":"2024-09-13T14:50:15.673209+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:24:06.048855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034405999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"261'100","reported_seq":1285,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.677528+0000","last_change":"2024-09-13T14:44:28.186029+0000","last_active":"2024-09-13T14:50:15.677528+0000","last_peered":"2024-09-13T14:50:15.677528+0000","last_clean":"2024-09-13T14:50:15.677528+0000","last_became_active":"2024-09-13T14:42:14.927771+0000","last_became_peered":"2024-09-13T14:42:14.927771+0000","last_unstale":"2024-09-13T14:50:15.677528+0000","last_undegraded":"2024-09-13T14:50:15.677528+0000","last_fullsized":"2024-09-13T14:50:15.677528+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:28.185949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:28.185949+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:38.363961+0000","scrub_duration":0.0018071129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3","version":"1174'6142","reported_seq":6167,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:43.176764+0000","last_change":"2024-09-13T14:49:22.119776+0000","last_active":"2024-09-13T14:50:43.176764+0000","last_peered":"2024-09-13T14:50:43.176764+0000","last_clean":"2024-09-13T14:50:43.176764+0000","last_became_active":"2024-09-13T14:49:22.119517+0000","last_became_peered":"2024-09-13T14:49:22.119517+0000","last_unstale":"2024-09-13T14:50:43.176764+0000","last_undegraded":"2024-09-13T14:50:43.176764+0000","last_fullsized":"2024-09-13T14:50:43.176764+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":6142,"log_dups_size":0,"ondisk_log_size":6142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:53.152882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402522112,"num_objects":6142,"num_object_clones":0,"num_object_copies":18426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6142,"num_write_kb":393088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":113,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.752083+0000","last_change":"2024-09-13T14:43:07.520176+0000","last_active":"2024-09-13T14:50:31.752083+0000","last_peered":"2024-09-13T14:50:31.752083+0000","last_clean":"2024-09-13T14:50:31.752083+0000","last_became_active":"2024-09-13T14:43:07.516748+0000","last_became_peered":"2024-09-13T14:43:07.516748+0000","last_unstale":"2024-09-13T14:50:31.752083+0000","last_undegraded":"2024-09-13T14:50:31.752083+0000","last_fullsized":"2024-09-13T14:50:31.752083+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:52:04.392821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":64,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.738799+0000","last_change":"2024-09-13T14:45:19.812353+0000","last_active":"2024-09-13T14:50:31.738799+0000","last_peered":"2024-09-13T14:50:31.738799+0000","last_clean":"2024-09-13T14:50:31.738799+0000","last_became_active":"2024-09-13T14:45:19.811743+0000","last_became_peered":"2024-09-13T14:45:19.811743+0000","last_unstale":"2024-09-13T14:50:31.738799+0000","last_undegraded":"2024-09-13T14:50:31.738799+0000","last_fullsized":"2024-09-13T14:50:31.738799+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:43.728170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"261'100","reported_seq":114,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.671440+0000","last_change":"2024-09-13T14:44:34.814157+0000","last_active":"2024-09-13T14:50:15.671440+0000","last_peered":"2024-09-13T14:50:15.671440+0000","last_clean":"2024-09-13T14:50:15.671440+0000","last_became_active":"2024-09-13T14:42:42.707803+0000","last_became_peered":"2024-09-13T14:42:42.707803+0000","last_unstale":"2024-09-13T14:50:15.671440+0000","last_undegraded":"2024-09-13T14:50:15.671440+0000","last_fullsized":"2024-09-13T14:50:15.671440+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:34.814070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:34.814070+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:38.408561+0000","scrub_duration":0.01686414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"1174'6179","reported_seq":6194,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:42.518539+0000","last_change":"2024-09-13T14:49:22.119578+0000","last_active":"2024-09-13T14:50:42.518539+0000","last_peered":"2024-09-13T14:50:42.518539+0000","last_clean":"2024-09-13T14:50:42.518539+0000","last_became_active":"2024-09-13T14:49:22.119280+0000","last_became_peered":"2024-09-13T14:49:22.119280+0000","last_unstale":"2024-09-13T14:50:42.518539+0000","last_undegraded":"2024-09-13T14:50:42.518539+0000","last_fullsized":"2024-09-13T14:50:42.518539+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":6179,"log_dups_size":0,"ondisk_log_size":6179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:13:15.669993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404946944,"num_objects":6179,"num_object_clones":0,"num_object_copies":18537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6179,"num_write_kb":395456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1144,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.683724+0000","last_change":"2024-09-13T14:44:03.855577+0000","last_active":"2024-09-13T14:50:15.683724+0000","last_peered":"2024-09-13T14:50:15.683724+0000","last_clean":"2024-09-13T14:50:15.683724+0000","last_became_active":"2024-09-13T14:44:03.855267+0000","last_became_peered":"2024-09-13T14:44:03.855267+0000","last_unstale":"2024-09-13T14:50:15.683724+0000","last_undegraded":"2024-09-13T14:50:15.683724+0000","last_fullsized":"2024-09-13T14:50:15.683724+0000","mapping_epoch":1140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:46.148814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1115,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.680841+0000","last_change":"2024-09-13T14:29:42.545815+0000","last_active":"2024-09-13T14:50:15.680841+0000","last_peered":"2024-09-13T14:50:15.680841+0000","last_clean":"2024-09-13T14:50:15.680841+0000","last_became_active":"2024-09-13T14:29:42.545508+0000","last_became_peered":"2024-09-13T14:29:42.545508+0000","last_unstale":"2024-09-13T14:50:15.680841+0000","last_undegraded":"2024-09-13T14:50:15.680841+0000","last_fullsized":"2024-09-13T14:50:15.680841+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:39.678381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"261'100","reported_seq":131,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.756923+0000","last_change":"2024-09-13T14:43:18.988482+0000","last_active":"2024-09-13T14:50:31.756923+0000","last_peered":"2024-09-13T14:50:31.756923+0000","last_clean":"2024-09-13T14:50:31.756923+0000","last_became_active":"2024-09-13T14:43:14.556754+0000","last_became_peered":"2024-09-13T14:43:14.556754+0000","last_unstale":"2024-09-13T14:50:31.756923+0000","last_undegraded":"2024-09-13T14:50:31.756923+0000","last_fullsized":"2024-09-13T14:50:31.756923+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:18.988405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:18.988405+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:45:01.726326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"1174'6216","reported_seq":6241,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:43.170621+0000","last_change":"2024-09-13T14:49:22.121230+0000","last_active":"2024-09-13T14:50:43.170621+0000","last_peered":"2024-09-13T14:50:43.170621+0000","last_clean":"2024-09-13T14:50:43.170621+0000","last_became_active":"2024-09-13T14:49:22.120836+0000","last_became_peered":"2024-09-13T14:49:22.120836+0000","last_unstale":"2024-09-13T14:50:43.170621+0000","last_undegraded":"2024-09-13T14:50:43.170621+0000","last_fullsized":"2024-09-13T14:50:43.170621+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":6216,"log_dups_size":0,"ondisk_log_size":6216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:21.843404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407371776,"num_objects":6216,"num_object_clones":0,"num_object_copies":18648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6216,"num_write_kb":397824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1161,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.683259+0000","last_change":"2024-09-13T14:43:07.515418+0000","last_active":"2024-09-13T14:50:15.683259+0000","last_peered":"2024-09-13T14:50:15.683259+0000","last_clean":"2024-09-13T14:50:15.683259+0000","last_became_active":"2024-09-13T14:43:07.515113+0000","last_became_peered":"2024-09-13T14:43:07.515113+0000","last_unstale":"2024-09-13T14:50:15.683259+0000","last_undegraded":"2024-09-13T14:50:15.683259+0000","last_fullsized":"2024-09-13T14:50:15.683259+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:10:25.810604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1128,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.680969+0000","last_change":"2024-09-13T14:43:32.636334+0000","last_active":"2024-09-13T14:50:15.680969+0000","last_peered":"2024-09-13T14:50:15.680969+0000","last_clean":"2024-09-13T14:50:15.680969+0000","last_became_active":"2024-09-13T14:43:32.635736+0000","last_became_peered":"2024-09-13T14:43:32.635736+0000","last_unstale":"2024-09-13T14:50:15.680969+0000","last_undegraded":"2024-09-13T14:50:15.680969+0000","last_fullsized":"2024-09-13T14:50:15.680969+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:49.076014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"261'100","reported_seq":119,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.673581+0000","last_change":"2024-09-13T14:45:19.805634+0000","last_active":"2024-09-13T14:50:15.673581+0000","last_peered":"2024-09-13T14:50:15.673581+0000","last_clean":"2024-09-13T14:50:15.673581+0000","last_became_active":"2024-09-13T14:45:19.805046+0000","last_became_peered":"2024-09-13T14:45:19.805046+0000","last_unstale":"2024-09-13T14:50:15.673581+0000","last_undegraded":"2024-09-13T14:50:15.673581+0000","last_fullsized":"2024-09-13T14:50:15.673581+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:32.752408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:32.752408+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:06:53.140239+0000","scrub_duration":0.0057731559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.0","version":"1174'5938","reported_seq":5963,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:39.683902+0000","last_change":"2024-09-13T14:49:22.118730+0000","last_active":"2024-09-13T14:50:39.683902+0000","last_peered":"2024-09-13T14:50:39.683902+0000","last_clean":"2024-09-13T14:50:39.683902+0000","last_became_active":"2024-09-13T14:49:22.118428+0000","last_became_peered":"2024-09-13T14:49:22.118428+0000","last_unstale":"2024-09-13T14:50:39.683902+0000","last_undegraded":"2024-09-13T14:50:39.683902+0000","last_fullsized":"2024-09-13T14:50:39.683902+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":5938,"log_dups_size":0,"ondisk_log_size":5938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:54.951267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389152768,"num_objects":5938,"num_object_clones":0,"num_object_copies":17814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5938,"num_write_kb":380032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1164,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.682556+0000","last_change":"2024-09-13T14:45:19.804441+0000","last_active":"2024-09-13T14:50:15.682556+0000","last_peered":"2024-09-13T14:50:15.682556+0000","last_clean":"2024-09-13T14:50:15.682556+0000","last_became_active":"2024-09-13T14:45:19.804039+0000","last_became_peered":"2024-09-13T14:45:19.804039+0000","last_unstale":"2024-09-13T14:50:15.682556+0000","last_undegraded":"2024-09-13T14:50:15.682556+0000","last_fullsized":"2024-09-13T14:50:15.682556+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:58:00.322042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":68,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.737990+0000","last_change":"2024-09-13T14:45:19.816666+0000","last_active":"2024-09-13T14:50:31.737990+0000","last_peered":"2024-09-13T14:50:31.737990+0000","last_clean":"2024-09-13T14:50:31.737990+0000","last_became_active":"2024-09-13T14:45:19.807224+0000","last_became_peered":"2024-09-13T14:45:19.807224+0000","last_unstale":"2024-09-13T14:50:31.737990+0000","last_undegraded":"2024-09-13T14:50:31.737990+0000","last_fullsized":"2024-09-13T14:50:31.737990+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:15:20.171682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"261'100","reported_seq":1304,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.676256+0000","last_change":"2024-09-13T14:45:19.798553+0000","last_active":"2024-09-13T14:50:15.676256+0000","last_peered":"2024-09-13T14:50:15.676256+0000","last_clean":"2024-09-13T14:50:15.676256+0000","last_became_active":"2024-09-13T14:45:19.798189+0000","last_became_peered":"2024-09-13T14:45:19.798189+0000","last_unstale":"2024-09-13T14:50:15.676256+0000","last_undegraded":"2024-09-13T14:50:15.676256+0000","last_fullsized":"2024-09-13T14:50:15.676256+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:29.169325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:29.169325+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:10.412666+0000","scrub_duration":0.0019358890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"1174'6204","reported_seq":6227,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:43.189576+0000","last_change":"2024-09-13T14:49:22.120585+0000","last_active":"2024-09-13T14:50:43.189576+0000","last_peered":"2024-09-13T14:50:43.189576+0000","last_clean":"2024-09-13T14:50:43.189576+0000","last_became_active":"2024-09-13T14:49:22.120301+0000","last_became_peered":"2024-09-13T14:49:22.120301+0000","last_unstale":"2024-09-13T14:50:43.189576+0000","last_undegraded":"2024-09-13T14:50:43.189576+0000","last_fullsized":"2024-09-13T14:50:43.189576+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":6204,"log_dups_size":0,"ondisk_log_size":6204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:20:25.183879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406585344,"num_objects":6204,"num_object_clones":0,"num_object_copies":18612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6204,"num_write_kb":397056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1732,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.736694+0000","last_change":"2024-09-13T14:43:07.515572+0000","last_active":"2024-09-13T14:50:31.736694+0000","last_peered":"2024-09-13T14:50:31.736694+0000","last_clean":"2024-09-13T14:50:31.736694+0000","last_became_active":"2024-09-13T14:43:07.515231+0000","last_became_peered":"2024-09-13T14:43:07.515231+0000","last_unstale":"2024-09-13T14:50:31.736694+0000","last_undegraded":"2024-09-13T14:50:31.736694+0000","last_fullsized":"2024-09-13T14:50:31.736694+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:57.966380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1157,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.674579+0000","last_change":"2024-09-13T14:45:19.804982+0000","last_active":"2024-09-13T14:50:15.674579+0000","last_peered":"2024-09-13T14:50:15.674579+0000","last_clean":"2024-09-13T14:50:15.674579+0000","last_became_active":"2024-09-13T14:45:19.804663+0000","last_became_peered":"2024-09-13T14:45:19.804663+0000","last_unstale":"2024-09-13T14:50:15.674579+0000","last_undegraded":"2024-09-13T14:50:15.674579+0000","last_fullsized":"2024-09-13T14:50:15.674579+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:43.211479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"261'100","reported_seq":394,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.683602+0000","last_change":"2024-09-13T14:43:42.881584+0000","last_active":"2024-09-13T14:50:15.683602+0000","last_peered":"2024-09-13T14:50:15.683602+0000","last_clean":"2024-09-13T14:50:15.683602+0000","last_became_active":"2024-09-13T14:43:14.562161+0000","last_became_peered":"2024-09-13T14:43:14.562161+0000","last_unstale":"2024-09-13T14:50:15.683602+0000","last_undegraded":"2024-09-13T14:50:15.683602+0000","last_fullsized":"2024-09-13T14:50:15.683602+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:42.881520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:42.881520+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:06:50.353442+0000","scrub_duration":0.0013804310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"1174'5863","reported_seq":5888,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:39.705166+0000","last_change":"2024-09-13T14:49:22.117570+0000","last_active":"2024-09-13T14:50:39.705166+0000","last_peered":"2024-09-13T14:50:39.705166+0000","last_clean":"2024-09-13T14:50:39.705166+0000","last_became_active":"2024-09-13T14:49:22.117259+0000","last_became_peered":"2024-09-13T14:49:22.117259+0000","last_unstale":"2024-09-13T14:50:39.705166+0000","last_undegraded":"2024-09-13T14:50:39.705166+0000","last_fullsized":"2024-09-13T14:50:39.705166+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":5863,"log_dups_size":0,"ondisk_log_size":5863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:59:23.669062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384237568,"num_objects":5863,"num_object_clones":0,"num_object_copies":17589,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5863,"num_write_kb":375232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1130,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.678621+0000","last_change":"2024-09-13T14:43:07.518120+0000","last_active":"2024-09-13T14:50:15.678621+0000","last_peered":"2024-09-13T14:50:15.678621+0000","last_clean":"2024-09-13T14:50:15.678621+0000","last_became_active":"2024-09-13T14:43:07.517361+0000","last_became_peered":"2024-09-13T14:43:07.517361+0000","last_unstale":"2024-09-13T14:50:15.678621+0000","last_undegraded":"2024-09-13T14:50:15.678621+0000","last_fullsized":"2024-09-13T14:50:15.678621+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:39.366473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":1112,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.676835+0000","last_change":"2024-09-13T14:19:25.871006+0000","last_active":"2024-09-13T14:50:15.676835+0000","last_peered":"2024-09-13T14:50:15.676835+0000","last_clean":"2024-09-13T14:50:15.676835+0000","last_became_active":"2024-09-13T14:19:25.870713+0000","last_became_peered":"2024-09-13T14:19:25.870713+0000","last_unstale":"2024-09-13T14:50:15.676835+0000","last_undegraded":"2024-09-13T14:50:15.676835+0000","last_fullsized":"2024-09-13T14:50:15.676835+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:34.372124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"261'100","reported_seq":388,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.675188+0000","last_change":"2024-09-13T14:49:21.723011+0000","last_active":"2024-09-13T14:50:15.675188+0000","last_peered":"2024-09-13T14:50:15.675188+0000","last_clean":"2024-09-13T14:50:15.675188+0000","last_became_active":"2024-09-13T14:43:32.636545+0000","last_became_peered":"2024-09-13T14:43:32.636545+0000","last_unstale":"2024-09-13T14:50:15.675188+0000","last_undegraded":"2024-09-13T14:50:15.675188+0000","last_fullsized":"2024-09-13T14:50:15.675188+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:49:21.722939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.722939+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:18:07.462147+0000","scrub_duration":0.0016642790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.5","version":"1174'6291","reported_seq":6306,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:42.811140+0000","last_change":"2024-09-13T14:49:22.120623+0000","last_active":"2024-09-13T14:50:42.811140+0000","last_peered":"2024-09-13T14:50:42.811140+0000","last_clean":"2024-09-13T14:50:42.811140+0000","last_became_active":"2024-09-13T14:49:22.120295+0000","last_became_peered":"2024-09-13T14:49:22.120295+0000","last_unstale":"2024-09-13T14:50:42.811140+0000","last_undegraded":"2024-09-13T14:50:42.811140+0000","last_fullsized":"2024-09-13T14:50:42.811140+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":6291,"log_dups_size":0,"ondisk_log_size":6291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.848357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412286976,"num_objects":6291,"num_object_clones":0,"num_object_copies":18873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6291,"num_write_kb":402624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1181,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.680827+0000","last_change":"2024-09-13T14:45:19.804868+0000","last_active":"2024-09-13T14:50:15.680827+0000","last_peered":"2024-09-13T14:50:15.680827+0000","last_clean":"2024-09-13T14:50:15.680827+0000","last_became_active":"2024-09-13T14:45:19.804520+0000","last_became_peered":"2024-09-13T14:45:19.804520+0000","last_unstale":"2024-09-13T14:50:15.680827+0000","last_undegraded":"2024-09-13T14:50:15.680827+0000","last_fullsized":"2024-09-13T14:50:15.680827+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:17:53.924629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":1158,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.678412+0000","last_change":"2024-09-13T14:43:07.533955+0000","last_active":"2024-09-13T14:50:15.678412+0000","last_peered":"2024-09-13T14:50:15.678412+0000","last_clean":"2024-09-13T14:50:15.678412+0000","last_became_active":"2024-09-13T14:43:07.516816+0000","last_became_peered":"2024-09-13T14:43:07.516816+0000","last_unstale":"2024-09-13T14:50:15.678412+0000","last_undegraded":"2024-09-13T14:50:15.678412+0000","last_fullsized":"2024-09-13T14:50:15.678412+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:26.232184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"261'100","reported_seq":68,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.737430+0000","last_change":"2024-09-13T14:45:19.821649+0000","last_active":"2024-09-13T14:50:31.737430+0000","last_peered":"2024-09-13T14:50:31.737430+0000","last_clean":"2024-09-13T14:50:31.737430+0000","last_became_active":"2024-09-13T14:45:19.812346+0000","last_became_peered":"2024-09-13T14:45:19.812346+0000","last_unstale":"2024-09-13T14:50:31.737430+0000","last_undegraded":"2024-09-13T14:50:31.737430+0000","last_fullsized":"2024-09-13T14:50:31.737430+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.597568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.597568+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:45.218892+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"1174'6092","reported_seq":6107,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:42.772154+0000","last_change":"2024-09-13T14:49:22.120890+0000","last_active":"2024-09-13T14:50:42.772154+0000","last_peered":"2024-09-13T14:50:42.772154+0000","last_clean":"2024-09-13T14:50:42.772154+0000","last_became_active":"2024-09-13T14:49:22.120420+0000","last_became_peered":"2024-09-13T14:49:22.120420+0000","last_unstale":"2024-09-13T14:50:42.772154+0000","last_undegraded":"2024-09-13T14:50:42.772154+0000","last_fullsized":"2024-09-13T14:50:42.772154+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":6092,"log_dups_size":0,"ondisk_log_size":6092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:03:49.271123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399245312,"num_objects":6092,"num_object_clones":0,"num_object_copies":18276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6092,"num_write_kb":389888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1099,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.673803+0000","last_change":"2024-09-13T14:19:26.887839+0000","last_active":"2024-09-13T14:50:15.673803+0000","last_peered":"2024-09-13T14:50:15.673803+0000","last_clean":"2024-09-13T14:50:15.673803+0000","last_became_active":"2024-09-13T14:19:26.887498+0000","last_became_peered":"2024-09-13T14:19:26.887498+0000","last_unstale":"2024-09-13T14:50:15.673803+0000","last_undegraded":"2024-09-13T14:50:15.673803+0000","last_fullsized":"2024-09-13T14:50:15.673803+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:24.475263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":1129,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.677428+0000","last_change":"2024-09-13T14:28:35.608607+0000","last_active":"2024-09-13T14:50:15.677428+0000","last_peered":"2024-09-13T14:50:15.677428+0000","last_clean":"2024-09-13T14:50:15.677428+0000","last_became_active":"2024-09-13T14:28:35.608254+0000","last_became_peered":"2024-09-13T14:28:35.608254+0000","last_unstale":"2024-09-13T14:50:15.677428+0000","last_undegraded":"2024-09-13T14:50:15.677428+0000","last_fullsized":"2024-09-13T14:50:15.677428+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:24.019173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"261'100","reported_seq":65,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.676490+0000","last_change":"2024-09-13T14:44:31.191431+0000","last_active":"2024-09-13T14:50:15.676490+0000","last_peered":"2024-09-13T14:50:15.676490+0000","last_clean":"2024-09-13T14:50:15.676490+0000","last_became_active":"2024-09-13T14:43:14.557376+0000","last_became_peered":"2024-09-13T14:43:14.557376+0000","last_unstale":"2024-09-13T14:50:15.676490+0000","last_undegraded":"2024-09-13T14:50:15.676490+0000","last_fullsized":"2024-09-13T14:50:15.676490+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.191326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.191326+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:15:27.352698+0000","scrub_duration":0.0025317360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1119,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.678712+0000","last_change":"2024-09-13T14:28:35.603724+0000","last_active":"2024-09-13T14:50:15.678712+0000","last_peered":"2024-09-13T14:50:15.678712+0000","last_clean":"2024-09-13T14:50:15.678712+0000","last_became_active":"2024-09-13T14:28:35.603316+0000","last_became_peered":"2024-09-13T14:28:35.603316+0000","last_unstale":"2024-09-13T14:50:15.678712+0000","last_undegraded":"2024-09-13T14:50:15.678712+0000","last_fullsized":"2024-09-13T14:50:15.678712+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:47.668223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":64,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.739360+0000","last_change":"2024-09-13T14:45:19.811466+0000","last_active":"2024-09-13T14:50:31.739360+0000","last_peered":"2024-09-13T14:50:31.739360+0000","last_clean":"2024-09-13T14:50:31.739360+0000","last_became_active":"2024-09-13T14:45:19.810617+0000","last_became_peered":"2024-09-13T14:45:19.810617+0000","last_unstale":"2024-09-13T14:50:31.739360+0000","last_undegraded":"2024-09-13T14:50:31.739360+0000","last_fullsized":"2024-09-13T14:50:31.739360+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:16.246669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"261'100","reported_seq":65,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.671565+0000","last_change":"2024-09-13T14:44:50.805306+0000","last_active":"2024-09-13T14:50:15.671565+0000","last_peered":"2024-09-13T14:50:15.671565+0000","last_clean":"2024-09-13T14:50:15.671565+0000","last_became_active":"2024-09-13T14:43:14.562922+0000","last_became_peered":"2024-09-13T14:43:14.562922+0000","last_unstale":"2024-09-13T14:50:15.671565+0000","last_undegraded":"2024-09-13T14:50:15.671565+0000","last_fullsized":"2024-09-13T14:50:15.671565+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:50.805210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:50.805210+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:28:38.529599+0000","scrub_duration":0.0042900009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1111,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.676448+0000","last_change":"2024-09-13T14:19:25.873088+0000","last_active":"2024-09-13T14:50:15.676448+0000","last_peered":"2024-09-13T14:50:15.676448+0000","last_clean":"2024-09-13T14:50:15.676448+0000","last_became_active":"2024-09-13T14:19:25.872781+0000","last_became_peered":"2024-09-13T14:19:25.872781+0000","last_unstale":"2024-09-13T14:50:15.676448+0000","last_undegraded":"2024-09-13T14:50:15.676448+0000","last_fullsized":"2024-09-13T14:50:15.676448+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:05.607382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1732,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.736817+0000","last_change":"2024-09-13T14:45:19.813876+0000","last_active":"2024-09-13T14:50:31.736817+0000","last_peered":"2024-09-13T14:50:31.736817+0000","last_clean":"2024-09-13T14:50:31.736817+0000","last_became_active":"2024-09-13T14:45:19.809136+0000","last_became_peered":"2024-09-13T14:45:19.809136+0000","last_unstale":"2024-09-13T14:50:31.736817+0000","last_undegraded":"2024-09-13T14:50:31.736817+0000","last_fullsized":"2024-09-13T14:50:31.736817+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:50.013528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"261'100","reported_seq":138,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.761415+0000","last_change":"2024-09-13T14:45:19.810654+0000","last_active":"2024-09-13T14:50:31.761415+0000","last_peered":"2024-09-13T14:50:31.761415+0000","last_clean":"2024-09-13T14:50:31.761415+0000","last_became_active":"2024-09-13T14:45:19.810066+0000","last_became_peered":"2024-09-13T14:45:19.810066+0000","last_unstale":"2024-09-13T14:50:31.761415+0000","last_undegraded":"2024-09-13T14:50:31.761415+0000","last_fullsized":"2024-09-13T14:50:31.761415+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:19.978464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:19.978464+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:33.639332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1779,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.739573+0000","last_change":"2024-09-13T14:45:19.809166+0000","last_active":"2024-09-13T14:50:31.739573+0000","last_peered":"2024-09-13T14:50:31.739573+0000","last_clean":"2024-09-13T14:50:31.739573+0000","last_became_active":"2024-09-13T14:45:19.808631+0000","last_became_peered":"2024-09-13T14:45:19.808631+0000","last_unstale":"2024-09-13T14:50:31.739573+0000","last_undegraded":"2024-09-13T14:50:31.739573+0000","last_fullsized":"2024-09-13T14:50:31.739573+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:38.822245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1136,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.674541+0000","last_change":"2024-09-13T14:45:19.804229+0000","last_active":"2024-09-13T14:50:15.674541+0000","last_peered":"2024-09-13T14:50:15.674541+0000","last_clean":"2024-09-13T14:50:15.674541+0000","last_became_active":"2024-09-13T14:45:19.803929+0000","last_became_peered":"2024-09-13T14:45:19.803929+0000","last_unstale":"2024-09-13T14:50:15.674541+0000","last_undegraded":"2024-09-13T14:50:15.674541+0000","last_fullsized":"2024-09-13T14:50:15.674541+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:33:53.675568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"261'100","reported_seq":65,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.673504+0000","last_change":"2024-09-13T14:44:42.654506+0000","last_active":"2024-09-13T14:50:15.673504+0000","last_peered":"2024-09-13T14:50:15.673504+0000","last_clean":"2024-09-13T14:50:15.673504+0000","last_became_active":"2024-09-13T14:43:14.563451+0000","last_became_peered":"2024-09-13T14:43:14.563451+0000","last_unstale":"2024-09-13T14:50:15.673504+0000","last_undegraded":"2024-09-13T14:50:15.673504+0000","last_fullsized":"2024-09-13T14:50:15.673504+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:42.654455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:42.654455+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:57:32.251597+0000","scrub_duration":0.001494097,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":1130,"reported_epoch":1173,"state":"active+clean","last_fresh":"2024-09-13T14:50:15.676954+0000","last_change":"2024-09-13T14:29:42.541093+0000","last_active":"2024-09-13T14:50:15.676954+0000","last_peered":"2024-09-13T14:50:15.676954+0000","last_clean":"2024-09-13T14:50:15.676954+0000","last_became_active":"2024-09-13T14:29:42.540671+0000","last_became_peered":"2024-09-13T14:29:42.540671+0000","last_unstale":"2024-09-13T14:50:15.676954+0000","last_undegraded":"2024-09-13T14:50:15.676954+0000","last_fullsized":"2024-09-13T14:50:15.676954+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:45.722376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1670,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.737679+0000","last_change":"2024-09-13T14:43:07.518679+0000","last_active":"2024-09-13T14:50:31.737679+0000","last_peered":"2024-09-13T14:50:31.737679+0000","last_clean":"2024-09-13T14:50:31.737679+0000","last_became_active":"2024-09-13T14:43:07.518333+0000","last_became_peered":"2024-09-13T14:43:07.518333+0000","last_unstale":"2024-09-13T14:50:31.737679+0000","last_undegraded":"2024-09-13T14:50:31.737679+0000","last_fullsized":"2024-09-13T14:50:31.737679+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:20.616274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"261'100","reported_seq":440,"reported_epoch":1174,"state":"active+clean","last_fresh":"2024-09-13T14:50:31.745726+0000","last_change":"2024-09-13T14:45:19.809268+0000","last_active":"2024-09-13T14:50:31.745726+0000","last_peered":"2024-09-13T14:50:31.745726+0000","last_clean":"2024-09-13T14:50:31.745726+0000","last_became_active":"2024-09-13T14:45:19.808772+0000","last_became_peered":"2024-09-13T14:45:19.808772+0000","last_unstale":"2024-09-13T14:50:31.745726+0000","last_undegraded":"2024-09-13T14:50:31.745726+0000","last_fullsized":"2024-09-13T14:50:31.745726+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:14.654603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:14.654603+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:40.964553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":6440222720,"num_objects":98270,"num_object_clones":0,"num_object_copies":294810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98270,"num_write_kb":6289280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19366871040,"data_stored":19366871040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98070,"ondisk_log_size":98070,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430608,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2290324,"kb_used_data":1992972,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":91454828,"statfs":{"total":95995035648,"available":93649743872,"internally_reserved":0,"allocated":2040803328,"data_stored":2037174221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626837,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2844104,"kb_used_data":2250048,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":90901048,"statfs":{"total":95995035648,"available":93082673152,"internally_reserved":0,"allocated":2304049152,"data_stored":2300431670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823066,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3475268,"kb_used_data":3173756,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":90269884,"statfs":{"total":95995035648,"available":92436361216,"internally_reserved":0,"allocated":3249926144,"data_stored":3246309120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,4,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019294,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2277836,"kb_used_data":1976324,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91467316,"statfs":{"total":95995035648,"available":93662531584,"internally_reserved":0,"allocated":2023755776,"data_stored":2020134247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059592951,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2589496,"kb_used_data":2308464,"kb_used_omap":26,"kb_used_meta":280997,"kb_avail":91155656,"statfs":{"total":95995035648,"available":93343391744,"internally_reserved":0,"allocated":2363867136,"data_stored":2360261307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":287741088},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":591,"seq":2538325672190,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2882496,"kb_used_data":2384760,"kb_used_omap":41,"kb_used_meta":497686,"kb_avail":90862656,"statfs":{"total":95995035648,"available":93043359744,"internally_reserved":0,"allocated":2441994240,"data_stored":2438381285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42651,"internal_metadata":509630821},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":557,"seq":2392296784133,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3221176,"kb_used_data":2790064,"kb_used_omap":32,"kb_used_meta":431071,"kb_avail":90523976,"statfs":{"total":95995035648,"available":92696551424,"internally_reserved":0,"allocated":2857025536,"data_stored":2853419652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33752,"internal_metadata":441416744},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":524,"seq":2250562863371,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2683964,"kb_used_data":2295860,"kb_used_omap":41,"kb_used_meta":388054,"kb_avail":91061188,"statfs":{"total":95995035648,"available":93246656512,"internally_reserved":0,"allocated":2350960640,"data_stored":2347351241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42694,"internal_metadata":397367610},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2344157184,"data_stored":2344157184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2868379648,"data_stored":2868379648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2447704064,"data_stored":2447704064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2370109440,"data_stored":2370109440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2027683840,"data_stored":2027683840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3253075968,"data_stored":3253075968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2017525760,"data_stored":2017525760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2038235136,"data_stored":2038235136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:50:47.489 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-13T14:50:47.489 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 51 2024-09-13T14:50:47.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:47.491 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:47.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:47.592 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:47.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:47.636 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:47.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:47.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:47.787 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:47.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:47.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:47.889 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:47.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:47.920 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:47.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:48.021 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:48.073 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:48.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:48.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:48.175 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:48.212 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:48.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:48.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:48.225 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:50:48.225 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:50:48.231 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 69 16 104588 104572 94.6998 56.5625 0.0278269 0.0105534 2024-09-13T14:50:48.315 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:48.360 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:48.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:48.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:48.461 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:48.508 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:48.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:48.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:48.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:48.643 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:48.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:48.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:48.745 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:48.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:48.790 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:48.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:48.811 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:50:48.892 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:48.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:48.931 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:48.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:49.032 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:49.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:49.080 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:49.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:49.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:49 smithi151 ceph-mon[104450]: pgmap v1721: 119 pgs: 119 active+clean; 6.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 53 MiB/s wr, 847 op/s 2024-09-13T14:50:49.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:49 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:50:49.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:49 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:50:49.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:49 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2225423337' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:49.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:49 smithi151 ceph-mon[108966]: pgmap v1721: 119 pgs: 119 active+clean; 6.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 53 MiB/s wr, 847 op/s 2024-09-13T14:50:49.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:49 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:50:49.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:49 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:50:49.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:49 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2225423337' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:49.115 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-13T14:50:49.181 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:49.231 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 70 16 105508 105492 94.1683 57.5 0.0211491 0.0106143 2024-09-13T14:50:49.232 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:49.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:49.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:49.334 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:49.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:49 smithi184 ceph-mon[92527]: pgmap v1721: 119 pgs: 119 active+clean; 6.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 53 MiB/s wr, 847 op/s 2024-09-13T14:50:49.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:49 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:50:49.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:49 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:50:49.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:49 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2225423337' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:49.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:49.376 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:49.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:49.475 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T14:50:49.477 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:49.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:49.501 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:49.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:49.603 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:49.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:49.666 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:49.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:49.767 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:49.792 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:49.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:49.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:49.894 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:49.935 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:49.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:49.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:50.037 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:50.079 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:50.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:50.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:50.181 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:50.209 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:50.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:50.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:50.232 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 71 16 106459 106443 93.6785 59.4375 0.0144534 0.0106698 2024-09-13T14:50:50.237 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:50 smithi151 ceph-mon[104450]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:50:50.237 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:50 smithi151 ceph-mon[104450]: Cluster is now healthy 2024-09-13T14:50:50.237 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:50 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2225423337' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:50:50.237 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:50 smithi151 ceph-mon[104450]: osdmap e1175: 8 total, 8 up, 8 in 2024-09-13T14:50:50.237 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:50 smithi151 ceph-mon[108966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:50:50.237 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:50 smithi151 ceph-mon[108966]: Cluster is now healthy 2024-09-13T14:50:50.237 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:50 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2225423337' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:50:50.237 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:50 smithi151 ceph-mon[108966]: osdmap e1175: 8 total, 8 up, 8 in 2024-09-13T14:50:50.311 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:50.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:50.343 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:50.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:50.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:50 smithi184 ceph-mon[92527]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:50:50.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:50 smithi184 ceph-mon[92527]: Cluster is now healthy 2024-09-13T14:50:50.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:50 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2225423337' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:50:50.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:50 smithi184 ceph-mon[92527]: osdmap e1175: 8 total, 8 up, 8 in 2024-09-13T14:50:50.354 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:50:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:50:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:50:50.444 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:50.479 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:50.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:50.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:50.562 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:50.588 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:50.613 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:50.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:50.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:50.715 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:50.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:50.760 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:50.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:50.861 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:50.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:50.900 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:50.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:51.001 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:51.013 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:51.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:51.061 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:51.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:51.162 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:51.189 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:51.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:51.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:51.232 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 72 16 107442 107426 93.2307 61.4375 0.0160332 0.0107208 2024-09-13T14:50:51.291 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:51.323 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:51.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:51.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:51.425 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:51.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:51.449 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:51.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:51.551 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:51.577 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:51.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:51.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:51.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:51 smithi184 ceph-mon[92527]: pgmap v1723: 119 pgs: 119 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 61 MiB/s wr, 979 op/s 2024-09-13T14:50:51.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:51 smithi151 ceph-mon[104450]: pgmap v1723: 119 pgs: 119 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 61 MiB/s wr, 979 op/s 2024-09-13T14:50:51.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:51 smithi151 ceph-mon[108966]: pgmap v1723: 119 pgs: 119 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 61 MiB/s wr, 979 op/s 2024-09-13T14:50:51.679 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:51.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:51.719 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:51.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:51.821 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:51.851 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:51.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:51.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:51.954 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:51.989 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:51.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:51.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:52.091 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:52.123 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:52.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:52.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:52.225 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:52.232 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 73 16 108389 108373 92.7643 59.1875 0.00833454 0.0107743 2024-09-13T14:50:52.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:52.254 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:52.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:52.355 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:52.393 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:52.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:52.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:52.495 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:52.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:52.528 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:52.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:52.605 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:50:52.630 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:52.657 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:52.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:52.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:52.759 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:52.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:52.789 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:52.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:52.889 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:52.917 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:52.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:52.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:53.019 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:53.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:53.058 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:53.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:53.159 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:53.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:53.187 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:53.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:53.232 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 74 16 109446 109430 92.4035 66.0625 0.00909502 0.0108167 2024-09-13T14:50:53.288 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:53.321 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:53.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:53.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:53.422 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:53.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:53.446 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:53.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:53.548 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:53.576 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:53.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:53.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:53.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:53 smithi184 ceph-mon[92527]: pgmap v1724: 119 pgs: 119 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 61 MiB/s wr, 978 op/s 2024-09-13T14:50:53.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:53 smithi151 ceph-mon[104450]: pgmap v1724: 119 pgs: 119 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 61 MiB/s wr, 978 op/s 2024-09-13T14:50:53.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:53 smithi151 ceph-mon[108966]: pgmap v1724: 119 pgs: 119 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 61 MiB/s wr, 978 op/s 2024-09-13T14:50:53.678 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:53.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:53.711 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:53.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:53.812 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:53.839 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:53.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:53.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:53.938 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:53.942 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:53.966 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:53.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:53.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:54.068 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:54.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:54.118 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:54.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:54.220 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:54.233 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 75 16 110396 110380 91.9621 59.375 0.00993944 0.0108686 2024-09-13T14:50:54.263 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:54.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:54.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:54.365 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:54.396 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:54.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:54.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:54.498 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:54.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:54.567 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:54.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:54.668 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:54.709 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:54.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:54.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:54.811 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:54.842 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:54.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:54.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:54.861 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:50:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:50:54] "GET /metrics HTTP/1.1" 200 36642 "" "Prometheus/2.51.0" 2024-09-13T14:50:54.944 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:54.975 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:54.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:54.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:55.077 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:55.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:55.114 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:55.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:55.215 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:55.233 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 76 15 111421 111406 91.5958 64.125 0.0161577 0.0109126 2024-09-13T14:50:55.248 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:55.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:55.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:55.351 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:55.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:55 smithi151 ceph-mon[104450]: pgmap v1725: 119 pgs: 119 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 61 MiB/s wr, 968 op/s 2024-09-13T14:50:55.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:55 smithi151 ceph-mon[108966]: pgmap v1725: 119 pgs: 119 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 61 MiB/s wr, 968 op/s 2024-09-13T14:50:55.388 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:55.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:55.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:55.490 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:55.523 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:55.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:55.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:55.608 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:55 smithi184 ceph-mon[92527]: pgmap v1725: 119 pgs: 119 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 61 MiB/s wr, 968 op/s 2024-09-13T14:50:55.625 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:55.665 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:55.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:55.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:55.767 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:55.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:55.806 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:55.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:55.907 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:55.962 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:55.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:55.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:56.063 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:56.092 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:56.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:56.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:56.194 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:56.233 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:56.233 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 77 16 112468 112452 91.2553 65.375 0.0269481 0.0109539 2024-09-13T14:50:56.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:56.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:56.335 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:56.370 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:56.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:56.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:56.472 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:56.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:56.538 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:56.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:56.556 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:56.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:56.686 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:56.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:56.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:56.788 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:56.816 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:56.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:56.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:56.918 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:56.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:56.954 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:56.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:57.054 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:57.086 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:57.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:57.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:57.188 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:57.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:57.223 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:57.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:57.233 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 78 16 113481 113465 90.897 63.3125 0.0182725 0.0109963 2024-09-13T14:50:57.324 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:57.351 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:57.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:57.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:57.417 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:57 smithi151 ceph-mon[104450]: pgmap v1726: 119 pgs: 119 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 60 MiB/s wr, 957 op/s 2024-09-13T14:50:57.418 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:57 smithi151 ceph-mon[108966]: pgmap v1726: 119 pgs: 119 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 60 MiB/s wr, 957 op/s 2024-09-13T14:50:57.453 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:57.490 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:57.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:57.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:57.593 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:57.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:57 smithi184 ceph-mon[92527]: pgmap v1726: 119 pgs: 119 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 60 MiB/s wr, 957 op/s 2024-09-13T14:50:57.629 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:57.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:57.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:57.731 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:57.760 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:57.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:57.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:57.864 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:57.893 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:57.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:57.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:57.994 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:58.023 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:58.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:58.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:58.125 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:58.162 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:58.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:58.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:58.233 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 79 16 114455 114439 90.517 60.875 0.0127615 0.011043 2024-09-13T14:50:58.264 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:58.290 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:58.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:58.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:58.392 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:58.419 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:58.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:58.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:58.521 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:58.560 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:58.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:58.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:58.662 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:58.712 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:58.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:58.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:58.814 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:58.854 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:58.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:58.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:58.905 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: specified pg_num 51 < pg_num_max 32 2024-09-13T14:50:58.937 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:50:58.937 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:50:58.956 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:58.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:58.989 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:58.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:59.090 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:59.135 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:59.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:59.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:59.234 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:2024-09-13T14:50:59.233363+0000 min lat: 0.00100112 max lat: 0.247171 avg lat: 0.0110879 2024-09-13T14:50:59.234 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:50:59.234 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 80 16 115427 115411 90.1449 60.75 0.0270659 0.0110879 2024-09-13T14:50:59.237 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:59.274 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:59.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:59.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:59.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:59 smithi151 ceph-mon[104450]: pgmap v1727: 119 pgs: 119 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 60 MiB/s wr, 960 op/s 2024-09-13T14:50:59.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:59 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1199846107' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"} : dispatch 2024-09-13T14:50:59.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:50:59 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"} : dispatch 2024-09-13T14:50:59.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:59 smithi151 ceph-mon[108966]: pgmap v1727: 119 pgs: 119 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 60 MiB/s wr, 960 op/s 2024-09-13T14:50:59.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:59 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1199846107' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"} : dispatch 2024-09-13T14:50:59.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:50:59 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"} : dispatch 2024-09-13T14:50:59.375 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:59.402 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:59.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:59.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:59.504 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:59.537 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:59.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:59.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:59.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:59 smithi184 ceph-mon[92527]: pgmap v1727: 119 pgs: 119 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 60 MiB/s wr, 960 op/s 2024-09-13T14:50:59.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:59 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1199846107' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"} : dispatch 2024-09-13T14:50:59.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:50:59 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"} : dispatch 2024-09-13T14:50:59.639 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:59.665 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:59.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:59.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:59.767 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:59.801 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:50:59.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:59.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:59.904 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:59.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:59.947 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:50:59.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:00.048 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:00.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:00.096 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:00.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:00.197 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:00.232 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:00.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:00.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:00.234 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 81 16 116446 116430 89.8182 63.6875 0.031178 0.0111285 2024-09-13T14:51:00.335 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:00.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:00.366 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:00.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:00.467 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:00.495 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:00.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:00.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:00.597 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:00.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:51:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:51:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:51:00.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:00.636 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:00.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:00.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:00.770 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:00.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:00.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:00.871 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:00.901 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:00.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:00.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:01.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:01.039 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:01.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:01.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:01.140 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:01.174 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:01.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:01.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:01.233 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 82 16 117508 117492 89.5323 66.375 0.0115562 0.0111644 2024-09-13T14:51:01.277 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:01.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:01.312 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:01.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:01.357 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T14:51:01.414 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:01.424 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:01 smithi151 ceph-mon[104450]: pgmap v1728: 119 pgs: 119 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 58 MiB/s wr, 925 op/s 2024-09-13T14:51:01.424 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:01 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/494096696' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:01.425 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:01 smithi151 ceph-mon[108966]: pgmap v1728: 119 pgs: 119 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 58 MiB/s wr, 925 op/s 2024-09-13T14:51:01.425 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:01 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/494096696' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:01.472 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:01.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:01.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:01.574 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:01.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:01 smithi184 ceph-mon[92527]: pgmap v1728: 119 pgs: 119 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 58 MiB/s wr, 925 op/s 2024-09-13T14:51:01.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:01 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/494096696' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:01.622 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:01.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:01.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:01.724 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:01.759 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:51:01.760 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:51:01.760 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:51:01.760 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:51:01.760 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:51:01.760 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:51:01.760 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:51:01.760 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:51:01.760 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:51:01.761 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:51:01.761 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:51:01.761 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:51:01.761 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:51:01.761 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:51:01.761 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:51:01.761 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:51:01.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:01.762 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:01.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:01.864 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:01.900 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:01.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:01.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:02.002 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:02.037 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:02.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:02.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:02.139 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:02.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:02.178 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:02.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:02.234 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 83 16 118481 118465 89.1862 60.8125 0.00862023 0.0112073 2024-09-13T14:51:02.279 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:02.310 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:02.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:02.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:02.411 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:02.465 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:02.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:02.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:02.567 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:02.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:02.602 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:02.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:02.704 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:02.741 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:02.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:02.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:02.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:51:02.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:02.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:02 smithi151 ceph-mon[104450]: pgmap v1729: 119 pgs: 119 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 61 MiB/s wr, 968 op/s 2024-09-13T14:51:02.803 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:51:02.803 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:02.803 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:02 smithi151 ceph-mon[108966]: pgmap v1729: 119 pgs: 119 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 61 MiB/s wr, 968 op/s 2024-09-13T14:51:02.843 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:02.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:51:02.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:02.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:02 smithi184 ceph-mon[92527]: pgmap v1729: 119 pgs: 119 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 61 MiB/s wr, 968 op/s 2024-09-13T14:51:02.872 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:02.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:02.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:02.974 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:03.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:03.003 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:03.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:03.104 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:03.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:03.148 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:03.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:03.234 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 84 16 119507 119491 88.8878 64.125 0.0111454 0.0112451 2024-09-13T14:51:03.250 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:03.288 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:03.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:03.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:03.390 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:03.407 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T14:51:03.422 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:03.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:03.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:03.524 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:03.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:03.564 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:03.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:03.665 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:03.700 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:03.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:03.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:03.802 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:03.835 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:03.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:03.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:03.936 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:03.963 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:03.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:03.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:04.065 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:04.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:04.125 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:04.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:04.226 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:04.234 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 85 16 120478 120462 88.5561 60.6875 0.0349525 0.0112872 2024-09-13T14:51:04.257 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:04.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:04.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:04.359 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:04.403 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:04.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:04.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:04.453 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:51:04.505 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:04.529 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:04.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:04.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:04.631 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:04.662 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:04.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:04.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:04.697 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-13T14:51:04.764 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:04.813 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:04.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:04.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:04.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:04 smithi151 ceph-mon[104450]: pgmap v1730: 119 pgs: 119 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-13T14:51:04.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:04 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3936862041' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:51:04.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:51:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:51:04] "GET /metrics HTTP/1.1" 200 36643 "" "Prometheus/2.51.0" 2024-09-13T14:51:04.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:04 smithi151 ceph-mon[108966]: pgmap v1730: 119 pgs: 119 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-13T14:51:04.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:04 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3936862041' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:51:04.917 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:04.943 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:04.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:04.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:05.045 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:05.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:05.075 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:05.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:05.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:04 smithi184 ceph-mon[92527]: pgmap v1730: 119 pgs: 119 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-13T14:51:05.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:04 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3936862041' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:51:05.176 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:05.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:05.219 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:05.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:05.234 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 86 16 121358 121342 88.1659 55 0.0223479 0.0113374 2024-09-13T14:51:05.320 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:05.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:05.360 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:05.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:05.461 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:05.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:05.503 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:05.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:05.604 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:05.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:05.679 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:05.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:05.780 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:05.802 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:05.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:05.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:05.904 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:05.943 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:05.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:05.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:06.046 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:06.089 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:06.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:06.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:06.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:05 smithi184 ceph-mon[92527]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:06.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:05 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3936862041' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:51:06.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:05 smithi184 ceph-mon[92527]: osdmap e1176: 8 total, 8 up, 8 in 2024-09-13T14:51:06.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:05 smithi151 ceph-mon[104450]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:06.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:05 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3936862041' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:51:06.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:05 smithi151 ceph-mon[104450]: osdmap e1176: 8 total, 8 up, 8 in 2024-09-13T14:51:06.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:05 smithi151 ceph-mon[108966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:06.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:05 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3936862041' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:51:06.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:05 smithi151 ceph-mon[108966]: osdmap e1176: 8 total, 8 up, 8 in 2024-09-13T14:51:06.191 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:06.224 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:06.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:06.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:06.234 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 87 16 122124 122108 87.7028 47.875 0.0187051 0.0113972 2024-09-13T14:51:06.326 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:06.359 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:06.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:06.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:06.461 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:06.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:06.483 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:06.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:06.584 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:06.620 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:06.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:06.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:06.722 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:06.763 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:06.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:06.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:06.865 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:06.893 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:06.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:06.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:06.995 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:07.026 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:07.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:07.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:07.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:06 smithi184 ceph-mon[92527]: pgmap v1732: 119 pgs: 119 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T14:51:07.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:06 smithi151 ceph-mon[104450]: pgmap v1732: 119 pgs: 119 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T14:51:07.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:06 smithi151 ceph-mon[108966]: pgmap v1732: 119 pgs: 119 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T14:51:07.128 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:07.170 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:07.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:07.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:07.234 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 88 16 123067 123051 87.3759 58.9375 0.00787963 0.0114396 2024-09-13T14:51:07.273 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:07.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:07.307 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:07.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:07.408 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:07.434 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:07.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:07.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:07.536 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:07.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:07.577 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:07.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:07.678 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:07.704 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:07.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:07.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:07.806 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:07.833 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:07.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:07.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:07.935 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:07.959 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:07.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:07.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:08.061 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:08.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:08.094 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:08.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:08.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:08.228 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:08.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:08.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:08.234 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 89 16 124002 123986 87.0507 58.4375 0.0209506 0.0114823 2024-09-13T14:51:08.330 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:08.364 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:08.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:08.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:08.409 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:51:08.409 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:51:08.409 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-13T14:51:08.410 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd out 4 2024-09-13T14:51:08.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:08.507 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:08.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:08.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:08.609 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:08.636 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:08.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:08.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:08.738 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:08.786 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:08.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:08.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:08.843 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:08.888 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:08.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:08.913 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:08.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:08.989 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:08 smithi151 ceph-mon[104450]: pgmap v1733: 119 pgs: 119 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T14:51:08.989 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:08 smithi151 ceph-mon[108966]: pgmap v1733: 119 pgs: 119 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T14:51:09.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:09.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:09.061 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:09.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:09.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:08 smithi184 ceph-mon[92527]: pgmap v1733: 119 pgs: 119 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T14:51:09.162 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:09.188 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:09.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:09.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:09.234 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Total time run: 90.0067 2024-09-13T14:51:09.235 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Total writes made: 124865 2024-09-13T14:51:09.235 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Write size: 65536 2024-09-13T14:51:09.235 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Object size: 65536 2024-09-13T14:51:09.235 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Bandwidth (MB/sec): 86.7054 2024-09-13T14:51:09.235 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Stddev Bandwidth: 36.4223 2024-09-13T14:51:09.235 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Max bandwidth (MB/sec): 223.312 2024-09-13T14:51:09.235 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Min bandwidth (MB/sec): 47.875 2024-09-13T14:51:09.235 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Average IOPS: 1387 2024-09-13T14:51:09.235 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Stddev IOPS: 582.757 2024-09-13T14:51:09.235 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Max IOPS: 3573 2024-09-13T14:51:09.235 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Min IOPS: 766 2024-09-13T14:51:09.235 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Average Latency(s): 0.0115291 2024-09-13T14:51:09.235 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Stddev Latency(s): 0.0108737 2024-09-13T14:51:09.236 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Max latency(s): 0.247171 2024-09-13T14:51:09.236 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Min latency(s): 0.00100112 2024-09-13T14:51:09.238 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T14:51:09.290 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:09.347 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:09.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:09.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:09.449 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:09.485 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:09.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:09.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:09.587 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:09.631 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:09.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:09.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:09.733 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:09.770 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:09.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:09.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:09.872 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:09.908 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:09.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:09.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:10.010 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:10.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:10.050 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:10.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:10.151 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:10.176 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:10.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:10.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:10.278 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:10.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:10.311 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:10.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:10.401 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:10.412 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:10.437 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:10.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:10.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:10.539 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:10.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:10.570 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:10.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:10.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:51:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:51:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:51:10.671 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:10.704 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:10.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:10.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:10.806 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:10.840 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:10.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:10.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:10.888 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:51:10.941 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:10.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:10.984 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:10.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:11.085 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:11.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:11.135 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:11.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:11.236 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:11.244 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:11.260 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:11.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:11.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:11.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:10 smithi184 ceph-mon[92527]: pgmap v1734: 119 pgs: 119 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 61 MiB/s wr, 982 op/s 2024-09-13T14:51:11.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:10 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:51:11.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:10 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:51:11.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:10 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:51:11.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:10 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:51:11.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:10 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:51:11.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:10 smithi151 ceph-mon[104450]: pgmap v1734: 119 pgs: 119 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 61 MiB/s wr, 982 op/s 2024-09-13T14:51:11.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:10 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:51:11.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:10 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:51:11.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:10 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:51:11.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:10 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:51:11.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:10 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:51:11.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:10 smithi151 ceph-mon[108966]: pgmap v1734: 119 pgs: 119 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 61 MiB/s wr, 982 op/s 2024-09-13T14:51:11.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:10 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:51:11.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:10 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:51:11.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:10 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:51:11.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:10 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:51:11.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:10 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:51:11.361 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:11.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:11.416 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:11.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:11.517 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:11.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:11.552 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:11.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:11.653 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:11.681 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:11.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:11.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:11.783 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:11.826 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:11.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:11.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:11.928 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:11.971 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:11.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:11.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:12.073 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:12.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:12.106 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:12.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:12.207 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:12.239 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:12.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:12.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:12.341 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:12.376 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:12.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:12.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:12.478 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:12.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:12.508 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:12.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:12.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:12.646 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:12.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:12.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:12.748 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:12.786 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:12.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:12.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:12.888 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:12.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:12.925 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:12.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:13.026 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:13.066 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:13.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:13.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:13.168 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:13.205 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:13.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:13.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:13.307 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:13.343 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:13.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:13.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:13.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:12 smithi184 ceph-mon[92527]: pgmap v1735: 119 pgs: 119 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 58 MiB/s wr, 956 op/s 2024-09-13T14:51:13.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:12 smithi151 ceph-mon[104450]: pgmap v1735: 119 pgs: 119 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 58 MiB/s wr, 956 op/s 2024-09-13T14:51:13.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:12 smithi151 ceph-mon[108966]: pgmap v1735: 119 pgs: 119 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 58 MiB/s wr, 956 op/s 2024-09-13T14:51:13.446 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:13.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:13.481 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:13.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:13.581 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:13.588 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:51:13.588 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-13T14:51:13.588 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-13T14:51:13.588 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-13T14:51:13.588 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:51:13.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:13.627 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:13.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:13.728 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:13.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:13.764 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:13.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:13.865 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:13.899 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:13.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:13.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:14.001 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:14.033 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:14.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:14.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:14.135 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:14.164 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:14.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:14.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:14.265 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:14.293 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T14:51:14.308 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:14.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:14.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:14.411 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:14.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:14.438 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:14.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:14.539 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:14.568 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:14.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:14.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:14.670 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:14.700 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:14.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:14.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:14.802 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:14.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:14.845 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:14.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:14.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:51:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:51:14] "GET /metrics HTTP/1.1" 200 36643 "" "Prometheus/2.51.0" 2024-09-13T14:51:14.946 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:14.971 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:14.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:14.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:15.073 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:15.096 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:51:15.096 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:51:15.108 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:15.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:15.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:15.128 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:51:15.129 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:51:15.160 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:51:15.160 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:51:15.161 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:14 smithi151 ceph-mon[104450]: pgmap v1736: 119 pgs: 119 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 42 MiB/s wr, 938 op/s 2024-09-13T14:51:15.161 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:14 smithi151 ceph-mon[108966]: pgmap v1736: 119 pgs: 119 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 42 MiB/s wr, 938 op/s 2024-09-13T14:51:15.199 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:51:15.199 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:51:15.211 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:15.229 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:51:15.230 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:51:15.249 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:15.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:15.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:15.260 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:51:15.260 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:51:15.291 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:51:15.291 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:51:15.308 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:15.323 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:51:15.323 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:51:15.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:15.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:14 smithi184 ceph-mon[92527]: pgmap v1736: 119 pgs: 119 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 42 MiB/s wr, 938 op/s 2024-09-13T14:51:15.410 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:15.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:15.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:15.512 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:15.550 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:15.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:15.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:15.645 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:15.651 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:15.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:15.681 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:15.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:15.782 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:15.822 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:15.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:15.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:15.924 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:15.967 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:15.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:15.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:16.068 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:16.101 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:16.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:16.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:16.202 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:16.232 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:16 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2971446447' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:16.233 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:16 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2971446447' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:16.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:16.244 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:16.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:16.345 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:16.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:16 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2971446447' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:16.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:16.379 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:16.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:16.480 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:16.515 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:16.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:16.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:16.617 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:16.653 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:16.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:16.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:16.755 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:16.786 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:16.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:16.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:16.889 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:16.925 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:16.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:16.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:17.027 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:17.058 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:17.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:17.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:17.159 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:17.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:17.204 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:17.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:17.305 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:17.343 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:17.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:17.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:17.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:17 smithi184 ceph-mon[92527]: pgmap v1737: 119 pgs: 119 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 94 B/s rd, 39 MiB/s wr, 1.04k op/s 2024-09-13T14:51:17.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:51:17.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:17.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:17 smithi151 ceph-mon[104450]: pgmap v1737: 119 pgs: 119 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 94 B/s rd, 39 MiB/s wr, 1.04k op/s 2024-09-13T14:51:17.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:51:17.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:17.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:17 smithi151 ceph-mon[108966]: pgmap v1737: 119 pgs: 119 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 94 B/s rd, 39 MiB/s wr, 1.04k op/s 2024-09-13T14:51:17.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:51:17.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:17.446 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:17.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:17.477 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:17.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:17.578 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:17.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:17.635 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:17.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:17.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:17.776 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:17.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:17.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:17.878 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:17.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:17.907 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:17.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:18.008 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:18.040 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:18.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:18.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:18.143 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:18.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:18.185 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:18.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:18.286 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:18.321 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:18.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:18.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:18.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:18 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a[104427]: 2024-09-13T14:51:18.109+0000 7fe9da14f640 -1 mon.a@0(leader).osd e1176 definitely_dead 0 2024-09-13T14:51:18.422 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:18.461 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:18.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:18.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:18.563 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:18.604 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:18.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:18.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:18.698 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:51:18.706 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:18.736 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:18.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:18.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:18.838 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:18.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:18.882 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:18.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:18.984 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:19.009 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:19.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:19.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:19.040 INFO:teuthology.orchestra.run.smithi151.stderr:marked out osd.4. 2024-09-13T14:51:19.112 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:19.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:19.135 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:19.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:19.236 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:19.259 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:19.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:19.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:19.272 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:19 smithi151 ceph-mon[104450]: pgmap v1738: 119 pgs: 119 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 26 MiB/s wr, 826 op/s 2024-09-13T14:51:19.272 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:19 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1973582648' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-13T14:51:19.273 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:19 smithi151 ceph-mon[104450]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-13T14:51:19.273 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:19 smithi151 ceph-mon[108966]: pgmap v1738: 119 pgs: 119 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 26 MiB/s wr, 826 op/s 2024-09-13T14:51:19.273 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:19 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1973582648' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-13T14:51:19.273 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:19 smithi151 ceph-mon[108966]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-13T14:51:19.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:19 smithi184 ceph-mon[92527]: pgmap v1738: 119 pgs: 119 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 26 MiB/s wr, 826 op/s 2024-09-13T14:51:19.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:19 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1973582648' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-13T14:51:19.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:19 smithi184 ceph-mon[92527]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-13T14:51:19.361 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:19.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:19.384 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:19.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:19.485 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:19.506 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:19.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:19.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:19.608 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:19.629 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:19.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:19.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:19.731 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:19.753 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:19.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:19.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:19.854 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:19.876 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:19.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:19.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:19.978 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:20.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.001 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:20.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:20.103 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:20.124 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:20.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:20.226 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:20.248 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:20.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:20.311 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:20 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1973582648' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-13T14:51:20.311 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:20 smithi151 ceph-mon[104450]: osdmap e1177: 8 total, 8 up, 7 in 2024-09-13T14:51:20.311 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:20 smithi151 ceph-mon[104450]: osdmap e1178: 8 total, 8 up, 7 in 2024-09-13T14:51:20.312 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:20 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1973582648' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-13T14:51:20.312 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:20 smithi151 ceph-mon[108966]: osdmap e1177: 8 total, 8 up, 7 in 2024-09-13T14:51:20.312 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:20 smithi151 ceph-mon[108966]: osdmap e1178: 8 total, 8 up, 7 in 2024-09-13T14:51:20.349 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:20.353 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:51:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:51:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:51:20.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:20 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1973582648' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-13T14:51:20.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:20 smithi184 ceph-mon[92527]: osdmap e1177: 8 total, 8 up, 7 in 2024-09-13T14:51:20.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:20 smithi184 ceph-mon[92527]: osdmap e1178: 8 total, 8 up, 7 in 2024-09-13T14:51:20.370 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:20.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:20.472 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:20.495 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:20.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:20.597 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:20.619 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:20.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:20.721 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:20.743 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:20.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:20.845 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:20.866 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:20.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:20.887 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-13T14:51:20.919 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:51:20.919 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:51:20.968 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:20.991 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:20.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:21.093 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:21.144 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:21.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:21.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:21.247 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:21.286 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:21.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:21.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:21.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:21 smithi184 ceph-mon[92527]: pgmap v1740: 119 pgs: 119 active+clean; 7.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 102 B/s rd, 13 MiB/s wr, 995 op/s 2024-09-13T14:51:21.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:21 smithi184 ceph-mon[92527]: 1.5 scrub starts 2024-09-13T14:51:21.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:21 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1553739031' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:21.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:21 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:21.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:21 smithi184 ceph-mon[92527]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:21.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:21 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:51:21.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:21 smithi184 ceph-mon[92527]: osdmap e1179: 8 total, 8 up, 7 in 2024-09-13T14:51:21.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:21 smithi151 ceph-mon[104450]: pgmap v1740: 119 pgs: 119 active+clean; 7.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 102 B/s rd, 13 MiB/s wr, 995 op/s 2024-09-13T14:51:21.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:21 smithi151 ceph-mon[104450]: 1.5 scrub starts 2024-09-13T14:51:21.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:21 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1553739031' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:21.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:21 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:21.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:21 smithi151 ceph-mon[104450]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:21.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:21 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:51:21.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:21 smithi151 ceph-mon[104450]: osdmap e1179: 8 total, 8 up, 7 in 2024-09-13T14:51:21.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:21 smithi151 ceph-mon[108966]: pgmap v1740: 119 pgs: 119 active+clean; 7.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 102 B/s rd, 13 MiB/s wr, 995 op/s 2024-09-13T14:51:21.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:21 smithi151 ceph-mon[108966]: 1.5 scrub starts 2024-09-13T14:51:21.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:21 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1553739031' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:21.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:21 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:21.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:21 smithi151 ceph-mon[108966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:21.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:21 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:51:21.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:21 smithi151 ceph-mon[108966]: osdmap e1179: 8 total, 8 up, 7 in 2024-09-13T14:51:21.448 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:21.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:21.490 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:21.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:21.591 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:21.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:21.625 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:21.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:21.727 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:21.763 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:21.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:21.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:21.865 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:21.903 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:21.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:21.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:21.906 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:22.005 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:22.034 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:22.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:22.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:22.136 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:22.169 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:22.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:22.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:22.217 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:22 smithi151 ceph-mon[108966]: 8.b starting backfill to osd.2 from (0'0,0'0] MAX to 1176'8499 2024-09-13T14:51:22.217 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:22 smithi151 ceph-mon[108966]: 8.2 starting backfill to osd.3 from (0'0,0'0] MAX to 1176'8468 2024-09-13T14:51:22.217 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:22 smithi151 ceph-mon[108966]: 8.4 starting backfill to osd.6 from (0'0,0'0] MAX to 1176'8322 2024-09-13T14:51:22.217 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:22 smithi151 ceph-mon[108966]: 8.5 starting backfill to osd.0 from (0'0,0'0] MAX to 1176'8502 2024-09-13T14:51:22.217 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:22 smithi151 ceph-mon[108966]: 8.e starting backfill to osd.7 from (0'0,0'0] MAX to 1176'8433 2024-09-13T14:51:22.217 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:22 smithi151 ceph-mon[104450]: 8.b starting backfill to osd.2 from (0'0,0'0] MAX to 1176'8499 2024-09-13T14:51:22.217 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:22 smithi151 ceph-mon[104450]: 8.2 starting backfill to osd.3 from (0'0,0'0] MAX to 1176'8468 2024-09-13T14:51:22.217 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:22 smithi151 ceph-mon[104450]: 8.4 starting backfill to osd.6 from (0'0,0'0] MAX to 1176'8322 2024-09-13T14:51:22.218 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:22 smithi151 ceph-mon[104450]: 8.5 starting backfill to osd.0 from (0'0,0'0] MAX to 1176'8502 2024-09-13T14:51:22.218 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:22 smithi151 ceph-mon[104450]: 8.e starting backfill to osd.7 from (0'0,0'0] MAX to 1176'8433 2024-09-13T14:51:22.271 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:22.304 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:22.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:22.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:22.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:22 smithi184 ceph-mon[92527]: 8.b starting backfill to osd.2 from (0'0,0'0] MAX to 1176'8499 2024-09-13T14:51:22.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:22 smithi184 ceph-mon[92527]: 8.2 starting backfill to osd.3 from (0'0,0'0] MAX to 1176'8468 2024-09-13T14:51:22.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:22 smithi184 ceph-mon[92527]: 8.4 starting backfill to osd.6 from (0'0,0'0] MAX to 1176'8322 2024-09-13T14:51:22.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:22 smithi184 ceph-mon[92527]: 8.5 starting backfill to osd.0 from (0'0,0'0] MAX to 1176'8502 2024-09-13T14:51:22.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:22 smithi184 ceph-mon[92527]: 8.e starting backfill to osd.7 from (0'0,0'0] MAX to 1176'8433 2024-09-13T14:51:22.405 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:22.452 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:22.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:22.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:22.554 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:22.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:22.586 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:22.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:22.687 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:22.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:22.725 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:22.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:22.826 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:22.859 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:22.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:22.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:22.962 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:22.996 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:22.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:22.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:23.098 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:23.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:23.130 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:23.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:23.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:23 smithi151 ceph-mon[104450]: pgmap v1743: 119 pgs: 1 unknown, 12 peering, 106 active+clean; 7.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 838 op/s 2024-09-13T14:51:23.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:23 smithi151 ceph-mon[104450]: Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-13T14:51:23.181 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:23 smithi151 ceph-mon[108966]: pgmap v1743: 119 pgs: 1 unknown, 12 peering, 106 active+clean; 7.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 838 op/s 2024-09-13T14:51:23.181 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:23 smithi151 ceph-mon[108966]: Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-13T14:51:23.221 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T14:51:23.231 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:23.299 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:23.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:23.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:23.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:23 smithi184 ceph-mon[92527]: pgmap v1743: 119 pgs: 1 unknown, 12 peering, 106 active+clean; 7.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 838 op/s 2024-09-13T14:51:23.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:23 smithi184 ceph-mon[92527]: Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-13T14:51:23.401 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:23.454 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:23.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:23.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:23.556 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:23.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:23.592 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:23.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:23.630 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:51:23.694 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:23.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:23.728 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:23.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:23.829 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:23.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:23.865 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:23.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:23.942 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:51:23.966 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:23.997 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:23.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:23.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:24.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:24.143 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:24.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:24.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:24.245 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:24.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:24.286 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:24.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:24.388 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:24.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:24.426 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:24.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:24.528 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:24.572 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:24.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:24.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:24.674 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:24.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:24.712 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:24.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:24.814 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:24.815 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:51:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:51:24] "GET /metrics HTTP/1.1" 200 36645 "" "Prometheus/2.51.0" 2024-09-13T14:51:24.868 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:24.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:24.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:24.934 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:24.970 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:24.994 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:51:24.994 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:51:25.005 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:25.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:25.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:25.033 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:51:25.033 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:51:25.068 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:51:25.068 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:51:25.102 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:51:25.102 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:51:25.107 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:25.132 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:51:25.133 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:51:25.135 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:25.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:25.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:25.168 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:51:25.168 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:51:25.179 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 6, 7, 1] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:51:25.179 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:51:25.180 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-13T14:51:25.180 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:51:25.201 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:51:25.202 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:51:25.230 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:51:25.230 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:51:25.237 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:25.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:25.266 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:25.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:25.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:25 smithi184 ceph-mon[92527]: pgmap v1744: 119 pgs: 2 active+remapped+backfilling, 1 unknown, 12 peering, 104 active+clean; 6.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 11 KiB/s rd, 25 KiB/s wr, 1.08k op/s; 13391/340599 objects misplaced (3.932%); 6.9 MiB/s, 1 keys/s, 113 objects/s recovering 2024-09-13T14:51:25.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:25 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/630472981' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:25.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:25 smithi151 ceph-mon[104450]: pgmap v1744: 119 pgs: 2 active+remapped+backfilling, 1 unknown, 12 peering, 104 active+clean; 6.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 11 KiB/s rd, 25 KiB/s wr, 1.08k op/s; 13391/340599 objects misplaced (3.932%); 6.9 MiB/s, 1 keys/s, 113 objects/s recovering 2024-09-13T14:51:25.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:25 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/630472981' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:25.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:25 smithi151 ceph-mon[108966]: pgmap v1744: 119 pgs: 2 active+remapped+backfilling, 1 unknown, 12 peering, 104 active+clean; 6.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 11 KiB/s rd, 25 KiB/s wr, 1.08k op/s; 13391/340599 objects misplaced (3.932%); 6.9 MiB/s, 1 keys/s, 113 objects/s recovering 2024-09-13T14:51:25.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:25 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/630472981' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:25.367 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:25.416 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:25.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:25.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:25.518 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:25.565 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:25.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:25.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:25.667 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:25.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:25.699 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:25.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:25.800 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:25.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:25.839 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:25.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:25.940 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:25.979 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:25.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:25.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:26.048 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:51:26.048 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [ 2024-09-13T14:51:26.048 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.048 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35940.0:140054 8.2 e1179/1178)", 2024-09-13T14:51:26.048 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:51:26.045293+0000", 2024-09-13T14:51:26.048 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.000119972, 2024-09-13T14:51:26.048 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000142998, 2024-09-13T14:51:26.049 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:51:26.049 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "queued for pg", 2024-09-13T14:51:26.049 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:51:26.049 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.049 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:51:26.049 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:51:26.045293+0000", 2024-09-13T14:51:26.049 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:26.049 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:26.049 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.049 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:51:26.049 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:51:26.045293+0000", 2024-09-13T14:51:26.050 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:26.050 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:26.050 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.050 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:51:26.050 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:51:26.045293+0000", 2024-09-13T14:51:26.050 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:51:26.050 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:26.050 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.050 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:51:26.050 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:51:26.045300+0000", 2024-09-13T14:51:26.050 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.9190000000000002e-06 2024-09-13T14:51:26.051 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:26.051 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.051 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:51:26.051 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:51:26.045314+0000", 2024-09-13T14:51:26.051 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4715999999999999e-05 2024-09-13T14:51:26.051 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:26.051 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.051 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:51:26.051 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:51:26.045321+0000", 2024-09-13T14:51:26.051 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3289999999999999e-06 2024-09-13T14:51:26.051 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:26.052 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:51:26.052 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:26.052 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:26.052 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.052 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35940.0:140043 8.2 e1179/1178 8:4a05c41a:::benchmark_data_smithi151_229097_object15175:head v 1179'8768, mlcod=1179'8766)", 2024-09-13T14:51:26.052 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:51:26.034984+0000", 2024-09-13T14:51:26.052 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.010429058, 2024-09-13T14:51:26.052 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.010521601, 2024-09-13T14:51:26.052 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:51:26.052 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:51:26.052 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:51:26.053 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.053 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:51:26.053 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:51:26.034984+0000", 2024-09-13T14:51:26.053 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:26.053 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:26.053 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.053 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:51:26.053 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:51:26.034984+0000", 2024-09-13T14:51:26.053 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:26.053 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:26.054 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.054 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:51:26.054 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:51:26.034983+0000", 2024-09-13T14:51:26.054 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:51:26.054 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:26.054 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.054 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:51:26.054 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:51:26.034992+0000", 2024-09-13T14:51:26.054 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.354e-06 2024-09-13T14:51:26.055 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:26.055 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.055 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:51:26.055 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:51:26.035008+0000", 2024-09-13T14:51:26.055 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6592e-05 2024-09-13T14:51:26.055 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:26.055 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.055 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:51:26.055 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:51:26.035015+0000", 2024-09-13T14:51:26.055 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.6159999999999998e-06 2024-09-13T14:51:26.056 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:26.056 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.056 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:51:26.056 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:51:26.035154+0000", 2024-09-13T14:51:26.056 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013877799999999999 2024-09-13T14:51:26.056 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:26.056 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.056 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:51:26.056 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:51:26.035211+0000", 2024-09-13T14:51:26.056 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6901000000000003e-05 2024-09-13T14:51:26.056 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:26.056 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:51:26.057 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:26.057 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:26.057 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.057 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35940.0:140047 8.2 e1179/1178 8:48032713:::benchmark_data_smithi151_229097_object15179:head v 1179'8769, mlcod=1179'8766)", 2024-09-13T14:51:26.057 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:51:26.037093+0000", 2024-09-13T14:51:26.057 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.0083201100000000004, 2024-09-13T14:51:26.057 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0084672979999999998, 2024-09-13T14:51:26.057 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:51:26.057 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:51:26.057 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:51:26.057 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.058 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:51:26.058 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:51:26.037093+0000", 2024-09-13T14:51:26.058 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:26.058 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:26.058 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.058 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:51:26.058 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:51:26.037093+0000", 2024-09-13T14:51:26.058 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:26.058 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:26.058 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.059 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:51:26.059 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:51:26.037093+0000", 2024-09-13T14:51:26.059 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:51:26.059 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:26.059 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.059 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:51:26.059 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:51:26.037099+0000", 2024-09-13T14:51:26.059 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.4350000000000002e-06 2024-09-13T14:51:26.059 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:26.059 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.059 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:51:26.059 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:51:26.037112+0000", 2024-09-13T14:51:26.060 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3178e-05 2024-09-13T14:51:26.060 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:26.060 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.060 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:51:26.060 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:51:26.037117+0000", 2024-09-13T14:51:26.060 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.2639999999999999e-06 2024-09-13T14:51:26.060 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:26.060 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.060 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:51:26.060 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:51:26.042497+0000", 2024-09-13T14:51:26.061 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0053793720000000003 2024-09-13T14:51:26.061 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:26.061 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.061 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:51:26.061 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:51:26.042529+0000", 2024-09-13T14:51:26.061 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.2119000000000002e-05 2024-09-13T14:51:26.061 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:26.061 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:51:26.061 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:26.062 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:26.062 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.062 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35940.0:140048 8.2 e1179/1178)", 2024-09-13T14:51:26.062 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:51:26.042198+0000", 2024-09-13T14:51:26.062 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.0032150310000000001, 2024-09-13T14:51:26.062 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.003416947, 2024-09-13T14:51:26.062 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:51:26.062 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "queued for pg", 2024-09-13T14:51:26.062 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:51:26.062 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.062 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:51:26.062 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:51:26.042198+0000", 2024-09-13T14:51:26.063 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:26.063 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:26.063 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.063 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:51:26.063 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:51:26.042198+0000", 2024-09-13T14:51:26.063 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:26.064 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:26.064 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.064 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:51:26.064 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:51:26.042198+0000", 2024-09-13T14:51:26.064 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:51:26.064 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:26.064 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.064 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:51:26.065 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:51:26.042204+0000", 2024-09-13T14:51:26.065 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.8700000000000003e-06 2024-09-13T14:51:26.065 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:26.065 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.065 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:51:26.065 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:51:26.042216+0000", 2024-09-13T14:51:26.065 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1613e-05 2024-09-13T14:51:26.065 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:26.065 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:26.066 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:51:26.066 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:51:26.042221+0000", 2024-09-13T14:51:26.066 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3770000000000004e-06 2024-09-13T14:51:26.066 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:26.066 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:51:26.066 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:26.066 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:26.066 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-13T14:51:26.066 INFO:teuthology.orchestra.run.smithi184.stdout: "num_ops": 4 2024-09-13T14:51:26.066 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:51:26.082 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:26.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:26.119 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:26.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:26.220 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:26.257 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:26.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:26.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:26.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:26 smithi184 ceph-mon[92527]: 1.5 scrub starts 2024-09-13T14:51:26.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:26 smithi151 ceph-mon[104450]: 1.5 scrub starts 2024-09-13T14:51:26.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:26 smithi151 ceph-mon[108966]: 1.5 scrub starts 2024-09-13T14:51:26.359 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:26.402 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:26.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:26.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:26.504 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:26.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:26.541 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:26.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:26.642 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:26.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:26.676 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:26.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:26.777 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:26.814 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:26.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:26.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:26.829 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T14:51:26.916 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:26.949 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:26.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:26.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:27.051 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:27.095 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:27.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:27.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:27.197 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:27.219 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:51:27.220 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:27.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:27.275 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:27.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:27.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:27 smithi151 ceph-mon[104450]: pgmap v1745: 119 pgs: 5 active+remapped+backfilling, 114 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 10 KiB/s rd, 23 KiB/s wr, 803 op/s; 33038/335589 objects misplaced (9.845%); 17 MiB/s, 1 keys/s, 274 objects/s recovering 2024-09-13T14:51:27.345 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:27 smithi151 ceph-mon[104450]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-09-13T14:51:27.345 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:27 smithi151 ceph-mon[108966]: pgmap v1745: 119 pgs: 5 active+remapped+backfilling, 114 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 10 KiB/s rd, 23 KiB/s wr, 803 op/s; 33038/335589 objects misplaced (9.845%); 17 MiB/s, 1 keys/s, 274 objects/s recovering 2024-09-13T14:51:27.345 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:27 smithi151 ceph-mon[108966]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-09-13T14:51:27.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:27 smithi184 ceph-mon[92527]: pgmap v1745: 119 pgs: 5 active+remapped+backfilling, 114 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 10 KiB/s rd, 23 KiB/s wr, 803 op/s; 33038/335589 objects misplaced (9.845%); 17 MiB/s, 1 keys/s, 274 objects/s recovering 2024-09-13T14:51:27.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:27 smithi184 ceph-mon[92527]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-09-13T14:51:27.376 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:27.412 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:27.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:27.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:27.515 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:27.560 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:27.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:27.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:27.662 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:27.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:27.696 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:27.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:27.797 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:27.831 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:27.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:27.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:27.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:27.970 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:27.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:27.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:28.072 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:28.103 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:28.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:28.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:28.205 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:28.223 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:51:28.239 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:28.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:28.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:28.341 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:28.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:28.408 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:28.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:28.509 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:28.540 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:28.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:28.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:28.642 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:28.683 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:28.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:28.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:28.785 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:28.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:28.819 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:28.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:28.921 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:28.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:28.960 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:28.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:29.062 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:29.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:29.095 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:29.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:29.117 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-13T14:51:29.196 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:29.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:29.241 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:29.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:29.248 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:29 smithi184 ceph-mon[92527]: pgmap v1746: 119 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfilling, 113 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 8.5 KiB/s rd, 18 KiB/s wr, 656 op/s; 33038/335589 objects misplaced (9.845%); 14 MiB/s, 1 keys/s, 224 objects/s recovering 2024-09-13T14:51:29.249 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:29 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1677225526' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:51:29.249 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:29 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:51:29.342 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:29.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:29 smithi151 ceph-mon[104450]: pgmap v1746: 119 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfilling, 113 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 8.5 KiB/s rd, 18 KiB/s wr, 656 op/s; 33038/335589 objects misplaced (9.845%); 14 MiB/s, 1 keys/s, 224 objects/s recovering 2024-09-13T14:51:29.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:29 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1677225526' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:51:29.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:29 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:51:29.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:29 smithi151 ceph-mon[108966]: pgmap v1746: 119 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfilling, 113 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 8.5 KiB/s rd, 18 KiB/s wr, 656 op/s; 33038/335589 objects misplaced (9.845%); 14 MiB/s, 1 keys/s, 224 objects/s recovering 2024-09-13T14:51:29.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:29 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1677225526' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:51:29.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:29 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:51:29.381 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:29.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:29.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:29.483 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:29.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:29.575 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:29.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:29.575 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:51:29.575 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [], 2024-09-13T14:51:29.575 INFO:teuthology.orchestra.run.smithi184.stdout: "complaint_time": 30, 2024-09-13T14:51:29.576 INFO:teuthology.orchestra.run.smithi184.stdout: "num_blocked_ops": 0 2024-09-13T14:51:29.576 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:51:29.675 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:29.719 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:29.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:29.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:29.821 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:29.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:29.858 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:29.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:29.959 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:29.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:29.998 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:29.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:30.100 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:30.139 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:30.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:30.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:30.194 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:30 smithi184 ceph-mon[92527]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:30.195 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:30 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:51:30.195 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:30 smithi184 ceph-mon[92527]: osdmap e1180: 8 total, 8 up, 7 in 2024-09-13T14:51:30.235 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T14:51:30.241 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:30.280 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:30.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:30.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:30.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:30 smithi151 ceph-mon[104450]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:30.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:30 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:51:30.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:30 smithi151 ceph-mon[104450]: osdmap e1180: 8 total, 8 up, 7 in 2024-09-13T14:51:30.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:30 smithi151 ceph-mon[108966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:30.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:30 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:51:30.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:30 smithi151 ceph-mon[108966]: osdmap e1180: 8 total, 8 up, 7 in 2024-09-13T14:51:30.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:30.405 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:30.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:30.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:30.506 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:30.549 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:30.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:30.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:30.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:51:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:51:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:51:30.653 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:30.680 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:51:30.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:30.692 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:30.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:30.793 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:30.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:30.815 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:30.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:30.916 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:30.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:30.955 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:30.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:31.056 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:31.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:31.098 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:31.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:31.199 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:31.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:31.242 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:31.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:31.343 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:31.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:31 smithi184 ceph-mon[92527]: pgmap v1748: 119 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfilling, 113 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 7.8 KiB/s rd, 17 KiB/s wr, 719 op/s; 30651/327198 objects misplaced (9.368%); 26 MiB/s, 1 keys/s, 417 objects/s recovering 2024-09-13T14:51:31.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:31.383 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:31.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:31.484 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:31.516 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:31.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:31.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:31.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:31 smithi151 ceph-mon[104450]: pgmap v1748: 119 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfilling, 113 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 7.8 KiB/s rd, 17 KiB/s wr, 719 op/s; 30651/327198 objects misplaced (9.368%); 26 MiB/s, 1 keys/s, 417 objects/s recovering 2024-09-13T14:51:31.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:31 smithi151 ceph-mon[108966]: pgmap v1748: 119 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfilling, 113 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 7.8 KiB/s rd, 17 KiB/s wr, 719 op/s; 30651/327198 objects misplaced (9.368%); 26 MiB/s, 1 keys/s, 417 objects/s recovering 2024-09-13T14:51:31.617 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:31.653 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:31.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:31.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:31.755 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:31.778 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:31.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:31.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:31.880 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:31.916 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:31.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:31.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:32.018 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:32.061 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:32.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:32.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:32.163 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:32.200 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:32.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:32.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:32.222 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:32 smithi184 ceph-mon[92527]: 1.5 scrub starts 2024-09-13T14:51:32.222 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:32 smithi184 ceph-mon[92527]: 1.5 scrub ok 2024-09-13T14:51:32.222 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:32.222 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:51:32.302 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:32.339 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:32.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:32.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:32.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:32.479 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:32.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:32.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:32.581 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:32.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:32 smithi151 ceph-mon[104450]: 1.5 scrub starts 2024-09-13T14:51:32.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:32 smithi151 ceph-mon[104450]: 1.5 scrub ok 2024-09-13T14:51:32.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:32.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:51:32.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:32 smithi151 ceph-mon[108966]: 1.5 scrub starts 2024-09-13T14:51:32.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:32 smithi151 ceph-mon[108966]: 1.5 scrub ok 2024-09-13T14:51:32.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:32.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:51:32.619 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:32.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:32.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:32.721 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:32.755 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:32.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:32.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:32.857 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:32.888 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:32.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:32.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:32.990 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:33.025 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:33.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:33.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:33.127 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:33.149 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:33.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:33.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:33.251 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:33.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:33.282 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:33.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:33.375 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:33 smithi151 ceph-mon[104450]: pgmap v1749: 119 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfilling, 113 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 6.8 KiB/s rd, 15 KiB/s wr, 827 op/s; 28466/321141 objects misplaced (8.864%); 31 MiB/s, 0 keys/s, 500 objects/s recovering 2024-09-13T14:51:33.376 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:33 smithi151 ceph-mon[108966]: pgmap v1749: 119 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfilling, 113 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 6.8 KiB/s rd, 15 KiB/s wr, 827 op/s; 28466/321141 objects misplaced (8.864%); 31 MiB/s, 0 keys/s, 500 objects/s recovering 2024-09-13T14:51:33.383 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:33.414 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:33.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:33.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:33.488 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:33 smithi184 ceph-mon[92527]: pgmap v1749: 119 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfilling, 113 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 6.8 KiB/s rd, 15 KiB/s wr, 827 op/s; 28466/321141 objects misplaced (8.864%); 31 MiB/s, 0 keys/s, 500 objects/s recovering 2024-09-13T14:51:33.516 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:33.548 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:33.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:33.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:33.650 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:33.688 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:33.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:33.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:33.789 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:33.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:33.827 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:33.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:33.928 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:33.959 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:33.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:33.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:34.061 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:34.092 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:34.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:34.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:34.194 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:34.238 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:34.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:34.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:34.340 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:34.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:34.380 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:34.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:34.481 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:34.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:34.515 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:34.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:34.616 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:34.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:34.648 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:34.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:34.749 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:34.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:34.778 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:34.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:34.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:51:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:51:34] "GET /metrics HTTP/1.1" 200 36696 "" "Prometheus/2.51.0" 2024-09-13T14:51:34.878 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:34.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:34.901 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:34.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:35.002 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:35.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:35.041 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:35.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:35.143 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:35.184 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:35.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:35.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:35.206 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:51:35.206 INFO:teuthology.orchestra.run.smithi184.stdout: "size": 20, 2024-09-13T14:51:35.207 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 600, 2024-09-13T14:51:35.207 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [ 2024-09-13T14:51:35.207 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.207 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:61889 7.1 7:8d476111:::benchmark_data_smithi151_190906_object61888:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1142)", 2024-09-13T14:51:35.207 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:44:33.520234+0000", 2024-09-13T14:51:35.207 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 420.96000787499997, 2024-09-13T14:51:35.207 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.245871268, 2024-09-13T14:51:35.207 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:51:35.207 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:35.207 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:51:35.207 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:51:35.207 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:51:35.208 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 61889 2024-09-13T14:51:35.208 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.208 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:51:35.208 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.208 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:51:35.208 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.520234+0000", 2024-09-13T14:51:35.208 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.208 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.208 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.208 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:51:35.208 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.520234+0000", 2024-09-13T14:51:35.208 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.208 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.209 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.209 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:51:35.209 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.520232+0000", 2024-09-13T14:51:35.209 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:51:35.209 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.209 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.209 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:51:35.209 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.520280+0000", 2024-09-13T14:51:35.209 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7960000000000002e-05 2024-09-13T14:51:35.209 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.209 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.209 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:51:35.210 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.520294+0000", 2024-09-13T14:51:35.210 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.348e-05 2024-09-13T14:51:35.210 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.210 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.210 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:51:35.210 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.520300+0000", 2024-09-13T14:51:35.210 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.2469999999999997e-06 2024-09-13T14:51:35.210 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.210 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.210 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:51:35.210 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.522368+0000", 2024-09-13T14:51:35.210 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002067752 2024-09-13T14:51:35.210 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.211 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.211 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:51:35.211 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.522518+0000", 2024-09-13T14:51:35.211 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000150655 2024-09-13T14:51:35.211 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.211 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.211 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:51:35.211 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.522682+0000", 2024-09-13T14:51:35.211 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000163714 2024-09-13T14:51:35.211 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.211 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.211 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:51:35.211 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.524824+0000", 2024-09-13T14:51:35.211 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0021421579999999999 2024-09-13T14:51:35.212 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.212 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.212 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.212 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.524916+0000", 2024-09-13T14:51:35.212 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.1238999999999998e-05 2024-09-13T14:51:35.212 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.212 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.212 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.212 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.766002+0000", 2024-09-13T14:51:35.212 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.24108671500000001 2024-09-13T14:51:35.212 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.212 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.212 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:51:35.213 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.766062+0000", 2024-09-13T14:51:35.213 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.9546000000000002e-05 2024-09-13T14:51:35.213 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.213 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.213 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:51:35.213 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.766105+0000", 2024-09-13T14:51:35.213 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3389000000000001e-05 2024-09-13T14:51:35.213 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.213 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:51:35.213 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.213 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.213 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.213 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:63289 7.1 7:8d93c16e:::benchmark_data_smithi151_190906_object63288:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1142)", 2024-09-13T14:51:35.213 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:44:34.960269+0000", 2024-09-13T14:51:35.214 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 419.51997264300002, 2024-09-13T14:51:35.214 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.137552386, 2024-09-13T14:51:35.214 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:51:35.214 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:35.214 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:51:35.214 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:51:35.214 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:51:35.214 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 63289 2024-09-13T14:51:35.214 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.214 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:51:35.214 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.214 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:51:35.214 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.960269+0000", 2024-09-13T14:51:35.214 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.215 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.215 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.215 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:51:35.215 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.960269+0000", 2024-09-13T14:51:35.215 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.215 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.215 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.215 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:51:35.215 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.960268+0000", 2024-09-13T14:51:35.215 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:51:35.215 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.215 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.215 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:51:35.215 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.961147+0000", 2024-09-13T14:51:35.216 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00087899099999999997 2024-09-13T14:51:35.216 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.216 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.216 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:51:35.216 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.961161+0000", 2024-09-13T14:51:35.216 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4022000000000001e-05 2024-09-13T14:51:35.216 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.216 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.216 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:51:35.216 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.961166+0000", 2024-09-13T14:51:35.216 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7100000000000004e-06 2024-09-13T14:51:35.216 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.216 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.217 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:51:35.217 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.961208+0000", 2024-09-13T14:51:35.217 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1582000000000003e-05 2024-09-13T14:51:35.217 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.217 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.217 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:51:35.217 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.961319+0000", 2024-09-13T14:51:35.217 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00011085499999999999 2024-09-13T14:51:35.217 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.217 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.217 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:51:35.217 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.961448+0000", 2024-09-13T14:51:35.217 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012894 2024-09-13T14:51:35.218 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.218 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.218 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.218 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.964844+0000", 2024-09-13T14:51:35.218 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0033967749999999999 2024-09-13T14:51:35.218 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.218 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.218 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:51:35.218 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.964850+0000", 2024-09-13T14:51:35.218 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8409999999999996e-06 2024-09-13T14:51:35.218 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.218 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.218 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.219 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:35.097681+0000", 2024-09-13T14:51:35.219 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.13283061900000001 2024-09-13T14:51:35.219 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.219 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.219 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:51:35.219 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:35.097778+0000", 2024-09-13T14:51:35.219 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.7297e-05 2024-09-13T14:51:35.219 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.219 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.219 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:51:35.219 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:35.097822+0000", 2024-09-13T14:51:35.219 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3387000000000003e-05 2024-09-13T14:51:35.219 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.220 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:51:35.220 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.220 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.220 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.220 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:65056 7.1 7:84e14700:::benchmark_data_smithi151_190906_object65055:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1142)", 2024-09-13T14:51:35.220 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:44:36.355488+0000", 2024-09-13T14:51:35.220 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 418.124754, 2024-09-13T14:51:35.220 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.15462773699999999, 2024-09-13T14:51:35.220 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:51:35.220 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:35.220 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:51:35.220 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:51:35.220 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:51:35.221 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 65056 2024-09-13T14:51:35.221 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.221 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:51:35.221 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.221 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:51:35.221 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:36.355488+0000", 2024-09-13T14:51:35.221 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.221 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.221 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.221 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:51:35.221 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:36.355488+0000", 2024-09-13T14:51:35.221 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.221 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.221 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.222 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:51:35.222 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:36.355486+0000", 2024-09-13T14:51:35.222 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:51:35.222 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.222 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.222 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:51:35.222 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:36.355520+0000", 2024-09-13T14:51:35.222 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3422e-05 2024-09-13T14:51:35.222 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.222 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.222 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:51:35.222 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:36.355536+0000", 2024-09-13T14:51:35.222 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6087e-05 2024-09-13T14:51:35.222 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.223 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.223 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:51:35.223 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:36.355542+0000", 2024-09-13T14:51:35.223 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1580000000000002e-06 2024-09-13T14:51:35.223 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.223 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.223 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:51:35.223 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:36.355602+0000", 2024-09-13T14:51:35.223 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.9670000000000003e-05 2024-09-13T14:51:35.223 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.223 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.223 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:51:35.223 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:36.355765+0000", 2024-09-13T14:51:35.223 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00016313499999999999 2024-09-13T14:51:35.224 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.224 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.224 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:51:35.224 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:36.355928+0000", 2024-09-13T14:51:35.224 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000163649 2024-09-13T14:51:35.224 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.224 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.224 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:51:35.224 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:36.358301+0000", 2024-09-13T14:51:35.224 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0023727599999999998 2024-09-13T14:51:35.224 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.224 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.224 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.224 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:36.359129+0000", 2024-09-13T14:51:35.225 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00082790499999999998 2024-09-13T14:51:35.225 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.225 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.225 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.225 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:36.510021+0000", 2024-09-13T14:51:35.225 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.15089190299999999 2024-09-13T14:51:35.225 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.225 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.225 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:51:35.225 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:36.510075+0000", 2024-09-13T14:51:35.225 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4274999999999998e-05 2024-09-13T14:51:35.225 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.225 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.225 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:51:35.226 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:36.510116+0000", 2024-09-13T14:51:35.226 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0367000000000002e-05 2024-09-13T14:51:35.226 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.226 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:51:35.226 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.226 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.226 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.226 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:97317 7.1 7:83be73bf:::benchmark_data_smithi151_190906_object97316:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1146)", 2024-09-13T14:51:35.226 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:45:03.741860+0000", 2024-09-13T14:51:35.226 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 390.73838201500001, 2024-09-13T14:51:35.226 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.15392760599999999, 2024-09-13T14:51:35.226 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:51:35.226 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:35.226 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:51:35.227 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:51:35.227 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:51:35.227 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 97317 2024-09-13T14:51:35.227 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.227 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:51:35.227 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.227 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:51:35.227 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.741860+0000", 2024-09-13T14:51:35.227 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.227 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.227 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.227 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:51:35.227 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.741860+0000", 2024-09-13T14:51:35.228 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.228 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.228 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.228 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:51:35.228 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.741859+0000", 2024-09-13T14:51:35.228 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:51:35.228 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.228 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.228 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:51:35.228 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.741901+0000", 2024-09-13T14:51:35.228 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1860000000000002e-05 2024-09-13T14:51:35.228 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.228 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.229 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:51:35.229 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.741911+0000", 2024-09-13T14:51:35.229 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.9469999999999998e-06 2024-09-13T14:51:35.229 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.229 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.229 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:51:35.229 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.741916+0000", 2024-09-13T14:51:35.229 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8119999999999996e-06 2024-09-13T14:51:35.229 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.229 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.229 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:51:35.229 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.754055+0000", 2024-09-13T14:51:35.229 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.012138062999999999 2024-09-13T14:51:35.229 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.230 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.230 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:51:35.230 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.754200+0000", 2024-09-13T14:51:35.230 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014585100000000001 2024-09-13T14:51:35.230 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.230 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.230 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:51:35.230 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.754339+0000", 2024-09-13T14:51:35.230 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000138646 2024-09-13T14:51:35.230 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.230 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.230 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:51:35.230 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.759977+0000", 2024-09-13T14:51:35.230 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0056384099999999999 2024-09-13T14:51:35.231 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.231 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.231 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.231 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.760380+0000", 2024-09-13T14:51:35.231 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00040259999999999997 2024-09-13T14:51:35.231 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.231 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.231 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.231 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.895650+0000", 2024-09-13T14:51:35.231 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.13526993100000001 2024-09-13T14:51:35.231 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.231 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.231 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:51:35.231 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.895747+0000", 2024-09-13T14:51:35.232 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.7399999999999996e-05 2024-09-13T14:51:35.232 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.232 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.232 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:51:35.232 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.895787+0000", 2024-09-13T14:51:35.232 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0070000000000001e-05 2024-09-13T14:51:35.232 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.232 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:51:35.232 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.232 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.232 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.232 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:97328 7.1 7:86c42084:::benchmark_data_smithi151_190906_object97327:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1146)", 2024-09-13T14:51:35.232 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:45:03.755640+0000", 2024-09-13T14:51:35.232 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 390.724601398, 2024-09-13T14:51:35.233 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.15131093500000001, 2024-09-13T14:51:35.233 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:51:35.233 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:35.233 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:51:35.233 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:51:35.233 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:51:35.233 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 97328 2024-09-13T14:51:35.233 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.233 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:51:35.233 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.233 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:51:35.233 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.755640+0000", 2024-09-13T14:51:35.233 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.233 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.234 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.234 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:51:35.234 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.755640+0000", 2024-09-13T14:51:35.234 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.234 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.234 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.234 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:51:35.234 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.755638+0000", 2024-09-13T14:51:35.234 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:51:35.234 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.234 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.234 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:51:35.234 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.755672+0000", 2024-09-13T14:51:35.234 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3645000000000001e-05 2024-09-13T14:51:35.235 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.235 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.235 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:51:35.235 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.755686+0000", 2024-09-13T14:51:35.235 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3720000000000001e-05 2024-09-13T14:51:35.235 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.235 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.235 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:51:35.235 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.755692+0000", 2024-09-13T14:51:35.235 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3759999999999996e-06 2024-09-13T14:51:35.235 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.235 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.235 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:51:35.236 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.756858+0000", 2024-09-13T14:51:35.236 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0011653189999999999 2024-09-13T14:51:35.236 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.236 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.236 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:51:35.236 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.757000+0000", 2024-09-13T14:51:35.236 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000142707 2024-09-13T14:51:35.236 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.236 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.236 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:51:35.236 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.757157+0000", 2024-09-13T14:51:35.236 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00015677900000000001 2024-09-13T14:51:35.236 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.236 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.237 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:51:35.237 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.759982+0000", 2024-09-13T14:51:35.237 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002824713 2024-09-13T14:51:35.237 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.237 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.237 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.237 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.760498+0000", 2024-09-13T14:51:35.237 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00051625900000000003 2024-09-13T14:51:35.237 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.237 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.237 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.237 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.906811+0000", 2024-09-13T14:51:35.237 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.146312677 2024-09-13T14:51:35.237 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.238 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.238 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:51:35.238 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.906907+0000", 2024-09-13T14:51:35.238 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.6183999999999999e-05 2024-09-13T14:51:35.238 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.238 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.238 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:51:35.238 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.906951+0000", 2024-09-13T14:51:35.238 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4499000000000002e-05 2024-09-13T14:51:35.238 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.238 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:51:35.238 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.238 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.238 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.238 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:141042 7.1 7:830a39ed:::benchmark_data_smithi151_190906_object20581:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1155)", 2024-09-13T14:51:35.239 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:46:05.945239+0000", 2024-09-13T14:51:35.239 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 328.53500301899999, 2024-09-13T14:51:35.239 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.149968298, 2024-09-13T14:51:35.239 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:51:35.239 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:35.239 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:51:35.239 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:51:35.239 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:51:35.239 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 141042 2024-09-13T14:51:35.239 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.239 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:51:35.239 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.239 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:51:35.239 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:05.945239+0000", 2024-09-13T14:51:35.240 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.240 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.240 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.240 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:51:35.240 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:05.945239+0000", 2024-09-13T14:51:35.240 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.240 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.240 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.240 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:51:35.240 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:05.945237+0000", 2024-09-13T14:51:35.240 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:51:35.240 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.240 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.240 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:51:35.241 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:05.945244+0000", 2024-09-13T14:51:35.241 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.8940000000000004e-06 2024-09-13T14:51:35.241 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.241 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.241 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:51:35.241 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:05.945256+0000", 2024-09-13T14:51:35.241 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1396000000000001e-05 2024-09-13T14:51:35.241 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.241 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.241 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:51:35.241 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:05.945262+0000", 2024-09-13T14:51:35.241 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1720000000000004e-06 2024-09-13T14:51:35.241 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.241 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.242 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:51:35.242 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:05.945302+0000", 2024-09-13T14:51:35.242 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0321999999999998e-05 2024-09-13T14:51:35.242 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.242 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.242 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:51:35.242 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:05.945399+0000", 2024-09-13T14:51:35.242 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.7164999999999996e-05 2024-09-13T14:51:35.242 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.242 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.242 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:51:35.242 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:05.945500+0000", 2024-09-13T14:51:35.242 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000100665 2024-09-13T14:51:35.242 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.243 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.243 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:51:35.243 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:05.948209+0000", 2024-09-13T14:51:35.243 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0027086990000000002 2024-09-13T14:51:35.243 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.243 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.243 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.243 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:05.948336+0000", 2024-09-13T14:51:35.243 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000127646 2024-09-13T14:51:35.243 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.243 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.243 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.243 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:05.949320+0000", 2024-09-13T14:51:35.243 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00098333699999999993 2024-09-13T14:51:35.243 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.244 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.244 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.244 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:06.095030+0000", 2024-09-13T14:51:35.244 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.14570997099999999 2024-09-13T14:51:35.244 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.244 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.244 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:51:35.244 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:06.095085+0000", 2024-09-13T14:51:35.244 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5194e-05 2024-09-13T14:51:35.244 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.244 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.244 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:51:35.244 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:06.095207+0000", 2024-09-13T14:51:35.244 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000122366 2024-09-13T14:51:35.245 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.245 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:51:35.245 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.245 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.245 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.245 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:195549 7.1 7:828894fe:::benchmark_data_smithi151_190906_object75088:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1160)", 2024-09-13T14:51:35.245 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:47:34.962584+0000", 2024-09-13T14:51:35.245 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 239.517657504, 2024-09-13T14:51:35.245 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.17580900499999999, 2024-09-13T14:51:35.245 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:51:35.245 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:35.245 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:51:35.245 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:51:35.245 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:51:35.246 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 195549 2024-09-13T14:51:35.246 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.246 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:51:35.246 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.246 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:51:35.246 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.962584+0000", 2024-09-13T14:51:35.246 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.246 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.246 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.246 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:51:35.246 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.962584+0000", 2024-09-13T14:51:35.246 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.246 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.246 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.246 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:51:35.247 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.962583+0000", 2024-09-13T14:51:35.247 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:51:35.247 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.247 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.247 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:51:35.247 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.962591+0000", 2024-09-13T14:51:35.247 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0269999999999995e-06 2024-09-13T14:51:35.247 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.247 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.247 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:51:35.247 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.962609+0000", 2024-09-13T14:51:35.247 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.8193999999999999e-05 2024-09-13T14:51:35.247 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.247 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.248 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:51:35.248 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.962617+0000", 2024-09-13T14:51:35.248 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.6190000000000002e-06 2024-09-13T14:51:35.248 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.248 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.248 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:51:35.248 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.962670+0000", 2024-09-13T14:51:35.248 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3047000000000002e-05 2024-09-13T14:51:35.248 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.248 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.248 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:51:35.248 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.962815+0000", 2024-09-13T14:51:35.248 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014494900000000001 2024-09-13T14:51:35.249 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.249 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.249 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:51:35.249 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.962973+0000", 2024-09-13T14:51:35.249 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000158252 2024-09-13T14:51:35.249 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.249 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.249 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.249 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.966688+0000", 2024-09-13T14:51:35.249 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0037152550000000002 2024-09-13T14:51:35.249 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.249 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.249 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:51:35.249 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.968210+0000", 2024-09-13T14:51:35.250 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001522277 2024-09-13T14:51:35.250 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.250 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.250 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.250 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.969011+0000", 2024-09-13T14:51:35.250 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00080025400000000003 2024-09-13T14:51:35.250 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.250 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.250 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.250 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:35.138207+0000", 2024-09-13T14:51:35.250 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.169196717 2024-09-13T14:51:35.251 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.251 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.251 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:51:35.251 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:35.138265+0000", 2024-09-13T14:51:35.251 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7268000000000001e-05 2024-09-13T14:51:35.251 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.251 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.251 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:51:35.251 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:35.138393+0000", 2024-09-13T14:51:35.251 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012880500000000001 2024-09-13T14:51:35.251 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.251 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:51:35.251 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.251 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.252 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.252 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:195563 7.1 7:8ea4174d:::benchmark_data_smithi151_190906_object75102:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1160)", 2024-09-13T14:51:35.252 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:47:34.981199+0000", 2024-09-13T14:51:35.252 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 239.499042968, 2024-09-13T14:51:35.252 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.161925608, 2024-09-13T14:51:35.252 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:51:35.252 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:35.252 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:51:35.252 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:51:35.252 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:51:35.252 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 195563 2024-09-13T14:51:35.252 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.252 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:51:35.252 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.253 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:51:35.253 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.981199+0000", 2024-09-13T14:51:35.253 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.253 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.253 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.253 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:51:35.253 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.981199+0000", 2024-09-13T14:51:35.253 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.253 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.253 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.253 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:51:35.253 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.981197+0000", 2024-09-13T14:51:35.253 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:51:35.253 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.254 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.254 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:51:35.254 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.981204+0000", 2024-09-13T14:51:35.254 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.5050000000000001e-06 2024-09-13T14:51:35.254 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.254 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.254 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:51:35.254 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.981218+0000", 2024-09-13T14:51:35.254 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4e-05 2024-09-13T14:51:35.254 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.254 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.254 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:51:35.254 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.981224+0000", 2024-09-13T14:51:35.254 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.964e-06 2024-09-13T14:51:35.255 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.255 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.255 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:51:35.255 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.981290+0000", 2024-09-13T14:51:35.255 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.5370999999999995e-05 2024-09-13T14:51:35.255 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.255 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.255 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:51:35.255 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.981453+0000", 2024-09-13T14:51:35.255 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00016340899999999999 2024-09-13T14:51:35.255 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.255 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.255 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:51:35.255 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.981553+0000", 2024-09-13T14:51:35.256 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.9429e-05 2024-09-13T14:51:35.256 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.256 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.256 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:51:35.256 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.984591+0000", 2024-09-13T14:51:35.256 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00303857 2024-09-13T14:51:35.256 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.256 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.256 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.256 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.985474+0000", 2024-09-13T14:51:35.256 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00088276399999999997 2024-09-13T14:51:35.256 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.256 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.256 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.257 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.986705+0000", 2024-09-13T14:51:35.257 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001230734 2024-09-13T14:51:35.257 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.257 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.257 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.257 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:35.142938+0000", 2024-09-13T14:51:35.257 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.15623298099999999 2024-09-13T14:51:35.257 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.257 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.257 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:51:35.257 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:35.142994+0000", 2024-09-13T14:51:35.257 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6759000000000003e-05 2024-09-13T14:51:35.257 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.257 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.257 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:51:35.258 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:35.143124+0000", 2024-09-13T14:51:35.258 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000130025 2024-09-13T14:51:35.258 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.258 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:51:35.258 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.258 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.258 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.258 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:196368 7.1 7:8b4e4f2b:::benchmark_data_smithi151_190906_object75907:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1160)", 2024-09-13T14:51:35.258 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:47:36.664845+0000", 2024-09-13T14:51:35.258 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 237.81539691899999, 2024-09-13T14:51:35.258 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.14392316499999999, 2024-09-13T14:51:35.258 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:51:35.258 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:35.258 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:51:35.259 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:51:35.259 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:51:35.259 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 196368 2024-09-13T14:51:35.259 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.259 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:51:35.259 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.259 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:51:35.259 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:36.664845+0000", 2024-09-13T14:51:35.259 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.259 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.259 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.259 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:51:35.259 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:36.664845+0000", 2024-09-13T14:51:35.259 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.260 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.260 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.260 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:51:35.260 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:36.664843+0000", 2024-09-13T14:51:35.260 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:51:35.260 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.260 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.260 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:51:35.260 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:36.664852+0000", 2024-09-13T14:51:35.260 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.4200000000000007e-06 2024-09-13T14:51:35.260 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.260 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.260 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:51:35.260 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:36.664867+0000", 2024-09-13T14:51:35.261 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4916000000000001e-05 2024-09-13T14:51:35.261 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.261 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.261 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:51:35.261 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:36.664885+0000", 2024-09-13T14:51:35.261 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.8802999999999999e-05 2024-09-13T14:51:35.261 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.261 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.261 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:51:35.261 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:36.664930+0000", 2024-09-13T14:51:35.261 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4440999999999997e-05 2024-09-13T14:51:35.261 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.261 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.261 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:51:35.262 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:36.665054+0000", 2024-09-13T14:51:35.262 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012421700000000001 2024-09-13T14:51:35.262 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.262 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.262 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:51:35.262 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:36.665142+0000", 2024-09-13T14:51:35.262 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.8202999999999996e-05 2024-09-13T14:51:35.262 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.262 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.262 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:51:35.262 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:36.667672+0000", 2024-09-13T14:51:35.262 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0025300790000000002 2024-09-13T14:51:35.262 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.262 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.263 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.263 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:36.667916+0000", 2024-09-13T14:51:35.263 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00024384799999999999 2024-09-13T14:51:35.263 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.263 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.263 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.263 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:36.668228+0000", 2024-09-13T14:51:35.263 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00031239199999999999 2024-09-13T14:51:35.263 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.263 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.263 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.263 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:36.808631+0000", 2024-09-13T14:51:35.263 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.14040271900000001 2024-09-13T14:51:35.263 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.264 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.264 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:51:35.264 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:36.808675+0000", 2024-09-13T14:51:35.264 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4045000000000003e-05 2024-09-13T14:51:35.264 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.264 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.264 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:51:35.264 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:36.808768+0000", 2024-09-13T14:51:35.264 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.2825999999999995e-05 2024-09-13T14:51:35.264 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.264 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:51:35.264 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.264 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.264 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.265 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:198497 7.1 7:8a05d87f:::benchmark_data_smithi151_190906_object78036:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1160)", 2024-09-13T14:51:35.265 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:47:40.151110+0000", 2024-09-13T14:51:35.265 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 234.32913136299999, 2024-09-13T14:51:35.265 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.35196765099999999, 2024-09-13T14:51:35.265 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:51:35.265 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:35.265 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:51:35.265 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:51:35.265 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:51:35.265 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 198497 2024-09-13T14:51:35.265 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.265 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:51:35.265 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.265 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:51:35.266 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.151110+0000", 2024-09-13T14:51:35.266 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.266 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.266 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.266 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:51:35.266 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.151110+0000", 2024-09-13T14:51:35.266 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.266 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.266 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.266 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:51:35.266 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.151109+0000", 2024-09-13T14:51:35.266 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:51:35.266 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.267 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.267 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:51:35.267 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.151116+0000", 2024-09-13T14:51:35.267 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.5180000000000001e-06 2024-09-13T14:51:35.267 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.267 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.267 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:51:35.267 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.151129+0000", 2024-09-13T14:51:35.267 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3219e-05 2024-09-13T14:51:35.267 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.267 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.267 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:51:35.267 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.151135+0000", 2024-09-13T14:51:35.267 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.2330000000000003e-06 2024-09-13T14:51:35.267 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.268 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.268 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:51:35.268 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.151180+0000", 2024-09-13T14:51:35.268 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4542000000000001e-05 2024-09-13T14:51:35.268 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.268 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.268 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:51:35.268 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.151270+0000", 2024-09-13T14:51:35.268 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.0049000000000005e-05 2024-09-13T14:51:35.268 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.268 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.268 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:51:35.268 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.151350+0000", 2024-09-13T14:51:35.269 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.9717999999999998e-05 2024-09-13T14:51:35.269 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.269 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.269 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.269 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.154133+0000", 2024-09-13T14:51:35.269 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0027833559999999999 2024-09-13T14:51:35.269 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.269 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.269 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:51:35.269 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.154148+0000", 2024-09-13T14:51:35.269 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.5121e-05 2024-09-13T14:51:35.269 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.269 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.269 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.269 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.160489+0000", 2024-09-13T14:51:35.270 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0063409779999999997 2024-09-13T14:51:35.270 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.270 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.270 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.270 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.502943+0000", 2024-09-13T14:51:35.270 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.34245401399999997 2024-09-13T14:51:35.270 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.270 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.270 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:51:35.270 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.502983+0000", 2024-09-13T14:51:35.270 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9666999999999998e-05 2024-09-13T14:51:35.270 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.270 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.270 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:51:35.271 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.503078+0000", 2024-09-13T14:51:35.271 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.5376000000000003e-05 2024-09-13T14:51:35.271 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.271 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:51:35.271 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.271 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.271 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.271 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:214311 7.1 7:8d2f8dc9:::benchmark_data_smithi151_190906_object93850:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1162)", 2024-09-13T14:51:35.271 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:48:06.281766+0000", 2024-09-13T14:51:35.271 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 208.19847533800001, 2024-09-13T14:51:35.271 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.22899525900000001, 2024-09-13T14:51:35.271 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:51:35.271 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:35.271 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:51:35.272 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:51:35.272 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:51:35.272 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 214311 2024-09-13T14:51:35.272 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.272 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:51:35.272 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.272 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:51:35.272 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:06.281766+0000", 2024-09-13T14:51:35.272 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.272 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.272 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.272 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:51:35.272 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:06.281766+0000", 2024-09-13T14:51:35.272 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.273 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.273 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.273 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:51:35.273 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:06.281764+0000", 2024-09-13T14:51:35.273 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999976 2024-09-13T14:51:35.273 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.273 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.273 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:51:35.273 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:06.281777+0000", 2024-09-13T14:51:35.273 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2425999999999999e-05 2024-09-13T14:51:35.273 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.273 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.273 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:51:35.273 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:06.281806+0000", 2024-09-13T14:51:35.274 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.9430000000000001e-05 2024-09-13T14:51:35.274 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.274 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.274 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:51:35.274 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:06.281817+0000", 2024-09-13T14:51:35.274 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1146e-05 2024-09-13T14:51:35.274 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.274 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.274 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:51:35.274 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:06.281936+0000", 2024-09-13T14:51:35.274 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000118638 2024-09-13T14:51:35.274 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.274 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.274 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:51:35.275 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:06.282112+0000", 2024-09-13T14:51:35.275 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000176427 2024-09-13T14:51:35.275 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.275 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.275 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:51:35.275 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:06.282215+0000", 2024-09-13T14:51:35.275 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000102833 2024-09-13T14:51:35.275 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.275 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.275 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:51:35.275 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:06.284660+0000", 2024-09-13T14:51:35.275 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0024443749999999999 2024-09-13T14:51:35.275 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.275 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.276 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.276 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:06.284945+0000", 2024-09-13T14:51:35.276 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000285548 2024-09-13T14:51:35.276 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.276 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.276 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.276 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:06.284986+0000", 2024-09-13T14:51:35.276 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0745999999999999e-05 2024-09-13T14:51:35.276 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.276 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.276 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.276 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:06.510584+0000", 2024-09-13T14:51:35.276 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.225598294 2024-09-13T14:51:35.276 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.277 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.277 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:51:35.277 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:06.510640+0000", 2024-09-13T14:51:35.277 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5458999999999999e-05 2024-09-13T14:51:35.277 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.277 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.277 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:51:35.277 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:06.510762+0000", 2024-09-13T14:51:35.277 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012219999999999999 2024-09-13T14:51:35.277 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.277 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:51:35.277 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.277 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.277 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.278 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:214937 7.1 7:8f95f2cd:::benchmark_data_smithi151_190906_object94476:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1162)", 2024-09-13T14:51:35.278 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:48:07.619610+0000", 2024-09-13T14:51:35.278 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 206.860631668, 2024-09-13T14:51:35.278 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.21546647699999999, 2024-09-13T14:51:35.278 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:51:35.278 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:35.278 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:51:35.278 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:51:35.278 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:51:35.278 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 214937 2024-09-13T14:51:35.278 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.278 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:51:35.278 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.278 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:51:35.279 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:07.619610+0000", 2024-09-13T14:51:35.279 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.279 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.279 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.279 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:51:35.279 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:07.619610+0000", 2024-09-13T14:51:35.279 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.279 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.279 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.279 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:51:35.279 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:07.619609+0000", 2024-09-13T14:51:35.279 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:51:35.279 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.279 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.280 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:51:35.280 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:07.619617+0000", 2024-09-13T14:51:35.280 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.841e-06 2024-09-13T14:51:35.280 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.280 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.280 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:51:35.280 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:07.619629+0000", 2024-09-13T14:51:35.280 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1901e-05 2024-09-13T14:51:35.280 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.280 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.280 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:51:35.280 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:07.619634+0000", 2024-09-13T14:51:35.280 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3489999999999999e-06 2024-09-13T14:51:35.280 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.281 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.281 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:51:35.281 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:07.619684+0000", 2024-09-13T14:51:35.281 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9642999999999999e-05 2024-09-13T14:51:35.281 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.281 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.281 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:51:35.281 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:07.619826+0000", 2024-09-13T14:51:35.281 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000142232 2024-09-13T14:51:35.281 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.281 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.281 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:51:35.281 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:07.619974+0000", 2024-09-13T14:51:35.281 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000148426 2024-09-13T14:51:35.282 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.282 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.282 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.282 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:07.622857+0000", 2024-09-13T14:51:35.282 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002882431 2024-09-13T14:51:35.282 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.282 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.282 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:51:35.282 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:07.623093+0000", 2024-09-13T14:51:35.282 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000236314 2024-09-13T14:51:35.282 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.282 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.282 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.282 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:07.626337+0000", 2024-09-13T14:51:35.283 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0032440960000000001 2024-09-13T14:51:35.283 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.283 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.283 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.283 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:07.834952+0000", 2024-09-13T14:51:35.283 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.20861507600000001 2024-09-13T14:51:35.283 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.283 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.283 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:51:35.283 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:07.834992+0000", 2024-09-13T14:51:35.283 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9854000000000002e-05 2024-09-13T14:51:35.283 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.283 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.283 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:51:35.284 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:07.835077+0000", 2024-09-13T14:51:35.284 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.4512999999999995e-05 2024-09-13T14:51:35.284 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.284 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:51:35.284 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.284 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.284 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.284 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:218140 7.1 7:888097b9:::benchmark_data_smithi151_190906_object97679:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1162)", 2024-09-13T14:51:35.284 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:48:13.176488+0000", 2024-09-13T14:51:35.284 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 201.30375382, 2024-09-13T14:51:35.284 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.22006838400000001, 2024-09-13T14:51:35.284 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:51:35.284 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:35.285 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:51:35.285 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:51:35.285 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:51:35.285 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 218140 2024-09-13T14:51:35.285 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.285 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:51:35.285 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.285 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:51:35.285 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.176488+0000", 2024-09-13T14:51:35.285 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.285 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.285 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.285 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:51:35.286 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.176488+0000", 2024-09-13T14:51:35.286 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.286 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.286 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.286 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:51:35.286 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.176486+0000", 2024-09-13T14:51:35.286 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999976 2024-09-13T14:51:35.286 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.286 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.286 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:51:35.286 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.176496+0000", 2024-09-13T14:51:35.286 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0445000000000001e-05 2024-09-13T14:51:35.286 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.286 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.287 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:51:35.287 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.176511+0000", 2024-09-13T14:51:35.287 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4611e-05 2024-09-13T14:51:35.287 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.287 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.287 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:51:35.287 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.176518+0000", 2024-09-13T14:51:35.287 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.3329999999999999e-06 2024-09-13T14:51:35.287 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.287 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.287 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:51:35.287 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.176565+0000", 2024-09-13T14:51:35.287 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.6823000000000002e-05 2024-09-13T14:51:35.287 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.288 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.288 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:51:35.288 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.176731+0000", 2024-09-13T14:51:35.288 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00016566399999999999 2024-09-13T14:51:35.288 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.288 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.288 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:51:35.288 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.176895+0000", 2024-09-13T14:51:35.288 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00016419199999999999 2024-09-13T14:51:35.288 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.288 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.288 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.288 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.183503+0000", 2024-09-13T14:51:35.288 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0066076640000000001 2024-09-13T14:51:35.289 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.289 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.289 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:51:35.289 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.183511+0000", 2024-09-13T14:51:35.289 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.2979999999999992e-06 2024-09-13T14:51:35.289 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.289 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.289 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.289 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.183548+0000", 2024-09-13T14:51:35.289 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.7453000000000002e-05 2024-09-13T14:51:35.289 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.289 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.289 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.289 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.396308+0000", 2024-09-13T14:51:35.290 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.21275965199999999 2024-09-13T14:51:35.290 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.290 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.290 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:51:35.290 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.396437+0000", 2024-09-13T14:51:35.290 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012852800000000001 2024-09-13T14:51:35.290 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.290 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.290 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:51:35.290 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.396556+0000", 2024-09-13T14:51:35.290 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000119867 2024-09-13T14:51:35.290 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.290 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:51:35.290 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.291 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.291 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.291 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:218143 7.1 7:82685e6a:::benchmark_data_smithi151_190906_object97682:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1162)", 2024-09-13T14:51:35.291 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:48:13.178718+0000", 2024-09-13T14:51:35.291 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 201.30152400200001, 2024-09-13T14:51:35.291 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.22125558100000001, 2024-09-13T14:51:35.291 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:51:35.291 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:35.291 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:51:35.291 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:51:35.291 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:51:35.291 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 218143 2024-09-13T14:51:35.291 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.291 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:51:35.292 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.292 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:51:35.292 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.178718+0000", 2024-09-13T14:51:35.292 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.292 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.292 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.292 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:51:35.292 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.178718+0000", 2024-09-13T14:51:35.292 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.292 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.292 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.292 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:51:35.292 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.178717+0000", 2024-09-13T14:51:35.292 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:51:35.293 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.293 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.293 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:51:35.293 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.178727+0000", 2024-09-13T14:51:35.293 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.9329999999999996e-06 2024-09-13T14:51:35.293 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.293 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.293 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:51:35.293 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.178740+0000", 2024-09-13T14:51:35.293 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3879999999999999e-05 2024-09-13T14:51:35.293 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.293 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.293 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:51:35.293 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.178757+0000", 2024-09-13T14:51:35.294 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6286999999999998e-05 2024-09-13T14:51:35.294 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.294 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.294 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:51:35.294 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.180149+0000", 2024-09-13T14:51:35.294 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0013918419999999999 2024-09-13T14:51:35.294 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.294 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.294 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:51:35.294 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.180229+0000", 2024-09-13T14:51:35.294 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0774999999999998e-05 2024-09-13T14:51:35.294 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.294 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.294 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:51:35.294 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.180311+0000", 2024-09-13T14:51:35.295 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.1723000000000003e-05 2024-09-13T14:51:35.295 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.295 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.295 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.295 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.183576+0000", 2024-09-13T14:51:35.295 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0032653869999999998 2024-09-13T14:51:35.295 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.295 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.295 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:51:35.295 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.183585+0000", 2024-09-13T14:51:35.295 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.8429999999999999e-06 2024-09-13T14:51:35.295 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.295 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.295 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.296 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.183735+0000", 2024-09-13T14:51:35.296 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014961600000000001 2024-09-13T14:51:35.296 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.296 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.296 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.296 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.399781+0000", 2024-09-13T14:51:35.296 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.21604623100000001 2024-09-13T14:51:35.296 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.296 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.296 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:51:35.296 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.399835+0000", 2024-09-13T14:51:35.296 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3831e-05 2024-09-13T14:51:35.296 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.296 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.297 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:51:35.297 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.399973+0000", 2024-09-13T14:51:35.297 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000138398 2024-09-13T14:51:35.297 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.297 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:51:35.297 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.297 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.297 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.297 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:219554 7.1 7:878273f9:::benchmark_data_smithi151_190906_object99093:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1162)", 2024-09-13T14:51:35.297 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:48:15.671473+0000", 2024-09-13T14:51:35.297 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 198.808768993, 2024-09-13T14:51:35.297 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.18064495799999999, 2024-09-13T14:51:35.297 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:51:35.297 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:35.298 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:51:35.298 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:51:35.298 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:51:35.298 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 219554 2024-09-13T14:51:35.298 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.298 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:51:35.298 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.298 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:51:35.298 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.671473+0000", 2024-09-13T14:51:35.298 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.298 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.298 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.298 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:51:35.298 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.671473+0000", 2024-09-13T14:51:35.299 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.299 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.299 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.299 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:51:35.299 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.671472+0000", 2024-09-13T14:51:35.299 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:51:35.299 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.299 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.299 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:51:35.299 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.671478+0000", 2024-09-13T14:51:35.299 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.9089999999999997e-06 2024-09-13T14:51:35.299 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.299 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.299 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:51:35.299 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.671489+0000", 2024-09-13T14:51:35.300 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.152e-05 2024-09-13T14:51:35.300 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.300 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.300 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:51:35.300 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.671493+0000", 2024-09-13T14:51:35.300 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3499999999999999e-06 2024-09-13T14:51:35.300 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.300 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.300 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:51:35.300 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.671542+0000", 2024-09-13T14:51:35.300 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.863e-05 2024-09-13T14:51:35.300 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.300 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.300 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:51:35.301 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.671688+0000", 2024-09-13T14:51:35.301 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000145403 2024-09-13T14:51:35.301 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.301 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.301 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:51:35.301 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.671847+0000", 2024-09-13T14:51:35.301 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000159576 2024-09-13T14:51:35.301 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.301 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.301 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.301 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.675007+0000", 2024-09-13T14:51:35.301 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0031600040000000001 2024-09-13T14:51:35.301 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.301 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.301 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:51:35.302 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.675201+0000", 2024-09-13T14:51:35.302 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00019353799999999999 2024-09-13T14:51:35.302 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.302 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.302 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.302 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.683113+0000", 2024-09-13T14:51:35.302 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0079122959999999992 2024-09-13T14:51:35.302 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.302 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.302 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.302 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.851973+0000", 2024-09-13T14:51:35.302 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.16886020500000001 2024-09-13T14:51:35.302 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.303 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.303 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:51:35.303 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.852012+0000", 2024-09-13T14:51:35.303 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9106999999999997e-05 2024-09-13T14:51:35.303 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.303 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.303 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:51:35.303 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.852118+0000", 2024-09-13T14:51:35.303 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00010555799999999999 2024-09-13T14:51:35.303 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.303 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:51:35.303 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.303 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.303 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.303 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:219556 7.1 7:8cd60c5c:::benchmark_data_smithi151_190906_object99095:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1162)", 2024-09-13T14:51:35.304 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:48:15.675894+0000", 2024-09-13T14:51:35.304 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 198.80434743399999, 2024-09-13T14:51:35.304 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.18125966499999999, 2024-09-13T14:51:35.304 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:51:35.304 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:35.304 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:51:35.304 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:51:35.304 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:51:35.304 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 219556 2024-09-13T14:51:35.304 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.304 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:51:35.304 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.304 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:51:35.304 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.675894+0000", 2024-09-13T14:51:35.305 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.305 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.305 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.305 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:51:35.305 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.675894+0000", 2024-09-13T14:51:35.305 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.305 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.305 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.305 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:51:35.305 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.675893+0000", 2024-09-13T14:51:35.305 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:51:35.306 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.306 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.306 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:51:35.306 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.675903+0000", 2024-09-13T14:51:35.306 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0209e-05 2024-09-13T14:51:35.306 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.306 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.306 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:51:35.306 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.675917+0000", 2024-09-13T14:51:35.306 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3592e-05 2024-09-13T14:51:35.306 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.306 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.306 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:51:35.306 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.675923+0000", 2024-09-13T14:51:35.307 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.0310000000000004e-06 2024-09-13T14:51:35.307 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.307 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.307 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:51:35.307 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.675966+0000", 2024-09-13T14:51:35.307 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.2697999999999997e-05 2024-09-13T14:51:35.307 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.307 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.307 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:51:35.307 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.676101+0000", 2024-09-13T14:51:35.307 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013541900000000001 2024-09-13T14:51:35.307 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.307 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.307 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:51:35.308 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.676251+0000", 2024-09-13T14:51:35.308 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000150321 2024-09-13T14:51:35.308 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.308 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.308 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.308 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.679360+0000", 2024-09-13T14:51:35.308 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.003108446 2024-09-13T14:51:35.308 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.308 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.308 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:51:35.308 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.679376+0000", 2024-09-13T14:51:35.308 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6490999999999998e-05 2024-09-13T14:51:35.308 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.308 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.309 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.309 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.683782+0000", 2024-09-13T14:51:35.309 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0044055079999999998 2024-09-13T14:51:35.309 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.309 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.309 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.309 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.856970+0000", 2024-09-13T14:51:35.309 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.17318771999999999 2024-09-13T14:51:35.309 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.309 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.309 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:51:35.309 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.857024+0000", 2024-09-13T14:51:35.309 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4577000000000003e-05 2024-09-13T14:51:35.309 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.310 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.310 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:51:35.310 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.857154+0000", 2024-09-13T14:51:35.310 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000129949 2024-09-13T14:51:35.310 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.310 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:51:35.310 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.310 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.310 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.310 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:219564 7.1 7:8f49224e:::benchmark_data_smithi151_190906_object99103:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1162)", 2024-09-13T14:51:35.310 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:48:15.684515+0000", 2024-09-13T14:51:35.310 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 198.79572642799999, 2024-09-13T14:51:35.310 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.17284477100000001, 2024-09-13T14:51:35.310 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:51:35.310 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:35.311 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:51:35.311 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:51:35.311 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:51:35.311 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 219564 2024-09-13T14:51:35.311 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.311 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:51:35.311 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.311 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:51:35.311 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.684515+0000", 2024-09-13T14:51:35.311 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.311 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.311 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.311 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:51:35.311 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.684515+0000", 2024-09-13T14:51:35.311 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.312 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.312 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.312 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:51:35.312 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.684514+0000", 2024-09-13T14:51:35.312 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:51:35.312 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.312 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.312 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:51:35.312 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.684521+0000", 2024-09-13T14:51:35.312 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.9929999999999998e-06 2024-09-13T14:51:35.312 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.312 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.312 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:51:35.312 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.684533+0000", 2024-09-13T14:51:35.313 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2046e-05 2024-09-13T14:51:35.313 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.313 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.313 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:51:35.313 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.684539+0000", 2024-09-13T14:51:35.313 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.268e-06 2024-09-13T14:51:35.313 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.313 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.313 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:51:35.313 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.684583+0000", 2024-09-13T14:51:35.313 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3494000000000001e-05 2024-09-13T14:51:35.313 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.313 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.314 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:51:35.314 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.684717+0000", 2024-09-13T14:51:35.314 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000134566 2024-09-13T14:51:35.314 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.314 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.314 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:51:35.314 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.684854+0000", 2024-09-13T14:51:35.314 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013663400000000001 2024-09-13T14:51:35.314 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.314 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.314 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.314 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.688009+0000", 2024-09-13T14:51:35.314 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.003155153 2024-09-13T14:51:35.314 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.314 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.315 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:51:35.315 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.688100+0000", 2024-09-13T14:51:35.315 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.0968e-05 2024-09-13T14:51:35.315 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.315 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.315 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.315 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.688293+0000", 2024-09-13T14:51:35.315 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00019281100000000001 2024-09-13T14:51:35.315 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.315 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.315 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.315 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.857198+0000", 2024-09-13T14:51:35.315 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.16890530500000001 2024-09-13T14:51:35.315 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.316 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.316 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:51:35.316 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.857251+0000", 2024-09-13T14:51:35.316 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.2417999999999999e-05 2024-09-13T14:51:35.316 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.316 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.316 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:51:35.316 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.857360+0000", 2024-09-13T14:51:35.316 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000109527 2024-09-13T14:51:35.316 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.316 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:51:35.316 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.316 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.317 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.317 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:239454 7.1 7:84cced3c:::benchmark_data_smithi151_190906_object118993:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1164)", 2024-09-13T14:51:35.317 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:48:48.828811+0000", 2024-09-13T14:51:35.317 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 165.651430487, 2024-09-13T14:51:35.317 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.129271951, 2024-09-13T14:51:35.317 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:51:35.317 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:35.317 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:51:35.317 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:51:35.317 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:51:35.317 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 239454 2024-09-13T14:51:35.317 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.317 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:51:35.317 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.318 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:51:35.318 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:48.828811+0000", 2024-09-13T14:51:35.318 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.318 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.318 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.318 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:51:35.318 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:48.828811+0000", 2024-09-13T14:51:35.318 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.318 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.318 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.318 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:51:35.318 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:48.828810+0000", 2024-09-13T14:51:35.318 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:51:35.318 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.319 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.319 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:51:35.319 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:48.828816+0000", 2024-09-13T14:51:35.319 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3759999999999996e-06 2024-09-13T14:51:35.319 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.319 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.319 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:51:35.319 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:48.828828+0000", 2024-09-13T14:51:35.319 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1289e-05 2024-09-13T14:51:35.319 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.319 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.319 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:51:35.319 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:48.828834+0000", 2024-09-13T14:51:35.319 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1190000000000002e-06 2024-09-13T14:51:35.319 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.320 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.320 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:51:35.320 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:48.830811+0000", 2024-09-13T14:51:35.320 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001977389 2024-09-13T14:51:35.320 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.320 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.320 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:51:35.320 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:48.830924+0000", 2024-09-13T14:51:35.320 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000112491 2024-09-13T14:51:35.320 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.320 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.320 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:51:35.320 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:48.831006+0000", 2024-09-13T14:51:35.321 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.1861999999999997e-05 2024-09-13T14:51:35.321 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.321 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.321 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:51:35.321 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:48.833720+0000", 2024-09-13T14:51:35.321 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0027140549999999999 2024-09-13T14:51:35.321 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.321 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.321 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.321 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:48.834270+0000", 2024-09-13T14:51:35.321 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00055069500000000003 2024-09-13T14:51:35.321 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.321 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.321 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.321 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:48.834682+0000", 2024-09-13T14:51:35.322 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00041196100000000002 2024-09-13T14:51:35.322 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.322 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.322 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.322 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:48.957932+0000", 2024-09-13T14:51:35.322 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.123249452 2024-09-13T14:51:35.322 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.322 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.322 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:51:35.322 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:48.957988+0000", 2024-09-13T14:51:35.322 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6354999999999998e-05 2024-09-13T14:51:35.322 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.322 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.322 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:51:35.323 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:48.958083+0000", 2024-09-13T14:51:35.323 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.5134000000000001e-05 2024-09-13T14:51:35.323 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.323 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:51:35.323 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.323 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.323 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.323 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:240176 7.1 7:8fe24462:::benchmark_data_smithi151_190906_object119715:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1164)", 2024-09-13T14:51:35.323 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:48:50.078607+0000", 2024-09-13T14:51:35.323 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 164.40163457200001, 2024-09-13T14:51:35.323 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.14877069000000001, 2024-09-13T14:51:35.323 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:51:35.323 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:35.323 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:51:35.324 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:51:35.324 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:51:35.324 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 240176 2024-09-13T14:51:35.324 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.324 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:51:35.324 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.324 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:51:35.324 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:50.078607+0000", 2024-09-13T14:51:35.324 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.324 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.324 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.324 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:51:35.324 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:50.078607+0000", 2024-09-13T14:51:35.324 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.324 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.325 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.325 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:51:35.325 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:50.078606+0000", 2024-09-13T14:51:35.325 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:51:35.325 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.325 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.325 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:51:35.325 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:50.078613+0000", 2024-09-13T14:51:35.325 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.2810000000000003e-06 2024-09-13T14:51:35.325 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.325 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.325 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:51:35.325 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:50.078626+0000", 2024-09-13T14:51:35.325 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3178e-05 2024-09-13T14:51:35.326 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.326 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.326 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:51:35.326 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:50.078631+0000", 2024-09-13T14:51:35.326 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.074e-06 2024-09-13T14:51:35.326 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.326 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.326 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:51:35.326 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:50.078685+0000", 2024-09-13T14:51:35.326 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3054999999999999e-05 2024-09-13T14:51:35.326 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.326 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.326 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:51:35.326 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:50.078780+0000", 2024-09-13T14:51:35.326 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.4982000000000006e-05 2024-09-13T14:51:35.327 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.327 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.327 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:51:35.327 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:50.078867+0000", 2024-09-13T14:51:35.327 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.7211000000000002e-05 2024-09-13T14:51:35.327 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.327 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.327 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:51:35.327 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:50.081423+0000", 2024-09-13T14:51:35.327 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002556647 2024-09-13T14:51:35.327 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.327 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.327 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.327 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:50.081514+0000", 2024-09-13T14:51:35.328 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.0669999999999998e-05 2024-09-13T14:51:35.328 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.328 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.328 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.328 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:50.081625+0000", 2024-09-13T14:51:35.328 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000111047 2024-09-13T14:51:35.328 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.328 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.328 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.328 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:50.227273+0000", 2024-09-13T14:51:35.328 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.14564764399999999 2024-09-13T14:51:35.328 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.328 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.328 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:51:35.329 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:50.227309+0000", 2024-09-13T14:51:35.329 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.6434999999999999e-05 2024-09-13T14:51:35.329 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.329 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.329 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:51:35.329 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:50.227378+0000", 2024-09-13T14:51:35.329 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.8786000000000002e-05 2024-09-13T14:51:35.329 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.329 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:51:35.329 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.329 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.329 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.329 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:42827 8.2 8:4433e755:::benchmark_data_smithi151_229097_object42826:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1172)", 2024-09-13T14:51:35.329 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:49:57.713598+0000", 2024-09-13T14:51:35.330 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 96.766644056000004, 2024-09-13T14:51:35.330 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.13415065700000001, 2024-09-13T14:51:35.330 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:51:35.330 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:35.330 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:51:35.330 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:51:35.330 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:51:35.330 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 42827 2024-09-13T14:51:35.330 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.330 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:51:35.330 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.330 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:51:35.330 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.713598+0000", 2024-09-13T14:51:35.330 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.331 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.331 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.331 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:51:35.331 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.713598+0000", 2024-09-13T14:51:35.331 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:51:35.331 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.331 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.331 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:51:35.331 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.713596+0000", 2024-09-13T14:51:35.331 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:51:35.331 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.331 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.331 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:51:35.331 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.713638+0000", 2024-09-13T14:51:35.331 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1916000000000003e-05 2024-09-13T14:51:35.332 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.332 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.332 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:51:35.332 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.713649+0000", 2024-09-13T14:51:35.332 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1114000000000001e-05 2024-09-13T14:51:35.332 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.332 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.332 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:51:35.332 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.713654+0000", 2024-09-13T14:51:35.332 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0879999999999997e-06 2024-09-13T14:51:35.332 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.332 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.332 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:51:35.332 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.713687+0000", 2024-09-13T14:51:35.333 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3478e-05 2024-09-13T14:51:35.333 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.333 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.333 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:51:35.333 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.713779+0000", 2024-09-13T14:51:35.333 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.2367999999999994e-05 2024-09-13T14:51:35.333 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.333 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.333 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:51:35.333 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.713880+0000", 2024-09-13T14:51:35.333 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000101043 2024-09-13T14:51:35.333 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.333 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.333 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:51:35.334 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.715459+0000", 2024-09-13T14:51:35.334 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001578238 2024-09-13T14:51:35.334 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.334 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.334 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.334 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.715892+0000", 2024-09-13T14:51:35.334 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000433485 2024-09-13T14:51:35.334 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.334 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.334 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:35.334 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.847624+0000", 2024-09-13T14:51:35.334 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.13173138700000001 2024-09-13T14:51:35.334 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.334 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.334 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:51:35.335 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.847712+0000", 2024-09-13T14:51:35.335 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.8432999999999998e-05 2024-09-13T14:51:35.335 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:51:35.335 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:51:35.335 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:51:35.335 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.847748+0000", 2024-09-13T14:51:35.335 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.6426e-05 2024-09-13T14:51:35.335 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.335 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:51:35.335 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.335 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:51:35.335 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:51:35.335 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:51:35.336 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:35.366 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:35.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:35.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:35.468 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:35.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:35.491 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:35.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:35.592 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:35.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:35 smithi184 ceph-mon[92527]: pgmap v1750: 119 pgs: 5 active+remapped+backfilling, 114 active+clean; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 909 op/s; 26093/313305 objects misplaced (8.328%); 39 MiB/s, 624 objects/s recovering 2024-09-13T14:51:35.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:35 smithi151 ceph-mon[104450]: pgmap v1750: 119 pgs: 5 active+remapped+backfilling, 114 active+clean; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 909 op/s; 26093/313305 objects misplaced (8.328%); 39 MiB/s, 624 objects/s recovering 2024-09-13T14:51:35.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:35 smithi151 ceph-mon[108966]: pgmap v1750: 119 pgs: 5 active+remapped+backfilling, 114 active+clean; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 909 op/s; 26093/313305 objects misplaced (8.328%); 39 MiB/s, 624 objects/s recovering 2024-09-13T14:51:35.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:35.622 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:35.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:35.724 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:35.755 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:35.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:35.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:35.857 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:35.884 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:35.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:35.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:35.988 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:36.018 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:36.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:36.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:36.120 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:36.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:36.152 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:36.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:36.253 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:36.288 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:36.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:36.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:36.390 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:36.424 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:36.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:36.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:36.526 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:36.578 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:36.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:36.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:36.680 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:36.708 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:36.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:36.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:36.810 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:36.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:36.845 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:36.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:36.946 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:36.988 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:36.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:36.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:37.092 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:37.126 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:37.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:37.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:37.229 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:37.250 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:37.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:37.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:37.352 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:37.384 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:37.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:37.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:37.486 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:37.519 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:37.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:37.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:37.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:37 smithi184 ceph-mon[92527]: pgmap v1751: 119 pgs: 5 active+remapped+backfilling, 114 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 927 op/s; 23763/307752 objects misplaced (7.721%); 42 MiB/s, 676 objects/s recovering 2024-09-13T14:51:37.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:37 smithi151 ceph-mon[104450]: pgmap v1751: 119 pgs: 5 active+remapped+backfilling, 114 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 927 op/s; 23763/307752 objects misplaced (7.721%); 42 MiB/s, 676 objects/s recovering 2024-09-13T14:51:37.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:37 smithi151 ceph-mon[108966]: pgmap v1751: 119 pgs: 5 active+remapped+backfilling, 114 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 927 op/s; 23763/307752 objects misplaced (7.721%); 42 MiB/s, 676 objects/s recovering 2024-09-13T14:51:37.621 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:37.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:37.652 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:37.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:37.753 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:37.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:37.793 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:37.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:37.894 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:37.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:37.927 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:37.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:38.028 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:38.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:38.064 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:38.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:38.165 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:38.198 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:38.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:38.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:38.300 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:38.345 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T14:51:38.346 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:38.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:38.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:38.448 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:38.489 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:38.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:38.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:38.591 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:38.629 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:38.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:38.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:38.731 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:38.732 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:51:38.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:38.769 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:38.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:38.870 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:38.896 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:38.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:38.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:38.998 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:39.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:39.026 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:39.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:39.127 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:39.163 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:39.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:39.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:39.264 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:39.298 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:39.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:39.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:39.394 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:39 smithi151 ceph-mon[104450]: pgmap v1752: 119 pgs: 5 active+remapped+backfilling, 114 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 927 op/s; 23763/307752 objects misplaced (7.721%); 42 MiB/s, 676 objects/s recovering 2024-09-13T14:51:39.394 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:39 smithi151 ceph-mon[108966]: pgmap v1752: 119 pgs: 5 active+remapped+backfilling, 114 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 927 op/s; 23763/307752 objects misplaced (7.721%); 42 MiB/s, 676 objects/s recovering 2024-09-13T14:51:39.400 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:39.433 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:39.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:39.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:39.534 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:39.569 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:39.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:39.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:39.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:39 smithi184 ceph-mon[92527]: pgmap v1752: 119 pgs: 5 active+remapped+backfilling, 114 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 927 op/s; 23763/307752 objects misplaced (7.721%); 42 MiB/s, 676 objects/s recovering 2024-09-13T14:51:39.672 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:39.708 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:39.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:39.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:39.813 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:39.853 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:39.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:39.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:39.956 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:39.984 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:39.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:39.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:40.086 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:40.123 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:40.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:40.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:40.225 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:40.258 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:40.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:40.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:40.359 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:40.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:40.393 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:40.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:40.494 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:40.515 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:40.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:40.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:40.575 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:51:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:51:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:51:40.617 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:40.660 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:40.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:40.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:40.763 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:40.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:40.801 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:40.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:40.902 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:40.937 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:40.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:40.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:41.039 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:41.079 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:41.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:41.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:41.181 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:41.214 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:41.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:41.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:41.216 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:41 smithi184 ceph-mon[92527]: pgmap v1753: 119 pgs: 5 active+remapped+backfilling, 114 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 841 op/s; 21601/300729 objects misplaced (7.183%); 40 MiB/s, 646 objects/s recovering 2024-09-13T14:51:41.245 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:51:41.245 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [], 2024-09-13T14:51:41.245 INFO:teuthology.orchestra.run.smithi184.stdout: "num_ops": 0 2024-09-13T14:51:41.245 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:51:41.316 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:41.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:41.342 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:41.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:41.443 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:41.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:41.478 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:41.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:41.579 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:41.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:41 smithi151 ceph-mon[104450]: pgmap v1753: 119 pgs: 5 active+remapped+backfilling, 114 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 841 op/s; 21601/300729 objects misplaced (7.183%); 40 MiB/s, 646 objects/s recovering 2024-09-13T14:51:41.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:41 smithi151 ceph-mon[108966]: pgmap v1753: 119 pgs: 5 active+remapped+backfilling, 114 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 841 op/s; 21601/300729 objects misplaced (7.183%); 40 MiB/s, 646 objects/s recovering 2024-09-13T14:51:41.623 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:41.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:41.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:41.725 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:41.751 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:41.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:41.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:41.853 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:41.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:41.886 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:41.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:41.988 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:42.027 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:42.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:42.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:42.129 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:42.177 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:42.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:42.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:42.267 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T14:51:42.280 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:42.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:42.325 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:42.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:42.427 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:42.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:42.461 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:42.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:42.562 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:42.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:42.608 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:42.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:42.666 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:51:42.678 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:42.709 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:42.748 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:42.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:42.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:42.851 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:42.879 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:42.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:42.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:42.980 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:43.024 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:43.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:43.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:43.093 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:51:43.095 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T14:51:43.126 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:43.155 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:43.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:43.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:43.249 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:43 smithi151 ceph-mon[104450]: pgmap v1754: 119 pgs: 5 active+remapped+backfilling, 114 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 874 op/s; 19394/295704 objects misplaced (6.559%); 44 MiB/s, 704 objects/s recovering 2024-09-13T14:51:43.249 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:43 smithi151 ceph-mon[108966]: pgmap v1754: 119 pgs: 5 active+remapped+backfilling, 114 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 874 op/s; 19394/295704 objects misplaced (6.559%); 44 MiB/s, 704 objects/s recovering 2024-09-13T14:51:43.257 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:43.292 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:43.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:43.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:43.394 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:43.429 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:43.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:43.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:43.531 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:43.564 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:43.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:43.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:43.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:43 smithi184 ceph-mon[92527]: pgmap v1754: 119 pgs: 5 active+remapped+backfilling, 114 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 874 op/s; 19394/295704 objects misplaced (6.559%); 44 MiB/s, 704 objects/s recovering 2024-09-13T14:51:43.666 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:43.704 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:43.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:43.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:43.806 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:43.837 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:43.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:43.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:43.939 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:43.979 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:43.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:43.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:44.081 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:44.115 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:51:44.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:44.122 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:44.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:44.223 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:44.255 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:44.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:44.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:44.265 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:44 smithi184 ceph-mon[92527]: from='client.55279 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:51:44.356 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:44.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:44.398 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:44.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:44.499 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:44.532 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:44.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:44.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:44.592 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:44 smithi151 ceph-mon[104450]: from='client.55279 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:51:44.593 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:44 smithi151 ceph-mon[108966]: from='client.55279 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:51:44.633 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:44.672 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:44.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:44.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:44.717 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:51:44.717 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [], 2024-09-13T14:51:44.717 INFO:teuthology.orchestra.run.smithi184.stdout: "complaint_time": 30, 2024-09-13T14:51:44.717 INFO:teuthology.orchestra.run.smithi184.stdout: "num_blocked_ops": 0 2024-09-13T14:51:44.718 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:51:44.773 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:44.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:44.830 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:44.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:44.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:51:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:51:44] "GET /metrics HTTP/1.1" 200 36696 "" "Prometheus/2.51.0" 2024-09-13T14:51:44.931 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:45.000 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:45.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:45.102 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:45.149 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:45.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:45.251 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:45.282 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:45.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:45.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:45 smithi151 ceph-mon[104450]: pgmap v1755: 119 pgs: 5 active+remapped+backfilling, 114 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 905 op/s; 17143/288516 objects misplaced (5.942%); 47 MiB/s, 758 objects/s recovering 2024-09-13T14:51:45.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:45 smithi151 ceph-mon[108966]: pgmap v1755: 119 pgs: 5 active+remapped+backfilling, 114 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 905 op/s; 17143/288516 objects misplaced (5.942%); 47 MiB/s, 758 objects/s recovering 2024-09-13T14:51:45.384 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:45.412 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:45.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:45.441 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:45 smithi184 ceph-mon[92527]: pgmap v1755: 119 pgs: 5 active+remapped+backfilling, 114 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 905 op/s; 17143/288516 objects misplaced (5.942%); 47 MiB/s, 758 objects/s recovering 2024-09-13T14:51:45.470 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T14:51:45.485 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1754,"stamp":"2024-09-13T14:51:41.592022+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6458619169,"num_objects":98568,"num_object_clones":0,"num_object_copies":295704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":19394,"num_objects_unfound":0,"num_objects_dirty":98568,"num_whiteouts":0,"num_read":21,"num_read_kb":52,"num_write":151263,"num_write_kb":7991503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12106,"num_bytes_recovered":791785596,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95417,"ondisk_log_size":95417,"up":357,"acting":357,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":358,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":21587580,"kb_used_data":18004228,"kb_used_omap":148,"kb_used_meta":3582955,"kb_avail":634628484,"statfs":{"total":671965249536,"available":649859567616,"internally_reserved":0,"allocated":18436329472,"data_stored":18410762394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":151697,"internal_metadata":3668946799},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,2,8,2,5,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-688010345,"num_objects":-10498,"num_object_clones":0,"num_object_copies":-31494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-11257,"num_objects_unfound":0,"num_objects_dirty":-10498,"num_whiteouts":0,"num_read":-6,"num_read_kb":-17,"num_write":10495,"num_write_kb":-37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8462,"num_bytes_recovered":554552215,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005400"},"pg_stats":[{"pgid":"1.1d","version":"261'100","reported_seq":139,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.118610+0000","last_change":"2024-09-13T14:51:19.907325+0000","last_active":"2024-09-13T14:51:29.118610+0000","last_peered":"2024-09-13T14:51:29.118610+0000","last_clean":"2024-09-13T14:51:29.118610+0000","last_became_active":"2024-09-13T14:51:19.906625+0000","last_became_peered":"2024-09-13T14:51:19.906625+0000","last_unstale":"2024-09-13T14:51:29.118610+0000","last_undegraded":"2024-09-13T14:51:29.118610+0000","last_fullsized":"2024-09-13T14:51:29.118610+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:17.988345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:17.988345+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:36.836053+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":1738,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.108500+0000","last_change":"2024-09-13T14:29:42.541263+0000","last_active":"2024-09-13T14:51:29.108500+0000","last_peered":"2024-09-13T14:51:29.108500+0000","last_clean":"2024-09-13T14:51:29.108500+0000","last_became_active":"2024-09-13T14:29:42.540873+0000","last_became_peered":"2024-09-13T14:29:42.540873+0000","last_unstale":"2024-09-13T14:51:29.108500+0000","last_undegraded":"2024-09-13T14:51:29.108500+0000","last_fullsized":"2024-09-13T14:51:29.108500+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:20.611206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1093,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.119924+0000","last_change":"2024-09-13T14:51:19.906192+0000","last_active":"2024-09-13T14:51:29.119924+0000","last_peered":"2024-09-13T14:51:29.119924+0000","last_clean":"2024-09-13T14:51:29.119924+0000","last_became_active":"2024-09-13T14:51:19.905896+0000","last_became_peered":"2024-09-13T14:51:19.905896+0000","last_unstale":"2024-09-13T14:51:29.119924+0000","last_undegraded":"2024-09-13T14:51:29.119924+0000","last_fullsized":"2024-09-13T14:51:29.119924+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:00:47.332946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":77,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.124261+0000","last_change":"2024-09-13T14:45:19.810203+0000","last_active":"2024-09-13T14:51:29.124261+0000","last_peered":"2024-09-13T14:51:29.124261+0000","last_clean":"2024-09-13T14:51:29.124261+0000","last_became_active":"2024-09-13T14:45:19.809693+0000","last_became_peered":"2024-09-13T14:45:19.809693+0000","last_unstale":"2024-09-13T14:51:29.124261+0000","last_undegraded":"2024-09-13T14:51:29.124261+0000","last_fullsized":"2024-09-13T14:51:29.124261+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:20.219673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":76,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.123102+0000","last_change":"2024-09-13T14:45:19.807991+0000","last_active":"2024-09-13T14:51:29.123102+0000","last_peered":"2024-09-13T14:51:29.123102+0000","last_clean":"2024-09-13T14:51:29.123102+0000","last_became_active":"2024-09-13T14:45:19.807662+0000","last_became_peered":"2024-09-13T14:45:19.807662+0000","last_unstale":"2024-09-13T14:51:29.123102+0000","last_undegraded":"2024-09-13T14:51:29.123102+0000","last_fullsized":"2024-09-13T14:51:29.123102+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:29.306108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"261'100","reported_seq":1251,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.888945+0000","last_change":"2024-09-13T14:44:54.259562+0000","last_active":"2024-09-13T14:51:20.888945+0000","last_peered":"2024-09-13T14:51:20.888945+0000","last_clean":"2024-09-13T14:51:20.888945+0000","last_became_active":"2024-09-13T14:43:14.563717+0000","last_became_peered":"2024-09-13T14:43:14.563717+0000","last_unstale":"2024-09-13T14:51:20.888945+0000","last_undegraded":"2024-09-13T14:51:20.888945+0000","last_fullsized":"2024-09-13T14:51:20.888945+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_clean_scrub_stamp":"2024-09-13T14:44:54.259482+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:43:52.729109+0000","scrub_duration":0.015265569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":1122,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.890857+0000","last_change":"2024-09-13T14:51:19.931014+0000","last_active":"2024-09-13T14:51:20.890857+0000","last_peered":"2024-09-13T14:51:20.890857+0000","last_clean":"2024-09-13T14:51:20.890857+0000","last_became_active":"2024-09-13T14:51:19.905511+0000","last_became_peered":"2024-09-13T14:51:19.905511+0000","last_unstale":"2024-09-13T14:51:20.890857+0000","last_undegraded":"2024-09-13T14:51:20.890857+0000","last_fullsized":"2024-09-13T14:51:20.890857+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:09:45.562438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":602,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.123497+0000","last_change":"2024-09-13T14:51:19.897388+0000","last_active":"2024-09-13T14:51:29.123497+0000","last_peered":"2024-09-13T14:51:29.123497+0000","last_clean":"2024-09-13T14:51:29.123497+0000","last_became_active":"2024-09-13T14:51:19.897092+0000","last_became_peered":"2024-09-13T14:51:19.897092+0000","last_unstale":"2024-09-13T14:51:29.123497+0000","last_undegraded":"2024-09-13T14:51:29.123497+0000","last_fullsized":"2024-09-13T14:51:29.123497+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:53.450593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1e","version":"261'100","reported_seq":94,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.124401+0000","last_change":"2024-09-13T14:45:35.947332+0000","last_active":"2024-09-13T14:51:29.124401+0000","last_peered":"2024-09-13T14:51:29.124401+0000","last_clean":"2024-09-13T14:51:29.124401+0000","last_became_active":"2024-09-13T14:45:19.810775+0000","last_became_peered":"2024-09-13T14:45:19.810775+0000","last_unstale":"2024-09-13T14:51:29.124401+0000","last_undegraded":"2024-09-13T14:51:29.124401+0000","last_fullsized":"2024-09-13T14:51:29.124401+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:35.947265+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:43:41.569795+0000","last_clean_scrub_stamp":"2024-09-13T14:45:35.947265+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:00.606370+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":1167,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.906053+0000","last_change":"2024-09-13T14:51:19.914073+0000","last_active":"2024-09-13T14:51:20.906053+0000","last_peered":"2024-09-13T14:51:20.906053+0000","last_clean":"2024-09-13T14:51:20.906053+0000","last_became_active":"2024-09-13T14:51:19.902618+0000","last_became_peered":"2024-09-13T14:51:19.902618+0000","last_unstale":"2024-09-13T14:51:20.906053+0000","last_undegraded":"2024-09-13T14:51:20.906053+0000","last_fullsized":"2024-09-13T14:51:20.906053+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:17:55.364334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1103,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.125571+0000","last_change":"2024-09-13T14:51:19.911521+0000","last_active":"2024-09-13T14:51:29.125571+0000","last_peered":"2024-09-13T14:51:29.125571+0000","last_clean":"2024-09-13T14:51:29.125571+0000","last_became_active":"2024-09-13T14:51:19.910935+0000","last_became_peered":"2024-09-13T14:51:19.910935+0000","last_unstale":"2024-09-13T14:51:29.125571+0000","last_undegraded":"2024-09-13T14:51:29.125571+0000","last_fullsized":"2024-09-13T14:51:29.125571+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:13.016819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":1729,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.109038+0000","last_change":"2024-09-13T14:51:19.909986+0000","last_active":"2024-09-13T14:51:29.109038+0000","last_peered":"2024-09-13T14:51:29.109038+0000","last_clean":"2024-09-13T14:51:29.109038+0000","last_became_active":"2024-09-13T14:51:19.909459+0000","last_became_peered":"2024-09-13T14:51:19.909459+0000","last_unstale":"2024-09-13T14:51:29.109038+0000","last_undegraded":"2024-09-13T14:51:29.109038+0000","last_fullsized":"2024-09-13T14:51:29.109038+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:44.484202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1744,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.119626+0000","last_change":"2024-09-13T14:45:19.808811+0000","last_active":"2024-09-13T14:51:29.119626+0000","last_peered":"2024-09-13T14:51:29.119626+0000","last_clean":"2024-09-13T14:51:29.119626+0000","last_became_active":"2024-09-13T14:45:19.808471+0000","last_became_peered":"2024-09-13T14:45:19.808471+0000","last_unstale":"2024-09-13T14:51:29.119626+0000","last_undegraded":"2024-09-13T14:51:29.119626+0000","last_fullsized":"2024-09-13T14:51:29.119626+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:53.638161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"261'100","reported_seq":76,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.123193+0000","last_change":"2024-09-13T14:45:19.813412+0000","last_active":"2024-09-13T14:51:29.123193+0000","last_peered":"2024-09-13T14:51:29.123193+0000","last_clean":"2024-09-13T14:51:29.123193+0000","last_became_active":"2024-09-13T14:45:19.812585+0000","last_became_peered":"2024-09-13T14:45:19.812585+0000","last_unstale":"2024-09-13T14:51:29.123193+0000","last_undegraded":"2024-09-13T14:51:29.123193+0000","last_fullsized":"2024-09-13T14:51:29.123193+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.614340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.614340+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:32:07.434109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":1112,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.118490+0000","last_change":"2024-09-13T14:51:19.909275+0000","last_active":"2024-09-13T14:51:29.118490+0000","last_peered":"2024-09-13T14:51:29.118490+0000","last_clean":"2024-09-13T14:51:29.118490+0000","last_became_active":"2024-09-13T14:51:19.906971+0000","last_became_peered":"2024-09-13T14:51:19.906971+0000","last_unstale":"2024-09-13T14:51:29.118490+0000","last_undegraded":"2024-09-13T14:51:29.118490+0000","last_fullsized":"2024-09-13T14:51:29.118490+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:44.343244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":83,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.123746+0000","last_change":"2024-09-13T14:51:19.902217+0000","last_active":"2024-09-13T14:51:29.123746+0000","last_peered":"2024-09-13T14:51:29.123746+0000","last_clean":"2024-09-13T14:51:29.123746+0000","last_became_active":"2024-09-13T14:51:19.901908+0000","last_became_peered":"2024-09-13T14:51:19.901908+0000","last_unstale":"2024-09-13T14:51:29.123746+0000","last_undegraded":"2024-09-13T14:51:29.123746+0000","last_fullsized":"2024-09-13T14:51:29.123746+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:32.001265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.18","version":"261'100","reported_seq":129,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.889700+0000","last_change":"2024-09-13T14:44:41.685537+0000","last_active":"2024-09-13T14:51:20.889700+0000","last_peered":"2024-09-13T14:51:20.889700+0000","last_clean":"2024-09-13T14:51:20.889700+0000","last_became_active":"2024-09-13T14:43:14.558502+0000","last_became_peered":"2024-09-13T14:43:14.558502+0000","last_unstale":"2024-09-13T14:51:20.889700+0000","last_undegraded":"2024-09-13T14:51:20.889700+0000","last_fullsized":"2024-09-13T14:51:20.889700+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1134,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:41.685124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:41.685124+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:37:58.412497+0000","scrub_duration":0.0075205970000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":132,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.118256+0000","last_change":"2024-09-13T14:51:19.909348+0000","last_active":"2024-09-13T14:51:29.118256+0000","last_peered":"2024-09-13T14:51:29.118256+0000","last_clean":"2024-09-13T14:51:29.118256+0000","last_became_active":"2024-09-13T14:51:19.908782+0000","last_became_peered":"2024-09-13T14:51:19.908782+0000","last_unstale":"2024-09-13T14:51:29.118256+0000","last_undegraded":"2024-09-13T14:51:29.118256+0000","last_fullsized":"2024-09-13T14:51:29.118256+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:56:57.088901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1760,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.112616+0000","last_change":"2024-09-13T14:45:19.815341+0000","last_active":"2024-09-13T14:51:29.112616+0000","last_peered":"2024-09-13T14:51:29.112616+0000","last_clean":"2024-09-13T14:51:29.112616+0000","last_became_active":"2024-09-13T14:45:19.810262+0000","last_became_peered":"2024-09-13T14:45:19.810262+0000","last_unstale":"2024-09-13T14:51:29.112616+0000","last_undegraded":"2024-09-13T14:51:29.112616+0000","last_fullsized":"2024-09-13T14:51:29.112616+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:39.847653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1b","version":"261'100","reported_seq":81,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.889334+0000","last_change":"2024-09-13T14:51:19.904448+0000","last_active":"2024-09-13T14:51:20.889334+0000","last_peered":"2024-09-13T14:51:20.889334+0000","last_clean":"2024-09-13T14:51:20.889334+0000","last_became_active":"2024-09-13T14:51:19.904096+0000","last_became_peered":"2024-09-13T14:51:19.904096+0000","last_unstale":"2024-09-13T14:51:20.889334+0000","last_undegraded":"2024-09-13T14:51:20.889334+0000","last_fullsized":"2024-09-13T14:51:20.889334+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:52.803242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:52.803242+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:47.588858+0000","scrub_duration":0.0019769660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":76,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.122897+0000","last_change":"2024-09-13T14:45:19.806623+0000","last_active":"2024-09-13T14:51:29.122897+0000","last_peered":"2024-09-13T14:51:29.122897+0000","last_clean":"2024-09-13T14:51:29.122897+0000","last_became_active":"2024-09-13T14:45:19.806219+0000","last_became_peered":"2024-09-13T14:45:19.806219+0000","last_unstale":"2024-09-13T14:51:29.122897+0000","last_undegraded":"2024-09-13T14:51:29.122897+0000","last_fullsized":"2024-09-13T14:51:29.122897+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:34:41.601201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1133,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.889393+0000","last_change":"2024-09-13T14:43:32.637751+0000","last_active":"2024-09-13T14:51:20.889393+0000","last_peered":"2024-09-13T14:51:20.889393+0000","last_clean":"2024-09-13T14:51:20.889393+0000","last_became_active":"2024-09-13T14:43:32.636901+0000","last_became_peered":"2024-09-13T14:43:32.636901+0000","last_unstale":"2024-09-13T14:51:20.889393+0000","last_undegraded":"2024-09-13T14:51:20.889393+0000","last_fullsized":"2024-09-13T14:51:20.889393+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:21.942419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1a","version":"261'100","reported_seq":16,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.888772+0000","last_change":"2024-09-13T14:51:19.903142+0000","last_active":"2024-09-13T14:51:20.888772+0000","last_peered":"2024-09-13T14:51:20.888772+0000","last_clean":"2024-09-13T14:51:20.888772+0000","last_became_active":"2024-09-13T14:51:19.902810+0000","last_became_peered":"2024-09-13T14:51:19.902810+0000","last_unstale":"2024-09-13T14:51:20.888772+0000","last_undegraded":"2024-09-13T14:51:20.888772+0000","last_fullsized":"2024-09-13T14:51:20.888772+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:41.909988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:41.909988+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":76,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.123087+0000","last_change":"2024-09-13T14:45:19.812071+0000","last_active":"2024-09-13T14:51:29.123087+0000","last_peered":"2024-09-13T14:51:29.123087+0000","last_clean":"2024-09-13T14:51:29.123087+0000","last_became_active":"2024-09-13T14:45:19.811336+0000","last_became_peered":"2024-09-13T14:45:19.811336+0000","last_unstale":"2024-09-13T14:51:29.123087+0000","last_undegraded":"2024-09-13T14:51:29.123087+0000","last_fullsized":"2024-09-13T14:51:29.123087+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:40:29.814608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1777,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.108369+0000","last_change":"2024-09-13T14:45:19.810478+0000","last_active":"2024-09-13T14:51:29.108369+0000","last_peered":"2024-09-13T14:51:29.108369+0000","last_clean":"2024-09-13T14:51:29.108369+0000","last_became_active":"2024-09-13T14:45:19.808278+0000","last_became_peered":"2024-09-13T14:45:19.808278+0000","last_unstale":"2024-09-13T14:51:29.108369+0000","last_undegraded":"2024-09-13T14:51:29.108369+0000","last_fullsized":"2024-09-13T14:51:29.108369+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:12.211089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"261'100","reported_seq":71,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.889547+0000","last_change":"2024-09-13T14:44:38.214961+0000","last_active":"2024-09-13T14:51:20.889547+0000","last_peered":"2024-09-13T14:51:20.889547+0000","last_clean":"2024-09-13T14:51:20.889547+0000","last_became_active":"2024-09-13T14:43:14.557883+0000","last_became_peered":"2024-09-13T14:43:14.557883+0000","last_unstale":"2024-09-13T14:51:20.889547+0000","last_undegraded":"2024-09-13T14:51:20.889547+0000","last_fullsized":"2024-09-13T14:51:20.889547+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:38.214865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:38.214865+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:49:36.013783+0000","scrub_duration":0.0016389639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1648,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.123351+0000","last_change":"2024-09-13T14:30:15.576921+0000","last_active":"2024-09-13T14:51:29.123351+0000","last_peered":"2024-09-13T14:51:29.123351+0000","last_clean":"2024-09-13T14:51:29.123351+0000","last_became_active":"2024-09-13T14:30:15.576607+0000","last_became_peered":"2024-09-13T14:30:15.576607+0000","last_unstale":"2024-09-13T14:51:29.123351+0000","last_undegraded":"2024-09-13T14:51:29.123351+0000","last_fullsized":"2024-09-13T14:51:29.123351+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:35.162460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1143,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.889244+0000","last_change":"2024-09-13T14:43:07.518801+0000","last_active":"2024-09-13T14:51:20.889244+0000","last_peered":"2024-09-13T14:51:20.889244+0000","last_clean":"2024-09-13T14:51:20.889244+0000","last_became_active":"2024-09-13T14:43:07.517610+0000","last_became_peered":"2024-09-13T14:43:07.517610+0000","last_unstale":"2024-09-13T14:51:20.889244+0000","last_undegraded":"2024-09-13T14:51:20.889244+0000","last_fullsized":"2024-09-13T14:51:20.889244+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:51.632254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"261'100","reported_seq":132,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.119490+0000","last_change":"2024-09-13T14:51:19.899337+0000","last_active":"2024-09-13T14:51:29.119490+0000","last_peered":"2024-09-13T14:51:29.119490+0000","last_clean":"2024-09-13T14:51:29.119490+0000","last_became_active":"2024-09-13T14:51:19.898806+0000","last_became_peered":"2024-09-13T14:51:19.898806+0000","last_unstale":"2024-09-13T14:51:29.119490+0000","last_undegraded":"2024-09-13T14:51:29.119490+0000","last_fullsized":"2024-09-13T14:51:29.119490+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.037517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.037517+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:04.124642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":1153,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.889315+0000","last_change":"2024-09-13T14:51:19.946789+0000","last_active":"2024-09-13T14:51:20.889315+0000","last_peered":"2024-09-13T14:51:20.889315+0000","last_clean":"2024-09-13T14:51:20.889315+0000","last_became_active":"2024-09-13T14:51:19.905677+0000","last_became_peered":"2024-09-13T14:51:19.905677+0000","last_unstale":"2024-09-13T14:51:20.889315+0000","last_undegraded":"2024-09-13T14:51:20.889315+0000","last_fullsized":"2024-09-13T14:51:20.889315+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:41:14.530433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":125,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.119008+0000","last_change":"2024-09-13T14:43:07.521576+0000","last_active":"2024-09-13T14:51:29.119008+0000","last_peered":"2024-09-13T14:51:29.119008+0000","last_clean":"2024-09-13T14:51:29.119008+0000","last_became_active":"2024-09-13T14:43:07.520478+0000","last_became_peered":"2024-09-13T14:43:07.520478+0000","last_unstale":"2024-09-13T14:51:29.119008+0000","last_undegraded":"2024-09-13T14:51:29.119008+0000","last_fullsized":"2024-09-13T14:51:29.119008+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:53.410135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"261'100","reported_seq":397,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.889313+0000","last_change":"2024-09-13T14:44:49.775941+0000","last_active":"2024-09-13T14:51:20.889313+0000","last_peered":"2024-09-13T14:51:20.889313+0000","last_clean":"2024-09-13T14:51:20.889313+0000","last_became_active":"2024-09-13T14:43:14.560297+0000","last_became_peered":"2024-09-13T14:43:14.560297+0000","last_unstale":"2024-09-13T14:51:20.889313+0000","last_undegraded":"2024-09-13T14:51:20.889313+0000","last_fullsized":"2024-09-13T14:51:20.889313+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:49.775844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:49.775844+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:15:53.138661+0000","scrub_duration":0.012350274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":333,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.889087+0000","last_change":"2024-09-13T14:51:19.922876+0000","last_active":"2024-09-13T14:51:20.889087+0000","last_peered":"2024-09-13T14:51:20.889087+0000","last_clean":"2024-09-13T14:51:20.889087+0000","last_became_active":"2024-09-13T14:51:19.897151+0000","last_became_peered":"2024-09-13T14:51:19.897151+0000","last_unstale":"2024-09-13T14:51:20.889087+0000","last_undegraded":"2024-09-13T14:51:20.889087+0000","last_fullsized":"2024-09-13T14:51:20.889087+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:53:08.858083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1093,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.122692+0000","last_change":"2024-09-13T14:51:19.913252+0000","last_active":"2024-09-13T14:51:29.122692+0000","last_peered":"2024-09-13T14:51:29.122692+0000","last_clean":"2024-09-13T14:51:29.122692+0000","last_became_active":"2024-09-13T14:51:19.912953+0000","last_became_peered":"2024-09-13T14:51:19.912953+0000","last_unstale":"2024-09-13T14:51:29.122692+0000","last_undegraded":"2024-09-13T14:51:29.122692+0000","last_fullsized":"2024-09-13T14:51:29.122692+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:39:27.026947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.16","version":"261'100","reported_seq":132,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.118912+0000","last_change":"2024-09-13T14:45:19.807461+0000","last_active":"2024-09-13T14:51:29.118912+0000","last_peered":"2024-09-13T14:51:29.118912+0000","last_clean":"2024-09-13T14:51:29.118912+0000","last_became_active":"2024-09-13T14:45:19.807117+0000","last_became_peered":"2024-09-13T14:45:19.807117+0000","last_unstale":"2024-09-13T14:51:29.118912+0000","last_undegraded":"2024-09-13T14:51:29.118912+0000","last_fullsized":"2024-09-13T14:51:29.118912+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.955197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.955197+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:03.531555+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.d","version":"1180'9590","reported_seq":9612,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:37.789929+0000","last_change":"2024-09-13T14:49:22.120980+0000","last_active":"2024-09-13T14:51:37.789929+0000","last_peered":"2024-09-13T14:51:37.789929+0000","last_clean":"2024-09-13T14:51:37.789929+0000","last_became_active":"2024-09-13T14:49:22.120684+0000","last_became_peered":"2024-09-13T14:49:22.120684+0000","last_unstale":"2024-09-13T14:51:37.789929+0000","last_undegraded":"2024-09-13T14:51:37.789929+0000","last_fullsized":"2024-09-13T14:51:37.789929+0000","mapping_epoch":1167,"log_start":"1172'3206","ondisk_log_start":"1172'3206","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":6384,"log_dups_size":0,"ondisk_log_size":6384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:49:34.732581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413859864,"num_objects":6316,"num_object_clones":0,"num_object_copies":18948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6316,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9590,"num_write_kb":508929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.890187+0000","last_change":"2024-09-13T14:51:19.904110+0000","last_active":"2024-09-13T14:51:20.890187+0000","last_peered":"2024-09-13T14:51:20.890187+0000","last_clean":"2024-09-13T14:51:20.890187+0000","last_became_active":"2024-09-13T14:51:19.903806+0000","last_became_peered":"2024-09-13T14:51:19.903806+0000","last_unstale":"2024-09-13T14:51:20.890187+0000","last_undegraded":"2024-09-13T14:51:20.890187+0000","last_fullsized":"2024-09-13T14:51:20.890187+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:11.557749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":605,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.889411+0000","last_change":"2024-09-13T14:43:32.635070+0000","last_active":"2024-09-13T14:51:20.889411+0000","last_peered":"2024-09-13T14:51:20.889411+0000","last_clean":"2024-09-13T14:51:20.889411+0000","last_became_active":"2024-09-13T14:43:32.634734+0000","last_became_peered":"2024-09-13T14:43:32.634734+0000","last_unstale":"2024-09-13T14:51:20.889411+0000","last_undegraded":"2024-09-13T14:51:20.889411+0000","last_fullsized":"2024-09-13T14:51:20.889411+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:33.107010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":1778,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.112761+0000","last_change":"2024-09-13T14:51:19.921328+0000","last_active":"2024-09-13T14:51:29.112761+0000","last_peered":"2024-09-13T14:51:29.112761+0000","last_clean":"2024-09-13T14:51:29.112761+0000","last_became_active":"2024-09-13T14:51:19.901833+0000","last_became_peered":"2024-09-13T14:51:19.901833+0000","last_unstale":"2024-09-13T14:51:29.112761+0000","last_undegraded":"2024-09-13T14:51:29.112761+0000","last_fullsized":"2024-09-13T14:51:29.112761+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:57.881817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"261'100","reported_seq":77,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.123366+0000","last_change":"2024-09-13T14:45:19.813527+0000","last_active":"2024-09-13T14:51:29.123366+0000","last_peered":"2024-09-13T14:51:29.123366+0000","last_clean":"2024-09-13T14:51:29.123366+0000","last_became_active":"2024-09-13T14:45:19.812834+0000","last_became_peered":"2024-09-13T14:45:19.812834+0000","last_unstale":"2024-09-13T14:51:29.123366+0000","last_undegraded":"2024-09-13T14:51:29.123366+0000","last_fullsized":"2024-09-13T14:51:29.123366+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.710192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.710192+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:08:55.152703+0000","scrub_duration":0.0019250910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"1180'9524","reported_seq":9562,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:39.921599+0000","last_change":"2024-09-13T14:49:22.114132+0000","last_active":"2024-09-13T14:51:39.921599+0000","last_peered":"2024-09-13T14:51:39.921599+0000","last_clean":"2024-09-13T14:51:39.921599+0000","last_became_active":"2024-09-13T14:49:22.113820+0000","last_became_peered":"2024-09-13T14:49:22.113820+0000","last_unstale":"2024-09-13T14:51:39.921599+0000","last_undegraded":"2024-09-13T14:51:39.921599+0000","last_fullsized":"2024-09-13T14:51:39.921599+0000","mapping_epoch":1167,"log_start":"1172'3546","ondisk_log_start":"1172'3546","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":5978,"log_dups_size":0,"ondisk_log_size":5978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:34:37.889567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404357120,"num_objects":6170,"num_object_clones":0,"num_object_copies":18510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9524,"num_write_kb":502208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":77,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.123179+0000","last_change":"2024-09-13T14:45:19.813025+0000","last_active":"2024-09-13T14:51:29.123179+0000","last_peered":"2024-09-13T14:51:29.123179+0000","last_clean":"2024-09-13T14:51:29.123179+0000","last_became_active":"2024-09-13T14:45:19.812714+0000","last_became_peered":"2024-09-13T14:45:19.812714+0000","last_unstale":"2024-09-13T14:51:29.123179+0000","last_undegraded":"2024-09-13T14:51:29.123179+0000","last_fullsized":"2024-09-13T14:51:29.123179+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:40.671012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":132,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.119170+0000","last_change":"2024-09-13T14:51:19.906672+0000","last_active":"2024-09-13T14:51:29.119170+0000","last_peered":"2024-09-13T14:51:29.119170+0000","last_clean":"2024-09-13T14:51:29.119170+0000","last_became_active":"2024-09-13T14:51:19.906296+0000","last_became_peered":"2024-09-13T14:51:19.906296+0000","last_unstale":"2024-09-13T14:51:29.119170+0000","last_undegraded":"2024-09-13T14:51:29.119170+0000","last_fullsized":"2024-09-13T14:51:29.119170+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:55:31.046356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":1680,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.126466+0000","last_change":"2024-09-13T14:30:15.581728+0000","last_active":"2024-09-13T14:51:29.126466+0000","last_peered":"2024-09-13T14:51:29.126466+0000","last_clean":"2024-09-13T14:51:29.126466+0000","last_became_active":"2024-09-13T14:30:15.581303+0000","last_became_peered":"2024-09-13T14:30:15.581303+0000","last_unstale":"2024-09-13T14:51:29.126466+0000","last_undegraded":"2024-09-13T14:51:29.126466+0000","last_fullsized":"2024-09-13T14:51:29.126466+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:10.027991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"261'100","reported_seq":32,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:31.310325+0000","last_change":"2024-09-13T14:51:31.310325+0000","last_active":"2024-09-13T14:51:31.310325+0000","last_peered":"2024-09-13T14:51:31.310325+0000","last_clean":"2024-09-13T14:51:31.310325+0000","last_became_active":"2024-09-13T14:51:19.899471+0000","last_became_peered":"2024-09-13T14:51:19.899471+0000","last_unstale":"2024-09-13T14:51:31.310325+0000","last_undegraded":"2024-09-13T14:51:31.310325+0000","last_fullsized":"2024-09-13T14:51:31.310325+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:51:31.310115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:51:31.310115+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:34:41.135431+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.f","version":"1180'9631","reported_seq":9668,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:39.811926+0000","last_change":"2024-09-13T14:49:22.118734+0000","last_active":"2024-09-13T14:51:39.811926+0000","last_peered":"2024-09-13T14:51:39.811926+0000","last_clean":"2024-09-13T14:51:39.811926+0000","last_became_active":"2024-09-13T14:49:22.118438+0000","last_became_peered":"2024-09-13T14:49:22.118438+0000","last_unstale":"2024-09-13T14:51:39.811926+0000","last_undegraded":"2024-09-13T14:51:39.811926+0000","last_fullsized":"2024-09-13T14:51:39.811926+0000","mapping_epoch":1167,"log_start":"1172'4358","ondisk_log_start":"1172'4358","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":5273,"log_dups_size":0,"ondisk_log_size":5273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:27:01.810072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401670144,"num_objects":6129,"num_object_clones":0,"num_object_copies":18387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9631,"num_write_kb":504320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1721,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.108993+0000","last_change":"2024-09-13T14:28:35.611508+0000","last_active":"2024-09-13T14:51:29.108993+0000","last_peered":"2024-09-13T14:51:29.108993+0000","last_clean":"2024-09-13T14:51:29.108993+0000","last_became_active":"2024-09-13T14:28:35.610106+0000","last_became_peered":"2024-09-13T14:28:35.610106+0000","last_unstale":"2024-09-13T14:51:29.108993+0000","last_undegraded":"2024-09-13T14:51:29.108993+0000","last_fullsized":"2024-09-13T14:51:29.108993+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:11.966958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1156,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.889619+0000","last_change":"2024-09-13T14:51:19.907402+0000","last_active":"2024-09-13T14:51:20.889619+0000","last_peered":"2024-09-13T14:51:20.889619+0000","last_clean":"2024-09-13T14:51:20.889619+0000","last_became_active":"2024-09-13T14:51:19.904775+0000","last_became_peered":"2024-09-13T14:51:19.904775+0000","last_unstale":"2024-09-13T14:51:20.889619+0000","last_undegraded":"2024-09-13T14:51:20.889619+0000","last_fullsized":"2024-09-13T14:51:20.889619+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:47:13.974154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":126,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.119629+0000","last_change":"2024-09-13T14:43:07.522750+0000","last_active":"2024-09-13T14:51:29.119629+0000","last_peered":"2024-09-13T14:51:29.119629+0000","last_clean":"2024-09-13T14:51:29.119629+0000","last_became_active":"2024-09-13T14:43:07.517267+0000","last_became_peered":"2024-09-13T14:43:07.517267+0000","last_unstale":"2024-09-13T14:51:29.119629+0000","last_undegraded":"2024-09-13T14:51:29.119629+0000","last_fullsized":"2024-09-13T14:51:29.119629+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:40.217748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"261'100","reported_seq":446,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.112398+0000","last_change":"2024-09-13T14:45:19.815050+0000","last_active":"2024-09-13T14:51:29.112398+0000","last_peered":"2024-09-13T14:51:29.112398+0000","last_clean":"2024-09-13T14:51:29.112398+0000","last_became_active":"2024-09-13T14:45:19.809732+0000","last_became_peered":"2024-09-13T14:45:19.809732+0000","last_unstale":"2024-09-13T14:51:29.112398+0000","last_undegraded":"2024-09-13T14:51:29.112398+0000","last_fullsized":"2024-09-13T14:51:29.112398+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:33.574443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:33.574443+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:51:40.261036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"1180'9502","reported_seq":13035,"reported_epoch":1180,"state":"active+remapped+backfilling","last_fresh":"2024-09-13T14:51:39.938203+0000","last_change":"2024-09-13T14:51:21.089870+0000","last_active":"2024-09-13T14:51:39.938203+0000","last_peered":"2024-09-13T14:51:39.938203+0000","last_clean":"2024-09-13T14:51:19.053567+0000","last_became_active":"2024-09-13T14:51:20.994825+0000","last_became_peered":"2024-09-13T14:51:20.994825+0000","last_unstale":"2024-09-13T14:51:39.938203+0000","last_undegraded":"2024-09-13T14:51:39.938203+0000","last_fullsized":"2024-09-13T14:51:39.938203+0000","mapping_epoch":1178,"log_start":"1172'3551","ondisk_log_start":"1172'3551","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":5951,"log_dups_size":0,"ondisk_log_size":5951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401342464,"num_objects":6124,"num_object_clones":0,"num_object_copies":18372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4503,"num_objects_unfound":0,"num_objects_dirty":6124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9502,"num_write_kb":500032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1742,"num_bytes_recovered":114098176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,4],"avail_no_missing":["0","2","4"],"object_location_counts":[{"shards":"0,2,4","objects":6124}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":133,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.118829+0000","last_change":"2024-09-13T14:51:19.910044+0000","last_active":"2024-09-13T14:51:29.118829+0000","last_peered":"2024-09-13T14:51:29.118829+0000","last_clean":"2024-09-13T14:51:29.118829+0000","last_became_active":"2024-09-13T14:51:19.908592+0000","last_became_peered":"2024-09-13T14:51:19.908592+0000","last_unstale":"2024-09-13T14:51:29.118829+0000","last_undegraded":"2024-09-13T14:51:29.118829+0000","last_fullsized":"2024-09-13T14:51:29.118829+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:16.467708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":126,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.119550+0000","last_change":"2024-09-13T14:43:07.525666+0000","last_active":"2024-09-13T14:51:29.119550+0000","last_peered":"2024-09-13T14:51:29.119550+0000","last_clean":"2024-09-13T14:51:29.119550+0000","last_became_active":"2024-09-13T14:43:07.524064+0000","last_became_peered":"2024-09-13T14:43:07.524064+0000","last_unstale":"2024-09-13T14:51:29.119550+0000","last_undegraded":"2024-09-13T14:51:29.119550+0000","last_fullsized":"2024-09-13T14:51:29.119550+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:10.270333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1160,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.890591+0000","last_change":"2024-09-13T14:45:19.804177+0000","last_active":"2024-09-13T14:51:20.890591+0000","last_peered":"2024-09-13T14:51:20.890591+0000","last_clean":"2024-09-13T14:51:20.890591+0000","last_became_active":"2024-09-13T14:45:19.803890+0000","last_became_peered":"2024-09-13T14:45:19.803890+0000","last_unstale":"2024-09-13T14:51:20.890591+0000","last_undegraded":"2024-09-13T14:51:20.890591+0000","last_fullsized":"2024-09-13T14:51:20.890591+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:25.140789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"261'100","reported_seq":93,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.126360+0000","last_change":"2024-09-13T14:45:36.965992+0000","last_active":"2024-09-13T14:51:29.126360+0000","last_peered":"2024-09-13T14:51:29.126360+0000","last_clean":"2024-09-13T14:51:29.126360+0000","last_became_active":"2024-09-13T14:45:19.811238+0000","last_became_peered":"2024-09-13T14:45:19.811238+0000","last_unstale":"2024-09-13T14:51:29.126360+0000","last_undegraded":"2024-09-13T14:51:29.126360+0000","last_fullsized":"2024-09-13T14:51:29.126360+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:36.965775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:45:36.965775+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:36:01.623251+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"1180'9551","reported_seq":9571,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:37.807320+0000","last_change":"2024-09-13T14:49:22.122277+0000","last_active":"2024-09-13T14:51:37.807320+0000","last_peered":"2024-09-13T14:51:37.807320+0000","last_clean":"2024-09-13T14:51:37.807320+0000","last_became_active":"2024-09-13T14:49:22.121980+0000","last_became_peered":"2024-09-13T14:49:22.121980+0000","last_unstale":"2024-09-13T14:51:37.807320+0000","last_undegraded":"2024-09-13T14:51:37.807320+0000","last_fullsized":"2024-09-13T14:51:37.807320+0000","mapping_epoch":1167,"log_start":"1172'3162","ondisk_log_start":"1172'3162","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":6389,"log_dups_size":0,"ondisk_log_size":6389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:58.275564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406650880,"num_objects":6205,"num_object_clones":0,"num_object_copies":18615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9551,"num_write_kb":504192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":59,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.905911+0000","last_change":"2024-09-13T14:45:19.806318+0000","last_active":"2024-09-13T14:51:20.905911+0000","last_peered":"2024-09-13T14:51:20.905911+0000","last_clean":"2024-09-13T14:51:20.905911+0000","last_became_active":"2024-09-13T14:45:19.806074+0000","last_became_peered":"2024-09-13T14:45:19.806074+0000","last_unstale":"2024-09-13T14:51:20.905911+0000","last_undegraded":"2024-09-13T14:51:20.905911+0000","last_fullsized":"2024-09-13T14:51:20.905911+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:21.679797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":490,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.109228+0000","last_change":"2024-09-13T14:51:19.910931+0000","last_active":"2024-09-13T14:51:29.109228+0000","last_peered":"2024-09-13T14:51:29.109228+0000","last_clean":"2024-09-13T14:51:29.109228+0000","last_became_active":"2024-09-13T14:51:19.910600+0000","last_became_peered":"2024-09-13T14:51:19.910600+0000","last_unstale":"2024-09-13T14:51:29.109228+0000","last_undegraded":"2024-09-13T14:51:29.109228+0000","last_fullsized":"2024-09-13T14:51:29.109228+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:57:45.304028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":1171,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.891011+0000","last_change":"2024-09-13T14:43:07.538919+0000","last_active":"2024-09-13T14:51:20.891011+0000","last_peered":"2024-09-13T14:51:20.891011+0000","last_clean":"2024-09-13T14:51:20.891011+0000","last_became_active":"2024-09-13T14:43:07.513109+0000","last_became_peered":"2024-09-13T14:43:07.513109+0000","last_unstale":"2024-09-13T14:51:20.891011+0000","last_undegraded":"2024-09-13T14:51:20.891011+0000","last_fullsized":"2024-09-13T14:51:20.891011+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:27.771720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"261'100","reported_seq":120,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.890057+0000","last_change":"2024-09-13T14:43:44.720333+0000","last_active":"2024-09-13T14:51:20.890057+0000","last_peered":"2024-09-13T14:51:20.890057+0000","last_clean":"2024-09-13T14:51:20.890057+0000","last_became_active":"2024-09-13T14:42:14.946591+0000","last_became_peered":"2024-09-13T14:42:14.946591+0000","last_unstale":"2024-09-13T14:51:20.890057+0000","last_undegraded":"2024-09-13T14:51:20.890057+0000","last_fullsized":"2024-09-13T14:51:20.890057+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:44.720258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:44.720258+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:33.281357+0000","scrub_duration":0.0018417920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"1180'9549","reported_seq":13615,"reported_epoch":1180,"state":"active+remapped+backfilling","last_fresh":"2024-09-13T14:51:39.809649+0000","last_change":"2024-09-13T14:51:21.027074+0000","last_active":"2024-09-13T14:51:39.809649+0000","last_peered":"2024-09-13T14:51:39.809649+0000","last_clean":"2024-09-13T14:51:19.032905+0000","last_became_active":"2024-09-13T14:51:20.995776+0000","last_became_peered":"2024-09-13T14:51:20.995776+0000","last_unstale":"2024-09-13T14:51:39.809649+0000","last_undegraded":"2024-09-13T14:51:39.809649+0000","last_fullsized":"2024-09-13T14:51:39.809649+0000","mapping_epoch":1178,"log_start":"1172'4236","ondisk_log_start":"1172'4236","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":5313,"log_dups_size":0,"ondisk_log_size":5313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401145856,"num_objects":6121,"num_object_clones":0,"num_object_copies":18363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4251,"num_objects_unfound":0,"num_objects_dirty":6121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9549,"num_write_kb":501440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2013,"num_bytes_recovered":131923968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,4],"avail_no_missing":["3","1","4"],"object_location_counts":[{"shards":"1,3,4","objects":6121}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":93,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.116007+0000","last_change":"2024-09-13T14:44:03.858211+0000","last_active":"2024-09-13T14:51:29.116007+0000","last_peered":"2024-09-13T14:51:29.116007+0000","last_clean":"2024-09-13T14:51:29.116007+0000","last_became_active":"2024-09-13T14:44:03.857928+0000","last_became_peered":"2024-09-13T14:44:03.857928+0000","last_unstale":"2024-09-13T14:51:29.116007+0000","last_undegraded":"2024-09-13T14:51:29.116007+0000","last_fullsized":"2024-09-13T14:51:29.116007+0000","mapping_epoch":1140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:28.428693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":126,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.120062+0000","last_change":"2024-09-13T14:43:07.516243+0000","last_active":"2024-09-13T14:51:29.120062+0000","last_peered":"2024-09-13T14:51:29.120062+0000","last_clean":"2024-09-13T14:51:29.120062+0000","last_became_active":"2024-09-13T14:43:07.515863+0000","last_became_peered":"2024-09-13T14:43:07.515863+0000","last_unstale":"2024-09-13T14:51:29.120062+0000","last_undegraded":"2024-09-13T14:51:29.120062+0000","last_fullsized":"2024-09-13T14:51:29.120062+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:08.085298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1762,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.122635+0000","last_change":"2024-09-13T14:45:19.812713+0000","last_active":"2024-09-13T14:51:29.122635+0000","last_peered":"2024-09-13T14:51:29.122635+0000","last_clean":"2024-09-13T14:51:29.122635+0000","last_became_active":"2024-09-13T14:45:19.812203+0000","last_became_peered":"2024-09-13T14:45:19.812203+0000","last_unstale":"2024-09-13T14:51:29.122635+0000","last_undegraded":"2024-09-13T14:51:29.122635+0000","last_fullsized":"2024-09-13T14:51:29.122635+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:30:46.149347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"261'100","reported_seq":1310,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.888678+0000","last_change":"2024-09-13T14:51:19.895551+0000","last_active":"2024-09-13T14:51:20.888678+0000","last_peered":"2024-09-13T14:51:20.888678+0000","last_clean":"2024-09-13T14:51:20.888678+0000","last_became_active":"2024-09-13T14:51:19.895204+0000","last_became_peered":"2024-09-13T14:51:19.895204+0000","last_unstale":"2024-09-13T14:51:20.888678+0000","last_undegraded":"2024-09-13T14:51:20.888678+0000","last_fullsized":"2024-09-13T14:51:20.888678+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:42.577243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:42.577243+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:24.225386+0000","scrub_duration":0.001367839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"1180'9598","reported_seq":9636,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:38.280304+0000","last_change":"2024-09-13T14:49:22.118690+0000","last_active":"2024-09-13T14:51:38.280304+0000","last_peered":"2024-09-13T14:51:38.280304+0000","last_clean":"2024-09-13T14:51:38.280304+0000","last_became_active":"2024-09-13T14:49:22.117901+0000","last_became_peered":"2024-09-13T14:49:22.117901+0000","last_unstale":"2024-09-13T14:51:38.280304+0000","last_undegraded":"2024-09-13T14:51:38.280304+0000","last_fullsized":"2024-09-13T14:51:38.280304+0000","mapping_epoch":1167,"log_start":"1172'4326","ondisk_log_start":"1172'4326","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":5272,"log_dups_size":0,"ondisk_log_size":5272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:28.098913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413794304,"num_objects":6314,"num_object_clones":0,"num_object_copies":18942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9598,"num_write_kb":509184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1756,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.112484+0000","last_change":"2024-09-13T14:43:07.520212+0000","last_active":"2024-09-13T14:51:29.112484+0000","last_peered":"2024-09-13T14:51:29.112484+0000","last_clean":"2024-09-13T14:51:29.112484+0000","last_became_active":"2024-09-13T14:43:07.519831+0000","last_became_peered":"2024-09-13T14:43:07.519831+0000","last_unstale":"2024-09-13T14:51:29.112484+0000","last_undegraded":"2024-09-13T14:51:29.112484+0000","last_fullsized":"2024-09-13T14:51:29.112484+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:20.990736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1786,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.109996+0000","last_change":"2024-09-13T14:51:19.900868+0000","last_active":"2024-09-13T14:51:29.109996+0000","last_peered":"2024-09-13T14:51:29.109996+0000","last_clean":"2024-09-13T14:51:29.109996+0000","last_became_active":"2024-09-13T14:51:19.900481+0000","last_became_peered":"2024-09-13T14:51:19.900481+0000","last_unstale":"2024-09-13T14:51:29.109996+0000","last_undegraded":"2024-09-13T14:51:29.109996+0000","last_fullsized":"2024-09-13T14:51:29.109996+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:27.227954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1183,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.888367+0000","last_change":"2024-09-13T14:43:07.516140+0000","last_active":"2024-09-13T14:51:20.888367+0000","last_peered":"2024-09-13T14:51:20.888367+0000","last_clean":"2024-09-13T14:51:20.888367+0000","last_became_active":"2024-09-13T14:43:07.515481+0000","last_became_peered":"2024-09-13T14:43:07.515481+0000","last_unstale":"2024-09-13T14:51:20.888367+0000","last_undegraded":"2024-09-13T14:51:20.888367+0000","last_fullsized":"2024-09-13T14:51:20.888367+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:43.706214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"261'100","reported_seq":145,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.123089+0000","last_change":"2024-09-13T14:51:19.911422+0000","last_active":"2024-09-13T14:51:29.123089+0000","last_peered":"2024-09-13T14:51:29.123089+0000","last_clean":"2024-09-13T14:51:29.123089+0000","last_became_active":"2024-09-13T14:51:19.908591+0000","last_became_peered":"2024-09-13T14:51:19.908591+0000","last_unstale":"2024-09-13T14:51:29.123089+0000","last_undegraded":"2024-09-13T14:51:29.123089+0000","last_fullsized":"2024-09-13T14:51:29.123089+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.008264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.008264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:18:44.837688+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"1180'9294","reported_seq":9315,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:37.779429+0000","last_change":"2024-09-13T14:49:22.119570+0000","last_active":"2024-09-13T14:51:37.779429+0000","last_peered":"2024-09-13T14:51:37.779429+0000","last_clean":"2024-09-13T14:51:37.779429+0000","last_became_active":"2024-09-13T14:49:22.119224+0000","last_became_peered":"2024-09-13T14:49:22.119224+0000","last_unstale":"2024-09-13T14:51:37.779429+0000","last_undegraded":"2024-09-13T14:51:37.779429+0000","last_fullsized":"2024-09-13T14:51:37.779429+0000","mapping_epoch":1167,"log_start":"1172'2874","ondisk_log_start":"1172'2874","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":6420,"log_dups_size":0,"ondisk_log_size":6420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:43.554416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401997824,"num_objects":6134,"num_object_clones":0,"num_object_copies":18402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9294,"num_write_kb":493696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":20,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.889388+0000","last_change":"2024-09-13T14:51:19.906135+0000","last_active":"2024-09-13T14:51:20.889388+0000","last_peered":"2024-09-13T14:51:20.889388+0000","last_clean":"2024-09-13T14:51:20.889388+0000","last_became_active":"2024-09-13T14:51:19.904330+0000","last_became_peered":"2024-09-13T14:51:19.904330+0000","last_unstale":"2024-09-13T14:51:20.889388+0000","last_undegraded":"2024-09-13T14:51:20.889388+0000","last_fullsized":"2024-09-13T14:51:20.889388+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:53.407536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":77,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.123377+0000","last_change":"2024-09-13T14:45:19.810888+0000","last_active":"2024-09-13T14:51:29.123377+0000","last_peered":"2024-09-13T14:51:29.123377+0000","last_clean":"2024-09-13T14:51:29.123377+0000","last_became_active":"2024-09-13T14:45:19.810486+0000","last_became_peered":"2024-09-13T14:45:19.810486+0000","last_unstale":"2024-09-13T14:51:29.123377+0000","last_undegraded":"2024-09-13T14:51:29.123377+0000","last_fullsized":"2024-09-13T14:51:29.123377+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:03.886942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":1136,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.889485+0000","last_change":"2024-09-13T14:29:42.539455+0000","last_active":"2024-09-13T14:51:20.889485+0000","last_peered":"2024-09-13T14:51:20.889485+0000","last_clean":"2024-09-13T14:51:20.889485+0000","last_became_active":"2024-09-13T14:29:42.539068+0000","last_became_peered":"2024-09-13T14:29:42.539068+0000","last_unstale":"2024-09-13T14:51:20.889485+0000","last_undegraded":"2024-09-13T14:51:20.889485+0000","last_fullsized":"2024-09-13T14:51:20.889485+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:24:06.048855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034405999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"261'100","reported_seq":1291,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.888447+0000","last_change":"2024-09-13T14:44:28.186029+0000","last_active":"2024-09-13T14:51:20.888447+0000","last_peered":"2024-09-13T14:51:20.888447+0000","last_clean":"2024-09-13T14:51:20.888447+0000","last_became_active":"2024-09-13T14:42:14.927771+0000","last_became_peered":"2024-09-13T14:42:14.927771+0000","last_unstale":"2024-09-13T14:51:20.888447+0000","last_undegraded":"2024-09-13T14:51:20.888447+0000","last_fullsized":"2024-09-13T14:51:20.888447+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:28.185949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:28.185949+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:38.363961+0000","scrub_duration":0.0018071129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3","version":"1180'9343","reported_seq":9381,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:38.284856+0000","last_change":"2024-09-13T14:49:22.119776+0000","last_active":"2024-09-13T14:51:38.284856+0000","last_peered":"2024-09-13T14:51:38.284856+0000","last_clean":"2024-09-13T14:51:38.284856+0000","last_became_active":"2024-09-13T14:49:22.119517+0000","last_became_peered":"2024-09-13T14:49:22.119517+0000","last_unstale":"2024-09-13T14:51:38.284856+0000","last_undegraded":"2024-09-13T14:51:38.284856+0000","last_fullsized":"2024-09-13T14:51:38.284856+0000","mapping_epoch":1167,"log_start":"1172'4007","ondisk_log_start":"1172'4007","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":5336,"log_dups_size":0,"ondisk_log_size":5336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:53.152882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405733376,"num_objects":6191,"num_object_clones":0,"num_object_copies":18573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9343,"num_write_kb":497088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":132,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.120493+0000","last_change":"2024-09-13T14:51:19.907839+0000","last_active":"2024-09-13T14:51:29.120493+0000","last_peered":"2024-09-13T14:51:29.120493+0000","last_clean":"2024-09-13T14:51:29.120493+0000","last_became_active":"2024-09-13T14:51:19.907418+0000","last_became_peered":"2024-09-13T14:51:19.907418+0000","last_unstale":"2024-09-13T14:51:29.120493+0000","last_undegraded":"2024-09-13T14:51:29.120493+0000","last_fullsized":"2024-09-13T14:51:29.120493+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:43:28.088269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":76,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.123272+0000","last_change":"2024-09-13T14:45:19.812353+0000","last_active":"2024-09-13T14:51:29.123272+0000","last_peered":"2024-09-13T14:51:29.123272+0000","last_clean":"2024-09-13T14:51:29.123272+0000","last_became_active":"2024-09-13T14:45:19.811743+0000","last_became_peered":"2024-09-13T14:45:19.811743+0000","last_unstale":"2024-09-13T14:51:29.123272+0000","last_undegraded":"2024-09-13T14:51:29.123272+0000","last_fullsized":"2024-09-13T14:51:29.123272+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:43.728170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"261'100","reported_seq":120,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.889739+0000","last_change":"2024-09-13T14:44:34.814157+0000","last_active":"2024-09-13T14:51:20.889739+0000","last_peered":"2024-09-13T14:51:20.889739+0000","last_clean":"2024-09-13T14:51:20.889739+0000","last_became_active":"2024-09-13T14:42:42.707803+0000","last_became_peered":"2024-09-13T14:42:42.707803+0000","last_unstale":"2024-09-13T14:51:20.889739+0000","last_undegraded":"2024-09-13T14:51:20.889739+0000","last_fullsized":"2024-09-13T14:51:20.889739+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:34.814070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:34.814070+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:38.408561+0000","scrub_duration":0.01686414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"1180'9503","reported_seq":13657,"reported_epoch":1180,"state":"active+remapped+backfilling","last_fresh":"2024-09-13T14:51:39.929034+0000","last_change":"2024-09-13T14:51:21.017992+0000","last_active":"2024-09-13T14:51:39.929034+0000","last_peered":"2024-09-13T14:51:39.929034+0000","last_clean":"2024-09-13T14:51:19.033262+0000","last_became_active":"2024-09-13T14:51:20.993124+0000","last_became_peered":"2024-09-13T14:51:20.993124+0000","last_unstale":"2024-09-13T14:51:39.929034+0000","last_undegraded":"2024-09-13T14:51:39.929034+0000","last_fullsized":"2024-09-13T14:51:39.929034+0000","mapping_epoch":1178,"log_start":"1172'3586","ondisk_log_start":"1172'3586","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":5917,"log_dups_size":0,"ondisk_log_size":5917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399310848,"num_objects":6093,"num_object_clones":0,"num_object_copies":18279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4171,"num_objects_unfound":0,"num_objects_dirty":6093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9503,"num_write_kb":499072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2063,"num_bytes_recovered":135200768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,4],"avail_no_missing":["0","4","5"],"object_location_counts":[{"shards":"0,4,5","objects":6093}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1119,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.118016+0000","last_change":"2024-09-13T14:51:19.895343+0000","last_active":"2024-09-13T14:51:29.118016+0000","last_peered":"2024-09-13T14:51:29.118016+0000","last_clean":"2024-09-13T14:51:29.118016+0000","last_became_active":"2024-09-13T14:51:19.895036+0000","last_became_peered":"2024-09-13T14:51:19.895036+0000","last_unstale":"2024-09-13T14:51:29.118016+0000","last_undegraded":"2024-09-13T14:51:29.118016+0000","last_fullsized":"2024-09-13T14:51:29.118016+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:40.018071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":554,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.122511+0000","last_change":"2024-09-13T14:51:19.905106+0000","last_active":"2024-09-13T14:51:29.122511+0000","last_peered":"2024-09-13T14:51:29.122511+0000","last_clean":"2024-09-13T14:51:29.122511+0000","last_became_active":"2024-09-13T14:51:19.904682+0000","last_became_peered":"2024-09-13T14:51:19.904682+0000","last_unstale":"2024-09-13T14:51:29.122511+0000","last_undegraded":"2024-09-13T14:51:29.122511+0000","last_fullsized":"2024-09-13T14:51:29.122511+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:17:23.764784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"261'100","reported_seq":143,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.122781+0000","last_change":"2024-09-13T14:43:18.988482+0000","last_active":"2024-09-13T14:51:29.122781+0000","last_peered":"2024-09-13T14:51:29.122781+0000","last_clean":"2024-09-13T14:51:29.122781+0000","last_became_active":"2024-09-13T14:43:14.556754+0000","last_became_peered":"2024-09-13T14:43:14.556754+0000","last_unstale":"2024-09-13T14:51:29.122781+0000","last_undegraded":"2024-09-13T14:51:29.122781+0000","last_fullsized":"2024-09-13T14:51:29.122781+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:18.988405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:18.988405+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:45:01.726326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"1180'9415","reported_seq":9453,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:38.261455+0000","last_change":"2024-09-13T14:49:22.121230+0000","last_active":"2024-09-13T14:51:38.261455+0000","last_peered":"2024-09-13T14:51:38.261455+0000","last_clean":"2024-09-13T14:51:38.261455+0000","last_became_active":"2024-09-13T14:49:22.120836+0000","last_became_peered":"2024-09-13T14:49:22.120836+0000","last_unstale":"2024-09-13T14:51:38.261455+0000","last_undegraded":"2024-09-13T14:51:38.261455+0000","last_fullsized":"2024-09-13T14:51:38.261455+0000","mapping_epoch":1167,"log_start":"1172'4103","ondisk_log_start":"1172'4103","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":5312,"log_dups_size":0,"ondisk_log_size":5312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:21.843404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409010176,"num_objects":6241,"num_object_clones":0,"num_object_copies":18723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9415,"num_write_kb":500992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1120,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.116556+0000","last_change":"2024-09-13T14:51:19.898670+0000","last_active":"2024-09-13T14:51:29.116556+0000","last_peered":"2024-09-13T14:51:29.116556+0000","last_clean":"2024-09-13T14:51:29.116556+0000","last_became_active":"2024-09-13T14:51:19.898316+0000","last_became_peered":"2024-09-13T14:51:19.898316+0000","last_unstale":"2024-09-13T14:51:29.116556+0000","last_undegraded":"2024-09-13T14:51:29.116556+0000","last_fullsized":"2024-09-13T14:51:29.116556+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:53:27.567679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1094,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.118918+0000","last_change":"2024-09-13T14:51:19.909652+0000","last_active":"2024-09-13T14:51:29.118918+0000","last_peered":"2024-09-13T14:51:29.118918+0000","last_clean":"2024-09-13T14:51:29.118918+0000","last_became_active":"2024-09-13T14:51:19.907411+0000","last_became_peered":"2024-09-13T14:51:19.907411+0000","last_unstale":"2024-09-13T14:51:29.118918+0000","last_undegraded":"2024-09-13T14:51:29.118918+0000","last_fullsized":"2024-09-13T14:51:29.118918+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:14:38.272707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"261'100","reported_seq":125,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.889534+0000","last_change":"2024-09-13T14:45:19.805634+0000","last_active":"2024-09-13T14:51:20.889534+0000","last_peered":"2024-09-13T14:51:20.889534+0000","last_clean":"2024-09-13T14:51:20.889534+0000","last_became_active":"2024-09-13T14:45:19.805046+0000","last_became_peered":"2024-09-13T14:45:19.805046+0000","last_unstale":"2024-09-13T14:51:20.889534+0000","last_undegraded":"2024-09-13T14:51:20.889534+0000","last_fullsized":"2024-09-13T14:51:20.889534+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:32.752408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:32.752408+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:06:53.140239+0000","scrub_duration":0.0057731559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.0","version":"1180'9316","reported_seq":9354,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:39.782913+0000","last_change":"2024-09-13T14:49:22.118730+0000","last_active":"2024-09-13T14:51:39.782913+0000","last_peered":"2024-09-13T14:51:39.782913+0000","last_clean":"2024-09-13T14:51:39.782913+0000","last_became_active":"2024-09-13T14:49:22.118428+0000","last_became_peered":"2024-09-13T14:49:22.118428+0000","last_unstale":"2024-09-13T14:51:39.782913+0000","last_undegraded":"2024-09-13T14:51:39.782913+0000","last_fullsized":"2024-09-13T14:51:39.782913+0000","mapping_epoch":1167,"log_start":"1172'3982","ondisk_log_start":"1172'3982","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":5334,"log_dups_size":0,"ondisk_log_size":5334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:54.951267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398458880,"num_objects":6080,"num_object_clones":0,"num_object_copies":18240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9316,"num_write_kb":492672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1147,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.123043+0000","last_change":"2024-09-13T14:51:19.900208+0000","last_active":"2024-09-13T14:51:29.123043+0000","last_peered":"2024-09-13T14:51:29.123043+0000","last_clean":"2024-09-13T14:51:29.123043+0000","last_became_active":"2024-09-13T14:51:19.899741+0000","last_became_peered":"2024-09-13T14:51:19.899741+0000","last_unstale":"2024-09-13T14:51:29.123043+0000","last_undegraded":"2024-09-13T14:51:29.123043+0000","last_fullsized":"2024-09-13T14:51:29.123043+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:31.828483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":91,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.125403+0000","last_change":"2024-09-13T14:51:19.929437+0000","last_active":"2024-09-13T14:51:29.125403+0000","last_peered":"2024-09-13T14:51:29.125403+0000","last_clean":"2024-09-13T14:51:29.125403+0000","last_became_active":"2024-09-13T14:51:19.911220+0000","last_became_peered":"2024-09-13T14:51:19.911220+0000","last_unstale":"2024-09-13T14:51:29.125403+0000","last_undegraded":"2024-09-13T14:51:29.125403+0000","last_fullsized":"2024-09-13T14:51:29.125403+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:14.405956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"261'100","reported_seq":1310,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.887926+0000","last_change":"2024-09-13T14:45:19.798553+0000","last_active":"2024-09-13T14:51:20.887926+0000","last_peered":"2024-09-13T14:51:20.887926+0000","last_clean":"2024-09-13T14:51:20.887926+0000","last_became_active":"2024-09-13T14:45:19.798189+0000","last_became_peered":"2024-09-13T14:45:19.798189+0000","last_unstale":"2024-09-13T14:51:20.887926+0000","last_undegraded":"2024-09-13T14:51:20.887926+0000","last_fullsized":"2024-09-13T14:51:20.887926+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:29.169325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:29.169325+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:10.412666+0000","scrub_duration":0.0019358890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"1180'9309","reported_seq":9345,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:38.303280+0000","last_change":"2024-09-13T14:49:22.120585+0000","last_active":"2024-09-13T14:51:38.303280+0000","last_peered":"2024-09-13T14:51:38.303280+0000","last_clean":"2024-09-13T14:51:38.303280+0000","last_became_active":"2024-09-13T14:49:22.120301+0000","last_became_peered":"2024-09-13T14:49:22.120301+0000","last_unstale":"2024-09-13T14:51:38.303280+0000","last_undegraded":"2024-09-13T14:51:38.303280+0000","last_fullsized":"2024-09-13T14:51:38.303280+0000","mapping_epoch":1167,"log_start":"1172'4042","ondisk_log_start":"1172'4042","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":5267,"log_dups_size":0,"ondisk_log_size":5267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:20:25.183879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401801216,"num_objects":6131,"num_object_clones":0,"num_object_copies":18393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9309,"num_write_kb":494080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1752,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.108878+0000","last_change":"2024-09-13T14:51:19.905621+0000","last_active":"2024-09-13T14:51:29.108878+0000","last_peered":"2024-09-13T14:51:29.108878+0000","last_clean":"2024-09-13T14:51:29.108878+0000","last_became_active":"2024-09-13T14:51:19.905274+0000","last_became_peered":"2024-09-13T14:51:19.905274+0000","last_unstale":"2024-09-13T14:51:29.108878+0000","last_undegraded":"2024-09-13T14:51:29.108878+0000","last_fullsized":"2024-09-13T14:51:29.108878+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:57.364606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1163,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.906135+0000","last_change":"2024-09-13T14:45:19.804982+0000","last_active":"2024-09-13T14:51:20.906135+0000","last_peered":"2024-09-13T14:51:20.906135+0000","last_clean":"2024-09-13T14:51:20.906135+0000","last_became_active":"2024-09-13T14:45:19.804663+0000","last_became_peered":"2024-09-13T14:45:19.804663+0000","last_unstale":"2024-09-13T14:51:20.906135+0000","last_undegraded":"2024-09-13T14:51:20.906135+0000","last_fullsized":"2024-09-13T14:51:20.906135+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:43.211479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"261'100","reported_seq":342,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.890279+0000","last_change":"2024-09-13T14:51:19.895244+0000","last_active":"2024-09-13T14:51:20.890279+0000","last_peered":"2024-09-13T14:51:20.890279+0000","last_clean":"2024-09-13T14:51:20.890279+0000","last_became_active":"2024-09-13T14:51:19.894667+0000","last_became_peered":"2024-09-13T14:51:19.894667+0000","last_unstale":"2024-09-13T14:51:20.890279+0000","last_undegraded":"2024-09-13T14:51:20.890279+0000","last_fullsized":"2024-09-13T14:51:20.890279+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:42.881520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:42.881520+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13417,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":17,"num_write":3,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":13417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.6","version":"1180'9236","reported_seq":9274,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:39.902724+0000","last_change":"2024-09-13T14:49:22.117570+0000","last_active":"2024-09-13T14:51:39.902724+0000","last_peered":"2024-09-13T14:51:39.902724+0000","last_clean":"2024-09-13T14:51:39.902724+0000","last_became_active":"2024-09-13T14:49:22.117259+0000","last_became_peered":"2024-09-13T14:49:22.117259+0000","last_unstale":"2024-09-13T14:51:39.902724+0000","last_undegraded":"2024-09-13T14:51:39.902724+0000","last_fullsized":"2024-09-13T14:51:39.902724+0000","mapping_epoch":1167,"log_start":"1172'3270","ondisk_log_start":"1172'3270","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":5966,"log_dups_size":0,"ondisk_log_size":5966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:59:23.669062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392560640,"num_objects":5990,"num_object_clones":0,"num_object_copies":17970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9236,"num_write_kb":487232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1136,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.890814+0000","last_change":"2024-09-13T14:43:07.518120+0000","last_active":"2024-09-13T14:51:20.890814+0000","last_peered":"2024-09-13T14:51:20.890814+0000","last_clean":"2024-09-13T14:51:20.890814+0000","last_became_active":"2024-09-13T14:43:07.517361+0000","last_became_peered":"2024-09-13T14:43:07.517361+0000","last_unstale":"2024-09-13T14:51:20.890814+0000","last_undegraded":"2024-09-13T14:51:20.890814+0000","last_fullsized":"2024-09-13T14:51:20.890814+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:39.366473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":1118,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.889465+0000","last_change":"2024-09-13T14:19:25.871006+0000","last_active":"2024-09-13T14:51:20.889465+0000","last_peered":"2024-09-13T14:51:20.889465+0000","last_clean":"2024-09-13T14:51:20.889465+0000","last_became_active":"2024-09-13T14:19:25.870713+0000","last_became_peered":"2024-09-13T14:19:25.870713+0000","last_unstale":"2024-09-13T14:51:20.889465+0000","last_undegraded":"2024-09-13T14:51:20.889465+0000","last_fullsized":"2024-09-13T14:51:20.889465+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:34.372124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"261'100","reported_seq":404,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.889346+0000","last_change":"2024-09-13T14:51:19.899481+0000","last_active":"2024-09-13T14:51:20.889346+0000","last_peered":"2024-09-13T14:51:20.889346+0000","last_clean":"2024-09-13T14:51:20.889346+0000","last_became_active":"2024-09-13T14:51:19.898919+0000","last_became_peered":"2024-09-13T14:51:19.898919+0000","last_unstale":"2024-09-13T14:51:20.889346+0000","last_undegraded":"2024-09-13T14:51:20.889346+0000","last_fullsized":"2024-09-13T14:51:20.889346+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:49:21.722939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.722939+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:22.739555+0000","scrub_duration":0.0016642790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.5","version":"1180'9527","reported_seq":13380,"reported_epoch":1180,"state":"active+remapped+backfilling","last_fresh":"2024-09-13T14:51:37.834725+0000","last_change":"2024-09-13T14:51:21.060776+0000","last_active":"2024-09-13T14:51:37.834725+0000","last_peered":"2024-09-13T14:51:37.834725+0000","last_clean":"2024-09-13T14:51:19.009276+0000","last_became_active":"2024-09-13T14:51:20.994052+0000","last_became_peered":"2024-09-13T14:51:20.994052+0000","last_unstale":"2024-09-13T14:51:37.834725+0000","last_undegraded":"2024-09-13T14:51:37.834725+0000","last_fullsized":"2024-09-13T14:51:37.834725+0000","mapping_epoch":1178,"log_start":"1172'3842","ondisk_log_start":"1172'3842","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":5685,"log_dups_size":0,"ondisk_log_size":5685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:10:42.329625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407044096,"num_objects":6211,"num_object_clones":0,"num_object_copies":18633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4439,"num_objects_unfound":0,"num_objects_dirty":6211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9527,"num_write_kb":503616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1908,"num_bytes_recovered":124977152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,4],"avail_no_missing":["6","4","7"],"object_location_counts":[{"shards":"4,6,7","objects":6211}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1164,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.122323+0000","last_change":"2024-09-13T14:51:19.900788+0000","last_active":"2024-09-13T14:51:29.122323+0000","last_peered":"2024-09-13T14:51:29.122323+0000","last_clean":"2024-09-13T14:51:29.122323+0000","last_became_active":"2024-09-13T14:51:19.900220+0000","last_became_peered":"2024-09-13T14:51:19.900220+0000","last_unstale":"2024-09-13T14:51:29.122323+0000","last_undegraded":"2024-09-13T14:51:29.122323+0000","last_fullsized":"2024-09-13T14:51:29.122323+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:38.189771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":1178,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.890641+0000","last_change":"2024-09-13T14:51:19.944221+0000","last_active":"2024-09-13T14:51:20.890641+0000","last_peered":"2024-09-13T14:51:20.890641+0000","last_clean":"2024-09-13T14:51:20.890641+0000","last_became_active":"2024-09-13T14:51:19.897813+0000","last_became_peered":"2024-09-13T14:51:19.897813+0000","last_unstale":"2024-09-13T14:51:20.890641+0000","last_undegraded":"2024-09-13T14:51:20.890641+0000","last_fullsized":"2024-09-13T14:51:20.890641+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:46:40.882318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"261'100","reported_seq":80,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.123182+0000","last_change":"2024-09-13T14:45:19.821649+0000","last_active":"2024-09-13T14:51:29.123182+0000","last_peered":"2024-09-13T14:51:29.123182+0000","last_clean":"2024-09-13T14:51:29.123182+0000","last_became_active":"2024-09-13T14:45:19.812346+0000","last_became_peered":"2024-09-13T14:45:19.812346+0000","last_unstale":"2024-09-13T14:51:29.123182+0000","last_undegraded":"2024-09-13T14:51:29.123182+0000","last_fullsized":"2024-09-13T14:51:29.123182+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.597568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.597568+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:45.218892+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"1180'9302","reported_seq":18039,"reported_epoch":1180,"state":"active+remapped+backfilling","last_fresh":"2024-09-13T14:51:37.826287+0000","last_change":"2024-09-13T14:51:21.031405+0000","last_active":"2024-09-13T14:51:37.826287+0000","last_peered":"2024-09-13T14:51:37.826287+0000","last_clean":"2024-09-13T14:51:19.037423+0000","last_became_active":"2024-09-13T14:51:20.992442+0000","last_became_peered":"2024-09-13T14:51:20.992442+0000","last_unstale":"2024-09-13T14:51:37.826287+0000","last_undegraded":"2024-09-13T14:51:37.826287+0000","last_fullsized":"2024-09-13T14:51:37.826287+0000","mapping_epoch":1178,"log_start":"1172'2840","ondisk_log_start":"1172'2840","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":6462,"log_dups_size":0,"ondisk_log_size":6462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:43:29.835909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399245312,"num_objects":6092,"num_object_clones":0,"num_object_copies":18276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2030,"num_objects_unfound":0,"num_objects_dirty":6092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9302,"num_write_kb":492608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4352,"num_bytes_recovered":285212672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,4],"avail_no_missing":["5","2","4"],"object_location_counts":[{"shards":"2,4,5","objects":6092}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1105,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.889778+0000","last_change":"2024-09-13T14:19:26.887839+0000","last_active":"2024-09-13T14:51:20.889778+0000","last_peered":"2024-09-13T14:51:20.889778+0000","last_clean":"2024-09-13T14:51:20.889778+0000","last_became_active":"2024-09-13T14:19:26.887498+0000","last_became_peered":"2024-09-13T14:19:26.887498+0000","last_unstale":"2024-09-13T14:51:20.889778+0000","last_undegraded":"2024-09-13T14:51:20.889778+0000","last_fullsized":"2024-09-13T14:51:20.889778+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:24.475263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":1149,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.889335+0000","last_change":"2024-09-13T14:51:19.948447+0000","last_active":"2024-09-13T14:51:20.889335+0000","last_peered":"2024-09-13T14:51:20.889335+0000","last_clean":"2024-09-13T14:51:20.889335+0000","last_became_active":"2024-09-13T14:51:19.908698+0000","last_became_peered":"2024-09-13T14:51:19.908698+0000","last_unstale":"2024-09-13T14:51:20.889335+0000","last_undegraded":"2024-09-13T14:51:20.889335+0000","last_fullsized":"2024-09-13T14:51:20.889335+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:02:56.829843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"261'100","reported_seq":81,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.889180+0000","last_change":"2024-09-13T14:51:19.899763+0000","last_active":"2024-09-13T14:51:20.889180+0000","last_peered":"2024-09-13T14:51:20.889180+0000","last_clean":"2024-09-13T14:51:20.889180+0000","last_became_active":"2024-09-13T14:51:19.899012+0000","last_became_peered":"2024-09-13T14:51:19.899012+0000","last_unstale":"2024-09-13T14:51:20.889180+0000","last_undegraded":"2024-09-13T14:51:20.889180+0000","last_fullsized":"2024-09-13T14:51:20.889180+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.191326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.191326+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:09.732098+0000","scrub_duration":0.0025317360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1125,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.890004+0000","last_change":"2024-09-13T14:28:35.603724+0000","last_active":"2024-09-13T14:51:20.890004+0000","last_peered":"2024-09-13T14:51:20.890004+0000","last_clean":"2024-09-13T14:51:20.890004+0000","last_became_active":"2024-09-13T14:28:35.603316+0000","last_became_peered":"2024-09-13T14:28:35.603316+0000","last_unstale":"2024-09-13T14:51:20.890004+0000","last_undegraded":"2024-09-13T14:51:20.890004+0000","last_fullsized":"2024-09-13T14:51:20.890004+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:47.668223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":76,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.126329+0000","last_change":"2024-09-13T14:45:19.811466+0000","last_active":"2024-09-13T14:51:29.126329+0000","last_peered":"2024-09-13T14:51:29.126329+0000","last_clean":"2024-09-13T14:51:29.126329+0000","last_became_active":"2024-09-13T14:45:19.810617+0000","last_became_peered":"2024-09-13T14:45:19.810617+0000","last_unstale":"2024-09-13T14:51:29.126329+0000","last_undegraded":"2024-09-13T14:51:29.126329+0000","last_fullsized":"2024-09-13T14:51:29.126329+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:16.246669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"261'100","reported_seq":71,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.890870+0000","last_change":"2024-09-13T14:44:50.805306+0000","last_active":"2024-09-13T14:51:20.890870+0000","last_peered":"2024-09-13T14:51:20.890870+0000","last_clean":"2024-09-13T14:51:20.890870+0000","last_became_active":"2024-09-13T14:43:14.562922+0000","last_became_peered":"2024-09-13T14:43:14.562922+0000","last_unstale":"2024-09-13T14:51:20.890870+0000","last_undegraded":"2024-09-13T14:51:20.890870+0000","last_fullsized":"2024-09-13T14:51:20.890870+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:50.805210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:50.805210+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:28:38.529599+0000","scrub_duration":0.0042900009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1127,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.888577+0000","last_change":"2024-09-13T14:51:19.892512+0000","last_active":"2024-09-13T14:51:20.888577+0000","last_peered":"2024-09-13T14:51:20.888577+0000","last_clean":"2024-09-13T14:51:20.888577+0000","last_became_active":"2024-09-13T14:51:19.892215+0000","last_became_peered":"2024-09-13T14:51:19.892215+0000","last_unstale":"2024-09-13T14:51:20.888577+0000","last_undegraded":"2024-09-13T14:51:20.888577+0000","last_fullsized":"2024-09-13T14:51:20.888577+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:51.369749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1744,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.112688+0000","last_change":"2024-09-13T14:45:19.813876+0000","last_active":"2024-09-13T14:51:29.112688+0000","last_peered":"2024-09-13T14:51:29.112688+0000","last_clean":"2024-09-13T14:51:29.112688+0000","last_became_active":"2024-09-13T14:45:19.809136+0000","last_became_peered":"2024-09-13T14:45:19.809136+0000","last_unstale":"2024-09-13T14:51:29.112688+0000","last_undegraded":"2024-09-13T14:51:29.112688+0000","last_fullsized":"2024-09-13T14:51:29.112688+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:50.013528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"261'100","reported_seq":157,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.122942+0000","last_change":"2024-09-13T14:51:19.912342+0000","last_active":"2024-09-13T14:51:29.122942+0000","last_peered":"2024-09-13T14:51:29.122942+0000","last_clean":"2024-09-13T14:51:29.122942+0000","last_became_active":"2024-09-13T14:51:19.909735+0000","last_became_peered":"2024-09-13T14:51:19.909735+0000","last_unstale":"2024-09-13T14:51:29.122942+0000","last_undegraded":"2024-09-13T14:51:29.122942+0000","last_fullsized":"2024-09-13T14:51:29.122942+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:19.978464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:19.978464+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:14.417715+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1791,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.108966+0000","last_change":"2024-09-13T14:45:19.809166+0000","last_active":"2024-09-13T14:51:29.108966+0000","last_peered":"2024-09-13T14:51:29.108966+0000","last_clean":"2024-09-13T14:51:29.108966+0000","last_became_active":"2024-09-13T14:45:19.808631+0000","last_became_peered":"2024-09-13T14:45:19.808631+0000","last_unstale":"2024-09-13T14:51:29.108966+0000","last_undegraded":"2024-09-13T14:51:29.108966+0000","last_fullsized":"2024-09-13T14:51:29.108966+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:38.822245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1142,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.890442+0000","last_change":"2024-09-13T14:45:19.804229+0000","last_active":"2024-09-13T14:51:20.890442+0000","last_peered":"2024-09-13T14:51:20.890442+0000","last_clean":"2024-09-13T14:51:20.890442+0000","last_became_active":"2024-09-13T14:45:19.803929+0000","last_became_peered":"2024-09-13T14:45:19.803929+0000","last_unstale":"2024-09-13T14:51:20.890442+0000","last_undegraded":"2024-09-13T14:51:20.890442+0000","last_fullsized":"2024-09-13T14:51:20.890442+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:33:53.675568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"261'100","reported_seq":71,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.889336+0000","last_change":"2024-09-13T14:44:42.654506+0000","last_active":"2024-09-13T14:51:20.889336+0000","last_peered":"2024-09-13T14:51:20.889336+0000","last_clean":"2024-09-13T14:51:20.889336+0000","last_became_active":"2024-09-13T14:43:14.563451+0000","last_became_peered":"2024-09-13T14:43:14.563451+0000","last_unstale":"2024-09-13T14:51:20.889336+0000","last_undegraded":"2024-09-13T14:51:20.889336+0000","last_fullsized":"2024-09-13T14:51:20.889336+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:42.654455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:42.654455+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:57:32.251597+0000","scrub_duration":0.001494097,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":1152,"reported_epoch":1179,"state":"active+clean","last_fresh":"2024-09-13T14:51:20.888881+0000","last_change":"2024-09-13T14:51:19.902458+0000","last_active":"2024-09-13T14:51:20.888881+0000","last_peered":"2024-09-13T14:51:20.888881+0000","last_clean":"2024-09-13T14:51:20.888881+0000","last_became_active":"2024-09-13T14:51:19.892094+0000","last_became_peered":"2024-09-13T14:51:19.892094+0000","last_unstale":"2024-09-13T14:51:20.888881+0000","last_undegraded":"2024-09-13T14:51:20.888881+0000","last_fullsized":"2024-09-13T14:51:20.888881+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:03:35.273889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1683,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.123417+0000","last_change":"2024-09-13T14:43:07.518679+0000","last_active":"2024-09-13T14:51:29.123417+0000","last_peered":"2024-09-13T14:51:29.123417+0000","last_clean":"2024-09-13T14:51:29.123417+0000","last_became_active":"2024-09-13T14:43:07.518333+0000","last_became_peered":"2024-09-13T14:43:07.518333+0000","last_unstale":"2024-09-13T14:51:29.123417+0000","last_undegraded":"2024-09-13T14:51:29.123417+0000","last_fullsized":"2024-09-13T14:51:29.123417+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:20.616274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"261'100","reported_seq":452,"reported_epoch":1180,"state":"active+clean","last_fresh":"2024-09-13T14:51:29.110039+0000","last_change":"2024-09-13T14:45:19.809268+0000","last_active":"2024-09-13T14:51:29.110039+0000","last_peered":"2024-09-13T14:51:29.110039+0000","last_clean":"2024-09-13T14:51:29.110039+0000","last_became_active":"2024-09-13T14:45:19.808772+0000","last_became_peered":"2024-09-13T14:45:19.808772+0000","last_unstale":"2024-09-13T14:51:29.110039+0000","last_undegraded":"2024-09-13T14:51:29.110039+0000","last_fullsized":"2024-09-13T14:51:29.110039+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:14.654603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:14.654603+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:40.964553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":6457983000,"num_objects":98542,"num_object_clones":0,"num_object_copies":295626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":19394,"num_objects_unfound":0,"num_objects_dirty":98542,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151190,"num_write_kb":7991361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12078,"num_bytes_recovered":791412736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20176908288,"data_stored":20176896072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92259,"ondisk_log_size":92259,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":4890,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":21480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":630620,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":51,"num_write":10,"num_write_kb":111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":367932,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430619,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2692268,"kb_used_data":2098212,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":91052884,"statfs":{"total":95995035648,"available":93238153216,"internally_reserved":0,"allocated":2148569088,"data_stored":2144901981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626848,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2998176,"kb_used_data":2408280,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":90746976,"statfs":{"total":95995035648,"available":92924903424,"internally_reserved":0,"allocated":2466078720,"data_stored":2462423248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823077,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3783004,"kb_used_data":3197268,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":89962148,"statfs":{"total":95995035648,"available":92121239552,"internally_reserved":0,"allocated":3274002432,"data_stored":3270347408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019305,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059592963,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2814732,"kb_used_data":2484804,"kb_used_omap":26,"kb_used_meta":329893,"kb_avail":90930420,"statfs":{"total":95995035648,"available":93112750080,"internally_reserved":0,"allocated":2544439296,"data_stored":2540799539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":337810592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":591,"seq":2538325672201,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3067236,"kb_used_data":2533404,"kb_used_omap":41,"kb_used_meta":533782,"kb_avail":90677916,"statfs":{"total":95995035648,"available":92854185984,"internally_reserved":0,"allocated":2594205696,"data_stored":2590547197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42651,"internal_metadata":546593125},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":557,"seq":2392296784144,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3330896,"kb_used_data":2796872,"kb_used_omap":32,"kb_used_meta":533791,"kb_avail":90414256,"statfs":{"total":95995035648,"available":92584198144,"internally_reserved":0,"allocated":2863996928,"data_stored":2860353042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33752,"internal_metadata":546602024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,4,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":524,"seq":2250562863383,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2901268,"kb_used_data":2485388,"kb_used_omap":41,"kb_used_meta":415830,"kb_avail":90843884,"statfs":{"total":95995035648,"available":93024137216,"internally_reserved":0,"allocated":2545037312,"data_stored":2541389979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42694,"internal_metadata":425810234},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2531590144,"data_stored":2531590144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2844590080,"data_stored":2844590080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2572619776,"data_stored":2572615704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2529165312,"data_stored":2529165312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2029912064,"data_stored":2029912064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3254128640,"data_stored":3254124568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2284978176,"data_stored":2284978176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2129924096,"data_stored":2129920024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:51:45.491 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-13T14:51:45.491 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 61 2024-09-13T14:51:45.514 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:45.537 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:45.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:45.639 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:45.662 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:45.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:45.745 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:51:45.764 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:45.792 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:45.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:45.894 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:45.916 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:45.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:46.016 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:51:46.017 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:51:46.018 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:46.041 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:51:46.042 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:51:46.054 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:46.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:46.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:46.057 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:46.068 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:51:46.068 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:51:46.098 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:51:46.098 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:51:46.124 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:51:46.124 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:51:46.146 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:51:46.146 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:51:46.156 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:46.166 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:51:46.168 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:51:46.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:46.184 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:46.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:46.194 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:51:46.195 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:51:46.286 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:46.325 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:46.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:46.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:46.335 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:46 smithi151 ceph-mon[104450]: osdmap e1181: 8 total, 8 up, 7 in 2024-09-13T14:51:46.335 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:46 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3912061548' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:46.335 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:46 smithi151 ceph-mon[108966]: osdmap e1181: 8 total, 8 up, 7 in 2024-09-13T14:51:46.335 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:46 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3912061548' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:46.427 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:46.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:46.457 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:46.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:46.558 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:46.594 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:46.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:46.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:46.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:46 smithi184 ceph-mon[92527]: osdmap e1181: 8 total, 8 up, 7 in 2024-09-13T14:51:46.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:46 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3912061548' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:46.696 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:46.725 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:46.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:46.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:46.827 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:46.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:46.858 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:46.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:46.959 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:47.014 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:47.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:47.117 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:47.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.149 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:47.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:47.249 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:47 smithi184 ceph-mon[92527]: pgmap v1757: 119 pgs: 5 active+remapped+backfilling, 114 active+clean; 5.7 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 841 op/s; 15166/282483 objects misplaced (5.369%); 43 MiB/s, 693 objects/s recovering 2024-09-13T14:51:47.249 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:47 smithi184 ceph-mon[92527]: osdmap e1182: 8 total, 8 up, 7 in 2024-09-13T14:51:47.250 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:47 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:47.250 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:47.291 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:47.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:47.393 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:47.425 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:47.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:47.527 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:47.556 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:47.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:47.598 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:47 smithi151 ceph-mon[108966]: pgmap v1757: 119 pgs: 5 active+remapped+backfilling, 114 active+clean; 5.7 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 841 op/s; 15166/282483 objects misplaced (5.369%); 43 MiB/s, 693 objects/s recovering 2024-09-13T14:51:47.598 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:47 smithi151 ceph-mon[108966]: osdmap e1182: 8 total, 8 up, 7 in 2024-09-13T14:51:47.598 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:47 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:47.598 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:47 smithi151 ceph-mon[104450]: pgmap v1757: 119 pgs: 5 active+remapped+backfilling, 114 active+clean; 5.7 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 841 op/s; 15166/282483 objects misplaced (5.369%); 43 MiB/s, 693 objects/s recovering 2024-09-13T14:51:47.598 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:47 smithi151 ceph-mon[104450]: osdmap e1182: 8 total, 8 up, 7 in 2024-09-13T14:51:47.598 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:47 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:47.658 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:47.695 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:47.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:47.797 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:47.841 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:47.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:47.849 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:51:47.849 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:51:47.943 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:47.977 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:47.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:48.079 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:48.118 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:48.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:48.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:48.220 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:48.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:48.256 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:48.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:48.357 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:48.396 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:48.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:48.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:48.498 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:48.546 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T14:51:48.550 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:48.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:48.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:48.652 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:48.703 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:48.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:48.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:48.805 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:48.841 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:48.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:48.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:48.943 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:48.971 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:51:48.994 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:48.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:48.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:49.096 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:49.130 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:49.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:49.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:49.232 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:49.269 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:49.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:49.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:49.371 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:49.378 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:49.410 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:49.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:49.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:49.449 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:49 smithi184 ceph-mon[92527]: pgmap v1759: 119 pgs: 5 active+remapped+backfilling, 114 active+clean; 5.7 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 759 op/s; 15166/282483 objects misplaced (5.369%); 40 MiB/s, 637 objects/s recovering 2024-09-13T14:51:49.479 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:49 smithi151 ceph-mon[108966]: pgmap v1759: 119 pgs: 5 active+remapped+backfilling, 114 active+clean; 5.7 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 759 op/s; 15166/282483 objects misplaced (5.369%); 40 MiB/s, 637 objects/s recovering 2024-09-13T14:51:49.479 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:49 smithi151 ceph-mon[104450]: pgmap v1759: 119 pgs: 5 active+remapped+backfilling, 114 active+clean; 5.7 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 759 op/s; 15166/282483 objects misplaced (5.369%); 40 MiB/s, 637 objects/s recovering 2024-09-13T14:51:49.512 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:49.554 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:49.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:49.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:49.656 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:49.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:49.687 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:49.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:49.788 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:49.824 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:49.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:49.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:49.926 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:49.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:49.961 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:49.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:50.062 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:50.094 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:50.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:50.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:50.197 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:50.234 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:50.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:50.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:50.336 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:50.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:50.369 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:50.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:50.470 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:50.476 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:51:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:51:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:51:50.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:50.511 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:50.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:50.612 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:50.639 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:50.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:50.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:50.741 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:50.782 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:50.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:50.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:50.884 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:50.917 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:50.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:50.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:51.020 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:51.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:51.064 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:51.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:51.145 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:51:51.145 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:51:51.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:51.206 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:51.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:51.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:51.308 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:51.333 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:51:51.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:51.334 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:51.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:51.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:51 smithi184 ceph-mon[92527]: pgmap v1760: 119 pgs: 4 active+remapped+backfilling, 115 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 845 op/s; 13448/275415 objects misplaced (4.883%); 39 MiB/s, 622 objects/s recovering 2024-09-13T14:51:51.435 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:51.461 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: specified pg_num 61 < pg_num_max 32 2024-09-13T14:51:51.461 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:51 smithi151 ceph-mon[104450]: pgmap v1760: 119 pgs: 4 active+remapped+backfilling, 115 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 845 op/s; 13448/275415 objects misplaced (4.883%); 39 MiB/s, 622 objects/s recovering 2024-09-13T14:51:51.461 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:51 smithi151 ceph-mon[108966]: pgmap v1760: 119 pgs: 4 active+remapped+backfilling, 115 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 845 op/s; 13448/275415 objects misplaced (4.883%); 39 MiB/s, 622 objects/s recovering 2024-09-13T14:51:51.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:51.462 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:51.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:51.563 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:51.625 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:51.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:51.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:51.727 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:51.773 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:51.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:51.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:51.833 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T14:51:51.875 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:51.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:51.924 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:51.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:52.025 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:52.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:52.059 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:52.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:52.160 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:52.208 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:51:52.209 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:52.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:52.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:52.311 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:52.349 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:52.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:52.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:52.452 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:52.486 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:52.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:52.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:52.502 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:52 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1012836673' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "61"} : dispatch 2024-09-13T14:51:52.502 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:52 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "61"} : dispatch 2024-09-13T14:51:52.589 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:52.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:52 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1012836673' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "61"} : dispatch 2024-09-13T14:51:52.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:52 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "61"} : dispatch 2024-09-13T14:51:52.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:52 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1012836673' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "61"} : dispatch 2024-09-13T14:51:52.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:52 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "61"} : dispatch 2024-09-13T14:51:52.616 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:52.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:52.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:52.718 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:52.753 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:52.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:52.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:52.768 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T14:51:52.855 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:52.891 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:52.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:52.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:52.993 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:53.028 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:53.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:53.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:53.130 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:53.177 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:53.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:53.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:53.279 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:53.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:53.310 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:53.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:53.335 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:53 smithi151 ceph-mon[104450]: pgmap v1761: 119 pgs: 4 active+remapped+backfilling, 115 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 765 op/s; 10666/270147 objects misplaced (3.948%); 42 MiB/s, 668 objects/s recovering 2024-09-13T14:51:53.335 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:53 smithi151 ceph-mon[108966]: pgmap v1761: 119 pgs: 4 active+remapped+backfilling, 115 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 765 op/s; 10666/270147 objects misplaced (3.948%); 42 MiB/s, 668 objects/s recovering 2024-09-13T14:51:53.346 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:53.410 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:53.455 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:53.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:53.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:53.512 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:53 smithi184 ceph-mon[92527]: pgmap v1761: 119 pgs: 4 active+remapped+backfilling, 115 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 765 op/s; 10666/270147 objects misplaced (3.948%); 42 MiB/s, 668 objects/s recovering 2024-09-13T14:51:53.558 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:53.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:53.595 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:53.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:53.696 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:53.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:53.735 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:53.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:53.836 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:53.874 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:53.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:53.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:53.977 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:54.015 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:54.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:54.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:54.117 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:54.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:54.152 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:54.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:54.179 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:51:54.179 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:51:54.230 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-13T14:51:54.230 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:54 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3438191538' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:54.230 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:54 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3438191538' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:54.253 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:54.301 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:54.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:54.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:54.403 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:54.442 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:54.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:54.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:54.513 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:54 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3438191538' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:54.544 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:54.585 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:54.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:54.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:54.687 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:54.723 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:54.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:54.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:54.825 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:54.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:51:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:51:54] "GET /metrics HTTP/1.1" 200 36679 "" "Prometheus/2.51.0" 2024-09-13T14:51:54.862 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:54.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:54.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:54.870 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T14:51:54.964 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:55.005 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:55.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:55.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:55.108 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:55.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:55.142 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:55.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:55.243 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:55.262 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:51:55.284 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:55.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:55.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:55.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:55.421 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:55.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:55.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:55.496 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:55 smithi151 ceph-mon[104450]: pgmap v1762: 119 pgs: 4 active+remapped+backfilling, 115 active+clean; 5.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 805 op/s; 9926/262128 objects misplaced (3.787%); 34 MiB/s, 543 objects/s recovering 2024-09-13T14:51:55.496 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:55 smithi151 ceph-mon[104450]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:51:55.496 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:55 smithi151 ceph-mon[104450]: Cluster is now healthy 2024-09-13T14:51:55.496 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:55 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3438191538' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:51:55.497 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:55 smithi151 ceph-mon[104450]: osdmap e1183: 8 total, 8 up, 7 in 2024-09-13T14:51:55.497 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:55 smithi151 ceph-mon[108966]: pgmap v1762: 119 pgs: 4 active+remapped+backfilling, 115 active+clean; 5.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 805 op/s; 9926/262128 objects misplaced (3.787%); 34 MiB/s, 543 objects/s recovering 2024-09-13T14:51:55.497 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:55 smithi151 ceph-mon[108966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:51:55.497 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:55 smithi151 ceph-mon[108966]: Cluster is now healthy 2024-09-13T14:51:55.497 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:55 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3438191538' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:51:55.497 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:55 smithi151 ceph-mon[108966]: osdmap e1183: 8 total, 8 up, 7 in 2024-09-13T14:51:55.523 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:55.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:55.561 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:55.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:55.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:55 smithi184 ceph-mon[92527]: pgmap v1762: 119 pgs: 4 active+remapped+backfilling, 115 active+clean; 5.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 805 op/s; 9926/262128 objects misplaced (3.787%); 34 MiB/s, 543 objects/s recovering 2024-09-13T14:51:55.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:55 smithi184 ceph-mon[92527]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:51:55.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:55 smithi184 ceph-mon[92527]: Cluster is now healthy 2024-09-13T14:51:55.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:55 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3438191538' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:51:55.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:55 smithi184 ceph-mon[92527]: osdmap e1183: 8 total, 8 up, 7 in 2024-09-13T14:51:55.662 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:55.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:55.707 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:55.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:55.809 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:55.834 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:55.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:55.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:55.937 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:55.975 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:55.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:55.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:56.077 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:56.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:56.115 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:56.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:56.217 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:56.263 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:56.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:56.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:56.365 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:56.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:56.402 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:56.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:56.503 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:56.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:56.540 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:56.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:56.641 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:56.674 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:56.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:56.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:56.776 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:56.814 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:56.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:56.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:56.916 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:56.952 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:56.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:56.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:57.053 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:57.075 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:57.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:57.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:57.177 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:57.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:57.207 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:57.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:57.308 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:57.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:57.347 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:57.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:57.446 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:57 smithi184 ceph-mon[92527]: pgmap v1764: 119 pgs: 4 active+remapped+backfilling, 115 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 943 op/s; 8113/255828 objects misplaced (3.171%); 40 MiB/s, 634 objects/s recovering 2024-09-13T14:51:57.448 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:57.478 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:51:57.478 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:51:57.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:57.486 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:57.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:57.587 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:57.611 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:57 smithi151 ceph-mon[104450]: pgmap v1764: 119 pgs: 4 active+remapped+backfilling, 115 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 943 op/s; 8113/255828 objects misplaced (3.171%); 40 MiB/s, 634 objects/s recovering 2024-09-13T14:51:57.612 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:57 smithi151 ceph-mon[108966]: pgmap v1764: 119 pgs: 4 active+remapped+backfilling, 115 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 943 op/s; 8113/255828 objects misplaced (3.171%); 40 MiB/s, 634 objects/s recovering 2024-09-13T14:51:57.617 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:57.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:57.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:57.719 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:57.749 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:57.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:57.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:57.769 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 6, 7, 1] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:51:57.770 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:51:57.770 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_3 2024-09-13T14:51:57.770 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:51:57.851 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:57.888 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:57.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:57.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:57.989 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:58.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:58.042 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:58.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:58.145 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:58.177 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:58.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:58.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:58.268 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T14:51:58.279 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:58.343 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:58.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:58.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:58.445 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:58.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:58.487 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:58.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:58.588 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:58.637 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:51:58.640 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:58.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:58.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:58.742 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:58.774 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:58.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:58.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:58.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:58.931 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:58.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:58.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:59.033 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:59.081 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:59.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:59.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:59.183 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:59.209 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:51:59.209 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:51:59.209 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:51:59.209 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:51:59.209 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:51:59.209 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:51:59.210 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:51:59.210 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:51:59.210 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:51:59.210 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:51:59.210 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:51:59.210 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:51:59.211 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:59.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:59.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:59.214 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:59.216 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:51:59.216 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:51:59.254 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:51:59.255 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:51:59.277 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:59 smithi151 ceph-mon[104450]: pgmap v1765: 119 pgs: 4 active+remapped+backfilling, 115 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 888 op/s; 8113/255828 objects misplaced (3.171%); 37 MiB/s, 597 objects/s recovering 2024-09-13T14:51:59.277 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:51:59 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3943318833' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:59.278 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:59 smithi151 ceph-mon[108966]: pgmap v1765: 119 pgs: 4 active+remapped+backfilling, 115 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 888 op/s; 8113/255828 objects misplaced (3.171%); 37 MiB/s, 597 objects/s recovering 2024-09-13T14:51:59.278 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:51:59 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3943318833' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:59.313 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:59.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:59.348 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:59.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:59.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:59 smithi184 ceph-mon[92527]: pgmap v1765: 119 pgs: 4 active+remapped+backfilling, 115 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 888 op/s; 8113/255828 objects misplaced (3.171%); 37 MiB/s, 597 objects/s recovering 2024-09-13T14:51:59.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:51:59 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3943318833' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:59.449 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:59.479 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:59.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:59.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:59.582 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:59.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:59.625 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:59.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:59.726 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:59.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:59.754 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:51:59.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:59.855 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:59.897 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:51:59.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:59.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:59.999 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:00.033 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:00.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:00.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:00.135 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:00.185 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:00.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:00.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:00.287 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:00.315 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:00.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:00.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:00.333 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:52:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:52:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:52:00.367 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:52:00.418 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:00.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:00.601 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:00.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:00.702 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:00.746 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:52:00.746 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [ 2024-09-13T14:52:00.746 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.746 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35940.0:171064 8.e e1183/1178 8:7a26ed3b:::benchmark_data_smithi151_229097_object46196:head v 1183'10702, mlcod=1183'10699)", 2024-09-13T14:52:00.746 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:00.727987+0000", 2024-09-13T14:52:00.746 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.0054279760000000002, 2024-09-13T14:52:00.746 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0054581259999999998, 2024-09-13T14:52:00.747 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:52:00.747 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:52:00.747 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:52:00.747 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.747 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:52:00.747 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.727987+0000", 2024-09-13T14:52:00.747 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:00.747 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.747 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.747 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:52:00.747 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.727987+0000", 2024-09-13T14:52:00.747 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:00.747 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.748 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.748 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:52:00.748 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.727986+0000", 2024-09-13T14:52:00.748 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:52:00.748 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.748 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.748 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:52:00.748 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.727998+0000", 2024-09-13T14:52:00.748 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.173e-05 2024-09-13T14:52:00.748 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.748 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.748 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:52:00.749 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.728016+0000", 2024-09-13T14:52:00.749 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.8644999999999999e-05 2024-09-13T14:52:00.749 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.749 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.749 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:52:00.749 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.728024+0000", 2024-09-13T14:52:00.749 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.3950000000000003e-06 2024-09-13T14:52:00.749 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.749 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.749 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:52:00.749 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.728229+0000", 2024-09-13T14:52:00.750 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00020505299999999999 2024-09-13T14:52:00.750 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.750 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.750 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:52:00.750 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.728286+0000", 2024-09-13T14:52:00.750 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7308000000000001e-05 2024-09-13T14:52:00.750 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:00.750 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:52:00.750 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:00.750 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.750 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.750 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35940.0:171072 8.c e1183/1167 8:3e10f35b:::benchmark_data_smithi151_229097_object46204:head v 1183'10678, mlcod=1183'10676)", 2024-09-13T14:52:00.751 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:00.730740+0000", 2024-09-13T14:52:00.751 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.002675139, 2024-09-13T14:52:00.751 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0027657659999999998, 2024-09-13T14:52:00.751 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:52:00.751 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:52:00.751 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:52:00.751 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.751 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:52:00.751 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.730740+0000", 2024-09-13T14:52:00.751 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:00.751 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.751 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.751 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:52:00.752 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.730740+0000", 2024-09-13T14:52:00.752 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:00.752 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.752 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.752 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:52:00.752 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.730739+0000", 2024-09-13T14:52:00.752 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:52:00.752 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.752 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.752 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:52:00.752 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.730751+0000", 2024-09-13T14:52:00.752 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1518999999999999e-05 2024-09-13T14:52:00.752 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.753 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.753 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:52:00.753 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.730768+0000", 2024-09-13T14:52:00.753 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.7572e-05 2024-09-13T14:52:00.753 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.753 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.753 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:52:00.753 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.730776+0000", 2024-09-13T14:52:00.753 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.3499999999999999e-06 2024-09-13T14:52:00.753 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.753 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.753 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:52:00.753 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.730881+0000", 2024-09-13T14:52:00.754 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000105667 2024-09-13T14:52:00.754 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.754 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.754 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:52:00.754 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.730912+0000", 2024-09-13T14:52:00.754 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.057e-05 2024-09-13T14:52:00.754 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:00.754 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:52:00.754 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:00.754 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.754 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.754 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.35940.0:171067 8.6 e1183/1167 8:69b926f2:::benchmark_data_smithi151_229097_object46199:head v 1183'10386, mlcod=1183'10384)", 2024-09-13T14:52:00.755 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:00.730920+0000", 2024-09-13T14:52:00.755 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.0024951919999999998, 2024-09-13T14:52:00.755 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0026795220000000002, 2024-09-13T14:52:00.755 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:52:00.755 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:52:00.755 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:52:00.755 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.755 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:52:00.755 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.730920+0000", 2024-09-13T14:52:00.755 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:00.755 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.755 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.755 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:52:00.755 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.730920+0000", 2024-09-13T14:52:00.756 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:00.756 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.756 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.756 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:52:00.756 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.730919+0000", 2024-09-13T14:52:00.756 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:52:00.756 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.756 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.756 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:52:00.756 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.730928+0000", 2024-09-13T14:52:00.757 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.3089999999999996e-06 2024-09-13T14:52:00.757 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.757 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.757 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:52:00.757 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.730937+0000", 2024-09-13T14:52:00.757 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.2099999999999999e-06 2024-09-13T14:52:00.757 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.757 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.757 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:52:00.757 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.730942+0000", 2024-09-13T14:52:00.757 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3040000000000004e-06 2024-09-13T14:52:00.758 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.758 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.758 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:52:00.758 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.731121+0000", 2024-09-13T14:52:00.758 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00017892299999999999 2024-09-13T14:52:00.758 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.758 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.758 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:52:00.758 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.731173+0000", 2024-09-13T14:52:00.758 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.2033999999999997e-05 2024-09-13T14:52:00.758 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:00.758 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:52:00.758 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:00.758 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.759 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.759 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "pg_backfill(progress 8.e e 1183/1183 lb 8:7a3989f7:::benchmark_data_smithi151_229097_object112819:head)", 2024-09-13T14:52:00.759 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:00.733592+0000", 2024-09-13T14:52:00.759 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 4294967295.9998231, 2024-09-13T14:52:00.759 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.6781999999999998e-05, 2024-09-13T14:52:00.759 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:52:00.759 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "queued for pg", 2024-09-13T14:52:00.759 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:52:00.759 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.759 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:52:00.759 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.733592+0000", 2024-09-13T14:52:00.759 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:00.759 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.760 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.760 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:52:00.760 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.733592+0000", 2024-09-13T14:52:00.760 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:00.760 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.760 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.760 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:52:00.760 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.733591+0000", 2024-09-13T14:52:00.760 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:52:00.760 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.760 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.760 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:52:00.760 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.733602+0000", 2024-09-13T14:52:00.760 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0560999999999999e-05 2024-09-13T14:52:00.761 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.761 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.761 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:52:00.761 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.733619+0000", 2024-09-13T14:52:00.761 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6963999999999999e-05 2024-09-13T14:52:00.761 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.761 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.761 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:52:00.761 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.733625+0000", 2024-09-13T14:52:00.761 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.6209999999999998e-06 2024-09-13T14:52:00.761 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:00.761 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:52:00.761 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:00.762 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.762 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.762 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "MOSDPGPush(8.e 1183/1178 [PushOp(8:7a3a1e66:::benchmark_data_smithi151_229097_object102998:head, version: 1174'6452, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:7a3a1e66:::benchmark_data_smithi151_229097_object102998:head@1174'6452, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:52:00.762 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:00.733636+0000", 2024-09-13T14:52:00.762 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 4294967295.9997797, 2024-09-13T14:52:00.762 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000100092, 2024-09-13T14:52:00.762 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:52:00.762 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "queued for pg", 2024-09-13T14:52:00.762 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:52:00.762 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.762 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:52:00.763 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.733636+0000", 2024-09-13T14:52:00.763 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:00.763 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.763 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.763 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:52:00.763 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.733636+0000", 2024-09-13T14:52:00.763 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:00.763 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.763 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.763 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:52:00.763 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.733635+0000", 2024-09-13T14:52:00.763 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:52:00.763 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.764 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.764 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:52:00.764 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.733681+0000", 2024-09-13T14:52:00.764 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5172000000000001e-05 2024-09-13T14:52:00.764 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.764 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.764 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:52:00.764 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.733695+0000", 2024-09-13T14:52:00.764 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4038e-05 2024-09-13T14:52:00.765 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:00.765 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:00.765 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:52:00.765 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:00.733704+0000", 2024-09-13T14:52:00.765 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.2390000000000008e-06 2024-09-13T14:52:00.765 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:00.765 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:52:00.765 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:00.765 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:00.765 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-13T14:52:00.765 INFO:teuthology.orchestra.run.smithi184.stdout: "num_ops": 5 2024-09-13T14:52:00.765 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:52:00.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:00.766 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:00.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:00.867 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:00.900 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:00.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:00.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:01.002 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:01.047 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:01.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:01.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:01.149 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:01.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:01.185 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:01.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:01.286 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:01.315 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:01.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:01.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:01.418 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:01.448 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:01.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:01.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:01.550 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:01.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:01.589 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:01.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:01.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:01 smithi184 ceph-mon[92527]: pgmap v1766: 119 pgs: 4 active+remapped+backfilling, 115 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 922 op/s; 7379/247716 objects misplaced (2.979%); 32 MiB/s, 506 objects/s recovering 2024-09-13T14:52:01.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:01 smithi151 ceph-mon[104450]: pgmap v1766: 119 pgs: 4 active+remapped+backfilling, 115 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 922 op/s; 7379/247716 objects misplaced (2.979%); 32 MiB/s, 506 objects/s recovering 2024-09-13T14:52:01.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:01 smithi151 ceph-mon[108966]: pgmap v1766: 119 pgs: 4 active+remapped+backfilling, 115 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 922 op/s; 7379/247716 objects misplaced (2.979%); 32 MiB/s, 506 objects/s recovering 2024-09-13T14:52:01.690 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:01.702 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:01.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:01.753 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:01.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:01.855 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:01.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:01.902 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:01.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:02.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:02.049 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:02.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:02.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:02.150 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:02.190 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:02.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:02.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:02.292 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:02.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:02.322 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:02.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:02.422 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:02.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:02.456 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:02.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:02.500 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:52:02.500 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:02.500 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:02 smithi151 ceph-mon[104450]: pgmap v1767: 119 pgs: 4 active+remapped+backfilling, 115 active+clean; 4.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 958 op/s; 5678/241386 objects misplaced (2.352%); 25 MiB/s, 406 objects/s recovering 2024-09-13T14:52:02.500 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:52:02.500 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:02.501 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:02 smithi151 ceph-mon[108966]: pgmap v1767: 119 pgs: 4 active+remapped+backfilling, 115 active+clean; 4.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 958 op/s; 5678/241386 objects misplaced (2.352%); 25 MiB/s, 406 objects/s recovering 2024-09-13T14:52:02.557 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:02.585 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:02.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:02.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:02.688 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:02.733 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:02.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:02.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:02.742 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:52:02.745 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T14:52:02.836 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:02.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:52:02.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:02.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:02 smithi184 ceph-mon[92527]: pgmap v1767: 119 pgs: 4 active+remapped+backfilling, 115 active+clean; 4.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 958 op/s; 5678/241386 objects misplaced (2.352%); 25 MiB/s, 406 objects/s recovering 2024-09-13T14:52:02.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:02.868 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:02.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:02.969 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:03.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:03.003 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:03.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:03.104 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:03.131 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:03.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:03.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:03.233 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:03.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:03.267 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:03.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:03.368 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:03.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:03.414 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:03.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:03.515 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:03.556 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:03.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:03.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:03.658 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:03.669 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:52:03.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:03.697 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:03.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:03.720 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:03 smithi184 ceph-mon[92527]: from='client.36063 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:52:03.720 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:03 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 5]} : dispatch 2024-09-13T14:52:03.720 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:03 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 5]} : dispatch 2024-09-13T14:52:03.734 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:03 smithi151 ceph-mon[104450]: from='client.36063 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:52:03.734 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:03 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 5]} : dispatch 2024-09-13T14:52:03.734 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:03 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 5]} : dispatch 2024-09-13T14:52:03.734 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:03 smithi151 ceph-mon[108966]: from='client.36063 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:52:03.735 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:03 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 5]} : dispatch 2024-09-13T14:52:03.735 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:03 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 5]} : dispatch 2024-09-13T14:52:03.752 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T14:52:03.799 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:03.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:03.838 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:03.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:03.939 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:04.004 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:04.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:04.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:04.100 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:52:04.106 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:04.143 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:04.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:04.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:04.245 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:04.275 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:04.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:04.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:04.377 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:04.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:04.406 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:04.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:04.472 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1767,"stamp":"2024-09-13T14:52:01.601757+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5272024353,"num_objects":80462,"num_object_clones":0,"num_object_copies":241386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5678,"num_objects_unfound":0,"num_objects_dirty":80462,"num_whiteouts":0,"num_read":21,"num_read_kb":52,"num_write":169369,"num_write_kb":7991503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23538,"num_bytes_recovered":1540927612,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95523,"ondisk_log_size":95523,"up":357,"acting":357,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":358,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":19427920,"kb_used_data":15434580,"kb_used_omap":148,"kb_used_meta":3992875,"kb_avail":636788144,"statfs":{"total":671965249536,"available":652071059456,"internally_reserved":0,"allocated":15805009920,"data_stored":15779277432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":151697,"internal_metadata":4088704879},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-628293632,"num_objects":-9587,"num_object_clones":0,"num_object_copies":-28761,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-4988,"num_objects_unfound":0,"num_objects_dirty":-9587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9587,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4064,"num_bytes_recovered":266338304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.005522"},"pg_stats":[{"pgid":"1.1d","version":"261'100","reported_seq":145,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.244998+0000","last_change":"2024-09-13T14:51:19.907325+0000","last_active":"2024-09-13T14:51:54.244998+0000","last_peered":"2024-09-13T14:51:54.244998+0000","last_clean":"2024-09-13T14:51:54.244998+0000","last_became_active":"2024-09-13T14:51:19.906625+0000","last_became_peered":"2024-09-13T14:51:19.906625+0000","last_unstale":"2024-09-13T14:51:54.244998+0000","last_undegraded":"2024-09-13T14:51:54.244998+0000","last_fullsized":"2024-09-13T14:51:54.244998+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:17.988345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:17.988345+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:36.836053+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":1744,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.239427+0000","last_change":"2024-09-13T14:29:42.541263+0000","last_active":"2024-09-13T14:51:54.239427+0000","last_peered":"2024-09-13T14:51:54.239427+0000","last_clean":"2024-09-13T14:51:54.239427+0000","last_became_active":"2024-09-13T14:29:42.540873+0000","last_became_peered":"2024-09-13T14:29:42.540873+0000","last_unstale":"2024-09-13T14:51:54.239427+0000","last_undegraded":"2024-09-13T14:51:54.239427+0000","last_fullsized":"2024-09-13T14:51:54.239427+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:20.611206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1099,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.253103+0000","last_change":"2024-09-13T14:51:19.906192+0000","last_active":"2024-09-13T14:51:54.253103+0000","last_peered":"2024-09-13T14:51:54.253103+0000","last_clean":"2024-09-13T14:51:54.253103+0000","last_became_active":"2024-09-13T14:51:19.905896+0000","last_became_peered":"2024-09-13T14:51:19.905896+0000","last_unstale":"2024-09-13T14:51:54.253103+0000","last_undegraded":"2024-09-13T14:51:54.253103+0000","last_fullsized":"2024-09-13T14:51:54.253103+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:00:47.332946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":83,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.236880+0000","last_change":"2024-09-13T14:45:19.810203+0000","last_active":"2024-09-13T14:51:54.236880+0000","last_peered":"2024-09-13T14:51:54.236880+0000","last_clean":"2024-09-13T14:51:54.236880+0000","last_became_active":"2024-09-13T14:45:19.809693+0000","last_became_peered":"2024-09-13T14:45:19.809693+0000","last_unstale":"2024-09-13T14:51:54.236880+0000","last_undegraded":"2024-09-13T14:51:54.236880+0000","last_fullsized":"2024-09-13T14:51:54.236880+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:20.219673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":82,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.235497+0000","last_change":"2024-09-13T14:45:19.807991+0000","last_active":"2024-09-13T14:51:54.235497+0000","last_peered":"2024-09-13T14:51:54.235497+0000","last_clean":"2024-09-13T14:51:54.235497+0000","last_became_active":"2024-09-13T14:45:19.807662+0000","last_became_peered":"2024-09-13T14:45:19.807662+0000","last_unstale":"2024-09-13T14:51:54.235497+0000","last_undegraded":"2024-09-13T14:51:54.235497+0000","last_fullsized":"2024-09-13T14:51:54.235497+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:29.306108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"261'100","reported_seq":1254,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.186216+0000","last_change":"2024-09-13T14:44:54.259562+0000","last_active":"2024-09-13T14:51:46.186216+0000","last_peered":"2024-09-13T14:51:46.186216+0000","last_clean":"2024-09-13T14:51:46.186216+0000","last_became_active":"2024-09-13T14:43:14.563717+0000","last_became_peered":"2024-09-13T14:43:14.563717+0000","last_unstale":"2024-09-13T14:51:46.186216+0000","last_undegraded":"2024-09-13T14:51:46.186216+0000","last_fullsized":"2024-09-13T14:51:46.186216+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_clean_scrub_stamp":"2024-09-13T14:44:54.259482+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:43:52.729109+0000","scrub_duration":0.015265569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":1125,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.191722+0000","last_change":"2024-09-13T14:51:19.931014+0000","last_active":"2024-09-13T14:51:46.191722+0000","last_peered":"2024-09-13T14:51:46.191722+0000","last_clean":"2024-09-13T14:51:46.191722+0000","last_became_active":"2024-09-13T14:51:19.905511+0000","last_became_peered":"2024-09-13T14:51:19.905511+0000","last_unstale":"2024-09-13T14:51:46.191722+0000","last_undegraded":"2024-09-13T14:51:46.191722+0000","last_fullsized":"2024-09-13T14:51:46.191722+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:09:45.562438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":608,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.232824+0000","last_change":"2024-09-13T14:51:19.897388+0000","last_active":"2024-09-13T14:51:54.232824+0000","last_peered":"2024-09-13T14:51:54.232824+0000","last_clean":"2024-09-13T14:51:54.232824+0000","last_became_active":"2024-09-13T14:51:19.897092+0000","last_became_peered":"2024-09-13T14:51:19.897092+0000","last_unstale":"2024-09-13T14:51:54.232824+0000","last_undegraded":"2024-09-13T14:51:54.232824+0000","last_fullsized":"2024-09-13T14:51:54.232824+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:53.450593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1e","version":"261'100","reported_seq":100,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.237023+0000","last_change":"2024-09-13T14:45:35.947332+0000","last_active":"2024-09-13T14:51:54.237023+0000","last_peered":"2024-09-13T14:51:54.237023+0000","last_clean":"2024-09-13T14:51:54.237023+0000","last_became_active":"2024-09-13T14:45:19.810775+0000","last_became_peered":"2024-09-13T14:45:19.810775+0000","last_unstale":"2024-09-13T14:51:54.237023+0000","last_undegraded":"2024-09-13T14:51:54.237023+0000","last_fullsized":"2024-09-13T14:51:54.237023+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:35.947265+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:43:41.569795+0000","last_clean_scrub_stamp":"2024-09-13T14:45:35.947265+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:00.606370+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":1170,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.195968+0000","last_change":"2024-09-13T14:51:19.914073+0000","last_active":"2024-09-13T14:51:46.195968+0000","last_peered":"2024-09-13T14:51:46.195968+0000","last_clean":"2024-09-13T14:51:46.195968+0000","last_became_active":"2024-09-13T14:51:19.902618+0000","last_became_peered":"2024-09-13T14:51:19.902618+0000","last_unstale":"2024-09-13T14:51:46.195968+0000","last_undegraded":"2024-09-13T14:51:46.195968+0000","last_fullsized":"2024-09-13T14:51:46.195968+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:17:55.364334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1109,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.255229+0000","last_change":"2024-09-13T14:51:19.911521+0000","last_active":"2024-09-13T14:51:54.255229+0000","last_peered":"2024-09-13T14:51:54.255229+0000","last_clean":"2024-09-13T14:51:54.255229+0000","last_became_active":"2024-09-13T14:51:19.910935+0000","last_became_peered":"2024-09-13T14:51:19.910935+0000","last_unstale":"2024-09-13T14:51:54.255229+0000","last_undegraded":"2024-09-13T14:51:54.255229+0000","last_fullsized":"2024-09-13T14:51:54.255229+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:13.016819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":1735,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.242669+0000","last_change":"2024-09-13T14:51:19.909986+0000","last_active":"2024-09-13T14:51:54.242669+0000","last_peered":"2024-09-13T14:51:54.242669+0000","last_clean":"2024-09-13T14:51:54.242669+0000","last_became_active":"2024-09-13T14:51:19.909459+0000","last_became_peered":"2024-09-13T14:51:19.909459+0000","last_unstale":"2024-09-13T14:51:54.242669+0000","last_undegraded":"2024-09-13T14:51:54.242669+0000","last_fullsized":"2024-09-13T14:51:54.242669+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:44.484202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1750,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.243741+0000","last_change":"2024-09-13T14:45:19.808811+0000","last_active":"2024-09-13T14:51:54.243741+0000","last_peered":"2024-09-13T14:51:54.243741+0000","last_clean":"2024-09-13T14:51:54.243741+0000","last_became_active":"2024-09-13T14:45:19.808471+0000","last_became_peered":"2024-09-13T14:45:19.808471+0000","last_unstale":"2024-09-13T14:51:54.243741+0000","last_undegraded":"2024-09-13T14:51:54.243741+0000","last_fullsized":"2024-09-13T14:51:54.243741+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:53.638161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"261'100","reported_seq":82,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.235581+0000","last_change":"2024-09-13T14:45:19.813412+0000","last_active":"2024-09-13T14:51:54.235581+0000","last_peered":"2024-09-13T14:51:54.235581+0000","last_clean":"2024-09-13T14:51:54.235581+0000","last_became_active":"2024-09-13T14:45:19.812585+0000","last_became_peered":"2024-09-13T14:45:19.812585+0000","last_unstale":"2024-09-13T14:51:54.235581+0000","last_undegraded":"2024-09-13T14:51:54.235581+0000","last_fullsized":"2024-09-13T14:51:54.235581+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.614340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.614340+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:32:07.434109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":1118,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.247319+0000","last_change":"2024-09-13T14:51:19.909275+0000","last_active":"2024-09-13T14:51:54.247319+0000","last_peered":"2024-09-13T14:51:54.247319+0000","last_clean":"2024-09-13T14:51:54.247319+0000","last_became_active":"2024-09-13T14:51:19.906971+0000","last_became_peered":"2024-09-13T14:51:19.906971+0000","last_unstale":"2024-09-13T14:51:54.247319+0000","last_undegraded":"2024-09-13T14:51:54.247319+0000","last_fullsized":"2024-09-13T14:51:54.247319+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:44.343244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":89,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.232377+0000","last_change":"2024-09-13T14:51:19.902217+0000","last_active":"2024-09-13T14:51:54.232377+0000","last_peered":"2024-09-13T14:51:54.232377+0000","last_clean":"2024-09-13T14:51:54.232377+0000","last_became_active":"2024-09-13T14:51:19.901908+0000","last_became_peered":"2024-09-13T14:51:19.901908+0000","last_unstale":"2024-09-13T14:51:54.232377+0000","last_undegraded":"2024-09-13T14:51:54.232377+0000","last_fullsized":"2024-09-13T14:51:54.232377+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:32.001265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.18","version":"261'100","reported_seq":132,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.191430+0000","last_change":"2024-09-13T14:44:41.685537+0000","last_active":"2024-09-13T14:51:46.191430+0000","last_peered":"2024-09-13T14:51:46.191430+0000","last_clean":"2024-09-13T14:51:46.191430+0000","last_became_active":"2024-09-13T14:43:14.558502+0000","last_became_peered":"2024-09-13T14:43:14.558502+0000","last_unstale":"2024-09-13T14:51:46.191430+0000","last_undegraded":"2024-09-13T14:51:46.191430+0000","last_fullsized":"2024-09-13T14:51:46.191430+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1134,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:41.685124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:41.685124+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:37:58.412497+0000","scrub_duration":0.0075205970000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":138,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.247332+0000","last_change":"2024-09-13T14:51:19.909348+0000","last_active":"2024-09-13T14:51:54.247332+0000","last_peered":"2024-09-13T14:51:54.247332+0000","last_clean":"2024-09-13T14:51:54.247332+0000","last_became_active":"2024-09-13T14:51:19.908782+0000","last_became_peered":"2024-09-13T14:51:19.908782+0000","last_unstale":"2024-09-13T14:51:54.247332+0000","last_undegraded":"2024-09-13T14:51:54.247332+0000","last_fullsized":"2024-09-13T14:51:54.247332+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:56:57.088901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1766,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.248623+0000","last_change":"2024-09-13T14:45:19.815341+0000","last_active":"2024-09-13T14:51:54.248623+0000","last_peered":"2024-09-13T14:51:54.248623+0000","last_clean":"2024-09-13T14:51:54.248623+0000","last_became_active":"2024-09-13T14:45:19.810262+0000","last_became_peered":"2024-09-13T14:45:19.810262+0000","last_unstale":"2024-09-13T14:51:54.248623+0000","last_undegraded":"2024-09-13T14:51:54.248623+0000","last_fullsized":"2024-09-13T14:51:54.248623+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:39.847653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1b","version":"261'100","reported_seq":84,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.191889+0000","last_change":"2024-09-13T14:51:19.904448+0000","last_active":"2024-09-13T14:51:46.191889+0000","last_peered":"2024-09-13T14:51:46.191889+0000","last_clean":"2024-09-13T14:51:46.191889+0000","last_became_active":"2024-09-13T14:51:19.904096+0000","last_became_peered":"2024-09-13T14:51:19.904096+0000","last_unstale":"2024-09-13T14:51:46.191889+0000","last_undegraded":"2024-09-13T14:51:46.191889+0000","last_fullsized":"2024-09-13T14:51:46.191889+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:52.803242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:52.803242+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:47.588858+0000","scrub_duration":0.0019769660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":82,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.233279+0000","last_change":"2024-09-13T14:45:19.806623+0000","last_active":"2024-09-13T14:51:54.233279+0000","last_peered":"2024-09-13T14:51:54.233279+0000","last_clean":"2024-09-13T14:51:54.233279+0000","last_became_active":"2024-09-13T14:45:19.806219+0000","last_became_peered":"2024-09-13T14:45:19.806219+0000","last_unstale":"2024-09-13T14:51:54.233279+0000","last_undegraded":"2024-09-13T14:51:54.233279+0000","last_fullsized":"2024-09-13T14:51:54.233279+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:34:41.601201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1136,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.191797+0000","last_change":"2024-09-13T14:43:32.637751+0000","last_active":"2024-09-13T14:51:46.191797+0000","last_peered":"2024-09-13T14:51:46.191797+0000","last_clean":"2024-09-13T14:51:46.191797+0000","last_became_active":"2024-09-13T14:43:32.636901+0000","last_became_peered":"2024-09-13T14:43:32.636901+0000","last_unstale":"2024-09-13T14:51:46.191797+0000","last_undegraded":"2024-09-13T14:51:46.191797+0000","last_fullsized":"2024-09-13T14:51:46.191797+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:21.942419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1a","version":"261'100","reported_seq":19,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.185865+0000","last_change":"2024-09-13T14:51:19.903142+0000","last_active":"2024-09-13T14:51:46.185865+0000","last_peered":"2024-09-13T14:51:46.185865+0000","last_clean":"2024-09-13T14:51:46.185865+0000","last_became_active":"2024-09-13T14:51:19.902810+0000","last_became_peered":"2024-09-13T14:51:19.902810+0000","last_unstale":"2024-09-13T14:51:46.185865+0000","last_undegraded":"2024-09-13T14:51:46.185865+0000","last_fullsized":"2024-09-13T14:51:46.185865+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:41.909988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:41.909988+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":82,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.233358+0000","last_change":"2024-09-13T14:45:19.812071+0000","last_active":"2024-09-13T14:51:54.233358+0000","last_peered":"2024-09-13T14:51:54.233358+0000","last_clean":"2024-09-13T14:51:54.233358+0000","last_became_active":"2024-09-13T14:45:19.811336+0000","last_became_peered":"2024-09-13T14:45:19.811336+0000","last_unstale":"2024-09-13T14:51:54.233358+0000","last_undegraded":"2024-09-13T14:51:54.233358+0000","last_fullsized":"2024-09-13T14:51:54.233358+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:40:29.814608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1783,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.239343+0000","last_change":"2024-09-13T14:45:19.810478+0000","last_active":"2024-09-13T14:51:54.239343+0000","last_peered":"2024-09-13T14:51:54.239343+0000","last_clean":"2024-09-13T14:51:54.239343+0000","last_became_active":"2024-09-13T14:45:19.808278+0000","last_became_peered":"2024-09-13T14:45:19.808278+0000","last_unstale":"2024-09-13T14:51:54.239343+0000","last_undegraded":"2024-09-13T14:51:54.239343+0000","last_fullsized":"2024-09-13T14:51:54.239343+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:12.211089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"261'100","reported_seq":74,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.186395+0000","last_change":"2024-09-13T14:44:38.214961+0000","last_active":"2024-09-13T14:51:46.186395+0000","last_peered":"2024-09-13T14:51:46.186395+0000","last_clean":"2024-09-13T14:51:46.186395+0000","last_became_active":"2024-09-13T14:43:14.557883+0000","last_became_peered":"2024-09-13T14:43:14.557883+0000","last_unstale":"2024-09-13T14:51:46.186395+0000","last_undegraded":"2024-09-13T14:51:46.186395+0000","last_fullsized":"2024-09-13T14:51:46.186395+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:38.214865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:38.214865+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:49:36.013783+0000","scrub_duration":0.0016389639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1654,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.235714+0000","last_change":"2024-09-13T14:30:15.576921+0000","last_active":"2024-09-13T14:51:54.235714+0000","last_peered":"2024-09-13T14:51:54.235714+0000","last_clean":"2024-09-13T14:51:54.235714+0000","last_became_active":"2024-09-13T14:30:15.576607+0000","last_became_peered":"2024-09-13T14:30:15.576607+0000","last_unstale":"2024-09-13T14:51:54.235714+0000","last_undegraded":"2024-09-13T14:51:54.235714+0000","last_fullsized":"2024-09-13T14:51:54.235714+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:35.162460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1146,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.191623+0000","last_change":"2024-09-13T14:43:07.518801+0000","last_active":"2024-09-13T14:51:46.191623+0000","last_peered":"2024-09-13T14:51:46.191623+0000","last_clean":"2024-09-13T14:51:46.191623+0000","last_became_active":"2024-09-13T14:43:07.517610+0000","last_became_peered":"2024-09-13T14:43:07.517610+0000","last_unstale":"2024-09-13T14:51:46.191623+0000","last_undegraded":"2024-09-13T14:51:46.191623+0000","last_fullsized":"2024-09-13T14:51:46.191623+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:51.632254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"261'100","reported_seq":138,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.243579+0000","last_change":"2024-09-13T14:51:19.899337+0000","last_active":"2024-09-13T14:51:54.243579+0000","last_peered":"2024-09-13T14:51:54.243579+0000","last_clean":"2024-09-13T14:51:54.243579+0000","last_became_active":"2024-09-13T14:51:19.898806+0000","last_became_peered":"2024-09-13T14:51:19.898806+0000","last_unstale":"2024-09-13T14:51:54.243579+0000","last_undegraded":"2024-09-13T14:51:54.243579+0000","last_fullsized":"2024-09-13T14:51:54.243579+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.037517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.037517+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:04.124642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":1156,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.186209+0000","last_change":"2024-09-13T14:51:19.946789+0000","last_active":"2024-09-13T14:51:46.186209+0000","last_peered":"2024-09-13T14:51:46.186209+0000","last_clean":"2024-09-13T14:51:46.186209+0000","last_became_active":"2024-09-13T14:51:19.905677+0000","last_became_peered":"2024-09-13T14:51:19.905677+0000","last_unstale":"2024-09-13T14:51:46.186209+0000","last_undegraded":"2024-09-13T14:51:46.186209+0000","last_fullsized":"2024-09-13T14:51:46.186209+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:41:14.530433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":131,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.247668+0000","last_change":"2024-09-13T14:43:07.521576+0000","last_active":"2024-09-13T14:51:54.247668+0000","last_peered":"2024-09-13T14:51:54.247668+0000","last_clean":"2024-09-13T14:51:54.247668+0000","last_became_active":"2024-09-13T14:43:07.520478+0000","last_became_peered":"2024-09-13T14:43:07.520478+0000","last_unstale":"2024-09-13T14:51:54.247668+0000","last_undegraded":"2024-09-13T14:51:54.247668+0000","last_fullsized":"2024-09-13T14:51:54.247668+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:53.410135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"261'100","reported_seq":400,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.191679+0000","last_change":"2024-09-13T14:44:49.775941+0000","last_active":"2024-09-13T14:51:46.191679+0000","last_peered":"2024-09-13T14:51:46.191679+0000","last_clean":"2024-09-13T14:51:46.191679+0000","last_became_active":"2024-09-13T14:43:14.560297+0000","last_became_peered":"2024-09-13T14:43:14.560297+0000","last_unstale":"2024-09-13T14:51:46.191679+0000","last_undegraded":"2024-09-13T14:51:46.191679+0000","last_fullsized":"2024-09-13T14:51:46.191679+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:49.775844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:49.775844+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:15:53.138661+0000","scrub_duration":0.012350274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":336,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.186356+0000","last_change":"2024-09-13T14:51:19.922876+0000","last_active":"2024-09-13T14:51:46.186356+0000","last_peered":"2024-09-13T14:51:46.186356+0000","last_clean":"2024-09-13T14:51:46.186356+0000","last_became_active":"2024-09-13T14:51:19.897151+0000","last_became_peered":"2024-09-13T14:51:19.897151+0000","last_unstale":"2024-09-13T14:51:46.186356+0000","last_undegraded":"2024-09-13T14:51:46.186356+0000","last_fullsized":"2024-09-13T14:51:46.186356+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:53:08.858083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1099,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.244860+0000","last_change":"2024-09-13T14:51:19.913252+0000","last_active":"2024-09-13T14:51:54.244860+0000","last_peered":"2024-09-13T14:51:54.244860+0000","last_clean":"2024-09-13T14:51:54.244860+0000","last_became_active":"2024-09-13T14:51:19.912953+0000","last_became_peered":"2024-09-13T14:51:19.912953+0000","last_unstale":"2024-09-13T14:51:54.244860+0000","last_undegraded":"2024-09-13T14:51:54.244860+0000","last_fullsized":"2024-09-13T14:51:54.244860+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:39:27.026947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.16","version":"261'100","reported_seq":138,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.245274+0000","last_change":"2024-09-13T14:45:19.807461+0000","last_active":"2024-09-13T14:51:54.245274+0000","last_peered":"2024-09-13T14:51:54.245274+0000","last_clean":"2024-09-13T14:51:54.245274+0000","last_became_active":"2024-09-13T14:45:19.807117+0000","last_became_peered":"2024-09-13T14:45:19.807117+0000","last_unstale":"2024-09-13T14:51:54.245274+0000","last_undegraded":"2024-09-13T14:51:54.245274+0000","last_fullsized":"2024-09-13T14:51:54.245274+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.955197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.955197+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:03.531555+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.d","version":"1183'10773","reported_seq":10798,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:57.854983+0000","last_change":"2024-09-13T14:49:22.120980+0000","last_active":"2024-09-13T14:51:57.854983+0000","last_peered":"2024-09-13T14:51:57.854983+0000","last_clean":"2024-09-13T14:51:57.854983+0000","last_became_active":"2024-09-13T14:49:22.120684+0000","last_became_peered":"2024-09-13T14:49:22.120684+0000","last_unstale":"2024-09-13T14:51:57.854983+0000","last_undegraded":"2024-09-13T14:51:57.854983+0000","last_fullsized":"2024-09-13T14:51:57.854983+0000","mapping_epoch":1167,"log_start":"1172'4306","ondisk_log_start":"1172'4306","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":6467,"log_dups_size":0,"ondisk_log_size":6467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:49:34.732581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336330776,"num_objects":5133,"num_object_clones":0,"num_object_copies":15399,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5133,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10773,"num_write_kb":508929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.191530+0000","last_change":"2024-09-13T14:51:19.904110+0000","last_active":"2024-09-13T14:51:46.191530+0000","last_peered":"2024-09-13T14:51:46.191530+0000","last_clean":"2024-09-13T14:51:46.191530+0000","last_became_active":"2024-09-13T14:51:19.903806+0000","last_became_peered":"2024-09-13T14:51:19.903806+0000","last_unstale":"2024-09-13T14:51:46.191530+0000","last_undegraded":"2024-09-13T14:51:46.191530+0000","last_fullsized":"2024-09-13T14:51:46.191530+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:11.557749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":608,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.186495+0000","last_change":"2024-09-13T14:43:32.635070+0000","last_active":"2024-09-13T14:51:46.186495+0000","last_peered":"2024-09-13T14:51:46.186495+0000","last_clean":"2024-09-13T14:51:46.186495+0000","last_became_active":"2024-09-13T14:43:32.634734+0000","last_became_peered":"2024-09-13T14:43:32.634734+0000","last_unstale":"2024-09-13T14:51:46.186495+0000","last_undegraded":"2024-09-13T14:51:46.186495+0000","last_fullsized":"2024-09-13T14:51:46.186495+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:33.107010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":1784,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.248770+0000","last_change":"2024-09-13T14:51:19.921328+0000","last_active":"2024-09-13T14:51:54.248770+0000","last_peered":"2024-09-13T14:51:54.248770+0000","last_clean":"2024-09-13T14:51:54.248770+0000","last_became_active":"2024-09-13T14:51:19.901833+0000","last_became_peered":"2024-09-13T14:51:19.901833+0000","last_unstale":"2024-09-13T14:51:54.248770+0000","last_undegraded":"2024-09-13T14:51:54.248770+0000","last_fullsized":"2024-09-13T14:51:54.248770+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:57.881817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"261'100","reported_seq":83,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.233328+0000","last_change":"2024-09-13T14:45:19.813527+0000","last_active":"2024-09-13T14:51:54.233328+0000","last_peered":"2024-09-13T14:51:54.233328+0000","last_clean":"2024-09-13T14:51:54.233328+0000","last_became_active":"2024-09-13T14:45:19.812834+0000","last_became_peered":"2024-09-13T14:45:19.812834+0000","last_unstale":"2024-09-13T14:51:54.233328+0000","last_undegraded":"2024-09-13T14:51:54.233328+0000","last_fullsized":"2024-09-13T14:51:54.233328+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.710192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.710192+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:08:55.152703+0000","scrub_duration":0.0019250910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"1183'10636","reported_seq":10680,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.845471+0000","last_change":"2024-09-13T14:49:22.114132+0000","last_active":"2024-09-13T14:51:59.845471+0000","last_peered":"2024-09-13T14:51:59.845471+0000","last_clean":"2024-09-13T14:51:59.845471+0000","last_became_active":"2024-09-13T14:49:22.113820+0000","last_became_peered":"2024-09-13T14:49:22.113820+0000","last_unstale":"2024-09-13T14:51:59.845471+0000","last_undegraded":"2024-09-13T14:51:59.845471+0000","last_fullsized":"2024-09-13T14:51:59.845471+0000","mapping_epoch":1167,"log_start":"1173'4746","ondisk_log_start":"1173'4746","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":5890,"log_dups_size":0,"ondisk_log_size":5890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:34:37.889567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331481088,"num_objects":5058,"num_object_clones":0,"num_object_copies":15174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10636,"num_write_kb":502208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":83,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.233402+0000","last_change":"2024-09-13T14:45:19.813025+0000","last_active":"2024-09-13T14:51:54.233402+0000","last_peered":"2024-09-13T14:51:54.233402+0000","last_clean":"2024-09-13T14:51:54.233402+0000","last_became_active":"2024-09-13T14:45:19.812714+0000","last_became_peered":"2024-09-13T14:45:19.812714+0000","last_unstale":"2024-09-13T14:51:54.233402+0000","last_undegraded":"2024-09-13T14:51:54.233402+0000","last_fullsized":"2024-09-13T14:51:54.233402+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:40.671012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":139,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.245543+0000","last_change":"2024-09-13T14:51:19.906672+0000","last_active":"2024-09-13T14:51:54.245543+0000","last_peered":"2024-09-13T14:51:54.245543+0000","last_clean":"2024-09-13T14:51:54.245543+0000","last_became_active":"2024-09-13T14:51:19.906296+0000","last_became_peered":"2024-09-13T14:51:19.906296+0000","last_unstale":"2024-09-13T14:51:54.245543+0000","last_undegraded":"2024-09-13T14:51:54.245543+0000","last_fullsized":"2024-09-13T14:51:54.245543+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:55:31.046356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":1687,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.239644+0000","last_change":"2024-09-13T14:30:15.581728+0000","last_active":"2024-09-13T14:51:54.239644+0000","last_peered":"2024-09-13T14:51:54.239644+0000","last_clean":"2024-09-13T14:51:54.239644+0000","last_became_active":"2024-09-13T14:30:15.581303+0000","last_became_peered":"2024-09-13T14:30:15.581303+0000","last_unstale":"2024-09-13T14:51:54.239644+0000","last_undegraded":"2024-09-13T14:51:54.239644+0000","last_fullsized":"2024-09-13T14:51:54.239644+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:10.027991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"261'100","reported_seq":38,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.242385+0000","last_change":"2024-09-13T14:51:31.310325+0000","last_active":"2024-09-13T14:51:54.242385+0000","last_peered":"2024-09-13T14:51:54.242385+0000","last_clean":"2024-09-13T14:51:54.242385+0000","last_became_active":"2024-09-13T14:51:19.899471+0000","last_became_peered":"2024-09-13T14:51:19.899471+0000","last_unstale":"2024-09-13T14:51:54.242385+0000","last_undegraded":"2024-09-13T14:51:54.242385+0000","last_fullsized":"2024-09-13T14:51:54.242385+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:51:31.310115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:51:31.310115+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:34:41.135431+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.f","version":"1183'10775","reported_seq":10817,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.889456+0000","last_change":"2024-09-13T14:49:22.118734+0000","last_active":"2024-09-13T14:51:59.889456+0000","last_peered":"2024-09-13T14:51:59.889456+0000","last_clean":"2024-09-13T14:51:59.889456+0000","last_became_active":"2024-09-13T14:49:22.118438+0000","last_became_peered":"2024-09-13T14:49:22.118438+0000","last_unstale":"2024-09-13T14:51:59.889456+0000","last_undegraded":"2024-09-13T14:51:59.889456+0000","last_fullsized":"2024-09-13T14:51:59.889456+0000","mapping_epoch":1167,"log_start":"1173'5458","ondisk_log_start":"1173'5458","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":5317,"log_dups_size":0,"ondisk_log_size":5317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:27:01.810072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326696960,"num_objects":4985,"num_object_clones":0,"num_object_copies":14955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10775,"num_write_kb":504320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1727,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.242579+0000","last_change":"2024-09-13T14:28:35.611508+0000","last_active":"2024-09-13T14:51:54.242579+0000","last_peered":"2024-09-13T14:51:54.242579+0000","last_clean":"2024-09-13T14:51:54.242579+0000","last_became_active":"2024-09-13T14:28:35.610106+0000","last_became_peered":"2024-09-13T14:28:35.610106+0000","last_unstale":"2024-09-13T14:51:54.242579+0000","last_undegraded":"2024-09-13T14:51:54.242579+0000","last_fullsized":"2024-09-13T14:51:54.242579+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:11.966958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1159,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.186470+0000","last_change":"2024-09-13T14:51:19.907402+0000","last_active":"2024-09-13T14:51:46.186470+0000","last_peered":"2024-09-13T14:51:46.186470+0000","last_clean":"2024-09-13T14:51:46.186470+0000","last_became_active":"2024-09-13T14:51:19.904775+0000","last_became_peered":"2024-09-13T14:51:19.904775+0000","last_unstale":"2024-09-13T14:51:46.186470+0000","last_undegraded":"2024-09-13T14:51:46.186470+0000","last_fullsized":"2024-09-13T14:51:46.186470+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:47:13.974154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":132,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.244591+0000","last_change":"2024-09-13T14:43:07.522750+0000","last_active":"2024-09-13T14:51:54.244591+0000","last_peered":"2024-09-13T14:51:54.244591+0000","last_clean":"2024-09-13T14:51:54.244591+0000","last_became_active":"2024-09-13T14:43:07.517267+0000","last_became_peered":"2024-09-13T14:43:07.517267+0000","last_unstale":"2024-09-13T14:51:54.244591+0000","last_undegraded":"2024-09-13T14:51:54.244591+0000","last_fullsized":"2024-09-13T14:51:54.244591+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:40.217748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"261'100","reported_seq":452,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.248408+0000","last_change":"2024-09-13T14:45:19.815050+0000","last_active":"2024-09-13T14:51:54.248408+0000","last_peered":"2024-09-13T14:51:54.248408+0000","last_clean":"2024-09-13T14:51:54.248408+0000","last_became_active":"2024-09-13T14:45:19.809732+0000","last_became_peered":"2024-09-13T14:45:19.809732+0000","last_unstale":"2024-09-13T14:51:54.248408+0000","last_undegraded":"2024-09-13T14:51:54.248408+0000","last_fullsized":"2024-09-13T14:51:54.248408+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:33.574443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:33.574443+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:51:40.261036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"1183'10657","reported_seq":18370,"reported_epoch":1183,"state":"active+remapped+backfilling","last_fresh":"2024-09-13T14:51:59.958865+0000","last_change":"2024-09-13T14:51:21.089870+0000","last_active":"2024-09-13T14:51:59.958865+0000","last_peered":"2024-09-13T14:51:59.958865+0000","last_clean":"2024-09-13T14:51:19.053567+0000","last_became_active":"2024-09-13T14:51:20.994825+0000","last_became_peered":"2024-09-13T14:51:20.994825+0000","last_unstale":"2024-09-13T14:51:59.958865+0000","last_undegraded":"2024-09-13T14:51:59.958865+0000","last_fullsized":"2024-09-13T14:51:59.958865+0000","mapping_epoch":1178,"log_start":"1173'4751","ondisk_log_start":"1173'4751","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":5906,"log_dups_size":0,"ondisk_log_size":5906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325648384,"num_objects":4969,"num_object_clones":0,"num_object_copies":14907,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1796,"num_objects_unfound":0,"num_objects_dirty":4969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10657,"num_write_kb":500032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3828,"num_bytes_recovered":250806272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,4],"avail_no_missing":["0","2","4"],"object_location_counts":[{"shards":"0,2,4","objects":4969}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":139,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.247800+0000","last_change":"2024-09-13T14:51:19.910044+0000","last_active":"2024-09-13T14:51:54.247800+0000","last_peered":"2024-09-13T14:51:54.247800+0000","last_clean":"2024-09-13T14:51:54.247800+0000","last_became_active":"2024-09-13T14:51:19.908592+0000","last_became_peered":"2024-09-13T14:51:19.908592+0000","last_unstale":"2024-09-13T14:51:54.247800+0000","last_undegraded":"2024-09-13T14:51:54.247800+0000","last_fullsized":"2024-09-13T14:51:54.247800+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:16.467708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":132,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.244510+0000","last_change":"2024-09-13T14:43:07.525666+0000","last_active":"2024-09-13T14:51:54.244510+0000","last_peered":"2024-09-13T14:51:54.244510+0000","last_clean":"2024-09-13T14:51:54.244510+0000","last_became_active":"2024-09-13T14:43:07.524064+0000","last_became_peered":"2024-09-13T14:43:07.524064+0000","last_unstale":"2024-09-13T14:51:54.244510+0000","last_undegraded":"2024-09-13T14:51:54.244510+0000","last_fullsized":"2024-09-13T14:51:54.244510+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:10.270333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1163,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.191743+0000","last_change":"2024-09-13T14:45:19.804177+0000","last_active":"2024-09-13T14:51:46.191743+0000","last_peered":"2024-09-13T14:51:46.191743+0000","last_clean":"2024-09-13T14:51:46.191743+0000","last_became_active":"2024-09-13T14:45:19.803890+0000","last_became_peered":"2024-09-13T14:45:19.803890+0000","last_unstale":"2024-09-13T14:51:46.191743+0000","last_undegraded":"2024-09-13T14:51:46.191743+0000","last_fullsized":"2024-09-13T14:51:46.191743+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:25.140789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"261'100","reported_seq":100,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.239488+0000","last_change":"2024-09-13T14:45:36.965992+0000","last_active":"2024-09-13T14:51:54.239488+0000","last_peered":"2024-09-13T14:51:54.239488+0000","last_clean":"2024-09-13T14:51:54.239488+0000","last_became_active":"2024-09-13T14:45:19.811238+0000","last_became_peered":"2024-09-13T14:45:19.811238+0000","last_unstale":"2024-09-13T14:51:54.239488+0000","last_undegraded":"2024-09-13T14:51:54.239488+0000","last_fullsized":"2024-09-13T14:51:54.239488+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:36.965775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:45:36.965775+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:36:01.623251+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"1183'10691","reported_seq":10714,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:57.868305+0000","last_change":"2024-09-13T14:49:22.122277+0000","last_active":"2024-09-13T14:51:57.868305+0000","last_peered":"2024-09-13T14:51:57.868305+0000","last_clean":"2024-09-13T14:51:57.868305+0000","last_became_active":"2024-09-13T14:49:22.121980+0000","last_became_peered":"2024-09-13T14:49:22.121980+0000","last_unstale":"2024-09-13T14:51:57.868305+0000","last_undegraded":"2024-09-13T14:51:57.868305+0000","last_fullsized":"2024-09-13T14:51:57.868305+0000","mapping_epoch":1167,"log_start":"1172'4262","ondisk_log_start":"1172'4262","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":6429,"log_dups_size":0,"ondisk_log_size":6429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:58.275564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331939840,"num_objects":5065,"num_object_clones":0,"num_object_copies":15195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10691,"num_write_kb":504192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":62,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.195843+0000","last_change":"2024-09-13T14:45:19.806318+0000","last_active":"2024-09-13T14:51:46.195843+0000","last_peered":"2024-09-13T14:51:46.195843+0000","last_clean":"2024-09-13T14:51:46.195843+0000","last_became_active":"2024-09-13T14:45:19.806074+0000","last_became_peered":"2024-09-13T14:45:19.806074+0000","last_unstale":"2024-09-13T14:51:46.195843+0000","last_undegraded":"2024-09-13T14:51:46.195843+0000","last_fullsized":"2024-09-13T14:51:46.195843+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:21.679797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":496,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.242894+0000","last_change":"2024-09-13T14:51:19.910931+0000","last_active":"2024-09-13T14:51:54.242894+0000","last_peered":"2024-09-13T14:51:54.242894+0000","last_clean":"2024-09-13T14:51:54.242894+0000","last_became_active":"2024-09-13T14:51:19.910600+0000","last_became_peered":"2024-09-13T14:51:19.910600+0000","last_unstale":"2024-09-13T14:51:54.242894+0000","last_undegraded":"2024-09-13T14:51:54.242894+0000","last_fullsized":"2024-09-13T14:51:54.242894+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:57:45.304028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":1174,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.191767+0000","last_change":"2024-09-13T14:43:07.538919+0000","last_active":"2024-09-13T14:51:46.191767+0000","last_peered":"2024-09-13T14:51:46.191767+0000","last_clean":"2024-09-13T14:51:46.191767+0000","last_became_active":"2024-09-13T14:43:07.513109+0000","last_became_peered":"2024-09-13T14:43:07.513109+0000","last_unstale":"2024-09-13T14:51:46.191767+0000","last_undegraded":"2024-09-13T14:51:46.191767+0000","last_fullsized":"2024-09-13T14:51:46.191767+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:27.771720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"261'100","reported_seq":123,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.191382+0000","last_change":"2024-09-13T14:43:44.720333+0000","last_active":"2024-09-13T14:51:46.191382+0000","last_peered":"2024-09-13T14:51:46.191382+0000","last_clean":"2024-09-13T14:51:46.191382+0000","last_became_active":"2024-09-13T14:42:14.946591+0000","last_became_peered":"2024-09-13T14:42:14.946591+0000","last_unstale":"2024-09-13T14:51:46.191382+0000","last_undegraded":"2024-09-13T14:51:46.191382+0000","last_fullsized":"2024-09-13T14:51:46.191382+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:44.720258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:44.720258+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:33.281357+0000","scrub_duration":0.0018417920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"1183'10683","reported_seq":19435,"reported_epoch":1183,"state":"active+remapped+backfilling","last_fresh":"2024-09-13T14:51:59.871447+0000","last_change":"2024-09-13T14:51:46.277518+0000","last_active":"2024-09-13T14:51:59.871447+0000","last_peered":"2024-09-13T14:51:59.871447+0000","last_clean":"2024-09-13T14:51:19.032905+0000","last_became_active":"2024-09-13T14:51:20.995776+0000","last_became_peered":"2024-09-13T14:51:20.995776+0000","last_unstale":"2024-09-13T14:51:59.871447+0000","last_undegraded":"2024-09-13T14:51:59.871447+0000","last_fullsized":"2024-09-13T14:51:59.871447+0000","mapping_epoch":1178,"log_start":"1173'5336","ondisk_log_start":"1173'5336","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":5347,"log_dups_size":0,"ondisk_log_size":5347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326828032,"num_objects":4987,"num_object_clones":0,"num_object_copies":14961,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1391,"num_objects_unfound":0,"num_objects_dirty":4987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10683,"num_write_kb":501440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4352,"num_bytes_recovered":285212672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,4],"avail_no_missing":["3","1","4"],"object_location_counts":[{"shards":"1,3,4","objects":4987}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":99,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.244226+0000","last_change":"2024-09-13T14:44:03.858211+0000","last_active":"2024-09-13T14:51:54.244226+0000","last_peered":"2024-09-13T14:51:54.244226+0000","last_clean":"2024-09-13T14:51:54.244226+0000","last_became_active":"2024-09-13T14:44:03.857928+0000","last_became_peered":"2024-09-13T14:44:03.857928+0000","last_unstale":"2024-09-13T14:51:54.244226+0000","last_undegraded":"2024-09-13T14:51:54.244226+0000","last_fullsized":"2024-09-13T14:51:54.244226+0000","mapping_epoch":1140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:28.428693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":132,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.253243+0000","last_change":"2024-09-13T14:43:07.516243+0000","last_active":"2024-09-13T14:51:54.253243+0000","last_peered":"2024-09-13T14:51:54.253243+0000","last_clean":"2024-09-13T14:51:54.253243+0000","last_became_active":"2024-09-13T14:43:07.515863+0000","last_became_peered":"2024-09-13T14:43:07.515863+0000","last_unstale":"2024-09-13T14:51:54.253243+0000","last_undegraded":"2024-09-13T14:51:54.253243+0000","last_fullsized":"2024-09-13T14:51:54.253243+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:08.085298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1768,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.245202+0000","last_change":"2024-09-13T14:45:19.812713+0000","last_active":"2024-09-13T14:51:54.245202+0000","last_peered":"2024-09-13T14:51:54.245202+0000","last_clean":"2024-09-13T14:51:54.245202+0000","last_became_active":"2024-09-13T14:45:19.812203+0000","last_became_peered":"2024-09-13T14:45:19.812203+0000","last_unstale":"2024-09-13T14:51:54.245202+0000","last_undegraded":"2024-09-13T14:51:54.245202+0000","last_fullsized":"2024-09-13T14:51:54.245202+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:30:46.149347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"261'100","reported_seq":1313,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.185755+0000","last_change":"2024-09-13T14:51:19.895551+0000","last_active":"2024-09-13T14:51:46.185755+0000","last_peered":"2024-09-13T14:51:46.185755+0000","last_clean":"2024-09-13T14:51:46.185755+0000","last_became_active":"2024-09-13T14:51:19.895204+0000","last_became_peered":"2024-09-13T14:51:19.895204+0000","last_unstale":"2024-09-13T14:51:46.185755+0000","last_undegraded":"2024-09-13T14:51:46.185755+0000","last_fullsized":"2024-09-13T14:51:46.185755+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:42.577243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:42.577243+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:24.225386+0000","scrub_duration":0.001367839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"1183'10754","reported_seq":10797,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.326658+0000","last_change":"2024-09-13T14:49:22.118690+0000","last_active":"2024-09-13T14:51:58.326658+0000","last_peered":"2024-09-13T14:51:58.326658+0000","last_clean":"2024-09-13T14:51:58.326658+0000","last_became_active":"2024-09-13T14:49:22.117901+0000","last_became_peered":"2024-09-13T14:49:22.117901+0000","last_unstale":"2024-09-13T14:51:58.326658+0000","last_undegraded":"2024-09-13T14:51:58.326658+0000","last_fullsized":"2024-09-13T14:51:58.326658+0000","mapping_epoch":1167,"log_start":"1173'5426","ondisk_log_start":"1173'5426","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":5328,"log_dups_size":0,"ondisk_log_size":5328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:28.098913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338034688,"num_objects":5158,"num_object_clones":0,"num_object_copies":15474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10754,"num_write_kb":509184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1762,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.248492+0000","last_change":"2024-09-13T14:43:07.520212+0000","last_active":"2024-09-13T14:51:54.248492+0000","last_peered":"2024-09-13T14:51:54.248492+0000","last_clean":"2024-09-13T14:51:54.248492+0000","last_became_active":"2024-09-13T14:43:07.519831+0000","last_became_peered":"2024-09-13T14:43:07.519831+0000","last_unstale":"2024-09-13T14:51:54.248492+0000","last_undegraded":"2024-09-13T14:51:54.248492+0000","last_fullsized":"2024-09-13T14:51:54.248492+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:20.990736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1792,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.243844+0000","last_change":"2024-09-13T14:51:19.900868+0000","last_active":"2024-09-13T14:51:54.243844+0000","last_peered":"2024-09-13T14:51:54.243844+0000","last_clean":"2024-09-13T14:51:54.243844+0000","last_became_active":"2024-09-13T14:51:19.900481+0000","last_became_peered":"2024-09-13T14:51:19.900481+0000","last_unstale":"2024-09-13T14:51:54.243844+0000","last_undegraded":"2024-09-13T14:51:54.243844+0000","last_fullsized":"2024-09-13T14:51:54.243844+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:27.227954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1186,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.185832+0000","last_change":"2024-09-13T14:43:07.516140+0000","last_active":"2024-09-13T14:51:46.185832+0000","last_peered":"2024-09-13T14:51:46.185832+0000","last_clean":"2024-09-13T14:51:46.185832+0000","last_became_active":"2024-09-13T14:43:07.515481+0000","last_became_peered":"2024-09-13T14:43:07.515481+0000","last_unstale":"2024-09-13T14:51:46.185832+0000","last_undegraded":"2024-09-13T14:51:46.185832+0000","last_fullsized":"2024-09-13T14:51:46.185832+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:43.706214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"261'100","reported_seq":151,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.245301+0000","last_change":"2024-09-13T14:51:19.911422+0000","last_active":"2024-09-13T14:51:54.245301+0000","last_peered":"2024-09-13T14:51:54.245301+0000","last_clean":"2024-09-13T14:51:54.245301+0000","last_became_active":"2024-09-13T14:51:19.908591+0000","last_became_peered":"2024-09-13T14:51:19.908591+0000","last_unstale":"2024-09-13T14:51:54.245301+0000","last_undegraded":"2024-09-13T14:51:54.245301+0000","last_fullsized":"2024-09-13T14:51:54.245301+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.008264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.008264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:18:44.837688+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"1183'10378","reported_seq":10402,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:57.834140+0000","last_change":"2024-09-13T14:49:22.119570+0000","last_active":"2024-09-13T14:51:57.834140+0000","last_peered":"2024-09-13T14:51:57.834140+0000","last_clean":"2024-09-13T14:51:57.834140+0000","last_became_active":"2024-09-13T14:49:22.119224+0000","last_became_peered":"2024-09-13T14:49:22.119224+0000","last_unstale":"2024-09-13T14:51:57.834140+0000","last_undegraded":"2024-09-13T14:51:57.834140+0000","last_fullsized":"2024-09-13T14:51:57.834140+0000","mapping_epoch":1167,"log_start":"1172'3974","ondisk_log_start":"1172'3974","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":6404,"log_dups_size":0,"ondisk_log_size":6404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:43.554416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330956800,"num_objects":5050,"num_object_clones":0,"num_object_copies":15150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10378,"num_write_kb":493696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":23,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.186235+0000","last_change":"2024-09-13T14:51:19.906135+0000","last_active":"2024-09-13T14:51:46.186235+0000","last_peered":"2024-09-13T14:51:46.186235+0000","last_clean":"2024-09-13T14:51:46.186235+0000","last_became_active":"2024-09-13T14:51:19.904330+0000","last_became_peered":"2024-09-13T14:51:19.904330+0000","last_unstale":"2024-09-13T14:51:46.186235+0000","last_undegraded":"2024-09-13T14:51:46.186235+0000","last_fullsized":"2024-09-13T14:51:46.186235+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:53.407536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":83,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.232672+0000","last_change":"2024-09-13T14:45:19.810888+0000","last_active":"2024-09-13T14:51:54.232672+0000","last_peered":"2024-09-13T14:51:54.232672+0000","last_clean":"2024-09-13T14:51:54.232672+0000","last_became_active":"2024-09-13T14:45:19.810486+0000","last_became_peered":"2024-09-13T14:45:19.810486+0000","last_unstale":"2024-09-13T14:51:54.232672+0000","last_undegraded":"2024-09-13T14:51:54.232672+0000","last_fullsized":"2024-09-13T14:51:54.232672+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:03.886942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":1139,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.191933+0000","last_change":"2024-09-13T14:29:42.539455+0000","last_active":"2024-09-13T14:51:46.191933+0000","last_peered":"2024-09-13T14:51:46.191933+0000","last_clean":"2024-09-13T14:51:46.191933+0000","last_became_active":"2024-09-13T14:29:42.539068+0000","last_became_peered":"2024-09-13T14:29:42.539068+0000","last_unstale":"2024-09-13T14:51:46.191933+0000","last_undegraded":"2024-09-13T14:51:46.191933+0000","last_fullsized":"2024-09-13T14:51:46.191933+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:24:06.048855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034405999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"261'100","reported_seq":1294,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.185855+0000","last_change":"2024-09-13T14:44:28.186029+0000","last_active":"2024-09-13T14:51:46.185855+0000","last_peered":"2024-09-13T14:51:46.185855+0000","last_clean":"2024-09-13T14:51:46.185855+0000","last_became_active":"2024-09-13T14:42:14.927771+0000","last_became_peered":"2024-09-13T14:42:14.927771+0000","last_unstale":"2024-09-13T14:51:46.185855+0000","last_undegraded":"2024-09-13T14:51:46.185855+0000","last_fullsized":"2024-09-13T14:51:46.185855+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:28.185949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:28.185949+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:38.363961+0000","scrub_duration":0.0018071129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3","version":"1183'10442","reported_seq":10485,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.343462+0000","last_change":"2024-09-13T14:49:22.119776+0000","last_active":"2024-09-13T14:51:58.343462+0000","last_peered":"2024-09-13T14:51:58.343462+0000","last_clean":"2024-09-13T14:51:58.343462+0000","last_became_active":"2024-09-13T14:49:22.119517+0000","last_became_peered":"2024-09-13T14:49:22.119517+0000","last_unstale":"2024-09-13T14:51:58.343462+0000","last_undegraded":"2024-09-13T14:51:58.343462+0000","last_fullsized":"2024-09-13T14:51:58.343462+0000","mapping_epoch":1167,"log_start":"1173'5107","ondisk_log_start":"1173'5107","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":5335,"log_dups_size":0,"ondisk_log_size":5335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:53.152882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333709312,"num_objects":5092,"num_object_clones":0,"num_object_copies":15276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10442,"num_write_kb":497088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":138,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.249463+0000","last_change":"2024-09-13T14:51:19.907839+0000","last_active":"2024-09-13T14:51:54.249463+0000","last_peered":"2024-09-13T14:51:54.249463+0000","last_clean":"2024-09-13T14:51:54.249463+0000","last_became_active":"2024-09-13T14:51:19.907418+0000","last_became_peered":"2024-09-13T14:51:19.907418+0000","last_unstale":"2024-09-13T14:51:54.249463+0000","last_undegraded":"2024-09-13T14:51:54.249463+0000","last_fullsized":"2024-09-13T14:51:54.249463+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:43:28.088269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":83,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.233499+0000","last_change":"2024-09-13T14:45:19.812353+0000","last_active":"2024-09-13T14:51:54.233499+0000","last_peered":"2024-09-13T14:51:54.233499+0000","last_clean":"2024-09-13T14:51:54.233499+0000","last_became_active":"2024-09-13T14:45:19.811743+0000","last_became_peered":"2024-09-13T14:45:19.811743+0000","last_unstale":"2024-09-13T14:51:54.233499+0000","last_undegraded":"2024-09-13T14:51:54.233499+0000","last_fullsized":"2024-09-13T14:51:54.233499+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:43.728170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"261'100","reported_seq":123,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.191524+0000","last_change":"2024-09-13T14:44:34.814157+0000","last_active":"2024-09-13T14:51:46.191524+0000","last_peered":"2024-09-13T14:51:46.191524+0000","last_clean":"2024-09-13T14:51:46.191524+0000","last_became_active":"2024-09-13T14:42:42.707803+0000","last_became_peered":"2024-09-13T14:42:42.707803+0000","last_unstale":"2024-09-13T14:51:46.191524+0000","last_undegraded":"2024-09-13T14:51:46.191524+0000","last_fullsized":"2024-09-13T14:51:46.191524+0000","mapping_epoch":1099,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:34.814070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:34.814070+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:38.408561+0000","scrub_duration":0.01686414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"1183'10679","reported_seq":19114,"reported_epoch":1183,"state":"active+remapped+backfilling","last_fresh":"2024-09-13T14:51:59.950621+0000","last_change":"2024-09-13T14:51:21.017992+0000","last_active":"2024-09-13T14:51:59.950621+0000","last_peered":"2024-09-13T14:51:59.950621+0000","last_clean":"2024-09-13T14:51:19.033262+0000","last_became_active":"2024-09-13T14:51:20.993124+0000","last_became_peered":"2024-09-13T14:51:20.993124+0000","last_unstale":"2024-09-13T14:51:59.950621+0000","last_undegraded":"2024-09-13T14:51:59.950621+0000","last_fullsized":"2024-09-13T14:51:59.950621+0000","mapping_epoch":1178,"log_start":"1173'4786","ondisk_log_start":"1173'4786","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":5893,"log_dups_size":0,"ondisk_log_size":5893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322240512,"num_objects":4917,"num_object_clones":0,"num_object_copies":14751,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1465,"num_objects_unfound":0,"num_objects_dirty":4917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10679,"num_write_kb":499072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4199,"num_bytes_recovered":275185664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,4],"avail_no_missing":["0","4","5"],"object_location_counts":[{"shards":"0,4,5","objects":4917}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1125,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.244479+0000","last_change":"2024-09-13T14:51:19.895343+0000","last_active":"2024-09-13T14:51:54.244479+0000","last_peered":"2024-09-13T14:51:54.244479+0000","last_clean":"2024-09-13T14:51:54.244479+0000","last_became_active":"2024-09-13T14:51:19.895036+0000","last_became_peered":"2024-09-13T14:51:19.895036+0000","last_unstale":"2024-09-13T14:51:54.244479+0000","last_undegraded":"2024-09-13T14:51:54.244479+0000","last_fullsized":"2024-09-13T14:51:54.244479+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:40.018071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":560,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.245077+0000","last_change":"2024-09-13T14:51:19.905106+0000","last_active":"2024-09-13T14:51:54.245077+0000","last_peered":"2024-09-13T14:51:54.245077+0000","last_clean":"2024-09-13T14:51:54.245077+0000","last_became_active":"2024-09-13T14:51:19.904682+0000","last_became_peered":"2024-09-13T14:51:19.904682+0000","last_unstale":"2024-09-13T14:51:54.245077+0000","last_undegraded":"2024-09-13T14:51:54.245077+0000","last_fullsized":"2024-09-13T14:51:54.245077+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:17:23.764784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"261'100","reported_seq":149,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.245019+0000","last_change":"2024-09-13T14:43:18.988482+0000","last_active":"2024-09-13T14:51:54.245019+0000","last_peered":"2024-09-13T14:51:54.245019+0000","last_clean":"2024-09-13T14:51:54.245019+0000","last_became_active":"2024-09-13T14:43:14.556754+0000","last_became_peered":"2024-09-13T14:43:14.556754+0000","last_unstale":"2024-09-13T14:51:54.245019+0000","last_undegraded":"2024-09-13T14:51:54.245019+0000","last_fullsized":"2024-09-13T14:51:54.245019+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:18.988405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:18.988405+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:45:01.726326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"1183'10553","reported_seq":10597,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.294186+0000","last_change":"2024-09-13T14:49:22.121230+0000","last_active":"2024-09-13T14:51:58.294186+0000","last_peered":"2024-09-13T14:51:58.294186+0000","last_clean":"2024-09-13T14:51:58.294186+0000","last_became_active":"2024-09-13T14:49:22.120836+0000","last_became_peered":"2024-09-13T14:49:22.120836+0000","last_unstale":"2024-09-13T14:51:58.294186+0000","last_undegraded":"2024-09-13T14:51:58.294186+0000","last_fullsized":"2024-09-13T14:51:58.294186+0000","mapping_epoch":1167,"log_start":"1173'5203","ondisk_log_start":"1173'5203","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":5350,"log_dups_size":0,"ondisk_log_size":5350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:21.843404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334430208,"num_objects":5103,"num_object_clones":0,"num_object_copies":15309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10553,"num_write_kb":500992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1126,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.242533+0000","last_change":"2024-09-13T14:51:19.898670+0000","last_active":"2024-09-13T14:51:54.242533+0000","last_peered":"2024-09-13T14:51:54.242533+0000","last_clean":"2024-09-13T14:51:54.242533+0000","last_became_active":"2024-09-13T14:51:19.898316+0000","last_became_peered":"2024-09-13T14:51:19.898316+0000","last_unstale":"2024-09-13T14:51:54.242533+0000","last_undegraded":"2024-09-13T14:51:54.242533+0000","last_fullsized":"2024-09-13T14:51:54.242533+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:53:27.567679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1100,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.247576+0000","last_change":"2024-09-13T14:51:19.909652+0000","last_active":"2024-09-13T14:51:54.247576+0000","last_peered":"2024-09-13T14:51:54.247576+0000","last_clean":"2024-09-13T14:51:54.247576+0000","last_became_active":"2024-09-13T14:51:19.907411+0000","last_became_peered":"2024-09-13T14:51:19.907411+0000","last_unstale":"2024-09-13T14:51:54.247576+0000","last_undegraded":"2024-09-13T14:51:54.247576+0000","last_fullsized":"2024-09-13T14:51:54.247576+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:14:38.272707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"261'100","reported_seq":128,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.191346+0000","last_change":"2024-09-13T14:45:19.805634+0000","last_active":"2024-09-13T14:51:46.191346+0000","last_peered":"2024-09-13T14:51:46.191346+0000","last_clean":"2024-09-13T14:51:46.191346+0000","last_became_active":"2024-09-13T14:45:19.805046+0000","last_became_peered":"2024-09-13T14:45:19.805046+0000","last_unstale":"2024-09-13T14:51:46.191346+0000","last_undegraded":"2024-09-13T14:51:46.191346+0000","last_fullsized":"2024-09-13T14:51:46.191346+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:32.752408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:32.752408+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:06:53.140239+0000","scrub_duration":0.0057731559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.0","version":"1183'10472","reported_seq":10516,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.855810+0000","last_change":"2024-09-13T14:49:22.118730+0000","last_active":"2024-09-13T14:51:59.855810+0000","last_peered":"2024-09-13T14:51:59.855810+0000","last_clean":"2024-09-13T14:51:59.855810+0000","last_became_active":"2024-09-13T14:49:22.118428+0000","last_became_peered":"2024-09-13T14:49:22.118428+0000","last_unstale":"2024-09-13T14:51:59.855810+0000","last_undegraded":"2024-09-13T14:51:59.855810+0000","last_fullsized":"2024-09-13T14:51:59.855810+0000","mapping_epoch":1167,"log_start":"1173'5182","ondisk_log_start":"1173'5182","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":5290,"log_dups_size":0,"ondisk_log_size":5290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:54.951267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322699264,"num_objects":4924,"num_object_clones":0,"num_object_copies":14772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10472,"num_write_kb":492672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1153,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.233052+0000","last_change":"2024-09-13T14:51:19.900208+0000","last_active":"2024-09-13T14:51:54.233052+0000","last_peered":"2024-09-13T14:51:54.233052+0000","last_clean":"2024-09-13T14:51:54.233052+0000","last_became_active":"2024-09-13T14:51:19.899741+0000","last_became_peered":"2024-09-13T14:51:19.899741+0000","last_unstale":"2024-09-13T14:51:54.233052+0000","last_undegraded":"2024-09-13T14:51:54.233052+0000","last_fullsized":"2024-09-13T14:51:54.233052+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:31.828483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":97,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.241789+0000","last_change":"2024-09-13T14:51:19.929437+0000","last_active":"2024-09-13T14:51:54.241789+0000","last_peered":"2024-09-13T14:51:54.241789+0000","last_clean":"2024-09-13T14:51:54.241789+0000","last_became_active":"2024-09-13T14:51:19.911220+0000","last_became_peered":"2024-09-13T14:51:19.911220+0000","last_unstale":"2024-09-13T14:51:54.241789+0000","last_undegraded":"2024-09-13T14:51:54.241789+0000","last_fullsized":"2024-09-13T14:51:54.241789+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:14.405956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"261'100","reported_seq":1313,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.185721+0000","last_change":"2024-09-13T14:45:19.798553+0000","last_active":"2024-09-13T14:51:46.185721+0000","last_peered":"2024-09-13T14:51:46.185721+0000","last_clean":"2024-09-13T14:51:46.185721+0000","last_became_active":"2024-09-13T14:45:19.798189+0000","last_became_peered":"2024-09-13T14:45:19.798189+0000","last_unstale":"2024-09-13T14:51:46.185721+0000","last_undegraded":"2024-09-13T14:51:46.185721+0000","last_fullsized":"2024-09-13T14:51:46.185721+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:29.169325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:29.169325+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:10.412666+0000","scrub_duration":0.0019358890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"1183'10440","reported_seq":10484,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.276695+0000","last_change":"2024-09-13T14:49:22.120585+0000","last_active":"2024-09-13T14:51:58.276695+0000","last_peered":"2024-09-13T14:51:58.276695+0000","last_clean":"2024-09-13T14:51:58.276695+0000","last_became_active":"2024-09-13T14:49:22.120301+0000","last_became_peered":"2024-09-13T14:49:22.120301+0000","last_unstale":"2024-09-13T14:51:58.276695+0000","last_undegraded":"2024-09-13T14:51:58.276695+0000","last_fullsized":"2024-09-13T14:51:58.276695+0000","mapping_epoch":1167,"log_start":"1173'5142","ondisk_log_start":"1173'5142","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":5298,"log_dups_size":0,"ondisk_log_size":5298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:20:25.183879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680000,"num_objects":5000,"num_object_clones":0,"num_object_copies":15000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10440,"num_write_kb":494080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1758,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.242487+0000","last_change":"2024-09-13T14:51:19.905621+0000","last_active":"2024-09-13T14:51:54.242487+0000","last_peered":"2024-09-13T14:51:54.242487+0000","last_clean":"2024-09-13T14:51:54.242487+0000","last_became_active":"2024-09-13T14:51:19.905274+0000","last_became_peered":"2024-09-13T14:51:19.905274+0000","last_unstale":"2024-09-13T14:51:54.242487+0000","last_undegraded":"2024-09-13T14:51:54.242487+0000","last_fullsized":"2024-09-13T14:51:54.242487+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:57.364606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1166,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.195993+0000","last_change":"2024-09-13T14:45:19.804982+0000","last_active":"2024-09-13T14:51:46.195993+0000","last_peered":"2024-09-13T14:51:46.195993+0000","last_clean":"2024-09-13T14:51:46.195993+0000","last_became_active":"2024-09-13T14:45:19.804663+0000","last_became_peered":"2024-09-13T14:45:19.804663+0000","last_unstale":"2024-09-13T14:51:46.195993+0000","last_undegraded":"2024-09-13T14:51:46.195993+0000","last_fullsized":"2024-09-13T14:51:46.195993+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:43.211479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"261'100","reported_seq":345,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.186479+0000","last_change":"2024-09-13T14:51:19.895244+0000","last_active":"2024-09-13T14:51:46.186479+0000","last_peered":"2024-09-13T14:51:46.186479+0000","last_clean":"2024-09-13T14:51:46.186479+0000","last_became_active":"2024-09-13T14:51:19.894667+0000","last_became_peered":"2024-09-13T14:51:19.894667+0000","last_unstale":"2024-09-13T14:51:46.186479+0000","last_undegraded":"2024-09-13T14:51:46.186479+0000","last_fullsized":"2024-09-13T14:51:46.186479+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:42.881520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:42.881520+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13417,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":17,"num_write":3,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":13417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.6","version":"1183'10336","reported_seq":10380,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.914927+0000","last_change":"2024-09-13T14:49:22.117570+0000","last_active":"2024-09-13T14:51:59.914927+0000","last_peered":"2024-09-13T14:51:59.914927+0000","last_clean":"2024-09-13T14:51:59.914927+0000","last_became_active":"2024-09-13T14:49:22.117259+0000","last_became_peered":"2024-09-13T14:49:22.117259+0000","last_unstale":"2024-09-13T14:51:59.914927+0000","last_undegraded":"2024-09-13T14:51:59.914927+0000","last_fullsized":"2024-09-13T14:51:59.914927+0000","mapping_epoch":1167,"log_start":"1173'4370","ondisk_log_start":"1173'4370","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":5966,"log_dups_size":0,"ondisk_log_size":5966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:59:23.669062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320471040,"num_objects":4890,"num_object_clones":0,"num_object_copies":14670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10336,"num_write_kb":487232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1139,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.190302+0000","last_change":"2024-09-13T14:43:07.518120+0000","last_active":"2024-09-13T14:51:46.190302+0000","last_peered":"2024-09-13T14:51:46.190302+0000","last_clean":"2024-09-13T14:51:46.190302+0000","last_became_active":"2024-09-13T14:43:07.517361+0000","last_became_peered":"2024-09-13T14:43:07.517361+0000","last_unstale":"2024-09-13T14:51:46.190302+0000","last_undegraded":"2024-09-13T14:51:46.190302+0000","last_fullsized":"2024-09-13T14:51:46.190302+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:39.366473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":1121,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.186310+0000","last_change":"2024-09-13T14:19:25.871006+0000","last_active":"2024-09-13T14:51:46.186310+0000","last_peered":"2024-09-13T14:51:46.186310+0000","last_clean":"2024-09-13T14:51:46.186310+0000","last_became_active":"2024-09-13T14:19:25.870713+0000","last_became_peered":"2024-09-13T14:19:25.870713+0000","last_unstale":"2024-09-13T14:51:46.186310+0000","last_undegraded":"2024-09-13T14:51:46.186310+0000","last_fullsized":"2024-09-13T14:51:46.186310+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:34.372124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"261'100","reported_seq":407,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.191937+0000","last_change":"2024-09-13T14:51:19.899481+0000","last_active":"2024-09-13T14:51:46.191937+0000","last_peered":"2024-09-13T14:51:46.191937+0000","last_clean":"2024-09-13T14:51:46.191937+0000","last_became_active":"2024-09-13T14:51:19.898919+0000","last_became_peered":"2024-09-13T14:51:19.898919+0000","last_unstale":"2024-09-13T14:51:46.191937+0000","last_undegraded":"2024-09-13T14:51:46.191937+0000","last_fullsized":"2024-09-13T14:51:46.191937+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:49:21.722939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.722939+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:22.739555+0000","scrub_duration":0.0016642790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.5","version":"1183'10622","reported_seq":20290,"reported_epoch":1183,"state":"active+remapped+backfilling","last_fresh":"2024-09-13T14:51:57.849899+0000","last_change":"2024-09-13T14:51:21.060776+0000","last_active":"2024-09-13T14:51:57.849899+0000","last_peered":"2024-09-13T14:51:57.849899+0000","last_clean":"2024-09-13T14:51:19.009276+0000","last_became_active":"2024-09-13T14:51:20.994052+0000","last_became_peered":"2024-09-13T14:51:20.994052+0000","last_unstale":"2024-09-13T14:51:57.849899+0000","last_undegraded":"2024-09-13T14:51:57.849899+0000","last_fullsized":"2024-09-13T14:51:57.849899+0000","mapping_epoch":1178,"log_start":"1173'4942","ondisk_log_start":"1173'4942","created":1167,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":5680,"log_dups_size":0,"ondisk_log_size":5680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:10:42.329625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335282176,"num_objects":5116,"num_object_clones":0,"num_object_copies":15348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1026,"num_objects_unfound":0,"num_objects_dirty":5116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10622,"num_write_kb":503616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4814,"num_bytes_recovered":315424768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,4],"avail_no_missing":["6","4","7"],"object_location_counts":[{"shards":"4,6,7","objects":5116}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1170,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.243688+0000","last_change":"2024-09-13T14:51:19.900788+0000","last_active":"2024-09-13T14:51:54.243688+0000","last_peered":"2024-09-13T14:51:54.243688+0000","last_clean":"2024-09-13T14:51:54.243688+0000","last_became_active":"2024-09-13T14:51:19.900220+0000","last_became_peered":"2024-09-13T14:51:19.900220+0000","last_unstale":"2024-09-13T14:51:54.243688+0000","last_undegraded":"2024-09-13T14:51:54.243688+0000","last_fullsized":"2024-09-13T14:51:54.243688+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:38.189771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":1181,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.186712+0000","last_change":"2024-09-13T14:51:19.944221+0000","last_active":"2024-09-13T14:51:46.186712+0000","last_peered":"2024-09-13T14:51:46.186712+0000","last_clean":"2024-09-13T14:51:46.186712+0000","last_became_active":"2024-09-13T14:51:19.897813+0000","last_became_peered":"2024-09-13T14:51:19.897813+0000","last_unstale":"2024-09-13T14:51:46.186712+0000","last_undegraded":"2024-09-13T14:51:46.186712+0000","last_fullsized":"2024-09-13T14:51:46.186712+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:46:40.882318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"261'100","reported_seq":86,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.233178+0000","last_change":"2024-09-13T14:45:19.821649+0000","last_active":"2024-09-13T14:51:54.233178+0000","last_peered":"2024-09-13T14:51:54.233178+0000","last_clean":"2024-09-13T14:51:54.233178+0000","last_became_active":"2024-09-13T14:45:19.812346+0000","last_became_peered":"2024-09-13T14:45:19.812346+0000","last_unstale":"2024-09-13T14:51:54.233178+0000","last_undegraded":"2024-09-13T14:51:54.233178+0000","last_fullsized":"2024-09-13T14:51:54.233178+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.597568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.597568+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:45.218892+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"1183'10405","reported_seq":23091,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:57.858188+0000","last_change":"2024-09-13T14:51:46.274974+0000","last_active":"2024-09-13T14:51:57.858188+0000","last_peered":"2024-09-13T14:51:57.858188+0000","last_clean":"2024-09-13T14:51:57.858188+0000","last_became_active":"2024-09-13T14:51:46.204915+0000","last_became_peered":"2024-09-13T14:51:46.204915+0000","last_unstale":"2024-09-13T14:51:57.858188+0000","last_undegraded":"2024-09-13T14:51:57.858188+0000","last_fullsized":"2024-09-13T14:51:57.858188+0000","mapping_epoch":1181,"log_start":"1172'3940","ondisk_log_start":"1172'3940","created":1167,"last_epoch_clean":1182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:49:21.089133+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.089133+0000","objects_scrubbed":0,"log_size":6465,"log_dups_size":0,"ondisk_log_size":6465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:26.197436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326959104,"num_objects":4989,"num_object_clones":0,"num_object_copies":14967,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10405,"num_write_kb":492608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6317,"num_bytes_recovered":413925376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1108,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.191476+0000","last_change":"2024-09-13T14:19:26.887839+0000","last_active":"2024-09-13T14:51:46.191476+0000","last_peered":"2024-09-13T14:51:46.191476+0000","last_clean":"2024-09-13T14:51:46.191476+0000","last_became_active":"2024-09-13T14:19:26.887498+0000","last_became_peered":"2024-09-13T14:19:26.887498+0000","last_unstale":"2024-09-13T14:51:46.191476+0000","last_undegraded":"2024-09-13T14:51:46.191476+0000","last_fullsized":"2024-09-13T14:51:46.191476+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:24.475263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":1152,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.186427+0000","last_change":"2024-09-13T14:51:19.948447+0000","last_active":"2024-09-13T14:51:46.186427+0000","last_peered":"2024-09-13T14:51:46.186427+0000","last_clean":"2024-09-13T14:51:46.186427+0000","last_became_active":"2024-09-13T14:51:19.908698+0000","last_became_peered":"2024-09-13T14:51:19.908698+0000","last_unstale":"2024-09-13T14:51:46.186427+0000","last_undegraded":"2024-09-13T14:51:46.186427+0000","last_fullsized":"2024-09-13T14:51:46.186427+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:02:56.829843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"261'100","reported_seq":84,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.186134+0000","last_change":"2024-09-13T14:51:19.899763+0000","last_active":"2024-09-13T14:51:46.186134+0000","last_peered":"2024-09-13T14:51:46.186134+0000","last_clean":"2024-09-13T14:51:46.186134+0000","last_became_active":"2024-09-13T14:51:19.899012+0000","last_became_peered":"2024-09-13T14:51:19.899012+0000","last_unstale":"2024-09-13T14:51:46.186134+0000","last_undegraded":"2024-09-13T14:51:46.186134+0000","last_fullsized":"2024-09-13T14:51:46.186134+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.191326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.191326+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:09.732098+0000","scrub_duration":0.0025317360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1128,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.189715+0000","last_change":"2024-09-13T14:28:35.603724+0000","last_active":"2024-09-13T14:51:46.189715+0000","last_peered":"2024-09-13T14:51:46.189715+0000","last_clean":"2024-09-13T14:51:46.189715+0000","last_became_active":"2024-09-13T14:28:35.603316+0000","last_became_peered":"2024-09-13T14:28:35.603316+0000","last_unstale":"2024-09-13T14:51:46.189715+0000","last_undegraded":"2024-09-13T14:51:46.189715+0000","last_fullsized":"2024-09-13T14:51:46.189715+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:47.668223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":82,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.239344+0000","last_change":"2024-09-13T14:45:19.811466+0000","last_active":"2024-09-13T14:51:54.239344+0000","last_peered":"2024-09-13T14:51:54.239344+0000","last_clean":"2024-09-13T14:51:54.239344+0000","last_became_active":"2024-09-13T14:45:19.810617+0000","last_became_peered":"2024-09-13T14:45:19.810617+0000","last_unstale":"2024-09-13T14:51:54.239344+0000","last_undegraded":"2024-09-13T14:51:54.239344+0000","last_fullsized":"2024-09-13T14:51:54.239344+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:16.246669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"261'100","reported_seq":74,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.191664+0000","last_change":"2024-09-13T14:44:50.805306+0000","last_active":"2024-09-13T14:51:46.191664+0000","last_peered":"2024-09-13T14:51:46.191664+0000","last_clean":"2024-09-13T14:51:46.191664+0000","last_became_active":"2024-09-13T14:43:14.562922+0000","last_became_peered":"2024-09-13T14:43:14.562922+0000","last_unstale":"2024-09-13T14:51:46.191664+0000","last_undegraded":"2024-09-13T14:51:46.191664+0000","last_fullsized":"2024-09-13T14:51:46.191664+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:50.805210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:50.805210+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:28:38.529599+0000","scrub_duration":0.0042900009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1130,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.185816+0000","last_change":"2024-09-13T14:51:19.892512+0000","last_active":"2024-09-13T14:51:46.185816+0000","last_peered":"2024-09-13T14:51:46.185816+0000","last_clean":"2024-09-13T14:51:46.185816+0000","last_became_active":"2024-09-13T14:51:19.892215+0000","last_became_peered":"2024-09-13T14:51:19.892215+0000","last_unstale":"2024-09-13T14:51:46.185816+0000","last_undegraded":"2024-09-13T14:51:46.185816+0000","last_fullsized":"2024-09-13T14:51:46.185816+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:51.369749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1750,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.248696+0000","last_change":"2024-09-13T14:45:19.813876+0000","last_active":"2024-09-13T14:51:54.248696+0000","last_peered":"2024-09-13T14:51:54.248696+0000","last_clean":"2024-09-13T14:51:54.248696+0000","last_became_active":"2024-09-13T14:45:19.809136+0000","last_became_peered":"2024-09-13T14:45:19.809136+0000","last_unstale":"2024-09-13T14:51:54.248696+0000","last_undegraded":"2024-09-13T14:51:54.248696+0000","last_fullsized":"2024-09-13T14:51:54.248696+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:50.013528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"261'100","reported_seq":163,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.245164+0000","last_change":"2024-09-13T14:51:19.912342+0000","last_active":"2024-09-13T14:51:54.245164+0000","last_peered":"2024-09-13T14:51:54.245164+0000","last_clean":"2024-09-13T14:51:54.245164+0000","last_became_active":"2024-09-13T14:51:19.909735+0000","last_became_peered":"2024-09-13T14:51:19.909735+0000","last_unstale":"2024-09-13T14:51:54.245164+0000","last_undegraded":"2024-09-13T14:51:54.245164+0000","last_fullsized":"2024-09-13T14:51:54.245164+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:19.978464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:19.978464+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:14.417715+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1798,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.244826+0000","last_change":"2024-09-13T14:45:19.809166+0000","last_active":"2024-09-13T14:51:54.244826+0000","last_peered":"2024-09-13T14:51:54.244826+0000","last_clean":"2024-09-13T14:51:54.244826+0000","last_became_active":"2024-09-13T14:45:19.808631+0000","last_became_peered":"2024-09-13T14:45:19.808631+0000","last_unstale":"2024-09-13T14:51:54.244826+0000","last_undegraded":"2024-09-13T14:51:54.244826+0000","last_fullsized":"2024-09-13T14:51:54.244826+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:38.822245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1145,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.191684+0000","last_change":"2024-09-13T14:45:19.804229+0000","last_active":"2024-09-13T14:51:46.191684+0000","last_peered":"2024-09-13T14:51:46.191684+0000","last_clean":"2024-09-13T14:51:46.191684+0000","last_became_active":"2024-09-13T14:45:19.803929+0000","last_became_peered":"2024-09-13T14:45:19.803929+0000","last_unstale":"2024-09-13T14:51:46.191684+0000","last_undegraded":"2024-09-13T14:51:46.191684+0000","last_fullsized":"2024-09-13T14:51:46.191684+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:33:53.675568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"261'100","reported_seq":74,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.191326+0000","last_change":"2024-09-13T14:44:42.654506+0000","last_active":"2024-09-13T14:51:46.191326+0000","last_peered":"2024-09-13T14:51:46.191326+0000","last_clean":"2024-09-13T14:51:46.191326+0000","last_became_active":"2024-09-13T14:43:14.563451+0000","last_became_peered":"2024-09-13T14:43:14.563451+0000","last_unstale":"2024-09-13T14:51:46.191326+0000","last_undegraded":"2024-09-13T14:51:46.191326+0000","last_fullsized":"2024-09-13T14:51:46.191326+0000","mapping_epoch":1128,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1129,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:42.654455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:42.654455+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:57:32.251597+0000","scrub_duration":0.001494097,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":1155,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.186183+0000","last_change":"2024-09-13T14:51:19.902458+0000","last_active":"2024-09-13T14:51:46.186183+0000","last_peered":"2024-09-13T14:51:46.186183+0000","last_clean":"2024-09-13T14:51:46.186183+0000","last_became_active":"2024-09-13T14:51:19.892094+0000","last_became_peered":"2024-09-13T14:51:19.892094+0000","last_unstale":"2024-09-13T14:51:46.186183+0000","last_undegraded":"2024-09-13T14:51:46.186183+0000","last_fullsized":"2024-09-13T14:51:46.186183+0000","mapping_epoch":1177,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:03:35.273889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1689,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.233498+0000","last_change":"2024-09-13T14:43:07.518679+0000","last_active":"2024-09-13T14:51:54.233498+0000","last_peered":"2024-09-13T14:51:54.233498+0000","last_clean":"2024-09-13T14:51:54.233498+0000","last_became_active":"2024-09-13T14:43:07.518333+0000","last_became_peered":"2024-09-13T14:43:07.518333+0000","last_unstale":"2024-09-13T14:51:54.233498+0000","last_undegraded":"2024-09-13T14:51:54.233498+0000","last_fullsized":"2024-09-13T14:51:54.233498+0000","mapping_epoch":1124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:20.616274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"261'100","reported_seq":458,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-13T14:51:54.243936+0000","last_change":"2024-09-13T14:45:19.809268+0000","last_active":"2024-09-13T14:51:54.243936+0000","last_peered":"2024-09-13T14:51:54.243936+0000","last_clean":"2024-09-13T14:51:54.243936+0000","last_became_active":"2024-09-13T14:45:19.808772+0000","last_became_peered":"2024-09-13T14:45:19.808772+0000","last_unstale":"2024-09-13T14:51:54.243936+0000","last_undegraded":"2024-09-13T14:51:54.243936+0000","last_fullsized":"2024-09-13T14:51:54.243936+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:14.654603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:14.654603+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:40.964553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":5271388184,"num_objects":80436,"num_object_clones":0,"num_object_copies":241308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5678,"num_objects_unfound":0,"num_objects_dirty":80436,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169296,"num_write_kb":7991361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23510,"num_bytes_recovered":1540554752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17200984064,"data_stored":17200971848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92365,"ondisk_log_size":92365,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":4890,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":21480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":630620,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":51,"num_write":10,"num_write_kb":111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":367932,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430623,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2439260,"kb_used_data":1853460,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":91305892,"statfs":{"total":95995035648,"available":93497233408,"internally_reserved":0,"allocated":1897943040,"data_stored":1894252303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626852,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3002196,"kb_used_data":2099464,"kb_used_omap":1,"kb_used_meta":902718,"kb_avail":90742956,"statfs":{"total":95995035648,"available":92920786944,"internally_reserved":0,"allocated":2149851136,"data_stored":2146172034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":924383576},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823081,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3214412,"kb_used_data":2628548,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":90530740,"statfs":{"total":95995035648,"available":92703477760,"internally_reserved":0,"allocated":2691633152,"data_stored":2687954498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019309,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059592967,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2507068,"kb_used_data":2150708,"kb_used_omap":26,"kb_used_meta":356261,"kb_avail":91238084,"statfs":{"total":95995035648,"available":93427798016,"internally_reserved":0,"allocated":2202324992,"data_stored":2198661605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":364811424},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":591,"seq":2538325672205,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2788308,"kb_used_data":2222540,"kb_used_omap":41,"kb_used_meta":565718,"kb_avail":90956844,"statfs":{"total":95995035648,"available":93139808256,"internally_reserved":0,"allocated":2275880960,"data_stored":2272198831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42651,"internal_metadata":579295589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":557,"seq":2392296784148,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2820800,"kb_used_data":2291128,"kb_used_omap":32,"kb_used_meta":529631,"kb_avail":90924352,"statfs":{"total":95995035648,"available":93106536448,"internally_reserved":0,"allocated":2346115072,"data_stored":2342447556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33752,"internal_metadata":542342184},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":524,"seq":2250562863387,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2655876,"kb_used_data":2188732,"kb_used_omap":41,"kb_used_meta":467094,"kb_avail":91089276,"statfs":{"total":95995035648,"available":93275418624,"internally_reserved":0,"allocated":2241261568,"data_stored":2237590605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42694,"internal_metadata":478304570},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2227896320,"data_stored":2227896320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2328821760,"data_stored":2328821760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2248937472,"data_stored":2248933400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2180448256,"data_stored":2180448256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1687683072,"data_stored":1687683072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2663780352,"data_stored":2663776280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1986723840,"data_stored":1986723840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1876692992,"data_stored":1876688920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:52:04.477 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_3 pgp_num 16 2024-09-13T14:52:04.507 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:04.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:04.531 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:04.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:04.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:04 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 5]}]': finished 2024-09-13T14:52:04.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:04 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 5]}]': finished 2024-09-13T14:52:04.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:04 smithi184 ceph-mon[92527]: osdmap e1184: 8 total, 8 up, 7 in 2024-09-13T14:52:04.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:04 smithi184 ceph-mon[92527]: pgmap v1769: 119 pgs: 4 active+remapped+backfilling, 115 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s; 4986/233394 objects misplaced (2.136%); 27 MiB/s, 437 objects/s recovering 2024-09-13T14:52:04.632 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:04.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:04.675 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:04.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:04.690 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:04.776 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:04.816 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:04.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:04.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:04.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:04 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 5]}]': finished 2024-09-13T14:52:04.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:04 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 5]}]': finished 2024-09-13T14:52:04.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:04 smithi151 ceph-mon[104450]: osdmap e1184: 8 total, 8 up, 7 in 2024-09-13T14:52:04.840 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:04 smithi151 ceph-mon[104450]: pgmap v1769: 119 pgs: 4 active+remapped+backfilling, 115 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s; 4986/233394 objects misplaced (2.136%); 27 MiB/s, 437 objects/s recovering 2024-09-13T14:52:04.840 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:52:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:52:04] "GET /metrics HTTP/1.1" 200 36681 "" "Prometheus/2.51.0" 2024-09-13T14:52:04.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:04 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 5]}]': finished 2024-09-13T14:52:04.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:04 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 5]}]': finished 2024-09-13T14:52:04.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:04 smithi151 ceph-mon[108966]: osdmap e1184: 8 total, 8 up, 7 in 2024-09-13T14:52:04.840 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:04 smithi151 ceph-mon[108966]: pgmap v1769: 119 pgs: 4 active+remapped+backfilling, 115 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s; 4986/233394 objects misplaced (2.136%); 27 MiB/s, 437 objects/s recovering 2024-09-13T14:52:04.918 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:04.943 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:04.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:04.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:05.045 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:05.079 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:05.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:05.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:05.180 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:05.201 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:05.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:05.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:05.303 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:05.328 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:05.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:05.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:05.431 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:05.463 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:05.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:05.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:05.565 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:05.596 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:05.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:05.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:05.698 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:05.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:05.729 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:05.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:05.830 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:05.835 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:05 smithi184 ceph-mon[92527]: osdmap e1185: 8 total, 8 up, 7 in 2024-09-13T14:52:05.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:05 smithi151 ceph-mon[104450]: osdmap e1185: 8 total, 8 up, 7 in 2024-09-13T14:52:05.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:05 smithi151 ceph-mon[108966]: osdmap e1185: 8 total, 8 up, 7 in 2024-09-13T14:52:05.859 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:05.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:05.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:05.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:05.997 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:05.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:05.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:06.099 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:06.127 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:06.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:06.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:06.229 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:06.257 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:06.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:06.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:06.359 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:06.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:06.395 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:06.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:06.497 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:06.527 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:06.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:06.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:06.629 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:06.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:06.657 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:06.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:06.726 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:06 smithi184 ceph-mon[92527]: pgmap v1771: 119 pgs: 4 active+remapped+backfilling, 115 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 1.18k op/s; 3421/227415 objects misplaced (1.504%); 31 MiB/s, 502 objects/s recovering 2024-09-13T14:52:06.757 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:06.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:06.780 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:06.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:06.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:06 smithi151 ceph-mon[104450]: pgmap v1771: 119 pgs: 4 active+remapped+backfilling, 115 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 1.18k op/s; 3421/227415 objects misplaced (1.504%); 31 MiB/s, 502 objects/s recovering 2024-09-13T14:52:06.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:06 smithi151 ceph-mon[108966]: pgmap v1771: 119 pgs: 4 active+remapped+backfilling, 115 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 1.18k op/s; 3421/227415 objects misplaced (1.504%); 31 MiB/s, 502 objects/s recovering 2024-09-13T14:52:06.880 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:06.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:06.905 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:06.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:07.006 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:07.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.028 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:07.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:07.129 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:07.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.152 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:07.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:07.253 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:07.279 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:07.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:07.381 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:07.410 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:07.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:07.512 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:07.534 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:07.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:07.589 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:07 smithi184 ceph-mon[92527]: osdmap e1186: 8 total, 8 up, 7 in 2024-09-13T14:52:07.637 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:07.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.683 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:07.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:07.784 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:07.831 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:07.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:07.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:07 smithi151 ceph-mon[104450]: osdmap e1186: 8 total, 8 up, 7 in 2024-09-13T14:52:07.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:07 smithi151 ceph-mon[108966]: osdmap e1186: 8 total, 8 up, 7 in 2024-09-13T14:52:07.934 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:07.972 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:07.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:08.074 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:08.102 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:08.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:08.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:08.204 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:08.207 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:52:08.208 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [], 2024-09-13T14:52:08.208 INFO:teuthology.orchestra.run.smithi184.stdout: "complaint_time": 30, 2024-09-13T14:52:08.208 INFO:teuthology.orchestra.run.smithi184.stdout: "num_blocked_ops": 0 2024-09-13T14:52:08.208 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:52:08.240 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:08.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:08.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:08.342 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:08.387 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:08.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:08.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:08.489 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:08.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:08.528 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:08.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:08.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:08 smithi151 ceph-mon[108966]: osdmap e1187: 8 total, 8 up, 7 in 2024-09-13T14:52:08.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:08 smithi151 ceph-mon[108966]: pgmap v1774: 119 pgs: 4 active+remapped+backfilling, 115 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 478 op/s; 3421/227415 objects misplaced (1.504%); 20 MiB/s, 317 objects/s recovering 2024-09-13T14:52:08.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:08 smithi151 ceph-mon[104450]: osdmap e1187: 8 total, 8 up, 7 in 2024-09-13T14:52:08.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:08 smithi151 ceph-mon[104450]: pgmap v1774: 119 pgs: 4 active+remapped+backfilling, 115 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 478 op/s; 3421/227415 objects misplaced (1.504%); 20 MiB/s, 317 objects/s recovering 2024-09-13T14:52:08.629 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:08.660 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:08.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:08.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:08.762 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:08.786 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:08.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:08.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:08.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:08 smithi184 ceph-mon[92527]: osdmap e1187: 8 total, 8 up, 7 in 2024-09-13T14:52:08.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:08 smithi184 ceph-mon[92527]: pgmap v1774: 119 pgs: 4 active+remapped+backfilling, 115 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 478 op/s; 3421/227415 objects misplaced (1.504%); 20 MiB/s, 317 objects/s recovering 2024-09-13T14:52:08.888 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:08.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:08.929 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:08.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:09.030 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:09.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:09.070 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:09.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:09.171 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:09.217 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:09.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:09.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:09.300 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T14:52:09.319 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:09.340 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:09.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:09.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:09.442 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:09.486 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:09.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:09.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:09.504 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-13T14:52:09.588 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:09.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:09 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/581268654' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:52:09.618 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:52:09.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:09.623 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:09.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:09.724 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:09.752 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:09.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:09.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:09.854 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:09.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:09 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/581268654' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:52:09.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:09 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/581268654' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:52:09.878 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:09.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:09.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:09.980 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:10.011 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:10.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:10.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:10.113 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:10.144 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:10.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:10.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:10.246 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:10.277 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:10.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:10.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:10.354 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:52:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:52:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:52:10.380 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:10.410 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:10.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:10.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:10.512 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:10.541 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:10.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:10.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:10.643 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:10.677 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:10.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:10.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:10.779 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:10.803 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:10.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:10.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:10.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:10 smithi184 ceph-mon[92527]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:10.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:10 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/581268654' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:52:10.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:10 smithi184 ceph-mon[92527]: osdmap e1188: 8 total, 8 up, 7 in 2024-09-13T14:52:10.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:10 smithi184 ceph-mon[92527]: pgmap v1776: 119 pgs: 3 active+remapped+backfilling, 116 active+clean; 4.5 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 769 op/s; 3087/221520 objects misplaced (1.394%); 20 MiB/s, 321 objects/s recovering 2024-09-13T14:52:10.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:10 smithi151 ceph-mon[108966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:10.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:10 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/581268654' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:52:10.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:10 smithi151 ceph-mon[108966]: osdmap e1188: 8 total, 8 up, 7 in 2024-09-13T14:52:10.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:10 smithi151 ceph-mon[108966]: pgmap v1776: 119 pgs: 3 active+remapped+backfilling, 116 active+clean; 4.5 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 769 op/s; 3087/221520 objects misplaced (1.394%); 20 MiB/s, 321 objects/s recovering 2024-09-13T14:52:10.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:10 smithi151 ceph-mon[104450]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:10.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:10 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/581268654' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:52:10.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:10 smithi151 ceph-mon[104450]: osdmap e1188: 8 total, 8 up, 7 in 2024-09-13T14:52:10.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:10 smithi151 ceph-mon[104450]: pgmap v1776: 119 pgs: 3 active+remapped+backfilling, 116 active+clean; 4.5 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 769 op/s; 3087/221520 objects misplaced (1.394%); 20 MiB/s, 321 objects/s recovering 2024-09-13T14:52:10.905 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:10.927 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:10.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:10.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:11.029 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:11.070 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:11.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:11.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:11.172 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:11.230 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:11.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:11.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:11.332 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:11.354 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:11.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:11.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:11.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:11.457 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:52:11.457 INFO:teuthology.orchestra.run.smithi184.stdout: "size": 20, 2024-09-13T14:52:11.457 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 600, 2024-09-13T14:52:11.457 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [ 2024-09-13T14:52:11.457 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.458 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:60535 7.5 7:ab646982:::benchmark_data_smithi151_190906_object60534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1142)", 2024-09-13T14:52:11.458 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:44:32.321485+0000", 2024-09-13T14:52:11.458 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 459.12279616299998, 2024-09-13T14:52:11.458 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.16541595000000001, 2024-09-13T14:52:11.458 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:52:11.458 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:11.458 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:52:11.458 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:52:11.458 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:52:11.458 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 60535 2024-09-13T14:52:11.458 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.458 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:52:11.459 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.459 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:52:11.459 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.321485+0000", 2024-09-13T14:52:11.459 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.459 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.459 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.459 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:52:11.459 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.321485+0000", 2024-09-13T14:52:11.459 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.459 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.459 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.460 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:52:11.460 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.321484+0000", 2024-09-13T14:52:11.460 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:52:11.460 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.460 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.460 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:52:11.460 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.322401+0000", 2024-09-13T14:52:11.460 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00091768700000000004 2024-09-13T14:52:11.460 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.460 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.460 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:52:11.460 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.322422+0000", 2024-09-13T14:52:11.461 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.1018000000000001e-05 2024-09-13T14:52:11.461 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.461 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.461 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:52:11.461 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.322429+0000", 2024-09-13T14:52:11.461 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3169999999999998e-06 2024-09-13T14:52:11.461 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.461 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.461 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:52:11.461 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.322920+0000", 2024-09-13T14:52:11.461 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00049131800000000003 2024-09-13T14:52:11.461 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.461 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.462 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:52:11.462 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.323083+0000", 2024-09-13T14:52:11.462 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00016288699999999999 2024-09-13T14:52:11.462 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.462 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.462 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:52:11.462 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.323276+0000", 2024-09-13T14:52:11.462 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00019269999999999999 2024-09-13T14:52:11.462 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.462 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.462 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:52:11.462 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.325333+0000", 2024-09-13T14:52:11.462 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002057686 2024-09-13T14:52:11.463 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.463 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.463 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.463 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.483240+0000", 2024-09-13T14:52:11.463 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.15790709 2024-09-13T14:52:11.463 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.463 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.463 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.463 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.486772+0000", 2024-09-13T14:52:11.463 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0035317740000000001 2024-09-13T14:52:11.463 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.463 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.463 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:52:11.463 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.486856+0000", 2024-09-13T14:52:11.464 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.4178000000000006e-05 2024-09-13T14:52:11.464 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.464 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.464 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:52:11.464 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.486901+0000", 2024-09-13T14:52:11.464 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5104e-05 2024-09-13T14:52:11.464 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.464 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:52:11.464 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.464 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.464 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.464 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:60537 7.c 7:3d857777:::benchmark_data_smithi151_190906_object60536:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1142)", 2024-09-13T14:52:11.465 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:44:32.323865+0000", 2024-09-13T14:52:11.465 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 459.12041643800001, 2024-09-13T14:52:11.465 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.156707083, 2024-09-13T14:52:11.465 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:52:11.465 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:11.465 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:52:11.465 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:52:11.465 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:52:11.465 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 60537 2024-09-13T14:52:11.465 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.465 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:52:11.465 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.465 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:52:11.465 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.323865+0000", 2024-09-13T14:52:11.466 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.466 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.466 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.466 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:52:11.466 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.323865+0000", 2024-09-13T14:52:11.466 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.466 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.466 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.466 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:52:11.466 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.323859+0000", 2024-09-13T14:52:11.466 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999938 2024-09-13T14:52:11.466 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.466 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.466 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:52:11.467 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.323914+0000", 2024-09-13T14:52:11.467 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4979000000000003e-05 2024-09-13T14:52:11.467 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.467 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.467 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:52:11.467 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.323927+0000", 2024-09-13T14:52:11.467 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3139000000000001e-05 2024-09-13T14:52:11.467 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.467 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.467 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:52:11.467 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.323934+0000", 2024-09-13T14:52:11.467 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7479999999999996e-06 2024-09-13T14:52:11.467 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.468 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.468 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:52:11.468 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.323973+0000", 2024-09-13T14:52:11.468 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9527000000000003e-05 2024-09-13T14:52:11.468 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.468 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.468 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:52:11.468 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.324064+0000", 2024-09-13T14:52:11.468 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.1040999999999998e-05 2024-09-13T14:52:11.468 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.468 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.468 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:52:11.468 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.324166+0000", 2024-09-13T14:52:11.469 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00010198 2024-09-13T14:52:11.469 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.469 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.469 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:52:11.469 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.326541+0000", 2024-09-13T14:52:11.469 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0023743530000000001 2024-09-13T14:52:11.469 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.469 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.469 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.469 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.326736+0000", 2024-09-13T14:52:11.469 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00019539899999999999 2024-09-13T14:52:11.469 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.469 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.470 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.470 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.480442+0000", 2024-09-13T14:52:11.470 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.153705644 2024-09-13T14:52:11.470 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.470 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.470 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:52:11.470 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.480491+0000", 2024-09-13T14:52:11.470 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9375000000000001e-05 2024-09-13T14:52:11.470 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.470 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.470 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:52:11.470 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:32.480572+0000", 2024-09-13T14:52:11.471 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.1205999999999995e-05 2024-09-13T14:52:11.471 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.471 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:52:11.471 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.471 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.471 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.471 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:61895 7.5 7:a46f2851:::benchmark_data_smithi151_190906_object61894:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1142)", 2024-09-13T14:52:11.471 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:44:33.523769+0000", 2024-09-13T14:52:11.471 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 457.920512879, 2024-09-13T14:52:11.471 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.24447361400000001, 2024-09-13T14:52:11.471 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:52:11.472 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:11.472 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:52:11.472 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:52:11.472 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:52:11.472 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 61895 2024-09-13T14:52:11.472 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.472 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:52:11.472 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.472 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:52:11.472 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.523769+0000", 2024-09-13T14:52:11.472 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.472 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.472 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.472 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:52:11.472 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.523769+0000", 2024-09-13T14:52:11.473 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.473 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.473 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.473 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:52:11.473 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.523765+0000", 2024-09-13T14:52:11.473 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999962 2024-09-13T14:52:11.473 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.473 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.473 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:52:11.473 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.523823+0000", 2024-09-13T14:52:11.473 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8334000000000001e-05 2024-09-13T14:52:11.473 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.473 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.473 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:52:11.474 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.523839+0000", 2024-09-13T14:52:11.474 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.5883999999999999e-05 2024-09-13T14:52:11.474 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.474 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.474 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:52:11.474 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.523845+0000", 2024-09-13T14:52:11.474 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.2890000000000003e-06 2024-09-13T14:52:11.474 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.474 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.474 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:52:11.474 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.523918+0000", 2024-09-13T14:52:11.474 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.3069000000000001e-05 2024-09-13T14:52:11.474 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.474 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.474 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:52:11.475 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.524056+0000", 2024-09-13T14:52:11.475 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000138007 2024-09-13T14:52:11.475 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.475 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.475 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:52:11.475 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.524210+0000", 2024-09-13T14:52:11.475 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00015325000000000001 2024-09-13T14:52:11.475 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.475 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.475 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:52:11.475 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.526052+0000", 2024-09-13T14:52:11.475 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001842764 2024-09-13T14:52:11.476 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.476 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.476 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.476 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.764788+0000", 2024-09-13T14:52:11.476 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.238735528 2024-09-13T14:52:11.476 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.476 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.476 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.476 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.768139+0000", 2024-09-13T14:52:11.476 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.003350829 2024-09-13T14:52:11.476 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.476 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.477 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:52:11.477 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.768197+0000", 2024-09-13T14:52:11.477 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7899999999999998e-05 2024-09-13T14:52:11.477 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.477 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.477 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:52:11.477 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:33.768242+0000", 2024-09-13T14:52:11.477 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5809999999999997e-05 2024-09-13T14:52:11.477 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.477 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:52:11.477 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.477 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.477 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.478 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:63024 7.5 7:a28b3da7:::benchmark_data_smithi151_190906_object63023:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1142)", 2024-09-13T14:52:11.478 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:44:34.633955+0000", 2024-09-13T14:52:11.478 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 456.81032621399999, 2024-09-13T14:52:11.478 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.14815014100000001, 2024-09-13T14:52:11.478 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:52:11.478 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:11.478 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:52:11.478 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:52:11.478 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:52:11.478 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 63024 2024-09-13T14:52:11.478 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.478 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:52:11.478 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.478 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:52:11.479 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.633955+0000", 2024-09-13T14:52:11.479 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.479 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.479 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.479 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:52:11.479 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.633955+0000", 2024-09-13T14:52:11.479 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.479 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.479 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.479 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:52:11.479 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.633954+0000", 2024-09-13T14:52:11.479 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:52:11.479 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.479 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.480 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:52:11.480 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.634020+0000", 2024-09-13T14:52:11.480 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.6606000000000006e-05 2024-09-13T14:52:11.480 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.480 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.480 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:52:11.480 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.634033+0000", 2024-09-13T14:52:11.480 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2574e-05 2024-09-13T14:52:11.480 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.480 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.480 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:52:11.480 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.634039+0000", 2024-09-13T14:52:11.480 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5219999999999998e-06 2024-09-13T14:52:11.481 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.481 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.481 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:52:11.481 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.634085+0000", 2024-09-13T14:52:11.481 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.6412000000000002e-05 2024-09-13T14:52:11.481 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.481 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.481 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:52:11.481 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.634226+0000", 2024-09-13T14:52:11.481 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014136199999999999 2024-09-13T14:52:11.481 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.481 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.481 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:52:11.481 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.634370+0000", 2024-09-13T14:52:11.481 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000143607 2024-09-13T14:52:11.482 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.482 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.482 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:52:11.482 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.636483+0000", 2024-09-13T14:52:11.482 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0021130279999999999 2024-09-13T14:52:11.482 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.482 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.482 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.482 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.753310+0000", 2024-09-13T14:52:11.482 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.116827134 2024-09-13T14:52:11.482 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.482 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.482 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.482 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.781847+0000", 2024-09-13T14:52:11.483 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.028536736 2024-09-13T14:52:11.483 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.483 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.483 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:52:11.483 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.782052+0000", 2024-09-13T14:52:11.483 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00020504 2024-09-13T14:52:11.483 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.483 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.483 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:52:11.483 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:34.782106+0000", 2024-09-13T14:52:11.483 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3667e-05 2024-09-13T14:52:11.483 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.483 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:52:11.483 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.484 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.484 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.484 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:66880 7.5 7:aba8be0f:::benchmark_data_smithi151_190906_object66879:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1142)", 2024-09-13T14:52:11.484 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:44:37.928179+0000", 2024-09-13T14:52:11.484 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 453.516102988, 2024-09-13T14:52:11.484 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.21699406800000001, 2024-09-13T14:52:11.484 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:52:11.484 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:11.484 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:52:11.484 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:52:11.484 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:52:11.484 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 66880 2024-09-13T14:52:11.484 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.484 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:52:11.484 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.485 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:52:11.485 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:37.928179+0000", 2024-09-13T14:52:11.485 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.485 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.485 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.485 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:52:11.485 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:37.928179+0000", 2024-09-13T14:52:11.485 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.485 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.485 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.485 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:52:11.485 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:37.928178+0000", 2024-09-13T14:52:11.485 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:52:11.485 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.486 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.486 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:52:11.486 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:37.928211+0000", 2024-09-13T14:52:11.486 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.2928000000000001e-05 2024-09-13T14:52:11.486 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.486 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.486 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:52:11.486 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:37.928221+0000", 2024-09-13T14:52:11.486 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0215e-05 2024-09-13T14:52:11.486 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.486 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.486 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:52:11.486 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:37.928227+0000", 2024-09-13T14:52:11.486 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.0859999999999998e-06 2024-09-13T14:52:11.487 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.487 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.487 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:52:11.487 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:37.930791+0000", 2024-09-13T14:52:11.487 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002564181 2024-09-13T14:52:11.487 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.487 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.487 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:52:11.487 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:37.930931+0000", 2024-09-13T14:52:11.487 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000140187 2024-09-13T14:52:11.487 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.487 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.487 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:52:11.487 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:37.931071+0000", 2024-09-13T14:52:11.488 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013932800000000001 2024-09-13T14:52:11.488 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.488 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.488 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:52:11.488 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:37.933828+0000", 2024-09-13T14:52:11.488 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0027570680000000001 2024-09-13T14:52:11.488 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.488 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.488 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.488 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:37.936581+0000", 2024-09-13T14:52:11.488 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002753279 2024-09-13T14:52:11.488 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.488 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.489 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.489 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:38.145102+0000", 2024-09-13T14:52:11.489 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.20852106500000001 2024-09-13T14:52:11.489 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.489 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.489 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:52:11.489 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:38.145145+0000", 2024-09-13T14:52:11.489 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3263999999999998e-05 2024-09-13T14:52:11.489 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.489 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.489 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:52:11.489 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:38.145173+0000", 2024-09-13T14:52:11.489 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.7389000000000002e-05 2024-09-13T14:52:11.489 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.489 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:52:11.490 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.490 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.490 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.490 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:93867 7.c 7:333b2c00:::benchmark_data_smithi151_190906_object93866:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1145)", 2024-09-13T14:52:11.490 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:44:59.912033+0000", 2024-09-13T14:52:11.490 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 431.532249057, 2024-09-13T14:52:11.490 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.16074064299999999, 2024-09-13T14:52:11.490 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:52:11.490 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:11.491 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:52:11.491 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:52:11.491 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:52:11.491 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 93867 2024-09-13T14:52:11.491 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.491 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:52:11.491 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.491 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:52:11.491 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.912033+0000", 2024-09-13T14:52:11.492 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.492 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.492 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.492 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:52:11.492 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.912033+0000", 2024-09-13T14:52:11.492 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.492 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.492 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.492 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:52:11.492 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.912025+0000", 2024-09-13T14:52:11.492 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999919 2024-09-13T14:52:11.492 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.492 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.493 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:52:11.493 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.912073+0000", 2024-09-13T14:52:11.493 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8429e-05 2024-09-13T14:52:11.493 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.493 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.493 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:52:11.493 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.912092+0000", 2024-09-13T14:52:11.493 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.8536999999999999e-05 2024-09-13T14:52:11.493 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.493 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.493 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:52:11.493 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.912098+0000", 2024-09-13T14:52:11.493 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.0190000000000003e-06 2024-09-13T14:52:11.493 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.494 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.494 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:52:11.494 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.912139+0000", 2024-09-13T14:52:11.494 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1306000000000001e-05 2024-09-13T14:52:11.494 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.494 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.494 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:52:11.494 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.912280+0000", 2024-09-13T14:52:11.494 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000140753 2024-09-13T14:52:11.494 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.494 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.494 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:52:11.494 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.912440+0000", 2024-09-13T14:52:11.495 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000160755 2024-09-13T14:52:11.495 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.495 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.495 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:52:11.495 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.919007+0000", 2024-09-13T14:52:11.495 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.006566934 2024-09-13T14:52:11.495 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.495 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.495 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.495 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.919285+0000", 2024-09-13T14:52:11.495 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00027790399999999999 2024-09-13T14:52:11.495 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.495 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.495 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.495 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.072630+0000", 2024-09-13T14:52:11.496 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.15334468400000001 2024-09-13T14:52:11.496 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.496 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.496 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:52:11.496 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.072731+0000", 2024-09-13T14:52:11.496 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00010129200000000001 2024-09-13T14:52:11.496 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.496 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.496 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:52:11.496 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.072773+0000", 2024-09-13T14:52:11.496 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.2006999999999999e-05 2024-09-13T14:52:11.496 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.496 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:52:11.496 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.497 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.497 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.497 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:93871 7.c 7:3f478944:::benchmark_data_smithi151_190906_object93870:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1145)", 2024-09-13T14:52:11.497 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:44:59.914328+0000", 2024-09-13T14:52:11.497 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 431.52995361299998, 2024-09-13T14:52:11.497 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.15858155500000001, 2024-09-13T14:52:11.497 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:52:11.497 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:11.497 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:52:11.497 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:52:11.497 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:52:11.497 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 93871 2024-09-13T14:52:11.497 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.497 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:52:11.498 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.498 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:52:11.498 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.914328+0000", 2024-09-13T14:52:11.498 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.498 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.498 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.498 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:52:11.498 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.914328+0000", 2024-09-13T14:52:11.498 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.498 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.498 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.498 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:52:11.498 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.914327+0000", 2024-09-13T14:52:11.499 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:52:11.499 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.499 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.499 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:52:11.499 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.914382+0000", 2024-09-13T14:52:11.499 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4993e-05 2024-09-13T14:52:11.499 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.499 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.499 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:52:11.499 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.914397+0000", 2024-09-13T14:52:11.499 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.5413999999999998e-05 2024-09-13T14:52:11.500 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.500 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.500 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:52:11.500 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.914403+0000", 2024-09-13T14:52:11.500 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.0889999999999996e-06 2024-09-13T14:52:11.500 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.500 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:11.502 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:52:11.502 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.915554+0000", 2024-09-13T14:52:11.502 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0011510260000000001 2024-09-13T14:52:11.502 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.502 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.502 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:52:11.502 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.915687+0000", 2024-09-13T14:52:11.502 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000132879 2024-09-13T14:52:11.503 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.503 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.503 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:52:11.503 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.915834+0000", 2024-09-13T14:52:11.503 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000147248 2024-09-13T14:52:11.503 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.503 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.503 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:52:11.503 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.919317+0000", 2024-09-13T14:52:11.503 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0034822799999999999 2024-09-13T14:52:11.503 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.503 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.503 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.503 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.919609+0000", 2024-09-13T14:52:11.504 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00029270099999999997 2024-09-13T14:52:11.504 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.504 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.504 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.504 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.072807+0000", 2024-09-13T14:52:11.504 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.15319735400000001 2024-09-13T14:52:11.504 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.504 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.504 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:52:11.504 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.072879+0000", 2024-09-13T14:52:11.504 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.1826e-05 2024-09-13T14:52:11.504 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.504 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.504 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:52:11.505 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.072910+0000", 2024-09-13T14:52:11.505 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.0920999999999998e-05 2024-09-13T14:52:11.505 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.505 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:52:11.505 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.505 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.505 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.505 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:93879 7.5 7:a47db476:::benchmark_data_smithi151_190906_object93878:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1145)", 2024-09-13T14:52:11.505 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:44:59.919741+0000", 2024-09-13T14:52:11.505 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 431.52454089600002, 2024-09-13T14:52:11.505 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.15622032599999999, 2024-09-13T14:52:11.505 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:52:11.505 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:11.505 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:52:11.506 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:52:11.506 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:52:11.506 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 93879 2024-09-13T14:52:11.506 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.506 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:52:11.506 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.506 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:52:11.506 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.919741+0000", 2024-09-13T14:52:11.506 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.506 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.506 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.506 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:52:11.507 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.919741+0000", 2024-09-13T14:52:11.507 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.507 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.507 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.507 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:52:11.507 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.919739+0000", 2024-09-13T14:52:11.507 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:52:11.507 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.507 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.507 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:52:11.507 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.919769+0000", 2024-09-13T14:52:11.507 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.0221000000000001e-05 2024-09-13T14:52:11.507 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.507 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.508 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:52:11.508 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.919784+0000", 2024-09-13T14:52:11.508 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.5308999999999999e-05 2024-09-13T14:52:11.508 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.508 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.508 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:52:11.508 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.919790+0000", 2024-09-13T14:52:11.508 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6899999999999997e-06 2024-09-13T14:52:11.508 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.508 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.508 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:52:11.508 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.919830+0000", 2024-09-13T14:52:11.508 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9993999999999997e-05 2024-09-13T14:52:11.509 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.509 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.509 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:52:11.509 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.919964+0000", 2024-09-13T14:52:11.509 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013362700000000001 2024-09-13T14:52:11.509 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.509 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.509 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:52:11.509 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.920098+0000", 2024-09-13T14:52:11.509 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013436899999999999 2024-09-13T14:52:11.509 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.509 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.510 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:52:11.510 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:44:59.923018+0000", 2024-09-13T14:52:11.510 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0029198119999999999 2024-09-13T14:52:11.510 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.510 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.510 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.510 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.074680+0000", 2024-09-13T14:52:11.510 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.15166197100000001 2024-09-13T14:52:11.510 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.510 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.510 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.510 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.075859+0000", 2024-09-13T14:52:11.510 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0011794310000000001 2024-09-13T14:52:11.510 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.511 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.511 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:52:11.511 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.075919+0000", 2024-09-13T14:52:11.511 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.9697999999999997e-05 2024-09-13T14:52:11.511 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.511 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.511 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:52:11.511 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:00.075961+0000", 2024-09-13T14:52:11.511 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.2005000000000002e-05 2024-09-13T14:52:11.511 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.511 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:52:11.511 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.511 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.511 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.512 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:97110 7.5 7:a3dd28eb:::benchmark_data_smithi151_190906_object97109:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1146)", 2024-09-13T14:52:11.512 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:45:03.423687+0000", 2024-09-13T14:52:11.512 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 428.02059437100002, 2024-09-13T14:52:11.512 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.141989755, 2024-09-13T14:52:11.512 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:52:11.512 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:11.512 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:52:11.512 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:52:11.512 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:52:11.512 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 97110 2024-09-13T14:52:11.512 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.512 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:52:11.512 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.513 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:52:11.513 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.423687+0000", 2024-09-13T14:52:11.513 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.513 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.513 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.513 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:52:11.513 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.423687+0000", 2024-09-13T14:52:11.513 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.513 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.513 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.513 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:52:11.514 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.423686+0000", 2024-09-13T14:52:11.514 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:52:11.514 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.514 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.514 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:52:11.514 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.423727+0000", 2024-09-13T14:52:11.514 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0964999999999998e-05 2024-09-13T14:52:11.514 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.514 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.514 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:52:11.514 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.423743+0000", 2024-09-13T14:52:11.514 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6764000000000001e-05 2024-09-13T14:52:11.514 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.514 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.515 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:52:11.515 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.423750+0000", 2024-09-13T14:52:11.515 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.2759999999999997e-06 2024-09-13T14:52:11.515 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.515 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.515 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:52:11.515 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.423794+0000", 2024-09-13T14:52:11.515 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4095999999999999e-05 2024-09-13T14:52:11.515 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.515 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.515 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:52:11.515 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.423926+0000", 2024-09-13T14:52:11.515 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013241899999999999 2024-09-13T14:52:11.515 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.516 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.516 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:52:11.516 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.424070+0000", 2024-09-13T14:52:11.516 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014382899999999999 2024-09-13T14:52:11.516 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.516 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.516 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:52:11.516 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.428093+0000", 2024-09-13T14:52:11.516 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0040235119999999999 2024-09-13T14:52:11.516 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.516 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.516 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.516 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.561023+0000", 2024-09-13T14:52:11.516 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.132929509 2024-09-13T14:52:11.517 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.517 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.517 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.517 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.565543+0000", 2024-09-13T14:52:11.517 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0045196780000000001 2024-09-13T14:52:11.517 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.517 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.517 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:52:11.517 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.565633+0000", 2024-09-13T14:52:11.517 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.0496999999999998e-05 2024-09-13T14:52:11.517 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.517 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.517 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:52:11.517 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.565677+0000", 2024-09-13T14:52:11.518 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3810000000000002e-05 2024-09-13T14:52:11.518 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.518 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:52:11.518 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.518 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.518 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.518 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:97113 7.c 7:390d7e9f:::benchmark_data_smithi151_190906_object97112:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1146)", 2024-09-13T14:52:11.518 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:45:03.425943+0000", 2024-09-13T14:52:11.518 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 428.01833875, 2024-09-13T14:52:11.518 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.141872581, 2024-09-13T14:52:11.518 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:52:11.518 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:11.518 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:52:11.519 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:52:11.519 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:52:11.519 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 97113 2024-09-13T14:52:11.519 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.519 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:52:11.519 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.519 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:52:11.519 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.425943+0000", 2024-09-13T14:52:11.519 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.519 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.519 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.519 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:52:11.519 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.425943+0000", 2024-09-13T14:52:11.519 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.520 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.520 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.520 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:52:11.520 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.425941+0000", 2024-09-13T14:52:11.520 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:52:11.520 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.520 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.520 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:52:11.520 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.426010+0000", 2024-09-13T14:52:11.520 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.8917000000000004e-05 2024-09-13T14:52:11.520 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.520 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.520 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:52:11.521 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.426026+0000", 2024-09-13T14:52:11.521 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6535999999999999e-05 2024-09-13T14:52:11.521 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.521 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.521 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:52:11.521 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.426034+0000", 2024-09-13T14:52:11.521 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.3420000000000002e-06 2024-09-13T14:52:11.521 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.521 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.521 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:52:11.521 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.426082+0000", 2024-09-13T14:52:11.521 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8439999999999997e-05 2024-09-13T14:52:11.521 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.521 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.522 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:52:11.522 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.426209+0000", 2024-09-13T14:52:11.522 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012715400000000001 2024-09-13T14:52:11.522 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.522 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.522 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:52:11.522 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.426347+0000", 2024-09-13T14:52:11.522 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000137689 2024-09-13T14:52:11.522 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.522 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.522 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:52:11.522 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.429590+0000", 2024-09-13T14:52:11.522 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0032433940000000001 2024-09-13T14:52:11.522 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.523 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.523 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.523 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.431300+0000", 2024-09-13T14:52:11.523 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0017094090000000001 2024-09-13T14:52:11.523 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.523 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.523 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.523 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.567733+0000", 2024-09-13T14:52:11.523 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.13643355200000001 2024-09-13T14:52:11.523 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.523 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.523 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:52:11.523 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.567777+0000", 2024-09-13T14:52:11.524 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3906000000000003e-05 2024-09-13T14:52:11.524 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.524 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.524 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:52:11.524 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.567815+0000", 2024-09-13T14:52:11.524 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8186999999999999e-05 2024-09-13T14:52:11.524 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.524 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:52:11.524 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.524 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.524 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.524 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:97120 7.c 7:31feb6db:::benchmark_data_smithi151_190906_object97119:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1146)", 2024-09-13T14:52:11.524 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:45:03.431613+0000", 2024-09-13T14:52:11.525 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 428.01266872100001, 2024-09-13T14:52:11.525 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.13969756899999999, 2024-09-13T14:52:11.525 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:52:11.525 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:11.525 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:52:11.525 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:52:11.525 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:52:11.525 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 97120 2024-09-13T14:52:11.525 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.525 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:52:11.525 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.525 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:52:11.525 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.431613+0000", 2024-09-13T14:52:11.526 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.526 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.526 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.526 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:52:11.526 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.431613+0000", 2024-09-13T14:52:11.526 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.526 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.526 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.526 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:52:11.526 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.431611+0000", 2024-09-13T14:52:11.526 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:52:11.526 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.526 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.526 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:52:11.527 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.431661+0000", 2024-09-13T14:52:11.527 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0099000000000003e-05 2024-09-13T14:52:11.527 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.527 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.527 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:52:11.527 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.431672+0000", 2024-09-13T14:52:11.527 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1161999999999999e-05 2024-09-13T14:52:11.527 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.527 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.527 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:52:11.527 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.431677+0000", 2024-09-13T14:52:11.527 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1250000000000003e-06 2024-09-13T14:52:11.527 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.527 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.528 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:52:11.528 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.431709+0000", 2024-09-13T14:52:11.528 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.2889999999999999e-05 2024-09-13T14:52:11.528 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.528 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.528 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:52:11.528 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.431813+0000", 2024-09-13T14:52:11.528 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00010348699999999999 2024-09-13T14:52:11.528 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.528 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.528 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:52:11.528 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.431922+0000", 2024-09-13T14:52:11.529 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000108633 2024-09-13T14:52:11.529 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.529 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.529 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:52:11.529 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.434449+0000", 2024-09-13T14:52:11.529 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0025274859999999998 2024-09-13T14:52:11.529 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.529 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.529 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.529 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.434644+0000", 2024-09-13T14:52:11.529 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00019509599999999999 2024-09-13T14:52:11.529 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.529 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.530 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.530 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.571212+0000", 2024-09-13T14:52:11.530 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.136568089 2024-09-13T14:52:11.530 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.530 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.530 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:52:11.530 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.571265+0000", 2024-09-13T14:52:11.530 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.2720999999999999e-05 2024-09-13T14:52:11.530 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.530 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.530 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:52:11.530 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.571310+0000", 2024-09-13T14:52:11.530 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5469999999999997e-05 2024-09-13T14:52:11.531 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.531 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:52:11.531 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.531 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.531 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.531 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:139341 7.5 7:ae3aab6d:::benchmark_data_smithi151_190906_object18880:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1155)", 2024-09-13T14:52:11.531 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:46:02.939175+0000", 2024-09-13T14:52:11.531 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 368.50510683300001, 2024-09-13T14:52:11.531 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.16508193299999999, 2024-09-13T14:52:11.531 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:52:11.531 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:11.531 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:52:11.531 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:52:11.532 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:52:11.532 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 139341 2024-09-13T14:52:11.532 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.532 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:52:11.532 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.532 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:52:11.532 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:02.939175+0000", 2024-09-13T14:52:11.532 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.532 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.532 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.532 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:52:11.532 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:02.939175+0000", 2024-09-13T14:52:11.532 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.533 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.533 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.533 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:52:11.533 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:02.939173+0000", 2024-09-13T14:52:11.533 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:52:11.533 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.533 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.533 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:52:11.533 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:02.939181+0000", 2024-09-13T14:52:11.533 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0809999999999994e-06 2024-09-13T14:52:11.533 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.533 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.533 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:52:11.534 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:02.939193+0000", 2024-09-13T14:52:11.534 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1909000000000001e-05 2024-09-13T14:52:11.534 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.534 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.534 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:52:11.534 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:02.939198+0000", 2024-09-13T14:52:11.534 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3019999999999996e-06 2024-09-13T14:52:11.534 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.534 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.534 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:52:11.534 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:02.939243+0000", 2024-09-13T14:52:11.534 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5426999999999997e-05 2024-09-13T14:52:11.534 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.535 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.535 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:52:11.535 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:02.939361+0000", 2024-09-13T14:52:11.535 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000118319 2024-09-13T14:52:11.535 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.535 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.535 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:52:11.535 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:02.939478+0000", 2024-09-13T14:52:11.535 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000116876 2024-09-13T14:52:11.535 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.535 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.535 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:52:11.535 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:02.942666+0000", 2024-09-13T14:52:11.535 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0031874490000000002 2024-09-13T14:52:11.536 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.536 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.536 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.536 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:03.088723+0000", 2024-09-13T14:52:11.536 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.146056993 2024-09-13T14:52:11.536 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.536 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.536 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.536 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:03.104167+0000", 2024-09-13T14:52:11.536 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.015444080000000001 2024-09-13T14:52:11.536 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.536 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.536 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:52:11.537 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:03.104215+0000", 2024-09-13T14:52:11.537 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8127000000000002e-05 2024-09-13T14:52:11.537 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.537 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.537 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:52:11.537 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:03.104257+0000", 2024-09-13T14:52:11.537 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1875999999999997e-05 2024-09-13T14:52:11.537 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.537 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:52:11.537 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.537 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.537 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.537 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:139348 7.5 7:a9fcabc8:::benchmark_data_smithi151_190906_object18887:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1155)", 2024-09-13T14:52:11.538 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:46:02.944758+0000", 2024-09-13T14:52:11.538 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 368.49952392799997, 2024-09-13T14:52:11.538 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.15962931599999999, 2024-09-13T14:52:11.538 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:52:11.538 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:11.538 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:52:11.538 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:52:11.538 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:52:11.538 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 139348 2024-09-13T14:52:11.538 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.538 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:52:11.538 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.538 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:52:11.538 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:02.944758+0000", 2024-09-13T14:52:11.538 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.539 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.539 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.539 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:52:11.539 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:02.944758+0000", 2024-09-13T14:52:11.539 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.539 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.539 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.539 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:52:11.539 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:02.944756+0000", 2024-09-13T14:52:11.539 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:52:11.539 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.539 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.539 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:52:11.539 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:02.944765+0000", 2024-09-13T14:52:11.540 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.6780000000000006e-06 2024-09-13T14:52:11.540 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.540 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.540 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:52:11.540 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:02.944777+0000", 2024-09-13T14:52:11.540 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1780000000000001e-05 2024-09-13T14:52:11.540 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.540 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.540 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:52:11.540 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:02.944782+0000", 2024-09-13T14:52:11.540 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1420000000000001e-06 2024-09-13T14:52:11.540 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.540 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.540 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:52:11.541 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:02.944815+0000", 2024-09-13T14:52:11.541 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3330000000000001e-05 2024-09-13T14:52:11.541 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.541 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.541 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:52:11.541 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:02.944902+0000", 2024-09-13T14:52:11.541 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.6176999999999998e-05 2024-09-13T14:52:11.541 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.541 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.541 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:52:11.541 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:02.944974+0000", 2024-09-13T14:52:11.541 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.2638999999999994e-05 2024-09-13T14:52:11.541 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.541 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.542 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:52:11.542 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:02.947161+0000", 2024-09-13T14:52:11.542 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0021867900000000001 2024-09-13T14:52:11.542 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.542 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.542 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.542 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:03.097026+0000", 2024-09-13T14:52:11.542 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.149864992 2024-09-13T14:52:11.542 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.542 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.542 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.542 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:03.104303+0000", 2024-09-13T14:52:11.542 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00727737 2024-09-13T14:52:11.543 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.543 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.543 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:52:11.543 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:03.104349+0000", 2024-09-13T14:52:11.543 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.6078999999999997e-05 2024-09-13T14:52:11.543 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.543 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.543 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:52:11.543 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:03.104387+0000", 2024-09-13T14:52:11.543 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.7579000000000001e-05 2024-09-13T14:52:11.543 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.543 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:52:11.543 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.543 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.544 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.544 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:195476 7.5 7:a666c979:::benchmark_data_smithi151_190906_object75015:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1160)", 2024-09-13T14:52:11.544 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:47:34.633167+0000", 2024-09-13T14:52:11.544 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 276.811114304, 2024-09-13T14:52:11.544 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.278040383, 2024-09-13T14:52:11.544 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:52:11.544 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:11.544 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:52:11.544 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:52:11.544 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:52:11.544 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 195476 2024-09-13T14:52:11.544 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.544 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:52:11.545 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.545 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:52:11.545 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.633167+0000", 2024-09-13T14:52:11.545 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.545 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.545 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.545 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:52:11.545 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.633167+0000", 2024-09-13T14:52:11.545 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.545 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.545 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.546 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:52:11.546 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.633166+0000", 2024-09-13T14:52:11.546 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:52:11.546 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.546 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.546 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:52:11.546 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.633172+0000", 2024-09-13T14:52:11.546 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.9560000000000002e-06 2024-09-13T14:52:11.546 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.546 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.546 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:52:11.546 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.633184+0000", 2024-09-13T14:52:11.547 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1615e-05 2024-09-13T14:52:11.547 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.547 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.547 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:52:11.547 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.633188+0000", 2024-09-13T14:52:11.547 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.2969999999999997e-06 2024-09-13T14:52:11.547 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.547 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.547 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:52:11.547 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.633231+0000", 2024-09-13T14:52:11.548 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3109999999999999e-05 2024-09-13T14:52:11.548 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.548 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.548 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:52:11.548 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.633307+0000", 2024-09-13T14:52:11.548 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.6138999999999997e-05 2024-09-13T14:52:11.548 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.548 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.548 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:52:11.548 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.633371+0000", 2024-09-13T14:52:11.548 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3775000000000005e-05 2024-09-13T14:52:11.548 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.549 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.549 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:52:11.549 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.635802+0000", 2024-09-13T14:52:11.549 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0024314419999999998 2024-09-13T14:52:11.549 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.549 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.549 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.549 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.648302+0000", 2024-09-13T14:52:11.549 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.012499613999999999 2024-09-13T14:52:11.549 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.549 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.549 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.549 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.911112+0000", 2024-09-13T14:52:11.550 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.26281038200000001 2024-09-13T14:52:11.550 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.550 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.550 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:52:11.550 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.911165+0000", 2024-09-13T14:52:11.550 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.2258e-05 2024-09-13T14:52:11.550 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.550 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.550 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:52:11.550 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.911208+0000", 2024-09-13T14:52:11.550 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3074999999999997e-05 2024-09-13T14:52:11.550 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.551 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:52:11.551 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.551 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.551 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.551 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:195494 7.5 7:ad93b618:::benchmark_data_smithi151_190906_object75033:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1160)", 2024-09-13T14:52:11.551 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:47:34.656435+0000", 2024-09-13T14:52:11.551 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 276.78784700199998, 2024-09-13T14:52:11.551 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.25947305300000001, 2024-09-13T14:52:11.551 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:52:11.551 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:11.551 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:52:11.551 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:52:11.551 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:52:11.551 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 195494 2024-09-13T14:52:11.552 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.552 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:52:11.552 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.552 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:52:11.552 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.656435+0000", 2024-09-13T14:52:11.552 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.552 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.552 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.552 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:52:11.552 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.656435+0000", 2024-09-13T14:52:11.552 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.552 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.552 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.552 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:52:11.553 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.656433+0000", 2024-09-13T14:52:11.553 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:52:11.553 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.553 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.553 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:52:11.553 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.656442+0000", 2024-09-13T14:52:11.553 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.3590000000000008e-06 2024-09-13T14:52:11.553 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.553 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.553 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:52:11.553 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.656454+0000", 2024-09-13T14:52:11.553 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.236e-05 2024-09-13T14:52:11.553 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.554 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.554 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:52:11.554 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.656458+0000", 2024-09-13T14:52:11.554 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.51e-06 2024-09-13T14:52:11.554 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.554 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.554 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:52:11.554 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.656497+0000", 2024-09-13T14:52:11.554 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8287999999999997e-05 2024-09-13T14:52:11.554 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.554 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.554 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:52:11.554 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.656590+0000", 2024-09-13T14:52:11.554 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.2812000000000005e-05 2024-09-13T14:52:11.555 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.555 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.555 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:52:11.555 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.656671+0000", 2024-09-13T14:52:11.555 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.1763999999999998e-05 2024-09-13T14:52:11.555 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.555 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.555 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:52:11.555 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.659440+0000", 2024-09-13T14:52:11.555 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0027690840000000002 2024-09-13T14:52:11.555 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.555 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.555 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.556 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.664543+0000", 2024-09-13T14:52:11.556 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0051029259999999998 2024-09-13T14:52:11.556 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.556 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.556 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.556 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.915817+0000", 2024-09-13T14:52:11.556 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.25127348799999999 2024-09-13T14:52:11.556 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.556 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.556 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:52:11.556 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.915861+0000", 2024-09-13T14:52:11.556 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4628e-05 2024-09-13T14:52:11.556 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.556 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.557 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:52:11.557 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.915908+0000", 2024-09-13T14:52:11.557 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.6159999999999999e-05 2024-09-13T14:52:11.557 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.557 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:52:11.557 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.557 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.557 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.557 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:195501 7.5 7:a7e7f373:::benchmark_data_smithi151_190906_object75040:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1160)", 2024-09-13T14:52:11.557 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:47:34.667132+0000", 2024-09-13T14:52:11.557 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 276.77714943199999, 2024-09-13T14:52:11.557 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.25459638400000001, 2024-09-13T14:52:11.557 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:52:11.557 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:11.558 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:52:11.558 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:52:11.558 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:52:11.558 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 195501 2024-09-13T14:52:11.558 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.558 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:52:11.558 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.558 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:52:11.558 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.667132+0000", 2024-09-13T14:52:11.558 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.558 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.558 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.558 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:52:11.558 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.667132+0000", 2024-09-13T14:52:11.559 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.559 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.559 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:11.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:11.560 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.560 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:52:11.560 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.667131+0000", 2024-09-13T14:52:11.560 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:52:11.560 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.560 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.561 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:52:11.561 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.667138+0000", 2024-09-13T14:52:11.561 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.1239999999999999e-06 2024-09-13T14:52:11.561 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.561 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.561 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:52:11.561 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.667151+0000", 2024-09-13T14:52:11.561 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2591e-05 2024-09-13T14:52:11.561 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.561 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.561 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:52:11.561 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.667155+0000", 2024-09-13T14:52:11.561 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.425e-06 2024-09-13T14:52:11.562 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.562 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.562 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:52:11.562 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.667187+0000", 2024-09-13T14:52:11.562 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.1814999999999999e-05 2024-09-13T14:52:11.562 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.562 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.562 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:52:11.562 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.667271+0000", 2024-09-13T14:52:11.562 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.3898000000000002e-05 2024-09-13T14:52:11.562 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.562 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.562 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:52:11.562 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.667346+0000", 2024-09-13T14:52:11.563 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.5086999999999994e-05 2024-09-13T14:52:11.563 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.563 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.563 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:52:11.563 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.669837+0000", 2024-09-13T14:52:11.563 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0024908310000000002 2024-09-13T14:52:11.563 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.563 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.563 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.563 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.670385+0000", 2024-09-13T14:52:11.563 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00054849800000000002 2024-09-13T14:52:11.563 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.563 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.564 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.564 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.921670+0000", 2024-09-13T14:52:11.564 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.25128460800000002 2024-09-13T14:52:11.564 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.564 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.564 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:52:11.564 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.921703+0000", 2024-09-13T14:52:11.564 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3327999999999997e-05 2024-09-13T14:52:11.564 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.564 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.564 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:52:11.564 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.921729+0000", 2024-09-13T14:52:11.564 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.5542999999999999e-05 2024-09-13T14:52:11.565 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.565 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:52:11.565 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.565 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.565 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.565 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:198717 7.5 7:a04aecc3:::benchmark_data_smithi151_190906_object78256:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1161)", 2024-09-13T14:52:11.565 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:47:40.844648+0000", 2024-09-13T14:52:11.565 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 270.59963326000002, 2024-09-13T14:52:11.565 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.13845824200000001, 2024-09-13T14:52:11.565 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:52:11.565 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:11.565 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:52:11.565 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:52:11.565 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:52:11.566 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 198717 2024-09-13T14:52:11.566 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.566 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:52:11.566 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.566 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:52:11.566 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.844648+0000", 2024-09-13T14:52:11.566 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.566 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.566 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.566 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:52:11.566 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.844648+0000", 2024-09-13T14:52:11.566 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.566 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.566 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.567 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:52:11.567 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.844647+0000", 2024-09-13T14:52:11.567 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:52:11.567 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.567 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.567 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:52:11.567 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.844654+0000", 2024-09-13T14:52:11.567 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.4459999999999998e-06 2024-09-13T14:52:11.567 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.567 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.567 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:52:11.567 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.844665+0000", 2024-09-13T14:52:11.568 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1630999999999999e-05 2024-09-13T14:52:11.568 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.568 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.568 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:52:11.568 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.844671+0000", 2024-09-13T14:52:11.568 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4419999999999997e-06 2024-09-13T14:52:11.568 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.568 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.568 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:52:11.568 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.844703+0000", 2024-09-13T14:52:11.568 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.2317000000000002e-05 2024-09-13T14:52:11.568 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.568 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.569 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:52:11.569 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.844790+0000", 2024-09-13T14:52:11.569 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.6692000000000003e-05 2024-09-13T14:52:11.569 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.569 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.569 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:52:11.569 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.844866+0000", 2024-09-13T14:52:11.569 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.6230999999999996e-05 2024-09-13T14:52:11.569 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.569 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.569 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:52:11.569 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.847400+0000", 2024-09-13T14:52:11.569 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0025342329999999999 2024-09-13T14:52:11.570 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.570 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.570 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.570 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.854311+0000", 2024-09-13T14:52:11.570 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0069111210000000001 2024-09-13T14:52:11.570 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.570 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.570 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.570 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.983006+0000", 2024-09-13T14:52:11.570 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.12869518199999999 2024-09-13T14:52:11.570 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.570 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.570 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:52:11.570 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.983063+0000", 2024-09-13T14:52:11.571 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6413999999999999e-05 2024-09-13T14:52:11.571 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.571 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.571 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:52:11.571 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.983107+0000", 2024-09-13T14:52:11.571 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3748999999999997e-05 2024-09-13T14:52:11.571 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.571 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:52:11.571 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.571 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.571 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.571 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:216097 7.5 7:aa855f38:::benchmark_data_smithi151_190906_object95636:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1162)", 2024-09-13T14:52:11.571 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:48:09.605767+0000", 2024-09-13T14:52:11.571 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 241.83851411399999, 2024-09-13T14:52:11.572 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.17497162499999999, 2024-09-13T14:52:11.572 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:52:11.572 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:11.572 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:52:11.572 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:52:11.572 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:52:11.572 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 216097 2024-09-13T14:52:11.572 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.572 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:52:11.572 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.572 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:52:11.572 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:09.605767+0000", 2024-09-13T14:52:11.572 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.572 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.573 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.573 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:52:11.573 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:09.605767+0000", 2024-09-13T14:52:11.573 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.573 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.573 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.573 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:52:11.573 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:09.605767+0000", 2024-09-13T14:52:11.573 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:52:11.573 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.573 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.573 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:52:11.573 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:09.605773+0000", 2024-09-13T14:52:11.573 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7020000000000004e-06 2024-09-13T14:52:11.574 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.574 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.574 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:52:11.574 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:09.605785+0000", 2024-09-13T14:52:11.574 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1161e-05 2024-09-13T14:52:11.574 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.574 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.574 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:52:11.574 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:09.605790+0000", 2024-09-13T14:52:11.574 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4480000000000002e-06 2024-09-13T14:52:11.574 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.574 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.574 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:52:11.575 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:09.608987+0000", 2024-09-13T14:52:11.575 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.003197175 2024-09-13T14:52:11.575 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.575 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.575 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:52:11.575 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:09.609111+0000", 2024-09-13T14:52:11.575 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012362000000000001 2024-09-13T14:52:11.575 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.575 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.575 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:52:11.575 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:09.609230+0000", 2024-09-13T14:52:11.575 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00011934099999999999 2024-09-13T14:52:11.575 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.576 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.576 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:52:11.576 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:09.612252+0000", 2024-09-13T14:52:11.576 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0030220580000000002 2024-09-13T14:52:11.576 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.576 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.576 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.576 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:09.624463+0000", 2024-09-13T14:52:11.576 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.012210441000000001 2024-09-13T14:52:11.576 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.576 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.576 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.576 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:09.780653+0000", 2024-09-13T14:52:11.577 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.15618985899999999 2024-09-13T14:52:11.577 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.577 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.577 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:52:11.577 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:09.780698+0000", 2024-09-13T14:52:11.577 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5843000000000001e-05 2024-09-13T14:52:11.577 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.577 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.577 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:52:11.577 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:09.780739+0000", 2024-09-13T14:52:11.577 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0750000000000001e-05 2024-09-13T14:52:11.577 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.578 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:52:11.578 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.578 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.578 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.578 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:216105 7.5 7:a2362ad5:::benchmark_data_smithi151_190906_object95644:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1162)", 2024-09-13T14:52:11.578 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:48:09.626101+0000", 2024-09-13T14:52:11.578 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 241.81818098299999, 2024-09-13T14:52:11.578 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.15478018900000001, 2024-09-13T14:52:11.578 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:52:11.578 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:11.578 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:52:11.578 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:52:11.578 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:52:11.578 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 216105 2024-09-13T14:52:11.579 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.579 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:52:11.579 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.579 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:52:11.579 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:09.626101+0000", 2024-09-13T14:52:11.579 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.579 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.579 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.579 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:52:11.579 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:09.626101+0000", 2024-09-13T14:52:11.579 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.579 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.579 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.579 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:52:11.579 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:09.626099+0000", 2024-09-13T14:52:11.580 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:52:11.580 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.580 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.580 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:52:11.580 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:09.626106+0000", 2024-09-13T14:52:11.580 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.2359999999999998e-06 2024-09-13T14:52:11.580 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.580 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.580 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:52:11.580 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:09.626120+0000", 2024-09-13T14:52:11.580 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.394e-05 2024-09-13T14:52:11.580 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.580 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.581 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:52:11.581 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:09.626126+0000", 2024-09-13T14:52:11.581 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.57e-06 2024-09-13T14:52:11.581 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.581 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.581 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:52:11.581 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:09.626156+0000", 2024-09-13T14:52:11.581 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.0598000000000001e-05 2024-09-13T14:52:11.581 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.581 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.581 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:52:11.581 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:09.626246+0000", 2024-09-13T14:52:11.581 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.9306000000000002e-05 2024-09-13T14:52:11.582 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.582 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.582 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:52:11.582 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:09.626325+0000", 2024-09-13T14:52:11.582 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.9047000000000004e-05 2024-09-13T14:52:11.582 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.582 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.582 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:52:11.582 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:09.628944+0000", 2024-09-13T14:52:11.582 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0026193129999999998 2024-09-13T14:52:11.582 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.582 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.582 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.582 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:09.633312+0000", 2024-09-13T14:52:11.583 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.004367976 2024-09-13T14:52:11.583 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.583 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.583 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.583 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:09.780797+0000", 2024-09-13T14:52:11.583 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.14748456200000001 2024-09-13T14:52:11.583 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.583 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.583 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:52:11.583 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:09.780842+0000", 2024-09-13T14:52:11.583 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5512000000000001e-05 2024-09-13T14:52:11.583 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.583 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.584 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:52:11.584 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:09.780881+0000", 2024-09-13T14:52:11.584 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8609000000000003e-05 2024-09-13T14:52:11.584 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.584 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:52:11.584 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.584 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.584 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.584 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:234454 7.5 7:afbc8f1d:::benchmark_data_smithi151_190906_object113993:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1164)", 2024-09-13T14:52:11.584 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:48:40.632040+0000", 2024-09-13T14:52:11.584 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 210.81224124799999, 2024-09-13T14:52:11.584 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.14530120699999999, 2024-09-13T14:52:11.584 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:52:11.585 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:11.585 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:52:11.585 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:52:11.585 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:52:11.585 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 234454 2024-09-13T14:52:11.585 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.585 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:52:11.585 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.585 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:52:11.585 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:40.632040+0000", 2024-09-13T14:52:11.585 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.585 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.585 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.585 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:52:11.586 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:40.632040+0000", 2024-09-13T14:52:11.586 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:52:11.586 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.586 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.586 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:52:11.586 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:40.632039+0000", 2024-09-13T14:52:11.586 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:52:11.586 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.586 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.586 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:52:11.586 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:40.632047+0000", 2024-09-13T14:52:11.586 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.1289999999999996e-06 2024-09-13T14:52:11.586 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.587 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.587 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:52:11.587 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:40.632062+0000", 2024-09-13T14:52:11.587 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4891e-05 2024-09-13T14:52:11.587 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.587 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.587 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:52:11.587 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:40.632068+0000", 2024-09-13T14:52:11.587 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1199999999999999e-06 2024-09-13T14:52:11.587 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.587 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.587 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:52:11.587 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:40.632110+0000", 2024-09-13T14:52:11.587 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.2691000000000002e-05 2024-09-13T14:52:11.588 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.588 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.588 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:52:11.588 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:40.632195+0000", 2024-09-13T14:52:11.588 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.4889999999999995e-05 2024-09-13T14:52:11.588 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.588 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.588 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:52:11.588 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:40.632275+0000", 2024-09-13T14:52:11.588 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.9603000000000003e-05 2024-09-13T14:52:11.588 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.588 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.588 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:52:11.589 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:40.634742+0000", 2024-09-13T14:52:11.589 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0024669840000000002 2024-09-13T14:52:11.589 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.589 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.589 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.589 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:40.635411+0000", 2024-09-13T14:52:11.589 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00066869800000000001 2024-09-13T14:52:11.589 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.589 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.589 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:11.589 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:40.777277+0000", 2024-09-13T14:52:11.590 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.14186668599999999 2024-09-13T14:52:11.590 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.590 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.590 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:52:11.590 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:40.777314+0000", 2024-09-13T14:52:11.590 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.6594000000000002e-05 2024-09-13T14:52:11.590 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:52:11.590 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:52:11.590 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:52:11.590 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:40.777342+0000", 2024-09-13T14:52:11.590 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.7622000000000001e-05 2024-09-13T14:52:11.590 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.590 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:52:11.591 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.591 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:52:11.591 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:52:11.591 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:52:11.660 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:11.682 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:11.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:11.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:11.784 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:11.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:11.806 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:11.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:11.836 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:11 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:52:11.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:11 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:52:11.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:11 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:52:11.908 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:11.929 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:11.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:11.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:12.031 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:12.053 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:12.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:12.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:12.155 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:12.176 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:12.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:12.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:12.278 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:12.299 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:12.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:12.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:12.355 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:52:12.402 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:12.425 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:12.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:12.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:12.527 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:12.550 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:12.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:12.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:12.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:12 smithi184 ceph-mon[92527]: osdmap e1189: 8 total, 8 up, 7 in 2024-09-13T14:52:12.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:12 smithi184 ceph-mon[92527]: pgmap v1778: 119 pgs: 3 active+remapped+backfilling, 116 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 706 op/s; 720/216531 objects misplaced (0.333%); 32 MiB/s, 508 objects/s recovering 2024-09-13T14:52:12.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:12 smithi151 ceph-mon[104450]: osdmap e1189: 8 total, 8 up, 7 in 2024-09-13T14:52:12.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:12 smithi151 ceph-mon[104450]: pgmap v1778: 119 pgs: 3 active+remapped+backfilling, 116 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 706 op/s; 720/216531 objects misplaced (0.333%); 32 MiB/s, 508 objects/s recovering 2024-09-13T14:52:12.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:12 smithi151 ceph-mon[108966]: osdmap e1189: 8 total, 8 up, 7 in 2024-09-13T14:52:12.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:12 smithi151 ceph-mon[108966]: pgmap v1778: 119 pgs: 3 active+remapped+backfilling, 116 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 706 op/s; 720/216531 objects misplaced (0.333%); 32 MiB/s, 508 objects/s recovering 2024-09-13T14:52:12.652 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:12.689 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:12.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:12.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:12.791 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:12.813 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:12.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:12.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:12.915 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:12.938 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:12.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:12.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:13.040 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:13.061 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:13.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:13.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:13.163 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:13.185 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:13.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:13.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:13.287 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:13.308 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:13.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:13.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:13.410 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:13.431 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:13.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:13.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:13.533 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:13.557 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:13.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:13.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:13.659 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:13.690 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:13.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:13.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:13.792 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:13.819 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:13.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:13.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:13.822 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:13 smithi184 ceph-mon[92527]: osdmap e1190: 8 total, 8 up, 7 in 2024-09-13T14:52:13.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:13 smithi151 ceph-mon[104450]: osdmap e1190: 8 total, 8 up, 7 in 2024-09-13T14:52:13.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:13 smithi151 ceph-mon[108966]: osdmap e1190: 8 total, 8 up, 7 in 2024-09-13T14:52:13.921 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:13.949 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:13.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:13.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:14.052 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:14.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:14.093 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:14.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:14.194 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:14.223 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:14.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:14.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:14.325 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:14.356 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:14.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:14.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:14.460 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:14.490 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:14.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:14.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:14.592 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:14.624 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:14.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:14.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:14.727 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:14.771 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:14.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:14.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:14.801 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:52:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:52:14] "GET /metrics HTTP/1.1" 200 36681 "" "Prometheus/2.51.0" 2024-09-13T14:52:14.801 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:14 smithi151 ceph-mon[108966]: osdmap e1191: 8 total, 8 up, 7 in 2024-09-13T14:52:14.802 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:14 smithi151 ceph-mon[108966]: pgmap v1781: 119 pgs: 3 active+remapped+backfilling, 116 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 859 op/s; 720/210948 objects misplaced (0.341%); 35 MiB/s, 555 objects/s recovering 2024-09-13T14:52:14.802 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:14 smithi151 ceph-mon[104450]: osdmap e1191: 8 total, 8 up, 7 in 2024-09-13T14:52:14.802 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:14 smithi151 ceph-mon[104450]: pgmap v1781: 119 pgs: 3 active+remapped+backfilling, 116 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 859 op/s; 720/210948 objects misplaced (0.341%); 35 MiB/s, 555 objects/s recovering 2024-09-13T14:52:14.810 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:14.813 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:14.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:14 smithi184 ceph-mon[92527]: osdmap e1191: 8 total, 8 up, 7 in 2024-09-13T14:52:14.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:14 smithi184 ceph-mon[92527]: pgmap v1781: 119 pgs: 3 active+remapped+backfilling, 116 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 859 op/s; 720/210948 objects misplaced (0.341%); 35 MiB/s, 555 objects/s recovering 2024-09-13T14:52:14.873 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:14.882 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:52:14.882 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:52:14.897 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:14.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:14.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:14.919 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:52:14.920 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:52:14.950 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:52:14.951 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:52:14.996 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:52:14.996 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:52:14.999 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:15.024 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:52:15.025 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:52:15.049 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:15.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:15.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:15.063 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:52:15.063 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:52:15.099 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:52:15.099 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:52:15.128 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:52:15.128 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:52:15.151 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:15.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:15.199 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:15.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:15.300 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:15.338 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:15.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:15.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:15.440 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:15.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:15.488 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:15.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:15.589 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:15.626 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:15.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:15.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:15.676 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:15 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/47018932' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:15.676 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:15 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/47018932' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:15.682 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:15 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/47018932' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:15.728 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:15.769 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:15.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:15.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:15.871 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:15.908 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:15.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:15.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:16.010 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:16.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:16.054 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:16.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:16.156 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:16.186 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:16.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:16.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:16.239 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:52:16.288 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:16.311 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:16.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:16.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:16.412 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:16.443 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:16.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:16.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:16.545 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:16.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:16.597 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:16.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:16.698 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:16.723 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:16.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:16.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:16.825 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:16.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:16 smithi184 ceph-mon[92527]: pgmap v1782: 119 pgs: 119 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 811 op/s; 31 MiB/s, 497 objects/s recovering 2024-09-13T14:52:16.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:52:16.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:16.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:16 smithi151 ceph-mon[104450]: pgmap v1782: 119 pgs: 119 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 811 op/s; 31 MiB/s, 497 objects/s recovering 2024-09-13T14:52:16.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:52:16.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:16.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:16 smithi151 ceph-mon[108966]: pgmap v1782: 119 pgs: 119 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 811 op/s; 31 MiB/s, 497 objects/s recovering 2024-09-13T14:52:16.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:52:16.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:16.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:16.865 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:16.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:16.966 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:16.988 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:16.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:16.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:17.090 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:17.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:17.125 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:17.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:17.226 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:17.262 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:17.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:17.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:17.364 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:17.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:17.402 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:17.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:17.503 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:17.558 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:17.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:17.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:17.660 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:17.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:17.685 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:17.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:17.786 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:17.836 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:17.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:17.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:17.938 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:17.971 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:17.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:17.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:18.015 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:52:18.015 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:52:18.015 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:52:18.016 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:52:18.073 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:18.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:18.106 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:18.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:18.207 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:18.246 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:18.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:18.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:18.348 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:18.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:52:18.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:52:18.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:18.371 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:18.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:18.472 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:18.507 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:18.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:18.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:18.609 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:18.645 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:18.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:18.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:18.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:18.782 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:18.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:18.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:18.884 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:18.917 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:18.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:18.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:18.982 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:52:19.019 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:19.063 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:19.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:19.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:19.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:19.207 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:19.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:19.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:19.310 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:19.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:19.336 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:19.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:19.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:18 smithi184 ceph-mon[92527]: pgmap v1783: 119 pgs: 119 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 527 op/s; 7.3 MiB/s, 116 objects/s recovering 2024-09-13T14:52:19.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:18 smithi151 ceph-mon[104450]: pgmap v1783: 119 pgs: 119 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 527 op/s; 7.3 MiB/s, 116 objects/s recovering 2024-09-13T14:52:19.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:18 smithi151 ceph-mon[108966]: pgmap v1783: 119 pgs: 119 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 527 op/s; 7.3 MiB/s, 116 objects/s recovering 2024-09-13T14:52:19.437 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:19.465 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:19.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:19.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:19.567 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:19.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:19.599 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:19.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:19.700 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:19.732 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:19.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:19.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:19.834 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:19.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:19.875 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:19.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:19.976 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:20.000 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:20.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:20.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:20.102 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:20.144 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:20.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:20.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:20.246 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:20.272 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:20.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:20.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:20.354 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:52:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:52:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:52:20.374 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:20.402 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:20.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:20.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:20.505 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:20.554 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:20.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:20.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:20.657 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:20.681 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:20.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:20.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:20.783 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:20.826 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:20.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:20.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:20.928 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:20.964 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:20.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:20.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:21.066 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:21.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:21.106 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:21.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:21.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:20 smithi151 ceph-mon[104450]: pgmap v1784: 119 pgs: 119 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 735 op/s; 5.5 MiB/s, 88 objects/s recovering 2024-09-13T14:52:21.165 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:20 smithi151 ceph-mon[108966]: pgmap v1784: 119 pgs: 119 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 735 op/s; 5.5 MiB/s, 88 objects/s recovering 2024-09-13T14:52:21.207 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:21.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:21.231 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:21.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:21.332 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:21.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:20 smithi184 ceph-mon[92527]: pgmap v1784: 119 pgs: 119 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 735 op/s; 5.5 MiB/s, 88 objects/s recovering 2024-09-13T14:52:21.355 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:21.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:21.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:21.457 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:21.479 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:21.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:21.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:21.581 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:21.607 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:21.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:21.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:21.709 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:21.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:21.733 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:21.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:21.834 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:21.859 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:21.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:21.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:21.962 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:21.999 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:21.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:22.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:22.007 INFO:teuthology.orchestra.run.smithi151.stderr:set pool 8 pgp_num to 16 2024-09-13T14:52:22.100 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:22.124 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:22.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:22.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:22.226 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:22.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:22.254 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:22.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:22.343 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:52:22.343 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:52:22.344 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:22 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2614990540' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T14:52:22.344 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:22 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:52:22.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:22 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2614990540' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T14:52:22.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:22 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:52:22.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:22 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2614990540' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T14:52:22.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:22 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:52:22.356 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:22.377 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:22.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:22.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:22.479 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:22.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:22.516 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:22.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:22.617 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:22.649 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:22.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:22.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:22.751 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:22.779 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:22.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:22.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:22.881 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:22.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:22.925 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:22.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:23.026 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:23.059 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:23.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:23.161 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:23.198 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:23.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:23.300 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:23.326 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:23.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:23.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:23 smithi184 ceph-mon[92527]: pgmap v1785: 119 pgs: 119 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 892 op/s; 4.9 MiB/s, 77 objects/s recovering 2024-09-13T14:52:23.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:23 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2614990540' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]': finished 2024-09-13T14:52:23.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:23 smithi184 ceph-mon[92527]: osdmap e1192: 8 total, 8 up, 7 in 2024-09-13T14:52:23.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:23 smithi151 ceph-mon[104450]: pgmap v1785: 119 pgs: 119 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 892 op/s; 4.9 MiB/s, 77 objects/s recovering 2024-09-13T14:52:23.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:23 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2614990540' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]': finished 2024-09-13T14:52:23.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:23 smithi151 ceph-mon[104450]: osdmap e1192: 8 total, 8 up, 7 in 2024-09-13T14:52:23.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:23 smithi151 ceph-mon[108966]: pgmap v1785: 119 pgs: 119 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 892 op/s; 4.9 MiB/s, 77 objects/s recovering 2024-09-13T14:52:23.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:23 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2614990540' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]': finished 2024-09-13T14:52:23.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:23 smithi151 ceph-mon[108966]: osdmap e1192: 8 total, 8 up, 7 in 2024-09-13T14:52:23.429 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:23.451 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T14:52:23.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.452 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:23.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:23.553 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:23.592 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:23.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:23.694 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:23.710 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:23.717 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:23.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:23.819 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:23.845 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:23.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:23.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:23.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.986 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:23.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:24.087 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:24.129 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:24.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:24.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:24.182 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:24.231 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:24.255 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:24.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:24.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:24.357 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:24.406 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:24.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:24.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:24.508 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:24.534 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:24.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:24.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:24.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:24.677 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:24.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:24.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:24.727 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:52:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:52:24] "GET /metrics HTTP/1.1" 200 36676 "" "Prometheus/2.51.0" 2024-09-13T14:52:24.779 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:24.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:24.805 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:24.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:24.906 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:24.948 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:24.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:24.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:25.050 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:25.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:25.088 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:25.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:25.189 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:25.215 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:25.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:25.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:25.318 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:25.342 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:25.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:25.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:25.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:25 smithi184 ceph-mon[92527]: pgmap v1787: 119 pgs: 119 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 908 op/s; 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:52:25.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:25 smithi151 ceph-mon[104450]: pgmap v1787: 119 pgs: 119 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 908 op/s; 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:52:25.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:25 smithi151 ceph-mon[108966]: pgmap v1787: 119 pgs: 119 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 908 op/s; 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:52:25.444 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:25.474 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:25.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:25.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:25.576 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:25.602 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:25.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:25.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:25.704 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:25.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:25.739 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:25.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:25.840 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:25.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:25.871 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:25.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:25.972 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:25.996 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:25.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:25.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:26.098 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:26.141 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:26.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:26.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:26.243 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:26.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:26.304 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:26.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:26.406 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:26.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:26.439 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:26.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:26.540 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:26.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:26.574 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:26.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:26.675 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:26.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:26.705 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:26.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:26.805 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:26.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:26.844 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:26.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:26.945 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:26.971 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:26.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:26.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:27.026 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:27.072 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:27.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:27.145 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:27.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:27.246 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:27.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:27.275 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:27.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:27.333 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:27 smithi151 ceph-mon[104450]: pgmap v1788: 119 pgs: 119 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s 2024-09-13T14:52:27.333 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:27 smithi151 ceph-mon[108966]: pgmap v1788: 119 pgs: 119 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s 2024-09-13T14:52:27.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:27 smithi184 ceph-mon[92527]: pgmap v1788: 119 pgs: 119 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s 2024-09-13T14:52:27.376 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:27.419 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:27.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:27.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:27.521 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:27.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:27.566 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:27.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:27.667 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:27.688 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:27.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:27.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:27.790 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:27.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:27.831 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:27.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:27.932 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:27.972 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:27.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:27.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:28.074 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:28.101 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:28.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:28.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:28.203 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:28.235 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:28.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:28.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:28.337 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:28.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:28 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/710595801' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:28.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:28 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:28.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:28 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/710595801' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:28.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:28 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:28.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:28 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/710595801' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:28.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:28 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:28.366 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:28.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:28.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:28.468 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:28.507 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:28.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:28.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:28.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:28.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:28.635 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:28.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:28.736 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:28.771 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:28.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:28.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:28.873 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:28.895 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:28.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:28.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:28.997 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:29.037 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:29.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:29.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:29.139 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:29.167 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:29.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:29.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:29.228 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 6, 7, 1] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:52:29.228 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:52:29.228 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-13T14:52:29.228 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:52:29.228 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:52:29.229 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-13T14:52:29.269 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:29.305 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:29.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:29.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:29.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:29 smithi184 ceph-mon[92527]: pgmap v1789: 119 pgs: 119 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s 2024-09-13T14:52:29.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:29 smithi184 ceph-mon[92527]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:29.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:29 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:52:29.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:29 smithi184 ceph-mon[92527]: osdmap e1193: 8 total, 8 up, 7 in 2024-09-13T14:52:29.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:29 smithi151 ceph-mon[104450]: pgmap v1789: 119 pgs: 119 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s 2024-09-13T14:52:29.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:29 smithi151 ceph-mon[104450]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:29.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:29 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:52:29.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:29 smithi151 ceph-mon[104450]: osdmap e1193: 8 total, 8 up, 7 in 2024-09-13T14:52:29.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:29 smithi151 ceph-mon[108966]: pgmap v1789: 119 pgs: 119 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s 2024-09-13T14:52:29.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:29 smithi151 ceph-mon[108966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:29.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:29 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:52:29.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:29 smithi151 ceph-mon[108966]: osdmap e1193: 8 total, 8 up, 7 in 2024-09-13T14:52:29.408 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:29.432 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:29.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:29.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:29.534 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:29.570 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:29.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:29.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:29.585 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:52:29.672 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:29.703 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:29.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:29.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:29.805 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:29.837 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:29.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:29.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:29.939 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:29.970 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:29.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:29.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:30.073 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:30.101 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:30.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:30.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:30.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:30.239 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:30.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:30.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:30.331 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:52:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:52:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:52:30.340 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:30.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:30.377 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:30.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:30.479 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:30.514 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:30.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:30.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:30.616 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:30.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:30.649 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:30.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:30.666 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-13T14:52:30.751 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:30.791 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:30.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:30.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:30.893 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:30.937 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:30.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:30.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:31.040 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:31.073 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:31.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:31.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:31.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:31 smithi184 ceph-mon[92527]: pgmap v1791: 119 pgs: 119 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s 2024-09-13T14:52:31.175 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:31.207 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:31.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:31.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:31.309 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:31.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:31.350 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:31.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:31.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:31 smithi151 ceph-mon[104450]: pgmap v1791: 119 pgs: 119 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s 2024-09-13T14:52:31.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:31 smithi151 ceph-mon[108966]: pgmap v1791: 119 pgs: 119 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s 2024-09-13T14:52:31.452 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:31.727 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:52:31.727 INFO:teuthology.orchestra.run.smithi184.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T14:52:31.727 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:52:31.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:31.764 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:31.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:31.865 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:31.904 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:31.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:31.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:32.005 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:32.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:32.035 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:32.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:32.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:32.186 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:32.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:32.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:32.288 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:32.314 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:32.314 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:32.323 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:32.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:32.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:32.426 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:32.465 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:32.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:32.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:32.568 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:32.607 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:32.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:32.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:32.709 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:32.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:32.750 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:32.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:32.852 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:32.884 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:32.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:32.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:32.986 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:33.025 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:33.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:33.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:33.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:33 smithi151 ceph-mon[104450]: pgmap v1792: 119 pgs: 119 active+clean; 3.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-13T14:52:33.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:33 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2246203696' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:33.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:33 smithi151 ceph-mon[108966]: pgmap v1792: 119 pgs: 119 active+clean; 3.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-13T14:52:33.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:33 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2246203696' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:33.127 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:33.139 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:52:33.139 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:52:33.140 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:52:33.140 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:52:33.140 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:52:33.140 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:52:33.140 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:52:33.140 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:52:33.140 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:52:33.140 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:52:33.140 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:52:33.140 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:52:33.141 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:52:33.141 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:52:33.141 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:52:33.141 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:52:33.169 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:33.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:33.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:33.270 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:33.310 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:33.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:33.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:33.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:33 smithi184 ceph-mon[92527]: pgmap v1792: 119 pgs: 119 active+clean; 3.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-13T14:52:33.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:33 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2246203696' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:33.411 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:33.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:33.458 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:33.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:33.559 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:33.595 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:33.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:33.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:33.696 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:33.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:33.731 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:33.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:33.831 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:52:33.834 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:33.925 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:33.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:33.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:34.026 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:34.052 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:34.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:34.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:34.154 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:34.182 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:34.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:34.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:34.283 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:34.320 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:34.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:34.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:34.422 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:34.461 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:34.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:34.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:34.563 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:34.608 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:34.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:34.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:34.627 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:52:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:52:34] "GET /metrics HTTP/1.1" 200 36659 "" "Prometheus/2.51.0" 2024-09-13T14:52:34.710 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:34.732 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:34.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:34.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:34.835 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:34.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:34.882 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:34.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:34.983 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:35.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:35.009 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:35.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:35.110 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:35.159 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:35.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:35.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:35.189 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:35 smithi151 ceph-mon[108966]: pgmap v1793: 119 pgs: 119 active+clean; 3.1 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-13T14:52:35.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:35 smithi151 ceph-mon[104450]: pgmap v1793: 119 pgs: 119 active+clean; 3.1 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-13T14:52:35.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:35.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:35.308 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:35.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:35.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:35 smithi184 ceph-mon[92527]: pgmap v1793: 119 pgs: 119 active+clean; 3.1 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-13T14:52:35.409 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:35.456 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:35.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:35.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:35.510 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:52:35.510 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-13T14:52:35.511 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-13T14:52:35.511 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-13T14:52:35.511 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:52:35.558 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:35.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:35.597 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:35.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:35.699 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:35.725 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:35.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:35.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:35.827 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:35.851 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:35.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:35.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:35.952 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:36.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:36.001 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:36.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:36.103 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:36.126 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:36.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:36.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:36.228 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:36.251 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:36.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:36.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:36.353 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:36.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:36.397 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:36.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:36.498 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:36.525 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:36.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:36.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:36.554 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T14:52:36.627 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:36.651 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:36.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:36.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:36.663 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:36.670 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:52:36.753 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:36.776 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:36.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:36.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:36.878 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:36.940 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:36.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:36.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:37.042 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:37.063 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:37.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:37.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:37.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:37 smithi151 ceph-mon[104450]: pgmap v1794: 119 pgs: 119 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-13T14:52:37.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:37 smithi151 ceph-mon[108966]: pgmap v1794: 119 pgs: 119 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-13T14:52:37.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:37.207 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:37.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:37.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:37.308 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:37.350 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:37.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:37.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:37.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:37 smithi184 ceph-mon[92527]: pgmap v1794: 119 pgs: 119 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-13T14:52:37.424 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 6, 7, 1] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:52:37.424 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:52:37.424 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 2, 3, 5, 6, 7, 1] 2024-09-13T14:52:37.424 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd out 5 2024-09-13T14:52:37.453 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:37.511 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:37.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:37.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:37.613 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:37.638 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:37.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:37.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:37.739 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:37.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:37.798 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:37.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:37.899 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:37.925 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:37.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:37.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:38.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:38.074 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:38.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:38.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:38.176 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:38.199 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:38.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:38.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:38.301 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:38.339 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:38.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:38.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:38.440 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:38.480 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:38.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:38.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:38.582 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:38.611 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:38.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:38.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:38.713 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:38.748 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:38.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:38.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:38.850 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:38.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:38.879 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:38.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:38.980 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:39.015 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:39.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:39.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:39.117 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:39.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:39.151 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:39.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:39.252 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:39.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:39.277 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:39.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:39.324 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:39 smithi151 ceph-mon[104450]: pgmap v1795: 119 pgs: 119 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-13T14:52:39.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:39 smithi151 ceph-mon[108966]: pgmap v1795: 119 pgs: 119 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-13T14:52:39.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:39 smithi184 ceph-mon[92527]: pgmap v1795: 119 pgs: 119 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-13T14:52:39.379 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:39.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:39.437 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:39.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:39.538 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:39.564 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:39.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:39.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:39.666 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:39.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:39.713 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:39.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:39.814 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:39.841 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:39.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:39.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:39.943 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:39.970 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:39.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:39.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:40.072 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:40.107 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:40.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:40.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:40.208 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:40.254 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:40.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:40.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:40.342 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:40.356 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:40.414 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:40 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:52:40.414 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:40 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:52:40.414 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:40 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:52:40.414 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:40 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:52:40.415 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:40 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:52:40.415 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:40 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:52:40.415 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:40 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:52:40.415 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:40 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:52:40.415 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:40 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:52:40.415 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:40 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:52:40.419 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:40.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:40.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:40.521 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:40.550 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:40.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:40.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:40.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:40 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:52:40.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:40 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:52:40.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:40 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:52:40.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:40 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:52:40.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:40 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:52:40.605 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:52:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:52:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:52:40.652 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:40.678 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:40.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:40.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:40.780 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:40.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:40.822 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:40.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:40.896 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:40.901 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:40.923 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:40.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:40.947 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:40.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:41.048 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:41.076 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:41.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:41.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:41.178 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:41.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:41.202 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:41.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:41.303 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:41.329 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:41.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:41.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:41.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:41 smithi151 ceph-mon[104450]: pgmap v1796: 119 pgs: 119 active+clean; 2.8 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 999 op/s 2024-09-13T14:52:41.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:41 smithi151 ceph-mon[108966]: pgmap v1796: 119 pgs: 119 active+clean; 2.8 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 999 op/s 2024-09-13T14:52:41.430 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:41.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:41.472 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:41.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:41.573 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:41.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:41 smithi184 ceph-mon[92527]: pgmap v1796: 119 pgs: 119 active+clean; 2.8 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 999 op/s 2024-09-13T14:52:41.609 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:41.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:41.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:41.711 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:41.732 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:41.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:41.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:41.834 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:41.872 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:41.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:41.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:41.974 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:42.001 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:42.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:42.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:42.103 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:42.147 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:42.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:42.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:42.242 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-13T14:52:42.249 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:42.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:42.281 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:42.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:42.383 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:42.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:42.405 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:42.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:42.509 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:42.544 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:42.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:42.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:42.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:42 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1721409431' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:52:42.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:42 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:52:42.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:42 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1721409431' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:52:42.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:42 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:52:42.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:42 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1721409431' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:52:42.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:42 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:52:42.646 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:42.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:42.681 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:42.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:42.781 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:42.818 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:42.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:42.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:42.921 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:42.950 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:42.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:42.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:43.052 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:43.076 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:43.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:43.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:43.178 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:43.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:43.213 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:43.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:43.314 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:43.341 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:43.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:43.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:43.443 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:43.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:43.480 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:43.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:43.581 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:43.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:43 smithi184 ceph-mon[92527]: pgmap v1797: 119 pgs: 119 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-13T14:52:43.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:43 smithi184 ceph-mon[92527]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:43.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:43 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:52:43.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:43 smithi184 ceph-mon[92527]: osdmap e1194: 8 total, 8 up, 7 in 2024-09-13T14:52:43.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:43 smithi151 ceph-mon[104450]: pgmap v1797: 119 pgs: 119 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-13T14:52:43.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:43 smithi151 ceph-mon[104450]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:43.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:43 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:52:43.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:43 smithi151 ceph-mon[104450]: osdmap e1194: 8 total, 8 up, 7 in 2024-09-13T14:52:43.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:43 smithi151 ceph-mon[108966]: pgmap v1797: 119 pgs: 119 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-13T14:52:43.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:43 smithi151 ceph-mon[108966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:43.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:43 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:52:43.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:43 smithi151 ceph-mon[108966]: osdmap e1194: 8 total, 8 up, 7 in 2024-09-13T14:52:43.613 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:43.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:43.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:43.715 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:43.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:43.740 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:43.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:43.841 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:43.880 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:43.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:43.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:43.981 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:44.007 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:44.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:44.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:44.109 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:44.137 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:44.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:44.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:44.239 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:44.266 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:44.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:44.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:44.368 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:44.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:44.407 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:44.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:44.508 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:44.543 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:44.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:44.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:44.592 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:44 smithi151 ceph-mon[104450]: 1.e scrub starts 2024-09-13T14:52:44.592 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:44 smithi151 ceph-mon[104450]: 1.e scrub ok 2024-09-13T14:52:44.592 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:44 smithi151 ceph-mon[108966]: 1.e scrub starts 2024-09-13T14:52:44.592 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:44 smithi151 ceph-mon[108966]: 1.e scrub ok 2024-09-13T14:52:44.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:44 smithi184 ceph-mon[92527]: 1.e scrub starts 2024-09-13T14:52:44.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:44 smithi184 ceph-mon[92527]: 1.e scrub ok 2024-09-13T14:52:44.645 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:44.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:44.675 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:44.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:44.776 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:44.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:44.804 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:44.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:44.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:52:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:52:44] "GET /metrics HTTP/1.1" 200 36659 "" "Prometheus/2.51.0" 2024-09-13T14:52:44.906 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:44.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:44.942 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:44.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:45.043 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:45.083 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:45.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:45.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:45.186 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:45.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:45.211 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:45.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:45.313 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:45.351 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:45.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:45.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:45.453 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:45.476 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:45.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:45.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:45.492 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:45 smithi151 ceph-mon[104450]: pgmap v1799: 119 pgs: 119 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 990 op/s 2024-09-13T14:52:45.492 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:45 smithi151 ceph-mon[108966]: pgmap v1799: 119 pgs: 119 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 990 op/s 2024-09-13T14:52:45.577 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:45.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:45.600 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:45.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:45.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:45 smithi184 ceph-mon[92527]: pgmap v1799: 119 pgs: 119 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 990 op/s 2024-09-13T14:52:45.701 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:45.727 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:45.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:45.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:45.829 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:45.863 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:45.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:45.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:45.965 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:45.990 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:45.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:45.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:46.093 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:46.129 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:46.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:46.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:46.231 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:46.259 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:46.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:46.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:46.361 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:46.399 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:46.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:46.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:46.501 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:46.541 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:46.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:46.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:46.642 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:46.686 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:46.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:46.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:46.788 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:46.813 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:46.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:46.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:46.915 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:46.940 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:46.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:46.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:47.042 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:47.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:47.070 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:47.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:47.171 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:47.201 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:47.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:47.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:47.302 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:47.337 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:47.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:47.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:47.425 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:47 smithi151 ceph-mon[104450]: pgmap v1800: 119 pgs: 119 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 972 op/s 2024-09-13T14:52:47.425 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:47 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:47.425 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:47 smithi151 ceph-mon[108966]: pgmap v1800: 119 pgs: 119 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 972 op/s 2024-09-13T14:52:47.425 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:47 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:47.439 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:47.475 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:47.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:47.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:47.577 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:47.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:47 smithi184 ceph-mon[92527]: pgmap v1800: 119 pgs: 119 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 972 op/s 2024-09-13T14:52:47.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:47 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:47.613 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:47.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:47.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:47.715 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:47.746 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:47.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:47.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:47.848 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:47.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:47.882 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:47.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:47.982 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:48.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:48.010 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:48.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:48.111 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:48.148 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:48.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:48.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:48.243 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:48 smithi151 ceph-mon[104450]: 1.1a scrub starts 2024-09-13T14:52:48.243 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:48 smithi151 ceph-mon[104450]: 1.1a scrub ok 2024-09-13T14:52:48.244 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:48 smithi151 ceph-mon[108966]: 1.1a scrub starts 2024-09-13T14:52:48.244 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:48 smithi151 ceph-mon[108966]: 1.1a scrub ok 2024-09-13T14:52:48.250 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:48.295 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:48.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:48.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:48.397 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:48.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:48.425 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:48.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:48.504 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:48 smithi184 ceph-mon[92527]: 1.1a scrub starts 2024-09-13T14:52:48.504 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:48 smithi184 ceph-mon[92527]: 1.1a scrub ok 2024-09-13T14:52:48.526 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:48.563 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:48.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:48.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:48.665 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:48.689 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:48.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:48.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:48.791 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:48.817 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:48.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:48.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:48.918 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:52:48.918 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-13T14:52:48.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-13T14:52:48.918 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-13T14:52:48.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.918 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:257683 6.10 6:0d4d0d4b:::benchmark_data_smithi151_155836_object125225:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1127)", 2024-09-13T14:52:48.918 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:13.538763+0000", 2024-09-13T14:52:48.918 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 575.37371729500001, 2024-09-13T14:52:48.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1550639309999999, 2024-09-13T14:52:48.919 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:52:48.919 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:52:48.919 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-13T14:52:48.919 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:52:48.919 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:52:48.919 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:52:48.919 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 257683 2024-09-13T14:52:48.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.919 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:52:48.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:52:48.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.538763+0000", 2024-09-13T14:52:48.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:48.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:52:48.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.538763+0000", 2024-09-13T14:52:48.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:48.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:52:48.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.538764+0000", 2024-09-13T14:52:48.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8059999999999999e-06 2024-09-13T14:52:48.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.921 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:52:48.921 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.538772+0000", 2024-09-13T14:52:48.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9610000000000005e-06 2024-09-13T14:52:48.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.921 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:52:48.921 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.538799+0000", 2024-09-13T14:52:48.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6308999999999998e-05 2024-09-13T14:52:48.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.921 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:52:48.921 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.538808+0000", 2024-09-13T14:52:48.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5210000000000008e-06 2024-09-13T14:52:48.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:48.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.538879+0000", 2024-09-13T14:52:48.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1119999999999997e-05 2024-09-13T14:52:48.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:52:48.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.539045+0000", 2024-09-13T14:52:48.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016596100000000001 2024-09-13T14:52:48.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:52:48.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.539664+0000", 2024-09-13T14:52:48.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00061883100000000003 2024-09-13T14:52:48.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:52:48.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.542365+0000", 2024-09-13T14:52:48.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027009109999999999 2024-09-13T14:52:48.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:48.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.543024+0000", 2024-09-13T14:52:48.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00065861000000000001 2024-09-13T14:52:48.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:48.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.549078+0000", 2024-09-13T14:52:48.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0060546599999999999 2024-09-13T14:52:48.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:48.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.549089+0000", 2024-09-13T14:52:48.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1021e-05 2024-09-13T14:52:48.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:48.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.551828+0000", 2024-09-13T14:52:48.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0027385710000001 2024-09-13T14:52:48.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:48.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.551840+0000", 2024-09-13T14:52:48.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2179e-05 2024-09-13T14:52:48.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:48.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.693758+0000", 2024-09-13T14:52:48.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.141917796 2024-09-13T14:52:48.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:52:48.926 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.693827+0000", 2024-09-13T14:52:48.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8674e-05 2024-09-13T14:52:48.926 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:48.926 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:52:48.926 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:48.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.926 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:257709 6.10 6:0bc2a4a6:::benchmark_data_smithi151_155836_object125251:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-13T14:52:48.926 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:13.582787+0000", 2024-09-13T14:52:48.926 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 575.32969303300001, 2024-09-13T14:52:48.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.119586478, 2024-09-13T14:52:48.927 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:52:48.927 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:52:48.927 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:48.927 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:52:48.927 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:52:48.927 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:52:48.927 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 257709 2024-09-13T14:52:48.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.927 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:52:48.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:52:48.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.582787+0000", 2024-09-13T14:52:48.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:48.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:52:48.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.582787+0000", 2024-09-13T14:52:48.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:48.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:52:48.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.582789+0000", 2024-09-13T14:52:48.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3700000000000002e-06 2024-09-13T14:52:48.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:52:48.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.582801+0000", 2024-09-13T14:52:48.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1606e-05 2024-09-13T14:52:48.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:52:48.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.582819+0000", 2024-09-13T14:52:48.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8074e-05 2024-09-13T14:52:48.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:52:48.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.582827+0000", 2024-09-13T14:52:48.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5660000000000007e-06 2024-09-13T14:52:48.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:48.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.582890+0000", 2024-09-13T14:52:48.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2322000000000004e-05 2024-09-13T14:52:48.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:48.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.582898+0000", 2024-09-13T14:52:48.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2179999999999999e-06 2024-09-13T14:52:48.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:48.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.551868+0000", 2024-09-13T14:52:48.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.96897046099999995 2024-09-13T14:52:48.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:48.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.551877+0000", 2024-09-13T14:52:48.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2940000000000006e-06 2024-09-13T14:52:48.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:48.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.694026+0000", 2024-09-13T14:52:48.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.142149455 2024-09-13T14:52:48.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:52:48.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.694107+0000", 2024-09-13T14:52:48.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0277000000000004e-05 2024-09-13T14:52:48.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:52:48.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.694558+0000", 2024-09-13T14:52:48.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000451929 2024-09-13T14:52:48.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:48.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702090+0000", 2024-09-13T14:52:48.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0075311750000000002 2024-09-13T14:52:48.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:52:48.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702111+0000", 2024-09-13T14:52:48.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1143999999999999e-05 2024-09-13T14:52:48.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:48.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702162+0000", 2024-09-13T14:52:48.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1102e-05 2024-09-13T14:52:48.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:48.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702208+0000", 2024-09-13T14:52:48.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6020000000000003e-05 2024-09-13T14:52:48.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:52:48.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702270+0000", 2024-09-13T14:52:48.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1797999999999993e-05 2024-09-13T14:52:48.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:52:48.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702373+0000", 2024-09-13T14:52:48.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103667 2024-09-13T14:52:48.934 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:48.934 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:52:48.934 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:48.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.935 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:257728 6.10 6:0d2b457a:::benchmark_data_smithi151_155836_object125270:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-13T14:52:48.935 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:13.598628+0000", 2024-09-13T14:52:48.935 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 575.31385176699996, 2024-09-13T14:52:48.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1041190269999999, 2024-09-13T14:52:48.935 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:52:48.935 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:52:48.935 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:48.935 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:52:48.935 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:52:48.935 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:52:48.935 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 257728 2024-09-13T14:52:48.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.935 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:52:48.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:52:48.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.598628+0000", 2024-09-13T14:52:48.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:48.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:52:48.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.598628+0000", 2024-09-13T14:52:48.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:48.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:52:48.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.598630+0000", 2024-09-13T14:52:48.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.874e-06 2024-09-13T14:52:48.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:52:48.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.598635+0000", 2024-09-13T14:52:48.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2780000000000001e-06 2024-09-13T14:52:48.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:52:48.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.598650+0000", 2024-09-13T14:52:48.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4844999999999999e-05 2024-09-13T14:52:48.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:52:48.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.598657+0000", 2024-09-13T14:52:48.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1169999999999998e-06 2024-09-13T14:52:48.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:48.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.598729+0000", 2024-09-13T14:52:48.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1586000000000002e-05 2024-09-13T14:52:48.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:48.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.598735+0000", 2024-09-13T14:52:48.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5300000000000002e-06 2024-09-13T14:52:48.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:48.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.551905+0000", 2024-09-13T14:52:48.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.95316925399999997 2024-09-13T14:52:48.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:48.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.551912+0000", 2024-09-13T14:52:48.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8590000000000005e-06 2024-09-13T14:52:48.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:48.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.697203+0000", 2024-09-13T14:52:48.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.145290602 2024-09-13T14:52:48.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:52:48.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.697330+0000", 2024-09-13T14:52:48.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012713399999999999 2024-09-13T14:52:48.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:52:48.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.697632+0000", 2024-09-13T14:52:48.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00030206900000000002 2024-09-13T14:52:48.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:52:48.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702118+0000", 2024-09-13T14:52:48.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0044853760000000001 2024-09-13T14:52:48.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:48.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702437+0000", 2024-09-13T14:52:48.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000318921 2024-09-13T14:52:48.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:48.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702489+0000", 2024-09-13T14:52:48.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2710000000000002e-05 2024-09-13T14:52:48.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:48.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702635+0000", 2024-09-13T14:52:48.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014527400000000001 2024-09-13T14:52:48.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:52:48.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702670+0000", 2024-09-13T14:52:48.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5197999999999997e-05 2024-09-13T14:52:48.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:52:48.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702747+0000", 2024-09-13T14:52:48.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7399999999999998e-05 2024-09-13T14:52:48.943 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:48.943 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:52:48.943 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:48.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.943 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:257772 6.10 6:0ebb2d78:::benchmark_data_smithi151_155836_object125314:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-13T14:52:48.943 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:13.644338+0000", 2024-09-13T14:52:48.943 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 575.26814187699995, 2024-09-13T14:52:48.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0617536259999998, 2024-09-13T14:52:48.944 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:52:48.944 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:52:48.944 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:48.944 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:52:48.944 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:52:48.944 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:52:48.944 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 257772 2024-09-13T14:52:48.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.944 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:52:48.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:52:48.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.644338+0000", 2024-09-13T14:52:48.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:48.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:52:48.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.644338+0000", 2024-09-13T14:52:48.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:48.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:52:48.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.644339+0000", 2024-09-13T14:52:48.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.339e-06 2024-09-13T14:52:48.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:52:48.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.644345+0000", 2024-09-13T14:52:48.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.31e-06 2024-09-13T14:52:48.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:52:48.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.644367+0000", 2024-09-13T14:52:48.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2107999999999999e-05 2024-09-13T14:52:48.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:52:48.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.644376+0000", 2024-09-13T14:52:48.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6930000000000002e-06 2024-09-13T14:52:48.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:48.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.644445+0000", 2024-09-13T14:52:48.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8768000000000003e-05 2024-09-13T14:52:48.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:48.947 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:48.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.644458+0000", 2024-09-13T14:52:48.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3213999999999999e-05 2024-09-13T14:52:48.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:48.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552174+0000", 2024-09-13T14:52:48.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.90771541899999997 2024-09-13T14:52:48.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:48.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552245+0000", 2024-09-13T14:52:48.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0680999999999996e-05 2024-09-13T14:52:48.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:48.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.699789+0000", 2024-09-13T14:52:48.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1475443649999999 2024-09-13T14:52:48.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:52:48.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.699933+0000", 2024-09-13T14:52:48.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000144356 2024-09-13T14:52:48.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:52:48.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.700163+0000", 2024-09-13T14:52:48.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00022942000000000001 2024-09-13T14:52:48.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:52:48.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702386+0000", 2024-09-13T14:52:48.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002222973 2024-09-13T14:52:48.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:48.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702550+0000", 2024-09-13T14:52:48.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016436399999999999 2024-09-13T14:52:48.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:48.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702591+0000", 2024-09-13T14:52:48.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0670000000000002e-05 2024-09-13T14:52:48.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:48.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.705911+0000", 2024-09-13T14:52:48.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033206339999999998 2024-09-13T14:52:48.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:52:48.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.705977+0000", 2024-09-13T14:52:48.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5827000000000005e-05 2024-09-13T14:52:48.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:52:48.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.706092+0000", 2024-09-13T14:52:48.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114485 2024-09-13T14:52:48.953 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:48.953 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:52:48.953 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:48.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.954 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:257796 6.10 6:08274f39:::benchmark_data_smithi151_155836_object125338:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-13T14:52:48.954 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:13.665405+0000", 2024-09-13T14:52:48.954 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 575.24707483500003, 2024-09-13T14:52:48.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0473038649999999, 2024-09-13T14:52:48.954 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:52:48.954 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:52:48.954 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:48.954 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:52:48.954 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:52:48.954 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:52:48.954 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 257796 2024-09-13T14:52:48.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.955 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:52:48.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:52:48.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.665405+0000", 2024-09-13T14:52:48.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:48.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:52:48.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.665405+0000", 2024-09-13T14:52:48.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:48.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:52:48.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.665407+0000", 2024-09-13T14:52:48.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.137e-06 2024-09-13T14:52:48.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:52:48.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.665415+0000", 2024-09-13T14:52:48.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5009999999999998e-06 2024-09-13T14:52:48.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:52:48.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.665435+0000", 2024-09-13T14:52:48.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9928999999999998e-05 2024-09-13T14:52:48.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:52:48.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.665443+0000", 2024-09-13T14:52:48.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6770000000000001e-06 2024-09-13T14:52:48.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:48.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.665518+0000", 2024-09-13T14:52:48.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4191999999999997e-05 2024-09-13T14:52:48.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:48.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.665529+0000", 2024-09-13T14:52:48.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.119e-05 2024-09-13T14:52:48.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:48.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552280+0000", 2024-09-13T14:52:48.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.88675172700000005 2024-09-13T14:52:48.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:48.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552289+0000", 2024-09-13T14:52:48.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1410000000000005e-06 2024-09-13T14:52:48.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:48.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702789+0000", 2024-09-13T14:52:48.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1505001939999999 2024-09-13T14:52:48.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:52:48.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.702931+0000", 2024-09-13T14:52:48.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000142201 2024-09-13T14:52:48.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:52:48.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.703228+0000", 2024-09-13T14:52:48.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00029683299999999998 2024-09-13T14:52:48.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:48.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.712398+0000", 2024-09-13T14:52:48.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0091702940000000007 2024-09-13T14:52:48.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:48.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.712453+0000", 2024-09-13T14:52:48.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4666000000000001e-05 2024-09-13T14:52:48.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:48.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.712507+0000", 2024-09-13T14:52:48.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4418e-05 2024-09-13T14:52:48.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:52:48.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.712556+0000", 2024-09-13T14:52:48.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8690000000000003e-05 2024-09-13T14:52:48.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:52:48.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.712592+0000", 2024-09-13T14:52:48.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6566999999999997e-05 2024-09-13T14:52:48.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:52:48.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.712709+0000", 2024-09-13T14:52:48.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116508 2024-09-13T14:52:48.962 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:48.962 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:52:48.962 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:48.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.962 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:257805 6.10 6:0dc62792:::benchmark_data_smithi151_155836_object125347:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-13T14:52:48.962 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:13.676320+0000", 2024-09-13T14:52:48.962 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 575.23615975999996, 2024-09-13T14:52:48.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0366515989999998, 2024-09-13T14:52:48.963 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:52:48.963 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:52:48.963 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:48.963 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:52:48.963 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:52:48.963 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:52:48.963 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 257805 2024-09-13T14:52:48.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.963 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:52:48.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:52:48.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.676320+0000", 2024-09-13T14:52:48.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:48.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:52:48.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.676320+0000", 2024-09-13T14:52:48.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:48.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:52:48.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.676322+0000", 2024-09-13T14:52:48.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.649e-06 2024-09-13T14:52:48.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:52:48.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.676328+0000", 2024-09-13T14:52:48.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1090000000000003e-06 2024-09-13T14:52:48.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:52:48.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.676344+0000", 2024-09-13T14:52:48.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6498999999999999e-05 2024-09-13T14:52:48.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:52:48.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.676351+0000", 2024-09-13T14:52:48.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8140000000000003e-06 2024-09-13T14:52:48.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:48.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.676407+0000", 2024-09-13T14:52:48.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6198e-05 2024-09-13T14:52:48.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:48.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.676415+0000", 2024-09-13T14:52:48.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9160000000000001e-06 2024-09-13T14:52:48.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:48.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552313+0000", 2024-09-13T14:52:48.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.87589755899999999 2024-09-13T14:52:48.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:48.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552320+0000", 2024-09-13T14:52:48.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0480000000000001e-06 2024-09-13T14:52:48.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:48.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.706131+0000", 2024-09-13T14:52:48.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.153810649 2024-09-13T14:52:48.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:52:48.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.706263+0000", 2024-09-13T14:52:48.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013223 2024-09-13T14:52:48.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:52:48.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.706719+0000", 2024-09-13T14:52:48.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045607500000000001 2024-09-13T14:52:48.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:48.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.712543+0000", 2024-09-13T14:52:48.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0058245789999999999 2024-09-13T14:52:48.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:52:48.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.712717+0000", 2024-09-13T14:52:48.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000173924 2024-09-13T14:52:48.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:48.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.712753+0000", 2024-09-13T14:52:48.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5793000000000001e-05 2024-09-13T14:52:48.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:48.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.712809+0000", 2024-09-13T14:52:48.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5949000000000003e-05 2024-09-13T14:52:48.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:52:48.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.712844+0000", 2024-09-13T14:52:48.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4761000000000001e-05 2024-09-13T14:52:48.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:52:48.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.712972+0000", 2024-09-13T14:52:48.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000127847 2024-09-13T14:52:48.971 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:48.971 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:52:48.971 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:48.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.971 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:257827 6.10 6:0d3dae4d:::benchmark_data_smithi151_155836_object125369:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-13T14:52:48.971 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:13.700521+0000", 2024-09-13T14:52:48.971 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 575.21195854400003, 2024-09-13T14:52:48.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0188779120000002, 2024-09-13T14:52:48.971 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:52:48.971 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:52:48.971 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:48.972 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:52:48.972 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:52:48.972 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:52:48.972 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 257827 2024-09-13T14:52:48.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.972 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:52:48.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:52:48.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.700521+0000", 2024-09-13T14:52:48.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:48.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:52:48.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.700521+0000", 2024-09-13T14:52:48.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:48.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:52:48.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.700522+0000", 2024-09-13T14:52:48.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1999999999999998e-07 2024-09-13T14:52:48.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:52:48.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.700527+0000", 2024-09-13T14:52:48.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3159999999999998e-06 2024-09-13T14:52:48.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:52:48.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.700535+0000", 2024-09-13T14:52:48.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1729999999999995e-06 2024-09-13T14:52:48.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:52:48.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.700541+0000", 2024-09-13T14:52:48.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6189999999999996e-06 2024-09-13T14:52:48.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:48.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.700603+0000", 2024-09-13T14:52:48.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1200999999999999e-05 2024-09-13T14:52:48.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:48.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.700611+0000", 2024-09-13T14:52:48.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4060000000000005e-06 2024-09-13T14:52:48.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:48.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552345+0000", 2024-09-13T14:52:48.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.85173386600000001 2024-09-13T14:52:48.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:48.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552352+0000", 2024-09-13T14:52:48.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4839999999999998e-06 2024-09-13T14:52:48.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:48.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.709531+0000", 2024-09-13T14:52:48.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.157178778 2024-09-13T14:52:48.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:52:48.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.709647+0000", 2024-09-13T14:52:48.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000115513 2024-09-13T14:52:48.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:52:48.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.710047+0000", 2024-09-13T14:52:48.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00040017699999999999 2024-09-13T14:52:48.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:52:48.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.712763+0000", 2024-09-13T14:52:48.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027165779999999999 2024-09-13T14:52:48.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:48.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.719122+0000", 2024-09-13T14:52:48.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0063590690000000002 2024-09-13T14:52:48.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:48.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.719177+0000", 2024-09-13T14:52:48.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4982000000000003e-05 2024-09-13T14:52:48.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:48.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.719219+0000", 2024-09-13T14:52:48.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1709999999999999e-05 2024-09-13T14:52:48.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:52:48.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.719282+0000", 2024-09-13T14:52:48.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3262000000000005e-05 2024-09-13T14:52:48.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:52:48.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.719399+0000", 2024-09-13T14:52:48.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116858 2024-09-13T14:52:48.979 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:48.979 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:52:48.979 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:48.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.979 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:257880 6.10 6:0d9bf4af:::benchmark_data_smithi151_155836_object125422:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-13T14:52:48.979 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:13.763399+0000", 2024-09-13T14:52:48.979 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 575.14908114800005, 2024-09-13T14:52:48.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.963654477, 2024-09-13T14:52:48.980 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:52:48.980 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:52:48.980 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:48.980 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:52:48.980 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:52:48.980 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:52:48.980 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 257880 2024-09-13T14:52:48.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.980 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:52:48.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:52:48.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.763399+0000", 2024-09-13T14:52:48.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:48.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:52:48.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.763399+0000", 2024-09-13T14:52:48.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:48.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:52:48.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.763400+0000", 2024-09-13T14:52:48.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6499999999999998e-07 2024-09-13T14:52:48.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:52:48.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.763404+0000", 2024-09-13T14:52:48.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9589999999999997e-06 2024-09-13T14:52:48.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:52:48.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.763415+0000", 2024-09-13T14:52:48.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1127e-05 2024-09-13T14:52:48.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:52:48.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.763422+0000", 2024-09-13T14:52:48.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4089999999999997e-06 2024-09-13T14:52:48.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:48.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.763483+0000", 2024-09-13T14:52:48.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0893000000000001e-05 2024-09-13T14:52:48.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:48.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.763491+0000", 2024-09-13T14:52:48.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9880000000000004e-06 2024-09-13T14:52:48.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:48.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552377+0000", 2024-09-13T14:52:48.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.78888635399999996 2024-09-13T14:52:48.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:48.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552385+0000", 2024-09-13T14:52:48.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5730000000000002e-06 2024-09-13T14:52:48.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:48.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.713023+0000", 2024-09-13T14:52:48.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.160637929 2024-09-13T14:52:48.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:52:48.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.713145+0000", 2024-09-13T14:52:48.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012216000000000001 2024-09-13T14:52:48.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:52:48.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.713535+0000", 2024-09-13T14:52:48.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039047000000000002 2024-09-13T14:52:48.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:52:48.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.719136+0000", 2024-09-13T14:52:48.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0056008610000000004 2024-09-13T14:52:48.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:48.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.726475+0000", 2024-09-13T14:52:48.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0073387640000000002 2024-09-13T14:52:48.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:48.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.726526+0000", 2024-09-13T14:52:48.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1286999999999999e-05 2024-09-13T14:52:48.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:48.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.726606+0000", 2024-09-13T14:52:48.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9840999999999997e-05 2024-09-13T14:52:48.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:52:48.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.726925+0000", 2024-09-13T14:52:48.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00031910899999999998 2024-09-13T14:52:48.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:52:48.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.727053+0000", 2024-09-13T14:52:48.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012788799999999999 2024-09-13T14:52:48.987 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:48.987 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:52:48.987 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:48.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.988 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:257940 6.10 6:09b2c975:::benchmark_data_smithi151_155836_object125482:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-13T14:52:48.988 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:13.834914+0000", 2024-09-13T14:52:48.988 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 575.07756635600003, 2024-09-13T14:52:48.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.892473069, 2024-09-13T14:52:48.988 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:52:48.988 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:52:48.988 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:48.988 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:52:48.988 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:52:48.988 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:52:48.988 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 257940 2024-09-13T14:52:48.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.988 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:52:48.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:52:48.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.834914+0000", 2024-09-13T14:52:48.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:48.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:52:48.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.834914+0000", 2024-09-13T14:52:48.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:48.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:52:48.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.834915+0000", 2024-09-13T14:52:48.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4529999999999999e-06 2024-09-13T14:52:48.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:52:48.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.834920+0000", 2024-09-13T14:52:48.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5560000000000001e-06 2024-09-13T14:52:48.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:52:48.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.834946+0000", 2024-09-13T14:52:48.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6834000000000001e-05 2024-09-13T14:52:48.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:52:48.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.834954+0000", 2024-09-13T14:52:48.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4830000000000002e-06 2024-09-13T14:52:48.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:48.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.835153+0000", 2024-09-13T14:52:48.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019867299999999999 2024-09-13T14:52:48.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:48.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.835165+0000", 2024-09-13T14:52:48.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.225e-05 2024-09-13T14:52:48.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.992 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:48.992 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552411+0000", 2024-09-13T14:52:48.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.71724599 2024-09-13T14:52:48.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:48.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:48.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552418+0000", 2024-09-13T14:52:48.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.277e-06 2024-09-13T14:52:48.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:48.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.716114+0000", 2024-09-13T14:52:48.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.163696184 2024-09-13T14:52:48.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:52:48.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.716215+0000", 2024-09-13T14:52:48.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101093 2024-09-13T14:52:48.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:52:48.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.716531+0000", 2024-09-13T14:52:48.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00031590900000000001 2024-09-13T14:52:48.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:52:48.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.719143+0000", 2024-09-13T14:52:48.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002611333 2024-09-13T14:52:48.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:48.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.726572+0000", 2024-09-13T14:52:48.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0074290759999999997 2024-09-13T14:52:48.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:48.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.727096+0000", 2024-09-13T14:52:48.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00052479899999999995 2024-09-13T14:52:48.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:48.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.727135+0000", 2024-09-13T14:52:48.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8365999999999998e-05 2024-09-13T14:52:48.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:52:48.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.727193+0000", 2024-09-13T14:52:48.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8394999999999999e-05 2024-09-13T14:52:48.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:52:48.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.727387+0000", 2024-09-13T14:52:48.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019339800000000001 2024-09-13T14:52:48.998 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:48.998 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:52:48.998 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:48.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.998 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:257961 6.10 6:0bf0500a:::benchmark_data_smithi151_155836_object125503:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-13T14:52:48.998 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:13.855136+0000", 2024-09-13T14:52:48.998 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 575.05734344300004, 2024-09-13T14:52:48.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.883789376, 2024-09-13T14:52:48.998 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:52:48.998 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:52:48.998 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:48.998 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:52:48.999 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:52:48.999 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:52:48.999 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 257961 2024-09-13T14:52:48.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.999 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:52:48.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:52:48.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.855136+0000", 2024-09-13T14:52:48.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:48.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:48.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:48.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:52:48.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.855136+0000", 2024-09-13T14:52:48.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:49.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:52:49.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.855138+0000", 2024-09-13T14:52:49.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1999999999999999e-06 2024-09-13T14:52:49.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:52:49.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.855142+0000", 2024-09-13T14:52:49.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3239999999999996e-06 2024-09-13T14:52:49.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:52:49.001 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.855153+0000", 2024-09-13T14:52:49.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1096e-05 2024-09-13T14:52:49.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:52:49.001 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.855160+0000", 2024-09-13T14:52:49.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9789999999999996e-06 2024-09-13T14:52:49.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:49.001 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.855220+0000", 2024-09-13T14:52:49.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0195000000000002e-05 2024-09-13T14:52:49.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:49.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.855229+0000", 2024-09-13T14:52:49.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7450000000000001e-06 2024-09-13T14:52:49.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:49.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552444+0000", 2024-09-13T14:52:49.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.69721484 2024-09-13T14:52:49.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:49.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552451+0000", 2024-09-13T14:52:49.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5120000000000002e-06 2024-09-13T14:52:49.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:49.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.719431+0000", 2024-09-13T14:52:49.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.166979166 2024-09-13T14:52:49.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:52:49.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.719556+0000", 2024-09-13T14:52:49.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012549699999999999 2024-09-13T14:52:49.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:52:49.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.719996+0000", 2024-09-13T14:52:49.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00044005699999999999 2024-09-13T14:52:49.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:52:49.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.726489+0000", 2024-09-13T14:52:49.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0064931499999999996 2024-09-13T14:52:49.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:49.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.727436+0000", 2024-09-13T14:52:49.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00094664900000000004 2024-09-13T14:52:49.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.005 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:49.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.738590+0000", 2024-09-13T14:52:49.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011154410999999999 2024-09-13T14:52:49.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.005 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:49.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.738673+0000", 2024-09-13T14:52:49.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2311000000000005e-05 2024-09-13T14:52:49.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.005 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:52:49.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.738783+0000", 2024-09-13T14:52:49.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000110529 2024-09-13T14:52:49.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:52:49.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.738926+0000", 2024-09-13T14:52:49.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014271499999999999 2024-09-13T14:52:49.006 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:49.006 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:52:49.006 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:49.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.006 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:258052 6.10 6:08f50a5c:::benchmark_data_smithi151_155836_object125594:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-13T14:52:49.006 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:13.975509+0000", 2024-09-13T14:52:49.006 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 574.93697082200003, 2024-09-13T14:52:49.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7637252050000001, 2024-09-13T14:52:49.007 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:52:49.007 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:52:49.007 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:49.007 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:52:49.007 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:52:49.007 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:52:49.007 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 258052 2024-09-13T14:52:49.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.007 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:52:49.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:52:49.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.975509+0000", 2024-09-13T14:52:49.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:49.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:52:49.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.975509+0000", 2024-09-13T14:52:49.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:49.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:52:49.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.975511+0000", 2024-09-13T14:52:49.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.438e-06 2024-09-13T14:52:49.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:52:49.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.975517+0000", 2024-09-13T14:52:49.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.55e-06 2024-09-13T14:52:49.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:52:49.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.975532+0000", 2024-09-13T14:52:49.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4935e-05 2024-09-13T14:52:49.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:52:49.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.975547+0000", 2024-09-13T14:52:49.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4927e-05 2024-09-13T14:52:49.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:49.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.975614+0000", 2024-09-13T14:52:49.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7502000000000005e-05 2024-09-13T14:52:49.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:49.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:13.975627+0000", 2024-09-13T14:52:49.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2982999999999999e-05 2024-09-13T14:52:49.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:49.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552478+0000", 2024-09-13T14:52:49.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.57685019599999998 2024-09-13T14:52:49.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:49.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552485+0000", 2024-09-13T14:52:49.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2099999999999996e-06 2024-09-13T14:52:49.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:49.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.722770+0000", 2024-09-13T14:52:49.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.170285639 2024-09-13T14:52:49.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:52:49.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.722894+0000", 2024-09-13T14:52:49.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000123898 2024-09-13T14:52:49.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:52:49.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.723290+0000", 2024-09-13T14:52:49.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039553100000000002 2024-09-13T14:52:49.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:52:49.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.726538+0000", 2024-09-13T14:52:49.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0032478350000000001 2024-09-13T14:52:49.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:49.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.738540+0000", 2024-09-13T14:52:49.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012001927000000001 2024-09-13T14:52:49.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:49.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.738633+0000", 2024-09-13T14:52:49.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3621000000000004e-05 2024-09-13T14:52:49.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:49.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.738976+0000", 2024-09-13T14:52:49.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00034300999999999998 2024-09-13T14:52:49.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:52:49.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.739124+0000", 2024-09-13T14:52:49.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014781500000000001 2024-09-13T14:52:49.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:52:49.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.739234+0000", 2024-09-13T14:52:49.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000110188 2024-09-13T14:52:49.015 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:49.015 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:52:49.015 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:49.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.015 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:258153 6.10 6:0bb5117f:::benchmark_data_smithi151_155836_object125695:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-13T14:52:49.015 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:14.087371+0000", 2024-09-13T14:52:49.015 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 574.82510901499995, 2024-09-13T14:52:49.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.659579189, 2024-09-13T14:52:49.015 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:52:49.015 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:52:49.016 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:49.016 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:52:49.016 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:52:49.016 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:52:49.016 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 258153 2024-09-13T14:52:49.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.016 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:52:49.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:52:49.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.087371+0000", 2024-09-13T14:52:49.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:49.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:52:49.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.087371+0000", 2024-09-13T14:52:49.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:49.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:52:49.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.087372+0000", 2024-09-13T14:52:49.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.567e-06 2024-09-13T14:52:49.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:52:49.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.087378+0000", 2024-09-13T14:52:49.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9200000000000001e-06 2024-09-13T14:52:49.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:52:49.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.087391+0000", 2024-09-13T14:52:49.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2313000000000001e-05 2024-09-13T14:52:49.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:52:49.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.087396+0000", 2024-09-13T14:52:49.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6969999999999998e-06 2024-09-13T14:52:49.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:49.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.087448+0000", 2024-09-13T14:52:49.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1969000000000003e-05 2024-09-13T14:52:49.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:49.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.087456+0000", 2024-09-13T14:52:49.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0940000000000008e-06 2024-09-13T14:52:49.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:49.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552511+0000", 2024-09-13T14:52:49.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.46505427399999999 2024-09-13T14:52:49.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:49.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552518+0000", 2024-09-13T14:52:49.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6960000000000005e-06 2024-09-13T14:52:49.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:49.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.727474+0000", 2024-09-13T14:52:49.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.174955296 2024-09-13T14:52:49.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:52:49.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.727607+0000", 2024-09-13T14:52:49.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013341899999999999 2024-09-13T14:52:49.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:52:49.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.728101+0000", 2024-09-13T14:52:49.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00049359700000000002 2024-09-13T14:52:49.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:49.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.739276+0000", 2024-09-13T14:52:49.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011175197 2024-09-13T14:52:49.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:49.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.739352+0000", 2024-09-13T14:52:49.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5711999999999996e-05 2024-09-13T14:52:49.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:52:49.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.746603+0000", 2024-09-13T14:52:49.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0072512039999999998 2024-09-13T14:52:49.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:49.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.746671+0000", 2024-09-13T14:52:49.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8139000000000006e-05 2024-09-13T14:52:49.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:52:49.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.746734+0000", 2024-09-13T14:52:49.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3295999999999998e-05 2024-09-13T14:52:49.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:52:49.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.746950+0000", 2024-09-13T14:52:49.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000215799 2024-09-13T14:52:49.023 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:49.023 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:52:49.023 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:49.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.023 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:258159 6.10 6:0bd99f59:::benchmark_data_smithi151_155836_object125701:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-13T14:52:49.023 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:14.093771+0000", 2024-09-13T14:52:49.023 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 574.81870898099999, 2024-09-13T14:52:49.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6632812779999999, 2024-09-13T14:52:49.024 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:52:49.024 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:52:49.024 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:49.024 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:52:49.024 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:52:49.024 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:52:49.024 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 258159 2024-09-13T14:52:49.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.024 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:52:49.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:52:49.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.093771+0000", 2024-09-13T14:52:49.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:49.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:52:49.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.093771+0000", 2024-09-13T14:52:49.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:49.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:52:49.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.093771+0000", 2024-09-13T14:52:49.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3399999999999999e-07 2024-09-13T14:52:49.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:52:49.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.093774+0000", 2024-09-13T14:52:49.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9579999999999999e-06 2024-09-13T14:52:49.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:52:49.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.093789+0000", 2024-09-13T14:52:49.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4372999999999999e-05 2024-09-13T14:52:49.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:52:49.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.093799+0000", 2024-09-13T14:52:49.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0061000000000001e-05 2024-09-13T14:52:49.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:49.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.093891+0000", 2024-09-13T14:52:49.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2108000000000006e-05 2024-09-13T14:52:49.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:49.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.093902+0000", 2024-09-13T14:52:49.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1147e-05 2024-09-13T14:52:49.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:49.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552544+0000", 2024-09-13T14:52:49.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.45864198499999997 2024-09-13T14:52:49.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:49.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552552+0000", 2024-09-13T14:52:49.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5809999999999999e-06 2024-09-13T14:52:49.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:49.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.731453+0000", 2024-09-13T14:52:49.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.178901524 2024-09-13T14:52:49.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:52:49.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.731594+0000", 2024-09-13T14:52:49.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014075000000000001 2024-09-13T14:52:49.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:52:49.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.731982+0000", 2024-09-13T14:52:49.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00038823200000000002 2024-09-13T14:52:49.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:49.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.739313+0000", 2024-09-13T14:52:49.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0073308289999999996 2024-09-13T14:52:49.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:52:49.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.746610+0000", 2024-09-13T14:52:49.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0072970980000000001 2024-09-13T14:52:49.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:49.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.747139+0000", 2024-09-13T14:52:49.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00052901100000000002 2024-09-13T14:52:49.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:49.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.755755+0000", 2024-09-13T14:52:49.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0086160409999999996 2024-09-13T14:52:49.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:52:49.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.756934+0000", 2024-09-13T14:52:49.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011788790000000001 2024-09-13T14:52:49.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:52:49.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.757052+0000", 2024-09-13T14:52:49.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118167 2024-09-13T14:52:49.031 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:49.031 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:52:49.031 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:49.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.031 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:258205 6.10 6:0d7fd54b:::benchmark_data_smithi151_155836_object125747:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-13T14:52:49.031 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:14.154924+0000", 2024-09-13T14:52:49.032 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 574.75755560100004, 2024-09-13T14:52:49.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.602350766, 2024-09-13T14:52:49.032 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:52:49.032 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:52:49.032 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:49.032 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:52:49.032 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:52:49.032 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:52:49.032 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 258205 2024-09-13T14:52:49.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.032 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:52:49.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:52:49.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.154924+0000", 2024-09-13T14:52:49.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:49.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:52:49.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.154924+0000", 2024-09-13T14:52:49.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:49.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:52:49.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.154925+0000", 2024-09-13T14:52:49.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.181e-06 2024-09-13T14:52:49.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:52:49.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.154930+0000", 2024-09-13T14:52:49.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0289999999999999e-06 2024-09-13T14:52:49.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:52:49.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.154941+0000", 2024-09-13T14:52:49.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1651000000000001e-05 2024-09-13T14:52:49.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:52:49.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.154948+0000", 2024-09-13T14:52:49.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0330000000000003e-06 2024-09-13T14:52:49.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:49.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.155012+0000", 2024-09-13T14:52:49.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3864999999999999e-05 2024-09-13T14:52:49.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:49.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.155025+0000", 2024-09-13T14:52:49.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2625e-05 2024-09-13T14:52:49.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:49.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552577+0000", 2024-09-13T14:52:49.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.39755244899999997 2024-09-13T14:52:49.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:49.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552584+0000", 2024-09-13T14:52:49.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3370000000000002e-06 2024-09-13T14:52:49.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:49.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.734466+0000", 2024-09-13T14:52:49.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.181881196 2024-09-13T14:52:49.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:52:49.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.734592+0000", 2024-09-13T14:52:49.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012631099999999999 2024-09-13T14:52:49.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:52:49.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.734982+0000", 2024-09-13T14:52:49.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000390252 2024-09-13T14:52:49.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:52:49.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.747156+0000", 2024-09-13T14:52:49.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012174114 2024-09-13T14:52:49.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:49.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.747200+0000", 2024-09-13T14:52:49.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3495999999999998e-05 2024-09-13T14:52:49.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:49.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.750743+0000", 2024-09-13T14:52:49.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0035429839999999999 2024-09-13T14:52:49.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:49.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.757115+0000", 2024-09-13T14:52:49.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0063716789999999999 2024-09-13T14:52:49.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:52:49.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.757174+0000", 2024-09-13T14:52:49.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9864999999999997e-05 2024-09-13T14:52:49.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:52:49.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.757275+0000", 2024-09-13T14:52:49.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010069900000000001 2024-09-13T14:52:49.039 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:49.039 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:52:49.039 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:49.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.039 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:258258 6.10 6:0ac2d4b7:::benchmark_data_smithi151_155836_object125800:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-13T14:52:49.040 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:14.255036+0000", 2024-09-13T14:52:49.040 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 574.65744347299994, 2024-09-13T14:52:49.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.502433766, 2024-09-13T14:52:49.040 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:52:49.040 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:52:49.040 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:49.041 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:52:49.041 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:52:49.041 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:52:49.041 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 258258 2024-09-13T14:52:49.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.041 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:52:49.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:52:49.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.255036+0000", 2024-09-13T14:52:49.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:49.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:52:49.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.255036+0000", 2024-09-13T14:52:49.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:49.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:52:49.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.255037+0000", 2024-09-13T14:52:49.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7000000000000003e-07 2024-09-13T14:52:49.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:52:49.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.255042+0000", 2024-09-13T14:52:49.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5210000000000004e-06 2024-09-13T14:52:49.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:52:49.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.255051+0000", 2024-09-13T14:52:49.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7739999999999993e-06 2024-09-13T14:52:49.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:52:49.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.255056+0000", 2024-09-13T14:52:49.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7960000000000003e-06 2024-09-13T14:52:49.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:49.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.255105+0000", 2024-09-13T14:52:49.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9925000000000001e-05 2024-09-13T14:52:49.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:49.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.255114+0000", 2024-09-13T14:52:49.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2900000000000002e-06 2024-09-13T14:52:49.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:49.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552610+0000", 2024-09-13T14:52:49.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29749615499999998 2024-09-13T14:52:49.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:49.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552617+0000", 2024-09-13T14:52:49.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3560000000000004e-06 2024-09-13T14:52:49.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:49.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.739380+0000", 2024-09-13T14:52:49.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.186762997 2024-09-13T14:52:49.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:52:49.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.739504+0000", 2024-09-13T14:52:49.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000123793 2024-09-13T14:52:49.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:52:49.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.740007+0000", 2024-09-13T14:52:49.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00050336499999999998 2024-09-13T14:52:49.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:52:49.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.747163+0000", 2024-09-13T14:52:49.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0071556759999999997 2024-09-13T14:52:49.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:49.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.747244+0000", 2024-09-13T14:52:49.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0593000000000006e-05 2024-09-13T14:52:49.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:49.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.750787+0000", 2024-09-13T14:52:49.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003543792 2024-09-13T14:52:49.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:49.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.757323+0000", 2024-09-13T14:52:49.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0065354250000000001 2024-09-13T14:52:49.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:52:49.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.757374+0000", 2024-09-13T14:52:49.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1622000000000001e-05 2024-09-13T14:52:49.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:52:49.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.757470+0000", 2024-09-13T14:52:49.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5716000000000003e-05 2024-09-13T14:52:49.049 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:49.049 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:52:49.049 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:49.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.049 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35289.0:258270 6.10 6:0a17d4dd:::benchmark_data_smithi151_155836_object125812:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-13T14:52:49.049 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:43:14.294755+0000", 2024-09-13T14:52:49.049 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 574.61772516999997, 2024-09-13T14:52:49.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.46420789, 2024-09-13T14:52:49.050 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:52:49.050 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:52:49.050 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:49.050 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:52:49.050 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35289", 2024-09-13T14:52:49.050 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1590185159", 2024-09-13T14:52:49.050 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 258270 2024-09-13T14:52:49.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.050 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:52:49.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.050 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:52:49.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.294755+0000", 2024-09-13T14:52:49.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:49.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.051 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:52:49.051 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.294755+0000", 2024-09-13T14:52:49.051 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:49.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.051 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:52:49.051 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.294757+0000", 2024-09-13T14:52:49.051 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9400000000000001e-06 2024-09-13T14:52:49.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.051 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:52:49.051 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.294764+0000", 2024-09-13T14:52:49.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3499999999999999e-06 2024-09-13T14:52:49.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.052 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:49.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:49.053 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:52:49.053 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.294782+0000", 2024-09-13T14:52:49.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8267999999999999e-05 2024-09-13T14:52:49.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:52:49.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.294798+0000", 2024-09-13T14:52:49.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5389000000000001e-05 2024-09-13T14:52:49.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:49.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.294856+0000", 2024-09-13T14:52:49.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8264000000000003e-05 2024-09-13T14:52:49.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:49.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.294865+0000", 2024-09-13T14:52:49.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.952e-06 2024-09-13T14:52:49.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:49.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552648+0000", 2024-09-13T14:52:49.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25778293299999999 2024-09-13T14:52:49.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:49.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:14.552656+0000", 2024-09-13T14:52:49.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6699999999999994e-06 2024-09-13T14:52:49.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:49.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.742910+0000", 2024-09-13T14:52:49.056 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.19025483 2024-09-13T14:52:49.056 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.056 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:52:49.056 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.743051+0000", 2024-09-13T14:52:49.056 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014111899999999999 2024-09-13T14:52:49.056 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.056 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:52:49.056 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.743528+0000", 2024-09-13T14:52:49.056 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000476298 2024-09-13T14:52:49.056 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.056 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:52:49.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.747212+0000", 2024-09-13T14:52:49.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0036839500000000001 2024-09-13T14:52:49.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:49.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.747369+0000", 2024-09-13T14:52:49.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015746600000000001 2024-09-13T14:52:49.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:49.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.757512+0000", 2024-09-13T14:52:49.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010142744 2024-09-13T14:52:49.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.058 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:49.058 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.758753+0000", 2024-09-13T14:52:49.058 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012413840000000001 2024-09-13T14:52:49.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.058 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:52:49.058 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.758861+0000", 2024-09-13T14:52:49.058 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010781900000000001 2024-09-13T14:52:49.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.059 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:52:49.059 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:43:15.758963+0000", 2024-09-13T14:52:49.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101514 2024-09-13T14:52:49.059 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:49.059 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:52:49.059 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:49.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.059 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:179768 8.2 8:4be553c0:::benchmark_data_smithi151_229097_object54900:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1189)", 2024-09-13T14:52:49.059 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:52:11.544389+0000", 2024-09-13T14:52:49.059 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.368090709000001, 2024-09-13T14:52:49.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.03863229, 2024-09-13T14:52:49.060 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:52:49.060 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:52:49.060 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:49.060 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:52:49.060 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T14:52:49.060 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:52:49.060 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 179768 2024-09-13T14:52:49.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.060 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:52:49.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.060 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:52:49.060 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.544389+0000", 2024-09-13T14:52:49.060 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:49.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:52:49.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.544389+0000", 2024-09-13T14:52:49.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:49.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:52:49.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.544390+0000", 2024-09-13T14:52:49.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.254e-06 2024-09-13T14:52:49.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.062 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:52:49.062 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.544395+0000", 2024-09-13T14:52:49.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.515e-06 2024-09-13T14:52:49.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.062 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:52:49.062 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.544409+0000", 2024-09-13T14:52:49.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4445e-05 2024-09-13T14:52:49.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.062 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:52:49.062 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.544417+0000", 2024-09-13T14:52:49.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.374e-06 2024-09-13T14:52:49.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.063 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:49.063 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.544467+0000", 2024-09-13T14:52:49.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0383999999999997e-05 2024-09-13T14:52:49.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.063 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:49.063 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.544474+0000", 2024-09-13T14:52:49.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9319999999999999e-06 2024-09-13T14:52:49.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.063 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:49.063 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.572593+0000", 2024-09-13T14:52:49.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.028119381 2024-09-13T14:52:49.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:52:49.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.572698+0000", 2024-09-13T14:52:49.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000104444 2024-09-13T14:52:49.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:52:49.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.572817+0000", 2024-09-13T14:52:49.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000119473 2024-09-13T14:52:49.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:49.065 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.582590+0000", 2024-09-13T14:52:49.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0097727140000000001 2024-09-13T14:52:49.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:49.065 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.582867+0000", 2024-09-13T14:52:49.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000276397 2024-09-13T14:52:49.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:52:49.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.582936+0000", 2024-09-13T14:52:49.066 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9743e-05 2024-09-13T14:52:49.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:52:49.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.582983+0000", 2024-09-13T14:52:49.068 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7012000000000003e-05 2024-09-13T14:52:49.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.068 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:52:49.068 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.583021+0000", 2024-09-13T14:52:49.068 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8222000000000001e-05 2024-09-13T14:52:49.068 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:49.068 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:52:49.068 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:49.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.068 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:179783 8.2 8:4958d2ee:::benchmark_data_smithi151_229097_object54915:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1189)", 2024-09-13T14:52:49.069 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:52:11.555141+0000", 2024-09-13T14:52:49.069 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.357338847000001, 2024-09-13T14:52:49.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0450786999999999, 2024-09-13T14:52:49.069 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:52:49.069 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:52:49.069 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:49.069 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:52:49.069 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T14:52:49.069 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:52:49.069 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 179783 2024-09-13T14:52:49.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.069 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:52:49.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.069 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:52:49.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.555141+0000", 2024-09-13T14:52:49.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:49.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:52:49.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.555141+0000", 2024-09-13T14:52:49.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:49.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:52:49.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.555142+0000", 2024-09-13T14:52:49.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4309999999999999e-06 2024-09-13T14:52:49.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.071 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:52:49.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.555147+0000", 2024-09-13T14:52:49.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6349999999999997e-06 2024-09-13T14:52:49.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.072 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:52:49.072 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.555161+0000", 2024-09-13T14:52:49.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3672999999999999e-05 2024-09-13T14:52:49.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.072 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:52:49.072 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.555169+0000", 2024-09-13T14:52:49.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2309999999999996e-06 2024-09-13T14:52:49.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.072 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:49.072 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.556122+0000", 2024-09-13T14:52:49.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00095323700000000001 2024-09-13T14:52:49.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.073 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:49.073 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.556131+0000", 2024-09-13T14:52:49.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.242e-06 2024-09-13T14:52:49.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.073 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:49.073 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.583362+0000", 2024-09-13T14:52:49.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.027230997 2024-09-13T14:52:49.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.074 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:52:49.074 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.583470+0000", 2024-09-13T14:52:49.074 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108377 2024-09-13T14:52:49.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.074 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:52:49.074 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.583587+0000", 2024-09-13T14:52:49.074 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011692 2024-09-13T14:52:49.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.074 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:49.074 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.593593+0000", 2024-09-13T14:52:49.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010006045999999999 2024-09-13T14:52:49.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:52:49.075 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.593640+0000", 2024-09-13T14:52:49.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6998e-05 2024-09-13T14:52:49.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:49.075 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.600119+0000", 2024-09-13T14:52:49.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0064793000000000003 2024-09-13T14:52:49.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:52:49.076 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.600174+0000", 2024-09-13T14:52:49.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5169e-05 2024-09-13T14:52:49.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.076 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:52:49.076 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.600220+0000", 2024-09-13T14:52:49.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5444000000000001e-05 2024-09-13T14:52:49.076 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:49.076 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:52:49.076 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:49.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.076 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:179785 8.2 8:4a59eb92:::benchmark_data_smithi151_229097_object54917:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1189)", 2024-09-13T14:52:49.076 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:52:11.557035+0000", 2024-09-13T14:52:49.077 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.355444448999997, 2024-09-13T14:52:49.077 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0433434539999999, 2024-09-13T14:52:49.077 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:52:49.077 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:52:49.077 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:49.077 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:52:49.077 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T14:52:49.077 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:52:49.078 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 179785 2024-09-13T14:52:49.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.078 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:52:49.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.078 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:52:49.078 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.557035+0000", 2024-09-13T14:52:49.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:49.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.078 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:52:49.078 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.557035+0000", 2024-09-13T14:52:49.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:49.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:52:49.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.557036+0000", 2024-09-13T14:52:49.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4900000000000005e-07 2024-09-13T14:52:49.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:52:49.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.557040+0000", 2024-09-13T14:52:49.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9369999999999997e-06 2024-09-13T14:52:49.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:52:49.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.557054+0000", 2024-09-13T14:52:49.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3835000000000001e-05 2024-09-13T14:52:49.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:52:49.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.557061+0000", 2024-09-13T14:52:49.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4059999999999999e-06 2024-09-13T14:52:49.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:49.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.557123+0000", 2024-09-13T14:52:49.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1369000000000001e-05 2024-09-13T14:52:49.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:49.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.557131+0000", 2024-09-13T14:52:49.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1650000000000006e-06 2024-09-13T14:52:49.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:49.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.586929+0000", 2024-09-13T14:52:49.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.029797485 2024-09-13T14:52:49.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:52:49.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.587042+0000", 2024-09-13T14:52:49.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000113081 2024-09-13T14:52:49.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:52:49.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.587155+0000", 2024-09-13T14:52:49.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000112961 2024-09-13T14:52:49.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:49.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.593630+0000", 2024-09-13T14:52:49.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0064749960000000002 2024-09-13T14:52:49.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:52:49.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.593645+0000", 2024-09-13T14:52:49.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.505e-05 2024-09-13T14:52:49.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:49.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.600273+0000", 2024-09-13T14:52:49.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0066280419999999998 2024-09-13T14:52:49.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:52:49.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.600340+0000", 2024-09-13T14:52:49.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7069000000000004e-05 2024-09-13T14:52:49.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:52:49.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.600379+0000", 2024-09-13T14:52:49.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9208999999999997e-05 2024-09-13T14:52:49.084 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:49.084 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:52:49.084 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:49.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.084 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:179793 8.2 8:43e8365d:::benchmark_data_smithi151_229097_object54925:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1189)", 2024-09-13T14:52:49.084 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:52:11.563160+0000", 2024-09-13T14:52:49.084 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.349320112999997, 2024-09-13T14:52:49.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.038487105, 2024-09-13T14:52:49.084 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:52:49.084 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:52:49.084 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:49.084 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:52:49.085 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T14:52:49.085 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:52:49.085 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 179793 2024-09-13T14:52:49.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.085 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:52:49.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.085 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:52:49.085 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.563160+0000", 2024-09-13T14:52:49.085 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:49.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.085 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:52:49.085 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.563160+0000", 2024-09-13T14:52:49.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:52:49.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:52:49.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.563161+0000", 2024-09-13T14:52:49.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.19e-06 2024-09-13T14:52:49.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:52:49.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.563165+0000", 2024-09-13T14:52:49.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.63e-06 2024-09-13T14:52:49.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:52:49.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.563175+0000", 2024-09-13T14:52:49.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0349e-05 2024-09-13T14:52:49.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:52:49.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.563182+0000", 2024-09-13T14:52:49.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.973e-06 2024-09-13T14:52:49.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:49.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.563238+0000", 2024-09-13T14:52:49.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6026999999999997e-05 2024-09-13T14:52:49.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:52:49.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.563246+0000", 2024-09-13T14:52:49.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8879999999999997e-06 2024-09-13T14:52:49.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:52:49.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.590033+0000", 2024-09-13T14:52:49.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.026787474 2024-09-13T14:52:49.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:52:49.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.590138+0000", 2024-09-13T14:52:49.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010491 2024-09-13T14:52:49.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:52:49.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.590245+0000", 2024-09-13T14:52:49.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000107218 2024-09-13T14:52:49.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:52:49.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.600063+0000", 2024-09-13T14:52:49.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0098180119999999992 2024-09-13T14:52:49.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:49.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.600425+0000", 2024-09-13T14:52:49.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000361166 2024-09-13T14:52:49.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:49.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.601528+0000", 2024-09-13T14:52:49.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001103775 2024-09-13T14:52:49.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:52:49.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.601608+0000", 2024-09-13T14:52:49.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9887000000000003e-05 2024-09-13T14:52:49.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:52:49.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:52:49.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:52:49.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.601647+0000", 2024-09-13T14:52:49.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8606000000000003e-05 2024-09-13T14:52:49.091 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:49.091 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:52:49.091 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:49.091 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:52:49.091 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:52:49.091 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:52:49.155 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:49.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:49.191 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:49.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:49.265 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:49 smithi151 ceph-mon[104450]: pgmap v1801: 119 pgs: 119 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 972 op/s 2024-09-13T14:52:49.266 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:49 smithi151 ceph-mon[108966]: pgmap v1801: 119 pgs: 119 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 972 op/s 2024-09-13T14:52:49.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:49.303 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:49 smithi184 ceph-mon[92527]: pgmap v1801: 119 pgs: 119 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 972 op/s 2024-09-13T14:52:49.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:49.330 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:49.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:49.431 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:49.468 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:49.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:49.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:49.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:49.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:49.609 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:49.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:49.710 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:49.740 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:49.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:49.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:49.842 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:49.871 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:49.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:49.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:49.973 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:50.016 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:50.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:50.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:50.024 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:52:50.118 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:50.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:50.158 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:50.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:50.259 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:50.283 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:50.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:50.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:50.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T14:52:50.385 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:50.412 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:50.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:50.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:50.513 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:50.536 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:50.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:50.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:50.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:52:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:52:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:52:50.638 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:50.665 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:50.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:50.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:50.766 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:50.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:50.839 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:50.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:50.857 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:52:50.857 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:52:50.894 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:52:50.894 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:52:50.925 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:52:50.925 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:52:50.940 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:50.957 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:52:50.957 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:52:50.967 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:50.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:50.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:50.985 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:52:50.985 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:52:51.018 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:52:51.019 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:52:51.061 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:52:51.061 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:52:51.068 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:51.102 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:52:51.102 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:52:51.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:51.121 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:51.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:51.222 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:51.259 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:51.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:51.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:51.361 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:51.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:51.399 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:51.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:51.500 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:51.530 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:51.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:51.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:51.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:51 smithi184 ceph-mon[92527]: pgmap v1802: 119 pgs: 119 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s 2024-09-13T14:52:51.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:51 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/4132528894' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:51.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:51 smithi151 ceph-mon[104450]: pgmap v1802: 119 pgs: 119 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s 2024-09-13T14:52:51.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:51 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/4132528894' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:51.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:51 smithi151 ceph-mon[108966]: pgmap v1802: 119 pgs: 119 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s 2024-09-13T14:52:51.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:51 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/4132528894' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:51.631 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:51.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:51.653 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:51.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:51.754 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:51.789 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:51.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:51.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:51.890 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:51.925 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:51.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:51.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:52.027 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:52.056 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:52.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:52.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:52.158 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:52.196 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:52.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:52.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:52.298 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:52.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:52.337 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:52.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:52.438 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:52.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:52.463 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:52.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:52.565 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:52.586 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:52.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:52.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:52.688 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:52.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:52.720 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:52.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:52.821 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:52.846 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:52.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:52.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:52.948 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:52.970 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:52.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:52.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:53.072 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:53.104 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:53.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:53.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:53.206 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:53.240 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:52 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a[104427]: 2024-09-13T14:52:52.971+0000 7fe9da14f640 -1 mon.a@0(leader).osd e1194 definitely_dead 0 2024-09-13T14:52:53.240 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:53 smithi151 ceph-mon[104450]: pgmap v1803: 119 pgs: 119 active+clean; 2.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-13T14:52:53.240 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:53 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3270743636' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-13T14:52:53.240 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:53 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-13T14:52:53.240 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:53 smithi151 ceph-mon[104450]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-13T14:52:53.240 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:53 smithi151 ceph-mon[108966]: pgmap v1803: 119 pgs: 119 active+clean; 2.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-13T14:52:53.240 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:53 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3270743636' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-13T14:52:53.241 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:53 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-13T14:52:53.241 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:53 smithi151 ceph-mon[108966]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-13T14:52:53.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:53.249 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:53.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:53.350 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:53.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:53.372 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:53.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:53.473 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:53.492 INFO:teuthology.orchestra.run.smithi151.stderr:marked out osd.5. 2024-09-13T14:52:53.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:53.496 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:53.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:53.597 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:53.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:53 smithi184 ceph-mon[92527]: pgmap v1803: 119 pgs: 119 active+clean; 2.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-13T14:52:53.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:53 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3270743636' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-13T14:52:53.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:53 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-13T14:52:53.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:53 smithi184 ceph-mon[92527]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-13T14:52:53.619 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:53.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:53.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:53.721 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:53.742 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:53.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:53.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:53.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:53.865 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:53.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:53.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:53.967 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:53.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:53.991 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:53.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:54.092 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:54.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:54.114 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:54.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:54.215 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:54.252 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:54.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:54.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:54.355 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:54.378 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:54.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:54.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:54.480 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:54.502 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:54.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:54.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:54.589 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:54 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-13T14:52:54.590 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:54 smithi151 ceph-mon[104450]: osdmap e1195: 8 total, 8 up, 6 in 2024-09-13T14:52:54.590 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:54 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-13T14:52:54.590 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:54 smithi151 ceph-mon[108966]: osdmap e1195: 8 total, 8 up, 6 in 2024-09-13T14:52:54.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:54 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-13T14:52:54.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:54 smithi184 ceph-mon[92527]: osdmap e1195: 8 total, 8 up, 6 in 2024-09-13T14:52:54.604 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:54.628 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:54.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:54.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:54.647 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:54.730 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:54.751 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:54.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:54.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:54.854 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:54.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:52:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:52:54] "GET /metrics HTTP/1.1" 200 36650 "" "Prometheus/2.51.0" 2024-09-13T14:52:54.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:54.880 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:54.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:54.922 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:54.981 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:55.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:55.004 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:55.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:55.105 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:55.131 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:55.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:55.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:55.233 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:55.262 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:55.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:55.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:55.364 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:55.387 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:55.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:55.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:55.490 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:55.511 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:55.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:55.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:55.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:55 smithi184 ceph-mon[92527]: pgmap v1805: 119 pgs: 13 peering, 106 active+clean; 1.9 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-13T14:52:55.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:55 smithi184 ceph-mon[92527]: Health check failed: Reduced data availability: 6 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-09-13T14:52:55.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:55 smithi184 ceph-mon[92527]: osdmap e1196: 8 total, 8 up, 6 in 2024-09-13T14:52:55.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:55 smithi184 ceph-mon[92527]: 1.a scrub starts 2024-09-13T14:52:55.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:55 smithi184 ceph-mon[92527]: 1.a scrub ok 2024-09-13T14:52:55.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:55 smithi151 ceph-mon[104450]: pgmap v1805: 119 pgs: 13 peering, 106 active+clean; 1.9 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-13T14:52:55.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:55 smithi151 ceph-mon[104450]: Health check failed: Reduced data availability: 6 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-09-13T14:52:55.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:55 smithi151 ceph-mon[104450]: osdmap e1196: 8 total, 8 up, 6 in 2024-09-13T14:52:55.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:55 smithi151 ceph-mon[104450]: 1.a scrub starts 2024-09-13T14:52:55.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:55 smithi151 ceph-mon[104450]: 1.a scrub ok 2024-09-13T14:52:55.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:55 smithi151 ceph-mon[108966]: pgmap v1805: 119 pgs: 13 peering, 106 active+clean; 1.9 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-13T14:52:55.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:55 smithi151 ceph-mon[108966]: Health check failed: Reduced data availability: 6 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-09-13T14:52:55.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:55 smithi151 ceph-mon[108966]: osdmap e1196: 8 total, 8 up, 6 in 2024-09-13T14:52:55.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:55 smithi151 ceph-mon[108966]: 1.a scrub starts 2024-09-13T14:52:55.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:55 smithi151 ceph-mon[108966]: 1.a scrub ok 2024-09-13T14:52:55.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:55.635 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:55.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:55.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:55.737 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:55.759 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:55.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:55.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:55.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:55.903 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:55.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:55.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:56.005 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:56.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:56.051 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:56.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:56.152 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:56.179 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:56.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:56.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:56.281 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:56.305 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:56.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:56.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:56.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:56 smithi151 ceph-mon[104450]: 1.17 scrub starts 2024-09-13T14:52:56.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:56 smithi151 ceph-mon[104450]: 1.17 scrub ok 2024-09-13T14:52:56.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:56 smithi151 ceph-mon[104450]: 1.12 scrub starts 2024-09-13T14:52:56.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:56 smithi151 ceph-mon[104450]: 1.12 scrub ok 2024-09-13T14:52:56.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:56 smithi151 ceph-mon[104450]: 8.0 starting backfill to osd.6 from (0'0,0'0] MAX to 1194'13608 2024-09-13T14:52:56.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:56 smithi151 ceph-mon[104450]: 8.8 starting backfill to osd.0 from (0'0,0'0] MAX to 1194'13884 2024-09-13T14:52:56.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:56 smithi151 ceph-mon[104450]: 8.d starting backfill to osd.6 from (0'0,0'0] MAX to 1194'14062 2024-09-13T14:52:56.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:56 smithi151 ceph-mon[104450]: osdmap e1197: 8 total, 8 up, 6 in 2024-09-13T14:52:56.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:56 smithi151 ceph-mon[104450]: 8.4 starting backfill to osd.3 from (0'0,0'0] MAX to 1194'13568 2024-09-13T14:52:56.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:56 smithi151 ceph-mon[104450]: 8.9 starting backfill to osd.3 from (0'0,0'0] MAX to 1194'13613 2024-09-13T14:52:56.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:56 smithi151 ceph-mon[104450]: 8.a starting backfill to osd.3 from (0'0,0'0] MAX to 1194'14120 2024-09-13T14:52:56.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:56 smithi151 ceph-mon[104450]: 8.3 starting backfill to osd.2 from (0'0,0'0] MAX to 1194'13675 2024-09-13T14:52:56.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:56 smithi151 ceph-mon[104450]: 8.2 starting backfill to osd.6 from (0'0,0'0] MAX to 1194'13737 2024-09-13T14:52:56.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:56 smithi151 ceph-mon[104450]: 1.8 scrub starts 2024-09-13T14:52:56.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:56 smithi151 ceph-mon[104450]: 1.8 scrub ok 2024-09-13T14:52:56.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:56 smithi151 ceph-mon[108966]: 1.17 scrub starts 2024-09-13T14:52:56.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:56 smithi151 ceph-mon[108966]: 1.17 scrub ok 2024-09-13T14:52:56.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:56 smithi151 ceph-mon[108966]: 1.12 scrub starts 2024-09-13T14:52:56.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:56 smithi151 ceph-mon[108966]: 1.12 scrub ok 2024-09-13T14:52:56.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:56 smithi151 ceph-mon[108966]: 8.0 starting backfill to osd.6 from (0'0,0'0] MAX to 1194'13608 2024-09-13T14:52:56.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:56 smithi151 ceph-mon[108966]: 8.8 starting backfill to osd.0 from (0'0,0'0] MAX to 1194'13884 2024-09-13T14:52:56.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:56 smithi151 ceph-mon[108966]: 8.d starting backfill to osd.6 from (0'0,0'0] MAX to 1194'14062 2024-09-13T14:52:56.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:56 smithi151 ceph-mon[108966]: osdmap e1197: 8 total, 8 up, 6 in 2024-09-13T14:52:56.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:56 smithi151 ceph-mon[108966]: 8.4 starting backfill to osd.3 from (0'0,0'0] MAX to 1194'13568 2024-09-13T14:52:56.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:56 smithi151 ceph-mon[108966]: 8.9 starting backfill to osd.3 from (0'0,0'0] MAX to 1194'13613 2024-09-13T14:52:56.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:56 smithi151 ceph-mon[108966]: 8.a starting backfill to osd.3 from (0'0,0'0] MAX to 1194'14120 2024-09-13T14:52:56.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:56 smithi151 ceph-mon[108966]: 8.3 starting backfill to osd.2 from (0'0,0'0] MAX to 1194'13675 2024-09-13T14:52:56.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:56 smithi151 ceph-mon[108966]: 8.2 starting backfill to osd.6 from (0'0,0'0] MAX to 1194'13737 2024-09-13T14:52:56.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:56 smithi151 ceph-mon[108966]: 1.8 scrub starts 2024-09-13T14:52:56.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:56 smithi151 ceph-mon[108966]: 1.8 scrub ok 2024-09-13T14:52:56.406 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:56.448 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:56.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:56.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:56.550 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:56.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:56.577 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:56.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:56.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:56 smithi184 ceph-mon[92527]: 1.17 scrub starts 2024-09-13T14:52:56.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:56 smithi184 ceph-mon[92527]: 1.17 scrub ok 2024-09-13T14:52:56.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:56 smithi184 ceph-mon[92527]: 1.12 scrub starts 2024-09-13T14:52:56.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:56 smithi184 ceph-mon[92527]: 1.12 scrub ok 2024-09-13T14:52:56.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:56 smithi184 ceph-mon[92527]: 8.0 starting backfill to osd.6 from (0'0,0'0] MAX to 1194'13608 2024-09-13T14:52:56.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:56 smithi184 ceph-mon[92527]: 8.8 starting backfill to osd.0 from (0'0,0'0] MAX to 1194'13884 2024-09-13T14:52:56.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:56 smithi184 ceph-mon[92527]: 8.d starting backfill to osd.6 from (0'0,0'0] MAX to 1194'14062 2024-09-13T14:52:56.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:56 smithi184 ceph-mon[92527]: osdmap e1197: 8 total, 8 up, 6 in 2024-09-13T14:52:56.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:56 smithi184 ceph-mon[92527]: 8.4 starting backfill to osd.3 from (0'0,0'0] MAX to 1194'13568 2024-09-13T14:52:56.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:56 smithi184 ceph-mon[92527]: 8.9 starting backfill to osd.3 from (0'0,0'0] MAX to 1194'13613 2024-09-13T14:52:56.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:56 smithi184 ceph-mon[92527]: 8.a starting backfill to osd.3 from (0'0,0'0] MAX to 1194'14120 2024-09-13T14:52:56.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:56 smithi184 ceph-mon[92527]: 8.3 starting backfill to osd.2 from (0'0,0'0] MAX to 1194'13675 2024-09-13T14:52:56.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:56 smithi184 ceph-mon[92527]: 8.2 starting backfill to osd.6 from (0'0,0'0] MAX to 1194'13737 2024-09-13T14:52:56.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:56 smithi184 ceph-mon[92527]: 1.8 scrub starts 2024-09-13T14:52:56.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:56 smithi184 ceph-mon[92527]: 1.8 scrub ok 2024-09-13T14:52:56.675 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:52:56.678 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:56.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:56.758 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:56.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:56.859 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:56.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:56.917 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:56.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:57.018 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:57.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:57.062 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:57.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:57.163 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:57.206 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:57.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:57.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:57.308 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:57.332 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:57.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:57.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:57.434 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:57.461 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:57.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:57.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:57.545 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:57 smithi151 ceph-mon[104450]: pgmap v1808: 119 pgs: 4 remapped+peering, 13 peering, 102 active+clean; 1.8 GiB data, 7.9 GiB used, 528 GiB / 536 GiB avail; 13 KiB/s rd, 29 KiB/s wr, 1.08k op/s; 10 KiB/s, 0 objects/s recovering 2024-09-13T14:52:57.545 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:57 smithi151 ceph-mon[104450]: 1.1b scrub starts 2024-09-13T14:52:57.545 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:57 smithi151 ceph-mon[104450]: 1.1b scrub ok 2024-09-13T14:52:57.545 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:57 smithi151 ceph-mon[104450]: 1.18 scrub starts 2024-09-13T14:52:57.546 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:57 smithi151 ceph-mon[104450]: 1.18 scrub ok 2024-09-13T14:52:57.546 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:57 smithi151 ceph-mon[108966]: pgmap v1808: 119 pgs: 4 remapped+peering, 13 peering, 102 active+clean; 1.8 GiB data, 7.9 GiB used, 528 GiB / 536 GiB avail; 13 KiB/s rd, 29 KiB/s wr, 1.08k op/s; 10 KiB/s, 0 objects/s recovering 2024-09-13T14:52:57.546 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:57 smithi151 ceph-mon[108966]: 1.1b scrub starts 2024-09-13T14:52:57.546 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:57 smithi151 ceph-mon[108966]: 1.1b scrub ok 2024-09-13T14:52:57.546 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:57 smithi151 ceph-mon[108966]: 1.18 scrub starts 2024-09-13T14:52:57.546 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:57 smithi151 ceph-mon[108966]: 1.18 scrub ok 2024-09-13T14:52:57.563 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:57.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:57 smithi184 ceph-mon[92527]: pgmap v1808: 119 pgs: 4 remapped+peering, 13 peering, 102 active+clean; 1.8 GiB data, 7.9 GiB used, 528 GiB / 536 GiB avail; 13 KiB/s rd, 29 KiB/s wr, 1.08k op/s; 10 KiB/s, 0 objects/s recovering 2024-09-13T14:52:57.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:57 smithi184 ceph-mon[92527]: 1.1b scrub starts 2024-09-13T14:52:57.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:57 smithi184 ceph-mon[92527]: 1.1b scrub ok 2024-09-13T14:52:57.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:57 smithi184 ceph-mon[92527]: 1.18 scrub starts 2024-09-13T14:52:57.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:57 smithi184 ceph-mon[92527]: 1.18 scrub ok 2024-09-13T14:52:57.605 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:57.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:57.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:57.708 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:57.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:57.733 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:57.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:57.834 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:57.877 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:57.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:57.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:57.979 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:58.006 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:58.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:58.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:58.108 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:58.133 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:58.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:58.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:58.235 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:58.261 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:58.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:58.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:58.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:58 smithi151 ceph-mon[104450]: 1.1 scrub starts 2024-09-13T14:52:58.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:58 smithi151 ceph-mon[104450]: 1.1 scrub ok 2024-09-13T14:52:58.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:58 smithi151 ceph-mon[108966]: 1.1 scrub starts 2024-09-13T14:52:58.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:58 smithi151 ceph-mon[108966]: 1.1 scrub ok 2024-09-13T14:52:58.363 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:58.389 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:58.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:58.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:58.491 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:58.531 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:58.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:58.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:58.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:58 smithi184 ceph-mon[92527]: 1.1 scrub starts 2024-09-13T14:52:58.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:58 smithi184 ceph-mon[92527]: 1.1 scrub ok 2024-09-13T14:52:58.634 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:58.660 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:58.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:58.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:58.761 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:52:58.762 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:52:58.763 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:58.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:58.817 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:58.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:58.918 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:58.945 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:58.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:58.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:59.047 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:59.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:59.090 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:59.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:59.192 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:59.219 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:59.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:59.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:59.228 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 6, 7, 1] out_osds: [4, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:52:59.228 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:52:59.229 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 2, 3, 6, 7, 1] 2024-09-13T14:52:59.229 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd out 1 2024-09-13T14:52:59.262 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-13T14:52:59.320 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:59.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:59.358 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:59.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:59.459 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:59.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:59.517 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:59.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:59.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:59 smithi184 ceph-mon[92527]: pgmap v1809: 119 pgs: 1 active+remapped+backfilling, 4 remapped+peering, 13 peering, 101 active+clean; 1.8 GiB data, 7.9 GiB used, 529 GiB / 536 GiB avail; 13 KiB/s rd, 29 KiB/s wr, 712 op/s; 1619/87228 objects misplaced (1.856%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:52:59.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:52:59 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/619558066' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:59.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:59 smithi151 ceph-mon[104450]: pgmap v1809: 119 pgs: 1 active+remapped+backfilling, 4 remapped+peering, 13 peering, 101 active+clean; 1.8 GiB data, 7.9 GiB used, 529 GiB / 536 GiB avail; 13 KiB/s rd, 29 KiB/s wr, 712 op/s; 1619/87228 objects misplaced (1.856%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:52:59.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:52:59 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/619558066' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:59.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:59 smithi151 ceph-mon[108966]: pgmap v1809: 119 pgs: 1 active+remapped+backfilling, 4 remapped+peering, 13 peering, 101 active+clean; 1.8 GiB data, 7.9 GiB used, 529 GiB / 536 GiB avail; 13 KiB/s rd, 29 KiB/s wr, 712 op/s; 1619/87228 objects misplaced (1.856%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:52:59.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:52:59 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/619558066' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:59.617 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:59.643 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:59.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:59.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:59.718 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T14:52:59.745 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:59.817 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:52:59.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:59.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:59.919 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:59.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:59.945 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:52:59.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:00.047 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:00.103 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:00.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:00.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:00.205 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:00.231 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:00.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:00.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:00.333 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:00.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:00 smithi151 ceph-mon[104450]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:53:00.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:00 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/619558066' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:53:00.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:00 smithi151 ceph-mon[104450]: osdmap e1198: 8 total, 8 up, 6 in 2024-09-13T14:53:00.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:00 smithi151 ceph-mon[108966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:53:00.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:00 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/619558066' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:53:00.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:00 smithi151 ceph-mon[108966]: osdmap e1198: 8 total, 8 up, 6 in 2024-09-13T14:53:00.373 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:00.374 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:00.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:00.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:00.476 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:00.529 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:00.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:00.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:00.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:00 smithi184 ceph-mon[92527]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:53:00.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:00 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/619558066' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:53:00.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:00 smithi184 ceph-mon[92527]: osdmap e1198: 8 total, 8 up, 6 in 2024-09-13T14:53:00.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:53:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:53:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:53:00.631 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:00.687 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:00.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:00.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:00.708 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:00.790 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:00.829 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:00.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:00.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:00.934 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:00.959 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:00.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:00.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:01.061 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:01.105 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:01.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:01.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:01.207 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:01.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:01.246 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:01.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:01.348 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:01.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:01 smithi151 ceph-mon[104450]: pgmap v1811: 119 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 4 remapped+peering, 111 active+clean; 1.7 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 2.0 KiB/s rd, 4.2 KiB/s wr, 398 op/s; 6398/84525 objects misplaced (7.569%); 4.5 MiB/s, 74 objects/s recovering 2024-09-13T14:53:01.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:01 smithi151 ceph-mon[104450]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 6 pgs peering) 2024-09-13T14:53:01.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:01 smithi151 ceph-mon[104450]: Cluster is now healthy 2024-09-13T14:53:01.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:01 smithi151 ceph-mon[108966]: pgmap v1811: 119 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 4 remapped+peering, 111 active+clean; 1.7 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 2.0 KiB/s rd, 4.2 KiB/s wr, 398 op/s; 6398/84525 objects misplaced (7.569%); 4.5 MiB/s, 74 objects/s recovering 2024-09-13T14:53:01.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:01 smithi151 ceph-mon[108966]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 6 pgs peering) 2024-09-13T14:53:01.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:01 smithi151 ceph-mon[108966]: Cluster is now healthy 2024-09-13T14:53:01.373 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:01.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:01.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:01.475 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:01.505 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:01.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:01.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:01.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:01 smithi184 ceph-mon[92527]: pgmap v1811: 119 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 4 remapped+peering, 111 active+clean; 1.7 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 2.0 KiB/s rd, 4.2 KiB/s wr, 398 op/s; 6398/84525 objects misplaced (7.569%); 4.5 MiB/s, 74 objects/s recovering 2024-09-13T14:53:01.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:01 smithi184 ceph-mon[92527]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 6 pgs peering) 2024-09-13T14:53:01.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:01 smithi184 ceph-mon[92527]: Cluster is now healthy 2024-09-13T14:53:01.607 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:01.632 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:01.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:01.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:01.734 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:01.766 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:01.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:01.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:01.868 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:01.893 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:01.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:01.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:01.995 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:02.034 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:02.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:02.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:02.136 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:02.164 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:02.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:02.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:02.269 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:02.312 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:02.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:02.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:02.414 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:02.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:53:02.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:02.635 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:02.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:02.648 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:53:02.648 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:02.648 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:02 smithi151 ceph-mon[104450]: pgmap v1812: 119 pgs: 4 active+remapped+backfill_wait, 4 active+remapped+backfilling, 111 active+clean; 1.6 GiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 617 op/s; 12077/77931 objects misplaced (15.497%); 9.6 MiB/s, 156 objects/s recovering 2024-09-13T14:53:02.649 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:53:02.649 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:02.649 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:02 smithi151 ceph-mon[108966]: pgmap v1812: 119 pgs: 4 active+remapped+backfill_wait, 4 active+remapped+backfilling, 111 active+clean; 1.6 GiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 617 op/s; 12077/77931 objects misplaced (15.497%); 9.6 MiB/s, 156 objects/s recovering 2024-09-13T14:53:02.736 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:02.798 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:02.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:02.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:02.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:53:02.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:02.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:02 smithi184 ceph-mon[92527]: pgmap v1812: 119 pgs: 4 active+remapped+backfill_wait, 4 active+remapped+backfilling, 111 active+clean; 1.6 GiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 617 op/s; 12077/77931 objects misplaced (15.497%); 9.6 MiB/s, 156 objects/s recovering 2024-09-13T14:53:02.899 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:02.957 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:02.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:02.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:03.058 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:03.084 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:03.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:03.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:03.174 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:03.185 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:03.212 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:03.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:03.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:03.314 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:03.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:03.380 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:03.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:03.481 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:03.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:03.506 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:03.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:03.604 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 14:53:03 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=cleanup t=2024-09-13T14:53:03.302247549Z level=info msg="Completed cleanup jobs" duration=12.678847ms 2024-09-13T14:53:03.607 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:03.634 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:03.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:03.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:03.735 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:03.775 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:03.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:03.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:03.877 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:03.903 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:03.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:03.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:04.005 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:04.033 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:04.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:04.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:04.134 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:04.185 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:04.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:04.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:04.287 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:04.312 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:04.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:04.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:04.417 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:04.456 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:04.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:04.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:04.558 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:04.582 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:04.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:04.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:04.683 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:53:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:53:04] "GET /metrics HTTP/1.1" 200 36681 "" "Prometheus/2.51.0" 2024-09-13T14:53:04.684 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:04.732 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:04.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:04.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:04.834 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:04.869 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:04.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:04.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:04.919 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:53:04.920 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:53:04.968 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:53:04.969 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:53:04.969 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:04 smithi151 ceph-mon[104450]: pgmap v1813: 119 pgs: 4 active+remapped+backfill_wait, 4 active+remapped+backfilling, 111 active+clean; 1.5 GiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 606 op/s; 10367/72213 objects misplaced (14.356%); 14 MiB/s, 223 objects/s recovering 2024-09-13T14:53:04.969 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:04 smithi151 ceph-mon[108966]: pgmap v1813: 119 pgs: 4 active+remapped+backfill_wait, 4 active+remapped+backfilling, 111 active+clean; 1.5 GiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 606 op/s; 10367/72213 objects misplaced (14.356%); 14 MiB/s, 223 objects/s recovering 2024-09-13T14:53:04.971 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:05.010 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:05.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:05.015 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:53:05.015 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:53:05.063 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:53:05.063 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:53:05.091 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:53:05.091 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:53:05.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:04 smithi184 ceph-mon[92527]: pgmap v1813: 119 pgs: 4 active+remapped+backfill_wait, 4 active+remapped+backfilling, 111 active+clean; 1.5 GiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 606 op/s; 10367/72213 objects misplaced (14.356%); 14 MiB/s, 223 objects/s recovering 2024-09-13T14:53:05.112 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:05.120 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:53:05.121 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:53:05.151 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:53:05.151 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:53:05.154 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:05.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:05.183 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:53:05.183 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:53:05.256 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:05.280 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:05.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:05.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:05.423 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:05.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:05.525 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:05.553 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:05.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:05.655 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:05.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.681 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:05.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:05.782 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:05.827 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:05.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:05.929 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:05.954 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:05.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:06.056 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:06.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:06.100 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:06.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:06.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:05 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1865646369' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:06.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:05 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1865646369' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:06.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:05 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1865646369' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:06.201 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:06.242 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:06.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:06.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:06.344 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:06.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:06.381 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:06.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:06.483 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:06.520 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:06.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:06.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:06.622 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:06.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:06.659 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:06.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:06.761 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:06.782 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:06.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:06.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:06.884 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:06.909 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:06.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:06.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:07.011 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:07.036 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:07.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:07.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:07.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:06 smithi184 ceph-mon[92527]: pgmap v1814: 119 pgs: 4 active+remapped+backfill_wait, 4 active+remapped+backfilling, 111 active+clean; 1.3 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 765 op/s; 8628/64536 objects misplaced (13.369%); 17 MiB/s, 267 objects/s recovering 2024-09-13T14:53:07.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:06 smithi151 ceph-mon[104450]: pgmap v1814: 119 pgs: 4 active+remapped+backfill_wait, 4 active+remapped+backfilling, 111 active+clean; 1.3 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 765 op/s; 8628/64536 objects misplaced (13.369%); 17 MiB/s, 267 objects/s recovering 2024-09-13T14:53:07.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:06 smithi151 ceph-mon[108966]: pgmap v1814: 119 pgs: 4 active+remapped+backfill_wait, 4 active+remapped+backfilling, 111 active+clean; 1.3 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 765 op/s; 8628/64536 objects misplaced (13.369%); 17 MiB/s, 267 objects/s recovering 2024-09-13T14:53:07.138 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:07.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:07.163 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:07.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:07.263 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:07.298 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:07.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:07.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:07.400 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:07.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:07.424 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:07.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:07.525 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:07.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:07.569 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:07.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:07.670 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:07.696 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:07.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:07.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:07.798 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:07.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:07.823 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:07.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:07.924 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:07.964 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:07.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:07.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:08.066 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:08.093 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:08.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:08.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:08.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:07 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:53:08.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:07 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:53:08.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:07 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:53:08.198 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:08.233 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:08.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:08.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:08.335 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:08.362 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:08.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:08.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:08.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:08.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:08.506 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:08.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:08.607 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:08.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:08.634 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:08.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:08.735 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:08.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:08.779 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:08.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:08.881 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:08.925 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:08.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:08.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:09.028 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:09.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.060 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:09.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:09.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:08 smithi184 ceph-mon[92527]: pgmap v1815: 119 pgs: 4 active+remapped+backfill_wait, 4 active+remapped+backfilling, 111 active+clean; 1.3 GiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 778 op/s; 8183/63801 objects misplaced (12.826%); 18 MiB/s, 285 objects/s recovering 2024-09-13T14:53:09.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:08 smithi151 ceph-mon[104450]: pgmap v1815: 119 pgs: 4 active+remapped+backfill_wait, 4 active+remapped+backfilling, 111 active+clean; 1.3 GiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 778 op/s; 8183/63801 objects misplaced (12.826%); 18 MiB/s, 285 objects/s recovering 2024-09-13T14:53:09.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:08 smithi151 ceph-mon[108966]: pgmap v1815: 119 pgs: 4 active+remapped+backfill_wait, 4 active+remapped+backfilling, 111 active+clean; 1.3 GiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 778 op/s; 8183/63801 objects misplaced (12.826%); 18 MiB/s, 285 objects/s recovering 2024-09-13T14:53:09.162 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:09.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.188 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:09.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:09.289 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:09.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.324 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:09.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:09.425 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:09.459 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:09.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:09.561 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:09.583 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:09.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:09.685 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:09.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.723 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:09.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:09.824 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:09.848 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:09.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:09.950 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:09.978 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:09.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:10.081 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:10.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:10.116 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:10.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:10.218 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:10.259 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:10.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:10.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:10.361 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:10.383 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:10.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:10.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:10.484 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:10.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:10.512 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:10.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:10.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:53:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:53:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:53:10.613 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:10.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:10.643 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:10.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:10.744 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:10.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:10.766 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:10.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:10.867 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:10.904 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:10.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:10.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:11.006 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:11.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:11.032 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:11.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:11.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:10 smithi184 ceph-mon[92527]: pgmap v1816: 119 pgs: 4 active+remapped+backfill_wait, 4 active+remapped+backfilling, 111 active+clean; 1.2 GiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 821 op/s; 7159/58962 objects misplaced (12.142%); 17 MiB/s, 276 objects/s recovering 2024-09-13T14:53:11.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:10 smithi151 ceph-mon[104450]: pgmap v1816: 119 pgs: 4 active+remapped+backfill_wait, 4 active+remapped+backfilling, 111 active+clean; 1.2 GiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 821 op/s; 7159/58962 objects misplaced (12.142%); 17 MiB/s, 276 objects/s recovering 2024-09-13T14:53:11.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:10 smithi151 ceph-mon[108966]: pgmap v1816: 119 pgs: 4 active+remapped+backfill_wait, 4 active+remapped+backfilling, 111 active+clean; 1.2 GiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 821 op/s; 7159/58962 objects misplaced (12.142%); 17 MiB/s, 276 objects/s recovering 2024-09-13T14:53:11.133 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:11.156 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:11.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:11.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:11.258 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:11.280 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:11.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:11.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:11.382 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:11.405 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:11.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:11.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:11.506 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:11.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:11.530 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:11.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:11.631 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:11.653 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:11.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:11.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:11.755 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:11.778 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:11.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:11.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:11.880 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:11.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:11.915 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:11.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:12.016 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:12.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:12.038 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:12.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:12.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:11 smithi184 ceph-mon[92527]: osdmap e1199: 8 total, 8 up, 6 in 2024-09-13T14:53:12.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:11 smithi151 ceph-mon[104450]: osdmap e1199: 8 total, 8 up, 6 in 2024-09-13T14:53:12.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:11 smithi151 ceph-mon[108966]: osdmap e1199: 8 total, 8 up, 6 in 2024-09-13T14:53:12.140 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:12.162 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:12.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:12.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:12.263 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:12.286 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:12.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:12.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:12.388 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:12.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:12.411 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:12.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:12.512 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:12.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:12.534 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:12.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:12.636 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:12.656 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:12.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:12.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:12.758 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:12.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:12.780 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:12.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:12.881 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:12.909 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:12.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:12.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:13.012 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:13.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:13.041 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:13.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:13.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:12 smithi184 ceph-mon[92527]: pgmap v1818: 119 pgs: 4 active+remapped+backfill_wait, 4 active+remapped+backfilling, 111 active+clean; 1.1 GiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 858 op/s; 5764/52179 objects misplaced (11.047%); 18 MiB/s, 285 objects/s recovering 2024-09-13T14:53:13.106 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:12 smithi184 ceph-mon[92527]: osdmap e1200: 8 total, 8 up, 6 in 2024-09-13T14:53:13.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:12 smithi151 ceph-mon[104450]: pgmap v1818: 119 pgs: 4 active+remapped+backfill_wait, 4 active+remapped+backfilling, 111 active+clean; 1.1 GiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 858 op/s; 5764/52179 objects misplaced (11.047%); 18 MiB/s, 285 objects/s recovering 2024-09-13T14:53:13.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:12 smithi151 ceph-mon[104450]: osdmap e1200: 8 total, 8 up, 6 in 2024-09-13T14:53:13.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:12 smithi151 ceph-mon[108966]: pgmap v1818: 119 pgs: 4 active+remapped+backfill_wait, 4 active+remapped+backfilling, 111 active+clean; 1.1 GiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 858 op/s; 5764/52179 objects misplaced (11.047%); 18 MiB/s, 285 objects/s recovering 2024-09-13T14:53:13.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:12 smithi151 ceph-mon[108966]: osdmap e1200: 8 total, 8 up, 6 in 2024-09-13T14:53:13.142 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:13.177 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:13.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:13.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:13.279 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:13.300 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:13.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:13.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:13.402 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:13.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:13.426 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:13.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:13.527 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:13.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:13.549 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:13.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:13.650 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:13.674 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:13.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:13.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:13.775 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:13.799 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:13.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:13.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:13.901 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:13.929 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:13.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:13.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:14.031 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:14.066 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:14.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:14.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:14.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:13 smithi184 ceph-mon[92527]: osdmap e1201: 8 total, 8 up, 6 in 2024-09-13T14:53:14.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:13 smithi184 ceph-mon[92527]: 1.10 scrub starts 2024-09-13T14:53:14.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:13 smithi184 ceph-mon[92527]: 1.10 scrub ok 2024-09-13T14:53:14.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:13 smithi151 ceph-mon[104450]: osdmap e1201: 8 total, 8 up, 6 in 2024-09-13T14:53:14.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:13 smithi151 ceph-mon[104450]: 1.10 scrub starts 2024-09-13T14:53:14.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:13 smithi151 ceph-mon[104450]: 1.10 scrub ok 2024-09-13T14:53:14.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:13 smithi151 ceph-mon[108966]: osdmap e1201: 8 total, 8 up, 6 in 2024-09-13T14:53:14.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:13 smithi151 ceph-mon[108966]: 1.10 scrub starts 2024-09-13T14:53:14.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:13 smithi151 ceph-mon[108966]: 1.10 scrub ok 2024-09-13T14:53:14.168 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:14.192 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:14.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:14.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:14.294 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:14.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:14.334 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:14.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:14.435 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:14.471 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:14.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:14.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:14.500 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:14.573 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:14.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:14.627 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:14.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:14.728 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:14.752 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:14.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:14.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:14.807 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:53:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:53:14] "GET /metrics HTTP/1.1" 200 36681 "" "Prometheus/2.51.0" 2024-09-13T14:53:14.854 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:14.898 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:14.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:14.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:14.999 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:15.042 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:15.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:15.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:15.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:14 smithi184 ceph-mon[92527]: pgmap v1821: 119 pgs: 1 active+clean+remapped, 1 peering, 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 112 active+clean; 1018 MiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 826 op/s; 3474/48915 objects misplaced (7.102%); 31 MiB/s, 489 objects/s recovering 2024-09-13T14:53:15.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:14 smithi184 ceph-mon[92527]: osdmap e1202: 8 total, 8 up, 6 in 2024-09-13T14:53:15.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:14 smithi151 ceph-mon[104450]: pgmap v1821: 119 pgs: 1 active+clean+remapped, 1 peering, 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 112 active+clean; 1018 MiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 826 op/s; 3474/48915 objects misplaced (7.102%); 31 MiB/s, 489 objects/s recovering 2024-09-13T14:53:15.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:14 smithi151 ceph-mon[104450]: osdmap e1202: 8 total, 8 up, 6 in 2024-09-13T14:53:15.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:14 smithi151 ceph-mon[108966]: pgmap v1821: 119 pgs: 1 active+clean+remapped, 1 peering, 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 112 active+clean; 1018 MiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 826 op/s; 3474/48915 objects misplaced (7.102%); 31 MiB/s, 489 objects/s recovering 2024-09-13T14:53:15.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:14 smithi151 ceph-mon[108966]: osdmap e1202: 8 total, 8 up, 6 in 2024-09-13T14:53:15.145 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:15.190 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:15.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:15.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:15.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:15.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:15.334 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:15.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:15.434 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:15.471 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:15.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:15.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:15.573 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:15.610 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:15.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:15.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:15.712 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:15.751 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:15.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:15.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:15.853 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:15.878 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:15.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:15.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:15.980 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:16.003 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:16.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:16.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:16.105 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:16.127 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:16.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:16.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:16.229 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:16.250 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:16.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:16.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:16.352 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:16.377 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:16.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:16.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:16.479 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:16.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:16.501 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:16.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:16.602 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:16.625 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:16.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:16.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:16.727 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:16.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:16.749 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:16.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:16.851 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:16.872 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:16.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:16.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:16.973 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:16.997 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:16.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:16.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:17.100 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:17.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:16 smithi184 ceph-mon[92527]: pgmap v1823: 119 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 1 peering, 2 active+remapped+backfill_wait, 114 active+clean; 947 MiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 459 op/s; 1885/45510 objects misplaced (4.142%); 42 MiB/s, 676 objects/s recovering 2024-09-13T14:53:17.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:16 smithi184 ceph-mon[92527]: osdmap e1203: 8 total, 8 up, 6 in 2024-09-13T14:53:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:17.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:16 smithi151 ceph-mon[104450]: pgmap v1823: 119 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 1 peering, 2 active+remapped+backfill_wait, 114 active+clean; 947 MiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 459 op/s; 1885/45510 objects misplaced (4.142%); 42 MiB/s, 676 objects/s recovering 2024-09-13T14:53:17.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:16 smithi151 ceph-mon[104450]: osdmap e1203: 8 total, 8 up, 6 in 2024-09-13T14:53:17.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:17.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:16 smithi151 ceph-mon[108966]: pgmap v1823: 119 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 1 peering, 2 active+remapped+backfill_wait, 114 active+clean; 947 MiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 459 op/s; 1885/45510 objects misplaced (4.142%); 42 MiB/s, 676 objects/s recovering 2024-09-13T14:53:17.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:16 smithi151 ceph-mon[108966]: osdmap e1203: 8 total, 8 up, 6 in 2024-09-13T14:53:17.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:17.124 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:17.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:17.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:17.226 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:17.258 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:53:17.258 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-13T14:53:17.258 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-13T14:53:17.259 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-13T14:53:17.259 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:53:17.264 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:17.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:17.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:17.366 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:17.389 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:17.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:17.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:17.491 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:17.515 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:17.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:17.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:17.617 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:17.641 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:17.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:17.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:17.743 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:17.768 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:17.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:17.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:17.869 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:17.907 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:17.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:17.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:18.009 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:18.034 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:18.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:18.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:18.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:17 smithi184 ceph-mon[92527]: osdmap e1204: 8 total, 8 up, 6 in 2024-09-13T14:53:18.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:17 smithi151 ceph-mon[108966]: osdmap e1204: 8 total, 8 up, 6 in 2024-09-13T14:53:18.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:17 smithi151 ceph-mon[104450]: osdmap e1204: 8 total, 8 up, 6 in 2024-09-13T14:53:18.136 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:18.159 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:18.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:18.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:18.187 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T14:53:18.261 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:18.285 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:18.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:18.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:18.387 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:18.409 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:18.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:18.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:18.511 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:18.562 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:18.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:18.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:18.664 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:18.688 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:18.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:18.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:18.790 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:18.813 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:18.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:18.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:18.915 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:18.952 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:18.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:18.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:19.053 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:19.076 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:19.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:19.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:19.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:18 smithi184 ceph-mon[92527]: pgmap v1826: 119 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 935 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 423 op/s; 1885/44931 objects misplaced (4.195%); 23 MiB/s, 371 objects/s recovering 2024-09-13T14:53:19.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:18 smithi151 ceph-mon[104450]: pgmap v1826: 119 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 935 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 423 op/s; 1885/44931 objects misplaced (4.195%); 23 MiB/s, 371 objects/s recovering 2024-09-13T14:53:19.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:18 smithi151 ceph-mon[108966]: pgmap v1826: 119 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 2 active+remapped+backfill_wait, 115 active+clean; 935 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 423 op/s; 1885/44931 objects misplaced (4.195%); 23 MiB/s, 371 objects/s recovering 2024-09-13T14:53:19.179 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:19.201 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:19.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:19.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:19.303 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:19.328 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:19.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:19.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:19.430 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:19.472 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:19.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:19.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:19.574 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:19.596 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:19.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:19.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:19.698 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:19.720 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:19.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:19.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:19.822 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:19.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:19.845 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:19.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:19.920 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:19.926 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-13T14:53:19.945 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:19.968 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:19.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:19.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:20.070 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:20.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:20.094 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:20.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:20.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:20.237 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:20.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:20.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:20.294 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:20 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a[104427]: 2024-09-13T14:53:19.996+0000 7fe9da14f640 -1 mon.a@0(leader).osd e1206 definitely_dead 0 2024-09-13T14:53:20.340 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:20.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:20.362 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:20.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:20.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:20.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:20.485 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:20.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:20.587 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:20.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:53:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:53:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:53:20.608 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:20.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:20.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:20.710 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:20.733 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:20.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:20.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:20.835 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:20.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:20.858 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:20.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:20.936 INFO:teuthology.orchestra.run.smithi151.stderr:marked out osd.1. 2024-09-13T14:53:20.958 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:20.965 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:20 smithi151 ceph-mon[104450]: pgmap v1827: 119 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_wait, 116 active+clean; 856 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 432 op/s; 1071/41136 objects misplaced (2.604%); 21 MiB/s, 338 objects/s recovering 2024-09-13T14:53:20.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:20 smithi151 ceph-mon[104450]: osdmap e1205: 8 total, 8 up, 6 in 2024-09-13T14:53:20.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:20 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3250108915' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:53:20.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:20 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:53:20.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:20 smithi151 ceph-mon[104450]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:53:20.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:20 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:53:20.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:20 smithi151 ceph-mon[104450]: osdmap e1206: 8 total, 8 up, 6 in 2024-09-13T14:53:20.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:20 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1489355751' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-13T14:53:20.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:20 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-13T14:53:20.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:20 smithi151 ceph-mon[104450]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-13T14:53:20.967 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:20 smithi151 ceph-mon[108966]: pgmap v1827: 119 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_wait, 116 active+clean; 856 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 432 op/s; 1071/41136 objects misplaced (2.604%); 21 MiB/s, 338 objects/s recovering 2024-09-13T14:53:20.967 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:20 smithi151 ceph-mon[108966]: osdmap e1205: 8 total, 8 up, 6 in 2024-09-13T14:53:20.967 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:20 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3250108915' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:53:20.967 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:20 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:53:20.967 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:20 smithi151 ceph-mon[108966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:53:20.967 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:20 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:53:20.967 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:20 smithi151 ceph-mon[108966]: osdmap e1206: 8 total, 8 up, 6 in 2024-09-13T14:53:20.967 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:20 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1489355751' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-13T14:53:20.967 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:20 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-13T14:53:20.967 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:20 smithi151 ceph-mon[108966]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-13T14:53:20.979 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:20.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:20.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:21.081 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:21.102 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:21.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:21.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:21.113 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:53:21.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:21.225 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:21.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:21.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:21.327 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:21.350 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:21.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:21.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:21.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:20 smithi184 ceph-mon[92527]: pgmap v1827: 119 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_wait, 116 active+clean; 856 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 432 op/s; 1071/41136 objects misplaced (2.604%); 21 MiB/s, 338 objects/s recovering 2024-09-13T14:53:21.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:20 smithi184 ceph-mon[92527]: osdmap e1205: 8 total, 8 up, 6 in 2024-09-13T14:53:21.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:20 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3250108915' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:53:21.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:20 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:53:21.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:20 smithi184 ceph-mon[92527]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:53:21.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:20 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:53:21.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:20 smithi184 ceph-mon[92527]: osdmap e1206: 8 total, 8 up, 6 in 2024-09-13T14:53:21.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:20 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1489355751' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-13T14:53:21.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:20 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-13T14:53:21.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:20 smithi184 ceph-mon[92527]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-13T14:53:21.451 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:21.473 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:21.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:21.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:21.576 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:21.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:21.599 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:21.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:21.700 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:21.721 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:21.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:21.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:21.823 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:21.844 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:21.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:21.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:21.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:21.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:21.982 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:21.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:22.083 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:22.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:22.106 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:22.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:22.208 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:22.225 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:22.231 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:22.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:22.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:22.332 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:22.352 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:21 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-13T14:53:22.352 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:21 smithi151 ceph-mon[104450]: osdmap e1207: 8 total, 8 up, 5 in 2024-09-13T14:53:22.352 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:21 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-13T14:53:22.352 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:21 smithi151 ceph-mon[108966]: osdmap e1207: 8 total, 8 up, 5 in 2024-09-13T14:53:22.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:21 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-13T14:53:22.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:21 smithi184 ceph-mon[92527]: osdmap e1207: 8 total, 8 up, 5 in 2024-09-13T14:53:22.355 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:22.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:22.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:22.458 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:22.479 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:22.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:22.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:22.581 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:22.605 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:22.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:22.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:22.706 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:22.728 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:22.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:22.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:22.738 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:53:22.830 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:22.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:22.855 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:22.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:22.956 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:22.979 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:22.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:22.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:23.082 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:23.105 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:23.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:23.107 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:22 smithi151 ceph-mon[104450]: pgmap v1831: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 722 MiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 0 B/s wr, 751 op/s; 115/34692 objects misplaced (0.331%); 20 MiB/s, 317 objects/s recovering 2024-09-13T14:53:23.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:22 smithi151 ceph-mon[104450]: osdmap e1208: 8 total, 8 up, 5 in 2024-09-13T14:53:23.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:22 smithi151 ceph-mon[108966]: pgmap v1831: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 722 MiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 0 B/s wr, 751 op/s; 115/34692 objects misplaced (0.331%); 20 MiB/s, 317 objects/s recovering 2024-09-13T14:53:23.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:22 smithi151 ceph-mon[108966]: osdmap e1208: 8 total, 8 up, 5 in 2024-09-13T14:53:23.206 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:23.228 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:23.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:23.330 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:23.353 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:23.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:23.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:22 smithi184 ceph-mon[92527]: pgmap v1831: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 722 MiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 0 B/s wr, 751 op/s; 115/34692 objects misplaced (0.331%); 20 MiB/s, 317 objects/s recovering 2024-09-13T14:53:23.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:22 smithi184 ceph-mon[92527]: osdmap e1208: 8 total, 8 up, 5 in 2024-09-13T14:53:23.455 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:23.481 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:23.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:23.583 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:23.608 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:23.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:23.710 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:23.733 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:23.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:23.834 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:23.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.861 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:23.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:23.962 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:23.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.989 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:23.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:24.090 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:24.116 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:24.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:24.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:24.218 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:24.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:24.244 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:24.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:24.245 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:23 smithi151 ceph-mon[104450]: 1.b scrub starts 2024-09-13T14:53:24.245 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:23 smithi151 ceph-mon[104450]: 8.3 starting backfill to osd.3 from (0'0,0'0] MAX to 1206'14860 2024-09-13T14:53:24.245 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:23 smithi151 ceph-mon[104450]: osdmap e1209: 8 total, 8 up, 5 in 2024-09-13T14:53:24.245 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:23 smithi151 ceph-mon[104450]: 8.b starting backfill to osd.7 from (0'0,0'0] MAX to 1206'15009 2024-09-13T14:53:24.245 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:23 smithi151 ceph-mon[104450]: 8.7 starting backfill to osd.2 from (0'0,0'0] MAX to 1206'14816 2024-09-13T14:53:24.245 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:23 smithi151 ceph-mon[104450]: 8.a starting backfill to osd.6 from (0'0,0'0] MAX to 1205'15265 2024-09-13T14:53:24.245 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:23 smithi151 ceph-mon[104450]: 8.0 starting backfill to osd.0 from (0'0,0'0] MAX to 1206'14766 2024-09-13T14:53:24.245 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:23 smithi151 ceph-mon[104450]: 8.f starting backfill to osd.6 from (0'0,0'0] MAX to 1206'15113 2024-09-13T14:53:24.246 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:23 smithi151 ceph-mon[108966]: 1.b scrub starts 2024-09-13T14:53:24.246 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:23 smithi151 ceph-mon[108966]: 8.3 starting backfill to osd.3 from (0'0,0'0] MAX to 1206'14860 2024-09-13T14:53:24.246 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:23 smithi151 ceph-mon[108966]: osdmap e1209: 8 total, 8 up, 5 in 2024-09-13T14:53:24.246 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:23 smithi151 ceph-mon[108966]: 8.b starting backfill to osd.7 from (0'0,0'0] MAX to 1206'15009 2024-09-13T14:53:24.246 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:23 smithi151 ceph-mon[108966]: 8.7 starting backfill to osd.2 from (0'0,0'0] MAX to 1206'14816 2024-09-13T14:53:24.246 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:23 smithi151 ceph-mon[108966]: 8.a starting backfill to osd.6 from (0'0,0'0] MAX to 1205'15265 2024-09-13T14:53:24.246 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:23 smithi151 ceph-mon[108966]: 8.0 starting backfill to osd.0 from (0'0,0'0] MAX to 1206'14766 2024-09-13T14:53:24.246 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:23 smithi151 ceph-mon[108966]: 8.f starting backfill to osd.6 from (0'0,0'0] MAX to 1206'15113 2024-09-13T14:53:24.345 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:24.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:23 smithi184 ceph-mon[92527]: 1.b scrub starts 2024-09-13T14:53:24.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:23 smithi184 ceph-mon[92527]: 8.3 starting backfill to osd.3 from (0'0,0'0] MAX to 1206'14860 2024-09-13T14:53:24.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:23 smithi184 ceph-mon[92527]: osdmap e1209: 8 total, 8 up, 5 in 2024-09-13T14:53:24.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:23 smithi184 ceph-mon[92527]: 8.b starting backfill to osd.7 from (0'0,0'0] MAX to 1206'15009 2024-09-13T14:53:24.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:23 smithi184 ceph-mon[92527]: 8.7 starting backfill to osd.2 from (0'0,0'0] MAX to 1206'14816 2024-09-13T14:53:24.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:23 smithi184 ceph-mon[92527]: 8.a starting backfill to osd.6 from (0'0,0'0] MAX to 1205'15265 2024-09-13T14:53:24.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:23 smithi184 ceph-mon[92527]: 8.0 starting backfill to osd.0 from (0'0,0'0] MAX to 1206'14766 2024-09-13T14:53:24.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:23 smithi184 ceph-mon[92527]: 8.f starting backfill to osd.6 from (0'0,0'0] MAX to 1206'15113 2024-09-13T14:53:24.385 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:24.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:24.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:24.488 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:24.513 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:24.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:24.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:24.615 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:24.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:24.651 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:24.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:24.752 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:24.775 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:24.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:24.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:24.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:53:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:53:24] "GET /metrics HTTP/1.1" 200 36660 "" "Prometheus/2.51.0" 2024-09-13T14:53:24.877 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:24.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:24.902 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:24.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:25.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:25.050 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:25.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:25.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:25.152 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:25.178 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:25.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:25.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:25.281 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:25.324 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:25.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:25.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:25.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:24 smithi184 ceph-mon[92527]: 8.1 starting backfill to osd.3 from (0'0,0'0] MAX to 1206'15000 2024-09-13T14:53:25.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:24 smithi184 ceph-mon[92527]: 1.16 scrub starts 2024-09-13T14:53:25.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:24 smithi184 ceph-mon[92527]: pgmap v1834: 119 pgs: 2 remapped+peering, 117 active+clean; 683 MiB data, 4.7 GiB used, 442 GiB / 447 GiB avail; 9.0 KiB/s rd, 20 KiB/s wr, 740 op/s; 16 MiB/s, 259 objects/s recovering 2024-09-13T14:53:25.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:24 smithi151 ceph-mon[108966]: 8.1 starting backfill to osd.3 from (0'0,0'0] MAX to 1206'15000 2024-09-13T14:53:25.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:24 smithi151 ceph-mon[108966]: 1.16 scrub starts 2024-09-13T14:53:25.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:24 smithi151 ceph-mon[108966]: pgmap v1834: 119 pgs: 2 remapped+peering, 117 active+clean; 683 MiB data, 4.7 GiB used, 442 GiB / 447 GiB avail; 9.0 KiB/s rd, 20 KiB/s wr, 740 op/s; 16 MiB/s, 259 objects/s recovering 2024-09-13T14:53:25.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:24 smithi151 ceph-mon[104450]: 8.1 starting backfill to osd.3 from (0'0,0'0] MAX to 1206'15000 2024-09-13T14:53:25.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:24 smithi151 ceph-mon[104450]: 1.16 scrub starts 2024-09-13T14:53:25.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:24 smithi151 ceph-mon[104450]: pgmap v1834: 119 pgs: 2 remapped+peering, 117 active+clean; 683 MiB data, 4.7 GiB used, 442 GiB / 447 GiB avail; 9.0 KiB/s rd, 20 KiB/s wr, 740 op/s; 16 MiB/s, 259 objects/s recovering 2024-09-13T14:53:25.392 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:25.426 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:25.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:25.450 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:25.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:25.551 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:25.576 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:25.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:25.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:25.678 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:25.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:25.702 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:25.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:25.804 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:25.828 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:25.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:25.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:25.930 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:25.953 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:25.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:25.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:26.055 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:26.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:26.080 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:26.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:26.181 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:26.224 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:26.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:26.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:26.326 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:26.351 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:26.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:26.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:26.453 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:26.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:26.489 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:26.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:26.590 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:26.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:26.632 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:26.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:26.685 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 6, 7] out_osds: [4, 5, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:53:26.685 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:53:26.686 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-09-13T14:53:26.686 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd in 5 2024-09-13T14:53:26.734 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:26.782 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:26.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:26.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:26.884 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:26.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:26.907 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:26.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:27.008 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:27.030 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:26 smithi151 ceph-mon[104450]: pgmap v1835: 119 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 2 remapped+peering, 112 active+clean; 582 MiB data, 4.5 GiB used, 443 GiB / 447 GiB avail; 11 KiB/s rd, 24 KiB/s wr, 775 op/s; 2511/27978 objects misplaced (8.975%); 12 MiB/s, 195 objects/s recovering 2024-09-13T14:53:27.030 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:26 smithi151 ceph-mon[108966]: pgmap v1835: 119 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 2 remapped+peering, 112 active+clean; 582 MiB data, 4.5 GiB used, 443 GiB / 447 GiB avail; 11 KiB/s rd, 24 KiB/s wr, 775 op/s; 2511/27978 objects misplaced (8.975%); 12 MiB/s, 195 objects/s recovering 2024-09-13T14:53:27.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:27.054 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:27.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:27.155 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:27.176 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:27.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:27.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:27.278 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:27.303 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:27.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:27.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:27.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:26 smithi184 ceph-mon[92527]: pgmap v1835: 119 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 2 remapped+peering, 112 active+clean; 582 MiB data, 4.5 GiB used, 443 GiB / 447 GiB avail; 11 KiB/s rd, 24 KiB/s wr, 775 op/s; 2511/27978 objects misplaced (8.975%); 12 MiB/s, 195 objects/s recovering 2024-09-13T14:53:27.406 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:27.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:27.450 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:27.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:27.551 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:27.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:27.592 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:27.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:27.693 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:27.725 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:27.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:27.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:27.827 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:27.865 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:27.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:27.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:27.967 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:28.004 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:28.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:28.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:28.106 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:28.111 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:53:28.111 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:53:28.111 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:53:28.111 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:53:28.111 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:53:28.111 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:53:28.112 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:53:28.112 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:53:28.112 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:53:28.112 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:53:28.112 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:53:28.112 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:53:28.112 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:53:28.112 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:53:28.113 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:53:28.113 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:53:28.130 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:28.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:28.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:28.232 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:28.257 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:28.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:28.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:28.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:27 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/26209126' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:28.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:27 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/26209126' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:28.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:27 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/26209126' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:28.359 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:28.380 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:28.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:28.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:28.481 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:28.508 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:28.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:28.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:28.611 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:28.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:28.633 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:28.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:28.734 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:28.756 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:28.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:28.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:28.857 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:28.879 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:28.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:28.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:28.981 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:29.002 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:29.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:29.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:29.103 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:29.128 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:29.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:29.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:29.230 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:29.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:29.253 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:29.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:29.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:28 smithi184 ceph-mon[92527]: pgmap v1836: 119 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 1 remapped+peering, 112 active+clean; 549 MiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 9.2 KiB/s rd, 20 KiB/s wr, 416 op/s; 2511/26418 objects misplaced (9.505%); 7.1 MiB/s, 114 objects/s recovering 2024-09-13T14:53:29.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:28 smithi184 ceph-mon[92527]: osdmap e1210: 8 total, 8 up, 5 in 2024-09-13T14:53:29.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:28 smithi184 ceph-mon[92527]: 1.b scrub starts 2024-09-13T14:53:29.355 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:29.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:28 smithi151 ceph-mon[104450]: pgmap v1836: 119 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 1 remapped+peering, 112 active+clean; 549 MiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 9.2 KiB/s rd, 20 KiB/s wr, 416 op/s; 2511/26418 objects misplaced (9.505%); 7.1 MiB/s, 114 objects/s recovering 2024-09-13T14:53:29.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:29 smithi151 ceph-mon[104450]: osdmap e1210: 8 total, 8 up, 5 in 2024-09-13T14:53:29.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:29 smithi151 ceph-mon[104450]: 1.b scrub starts 2024-09-13T14:53:29.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:28 smithi151 ceph-mon[108966]: pgmap v1836: 119 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 1 remapped+peering, 112 active+clean; 549 MiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 9.2 KiB/s rd, 20 KiB/s wr, 416 op/s; 2511/26418 objects misplaced (9.505%); 7.1 MiB/s, 114 objects/s recovering 2024-09-13T14:53:29.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:29 smithi151 ceph-mon[108966]: osdmap e1210: 8 total, 8 up, 5 in 2024-09-13T14:53:29.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:29 smithi151 ceph-mon[108966]: 1.b scrub starts 2024-09-13T14:53:29.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:29.378 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:29.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:29.479 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:29.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:29.505 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:29.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:29.606 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:29.632 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:29.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:29.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:29.735 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:29.761 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:29.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:29.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:29.863 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:29.884 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:29.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:29.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:29.986 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:30.014 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:30.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:30.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:30.115 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:30.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:30.141 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:30.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:30.243 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:30.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:30.266 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:30.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:30.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:29 smithi184 ceph-mon[92527]: osdmap e1211: 8 total, 8 up, 5 in 2024-09-13T14:53:30.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:29 smithi184 ceph-mon[92527]: 1.16 scrub starts 2024-09-13T14:53:30.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:29 smithi184 ceph-mon[92527]: osdmap e1212: 8 total, 8 up, 5 in 2024-09-13T14:53:30.354 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:53:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:53:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:53:30.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:29 smithi151 ceph-mon[108966]: osdmap e1211: 8 total, 8 up, 5 in 2024-09-13T14:53:30.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:29 smithi151 ceph-mon[108966]: 1.16 scrub starts 2024-09-13T14:53:30.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:29 smithi151 ceph-mon[108966]: osdmap e1212: 8 total, 8 up, 5 in 2024-09-13T14:53:30.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:29 smithi151 ceph-mon[104450]: osdmap e1211: 8 total, 8 up, 5 in 2024-09-13T14:53:30.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:29 smithi151 ceph-mon[104450]: 1.16 scrub starts 2024-09-13T14:53:30.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:29 smithi151 ceph-mon[104450]: osdmap e1212: 8 total, 8 up, 5 in 2024-09-13T14:53:30.367 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:30.388 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:30.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:30.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:30.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:30.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:30.521 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:30.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:30.622 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:30.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:30.645 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:30.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:30.746 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:30.768 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:30.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:30.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:30.869 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:30.891 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:30.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:30.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:30.993 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:31.017 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:31.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:31.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:31.090 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:53:31.090 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-13T14:53:31.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-13T14:53:31.090 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-13T14:53:31.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.090 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:233760 8.3 8:cf463d1b:::benchmark_data_smithi151_229097_object108892:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1200)", 2024-09-13T14:53:31.090 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:11.843388+0000", 2024-09-13T14:53:31.090 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.104643979, 2024-09-13T14:53:31.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.022823689, 2024-09-13T14:53:31.090 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:53:31.091 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:53:31.091 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:31.091 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:53:31.091 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T14:53:31.091 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:53:31.091 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 233760 2024-09-13T14:53:31.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.091 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:53:31.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:53:31.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.843388+0000", 2024-09-13T14:53:31.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:53:31.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.843388+0000", 2024-09-13T14:53:31.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:53:31.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.843389+0000", 2024-09-13T14:53:31.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8599999999999997e-07 2024-09-13T14:53:31.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:53:31.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.843393+0000", 2024-09-13T14:53:31.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2869999999999998e-06 2024-09-13T14:53:31.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.093 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:53:31.093 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.843405+0000", 2024-09-13T14:53:31.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.115e-05 2024-09-13T14:53:31.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.093 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:53:31.093 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.843412+0000", 2024-09-13T14:53:31.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1110000000000002e-06 2024-09-13T14:53:31.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.843572+0000", 2024-09-13T14:53:31.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016003299999999999 2024-09-13T14:53:31.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:53:31.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.843579+0000", 2024-09-13T14:53:31.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7510000000000008e-06 2024-09-13T14:53:31.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.860470+0000", 2024-09-13T14:53:31.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0168901340000001 2024-09-13T14:53:31.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:53:31.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.860595+0000", 2024-09-13T14:53:31.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000125117 2024-09-13T14:53:31.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:53:31.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.860732+0000", 2024-09-13T14:53:31.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013740700000000001 2024-09-13T14:53:31.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:31.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.863073+0000", 2024-09-13T14:53:31.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002340899 2024-09-13T14:53:31.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:53:31.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.863088+0000", 2024-09-13T14:53:31.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4558e-05 2024-09-13T14:53:31.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:31.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.866115+0000", 2024-09-13T14:53:31.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030279349999999998 2024-09-13T14:53:31.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:53:31.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.866165+0000", 2024-09-13T14:53:31.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9929999999999998e-05 2024-09-13T14:53:31.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:53:31.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.866212+0000", 2024-09-13T14:53:31.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6490999999999999e-05 2024-09-13T14:53:31.097 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.098 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:53:31.098 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.098 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:233761 8.3 8:ceda9a69:::benchmark_data_smithi151_229097_object108893:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1200)", 2024-09-13T14:53:31.098 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:11.846412+0000", 2024-09-13T14:53:31.098 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.101619949, 2024-09-13T14:53:31.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0200265180000001, 2024-09-13T14:53:31.098 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:53:31.098 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:53:31.098 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:31.098 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:53:31.098 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T14:53:31.098 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:53:31.099 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 233761 2024-09-13T14:53:31.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.099 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:53:31.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:53:31.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.846412+0000", 2024-09-13T14:53:31.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:53:31.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.846412+0000", 2024-09-13T14:53:31.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:53:31.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.846414+0000", 2024-09-13T14:53:31.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3909999999999999e-06 2024-09-13T14:53:31.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:53:31.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.846419+0000", 2024-09-13T14:53:31.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3929999999999999e-06 2024-09-13T14:53:31.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:53:31.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.846430+0000", 2024-09-13T14:53:31.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1178999999999999e-05 2024-09-13T14:53:31.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:53:31.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.846436+0000", 2024-09-13T14:53:31.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3940000000000004e-06 2024-09-13T14:53:31.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.846493+0000", 2024-09-13T14:53:31.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7182999999999998e-05 2024-09-13T14:53:31.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:53:31.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.846502+0000", 2024-09-13T14:53:31.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7499999999999992e-06 2024-09-13T14:53:31.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.863127+0000", 2024-09-13T14:53:31.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0166255980000001 2024-09-13T14:53:31.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:53:31.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.863205+0000", 2024-09-13T14:53:31.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7682000000000006e-05 2024-09-13T14:53:31.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:53:31.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.863272+0000", 2024-09-13T14:53:31.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6857e-05 2024-09-13T14:53:31.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:31.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.866264+0000", 2024-09-13T14:53:31.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002991919 2024-09-13T14:53:31.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:53:31.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.866354+0000", 2024-09-13T14:53:31.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9981999999999993e-05 2024-09-13T14:53:31.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:31.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.866376+0000", 2024-09-13T14:53:31.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2708e-05 2024-09-13T14:53:31.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:53:31.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.866404+0000", 2024-09-13T14:53:31.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7871000000000001e-05 2024-09-13T14:53:31.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:53:31.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.866439+0000", 2024-09-13T14:53:31.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4610999999999998e-05 2024-09-13T14:53:31.105 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.105 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:53:31.105 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.105 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:233782 8.3 8:ce70c30b:::benchmark_data_smithi151_229097_object108914:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1200)", 2024-09-13T14:53:31.105 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:11.870138+0000", 2024-09-13T14:53:31.105 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.077893905, 2024-09-13T14:53:31.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.99643160399999997, 2024-09-13T14:53:31.105 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:53:31.105 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:53:31.105 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:31.105 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:53:31.105 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T14:53:31.106 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:53:31.106 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 233782 2024-09-13T14:53:31.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.106 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:53:31.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:53:31.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.870138+0000", 2024-09-13T14:53:31.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:53:31.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.870138+0000", 2024-09-13T14:53:31.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:53:31.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.870140+0000", 2024-09-13T14:53:31.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2780000000000001e-06 2024-09-13T14:53:31.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:53:31.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.870145+0000", 2024-09-13T14:53:31.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5790000000000003e-06 2024-09-13T14:53:31.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:53:31.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.870161+0000", 2024-09-13T14:53:31.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5667000000000001e-05 2024-09-13T14:53:31.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:53:31.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.870169+0000", 2024-09-13T14:53:31.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8560000000000007e-06 2024-09-13T14:53:31.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.870243+0000", 2024-09-13T14:53:31.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4345999999999996e-05 2024-09-13T14:53:31.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:53:31.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.870265+0000", 2024-09-13T14:53:31.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2073e-05 2024-09-13T14:53:31.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.864725+0000", 2024-09-13T14:53:31.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.99446032399999995 2024-09-13T14:53:31.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:53:31.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.864836+0000", 2024-09-13T14:53:31.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000111048 2024-09-13T14:53:31.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:53:31.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.864954+0000", 2024-09-13T14:53:31.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117349 2024-09-13T14:53:31.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:31.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.866482+0000", 2024-09-13T14:53:31.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015280809999999999 2024-09-13T14:53:31.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:53:31.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.866490+0000", 2024-09-13T14:53:31.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1769999999999998e-06 2024-09-13T14:53:31.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:31.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.866517+0000", 2024-09-13T14:53:31.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7381000000000001e-05 2024-09-13T14:53:31.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:53:31.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.866545+0000", 2024-09-13T14:53:31.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8e-05 2024-09-13T14:53:31.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:53:31.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.866570+0000", 2024-09-13T14:53:31.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4445000000000001e-05 2024-09-13T14:53:31.112 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.112 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:53:31.112 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.112 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:233799 8.3 8:c5d44874:::benchmark_data_smithi151_229097_object108931:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1200)", 2024-09-13T14:53:31.112 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:11.891427+0000", 2024-09-13T14:53:31.112 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.056605554000001, 2024-09-13T14:53:31.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.97882745000000004, 2024-09-13T14:53:31.112 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:53:31.112 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:53:31.113 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:31.113 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:53:31.113 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T14:53:31.113 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:53:31.113 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 233799 2024-09-13T14:53:31.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.113 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:53:31.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:53:31.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.891427+0000", 2024-09-13T14:53:31.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:53:31.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.891427+0000", 2024-09-13T14:53:31.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:53:31.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.891428+0000", 2024-09-13T14:53:31.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.046e-06 2024-09-13T14:53:31.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:53:31.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.891433+0000", 2024-09-13T14:53:31.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.395e-06 2024-09-13T14:53:31.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:53:31.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.891444+0000", 2024-09-13T14:53:31.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1215e-05 2024-09-13T14:53:31.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:53:31.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.891451+0000", 2024-09-13T14:53:31.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6969999999999996e-06 2024-09-13T14:53:31.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.891539+0000", 2024-09-13T14:53:31.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7714999999999996e-05 2024-09-13T14:53:31.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:53:31.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.891548+0000", 2024-09-13T14:53:31.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1929999999999999e-06 2024-09-13T14:53:31.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.866613+0000", 2024-09-13T14:53:31.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.97506510999999996 2024-09-13T14:53:31.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:53:31.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.866712+0000", 2024-09-13T14:53:31.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8746000000000006e-05 2024-09-13T14:53:31.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:53:31.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.866831+0000", 2024-09-13T14:53:31.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000119559 2024-09-13T14:53:31.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:31.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.870040+0000", 2024-09-13T14:53:31.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003208364 2024-09-13T14:53:31.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:31.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.870098+0000", 2024-09-13T14:53:31.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8482e-05 2024-09-13T14:53:31.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:53:31.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.870157+0000", 2024-09-13T14:53:31.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9302000000000002e-05 2024-09-13T14:53:31.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:53:31.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.870210+0000", 2024-09-13T14:53:31.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2374999999999999e-05 2024-09-13T14:53:31.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:53:31.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.870254+0000", 2024-09-13T14:53:31.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4251e-05 2024-09-13T14:53:31.119 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.119 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:53:31.119 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.119 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:233801 8.3 8:c6282ca0:::benchmark_data_smithi151_229097_object108933:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1200)", 2024-09-13T14:53:31.119 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:11.891480+0000", 2024-09-13T14:53:31.120 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.056551894, 2024-09-13T14:53:31.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.98152921800000004, 2024-09-13T14:53:31.120 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:53:31.120 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:53:31.120 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:31.120 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:53:31.120 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T14:53:31.120 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:53:31.120 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 233801 2024-09-13T14:53:31.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.120 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:53:31.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:53:31.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.891480+0000", 2024-09-13T14:53:31.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:53:31.121 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.891480+0000", 2024-09-13T14:53:31.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:53:31.121 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.891481+0000", 2024-09-13T14:53:31.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3399999999999999e-07 2024-09-13T14:53:31.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:53:31.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.891483+0000", 2024-09-13T14:53:31.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.667e-06 2024-09-13T14:53:31.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:53:31.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.891491+0000", 2024-09-13T14:53:31.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0480000000000001e-06 2024-09-13T14:53:31.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:53:31.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.891496+0000", 2024-09-13T14:53:31.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.463e-06 2024-09-13T14:53:31.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.891574+0000", 2024-09-13T14:53:31.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7535999999999998e-05 2024-09-13T14:53:31.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:53:31.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.891579+0000", 2024-09-13T14:53:31.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6860000000000003e-06 2024-09-13T14:53:31.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.868294+0000", 2024-09-13T14:53:31.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.97671498700000003 2024-09-13T14:53:31.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:53:31.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.868403+0000", 2024-09-13T14:53:31.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108663 2024-09-13T14:53:31.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:53:31.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.868521+0000", 2024-09-13T14:53:31.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011804800000000001 2024-09-13T14:53:31.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:31.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.870145+0000", 2024-09-13T14:53:31.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016239399999999999 2024-09-13T14:53:31.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:31.125 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.870311+0000", 2024-09-13T14:53:31.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000165668 2024-09-13T14:53:31.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:53:31.125 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.872917+0000", 2024-09-13T14:53:31.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002606311 2024-09-13T14:53:31.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:53:31.125 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.872966+0000", 2024-09-13T14:53:31.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9450000000000003e-05 2024-09-13T14:53:31.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:53:31.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.873010+0000", 2024-09-13T14:53:31.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3217000000000003e-05 2024-09-13T14:53:31.126 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.126 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:53:31.126 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.126 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:233812 8.3 8:ccf1ae68:::benchmark_data_smithi151_229097_object108944:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1200)", 2024-09-13T14:53:31.126 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:11.900068+0000", 2024-09-13T14:53:31.127 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.04796398, 2024-09-13T14:53:31.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.97891498700000001, 2024-09-13T14:53:31.127 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:53:31.127 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:53:31.127 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:31.127 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:53:31.127 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T14:53:31.127 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:53:31.127 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 233812 2024-09-13T14:53:31.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.127 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:53:31.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:53:31.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.900068+0000", 2024-09-13T14:53:31.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:53:31.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.900068+0000", 2024-09-13T14:53:31.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:53:31.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.900070+0000", 2024-09-13T14:53:31.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7799999999999999e-06 2024-09-13T14:53:31.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:53:31.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.900076+0000", 2024-09-13T14:53:31.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9560000000000002e-06 2024-09-13T14:53:31.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:53:31.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.900095+0000", 2024-09-13T14:53:31.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8694000000000001e-05 2024-09-13T14:53:31.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:53:31.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.900103+0000", 2024-09-13T14:53:31.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0369999999999995e-06 2024-09-13T14:53:31.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.900177+0000", 2024-09-13T14:53:31.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4072999999999994e-05 2024-09-13T14:53:31.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:53:31.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.900188+0000", 2024-09-13T14:53:31.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1727e-05 2024-09-13T14:53:31.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.870356+0000", 2024-09-13T14:53:31.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.97016793899999998 2024-09-13T14:53:31.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:53:31.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.870467+0000", 2024-09-13T14:53:31.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000110862 2024-09-13T14:53:31.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:53:31.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.870591+0000", 2024-09-13T14:53:31.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012386799999999999 2024-09-13T14:53:31.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:31.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.876004+0000", 2024-09-13T14:53:31.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0054126069999999998 2024-09-13T14:53:31.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:31.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.876083+0000", 2024-09-13T14:53:31.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9511999999999994e-05 2024-09-13T14:53:31.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:53:31.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.878893+0000", 2024-09-13T14:53:31.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028092519999999999 2024-09-13T14:53:31.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:53:31.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.878942+0000", 2024-09-13T14:53:31.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9410999999999998e-05 2024-09-13T14:53:31.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:53:31.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.878983+0000", 2024-09-13T14:53:31.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1269000000000001e-05 2024-09-13T14:53:31.133 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.134 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:53:31.134 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.134 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:233861 8.3 8:c888d867:::benchmark_data_smithi151_229097_object108993:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1200)", 2024-09-13T14:53:31.134 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:11.969681+0000", 2024-09-13T14:53:31.134 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.978351376999999, 2024-09-13T14:53:31.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.91056891699999998, 2024-09-13T14:53:31.134 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:53:31.134 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:53:31.134 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:31.134 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:53:31.134 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T14:53:31.134 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:53:31.135 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 233861 2024-09-13T14:53:31.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.135 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:53:31.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:53:31.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.969681+0000", 2024-09-13T14:53:31.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:53:31.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.969681+0000", 2024-09-13T14:53:31.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:53:31.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.969682+0000", 2024-09-13T14:53:31.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3600000000000003e-07 2024-09-13T14:53:31.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:53:31.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.969686+0000", 2024-09-13T14:53:31.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3930000000000001e-06 2024-09-13T14:53:31.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:53:31.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.969694+0000", 2024-09-13T14:53:31.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9619999999999993e-06 2024-09-13T14:53:31.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:53:31.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.969700+0000", 2024-09-13T14:53:31.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5819999999999998e-06 2024-09-13T14:53:31.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.969770+0000", 2024-09-13T14:53:31.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9992000000000003e-05 2024-09-13T14:53:31.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:53:31.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.969797+0000", 2024-09-13T14:53:31.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7002999999999999e-05 2024-09-13T14:53:31.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.873048+0000", 2024-09-13T14:53:31.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.90325082800000001 2024-09-13T14:53:31.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:53:31.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.873148+0000", 2024-09-13T14:53:31.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9809999999999995e-05 2024-09-13T14:53:31.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:53:31.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.873257+0000", 2024-09-13T14:53:31.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109076 2024-09-13T14:53:31.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:31.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.876052+0000", 2024-09-13T14:53:31.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027945280000000001 2024-09-13T14:53:31.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:53:31.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.878990+0000", 2024-09-13T14:53:31.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0029379440000000001 2024-09-13T14:53:31.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:31.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.880096+0000", 2024-09-13T14:53:31.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001106599 2024-09-13T14:53:31.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:53:31.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.880209+0000", 2024-09-13T14:53:31.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000113189 2024-09-13T14:53:31.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:53:31.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:12.880250+0000", 2024-09-13T14:53:31.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0275000000000003e-05 2024-09-13T14:53:31.140 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.141 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:53:31.141 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.141 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:238805 8.a 8.a72421a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1206)", 2024-09-13T14:53:31.141 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:20.014865+0000", 2024-09-13T14:53:31.141 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 6.9331674740000002, 2024-09-13T14:53:31.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.91208313900000004, 2024-09-13T14:53:31.141 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:53:31.141 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:53:31.141 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-13T14:53:31.141 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:53:31.141 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T14:53:31.141 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:53:31.142 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 238805 2024-09-13T14:53:31.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.142 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:53:31.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:53:31.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.014865+0000", 2024-09-13T14:53:31.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:53:31.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.014865+0000", 2024-09-13T14:53:31.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:53:31.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.014866+0000", 2024-09-13T14:53:31.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2810000000000001e-06 2024-09-13T14:53:31.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:53:31.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.014870+0000", 2024-09-13T14:53:31.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1180000000000002e-06 2024-09-13T14:53:31.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:53:31.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.014883+0000", 2024-09-13T14:53:31.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2889e-05 2024-09-13T14:53:31.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:53:31.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.014890+0000", 2024-09-13T14:53:31.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0809999999999996e-06 2024-09-13T14:53:31.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.014941+0000", 2024-09-13T14:53:31.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0541999999999998e-05 2024-09-13T14:53:31.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:53:31.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.014948+0000", 2024-09-13T14:53:31.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6320000000000008e-06 2024-09-13T14:53:31.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.926938+0000", 2024-09-13T14:53:31.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.91198999000000003 2024-09-13T14:53:31.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:53:31.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.926948+0000", 2024-09-13T14:53:31.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6059999999999992e-06 2024-09-13T14:53:31.145 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.145 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:53:31.145 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.146 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:238844 8.a 8.847533da (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1206)", 2024-09-13T14:53:31.146 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:20.043551+0000", 2024-09-13T14:53:31.146 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 6.9044808069999997, 2024-09-13T14:53:31.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.88342225699999999, 2024-09-13T14:53:31.146 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:53:31.146 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:53:31.146 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-13T14:53:31.146 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:53:31.146 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T14:53:31.146 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:53:31.146 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 238844 2024-09-13T14:53:31.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.146 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:53:31.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:53:31.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.043551+0000", 2024-09-13T14:53:31.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:53:31.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.043551+0000", 2024-09-13T14:53:31.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:53:31.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.043552+0000", 2024-09-13T14:53:31.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5499999999999998e-07 2024-09-13T14:53:31.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:53:31.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.043554+0000", 2024-09-13T14:53:31.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1799999999999999e-06 2024-09-13T14:53:31.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:53:31.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.043560+0000", 2024-09-13T14:53:31.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7599999999999999e-06 2024-09-13T14:53:31.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:53:31.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.043565+0000", 2024-09-13T14:53:31.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3199999999999999e-06 2024-09-13T14:53:31.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.043615+0000", 2024-09-13T14:53:31.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0033e-05 2024-09-13T14:53:31.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:53:31.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.043623+0000", 2024-09-13T14:53:31.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2530000000000004e-06 2024-09-13T14:53:31.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.926967+0000", 2024-09-13T14:53:31.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.88334385100000001 2024-09-13T14:53:31.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:53:31.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.926974+0000", 2024-09-13T14:53:31.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4049999999999997e-06 2024-09-13T14:53:31.150 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.151 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:53:31.151 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.151 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:238916 8.a 8.13accb8a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1206)", 2024-09-13T14:53:31.151 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:20.108942+0000", 2024-09-13T14:53:31.151 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 6.839089864, 2024-09-13T14:53:31.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.81806223199999994, 2024-09-13T14:53:31.151 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:53:31.151 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:53:31.151 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-13T14:53:31.151 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:53:31.151 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T14:53:31.151 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:53:31.152 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 238916 2024-09-13T14:53:31.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.152 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:53:31.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:53:31.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.108942+0000", 2024-09-13T14:53:31.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:53:31.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.108942+0000", 2024-09-13T14:53:31.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:53:31.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.108943+0000", 2024-09-13T14:53:31.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7100000000000011e-07 2024-09-13T14:53:31.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:53:31.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.108948+0000", 2024-09-13T14:53:31.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8899999999999998e-06 2024-09-13T14:53:31.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:53:31.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.108960+0000", 2024-09-13T14:53:31.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1969e-05 2024-09-13T14:53:31.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:53:31.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.108968+0000", 2024-09-13T14:53:31.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6009999999999997e-06 2024-09-13T14:53:31.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.109022+0000", 2024-09-13T14:53:31.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4669000000000001e-05 2024-09-13T14:53:31.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:53:31.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.109031+0000", 2024-09-13T14:53:31.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8109999999999992e-06 2024-09-13T14:53:31.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.926998+0000", 2024-09-13T14:53:31.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.817966631 2024-09-13T14:53:31.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:53:31.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.927005+0000", 2024-09-13T14:53:31.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6900000000000003e-06 2024-09-13T14:53:31.155 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.155 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:53:31.155 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.156 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:238931 8.a 8.1d164f1a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1206)", 2024-09-13T14:53:31.156 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:20.122172+0000", 2024-09-13T14:53:31.156 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 6.8258605550000002, 2024-09-13T14:53:31.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.80485703399999997, 2024-09-13T14:53:31.156 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:53:31.156 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:53:31.156 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-13T14:53:31.156 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:53:31.156 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T14:53:31.156 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:53:31.156 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 238931 2024-09-13T14:53:31.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.157 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:53:31.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:53:31.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.122172+0000", 2024-09-13T14:53:31.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:53:31.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.122172+0000", 2024-09-13T14:53:31.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:53:31.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.122173+0000", 2024-09-13T14:53:31.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4169999999999999e-06 2024-09-13T14:53:31.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:53:31.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.122178+0000", 2024-09-13T14:53:31.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7849999999999999e-06 2024-09-13T14:53:31.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:53:31.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.122192+0000", 2024-09-13T14:53:31.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4354e-05 2024-09-13T14:53:31.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:53:31.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.122200+0000", 2024-09-13T14:53:31.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6969999999999993e-06 2024-09-13T14:53:31.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.122262+0000", 2024-09-13T14:53:31.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1890000000000005e-05 2024-09-13T14:53:31.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:53:31.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.122271+0000", 2024-09-13T14:53:31.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0999999999999993e-06 2024-09-13T14:53:31.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.927022+0000", 2024-09-13T14:53:31.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.80475141400000005 2024-09-13T14:53:31.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:53:31.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.927029+0000", 2024-09-13T14:53:31.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3770000000000001e-06 2024-09-13T14:53:31.160 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.160 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:53:31.160 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.160 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:238958 8.a 8.f25c630a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1206)", 2024-09-13T14:53:31.161 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:20.145272+0000", 2024-09-13T14:53:31.161 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 6.8027599680000002, 2024-09-13T14:53:31.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.78178113699999996, 2024-09-13T14:53:31.161 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:53:31.161 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:53:31.161 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-13T14:53:31.161 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:53:31.161 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T14:53:31.161 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:53:31.161 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 238958 2024-09-13T14:53:31.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.161 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:53:31.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:53:31.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.145272+0000", 2024-09-13T14:53:31.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:53:31.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.145272+0000", 2024-09-13T14:53:31.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:53:31.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.145273+0000", 2024-09-13T14:53:31.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.017e-06 2024-09-13T14:53:31.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:53:31.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.145276+0000", 2024-09-13T14:53:31.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2270000000000002e-06 2024-09-13T14:53:31.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:53:31.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.145284+0000", 2024-09-13T14:53:31.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9889999999999992e-06 2024-09-13T14:53:31.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:53:31.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.145290+0000", 2024-09-13T14:53:31.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5539999999999997e-06 2024-09-13T14:53:31.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.145355+0000", 2024-09-13T14:53:31.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4883999999999997e-05 2024-09-13T14:53:31.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:53:31.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.145363+0000", 2024-09-13T14:53:31.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3839999999999997e-06 2024-09-13T14:53:31.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.927046+0000", 2024-09-13T14:53:31.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.78168269199999996 2024-09-13T14:53:31.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:53:31.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.927053+0000", 2024-09-13T14:53:31.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3900000000000004e-06 2024-09-13T14:53:31.165 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.165 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:53:31.165 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.165 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:239020 8.a 8.2620200a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1206)", 2024-09-13T14:53:31.165 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:20.208165+0000", 2024-09-13T14:53:31.165 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 6.7398672020000001, 2024-09-13T14:53:31.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.71891390399999999, 2024-09-13T14:53:31.166 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:53:31.166 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:53:31.166 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-13T14:53:31.166 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:53:31.166 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T14:53:31.166 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:53:31.166 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 239020 2024-09-13T14:53:31.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.166 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:53:31.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:53:31.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.208165+0000", 2024-09-13T14:53:31.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:53:31.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.208165+0000", 2024-09-13T14:53:31.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:53:31.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.208167+0000", 2024-09-13T14:53:31.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.849e-06 2024-09-13T14:53:31.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:53:31.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.208174+0000", 2024-09-13T14:53:31.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6889999999999998e-06 2024-09-13T14:53:31.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:53:31.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.208187+0000", 2024-09-13T14:53:31.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3565e-05 2024-09-13T14:53:31.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:53:31.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.208196+0000", 2024-09-13T14:53:31.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4600000000000003e-06 2024-09-13T14:53:31.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.208265+0000", 2024-09-13T14:53:31.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9045e-05 2024-09-13T14:53:31.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:53:31.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.208274+0000", 2024-09-13T14:53:31.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3959999999999995e-06 2024-09-13T14:53:31.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.927072+0000", 2024-09-13T14:53:31.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.71879802599999998 2024-09-13T14:53:31.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:53:31.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.927079+0000", 2024-09-13T14:53:31.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8739999999999997e-06 2024-09-13T14:53:31.170 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.170 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:53:31.170 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.170 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:239022 8.a 8.52ede99a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1206)", 2024-09-13T14:53:31.171 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:20.208331+0000", 2024-09-13T14:53:31.171 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 6.7397008740000004, 2024-09-13T14:53:31.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.71877201999999996, 2024-09-13T14:53:31.171 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:53:31.171 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:53:31.171 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-13T14:53:31.171 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:53:31.171 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T14:53:31.171 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:53:31.171 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 239022 2024-09-13T14:53:31.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.171 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:53:31.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:53:31.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.208331+0000", 2024-09-13T14:53:31.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:53:31.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.208331+0000", 2024-09-13T14:53:31.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:53:31.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.208332+0000", 2024-09-13T14:53:31.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5600000000000004e-07 2024-09-13T14:53:31.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:53:31.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.208337+0000", 2024-09-13T14:53:31.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.617e-06 2024-09-13T14:53:31.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:53:31.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.208347+0000", 2024-09-13T14:53:31.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7310000000000005e-06 2024-09-13T14:53:31.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:53:31.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.208353+0000", 2024-09-13T14:53:31.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9319999999999999e-06 2024-09-13T14:53:31.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.209135+0000", 2024-09-13T14:53:31.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00078141199999999999 2024-09-13T14:53:31.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:53:31.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.209143+0000", 2024-09-13T14:53:31.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4409999999999993e-06 2024-09-13T14:53:31.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.927097+0000", 2024-09-13T14:53:31.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.71795370599999997 2024-09-13T14:53:31.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:53:31.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.927103+0000", 2024-09-13T14:53:31.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3249999999999996e-06 2024-09-13T14:53:31.175 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.175 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:53:31.175 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.176 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:239024 8.a 8.1aa52fca (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1206)", 2024-09-13T14:53:31.176 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:20.212121+0000", 2024-09-13T14:53:31.176 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 6.7359108409999999, 2024-09-13T14:53:31.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.71500491300000002, 2024-09-13T14:53:31.176 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:53:31.176 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:53:31.176 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-13T14:53:31.176 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:53:31.176 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T14:53:31.176 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:53:31.176 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 239024 2024-09-13T14:53:31.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.177 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:53:31.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:53:31.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.212121+0000", 2024-09-13T14:53:31.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:53:31.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.212121+0000", 2024-09-13T14:53:31.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:53:31.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.212123+0000", 2024-09-13T14:53:31.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.372e-06 2024-09-13T14:53:31.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:53:31.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.212129+0000", 2024-09-13T14:53:31.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1120000000000001e-06 2024-09-13T14:53:31.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:53:31.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.212145+0000", 2024-09-13T14:53:31.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5970000000000001e-05 2024-09-13T14:53:31.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:53:31.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.212153+0000", 2024-09-13T14:53:31.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8059999999999995e-06 2024-09-13T14:53:31.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.212245+0000", 2024-09-13T14:53:31.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2724000000000002e-05 2024-09-13T14:53:31.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:53:31.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.212255+0000", 2024-09-13T14:53:31.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2e-06 2024-09-13T14:53:31.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.927120+0000", 2024-09-13T14:53:31.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.71486553500000005 2024-09-13T14:53:31.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:53:31.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.927126+0000", 2024-09-13T14:53:31.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1940000000000003e-06 2024-09-13T14:53:31.180 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.180 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:53:31.180 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.181 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:239045 8.a 8.d27f3aa (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1206)", 2024-09-13T14:53:31.181 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:20.232577+0000", 2024-09-13T14:53:31.181 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 6.7154549000000001, 2024-09-13T14:53:31.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.69457231699999999, 2024-09-13T14:53:31.181 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:53:31.181 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:53:31.181 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-13T14:53:31.181 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:53:31.181 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T14:53:31.181 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:53:31.181 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 239045 2024-09-13T14:53:31.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.182 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:53:31.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:53:31.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.232577+0000", 2024-09-13T14:53:31.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:53:31.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.232577+0000", 2024-09-13T14:53:31.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:53:31.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.232579+0000", 2024-09-13T14:53:31.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.508e-06 2024-09-13T14:53:31.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:53:31.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.232583+0000", 2024-09-13T14:53:31.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6090000000000003e-06 2024-09-13T14:53:31.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:53:31.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.232596+0000", 2024-09-13T14:53:31.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2860000000000001e-05 2024-09-13T14:53:31.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.184 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:53:31.184 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.232604+0000", 2024-09-13T14:53:31.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5989999999999996e-06 2024-09-13T14:53:31.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.184 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.184 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.232660+0000", 2024-09-13T14:53:31.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6452000000000001e-05 2024-09-13T14:53:31.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:53:31.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.232669+0000", 2024-09-13T14:53:31.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6850000000000007e-06 2024-09-13T14:53:31.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.927143+0000", 2024-09-13T14:53:31.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.69447447399999995 2024-09-13T14:53:31.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:53:31.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.927150+0000", 2024-09-13T14:53:31.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1299999999999998e-06 2024-09-13T14:53:31.185 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.186 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:53:31.186 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.186 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:239060 8.a 8.bd7699ea (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1206)", 2024-09-13T14:53:31.186 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:20.247912+0000", 2024-09-13T14:53:31.186 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 6.7001202939999995, 2024-09-13T14:53:31.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.67926183500000004, 2024-09-13T14:53:31.186 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:53:31.186 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:53:31.186 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-13T14:53:31.186 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:53:31.186 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T14:53:31.186 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:53:31.187 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 239060 2024-09-13T14:53:31.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.187 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:53:31.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:53:31.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.247912+0000", 2024-09-13T14:53:31.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:53:31.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.247912+0000", 2024-09-13T14:53:31.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:53:31.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.247913+0000", 2024-09-13T14:53:31.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.42e-06 2024-09-13T14:53:31.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:53:31.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.247919+0000", 2024-09-13T14:53:31.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7939999999999999e-06 2024-09-13T14:53:31.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:53:31.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.247933+0000", 2024-09-13T14:53:31.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4379000000000001e-05 2024-09-13T14:53:31.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:53:31.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.247942+0000", 2024-09-13T14:53:31.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2770000000000005e-06 2024-09-13T14:53:31.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.248155+0000", 2024-09-13T14:53:31.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00021320000000000001 2024-09-13T14:53:31.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:53:31.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.248161+0000", 2024-09-13T14:53:31.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5489999999999997e-06 2024-09-13T14:53:31.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.927168+0000", 2024-09-13T14:53:31.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.67900714100000004 2024-09-13T14:53:31.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:53:31.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.927174+0000", 2024-09-13T14:53:31.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0750000000000003e-06 2024-09-13T14:53:31.190 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.190 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:53:31.190 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.191 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:239086 8.a 8.ad92d68a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1206)", 2024-09-13T14:53:31.191 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:20.277405+0000", 2024-09-13T14:53:31.191 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 6.6706270510000003, 2024-09-13T14:53:31.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.64979209599999999, 2024-09-13T14:53:31.191 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:53:31.191 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:53:31.191 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-13T14:53:31.191 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:53:31.191 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T14:53:31.191 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:53:31.191 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 239086 2024-09-13T14:53:31.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.191 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:53:31.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:53:31.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.277405+0000", 2024-09-13T14:53:31.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:53:31.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.277405+0000", 2024-09-13T14:53:31.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:53:31.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.277407+0000", 2024-09-13T14:53:31.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8679999999999999e-06 2024-09-13T14:53:31.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:53:31.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.277414+0000", 2024-09-13T14:53:31.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2289999999999997e-06 2024-09-13T14:53:31.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:53:31.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.277436+0000", 2024-09-13T14:53:31.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2269e-05 2024-09-13T14:53:31.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:53:31.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.277446+0000", 2024-09-13T14:53:31.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.268e-06 2024-09-13T14:53:31.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.277517+0000", 2024-09-13T14:53:31.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1555000000000002e-05 2024-09-13T14:53:31.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:53:31.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.277528+0000", 2024-09-13T14:53:31.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1045e-05 2024-09-13T14:53:31.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.927191+0000", 2024-09-13T14:53:31.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.649662513 2024-09-13T14:53:31.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:53:31.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.927197+0000", 2024-09-13T14:53:31.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3489999999999997e-06 2024-09-13T14:53:31.195 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.195 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:53:31.195 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.195 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:239099 8.a 8.80cae5aa (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1206)", 2024-09-13T14:53:31.195 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:20.289651+0000", 2024-09-13T14:53:31.196 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 6.6583813660000004, 2024-09-13T14:53:31.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.63757031799999997, 2024-09-13T14:53:31.196 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:53:31.196 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:53:31.196 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-13T14:53:31.196 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:53:31.196 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T14:53:31.196 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:53:31.196 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 239099 2024-09-13T14:53:31.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.196 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:53:31.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.196 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:53:31.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.289651+0000", 2024-09-13T14:53:31.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.197 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.197 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:53:31.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.289651+0000", 2024-09-13T14:53:31.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.197 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.197 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:53:31.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.289652+0000", 2024-09-13T14:53:31.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4500000000000001e-06 2024-09-13T14:53:31.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.197 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.198 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:53:31.198 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.289657+0000", 2024-09-13T14:53:31.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1649999999999997e-06 2024-09-13T14:53:31.198 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.198 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:53:31.198 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.289670+0000", 2024-09-13T14:53:31.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2211999999999999e-05 2024-09-13T14:53:31.198 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.198 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:53:31.198 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.289677+0000", 2024-09-13T14:53:31.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3359999999999997e-06 2024-09-13T14:53:31.198 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.199 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.289732+0000", 2024-09-13T14:53:31.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5481e-05 2024-09-13T14:53:31.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.199 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:53:31.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.289742+0000", 2024-09-13T14:53:31.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2480000000000002e-06 2024-09-13T14:53:31.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.199 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.927214+0000", 2024-09-13T14:53:31.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.63747262000000005 2024-09-13T14:53:31.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:53:31.200 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.927221+0000", 2024-09-13T14:53:31.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8059999999999997e-06 2024-09-13T14:53:31.200 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.200 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:53:31.200 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.200 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:239125 8.a 8.2d5158ea (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1206)", 2024-09-13T14:53:31.200 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:20.321010+0000", 2024-09-13T14:53:31.200 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 6.6270222749999999, 2024-09-13T14:53:31.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.60623415800000002, 2024-09-13T14:53:31.201 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:53:31.201 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:53:31.201 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-13T14:53:31.201 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:53:31.201 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T14:53:31.201 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:53:31.201 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 239125 2024-09-13T14:53:31.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.201 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:53:31.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:53:31.202 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.321010+0000", 2024-09-13T14:53:31.202 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.202 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.202 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:53:31.202 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.321010+0000", 2024-09-13T14:53:31.202 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:53:31.202 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.202 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:53:31.202 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.321011+0000", 2024-09-13T14:53:31.202 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1349999999999999e-06 2024-09-13T14:53:31.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.203 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:53:31.203 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.321016+0000", 2024-09-13T14:53:31.203 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.51e-06 2024-09-13T14:53:31.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.203 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:53:31.203 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.321028+0000", 2024-09-13T14:53:31.203 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2917e-05 2024-09-13T14:53:31.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.203 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:53:31.203 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.321035+0000", 2024-09-13T14:53:31.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.173e-06 2024-09-13T14:53:31.204 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.204 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.204 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.204 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.321075+0000", 2024-09-13T14:53:31.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0778000000000002e-05 2024-09-13T14:53:31.204 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.204 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.204 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:53:31.204 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.321081+0000", 2024-09-13T14:53:31.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8119999999999996e-06 2024-09-13T14:53:31.204 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.204 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.205 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:53:31.205 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.927238+0000", 2024-09-13T14:53:31.205 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.60615646400000001 2024-09-13T14:53:31.205 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:53:31.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:53:31.205 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:53:31.205 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:20.927244+0000", 2024-09-13T14:53:31.205 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3690000000000003e-06 2024-09-13T14:53:31.205 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.205 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:53:31.205 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.209 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:53:31.209 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:53:31.209 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:53:31.210 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:31.231 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:31.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:31.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:31.332 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:31.353 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:31.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:30 smithi184 ceph-mon[92527]: pgmap v1839: 119 pgs: 2 active+clean+remapped, 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 112 active+clean; 500 MiB data, 4.2 GiB used, 443 GiB / 447 GiB avail; 4.2 KiB/s rd, 9.2 KiB/s wr, 439 op/s; 2511/24075 objects misplaced (10.430%); 11 MiB/s, 178 objects/s recovering 2024-09-13T14:53:31.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:30 smithi184 ceph-mon[92527]: osdmap e1213: 8 total, 8 up, 5 in 2024-09-13T14:53:31.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:31.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:31.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:30 smithi151 ceph-mon[104450]: pgmap v1839: 119 pgs: 2 active+clean+remapped, 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 112 active+clean; 500 MiB data, 4.2 GiB used, 443 GiB / 447 GiB avail; 4.2 KiB/s rd, 9.2 KiB/s wr, 439 op/s; 2511/24075 objects misplaced (10.430%); 11 MiB/s, 178 objects/s recovering 2024-09-13T14:53:31.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:30 smithi151 ceph-mon[104450]: osdmap e1213: 8 total, 8 up, 5 in 2024-09-13T14:53:31.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:30 smithi151 ceph-mon[108966]: pgmap v1839: 119 pgs: 2 active+clean+remapped, 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 112 active+clean; 500 MiB data, 4.2 GiB used, 443 GiB / 447 GiB avail; 4.2 KiB/s rd, 9.2 KiB/s wr, 439 op/s; 2511/24075 objects misplaced (10.430%); 11 MiB/s, 178 objects/s recovering 2024-09-13T14:53:31.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:30 smithi151 ceph-mon[108966]: osdmap e1213: 8 total, 8 up, 5 in 2024-09-13T14:53:31.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:31.478 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:31.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:31.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:31.580 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:31.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:31.603 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:31.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:31.703 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:31.725 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:31.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:31.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:31.827 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:31.852 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:31.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:31.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:31.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:31.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:31.985 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:31.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:32.086 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:32.130 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:32.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:32.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:32.232 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:32.267 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:32.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:32.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:32.369 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:32.393 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:32.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:32.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:32.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:32.503 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T14:53:32.531 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:32.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:32.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:32.633 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:32.674 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:32.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:32.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:32.776 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:32.799 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:32.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:32.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:53:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:32 smithi184 ceph-mon[92527]: pgmap v1842: 119 pgs: 1 active+remapped+backfilling, 1 peering, 2 active+clean+remapped, 115 active+clean; 377 MiB data, 4.1 GiB used, 443 GiB / 447 GiB avail; 0 B/s wr, 690 op/s; 149/18132 objects misplaced (0.822%); 34 MiB/s, 550 objects/s recovering 2024-09-13T14:53:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:32 smithi184 ceph-mon[92527]: osdmap e1214: 8 total, 8 up, 5 in 2024-09-13T14:53:32.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:53:32.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:32.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:32 smithi151 ceph-mon[108966]: pgmap v1842: 119 pgs: 1 active+remapped+backfilling, 1 peering, 2 active+clean+remapped, 115 active+clean; 377 MiB data, 4.1 GiB used, 443 GiB / 447 GiB avail; 0 B/s wr, 690 op/s; 149/18132 objects misplaced (0.822%); 34 MiB/s, 550 objects/s recovering 2024-09-13T14:53:32.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:32 smithi151 ceph-mon[108966]: osdmap e1214: 8 total, 8 up, 5 in 2024-09-13T14:53:32.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:53:32.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:32.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:32 smithi151 ceph-mon[104450]: pgmap v1842: 119 pgs: 1 active+remapped+backfilling, 1 peering, 2 active+clean+remapped, 115 active+clean; 377 MiB data, 4.1 GiB used, 443 GiB / 447 GiB avail; 0 B/s wr, 690 op/s; 149/18132 objects misplaced (0.822%); 34 MiB/s, 550 objects/s recovering 2024-09-13T14:53:32.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:32 smithi151 ceph-mon[104450]: osdmap e1214: 8 total, 8 up, 5 in 2024-09-13T14:53:32.901 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:32.928 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:32.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:32.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:33.030 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:33.067 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:33.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:33.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:33.168 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:33.202 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:33.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:33.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:33.304 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:33.329 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:33.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:33.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:33.430 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:33.474 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:33.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:33.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:33.576 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:33.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:33.609 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:33.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:33.710 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:33.746 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:33.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:33.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:33.848 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:33.869 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:33.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:33.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:33.970 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:33.997 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:33.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:33.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:34.099 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:34.128 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:34.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:34.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:34.230 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:34.254 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:34.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:34.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:34.356 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:34.392 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:34.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:34.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:34.494 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:34.518 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:34.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:34.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:34.620 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:34.644 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:34.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:34.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:34.692 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:53:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:53:34] "GET /metrics HTTP/1.1" 200 36675 "" "Prometheus/2.51.0" 2024-09-13T14:53:34.746 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:34.770 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:34.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:34.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:34.871 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:34.893 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:34.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:34.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:34.995 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:35.020 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:35.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:35.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:35.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:34 smithi184 ceph-mon[92527]: 1.b scrub starts 2024-09-13T14:53:35.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:34 smithi184 ceph-mon[92527]: pgmap v1844: 119 pgs: 1 active+remapped+backfilling, 1 peering, 117 active+clean; 337 MiB data, 4.0 GiB used, 443 GiB / 447 GiB avail; 0 B/s wr, 559 op/s; 149/16209 objects misplaced (0.919%); 22 MiB/s, 355 objects/s recovering 2024-09-13T14:53:35.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:34 smithi151 ceph-mon[104450]: 1.b scrub starts 2024-09-13T14:53:35.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:34 smithi151 ceph-mon[104450]: pgmap v1844: 119 pgs: 1 active+remapped+backfilling, 1 peering, 117 active+clean; 337 MiB data, 4.0 GiB used, 443 GiB / 447 GiB avail; 0 B/s wr, 559 op/s; 149/16209 objects misplaced (0.919%); 22 MiB/s, 355 objects/s recovering 2024-09-13T14:53:35.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:34 smithi151 ceph-mon[108966]: 1.b scrub starts 2024-09-13T14:53:35.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:34 smithi151 ceph-mon[108966]: pgmap v1844: 119 pgs: 1 active+remapped+backfilling, 1 peering, 117 active+clean; 337 MiB data, 4.0 GiB used, 443 GiB / 447 GiB avail; 0 B/s wr, 559 op/s; 149/16209 objects misplaced (0.919%); 22 MiB/s, 355 objects/s recovering 2024-09-13T14:53:35.123 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:35.148 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:35.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:35.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:35.251 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:35.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:35.291 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:35.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:35.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:35.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:35.424 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:35.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:35.525 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:35.546 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:35.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:35.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:35.648 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:35.674 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:35.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:35.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:35.719 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-13T14:53:35.776 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:35.810 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:35.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:35.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:35.912 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:35.934 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:35.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:35.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:36.036 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:36.070 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:36.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:36.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:36.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:35 smithi184 ceph-mon[92527]: 1.16 scrub starts 2024-09-13T14:53:36.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:35 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1417436473' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:53:36.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:35 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:53:36.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:35 smithi151 ceph-mon[104450]: 1.16 scrub starts 2024-09-13T14:53:36.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:35 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1417436473' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:53:36.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:35 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:53:36.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:35 smithi151 ceph-mon[108966]: 1.16 scrub starts 2024-09-13T14:53:36.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:35 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1417436473' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:53:36.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:35 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:53:36.171 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:36.194 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:36.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:36.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:36.296 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:36.320 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:36.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:36.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:36.422 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:36.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:36.462 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:36.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:36.563 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:36.586 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:36.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:36.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:36.654 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Removed 124865 objects 2024-09-13T14:53:36.654 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Clean up completed and total clean up time :147.415 2024-09-13T14:53:36.688 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:36.709 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:36.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:36.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:36.811 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:36.833 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:36.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:36.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:36.936 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:36.957 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:36.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:36.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:37.059 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:37.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:37.081 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:37.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:37.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:36 smithi184 ceph-mon[92527]: pgmap v1845: 119 pgs: 2 active+clean+scrubbing, 117 active+clean; 158 MiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 0 B/s wr, 912 op/s; 19 MiB/s, 301 objects/s recovering 2024-09-13T14:53:37.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:36 smithi184 ceph-mon[92527]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:53:37.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:36 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:53:37.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:36 smithi184 ceph-mon[92527]: osdmap e1215: 8 total, 8 up, 5 in 2024-09-13T14:53:37.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:36 smithi151 ceph-mon[104450]: pgmap v1845: 119 pgs: 2 active+clean+scrubbing, 117 active+clean; 158 MiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 0 B/s wr, 912 op/s; 19 MiB/s, 301 objects/s recovering 2024-09-13T14:53:37.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:36 smithi151 ceph-mon[104450]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:53:37.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:36 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:53:37.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:36 smithi151 ceph-mon[104450]: osdmap e1215: 8 total, 8 up, 5 in 2024-09-13T14:53:37.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:36 smithi151 ceph-mon[108966]: pgmap v1845: 119 pgs: 2 active+clean+scrubbing, 117 active+clean; 158 MiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 0 B/s wr, 912 op/s; 19 MiB/s, 301 objects/s recovering 2024-09-13T14:53:37.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:36 smithi151 ceph-mon[108966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:53:37.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:36 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:53:37.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:36 smithi151 ceph-mon[108966]: osdmap e1215: 8 total, 8 up, 5 in 2024-09-13T14:53:37.182 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:37.204 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:37.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:37.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:37.211 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-09-13T14:53:37.211 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-09-13T14:53:37.212 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:37.217 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:37.305 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:37.327 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:37.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:37.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:37.429 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:37.450 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:37.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:37.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:37.552 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:37.574 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:37.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:37.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:37.676 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:37.698 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:37.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:37.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:37.800 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:37.821 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:37.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:37.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:37.923 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:37.945 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:37.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:37.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:38.047 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:38.068 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:38.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:38.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:38.170 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:38.190 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:38.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:38.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:38.292 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:38.313 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:38.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:38.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:38.415 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:38.437 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:38.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:38.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:38.539 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:38.562 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:38.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:38.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:38.609 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:38.664 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:38.672 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:53:38.686 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:38.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:38.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:38.788 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:38.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:38.811 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:38.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:38.912 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:38.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:38.935 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:38.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:39.008 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:38 smithi151 ceph-mon[104450]: pgmap v1847: 119 pgs: 2 active+clean+scrubbing, 117 active+clean; 122 MiB data, 3.5 GiB used, 443 GiB / 447 GiB avail; 0 B/s wr, 606 op/s; 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:53:39.008 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:38 smithi151 ceph-mon[108966]: pgmap v1847: 119 pgs: 2 active+clean+scrubbing, 117 active+clean; 122 MiB data, 3.5 GiB used, 443 GiB / 447 GiB avail; 0 B/s wr, 606 op/s; 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:53:39.036 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:39.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:39.063 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:39.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:39.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:38 smithi184 ceph-mon[92527]: pgmap v1847: 119 pgs: 2 active+clean+scrubbing, 117 active+clean; 122 MiB data, 3.5 GiB used, 443 GiB / 447 GiB avail; 0 B/s wr, 606 op/s; 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:53:39.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:39.186 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:39.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:39.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:39.200 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:53:39.288 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:39.313 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:39.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:39.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:39.415 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:39.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:39.442 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:39.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:39.543 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:39.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:39.565 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:39.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:39.666 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:39.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:39.694 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:39.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:39.795 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:39.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:39.819 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:39.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:39.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:39 smithi151 ceph-mon[104450]: 1.b scrub starts 2024-09-13T14:53:39.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:39 smithi151 ceph-mon[108966]: 1.b scrub starts 2024-09-13T14:53:39.921 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:39.943 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:39.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:39.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:40.045 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:40.066 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:40.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:40.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:40.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:39 smithi184 ceph-mon[92527]: 1.b scrub starts 2024-09-13T14:53:40.168 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:40.190 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:40.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:40.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:40.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:40.314 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:40.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:40.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:40.416 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:40.437 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:40.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:40.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:40.539 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:40.561 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:40.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:40.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:40.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:53:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:53:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:53:40.635 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:53:40.636 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:53:40.663 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:40.684 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:40.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:40.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:40.786 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:40.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:40.809 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:40.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:40.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:40 smithi151 ceph-mon[104450]: 1.16 scrub starts 2024-09-13T14:53:40.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:40 smithi151 ceph-mon[104450]: pgmap v1848: 119 pgs: 2 active+clean+scrubbing, 117 active+clean; 69 MiB data, 3.4 GiB used, 444 GiB / 447 GiB avail; 0 B/s wr, 615 op/s; 1.2 MiB/s, 18 objects/s recovering 2024-09-13T14:53:40.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:40 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:53:40.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:40 smithi151 ceph-mon[108966]: 1.16 scrub starts 2024-09-13T14:53:40.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:40 smithi151 ceph-mon[108966]: pgmap v1848: 119 pgs: 2 active+clean+scrubbing, 117 active+clean; 69 MiB data, 3.4 GiB used, 444 GiB / 447 GiB avail; 0 B/s wr, 615 op/s; 1.2 MiB/s, 18 objects/s recovering 2024-09-13T14:53:40.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:40 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:53:40.910 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:40.932 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:40.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:40.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:41.034 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:41.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:41.056 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:41.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:41.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:40 smithi184 ceph-mon[92527]: 1.16 scrub starts 2024-09-13T14:53:41.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:40 smithi184 ceph-mon[92527]: pgmap v1848: 119 pgs: 2 active+clean+scrubbing, 117 active+clean; 69 MiB data, 3.4 GiB used, 444 GiB / 447 GiB avail; 0 B/s wr, 615 op/s; 1.2 MiB/s, 18 objects/s recovering 2024-09-13T14:53:41.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:40 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:53:41.157 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:41.179 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:41.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:41.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:41.240 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T14:53:41.281 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:41.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:41.303 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:41.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:41.404 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:41.425 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:41.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:41.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:41.527 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:41.549 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:41.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:41.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:41.651 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:41.672 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:41.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:41.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:41.774 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:41.798 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:41.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:41.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:41.900 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:41.921 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:41.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:41.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:42.023 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:42.044 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:42.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:42.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:42.146 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:42.167 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:42.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:42.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:42.269 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:42.289 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:42.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:42.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:42.331 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:42.343 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:42.392 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:42.414 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:42.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:42.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:42.516 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:42.537 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:42.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:42.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:42.639 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:42.660 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:42.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:42.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:42.762 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:42.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:42.785 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:42.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:42.886 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:42.908 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:42.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:42.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:42.924 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:42 smithi151 ceph-mon[104450]: pgmap v1849: 119 pgs: 2 active+clean+scrubbing, 117 active+clean; 630 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail; 0 B/s wr, 619 op/s; 975 KiB/s, 15 objects/s recovering 2024-09-13T14:53:42.924 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:42 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:53:42.924 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:42 smithi151 ceph-mon[108966]: pgmap v1849: 119 pgs: 2 active+clean+scrubbing, 117 active+clean; 630 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail; 0 B/s wr, 619 op/s; 975 KiB/s, 15 objects/s recovering 2024-09-13T14:53:42.925 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:42 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:53:43.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:43.032 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:43.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:43.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:43.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:42 smithi184 ceph-mon[92527]: pgmap v1849: 119 pgs: 2 active+clean+scrubbing, 117 active+clean; 630 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail; 0 B/s wr, 619 op/s; 975 KiB/s, 15 objects/s recovering 2024-09-13T14:53:43.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:42 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:53:43.133 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:43.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:43.156 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:43.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:43.257 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:43.279 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:43.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:43.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:43.381 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:43.404 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:43.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:43.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:43.506 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:43.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:43.529 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:43.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:43.630 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:43.654 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:43.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:43.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:43.756 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:43.778 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:43.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:43.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:43.880 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:43.919 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:43.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:43.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:44.021 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:44.043 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:44.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:44.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:44.144 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:44.165 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:44.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:44.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:44.267 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:44.289 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:44.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:44.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:44.390 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:44.414 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:44.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:44.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:44.515 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:44.536 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:44.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:44.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:44.637 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:44.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:44.659 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:44.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:44.758 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:53:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:53:44] "GET /metrics HTTP/1.1" 200 36675 "" "Prometheus/2.51.0" 2024-09-13T14:53:44.761 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:44.783 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:44.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:44.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:44.885 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:44.907 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:44.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:44.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:45.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:45.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:45.032 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:45.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:45.091 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:44 smithi151 ceph-mon[104450]: 1.b scrub starts 2024-09-13T14:53:45.091 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:44 smithi151 ceph-mon[104450]: pgmap v1850: 119 pgs: 2 active+clean+scrubbing, 117 active+clean; 630 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail; 0 B/s wr, 537 op/s; 947 KiB/s, 14 objects/s recovering 2024-09-13T14:53:45.091 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:44 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:53:45.091 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:44 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:53:45.092 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:44 smithi151 ceph-mon[108966]: 1.b scrub starts 2024-09-13T14:53:45.092 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:44 smithi151 ceph-mon[108966]: pgmap v1850: 119 pgs: 2 active+clean+scrubbing, 117 active+clean; 630 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail; 0 B/s wr, 537 op/s; 947 KiB/s, 14 objects/s recovering 2024-09-13T14:53:45.092 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:44 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:53:45.092 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:44 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:53:45.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:44 smithi184 ceph-mon[92527]: 1.b scrub starts 2024-09-13T14:53:45.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:44 smithi184 ceph-mon[92527]: pgmap v1850: 119 pgs: 2 active+clean+scrubbing, 117 active+clean; 630 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail; 0 B/s wr, 537 op/s; 947 KiB/s, 14 objects/s recovering 2024-09-13T14:53:45.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:44 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:53:45.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:44 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:53:45.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:45.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:45.157 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:45.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:45.258 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:45.284 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:45.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:45.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:45.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:45 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a[104427]: 2024-09-13T14:53:45.143+0000 7fe9da14f640 -1 mon.a@0(leader).osd e1215 definitely_dead 0 2024-09-13T14:53:45.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:45.407 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:45.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:45.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:45.509 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:45.535 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:45.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:45.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:45.637 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:45.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:45.659 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:45.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:45.760 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:45.765 INFO:teuthology.orchestra.run.smithi151.stderr:marked in osd.5. 2024-09-13T14:53:45.771 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_3' removed 2024-09-13T14:53:45.795 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:45.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:45.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:45.897 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:45.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:45.919 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:45.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:46.020 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:46.041 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:46.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:46.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:46.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:45 smithi184 ceph-mon[92527]: 1.16 scrub starts 2024-09-13T14:53:46.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:45 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3647212409' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:53:46.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:45 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:53:46.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:45 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3002671999' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-13T14:53:46.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:45 smithi151 ceph-mon[104450]: 1.16 scrub starts 2024-09-13T14:53:46.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:45 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3647212409' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:53:46.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:45 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:53:46.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:45 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3002671999' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-13T14:53:46.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:45 smithi151 ceph-mon[108966]: 1.16 scrub starts 2024-09-13T14:53:46.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:45 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3647212409' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:53:46.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:45 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:53:46.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:45 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3002671999' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-13T14:53:46.143 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:46.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:46.165 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:46.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:46.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:46.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:46.289 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:46.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:46.390 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:46.412 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:46.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:46.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:46.514 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:46.537 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:46.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:46.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:46.639 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:46.662 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:46.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:46.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:46.764 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:46.793 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:46.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:46.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:46.842 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T14:53:46.843 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T14:53:46.843 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-09-13T14:53:46.843 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_4 16 2024-09-13T14:53:46.895 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:46.916 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:46.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:46.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:47.018 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:47.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:47.044 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:47.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:46 smithi184 ceph-mon[92527]: pgmap v1851: 119 pgs: 2 active+clean+scrubbing, 117 active+clean; 630 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail; 0 B/s wr, 252 op/s 2024-09-13T14:53:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:46 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:53:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:46 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3002671999' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-13T14:53:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:46 smithi184 ceph-mon[92527]: osdmap e1216: 8 total, 8 up, 6 in 2024-09-13T14:53:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:46 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/4289709316' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:53:47.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:46 smithi151 ceph-mon[104450]: pgmap v1851: 119 pgs: 2 active+clean+scrubbing, 117 active+clean; 630 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail; 0 B/s wr, 252 op/s 2024-09-13T14:53:47.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:46 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:53:47.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:46 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3002671999' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-13T14:53:47.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:46 smithi151 ceph-mon[104450]: osdmap e1216: 8 total, 8 up, 6 in 2024-09-13T14:53:47.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:47.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:46 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/4289709316' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:53:47.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:46 smithi151 ceph-mon[108966]: pgmap v1851: 119 pgs: 2 active+clean+scrubbing, 117 active+clean; 630 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail; 0 B/s wr, 252 op/s 2024-09-13T14:53:47.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:46 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:53:47.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:46 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3002671999' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-13T14:53:47.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:46 smithi151 ceph-mon[108966]: osdmap e1216: 8 total, 8 up, 6 in 2024-09-13T14:53:47.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:47.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:46 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/4289709316' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:53:47.128 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-13T14:53:47.145 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:47.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:47.167 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:47.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:47.268 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:47.290 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:47.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:47.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:47.325 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:47.392 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:47.418 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:47.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:47.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:47.520 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:47.545 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:47.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:47.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:47.647 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:47.669 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:47.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:47.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:47.770 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:47.796 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:47.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:47.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:47.898 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:47.922 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:47.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:47.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:48.002 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:47 smithi151 ceph-mon[104450]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:53:48.002 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:47 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/4289709316' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:53:48.002 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:47 smithi151 ceph-mon[104450]: osdmap e1217: 8 total, 8 up, 6 in 2024-09-13T14:53:48.002 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:47 smithi151 ceph-mon[108966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:53:48.003 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:47 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/4289709316' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:53:48.003 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:47 smithi151 ceph-mon[108966]: osdmap e1217: 8 total, 8 up, 6 in 2024-09-13T14:53:48.024 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:48.047 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:48.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:48.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:48.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:47 smithi184 ceph-mon[92527]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:53:48.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:47 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/4289709316' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:53:48.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:47 smithi184 ceph-mon[92527]: osdmap e1217: 8 total, 8 up, 6 in 2024-09-13T14:53:48.150 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:48.172 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:48.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:48.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:48.274 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:48.296 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:48.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:48.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:48.400 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:48.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:48.426 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:48.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:48.527 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:48.549 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:48.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:48.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:48.651 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:48.673 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:48.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:48.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:48.714 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-09-13T14:53:48.776 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:48.797 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:48.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:48.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:48.898 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:48.909 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:53:48.909 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:53:48.915 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:53:48.915 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:53:48.920 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:48.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:48.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:48.931 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:53:48.931 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:53:48.948 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:53:48.948 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:53:48.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:48 smithi151 ceph-mon[104450]: 1.b scrub starts 2024-09-13T14:53:48.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:48 smithi151 ceph-mon[104450]: 1.b scrub ok 2024-09-13T14:53:48.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:48 smithi151 ceph-mon[104450]: pgmap v1854: 103 pgs: 2 active+clean+scrubbing, 101 active+clean; 630 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail 2024-09-13T14:53:48.949 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:48 smithi151 ceph-mon[108966]: 1.b scrub starts 2024-09-13T14:53:48.949 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:48 smithi151 ceph-mon[108966]: 1.b scrub ok 2024-09-13T14:53:48.949 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:48 smithi151 ceph-mon[108966]: pgmap v1854: 103 pgs: 2 active+clean+scrubbing, 101 active+clean; 630 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail 2024-09-13T14:53:48.963 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:53:48.963 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:53:48.978 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:53:48.978 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:53:48.993 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:53:48.994 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:53:49.009 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:53:49.012 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:53:49.021 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:49.043 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:49.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:49.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:49.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:48 smithi184 ceph-mon[92527]: 1.b scrub starts 2024-09-13T14:53:49.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:48 smithi184 ceph-mon[92527]: 1.b scrub ok 2024-09-13T14:53:49.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:48 smithi184 ceph-mon[92527]: pgmap v1854: 103 pgs: 2 active+clean+scrubbing, 101 active+clean; 630 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail 2024-09-13T14:53:49.145 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:49.167 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:49.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:49.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:49.269 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:49.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:49.292 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:49.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:49.393 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:49.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:49.415 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:49.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:49.516 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:49.539 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:49.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:49.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:49.641 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:49.664 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:49.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:49.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:49.723 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:53:49.766 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:49.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:49.790 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:49.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:49.891 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:49.915 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:49.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:49.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:49.921 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:49 smithi151 ceph-mon[104450]: 1.16 scrub starts 2024-09-13T14:53:49.922 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:49 smithi151 ceph-mon[104450]: 1.16 scrub ok 2024-09-13T14:53:49.922 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:49 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1771337879' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:49.922 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:49 smithi151 ceph-mon[104450]: 1.a scrub starts 2024-09-13T14:53:49.922 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:49 smithi151 ceph-mon[104450]: 1.a scrub ok 2024-09-13T14:53:49.922 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:49 smithi151 ceph-mon[108966]: 1.16 scrub starts 2024-09-13T14:53:49.922 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:49 smithi151 ceph-mon[108966]: 1.16 scrub ok 2024-09-13T14:53:49.922 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:49 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1771337879' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:49.922 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:49 smithi151 ceph-mon[108966]: 1.a scrub starts 2024-09-13T14:53:49.922 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:49 smithi151 ceph-mon[108966]: 1.a scrub ok 2024-09-13T14:53:50.017 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:50.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:50.040 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:50.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:50.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:49 smithi184 ceph-mon[92527]: 1.16 scrub starts 2024-09-13T14:53:50.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:49 smithi184 ceph-mon[92527]: 1.16 scrub ok 2024-09-13T14:53:50.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:49 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1771337879' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:50.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:49 smithi184 ceph-mon[92527]: 1.a scrub starts 2024-09-13T14:53:50.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:49 smithi184 ceph-mon[92527]: 1.a scrub ok 2024-09-13T14:53:50.141 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:50.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:50.163 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:50.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:50.264 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:50.286 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:50.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:50.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:50.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:50.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:50.410 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:50.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:50.511 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:50.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:50.533 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:50.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:50.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:53:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:53:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:53:50.634 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:50.658 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:50.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:50.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:50.709 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:50.728 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:50.759 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:50.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:50.781 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:50.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:50.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:50.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:50.905 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:50.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:51.006 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:51.030 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:51.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:51.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:51.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:50 smithi151 ceph-mon[104450]: pgmap v1855: 103 pgs: 2 active+clean+scrubbing, 101 active+clean; 656 KiB data, 3.9 GiB used, 533 GiB / 536 GiB avail; 4.1 KiB/s rd, 1 op/s 2024-09-13T14:53:51.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:50 smithi151 ceph-mon[104450]: 1.10 scrub starts 2024-09-13T14:53:51.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:50 smithi151 ceph-mon[104450]: 1.10 scrub ok 2024-09-13T14:53:51.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:50 smithi151 ceph-mon[108966]: pgmap v1855: 103 pgs: 2 active+clean+scrubbing, 101 active+clean; 656 KiB data, 3.9 GiB used, 533 GiB / 536 GiB avail; 4.1 KiB/s rd, 1 op/s 2024-09-13T14:53:51.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:50 smithi151 ceph-mon[108966]: 1.10 scrub starts 2024-09-13T14:53:51.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:50 smithi151 ceph-mon[108966]: 1.10 scrub ok 2024-09-13T14:53:51.132 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:51.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:51.156 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:51.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:51.257 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:51.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:51.279 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:51.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:51.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:50 smithi184 ceph-mon[92527]: pgmap v1855: 103 pgs: 2 active+clean+scrubbing, 101 active+clean; 656 KiB data, 3.9 GiB used, 533 GiB / 536 GiB avail; 4.1 KiB/s rd, 1 op/s 2024-09-13T14:53:51.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:50 smithi184 ceph-mon[92527]: 1.10 scrub starts 2024-09-13T14:53:51.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:50 smithi184 ceph-mon[92527]: 1.10 scrub ok 2024-09-13T14:53:51.380 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:51.405 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:51.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:51.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:51.507 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:51.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:51.530 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:51.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:51.631 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:51.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:51.653 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:51.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:51.675 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:53:51.754 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:51.779 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:51.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:51.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:51.884 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:51.907 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:51.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:51.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:52.009 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:52.034 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:52.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:52.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:52.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:52.159 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:52.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:52.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:52.261 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:52.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:52.288 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:52.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:52.389 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:52.411 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:52.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:52.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:52.513 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:52.534 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:52.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:52.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:52.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:52.659 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:52.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:52.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:52.761 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:52.782 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:52.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:52.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:52.884 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:52.905 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:52.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:52.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:53.007 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:53.034 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:53.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:53.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:53.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:53.158 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:53.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:53.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:53.162 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:53:53.162 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-13T14:53:53.163 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-13T14:53:53.163 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-13T14:53:53.163 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:53:53.259 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:53.286 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:53.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:53.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:53.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:52 smithi184 ceph-mon[92527]: pgmap v1856: 103 pgs: 103 active+clean; 634 KiB data, 3.9 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T14:53:53.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:52 smithi151 ceph-mon[104450]: pgmap v1856: 103 pgs: 103 active+clean; 634 KiB data, 3.9 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T14:53:53.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:52 smithi151 ceph-mon[108966]: pgmap v1856: 103 pgs: 103 active+clean; 634 KiB data, 3.9 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T14:53:53.388 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:53.413 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:53.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:53.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:53.515 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:53.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:53.538 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:53.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:53.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:53.660 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:53.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:53.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:53.715 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 6, 7, 5] out_osds: [4, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:53:53.715 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:53:53.716 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-09-13T14:53:53.716 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd in 1 2024-09-13T14:53:53.762 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:53.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:53.785 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:53.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:53.886 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:53.911 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:53.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:53.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:54.012 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:54.036 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:54.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:54.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:54.140 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:54.162 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:54.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:54.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:54.264 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:54.285 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:54.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:54.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:54.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:54.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:54.409 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:54.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:54.510 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:54.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:54.537 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:54.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:54.638 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:54.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:54.661 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:54.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:54.762 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:54.784 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:54.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:54.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:54.817 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:53:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:53:54] "GET /metrics HTTP/1.1" 200 34299 "" "Prometheus/2.51.0" 2024-09-13T14:53:54.850 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T14:53:54.856 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:54.857 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:54.887 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:54.921 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:54.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:54.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:55.023 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:55.044 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:55.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:55.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:55.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:54 smithi151 ceph-mon[104450]: pgmap v1857: 103 pgs: 103 active+clean; 608 KiB data, 3.8 GiB used, 533 GiB / 536 GiB avail 2024-09-13T14:53:55.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:54 smithi151 ceph-mon[108966]: pgmap v1857: 103 pgs: 103 active+clean; 608 KiB data, 3.8 GiB used, 533 GiB / 536 GiB avail 2024-09-13T14:53:55.146 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:55.171 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:55.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:55.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:55.273 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:55.295 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:55.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:55.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:55.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:54 smithi184 ceph-mon[92527]: pgmap v1857: 103 pgs: 103 active+clean; 608 KiB data, 3.8 GiB used, 533 GiB / 536 GiB avail 2024-09-13T14:53:55.396 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:55.419 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:55.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:55.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:55.521 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:55.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:55.549 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:55.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:55.650 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:55.675 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:55.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:55.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:55.777 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:55.802 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:55.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:55.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:55.905 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:55.930 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:55.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:55.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:55.953 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_4' created 2024-09-13T14:53:56.033 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:56.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:56.056 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:56.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:56.157 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:56.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:56.182 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:56.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:56.283 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:56.309 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:56.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:56.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:56.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:55 smithi184 ceph-mon[92527]: 1.12 scrub starts 2024-09-13T14:53:56.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:55 smithi184 ceph-mon[92527]: 1.12 scrub ok 2024-09-13T14:53:56.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:55 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1091275153' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-13T14:53:56.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:55 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-13T14:53:56.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:55 smithi151 ceph-mon[108966]: 1.12 scrub starts 2024-09-13T14:53:56.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:55 smithi151 ceph-mon[108966]: 1.12 scrub ok 2024-09-13T14:53:56.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:55 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1091275153' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-13T14:53:56.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:55 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-13T14:53:56.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:55 smithi151 ceph-mon[104450]: 1.12 scrub starts 2024-09-13T14:53:56.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:55 smithi151 ceph-mon[104450]: 1.12 scrub ok 2024-09-13T14:53:56.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:55 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1091275153' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-13T14:53:56.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:55 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-13T14:53:56.411 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:56.437 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:56.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:56.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:56.538 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:56.563 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:56.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:56.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:56.665 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:56.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:56.688 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:56.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:56.789 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:56.818 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:56.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:56.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:56.920 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:56.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:56.942 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:56.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:57.043 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:57.064 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:57.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:57.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:57.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:56 smithi151 ceph-mon[104450]: pgmap v1858: 103 pgs: 103 active+clean; 608 KiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-13T14:53:57.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:56 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-13T14:53:57.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:56 smithi151 ceph-mon[104450]: osdmap e1218: 8 total, 8 up, 6 in 2024-09-13T14:53:57.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:56 smithi151 ceph-mon[108966]: pgmap v1858: 103 pgs: 103 active+clean; 608 KiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-13T14:53:57.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:56 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-13T14:53:57.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:56 smithi151 ceph-mon[108966]: osdmap e1218: 8 total, 8 up, 6 in 2024-09-13T14:53:57.167 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:57.193 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:57.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:57.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:57.295 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:57.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:57.322 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:57.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:57.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:56 smithi184 ceph-mon[92527]: pgmap v1858: 103 pgs: 103 active+clean; 608 KiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-13T14:53:57.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:56 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-13T14:53:57.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:56 smithi184 ceph-mon[92527]: osdmap e1218: 8 total, 8 up, 6 in 2024-09-13T14:53:57.423 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:57.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:57.446 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:57.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:57.547 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:57.569 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:57.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:57.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:57.671 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:57.696 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:57.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:57.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:57.797 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:57.824 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:57.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:57.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:57.926 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:57.951 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:57.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:57.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:58.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:58.075 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:58.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:58.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:58.177 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:58.203 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:58.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:58.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:58.305 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:58.329 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:58.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:58.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:58.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:57 smithi184 ceph-mon[92527]: osdmap e1219: 8 total, 8 up, 6 in 2024-09-13T14:53:58.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:57 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1669663828' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:53:58.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:57 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:53:58.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:57 smithi151 ceph-mon[104450]: osdmap e1219: 8 total, 8 up, 6 in 2024-09-13T14:53:58.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:57 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1669663828' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:53:58.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:57 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:53:58.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:57 smithi151 ceph-mon[108966]: osdmap e1219: 8 total, 8 up, 6 in 2024-09-13T14:53:58.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:57 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1669663828' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:53:58.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:57 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:53:58.392 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-13T14:53:58.432 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:58.458 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:58.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:58.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:58.559 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:58.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:58.585 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:58.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:58.686 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:58.713 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:58.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:58.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:58.815 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:58.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:58.842 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:58.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:58.943 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:58.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:58.971 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:58.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:59.073 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:59.099 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:59.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:59.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:59.201 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:59.223 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:59.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:59.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:59.229 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:58 smithi151 ceph-mon[104450]: pgmap v1861: 119 pgs: 15 unknown, 104 active+clean; 608 KiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-13T14:53:59.230 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:58 smithi151 ceph-mon[104450]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:53:59.230 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:58 smithi151 ceph-mon[104450]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:53:59.230 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:58 smithi151 ceph-mon[104450]: Cluster is now healthy 2024-09-13T14:53:59.230 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:58 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:53:59.230 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:53:58 smithi151 ceph-mon[104450]: osdmap e1220: 8 total, 8 up, 6 in 2024-09-13T14:53:59.230 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:58 smithi151 ceph-mon[108966]: pgmap v1861: 119 pgs: 15 unknown, 104 active+clean; 608 KiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-13T14:53:59.230 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:58 smithi151 ceph-mon[108966]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:53:59.230 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:58 smithi151 ceph-mon[108966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:53:59.230 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:58 smithi151 ceph-mon[108966]: Cluster is now healthy 2024-09-13T14:53:59.230 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:58 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:53:59.230 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:53:58 smithi151 ceph-mon[108966]: osdmap e1220: 8 total, 8 up, 6 in 2024-09-13T14:53:59.324 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:59.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:59.346 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:59.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:59.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:58 smithi184 ceph-mon[92527]: pgmap v1861: 119 pgs: 15 unknown, 104 active+clean; 608 KiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-13T14:53:59.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:58 smithi184 ceph-mon[92527]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:53:59.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:58 smithi184 ceph-mon[92527]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:53:59.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:58 smithi184 ceph-mon[92527]: Cluster is now healthy 2024-09-13T14:53:59.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:58 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:53:59.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:53:58 smithi184 ceph-mon[92527]: osdmap e1220: 8 total, 8 up, 6 in 2024-09-13T14:53:59.447 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:59.471 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:59.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:59.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:59.573 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:59.596 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:53:59.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:59.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:59.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-09-13T14:53:59.698 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:59.720 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:59.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:59.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:59.822 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:59.844 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:59.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:59.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:59.946 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:59.968 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:53:59.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:59.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:00.070 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:00.094 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:00.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:00.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:00.196 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:00.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:00.221 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:00.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:00.322 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:00.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:00.346 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:00.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:00.447 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:00.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:00.473 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:00.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:00.575 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:00.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:00.598 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:00.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:00.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:54:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:54:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:54:00.656 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:00.699 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:00.725 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:00.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:00.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:00.827 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:00.849 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:00.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:00.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:00.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:00.973 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:00.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:00.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:01.075 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:01.100 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:01.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:01.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:01.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:00 smithi151 ceph-mon[104450]: pgmap v1863: 119 pgs: 8 unknown, 111 active+clean; 582 KiB data, 3.8 GiB used, 533 GiB / 536 GiB avail 2024-09-13T14:54:01.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:00 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2882156009' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:01.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:00 smithi151 ceph-mon[108966]: pgmap v1863: 119 pgs: 8 unknown, 111 active+clean; 582 KiB data, 3.8 GiB used, 533 GiB / 536 GiB avail 2024-09-13T14:54:01.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:00 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2882156009' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:01.201 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:01.227 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:01.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:01.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:01.329 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:01.353 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:01.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:01.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:01.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:00 smithi184 ceph-mon[92527]: pgmap v1863: 119 pgs: 8 unknown, 111 active+clean; 582 KiB data, 3.8 GiB used, 533 GiB / 536 GiB avail 2024-09-13T14:54:01.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:00 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2882156009' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:01.455 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:01.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:01.481 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:01.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:01.582 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:01.604 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:01.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:01.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:01.706 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:01.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:01.729 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:01.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:01.831 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:01.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:01.854 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:01.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:01.892 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:54:01.893 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:54:01.893 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:54:01.893 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:54:01.893 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:54:01.893 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:54:01.894 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:54:01.894 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:54:01.894 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:54:01.894 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:54:01.894 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:54:01.894 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:54:01.894 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:54:01.894 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:54:01.894 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:54:01.895 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:54:01.955 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:01.981 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:01.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:01.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:02.083 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:02.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:02.105 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:02.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:02.207 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:02.228 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:02.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:02.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:02.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:54:02.330 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:02.355 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:02.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:02.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:02.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:02.478 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:02.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:02.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:02.581 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:02.603 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:02.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:02.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:02.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:54:02.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:02.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:02 smithi151 ceph-mon[104450]: pgmap v1864: 119 pgs: 119 active+clean; 582 KiB data, 3.8 GiB used, 533 GiB / 536 GiB avail 2024-09-13T14:54:02.699 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:54:02.699 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:02.699 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:02 smithi151 ceph-mon[108966]: pgmap v1864: 119 pgs: 119 active+clean; 582 KiB data, 3.8 GiB used, 533 GiB / 536 GiB avail 2024-09-13T14:54:02.705 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:02.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:02.729 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:02.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:02.830 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:02.851 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:02.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:02.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:02.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:54:02.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:02.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:02 smithi184 ceph-mon[92527]: pgmap v1864: 119 pgs: 119 active+clean; 582 KiB data, 3.8 GiB used, 533 GiB / 536 GiB avail 2024-09-13T14:54:02.953 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:02.979 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:02.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:02.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:03.081 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:03.103 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:03.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:03.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:03.205 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:03.231 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:03.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:03.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:03.333 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:03.359 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:03.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:03.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:03.461 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:03.487 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:03.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:03.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:03.588 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:03.610 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:03.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:03.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:03.712 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:03.738 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:03.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:03.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:03.826 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:03 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [5, 6]} : dispatch 2024-09-13T14:54:03.826 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:03 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.15", "id": [7, 2]} : dispatch 2024-09-13T14:54:03.826 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:03 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [0, 6]} : dispatch 2024-09-13T14:54:03.827 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:03 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [5, 6]} : dispatch 2024-09-13T14:54:03.827 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:03 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.15", "id": [7, 2]} : dispatch 2024-09-13T14:54:03.827 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:03 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [0, 6]} : dispatch 2024-09-13T14:54:03.841 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:03.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:03 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [5, 6]} : dispatch 2024-09-13T14:54:03.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:03 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.15", "id": [7, 2]} : dispatch 2024-09-13T14:54:03.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:03 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [0, 6]} : dispatch 2024-09-13T14:54:03.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:03.867 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:03.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:03.969 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:03.994 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:03.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:03.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:04.096 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:04.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a[104427]: 2024-09-13T14:54:04.012+0000 7fe9da14f640 -1 mon.a@0(leader).osd e1221 definitely_dead 0 2024-09-13T14:54:04.120 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:04.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:04.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:04.222 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:04.244 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:04.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:04.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:04.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:04.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:04.368 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:04.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:04.469 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:04.495 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:04.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:04.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:04.511 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:54:04.579 INFO:teuthology.orchestra.run.smithi151.stderr:marked in osd.1. 2024-09-13T14:54:04.596 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:04.621 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:04.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:04.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:04.722 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:04.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:04.747 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:04.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:04.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:04 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [5, 6]}]': finished 2024-09-13T14:54:04.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:04 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.15", "id": [7, 2]}]': finished 2024-09-13T14:54:04.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:04 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [0, 6]}]': finished 2024-09-13T14:54:04.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:04 smithi151 ceph-mon[104450]: osdmap e1221: 8 total, 8 up, 6 in 2024-09-13T14:54:04.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:04 smithi151 ceph-mon[104450]: pgmap v1866: 119 pgs: 119 active+clean; 582 KiB data, 3.8 GiB used, 533 GiB / 536 GiB avail 2024-09-13T14:54:04.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:04 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1874318537' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-13T14:54:04.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:04 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-13T14:54:04.805 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:54:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:54:04] "GET /metrics HTTP/1.1" 200 36574 "" "Prometheus/2.51.0" 2024-09-13T14:54:04.805 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:04 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [5, 6]}]': finished 2024-09-13T14:54:04.805 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:04 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.15", "id": [7, 2]}]': finished 2024-09-13T14:54:04.806 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:04 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [0, 6]}]': finished 2024-09-13T14:54:04.806 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:04 smithi151 ceph-mon[108966]: osdmap e1221: 8 total, 8 up, 6 in 2024-09-13T14:54:04.806 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:04 smithi151 ceph-mon[108966]: pgmap v1866: 119 pgs: 119 active+clean; 582 KiB data, 3.8 GiB used, 533 GiB / 536 GiB avail 2024-09-13T14:54:04.806 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:04 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1874318537' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-13T14:54:04.806 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:04 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-13T14:54:04.847 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:04.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:04 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [5, 6]}]': finished 2024-09-13T14:54:04.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:04 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.15", "id": [7, 2]}]': finished 2024-09-13T14:54:04.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:04 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [0, 6]}]': finished 2024-09-13T14:54:04.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:04 smithi184 ceph-mon[92527]: osdmap e1221: 8 total, 8 up, 6 in 2024-09-13T14:54:04.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:04 smithi184 ceph-mon[92527]: pgmap v1866: 119 pgs: 119 active+clean; 582 KiB data, 3.8 GiB used, 533 GiB / 536 GiB avail 2024-09-13T14:54:04.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:04 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1874318537' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-13T14:54:04.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:04 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-13T14:54:04.869 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:04.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:04.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:04.971 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:04.993 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:04.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:04.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:05.095 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:05.116 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:05.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:05.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:05.317 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:05.343 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:05.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:05.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:05.445 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:05.470 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:05.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:05.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:05.571 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:05.602 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:05.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:05.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:05.617 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-09-13T14:54:05.705 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:05.726 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:05.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:05.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:05.828 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:05.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:05.853 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:05.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:05.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:05 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-13T14:54:05.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:05 smithi184 ceph-mon[92527]: osdmap e1222: 8 total, 8 up, 7 in 2024-09-13T14:54:05.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:05 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-13T14:54:05.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:05 smithi151 ceph-mon[104450]: osdmap e1222: 8 total, 8 up, 7 in 2024-09-13T14:54:05.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:05 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-13T14:54:05.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:05 smithi151 ceph-mon[108966]: osdmap e1222: 8 total, 8 up, 7 in 2024-09-13T14:54:05.954 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:05.978 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:05.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:05.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:06.079 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:06.103 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:06.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:06.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:06.127 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:54:06.127 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:54:06.205 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:06.230 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:06.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:06.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:06.332 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:06.358 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:06.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:06.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:06.460 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:06.486 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:06.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:06.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:06.588 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:06.614 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:06.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:06.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:06.716 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:06.742 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:06.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:06.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:06.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:06.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:06 smithi184 ceph-mon[92527]: osdmap e1223: 8 total, 8 up, 7 in 2024-09-13T14:54:06.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:06 smithi184 ceph-mon[92527]: pgmap v1869: 119 pgs: 15 peering, 104 active+clean; 582 KiB data, 3.8 GiB used, 533 GiB / 536 GiB avail 2024-09-13T14:54:06.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:06 smithi151 ceph-mon[104450]: osdmap e1223: 8 total, 8 up, 7 in 2024-09-13T14:54:06.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:06 smithi151 ceph-mon[104450]: pgmap v1869: 119 pgs: 15 peering, 104 active+clean; 582 KiB data, 3.8 GiB used, 533 GiB / 536 GiB avail 2024-09-13T14:54:06.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:06 smithi151 ceph-mon[108966]: osdmap e1223: 8 total, 8 up, 7 in 2024-09-13T14:54:06.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:06 smithi151 ceph-mon[108966]: pgmap v1869: 119 pgs: 15 peering, 104 active+clean; 582 KiB data, 3.8 GiB used, 533 GiB / 536 GiB avail 2024-09-13T14:54:06.865 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:06.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:06.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:06.967 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:06.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:06.994 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:06.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:07.095 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:07.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:07.117 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:07.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:07.218 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:07.243 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:07.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:07.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:07.258 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T14:54:07.345 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:07.370 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:07.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:07.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:07.472 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:07.495 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:07.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:07.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:07.597 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:07.618 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:07.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:07.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:07.720 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:07.732 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:07.734 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:07.745 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:07.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:07.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:07.774 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:07 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:54:07.774 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:07 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:54:07.774 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:07 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:54:07.774 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:07 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:54:07.774 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:07 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:54:07.774 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:07 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:54:07.778 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:07.848 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:07.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:07.876 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:07.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:07.977 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:07.998 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:07.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:07.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:08.100 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:08.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:07 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:54:08.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:07 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:54:08.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:07 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:54:08.126 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:08.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:08.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:08.228 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:08.251 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:08.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:08.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:08.352 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:08.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:08.377 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:08.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:08.478 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:08.502 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:08.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:08.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:08.603 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:08.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:08.626 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:08.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:08.727 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:08.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:08.749 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:08.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:08.850 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:08.875 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:08.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:08.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:08.977 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:09.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:09.002 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:09.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:09.085 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:08 smithi151 ceph-mon[104450]: pgmap v1870: 119 pgs: 15 peering, 104 active+clean; 582 KiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 6 B/s, 0 objects/s recovering 2024-09-13T14:54:09.085 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:08 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:54:09.085 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:08 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:54:09.085 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:08 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:54:09.085 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:08 smithi151 ceph-mon[108966]: pgmap v1870: 119 pgs: 15 peering, 104 active+clean; 582 KiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 6 B/s, 0 objects/s recovering 2024-09-13T14:54:09.085 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:08 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:54:09.085 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:08 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:54:09.085 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:08 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:54:09.103 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:09.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:08 smithi184 ceph-mon[92527]: pgmap v1870: 119 pgs: 15 peering, 104 active+clean; 582 KiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 6 B/s, 0 objects/s recovering 2024-09-13T14:54:09.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:08 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:54:09.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:08 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:54:09.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:08 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:54:09.129 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:09.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:09.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:09.231 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:09.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:09.253 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:09.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:09.354 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:09.376 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:09.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:09.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:09.478 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:09.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:09.505 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:09.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:09.606 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:09.629 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:09.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:09.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:09.732 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:09.757 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:09.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:09.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:09.859 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:09.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:09.881 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:09.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:09.982 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:10.009 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:10.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:10.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:10.113 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:10.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:10.135 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:10.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:10.236 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:10.261 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:10.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:10.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:10.363 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:10.385 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:10.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:10.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:10.487 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:10.508 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:10.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:10.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:10.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:54:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:54:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:54:10.611 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:10.618 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 6, 7, 5, 1] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:54:10.618 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:54:10.618 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-13T14:54:10.619 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:54:10.619 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:54:10.619 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-09-13T14:54:10.638 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:10.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:10.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:10.741 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:10.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:10.764 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:10.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:10.866 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:10.888 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:10.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:10.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:10.990 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:11.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:11.013 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:11.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:11.093 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:10 smithi151 ceph-mon[104450]: pgmap v1871: 119 pgs: 15 peering, 104 active+clean; 582 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 6 B/s, 0 objects/s recovering 2024-09-13T14:54:11.093 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:10 smithi151 ceph-mon[108966]: pgmap v1871: 119 pgs: 15 peering, 104 active+clean; 582 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 6 B/s, 0 objects/s recovering 2024-09-13T14:54:11.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:10 smithi184 ceph-mon[92527]: pgmap v1871: 119 pgs: 15 peering, 104 active+clean; 582 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 6 B/s, 0 objects/s recovering 2024-09-13T14:54:11.115 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:11.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:11.143 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:11.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:11.201 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:11.244 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:11.269 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:11.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:11.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:11.371 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:11.396 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:11.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:11.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:11.498 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:11.522 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:11.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:11.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:11.623 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:11.650 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:11.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:11.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:11.753 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:11.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:11.777 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:11.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:11.878 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:11.904 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:11.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:11.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:12.007 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:12.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:12.029 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:12.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:12.130 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:12.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:12.152 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:12.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:12.253 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:12.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:12.276 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:12.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:12.377 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:12.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:12.404 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:12.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:12.505 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:12.526 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:12.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:12.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:12.628 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:12.654 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:12.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:12.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:12.710 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:12.756 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:12.777 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:12.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:12.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:12.787 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-13T14:54:12.879 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:12.902 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:12.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:12.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:13.004 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:13.030 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:13.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:13.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:13.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:12 smithi184 ceph-mon[92527]: pgmap v1872: 119 pgs: 119 active+clean; 582 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 5 B/s, 0 objects/s recovering 2024-09-13T14:54:13.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:12 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/302165555' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:54:13.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:12 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:54:13.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:12 smithi151 ceph-mon[104450]: pgmap v1872: 119 pgs: 119 active+clean; 582 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 5 B/s, 0 objects/s recovering 2024-09-13T14:54:13.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:12 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/302165555' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:54:13.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:12 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:54:13.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:12 smithi151 ceph-mon[108966]: pgmap v1872: 119 pgs: 119 active+clean; 582 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 5 B/s, 0 objects/s recovering 2024-09-13T14:54:13.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:12 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/302165555' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:54:13.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:12 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:54:13.132 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:13.156 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:13.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:13.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:13.258 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:13.282 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:13.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:13.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:13.384 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:13.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:13.409 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:13.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:13.510 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:13.533 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:13.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:13.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:13.635 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:13.658 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:13.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:13.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:13.760 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:13.787 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:13.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:13.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:13.870 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:54:13.870 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:54:13.886 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:54:13.886 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:54:13.889 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:13.902 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:54:13.903 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:54:13.903 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:13 smithi151 ceph-mon[104450]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:13.903 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:13 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:54:13.903 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:13 smithi151 ceph-mon[104450]: osdmap e1224: 8 total, 8 up, 7 in 2024-09-13T14:54:13.903 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:13 smithi151 ceph-mon[108966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:13.904 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:13 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:54:13.904 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:13 smithi151 ceph-mon[108966]: osdmap e1224: 8 total, 8 up, 7 in 2024-09-13T14:54:13.910 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:13.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:13.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:13.919 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:54:13.919 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:54:13.936 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:54:13.936 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:54:13.952 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:54:13.952 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:54:13.968 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:54:13.968 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:54:13.984 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:54:13.984 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:54:14.012 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:14.038 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:14.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:14.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:14.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:13 smithi184 ceph-mon[92527]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:14.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:13 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:54:14.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:13 smithi184 ceph-mon[92527]: osdmap e1224: 8 total, 8 up, 7 in 2024-09-13T14:54:14.140 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:14.166 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:14.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:14.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:14.268 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:14.289 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:14.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:14.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:14.391 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:14.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:14.418 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:14.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:14.519 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:14.540 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:14.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:14.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:14.642 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:14.668 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:14.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:14.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:14.727 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:54:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:54:14] "GET /metrics HTTP/1.1" 200 36574 "" "Prometheus/2.51.0" 2024-09-13T14:54:14.770 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:14.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:14.793 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:14.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:14.895 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:14.920 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:14.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:14.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:15.022 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:15.047 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:15.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:15.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:15.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:14 smithi184 ceph-mon[92527]: pgmap v1874: 119 pgs: 119 active+clean; 582 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 5 B/s, 0 objects/s recovering 2024-09-13T14:54:15.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:14 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/83673909' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:15.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:14 smithi151 ceph-mon[108966]: pgmap v1874: 119 pgs: 119 active+clean; 582 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 5 B/s, 0 objects/s recovering 2024-09-13T14:54:15.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:14 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/83673909' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:15.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:14 smithi151 ceph-mon[104450]: pgmap v1874: 119 pgs: 119 active+clean; 582 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 5 B/s, 0 objects/s recovering 2024-09-13T14:54:15.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:14 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/83673909' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:15.149 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:15.174 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:15.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:15.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:15.276 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:15.298 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:15.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:15.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:15.400 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:15.422 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:15.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:15.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:15.524 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:15.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:15.546 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:15.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:15.647 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:15.665 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:54:15.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:15.670 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:15.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:15.771 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:15.797 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:15.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:15.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:15.808 INFO:teuthology.orchestra.run.smithi151.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-09-13T14:54:15.899 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:15.923 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:15.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:15.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:16.025 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:16.047 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:16.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:16.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:16.075 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:15 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3151705673' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:54:16.075 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:15 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:54:16.076 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:15 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3151705673' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:54:16.076 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:15 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:54:16.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:15 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3151705673' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:54:16.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:15 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:54:16.149 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:16.153 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:54:16.153 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-13T14:54:16.153 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-13T14:54:16.153 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:54:16.175 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:16.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:16.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:16.277 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:16.302 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:16.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:16.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:16.404 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:16.426 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:16.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:16.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:16.528 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:16.553 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:16.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:16.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:16.655 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:16.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:16.681 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:16.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:16.752 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:54:16.782 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:16.807 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:16.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:16.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:16.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T14:54:16.909 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:16.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:16.934 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:16.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:17.035 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:17.059 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:17.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:17.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:17.093 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:16 smithi151 ceph-mon[104450]: pgmap v1875: 119 pgs: 119 active+clean; 582 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 4 B/s, 0 objects/s recovering 2024-09-13T14:54:17.093 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:16 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:54:17.093 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:16 smithi151 ceph-mon[104450]: osdmap e1225: 8 total, 8 up, 7 in 2024-09-13T14:54:17.093 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:54:17.093 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:17.093 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:54:17.094 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:16 smithi151 ceph-mon[108966]: pgmap v1875: 119 pgs: 119 active+clean; 582 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 4 B/s, 0 objects/s recovering 2024-09-13T14:54:17.094 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:16 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:54:17.094 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:16 smithi151 ceph-mon[108966]: osdmap e1225: 8 total, 8 up, 7 in 2024-09-13T14:54:17.094 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:54:17.094 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:17.094 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:54:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:16 smithi184 ceph-mon[92527]: pgmap v1875: 119 pgs: 119 active+clean; 582 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 4 B/s, 0 objects/s recovering 2024-09-13T14:54:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:16 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:54:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:16 smithi184 ceph-mon[92527]: osdmap e1225: 8 total, 8 up, 7 in 2024-09-13T14:54:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:54:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:54:17.161 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:17.182 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:17.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:17.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:17.284 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:17.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:17.307 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:17.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:17.408 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:17.434 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:17.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:17.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:17.537 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:17.563 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:17.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:17.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:17.665 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:17.691 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:17.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:17.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:17.793 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:17.815 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:17.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:17.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:17.921 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:17.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:17.943 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:17.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:18.045 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:18.048 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:17 smithi151 ceph-mon[104450]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:54:18.048 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:17 smithi151 ceph-mon[108966]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:54:18.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:18.068 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:18.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:18.169 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:18.191 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:18.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:18.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:18.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:18.314 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:18.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:18.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:18.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:17 smithi184 ceph-mon[92527]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:54:18.417 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:18.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:18.444 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:18.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:18.545 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:18.567 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:18.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:18.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:18.669 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:18.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:18.692 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:18.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:18.792 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:18.814 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:18.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:18.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:18.916 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:18.939 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:18.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:18.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:19.029 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:18 smithi151 ceph-mon[104450]: pgmap v1877: 119 pgs: 119 active+clean; 582 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-13T14:54:19.030 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:18 smithi151 ceph-mon[108966]: pgmap v1877: 119 pgs: 119 active+clean; 582 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-13T14:54:19.041 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:19.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:19.065 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:19.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:19.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:18 smithi184 ceph-mon[92527]: pgmap v1877: 119 pgs: 119 active+clean; 582 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-13T14:54:19.166 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:19.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:19.189 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:19.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:19.288 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:54:19.288 INFO:teuthology.orchestra.run.smithi151.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T14:54:19.288 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:54:19.290 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:19.313 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:19.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:19.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:19.415 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:19.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:19.437 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:19.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:19.538 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:19.560 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:19.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:19.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:19.662 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:19.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:19.685 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:19.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:19.770 DEBUG:teuthology.orchestra.run.smithi151:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-09-13T14:54:19.772 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T14:54:19.772 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T14:54:19.786 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:19.807 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:19.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:19.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:19.909 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:19.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:19.932 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:19.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:19.932 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:19.933 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:19.955 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:hints = 1 2024-09-13T14:54:19.955 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T14:54:19.956 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Object prefix: benchmark_data_smithi151_261954 2024-09-13T14:54:19.956 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:54:19.956 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 0 16 16 0 0 0 - 0 2024-09-13T14:54:19.993 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:20.032 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:20.081 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:20.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:20.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:20.183 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:20.220 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:20.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:20.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:20.322 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:20.362 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:20.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:20.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:20.464 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:20.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:20.504 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:20.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:20.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:54:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:54:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:54:20.605 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:20.638 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:20.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:20.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:20.741 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:20.778 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:20.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:20.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:20.880 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:20.916 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:20.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:20.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:20.956 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 1 16 3778 3762 234.994 235.125 0.00431821 0.00424163 2024-09-13T14:54:21.019 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:21.050 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:21.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:21.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:21.152 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:21.153 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:20 smithi151 ceph-mon[104450]: pgmap v1878: 119 pgs: 119 active+clean; 582 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail 2024-09-13T14:54:21.153 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:20 smithi151 ceph-mon[108966]: pgmap v1878: 119 pgs: 119 active+clean; 582 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail 2024-09-13T14:54:21.182 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:21.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:21.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:21.284 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:21.314 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:21.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:21.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:21.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:20 smithi184 ceph-mon[92527]: pgmap v1878: 119 pgs: 119 active+clean; 582 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail 2024-09-13T14:54:21.416 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:21.448 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:21.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:21.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:21.550 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:21.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:21.581 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:21.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:21.682 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:21.719 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:21.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:21.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:21.820 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:21.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:21.842 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:21.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:21.943 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:21.956 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 2 16 7170 7154 223.49 212 0.00548214 0.00446555 2024-09-13T14:54:21.976 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:21.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:21.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:22.078 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:22.122 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:22.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:22.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:22.224 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:22.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:22.262 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:22.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:22.363 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:22.399 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:22.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:22.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:22.501 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:22.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:22.533 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:22.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:22.635 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:22.674 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:22.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:22.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:22.776 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:22.797 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:22.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:22.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:22.900 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:22.932 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:22.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:22.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:22.957 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 3 16 10062 10046 209.24 180.75 0.00352729 0.00477164 2024-09-13T14:54:23.034 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:23.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:23.072 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:23.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:23.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:22 smithi151 ceph-mon[108966]: pgmap v1879: 119 pgs: 119 active+clean; 24 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 2.6 MiB/s wr, 42 op/s 2024-09-13T14:54:23.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:22 smithi151 ceph-mon[104450]: pgmap v1879: 119 pgs: 119 active+clean; 24 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 2.6 MiB/s wr, 42 op/s 2024-09-13T14:54:23.174 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:23.203 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:23.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:23.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:23.305 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:23.344 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:23.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:23.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:23.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:22 smithi184 ceph-mon[92527]: pgmap v1879: 119 pgs: 119 active+clean; 24 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 2.6 MiB/s wr, 42 op/s 2024-09-13T14:54:23.446 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:23.483 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:23.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:23.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:23.585 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:23.622 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:23.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:23.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:23.725 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:23.761 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:23.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:23.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:23.863 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:23.894 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:23.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:23.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:23.957 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 4 16 13214 13198 206.176 197 0.00309191 0.00484294 2024-09-13T14:54:23.996 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:24.033 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:24.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:24.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:24.135 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:24.173 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:24.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:24.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:24.277 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:24.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:24.320 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:24.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:24.421 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:24.452 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:24.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:24.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:24.554 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:24.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:24.585 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:24.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:24.686 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:24.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:24.709 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:24.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:24.810 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:24.849 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:24.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:24.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:24.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:54:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:54:24] "GET /metrics HTTP/1.1" 200 36572 "" "Prometheus/2.51.0" 2024-09-13T14:54:24.886 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-13T14:54:24.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:24.952 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 6, 7, 5, 1] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:54:24.952 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:54:24.952 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-13T14:54:24.953 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:54:24.953 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:54:24.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-09-13T14:54:24.957 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 5 16 15730 15714 196.378 157.25 0.00702104 0.00508403 2024-09-13T14:54:24.989 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:24.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:24.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:25.091 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:25.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:25.123 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:25.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:25.224 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:25.252 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:25.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:25.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:25.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:24 smithi184 ceph-mon[92527]: pgmap v1880: 119 pgs: 119 active+clean; 323 MiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 32 MiB/s wr, 516 op/s 2024-09-13T14:54:25.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:24 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1201486846' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:54:25.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:24 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:54:25.354 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:25.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:24 smithi151 ceph-mon[104450]: pgmap v1880: 119 pgs: 119 active+clean; 323 MiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 32 MiB/s wr, 516 op/s 2024-09-13T14:54:25.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:24 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1201486846' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:54:25.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:24 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:54:25.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:24 smithi151 ceph-mon[108966]: pgmap v1880: 119 pgs: 119 active+clean; 323 MiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 32 MiB/s wr, 516 op/s 2024-09-13T14:54:25.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:24 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1201486846' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:54:25.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:24 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:54:25.395 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:25.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:25.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:25.498 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:25.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:25.537 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:25.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:25.638 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:25.668 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:25.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:25.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:25.770 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:25.799 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:25.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:25.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:25.901 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:25.933 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:25.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:25.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:25.958 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 6 16 18398 18382 191.413 166.75 0.00511238 0.00521766 2024-09-13T14:54:26.035 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:26.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:26.068 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:26.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:26.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:25 smithi151 ceph-mon[104450]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:26.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:25 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:54:26.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:25 smithi151 ceph-mon[104450]: osdmap e1226: 8 total, 8 up, 7 in 2024-09-13T14:54:26.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:25 smithi151 ceph-mon[108966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:26.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:25 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:54:26.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:25 smithi151 ceph-mon[108966]: osdmap e1226: 8 total, 8 up, 7 in 2024-09-13T14:54:26.169 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:26.200 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:26.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:26.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:26.302 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:26.343 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:26.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:26.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:26.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:25 smithi184 ceph-mon[92527]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:26.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:25 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:54:26.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:25 smithi184 ceph-mon[92527]: osdmap e1226: 8 total, 8 up, 7 in 2024-09-13T14:54:26.444 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:26.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:26.485 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:26.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:26.586 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:26.612 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:26.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:26.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:26.714 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:26.749 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:26.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:26.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:26.852 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:26.879 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:26.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:26.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:26.958 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 7 16 20995 20979 187.255 162.312 0.00819793 0.00533443 2024-09-13T14:54:26.981 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:27.011 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:27.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:27.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:27.041 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:26 smithi151 ceph-mon[108966]: pgmap v1882: 119 pgs: 119 active+clean; 744 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 75 MiB/s wr, 1.21k op/s 2024-09-13T14:54:27.042 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:26 smithi151 ceph-mon[104450]: pgmap v1882: 119 pgs: 119 active+clean; 744 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 75 MiB/s wr, 1.21k op/s 2024-09-13T14:54:27.113 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:27.149 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:27.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:27.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:27.170 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:27.211 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:54:27.211 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-13T14:54:27.211 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-13T14:54:27.211 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-13T14:54:27.211 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:54:27.251 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:27.281 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:27.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:27.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:27.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:26 smithi184 ceph-mon[92527]: pgmap v1882: 119 pgs: 119 active+clean; 744 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 75 MiB/s wr, 1.21k op/s 2024-09-13T14:54:27.384 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:27.415 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:27.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:27.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:27.517 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:27.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:27.547 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:27.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:27.648 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:27.687 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:27.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:27.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:27.789 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:27.820 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:27.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:27.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:27.922 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:27.943 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:27.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:27.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:27.958 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 8 16 23312 23296 181.949 144.812 0.00165544 0.00546071 2024-09-13T14:54:28.045 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:28.075 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:28.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:28.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:28.177 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:28.217 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:28.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:28.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:28.319 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:28.356 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:28.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:28.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:28.458 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:28.483 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:28.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:28.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:28.585 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:28.606 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:28.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:28.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:28.708 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:28.734 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:28.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:28.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:28.836 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:28.870 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:28.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:28.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:28.904 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:54:28.958 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 9 16 25098 25082 174.134 111.625 0.0128013 0.00573654 2024-09-13T14:54:28.973 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:29.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:29.016 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:29.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:29.117 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:29.148 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:29.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:29.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:29.191 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:28 smithi184 ceph-mon[92527]: pgmap v1883: 119 pgs: 119 active+clean; 1.1 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-09-13T14:54:29.251 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:29.281 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:29.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:29.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:29.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:28 smithi151 ceph-mon[108966]: pgmap v1883: 119 pgs: 119 active+clean; 1.1 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-09-13T14:54:29.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:28 smithi151 ceph-mon[104450]: pgmap v1883: 119 pgs: 119 active+clean; 1.1 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-09-13T14:54:29.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:29.422 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:29.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:29.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:29.524 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:29.545 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:29.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:29.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:29.646 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:29.683 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:29.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:29.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:29.785 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:29.822 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:29.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:29.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:29.924 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:29.954 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:29.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:29.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:29.958 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 10 16 26901 26885 167.99 112.688 0.00486649 0.00594713 2024-09-13T14:54:30.056 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:30.077 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:30.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:30.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:30.178 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:30.210 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:30.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:30.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:30.312 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:30.341 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:30.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:30.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:30.443 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:30.464 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:30.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:30.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:30.566 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:30.596 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:30.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:30.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:30.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:54:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:54:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:54:30.698 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:30.728 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:30.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:30.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:30.830 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:30.851 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:30.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:30.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:30.953 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:30.958 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 11 16 28602 28586 162.383 106.312 0.00710994 0.00615186 2024-09-13T14:54:30.982 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:30.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:30.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:31.084 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:31.120 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:31.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:31.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:31.221 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:31.260 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:31.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:31.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:31.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:30 smithi184 ceph-mon[92527]: pgmap v1884: 119 pgs: 119 active+clean; 1.2 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-13T14:54:31.362 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:31.363 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:30 smithi151 ceph-mon[104450]: pgmap v1884: 119 pgs: 119 active+clean; 1.2 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-13T14:54:31.363 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:30 smithi151 ceph-mon[108966]: pgmap v1884: 119 pgs: 119 active+clean; 1.2 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-13T14:54:31.391 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:31.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:31.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:31.493 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:31.513 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:31.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:31.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:31.615 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:31.653 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:31.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:31.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:31.755 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:31.776 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:31.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:31.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:31.877 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:31.903 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:31.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:31.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:31.958 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 12 16 30308 30292 157.736 106.625 0.00517576 0.00633411 2024-09-13T14:54:32.005 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:32.038 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:32.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:32.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:32.141 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:32.162 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:32.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:32.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:32.264 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:32.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:32.303 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:32.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:32.404 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:32.435 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:32.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:32.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:32.537 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:32.558 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:32.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:32.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:32.660 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:32.691 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:32.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:32.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:32.792 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:32.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:32.830 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:32.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:54:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:32 smithi184 ceph-mon[92527]: pgmap v1885: 119 pgs: 119 active+clean; 1.5 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-09-13T14:54:32.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:54:32.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:32.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:32 smithi151 ceph-mon[104450]: pgmap v1885: 119 pgs: 119 active+clean; 1.5 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-09-13T14:54:32.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:54:32.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:32.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:32 smithi151 ceph-mon[108966]: pgmap v1885: 119 pgs: 119 active+clean; 1.5 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-09-13T14:54:32.932 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:32.958 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 13 16 32161 32145 154.511 115.812 0.00650576 0.00646532 2024-09-13T14:54:32.969 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:32.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:32.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:33.071 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:33.092 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:33.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:33.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:33.194 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:33.223 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:33.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:33.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:33.324 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:33.357 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:33.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:33.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:33.459 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:33.496 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:33.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:33.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:33.598 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:33.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:33.625 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:33.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:33.726 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:33.763 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:33.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:33.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:33.865 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:33.901 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:33.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:33.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:33.959 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 14 16 33802 33786 150.799 102.562 0.00189323 0.00661179 2024-09-13T14:54:34.004 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:34.034 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:34.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:34.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:34.171 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:34.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:34.273 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:34.294 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:34.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:34.397 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:34.433 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:34.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:34.535 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:34.572 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:34.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:34.674 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:34.705 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:34.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:34.807 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:34.828 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:34.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:34.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:34 smithi151 ceph-mon[104450]: pgmap v1886: 119 pgs: 119 active+clean; 1.8 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 152 MiB/s wr, 2.43k op/s 2024-09-13T14:54:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:34 smithi151 ceph-mon[108966]: pgmap v1886: 119 pgs: 119 active+clean; 1.8 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 152 MiB/s wr, 2.43k op/s 2024-09-13T14:54:34.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:54:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:54:34] "GET /metrics HTTP/1.1" 200 36637 "" "Prometheus/2.51.0" 2024-09-13T14:54:34.930 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:34.959 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 15 16 35341 35325 147.158 96.1875 0.0075726 0.00678966 2024-09-13T14:54:34.959 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:34.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:35.061 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:35.090 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:35.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:35.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:35.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:34 smithi184 ceph-mon[92527]: pgmap v1886: 119 pgs: 119 active+clean; 1.8 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 152 MiB/s wr, 2.43k op/s 2024-09-13T14:54:35.192 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:35.237 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:35.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:35.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:35.339 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:35.368 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:35.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:35.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:35.469 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:35.510 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:35.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:35.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:35.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:35.633 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:35.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:35.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:35.735 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:35.771 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:35.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:35.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:35.873 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:35.909 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:35.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:35.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:35.959 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 16 16 36989 36973 144.398 103 0.00769413 0.00691986 2024-09-13T14:54:36.012 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:36.033 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:36.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:36.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:36.135 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:36.396 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:36.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:36.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:36.498 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:36.542 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:36.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:36.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:36.644 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:36.672 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:36.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:36.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:36.774 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:36.802 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:36.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:36.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:36.904 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:36.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:36.934 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:36.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:36.959 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 17 16 38387 38371 141.044 87.375 0.0207788 0.00708407 2024-09-13T14:54:37.035 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:37.062 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:37.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:37.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:37.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:36 smithi184 ceph-mon[92527]: pgmap v1887: 119 pgs: 119 active+clean; 2.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 163 MiB/s wr, 2.60k op/s 2024-09-13T14:54:37.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:36 smithi151 ceph-mon[104450]: pgmap v1887: 119 pgs: 119 active+clean; 2.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 163 MiB/s wr, 2.60k op/s 2024-09-13T14:54:37.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:36 smithi151 ceph-mon[108966]: pgmap v1887: 119 pgs: 119 active+clean; 2.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 163 MiB/s wr, 2.60k op/s 2024-09-13T14:54:37.165 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:37.194 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:37.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:37.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:37.295 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:37.324 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:37.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:37.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:37.426 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:37.454 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:37.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:37.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:37.556 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:37.563 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:54:37.563 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:54:37.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:37.598 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:37.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:37.609 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:54:37.609 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:54:37.641 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:54:37.641 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:54:37.677 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:54:37.677 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:54:37.700 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:37.707 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:54:37.707 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:54:37.737 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:54:37.738 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:54:37.752 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:37.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:37.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:37.770 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:54:37.771 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:54:37.803 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:54:37.804 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:54:37.822 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:37 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1654665985' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:37.823 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:37 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1654665985' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:37.855 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:37.889 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:37.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:37.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:37.959 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 18 16 40000 39984 138.808 100.812 0.0128246 0.00719812 2024-09-13T14:54:37.992 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:38.026 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:38.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:38.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:38.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:37 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1654665985' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:38.125 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T14:54:38.129 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:38.154 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:38.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:38.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:38.256 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:38.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:38.329 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:38.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:38.430 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:38.432 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:54:38.432 INFO:teuthology.orchestra.run.smithi151.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T14:54:38.432 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:54:38.453 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:38.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:38.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:38.555 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:38.585 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:38.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:38.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:38.687 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:38.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:38.729 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:38.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:38.830 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:38.858 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:38.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:38.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:38.959 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 19 16 41659 41643 136.959 103.688 0.00535007 0.00729601 2024-09-13T14:54:38.959 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:38.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:38.997 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:38.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:39.098 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:39.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:38 smithi184 ceph-mon[92527]: pgmap v1888: 119 pgs: 119 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 129 MiB/s wr, 2.06k op/s 2024-09-13T14:54:39.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:38 smithi151 ceph-mon[108966]: pgmap v1888: 119 pgs: 119 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 129 MiB/s wr, 2.06k op/s 2024-09-13T14:54:39.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:38 smithi151 ceph-mon[104450]: pgmap v1888: 119 pgs: 119 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 129 MiB/s wr, 2.06k op/s 2024-09-13T14:54:39.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:39.138 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:39.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:39.239 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:39.265 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:39.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:39.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:39.367 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:39.410 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:39.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:39.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:39.512 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:39.553 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:39.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:39.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:39.655 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:39.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:39.702 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:39.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:39.803 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:39.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:39.833 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:39.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:39.934 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:39.960 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:2024-09-13T14:54:39.959882+0000 min lat: 0.00116648 max lat: 0.207775 avg lat: 0.00736057 2024-09-13T14:54:39.960 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:54:39.960 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 20 16 43468 43452 135.759 113.062 0.00686987 0.00736057 2024-09-13T14:54:39.969 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:39.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:39.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:40.055 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:40.070 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:40.071 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:40.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:40.132 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:40.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:40.233 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:40.262 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:40.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:40.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:40.364 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:40.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:40.409 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:40.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:40.510 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:40.541 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:40.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:40.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:40.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:54:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:54:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:54:40.643 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:40.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:40.681 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:40.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:40.782 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:40.811 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:40.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:40.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:40.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:40 smithi151 ceph-mon[104450]: pgmap v1889: 119 pgs: 119 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 103 MiB/s wr, 1.64k op/s 2024-09-13T14:54:40.842 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:40 smithi151 ceph-mon[108966]: pgmap v1889: 119 pgs: 119 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 103 MiB/s wr, 1.64k op/s 2024-09-13T14:54:40.913 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:40.952 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:40.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:40.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:40.960 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 21 16 45225 45209 134.523 109.812 0.00929514 0.00742839 2024-09-13T14:54:41.054 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:41.089 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:41.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:41.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:41.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:40 smithi184 ceph-mon[92527]: pgmap v1889: 119 pgs: 119 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 103 MiB/s wr, 1.64k op/s 2024-09-13T14:54:41.191 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:41.218 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:41.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:41.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:41.321 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:41.350 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:41.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:41.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:41.452 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:41.483 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:41.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:41.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:41.584 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:41.611 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:41.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:41.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:41.713 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:41.756 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:41.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:41.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:41.858 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:41.909 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:41.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:41.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:41.960 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 22 16 46997 46981 133.441 110.75 0.0105567 0.00748779 2024-09-13T14:54:42.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:42.028 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:54:42.060 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:42.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:42.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:42.163 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:42.204 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:42.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:42.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:42.306 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:42.336 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:42.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:42.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:42.437 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:42.468 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:42.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:42.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:42.570 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:42.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:42.596 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:42.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:42.697 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:42.737 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:42.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:42.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:42.839 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:42.869 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:42.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:42.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:42.960 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 23 16 48523 48507 131.785 95.375 0.00415492 0.00757222 2024-09-13T14:54:42.971 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:42.994 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:42.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:42.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:43.096 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:42 smithi184 ceph-mon[92527]: pgmap v1890: 119 pgs: 119 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-13T14:54:43.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:42 smithi151 ceph-mon[104450]: pgmap v1890: 119 pgs: 119 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-13T14:54:43.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:42 smithi151 ceph-mon[108966]: pgmap v1890: 119 pgs: 119 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-13T14:54:43.127 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:43.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:43.229 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:43.266 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:43.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:43.368 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:43.400 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:43.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:43.502 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:43.529 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:43.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:43.616 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:54:43.617 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:54:43.631 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:43.671 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:43.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:43.773 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:43.806 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:43.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:43.908 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:43.947 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:43.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:43.960 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 24 16 50162 50146 130.562 102.438 0.0184244 0.0076538 2024-09-13T14:54:44.050 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:44.077 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:44.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:44.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:44.179 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:44.215 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:44.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:44.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:44.302 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 6, 7, 5, 1] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:54:44.303 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:54:44.303 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-09-13T14:54:44.303 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd in 4 2024-09-13T14:54:44.316 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:44.341 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:44.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:44.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:44.443 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:44.471 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:44.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:44.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:44.573 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:44.600 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:44.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:44.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:44.702 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:44.732 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:44.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:44.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:44.834 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:44.857 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:44 smithi151 ceph-mon[104450]: pgmap v1891: 119 pgs: 119 active+clean; 2.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-13T14:54:44.857 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:54:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:54:44] "GET /metrics HTTP/1.1" 200 36637 "" "Prometheus/2.51.0" 2024-09-13T14:54:44.857 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:44 smithi151 ceph-mon[108966]: pgmap v1891: 119 pgs: 119 active+clean; 2.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-13T14:54:44.858 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T14:54:44.870 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:44.889 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:44.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:44.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:44.961 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 25 15 51734 51719 129.272 98.3125 0.00785584 0.00773049 2024-09-13T14:54:44.991 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:45.035 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:45.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:45.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:45.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:44 smithi184 ceph-mon[92527]: pgmap v1891: 119 pgs: 119 active+clean; 2.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-13T14:54:45.137 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:45.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:45.174 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:45.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:45.267 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:54:45.275 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:45.312 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:45.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:45.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:45.414 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:45.454 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:45.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:45.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:45.556 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:45.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:45.603 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:45.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:45.704 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:45.728 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:45.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:45.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:45.830 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:45.864 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:45.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:45.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:45.961 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 26 16 53232 53216 127.899 93.5625 0.00723864 0.00781343 2024-09-13T14:54:45.965 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:45.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:46.000 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:46.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:46.101 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:46.140 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:46.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:46.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:46.242 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:46.264 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:46.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:46.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:46.366 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:46.394 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:46.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:46.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:46.496 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:46.536 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:46.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:46.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:46.638 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:46.677 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:46.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:46.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:46.748 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-13T14:54:46.778 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:46.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:46.829 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:46.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:46.930 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:46.961 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 27 16 54615 54599 126.362 86.4375 0.0112133 0.00790944 2024-09-13T14:54:46.969 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:46.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:46.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:47.071 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:46 smithi184 ceph-mon[92527]: pgmap v1892: 119 pgs: 119 active+clean; 3.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-09-13T14:54:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:46 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1549744411' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:54:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:46 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:54:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:47.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:46 smithi151 ceph-mon[104450]: pgmap v1892: 119 pgs: 119 active+clean; 3.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-09-13T14:54:47.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:46 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1549744411' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:54:47.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:46 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:54:47.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:47.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:46 smithi151 ceph-mon[108966]: pgmap v1892: 119 pgs: 119 active+clean; 3.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-09-13T14:54:47.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:46 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1549744411' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:54:47.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:46 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:54:47.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:47.111 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:47.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:47.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:47.213 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:47.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:47.244 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:47.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:47.345 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:47.375 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:47.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:47.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:47.477 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:47.510 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:47.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:47.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:47.565 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:47.612 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:47.644 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:47.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:47.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:47.746 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:47.775 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:47.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:47.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:47.877 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:47.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:47.919 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:47.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:47.961 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 28 16 56162 56146 125.302 96.6875 0.00740593 0.00797475 2024-09-13T14:54:48.021 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:48.030 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:47 smithi151 ceph-mon[104450]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:48.030 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:47 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:54:48.030 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:47 smithi151 ceph-mon[104450]: osdmap e1227: 8 total, 8 up, 7 in 2024-09-13T14:54:48.030 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:47 smithi151 ceph-mon[108966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:48.030 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:47 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:54:48.030 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:47 smithi151 ceph-mon[108966]: osdmap e1227: 8 total, 8 up, 7 in 2024-09-13T14:54:48.060 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:48.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:48.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:48.065 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:47 smithi184 ceph-mon[92527]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:48.065 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:47 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:54:48.065 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:47 smithi184 ceph-mon[92527]: osdmap e1227: 8 total, 8 up, 7 in 2024-09-13T14:54:48.162 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:48.206 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:48.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:48.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:48.308 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:48.333 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:48.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:48.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:48.435 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:48.474 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:48.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:48.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:48.576 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:48.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:48.618 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:48.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:48.679 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:54:48.679 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [], 2024-09-13T14:54:48.680 INFO:teuthology.orchestra.run.smithi184.stdout: "num_ops": 0 2024-09-13T14:54:48.680 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:54:48.719 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:48.753 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:48.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:48.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:48.855 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:48.876 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:48.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:48.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:48.961 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 29 16 57444 57428 123.744 80.125 0.0281656 0.00807556 2024-09-13T14:54:48.978 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:49.005 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:49.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:49.013 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:48 smithi184 ceph-mon[92527]: pgmap v1894: 119 pgs: 119 active+clean; 3.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-09-13T14:54:49.013 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:48 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/4071130422' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:49.107 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:49.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:48 smithi151 ceph-mon[104450]: pgmap v1894: 119 pgs: 119 active+clean; 3.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-09-13T14:54:49.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:48 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/4071130422' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:49.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:48 smithi151 ceph-mon[108966]: pgmap v1894: 119 pgs: 119 active+clean; 3.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-09-13T14:54:49.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:48 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/4071130422' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:49.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.145 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:49.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:49.246 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:49.284 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:49.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:49.362 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T14:54:49.386 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:49.433 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:49.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:49.535 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:49.536 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:54:49.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.574 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:49.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:49.675 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:49.696 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:49.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:49.738 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:54:49.798 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:49.838 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:49.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:49.940 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:49.961 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 30 16 58696 58680 122.227 78.25 0.0101903 0.00817589 2024-09-13T14:54:49.971 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:49.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:50.073 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:50.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:50.112 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:50.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:50.213 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:50.255 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:50.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:50.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:50.328 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:54:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:54:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:54:50.358 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:50.384 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:50.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:50.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:50.486 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:50.509 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:50.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:50.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:50.611 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:50.641 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:50.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:50.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:50.742 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:50.763 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:50.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:50.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:50.865 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:50.886 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:50.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:50.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:50.962 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 31 16 59868 59852 120.647 73.25 0.00877648 0.00828365 2024-09-13T14:54:50.988 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:51.025 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:51.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:51.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:50 smithi184 ceph-mon[92527]: pgmap v1895: 119 pgs: 119 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-09-13T14:54:51.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:50 smithi151 ceph-mon[104450]: pgmap v1895: 119 pgs: 119 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-09-13T14:54:51.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:50 smithi151 ceph-mon[108966]: pgmap v1895: 119 pgs: 119 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-09-13T14:54:51.127 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:51.160 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:51.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:51.261 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:51.296 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:51.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:51.398 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:51.418 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:51.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:51.520 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:51.549 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:51.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:51.651 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:51.680 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:51.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:51.782 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:51.810 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:51.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:51.912 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:51.939 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:51.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:51.961 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 32 16 61171 61155 119.421 81.4375 0.00437211 0.0083588 2024-09-13T14:54:52.040 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:52.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:52.062 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:52.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:52.163 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:52.190 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:52.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:52.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:52.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:52.313 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:52.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:52.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:52.415 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:52.439 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:52.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:52.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:52.541 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:52.575 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:52.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:52.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:52.677 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:52.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:52.718 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:52.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:52.819 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:52.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:52 smithi184 ceph-mon[92527]: pgmap v1896: 119 pgs: 119 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-13T14:54:52.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:52.865 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:52.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:52.962 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 33 16 62027 62011 117.424 53.5 0.0120179 0.00851115 2024-09-13T14:54:52.966 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:53.002 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:53.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:53.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:53.103 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:53.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:52 smithi151 ceph-mon[104450]: pgmap v1896: 119 pgs: 119 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-13T14:54:53.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:52 smithi151 ceph-mon[108966]: pgmap v1896: 119 pgs: 119 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-13T14:54:53.128 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:53.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:53.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:53.167 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:54:53.167 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [], 2024-09-13T14:54:53.168 INFO:teuthology.orchestra.run.smithi184.stdout: "complaint_time": 30, 2024-09-13T14:54:53.168 INFO:teuthology.orchestra.run.smithi184.stdout: "num_blocked_ops": 0 2024-09-13T14:54:53.168 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:54:53.230 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:53.261 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:53.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:53.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:53.363 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:53.398 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:53.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:53.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:53.500 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:53.521 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:53.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:53.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:53.624 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:53.645 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:53.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:53.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:53.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:53.768 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:53.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:53.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:53.802 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T14:54:53.870 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:53.900 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:53.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:53.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:53.962 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 34 16 62863 62847 115.507 52.25 0.00946305 0.00865135 2024-09-13T14:54:54.001 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:54.039 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:54.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:54.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:54.141 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:54.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:54.165 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:54.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:54.173 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:54:54.266 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:54.287 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:54.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:54.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:54.306 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:54:54.306 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:54:54.306 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:54:54.306 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:54:54.306 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:54:54.307 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:54:54.307 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:54:54.307 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:54:54.307 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:54:54.307 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:54:54.307 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:54:54.307 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:54:54.308 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:54:54.308 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:54:54.308 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:54:54.308 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:54:54.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:54.422 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:54.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:54.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:54.524 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:54.552 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:54.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:54.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:54.654 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:54.688 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:54.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:54.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:54.790 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:54.827 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:54.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:54.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:54 smithi151 ceph-mon[104450]: pgmap v1897: 119 pgs: 119 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-13T14:54:54.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:54 smithi151 ceph-mon[108966]: pgmap v1897: 119 pgs: 119 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-13T14:54:54.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:54:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:54:54] "GET /metrics HTTP/1.1" 200 36642 "" "Prometheus/2.51.0" 2024-09-13T14:54:54.929 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:54.957 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:54.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:54.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:54.962 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 35 16 64021 64005 114.275 72.375 0.0154473 0.008745 2024-09-13T14:54:55.059 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:55.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:55.089 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:55.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:55.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:54 smithi184 ceph-mon[92527]: pgmap v1897: 119 pgs: 119 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-13T14:54:55.190 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:55.225 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:55.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:55.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:55.327 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:55.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:55.362 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:55.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:55.463 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:55.511 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:55.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:55.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:55.613 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:55.652 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:55.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:55.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:55.754 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:55.790 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:55.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:55.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:55.892 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:55.927 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:55.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:55.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:55.962 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 36 16 65485 65469 113.642 91.5 0.00516418 0.00879429 2024-09-13T14:54:56.028 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:56.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:56.071 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:56.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:56.108 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:54:56.108 INFO:teuthology.orchestra.run.smithi184.stdout: "size": 20, 2024-09-13T14:54:56.109 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 600, 2024-09-13T14:54:56.109 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [ 2024-09-13T14:54:56.109 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.109 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:97317 7.1 7:83be73bf:::benchmark_data_smithi151_190906_object97316:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1146)", 2024-09-13T14:54:56.109 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:45:03.741860+0000", 2024-09-13T14:54:56.109 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 592.36307774600004, 2024-09-13T14:54:56.109 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.15392760599999999, 2024-09-13T14:54:56.109 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:54:56.109 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:56.109 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:54:56.109 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:54:56.109 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:54:56.109 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 97317 2024-09-13T14:54:56.110 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.110 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:54:56.110 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.110 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:54:56.110 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.741860+0000", 2024-09-13T14:54:56.110 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.110 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.110 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.110 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:54:56.110 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.741860+0000", 2024-09-13T14:54:56.110 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.110 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.110 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.111 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:54:56.111 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.741859+0000", 2024-09-13T14:54:56.111 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:54:56.111 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.111 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.111 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:54:56.111 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.741901+0000", 2024-09-13T14:54:56.111 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1860000000000002e-05 2024-09-13T14:54:56.111 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.111 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.111 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:54:56.111 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.741911+0000", 2024-09-13T14:54:56.112 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.9469999999999998e-06 2024-09-13T14:54:56.112 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.112 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.112 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:54:56.112 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.741916+0000", 2024-09-13T14:54:56.112 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8119999999999996e-06 2024-09-13T14:54:56.112 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.112 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.112 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:54:56.112 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.754055+0000", 2024-09-13T14:54:56.112 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.012138062999999999 2024-09-13T14:54:56.112 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.112 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.113 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:54:56.113 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.754200+0000", 2024-09-13T14:54:56.113 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014585100000000001 2024-09-13T14:54:56.113 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.113 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.113 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:54:56.113 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.754339+0000", 2024-09-13T14:54:56.113 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000138646 2024-09-13T14:54:56.113 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.113 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.113 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:54:56.113 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.759977+0000", 2024-09-13T14:54:56.114 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0056384099999999999 2024-09-13T14:54:56.114 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.114 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.114 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.114 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.760380+0000", 2024-09-13T14:54:56.114 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00040259999999999997 2024-09-13T14:54:56.114 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.114 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.114 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.114 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.895650+0000", 2024-09-13T14:54:56.114 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.13526993100000001 2024-09-13T14:54:56.114 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.114 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.114 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:54:56.114 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.895747+0000", 2024-09-13T14:54:56.115 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.7399999999999996e-05 2024-09-13T14:54:56.115 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.115 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.115 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:54:56.115 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.895787+0000", 2024-09-13T14:54:56.115 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0070000000000001e-05 2024-09-13T14:54:56.115 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.115 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:54:56.115 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.115 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.115 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.115 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:97328 7.1 7:86c42084:::benchmark_data_smithi151_190906_object97327:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1146)", 2024-09-13T14:54:56.115 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:45:03.755640+0000", 2024-09-13T14:54:56.115 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 592.34929712899998, 2024-09-13T14:54:56.116 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.15131093500000001, 2024-09-13T14:54:56.116 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:54:56.116 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:56.116 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:54:56.116 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:54:56.116 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:54:56.116 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 97328 2024-09-13T14:54:56.116 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.116 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:54:56.116 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.116 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:54:56.116 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.755640+0000", 2024-09-13T14:54:56.116 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.116 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.116 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.117 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:54:56.117 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.755640+0000", 2024-09-13T14:54:56.117 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.117 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.117 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.117 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:54:56.117 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.755638+0000", 2024-09-13T14:54:56.117 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:54:56.117 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.117 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.117 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:54:56.117 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.755672+0000", 2024-09-13T14:54:56.117 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3645000000000001e-05 2024-09-13T14:54:56.117 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.118 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.118 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:54:56.118 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.755686+0000", 2024-09-13T14:54:56.118 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3720000000000001e-05 2024-09-13T14:54:56.118 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.118 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.118 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:54:56.118 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.755692+0000", 2024-09-13T14:54:56.118 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3759999999999996e-06 2024-09-13T14:54:56.118 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.118 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.118 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:54:56.118 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.756858+0000", 2024-09-13T14:54:56.118 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0011653189999999999 2024-09-13T14:54:56.118 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.119 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.119 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:54:56.119 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.757000+0000", 2024-09-13T14:54:56.119 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000142707 2024-09-13T14:54:56.119 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.119 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.119 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:54:56.119 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.757157+0000", 2024-09-13T14:54:56.119 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00015677900000000001 2024-09-13T14:54:56.119 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.119 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.119 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:54:56.119 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.759982+0000", 2024-09-13T14:54:56.119 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002824713 2024-09-13T14:54:56.120 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.120 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.120 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.120 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.760498+0000", 2024-09-13T14:54:56.120 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00051625900000000003 2024-09-13T14:54:56.120 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.120 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.120 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.120 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.906811+0000", 2024-09-13T14:54:56.120 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.146312677 2024-09-13T14:54:56.120 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.120 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.120 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:54:56.120 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.906907+0000", 2024-09-13T14:54:56.121 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.6183999999999999e-05 2024-09-13T14:54:56.121 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.121 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.121 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:54:56.121 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:45:03.906951+0000", 2024-09-13T14:54:56.121 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4499000000000002e-05 2024-09-13T14:54:56.121 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.121 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:54:56.121 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.121 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.121 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.121 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:141042 7.1 7:830a39ed:::benchmark_data_smithi151_190906_object20581:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1155)", 2024-09-13T14:54:56.121 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:46:05.945239+0000", 2024-09-13T14:54:56.121 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 530.15969874999996, 2024-09-13T14:54:56.122 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.149968298, 2024-09-13T14:54:56.122 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:54:56.122 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:56.122 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:54:56.122 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:54:56.122 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:54:56.123 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 141042 2024-09-13T14:54:56.123 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.123 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:54:56.123 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.123 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:54:56.123 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:05.945239+0000", 2024-09-13T14:54:56.123 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.123 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.123 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.123 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:54:56.124 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:05.945239+0000", 2024-09-13T14:54:56.124 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.124 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.124 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.124 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:54:56.124 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:05.945237+0000", 2024-09-13T14:54:56.124 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:56.124 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.124 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.124 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:54:56.124 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:05.945244+0000", 2024-09-13T14:54:56.124 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.8940000000000004e-06 2024-09-13T14:54:56.124 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.125 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.125 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:54:56.125 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:05.945256+0000", 2024-09-13T14:54:56.125 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1396000000000001e-05 2024-09-13T14:54:56.125 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.125 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.125 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:54:56.125 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:05.945262+0000", 2024-09-13T14:54:56.125 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1720000000000004e-06 2024-09-13T14:54:56.125 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.125 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.125 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:54:56.125 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:05.945302+0000", 2024-09-13T14:54:56.125 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0321999999999998e-05 2024-09-13T14:54:56.126 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.126 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.126 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:54:56.126 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:05.945399+0000", 2024-09-13T14:54:56.126 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.7164999999999996e-05 2024-09-13T14:54:56.126 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.126 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.126 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:54:56.126 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:05.945500+0000", 2024-09-13T14:54:56.126 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000100665 2024-09-13T14:54:56.126 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.126 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.126 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:54:56.126 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:05.948209+0000", 2024-09-13T14:54:56.127 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0027086990000000002 2024-09-13T14:54:56.127 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.127 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.127 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.127 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:05.948336+0000", 2024-09-13T14:54:56.127 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000127646 2024-09-13T14:54:56.127 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.127 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.127 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.127 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:05.949320+0000", 2024-09-13T14:54:56.127 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00098333699999999993 2024-09-13T14:54:56.127 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.127 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.127 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.128 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:06.095030+0000", 2024-09-13T14:54:56.128 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.14570997099999999 2024-09-13T14:54:56.128 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.128 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.128 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:54:56.128 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:06.095085+0000", 2024-09-13T14:54:56.128 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5194e-05 2024-09-13T14:54:56.128 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.128 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.128 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:54:56.128 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:46:06.095207+0000", 2024-09-13T14:54:56.128 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000122366 2024-09-13T14:54:56.128 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.129 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:54:56.129 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.129 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.129 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.129 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:195549 7.1 7:828894fe:::benchmark_data_smithi151_190906_object75088:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1160)", 2024-09-13T14:54:56.129 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:47:34.962584+0000", 2024-09-13T14:54:56.129 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 441.14235323499997, 2024-09-13T14:54:56.129 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.17580900499999999, 2024-09-13T14:54:56.129 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:54:56.129 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:56.129 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:54:56.129 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:54:56.129 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:54:56.130 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 195549 2024-09-13T14:54:56.130 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.130 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:54:56.130 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.130 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:54:56.130 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.962584+0000", 2024-09-13T14:54:56.130 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.130 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.130 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.130 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:54:56.130 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.962584+0000", 2024-09-13T14:54:56.130 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.130 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.131 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.131 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:54:56.131 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.962583+0000", 2024-09-13T14:54:56.131 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:56.131 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.131 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.131 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:54:56.131 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.962591+0000", 2024-09-13T14:54:56.131 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0269999999999995e-06 2024-09-13T14:54:56.131 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.131 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.131 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:54:56.131 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.962609+0000", 2024-09-13T14:54:56.131 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.8193999999999999e-05 2024-09-13T14:54:56.132 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.132 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.132 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:54:56.132 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.962617+0000", 2024-09-13T14:54:56.132 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.6190000000000002e-06 2024-09-13T14:54:56.132 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.132 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.132 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:54:56.132 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.962670+0000", 2024-09-13T14:54:56.132 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3047000000000002e-05 2024-09-13T14:54:56.133 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.133 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.133 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:54:56.133 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.962815+0000", 2024-09-13T14:54:56.133 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014494900000000001 2024-09-13T14:54:56.133 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.133 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.133 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:54:56.133 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.962973+0000", 2024-09-13T14:54:56.133 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000158252 2024-09-13T14:54:56.133 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.133 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.133 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.134 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.966688+0000", 2024-09-13T14:54:56.134 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0037152550000000002 2024-09-13T14:54:56.134 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.134 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.134 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:54:56.134 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.968210+0000", 2024-09-13T14:54:56.134 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001522277 2024-09-13T14:54:56.134 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.134 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.134 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.134 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.969011+0000", 2024-09-13T14:54:56.134 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00080025400000000003 2024-09-13T14:54:56.134 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.135 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.135 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.135 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:35.138207+0000", 2024-09-13T14:54:56.135 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.169196717 2024-09-13T14:54:56.135 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.135 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.135 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:54:56.135 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:35.138265+0000", 2024-09-13T14:54:56.135 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7268000000000001e-05 2024-09-13T14:54:56.135 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.135 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.135 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:54:56.135 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:35.138393+0000", 2024-09-13T14:54:56.135 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012880500000000001 2024-09-13T14:54:56.136 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.136 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:54:56.136 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.136 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.136 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.136 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:195563 7.1 7:8ea4174d:::benchmark_data_smithi151_190906_object75102:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1160)", 2024-09-13T14:54:56.136 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:47:34.981199+0000", 2024-09-13T14:54:56.136 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 441.123738699, 2024-09-13T14:54:56.136 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.161925608, 2024-09-13T14:54:56.136 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:54:56.136 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:56.136 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:54:56.136 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:54:56.137 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:54:56.137 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 195563 2024-09-13T14:54:56.137 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.137 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:54:56.137 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.137 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:54:56.137 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.981199+0000", 2024-09-13T14:54:56.137 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.137 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.137 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.137 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:54:56.137 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.981199+0000", 2024-09-13T14:54:56.137 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.138 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.138 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.138 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:54:56.138 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.981197+0000", 2024-09-13T14:54:56.138 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:54:56.138 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.138 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.138 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:54:56.138 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.981204+0000", 2024-09-13T14:54:56.138 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.5050000000000001e-06 2024-09-13T14:54:56.138 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.138 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.138 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:54:56.139 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.981218+0000", 2024-09-13T14:54:56.139 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4e-05 2024-09-13T14:54:56.139 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.139 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.139 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:54:56.139 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.981224+0000", 2024-09-13T14:54:56.139 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.964e-06 2024-09-13T14:54:56.139 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.139 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.139 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:54:56.139 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.981290+0000", 2024-09-13T14:54:56.139 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.5370999999999995e-05 2024-09-13T14:54:56.139 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.139 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.139 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:54:56.140 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.981453+0000", 2024-09-13T14:54:56.140 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00016340899999999999 2024-09-13T14:54:56.140 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.140 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.140 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:54:56.140 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.981553+0000", 2024-09-13T14:54:56.140 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.9429e-05 2024-09-13T14:54:56.140 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.140 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.140 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:54:56.140 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.984591+0000", 2024-09-13T14:54:56.140 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00303857 2024-09-13T14:54:56.140 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.140 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.141 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.141 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.985474+0000", 2024-09-13T14:54:56.141 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00088276399999999997 2024-09-13T14:54:56.141 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.141 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.141 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.141 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:34.986705+0000", 2024-09-13T14:54:56.141 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001230734 2024-09-13T14:54:56.141 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.141 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.141 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.141 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:35.142938+0000", 2024-09-13T14:54:56.141 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.15623298099999999 2024-09-13T14:54:56.142 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.142 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.142 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:54:56.142 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:35.142994+0000", 2024-09-13T14:54:56.142 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6759000000000003e-05 2024-09-13T14:54:56.142 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.142 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.142 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:54:56.142 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:35.143124+0000", 2024-09-13T14:54:56.142 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000130025 2024-09-13T14:54:56.142 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.142 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:54:56.142 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.142 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.143 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.143 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:196368 7.1 7:8b4e4f2b:::benchmark_data_smithi151_190906_object75907:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1160)", 2024-09-13T14:54:56.143 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:47:36.664845+0000", 2024-09-13T14:54:56.143 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 439.44009265, 2024-09-13T14:54:56.143 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.14392316499999999, 2024-09-13T14:54:56.143 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:54:56.143 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:56.143 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:54:56.143 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:54:56.143 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:54:56.143 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 196368 2024-09-13T14:54:56.143 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.143 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:54:56.144 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.144 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:54:56.144 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:36.664845+0000", 2024-09-13T14:54:56.144 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.144 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.144 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.144 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:54:56.144 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:36.664845+0000", 2024-09-13T14:54:56.144 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.144 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.144 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.144 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:54:56.144 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:36.664843+0000", 2024-09-13T14:54:56.145 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:54:56.145 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.145 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.145 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:54:56.145 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:36.664852+0000", 2024-09-13T14:54:56.145 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.4200000000000007e-06 2024-09-13T14:54:56.145 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.145 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.145 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:54:56.145 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:36.664867+0000", 2024-09-13T14:54:56.145 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4916000000000001e-05 2024-09-13T14:54:56.145 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.145 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.145 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:54:56.146 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:36.664885+0000", 2024-09-13T14:54:56.146 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.8802999999999999e-05 2024-09-13T14:54:56.146 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.146 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.146 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:54:56.146 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:36.664930+0000", 2024-09-13T14:54:56.146 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4440999999999997e-05 2024-09-13T14:54:56.146 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.146 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.146 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:54:56.146 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:36.665054+0000", 2024-09-13T14:54:56.146 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012421700000000001 2024-09-13T14:54:56.146 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.146 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.147 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:54:56.147 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:36.665142+0000", 2024-09-13T14:54:56.147 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.8202999999999996e-05 2024-09-13T14:54:56.147 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.147 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.147 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:54:56.147 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:36.667672+0000", 2024-09-13T14:54:56.147 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0025300790000000002 2024-09-13T14:54:56.147 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.147 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.147 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.147 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:36.667916+0000", 2024-09-13T14:54:56.147 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00024384799999999999 2024-09-13T14:54:56.147 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.148 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.148 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.148 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:36.668228+0000", 2024-09-13T14:54:56.148 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00031239199999999999 2024-09-13T14:54:56.148 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.148 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.148 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.148 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:36.808631+0000", 2024-09-13T14:54:56.148 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.14040271900000001 2024-09-13T14:54:56.148 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.148 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.148 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:54:56.148 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:36.808675+0000", 2024-09-13T14:54:56.148 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4045000000000003e-05 2024-09-13T14:54:56.149 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.149 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.149 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:54:56.149 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:36.808768+0000", 2024-09-13T14:54:56.149 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.2825999999999995e-05 2024-09-13T14:54:56.149 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.149 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:54:56.149 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.149 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.149 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.149 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:198497 7.1 7:8a05d87f:::benchmark_data_smithi151_190906_object78036:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1160)", 2024-09-13T14:54:56.149 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:47:40.151110+0000", 2024-09-13T14:54:56.149 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 435.95382709400002, 2024-09-13T14:54:56.149 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.35196765099999999, 2024-09-13T14:54:56.150 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:54:56.150 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:56.150 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:54:56.150 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:54:56.150 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:54:56.150 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 198497 2024-09-13T14:54:56.150 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.150 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:54:56.150 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.150 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:54:56.150 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.151110+0000", 2024-09-13T14:54:56.150 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.150 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.150 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.150 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:54:56.151 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.151110+0000", 2024-09-13T14:54:56.151 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.151 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.151 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.151 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:54:56.151 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.151109+0000", 2024-09-13T14:54:56.151 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:54:56.151 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.151 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.151 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:54:56.151 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.151116+0000", 2024-09-13T14:54:56.151 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.5180000000000001e-06 2024-09-13T14:54:56.151 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.151 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.151 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:54:56.152 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.151129+0000", 2024-09-13T14:54:56.152 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3219e-05 2024-09-13T14:54:56.152 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.152 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.152 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:54:56.152 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.151135+0000", 2024-09-13T14:54:56.152 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.2330000000000003e-06 2024-09-13T14:54:56.152 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.152 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.152 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:54:56.152 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.151180+0000", 2024-09-13T14:54:56.152 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4542000000000001e-05 2024-09-13T14:54:56.152 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.152 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.153 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:54:56.153 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.151270+0000", 2024-09-13T14:54:56.153 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.0049000000000005e-05 2024-09-13T14:54:56.153 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.153 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.153 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:54:56.153 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.151350+0000", 2024-09-13T14:54:56.153 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.9717999999999998e-05 2024-09-13T14:54:56.153 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.153 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.153 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.153 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.154133+0000", 2024-09-13T14:54:56.153 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0027833559999999999 2024-09-13T14:54:56.153 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.153 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.154 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:54:56.154 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.154148+0000", 2024-09-13T14:54:56.154 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.5121e-05 2024-09-13T14:54:56.154 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.154 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.154 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.154 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.160489+0000", 2024-09-13T14:54:56.154 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0063409779999999997 2024-09-13T14:54:56.154 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.154 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.154 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.154 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.502943+0000", 2024-09-13T14:54:56.154 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.34245401399999997 2024-09-13T14:54:56.155 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.155 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.155 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:54:56.155 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.502983+0000", 2024-09-13T14:54:56.155 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9666999999999998e-05 2024-09-13T14:54:56.155 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.155 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.155 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:54:56.155 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:47:40.503078+0000", 2024-09-13T14:54:56.155 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.5376000000000003e-05 2024-09-13T14:54:56.155 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.155 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:54:56.155 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.155 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.155 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.156 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:214311 7.1 7:8d2f8dc9:::benchmark_data_smithi151_190906_object93850:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1162)", 2024-09-13T14:54:56.156 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:48:06.281766+0000", 2024-09-13T14:54:56.156 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 409.82317106900001, 2024-09-13T14:54:56.156 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.22899525900000001, 2024-09-13T14:54:56.156 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:54:56.156 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:56.156 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:54:56.156 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:54:56.156 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:54:56.156 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 214311 2024-09-13T14:54:56.156 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.156 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:54:56.156 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.156 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:54:56.157 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:06.281766+0000", 2024-09-13T14:54:56.157 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.157 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.157 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.157 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:54:56.157 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:06.281766+0000", 2024-09-13T14:54:56.157 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.157 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.157 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.157 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:54:56.157 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:06.281764+0000", 2024-09-13T14:54:56.157 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999976 2024-09-13T14:54:56.157 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.158 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.158 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:54:56.158 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:06.281777+0000", 2024-09-13T14:54:56.158 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2425999999999999e-05 2024-09-13T14:54:56.158 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.158 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.158 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:54:56.158 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:06.281806+0000", 2024-09-13T14:54:56.158 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.9430000000000001e-05 2024-09-13T14:54:56.158 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.158 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.158 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:54:56.158 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:06.281817+0000", 2024-09-13T14:54:56.159 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1146e-05 2024-09-13T14:54:56.159 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.159 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.159 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:54:56.159 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:06.281936+0000", 2024-09-13T14:54:56.159 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000118638 2024-09-13T14:54:56.159 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.159 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.159 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:54:56.159 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:06.282112+0000", 2024-09-13T14:54:56.159 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000176427 2024-09-13T14:54:56.159 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.159 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.159 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:54:56.159 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:06.282215+0000", 2024-09-13T14:54:56.160 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000102833 2024-09-13T14:54:56.160 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.160 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.160 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:54:56.160 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:06.284660+0000", 2024-09-13T14:54:56.160 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0024443749999999999 2024-09-13T14:54:56.160 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.160 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.160 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.160 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:06.284945+0000", 2024-09-13T14:54:56.160 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000285548 2024-09-13T14:54:56.160 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.160 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.160 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.161 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:06.284986+0000", 2024-09-13T14:54:56.161 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0745999999999999e-05 2024-09-13T14:54:56.161 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.161 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.161 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.161 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:06.510584+0000", 2024-09-13T14:54:56.161 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.225598294 2024-09-13T14:54:56.161 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.161 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.161 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:54:56.161 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:06.510640+0000", 2024-09-13T14:54:56.161 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5458999999999999e-05 2024-09-13T14:54:56.161 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.161 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.162 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:54:56.162 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:06.510762+0000", 2024-09-13T14:54:56.162 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012219999999999999 2024-09-13T14:54:56.162 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.162 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:54:56.162 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.162 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.162 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.162 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:214937 7.1 7:8f95f2cd:::benchmark_data_smithi151_190906_object94476:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1162)", 2024-09-13T14:54:56.162 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:48:07.619610+0000", 2024-09-13T14:54:56.162 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 408.48532739900003, 2024-09-13T14:54:56.162 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.21546647699999999, 2024-09-13T14:54:56.162 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:54:56.162 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:56.163 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:54:56.163 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:54:56.163 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:54:56.163 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 214937 2024-09-13T14:54:56.163 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.163 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:54:56.163 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.163 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:54:56.163 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:07.619610+0000", 2024-09-13T14:54:56.163 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.163 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.163 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.163 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:54:56.163 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:07.619610+0000", 2024-09-13T14:54:56.163 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.164 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.164 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.164 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:54:56.164 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:07.619609+0000", 2024-09-13T14:54:56.164 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:56.164 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.164 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.164 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:54:56.164 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:07.619617+0000", 2024-09-13T14:54:56.164 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.841e-06 2024-09-13T14:54:56.164 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.164 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.164 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:54:56.164 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:07.619629+0000", 2024-09-13T14:54:56.165 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1901e-05 2024-09-13T14:54:56.166 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.166 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.166 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:54:56.166 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:07.619634+0000", 2024-09-13T14:54:56.166 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3489999999999999e-06 2024-09-13T14:54:56.166 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.166 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.166 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:54:56.166 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:07.619684+0000", 2024-09-13T14:54:56.166 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9642999999999999e-05 2024-09-13T14:54:56.166 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.166 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.166 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:54:56.166 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:07.619826+0000", 2024-09-13T14:54:56.167 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000142232 2024-09-13T14:54:56.167 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.167 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.167 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:54:56.167 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:07.619974+0000", 2024-09-13T14:54:56.167 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000148426 2024-09-13T14:54:56.167 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.167 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.167 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.167 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:07.622857+0000", 2024-09-13T14:54:56.167 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002882431 2024-09-13T14:54:56.167 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.167 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.167 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:54:56.168 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:07.623093+0000", 2024-09-13T14:54:56.168 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000236314 2024-09-13T14:54:56.168 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.168 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.168 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.168 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:07.626337+0000", 2024-09-13T14:54:56.168 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0032440960000000001 2024-09-13T14:54:56.168 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.168 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.168 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.168 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:07.834952+0000", 2024-09-13T14:54:56.168 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.20861507600000001 2024-09-13T14:54:56.168 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.169 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.169 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:54:56.169 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:07.834992+0000", 2024-09-13T14:54:56.169 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9854000000000002e-05 2024-09-13T14:54:56.169 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.169 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.169 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:54:56.169 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:07.835077+0000", 2024-09-13T14:54:56.169 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.4512999999999995e-05 2024-09-13T14:54:56.169 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.169 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:54:56.169 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.169 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.169 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.170 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:218140 7.1 7:888097b9:::benchmark_data_smithi151_190906_object97679:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1162)", 2024-09-13T14:54:56.170 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:48:13.176488+0000", 2024-09-13T14:54:56.170 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 402.92844955100003, 2024-09-13T14:54:56.170 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.22006838400000001, 2024-09-13T14:54:56.170 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:54:56.170 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:56.170 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:54:56.170 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:54:56.170 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:54:56.170 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 218140 2024-09-13T14:54:56.170 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.170 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:54:56.170 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.170 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:54:56.170 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.176488+0000", 2024-09-13T14:54:56.171 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.171 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.171 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.171 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:54:56.171 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.176488+0000", 2024-09-13T14:54:56.171 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.171 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.171 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.171 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:54:56.171 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.176486+0000", 2024-09-13T14:54:56.171 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999976 2024-09-13T14:54:56.171 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.171 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.171 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:54:56.171 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.176496+0000", 2024-09-13T14:54:56.172 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0445000000000001e-05 2024-09-13T14:54:56.172 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.172 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.172 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:54:56.172 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.176511+0000", 2024-09-13T14:54:56.172 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4611e-05 2024-09-13T14:54:56.172 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.172 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.172 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:54:56.172 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.176518+0000", 2024-09-13T14:54:56.172 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.3329999999999999e-06 2024-09-13T14:54:56.172 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.172 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.172 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:54:56.173 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.176565+0000", 2024-09-13T14:54:56.173 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.6823000000000002e-05 2024-09-13T14:54:56.173 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.173 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.173 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:54:56.173 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.176731+0000", 2024-09-13T14:54:56.173 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00016566399999999999 2024-09-13T14:54:56.173 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.173 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.173 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:54:56.173 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.176895+0000", 2024-09-13T14:54:56.173 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00016419199999999999 2024-09-13T14:54:56.173 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.173 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.174 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.174 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.183503+0000", 2024-09-13T14:54:56.174 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0066076640000000001 2024-09-13T14:54:56.174 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.174 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.174 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:54:56.174 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.183511+0000", 2024-09-13T14:54:56.174 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.2979999999999992e-06 2024-09-13T14:54:56.174 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.174 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.174 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.174 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.183548+0000", 2024-09-13T14:54:56.174 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.7453000000000002e-05 2024-09-13T14:54:56.174 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.175 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.175 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.175 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.396308+0000", 2024-09-13T14:54:56.175 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.21275965199999999 2024-09-13T14:54:56.175 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.175 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.175 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:54:56.175 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.396437+0000", 2024-09-13T14:54:56.175 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012852800000000001 2024-09-13T14:54:56.175 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.175 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.175 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:54:56.175 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.396556+0000", 2024-09-13T14:54:56.176 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000119867 2024-09-13T14:54:56.176 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.176 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:54:56.176 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.176 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.176 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.176 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:218143 7.1 7:82685e6a:::benchmark_data_smithi151_190906_object97682:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1162)", 2024-09-13T14:54:56.176 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:48:13.178718+0000", 2024-09-13T14:54:56.176 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 402.92621973299998, 2024-09-13T14:54:56.176 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.22125558100000001, 2024-09-13T14:54:56.176 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:54:56.176 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:56.176 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:54:56.176 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:54:56.177 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:54:56.177 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 218143 2024-09-13T14:54:56.177 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.177 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:54:56.177 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.177 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:54:56.177 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.178718+0000", 2024-09-13T14:54:56.177 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.177 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.177 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.177 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:54:56.177 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.178718+0000", 2024-09-13T14:54:56.177 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.178 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.178 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.178 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:54:56.178 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.178717+0000", 2024-09-13T14:54:56.178 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:54:56.178 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.178 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.178 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:54:56.178 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.178727+0000", 2024-09-13T14:54:56.178 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.9329999999999996e-06 2024-09-13T14:54:56.178 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.178 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.178 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:54:56.178 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.178740+0000", 2024-09-13T14:54:56.178 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3879999999999999e-05 2024-09-13T14:54:56.179 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.179 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.179 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:54:56.179 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.178757+0000", 2024-09-13T14:54:56.179 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6286999999999998e-05 2024-09-13T14:54:56.179 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.179 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.179 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:54:56.179 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.180149+0000", 2024-09-13T14:54:56.179 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0013918419999999999 2024-09-13T14:54:56.179 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.179 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.179 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:54:56.180 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.180229+0000", 2024-09-13T14:54:56.180 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0774999999999998e-05 2024-09-13T14:54:56.180 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.180 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.180 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:54:56.180 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.180311+0000", 2024-09-13T14:54:56.180 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.1723000000000003e-05 2024-09-13T14:54:56.180 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.180 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.180 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.180 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.183576+0000", 2024-09-13T14:54:56.180 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0032653869999999998 2024-09-13T14:54:56.180 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.181 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.181 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:54:56.181 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.183585+0000", 2024-09-13T14:54:56.181 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.8429999999999999e-06 2024-09-13T14:54:56.181 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.181 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.181 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.181 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.183735+0000", 2024-09-13T14:54:56.181 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014961600000000001 2024-09-13T14:54:56.181 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.181 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.181 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.181 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.399781+0000", 2024-09-13T14:54:56.181 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.21604623100000001 2024-09-13T14:54:56.182 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.182 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.182 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:54:56.182 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.399835+0000", 2024-09-13T14:54:56.182 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3831e-05 2024-09-13T14:54:56.182 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.182 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.182 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:54:56.182 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:13.399973+0000", 2024-09-13T14:54:56.182 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000138398 2024-09-13T14:54:56.182 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.182 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:54:56.182 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.182 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.182 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.183 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:219554 7.1 7:878273f9:::benchmark_data_smithi151_190906_object99093:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1162)", 2024-09-13T14:54:56.183 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:48:15.671473+0000", 2024-09-13T14:54:56.183 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 400.43346472399998, 2024-09-13T14:54:56.183 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.18064495799999999, 2024-09-13T14:54:56.183 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:54:56.183 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:56.183 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:54:56.183 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:54:56.183 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:54:56.183 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 219554 2024-09-13T14:54:56.183 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.183 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:54:56.183 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.183 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:54:56.183 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.671473+0000", 2024-09-13T14:54:56.184 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.184 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.184 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.184 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:54:56.184 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.671473+0000", 2024-09-13T14:54:56.184 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.184 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.184 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.184 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:54:56.184 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.671472+0000", 2024-09-13T14:54:56.184 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:54:56.184 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.184 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.184 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:54:56.185 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.671478+0000", 2024-09-13T14:54:56.185 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.9089999999999997e-06 2024-09-13T14:54:56.185 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.185 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.185 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:54:56.185 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.671489+0000", 2024-09-13T14:54:56.185 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.152e-05 2024-09-13T14:54:56.185 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.185 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.185 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:54:56.185 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.671493+0000", 2024-09-13T14:54:56.185 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3499999999999999e-06 2024-09-13T14:54:56.185 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.185 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.186 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:54:56.186 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.671542+0000", 2024-09-13T14:54:56.186 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.863e-05 2024-09-13T14:54:56.186 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.186 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.186 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:54:56.186 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.671688+0000", 2024-09-13T14:54:56.186 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000145403 2024-09-13T14:54:56.186 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.186 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.186 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:54:56.186 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.671847+0000", 2024-09-13T14:54:56.186 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000159576 2024-09-13T14:54:56.187 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.187 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.187 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.187 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.675007+0000", 2024-09-13T14:54:56.187 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0031600040000000001 2024-09-13T14:54:56.187 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.187 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.187 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:54:56.187 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.675201+0000", 2024-09-13T14:54:56.187 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00019353799999999999 2024-09-13T14:54:56.187 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.187 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.187 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.187 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.683113+0000", 2024-09-13T14:54:56.188 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0079122959999999992 2024-09-13T14:54:56.188 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.188 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.188 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.188 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.851973+0000", 2024-09-13T14:54:56.188 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.16886020500000001 2024-09-13T14:54:56.188 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.188 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.188 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:54:56.188 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.852012+0000", 2024-09-13T14:54:56.188 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9106999999999997e-05 2024-09-13T14:54:56.188 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.188 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.188 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:54:56.188 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.852118+0000", 2024-09-13T14:54:56.189 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00010555799999999999 2024-09-13T14:54:56.189 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.189 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:54:56.189 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.189 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.189 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.189 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:219556 7.1 7:8cd60c5c:::benchmark_data_smithi151_190906_object99095:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1162)", 2024-09-13T14:54:56.189 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:48:15.675894+0000", 2024-09-13T14:54:56.189 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 400.429043165, 2024-09-13T14:54:56.189 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.18125966499999999, 2024-09-13T14:54:56.189 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:54:56.189 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:56.189 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:54:56.189 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:54:56.189 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:54:56.190 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 219556 2024-09-13T14:54:56.190 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.190 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:54:56.190 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.190 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:54:56.190 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.675894+0000", 2024-09-13T14:54:56.190 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.190 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.190 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.190 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:54:56.190 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.675894+0000", 2024-09-13T14:54:56.190 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.190 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.190 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.190 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:54:56.191 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.675893+0000", 2024-09-13T14:54:56.191 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:56.191 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.191 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.191 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:54:56.191 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.675903+0000", 2024-09-13T14:54:56.191 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0209e-05 2024-09-13T14:54:56.191 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.191 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.191 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:54:56.191 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.675917+0000", 2024-09-13T14:54:56.191 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3592e-05 2024-09-13T14:54:56.191 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.191 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.192 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:54:56.192 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.675923+0000", 2024-09-13T14:54:56.192 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.0310000000000004e-06 2024-09-13T14:54:56.192 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.192 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.192 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:54:56.192 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.675966+0000", 2024-09-13T14:54:56.192 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.2697999999999997e-05 2024-09-13T14:54:56.192 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.192 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.192 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:54:56.192 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.676101+0000", 2024-09-13T14:54:56.192 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013541900000000001 2024-09-13T14:54:56.192 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.193 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.193 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:54:56.193 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.676251+0000", 2024-09-13T14:54:56.193 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000150321 2024-09-13T14:54:56.193 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.193 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.193 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.193 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.679360+0000", 2024-09-13T14:54:56.193 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.003108446 2024-09-13T14:54:56.193 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.193 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.193 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:54:56.193 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.679376+0000", 2024-09-13T14:54:56.193 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6490999999999998e-05 2024-09-13T14:54:56.193 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.194 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.194 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.194 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.683782+0000", 2024-09-13T14:54:56.194 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0044055079999999998 2024-09-13T14:54:56.194 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.194 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.194 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.194 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.856970+0000", 2024-09-13T14:54:56.194 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.17318771999999999 2024-09-13T14:54:56.194 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.194 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.194 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:54:56.194 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.857024+0000", 2024-09-13T14:54:56.194 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4577000000000003e-05 2024-09-13T14:54:56.194 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.195 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.195 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:54:56.195 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.857154+0000", 2024-09-13T14:54:56.195 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000129949 2024-09-13T14:54:56.195 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.195 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:54:56.195 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.195 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.195 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.195 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:219564 7.1 7:8f49224e:::benchmark_data_smithi151_190906_object99103:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1162)", 2024-09-13T14:54:56.195 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:48:15.684515+0000", 2024-09-13T14:54:56.195 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 400.420422159, 2024-09-13T14:54:56.195 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.17284477100000001, 2024-09-13T14:54:56.195 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:54:56.196 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:56.196 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:54:56.196 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:54:56.196 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:54:56.196 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 219564 2024-09-13T14:54:56.196 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.196 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:54:56.196 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.196 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:54:56.196 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.684515+0000", 2024-09-13T14:54:56.196 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.196 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.196 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.196 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:54:56.197 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.684515+0000", 2024-09-13T14:54:56.197 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.197 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.197 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.197 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:54:56.197 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.684514+0000", 2024-09-13T14:54:56.197 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:56.197 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.197 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.197 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:54:56.197 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.684521+0000", 2024-09-13T14:54:56.197 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.9929999999999998e-06 2024-09-13T14:54:56.197 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.198 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.198 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:54:56.198 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.684533+0000", 2024-09-13T14:54:56.198 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2046e-05 2024-09-13T14:54:56.198 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.198 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.198 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:54:56.198 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.684539+0000", 2024-09-13T14:54:56.198 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.268e-06 2024-09-13T14:54:56.198 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.198 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.198 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:54:56.198 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.684583+0000", 2024-09-13T14:54:56.198 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3494000000000001e-05 2024-09-13T14:54:56.198 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.199 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.199 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:54:56.199 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.684717+0000", 2024-09-13T14:54:56.199 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000134566 2024-09-13T14:54:56.199 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.199 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.199 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:54:56.199 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.684854+0000", 2024-09-13T14:54:56.199 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013663400000000001 2024-09-13T14:54:56.199 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.199 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.199 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.199 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.688009+0000", 2024-09-13T14:54:56.199 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.003155153 2024-09-13T14:54:56.199 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.200 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.200 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:54:56.200 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.688100+0000", 2024-09-13T14:54:56.200 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.0968e-05 2024-09-13T14:54:56.200 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.200 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.200 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.200 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.688293+0000", 2024-09-13T14:54:56.200 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00019281100000000001 2024-09-13T14:54:56.200 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.200 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.200 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.200 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.857198+0000", 2024-09-13T14:54:56.200 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.16890530500000001 2024-09-13T14:54:56.201 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.201 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.201 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:54:56.201 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.857251+0000", 2024-09-13T14:54:56.201 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.2417999999999999e-05 2024-09-13T14:54:56.201 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.201 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.201 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:54:56.201 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:15.857360+0000", 2024-09-13T14:54:56.201 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000109527 2024-09-13T14:54:56.201 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.201 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:54:56.201 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.201 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.202 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.202 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:239454 7.1 7:84cced3c:::benchmark_data_smithi151_190906_object118993:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1164)", 2024-09-13T14:54:56.202 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:48:48.828811+0000", 2024-09-13T14:54:56.202 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 367.276126218, 2024-09-13T14:54:56.202 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.129271951, 2024-09-13T14:54:56.202 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:54:56.202 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:56.202 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:54:56.202 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:54:56.202 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:54:56.202 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 239454 2024-09-13T14:54:56.202 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.202 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:54:56.202 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.202 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:54:56.203 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:48.828811+0000", 2024-09-13T14:54:56.203 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.203 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.203 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.203 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:54:56.203 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:48.828811+0000", 2024-09-13T14:54:56.203 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.203 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.203 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.203 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:54:56.203 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:48.828810+0000", 2024-09-13T14:54:56.203 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:56.203 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.203 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.203 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:54:56.204 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:48.828816+0000", 2024-09-13T14:54:56.204 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3759999999999996e-06 2024-09-13T14:54:56.204 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.204 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.204 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:54:56.204 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:48.828828+0000", 2024-09-13T14:54:56.204 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1289e-05 2024-09-13T14:54:56.204 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.204 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.204 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:54:56.204 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:48.828834+0000", 2024-09-13T14:54:56.204 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1190000000000002e-06 2024-09-13T14:54:56.204 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.204 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.205 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:54:56.205 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:48.830811+0000", 2024-09-13T14:54:56.205 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001977389 2024-09-13T14:54:56.205 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.205 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.205 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:54:56.205 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:48.830924+0000", 2024-09-13T14:54:56.205 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000112491 2024-09-13T14:54:56.205 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.205 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.205 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:54:56.205 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:48.831006+0000", 2024-09-13T14:54:56.205 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.1861999999999997e-05 2024-09-13T14:54:56.205 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.205 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.206 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:54:56.206 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:48.833720+0000", 2024-09-13T14:54:56.206 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0027140549999999999 2024-09-13T14:54:56.206 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.206 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.206 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.206 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:48.834270+0000", 2024-09-13T14:54:56.206 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00055069500000000003 2024-09-13T14:54:56.206 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.206 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.206 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.206 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:48.834682+0000", 2024-09-13T14:54:56.206 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00041196100000000002 2024-09-13T14:54:56.207 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.207 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.207 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.207 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:48.957932+0000", 2024-09-13T14:54:56.207 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.123249452 2024-09-13T14:54:56.207 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.207 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.207 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:54:56.207 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:48.957988+0000", 2024-09-13T14:54:56.207 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6354999999999998e-05 2024-09-13T14:54:56.207 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.207 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.207 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:54:56.207 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:48.958083+0000", 2024-09-13T14:54:56.207 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.5134000000000001e-05 2024-09-13T14:54:56.208 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.208 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:54:56.208 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.208 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.208 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.208 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.45494.0:240176 7.1 7:8fe24462:::benchmark_data_smithi151_190906_object119715:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1164)", 2024-09-13T14:54:56.208 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:48:50.078607+0000", 2024-09-13T14:54:56.208 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 366.02633030300001, 2024-09-13T14:54:56.208 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.14877069000000001, 2024-09-13T14:54:56.208 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:54:56.208 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:56.208 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:54:56.208 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.45494", 2024-09-13T14:54:56.208 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3078772316", 2024-09-13T14:54:56.209 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 240176 2024-09-13T14:54:56.209 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.209 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:54:56.209 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.209 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:54:56.209 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:50.078607+0000", 2024-09-13T14:54:56.209 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.209 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.209 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.209 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:54:56.209 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:50.078607+0000", 2024-09-13T14:54:56.209 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.209 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.209 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.210 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:54:56.210 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:50.078606+0000", 2024-09-13T14:54:56.210 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:56.210 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.210 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.210 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:54:56.210 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:50.078613+0000", 2024-09-13T14:54:56.210 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.2810000000000003e-06 2024-09-13T14:54:56.210 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.210 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.210 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:54:56.210 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:50.078626+0000", 2024-09-13T14:54:56.210 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3178e-05 2024-09-13T14:54:56.210 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.211 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.211 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:54:56.211 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:50.078631+0000", 2024-09-13T14:54:56.211 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.074e-06 2024-09-13T14:54:56.211 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.212 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.212 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:54:56.212 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:50.078685+0000", 2024-09-13T14:54:56.212 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3054999999999999e-05 2024-09-13T14:54:56.212 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.213 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.213 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:54:56.213 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:50.078780+0000", 2024-09-13T14:54:56.213 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.4982000000000006e-05 2024-09-13T14:54:56.213 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.213 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.213 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:54:56.213 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:50.078867+0000", 2024-09-13T14:54:56.213 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.7211000000000002e-05 2024-09-13T14:54:56.213 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.213 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.213 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:54:56.213 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:50.081423+0000", 2024-09-13T14:54:56.213 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002556647 2024-09-13T14:54:56.213 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.214 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.214 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.214 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:50.081514+0000", 2024-09-13T14:54:56.214 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.0669999999999998e-05 2024-09-13T14:54:56.214 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.214 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.214 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.214 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:50.081625+0000", 2024-09-13T14:54:56.214 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000111047 2024-09-13T14:54:56.214 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.214 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.214 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.214 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:50.227273+0000", 2024-09-13T14:54:56.214 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.14564764399999999 2024-09-13T14:54:56.214 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.215 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.215 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:54:56.215 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:50.227309+0000", 2024-09-13T14:54:56.215 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.6434999999999999e-05 2024-09-13T14:54:56.215 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.215 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.215 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:54:56.215 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:48:50.227378+0000", 2024-09-13T14:54:56.215 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.8786000000000002e-05 2024-09-13T14:54:56.215 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.215 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:54:56.215 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.215 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.215 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.215 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:42827 8.2 8:4433e755:::benchmark_data_smithi151_229097_object42826:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1172)", 2024-09-13T14:54:56.216 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:49:57.713598+0000", 2024-09-13T14:54:56.216 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 298.39133978699999, 2024-09-13T14:54:56.216 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.13415065700000001, 2024-09-13T14:54:56.216 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:54:56.216 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:56.216 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:54:56.216 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:54:56.216 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:54:56.216 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 42827 2024-09-13T14:54:56.216 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.216 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:54:56.216 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.216 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:54:56.216 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.713598+0000", 2024-09-13T14:54:56.217 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.217 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.217 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.217 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:54:56.217 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.713598+0000", 2024-09-13T14:54:56.217 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:54:56.217 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.217 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.217 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:54:56.217 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.713596+0000", 2024-09-13T14:54:56.217 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:56.217 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.217 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.217 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:54:56.217 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.713638+0000", 2024-09-13T14:54:56.218 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1916000000000003e-05 2024-09-13T14:54:56.218 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.218 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.218 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:54:56.218 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.713649+0000", 2024-09-13T14:54:56.218 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1114000000000001e-05 2024-09-13T14:54:56.218 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.218 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.218 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:54:56.218 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.713654+0000", 2024-09-13T14:54:56.218 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0879999999999997e-06 2024-09-13T14:54:56.218 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.218 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.219 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:54:56.219 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.713687+0000", 2024-09-13T14:54:56.219 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3478e-05 2024-09-13T14:54:56.219 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.219 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.219 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:54:56.219 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.713779+0000", 2024-09-13T14:54:56.219 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.2367999999999994e-05 2024-09-13T14:54:56.219 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.219 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.219 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:54:56.219 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.713880+0000", 2024-09-13T14:54:56.219 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000101043 2024-09-13T14:54:56.219 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.219 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.220 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:54:56.220 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.715459+0000", 2024-09-13T14:54:56.220 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001578238 2024-09-13T14:54:56.220 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.220 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.220 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.220 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.715892+0000", 2024-09-13T14:54:56.220 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000433485 2024-09-13T14:54:56.220 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.220 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.220 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:56.220 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.847624+0000", 2024-09-13T14:54:56.220 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.13173138700000001 2024-09-13T14:54:56.221 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.221 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.221 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:54:56.221 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.847712+0000", 2024-09-13T14:54:56.221 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.8432999999999998e-05 2024-09-13T14:54:56.221 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:54:56.221 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:54:56.221 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:54:56.221 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.847748+0000", 2024-09-13T14:54:56.221 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.6426e-05 2024-09-13T14:54:56.221 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.221 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:54:56.221 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.221 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:54:56.222 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:54:56.222 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:54:56.222 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:56.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:56.264 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:56.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:56.365 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:56.401 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:56.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:56.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:56.503 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:56.543 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:56.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:56.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:56.645 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:56.686 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:56.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:56.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:56.788 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:56.816 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:56.818 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:56.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:56.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:56.920 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:56.962 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 37 16 66763 66747 112.729 79.875 0.0105728 0.00886519 2024-09-13T14:54:56.965 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:56 smithi151 ceph-mon[104450]: pgmap v1898: 119 pgs: 119 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-09-13T14:54:56.966 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:56 smithi151 ceph-mon[108966]: pgmap v1898: 119 pgs: 119 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-09-13T14:54:56.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:56.967 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:56.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:57.067 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:57.094 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:57.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:57.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:57.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:56 smithi184 ceph-mon[92527]: pgmap v1898: 119 pgs: 119 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-09-13T14:54:57.196 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:57.239 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:57.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:57.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:57.341 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:57.366 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:57.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:57.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:57.468 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:57.498 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:57.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:57.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:57.600 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:57.601 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:54:57.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:57 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a[104427]: 2024-09-13T14:54:57.223+0000 7fe9da14f640 -1 mon.a@0(leader).osd e1227 definitely_dead 0 2024-09-13T14:54:57.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:57.645 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:57.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:57.747 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:57.775 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:57.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:57.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:57.811 INFO:teuthology.orchestra.run.smithi151.stderr:marked in osd.4. 2024-09-13T14:54:57.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:57.898 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:57.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:57.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:57.962 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 38 16 67739 67723 111.368 61 0.00327727 0.00894827 2024-09-13T14:54:58.000 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:58.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:58.022 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:58.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:58.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:57 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/168171877' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-13T14:54:58.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:57 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-13T14:54:58.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:57 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/168171877' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-13T14:54:58.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:57 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-13T14:54:58.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:57 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/168171877' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-13T14:54:58.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:57 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-13T14:54:58.123 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:58.144 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:58.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:58.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:58.246 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:58.257 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T14:54:58.267 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:58.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:58.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:58.369 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:58.390 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:58.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:58.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:58.432 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-09-13T14:54:58.492 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:58.498 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:54:58.514 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:58.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:58.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:58.615 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:58.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:58.638 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:58.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:58.739 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:58.760 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:58.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:58.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:58.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:58.883 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:58.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:58.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:58.962 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 39 16 67739 67723 108.512 0 - 0.00894827 2024-09-13T14:54:58.963 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:58 smithi184 ceph-mon[92527]: pgmap v1899: 119 pgs: 119 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-13T14:54:58.963 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:58 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-13T14:54:58.963 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:58 smithi184 ceph-mon[92527]: osdmap e1228: 8 total, 8 up, 8 in 2024-09-13T14:54:58.985 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:59.006 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:59.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:59.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:59.077 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:58 smithi151 ceph-mon[104450]: pgmap v1899: 119 pgs: 119 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-13T14:54:59.077 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:58 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-13T14:54:59.077 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:58 smithi151 ceph-mon[104450]: osdmap e1228: 8 total, 8 up, 8 in 2024-09-13T14:54:59.077 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:58 smithi151 ceph-mon[108966]: pgmap v1899: 119 pgs: 119 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-13T14:54:59.077 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:58 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-13T14:54:59.077 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:58 smithi151 ceph-mon[108966]: osdmap e1228: 8 total, 8 up, 8 in 2024-09-13T14:54:59.078 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:59.108 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:59.131 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:59.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:59.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:59.232 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:59.254 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:59.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:59.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:59.356 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:59.379 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:59.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:59.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:59.481 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:59.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:59.504 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:59.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:59.605 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:59.626 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:59.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:59.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:59.728 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:59.749 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:54:59.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:59.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:59.805 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-13T14:54:59.851 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:59.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:59.873 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:59.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:59.963 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:2024-09-13T14:54:59.962326+0000 min lat: 0.00116648 max lat: 0.321231 avg lat: 0.00894827 2024-09-13T14:54:59.963 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:54:59.963 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 40 16 67739 67723 105.8 0 - 0.00894827 2024-09-13T14:54:59.974 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:59.997 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:54:59.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:59.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:00.100 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:00.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:59 smithi184 ceph-mon[92527]: osdmap e1229: 8 total, 8 up, 8 in 2024-09-13T14:55:00.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:59 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/860380279' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:00.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:59 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:00.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:59 smithi184 ceph-mon[92527]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:55:00.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:59 smithi184 ceph-mon[92527]: Cluster is now healthy 2024-09-13T14:55:00.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:59 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:55:00.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:54:59 smithi184 ceph-mon[92527]: osdmap e1230: 8 total, 8 up, 8 in 2024-09-13T14:55:00.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:59 smithi151 ceph-mon[104450]: osdmap e1229: 8 total, 8 up, 8 in 2024-09-13T14:55:00.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:59 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/860380279' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:00.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:59 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:00.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:59 smithi151 ceph-mon[104450]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:55:00.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:59 smithi151 ceph-mon[104450]: Cluster is now healthy 2024-09-13T14:55:00.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:59 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:55:00.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:54:59 smithi151 ceph-mon[104450]: osdmap e1230: 8 total, 8 up, 8 in 2024-09-13T14:55:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:59 smithi151 ceph-mon[108966]: osdmap e1229: 8 total, 8 up, 8 in 2024-09-13T14:55:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:59 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/860380279' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:00.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:59 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:00.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:59 smithi151 ceph-mon[108966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:55:00.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:59 smithi151 ceph-mon[108966]: Cluster is now healthy 2024-09-13T14:55:00.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:59 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:55:00.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:54:59 smithi151 ceph-mon[108966]: osdmap e1230: 8 total, 8 up, 8 in 2024-09-13T14:55:00.143 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:00.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:00.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:00.246 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:00.292 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:00.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:00.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:00.394 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:00.430 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:00.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:00.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:00.511 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:55:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:55:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:55:00.532 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:00.573 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:00.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:00.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:00.675 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:00.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:00.714 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:00.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:00.816 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:00.833 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:55:00.833 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [ 2024-09-13T14:55:00.833 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.833 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:68670 9.6 9:69191eda:::benchmark_data_smithi151_261954_object68669:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1230)", 2024-09-13T14:55:00.833 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:55:00.819923+0000", 2024-09-13T14:55:00.834 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.010504209, 2024-09-13T14:55:00.834 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.010541165, 2024-09-13T14:55:00.834 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:00.834 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:55:00.834 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:00.835 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T14:55:00.835 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:55:00.835 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 68670 2024-09-13T14:55:00.835 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.836 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:00.836 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.836 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:00.836 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.819923+0000", 2024-09-13T14:55:00.836 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:00.836 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.837 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.837 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:00.837 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.819923+0000", 2024-09-13T14:55:00.837 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:00.837 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.838 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.838 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:00.838 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.819922+0000", 2024-09-13T14:55:00.838 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:55:00.838 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.839 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.839 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:00.839 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.819946+0000", 2024-09-13T14:55:00.839 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.4440999999999999e-05 2024-09-13T14:55:00.839 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.839 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.840 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:00.840 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.819956+0000", 2024-09-13T14:55:00.840 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.4509999999999998e-06 2024-09-13T14:55:00.840 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.840 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.841 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:00.841 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.819960+0000", 2024-09-13T14:55:00.841 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4880000000000001e-06 2024-09-13T14:55:00.841 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.841 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.842 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:00.842 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.821547+0000", 2024-09-13T14:55:00.842 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0015870019999999999 2024-09-13T14:55:00.842 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.842 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.842 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:00.843 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.821687+0000", 2024-09-13T14:55:00.843 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013944700000000001 2024-09-13T14:55:00.843 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.843 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.843 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:55:00.844 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.821826+0000", 2024-09-13T14:55:00.844 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000139202 2024-09-13T14:55:00.844 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.844 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.844 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:00.844 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.825094+0000", 2024-09-13T14:55:00.845 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0032682010000000001 2024-09-13T14:55:00.845 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.845 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.845 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:55:00.845 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.826129+0000", 2024-09-13T14:55:00.845 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0010349689999999999 2024-09-13T14:55:00.846 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:00.846 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:00.846 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:00.846 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.847 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.847 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:68674 9.a 9:50ea69d1:::benchmark_data_smithi151_261954_object68673:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1230)", 2024-09-13T14:55:00.847 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:55:00.820697+0000", 2024-09-13T14:55:00.847 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.0097302420000000001, 2024-09-13T14:55:00.847 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0098584290000000002, 2024-09-13T14:55:00.848 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:00.848 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:55:00.848 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:00.848 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T14:55:00.848 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:55:00.848 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 68674 2024-09-13T14:55:00.848 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.848 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:00.848 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.848 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:00.848 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.820697+0000", 2024-09-13T14:55:00.848 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:00.848 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.849 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.849 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:00.849 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.820697+0000", 2024-09-13T14:55:00.849 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:00.849 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.849 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.849 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:00.849 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.820697+0000", 2024-09-13T14:55:00.849 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:55:00.849 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.850 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.850 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:00.850 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.820741+0000", 2024-09-13T14:55:00.850 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4039999999999998e-05 2024-09-13T14:55:00.850 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.850 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.850 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:00.850 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.820754+0000", 2024-09-13T14:55:00.850 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3576e-05 2024-09-13T14:55:00.850 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.851 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.851 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:00.851 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.820760+0000", 2024-09-13T14:55:00.851 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7969999999999997e-06 2024-09-13T14:55:00.851 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.851 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.851 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:00.851 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.820808+0000", 2024-09-13T14:55:00.851 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7970999999999999e-05 2024-09-13T14:55:00.851 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.851 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.851 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:00.851 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.820968+0000", 2024-09-13T14:55:00.851 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000160242 2024-09-13T14:55:00.852 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.852 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.852 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:55:00.852 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.821160+0000", 2024-09-13T14:55:00.852 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000191526 2024-09-13T14:55:00.852 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.852 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.852 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:55:00.852 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.827065+0000", 2024-09-13T14:55:00.852 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0059049940000000002 2024-09-13T14:55:00.852 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.852 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.852 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:00.852 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.830462+0000", 2024-09-13T14:55:00.853 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0033977180000000001 2024-09-13T14:55:00.853 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:00.853 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:00.853 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:00.853 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.853 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.853 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:68677 9.f 9:f9d04019:::benchmark_data_smithi151_261954_object68676:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1230)", 2024-09-13T14:55:00.853 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:55:00.823619+0000", 2024-09-13T14:55:00.853 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.0068089889999999997, 2024-09-13T14:55:00.853 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0070176279999999997, 2024-09-13T14:55:00.853 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:00.853 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:55:00.854 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:00.854 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T14:55:00.854 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:55:00.854 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 68677 2024-09-13T14:55:00.854 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.854 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:00.854 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.854 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:00.854 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.823619+0000", 2024-09-13T14:55:00.854 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:00.854 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.854 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.855 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:00.855 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.823619+0000", 2024-09-13T14:55:00.855 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:00.855 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.855 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.855 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:00.855 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.823617+0000", 2024-09-13T14:55:00.855 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:55:00.855 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.855 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.855 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:00.855 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.823727+0000", 2024-09-13T14:55:00.855 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00011021799999999999 2024-09-13T14:55:00.856 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.856 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.856 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:00.856 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.823749+0000", 2024-09-13T14:55:00.856 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.122e-05 2024-09-13T14:55:00.856 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.856 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.856 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:00.856 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.823763+0000", 2024-09-13T14:55:00.856 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4669e-05 2024-09-13T14:55:00.857 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.857 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.857 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:00.857 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.823896+0000", 2024-09-13T14:55:00.858 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013228799999999999 2024-09-13T14:55:00.858 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.858 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.858 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:00.858 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.824011+0000", 2024-09-13T14:55:00.858 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000115742 2024-09-13T14:55:00.858 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.858 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.858 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3", 2024-09-13T14:55:00.858 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.824132+0000", 2024-09-13T14:55:00.858 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000121075 2024-09-13T14:55:00.858 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:00.858 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:00.859 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:00.859 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.859 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.859 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:68679 9.a 9:53662947:::benchmark_data_smithi151_261954_object68678:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1230)", 2024-09-13T14:55:00.859 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:55:00.823777+0000", 2024-09-13T14:55:00.859 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.0066506869999999997, 2024-09-13T14:55:00.859 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0069290460000000003, 2024-09-13T14:55:00.859 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:00.859 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:55:00.859 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:00.859 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T14:55:00.859 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:55:00.860 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 68679 2024-09-13T14:55:00.860 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.860 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:00.860 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.860 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:00.860 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.823777+0000", 2024-09-13T14:55:00.860 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:00.860 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.860 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.860 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:00.860 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.823777+0000", 2024-09-13T14:55:00.861 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:00.861 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.861 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.861 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:00.861 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.823777+0000", 2024-09-13T14:55:00.861 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:55:00.861 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.861 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.861 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:00.861 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.823830+0000", 2024-09-13T14:55:00.861 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3306000000000001e-05 2024-09-13T14:55:00.861 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.861 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.862 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:00.862 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.823837+0000", 2024-09-13T14:55:00.862 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.3019999999999997e-06 2024-09-13T14:55:00.862 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.862 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.862 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:00.862 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.823841+0000", 2024-09-13T14:55:00.862 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8909999999999997e-06 2024-09-13T14:55:00.862 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.862 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.862 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:00.862 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.824164+0000", 2024-09-13T14:55:00.862 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00032322399999999999 2024-09-13T14:55:00.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.863 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.863 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:00.863 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.824279+0000", 2024-09-13T14:55:00.863 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00011515600000000001 2024-09-13T14:55:00.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.863 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.863 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:55:00.863 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.824401+0000", 2024-09-13T14:55:00.863 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000121307 2024-09-13T14:55:00.863 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:00.863 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:00.863 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:00.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.864 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.864 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.46034.0:68678 9.d e1230/1222 9:b90d61e7:::benchmark_data_smithi151_261954_object68677:head v 1230'4294, mlcod=1230'4292)", 2024-09-13T14:55:00.864 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:55:00.826185+0000", 2024-09-13T14:55:00.864 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.0042424619999999998, 2024-09-13T14:55:00.864 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0045866939999999997, 2024-09-13T14:55:00.864 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:00.864 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:55:00.864 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:00.864 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.864 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:00.864 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.826185+0000", 2024-09-13T14:55:00.864 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:00.865 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.865 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.865 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:00.865 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.826185+0000", 2024-09-13T14:55:00.865 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:00.865 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.865 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.865 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:00.865 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.826185+0000", 2024-09-13T14:55:00.865 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:55:00.865 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.865 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.865 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:00.866 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.826216+0000", 2024-09-13T14:55:00.866 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.1958000000000001e-05 2024-09-13T14:55:00.866 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.866 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.866 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:00.866 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.826232+0000", 2024-09-13T14:55:00.866 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.5443000000000001e-05 2024-09-13T14:55:00.866 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.866 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.866 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:00.866 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.826238+0000", 2024-09-13T14:55:00.866 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.5200000000000003e-06 2024-09-13T14:55:00.866 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.867 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.867 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:00.867 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.827647+0000", 2024-09-13T14:55:00.867 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001408093 2024-09-13T14:55:00.867 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.867 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.867 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:00.867 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.827666+0000", 2024-09-13T14:55:00.867 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.9026999999999999e-05 2024-09-13T14:55:00.867 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:00.867 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:00.867 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:00.867 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.868 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.868 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:68683 9.6 9:6d9e62f6:::benchmark_data_smithi151_261954_object68682:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1230)", 2024-09-13T14:55:00.868 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:55:00.826834+0000", 2024-09-13T14:55:00.868 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.0035932289999999999, 2024-09-13T14:55:00.868 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0039944209999999997, 2024-09-13T14:55:00.868 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:00.868 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:55:00.868 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:00.868 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T14:55:00.868 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:55:00.868 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 68683 2024-09-13T14:55:00.868 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.869 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:00.869 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.869 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:00.869 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.826834+0000", 2024-09-13T14:55:00.869 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:00.869 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.869 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.869 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:00.869 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.826834+0000", 2024-09-13T14:55:00.869 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:00.869 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.869 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.870 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:00.870 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.826833+0000", 2024-09-13T14:55:00.870 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:55:00.870 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.870 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.870 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:00.870 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.826914+0000", 2024-09-13T14:55:00.870 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0890000000000006e-05 2024-09-13T14:55:00.870 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.870 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.870 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:00.871 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.826931+0000", 2024-09-13T14:55:00.871 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.7620000000000001e-05 2024-09-13T14:55:00.871 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.871 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.871 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:00.871 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.826938+0000", 2024-09-13T14:55:00.871 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.4180000000000002e-06 2024-09-13T14:55:00.871 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.871 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.871 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:00.871 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.827008+0000", 2024-09-13T14:55:00.871 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.0382999999999993e-05 2024-09-13T14:55:00.871 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.872 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.872 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:00.872 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.827149+0000", 2024-09-13T14:55:00.872 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000140977 2024-09-13T14:55:00.872 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.872 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.872 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:55:00.872 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.827338+0000", 2024-09-13T14:55:00.872 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000188677 2024-09-13T14:55:00.872 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:00.872 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:00.872 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:00.872 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.873 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.873 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.46034.0:68676 9.5 e1230/1218 9:afd87c78:::benchmark_data_smithi151_261954_object68675:head v 1230'4240, mlcod=1230'4238)", 2024-09-13T14:55:00.873 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:55:00.829169+0000", 2024-09-13T14:55:00.873 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.001259074, 2024-09-13T14:55:00.873 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001740445, 2024-09-13T14:55:00.873 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:00.873 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:55:00.873 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:00.873 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.873 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:00.873 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.829169+0000", 2024-09-13T14:55:00.873 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:00.873 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.874 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.874 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:00.874 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.829169+0000", 2024-09-13T14:55:00.874 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:00.874 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.874 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.874 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:00.874 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.829166+0000", 2024-09-13T14:55:00.874 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999976 2024-09-13T14:55:00.874 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.874 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.874 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:00.875 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.829218+0000", 2024-09-13T14:55:00.875 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1996000000000002e-05 2024-09-13T14:55:00.875 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.875 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.875 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:00.875 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.829233+0000", 2024-09-13T14:55:00.875 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.447e-05 2024-09-13T14:55:00.875 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.875 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.875 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:00.875 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.829241+0000", 2024-09-13T14:55:00.875 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.9249999999999995e-06 2024-09-13T14:55:00.876 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.876 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.876 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:00.876 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.829304+0000", 2024-09-13T14:55:00.876 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3600000000000001e-05 2024-09-13T14:55:00.876 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.876 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.876 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:00.876 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.829326+0000", 2024-09-13T14:55:00.876 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.1523e-05 2024-09-13T14:55:00.876 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:00.876 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:00.876 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:00.877 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.877 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.877 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:68666 9.f 9:f8527198:::benchmark_data_smithi151_261954_object68665:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1230)", 2024-09-13T14:55:00.877 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:55:00.813638+0000", 2024-09-13T14:55:00.877 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.016789400999999999, 2024-09-13T14:55:00.877 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.017330034000000001, 2024-09-13T14:55:00.877 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:00.877 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:55:00.877 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:00.877 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T14:55:00.877 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:55:00.877 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 68666 2024-09-13T14:55:00.877 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.878 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:00.878 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.878 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:00.878 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.813638+0000", 2024-09-13T14:55:00.878 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:00.878 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.878 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.878 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:00.878 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.813638+0000", 2024-09-13T14:55:00.878 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:00.878 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.878 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.879 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:00.879 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.813638+0000", 2024-09-13T14:55:00.879 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:55:00.879 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.879 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.879 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:00.879 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.813711+0000", 2024-09-13T14:55:00.879 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.3755000000000001e-05 2024-09-13T14:55:00.879 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.879 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.879 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:00.879 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.813720+0000", 2024-09-13T14:55:00.879 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.6600000000000001e-06 2024-09-13T14:55:00.879 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.880 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.880 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:00.880 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.813726+0000", 2024-09-13T14:55:00.880 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5759999999999996e-06 2024-09-13T14:55:00.880 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.880 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.880 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:00.880 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.818819+0000", 2024-09-13T14:55:00.880 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0050934509999999997 2024-09-13T14:55:00.880 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.880 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.880 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:00.880 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.818963+0000", 2024-09-13T14:55:00.880 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014359900000000001 2024-09-13T14:55:00.881 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.881 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.881 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3", 2024-09-13T14:55:00.881 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.819106+0000", 2024-09-13T14:55:00.881 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014310700000000001 2024-09-13T14:55:00.881 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.881 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.881 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:55:00.881 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.822115+0000", 2024-09-13T14:55:00.881 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0030095370000000001 2024-09-13T14:55:00.881 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:00.881 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:00.882 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:00.882 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.830302+0000", 2024-09-13T14:55:00.882 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0081864650000000004 2024-09-13T14:55:00.882 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:00.882 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:00.882 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:00.882 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:00.882 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-13T14:55:00.882 INFO:teuthology.orchestra.run.smithi184.stdout: "num_ops": 8 2024-09-13T14:55:00.882 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:55:00.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:00.883 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:00.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:00.962 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 41 16 68821 68805 104.869 22.5417 0.00812093 0.00953007 2024-09-13T14:55:00.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:01.019 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:01.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:01.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:01.029 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:00 smithi151 ceph-mon[104450]: pgmap v1902: 119 pgs: 8 peering, 111 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-13T14:55:01.030 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:00 smithi151 ceph-mon[108966]: pgmap v1902: 119 pgs: 8 peering, 111 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-13T14:55:01.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:00 smithi184 ceph-mon[92527]: pgmap v1902: 119 pgs: 8 peering, 111 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-13T14:55:01.124 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:01.162 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:01.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:01.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:01.264 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:01.299 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:01.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:01.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:01.402 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:01.441 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:01.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:01.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:01.543 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:01.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:01.582 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:01.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:01.678 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T14:55:01.683 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:01.726 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:01.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:01.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:01.827 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:01.872 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:01.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:01.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:01.963 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 42 16 69985 69969 104.104 72.75 0.0181638 0.00959943 2024-09-13T14:55:01.974 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:02.005 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:02.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:02.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:02.107 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:55:02.107 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:02.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:02.162 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:02.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:02.264 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:02.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:02.312 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:02.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:02.413 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:02.430 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:55:02.430 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:55:02.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:02.454 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:02.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:02.457 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:55:02.457 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:55:02.487 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:55:02.488 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:55:02.521 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:55:02.522 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:55:02.522 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:55:02.522 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:02.523 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:02 smithi151 ceph-mon[104450]: pgmap v1904: 119 pgs: 1 active+recovering+undersized+remapped, 8 peering, 110 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 49 MiB/s wr, 777 op/s; 4290/202518 objects misplaced (2.118%); 53 KiB/s, 1 objects/s recovering 2024-09-13T14:55:02.523 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:02 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3381262126' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:02.523 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:55:02.523 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:02.523 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:02 smithi151 ceph-mon[108966]: pgmap v1904: 119 pgs: 1 active+recovering+undersized+remapped, 8 peering, 110 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 49 MiB/s wr, 777 op/s; 4290/202518 objects misplaced (2.118%); 53 KiB/s, 1 objects/s recovering 2024-09-13T14:55:02.523 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:02 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3381262126' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:02.546 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:55:02.546 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:55:02.556 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:02.577 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:55:02.578 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:55:02.587 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:02.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:02.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:02.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:55:02.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:02.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:02 smithi184 ceph-mon[92527]: pgmap v1904: 119 pgs: 1 active+recovering+undersized+remapped, 8 peering, 110 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 49 MiB/s wr, 777 op/s; 4290/202518 objects misplaced (2.118%); 53 KiB/s, 1 objects/s recovering 2024-09-13T14:55:02.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:02 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3381262126' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:02.607 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:55:02.607 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:55:02.637 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:55:02.638 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:55:02.689 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:02.725 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:02.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:02.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:02.827 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:02.862 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:02.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:02.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:02.963 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 43 16 71133 71117 103.351 71.75 0.00343686 0.00966794 2024-09-13T14:55:02.964 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:02.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:02.995 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:02.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:03.072 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:55:03.096 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:03.128 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:03.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:03.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:03.230 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:03.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:03.274 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:03.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:03.375 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:03.433 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 6, 7, 5, 1, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:55:03.433 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:55:03.433 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 2, 3, 6, 7, 5, 1, 4] 2024-09-13T14:55:03.433 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd out 5 2024-09-13T14:55:03.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:03.437 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:03.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:03.538 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:03.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:03.580 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:03.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:03.681 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:03.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:03.704 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:03.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:03.805 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:03.843 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:03.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:03.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:03.945 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:03.962 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 44 16 72170 72154 102.475 64.8125 0.00731866 0.00975286 2024-09-13T14:55:03.993 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:03.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:03.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:04.047 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:03 smithi184 ceph-mon[92527]: Health check failed: Degraded data redundancy: 8506/208485 objects degraded (4.080%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:04.071 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:55:04.072 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:55:04.095 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:04.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:03 smithi151 ceph-mon[104450]: Health check failed: Degraded data redundancy: 8506/208485 objects degraded (4.080%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:04.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:03 smithi151 ceph-mon[108966]: Health check failed: Degraded data redundancy: 8506/208485 objects degraded (4.080%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:04.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:04.132 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:04.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:04.234 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:04.269 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:04.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:04.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:04.370 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:04.398 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:04.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:04.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:04.442 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:04.501 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:04.512 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:04.531 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:04.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:04.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:04.633 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:04.673 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:04.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:04.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:04.775 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:04.782 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T14:55:04.806 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:04.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:04.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:04.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:04 smithi151 ceph-mon[104450]: pgmap v1905: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 116 active+clean; 4.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 2.8 KiB/s rd, 43 MiB/s wr, 688 op/s; 8506/208485 objects degraded (4.080%); 4290/208485 objects misplaced (2.058%); 56 KiB/s, 1 keys/s, 2 objects/s recovering 2024-09-13T14:55:04.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:55:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:55:04] "GET /metrics HTTP/1.1" 200 36640 "" "Prometheus/2.51.0" 2024-09-13T14:55:04.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:04 smithi151 ceph-mon[108966]: pgmap v1905: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 116 active+clean; 4.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 2.8 KiB/s rd, 43 MiB/s wr, 688 op/s; 8506/208485 objects degraded (4.080%); 4290/208485 objects misplaced (2.058%); 56 KiB/s, 1 keys/s, 2 objects/s recovering 2024-09-13T14:55:04.908 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:04.950 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:04.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:04.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:04.963 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 45 16 73162 73146 101.576 62 0.00892534 0.00983846 2024-09-13T14:55:05.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:05.090 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:05.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:05.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:05.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:04 smithi184 ceph-mon[92527]: pgmap v1905: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 116 active+clean; 4.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 2.8 KiB/s rd, 43 MiB/s wr, 688 op/s; 8506/208485 objects degraded (4.080%); 4290/208485 objects misplaced (2.058%); 56 KiB/s, 1 keys/s, 2 objects/s recovering 2024-09-13T14:55:05.160 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:55:05.192 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:05.213 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:05.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:05.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:05.315 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:05.353 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:05.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:05.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:05.455 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:05.487 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:05.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:05.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:05.589 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:05.618 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:05.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:05.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:05.720 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:05.755 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:05.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:05.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:05.857 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:05.894 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:05.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:05.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:05.963 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 46 16 73983 73967 100.483 51.3125 0.00294454 0.00992865 2024-09-13T14:55:05.996 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:06.029 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:06.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:06.131 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:06.153 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:06.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:06.256 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:06.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.296 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:06.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:06.333 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:55:06.397 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:06.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.430 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:06.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:06.532 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:06.564 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:06.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:06.669 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:06.705 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:06.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:06.807 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:06.850 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:06.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:06.931 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:06 smithi184 ceph-mon[92527]: pgmap v1906: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 2.2 KiB/s rd, 51 MiB/s wr, 817 op/s; 8506/215385 objects degraded (3.949%); 4194/215385 objects misplaced (1.947%); 824 KiB/s, 1 keys/s, 14 objects/s recovering 2024-09-13T14:55:06.952 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:06.963 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 47 16 74941 74925 99.6191 59.875 0.0278908 0.0100313 2024-09-13T14:55:06.981 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:06.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:07.083 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:07.169 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:06 smithi151 ceph-mon[104450]: pgmap v1906: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 2.2 KiB/s rd, 51 MiB/s wr, 817 op/s; 8506/215385 objects degraded (3.949%); 4194/215385 objects misplaced (1.947%); 824 KiB/s, 1 keys/s, 14 objects/s recovering 2024-09-13T14:55:07.169 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:06 smithi151 ceph-mon[108966]: pgmap v1906: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 2.2 KiB/s rd, 51 MiB/s wr, 817 op/s; 8506/215385 objects degraded (3.949%); 4194/215385 objects misplaced (1.947%); 824 KiB/s, 1 keys/s, 14 objects/s recovering 2024-09-13T14:55:07.202 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:07.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:07.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:07.304 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:07.328 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:07.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:07.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:07.360 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:55:07.360 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:55:07.430 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:07.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:07.478 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:07.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:07.579 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:07.611 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:07.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:07.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:07.714 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:07.751 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:07.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:07.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:07.852 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:07.880 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:07.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:07.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:07.963 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 48 16 75910 75894 98.8054 60.5625 0.0132566 0.0101153 2024-09-13T14:55:07.982 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:08.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:08.018 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:08.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:08.051 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T14:55:08.119 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:08.148 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:08.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:08.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:08.250 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:08.277 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:08.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:08.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:08.379 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:08.383 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:55:08.402 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:08.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:08.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:08.504 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:08.546 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:08.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:08.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:08.648 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:08.678 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:08.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:08.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:08.780 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:08.815 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:08.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:08.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:08.917 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:08.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:08.958 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:08.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:08.963 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 49 16 76768 76752 97.8833 53.625 0.0121564 0.0102109 2024-09-13T14:55:09.060 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:09.098 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:09.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:09.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:09.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:08 smithi184 ceph-mon[92527]: pgmap v1907: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.9 KiB/s rd, 56 MiB/s wr, 902 op/s; 8506/221892 objects degraded (3.833%); 4194/221892 objects misplaced (1.890%); 731 KiB/s, 1 keys/s, 12 objects/s recovering 2024-09-13T14:55:09.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:08 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:55:09.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:08 smithi151 ceph-mon[104450]: pgmap v1907: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.9 KiB/s rd, 56 MiB/s wr, 902 op/s; 8506/221892 objects degraded (3.833%); 4194/221892 objects misplaced (1.890%); 731 KiB/s, 1 keys/s, 12 objects/s recovering 2024-09-13T14:55:09.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:08 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:55:09.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:08 smithi151 ceph-mon[108966]: pgmap v1907: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.9 KiB/s rd, 56 MiB/s wr, 902 op/s; 8506/221892 objects degraded (3.833%); 4194/221892 objects misplaced (1.890%); 731 KiB/s, 1 keys/s, 12 objects/s recovering 2024-09-13T14:55:09.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:08 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:55:09.200 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:09.256 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:09.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:09.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:09.358 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:09.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:09.391 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:09.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:09.492 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:09.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:09.529 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:09.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:09.630 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:09.652 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:09.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:09.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:09.754 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:09.786 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:09.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:09.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:09.888 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:09.922 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:09.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:09.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:09.963 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 50 16 77812 77796 97.2305 65.25 0.00784378 0.0102799 2024-09-13T14:55:10.024 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:10.028 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:09 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 8506/223674 objects degraded (3.803%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:10.086 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:10.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:10.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:10.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:09 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 8506/223674 objects degraded (3.803%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:10.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:09 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 8506/223674 objects degraded (3.803%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:10.188 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:10.220 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:10.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:10.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:10.287 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:55:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:55:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:55:10.322 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:10.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:10.363 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:10.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:10.464 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:10.505 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:10.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:10.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:10.525 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:55:10.525 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:55:10.607 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:10.636 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:10.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:10.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:10.738 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:10.770 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:10.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:10.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:10.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:10.908 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:10.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:10.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:10.963 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 51 15 78825 78810 96.5666 63.375 0.0132399 0.0103492 2024-09-13T14:55:11.011 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:11.051 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:11.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:11.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:11.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:10 smithi184 ceph-mon[92527]: pgmap v1908: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 54 MiB/s wr, 858 op/s; 8506/223674 objects degraded (3.803%); 4194/223674 objects misplaced (1.875%); 647 KiB/s, 0 keys/s, 11 objects/s recovering 2024-09-13T14:55:11.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:10 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:55:11.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:10 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:55:11.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:10 smithi151 ceph-mon[104450]: pgmap v1908: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 54 MiB/s wr, 858 op/s; 8506/223674 objects degraded (3.803%); 4194/223674 objects misplaced (1.875%); 647 KiB/s, 0 keys/s, 11 objects/s recovering 2024-09-13T14:55:11.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:10 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:55:11.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:10 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:55:11.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:10 smithi151 ceph-mon[108966]: pgmap v1908: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 54 MiB/s wr, 858 op/s; 8506/223674 objects degraded (3.803%); 4194/223674 objects misplaced (1.875%); 647 KiB/s, 0 keys/s, 11 objects/s recovering 2024-09-13T14:55:11.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:10 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:55:11.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:10 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:55:11.152 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:11.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:11.195 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:11.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:11.296 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:11.333 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:11.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:11.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:11.392 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T14:55:11.435 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:11.466 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:11.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:11.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:11.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:11.603 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:11.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:11.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:11.705 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:11.755 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:11.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:11.759 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:11.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:11.861 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:11.892 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:55:11.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:11.914 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:11.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:11.963 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 52 16 79821 79805 95.9052 62.1875 0.0100007 0.0104217 2024-09-13T14:55:12.014 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:12.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:12.050 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:12.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:12.151 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:12.198 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:12.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:12.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:12.301 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:12.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:12.350 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:12.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:12.451 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:12.487 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:12.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:12.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:12.588 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:12.624 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:12.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:12.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:12.726 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:12.764 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:12.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:12.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:12.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:12 smithi184 ceph-mon[92527]: pgmap v1909: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.4 KiB/s rd, 56 MiB/s wr, 898 op/s; 8506/229893 objects degraded (3.700%); 4076/229893 objects misplaced (1.773%); 1.2 MiB/s, 0 keys/s, 19 objects/s recovering 2024-09-13T14:55:12.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:12 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:55:12.865 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:12.903 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:12.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:12.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:12.964 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 53 16 80869 80853 95.3315 65.5 0.00578121 0.0104835 2024-09-13T14:55:13.007 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:13.030 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:12 smithi151 ceph-mon[104450]: pgmap v1909: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.4 KiB/s rd, 56 MiB/s wr, 898 op/s; 8506/229893 objects degraded (3.700%); 4076/229893 objects misplaced (1.773%); 1.2 MiB/s, 0 keys/s, 19 objects/s recovering 2024-09-13T14:55:13.030 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:12 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:55:13.030 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:12 smithi151 ceph-mon[108966]: pgmap v1909: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.4 KiB/s rd, 56 MiB/s wr, 898 op/s; 8506/229893 objects degraded (3.700%); 4076/229893 objects misplaced (1.773%); 1.2 MiB/s, 0 keys/s, 19 objects/s recovering 2024-09-13T14:55:13.030 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:12 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:55:13.041 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:13.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:13.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:13.143 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:13.180 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:13.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:13.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:13.282 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:13.345 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:13.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:13.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:13.447 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:13.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:13.483 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:13.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:13.584 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:13.625 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:13.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:13.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:13.727 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:13.761 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:13.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:13.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:13.822 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-13T14:55:13.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:13 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3155223061' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:55:13.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:13 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:55:13.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:13 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3155223061' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:55:13.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:13 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:55:13.863 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:13.906 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:13.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:13.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:13.964 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 54 16 81874 81858 94.7292 62.8125 0.0174925 0.010551 2024-09-13T14:55:14.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:14.046 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:14.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:14.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:14.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:13 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3155223061' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:55:14.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:13 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:55:14.148 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:14.185 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:14.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:14.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:14.287 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:14.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:14.329 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:14.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:14.430 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:14.457 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:14.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:14.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:14.560 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:14.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:14.598 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:14.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:14.699 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:14.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:14.735 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:14.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:14.751 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:55:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:55:14] "GET /metrics HTTP/1.1" 200 36640 "" "Prometheus/2.51.0" 2024-09-13T14:55:14.835 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:14.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:14.872 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:14.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:14.964 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 55 16 82891 82875 94.1625 63.5625 0.0136476 0.010614 2024-09-13T14:55:14.973 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:15.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:15.016 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:15.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:15.040 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:14 smithi184 ceph-mon[92527]: pgmap v1910: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.4 KiB/s rd, 62 MiB/s wr, 989 op/s; 8506/238131 objects degraded (3.572%); 4076/238131 objects misplaced (1.712%); 1.1 MiB/s, 0 keys/s, 18 objects/s recovering 2024-09-13T14:55:15.040 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:14 smithi184 ceph-mon[92527]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:55:15.041 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:14 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:55:15.041 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:14 smithi184 ceph-mon[92527]: osdmap e1231: 8 total, 8 up, 8 in 2024-09-13T14:55:15.062 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:14 smithi151 ceph-mon[108966]: pgmap v1910: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.4 KiB/s rd, 62 MiB/s wr, 989 op/s; 8506/238131 objects degraded (3.572%); 4076/238131 objects misplaced (1.712%); 1.1 MiB/s, 0 keys/s, 18 objects/s recovering 2024-09-13T14:55:15.062 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:14 smithi151 ceph-mon[108966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:55:15.062 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:14 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:55:15.062 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:14 smithi151 ceph-mon[108966]: osdmap e1231: 8 total, 8 up, 8 in 2024-09-13T14:55:15.063 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:14 smithi151 ceph-mon[104450]: pgmap v1910: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.4 KiB/s rd, 62 MiB/s wr, 989 op/s; 8506/238131 objects degraded (3.572%); 4076/238131 objects misplaced (1.712%); 1.1 MiB/s, 0 keys/s, 18 objects/s recovering 2024-09-13T14:55:15.063 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:14 smithi151 ceph-mon[104450]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:55:15.063 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:14 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:55:15.063 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:14 smithi151 ceph-mon[104450]: osdmap e1231: 8 total, 8 up, 8 in 2024-09-13T14:55:15.063 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a[104427]: 2024-09-13T14:55:14.911+0000 7fe9da14f640 -1 mon.a@0(leader).osd e1231 definitely_dead 0 2024-09-13T14:55:15.117 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:15.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:15.153 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:15.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:15.169 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:55:15.169 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:55:15.255 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:15.285 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:15.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:15.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:15.387 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:15.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:15.426 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:15.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:15.527 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:15.561 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:15.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:15.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:15.665 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:15.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:15.696 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:15.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:15.797 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:15.835 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:15.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:15.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:15.840 INFO:teuthology.orchestra.run.smithi151.stderr:marked out osd.5. 2024-09-13T14:55:15.881 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T14:55:15.937 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:15.960 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:15.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:15.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:15.964 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 56 16 83749 83733 93.4385 53.625 0.00734554 0.0106796 2024-09-13T14:55:16.062 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:16.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:16.085 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:16.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:16.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:15 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3705703734' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-13T14:55:16.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:15 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-13T14:55:16.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:15 smithi184 ceph-mon[92527]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-13T14:55:16.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:15 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 8506/238131 objects degraded (3.572%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:16.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:15 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3352279957' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:16.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:15 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3705703734' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-13T14:55:16.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:15 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-13T14:55:16.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:15 smithi151 ceph-mon[104450]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-13T14:55:16.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:15 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 8506/238131 objects degraded (3.572%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:16.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:15 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3352279957' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:16.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:15 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3705703734' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-13T14:55:16.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:15 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-13T14:55:16.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:15 smithi151 ceph-mon[108966]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-13T14:55:16.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:15 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 8506/238131 objects degraded (3.572%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:16.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:15 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3352279957' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:16.187 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:16.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:16.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:16.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:16.312 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:16.333 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:16.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:16.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:16.435 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:16.442 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:55:16.457 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:16.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:16.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:16.559 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:16.581 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:16.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:16.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:16.682 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:16.704 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:16.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:16.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:16.806 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:16.827 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:16.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:16.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:16.871 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:55:16.871 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:55:16.871 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:55:16.871 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:55:16.872 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:55:16.872 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:55:16.872 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:55:16.872 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:55:16.872 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:55:16.872 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:55:16.872 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:55:16.873 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:55:16.873 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:55:16.873 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:55:16.873 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:55:16.873 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:55:16.929 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:16.950 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:16.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:16.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:16.964 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 57 16 83749 83733 91.7993 0 - 0.0106796 2024-09-13T14:55:17.052 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:17.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.075 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:17.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:16 smithi184 ceph-mon[92527]: pgmap v1912: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 61 MiB/s wr, 978 op/s; 8506/244764 objects degraded (3.475%); 3978/244764 objects misplaced (1.625%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:55:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:16 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-13T14:55:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:16 smithi184 ceph-mon[92527]: osdmap e1232: 8 total, 8 up, 7 in 2024-09-13T14:55:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:55:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:17.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:16 smithi151 ceph-mon[104450]: pgmap v1912: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 61 MiB/s wr, 978 op/s; 8506/244764 objects degraded (3.475%); 3978/244764 objects misplaced (1.625%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:55:17.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:16 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-13T14:55:17.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:16 smithi151 ceph-mon[104450]: osdmap e1232: 8 total, 8 up, 7 in 2024-09-13T14:55:17.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:55:17.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:17.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:16 smithi151 ceph-mon[108966]: pgmap v1912: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 61 MiB/s wr, 978 op/s; 8506/244764 objects degraded (3.475%); 3978/244764 objects misplaced (1.625%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:55:17.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:16 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-13T14:55:17.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:16 smithi151 ceph-mon[108966]: osdmap e1232: 8 total, 8 up, 7 in 2024-09-13T14:55:17.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:55:17.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:17.176 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:17.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.198 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:17.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:17.301 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:17.323 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:17.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:17.424 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:17.448 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:17.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:17.550 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:17.572 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:17.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:17.674 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:17.695 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:17.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:17.797 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:17.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.820 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:17.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:17.921 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:17.942 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:17.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:17.964 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 58 16 83749 83733 90.2167 0 - 0.0106796 2024-09-13T14:55:18.044 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:18.099 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:18.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:18.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:18.201 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:18.208 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:17 smithi184 ceph-mon[92527]: osdmap e1233: 8 total, 8 up, 7 in 2024-09-13T14:55:18.208 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:17 smithi184 ceph-mon[92527]: osdmap e1234: 8 total, 8 up, 7 in 2024-09-13T14:55:18.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:18.239 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:18.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:18.340 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:18.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:17 smithi151 ceph-mon[104450]: osdmap e1233: 8 total, 8 up, 7 in 2024-09-13T14:55:18.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:17 smithi151 ceph-mon[104450]: osdmap e1234: 8 total, 8 up, 7 in 2024-09-13T14:55:18.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:17 smithi151 ceph-mon[108966]: osdmap e1233: 8 total, 8 up, 7 in 2024-09-13T14:55:18.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:17 smithi151 ceph-mon[108966]: osdmap e1234: 8 total, 8 up, 7 in 2024-09-13T14:55:18.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:18.382 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:18.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:18.483 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:18.510 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:18.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:18.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:18.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:18.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:18.663 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:18.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:18.765 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:18.790 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:18.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:18.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:18.893 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:18.931 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:18.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:18.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:18.964 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 59 16 84175 84159 89.1389 8.875 0.0551718 0.0112118 2024-09-13T14:55:19.033 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:19.060 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:19.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:19.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:19.162 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:19.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:19.200 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:19.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:19.301 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:19.338 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:19.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:19.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:19.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:18 smithi184 ceph-mon[92527]: pgmap v1915: 119 pgs: 3 remapped+peering, 2 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 5.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s; 8508/248307 objects degraded (3.426%); 3978/248307 objects misplaced (1.602%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:55:19.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:18 smithi151 ceph-mon[104450]: pgmap v1915: 119 pgs: 3 remapped+peering, 2 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 5.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s; 8508/248307 objects degraded (3.426%); 3978/248307 objects misplaced (1.602%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:55:19.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:18 smithi151 ceph-mon[108966]: pgmap v1915: 119 pgs: 3 remapped+peering, 2 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 5.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s; 8508/248307 objects degraded (3.426%); 3978/248307 objects misplaced (1.602%); 1.0 MiB/s, 16 objects/s recovering 2024-09-13T14:55:19.440 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:19.470 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:19.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:19.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:19.572 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:19.611 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:19.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:19.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:19.713 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:19.769 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:19.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:19.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:19.870 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:19.903 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:19.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:19.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:19.964 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:2024-09-13T14:55:19.964172+0000 min lat: 0.00116648 max lat: 2.46778 avg lat: 0.0113193 2024-09-13T14:55:19.964 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:55:19.964 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 60 16 84781 84765 88.2844 37.875 0.0237004 0.0113193 2024-09-13T14:55:20.005 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:20.046 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:20.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:20.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:20.148 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:20.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:20.208 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:20.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:20.309 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:20.348 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:20.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:20.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:20.450 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:20.479 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:20.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:20.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:20.581 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:20.603 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:55:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:55:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:55:20.606 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:20.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:20.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:20.708 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:20.758 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:20.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:20.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:20.861 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:20.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:20.895 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:20.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:20.964 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 61 16 85351 85335 87.4211 35.625 0.0239498 0.0114305 2024-09-13T14:55:20.997 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:21.022 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:21.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:21.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:21.049 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:20 smithi151 ceph-mon[104450]: pgmap v1917: 119 pgs: 3 remapped+peering, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 45 MiB/s wr, 716 op/s; 14636/250722 objects degraded (5.838%); 8241/250722 objects misplaced (3.287%); 1.1 MiB/s, 18 objects/s recovering 2024-09-13T14:55:21.050 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:20 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 14636/250722 objects degraded (5.838%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:21.050 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:20 smithi151 ceph-mon[108966]: pgmap v1917: 119 pgs: 3 remapped+peering, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 45 MiB/s wr, 716 op/s; 14636/250722 objects degraded (5.838%); 8241/250722 objects misplaced (3.287%); 1.1 MiB/s, 18 objects/s recovering 2024-09-13T14:55:21.050 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:20 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 14636/250722 objects degraded (5.838%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:21.124 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:21.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:21.164 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:21.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:21.265 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:21.290 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:21.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:21.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:21.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:20 smithi184 ceph-mon[92527]: pgmap v1917: 119 pgs: 3 remapped+peering, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 45 MiB/s wr, 716 op/s; 14636/250722 objects degraded (5.838%); 8241/250722 objects misplaced (3.287%); 1.1 MiB/s, 18 objects/s recovering 2024-09-13T14:55:21.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:20 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 14636/250722 objects degraded (5.838%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:21.393 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:21.439 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:55:21.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:21.484 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:21.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:21.585 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:21.637 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:21.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:21.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:21.740 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:21.762 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:21.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:21.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:21.864 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:21.887 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:21.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:21.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:21.964 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 62 16 85745 85729 86.4083 24.625 0.225954 0.0115351 2024-09-13T14:55:21.989 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:22.015 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:22.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:22.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:22.117 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:22.178 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:22.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:22.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:22.283 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:22.355 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:22.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:22.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:22.457 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:22.480 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:22.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:22.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:22.582 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:22.624 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:22.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:22.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:22.726 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:22.772 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:22.804 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:22.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:22.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:22.906 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:22.966 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 63 16 86267 86251 85.5528 32.625 0.0286411 0.0116813 2024-09-13T14:55:22.966 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:22.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:22.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:23.068 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:23.097 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:55:23.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:23.098 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:23.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:23.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:22 smithi184 ceph-mon[92527]: pgmap v1918: 119 pgs: 3 remapped+peering, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 104 active+clean; 5.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 29 MiB/s wr, 459 op/s; 33975/253041 objects degraded (13.427%); 9462/253041 objects misplaced (3.739%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:55:23.138 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:22 smithi151 ceph-mon[104450]: pgmap v1918: 119 pgs: 3 remapped+peering, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 104 active+clean; 5.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 29 MiB/s wr, 459 op/s; 33975/253041 objects degraded (13.427%); 9462/253041 objects misplaced (3.739%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:55:23.139 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:22 smithi151 ceph-mon[108966]: pgmap v1918: 119 pgs: 3 remapped+peering, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 104 active+clean; 5.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 29 MiB/s wr, 459 op/s; 33975/253041 objects degraded (13.427%); 9462/253041 objects misplaced (3.739%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:55:23.199 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:23.226 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:23.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:23.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:23.328 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:23.409 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:23.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:23.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:23.511 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:23.538 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:23.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:23.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:23.640 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:23.704 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:23.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:23.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:23.806 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:23.851 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:23.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:23.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:23.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:23.966 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 64 16 86802 86786 84.7385 33.4375 0.0492332 0.0117928 2024-09-13T14:55:24.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:24.005 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:24.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:24.107 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:24.152 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:24.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:24.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:24.254 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:24.307 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:24.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:24.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:24.351 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:55:24.351 INFO:teuthology.orchestra.run.smithi184.stdout: "size": 20, 2024-09-13T14:55:24.351 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 600, 2024-09-13T14:55:24.351 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [ 2024-09-13T14:55:24.351 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.351 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176106 8.5 8:a0ea7fc3:::benchmark_data_smithi151_229097_object51238:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1185)", 2024-09-13T14:55:24.352 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.478783+0000", 2024-09-13T14:55:24.352 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 197.018573632, 2024-09-13T14:55:24.352 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0704500290000001, 2024-09-13T14:55:24.352 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:24.352 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "reached pg", 2024-09-13T14:55:24.352 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:24.352 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:55:24.352 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:55:24.352 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176106 2024-09-13T14:55:24.352 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.352 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:24.352 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.353 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:24.353 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.478783+0000", 2024-09-13T14:55:24.353 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.353 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.353 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.353 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:24.353 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.478783+0000", 2024-09-13T14:55:24.353 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.353 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.353 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.353 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:24.353 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.478781+0000", 2024-09-13T14:55:24.353 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:55:24.354 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.354 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.354 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:24.354 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.478788+0000", 2024-09-13T14:55:24.354 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.1450000000000002e-06 2024-09-13T14:55:24.354 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.354 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.354 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:24.354 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.478805+0000", 2024-09-13T14:55:24.354 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6324000000000002e-05 2024-09-13T14:55:24.354 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.354 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.354 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:24.355 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.478812+0000", 2024-09-13T14:55:24.355 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.9059999999999996e-06 2024-09-13T14:55:24.355 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.355 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.355 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.355 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.478866+0000", 2024-09-13T14:55:24.355 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4895000000000002e-05 2024-09-13T14:55:24.355 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.355 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.355 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:24.355 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.479030+0000", 2024-09-13T14:55:24.355 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000163855 2024-09-13T14:55:24.355 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.356 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.356 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,4,7", 2024-09-13T14:55:24.356 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.479187+0000", 2024-09-13T14:55:24.356 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000157152 2024-09-13T14:55:24.356 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.356 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.356 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.356 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.485019+0000", 2024-09-13T14:55:24.356 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0058318290000000002 2024-09-13T14:55:24.356 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.356 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.357 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:55:24.357 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.485030+0000", 2024-09-13T14:55:24.357 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0482999999999999e-05 2024-09-13T14:55:24.357 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.357 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.357 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.357 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.511091+0000", 2024-09-13T14:55:24.357 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.026061415 2024-09-13T14:55:24.357 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.357 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.357 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for missing object", 2024-09-13T14:55:24.357 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.511214+0000", 2024-09-13T14:55:24.357 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012329499999999999 2024-09-13T14:55:24.358 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.358 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.358 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.358 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549141+0000", 2024-09-13T14:55:24.358 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.037926540000000002 2024-09-13T14:55:24.358 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.358 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.358 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:55:24.358 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549233+0000", 2024-09-13T14:55:24.358 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.1943999999999999e-05 2024-09-13T14:55:24.358 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.358 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:24.359 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.359 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.359 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.359 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176132 8.5 8:a8107e76:::benchmark_data_smithi151_229097_object51264:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1186)", 2024-09-13T14:55:24.359 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.508435+0000", 2024-09-13T14:55:24.359 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 196.98892119300001, 2024-09-13T14:55:24.359 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0887406580000001, 2024-09-13T14:55:24.359 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:24.359 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:24.359 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:24.359 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:55:24.359 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:55:24.359 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176132 2024-09-13T14:55:24.359 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.359 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:24.360 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.360 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:24.360 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.508435+0000", 2024-09-13T14:55:24.360 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.360 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.360 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.361 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:24.361 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.508435+0000", 2024-09-13T14:55:24.361 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.361 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.361 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.361 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:24.361 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.508433+0000", 2024-09-13T14:55:24.361 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:55:24.361 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.361 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.361 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:24.361 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.508441+0000", 2024-09-13T14:55:24.362 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.6699999999999994e-06 2024-09-13T14:55:24.362 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.362 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.362 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:24.362 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.508461+0000", 2024-09-13T14:55:24.362 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.0315000000000001e-05 2024-09-13T14:55:24.362 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.362 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.362 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:24.362 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.508469+0000", 2024-09-13T14:55:24.362 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.7610000000000007e-06 2024-09-13T14:55:24.362 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.362 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.363 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.363 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.508522+0000", 2024-09-13T14:55:24.363 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3053000000000002e-05 2024-09-13T14:55:24.363 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.363 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.363 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:55:24.363 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.508531+0000", 2024-09-13T14:55:24.363 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.2529999999999993e-06 2024-09-13T14:55:24.363 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.363 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.364 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.364 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.511245+0000", 2024-09-13T14:55:24.364 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.002713752 2024-09-13T14:55:24.364 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.364 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.365 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:24.365 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.511318+0000", 2024-09-13T14:55:24.365 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.2512e-05 2024-09-13T14:55:24.365 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.366 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.366 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:55:24.366 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.511392+0000", 2024-09-13T14:55:24.366 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.4664000000000002e-05 2024-09-13T14:55:24.366 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.366 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.366 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:55:24.366 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.519958+0000", 2024-09-13T14:55:24.366 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0085655379999999993 2024-09-13T14:55:24.366 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.366 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.366 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.366 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.548866+0000", 2024-09-13T14:55:24.367 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.028907676 2024-09-13T14:55:24.367 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.367 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.367 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.367 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.597007+0000", 2024-09-13T14:55:24.367 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.048141679999999999 2024-09-13T14:55:24.367 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.367 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.367 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:55:24.367 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.597062+0000", 2024-09-13T14:55:24.367 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4799000000000002e-05 2024-09-13T14:55:24.367 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.367 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.367 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:55:24.368 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.597176+0000", 2024-09-13T14:55:24.368 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00011391399999999999 2024-09-13T14:55:24.368 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.368 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:24.368 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.368 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.368 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.368 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176144 8.5 8:a9c768c7:::benchmark_data_smithi151_229097_object51276:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1186)", 2024-09-13T14:55:24.368 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.520194+0000", 2024-09-13T14:55:24.368 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 196.97716265599999, 2024-09-13T14:55:24.368 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0820666210000001, 2024-09-13T14:55:24.368 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:24.369 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:24.369 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:24.369 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:55:24.369 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:55:24.369 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176144 2024-09-13T14:55:24.369 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.369 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:24.369 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.369 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:24.369 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.520194+0000", 2024-09-13T14:55:24.369 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.369 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.369 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.370 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:24.370 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.520194+0000", 2024-09-13T14:55:24.370 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.370 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.370 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.370 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:24.370 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.520192+0000", 2024-09-13T14:55:24.370 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:55:24.370 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.370 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.370 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:24.370 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.520202+0000", 2024-09-13T14:55:24.370 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.6059999999999992e-06 2024-09-13T14:55:24.370 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.371 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.371 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:24.371 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.520215+0000", 2024-09-13T14:55:24.371 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3455000000000001e-05 2024-09-13T14:55:24.371 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.371 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.371 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:24.371 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.520222+0000", 2024-09-13T14:55:24.371 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3149999999999997e-06 2024-09-13T14:55:24.371 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.371 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.371 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.371 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.520272+0000", 2024-09-13T14:55:24.371 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9869e-05 2024-09-13T14:55:24.372 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.372 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.372 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:55:24.372 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.520279+0000", 2024-09-13T14:55:24.372 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.6660000000000008e-06 2024-09-13T14:55:24.372 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.372 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.372 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.372 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.513595+0000", 2024-09-13T14:55:24.372 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.99331558900000005 2024-09-13T14:55:24.372 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.372 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.372 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:24.373 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.513707+0000", 2024-09-13T14:55:24.373 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000112247 2024-09-13T14:55:24.373 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.373 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.373 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:55:24.373 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.513842+0000", 2024-09-13T14:55:24.373 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013462500000000001 2024-09-13T14:55:24.373 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.373 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.373 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:55:24.373 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.524620+0000", 2024-09-13T14:55:24.373 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.010777927 2024-09-13T14:55:24.373 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.374 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.374 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.374 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549270+0000", 2024-09-13T14:55:24.374 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.024650726000000001 2024-09-13T14:55:24.374 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.374 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.374 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.374 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.602098+0000", 2024-09-13T14:55:24.374 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.052828167000000002 2024-09-13T14:55:24.374 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.374 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.374 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:55:24.374 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.602150+0000", 2024-09-13T14:55:24.375 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1554999999999997e-05 2024-09-13T14:55:24.375 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.375 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.375 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:55:24.375 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.602261+0000", 2024-09-13T14:55:24.375 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000110504 2024-09-13T14:55:24.375 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.375 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:24.375 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.375 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.375 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.375 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176145 8.5 8:a3632e40:::benchmark_data_smithi151_229097_object51277:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1186)", 2024-09-13T14:55:24.375 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.520581+0000", 2024-09-13T14:55:24.375 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 196.97677554000001, 2024-09-13T14:55:24.376 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0818620189999999, 2024-09-13T14:55:24.376 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:24.376 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:24.376 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:24.376 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:55:24.376 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:55:24.376 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176145 2024-09-13T14:55:24.376 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.376 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:24.376 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.376 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:24.376 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.520581+0000", 2024-09-13T14:55:24.376 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.377 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.377 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.377 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:24.377 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.520581+0000", 2024-09-13T14:55:24.377 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.377 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.377 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.377 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:24.377 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.520580+0000", 2024-09-13T14:55:24.377 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:55:24.377 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.377 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.377 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:24.377 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.520585+0000", 2024-09-13T14:55:24.378 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.2240000000000003e-06 2024-09-13T14:55:24.378 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.378 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.378 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:24.378 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.520596+0000", 2024-09-13T14:55:24.378 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0985000000000001e-05 2024-09-13T14:55:24.378 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.378 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.378 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:24.378 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.520602+0000", 2024-09-13T14:55:24.378 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3870000000000003e-06 2024-09-13T14:55:24.378 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.378 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.379 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.379 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.521409+0000", 2024-09-13T14:55:24.379 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00080681600000000002 2024-09-13T14:55:24.379 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.379 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.379 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:55:24.379 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.521415+0000", 2024-09-13T14:55:24.379 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.8290000000000002e-06 2024-09-13T14:55:24.379 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.379 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.379 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.379 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.515883+0000", 2024-09-13T14:55:24.379 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.99446738599999995 2024-09-13T14:55:24.379 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.380 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.380 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:24.380 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.515999+0000", 2024-09-13T14:55:24.380 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000116388 2024-09-13T14:55:24.380 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.380 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.380 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:55:24.380 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.516074+0000", 2024-09-13T14:55:24.380 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.4924000000000003e-05 2024-09-13T14:55:24.380 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.380 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.380 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:55:24.380 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.524628+0000", 2024-09-13T14:55:24.381 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0085543259999999992 2024-09-13T14:55:24.381 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.381 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.381 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.381 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549302+0000", 2024-09-13T14:55:24.381 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.024673249000000001 2024-09-13T14:55:24.381 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.381 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.381 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.381 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.602311+0000", 2024-09-13T14:55:24.381 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.053008936 2024-09-13T14:55:24.381 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.381 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.382 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:55:24.382 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.602344+0000", 2024-09-13T14:55:24.382 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3071000000000002e-05 2024-09-13T14:55:24.382 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.382 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.382 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:55:24.382 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.602443+0000", 2024-09-13T14:55:24.382 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.9426000000000007e-05 2024-09-13T14:55:24.382 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.382 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:24.382 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.382 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.383 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.383 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176149 8.5 8:ad876389:::benchmark_data_smithi151_229097_object51281:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1186)", 2024-09-13T14:55:24.383 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.522270+0000", 2024-09-13T14:55:24.383 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 196.97508633199999, 2024-09-13T14:55:24.383 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0803534560000001, 2024-09-13T14:55:24.383 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:24.383 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:24.383 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:24.383 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:55:24.383 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:55:24.383 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176149 2024-09-13T14:55:24.383 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.383 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:24.384 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.384 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:24.384 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.522270+0000", 2024-09-13T14:55:24.384 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.384 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.384 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.384 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:24.384 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.522270+0000", 2024-09-13T14:55:24.384 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.384 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.384 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.384 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:24.384 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.522269+0000", 2024-09-13T14:55:24.385 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:55:24.385 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.385 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.385 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:24.385 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.522277+0000", 2024-09-13T14:55:24.385 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.5159999999999997e-06 2024-09-13T14:55:24.385 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.385 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.385 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:24.385 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.522292+0000", 2024-09-13T14:55:24.385 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4927e-05 2024-09-13T14:55:24.385 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.386 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.386 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:24.386 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.522297+0000", 2024-09-13T14:55:24.386 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4809999999999997e-06 2024-09-13T14:55:24.386 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.386 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.386 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.386 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.522335+0000", 2024-09-13T14:55:24.386 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.7545000000000001e-05 2024-09-13T14:55:24.386 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.386 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.387 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:55:24.387 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.522341+0000", 2024-09-13T14:55:24.387 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.0460000000000002e-06 2024-09-13T14:55:24.387 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.387 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.387 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.387 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.517970+0000", 2024-09-13T14:55:24.387 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.99562924100000005 2024-09-13T14:55:24.387 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.387 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.387 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:24.387 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.518057+0000", 2024-09-13T14:55:24.387 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.6854000000000006e-05 2024-09-13T14:55:24.388 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.388 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.388 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:55:24.388 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.518128+0000", 2024-09-13T14:55:24.388 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.1171e-05 2024-09-13T14:55:24.388 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.388 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.388 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:55:24.388 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.529279+0000", 2024-09-13T14:55:24.388 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.011150764000000001 2024-09-13T14:55:24.388 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.388 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.388 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.389 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549331+0000", 2024-09-13T14:55:24.389 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.020052303 2024-09-13T14:55:24.389 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.389 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.389 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.389 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.602480+0000", 2024-09-13T14:55:24.389 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.053148753999999999 2024-09-13T14:55:24.389 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.389 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.389 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:55:24.389 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.602524+0000", 2024-09-13T14:55:24.389 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4388000000000003e-05 2024-09-13T14:55:24.389 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.389 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.390 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:55:24.390 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.602624+0000", 2024-09-13T14:55:24.390 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.9468000000000004e-05 2024-09-13T14:55:24.390 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.390 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:24.390 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.390 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.390 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.390 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176153 8.5 8:aba5861e:::benchmark_data_smithi151_229097_object51285:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1186)", 2024-09-13T14:55:24.390 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.525003+0000", 2024-09-13T14:55:24.390 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 196.97235361099999, 2024-09-13T14:55:24.390 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0909296319999999, 2024-09-13T14:55:24.390 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:24.390 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:24.391 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:24.391 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:55:24.391 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:55:24.391 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176153 2024-09-13T14:55:24.391 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.391 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:24.391 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.391 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:24.391 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.525003+0000", 2024-09-13T14:55:24.391 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.391 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.392 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.392 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:24.392 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.525003+0000", 2024-09-13T14:55:24.392 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.392 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.392 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.392 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:24.392 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.525001+0000", 2024-09-13T14:55:24.392 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:55:24.392 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.392 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.392 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:24.392 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.525009+0000", 2024-09-13T14:55:24.393 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.7230000000000004e-06 2024-09-13T14:55:24.393 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.393 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.393 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:24.393 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.525028+0000", 2024-09-13T14:55:24.393 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.8882999999999999e-05 2024-09-13T14:55:24.393 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.393 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.393 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:24.393 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.525034+0000", 2024-09-13T14:55:24.393 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.8020000000000003e-06 2024-09-13T14:55:24.393 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.393 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.393 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.394 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.525084+0000", 2024-09-13T14:55:24.394 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0086999999999997e-05 2024-09-13T14:55:24.394 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.394 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.394 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:55:24.394 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.525091+0000", 2024-09-13T14:55:24.394 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.6789999999999999e-06 2024-09-13T14:55:24.394 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.394 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.394 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.394 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.519980+0000", 2024-09-13T14:55:24.394 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.99488926600000005 2024-09-13T14:55:24.394 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.395 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.395 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:24.395 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.520088+0000", 2024-09-13T14:55:24.395 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00010715800000000001 2024-09-13T14:55:24.395 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.395 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.395 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:55:24.395 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.520221+0000", 2024-09-13T14:55:24.395 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013349799999999999 2024-09-13T14:55:24.395 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.395 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.395 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:55:24.395 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.529287+0000", 2024-09-13T14:55:24.396 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0090658609999999997 2024-09-13T14:55:24.396 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.396 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.396 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.396 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549357+0000", 2024-09-13T14:55:24.396 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.020070523999999999 2024-09-13T14:55:24.396 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.396 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.396 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.396 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.615738+0000", 2024-09-13T14:55:24.396 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.066380272000000004 2024-09-13T14:55:24.396 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.396 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.397 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:55:24.397 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.615801+0000", 2024-09-13T14:55:24.397 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3355999999999994e-05 2024-09-13T14:55:24.397 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.397 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.397 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:55:24.397 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.615933+0000", 2024-09-13T14:55:24.397 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013155099999999999 2024-09-13T14:55:24.397 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.397 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:24.397 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.397 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.397 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.398 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176156 8.5 8:ae72b766:::benchmark_data_smithi151_229097_object51288:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1186)", 2024-09-13T14:55:24.398 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.527548+0000", 2024-09-13T14:55:24.398 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 196.96980859000001, 2024-09-13T14:55:24.398 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0885104029999999, 2024-09-13T14:55:24.398 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:24.398 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:24.398 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:24.398 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:55:24.398 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:55:24.398 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176156 2024-09-13T14:55:24.398 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.398 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:24.398 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.399 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:24.399 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.527548+0000", 2024-09-13T14:55:24.399 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.399 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.399 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.399 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:24.399 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.527548+0000", 2024-09-13T14:55:24.399 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.399 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.399 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.399 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:24.399 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.527547+0000", 2024-09-13T14:55:24.399 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:55:24.399 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.400 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.400 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:24.400 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.527553+0000", 2024-09-13T14:55:24.400 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.601e-06 2024-09-13T14:55:24.400 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.400 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.400 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:24.400 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.527571+0000", 2024-09-13T14:55:24.400 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.8229000000000001e-05 2024-09-13T14:55:24.400 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.400 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.400 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:24.400 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.527578+0000", 2024-09-13T14:55:24.401 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.6889999999999998e-06 2024-09-13T14:55:24.401 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.401 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.401 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.401 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.527637+0000", 2024-09-13T14:55:24.401 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8743000000000003e-05 2024-09-13T14:55:24.401 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.401 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.401 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:55:24.401 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.527644+0000", 2024-09-13T14:55:24.401 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.2060000000000001e-06 2024-09-13T14:55:24.401 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.401 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.402 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.402 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.522614+0000", 2024-09-13T14:55:24.402 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.99496983999999999 2024-09-13T14:55:24.402 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.402 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.402 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:24.402 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.522740+0000", 2024-09-13T14:55:24.402 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012576699999999999 2024-09-13T14:55:24.402 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.402 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.402 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:55:24.402 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.522822+0000", 2024-09-13T14:55:24.402 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.2529999999999998e-05 2024-09-13T14:55:24.403 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.403 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.403 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:55:24.403 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.533089+0000", 2024-09-13T14:55:24.403 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.010266688 2024-09-13T14:55:24.403 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.403 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.403 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.403 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549385+0000", 2024-09-13T14:55:24.404 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.016296496000000001 2024-09-13T14:55:24.404 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.404 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.404 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.404 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.615972+0000", 2024-09-13T14:55:24.404 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.066586874000000004 2024-09-13T14:55:24.404 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.404 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.404 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:55:24.404 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.616005+0000", 2024-09-13T14:55:24.404 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3263999999999999e-05 2024-09-13T14:55:24.404 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.404 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.404 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:55:24.405 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.616058+0000", 2024-09-13T14:55:24.405 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.2939000000000003e-05 2024-09-13T14:55:24.405 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.405 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:24.405 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.405 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.405 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.405 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176165 8.5 8:acd013d5:::benchmark_data_smithi151_229097_object51297:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1186)", 2024-09-13T14:55:24.405 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.537453+0000", 2024-09-13T14:55:24.405 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 196.95990306600001, 2024-09-13T14:55:24.405 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0843085750000001, 2024-09-13T14:55:24.405 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:24.406 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:24.406 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:24.406 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:55:24.406 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:55:24.406 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176165 2024-09-13T14:55:24.406 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.406 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:24.406 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.406 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:24.406 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.537453+0000", 2024-09-13T14:55:24.406 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.406 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.406 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.406 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:24.407 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.537453+0000", 2024-09-13T14:55:24.407 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.407 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.407 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.407 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:24.407 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.537452+0000", 2024-09-13T14:55:24.407 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:55:24.407 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.407 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.407 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:24.407 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.537459+0000", 2024-09-13T14:55:24.407 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7510000000000002e-06 2024-09-13T14:55:24.407 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.408 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.408 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:24.408 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.537469+0000", 2024-09-13T14:55:24.408 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.9890000000000004e-06 2024-09-13T14:55:24.408 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.408 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.408 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:24.408 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.537473+0000", 2024-09-13T14:55:24.408 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4889999999999997e-06 2024-09-13T14:55:24.408 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.408 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.408 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.408 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.537508+0000", 2024-09-13T14:55:24.409 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.4484999999999999e-05 2024-09-13T14:55:24.409 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.409 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.409 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:55:24.409 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.537513+0000", 2024-09-13T14:55:24.409 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7309999999999998e-06 2024-09-13T14:55:24.409 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.409 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.409 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.409 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.524669+0000", 2024-09-13T14:55:24.409 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.98715564700000002 2024-09-13T14:55:24.409 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.409 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.410 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:24.410 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.524783+0000", 2024-09-13T14:55:24.410 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00011410599999999999 2024-09-13T14:55:24.410 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.410 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.410 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:55:24.410 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.524910+0000", 2024-09-13T14:55:24.410 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000126747 2024-09-13T14:55:24.410 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.410 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.410 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:55:24.410 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.533097+0000", 2024-09-13T14:55:24.410 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0081872630000000002 2024-09-13T14:55:24.410 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.411 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.411 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.411 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549417+0000", 2024-09-13T14:55:24.411 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.016319314000000001 2024-09-13T14:55:24.411 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.411 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.411 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.411 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.621591+0000", 2024-09-13T14:55:24.411 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.072174490999999993 2024-09-13T14:55:24.411 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.411 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.411 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:55:24.411 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.621653+0000", 2024-09-13T14:55:24.412 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.2175999999999996e-05 2024-09-13T14:55:24.412 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.412 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.412 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:55:24.412 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.621762+0000", 2024-09-13T14:55:24.412 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000108867 2024-09-13T14:55:24.412 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.412 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:24.412 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.412 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.412 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.412 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176168 8.5 8:a528538c:::benchmark_data_smithi151_229097_object51300:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1186)", 2024-09-13T14:55:24.412 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.538334+0000", 2024-09-13T14:55:24.413 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 196.959022398, 2024-09-13T14:55:24.413 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0836694440000001, 2024-09-13T14:55:24.413 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:24.413 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:24.413 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:24.413 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:55:24.413 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:55:24.413 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176168 2024-09-13T14:55:24.413 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.413 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:24.413 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.413 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:24.413 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.538334+0000", 2024-09-13T14:55:24.414 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.414 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.414 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.414 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:24.414 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.538334+0000", 2024-09-13T14:55:24.414 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.414 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.414 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.414 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:24.414 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.538334+0000", 2024-09-13T14:55:24.414 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:55:24.414 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.414 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.414 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:24.415 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.538337+0000", 2024-09-13T14:55:24.415 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8879999999999999e-06 2024-09-13T14:55:24.415 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.415 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.415 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:24.415 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.538348+0000", 2024-09-13T14:55:24.415 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.095e-05 2024-09-13T14:55:24.415 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.415 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.415 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:24.415 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.538352+0000", 2024-09-13T14:55:24.415 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.4000000000000001e-06 2024-09-13T14:55:24.415 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.416 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.416 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.416 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.538384+0000", 2024-09-13T14:55:24.416 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.1996000000000003e-05 2024-09-13T14:55:24.416 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.416 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.416 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:55:24.416 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.538389+0000", 2024-09-13T14:55:24.416 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8849999999999998e-06 2024-09-13T14:55:24.416 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.416 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.416 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.416 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.527222+0000", 2024-09-13T14:55:24.417 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.98883309200000002 2024-09-13T14:55:24.417 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.417 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.417 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:24.417 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.527302+0000", 2024-09-13T14:55:24.417 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0692000000000006e-05 2024-09-13T14:55:24.417 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.417 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.417 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:55:24.417 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.527381+0000", 2024-09-13T14:55:24.417 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.8848000000000001e-05 2024-09-13T14:55:24.417 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.417 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.417 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:55:24.418 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.536903+0000", 2024-09-13T14:55:24.418 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0095215600000000001 2024-09-13T14:55:24.418 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.418 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.418 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.418 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549447+0000", 2024-09-13T14:55:24.418 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.012544092 2024-09-13T14:55:24.418 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.418 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.418 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.418 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.621827+0000", 2024-09-13T14:55:24.418 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.072379599000000003 2024-09-13T14:55:24.418 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.418 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.419 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:55:24.419 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.621888+0000", 2024-09-13T14:55:24.419 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1741000000000004e-05 2024-09-13T14:55:24.419 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.419 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.419 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:55:24.419 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.622004+0000", 2024-09-13T14:55:24.419 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0001153 2024-09-13T14:55:24.419 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.419 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:24.419 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.419 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.419 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.419 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176178 8.5 8:a790fc8b:::benchmark_data_smithi151_229097_object51310:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1186)", 2024-09-13T14:55:24.420 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.549688+0000", 2024-09-13T14:55:24.420 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 196.94766827999999, 2024-09-13T14:55:24.420 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0725186920000001, 2024-09-13T14:55:24.420 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:24.420 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:24.420 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:24.420 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:55:24.420 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:55:24.420 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176178 2024-09-13T14:55:24.420 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.420 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:24.420 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.420 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:24.420 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.549688+0000", 2024-09-13T14:55:24.421 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.421 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.421 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.421 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:24.421 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.549688+0000", 2024-09-13T14:55:24.421 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.421 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.421 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.421 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:24.421 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.549687+0000", 2024-09-13T14:55:24.421 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:55:24.421 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.421 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.422 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:24.422 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.549692+0000", 2024-09-13T14:55:24.422 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5300000000000004e-06 2024-09-13T14:55:24.422 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.422 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.422 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:24.422 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.549703+0000", 2024-09-13T14:55:24.422 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0339999999999999e-05 2024-09-13T14:55:24.422 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.422 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.422 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:24.422 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.549707+0000", 2024-09-13T14:55:24.422 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.6759999999999998e-06 2024-09-13T14:55:24.422 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.423 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.423 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.423 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.549753+0000", 2024-09-13T14:55:24.423 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5949999999999999e-05 2024-09-13T14:55:24.423 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.423 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.423 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:55:24.423 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.549762+0000", 2024-09-13T14:55:24.423 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.2309999999999996e-06 2024-09-13T14:55:24.423 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.423 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.423 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.423 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.529331+0000", 2024-09-13T14:55:24.423 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.97956943299999999 2024-09-13T14:55:24.423 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.424 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.424 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:24.424 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.529426+0000", 2024-09-13T14:55:24.424 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.4649999999999997e-05 2024-09-13T14:55:24.424 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.424 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.424 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:55:24.424 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.529529+0000", 2024-09-13T14:55:24.424 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000103084 2024-09-13T14:55:24.424 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.424 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.424 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:55:24.424 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.536911+0000", 2024-09-13T14:55:24.424 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0073820359999999998 2024-09-13T14:55:24.425 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.425 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.425 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.425 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549476+0000", 2024-09-13T14:55:24.425 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.012565270999999999 2024-09-13T14:55:24.425 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.425 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.425 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.425 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.622062+0000", 2024-09-13T14:55:24.425 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.072586176000000002 2024-09-13T14:55:24.425 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.425 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.425 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:55:24.426 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.622112+0000", 2024-09-13T14:55:24.426 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.986e-05 2024-09-13T14:55:24.426 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.426 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.426 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:55:24.426 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.622207+0000", 2024-09-13T14:55:24.426 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.4876000000000004e-05 2024-09-13T14:55:24.426 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.426 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:24.426 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.426 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.426 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.426 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176217 8.5 8:a0066b24:::benchmark_data_smithi151_229097_object51349:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1186)", 2024-09-13T14:55:24.427 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.618234+0000", 2024-09-13T14:55:24.427 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 196.87912267799999, 2024-09-13T14:55:24.427 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.004108939, 2024-09-13T14:55:24.427 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:24.427 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:24.427 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:24.427 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:55:24.427 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:55:24.427 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176217 2024-09-13T14:55:24.427 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.427 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:24.427 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.427 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:24.427 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618234+0000", 2024-09-13T14:55:24.428 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.428 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.428 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.428 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:24.428 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618234+0000", 2024-09-13T14:55:24.428 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.428 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.428 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.428 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:24.428 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618232+0000", 2024-09-13T14:55:24.428 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:55:24.428 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.428 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.428 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:24.429 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618240+0000", 2024-09-13T14:55:24.429 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0639999999999994e-06 2024-09-13T14:55:24.429 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.429 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.429 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:24.429 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618252+0000", 2024-09-13T14:55:24.429 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1773000000000001e-05 2024-09-13T14:55:24.429 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.429 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.429 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:24.429 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618257+0000", 2024-09-13T14:55:24.429 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7820000000000001e-06 2024-09-13T14:55:24.429 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.429 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.430 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.430 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618302+0000", 2024-09-13T14:55:24.430 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5296999999999997e-05 2024-09-13T14:55:24.430 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.430 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.430 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:55:24.430 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618311+0000", 2024-09-13T14:55:24.430 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.7839999999999992e-06 2024-09-13T14:55:24.430 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.430 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.431 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.431 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.531326+0000", 2024-09-13T14:55:24.431 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.913014624 2024-09-13T14:55:24.431 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.431 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.431 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:24.431 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.531393+0000", 2024-09-13T14:55:24.431 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7386999999999997e-05 2024-09-13T14:55:24.431 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.431 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.431 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:55:24.431 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.531460+0000", 2024-09-13T14:55:24.431 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7428000000000006e-05 2024-09-13T14:55:24.432 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.432 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.432 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:55:24.432 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.540413+0000", 2024-09-13T14:55:24.432 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0089529420000000002 2024-09-13T14:55:24.432 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.432 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.432 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.432 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549505+0000", 2024-09-13T14:55:24.432 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0090913369999999997 2024-09-13T14:55:24.432 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.432 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.432 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.432 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.622257+0000", 2024-09-13T14:55:24.433 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.072752467000000001 2024-09-13T14:55:24.433 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.433 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.433 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:55:24.433 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.622287+0000", 2024-09-13T14:55:24.433 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.9539e-05 2024-09-13T14:55:24.433 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.433 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.433 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:55:24.433 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.622343+0000", 2024-09-13T14:55:24.433 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6140000000000001e-05 2024-09-13T14:55:24.433 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.433 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:24.433 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.434 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.434 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.434 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176218 8.5 8:a137a2c8:::benchmark_data_smithi151_229097_object51350:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1186)", 2024-09-13T14:55:24.434 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.618308+0000", 2024-09-13T14:55:24.434 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 196.87904817899999, 2024-09-13T14:55:24.434 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0269271740000001, 2024-09-13T14:55:24.434 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:24.434 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:24.434 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:24.434 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:55:24.434 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:55:24.434 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176218 2024-09-13T14:55:24.435 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.435 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:24.435 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.435 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:24.435 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618308+0000", 2024-09-13T14:55:24.435 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.435 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.435 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.435 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:24.435 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618308+0000", 2024-09-13T14:55:24.435 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.435 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.435 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.436 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:24.436 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618308+0000", 2024-09-13T14:55:24.436 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:55:24.436 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.436 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.436 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:24.436 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618311+0000", 2024-09-13T14:55:24.436 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.723e-06 2024-09-13T14:55:24.436 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.436 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.436 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:24.436 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618318+0000", 2024-09-13T14:55:24.436 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.1470000000000004e-06 2024-09-13T14:55:24.436 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.437 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.437 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:24.437 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618321+0000", 2024-09-13T14:55:24.437 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.0489999999999999e-06 2024-09-13T14:55:24.437 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.437 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.437 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.437 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618360+0000", 2024-09-13T14:55:24.437 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8761999999999999e-05 2024-09-13T14:55:24.437 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.437 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.437 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:55:24.437 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618366+0000", 2024-09-13T14:55:24.437 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1059999999999996e-06 2024-09-13T14:55:24.438 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.438 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.438 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.438 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.533123+0000", 2024-09-13T14:55:24.438 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.91475721600000004 2024-09-13T14:55:24.438 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.438 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.438 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:24.438 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.533193+0000", 2024-09-13T14:55:24.438 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.9752999999999994e-05 2024-09-13T14:55:24.438 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.438 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.438 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:55:24.439 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.533274+0000", 2024-09-13T14:55:24.439 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0891999999999997e-05 2024-09-13T14:55:24.439 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.439 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.439 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:55:24.439 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.540421+0000", 2024-09-13T14:55:24.439 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0071469419999999999 2024-09-13T14:55:24.439 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.439 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.439 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.439 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549533+0000", 2024-09-13T14:55:24.439 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0091122310000000002 2024-09-13T14:55:24.439 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.439 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.440 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.440 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.644975+0000", 2024-09-13T14:55:24.440 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.095442461000000006 2024-09-13T14:55:24.440 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.440 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.440 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:55:24.440 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.645120+0000", 2024-09-13T14:55:24.440 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000144647 2024-09-13T14:55:24.440 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.440 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.440 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:55:24.440 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.645236+0000", 2024-09-13T14:55:24.440 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000115633 2024-09-13T14:55:24.441 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.441 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:24.441 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.441 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.441 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.441 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176222 8.5 8:aefbc882:::benchmark_data_smithi151_229097_object51354:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1186)", 2024-09-13T14:55:24.441 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.621980+0000", 2024-09-13T14:55:24.441 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 196.87537703300001, 2024-09-13T14:55:24.441 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.023445849, 2024-09-13T14:55:24.441 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:24.441 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:24.441 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:24.441 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:55:24.441 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:55:24.442 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176222 2024-09-13T14:55:24.442 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.442 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:24.442 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.442 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:24.442 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.621980+0000", 2024-09-13T14:55:24.442 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.442 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.442 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.442 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:24.442 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.621980+0000", 2024-09-13T14:55:24.442 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.442 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.443 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.443 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:24.443 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.621978+0000", 2024-09-13T14:55:24.443 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:55:24.443 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.443 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.443 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:24.443 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.621986+0000", 2024-09-13T14:55:24.443 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.7589999999999997e-06 2024-09-13T14:55:24.443 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.443 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.443 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:24.443 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.621997+0000", 2024-09-13T14:55:24.443 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1202e-05 2024-09-13T14:55:24.444 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.444 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.444 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:24.444 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.622002+0000", 2024-09-13T14:55:24.444 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3880000000000002e-06 2024-09-13T14:55:24.444 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.444 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.444 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.444 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.622034+0000", 2024-09-13T14:55:24.444 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.2437000000000001e-05 2024-09-13T14:55:24.444 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.444 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.444 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:55:24.445 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.622039+0000", 2024-09-13T14:55:24.445 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.6739999999999996e-06 2024-09-13T14:55:24.445 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.445 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.445 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.445 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.535015+0000", 2024-09-13T14:55:24.445 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.91297645500000002 2024-09-13T14:55:24.445 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.445 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.445 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:24.445 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.535097+0000", 2024-09-13T14:55:24.445 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.1742000000000004e-05 2024-09-13T14:55:24.445 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.445 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.446 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:55:24.446 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.535208+0000", 2024-09-13T14:55:24.446 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000110535 2024-09-13T14:55:24.446 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.446 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.446 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:55:24.446 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.548717+0000", 2024-09-13T14:55:24.446 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.013509241999999999 2024-09-13T14:55:24.446 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.446 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.446 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.446 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549562+0000", 2024-09-13T14:55:24.446 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000845245 2024-09-13T14:55:24.446 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.447 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.447 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.447 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.645261+0000", 2024-09-13T14:55:24.447 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.095698762000000007 2024-09-13T14:55:24.447 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.447 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.447 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:55:24.447 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.645330+0000", 2024-09-13T14:55:24.447 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.9472000000000002e-05 2024-09-13T14:55:24.447 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.447 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.447 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:55:24.447 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.645425+0000", 2024-09-13T14:55:24.447 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.4977999999999998e-05 2024-09-13T14:55:24.447 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.448 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:24.448 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.448 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.448 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176237 8.5 8:ada8226f:::benchmark_data_smithi151_229097_object51369:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1186)", 2024-09-13T14:55:24.448 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.781570+0000", 2024-09-13T14:55:24.448 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 196.71578663099999, 2024-09-13T14:55:24.448 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.86405514800000005, 2024-09-13T14:55:24.448 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:24.448 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:24.448 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:24.448 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:55:24.448 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:55:24.448 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176237 2024-09-13T14:55:24.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.449 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:24.449 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.449 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:24.449 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.781570+0000", 2024-09-13T14:55:24.449 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.449 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.449 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.449 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:24.449 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.781570+0000", 2024-09-13T14:55:24.449 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.449 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.449 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.449 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:24.449 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.781567+0000", 2024-09-13T14:55:24.450 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999971 2024-09-13T14:55:24.450 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.450 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.450 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:24.450 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.781584+0000", 2024-09-13T14:55:24.450 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.7093000000000001e-05 2024-09-13T14:55:24.450 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.450 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.450 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:24.451 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.781624+0000", 2024-09-13T14:55:24.451 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0039000000000001e-05 2024-09-13T14:55:24.451 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.451 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.451 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:24.451 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.781639+0000", 2024-09-13T14:55:24.451 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4392999999999999e-05 2024-09-13T14:55:24.451 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.451 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.451 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.451 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.781740+0000", 2024-09-13T14:55:24.451 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000101418 2024-09-13T14:55:24.451 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.451 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.452 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:55:24.452 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.781755+0000", 2024-09-13T14:55:24.452 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4776e-05 2024-09-13T14:55:24.452 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.452 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.452 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.452 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.536949+0000", 2024-09-13T14:55:24.452 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.75519378100000001 2024-09-13T14:55:24.452 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.452 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.452 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:24.452 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.537045+0000", 2024-09-13T14:55:24.452 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.6409000000000005e-05 2024-09-13T14:55:24.453 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.453 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.453 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:55:24.453 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.537125+0000", 2024-09-13T14:55:24.453 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0263e-05 2024-09-13T14:55:24.453 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.453 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.453 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:55:24.453 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.548721+0000", 2024-09-13T14:55:24.453 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.011595611 2024-09-13T14:55:24.453 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.453 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.453 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.454 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549590+0000", 2024-09-13T14:55:24.454 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00086873600000000005 2024-09-13T14:55:24.454 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.454 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.454 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.454 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.645464+0000", 2024-09-13T14:55:24.454 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.095874803999999994 2024-09-13T14:55:24.454 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.454 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.454 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:55:24.454 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.645533+0000", 2024-09-13T14:55:24.454 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.8547999999999995e-05 2024-09-13T14:55:24.454 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.454 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.455 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:55:24.455 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.645625+0000", 2024-09-13T14:55:24.455 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.2176999999999995e-05 2024-09-13T14:55:24.455 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.455 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:24.455 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.455 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.455 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.455 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176259 8.5 8:afd58153:::benchmark_data_smithi151_229097_object51391:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1186)", 2024-09-13T14:55:24.455 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.831139+0000", 2024-09-13T14:55:24.455 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 196.66621745, 2024-09-13T14:55:24.455 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.81465061999999999, 2024-09-13T14:55:24.455 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:24.455 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:24.455 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:24.456 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:55:24.456 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:55:24.456 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176259 2024-09-13T14:55:24.456 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.456 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:24.456 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.456 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:24.456 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.831139+0000", 2024-09-13T14:55:24.456 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.456 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.456 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.456 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:24.456 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.831139+0000", 2024-09-13T14:55:24.457 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.457 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.457 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.457 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:24.457 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.831138+0000", 2024-09-13T14:55:24.457 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:55:24.457 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.457 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.457 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:24.457 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.831149+0000", 2024-09-13T14:55:24.457 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1037e-05 2024-09-13T14:55:24.457 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.457 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.458 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:24.458 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.831161+0000", 2024-09-13T14:55:24.458 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1969e-05 2024-09-13T14:55:24.458 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.460 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.461 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:24.461 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.831165+0000", 2024-09-13T14:55:24.461 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5770000000000004e-06 2024-09-13T14:55:24.461 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.461 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.461 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.461 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.831207+0000", 2024-09-13T14:55:24.461 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.189e-05 2024-09-13T14:55:24.461 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.461 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.462 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:55:24.462 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.831216+0000", 2024-09-13T14:55:24.462 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.7090000000000008e-06 2024-09-13T14:55:24.462 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.462 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.462 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.462 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.538743+0000", 2024-09-13T14:55:24.462 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.70752717899999995 2024-09-13T14:55:24.462 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.462 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.462 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:24.462 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.538861+0000", 2024-09-13T14:55:24.462 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00011845699999999999 2024-09-13T14:55:24.463 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.463 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.463 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:55:24.463 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.538998+0000", 2024-09-13T14:55:24.463 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013685999999999999 2024-09-13T14:55:24.463 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.463 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.463 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:55:24.463 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.548827+0000", 2024-09-13T14:55:24.463 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0098285239999999999 2024-09-13T14:55:24.463 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.463 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.463 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.463 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549617+0000", 2024-09-13T14:55:24.464 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00079065899999999998 2024-09-13T14:55:24.464 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.464 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.464 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.464 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.645663+0000", 2024-09-13T14:55:24.464 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.096045084000000003 2024-09-13T14:55:24.464 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.464 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.464 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:55:24.464 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.645708+0000", 2024-09-13T14:55:24.464 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5215e-05 2024-09-13T14:55:24.464 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.464 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.464 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:55:24.464 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.645790+0000", 2024-09-13T14:55:24.465 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.1987e-05 2024-09-13T14:55:24.465 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.465 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:24.465 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.465 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.465 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.465 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176290 8.5 8:af9a2a07:::benchmark_data_smithi151_229097_object51422:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1186)", 2024-09-13T14:55:24.465 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.946125+0000", 2024-09-13T14:55:24.465 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 196.55123177199999, 2024-09-13T14:55:24.465 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.69983572299999997, 2024-09-13T14:55:24.465 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:24.465 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:24.465 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:24.465 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:55:24.466 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:55:24.466 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176290 2024-09-13T14:55:24.466 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.466 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:24.466 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.466 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:24.466 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.946125+0000", 2024-09-13T14:55:24.466 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.466 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.466 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.466 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:24.466 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.946125+0000", 2024-09-13T14:55:24.466 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.466 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.466 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.467 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:24.467 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.946124+0000", 2024-09-13T14:55:24.467 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:55:24.467 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.467 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.467 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:24.467 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.946130+0000", 2024-09-13T14:55:24.467 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.5989999999999998e-06 2024-09-13T14:55:24.467 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.467 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.467 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:24.467 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.946142+0000", 2024-09-13T14:55:24.467 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2257e-05 2024-09-13T14:55:24.467 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.467 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.468 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:24.468 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.946147+0000", 2024-09-13T14:55:24.468 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4100000000000001e-06 2024-09-13T14:55:24.468 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.468 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.468 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.468 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.946184+0000", 2024-09-13T14:55:24.468 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.7633999999999999e-05 2024-09-13T14:55:24.468 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.468 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.468 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:55:24.468 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.946191+0000", 2024-09-13T14:55:24.468 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.652e-06 2024-09-13T14:55:24.469 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.469 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.469 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.469 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.540453+0000", 2024-09-13T14:55:24.469 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.59426235999999999 2024-09-13T14:55:24.469 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.469 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.469 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:24.469 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.540521+0000", 2024-09-13T14:55:24.469 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7968999999999999e-05 2024-09-13T14:55:24.469 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.469 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.469 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:55:24.470 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.540613+0000", 2024-09-13T14:55:24.470 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.1310000000000005e-05 2024-09-13T14:55:24.470 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.470 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.470 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:55:24.470 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.548831+0000", 2024-09-13T14:55:24.470 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.008218108 2024-09-13T14:55:24.470 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.470 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.470 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.470 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549644+0000", 2024-09-13T14:55:24.470 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00081339800000000001 2024-09-13T14:55:24.470 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.470 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.471 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.471 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.645820+0000", 2024-09-13T14:55:24.471 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.096175694000000006 2024-09-13T14:55:24.471 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.471 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.471 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:55:24.471 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.645856+0000", 2024-09-13T14:55:24.471 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.6072999999999998e-05 2024-09-13T14:55:24.471 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.471 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.471 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:55:24.471 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.645961+0000", 2024-09-13T14:55:24.471 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000104467 2024-09-13T14:55:24.471 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.472 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:24.472 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.472 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.472 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.472 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:242830 8.3 8:ce75ae96:::benchmark_data_smithi151_229097_object117962:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1209)", 2024-09-13T14:55:24.472 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:53:27.088442+0000", 2024-09-13T14:55:24.472 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 116.408914256, 2024-09-13T14:55:24.472 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.21789556800000001, 2024-09-13T14:55:24.472 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:24.472 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:24.472 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:24.472 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:55:24.472 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:55:24.472 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 242830 2024-09-13T14:55:24.473 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.473 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:24.473 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.473 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:24.473 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.088442+0000", 2024-09-13T14:55:24.473 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.473 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.473 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.473 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:24.473 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.088442+0000", 2024-09-13T14:55:24.473 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.473 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.473 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.474 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:24.474 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.088441+0000", 2024-09-13T14:55:24.474 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:55:24.474 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.474 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.474 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:24.474 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.088446+0000", 2024-09-13T14:55:24.474 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9150000000000004e-06 2024-09-13T14:55:24.474 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.474 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.474 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:24.474 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.088455+0000", 2024-09-13T14:55:24.474 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.1779999999999992e-06 2024-09-13T14:55:24.475 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.475 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.475 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:24.475 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.088459+0000", 2024-09-13T14:55:24.475 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8519999999999997e-06 2024-09-13T14:55:24.475 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.475 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.475 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.475 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.088497+0000", 2024-09-13T14:55:24.475 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.7601000000000001e-05 2024-09-13T14:55:24.475 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.475 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.475 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:24.475 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.088622+0000", 2024-09-13T14:55:24.476 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012506999999999999 2024-09-13T14:55:24.476 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.476 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.476 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2,3", 2024-09-13T14:55:24.476 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.088767+0000", 2024-09-13T14:55:24.476 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014500799999999999 2024-09-13T14:55:24.476 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.476 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.476 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:55:24.476 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.090579+0000", 2024-09-13T14:55:24.476 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0018122749999999999 2024-09-13T14:55:24.476 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.476 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.477 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.477 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.095175+0000", 2024-09-13T14:55:24.477 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0045959140000000004 2024-09-13T14:55:24.477 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.477 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.477 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.477 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.095460+0000", 2024-09-13T14:55:24.477 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00028527499999999998 2024-09-13T14:55:24.477 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.477 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.477 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.477 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.306235+0000", 2024-09-13T14:55:24.477 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.210775188 2024-09-13T14:55:24.477 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.478 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.478 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:55:24.478 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.306277+0000", 2024-09-13T14:55:24.478 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1529000000000002e-05 2024-09-13T14:55:24.478 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.478 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.478 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:55:24.478 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.306338+0000", 2024-09-13T14:55:24.478 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.0970000000000001e-05 2024-09-13T14:55:24.478 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.478 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:24.478 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.478 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.479 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.479 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:242848 8.3 8:c423cd6a:::benchmark_data_smithi151_229097_object117980:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1209)", 2024-09-13T14:55:24.479 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:53:27.104877+0000", 2024-09-13T14:55:24.479 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 116.39247985199999, 2024-09-13T14:55:24.479 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.218461392, 2024-09-13T14:55:24.479 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:24.479 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:24.479 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:24.479 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:55:24.479 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:55:24.479 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 242848 2024-09-13T14:55:24.479 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.479 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:24.479 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.480 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:24.480 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.104877+0000", 2024-09-13T14:55:24.480 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.480 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.480 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.480 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:24.480 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.104877+0000", 2024-09-13T14:55:24.480 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.480 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.480 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.480 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:24.480 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.104875+0000", 2024-09-13T14:55:24.480 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:55:24.480 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.481 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.481 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:24.481 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.104880+0000", 2024-09-13T14:55:24.481 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1900000000000003e-06 2024-09-13T14:55:24.481 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.481 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.481 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:24.481 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.104892+0000", 2024-09-13T14:55:24.481 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1664e-05 2024-09-13T14:55:24.481 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.481 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.481 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:24.481 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.104897+0000", 2024-09-13T14:55:24.481 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4500000000000003e-06 2024-09-13T14:55:24.482 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.482 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.482 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.482 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.104943+0000", 2024-09-13T14:55:24.482 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5284000000000002e-05 2024-09-13T14:55:24.482 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.482 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.482 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:24.482 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.105057+0000", 2024-09-13T14:55:24.482 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00011414200000000001 2024-09-13T14:55:24.482 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.482 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.482 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2,3", 2024-09-13T14:55:24.483 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.105181+0000", 2024-09-13T14:55:24.483 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012442599999999999 2024-09-13T14:55:24.483 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.483 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.483 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:55:24.483 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.107094+0000", 2024-09-13T14:55:24.483 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0019128680000000001 2024-09-13T14:55:24.483 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.483 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.483 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.483 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.107743+0000", 2024-09-13T14:55:24.483 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00064880199999999997 2024-09-13T14:55:24.483 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.483 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.484 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.484 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.110272+0000", 2024-09-13T14:55:24.484 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0025289739999999998 2024-09-13T14:55:24.484 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.484 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.484 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.484 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.323241+0000", 2024-09-13T14:55:24.484 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.21296933700000001 2024-09-13T14:55:24.484 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.484 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.484 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:55:24.484 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.323271+0000", 2024-09-13T14:55:24.484 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.0009e-05 2024-09-13T14:55:24.484 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.485 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.485 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:55:24.485 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.323338+0000", 2024-09-13T14:55:24.485 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.6878000000000006e-05 2024-09-13T14:55:24.485 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.485 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:24.485 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.485 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.485 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.485 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:242856 8.3 8:cdfc04d4:::benchmark_data_smithi151_229097_object117988:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1209)", 2024-09-13T14:55:24.485 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:53:27.112383+0000", 2024-09-13T14:55:24.485 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 116.38497354, 2024-09-13T14:55:24.485 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.21109143, 2024-09-13T14:55:24.486 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:24.486 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:24.486 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:24.486 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:55:24.486 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:55:24.486 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 242856 2024-09-13T14:55:24.486 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.486 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:24.486 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.486 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:24.486 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.112383+0000", 2024-09-13T14:55:24.486 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.486 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.487 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.487 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:24.487 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.112383+0000", 2024-09-13T14:55:24.487 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.487 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.487 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.487 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:24.487 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.112382+0000", 2024-09-13T14:55:24.487 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:55:24.487 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.487 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.487 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:24.487 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.112389+0000", 2024-09-13T14:55:24.488 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.9779999999999999e-06 2024-09-13T14:55:24.488 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.488 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.488 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:24.488 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.112398+0000", 2024-09-13T14:55:24.488 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.1109999999999997e-06 2024-09-13T14:55:24.488 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.488 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.488 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:24.488 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.112402+0000", 2024-09-13T14:55:24.488 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1529999999999999e-06 2024-09-13T14:55:24.488 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.488 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.489 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.489 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.112439+0000", 2024-09-13T14:55:24.489 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.6874e-05 2024-09-13T14:55:24.489 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.489 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.489 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:24.489 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.112519+0000", 2024-09-13T14:55:24.489 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0414999999999995e-05 2024-09-13T14:55:24.489 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.489 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.489 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2,3", 2024-09-13T14:55:24.489 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.112600+0000", 2024-09-13T14:55:24.489 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0702000000000001e-05 2024-09-13T14:55:24.490 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.490 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.490 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:55:24.490 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.114201+0000", 2024-09-13T14:55:24.490 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0016011650000000001 2024-09-13T14:55:24.490 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.490 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.490 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.490 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.115123+0000", 2024-09-13T14:55:24.490 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00092186899999999997 2024-09-13T14:55:24.490 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.490 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.490 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.491 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.115150+0000", 2024-09-13T14:55:24.491 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.6780000000000001e-05 2024-09-13T14:55:24.491 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.491 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.491 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.491 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.323373+0000", 2024-09-13T14:55:24.491 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.20822268499999999 2024-09-13T14:55:24.491 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.491 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.491 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:55:24.491 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.323404+0000", 2024-09-13T14:55:24.491 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.1711999999999997e-05 2024-09-13T14:55:24.491 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.492 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.492 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:55:24.492 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.323474+0000", 2024-09-13T14:55:24.492 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.0078000000000002e-05 2024-09-13T14:55:24.492 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.492 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:24.492 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.492 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.492 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.492 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:242863 8.3 8:cbf9558a:::benchmark_data_smithi151_229097_object117995:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1209)", 2024-09-13T14:55:24.492 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:53:27.124767+0000", 2024-09-13T14:55:24.492 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 116.372589921, 2024-09-13T14:55:24.492 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.19882012600000001, 2024-09-13T14:55:24.493 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:24.493 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:24.493 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:24.493 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:55:24.493 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:55:24.493 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 242863 2024-09-13T14:55:24.493 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.493 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:24.493 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.493 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:24.493 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.124767+0000", 2024-09-13T14:55:24.493 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.493 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.494 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.494 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:24.494 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.124767+0000", 2024-09-13T14:55:24.494 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:24.494 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.494 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.494 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:24.494 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.124766+0000", 2024-09-13T14:55:24.494 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:55:24.494 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.494 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.494 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:24.494 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.124769+0000", 2024-09-13T14:55:24.495 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.6770000000000001e-06 2024-09-13T14:55:24.495 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.495 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.495 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:24.495 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.124778+0000", 2024-09-13T14:55:24.495 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.3999999999999992e-06 2024-09-13T14:55:24.495 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.495 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.495 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:24.495 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.124781+0000", 2024-09-13T14:55:24.495 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.535e-06 2024-09-13T14:55:24.495 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.495 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.496 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:24.496 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.125014+0000", 2024-09-13T14:55:24.496 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00023265300000000001 2024-09-13T14:55:24.496 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.496 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.496 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:24.496 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.125092+0000", 2024-09-13T14:55:24.496 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.8208999999999996e-05 2024-09-13T14:55:24.496 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.496 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.496 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2,3", 2024-09-13T14:55:24.496 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.125176+0000", 2024-09-13T14:55:24.496 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.3591999999999995e-05 2024-09-13T14:55:24.496 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.497 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.497 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:55:24.497 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.126841+0000", 2024-09-13T14:55:24.497 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0016657919999999999 2024-09-13T14:55:24.497 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.497 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.497 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.497 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.127135+0000", 2024-09-13T14:55:24.497 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00029333799999999998 2024-09-13T14:55:24.497 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.497 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.497 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.497 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.127743+0000", 2024-09-13T14:55:24.498 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00060855200000000005 2024-09-13T14:55:24.498 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.498 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.498 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:24.498 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.323507+0000", 2024-09-13T14:55:24.498 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.195763628 2024-09-13T14:55:24.498 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.498 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.498 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:55:24.498 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.323533+0000", 2024-09-13T14:55:24.498 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.6466999999999999e-05 2024-09-13T14:55:24.498 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:24.498 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:24.498 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:55:24.498 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:53:27.323587+0000", 2024-09-13T14:55:24.499 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3294000000000002e-05 2024-09-13T14:55:24.499 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.499 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:24.499 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.499 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:24.499 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:24.499 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:55:24.499 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:24.527 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:24.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:24.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:24.629 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:24.667 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:24.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:24.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:24.769 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:24.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:24.815 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:24.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:24.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:55:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:55:24] "GET /metrics HTTP/1.1" 200 36671 "" "Prometheus/2.51.0" 2024-09-13T14:55:24.916 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:24.945 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:24.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:24.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:24.966 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 65 16 87374 87358 83.9848 35.75 0.0253044 0.0118998 2024-09-13T14:55:25.047 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:25.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:25.075 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:25.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:25.132 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:24 smithi151 ceph-mon[108966]: pgmap v1919: 119 pgs: 2 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 27 MiB/s wr, 439 op/s; 44483/255102 objects degraded (17.437%); 14704/255102 objects misplaced (5.764%); 2.9 MiB/s, 46 objects/s recovering 2024-09-13T14:55:25.132 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:24 smithi151 ceph-mon[104450]: pgmap v1919: 119 pgs: 2 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 27 MiB/s wr, 439 op/s; 44483/255102 objects degraded (17.437%); 14704/255102 objects misplaced (5.764%); 2.9 MiB/s, 46 objects/s recovering 2024-09-13T14:55:25.157 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:25.176 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:25.212 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 6, 7, 1, 4] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:55:25.212 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:55:25.212 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-09-13T14:55:25.212 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd in 5 2024-09-13T14:55:25.224 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:25.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:25.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:25.326 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:25.350 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:25.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:25.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:25.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:24 smithi184 ceph-mon[92527]: pgmap v1919: 119 pgs: 2 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 27 MiB/s wr, 439 op/s; 44483/255102 objects degraded (17.437%); 14704/255102 objects misplaced (5.764%); 2.9 MiB/s, 46 objects/s recovering 2024-09-13T14:55:25.435 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T14:55:25.453 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:25.507 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:25.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:25.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:25.610 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:25.646 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:25.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:25.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:25.748 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:25.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:25.804 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:25.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:25.820 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:55:25.905 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:25.938 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:25.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:25.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:25.969 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 66 16 87895 87879 83.2033 32.5625 0.0183452 0.0120115 2024-09-13T14:55:26.040 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:26.061 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:26.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:26.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:26.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:25 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 44483/255102 objects degraded (17.437%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:26.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:25 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:55:26.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:25 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:55:26.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:25 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 44483/255102 objects degraded (17.437%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:26.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:25 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:55:26.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:25 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:55:26.140 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:25 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 44483/255102 objects degraded (17.437%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:26.140 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:25 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:55:26.140 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:25 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:55:26.163 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:26.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:26.208 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:26.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:26.309 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:26.336 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:26.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:26.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:26.437 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:26.462 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:26.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:26.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:26.564 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:26.601 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:26.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:26.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:26.702 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:26.726 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:26.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:26.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:26.828 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:26.856 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:26.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:26.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:26.959 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:26.968 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 67 16 88423 88407 82.4541 33 0.0413143 0.0121206 2024-09-13T14:55:26.982 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:26.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:26.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:27.004 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:26 smithi151 ceph-mon[104450]: pgmap v1920: 119 pgs: 2 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 37 MiB/s wr, 593 op/s; 44483/260490 objects degraded (17.077%); 14476/260490 objects misplaced (5.557%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:55:27.005 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:26 smithi151 ceph-mon[108966]: pgmap v1920: 119 pgs: 2 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 37 MiB/s wr, 593 op/s; 44483/260490 objects degraded (17.077%); 14476/260490 objects misplaced (5.557%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:55:27.083 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:27.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:26 smithi184 ceph-mon[92527]: pgmap v1920: 119 pgs: 2 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 37 MiB/s wr, 593 op/s; 44483/260490 objects degraded (17.077%); 14476/260490 objects misplaced (5.557%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:55:27.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:27.110 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:27.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:27.211 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:27.239 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:27.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:27.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:27.341 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:27.372 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:27.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:27.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:27.473 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:27.501 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:27.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:27.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:27.603 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:27.646 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:27.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:27.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:27.707 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:55:27.708 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [ 2024-09-13T14:55:27.708 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.708 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:88805 9.7 9:ec3b9393:::benchmark_data_smithi151_261954_object88804:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1234)", 2024-09-13T14:55:27.708 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:55:27.687995+0000", 2024-09-13T14:55:27.708 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.017103229000000001, 2024-09-13T14:55:27.708 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.017129819000000001, 2024-09-13T14:55:27.708 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:27.708 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:55:27.708 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:27.708 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T14:55:27.709 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:55:27.709 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 88805 2024-09-13T14:55:27.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.709 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:27.709 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.709 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:27.709 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.687995+0000", 2024-09-13T14:55:27.709 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:27.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.709 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.709 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:27.709 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.687995+0000", 2024-09-13T14:55:27.710 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:27.710 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.710 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.710 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:27.710 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.687986+0000", 2024-09-13T14:55:27.710 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999909 2024-09-13T14:55:27.710 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.710 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.710 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:27.710 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.688038+0000", 2024-09-13T14:55:27.710 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1659000000000001e-05 2024-09-13T14:55:27.710 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.710 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.710 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:27.711 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.688052+0000", 2024-09-13T14:55:27.711 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4127999999999999e-05 2024-09-13T14:55:27.711 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.711 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.711 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:27.711 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.688057+0000", 2024-09-13T14:55:27.711 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6149999999999996e-06 2024-09-13T14:55:27.711 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.711 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.711 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:27.711 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.688112+0000", 2024-09-13T14:55:27.711 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5087999999999998e-05 2024-09-13T14:55:27.711 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.711 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.712 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:27.712 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.688282+0000", 2024-09-13T14:55:27.712 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000169693 2024-09-13T14:55:27.712 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.712 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.712 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:55:27.712 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.688475+0000", 2024-09-13T14:55:27.712 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00019237599999999999 2024-09-13T14:55:27.712 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.712 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.712 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:55:27.712 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.691729+0000", 2024-09-13T14:55:27.713 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0032542999999999999 2024-09-13T14:55:27.713 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.713 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.713 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:27.713 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.700705+0000", 2024-09-13T14:55:27.713 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0089757459999999997 2024-09-13T14:55:27.713 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:27.713 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:27.713 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:27.713 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.713 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.713 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:88808 9.7 9:ef339006:::benchmark_data_smithi151_261954_object88807:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1234)", 2024-09-13T14:55:27.714 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:55:27.692702+0000", 2024-09-13T14:55:27.714 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.012396244000000001, 2024-09-13T14:55:27.714 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.012478502000000001, 2024-09-13T14:55:27.714 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:27.714 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:55:27.714 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:27.714 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T14:55:27.714 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:55:27.714 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 88808 2024-09-13T14:55:27.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.714 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:27.714 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.714 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:27.714 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.692702+0000", 2024-09-13T14:55:27.715 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:27.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.715 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.715 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:27.715 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.692702+0000", 2024-09-13T14:55:27.715 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:27.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.715 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.715 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:27.715 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.692700+0000", 2024-09-13T14:55:27.715 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:55:27.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.715 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.715 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:27.715 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.692744+0000", 2024-09-13T14:55:27.716 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3281000000000001e-05 2024-09-13T14:55:27.716 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.716 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.716 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:27.716 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.692759+0000", 2024-09-13T14:55:27.716 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.5007999999999999e-05 2024-09-13T14:55:27.716 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.716 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.716 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:27.716 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.692766+0000", 2024-09-13T14:55:27.716 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.1999999999999997e-06 2024-09-13T14:55:27.716 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.716 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.717 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:27.717 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.692812+0000", 2024-09-13T14:55:27.717 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.6218999999999999e-05 2024-09-13T14:55:27.717 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.717 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.717 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:27.717 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.692973+0000", 2024-09-13T14:55:27.717 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000160669 2024-09-13T14:55:27.717 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.717 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.717 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:55:27.717 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.693162+0000", 2024-09-13T14:55:27.717 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000189746 2024-09-13T14:55:27.717 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.718 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.718 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:55:27.718 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.696105+0000", 2024-09-13T14:55:27.718 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0029421009999999999 2024-09-13T14:55:27.718 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.718 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.718 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:27.718 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.700994+0000", 2024-09-13T14:55:27.718 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0048892149999999997 2024-09-13T14:55:27.718 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:27.718 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:27.718 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:27.718 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.719 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.719 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:88810 9.d 9:b9edcc19:::benchmark_data_smithi151_261954_object88809:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1234)", 2024-09-13T14:55:27.719 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:55:27.696087+0000", 2024-09-13T14:55:27.719 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.0090116569999999993, 2024-09-13T14:55:27.719 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0091398590000000002, 2024-09-13T14:55:27.719 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:27.719 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:55:27.719 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:27.719 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T14:55:27.719 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:55:27.719 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 88810 2024-09-13T14:55:27.719 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.719 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:27.720 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.720 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:27.720 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.696087+0000", 2024-09-13T14:55:27.720 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:27.720 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.720 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.720 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:27.720 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.696087+0000", 2024-09-13T14:55:27.720 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:27.720 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.721 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.721 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:27.721 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.696086+0000", 2024-09-13T14:55:27.721 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:55:27.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.721 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.721 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:27.721 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.696146+0000", 2024-09-13T14:55:27.721 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.0053000000000003e-05 2024-09-13T14:55:27.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.721 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.721 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:27.722 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.696154+0000", 2024-09-13T14:55:27.722 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.4849999999999992e-06 2024-09-13T14:55:27.722 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.722 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.722 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:27.722 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.696159+0000", 2024-09-13T14:55:27.722 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.113e-06 2024-09-13T14:55:27.723 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.723 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.723 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:27.723 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.696203+0000", 2024-09-13T14:55:27.723 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4008000000000003e-05 2024-09-13T14:55:27.723 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.723 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.723 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:27.723 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.696373+0000", 2024-09-13T14:55:27.723 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00016940100000000001 2024-09-13T14:55:27.723 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.724 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.724 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:55:27.724 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.696562+0000", 2024-09-13T14:55:27.724 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00018954599999999999 2024-09-13T14:55:27.724 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.724 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.724 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:55:27.724 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.702311+0000", 2024-09-13T14:55:27.724 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0057485940000000001 2024-09-13T14:55:27.724 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:27.724 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:27.724 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:27.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.725 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.725 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:88811 9.e 9:7bc5e31b:::benchmark_data_smithi151_261954_object88810:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1234)", 2024-09-13T14:55:27.725 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:55:27.696169+0000", 2024-09-13T14:55:27.725 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.0089293770000000005, 2024-09-13T14:55:27.725 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0091160860000000007, 2024-09-13T14:55:27.725 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:27.725 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:55:27.725 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:27.725 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T14:55:27.725 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:55:27.725 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 88811 2024-09-13T14:55:27.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.726 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:27.726 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.726 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:27.726 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.696169+0000", 2024-09-13T14:55:27.726 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:27.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.726 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.726 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:27.726 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.696169+0000", 2024-09-13T14:55:27.726 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:27.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.726 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.727 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:27.727 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.696168+0000", 2024-09-13T14:55:27.727 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:55:27.727 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.727 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.727 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:27.727 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.696234+0000", 2024-09-13T14:55:27.727 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.6047e-05 2024-09-13T14:55:27.727 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.727 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.727 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:27.727 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.696243+0000", 2024-09-13T14:55:27.727 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.4530000000000002e-06 2024-09-13T14:55:27.727 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.728 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.728 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:27.728 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.696248+0000", 2024-09-13T14:55:27.728 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8380000000000001e-06 2024-09-13T14:55:27.728 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.728 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.728 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:27.728 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.699440+0000", 2024-09-13T14:55:27.728 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0031924480000000001 2024-09-13T14:55:27.728 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.728 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.728 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:27.728 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.699616+0000", 2024-09-13T14:55:27.728 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00017637299999999999 2024-09-13T14:55:27.729 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.729 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.729 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:55:27.729 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.699799+0000", 2024-09-13T14:55:27.729 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00018296299999999999 2024-09-13T14:55:27.729 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:27.729 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:27.729 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:27.729 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.729 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.729 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:88812 9.d 9:b1d12568:::benchmark_data_smithi151_261954_object88811:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1234)", 2024-09-13T14:55:27.729 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:55:27.696257+0000", 2024-09-13T14:55:27.729 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.0088414189999999997, 2024-09-13T14:55:27.729 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0090750599999999994, 2024-09-13T14:55:27.730 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:27.730 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:55:27.730 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:27.730 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T14:55:27.730 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:55:27.730 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 88812 2024-09-13T14:55:27.730 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.730 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:27.730 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.730 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:27.730 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.696257+0000", 2024-09-13T14:55:27.730 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:27.730 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.730 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.730 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:27.731 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.696257+0000", 2024-09-13T14:55:27.731 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:27.731 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.731 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.731 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:27.731 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.696256+0000", 2024-09-13T14:55:27.731 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:55:27.731 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.731 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.731 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:27.731 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.696305+0000", 2024-09-13T14:55:27.731 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8779999999999997e-05 2024-09-13T14:55:27.731 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.731 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.731 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:27.732 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.696312+0000", 2024-09-13T14:55:27.732 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.1439999999999997e-06 2024-09-13T14:55:27.732 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.732 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.732 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:27.732 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.696316+0000", 2024-09-13T14:55:27.732 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3499999999999999e-06 2024-09-13T14:55:27.732 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.732 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.732 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:27.732 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.699529+0000", 2024-09-13T14:55:27.732 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0032125700000000001 2024-09-13T14:55:27.732 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.732 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.732 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:27.733 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.699657+0000", 2024-09-13T14:55:27.733 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012809099999999999 2024-09-13T14:55:27.733 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.733 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.733 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:55:27.733 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.699763+0000", 2024-09-13T14:55:27.733 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000105506 2024-09-13T14:55:27.733 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.733 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.733 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:55:27.733 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.702462+0000", 2024-09-13T14:55:27.733 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0026997179999999998 2024-09-13T14:55:27.733 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:27.733 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:27.733 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:88813 9.e 9:7cc7a935:::benchmark_data_smithi151_261954_object88812:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1234)", 2024-09-13T14:55:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:55:27.699270+0000", 2024-09-13T14:55:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.0058279170000000002, 2024-09-13T14:55:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0061312709999999998, 2024-09-13T14:55:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:55:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T14:55:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:55:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 88813 2024-09-13T14:55:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:27.735 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.735 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:27.735 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.699270+0000", 2024-09-13T14:55:27.735 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:27.735 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.735 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.735 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:27.735 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.699270+0000", 2024-09-13T14:55:27.735 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:27.735 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.735 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.735 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:27.736 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.699268+0000", 2024-09-13T14:55:27.736 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:55:27.736 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.736 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.736 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:27.736 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.699301+0000", 2024-09-13T14:55:27.736 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3040000000000002e-05 2024-09-13T14:55:27.736 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.736 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.736 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:27.736 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.699314+0000", 2024-09-13T14:55:27.736 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2836999999999999e-05 2024-09-13T14:55:27.736 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.737 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.737 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:27.737 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.699336+0000", 2024-09-13T14:55:27.737 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.1671999999999999e-05 2024-09-13T14:55:27.737 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.737 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.737 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:27.737 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.702668+0000", 2024-09-13T14:55:27.737 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0033324000000000001 2024-09-13T14:55:27.737 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.737 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.737 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:27.737 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.702819+0000", 2024-09-13T14:55:27.738 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00015087799999999999 2024-09-13T14:55:27.738 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.738 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.738 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:55:27.738 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.703013+0000", 2024-09-13T14:55:27.738 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000194253 2024-09-13T14:55:27.738 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:27.738 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:27.738 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:27.738 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.738 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.738 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop_reply(client.46034.0:88810 9.d e1234/1233)", 2024-09-13T14:55:27.738 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:55:27.705408+0000", 2024-09-13T14:55:27.738 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 4294967295.9996905, 2024-09-13T14:55:27.739 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1977000000000001e-05, 2024-09-13T14:55:27.739 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:27.739 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "queued for pg", 2024-09-13T14:55:27.739 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:27.739 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.739 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:27.739 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.705408+0000", 2024-09-13T14:55:27.739 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:27.739 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.739 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.739 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:27.739 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.705408+0000", 2024-09-13T14:55:27.739 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:27.740 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.740 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.740 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:27.740 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.705407+0000", 2024-09-13T14:55:27.740 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:55:27.740 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.740 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.740 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:27.740 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.705416+0000", 2024-09-13T14:55:27.740 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.7139999999999999e-06 2024-09-13T14:55:27.740 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.740 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.740 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.705427+0000", 2024-09-13T14:55:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1627999999999999e-05 2024-09-13T14:55:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.705433+0000", 2024-09-13T14:55:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.9000000000000003e-06 2024-09-13T14:55:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:88795 9.7 9:e7a606ce:::benchmark_data_smithi151_261954_object88794:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1234)", 2024-09-13T14:55:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:55:27.670115+0000", 2024-09-13T14:55:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.034983272000000003, 2024-09-13T14:55:27.742 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.035392158, 2024-09-13T14:55:27.742 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:55:27.742 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:55:27.742 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:55:27.742 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T14:55:27.742 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:55:27.742 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 88795 2024-09-13T14:55:27.742 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.742 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:55:27.742 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.742 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:55:27.742 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.670115+0000", 2024-09-13T14:55:27.742 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:27.743 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.743 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.743 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:55:27.743 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.670115+0000", 2024-09-13T14:55:27.743 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:55:27.743 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.743 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.743 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:55:27.743 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.670113+0000", 2024-09-13T14:55:27.743 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:55:27.743 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.743 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.743 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:55:27.743 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.670142+0000", 2024-09-13T14:55:27.744 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.9363e-05 2024-09-13T14:55:27.744 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.744 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.744 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:55:27.744 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.670158+0000", 2024-09-13T14:55:27.744 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.5783000000000001e-05 2024-09-13T14:55:27.744 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.744 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.744 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:55:27.744 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.670165+0000", 2024-09-13T14:55:27.744 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.5899999999999996e-06 2024-09-13T14:55:27.744 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.744 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.745 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:55:27.745 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.670206+0000", 2024-09-13T14:55:27.745 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1325000000000002e-05 2024-09-13T14:55:27.745 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.745 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.745 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:55:27.745 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.670324+0000", 2024-09-13T14:55:27.745 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00011756600000000001 2024-09-13T14:55:27.745 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.745 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.745 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:55:27.745 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.670451+0000", 2024-09-13T14:55:27.745 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012722800000000001 2024-09-13T14:55:27.745 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.746 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.746 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:55:27.746 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.673342+0000", 2024-09-13T14:55:27.746 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002891336 2024-09-13T14:55:27.746 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:55:27.746 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:55:27.746 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:27.746 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:27.677026+0000", 2024-09-13T14:55:27.746 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.003684292 2024-09-13T14:55:27.746 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:27.746 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:55:27.746 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:27.746 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:55:27.746 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-13T14:55:27.747 INFO:teuthology.orchestra.run.smithi184.stdout: "num_ops": 8 2024-09-13T14:55:27.747 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:55:27.748 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:27.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:27.792 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:27.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:27.893 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:27.905 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-13T14:55:27.922 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:27.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:27.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:27.969 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 68 16 88940 88924 81.7167 32.3125 0.0174313 0.0122274 2024-09-13T14:55:28.023 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:28.047 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:28.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:28.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:28.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:27 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3444917563' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:28.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:27 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3444917563' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:28.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:27 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3444917563' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:28.149 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:28.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:28.185 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:28.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:28.286 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:28.309 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:28.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:28.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:28.411 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:28.453 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:28.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:28.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:28.545 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T14:55:28.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:28.600 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:28.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:28.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:28.702 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:28.748 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:28.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:28.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:28.850 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:28.880 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:55:28.882 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:28.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:28.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:28.968 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 69 16 89520 89504 81.0578 36.25 0.0452001 0.0123291 2024-09-13T14:55:28.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:29.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:29.015 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:29.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:29.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:28 smithi151 ceph-mon[104450]: pgmap v1921: 119 pgs: 2 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 31 MiB/s wr, 489 op/s; 44483/262989 objects degraded (16.914%); 14347/262989 objects misplaced (5.455%); 4.5 MiB/s, 71 objects/s recovering 2024-09-13T14:55:29.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:28 smithi151 ceph-mon[104450]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:55:29.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:28 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3444917563' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:55:29.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:28 smithi151 ceph-mon[104450]: osdmap e1235: 8 total, 8 up, 7 in 2024-09-13T14:55:29.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:28 smithi151 ceph-mon[108966]: pgmap v1921: 119 pgs: 2 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 31 MiB/s wr, 489 op/s; 44483/262989 objects degraded (16.914%); 14347/262989 objects misplaced (5.455%); 4.5 MiB/s, 71 objects/s recovering 2024-09-13T14:55:29.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:28 smithi151 ceph-mon[108966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:55:29.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:28 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3444917563' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:55:29.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:28 smithi151 ceph-mon[108966]: osdmap e1235: 8 total, 8 up, 7 in 2024-09-13T14:55:29.116 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:29.154 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:29.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:29.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:29.168 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:28 smithi184 ceph-mon[92527]: pgmap v1921: 119 pgs: 2 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 31 MiB/s wr, 489 op/s; 44483/262989 objects degraded (16.914%); 14347/262989 objects misplaced (5.455%); 4.5 MiB/s, 71 objects/s recovering 2024-09-13T14:55:29.168 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:28 smithi184 ceph-mon[92527]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:55:29.168 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:28 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3444917563' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:55:29.168 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:28 smithi184 ceph-mon[92527]: osdmap e1235: 8 total, 8 up, 7 in 2024-09-13T14:55:29.256 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:29.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:29.294 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:29.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:29.395 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:29.446 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:29.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:29.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:29.547 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:29.595 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:29.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:29.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:29.697 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:29.762 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:29.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:29.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:29.864 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:29.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:29.938 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:29.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:29.972 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 70 16 90058 90042 80.3787 33.625 0.0379213 0.0124339 2024-09-13T14:55:30.040 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:30.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:30.080 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:30.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:30.181 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:30.222 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:30.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:30.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:30.324 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:30.352 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:30.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:30.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:30.454 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:30.478 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:30.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:30.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:30.509 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:55:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:55:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:55:30.580 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:30.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:30 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:55:30.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:30 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:55:30.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:30 smithi151 ceph-mon[108966]: pgmap v1923: 119 pgs: 2 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 30 MiB/s wr, 475 op/s; 44483/264981 objects degraded (16.787%); 14223/264981 objects misplaced (5.368%); 5.2 MiB/s, 83 objects/s recovering 2024-09-13T14:55:30.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:30 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 44483/262989 objects degraded (16.914%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:30.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:30 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:55:30.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:30 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:55:30.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:30 smithi151 ceph-mon[104450]: pgmap v1923: 119 pgs: 2 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 30 MiB/s wr, 475 op/s; 44483/264981 objects degraded (16.787%); 14223/264981 objects misplaced (5.368%); 5.2 MiB/s, 83 objects/s recovering 2024-09-13T14:55:30.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:30 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 44483/262989 objects degraded (16.914%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:30.638 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:30.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:30.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:30.740 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:30.765 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:30.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:30.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:30.852 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:30 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:55:30.852 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:30 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:55:30.852 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:30 smithi184 ceph-mon[92527]: pgmap v1923: 119 pgs: 2 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 30 MiB/s wr, 475 op/s; 44483/264981 objects degraded (16.787%); 14223/264981 objects misplaced (5.368%); 5.2 MiB/s, 83 objects/s recovering 2024-09-13T14:55:30.852 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:30 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 44483/262989 objects degraded (16.914%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:30.862 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:55:30.862 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [], 2024-09-13T14:55:30.862 INFO:teuthology.orchestra.run.smithi184.stdout: "complaint_time": 30, 2024-09-13T14:55:30.862 INFO:teuthology.orchestra.run.smithi184.stdout: "num_blocked_ops": 0 2024-09-13T14:55:30.862 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:55:30.868 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:30.894 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:30.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:30.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:30.971 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 71 16 90582 90566 79.7069 32.75 0.00634032 0.0125388 2024-09-13T14:55:30.996 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:31.022 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:31.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:31.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:31.123 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:31.148 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:31.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:31.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:31.250 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:31.278 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:31.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:31.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:31.381 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:31.409 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:31.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:31.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:31.482 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:31.510 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:31.525 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T14:55:31.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:31.592 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:31.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:31.693 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:31.723 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:31.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:31.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:31.789 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:55:31.789 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:55:31.790 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:55:31.790 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:31.790 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:55:31.790 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:55:31.790 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:55:31.790 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:31.825 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:31.849 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:55:31.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:55:31.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:55:31.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:55:31.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:31.863 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:31.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:31.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:31.965 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:31.971 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 72 16 91114 91098 79.0617 33.25 0.0618823 0.0126413 2024-09-13T14:55:31.989 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:31.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:31.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:32.092 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:32.139 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:32.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:32.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:32.241 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:32.284 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:32.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:32.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:32.330 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:55:32.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:32.438 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:32.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:32.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:32.534 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:55:32.534 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:55:32.541 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:32.592 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:55:32.593 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:55:32.603 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:32.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:32.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:32.647 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:32 smithi151 ceph-mon[104450]: pgmap v1924: 119 pgs: 2 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 33 MiB/s wr, 521 op/s; 44483/268695 objects degraded (16.555%); 14139/268695 objects misplaced (5.262%); 4.1 MiB/s, 65 objects/s recovering 2024-09-13T14:55:32.648 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:32 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/315103683' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:32.648 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:32 smithi151 ceph-mon[108966]: pgmap v1924: 119 pgs: 2 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 33 MiB/s wr, 521 op/s; 44483/268695 objects degraded (16.555%); 14139/268695 objects misplaced (5.262%); 4.1 MiB/s, 65 objects/s recovering 2024-09-13T14:55:32.648 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:32 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/315103683' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:32.648 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:55:32.649 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:55:32.679 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:55:32.679 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:55:32.706 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:32.716 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:55:32.716 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:55:32.732 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:32.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:32.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:32.747 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:55:32.747 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:55:32.776 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:55:32.776 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:55:32.800 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:32 smithi184 ceph-mon[92527]: pgmap v1924: 119 pgs: 2 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 33 MiB/s wr, 521 op/s; 44483/268695 objects degraded (16.555%); 14139/268695 objects misplaced (5.262%); 4.1 MiB/s, 65 objects/s recovering 2024-09-13T14:55:32.800 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:32 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/315103683' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:32.814 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:55:32.814 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:55:32.834 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:32.860 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:32.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:32.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:32.962 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:32.971 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 73 16 91579 91563 78.3769 29.0625 0.125524 0.0127488 2024-09-13T14:55:32.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:32.993 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:32.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:33.094 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:33.124 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:33.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:33.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:33.226 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:33.248 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:33.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:33.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:33.350 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:33.378 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:33.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:33.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:33.479 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:33.508 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:33.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:33.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:33.610 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:33.653 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:33.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:33.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:33.743 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:55:33.743 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:55:33.755 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:33.802 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:33.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:33.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:33.904 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:33.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:33.948 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:33.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:33.971 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 74 16 92130 92114 77.7831 34.4375 0.0187587 0.0128485 2024-09-13T14:55:34.049 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:34.089 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:34.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:34.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:34.191 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:34.217 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:34.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:34.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:34.318 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:34.368 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:34.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:34.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:34.469 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:34.482 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:55:34.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:34.520 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:34.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:34.621 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:34.647 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:34.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:34.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:34.749 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:34.822 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:34.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:34.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:34.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:34 smithi151 ceph-mon[104450]: pgmap v1925: 119 pgs: 2 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 33 MiB/s wr, 535 op/s; 44483/271176 objects degraded (16.404%); 14002/271176 objects misplaced (5.163%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:55:34.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:55:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:55:34] "GET /metrics HTTP/1.1" 200 36682 "" "Prometheus/2.51.0" 2024-09-13T14:55:34.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:34 smithi151 ceph-mon[108966]: pgmap v1925: 119 pgs: 2 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 33 MiB/s wr, 535 op/s; 44483/271176 objects degraded (16.404%); 14002/271176 objects misplaced (5.163%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:55:34.925 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:34.969 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:34.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:34.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:34.971 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 75 16 92629 92613 77.1619 31.1875 0.0232052 0.0129537 2024-09-13T14:55:35.070 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:35.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:34 smithi184 ceph-mon[92527]: pgmap v1925: 119 pgs: 2 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 33 MiB/s wr, 535 op/s; 44483/271176 objects degraded (16.404%); 14002/271176 objects misplaced (5.163%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:55:35.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:35.134 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:35.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:35.234 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:35.274 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:35.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:35.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:35.376 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:35.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:35.400 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:35.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:35.501 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:35.528 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:35.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:35.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:35.630 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:35.652 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:35.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:35.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:35.754 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:35.775 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:35.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:35.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:35.877 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:35.903 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:35.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:35.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:35.971 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 76 16 93010 92994 76.46 23.8125 0.0435702 0.013072 2024-09-13T14:55:36.006 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:36.029 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:36.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:36.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:36.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:35 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 44483/271176 objects degraded (16.404%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:36.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:35 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 44483/271176 objects degraded (16.404%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:36.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:35 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 44483/271176 objects degraded (16.404%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:36.131 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:36.154 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:36.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:36.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:36.256 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:36.282 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:36.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:36.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:36.384 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:36.408 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:36.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:36.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:36.510 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:36.563 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:36.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:36.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:36.665 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:36.692 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:36.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:36.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:36.794 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:36.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:36.833 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:36.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:36.934 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:36.965 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:36.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:36.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:36.971 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 77 16 93576 93560 75.9264 35.375 0.0259342 0.013164 2024-09-13T14:55:37.066 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:37.092 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:37.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:37.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:37.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:36 smithi184 ceph-mon[92527]: pgmap v1926: 119 pgs: 2 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 33 MiB/s wr, 532 op/s; 44483/276486 objects degraded (16.089%); 13812/276486 objects misplaced (4.996%); 4.1 MiB/s, 66 objects/s recovering 2024-09-13T14:55:37.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:36 smithi151 ceph-mon[104450]: pgmap v1926: 119 pgs: 2 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 33 MiB/s wr, 532 op/s; 44483/276486 objects degraded (16.089%); 13812/276486 objects misplaced (4.996%); 4.1 MiB/s, 66 objects/s recovering 2024-09-13T14:55:37.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:36 smithi151 ceph-mon[108966]: pgmap v1926: 119 pgs: 2 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 33 MiB/s wr, 532 op/s; 44483/276486 objects degraded (16.089%); 13812/276486 objects misplaced (4.996%); 4.1 MiB/s, 66 objects/s recovering 2024-09-13T14:55:37.194 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:37.218 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:37.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:37.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:37.320 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:37.344 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:37.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:37.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:37.446 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:37.472 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:37.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:37.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:37.575 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:37.616 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:37.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:37.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:37.718 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:37.742 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:37.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:37.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:37.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:37.891 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:37.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:37.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:37.971 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 78 16 94109 94093 75.3801 33.3125 0.0179014 0.0132596 2024-09-13T14:55:37.994 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:38.021 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:38.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:38.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:38.123 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:38.152 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:38.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:38.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:38.254 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:38.289 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:38.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:38.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:38.391 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:38.418 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:38.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:38.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:38.520 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:38.578 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:38.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:38.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:38.680 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:38.704 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:38.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:38.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:38.806 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:38.843 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:38.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:38.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:38.945 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:38.971 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 79 16 94672 94656 74.8713 35.1875 0.0246124 0.0133497 2024-09-13T14:55:38.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:38.974 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:38.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:39.025 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:38 smithi151 ceph-mon[104450]: pgmap v1927: 119 pgs: 2 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 33 MiB/s wr, 526 op/s; 44483/278796 objects degraded (15.955%); 13659/278796 objects misplaced (4.899%); 4.3 MiB/s, 68 objects/s recovering 2024-09-13T14:55:39.026 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:38 smithi151 ceph-mon[108966]: pgmap v1927: 119 pgs: 2 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 33 MiB/s wr, 526 op/s; 44483/278796 objects degraded (15.955%); 13659/278796 objects misplaced (4.899%); 4.3 MiB/s, 68 objects/s recovering 2024-09-13T14:55:39.065 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:39.075 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:39.083 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:39.100 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:39.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:39.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:39.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:38 smithi184 ceph-mon[92527]: pgmap v1927: 119 pgs: 2 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 33 MiB/s wr, 526 op/s; 44483/278796 objects degraded (15.955%); 13659/278796 objects misplaced (4.899%); 4.3 MiB/s, 68 objects/s recovering 2024-09-13T14:55:39.201 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:39.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:39.234 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:39.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:39.335 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:39.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:39.413 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:39.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:39.514 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:39.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:39.558 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:39.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:39.659 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:39.685 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:39.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:39.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:39.786 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:39.809 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:39.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:39.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:39.910 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:39.935 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:39.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:39.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:39.971 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:2024-09-13T14:55:39.971369+0000 min lat: 0.00116648 max lat: 2.46778 avg lat: 0.013446 2024-09-13T14:55:39.972 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:55:39.972 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 80 16 95184 95168 74.3355 32 0.034403 0.013446 2024-09-13T14:55:40.036 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:40.059 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:40.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:40.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:40.161 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:40.200 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:40.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:40.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:40.271 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:40 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a[104427]: 2024-09-13T14:55:40.155+0000 7fe9da14f640 -1 mon.a@0(leader).osd e1235 definitely_dead 0 2024-09-13T14:55:40.302 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:40.328 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:40.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:40.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:40.430 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:40.465 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:40.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:40.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:40.566 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:40.593 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:40.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:40.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:40.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:55:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:55:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:55:40.695 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:40.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:40.740 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:40.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:40.766 INFO:teuthology.orchestra.run.smithi151.stderr:marked in osd.5. 2024-09-13T14:55:40.841 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:40.867 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:40.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:40.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:40.969 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:40.972 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 81 16 95607 95591 73.7441 26.4375 0.0322707 0.013522 2024-09-13T14:55:40.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:40.991 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:40.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:41.069 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:55:41.092 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:41.093 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:40 smithi151 ceph-mon[104450]: pgmap v1928: 119 pgs: 2 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 31 MiB/s wr, 497 op/s; 44483/280563 objects degraded (15.855%); 13516/280563 objects misplaced (4.817%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:55:41.093 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:40 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 44483/280563 objects degraded (15.855%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:41.093 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:40 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1465843037' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-13T14:55:41.093 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:40 smithi151 ceph-mon[108966]: pgmap v1928: 119 pgs: 2 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 31 MiB/s wr, 497 op/s; 44483/280563 objects degraded (15.855%); 13516/280563 objects misplaced (4.817%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:55:41.093 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:40 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 44483/280563 objects degraded (15.855%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:41.094 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:40 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1465843037' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-13T14:55:41.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:40 smithi184 ceph-mon[92527]: pgmap v1928: 119 pgs: 2 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 31 MiB/s wr, 497 op/s; 44483/280563 objects degraded (15.855%); 13516/280563 objects misplaced (4.817%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:55:41.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:40 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 44483/280563 objects degraded (15.855%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:41.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:40 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1465843037' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-13T14:55:41.113 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:41.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:41.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:41.215 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:41.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:41.237 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:41.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:41.338 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:41.361 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:41.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:41.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:41.462 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:41.484 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:41.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:41.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:41.586 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:41.607 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:41.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:41.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:41.709 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:41.730 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:41.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:41.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:41.832 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:41.862 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:41.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:41.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:41.964 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:41.971 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 82 16 95626 95610 72.8594 1.1875 0.00378309 0.0136428 2024-09-13T14:55:41.989 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:41.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:41.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:42.091 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:42.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:41 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1465843037' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-13T14:55:42.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:41 smithi184 ceph-mon[92527]: osdmap e1236: 8 total, 8 up, 8 in 2024-09-13T14:55:42.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:41 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1465843037' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-13T14:55:42.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:41 smithi151 ceph-mon[104450]: osdmap e1236: 8 total, 8 up, 8 in 2024-09-13T14:55:42.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:41 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1465843037' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-13T14:55:42.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:41 smithi151 ceph-mon[108966]: osdmap e1236: 8 total, 8 up, 8 in 2024-09-13T14:55:42.130 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:42.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:42.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:42.232 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:42.270 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:42.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:42.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:42.372 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:42.406 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:42.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:42.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:42.507 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:42.540 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:42.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:42.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:42.642 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:42.671 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:42.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:42.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:42.773 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:42.807 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:42.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:42.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:42.910 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:42.943 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:42.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:42.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:42.972 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 83 16 96537 96521 72.6675 56.9375 0.00899678 0.0137558 2024-09-13T14:55:43.046 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:43.085 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:43.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:43.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:43.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:42 smithi184 ceph-mon[92527]: pgmap v1930: 119 pgs: 2 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 32 MiB/s wr, 510 op/s; 44483/284007 objects degraded (15.663%); 13419/284007 objects misplaced (4.725%); 4.5 MiB/s, 71 objects/s recovering 2024-09-13T14:55:43.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:42 smithi184 ceph-mon[92527]: osdmap e1237: 8 total, 8 up, 8 in 2024-09-13T14:55:43.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:42 smithi151 ceph-mon[104450]: pgmap v1930: 119 pgs: 2 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 32 MiB/s wr, 510 op/s; 44483/284007 objects degraded (15.663%); 13419/284007 objects misplaced (4.725%); 4.5 MiB/s, 71 objects/s recovering 2024-09-13T14:55:43.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:42 smithi151 ceph-mon[104450]: osdmap e1237: 8 total, 8 up, 8 in 2024-09-13T14:55:43.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:42 smithi151 ceph-mon[108966]: pgmap v1930: 119 pgs: 2 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 32 MiB/s wr, 510 op/s; 44483/284007 objects degraded (15.663%); 13419/284007 objects misplaced (4.725%); 4.5 MiB/s, 71 objects/s recovering 2024-09-13T14:55:43.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:42 smithi151 ceph-mon[108966]: osdmap e1237: 8 total, 8 up, 8 in 2024-09-13T14:55:43.187 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:43.224 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:43.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:43.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:43.326 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:43.357 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:43.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:43.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:43.459 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:43.493 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:43.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:43.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:43.595 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:43.610 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-09-13T14:55:43.622 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:43.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:43.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:43.724 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:43.775 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:43.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:43.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:43.877 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:43.913 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:43.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:43.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:43.972 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 84 16 97510 97494 72.5263 60.8125 0.010468 0.0137825 2024-09-13T14:55:44.014 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:44.055 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:44.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:44.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:44.157 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:44.189 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:44.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:44.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:44.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:44.340 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:44.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:44.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:44.442 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:44.480 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:44.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:44.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:44.582 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:44.615 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:44.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:44.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:44.717 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:44.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:44.771 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:44.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:44.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:55:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:55:44] "GET /metrics HTTP/1.1" 200 36682 "" "Prometheus/2.51.0" 2024-09-13T14:55:44.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:44 smithi151 ceph-mon[108966]: pgmap v1932: 119 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 25 MiB/s wr, 401 op/s; 36310/286131 objects degraded (12.690%); 8596/286131 objects misplaced (3.004%); 3.8 MiB/s, 62 objects/s recovering 2024-09-13T14:55:44.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:44 smithi151 ceph-mon[104450]: pgmap v1932: 119 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 25 MiB/s wr, 401 op/s; 36310/286131 objects degraded (12.690%); 8596/286131 objects misplaced (3.004%); 3.8 MiB/s, 62 objects/s recovering 2024-09-13T14:55:44.872 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:44.908 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:44.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:44.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:44.972 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 85 16 98441 98425 72.3573 58.1875 0.00892778 0.0138142 2024-09-13T14:55:45.011 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:45.043 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:45.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:45.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:45.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:44 smithi184 ceph-mon[92527]: pgmap v1932: 119 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 25 MiB/s wr, 401 op/s; 36310/286131 objects degraded (12.690%); 8596/286131 objects misplaced (3.004%); 3.8 MiB/s, 62 objects/s recovering 2024-09-13T14:55:45.145 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:45.180 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:45.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:45.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:45.282 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:45.319 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:45.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:45.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:45.422 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:45.454 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:45.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:45.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:45.462 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:45.516 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:55:45.516 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:55:45.555 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:45.585 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:45.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:45.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:45.687 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:45.722 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:45.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:45.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:45.824 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:45.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:45.859 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:45.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:45.960 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:45.975 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 86 16 99319 99303 72.1519 54.875 0.0224251 0.0138536 2024-09-13T14:55:45.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:45.995 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:45.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:46.097 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:46.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:45 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 36310/286131 objects degraded (12.690%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:46.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:45 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 36310/286131 objects degraded (12.690%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:46.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:45 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 36310/286131 objects degraded (12.690%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:46.126 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:46.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:46.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:46.228 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:46.265 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:46.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:46.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:46.367 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:46.406 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:46.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:46.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:46.508 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:46.539 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:46.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:46.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:46.641 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:46.671 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:46.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:46.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:46.773 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:46.812 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:46.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:46.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:46.825 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-13T14:55:46.915 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:46.948 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:46.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:46.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:46.975 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 87 16 100216 100200 71.9669 56.0625 0.0127671 0.0138898 2024-09-13T14:55:47.001 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:46 smithi151 ceph-mon[104450]: pgmap v1933: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 37 MiB/s wr, 584 op/s; 13829/292833 objects degraded (4.722%); 4180/292833 objects misplaced (1.427%); 6.4 MiB/s, 104 objects/s recovering 2024-09-13T14:55:47.001 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:46 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/236435789' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:55:47.001 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:46 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:55:47.001 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:47.002 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:46 smithi151 ceph-mon[108966]: pgmap v1933: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 37 MiB/s wr, 584 op/s; 13829/292833 objects degraded (4.722%); 4180/292833 objects misplaced (1.427%); 6.4 MiB/s, 104 objects/s recovering 2024-09-13T14:55:47.002 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:46 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/236435789' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:55:47.002 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:46 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:55:47.002 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:47.050 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:47.090 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:47.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:47.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:46 smithi184 ceph-mon[92527]: pgmap v1933: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 37 MiB/s wr, 584 op/s; 13829/292833 objects degraded (4.722%); 4180/292833 objects misplaced (1.427%); 6.4 MiB/s, 104 objects/s recovering 2024-09-13T14:55:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:46 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/236435789' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:55:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:46 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:55:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:47.192 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:47.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:47.235 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:47.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:47.336 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:47.372 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:47.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:47.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:47.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T14:55:47.474 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:47.506 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:47.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:47.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:47.608 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:47.642 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:47.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:47.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:47.744 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:47.775 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:47.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:47.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:47.798 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:47 smithi151 ceph-mon[108966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:55:47.798 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:47 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:55:47.798 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:47 smithi151 ceph-mon[108966]: osdmap e1238: 8 total, 8 up, 8 in 2024-09-13T14:55:47.877 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:47.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:47.925 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:47.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:47.978 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 88 16 101122 101106 71.7902 56.625 0.0210678 0.0139236 2024-09-13T14:55:48.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:48.066 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:48.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:48.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:48.077 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:47 smithi151 ceph-mon[104450]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:55:48.077 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:47 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:55:48.077 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:47 smithi151 ceph-mon[104450]: osdmap e1238: 8 total, 8 up, 8 in 2024-09-13T14:55:48.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:47 smithi184 ceph-mon[92527]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:55:48.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:47 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:55:48.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:47 smithi184 ceph-mon[92527]: osdmap e1238: 8 total, 8 up, 8 in 2024-09-13T14:55:48.168 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:48.202 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:48.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:48.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:48.303 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:48.346 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:48.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:48.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:48.448 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:48.482 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:48.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:48.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:48.584 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:48.611 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 6, 7, 1, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:55:48.611 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:55:48.611 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-13T14:55:48.612 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:55:48.612 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:55:48.612 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-09-13T14:55:48.632 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:48.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:48.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:48.734 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:48.795 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:48.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:48.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:48.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:48 smithi184 ceph-mon[92527]: pgmap v1935: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 40 MiB/s wr, 633 op/s; 13829/297153 objects degraded (4.654%); 4180/297153 objects misplaced (1.407%); 5.3 MiB/s, 85 objects/s recovering 2024-09-13T14:55:48.898 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:48.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:48.927 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:48.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:48.979 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 89 16 102003 101987 71.6017 55.0625 0.0164362 0.0139595 2024-09-13T14:55:49.028 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:49.030 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:49.062 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:55:49.071 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:49.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:49.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:49.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:48 smithi151 ceph-mon[104450]: pgmap v1935: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 40 MiB/s wr, 633 op/s; 13829/297153 objects degraded (4.654%); 4180/297153 objects misplaced (1.407%); 5.3 MiB/s, 85 objects/s recovering 2024-09-13T14:55:49.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:48 smithi151 ceph-mon[108966]: pgmap v1935: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 40 MiB/s wr, 633 op/s; 13829/297153 objects degraded (4.654%); 4180/297153 objects misplaced (1.407%); 5.3 MiB/s, 85 objects/s recovering 2024-09-13T14:55:49.173 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:49.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:49.225 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:49.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:49.326 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:49.356 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:49.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:49.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:49.458 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:49.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:49.494 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:49.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:49.595 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:49.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:49.637 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:49.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:49.738 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:49.766 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:49.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:49.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:49.868 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:49.911 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:49.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:49.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:49.979 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Total time run: 90.0149 2024-09-13T14:55:49.979 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Total writes made: 102853 2024-09-13T14:55:49.979 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Write size: 65536 2024-09-13T14:55:49.979 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Object size: 65536 2024-09-13T14:55:49.979 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Bandwidth (MB/sec): 71.4139 2024-09-13T14:55:49.979 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Stddev Bandwidth: 47.106 2024-09-13T14:55:49.979 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Max bandwidth (MB/sec): 235.125 2024-09-13T14:55:49.979 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Min bandwidth (MB/sec): 0 2024-09-13T14:55:49.979 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Average IOPS: 1142 2024-09-13T14:55:49.979 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Stddev IOPS: 753.704 2024-09-13T14:55:49.980 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Max IOPS: 3762 2024-09-13T14:55:49.980 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Min IOPS: 0 2024-09-13T14:55:49.980 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Average Latency(s): 0.0139974 2024-09-13T14:55:49.980 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Stddev Latency(s): 0.0452069 2024-09-13T14:55:49.980 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Max latency(s): 2.46778 2024-09-13T14:55:49.980 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Min latency(s): 0.00116648 2024-09-13T14:55:49.986 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T14:55:50.013 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:50.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:50.052 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:50.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:50.153 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:50.191 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:50.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:50.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:50.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:50.325 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:50.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:50.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:50.427 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:50.465 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:50.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:50.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:50.566 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:50.575 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:55:50.599 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:50.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:50.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:50.605 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:55:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:55:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:55:50.701 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:50.738 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:50.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:50.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:50.840 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:50.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:50.872 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:50.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:50.941 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:50 smithi184 ceph-mon[92527]: pgmap v1936: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 43 MiB/s wr, 687 op/s; 13829/300522 objects degraded (4.602%); 4180/300522 objects misplaced (1.391%); 4.6 MiB/s, 74 objects/s recovering 2024-09-13T14:55:50.942 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:50 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 13829/300522 objects degraded (4.602%), 12 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:50.973 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:51.005 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:50 smithi151 ceph-mon[104450]: pgmap v1936: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 43 MiB/s wr, 687 op/s; 13829/300522 objects degraded (4.602%); 4180/300522 objects misplaced (1.391%); 4.6 MiB/s, 74 objects/s recovering 2024-09-13T14:55:51.005 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:50 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 13829/300522 objects degraded (4.602%), 12 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:51.006 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:50 smithi151 ceph-mon[108966]: pgmap v1936: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 43 MiB/s wr, 687 op/s; 13829/300522 objects degraded (4.602%); 4180/300522 objects misplaced (1.391%); 4.6 MiB/s, 74 objects/s recovering 2024-09-13T14:55:51.006 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:50 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 13829/300522 objects degraded (4.602%), 12 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:51.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:51.009 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:51.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:51.110 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:51.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:51.149 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:51.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:51.164 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:55:51.164 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:55:51.164 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:55:51.164 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:55:51.164 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:55:51.164 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:55:51.164 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:55:51.164 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:55:51.165 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:55:51.165 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:55:51.173 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:55:51.173 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:55:51.204 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:55:51.205 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:55:51.237 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:55:51.237 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:55:51.250 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:51.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:51.282 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:51.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:51.383 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:51.415 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:51.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:51.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:51.518 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:51.549 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:51.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:51.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:51.651 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:51.690 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:51.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:51.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:51.792 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:51.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:51.829 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:51.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:51.931 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:51.952 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:51.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:51.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:52.054 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:52.084 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:52.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:52.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:52.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:51 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3173754472' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:52.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:51 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3173754472' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:52.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:51 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3173754472' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:52.136 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:55:52.136 INFO:teuthology.orchestra.run.smithi184.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T14:55:52.136 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:55:52.191 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:52.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:52.223 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:52.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:52.324 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:52.358 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:52.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:52.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:52.460 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:52.501 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:52.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:52.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:52.603 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:52.624 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:52.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:52.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:52.726 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:52.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:52.768 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:52.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:52.869 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:52.899 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:52.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:52.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:53.001 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:53.040 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:53.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:53.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:53.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:52 smithi184 ceph-mon[92527]: pgmap v1937: 119 pgs: 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 46 MiB/s wr, 753 op/s; 13829/305049 objects degraded (4.533%); 3679/305049 objects misplaced (1.206%); 6.8 MiB/s, 110 objects/s recovering 2024-09-13T14:55:53.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:52 smithi151 ceph-mon[108966]: pgmap v1937: 119 pgs: 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 46 MiB/s wr, 753 op/s; 13829/305049 objects degraded (4.533%); 3679/305049 objects misplaced (1.206%); 6.8 MiB/s, 110 objects/s recovering 2024-09-13T14:55:53.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:52 smithi151 ceph-mon[104450]: pgmap v1937: 119 pgs: 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 46 MiB/s wr, 753 op/s; 13829/305049 objects degraded (4.533%); 3679/305049 objects misplaced (1.206%); 6.8 MiB/s, 110 objects/s recovering 2024-09-13T14:55:53.142 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:53.164 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:53.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:53.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:53.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:53.288 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:53.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:53.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:53.390 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:53.411 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:53.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:53.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:53.513 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:53.534 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:53.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:53.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:53.635 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:53.657 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:53.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:53.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:53.759 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:53.780 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:53.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:53.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:53.882 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:53.903 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:53.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:53.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:54.005 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:54.041 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:54.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:54.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:54.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:53 smithi151 ceph-mon[108966]: osdmap e1239: 8 total, 8 up, 8 in 2024-09-13T14:55:54.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:53 smithi151 ceph-mon[104450]: osdmap e1239: 8 total, 8 up, 8 in 2024-09-13T14:55:54.143 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:54.171 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:54.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:54.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:54.273 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:54.313 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:54.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:54.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:54.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:53 smithi184 ceph-mon[92527]: osdmap e1239: 8 total, 8 up, 8 in 2024-09-13T14:55:54.415 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:54.447 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:54.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:54.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:54.549 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:54.585 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:54.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:54.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:54.686 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:54.719 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:54.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:54.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:54.820 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:54.857 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:55:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:55:54] "GET /metrics HTTP/1.1" 200 36668 "" "Prometheus/2.51.0" 2024-09-13T14:55:54.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:54.860 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:54.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:54.961 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:54.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:54.995 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:54.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:55.096 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:55.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:54 smithi151 ceph-mon[104450]: pgmap v1939: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 127 B/s rd, 39 MiB/s wr, 739 op/s; 13829/305274 objects degraded (4.530%); 3679/305274 objects misplaced (1.205%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:55:55.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:54 smithi151 ceph-mon[104450]: osdmap e1240: 8 total, 8 up, 8 in 2024-09-13T14:55:55.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:54 smithi151 ceph-mon[108966]: pgmap v1939: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 127 B/s rd, 39 MiB/s wr, 739 op/s; 13829/305274 objects degraded (4.530%); 3679/305274 objects misplaced (1.205%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:55:55.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:54 smithi151 ceph-mon[108966]: osdmap e1240: 8 total, 8 up, 8 in 2024-09-13T14:55:55.129 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:55.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:55.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:55.231 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:55.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:55.283 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:55.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:55.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:54 smithi184 ceph-mon[92527]: pgmap v1939: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 127 B/s rd, 39 MiB/s wr, 739 op/s; 13829/305274 objects degraded (4.530%); 3679/305274 objects misplaced (1.205%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:55:55.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:54 smithi184 ceph-mon[92527]: osdmap e1240: 8 total, 8 up, 8 in 2024-09-13T14:55:55.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:55.417 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:55.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:55.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:55.519 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:55.546 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:55.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:55.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:55.648 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:55.669 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:55.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:55.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:55.771 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:55.792 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:55.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:55.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:55.894 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:55.927 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:55.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:55.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:56.029 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:56.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:56.064 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:56.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:56.117 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:55 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 13829/305274 objects degraded (4.530%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:56.121 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:55 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 13829/305274 objects degraded (4.530%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:56.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:56.195 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:56.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:56.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:56.297 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:56.332 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:56.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:56.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:56.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:55 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 13829/305274 objects degraded (4.530%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:56.434 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:56.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:56.466 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:56.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:56.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:56.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:56.594 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:56.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:56.695 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:56.731 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:56.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:56.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:56.833 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:56.869 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:56.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:56.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:56.971 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:56.991 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:56.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:56.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:57.093 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:57.124 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:57.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:57.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:57.226 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:57.258 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:57.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:57.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:57.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:56 smithi184 ceph-mon[92527]: pgmap v1941: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 127 B/s rd, 30 MiB/s wr, 873 op/s; 13829/299166 objects degraded (4.623%); 3336/299166 objects misplaced (1.115%); 6.6 MiB/s, 105 objects/s recovering 2024-09-13T14:55:57.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:56 smithi151 ceph-mon[108966]: pgmap v1941: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 127 B/s rd, 30 MiB/s wr, 873 op/s; 13829/299166 objects degraded (4.623%); 3336/299166 objects misplaced (1.115%); 6.6 MiB/s, 105 objects/s recovering 2024-09-13T14:55:57.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:56 smithi151 ceph-mon[104450]: pgmap v1941: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 127 B/s rd, 30 MiB/s wr, 873 op/s; 13829/299166 objects degraded (4.623%); 3336/299166 objects misplaced (1.115%); 6.6 MiB/s, 105 objects/s recovering 2024-09-13T14:55:57.360 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:57.398 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:57.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:57.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:57.500 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:57.536 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:57.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:57.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:57.638 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:57.659 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:57.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:57.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:57.761 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:57.801 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:57.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:57.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:57.903 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:57.938 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:57.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:57.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:58.031 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 6, 7, 1, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:55:58.031 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:55:58.031 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-13T14:55:58.031 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:55:58.031 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:55:58.032 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-09-13T14:55:58.040 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:58.064 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:58.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:58.161 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:55:58.165 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:58.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.196 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:58.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:58.297 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:58.333 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:58.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:58.435 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:58.456 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:58.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:58.558 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:58.580 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:58.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:58.682 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:58.715 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:58.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:58.817 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:58.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.863 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:58.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:58.964 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:58.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.999 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:58.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:59.100 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:59.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:55:58 smithi151 ceph-mon[104450]: pgmap v1942: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 127 B/s rd, 21 MiB/s wr, 877 op/s; 13829/295689 objects degraded (4.677%); 3336/295689 objects misplaced (1.128%); 6.6 MiB/s, 105 objects/s recovering 2024-09-13T14:55:59.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:55:58 smithi151 ceph-mon[108966]: pgmap v1942: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 127 B/s rd, 21 MiB/s wr, 877 op/s; 13829/295689 objects degraded (4.677%); 3336/295689 objects misplaced (1.128%); 6.6 MiB/s, 105 objects/s recovering 2024-09-13T14:55:59.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:59.131 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:59.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:59.233 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:59.253 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:59.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:59.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:59.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:55:58 smithi184 ceph-mon[92527]: pgmap v1942: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 127 B/s rd, 21 MiB/s wr, 877 op/s; 13829/295689 objects degraded (4.677%); 3336/295689 objects misplaced (1.128%); 6.6 MiB/s, 105 objects/s recovering 2024-09-13T14:55:59.355 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:59.386 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:59.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:59.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:59.488 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:59.526 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:59.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:59.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:59.630 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:59.669 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:55:59.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:59.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:59.771 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:59.807 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:59.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:59.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:59.909 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:59.950 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:55:59.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:59.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:00.052 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:00.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:00.089 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:00.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:00.190 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:00.220 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:00.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:00.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:00.322 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:00.359 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:00.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:00.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:00.461 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:00.494 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:00.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:00.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:00.596 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:00.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:56:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:56:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:56:00.627 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:00.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:00.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:00.730 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:00.765 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:00.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:00.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:00.867 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:00.912 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:00.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:00.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:01.014 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:01.048 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:01.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:01.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:01.150 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:01.165 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:56:01.165 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-13T14:56:01.165 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-13T14:56:01.165 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-13T14:56:01.165 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:56:01.166 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:00 smithi151 ceph-mon[104450]: pgmap v1943: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.0 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 127 B/s rd, 7.6 MiB/s wr, 759 op/s; 13829/292623 objects degraded (4.726%); 3336/292623 objects misplaced (1.140%); 2.7 MiB/s, 42 objects/s recovering 2024-09-13T14:56:01.166 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:00 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 13829/292623 objects degraded (4.726%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:56:01.166 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:00 smithi151 ceph-mon[108966]: pgmap v1943: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.0 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 127 B/s rd, 7.6 MiB/s wr, 759 op/s; 13829/292623 objects degraded (4.726%); 3336/292623 objects misplaced (1.140%); 2.7 MiB/s, 42 objects/s recovering 2024-09-13T14:56:01.167 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:00 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 13829/292623 objects degraded (4.726%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:56:01.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:01.183 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:01.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:01.284 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:01.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:01.326 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:01.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:01.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:00 smithi184 ceph-mon[92527]: pgmap v1943: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.0 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 127 B/s rd, 7.6 MiB/s wr, 759 op/s; 13829/292623 objects degraded (4.726%); 3336/292623 objects misplaced (1.140%); 2.7 MiB/s, 42 objects/s recovering 2024-09-13T14:56:01.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:00 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 13829/292623 objects degraded (4.726%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:56:01.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:01.463 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:01.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:01.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:01.564 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:01.604 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:01.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:01.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:01.706 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:01.737 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:01.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:01.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:01.839 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:01.857 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:01.866 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:01.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:01.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:01.968 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:02.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:02.002 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:02.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:02.103 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:02.142 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:02.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:02.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:02.244 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:02.278 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:02.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:02.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:02.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:01 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:02.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:01 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:02.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:01 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:02.380 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:02.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:02.429 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:02.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:02.482 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T14:56:02.516 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:02.529 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:02.531 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:02.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:02.596 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:02.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:02.697 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:02.727 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:02.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:02.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:02.829 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:02.871 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:02.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:02.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:02.973 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:03.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:03.011 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:03.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:03.036 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:02 smithi151 ceph-mon[108966]: pgmap v1944: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.7 MiB/s wr, 751 op/s; 13829/286770 objects degraded (4.822%); 3177/286770 objects misplaced (1.108%); 3.5 MiB/s, 56 objects/s recovering 2024-09-13T14:56:03.037 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:02 smithi151 ceph-mon[104450]: pgmap v1944: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.7 MiB/s wr, 751 op/s; 13829/286770 objects degraded (4.822%); 3177/286770 objects misplaced (1.108%); 3.5 MiB/s, 56 objects/s recovering 2024-09-13T14:56:03.112 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:03.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:03.165 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:03.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:03.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:03.305 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:03.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:03.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:03.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:02 smithi184 ceph-mon[92527]: pgmap v1944: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.7 MiB/s wr, 751 op/s; 13829/286770 objects degraded (4.822%); 3177/286770 objects misplaced (1.108%); 3.5 MiB/s, 56 objects/s recovering 2024-09-13T14:56:03.407 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:03.444 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:03.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:03.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:03.546 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:03.582 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:03.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:03.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:03.684 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:03.718 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:03.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:03.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:03.820 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:03.857 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:03.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:03.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:03.959 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:04.004 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:04.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:04.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:04.107 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:04.136 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:04.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:04.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:04.238 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:04.267 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:04.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:04.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:04.369 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:04.405 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:04.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:04.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:04.507 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:04.540 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:04.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:04.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:04.642 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:04.684 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:04.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:04.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:04.725 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:56:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:56:04] "GET /metrics HTTP/1.1" 200 36680 "" "Prometheus/2.51.0" 2024-09-13T14:56:04.786 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:04.820 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:04.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:04.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:04.922 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:04.964 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:04.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:04.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:05.066 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:05.101 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:05.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:05.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:05.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:04 smithi151 ceph-mon[104450]: pgmap v1945: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.5 MiB/s wr, 813 op/s; 13829/282297 objects degraded (4.899%); 3177/282297 objects misplaced (1.125%); 3.1 MiB/s, 50 objects/s recovering 2024-09-13T14:56:05.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:04 smithi151 ceph-mon[108966]: pgmap v1945: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.5 MiB/s wr, 813 op/s; 13829/282297 objects degraded (4.899%); 3177/282297 objects misplaced (1.125%); 3.1 MiB/s, 50 objects/s recovering 2024-09-13T14:56:05.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:05.204 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:05.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:05.250 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:05.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:05.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:05.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:04 smithi184 ceph-mon[92527]: pgmap v1945: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.5 MiB/s wr, 813 op/s; 13829/282297 objects degraded (4.899%); 3177/282297 objects misplaced (1.125%); 3.1 MiB/s, 50 objects/s recovering 2024-09-13T14:56:05.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:05.401 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:05.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:05.502 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:05.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:05.543 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:05.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:05.644 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:05.669 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:05.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:05.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:05.771 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:05.810 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:05.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:05.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:05.912 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:05.953 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:05.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:05.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:06.055 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:06.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:06.100 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:06.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:06.175 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:05 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 13829/282297 objects degraded (4.899%), 11 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:06.175 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:05 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 13829/282297 objects degraded (4.899%), 11 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:06.201 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:06.235 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:06.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:06.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:06.337 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:06.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:05 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 13829/282297 objects degraded (4.899%), 11 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:06.372 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:06.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:06.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:06.473 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:06.508 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:06.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:06.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:06.610 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:06.636 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:06.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:06.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:06.738 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:06.771 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:06.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:06.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:06.873 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:06.908 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:06.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:06.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:07.010 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:07.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.042 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:07.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:07.143 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:07.181 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:07.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:07.283 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:07.312 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:07.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:07.324 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:06 smithi151 ceph-mon[104450]: pgmap v1946: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.3 MiB/s wr, 973 op/s; 13829/272115 objects degraded (5.082%); 3041/272115 objects misplaced (1.118%); 3.4 MiB/s, 53 objects/s recovering 2024-09-13T14:56:07.325 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:06 smithi151 ceph-mon[108966]: pgmap v1946: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.3 MiB/s wr, 973 op/s; 13829/272115 objects degraded (5.082%); 3041/272115 objects misplaced (1.118%); 3.4 MiB/s, 53 objects/s recovering 2024-09-13T14:56:07.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:06 smithi184 ceph-mon[92527]: pgmap v1946: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.3 MiB/s wr, 973 op/s; 13829/272115 objects degraded (5.082%); 3041/272115 objects misplaced (1.118%); 3.4 MiB/s, 53 objects/s recovering 2024-09-13T14:56:07.414 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:07.434 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:07.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:07.536 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:07.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.575 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:07.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:07.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:07.704 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:07.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:07.806 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:07.837 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:07.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:07.939 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:07.967 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:07.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:08.069 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:08.103 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:08.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:08.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:08.205 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:08.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:08.231 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:08.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:08.332 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:08.368 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:08.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:08.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:08.470 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:08.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:08.506 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:08.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:08.607 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:08.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:08.636 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:08.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:08.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:08.777 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:08.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:08.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:08.917 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:08.954 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:08.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:08.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:09.056 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:09.095 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:09.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:09.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:09.197 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:09.222 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:09.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:09.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:09.323 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:09.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:08 smithi184 ceph-mon[92527]: pgmap v1947: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 884 op/s; 13829/267330 objects degraded (5.173%); 3041/267330 objects misplaced (1.138%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:56:09.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:08 smithi151 ceph-mon[108966]: pgmap v1947: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 884 op/s; 13829/267330 objects degraded (5.173%); 3041/267330 objects misplaced (1.138%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:56:09.359 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:09.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:08 smithi151 ceph-mon[104450]: pgmap v1947: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 884 op/s; 13829/267330 objects degraded (5.173%); 3041/267330 objects misplaced (1.138%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:56:09.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:09.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:09.462 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:09.496 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:09.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:09.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:09.598 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:09.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:09.651 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:09.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:09.752 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:09.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:09.806 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:09.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:09.907 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:09.922 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:56:09.922 INFO:teuthology.orchestra.run.smithi151.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T14:56:09.922 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:56:09.943 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:09.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:09.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:10.046 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:10.083 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:10.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:10.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:10.185 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:10.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:10.219 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:10.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:10.320 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:10.358 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:10.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:10.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:10.461 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:10.496 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:10.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:10.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:10.598 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:10.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:56:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:56:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:56:10.627 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:10.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:10.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:10.729 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:10.757 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:10.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:10.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:10.859 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:10.896 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:10.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:10.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:10.998 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:11.040 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:11.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:11.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:11.082 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:10 smithi151 ceph-mon[108966]: pgmap v1948: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 892 op/s; 13829/263532 objects degraded (5.248%); 3041/263532 objects misplaced (1.154%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:56:11.083 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:10 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 13829/263532 objects degraded (5.248%), 11 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:11.083 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:10 smithi151 ceph-mon[104450]: pgmap v1948: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 892 op/s; 13829/263532 objects degraded (5.248%); 3041/263532 objects misplaced (1.154%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:56:11.083 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:10 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 13829/263532 objects degraded (5.248%), 11 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:11.142 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:11.170 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:11.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:11.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:11.272 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:11.313 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:11.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:11.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:11.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:10 smithi184 ceph-mon[92527]: pgmap v1948: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 892 op/s; 13829/263532 objects degraded (5.248%); 3041/263532 objects misplaced (1.154%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:56:11.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:10 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 13829/263532 objects degraded (5.248%), 11 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:11.415 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:11.461 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:11.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:11.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:11.563 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:11.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:11.599 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:11.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:11.700 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:11.744 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:11.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:11.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:11.846 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:11.879 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:11.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:11.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:11.943 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-13T14:56:11.943 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:11 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1460666018' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:11.943 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:11 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2345077820' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:11.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:11 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1460666018' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:11.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:11 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2345077820' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:11.981 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:12.008 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:12.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:12.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:12.109 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:12.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:12.138 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:12.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:12.239 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:12.270 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:12.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:12.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:12.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:11 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1460666018' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:12.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:11 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2345077820' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:12.372 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:12.412 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:12.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:12.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:12.514 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:12.545 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:12.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:12.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:12.647 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:12.680 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:12.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:12.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:12.782 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:12.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:12.816 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:12.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:12.917 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:12.953 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:12.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:12.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:13.055 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:13.066 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:56:13.066 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:56:13.066 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:56:13.066 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:56:13.066 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:56:13.066 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:56:13.066 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:56:13.066 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:56:13.067 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:56:13.067 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:56:13.067 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:56:13.067 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:56:13.067 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:56:13.067 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:56:13.067 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:56:13.067 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:56:13.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:13.089 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:13.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:13.190 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:13.227 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:13.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:13.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:13.328 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:13.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:12 smithi184 ceph-mon[92527]: pgmap v1949: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 984 op/s; 13829/257181 objects degraded (5.377%); 2898/257181 objects misplaced (1.127%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:56:13.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:12 smithi184 ceph-mon[92527]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:56:13.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:12 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2345077820' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:56:13.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:12 smithi184 ceph-mon[92527]: osdmap e1241: 8 total, 8 up, 8 in 2024-09-13T14:56:13.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:12 smithi151 ceph-mon[104450]: pgmap v1949: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 984 op/s; 13829/257181 objects degraded (5.377%); 2898/257181 objects misplaced (1.127%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:56:13.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:12 smithi151 ceph-mon[104450]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:56:13.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:12 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2345077820' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:56:13.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:12 smithi151 ceph-mon[104450]: osdmap e1241: 8 total, 8 up, 8 in 2024-09-13T14:56:13.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:12 smithi151 ceph-mon[108966]: pgmap v1949: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 984 op/s; 13829/257181 objects degraded (5.377%); 2898/257181 objects misplaced (1.127%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:56:13.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:12 smithi151 ceph-mon[108966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:56:13.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:12 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2345077820' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:56:13.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:12 smithi151 ceph-mon[108966]: osdmap e1241: 8 total, 8 up, 8 in 2024-09-13T14:56:13.360 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:13.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:13.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:13.462 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:13.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:13.488 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:13.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:13.589 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:13.625 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:13.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:13.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:13.727 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:13.761 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:13.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:13.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:13.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:13.900 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:13.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:13.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:14.002 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:14.042 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:14.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:14.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:14.144 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:14.182 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:14.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:14.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:14.284 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:14.320 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:14.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:14.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:14.424 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:14.459 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:14.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:14.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:14.560 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:14.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:14.594 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:14.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:14.695 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:14.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:14.729 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:14.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:14.830 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:14.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:56:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:56:14] "GET /metrics HTTP/1.1" 200 36680 "" "Prometheus/2.51.0" 2024-09-13T14:56:14.869 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:14.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:14.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:14.971 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:14.991 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:56:15.007 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:15.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:15.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:15.109 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:15.134 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:15.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:15.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:15.236 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:15.272 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:15.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:15.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:15.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:14 smithi184 ceph-mon[92527]: pgmap v1951: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 984 op/s; 13829/252762 objects degraded (5.471%); 2898/252762 objects misplaced (1.147%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T14:56:15.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:14 smithi151 ceph-mon[104450]: pgmap v1951: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 984 op/s; 13829/252762 objects degraded (5.471%); 2898/252762 objects misplaced (1.147%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T14:56:15.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:14 smithi151 ceph-mon[108966]: pgmap v1951: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 984 op/s; 13829/252762 objects degraded (5.471%); 2898/252762 objects misplaced (1.147%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T14:56:15.375 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:15.414 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:15.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:15.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:15.516 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:15.548 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:15.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:15.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:15.650 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:15.689 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:15.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:15.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:15.791 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:15.820 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:15.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:15.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:15.922 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:15.945 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:15.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:15.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:16.047 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:16.084 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:16.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:16.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:16.186 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:16.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:16.232 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:16.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:16.271 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:15 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 13829/252762 objects degraded (5.471%), 11 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:16.271 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:15 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 13829/252762 objects degraded (5.471%), 11 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:16.334 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:16.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:15 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 13829/252762 objects degraded (5.471%), 11 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:16.365 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:16.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:16.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:16.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:16.498 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:16.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:16.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:16.599 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:16.629 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:16.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:16.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:16.731 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:16.788 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:16.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:16.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:16.890 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:16.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:16.922 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:16.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:16.988 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:16 smithi151 ceph-mon[104450]: pgmap v1952: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 865 op/s; 13829/246132 objects degraded (5.619%); 2898/246132 objects misplaced (1.177%); 915 KiB/s, 14 objects/s recovering 2024-09-13T14:56:16.988 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:56:16.988 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:16.988 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:16 smithi151 ceph-mon[108966]: pgmap v1952: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 865 op/s; 13829/246132 objects degraded (5.619%); 2898/246132 objects misplaced (1.177%); 915 KiB/s, 14 objects/s recovering 2024-09-13T14:56:16.988 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:56:16.988 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:17.024 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:17.058 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:17.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:17.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:17.144 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 6, 7, 1, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:56:17.144 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T14:56:17.161 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:17.192 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:17.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:17.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:17.294 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:17.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:16 smithi184 ceph-mon[92527]: pgmap v1952: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 865 op/s; 13829/246132 objects degraded (5.619%); 2898/246132 objects misplaced (1.177%); 915 KiB/s, 14 objects/s recovering 2024-09-13T14:56:17.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:56:17.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:17.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:17.355 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:17.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:17.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:17.483 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:56:17.483 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:56:17.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:17.521 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:17.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:17.621 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:17.661 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:17.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:17.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:17.763 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:17.796 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:17.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:17.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:17.898 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:17.931 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:17.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:17.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:18.032 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:18.068 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:18.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:18.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:18.144 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:56:18.171 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:18.207 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:18.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:18.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:18.309 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:18.365 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:18.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:18.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:18.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T14:56:18.448 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:18.463 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:18.467 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:18.498 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:18.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:18.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:18.600 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:18.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:18.674 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:18.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:18.775 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:18.819 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:18.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:18.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:18.921 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:18.959 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:18.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:18.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:19.061 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:19.093 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:19.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:19.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:19.195 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:19.229 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:19.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:19.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:19.285 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:18 smithi151 ceph-mon[104450]: pgmap v1953: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 983 op/s; 13829/237819 objects degraded (5.815%); 2755/237819 objects misplaced (1.158%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:56:19.285 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:18 smithi151 ceph-mon[108966]: pgmap v1953: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 983 op/s; 13829/237819 objects degraded (5.815%); 2755/237819 objects misplaced (1.158%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:56:19.331 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:19.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:18 smithi184 ceph-mon[92527]: pgmap v1953: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 983 op/s; 13829/237819 objects degraded (5.815%); 2755/237819 objects misplaced (1.158%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:56:19.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:19.370 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:19.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:19.471 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:19.503 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:19.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:19.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:19.605 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:19.646 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:19.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:19.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:19.748 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:19.782 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:19.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:19.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:19.884 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:19.921 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:19.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:19.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:20.025 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:20.067 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:20.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:20.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:20.168 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:20.210 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:20.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:20.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:20.311 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:20.340 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:20.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:20.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:20.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:20 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 13829/237819 objects degraded (5.815%), 11 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:20.354 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:56:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:56:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:56:20.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:20 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 13829/237819 objects degraded (5.815%), 11 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:20.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:20 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 13829/237819 objects degraded (5.815%), 11 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:20.442 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:20.473 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:20.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:20.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:20.575 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:20.610 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:20.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:20.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:20.712 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:20.740 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:20.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:20.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:20.842 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:20.874 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:20.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:20.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:20.976 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:21.010 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:21.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:21.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:21.112 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:21.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:21.148 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:21.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:21.249 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:21.270 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:21.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:21.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:21.296 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:21 smithi184 ceph-mon[92527]: pgmap v1954: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 974 op/s; 13829/234273 objects degraded (5.903%); 2755/234273 objects misplaced (1.176%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:56:21.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:21 smithi151 ceph-mon[104450]: pgmap v1954: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 974 op/s; 13829/234273 objects degraded (5.903%); 2755/234273 objects misplaced (1.176%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:56:21.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:21 smithi151 ceph-mon[108966]: pgmap v1954: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 974 op/s; 13829/234273 objects degraded (5.903%); 2755/234273 objects misplaced (1.176%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:56:21.372 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:21.405 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:21.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:21.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:21.507 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:21.540 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:21.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:21.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:21.642 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:21.678 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:21.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:21.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:21.780 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:21.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:21.818 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:21.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:21.920 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:21.950 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:21.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:21.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:22.052 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:22.084 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:22.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:22.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:22.186 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:22.232 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:22.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:22.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:22.334 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:22.364 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:22.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:22.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:22.466 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:22.501 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:22.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:22.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:22.602 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:22.627 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:22.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:22.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:22.729 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:22.751 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:22.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:22.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:22.852 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:22.873 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:22.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:22.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:22.975 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:23.005 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:23.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:23.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:23.107 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:23.127 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:23.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:23.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:23.229 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:23.262 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:23.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:23.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:23.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:23 smithi184 ceph-mon[92527]: pgmap v1955: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 976 op/s; 13956/227883 objects degraded (6.124%); 2458/227883 objects misplaced (1.079%); 2.0 MiB/s, 31 objects/s recovering 2024-09-13T14:56:23.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:23 smithi151 ceph-mon[104450]: pgmap v1955: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 976 op/s; 13956/227883 objects degraded (6.124%); 2458/227883 objects misplaced (1.079%); 2.0 MiB/s, 31 objects/s recovering 2024-09-13T14:56:23.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:23 smithi151 ceph-mon[108966]: pgmap v1955: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 976 op/s; 13956/227883 objects degraded (6.124%); 2458/227883 objects misplaced (1.079%); 2.0 MiB/s, 31 objects/s recovering 2024-09-13T14:56:23.364 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:23.393 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:23.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:23.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:23.495 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:23.526 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:23.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:23.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:23.628 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:23.661 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:23.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:23.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:23.762 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:23.783 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:23.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:23.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:23.885 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:23.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:23.923 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:23.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:24.023 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:24.053 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:24.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:24.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:24.155 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:24.195 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:24.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:24.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:24.297 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:24.338 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:24.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:24.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:24.440 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:24.473 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:24.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:24.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:24.575 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:24.606 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:24.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:24.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:24.708 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:24.737 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:24.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:24.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:24.839 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:24.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:56:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:56:24] "GET /metrics HTTP/1.1" 200 36675 "" "Prometheus/2.51.0" 2024-09-13T14:56:24.873 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:24.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:24.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:24.974 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:25.006 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:25.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:25.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:25.108 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:25.140 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:25.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:25.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:25.242 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:25.277 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:25.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:25.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:25.302 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:25 smithi151 ceph-mon[104450]: pgmap v1956: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 955 op/s; 13956/223470 objects degraded (6.245%); 2458/223470 objects misplaced (1.100%); 1.7 MiB/s, 26 objects/s recovering 2024-09-13T14:56:25.302 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:25 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 13956/223470 objects degraded (6.245%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:25.302 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:25 smithi151 ceph-mon[108966]: pgmap v1956: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 955 op/s; 13956/223470 objects degraded (6.245%); 2458/223470 objects misplaced (1.100%); 1.7 MiB/s, 26 objects/s recovering 2024-09-13T14:56:25.302 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:25 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 13956/223470 objects degraded (6.245%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:25.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:25 smithi184 ceph-mon[92527]: pgmap v1956: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 955 op/s; 13956/223470 objects degraded (6.245%); 2458/223470 objects misplaced (1.100%); 1.7 MiB/s, 26 objects/s recovering 2024-09-13T14:56:25.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:25 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 13956/223470 objects degraded (6.245%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:25.379 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:25.400 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:25.405 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:25.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:25.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:25.506 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:25.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:25.551 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:25.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:25.652 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:25.690 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:25.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:25.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:25.792 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:25.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:25.831 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:25.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:25.932 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:25.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:25.972 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:25.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:26.073 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:26.126 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:26.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:26.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:26.228 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:26.269 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:26.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:26.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:26.372 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:26.408 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:26.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:26.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:26.510 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:26.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:26.549 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:26.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:26.650 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:26.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:26.699 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:26.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:26.801 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:26.828 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:26.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:26.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:26.929 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:26.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:26.968 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:26.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:27.069 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:27.105 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:27.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:27.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:27.141 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:27.208 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:27.235 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:27.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:27.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:27.251 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:27 smithi151 ceph-mon[104450]: pgmap v1957: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 979 op/s; 13956/217482 objects degraded (6.417%); 2458/217482 objects misplaced (1.130%); 1.6 MiB/s, 26 objects/s recovering 2024-09-13T14:56:27.251 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:27 smithi151 ceph-mon[108966]: pgmap v1957: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 979 op/s; 13956/217482 objects degraded (6.417%); 2458/217482 objects misplaced (1.130%); 1.6 MiB/s, 26 objects/s recovering 2024-09-13T14:56:27.337 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:27.357 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:27 smithi184 ceph-mon[92527]: pgmap v1957: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 979 op/s; 13956/217482 objects degraded (6.417%); 2458/217482 objects misplaced (1.130%); 1.6 MiB/s, 26 objects/s recovering 2024-09-13T14:56:27.377 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:27.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:27.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:27.479 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:27.514 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:27.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:27.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:27.616 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:27.651 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:27.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:27.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:27.753 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:27.786 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:27.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:27.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:27.888 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:27.921 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:27.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:27.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:28.023 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:28.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:28.057 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:28.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:28.096 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T14:56:28.158 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:28.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:28.226 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:28.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:28.327 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:28.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:28 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/560880846' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:56:28.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:28 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:56:28.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:28 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/560880846' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:56:28.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:28 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:56:28.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:28 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/560880846' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:56:28.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:28 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:56:28.369 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:28.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:28.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:28.471 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:28.508 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:28.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:28.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:28.610 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:28.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:28.645 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:28.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:28.746 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:28.780 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:28.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:28.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:28.882 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:28.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:28.931 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:28.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:29.033 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:29.070 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:29.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:29.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:29.172 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:29.214 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:29.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:29.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:29.315 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:29 smithi151 ceph-mon[104450]: pgmap v1958: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 986 op/s; 14039/210612 objects degraded (6.666%); 2172/210612 objects misplaced (1.031%); 2.7 MiB/s, 42 objects/s recovering 2024-09-13T14:56:29.315 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:29 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:56:29.315 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:29 smithi151 ceph-mon[104450]: osdmap e1242: 8 total, 8 up, 8 in 2024-09-13T14:56:29.315 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:29 smithi151 ceph-mon[108966]: pgmap v1958: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 986 op/s; 14039/210612 objects degraded (6.666%); 2172/210612 objects misplaced (1.031%); 2.7 MiB/s, 42 objects/s recovering 2024-09-13T14:56:29.315 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:29 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:56:29.315 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:29 smithi151 ceph-mon[108966]: osdmap e1242: 8 total, 8 up, 8 in 2024-09-13T14:56:29.317 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:29.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:29 smithi184 ceph-mon[92527]: pgmap v1958: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 986 op/s; 14039/210612 objects degraded (6.666%); 2172/210612 objects misplaced (1.031%); 2.7 MiB/s, 42 objects/s recovering 2024-09-13T14:56:29.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:29 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:56:29.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:29 smithi184 ceph-mon[92527]: osdmap e1242: 8 total, 8 up, 8 in 2024-09-13T14:56:29.365 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:29.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:29.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:29.466 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:29.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:29.501 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:29.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:29.602 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:29.641 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:29.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:29.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:29.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:29.772 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:29.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:29.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:29.874 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:29.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:29.906 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:29.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:29.913 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T14:56:30.006 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:30.064 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:30.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:30.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:30.079 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-13T14:56:30.126 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:30 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3319637904' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:56:30.126 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:30 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:56:30.126 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:30 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 14039/210612 objects degraded (6.666%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:30.127 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:30 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3319637904' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:56:30.127 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:30 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:56:30.127 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:30 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 14039/210612 objects degraded (6.666%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:30.166 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:30.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:30.202 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:30.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:30.205 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:56:30.205 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:56:30.240 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:56:30.240 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:56:30.269 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:56:30.270 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:56:30.302 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:56:30.302 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:56:30.302 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:30.332 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:56:30.333 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:56:30.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:30.350 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:30.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:30.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:30 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3319637904' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:56:30.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:30 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:56:30.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:30 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 14039/210612 objects degraded (6.666%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:30.354 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:56:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:56:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:56:30.371 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:56:30.371 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:56:30.409 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:56:30.409 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:56:30.443 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:56:30.443 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:56:30.451 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:30.483 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:30.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:30.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:30.585 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:30.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:30.624 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:30.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:30.725 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:30.759 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:30.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:30.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:30.860 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:30.902 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:30.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:30.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:31.004 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:31.040 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:31.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:31.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:31.142 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:31.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:31.182 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:31.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:31.283 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:31.330 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:31 smithi151 ceph-mon[104450]: pgmap v1960: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.2 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 951 op/s; 14039/205722 objects degraded (6.824%); 2172/205722 objects misplaced (1.056%); 2.3 MiB/s, 37 objects/s recovering 2024-09-13T14:56:31.330 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:31 smithi151 ceph-mon[104450]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:31.331 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:31 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:56:31.331 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:31 smithi151 ceph-mon[104450]: osdmap e1243: 8 total, 8 up, 8 in 2024-09-13T14:56:31.331 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:31 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/140532704' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:31.331 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:56:31.331 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:31 smithi151 ceph-mon[108966]: pgmap v1960: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.2 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 951 op/s; 14039/205722 objects degraded (6.824%); 2172/205722 objects misplaced (1.056%); 2.3 MiB/s, 37 objects/s recovering 2024-09-13T14:56:31.331 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:31 smithi151 ceph-mon[108966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:31.331 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:31 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:56:31.331 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:31 smithi151 ceph-mon[108966]: osdmap e1243: 8 total, 8 up, 8 in 2024-09-13T14:56:31.331 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:31 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/140532704' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:31.331 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:56:31.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:31.348 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:31.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:31.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:31 smithi184 ceph-mon[92527]: pgmap v1960: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.2 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 951 op/s; 14039/205722 objects degraded (6.824%); 2172/205722 objects misplaced (1.056%); 2.3 MiB/s, 37 objects/s recovering 2024-09-13T14:56:31.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:31 smithi184 ceph-mon[92527]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:31.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:31 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:56:31.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:31 smithi184 ceph-mon[92527]: osdmap e1243: 8 total, 8 up, 8 in 2024-09-13T14:56:31.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:31 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/140532704' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:31.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:56:31.449 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:31.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:31.491 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:31.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:31.592 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:31.636 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:31.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:31.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:31.738 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:31.781 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:31.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:31.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:31.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:31.921 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:31.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:31.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:32.023 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:32.051 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:32.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:32.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:32.153 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:32.194 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:32.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:32.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:32.296 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:32.332 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:32.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:32.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:32.434 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:32.472 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:32.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:32.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:32.575 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:32.627 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:32.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:32.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:32.653 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:56:32.654 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:32.654 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:56:32.654 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:56:32.654 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:32 smithi151 ceph-mon[104450]: pgmap v1962: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 14172/199125 objects degraded (7.117%); 1866/199125 objects misplaced (0.937%); 2.9 MiB/s, 46 objects/s recovering 2024-09-13T14:56:32.654 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:56:32.654 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:32.654 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:56:32.654 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:56:32.654 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:32 smithi151 ceph-mon[108966]: pgmap v1962: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 14172/199125 objects degraded (7.117%); 1866/199125 objects misplaced (0.937%); 2.9 MiB/s, 46 objects/s recovering 2024-09-13T14:56:32.729 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:32.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:32.769 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:32.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:32.852 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:56:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:56:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:56:32.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:56:32.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:32 smithi184 ceph-mon[92527]: pgmap v1962: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 14172/199125 objects degraded (7.117%); 1866/199125 objects misplaced (0.937%); 2.9 MiB/s, 46 objects/s recovering 2024-09-13T14:56:32.870 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:32.902 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:32.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:32.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:33.004 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:33.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:33.034 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:33.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:33.135 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:33.179 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:56:33.179 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-13T14:56:33.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.180 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.46034.0:143152 9.4 9:2af8d507:::benchmark_data_smithi151_261954_object40296:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1243)", 2024-09-13T14:56:33.180 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:56:33.154767+0000", 2024-09-13T14:56:33.180 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.017730465000000001, 2024-09-13T14:56:33.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017762191, 2024-09-13T14:56:33.180 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:56:33.180 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:56:33.180 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:33.180 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:56:33.180 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.46034", 2024-09-13T14:56:33.180 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:56:33.180 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 143152 2024-09-13T14:56:33.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:56:33.180 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:56:33.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:56:33.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:56:33.154767+0000", 2024-09-13T14:56:33.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:56:33.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:56:33.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:56:33.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:56:33.154767+0000", 2024-09-13T14:56:33.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:56:33.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:56:33.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:56:33.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:56:33.154769+0000", 2024-09-13T14:56:33.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3969999999999999e-06 2024-09-13T14:56:33.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:56:33.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:56:33.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:56:33.154774+0000", 2024-09-13T14:56:33.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4859999999999997e-06 2024-09-13T14:56:33.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:56:33.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:56:33.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:56:33.155583+0000", 2024-09-13T14:56:33.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00080909000000000005 2024-09-13T14:56:33.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:56:33.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:56:33.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:56:33.155594+0000", 2024-09-13T14:56:33.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1185e-05 2024-09-13T14:56:33.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:56:33.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.184 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:56:33.184 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:56:33.156215+0000", 2024-09-13T14:56:33.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000620124 2024-09-13T14:56:33.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:56:33.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.184 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:56:33.184 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:56:33.156684+0000", 2024-09-13T14:56:33.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046934899999999998 2024-09-13T14:56:33.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:56:33.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.184 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:56:33.184 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:56:33.157143+0000", 2024-09-13T14:56:33.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045944399999999999 2024-09-13T14:56:33.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:56:33.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:56:33.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:56:33.160988+0000", 2024-09-13T14:56:33.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0038448979999999998 2024-09-13T14:56:33.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:56:33.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:33.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:56:33.167310+0000", 2024-09-13T14:56:33.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0063218520000000002 2024-09-13T14:56:33.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:56:33.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:33.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:56:33.170172+0000", 2024-09-13T14:56:33.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028619750000000001 2024-09-13T14:56:33.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:56:33.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:56:33.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:56:33.170320+0000", 2024-09-13T14:56:33.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000148085 2024-09-13T14:56:33.186 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:56:33.186 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:56:33.186 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:56:33.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:56:33.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.186 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.46034.0:143158 9.2 e1243/1222 9:4659dad9:::benchmark_data_smithi151_261954_object40302:head v 1243'8980, mlcod=1243'8978)", 2024-09-13T14:56:33.187 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:56:33.169876+0000", 2024-09-13T14:56:33.187 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0026214160000000001, 2024-09-13T14:56:33.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027428589999999998, 2024-09-13T14:56:33.187 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:56:33.187 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:56:33.187 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-13T14:56:33.187 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:56:33.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:56:33.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:56:33.169876+0000", 2024-09-13T14:56:33.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:56:33.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:56:33.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:56:33.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:56:33.169876+0000", 2024-09-13T14:56:33.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:56:33.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:56:33.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:56:33.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:56:33.169878+0000", 2024-09-13T14:56:33.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3650000000000001e-06 2024-09-13T14:56:33.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:56:33.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:56:33.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:56:33.169887+0000", 2024-09-13T14:56:33.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0960000000000007e-06 2024-09-13T14:56:33.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:56:33.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:56:33.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:56:33.169907+0000", 2024-09-13T14:56:33.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0171000000000001e-05 2024-09-13T14:56:33.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:56:33.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:56:33.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:56:33.169915+0000", 2024-09-13T14:56:33.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.388e-06 2024-09-13T14:56:33.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:56:33.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:56:33.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:56:33.171502+0000", 2024-09-13T14:56:33.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015867920000000001 2024-09-13T14:56:33.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:56:33.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:56:33.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:56:33.171546+0000", 2024-09-13T14:56:33.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3395e-05 2024-09-13T14:56:33.190 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:56:33.190 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:56:33.190 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:56:33.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:56:33.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.191 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.46034.0:143152 9.4 e1243/1229 ondisk, result = 0)", 2024-09-13T14:56:33.191 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:56:33.169938+0000", 2024-09-13T14:56:33.191 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0025597319999999999, 2024-09-13T14:56:33.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027320119999999998, 2024-09-13T14:56:33.191 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:56:33.191 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:56:33.191 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-13T14:56:33.191 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:56:33.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:56:33.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:56:33.169938+0000", 2024-09-13T14:56:33.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:56:33.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:56:33.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:56:33.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:56:33.169938+0000", 2024-09-13T14:56:33.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:56:33.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:56:33.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:56:33.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:56:33.169938+0000", 2024-09-13T14:56:33.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9999999999999999e-07 2024-09-13T14:56:33.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:56:33.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:56:33.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:56:33.169941+0000", 2024-09-13T14:56:33.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.604e-06 2024-09-13T14:56:33.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:56:33.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:56:33.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:56:33.169947+0000", 2024-09-13T14:56:33.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8499999999999999e-06 2024-09-13T14:56:33.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:56:33.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:56:33.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:56:33.169951+0000", 2024-09-13T14:56:33.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2849999999999996e-06 2024-09-13T14:56:33.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:56:33.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:56:33.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:56:33.170130+0000", 2024-09-13T14:56:33.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000179223 2024-09-13T14:56:33.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:56:33.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:56:33.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:56:33.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:56:33.170153+0000", 2024-09-13T14:56:33.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2370000000000001e-05 2024-09-13T14:56:33.194 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:56:33.194 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:56:33.194 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:56:33.194 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:56:33.194 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-13T14:56:33.195 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 3 2024-09-13T14:56:33.195 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:56:33.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:33.214 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:33.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:33.315 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:33.368 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:33.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:33.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:33.470 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:33.508 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:33.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:33.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:33.610 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:33.666 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:33.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:33.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:33.768 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:33.805 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:33.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:33.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:33.907 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:33.930 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:33.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:33.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:34.032 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:34.070 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:34.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:34.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:34.172 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:34.205 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:34.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:34.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:34.307 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:34.347 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:34.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:34.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:34.369 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:34.402 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:56:34.449 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:34.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:34.495 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:34.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:34.596 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:34.635 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:34.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:34.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:34.736 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:34.760 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:56:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:56:34] "GET /metrics HTTP/1.1" 200 36681 "" "Prometheus/2.51.0" 2024-09-13T14:56:34.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:34.783 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:34.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:34.884 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:34.927 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:34.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:34.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:35.025 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T14:56:35.026 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:34 smithi151 ceph-mon[104450]: pgmap v1963: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 954 op/s; 14172/194568 objects degraded (7.284%); 1866/194568 objects misplaced (0.959%); 2.9 MiB/s, 46 objects/s recovering 2024-09-13T14:56:35.027 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:34 smithi151 ceph-mon[108966]: pgmap v1963: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 954 op/s; 14172/194568 objects degraded (7.284%); 1866/194568 objects misplaced (0.959%); 2.9 MiB/s, 46 objects/s recovering 2024-09-13T14:56:35.028 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:35.069 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:34 smithi184 ceph-mon[92527]: pgmap v1963: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 954 op/s; 14172/194568 objects degraded (7.284%); 1866/194568 objects misplaced (0.959%); 2.9 MiB/s, 46 objects/s recovering 2024-09-13T14:56:35.111 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:35.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:35.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:35.212 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:35.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:35.271 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:35.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:35.372 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:35.410 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:35.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:35.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:35.512 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:35.541 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:35.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:35.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:35.643 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:35.671 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:35.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:35.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:35.773 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:35.798 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:35.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:35.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:35.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:35 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 14172/194568 objects degraded (7.284%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:35.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:35 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 14172/194568 objects degraded (7.284%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:35.900 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:35.933 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:35.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:35.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:36.034 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:36.048 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:35 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 14172/194568 objects degraded (7.284%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:36.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:36.067 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:36.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:36.168 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:36.207 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:36.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:36.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:36.309 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:36.353 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:36.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:36.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:36.455 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:36.490 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:36.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:36.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:36.591 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:36.597 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:36.659 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:36.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:36.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:36.761 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:36.794 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:36.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:36.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:36.896 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:36.927 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:36.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:36.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:37.029 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:37.065 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:37.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:37.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:37.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:36 smithi184 ceph-mon[92527]: pgmap v1964: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 918 op/s; 14172/188556 objects degraded (7.516%); 1866/188556 objects misplaced (0.990%); 1.4 MiB/s, 21 objects/s recovering 2024-09-13T14:56:37.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:36 smithi151 ceph-mon[104450]: pgmap v1964: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 918 op/s; 14172/188556 objects degraded (7.516%); 1866/188556 objects misplaced (0.990%); 1.4 MiB/s, 21 objects/s recovering 2024-09-13T14:56:37.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:36 smithi151 ceph-mon[108966]: pgmap v1964: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 918 op/s; 14172/188556 objects degraded (7.516%); 1866/188556 objects misplaced (0.990%); 1.4 MiB/s, 21 objects/s recovering 2024-09-13T14:56:37.168 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:37.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:37.208 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:37.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:37.309 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:37.361 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:37.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:37.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:37.462 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:37.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:37.504 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:37.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:37.570 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:37.605 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:37.632 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:37.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:37.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:37.734 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:37.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:37.786 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:37.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:37.887 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:37.915 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:37.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:37.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:38.017 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:38.056 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:38.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:38.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:38.158 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:38.221 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:38.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:38.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:38.323 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:38.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:38.359 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:38.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:38.460 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:38.493 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:38.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:38.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:38.595 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:38.615 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:38.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:38.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:38.717 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:38.751 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:38.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:38.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:38.853 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:38.891 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:38.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:38.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:38.992 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:39.025 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:39.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:39.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:39.126 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:39.167 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:39.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:39.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:39.269 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:39.302 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:39.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:39.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:39.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:38 smithi184 ceph-mon[92527]: pgmap v1965: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 964 op/s; 14296/181770 objects degraded (7.865%); 1561/181770 objects misplaced (0.859%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:56:39.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:38 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:56:39.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:38 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:56:39.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:38 smithi151 ceph-mon[104450]: pgmap v1965: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 964 op/s; 14296/181770 objects degraded (7.865%); 1561/181770 objects misplaced (0.859%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:56:39.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:38 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:56:39.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:38 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:56:39.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:38 smithi151 ceph-mon[108966]: pgmap v1965: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 964 op/s; 14296/181770 objects degraded (7.865%); 1561/181770 objects misplaced (0.859%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:56:39.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:38 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:56:39.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:38 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:56:39.404 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:39.440 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:39.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:39.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:39.543 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:39.583 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:39.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:39.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:39.685 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:39.716 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:39.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:39.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:39.818 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:39.851 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:39.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:39.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:39.953 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:39.977 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:39.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:39.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:40.079 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:40.110 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:40.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:40.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:40.212 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:40.248 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:40.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:40.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:40.350 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:40.371 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:40.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:40.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:40.473 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:40.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:40.507 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:40.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:40.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:56:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:56:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:56:40.608 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:40.642 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:40.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:40.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:40.744 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:40.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:40.781 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:40.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:40.882 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:40.918 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:40.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:40.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:41.020 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:41.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:41.063 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:41.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:41.164 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:41.195 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:41.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:41.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:41.297 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:41.325 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:41.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:41.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:41.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:40 smithi184 ceph-mon[92527]: pgmap v1966: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 939 op/s; 14296/177516 objects degraded (8.053%); 1561/177516 objects misplaced (0.879%); 2.2 MiB/s, 35 objects/s recovering 2024-09-13T14:56:41.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:40 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 14296/177516 objects degraded (8.053%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:41.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:40 smithi151 ceph-mon[104450]: pgmap v1966: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 939 op/s; 14296/177516 objects degraded (8.053%); 1561/177516 objects misplaced (0.879%); 2.2 MiB/s, 35 objects/s recovering 2024-09-13T14:56:41.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:40 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 14296/177516 objects degraded (8.053%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:41.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:40 smithi151 ceph-mon[108966]: pgmap v1966: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 939 op/s; 14296/177516 objects degraded (8.053%); 1561/177516 objects misplaced (0.879%); 2.2 MiB/s, 35 objects/s recovering 2024-09-13T14:56:41.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:40 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 14296/177516 objects degraded (8.053%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:41.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:41.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:41.468 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:41.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:41.569 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:41.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:41.611 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:41.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:41.712 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:41.745 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:41.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:41.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:41.847 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:41.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:41.878 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:41.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:41.980 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:42.012 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:42.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:42.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:42.114 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:42.152 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:42.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:42.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:42.254 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:42.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:42.281 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:42.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:42.382 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:42.422 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:42.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:42.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:42.524 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:42.562 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:42.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:42.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:42.664 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:42.692 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:42.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:42.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:42.795 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:42.832 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:42.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:42.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:42.934 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:42.962 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:42.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:42.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:43.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:43.086 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:43.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:43.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:43.188 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:43.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:43.228 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:43.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:43.329 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:43.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:42 smithi184 ceph-mon[92527]: pgmap v1967: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 972 op/s; 14385/171786 objects degraded (8.374%); 1249/171786 objects misplaced (0.727%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:56:43.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:42 smithi151 ceph-mon[104450]: pgmap v1967: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 972 op/s; 14385/171786 objects degraded (8.374%); 1249/171786 objects misplaced (0.727%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:56:43.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:42 smithi151 ceph-mon[108966]: pgmap v1967: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 972 op/s; 14385/171786 objects degraded (8.374%); 1249/171786 objects misplaced (0.727%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:56:43.361 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:43.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:43.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:43.464 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:43.495 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:43.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:43.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:43.597 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:43.632 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:43.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:43.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:43.734 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:43.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:43.776 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:43.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:43.877 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:43.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:43.916 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:43.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:44.017 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:44.047 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:44.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:44.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:44.149 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:44.181 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:44.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:44.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:44.283 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:44.315 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:44.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:44.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:44.417 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:44.447 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:44.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:44.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:44.549 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:44.579 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:44.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:44.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:44.681 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:44.720 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:44.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:44.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:44.823 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:44.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:56:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:56:44] "GET /metrics HTTP/1.1" 200 36681 "" "Prometheus/2.51.0" 2024-09-13T14:56:44.860 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:44.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:44.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:44.962 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:44.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:44.998 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:44.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:45.100 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:45.120 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:45.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:45.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:45.222 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:45.255 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:45.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:45.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:45.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:44 smithi184 ceph-mon[92527]: pgmap v1968: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 878 op/s; 14385/167475 objects degraded (8.589%); 1249/167475 objects misplaced (0.746%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:56:45.357 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:45.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:44 smithi151 ceph-mon[104450]: pgmap v1968: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 878 op/s; 14385/167475 objects degraded (8.589%); 1249/167475 objects misplaced (0.746%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:56:45.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:44 smithi151 ceph-mon[108966]: pgmap v1968: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 878 op/s; 14385/167475 objects degraded (8.589%); 1249/167475 objects misplaced (0.746%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:56:45.396 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:45.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:45.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:45.498 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:45.534 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:45.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:45.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:45.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:45.671 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:45.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:45.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:45.773 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:45.795 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:45.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:45.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:45.897 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:45.936 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:45.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:45.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:46.039 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:46.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:46.084 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:46.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:46.185 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:46.221 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:46.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:46.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:46.323 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:46.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:46.349 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:46.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:46.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:45 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 14385/167475 objects degraded (8.589%), 12 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:46.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:45 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 14385/167475 objects degraded (8.589%), 12 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:46.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:45 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 14385/167475 objects degraded (8.589%), 12 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:46.450 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:46.484 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:46.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:46.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:46.586 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:46.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:46.621 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:46.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:46.722 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:46.760 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:46.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:46.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:46.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:46.898 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:46.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:46.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:47.000 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:47.040 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:47.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:47.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:47.142 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:47.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:47.184 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:47.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:47.285 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:47.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:47.318 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:47.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:47.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:46 smithi184 ceph-mon[92527]: pgmap v1969: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 924 op/s; 14385/161253 objects degraded (8.921%); 1249/161253 objects misplaced (0.775%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:56:47.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:47.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:46 smithi151 ceph-mon[104450]: pgmap v1969: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 924 op/s; 14385/161253 objects degraded (8.921%); 1249/161253 objects misplaced (0.775%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:56:47.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:47.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:46 smithi151 ceph-mon[108966]: pgmap v1969: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 924 op/s; 14385/161253 objects degraded (8.921%); 1249/161253 objects misplaced (0.775%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:56:47.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:47.419 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:47.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:47.459 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:47.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:47.540 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:47.560 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:47.616 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:47.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:47.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:47.718 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:47.784 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:47.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:47.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:47.885 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:47.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:47.919 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:47.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:48.020 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:48.052 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:48.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:48.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:48.154 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:48.188 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:48.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:48.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:48.289 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:48.334 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:48.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:48.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:48.436 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:48.469 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:48.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:48.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:48.571 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:48.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:48.613 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:48.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:48.714 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:48.754 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:48.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:48.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:48.856 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:48.888 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:48.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:48.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:48.925 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T14:56:48.989 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:49.018 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:49.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:49.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:49.120 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:49.154 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:49.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:49.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:49.256 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:49.272 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:48 smithi184 ceph-mon[92527]: pgmap v1970: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 956 op/s; 14408/154080 objects degraded (9.351%); 993/154080 objects misplaced (0.644%); 3.3 MiB/s, 53 objects/s recovering 2024-09-13T14:56:49.272 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:48 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2836588999' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:56:49.272 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:48 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:56:49.295 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:49.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:49.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:49.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:48 smithi151 ceph-mon[108966]: pgmap v1970: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 956 op/s; 14408/154080 objects degraded (9.351%); 993/154080 objects misplaced (0.644%); 3.3 MiB/s, 53 objects/s recovering 2024-09-13T14:56:49.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:48 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2836588999' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:56:49.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:48 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:56:49.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:48 smithi151 ceph-mon[104450]: pgmap v1970: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 956 op/s; 14408/154080 objects degraded (9.351%); 993/154080 objects misplaced (0.644%); 3.3 MiB/s, 53 objects/s recovering 2024-09-13T14:56:49.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:48 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2836588999' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:56:49.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:48 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:56:49.397 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:49.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:49.439 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:49.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:49.540 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:49.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:49.575 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:49.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:49.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:49.713 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:49.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:49.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:49.815 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:49.853 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:49.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:49.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:49.955 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:49.985 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:49.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:49.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:49.988 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:49 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T14:56:49.988 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:49 smithi151 ceph-mon[104450]: osdmap e1244: 8 total, 8 up, 8 in 2024-09-13T14:56:49.988 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:49 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T14:56:49.989 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:49 smithi151 ceph-mon[108966]: osdmap e1244: 8 total, 8 up, 8 in 2024-09-13T14:56:50.087 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:50.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:50.134 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:50.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:50.213 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:49 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T14:56:50.213 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:49 smithi184 ceph-mon[92527]: osdmap e1244: 8 total, 8 up, 8 in 2024-09-13T14:56:50.235 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:50.269 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:50.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:50.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:50.371 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:50.411 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:50.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:50.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:50.513 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:50.543 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:50.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:50.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:50.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:56:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:56:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:56:50.645 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:50.676 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:50.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:50.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:50.777 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:50.817 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:50.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:50.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:50.919 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:50.952 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:50.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:50.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:50.953 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-13T14:56:51.054 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:51.095 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:51.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:51.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:51.197 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:51.229 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:50 smithi151 ceph-mon[104450]: pgmap v1972: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 930 op/s; 14408/149568 objects degraded (9.633%); 993/149568 objects misplaced (0.664%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:56:51.230 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:50 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 14408/149568 objects degraded (9.633%), 12 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:51.230 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:50 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3590081952' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:51.230 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:50 smithi151 ceph-mon[108966]: pgmap v1972: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 930 op/s; 14408/149568 objects degraded (9.633%); 993/149568 objects misplaced (0.664%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:56:51.230 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:50 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 14408/149568 objects degraded (9.633%), 12 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:51.230 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:50 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3590081952' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:51.232 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:51.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:51.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:51.334 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:51.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:50 smithi184 ceph-mon[92527]: pgmap v1972: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 930 op/s; 14408/149568 objects degraded (9.633%); 993/149568 objects misplaced (0.664%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:56:51.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:50 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 14408/149568 objects degraded (9.633%), 12 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:51.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:50 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3590081952' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:51.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:51.378 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:51.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:51.480 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:51.510 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:51.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:51.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:51.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:51.648 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:51.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:51.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:51.712 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T14:56:51.750 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:51.812 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:51.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:51.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:51.914 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:51.948 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:51.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:51.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:52.050 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:52.085 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:52.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:52.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:52.187 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:52.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:52.229 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:52.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:52.330 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:52.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:51 smithi184 ceph-mon[92527]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:52.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:51 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3590081952' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:56:52.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:51 smithi184 ceph-mon[92527]: osdmap e1245: 8 total, 8 up, 8 in 2024-09-13T14:56:52.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:51 smithi151 ceph-mon[104450]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:52.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:51 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3590081952' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:56:52.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:51 smithi151 ceph-mon[104450]: osdmap e1245: 8 total, 8 up, 8 in 2024-09-13T14:56:52.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:51 smithi151 ceph-mon[108966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:52.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:51 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3590081952' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:56:52.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:51 smithi151 ceph-mon[108966]: osdmap e1245: 8 total, 8 up, 8 in 2024-09-13T14:56:52.368 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:52.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:52.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:52.471 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:52.499 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:52.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:52.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:52.601 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:52.640 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:52.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:52.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:52.742 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:52.775 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:52.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:52.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:52.877 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:52.911 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:52.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:52.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:53.013 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:53.050 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:53.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:53.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:53.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:52 smithi151 ceph-mon[104450]: pgmap v1974: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 14511/143064 objects degraded (10.143%); 663/143064 objects misplaced (0.463%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:56:53.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:52 smithi151 ceph-mon[108966]: pgmap v1974: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 14511/143064 objects degraded (10.143%); 663/143064 objects misplaced (0.463%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:56:53.152 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:53.185 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:53.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:53.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:53.287 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:53.324 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:53.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:53.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:53.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:52 smithi184 ceph-mon[92527]: pgmap v1974: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 14511/143064 objects degraded (10.143%); 663/143064 objects misplaced (0.463%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:56:53.426 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:53.463 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:53.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:53.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:53.564 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:53.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:53.604 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:53.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:53.705 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:53.743 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:53.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:53.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:53.842 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:53.845 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:53.903 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:53.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:53.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:54.005 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:54.049 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:54.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:54.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:54.151 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:54.174 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:56:54.174 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-13T14:56:54.175 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-13T14:56:54.175 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-13T14:56:54.175 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:56:54.216 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:54.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:54.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:54.318 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:54.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:54.367 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:54.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:54.468 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:54.490 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:56:54.506 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:54.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:54.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:54.560 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:56:54.560 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:56:54.560 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:56:54.560 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:56:54.561 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:56:54.561 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:56:54.561 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:56:54.561 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:56:54.565 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:56:54.565 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:56:54.602 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:56:54.602 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:56:54.608 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:54.634 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:54.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:54.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:54.637 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:56:54.637 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:56:54.673 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:56:54.673 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:56:54.735 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:54.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:54.790 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:54.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:54.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:56:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:56:54] "GET /metrics HTTP/1.1" 200 36680 "" "Prometheus/2.51.0" 2024-09-13T14:56:54.891 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:54.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:54.926 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:54.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:55.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:55.066 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:55.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:55.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:55.168 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:55.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:55.201 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:55.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:55.302 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:55.340 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:55.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:55.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:55.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:54 smithi184 ceph-mon[92527]: pgmap v1975: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 2.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 955 op/s; 14511/138294 objects degraded (10.493%); 663/138294 objects misplaced (0.479%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:56:55.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:54 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3881823511' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:55.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:54 smithi151 ceph-mon[108966]: pgmap v1975: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 2.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 955 op/s; 14511/138294 objects degraded (10.493%); 663/138294 objects misplaced (0.479%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:56:55.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:54 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3881823511' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:55.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:54 smithi151 ceph-mon[104450]: pgmap v1975: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 2.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 955 op/s; 14511/138294 objects degraded (10.493%); 663/138294 objects misplaced (0.479%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:56:55.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:54 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3881823511' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:55.442 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:55.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:55.483 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:55.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:55.584 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:55.625 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:55.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:55.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:55.728 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:55.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:55.763 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:55.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:55.864 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:55.903 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:55.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:55.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:56.005 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:56.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:56.040 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:56.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:56.141 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:56.177 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:56.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:56.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:56.279 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:56.308 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:56.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:56.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:56.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:55 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 14511/138294 objects degraded (10.493%), 12 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:56.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:55 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 14511/138294 objects degraded (10.493%), 12 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:56.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:55 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 14511/138294 objects degraded (10.493%), 12 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:56:56.410 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:56.446 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:56.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:56.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:56.548 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:56.589 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:56.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:56.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:56.691 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:56.725 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:56.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:56.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:56.827 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:56.859 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:56.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:56.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:56.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:57.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:57.003 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:57.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:57.104 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:57.138 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:57.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:57.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:57.240 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:57.274 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:57.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:57.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:57.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:56 smithi184 ceph-mon[92527]: pgmap v1976: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 2.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 926 op/s; 14511/131835 objects degraded (11.007%); 663/131835 objects misplaced (0.503%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:56:57.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:56 smithi151 ceph-mon[104450]: pgmap v1976: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 2.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 926 op/s; 14511/131835 objects degraded (11.007%); 663/131835 objects misplaced (0.503%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:56:57.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:56 smithi151 ceph-mon[108966]: pgmap v1976: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 2.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 926 op/s; 14511/131835 objects degraded (11.007%); 663/131835 objects misplaced (0.503%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:56:57.377 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:57.418 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:57.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:57.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:57.520 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:57.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:57.562 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:57.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:57.663 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:57.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:57.700 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:57.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:57.801 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:57.840 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:57.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:57.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:57.943 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:57.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:57.983 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:57.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:58.084 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:58.119 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:58.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:58.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:58.221 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:58.267 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:58.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:58.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:58.369 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:58.411 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:58.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:58.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:58.513 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:58.555 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:58.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:58.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:58.658 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:58.693 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:58.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:58.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:58.794 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:58.829 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:58.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:58.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:58.922 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T14:56:58.932 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:59.020 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:59.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:59.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:59.122 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:59.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:59.164 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:59.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:59.265 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:59.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:59.305 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:59.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:59.337 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:56:58 smithi151 ceph-mon[104450]: pgmap v1977: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 14567/124260 objects degraded (11.723%); 341/124260 objects misplaced (0.274%); 3.5 MiB/s, 56 objects/s recovering 2024-09-13T14:56:59.337 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:56:59 smithi151 ceph-mon[108966]: pgmap v1977: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 14567/124260 objects degraded (11.723%); 341/124260 objects misplaced (0.274%); 3.5 MiB/s, 56 objects/s recovering 2024-09-13T14:56:59.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:56:58 smithi184 ceph-mon[92527]: pgmap v1977: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 14567/124260 objects degraded (11.723%); 341/124260 objects misplaced (0.274%); 3.5 MiB/s, 56 objects/s recovering 2024-09-13T14:56:59.407 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:59.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:59.442 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:59.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:59.543 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:59.585 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:56:59.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:59.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:59.687 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:59.715 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:59.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:59.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:59.817 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:59.853 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:56:59.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:59.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:59.873 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:56:59.955 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:59.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:00.000 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:00.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:00.101 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:00.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:00 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 14567/124260 objects degraded (11.723%), 12 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:00 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 14567/124260 objects degraded (11.723%), 12 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:00.130 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:00.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:00.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:00.232 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:00.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:00.270 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:00.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:00.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:00 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 14567/124260 objects degraded (11.723%), 12 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:00.354 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:57:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:57:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:57:00.371 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:00.385 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:00.473 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:00.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:00.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:00.575 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:00.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:00.613 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:00.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:00.714 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:00.746 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:00.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:00.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:00.847 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:00.889 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:00.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:00.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:00.991 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:01.026 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:01.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:01.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:01.087 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:01 smithi151 ceph-mon[104450]: pgmap v1978: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 14567/119352 objects degraded (12.205%); 341/119352 objects misplaced (0.286%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:57:01.088 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:01 smithi151 ceph-mon[108966]: pgmap v1978: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 14567/119352 objects degraded (12.205%); 341/119352 objects misplaced (0.286%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:57:01.128 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:01.169 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:01.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:01.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:01.271 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:01.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:01.313 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:01.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:01.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:01 smithi184 ceph-mon[92527]: pgmap v1978: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 14567/119352 objects degraded (12.205%); 341/119352 objects misplaced (0.286%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:57:01.415 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:01.455 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:01.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:01.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:01.557 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:01.589 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:01.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:01.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:01.691 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:01.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:01.728 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:01.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:01.829 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:01.867 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:01.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:01.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:01.969 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:02.014 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:02.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:02.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:02.116 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:02.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:02.153 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:02.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:02.254 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:02.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:02.287 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:02.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:02.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:02.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:02.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:02.389 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:02.425 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:02.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:02.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:02.527 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:02.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:02.561 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:02.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:02.662 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:02.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:02.704 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:02.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:02.805 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:02.840 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:02.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:02.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:02.942 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:02.977 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:02.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:02.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:03.079 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:03.113 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:03.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:03.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:03.215 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:03.222 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T14:57:03.246 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:03.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:03.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:03.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:03.356 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:03 smithi184 ceph-mon[92527]: pgmap v1979: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 2.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 948 op/s; 14597/112422 objects degraded (12.984%); 3.3 MiB/s, 53 objects/s recovering 2024-09-13T14:57:03.356 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:03 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/216324336' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:57:03.356 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:03 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:57:03.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:03 smithi151 ceph-mon[104450]: pgmap v1979: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 2.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 948 op/s; 14597/112422 objects degraded (12.984%); 3.3 MiB/s, 53 objects/s recovering 2024-09-13T14:57:03.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:03 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/216324336' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:57:03.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:03 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:57:03.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:03 smithi151 ceph-mon[108966]: pgmap v1979: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 2.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 948 op/s; 14597/112422 objects degraded (12.984%); 3.3 MiB/s, 53 objects/s recovering 2024-09-13T14:57:03.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:03 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/216324336' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:57:03.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:03 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:57:03.389 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:03.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:03.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:03.491 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:03.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:03.529 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:03.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:03.630 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:03.670 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:03.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:03.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:03.772 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:03.809 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:03.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:03.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:03.911 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:03.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:03.933 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:03.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:04.034 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:04.074 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:04.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:04.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:04.176 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:04.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:04.201 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:04.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:04.302 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:04.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:04.342 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:04.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:04.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:04 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:57:04.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:04 smithi184 ceph-mon[92527]: osdmap e1246: 8 total, 8 up, 8 in 2024-09-13T14:57:04.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:04 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:57:04.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:04 smithi151 ceph-mon[104450]: osdmap e1246: 8 total, 8 up, 8 in 2024-09-13T14:57:04.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:04 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:57:04.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:04 smithi151 ceph-mon[108966]: osdmap e1246: 8 total, 8 up, 8 in 2024-09-13T14:57:04.443 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:04.475 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:04.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:04.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:04.578 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:04.599 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:04.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:04.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:04.701 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:04.734 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:04.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:04.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:04.836 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:04.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:57:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:57:04] "GET /metrics HTTP/1.1" 200 36678 "" "Prometheus/2.51.0" 2024-09-13T14:57:04.869 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:04.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:04.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:04.971 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:05.009 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:05.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:05.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:05.111 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:05.149 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:05.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:05.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:05.250 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:05.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:05.296 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:05.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:05.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:05 smithi184 ceph-mon[92527]: pgmap v1981: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 2.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 14597/107451 objects degraded (13.585%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:57:05.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:05 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 14597/107451 objects degraded (13.585%), 12 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:05.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:05 smithi151 ceph-mon[104450]: pgmap v1981: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 2.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 14597/107451 objects degraded (13.585%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:57:05.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:05 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 14597/107451 objects degraded (13.585%), 12 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:05.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:05 smithi151 ceph-mon[108966]: pgmap v1981: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 2.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 14597/107451 objects degraded (13.585%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:57:05.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:05 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 14597/107451 objects degraded (13.585%), 12 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:05.397 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:05.430 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:05.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:05.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:05.532 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:05.571 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:05.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:05.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:05.673 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:05.713 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:05.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:05.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:05.815 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:05.848 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:05.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:05.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:05.949 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:05.979 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:05.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:05.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:06.081 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:06.122 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:06.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:06.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:06.224 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:06.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:06.259 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:06.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:06.360 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:06.395 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:06.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:06.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:06.497 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:06.534 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:06.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:06.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:06.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:06.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:06.678 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:06.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:06.779 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:06.815 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:06.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:06.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:06.917 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:06.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:06.954 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:06.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:07.055 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:07.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:07.078 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:07.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:07.179 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:07.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:07.218 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:07.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:07.319 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:07.347 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:07.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:07.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:07.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:07 smithi184 ceph-mon[92527]: pgmap v1982: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 2.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 14597/100920 objects degraded (14.464%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:57:07.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:07 smithi151 ceph-mon[104450]: pgmap v1982: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 2.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 14597/100920 objects degraded (14.464%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:57:07.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:07 smithi151 ceph-mon[108966]: pgmap v1982: 119 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 2.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 14597/100920 objects degraded (14.464%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:57:07.449 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:07.491 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:07.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:07.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:07.597 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:07.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:07.625 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:07.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:07.726 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:07.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:07.763 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:07.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:07.864 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:07.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:07.902 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:07.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:08.003 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:08.035 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:08.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:08.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:08.137 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:08.168 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:08.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:08.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:08.270 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:08.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:08.307 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:08.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:08.408 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:08.446 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:08.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:08.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:08.548 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:08.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:08.579 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:08.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:08.680 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:08.715 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:08.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:08.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:08.817 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:08.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:08.861 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:08.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:08.963 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:08.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:08.998 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:08.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:09.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:09.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:09.133 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:09.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:09.234 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:09.269 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:09.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:09.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:09.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:09 smithi184 ceph-mon[92527]: pgmap v1983: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 14245/93669 objects degraded (15.208%); 1.9 MiB/s, 66 objects/s recovering 2024-09-13T14:57:09.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:09 smithi151 ceph-mon[104450]: pgmap v1983: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 14245/93669 objects degraded (15.208%); 1.9 MiB/s, 66 objects/s recovering 2024-09-13T14:57:09.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:09 smithi151 ceph-mon[108966]: pgmap v1983: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 14245/93669 objects degraded (15.208%); 1.9 MiB/s, 66 objects/s recovering 2024-09-13T14:57:09.371 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:09.403 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:09.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:09.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:09.505 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:09.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:09.527 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:09.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:09.616 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T14:57:09.628 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:09.661 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:09.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:09.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:09.763 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:09.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:09.806 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:09.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:09.907 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:09.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:09.950 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:09.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:10.051 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:10.117 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:10.119 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:10.136 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:10.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:10.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:10.238 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:10.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:10.300 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:10.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:10.328 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:10 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 14245/93669 objects degraded (15.208%), 12 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:10.328 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:10 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 14245/93669 objects degraded (15.208%), 12 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:10.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:10 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 14245/93669 objects degraded (15.208%), 12 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:10.354 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:57:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:57:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:57:10.401 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:10.436 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:10.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:10.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:10.537 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:10.570 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:10.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:10.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:10.675 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:10.709 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:10.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:10.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:10.811 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:10.875 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:10.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:10.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:10.977 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:11.021 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:11.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:11.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:11.123 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:11.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:11.175 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:11.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:11.276 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:11.302 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:11.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:11.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:11.343 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:11 smithi151 ceph-mon[104450]: pgmap v1984: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 14245/88890 objects degraded (16.025%); 1.9 MiB/s, 66 objects/s recovering 2024-09-13T14:57:11.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:11 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:57:11.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:11 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:57:11.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:11 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:57:11.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:11 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:57:11.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:11 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:57:11.344 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:11 smithi151 ceph-mon[108966]: pgmap v1984: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 14245/88890 objects degraded (16.025%); 1.9 MiB/s, 66 objects/s recovering 2024-09-13T14:57:11.344 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:11 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:57:11.344 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:11 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:57:11.345 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:11 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:57:11.345 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:11 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:57:11.345 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:11 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:57:11.405 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:11.429 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:11.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:11.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:11.531 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:11.553 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:11.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:11.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:11.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:11 smithi184 ceph-mon[92527]: pgmap v1984: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 14245/88890 objects degraded (16.025%); 1.9 MiB/s, 66 objects/s recovering 2024-09-13T14:57:11.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:11 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:57:11.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:11 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:57:11.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:11 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:57:11.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:11 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:57:11.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:11 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:57:11.656 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:11.677 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:11.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:11.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:11.779 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:11.800 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:11.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:11.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:11.902 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:11.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:11.924 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:11.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:12.025 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:12.046 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:12.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:12.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:12.148 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:12.170 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:12.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:12.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:12.272 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:12.308 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:12.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:12.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:12.410 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:12.432 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:12.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:12.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:12.534 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:12.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:12.574 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:12.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:12.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:12 smithi184 ceph-mon[92527]: osdmap e1247: 8 total, 8 up, 8 in 2024-09-13T14:57:12.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:12 smithi151 ceph-mon[104450]: osdmap e1247: 8 total, 8 up, 8 in 2024-09-13T14:57:12.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:12 smithi151 ceph-mon[108966]: osdmap e1247: 8 total, 8 up, 8 in 2024-09-13T14:57:12.675 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:12.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:12.712 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:12.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:12.813 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:12.839 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:12.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:12.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:12.941 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:12.975 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:12.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:12.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:13.077 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:13.114 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:13.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:13.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:13.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:13.249 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:13.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:13.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:13.350 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:13.372 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:13.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:13.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:13.474 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:13.507 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:13.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:13.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:13.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:13 smithi184 ceph-mon[92527]: pgmap v1986: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 952 op/s; 13852/82587 objects degraded (16.773%); 0 B/s, 85 objects/s recovering 2024-09-13T14:57:13.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:13 smithi184 ceph-mon[92527]: osdmap e1248: 8 total, 8 up, 8 in 2024-09-13T14:57:13.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:13 smithi151 ceph-mon[104450]: pgmap v1986: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 952 op/s; 13852/82587 objects degraded (16.773%); 0 B/s, 85 objects/s recovering 2024-09-13T14:57:13.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:13 smithi151 ceph-mon[104450]: osdmap e1248: 8 total, 8 up, 8 in 2024-09-13T14:57:13.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:13 smithi151 ceph-mon[108966]: pgmap v1986: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 952 op/s; 13852/82587 objects degraded (16.773%); 0 B/s, 85 objects/s recovering 2024-09-13T14:57:13.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:13 smithi151 ceph-mon[108966]: osdmap e1248: 8 total, 8 up, 8 in 2024-09-13T14:57:13.610 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:13.647 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:13.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:13.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:13.749 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:13.789 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:13.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:13.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:13.891 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:13.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:13.926 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:13.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:14.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:14.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:14.060 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:14.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:14.161 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:14.198 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:14.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:14.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:14.300 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:14.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:14.330 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:14.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:14.431 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:14.473 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:14.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:14.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:14.575 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:14.597 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:14.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:14.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:14.699 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:14.735 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:14.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:14.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:14.837 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:14.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:57:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:57:14] "GET /metrics HTTP/1.1" 200 36678 "" "Prometheus/2.51.0" 2024-09-13T14:57:14.876 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:14.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:14.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:14.978 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:15.007 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:15.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:15.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:15.109 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:15.143 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:15.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:15.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:15.245 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:15.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:15.285 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:15.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:15.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:15.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:15.431 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:15.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:15.533 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:15.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:15.574 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:15.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:15.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:15 smithi184 ceph-mon[92527]: pgmap v1988: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 922 op/s; 13851/78762 objects degraded (17.586%); 32 KiB/s, 93 objects/s recovering 2024-09-13T14:57:15.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:15 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 13851/78762 objects degraded (17.586%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:15.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:15 smithi151 ceph-mon[104450]: pgmap v1988: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 922 op/s; 13851/78762 objects degraded (17.586%); 32 KiB/s, 93 objects/s recovering 2024-09-13T14:57:15.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:15 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 13851/78762 objects degraded (17.586%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:15.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:15 smithi151 ceph-mon[108966]: pgmap v1988: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 922 op/s; 13851/78762 objects degraded (17.586%); 32 KiB/s, 93 objects/s recovering 2024-09-13T14:57:15.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:15 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 13851/78762 objects degraded (17.586%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:15.675 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:15.717 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:15.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:15.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:15.819 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:15.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:15.862 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:15.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:15.963 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:15.996 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:15.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:15.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:16.100 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:16.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:16.139 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:16.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:16.240 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:16.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:16.279 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:16.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:16.380 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:16.412 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:16.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:16.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:16.514 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:16.548 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:16.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:16.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:16.651 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:16.695 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:16.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:16.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:16.797 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:16.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:16.836 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:16.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:16.937 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:16.960 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:16.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:16.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:17.062 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:17.096 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:17.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:17.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:17.198 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:17.242 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:17.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:17.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:17.344 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:17.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:17.383 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:17.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:17.485 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:17.523 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:17.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:17.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:17.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:17 smithi184 ceph-mon[92527]: pgmap v1989: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 830 op/s; 12899/73722 objects degraded (17.497%); 263/73722 objects misplaced (0.357%); 3.8 MiB/s, 135 objects/s recovering 2024-09-13T14:57:17.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:17.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:17 smithi151 ceph-mon[104450]: pgmap v1989: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 830 op/s; 12899/73722 objects degraded (17.497%); 263/73722 objects misplaced (0.357%); 3.8 MiB/s, 135 objects/s recovering 2024-09-13T14:57:17.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:17.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:17 smithi151 ceph-mon[108966]: pgmap v1989: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 830 op/s; 12899/73722 objects degraded (17.497%); 263/73722 objects misplaced (0.357%); 3.8 MiB/s, 135 objects/s recovering 2024-09-13T14:57:17.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:17.624 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:17.663 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:17.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:17.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:17.766 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:17.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:17.814 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:17.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:17.915 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:17.952 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:17.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:17.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:18.054 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:18.086 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:18.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:18.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:18.188 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:18.225 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:18.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:18.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:18.327 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:18.358 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:18.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:18.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:18.460 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:18.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:18.515 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:18.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:18.616 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:18.652 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:18.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:18.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:18.754 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:18.796 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:18.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:18.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:18.899 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:18.905 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:18.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:18.926 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:18.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:19.027 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:19.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:19.057 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:19.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:19.158 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:19.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:19.199 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:19.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:19.300 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:19.334 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:19.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:19.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:19.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:19 smithi151 ceph-mon[104450]: pgmap v1990: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 1.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 908 op/s; 12876/67074 objects degraded (19.197%); 263/67074 objects misplaced (0.392%); 3.8 MiB/s, 138 objects/s recovering 2024-09-13T14:57:19.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:19 smithi151 ceph-mon[108966]: pgmap v1990: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 1.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 908 op/s; 12876/67074 objects degraded (19.197%); 263/67074 objects misplaced (0.392%); 3.8 MiB/s, 138 objects/s recovering 2024-09-13T14:57:19.405 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:19 smithi184 ceph-mon[92527]: pgmap v1990: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 1.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 908 op/s; 12876/67074 objects degraded (19.197%); 263/67074 objects misplaced (0.392%); 3.8 MiB/s, 138 objects/s recovering 2024-09-13T14:57:19.438 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:19.459 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:19.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:19.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:19.561 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:19.585 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:19.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:19.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:19.686 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:19.708 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:19.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:19.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:19.810 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:19.831 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:19.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:19.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:19.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:19.959 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:19.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:19.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:20.061 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:20.083 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:20.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:20.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:20.180 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-13T14:57:20.185 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:20.206 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:20.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:20.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:20.309 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:20.340 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:20.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:20.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:20.447 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:20.481 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:20.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:20.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:20.584 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:20.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:57:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:57:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:57:20.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:20 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/832370900' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:57:20.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:20 smithi184 ceph-mon[92527]: osdmap e1249: 8 total, 8 up, 8 in 2024-09-13T14:57:20.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:20 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:57:20.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:20 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 12876/67074 objects degraded (19.197%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:20.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:20 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/832370900' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:57:20.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:20 smithi151 ceph-mon[104450]: osdmap e1249: 8 total, 8 up, 8 in 2024-09-13T14:57:20.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:20 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:57:20.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:20 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 12876/67074 objects degraded (19.197%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:20.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:20 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/832370900' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:57:20.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:20 smithi151 ceph-mon[108966]: osdmap e1249: 8 total, 8 up, 8 in 2024-09-13T14:57:20.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:20 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:57:20.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:20 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 12876/67074 objects degraded (19.197%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:20.621 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:20.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:20.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:20.723 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:20.759 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:20.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:20.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:20.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:20.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:20.899 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:20.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:21.000 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:21.039 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:21.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:21.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:21.141 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:21.173 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:21.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:21.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:21.275 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:21.310 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:21.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:21.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:21.412 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:21.439 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:21.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:21.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:21.541 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:21.579 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:21.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:21.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:21.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:21 smithi184 ceph-mon[92527]: pgmap v1992: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 832 op/s; 12684/62604 objects degraded (20.261%); 263/62604 objects misplaced (0.420%); 3.8 MiB/s, 113 objects/s recovering 2024-09-13T14:57:21.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:21 smithi184 ceph-mon[92527]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:57:21.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:21 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:57:21.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:21 smithi184 ceph-mon[92527]: osdmap e1250: 8 total, 8 up, 8 in 2024-09-13T14:57:21.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:21 smithi151 ceph-mon[104450]: pgmap v1992: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 832 op/s; 12684/62604 objects degraded (20.261%); 263/62604 objects misplaced (0.420%); 3.8 MiB/s, 113 objects/s recovering 2024-09-13T14:57:21.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:21 smithi151 ceph-mon[104450]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:57:21.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:21 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:57:21.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:21 smithi151 ceph-mon[104450]: osdmap e1250: 8 total, 8 up, 8 in 2024-09-13T14:57:21.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:21 smithi151 ceph-mon[108966]: pgmap v1992: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 832 op/s; 12684/62604 objects degraded (20.261%); 263/62604 objects misplaced (0.420%); 3.8 MiB/s, 113 objects/s recovering 2024-09-13T14:57:21.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:21 smithi151 ceph-mon[108966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:57:21.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:21 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:57:21.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:21 smithi151 ceph-mon[108966]: osdmap e1250: 8 total, 8 up, 8 in 2024-09-13T14:57:21.681 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:21.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:21.718 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:21.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:21.819 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:21.845 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:57:21.845 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:57:21.849 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:21.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:21.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:21.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:21.991 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:21.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:21.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:22.092 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:22.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:22.121 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:22.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:22.222 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:22.252 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:22.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:22.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:22.354 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:22.385 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:22.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:22.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:22.487 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:22.517 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:22.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:22.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:22.619 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:22.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:22.660 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:22.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:22.761 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:22.794 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:22.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:22.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:22.896 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:22.922 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:22.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:22.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:23.024 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:23.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:23.054 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:23.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:23.155 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:23.189 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:23.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:23.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:23.291 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:23.324 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:23.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:23.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:23.426 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:23.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:23 smithi151 ceph-mon[104450]: pgmap v1994: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 902 op/s; 12684/57096 objects degraded (22.215%); 5.8 MiB/s, 145 objects/s recovering 2024-09-13T14:57:23.450 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:23 smithi151 ceph-mon[108966]: pgmap v1994: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 902 op/s; 12684/57096 objects degraded (22.215%); 5.8 MiB/s, 145 objects/s recovering 2024-09-13T14:57:23.452 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T14:57:23.457 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:23.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:23.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:23.559 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:23.595 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:23.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:23.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:23.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:23 smithi184 ceph-mon[92527]: pgmap v1994: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 902 op/s; 12684/57096 objects degraded (22.215%); 5.8 MiB/s, 145 objects/s recovering 2024-09-13T14:57:23.697 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:23.725 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:23.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:23.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:23.826 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:23.866 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:23.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:23.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:23.968 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:24.000 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:24.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:24.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:24.102 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:24.137 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:24.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:24.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:24.239 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:24.277 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:24.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:24.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:24.379 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:24.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:24.415 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:24.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:24.516 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:24.551 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:24.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:24.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:24.653 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:24.682 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:24.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:24.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:24.729 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:57:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:57:24] "GET /metrics HTTP/1.1" 200 36682 "" "Prometheus/2.51.0" 2024-09-13T14:57:24.784 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:24.818 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:24.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:24.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:24.920 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:24.963 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:24.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:24.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:25.065 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:25.094 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:25.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:25.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:25.196 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:25.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:25.247 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:25.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:25.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:25.377 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:25 smithi151 ceph-mon[104450]: pgmap v1995: 119 pgs: 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 1.1 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 857 op/s; 11902/53124 objects degraded (22.404%); 465/53124 objects misplaced (0.875%); 4.5 MiB/s, 99 objects/s recovering 2024-09-13T14:57:25.378 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:25 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 11902/53124 objects degraded (22.404%), 9 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:25.378 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:25 smithi151 ceph-mon[108966]: pgmap v1995: 119 pgs: 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 1.1 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 857 op/s; 11902/53124 objects degraded (22.404%); 465/53124 objects misplaced (0.875%); 4.5 MiB/s, 99 objects/s recovering 2024-09-13T14:57:25.378 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:25 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 11902/53124 objects degraded (22.404%), 9 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:25.388 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:25.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:25.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:25.479 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:57:25.479 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:57:25.490 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:25.519 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:57:25.519 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:57:25.533 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:25.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:25.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:25.551 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:57:25.551 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:57:25.587 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:57:25.587 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:57:25.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:25 smithi184 ceph-mon[92527]: pgmap v1995: 119 pgs: 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 1.1 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 857 op/s; 11902/53124 objects degraded (22.404%); 465/53124 objects misplaced (0.875%); 4.5 MiB/s, 99 objects/s recovering 2024-09-13T14:57:25.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:25 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 11902/53124 objects degraded (22.404%), 9 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:25.620 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:57:25.620 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:57:25.636 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:25.652 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:57:25.652 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:57:25.665 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:25.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:25.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:25.688 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:57:25.688 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:57:25.729 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:57:25.730 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:57:25.767 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:25.805 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:25.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:25.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:25.907 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:25.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:25.943 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:25.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:25.988 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:57:26.044 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:26.069 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:26.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:26.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:26.171 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:26.228 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:26.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:26.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:26.329 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:26.363 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:26.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:26.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:26.465 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:26.501 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:26.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:26.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:26.603 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:26.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:26 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2671183984' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:26.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:26 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2671183984' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:26.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:26 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2671183984' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:26.631 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:26.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:26.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:26.733 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:26.769 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:26.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:26.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:26.871 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:26.906 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:26.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:26.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:27.007 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:27.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:27.048 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:27.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:27.149 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:27.171 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:27.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:27.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:27.273 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:27.305 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:27.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:27.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:27.407 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:27.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:27.445 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:27.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:27.546 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:27.571 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:27.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:27.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:27.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:27 smithi184 ceph-mon[92527]: pgmap v1996: 119 pgs: 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 985 MiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 821 op/s; 11595/47355 objects degraded (24.485%); 465/47355 objects misplaced (0.982%); 4.5 MiB/s, 134 objects/s recovering 2024-09-13T14:57:27.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:27 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3562085010' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:57:27.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:27 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:57:27.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:27 smithi151 ceph-mon[104450]: pgmap v1996: 119 pgs: 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 985 MiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 821 op/s; 11595/47355 objects degraded (24.485%); 465/47355 objects misplaced (0.982%); 4.5 MiB/s, 134 objects/s recovering 2024-09-13T14:57:27.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:27 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3562085010' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:57:27.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:27 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:57:27.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:27 smithi151 ceph-mon[108966]: pgmap v1996: 119 pgs: 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 985 MiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 821 op/s; 11595/47355 objects degraded (24.485%); 465/47355 objects misplaced (0.982%); 4.5 MiB/s, 134 objects/s recovering 2024-09-13T14:57:27.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:27 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3562085010' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:57:27.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:27 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:57:27.673 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:27.695 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:27.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:27.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:27.797 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:27.819 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:27.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:27.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:27.921 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:27.953 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:27.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:27.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:28.054 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:28.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:28.098 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:28.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:28.199 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:28.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:28.222 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:28.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:28.324 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:28.363 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:28.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:28.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:28.465 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:28.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:28.489 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:28.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:28.590 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:28.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:28 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:57:28.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:28 smithi184 ceph-mon[92527]: osdmap e1251: 8 total, 8 up, 8 in 2024-09-13T14:57:28.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:28 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:57:28.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:28 smithi151 ceph-mon[104450]: osdmap e1251: 8 total, 8 up, 8 in 2024-09-13T14:57:28.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:28 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:57:28.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:28 smithi151 ceph-mon[108966]: osdmap e1251: 8 total, 8 up, 8 in 2024-09-13T14:57:28.630 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:28.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:28.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:28.732 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:28.764 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:28.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:28.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:28.866 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:28.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:28.903 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:28.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:29.004 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:29.040 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:29.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:29.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:29.142 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:29.179 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:29.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:29.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:29.281 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:29.317 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:29.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:29.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:29.419 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:29.455 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:29.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:29.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:29.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:29.591 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:29.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:29.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:29.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:29 smithi184 ceph-mon[92527]: pgmap v1998: 119 pgs: 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 838 MiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 929 op/s; 11595/40290 objects degraded (28.779%); 465/40290 objects misplaced (1.154%); 4.5 MiB/s, 110 objects/s recovering 2024-09-13T14:57:29.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:29 smithi151 ceph-mon[104450]: pgmap v1998: 119 pgs: 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 838 MiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 929 op/s; 11595/40290 objects degraded (28.779%); 465/40290 objects misplaced (1.154%); 4.5 MiB/s, 110 objects/s recovering 2024-09-13T14:57:29.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:29 smithi151 ceph-mon[108966]: pgmap v1998: 119 pgs: 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 838 MiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 929 op/s; 11595/40290 objects degraded (28.779%); 465/40290 objects misplaced (1.154%); 4.5 MiB/s, 110 objects/s recovering 2024-09-13T14:57:29.693 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:29.736 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:29.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:29.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:29.838 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:29.876 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:29.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:29.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:29.978 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:30.017 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:30.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:30.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:30.118 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:30.157 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:30.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:30.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:30.258 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:30.290 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:30.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:30.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:30.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:30.423 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:30.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:30.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:30.525 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:30.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:30.559 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:30.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:30.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:30 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 11595/40290 objects degraded (28.779%), 9 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:30.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:57:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:57:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:57:30.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:30 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 11595/40290 objects degraded (28.779%), 9 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:30.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:30 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 11595/40290 objects degraded (28.779%), 9 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:30.660 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:30.689 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:30.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:30.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:30.790 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:30.825 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:30.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:30.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:30.868 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T14:57:30.927 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:30.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:30.960 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:30.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:31.061 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:31.068 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:31.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:31.116 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:31.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:31.217 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:31.253 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:31.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:31.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:31.354 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:31.377 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:31.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:31.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:31.479 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:31.501 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:31.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:31.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:31.603 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:31.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:31 smithi184 ceph-mon[92527]: pgmap v1999: 119 pgs: 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 749 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 928 op/s; 11345/35982 objects degraded (31.530%); 131/35982 objects misplaced (0.364%); 6.0 MiB/s, 153 objects/s recovering 2024-09-13T14:57:31.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:31 smithi151 ceph-mon[104450]: pgmap v1999: 119 pgs: 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 749 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 928 op/s; 11345/35982 objects degraded (31.530%); 131/35982 objects misplaced (0.364%); 6.0 MiB/s, 153 objects/s recovering 2024-09-13T14:57:31.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:31 smithi151 ceph-mon[108966]: pgmap v1999: 119 pgs: 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 749 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 928 op/s; 11345/35982 objects degraded (31.530%); 131/35982 objects misplaced (0.364%); 6.0 MiB/s, 153 objects/s recovering 2024-09-13T14:57:31.625 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:31.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:31.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:31.691 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T14:57:31.727 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:31.749 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:31.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:31.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:31.850 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:31.874 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:31.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:31.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:31.976 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:31.999 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:31.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:32.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:32.100 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:32.126 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:32.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:32.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:32.228 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:32.253 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:32.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:32.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:32.354 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:32.377 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:32.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:32.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:32.479 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:32.517 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:32.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:32.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:32.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:32 smithi184 ceph-mon[92527]: osdmap e1252: 8 total, 8 up, 8 in 2024-09-13T14:57:32.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:32.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:32 smithi151 ceph-mon[104450]: osdmap e1252: 8 total, 8 up, 8 in 2024-09-13T14:57:32.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:32.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:32 smithi151 ceph-mon[108966]: osdmap e1252: 8 total, 8 up, 8 in 2024-09-13T14:57:32.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:32.619 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:32.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:32.661 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:32.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:32.762 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:32.791 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:57:32.795 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:32.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:32.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:32.897 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:32.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:32.935 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:32.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:33.036 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:33.076 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:33.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:33.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:33.177 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:33.198 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:33.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:33.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:33.300 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:33.336 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:33.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:33.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:33.438 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:33.465 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:33.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:33.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:33.567 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:33.600 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:33.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:33.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:33.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:33 smithi184 ceph-mon[92527]: pgmap v2001: 119 pgs: 3 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 624 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 963 op/s; 11244/29982 objects degraded (37.503%); 184/29982 objects misplaced (0.614%); 3.0 MiB/s, 117 objects/s recovering 2024-09-13T14:57:33.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:33 smithi184 ceph-mon[92527]: osdmap e1253: 8 total, 8 up, 8 in 2024-09-13T14:57:33.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:33 smithi151 ceph-mon[104450]: pgmap v2001: 119 pgs: 3 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 624 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 963 op/s; 11244/29982 objects degraded (37.503%); 184/29982 objects misplaced (0.614%); 3.0 MiB/s, 117 objects/s recovering 2024-09-13T14:57:33.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:33 smithi151 ceph-mon[104450]: osdmap e1253: 8 total, 8 up, 8 in 2024-09-13T14:57:33.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:33 smithi151 ceph-mon[108966]: pgmap v2001: 119 pgs: 3 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 624 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 963 op/s; 11244/29982 objects degraded (37.503%); 184/29982 objects misplaced (0.614%); 3.0 MiB/s, 117 objects/s recovering 2024-09-13T14:57:33.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:33 smithi151 ceph-mon[108966]: osdmap e1253: 8 total, 8 up, 8 in 2024-09-13T14:57:33.702 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:33.736 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:33.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:33.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:33.838 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:33.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:33.880 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:33.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:33.981 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:34.021 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:34.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:34.123 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:34.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.162 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:34.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:34.263 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:34.296 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:34.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:34.398 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:34.435 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:34.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:34.537 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:34.568 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:34.572 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:34.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:34.674 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:34.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.705 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:34.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:34.806 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:34.842 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:34.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:34.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:57:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:57:34] "GET /metrics HTTP/1.1" 200 36669 "" "Prometheus/2.51.0" 2024-09-13T14:57:34.944 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:34.981 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:34.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:35.083 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:35.118 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:35.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:35.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:35.221 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:35.243 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:35.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:35.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:35.345 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:35.367 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:35.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:35.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:35.469 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:35.492 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:35.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:35.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:35.594 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:35.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:35 smithi184 ceph-mon[92527]: pgmap v2003: 119 pgs: 1 active+degraded, 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 549 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 880 op/s; 11234/26406 objects degraded (42.543%); 53/26406 objects misplaced (0.201%); 4.9 MiB/s, 118 objects/s recovering 2024-09-13T14:57:35.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:35 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 11234/26406 objects degraded (42.543%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:35.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:35 smithi184 ceph-mon[92527]: osdmap e1254: 8 total, 8 up, 8 in 2024-09-13T14:57:35.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:35 smithi151 ceph-mon[104450]: pgmap v2003: 119 pgs: 1 active+degraded, 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 549 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 880 op/s; 11234/26406 objects degraded (42.543%); 53/26406 objects misplaced (0.201%); 4.9 MiB/s, 118 objects/s recovering 2024-09-13T14:57:35.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:35 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 11234/26406 objects degraded (42.543%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:35.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:35 smithi151 ceph-mon[104450]: osdmap e1254: 8 total, 8 up, 8 in 2024-09-13T14:57:35.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:35 smithi151 ceph-mon[108966]: pgmap v2003: 119 pgs: 1 active+degraded, 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 549 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 880 op/s; 11234/26406 objects degraded (42.543%); 53/26406 objects misplaced (0.201%); 4.9 MiB/s, 118 objects/s recovering 2024-09-13T14:57:35.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:35 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 11234/26406 objects degraded (42.543%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:35.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:35 smithi151 ceph-mon[108966]: osdmap e1254: 8 total, 8 up, 8 in 2024-09-13T14:57:35.615 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:35.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:35.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:35.717 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:35.739 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:35.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:35.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:35.841 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:35.862 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:35.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:35.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:35.964 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:35.986 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:35.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:35.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:36.088 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:36.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:36.121 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:36.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:36.222 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:36.257 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:36.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:36.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:36.359 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:36.394 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:36.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:36.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:36.495 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:36.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:36.533 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:36.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:36.634 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:36.669 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:36.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:36.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:36.771 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:36.801 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:36.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:36.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:36.903 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:36.936 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:36.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:36.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:37.038 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:37.076 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:37.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:37.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:37.178 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:37.216 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:37.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:37.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:37.249 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:36 smithi151 ceph-mon[104450]: pgmap v2005: 119 pgs: 1 active+degraded, 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 448 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 800 op/s; 10752/21567 objects degraded (49.854%); 53/21567 objects misplaced (0.246%); 1.9 MiB/s, 111 objects/s recovering 2024-09-13T14:57:37.249 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:36 smithi151 ceph-mon[104450]: osdmap e1255: 8 total, 8 up, 8 in 2024-09-13T14:57:37.249 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:36 smithi151 ceph-mon[108966]: pgmap v2005: 119 pgs: 1 active+degraded, 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 448 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 800 op/s; 10752/21567 objects degraded (49.854%); 53/21567 objects misplaced (0.246%); 1.9 MiB/s, 111 objects/s recovering 2024-09-13T14:57:37.249 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:36 smithi151 ceph-mon[108966]: osdmap e1255: 8 total, 8 up, 8 in 2024-09-13T14:57:37.292 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:37.318 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:37.356 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:36 smithi184 ceph-mon[92527]: pgmap v2005: 119 pgs: 1 active+degraded, 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 448 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 800 op/s; 10752/21567 objects degraded (49.854%); 53/21567 objects misplaced (0.246%); 1.9 MiB/s, 111 objects/s recovering 2024-09-13T14:57:37.356 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:36 smithi184 ceph-mon[92527]: osdmap e1255: 8 total, 8 up, 8 in 2024-09-13T14:57:37.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:37.366 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:37.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:37.467 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:37.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:37.507 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:37.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:37.607 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:37.640 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:37.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:37.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:37.742 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:37.777 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:37.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:37.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:37.879 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:37.918 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:37.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:37.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:38.019 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:38.057 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:38.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:38.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:38.160 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:38.200 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:38.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:38.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:38.302 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:38.336 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:38.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:38.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:38.438 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:38.468 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:38.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:38.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:38.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:38.605 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:38.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:38.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:38.707 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:38.744 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:38.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:38.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:38.846 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:38.881 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:38.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:38.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:38.983 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:39.023 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:39.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:39.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:39.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:38 smithi151 ceph-mon[104450]: pgmap v2007: 119 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 319 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 812 op/s; 10113/15360 objects degraded (65.840%); 1.9 MiB/s, 219 objects/s recovering 2024-09-13T14:57:39.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:38 smithi151 ceph-mon[108966]: pgmap v2007: 119 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 319 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 812 op/s; 10113/15360 objects degraded (65.840%); 1.9 MiB/s, 219 objects/s recovering 2024-09-13T14:57:39.124 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:39.159 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:39.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:39.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:39.261 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:39.303 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:39.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:39.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:39.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:38 smithi184 ceph-mon[92527]: pgmap v2007: 119 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 319 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 812 op/s; 10113/15360 objects degraded (65.840%); 1.9 MiB/s, 219 objects/s recovering 2024-09-13T14:57:39.405 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:39.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:39.439 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:39.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:39.540 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:39.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:39.582 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:39.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:39.683 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:39.724 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:39.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:39.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:39.779 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:57:39.779 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-13T14:57:39.780 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:57:39.780 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.46034.0:203876 9.6 e1255/1252 9:60b816cf:::benchmark_data_smithi151_261954_object101020:head v 1255'12860, mlcod=1255'12857)", 2024-09-13T14:57:39.780 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:57:39.260213+0000", 2024-09-13T14:57:39.780 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.005994638, 2024-09-13T14:57:39.780 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.006186464, 2024-09-13T14:57:39.780 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:57:39.780 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:57:39.780 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-13T14:57:39.780 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:57:39.780 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:57:39.780 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:57:39.780 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:39.260213+0000", 2024-09-13T14:57:39.781 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:57:39.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:57:39.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:57:39.781 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:57:39.781 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:39.260213+0000", 2024-09-13T14:57:39.781 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:57:39.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:57:39.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:57:39.781 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:57:39.781 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:39.260215+0000", 2024-09-13T14:57:39.781 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.593e-06 2024-09-13T14:57:39.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:57:39.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:57:39.781 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:57:39.782 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:39.260223+0000", 2024-09-13T14:57:39.782 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5059999999999995e-06 2024-09-13T14:57:39.782 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:57:39.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:57:39.782 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:57:39.782 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:39.260240+0000", 2024-09-13T14:57:39.782 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7201999999999999e-05 2024-09-13T14:57:39.782 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:57:39.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:57:39.782 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:57:39.782 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:39.260249+0000", 2024-09-13T14:57:39.782 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9120000000000004e-06 2024-09-13T14:57:39.782 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:57:39.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:57:39.782 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:57:39.783 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:39.260440+0000", 2024-09-13T14:57:39.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000190392 2024-09-13T14:57:39.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:57:39.783 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:57:39.783 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:57:39.783 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:39.260489+0000", 2024-09-13T14:57:39.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9555000000000002e-05 2024-09-13T14:57:39.783 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:57:39.783 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:57:39.783 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:57:39.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:57:39.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:57:39.784 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.46034.0:203885 9.2 e1255/1222)", 2024-09-13T14:57:39.784 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:57:39.263942+0000", 2024-09-13T14:57:39.784 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0022656579999999998, 2024-09-13T14:57:39.784 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0025090070000000002, 2024-09-13T14:57:39.784 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:57:39.784 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:57:39.784 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "queued for pg", 2024-09-13T14:57:39.784 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:57:39.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:57:39.784 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:57:39.784 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:39.263942+0000", 2024-09-13T14:57:39.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:57:39.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:57:39.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:57:39.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:57:39.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:39.263942+0000", 2024-09-13T14:57:39.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:57:39.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:57:39.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:57:39.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:57:39.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:39.263943+0000", 2024-09-13T14:57:39.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2020000000000001e-06 2024-09-13T14:57:39.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:57:39.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:57:39.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:57:39.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:39.263951+0000", 2024-09-13T14:57:39.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7719999999999994e-06 2024-09-13T14:57:39.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:57:39.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:57:39.786 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:57:39.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:39.263970+0000", 2024-09-13T14:57:39.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8715e-05 2024-09-13T14:57:39.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:57:39.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:57:39.786 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:57:39.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:39.263979+0000", 2024-09-13T14:57:39.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2820000000000002e-06 2024-09-13T14:57:39.786 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:57:39.786 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:57:39.787 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:57:39.787 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:57:39.787 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:57:39.787 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.46034.0:203878 9.6 e1255/1252)", 2024-09-13T14:57:39.787 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:57:39.265948+0000", 2024-09-13T14:57:39.787 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.00025984700000000001, 2024-09-13T14:57:39.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00062924599999999997, 2024-09-13T14:57:39.787 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:57:39.787 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:57:39.787 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "queued for pg", 2024-09-13T14:57:39.787 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:57:39.787 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:57:39.787 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:57:39.787 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:39.265948+0000", 2024-09-13T14:57:39.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:57:39.788 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:57:39.788 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:57:39.788 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:57:39.788 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:39.265948+0000", 2024-09-13T14:57:39.788 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:57:39.788 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:57:39.788 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:57:39.788 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:57:39.788 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:39.265949+0000", 2024-09-13T14:57:39.788 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3349999999999999e-06 2024-09-13T14:57:39.788 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:57:39.788 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:57:39.788 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:57:39.788 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:39.265959+0000", 2024-09-13T14:57:39.789 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4930000000000004e-06 2024-09-13T14:57:39.789 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:57:39.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:57:39.789 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:57:39.789 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:39.265977+0000", 2024-09-13T14:57:39.789 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8800999999999998e-05 2024-09-13T14:57:39.789 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:57:39.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:57:39.789 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:57:39.789 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:39.265987+0000", 2024-09-13T14:57:39.789 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9580000000000002e-06 2024-09-13T14:57:39.789 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:57:39.789 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:57:39.789 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:57:39.789 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:57:39.790 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-13T14:57:39.790 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 3 2024-09-13T14:57:39.790 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:57:39.826 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:39.860 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:39.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:39.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:39.961 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:39.994 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:39.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:39.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:40.096 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:40.120 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:40.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:40.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:40.222 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:40.243 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:40.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:40.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:40.345 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:40.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:40 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 10113/15360 objects degraded (65.840%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:40.354 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:57:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:57:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:57:40.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:40 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 10113/15360 objects degraded (65.840%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:40.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:40 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 10113/15360 objects degraded (65.840%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:40.378 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:40.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:40.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:40.481 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:40.523 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:40.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:40.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:40.625 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:40.663 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:40.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:40.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:40.765 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:40.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:40.795 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:40.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:40.896 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:40.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:40.929 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:40.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:41.030 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:41.071 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:41.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:41.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:41.173 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:41.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:41.205 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:41.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:41.307 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:41.328 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:41.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:41.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:41.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:41 smithi184 ceph-mon[92527]: pgmap v2008: 119 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 252 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 794 op/s; 9710/12129 objects degraded (80.056%); 1.5 MiB/s, 229 objects/s recovering 2024-09-13T14:57:41.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:41 smithi151 ceph-mon[104450]: pgmap v2008: 119 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 252 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 794 op/s; 9710/12129 objects degraded (80.056%); 1.5 MiB/s, 229 objects/s recovering 2024-09-13T14:57:41.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:41 smithi151 ceph-mon[108966]: pgmap v2008: 119 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 252 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 794 op/s; 9710/12129 objects degraded (80.056%); 1.5 MiB/s, 229 objects/s recovering 2024-09-13T14:57:41.430 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:41.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:41.453 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:41.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:41.554 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:41.576 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:41.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:41.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:41.678 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:41.700 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:41.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:41.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:41.802 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:41.823 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:41.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:41.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:41.925 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:41.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:41.946 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:41.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:42.047 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:42.070 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:42.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:42.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:42.171 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:42.193 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:42.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:42.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:42.204 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Removed 102853 objects 2024-09-13T14:57:42.205 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Clean up completed and total clean up time :112.219 2024-09-13T14:57:42.294 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:42.317 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:42.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:42.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:42.329 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:42 smithi151 ceph-mon[104450]: osdmap e1256: 8 total, 8 up, 8 in 2024-09-13T14:57:42.329 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:42 smithi151 ceph-mon[108966]: osdmap e1256: 8 total, 8 up, 8 in 2024-09-13T14:57:42.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:42 smithi184 ceph-mon[92527]: osdmap e1256: 8 total, 8 up, 8 in 2024-09-13T14:57:42.419 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:42.441 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:42.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:42.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:42.543 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:42.565 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:42.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:42.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:42.667 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:42.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:42.689 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:42.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:42.790 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:42.811 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:42.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:42.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:42.913 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:42.934 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:42.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:42.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:43.036 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:43.057 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:43.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:43.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:43.159 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:43.180 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:43.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:43.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:43.283 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:43.304 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:43.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:43.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:43.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:43 smithi184 ceph-mon[92527]: pgmap v2010: 119 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 112 active+clean; 120 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 877 op/s; 9095/5832 objects degraded (155.950%); 503 KiB/s, 269 objects/s recovering 2024-09-13T14:57:43.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:43 smithi184 ceph-mon[92527]: osdmap e1257: 8 total, 8 up, 8 in 2024-09-13T14:57:43.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:43 smithi151 ceph-mon[104450]: pgmap v2010: 119 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 112 active+clean; 120 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 877 op/s; 9095/5832 objects degraded (155.950%); 503 KiB/s, 269 objects/s recovering 2024-09-13T14:57:43.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:43 smithi151 ceph-mon[104450]: osdmap e1257: 8 total, 8 up, 8 in 2024-09-13T14:57:43.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:43 smithi151 ceph-mon[108966]: pgmap v2010: 119 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 112 active+clean; 120 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 877 op/s; 9095/5832 objects degraded (155.950%); 503 KiB/s, 269 objects/s recovering 2024-09-13T14:57:43.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:43 smithi151 ceph-mon[108966]: osdmap e1257: 8 total, 8 up, 8 in 2024-09-13T14:57:43.405 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:43.426 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:43.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:43.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:43.528 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:43.550 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:43.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:43.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:43.652 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:43.672 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:43.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:43.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:43.774 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:43.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:43.796 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:43.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:43.897 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:43.918 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:43.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:43.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:44.020 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:44.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:44.044 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:44.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:44.145 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:44.166 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:44.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:44.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:44.268 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:44.289 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:44.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:44.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:44.391 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:44.413 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:44.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:44.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:44.517 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:44.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:44.539 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:44.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:44.640 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:44.661 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:44.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:44.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:44.763 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:44.784 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:44.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:44.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:44.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:57:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:57:44] "GET /metrics HTTP/1.1" 200 36669 "" "Prometheus/2.51.0" 2024-09-13T14:57:44.886 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:44.907 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:44.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:44.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:45.010 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:45.032 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:45.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:45.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:45.102 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T14:57:45.133 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:45.171 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:45.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:45.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:45.273 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:45.296 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:45.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:45.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:45.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:45 smithi184 ceph-mon[92527]: pgmap v2012: 119 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 112 active+clean; 45 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 731 op/s; 9095/2211 objects degraded (411.352%); 0 B/s, 131 objects/s recovering 2024-09-13T14:57:45.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:45 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 9095/2211 objects degraded (411.352%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:45.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:45 smithi151 ceph-mon[104450]: pgmap v2012: 119 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 112 active+clean; 45 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 731 op/s; 9095/2211 objects degraded (411.352%); 0 B/s, 131 objects/s recovering 2024-09-13T14:57:45.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:45 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 9095/2211 objects degraded (411.352%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:45.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:45 smithi151 ceph-mon[108966]: pgmap v2012: 119 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 112 active+clean; 45 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 731 op/s; 9095/2211 objects degraded (411.352%); 0 B/s, 131 objects/s recovering 2024-09-13T14:57:45.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:45 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 9095/2211 objects degraded (411.352%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:45.398 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:45.419 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:45.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:45.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:45.521 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:45.542 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:45.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:45.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:45.644 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:45.665 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:45.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:45.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:45.767 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:45.789 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:45.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:45.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:45.891 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:45.913 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:45.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:45.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:46.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:46.037 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:46.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:46.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:46.139 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:46.160 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:46.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:46.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:46.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:46.283 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:46.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:46.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:46.360 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:46.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:46.410 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:46.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:46.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:46.512 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:46.533 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:46.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:46.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:46.635 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:46.657 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:46.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:46.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:46.682 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:46.759 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:46.782 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:46.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:46.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:46.883 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:46.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:46.905 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:46.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:47.006 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:47.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:47.029 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:47.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:47.041 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T14:57:47.130 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:47.152 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:47.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:47.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:47.216 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-09-13T14:57:47.217 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-09-13T14:57:47.254 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:47.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:47.283 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:47.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:47.337 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:47 smithi151 ceph-mon[108966]: pgmap v2013: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 113 active+clean; 4.2 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 629 op/s; 8089/252 objects degraded (3209.921%); 0 B/s, 252 objects/s recovering 2024-09-13T14:57:47.337 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:47 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:47.337 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:47 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3159441784' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:57:47.337 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:47 smithi151 ceph-mon[104450]: pgmap v2013: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 113 active+clean; 4.2 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 629 op/s; 8089/252 objects degraded (3209.921%); 0 B/s, 252 objects/s recovering 2024-09-13T14:57:47.337 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:47 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:47.337 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:47 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3159441784' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:57:47.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:47 smithi184 ceph-mon[92527]: pgmap v2013: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 113 active+clean; 4.2 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 629 op/s; 8089/252 objects degraded (3209.921%); 0 B/s, 252 objects/s recovering 2024-09-13T14:57:47.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:47 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:47.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:47 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3159441784' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:57:47.384 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:47.405 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:47.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:47.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:47.507 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:47.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:47.529 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:47.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:47.630 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:47.652 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:47.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:47.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:47.754 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:47.775 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:47.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:47.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:47.877 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:47.897 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:47.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:47.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:47.999 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:48.021 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:48.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:48.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:48.123 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:48.146 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:48.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:48.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:48.248 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:48.270 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:48.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:48.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:48.297 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:48 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3159441784' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T14:57:48.297 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:48 smithi151 ceph-mon[104450]: osdmap e1258: 8 total, 8 up, 8 in 2024-09-13T14:57:48.297 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:48 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3159441784' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T14:57:48.298 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:48 smithi151 ceph-mon[108966]: osdmap e1258: 8 total, 8 up, 8 in 2024-09-13T14:57:48.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:48 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3159441784' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T14:57:48.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:48 smithi184 ceph-mon[92527]: osdmap e1258: 8 total, 8 up, 8 in 2024-09-13T14:57:48.372 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:48.395 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:48.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:48.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:48.497 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:48.521 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:48.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:48.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:48.623 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:48.645 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:48.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:48.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:48.747 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:48.769 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:48.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:48.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:48.871 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:48.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:48.895 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:48.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:48.996 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:49.018 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:49.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:49.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:49.120 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:49.143 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:49.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:49.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:49.245 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:49.268 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:49.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:49.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:49.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:49 smithi184 ceph-mon[92527]: pgmap v2015: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 113 active+clean; 595 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 285 op/s; 8089/78 objects degraded (10370.513%); 0 B/s, 149 objects/s recovering 2024-09-13T14:57:49.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:49 smithi184 ceph-mon[92527]: osdmap e1259: 8 total, 8 up, 8 in 2024-09-13T14:57:49.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:49 smithi151 ceph-mon[104450]: pgmap v2015: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 113 active+clean; 595 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 285 op/s; 8089/78 objects degraded (10370.513%); 0 B/s, 149 objects/s recovering 2024-09-13T14:57:49.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:49 smithi151 ceph-mon[104450]: osdmap e1259: 8 total, 8 up, 8 in 2024-09-13T14:57:49.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:49 smithi151 ceph-mon[108966]: pgmap v2015: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 113 active+clean; 595 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 285 op/s; 8089/78 objects degraded (10370.513%); 0 B/s, 149 objects/s recovering 2024-09-13T14:57:49.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:49 smithi151 ceph-mon[108966]: osdmap e1259: 8 total, 8 up, 8 in 2024-09-13T14:57:49.370 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:49.393 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:49.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:49.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:49.495 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:49.517 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:49.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:49.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:49.619 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:49.644 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:49.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:49.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:49.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:49.768 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:49.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:49.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:49.870 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:49.892 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:49.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:49.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:49.994 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:50.017 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:50.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:50.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:50.119 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:50.142 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:50.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:50.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:50.244 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:50.266 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:50.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:50.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:50.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:50 smithi184 ceph-mon[92527]: osdmap e1260: 8 total, 8 up, 8 in 2024-09-13T14:57:50.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:50 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 8089/78 objects degraded (10370.513%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:50.354 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:57:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:57:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:57:50.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:50 smithi151 ceph-mon[104450]: osdmap e1260: 8 total, 8 up, 8 in 2024-09-13T14:57:50.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:50 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 8089/78 objects degraded (10370.513%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:50.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:50 smithi151 ceph-mon[108966]: osdmap e1260: 8 total, 8 up, 8 in 2024-09-13T14:57:50.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:50 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 8089/78 objects degraded (10370.513%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:57:50.368 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:50.391 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:50.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:50.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:50.493 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:50.515 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:50.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:50.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:50.617 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:50.638 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:50.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:50.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:50.740 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:50.762 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:50.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:50.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:50.864 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:50.886 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:50.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:50.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:50.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:50.988 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:51.011 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:51.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:51.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:51.067 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-13T14:57:51.113 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:51.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:51.137 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:51.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:51.238 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:51.261 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:51.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:51.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:51.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:51 smithi184 ceph-mon[92527]: pgmap v2018: 119 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 114 active+clean; 595 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 118 op/s; 7237/78 objects degraded (9278.205%); 0 B/s, 309 objects/s recovering 2024-09-13T14:57:51.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:51 smithi184 ceph-mon[92527]: osdmap e1261: 8 total, 8 up, 8 in 2024-09-13T14:57:51.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:51 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/644450330' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:51.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:51 smithi151 ceph-mon[104450]: pgmap v2018: 119 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 114 active+clean; 595 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 118 op/s; 7237/78 objects degraded (9278.205%); 0 B/s, 309 objects/s recovering 2024-09-13T14:57:51.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:51 smithi151 ceph-mon[104450]: osdmap e1261: 8 total, 8 up, 8 in 2024-09-13T14:57:51.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:51 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/644450330' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:51.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:51 smithi151 ceph-mon[108966]: pgmap v2018: 119 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 114 active+clean; 595 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 118 op/s; 7237/78 objects degraded (9278.205%); 0 B/s, 309 objects/s recovering 2024-09-13T14:57:51.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:51 smithi151 ceph-mon[108966]: osdmap e1261: 8 total, 8 up, 8 in 2024-09-13T14:57:51.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:51 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/644450330' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:51.363 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:51.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:51.388 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:51.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:51.489 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:51.510 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:51.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:51.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:51.612 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:51.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:51.634 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:51.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:51.735 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:51.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:51.757 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:51.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:51.831 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:57:51.831 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-13T14:57:51.831 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-13T14:57:51.831 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-13T14:57:51.831 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:57:51.858 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:51.866 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T14:57:51.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:51.880 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:51.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:51.982 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:52.005 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:52.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:52.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:52.107 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:52.128 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:52.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:52.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:52.230 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:52.252 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:52.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:52.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:52.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:52 smithi184 ceph-mon[92527]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:57:52.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:52 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/644450330' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:57:52.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:52 smithi184 ceph-mon[92527]: osdmap e1262: 8 total, 8 up, 8 in 2024-09-13T14:57:52.354 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:52.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:52 smithi151 ceph-mon[108966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:57:52.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:52 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/644450330' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:57:52.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:52 smithi151 ceph-mon[108966]: osdmap e1262: 8 total, 8 up, 8 in 2024-09-13T14:57:52.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:52 smithi151 ceph-mon[104450]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:57:52.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:52 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/644450330' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:57:52.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:52 smithi151 ceph-mon[104450]: osdmap e1262: 8 total, 8 up, 8 in 2024-09-13T14:57:52.376 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:52.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:52.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:52.478 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:52.500 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:52.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:52.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:52.602 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:52.623 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:52.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:52.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:52.725 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:52.747 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:52.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:52.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:52.849 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:52.873 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:52.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:52.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:52.974 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:52.996 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:52.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:52.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:53.098 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:53.123 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:53.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:53.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:53.225 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:53.247 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:53.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:53.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:53.349 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:53.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:53 smithi184 ceph-mon[92527]: pgmap v2021: 119 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 114 active+clean; 595 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 7237/78 objects degraded (9278.205%); 0 B/s, 212 objects/s recovering 2024-09-13T14:57:53.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:53 smithi184 ceph-mon[92527]: osdmap e1263: 8 total, 8 up, 8 in 2024-09-13T14:57:53.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:53 smithi151 ceph-mon[108966]: pgmap v2021: 119 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 114 active+clean; 595 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 7237/78 objects degraded (9278.205%); 0 B/s, 212 objects/s recovering 2024-09-13T14:57:53.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:53 smithi151 ceph-mon[108966]: osdmap e1263: 8 total, 8 up, 8 in 2024-09-13T14:57:53.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:53 smithi151 ceph-mon[104450]: pgmap v2021: 119 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 114 active+clean; 595 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 7237/78 objects degraded (9278.205%); 0 B/s, 212 objects/s recovering 2024-09-13T14:57:53.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:53 smithi151 ceph-mon[104450]: osdmap e1263: 8 total, 8 up, 8 in 2024-09-13T14:57:53.370 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:53.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:53.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:53.472 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:53.495 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:53.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:53.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:53.596 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:53.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:53.619 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:53.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:53.720 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:53.742 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:53.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:53.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:53.843 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:53.865 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:53.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:53.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:53.967 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:53.985 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T14:57:53.991 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:53.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:53.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:54.093 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:54.115 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:54.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:54.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:54.217 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:54.241 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:54.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:54.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:54.343 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:54.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:54 smithi184 ceph-mon[92527]: osdmap e1264: 8 total, 8 up, 8 in 2024-09-13T14:57:54.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:54 smithi151 ceph-mon[104450]: osdmap e1264: 8 total, 8 up, 8 in 2024-09-13T14:57:54.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:54 smithi151 ceph-mon[108966]: osdmap e1264: 8 total, 8 up, 8 in 2024-09-13T14:57:54.365 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:54.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:54.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:54.467 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:54.489 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:54.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:54.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:54.591 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:54.612 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:54.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:54.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:54.697 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:57:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:57:54] "GET /metrics HTTP/1.1" 200 36650 "" "Prometheus/2.51.0" 2024-09-13T14:57:54.714 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:54.736 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:54.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:54.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:54.838 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:54.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:54.862 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:54.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:54.963 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:54.985 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:54.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:54.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:55.087 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:55.109 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:55.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:55.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:55.195 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:55.196 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:55.210 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:55.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:55.234 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:55.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:55.280 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:55 smithi151 ceph-mon[108966]: pgmap v2024: 119 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 117 active+clean; 595 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 3949/78 objects degraded (5062.821%); 0 B/s, 444 objects/s recovering 2024-09-13T14:57:55.281 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:55 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 3949/78 objects degraded (5062.821%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:57:55.281 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:55 smithi151 ceph-mon[104450]: pgmap v2024: 119 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 117 active+clean; 595 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 3949/78 objects degraded (5062.821%); 0 B/s, 444 objects/s recovering 2024-09-13T14:57:55.281 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:55 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 3949/78 objects degraded (5062.821%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:57:55.336 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:55.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:55 smithi184 ceph-mon[92527]: pgmap v2024: 119 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 117 active+clean; 595 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 3949/78 objects degraded (5062.821%); 0 B/s, 444 objects/s recovering 2024-09-13T14:57:55.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:55 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 3949/78 objects degraded (5062.821%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:57:55.358 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:55.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:55.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:55.381 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:57:55.381 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:57:55.397 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:57:55.397 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:57:55.425 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:57:55.425 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:57:55.441 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:57:55.441 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:57:55.460 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:57:55.460 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:57:55.460 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:55.475 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:57:55.476 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:57:55.481 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:55.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:55.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:55.491 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:57:55.491 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:57:55.507 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:57:55.507 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:57:55.583 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:55.600 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:57:55.606 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:55.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:55.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:55.708 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:55.729 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:55.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:55.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:55.831 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:55.853 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:55.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:55.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:55.955 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:55.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:55.977 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:55.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:56.078 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:56.101 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:56.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:56.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:56.106 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_4' removed 2024-09-13T14:57:56.204 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:56.225 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:56.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:56.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:56.327 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:56.348 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:56 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/866143609' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:56.348 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:56 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2823469986' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:57:56.348 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:56 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:57:56.348 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:56 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/866143609' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:56.348 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:56 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2823469986' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:57:56.349 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:56 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:57:56.351 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:56.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:56.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:56.453 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:56.474 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:56.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:56.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:56.576 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:56.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:56.598 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:56.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:56.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:56 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/866143609' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:56.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:56 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2823469986' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:57:56.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:56 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:57:56.698 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:56.727 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:56.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:56.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:56.829 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:56.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:56.851 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:56.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:56.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:56.974 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:56.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:56.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:57.072 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T14:57:57.073 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T14:57:57.073 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_5 2024-09-13T14:57:57.073 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_5 16 2024-09-13T14:57:57.076 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:57.137 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:57.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:57.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:57.239 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:57.261 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:57.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:57.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:57.311 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:57 smithi151 ceph-mon[104450]: pgmap v2025: 119 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 117 active+clean; 595 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 3949/78 objects degraded (5062.821%); 0 B/s, 313 objects/s recovering 2024-09-13T14:57:57.311 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:57 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:57:57.312 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:57 smithi151 ceph-mon[104450]: osdmap e1265: 8 total, 8 up, 8 in 2024-09-13T14:57:57.312 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:57 smithi151 ceph-mon[108966]: pgmap v2025: 119 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 117 active+clean; 595 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 3949/78 objects degraded (5062.821%); 0 B/s, 313 objects/s recovering 2024-09-13T14:57:57.312 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:57 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:57:57.312 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:57 smithi151 ceph-mon[108966]: osdmap e1265: 8 total, 8 up, 8 in 2024-09-13T14:57:57.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:57 smithi184 ceph-mon[92527]: pgmap v2025: 119 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 117 active+clean; 595 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 3949/78 objects degraded (5062.821%); 0 B/s, 313 objects/s recovering 2024-09-13T14:57:57.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:57 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:57:57.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:57 smithi184 ceph-mon[92527]: osdmap e1265: 8 total, 8 up, 8 in 2024-09-13T14:57:57.362 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:57.385 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:57.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:57.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:57.487 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:57.509 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:57.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:57.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:57.611 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:57.635 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:57.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:57.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:57.736 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:57.761 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:57.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:57.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:57.863 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:57.886 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:57.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:57.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:57.988 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:58.009 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:58.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:58.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:58.111 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:58.133 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:58.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:58.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:58.234 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:58.260 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:58.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:58.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:58.357 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:58 smithi151 ceph-mon[104450]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3949/78 objects degraded (5062.821%), 1 pg degraded, 1 pg undersized) 2024-09-13T14:57:58.357 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:58 smithi151 ceph-mon[104450]: Cluster is now healthy 2024-09-13T14:57:58.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:58 smithi151 ceph-mon[108966]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3949/78 objects degraded (5062.821%), 1 pg degraded, 1 pg undersized) 2024-09-13T14:57:58.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:58 smithi151 ceph-mon[108966]: Cluster is now healthy 2024-09-13T14:57:58.361 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:58.387 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:58.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:58.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:58.488 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:58.511 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:58.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:58.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:58.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:58 smithi184 ceph-mon[92527]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3949/78 objects degraded (5062.821%), 1 pg degraded, 1 pg undersized) 2024-09-13T14:57:58.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:58 smithi184 ceph-mon[92527]: Cluster is now healthy 2024-09-13T14:57:58.613 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:58.638 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:58.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:58.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:58.740 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:58.762 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:58.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:58.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:58.864 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:58.885 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:58.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:58.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:58.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:59.008 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:59.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:59.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:59.110 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:59.133 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:59.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:59.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:59.235 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:59.261 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:59.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:59.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:59.351 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:57:59.351 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:57:59.363 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:59.386 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:59.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:59.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:59.488 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:59.514 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:57:59.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:59.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:59.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:57:59 smithi184 ceph-mon[92527]: pgmap v2027: 103 pgs: 103 active+clean; 595 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:57:59.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:57:59 smithi151 ceph-mon[104450]: pgmap v2027: 103 pgs: 103 active+clean; 595 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:57:59.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:57:59 smithi151 ceph-mon[108966]: pgmap v2027: 103 pgs: 103 active+clean; 595 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:57:59.616 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:59.638 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:59.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:59.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:59.740 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:59.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:59.762 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:59.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:59.863 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:59.884 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:57:59.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:59.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:59.987 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:00.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:00.013 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:00.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:00.114 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:00.126 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T14:58:00.139 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:00.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:00.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:00.202 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:00.203 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:00.240 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:00.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:00.262 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:00.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:00.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:00 smithi151 ceph-mon[104450]: 1.e scrub starts 2024-09-13T14:58:00.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:00 smithi151 ceph-mon[104450]: 1.e scrub ok 2024-09-13T14:58:00.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:00 smithi151 ceph-mon[108966]: 1.e scrub starts 2024-09-13T14:58:00.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:00 smithi151 ceph-mon[108966]: 1.e scrub ok 2024-09-13T14:58:00.363 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:00.384 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:00.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:00.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:00.486 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:00.512 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:00.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:00.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:00.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:00 smithi184 ceph-mon[92527]: 1.e scrub starts 2024-09-13T14:58:00.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:00 smithi184 ceph-mon[92527]: 1.e scrub ok 2024-09-13T14:58:00.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:58:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:58:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:58:00.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:00.636 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:00.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:00.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:00.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:00.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:00.760 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:00.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:00.861 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:00.887 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:00.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:00.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:00.990 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:01.016 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:01.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:01.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:01.118 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:01.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:01.140 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:01.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:01.241 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:01.263 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:01.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:01.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:01.365 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:01.386 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:01.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:01.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:01.420 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:01 smithi151 ceph-mon[104450]: pgmap v2028: 103 pgs: 103 active+clean; 595 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:58:01.420 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:01 smithi151 ceph-mon[108966]: pgmap v2028: 103 pgs: 103 active+clean; 595 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:58:01.488 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:01.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:01.512 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:01.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:01.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:01 smithi184 ceph-mon[92527]: pgmap v2028: 103 pgs: 103 active+clean; 595 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:58:01.612 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:01.638 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:01.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:01.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:01.741 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:01.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:01.763 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:01.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:01.864 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:01.890 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:01.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:01.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:01.992 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:02.018 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:02.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:02.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:02.120 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:02.142 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:02.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:02.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:02.176 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T14:58:02.244 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:02.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:02.268 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:02.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:02.369 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:02.390 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:02.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:02.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:02.493 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:02.518 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:02.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:02.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:02.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:02 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3418349762' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:58:02.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:02.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:02.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:02.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:02 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3418349762' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:58:02.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:02.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:02.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:02.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:02 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3418349762' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:58:02.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:02.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:02.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:02.620 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:02.641 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:02.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:02.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:02.743 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:02.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:02.769 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:02.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:02.835 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:58:02.870 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:02.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:02.901 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:02.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:03.002 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:03.026 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:03.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:03.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:03.128 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:03.154 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:03.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:03.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:03.256 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:03.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:03.280 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:03.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:03.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:03 smithi151 ceph-mon[108966]: pgmap v2029: 103 pgs: 103 active+clean; 595 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:58:03.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:03 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3418349762' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T14:58:03.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:03 smithi151 ceph-mon[108966]: osdmap e1266: 8 total, 8 up, 8 in 2024-09-13T14:58:03.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:03 smithi151 ceph-mon[104450]: pgmap v2029: 103 pgs: 103 active+clean; 595 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:58:03.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:03 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3418349762' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T14:58:03.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:03 smithi151 ceph-mon[104450]: osdmap e1266: 8 total, 8 up, 8 in 2024-09-13T14:58:03.381 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:03.402 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:03.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:03.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:03.504 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:03.530 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:03.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:03.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:03.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:03 smithi184 ceph-mon[92527]: pgmap v2029: 103 pgs: 103 active+clean; 595 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:58:03.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:03 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3418349762' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T14:58:03.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:03 smithi184 ceph-mon[92527]: osdmap e1266: 8 total, 8 up, 8 in 2024-09-13T14:58:03.631 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:03.653 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:03.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:03.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:03.755 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:03.777 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:03.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:03.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:03.879 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:03.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:03.906 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:03.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:04.007 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:04.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:04.033 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:04.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:04.134 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:04.158 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:04.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:04.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:04.260 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:04.282 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:04.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:04.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:04.384 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:04.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:04.410 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:04.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:04.435 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:04 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]} : dispatch 2024-09-13T14:58:04.435 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:04 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]} : dispatch 2024-09-13T14:58:04.435 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:04 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-13T14:58:04.435 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:04 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]} : dispatch 2024-09-13T14:58:04.436 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:04 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]} : dispatch 2024-09-13T14:58:04.436 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:04 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-13T14:58:04.510 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:04.532 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:04.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:04.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:04.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:04 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]} : dispatch 2024-09-13T14:58:04.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:04 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]} : dispatch 2024-09-13T14:58:04.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:04 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-13T14:58:04.613 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:04.634 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:04.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-13T14:58:04.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:04.659 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:04.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:04.737 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:58:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:58:04] "GET /metrics HTTP/1.1" 200 34298 "" "Prometheus/2.51.0" 2024-09-13T14:58:04.760 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:04.784 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:04.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:04.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:04.886 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:04.898 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:04.911 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:04.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:04.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:05.013 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:05.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:05.043 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:05.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:05.144 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:05.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:05.170 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:05.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:05.186 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_5' created 2024-09-13T14:58:05.260 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:05 smithi151 ceph-mon[104450]: pgmap v2031: 103 pgs: 103 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:58:05.260 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:05 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-09-13T14:58:05.260 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:05 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]': finished 2024-09-13T14:58:05.260 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:05 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-13T14:58:05.260 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:05 smithi151 ceph-mon[104450]: osdmap e1267: 8 total, 8 up, 8 in 2024-09-13T14:58:05.260 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:05 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3617100776' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-13T14:58:05.260 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:05 smithi151 ceph-mon[108966]: pgmap v2031: 103 pgs: 103 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:58:05.260 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:05 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-09-13T14:58:05.261 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:05 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]': finished 2024-09-13T14:58:05.261 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:05 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-13T14:58:05.261 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:05 smithi151 ceph-mon[108966]: osdmap e1267: 8 total, 8 up, 8 in 2024-09-13T14:58:05.261 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:05 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3617100776' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-13T14:58:05.271 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:05.293 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:05.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:05.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:05.395 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:05.420 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:05.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:05.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:05.526 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:05.548 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:05.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:05.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:05.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:05 smithi184 ceph-mon[92527]: pgmap v2031: 103 pgs: 103 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:58:05.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:05 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-09-13T14:58:05.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:05 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]': finished 2024-09-13T14:58:05.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:05 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-13T14:58:05.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:05 smithi184 ceph-mon[92527]: osdmap e1267: 8 total, 8 up, 8 in 2024-09-13T14:58:05.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:05 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3617100776' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-13T14:58:05.650 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:05.676 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:05.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:05.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:05.778 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:05.803 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:05.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:05.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:05.905 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:05.929 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:05.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:05.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:06.031 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:06.052 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:06.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:06.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:06.154 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:06.175 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:06.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:06.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:06.277 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:06.299 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:06.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:06.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:06.319 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:06 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3617100776' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-13T14:58:06.319 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:06 smithi151 ceph-mon[104450]: osdmap e1268: 8 total, 8 up, 8 in 2024-09-13T14:58:06.320 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:06 smithi151 ceph-mon[104450]: osdmap e1269: 8 total, 8 up, 8 in 2024-09-13T14:58:06.320 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:06 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3617100776' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-13T14:58:06.320 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:06 smithi151 ceph-mon[108966]: osdmap e1268: 8 total, 8 up, 8 in 2024-09-13T14:58:06.320 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:06 smithi151 ceph-mon[108966]: osdmap e1269: 8 total, 8 up, 8 in 2024-09-13T14:58:06.402 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:06.423 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:06.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:06.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:06.525 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:06.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:06.549 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:06.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:06.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:06 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3617100776' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-13T14:58:06.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:06 smithi184 ceph-mon[92527]: osdmap e1268: 8 total, 8 up, 8 in 2024-09-13T14:58:06.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:06 smithi184 ceph-mon[92527]: osdmap e1269: 8 total, 8 up, 8 in 2024-09-13T14:58:06.651 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:06.677 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:06.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:06.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:06.779 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:06.805 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:06.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:06.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:06.907 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:06.929 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:06.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:06.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:07.030 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:07.052 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:07.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:07.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:07.154 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:07.180 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:07.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:07.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:07.206 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-13T14:58:07.282 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:07.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:07.308 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:07.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:07.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:07 smithi151 ceph-mon[104450]: pgmap v2034: 119 pgs: 2 creating+peering, 14 unknown, 103 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:58:07.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:07 smithi151 ceph-mon[104450]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:58:07.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:07 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3701209012' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:58:07.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:07 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:58:07.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:07 smithi151 ceph-mon[104450]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:07.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:07 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:58:07.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:07 smithi151 ceph-mon[108966]: pgmap v2034: 119 pgs: 2 creating+peering, 14 unknown, 103 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:58:07.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:07 smithi151 ceph-mon[108966]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:58:07.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:07 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3701209012' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:58:07.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:07 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:58:07.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:07 smithi151 ceph-mon[108966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:07.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:07 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:58:07.410 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:07.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:07.434 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:07.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:07.535 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:07.542 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_5 rados --yes-i-really-mean-it || true 2024-09-13T14:58:07.561 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:07.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:07.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:07.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:07 smithi184 ceph-mon[92527]: pgmap v2034: 119 pgs: 2 creating+peering, 14 unknown, 103 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:58:07.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:07 smithi184 ceph-mon[92527]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:58:07.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:07 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3701209012' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:58:07.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:07 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:58:07.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:07 smithi184 ceph-mon[92527]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:07.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:07 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:58:07.663 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:07.689 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:07.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:07.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:07.774 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:07.792 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:07.816 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:07.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:07.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:07.918 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:07.940 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:07.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:07.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:08.042 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:08.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.069 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:08.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:08.170 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:08.192 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:08.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:08.294 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:08.316 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:08.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:08.418 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:08.443 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:08.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:08.545 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:08.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.571 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:08.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:08.591 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:08 smithi151 ceph-mon[104450]: osdmap e1270: 8 total, 8 up, 8 in 2024-09-13T14:58:08.591 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:08 smithi151 ceph-mon[108966]: osdmap e1270: 8 total, 8 up, 8 in 2024-09-13T14:58:08.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:08 smithi184 ceph-mon[92527]: osdmap e1270: 8 total, 8 up, 8 in 2024-09-13T14:58:08.672 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:08.694 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:08.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:08.795 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:08.817 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:08.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:08.873 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:08.919 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:08.944 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:08.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:09.046 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:09.068 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:09.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:09.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:09.170 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:09.196 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:09.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:09.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:09.298 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:09.324 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:09.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:09.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:09.426 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:09.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:09.452 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:09.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:09.553 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:09.575 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:09.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:09.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:09.589 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:09 smithi151 ceph-mon[108966]: pgmap v2037: 119 pgs: 3 creating+peering, 4 unknown, 112 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:58:09.590 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:09 smithi151 ceph-mon[104450]: pgmap v2037: 119 pgs: 3 creating+peering, 4 unknown, 112 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:58:09.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:09 smithi184 ceph-mon[92527]: pgmap v2037: 119 pgs: 3 creating+peering, 4 unknown, 112 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:58:09.677 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:09.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:09.700 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:09.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:09.801 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:09.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:09.824 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:09.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:09.925 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:09.947 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:09.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:09.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:10.049 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:10.074 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:10.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:10.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:10.141 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:58:10.142 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:58:10.158 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:58:10.158 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:58:10.173 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:58:10.174 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:58:10.176 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:10.190 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:58:10.190 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:58:10.191 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:58:10.198 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:10.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:10.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:10.206 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:58:10.206 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:58:10.223 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:58:10.223 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:58:10.239 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:58:10.240 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:58:10.255 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:58:10.256 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:58:10.301 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:10.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:10.324 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:10.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:10.425 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:10.444 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:10 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3686112035' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:10.444 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:10 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3686112035' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:10.446 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:10.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:10.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:10.548 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:10.574 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:10.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:10.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:10.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:10 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3686112035' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:10.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:58:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:58:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:58:10.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:10.698 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:10.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:10.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:10.800 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:10.822 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:10.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:10.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:10.925 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:10.952 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:10.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:10.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:11.054 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:11.075 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:11.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:11.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:11.177 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:11.203 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:11.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:11.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:11.306 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:11.332 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:11.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:11.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:11.434 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:11.456 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:11.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:11.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:11.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:11 smithi151 ceph-mon[104450]: pgmap v2038: 119 pgs: 3 creating+peering, 116 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:58:11.556 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:11 smithi151 ceph-mon[108966]: pgmap v2038: 119 pgs: 3 creating+peering, 116 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:58:11.558 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:11.580 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:11.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:11.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:11.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:11 smithi184 ceph-mon[92527]: pgmap v2038: 119 pgs: 3 creating+peering, 116 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:58:11.682 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:11.704 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:11.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:11.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:11.806 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:11.828 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:11.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:11.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:11.930 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:11.962 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:11.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:11.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:12.064 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:12.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:12.087 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:12.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:12.190 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:12.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:12.213 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:12.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:12.314 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:12.337 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:12.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:12.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:12.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:12 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:58:12.439 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:12.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:12.462 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:12.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:12.563 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:12.585 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:12.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:12.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:12.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:12 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:58:12.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:12 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:58:12.687 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:12.709 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:12.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:12.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:12.811 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:12.832 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:12.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:12.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:12.934 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:12.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:12.956 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:12.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:13.057 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:13.080 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:13.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:13.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:13.095 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:58:13.095 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-13T14:58:13.095 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-13T14:58:13.095 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:58:13.182 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:13.203 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:13.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:13.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:13.305 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:13.329 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:13.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:13.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:13.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:13 smithi184 ceph-mon[92527]: pgmap v2039: 119 pgs: 119 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:58:13.431 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:13.453 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:13.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:13.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:13.553 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:13 smithi151 ceph-mon[104450]: pgmap v2039: 119 pgs: 119 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:58:13.554 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:13 smithi151 ceph-mon[108966]: pgmap v2039: 119 pgs: 119 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:58:13.555 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:13.577 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:13.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:13.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:13.679 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:13.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:13.701 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:13.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:13.802 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:13.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:13.825 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:13.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:13.926 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:13.947 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:13.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:13.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:14.049 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:14.071 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:14.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:14.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:14.173 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:14.193 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:14.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:14.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:14.241 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T14:58:14.295 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:14.317 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:14.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:14.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:14.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:58:14.407 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:14 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1215905989' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:58:14.408 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:14 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1215905989' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:58:14.419 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:14.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:14.443 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:14.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:14.500 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:14.541 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:14 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1215905989' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:58:14.544 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:14.565 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:14.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:14.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:14.667 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:14.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:14.691 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:14.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:14.792 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:14.814 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:14.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:14.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:14.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:58:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:58:14] "GET /metrics HTTP/1.1" 200 34298 "" "Prometheus/2.51.0" 2024-09-13T14:58:14.882 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T14:58:14.916 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:14.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:14.941 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:14.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:15.043 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:15.066 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:15.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:15.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:15.168 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:15.190 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:15.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:15.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:15.246 INFO:teuthology.orchestra.run.smithi151.stderr:enabled application 'rados' on pool 'unique_pool_5' 2024-09-13T14:58:15.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:15.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:15.316 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:15.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:15.418 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:15.439 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:15.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:15.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:15.504 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:15 smithi184 ceph-mon[92527]: pgmap v2040: 119 pgs: 119 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:58:15.504 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:15 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1215905989' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:58:15.505 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:15 smithi184 ceph-mon[92527]: osdmap e1271: 8 total, 8 up, 8 in 2024-09-13T14:58:15.505 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:15 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3538905562' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:58:15.505 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:15 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:58:15.541 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:15.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:15.564 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:15.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:15.580 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:15 smithi151 ceph-mon[108966]: pgmap v2040: 119 pgs: 119 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:58:15.580 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:15 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1215905989' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:58:15.580 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:15 smithi151 ceph-mon[108966]: osdmap e1271: 8 total, 8 up, 8 in 2024-09-13T14:58:15.580 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:15 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3538905562' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:58:15.580 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:15 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:58:15.581 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:15 smithi151 ceph-mon[104450]: pgmap v2040: 119 pgs: 119 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:58:15.581 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:15 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1215905989' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:58:15.581 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:15 smithi151 ceph-mon[104450]: osdmap e1271: 8 total, 8 up, 8 in 2024-09-13T14:58:15.581 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:15 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3538905562' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:58:15.581 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:15 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:58:15.608 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T14:58:15.665 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:15.688 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:15.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:15.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:15.790 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:15.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:15.811 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:15.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:15.912 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:15.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:15.935 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:15.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:16.030 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:16.036 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:16.060 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:16.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:16.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:16.162 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:16.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:16.185 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:16.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:16.286 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:16.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:16.309 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:16.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:16.352 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:16 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:58:16.352 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:16 smithi151 ceph-mon[108966]: osdmap e1272: 8 total, 8 up, 8 in 2024-09-13T14:58:16.352 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:16.352 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:16.352 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:16 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:58:16.352 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:16 smithi151 ceph-mon[104450]: osdmap e1272: 8 total, 8 up, 8 in 2024-09-13T14:58:16.352 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:16.352 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:16.410 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:16.432 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:16.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:16.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:16.534 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:16.556 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:16.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:16.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:16.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:16 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:58:16.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:16 smithi184 ceph-mon[92527]: osdmap e1272: 8 total, 8 up, 8 in 2024-09-13T14:58:16.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:16.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:16.658 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:16.679 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:16.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:16.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:16.782 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:16.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:16.804 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:16.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:16.905 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:16.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:16.927 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:16.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:17.029 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:17.050 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:17.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:17.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:17.055 DEBUG:teuthology.orchestra.run.smithi151:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_5 bench 90 write' 2024-09-13T14:58:17.058 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T14:58:17.058 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T14:58:17.152 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:17.174 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:17.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:17.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:17.185 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:hints = 1 2024-09-13T14:58:17.185 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T14:58:17.186 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Object prefix: benchmark_data_smithi151_288181 2024-09-13T14:58:17.186 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:58:17.186 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 0 0 0 0 0 0 - 0 2024-09-13T14:58:17.275 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:17.309 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:17.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:17.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:17.411 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:17.442 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:17.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:17.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:17.544 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:17.579 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:17.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:17.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:17.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:17 smithi184 ceph-mon[92527]: pgmap v2043: 119 pgs: 119 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:58:17.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:17 smithi184 ceph-mon[92527]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:58:17.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:17.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:17.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:17.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:17.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:17.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:17 smithi151 ceph-mon[104450]: pgmap v2043: 119 pgs: 119 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:58:17.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:17 smithi151 ceph-mon[104450]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:58:17.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:17.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:17.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:17.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:17.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:17.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:17 smithi151 ceph-mon[108966]: pgmap v2043: 119 pgs: 119 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:58:17.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:17 smithi151 ceph-mon[108966]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:58:17.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:17.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:17.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:17.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:17.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:17.682 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:17.716 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:17.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:17.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:17.818 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:17.855 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:17.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:17.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:17.957 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:17.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:17.993 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:17.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:18.094 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:18.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:18.130 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:18.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:18.186 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 1 16 4027 4011 250.673 250.688 0.00367535 0.00397934 2024-09-13T14:58:18.232 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:18.266 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:18.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:18.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:18.368 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:18.400 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:18.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:18.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:18.502 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:18.535 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:18.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:18.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:18.637 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:18.672 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:18.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:18.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:18.774 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:18.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:18.812 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:18.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:18.913 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:18.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:18.947 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:18.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:19.049 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:19.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:19.084 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:19.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:19.186 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:19.186 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 2 16 7673 7657 239.264 227.875 0.00468818 0.00417026 2024-09-13T14:58:19.224 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:19.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:19.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:19.326 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:19.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:19.365 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:19.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:19.466 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:19.503 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:19.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:19.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:19.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:19 smithi184 ceph-mon[92527]: pgmap v2044: 119 pgs: 119 active+clean; 10 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 MiB/s wr, 19 op/s 2024-09-13T14:58:19.605 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:19.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:19 smithi151 ceph-mon[104450]: pgmap v2044: 119 pgs: 119 active+clean; 10 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 MiB/s wr, 19 op/s 2024-09-13T14:58:19.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:19 smithi151 ceph-mon[108966]: pgmap v2044: 119 pgs: 119 active+clean; 10 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 MiB/s wr, 19 op/s 2024-09-13T14:58:19.641 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:19.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:19.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:19.743 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:19.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:19.782 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:19.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:19.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:19.921 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:19.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:19.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:20.023 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:20.058 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:20.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:20.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:20.160 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:20.186 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 3 16 10747 10731 223.545 192.125 0.00859143 0.00446556 2024-09-13T14:58:20.199 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:20.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:20.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:20.301 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:20.336 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:20.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:20.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:20.438 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:20.471 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:20.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:20.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:20.574 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:20.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:58:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:58:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:58:20.606 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:20.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:20.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:20.708 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:20.744 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:20.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:20.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:20.846 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:20.886 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:20.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:20.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:20.988 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:21.015 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:21.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:21.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:21.117 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:21.154 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:21.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:21.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:21.186 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 4 16 13697 13681 213.748 184.375 0.00712048 0.0046714 2024-09-13T14:58:21.256 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:21.294 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:21.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:21.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:21.396 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:21.428 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:21.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:21.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:21.529 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:21.566 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:21.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:21.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:21.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:21 smithi184 ceph-mon[92527]: pgmap v2045: 119 pgs: 119 active+clean; 94 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 12 MiB/s wr, 186 op/s 2024-09-13T14:58:21.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:21 smithi151 ceph-mon[104450]: pgmap v2045: 119 pgs: 119 active+clean; 94 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 12 MiB/s wr, 186 op/s 2024-09-13T14:58:21.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:21 smithi151 ceph-mon[108966]: pgmap v2045: 119 pgs: 119 active+clean; 94 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 12 MiB/s wr, 186 op/s 2024-09-13T14:58:21.668 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:21.704 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:21.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:21.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:21.806 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:21.840 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:21.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:21.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:21.942 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:21.973 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:21.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:21.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:22.075 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:22.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:22.112 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:22.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:22.186 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 5 16 16467 16451 205.62 173.125 0.00395149 0.00485743 2024-09-13T14:58:22.213 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:22.247 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:22.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:22.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:22.349 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:22.389 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:22.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:22.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:22.494 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:22.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:22.529 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:22.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:22.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:22 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:22.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:22 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:22.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:22 smithi151 ceph-mon[104450]: pgmap v2046: 119 pgs: 119 active+clean; 238 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 30 MiB/s wr, 474 op/s 2024-09-13T14:58:22.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:22 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:22.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:22 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:22.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:22 smithi151 ceph-mon[108966]: pgmap v2046: 119 pgs: 119 active+clean; 238 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 30 MiB/s wr, 474 op/s 2024-09-13T14:58:22.630 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:22.662 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:22.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:22.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:22.764 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:22.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:22.787 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:22.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:22.790 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:22 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:22.790 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:22 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:22.790 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:22 smithi184 ceph-mon[92527]: pgmap v2046: 119 pgs: 119 active+clean; 238 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 30 MiB/s wr, 474 op/s 2024-09-13T14:58:22.888 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:22.920 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:22.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:22.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:23.022 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:23.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:23.044 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:23.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:23.145 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:23.180 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:23.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:23.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:23.186 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 6 16 18771 18755 195.348 144 0.00383362 0.0051127 2024-09-13T14:58:23.281 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:23.318 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:23.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:23.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:23.420 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:23.440 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:23.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:23.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:23.542 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:23.580 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:23.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:23.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:23.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:23 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3946669319' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:23.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:23 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3946669319' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:23.681 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:23.716 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:23.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:23.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:23.818 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:23.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:23.845 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:23.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:23.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:23 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3946669319' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:23.946 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:23.983 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:23.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:23.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:24.085 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:24.109 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:24.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:24.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:24.186 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 7 15 20838 20823 185.904 129.25 0.00879393 0.00537306 2024-09-13T14:58:24.211 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:24.244 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:24.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:24.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:24.346 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:24.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:24.379 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:24.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:24.479 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:24.518 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:24.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:24.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:24.620 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:24.654 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:24.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:24.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:24.728 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-13T14:58:24.728 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:24 smithi151 ceph-mon[104450]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:24.729 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:24 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3946669319' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:58:24.729 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:24 smithi151 ceph-mon[104450]: osdmap e1273: 8 total, 8 up, 8 in 2024-09-13T14:58:24.729 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:24 smithi151 ceph-mon[104450]: pgmap v2048: 119 pgs: 119 active+clean; 932 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 110 MiB/s wr, 1.75k op/s 2024-09-13T14:58:24.729 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:58:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:58:24] "GET /metrics HTTP/1.1" 200 36631 "" "Prometheus/2.51.0" 2024-09-13T14:58:24.730 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:24 smithi151 ceph-mon[108966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:24.730 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:24 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3946669319' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:58:24.730 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:24 smithi151 ceph-mon[108966]: osdmap e1273: 8 total, 8 up, 8 in 2024-09-13T14:58:24.730 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:24 smithi151 ceph-mon[108966]: pgmap v2048: 119 pgs: 119 active+clean; 932 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 110 MiB/s wr, 1.75k op/s 2024-09-13T14:58:24.756 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:24.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:24.785 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:24.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:24.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:24 smithi184 ceph-mon[92527]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:24.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:24 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3946669319' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:58:24.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:24 smithi184 ceph-mon[92527]: osdmap e1273: 8 total, 8 up, 8 in 2024-09-13T14:58:24.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:24 smithi184 ceph-mon[92527]: pgmap v2048: 119 pgs: 119 active+clean; 932 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 110 MiB/s wr, 1.75k op/s 2024-09-13T14:58:24.887 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:24.922 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:24.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:24.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:25.025 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:25.046 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:25.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:25.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:25.148 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:25.169 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:25.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:25.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:25.186 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 8 16 22967 22951 179.289 133 0.00163888 0.00550959 2024-09-13T14:58:25.271 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:25.292 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:25.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:25.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:25.394 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:25.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:25.430 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:25.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:25.531 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:25.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:25.564 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:25.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:25.666 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:25.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:25.700 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:25.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:25.801 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:25.836 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:25.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:25.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:25.938 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:25.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:25.966 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:25.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:26.067 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:26.103 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:26.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:26.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:26.187 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 9 16 24863 24847 172.533 118.5 0.00424781 0.00579045 2024-09-13T14:58:26.205 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:26.226 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:26.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:26.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:26.328 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:26.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:26.366 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:26.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:26.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:26.507 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:26.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:26.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:26.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:26.630 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:26.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:26.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:26.733 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:26.767 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:26.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:26.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:26.787 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:26.788 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:26.869 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:26.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:26.914 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:26.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:27.015 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:27.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:27.046 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:27.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:27.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:26 smithi184 ceph-mon[92527]: pgmap v2049: 119 pgs: 119 active+clean; 1.1 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 112 MiB/s wr, 1.80k op/s 2024-09-13T14:58:27.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:26 smithi151 ceph-mon[104450]: pgmap v2049: 119 pgs: 119 active+clean; 1.1 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 112 MiB/s wr, 1.80k op/s 2024-09-13T14:58:27.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:26 smithi151 ceph-mon[108966]: pgmap v2049: 119 pgs: 119 active+clean; 1.1 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 112 MiB/s wr, 1.80k op/s 2024-09-13T14:58:27.147 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:27.158 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:58:27.158 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:58:27.187 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 10 15 26555 26540 165.86 105.812 0.00504655 0.00602367 2024-09-13T14:58:27.191 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:27.192 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:58:27.192 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:58:27.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:27.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:27.225 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:58:27.225 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:58:27.254 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:58:27.254 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:58:27.295 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:27.317 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:58:27.318 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:58:27.322 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:27.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:27.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:27.334 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:58:27.334 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:58:27.364 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:58:27.365 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:58:27.396 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:58:27.397 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:58:27.424 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:27.458 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:27.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:27.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:27.560 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:27.597 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:27.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:27.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:27.699 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:27.731 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:27.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:27.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:27.833 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:27.868 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:27.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:27.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:27.970 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:28.006 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:28.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:28.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:27 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1265977662' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:28.107 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:28.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:27 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1265977662' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:28.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:27 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1265977662' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:28.140 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:28.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:28.187 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 11 16 28897 28881 164.082 146.312 0.00763575 0.00608954 2024-09-13T14:58:28.242 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:28.277 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:28.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:28.379 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:28.412 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:28.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:28.514 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:28.551 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:28.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:28.654 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:28.689 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:28.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:28.791 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:28.826 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:28.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:28.929 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:28.965 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:28.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:29.067 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:29.076 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:28 smithi151 ceph-mon[104450]: pgmap v2050: 119 pgs: 119 active+clean; 1.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 151 MiB/s wr, 2.42k op/s 2024-09-13T14:58:29.077 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:28 smithi151 ceph-mon[108966]: pgmap v2050: 119 pgs: 119 active+clean; 1.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 151 MiB/s wr, 2.42k op/s 2024-09-13T14:58:29.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:28 smithi184 ceph-mon[92527]: pgmap v2050: 119 pgs: 119 active+clean; 1.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 151 MiB/s wr, 2.42k op/s 2024-09-13T14:58:29.105 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:29.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:29.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:29.188 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 12 16 31242 31226 162.609 146.562 0.00632191 0.00614385 2024-09-13T14:58:29.208 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:29.243 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:29.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:29.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:29.345 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:29.381 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:29.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:29.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:29.483 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:29.519 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:29.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:29.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:29.621 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:29.657 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:29.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:29.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:29.759 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:29.796 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:29.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:29.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:29.898 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:29.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:29.933 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:29.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:30.035 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:30.070 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:30.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:30.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:30.173 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:30.188 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 13 16 33571 33555 161.295 145.562 0.00493171 0.00619224 2024-09-13T14:58:30.208 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:30.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:30.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:30.310 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:30.344 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:30.344 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:58:30.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:30.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:30.446 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:30.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:30.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:30.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:30.598 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:30.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:58:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:58:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:58:30.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:30.621 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:30.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:30.723 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:30.761 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:30.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:30.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:30.863 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:30.899 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:30.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:30.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:31.001 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:31.034 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:31.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:31.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:31.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:30 smithi184 ceph-mon[92527]: pgmap v2051: 119 pgs: 119 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-09-13T14:58:31.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:30 smithi151 ceph-mon[104450]: pgmap v2051: 119 pgs: 119 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-09-13T14:58:31.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:30 smithi151 ceph-mon[108966]: pgmap v2051: 119 pgs: 119 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-09-13T14:58:31.136 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:31.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:31.173 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:31.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:31.188 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 14 16 35611 35595 158.88 127.5 0.00526254 0.00628885 2024-09-13T14:58:31.274 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:31.309 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:31.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:31.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:31.411 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:31.441 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:31.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:31.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:31.543 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:31.563 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:31.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:31.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:31.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:58:31.666 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:31.693 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:31.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:31.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:31.795 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:31.821 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:31.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:31.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:31.924 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:31.946 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:31.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:31.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:32.048 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:32.070 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:32.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:32.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:32.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:32.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:32.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:32.172 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:32.188 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 15 16 36182 36166 150.668 35.6875 0.00287484 0.00639992 2024-09-13T14:58:32.196 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:32.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:32.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:32.298 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:32.336 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:32.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:32.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:32.438 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:32.467 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:32.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:32.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:32.569 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:32.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:32.610 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:32.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:32.711 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:32.745 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:32.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:32.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:32.847 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:32.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:32 smithi151 ceph-mon[104450]: pgmap v2052: 119 pgs: 119 active+clean; 1.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 157 MiB/s wr, 2.50k op/s 2024-09-13T14:58:32.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:32 smithi151 ceph-mon[108966]: pgmap v2052: 119 pgs: 119 active+clean; 1.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 157 MiB/s wr, 2.50k op/s 2024-09-13T14:58:32.885 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:32.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:32.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:32.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:33.022 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:33.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:33.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:33.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:32 smithi184 ceph-mon[92527]: pgmap v2052: 119 pgs: 119 active+clean; 1.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 157 MiB/s wr, 2.50k op/s 2024-09-13T14:58:33.124 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:33.155 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:33.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:33.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:33.189 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 16 16 38033 38017 148.476 115.688 0.00793178 0.00673022 2024-09-13T14:58:33.257 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:33.291 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:33.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:33.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:33.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:33.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:33.430 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:33.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:33.531 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:33.563 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:33.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:33.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:33.665 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:33.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:33.699 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:33.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:33.801 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:33.857 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:33.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:33.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:33.958 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:33.996 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:33.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:33.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:34.098 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:34.141 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:34.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:34.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:34.189 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 17 16 40032 40016 147.091 124.938 0.00611123 0.0067931 2024-09-13T14:58:34.243 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:34.274 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:34.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:34.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:34.377 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:34.413 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:34.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:34.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:34.515 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:34.540 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:34.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:34.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:34.642 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:34.675 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:34.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:34.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:34.777 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:34.808 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:34.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:34.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:34.851 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T14:58:34.852 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:58:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:58:34] "GET /metrics HTTP/1.1" 200 36698 "" "Prometheus/2.51.0" 2024-09-13T14:58:34.910 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:34.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:34.950 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:34.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:35.052 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:35.089 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:35.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:35.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:35.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:34 smithi151 ceph-mon[104450]: pgmap v2053: 119 pgs: 119 active+clean; 2.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 129 MiB/s wr, 2.06k op/s 2024-09-13T14:58:35.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:34 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3625885218' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:58:35.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:34 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:58:35.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:34 smithi151 ceph-mon[108966]: pgmap v2053: 119 pgs: 119 active+clean; 2.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 129 MiB/s wr, 2.06k op/s 2024-09-13T14:58:35.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:34 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3625885218' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:58:35.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:34 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:58:35.189 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 18 16 41779 41763 144.985 109.188 0.0056693 0.00689198 2024-09-13T14:58:35.191 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:35.230 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:35.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:35.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:35.331 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:35.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:34 smithi184 ceph-mon[92527]: pgmap v2053: 119 pgs: 119 active+clean; 2.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 129 MiB/s wr, 2.06k op/s 2024-09-13T14:58:35.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:34 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3625885218' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:58:35.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:34 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:58:35.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:35.384 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:35.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:35.485 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:35.490 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T14:58:35.491 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:58:35.517 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:35.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:35.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:35.619 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:35.650 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:35.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:35.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:35.752 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:35.796 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:35.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:35.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:35.898 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:35.920 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:35.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:35.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:36.022 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:36.054 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:36.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:36.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:36.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:35 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:58:36.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:35 smithi151 ceph-mon[104450]: osdmap e1274: 8 total, 8 up, 8 in 2024-09-13T14:58:36.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:35 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:58:36.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:35 smithi151 ceph-mon[108966]: osdmap e1274: 8 total, 8 up, 8 in 2024-09-13T14:58:36.157 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:36.189 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 19 16 43309 43293 142.387 95.625 0.00398384 0.0070174 2024-09-13T14:58:36.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:36.190 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:36.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:36.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:36.330 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:36.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:36.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:36.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:35 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:58:36.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:35 smithi184 ceph-mon[92527]: osdmap e1274: 8 total, 8 up, 8 in 2024-09-13T14:58:36.432 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:36.453 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:36.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:36.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:36.555 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:36.587 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:36.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:36.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:36.689 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:36.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:36.725 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:36.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:36.826 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:36.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:36.848 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:36.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:36.949 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:36.984 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:36.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:36.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:37.086 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:37.124 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:37.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:37.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:37.189 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:2024-09-13T14:58:37.189125+0000 min lat: 0.00119391 max lat: 0.812283 avg lat: 0.00717001 2024-09-13T14:58:37.189 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:58:37.189 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 20 16 44624 44608 139.377 82.1875 0.011013 0.00717001 2024-09-13T14:58:37.226 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:37.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:37.264 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:37.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:37.338 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:36 smithi151 ceph-mon[104450]: pgmap v2055: 119 pgs: 119 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 120 MiB/s wr, 1.92k op/s 2024-09-13T14:58:37.339 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:36 smithi151 ceph-mon[108966]: pgmap v2055: 119 pgs: 119 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 120 MiB/s wr, 1.92k op/s 2024-09-13T14:58:37.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:36 smithi184 ceph-mon[92527]: pgmap v2055: 119 pgs: 119 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 120 MiB/s wr, 1.92k op/s 2024-09-13T14:58:37.365 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:37.403 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:37.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:37.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:37.505 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:37.537 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:37.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:37.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:37.597 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:37.640 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:37.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:37.668 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:37.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:37.769 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:37.801 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:37.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:37.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:37.904 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:37.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:37.944 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:37.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:38.045 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:38.075 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:38.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:38.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:38.177 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:38.189 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 21 16 46446 46430 138.162 113.875 0.0102183 0.00723268 2024-09-13T14:58:38.209 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:38.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:38.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:38.240 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T14:58:38.240 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:58:38.311 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:38.340 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:38.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:38.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:38.442 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:38.475 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:38.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:38.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:38.577 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:38.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:38.606 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:38.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:38.707 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:38.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:38.739 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:38.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:38.840 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:38.882 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:38.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:38.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:38.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:39.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:39.023 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:39.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:39.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:38 smithi151 ceph-mon[104450]: pgmap v2056: 119 pgs: 119 active+clean; 2.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-09-13T14:58:39.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:38 smithi151 ceph-mon[108966]: pgmap v2056: 119 pgs: 119 active+clean; 2.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-09-13T14:58:39.123 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:39.154 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:39.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:39.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:39.189 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 22 16 48184 48168 136.819 108.625 0.00770313 0.00730432 2024-09-13T14:58:39.256 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:39.275 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T14:58:39.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:39.285 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:39.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:39.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:38 smithi184 ceph-mon[92527]: pgmap v2056: 119 pgs: 119 active+clean; 2.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-09-13T14:58:39.387 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:39.429 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:39.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:39.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:39.531 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:39.565 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:39.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:39.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:39.667 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:39.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:39.718 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:39.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:39.819 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:39.857 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:39.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:39.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:39.960 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:39.973 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:39.986 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:39.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:39.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:40.087 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:40.131 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:40.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:40.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:40.189 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 23 16 49907 49891 135.552 107.688 0.00599263 0.00737226 2024-09-13T14:58:40.232 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:40.263 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:40.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:40.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:40.365 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:40.399 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:40.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:40.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:40.502 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:40.538 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:40.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:40.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:40.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:58:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:58:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:58:40.641 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:40.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:40.677 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:40.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:40.701 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:40.777 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:40.809 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:40.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:40.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:40.911 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:40.943 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:40.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:40.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:41.045 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:41.083 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:41.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:41.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:41.185 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:41.189 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 24 16 51619 51603 134.362 107 0.00643355 0.00743823 2024-09-13T14:58:41.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:41.224 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:41.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:41.238 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:40 smithi151 ceph-mon[104450]: pgmap v2057: 119 pgs: 119 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-09-13T14:58:41.238 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:40 smithi151 ceph-mon[108966]: pgmap v2057: 119 pgs: 119 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-09-13T14:58:41.325 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:41.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:40 smithi184 ceph-mon[92527]: pgmap v2057: 119 pgs: 119 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-09-13T14:58:41.358 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:41.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:41.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:41.460 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:41.494 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:41.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:41.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:41.596 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:41.633 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:41.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:41.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:41.735 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:41.770 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:41.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:41.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:41.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:41.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:41.913 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:41.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:42.015 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:42.042 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:42.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:42.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:42.144 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:42.190 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 25 16 53319 53303 133.237 106.25 0.0143122 0.00749954 2024-09-13T14:58:42.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:42.197 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:42.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:42.298 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:42.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:42.329 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:42.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:42.430 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:42.474 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:42.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:42.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:42.576 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:42.597 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:42.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:42.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:42.699 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:42.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:42.732 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:42.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:42.833 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:42.868 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:42.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:42.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:42.970 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:43.008 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:43.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:43.110 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:43.144 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:43.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:43.190 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 26 16 54915 54899 131.949 99.75 0.00481772 0.00757382 2024-09-13T14:58:43.203 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:42 smithi151 ceph-mon[104450]: pgmap v2058: 119 pgs: 119 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-09-13T14:58:43.203 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:42 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2074644284' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:43.204 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:42 smithi151 ceph-mon[108966]: pgmap v2058: 119 pgs: 119 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-09-13T14:58:43.204 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:42 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2074644284' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:43.245 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:43.279 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:43.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:43.356 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:42 smithi184 ceph-mon[92527]: pgmap v2058: 119 pgs: 119 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-09-13T14:58:43.356 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:42 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2074644284' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:43.382 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:43.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.415 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:43.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:43.516 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:43.551 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:43.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:43.653 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:43.691 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:43.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:43.755 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:58:43.756 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:58:43.756 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:58:43.756 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:58:43.756 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:58:43.756 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:58:43.756 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:58:43.756 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:58:43.756 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:58:43.756 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:58:43.757 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:58:43.757 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:58:43.757 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:58:43.757 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:58:43.757 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:58:43.757 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:58:43.795 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:43.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.832 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:43.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:43.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:43.974 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:43.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:44.076 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:44.108 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:44.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:44.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:44.190 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 27 16 56618 56602 131.004 106.438 0.00873858 0.00762829 2024-09-13T14:58:44.209 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:44.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:44.246 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:44.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:44.347 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:44.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:44.387 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:44.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:44.488 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:44.527 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:44.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:44.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:44.629 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:44.662 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:44.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:44.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:44.764 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:44.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:44.804 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:44.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:44.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:58:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:58:44] "GET /metrics HTTP/1.1" 200 36698 "" "Prometheus/2.51.0" 2024-09-13T14:58:44.905 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:44.931 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:44.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:44.945 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:44.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:45.046 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:45.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:45.081 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:45.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:45.163 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:44 smithi151 ceph-mon[104450]: pgmap v2059: 119 pgs: 119 active+clean; 3.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-09-13T14:58:45.163 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:44 smithi151 ceph-mon[108966]: pgmap v2059: 119 pgs: 119 active+clean; 3.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-09-13T14:58:45.184 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:45.190 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 28 16 58234 58218 129.932 101 0.0129647 0.0076915 2024-09-13T14:58:45.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:45.225 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:45.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:45.326 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:45.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:44 smithi184 ceph-mon[92527]: pgmap v2059: 119 pgs: 119 active+clean; 3.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-09-13T14:58:45.364 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:45.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:45.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:45.466 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:45.496 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:45.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:45.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:45.598 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:45.632 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:45.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:45.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:45.734 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:45.776 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:45.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:45.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:45.878 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:45.906 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:45.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:45.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:46.009 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:46.045 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:46.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:46.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:46.146 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:46.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:46.187 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:46.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:46.190 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 29 16 59735 59719 128.686 93.8125 0.00660599 0.00776455 2024-09-13T14:58:46.229 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:58:46.231 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T14:58:46.289 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:46.319 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:46.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:46.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:46.421 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:46.461 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:46.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:46.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:46.517 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:58:46.563 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:46.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:46.593 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:46.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:46.694 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:46.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:46.746 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:46.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:46.847 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:46.876 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:46.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:46.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:46.898 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-13T14:58:46.979 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:47.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.011 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:47.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:47.083 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:46 smithi151 ceph-mon[104450]: pgmap v2060: 119 pgs: 119 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 97 MiB/s wr, 1.56k op/s 2024-09-13T14:58:47.083 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:46 smithi151 ceph-mon[104450]: from='client.55585 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:58:47.083 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:47.083 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:46 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3604174723' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:58:47.083 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:46 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:58:47.084 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:46 smithi151 ceph-mon[108966]: pgmap v2060: 119 pgs: 119 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 97 MiB/s wr, 1.56k op/s 2024-09-13T14:58:47.084 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:46 smithi151 ceph-mon[108966]: from='client.55585 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:58:47.084 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:47.084 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:46 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3604174723' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:58:47.084 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:46 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:58:47.112 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:47.159 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:47.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:47.190 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 30 16 60996 60980 127.024 78.8125 0.00428245 0.0078673 2024-09-13T14:58:47.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:47.299 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:47.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:47.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:46 smithi184 ceph-mon[92527]: pgmap v2060: 119 pgs: 119 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 97 MiB/s wr, 1.56k op/s 2024-09-13T14:58:47.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:46 smithi184 ceph-mon[92527]: from='client.55585 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:58:47.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:47.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:46 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3604174723' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:58:47.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:46 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:58:47.401 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:47.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.436 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:47.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:47.537 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:47.569 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:47.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:47.672 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:47.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.707 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:47.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:47.808 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:47.845 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:47.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:47.949 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:47.984 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:47.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:48.086 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:48.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:47 smithi151 ceph-mon[108966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:48.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:47 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:58:48.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:47 smithi151 ceph-mon[108966]: osdmap e1275: 8 total, 8 up, 8 in 2024-09-13T14:58:48.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:47 smithi151 ceph-mon[104450]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:48.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:47 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:58:48.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:47 smithi151 ceph-mon[104450]: osdmap e1275: 8 total, 8 up, 8 in 2024-09-13T14:58:48.115 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:48.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:48.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:48.190 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 31 16 62575 62559 126.109 98.6875 0.0107363 0.00792471 2024-09-13T14:58:48.217 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:48.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:48.253 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:48.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:48.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:47 smithi184 ceph-mon[92527]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:48.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:47 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:58:48.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:47 smithi184 ceph-mon[92527]: osdmap e1275: 8 total, 8 up, 8 in 2024-09-13T14:58:48.354 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:48.389 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:48.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:48.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:48.491 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:48.512 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:48.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:48.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:48.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:48.652 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:48.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:48.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:48.754 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:48.784 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:48.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:48.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:48.886 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:48.919 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:48.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:48.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:49.020 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:49.053 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:49.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:49.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:49.155 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:49.190 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 32 16 63918 63902 124.791 83.9375 0.00560313 0.0080089 2024-09-13T14:58:49.193 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:49.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:49.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:49.254 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:48 smithi184 ceph-mon[92527]: pgmap v2062: 119 pgs: 119 active+clean; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-09-13T14:58:49.295 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:49.327 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:49.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:49.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:49.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:48 smithi151 ceph-mon[108966]: pgmap v2062: 119 pgs: 119 active+clean; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-09-13T14:58:49.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:48 smithi151 ceph-mon[104450]: pgmap v2062: 119 pgs: 119 active+clean; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-09-13T14:58:49.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:49.460 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:49.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:49.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:49.562 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:49.600 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:49.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:49.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:49.702 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:49.737 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:49.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:49.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:49.839 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:49.873 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:49.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:49.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:49.975 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:50.006 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:50.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:50.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:50.109 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:50.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:50.142 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:50.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:50.190 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 33 16 65296 65280 123.619 86.125 0.00547361 0.0080837 2024-09-13T14:58:50.243 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:50.285 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:50.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:50.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:50.354 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:58:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:58:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:58:50.387 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:50.415 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:50.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:50.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:50.517 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:50.554 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:50.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:50.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:50.656 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:50.692 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:50.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:50.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:50.793 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:50.825 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:50.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:50.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:50.926 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:50.964 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:50.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:50.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:51.066 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:51.093 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:51.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:51.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:51.190 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 34 16 66608 66592 122.395 82 0.0033267 0.00816209 2024-09-13T14:58:51.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:51.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:51.216 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:51.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:51.317 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:51.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:51.344 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:51.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:51.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:50 smithi184 ceph-mon[92527]: pgmap v2063: 119 pgs: 119 active+clean; 3.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-13T14:58:51.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:50 smithi151 ceph-mon[108966]: pgmap v2063: 119 pgs: 119 active+clean; 3.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-13T14:58:51.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:50 smithi151 ceph-mon[104450]: pgmap v2063: 119 pgs: 119 active+clean; 3.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-13T14:58:51.445 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:51.483 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:51.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:51.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:51.585 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:51.625 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:51.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:51.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:51.727 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:51.762 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:51.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:51.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:51.864 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:51.897 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:51.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:51.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:51.999 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:52.031 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:52.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:52.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:52.133 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:52.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:52.189 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:52.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:52.191 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 35 16 67828 67812 121.076 76.25 0.00830981 0.00825432 2024-09-13T14:58:52.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:52.326 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:52.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:52.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:52.428 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:52.462 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:52.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:52.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:52.565 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:52.601 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:52.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:52.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:52.703 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:52.735 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:52.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:52.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:52.837 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:52.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:52.870 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:52.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:52.971 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:53.010 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:53.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:53.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:53.112 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:53.148 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:53.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:53.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:53.191 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 36 16 69112 69096 119.941 80.25 0.00596711 0.00833272 2024-09-13T14:58:53.250 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:53.283 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:53.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:53.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:53.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:52 smithi184 ceph-mon[92527]: pgmap v2064: 119 pgs: 119 active+clean; 3.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-09-13T14:58:53.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:52 smithi151 ceph-mon[104450]: pgmap v2064: 119 pgs: 119 active+clean; 3.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-09-13T14:58:53.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:52 smithi151 ceph-mon[108966]: pgmap v2064: 119 pgs: 119 active+clean; 3.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-09-13T14:58:53.388 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:53.423 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:53.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:53.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:53.443 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":2060,"stamp":"2024-09-13T14:58:45.757834+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3548846029,"num_objects":54168,"num_object_clones":0,"num_object_copies":162504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54205,"num_write_kb":3465119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":860775,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57300,"ondisk_log_size":57300,"up":357,"acting":357,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":357,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":16707000,"kb_used_data":10314936,"kb_used_omap":149,"kb_used_meta":6391850,"kb_avail":733254216,"statfs":{"total":767960285184,"available":750852317184,"internally_reserved":0,"allocated":10562494464,"data_stored":10541356999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":153401,"internal_metadata":6545254599},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1114636288,"num_objects":17008,"num_object_clones":0,"num_object_copies":51024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17008,"num_write_kb":1088512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.908772"},"pg_stats":[{"pgid":"1.1d","version":"261'100","reported_seq":122,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.860952+0000","last_change":"2024-09-13T14:54:58.831284+0000","last_active":"2024-09-13T14:58:34.860952+0000","last_peered":"2024-09-13T14:58:34.860952+0000","last_clean":"2024-09-13T14:58:34.860952+0000","last_became_active":"2024-09-13T14:54:58.830090+0000","last_became_peered":"2024-09-13T14:54:58.830090+0000","last_unstale":"2024-09-13T14:58:34.860952+0000","last_undegraded":"2024-09-13T14:58:34.860952+0000","last_fullsized":"2024-09-13T14:58:34.860952+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:17.988345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:17.988345+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:02:05.633533+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":1984,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.853158+0000","last_change":"2024-09-13T14:55:52.137133+0000","last_active":"2024-09-13T14:58:34.853158+0000","last_peered":"2024-09-13T14:58:34.853158+0000","last_clean":"2024-09-13T14:58:34.853158+0000","last_became_active":"2024-09-13T14:55:41.862495+0000","last_became_peered":"2024-09-13T14:55:41.862495+0000","last_unstale":"2024-09-13T14:58:34.853158+0000","last_undegraded":"2024-09-13T14:58:34.853158+0000","last_fullsized":"2024-09-13T14:58:34.853158+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:51:35.574651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2280,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":65,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.515657+0000","last_change":"2024-09-13T14:58:05.196679+0000","last_active":"2024-09-13T14:58:23.515657+0000","last_peered":"2024-09-13T14:58:23.515657+0000","last_clean":"2024-09-13T14:58:23.515657+0000","last_became_active":"2024-09-13T14:58:05.196297+0000","last_became_peered":"2024-09-13T14:58:05.196297+0000","last_unstale":"2024-09-13T14:58:23.515657+0000","last_undegraded":"2024-09-13T14:58:23.515657+0000","last_fullsized":"2024-09-13T14:58:23.515657+0000","mapping_epoch":1267,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:56:17.644420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":268,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.864453+0000","last_change":"2024-09-13T14:45:19.810203+0000","last_active":"2024-09-13T14:58:34.864453+0000","last_peered":"2024-09-13T14:58:34.864453+0000","last_clean":"2024-09-13T14:58:34.864453+0000","last_became_active":"2024-09-13T14:45:19.809693+0000","last_became_peered":"2024-09-13T14:45:19.809693+0000","last_unstale":"2024-09-13T14:58:34.864453+0000","last_undegraded":"2024-09-13T14:58:34.864453+0000","last_fullsized":"2024-09-13T14:58:34.864453+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:20.219673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":267,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.862553+0000","last_change":"2024-09-13T14:45:19.807991+0000","last_active":"2024-09-13T14:58:34.862553+0000","last_peered":"2024-09-13T14:58:34.862553+0000","last_clean":"2024-09-13T14:58:34.862553+0000","last_became_active":"2024-09-13T14:45:19.807662+0000","last_became_peered":"2024-09-13T14:45:19.807662+0000","last_unstale":"2024-09-13T14:58:34.862553+0000","last_undegraded":"2024-09-13T14:58:34.862553+0000","last_fullsized":"2024-09-13T14:58:34.862553+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:29.306108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"261'100","reported_seq":1375,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.512361+0000","last_change":"2024-09-13T14:54:05.604842+0000","last_active":"2024-09-13T14:58:23.512361+0000","last_peered":"2024-09-13T14:58:23.512361+0000","last_clean":"2024-09-13T14:58:23.512361+0000","last_became_active":"2024-09-13T14:54:05.604072+0000","last_became_peered":"2024-09-13T14:54:05.604072+0000","last_unstale":"2024-09-13T14:58:23.512361+0000","last_undegraded":"2024-09-13T14:58:23.512361+0000","last_fullsized":"2024-09-13T14:58:23.512361+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_clean_scrub_stamp":"2024-09-13T14:44:54.259482+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:41:25.897474+0000","scrub_duration":0.015265569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":51,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.519354+0000","last_change":"2024-09-13T14:55:41.870022+0000","last_active":"2024-09-13T14:58:23.519354+0000","last_peered":"2024-09-13T14:58:23.519354+0000","last_clean":"2024-09-13T14:58:23.519354+0000","last_became_active":"2024-09-13T14:55:41.786811+0000","last_became_peered":"2024-09-13T14:55:41.786811+0000","last_unstale":"2024-09-13T14:58:23.519354+0000","last_undegraded":"2024-09-13T14:58:23.519354+0000","last_fullsized":"2024-09-13T14:58:23.519354+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:38:18.373670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":55,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.515708+0000","last_change":"2024-09-13T14:54:58.830988+0000","last_active":"2024-09-13T14:58:23.515708+0000","last_peered":"2024-09-13T14:58:23.515708+0000","last_clean":"2024-09-13T14:58:23.515708+0000","last_became_active":"2024-09-13T14:54:58.830686+0000","last_became_peered":"2024-09-13T14:54:58.830686+0000","last_unstale":"2024-09-13T14:58:23.515708+0000","last_undegraded":"2024-09-13T14:58:23.515708+0000","last_fullsized":"2024-09-13T14:58:23.515708+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:37.924642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"261'100","reported_seq":285,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.864596+0000","last_change":"2024-09-13T14:45:35.947332+0000","last_active":"2024-09-13T14:58:34.864596+0000","last_peered":"2024-09-13T14:58:34.864596+0000","last_clean":"2024-09-13T14:58:34.864596+0000","last_became_active":"2024-09-13T14:45:19.810775+0000","last_became_peered":"2024-09-13T14:45:19.810775+0000","last_unstale":"2024-09-13T14:58:34.864596+0000","last_undegraded":"2024-09-13T14:58:34.864596+0000","last_fullsized":"2024-09-13T14:58:34.864596+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:35.947265+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:43:41.569795+0000","last_clean_scrub_stamp":"2024-09-13T14:45:35.947265+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:00.606370+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":51,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.519378+0000","last_change":"2024-09-13T14:55:41.901299+0000","last_active":"2024-09-13T14:58:23.519378+0000","last_peered":"2024-09-13T14:58:23.519378+0000","last_clean":"2024-09-13T14:58:23.519378+0000","last_became_active":"2024-09-13T14:55:41.866844+0000","last_became_peered":"2024-09-13T14:55:41.866844+0000","last_unstale":"2024-09-13T14:58:23.519378+0000","last_undegraded":"2024-09-13T14:58:23.519378+0000","last_fullsized":"2024-09-13T14:58:23.519378+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:44.106308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":55,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.516329+0000","last_change":"2024-09-13T14:54:58.830335+0000","last_active":"2024-09-13T14:58:23.516329+0000","last_peered":"2024-09-13T14:58:23.516329+0000","last_clean":"2024-09-13T14:58:23.516329+0000","last_became_active":"2024-09-13T14:54:58.829978+0000","last_became_peered":"2024-09-13T14:54:58.829978+0000","last_unstale":"2024-09-13T14:58:23.516329+0000","last_undegraded":"2024-09-13T14:58:23.516329+0000","last_fullsized":"2024-09-13T14:58:23.516329+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:46.608799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":1941,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.852988+0000","last_change":"2024-09-13T14:54:58.832660+0000","last_active":"2024-09-13T14:58:34.852988+0000","last_peered":"2024-09-13T14:58:34.852988+0000","last_clean":"2024-09-13T14:58:34.852988+0000","last_became_active":"2024-09-13T14:54:58.831770+0000","last_became_peered":"2024-09-13T14:54:58.831770+0000","last_unstale":"2024-09-13T14:58:34.852988+0000","last_undegraded":"2024-09-13T14:58:34.852988+0000","last_fullsized":"2024-09-13T14:58:34.852988+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:07:14.004187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1949,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.866141+0000","last_change":"2024-09-13T14:54:05.592484+0000","last_active":"2024-09-13T14:58:34.866141+0000","last_peered":"2024-09-13T14:58:34.866141+0000","last_clean":"2024-09-13T14:58:34.866141+0000","last_became_active":"2024-09-13T14:54:05.592159+0000","last_became_peered":"2024-09-13T14:54:05.592159+0000","last_unstale":"2024-09-13T14:58:34.866141+0000","last_undegraded":"2024-09-13T14:58:34.866141+0000","last_fullsized":"2024-09-13T14:58:34.866141+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:02.607037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"261'100","reported_seq":267,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.862642+0000","last_change":"2024-09-13T14:45:19.813412+0000","last_active":"2024-09-13T14:58:34.862642+0000","last_peered":"2024-09-13T14:58:34.862642+0000","last_clean":"2024-09-13T14:58:34.862642+0000","last_became_active":"2024-09-13T14:45:19.812585+0000","last_became_peered":"2024-09-13T14:45:19.812585+0000","last_unstale":"2024-09-13T14:58:34.862642+0000","last_undegraded":"2024-09-13T14:58:34.862642+0000","last_fullsized":"2024-09-13T14:58:34.862642+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.614340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.614340+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:32:07.434109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":55,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.516177+0000","last_change":"2024-09-13T14:54:58.831851+0000","last_active":"2024-09-13T14:58:23.516177+0000","last_peered":"2024-09-13T14:58:23.516177+0000","last_clean":"2024-09-13T14:58:23.516177+0000","last_became_active":"2024-09-13T14:54:58.831477+0000","last_became_peered":"2024-09-13T14:54:58.831477+0000","last_unstale":"2024-09-13T14:58:23.516177+0000","last_undegraded":"2024-09-13T14:58:23.516177+0000","last_fullsized":"2024-09-13T14:58:23.516177+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:42.562225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":302,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.864713+0000","last_change":"2024-09-13T14:54:58.827147+0000","last_active":"2024-09-13T14:58:34.864713+0000","last_peered":"2024-09-13T14:58:34.864713+0000","last_clean":"2024-09-13T14:58:34.864713+0000","last_became_active":"2024-09-13T14:54:58.826783+0000","last_became_peered":"2024-09-13T14:54:58.826783+0000","last_unstale":"2024-09-13T14:58:34.864713+0000","last_undegraded":"2024-09-13T14:58:34.864713+0000","last_fullsized":"2024-09-13T14:58:34.864713+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:40:27.819584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.18","version":"261'100","reported_seq":51,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.518687+0000","last_change":"2024-09-13T14:57:11.038947+0000","last_active":"2024-09-13T14:58:23.518687+0000","last_peered":"2024-09-13T14:58:23.518687+0000","last_clean":"2024-09-13T14:58:23.518687+0000","last_became_active":"2024-09-13T14:55:41.838842+0000","last_became_peered":"2024-09-13T14:55:41.838842+0000","last_unstale":"2024-09-13T14:58:23.518687+0000","last_undegraded":"2024-09-13T14:58:23.518687+0000","last_fullsized":"2024-09-13T14:58:23.518687+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1246,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:56.548855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:56.548855+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:52.715319+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":122,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.858173+0000","last_change":"2024-09-13T14:54:58.821234+0000","last_active":"2024-09-13T14:58:34.858173+0000","last_peered":"2024-09-13T14:58:34.858173+0000","last_clean":"2024-09-13T14:58:34.858173+0000","last_became_active":"2024-09-13T14:54:58.820948+0000","last_became_peered":"2024-09-13T14:54:58.820948+0000","last_unstale":"2024-09-13T14:58:34.858173+0000","last_undegraded":"2024-09-13T14:58:34.858173+0000","last_fullsized":"2024-09-13T14:58:34.858173+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:58:51.172714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1952,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.853565+0000","last_change":"2024-09-13T14:45:19.815341+0000","last_active":"2024-09-13T14:58:34.853565+0000","last_peered":"2024-09-13T14:58:34.853565+0000","last_clean":"2024-09-13T14:58:34.853565+0000","last_became_active":"2024-09-13T14:45:19.810262+0000","last_became_peered":"2024-09-13T14:45:19.810262+0000","last_unstale":"2024-09-13T14:58:34.853565+0000","last_undegraded":"2024-09-13T14:58:34.853565+0000","last_fullsized":"2024-09-13T14:58:34.853565+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:39.847653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1b","version":"261'100","reported_seq":47,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.519369+0000","last_change":"2024-09-13T14:55:41.919999+0000","last_active":"2024-09-13T14:58:23.519369+0000","last_peered":"2024-09-13T14:58:23.519369+0000","last_clean":"2024-09-13T14:58:23.519369+0000","last_became_active":"2024-09-13T14:55:41.919444+0000","last_became_peered":"2024-09-13T14:55:41.919444+0000","last_unstale":"2024-09-13T14:58:23.519369+0000","last_undegraded":"2024-09-13T14:58:23.519369+0000","last_fullsized":"2024-09-13T14:58:23.519369+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:55.988431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:55.988431+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:22:19.385914+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":281,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.862546+0000","last_change":"2024-09-13T14:54:05.606614+0000","last_active":"2024-09-13T14:58:34.862546+0000","last_peered":"2024-09-13T14:58:34.862546+0000","last_clean":"2024-09-13T14:58:34.862546+0000","last_became_active":"2024-09-13T14:54:05.605524+0000","last_became_peered":"2024-09-13T14:54:05.605524+0000","last_unstale":"2024-09-13T14:58:34.862546+0000","last_undegraded":"2024-09-13T14:58:34.862546+0000","last_fullsized":"2024-09-13T14:58:34.862546+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:59.484071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":47,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.519654+0000","last_change":"2024-09-13T14:55:41.883426+0000","last_active":"2024-09-13T14:58:23.519654+0000","last_peered":"2024-09-13T14:58:23.519654+0000","last_clean":"2024-09-13T14:58:23.519654+0000","last_became_active":"2024-09-13T14:55:41.880724+0000","last_became_peered":"2024-09-13T14:55:41.880724+0000","last_unstale":"2024-09-13T14:58:23.519654+0000","last_undegraded":"2024-09-13T14:58:23.519654+0000","last_fullsized":"2024-09-13T14:58:23.519654+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:17.920508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1a","version":"261'100","reported_seq":55,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.516039+0000","last_change":"2024-09-13T14:54:58.831623+0000","last_active":"2024-09-13T14:58:23.516039+0000","last_peered":"2024-09-13T14:58:23.516039+0000","last_clean":"2024-09-13T14:58:23.516039+0000","last_became_active":"2024-09-13T14:54:58.831026+0000","last_became_peered":"2024-09-13T14:54:58.831026+0000","last_unstale":"2024-09-13T14:58:23.516039+0000","last_undegraded":"2024-09-13T14:58:23.516039+0000","last_fullsized":"2024-09-13T14:58:23.516039+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:46.985510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:46.985510+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:27:24.529132+0000","scrub_duration":0.0030525999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":288,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.862637+0000","last_change":"2024-09-13T14:54:05.606882+0000","last_active":"2024-09-13T14:58:34.862637+0000","last_peered":"2024-09-13T14:58:34.862637+0000","last_clean":"2024-09-13T14:58:34.862637+0000","last_became_active":"2024-09-13T14:54:05.605912+0000","last_became_peered":"2024-09-13T14:54:05.605912+0000","last_unstale":"2024-09-13T14:58:34.862637+0000","last_undegraded":"2024-09-13T14:58:34.862637+0000","last_fullsized":"2024-09-13T14:58:34.862637+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:05.445329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1996,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.853071+0000","last_change":"2024-09-13T14:55:41.868797+0000","last_active":"2024-09-13T14:58:34.853071+0000","last_peered":"2024-09-13T14:58:34.853071+0000","last_clean":"2024-09-13T14:58:34.853071+0000","last_became_active":"2024-09-13T14:55:41.865613+0000","last_became_peered":"2024-09-13T14:55:41.865613+0000","last_unstale":"2024-09-13T14:58:34.853071+0000","last_undegraded":"2024-09-13T14:58:34.853071+0000","last_fullsized":"2024-09-13T14:58:34.853071+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:16:29.230176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"261'100","reported_seq":225,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.512661+0000","last_change":"2024-09-13T14:55:41.868936+0000","last_active":"2024-09-13T14:58:23.512661+0000","last_peered":"2024-09-13T14:58:23.512661+0000","last_clean":"2024-09-13T14:58:23.512661+0000","last_became_active":"2024-09-13T14:55:41.865505+0000","last_became_peered":"2024-09-13T14:55:41.865505+0000","last_unstale":"2024-09-13T14:58:23.512661+0000","last_undegraded":"2024-09-13T14:58:23.512661+0000","last_fullsized":"2024-09-13T14:58:23.512661+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:38.214865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:38.214865+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:45:08.788150+0000","scrub_duration":0.0016389639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1839,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.863219+0000","last_change":"2024-09-13T14:30:15.576921+0000","last_active":"2024-09-13T14:58:34.863219+0000","last_peered":"2024-09-13T14:58:34.863219+0000","last_clean":"2024-09-13T14:58:34.863219+0000","last_became_active":"2024-09-13T14:30:15.576607+0000","last_became_peered":"2024-09-13T14:30:15.576607+0000","last_unstale":"2024-09-13T14:58:34.863219+0000","last_undegraded":"2024-09-13T14:58:34.863219+0000","last_fullsized":"2024-09-13T14:58:34.863219+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:35.162460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":47,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.519503+0000","last_change":"2024-09-13T14:55:41.868620+0000","last_active":"2024-09-13T14:58:23.519503+0000","last_peered":"2024-09-13T14:58:23.519503+0000","last_clean":"2024-09-13T14:58:23.519503+0000","last_became_active":"2024-09-13T14:55:41.868289+0000","last_became_peered":"2024-09-13T14:55:41.868289+0000","last_unstale":"2024-09-13T14:58:23.519503+0000","last_undegraded":"2024-09-13T14:58:23.519503+0000","last_fullsized":"2024-09-13T14:58:23.519503+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:17:28.109694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"261'100","reported_seq":344,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.865938+0000","last_change":"2024-09-13T14:54:58.824599+0000","last_active":"2024-09-13T14:58:34.865938+0000","last_peered":"2024-09-13T14:58:34.865938+0000","last_clean":"2024-09-13T14:58:34.865938+0000","last_became_active":"2024-09-13T14:54:58.824304+0000","last_became_peered":"2024-09-13T14:54:58.824304+0000","last_unstale":"2024-09-13T14:58:34.865938+0000","last_undegraded":"2024-09-13T14:58:34.865938+0000","last_fullsized":"2024-09-13T14:58:34.865938+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.037517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.037517+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:42.296498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":136,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.866763+0000","last_change":"2024-09-13T14:54:58.836713+0000","last_active":"2024-09-13T14:58:34.866763+0000","last_peered":"2024-09-13T14:58:34.866763+0000","last_clean":"2024-09-13T14:58:34.866763+0000","last_became_active":"2024-09-13T14:54:58.827723+0000","last_became_peered":"2024-09-13T14:54:58.827723+0000","last_unstale":"2024-09-13T14:58:34.866763+0000","last_undegraded":"2024-09-13T14:58:34.866763+0000","last_fullsized":"2024-09-13T14:58:34.866763+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:10:25.743550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":115,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.859281+0000","last_change":"2024-09-13T14:54:05.614569+0000","last_active":"2024-09-13T14:58:34.859281+0000","last_peered":"2024-09-13T14:58:34.859281+0000","last_clean":"2024-09-13T14:58:34.859281+0000","last_became_active":"2024-09-13T14:54:05.611810+0000","last_became_peered":"2024-09-13T14:54:05.611810+0000","last_unstale":"2024-09-13T14:58:34.859281+0000","last_undegraded":"2024-09-13T14:58:34.859281+0000","last_fullsized":"2024-09-13T14:58:34.859281+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:46:02.721438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"261'100","reported_seq":47,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.519623+0000","last_change":"2024-09-13T14:55:41.860977+0000","last_active":"2024-09-13T14:58:23.519623+0000","last_peered":"2024-09-13T14:58:23.519623+0000","last_clean":"2024-09-13T14:58:23.519623+0000","last_became_active":"2024-09-13T14:55:41.859488+0000","last_became_peered":"2024-09-13T14:55:41.859488+0000","last_unstale":"2024-09-13T14:58:23.519623+0000","last_undegraded":"2024-09-13T14:58:23.519623+0000","last_fullsized":"2024-09-13T14:58:23.519623+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:54.933410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:54.933410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:48.738084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":65,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.516242+0000","last_change":"2024-09-13T14:54:58.838308+0000","last_active":"2024-09-13T14:58:23.516242+0000","last_peered":"2024-09-13T14:58:23.516242+0000","last_clean":"2024-09-13T14:58:23.516242+0000","last_became_active":"2024-09-13T14:54:58.825174+0000","last_became_peered":"2024-09-13T14:54:58.825174+0000","last_unstale":"2024-09-13T14:58:23.516242+0000","last_undegraded":"2024-09-13T14:58:23.516242+0000","last_fullsized":"2024-09-13T14:58:23.516242+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:59:20.787507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":55,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.516477+0000","last_change":"2024-09-13T14:54:58.831327+0000","last_active":"2024-09-13T14:58:23.516477+0000","last_peered":"2024-09-13T14:58:23.516477+0000","last_clean":"2024-09-13T14:58:23.516477+0000","last_became_active":"2024-09-13T14:54:58.830414+0000","last_became_peered":"2024-09-13T14:54:58.830414+0000","last_unstale":"2024-09-13T14:58:23.516477+0000","last_undegraded":"2024-09-13T14:58:23.516477+0000","last_fullsized":"2024-09-13T14:58:23.516477+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:12.501968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"261'100","reported_seq":115,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.859572+0000","last_change":"2024-09-13T14:54:05.613604+0000","last_active":"2024-09-13T14:58:34.859572+0000","last_peered":"2024-09-13T14:58:34.859572+0000","last_clean":"2024-09-13T14:58:34.859572+0000","last_became_active":"2024-09-13T14:54:05.613247+0000","last_became_peered":"2024-09-13T14:54:05.613247+0000","last_unstale":"2024-09-13T14:58:34.859572+0000","last_undegraded":"2024-09-13T14:58:34.859572+0000","last_fullsized":"2024-09-13T14:58:34.859572+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:48.363045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:48.363045+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:17.100985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.f","version":"1274'3353","reported_seq":3367,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:42.275476+0000","last_change":"2024-09-13T14:58:06.200417+0000","last_active":"2024-09-13T14:58:42.275476+0000","last_peered":"2024-09-13T14:58:42.275476+0000","last_clean":"2024-09-13T14:58:42.275476+0000","last_became_active":"2024-09-13T14:58:06.200062+0000","last_became_peered":"2024-09-13T14:58:06.200062+0000","last_unstale":"2024-09-13T14:58:42.275476+0000","last_undegraded":"2024-09-13T14:58:42.275476+0000","last_fullsized":"2024-09-13T14:58:42.275476+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":3353,"log_dups_size":0,"ondisk_log_size":3353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:53:26.095129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219742208,"num_objects":3353,"num_object_clones":0,"num_object_copies":10059,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3353,"num_write_kb":214592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":75,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.516090+0000","last_change":"2024-09-13T14:55:41.788238+0000","last_active":"2024-09-13T14:58:23.516090+0000","last_peered":"2024-09-13T14:58:23.516090+0000","last_clean":"2024-09-13T14:58:23.516090+0000","last_became_active":"2024-09-13T14:55:41.786422+0000","last_became_peered":"2024-09-13T14:55:41.786422+0000","last_unstale":"2024-09-13T14:58:23.516090+0000","last_undegraded":"2024-09-13T14:58:23.516090+0000","last_fullsized":"2024-09-13T14:58:23.516090+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:13:55.167916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":699,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.513216+0000","last_change":"2024-09-13T14:43:32.635070+0000","last_active":"2024-09-13T14:58:23.513216+0000","last_peered":"2024-09-13T14:58:23.513216+0000","last_clean":"2024-09-13T14:58:23.513216+0000","last_became_active":"2024-09-13T14:43:32.634734+0000","last_became_peered":"2024-09-13T14:43:32.634734+0000","last_unstale":"2024-09-13T14:58:23.513216+0000","last_undegraded":"2024-09-13T14:58:23.513216+0000","last_fullsized":"2024-09-13T14:58:23.513216+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:33.107010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":2013,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.853673+0000","last_change":"2024-09-13T14:54:58.846704+0000","last_active":"2024-09-13T14:58:34.853673+0000","last_peered":"2024-09-13T14:58:34.853673+0000","last_clean":"2024-09-13T14:58:34.853673+0000","last_became_active":"2024-09-13T14:54:58.830897+0000","last_became_peered":"2024-09-13T14:54:58.830897+0000","last_unstale":"2024-09-13T14:58:34.853673+0000","last_undegraded":"2024-09-13T14:58:34.853673+0000","last_fullsized":"2024-09-13T14:58:34.853673+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:31:37.389598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"261'100","reported_seq":296,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.863996+0000","last_change":"2024-09-13T14:55:41.845434+0000","last_active":"2024-09-13T14:58:34.863996+0000","last_peered":"2024-09-13T14:58:34.863996+0000","last_clean":"2024-09-13T14:58:34.863996+0000","last_became_active":"2024-09-13T14:55:41.845155+0000","last_became_peered":"2024-09-13T14:55:41.845155+0000","last_unstale":"2024-09-13T14:58:34.863996+0000","last_undegraded":"2024-09-13T14:58:34.863996+0000","last_fullsized":"2024-09-13T14:58:34.863996+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.710192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.710192+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:32.286774+0000","scrub_duration":0.0019250910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"1274'3293","reported_seq":3307,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:42.291129+0000","last_change":"2024-09-13T14:58:06.201145+0000","last_active":"2024-09-13T14:58:42.291129+0000","last_peered":"2024-09-13T14:58:42.291129+0000","last_clean":"2024-09-13T14:58:42.291129+0000","last_became_active":"2024-09-13T14:58:06.200769+0000","last_became_peered":"2024-09-13T14:58:06.200769+0000","last_unstale":"2024-09-13T14:58:42.291129+0000","last_undegraded":"2024-09-13T14:58:42.291129+0000","last_fullsized":"2024-09-13T14:58:42.291129+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":3293,"log_dups_size":0,"ondisk_log_size":3293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:18:09.369835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215810048,"num_objects":3293,"num_object_clones":0,"num_object_copies":9879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3293,"num_write_kb":210752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":170,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.862382+0000","last_change":"2024-09-13T14:52:54.233894+0000","last_active":"2024-09-13T14:58:34.862382+0000","last_peered":"2024-09-13T14:58:34.862382+0000","last_clean":"2024-09-13T14:58:34.862382+0000","last_became_active":"2024-09-13T14:52:54.233601+0000","last_became_peered":"2024-09-13T14:52:54.233601+0000","last_unstale":"2024-09-13T14:58:34.862382+0000","last_undegraded":"2024-09-13T14:58:34.862382+0000","last_fullsized":"2024-09-13T14:58:34.862382+0000","mapping_epoch":1195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:17:24.364701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":123,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.859665+0000","last_change":"2024-09-13T14:54:58.825887+0000","last_active":"2024-09-13T14:58:34.859665+0000","last_peered":"2024-09-13T14:58:34.859665+0000","last_clean":"2024-09-13T14:58:34.859665+0000","last_became_active":"2024-09-13T14:54:58.825350+0000","last_became_peered":"2024-09-13T14:54:58.825350+0000","last_unstale":"2024-09-13T14:58:34.859665+0000","last_undegraded":"2024-09-13T14:58:34.859665+0000","last_fullsized":"2024-09-13T14:58:34.859665+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:16.669031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":1872,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.864395+0000","last_change":"2024-09-13T14:30:15.581728+0000","last_active":"2024-09-13T14:58:34.864395+0000","last_peered":"2024-09-13T14:58:34.864395+0000","last_clean":"2024-09-13T14:58:34.864395+0000","last_became_active":"2024-09-13T14:30:15.581303+0000","last_became_peered":"2024-09-13T14:30:15.581303+0000","last_unstale":"2024-09-13T14:58:34.864395+0000","last_undegraded":"2024-09-13T14:58:34.864395+0000","last_fullsized":"2024-09-13T14:58:34.864395+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:10.027991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"261'100","reported_seq":55,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.516574+0000","last_change":"2024-09-13T14:54:58.822940+0000","last_active":"2024-09-13T14:58:23.516574+0000","last_peered":"2024-09-13T14:58:23.516574+0000","last_clean":"2024-09-13T14:58:23.516574+0000","last_became_active":"2024-09-13T14:54:58.822551+0000","last_became_peered":"2024-09-13T14:54:58.822551+0000","last_unstale":"2024-09-13T14:58:23.516574+0000","last_undegraded":"2024-09-13T14:58:23.516574+0000","last_fullsized":"2024-09-13T14:58:23.516574+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:51:31.310115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:51:31.310115+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:05:04.280014+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"1274'3432","reported_seq":3446,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:42.286254+0000","last_change":"2024-09-13T14:58:06.203698+0000","last_active":"2024-09-13T14:58:42.286254+0000","last_peered":"2024-09-13T14:58:42.286254+0000","last_clean":"2024-09-13T14:58:42.286254+0000","last_became_active":"2024-09-13T14:58:06.203402+0000","last_became_peered":"2024-09-13T14:58:06.203402+0000","last_unstale":"2024-09-13T14:58:42.286254+0000","last_undegraded":"2024-09-13T14:58:42.286254+0000","last_fullsized":"2024-09-13T14:58:42.286254+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":3432,"log_dups_size":0,"ondisk_log_size":3432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:59:27.300867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224919552,"num_objects":3432,"num_object_clones":0,"num_object_copies":10296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3432,"num_write_kb":219648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1947,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.852901+0000","last_change":"2024-09-13T14:55:41.914117+0000","last_active":"2024-09-13T14:58:34.852901+0000","last_peered":"2024-09-13T14:58:34.852901+0000","last_clean":"2024-09-13T14:58:34.852901+0000","last_became_active":"2024-09-13T14:55:41.913777+0000","last_became_peered":"2024-09-13T14:55:41.913777+0000","last_unstale":"2024-09-13T14:58:34.852901+0000","last_undegraded":"2024-09-13T14:58:34.852901+0000","last_fullsized":"2024-09-13T14:58:34.852901+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:40:31.224332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1280,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.512715+0000","last_change":"2024-09-13T14:54:58.825971+0000","last_active":"2024-09-13T14:58:23.512715+0000","last_peered":"2024-09-13T14:58:23.512715+0000","last_clean":"2024-09-13T14:58:23.512715+0000","last_became_active":"2024-09-13T14:54:58.822356+0000","last_became_peered":"2024-09-13T14:54:58.822356+0000","last_unstale":"2024-09-13T14:58:23.512715+0000","last_undegraded":"2024-09-13T14:58:23.512715+0000","last_fullsized":"2024-09-13T14:58:23.512715+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:14.163203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":130,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.859426+0000","last_change":"2024-09-13T14:55:41.892125+0000","last_active":"2024-09-13T14:58:34.859426+0000","last_peered":"2024-09-13T14:58:34.859426+0000","last_clean":"2024-09-13T14:58:34.859426+0000","last_became_active":"2024-09-13T14:55:41.891812+0000","last_became_peered":"2024-09-13T14:55:41.891812+0000","last_unstale":"2024-09-13T14:58:34.859426+0000","last_undegraded":"2024-09-13T14:58:34.859426+0000","last_fullsized":"2024-09-13T14:58:34.859426+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:03.359541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"261'100","reported_seq":638,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.853422+0000","last_change":"2024-09-13T14:45:19.815050+0000","last_active":"2024-09-13T14:58:34.853422+0000","last_peered":"2024-09-13T14:58:34.853422+0000","last_clean":"2024-09-13T14:58:34.853422+0000","last_became_active":"2024-09-13T14:45:19.809732+0000","last_became_peered":"2024-09-13T14:45:19.809732+0000","last_unstale":"2024-09-13T14:58:34.853422+0000","last_undegraded":"2024-09-13T14:58:34.853422+0000","last_fullsized":"2024-09-13T14:58:34.853422+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:33.574443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:33.574443+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:51:40.261036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.c","version":"1274'3317","reported_seq":3331,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:42.228751+0000","last_change":"2024-09-13T14:58:06.202247+0000","last_active":"2024-09-13T14:58:42.228751+0000","last_peered":"2024-09-13T14:58:42.228751+0000","last_clean":"2024-09-13T14:58:42.228751+0000","last_became_active":"2024-09-13T14:58:06.201537+0000","last_became_peered":"2024-09-13T14:58:06.201537+0000","last_unstale":"2024-09-13T14:58:42.228751+0000","last_undegraded":"2024-09-13T14:58:42.228751+0000","last_fullsized":"2024-09-13T14:58:42.228751+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":3317,"log_dups_size":0,"ondisk_log_size":3317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:45:06.849570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217382912,"num_objects":3317,"num_object_clones":0,"num_object_copies":9951,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3317,"num_write_kb":212288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":137,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.859610+0000","last_change":"2024-09-13T14:55:41.929999+0000","last_active":"2024-09-13T14:58:34.859610+0000","last_peered":"2024-09-13T14:58:34.859610+0000","last_clean":"2024-09-13T14:58:34.859610+0000","last_became_active":"2024-09-13T14:55:41.926393+0000","last_became_peered":"2024-09-13T14:55:41.926393+0000","last_unstale":"2024-09-13T14:58:34.859610+0000","last_undegraded":"2024-09-13T14:58:34.859610+0000","last_fullsized":"2024-09-13T14:58:34.859610+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:16:40.228234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":116,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.859336+0000","last_change":"2024-09-13T14:54:05.605111+0000","last_active":"2024-09-13T14:58:34.859336+0000","last_peered":"2024-09-13T14:58:34.859336+0000","last_clean":"2024-09-13T14:58:34.859336+0000","last_became_active":"2024-09-13T14:54:05.599852+0000","last_became_peered":"2024-09-13T14:54:05.599852+0000","last_unstale":"2024-09-13T14:58:34.859336+0000","last_undegraded":"2024-09-13T14:58:34.859336+0000","last_fullsized":"2024-09-13T14:58:34.859336+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:11.238519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":47,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.519315+0000","last_change":"2024-09-13T14:55:41.839293+0000","last_active":"2024-09-13T14:58:23.519315+0000","last_peered":"2024-09-13T14:58:23.519315+0000","last_clean":"2024-09-13T14:58:23.519315+0000","last_became_active":"2024-09-13T14:55:41.838820+0000","last_became_peered":"2024-09-13T14:55:41.838820+0000","last_unstale":"2024-09-13T14:58:23.519315+0000","last_undegraded":"2024-09-13T14:58:23.519315+0000","last_fullsized":"2024-09-13T14:58:23.519315+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:14:27.811437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"261'100","reported_seq":285,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.864202+0000","last_change":"2024-09-13T14:45:36.965992+0000","last_active":"2024-09-13T14:58:34.864202+0000","last_peered":"2024-09-13T14:58:34.864202+0000","last_clean":"2024-09-13T14:58:34.864202+0000","last_became_active":"2024-09-13T14:45:19.811238+0000","last_became_peered":"2024-09-13T14:45:19.811238+0000","last_unstale":"2024-09-13T14:58:34.864202+0000","last_undegraded":"2024-09-13T14:58:34.864202+0000","last_fullsized":"2024-09-13T14:58:34.864202+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:36.965775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:45:36.965775+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:36:01.623251+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"1274'3339","reported_seq":3353,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:42.289174+0000","last_change":"2024-09-13T14:58:06.203687+0000","last_active":"2024-09-13T14:58:42.289174+0000","last_peered":"2024-09-13T14:58:42.289174+0000","last_clean":"2024-09-13T14:58:42.289174+0000","last_became_active":"2024-09-13T14:58:06.203340+0000","last_became_peered":"2024-09-13T14:58:06.203340+0000","last_unstale":"2024-09-13T14:58:42.289174+0000","last_undegraded":"2024-09-13T14:58:42.289174+0000","last_fullsized":"2024-09-13T14:58:42.289174+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":3339,"log_dups_size":0,"ondisk_log_size":3339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:17:52.886190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218824704,"num_objects":3339,"num_object_clones":0,"num_object_copies":10017,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3339,"num_write_kb":213696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":116,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.859267+0000","last_change":"2024-09-13T14:54:05.596068+0000","last_active":"2024-09-13T14:58:34.859267+0000","last_peered":"2024-09-13T14:58:34.859267+0000","last_clean":"2024-09-13T14:58:34.859267+0000","last_became_active":"2024-09-13T14:54:05.595743+0000","last_became_peered":"2024-09-13T14:54:05.595743+0000","last_unstale":"2024-09-13T14:58:34.859267+0000","last_undegraded":"2024-09-13T14:58:34.859267+0000","last_fullsized":"2024-09-13T14:58:34.859267+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:15:39.660370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":55,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.516417+0000","last_change":"2024-09-13T14:54:58.831309+0000","last_active":"2024-09-13T14:58:23.516417+0000","last_peered":"2024-09-13T14:58:23.516417+0000","last_clean":"2024-09-13T14:58:23.516417+0000","last_became_active":"2024-09-13T14:54:58.830950+0000","last_became_peered":"2024-09-13T14:54:58.830950+0000","last_unstale":"2024-09-13T14:58:23.516417+0000","last_undegraded":"2024-09-13T14:58:23.516417+0000","last_fullsized":"2024-09-13T14:58:23.516417+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:16:54.838006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":96,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.519256+0000","last_change":"2024-09-13T14:55:41.889668+0000","last_active":"2024-09-13T14:58:23.519256+0000","last_peered":"2024-09-13T14:58:23.519256+0000","last_clean":"2024-09-13T14:58:23.519256+0000","last_became_active":"2024-09-13T14:55:41.889195+0000","last_became_peered":"2024-09-13T14:55:41.889195+0000","last_unstale":"2024-09-13T14:58:23.519256+0000","last_undegraded":"2024-09-13T14:58:23.519256+0000","last_fullsized":"2024-09-13T14:58:23.519256+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:52.765208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"261'100","reported_seq":47,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.518668+0000","last_change":"2024-09-13T14:55:41.897451+0000","last_active":"2024-09-13T14:58:23.518668+0000","last_peered":"2024-09-13T14:58:23.518668+0000","last_clean":"2024-09-13T14:58:23.518668+0000","last_became_active":"2024-09-13T14:55:41.896485+0000","last_became_peered":"2024-09-13T14:55:41.896485+0000","last_unstale":"2024-09-13T14:58:23.518668+0000","last_undegraded":"2024-09-13T14:58:23.518668+0000","last_fullsized":"2024-09-13T14:58:23.518668+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:57.024823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:57.024823+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:25:55.484738+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"1274'3399","reported_seq":3413,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:42.273164+0000","last_change":"2024-09-13T14:58:06.202180+0000","last_active":"2024-09-13T14:58:42.273164+0000","last_peered":"2024-09-13T14:58:42.273164+0000","last_clean":"2024-09-13T14:58:42.273164+0000","last_became_active":"2024-09-13T14:58:06.201886+0000","last_became_peered":"2024-09-13T14:58:06.201886+0000","last_unstale":"2024-09-13T14:58:42.273164+0000","last_undegraded":"2024-09-13T14:58:42.273164+0000","last_fullsized":"2024-09-13T14:58:42.273164+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":3399,"log_dups_size":0,"ondisk_log_size":3399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:48.418531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222756864,"num_objects":3399,"num_object_clones":0,"num_object_copies":10197,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3399,"num_write_kb":217536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":130,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.860584+0000","last_change":"2024-09-13T14:55:41.876729+0000","last_active":"2024-09-13T14:58:34.860584+0000","last_peered":"2024-09-13T14:58:34.860584+0000","last_clean":"2024-09-13T14:58:34.860584+0000","last_became_active":"2024-09-13T14:55:41.876280+0000","last_became_peered":"2024-09-13T14:55:41.876280+0000","last_unstale":"2024-09-13T14:58:34.860584+0000","last_undegraded":"2024-09-13T14:58:34.860584+0000","last_fullsized":"2024-09-13T14:58:34.860584+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:39:30.308792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":116,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.858147+0000","last_change":"2024-09-13T14:54:05.614107+0000","last_active":"2024-09-13T14:58:34.858147+0000","last_peered":"2024-09-13T14:58:34.858147+0000","last_clean":"2024-09-13T14:58:34.858147+0000","last_became_active":"2024-09-13T14:54:05.613429+0000","last_became_peered":"2024-09-13T14:54:05.613429+0000","last_unstale":"2024-09-13T14:58:34.858147+0000","last_undegraded":"2024-09-13T14:58:34.858147+0000","last_fullsized":"2024-09-13T14:58:34.858147+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:51.312413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1953,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.866064+0000","last_change":"2024-09-13T14:45:19.812713+0000","last_active":"2024-09-13T14:58:34.866064+0000","last_peered":"2024-09-13T14:58:34.866064+0000","last_clean":"2024-09-13T14:58:34.866064+0000","last_became_active":"2024-09-13T14:45:19.812203+0000","last_became_peered":"2024-09-13T14:45:19.812203+0000","last_unstale":"2024-09-13T14:58:34.866064+0000","last_undegraded":"2024-09-13T14:58:34.866064+0000","last_fullsized":"2024-09-13T14:58:34.866064+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:30:46.149347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"261'100","reported_seq":1414,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.512054+0000","last_change":"2024-09-13T14:54:58.820128+0000","last_active":"2024-09-13T14:58:23.512054+0000","last_peered":"2024-09-13T14:58:23.512054+0000","last_clean":"2024-09-13T14:58:23.512054+0000","last_became_active":"2024-09-13T14:54:58.819324+0000","last_became_peered":"2024-09-13T14:54:58.819324+0000","last_unstale":"2024-09-13T14:58:23.512054+0000","last_undegraded":"2024-09-13T14:58:23.512054+0000","last_fullsized":"2024-09-13T14:58:23.512054+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:42.577243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:42.577243+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:50:11.323977+0000","scrub_duration":0.001367839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"1274'3497","reported_seq":3519,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:43.606290+0000","last_change":"2024-09-13T14:58:06.203607+0000","last_active":"2024-09-13T14:58:43.606290+0000","last_peered":"2024-09-13T14:58:43.606290+0000","last_clean":"2024-09-13T14:58:43.606290+0000","last_became_active":"2024-09-13T14:58:06.203316+0000","last_became_peered":"2024-09-13T14:58:06.203316+0000","last_unstale":"2024-09-13T14:58:43.606290+0000","last_undegraded":"2024-09-13T14:58:43.606290+0000","last_fullsized":"2024-09-13T14:58:43.606290+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":3497,"log_dups_size":0,"ondisk_log_size":3497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:55.731243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229179392,"num_objects":3497,"num_object_clones":0,"num_object_copies":10491,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3497,"num_write_kb":223808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1961,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.853464+0000","last_change":"2024-09-13T14:54:05.607857+0000","last_active":"2024-09-13T14:58:34.853464+0000","last_peered":"2024-09-13T14:58:34.853464+0000","last_clean":"2024-09-13T14:58:34.853464+0000","last_became_active":"2024-09-13T14:54:05.607112+0000","last_became_peered":"2024-09-13T14:54:05.607112+0000","last_unstale":"2024-09-13T14:58:34.853464+0000","last_undegraded":"2024-09-13T14:58:34.853464+0000","last_fullsized":"2024-09-13T14:58:34.853464+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:41:23.733552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1998,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.864153+0000","last_change":"2024-09-13T14:54:58.832430+0000","last_active":"2024-09-13T14:58:34.864153+0000","last_peered":"2024-09-13T14:58:34.864153+0000","last_clean":"2024-09-13T14:58:34.864153+0000","last_became_active":"2024-09-13T14:54:58.832091+0000","last_became_peered":"2024-09-13T14:54:58.832091+0000","last_unstale":"2024-09-13T14:58:34.864153+0000","last_undegraded":"2024-09-13T14:58:34.864153+0000","last_fullsized":"2024-09-13T14:58:34.864153+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:50.845541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1297,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.512315+0000","last_change":"2024-09-13T14:54:05.604586+0000","last_active":"2024-09-13T14:58:23.512315+0000","last_peered":"2024-09-13T14:58:23.512315+0000","last_clean":"2024-09-13T14:58:23.512315+0000","last_became_active":"2024-09-13T14:54:05.603632+0000","last_became_peered":"2024-09-13T14:54:05.603632+0000","last_unstale":"2024-09-13T14:58:23.512315+0000","last_undegraded":"2024-09-13T14:58:23.512315+0000","last_fullsized":"2024-09-13T14:58:23.512315+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:33.197882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"261'100","reported_seq":123,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.861944+0000","last_change":"2024-09-13T14:54:58.828206+0000","last_active":"2024-09-13T14:58:34.861944+0000","last_peered":"2024-09-13T14:58:34.861944+0000","last_clean":"2024-09-13T14:58:34.861944+0000","last_became_active":"2024-09-13T14:54:58.827560+0000","last_became_peered":"2024-09-13T14:54:58.827560+0000","last_unstale":"2024-09-13T14:58:34.861944+0000","last_undegraded":"2024-09-13T14:58:34.861944+0000","last_fullsized":"2024-09-13T14:58:34.861944+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.008264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.008264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:23.090746+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"1274'3468","reported_seq":3489,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:44.174629+0000","last_change":"2024-09-13T14:58:06.200816+0000","last_active":"2024-09-13T14:58:44.174629+0000","last_peered":"2024-09-13T14:58:44.174629+0000","last_clean":"2024-09-13T14:58:44.174629+0000","last_became_active":"2024-09-13T14:58:06.200474+0000","last_became_peered":"2024-09-13T14:58:06.200474+0000","last_unstale":"2024-09-13T14:58:44.174629+0000","last_undegraded":"2024-09-13T14:58:44.174629+0000","last_fullsized":"2024-09-13T14:58:44.174629+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":3468,"log_dups_size":0,"ondisk_log_size":3468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:55.523165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227278848,"num_objects":3468,"num_object_clones":0,"num_object_copies":10404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3468,"num_write_kb":221952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":55,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.516571+0000","last_change":"2024-09-13T14:54:58.828591+0000","last_active":"2024-09-13T14:58:23.516571+0000","last_peered":"2024-09-13T14:58:23.516571+0000","last_clean":"2024-09-13T14:58:23.516571+0000","last_became_active":"2024-09-13T14:54:58.828212+0000","last_became_peered":"2024-09-13T14:54:58.828212+0000","last_unstale":"2024-09-13T14:58:23.516571+0000","last_undegraded":"2024-09-13T14:58:23.516571+0000","last_fullsized":"2024-09-13T14:58:23.516571+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:50:15.315121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":303,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.863013+0000","last_change":"2024-09-13T14:55:41.835887+0000","last_active":"2024-09-13T14:58:34.863013+0000","last_peered":"2024-09-13T14:58:34.863013+0000","last_clean":"2024-09-13T14:58:34.863013+0000","last_became_active":"2024-09-13T14:55:41.834413+0000","last_became_peered":"2024-09-13T14:55:41.834413+0000","last_unstale":"2024-09-13T14:58:34.863013+0000","last_undegraded":"2024-09-13T14:58:34.863013+0000","last_fullsized":"2024-09-13T14:58:34.863013+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:01.850578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":82,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.519445+0000","last_change":"2024-09-13T14:55:41.839256+0000","last_active":"2024-09-13T14:58:23.519445+0000","last_peered":"2024-09-13T14:58:23.519445+0000","last_clean":"2024-09-13T14:58:23.519445+0000","last_became_active":"2024-09-13T14:55:41.838918+0000","last_became_peered":"2024-09-13T14:55:41.838918+0000","last_unstale":"2024-09-13T14:58:23.519445+0000","last_undegraded":"2024-09-13T14:58:23.519445+0000","last_fullsized":"2024-09-13T14:58:23.519445+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:31:58.442849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"261'100","reported_seq":1385,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.512244+0000","last_change":"2024-09-13T14:44:28.186029+0000","last_active":"2024-09-13T14:58:23.512244+0000","last_peered":"2024-09-13T14:58:23.512244+0000","last_clean":"2024-09-13T14:58:23.512244+0000","last_became_active":"2024-09-13T14:42:14.927771+0000","last_became_peered":"2024-09-13T14:42:14.927771+0000","last_unstale":"2024-09-13T14:58:23.512244+0000","last_undegraded":"2024-09-13T14:58:23.512244+0000","last_fullsized":"2024-09-13T14:58:23.512244+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:28.185949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:28.185949+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:38.363961+0000","scrub_duration":0.0018071129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"1274'3378","reported_seq":3389,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:42.295301+0000","last_change":"2024-09-13T14:58:06.204014+0000","last_active":"2024-09-13T14:58:42.295301+0000","last_peered":"2024-09-13T14:58:42.295301+0000","last_clean":"2024-09-13T14:58:42.295301+0000","last_became_active":"2024-09-13T14:58:06.203193+0000","last_became_peered":"2024-09-13T14:58:06.203193+0000","last_unstale":"2024-09-13T14:58:42.295301+0000","last_undegraded":"2024-09-13T14:58:42.295301+0000","last_fullsized":"2024-09-13T14:58:42.295301+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":3378,"log_dups_size":0,"ondisk_log_size":3378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:05:05.742511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221380608,"num_objects":3378,"num_object_clones":0,"num_object_copies":10134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3378,"num_write_kb":216192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":123,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.860718+0000","last_change":"2024-09-13T14:54:58.829485+0000","last_active":"2024-09-13T14:58:34.860718+0000","last_peered":"2024-09-13T14:58:34.860718+0000","last_clean":"2024-09-13T14:58:34.860718+0000","last_became_active":"2024-09-13T14:54:58.829141+0000","last_became_peered":"2024-09-13T14:54:58.829141+0000","last_unstale":"2024-09-13T14:58:34.860718+0000","last_undegraded":"2024-09-13T14:58:34.860718+0000","last_fullsized":"2024-09-13T14:58:34.860718+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:10:05.474311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":289,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.862725+0000","last_change":"2024-09-13T14:54:05.609380+0000","last_active":"2024-09-13T14:58:34.862725+0000","last_peered":"2024-09-13T14:58:34.862725+0000","last_clean":"2024-09-13T14:58:34.862725+0000","last_became_active":"2024-09-13T14:54:05.606282+0000","last_became_peered":"2024-09-13T14:54:05.606282+0000","last_unstale":"2024-09-13T14:58:34.862725+0000","last_undegraded":"2024-09-13T14:58:34.862725+0000","last_fullsized":"2024-09-13T14:58:34.862725+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:24:34.588847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"261'100","reported_seq":47,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.519072+0000","last_change":"2024-09-13T14:55:41.784086+0000","last_active":"2024-09-13T14:58:23.519072+0000","last_peered":"2024-09-13T14:58:23.519072+0000","last_clean":"2024-09-13T14:58:23.519072+0000","last_became_active":"2024-09-13T14:55:41.783492+0000","last_became_peered":"2024-09-13T14:55:41.783492+0000","last_unstale":"2024-09-13T14:58:23.519072+0000","last_undegraded":"2024-09-13T14:58:23.519072+0000","last_fullsized":"2024-09-13T14:58:23.519072+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:49.280478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:49.280478+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:16:55.041094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"1274'3295","reported_seq":3309,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:42.268328+0000","last_change":"2024-09-13T14:58:06.200769+0000","last_active":"2024-09-13T14:58:42.268328+0000","last_peered":"2024-09-13T14:58:42.268328+0000","last_clean":"2024-09-13T14:58:42.268328+0000","last_became_active":"2024-09-13T14:58:06.200457+0000","last_became_peered":"2024-09-13T14:58:06.200457+0000","last_unstale":"2024-09-13T14:58:42.268328+0000","last_undegraded":"2024-09-13T14:58:42.268328+0000","last_fullsized":"2024-09-13T14:58:42.268328+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":3295,"log_dups_size":0,"ondisk_log_size":3295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:25:18.549925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215941120,"num_objects":3295,"num_object_clones":0,"num_object_copies":9885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3295,"num_write_kb":210880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":55,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.516829+0000","last_change":"2024-09-13T14:54:58.829851+0000","last_active":"2024-09-13T14:58:23.516829+0000","last_peered":"2024-09-13T14:58:23.516829+0000","last_clean":"2024-09-13T14:58:23.516829+0000","last_became_active":"2024-09-13T14:54:58.829565+0000","last_became_peered":"2024-09-13T14:54:58.829565+0000","last_unstale":"2024-09-13T14:58:23.516829+0000","last_undegraded":"2024-09-13T14:58:23.516829+0000","last_fullsized":"2024-09-13T14:58:23.516829+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:00:19.211398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":55,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.516364+0000","last_change":"2024-09-13T14:54:58.827628+0000","last_active":"2024-09-13T14:58:23.516364+0000","last_peered":"2024-09-13T14:58:23.516364+0000","last_clean":"2024-09-13T14:58:23.516364+0000","last_became_active":"2024-09-13T14:54:58.827371+0000","last_became_peered":"2024-09-13T14:54:58.827371+0000","last_unstale":"2024-09-13T14:58:23.516364+0000","last_undegraded":"2024-09-13T14:58:23.516364+0000","last_fullsized":"2024-09-13T14:58:23.516364+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:21:17.887314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"261'100","reported_seq":130,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.862070+0000","last_change":"2024-09-13T14:55:41.949447+0000","last_active":"2024-09-13T14:58:34.862070+0000","last_peered":"2024-09-13T14:58:34.862070+0000","last_clean":"2024-09-13T14:58:34.862070+0000","last_became_active":"2024-09-13T14:55:41.949062+0000","last_became_peered":"2024-09-13T14:55:41.949062+0000","last_unstale":"2024-09-13T14:58:34.862070+0000","last_undegraded":"2024-09-13T14:58:34.862070+0000","last_fullsized":"2024-09-13T14:58:34.862070+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:47.405845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:47.405845+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:41:06.083975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3","version":"1274'3334","reported_seq":3357,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:42.974309+0000","last_change":"2024-09-13T14:58:06.200017+0000","last_active":"2024-09-13T14:58:42.974309+0000","last_peered":"2024-09-13T14:58:42.974309+0000","last_clean":"2024-09-13T14:58:42.974309+0000","last_became_active":"2024-09-13T14:58:06.199665+0000","last_became_peered":"2024-09-13T14:58:06.199665+0000","last_unstale":"2024-09-13T14:58:42.974309+0000","last_undegraded":"2024-09-13T14:58:42.974309+0000","last_fullsized":"2024-09-13T14:58:42.974309+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":3334,"log_dups_size":0,"ondisk_log_size":3334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:09.742112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218497024,"num_objects":3334,"num_object_clones":0,"num_object_copies":10002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3334,"num_write_kb":213376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":55,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.516790+0000","last_change":"2024-09-13T14:54:58.832125+0000","last_active":"2024-09-13T14:58:23.516790+0000","last_peered":"2024-09-13T14:58:23.516790+0000","last_clean":"2024-09-13T14:58:23.516790+0000","last_became_active":"2024-09-13T14:54:58.831788+0000","last_became_peered":"2024-09-13T14:54:58.831788+0000","last_unstale":"2024-09-13T14:58:23.516790+0000","last_undegraded":"2024-09-13T14:58:23.516790+0000","last_fullsized":"2024-09-13T14:58:23.516790+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:40.774847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":55,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.515956+0000","last_change":"2024-09-13T14:54:58.831785+0000","last_active":"2024-09-13T14:58:23.515956+0000","last_peered":"2024-09-13T14:58:23.515956+0000","last_clean":"2024-09-13T14:58:23.515956+0000","last_became_active":"2024-09-13T14:54:58.831331+0000","last_became_peered":"2024-09-13T14:54:58.831331+0000","last_unstale":"2024-09-13T14:58:23.515956+0000","last_undegraded":"2024-09-13T14:58:23.515956+0000","last_fullsized":"2024-09-13T14:58:23.515956+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:45.394867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"261'100","reported_seq":47,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.518475+0000","last_change":"2024-09-13T14:55:41.844203+0000","last_active":"2024-09-13T14:58:23.518475+0000","last_peered":"2024-09-13T14:58:23.518475+0000","last_clean":"2024-09-13T14:58:23.518475+0000","last_became_active":"2024-09-13T14:55:41.843897+0000","last_became_peered":"2024-09-13T14:55:41.843897+0000","last_unstale":"2024-09-13T14:58:23.518475+0000","last_undegraded":"2024-09-13T14:58:23.518475+0000","last_fullsized":"2024-09-13T14:58:23.518475+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:55.816857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:55.816857+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:42:54.571599+0000","scrub_duration":270,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2","version":"1274'3592","reported_seq":3615,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:44.170174+0000","last_change":"2024-09-13T14:58:06.201242+0000","last_active":"2024-09-13T14:58:44.170174+0000","last_peered":"2024-09-13T14:58:44.170174+0000","last_clean":"2024-09-13T14:58:44.170174+0000","last_became_active":"2024-09-13T14:58:06.200965+0000","last_became_peered":"2024-09-13T14:58:06.200965+0000","last_unstale":"2024-09-13T14:58:44.170174+0000","last_undegraded":"2024-09-13T14:58:44.170174+0000","last_fullsized":"2024-09-13T14:58:44.170174+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":3592,"log_dups_size":0,"ondisk_log_size":3592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:13:13.810742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235405312,"num_objects":3592,"num_object_clones":0,"num_object_copies":10776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3592,"num_write_kb":229888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":55,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.516935+0000","last_change":"2024-09-13T14:54:58.827402+0000","last_active":"2024-09-13T14:58:23.516935+0000","last_peered":"2024-09-13T14:58:23.516935+0000","last_clean":"2024-09-13T14:58:23.516935+0000","last_became_active":"2024-09-13T14:54:58.827119+0000","last_became_peered":"2024-09-13T14:54:58.827119+0000","last_unstale":"2024-09-13T14:58:23.516935+0000","last_undegraded":"2024-09-13T14:58:23.516935+0000","last_fullsized":"2024-09-13T14:58:23.516935+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:06.078013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":315,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.863451+0000","last_change":"2024-09-13T14:54:58.843575+0000","last_active":"2024-09-13T14:58:34.863451+0000","last_peered":"2024-09-13T14:58:34.863451+0000","last_clean":"2024-09-13T14:58:34.863451+0000","last_became_active":"2024-09-13T14:54:58.822033+0000","last_became_peered":"2024-09-13T14:54:58.822033+0000","last_unstale":"2024-09-13T14:58:34.863451+0000","last_undegraded":"2024-09-13T14:58:34.863451+0000","last_fullsized":"2024-09-13T14:58:34.863451+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:44.356090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"261'100","reported_seq":1404,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.512090+0000","last_change":"2024-09-13T14:45:19.798553+0000","last_active":"2024-09-13T14:58:23.512090+0000","last_peered":"2024-09-13T14:58:23.512090+0000","last_clean":"2024-09-13T14:58:23.512090+0000","last_became_active":"2024-09-13T14:45:19.798189+0000","last_became_peered":"2024-09-13T14:45:19.798189+0000","last_unstale":"2024-09-13T14:58:23.512090+0000","last_undegraded":"2024-09-13T14:58:23.512090+0000","last_fullsized":"2024-09-13T14:58:23.512090+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:29.169325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:29.169325+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:10.412666+0000","scrub_duration":0.0019358890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"1274'3447","reported_seq":3469,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:42.987958+0000","last_change":"2024-09-13T14:58:06.201392+0000","last_active":"2024-09-13T14:58:42.987958+0000","last_peered":"2024-09-13T14:58:42.987958+0000","last_clean":"2024-09-13T14:58:42.987958+0000","last_became_active":"2024-09-13T14:58:06.201088+0000","last_became_peered":"2024-09-13T14:58:06.201088+0000","last_unstale":"2024-09-13T14:58:42.987958+0000","last_undegraded":"2024-09-13T14:58:42.987958+0000","last_fullsized":"2024-09-13T14:58:42.987958+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":3447,"log_dups_size":0,"ondisk_log_size":3447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:44:35.707112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225902592,"num_objects":3447,"num_object_clones":0,"num_object_copies":10341,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3447,"num_write_kb":220608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1971,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.852805+0000","last_change":"2024-09-13T14:54:58.834208+0000","last_active":"2024-09-13T14:58:34.852805+0000","last_peered":"2024-09-13T14:58:34.852805+0000","last_clean":"2024-09-13T14:58:34.852805+0000","last_became_active":"2024-09-13T14:54:58.833071+0000","last_became_peered":"2024-09-13T14:54:58.833071+0000","last_unstale":"2024-09-13T14:58:34.852805+0000","last_undegraded":"2024-09-13T14:58:34.852805+0000","last_fullsized":"2024-09-13T14:58:34.852805+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:25:27.099352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":47,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.519236+0000","last_change":"2024-09-13T14:55:41.848811+0000","last_active":"2024-09-13T14:58:23.519236+0000","last_peered":"2024-09-13T14:58:23.519236+0000","last_clean":"2024-09-13T14:58:23.519236+0000","last_became_active":"2024-09-13T14:55:41.848542+0000","last_became_peered":"2024-09-13T14:55:41.848542+0000","last_unstale":"2024-09-13T14:58:23.519236+0000","last_undegraded":"2024-09-13T14:58:23.519236+0000","last_fullsized":"2024-09-13T14:58:23.519236+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:48.534022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"261'100","reported_seq":65,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.516787+0000","last_change":"2024-09-13T14:57:59.015556+0000","last_active":"2024-09-13T14:58:23.516787+0000","last_peered":"2024-09-13T14:58:23.516787+0000","last_clean":"2024-09-13T14:58:23.516787+0000","last_became_active":"2024-09-13T14:54:58.826820+0000","last_became_peered":"2024-09-13T14:54:58.826820+0000","last_unstale":"2024-09-13T14:58:23.516787+0000","last_undegraded":"2024-09-13T14:58:23.516787+0000","last_fullsized":"2024-09-13T14:58:23.516787+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:57:59.015484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:57:59.015484+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:49.345864+0000","scrub_duration":0.0015442190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"1274'3501","reported_seq":3524,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:43.590481+0000","last_change":"2024-09-13T14:58:06.202476+0000","last_active":"2024-09-13T14:58:43.590481+0000","last_peered":"2024-09-13T14:58:43.590481+0000","last_clean":"2024-09-13T14:58:43.590481+0000","last_became_active":"2024-09-13T14:58:06.202211+0000","last_became_peered":"2024-09-13T14:58:06.202211+0000","last_unstale":"2024-09-13T14:58:43.590481+0000","last_undegraded":"2024-09-13T14:58:43.590481+0000","last_fullsized":"2024-09-13T14:58:43.590481+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":3501,"log_dups_size":0,"ondisk_log_size":3501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:06:50.307820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229441536,"num_objects":3501,"num_object_clones":0,"num_object_copies":10503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3501,"num_write_kb":224064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1270,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.520148+0000","last_change":"2024-09-13T14:58:05.193719+0000","last_active":"2024-09-13T14:58:23.520148+0000","last_peered":"2024-09-13T14:58:23.520148+0000","last_clean":"2024-09-13T14:58:23.520148+0000","last_became_active":"2024-09-13T14:58:05.193367+0000","last_became_peered":"2024-09-13T14:58:05.193367+0000","last_unstale":"2024-09-13T14:58:23.520148+0000","last_undegraded":"2024-09-13T14:58:23.520148+0000","last_fullsized":"2024-09-13T14:58:23.520148+0000","mapping_epoch":1267,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:38:46.329171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":1268,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.512645+0000","last_change":"2024-09-13T14:55:41.919407+0000","last_active":"2024-09-13T14:58:23.512645+0000","last_peered":"2024-09-13T14:58:23.512645+0000","last_clean":"2024-09-13T14:58:23.512645+0000","last_became_active":"2024-09-13T14:55:41.793557+0000","last_became_peered":"2024-09-13T14:55:41.793557+0000","last_unstale":"2024-09-13T14:58:23.512645+0000","last_undegraded":"2024-09-13T14:58:23.512645+0000","last_fullsized":"2024-09-13T14:58:23.512645+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:57:51.874930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"261'100","reported_seq":47,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.519694+0000","last_change":"2024-09-13T14:55:41.785240+0000","last_active":"2024-09-13T14:58:23.519694+0000","last_peered":"2024-09-13T14:58:23.519694+0000","last_clean":"2024-09-13T14:58:23.519694+0000","last_became_active":"2024-09-13T14:55:41.784916+0000","last_became_peered":"2024-09-13T14:55:41.784916+0000","last_unstale":"2024-09-13T14:58:23.519694+0000","last_undegraded":"2024-09-13T14:58:23.519694+0000","last_fullsized":"2024-09-13T14:58:23.519694+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:49:21.722939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.722939+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:57.374891+0000","scrub_duration":0.0016642790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"1274'3158","reported_seq":3181,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:40.948324+0000","last_change":"2024-09-13T14:58:06.199838+0000","last_active":"2024-09-13T14:58:40.948324+0000","last_peered":"2024-09-13T14:58:40.948324+0000","last_clean":"2024-09-13T14:58:40.948324+0000","last_became_active":"2024-09-13T14:58:06.199417+0000","last_became_peered":"2024-09-13T14:58:06.199417+0000","last_unstale":"2024-09-13T14:58:40.948324+0000","last_undegraded":"2024-09-13T14:58:40.948324+0000","last_fullsized":"2024-09-13T14:58:40.948324+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":3158,"log_dups_size":0,"ondisk_log_size":3158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:14:49.667560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206962688,"num_objects":3158,"num_object_clones":0,"num_object_copies":9474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3158,"num_write_kb":202112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":65,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.516290+0000","last_change":"2024-09-13T14:58:05.197375+0000","last_active":"2024-09-13T14:58:23.516290+0000","last_peered":"2024-09-13T14:58:23.516290+0000","last_clean":"2024-09-13T14:58:23.516290+0000","last_became_active":"2024-09-13T14:58:05.197085+0000","last_became_peered":"2024-09-13T14:58:05.197085+0000","last_unstale":"2024-09-13T14:58:23.516290+0000","last_undegraded":"2024-09-13T14:58:23.516290+0000","last_fullsized":"2024-09-13T14:58:23.516290+0000","mapping_epoch":1267,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:09:33.979543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":1328,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.512982+0000","last_change":"2024-09-13T14:54:58.841801+0000","last_active":"2024-09-13T14:58:23.512982+0000","last_peered":"2024-09-13T14:58:23.512982+0000","last_clean":"2024-09-13T14:58:23.512982+0000","last_became_active":"2024-09-13T14:54:58.820336+0000","last_became_peered":"2024-09-13T14:54:58.820336+0000","last_unstale":"2024-09-13T14:58:23.512982+0000","last_undegraded":"2024-09-13T14:58:23.512982+0000","last_fullsized":"2024-09-13T14:58:23.512982+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:19:59.656874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"261'100","reported_seq":271,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.863845+0000","last_change":"2024-09-13T14:45:19.821649+0000","last_active":"2024-09-13T14:58:34.863845+0000","last_peered":"2024-09-13T14:58:34.863845+0000","last_clean":"2024-09-13T14:58:34.863845+0000","last_became_active":"2024-09-13T14:45:19.812346+0000","last_became_peered":"2024-09-13T14:45:19.812346+0000","last_unstale":"2024-09-13T14:58:34.863845+0000","last_undegraded":"2024-09-13T14:58:34.863845+0000","last_fullsized":"2024-09-13T14:58:34.863845+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.597568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.597568+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:45.218892+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"1274'3339","reported_seq":3353,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:42.285650+0000","last_change":"2024-09-13T14:58:06.203821+0000","last_active":"2024-09-13T14:58:42.285650+0000","last_peered":"2024-09-13T14:58:42.285650+0000","last_clean":"2024-09-13T14:58:42.285650+0000","last_became_active":"2024-09-13T14:58:06.203460+0000","last_became_peered":"2024-09-13T14:58:06.203460+0000","last_unstale":"2024-09-13T14:58:42.285650+0000","last_undegraded":"2024-09-13T14:58:42.285650+0000","last_fullsized":"2024-09-13T14:58:42.285650+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":3339,"log_dups_size":0,"ondisk_log_size":3339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:46:54.957926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218824704,"num_objects":3339,"num_object_clones":0,"num_object_copies":10017,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3339,"num_write_kb":213696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":47,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.518615+0000","last_change":"2024-09-13T14:55:41.787477+0000","last_active":"2024-09-13T14:58:23.518615+0000","last_peered":"2024-09-13T14:58:23.518615+0000","last_clean":"2024-09-13T14:58:23.518615+0000","last_became_active":"2024-09-13T14:55:41.786674+0000","last_became_peered":"2024-09-13T14:55:41.786674+0000","last_unstale":"2024-09-13T14:58:23.518615+0000","last_undegraded":"2024-09-13T14:58:23.518615+0000","last_fullsized":"2024-09-13T14:58:23.518615+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:56.263938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":1299,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.513143+0000","last_change":"2024-09-13T14:54:58.833271+0000","last_active":"2024-09-13T14:58:23.513143+0000","last_peered":"2024-09-13T14:58:23.513143+0000","last_clean":"2024-09-13T14:58:23.513143+0000","last_became_active":"2024-09-13T14:54:58.819611+0000","last_became_peered":"2024-09-13T14:54:58.819611+0000","last_unstale":"2024-09-13T14:58:23.513143+0000","last_undegraded":"2024-09-13T14:58:23.513143+0000","last_fullsized":"2024-09-13T14:58:23.513143+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:52:37.530890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"261'100","reported_seq":205,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.512562+0000","last_change":"2024-09-13T14:54:58.826831+0000","last_active":"2024-09-13T14:58:23.512562+0000","last_peered":"2024-09-13T14:58:23.512562+0000","last_clean":"2024-09-13T14:58:23.512562+0000","last_became_active":"2024-09-13T14:54:58.826370+0000","last_became_peered":"2024-09-13T14:54:58.826370+0000","last_unstale":"2024-09-13T14:58:23.512562+0000","last_undegraded":"2024-09-13T14:58:23.512562+0000","last_fullsized":"2024-09-13T14:58:23.512562+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.191326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.191326+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:09:06.771779+0000","scrub_duration":0.0025317360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1299,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.520110+0000","last_change":"2024-09-13T14:55:41.925458+0000","last_active":"2024-09-13T14:58:23.520110+0000","last_peered":"2024-09-13T14:58:23.520110+0000","last_clean":"2024-09-13T14:58:23.520110+0000","last_became_active":"2024-09-13T14:55:41.859331+0000","last_became_peered":"2024-09-13T14:55:41.859331+0000","last_unstale":"2024-09-13T14:58:23.520110+0000","last_undegraded":"2024-09-13T14:58:23.520110+0000","last_fullsized":"2024-09-13T14:58:23.520110+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:19:20.962759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":281,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.864118+0000","last_change":"2024-09-13T14:54:05.604259+0000","last_active":"2024-09-13T14:58:34.864118+0000","last_peered":"2024-09-13T14:58:34.864118+0000","last_clean":"2024-09-13T14:58:34.864118+0000","last_became_active":"2024-09-13T14:54:05.603890+0000","last_became_peered":"2024-09-13T14:54:05.603890+0000","last_unstale":"2024-09-13T14:58:34.864118+0000","last_undegraded":"2024-09-13T14:58:34.864118+0000","last_fullsized":"2024-09-13T14:58:34.864118+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:14.847855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"261'100","reported_seq":47,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.519176+0000","last_change":"2024-09-13T14:55:41.889497+0000","last_active":"2024-09-13T14:58:23.519176+0000","last_peered":"2024-09-13T14:58:23.519176+0000","last_clean":"2024-09-13T14:58:23.519176+0000","last_became_active":"2024-09-13T14:55:41.888766+0000","last_became_peered":"2024-09-13T14:55:41.888766+0000","last_unstale":"2024-09-13T14:58:23.519176+0000","last_undegraded":"2024-09-13T14:58:23.519176+0000","last_fullsized":"2024-09-13T14:58:23.519176+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:55.378404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:55.378404+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:50.289550+0000","scrub_duration":0.0016008439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1231,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.512174+0000","last_change":"2024-09-13T14:54:58.825419+0000","last_active":"2024-09-13T14:58:23.512174+0000","last_peered":"2024-09-13T14:58:23.512174+0000","last_clean":"2024-09-13T14:58:23.512174+0000","last_became_active":"2024-09-13T14:54:58.820338+0000","last_became_peered":"2024-09-13T14:54:58.820338+0000","last_unstale":"2024-09-13T14:58:23.512174+0000","last_undegraded":"2024-09-13T14:58:23.512174+0000","last_fullsized":"2024-09-13T14:58:23.512174+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:09.298279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1935,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.853603+0000","last_change":"2024-09-13T14:45:19.813876+0000","last_active":"2024-09-13T14:58:34.853603+0000","last_peered":"2024-09-13T14:58:34.853603+0000","last_clean":"2024-09-13T14:58:34.853603+0000","last_became_active":"2024-09-13T14:45:19.809136+0000","last_became_peered":"2024-09-13T14:45:19.809136+0000","last_unstale":"2024-09-13T14:58:34.853603+0000","last_undegraded":"2024-09-13T14:58:34.853603+0000","last_fullsized":"2024-09-13T14:58:34.853603+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:50.013528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"261'100","reported_seq":122,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.861740+0000","last_change":"2024-09-13T14:54:58.828578+0000","last_active":"2024-09-13T14:58:34.861740+0000","last_peered":"2024-09-13T14:58:34.861740+0000","last_clean":"2024-09-13T14:58:34.861740+0000","last_became_active":"2024-09-13T14:54:58.828004+0000","last_became_peered":"2024-09-13T14:54:58.828004+0000","last_unstale":"2024-09-13T14:58:34.861740+0000","last_undegraded":"2024-09-13T14:58:34.861740+0000","last_fullsized":"2024-09-13T14:58:34.861740+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:19.978464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:19.978464+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:52:31.236171+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1982,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.858682+0000","last_change":"2024-09-13T14:45:19.809166+0000","last_active":"2024-09-13T14:58:34.858682+0000","last_peered":"2024-09-13T14:58:34.858682+0000","last_clean":"2024-09-13T14:58:34.858682+0000","last_became_active":"2024-09-13T14:45:19.808631+0000","last_became_peered":"2024-09-13T14:45:19.808631+0000","last_unstale":"2024-09-13T14:58:34.858682+0000","last_undegraded":"2024-09-13T14:58:34.858682+0000","last_fullsized":"2024-09-13T14:58:34.858682+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:38.822245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":47,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.519449+0000","last_change":"2024-09-13T14:55:41.835069+0000","last_active":"2024-09-13T14:58:23.519449+0000","last_peered":"2024-09-13T14:58:23.519449+0000","last_clean":"2024-09-13T14:58:23.519449+0000","last_became_active":"2024-09-13T14:55:41.834766+0000","last_became_peered":"2024-09-13T14:55:41.834766+0000","last_unstale":"2024-09-13T14:58:23.519449+0000","last_undegraded":"2024-09-13T14:58:23.519449+0000","last_fullsized":"2024-09-13T14:58:23.519449+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:02:16.216617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"261'100","reported_seq":47,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.518548+0000","last_change":"2024-09-13T14:55:41.787716+0000","last_active":"2024-09-13T14:58:23.518548+0000","last_peered":"2024-09-13T14:58:23.518548+0000","last_clean":"2024-09-13T14:58:23.518548+0000","last_became_active":"2024-09-13T14:55:41.787407+0000","last_became_peered":"2024-09-13T14:55:41.787407+0000","last_unstale":"2024-09-13T14:58:23.518548+0000","last_undegraded":"2024-09-13T14:58:23.518548+0000","last_fullsized":"2024-09-13T14:58:23.518548+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:50.327393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:50.327393+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:43:30.234863+0000","scrub_duration":0.0011508309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":1262,"reported_epoch":1273,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.512478+0000","last_change":"2024-09-13T14:54:58.845372+0000","last_active":"2024-09-13T14:58:23.512478+0000","last_peered":"2024-09-13T14:58:23.512478+0000","last_clean":"2024-09-13T14:58:23.512478+0000","last_became_active":"2024-09-13T14:54:58.822960+0000","last_became_peered":"2024-09-13T14:54:58.822960+0000","last_unstale":"2024-09-13T14:58:23.512478+0000","last_undegraded":"2024-09-13T14:58:23.512478+0000","last_fullsized":"2024-09-13T14:58:23.512478+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:19:30.257102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1888,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.864094+0000","last_change":"2024-09-13T14:54:05.605595+0000","last_active":"2024-09-13T14:58:34.864094+0000","last_peered":"2024-09-13T14:58:34.864094+0000","last_clean":"2024-09-13T14:58:34.864094+0000","last_became_active":"2024-09-13T14:54:05.604916+0000","last_became_peered":"2024-09-13T14:54:05.604916+0000","last_unstale":"2024-09-13T14:58:34.864094+0000","last_undegraded":"2024-09-13T14:58:34.864094+0000","last_fullsized":"2024-09-13T14:58:34.864094+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:06:22.358902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"261'100","reported_seq":678,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.857565+0000","last_change":"2024-09-13T14:55:41.878403+0000","last_active":"2024-09-13T14:58:34.857565+0000","last_peered":"2024-09-13T14:58:34.857565+0000","last_clean":"2024-09-13T14:58:34.857565+0000","last_became_active":"2024-09-13T14:55:41.878071+0000","last_became_peered":"2024-09-13T14:55:41.878071+0000","last_unstale":"2024-09-13T14:58:34.857565+0000","last_undegraded":"2024-09-13T14:58:34.857565+0000","last_fullsized":"2024-09-13T14:58:34.857565+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:14.654603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:14.654603+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:09:56.313293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":3548250112,"num_objects":54142,"num_object_clones":0,"num_object_copies":162426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54142,"num_write_kb":3465088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10624237568,"data_stored":10624237568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54142,"ondisk_log_size":54142,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":7662,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":853056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430704,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1753260,"kb_used_data":854180,"kb_used_omap":1,"kb_used_meta":899070,"kb_avail":91991892,"statfs":{"total":95995035648,"available":94199697408,"internally_reserved":0,"allocated":874680320,"data_stored":872027948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":920648024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626933,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2389540,"kb_used_data":1495132,"kb_used_omap":1,"kb_used_meta":894398,"kb_avail":91355612,"statfs":{"total":95995035648,"available":93548146688,"internally_reserved":0,"allocated":1531015168,"data_stored":1528370407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":915863896},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823162,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1976992,"kb_used_data":1073304,"kb_used_omap":1,"kb_used_meta":903678,"kb_avail":91768160,"statfs":{"total":95995035648,"available":93970595840,"internally_reserved":0,"allocated":1099063296,"data_stored":1096419001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":925366616},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019390,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2600936,"kb_used_data":1713568,"kb_used_omap":1,"kb_used_meta":887358,"kb_avail":91144216,"statfs":{"total":95995035648,"available":93331677184,"internally_reserved":0,"allocated":1754693632,"data_stored":1752044832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":908654936},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059593047,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1650132,"kb_used_data":1006412,"kb_used_omap":26,"kb_used_meta":643685,"kb_avail":92095020,"statfs":{"total":95995035648,"available":94305300480,"internally_reserved":0,"allocated":1030565888,"data_stored":1027932780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":659133600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":591,"seq":2538325672286,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2041376,"kb_used_data":1315352,"kb_used_omap":41,"kb_used_meta":725974,"kb_avail":91703776,"statfs":{"total":95995035648,"available":93904666624,"internally_reserved":0,"allocated":1346920448,"data_stored":1344276200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42651,"internal_metadata":743397733},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":557,"seq":2392296784229,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1446100,"kb_used_data":676876,"kb_used_omap":32,"kb_used_meta":769183,"kb_avail":92299052,"statfs":{"total":95995035648,"available":94514229248,"internally_reserved":0,"allocated":693121024,"data_stored":690487845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33752,"internal_metadata":787643432},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":524,"seq":2250562863468,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2848664,"kb_used_data":2180112,"kb_used_omap":41,"kb_used_meta":668502,"kb_avail":90896488,"statfs":{"total":95995035648,"available":93078003712,"internally_reserved":0,"allocated":2232434688,"data_stored":2229797986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42694,"internal_metadata":684546362},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2292383744,"data_stored":2292383744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":690749440,"data_stored":690749440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1339359232,"data_stored":1339359232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1045889024,"data_stored":1045889024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1757151232,"data_stored":1757151232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1091633152,"data_stored":1091633152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1526005760,"data_stored":1526005760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":881065984,"data_stored":881065984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:58:53.449 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:58:53.525 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:53.555 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:53.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:53.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:53.657 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:53.692 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:53.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:53.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:53.797 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:53.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:53.833 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:53.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:53.921 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:53.934 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:53.962 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:53.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:53.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:54.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:54.110 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:54.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:54.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:54.167 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:58:54.192 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 37 16 70459 70443 118.972 84.1875 0.0266822 0.00839964 2024-09-13T14:58:54.212 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:54.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:54.237 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:54.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:54.338 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:54.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:54.367 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:54.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:54.468 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:54.501 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:54.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:54.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:54.603 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:54.643 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:54.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:54.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:54.745 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:54.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:54.782 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:54.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:54.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:58:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:58:54] "GET /metrics HTTP/1.1" 200 36701 "" "Prometheus/2.51.0" 2024-09-13T14:58:54.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:54.919 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:54.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:54.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:55.021 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:55.056 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:55.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:55.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:55.158 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:55.192 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 38 16 71719 71703 117.913 78.75 0.0105009 0.00847492 2024-09-13T14:58:55.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:55.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:55.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:55.227 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:58:55.227 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-13T14:58:55.227 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-13T14:58:55.227 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-13T14:58:55.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.228 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:179787 8.b 8:dc70ed76:::benchmark_data_smithi151_229097_object54919:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1189)", 2024-09-13T14:58:55.228 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:52:11.559158+0000", 2024-09-13T14:58:55.228 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 403.66376492199998, 2024-09-13T14:58:55.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.012651425, 2024-09-13T14:58:55.228 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:58:55.228 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:58:55.228 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:55.228 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:58:55.228 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T14:58:55.228 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:58:55.228 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 179787 2024-09-13T14:58:55.228 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.228 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:58:55.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.229 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:58:55.229 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.559158+0000", 2024-09-13T14:58:55.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.229 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.229 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:58:55.229 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.559158+0000", 2024-09-13T14:58:55.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.229 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.229 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:58:55.229 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.559159+0000", 2024-09-13T14:58:55.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.161e-06 2024-09-13T14:58:55.230 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.230 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.230 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:58:55.230 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.561118+0000", 2024-09-13T14:58:55.230 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019583280000000001 2024-09-13T14:58:55.230 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.230 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.230 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:58:55.230 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.561132+0000", 2024-09-13T14:58:55.230 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4139e-05 2024-09-13T14:58:55.230 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.230 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.230 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:58:55.230 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.561140+0000", 2024-09-13T14:58:55.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8699999999999992e-06 2024-09-13T14:58:55.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.561251+0000", 2024-09-13T14:58:55.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000111113 2024-09-13T14:58:55.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:58:55.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.561269+0000", 2024-09-13T14:58:55.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7951000000000001e-05 2024-09-13T14:58:55.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.232 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.562339+0000", 2024-09-13T14:58:55.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0010704989999999 2024-09-13T14:58:55.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.232 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:58:55.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.562475+0000", 2024-09-13T14:58:55.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000136038 2024-09-13T14:58:55.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.232 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:58:55.233 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.562622+0000", 2024-09-13T14:58:55.233 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014687899999999999 2024-09-13T14:58:55.233 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.233 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.233 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.568665+0000", 2024-09-13T14:58:55.233 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0060433170000000003 2024-09-13T14:58:55.233 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.233 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.234 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.568733+0000", 2024-09-13T14:58:55.234 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7955999999999998e-05 2024-09-13T14:58:55.234 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.234 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.234 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:58:55.234 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.571695+0000", 2024-09-13T14:58:55.234 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002961722 2024-09-13T14:58:55.234 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.234 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.234 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:58:55.234 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.571749+0000", 2024-09-13T14:58:55.234 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4112e-05 2024-09-13T14:58:55.234 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.234 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.235 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:58:55.235 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.571809+0000", 2024-09-13T14:58:55.235 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0340000000000002e-05 2024-09-13T14:58:55.235 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.235 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:58:55.235 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.235 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.235 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:179816 8.b 8:d0f51985:::benchmark_data_smithi151_229097_object54948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1189)", 2024-09-13T14:58:55.235 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:52:11.584714+0000", 2024-09-13T14:58:55.235 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 403.63820890800002, 2024-09-13T14:58:55.235 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.98744244400000003, 2024-09-13T14:58:55.235 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:58:55.235 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:58:55.235 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:55.236 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:58:55.236 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T14:58:55.236 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:58:55.236 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 179816 2024-09-13T14:58:55.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.236 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:58:55.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:58:55.236 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.584714+0000", 2024-09-13T14:58:55.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:58:55.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.584714+0000", 2024-09-13T14:58:55.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.237 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.237 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:58:55.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.584716+0000", 2024-09-13T14:58:55.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.697e-06 2024-09-13T14:58:55.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.237 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.237 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:58:55.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.584722+0000", 2024-09-13T14:58:55.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8000000000000004e-06 2024-09-13T14:58:55.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.238 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.238 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:58:55.238 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.584738+0000", 2024-09-13T14:58:55.238 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6294999999999999e-05 2024-09-13T14:58:55.238 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.238 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.238 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:58:55.238 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.584747+0000", 2024-09-13T14:58:55.238 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7779999999999996e-06 2024-09-13T14:58:55.238 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.238 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.238 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.238 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.584813+0000", 2024-09-13T14:58:55.239 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6781999999999999e-05 2024-09-13T14:58:55.239 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.239 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:58:55.239 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.584822+0000", 2024-09-13T14:58:55.239 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3950000000000001e-06 2024-09-13T14:58:55.239 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.239 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.239 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.566091+0000", 2024-09-13T14:58:55.239 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.98126905799999997 2024-09-13T14:58:55.239 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.239 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:58:55.240 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.566208+0000", 2024-09-13T14:58:55.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116889 2024-09-13T14:58:55.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.240 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:58:55.240 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.566340+0000", 2024-09-13T14:58:55.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013193999999999999 2024-09-13T14:58:55.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.240 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.240 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.571968+0000", 2024-09-13T14:58:55.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0056284339999999999 2024-09-13T14:58:55.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.241 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:58:55.241 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.571985+0000", 2024-09-13T14:58:55.241 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7269e-05 2024-09-13T14:58:55.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.241 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.241 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.572064+0000", 2024-09-13T14:58:55.241 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8638000000000002e-05 2024-09-13T14:58:55.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.241 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:58:55.241 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.572114+0000", 2024-09-13T14:58:55.242 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9697000000000002e-05 2024-09-13T14:58:55.242 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.242 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.242 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:58:55.242 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.572156+0000", 2024-09-13T14:58:55.242 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2772000000000003e-05 2024-09-13T14:58:55.242 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.243 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:58:55.243 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.243 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.243 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.243 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:233538 8.0 8:0a49c4c4:::benchmark_data_smithi151_229097_object108670:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1198)", 2024-09-13T14:58:55.243 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:10.775515+0000", 2024-09-13T14:58:55.243 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 344.44740824799999, 2024-09-13T14:58:55.243 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.049572129, 2024-09-13T14:58:55.243 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:58:55.243 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:58:55.243 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-13T14:58:55.243 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:58:55.244 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T14:58:55.244 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:58:55.244 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 233538 2024-09-13T14:58:55.244 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.244 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:58:55.244 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.244 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:58:55.244 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:10.775515+0000", 2024-09-13T14:58:55.244 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.244 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.244 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.244 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:58:55.245 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:10.775515+0000", 2024-09-13T14:58:55.245 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.245 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.245 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.245 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:58:55.245 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:10.775517+0000", 2024-09-13T14:58:55.245 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1849999999999998e-06 2024-09-13T14:58:55.245 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.245 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.245 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:58:55.245 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:10.775523+0000", 2024-09-13T14:58:55.245 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0650000000000004e-06 2024-09-13T14:58:55.245 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.246 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:58:55.246 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:10.778499+0000", 2024-09-13T14:58:55.246 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0029762870000000002 2024-09-13T14:58:55.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.246 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:58:55.246 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:10.778518+0000", 2024-09-13T14:58:55.246 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8595000000000001e-05 2024-09-13T14:58:55.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.246 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.247 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:10.779421+0000", 2024-09-13T14:58:55.247 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00090357700000000003 2024-09-13T14:58:55.247 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.247 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.247 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:58:55.247 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:10.779525+0000", 2024-09-13T14:58:55.247 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103723 2024-09-13T14:58:55.247 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.247 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.247 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,5,6", 2024-09-13T14:58:55.247 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:10.779628+0000", 2024-09-13T14:58:55.248 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010325199999999999 2024-09-13T14:58:55.248 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.248 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.248 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:10.793970+0000", 2024-09-13T14:58:55.248 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01434116 2024-09-13T14:58:55.248 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.248 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.248 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:10.794088+0000", 2024-09-13T14:58:55.248 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117964 2024-09-13T14:58:55.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.249 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.249 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:10.794382+0000", 2024-09-13T14:58:55.249 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000294128 2024-09-13T14:58:55.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.249 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.249 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:10.810965+0000", 2024-09-13T14:58:55.249 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016583131000000001 2024-09-13T14:58:55.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.250 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.250 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:58:55.250 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:10.810979+0000", 2024-09-13T14:58:55.250 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4406999999999999e-05 2024-09-13T14:58:55.250 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.250 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.250 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.250 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.824984+0000", 2024-09-13T14:58:55.250 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0140044749999999 2024-09-13T14:58:55.250 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.250 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.250 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:58:55.250 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.825087+0000", 2024-09-13T14:58:55.251 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010318 2024-09-13T14:58:55.251 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.251 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:58:55.251 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.251 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.251 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.251 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:233595 8.0 8:0677edea:::benchmark_data_smithi151_229097_object108727:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1199)", 2024-09-13T14:58:55.251 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:10.843532+0000", 2024-09-13T14:58:55.251 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 344.37939110299999, 2024-09-13T14:58:55.252 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.98704009699999995, 2024-09-13T14:58:55.252 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:58:55.252 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:58:55.252 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:55.252 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:58:55.252 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T14:58:55.252 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:58:55.252 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 233595 2024-09-13T14:58:55.252 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.252 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:58:55.252 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.252 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:58:55.252 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:10.843532+0000", 2024-09-13T14:58:55.253 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.253 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.253 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.253 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:58:55.253 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:10.843532+0000", 2024-09-13T14:58:55.253 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.253 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.253 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.253 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:58:55.253 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:10.843532+0000", 2024-09-13T14:58:55.253 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0600000000000001e-07 2024-09-13T14:58:55.253 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.253 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.253 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:58:55.254 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:10.843535+0000", 2024-09-13T14:58:55.254 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2919999999999998e-06 2024-09-13T14:58:55.254 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.254 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.254 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:58:55.254 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:10.843540+0000", 2024-09-13T14:58:55.254 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7250000000000002e-06 2024-09-13T14:58:55.254 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.254 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.254 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:58:55.254 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:10.843545+0000", 2024-09-13T14:58:55.254 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8029999999999996e-06 2024-09-13T14:58:55.255 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.255 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.255 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.255 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:10.843838+0000", 2024-09-13T14:58:55.255 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00029333000000000001 2024-09-13T14:58:55.255 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.255 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.255 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:58:55.255 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:10.843846+0000", 2024-09-13T14:58:55.255 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3180000000000001e-06 2024-09-13T14:58:55.255 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.255 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.256 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.256 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.825594+0000", 2024-09-13T14:58:55.256 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.98174796900000005 2024-09-13T14:58:55.256 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.256 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.256 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:58:55.256 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.825695+0000", 2024-09-13T14:58:55.256 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101178 2024-09-13T14:58:55.256 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.256 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.256 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:58:55.256 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.825820+0000", 2024-09-13T14:58:55.257 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000125179 2024-09-13T14:58:55.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.257 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.257 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.257 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.830309+0000", 2024-09-13T14:58:55.257 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0044893839999999999 2024-09-13T14:58:55.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.257 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.257 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:58:55.257 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.830402+0000", 2024-09-13T14:58:55.257 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2724000000000002e-05 2024-09-13T14:58:55.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.258 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.258 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.258 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.830482+0000", 2024-09-13T14:58:55.258 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9923e-05 2024-09-13T14:58:55.258 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.258 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.258 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:58:55.258 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.830534+0000", 2024-09-13T14:58:55.259 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:54 smithi151 ceph-mon[104450]: pgmap v2065: 119 pgs: 119 active+clean; 4.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-09-13T14:58:55.259 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:54 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:55.259 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:54 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:55.259 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:54 smithi151 ceph-mon[108966]: pgmap v2065: 119 pgs: 119 active+clean; 4.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-09-13T14:58:55.259 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:54 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:55.259 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:54 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:55.260 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1913999999999998e-05 2024-09-13T14:58:55.260 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.260 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.261 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:58:55.261 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.830572+0000", 2024-09-13T14:58:55.261 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7951999999999998e-05 2024-09-13T14:58:55.261 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.261 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:58:55.261 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.261 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.261 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.262 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:233613 8.0 8:0065e074:::benchmark_data_smithi151_229097_object108745:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1199)", 2024-09-13T14:58:55.262 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:10.858380+0000", 2024-09-13T14:58:55.262 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 344.364543169, 2024-09-13T14:58:55.262 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.97491249199999996, 2024-09-13T14:58:55.262 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:58:55.262 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:58:55.262 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:55.262 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:58:55.262 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T14:58:55.262 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:58:55.262 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 233613 2024-09-13T14:58:55.262 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.262 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:58:55.262 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.263 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:58:55.263 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:10.858380+0000", 2024-09-13T14:58:55.263 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.263 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.263 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.263 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:58:55.263 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:10.858380+0000", 2024-09-13T14:58:55.263 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.263 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.263 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.263 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:58:55.263 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:10.858380+0000", 2024-09-13T14:58:55.264 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4700000000000002e-07 2024-09-13T14:58:55.264 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.264 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.264 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:58:55.264 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:10.858383+0000", 2024-09-13T14:58:55.264 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3319999999999999e-06 2024-09-13T14:58:55.264 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.264 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.264 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:58:55.264 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:10.858392+0000", 2024-09-13T14:58:55.264 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4769999999999992e-06 2024-09-13T14:58:55.264 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.264 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.265 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:58:55.265 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:10.858397+0000", 2024-09-13T14:58:55.265 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.967e-06 2024-09-13T14:58:55.265 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.265 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.265 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.265 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:10.858512+0000", 2024-09-13T14:58:55.265 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000115122 2024-09-13T14:58:55.265 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.265 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.265 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:58:55.265 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:10.858520+0000", 2024-09-13T14:58:55.265 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5569999999999998e-06 2024-09-13T14:58:55.266 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.266 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.266 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.266 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.827841+0000", 2024-09-13T14:58:55.266 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.96932171199999995 2024-09-13T14:58:55.266 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.266 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.266 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:58:55.266 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.827949+0000", 2024-09-13T14:58:55.266 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000107701 2024-09-13T14:58:55.266 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.266 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.266 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:58:55.267 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.828066+0000", 2024-09-13T14:58:55.267 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116754 2024-09-13T14:58:55.267 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.267 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.267 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.267 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.830442+0000", 2024-09-13T14:58:55.267 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002376043 2024-09-13T14:58:55.267 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.267 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.267 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.267 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.833110+0000", 2024-09-13T14:58:55.267 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002668377 2024-09-13T14:58:55.267 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.267 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.268 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:58:55.268 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.833198+0000", 2024-09-13T14:58:55.268 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7929999999999993e-05 2024-09-13T14:58:55.268 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.268 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.268 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:58:55.268 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.833251+0000", 2024-09-13T14:58:55.268 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.287e-05 2024-09-13T14:58:55.268 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.268 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.268 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:58:55.268 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:11.833292+0000", 2024-09-13T14:58:55.268 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1202999999999998e-05 2024-09-13T14:58:55.269 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.269 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:58:55.269 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.269 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.269 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.269 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.46034.0:67676 9.c 9:3a45e7d2:::benchmark_data_smithi151_261954_object67675:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1227)", 2024-09-13T14:58:55.269 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:54:57.800212+0000", 2024-09-13T14:58:55.269 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 237.42271138500001, 2024-09-13T14:58:55.269 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2215523739999998, 2024-09-13T14:58:55.269 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:58:55.269 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:58:55.269 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-13T14:58:55.269 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:58:55.270 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.46034", 2024-09-13T14:58:55.270 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:58:55.270 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 67676 2024-09-13T14:58:55.270 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.270 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:58:55.270 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.270 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:58:55.270 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:57.800212+0000", 2024-09-13T14:58:55.270 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.270 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.270 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.270 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:58:55.270 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:57.800212+0000", 2024-09-13T14:58:55.271 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:58:55.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:57.800215+0000", 2024-09-13T14:58:55.271 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5090000000000002e-06 2024-09-13T14:58:55.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:58:55.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:57.800245+0000", 2024-09-13T14:58:55.271 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9782e-05 2024-09-13T14:58:55.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.272 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:58:55.272 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:57.800263+0000", 2024-09-13T14:58:55.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8139e-05 2024-09-13T14:58:55.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.272 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.272 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:58:55.272 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:57.800272+0000", 2024-09-13T14:58:55.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.619e-06 2024-09-13T14:58:55.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.272 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.272 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.272 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:57.800385+0000", 2024-09-13T14:58:55.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000113684 2024-09-13T14:58:55.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.273 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:58:55.273 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:57.800550+0000", 2024-09-13T14:58:55.273 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000164916 2024-09-13T14:58:55.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.273 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:58:55.273 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:57.800696+0000", 2024-09-13T14:58:55.273 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014565699999999999 2024-09-13T14:58:55.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.273 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:58:55.274 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:57.809321+0000", 2024-09-13T14:58:55.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0086250470000000003 2024-09-13T14:58:55.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.274 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.274 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.274 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:57.832914+0000", 2024-09-13T14:58:55.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.023592722 2024-09-13T14:58:55.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.274 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.274 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:58:55.274 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:57.832924+0000", 2024-09-13T14:58:55.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0375e-05 2024-09-13T14:58:55.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.275 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.275 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.275 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:58.809562+0000", 2024-09-13T14:58:55.275 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.976637796 2024-09-13T14:58:55.275 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.275 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.275 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:58:55.275 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:58.809574+0000", 2024-09-13T14:58:55.275 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2473000000000001e-05 2024-09-13T14:58:55.275 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.275 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.275 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.276 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:00.008205+0000", 2024-09-13T14:58:55.276 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1986307 2024-09-13T14:58:55.276 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.276 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.276 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for degraded object", 2024-09-13T14:58:55.276 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:00.009447+0000", 2024-09-13T14:58:55.276 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012419620000000001 2024-09-13T14:58:55.276 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.276 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.276 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.276 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:00.021647+0000", 2024-09-13T14:58:55.276 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012200183 2024-09-13T14:58:55.276 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.277 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.277 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:58:55.277 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:00.021764+0000", 2024-09-13T14:58:55.277 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011681 2024-09-13T14:58:55.277 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.277 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:58:55.277 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.277 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.277 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.277 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.46034.0:67735 9.c 9:35268c4d:::benchmark_data_smithi151_261954_object67734:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1228)", 2024-09-13T14:58:55.277 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:54:57.872308+0000", 2024-09-13T14:58:55.277 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 237.35061496200001, 2024-09-13T14:58:55.277 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1522546939999998, 2024-09-13T14:58:55.278 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:58:55.278 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:58:55.278 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:55.278 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:58:55.278 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.46034", 2024-09-13T14:58:55.278 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:58:55.278 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 67735 2024-09-13T14:58:55.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.278 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:58:55.278 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.278 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:58:55.278 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:57.872308+0000", 2024-09-13T14:58:55.278 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.279 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.279 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:58:55.279 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:57.872308+0000", 2024-09-13T14:58:55.279 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.279 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.279 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.279 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:58:55.279 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:57.872309+0000", 2024-09-13T14:58:55.279 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1680000000000001e-06 2024-09-13T14:58:55.279 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.279 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.280 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:58:55.280 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:57.872329+0000", 2024-09-13T14:58:55.280 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9681999999999999e-05 2024-09-13T14:58:55.280 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.280 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.280 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:58:55.280 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:57.872345+0000", 2024-09-13T14:58:55.280 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5767e-05 2024-09-13T14:58:55.280 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.280 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.280 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:58:55.281 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:57.872352+0000", 2024-09-13T14:58:55.281 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9099999999999999e-06 2024-09-13T14:58:55.281 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.281 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.281 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.281 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:57.872413+0000", 2024-09-13T14:58:55.281 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1359000000000007e-05 2024-09-13T14:58:55.281 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.281 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.281 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:58:55.281 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:57.872423+0000", 2024-09-13T14:58:55.281 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0492999999999999e-05 2024-09-13T14:58:55.281 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.281 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.282 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.282 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:58.809599+0000", 2024-09-13T14:58:55.282 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.93717527499999997 2024-09-13T14:58:55.282 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.282 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:58:55.282 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:58.809606+0000", 2024-09-13T14:58:55.282 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3599999999999998e-06 2024-09-13T14:58:55.282 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.282 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.282 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:00.012131+0000", 2024-09-13T14:58:55.282 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.202525297 2024-09-13T14:58:55.283 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.283 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.283 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:58:55.283 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:00.012239+0000", 2024-09-13T14:58:55.283 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108193 2024-09-13T14:58:55.283 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.283 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.283 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-13T14:58:55.283 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:00.012688+0000", 2024-09-13T14:58:55.283 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000448114 2024-09-13T14:58:55.283 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.283 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.283 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:58:55.283 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:00.021761+0000", 2024-09-13T14:58:55.284 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0090736749999999998 2024-09-13T14:58:55.284 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.284 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.284 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.284 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:00.021881+0000", 2024-09-13T14:58:55.284 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000119259 2024-09-13T14:58:55.284 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.284 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.284 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.284 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:00.021951+0000", 2024-09-13T14:58:55.284 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0154999999999995e-05 2024-09-13T14:58:55.284 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.284 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.285 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:58:55.285 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:00.021992+0000", 2024-09-13T14:58:55.285 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1217999999999997e-05 2024-09-13T14:58:55.285 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.285 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.285 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:58:55.285 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:00.024563+0000", 2024-09-13T14:58:55.285 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0025707690000000001 2024-09-13T14:58:55.285 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.285 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:58:55.285 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.285 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.285 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.286 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.46034.0:67739 9.c 9:3b1bf703:::benchmark_data_smithi151_261954_object67738:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1228)", 2024-09-13T14:58:55.286 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:54:57.879532+0000", 2024-09-13T14:58:55.286 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 237.34339117600001, 2024-09-13T14:58:55.286 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1733456470000001, 2024-09-13T14:58:55.286 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:58:55.286 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:58:55.286 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:55.286 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:58:55.286 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.46034", 2024-09-13T14:58:55.286 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:58:55.286 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 67739 2024-09-13T14:58:55.286 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.286 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:58:55.286 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.287 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:58:55.287 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:57.879532+0000", 2024-09-13T14:58:55.287 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.287 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.287 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:58:55.287 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:57.879532+0000", 2024-09-13T14:58:55.287 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.287 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.287 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:58:55.287 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:57.879533+0000", 2024-09-13T14:58:55.287 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5340000000000001e-06 2024-09-13T14:58:55.287 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.288 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:58:55.288 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:57.879554+0000", 2024-09-13T14:58:55.288 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1039e-05 2024-09-13T14:58:55.288 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.288 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.288 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:58:55.288 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:57.879572+0000", 2024-09-13T14:58:55.288 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7362000000000001e-05 2024-09-13T14:58:55.288 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.288 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.288 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:58:55.288 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:57.879579+0000", 2024-09-13T14:58:55.288 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5109999999999997e-06 2024-09-13T14:58:55.288 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.289 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.289 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.289 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:57.879651+0000", 2024-09-13T14:58:55.289 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1878999999999994e-05 2024-09-13T14:58:55.289 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.289 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.289 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:58:55.289 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:57.879664+0000", 2024-09-13T14:58:55.289 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2867000000000001e-05 2024-09-13T14:58:55.289 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.289 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.289 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.289 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:58.809628+0000", 2024-09-13T14:58:55.290 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.92996435099999997 2024-09-13T14:58:55.290 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.290 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.290 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:58:55.290 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:54:58.809636+0000", 2024-09-13T14:58:55.290 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1940000000000001e-06 2024-09-13T14:58:55.290 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.290 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.290 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.290 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:00.028432+0000", 2024-09-13T14:58:55.290 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.218796046 2024-09-13T14:58:55.290 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.290 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.291 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:58:55.291 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:00.028590+0000", 2024-09-13T14:58:55.291 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015813000000000001 2024-09-13T14:58:55.291 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.291 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.291 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-13T14:58:55.291 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:00.028741+0000", 2024-09-13T14:58:55.291 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000151209 2024-09-13T14:58:55.291 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.291 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.291 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.291 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:00.041222+0000", 2024-09-13T14:58:55.291 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012480694000000001 2024-09-13T14:58:55.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.292 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.292 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:58:55.292 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:00.041323+0000", 2024-09-13T14:58:55.292 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101396 2024-09-13T14:58:55.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.292 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.292 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.292 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:00.049479+0000", 2024-09-13T14:58:55.292 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0081557680000000007 2024-09-13T14:58:55.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.292 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.292 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:58:55.293 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:00.049549+0000", 2024-09-13T14:58:55.293 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0145e-05 2024-09-13T14:58:55.293 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.293 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.293 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:58:55.293 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:00.052877+0000", 2024-09-13T14:58:55.293 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003328522 2024-09-13T14:58:55.293 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.293 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:58:55.293 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.293 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.293 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.293 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.46034.0:83727 9.8 9:1a39a8bd:::benchmark_data_smithi151_261954_object83726:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1231)", 2024-09-13T14:58:55.293 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:55:15.835101+0000", 2024-09-13T14:58:55.294 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 219.38782201800001, 2024-09-13T14:58:55.294 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3707149209999998, 2024-09-13T14:58:55.294 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:58:55.294 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:58:55.294 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-13T14:58:55.294 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:58:55.294 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.46034", 2024-09-13T14:58:55.294 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:58:55.294 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 83727 2024-09-13T14:58:55.294 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.294 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:58:55.294 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.294 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:58:55.294 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.835101+0000", 2024-09-13T14:58:55.295 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.295 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.295 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.295 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:58:55.295 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.835101+0000", 2024-09-13T14:58:55.295 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.295 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.295 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.295 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:58:55.295 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.835108+0000", 2024-09-13T14:58:55.295 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7460000000000003e-06 2024-09-13T14:58:55.295 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.295 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.296 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:58:55.296 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.835134+0000", 2024-09-13T14:58:55.296 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.639e-05 2024-09-13T14:58:55.296 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.296 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.296 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:58:55.296 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.835155+0000", 2024-09-13T14:58:55.296 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0678000000000001e-05 2024-09-13T14:58:55.296 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.296 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.296 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:58:55.296 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.835167+0000", 2024-09-13T14:58:55.296 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1909000000000001e-05 2024-09-13T14:58:55.297 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.297 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.297 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.297 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.838510+0000", 2024-09-13T14:58:55.297 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003342935 2024-09-13T14:58:55.297 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.297 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.297 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:58:55.297 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.839152+0000", 2024-09-13T14:58:55.297 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00064242100000000001 2024-09-13T14:58:55.297 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.297 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.297 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:58:55.297 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.839294+0000", 2024-09-13T14:58:55.297 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014248599999999999 2024-09-13T14:58:55.298 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.298 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.298 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:58:55.298 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.849661+0000", 2024-09-13T14:58:55.298 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010366288 2024-09-13T14:58:55.298 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.298 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.298 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.298 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.872938+0000", 2024-09-13T14:58:55.298 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.023277507999999999 2024-09-13T14:58:55.298 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.298 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.298 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:58:55.299 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.872952+0000", 2024-09-13T14:58:55.299 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3448999999999999e-05 2024-09-13T14:58:55.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.299 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.299 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.299 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:16.855094+0000", 2024-09-13T14:58:55.299 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.98214216799999998 2024-09-13T14:58:55.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.299 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.299 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:58:55.299 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:16.855104+0000", 2024-09-13T14:58:55.299 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0387999999999999e-05 2024-09-13T14:58:55.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.300 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.300 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.300 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.205693+0000", 2024-09-13T14:58:55.300 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.350588624 2024-09-13T14:58:55.300 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.300 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.300 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:58:55.300 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.205816+0000", 2024-09-13T14:58:55.300 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012293100000000001 2024-09-13T14:58:55.300 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.300 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:58:55.300 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.301 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.301 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.301 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.46034.0:83729 9.8 9:19445c2a:::benchmark_data_smithi151_261954_object83728:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1231)", 2024-09-13T14:58:55.301 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:55:15.835610+0000", 2024-09-13T14:58:55.301 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 219.387312549, 2024-09-13T14:58:55.301 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.465796101, 2024-09-13T14:58:55.301 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:58:55.301 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:58:55.301 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-13T14:58:55.301 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:58:55.301 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.46034", 2024-09-13T14:58:55.301 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:58:55.301 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 83729 2024-09-13T14:58:55.301 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.302 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:58:55.302 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.302 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:58:55.302 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.835610+0000", 2024-09-13T14:58:55.302 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.302 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.302 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.302 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:58:55.302 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.835610+0000", 2024-09-13T14:58:55.302 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.302 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.302 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.302 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:58:55.303 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.835611+0000", 2024-09-13T14:58:55.303 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7199999999999998e-07 2024-09-13T14:58:55.303 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.303 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.303 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:58:55.303 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.835631+0000", 2024-09-13T14:58:55.303 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9724e-05 2024-09-13T14:58:55.303 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.303 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.303 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:58:55.303 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.835641+0000", 2024-09-13T14:58:55.303 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6350000000000001e-06 2024-09-13T14:58:55.303 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.304 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.304 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:58:55.304 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.835647+0000", 2024-09-13T14:58:55.304 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4840000000000001e-06 2024-09-13T14:58:55.304 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.304 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.305 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.305 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.850413+0000", 2024-09-13T14:58:55.305 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014765597 2024-09-13T14:58:55.305 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.305 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.305 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:58:55.305 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.850568+0000", 2024-09-13T14:58:55.305 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000155845 2024-09-13T14:58:55.305 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.306 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:58:55.306 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.850713+0000", 2024-09-13T14:58:55.306 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014484900000000001 2024-09-13T14:58:55.306 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.306 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.306 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.873357+0000", 2024-09-13T14:58:55.306 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.022643447000000001 2024-09-13T14:58:55.306 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.306 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:58:55.306 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.873365+0000", 2024-09-13T14:58:55.307 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7139999999999999e-06 2024-09-13T14:58:55.307 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.307 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.307 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.307 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:16.856952+0000", 2024-09-13T14:58:55.307 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.98358636700000002 2024-09-13T14:58:55.307 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.307 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.307 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:58:55.307 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:16.856964+0000", 2024-09-13T14:58:55.307 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1768e-05 2024-09-13T14:58:55.307 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.308 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.308 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.308 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.275539+0000", 2024-09-13T14:58:55.308 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.418575154 2024-09-13T14:58:55.308 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.308 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.308 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for degraded object", 2024-09-13T14:58:55.308 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.276421+0000", 2024-09-13T14:58:55.308 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00088247399999999995 2024-09-13T14:58:55.308 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.308 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.308 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.309 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.301302+0000", 2024-09-13T14:58:55.309 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.024881210000000001 2024-09-13T14:58:55.309 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.309 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.309 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:58:55.309 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.301406+0000", 2024-09-13T14:58:55.309 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010406100000000001 2024-09-13T14:58:55.309 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.309 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:58:55.309 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.309 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.309 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.310 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.46034.0:83745 9.8 9:1e05f279:::benchmark_data_smithi151_261954_object83744:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1232)", 2024-09-13T14:58:55.310 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:55:15.904152+0000", 2024-09-13T14:58:55.310 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 219.31877117499999, 2024-09-13T14:58:55.310 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4080624049999999, 2024-09-13T14:58:55.310 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:58:55.310 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:58:55.310 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:55.310 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:58:55.310 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.46034", 2024-09-13T14:58:55.310 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:58:55.310 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 83745 2024-09-13T14:58:55.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.310 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:58:55.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:58:55.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.904152+0000", 2024-09-13T14:58:55.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:58:55.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.904152+0000", 2024-09-13T14:58:55.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:58:55.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.904153+0000", 2024-09-13T14:58:55.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.285e-06 2024-09-13T14:58:55.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.312 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:58:55.312 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.904177+0000", 2024-09-13T14:58:55.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4216e-05 2024-09-13T14:58:55.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.312 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:58:55.312 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.904190+0000", 2024-09-13T14:58:55.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2928999999999999e-05 2024-09-13T14:58:55.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.313 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:58:55.313 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.904197+0000", 2024-09-13T14:58:55.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9600000000000003e-06 2024-09-13T14:58:55.313 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.313 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.313 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.904244+0000", 2024-09-13T14:58:55.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7290999999999998e-05 2024-09-13T14:58:55.313 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.314 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:58:55.314 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.904251+0000", 2024-09-13T14:58:55.314 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6379999999999998e-06 2024-09-13T14:58:55.314 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.314 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.314 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:16.857027+0000", 2024-09-13T14:58:55.314 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.95277572499999996 2024-09-13T14:58:55.314 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.314 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:58:55.314 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:16.857036+0000", 2024-09-13T14:58:55.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8929999999999994e-06 2024-09-13T14:58:55.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.276537+0000", 2024-09-13T14:58:55.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.419501181 2024-09-13T14:58:55.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:58:55.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.276706+0000", 2024-09-13T14:58:55.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016901000000000001 2024-09-13T14:58:55.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:58:55.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.277266+0000", 2024-09-13T14:58:55.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00056001499999999997 2024-09-13T14:58:55.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.301519+0000", 2024-09-13T14:58:55.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.024252863 2024-09-13T14:58:55.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.301561+0000", 2024-09-13T14:58:55.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2700000000000001e-05 2024-09-13T14:58:55.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:58:55.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.307821+0000", 2024-09-13T14:58:55.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0062590730000000004 2024-09-13T14:58:55.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.317 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:58:55.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.307879+0000", 2024-09-13T14:58:55.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8350999999999997e-05 2024-09-13T14:58:55.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.317 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:58:55.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.312214+0000", 2024-09-13T14:58:55.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0043352750000000004 2024-09-13T14:58:55.317 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.318 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:58:55.318 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.318 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.318 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.318 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.46034.0:200313 9.8 9:14628d6c:::benchmark_data_smithi151_261954_object97457:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1253)", 2024-09-13T14:58:55.318 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:57:34.978598+0000", 2024-09-13T14:58:55.318 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 80.244324641000006, 2024-09-13T14:58:55.318 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0422188750000001, 2024-09-13T14:58:55.318 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:58:55.318 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:58:55.318 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-13T14:58:55.318 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:58:55.318 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.46034", 2024-09-13T14:58:55.319 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:58:55.319 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 200313 2024-09-13T14:58:55.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.319 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:58:55.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.319 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:58:55.319 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:34.978598+0000", 2024-09-13T14:58:55.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.319 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:58:55.319 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:34.978598+0000", 2024-09-13T14:58:55.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:58:55.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:34.978600+0000", 2024-09-13T14:58:55.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2899999999999999e-06 2024-09-13T14:58:55.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:58:55.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:34.978604+0000", 2024-09-13T14:58:55.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8749999999999999e-06 2024-09-13T14:58:55.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:58:55.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:34.978661+0000", 2024-09-13T14:58:55.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6752999999999997e-05 2024-09-13T14:58:55.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:58:55.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:34.978684+0000", 2024-09-13T14:58:55.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2993000000000002e-05 2024-09-13T14:58:55.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:34.994896+0000", 2024-09-13T14:58:55.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016211817999999999 2024-09-13T14:58:55.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:58:55.322 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:34.995038+0000", 2024-09-13T14:58:55.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014216000000000001 2024-09-13T14:58:55.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.322 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:58:55.322 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:34.995168+0000", 2024-09-13T14:58:55.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013023000000000001 2024-09-13T14:58:55.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.322 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.322 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.009534+0000", 2024-09-13T14:58:55.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014365559999999999 2024-09-13T14:58:55.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:58:55.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.009540+0000", 2024-09-13T14:58:55.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9839999999999998e-06 2024-09-13T14:58:55.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.020738+0000", 2024-09-13T14:58:55.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0111981430000001 2024-09-13T14:58:55.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:58:55.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.020817+0000", 2024-09-13T14:58:55.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9068999999999998e-05 2024-09-13T14:58:55.323 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.324 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:58:55.324 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.324 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.46034.0:200368 9.8 9:1614d357:::benchmark_data_smithi151_261954_object97512:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1254)", 2024-09-13T14:58:55.324 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:57:35.039528+0000", 2024-09-13T14:58:55.324 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 80.183395102000006, 2024-09-13T14:58:55.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.98955691000000001, 2024-09-13T14:58:55.324 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:58:55.324 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:58:55.324 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:55.324 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:58:55.324 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.46034", 2024-09-13T14:58:55.325 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:58:55.325 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 200368 2024-09-13T14:58:55.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.325 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:58:55.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.325 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:58:55.325 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.039528+0000", 2024-09-13T14:58:55.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.325 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:58:55.325 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.039528+0000", 2024-09-13T14:58:55.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:58:55.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.039529+0000", 2024-09-13T14:58:55.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.19e-06 2024-09-13T14:58:55.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:58:55.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.039534+0000", 2024-09-13T14:58:55.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9250000000000003e-06 2024-09-13T14:58:55.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:58:55.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.039550+0000", 2024-09-13T14:58:55.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5747e-05 2024-09-13T14:58:55.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:58:55.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.039557+0000", 2024-09-13T14:58:55.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.537e-06 2024-09-13T14:58:55.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.039634+0000", 2024-09-13T14:58:55.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6333000000000003e-05 2024-09-13T14:58:55.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:58:55.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.039643+0000", 2024-09-13T14:58:55.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5650000000000007e-06 2024-09-13T14:58:55.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.021229+0000", 2024-09-13T14:58:55.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.98158553299999995 2024-09-13T14:58:55.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:58:55.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.021339+0000", 2024-09-13T14:58:55.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000110402 2024-09-13T14:58:55.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:58:55.329 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.021458+0000", 2024-09-13T14:58:55.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118445 2024-09-13T14:58:55.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.329 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.028852+0000", 2024-09-13T14:58:55.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0073948210000000002 2024-09-13T14:58:55.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.329 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.028918+0000", 2024-09-13T14:58:55.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5522e-05 2024-09-13T14:58:55.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:58:55.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.028987+0000", 2024-09-13T14:58:55.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9136999999999999e-05 2024-09-13T14:58:55.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:58:55.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.029041+0000", 2024-09-13T14:58:55.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3943000000000002e-05 2024-09-13T14:58:55.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:58:55.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.029085+0000", 2024-09-13T14:58:55.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3810000000000002e-05 2024-09-13T14:58:55.331 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.331 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:58:55.331 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.331 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.46034.0:200370 9.8 9:176c4228:::benchmark_data_smithi151_261954_object97514:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1254)", 2024-09-13T14:58:55.331 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:57:35.040542+0000", 2024-09-13T14:58:55.331 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 80.182380832999996, 2024-09-13T14:58:55.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.98866303600000005, 2024-09-13T14:58:55.331 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:58:55.331 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:58:55.331 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:55.332 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:58:55.332 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.46034", 2024-09-13T14:58:55.332 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:58:55.332 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 200370 2024-09-13T14:58:55.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.332 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:58:55.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:58:55.332 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.040542+0000", 2024-09-13T14:58:55.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:58:55.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.040542+0000", 2024-09-13T14:58:55.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.333 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:58:55.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.040543+0000", 2024-09-13T14:58:55.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6499999999999999e-07 2024-09-13T14:58:55.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.333 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:58:55.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.040546+0000", 2024-09-13T14:58:55.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8459999999999999e-06 2024-09-13T14:58:55.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:58:55.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.040555+0000", 2024-09-13T14:58:55.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8039999999999996e-06 2024-09-13T14:58:55.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:58:55.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.040561+0000", 2024-09-13T14:58:55.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0039999999999996e-06 2024-09-13T14:58:55.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.040605+0000", 2024-09-13T14:58:55.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.392e-05 2024-09-13T14:58:55.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:58:55.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.040613+0000", 2024-09-13T14:58:55.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6510000000000001e-06 2024-09-13T14:58:55.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.023922+0000", 2024-09-13T14:58:55.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.98330905899999999 2024-09-13T14:58:55.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:58:55.336 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.024037+0000", 2024-09-13T14:58:55.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000115202 2024-09-13T14:58:55.336 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.336 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:58:55.336 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.024104+0000", 2024-09-13T14:58:55.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6537000000000003e-05 2024-09-13T14:58:55.336 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.336 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.336 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.028973+0000", 2024-09-13T14:58:55.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0048694189999999998 2024-09-13T14:58:55.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:58:55.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.029095+0000", 2024-09-13T14:58:55.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012183799999999999 2024-09-13T14:58:55.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.029148+0000", 2024-09-13T14:58:55.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2549000000000001e-05 2024-09-13T14:58:55.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:58:55.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.029180+0000", 2024-09-13T14:58:55.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2777000000000001e-05 2024-09-13T14:58:55.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.338 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:58:55.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.029205+0000", 2024-09-13T14:58:55.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4765000000000001e-05 2024-09-13T14:58:55.338 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.338 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:58:55.338 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.338 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.46034.0:200375 9.8 9:15a9e705:::benchmark_data_smithi151_261954_object97519:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1254)", 2024-09-13T14:58:55.339 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:57:35.043856+0000", 2024-09-13T14:58:55.339 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 80.179067126000007, 2024-09-13T14:58:55.339 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.98555630599999999, 2024-09-13T14:58:55.339 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:58:55.339 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:58:55.339 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:55.339 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:58:55.339 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.46034", 2024-09-13T14:58:55.339 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:58:55.339 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 200375 2024-09-13T14:58:55.339 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.339 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:58:55.339 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.340 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:58:55.340 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.043856+0000", 2024-09-13T14:58:55.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.340 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:58:55.340 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.043856+0000", 2024-09-13T14:58:55.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.340 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:58:55.340 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.043857+0000", 2024-09-13T14:58:55.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0160000000000001e-06 2024-09-13T14:58:55.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.341 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:58:55.341 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.043861+0000", 2024-09-13T14:58:55.341 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6399999999999996e-06 2024-09-13T14:58:55.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.341 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:58:55.341 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.043873+0000", 2024-09-13T14:58:55.341 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1986e-05 2024-09-13T14:58:55.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.341 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:58:55.341 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.043881+0000", 2024-09-13T14:58:55.341 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4200000000000001e-06 2024-09-13T14:58:55.342 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.342 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.342 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.043941+0000", 2024-09-13T14:58:55.342 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.003e-05 2024-09-13T14:58:55.342 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.342 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:58:55.342 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.043950+0000", 2024-09-13T14:58:55.342 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8759999999999994e-06 2024-09-13T14:58:55.342 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.342 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.343 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.026289+0000", 2024-09-13T14:58:55.343 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.98233884400000004 2024-09-13T14:58:55.343 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.343 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.343 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:58:55.343 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.026407+0000", 2024-09-13T14:58:55.343 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011851 2024-09-13T14:58:55.343 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.343 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.343 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:58:55.343 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.026528+0000", 2024-09-13T14:58:55.343 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012102799999999999 2024-09-13T14:58:55.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.344 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.344 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.029267+0000", 2024-09-13T14:58:55.344 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027390499999999998 2024-09-13T14:58:55.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.344 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:58:55.344 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.029280+0000", 2024-09-13T14:58:55.344 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2713e-05 2024-09-13T14:58:55.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.344 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.029336+0000", 2024-09-13T14:58:55.345 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6518000000000003e-05 2024-09-13T14:58:55.345 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.345 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.345 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:58:55.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.029386+0000", 2024-09-13T14:58:55.345 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9938999999999997e-05 2024-09-13T14:58:55.345 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.345 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.345 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:58:55.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.029412+0000", 2024-09-13T14:58:55.345 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5735999999999999e-05 2024-09-13T14:58:55.345 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.345 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:58:55.346 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.346 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.46034.0:200377 9.8 9:16aeef32:::benchmark_data_smithi151_261954_object97521:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1254)", 2024-09-13T14:58:55.346 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:57:35.045522+0000", 2024-09-13T14:58:55.346 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 80.177401226000001, 2024-09-13T14:58:55.346 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.99413063400000001, 2024-09-13T14:58:55.346 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:58:55.346 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:58:55.346 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:55.346 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:58:55.346 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.46034", 2024-09-13T14:58:55.346 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:58:55.347 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 200377 2024-09-13T14:58:55.347 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.347 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:58:55.347 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.347 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:58:55.347 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.045522+0000", 2024-09-13T14:58:55.347 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.347 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.347 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.347 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:58:55.347 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.045522+0000", 2024-09-13T14:58:55.347 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.347 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.347 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.348 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:58:55.348 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.045523+0000", 2024-09-13T14:58:55.348 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.31e-06 2024-09-13T14:58:55.348 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.348 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.348 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:58:55.348 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.045528+0000", 2024-09-13T14:58:55.348 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:55.349 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3430000000000003e-06 2024-09-13T14:58:55.349 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.349 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.350 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:58:55.350 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.045542+0000", 2024-09-13T14:58:55.350 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3465999999999999e-05 2024-09-13T14:58:55.350 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.350 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.350 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:58:55.350 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.045549+0000", 2024-09-13T14:58:55.350 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3939999999999998e-06 2024-09-13T14:58:55.350 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.350 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.350 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.350 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.045609+0000", 2024-09-13T14:58:55.351 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9346999999999999e-05 2024-09-13T14:58:55.351 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.351 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.351 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:58:55.351 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.045618+0000", 2024-09-13T14:58:55.351 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4180000000000003e-06 2024-09-13T14:58:55.351 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.351 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.351 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.351 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.029477+0000", 2024-09-13T14:58:55.351 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.98385874100000004 2024-09-13T14:58:55.351 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:58:55.352 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.029585+0000", 2024-09-13T14:58:55.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000107936 2024-09-13T14:58:55.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:58:55.352 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.029686+0000", 2024-09-13T14:58:55.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010132399999999999 2024-09-13T14:58:55.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.353 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:58:55.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.039442+0000", 2024-09-13T14:58:55.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0097556729999999994 2024-09-13T14:58:55.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.353 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.039477+0000", 2024-09-13T14:58:55.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4866999999999997e-05 2024-09-13T14:58:55.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.354 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.354 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.039546+0000", 2024-09-13T14:58:55.354 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9882999999999995e-05 2024-09-13T14:58:55.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.354 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:58:55.354 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.039606+0000", 2024-09-13T14:58:55.354 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9621999999999999e-05 2024-09-13T14:58:55.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.354 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:58:55.354 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.039652+0000", 2024-09-13T14:58:55.355 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6310000000000002e-05 2024-09-13T14:58:55.355 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.355 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:58:55.355 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.355 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.46034.0:200391 9.8 9:11db4d5b:::benchmark_data_smithi151_261954_object97535:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1254)", 2024-09-13T14:58:55.355 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:57:35.058283+0000", 2024-09-13T14:58:55.355 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 80.164640114999997, 2024-09-13T14:58:55.355 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.98158972700000002, 2024-09-13T14:58:55.355 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:58:55.355 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:58:55.355 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:55.356 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:58:55.356 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.46034", 2024-09-13T14:58:55.356 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:58:55.356 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 200391 2024-09-13T14:58:55.356 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.356 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:58:55.356 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.356 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:58:55.356 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.058283+0000", 2024-09-13T14:58:55.356 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.356 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.356 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.356 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:58:55.356 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.058283+0000", 2024-09-13T14:58:55.357 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.357 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:58:55.357 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.058283+0000", 2024-09-13T14:58:55.357 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9699999999999996e-07 2024-09-13T14:58:55.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.357 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:58:55.357 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.058286+0000", 2024-09-13T14:58:55.357 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.531e-06 2024-09-13T14:58:55.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.357 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:58:55.358 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.058293+0000", 2024-09-13T14:58:55.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9450000000000004e-06 2024-09-13T14:58:55.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.358 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.358 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:58:55.358 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.058299+0000", 2024-09-13T14:58:55.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.806e-06 2024-09-13T14:58:55.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.358 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.358 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.358 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.058378+0000", 2024-09-13T14:58:55.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9179999999999997e-05 2024-09-13T14:58:55.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.359 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:58:55.359 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.058388+0000", 2024-09-13T14:58:55.359 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0219e-05 2024-09-13T14:58:55.359 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.359 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.359 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.031872+0000", 2024-09-13T14:58:55.359 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.97348421500000004 2024-09-13T14:58:55.359 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.359 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:58:55.359 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.031945+0000", 2024-09-13T14:58:55.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2575000000000002e-05 2024-09-13T14:58:55.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.360 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:58:55.360 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.032050+0000", 2024-09-13T14:58:55.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010551 2024-09-13T14:58:55.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.360 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:58:55.360 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.039488+0000", 2024-09-13T14:58:55.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0074379750000000003 2024-09-13T14:58:55.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.361 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.039694+0000", 2024-09-13T14:58:55.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00020611499999999999 2024-09-13T14:58:55.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.361 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.039741+0000", 2024-09-13T14:58:55.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6258000000000003e-05 2024-09-13T14:58:55.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.361 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:58:55.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.039836+0000", 2024-09-13T14:58:55.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4948e-05 2024-09-13T14:58:55.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:58:55.362 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.039873+0000", 2024-09-13T14:58:55.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6952999999999997e-05 2024-09-13T14:58:55.362 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.362 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:58:55.362 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.362 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.46034.0:200394 9.8 9:1d116ec0:::benchmark_data_smithi151_261954_object97538:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1254)", 2024-09-13T14:58:55.362 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:57:35.061062+0000", 2024-09-13T14:58:55.362 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 80.161861395000003, 2024-09-13T14:58:55.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.98698136800000003, 2024-09-13T14:58:55.363 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:58:55.363 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:58:55.363 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:55.363 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:58:55.363 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.46034", 2024-09-13T14:58:55.363 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:58:55.363 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 200394 2024-09-13T14:58:55.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.363 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:58:55.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.363 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:58:55.363 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.061062+0000", 2024-09-13T14:58:55.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:58:55.364 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.061062+0000", 2024-09-13T14:58:55.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:58:55.364 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.061063+0000", 2024-09-13T14:58:55.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3650000000000001e-06 2024-09-13T14:58:55.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:58:55.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.061068+0000", 2024-09-13T14:58:55.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2299999999999999e-06 2024-09-13T14:58:55.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:58:55.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.061084+0000", 2024-09-13T14:58:55.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6089000000000001e-05 2024-09-13T14:58:55.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:58:55.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.061092+0000", 2024-09-13T14:58:55.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7319999999999998e-06 2024-09-13T14:58:55.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.061148+0000", 2024-09-13T14:58:55.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5612999999999998e-05 2024-09-13T14:58:55.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:58:55.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.061157+0000", 2024-09-13T14:58:55.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0010000000000007e-06 2024-09-13T14:58:55.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.034283+0000", 2024-09-13T14:58:55.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.97312608899999997 2024-09-13T14:58:55.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:58:55.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.034358+0000", 2024-09-13T14:58:55.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5512999999999993e-05 2024-09-13T14:58:55.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:58:55.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.034420+0000", 2024-09-13T14:58:55.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2201999999999998e-05 2024-09-13T14:58:55.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:58:55.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.039494+0000", 2024-09-13T14:58:55.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0050731279999999997 2024-09-13T14:58:55.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.039916+0000", 2024-09-13T14:58:55.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000422512 2024-09-13T14:58:55.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.369 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.047925+0000", 2024-09-13T14:58:55.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0080094510000000008 2024-09-13T14:58:55.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.369 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:58:55.369 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.047994+0000", 2024-09-13T14:58:55.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8295999999999998e-05 2024-09-13T14:58:55.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.369 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:58:55.369 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.048043+0000", 2024-09-13T14:58:55.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9147000000000002e-05 2024-09-13T14:58:55.370 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.370 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:58:55.370 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.370 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.46034.0:200403 9.8 9:12dd93db:::benchmark_data_smithi151_261954_object97547:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1254)", 2024-09-13T14:58:55.370 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:57:35.068866+0000", 2024-09-13T14:58:55.370 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 80.154057183000006, 2024-09-13T14:58:55.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.97935454499999997, 2024-09-13T14:58:55.370 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:58:55.370 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:58:55.370 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:55.370 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:58:55.370 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.46034", 2024-09-13T14:58:55.371 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:58:55.371 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 200403 2024-09-13T14:58:55.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.371 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:58:55.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:58:55.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.068866+0000", 2024-09-13T14:58:55.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:58:55.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.068866+0000", 2024-09-13T14:58:55.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:58:55.372 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.068867+0000", 2024-09-13T14:58:55.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4499999999999996e-07 2024-09-13T14:58:55.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:58:55.372 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.068870+0000", 2024-09-13T14:58:55.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.676e-06 2024-09-13T14:58:55.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:58:55.372 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.068879+0000", 2024-09-13T14:58:55.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6920000000000008e-06 2024-09-13T14:58:55.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:58:55.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.068885+0000", 2024-09-13T14:58:55.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1489999999999999e-06 2024-09-13T14:58:55.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.068943+0000", 2024-09-13T14:58:55.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8103999999999998e-05 2024-09-13T14:58:55.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:58:55.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.068951+0000", 2024-09-13T14:58:55.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2060000000000007e-06 2024-09-13T14:58:55.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.036744+0000", 2024-09-13T14:58:55.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.96779236899999999 2024-09-13T14:58:55.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:58:55.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.036869+0000", 2024-09-13T14:58:55.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012578399999999999 2024-09-13T14:58:55.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:58:55.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.036989+0000", 2024-09-13T14:58:55.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011989699999999999 2024-09-13T14:58:55.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.039968+0000", 2024-09-13T14:58:55.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00297868 2024-09-13T14:58:55.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:58:55.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.042401+0000", 2024-09-13T14:58:55.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0024325760000000001 2024-09-13T14:58:55.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.048100+0000", 2024-09-13T14:58:55.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0056989199999999997 2024-09-13T14:58:55.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:58:55.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.048182+0000", 2024-09-13T14:58:55.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1979999999999998e-05 2024-09-13T14:58:55.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:58:55.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.048220+0000", 2024-09-13T14:58:55.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8766999999999997e-05 2024-09-13T14:58:55.377 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.377 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:58:55.377 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.377 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.46034.0:200409 9.8 9:15a8ee4f:::benchmark_data_smithi151_261954_object97553:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1254)", 2024-09-13T14:58:55.377 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:57:35.074463+0000", 2024-09-13T14:58:55.377 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 80.148459892000005, 2024-09-13T14:58:55.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.97936559700000003, 2024-09-13T14:58:55.377 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:58:55.377 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:58:55.377 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:55.377 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:58:55.378 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.46034", 2024-09-13T14:58:55.378 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:58:55.378 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 200409 2024-09-13T14:58:55.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.378 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:58:55.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:58:55.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.074463+0000", 2024-09-13T14:58:55.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:58:55.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.074463+0000", 2024-09-13T14:58:55.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:58:55.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:58:55.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.074464+0000", 2024-09-13T14:58:55.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4410000000000001e-06 2024-09-13T14:58:55.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:58:55.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.074470+0000", 2024-09-13T14:58:55.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5639999999999996e-06 2024-09-13T14:58:55.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:58:55.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.074486+0000", 2024-09-13T14:58:55.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6390999999999999e-05 2024-09-13T14:58:55.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:58:55.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.074495+0000", 2024-09-13T14:58:55.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.337e-06 2024-09-13T14:58:55.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.074592+0000", 2024-09-13T14:58:55.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7668000000000001e-05 2024-09-13T14:58:55.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.381 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T14:58:55.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.074602+0000", 2024-09-13T14:58:55.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7739999999999999e-06 2024-09-13T14:58:55.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.381 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:58:55.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.040006+0000", 2024-09-13T14:58:55.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.96540329899999999 2024-09-13T14:58:55.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.381 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:58:55.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.040099+0000", 2024-09-13T14:58:55.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3348000000000001e-05 2024-09-13T14:58:55.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:58:55.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.040192+0000", 2024-09-13T14:58:55.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2989e-05 2024-09-13T14:58:55.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:58:55.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.048232+0000", 2024-09-13T14:58:55.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0080400160000000005 2024-09-13T14:58:55.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.048275+0000", 2024-09-13T14:58:55.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3547000000000002e-05 2024-09-13T14:58:55.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:55.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.053722+0000", 2024-09-13T14:58:55.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0054461630000000004 2024-09-13T14:58:55.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T14:58:55.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.053776+0000", 2024-09-13T14:58:55.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4061999999999998e-05 2024-09-13T14:58:55.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:58:55.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:58:55.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T14:58:55.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.053829+0000", 2024-09-13T14:58:55.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2998000000000003e-05 2024-09-13T14:58:55.384 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.384 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:58:55.384 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.384 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:58:55.384 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:58:55.384 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:58:55.385 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:54 smithi184 ceph-mon[92527]: pgmap v2065: 119 pgs: 119 active+clean; 4.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-09-13T14:58:55.385 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:54 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:55.385 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:54 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:55.405 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:55.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:55.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:55.506 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:55.548 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:55.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:55.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:55.650 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:55.685 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:55.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:55.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:55.787 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:55.822 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:55.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:55.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:55.924 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:55.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:55.959 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:55.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:56.060 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:56.098 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:56.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:56.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:56.192 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 39 16 72976 72960 116.904 78.5625 0.00977051 0.00854961 2024-09-13T14:58:56.201 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:56.235 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:56.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:56.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:56.337 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:56.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:56.375 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:56.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:56.476 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:56.500 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:56.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:56.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:56.602 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:56.635 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:56.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:56.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:56.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:56.773 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:56.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:56.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:56.875 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:56.909 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:56.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:56.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:57.011 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:57.048 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:57.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:57.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:57.093 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:57.149 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:57.185 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:57.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:57.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:57.194 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:2024-09-13T14:58:57.194020+0000 min lat: 0.00119391 max lat: 0.812283 avg lat: 0.00861713 2024-09-13T14:58:57.194 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:58:57.194 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 40 16 74255 74239 115.975 79.9375 0.00788392 0.00861713 2024-09-13T14:58:57.281 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:56 smithi151 ceph-mon[104450]: pgmap v2066: 119 pgs: 119 active+clean; 4.2 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-13T14:58:57.281 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:56 smithi151 ceph-mon[108966]: pgmap v2066: 119 pgs: 119 active+clean; 4.2 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-13T14:58:57.287 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:57.313 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:57.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:57.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:57.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:56 smithi184 ceph-mon[92527]: pgmap v2066: 119 pgs: 119 active+clean; 4.2 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-13T14:58:57.414 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:57.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:57.453 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:57.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:57.554 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:57.591 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:57.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:57.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:57.693 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:57.732 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:57.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:57.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:57.834 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:57.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:57.880 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:57.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:57.981 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:58.018 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:58.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:58.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:58.120 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:58.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:58.161 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:58.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:58.194 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 41 16 75596 75580 115.19 83.8125 0.00805212 0.0086766 2024-09-13T14:58:58.263 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:58.301 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:58.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:58.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:58.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:57 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:58:58.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:57 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:58:58.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:57 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:58.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:57 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:58:58.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:57 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:58:58.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:57 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:58.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:57 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:58:58.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:57 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:58:58.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:57 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:58:58.403 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:58.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:58.438 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:58.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:58.492 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T14:58:58.539 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:58.577 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:58.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:58.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:58.679 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:58.718 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:58.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:58.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:58.820 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:58.854 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:58.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:58.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:58.932 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:58:58.955 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:58.993 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:58.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:58.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:59.095 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:59.145 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:59.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:59.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:59.195 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 42 16 76915 76899 114.41 82.4375 0.00757372 0.00873516 2024-09-13T14:58:59.248 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:59.301 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:59.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:59.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:59.404 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:59.445 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:59.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:59.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:59.547 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:59.585 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:58:59.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:59.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:59.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:58:59 smithi184 ceph-mon[92527]: pgmap v2067: 119 pgs: 119 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-09-13T14:58:59.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:58:59 smithi151 ceph-mon[104450]: pgmap v2067: 119 pgs: 119 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-09-13T14:58:59.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:58:59 smithi151 ceph-mon[108966]: pgmap v2067: 119 pgs: 119 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-09-13T14:58:59.687 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:59.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:59.729 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:59.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:59.831 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:59.862 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:58:59.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:59.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:59.904 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:58:59.905 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:58:59.941 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:58:59.941 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:58:59.964 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:59.979 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:58:59.979 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:59:00.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:00.006 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:00.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:00.017 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:59:00.018 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:59:00.054 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:59:00.054 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:59:00.084 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:59:00.084 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:59:00.107 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:00.119 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:59:00.119 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:59:00.150 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:00.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:00.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:00.153 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:59:00.153 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:59:00.188 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:00.194 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 43 16 78180 78164 113.588 79.0625 0.0211533 0.00879863 2024-09-13T14:59:00.253 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:00.287 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:00.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:00.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:00.389 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:00.423 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:00.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:00.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:00.505 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:00 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3648777728' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:00.506 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:00 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3648777728' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:00.514 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:00 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3648777728' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:00.514 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:59:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:59:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:59:00.525 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:00.558 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:00.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:00.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:00.661 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:00.701 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:00.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:00.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:00.803 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:00.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:00.841 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:00.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:00.942 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:00.963 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:00.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:00.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:01.065 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:01.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:01.109 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:01.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:01.194 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 44 16 79361 79345 112.684 73.8125 0.00611332 0.00886914 2024-09-13T14:59:01.211 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:01.238 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:59:01.239 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:59:01.250 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:01.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:01.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:01.352 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:01.391 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:01.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:01.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:01.494 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:01.532 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:01.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:01.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:01.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:01 smithi151 ceph-mon[104450]: pgmap v2068: 119 pgs: 119 active+clean; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-13T14:59:01.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:01 smithi151 ceph-mon[108966]: pgmap v2068: 119 pgs: 119 active+clean; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-13T14:59:01.634 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:01.669 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:01.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:01.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:01.771 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:01.803 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:01.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:01.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:01.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:01 smithi184 ceph-mon[92527]: pgmap v2068: 119 pgs: 119 active+clean; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-13T14:59:01.906 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:01.941 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T14:59:01.951 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:01.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:01.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:02.054 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:02.084 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:02.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:02.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:02.186 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:02.194 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 45 16 80561 80545 111.846 75 0.0115608 0.00893583 2024-09-13T14:59:02.216 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:02.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:02.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:02.318 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:02.357 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:59:02.359 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:02.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:02.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:02.461 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:02.492 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:02.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:02.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:02.571 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:02.571 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:02.594 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:02.630 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:02.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:02.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:02.635 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:02.733 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:02.771 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:02.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:02.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:02.873 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:02.906 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:02.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:02.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:03.008 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:03.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:03.048 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:03.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:03.150 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:03.195 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 46 16 81826 81810 111.133 79.0625 0.0107382 0.00899329 2024-09-13T14:59:03.226 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:03.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:03.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:03.330 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:03.367 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:03.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:03.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:03.469 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:03.507 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:03.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:03.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:03.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:03 smithi184 ceph-mon[92527]: pgmap v2069: 119 pgs: 119 active+clean; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 70 MiB/s wr, 1.13k op/s 2024-09-13T14:59:03.609 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:03.645 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:03.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:03.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:03.661 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:03 smithi151 ceph-mon[108966]: pgmap v2069: 119 pgs: 119 active+clean; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 70 MiB/s wr, 1.13k op/s 2024-09-13T14:59:03.661 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:03 smithi151 ceph-mon[104450]: pgmap v2069: 119 pgs: 119 active+clean; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 70 MiB/s wr, 1.13k op/s 2024-09-13T14:59:03.747 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:03.783 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:03.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:03.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:03.885 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:03.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:03.928 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:03.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:04.029 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:04.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:04.072 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:04.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:04.162 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:59:04.164 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T14:59:04.174 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:04.195 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 47 16 83042 83026 110.385 76 0.0205283 0.00905405 2024-09-13T14:59:04.212 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:04.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:04.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:04.314 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:04.322 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:59:04.322 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:59:04.360 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:04.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:04.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:04.462 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:04.498 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:04.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:04.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:04.600 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:04.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:04 smithi184 ceph-mon[92527]: pgmap v2070: 119 pgs: 119 active+clean; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-13T14:59:04.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:04 smithi184 ceph-mon[92527]: from='client.46265 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:59:04.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:04.639 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:04.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:04.740 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:04 smithi151 ceph-mon[104450]: pgmap v2070: 119 pgs: 119 active+clean; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-13T14:59:04.740 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:04 smithi151 ceph-mon[104450]: from='client.46265 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:59:04.741 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:04 smithi151 ceph-mon[108966]: pgmap v2070: 119 pgs: 119 active+clean; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-13T14:59:04.741 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:04 smithi151 ceph-mon[108966]: from='client.46265 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:59:04.741 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:59:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:59:04] "GET /metrics HTTP/1.1" 200 36699 "" "Prometheus/2.51.0" 2024-09-13T14:59:04.741 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:04.776 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:04.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:04.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:04.878 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:04.911 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:04.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:04.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:05.014 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:05.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:05.050 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:05.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:05.151 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:05.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:05.174 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:05.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:05.195 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 48 16 83952 83936 109.27 56.875 0.00439629 0.0091242 2024-09-13T14:59:05.275 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:05.297 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:05.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:05.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:05.399 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:05.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:05.444 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:05.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:05.545 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:05.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:05.568 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:05.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:05.572 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:59:05.668 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:05.699 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:05.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:05.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:05.801 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:05.821 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:05.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:05.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:05.923 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:05.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:05.959 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:05.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:06.061 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:06.093 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:06.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:06.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:06.195 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:06.196 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 49 16 84743 84727 108.049 49.4375 0.00901074 0.00924339 2024-09-13T14:59:06.227 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:06.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:06.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:06.329 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:06.351 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:06.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:06.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:06.452 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:06.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:06.475 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:06.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:06.576 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:06.608 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:06.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:06.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:06.710 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:06.731 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:06.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:06.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:06.833 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:06.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:06.864 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:06.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:06.965 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:06.997 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:06.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:06.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:07.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:07.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:06 smithi184 ceph-mon[92527]: pgmap v2071: 119 pgs: 119 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-13T14:59:07.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:06 smithi151 ceph-mon[104450]: pgmap v2071: 119 pgs: 119 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-13T14:59:07.112 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:06 smithi151 ceph-mon[108966]: pgmap v2071: 119 pgs: 119 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-13T14:59:07.139 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:07.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:07.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:07.196 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 50 16 85543 85527 106.888 50 0.0122443 0.00935135 2024-09-13T14:59:07.242 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:07.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:07.281 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:07.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:07.382 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:07.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:07.419 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:07.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:07.425 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":2070,"stamp":"2024-09-13T14:59:03.765623+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5273163725,"num_objects":80479,"num_object_clones":0,"num_object_copies":241437,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80516,"num_write_kb":5149023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":860775,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83611,"ondisk_log_size":83611,"up":357,"acting":357,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":357,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":21984408,"kb_used_data":15431320,"kb_used_omap":149,"kb_used_meta":6552874,"kb_avail":727976808,"statfs":{"total":767960285184,"available":745448251392,"internally_reserved":0,"allocated":15801671680,"data_stored":15780502815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":153401,"internal_metadata":6710143175},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,5,7],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1196032000,"num_objects":18250,"num_object_clones":0,"num_object_copies":54750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18250,"num_write_kb":1168000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006180"},"pg_stats":[{"pgid":"1.1d","version":"261'100","reported_seq":124,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.900438+0000","last_change":"2024-09-13T14:54:58.831284+0000","last_active":"2024-09-13T14:58:46.900438+0000","last_peered":"2024-09-13T14:58:46.900438+0000","last_clean":"2024-09-13T14:58:46.900438+0000","last_became_active":"2024-09-13T14:54:58.830090+0000","last_became_peered":"2024-09-13T14:54:58.830090+0000","last_unstale":"2024-09-13T14:58:46.900438+0000","last_undegraded":"2024-09-13T14:58:46.900438+0000","last_fullsized":"2024-09-13T14:58:46.900438+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:17.988345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:17.988345+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:02:05.633533+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":1986,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.906449+0000","last_change":"2024-09-13T14:55:52.137133+0000","last_active":"2024-09-13T14:58:46.906449+0000","last_peered":"2024-09-13T14:58:46.906449+0000","last_clean":"2024-09-13T14:58:46.906449+0000","last_became_active":"2024-09-13T14:55:41.862495+0000","last_became_peered":"2024-09-13T14:55:41.862495+0000","last_unstale":"2024-09-13T14:58:46.906449+0000","last_undegraded":"2024-09-13T14:58:46.906449+0000","last_fullsized":"2024-09-13T14:58:46.906449+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:51:35.574651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2280,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":66,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.857498+0000","last_change":"2024-09-13T14:58:05.196679+0000","last_active":"2024-09-13T14:58:34.857498+0000","last_peered":"2024-09-13T14:58:34.857498+0000","last_clean":"2024-09-13T14:58:34.857498+0000","last_became_active":"2024-09-13T14:58:05.196297+0000","last_became_peered":"2024-09-13T14:58:05.196297+0000","last_unstale":"2024-09-13T14:58:34.857498+0000","last_undegraded":"2024-09-13T14:58:34.857498+0000","last_fullsized":"2024-09-13T14:58:34.857498+0000","mapping_epoch":1267,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:56:17.644420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":270,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.909171+0000","last_change":"2024-09-13T14:45:19.810203+0000","last_active":"2024-09-13T14:58:46.909171+0000","last_peered":"2024-09-13T14:58:46.909171+0000","last_clean":"2024-09-13T14:58:46.909171+0000","last_became_active":"2024-09-13T14:45:19.809693+0000","last_became_peered":"2024-09-13T14:45:19.809693+0000","last_unstale":"2024-09-13T14:58:46.909171+0000","last_undegraded":"2024-09-13T14:58:46.909171+0000","last_fullsized":"2024-09-13T14:58:46.909171+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:20.219673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":269,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.908498+0000","last_change":"2024-09-13T14:45:19.807991+0000","last_active":"2024-09-13T14:58:46.908498+0000","last_peered":"2024-09-13T14:58:46.908498+0000","last_clean":"2024-09-13T14:58:46.908498+0000","last_became_active":"2024-09-13T14:45:19.807662+0000","last_became_peered":"2024-09-13T14:45:19.807662+0000","last_unstale":"2024-09-13T14:58:46.908498+0000","last_undegraded":"2024-09-13T14:58:46.908498+0000","last_fullsized":"2024-09-13T14:58:46.908498+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:29.306108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"261'100","reported_seq":1376,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.850362+0000","last_change":"2024-09-13T14:54:05.604842+0000","last_active":"2024-09-13T14:58:34.850362+0000","last_peered":"2024-09-13T14:58:34.850362+0000","last_clean":"2024-09-13T14:58:34.850362+0000","last_became_active":"2024-09-13T14:54:05.604072+0000","last_became_peered":"2024-09-13T14:54:05.604072+0000","last_unstale":"2024-09-13T14:58:34.850362+0000","last_undegraded":"2024-09-13T14:58:34.850362+0000","last_fullsized":"2024-09-13T14:58:34.850362+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_clean_scrub_stamp":"2024-09-13T14:44:54.259482+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:41:25.897474+0000","scrub_duration":0.015265569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":52,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.852629+0000","last_change":"2024-09-13T14:55:41.870022+0000","last_active":"2024-09-13T14:58:34.852629+0000","last_peered":"2024-09-13T14:58:34.852629+0000","last_clean":"2024-09-13T14:58:34.852629+0000","last_became_active":"2024-09-13T14:55:41.786811+0000","last_became_peered":"2024-09-13T14:55:41.786811+0000","last_unstale":"2024-09-13T14:58:34.852629+0000","last_undegraded":"2024-09-13T14:58:34.852629+0000","last_fullsized":"2024-09-13T14:58:34.852629+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:38:18.373670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":56,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.857822+0000","last_change":"2024-09-13T14:54:58.830988+0000","last_active":"2024-09-13T14:58:34.857822+0000","last_peered":"2024-09-13T14:58:34.857822+0000","last_clean":"2024-09-13T14:58:34.857822+0000","last_became_active":"2024-09-13T14:54:58.830686+0000","last_became_peered":"2024-09-13T14:54:58.830686+0000","last_unstale":"2024-09-13T14:58:34.857822+0000","last_undegraded":"2024-09-13T14:58:34.857822+0000","last_fullsized":"2024-09-13T14:58:34.857822+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:37.924642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"261'100","reported_seq":287,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.909333+0000","last_change":"2024-09-13T14:45:35.947332+0000","last_active":"2024-09-13T14:58:46.909333+0000","last_peered":"2024-09-13T14:58:46.909333+0000","last_clean":"2024-09-13T14:58:46.909333+0000","last_became_active":"2024-09-13T14:45:19.810775+0000","last_became_peered":"2024-09-13T14:45:19.810775+0000","last_unstale":"2024-09-13T14:58:46.909333+0000","last_undegraded":"2024-09-13T14:58:46.909333+0000","last_fullsized":"2024-09-13T14:58:46.909333+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:35.947265+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:43:41.569795+0000","last_clean_scrub_stamp":"2024-09-13T14:45:35.947265+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:00.606370+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":52,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.852480+0000","last_change":"2024-09-13T14:55:41.901299+0000","last_active":"2024-09-13T14:58:34.852480+0000","last_peered":"2024-09-13T14:58:34.852480+0000","last_clean":"2024-09-13T14:58:34.852480+0000","last_became_active":"2024-09-13T14:55:41.866844+0000","last_became_peered":"2024-09-13T14:55:41.866844+0000","last_unstale":"2024-09-13T14:58:34.852480+0000","last_undegraded":"2024-09-13T14:58:34.852480+0000","last_fullsized":"2024-09-13T14:58:34.852480+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:44.106308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":56,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.857067+0000","last_change":"2024-09-13T14:54:58.830335+0000","last_active":"2024-09-13T14:58:34.857067+0000","last_peered":"2024-09-13T14:58:34.857067+0000","last_clean":"2024-09-13T14:58:34.857067+0000","last_became_active":"2024-09-13T14:54:58.829978+0000","last_became_peered":"2024-09-13T14:54:58.829978+0000","last_unstale":"2024-09-13T14:58:34.857067+0000","last_undegraded":"2024-09-13T14:58:34.857067+0000","last_fullsized":"2024-09-13T14:58:34.857067+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:46.608799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":1943,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.908141+0000","last_change":"2024-09-13T14:54:58.832660+0000","last_active":"2024-09-13T14:58:46.908141+0000","last_peered":"2024-09-13T14:58:46.908141+0000","last_clean":"2024-09-13T14:58:46.908141+0000","last_became_active":"2024-09-13T14:54:58.831770+0000","last_became_peered":"2024-09-13T14:54:58.831770+0000","last_unstale":"2024-09-13T14:58:46.908141+0000","last_undegraded":"2024-09-13T14:58:46.908141+0000","last_fullsized":"2024-09-13T14:58:46.908141+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:07:14.004187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1951,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.912391+0000","last_change":"2024-09-13T14:54:05.592484+0000","last_active":"2024-09-13T14:58:46.912391+0000","last_peered":"2024-09-13T14:58:46.912391+0000","last_clean":"2024-09-13T14:58:46.912391+0000","last_became_active":"2024-09-13T14:54:05.592159+0000","last_became_peered":"2024-09-13T14:54:05.592159+0000","last_unstale":"2024-09-13T14:58:46.912391+0000","last_undegraded":"2024-09-13T14:58:46.912391+0000","last_fullsized":"2024-09-13T14:58:46.912391+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:02.607037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"261'100","reported_seq":269,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.908586+0000","last_change":"2024-09-13T14:45:19.813412+0000","last_active":"2024-09-13T14:58:46.908586+0000","last_peered":"2024-09-13T14:58:46.908586+0000","last_clean":"2024-09-13T14:58:46.908586+0000","last_became_active":"2024-09-13T14:45:19.812585+0000","last_became_peered":"2024-09-13T14:45:19.812585+0000","last_unstale":"2024-09-13T14:58:46.908586+0000","last_undegraded":"2024-09-13T14:58:46.908586+0000","last_fullsized":"2024-09-13T14:58:46.908586+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.614340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.614340+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:32:07.434109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":56,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.857908+0000","last_change":"2024-09-13T14:54:58.831851+0000","last_active":"2024-09-13T14:58:34.857908+0000","last_peered":"2024-09-13T14:58:34.857908+0000","last_clean":"2024-09-13T14:58:34.857908+0000","last_became_active":"2024-09-13T14:54:58.831477+0000","last_became_peered":"2024-09-13T14:54:58.831477+0000","last_unstale":"2024-09-13T14:58:34.857908+0000","last_undegraded":"2024-09-13T14:58:34.857908+0000","last_fullsized":"2024-09-13T14:58:34.857908+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:42.562225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":304,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.908704+0000","last_change":"2024-09-13T14:54:58.827147+0000","last_active":"2024-09-13T14:58:46.908704+0000","last_peered":"2024-09-13T14:58:46.908704+0000","last_clean":"2024-09-13T14:58:46.908704+0000","last_became_active":"2024-09-13T14:54:58.826783+0000","last_became_peered":"2024-09-13T14:54:58.826783+0000","last_unstale":"2024-09-13T14:58:46.908704+0000","last_undegraded":"2024-09-13T14:58:46.908704+0000","last_fullsized":"2024-09-13T14:58:46.908704+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:40:27.819584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.18","version":"261'100","reported_seq":52,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.851879+0000","last_change":"2024-09-13T14:57:11.038947+0000","last_active":"2024-09-13T14:58:34.851879+0000","last_peered":"2024-09-13T14:58:34.851879+0000","last_clean":"2024-09-13T14:58:34.851879+0000","last_became_active":"2024-09-13T14:55:41.838842+0000","last_became_peered":"2024-09-13T14:55:41.838842+0000","last_unstale":"2024-09-13T14:58:34.851879+0000","last_undegraded":"2024-09-13T14:58:34.851879+0000","last_fullsized":"2024-09-13T14:58:34.851879+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1246,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:56.548855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:56.548855+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:52.715319+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":125,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.900016+0000","last_change":"2024-09-13T14:54:58.821234+0000","last_active":"2024-09-13T14:58:46.900016+0000","last_peered":"2024-09-13T14:58:46.900016+0000","last_clean":"2024-09-13T14:58:46.900016+0000","last_became_active":"2024-09-13T14:54:58.820948+0000","last_became_peered":"2024-09-13T14:54:58.820948+0000","last_unstale":"2024-09-13T14:58:46.900016+0000","last_undegraded":"2024-09-13T14:58:46.900016+0000","last_fullsized":"2024-09-13T14:58:46.900016+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:58:51.172714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1954,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.907221+0000","last_change":"2024-09-13T14:45:19.815341+0000","last_active":"2024-09-13T14:58:46.907221+0000","last_peered":"2024-09-13T14:58:46.907221+0000","last_clean":"2024-09-13T14:58:46.907221+0000","last_became_active":"2024-09-13T14:45:19.810262+0000","last_became_peered":"2024-09-13T14:45:19.810262+0000","last_unstale":"2024-09-13T14:58:46.907221+0000","last_undegraded":"2024-09-13T14:58:46.907221+0000","last_fullsized":"2024-09-13T14:58:46.907221+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:39.847653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1b","version":"261'100","reported_seq":48,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.852696+0000","last_change":"2024-09-13T14:55:41.919999+0000","last_active":"2024-09-13T14:58:34.852696+0000","last_peered":"2024-09-13T14:58:34.852696+0000","last_clean":"2024-09-13T14:58:34.852696+0000","last_became_active":"2024-09-13T14:55:41.919444+0000","last_became_peered":"2024-09-13T14:55:41.919444+0000","last_unstale":"2024-09-13T14:58:34.852696+0000","last_undegraded":"2024-09-13T14:58:34.852696+0000","last_fullsized":"2024-09-13T14:58:34.852696+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:55.988431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:55.988431+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:22:19.385914+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":283,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.908144+0000","last_change":"2024-09-13T14:54:05.606614+0000","last_active":"2024-09-13T14:58:46.908144+0000","last_peered":"2024-09-13T14:58:46.908144+0000","last_clean":"2024-09-13T14:58:46.908144+0000","last_became_active":"2024-09-13T14:54:05.605524+0000","last_became_peered":"2024-09-13T14:54:05.605524+0000","last_unstale":"2024-09-13T14:58:46.908144+0000","last_undegraded":"2024-09-13T14:58:46.908144+0000","last_fullsized":"2024-09-13T14:58:46.908144+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:59.484071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":48,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.852902+0000","last_change":"2024-09-13T14:55:41.883426+0000","last_active":"2024-09-13T14:58:34.852902+0000","last_peered":"2024-09-13T14:58:34.852902+0000","last_clean":"2024-09-13T14:58:34.852902+0000","last_became_active":"2024-09-13T14:55:41.880724+0000","last_became_peered":"2024-09-13T14:55:41.880724+0000","last_unstale":"2024-09-13T14:58:34.852902+0000","last_undegraded":"2024-09-13T14:58:34.852902+0000","last_fullsized":"2024-09-13T14:58:34.852902+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:17.920508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1a","version":"261'100","reported_seq":56,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.857965+0000","last_change":"2024-09-13T14:54:58.831623+0000","last_active":"2024-09-13T14:58:34.857965+0000","last_peered":"2024-09-13T14:58:34.857965+0000","last_clean":"2024-09-13T14:58:34.857965+0000","last_became_active":"2024-09-13T14:54:58.831026+0000","last_became_peered":"2024-09-13T14:54:58.831026+0000","last_unstale":"2024-09-13T14:58:34.857965+0000","last_undegraded":"2024-09-13T14:58:34.857965+0000","last_fullsized":"2024-09-13T14:58:34.857965+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:46.985510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:46.985510+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:27:24.529132+0000","scrub_duration":0.0030525999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":290,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.908149+0000","last_change":"2024-09-13T14:54:05.606882+0000","last_active":"2024-09-13T14:58:46.908149+0000","last_peered":"2024-09-13T14:58:46.908149+0000","last_clean":"2024-09-13T14:58:46.908149+0000","last_became_active":"2024-09-13T14:54:05.605912+0000","last_became_peered":"2024-09-13T14:54:05.605912+0000","last_unstale":"2024-09-13T14:58:46.908149+0000","last_undegraded":"2024-09-13T14:58:46.908149+0000","last_fullsized":"2024-09-13T14:58:46.908149+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:05.445329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1998,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.906364+0000","last_change":"2024-09-13T14:55:41.868797+0000","last_active":"2024-09-13T14:58:46.906364+0000","last_peered":"2024-09-13T14:58:46.906364+0000","last_clean":"2024-09-13T14:58:46.906364+0000","last_became_active":"2024-09-13T14:55:41.865613+0000","last_became_peered":"2024-09-13T14:55:41.865613+0000","last_unstale":"2024-09-13T14:58:46.906364+0000","last_undegraded":"2024-09-13T14:58:46.906364+0000","last_fullsized":"2024-09-13T14:58:46.906364+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:16:29.230176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"261'100","reported_seq":226,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.850787+0000","last_change":"2024-09-13T14:55:41.868936+0000","last_active":"2024-09-13T14:58:34.850787+0000","last_peered":"2024-09-13T14:58:34.850787+0000","last_clean":"2024-09-13T14:58:34.850787+0000","last_became_active":"2024-09-13T14:55:41.865505+0000","last_became_peered":"2024-09-13T14:55:41.865505+0000","last_unstale":"2024-09-13T14:58:34.850787+0000","last_undegraded":"2024-09-13T14:58:34.850787+0000","last_fullsized":"2024-09-13T14:58:34.850787+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:38.214865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:38.214865+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:45:08.788150+0000","scrub_duration":0.0016389639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1841,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.908734+0000","last_change":"2024-09-13T14:30:15.576921+0000","last_active":"2024-09-13T14:58:46.908734+0000","last_peered":"2024-09-13T14:58:46.908734+0000","last_clean":"2024-09-13T14:58:46.908734+0000","last_became_active":"2024-09-13T14:30:15.576607+0000","last_became_peered":"2024-09-13T14:30:15.576607+0000","last_unstale":"2024-09-13T14:58:46.908734+0000","last_undegraded":"2024-09-13T14:58:46.908734+0000","last_fullsized":"2024-09-13T14:58:46.908734+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:35.162460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":48,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.852535+0000","last_change":"2024-09-13T14:55:41.868620+0000","last_active":"2024-09-13T14:58:34.852535+0000","last_peered":"2024-09-13T14:58:34.852535+0000","last_clean":"2024-09-13T14:58:34.852535+0000","last_became_active":"2024-09-13T14:55:41.868289+0000","last_became_peered":"2024-09-13T14:55:41.868289+0000","last_unstale":"2024-09-13T14:58:34.852535+0000","last_undegraded":"2024-09-13T14:58:34.852535+0000","last_fullsized":"2024-09-13T14:58:34.852535+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:17:28.109694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"261'100","reported_seq":346,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.912231+0000","last_change":"2024-09-13T14:54:58.824599+0000","last_active":"2024-09-13T14:58:46.912231+0000","last_peered":"2024-09-13T14:58:46.912231+0000","last_clean":"2024-09-13T14:58:46.912231+0000","last_became_active":"2024-09-13T14:54:58.824304+0000","last_became_peered":"2024-09-13T14:54:58.824304+0000","last_unstale":"2024-09-13T14:58:46.912231+0000","last_undegraded":"2024-09-13T14:58:46.912231+0000","last_fullsized":"2024-09-13T14:58:46.912231+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.037517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.037517+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:42.296498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":139,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.912268+0000","last_change":"2024-09-13T14:54:58.836713+0000","last_active":"2024-09-13T14:58:46.912268+0000","last_peered":"2024-09-13T14:58:46.912268+0000","last_clean":"2024-09-13T14:58:46.912268+0000","last_became_active":"2024-09-13T14:54:58.827723+0000","last_became_peered":"2024-09-13T14:54:58.827723+0000","last_unstale":"2024-09-13T14:58:46.912268+0000","last_undegraded":"2024-09-13T14:58:46.912268+0000","last_fullsized":"2024-09-13T14:58:46.912268+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:10:25.743550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":118,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.900300+0000","last_change":"2024-09-13T14:54:05.614569+0000","last_active":"2024-09-13T14:58:46.900300+0000","last_peered":"2024-09-13T14:58:46.900300+0000","last_clean":"2024-09-13T14:58:46.900300+0000","last_became_active":"2024-09-13T14:54:05.611810+0000","last_became_peered":"2024-09-13T14:54:05.611810+0000","last_unstale":"2024-09-13T14:58:46.900300+0000","last_undegraded":"2024-09-13T14:58:46.900300+0000","last_fullsized":"2024-09-13T14:58:46.900300+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:46:02.721438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"261'100","reported_seq":48,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.852859+0000","last_change":"2024-09-13T14:55:41.860977+0000","last_active":"2024-09-13T14:58:34.852859+0000","last_peered":"2024-09-13T14:58:34.852859+0000","last_clean":"2024-09-13T14:58:34.852859+0000","last_became_active":"2024-09-13T14:55:41.859488+0000","last_became_peered":"2024-09-13T14:55:41.859488+0000","last_unstale":"2024-09-13T14:58:34.852859+0000","last_undegraded":"2024-09-13T14:58:34.852859+0000","last_fullsized":"2024-09-13T14:58:34.852859+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:54.933410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:54.933410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:48.738084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":66,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.857309+0000","last_change":"2024-09-13T14:54:58.838308+0000","last_active":"2024-09-13T14:58:34.857309+0000","last_peered":"2024-09-13T14:58:34.857309+0000","last_clean":"2024-09-13T14:58:34.857309+0000","last_became_active":"2024-09-13T14:54:58.825174+0000","last_became_peered":"2024-09-13T14:54:58.825174+0000","last_unstale":"2024-09-13T14:58:34.857309+0000","last_undegraded":"2024-09-13T14:58:34.857309+0000","last_fullsized":"2024-09-13T14:58:34.857309+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:59:20.787507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":56,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.857161+0000","last_change":"2024-09-13T14:54:58.831327+0000","last_active":"2024-09-13T14:58:34.857161+0000","last_peered":"2024-09-13T14:58:34.857161+0000","last_clean":"2024-09-13T14:58:34.857161+0000","last_became_active":"2024-09-13T14:54:58.830414+0000","last_became_peered":"2024-09-13T14:54:58.830414+0000","last_unstale":"2024-09-13T14:58:34.857161+0000","last_undegraded":"2024-09-13T14:58:34.857161+0000","last_fullsized":"2024-09-13T14:58:34.857161+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:12.501968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"261'100","reported_seq":118,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.901275+0000","last_change":"2024-09-13T14:54:05.613604+0000","last_active":"2024-09-13T14:58:46.901275+0000","last_peered":"2024-09-13T14:58:46.901275+0000","last_clean":"2024-09-13T14:58:46.901275+0000","last_became_active":"2024-09-13T14:54:05.613247+0000","last_became_peered":"2024-09-13T14:54:05.613247+0000","last_unstale":"2024-09-13T14:58:46.901275+0000","last_undegraded":"2024-09-13T14:58:46.901275+0000","last_fullsized":"2024-09-13T14:58:46.901275+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:48.363045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:48.363045+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:17.100985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.f","version":"1275'5037","reported_seq":5052,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:59:02.285866+0000","last_change":"2024-09-13T14:58:06.200417+0000","last_active":"2024-09-13T14:59:02.285866+0000","last_peered":"2024-09-13T14:59:02.285866+0000","last_clean":"2024-09-13T14:59:02.285866+0000","last_became_active":"2024-09-13T14:58:06.200062+0000","last_became_peered":"2024-09-13T14:58:06.200062+0000","last_unstale":"2024-09-13T14:59:02.285866+0000","last_undegraded":"2024-09-13T14:59:02.285866+0000","last_fullsized":"2024-09-13T14:59:02.285866+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":5037,"log_dups_size":0,"ondisk_log_size":5037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:53:26.095129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330104832,"num_objects":5037,"num_object_clones":0,"num_object_copies":15111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5037,"num_write_kb":322368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":76,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.856887+0000","last_change":"2024-09-13T14:55:41.788238+0000","last_active":"2024-09-13T14:58:34.856887+0000","last_peered":"2024-09-13T14:58:34.856887+0000","last_clean":"2024-09-13T14:58:34.856887+0000","last_became_active":"2024-09-13T14:55:41.786422+0000","last_became_peered":"2024-09-13T14:55:41.786422+0000","last_unstale":"2024-09-13T14:58:34.856887+0000","last_undegraded":"2024-09-13T14:58:34.856887+0000","last_fullsized":"2024-09-13T14:58:34.856887+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:13:55.167916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":700,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.850989+0000","last_change":"2024-09-13T14:43:32.635070+0000","last_active":"2024-09-13T14:58:34.850989+0000","last_peered":"2024-09-13T14:58:34.850989+0000","last_clean":"2024-09-13T14:58:34.850989+0000","last_became_active":"2024-09-13T14:43:32.634734+0000","last_became_peered":"2024-09-13T14:43:32.634734+0000","last_unstale":"2024-09-13T14:58:34.850989+0000","last_undegraded":"2024-09-13T14:58:34.850989+0000","last_fullsized":"2024-09-13T14:58:34.850989+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:33.107010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":2016,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.907418+0000","last_change":"2024-09-13T14:54:58.846704+0000","last_active":"2024-09-13T14:58:46.907418+0000","last_peered":"2024-09-13T14:58:46.907418+0000","last_clean":"2024-09-13T14:58:46.907418+0000","last_became_active":"2024-09-13T14:54:58.830897+0000","last_became_peered":"2024-09-13T14:54:58.830897+0000","last_unstale":"2024-09-13T14:58:46.907418+0000","last_undegraded":"2024-09-13T14:58:46.907418+0000","last_fullsized":"2024-09-13T14:58:46.907418+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:31:37.389598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"261'100","reported_seq":298,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.908771+0000","last_change":"2024-09-13T14:55:41.845434+0000","last_active":"2024-09-13T14:58:46.908771+0000","last_peered":"2024-09-13T14:58:46.908771+0000","last_clean":"2024-09-13T14:58:46.908771+0000","last_became_active":"2024-09-13T14:55:41.845155+0000","last_became_peered":"2024-09-13T14:55:41.845155+0000","last_unstale":"2024-09-13T14:58:46.908771+0000","last_undegraded":"2024-09-13T14:58:46.908771+0000","last_fullsized":"2024-09-13T14:58:46.908771+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.710192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.710192+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:32.286774+0000","scrub_duration":0.0019250910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"1275'4966","reported_seq":4981,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:59:02.269889+0000","last_change":"2024-09-13T14:58:06.201145+0000","last_active":"2024-09-13T14:59:02.269889+0000","last_peered":"2024-09-13T14:59:02.269889+0000","last_clean":"2024-09-13T14:59:02.269889+0000","last_became_active":"2024-09-13T14:58:06.200769+0000","last_became_peered":"2024-09-13T14:58:06.200769+0000","last_unstale":"2024-09-13T14:59:02.269889+0000","last_undegraded":"2024-09-13T14:59:02.269889+0000","last_fullsized":"2024-09-13T14:59:02.269889+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":4966,"log_dups_size":0,"ondisk_log_size":4966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:18:09.369835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325451776,"num_objects":4966,"num_object_clones":0,"num_object_copies":14898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4966,"num_write_kb":317824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":173,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.908097+0000","last_change":"2024-09-13T14:52:54.233894+0000","last_active":"2024-09-13T14:58:46.908097+0000","last_peered":"2024-09-13T14:58:46.908097+0000","last_clean":"2024-09-13T14:58:46.908097+0000","last_became_active":"2024-09-13T14:52:54.233601+0000","last_became_peered":"2024-09-13T14:52:54.233601+0000","last_unstale":"2024-09-13T14:58:46.908097+0000","last_undegraded":"2024-09-13T14:58:46.908097+0000","last_fullsized":"2024-09-13T14:58:46.908097+0000","mapping_epoch":1195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:17:24.364701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":125,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.901321+0000","last_change":"2024-09-13T14:54:58.825887+0000","last_active":"2024-09-13T14:58:46.901321+0000","last_peered":"2024-09-13T14:58:46.901321+0000","last_clean":"2024-09-13T14:58:46.901321+0000","last_became_active":"2024-09-13T14:54:58.825350+0000","last_became_peered":"2024-09-13T14:54:58.825350+0000","last_unstale":"2024-09-13T14:58:46.901321+0000","last_undegraded":"2024-09-13T14:58:46.901321+0000","last_fullsized":"2024-09-13T14:58:46.901321+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:16.669031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":1874,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.912247+0000","last_change":"2024-09-13T14:30:15.581728+0000","last_active":"2024-09-13T14:58:46.912247+0000","last_peered":"2024-09-13T14:58:46.912247+0000","last_clean":"2024-09-13T14:58:46.912247+0000","last_became_active":"2024-09-13T14:30:15.581303+0000","last_became_peered":"2024-09-13T14:30:15.581303+0000","last_unstale":"2024-09-13T14:58:46.912247+0000","last_undegraded":"2024-09-13T14:58:46.912247+0000","last_fullsized":"2024-09-13T14:58:46.912247+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:10.027991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"261'100","reported_seq":56,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.857333+0000","last_change":"2024-09-13T14:54:58.822940+0000","last_active":"2024-09-13T14:58:34.857333+0000","last_peered":"2024-09-13T14:58:34.857333+0000","last_clean":"2024-09-13T14:58:34.857333+0000","last_became_active":"2024-09-13T14:54:58.822551+0000","last_became_peered":"2024-09-13T14:54:58.822551+0000","last_unstale":"2024-09-13T14:58:34.857333+0000","last_undegraded":"2024-09-13T14:58:34.857333+0000","last_fullsized":"2024-09-13T14:58:34.857333+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:51:31.310115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:51:31.310115+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:05:04.280014+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"1275'5133","reported_seq":5148,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:59:02.276255+0000","last_change":"2024-09-13T14:58:06.203698+0000","last_active":"2024-09-13T14:59:02.276255+0000","last_peered":"2024-09-13T14:59:02.276255+0000","last_clean":"2024-09-13T14:59:02.276255+0000","last_became_active":"2024-09-13T14:58:06.203402+0000","last_became_peered":"2024-09-13T14:58:06.203402+0000","last_unstale":"2024-09-13T14:59:02.276255+0000","last_undegraded":"2024-09-13T14:59:02.276255+0000","last_fullsized":"2024-09-13T14:59:02.276255+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":5133,"log_dups_size":0,"ondisk_log_size":5133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:59:27.300867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336396288,"num_objects":5133,"num_object_clones":0,"num_object_copies":15399,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5133,"num_write_kb":328512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1949,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.908019+0000","last_change":"2024-09-13T14:55:41.914117+0000","last_active":"2024-09-13T14:58:46.908019+0000","last_peered":"2024-09-13T14:58:46.908019+0000","last_clean":"2024-09-13T14:58:46.908019+0000","last_became_active":"2024-09-13T14:55:41.913777+0000","last_became_peered":"2024-09-13T14:55:41.913777+0000","last_unstale":"2024-09-13T14:58:46.908019+0000","last_undegraded":"2024-09-13T14:58:46.908019+0000","last_fullsized":"2024-09-13T14:58:46.908019+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:40:31.224332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1281,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.850869+0000","last_change":"2024-09-13T14:54:58.825971+0000","last_active":"2024-09-13T14:58:34.850869+0000","last_peered":"2024-09-13T14:58:34.850869+0000","last_clean":"2024-09-13T14:58:34.850869+0000","last_became_active":"2024-09-13T14:54:58.822356+0000","last_became_peered":"2024-09-13T14:54:58.822356+0000","last_unstale":"2024-09-13T14:58:34.850869+0000","last_undegraded":"2024-09-13T14:58:34.850869+0000","last_fullsized":"2024-09-13T14:58:34.850869+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:14.163203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":132,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.901485+0000","last_change":"2024-09-13T14:55:41.892125+0000","last_active":"2024-09-13T14:58:46.901485+0000","last_peered":"2024-09-13T14:58:46.901485+0000","last_clean":"2024-09-13T14:58:46.901485+0000","last_became_active":"2024-09-13T14:55:41.891812+0000","last_became_peered":"2024-09-13T14:55:41.891812+0000","last_unstale":"2024-09-13T14:58:46.901485+0000","last_undegraded":"2024-09-13T14:58:46.901485+0000","last_fullsized":"2024-09-13T14:58:46.901485+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:03.359541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"261'100","reported_seq":640,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.906881+0000","last_change":"2024-09-13T14:45:19.815050+0000","last_active":"2024-09-13T14:58:46.906881+0000","last_peered":"2024-09-13T14:58:46.906881+0000","last_clean":"2024-09-13T14:58:46.906881+0000","last_became_active":"2024-09-13T14:45:19.809732+0000","last_became_peered":"2024-09-13T14:45:19.809732+0000","last_unstale":"2024-09-13T14:58:46.906881+0000","last_undegraded":"2024-09-13T14:58:46.906881+0000","last_fullsized":"2024-09-13T14:58:46.906881+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:33.574443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:33.574443+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:51:40.261036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.c","version":"1275'4955","reported_seq":4970,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:59:02.258223+0000","last_change":"2024-09-13T14:58:06.202247+0000","last_active":"2024-09-13T14:59:02.258223+0000","last_peered":"2024-09-13T14:59:02.258223+0000","last_clean":"2024-09-13T14:59:02.258223+0000","last_became_active":"2024-09-13T14:58:06.201537+0000","last_became_peered":"2024-09-13T14:58:06.201537+0000","last_unstale":"2024-09-13T14:59:02.258223+0000","last_undegraded":"2024-09-13T14:59:02.258223+0000","last_fullsized":"2024-09-13T14:59:02.258223+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":4955,"log_dups_size":0,"ondisk_log_size":4955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:45:06.849570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324730880,"num_objects":4955,"num_object_clones":0,"num_object_copies":14865,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4955,"num_write_kb":317120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":139,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.900609+0000","last_change":"2024-09-13T14:55:41.929999+0000","last_active":"2024-09-13T14:58:46.900609+0000","last_peered":"2024-09-13T14:58:46.900609+0000","last_clean":"2024-09-13T14:58:46.900609+0000","last_became_active":"2024-09-13T14:55:41.926393+0000","last_became_peered":"2024-09-13T14:55:41.926393+0000","last_unstale":"2024-09-13T14:58:46.900609+0000","last_undegraded":"2024-09-13T14:58:46.900609+0000","last_fullsized":"2024-09-13T14:58:46.900609+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:16:40.228234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":118,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.901239+0000","last_change":"2024-09-13T14:54:05.605111+0000","last_active":"2024-09-13T14:58:46.901239+0000","last_peered":"2024-09-13T14:58:46.901239+0000","last_clean":"2024-09-13T14:58:46.901239+0000","last_became_active":"2024-09-13T14:54:05.599852+0000","last_became_peered":"2024-09-13T14:54:05.599852+0000","last_unstale":"2024-09-13T14:58:46.901239+0000","last_undegraded":"2024-09-13T14:58:46.901239+0000","last_fullsized":"2024-09-13T14:58:46.901239+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:11.238519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":48,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.852601+0000","last_change":"2024-09-13T14:55:41.839293+0000","last_active":"2024-09-13T14:58:34.852601+0000","last_peered":"2024-09-13T14:58:34.852601+0000","last_clean":"2024-09-13T14:58:34.852601+0000","last_became_active":"2024-09-13T14:55:41.838820+0000","last_became_peered":"2024-09-13T14:55:41.838820+0000","last_unstale":"2024-09-13T14:58:34.852601+0000","last_undegraded":"2024-09-13T14:58:34.852601+0000","last_fullsized":"2024-09-13T14:58:34.852601+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:14:27.811437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"261'100","reported_seq":287,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.912148+0000","last_change":"2024-09-13T14:45:36.965992+0000","last_active":"2024-09-13T14:58:46.912148+0000","last_peered":"2024-09-13T14:58:46.912148+0000","last_clean":"2024-09-13T14:58:46.912148+0000","last_became_active":"2024-09-13T14:45:19.811238+0000","last_became_peered":"2024-09-13T14:45:19.811238+0000","last_unstale":"2024-09-13T14:58:46.912148+0000","last_undegraded":"2024-09-13T14:58:46.912148+0000","last_fullsized":"2024-09-13T14:58:46.912148+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:36.965775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:45:36.965775+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:36:01.623251+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"1275'4981","reported_seq":4996,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:59:02.288450+0000","last_change":"2024-09-13T14:58:06.203687+0000","last_active":"2024-09-13T14:59:02.288450+0000","last_peered":"2024-09-13T14:59:02.288450+0000","last_clean":"2024-09-13T14:59:02.288450+0000","last_became_active":"2024-09-13T14:58:06.203340+0000","last_became_peered":"2024-09-13T14:58:06.203340+0000","last_unstale":"2024-09-13T14:59:02.288450+0000","last_undegraded":"2024-09-13T14:59:02.288450+0000","last_fullsized":"2024-09-13T14:59:02.288450+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":4981,"log_dups_size":0,"ondisk_log_size":4981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:17:52.886190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326434816,"num_objects":4981,"num_object_clones":0,"num_object_copies":14943,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4981,"num_write_kb":318784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":118,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.900665+0000","last_change":"2024-09-13T14:54:05.596068+0000","last_active":"2024-09-13T14:58:46.900665+0000","last_peered":"2024-09-13T14:58:46.900665+0000","last_clean":"2024-09-13T14:58:46.900665+0000","last_became_active":"2024-09-13T14:54:05.595743+0000","last_became_peered":"2024-09-13T14:54:05.595743+0000","last_unstale":"2024-09-13T14:58:46.900665+0000","last_undegraded":"2024-09-13T14:58:46.900665+0000","last_fullsized":"2024-09-13T14:58:46.900665+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:15:39.660370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":56,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.856668+0000","last_change":"2024-09-13T14:54:58.831309+0000","last_active":"2024-09-13T14:58:34.856668+0000","last_peered":"2024-09-13T14:58:34.856668+0000","last_clean":"2024-09-13T14:58:34.856668+0000","last_became_active":"2024-09-13T14:54:58.830950+0000","last_became_peered":"2024-09-13T14:54:58.830950+0000","last_unstale":"2024-09-13T14:58:34.856668+0000","last_undegraded":"2024-09-13T14:58:34.856668+0000","last_fullsized":"2024-09-13T14:58:34.856668+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:16:54.838006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":97,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.852365+0000","last_change":"2024-09-13T14:55:41.889668+0000","last_active":"2024-09-13T14:58:34.852365+0000","last_peered":"2024-09-13T14:58:34.852365+0000","last_clean":"2024-09-13T14:58:34.852365+0000","last_became_active":"2024-09-13T14:55:41.889195+0000","last_became_peered":"2024-09-13T14:55:41.889195+0000","last_unstale":"2024-09-13T14:58:34.852365+0000","last_undegraded":"2024-09-13T14:58:34.852365+0000","last_fullsized":"2024-09-13T14:58:34.852365+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:52.765208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"261'100","reported_seq":48,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.852696+0000","last_change":"2024-09-13T14:55:41.897451+0000","last_active":"2024-09-13T14:58:34.852696+0000","last_peered":"2024-09-13T14:58:34.852696+0000","last_clean":"2024-09-13T14:58:34.852696+0000","last_became_active":"2024-09-13T14:55:41.896485+0000","last_became_peered":"2024-09-13T14:55:41.896485+0000","last_unstale":"2024-09-13T14:58:34.852696+0000","last_undegraded":"2024-09-13T14:58:34.852696+0000","last_fullsized":"2024-09-13T14:58:34.852696+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:57.024823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:57.024823+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:25:55.484738+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"1275'5055","reported_seq":5070,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:59:02.312260+0000","last_change":"2024-09-13T14:58:06.202180+0000","last_active":"2024-09-13T14:59:02.312260+0000","last_peered":"2024-09-13T14:59:02.312260+0000","last_clean":"2024-09-13T14:59:02.312260+0000","last_became_active":"2024-09-13T14:58:06.201886+0000","last_became_peered":"2024-09-13T14:58:06.201886+0000","last_unstale":"2024-09-13T14:59:02.312260+0000","last_undegraded":"2024-09-13T14:59:02.312260+0000","last_fullsized":"2024-09-13T14:59:02.312260+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":5055,"log_dups_size":0,"ondisk_log_size":5055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:48.418531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331284480,"num_objects":5055,"num_object_clones":0,"num_object_copies":15165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5055,"num_write_kb":323520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":132,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.900934+0000","last_change":"2024-09-13T14:55:41.876729+0000","last_active":"2024-09-13T14:58:46.900934+0000","last_peered":"2024-09-13T14:58:46.900934+0000","last_clean":"2024-09-13T14:58:46.900934+0000","last_became_active":"2024-09-13T14:55:41.876280+0000","last_became_peered":"2024-09-13T14:55:41.876280+0000","last_unstale":"2024-09-13T14:58:46.900934+0000","last_undegraded":"2024-09-13T14:58:46.900934+0000","last_fullsized":"2024-09-13T14:58:46.900934+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:39:30.308792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":118,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.901971+0000","last_change":"2024-09-13T14:54:05.614107+0000","last_active":"2024-09-13T14:58:46.901971+0000","last_peered":"2024-09-13T14:58:46.901971+0000","last_clean":"2024-09-13T14:58:46.901971+0000","last_became_active":"2024-09-13T14:54:05.613429+0000","last_became_peered":"2024-09-13T14:54:05.613429+0000","last_unstale":"2024-09-13T14:58:46.901971+0000","last_undegraded":"2024-09-13T14:58:46.901971+0000","last_fullsized":"2024-09-13T14:58:46.901971+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:51.312413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1956,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.912513+0000","last_change":"2024-09-13T14:45:19.812713+0000","last_active":"2024-09-13T14:58:46.912513+0000","last_peered":"2024-09-13T14:58:46.912513+0000","last_clean":"2024-09-13T14:58:46.912513+0000","last_became_active":"2024-09-13T14:45:19.812203+0000","last_became_peered":"2024-09-13T14:45:19.812203+0000","last_unstale":"2024-09-13T14:58:46.912513+0000","last_undegraded":"2024-09-13T14:58:46.912513+0000","last_fullsized":"2024-09-13T14:58:46.912513+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:30:46.149347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"261'100","reported_seq":1415,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.849639+0000","last_change":"2024-09-13T14:54:58.820128+0000","last_active":"2024-09-13T14:58:34.849639+0000","last_peered":"2024-09-13T14:58:34.849639+0000","last_clean":"2024-09-13T14:58:34.849639+0000","last_became_active":"2024-09-13T14:54:58.819324+0000","last_became_peered":"2024-09-13T14:54:58.819324+0000","last_unstale":"2024-09-13T14:58:34.849639+0000","last_undegraded":"2024-09-13T14:58:34.849639+0000","last_fullsized":"2024-09-13T14:58:34.849639+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:42.577243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:42.577243+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:50:11.323977+0000","scrub_duration":0.001367839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"1275'5156","reported_seq":5181,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:59:03.605853+0000","last_change":"2024-09-13T14:58:06.203607+0000","last_active":"2024-09-13T14:59:03.605853+0000","last_peered":"2024-09-13T14:59:03.605853+0000","last_clean":"2024-09-13T14:59:03.605853+0000","last_became_active":"2024-09-13T14:58:06.203316+0000","last_became_peered":"2024-09-13T14:58:06.203316+0000","last_unstale":"2024-09-13T14:59:03.605853+0000","last_undegraded":"2024-09-13T14:59:03.605853+0000","last_fullsized":"2024-09-13T14:59:03.605853+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":5156,"log_dups_size":0,"ondisk_log_size":5156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:55.731243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337903616,"num_objects":5156,"num_object_clones":0,"num_object_copies":15468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5156,"num_write_kb":329984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1964,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.907175+0000","last_change":"2024-09-13T14:54:05.607857+0000","last_active":"2024-09-13T14:58:46.907175+0000","last_peered":"2024-09-13T14:58:46.907175+0000","last_clean":"2024-09-13T14:58:46.907175+0000","last_became_active":"2024-09-13T14:54:05.607112+0000","last_became_peered":"2024-09-13T14:54:05.607112+0000","last_unstale":"2024-09-13T14:58:46.907175+0000","last_undegraded":"2024-09-13T14:58:46.907175+0000","last_fullsized":"2024-09-13T14:58:46.907175+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:41:23.733552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2000,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.906132+0000","last_change":"2024-09-13T14:54:58.832430+0000","last_active":"2024-09-13T14:58:46.906132+0000","last_peered":"2024-09-13T14:58:46.906132+0000","last_clean":"2024-09-13T14:58:46.906132+0000","last_became_active":"2024-09-13T14:54:58.832091+0000","last_became_peered":"2024-09-13T14:54:58.832091+0000","last_unstale":"2024-09-13T14:58:46.906132+0000","last_undegraded":"2024-09-13T14:58:46.906132+0000","last_fullsized":"2024-09-13T14:58:46.906132+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:50.845541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1298,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.849690+0000","last_change":"2024-09-13T14:54:05.604586+0000","last_active":"2024-09-13T14:58:34.849690+0000","last_peered":"2024-09-13T14:58:34.849690+0000","last_clean":"2024-09-13T14:58:34.849690+0000","last_became_active":"2024-09-13T14:54:05.603632+0000","last_became_peered":"2024-09-13T14:54:05.603632+0000","last_unstale":"2024-09-13T14:58:34.849690+0000","last_undegraded":"2024-09-13T14:58:34.849690+0000","last_fullsized":"2024-09-13T14:58:34.849690+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:33.197882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"261'100","reported_seq":125,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.903578+0000","last_change":"2024-09-13T14:54:58.828206+0000","last_active":"2024-09-13T14:58:46.903578+0000","last_peered":"2024-09-13T14:58:46.903578+0000","last_clean":"2024-09-13T14:58:46.903578+0000","last_became_active":"2024-09-13T14:54:58.827560+0000","last_became_peered":"2024-09-13T14:54:58.827560+0000","last_unstale":"2024-09-13T14:58:46.903578+0000","last_undegraded":"2024-09-13T14:58:46.903578+0000","last_fullsized":"2024-09-13T14:58:46.903578+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.008264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.008264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:23.090746+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"1275'4886","reported_seq":4911,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:59.172779+0000","last_change":"2024-09-13T14:58:06.200816+0000","last_active":"2024-09-13T14:58:59.172779+0000","last_peered":"2024-09-13T14:58:59.172779+0000","last_clean":"2024-09-13T14:58:59.172779+0000","last_became_active":"2024-09-13T14:58:06.200474+0000","last_became_peered":"2024-09-13T14:58:06.200474+0000","last_unstale":"2024-09-13T14:58:59.172779+0000","last_undegraded":"2024-09-13T14:58:59.172779+0000","last_fullsized":"2024-09-13T14:58:59.172779+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":4886,"log_dups_size":0,"ondisk_log_size":4886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:55.523165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320208896,"num_objects":4886,"num_object_clones":0,"num_object_copies":14658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4886,"num_write_kb":312704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":56,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.857331+0000","last_change":"2024-09-13T14:54:58.828591+0000","last_active":"2024-09-13T14:58:34.857331+0000","last_peered":"2024-09-13T14:58:34.857331+0000","last_clean":"2024-09-13T14:58:34.857331+0000","last_became_active":"2024-09-13T14:54:58.828212+0000","last_became_peered":"2024-09-13T14:54:58.828212+0000","last_unstale":"2024-09-13T14:58:34.857331+0000","last_undegraded":"2024-09-13T14:58:34.857331+0000","last_fullsized":"2024-09-13T14:58:34.857331+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:50:15.315121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":305,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.908894+0000","last_change":"2024-09-13T14:55:41.835887+0000","last_active":"2024-09-13T14:58:46.908894+0000","last_peered":"2024-09-13T14:58:46.908894+0000","last_clean":"2024-09-13T14:58:46.908894+0000","last_became_active":"2024-09-13T14:55:41.834413+0000","last_became_peered":"2024-09-13T14:55:41.834413+0000","last_unstale":"2024-09-13T14:58:46.908894+0000","last_undegraded":"2024-09-13T14:58:46.908894+0000","last_fullsized":"2024-09-13T14:58:46.908894+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:01.850578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":83,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.852776+0000","last_change":"2024-09-13T14:55:41.839256+0000","last_active":"2024-09-13T14:58:34.852776+0000","last_peered":"2024-09-13T14:58:34.852776+0000","last_clean":"2024-09-13T14:58:34.852776+0000","last_became_active":"2024-09-13T14:55:41.838918+0000","last_became_peered":"2024-09-13T14:55:41.838918+0000","last_unstale":"2024-09-13T14:58:34.852776+0000","last_undegraded":"2024-09-13T14:58:34.852776+0000","last_fullsized":"2024-09-13T14:58:34.852776+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:31:58.442849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"261'100","reported_seq":1386,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.849781+0000","last_change":"2024-09-13T14:44:28.186029+0000","last_active":"2024-09-13T14:58:34.849781+0000","last_peered":"2024-09-13T14:58:34.849781+0000","last_clean":"2024-09-13T14:58:34.849781+0000","last_became_active":"2024-09-13T14:42:14.927771+0000","last_became_peered":"2024-09-13T14:42:14.927771+0000","last_unstale":"2024-09-13T14:58:34.849781+0000","last_undegraded":"2024-09-13T14:58:34.849781+0000","last_fullsized":"2024-09-13T14:58:34.849781+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:28.185949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:28.185949+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:38.363961+0000","scrub_duration":0.0018071129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"1275'5085","reported_seq":5100,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:59:02.263878+0000","last_change":"2024-09-13T14:58:06.204014+0000","last_active":"2024-09-13T14:59:02.263878+0000","last_peered":"2024-09-13T14:59:02.263878+0000","last_clean":"2024-09-13T14:59:02.263878+0000","last_became_active":"2024-09-13T14:58:06.203193+0000","last_became_peered":"2024-09-13T14:58:06.203193+0000","last_unstale":"2024-09-13T14:59:02.263878+0000","last_undegraded":"2024-09-13T14:59:02.263878+0000","last_fullsized":"2024-09-13T14:59:02.263878+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":5085,"log_dups_size":0,"ondisk_log_size":5085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:05:05.742511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333250560,"num_objects":5085,"num_object_clones":0,"num_object_copies":15255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5085,"num_write_kb":325440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":125,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.901004+0000","last_change":"2024-09-13T14:54:58.829485+0000","last_active":"2024-09-13T14:58:46.901004+0000","last_peered":"2024-09-13T14:58:46.901004+0000","last_clean":"2024-09-13T14:58:46.901004+0000","last_became_active":"2024-09-13T14:54:58.829141+0000","last_became_peered":"2024-09-13T14:54:58.829141+0000","last_unstale":"2024-09-13T14:58:46.901004+0000","last_undegraded":"2024-09-13T14:58:46.901004+0000","last_fullsized":"2024-09-13T14:58:46.901004+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:10:05.474311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":291,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.908257+0000","last_change":"2024-09-13T14:54:05.609380+0000","last_active":"2024-09-13T14:58:46.908257+0000","last_peered":"2024-09-13T14:58:46.908257+0000","last_clean":"2024-09-13T14:58:46.908257+0000","last_became_active":"2024-09-13T14:54:05.606282+0000","last_became_peered":"2024-09-13T14:54:05.606282+0000","last_unstale":"2024-09-13T14:58:46.908257+0000","last_undegraded":"2024-09-13T14:58:46.908257+0000","last_fullsized":"2024-09-13T14:58:46.908257+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:24:34.588847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"261'100","reported_seq":48,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.852215+0000","last_change":"2024-09-13T14:55:41.784086+0000","last_active":"2024-09-13T14:58:34.852215+0000","last_peered":"2024-09-13T14:58:34.852215+0000","last_clean":"2024-09-13T14:58:34.852215+0000","last_became_active":"2024-09-13T14:55:41.783492+0000","last_became_peered":"2024-09-13T14:55:41.783492+0000","last_unstale":"2024-09-13T14:58:34.852215+0000","last_undegraded":"2024-09-13T14:58:34.852215+0000","last_fullsized":"2024-09-13T14:58:34.852215+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:49.280478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:49.280478+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:16:55.041094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"1275'5046","reported_seq":5061,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:59:02.275997+0000","last_change":"2024-09-13T14:58:06.200769+0000","last_active":"2024-09-13T14:59:02.275997+0000","last_peered":"2024-09-13T14:59:02.275997+0000","last_clean":"2024-09-13T14:59:02.275997+0000","last_became_active":"2024-09-13T14:58:06.200457+0000","last_became_peered":"2024-09-13T14:58:06.200457+0000","last_unstale":"2024-09-13T14:59:02.275997+0000","last_undegraded":"2024-09-13T14:59:02.275997+0000","last_fullsized":"2024-09-13T14:59:02.275997+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":5046,"log_dups_size":0,"ondisk_log_size":5046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:25:18.549925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330694656,"num_objects":5046,"num_object_clones":0,"num_object_copies":15138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5046,"num_write_kb":322944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":56,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.857702+0000","last_change":"2024-09-13T14:54:58.829851+0000","last_active":"2024-09-13T14:58:34.857702+0000","last_peered":"2024-09-13T14:58:34.857702+0000","last_clean":"2024-09-13T14:58:34.857702+0000","last_became_active":"2024-09-13T14:54:58.829565+0000","last_became_peered":"2024-09-13T14:54:58.829565+0000","last_unstale":"2024-09-13T14:58:34.857702+0000","last_undegraded":"2024-09-13T14:58:34.857702+0000","last_fullsized":"2024-09-13T14:58:34.857702+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:00:19.211398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":56,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.857557+0000","last_change":"2024-09-13T14:54:58.827628+0000","last_active":"2024-09-13T14:58:34.857557+0000","last_peered":"2024-09-13T14:58:34.857557+0000","last_clean":"2024-09-13T14:58:34.857557+0000","last_became_active":"2024-09-13T14:54:58.827371+0000","last_became_peered":"2024-09-13T14:54:58.827371+0000","last_unstale":"2024-09-13T14:58:34.857557+0000","last_undegraded":"2024-09-13T14:58:34.857557+0000","last_fullsized":"2024-09-13T14:58:34.857557+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:21:17.887314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"261'100","reported_seq":132,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.903861+0000","last_change":"2024-09-13T14:55:41.949447+0000","last_active":"2024-09-13T14:58:46.903861+0000","last_peered":"2024-09-13T14:58:46.903861+0000","last_clean":"2024-09-13T14:58:46.903861+0000","last_became_active":"2024-09-13T14:55:41.949062+0000","last_became_peered":"2024-09-13T14:55:41.949062+0000","last_unstale":"2024-09-13T14:58:46.903861+0000","last_undegraded":"2024-09-13T14:58:46.903861+0000","last_fullsized":"2024-09-13T14:58:46.903861+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:47.405845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:47.405845+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:41:06.083975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3","version":"1275'5056","reported_seq":5081,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:59:02.994867+0000","last_change":"2024-09-13T14:58:06.200017+0000","last_active":"2024-09-13T14:59:02.994867+0000","last_peered":"2024-09-13T14:59:02.994867+0000","last_clean":"2024-09-13T14:59:02.994867+0000","last_became_active":"2024-09-13T14:58:06.199665+0000","last_became_peered":"2024-09-13T14:58:06.199665+0000","last_unstale":"2024-09-13T14:59:02.994867+0000","last_undegraded":"2024-09-13T14:59:02.994867+0000","last_fullsized":"2024-09-13T14:59:02.994867+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":5056,"log_dups_size":0,"ondisk_log_size":5056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:09.742112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331350016,"num_objects":5056,"num_object_clones":0,"num_object_copies":15168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5056,"num_write_kb":323584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":56,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.857514+0000","last_change":"2024-09-13T14:54:58.832125+0000","last_active":"2024-09-13T14:58:34.857514+0000","last_peered":"2024-09-13T14:58:34.857514+0000","last_clean":"2024-09-13T14:58:34.857514+0000","last_became_active":"2024-09-13T14:54:58.831788+0000","last_became_peered":"2024-09-13T14:54:58.831788+0000","last_unstale":"2024-09-13T14:58:34.857514+0000","last_undegraded":"2024-09-13T14:58:34.857514+0000","last_fullsized":"2024-09-13T14:58:34.857514+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:40.774847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":56,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.857916+0000","last_change":"2024-09-13T14:54:58.831785+0000","last_active":"2024-09-13T14:58:34.857916+0000","last_peered":"2024-09-13T14:58:34.857916+0000","last_clean":"2024-09-13T14:58:34.857916+0000","last_became_active":"2024-09-13T14:54:58.831331+0000","last_became_peered":"2024-09-13T14:54:58.831331+0000","last_unstale":"2024-09-13T14:58:34.857916+0000","last_undegraded":"2024-09-13T14:58:34.857916+0000","last_fullsized":"2024-09-13T14:58:34.857916+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:45.394867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"261'100","reported_seq":48,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.851704+0000","last_change":"2024-09-13T14:55:41.844203+0000","last_active":"2024-09-13T14:58:34.851704+0000","last_peered":"2024-09-13T14:58:34.851704+0000","last_clean":"2024-09-13T14:58:34.851704+0000","last_became_active":"2024-09-13T14:55:41.843897+0000","last_became_peered":"2024-09-13T14:55:41.843897+0000","last_unstale":"2024-09-13T14:58:34.851704+0000","last_undegraded":"2024-09-13T14:58:34.851704+0000","last_fullsized":"2024-09-13T14:58:34.851704+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:55.816857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:55.816857+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:42:54.571599+0000","scrub_duration":270,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2","version":"1275'4889","reported_seq":4914,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:59.146132+0000","last_change":"2024-09-13T14:58:06.201242+0000","last_active":"2024-09-13T14:58:59.146132+0000","last_peered":"2024-09-13T14:58:59.146132+0000","last_clean":"2024-09-13T14:58:59.146132+0000","last_became_active":"2024-09-13T14:58:06.200965+0000","last_became_peered":"2024-09-13T14:58:06.200965+0000","last_unstale":"2024-09-13T14:58:59.146132+0000","last_undegraded":"2024-09-13T14:58:59.146132+0000","last_fullsized":"2024-09-13T14:58:59.146132+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":4889,"log_dups_size":0,"ondisk_log_size":4889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:13:13.810742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320405504,"num_objects":4889,"num_object_clones":0,"num_object_copies":14667,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4889,"num_write_kb":312896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":56,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.857440+0000","last_change":"2024-09-13T14:54:58.827402+0000","last_active":"2024-09-13T14:58:34.857440+0000","last_peered":"2024-09-13T14:58:34.857440+0000","last_clean":"2024-09-13T14:58:34.857440+0000","last_became_active":"2024-09-13T14:54:58.827119+0000","last_became_peered":"2024-09-13T14:54:58.827119+0000","last_unstale":"2024-09-13T14:58:34.857440+0000","last_undegraded":"2024-09-13T14:58:34.857440+0000","last_fullsized":"2024-09-13T14:58:34.857440+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:06.078013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":318,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.908510+0000","last_change":"2024-09-13T14:54:58.843575+0000","last_active":"2024-09-13T14:58:46.908510+0000","last_peered":"2024-09-13T14:58:46.908510+0000","last_clean":"2024-09-13T14:58:46.908510+0000","last_became_active":"2024-09-13T14:54:58.822033+0000","last_became_peered":"2024-09-13T14:54:58.822033+0000","last_unstale":"2024-09-13T14:58:46.908510+0000","last_undegraded":"2024-09-13T14:58:46.908510+0000","last_fullsized":"2024-09-13T14:58:46.908510+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:44.356090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"261'100","reported_seq":1405,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.849644+0000","last_change":"2024-09-13T14:45:19.798553+0000","last_active":"2024-09-13T14:58:34.849644+0000","last_peered":"2024-09-13T14:58:34.849644+0000","last_clean":"2024-09-13T14:58:34.849644+0000","last_became_active":"2024-09-13T14:45:19.798189+0000","last_became_peered":"2024-09-13T14:45:19.798189+0000","last_unstale":"2024-09-13T14:58:34.849644+0000","last_undegraded":"2024-09-13T14:58:34.849644+0000","last_fullsized":"2024-09-13T14:58:34.849644+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:29.169325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:29.169325+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:10.412666+0000","scrub_duration":0.0019358890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"1275'5108","reported_seq":5133,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:59:03.017899+0000","last_change":"2024-09-13T14:58:06.201392+0000","last_active":"2024-09-13T14:59:03.017899+0000","last_peered":"2024-09-13T14:59:03.017899+0000","last_clean":"2024-09-13T14:59:03.017899+0000","last_became_active":"2024-09-13T14:58:06.201088+0000","last_became_peered":"2024-09-13T14:58:06.201088+0000","last_unstale":"2024-09-13T14:59:03.017899+0000","last_undegraded":"2024-09-13T14:59:03.017899+0000","last_fullsized":"2024-09-13T14:59:03.017899+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":5108,"log_dups_size":0,"ondisk_log_size":5108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:44:35.707112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334757888,"num_objects":5108,"num_object_clones":0,"num_object_copies":15324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5108,"num_write_kb":326912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1973,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.908405+0000","last_change":"2024-09-13T14:54:58.834208+0000","last_active":"2024-09-13T14:58:46.908405+0000","last_peered":"2024-09-13T14:58:46.908405+0000","last_clean":"2024-09-13T14:58:46.908405+0000","last_became_active":"2024-09-13T14:54:58.833071+0000","last_became_peered":"2024-09-13T14:54:58.833071+0000","last_unstale":"2024-09-13T14:58:46.908405+0000","last_undegraded":"2024-09-13T14:58:46.908405+0000","last_fullsized":"2024-09-13T14:58:46.908405+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:25:27.099352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":48,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.852395+0000","last_change":"2024-09-13T14:55:41.848811+0000","last_active":"2024-09-13T14:58:34.852395+0000","last_peered":"2024-09-13T14:58:34.852395+0000","last_clean":"2024-09-13T14:58:34.852395+0000","last_became_active":"2024-09-13T14:55:41.848542+0000","last_became_peered":"2024-09-13T14:55:41.848542+0000","last_unstale":"2024-09-13T14:58:34.852395+0000","last_undegraded":"2024-09-13T14:58:34.852395+0000","last_fullsized":"2024-09-13T14:58:34.852395+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:48.534022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"261'100","reported_seq":66,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.857703+0000","last_change":"2024-09-13T14:57:59.015556+0000","last_active":"2024-09-13T14:58:34.857703+0000","last_peered":"2024-09-13T14:58:34.857703+0000","last_clean":"2024-09-13T14:58:34.857703+0000","last_became_active":"2024-09-13T14:54:58.826820+0000","last_became_peered":"2024-09-13T14:54:58.826820+0000","last_unstale":"2024-09-13T14:58:34.857703+0000","last_undegraded":"2024-09-13T14:58:34.857703+0000","last_fullsized":"2024-09-13T14:58:34.857703+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:57:59.015484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:57:59.015484+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:49.345864+0000","scrub_duration":0.0015442190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"1275'5168","reported_seq":5192,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:59:03.642559+0000","last_change":"2024-09-13T14:58:06.202476+0000","last_active":"2024-09-13T14:59:03.642559+0000","last_peered":"2024-09-13T14:59:03.642559+0000","last_clean":"2024-09-13T14:59:03.642559+0000","last_became_active":"2024-09-13T14:58:06.202211+0000","last_became_peered":"2024-09-13T14:58:06.202211+0000","last_unstale":"2024-09-13T14:59:03.642559+0000","last_undegraded":"2024-09-13T14:59:03.642559+0000","last_fullsized":"2024-09-13T14:59:03.642559+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":5168,"log_dups_size":0,"ondisk_log_size":5168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:06:50.307820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338690048,"num_objects":5168,"num_object_clones":0,"num_object_copies":15504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5168,"num_write_kb":330752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1271,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.855511+0000","last_change":"2024-09-13T14:58:05.193719+0000","last_active":"2024-09-13T14:58:34.855511+0000","last_peered":"2024-09-13T14:58:34.855511+0000","last_clean":"2024-09-13T14:58:34.855511+0000","last_became_active":"2024-09-13T14:58:05.193367+0000","last_became_peered":"2024-09-13T14:58:05.193367+0000","last_unstale":"2024-09-13T14:58:34.855511+0000","last_undegraded":"2024-09-13T14:58:34.855511+0000","last_fullsized":"2024-09-13T14:58:34.855511+0000","mapping_epoch":1267,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:38:46.329171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":1269,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.850776+0000","last_change":"2024-09-13T14:55:41.919407+0000","last_active":"2024-09-13T14:58:34.850776+0000","last_peered":"2024-09-13T14:58:34.850776+0000","last_clean":"2024-09-13T14:58:34.850776+0000","last_became_active":"2024-09-13T14:55:41.793557+0000","last_became_peered":"2024-09-13T14:55:41.793557+0000","last_unstale":"2024-09-13T14:58:34.850776+0000","last_undegraded":"2024-09-13T14:58:34.850776+0000","last_fullsized":"2024-09-13T14:58:34.850776+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:57:51.874930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"261'100","reported_seq":48,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.852946+0000","last_change":"2024-09-13T14:55:41.785240+0000","last_active":"2024-09-13T14:58:34.852946+0000","last_peered":"2024-09-13T14:58:34.852946+0000","last_clean":"2024-09-13T14:58:34.852946+0000","last_became_active":"2024-09-13T14:55:41.784916+0000","last_became_peered":"2024-09-13T14:55:41.784916+0000","last_unstale":"2024-09-13T14:58:34.852946+0000","last_undegraded":"2024-09-13T14:58:34.852946+0000","last_fullsized":"2024-09-13T14:58:34.852946+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:49:21.722939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.722939+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:57.374891+0000","scrub_duration":0.0016642790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"1275'4895","reported_seq":4915,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:59:00.904559+0000","last_change":"2024-09-13T14:58:06.199838+0000","last_active":"2024-09-13T14:59:00.904559+0000","last_peered":"2024-09-13T14:59:00.904559+0000","last_clean":"2024-09-13T14:59:00.904559+0000","last_became_active":"2024-09-13T14:58:06.199417+0000","last_became_peered":"2024-09-13T14:58:06.199417+0000","last_unstale":"2024-09-13T14:59:00.904559+0000","last_undegraded":"2024-09-13T14:59:00.904559+0000","last_fullsized":"2024-09-13T14:59:00.904559+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":4895,"log_dups_size":0,"ondisk_log_size":4895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:14:49.667560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320798720,"num_objects":4895,"num_object_clones":0,"num_object_copies":14685,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4895,"num_write_kb":313280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":66,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.857116+0000","last_change":"2024-09-13T14:58:05.197375+0000","last_active":"2024-09-13T14:58:34.857116+0000","last_peered":"2024-09-13T14:58:34.857116+0000","last_clean":"2024-09-13T14:58:34.857116+0000","last_became_active":"2024-09-13T14:58:05.197085+0000","last_became_peered":"2024-09-13T14:58:05.197085+0000","last_unstale":"2024-09-13T14:58:34.857116+0000","last_undegraded":"2024-09-13T14:58:34.857116+0000","last_fullsized":"2024-09-13T14:58:34.857116+0000","mapping_epoch":1267,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:09:33.979543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":1329,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.851009+0000","last_change":"2024-09-13T14:54:58.841801+0000","last_active":"2024-09-13T14:58:34.851009+0000","last_peered":"2024-09-13T14:58:34.851009+0000","last_clean":"2024-09-13T14:58:34.851009+0000","last_became_active":"2024-09-13T14:54:58.820336+0000","last_became_peered":"2024-09-13T14:54:58.820336+0000","last_unstale":"2024-09-13T14:58:34.851009+0000","last_undegraded":"2024-09-13T14:58:34.851009+0000","last_fullsized":"2024-09-13T14:58:34.851009+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:19:59.656874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"261'100","reported_seq":274,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.908676+0000","last_change":"2024-09-13T14:45:19.821649+0000","last_active":"2024-09-13T14:58:46.908676+0000","last_peered":"2024-09-13T14:58:46.908676+0000","last_clean":"2024-09-13T14:58:46.908676+0000","last_became_active":"2024-09-13T14:45:19.812346+0000","last_became_peered":"2024-09-13T14:45:19.812346+0000","last_unstale":"2024-09-13T14:58:46.908676+0000","last_undegraded":"2024-09-13T14:58:46.908676+0000","last_fullsized":"2024-09-13T14:58:46.908676+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.597568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.597568+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:45.218892+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"1275'5037","reported_seq":5052,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:59:02.303812+0000","last_change":"2024-09-13T14:58:06.203821+0000","last_active":"2024-09-13T14:59:02.303812+0000","last_peered":"2024-09-13T14:59:02.303812+0000","last_clean":"2024-09-13T14:59:02.303812+0000","last_became_active":"2024-09-13T14:58:06.203460+0000","last_became_peered":"2024-09-13T14:58:06.203460+0000","last_unstale":"2024-09-13T14:59:02.303812+0000","last_undegraded":"2024-09-13T14:59:02.303812+0000","last_fullsized":"2024-09-13T14:59:02.303812+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":5037,"log_dups_size":0,"ondisk_log_size":5037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:46:54.957926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330104832,"num_objects":5037,"num_object_clones":0,"num_object_copies":15111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5037,"num_write_kb":322368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":48,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.851829+0000","last_change":"2024-09-13T14:55:41.787477+0000","last_active":"2024-09-13T14:58:34.851829+0000","last_peered":"2024-09-13T14:58:34.851829+0000","last_clean":"2024-09-13T14:58:34.851829+0000","last_became_active":"2024-09-13T14:55:41.786674+0000","last_became_peered":"2024-09-13T14:55:41.786674+0000","last_unstale":"2024-09-13T14:58:34.851829+0000","last_undegraded":"2024-09-13T14:58:34.851829+0000","last_fullsized":"2024-09-13T14:58:34.851829+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:56.263938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":1300,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.850898+0000","last_change":"2024-09-13T14:54:58.833271+0000","last_active":"2024-09-13T14:58:34.850898+0000","last_peered":"2024-09-13T14:58:34.850898+0000","last_clean":"2024-09-13T14:58:34.850898+0000","last_became_active":"2024-09-13T14:54:58.819611+0000","last_became_peered":"2024-09-13T14:54:58.819611+0000","last_unstale":"2024-09-13T14:58:34.850898+0000","last_undegraded":"2024-09-13T14:58:34.850898+0000","last_fullsized":"2024-09-13T14:58:34.850898+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:52:37.530890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"261'100","reported_seq":206,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.850602+0000","last_change":"2024-09-13T14:54:58.826831+0000","last_active":"2024-09-13T14:58:34.850602+0000","last_peered":"2024-09-13T14:58:34.850602+0000","last_clean":"2024-09-13T14:58:34.850602+0000","last_became_active":"2024-09-13T14:54:58.826370+0000","last_became_peered":"2024-09-13T14:54:58.826370+0000","last_unstale":"2024-09-13T14:58:34.850602+0000","last_undegraded":"2024-09-13T14:58:34.850602+0000","last_fullsized":"2024-09-13T14:58:34.850602+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.191326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.191326+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:09:06.771779+0000","scrub_duration":0.0025317360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1300,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.855537+0000","last_change":"2024-09-13T14:55:41.925458+0000","last_active":"2024-09-13T14:58:34.855537+0000","last_peered":"2024-09-13T14:58:34.855537+0000","last_clean":"2024-09-13T14:58:34.855537+0000","last_became_active":"2024-09-13T14:55:41.859331+0000","last_became_peered":"2024-09-13T14:55:41.859331+0000","last_unstale":"2024-09-13T14:58:34.855537+0000","last_undegraded":"2024-09-13T14:58:34.855537+0000","last_fullsized":"2024-09-13T14:58:34.855537+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:19:20.962759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":283,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.912120+0000","last_change":"2024-09-13T14:54:05.604259+0000","last_active":"2024-09-13T14:58:46.912120+0000","last_peered":"2024-09-13T14:58:46.912120+0000","last_clean":"2024-09-13T14:58:46.912120+0000","last_became_active":"2024-09-13T14:54:05.603890+0000","last_became_peered":"2024-09-13T14:54:05.603890+0000","last_unstale":"2024-09-13T14:58:46.912120+0000","last_undegraded":"2024-09-13T14:58:46.912120+0000","last_fullsized":"2024-09-13T14:58:46.912120+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:14.847855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"261'100","reported_seq":48,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.852289+0000","last_change":"2024-09-13T14:55:41.889497+0000","last_active":"2024-09-13T14:58:34.852289+0000","last_peered":"2024-09-13T14:58:34.852289+0000","last_clean":"2024-09-13T14:58:34.852289+0000","last_became_active":"2024-09-13T14:55:41.888766+0000","last_became_peered":"2024-09-13T14:55:41.888766+0000","last_unstale":"2024-09-13T14:58:34.852289+0000","last_undegraded":"2024-09-13T14:58:34.852289+0000","last_fullsized":"2024-09-13T14:58:34.852289+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:55.378404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:55.378404+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:50.289550+0000","scrub_duration":0.0016008439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1232,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.849782+0000","last_change":"2024-09-13T14:54:58.825419+0000","last_active":"2024-09-13T14:58:34.849782+0000","last_peered":"2024-09-13T14:58:34.849782+0000","last_clean":"2024-09-13T14:58:34.849782+0000","last_became_active":"2024-09-13T14:54:58.820338+0000","last_became_peered":"2024-09-13T14:54:58.820338+0000","last_unstale":"2024-09-13T14:58:34.849782+0000","last_undegraded":"2024-09-13T14:58:34.849782+0000","last_fullsized":"2024-09-13T14:58:34.849782+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:09.298279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1937,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.907296+0000","last_change":"2024-09-13T14:45:19.813876+0000","last_active":"2024-09-13T14:58:46.907296+0000","last_peered":"2024-09-13T14:58:46.907296+0000","last_clean":"2024-09-13T14:58:46.907296+0000","last_became_active":"2024-09-13T14:45:19.809136+0000","last_became_peered":"2024-09-13T14:45:19.809136+0000","last_unstale":"2024-09-13T14:58:46.907296+0000","last_undegraded":"2024-09-13T14:58:46.907296+0000","last_fullsized":"2024-09-13T14:58:46.907296+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:50.013528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"261'100","reported_seq":125,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.903476+0000","last_change":"2024-09-13T14:54:58.828578+0000","last_active":"2024-09-13T14:58:46.903476+0000","last_peered":"2024-09-13T14:58:46.903476+0000","last_clean":"2024-09-13T14:58:46.903476+0000","last_became_active":"2024-09-13T14:54:58.828004+0000","last_became_peered":"2024-09-13T14:54:58.828004+0000","last_unstale":"2024-09-13T14:58:46.903476+0000","last_undegraded":"2024-09-13T14:58:46.903476+0000","last_fullsized":"2024-09-13T14:58:46.903476+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:19.978464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:19.978464+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:52:31.236171+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1985,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.907910+0000","last_change":"2024-09-13T14:45:19.809166+0000","last_active":"2024-09-13T14:58:46.907910+0000","last_peered":"2024-09-13T14:58:46.907910+0000","last_clean":"2024-09-13T14:58:46.907910+0000","last_became_active":"2024-09-13T14:45:19.808631+0000","last_became_peered":"2024-09-13T14:45:19.808631+0000","last_unstale":"2024-09-13T14:58:46.907910+0000","last_undegraded":"2024-09-13T14:58:46.907910+0000","last_fullsized":"2024-09-13T14:58:46.907910+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:38.822245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":48,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.852670+0000","last_change":"2024-09-13T14:55:41.835069+0000","last_active":"2024-09-13T14:58:34.852670+0000","last_peered":"2024-09-13T14:58:34.852670+0000","last_clean":"2024-09-13T14:58:34.852670+0000","last_became_active":"2024-09-13T14:55:41.834766+0000","last_became_peered":"2024-09-13T14:55:41.834766+0000","last_unstale":"2024-09-13T14:58:34.852670+0000","last_undegraded":"2024-09-13T14:58:34.852670+0000","last_fullsized":"2024-09-13T14:58:34.852670+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:02:16.216617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"261'100","reported_seq":48,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.851787+0000","last_change":"2024-09-13T14:55:41.787716+0000","last_active":"2024-09-13T14:58:34.851787+0000","last_peered":"2024-09-13T14:58:34.851787+0000","last_clean":"2024-09-13T14:58:34.851787+0000","last_became_active":"2024-09-13T14:55:41.787407+0000","last_became_peered":"2024-09-13T14:55:41.787407+0000","last_unstale":"2024-09-13T14:58:34.851787+0000","last_undegraded":"2024-09-13T14:58:34.851787+0000","last_fullsized":"2024-09-13T14:58:34.851787+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:50.327393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:50.327393+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:43:30.234863+0000","scrub_duration":0.0011508309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":1263,"reported_epoch":1274,"state":"active+clean","last_fresh":"2024-09-13T14:58:34.850040+0000","last_change":"2024-09-13T14:54:58.845372+0000","last_active":"2024-09-13T14:58:34.850040+0000","last_peered":"2024-09-13T14:58:34.850040+0000","last_clean":"2024-09-13T14:58:34.850040+0000","last_became_active":"2024-09-13T14:54:58.822960+0000","last_became_peered":"2024-09-13T14:54:58.822960+0000","last_unstale":"2024-09-13T14:58:34.850040+0000","last_undegraded":"2024-09-13T14:58:34.850040+0000","last_fullsized":"2024-09-13T14:58:34.850040+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:19:30.257102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1890,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.908944+0000","last_change":"2024-09-13T14:54:05.605595+0000","last_active":"2024-09-13T14:58:46.908944+0000","last_peered":"2024-09-13T14:58:46.908944+0000","last_clean":"2024-09-13T14:58:46.908944+0000","last_became_active":"2024-09-13T14:54:05.604916+0000","last_became_peered":"2024-09-13T14:54:05.604916+0000","last_unstale":"2024-09-13T14:58:46.908944+0000","last_undegraded":"2024-09-13T14:58:46.908944+0000","last_fullsized":"2024-09-13T14:58:46.908944+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:06:22.358902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"261'100","reported_seq":680,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.906234+0000","last_change":"2024-09-13T14:55:41.878403+0000","last_active":"2024-09-13T14:58:46.906234+0000","last_peered":"2024-09-13T14:58:46.906234+0000","last_clean":"2024-09-13T14:58:46.906234+0000","last_became_active":"2024-09-13T14:55:41.878071+0000","last_became_peered":"2024-09-13T14:55:41.878071+0000","last_unstale":"2024-09-13T14:58:46.906234+0000","last_undegraded":"2024-09-13T14:58:46.906234+0000","last_fullsized":"2024-09-13T14:58:46.906234+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:14.654603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:14.654603+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:09:56.313293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":5272567808,"num_objects":80453,"num_object_clones":0,"num_object_copies":241359,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80453,"num_write_kb":5148992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15878717440,"data_stored":15878717440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80453,"ondisk_log_size":80453,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":7662,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":853056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430708,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2179256,"kb_used_data":1288496,"kb_used_omap":1,"kb_used_meta":890750,"kb_avail":91565896,"statfs":{"total":95995035648,"available":93763477504,"internally_reserved":0,"allocated":1319419904,"data_stored":1316763607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":912128344},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626937,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3159344,"kb_used_data":2256616,"kb_used_omap":1,"kb_used_meta":902718,"kb_avail":90585808,"statfs":{"total":95995035648,"available":92759867392,"internally_reserved":0,"allocated":2310774784,"data_stored":2308126098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":924383576},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823166,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2506476,"kb_used_data":1606948,"kb_used_omap":1,"kb_used_meta":899518,"kb_avail":91238676,"statfs":{"total":95995035648,"available":93428404224,"internally_reserved":0,"allocated":1645514752,"data_stored":1642866532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":921106776},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019394,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3476148,"kb_used_data":2576300,"kb_used_omap":1,"kb_used_meta":899838,"kb_avail":90269004,"statfs":{"total":95995035648,"available":92435460096,"internally_reserved":0,"allocated":2638131200,"data_stored":2635478475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":921434456},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059593051,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2248416,"kb_used_data":1568600,"kb_used_omap":26,"kb_used_meta":679781,"kb_avail":91496736,"statfs":{"total":95995035648,"available":93692657664,"internally_reserved":0,"allocated":1606246400,"data_stored":1603609367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":696095904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":591,"seq":2538325672290,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2699244,"kb_used_data":1937124,"kb_used_omap":41,"kb_used_meta":762070,"kb_avail":91045908,"statfs":{"total":95995035648,"available":93231009792,"internally_reserved":0,"allocated":1983614976,"data_stored":1980966803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42651,"internal_metadata":780360037},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":557,"seq":2392296784232,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1705312,"kb_used_data":936088,"kb_used_omap":32,"kb_used_meta":769183,"kb_avail":92039840,"statfs":{"total":95995035648,"available":94248796160,"internally_reserved":0,"allocated":958554112,"data_stored":955917008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33752,"internal_metadata":787643432},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":524,"seq":2250562863472,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4010212,"kb_used_data":3261148,"kb_used_omap":41,"kb_used_meta":749014,"kb_avail":89734940,"statfs":{"total":95995035648,"available":91888578560,"internally_reserved":0,"allocated":3339415552,"data_stored":3336774925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42694,"internal_metadata":766990650},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3387162624,"data_stored":3387162624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":954531840,"data_stored":954531840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1998454784,"data_stored":1998454784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1616510976,"data_stored":1616510976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2648571904,"data_stored":2648571904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1643446272,"data_stored":1643446272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2311061504,"data_stored":2311061504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1318977536,"data_stored":1318977536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:59:07.431 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T14:59:07.431 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:59:07.431 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-13T14:59:07.432 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:59:07.521 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:07.547 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:07.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:07.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:07.649 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:07.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:07.690 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:07.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:07.752 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:07.792 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:07.816 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:07.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:07.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:07.918 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:07.921 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T14:59:07.963 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:07.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:07.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:08.065 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:08.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:08.113 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:08.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:08.196 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 51 16 86559 86543 106.037 63.5 0.0217244 0.00942603 2024-09-13T14:59:08.215 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:08.251 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:08.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:08.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:08.312 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:59:08.353 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:08.386 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:08.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:08.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:08.488 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:08.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:08.526 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:08.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:08.627 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:08.679 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:08.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:08.686 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:08.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:08.787 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:08.820 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:08.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:08.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:08.834 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-13T14:59:08.923 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:08.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:08.957 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:08.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:09.058 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:09.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:09.100 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:09.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:09.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:08 smithi184 ceph-mon[92527]: pgmap v2072: 119 pgs: 119 active+clean; 5.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-13T14:59:09.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:08 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/26390219' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:09.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:08 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:09.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:08 smithi151 ceph-mon[104450]: pgmap v2072: 119 pgs: 119 active+clean; 5.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-13T14:59:09.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:08 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/26390219' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:09.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:08 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:09.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:08 smithi151 ceph-mon[108966]: pgmap v2072: 119 pgs: 119 active+clean; 5.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-13T14:59:09.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:08 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/26390219' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:09.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:08 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:09.196 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 52 16 87722 87706 105.396 72.6875 0.0120872 0.00948312 2024-09-13T14:59:09.200 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:09.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:09.238 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:09.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:09.339 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:09.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:09.383 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:09.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:09.484 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:09.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:09.524 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:09.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:09.625 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:09.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:09.667 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:09.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:09.769 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:09.807 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:09.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:09.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:09.888 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:09 smithi151 ceph-mon[104450]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:59:09.888 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:09 smithi151 ceph-mon[104450]: Cluster is now healthy 2024-09-13T14:59:09.888 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:09 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:59:09.888 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:09 smithi151 ceph-mon[104450]: osdmap e1276: 8 total, 8 up, 8 in 2024-09-13T14:59:09.888 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:09 smithi151 ceph-mon[108966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:59:09.889 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:09 smithi151 ceph-mon[108966]: Cluster is now healthy 2024-09-13T14:59:09.889 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:09 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:59:09.889 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:09 smithi151 ceph-mon[108966]: osdmap e1276: 8 total, 8 up, 8 in 2024-09-13T14:59:09.910 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:09.946 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:09.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:09.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:10.048 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:10.087 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:10.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:10.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:10.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:09 smithi184 ceph-mon[92527]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:59:10.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:09 smithi184 ceph-mon[92527]: Cluster is now healthy 2024-09-13T14:59:10.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:09 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:59:10.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:09 smithi184 ceph-mon[92527]: osdmap e1276: 8 total, 8 up, 8 in 2024-09-13T14:59:10.189 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:10.196 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 53 16 88885 88869 104.778 72.6875 0.00928912 0.00953785 2024-09-13T14:59:10.213 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:59:10.213 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:59:10.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:10.235 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:10.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:10.249 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:59:10.249 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:59:10.281 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:59:10.282 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:59:10.312 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:59:10.313 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:59:10.336 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:10.343 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:59:10.343 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:59:10.369 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:59:10.369 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:59:10.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:10.377 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:10.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:10.405 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:59:10.405 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:59:10.409 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:59:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:59:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:59:10.433 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:59:10.433 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:59:10.478 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:10.517 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:10.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:10.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:10.556 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:59:10.556 INFO:teuthology.orchestra.run.smithi184.stdout: "size": 20, 2024-09-13T14:59:10.556 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 600, 2024-09-13T14:59:10.556 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [ 2024-09-13T14:59:10.556 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.557 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:42827 8.2 8:4433e755:::benchmark_data_smithi151_229097_object42826:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1172)", 2024-09-13T14:59:10.557 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:49:57.713598+0000", 2024-09-13T14:59:10.557 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 552.83335736200002, 2024-09-13T14:59:10.557 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.13415065700000001, 2024-09-13T14:59:10.557 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:10.557 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:10.557 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:10.557 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:59:10.557 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:59:10.557 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 42827 2024-09-13T14:59:10.557 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.557 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:10.557 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.558 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:10.558 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.713598+0000", 2024-09-13T14:59:10.558 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.558 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.558 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.558 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:10.558 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.713598+0000", 2024-09-13T14:59:10.558 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.558 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.558 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.558 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:10.558 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.713596+0000", 2024-09-13T14:59:10.559 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:10.559 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.559 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.559 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:10.559 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.713638+0000", 2024-09-13T14:59:10.559 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1916000000000003e-05 2024-09-13T14:59:10.559 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.559 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.559 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:10.559 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.713649+0000", 2024-09-13T14:59:10.559 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1114000000000001e-05 2024-09-13T14:59:10.559 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.559 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.559 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:10.560 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.713654+0000", 2024-09-13T14:59:10.560 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0879999999999997e-06 2024-09-13T14:59:10.560 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.560 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.560 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:10.560 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.713687+0000", 2024-09-13T14:59:10.560 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3478e-05 2024-09-13T14:59:10.560 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.560 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.560 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:10.560 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.713779+0000", 2024-09-13T14:59:10.560 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.2367999999999994e-05 2024-09-13T14:59:10.560 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.561 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.561 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:59:10.561 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.713880+0000", 2024-09-13T14:59:10.561 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000101043 2024-09-13T14:59:10.561 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.561 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.561 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:10.561 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.715459+0000", 2024-09-13T14:59:10.561 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001578238 2024-09-13T14:59:10.561 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.561 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.561 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:10.561 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.715892+0000", 2024-09-13T14:59:10.562 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000433485 2024-09-13T14:59:10.562 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.562 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.562 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:10.562 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.847624+0000", 2024-09-13T14:59:10.562 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.13173138700000001 2024-09-13T14:59:10.562 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.562 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.562 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:10.562 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.847712+0000", 2024-09-13T14:59:10.562 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.8432999999999998e-05 2024-09-13T14:59:10.562 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.562 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.563 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:10.563 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:49:57.847748+0000", 2024-09-13T14:59:10.563 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.6426e-05 2024-09-13T14:59:10.563 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.563 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:10.563 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.563 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.563 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.563 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "MOSDPGPush(9.c 1230/1229 [PushOp(9:3fe1da54:::benchmark_data_smithi151_261954_object34:head, version: 1225'3, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:3fe1da54:::benchmark_data_smithi151_261954_object34:head@1225'3, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:59:10.563 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:55:00.127225+0000", 2024-09-13T14:59:10.563 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 250.419730077, 2024-09-13T14:59:10.563 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.117964821, 2024-09-13T14:59:10.563 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:10.564 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:59:10.564 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:10.564 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.564 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:10.564 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.127225+0000", 2024-09-13T14:59:10.564 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.564 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.564 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.564 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:10.564 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.127225+0000", 2024-09-13T14:59:10.564 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.564 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.564 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.565 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:10.565 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.127225+0000", 2024-09-13T14:59:10.565 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:59:10.565 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.565 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.565 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:10.565 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.127290+0000", 2024-09-13T14:59:10.565 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.5216999999999996e-05 2024-09-13T14:59:10.565 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.565 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.565 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:10.565 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.127306+0000", 2024-09-13T14:59:10.565 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6280999999999999e-05 2024-09-13T14:59:10.565 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.566 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.566 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:10.566 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.127314+0000", 2024-09-13T14:59:10.566 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.1210000000000007e-06 2024-09-13T14:59:10.566 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.566 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.566 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:10.566 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.233431+0000", 2024-09-13T14:59:10.566 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.106116483 2024-09-13T14:59:10.566 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.566 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.566 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:10.566 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.233449+0000", 2024-09-13T14:59:10.566 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.8281999999999999e-05 2024-09-13T14:59:10.567 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.567 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.567 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:10.567 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.245190+0000", 2024-09-13T14:59:10.567 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.011740980999999999 2024-09-13T14:59:10.567 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.567 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:10.567 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.567 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.567 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.567 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "MOSDPGPush(9.c 1230/1229 [PushOp(9:398315b8:::benchmark_data_smithi151_261954_object276:head, version: 1225'16, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:398315b8:::benchmark_data_smithi151_261954_object276:head@1225'16, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:59:10.567 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:55:00.827145+0000", 2024-09-13T14:59:10.568 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 249.719809814, 2024-09-13T14:59:10.568 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.10624096299999999, 2024-09-13T14:59:10.568 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:10.568 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:59:10.568 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:10.568 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.568 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:10.568 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.827145+0000", 2024-09-13T14:59:10.568 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.568 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.568 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.568 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:10.568 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.827145+0000", 2024-09-13T14:59:10.568 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.569 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.569 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.569 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:10.569 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.827144+0000", 2024-09-13T14:59:10.569 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:10.569 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.569 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.569 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:10.569 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.827203+0000", 2024-09-13T14:59:10.569 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8881000000000001e-05 2024-09-13T14:59:10.569 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.569 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.569 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:10.570 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.827230+0000", 2024-09-13T14:59:10.570 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.7475e-05 2024-09-13T14:59:10.570 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.570 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.570 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:10.570 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.827241+0000", 2024-09-13T14:59:10.570 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0689999999999999e-05 2024-09-13T14:59:10.570 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.570 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.570 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:10.570 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.916971+0000", 2024-09-13T14:59:10.570 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.089729472000000005 2024-09-13T14:59:10.570 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.571 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.571 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:10.571 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.916977+0000", 2024-09-13T14:59:10.571 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1149999999999999e-06 2024-09-13T14:59:10.571 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.571 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.571 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:10.571 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.933386+0000", 2024-09-13T14:59:10.571 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.016409639 2024-09-13T14:59:10.571 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.571 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:10.571 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.572 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.572 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.572 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "MOSDPGPush(9.c 1234/1229 [PushOp(9:383b629e:::benchmark_data_smithi151_261954_object8007:head, version: 1225'537, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:383b629e:::benchmark_data_smithi151_261954_object8007:head@1225'537, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:59:10.572 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:55:17.970502+0000", 2024-09-13T14:59:10.572 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 232.57645311600001, 2024-09-13T14:59:10.572 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.21509233799999999, 2024-09-13T14:59:10.573 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:10.573 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:59:10.573 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:10.573 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.573 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:10.573 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:17.970502+0000", 2024-09-13T14:59:10.573 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.573 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.573 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.573 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:10.573 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:17.970502+0000", 2024-09-13T14:59:10.573 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.574 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.574 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.574 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:10.574 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:17.970501+0000", 2024-09-13T14:59:10.574 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:10.574 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.574 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.574 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:10.574 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:17.970545+0000", 2024-09-13T14:59:10.574 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4351000000000003e-05 2024-09-13T14:59:10.574 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.574 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.574 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:10.574 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:18.183178+0000", 2024-09-13T14:59:10.575 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.21263328000000001 2024-09-13T14:59:10.575 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.575 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.575 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:10.575 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:18.183203+0000", 2024-09-13T14:59:10.575 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.5021e-05 2024-09-13T14:59:10.575 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.575 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.575 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:10.575 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:18.183276+0000", 2024-09-13T14:59:10.575 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.3051000000000002e-05 2024-09-13T14:59:10.575 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.575 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.575 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:10.576 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:18.183285+0000", 2024-09-13T14:59:10.576 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.1789999999999997e-06 2024-09-13T14:59:10.576 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.576 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.576 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:10.576 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:18.185594+0000", 2024-09-13T14:59:10.576 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0023089360000000001 2024-09-13T14:59:10.576 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.576 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:10.576 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.576 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.576 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.576 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:18468 10.e 10:74ea4b05:::benchmark_data_smithi151_288181_object18467:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1272)", 2024-09-13T14:59:10.577 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:58:22.987531+0000", 2024-09-13T14:59:10.577 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 47.559424186000001, 2024-09-13T14:59:10.577 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.106306742, 2024-09-13T14:59:10.577 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:10.577 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:10.577 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:10.577 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T14:59:10.577 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T14:59:10.578 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 18468 2024-09-13T14:59:10.578 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.578 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:10.578 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.578 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:10.578 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.987531+0000", 2024-09-13T14:59:10.578 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.578 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.578 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.578 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:10.578 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.987531+0000", 2024-09-13T14:59:10.578 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.578 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.579 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.579 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:10.579 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.987530+0000", 2024-09-13T14:59:10.579 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:59:10.579 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.579 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.579 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:10.579 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.987587+0000", 2024-09-13T14:59:10.579 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6110000000000003e-05 2024-09-13T14:59:10.579 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.579 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.579 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:10.579 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.987594+0000", 2024-09-13T14:59:10.579 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.0310000000000001e-06 2024-09-13T14:59:10.580 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.580 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.580 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:10.580 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.987597+0000", 2024-09-13T14:59:10.580 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8589999999999998e-06 2024-09-13T14:59:10.580 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.580 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.580 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:10.580 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.988621+0000", 2024-09-13T14:59:10.580 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001023298 2024-09-13T14:59:10.580 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.580 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.580 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:10.581 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.988722+0000", 2024-09-13T14:59:10.581 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000100781 2024-09-13T14:59:10.581 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.581 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.581 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:59:10.581 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.988833+0000", 2024-09-13T14:59:10.581 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00011194800000000001 2024-09-13T14:59:10.581 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.581 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.581 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:10.581 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.989684+0000", 2024-09-13T14:59:10.581 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00085093199999999999 2024-09-13T14:59:10.581 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.581 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.582 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:10.582 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.990048+0000", 2024-09-13T14:59:10.582 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00036384699999999999 2024-09-13T14:59:10.582 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.582 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.582 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:10.582 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.993255+0000", 2024-09-13T14:59:10.582 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0032070559999999998 2024-09-13T14:59:10.582 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.582 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.582 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:10.582 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.093779+0000", 2024-09-13T14:59:10.582 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.100523494 2024-09-13T14:59:10.582 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.583 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.583 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:10.583 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.093838+0000", 2024-09-13T14:59:10.583 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8907999999999999e-05 2024-09-13T14:59:10.583 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.583 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:10.583 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.583 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.583 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.583 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:18472 10.d 10:b1ed443b:::benchmark_data_smithi151_288181_object18471:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1272)", 2024-09-13T14:59:10.583 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:58:22.989741+0000", 2024-09-13T14:59:10.583 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 47.557213816000001, 2024-09-13T14:59:10.583 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.10466682400000001, 2024-09-13T14:59:10.584 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:10.584 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:10.584 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:10.584 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T14:59:10.584 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T14:59:10.584 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 18472 2024-09-13T14:59:10.584 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.584 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:10.584 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.584 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:10.584 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.989741+0000", 2024-09-13T14:59:10.584 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.584 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.584 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.585 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:10.585 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.989741+0000", 2024-09-13T14:59:10.585 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.585 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.585 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.585 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:10.585 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.989740+0000", 2024-09-13T14:59:10.585 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:10.585 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.585 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.585 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:10.585 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.989779+0000", 2024-09-13T14:59:10.585 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9044000000000001e-05 2024-09-13T14:59:10.585 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.586 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.586 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:10.586 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.989793+0000", 2024-09-13T14:59:10.586 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.367e-05 2024-09-13T14:59:10.586 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.586 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.586 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:10.586 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.989799+0000", 2024-09-13T14:59:10.586 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.9660000000000001e-06 2024-09-13T14:59:10.586 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.586 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.586 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:10.586 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.989837+0000", 2024-09-13T14:59:10.587 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8331999999999998e-05 2024-09-13T14:59:10.587 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.587 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.587 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:10.587 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.989952+0000", 2024-09-13T14:59:10.587 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000114406 2024-09-13T14:59:10.587 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.587 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.587 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:59:10.587 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.990078+0000", 2024-09-13T14:59:10.587 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000126258 2024-09-13T14:59:10.587 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.587 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.588 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:10.588 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.990935+0000", 2024-09-13T14:59:10.588 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00085728200000000001 2024-09-13T14:59:10.588 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.588 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.588 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:10.588 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.990993+0000", 2024-09-13T14:59:10.588 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7491999999999998e-05 2024-09-13T14:59:10.588 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.588 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.588 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:10.588 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.094319+0000", 2024-09-13T14:59:10.588 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.10332611899999999 2024-09-13T14:59:10.589 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.589 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.589 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:10.589 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.094372+0000", 2024-09-13T14:59:10.589 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3026999999999999e-05 2024-09-13T14:59:10.589 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.589 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.589 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:10.589 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.094408+0000", 2024-09-13T14:59:10.589 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.6423e-05 2024-09-13T14:59:10.589 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.589 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:10.589 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.590 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.590 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.590 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:18480 10.d 10:b337b199:::benchmark_data_smithi151_288181_object18479:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1272)", 2024-09-13T14:59:10.590 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:58:22.991015+0000", 2024-09-13T14:59:10.590 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 47.555940591999999, 2024-09-13T14:59:10.590 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.103508788, 2024-09-13T14:59:10.590 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:10.590 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:10.590 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:10.590 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T14:59:10.590 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T14:59:10.590 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 18480 2024-09-13T14:59:10.590 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.591 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:10.591 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.591 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:10.591 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.991015+0000", 2024-09-13T14:59:10.591 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.591 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.591 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.591 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:10.591 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.991015+0000", 2024-09-13T14:59:10.591 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.591 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.592 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.592 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:10.592 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.991014+0000", 2024-09-13T14:59:10.592 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:59:10.592 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.592 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.592 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:10.592 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.991084+0000", 2024-09-13T14:59:10.592 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.0259000000000006e-05 2024-09-13T14:59:10.592 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.592 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.592 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:10.592 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.991093+0000", 2024-09-13T14:59:10.592 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.5320000000000006e-06 2024-09-13T14:59:10.593 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.593 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.593 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:10.593 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.991098+0000", 2024-09-13T14:59:10.593 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9840000000000001e-06 2024-09-13T14:59:10.593 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.593 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.593 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:10.593 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.991135+0000", 2024-09-13T14:59:10.593 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.7358999999999999e-05 2024-09-13T14:59:10.593 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.593 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.593 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:10.593 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.991232+0000", 2024-09-13T14:59:10.594 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.6510999999999998e-05 2024-09-13T14:59:10.594 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.594 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.594 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:59:10.594 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.991664+0000", 2024-09-13T14:59:10.594 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00043206000000000001 2024-09-13T14:59:10.594 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.594 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.594 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:10.594 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.992730+0000", 2024-09-13T14:59:10.594 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0010663210000000001 2024-09-13T14:59:10.594 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.594 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.595 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:10.595 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.992799+0000", 2024-09-13T14:59:10.595 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.8973000000000005e-05 2024-09-13T14:59:10.595 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.595 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.595 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:10.595 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.094452+0000", 2024-09-13T14:59:10.595 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.101653483 2024-09-13T14:59:10.595 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.595 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.595 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:10.595 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.094492+0000", 2024-09-13T14:59:10.595 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9888999999999997e-05 2024-09-13T14:59:10.596 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.596 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.596 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:10.596 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.094523+0000", 2024-09-13T14:59:10.596 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.1106999999999999e-05 2024-09-13T14:59:10.596 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.596 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:10.596 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.596 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.596 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.596 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.36396.0:18477 10.4 e1272/1268 10:2a34aac8:::benchmark_data_smithi151_288181_object18476:head v 1272'1145, mlcod=1272'1141)", 2024-09-13T14:59:10.596 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:58:22.992454+0000", 2024-09-13T14:59:10.596 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 47.554500711999999, 2024-09-13T14:59:10.597 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.102146037, 2024-09-13T14:59:10.597 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:10.597 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:10.597 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:10.597 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.597 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:10.597 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.992454+0000", 2024-09-13T14:59:10.597 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.597 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.597 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.597 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:10.598 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.992454+0000", 2024-09-13T14:59:10.598 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.598 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.598 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.598 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:10.599 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.992454+0000", 2024-09-13T14:59:10.599 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:59:10.599 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.599 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.599 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:10.599 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.992528+0000", 2024-09-13T14:59:10.599 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.4195000000000004e-05 2024-09-13T14:59:10.599 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.599 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.599 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:10.599 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.992538+0000", 2024-09-13T14:59:10.599 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0573e-05 2024-09-13T14:59:10.599 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.600 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.600 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:10.600 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.992545+0000", 2024-09-13T14:59:10.600 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3080000000000004e-06 2024-09-13T14:59:10.600 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.600 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.600 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:10.600 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.992589+0000", 2024-09-13T14:59:10.600 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3890999999999997e-05 2024-09-13T14:59:10.600 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.600 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.600 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:10.600 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.992627+0000", 2024-09-13T14:59:10.601 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8303000000000002e-05 2024-09-13T14:59:10.601 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.601 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.601 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:10.601 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.993338+0000", 2024-09-13T14:59:10.601 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00071061300000000002 2024-09-13T14:59:10.601 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.601 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.601 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:10.601 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.094600+0000", 2024-09-13T14:59:10.601 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.101262931 2024-09-13T14:59:10.601 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.601 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:10.602 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.602 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.602 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.602 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:18482 10.d 10:b996de7a:::benchmark_data_smithi151_288181_object18481:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1272)", 2024-09-13T14:59:10.602 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:58:22.993384+0000", 2024-09-13T14:59:10.602 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 47.553571417000001, 2024-09-13T14:59:10.602 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.108046376, 2024-09-13T14:59:10.602 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:10.602 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:10.602 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:10.602 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T14:59:10.602 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T14:59:10.602 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 18482 2024-09-13T14:59:10.603 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.603 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:10.603 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.603 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:10.603 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.993384+0000", 2024-09-13T14:59:10.603 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.603 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.603 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.603 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:10.603 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.993384+0000", 2024-09-13T14:59:10.604 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.604 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.604 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.604 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:10.604 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.993383+0000", 2024-09-13T14:59:10.604 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:59:10.604 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.604 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.604 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:10.604 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.993441+0000", 2024-09-13T14:59:10.604 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7629e-05 2024-09-13T14:59:10.604 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.604 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.605 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:10.605 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.093761+0000", 2024-09-13T14:59:10.605 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.10031998 2024-09-13T14:59:10.605 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.605 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.605 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:10.605 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.093769+0000", 2024-09-13T14:59:10.605 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.3529999999999995e-06 2024-09-13T14:59:10.605 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.605 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.605 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:10.605 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.094544+0000", 2024-09-13T14:59:10.605 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00077534800000000003 2024-09-13T14:59:10.605 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.606 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.606 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:10.606 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.094642+0000", 2024-09-13T14:59:10.606 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.7359000000000001e-05 2024-09-13T14:59:10.606 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.606 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.606 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:59:10.606 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.094765+0000", 2024-09-13T14:59:10.606 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000123449 2024-09-13T14:59:10.606 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.606 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.606 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:10.606 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.096098+0000", 2024-09-13T14:59:10.606 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001332684 2024-09-13T14:59:10.607 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.607 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.607 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:10.607 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.096533+0000", 2024-09-13T14:59:10.607 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00043552499999999997 2024-09-13T14:59:10.607 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.607 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.607 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:10.607 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.101359+0000", 2024-09-13T14:59:10.607 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0048257129999999997 2024-09-13T14:59:10.607 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.607 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.607 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:10.607 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.101401+0000", 2024-09-13T14:59:10.607 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.2052000000000003e-05 2024-09-13T14:59:10.608 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.608 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.608 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:10.608 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.101430+0000", 2024-09-13T14:59:10.608 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.9053000000000001e-05 2024-09-13T14:59:10.608 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.608 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:10.608 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.608 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.608 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.608 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.36396.0:18481 10.2 e1272/1268 10:4f41fee5:::benchmark_data_smithi151_288181_object18480:head v 1272'1179, mlcod=1272'1177)", 2024-09-13T14:59:10.608 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:58:22.993684+0000", 2024-09-13T14:59:10.608 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 47.553271015, 2024-09-13T14:59:10.608 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.10432588600000001, 2024-09-13T14:59:10.609 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:10.609 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:10.609 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:10.609 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.609 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:10.609 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.993684+0000", 2024-09-13T14:59:10.609 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.609 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.609 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.609 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:10.609 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.993684+0000", 2024-09-13T14:59:10.609 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.609 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.609 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.610 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:10.610 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.993684+0000", 2024-09-13T14:59:10.610 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:59:10.610 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.610 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.610 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:10.610 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:22.993747+0000", 2024-09-13T14:59:10.610 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.2983000000000003e-05 2024-09-13T14:59:10.610 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.610 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.610 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:10.610 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.093755+0000", 2024-09-13T14:59:10.610 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.100008872 2024-09-13T14:59:10.610 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.611 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.611 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:10.611 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.093767+0000", 2024-09-13T14:59:10.611 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2058999999999999e-05 2024-09-13T14:59:10.611 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.611 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.611 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:10.611 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.093835+0000", 2024-09-13T14:59:10.611 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7063999999999994e-05 2024-09-13T14:59:10.611 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.611 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.611 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:10.611 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.094270+0000", 2024-09-13T14:59:10.612 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00043549300000000001 2024-09-13T14:59:10.612 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.612 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.612 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:10.612 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.097914+0000", 2024-09-13T14:59:10.612 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.003644006 2024-09-13T14:59:10.612 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.612 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.612 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:10.612 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.098010+0000", 2024-09-13T14:59:10.612 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.5946999999999995e-05 2024-09-13T14:59:10.612 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.613 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:10.613 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.613 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.613 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.613 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:19312 10.d 10:b92db777:::benchmark_data_smithi151_288181_object19311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1272)", 2024-09-13T14:59:10.613 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:58:23.380454+0000", 2024-09-13T14:59:10.613 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 47.166500649, 2024-09-13T14:59:10.613 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.12024602400000001, 2024-09-13T14:59:10.613 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:10.613 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:10.613 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:10.613 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T14:59:10.613 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T14:59:10.614 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 19312 2024-09-13T14:59:10.614 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.614 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:10.614 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.614 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:10.614 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.380454+0000", 2024-09-13T14:59:10.614 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.614 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.614 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.614 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:10.614 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.380454+0000", 2024-09-13T14:59:10.614 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.615 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.615 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.615 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:10.615 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.380454+0000", 2024-09-13T14:59:10.615 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:59:10.615 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.615 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.615 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:10.615 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.380482+0000", 2024-09-13T14:59:10.615 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.8411999999999998e-05 2024-09-13T14:59:10.615 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.616 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.616 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:10.616 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.380496+0000", 2024-09-13T14:59:10.616 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3562e-05 2024-09-13T14:59:10.616 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.616 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.616 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:10.616 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.380501+0000", 2024-09-13T14:59:10.616 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6640000000000003e-06 2024-09-13T14:59:10.616 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.616 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.616 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:10.617 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.381376+0000", 2024-09-13T14:59:10.617 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00087495300000000004 2024-09-13T14:59:10.617 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.617 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.617 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:10.617 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.382142+0000", 2024-09-13T14:59:10.617 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00076627900000000003 2024-09-13T14:59:10.617 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.617 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.617 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:59:10.617 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.382307+0000", 2024-09-13T14:59:10.617 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00016443 2024-09-13T14:59:10.617 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.618 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.618 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:10.618 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.384685+0000", 2024-09-13T14:59:10.618 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0023778010000000001 2024-09-13T14:59:10.618 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.618 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.618 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:10.618 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.385449+0000", 2024-09-13T14:59:10.618 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00076400899999999996 2024-09-13T14:59:10.618 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.618 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.618 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:10.619 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.500632+0000", 2024-09-13T14:59:10.619 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.115183041 2024-09-13T14:59:10.619 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.619 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.619 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:10.619 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.500673+0000", 2024-09-13T14:59:10.619 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1507000000000001e-05 2024-09-13T14:59:10.619 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.619 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.620 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:10.620 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:23.500701+0000", 2024-09-13T14:59:10.620 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.73e-05 2024-09-13T14:59:10.620 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.620 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:10.620 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.620 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.620 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.620 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:25645 10.1 10:87b20178:::benchmark_data_smithi151_288181_object25644:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1273)", 2024-09-13T14:59:10.620 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:58:26.582210+0000", 2024-09-13T14:59:10.620 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 43.964744848999999, 2024-09-13T14:59:10.620 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.11706662299999999, 2024-09-13T14:59:10.620 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:10.621 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:10.621 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:10.621 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T14:59:10.621 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T14:59:10.621 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 25645 2024-09-13T14:59:10.621 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.621 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:10.621 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.621 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:10.621 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:26.582210+0000", 2024-09-13T14:59:10.621 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.621 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.621 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.622 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:10.622 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:26.582210+0000", 2024-09-13T14:59:10.622 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.622 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.622 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.622 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:10.622 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:26.582209+0000", 2024-09-13T14:59:10.622 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:59:10.622 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.622 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.622 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:10.622 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:26.582239+0000", 2024-09-13T14:59:10.622 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.9260000000000001e-05 2024-09-13T14:59:10.622 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.623 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.623 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:10.623 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:26.582249+0000", 2024-09-13T14:59:10.623 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0883999999999999e-05 2024-09-13T14:59:10.623 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.623 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.623 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:10.623 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:26.582255+0000", 2024-09-13T14:59:10.623 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7769999999999999e-06 2024-09-13T14:59:10.623 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.623 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.623 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:10.623 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:26.582296+0000", 2024-09-13T14:59:10.623 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0788000000000003e-05 2024-09-13T14:59:10.624 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.624 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.624 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:10.624 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:26.582380+0000", 2024-09-13T14:59:10.624 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.4284999999999997e-05 2024-09-13T14:59:10.624 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.624 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.624 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:59:10.624 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:26.582473+0000", 2024-09-13T14:59:10.624 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.3121999999999994e-05 2024-09-13T14:59:10.624 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.624 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.624 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:10.624 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:26.583398+0000", 2024-09-13T14:59:10.625 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00092416600000000003 2024-09-13T14:59:10.625 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.625 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.625 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:10.625 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:26.584026+0000", 2024-09-13T14:59:10.625 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00062881899999999997 2024-09-13T14:59:10.625 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.625 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.625 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:10.625 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:26.699158+0000", 2024-09-13T14:59:10.625 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.115131812 2024-09-13T14:59:10.625 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.626 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.626 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:10.626 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:26.699243+0000", 2024-09-13T14:59:10.626 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.5053e-05 2024-09-13T14:59:10.626 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.626 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.626 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:10.626 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:26.699277+0000", 2024-09-13T14:59:10.626 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3654e-05 2024-09-13T14:59:10.626 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.626 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:10.626 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.626 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.626 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.626 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:25651 10.1 10:857a014d:::benchmark_data_smithi151_288181_object25650:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1273)", 2024-09-13T14:59:10.627 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:58:26.589980+0000", 2024-09-13T14:59:10.627 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 43.956975409000002, 2024-09-13T14:59:10.627 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.110273857, 2024-09-13T14:59:10.627 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:10.627 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:10.627 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:10.627 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T14:59:10.627 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T14:59:10.627 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 25651 2024-09-13T14:59:10.627 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.627 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:10.627 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.627 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:10.627 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:26.589980+0000", 2024-09-13T14:59:10.628 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.628 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.628 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.628 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:10.628 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:26.589980+0000", 2024-09-13T14:59:10.628 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.628 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.628 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.628 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:10.628 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:26.589979+0000", 2024-09-13T14:59:10.628 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:59:10.628 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.628 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.628 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:10.629 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:26.590065+0000", 2024-09-13T14:59:10.629 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.6211000000000005e-05 2024-09-13T14:59:10.629 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.629 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.629 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:10.629 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:26.590081+0000", 2024-09-13T14:59:10.629 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.5807999999999998e-05 2024-09-13T14:59:10.629 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.629 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.629 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:10.629 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:26.590087+0000", 2024-09-13T14:59:10.630 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.5239999999999997e-06 2024-09-13T14:59:10.630 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.630 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.630 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:10.630 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:26.590135+0000", 2024-09-13T14:59:10.630 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7973000000000003e-05 2024-09-13T14:59:10.630 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.630 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.630 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:10.630 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:26.590262+0000", 2024-09-13T14:59:10.630 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012679799999999999 2024-09-13T14:59:10.630 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.630 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.630 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:59:10.631 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:26.590357+0000", 2024-09-13T14:59:10.631 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.4718999999999999e-05 2024-09-13T14:59:10.631 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.631 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.631 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:10.631 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:26.591379+0000", 2024-09-13T14:59:10.631 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0010219820000000001 2024-09-13T14:59:10.631 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.631 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.631 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:10.631 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:26.591578+0000", 2024-09-13T14:59:10.631 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000199166 2024-09-13T14:59:10.631 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.632 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.632 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:10.632 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:26.699308+0000", 2024-09-13T14:59:10.632 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.10773037100000001 2024-09-13T14:59:10.632 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.632 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.632 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:10.632 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:26.700149+0000", 2024-09-13T14:59:10.632 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00084032599999999996 2024-09-13T14:59:10.632 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.632 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.632 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:10.632 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:26.700254+0000", 2024-09-13T14:59:10.633 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000104999 2024-09-13T14:59:10.633 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.633 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:10.633 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.633 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.633 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.633 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:26470 10.d 10:b6276f15:::benchmark_data_smithi151_288181_object26469:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1273)", 2024-09-13T14:59:10.633 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:58:27.057392+0000", 2024-09-13T14:59:10.633 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 43.489563052000001, 2024-09-13T14:59:10.633 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.103885633, 2024-09-13T14:59:10.633 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:10.633 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:10.633 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:10.633 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T14:59:10.634 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T14:59:10.634 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 26470 2024-09-13T14:59:10.634 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.634 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:10.634 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.634 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:10.634 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:27.057392+0000", 2024-09-13T14:59:10.634 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.634 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.634 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.634 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:10.634 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:27.057392+0000", 2024-09-13T14:59:10.634 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.634 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.635 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.635 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:10.635 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:27.057390+0000", 2024-09-13T14:59:10.635 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:10.635 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.635 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.635 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:10.635 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:27.057442+0000", 2024-09-13T14:59:10.635 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1885000000000002e-05 2024-09-13T14:59:10.635 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.635 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.635 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:10.635 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:27.057457+0000", 2024-09-13T14:59:10.635 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4481000000000001e-05 2024-09-13T14:59:10.636 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.636 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.636 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:10.636 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:27.057463+0000", 2024-09-13T14:59:10.636 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.6109999999999999e-06 2024-09-13T14:59:10.636 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.636 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.636 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:10.636 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:27.057495+0000", 2024-09-13T14:59:10.636 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.1276999999999999e-05 2024-09-13T14:59:10.636 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.636 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.636 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:10.637 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:27.057584+0000", 2024-09-13T14:59:10.637 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.9442000000000002e-05 2024-09-13T14:59:10.637 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.637 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.637 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:59:10.637 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:27.057690+0000", 2024-09-13T14:59:10.637 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00010539699999999999 2024-09-13T14:59:10.637 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.637 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.637 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:10.637 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:27.059976+0000", 2024-09-13T14:59:10.637 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002286817 2024-09-13T14:59:10.637 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.637 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.638 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:10.638 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:27.060070+0000", 2024-09-13T14:59:10.638 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.3919000000000007e-05 2024-09-13T14:59:10.638 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.638 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.638 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:10.638 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:27.161168+0000", 2024-09-13T14:59:10.638 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.101097442 2024-09-13T14:59:10.638 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.638 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.638 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:10.638 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:27.161244+0000", 2024-09-13T14:59:10.638 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.6144999999999997e-05 2024-09-13T14:59:10.639 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.639 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.639 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:10.639 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:27.161278+0000", 2024-09-13T14:59:10.639 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3843000000000001e-05 2024-09-13T14:59:10.639 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.639 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:10.639 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.639 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.639 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.639 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:26472 10.d 10:b076cb80:::benchmark_data_smithi151_288181_object26471:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1273)", 2024-09-13T14:59:10.639 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:58:27.057474+0000", 2024-09-13T14:59:10.639 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 43.489481541000004, 2024-09-13T14:59:10.639 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.105677513, 2024-09-13T14:59:10.640 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:10.640 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:10.640 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:10.640 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T14:59:10.640 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T14:59:10.640 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 26472 2024-09-13T14:59:10.640 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.640 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:10.640 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.640 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:10.640 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:27.057474+0000", 2024-09-13T14:59:10.640 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.640 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.641 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.641 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:10.641 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:27.057474+0000", 2024-09-13T14:59:10.641 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.641 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.641 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.641 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:10.641 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:27.057473+0000", 2024-09-13T14:59:10.641 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:59:10.641 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.641 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.641 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:10.643 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:10.643 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:27.057512+0000", 2024-09-13T14:59:10.643 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9341999999999997e-05 2024-09-13T14:59:10.643 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.643 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.643 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:10.643 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:27.057519+0000", 2024-09-13T14:59:10.643 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.0720000000000002e-06 2024-09-13T14:59:10.643 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.643 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.644 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:10.644 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:27.057523+0000", 2024-09-13T14:59:10.644 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9979999999999997e-06 2024-09-13T14:59:10.644 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.644 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.644 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:10.644 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:27.058636+0000", 2024-09-13T14:59:10.644 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001113066 2024-09-13T14:59:10.644 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.644 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.644 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:10.644 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:27.058897+0000", 2024-09-13T14:59:10.644 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00026037100000000002 2024-09-13T14:59:10.645 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.645 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.645 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:59:10.645 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:27.059028+0000", 2024-09-13T14:59:10.645 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013090500000000001 2024-09-13T14:59:10.645 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.645 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.645 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:10.645 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:27.060433+0000", 2024-09-13T14:59:10.645 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001405442 2024-09-13T14:59:10.645 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.645 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.645 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:10.646 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:27.060905+0000", 2024-09-13T14:59:10.646 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00047207199999999998 2024-09-13T14:59:10.646 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.646 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.646 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:10.646 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:27.163039+0000", 2024-09-13T14:59:10.646 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.10213367399999999 2024-09-13T14:59:10.646 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.646 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.646 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:10.646 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:27.163118+0000", 2024-09-13T14:59:10.646 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.8561999999999998e-05 2024-09-13T14:59:10.646 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.646 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.647 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:10.647 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:27.163151+0000", 2024-09-13T14:59:10.647 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3584000000000002e-05 2024-09-13T14:59:10.647 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.647 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:10.647 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.647 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.647 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.647 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:66595 10.e 10:7dc86aed:::benchmark_data_smithi151_288181_object66594:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1275)", 2024-09-13T14:59:10.647 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:58:51.175004+0000", 2024-09-13T14:59:10.647 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 19.371950775999998, 2024-09-13T14:59:10.647 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.16272383100000001, 2024-09-13T14:59:10.647 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:10.647 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:10.647 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:10.648 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T14:59:10.648 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T14:59:10.648 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 66595 2024-09-13T14:59:10.648 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.648 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:10.648 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.648 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:10.648 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.175004+0000", 2024-09-13T14:59:10.648 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.648 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.648 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.648 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:10.648 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.175004+0000", 2024-09-13T14:59:10.648 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.649 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.649 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.649 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:10.649 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.175003+0000", 2024-09-13T14:59:10.649 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:59:10.649 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.649 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.649 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:10.649 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.175062+0000", 2024-09-13T14:59:10.649 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.9284999999999999e-05 2024-09-13T14:59:10.649 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.649 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.649 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:10.650 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.175077+0000", 2024-09-13T14:59:10.650 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4917e-05 2024-09-13T14:59:10.650 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.650 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.650 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:10.650 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.175093+0000", 2024-09-13T14:59:10.650 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6172e-05 2024-09-13T14:59:10.650 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.650 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.650 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:10.650 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.175133+0000", 2024-09-13T14:59:10.650 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9919000000000002e-05 2024-09-13T14:59:10.650 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.650 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.651 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:10.651 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.175274+0000", 2024-09-13T14:59:10.651 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014076200000000001 2024-09-13T14:59:10.651 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.651 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.651 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:59:10.651 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.175410+0000", 2024-09-13T14:59:10.651 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000136704 2024-09-13T14:59:10.651 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.651 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.651 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:10.651 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.178053+0000", 2024-09-13T14:59:10.651 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0026429219999999998 2024-09-13T14:59:10.651 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.651 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.651 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:10.652 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.178197+0000", 2024-09-13T14:59:10.652 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000143917 2024-09-13T14:59:10.652 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.652 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.652 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:10.652 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.337669+0000", 2024-09-13T14:59:10.652 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.15947213599999999 2024-09-13T14:59:10.652 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.652 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.652 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:10.652 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.337704+0000", 2024-09-13T14:59:10.652 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.4783000000000002e-05 2024-09-13T14:59:10.652 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.652 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.652 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:10.653 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.337728+0000", 2024-09-13T14:59:10.653 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.4049999999999998e-05 2024-09-13T14:59:10.653 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.653 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:10.653 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.653 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.653 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.653 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:66607 10.e 10:7ef74f71:::benchmark_data_smithi151_288181_object66606:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1275)", 2024-09-13T14:59:10.653 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:58:51.188011+0000", 2024-09-13T14:59:10.653 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 19.358943884999999, 2024-09-13T14:59:10.653 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.14979794199999999, 2024-09-13T14:59:10.653 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:10.653 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:10.653 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:10.654 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T14:59:10.654 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T14:59:10.654 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 66607 2024-09-13T14:59:10.654 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.654 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:10.654 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.654 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:10.654 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.188011+0000", 2024-09-13T14:59:10.654 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.654 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.654 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.654 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:10.654 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.188011+0000", 2024-09-13T14:59:10.654 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.654 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.655 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.655 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:10.655 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.188010+0000", 2024-09-13T14:59:10.655 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:59:10.655 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.655 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.655 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:10.655 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.188066+0000", 2024-09-13T14:59:10.655 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5597999999999999e-05 2024-09-13T14:59:10.655 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.655 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.655 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:10.655 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.188082+0000", 2024-09-13T14:59:10.655 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.5767999999999999e-05 2024-09-13T14:59:10.656 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.656 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.656 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:10.656 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.188088+0000", 2024-09-13T14:59:10.656 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3380000000000001e-06 2024-09-13T14:59:10.656 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.656 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.656 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:10.656 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.188131+0000", 2024-09-13T14:59:10.656 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.2654000000000002e-05 2024-09-13T14:59:10.656 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.656 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.656 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:10.656 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.188252+0000", 2024-09-13T14:59:10.656 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000120926 2024-09-13T14:59:10.657 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.657 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.657 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:59:10.657 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.188396+0000", 2024-09-13T14:59:10.657 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014447799999999999 2024-09-13T14:59:10.657 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.657 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.657 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:10.657 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.190400+0000", 2024-09-13T14:59:10.657 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0020043790000000001 2024-09-13T14:59:10.657 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.657 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.657 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:10.657 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.190771+0000", 2024-09-13T14:59:10.658 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00037063800000000001 2024-09-13T14:59:10.658 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.658 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.658 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:10.658 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.337758+0000", 2024-09-13T14:59:10.658 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.146986755 2024-09-13T14:59:10.658 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.658 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.658 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:10.658 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.337788+0000", 2024-09-13T14:59:10.658 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.0218000000000001e-05 2024-09-13T14:59:10.658 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.658 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.658 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:10.659 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.337809+0000", 2024-09-13T14:59:10.659 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.1169999999999999e-05 2024-09-13T14:59:10.659 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.659 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:10.659 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.659 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.659 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.659 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:68559 10.d 10:b7ddfe54:::benchmark_data_smithi151_288181_object68558:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1275)", 2024-09-13T14:59:10.659 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:58:52.739423+0000", 2024-09-13T14:59:10.659 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 17.807532452, 2024-09-13T14:59:10.659 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.107598192, 2024-09-13T14:59:10.659 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:10.659 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:10.660 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:10.660 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T14:59:10.660 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T14:59:10.660 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 68559 2024-09-13T14:59:10.660 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.660 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:10.660 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.660 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:10.660 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.739423+0000", 2024-09-13T14:59:10.660 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.660 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.660 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.660 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:10.660 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.739423+0000", 2024-09-13T14:59:10.660 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.661 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.661 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.661 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:10.661 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.739421+0000", 2024-09-13T14:59:10.661 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:59:10.661 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.661 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.661 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:10.661 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.739484+0000", 2024-09-13T14:59:10.661 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3077999999999995e-05 2024-09-13T14:59:10.661 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.661 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.661 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:10.662 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.739499+0000", 2024-09-13T14:59:10.662 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4876e-05 2024-09-13T14:59:10.662 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.662 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.662 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:10.662 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.739505+0000", 2024-09-13T14:59:10.662 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1079999999999998e-06 2024-09-13T14:59:10.662 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.662 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.662 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:10.662 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.739553+0000", 2024-09-13T14:59:10.662 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8273999999999999e-05 2024-09-13T14:59:10.662 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.662 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.663 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:10.663 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.739673+0000", 2024-09-13T14:59:10.663 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000119951 2024-09-13T14:59:10.663 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.663 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.663 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:59:10.663 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.739810+0000", 2024-09-13T14:59:10.663 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000136935 2024-09-13T14:59:10.663 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.663 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.663 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:10.663 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.746309+0000", 2024-09-13T14:59:10.663 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0064988820000000001 2024-09-13T14:59:10.663 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.663 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.664 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:10.664 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.746347+0000", 2024-09-13T14:59:10.664 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.7703000000000001e-05 2024-09-13T14:59:10.664 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.664 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.664 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:10.664 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.846868+0000", 2024-09-13T14:59:10.664 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.100521178 2024-09-13T14:59:10.664 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.664 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.664 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:10.664 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.846981+0000", 2024-09-13T14:59:10.664 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000113349 2024-09-13T14:59:10.664 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.664 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.665 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:10.665 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.847021+0000", 2024-09-13T14:59:10.665 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9719999999999999e-05 2024-09-13T14:59:10.665 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.665 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:10.665 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.665 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.665 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.665 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:68563 10.d 10:be558d9f:::benchmark_data_smithi151_288181_object68562:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1275)", 2024-09-13T14:59:10.665 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:58:52.741662+0000", 2024-09-13T14:59:10.665 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 17.805292894000001, 2024-09-13T14:59:10.665 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.105501162, 2024-09-13T14:59:10.665 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:10.666 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:10.666 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:10.666 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T14:59:10.666 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T14:59:10.666 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 68563 2024-09-13T14:59:10.666 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.666 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:10.666 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.666 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:10.666 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.741662+0000", 2024-09-13T14:59:10.666 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.666 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.666 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.666 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:10.667 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.741662+0000", 2024-09-13T14:59:10.667 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.667 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.667 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.667 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:10.667 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.741661+0000", 2024-09-13T14:59:10.667 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:59:10.667 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.667 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.667 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:10.667 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.741729+0000", 2024-09-13T14:59:10.667 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7747e-05 2024-09-13T14:59:10.667 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.668 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.668 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:10.668 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.741744+0000", 2024-09-13T14:59:10.668 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4448e-05 2024-09-13T14:59:10.668 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.668 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.668 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:10.668 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.741749+0000", 2024-09-13T14:59:10.668 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3920000000000002e-06 2024-09-13T14:59:10.668 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.668 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.668 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:10.668 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.742218+0000", 2024-09-13T14:59:10.669 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00046943799999999997 2024-09-13T14:59:10.669 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.669 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.669 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:10.669 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.742354+0000", 2024-09-13T14:59:10.669 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013558799999999999 2024-09-13T14:59:10.669 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.669 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.669 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:59:10.669 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.742492+0000", 2024-09-13T14:59:10.669 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000137861 2024-09-13T14:59:10.669 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.669 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.669 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:10.670 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.746352+0000", 2024-09-13T14:59:10.670 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0038604139999999999 2024-09-13T14:59:10.670 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.670 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.670 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:10.670 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.746396+0000", 2024-09-13T14:59:10.670 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3325000000000003e-05 2024-09-13T14:59:10.670 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.670 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.670 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:10.670 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.847056+0000", 2024-09-13T14:59:10.670 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.100660598 2024-09-13T14:59:10.670 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.670 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.671 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:10.671 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.847128+0000", 2024-09-13T14:59:10.671 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.1466999999999999e-05 2024-09-13T14:59:10.671 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.671 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.671 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:10.671 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.847163+0000", 2024-09-13T14:59:10.671 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.5821999999999997e-05 2024-09-13T14:59:10.671 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.671 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:10.671 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.671 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.671 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.671 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:68577 10.d 10:b2345661:::benchmark_data_smithi151_288181_object68576:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1275)", 2024-09-13T14:59:10.672 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:58:52.751511+0000", 2024-09-13T14:59:10.672 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 17.795444516, 2024-09-13T14:59:10.672 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.101698181, 2024-09-13T14:59:10.672 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:10.672 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:10.672 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:10.672 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T14:59:10.672 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T14:59:10.672 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 68577 2024-09-13T14:59:10.672 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.672 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:10.672 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.672 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:10.673 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.751511+0000", 2024-09-13T14:59:10.673 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.673 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.673 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.673 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:10.673 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.751511+0000", 2024-09-13T14:59:10.673 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:10.673 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.673 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.673 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:10.673 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.751509+0000", 2024-09-13T14:59:10.673 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:10.673 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.673 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.674 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:10.674 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.751596+0000", 2024-09-13T14:59:10.674 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.6547999999999999e-05 2024-09-13T14:59:10.674 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.674 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.674 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:10.674 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.751611+0000", 2024-09-13T14:59:10.674 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4802e-05 2024-09-13T14:59:10.674 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.674 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.674 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:10.674 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.751617+0000", 2024-09-13T14:59:10.674 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.066e-06 2024-09-13T14:59:10.675 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.675 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.675 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:10.675 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.751658+0000", 2024-09-13T14:59:10.675 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0744000000000002e-05 2024-09-13T14:59:10.675 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.675 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.675 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:10.675 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.751769+0000", 2024-09-13T14:59:10.675 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00011186 2024-09-13T14:59:10.675 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.675 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.675 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:59:10.676 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.751901+0000", 2024-09-13T14:59:10.676 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013117399999999999 2024-09-13T14:59:10.676 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.676 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.676 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:10.676 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.754176+0000", 2024-09-13T14:59:10.676 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002275661 2024-09-13T14:59:10.676 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.676 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.676 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:10.676 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.754324+0000", 2024-09-13T14:59:10.676 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014753300000000001 2024-09-13T14:59:10.676 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.676 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.677 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:10.677 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.853005+0000", 2024-09-13T14:59:10.677 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.098680806999999995 2024-09-13T14:59:10.677 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.677 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.677 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:10.677 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.853165+0000", 2024-09-13T14:59:10.677 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00015992 2024-09-13T14:59:10.677 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:10.677 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:10.677 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:10.677 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:52.853209+0000", 2024-09-13T14:59:10.677 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4289000000000002e-05 2024-09-13T14:59:10.677 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.678 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:10.678 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.678 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:10.678 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:10.678 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:59:10.713 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:10.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:10.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:10.814 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:10.855 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:10.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:10.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:10.957 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:10.994 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:10.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:10.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:11.099 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:11.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:10 smithi151 ceph-mon[104450]: pgmap v2074: 119 pgs: 119 active+clean; 5.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-13T14:59:11.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:10 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/143109077' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:11.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:10 smithi151 ceph-mon[108966]: pgmap v2074: 119 pgs: 119 active+clean; 5.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-13T14:59:11.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:10 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/143109077' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:11.134 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:11.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:11.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:11.196 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 54 16 90025 90009 104.157 71.25 0.00948162 0.00959598 2024-09-13T14:59:11.236 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:11.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:11.273 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:11.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:11.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:10 smithi184 ceph-mon[92527]: pgmap v2074: 119 pgs: 119 active+clean; 5.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-13T14:59:11.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:10 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/143109077' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:11.374 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:11.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:11.403 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:11.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:11.504 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:11.535 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T14:59:11.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:11.542 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:11.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:11.643 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:11.691 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:11.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:11.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:11.793 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:11.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:11.826 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:11.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:11.927 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:11.959 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:11.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:11.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:12.009 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:59:12.061 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:12.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:12.110 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:12.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:12.196 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 55 16 91188 91172 103.585 72.6875 0.0142223 0.00964902 2024-09-13T14:59:12.212 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:12.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:12.250 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:12.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:12.351 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:12.397 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:12.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:12.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:12.499 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:12.543 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:12.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:12.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:12.645 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:12.678 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:12.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:12.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:12.780 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:12.820 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:12.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:12.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:12.923 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:12.962 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:12.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:12.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:13.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:13.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:13.103 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:13.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:13.197 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 56 16 92374 92358 103.058 74.125 0.010909 0.00969834 2024-09-13T14:59:13.204 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:13.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:13.252 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:13.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:13.277 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:12 smithi151 ceph-mon[104450]: pgmap v2075: 119 pgs: 119 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-13T14:59:13.278 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:12 smithi151 ceph-mon[108966]: pgmap v2075: 119 pgs: 119 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-13T14:59:13.311 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:12 smithi184 ceph-mon[92527]: pgmap v2075: 119 pgs: 119 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-13T14:59:13.319 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:59:13.353 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:13.404 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:13.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:13.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:13.506 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:13.533 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:13.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:13.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:13.635 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:13.692 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:13.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:13.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:13.794 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:13.826 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:13.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:13.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:13.871 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:13.901 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:59:13.927 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:13.961 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:13.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:13.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:13.963 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:59:13.963 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [ 2024-09-13T14:59:13.963 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:13.963 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.36396.0:93161 10.3 e1276/1268 10:ce65bc35:::benchmark_data_smithi151_288181_object93160:head v 1276'5830, mlcod=1276'5828)", 2024-09-13T14:59:13.963 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:59:13.958965+0000", 2024-09-13T14:59:13.963 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.0019905589999999998, 2024-09-13T14:59:13.963 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002020104, 2024-09-13T14:59:13.963 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:13.963 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T14:59:13.963 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:13.963 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:13.963 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:13.964 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:13.958965+0000", 2024-09-13T14:59:13.964 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:13.964 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:13.964 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:13.964 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:13.964 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:13.958965+0000", 2024-09-13T14:59:13.964 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:13.964 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:13.964 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:13.964 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:13.964 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:13.958963+0000", 2024-09-13T14:59:13.964 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:13.964 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:13.965 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:13.965 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:13.965 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:13.959008+0000", 2024-09-13T14:59:13.965 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4702e-05 2024-09-13T14:59:13.965 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:13.965 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:13.965 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:13.965 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:13.959031+0000", 2024-09-13T14:59:13.965 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.3187999999999999e-05 2024-09-13T14:59:13.965 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:13.965 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:13.965 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:13.965 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:13.959038+0000", 2024-09-13T14:59:13.966 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.9920000000000001e-06 2024-09-13T14:59:13.966 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:13.966 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:13.966 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:13.966 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:13.959088+0000", 2024-09-13T14:59:13.966 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9743000000000001e-05 2024-09-13T14:59:13.966 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:13.966 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:13.966 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:13.966 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:13.959111+0000", 2024-09-13T14:59:13.966 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.3218e-05 2024-09-13T14:59:13.966 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:13.966 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:13.967 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:13.967 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:13.967 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-13T14:59:13.967 INFO:teuthology.orchestra.run.smithi184.stdout: "num_ops": 1 2024-09-13T14:59:13.967 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:59:14.063 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:14.093 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:14.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:14.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:14.155 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T14:59:14.156 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T14:59:14.194 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:14.197 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 57 15 93376 93361 102.35 62.6875 0.00728538 0.00976494 2024-09-13T14:59:14.221 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:14.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:14.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:14.323 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:14.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:14.365 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:14.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:14.466 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:14.502 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:14.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:14.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:14.604 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:14.635 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:14.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:14.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:14.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:14.777 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:14.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:14.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:14.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:59:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:59:14] "GET /metrics HTTP/1.1" 200 36699 "" "Prometheus/2.51.0" 2024-09-13T14:59:14.871 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T14:59:14.879 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:14.928 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:14.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:14.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:15.030 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:15.070 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:15.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:15.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:15.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:14 smithi184 ceph-mon[92527]: pgmap v2076: 119 pgs: 119 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-13T14:59:15.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:14 smithi184 ceph-mon[92527]: from='client.36447 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:59:15.172 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:15.197 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 58 16 94494 94478 101.789 69.8125 0.00604925 0.00981848 2024-09-13T14:59:15.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:15.232 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:15.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:15.276 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:14 smithi151 ceph-mon[104450]: pgmap v2076: 119 pgs: 119 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-13T14:59:15.277 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:14 smithi151 ceph-mon[104450]: from='client.36447 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:59:15.277 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:14 smithi151 ceph-mon[108966]: pgmap v2076: 119 pgs: 119 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-13T14:59:15.277 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:14 smithi151 ceph-mon[108966]: from='client.36447 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:59:15.303 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:59:15.333 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:15.386 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:15.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:15.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:15.487 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:15.535 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:15.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:15.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:15.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:15.685 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:15.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:15.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:15.787 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:15.821 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:15.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:15.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:15.923 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:15.962 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:15.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:15.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:16.064 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:16.098 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:16.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:16.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:16.197 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 59 16 95637 95621 101.274 71.4375 0.0203988 0.00986937 2024-09-13T14:59:16.200 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:16.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:16.241 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:16.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:16.342 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:16.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:16.374 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:16.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:16.475 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:16.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:16.525 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:16.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:16.626 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:16.679 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:16.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:16.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:16.781 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:16.815 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:16.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:16.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:16.917 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:16.956 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:16.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:16.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:16.967 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":2076,"stamp":"2024-09-13T14:59:13.769652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5995501517,"num_objects":91501,"num_object_clones":0,"num_object_copies":274503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91538,"num_write_kb":5854431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":860775,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94633,"ondisk_log_size":94633,"up":357,"acting":357,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":357,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":24076216,"kb_used_data":17528632,"kb_used_omap":149,"kb_used_meta":6547370,"kb_avail":725885000,"statfs":{"total":767960285184,"available":743306240000,"internally_reserved":0,"allocated":17949319168,"data_stored":17928118903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":153401,"internal_metadata":6704507079},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,3,5,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":722337792,"num_objects":11022,"num_object_clones":0,"num_object_copies":33066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11022,"num_write_kb":705408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004029"},"pg_stats":[{"pgid":"1.1d","version":"261'100","reported_seq":127,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.840076+0000","last_change":"2024-09-13T14:54:58.831284+0000","last_active":"2024-09-13T14:59:08.840076+0000","last_peered":"2024-09-13T14:59:08.840076+0000","last_clean":"2024-09-13T14:59:08.840076+0000","last_became_active":"2024-09-13T14:54:58.830090+0000","last_became_peered":"2024-09-13T14:54:58.830090+0000","last_unstale":"2024-09-13T14:59:08.840076+0000","last_undegraded":"2024-09-13T14:59:08.840076+0000","last_fullsized":"2024-09-13T14:59:08.840076+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:17.988345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:17.988345+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:02:05.633533+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":1988,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.834282+0000","last_change":"2024-09-13T14:55:52.137133+0000","last_active":"2024-09-13T14:59:08.834282+0000","last_peered":"2024-09-13T14:59:08.834282+0000","last_clean":"2024-09-13T14:59:08.834282+0000","last_became_active":"2024-09-13T14:55:41.862495+0000","last_became_peered":"2024-09-13T14:55:41.862495+0000","last_unstale":"2024-09-13T14:59:08.834282+0000","last_undegraded":"2024-09-13T14:59:08.834282+0000","last_fullsized":"2024-09-13T14:59:08.834282+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:51:35.574651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2280,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":67,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.903258+0000","last_change":"2024-09-13T14:58:05.196679+0000","last_active":"2024-09-13T14:58:46.903258+0000","last_peered":"2024-09-13T14:58:46.903258+0000","last_clean":"2024-09-13T14:58:46.903258+0000","last_became_active":"2024-09-13T14:58:05.196297+0000","last_became_peered":"2024-09-13T14:58:05.196297+0000","last_unstale":"2024-09-13T14:58:46.903258+0000","last_undegraded":"2024-09-13T14:58:46.903258+0000","last_fullsized":"2024-09-13T14:58:46.903258+0000","mapping_epoch":1267,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:56:17.644420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":272,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.839924+0000","last_change":"2024-09-13T14:45:19.810203+0000","last_active":"2024-09-13T14:59:08.839924+0000","last_peered":"2024-09-13T14:59:08.839924+0000","last_clean":"2024-09-13T14:59:08.839924+0000","last_became_active":"2024-09-13T14:45:19.809693+0000","last_became_peered":"2024-09-13T14:45:19.809693+0000","last_unstale":"2024-09-13T14:59:08.839924+0000","last_undegraded":"2024-09-13T14:59:08.839924+0000","last_fullsized":"2024-09-13T14:59:08.839924+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:20.219673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":271,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.844094+0000","last_change":"2024-09-13T14:45:19.807991+0000","last_active":"2024-09-13T14:59:08.844094+0000","last_peered":"2024-09-13T14:59:08.844094+0000","last_clean":"2024-09-13T14:59:08.844094+0000","last_became_active":"2024-09-13T14:45:19.807662+0000","last_became_peered":"2024-09-13T14:45:19.807662+0000","last_unstale":"2024-09-13T14:59:08.844094+0000","last_undegraded":"2024-09-13T14:59:08.844094+0000","last_fullsized":"2024-09-13T14:59:08.844094+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:29.306108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"261'100","reported_seq":1377,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.894819+0000","last_change":"2024-09-13T14:54:05.604842+0000","last_active":"2024-09-13T14:58:46.894819+0000","last_peered":"2024-09-13T14:58:46.894819+0000","last_clean":"2024-09-13T14:58:46.894819+0000","last_became_active":"2024-09-13T14:54:05.604072+0000","last_became_peered":"2024-09-13T14:54:05.604072+0000","last_unstale":"2024-09-13T14:58:46.894819+0000","last_undegraded":"2024-09-13T14:58:46.894819+0000","last_fullsized":"2024-09-13T14:58:46.894819+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_clean_scrub_stamp":"2024-09-13T14:44:54.259482+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:41:25.897474+0000","scrub_duration":0.015265569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":53,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.986756+0000","last_change":"2024-09-13T14:55:41.870022+0000","last_active":"2024-09-13T14:58:46.986756+0000","last_peered":"2024-09-13T14:58:46.986756+0000","last_clean":"2024-09-13T14:58:46.986756+0000","last_became_active":"2024-09-13T14:55:41.786811+0000","last_became_peered":"2024-09-13T14:55:41.786811+0000","last_unstale":"2024-09-13T14:58:46.986756+0000","last_undegraded":"2024-09-13T14:58:46.986756+0000","last_fullsized":"2024-09-13T14:58:46.986756+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:38:18.373670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":57,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.969164+0000","last_change":"2024-09-13T14:54:58.830988+0000","last_active":"2024-09-13T14:58:46.969164+0000","last_peered":"2024-09-13T14:58:46.969164+0000","last_clean":"2024-09-13T14:58:46.969164+0000","last_became_active":"2024-09-13T14:54:58.830686+0000","last_became_peered":"2024-09-13T14:54:58.830686+0000","last_unstale":"2024-09-13T14:58:46.969164+0000","last_undegraded":"2024-09-13T14:58:46.969164+0000","last_fullsized":"2024-09-13T14:58:46.969164+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:37.924642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"261'100","reported_seq":289,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.840127+0000","last_change":"2024-09-13T14:45:35.947332+0000","last_active":"2024-09-13T14:59:08.840127+0000","last_peered":"2024-09-13T14:59:08.840127+0000","last_clean":"2024-09-13T14:59:08.840127+0000","last_became_active":"2024-09-13T14:45:19.810775+0000","last_became_peered":"2024-09-13T14:45:19.810775+0000","last_unstale":"2024-09-13T14:59:08.840127+0000","last_undegraded":"2024-09-13T14:59:08.840127+0000","last_fullsized":"2024-09-13T14:59:08.840127+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:35.947265+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:43:41.569795+0000","last_clean_scrub_stamp":"2024-09-13T14:45:35.947265+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:00.606370+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":53,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.987052+0000","last_change":"2024-09-13T14:55:41.901299+0000","last_active":"2024-09-13T14:58:46.987052+0000","last_peered":"2024-09-13T14:58:46.987052+0000","last_clean":"2024-09-13T14:58:46.987052+0000","last_became_active":"2024-09-13T14:55:41.866844+0000","last_became_peered":"2024-09-13T14:55:41.866844+0000","last_unstale":"2024-09-13T14:58:46.987052+0000","last_undegraded":"2024-09-13T14:58:46.987052+0000","last_fullsized":"2024-09-13T14:58:46.987052+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:44.106308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":57,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.969473+0000","last_change":"2024-09-13T14:54:58.830335+0000","last_active":"2024-09-13T14:58:46.969473+0000","last_peered":"2024-09-13T14:58:46.969473+0000","last_clean":"2024-09-13T14:58:46.969473+0000","last_became_active":"2024-09-13T14:54:58.829978+0000","last_became_peered":"2024-09-13T14:54:58.829978+0000","last_unstale":"2024-09-13T14:58:46.969473+0000","last_undegraded":"2024-09-13T14:58:46.969473+0000","last_fullsized":"2024-09-13T14:58:46.969473+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:46.608799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":1945,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.833920+0000","last_change":"2024-09-13T14:54:58.832660+0000","last_active":"2024-09-13T14:59:08.833920+0000","last_peered":"2024-09-13T14:59:08.833920+0000","last_clean":"2024-09-13T14:59:08.833920+0000","last_became_active":"2024-09-13T14:54:58.831770+0000","last_became_peered":"2024-09-13T14:54:58.831770+0000","last_unstale":"2024-09-13T14:59:08.833920+0000","last_undegraded":"2024-09-13T14:59:08.833920+0000","last_fullsized":"2024-09-13T14:59:08.833920+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:07:14.004187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1953,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.835590+0000","last_change":"2024-09-13T14:54:05.592484+0000","last_active":"2024-09-13T14:59:08.835590+0000","last_peered":"2024-09-13T14:59:08.835590+0000","last_clean":"2024-09-13T14:59:08.835590+0000","last_became_active":"2024-09-13T14:54:05.592159+0000","last_became_peered":"2024-09-13T14:54:05.592159+0000","last_unstale":"2024-09-13T14:59:08.835590+0000","last_undegraded":"2024-09-13T14:59:08.835590+0000","last_fullsized":"2024-09-13T14:59:08.835590+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:02.607037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"261'100","reported_seq":271,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.844202+0000","last_change":"2024-09-13T14:45:19.813412+0000","last_active":"2024-09-13T14:59:08.844202+0000","last_peered":"2024-09-13T14:59:08.844202+0000","last_clean":"2024-09-13T14:59:08.844202+0000","last_became_active":"2024-09-13T14:45:19.812585+0000","last_became_peered":"2024-09-13T14:45:19.812585+0000","last_unstale":"2024-09-13T14:59:08.844202+0000","last_undegraded":"2024-09-13T14:59:08.844202+0000","last_fullsized":"2024-09-13T14:59:08.844202+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.614340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.614340+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:32:07.434109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":57,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.969302+0000","last_change":"2024-09-13T14:54:58.831851+0000","last_active":"2024-09-13T14:58:46.969302+0000","last_peered":"2024-09-13T14:58:46.969302+0000","last_clean":"2024-09-13T14:58:46.969302+0000","last_became_active":"2024-09-13T14:54:58.831477+0000","last_became_peered":"2024-09-13T14:54:58.831477+0000","last_unstale":"2024-09-13T14:58:46.969302+0000","last_undegraded":"2024-09-13T14:58:46.969302+0000","last_fullsized":"2024-09-13T14:58:46.969302+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:42.562225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":306,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.850240+0000","last_change":"2024-09-13T14:54:58.827147+0000","last_active":"2024-09-13T14:59:08.850240+0000","last_peered":"2024-09-13T14:59:08.850240+0000","last_clean":"2024-09-13T14:59:08.850240+0000","last_became_active":"2024-09-13T14:54:58.826783+0000","last_became_peered":"2024-09-13T14:54:58.826783+0000","last_unstale":"2024-09-13T14:59:08.850240+0000","last_undegraded":"2024-09-13T14:59:08.850240+0000","last_fullsized":"2024-09-13T14:59:08.850240+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:40:27.819584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.18","version":"261'100","reported_seq":53,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.987311+0000","last_change":"2024-09-13T14:57:11.038947+0000","last_active":"2024-09-13T14:58:46.987311+0000","last_peered":"2024-09-13T14:58:46.987311+0000","last_clean":"2024-09-13T14:58:46.987311+0000","last_became_active":"2024-09-13T14:55:41.838842+0000","last_became_peered":"2024-09-13T14:55:41.838842+0000","last_unstale":"2024-09-13T14:58:46.987311+0000","last_undegraded":"2024-09-13T14:58:46.987311+0000","last_fullsized":"2024-09-13T14:58:46.987311+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1246,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:56.548855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:56.548855+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:52.715319+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":127,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.843569+0000","last_change":"2024-09-13T14:54:58.821234+0000","last_active":"2024-09-13T14:59:08.843569+0000","last_peered":"2024-09-13T14:59:08.843569+0000","last_clean":"2024-09-13T14:59:08.843569+0000","last_became_active":"2024-09-13T14:54:58.820948+0000","last_became_peered":"2024-09-13T14:54:58.820948+0000","last_unstale":"2024-09-13T14:59:08.843569+0000","last_undegraded":"2024-09-13T14:59:08.843569+0000","last_fullsized":"2024-09-13T14:59:08.843569+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:58:51.172714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1956,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.834781+0000","last_change":"2024-09-13T14:45:19.815341+0000","last_active":"2024-09-13T14:59:08.834781+0000","last_peered":"2024-09-13T14:59:08.834781+0000","last_clean":"2024-09-13T14:59:08.834781+0000","last_became_active":"2024-09-13T14:45:19.810262+0000","last_became_peered":"2024-09-13T14:45:19.810262+0000","last_unstale":"2024-09-13T14:59:08.834781+0000","last_undegraded":"2024-09-13T14:59:08.834781+0000","last_fullsized":"2024-09-13T14:59:08.834781+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:39.847653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1b","version":"261'100","reported_seq":49,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.987947+0000","last_change":"2024-09-13T14:55:41.919999+0000","last_active":"2024-09-13T14:58:46.987947+0000","last_peered":"2024-09-13T14:58:46.987947+0000","last_clean":"2024-09-13T14:58:46.987947+0000","last_became_active":"2024-09-13T14:55:41.919444+0000","last_became_peered":"2024-09-13T14:55:41.919444+0000","last_unstale":"2024-09-13T14:58:46.987947+0000","last_undegraded":"2024-09-13T14:58:46.987947+0000","last_fullsized":"2024-09-13T14:58:46.987947+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:55.988431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:55.988431+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:22:19.385914+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":285,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.840840+0000","last_change":"2024-09-13T14:54:05.606614+0000","last_active":"2024-09-13T14:59:08.840840+0000","last_peered":"2024-09-13T14:59:08.840840+0000","last_clean":"2024-09-13T14:59:08.840840+0000","last_became_active":"2024-09-13T14:54:05.605524+0000","last_became_peered":"2024-09-13T14:54:05.605524+0000","last_unstale":"2024-09-13T14:59:08.840840+0000","last_undegraded":"2024-09-13T14:59:08.840840+0000","last_fullsized":"2024-09-13T14:59:08.840840+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:59.484071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":49,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.987421+0000","last_change":"2024-09-13T14:55:41.883426+0000","last_active":"2024-09-13T14:58:46.987421+0000","last_peered":"2024-09-13T14:58:46.987421+0000","last_clean":"2024-09-13T14:58:46.987421+0000","last_became_active":"2024-09-13T14:55:41.880724+0000","last_became_peered":"2024-09-13T14:55:41.880724+0000","last_unstale":"2024-09-13T14:58:46.987421+0000","last_undegraded":"2024-09-13T14:58:46.987421+0000","last_fullsized":"2024-09-13T14:58:46.987421+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:17.920508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1a","version":"261'100","reported_seq":57,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.969443+0000","last_change":"2024-09-13T14:54:58.831623+0000","last_active":"2024-09-13T14:58:46.969443+0000","last_peered":"2024-09-13T14:58:46.969443+0000","last_clean":"2024-09-13T14:58:46.969443+0000","last_became_active":"2024-09-13T14:54:58.831026+0000","last_became_peered":"2024-09-13T14:54:58.831026+0000","last_unstale":"2024-09-13T14:58:46.969443+0000","last_undegraded":"2024-09-13T14:58:46.969443+0000","last_fullsized":"2024-09-13T14:58:46.969443+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:46.985510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:46.985510+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:27:24.529132+0000","scrub_duration":0.0030525999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":292,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.840955+0000","last_change":"2024-09-13T14:54:05.606882+0000","last_active":"2024-09-13T14:59:08.840955+0000","last_peered":"2024-09-13T14:59:08.840955+0000","last_clean":"2024-09-13T14:59:08.840955+0000","last_became_active":"2024-09-13T14:54:05.605912+0000","last_became_peered":"2024-09-13T14:54:05.605912+0000","last_unstale":"2024-09-13T14:59:08.840955+0000","last_undegraded":"2024-09-13T14:59:08.840955+0000","last_fullsized":"2024-09-13T14:59:08.840955+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:05.445329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2000,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.834192+0000","last_change":"2024-09-13T14:55:41.868797+0000","last_active":"2024-09-13T14:59:08.834192+0000","last_peered":"2024-09-13T14:59:08.834192+0000","last_clean":"2024-09-13T14:59:08.834192+0000","last_became_active":"2024-09-13T14:55:41.865613+0000","last_became_peered":"2024-09-13T14:55:41.865613+0000","last_unstale":"2024-09-13T14:59:08.834192+0000","last_undegraded":"2024-09-13T14:59:08.834192+0000","last_fullsized":"2024-09-13T14:59:08.834192+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:16:29.230176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"261'100","reported_seq":227,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.966765+0000","last_change":"2024-09-13T14:55:41.868936+0000","last_active":"2024-09-13T14:58:46.966765+0000","last_peered":"2024-09-13T14:58:46.966765+0000","last_clean":"2024-09-13T14:58:46.966765+0000","last_became_active":"2024-09-13T14:55:41.865505+0000","last_became_peered":"2024-09-13T14:55:41.865505+0000","last_unstale":"2024-09-13T14:58:46.966765+0000","last_undegraded":"2024-09-13T14:58:46.966765+0000","last_fullsized":"2024-09-13T14:58:46.966765+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:38.214865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:38.214865+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:45:08.788150+0000","scrub_duration":0.0016389639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1843,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.844397+0000","last_change":"2024-09-13T14:30:15.576921+0000","last_active":"2024-09-13T14:59:08.844397+0000","last_peered":"2024-09-13T14:59:08.844397+0000","last_clean":"2024-09-13T14:59:08.844397+0000","last_became_active":"2024-09-13T14:30:15.576607+0000","last_became_peered":"2024-09-13T14:30:15.576607+0000","last_unstale":"2024-09-13T14:59:08.844397+0000","last_undegraded":"2024-09-13T14:59:08.844397+0000","last_fullsized":"2024-09-13T14:59:08.844397+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:35.162460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":49,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.987496+0000","last_change":"2024-09-13T14:55:41.868620+0000","last_active":"2024-09-13T14:58:46.987496+0000","last_peered":"2024-09-13T14:58:46.987496+0000","last_clean":"2024-09-13T14:58:46.987496+0000","last_became_active":"2024-09-13T14:55:41.868289+0000","last_became_peered":"2024-09-13T14:55:41.868289+0000","last_unstale":"2024-09-13T14:58:46.987496+0000","last_undegraded":"2024-09-13T14:58:46.987496+0000","last_fullsized":"2024-09-13T14:58:46.987496+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:17:28.109694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"261'100","reported_seq":348,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.835445+0000","last_change":"2024-09-13T14:54:58.824599+0000","last_active":"2024-09-13T14:59:08.835445+0000","last_peered":"2024-09-13T14:59:08.835445+0000","last_clean":"2024-09-13T14:59:08.835445+0000","last_became_active":"2024-09-13T14:54:58.824304+0000","last_became_peered":"2024-09-13T14:54:58.824304+0000","last_unstale":"2024-09-13T14:59:08.835445+0000","last_undegraded":"2024-09-13T14:59:08.835445+0000","last_fullsized":"2024-09-13T14:59:08.835445+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.037517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.037517+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:42.296498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":141,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.845106+0000","last_change":"2024-09-13T14:54:58.836713+0000","last_active":"2024-09-13T14:59:08.845106+0000","last_peered":"2024-09-13T14:59:08.845106+0000","last_clean":"2024-09-13T14:59:08.845106+0000","last_became_active":"2024-09-13T14:54:58.827723+0000","last_became_peered":"2024-09-13T14:54:58.827723+0000","last_unstale":"2024-09-13T14:59:08.845106+0000","last_undegraded":"2024-09-13T14:59:08.845106+0000","last_fullsized":"2024-09-13T14:59:08.845106+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:10:25.743550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":120,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.841004+0000","last_change":"2024-09-13T14:54:05.614569+0000","last_active":"2024-09-13T14:59:08.841004+0000","last_peered":"2024-09-13T14:59:08.841004+0000","last_clean":"2024-09-13T14:59:08.841004+0000","last_became_active":"2024-09-13T14:54:05.611810+0000","last_became_peered":"2024-09-13T14:54:05.611810+0000","last_unstale":"2024-09-13T14:59:08.841004+0000","last_undegraded":"2024-09-13T14:59:08.841004+0000","last_fullsized":"2024-09-13T14:59:08.841004+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:46:02.721438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"261'100","reported_seq":49,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.987414+0000","last_change":"2024-09-13T14:55:41.860977+0000","last_active":"2024-09-13T14:58:46.987414+0000","last_peered":"2024-09-13T14:58:46.987414+0000","last_clean":"2024-09-13T14:58:46.987414+0000","last_became_active":"2024-09-13T14:55:41.859488+0000","last_became_peered":"2024-09-13T14:55:41.859488+0000","last_unstale":"2024-09-13T14:58:46.987414+0000","last_undegraded":"2024-09-13T14:58:46.987414+0000","last_fullsized":"2024-09-13T14:58:46.987414+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:54.933410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:54.933410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:48.738084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":67,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.970005+0000","last_change":"2024-09-13T14:54:58.838308+0000","last_active":"2024-09-13T14:58:46.970005+0000","last_peered":"2024-09-13T14:58:46.970005+0000","last_clean":"2024-09-13T14:58:46.970005+0000","last_became_active":"2024-09-13T14:54:58.825174+0000","last_became_peered":"2024-09-13T14:54:58.825174+0000","last_unstale":"2024-09-13T14:58:46.970005+0000","last_undegraded":"2024-09-13T14:58:46.970005+0000","last_fullsized":"2024-09-13T14:58:46.970005+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:59:20.787507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":57,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.969773+0000","last_change":"2024-09-13T14:54:58.831327+0000","last_active":"2024-09-13T14:58:46.969773+0000","last_peered":"2024-09-13T14:58:46.969773+0000","last_clean":"2024-09-13T14:58:46.969773+0000","last_became_active":"2024-09-13T14:54:58.830414+0000","last_became_peered":"2024-09-13T14:54:58.830414+0000","last_unstale":"2024-09-13T14:58:46.969773+0000","last_undegraded":"2024-09-13T14:58:46.969773+0000","last_fullsized":"2024-09-13T14:58:46.969773+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:12.501968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"261'100","reported_seq":120,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.844578+0000","last_change":"2024-09-13T14:54:05.613604+0000","last_active":"2024-09-13T14:59:08.844578+0000","last_peered":"2024-09-13T14:59:08.844578+0000","last_clean":"2024-09-13T14:59:08.844578+0000","last_became_active":"2024-09-13T14:54:05.613247+0000","last_became_peered":"2024-09-13T14:54:05.613247+0000","last_unstale":"2024-09-13T14:59:08.844578+0000","last_undegraded":"2024-09-13T14:59:08.844578+0000","last_fullsized":"2024-09-13T14:59:08.844578+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:48.363045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:48.363045+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:17.100985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.f","version":"1276'5697","reported_seq":5713,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:12.247930+0000","last_change":"2024-09-13T14:58:06.200417+0000","last_active":"2024-09-13T14:59:12.247930+0000","last_peered":"2024-09-13T14:59:12.247930+0000","last_clean":"2024-09-13T14:59:12.247930+0000","last_became_active":"2024-09-13T14:58:06.200062+0000","last_became_peered":"2024-09-13T14:58:06.200062+0000","last_unstale":"2024-09-13T14:59:12.247930+0000","last_undegraded":"2024-09-13T14:59:12.247930+0000","last_fullsized":"2024-09-13T14:59:12.247930+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":5697,"log_dups_size":0,"ondisk_log_size":5697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:53:26.095129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373358592,"num_objects":5697,"num_object_clones":0,"num_object_copies":17091,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5697,"num_write_kb":364608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":77,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.966451+0000","last_change":"2024-09-13T14:55:41.788238+0000","last_active":"2024-09-13T14:58:46.966451+0000","last_peered":"2024-09-13T14:58:46.966451+0000","last_clean":"2024-09-13T14:58:46.966451+0000","last_became_active":"2024-09-13T14:55:41.786422+0000","last_became_peered":"2024-09-13T14:55:41.786422+0000","last_unstale":"2024-09-13T14:58:46.966451+0000","last_undegraded":"2024-09-13T14:58:46.966451+0000","last_fullsized":"2024-09-13T14:58:46.966451+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:13:55.167916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":701,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.967633+0000","last_change":"2024-09-13T14:43:32.635070+0000","last_active":"2024-09-13T14:58:46.967633+0000","last_peered":"2024-09-13T14:58:46.967633+0000","last_clean":"2024-09-13T14:58:46.967633+0000","last_became_active":"2024-09-13T14:43:32.634734+0000","last_became_peered":"2024-09-13T14:43:32.634734+0000","last_unstale":"2024-09-13T14:58:46.967633+0000","last_undegraded":"2024-09-13T14:58:46.967633+0000","last_fullsized":"2024-09-13T14:58:46.967633+0000","mapping_epoch":1131,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:33.107010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":2018,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.834986+0000","last_change":"2024-09-13T14:54:58.846704+0000","last_active":"2024-09-13T14:59:08.834986+0000","last_peered":"2024-09-13T14:59:08.834986+0000","last_clean":"2024-09-13T14:59:08.834986+0000","last_became_active":"2024-09-13T14:54:58.830897+0000","last_became_peered":"2024-09-13T14:54:58.830897+0000","last_unstale":"2024-09-13T14:59:08.834986+0000","last_undegraded":"2024-09-13T14:59:08.834986+0000","last_fullsized":"2024-09-13T14:59:08.834986+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:31:37.389598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"261'100","reported_seq":300,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.843362+0000","last_change":"2024-09-13T14:55:41.845434+0000","last_active":"2024-09-13T14:59:08.843362+0000","last_peered":"2024-09-13T14:59:08.843362+0000","last_clean":"2024-09-13T14:59:08.843362+0000","last_became_active":"2024-09-13T14:55:41.845155+0000","last_became_peered":"2024-09-13T14:55:41.845155+0000","last_unstale":"2024-09-13T14:59:08.843362+0000","last_undegraded":"2024-09-13T14:59:08.843362+0000","last_fullsized":"2024-09-13T14:59:08.843362+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.710192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.710192+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:32.286774+0000","scrub_duration":0.0019250910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"1276'5596","reported_seq":5611,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:12.300921+0000","last_change":"2024-09-13T14:58:06.201145+0000","last_active":"2024-09-13T14:59:12.300921+0000","last_peered":"2024-09-13T14:59:12.300921+0000","last_clean":"2024-09-13T14:59:12.300921+0000","last_became_active":"2024-09-13T14:58:06.200769+0000","last_became_peered":"2024-09-13T14:58:06.200769+0000","last_unstale":"2024-09-13T14:59:12.300921+0000","last_undegraded":"2024-09-13T14:59:12.300921+0000","last_fullsized":"2024-09-13T14:59:12.300921+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":5596,"log_dups_size":0,"ondisk_log_size":5596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:18:09.369835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366739456,"num_objects":5596,"num_object_clones":0,"num_object_copies":16788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5596,"num_write_kb":358144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":175,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.840659+0000","last_change":"2024-09-13T14:52:54.233894+0000","last_active":"2024-09-13T14:59:08.840659+0000","last_peered":"2024-09-13T14:59:08.840659+0000","last_clean":"2024-09-13T14:59:08.840659+0000","last_became_active":"2024-09-13T14:52:54.233601+0000","last_became_peered":"2024-09-13T14:52:54.233601+0000","last_unstale":"2024-09-13T14:59:08.840659+0000","last_undegraded":"2024-09-13T14:59:08.840659+0000","last_fullsized":"2024-09-13T14:59:08.840659+0000","mapping_epoch":1195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:17:24.364701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":127,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.844582+0000","last_change":"2024-09-13T14:54:58.825887+0000","last_active":"2024-09-13T14:59:08.844582+0000","last_peered":"2024-09-13T14:59:08.844582+0000","last_clean":"2024-09-13T14:59:08.844582+0000","last_became_active":"2024-09-13T14:54:58.825350+0000","last_became_peered":"2024-09-13T14:54:58.825350+0000","last_unstale":"2024-09-13T14:59:08.844582+0000","last_undegraded":"2024-09-13T14:59:08.844582+0000","last_fullsized":"2024-09-13T14:59:08.844582+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:16.669031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":1876,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.845393+0000","last_change":"2024-09-13T14:30:15.581728+0000","last_active":"2024-09-13T14:59:08.845393+0000","last_peered":"2024-09-13T14:59:08.845393+0000","last_clean":"2024-09-13T14:59:08.845393+0000","last_became_active":"2024-09-13T14:30:15.581303+0000","last_became_peered":"2024-09-13T14:30:15.581303+0000","last_unstale":"2024-09-13T14:59:08.845393+0000","last_undegraded":"2024-09-13T14:59:08.845393+0000","last_fullsized":"2024-09-13T14:59:08.845393+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:10.027991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"261'100","reported_seq":57,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.972106+0000","last_change":"2024-09-13T14:54:58.822940+0000","last_active":"2024-09-13T14:58:46.972106+0000","last_peered":"2024-09-13T14:58:46.972106+0000","last_clean":"2024-09-13T14:58:46.972106+0000","last_became_active":"2024-09-13T14:54:58.822551+0000","last_became_peered":"2024-09-13T14:54:58.822551+0000","last_unstale":"2024-09-13T14:58:46.972106+0000","last_undegraded":"2024-09-13T14:58:46.972106+0000","last_fullsized":"2024-09-13T14:58:46.972106+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:51:31.310115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:51:31.310115+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:05:04.280014+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"1276'5786","reported_seq":5802,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:12.293157+0000","last_change":"2024-09-13T14:58:06.203698+0000","last_active":"2024-09-13T14:59:12.293157+0000","last_peered":"2024-09-13T14:59:12.293157+0000","last_clean":"2024-09-13T14:59:12.293157+0000","last_became_active":"2024-09-13T14:58:06.203402+0000","last_became_peered":"2024-09-13T14:58:06.203402+0000","last_unstale":"2024-09-13T14:59:12.293157+0000","last_undegraded":"2024-09-13T14:59:12.293157+0000","last_fullsized":"2024-09-13T14:59:12.293157+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":5786,"log_dups_size":0,"ondisk_log_size":5786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:59:27.300867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379191296,"num_objects":5786,"num_object_clones":0,"num_object_copies":17358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5786,"num_write_kb":370304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1951,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.833883+0000","last_change":"2024-09-13T14:55:41.914117+0000","last_active":"2024-09-13T14:59:08.833883+0000","last_peered":"2024-09-13T14:59:08.833883+0000","last_clean":"2024-09-13T14:59:08.833883+0000","last_became_active":"2024-09-13T14:55:41.913777+0000","last_became_peered":"2024-09-13T14:55:41.913777+0000","last_unstale":"2024-09-13T14:59:08.833883+0000","last_undegraded":"2024-09-13T14:59:08.833883+0000","last_fullsized":"2024-09-13T14:59:08.833883+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:40:31.224332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1282,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.966846+0000","last_change":"2024-09-13T14:54:58.825971+0000","last_active":"2024-09-13T14:58:46.966846+0000","last_peered":"2024-09-13T14:58:46.966846+0000","last_clean":"2024-09-13T14:58:46.966846+0000","last_became_active":"2024-09-13T14:54:58.822356+0000","last_became_peered":"2024-09-13T14:54:58.822356+0000","last_unstale":"2024-09-13T14:58:46.966846+0000","last_undegraded":"2024-09-13T14:58:46.966846+0000","last_fullsized":"2024-09-13T14:58:46.966846+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:14.163203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":134,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.839760+0000","last_change":"2024-09-13T14:55:41.892125+0000","last_active":"2024-09-13T14:59:08.839760+0000","last_peered":"2024-09-13T14:59:08.839760+0000","last_clean":"2024-09-13T14:59:08.839760+0000","last_became_active":"2024-09-13T14:55:41.891812+0000","last_became_peered":"2024-09-13T14:55:41.891812+0000","last_unstale":"2024-09-13T14:59:08.839760+0000","last_undegraded":"2024-09-13T14:59:08.839760+0000","last_fullsized":"2024-09-13T14:59:08.839760+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:03.359541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"261'100","reported_seq":642,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.834583+0000","last_change":"2024-09-13T14:45:19.815050+0000","last_active":"2024-09-13T14:59:08.834583+0000","last_peered":"2024-09-13T14:59:08.834583+0000","last_clean":"2024-09-13T14:59:08.834583+0000","last_became_active":"2024-09-13T14:45:19.809732+0000","last_became_peered":"2024-09-13T14:45:19.809732+0000","last_unstale":"2024-09-13T14:59:08.834583+0000","last_undegraded":"2024-09-13T14:59:08.834583+0000","last_fullsized":"2024-09-13T14:59:08.834583+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:33.574443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:33.574443+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:51:40.261036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.c","version":"1276'5646","reported_seq":5662,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:12.241063+0000","last_change":"2024-09-13T14:58:06.202247+0000","last_active":"2024-09-13T14:59:12.241063+0000","last_peered":"2024-09-13T14:59:12.241063+0000","last_clean":"2024-09-13T14:59:12.241063+0000","last_became_active":"2024-09-13T14:58:06.201537+0000","last_became_peered":"2024-09-13T14:58:06.201537+0000","last_unstale":"2024-09-13T14:59:12.241063+0000","last_undegraded":"2024-09-13T14:59:12.241063+0000","last_fullsized":"2024-09-13T14:59:12.241063+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":5646,"log_dups_size":0,"ondisk_log_size":5646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:45:06.849570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370016256,"num_objects":5646,"num_object_clones":0,"num_object_copies":16938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5646,"num_write_kb":361344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":141,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.841145+0000","last_change":"2024-09-13T14:55:41.929999+0000","last_active":"2024-09-13T14:59:08.841145+0000","last_peered":"2024-09-13T14:59:08.841145+0000","last_clean":"2024-09-13T14:59:08.841145+0000","last_became_active":"2024-09-13T14:55:41.926393+0000","last_became_peered":"2024-09-13T14:55:41.926393+0000","last_unstale":"2024-09-13T14:59:08.841145+0000","last_undegraded":"2024-09-13T14:59:08.841145+0000","last_fullsized":"2024-09-13T14:59:08.841145+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:16:40.228234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":120,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.839684+0000","last_change":"2024-09-13T14:54:05.605111+0000","last_active":"2024-09-13T14:59:08.839684+0000","last_peered":"2024-09-13T14:59:08.839684+0000","last_clean":"2024-09-13T14:59:08.839684+0000","last_became_active":"2024-09-13T14:54:05.599852+0000","last_became_peered":"2024-09-13T14:54:05.599852+0000","last_unstale":"2024-09-13T14:59:08.839684+0000","last_undegraded":"2024-09-13T14:59:08.839684+0000","last_fullsized":"2024-09-13T14:59:08.839684+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:11.238519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":49,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.986682+0000","last_change":"2024-09-13T14:55:41.839293+0000","last_active":"2024-09-13T14:58:46.986682+0000","last_peered":"2024-09-13T14:58:46.986682+0000","last_clean":"2024-09-13T14:58:46.986682+0000","last_became_active":"2024-09-13T14:55:41.838820+0000","last_became_peered":"2024-09-13T14:55:41.838820+0000","last_unstale":"2024-09-13T14:58:46.986682+0000","last_undegraded":"2024-09-13T14:58:46.986682+0000","last_fullsized":"2024-09-13T14:58:46.986682+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:14:27.811437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"261'100","reported_seq":289,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.845232+0000","last_change":"2024-09-13T14:45:36.965992+0000","last_active":"2024-09-13T14:59:08.845232+0000","last_peered":"2024-09-13T14:59:08.845232+0000","last_clean":"2024-09-13T14:59:08.845232+0000","last_became_active":"2024-09-13T14:45:19.811238+0000","last_became_peered":"2024-09-13T14:45:19.811238+0000","last_unstale":"2024-09-13T14:59:08.845232+0000","last_undegraded":"2024-09-13T14:59:08.845232+0000","last_fullsized":"2024-09-13T14:59:08.845232+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:36.965775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:45:36.965775+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:36:01.623251+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"1276'5634","reported_seq":5650,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:12.301600+0000","last_change":"2024-09-13T14:58:06.203687+0000","last_active":"2024-09-13T14:59:12.301600+0000","last_peered":"2024-09-13T14:59:12.301600+0000","last_clean":"2024-09-13T14:59:12.301600+0000","last_became_active":"2024-09-13T14:58:06.203340+0000","last_became_peered":"2024-09-13T14:58:06.203340+0000","last_unstale":"2024-09-13T14:59:12.301600+0000","last_undegraded":"2024-09-13T14:59:12.301600+0000","last_fullsized":"2024-09-13T14:59:12.301600+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":5634,"log_dups_size":0,"ondisk_log_size":5634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:17:52.886190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369229824,"num_objects":5634,"num_object_clones":0,"num_object_copies":16902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5634,"num_write_kb":360576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":120,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.839595+0000","last_change":"2024-09-13T14:54:05.596068+0000","last_active":"2024-09-13T14:59:08.839595+0000","last_peered":"2024-09-13T14:59:08.839595+0000","last_clean":"2024-09-13T14:59:08.839595+0000","last_became_active":"2024-09-13T14:54:05.595743+0000","last_became_peered":"2024-09-13T14:54:05.595743+0000","last_unstale":"2024-09-13T14:59:08.839595+0000","last_undegraded":"2024-09-13T14:59:08.839595+0000","last_fullsized":"2024-09-13T14:59:08.839595+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:15:39.660370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":57,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.969691+0000","last_change":"2024-09-13T14:54:58.831309+0000","last_active":"2024-09-13T14:58:46.969691+0000","last_peered":"2024-09-13T14:58:46.969691+0000","last_clean":"2024-09-13T14:58:46.969691+0000","last_became_active":"2024-09-13T14:54:58.830950+0000","last_became_peered":"2024-09-13T14:54:58.830950+0000","last_unstale":"2024-09-13T14:58:46.969691+0000","last_undegraded":"2024-09-13T14:58:46.969691+0000","last_fullsized":"2024-09-13T14:58:46.969691+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:16:54.838006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":98,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.987361+0000","last_change":"2024-09-13T14:55:41.889668+0000","last_active":"2024-09-13T14:58:46.987361+0000","last_peered":"2024-09-13T14:58:46.987361+0000","last_clean":"2024-09-13T14:58:46.987361+0000","last_became_active":"2024-09-13T14:55:41.889195+0000","last_became_peered":"2024-09-13T14:55:41.889195+0000","last_unstale":"2024-09-13T14:58:46.987361+0000","last_undegraded":"2024-09-13T14:58:46.987361+0000","last_fullsized":"2024-09-13T14:58:46.987361+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:52.765208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"261'100","reported_seq":49,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.987179+0000","last_change":"2024-09-13T14:55:41.897451+0000","last_active":"2024-09-13T14:58:46.987179+0000","last_peered":"2024-09-13T14:58:46.987179+0000","last_clean":"2024-09-13T14:58:46.987179+0000","last_became_active":"2024-09-13T14:55:41.896485+0000","last_became_peered":"2024-09-13T14:55:41.896485+0000","last_unstale":"2024-09-13T14:58:46.987179+0000","last_undegraded":"2024-09-13T14:58:46.987179+0000","last_fullsized":"2024-09-13T14:58:46.987179+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:57.024823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:57.024823+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:25:55.484738+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"1276'5697","reported_seq":5713,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:12.285002+0000","last_change":"2024-09-13T14:58:06.202180+0000","last_active":"2024-09-13T14:59:12.285002+0000","last_peered":"2024-09-13T14:59:12.285002+0000","last_clean":"2024-09-13T14:59:12.285002+0000","last_became_active":"2024-09-13T14:58:06.201886+0000","last_became_peered":"2024-09-13T14:58:06.201886+0000","last_unstale":"2024-09-13T14:59:12.285002+0000","last_undegraded":"2024-09-13T14:59:12.285002+0000","last_fullsized":"2024-09-13T14:59:12.285002+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":5697,"log_dups_size":0,"ondisk_log_size":5697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:48.418531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373358592,"num_objects":5697,"num_object_clones":0,"num_object_copies":17091,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5697,"num_write_kb":364608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":134,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.844283+0000","last_change":"2024-09-13T14:55:41.876729+0000","last_active":"2024-09-13T14:59:08.844283+0000","last_peered":"2024-09-13T14:59:08.844283+0000","last_clean":"2024-09-13T14:59:08.844283+0000","last_became_active":"2024-09-13T14:55:41.876280+0000","last_became_peered":"2024-09-13T14:55:41.876280+0000","last_unstale":"2024-09-13T14:59:08.844283+0000","last_undegraded":"2024-09-13T14:59:08.844283+0000","last_fullsized":"2024-09-13T14:59:08.844283+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:39:30.308792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":120,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.842676+0000","last_change":"2024-09-13T14:54:05.614107+0000","last_active":"2024-09-13T14:59:08.842676+0000","last_peered":"2024-09-13T14:59:08.842676+0000","last_clean":"2024-09-13T14:59:08.842676+0000","last_became_active":"2024-09-13T14:54:05.613429+0000","last_became_peered":"2024-09-13T14:54:05.613429+0000","last_unstale":"2024-09-13T14:59:08.842676+0000","last_undegraded":"2024-09-13T14:59:08.842676+0000","last_fullsized":"2024-09-13T14:59:08.842676+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:51.312413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1958,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.846121+0000","last_change":"2024-09-13T14:45:19.812713+0000","last_active":"2024-09-13T14:59:08.846121+0000","last_peered":"2024-09-13T14:59:08.846121+0000","last_clean":"2024-09-13T14:59:08.846121+0000","last_became_active":"2024-09-13T14:45:19.812203+0000","last_became_peered":"2024-09-13T14:45:19.812203+0000","last_unstale":"2024-09-13T14:59:08.846121+0000","last_undegraded":"2024-09-13T14:59:08.846121+0000","last_fullsized":"2024-09-13T14:59:08.846121+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:30:46.149347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"261'100","reported_seq":1416,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.894340+0000","last_change":"2024-09-13T14:54:58.820128+0000","last_active":"2024-09-13T14:58:46.894340+0000","last_peered":"2024-09-13T14:58:46.894340+0000","last_clean":"2024-09-13T14:58:46.894340+0000","last_became_active":"2024-09-13T14:54:58.819324+0000","last_became_peered":"2024-09-13T14:54:58.819324+0000","last_unstale":"2024-09-13T14:58:46.894340+0000","last_undegraded":"2024-09-13T14:58:46.894340+0000","last_fullsized":"2024-09-13T14:58:46.894340+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:42.577243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:42.577243+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:50:11.323977+0000","scrub_duration":0.001367839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"1276'5775","reported_seq":5802,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:13.672353+0000","last_change":"2024-09-13T14:58:06.203607+0000","last_active":"2024-09-13T14:59:13.672353+0000","last_peered":"2024-09-13T14:59:13.672353+0000","last_clean":"2024-09-13T14:59:13.672353+0000","last_became_active":"2024-09-13T14:58:06.203316+0000","last_became_peered":"2024-09-13T14:58:06.203316+0000","last_unstale":"2024-09-13T14:59:13.672353+0000","last_undegraded":"2024-09-13T14:59:13.672353+0000","last_fullsized":"2024-09-13T14:59:13.672353+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":5775,"log_dups_size":0,"ondisk_log_size":5775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:55.731243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378470400,"num_objects":5775,"num_object_clones":0,"num_object_copies":17325,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5775,"num_write_kb":369600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1966,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.834701+0000","last_change":"2024-09-13T14:54:05.607857+0000","last_active":"2024-09-13T14:59:08.834701+0000","last_peered":"2024-09-13T14:59:08.834701+0000","last_clean":"2024-09-13T14:59:08.834701+0000","last_became_active":"2024-09-13T14:54:05.607112+0000","last_became_peered":"2024-09-13T14:54:05.607112+0000","last_unstale":"2024-09-13T14:59:08.834701+0000","last_undegraded":"2024-09-13T14:59:08.834701+0000","last_fullsized":"2024-09-13T14:59:08.834701+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:41:23.733552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2002,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.834344+0000","last_change":"2024-09-13T14:54:58.832430+0000","last_active":"2024-09-13T14:59:08.834344+0000","last_peered":"2024-09-13T14:59:08.834344+0000","last_clean":"2024-09-13T14:59:08.834344+0000","last_became_active":"2024-09-13T14:54:58.832091+0000","last_became_peered":"2024-09-13T14:54:58.832091+0000","last_unstale":"2024-09-13T14:59:08.834344+0000","last_undegraded":"2024-09-13T14:59:08.834344+0000","last_fullsized":"2024-09-13T14:59:08.834344+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:50.845541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1299,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.966841+0000","last_change":"2024-09-13T14:54:05.604586+0000","last_active":"2024-09-13T14:58:46.966841+0000","last_peered":"2024-09-13T14:58:46.966841+0000","last_clean":"2024-09-13T14:58:46.966841+0000","last_became_active":"2024-09-13T14:54:05.603632+0000","last_became_peered":"2024-09-13T14:54:05.603632+0000","last_unstale":"2024-09-13T14:58:46.966841+0000","last_undegraded":"2024-09-13T14:58:46.966841+0000","last_fullsized":"2024-09-13T14:58:46.966841+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:33.197882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"261'100","reported_seq":127,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.844095+0000","last_change":"2024-09-13T14:54:58.828206+0000","last_active":"2024-09-13T14:59:08.844095+0000","last_peered":"2024-09-13T14:59:08.844095+0000","last_clean":"2024-09-13T14:59:08.844095+0000","last_became_active":"2024-09-13T14:54:58.827560+0000","last_became_peered":"2024-09-13T14:54:58.827560+0000","last_unstale":"2024-09-13T14:59:08.844095+0000","last_undegraded":"2024-09-13T14:59:08.844095+0000","last_fullsized":"2024-09-13T14:59:08.844095+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.008264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.008264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:23.090746+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"1276'5779","reported_seq":5806,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:11.889353+0000","last_change":"2024-09-13T14:58:06.200816+0000","last_active":"2024-09-13T14:59:11.889353+0000","last_peered":"2024-09-13T14:59:11.889353+0000","last_clean":"2024-09-13T14:59:11.889353+0000","last_became_active":"2024-09-13T14:58:06.200474+0000","last_became_peered":"2024-09-13T14:58:06.200474+0000","last_unstale":"2024-09-13T14:59:11.889353+0000","last_undegraded":"2024-09-13T14:59:11.889353+0000","last_fullsized":"2024-09-13T14:59:11.889353+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":5779,"log_dups_size":0,"ondisk_log_size":5779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:55.523165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378732544,"num_objects":5779,"num_object_clones":0,"num_object_copies":17337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5779,"num_write_kb":369856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":57,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.987733+0000","last_change":"2024-09-13T14:54:58.828591+0000","last_active":"2024-09-13T14:58:46.987733+0000","last_peered":"2024-09-13T14:58:46.987733+0000","last_clean":"2024-09-13T14:58:46.987733+0000","last_became_active":"2024-09-13T14:54:58.828212+0000","last_became_peered":"2024-09-13T14:54:58.828212+0000","last_unstale":"2024-09-13T14:58:46.987733+0000","last_undegraded":"2024-09-13T14:58:46.987733+0000","last_fullsized":"2024-09-13T14:58:46.987733+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:50:15.315121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":307,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.839596+0000","last_change":"2024-09-13T14:55:41.835887+0000","last_active":"2024-09-13T14:59:08.839596+0000","last_peered":"2024-09-13T14:59:08.839596+0000","last_clean":"2024-09-13T14:59:08.839596+0000","last_became_active":"2024-09-13T14:55:41.834413+0000","last_became_peered":"2024-09-13T14:55:41.834413+0000","last_unstale":"2024-09-13T14:59:08.839596+0000","last_undegraded":"2024-09-13T14:59:08.839596+0000","last_fullsized":"2024-09-13T14:59:08.839596+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:01.850578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":84,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.988029+0000","last_change":"2024-09-13T14:55:41.839256+0000","last_active":"2024-09-13T14:58:46.988029+0000","last_peered":"2024-09-13T14:58:46.988029+0000","last_clean":"2024-09-13T14:58:46.988029+0000","last_became_active":"2024-09-13T14:55:41.838918+0000","last_became_peered":"2024-09-13T14:55:41.838918+0000","last_unstale":"2024-09-13T14:58:46.988029+0000","last_undegraded":"2024-09-13T14:58:46.988029+0000","last_fullsized":"2024-09-13T14:58:46.988029+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:31:58.442849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"261'100","reported_seq":1387,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.966778+0000","last_change":"2024-09-13T14:44:28.186029+0000","last_active":"2024-09-13T14:58:46.966778+0000","last_peered":"2024-09-13T14:58:46.966778+0000","last_clean":"2024-09-13T14:58:46.966778+0000","last_became_active":"2024-09-13T14:42:14.927771+0000","last_became_peered":"2024-09-13T14:42:14.927771+0000","last_unstale":"2024-09-13T14:58:46.966778+0000","last_undegraded":"2024-09-13T14:58:46.966778+0000","last_fullsized":"2024-09-13T14:58:46.966778+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1117,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:28.185949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:28.185949+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:38.363961+0000","scrub_duration":0.0018071129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"1276'5737","reported_seq":5753,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:12.312377+0000","last_change":"2024-09-13T14:58:06.204014+0000","last_active":"2024-09-13T14:59:12.312377+0000","last_peered":"2024-09-13T14:59:12.312377+0000","last_clean":"2024-09-13T14:59:12.312377+0000","last_became_active":"2024-09-13T14:58:06.203193+0000","last_became_peered":"2024-09-13T14:58:06.203193+0000","last_unstale":"2024-09-13T14:59:12.312377+0000","last_undegraded":"2024-09-13T14:59:12.312377+0000","last_fullsized":"2024-09-13T14:59:12.312377+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":5737,"log_dups_size":0,"ondisk_log_size":5737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:05:05.742511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375980032,"num_objects":5737,"num_object_clones":0,"num_object_copies":17211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5737,"num_write_kb":367168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":127,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.844369+0000","last_change":"2024-09-13T14:54:58.829485+0000","last_active":"2024-09-13T14:59:08.844369+0000","last_peered":"2024-09-13T14:59:08.844369+0000","last_clean":"2024-09-13T14:59:08.844369+0000","last_became_active":"2024-09-13T14:54:58.829141+0000","last_became_peered":"2024-09-13T14:54:58.829141+0000","last_unstale":"2024-09-13T14:59:08.844369+0000","last_undegraded":"2024-09-13T14:59:08.844369+0000","last_fullsized":"2024-09-13T14:59:08.844369+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:10:05.474311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":293,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.841061+0000","last_change":"2024-09-13T14:54:05.609380+0000","last_active":"2024-09-13T14:59:08.841061+0000","last_peered":"2024-09-13T14:59:08.841061+0000","last_clean":"2024-09-13T14:59:08.841061+0000","last_became_active":"2024-09-13T14:54:05.606282+0000","last_became_peered":"2024-09-13T14:54:05.606282+0000","last_unstale":"2024-09-13T14:59:08.841061+0000","last_undegraded":"2024-09-13T14:59:08.841061+0000","last_fullsized":"2024-09-13T14:59:08.841061+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:24:34.588847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"261'100","reported_seq":49,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.986841+0000","last_change":"2024-09-13T14:55:41.784086+0000","last_active":"2024-09-13T14:58:46.986841+0000","last_peered":"2024-09-13T14:58:46.986841+0000","last_clean":"2024-09-13T14:58:46.986841+0000","last_became_active":"2024-09-13T14:55:41.783492+0000","last_became_peered":"2024-09-13T14:55:41.783492+0000","last_unstale":"2024-09-13T14:58:46.986841+0000","last_undegraded":"2024-09-13T14:58:46.986841+0000","last_fullsized":"2024-09-13T14:58:46.986841+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:49.280478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:49.280478+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:16:55.041094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"1276'5724","reported_seq":5740,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:12.309130+0000","last_change":"2024-09-13T14:58:06.200769+0000","last_active":"2024-09-13T14:59:12.309130+0000","last_peered":"2024-09-13T14:59:12.309130+0000","last_clean":"2024-09-13T14:59:12.309130+0000","last_became_active":"2024-09-13T14:58:06.200457+0000","last_became_peered":"2024-09-13T14:58:06.200457+0000","last_unstale":"2024-09-13T14:59:12.309130+0000","last_undegraded":"2024-09-13T14:59:12.309130+0000","last_fullsized":"2024-09-13T14:59:12.309130+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":5724,"log_dups_size":0,"ondisk_log_size":5724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:25:18.549925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375128064,"num_objects":5724,"num_object_clones":0,"num_object_copies":17172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5724,"num_write_kb":366336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":57,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.990151+0000","last_change":"2024-09-13T14:54:58.829851+0000","last_active":"2024-09-13T14:58:46.990151+0000","last_peered":"2024-09-13T14:58:46.990151+0000","last_clean":"2024-09-13T14:58:46.990151+0000","last_became_active":"2024-09-13T14:54:58.829565+0000","last_became_peered":"2024-09-13T14:54:58.829565+0000","last_unstale":"2024-09-13T14:58:46.990151+0000","last_undegraded":"2024-09-13T14:58:46.990151+0000","last_fullsized":"2024-09-13T14:58:46.990151+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:00:19.211398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":57,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.966649+0000","last_change":"2024-09-13T14:54:58.827628+0000","last_active":"2024-09-13T14:58:46.966649+0000","last_peered":"2024-09-13T14:58:46.966649+0000","last_clean":"2024-09-13T14:58:46.966649+0000","last_became_active":"2024-09-13T14:54:58.827371+0000","last_became_peered":"2024-09-13T14:54:58.827371+0000","last_unstale":"2024-09-13T14:58:46.966649+0000","last_undegraded":"2024-09-13T14:58:46.966649+0000","last_fullsized":"2024-09-13T14:58:46.966649+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:21:17.887314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"261'100","reported_seq":134,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.844180+0000","last_change":"2024-09-13T14:55:41.949447+0000","last_active":"2024-09-13T14:59:08.844180+0000","last_peered":"2024-09-13T14:59:08.844180+0000","last_clean":"2024-09-13T14:59:08.844180+0000","last_became_active":"2024-09-13T14:55:41.949062+0000","last_became_peered":"2024-09-13T14:55:41.949062+0000","last_unstale":"2024-09-13T14:59:08.844180+0000","last_undegraded":"2024-09-13T14:59:08.844180+0000","last_fullsized":"2024-09-13T14:59:08.844180+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:47.405845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:47.405845+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:41:06.083975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3","version":"1276'5764","reported_seq":5791,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:13.019856+0000","last_change":"2024-09-13T14:58:06.200017+0000","last_active":"2024-09-13T14:59:13.019856+0000","last_peered":"2024-09-13T14:59:13.019856+0000","last_clean":"2024-09-13T14:59:13.019856+0000","last_became_active":"2024-09-13T14:58:06.199665+0000","last_became_peered":"2024-09-13T14:58:06.199665+0000","last_unstale":"2024-09-13T14:59:13.019856+0000","last_undegraded":"2024-09-13T14:59:13.019856+0000","last_fullsized":"2024-09-13T14:59:13.019856+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":5764,"log_dups_size":0,"ondisk_log_size":5764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:09.742112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":377749504,"num_objects":5764,"num_object_clones":0,"num_object_copies":17292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5764,"num_write_kb":368896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":57,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.988632+0000","last_change":"2024-09-13T14:54:58.832125+0000","last_active":"2024-09-13T14:58:46.988632+0000","last_peered":"2024-09-13T14:58:46.988632+0000","last_clean":"2024-09-13T14:58:46.988632+0000","last_became_active":"2024-09-13T14:54:58.831788+0000","last_became_peered":"2024-09-13T14:54:58.831788+0000","last_unstale":"2024-09-13T14:58:46.988632+0000","last_undegraded":"2024-09-13T14:58:46.988632+0000","last_fullsized":"2024-09-13T14:58:46.988632+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:40.774847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":57,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.969374+0000","last_change":"2024-09-13T14:54:58.831785+0000","last_active":"2024-09-13T14:58:46.969374+0000","last_peered":"2024-09-13T14:58:46.969374+0000","last_clean":"2024-09-13T14:58:46.969374+0000","last_became_active":"2024-09-13T14:54:58.831331+0000","last_became_peered":"2024-09-13T14:54:58.831331+0000","last_unstale":"2024-09-13T14:58:46.969374+0000","last_undegraded":"2024-09-13T14:58:46.969374+0000","last_fullsized":"2024-09-13T14:58:46.969374+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:45.394867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"261'100","reported_seq":49,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.910320+0000","last_change":"2024-09-13T14:55:41.844203+0000","last_active":"2024-09-13T14:58:46.910320+0000","last_peered":"2024-09-13T14:58:46.910320+0000","last_clean":"2024-09-13T14:58:46.910320+0000","last_became_active":"2024-09-13T14:55:41.843897+0000","last_became_peered":"2024-09-13T14:55:41.843897+0000","last_unstale":"2024-09-13T14:58:46.910320+0000","last_undegraded":"2024-09-13T14:58:46.910320+0000","last_fullsized":"2024-09-13T14:58:46.910320+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:55.816857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:55.816857+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:42:54.571599+0000","scrub_duration":270,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2","version":"1276'5761","reported_seq":5788,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:11.873564+0000","last_change":"2024-09-13T14:58:06.201242+0000","last_active":"2024-09-13T14:59:11.873564+0000","last_peered":"2024-09-13T14:59:11.873564+0000","last_clean":"2024-09-13T14:59:11.873564+0000","last_became_active":"2024-09-13T14:58:06.200965+0000","last_became_peered":"2024-09-13T14:58:06.200965+0000","last_unstale":"2024-09-13T14:59:11.873564+0000","last_undegraded":"2024-09-13T14:59:11.873564+0000","last_fullsized":"2024-09-13T14:59:11.873564+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":5761,"log_dups_size":0,"ondisk_log_size":5761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:13:13.810742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":377552896,"num_objects":5761,"num_object_clones":0,"num_object_copies":17283,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5761,"num_write_kb":368704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":57,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.970130+0000","last_change":"2024-09-13T14:54:58.827402+0000","last_active":"2024-09-13T14:58:46.970130+0000","last_peered":"2024-09-13T14:58:46.970130+0000","last_clean":"2024-09-13T14:58:46.970130+0000","last_became_active":"2024-09-13T14:54:58.827119+0000","last_became_peered":"2024-09-13T14:54:58.827119+0000","last_unstale":"2024-09-13T14:58:46.970130+0000","last_undegraded":"2024-09-13T14:58:46.970130+0000","last_fullsized":"2024-09-13T14:58:46.970130+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:06.078013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":320,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.842768+0000","last_change":"2024-09-13T14:54:58.843575+0000","last_active":"2024-09-13T14:59:08.842768+0000","last_peered":"2024-09-13T14:59:08.842768+0000","last_clean":"2024-09-13T14:59:08.842768+0000","last_became_active":"2024-09-13T14:54:58.822033+0000","last_became_peered":"2024-09-13T14:54:58.822033+0000","last_unstale":"2024-09-13T14:59:08.842768+0000","last_undegraded":"2024-09-13T14:59:08.842768+0000","last_fullsized":"2024-09-13T14:59:08.842768+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:44.356090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"261'100","reported_seq":1406,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.966903+0000","last_change":"2024-09-13T14:45:19.798553+0000","last_active":"2024-09-13T14:58:46.966903+0000","last_peered":"2024-09-13T14:58:46.966903+0000","last_clean":"2024-09-13T14:58:46.966903+0000","last_became_active":"2024-09-13T14:45:19.798189+0000","last_became_peered":"2024-09-13T14:45:19.798189+0000","last_unstale":"2024-09-13T14:58:46.966903+0000","last_undegraded":"2024-09-13T14:58:46.966903+0000","last_fullsized":"2024-09-13T14:58:46.966903+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:29.169325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:29.169325+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:10.412666+0000","scrub_duration":0.0019358890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"1276'5777","reported_seq":5804,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:13.026435+0000","last_change":"2024-09-13T14:58:06.201392+0000","last_active":"2024-09-13T14:59:13.026435+0000","last_peered":"2024-09-13T14:59:13.026435+0000","last_clean":"2024-09-13T14:59:13.026435+0000","last_became_active":"2024-09-13T14:58:06.201088+0000","last_became_peered":"2024-09-13T14:58:06.201088+0000","last_unstale":"2024-09-13T14:59:13.026435+0000","last_undegraded":"2024-09-13T14:59:13.026435+0000","last_fullsized":"2024-09-13T14:59:13.026435+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":5777,"log_dups_size":0,"ondisk_log_size":5777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:44:35.707112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378601472,"num_objects":5777,"num_object_clones":0,"num_object_copies":17331,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5777,"num_write_kb":369728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1975,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.833815+0000","last_change":"2024-09-13T14:54:58.834208+0000","last_active":"2024-09-13T14:59:08.833815+0000","last_peered":"2024-09-13T14:59:08.833815+0000","last_clean":"2024-09-13T14:59:08.833815+0000","last_became_active":"2024-09-13T14:54:58.833071+0000","last_became_peered":"2024-09-13T14:54:58.833071+0000","last_unstale":"2024-09-13T14:59:08.833815+0000","last_undegraded":"2024-09-13T14:59:08.833815+0000","last_fullsized":"2024-09-13T14:59:08.833815+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:25:27.099352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":49,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.986924+0000","last_change":"2024-09-13T14:55:41.848811+0000","last_active":"2024-09-13T14:58:46.986924+0000","last_peered":"2024-09-13T14:58:46.986924+0000","last_clean":"2024-09-13T14:58:46.986924+0000","last_became_active":"2024-09-13T14:55:41.848542+0000","last_became_peered":"2024-09-13T14:55:41.848542+0000","last_unstale":"2024-09-13T14:58:46.986924+0000","last_undegraded":"2024-09-13T14:58:46.986924+0000","last_fullsized":"2024-09-13T14:58:46.986924+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:48.534022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"261'100","reported_seq":67,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.990064+0000","last_change":"2024-09-13T14:57:59.015556+0000","last_active":"2024-09-13T14:58:46.990064+0000","last_peered":"2024-09-13T14:58:46.990064+0000","last_clean":"2024-09-13T14:58:46.990064+0000","last_became_active":"2024-09-13T14:54:58.826820+0000","last_became_peered":"2024-09-13T14:54:58.826820+0000","last_unstale":"2024-09-13T14:58:46.990064+0000","last_undegraded":"2024-09-13T14:58:46.990064+0000","last_fullsized":"2024-09-13T14:58:46.990064+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:57:59.015484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:57:59.015484+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:49.345864+0000","scrub_duration":0.0015442190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"1276'5842","reported_seq":5869,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:13.616371+0000","last_change":"2024-09-13T14:58:06.202476+0000","last_active":"2024-09-13T14:59:13.616371+0000","last_peered":"2024-09-13T14:59:13.616371+0000","last_clean":"2024-09-13T14:59:13.616371+0000","last_became_active":"2024-09-13T14:58:06.202211+0000","last_became_peered":"2024-09-13T14:58:06.202211+0000","last_unstale":"2024-09-13T14:59:13.616371+0000","last_undegraded":"2024-09-13T14:59:13.616371+0000","last_fullsized":"2024-09-13T14:59:13.616371+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":5842,"log_dups_size":0,"ondisk_log_size":5842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:06:50.307820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382861312,"num_objects":5842,"num_object_clones":0,"num_object_copies":17526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5842,"num_write_kb":373888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1272,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.967701+0000","last_change":"2024-09-13T14:58:05.193719+0000","last_active":"2024-09-13T14:58:46.967701+0000","last_peered":"2024-09-13T14:58:46.967701+0000","last_clean":"2024-09-13T14:58:46.967701+0000","last_became_active":"2024-09-13T14:58:05.193367+0000","last_became_peered":"2024-09-13T14:58:05.193367+0000","last_unstale":"2024-09-13T14:58:46.967701+0000","last_undegraded":"2024-09-13T14:58:46.967701+0000","last_fullsized":"2024-09-13T14:58:46.967701+0000","mapping_epoch":1267,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:38:46.329171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":1270,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.966679+0000","last_change":"2024-09-13T14:55:41.919407+0000","last_active":"2024-09-13T14:58:46.966679+0000","last_peered":"2024-09-13T14:58:46.966679+0000","last_clean":"2024-09-13T14:58:46.966679+0000","last_became_active":"2024-09-13T14:55:41.793557+0000","last_became_peered":"2024-09-13T14:55:41.793557+0000","last_unstale":"2024-09-13T14:58:46.966679+0000","last_undegraded":"2024-09-13T14:58:46.966679+0000","last_fullsized":"2024-09-13T14:58:46.966679+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:57:51.874930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"261'100","reported_seq":49,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.987507+0000","last_change":"2024-09-13T14:55:41.785240+0000","last_active":"2024-09-13T14:58:46.987507+0000","last_peered":"2024-09-13T14:58:46.987507+0000","last_clean":"2024-09-13T14:58:46.987507+0000","last_became_active":"2024-09-13T14:55:41.784916+0000","last_became_peered":"2024-09-13T14:55:41.784916+0000","last_unstale":"2024-09-13T14:58:46.987507+0000","last_undegraded":"2024-09-13T14:58:46.987507+0000","last_fullsized":"2024-09-13T14:58:46.987507+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:49:21.722939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.722939+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:57.374891+0000","scrub_duration":0.0016642790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"1276'5539","reported_seq":5566,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:10.982884+0000","last_change":"2024-09-13T14:58:06.199838+0000","last_active":"2024-09-13T14:59:10.982884+0000","last_peered":"2024-09-13T14:59:10.982884+0000","last_clean":"2024-09-13T14:59:10.982884+0000","last_became_active":"2024-09-13T14:58:06.199417+0000","last_became_peered":"2024-09-13T14:58:06.199417+0000","last_unstale":"2024-09-13T14:59:10.982884+0000","last_undegraded":"2024-09-13T14:59:10.982884+0000","last_fullsized":"2024-09-13T14:59:10.982884+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":5539,"log_dups_size":0,"ondisk_log_size":5539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:14:49.667560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363003904,"num_objects":5539,"num_object_clones":0,"num_object_copies":16617,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5539,"num_write_kb":354496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":67,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.966576+0000","last_change":"2024-09-13T14:58:05.197375+0000","last_active":"2024-09-13T14:58:46.966576+0000","last_peered":"2024-09-13T14:58:46.966576+0000","last_clean":"2024-09-13T14:58:46.966576+0000","last_became_active":"2024-09-13T14:58:05.197085+0000","last_became_peered":"2024-09-13T14:58:05.197085+0000","last_unstale":"2024-09-13T14:58:46.966576+0000","last_undegraded":"2024-09-13T14:58:46.966576+0000","last_fullsized":"2024-09-13T14:58:46.966576+0000","mapping_epoch":1267,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:09:33.979543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":1330,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.967225+0000","last_change":"2024-09-13T14:54:58.841801+0000","last_active":"2024-09-13T14:58:46.967225+0000","last_peered":"2024-09-13T14:58:46.967225+0000","last_clean":"2024-09-13T14:58:46.967225+0000","last_became_active":"2024-09-13T14:54:58.820336+0000","last_became_peered":"2024-09-13T14:54:58.820336+0000","last_unstale":"2024-09-13T14:58:46.967225+0000","last_undegraded":"2024-09-13T14:58:46.967225+0000","last_fullsized":"2024-09-13T14:58:46.967225+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:19:59.656874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"261'100","reported_seq":276,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.843261+0000","last_change":"2024-09-13T14:45:19.821649+0000","last_active":"2024-09-13T14:59:08.843261+0000","last_peered":"2024-09-13T14:59:08.843261+0000","last_clean":"2024-09-13T14:59:08.843261+0000","last_became_active":"2024-09-13T14:45:19.812346+0000","last_became_peered":"2024-09-13T14:45:19.812346+0000","last_unstale":"2024-09-13T14:59:08.843261+0000","last_undegraded":"2024-09-13T14:59:08.843261+0000","last_fullsized":"2024-09-13T14:59:08.843261+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.597568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.597568+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:45.218892+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"1276'5721","reported_seq":5737,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:12.293541+0000","last_change":"2024-09-13T14:58:06.203821+0000","last_active":"2024-09-13T14:59:12.293541+0000","last_peered":"2024-09-13T14:59:12.293541+0000","last_clean":"2024-09-13T14:59:12.293541+0000","last_became_active":"2024-09-13T14:58:06.203460+0000","last_became_peered":"2024-09-13T14:58:06.203460+0000","last_unstale":"2024-09-13T14:59:12.293541+0000","last_undegraded":"2024-09-13T14:59:12.293541+0000","last_fullsized":"2024-09-13T14:59:12.293541+0000","mapping_epoch":1268,"log_start":"0'0","ondisk_log_start":"0'0","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":5721,"log_dups_size":0,"ondisk_log_size":5721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:46:54.957926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374931456,"num_objects":5721,"num_object_clones":0,"num_object_copies":17163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5721,"num_write_kb":366144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":49,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.910400+0000","last_change":"2024-09-13T14:55:41.787477+0000","last_active":"2024-09-13T14:58:46.910400+0000","last_peered":"2024-09-13T14:58:46.910400+0000","last_clean":"2024-09-13T14:58:46.910400+0000","last_became_active":"2024-09-13T14:55:41.786674+0000","last_became_peered":"2024-09-13T14:55:41.786674+0000","last_unstale":"2024-09-13T14:58:46.910400+0000","last_undegraded":"2024-09-13T14:58:46.910400+0000","last_fullsized":"2024-09-13T14:58:46.910400+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:56.263938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":1301,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.967557+0000","last_change":"2024-09-13T14:54:58.833271+0000","last_active":"2024-09-13T14:58:46.967557+0000","last_peered":"2024-09-13T14:58:46.967557+0000","last_clean":"2024-09-13T14:58:46.967557+0000","last_became_active":"2024-09-13T14:54:58.819611+0000","last_became_peered":"2024-09-13T14:54:58.819611+0000","last_unstale":"2024-09-13T14:58:46.967557+0000","last_undegraded":"2024-09-13T14:58:46.967557+0000","last_fullsized":"2024-09-13T14:58:46.967557+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:52:37.530890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"261'100","reported_seq":207,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.894855+0000","last_change":"2024-09-13T14:54:58.826831+0000","last_active":"2024-09-13T14:58:46.894855+0000","last_peered":"2024-09-13T14:58:46.894855+0000","last_clean":"2024-09-13T14:58:46.894855+0000","last_became_active":"2024-09-13T14:54:58.826370+0000","last_became_peered":"2024-09-13T14:54:58.826370+0000","last_unstale":"2024-09-13T14:58:46.894855+0000","last_undegraded":"2024-09-13T14:58:46.894855+0000","last_fullsized":"2024-09-13T14:58:46.894855+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.191326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.191326+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:09:06.771779+0000","scrub_duration":0.0025317360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1301,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.968077+0000","last_change":"2024-09-13T14:55:41.925458+0000","last_active":"2024-09-13T14:58:46.968077+0000","last_peered":"2024-09-13T14:58:46.968077+0000","last_clean":"2024-09-13T14:58:46.968077+0000","last_became_active":"2024-09-13T14:55:41.859331+0000","last_became_peered":"2024-09-13T14:55:41.859331+0000","last_unstale":"2024-09-13T14:58:46.968077+0000","last_undegraded":"2024-09-13T14:58:46.968077+0000","last_fullsized":"2024-09-13T14:58:46.968077+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:19:20.962759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":285,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.845138+0000","last_change":"2024-09-13T14:54:05.604259+0000","last_active":"2024-09-13T14:59:08.845138+0000","last_peered":"2024-09-13T14:59:08.845138+0000","last_clean":"2024-09-13T14:59:08.845138+0000","last_became_active":"2024-09-13T14:54:05.603890+0000","last_became_peered":"2024-09-13T14:54:05.603890+0000","last_unstale":"2024-09-13T14:59:08.845138+0000","last_undegraded":"2024-09-13T14:59:08.845138+0000","last_fullsized":"2024-09-13T14:59:08.845138+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:14.847855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"261'100","reported_seq":49,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.986971+0000","last_change":"2024-09-13T14:55:41.889497+0000","last_active":"2024-09-13T14:58:46.986971+0000","last_peered":"2024-09-13T14:58:46.986971+0000","last_clean":"2024-09-13T14:58:46.986971+0000","last_became_active":"2024-09-13T14:55:41.888766+0000","last_became_peered":"2024-09-13T14:55:41.888766+0000","last_unstale":"2024-09-13T14:58:46.986971+0000","last_undegraded":"2024-09-13T14:58:46.986971+0000","last_fullsized":"2024-09-13T14:58:46.986971+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:55.378404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:55.378404+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:50.289550+0000","scrub_duration":0.0016008439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1233,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.966608+0000","last_change":"2024-09-13T14:54:58.825419+0000","last_active":"2024-09-13T14:58:46.966608+0000","last_peered":"2024-09-13T14:58:46.966608+0000","last_clean":"2024-09-13T14:58:46.966608+0000","last_became_active":"2024-09-13T14:54:58.820338+0000","last_became_peered":"2024-09-13T14:54:58.820338+0000","last_unstale":"2024-09-13T14:58:46.966608+0000","last_undegraded":"2024-09-13T14:58:46.966608+0000","last_fullsized":"2024-09-13T14:58:46.966608+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:09.298279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1939,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.834868+0000","last_change":"2024-09-13T14:45:19.813876+0000","last_active":"2024-09-13T14:59:08.834868+0000","last_peered":"2024-09-13T14:59:08.834868+0000","last_clean":"2024-09-13T14:59:08.834868+0000","last_became_active":"2024-09-13T14:45:19.809136+0000","last_became_peered":"2024-09-13T14:45:19.809136+0000","last_unstale":"2024-09-13T14:59:08.834868+0000","last_undegraded":"2024-09-13T14:59:08.834868+0000","last_fullsized":"2024-09-13T14:59:08.834868+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:50.013528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"261'100","reported_seq":127,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.843967+0000","last_change":"2024-09-13T14:54:58.828578+0000","last_active":"2024-09-13T14:59:08.843967+0000","last_peered":"2024-09-13T14:59:08.843967+0000","last_clean":"2024-09-13T14:59:08.843967+0000","last_became_active":"2024-09-13T14:54:58.828004+0000","last_became_peered":"2024-09-13T14:54:58.828004+0000","last_unstale":"2024-09-13T14:59:08.843967+0000","last_undegraded":"2024-09-13T14:59:08.843967+0000","last_fullsized":"2024-09-13T14:59:08.843967+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:19.978464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:19.978464+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:52:31.236171+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1987,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.834438+0000","last_change":"2024-09-13T14:45:19.809166+0000","last_active":"2024-09-13T14:59:08.834438+0000","last_peered":"2024-09-13T14:59:08.834438+0000","last_clean":"2024-09-13T14:59:08.834438+0000","last_became_active":"2024-09-13T14:45:19.808631+0000","last_became_peered":"2024-09-13T14:45:19.808631+0000","last_unstale":"2024-09-13T14:59:08.834438+0000","last_undegraded":"2024-09-13T14:59:08.834438+0000","last_fullsized":"2024-09-13T14:59:08.834438+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:38.822245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":49,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.986826+0000","last_change":"2024-09-13T14:55:41.835069+0000","last_active":"2024-09-13T14:58:46.986826+0000","last_peered":"2024-09-13T14:58:46.986826+0000","last_clean":"2024-09-13T14:58:46.986826+0000","last_became_active":"2024-09-13T14:55:41.834766+0000","last_became_peered":"2024-09-13T14:55:41.834766+0000","last_unstale":"2024-09-13T14:58:46.986826+0000","last_undegraded":"2024-09-13T14:58:46.986826+0000","last_fullsized":"2024-09-13T14:58:46.986826+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:02:16.216617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"261'100","reported_seq":49,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.910387+0000","last_change":"2024-09-13T14:55:41.787716+0000","last_active":"2024-09-13T14:58:46.910387+0000","last_peered":"2024-09-13T14:58:46.910387+0000","last_clean":"2024-09-13T14:58:46.910387+0000","last_became_active":"2024-09-13T14:55:41.787407+0000","last_became_peered":"2024-09-13T14:55:41.787407+0000","last_unstale":"2024-09-13T14:58:46.910387+0000","last_undegraded":"2024-09-13T14:58:46.910387+0000","last_fullsized":"2024-09-13T14:58:46.910387+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:50.327393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:50.327393+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:43:30.234863+0000","scrub_duration":0.0011508309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":1264,"reported_epoch":1275,"state":"active+clean","last_fresh":"2024-09-13T14:58:46.894792+0000","last_change":"2024-09-13T14:54:58.845372+0000","last_active":"2024-09-13T14:58:46.894792+0000","last_peered":"2024-09-13T14:58:46.894792+0000","last_clean":"2024-09-13T14:58:46.894792+0000","last_became_active":"2024-09-13T14:54:58.822960+0000","last_became_peered":"2024-09-13T14:54:58.822960+0000","last_unstale":"2024-09-13T14:58:46.894792+0000","last_undegraded":"2024-09-13T14:58:46.894792+0000","last_fullsized":"2024-09-13T14:58:46.894792+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:19:30.257102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1892,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.843394+0000","last_change":"2024-09-13T14:54:05.605595+0000","last_active":"2024-09-13T14:59:08.843394+0000","last_peered":"2024-09-13T14:59:08.843394+0000","last_clean":"2024-09-13T14:59:08.843394+0000","last_became_active":"2024-09-13T14:54:05.604916+0000","last_became_peered":"2024-09-13T14:54:05.604916+0000","last_unstale":"2024-09-13T14:59:08.843394+0000","last_undegraded":"2024-09-13T14:59:08.843394+0000","last_fullsized":"2024-09-13T14:59:08.843394+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:06:22.358902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"261'100","reported_seq":682,"reported_epoch":1276,"state":"active+clean","last_fresh":"2024-09-13T14:59:08.834430+0000","last_change":"2024-09-13T14:55:41.878403+0000","last_active":"2024-09-13T14:59:08.834430+0000","last_peered":"2024-09-13T14:59:08.834430+0000","last_clean":"2024-09-13T14:59:08.834430+0000","last_became_active":"2024-09-13T14:55:41.878071+0000","last_became_peered":"2024-09-13T14:55:41.878071+0000","last_unstale":"2024-09-13T14:59:08.834430+0000","last_undegraded":"2024-09-13T14:59:08.834430+0000","last_fullsized":"2024-09-13T14:59:08.834430+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:14.654603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:14.654603+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:09:56.313293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":5994905600,"num_objects":91475,"num_object_clones":0,"num_object_copies":274425,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91475,"num_write_kb":5854400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18000445440,"data_stored":18000445440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91475,"ondisk_log_size":91475,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":7662,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":853056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430710,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2341444,"kb_used_data":1454844,"kb_used_omap":1,"kb_used_meta":886590,"kb_avail":91403708,"statfs":{"total":95995035648,"available":93597396992,"internally_reserved":0,"allocated":1489760256,"data_stored":1487100034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":907868504},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626939,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3459836,"kb_used_data":2557108,"kb_used_omap":1,"kb_used_meta":902718,"kb_avail":90285316,"statfs":{"total":95995035648,"available":92452163584,"internally_reserved":0,"allocated":2618478592,"data_stored":2615825981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":924383576},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823168,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2714616,"kb_used_data":1819248,"kb_used_omap":1,"kb_used_meta":895358,"kb_avail":91030536,"statfs":{"total":95995035648,"available":93215268864,"internally_reserved":0,"allocated":1862909952,"data_stored":1860257807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":916846936},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019396,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3805888,"kb_used_data":2918520,"kb_used_omap":1,"kb_used_meta":887358,"kb_avail":89939264,"statfs":{"total":95995035648,"available":92097806336,"internally_reserved":0,"allocated":2988564480,"data_stored":2985907830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":908654936},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059593053,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2476460,"kb_used_data":1782756,"kb_used_omap":26,"kb_used_meta":693669,"kb_avail":91268692,"statfs":{"total":95995035648,"available":93459140608,"internally_reserved":0,"allocated":1825542144,"data_stored":1822901186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":710317216},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":591,"seq":2538325672292,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2953912,"kb_used_data":2191792,"kb_used_omap":41,"kb_used_meta":762070,"kb_avail":90791240,"statfs":{"total":95995035648,"available":92970229760,"internally_reserved":0,"allocated":2244395008,"data_stored":2241742910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42651,"internal_metadata":780360037},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":557,"seq":2392296784235,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1893164,"kb_used_data":1110052,"kb_used_omap":32,"kb_used_meta":783071,"kb_avail":91851988,"statfs":{"total":95995035648,"available":94056435712,"internally_reserved":0,"allocated":1136693248,"data_stored":1134052219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33752,"internal_metadata":801864744},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":524,"seq":2250562863474,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4430896,"kb_used_data":3694312,"kb_used_omap":41,"kb_used_meta":736534,"kb_avail":89314256,"statfs":{"total":95995035648,"available":91457798144,"internally_reserved":0,"allocated":3782975488,"data_stored":3780330936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42694,"internal_metadata":754211130},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3816947712,"data_stored":3816947712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1129775104,"data_stored":1129775104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2254831616,"data_stored":2254831616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1835139072,"data_stored":1835139072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2997747712,"data_stored":2997747712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1861877760,"data_stored":1861877760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2613510144,"data_stored":2613510144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1490616320,"data_stored":1490616320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:59:16.972 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pgp_num 61 2024-09-13T14:59:17.058 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:17.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:17.084 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:17.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:16 smithi184 ceph-mon[92527]: pgmap v2077: 119 pgs: 119 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-13T14:59:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:59:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:17.186 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:17.197 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:2024-09-13T14:59:17.197108+0000 min lat: 0.00119391 max lat: 0.812283 avg lat: 0.00992993 2024-09-13T14:59:17.197 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:59:17.197 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 60 16 96623 96607 100.613 61.625 0.0142303 0.00992993 2024-09-13T14:59:17.216 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:17.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:17.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:17.217 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:17.234 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:16 smithi151 ceph-mon[108966]: pgmap v2077: 119 pgs: 119 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-13T14:59:17.235 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:59:17.235 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:17.235 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:16 smithi151 ceph-mon[104450]: pgmap v2077: 119 pgs: 119 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-13T14:59:17.235 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:59:17.236 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:17.317 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:17.356 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:17.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:17.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:17.458 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:17.496 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:17.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:17.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:17.598 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:17.632 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:17.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:17.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:17.734 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:17.771 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:17.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:17.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:17.794 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:59:17.795 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [], 2024-09-13T14:59:17.795 INFO:teuthology.orchestra.run.smithi184.stdout: "complaint_time": 30, 2024-09-13T14:59:17.795 INFO:teuthology.orchestra.run.smithi184.stdout: "num_blocked_ops": 0 2024-09-13T14:59:17.795 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:59:17.873 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:17.917 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:17.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:17.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:18.019 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:18.063 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:18.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:18.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:18.166 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:18.197 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 61 16 97666 97650 100.033 65.1875 0.0105445 0.00999098 2024-09-13T14:59:18.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:18.214 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:18.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:18.315 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:18.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:18.352 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:18.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:18.453 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:18.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:18.488 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:18.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:18.494 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T14:59:18.589 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:18.629 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:18.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:18.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:18.730 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:18.774 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:18.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:18.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:18.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:18.897 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:59:18.903 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-13T14:59:18.912 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:18.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:18.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:19.013 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:19.052 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:19.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:19.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:19.154 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:19.173 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:18 smithi184 ceph-mon[92527]: pgmap v2078: 119 pgs: 119 active+clean; 5.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-13T14:59:19.173 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:18 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3586569644' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:59:19.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:19.194 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:19.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:19.197 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 62 16 98645 98629 99.406 61.1875 0.0106858 0.0100546 2024-09-13T14:59:19.295 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:19.334 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:19.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:19.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:19.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:18 smithi151 ceph-mon[104450]: pgmap v2078: 119 pgs: 119 active+clean; 5.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-13T14:59:19.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:18 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3586569644' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:59:19.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:18 smithi151 ceph-mon[108966]: pgmap v2078: 119 pgs: 119 active+clean; 5.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-13T14:59:19.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:18 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3586569644' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:59:19.436 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:19.472 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:19.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:19.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:19.574 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:19.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:19.601 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:19.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:19.702 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:19.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:19.724 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:19.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:19.825 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:19.865 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:19.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:19.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:19.967 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:19.989 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:19.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:19.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:20.090 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:20.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:19 smithi151 ceph-mon[104450]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:20.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:19 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3586569644' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:59:20.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:19 smithi151 ceph-mon[104450]: osdmap e1277: 8 total, 8 up, 8 in 2024-09-13T14:59:20.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:19 smithi151 ceph-mon[108966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:20.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:19 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3586569644' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:59:20.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:19 smithi151 ceph-mon[108966]: osdmap e1277: 8 total, 8 up, 8 in 2024-09-13T14:59:20.114 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:20.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:20.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:20.197 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 63 16 99311 99295 98.4888 41.625 0.00508019 0.0101481 2024-09-13T14:59:20.214 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:19 smithi184 ceph-mon[92527]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:20.214 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:19 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3586569644' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:59:20.214 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:19 smithi184 ceph-mon[92527]: osdmap e1277: 8 total, 8 up, 8 in 2024-09-13T14:59:20.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:20.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:20.238 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:20.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:20.339 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:20.378 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:20.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:20.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:20.480 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:20.514 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:20.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:20.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:20.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:59:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:59:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:59:20.617 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:20.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:20.655 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:20.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:20.756 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:20.791 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:20.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:20.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:20.893 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:20.931 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:20.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:20.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:21.033 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:21.065 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:21.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:21.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:21.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:20 smithi184 ceph-mon[92527]: pgmap v2080: 119 pgs: 119 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-13T14:59:21.166 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:21.167 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:20 smithi151 ceph-mon[108966]: pgmap v2080: 119 pgs: 119 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-13T14:59:21.168 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:20 smithi151 ceph-mon[104450]: pgmap v2080: 119 pgs: 119 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-13T14:59:21.197 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 64 16 100256 100240 97.8728 59.0625 0.0134053 0.0102123 2024-09-13T14:59:21.203 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:21.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:21.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:21.305 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:21.344 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:21.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:21.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:21.446 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:21.480 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:21.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:21.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:21.581 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:21.616 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:21.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:21.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:21.719 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:21.761 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:21.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:21.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:21.863 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:21.902 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:21.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:21.903 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:21.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:22.004 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:22.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:22.047 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:22.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:22.148 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:22.183 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:22.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:22.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:22.198 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 65 16 101348 101332 97.417 68.25 0.00956539 0.0102607 2024-09-13T14:59:22.285 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:22.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:22.322 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:22.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:22.423 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:22.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:22.464 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:22.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:22.565 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:22.607 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:22.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:22.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:22.708 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:22.745 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:22.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:22.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:22.847 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:22.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:22.885 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:22.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:22.986 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:23.022 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:23.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:23.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:23.038 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:22 smithi151 ceph-mon[104450]: pgmap v2081: 119 pgs: 119 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-13T14:59:23.038 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:22 smithi151 ceph-mon[108966]: pgmap v2081: 119 pgs: 119 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-13T14:59:23.060 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:59:23.061 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:59:23.088 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:59:23.088 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:59:23.112 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:59:23.112 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:59:23.124 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:23.152 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:59:23.153 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:59:23.161 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:23.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:23.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:23.175 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:59:23.175 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:59:23.191 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:59:23.191 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:59:23.197 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 66 16 102313 102297 96.8548 60.3125 0.00375729 0.0103123 2024-09-13T14:59:23.207 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:59:23.207 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:59:23.263 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:23.284 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:23.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:23.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:23.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:22 smithi184 ceph-mon[92527]: pgmap v2081: 119 pgs: 119 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-13T14:59:23.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:23.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:23.409 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:23.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:23.474 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:59:23.475 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:59:23.510 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:23.539 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:23.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:23.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:23.586 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:59:23.586 INFO:teuthology.orchestra.run.smithi184.stdout: "size": 20, 2024-09-13T14:59:23.586 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 600, 2024-09-13T14:59:23.586 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [ 2024-09-13T14:59:23.586 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.586 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197603 9.6 9:67850b99:::benchmark_data_smithi151_261954_object94747:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T14:59:23.586 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.242955+0000", 2024-09-13T14:59:23.586 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 111.80708617099999, 2024-09-13T14:59:23.587 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1339466739999999, 2024-09-13T14:59:23.587 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:23.587 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "reached pg", 2024-09-13T14:59:23.587 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:23.587 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T14:59:23.587 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:59:23.587 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197603 2024-09-13T14:59:23.587 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.587 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:23.587 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.587 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:23.587 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.242955+0000", 2024-09-13T14:59:23.587 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.587 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.588 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.588 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:23.588 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.242955+0000", 2024-09-13T14:59:23.588 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.588 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.588 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.588 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:23.588 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.242953+0000", 2024-09-13T14:59:23.588 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:23.588 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.588 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.588 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:23.588 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.242966+0000", 2024-09-13T14:59:23.589 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2500999999999999e-05 2024-09-13T14:59:23.589 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.589 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.589 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:23.589 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.242979+0000", 2024-09-13T14:59:23.589 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3421000000000001e-05 2024-09-13T14:59:23.589 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.589 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.589 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:23.589 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.242985+0000", 2024-09-13T14:59:23.589 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5960000000000003e-06 2024-09-13T14:59:23.589 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.589 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.590 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.590 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.245536+0000", 2024-09-13T14:59:23.590 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002551598 2024-09-13T14:59:23.590 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.590 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.590 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op must wait for map", 2024-09-13T14:59:23.590 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.245541+0000", 2024-09-13T14:59:23.590 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3510000000000004e-06 2024-09-13T14:59:23.590 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.590 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.590 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.590 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.246376+0000", 2024-09-13T14:59:23.590 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00083585899999999999 2024-09-13T14:59:23.590 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.591 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.591 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:23.591 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.246385+0000", 2024-09-13T14:59:23.591 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.8689999999999993e-06 2024-09-13T14:59:23.591 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.591 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.591 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.591 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.303041+0000", 2024-09-13T14:59:23.591 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0566552010000001 2024-09-13T14:59:23.591 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.591 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.591 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for missing object", 2024-09-13T14:59:23.591 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.303149+0000", 2024-09-13T14:59:23.592 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00010892599999999999 2024-09-13T14:59:23.592 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.592 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.592 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.592 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.376838+0000", 2024-09-13T14:59:23.592 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.073688742000000002 2024-09-13T14:59:23.592 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.592 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.592 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:23.592 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.376901+0000", 2024-09-13T14:59:23.592 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3144000000000004e-05 2024-09-13T14:59:23.592 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.592 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:23.592 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.593 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.593 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.593 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197608 9.6 9:664669c4:::benchmark_data_smithi151_261954_object94752:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T14:59:23.593 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.248591+0000", 2024-09-13T14:59:23.593 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 111.80144948, 2024-09-13T14:59:23.593 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1402739749999999, 2024-09-13T14:59:23.593 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:23.593 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:23.593 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:23.593 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T14:59:23.593 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:59:23.593 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197608 2024-09-13T14:59:23.593 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.594 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:23.594 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.594 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:23.594 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.248591+0000", 2024-09-13T14:59:23.594 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.594 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.594 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.594 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:23.594 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.248591+0000", 2024-09-13T14:59:23.594 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.594 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.594 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.594 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:23.595 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.248590+0000", 2024-09-13T14:59:23.595 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:23.595 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.595 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.595 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:23.595 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.248597+0000", 2024-09-13T14:59:23.595 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.4560000000000003e-06 2024-09-13T14:59:23.595 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.595 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.595 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:23.595 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.248610+0000", 2024-09-13T14:59:23.595 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.276e-05 2024-09-13T14:59:23.595 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.595 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.596 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:23.596 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.248615+0000", 2024-09-13T14:59:23.596 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7820000000000001e-06 2024-09-13T14:59:23.596 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.596 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.596 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.596 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.248958+0000", 2024-09-13T14:59:23.596 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00034373999999999999 2024-09-13T14:59:23.596 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.596 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.596 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:23.596 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.248985+0000", 2024-09-13T14:59:23.596 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.6902000000000001e-05 2024-09-13T14:59:23.597 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.597 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.597 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.597 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.303170+0000", 2024-09-13T14:59:23.597 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0541845750000001 2024-09-13T14:59:23.597 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.597 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.597 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:23.597 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.303270+0000", 2024-09-13T14:59:23.597 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00010008700000000001 2024-09-13T14:59:23.597 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.597 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.597 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:23.598 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.303848+0000", 2024-09-13T14:59:23.598 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00057828600000000001 2024-09-13T14:59:23.598 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.598 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.598 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:23.598 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.325700+0000", 2024-09-13T14:59:23.598 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.021852136000000001 2024-09-13T14:59:23.598 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.598 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.598 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.598 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.384239+0000", 2024-09-13T14:59:23.598 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.058538245000000003 2024-09-13T14:59:23.598 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.598 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.599 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.599 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.388686+0000", 2024-09-13T14:59:23.599 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0044472849999999996 2024-09-13T14:59:23.599 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.599 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.599 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:23.599 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.388740+0000", 2024-09-13T14:59:23.599 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3934999999999998e-05 2024-09-13T14:59:23.599 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.599 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.599 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:23.599 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.388865+0000", 2024-09-13T14:59:23.599 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012541200000000001 2024-09-13T14:59:23.599 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.600 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:23.600 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.600 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.600 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.600 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197611 9.6 9:6ed0f261:::benchmark_data_smithi151_261954_object94755:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T14:59:23.600 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.250949+0000", 2024-09-13T14:59:23.600 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 111.799091905, 2024-09-13T14:59:23.600 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1381295499999999, 2024-09-13T14:59:23.600 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:23.600 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:23.600 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:23.600 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T14:59:23.601 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:59:23.601 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197611 2024-09-13T14:59:23.601 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.601 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:23.601 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.601 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:23.601 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.250949+0000", 2024-09-13T14:59:23.601 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.601 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.601 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.601 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:23.601 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.250949+0000", 2024-09-13T14:59:23.601 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.601 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.602 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.602 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:23.602 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.250948+0000", 2024-09-13T14:59:23.602 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:23.602 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.602 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.602 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:23.602 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.250953+0000", 2024-09-13T14:59:23.602 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8359999999999997e-06 2024-09-13T14:59:23.602 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.602 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.602 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:23.603 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.250967+0000", 2024-09-13T14:59:23.603 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3585e-05 2024-09-13T14:59:23.603 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.603 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.603 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:23.603 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.250973+0000", 2024-09-13T14:59:23.603 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8000000000000004e-06 2024-09-13T14:59:23.603 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.603 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.603 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.603 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.253272+0000", 2024-09-13T14:59:23.603 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002299125 2024-09-13T14:59:23.603 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.603 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.604 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:23.604 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.253280+0000", 2024-09-13T14:59:23.604 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.8630000000000008e-06 2024-09-13T14:59:23.604 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.604 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.604 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.604 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.316907+0000", 2024-09-13T14:59:23.604 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.063627122 2024-09-13T14:59:23.604 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.604 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.604 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:23.604 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.316994+0000", 2024-09-13T14:59:23.604 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.7138000000000005e-05 2024-09-13T14:59:23.604 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.605 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.605 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:23.605 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.317078+0000", 2024-09-13T14:59:23.605 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.3681000000000001e-05 2024-09-13T14:59:23.605 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.605 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.605 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:23.605 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.325728+0000", 2024-09-13T14:59:23.605 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0086505990000000001 2024-09-13T14:59:23.605 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.605 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.605 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.605 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.384264+0000", 2024-09-13T14:59:23.606 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.058535255000000001 2024-09-13T14:59:23.606 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.606 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.606 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.606 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.388929+0000", 2024-09-13T14:59:23.606 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0046659789999999998 2024-09-13T14:59:23.606 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.606 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.606 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:23.606 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.388978+0000", 2024-09-13T14:59:23.606 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8105000000000001e-05 2024-09-13T14:59:23.606 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.606 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.606 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:23.607 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.389079+0000", 2024-09-13T14:59:23.607 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000100908 2024-09-13T14:59:23.607 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.607 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:23.607 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.607 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.607 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.607 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197615 9.6 9:6509112f:::benchmark_data_smithi151_261954_object94759:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T14:59:23.607 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.252664+0000", 2024-09-13T14:59:23.607 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 111.797376871, 2024-09-13T14:59:23.607 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1437066060000001, 2024-09-13T14:59:23.607 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:23.607 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:23.607 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:23.608 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T14:59:23.608 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:59:23.608 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197615 2024-09-13T14:59:23.608 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.608 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:23.608 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.608 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:23.608 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.252664+0000", 2024-09-13T14:59:23.608 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.608 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.608 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.608 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:23.608 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.252664+0000", 2024-09-13T14:59:23.609 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.609 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.609 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.609 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:23.609 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.252663+0000", 2024-09-13T14:59:23.609 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:59:23.609 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.609 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.609 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:23.609 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.252672+0000", 2024-09-13T14:59:23.609 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.8130000000000002e-06 2024-09-13T14:59:23.609 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.609 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.609 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:23.610 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.252686+0000", 2024-09-13T14:59:23.610 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4126e-05 2024-09-13T14:59:23.610 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.610 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.610 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:23.610 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.252692+0000", 2024-09-13T14:59:23.610 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.4620000000000001e-06 2024-09-13T14:59:23.610 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.610 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.610 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.610 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.253301+0000", 2024-09-13T14:59:23.610 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00060926600000000002 2024-09-13T14:59:23.610 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.611 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.611 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:23.611 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.253308+0000", 2024-09-13T14:59:23.611 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3550000000000001e-06 2024-09-13T14:59:23.611 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.611 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.611 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.611 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.319292+0000", 2024-09-13T14:59:23.611 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.065983992 2024-09-13T14:59:23.611 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.611 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.611 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:23.611 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.319387+0000", 2024-09-13T14:59:23.611 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.4790000000000006e-05 2024-09-13T14:59:23.612 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.612 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.612 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:23.612 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.319457+0000", 2024-09-13T14:59:23.612 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.0302999999999994e-05 2024-09-13T14:59:23.612 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.612 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.612 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:23.612 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.329056+0000", 2024-09-13T14:59:23.612 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0095986660000000005 2024-09-13T14:59:23.612 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.612 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.612 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.612 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.390210+0000", 2024-09-13T14:59:23.613 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.061154497000000002 2024-09-13T14:59:23.613 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.613 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.613 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.613 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.396285+0000", 2024-09-13T14:59:23.613 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0060744579999999996 2024-09-13T14:59:23.613 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.613 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.613 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:23.613 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.396330+0000", 2024-09-13T14:59:23.613 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5677999999999999e-05 2024-09-13T14:59:23.613 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.613 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.614 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:23.614 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.396371+0000", 2024-09-13T14:59:23.614 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0364999999999997e-05 2024-09-13T14:59:23.614 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.614 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:23.614 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.614 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.614 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.614 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197628 9.6 9:6d4e4d04:::benchmark_data_smithi151_261954_object94772:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T14:59:23.614 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.268145+0000", 2024-09-13T14:59:23.614 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 111.781896347, 2024-09-13T14:59:23.614 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.128359632, 2024-09-13T14:59:23.614 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:23.615 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:23.615 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:23.615 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T14:59:23.615 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:59:23.615 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197628 2024-09-13T14:59:23.615 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.615 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:23.615 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.615 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:23.615 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.268145+0000", 2024-09-13T14:59:23.615 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.615 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.615 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.616 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:23.616 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.268145+0000", 2024-09-13T14:59:23.616 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.616 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.616 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.616 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:23.616 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.268143+0000", 2024-09-13T14:59:23.616 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:59:23.616 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.616 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.616 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:23.616 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.268150+0000", 2024-09-13T14:59:23.616 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.2069999999999998e-06 2024-09-13T14:59:23.617 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.617 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.617 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:23.617 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.268166+0000", 2024-09-13T14:59:23.617 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.5611999999999999e-05 2024-09-13T14:59:23.617 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.617 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.617 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:23.617 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.268176+0000", 2024-09-13T14:59:23.617 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0108999999999999e-05 2024-09-13T14:59:23.617 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.617 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.617 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.618 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.274003+0000", 2024-09-13T14:59:23.618 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0058268690000000001 2024-09-13T14:59:23.618 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.618 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.618 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:23.618 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.274013+0000", 2024-09-13T14:59:23.618 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0909e-05 2024-09-13T14:59:23.618 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.618 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.618 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.618 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.321130+0000", 2024-09-13T14:59:23.618 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.047116344 2024-09-13T14:59:23.618 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.619 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.619 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:23.619 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.321198+0000", 2024-09-13T14:59:23.619 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7824000000000007e-05 2024-09-13T14:59:23.619 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.619 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.619 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:23.619 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.321265+0000", 2024-09-13T14:59:23.619 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7514000000000004e-05 2024-09-13T14:59:23.619 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.619 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.619 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:23.619 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.329063+0000", 2024-09-13T14:59:23.620 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0077980139999999998 2024-09-13T14:59:23.620 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.620 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.620 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.620 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.390255+0000", 2024-09-13T14:59:23.620 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.061192150000000001 2024-09-13T14:59:23.620 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.620 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.620 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.620 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.396426+0000", 2024-09-13T14:59:23.620 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0061704300000000002 2024-09-13T14:59:23.620 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.620 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.621 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:23.621 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.396465+0000", 2024-09-13T14:59:23.621 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9487000000000003e-05 2024-09-13T14:59:23.621 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.621 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.621 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:23.621 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.396504+0000", 2024-09-13T14:59:23.621 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8927999999999997e-05 2024-09-13T14:59:23.621 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.622 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:23.622 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.622 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.622 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.622 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197632 9.6 9:650414d3:::benchmark_data_smithi151_261954_object94776:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T14:59:23.622 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.269941+0000", 2024-09-13T14:59:23.622 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 111.780099605, 2024-09-13T14:59:23.622 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.130399621, 2024-09-13T14:59:23.622 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:23.622 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:23.622 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:23.622 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T14:59:23.622 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:59:23.623 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197632 2024-09-13T14:59:23.623 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.623 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:23.623 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.623 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:23.623 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.269941+0000", 2024-09-13T14:59:23.623 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.623 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.623 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.624 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:23.624 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.269941+0000", 2024-09-13T14:59:23.624 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.624 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.624 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.624 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:23.624 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.269940+0000", 2024-09-13T14:59:23.624 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:59:23.624 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.624 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.624 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:23.625 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.269948+0000", 2024-09-13T14:59:23.625 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.6009999999999997e-06 2024-09-13T14:59:23.625 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.625 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.625 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:23.625 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.269961+0000", 2024-09-13T14:59:23.625 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3560000000000001e-05 2024-09-13T14:59:23.625 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.625 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.625 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:23.626 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.269967+0000", 2024-09-13T14:59:23.626 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8109999999999999e-06 2024-09-13T14:59:23.626 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.626 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.626 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.626 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.274093+0000", 2024-09-13T14:59:23.626 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0041263710000000002 2024-09-13T14:59:23.626 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.626 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.626 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:23.626 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.274101+0000", 2024-09-13T14:59:23.626 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.5009999999999998e-06 2024-09-13T14:59:23.626 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.627 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.627 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.627 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.322701+0000", 2024-09-13T14:59:23.627 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0486003740000001 2024-09-13T14:59:23.627 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.627 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.627 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:23.627 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.322765+0000", 2024-09-13T14:59:23.627 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3381999999999997e-05 2024-09-13T14:59:23.627 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.627 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.627 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:23.627 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.322834+0000", 2024-09-13T14:59:23.628 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.9497000000000002e-05 2024-09-13T14:59:23.628 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.628 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.628 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:23.628 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.329070+0000", 2024-09-13T14:59:23.628 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0062355609999999997 2024-09-13T14:59:23.628 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.628 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.628 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.628 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.397613+0000", 2024-09-13T14:59:23.628 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.068543525999999994 2024-09-13T14:59:23.628 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.628 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.629 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.629 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.400251+0000", 2024-09-13T14:59:23.629 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002637805 2024-09-13T14:59:23.629 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.629 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.629 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:23.629 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.400302+0000", 2024-09-13T14:59:23.629 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0559999999999997e-05 2024-09-13T14:59:23.629 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.629 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.629 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:23.629 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.400341+0000", 2024-09-13T14:59:23.629 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9203999999999999e-05 2024-09-13T14:59:23.630 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.630 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:23.630 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.630 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.630 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.630 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197636 9.6 9:623919ba:::benchmark_data_smithi151_261954_object94780:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T14:59:23.630 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.275235+0000", 2024-09-13T14:59:23.630 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 111.774805429, 2024-09-13T14:59:23.630 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.126014793, 2024-09-13T14:59:23.630 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:23.630 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:23.630 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:23.630 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T14:59:23.631 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:59:23.631 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197636 2024-09-13T14:59:23.631 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.631 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:23.631 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.631 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:23.631 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.275235+0000", 2024-09-13T14:59:23.631 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.631 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.631 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.631 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:23.631 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.275235+0000", 2024-09-13T14:59:23.631 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.631 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.632 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.632 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:23.632 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.275234+0000", 2024-09-13T14:59:23.632 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:59:23.632 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.632 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.632 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:23.632 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.275239+0000", 2024-09-13T14:59:23.632 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1749999999999996e-06 2024-09-13T14:59:23.632 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.632 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.632 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:23.632 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.275248+0000", 2024-09-13T14:59:23.633 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.662e-06 2024-09-13T14:59:23.633 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.633 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.633 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:23.633 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.275253+0000", 2024-09-13T14:59:23.633 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9910000000000002e-06 2024-09-13T14:59:23.633 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.633 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.633 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.633 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.278310+0000", 2024-09-13T14:59:23.633 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0030561 2024-09-13T14:59:23.633 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.634 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.634 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:23.634 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.278320+0000", 2024-09-13T14:59:23.634 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.042e-05 2024-09-13T14:59:23.634 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.634 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.634 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.634 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.324240+0000", 2024-09-13T14:59:23.634 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.045920113 2024-09-13T14:59:23.634 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.634 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.634 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:23.634 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.324321+0000", 2024-09-13T14:59:23.634 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.1411999999999999e-05 2024-09-13T14:59:23.635 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.635 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.635 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:23.635 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.324396+0000", 2024-09-13T14:59:23.635 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.4723999999999998e-05 2024-09-13T14:59:23.635 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.635 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.635 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:23.635 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.333518+0000", 2024-09-13T14:59:23.635 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0091214390000000003 2024-09-13T14:59:23.635 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.635 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.636 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.636 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.400387+0000", 2024-09-13T14:59:23.636 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.066869291999999997 2024-09-13T14:59:23.636 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.636 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.636 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.636 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.401174+0000", 2024-09-13T14:59:23.636 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00078730300000000005 2024-09-13T14:59:23.636 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.636 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.636 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:23.636 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.401217+0000", 2024-09-13T14:59:23.636 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.2880999999999998e-05 2024-09-13T14:59:23.637 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.637 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.637 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:23.637 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.401250+0000", 2024-09-13T14:59:23.637 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3105999999999998e-05 2024-09-13T14:59:23.637 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.637 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:23.637 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.637 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.637 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.637 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197641 9.6 9:6b2553dd:::benchmark_data_smithi151_261954_object94785:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T14:59:23.637 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.278888+0000", 2024-09-13T14:59:23.637 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 111.771152474, 2024-09-13T14:59:23.638 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1274783509999999, 2024-09-13T14:59:23.638 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:23.638 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:23.638 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:23.638 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T14:59:23.638 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:59:23.638 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197641 2024-09-13T14:59:23.638 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.638 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:23.638 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.638 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:23.638 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.278888+0000", 2024-09-13T14:59:23.638 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.639 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.639 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.639 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:23.639 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.278888+0000", 2024-09-13T14:59:23.639 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.639 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.639 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.639 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:23.639 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.278887+0000", 2024-09-13T14:59:23.639 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:23.639 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.639 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.639 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:23.639 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.278896+0000", 2024-09-13T14:59:23.640 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.3260000000000002e-06 2024-09-13T14:59:23.640 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.640 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.640 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:23.640 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.278910+0000", 2024-09-13T14:59:23.640 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4234e-05 2024-09-13T14:59:23.640 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.640 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.640 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:23.640 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.278916+0000", 2024-09-13T14:59:23.640 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5160000000000002e-06 2024-09-13T14:59:23.640 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.640 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.641 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.641 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.278966+0000", 2024-09-13T14:59:23.641 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9969999999999998e-05 2024-09-13T14:59:23.641 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.641 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.641 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:23.641 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.278976+0000", 2024-09-13T14:59:23.641 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.7880000000000001e-06 2024-09-13T14:59:23.641 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.641 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.641 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.641 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.325763+0000", 2024-09-13T14:59:23.641 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0467872410000001 2024-09-13T14:59:23.641 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.642 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.642 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:23.642 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.325830+0000", 2024-09-13T14:59:23.642 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.6948999999999999e-05 2024-09-13T14:59:23.642 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.642 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.642 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:23.642 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.325905+0000", 2024-09-13T14:59:23.642 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.5026999999999998e-05 2024-09-13T14:59:23.642 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.642 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.642 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:23.642 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.333522+0000", 2024-09-13T14:59:23.642 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0076173600000000001 2024-09-13T14:59:23.643 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.643 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.643 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.643 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.401291+0000", 2024-09-13T14:59:23.643 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.067768703999999999 2024-09-13T14:59:23.643 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.643 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.643 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.643 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.406235+0000", 2024-09-13T14:59:23.643 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0049436469999999998 2024-09-13T14:59:23.643 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.643 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.643 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:23.644 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.406316+0000", 2024-09-13T14:59:23.644 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.1417999999999999e-05 2024-09-13T14:59:23.644 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.644 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.644 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:23.644 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.406367+0000", 2024-09-13T14:59:23.644 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0739000000000003e-05 2024-09-13T14:59:23.644 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.644 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:23.644 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.644 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.644 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.644 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197648 9.6 9:680e8575:::benchmark_data_smithi151_261954_object94792:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T14:59:23.645 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.286808+0000", 2024-09-13T14:59:23.645 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 111.763232809, 2024-09-13T14:59:23.645 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.11968004, 2024-09-13T14:59:23.645 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:23.645 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:23.645 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:23.645 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T14:59:23.645 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:59:23.645 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197648 2024-09-13T14:59:23.645 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.645 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:23.645 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.645 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:23.645 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.286808+0000", 2024-09-13T14:59:23.646 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.646 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.646 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.646 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:23.646 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.286808+0000", 2024-09-13T14:59:23.646 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.646 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.646 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.646 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:23.646 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.286806+0000", 2024-09-13T14:59:23.646 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:59:23.646 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.646 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.647 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:23.647 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.286813+0000", 2024-09-13T14:59:23.647 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.3490000000000003e-06 2024-09-13T14:59:23.647 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.647 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.647 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:23.647 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.286826+0000", 2024-09-13T14:59:23.647 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2367e-05 2024-09-13T14:59:23.647 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.647 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.647 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:23.647 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.286832+0000", 2024-09-13T14:59:23.647 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8640000000000001e-06 2024-09-13T14:59:23.647 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.648 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.648 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.648 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.287248+0000", 2024-09-13T14:59:23.648 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00041674399999999998 2024-09-13T14:59:23.648 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.648 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.648 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:23.648 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.287312+0000", 2024-09-13T14:59:23.648 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3938999999999999e-05 2024-09-13T14:59:23.648 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.648 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.648 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.648 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.327320+0000", 2024-09-13T14:59:23.649 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0400074850000001 2024-09-13T14:59:23.649 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.649 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.649 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:23.649 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.327407+0000", 2024-09-13T14:59:23.649 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.7183999999999997e-05 2024-09-13T14:59:23.649 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.649 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.649 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:23.649 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.327486+0000", 2024-09-13T14:59:23.649 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.9269000000000003e-05 2024-09-13T14:59:23.649 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.649 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.649 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:23.650 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.335020+0000", 2024-09-13T14:59:23.650 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.007533891 2024-09-13T14:59:23.650 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.650 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.650 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.650 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.405114+0000", 2024-09-13T14:59:23.650 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.070093875 2024-09-13T14:59:23.650 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.650 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.650 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.650 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.406409+0000", 2024-09-13T14:59:23.650 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0012952319999999999 2024-09-13T14:59:23.651 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.651 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.651 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:23.651 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.406454+0000", 2024-09-13T14:59:23.651 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4342999999999999e-05 2024-09-13T14:59:23.651 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.651 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.651 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:23.651 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.406488+0000", 2024-09-13T14:59:23.651 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.4419999999999999e-05 2024-09-13T14:59:23.651 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.651 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:23.651 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.651 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.652 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.652 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197655 9.6 9:6ad42e07:::benchmark_data_smithi151_261954_object94799:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T14:59:23.652 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.294574+0000", 2024-09-13T14:59:23.652 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 111.75546734700001, 2024-09-13T14:59:23.652 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1215945810000001, 2024-09-13T14:59:23.652 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:23.652 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:23.652 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:23.652 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T14:59:23.652 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:59:23.652 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197655 2024-09-13T14:59:23.652 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.652 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:23.653 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.653 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:23.653 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.294574+0000", 2024-09-13T14:59:23.653 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.653 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.653 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.653 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:23.653 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.294574+0000", 2024-09-13T14:59:23.653 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.653 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.653 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.653 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:23.653 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.294572+0000", 2024-09-13T14:59:23.654 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:23.654 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.654 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.654 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:23.654 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.294586+0000", 2024-09-13T14:59:23.654 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.402e-05 2024-09-13T14:59:23.654 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.654 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.654 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:23.654 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.294597+0000", 2024-09-13T14:59:23.654 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1353e-05 2024-09-13T14:59:23.654 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.654 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.654 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:23.655 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.294603+0000", 2024-09-13T14:59:23.655 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4110000000000004e-06 2024-09-13T14:59:23.655 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.655 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.655 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.655 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.294671+0000", 2024-09-13T14:59:23.655 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.8596000000000005e-05 2024-09-13T14:59:23.655 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.655 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.655 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:23.655 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.294690+0000", 2024-09-13T14:59:23.655 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.8448e-05 2024-09-13T14:59:23.655 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.655 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.656 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.656 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.329097+0000", 2024-09-13T14:59:23.656 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.034406905 2024-09-13T14:59:23.656 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.656 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.656 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:23.656 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.329164+0000", 2024-09-13T14:59:23.656 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.6860999999999996e-05 2024-09-13T14:59:23.656 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.656 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.656 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:23.656 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.329229+0000", 2024-09-13T14:59:23.656 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.5568000000000006e-05 2024-09-13T14:59:23.657 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.657 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.657 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:23.657 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.335029+0000", 2024-09-13T14:59:23.657 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0057995770000000002 2024-09-13T14:59:23.657 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.657 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.657 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.657 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.405257+0000", 2024-09-13T14:59:23.657 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.070228192999999994 2024-09-13T14:59:23.657 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.657 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.657 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.657 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.416082+0000", 2024-09-13T14:59:23.658 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.010825128999999999 2024-09-13T14:59:23.658 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.658 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.658 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:23.658 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.416130+0000", 2024-09-13T14:59:23.658 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8223000000000002e-05 2024-09-13T14:59:23.658 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.658 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.658 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:23.658 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.416168+0000", 2024-09-13T14:59:23.658 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.7808999999999997e-05 2024-09-13T14:59:23.658 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.658 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:23.659 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.659 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.659 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.659 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197661 9.6 9:6ea81a69:::benchmark_data_smithi151_261954_object94805:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T14:59:23.659 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.305951+0000", 2024-09-13T14:59:23.659 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 111.74408995100001, 2024-09-13T14:59:23.659 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.114294699, 2024-09-13T14:59:23.659 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:23.659 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:23.659 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:23.659 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T14:59:23.659 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:59:23.659 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197661 2024-09-13T14:59:23.660 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.660 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:23.660 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.660 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:23.660 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.305951+0000", 2024-09-13T14:59:23.660 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.660 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.660 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.660 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:23.660 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.305951+0000", 2024-09-13T14:59:23.660 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.660 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.660 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.660 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:23.661 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.305949+0000", 2024-09-13T14:59:23.661 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:59:23.661 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.661 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.661 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:23.661 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.305957+0000", 2024-09-13T14:59:23.661 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.994e-06 2024-09-13T14:59:23.661 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.661 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.661 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:23.661 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.305976+0000", 2024-09-13T14:59:23.661 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.8349999999999999e-05 2024-09-13T14:59:23.661 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.662 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.662 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:23.662 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.305982+0000", 2024-09-13T14:59:23.662 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.336e-06 2024-09-13T14:59:23.662 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.662 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.662 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.662 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.321123+0000", 2024-09-13T14:59:23.662 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.015140614 2024-09-13T14:59:23.662 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.662 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.662 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:23.662 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.321132+0000", 2024-09-13T14:59:23.662 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.533e-06 2024-09-13T14:59:23.663 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.663 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.663 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.663 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.330506+0000", 2024-09-13T14:59:23.663 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0093742370000001 2024-09-13T14:59:23.663 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.663 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.663 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:23.663 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.330565+0000", 2024-09-13T14:59:23.663 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8239000000000003e-05 2024-09-13T14:59:23.663 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.663 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.663 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:23.664 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.330629+0000", 2024-09-13T14:59:23.664 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.4053999999999994e-05 2024-09-13T14:59:23.664 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.664 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.664 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:23.664 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.335035+0000", 2024-09-13T14:59:23.664 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0044066849999999996 2024-09-13T14:59:23.664 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.664 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.664 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.664 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.415527+0000", 2024-09-13T14:59:23.664 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.080491317000000007 2024-09-13T14:59:23.664 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.665 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.665 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.665 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.420160+0000", 2024-09-13T14:59:23.665 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0046338359999999997 2024-09-13T14:59:23.665 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.665 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.665 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:23.665 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.420204+0000", 2024-09-13T14:59:23.665 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3850000000000002e-05 2024-09-13T14:59:23.665 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.665 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.665 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:23.665 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.420246+0000", 2024-09-13T14:59:23.665 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1322999999999997e-05 2024-09-13T14:59:23.666 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.666 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:23.666 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.666 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.666 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.666 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197663 9.6 9:6cc501ca:::benchmark_data_smithi151_261954_object94807:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T14:59:23.666 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.306393+0000", 2024-09-13T14:59:23.666 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 111.743648245, 2024-09-13T14:59:23.666 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1249737289999999, 2024-09-13T14:59:23.666 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:23.666 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:23.666 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:23.666 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T14:59:23.667 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:59:23.667 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197663 2024-09-13T14:59:23.667 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.667 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:23.667 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.667 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:23.667 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.306393+0000", 2024-09-13T14:59:23.667 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.667 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.667 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.667 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:23.667 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.306393+0000", 2024-09-13T14:59:23.667 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.667 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.668 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.668 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:23.668 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.306392+0000", 2024-09-13T14:59:23.668 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:59:23.668 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.668 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.668 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:23.668 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.306400+0000", 2024-09-13T14:59:23.668 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.4349999999999997e-06 2024-09-13T14:59:23.668 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.668 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.668 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:23.668 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.306412+0000", 2024-09-13T14:59:23.669 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2131e-05 2024-09-13T14:59:23.669 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.669 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.669 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:23.669 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.306420+0000", 2024-09-13T14:59:23.669 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.644e-06 2024-09-13T14:59:23.669 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.669 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.669 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.669 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.321161+0000", 2024-09-13T14:59:23.669 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.014741291 2024-09-13T14:59:23.669 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.669 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.670 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:23.670 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.321167+0000", 2024-09-13T14:59:23.670 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1399999999999997e-06 2024-09-13T14:59:23.670 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.670 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.670 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.670 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.331935+0000", 2024-09-13T14:59:23.670 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.010767926 2024-09-13T14:59:23.670 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.670 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.670 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:23.670 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.332015+0000", 2024-09-13T14:59:23.670 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.9986000000000003e-05 2024-09-13T14:59:23.670 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.671 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.671 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:23.671 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.332091+0000", 2024-09-13T14:59:23.671 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.6066e-05 2024-09-13T14:59:23.671 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.671 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.671 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:23.671 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.338252+0000", 2024-09-13T14:59:23.671 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0061609170000000001 2024-09-13T14:59:23.671 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.671 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.671 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.671 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.427278+0000", 2024-09-13T14:59:23.671 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.089025608000000006 2024-09-13T14:59:23.672 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.672 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.672 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.672 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.431274+0000", 2024-09-13T14:59:23.672 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.003996108 2024-09-13T14:59:23.672 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.672 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.672 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:23.672 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.431324+0000", 2024-09-13T14:59:23.672 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9848000000000001e-05 2024-09-13T14:59:23.672 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.672 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.672 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:23.673 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.431366+0000", 2024-09-13T14:59:23.673 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.2619e-05 2024-09-13T14:59:23.673 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.673 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:23.673 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.673 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.673 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.673 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197673 9.6 9:6e5dc1cf:::benchmark_data_smithi151_261954_object94817:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T14:59:23.673 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.319349+0000", 2024-09-13T14:59:23.673 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 111.730691742, 2024-09-13T14:59:23.673 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1161776160000001, 2024-09-13T14:59:23.673 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:23.673 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:23.674 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:23.674 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T14:59:23.674 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:59:23.674 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197673 2024-09-13T14:59:23.674 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.674 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:23.674 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.674 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:23.674 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.319349+0000", 2024-09-13T14:59:23.674 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.674 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.674 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.674 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:23.675 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.319349+0000", 2024-09-13T14:59:23.675 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.675 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.675 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.675 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:23.675 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.319347+0000", 2024-09-13T14:59:23.675 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:59:23.675 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.675 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.675 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:23.675 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.319354+0000", 2024-09-13T14:59:23.675 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.1110000000000002e-06 2024-09-13T14:59:23.675 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.675 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.676 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:23.676 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.319369+0000", 2024-09-13T14:59:23.676 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4137e-05 2024-09-13T14:59:23.676 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.676 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.676 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:23.676 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.319375+0000", 2024-09-13T14:59:23.676 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.4080000000000003e-06 2024-09-13T14:59:23.676 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.676 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.676 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.676 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.321187+0000", 2024-09-13T14:59:23.676 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001811937 2024-09-13T14:59:23.676 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.677 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.677 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:23.677 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.321193+0000", 2024-09-13T14:59:23.677 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1120000000000001e-06 2024-09-13T14:59:23.677 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.677 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.677 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.677 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.333564+0000", 2024-09-13T14:59:23.677 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0123709649999999 2024-09-13T14:59:23.677 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.677 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.677 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:23.678 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.333648+0000", 2024-09-13T14:59:23.678 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.3926999999999998e-05 2024-09-13T14:59:23.678 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.678 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.678 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:23.678 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.333724+0000", 2024-09-13T14:59:23.678 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.6155000000000005e-05 2024-09-13T14:59:23.678 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.678 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.678 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:23.678 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.339722+0000", 2024-09-13T14:59:23.678 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0059980229999999999 2024-09-13T14:59:23.678 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.679 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.679 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.679 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.427375+0000", 2024-09-13T14:59:23.679 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.087653178999999998 2024-09-13T14:59:23.679 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.679 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.679 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.679 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.435449+0000", 2024-09-13T14:59:23.679 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.008073172 2024-09-13T14:59:23.679 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.679 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.679 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:23.679 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.435490+0000", 2024-09-13T14:59:23.680 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1168000000000003e-05 2024-09-13T14:59:23.680 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.680 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.680 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:23.680 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.435527+0000", 2024-09-13T14:59:23.680 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.7069e-05 2024-09-13T14:59:23.680 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.680 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:23.680 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.680 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.680 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.680 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197677 9.6 9:6ff7b7d6:::benchmark_data_smithi151_261954_object94821:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T14:59:23.680 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.324907+0000", 2024-09-13T14:59:23.681 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 111.725133519, 2024-09-13T14:59:23.681 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.110717366, 2024-09-13T14:59:23.681 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:23.681 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:23.681 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:23.681 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T14:59:23.681 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:59:23.681 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197677 2024-09-13T14:59:23.681 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.681 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:23.681 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.681 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:23.681 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.324907+0000", 2024-09-13T14:59:23.682 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.682 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.682 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.682 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:23.682 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.324907+0000", 2024-09-13T14:59:23.682 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.682 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.682 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.682 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:23.682 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.324906+0000", 2024-09-13T14:59:23.682 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:23.682 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.683 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.683 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:23.683 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.324915+0000", 2024-09-13T14:59:23.683 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.2089999999999994e-06 2024-09-13T14:59:23.683 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.683 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.683 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:23.683 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.324927+0000", 2024-09-13T14:59:23.683 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1759999999999999e-05 2024-09-13T14:59:23.683 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.683 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.683 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:23.683 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.324932+0000", 2024-09-13T14:59:23.684 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0950000000000003e-06 2024-09-13T14:59:23.684 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.684 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.684 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.684 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.324978+0000", 2024-09-13T14:59:23.684 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5920000000000001e-05 2024-09-13T14:59:23.684 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.684 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.684 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:23.684 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.324985+0000", 2024-09-13T14:59:23.684 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.6399999999999997e-06 2024-09-13T14:59:23.684 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.684 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.685 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.685 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.335071+0000", 2024-09-13T14:59:23.685 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0100860730000001 2024-09-13T14:59:23.685 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.685 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.685 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:23.685 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.335145+0000", 2024-09-13T14:59:23.685 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.3349000000000005e-05 2024-09-13T14:59:23.685 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.685 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.685 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:23.685 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.335213+0000", 2024-09-13T14:59:23.685 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7719000000000007e-05 2024-09-13T14:59:23.686 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.686 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.686 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:23.686 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.339729+0000", 2024-09-13T14:59:23.686 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.004516066 2024-09-13T14:59:23.686 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.686 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.686 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.686 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.427485+0000", 2024-09-13T14:59:23.686 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.087756043000000006 2024-09-13T14:59:23.686 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.686 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.686 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.686 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.435557+0000", 2024-09-13T14:59:23.687 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.008072648 2024-09-13T14:59:23.687 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.687 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.687 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:23.687 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.435593+0000", 2024-09-13T14:59:23.687 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.5373999999999997e-05 2024-09-13T14:59:23.687 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.687 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.687 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:23.687 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.435625+0000", 2024-09-13T14:59:23.687 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.2020000000000002e-05 2024-09-13T14:59:23.687 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.687 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:23.688 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.688 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.688 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.688 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197692 9.6 9:6ccfa49f:::benchmark_data_smithi151_261954_object94836:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T14:59:23.688 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.354825+0000", 2024-09-13T14:59:23.688 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 111.695215379, 2024-09-13T14:59:23.688 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0808906490000001, 2024-09-13T14:59:23.688 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:23.688 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:23.688 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:23.688 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T14:59:23.688 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:59:23.688 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197692 2024-09-13T14:59:23.689 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.689 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:23.689 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.689 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:23.689 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.354825+0000", 2024-09-13T14:59:23.689 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.689 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.689 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.689 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:23.689 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.354825+0000", 2024-09-13T14:59:23.689 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.689 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.689 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.690 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:23.690 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.354824+0000", 2024-09-13T14:59:23.690 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:23.690 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.690 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.690 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:23.690 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.354829+0000", 2024-09-13T14:59:23.690 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.2789999999999998e-06 2024-09-13T14:59:23.690 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.690 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.690 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:23.690 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.354839+0000", 2024-09-13T14:59:23.690 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0052e-05 2024-09-13T14:59:23.690 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.691 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.691 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:23.691 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.354844+0000", 2024-09-13T14:59:23.691 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5140000000000003e-06 2024-09-13T14:59:23.691 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.691 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.691 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.691 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.354895+0000", 2024-09-13T14:59:23.691 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0875999999999999e-05 2024-09-13T14:59:23.691 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.691 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.691 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:23.691 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.354902+0000", 2024-09-13T14:59:23.692 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.6899999999999992e-06 2024-09-13T14:59:23.692 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.692 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.692 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.692 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.336702+0000", 2024-09-13T14:59:23.692 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.98179912499999999 2024-09-13T14:59:23.692 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.692 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.692 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:23.692 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.336790+0000", 2024-09-13T14:59:23.692 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.7943999999999997e-05 2024-09-13T14:59:23.692 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.692 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.692 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:23.693 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.336870+0000", 2024-09-13T14:59:23.693 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0018000000000005e-05 2024-09-13T14:59:23.693 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.693 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.693 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:23.693 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.339822+0000", 2024-09-13T14:59:23.693 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0029526470000000001 2024-09-13T14:59:23.693 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.693 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.693 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.693 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.427588+0000", 2024-09-13T14:59:23.693 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.087765459000000004 2024-09-13T14:59:23.693 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.694 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.694 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.694 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.435654+0000", 2024-09-13T14:59:23.694 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0080660839999999994 2024-09-13T14:59:23.694 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.694 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.694 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:23.694 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.435687+0000", 2024-09-13T14:59:23.694 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3231000000000001e-05 2024-09-13T14:59:23.694 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.694 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.694 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:23.694 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.435716+0000", 2024-09-13T14:59:23.694 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.9193999999999999e-05 2024-09-13T14:59:23.695 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.695 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:23.695 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.695 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.695 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.695 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197694 9.6 9:63889f28:::benchmark_data_smithi151_261954_object94838:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T14:59:23.695 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.360247+0000", 2024-09-13T14:59:23.695 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 111.689794116, 2024-09-13T14:59:23.695 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.075543398, 2024-09-13T14:59:23.695 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:23.695 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:23.695 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:23.695 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T14:59:23.695 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:59:23.696 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197694 2024-09-13T14:59:23.696 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.696 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:23.696 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.696 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:23.696 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.360247+0000", 2024-09-13T14:59:23.696 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.696 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.696 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.696 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:23.696 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.360247+0000", 2024-09-13T14:59:23.696 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.696 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.696 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.697 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:23.697 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.360245+0000", 2024-09-13T14:59:23.697 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:23.697 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.697 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.697 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:23.697 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.360250+0000", 2024-09-13T14:59:23.697 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8670000000000002e-06 2024-09-13T14:59:23.697 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.697 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.698 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:23.698 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.360270+0000", 2024-09-13T14:59:23.698 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.9613e-05 2024-09-13T14:59:23.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.698 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.698 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:23.698 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.360275+0000", 2024-09-13T14:59:23.698 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9400000000000001e-06 2024-09-13T14:59:23.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.698 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.698 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.698 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.360331+0000", 2024-09-13T14:59:23.698 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6437000000000002e-05 2024-09-13T14:59:23.699 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.699 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.699 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:23.699 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.360341+0000", 2024-09-13T14:59:23.699 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0128e-05 2024-09-13T14:59:23.699 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.699 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.699 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.699 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.338291+0000", 2024-09-13T14:59:23.699 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.97794926900000001 2024-09-13T14:59:23.699 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.699 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.699 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:23.699 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.338371+0000", 2024-09-13T14:59:23.700 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0735999999999994e-05 2024-09-13T14:59:23.700 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.700 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.700 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:23.700 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.338449+0000", 2024-09-13T14:59:23.700 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.8010999999999996e-05 2024-09-13T14:59:23.700 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.700 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.700 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:23.700 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.366705+0000", 2024-09-13T14:59:23.700 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.028255118999999999 2024-09-13T14:59:23.700 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.700 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.700 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.701 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.427612+0000", 2024-09-13T14:59:23.701 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.060906982999999998 2024-09-13T14:59:23.701 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.704 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.704 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.704 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.435740+0000", 2024-09-13T14:59:23.704 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0081282839999999995 2024-09-13T14:59:23.704 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.704 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.704 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:23.704 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.435767+0000", 2024-09-13T14:59:23.704 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.7004000000000001e-05 2024-09-13T14:59:23.704 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.704 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.704 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:23.705 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.435790+0000", 2024-09-13T14:59:23.705 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.3312999999999999e-05 2024-09-13T14:59:23.705 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.705 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:23.705 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.705 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.705 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.705 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:204666 9.6 9:6c15a554:::benchmark_data_smithi151_261954_object101810:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1255)", 2024-09-13T14:59:23.705 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:39.992837+0000", 2024-09-13T14:59:23.705 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 103.057204126, 2024-09-13T14:59:23.705 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.287841291, 2024-09-13T14:59:23.705 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:23.705 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:23.706 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:23.706 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T14:59:23.706 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:59:23.706 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 204666 2024-09-13T14:59:23.706 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.706 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:23.706 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.706 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:23.706 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:39.992837+0000", 2024-09-13T14:59:23.706 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.706 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.706 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.706 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:23.706 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:39.992837+0000", 2024-09-13T14:59:23.707 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.707 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.707 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.707 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:23.707 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:39.992835+0000", 2024-09-13T14:59:23.707 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999976 2024-09-13T14:59:23.707 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.707 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.707 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:23.707 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:39.992845+0000", 2024-09-13T14:59:23.707 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0101e-05 2024-09-13T14:59:23.707 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.707 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.707 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:23.708 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:39.992858+0000", 2024-09-13T14:59:23.708 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2879e-05 2024-09-13T14:59:23.708 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.708 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.708 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:23.708 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:39.992863+0000", 2024-09-13T14:59:23.708 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6520000000000003e-06 2024-09-13T14:59:23.708 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.708 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.708 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.708 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:39.992916+0000", 2024-09-13T14:59:23.708 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3201000000000001e-05 2024-09-13T14:59:23.708 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.709 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.709 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:23.709 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:39.993030+0000", 2024-09-13T14:59:23.709 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000113307 2024-09-13T14:59:23.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.709 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.709 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:23.709 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:39.993148+0000", 2024-09-13T14:59:23.709 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000118467 2024-09-13T14:59:23.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.709 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.709 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:23.709 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:40.003730+0000", 2024-09-13T14:59:23.710 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.010581633999999999 2024-09-13T14:59:23.710 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.710 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.710 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.710 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:40.003762+0000", 2024-09-13T14:59:23.710 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.1931000000000003e-05 2024-09-13T14:59:23.710 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.710 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.710 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.710 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:40.280604+0000", 2024-09-13T14:59:23.710 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.27684178999999998 2024-09-13T14:59:23.710 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.710 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.710 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:23.711 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:40.280636+0000", 2024-09-13T14:59:23.711 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.2780000000000001e-05 2024-09-13T14:59:23.711 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.711 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.711 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:23.711 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:40.280678+0000", 2024-09-13T14:59:23.711 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1715000000000003e-05 2024-09-13T14:59:23.711 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.711 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:23.711 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.711 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.711 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.711 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:204677 9.6 9:696ebb35:::benchmark_data_smithi151_261954_object101821:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1255)", 2024-09-13T14:59:23.711 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:40.004049+0000", 2024-09-13T14:59:23.712 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 103.045991989, 2024-09-13T14:59:23.712 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.27671621099999999, 2024-09-13T14:59:23.712 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:23.712 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:23.712 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:23.712 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T14:59:23.712 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T14:59:23.712 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 204677 2024-09-13T14:59:23.712 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.712 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:23.712 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.712 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:23.712 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:40.004049+0000", 2024-09-13T14:59:23.713 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.713 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.713 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.713 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:23.713 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:40.004049+0000", 2024-09-13T14:59:23.713 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.713 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.713 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.713 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:23.713 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:40.004048+0000", 2024-09-13T14:59:23.713 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:23.713 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.713 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.713 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:23.714 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:40.004055+0000", 2024-09-13T14:59:23.714 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.6690000000000006e-06 2024-09-13T14:59:23.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.714 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.714 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:23.714 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:40.004065+0000", 2024-09-13T14:59:23.714 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.9539999999999999e-06 2024-09-13T14:59:23.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.714 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.714 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:23.714 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:40.004071+0000", 2024-09-13T14:59:23.714 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3199999999999999e-06 2024-09-13T14:59:23.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.714 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.715 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.715 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:40.004151+0000", 2024-09-13T14:59:23.715 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0167999999999995e-05 2024-09-13T14:59:23.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.715 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.715 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:23.715 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:40.004298+0000", 2024-09-13T14:59:23.715 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014701800000000001 2024-09-13T14:59:23.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.715 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.715 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:23.715 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:40.004423+0000", 2024-09-13T14:59:23.715 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012567500000000001 2024-09-13T14:59:23.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.716 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.716 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:23.716 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:40.011351+0000", 2024-09-13T14:59:23.716 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0069270690000000001 2024-09-13T14:59:23.716 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.716 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.716 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.716 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:40.012173+0000", 2024-09-13T14:59:23.716 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00082271999999999998 2024-09-13T14:59:23.716 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.716 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.716 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.716 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:40.280713+0000", 2024-09-13T14:59:23.716 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.26853982599999998 2024-09-13T14:59:23.717 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.717 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.717 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:23.717 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:40.280743+0000", 2024-09-13T14:59:23.717 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.9886000000000002e-05 2024-09-13T14:59:23.717 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.717 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.717 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:23.717 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:40.280765+0000", 2024-09-13T14:59:23.717 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.2099e-05 2024-09-13T14:59:23.717 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.717 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:23.717 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.717 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.718 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.718 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:99175 10.9 10:90d922ad:::benchmark_data_smithi151_288181_object99174:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1277)", 2024-09-13T14:59:23.718 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:59:19.831592+0000", 2024-09-13T14:59:23.718 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 3.2184487580000001, 2024-09-13T14:59:23.718 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.28007362099999999, 2024-09-13T14:59:23.718 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:23.718 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:23.718 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:23.718 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T14:59:23.718 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T14:59:23.718 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 99175 2024-09-13T14:59:23.718 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.718 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:23.719 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.719 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:23.719 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.831592+0000", 2024-09-13T14:59:23.719 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.719 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.719 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.719 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:23.719 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.831592+0000", 2024-09-13T14:59:23.719 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.719 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.719 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.719 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:23.719 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.831591+0000", 2024-09-13T14:59:23.720 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:23.720 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.720 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.720 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:23.720 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.831634+0000", 2024-09-13T14:59:23.720 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3673e-05 2024-09-13T14:59:23.720 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.720 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.720 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:23.720 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.831650+0000", 2024-09-13T14:59:23.720 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.537e-05 2024-09-13T14:59:23.720 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.720 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.721 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:23.721 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.831657+0000", 2024-09-13T14:59:23.721 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.3860000000000001e-06 2024-09-13T14:59:23.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.721 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.721 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.721 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.831697+0000", 2024-09-13T14:59:23.721 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0107000000000001e-05 2024-09-13T14:59:23.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.721 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.722 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:23.722 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.831788+0000", 2024-09-13T14:59:23.722 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.0734000000000002e-05 2024-09-13T14:59:23.722 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.722 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.722 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,7", 2024-09-13T14:59:23.722 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.831885+0000", 2024-09-13T14:59:23.722 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.7294999999999996e-05 2024-09-13T14:59:23.722 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.722 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.722 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:23.722 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.834495+0000", 2024-09-13T14:59:23.722 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0026095900000000002 2024-09-13T14:59:23.723 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.723 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.723 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.723 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.835682+0000", 2024-09-13T14:59:23.723 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0011867799999999999 2024-09-13T14:59:23.723 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.723 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.723 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.723 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:20.111530+0000", 2024-09-13T14:59:23.723 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.27584882799999999 2024-09-13T14:59:23.724 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.724 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.724 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:23.724 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:20.111622+0000", 2024-09-13T14:59:23.724 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.1657999999999996e-05 2024-09-13T14:59:23.724 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.724 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.724 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:23.724 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:20.111666+0000", 2024-09-13T14:59:23.724 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3739999999999998e-05 2024-09-13T14:59:23.724 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.724 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:23.724 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.725 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.725 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:99196 10.9 10:94dd644b:::benchmark_data_smithi151_288181_object99195:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1277)", 2024-09-13T14:59:23.725 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:59:19.860330+0000", 2024-09-13T14:59:23.725 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 3.1897112239999998, 2024-09-13T14:59:23.725 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.25205488999999998, 2024-09-13T14:59:23.725 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:23.725 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:23.725 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:23.725 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T14:59:23.726 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T14:59:23.726 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 99196 2024-09-13T14:59:23.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.726 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:23.726 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.726 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:23.726 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.860330+0000", 2024-09-13T14:59:23.726 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.726 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.726 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:23.726 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.860330+0000", 2024-09-13T14:59:23.726 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:23.727 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.727 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.727 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:23.727 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.860328+0000", 2024-09-13T14:59:23.727 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:23.727 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.727 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.727 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:23.727 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.860381+0000", 2024-09-13T14:59:23.727 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.2859999999999999e-05 2024-09-13T14:59:23.727 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.727 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.727 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:23.728 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.860394+0000", 2024-09-13T14:59:23.728 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.259e-05 2024-09-13T14:59:23.728 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.728 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.728 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:23.728 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.860401+0000", 2024-09-13T14:59:23.728 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.798e-06 2024-09-13T14:59:23.728 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.728 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.728 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:23.728 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.860435+0000", 2024-09-13T14:59:23.728 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.4230000000000003e-05 2024-09-13T14:59:23.728 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.729 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.729 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:23.729 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.860533+0000", 2024-09-13T14:59:23.729 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.7850000000000007e-05 2024-09-13T14:59:23.729 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.729 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.729 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,7", 2024-09-13T14:59:23.729 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.860625+0000", 2024-09-13T14:59:23.729 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.2081000000000001e-05 2024-09-13T14:59:23.729 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.729 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.729 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:23.729 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.865782+0000", 2024-09-13T14:59:23.729 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0051574890000000003 2024-09-13T14:59:23.730 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.730 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.730 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.730 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.867977+0000", 2024-09-13T14:59:23.730 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0021951639999999999 2024-09-13T14:59:23.730 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.730 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.730 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:23.730 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:20.112307+0000", 2024-09-13T14:59:23.730 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.24432907500000001 2024-09-13T14:59:23.730 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.730 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.730 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:23.730 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:20.112351+0000", 2024-09-13T14:59:23.731 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4861000000000003e-05 2024-09-13T14:59:23.731 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:23.731 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:23.731 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:23.731 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:20.112385+0000", 2024-09-13T14:59:23.731 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3102999999999998e-05 2024-09-13T14:59:23.731 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.731 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:23.731 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.731 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:23.731 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:23.731 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:59:23.732 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:23.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:23.771 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:23.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:23.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:23.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:23.910 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:23.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:24.011 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:24.049 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:24.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:24.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:24.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:23 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/384468227' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:24.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:23 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/384468227' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:24.151 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:24.186 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:24.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:24.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:24.198 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 67 16 103118 103102 96.1601 50.3125 0.0101589 0.0103939 2024-09-13T14:59:24.289 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:24.323 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:24.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:24.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:24.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:23 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/384468227' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:24.424 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:24.461 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:24.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:24.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:24.474 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:59:24.564 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:24.592 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:24.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:24.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:24.694 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:24.723 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:24.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:24.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:24.826 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:24.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:59:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:59:24] "GET /metrics HTTP/1.1" 200 36698 "" "Prometheus/2.51.0" 2024-09-13T14:59:24.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:24.867 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:24.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:24.968 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:25.003 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:25.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:25.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:25.088 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T14:59:25.105 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:25.136 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:25.176 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:25.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:25.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:25.198 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 68 16 104152 104136 95.6964 64.625 0.0182195 0.0104433 2024-09-13T14:59:25.224 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:24 smithi151 ceph-mon[108966]: pgmap v2082: 119 pgs: 119 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T14:59:25.224 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:24 smithi151 ceph-mon[104450]: pgmap v2082: 119 pgs: 119 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T14:59:25.278 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:25.310 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:25.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:25.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:25.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:24 smithi184 ceph-mon[92527]: pgmap v2082: 119 pgs: 119 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T14:59:25.412 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:25.455 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:25.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:25.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:25.507 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:59:25.558 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:25.600 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:25.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:25.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:25.702 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:25.741 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:25.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:25.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:25.843 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:25.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:25.883 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:25.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:25.984 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:26.012 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:26.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:26.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:26.115 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:26.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:26.154 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:26.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:26.198 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 69 16 105159 105143 95.2216 62.9375 0.0117556 0.0104959 2024-09-13T14:59:26.255 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:26.295 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:26.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:26.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:26.323 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: specified pgp_num 61 > pg_num 31 2024-09-13T14:59:26.397 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:26.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:26.431 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:26.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:26.532 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:26.568 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:26.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:26.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:26.670 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:26.705 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:26.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:26.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:26.807 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:26.847 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:26.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:26.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:26.949 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:26.984 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:26.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:26.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:27.086 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:27.092 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:59:27.098 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:26 smithi184 ceph-mon[92527]: pgmap v2083: 119 pgs: 119 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T14:59:27.098 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:26 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3813686978' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-13T14:59:27.098 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:26 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-13T14:59:27.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:26 smithi151 ceph-mon[108966]: pgmap v2083: 119 pgs: 119 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T14:59:27.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:26 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3813686978' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-13T14:59:27.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:26 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-13T14:59:27.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:26 smithi151 ceph-mon[104450]: pgmap v2083: 119 pgs: 119 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T14:59:27.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:26 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3813686978' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-13T14:59:27.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:26 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-13T14:59:27.140 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:27.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:27.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:27.201 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 70 16 106234 106218 94.8171 67.1875 0.006126 0.0105411 2024-09-13T14:59:27.243 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:27.288 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:27.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:27.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:27.390 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:27.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:27.434 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:27.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:27.535 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:27.542 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:59:27.542 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:59:27.573 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:27.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:27.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:27.675 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:27.724 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:27.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:27.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:27.826 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:27.835 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T14:59:27.871 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:27.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:27.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:27.909 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:27.973 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:28.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:28.018 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:28.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:28.119 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:28.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:28.154 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:28.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:28.201 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 71 16 107246 107230 94.3719 63.25 0.0188942 0.0105914 2024-09-13T14:59:28.249 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T14:59:28.255 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:28.296 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:28.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:28.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:28.398 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:28.427 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:28.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:28.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:28.529 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:28.568 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:28.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:28.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:28.654 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:59:28.669 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:28.704 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:28.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:28.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:28.806 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:28.839 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:28.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:28.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:28.941 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:28.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:28.981 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:28.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:29.082 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:29.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:28 smithi184 ceph-mon[92527]: pgmap v2084: 119 pgs: 119 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 60 MiB/s wr, 966 op/s 2024-09-13T14:59:29.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:28 smithi151 ceph-mon[104450]: pgmap v2084: 119 pgs: 119 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 60 MiB/s wr, 966 op/s 2024-09-13T14:59:29.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:28 smithi151 ceph-mon[108966]: pgmap v2084: 119 pgs: 119 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 60 MiB/s wr, 966 op/s 2024-09-13T14:59:29.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:29.113 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:29.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:29.201 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 72 16 108240 108224 93.924 62.125 0.0177113 0.0106419 2024-09-13T14:59:29.214 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:29.252 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:29.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:29.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:29.354 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:29.390 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:29.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:29.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:29.492 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:29.524 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:29.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:29.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:29.626 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:29.662 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:29.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:29.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:29.764 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:29.801 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:29.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:29.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:29.904 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:29.940 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:29.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:29.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:29.969 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-13T14:59:30.042 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:30.081 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:30.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:30.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:30.183 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:30.202 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 73 16 109305 109289 93.5491 66.5625 0.0134905 0.010685 2024-09-13T14:59:30.213 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:30.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:30.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:30.215 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:29 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2888008317' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:30.215 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:29 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:30.315 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:30.354 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:30.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:30.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:30.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:29 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2888008317' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:30.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:29 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:30.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:29 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2888008317' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:30.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:29 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:30.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:30.471 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:59:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:59:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:59:30.490 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:30.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:30.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:30.592 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:30.629 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:59:30.629 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:59:30.637 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:30.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:30.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:30.739 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:30.772 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:30.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:30.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:30.874 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:30.910 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:30.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:30.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:31.012 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:31.023 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:30 smithi151 ceph-mon[104450]: pgmap v2085: 119 pgs: 119 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 55 MiB/s wr, 882 op/s 2024-09-13T14:59:31.023 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:30 smithi151 ceph-mon[104450]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:31.023 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:30 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:59:31.023 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:30 smithi151 ceph-mon[104450]: osdmap e1278: 8 total, 8 up, 8 in 2024-09-13T14:59:31.025 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:30 smithi151 ceph-mon[108966]: pgmap v2085: 119 pgs: 119 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 55 MiB/s wr, 882 op/s 2024-09-13T14:59:31.025 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:30 smithi151 ceph-mon[108966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:31.025 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:30 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:59:31.026 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:30 smithi151 ceph-mon[108966]: osdmap e1278: 8 total, 8 up, 8 in 2024-09-13T14:59:31.056 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:31.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:31.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:31.104 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:59:31.105 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:59:31.137 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:59:31.138 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:59:31.158 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:31.171 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:59:31.171 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:59:31.201 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:31.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:31.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:31.202 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 74 16 110341 110325 93.1599 64.75 0.00940992 0.0107288 2024-09-13T14:59:31.211 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:59:31.212 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:59:31.256 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:59:31.256 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:59:31.284 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:59:31.284 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:59:31.302 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:31.310 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:30 smithi184 ceph-mon[92527]: pgmap v2085: 119 pgs: 119 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 55 MiB/s wr, 882 op/s 2024-09-13T14:59:31.310 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:30 smithi184 ceph-mon[92527]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:31.310 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:30 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:59:31.311 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:30 smithi184 ceph-mon[92527]: osdmap e1278: 8 total, 8 up, 8 in 2024-09-13T14:59:31.319 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:59:31.320 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:59:31.344 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:31.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:31.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:31.349 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:59:31.349 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:59:31.356 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T14:59:31.447 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:31.496 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:31.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:31.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:31.598 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:31.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:31.645 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:31.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:31.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:31.784 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:31.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:31.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:31.823 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:59:31.886 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:31.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:31.934 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:31.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:32.035 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:32.064 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:32.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:32.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:32.166 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:32.202 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 75 16 111394 111378 92.7952 65.8125 0.0150309 0.0107717 2024-09-13T14:59:32.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:32.206 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:32.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:32.307 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:32.332 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:31 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1008684866' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:32.333 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:59:32.333 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:32.349 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:32.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:32.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:32.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:31 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1008684866' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:32.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:59:32.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:32.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:31 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1008684866' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:32.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T14:59:32.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:32.451 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:32.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:32.483 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:32.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:32.585 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:32.623 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:32.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:32.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:32.725 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:32.766 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:32.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:32.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:32.775 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:59:32.836 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 6, 7, 1, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:59:32.837 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:59:32.837 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-13T14:59:32.837 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:59:32.837 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:59:32.837 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-13T14:59:32.868 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:32.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:32.938 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:32.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:33.039 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:33.083 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:33.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:33.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:33.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:32 smithi151 ceph-mon[104450]: pgmap v2087: 119 pgs: 119 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 60 MiB/s wr, 962 op/s 2024-09-13T14:59:33.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:32 smithi151 ceph-mon[108966]: pgmap v2087: 119 pgs: 119 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 60 MiB/s wr, 962 op/s 2024-09-13T14:59:33.186 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:33.198 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:32 smithi184 ceph-mon[92527]: pgmap v2087: 119 pgs: 119 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 60 MiB/s wr, 962 op/s 2024-09-13T14:59:33.202 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 76 16 112437 112421 92.4317 65.1875 0.0158979 0.0108137 2024-09-13T14:59:33.218 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:33.224 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:33.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:33.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:33.326 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:33.342 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:59:33.363 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:33.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:33.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:33.465 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:33.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:33.496 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:33.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:33.598 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:33.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:33.641 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:33.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:33.742 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:33.775 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:33.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:33.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:33.877 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:33.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:33.916 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:33.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:34.017 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:34.054 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:34.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:34.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:34.067 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:59:34.157 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:34.404 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 77 16 113417 113401 92.0267 61.25 0.0352425 0.0108611 2024-09-13T14:59:34.441 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:34.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:34.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:34.543 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:34.579 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:34.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:34.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:34.681 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:34.695 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:59:34.695 INFO:teuthology.orchestra.run.smithi184.stdout: "size": 20, 2024-09-13T14:59:34.695 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 600, 2024-09-13T14:59:34.696 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [ 2024-09-13T14:59:34.696 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.696 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176106 8.5 8:a0ea7fc3:::benchmark_data_smithi151_229097_object51238:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1185)", 2024-09-13T14:59:34.696 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.478783+0000", 2024-09-13T14:59:34.696 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 448.12510616100002, 2024-09-13T14:59:34.696 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0704500290000001, 2024-09-13T14:59:34.696 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:34.696 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "reached pg", 2024-09-13T14:59:34.696 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:34.696 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:59:34.696 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:59:34.696 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176106 2024-09-13T14:59:34.696 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.697 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:34.697 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.697 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:34.697 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.478783+0000", 2024-09-13T14:59:34.697 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.697 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.697 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.697 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:34.697 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.478783+0000", 2024-09-13T14:59:34.697 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.697 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.697 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.697 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:34.697 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.478781+0000", 2024-09-13T14:59:34.698 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:59:34.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.698 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.698 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:34.698 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.478788+0000", 2024-09-13T14:59:34.698 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.1450000000000002e-06 2024-09-13T14:59:34.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.698 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.698 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:34.698 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.478805+0000", 2024-09-13T14:59:34.698 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6324000000000002e-05 2024-09-13T14:59:34.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.698 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.698 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:34.699 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.478812+0000", 2024-09-13T14:59:34.699 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.9059999999999996e-06 2024-09-13T14:59:34.699 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.699 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.699 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.699 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.478866+0000", 2024-09-13T14:59:34.699 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4895000000000002e-05 2024-09-13T14:59:34.699 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.699 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.699 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:34.699 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.479030+0000", 2024-09-13T14:59:34.699 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000163855 2024-09-13T14:59:34.699 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.699 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.700 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,4,7", 2024-09-13T14:59:34.700 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.479187+0000", 2024-09-13T14:59:34.700 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000157152 2024-09-13T14:59:34.700 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.700 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.700 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.700 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.485019+0000", 2024-09-13T14:59:34.700 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0058318290000000002 2024-09-13T14:59:34.700 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.700 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.700 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:34.700 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.485030+0000", 2024-09-13T14:59:34.700 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0482999999999999e-05 2024-09-13T14:59:34.700 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.701 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.701 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.701 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.511091+0000", 2024-09-13T14:59:34.701 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.026061415 2024-09-13T14:59:34.701 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.701 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.701 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for missing object", 2024-09-13T14:59:34.701 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.511214+0000", 2024-09-13T14:59:34.701 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012329499999999999 2024-09-13T14:59:34.701 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.701 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.701 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.701 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549141+0000", 2024-09-13T14:59:34.702 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.037926540000000002 2024-09-13T14:59:34.702 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.702 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.702 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:34.702 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549233+0000", 2024-09-13T14:59:34.702 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.1943999999999999e-05 2024-09-13T14:59:34.702 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.702 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:34.702 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.702 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.702 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.702 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176132 8.5 8:a8107e76:::benchmark_data_smithi151_229097_object51264:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1186)", 2024-09-13T14:59:34.702 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.508435+0000", 2024-09-13T14:59:34.702 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 448.095453722, 2024-09-13T14:59:34.703 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0887406580000001, 2024-09-13T14:59:34.703 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:34.703 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:34.703 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:34.703 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:59:34.703 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:59:34.703 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176132 2024-09-13T14:59:34.703 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.703 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:34.703 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.703 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:34.703 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.508435+0000", 2024-09-13T14:59:34.703 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.703 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.703 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.704 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:34.704 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.508435+0000", 2024-09-13T14:59:34.704 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.704 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.704 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.704 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:34.704 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.508433+0000", 2024-09-13T14:59:34.704 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:59:34.704 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.704 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.704 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:34.704 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.508441+0000", 2024-09-13T14:59:34.704 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.6699999999999994e-06 2024-09-13T14:59:34.705 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.705 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.705 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:34.705 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.508461+0000", 2024-09-13T14:59:34.705 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.0315000000000001e-05 2024-09-13T14:59:34.705 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.705 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.705 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:34.705 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.508469+0000", 2024-09-13T14:59:34.705 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.7610000000000007e-06 2024-09-13T14:59:34.705 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.705 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.705 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.705 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.508522+0000", 2024-09-13T14:59:34.705 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3053000000000002e-05 2024-09-13T14:59:34.706 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.706 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.706 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:34.706 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.508531+0000", 2024-09-13T14:59:34.706 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.2529999999999993e-06 2024-09-13T14:59:34.706 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.706 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.706 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.706 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.511245+0000", 2024-09-13T14:59:34.706 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.002713752 2024-09-13T14:59:34.706 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.706 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.706 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:34.706 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.511318+0000", 2024-09-13T14:59:34.707 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.2512e-05 2024-09-13T14:59:34.707 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.707 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.707 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:59:34.707 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.511392+0000", 2024-09-13T14:59:34.707 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.4664000000000002e-05 2024-09-13T14:59:34.707 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.707 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.707 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:34.707 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.519958+0000", 2024-09-13T14:59:34.707 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0085655379999999993 2024-09-13T14:59:34.707 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.707 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.707 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.707 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.548866+0000", 2024-09-13T14:59:34.708 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.028907676 2024-09-13T14:59:34.708 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.708 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.708 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.708 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.597007+0000", 2024-09-13T14:59:34.708 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.048141679999999999 2024-09-13T14:59:34.708 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.708 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.708 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:34.708 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.597062+0000", 2024-09-13T14:59:34.708 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4799000000000002e-05 2024-09-13T14:59:34.708 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.708 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.708 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:34.709 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.597176+0000", 2024-09-13T14:59:34.709 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00011391399999999999 2024-09-13T14:59:34.709 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.709 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:34.709 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.709 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.709 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176144 8.5 8:a9c768c7:::benchmark_data_smithi151_229097_object51276:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1186)", 2024-09-13T14:59:34.709 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.520194+0000", 2024-09-13T14:59:34.709 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 448.08369518500001, 2024-09-13T14:59:34.709 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0820666210000001, 2024-09-13T14:59:34.709 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:34.709 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:34.710 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:34.710 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:59:34.710 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:59:34.710 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176144 2024-09-13T14:59:34.710 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.710 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:34.710 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.710 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:34.710 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.520194+0000", 2024-09-13T14:59:34.710 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.710 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.710 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.710 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:34.710 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.520194+0000", 2024-09-13T14:59:34.710 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.711 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.711 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.711 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:34.711 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.520192+0000", 2024-09-13T14:59:34.711 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:34.711 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.711 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.711 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:34.711 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.520202+0000", 2024-09-13T14:59:34.711 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.6059999999999992e-06 2024-09-13T14:59:34.711 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.711 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.711 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:34.711 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.520215+0000", 2024-09-13T14:59:34.712 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3455000000000001e-05 2024-09-13T14:59:34.712 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.712 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.712 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:34.712 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.520222+0000", 2024-09-13T14:59:34.712 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3149999999999997e-06 2024-09-13T14:59:34.712 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.712 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.712 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.712 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.520272+0000", 2024-09-13T14:59:34.712 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9869e-05 2024-09-13T14:59:34.712 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.712 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.712 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:34.713 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.520279+0000", 2024-09-13T14:59:34.713 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.6660000000000008e-06 2024-09-13T14:59:34.713 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.713 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.713 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.713 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.513595+0000", 2024-09-13T14:59:34.713 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.99331558900000005 2024-09-13T14:59:34.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.714 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.714 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:34.714 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.513707+0000", 2024-09-13T14:59:34.714 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000112247 2024-09-13T14:59:34.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.714 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.714 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:59:34.714 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.513842+0000", 2024-09-13T14:59:34.715 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013462500000000001 2024-09-13T14:59:34.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.715 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.715 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:34.715 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.524620+0000", 2024-09-13T14:59:34.715 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.010777927 2024-09-13T14:59:34.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.715 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.715 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.715 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549270+0000", 2024-09-13T14:59:34.715 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.024650726000000001 2024-09-13T14:59:34.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.715 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.715 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.716 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.602098+0000", 2024-09-13T14:59:34.716 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.052828167000000002 2024-09-13T14:59:34.716 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.716 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.716 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:34.716 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.602150+0000", 2024-09-13T14:59:34.716 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1554999999999997e-05 2024-09-13T14:59:34.716 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.716 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.716 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:34.716 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.602261+0000", 2024-09-13T14:59:34.717 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000110504 2024-09-13T14:59:34.717 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.717 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:34.717 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.717 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.717 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.717 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176145 8.5 8:a3632e40:::benchmark_data_smithi151_229097_object51277:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1186)", 2024-09-13T14:59:34.717 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.520581+0000", 2024-09-13T14:59:34.717 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 448.083308069, 2024-09-13T14:59:34.717 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0818620189999999, 2024-09-13T14:59:34.717 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:34.717 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:34.717 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:34.717 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:59:34.718 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:59:34.718 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176145 2024-09-13T14:59:34.718 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.718 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:34.718 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.718 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:34.718 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.520581+0000", 2024-09-13T14:59:34.718 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.718 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.718 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.718 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:34.718 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.520581+0000", 2024-09-13T14:59:34.718 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.718 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.718 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.719 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:34.719 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.520580+0000", 2024-09-13T14:59:34.719 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:59:34.719 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.719 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.719 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:34.719 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.520585+0000", 2024-09-13T14:59:34.719 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.2240000000000003e-06 2024-09-13T14:59:34.719 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.719 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.719 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:34.719 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.520596+0000", 2024-09-13T14:59:34.719 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0985000000000001e-05 2024-09-13T14:59:34.719 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.720 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.720 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:34.720 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.520602+0000", 2024-09-13T14:59:34.720 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3870000000000003e-06 2024-09-13T14:59:34.720 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.720 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.720 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.720 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.521409+0000", 2024-09-13T14:59:34.720 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00080681600000000002 2024-09-13T14:59:34.720 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.720 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.720 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:34.720 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.521415+0000", 2024-09-13T14:59:34.720 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.8290000000000002e-06 2024-09-13T14:59:34.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.721 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.721 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.721 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.515883+0000", 2024-09-13T14:59:34.721 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.99446738599999995 2024-09-13T14:59:34.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.721 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.721 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:34.721 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.515999+0000", 2024-09-13T14:59:34.721 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000116388 2024-09-13T14:59:34.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.721 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.721 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:59:34.721 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.516074+0000", 2024-09-13T14:59:34.721 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.4924000000000003e-05 2024-09-13T14:59:34.722 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.722 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.722 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:34.722 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.524628+0000", 2024-09-13T14:59:34.722 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0085543259999999992 2024-09-13T14:59:34.722 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.722 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.722 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.722 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549302+0000", 2024-09-13T14:59:34.722 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.024673249000000001 2024-09-13T14:59:34.722 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.722 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.722 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.723 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.602311+0000", 2024-09-13T14:59:34.723 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.053008936 2024-09-13T14:59:34.723 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.723 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.723 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:34.723 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.602344+0000", 2024-09-13T14:59:34.723 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3071000000000002e-05 2024-09-13T14:59:34.723 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.723 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.723 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:34.723 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.602443+0000", 2024-09-13T14:59:34.723 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.9426000000000007e-05 2024-09-13T14:59:34.723 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.723 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:34.724 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.724 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.724 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.724 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176149 8.5 8:ad876389:::benchmark_data_smithi151_229097_object51281:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1186)", 2024-09-13T14:59:34.724 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.522270+0000", 2024-09-13T14:59:34.724 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 448.08161886099998, 2024-09-13T14:59:34.724 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0803534560000001, 2024-09-13T14:59:34.724 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:34.724 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:34.724 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:34.724 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:59:34.724 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:59:34.724 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176149 2024-09-13T14:59:34.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.725 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:34.725 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.725 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:34.725 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.522270+0000", 2024-09-13T14:59:34.725 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.725 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.725 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:34.725 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.522270+0000", 2024-09-13T14:59:34.725 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.725 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.725 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:34.726 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.522269+0000", 2024-09-13T14:59:34.726 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:59:34.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.726 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.726 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:34.726 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.522277+0000", 2024-09-13T14:59:34.726 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.5159999999999997e-06 2024-09-13T14:59:34.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.726 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.726 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:34.726 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.522292+0000", 2024-09-13T14:59:34.726 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4927e-05 2024-09-13T14:59:34.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.726 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.727 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:34.727 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.522297+0000", 2024-09-13T14:59:34.727 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4809999999999997e-06 2024-09-13T14:59:34.727 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.727 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.727 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.727 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.522335+0000", 2024-09-13T14:59:34.727 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.7545000000000001e-05 2024-09-13T14:59:34.727 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.727 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.727 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:34.727 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.522341+0000", 2024-09-13T14:59:34.727 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.0460000000000002e-06 2024-09-13T14:59:34.727 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.728 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.728 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.728 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.517970+0000", 2024-09-13T14:59:34.728 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.99562924100000005 2024-09-13T14:59:34.728 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.728 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.728 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:34.728 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.518057+0000", 2024-09-13T14:59:34.728 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.6854000000000006e-05 2024-09-13T14:59:34.728 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.728 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.728 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:59:34.728 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.518128+0000", 2024-09-13T14:59:34.729 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.1171e-05 2024-09-13T14:59:34.729 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.729 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.729 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:34.729 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.529279+0000", 2024-09-13T14:59:34.729 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.011150764000000001 2024-09-13T14:59:34.729 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.729 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.729 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.729 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549331+0000", 2024-09-13T14:59:34.729 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.020052303 2024-09-13T14:59:34.729 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.729 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.729 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.730 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.602480+0000", 2024-09-13T14:59:34.730 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.053148753999999999 2024-09-13T14:59:34.730 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.730 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.730 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:34.730 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.602524+0000", 2024-09-13T14:59:34.730 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4388000000000003e-05 2024-09-13T14:59:34.730 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.730 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.730 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:34.730 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.602624+0000", 2024-09-13T14:59:34.730 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.9468000000000004e-05 2024-09-13T14:59:34.730 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.730 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:34.730 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.731 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.731 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.731 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176153 8.5 8:aba5861e:::benchmark_data_smithi151_229097_object51285:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1186)", 2024-09-13T14:59:34.731 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.525003+0000", 2024-09-13T14:59:34.731 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 448.07888614000001, 2024-09-13T14:59:34.731 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0909296319999999, 2024-09-13T14:59:34.731 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:34.731 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:34.731 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:34.731 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:59:34.731 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:59:34.731 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176153 2024-09-13T14:59:34.731 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.731 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:34.732 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.732 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:34.732 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.525003+0000", 2024-09-13T14:59:34.732 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.732 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.732 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.732 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:34.732 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.525003+0000", 2024-09-13T14:59:34.732 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.732 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.732 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.732 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:34.732 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.525001+0000", 2024-09-13T14:59:34.732 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:59:34.733 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.733 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.733 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:34.733 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.525009+0000", 2024-09-13T14:59:34.733 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.7230000000000004e-06 2024-09-13T14:59:34.733 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.733 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.733 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:34.733 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.525028+0000", 2024-09-13T14:59:34.733 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.8882999999999999e-05 2024-09-13T14:59:34.733 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.733 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.733 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:34.733 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.525034+0000", 2024-09-13T14:59:34.734 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.8020000000000003e-06 2024-09-13T14:59:34.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.734 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.734 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.734 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.525084+0000", 2024-09-13T14:59:34.734 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0086999999999997e-05 2024-09-13T14:59:34.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.734 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.734 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:34.734 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.525091+0000", 2024-09-13T14:59:34.734 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.6789999999999999e-06 2024-09-13T14:59:34.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.734 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.734 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.734 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.519980+0000", 2024-09-13T14:59:34.735 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.99488926600000005 2024-09-13T14:59:34.735 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.735 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.735 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:34.735 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.520088+0000", 2024-09-13T14:59:34.735 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00010715800000000001 2024-09-13T14:59:34.735 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.735 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.735 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:59:34.735 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.520221+0000", 2024-09-13T14:59:34.735 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013349799999999999 2024-09-13T14:59:34.735 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.735 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.735 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:34.736 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.529287+0000", 2024-09-13T14:59:34.736 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0090658609999999997 2024-09-13T14:59:34.736 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.736 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.736 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.736 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549357+0000", 2024-09-13T14:59:34.736 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.020070523999999999 2024-09-13T14:59:34.736 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.736 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.736 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.736 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.615738+0000", 2024-09-13T14:59:34.736 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.066380272000000004 2024-09-13T14:59:34.736 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.736 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.737 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:34.737 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.615801+0000", 2024-09-13T14:59:34.737 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3355999999999994e-05 2024-09-13T14:59:34.737 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.737 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.737 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:34.737 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.615933+0000", 2024-09-13T14:59:34.737 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013155099999999999 2024-09-13T14:59:34.737 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.737 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:34.737 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.737 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.737 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.738 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176156 8.5 8:ae72b766:::benchmark_data_smithi151_229097_object51288:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1186)", 2024-09-13T14:59:34.738 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.527548+0000", 2024-09-13T14:59:34.738 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 448.07634111900001, 2024-09-13T14:59:34.738 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0885104029999999, 2024-09-13T14:59:34.738 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:34.738 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:34.738 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:34.738 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:59:34.738 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:59:34.738 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176156 2024-09-13T14:59:34.738 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.738 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:34.738 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.738 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:34.739 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.527548+0000", 2024-09-13T14:59:34.739 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.739 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.739 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.739 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:34.739 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.527548+0000", 2024-09-13T14:59:34.739 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.739 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.739 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.739 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:34.739 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.527547+0000", 2024-09-13T14:59:34.739 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:34.739 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.739 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.740 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:34.740 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.527553+0000", 2024-09-13T14:59:34.740 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.601e-06 2024-09-13T14:59:34.740 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.740 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.740 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:34.740 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.527571+0000", 2024-09-13T14:59:34.740 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.8229000000000001e-05 2024-09-13T14:59:34.740 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.740 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.740 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:34.740 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.527578+0000", 2024-09-13T14:59:34.740 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.6889999999999998e-06 2024-09-13T14:59:34.740 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.741 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.741 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.741 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.527637+0000", 2024-09-13T14:59:34.741 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8743000000000003e-05 2024-09-13T14:59:34.741 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.741 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.741 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:34.741 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.527644+0000", 2024-09-13T14:59:34.741 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.2060000000000001e-06 2024-09-13T14:59:34.741 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.741 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.741 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.741 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.522614+0000", 2024-09-13T14:59:34.742 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.99496983999999999 2024-09-13T14:59:34.742 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.742 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.742 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:34.742 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.522740+0000", 2024-09-13T14:59:34.742 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012576699999999999 2024-09-13T14:59:34.742 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.742 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.742 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:59:34.742 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.522822+0000", 2024-09-13T14:59:34.742 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.2529999999999998e-05 2024-09-13T14:59:34.742 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.742 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.742 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:34.742 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.533089+0000", 2024-09-13T14:59:34.743 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.010266688 2024-09-13T14:59:34.743 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.743 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.743 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.743 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549385+0000", 2024-09-13T14:59:34.743 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.016296496000000001 2024-09-13T14:59:34.743 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.743 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.743 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.743 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.615972+0000", 2024-09-13T14:59:34.743 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.066586874000000004 2024-09-13T14:59:34.743 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.743 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.743 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:34.744 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.616005+0000", 2024-09-13T14:59:34.744 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3263999999999999e-05 2024-09-13T14:59:34.744 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.744 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.744 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:34.744 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.616058+0000", 2024-09-13T14:59:34.744 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.2939000000000003e-05 2024-09-13T14:59:34.744 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.744 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:34.744 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.744 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.744 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.744 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176165 8.5 8:acd013d5:::benchmark_data_smithi151_229097_object51297:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1186)", 2024-09-13T14:59:34.745 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.537453+0000", 2024-09-13T14:59:34.745 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 448.06643559499997, 2024-09-13T14:59:34.745 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0843085750000001, 2024-09-13T14:59:34.745 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:34.745 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:34.745 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:34.745 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:59:34.745 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:59:34.745 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176165 2024-09-13T14:59:34.745 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.745 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:34.745 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.745 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:34.745 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.537453+0000", 2024-09-13T14:59:34.746 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.746 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.746 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.746 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:34.746 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.537453+0000", 2024-09-13T14:59:34.746 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.746 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.746 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.746 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:34.746 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.537452+0000", 2024-09-13T14:59:34.746 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:34.746 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.746 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.746 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:34.747 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.537459+0000", 2024-09-13T14:59:34.747 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7510000000000002e-06 2024-09-13T14:59:34.747 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.747 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.747 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:34.747 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.537469+0000", 2024-09-13T14:59:34.747 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.9890000000000004e-06 2024-09-13T14:59:34.747 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.747 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.747 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:34.747 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.537473+0000", 2024-09-13T14:59:34.747 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4889999999999997e-06 2024-09-13T14:59:34.747 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.747 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.748 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.748 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.537508+0000", 2024-09-13T14:59:34.748 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.4484999999999999e-05 2024-09-13T14:59:34.748 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.748 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.748 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:34.748 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.537513+0000", 2024-09-13T14:59:34.748 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7309999999999998e-06 2024-09-13T14:59:34.748 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.748 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.748 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.748 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.524669+0000", 2024-09-13T14:59:34.748 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.98715564700000002 2024-09-13T14:59:34.749 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.749 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.749 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:34.749 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.524783+0000", 2024-09-13T14:59:34.749 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00011410599999999999 2024-09-13T14:59:34.749 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.749 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.749 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:59:34.749 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.524910+0000", 2024-09-13T14:59:34.749 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000126747 2024-09-13T14:59:34.749 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.749 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.749 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:34.749 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.533097+0000", 2024-09-13T14:59:34.750 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0081872630000000002 2024-09-13T14:59:34.750 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.750 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.750 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.750 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549417+0000", 2024-09-13T14:59:34.750 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.016319314000000001 2024-09-13T14:59:34.750 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.750 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.750 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.750 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.621591+0000", 2024-09-13T14:59:34.750 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.072174490999999993 2024-09-13T14:59:34.750 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.750 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.750 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:34.750 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.621653+0000", 2024-09-13T14:59:34.751 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.2175999999999996e-05 2024-09-13T14:59:34.751 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.751 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.751 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:34.751 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.621762+0000", 2024-09-13T14:59:34.751 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000108867 2024-09-13T14:59:34.751 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.751 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:34.751 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.751 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.751 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.751 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176168 8.5 8:a528538c:::benchmark_data_smithi151_229097_object51300:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1186)", 2024-09-13T14:59:34.751 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.538334+0000", 2024-09-13T14:59:34.752 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 448.06555492699999, 2024-09-13T14:59:34.752 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0836694440000001, 2024-09-13T14:59:34.752 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:34.752 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:34.752 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:34.752 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:59:34.752 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:59:34.752 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176168 2024-09-13T14:59:34.752 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.752 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:34.752 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.752 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:34.752 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.538334+0000", 2024-09-13T14:59:34.752 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.753 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.753 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.753 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:34.753 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.538334+0000", 2024-09-13T14:59:34.753 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.753 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.753 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.753 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:34.753 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.538334+0000", 2024-09-13T14:59:34.753 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:59:34.753 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.753 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.753 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:34.754 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.538337+0000", 2024-09-13T14:59:34.754 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8879999999999999e-06 2024-09-13T14:59:34.754 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.754 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.754 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:34.754 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.538348+0000", 2024-09-13T14:59:34.754 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.095e-05 2024-09-13T14:59:34.754 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.754 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.754 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:34.754 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.538352+0000", 2024-09-13T14:59:34.754 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.4000000000000001e-06 2024-09-13T14:59:34.754 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.754 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.755 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.755 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.538384+0000", 2024-09-13T14:59:34.755 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.1996000000000003e-05 2024-09-13T14:59:34.755 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.755 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.755 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:34.755 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.538389+0000", 2024-09-13T14:59:34.755 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8849999999999998e-06 2024-09-13T14:59:34.755 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.755 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.755 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.755 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.527222+0000", 2024-09-13T14:59:34.755 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.98883309200000002 2024-09-13T14:59:34.755 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.756 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.756 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:34.756 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.527302+0000", 2024-09-13T14:59:34.756 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0692000000000006e-05 2024-09-13T14:59:34.756 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.756 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.756 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:59:34.756 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.527381+0000", 2024-09-13T14:59:34.756 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.8848000000000001e-05 2024-09-13T14:59:34.756 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.756 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.756 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:34.756 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.536903+0000", 2024-09-13T14:59:34.756 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0095215600000000001 2024-09-13T14:59:34.757 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.757 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.757 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.757 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549447+0000", 2024-09-13T14:59:34.757 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.012544092 2024-09-13T14:59:34.757 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.757 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.757 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.757 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.621827+0000", 2024-09-13T14:59:34.757 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.072379599000000003 2024-09-13T14:59:34.757 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.757 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.757 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:34.757 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.621888+0000", 2024-09-13T14:59:34.758 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1741000000000004e-05 2024-09-13T14:59:34.758 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.758 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.758 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:34.758 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.622004+0000", 2024-09-13T14:59:34.758 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0001153 2024-09-13T14:59:34.758 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.758 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:34.758 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.758 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.758 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.758 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176178 8.5 8:a790fc8b:::benchmark_data_smithi151_229097_object51310:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1186)", 2024-09-13T14:59:34.758 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.549688+0000", 2024-09-13T14:59:34.758 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 448.05420080900001, 2024-09-13T14:59:34.758 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0725186920000001, 2024-09-13T14:59:34.759 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:34.759 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:34.759 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:34.759 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:59:34.759 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:59:34.759 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176178 2024-09-13T14:59:34.759 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.759 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:34.759 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.759 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:34.759 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.549688+0000", 2024-09-13T14:59:34.759 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.759 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.759 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.760 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:34.760 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.549688+0000", 2024-09-13T14:59:34.760 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.760 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.760 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.760 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:34.760 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.549687+0000", 2024-09-13T14:59:34.760 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:34.760 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.760 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.760 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:34.760 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.549692+0000", 2024-09-13T14:59:34.760 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5300000000000004e-06 2024-09-13T14:59:34.760 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.761 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.761 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:34.761 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.549703+0000", 2024-09-13T14:59:34.761 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0339999999999999e-05 2024-09-13T14:59:34.761 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.761 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.761 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:34.761 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.549707+0000", 2024-09-13T14:59:34.761 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.6759999999999998e-06 2024-09-13T14:59:34.761 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.761 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.761 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.761 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.549753+0000", 2024-09-13T14:59:34.761 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5949999999999999e-05 2024-09-13T14:59:34.761 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.762 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.762 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:34.762 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.549762+0000", 2024-09-13T14:59:34.762 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.2309999999999996e-06 2024-09-13T14:59:34.762 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.762 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.762 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.762 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.529331+0000", 2024-09-13T14:59:34.762 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.97956943299999999 2024-09-13T14:59:34.762 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.762 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.762 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:34.762 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.529426+0000", 2024-09-13T14:59:34.762 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.4649999999999997e-05 2024-09-13T14:59:34.763 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.763 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.763 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:59:34.763 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.529529+0000", 2024-09-13T14:59:34.763 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000103084 2024-09-13T14:59:34.763 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.763 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.763 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:34.763 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.536911+0000", 2024-09-13T14:59:34.763 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0073820359999999998 2024-09-13T14:59:34.763 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.763 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.763 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.763 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549476+0000", 2024-09-13T14:59:34.764 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.012565270999999999 2024-09-13T14:59:34.764 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.764 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.764 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.764 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.622062+0000", 2024-09-13T14:59:34.764 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.072586176000000002 2024-09-13T14:59:34.764 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.764 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.764 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:34.764 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.622112+0000", 2024-09-13T14:59:34.764 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.986e-05 2024-09-13T14:59:34.764 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.764 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.765 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:34.765 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.622207+0000", 2024-09-13T14:59:34.765 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.4876000000000004e-05 2024-09-13T14:59:34.765 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.765 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:34.765 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.765 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.765 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.765 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176217 8.5 8:a0066b24:::benchmark_data_smithi151_229097_object51349:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1186)", 2024-09-13T14:59:34.765 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.618234+0000", 2024-09-13T14:59:34.766 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 447.98565520699998, 2024-09-13T14:59:34.766 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.004108939, 2024-09-13T14:59:34.766 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:34.766 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:34.766 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:34.766 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:59:34.766 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:59:34.766 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176217 2024-09-13T14:59:34.766 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.766 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:34.766 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.766 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:34.766 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618234+0000", 2024-09-13T14:59:34.767 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.767 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.767 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.767 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:34.767 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618234+0000", 2024-09-13T14:59:34.767 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.767 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.767 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.767 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:34.767 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618232+0000", 2024-09-13T14:59:34.767 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:34.767 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.767 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.768 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:34.768 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618240+0000", 2024-09-13T14:59:34.768 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0639999999999994e-06 2024-09-13T14:59:34.768 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.768 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.768 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:34.768 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618252+0000", 2024-09-13T14:59:34.768 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1773000000000001e-05 2024-09-13T14:59:34.768 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.768 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.768 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:34.768 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618257+0000", 2024-09-13T14:59:34.768 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7820000000000001e-06 2024-09-13T14:59:34.768 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.769 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.769 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.769 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618302+0000", 2024-09-13T14:59:34.769 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5296999999999997e-05 2024-09-13T14:59:34.769 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.769 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.769 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:34.769 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618311+0000", 2024-09-13T14:59:34.769 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.7839999999999992e-06 2024-09-13T14:59:34.769 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.769 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.769 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.769 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.531326+0000", 2024-09-13T14:59:34.770 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.913014624 2024-09-13T14:59:34.770 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.770 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.770 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:34.770 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.531393+0000", 2024-09-13T14:59:34.770 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7386999999999997e-05 2024-09-13T14:59:34.770 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.770 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.770 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:59:34.770 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.531460+0000", 2024-09-13T14:59:34.770 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7428000000000006e-05 2024-09-13T14:59:34.770 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.770 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.770 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:34.771 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.540413+0000", 2024-09-13T14:59:34.771 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0089529420000000002 2024-09-13T14:59:34.771 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.771 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.771 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.771 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549505+0000", 2024-09-13T14:59:34.771 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0090913369999999997 2024-09-13T14:59:34.771 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.771 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.771 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.771 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.622257+0000", 2024-09-13T14:59:34.771 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.072752467000000001 2024-09-13T14:59:34.771 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.771 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.772 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:34.772 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.622287+0000", 2024-09-13T14:59:34.772 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.9539e-05 2024-09-13T14:59:34.772 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.772 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.772 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:34.772 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.622343+0000", 2024-09-13T14:59:34.772 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6140000000000001e-05 2024-09-13T14:59:34.772 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.772 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:34.772 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.772 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.772 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.772 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176218 8.5 8:a137a2c8:::benchmark_data_smithi151_229097_object51350:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1186)", 2024-09-13T14:59:34.773 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.618308+0000", 2024-09-13T14:59:34.773 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 447.98558070799999, 2024-09-13T14:59:34.773 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0269271740000001, 2024-09-13T14:59:34.773 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:34.773 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:34.773 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:34.773 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:59:34.773 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:59:34.773 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176218 2024-09-13T14:59:34.773 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.773 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:34.773 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.773 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:34.773 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618308+0000", 2024-09-13T14:59:34.774 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.774 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.774 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.774 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:34.774 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618308+0000", 2024-09-13T14:59:34.774 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.774 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.774 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.774 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:34.774 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618308+0000", 2024-09-13T14:59:34.774 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T14:59:34.774 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.774 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.774 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:34.775 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618311+0000", 2024-09-13T14:59:34.775 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.723e-06 2024-09-13T14:59:34.775 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.775 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.775 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:34.775 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618318+0000", 2024-09-13T14:59:34.775 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.1470000000000004e-06 2024-09-13T14:59:34.775 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.775 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.775 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:34.775 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618321+0000", 2024-09-13T14:59:34.775 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.0489999999999999e-06 2024-09-13T14:59:34.775 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.775 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.775 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.776 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618360+0000", 2024-09-13T14:59:34.776 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8761999999999999e-05 2024-09-13T14:59:34.776 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.776 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.776 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:34.776 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.618366+0000", 2024-09-13T14:59:34.776 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1059999999999996e-06 2024-09-13T14:59:34.776 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.776 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.776 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.777 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.533123+0000", 2024-09-13T14:59:34.777 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.91475721600000004 2024-09-13T14:59:34.777 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.777 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.777 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:34.777 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.533193+0000", 2024-09-13T14:59:34.777 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.9752999999999994e-05 2024-09-13T14:59:34.777 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.777 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.777 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:59:34.777 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.533274+0000", 2024-09-13T14:59:34.777 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0891999999999997e-05 2024-09-13T14:59:34.777 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.777 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.778 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:34.778 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.540421+0000", 2024-09-13T14:59:34.778 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0071469419999999999 2024-09-13T14:59:34.778 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.778 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.778 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.778 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549533+0000", 2024-09-13T14:59:34.778 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0091122310000000002 2024-09-13T14:59:34.778 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.778 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.778 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.778 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.644975+0000", 2024-09-13T14:59:34.778 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.095442461000000006 2024-09-13T14:59:34.778 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.779 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.779 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:34.779 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.645120+0000", 2024-09-13T14:59:34.779 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000144647 2024-09-13T14:59:34.779 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.779 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.779 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:34.779 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.645236+0000", 2024-09-13T14:59:34.779 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000115633 2024-09-13T14:59:34.779 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.779 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:34.779 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.779 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.779 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.780 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176222 8.5 8:aefbc882:::benchmark_data_smithi151_229097_object51354:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1186)", 2024-09-13T14:59:34.780 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.621980+0000", 2024-09-13T14:59:34.780 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 447.981909562, 2024-09-13T14:59:34.780 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.023445849, 2024-09-13T14:59:34.780 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:34.780 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:34.780 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:34.780 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:59:34.780 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:59:34.780 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176222 2024-09-13T14:59:34.780 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.780 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:34.780 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.780 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:34.781 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.621980+0000", 2024-09-13T14:59:34.781 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.781 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.781 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.781 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:34.781 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.621980+0000", 2024-09-13T14:59:34.781 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.781 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.781 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.781 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:34.781 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.621978+0000", 2024-09-13T14:59:34.781 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:59:34.781 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.781 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.782 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:34.782 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.621986+0000", 2024-09-13T14:59:34.782 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.7589999999999997e-06 2024-09-13T14:59:34.782 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.782 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.782 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:34.782 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.621997+0000", 2024-09-13T14:59:34.782 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1202e-05 2024-09-13T14:59:34.782 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.782 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.782 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:34.782 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.622002+0000", 2024-09-13T14:59:34.782 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3880000000000002e-06 2024-09-13T14:59:34.782 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.783 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.783 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.783 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.622034+0000", 2024-09-13T14:59:34.783 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.2437000000000001e-05 2024-09-13T14:59:34.783 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.783 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.783 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:34.783 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.622039+0000", 2024-09-13T14:59:34.783 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.6739999999999996e-06 2024-09-13T14:59:34.783 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.783 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.783 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.783 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.535015+0000", 2024-09-13T14:59:34.784 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.91297645500000002 2024-09-13T14:59:34.784 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.784 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.784 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:34.784 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.535097+0000", 2024-09-13T14:59:34.784 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.1742000000000004e-05 2024-09-13T14:59:34.784 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.784 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.784 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:59:34.784 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.535208+0000", 2024-09-13T14:59:34.784 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000110535 2024-09-13T14:59:34.784 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.784 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.784 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:34.785 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.548717+0000", 2024-09-13T14:59:34.785 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.013509241999999999 2024-09-13T14:59:34.785 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.785 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.785 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.785 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549562+0000", 2024-09-13T14:59:34.785 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000845245 2024-09-13T14:59:34.785 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.785 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.785 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.785 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.645261+0000", 2024-09-13T14:59:34.785 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.095698762000000007 2024-09-13T14:59:34.785 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.785 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.786 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:34.786 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.645330+0000", 2024-09-13T14:59:34.786 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.9472000000000002e-05 2024-09-13T14:59:34.786 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.786 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.786 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:34.786 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.645425+0000", 2024-09-13T14:59:34.786 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.4977999999999998e-05 2024-09-13T14:59:34.786 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.786 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:34.786 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.786 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.786 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.786 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176237 8.5 8:ada8226f:::benchmark_data_smithi151_229097_object51369:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1186)", 2024-09-13T14:59:34.787 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.781570+0000", 2024-09-13T14:59:34.787 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 447.82231916000001, 2024-09-13T14:59:34.787 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.86405514800000005, 2024-09-13T14:59:34.787 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:34.787 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:34.788 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:34.789 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:59:34.789 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:59:34.789 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176237 2024-09-13T14:59:34.789 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.789 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:34.789 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.789 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:34.789 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.781570+0000", 2024-09-13T14:59:34.789 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.789 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.789 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.789 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:34.789 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.781570+0000", 2024-09-13T14:59:34.790 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.790 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.790 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.790 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:34.790 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.781567+0000", 2024-09-13T14:59:34.790 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999971 2024-09-13T14:59:34.790 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.790 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.790 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:34.790 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.781584+0000", 2024-09-13T14:59:34.790 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.7093000000000001e-05 2024-09-13T14:59:34.790 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.790 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.790 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:34.791 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.781624+0000", 2024-09-13T14:59:34.791 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0039000000000001e-05 2024-09-13T14:59:34.791 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.791 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.791 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:34.791 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.781639+0000", 2024-09-13T14:59:34.791 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4392999999999999e-05 2024-09-13T14:59:34.791 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.791 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.791 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.791 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.781740+0000", 2024-09-13T14:59:34.791 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000101418 2024-09-13T14:59:34.791 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.791 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.791 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:34.792 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.781755+0000", 2024-09-13T14:59:34.792 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4776e-05 2024-09-13T14:59:34.792 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.792 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.792 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.792 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.536949+0000", 2024-09-13T14:59:34.792 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.75519378100000001 2024-09-13T14:59:34.792 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.792 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.792 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:34.792 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.537045+0000", 2024-09-13T14:59:34.792 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.6409000000000005e-05 2024-09-13T14:59:34.792 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.792 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.793 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:59:34.793 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.537125+0000", 2024-09-13T14:59:34.793 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0263e-05 2024-09-13T14:59:34.793 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.793 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.793 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:34.793 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.548721+0000", 2024-09-13T14:59:34.793 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.011595611 2024-09-13T14:59:34.793 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.793 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.793 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.793 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549590+0000", 2024-09-13T14:59:34.793 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00086873600000000005 2024-09-13T14:59:34.793 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.794 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.794 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.794 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.645464+0000", 2024-09-13T14:59:34.794 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.095874803999999994 2024-09-13T14:59:34.794 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.794 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.794 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:34.794 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.645533+0000", 2024-09-13T14:59:34.794 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.8547999999999995e-05 2024-09-13T14:59:34.794 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.794 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.794 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:34.794 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.645625+0000", 2024-09-13T14:59:34.795 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.2176999999999995e-05 2024-09-13T14:59:34.795 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.795 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:34.795 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.795 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.795 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.795 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176259 8.5 8:afd58153:::benchmark_data_smithi151_229097_object51391:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1186)", 2024-09-13T14:59:34.795 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.831139+0000", 2024-09-13T14:59:34.795 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 447.77274997900003, 2024-09-13T14:59:34.795 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.81465061999999999, 2024-09-13T14:59:34.795 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:34.795 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:34.795 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:34.796 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:59:34.796 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:59:34.796 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176259 2024-09-13T14:59:34.796 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.796 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:34.796 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.796 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:34.796 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.831139+0000", 2024-09-13T14:59:34.796 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.796 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.796 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.796 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:34.796 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.831139+0000", 2024-09-13T14:59:34.796 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.797 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.797 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.797 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:34.797 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.831138+0000", 2024-09-13T14:59:34.797 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:34.797 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.797 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.797 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:34.797 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.831149+0000", 2024-09-13T14:59:34.797 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1037e-05 2024-09-13T14:59:34.797 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.797 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.797 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:34.798 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.831161+0000", 2024-09-13T14:59:34.798 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1969e-05 2024-09-13T14:59:34.798 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.798 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.798 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:34.798 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.831165+0000", 2024-09-13T14:59:34.798 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5770000000000004e-06 2024-09-13T14:59:34.798 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.798 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.798 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.798 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.831207+0000", 2024-09-13T14:59:34.798 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.189e-05 2024-09-13T14:59:34.798 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.798 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.798 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:34.799 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.831216+0000", 2024-09-13T14:59:34.799 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.7090000000000008e-06 2024-09-13T14:59:34.799 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.799 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.799 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.799 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.538743+0000", 2024-09-13T14:59:34.799 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.70752717899999995 2024-09-13T14:59:34.799 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.799 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.799 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:34.799 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.538861+0000", 2024-09-13T14:59:34.799 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00011845699999999999 2024-09-13T14:59:34.799 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.799 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.800 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:59:34.800 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.538998+0000", 2024-09-13T14:59:34.800 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013685999999999999 2024-09-13T14:59:34.800 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.800 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.800 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:34.800 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.548827+0000", 2024-09-13T14:59:34.800 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0098285239999999999 2024-09-13T14:59:34.800 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.800 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.800 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.800 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549617+0000", 2024-09-13T14:59:34.800 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00079065899999999998 2024-09-13T14:59:34.800 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.801 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.801 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.801 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.645663+0000", 2024-09-13T14:59:34.801 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.096045084000000003 2024-09-13T14:59:34.801 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.801 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.801 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:34.801 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.645708+0000", 2024-09-13T14:59:34.801 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5215e-05 2024-09-13T14:59:34.801 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.801 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.801 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:34.801 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.645790+0000", 2024-09-13T14:59:34.801 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.1987e-05 2024-09-13T14:59:34.802 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.802 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:34.802 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.802 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.802 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.802 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.35940.0:176290 8.5 8:af9a2a07:::benchmark_data_smithi151_229097_object51422:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1186)", 2024-09-13T14:59:34.802 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:52:06.946125+0000", 2024-09-13T14:59:34.802 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 447.65776430099999, 2024-09-13T14:59:34.802 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.69983572299999997, 2024-09-13T14:59:34.802 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:34.802 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:34.802 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:34.802 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.35940", 2024-09-13T14:59:34.802 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T14:59:34.802 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 176290 2024-09-13T14:59:34.803 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.803 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:34.803 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.803 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:34.803 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.946125+0000", 2024-09-13T14:59:34.803 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.803 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.803 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.803 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:34.803 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.946125+0000", 2024-09-13T14:59:34.803 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.803 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.803 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.803 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:34.804 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.946124+0000", 2024-09-13T14:59:34.804 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:34.804 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.804 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.804 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:34.804 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.946130+0000", 2024-09-13T14:59:34.804 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.5989999999999998e-06 2024-09-13T14:59:34.804 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.804 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.804 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:34.804 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.946142+0000", 2024-09-13T14:59:34.804 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2257e-05 2024-09-13T14:59:34.804 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.804 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.805 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:34.805 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.946147+0000", 2024-09-13T14:59:34.805 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4100000000000001e-06 2024-09-13T14:59:34.805 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.805 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.805 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.805 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.946184+0000", 2024-09-13T14:59:34.805 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.7633999999999999e-05 2024-09-13T14:59:34.805 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.805 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.805 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T14:59:34.805 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:06.946191+0000", 2024-09-13T14:59:34.805 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.652e-06 2024-09-13T14:59:34.805 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.805 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.806 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.806 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.540453+0000", 2024-09-13T14:59:34.806 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.59426235999999999 2024-09-13T14:59:34.806 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.806 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.806 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:34.806 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.540521+0000", 2024-09-13T14:59:34.806 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7968999999999999e-05 2024-09-13T14:59:34.806 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.806 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.806 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:59:34.806 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.540613+0000", 2024-09-13T14:59:34.806 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.1310000000000005e-05 2024-09-13T14:59:34.806 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.807 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.807 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:34.807 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.548831+0000", 2024-09-13T14:59:34.807 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.008218108 2024-09-13T14:59:34.807 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.807 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.807 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.807 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.549644+0000", 2024-09-13T14:59:34.807 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00081339800000000001 2024-09-13T14:59:34.807 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.807 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.807 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.807 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.645820+0000", 2024-09-13T14:59:34.807 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.096175694000000006 2024-09-13T14:59:34.808 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.808 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.808 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:34.808 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.645856+0000", 2024-09-13T14:59:34.808 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.6072999999999998e-05 2024-09-13T14:59:34.808 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.808 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.808 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:34.808 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:52:07.645961+0000", 2024-09-13T14:59:34.808 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000104467 2024-09-13T14:59:34.808 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.808 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:34.808 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.808 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.809 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.809 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:22917 10.f 10:f03fdad5:::benchmark_data_smithi151_288181_object22916:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1273)", 2024-09-13T14:59:34.809 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:58:25.085696+0000", 2024-09-13T14:59:34.809 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 69.518192776000006, 2024-09-13T14:59:34.809 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.23906908900000001, 2024-09-13T14:59:34.809 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:34.809 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:34.809 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:34.809 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T14:59:34.809 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T14:59:34.809 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 22917 2024-09-13T14:59:34.809 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.809 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:34.809 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.810 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:34.810 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.085696+0000", 2024-09-13T14:59:34.810 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.810 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.810 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.810 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:34.810 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.085696+0000", 2024-09-13T14:59:34.810 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.810 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.810 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.810 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:34.810 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.085695+0000", 2024-09-13T14:59:34.810 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:34.810 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.811 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.811 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:34.811 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.085730+0000", 2024-09-13T14:59:34.811 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.5237000000000002e-05 2024-09-13T14:59:34.811 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.811 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.811 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:34.811 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.085779+0000", 2024-09-13T14:59:34.811 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9027999999999999e-05 2024-09-13T14:59:34.811 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.811 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.811 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:34.811 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.085803+0000", 2024-09-13T14:59:34.811 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.4006999999999999e-05 2024-09-13T14:59:34.812 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.812 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.812 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.812 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.085856+0000", 2024-09-13T14:59:34.812 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.2911999999999998e-05 2024-09-13T14:59:34.812 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.812 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.812 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:34.812 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.085976+0000", 2024-09-13T14:59:34.812 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012017 2024-09-13T14:59:34.812 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.812 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.812 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:59:34.812 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.086109+0000", 2024-09-13T14:59:34.813 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013235499999999999 2024-09-13T14:59:34.813 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.813 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.813 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:34.813 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.087815+0000", 2024-09-13T14:59:34.813 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0017059810000000001 2024-09-13T14:59:34.813 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.813 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.813 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.814 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.088070+0000", 2024-09-13T14:59:34.814 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00025514400000000001 2024-09-13T14:59:34.814 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.814 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.814 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.814 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.324668+0000", 2024-09-13T14:59:34.814 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.23659777400000001 2024-09-13T14:59:34.814 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.814 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.814 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:34.814 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.324725+0000", 2024-09-13T14:59:34.814 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7327000000000002e-05 2024-09-13T14:59:34.814 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.815 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.815 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:34.815 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.324765+0000", 2024-09-13T14:59:34.815 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0467999999999999e-05 2024-09-13T14:59:34.815 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.815 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:34.815 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.815 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.815 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.815 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:22929 10.f 10:f8fdbdee:::benchmark_data_smithi151_288181_object22928:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1273)", 2024-09-13T14:59:34.815 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:58:25.089916+0000", 2024-09-13T14:59:34.815 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 69.513973433999993, 2024-09-13T14:59:34.815 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.235597062, 2024-09-13T14:59:34.815 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:34.816 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:34.816 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:34.816 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T14:59:34.816 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T14:59:34.816 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 22929 2024-09-13T14:59:34.816 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.816 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:34.816 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.816 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:34.816 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.089916+0000", 2024-09-13T14:59:34.816 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.816 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.816 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.816 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:34.817 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.089916+0000", 2024-09-13T14:59:34.817 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.817 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.817 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.817 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:34.817 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.089914+0000", 2024-09-13T14:59:34.817 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T14:59:34.817 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.817 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.817 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:34.817 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.089947+0000", 2024-09-13T14:59:34.817 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3559999999999997e-05 2024-09-13T14:59:34.817 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.817 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.818 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:34.818 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.089965+0000", 2024-09-13T14:59:34.818 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.8085999999999999e-05 2024-09-13T14:59:34.818 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.818 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.818 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:34.818 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.089972+0000", 2024-09-13T14:59:34.818 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.8519999999999998e-06 2024-09-13T14:59:34.818 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.818 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.818 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.818 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.090021+0000", 2024-09-13T14:59:34.818 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8909000000000002e-05 2024-09-13T14:59:34.818 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.819 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.819 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:34.819 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.090138+0000", 2024-09-13T14:59:34.819 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000116902 2024-09-13T14:59:34.819 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.819 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.819 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:59:34.819 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.090271+0000", 2024-09-13T14:59:34.819 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013331700000000001 2024-09-13T14:59:34.819 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.819 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.819 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.819 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.091905+0000", 2024-09-13T14:59:34.819 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001633926 2024-09-13T14:59:34.819 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.820 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.820 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:34.820 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.092166+0000", 2024-09-13T14:59:34.820 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00026091299999999999 2024-09-13T14:59:34.820 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.820 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.820 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.820 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.325435+0000", 2024-09-13T14:59:34.820 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.23326865599999999 2024-09-13T14:59:34.820 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.820 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.820 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:34.820 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.325479+0000", 2024-09-13T14:59:34.821 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3795000000000003e-05 2024-09-13T14:59:34.821 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.821 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.821 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:34.821 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.325513+0000", 2024-09-13T14:59:34.821 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3983999999999999e-05 2024-09-13T14:59:34.821 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.821 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:34.821 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.821 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.821 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.821 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:22938 10.f 10:f5c17348:::benchmark_data_smithi151_288181_object22937:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1273)", 2024-09-13T14:59:34.821 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:58:25.092672+0000", 2024-09-13T14:59:34.821 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 69.511217060000007, 2024-09-13T14:59:34.822 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.235104329, 2024-09-13T14:59:34.822 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:34.822 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:34.822 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:34.822 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T14:59:34.822 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T14:59:34.822 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 22938 2024-09-13T14:59:34.822 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.822 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:34.822 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.822 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:34.822 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.092672+0000", 2024-09-13T14:59:34.822 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.822 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.823 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.823 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:34.823 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.092672+0000", 2024-09-13T14:59:34.823 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.823 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.823 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.823 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:34.823 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.092671+0000", 2024-09-13T14:59:34.823 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:59:34.823 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.823 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.823 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:34.823 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.092708+0000", 2024-09-13T14:59:34.823 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.7457999999999999e-05 2024-09-13T14:59:34.824 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.824 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.824 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:34.824 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.092724+0000", 2024-09-13T14:59:34.824 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.5974999999999999e-05 2024-09-13T14:59:34.824 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.824 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.824 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:34.824 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.092730+0000", 2024-09-13T14:59:34.824 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.835e-06 2024-09-13T14:59:34.824 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.824 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.824 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.824 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.093512+0000", 2024-09-13T14:59:34.825 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00078151000000000002 2024-09-13T14:59:34.825 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.825 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.825 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:34.825 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.093664+0000", 2024-09-13T14:59:34.825 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00015232700000000001 2024-09-13T14:59:34.825 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.825 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.825 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:59:34.825 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.093793+0000", 2024-09-13T14:59:34.825 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012876200000000001 2024-09-13T14:59:34.825 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.825 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.825 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:34.826 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.094933+0000", 2024-09-13T14:59:34.826 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0011403470000000001 2024-09-13T14:59:34.826 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.826 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.826 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.826 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.095172+0000", 2024-09-13T14:59:34.826 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000238915 2024-09-13T14:59:34.826 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.826 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.826 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.826 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.327704+0000", 2024-09-13T14:59:34.826 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.232531548 2024-09-13T14:59:34.826 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.826 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.826 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:34.827 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.327744+0000", 2024-09-13T14:59:34.827 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0460999999999998e-05 2024-09-13T14:59:34.827 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.827 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.827 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:34.827 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.327776+0000", 2024-09-13T14:59:34.827 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.2200999999999999e-05 2024-09-13T14:59:34.827 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.827 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:34.827 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.827 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.827 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.827 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:22959 10.f 10:f324ddcf:::benchmark_data_smithi151_288181_object22958:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1273)", 2024-09-13T14:59:34.827 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:58:25.103251+0000", 2024-09-13T14:59:34.828 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 69.500638237999993, 2024-09-13T14:59:34.828 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.22474344900000001, 2024-09-13T14:59:34.828 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T14:59:34.828 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:34.828 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T14:59:34.828 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T14:59:34.828 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T14:59:34.828 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 22959 2024-09-13T14:59:34.828 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.828 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T14:59:34.828 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.828 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T14:59:34.828 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.103251+0000", 2024-09-13T14:59:34.828 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.829 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.829 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.829 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T14:59:34.829 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.103251+0000", 2024-09-13T14:59:34.829 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T14:59:34.829 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.829 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.829 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T14:59:34.829 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.103250+0000", 2024-09-13T14:59:34.829 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T14:59:34.829 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.829 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.829 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T14:59:34.829 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.103314+0000", 2024-09-13T14:59:34.830 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3723e-05 2024-09-13T14:59:34.830 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.830 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.830 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T14:59:34.830 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.103328+0000", 2024-09-13T14:59:34.830 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4384e-05 2024-09-13T14:59:34.830 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.830 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.830 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T14:59:34.830 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.103334+0000", 2024-09-13T14:59:34.830 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.48e-06 2024-09-13T14:59:34.830 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.830 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.830 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T14:59:34.831 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.103387+0000", 2024-09-13T14:59:34.831 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3486000000000003e-05 2024-09-13T14:59:34.831 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.831 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.831 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T14:59:34.831 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.103512+0000", 2024-09-13T14:59:34.831 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012517999999999999 2024-09-13T14:59:34.831 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.831 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.831 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:59:34.831 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.103643+0000", 2024-09-13T14:59:34.831 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000130717 2024-09-13T14:59:34.831 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.831 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.832 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T14:59:34.832 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.104645+0000", 2024-09-13T14:59:34.832 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0010017069999999999 2024-09-13T14:59:34.832 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.832 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.832 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.832 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.104898+0000", 2024-09-13T14:59:34.832 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00025336199999999998 2024-09-13T14:59:34.832 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.832 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.832 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:34.832 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.327836+0000", 2024-09-13T14:59:34.832 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.22293832499999999 2024-09-13T14:59:34.832 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.833 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.833 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T14:59:34.833 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.327949+0000", 2024-09-13T14:59:34.833 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000112658 2024-09-13T14:59:34.833 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T14:59:34.833 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T14:59:34.833 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T14:59:34.833 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:25.327994+0000", 2024-09-13T14:59:34.833 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5336000000000001e-05 2024-09-13T14:59:34.833 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.833 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:34.833 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.833 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T14:59:34.833 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T14:59:34.833 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:59:34.834 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:59:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:59:34] "GET /metrics HTTP/1.1" 200 36698 "" "Prometheus/2.51.0" 2024-09-13T14:59:34.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:34.835 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:34.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:34.936 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:34.972 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:34.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:34.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:35.074 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:35.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:34 smithi151 ceph-mon[108966]: pgmap v2088: 119 pgs: 119 active+clean; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T14:59:35.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:34 smithi151 ceph-mon[104450]: pgmap v2088: 119 pgs: 119 active+clean; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T14:59:35.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:35.118 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:35.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:35.203 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 78 16 114436 114420 91.6633 63.6875 0.0101008 0.0109044 2024-09-13T14:59:35.219 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:35.254 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:35.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:35.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:35.260 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:34 smithi184 ceph-mon[92527]: pgmap v2088: 119 pgs: 119 active+clean; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T14:59:35.356 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:35.382 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:35.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:35.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:35.484 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:35.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:35.528 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:35.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:35.629 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:35.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:35.682 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:35.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:35.783 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:35.790 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T14:59:35.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:35.813 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:35.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:35.914 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:35.932 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:35.979 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:35.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:35.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:36.081 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:36.131 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:36.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:36.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:36.202 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 79 16 115415 115399 91.2775 61.1875 0.00995775 0.0109507 2024-09-13T14:59:36.233 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:36.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:36.263 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:36.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:36.364 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:36.402 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:36.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:36.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:36.504 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:36.530 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:36.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:36.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:36.632 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:36.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:36.678 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:36.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:36.779 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:36.813 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:36.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:36.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:36.830 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:59:36.915 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:36.958 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:36.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:36.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:36.999 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-13T14:59:37.060 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:37.098 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:37.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:37.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:37.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:36 smithi184 ceph-mon[92527]: pgmap v2089: 119 pgs: 119 active+clean; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T14:59:37.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:36 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2005453406' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:59:37.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:36 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:59:37.199 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:37.202 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:2024-09-13T14:59:37.202451+0000 min lat: 0.00119391 max lat: 0.812283 avg lat: 0.0109978 2024-09-13T14:59:37.202 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:59:37.203 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 80 16 116376 116360 90.8873 60.0625 0.0319233 0.0109978 2024-09-13T14:59:37.227 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:37.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:37.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:37.329 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:37.340 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:36 smithi151 ceph-mon[104450]: pgmap v2089: 119 pgs: 119 active+clean; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T14:59:37.341 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:36 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2005453406' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:59:37.341 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:36 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:59:37.341 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:36 smithi151 ceph-mon[108966]: pgmap v2089: 119 pgs: 119 active+clean; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T14:59:37.341 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:36 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2005453406' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:59:37.341 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:36 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:59:37.374 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:37.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:37.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:37.476 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:37.508 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:37.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:37.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:37.580 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T14:59:37.580 INFO:teuthology.orchestra.run.smithi184.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T14:59:37.580 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T14:59:37.610 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:37.643 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:37.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:37.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:37.745 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:37.780 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:37.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:37.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:37.882 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:37.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:37.923 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:37.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:38.024 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:38.068 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:38.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:38.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:38.169 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:38.204 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 81 16 117327 117311 90.4979 59.4375 0.014943 0.011045 2024-09-13T14:59:38.209 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:38.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:38.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:38.253 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:37 smithi151 ceph-mon[104450]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:38.253 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:37 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:59:38.253 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:37 smithi151 ceph-mon[104450]: osdmap e1279: 8 total, 8 up, 8 in 2024-09-13T14:59:38.253 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:37 smithi151 ceph-mon[108966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:38.253 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:37 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:59:38.253 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:37 smithi151 ceph-mon[108966]: osdmap e1279: 8 total, 8 up, 8 in 2024-09-13T14:59:38.311 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:38.340 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:37 smithi184 ceph-mon[92527]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:38.341 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:37 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:59:38.341 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:37 smithi184 ceph-mon[92527]: osdmap e1279: 8 total, 8 up, 8 in 2024-09-13T14:59:38.349 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:38.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:38.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:38.451 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:38.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:38.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:38.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:38.596 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:38.635 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:38.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:38.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:38.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:38.775 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:38.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:38.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:38.839 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:59:38.840 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:59:38.877 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:38.879 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:59:38.879 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:59:38.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:38.912 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:38.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:38.919 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:59:38.919 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:59:38.958 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:59:38.958 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:59:38.991 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:59:38.992 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:59:39.013 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:39.025 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:59:39.025 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:59:39.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:39.046 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:39.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:39.058 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:59:39.059 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:59:39.087 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:59:39.087 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:59:39.147 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:39.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:39.194 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:39.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:39.204 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 82 16 118254 118238 90.1008 57.9375 0.00666855 0.011093 2024-09-13T14:59:39.296 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:39.310 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:38 smithi184 ceph-mon[92527]: pgmap v2091: 119 pgs: 119 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-13T14:59:39.311 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:38 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/460218799' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:39.339 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:39.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:39.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:39.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:38 smithi151 ceph-mon[104450]: pgmap v2091: 119 pgs: 119 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-13T14:59:39.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:38 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/460218799' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:39.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:38 smithi151 ceph-mon[108966]: pgmap v2091: 119 pgs: 119 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-13T14:59:39.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:38 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/460218799' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:39.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:39.481 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:39.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:39.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:39.583 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:39.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:39.620 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:39.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:39.721 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:39.753 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:59:39.754 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:59:39.758 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:39.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:39.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:39.860 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:39.881 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:39.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:39.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:39.983 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:40.015 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:40.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:40.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:40.106 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:59:40.117 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:40.195 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:40.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:40.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:40.204 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 83 16 119184 119168 89.7151 58.125 0.0229566 0.0111417 2024-09-13T14:59:40.297 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:40.327 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:40.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:40.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:40.430 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:40.465 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:40.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:40.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:40.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:40.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:40.588 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:40.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:40.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:59:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:59:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:59:40.689 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:40.723 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:40.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:40.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:40.825 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:40.855 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:40.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:40.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:40.957 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:40.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:40.994 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:40.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:41.095 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:41.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:41 smithi151 ceph-mon[104450]: pgmap v2092: 119 pgs: 119 active+clean; 7.1 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-13T14:59:41.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:41 smithi151 ceph-mon[108966]: pgmap v2092: 119 pgs: 119 active+clean; 7.1 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-13T14:59:41.117 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:41.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:41.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:41.204 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 84 16 120003 119987 89.2564 51.1875 0.0073528 0.0111972 2024-09-13T14:59:41.219 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:41.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:41.260 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:41.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:41.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:41 smithi184 ceph-mon[92527]: pgmap v2092: 119 pgs: 119 active+clean; 7.1 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-13T14:59:41.361 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:41.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:41.382 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:41.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:41.483 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:41.504 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:41.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:41.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:41.606 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:41.662 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:41.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:41.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:41.763 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:41.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:41.789 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:41.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:41.890 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:41.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:41.935 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:41.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:42.036 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:42.079 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:42.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:42.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:42.180 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:42.204 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 85 16 120755 120739 88.7593 47 0.0152948 0.0112615 2024-09-13T14:59:42.233 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:42.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:42.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:42.335 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:42.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:42.377 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:42.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:42.478 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:42.514 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:42.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:42.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:42.616 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:42.638 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:42.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:42.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:42.739 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:42.775 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:42.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:42.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:42.878 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:42.924 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:42.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:42.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:43.026 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:43.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:43.063 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:43.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:43.164 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:43.186 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:43.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:43.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:43.204 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 86 16 121589 121573 88.3333 52.125 0.0586376 0.011316 2024-09-13T14:59:43.288 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:43.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:43.314 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:43.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:43.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:43 smithi184 ceph-mon[92527]: pgmap v2093: 119 pgs: 119 active+clean; 7.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T14:59:43.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:43 smithi151 ceph-mon[104450]: pgmap v2093: 119 pgs: 119 active+clean; 7.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T14:59:43.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:43 smithi151 ceph-mon[108966]: pgmap v2093: 119 pgs: 119 active+clean; 7.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T14:59:43.415 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:43.456 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:43.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:43.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:43.558 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:43.591 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:43.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:43.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:43.693 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:43.727 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:43.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:43.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:43.829 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:43.870 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:43.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:43.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:43.972 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:44.006 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:44.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:44.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:44.108 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:44.142 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:44.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:44.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:44.144 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 6, 7, 1, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:59:44.144 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:59:44.144 INFO:tasks.thrashosds.thrasher:Reweighting osd 7 to 0.10923013639905937 2024-09-13T14:59:44.144 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 7 0.10923013639905937 2024-09-13T14:59:44.197 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T14:59:44.205 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 87 16 122613 122597 88.0536 64 0.0170961 0.0113515 2024-09-13T14:59:44.244 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:44.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:44.284 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:44.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:44.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:44.425 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:44.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:44.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:44.514 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:44.527 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:44.577 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:59:44.583 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:44.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:44.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:44.685 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:44.719 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:44.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:44.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:44.821 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:44.833 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:59:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:59:44] "GET /metrics HTTP/1.1" 200 36698 "" "Prometheus/2.51.0" 2024-09-13T14:59:44.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:44.878 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:44.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:44.979 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:45.014 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:45.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:45.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:45.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:45 smithi151 ceph-mon[104450]: pgmap v2094: 119 pgs: 119 active+clean; 7.4 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 58 MiB/s wr, 931 op/s 2024-09-13T14:59:45.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:45 smithi151 ceph-mon[108966]: pgmap v2094: 119 pgs: 119 active+clean; 7.4 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 58 MiB/s wr, 931 op/s 2024-09-13T14:59:45.116 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:45.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:45.157 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:45.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:45.205 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 88 16 123597 123581 87.7518 61.5 0.00584894 0.0113907 2024-09-13T14:59:45.257 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:45.292 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:45.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:45.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:45.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:45 smithi184 ceph-mon[92527]: pgmap v2094: 119 pgs: 119 active+clean; 7.4 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 58 MiB/s wr, 931 op/s 2024-09-13T14:59:45.393 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:45.401 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:45.436 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:45.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:45.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:45.539 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:45.573 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:45.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:45.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:45.675 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:45.707 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:45.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:45.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:45.809 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:45.844 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:45.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:45.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:45.949 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:46.000 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:46.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:46.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:46.102 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:46.144 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:46.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:46.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:46.205 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 89 16 124639 124623 87.4975 65.125 0.0156168 0.0114238 2024-09-13T14:59:46.246 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:46.286 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:46.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:46.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:46.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:46.421 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:46.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:46.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:46.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:59:46.487 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:59:46.487 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:59:46.523 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:46.580 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:46.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:46.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:46.682 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:46.725 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:46.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:46.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:46.827 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:46.864 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:46.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:46.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:46.966 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:46.998 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:46.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:46.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:47.100 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:47.103 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:47 smithi184 ceph-mon[92527]: pgmap v2095: 119 pgs: 119 active+clean; 7.4 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 58 MiB/s wr, 931 op/s 2024-09-13T14:59:47.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:47 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:47.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:47 smithi151 ceph-mon[104450]: pgmap v2095: 119 pgs: 119 active+clean; 7.4 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 58 MiB/s wr, 931 op/s 2024-09-13T14:59:47.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:47 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:47.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:47 smithi151 ceph-mon[108966]: pgmap v2095: 119 pgs: 119 active+clean; 7.4 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 58 MiB/s wr, 931 op/s 2024-09-13T14:59:47.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:47 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:47.132 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:47.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:47.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:47.166 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T14:59:47.205 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Total time run: 90.0095 2024-09-13T14:59:47.205 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Total writes made: 125610 2024-09-13T14:59:47.205 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Write size: 65536 2024-09-13T14:59:47.205 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Object size: 65536 2024-09-13T14:59:47.205 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Bandwidth (MB/sec): 87.22 2024-09-13T14:59:47.205 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Stddev Bandwidth: 38.9878 2024-09-13T14:59:47.206 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Max bandwidth (MB/sec): 250.688 2024-09-13T14:59:47.206 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Min bandwidth (MB/sec): 35.6875 2024-09-13T14:59:47.206 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Average IOPS: 1395 2024-09-13T14:59:47.206 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Stddev IOPS: 623.806 2024-09-13T14:59:47.206 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Max IOPS: 4011 2024-09-13T14:59:47.206 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Min IOPS: 571 2024-09-13T14:59:47.206 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Average Latency(s): 0.0114609 2024-09-13T14:59:47.206 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Stddev Latency(s): 0.0139685 2024-09-13T14:59:47.206 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Max latency(s): 0.812283 2024-09-13T14:59:47.206 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Min latency(s): 0.00119391 2024-09-13T14:59:47.210 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T14:59:47.234 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:47.284 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:47.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:47.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:47.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:47.430 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:47.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:47.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:47.531 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:47.574 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:47.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:47.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:47.652 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:59:47.675 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:47.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:47.723 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:47.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:47.824 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:47.855 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:47.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:47.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:47.958 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:47.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:47.998 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:47.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:48.100 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:48.128 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:48.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:48.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:48.231 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:48.260 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:48.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:48.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:48.361 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:48.393 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:48.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:48.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:48.494 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:48.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:48.528 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:48.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:48.629 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:48.657 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:48.657 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:48.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:48.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:48.759 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:48.791 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:48.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:48.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:48.893 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:48.928 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:48.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:48.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:49.030 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:49.047 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.7 to 0.10923 (1bf6) 2024-09-13T14:59:49.054 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:49 smithi151 ceph-mon[104450]: pgmap v2096: 119 pgs: 119 active+clean; 7.6 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 94 B/s rd, 69 MiB/s wr, 1.12k op/s 2024-09-13T14:59:49.054 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:49 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2574153102' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:49.054 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:49 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:49.054 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:49 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2664587055' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 0.10923013639905937} : dispatch 2024-09-13T14:59:49.054 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:49 smithi151 ceph-mon[108966]: pgmap v2096: 119 pgs: 119 active+clean; 7.6 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 94 B/s rd, 69 MiB/s wr, 1.12k op/s 2024-09-13T14:59:49.054 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:49 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2574153102' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:49.055 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:49 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:49.055 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:49 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2664587055' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 0.10923013639905937} : dispatch 2024-09-13T14:59:49.055 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-13T14:59:49.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:49.073 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:49.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:49.174 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:49.196 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:49.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:49.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:49.298 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:49.319 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:49.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:49.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:49.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:49 smithi184 ceph-mon[92527]: pgmap v2096: 119 pgs: 119 active+clean; 7.6 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 94 B/s rd, 69 MiB/s wr, 1.12k op/s 2024-09-13T14:59:49.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:49 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2574153102' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:49.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:49 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:49.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:49 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2664587055' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 0.10923013639905937} : dispatch 2024-09-13T14:59:49.422 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:49.443 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:49.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:49.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:49.545 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:49.566 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:49.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:49.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:49.668 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:49.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:49.691 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:49.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:49.792 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:49.814 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:49.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:49.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:49.916 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:49.937 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:49.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:49.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:50.014 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T14:59:50.014 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:59:50.040 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:50.069 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:50.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:50.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:50.171 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:50.192 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:50.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:50.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:50.293 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 14:59:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T14:59:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T14:59:50.293 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:50 smithi184 ceph-mon[92527]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:59:50.293 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:50 smithi184 ceph-mon[92527]: Cluster is now healthy 2024-09-13T14:59:50.294 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:50 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:59:50.294 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:50 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2664587055' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 0.10923013639905937}]': finished 2024-09-13T14:59:50.294 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:50 smithi184 ceph-mon[92527]: osdmap e1280: 8 total, 8 up, 8 in 2024-09-13T14:59:50.294 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:50 smithi184 ceph-mon[92527]: osdmap e1281: 8 total, 8 up, 8 in 2024-09-13T14:59:50.294 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:50.316 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:50.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:50.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:50.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:50 smithi151 ceph-mon[104450]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:59:50.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:50 smithi151 ceph-mon[104450]: Cluster is now healthy 2024-09-13T14:59:50.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:50 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:59:50.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:50 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2664587055' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 0.10923013639905937}]': finished 2024-09-13T14:59:50.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:50 smithi151 ceph-mon[104450]: osdmap e1280: 8 total, 8 up, 8 in 2024-09-13T14:59:50.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:50 smithi151 ceph-mon[104450]: osdmap e1281: 8 total, 8 up, 8 in 2024-09-13T14:59:50.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:50 smithi151 ceph-mon[108966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:59:50.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:50 smithi151 ceph-mon[108966]: Cluster is now healthy 2024-09-13T14:59:50.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:50 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:59:50.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:50 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2664587055' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 0.10923013639905937}]': finished 2024-09-13T14:59:50.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:50 smithi151 ceph-mon[108966]: osdmap e1280: 8 total, 8 up, 8 in 2024-09-13T14:59:50.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:50 smithi151 ceph-mon[108966]: osdmap e1281: 8 total, 8 up, 8 in 2024-09-13T14:59:50.418 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:50.439 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:50.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:50.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:50.541 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:50.562 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:50.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:50.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:50.664 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:50.685 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:50.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:50.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:50.787 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:50.808 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:50.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:50.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:50.910 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:50.931 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:50.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:50.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:51.033 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:51.063 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:51.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:51.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:51.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:51 smithi184 ceph-mon[92527]: pgmap v2098: 119 pgs: 119 active+clean; 7.6 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 102 B/s rd, 56 MiB/s wr, 928 op/s 2024-09-13T14:59:51.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:51 smithi184 ceph-mon[92527]: osdmap e1282: 8 total, 8 up, 8 in 2024-09-13T14:59:51.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:51.199 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:51.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:51.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:51.301 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:51.347 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:51.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:51.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:51.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:59:51.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:51 smithi151 ceph-mon[104450]: pgmap v2098: 119 pgs: 119 active+clean; 7.6 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 102 B/s rd, 56 MiB/s wr, 928 op/s 2024-09-13T14:59:51.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:51 smithi151 ceph-mon[104450]: osdmap e1282: 8 total, 8 up, 8 in 2024-09-13T14:59:51.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:51 smithi151 ceph-mon[108966]: pgmap v2098: 119 pgs: 119 active+clean; 7.6 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 102 B/s rd, 56 MiB/s wr, 928 op/s 2024-09-13T14:59:51.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:51 smithi151 ceph-mon[108966]: osdmap e1282: 8 total, 8 up, 8 in 2024-09-13T14:59:51.449 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:51.482 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:51.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:51.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:51.584 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:51.626 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:51.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:51.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:51.728 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:51.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:51.763 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:51.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:51.864 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:51.899 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:51.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:51.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:52.001 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:52.042 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:52.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:52.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:52.144 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:52.185 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:52.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:52.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:52.286 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:52.305 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:52.333 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:52 smithi184 ceph-mon[92527]: 10.9 starting backfill to osd.4 from (0'0,0'0] MAX to 1279'7862 2024-09-13T14:59:52.333 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:52 smithi184 ceph-mon[92527]: 10.c starting backfill to osd.3 from (0'0,0'0] MAX to 1279'7943 2024-09-13T14:59:52.333 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:52 smithi184 ceph-mon[92527]: 10.8 starting backfill to osd.5 from (0'0,0'0] MAX to 1279'7978 2024-09-13T14:59:52.333 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:52 smithi184 ceph-mon[92527]: 1.0 scrub starts 2024-09-13T14:59:52.333 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:52 smithi184 ceph-mon[92527]: 1.0 scrub ok 2024-09-13T14:59:52.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:52.337 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:52.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:52.349 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T14:59:52.349 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:59:52.350 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:52 smithi151 ceph-mon[104450]: 10.9 starting backfill to osd.4 from (0'0,0'0] MAX to 1279'7862 2024-09-13T14:59:52.350 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:52 smithi151 ceph-mon[104450]: 10.c starting backfill to osd.3 from (0'0,0'0] MAX to 1279'7943 2024-09-13T14:59:52.350 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:52 smithi151 ceph-mon[104450]: 10.8 starting backfill to osd.5 from (0'0,0'0] MAX to 1279'7978 2024-09-13T14:59:52.350 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:52 smithi151 ceph-mon[104450]: 1.0 scrub starts 2024-09-13T14:59:52.350 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:52 smithi151 ceph-mon[104450]: 1.0 scrub ok 2024-09-13T14:59:52.350 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:52 smithi151 ceph-mon[108966]: 10.9 starting backfill to osd.4 from (0'0,0'0] MAX to 1279'7862 2024-09-13T14:59:52.350 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:52 smithi151 ceph-mon[108966]: 10.c starting backfill to osd.3 from (0'0,0'0] MAX to 1279'7943 2024-09-13T14:59:52.350 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:52 smithi151 ceph-mon[108966]: 10.8 starting backfill to osd.5 from (0'0,0'0] MAX to 1279'7978 2024-09-13T14:59:52.351 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:52 smithi151 ceph-mon[108966]: 1.0 scrub starts 2024-09-13T14:59:52.351 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:52 smithi151 ceph-mon[108966]: 1.0 scrub ok 2024-09-13T14:59:52.387 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T14:59:52.387 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:59:52.407 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:59:52.422 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T14:59:52.422 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:59:52.439 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:52.458 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T14:59:52.458 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:59:52.471 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:52.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:52.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:52.490 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T14:59:52.491 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:59:52.525 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T14:59:52.525 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:59:52.559 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T14:59:52.559 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:59:52.573 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:52.592 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T14:59:52.592 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:59:52.637 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:52.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:52.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:52.739 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:52.781 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:52.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:52.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:52.883 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:52.921 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:52.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:52.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:53.023 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:53.056 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:53.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:53.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:53.158 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:53.183 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:53.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:53.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:53.284 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:53.319 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:53.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:53.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:53.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:53 smithi184 ceph-mon[92527]: pgmap v2101: 119 pgs: 119 active+clean; 7.6 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 170 B/s rd, 48 MiB/s wr, 843 op/s 2024-09-13T14:59:53.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:53 smithi184 ceph-mon[92527]: 1.2 scrub starts 2024-09-13T14:59:53.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:53 smithi184 ceph-mon[92527]: 1.2 scrub ok 2024-09-13T14:59:53.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:53 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/899078124' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:53.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:53 smithi151 ceph-mon[104450]: pgmap v2101: 119 pgs: 119 active+clean; 7.6 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 170 B/s rd, 48 MiB/s wr, 843 op/s 2024-09-13T14:59:53.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:53 smithi151 ceph-mon[104450]: 1.2 scrub starts 2024-09-13T14:59:53.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:53 smithi151 ceph-mon[104450]: 1.2 scrub ok 2024-09-13T14:59:53.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:53 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/899078124' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:53.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:53 smithi151 ceph-mon[108966]: pgmap v2101: 119 pgs: 119 active+clean; 7.6 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 170 B/s rd, 48 MiB/s wr, 843 op/s 2024-09-13T14:59:53.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:53 smithi151 ceph-mon[108966]: 1.2 scrub starts 2024-09-13T14:59:53.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:53 smithi151 ceph-mon[108966]: 1.2 scrub ok 2024-09-13T14:59:53.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:53 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/899078124' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:53.421 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:53.450 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:53.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:53.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:53.552 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:53.584 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:53.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:53.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:53.686 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:53.717 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:53.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:53.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:53.819 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:53.854 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:53.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:53.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:53.956 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:53.995 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:53.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:53.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:54.097 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:54.133 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:54.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:54.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:54.235 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:54.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:54.267 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:54.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:54.369 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:54.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:54.403 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:54.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:54.504 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:54.540 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:54.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:54.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:54.642 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:54.677 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:54.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:54.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:54.779 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:54.809 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:54.850 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:54.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:54.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:54.860 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 14:59:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:14:59:54] "GET /metrics HTTP/1.1" 200 36699 "" "Prometheus/2.51.0" 2024-09-13T14:59:54.952 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:55.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:55.010 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:55.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:55.111 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:55.147 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:55.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:55.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:55.249 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:55.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:55.282 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:55.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:55.349 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:55 smithi151 ceph-mon[104450]: pgmap v2102: 119 pgs: 2 active+remapped+backfilling, 117 active+clean; 7.5 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 11 MiB/s wr, 626 op/s; 14901/368502 objects misplaced (4.044%); 2.9 MiB/s, 46 objects/s recovering 2024-09-13T14:59:55.349 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:55 smithi151 ceph-mon[108966]: pgmap v2102: 119 pgs: 2 active+remapped+backfilling, 117 active+clean; 7.5 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 11 MiB/s wr, 626 op/s; 14901/368502 objects misplaced (4.044%); 2.9 MiB/s, 46 objects/s recovering 2024-09-13T14:59:55.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:55 smithi184 ceph-mon[92527]: pgmap v2102: 119 pgs: 2 active+remapped+backfilling, 117 active+clean; 7.5 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 11 MiB/s wr, 626 op/s; 14901/368502 objects misplaced (4.044%); 2.9 MiB/s, 46 objects/s recovering 2024-09-13T14:59:55.383 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:55.421 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:55.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:55.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:55.523 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:55.564 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:55.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:55.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:55.666 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:55.701 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:55.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:55.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:55.803 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:55.839 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:55.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:55.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:55.941 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:55.979 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:55.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:55.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:56.081 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:56.094 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 6, 7, 1, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:59:56.094 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:59:56.094 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 2, 3, 6, 7, 1, 4, 5] 2024-09-13T14:59:56.094 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd out 0 2024-09-13T14:59:56.117 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:56.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:56.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:56.161 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:59:56.219 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:56.277 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:56.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:56.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:56.379 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:56.466 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:56.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:56.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:56.568 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:56.609 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:56.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:56.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:56.712 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:56.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:56.759 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:56.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:56.772 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T14:59:56.772 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-13T14:59:56.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.772 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.36396.0:132794 10.8 10:1ef16329:::benchmark_data_smithi151_288181_object7181:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1282)", 2024-09-13T14:59:56.772 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:59:56.367442+0000", 2024-09-13T14:59:56.772 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.030930124, 2024-09-13T14:59:56.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.030958942, 2024-09-13T14:59:56.773 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:59:56.773 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:59:56.773 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:59:56.773 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:59:56.773 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.36396", 2024-09-13T14:59:56.773 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T14:59:56.773 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 132794 2024-09-13T14:59:56.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.773 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:59:56.773 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.773 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:59:56.773 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.367442+0000", 2024-09-13T14:59:56.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:59:56.774 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.774 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:59:56.774 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.367442+0000", 2024-09-13T14:59:56.774 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:59:56.774 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.774 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:59:56.774 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.367444+0000", 2024-09-13T14:59:56.774 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1349999999999999e-06 2024-09-13T14:59:56.774 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:59:56.775 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.367448+0000", 2024-09-13T14:59:56.775 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8600000000000003e-06 2024-09-13T14:59:56.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:59:56.775 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.367459+0000", 2024-09-13T14:59:56.775 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0648e-05 2024-09-13T14:59:56.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:59:56.776 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.367465+0000", 2024-09-13T14:59:56.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1449999999999996e-06 2024-09-13T14:59:56.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.776 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.776 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:59:56.776 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.369902+0000", 2024-09-13T14:59:56.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0024370300000000002 2024-09-13T14:59:56.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.776 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.776 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:59:56.776 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.370043+0000", 2024-09-13T14:59:56.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014042300000000001 2024-09-13T14:59:56.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.777 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,5,7", 2024-09-13T14:59:56.777 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.370185+0000", 2024-09-13T14:59:56.777 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000142898 2024-09-13T14:59:56.777 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.777 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T14:59:56.777 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.376852+0000", 2024-09-13T14:59:56.777 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0066666390000000002 2024-09-13T14:59:56.777 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.777 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:56.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.377494+0000", 2024-09-13T14:59:56.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00064229700000000001 2024-09-13T14:59:56.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:56.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.381379+0000", 2024-09-13T14:59:56.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0038846089999999998 2024-09-13T14:59:56.778 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:59:56.778 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:59:56.778 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:59:56.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.778 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.36396.0:132801 10.c e1282/1281 10:3d7b0d22:::benchmark_data_smithi151_288181_object7188:head v 1282'8282, mlcod=1282'8280)", 2024-09-13T14:59:56.779 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:59:56.377540+0000", 2024-09-13T14:59:56.779 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.020832015999999998, 2024-09-13T14:59:56.779 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020948205000000001, 2024-09-13T14:59:56.779 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:59:56.779 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:59:56.779 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-13T14:59:56.779 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:59:56.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.779 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:59:56.779 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.377540+0000", 2024-09-13T14:59:56.779 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:59:56.779 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.779 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:59:56.779 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.377540+0000", 2024-09-13T14:59:56.780 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:59:56.780 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.780 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.780 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:59:56.780 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.377541+0000", 2024-09-13T14:59:56.780 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1600000000000001e-07 2024-09-13T14:59:56.780 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.780 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.780 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:59:56.780 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.377548+0000", 2024-09-13T14:59:56.780 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2559999999999996e-06 2024-09-13T14:59:56.780 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.780 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.781 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:59:56.781 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.377556+0000", 2024-09-13T14:59:56.781 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2320000000000001e-06 2024-09-13T14:59:56.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.781 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:59:56.781 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.377561+0000", 2024-09-13T14:59:56.781 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9919999999999998e-06 2024-09-13T14:59:56.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.781 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:59:56.781 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.377702+0000", 2024-09-13T14:59:56.781 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014037399999999999 2024-09-13T14:59:56.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.782 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:59:56.782 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.377726+0000", 2024-09-13T14:59:56.782 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4304e-05 2024-09-13T14:59:56.782 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:59:56.782 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:59:56.782 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:59:56.782 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.782 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.36396.0:132805 10.4 10:2e4b134b:::benchmark_data_smithi151_288181_object7192:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1282)", 2024-09-13T14:59:56.782 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:59:56.391148+0000", 2024-09-13T14:59:56.782 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0072242160000000003, 2024-09-13T14:59:56.782 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0073914209999999996, 2024-09-13T14:59:56.782 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:59:56.782 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:59:56.783 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:59:56.783 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T14:59:56.783 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.36396", 2024-09-13T14:59:56.783 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T14:59:56.783 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 132805 2024-09-13T14:59:56.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.783 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:59:56.783 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.783 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:59:56.783 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.391148+0000", 2024-09-13T14:59:56.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:59:56.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.783 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.784 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:59:56.784 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.391148+0000", 2024-09-13T14:59:56.784 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:59:56.784 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.784 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:59:56.784 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.391151+0000", 2024-09-13T14:59:56.784 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6740000000000001e-06 2024-09-13T14:59:56.784 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.784 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:59:56.784 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.391160+0000", 2024-09-13T14:59:56.784 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0399999999999998e-06 2024-09-13T14:59:56.784 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:59:56.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.391188+0000", 2024-09-13T14:59:56.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7834000000000002e-05 2024-09-13T14:59:56.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:59:56.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.391199+0000", 2024-09-13T14:59:56.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0919e-05 2024-09-13T14:59:56.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:59:56.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.397836+0000", 2024-09-13T14:59:56.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00663767 2024-09-13T14:59:56.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.786 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:59:56.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.397964+0000", 2024-09-13T14:59:56.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000128059 2024-09-13T14:59:56.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.786 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:59:56.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.398081+0000", 2024-09-13T14:59:56.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116458 2024-09-13T14:59:56.786 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:59:56.786 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:59:56.787 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:59:56.787 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.787 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.787 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.36396.0:132808 10.c e1282/1281 10:3456c606:::benchmark_data_smithi151_288181_object7195:head v 1282'8283, mlcod=1282'8280)", 2024-09-13T14:59:56.787 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:59:56.391239+0000", 2024-09-13T14:59:56.787 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0071331440000000001, 2024-09-13T14:59:56.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0073624930000000003, 2024-09-13T14:59:56.787 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:59:56.787 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:59:56.787 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-13T14:59:56.787 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:59:56.787 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.787 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:59:56.788 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.391239+0000", 2024-09-13T14:59:56.788 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:59:56.788 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.788 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.788 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:59:56.788 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.391239+0000", 2024-09-13T14:59:56.788 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:59:56.788 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.788 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.788 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:59:56.788 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.391241+0000", 2024-09-13T14:59:56.789 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2890000000000001e-06 2024-09-13T14:59:56.789 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.789 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:59:56.789 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.391246+0000", 2024-09-13T14:59:56.789 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6899999999999997e-06 2024-09-13T14:59:56.789 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.789 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:59:56.789 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.391254+0000", 2024-09-13T14:59:56.789 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1720000000000007e-06 2024-09-13T14:59:56.789 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.790 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:59:56.790 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.391260+0000", 2024-09-13T14:59:56.790 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6500000000000001e-06 2024-09-13T14:59:56.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.790 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:59:56.790 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.395023+0000", 2024-09-13T14:59:56.790 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037627799999999999 2024-09-13T14:59:56.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.790 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:59:56.790 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.395046+0000", 2024-09-13T14:59:56.790 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3575999999999999e-05 2024-09-13T14:59:56.791 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:59:56.791 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:59:56.791 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:59:56.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.791 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.36396.0:132807 10.5 e1282/1268)", 2024-09-13T14:59:56.791 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:59:56.391284+0000", 2024-09-13T14:59:56.791 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0070888569999999996, 2024-09-13T14:59:56.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0073609950000000004, 2024-09-13T14:59:56.791 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:59:56.791 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:59:56.791 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "queued for pg", 2024-09-13T14:59:56.791 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:59:56.792 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.792 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:59:56.792 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.391284+0000", 2024-09-13T14:59:56.792 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:59:56.792 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.792 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.792 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:59:56.792 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.391284+0000", 2024-09-13T14:59:56.792 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:59:56.792 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.792 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.792 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:59:56.792 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.391284+0000", 2024-09-13T14:59:56.792 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1699999999999999e-07 2024-09-13T14:59:56.793 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.793 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.793 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:59:56.793 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.391288+0000", 2024-09-13T14:59:56.793 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1500000000000001e-06 2024-09-13T14:59:56.793 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.793 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.793 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:59:56.793 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.391295+0000", 2024-09-13T14:59:56.793 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7050000000000002e-06 2024-09-13T14:59:56.793 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.793 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.793 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:59:56.793 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.391299+0000", 2024-09-13T14:59:56.794 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3749999999999996e-06 2024-09-13T14:59:56.794 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:59:56.794 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:59:56.794 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:59:56.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.794 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.794 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.36396.0:132806 10.a e1282/1268 10:5e5b6f2e:::benchmark_data_smithi151_288181_object7193:head v 1282'8235, mlcod=1282'8233)", 2024-09-13T14:59:56.794 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:59:56.393864+0000", 2024-09-13T14:59:56.794 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0045086010000000001, 2024-09-13T14:59:56.794 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0048198989999999999, 2024-09-13T14:59:56.794 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T14:59:56.794 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T14:59:56.794 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-13T14:59:56.794 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T14:59:56.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.795 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T14:59:56.795 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.393864+0000", 2024-09-13T14:59:56.795 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:59:56.795 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.795 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T14:59:56.795 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.393864+0000", 2024-09-13T14:59:56.795 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T14:59:56.795 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.795 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T14:59:56.795 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.393865+0000", 2024-09-13T14:59:56.796 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.468e-06 2024-09-13T14:59:56.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.796 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T14:59:56.796 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.393874+0000", 2024-09-13T14:59:56.796 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2010000000000005e-06 2024-09-13T14:59:56.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.796 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T14:59:56.796 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.393889+0000", 2024-09-13T14:59:56.796 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4749999999999999e-05 2024-09-13T14:59:56.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.796 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T14:59:56.797 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.393898+0000", 2024-09-13T14:59:56.797 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6710000000000005e-06 2024-09-13T14:59:56.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.797 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T14:59:56.797 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.393976+0000", 2024-09-13T14:59:56.797 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8492999999999995e-05 2024-09-13T14:59:56.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T14:59:56.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T14:59:56.797 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T14:59:56.797 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:59:56.393996+0000", 2024-09-13T14:59:56.797 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.914e-05 2024-09-13T14:59:56.797 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:59:56.797 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T14:59:56.798 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:59:56.798 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T14:59:56.798 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-13T14:59:56.798 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 6 2024-09-13T14:59:56.798 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T14:59:56.860 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:56.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:56.902 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:56.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:57.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:57.043 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:57.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:57.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:57.145 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:57.179 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:57.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:57.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:57.281 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:57.318 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:57.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:57.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:57.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:57 smithi184 ceph-mon[92527]: pgmap v2103: 119 pgs: 3 active+remapped+backfilling, 116 active+clean; 7.5 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 942 KiB/s wr, 431 op/s; 22194/367356 objects misplaced (6.042%); 5.4 MiB/s, 87 objects/s recovering 2024-09-13T14:59:57.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:57 smithi151 ceph-mon[104450]: pgmap v2103: 119 pgs: 3 active+remapped+backfilling, 116 active+clean; 7.5 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 942 KiB/s wr, 431 op/s; 22194/367356 objects misplaced (6.042%); 5.4 MiB/s, 87 objects/s recovering 2024-09-13T14:59:57.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:57 smithi151 ceph-mon[108966]: pgmap v2103: 119 pgs: 3 active+remapped+backfilling, 116 active+clean; 7.5 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 942 KiB/s wr, 431 op/s; 22194/367356 objects misplaced (6.042%); 5.4 MiB/s, 87 objects/s recovering 2024-09-13T14:59:57.420 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:57.459 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:57.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:57.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:57.561 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:57.583 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T14:59:57.638 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:57.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:57.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:57.740 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:57.776 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:57.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:57.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:57.878 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:57.917 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:57.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:57.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:58.018 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:58.065 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:58.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:58.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:58.167 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:58.193 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:58 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:59:58.194 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:58 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:59:58.204 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:58.205 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:58.213 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:58.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:58.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:58.316 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:58.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:58 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:59:58.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:58.365 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:58.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:58.466 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:58.524 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:58.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:58.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:58.626 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:58.658 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:58.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:58.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:58.762 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:58.796 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:58.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:58.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:58.898 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:58.945 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:58.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:58.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:59.047 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:59.080 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:59.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:59.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:59.118 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-13T14:59:59.182 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:59.220 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:59.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:59.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:59.322 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:59.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:59 smithi184 ceph-mon[92527]: pgmap v2104: 119 pgs: 3 active+remapped+backfilling, 116 active+clean; 7.2 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 792 KiB/s wr, 847 op/s; 20046/355719 objects misplaced (5.635%); 17 MiB/s, 266 objects/s recovering 2024-09-13T14:59:59.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 14:59:59 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2557853608' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:59:59.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:59 smithi151 ceph-mon[104450]: pgmap v2104: 119 pgs: 3 active+remapped+backfilling, 116 active+clean; 7.2 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 792 KiB/s wr, 847 op/s; 20046/355719 objects misplaced (5.635%); 17 MiB/s, 266 objects/s recovering 2024-09-13T14:59:59.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 14:59:59 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2557853608' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:59:59.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:59 smithi151 ceph-mon[108966]: pgmap v2104: 119 pgs: 3 active+remapped+backfilling, 116 active+clean; 7.2 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 792 KiB/s wr, 847 op/s; 20046/355719 objects misplaced (5.635%); 17 MiB/s, 266 objects/s recovering 2024-09-13T14:59:59.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 14:59:59 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2557853608' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:59:59.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:59.361 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:59.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:59.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:59.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T14:59:59.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:59.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:59.597 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:59.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:59.653 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:59.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:59.754 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:59.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:59.797 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:59.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:59.898 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:59.942 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T14:59:59.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:59.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:59.991 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:00.044 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:00.091 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:00.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:00.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:00.193 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:00.215 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:00 smithi184 ceph-mon[92527]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:00:00.215 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:00 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2557853608' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:00:00.215 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:00 smithi184 ceph-mon[92527]: osdmap e1283: 8 total, 8 up, 8 in 2024-09-13T15:00:00.215 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:00 smithi184 ceph-mon[92527]: overall HEALTH_WARN noscrub flag(s) set 2024-09-13T15:00:00.228 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:00.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:00.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:00.330 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:00.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:00 smithi151 ceph-mon[104450]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:00:00.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:00 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2557853608' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:00:00.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:00 smithi151 ceph-mon[104450]: osdmap e1283: 8 total, 8 up, 8 in 2024-09-13T15:00:00.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:00 smithi151 ceph-mon[104450]: overall HEALTH_WARN noscrub flag(s) set 2024-09-13T15:00:00.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:00 smithi151 ceph-mon[108966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:00:00.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:00 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2557853608' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:00:00.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:00 smithi151 ceph-mon[108966]: osdmap e1283: 8 total, 8 up, 8 in 2024-09-13T15:00:00.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:00 smithi151 ceph-mon[108966]: overall HEALTH_WARN noscrub flag(s) set 2024-09-13T15:00:00.363 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:00.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:00.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:00.464 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:00.489 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:00:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:00:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:00:00.499 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:00.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:00.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:00.601 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:00.648 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:00.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:00.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:00.751 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:00.795 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:00.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:00.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:00.897 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:00.930 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:00.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:00.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:01.032 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:01.061 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:01.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:01.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:01.132 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:01 smithi151 ceph-mon[104450]: pgmap v2106: 119 pgs: 3 active+remapped+backfilling, 116 active+clean; 7.2 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 723 KiB/s wr, 917 op/s; 19151/351951 objects misplaced (5.441%); 20 MiB/s, 312 objects/s recovering 2024-09-13T15:00:01.132 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:01 smithi151 ceph-mon[108966]: pgmap v2106: 119 pgs: 3 active+remapped+backfilling, 116 active+clean; 7.2 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 723 KiB/s wr, 917 op/s; 19151/351951 objects misplaced (5.441%); 20 MiB/s, 312 objects/s recovering 2024-09-13T15:00:01.164 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:01.201 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:01.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:01.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:01.303 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:01.304 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:01 smithi184 ceph-mon[92527]: pgmap v2106: 119 pgs: 3 active+remapped+backfilling, 116 active+clean; 7.2 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 723 KiB/s wr, 917 op/s; 19151/351951 objects misplaced (5.441%); 20 MiB/s, 312 objects/s recovering 2024-09-13T15:00:01.341 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:01.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:01.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:01.442 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:01.477 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:01.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:01.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:01.578 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:01.616 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:01.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:01.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:01.718 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:01.777 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:01.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:01.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:01.880 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:01.929 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:01.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:01.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:02.031 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:02.077 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:02.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:02.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:02.178 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:02.219 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:02.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:02.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:02.321 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:02.352 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:02.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:02.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:02.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:02.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:02.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:02.454 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:02.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:02.497 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:02.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:02.598 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:02.635 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:02.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:02.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:02.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:02.780 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:02.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:02.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:02.882 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:02.921 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:02.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:02.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:03.022 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:03.059 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:03.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:03.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:03.161 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:03.200 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:03.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:03.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:03.302 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:03.336 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:03.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:03.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:03.383 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:03 smithi184 ceph-mon[92527]: pgmap v2107: 119 pgs: 3 active+remapped+backfilling, 116 active+clean; 7.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 218 KiB/s wr, 818 op/s; 19151/351000 objects misplaced (5.456%); 17 MiB/s, 274 objects/s recovering 2024-09-13T15:00:03.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:03 smithi151 ceph-mon[104450]: pgmap v2107: 119 pgs: 3 active+remapped+backfilling, 116 active+clean; 7.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 218 KiB/s wr, 818 op/s; 19151/351000 objects misplaced (5.456%); 17 MiB/s, 274 objects/s recovering 2024-09-13T15:00:03.438 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:03 smithi151 ceph-mon[108966]: pgmap v2107: 119 pgs: 3 active+remapped+backfilling, 116 active+clean; 7.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 218 KiB/s wr, 818 op/s; 19151/351000 objects misplaced (5.456%); 17 MiB/s, 274 objects/s recovering 2024-09-13T15:00:03.439 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:03.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:03.487 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:03.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:03.588 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:03.624 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:03.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:03.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:03.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:03 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a[104427]: 2024-09-13T15:00:03.434+0000 7fe9da14f640 -1 mon.a@0(leader).osd e1283 definitely_dead 0 2024-09-13T15:00:03.725 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:03.756 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:03.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:03.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:03.858 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:03.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:03.904 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:03.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:03.937 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T15:00:04.005 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:04.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:04.046 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:04.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:04.147 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:04.149 INFO:teuthology.orchestra.run.smithi151.stderr:marked out osd.0. 2024-09-13T15:00:04.202 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:04.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:04.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:04.259 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:04 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1907976077' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-09-13T15:00:04.259 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:04 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-09-13T15:00:04.259 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:04 smithi151 ceph-mon[104450]: Client client.admin marked osd.0 out, while it was still marked up 2024-09-13T15:00:04.259 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:04 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:00:04.259 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:04 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:00:04.260 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:04 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1907976077' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-09-13T15:00:04.260 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:04 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-09-13T15:00:04.260 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:04 smithi151 ceph-mon[108966]: Client client.admin marked osd.0 out, while it was still marked up 2024-09-13T15:00:04.260 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:04 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:00:04.260 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:04 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:00:04.304 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:04.327 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:04.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:04.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:04.429 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:04.452 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:04.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:04.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:04.554 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:04.577 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:04.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:04.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:04.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:04 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1907976077' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-09-13T15:00:04.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:04 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-09-13T15:00:04.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:04 smithi184 ceph-mon[92527]: Client client.admin marked osd.0 out, while it was still marked up 2024-09-13T15:00:04.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:04 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:00:04.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:04 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:00:04.679 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:04.703 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:04.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:04.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:04.804 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:00:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:00:04] "GET /metrics HTTP/1.1" 200 36752 "" "Prometheus/2.51.0" 2024-09-13T15:00:04.805 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:04.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:04.828 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:04.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:04.929 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:04.951 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:04.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:04.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:05.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:05.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:05.992 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:05.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:05.993 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:05 smithi151 ceph-mon[108966]: pgmap v2108: 119 pgs: 3 active+remapped+backfilling, 116 active+clean; 6.9 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 988 op/s; 17316/338829 objects misplaced (5.111%); 24 MiB/s, 376 objects/s recovering 2024-09-13T15:00:05.993 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:05 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-09-13T15:00:05.993 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:05 smithi151 ceph-mon[108966]: osdmap e1284: 8 total, 8 up, 7 in 2024-09-13T15:00:05.993 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:05 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]} : dispatch 2024-09-13T15:00:05.993 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:05 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2393022658' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:05.993 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:05 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-09-13T15:00:05.993 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:05 smithi151 ceph-mon[108966]: osdmap e1285: 8 total, 8 up, 7 in 2024-09-13T15:00:05.993 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:05 smithi151 ceph-mon[104450]: pgmap v2108: 119 pgs: 3 active+remapped+backfilling, 116 active+clean; 6.9 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 988 op/s; 17316/338829 objects misplaced (5.111%); 24 MiB/s, 376 objects/s recovering 2024-09-13T15:00:05.993 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:05 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-09-13T15:00:05.994 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:05 smithi151 ceph-mon[104450]: osdmap e1284: 8 total, 8 up, 7 in 2024-09-13T15:00:05.994 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:05 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]} : dispatch 2024-09-13T15:00:05.994 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:05 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2393022658' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:05.994 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:05 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-09-13T15:00:05.994 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:05 smithi151 ceph-mon[104450]: osdmap e1285: 8 total, 8 up, 7 in 2024-09-13T15:00:05.994 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T15:00:05.994 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T15:00:05.994 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T15:00:05.994 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T15:00:05.995 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T15:00:05.995 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T15:00:05.995 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T15:00:05.995 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T15:00:05.995 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:00:05.995 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:00:05.995 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:00:05.995 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:00:05.995 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:00:05.995 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:00:05.995 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:00:05.996 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:00:05.996 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:05 smithi184 ceph-mon[92527]: pgmap v2108: 119 pgs: 3 active+remapped+backfilling, 116 active+clean; 6.9 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 988 op/s; 17316/338829 objects misplaced (5.111%); 24 MiB/s, 376 objects/s recovering 2024-09-13T15:00:05.996 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:05 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-09-13T15:00:05.996 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:05 smithi184 ceph-mon[92527]: osdmap e1284: 8 total, 8 up, 7 in 2024-09-13T15:00:05.996 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:05 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]} : dispatch 2024-09-13T15:00:05.996 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:05 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2393022658' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:05.996 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:05 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-09-13T15:00:05.997 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:05 smithi184 ceph-mon[92527]: osdmap e1285: 8 total, 8 up, 7 in 2024-09-13T15:00:06.093 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:06.127 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:06.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:06.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:06.228 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:06.268 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:06.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:06.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:06.370 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:06.406 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:06.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:06.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:06.510 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:06.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:06.550 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:06.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:06.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:06 smithi184 ceph-mon[92527]: 10.c continuing backfill to osd.3 from (1274'3386,1283'8745] 10:32e895ce:::benchmark_data_smithi151_288181_object59958:head to 1283'8745 2024-09-13T15:00:06.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:06 smithi184 ceph-mon[92527]: 10.c starting backfill to osd.6 from (0'0,0'0] MAX to 1283'8745 2024-09-13T15:00:06.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:06 smithi184 ceph-mon[92527]: osdmap e1286: 8 total, 8 up, 7 in 2024-09-13T15:00:06.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:06 smithi151 ceph-mon[104450]: 10.c continuing backfill to osd.3 from (1274'3386,1283'8745] 10:32e895ce:::benchmark_data_smithi151_288181_object59958:head to 1283'8745 2024-09-13T15:00:06.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:06 smithi151 ceph-mon[104450]: 10.c starting backfill to osd.6 from (0'0,0'0] MAX to 1283'8745 2024-09-13T15:00:06.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:06 smithi151 ceph-mon[104450]: osdmap e1286: 8 total, 8 up, 7 in 2024-09-13T15:00:06.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:06 smithi151 ceph-mon[108966]: 10.c continuing backfill to osd.3 from (1274'3386,1283'8745] 10:32e895ce:::benchmark_data_smithi151_288181_object59958:head to 1283'8745 2024-09-13T15:00:06.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:06 smithi151 ceph-mon[108966]: 10.c starting backfill to osd.6 from (0'0,0'0] MAX to 1283'8745 2024-09-13T15:00:06.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:06 smithi151 ceph-mon[108966]: osdmap e1286: 8 total, 8 up, 7 in 2024-09-13T15:00:06.652 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:06.687 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:06.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:06.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:06.789 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:06.827 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:06.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:06.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:06.929 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:06.963 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:06.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:06.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:07.065 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:07.102 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:07.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:07.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:07.204 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:07.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:07.241 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:07.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:07.342 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:07.364 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:07.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:07.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:07.466 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:07.500 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:07.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:07.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:07.602 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:07.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:07 smithi184 ceph-mon[92527]: pgmap v2111: 119 pgs: 3 active+remapped+backfilling, 116 active+clean; 6.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 748 op/s; 16445/336975 objects misplaced (4.880%); 18 MiB/s, 287 objects/s recovering 2024-09-13T15:00:07.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:07 smithi184 ceph-mon[92527]: 1.0 scrub starts 2024-09-13T15:00:07.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:07 smithi184 ceph-mon[92527]: 10.1 starting backfill to osd.1 from (0'0,0'0] MAX to 1283'8858 2024-09-13T15:00:07.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:07 smithi184 ceph-mon[92527]: 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 1283'8884 2024-09-13T15:00:07.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:07 smithi184 ceph-mon[92527]: 10.8 starting backfill to osd.4 from (0'0,0'0] MAX to 1283'8810 2024-09-13T15:00:07.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:07 smithi184 ceph-mon[92527]: 10.4 starting backfill to osd.1 from (0'0,0'0] MAX to 1283'8842 2024-09-13T15:00:07.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:07 smithi184 ceph-mon[92527]: 10.6 starting backfill to osd.1 from (0'0,0'0] MAX to 1283'8786 2024-09-13T15:00:07.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:07 smithi184 ceph-mon[92527]: 10.2 starting backfill to osd.3 from (0'0,0'0] MAX to 1283'8917 2024-09-13T15:00:07.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:07 smithi184 ceph-mon[92527]: 10.b starting backfill to osd.4 from (0'0,0'0] MAX to 1283'8818 2024-09-13T15:00:07.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:07 smithi184 ceph-mon[92527]: 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 1283'8718 2024-09-13T15:00:07.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:07 smithi184 ceph-mon[92527]: 10.5 starting backfill to osd.5 from (0'0,0'0] MAX to 1283'8725 2024-09-13T15:00:07.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:07 smithi184 ceph-mon[92527]: 10.8 continuing backfill to osd.5 from (1274'2656,1283'8810] 10:1369da19:::benchmark_data_smithi151_288181_object21246:head to 1283'8810 2024-09-13T15:00:07.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:07 smithi151 ceph-mon[104450]: pgmap v2111: 119 pgs: 3 active+remapped+backfilling, 116 active+clean; 6.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 748 op/s; 16445/336975 objects misplaced (4.880%); 18 MiB/s, 287 objects/s recovering 2024-09-13T15:00:07.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:07 smithi151 ceph-mon[104450]: 1.0 scrub starts 2024-09-13T15:00:07.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:07 smithi151 ceph-mon[104450]: 10.1 starting backfill to osd.1 from (0'0,0'0] MAX to 1283'8858 2024-09-13T15:00:07.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:07 smithi151 ceph-mon[104450]: 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 1283'8884 2024-09-13T15:00:07.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:07 smithi151 ceph-mon[104450]: 10.8 starting backfill to osd.4 from (0'0,0'0] MAX to 1283'8810 2024-09-13T15:00:07.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:07 smithi151 ceph-mon[104450]: 10.4 starting backfill to osd.1 from (0'0,0'0] MAX to 1283'8842 2024-09-13T15:00:07.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:07 smithi151 ceph-mon[104450]: 10.6 starting backfill to osd.1 from (0'0,0'0] MAX to 1283'8786 2024-09-13T15:00:07.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:07 smithi151 ceph-mon[104450]: 10.2 starting backfill to osd.3 from (0'0,0'0] MAX to 1283'8917 2024-09-13T15:00:07.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:07 smithi151 ceph-mon[104450]: 10.b starting backfill to osd.4 from (0'0,0'0] MAX to 1283'8818 2024-09-13T15:00:07.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:07 smithi151 ceph-mon[104450]: 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 1283'8718 2024-09-13T15:00:07.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:07 smithi151 ceph-mon[104450]: 10.5 starting backfill to osd.5 from (0'0,0'0] MAX to 1283'8725 2024-09-13T15:00:07.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:07 smithi151 ceph-mon[104450]: 10.8 continuing backfill to osd.5 from (1274'2656,1283'8810] 10:1369da19:::benchmark_data_smithi151_288181_object21246:head to 1283'8810 2024-09-13T15:00:07.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:07 smithi151 ceph-mon[108966]: pgmap v2111: 119 pgs: 3 active+remapped+backfilling, 116 active+clean; 6.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 748 op/s; 16445/336975 objects misplaced (4.880%); 18 MiB/s, 287 objects/s recovering 2024-09-13T15:00:07.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:07 smithi151 ceph-mon[108966]: 1.0 scrub starts 2024-09-13T15:00:07.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:07 smithi151 ceph-mon[108966]: 10.1 starting backfill to osd.1 from (0'0,0'0] MAX to 1283'8858 2024-09-13T15:00:07.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:07 smithi151 ceph-mon[108966]: 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 1283'8884 2024-09-13T15:00:07.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:07 smithi151 ceph-mon[108966]: 10.8 starting backfill to osd.4 from (0'0,0'0] MAX to 1283'8810 2024-09-13T15:00:07.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:07 smithi151 ceph-mon[108966]: 10.4 starting backfill to osd.1 from (0'0,0'0] MAX to 1283'8842 2024-09-13T15:00:07.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:07 smithi151 ceph-mon[108966]: 10.6 starting backfill to osd.1 from (0'0,0'0] MAX to 1283'8786 2024-09-13T15:00:07.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:07 smithi151 ceph-mon[108966]: 10.2 starting backfill to osd.3 from (0'0,0'0] MAX to 1283'8917 2024-09-13T15:00:07.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:07 smithi151 ceph-mon[108966]: 10.b starting backfill to osd.4 from (0'0,0'0] MAX to 1283'8818 2024-09-13T15:00:07.611 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:07 smithi151 ceph-mon[108966]: 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 1283'8718 2024-09-13T15:00:07.611 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:07 smithi151 ceph-mon[108966]: 10.5 starting backfill to osd.5 from (0'0,0'0] MAX to 1283'8725 2024-09-13T15:00:07.611 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:07 smithi151 ceph-mon[108966]: 10.8 continuing backfill to osd.5 from (1274'2656,1283'8810] 10:1369da19:::benchmark_data_smithi151_288181_object21246:head to 1283'8810 2024-09-13T15:00:07.642 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:07.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:07.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:07.744 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:07.788 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:07.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:07.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:07.890 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:07.921 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:07.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:07.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:08.023 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:08.053 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:08.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:08.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:08.155 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:08.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:08.205 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:08.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:08.306 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:08.350 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:08.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:08.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:08.452 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:08.489 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:08.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:08.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:08.591 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:08.626 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:08.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:08.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:08.728 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:08.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:08.768 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:08.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:08.869 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:08.901 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:08.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:08.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:09.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:09.039 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:09.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:09.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:09.141 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:09.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:09.180 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:09.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:09.281 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:09.320 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:09.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:09.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:09.422 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:09.455 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:09.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:09.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:09.557 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:09.596 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:09.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:09.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:09.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:09 smithi184 ceph-mon[92527]: pgmap v2113: 119 pgs: 6 active+remapped+backfill_wait, 4 active+remapped+backfilling, 109 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 5.5 KiB/s rd, 12 KiB/s wr, 1.17k op/s; 68543/329952 objects misplaced (20.774%); 47 MiB/s, 758 objects/s recovering 2024-09-13T15:00:09.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:09 smithi151 ceph-mon[104450]: pgmap v2113: 119 pgs: 6 active+remapped+backfill_wait, 4 active+remapped+backfilling, 109 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 5.5 KiB/s rd, 12 KiB/s wr, 1.17k op/s; 68543/329952 objects misplaced (20.774%); 47 MiB/s, 758 objects/s recovering 2024-09-13T15:00:09.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:09 smithi151 ceph-mon[108966]: pgmap v2113: 119 pgs: 6 active+remapped+backfill_wait, 4 active+remapped+backfilling, 109 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 5.5 KiB/s rd, 12 KiB/s wr, 1.17k op/s; 68543/329952 objects misplaced (20.774%); 47 MiB/s, 758 objects/s recovering 2024-09-13T15:00:09.698 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:09.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:09.740 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:09.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:09.841 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:09.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:09.878 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:09.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:09.979 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:10.021 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:10.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:10.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:10.117 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:00:10.122 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:10.205 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:10.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:10.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:10.307 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:10.346 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:10.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:10.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:10.448 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:10.482 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:10.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:10.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:10.584 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:10.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:00:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:00:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:00:10.620 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:10.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:10.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:10.681 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T15:00:10.682 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:00:10.721 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:10.758 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:10.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:10.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:10.860 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:10.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:10.896 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:10.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:10.997 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:11.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:11.037 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:11.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:11.138 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:11.178 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:11.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:11.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:11.280 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:11.301 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:11.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:11.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:11.403 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:11.436 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:11.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:11.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:11.537 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:11.576 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:11.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:11.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:11.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:11 smithi184 ceph-mon[92527]: pgmap v2114: 119 pgs: 8 active+remapped+backfill_wait, 3 active+remapped+backfilling, 108 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 5.5 KiB/s rd, 12 KiB/s wr, 579 op/s; 82008/328446 objects misplaced (24.968%); 34 MiB/s, 546 objects/s recovering 2024-09-13T15:00:11.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:11 smithi151 ceph-mon[104450]: pgmap v2114: 119 pgs: 8 active+remapped+backfill_wait, 3 active+remapped+backfilling, 108 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 5.5 KiB/s rd, 12 KiB/s wr, 579 op/s; 82008/328446 objects misplaced (24.968%); 34 MiB/s, 546 objects/s recovering 2024-09-13T15:00:11.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:11 smithi151 ceph-mon[108966]: pgmap v2114: 119 pgs: 8 active+remapped+backfill_wait, 3 active+remapped+backfilling, 108 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 5.5 KiB/s rd, 12 KiB/s wr, 579 op/s; 82008/328446 objects misplaced (24.968%); 34 MiB/s, 546 objects/s recovering 2024-09-13T15:00:11.678 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:11.713 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:11.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:11.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:11.815 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:11.850 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:11.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:11.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:11.951 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:11.979 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:11.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:11.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:12.081 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:12.136 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:12.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:12.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:12.238 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:12.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:12.278 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:12.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:12.379 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:12.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:12.415 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:12.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:12.516 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:12.551 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:12.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:12.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:12.653 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:12.707 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T15:00:12.720 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:12.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:12.721 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:12.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:12.822 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:12.857 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:12.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:12.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:12.959 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:13.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:13.043 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:13.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:13.144 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:13.149 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:13.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:13.174 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:13.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:13.209 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:13 smithi151 ceph-mon[108966]: pgmap v2115: 119 pgs: 8 active+remapped+backfill_wait, 3 active+remapped+backfilling, 108 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 4.3 KiB/s rd, 9.7 KiB/s wr, 415 op/s; 82008/327489 objects misplaced (25.041%); 22 MiB/s, 348 objects/s recovering 2024-09-13T15:00:13.209 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:13 smithi151 ceph-mon[108966]: 1.0 scrub starts 2024-09-13T15:00:13.274 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:13.312 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:13.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:13.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:13.414 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:13.447 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:13.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:13.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:13.549 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:13.582 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:13.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:13.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:13.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:13 smithi184 ceph-mon[92527]: pgmap v2115: 119 pgs: 8 active+remapped+backfill_wait, 3 active+remapped+backfilling, 108 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 4.3 KiB/s rd, 9.7 KiB/s wr, 415 op/s; 82008/327489 objects misplaced (25.041%); 22 MiB/s, 348 objects/s recovering 2024-09-13T15:00:13.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:13 smithi184 ceph-mon[92527]: 1.0 scrub starts 2024-09-13T15:00:13.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:13 smithi151 ceph-mon[104450]: pgmap v2115: 119 pgs: 8 active+remapped+backfill_wait, 3 active+remapped+backfilling, 108 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 4.3 KiB/s rd, 9.7 KiB/s wr, 415 op/s; 82008/327489 objects misplaced (25.041%); 22 MiB/s, 348 objects/s recovering 2024-09-13T15:00:13.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:13 smithi151 ceph-mon[104450]: 1.0 scrub starts 2024-09-13T15:00:13.685 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:13.722 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:13.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:13.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:13.824 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:13.863 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:13.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:13.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:13.966 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:14.003 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:14.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:14.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:14.105 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:14.141 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:14.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:14.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:14.243 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:14.281 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:14.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:14.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:14.294 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 6, 7, 1, 4, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T15:00:14.294 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:00:14.295 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [2, 3, 6, 7, 1, 4, 5] 2024-09-13T15:00:14.295 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd out 4 2024-09-13T15:00:14.383 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:14.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:14.416 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:14.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:14.517 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:14.552 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:14.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:14.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:14.654 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:14.693 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:14.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:14.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:14.795 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:14.834 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:14.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:14.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:14.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:00:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:00:14] "GET /metrics HTTP/1.1" 200 36752 "" "Prometheus/2.51.0" 2024-09-13T15:00:14.935 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:14.971 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:14.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:14.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:15.073 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:15.113 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:15.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:15.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:15.215 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:15.258 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:15.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:15.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:15.359 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:15.397 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:15.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:15.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:15.499 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:15.538 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:15.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:15.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:15.563 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:15 smithi151 ceph-mon[104450]: pgmap v2116: 119 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_wait, 3 active+remapped+backfilling, 107 active+clean; 6.4 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 3.8 KiB/s rd, 8.4 KiB/s wr, 901 op/s; 76260/313323 objects misplaced (24.339%); 34 MiB/s, 547 objects/s recovering 2024-09-13T15:00:15.563 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:15 smithi151 ceph-mon[108966]: pgmap v2116: 119 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_wait, 3 active+remapped+backfilling, 107 active+clean; 6.4 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 3.8 KiB/s rd, 8.4 KiB/s wr, 901 op/s; 76260/313323 objects misplaced (24.339%); 34 MiB/s, 547 objects/s recovering 2024-09-13T15:00:15.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:15 smithi184 ceph-mon[92527]: pgmap v2116: 119 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_wait, 3 active+remapped+backfilling, 107 active+clean; 6.4 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 3.8 KiB/s rd, 8.4 KiB/s wr, 901 op/s; 76260/313323 objects misplaced (24.339%); 34 MiB/s, 547 objects/s recovering 2024-09-13T15:00:15.639 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:15.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:15.683 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:15.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:15.784 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:15.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:15.826 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:15.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:15.927 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:15.962 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:15.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:15.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:16.064 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:16.097 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:16.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:16.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:16.199 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:16.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:16.242 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:16.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:16.343 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:16.381 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:16.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:16.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:16.483 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:16.522 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:16.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:16.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:16.624 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:16.662 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:16.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:16.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:16.764 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:16.800 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:16.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:16.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:16.902 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:16.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:16.933 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:16.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:17.034 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:17.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:17.068 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:17.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:17.169 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:17.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:17.203 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:17.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:17.304 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:17.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:17.338 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:17.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:17.439 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:17.486 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:17.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:17.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:17.553 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:17.587 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:17.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:17 smithi184 ceph-mon[92527]: pgmap v2117: 119 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_wait, 3 active+remapped+backfilling, 107 active+clean; 6.4 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 3.3 KiB/s rd, 7.4 KiB/s wr, 790 op/s; 76260/313323 objects misplaced (24.339%); 30 MiB/s, 480 objects/s recovering 2024-09-13T15:00:17.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:17.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:17 smithi151 ceph-mon[108966]: pgmap v2117: 119 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_wait, 3 active+remapped+backfilling, 107 active+clean; 6.4 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 3.3 KiB/s rd, 7.4 KiB/s wr, 790 op/s; 76260/313323 objects misplaced (24.339%); 30 MiB/s, 480 objects/s recovering 2024-09-13T15:00:17.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:17.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:17 smithi151 ceph-mon[104450]: pgmap v2117: 119 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_wait, 3 active+remapped+backfilling, 107 active+clean; 6.4 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 3.3 KiB/s rd, 7.4 KiB/s wr, 790 op/s; 76260/313323 objects misplaced (24.339%); 30 MiB/s, 480 objects/s recovering 2024-09-13T15:00:17.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:17.621 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:17.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:17.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:17.722 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:17.756 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:17.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:17.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:17.857 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:17.893 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:17.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:17.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:17.995 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:18.031 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:18.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:18.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:18.053 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:18.133 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:18.182 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:18.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:18.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:18.284 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:18.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:18.320 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:18.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:18.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:18 smithi151 ceph-mon[104450]: 1.0 scrub starts 2024-09-13T15:00:18.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:18 smithi151 ceph-mon[108966]: 1.0 scrub starts 2024-09-13T15:00:18.421 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:18.453 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:18.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:18.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:18.554 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:18.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:18.587 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:18.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:18.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:18 smithi184 ceph-mon[92527]: 1.0 scrub starts 2024-09-13T15:00:18.688 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:18.725 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:18.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:18.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:18.827 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:18.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:18.861 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:18.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:18.963 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:18.995 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:18.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:18.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:19.097 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:19.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:19.130 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:19.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:19.231 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:19.267 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:19.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:19.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:19.315 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:19 smithi151 ceph-mon[104450]: pgmap v2118: 119 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_wait, 3 active+remapped+backfilling, 107 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 2.8 KiB/s rd, 6.3 KiB/s wr, 992 op/s; 71552/301995 objects misplaced (23.693%); 36 MiB/s, 583 objects/s recovering 2024-09-13T15:00:19.315 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:19 smithi151 ceph-mon[108966]: pgmap v2118: 119 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_wait, 3 active+remapped+backfilling, 107 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 2.8 KiB/s rd, 6.3 KiB/s wr, 992 op/s; 71552/301995 objects misplaced (23.693%); 36 MiB/s, 583 objects/s recovering 2024-09-13T15:00:19.369 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:19.404 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:19.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:19.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:19.505 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:19 smithi184 ceph-mon[92527]: pgmap v2118: 119 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_wait, 3 active+remapped+backfilling, 107 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 2.8 KiB/s rd, 6.3 KiB/s wr, 992 op/s; 71552/301995 objects misplaced (23.693%); 36 MiB/s, 583 objects/s recovering 2024-09-13T15:00:19.506 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:19.544 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:19.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:19.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:19.646 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:19.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:19.683 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:19.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:19.784 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:19.832 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:19.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:19.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:19.934 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:19.977 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:19.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:19.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:20.079 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:20.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:20.116 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:20.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:20.217 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:20.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:20.253 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:20.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:20.354 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:20.401 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:20.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:20.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:20.504 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:20.536 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:20.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:20.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:20.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:00:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:00:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:00:20.638 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:20.682 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:20.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:20.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:20.784 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:20.823 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:20.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:20.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:20.925 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:20.959 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:20.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:20.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:21.061 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:21.100 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:21.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:21.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:21.202 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:21.247 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:21.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:21.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:21.348 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:21.380 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:21.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:21.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:21.398 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T15:00:21.398 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:00:21.431 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T15:00:21.432 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:00:21.461 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T15:00:21.461 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:00:21.481 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:21.493 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T15:00:21.493 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:00:21.494 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:21 smithi151 ceph-mon[104450]: pgmap v2119: 119 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_wait, 3 active+remapped+backfilling, 107 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 879 op/s; 70662/298287 objects misplaced (23.689%); 22 MiB/s, 351 objects/s recovering 2024-09-13T15:00:21.494 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:21 smithi151 ceph-mon[108966]: pgmap v2119: 119 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_wait, 3 active+remapped+backfilling, 107 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 879 op/s; 70662/298287 objects misplaced (23.689%); 22 MiB/s, 351 objects/s recovering 2024-09-13T15:00:21.521 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:21.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:21.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:21.541 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T15:00:21.542 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:00:21.579 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T15:00:21.579 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:00:21.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:21 smithi184 ceph-mon[92527]: pgmap v2119: 119 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_wait, 3 active+remapped+backfilling, 107 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 879 op/s; 70662/298287 objects misplaced (23.689%); 22 MiB/s, 351 objects/s recovering 2024-09-13T15:00:21.610 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T15:00:21.610 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:00:21.623 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:21.647 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T15:00:21.647 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:00:21.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:21.656 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:21.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:21.757 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:21.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:21.796 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:21.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:21.897 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:21.937 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:21.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:21.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:22.041 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:22.076 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:22.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:22.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:22.178 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:22.220 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:22.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:22.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:22.322 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:22.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:22.358 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:22.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:22.459 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:22.501 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:22.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:22.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:22.519 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:22 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3300505647' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:22.519 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:22 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3300505647' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:22.603 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:22.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:22 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3300505647' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:22.642 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:22.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:22.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:22.744 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:22.786 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:22.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:22.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:22.888 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:22.917 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:22.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:22.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:23.019 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:23.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:23.062 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:23.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:23.163 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:23.198 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:23.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:23.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:23.300 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:23.329 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:23.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:23.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:23.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:23 smithi151 ceph-mon[104450]: pgmap v2120: 119 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_wait, 3 active+remapped+backfilling, 107 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 837 op/s; 70662/298287 objects misplaced (23.689%); 22 MiB/s, 349 objects/s recovering 2024-09-13T15:00:23.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:23 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/869256008' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:00:23.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:23 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:00:23.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:23 smithi151 ceph-mon[108966]: pgmap v2120: 119 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_wait, 3 active+remapped+backfilling, 107 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 837 op/s; 70662/298287 objects misplaced (23.689%); 22 MiB/s, 349 objects/s recovering 2024-09-13T15:00:23.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:23 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/869256008' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:00:23.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:23 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:00:23.431 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:23.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:23.463 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:23.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:23.503 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-13T15:00:23.565 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:23.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:23.597 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:23.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:23.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:23 smithi184 ceph-mon[92527]: pgmap v2120: 119 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_wait, 3 active+remapped+backfilling, 107 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 837 op/s; 70662/298287 objects misplaced (23.689%); 22 MiB/s, 349 objects/s recovering 2024-09-13T15:00:23.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:23 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/869256008' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:00:23.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:23 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:00:23.698 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:23.738 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:23.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:23.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:23.840 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:23.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:23 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a[104427]: 2024-09-13T15:00:23.606+0000 7fe9da14f640 -1 mon.a@0(leader).osd e1287 definitely_dead 0 2024-09-13T15:00:23.871 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:23.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:23.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:23.973 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:24.008 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:24.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:24.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:24.110 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:24.145 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:24.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:24.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:24.248 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:24.322 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:24.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:24.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:24.424 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:24.473 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:24.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:24.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:24.498 INFO:teuthology.orchestra.run.smithi151.stderr:marked out osd.4. 2024-09-13T15:00:24.521 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:24 smithi151 ceph-mon[104450]: 1.0 scrub starts 2024-09-13T15:00:24.521 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:24 smithi151 ceph-mon[104450]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:00:24.521 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:24 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:00:24.521 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:24 smithi151 ceph-mon[104450]: osdmap e1287: 8 total, 8 up, 7 in 2024-09-13T15:00:24.521 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:24 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:00:24.521 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:24 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:00:24.521 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:24 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/4240225879' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-13T15:00:24.521 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:24 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-13T15:00:24.521 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:24 smithi151 ceph-mon[104450]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-13T15:00:24.522 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:24 smithi151 ceph-mon[108966]: 1.0 scrub starts 2024-09-13T15:00:24.522 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:24 smithi151 ceph-mon[108966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:00:24.522 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:24 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:00:24.522 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:24 smithi151 ceph-mon[108966]: osdmap e1287: 8 total, 8 up, 7 in 2024-09-13T15:00:24.522 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:24 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:00:24.522 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:24 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:00:24.522 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:24 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/4240225879' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-13T15:00:24.522 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:24 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-13T15:00:24.522 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:24 smithi151 ceph-mon[108966]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-13T15:00:24.562 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:00:24.576 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:24.599 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:24.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:24.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:24.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:24 smithi184 ceph-mon[92527]: 1.0 scrub starts 2024-09-13T15:00:24.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:24 smithi184 ceph-mon[92527]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:00:24.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:24 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:00:24.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:24 smithi184 ceph-mon[92527]: osdmap e1287: 8 total, 8 up, 7 in 2024-09-13T15:00:24.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:24 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:00:24.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:24 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:00:24.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:24 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/4240225879' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-13T15:00:24.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:24 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-13T15:00:24.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:24 smithi184 ceph-mon[92527]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-13T15:00:24.701 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:24.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:24.726 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:24.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:24.827 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:24.849 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:24.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:24.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:24.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:00:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:00:24] "GET /metrics HTTP/1.1" 200 36759 "" "Prometheus/2.51.0" 2024-09-13T15:00:24.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:24.976 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:24.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:24.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:25.078 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:25.111 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:25.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:25.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:25.213 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:25.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:25.254 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:25.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:25.356 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:25.382 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:25.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:25.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:25.484 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:25.508 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:25.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:25.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:25.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:25 smithi184 ceph-mon[92527]: pgmap v2122: 119 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_wait, 3 active+remapped+backfilling, 107 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 996 op/s; 65218/283419 objects misplaced (23.011%); 25 MiB/s, 399 objects/s recovering 2024-09-13T15:00:25.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:25 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-13T15:00:25.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:25 smithi184 ceph-mon[92527]: osdmap e1288: 8 total, 8 up, 6 in 2024-09-13T15:00:25.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:25 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:00:25.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:25 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:00:25.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:25 smithi184 ceph-mon[92527]: osdmap e1289: 8 total, 8 up, 6 in 2024-09-13T15:00:25.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:25 smithi151 ceph-mon[104450]: pgmap v2122: 119 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_wait, 3 active+remapped+backfilling, 107 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 996 op/s; 65218/283419 objects misplaced (23.011%); 25 MiB/s, 399 objects/s recovering 2024-09-13T15:00:25.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:25 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-13T15:00:25.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:25 smithi151 ceph-mon[104450]: osdmap e1288: 8 total, 8 up, 6 in 2024-09-13T15:00:25.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:25 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:00:25.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:25 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:00:25.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:25 smithi151 ceph-mon[104450]: osdmap e1289: 8 total, 8 up, 6 in 2024-09-13T15:00:25.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:25 smithi151 ceph-mon[108966]: pgmap v2122: 119 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_wait, 3 active+remapped+backfilling, 107 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 996 op/s; 65218/283419 objects misplaced (23.011%); 25 MiB/s, 399 objects/s recovering 2024-09-13T15:00:25.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:25 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-13T15:00:25.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:25 smithi151 ceph-mon[108966]: osdmap e1288: 8 total, 8 up, 6 in 2024-09-13T15:00:25.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:25 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:00:25.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:25 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:00:25.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:25 smithi151 ceph-mon[108966]: osdmap e1289: 8 total, 8 up, 6 in 2024-09-13T15:00:25.610 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:25.634 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:25.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:25.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:25.718 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:25.736 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:25.763 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:25.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:25.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:25.865 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:25.890 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:25.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:25.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:25.992 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:26.015 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:26.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:26.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:26.117 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:26.147 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:26.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:26.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:26.249 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:26.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:26.292 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:26.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:26.359 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:00:26.393 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:26.445 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:26.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:26.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:26.521 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:26 smithi151 ceph-mon[104450]: 10.8 starting backfill to osd.3 from (0'0,0'0] MAX to 1287'9942 2024-09-13T15:00:26.521 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:26 smithi151 ceph-mon[104450]: 10.9 starting backfill to osd.3 from (0'0,0'0] MAX to 1287'9865 2024-09-13T15:00:26.521 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:26 smithi151 ceph-mon[104450]: 10.c starting backfill to osd.1 from (0'0,0'0] MAX to 1287'9838 2024-09-13T15:00:26.522 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:26 smithi151 ceph-mon[104450]: 10.d continuing backfill to osd.3 from (1275'5297,1287'10044] MIN to 1287'10044 2024-09-13T15:00:26.522 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:26 smithi151 ceph-mon[104450]: 10.d starting backfill to osd.7 from (0'0,0'0] MAX to 1287'10044 2024-09-13T15:00:26.522 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:26 smithi151 ceph-mon[104450]: 10.2 continuing backfill to osd.3 from (1275'5256,1287'10043] MIN to 1287'10043 2024-09-13T15:00:26.522 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:26 smithi151 ceph-mon[104450]: 10.2 starting backfill to osd.6 from (0'0,0'0] MAX to 1287'10043 2024-09-13T15:00:26.522 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:26 smithi151 ceph-mon[104450]: 10.b starting backfill to osd.2 from (0'0,0'0] MAX to 1287'9878 2024-09-13T15:00:26.522 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:26 smithi151 ceph-mon[104450]: 10.6 continuing backfill to osd.1 from (1275'4623,1287'9910] 10:65b386ac:::benchmark_data_smithi151_288181_object61855:head to 1287'9910 2024-09-13T15:00:26.522 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:26 smithi151 ceph-mon[104450]: 10.6 starting backfill to osd.6 from (0'0,0'0] MAX to 1287'9910 2024-09-13T15:00:26.523 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:26 smithi151 ceph-mon[104450]: 10.c continuing backfill to osd.3 from (1275'5059,1287'9838] 10:32e895ce:::benchmark_data_smithi151_288181_object59958:head to 1287'9838 2024-09-13T15:00:26.523 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:26 smithi151 ceph-mon[104450]: 10.c continuing backfill to osd.6 from (1275'5059,1287'9838] 10:314bea60:::benchmark_data_smithi151_288181_object34131:head to 1287'9838 2024-09-13T15:00:26.523 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:26 smithi151 ceph-mon[104450]: 10.8 continuing backfill to osd.5 from (1274'3756,1287'9942] 10:1369da19:::benchmark_data_smithi151_288181_object21246:head to 1287'9942 2024-09-13T15:00:26.523 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:26 smithi151 ceph-mon[104450]: 1.e scrub starts 2024-09-13T15:00:26.523 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:26 smithi151 ceph-mon[104450]: osdmap e1290: 8 total, 8 up, 6 in 2024-09-13T15:00:26.524 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:26 smithi151 ceph-mon[108966]: 10.8 starting backfill to osd.3 from (0'0,0'0] MAX to 1287'9942 2024-09-13T15:00:26.524 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:26 smithi151 ceph-mon[108966]: 10.9 starting backfill to osd.3 from (0'0,0'0] MAX to 1287'9865 2024-09-13T15:00:26.524 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:26 smithi151 ceph-mon[108966]: 10.c starting backfill to osd.1 from (0'0,0'0] MAX to 1287'9838 2024-09-13T15:00:26.524 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:26 smithi151 ceph-mon[108966]: 10.d continuing backfill to osd.3 from (1275'5297,1287'10044] MIN to 1287'10044 2024-09-13T15:00:26.525 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:26 smithi151 ceph-mon[108966]: 10.d starting backfill to osd.7 from (0'0,0'0] MAX to 1287'10044 2024-09-13T15:00:26.525 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:26 smithi151 ceph-mon[108966]: 10.2 continuing backfill to osd.3 from (1275'5256,1287'10043] MIN to 1287'10043 2024-09-13T15:00:26.525 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:26 smithi151 ceph-mon[108966]: 10.2 starting backfill to osd.6 from (0'0,0'0] MAX to 1287'10043 2024-09-13T15:00:26.525 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:26 smithi151 ceph-mon[108966]: 10.b starting backfill to osd.2 from (0'0,0'0] MAX to 1287'9878 2024-09-13T15:00:26.526 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:26 smithi151 ceph-mon[108966]: 10.6 continuing backfill to osd.1 from (1275'4623,1287'9910] 10:65b386ac:::benchmark_data_smithi151_288181_object61855:head to 1287'9910 2024-09-13T15:00:26.526 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:26 smithi151 ceph-mon[108966]: 10.6 starting backfill to osd.6 from (0'0,0'0] MAX to 1287'9910 2024-09-13T15:00:26.526 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:26 smithi151 ceph-mon[108966]: 10.c continuing backfill to osd.3 from (1275'5059,1287'9838] 10:32e895ce:::benchmark_data_smithi151_288181_object59958:head to 1287'9838 2024-09-13T15:00:26.527 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:26 smithi151 ceph-mon[108966]: 10.c continuing backfill to osd.6 from (1275'5059,1287'9838] 10:314bea60:::benchmark_data_smithi151_288181_object34131:head to 1287'9838 2024-09-13T15:00:26.527 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:26 smithi151 ceph-mon[108966]: 10.8 continuing backfill to osd.5 from (1274'3756,1287'9942] 10:1369da19:::benchmark_data_smithi151_288181_object21246:head to 1287'9942 2024-09-13T15:00:26.527 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:26 smithi151 ceph-mon[108966]: 1.e scrub starts 2024-09-13T15:00:26.527 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:26 smithi151 ceph-mon[108966]: osdmap e1290: 8 total, 8 up, 6 in 2024-09-13T15:00:26.548 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:26.576 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:26.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:26.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:26.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:26 smithi184 ceph-mon[92527]: 10.8 starting backfill to osd.3 from (0'0,0'0] MAX to 1287'9942 2024-09-13T15:00:26.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:26 smithi184 ceph-mon[92527]: 10.9 starting backfill to osd.3 from (0'0,0'0] MAX to 1287'9865 2024-09-13T15:00:26.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:26 smithi184 ceph-mon[92527]: 10.c starting backfill to osd.1 from (0'0,0'0] MAX to 1287'9838 2024-09-13T15:00:26.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:26 smithi184 ceph-mon[92527]: 10.d continuing backfill to osd.3 from (1275'5297,1287'10044] MIN to 1287'10044 2024-09-13T15:00:26.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:26 smithi184 ceph-mon[92527]: 10.d starting backfill to osd.7 from (0'0,0'0] MAX to 1287'10044 2024-09-13T15:00:26.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:26 smithi184 ceph-mon[92527]: 10.2 continuing backfill to osd.3 from (1275'5256,1287'10043] MIN to 1287'10043 2024-09-13T15:00:26.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:26 smithi184 ceph-mon[92527]: 10.2 starting backfill to osd.6 from (0'0,0'0] MAX to 1287'10043 2024-09-13T15:00:26.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:26 smithi184 ceph-mon[92527]: 10.b starting backfill to osd.2 from (0'0,0'0] MAX to 1287'9878 2024-09-13T15:00:26.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:26 smithi184 ceph-mon[92527]: 10.6 continuing backfill to osd.1 from (1275'4623,1287'9910] 10:65b386ac:::benchmark_data_smithi151_288181_object61855:head to 1287'9910 2024-09-13T15:00:26.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:26 smithi184 ceph-mon[92527]: 10.6 starting backfill to osd.6 from (0'0,0'0] MAX to 1287'9910 2024-09-13T15:00:26.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:26 smithi184 ceph-mon[92527]: 10.c continuing backfill to osd.3 from (1275'5059,1287'9838] 10:32e895ce:::benchmark_data_smithi151_288181_object59958:head to 1287'9838 2024-09-13T15:00:26.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:26 smithi184 ceph-mon[92527]: 10.c continuing backfill to osd.6 from (1275'5059,1287'9838] 10:314bea60:::benchmark_data_smithi151_288181_object34131:head to 1287'9838 2024-09-13T15:00:26.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:26 smithi184 ceph-mon[92527]: 10.8 continuing backfill to osd.5 from (1274'3756,1287'9942] 10:1369da19:::benchmark_data_smithi151_288181_object21246:head to 1287'9942 2024-09-13T15:00:26.606 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:26 smithi184 ceph-mon[92527]: 1.e scrub starts 2024-09-13T15:00:26.606 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:26 smithi184 ceph-mon[92527]: osdmap e1290: 8 total, 8 up, 6 in 2024-09-13T15:00:26.679 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:26.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:26.711 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:26.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:26.812 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:26.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:26.877 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:26.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:26.979 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:27.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:27.025 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:27.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:27.126 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:27.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:27.175 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:27.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:27.277 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:27.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:27.318 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:27.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:27.419 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:27.454 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:27.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:27.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:27.538 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:27 smithi151 ceph-mon[108966]: pgmap v2125: 119 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_wait, 3 active+remapped+backfilling, 107 active+clean; 5.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 825 op/s; 65218/283419 objects misplaced (23.011%); 20 MiB/s, 322 objects/s recovering 2024-09-13T15:00:27.538 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:27 smithi151 ceph-mon[108966]: 1.2 scrub starts 2024-09-13T15:00:27.539 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:27 smithi151 ceph-mon[108966]: 10.1 continuing backfill to osd.1 from (1275'5171,1287'9946] MIN to 1287'9946 2024-09-13T15:00:27.539 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:27 smithi151 ceph-mon[108966]: 10.1 starting backfill to osd.2 from (0'0,0'0] MAX to 1287'9946 2024-09-13T15:00:27.539 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:27 smithi151 ceph-mon[108966]: 10.e starting backfill to osd.5 from (0'0,0'0] MAX to 1287'9718 2024-09-13T15:00:27.539 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:27 smithi151 ceph-mon[108966]: 10.4 continuing backfill to osd.1 from (1275'5255,1287'9957] MIN to 1287'9957 2024-09-13T15:00:27.539 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:27 smithi151 ceph-mon[108966]: 10.4 starting backfill to osd.2 from (0'0,0'0] MAX to 1287'9957 2024-09-13T15:00:27.539 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:27 smithi151 ceph-mon[108966]: 10.0 starting backfill to osd.5 from (0'0,0'0] MAX to 1287'9934 2024-09-13T15:00:27.539 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:27 smithi151 ceph-mon[104450]: pgmap v2125: 119 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_wait, 3 active+remapped+backfilling, 107 active+clean; 5.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 825 op/s; 65218/283419 objects misplaced (23.011%); 20 MiB/s, 322 objects/s recovering 2024-09-13T15:00:27.539 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:27 smithi151 ceph-mon[104450]: 1.2 scrub starts 2024-09-13T15:00:27.539 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:27 smithi151 ceph-mon[104450]: 10.1 continuing backfill to osd.1 from (1275'5171,1287'9946] MIN to 1287'9946 2024-09-13T15:00:27.539 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:27 smithi151 ceph-mon[104450]: 10.1 starting backfill to osd.2 from (0'0,0'0] MAX to 1287'9946 2024-09-13T15:00:27.539 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:27 smithi151 ceph-mon[104450]: 10.e starting backfill to osd.5 from (0'0,0'0] MAX to 1287'9718 2024-09-13T15:00:27.540 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:27 smithi151 ceph-mon[104450]: 10.4 continuing backfill to osd.1 from (1275'5255,1287'9957] MIN to 1287'9957 2024-09-13T15:00:27.540 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:27 smithi151 ceph-mon[104450]: 10.4 starting backfill to osd.2 from (0'0,0'0] MAX to 1287'9957 2024-09-13T15:00:27.540 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:27 smithi151 ceph-mon[104450]: 10.0 starting backfill to osd.5 from (0'0,0'0] MAX to 1287'9934 2024-09-13T15:00:27.556 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:27.593 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:27.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:27.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:27.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:27 smithi184 ceph-mon[92527]: pgmap v2125: 119 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_wait, 3 active+remapped+backfilling, 107 active+clean; 5.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 825 op/s; 65218/283419 objects misplaced (23.011%); 20 MiB/s, 322 objects/s recovering 2024-09-13T15:00:27.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:27 smithi184 ceph-mon[92527]: 1.2 scrub starts 2024-09-13T15:00:27.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:27 smithi184 ceph-mon[92527]: 10.1 continuing backfill to osd.1 from (1275'5171,1287'9946] MIN to 1287'9946 2024-09-13T15:00:27.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:27 smithi184 ceph-mon[92527]: 10.1 starting backfill to osd.2 from (0'0,0'0] MAX to 1287'9946 2024-09-13T15:00:27.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:27 smithi184 ceph-mon[92527]: 10.e starting backfill to osd.5 from (0'0,0'0] MAX to 1287'9718 2024-09-13T15:00:27.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:27 smithi184 ceph-mon[92527]: 10.4 continuing backfill to osd.1 from (1275'5255,1287'9957] MIN to 1287'9957 2024-09-13T15:00:27.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:27 smithi184 ceph-mon[92527]: 10.4 starting backfill to osd.2 from (0'0,0'0] MAX to 1287'9957 2024-09-13T15:00:27.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:27 smithi184 ceph-mon[92527]: 10.0 starting backfill to osd.5 from (0'0,0'0] MAX to 1287'9934 2024-09-13T15:00:27.695 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:27.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:27.742 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:27.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:27.843 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:27.876 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:27.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:27.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:27.978 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:28.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:28.016 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:28.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:28.079 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T15:00:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-13T15:00:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-13T15:00:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-13T15:00:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:179783 8.2 8:4958d2ee:::benchmark_data_smithi151_229097_object54915:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1189)", 2024-09-13T15:00:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:52:11.555141+0000", 2024-09-13T15:00:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 496.52020911300002, 2024-09-13T15:00:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0450786999999999, 2024-09-13T15:00:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:00:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:00:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:00:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T15:00:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T15:00:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 179783 2024-09-13T15:00:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:00:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:00:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.555141+0000", 2024-09-13T15:00:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:00:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.555141+0000", 2024-09-13T15:00:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:00:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.555142+0000", 2024-09-13T15:00:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4309999999999999e-06 2024-09-13T15:00:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:00:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.555147+0000", 2024-09-13T15:00:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6349999999999997e-06 2024-09-13T15:00:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:00:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.555161+0000", 2024-09-13T15:00:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3672999999999999e-05 2024-09-13T15:00:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:00:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.555169+0000", 2024-09-13T15:00:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2309999999999996e-06 2024-09-13T15:00:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.556122+0000", 2024-09-13T15:00:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00095323700000000001 2024-09-13T15:00:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:00:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.556131+0000", 2024-09-13T15:00:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.242e-06 2024-09-13T15:00:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.583362+0000", 2024-09-13T15:00:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.027230997 2024-09-13T15:00:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:00:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.583470+0000", 2024-09-13T15:00:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108377 2024-09-13T15:00:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:00:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.583587+0000", 2024-09-13T15:00:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011692 2024-09-13T15:00:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.593593+0000", 2024-09-13T15:00:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010006045999999999 2024-09-13T15:00:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:00:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.593640+0000", 2024-09-13T15:00:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6998e-05 2024-09-13T15:00:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.600119+0000", 2024-09-13T15:00:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0064793000000000003 2024-09-13T15:00:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:00:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.600174+0000", 2024-09-13T15:00:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5169e-05 2024-09-13T15:00:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:00:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.600220+0000", 2024-09-13T15:00:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5444000000000001e-05 2024-09-13T15:00:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:00:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:179785 8.2 8:4a59eb92:::benchmark_data_smithi151_229097_object54917:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1189)", 2024-09-13T15:00:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:52:11.557035+0000", 2024-09-13T15:00:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 496.51831471499997, 2024-09-13T15:00:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0433434539999999, 2024-09-13T15:00:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:00:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:00:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:00:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T15:00:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T15:00:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 179785 2024-09-13T15:00:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:00:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:00:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.557035+0000", 2024-09-13T15:00:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:00:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.557035+0000", 2024-09-13T15:00:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:00:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.557036+0000", 2024-09-13T15:00:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4900000000000005e-07 2024-09-13T15:00:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:00:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.557040+0000", 2024-09-13T15:00:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9369999999999997e-06 2024-09-13T15:00:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:00:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.557054+0000", 2024-09-13T15:00:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3835000000000001e-05 2024-09-13T15:00:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:00:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.557061+0000", 2024-09-13T15:00:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4059999999999999e-06 2024-09-13T15:00:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.557123+0000", 2024-09-13T15:00:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1369000000000001e-05 2024-09-13T15:00:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:00:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:11.557131+0000", 2024-09-13T15:00:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1650000000000006e-06 2024-09-13T15:00:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.586929+0000", 2024-09-13T15:00:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.029797485 2024-09-13T15:00:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:00:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.587042+0000", 2024-09-13T15:00:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000113081 2024-09-13T15:00:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:00:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.587155+0000", 2024-09-13T15:00:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000112961 2024-09-13T15:00:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.593630+0000", 2024-09-13T15:00:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0064749960000000002 2024-09-13T15:00:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:00:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.593645+0000", 2024-09-13T15:00:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.505e-05 2024-09-13T15:00:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.600273+0000", 2024-09-13T15:00:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0066280419999999998 2024-09-13T15:00:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:00:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.600340+0000", 2024-09-13T15:00:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7069000000000004e-05 2024-09-13T15:00:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:00:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:52:12.600379+0000", 2024-09-13T15:00:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9208999999999997e-05 2024-09-13T15:00:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:00:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:235893 8.9 8:9de210c4:::benchmark_data_smithi151_229097_object111025:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1203)", 2024-09-13T15:00:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:15.854275+0000", 2024-09-13T15:00:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 432.22107551599998, 2024-09-13T15:00:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1044303580000001, 2024-09-13T15:00:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:00:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:00:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:00:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T15:00:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T15:00:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 235893 2024-09-13T15:00:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:00:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:00:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.854275+0000", 2024-09-13T15:00:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:00:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.854275+0000", 2024-09-13T15:00:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:00:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.854276+0000", 2024-09-13T15:00:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1399999999999995e-07 2024-09-13T15:00:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:00:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.854280+0000", 2024-09-13T15:00:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6399999999999996e-06 2024-09-13T15:00:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:00:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.854292+0000", 2024-09-13T15:00:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1596e-05 2024-09-13T15:00:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:00:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.854301+0000", 2024-09-13T15:00:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0359999999999995e-06 2024-09-13T15:00:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.856832+0000", 2024-09-13T15:00:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0025309519999999999 2024-09-13T15:00:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:00:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.856839+0000", 2024-09-13T15:00:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4499999999999998e-06 2024-09-13T15:00:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.955517+0000", 2024-09-13T15:00:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0986773240000001 2024-09-13T15:00:28.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:00:28.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.955603+0000", 2024-09-13T15:00:28.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6595999999999996e-05 2024-09-13T15:00:28.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,6", 2024-09-13T15:00:28.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.955709+0000", 2024-09-13T15:00:28.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105899 2024-09-13T15:00:28.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.958555+0000", 2024-09-13T15:00:28.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028462119999999999 2024-09-13T15:00:28.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:00:28.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.958570+0000", 2024-09-13T15:00:28.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4888999999999999e-05 2024-09-13T15:00:28.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.958622+0000", 2024-09-13T15:00:28.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1595000000000003e-05 2024-09-13T15:00:28.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:00:28.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.958658+0000", 2024-09-13T15:00:28.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6480000000000003e-05 2024-09-13T15:00:28.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:00:28.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.958705+0000", 2024-09-13T15:00:28.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6774999999999999e-05 2024-09-13T15:00:28.102 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.102 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:00:28.102 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.102 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:235934 8.9 8:9e1a02bc:::benchmark_data_smithi151_229097_object111066:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1203)", 2024-09-13T15:00:28.102 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:15.893447+0000", 2024-09-13T15:00:28.103 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 432.181903551, 2024-09-13T15:00:28.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.069973793, 2024-09-13T15:00:28.103 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:00:28.103 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:00:28.103 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:28.103 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:00:28.103 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T15:00:28.103 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T15:00:28.103 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 235934 2024-09-13T15:00:28.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.103 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:00:28.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:00:28.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.893447+0000", 2024-09-13T15:00:28.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:00:28.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.893447+0000", 2024-09-13T15:00:28.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:00:28.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.893448+0000", 2024-09-13T15:00:28.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2020000000000001e-06 2024-09-13T15:00:28.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:00:28.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.893453+0000", 2024-09-13T15:00:28.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9150000000000004e-06 2024-09-13T15:00:28.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:00:28.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.893463+0000", 2024-09-13T15:00:28.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0695e-05 2024-09-13T15:00:28.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:00:28.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.893470+0000", 2024-09-13T15:00:28.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2999999999999998e-06 2024-09-13T15:00:28.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.895021+0000", 2024-09-13T15:00:28.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015513510000000001 2024-09-13T15:00:28.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:00:28.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.895029+0000", 2024-09-13T15:00:28.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7670000000000003e-06 2024-09-13T15:00:28.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.958753+0000", 2024-09-13T15:00:28.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.063724602 2024-09-13T15:00:28.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:00:28.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.958874+0000", 2024-09-13T15:00:28.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012048300000000001 2024-09-13T15:00:28.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,6", 2024-09-13T15:00:28.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.958996+0000", 2024-09-13T15:00:28.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000122263 2024-09-13T15:00:28.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.963220+0000", 2024-09-13T15:00:28.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0042233879999999998 2024-09-13T15:00:28.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.963336+0000", 2024-09-13T15:00:28.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116451 2024-09-13T15:00:28.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:00:28.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.963344+0000", 2024-09-13T15:00:28.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0139999999999998e-06 2024-09-13T15:00:28.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:00:28.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.963384+0000", 2024-09-13T15:00:28.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9756999999999999e-05 2024-09-13T15:00:28.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:00:28.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.963420+0000", 2024-09-13T15:00:28.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6604999999999999e-05 2024-09-13T15:00:28.109 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.109 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:00:28.109 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.110 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:235937 8.9 8:96f0bdc7:::benchmark_data_smithi151_229097_object111069:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1203)", 2024-09-13T15:00:28.110 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:15.894610+0000", 2024-09-13T15:00:28.110 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 432.18074042199999, 2024-09-13T15:00:28.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0690157899999999, 2024-09-13T15:00:28.110 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:00:28.110 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:00:28.110 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:28.110 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:00:28.110 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T15:00:28.110 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T15:00:28.110 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 235937 2024-09-13T15:00:28.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.111 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:00:28.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:00:28.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.894610+0000", 2024-09-13T15:00:28.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:00:28.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.894610+0000", 2024-09-13T15:00:28.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:00:28.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.894611+0000", 2024-09-13T15:00:28.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9599999999999998e-07 2024-09-13T15:00:28.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:00:28.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.894615+0000", 2024-09-13T15:00:28.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.374e-06 2024-09-13T15:00:28.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:00:28.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.894633+0000", 2024-09-13T15:00:28.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8014999999999999e-05 2024-09-13T15:00:28.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:00:28.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.894641+0000", 2024-09-13T15:00:28.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5259999999999996e-06 2024-09-13T15:00:28.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.895049+0000", 2024-09-13T15:00:28.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00040882400000000001 2024-09-13T15:00:28.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:00:28.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.895055+0000", 2024-09-13T15:00:28.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3460000000000001e-06 2024-09-13T15:00:28.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.961125+0000", 2024-09-13T15:00:28.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.066070098 2024-09-13T15:00:28.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:00:28.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.961224+0000", 2024-09-13T15:00:28.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8894000000000006e-05 2024-09-13T15:00:28.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,6", 2024-09-13T15:00:28.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.961293+0000", 2024-09-13T15:00:28.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9479000000000003e-05 2024-09-13T15:00:28.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.963473+0000", 2024-09-13T15:00:28.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021798410000000001 2024-09-13T15:00:28.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.963524+0000", 2024-09-13T15:00:28.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0769000000000001e-05 2024-09-13T15:00:28.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:00:28.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.963537+0000", 2024-09-13T15:00:28.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.364e-05 2024-09-13T15:00:28.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:00:28.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.963581+0000", 2024-09-13T15:00:28.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4017999999999998e-05 2024-09-13T15:00:28.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:00:28.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.963626+0000", 2024-09-13T15:00:28.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4070000000000003e-05 2024-09-13T15:00:28.117 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.117 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:00:28.117 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.117 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:235938 8.9 8:9ef41bc5:::benchmark_data_smithi151_229097_object111070:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1203)", 2024-09-13T15:00:28.117 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:15.896036+0000", 2024-09-13T15:00:28.117 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 432.17931434500002, 2024-09-13T15:00:28.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0761522379999999, 2024-09-13T15:00:28.117 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:00:28.117 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:00:28.117 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:28.118 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:00:28.118 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T15:00:28.118 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T15:00:28.118 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 235938 2024-09-13T15:00:28.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.118 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:00:28.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:00:28.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.896036+0000", 2024-09-13T15:00:28.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:00:28.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.896036+0000", 2024-09-13T15:00:28.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:00:28.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.896037+0000", 2024-09-13T15:00:28.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0690000000000001e-06 2024-09-13T15:00:28.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:00:28.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.896041+0000", 2024-09-13T15:00:28.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1899999999999997e-06 2024-09-13T15:00:28.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:00:28.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.896053+0000", 2024-09-13T15:00:28.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1704e-05 2024-09-13T15:00:28.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:00:28.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.896060+0000", 2024-09-13T15:00:28.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7360000000000004e-06 2024-09-13T15:00:28.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.896111+0000", 2024-09-13T15:00:28.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1544e-05 2024-09-13T15:00:28.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:00:28.121 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.896119+0000", 2024-09-13T15:00:28.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7719999999999994e-06 2024-09-13T15:00:28.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.121 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.963668+0000", 2024-09-13T15:00:28.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.067549584 2024-09-13T15:00:28.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:00:28.121 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.963756+0000", 2024-09-13T15:00:28.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7744999999999994e-05 2024-09-13T15:00:28.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,6", 2024-09-13T15:00:28.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.963858+0000", 2024-09-13T15:00:28.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101939 2024-09-13T15:00:28.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.972028+0000", 2024-09-13T15:00:28.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0081694520000000007 2024-09-13T15:00:28.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.972077+0000", 2024-09-13T15:00:28.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9166000000000003e-05 2024-09-13T15:00:28.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:00:28.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.972132+0000", 2024-09-13T15:00:28.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4795e-05 2024-09-13T15:00:28.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:00:28.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.972163+0000", 2024-09-13T15:00:28.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1544000000000002e-05 2024-09-13T15:00:28.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:00:28.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.972188+0000", 2024-09-13T15:00:28.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4998e-05 2024-09-13T15:00:28.124 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.124 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:00:28.124 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.125 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:235940 8.9 8:91a19492:::benchmark_data_smithi151_229097_object111072:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1203)", 2024-09-13T15:00:28.125 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:15.899895+0000", 2024-09-13T15:00:28.125 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 432.17545543799997, 2024-09-13T15:00:28.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.07244626, 2024-09-13T15:00:28.125 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:00:28.125 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:00:28.125 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:28.125 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:00:28.125 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T15:00:28.125 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T15:00:28.125 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 235940 2024-09-13T15:00:28.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.125 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:00:28.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:00:28.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.899895+0000", 2024-09-13T15:00:28.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:00:28.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.899895+0000", 2024-09-13T15:00:28.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:00:28.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.899896+0000", 2024-09-13T15:00:28.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.362e-06 2024-09-13T15:00:28.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:00:28.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.899901+0000", 2024-09-13T15:00:28.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9799999999999998e-06 2024-09-13T15:00:28.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:00:28.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.899923+0000", 2024-09-13T15:00:28.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1897999999999999e-05 2024-09-13T15:00:28.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:00:28.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.899945+0000", 2024-09-13T15:00:28.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.232e-05 2024-09-13T15:00:28.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.899999+0000", 2024-09-13T15:00:28.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3628999999999998e-05 2024-09-13T15:00:28.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:00:28.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.900008+0000", 2024-09-13T15:00:28.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6170000000000007e-06 2024-09-13T15:00:28.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.966192+0000", 2024-09-13T15:00:28.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0661841620000001 2024-09-13T15:00:28.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:00:28.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.966344+0000", 2024-09-13T15:00:28.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015264599999999999 2024-09-13T15:00:28.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,6", 2024-09-13T15:00:28.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.966472+0000", 2024-09-13T15:00:28.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012720799999999999 2024-09-13T15:00:28.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.972117+0000", 2024-09-13T15:00:28.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0056456429999999997 2024-09-13T15:00:28.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:00:28.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.972196+0000", 2024-09-13T15:00:28.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9030999999999996e-05 2024-09-13T15:00:28.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.972250+0000", 2024-09-13T15:00:28.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3684999999999998e-05 2024-09-13T15:00:28.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:00:28.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.972302+0000", 2024-09-13T15:00:28.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2568999999999998e-05 2024-09-13T15:00:28.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:00:28.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.972341+0000", 2024-09-13T15:00:28.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8510000000000002e-05 2024-09-13T15:00:28.132 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.132 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:00:28.132 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.132 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:235945 8.9 8:9bba0f02:::benchmark_data_smithi151_229097_object111077:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1203)", 2024-09-13T15:00:28.132 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:15.903099+0000", 2024-09-13T15:00:28.132 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 432.17225122600001, 2024-09-13T15:00:28.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.079928687, 2024-09-13T15:00:28.132 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:00:28.133 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:00:28.133 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:28.133 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:00:28.133 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T15:00:28.133 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T15:00:28.133 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 235945 2024-09-13T15:00:28.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.133 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:00:28.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:00:28.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.903099+0000", 2024-09-13T15:00:28.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:00:28.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.903099+0000", 2024-09-13T15:00:28.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:00:28.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.903100+0000", 2024-09-13T15:00:28.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.313e-06 2024-09-13T15:00:28.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:00:28.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.903105+0000", 2024-09-13T15:00:28.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0470000000000001e-06 2024-09-13T15:00:28.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:00:28.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.903123+0000", 2024-09-13T15:00:28.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7785999999999999e-05 2024-09-13T15:00:28.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:00:28.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.903131+0000", 2024-09-13T15:00:28.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6569999999999997e-06 2024-09-13T15:00:28.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.903200+0000", 2024-09-13T15:00:28.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8781000000000004e-05 2024-09-13T15:00:28.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:00:28.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.903209+0000", 2024-09-13T15:00:28.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9369999999999999e-06 2024-09-13T15:00:28.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.969039+0000", 2024-09-13T15:00:28.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.065829452 2024-09-13T15:00:28.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:00:28.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.969144+0000", 2024-09-13T15:00:28.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010464800000000001 2024-09-13T15:00:28.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,6", 2024-09-13T15:00:28.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.969257+0000", 2024-09-13T15:00:28.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000113226 2024-09-13T15:00:28.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.972381+0000", 2024-09-13T15:00:28.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031246630000000002 2024-09-13T15:00:28.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.972423+0000", 2024-09-13T15:00:28.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1640000000000001e-05 2024-09-13T15:00:28.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.138 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:28.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:00:28.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.982916+0000", 2024-09-13T15:00:28.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010492417 2024-09-13T15:00:28.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:00:28.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.982963+0000", 2024-09-13T15:00:28.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7583000000000002e-05 2024-09-13T15:00:28.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:00:28.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.983028+0000", 2024-09-13T15:00:28.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4536999999999996e-05 2024-09-13T15:00:28.141 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.141 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:00:28.141 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.141 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:235946 8.9 8:90fa6c0d:::benchmark_data_smithi151_229097_object111078:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1203)", 2024-09-13T15:00:28.141 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:15.903582+0000", 2024-09-13T15:00:28.141 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 432.17176790399998, 2024-09-13T15:00:28.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0844198309999999, 2024-09-13T15:00:28.141 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:00:28.141 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:00:28.141 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:28.142 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:00:28.142 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T15:00:28.142 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T15:00:28.142 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 235946 2024-09-13T15:00:28.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.142 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:00:28.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:00:28.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.903582+0000", 2024-09-13T15:00:28.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:00:28.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.903582+0000", 2024-09-13T15:00:28.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:00:28.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.903583+0000", 2024-09-13T15:00:28.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0699999999999996e-07 2024-09-13T15:00:28.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:00:28.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.903587+0000", 2024-09-13T15:00:28.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.352e-06 2024-09-13T15:00:28.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:00:28.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.903603+0000", 2024-09-13T15:00:28.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5729000000000001e-05 2024-09-13T15:00:28.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:00:28.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.903610+0000", 2024-09-13T15:00:28.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2599999999999999e-06 2024-09-13T15:00:28.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.903847+0000", 2024-09-13T15:00:28.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00023640600000000001 2024-09-13T15:00:28.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:00:28.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.903855+0000", 2024-09-13T15:00:28.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0479999999999999e-06 2024-09-13T15:00:28.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.972453+0000", 2024-09-13T15:00:28.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.068598269 2024-09-13T15:00:28.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:00:28.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.972566+0000", 2024-09-13T15:00:28.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000112525 2024-09-13T15:00:28.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,6", 2024-09-13T15:00:28.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.972671+0000", 2024-09-13T15:00:28.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105058 2024-09-13T15:00:28.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:00:28.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.987783+0000", 2024-09-13T15:00:28.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01511214 2024-09-13T15:00:28.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.987842+0000", 2024-09-13T15:00:28.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9534999999999998e-05 2024-09-13T15:00:28.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.987921+0000", 2024-09-13T15:00:28.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8563e-05 2024-09-13T15:00:28.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:00:28.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.987965+0000", 2024-09-13T15:00:28.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3992000000000002e-05 2024-09-13T15:00:28.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:00:28.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.988002+0000", 2024-09-13T15:00:28.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7147000000000002e-05 2024-09-13T15:00:28.148 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.148 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:00:28.148 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.148 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:235960 8.9 8:96ec1cf2:::benchmark_data_smithi151_229097_object111092:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1203)", 2024-09-13T15:00:28.148 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:15.916531+0000", 2024-09-13T15:00:28.148 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 432.15881942999999, 2024-09-13T15:00:28.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0716423859999999, 2024-09-13T15:00:28.149 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:00:28.149 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:00:28.149 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:28.149 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:00:28.149 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T15:00:28.149 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T15:00:28.149 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 235960 2024-09-13T15:00:28.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.149 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:00:28.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:00:28.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.916531+0000", 2024-09-13T15:00:28.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:00:28.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.916531+0000", 2024-09-13T15:00:28.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:00:28.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.916531+0000", 2024-09-13T15:00:28.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7899999999999998e-07 2024-09-13T15:00:28.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:00:28.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.916535+0000", 2024-09-13T15:00:28.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.411e-06 2024-09-13T15:00:28.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:00:28.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.916543+0000", 2024-09-13T15:00:28.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4670000000000004e-06 2024-09-13T15:00:28.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:00:28.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.916549+0000", 2024-09-13T15:00:28.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9580000000000004e-06 2024-09-13T15:00:28.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.916629+0000", 2024-09-13T15:00:28.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9606999999999999e-05 2024-09-13T15:00:28.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:00:28.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.916636+0000", 2024-09-13T15:00:28.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.255e-06 2024-09-13T15:00:28.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.975378+0000", 2024-09-13T15:00:28.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.058741715 2024-09-13T15:00:28.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:00:28.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.975495+0000", 2024-09-13T15:00:28.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117531 2024-09-13T15:00:28.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,6", 2024-09-13T15:00:28.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.975601+0000", 2024-09-13T15:00:28.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000106075 2024-09-13T15:00:28.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:00:28.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.987789+0000", 2024-09-13T15:00:28.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012187055 2024-09-13T15:00:28.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.987886+0000", 2024-09-13T15:00:28.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7342000000000004e-05 2024-09-13T15:00:28.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.988054+0000", 2024-09-13T15:00:28.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000168095 2024-09-13T15:00:28.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:00:28.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.988137+0000", 2024-09-13T15:00:28.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2750999999999994e-05 2024-09-13T15:00:28.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:00:28.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.988173+0000", 2024-09-13T15:00:28.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6445000000000001e-05 2024-09-13T15:00:28.155 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.155 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:00:28.155 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.155 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:235991 8.9 8:9a50ea89:::benchmark_data_smithi151_229097_object111123:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1203)", 2024-09-13T15:00:28.155 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:15.947107+0000", 2024-09-13T15:00:28.156 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 432.12824330299998, 2024-09-13T15:00:28.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.041216291, 2024-09-13T15:00:28.156 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:00:28.156 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:00:28.156 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:28.156 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:00:28.156 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T15:00:28.156 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T15:00:28.156 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 235991 2024-09-13T15:00:28.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.156 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:00:28.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:00:28.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.947107+0000", 2024-09-13T15:00:28.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:00:28.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.947107+0000", 2024-09-13T15:00:28.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:00:28.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.947108+0000", 2024-09-13T15:00:28.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4160000000000001e-06 2024-09-13T15:00:28.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:00:28.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.947114+0000", 2024-09-13T15:00:28.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6400000000000002e-06 2024-09-13T15:00:28.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:00:28.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.947136+0000", 2024-09-13T15:00:28.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2172e-05 2024-09-13T15:00:28.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:00:28.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.947144+0000", 2024-09-13T15:00:28.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8310000000000001e-06 2024-09-13T15:00:28.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.947207+0000", 2024-09-13T15:00:28.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2661000000000002e-05 2024-09-13T15:00:28.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:00:28.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.947215+0000", 2024-09-13T15:00:28.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7139999999999999e-06 2024-09-13T15:00:28.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.978225+0000", 2024-09-13T15:00:28.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0310099699999999 2024-09-13T15:00:28.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:00:28.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.978566+0000", 2024-09-13T15:00:28.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00034066599999999999 2024-09-13T15:00:28.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,6", 2024-09-13T15:00:28.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.980078+0000", 2024-09-13T15:00:28.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001511734 2024-09-13T15:00:28.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:00:28.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.987803+0000", 2024-09-13T15:00:28.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0077258040000000002 2024-09-13T15:00:28.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.988214+0000", 2024-09-13T15:00:28.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00041007400000000002 2024-09-13T15:00:28.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.988247+0000", 2024-09-13T15:00:28.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3107e-05 2024-09-13T15:00:28.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:00:28.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.988291+0000", 2024-09-13T15:00:28.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4569e-05 2024-09-13T15:00:28.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:00:28.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.988323+0000", 2024-09-13T15:00:28.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1933000000000001e-05 2024-09-13T15:00:28.162 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.162 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:00:28.162 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.163 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.35940.0:235993 8.9 8:9f3d5edd:::benchmark_data_smithi151_229097_object111125:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1203)", 2024-09-13T15:00:28.163 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:53:15.947561+0000", 2024-09-13T15:00:28.163 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 432.12778933700002, 2024-09-13T15:00:28.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0523244789999999, 2024-09-13T15:00:28.163 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:00:28.163 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:00:28.163 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:28.163 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:00:28.163 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.35940", 2024-09-13T15:00:28.163 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/728654974", 2024-09-13T15:00:28.163 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 235993 2024-09-13T15:00:28.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.164 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:00:28.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:00:28.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.947561+0000", 2024-09-13T15:00:28.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:00:28.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.947561+0000", 2024-09-13T15:00:28.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:00:28.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.947562+0000", 2024-09-13T15:00:28.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7599999999999996e-07 2024-09-13T15:00:28.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:00:28.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.947566+0000", 2024-09-13T15:00:28.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9840000000000003e-06 2024-09-13T15:00:28.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:00:28.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.947575+0000", 2024-09-13T15:00:28.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7589999999999992e-06 2024-09-13T15:00:28.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:00:28.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.947582+0000", 2024-09-13T15:00:28.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9990000000000002e-06 2024-09-13T15:00:28.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.947632+0000", 2024-09-13T15:00:28.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9956000000000001e-05 2024-09-13T15:00:28.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:00:28.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:15.947639+0000", 2024-09-13T15:00:28.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7329999999999997e-06 2024-09-13T15:00:28.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.983060+0000", 2024-09-13T15:00:28.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.035421379 2024-09-13T15:00:28.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:00:28.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.983152+0000", 2024-09-13T15:00:28.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1889000000000001e-05 2024-09-13T15:00:28.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,6", 2024-09-13T15:00:28.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.983262+0000", 2024-09-13T15:00:28.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109794 2024-09-13T15:00:28.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.988555+0000", 2024-09-13T15:00:28.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0052928690000000004 2024-09-13T15:00:28.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.988590+0000", 2024-09-13T15:00:28.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4962999999999997e-05 2024-09-13T15:00:28.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:00:28.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.996880+0000", 2024-09-13T15:00:28.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0082900270000000002 2024-09-13T15:00:28.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:00:28.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.999843+0000", 2024-09-13T15:00:28.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002963275 2024-09-13T15:00:28.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:00:28.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:53:16.999885+0000", 2024-09-13T15:00:28.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2076000000000002e-05 2024-09-13T15:00:28.170 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.170 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:00:28.170 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.170 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.46034.0:83738 9.b 9:ddb065ee:::benchmark_data_smithi151_261954_object83737:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1232)", 2024-09-13T15:00:28.170 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:55:15.874313+0000", 2024-09-13T15:00:28.170 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 312.20103688699999, 2024-09-13T15:00:28.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3916816390000002, 2024-09-13T15:00:28.170 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:00:28.171 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:00:28.171 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:28.171 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:00:28.171 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.46034", 2024-09-13T15:00:28.171 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:00:28.171 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 83738 2024-09-13T15:00:28.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.171 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:00:28.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:00:28.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.874313+0000", 2024-09-13T15:00:28.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:00:28.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.874313+0000", 2024-09-13T15:00:28.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:00:28.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.874315+0000", 2024-09-13T15:00:28.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.629e-06 2024-09-13T15:00:28.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:00:28.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.874339+0000", 2024-09-13T15:00:28.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4338e-05 2024-09-13T15:00:28.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:00:28.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.874352+0000", 2024-09-13T15:00:28.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2374e-05 2024-09-13T15:00:28.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:00:28.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.874358+0000", 2024-09-13T15:00:28.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6309999999999997e-06 2024-09-13T15:00:28.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.874808+0000", 2024-09-13T15:00:28.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00044960600000000002 2024-09-13T15:00:28.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:00:28.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.874818+0000", 2024-09-13T15:00:28.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.007e-05 2024-09-13T15:00:28.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:16.844952+0000", 2024-09-13T15:00:28.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.970134422 2024-09-13T15:00:28.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:00:28.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:16.844962+0000", 2024-09-13T15:00:28.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4900000000000006e-06 2024-09-13T15:00:28.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.245123+0000", 2024-09-13T15:00:28.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4001615359999999 2024-09-13T15:00:28.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:00:28.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.245278+0000", 2024-09-13T15:00:28.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015417100000000001 2024-09-13T15:00:28.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,6", 2024-09-13T15:00:28.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.245424+0000", 2024-09-13T15:00:28.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014675800000000001 2024-09-13T15:00:28.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.253008+0000", 2024-09-13T15:00:28.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0075839590000000004 2024-09-13T15:00:28.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.260639+0000", 2024-09-13T15:00:28.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0076307650000000003 2024-09-13T15:00:28.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:00:28.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.260874+0000", 2024-09-13T15:00:28.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00023486899999999999 2024-09-13T15:00:28.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:00:28.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.261051+0000", 2024-09-13T15:00:28.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017676399999999999 2024-09-13T15:00:28.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:00:28.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.265995+0000", 2024-09-13T15:00:28.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0049442569999999996 2024-09-13T15:00:28.179 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.179 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:00:28.179 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.179 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.46034.0:83739 9.b 9:dd8f5619:::benchmark_data_smithi151_261954_object83738:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1232)", 2024-09-13T15:00:28.179 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:55:15.875383+0000", 2024-09-13T15:00:28.179 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 312.19996710100003, 2024-09-13T15:00:28.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.395487438, 2024-09-13T15:00:28.179 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:00:28.179 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:00:28.179 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:28.179 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:00:28.179 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.46034", 2024-09-13T15:00:28.180 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:00:28.180 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 83739 2024-09-13T15:00:28.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.180 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:00:28.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:00:28.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.875383+0000", 2024-09-13T15:00:28.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:00:28.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.875383+0000", 2024-09-13T15:00:28.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:00:28.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.875385+0000", 2024-09-13T15:00:28.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4959999999999999e-06 2024-09-13T15:00:28.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:00:28.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.875410+0000", 2024-09-13T15:00:28.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5817e-05 2024-09-13T15:00:28.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:00:28.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.875423+0000", 2024-09-13T15:00:28.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2609000000000001e-05 2024-09-13T15:00:28.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:00:28.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.875430+0000", 2024-09-13T15:00:28.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8580000000000002e-06 2024-09-13T15:00:28.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.875495+0000", 2024-09-13T15:00:28.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4703999999999996e-05 2024-09-13T15:00:28.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:00:28.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.875503+0000", 2024-09-13T15:00:28.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6400000000000003e-06 2024-09-13T15:00:28.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:16.844986+0000", 2024-09-13T15:00:28.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:28.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.96948313399999997 2024-09-13T15:00:28.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:00:28.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:16.844993+0000", 2024-09-13T15:00:28.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8129999999999998e-06 2024-09-13T15:00:28.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.253047+0000", 2024-09-13T15:00:28.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.408053913 2024-09-13T15:00:28.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:00:28.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.253208+0000", 2024-09-13T15:00:28.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000160564 2024-09-13T15:00:28.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,6", 2024-09-13T15:00:28.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.253369+0000", 2024-09-13T15:00:28.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016175900000000001 2024-09-13T15:00:28.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.260824+0000", 2024-09-13T15:00:28.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0074545280000000002 2024-09-13T15:00:28.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:00:28.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.266015+0000", 2024-09-13T15:00:28.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0051908279999999998 2024-09-13T15:00:28.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.266551+0000", 2024-09-13T15:00:28.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00053653100000000003 2024-09-13T15:00:28.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:00:28.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.266619+0000", 2024-09-13T15:00:28.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7319999999999999e-05 2024-09-13T15:00:28.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:00:28.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.270871+0000", 2024-09-13T15:00:28.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0042519239999999998 2024-09-13T15:00:28.188 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.188 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:00:28.188 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.189 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.46034.0:83744 9.5 9:a28ae00e:::benchmark_data_smithi151_261954_object83743:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1232)", 2024-09-13T15:00:28.189 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:55:15.900982+0000", 2024-09-13T15:00:28.189 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 312.174368517, 2024-09-13T15:00:28.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2424329350000001, 2024-09-13T15:00:28.189 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:00:28.189 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:00:28.189 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:28.189 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:00:28.189 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.46034", 2024-09-13T15:00:28.189 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:00:28.189 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 83744 2024-09-13T15:00:28.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.189 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:00:28.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:00:28.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.900982+0000", 2024-09-13T15:00:28.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:00:28.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.900982+0000", 2024-09-13T15:00:28.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:00:28.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.900983+0000", 2024-09-13T15:00:28.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7069999999999999e-06 2024-09-13T15:00:28.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:00:28.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.901011+0000", 2024-09-13T15:00:28.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7189999999999999e-05 2024-09-13T15:00:28.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:00:28.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.901027+0000", 2024-09-13T15:00:28.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6730000000000001e-05 2024-09-13T15:00:28.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:00:28.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.901034+0000", 2024-09-13T15:00:28.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1029999999999996e-06 2024-09-13T15:00:28.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.901093+0000", 2024-09-13T15:00:28.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8640000000000001e-05 2024-09-13T15:00:28.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:00:28.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.901101+0000", 2024-09-13T15:00:28.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4900000000000003e-06 2024-09-13T15:00:28.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:16.840664+0000", 2024-09-13T15:00:28.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.93956368999999995 2024-09-13T15:00:28.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:00:28.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:16.840676+0000", 2024-09-13T15:00:28.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1901e-05 2024-09-13T15:00:28.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.131975+0000", 2024-09-13T15:00:28.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.291299121 2024-09-13T15:00:28.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:00:28.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.132257+0000", 2024-09-13T15:00:28.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028212099999999998 2024-09-13T15:00:28.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-13T15:00:28.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.132450+0000", 2024-09-13T15:00:28.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019222900000000001 2024-09-13T15:00:28.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.139668+0000", 2024-09-13T15:00:28.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.007218449 2024-09-13T15:00:28.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.139804+0000", 2024-09-13T15:00:28.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013552700000000001 2024-09-13T15:00:28.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:00:28.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.139821+0000", 2024-09-13T15:00:28.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7149000000000001e-05 2024-09-13T15:00:28.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.196 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:00:28.196 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.139885+0000", 2024-09-13T15:00:28.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.436e-05 2024-09-13T15:00:28.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.196 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:00:28.196 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.143415+0000", 2024-09-13T15:00:28.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0035295280000000001 2024-09-13T15:00:28.197 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.197 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:00:28.197 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.197 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.197 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.46034.0:83746 9.b 9:dc9ec253:::benchmark_data_smithi151_261954_object83745:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1232)", 2024-09-13T15:00:28.197 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:55:15.906630+0000", 2024-09-13T15:00:28.197 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 312.16872004599998, 2024-09-13T15:00:28.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3735334429999999, 2024-09-13T15:00:28.197 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:00:28.197 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:00:28.197 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:28.197 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:00:28.197 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.46034", 2024-09-13T15:00:28.198 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:00:28.198 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 83746 2024-09-13T15:00:28.198 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.198 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:00:28.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.198 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:00:28.198 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.906630+0000", 2024-09-13T15:00:28.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.198 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.198 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:00:28.198 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.906630+0000", 2024-09-13T15:00:28.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.198 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.199 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:00:28.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.906632+0000", 2024-09-13T15:00:28.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0090000000000002e-06 2024-09-13T15:00:28.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.199 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:00:28.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.906657+0000", 2024-09-13T15:00:28.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5181000000000002e-05 2024-09-13T15:00:28.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.199 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:00:28.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.906674+0000", 2024-09-13T15:00:28.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6650000000000002e-05 2024-09-13T15:00:28.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:00:28.200 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.906680+0000", 2024-09-13T15:00:28.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3450000000000002e-06 2024-09-13T15:00:28.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.200 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.906736+0000", 2024-09-13T15:00:28.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5782000000000002e-05 2024-09-13T15:00:28.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:00:28.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:15.906743+0000", 2024-09-13T15:00:28.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.781e-06 2024-09-13T15:00:28.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:16.845016+0000", 2024-09-13T15:00:28.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.938272997 2024-09-13T15:00:28.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:00:28.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:16.845023+0000", 2024-09-13T15:00:28.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7390000000000002e-06 2024-09-13T15:00:28.202 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.202 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.202 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.256465+0000", 2024-09-13T15:00:28.202 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4114422229999999 2024-09-13T15:00:28.202 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.202 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:00:28.202 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.256638+0000", 2024-09-13T15:00:28.202 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017350500000000001 2024-09-13T15:00:28.202 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.202 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,6", 2024-09-13T15:00:28.203 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.256816+0000", 2024-09-13T15:00:28.203 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000177911 2024-09-13T15:00:28.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.203 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.203 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.266076+0000", 2024-09-13T15:00:28.203 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0092594549999999998 2024-09-13T15:00:28.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.203 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:00:28.203 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.266104+0000", 2024-09-13T15:00:28.203 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8147999999999999e-05 2024-09-13T15:00:28.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.204 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.204 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.275822+0000", 2024-09-13T15:00:28.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.009717926 2024-09-13T15:00:28.204 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.204 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.204 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:00:28.204 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.275884+0000", 2024-09-13T15:00:28.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1729999999999993e-05 2024-09-13T15:00:28.204 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.204 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.204 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:00:28.204 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:18.280164+0000", 2024-09-13T15:00:28.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0042800609999999999 2024-09-13T15:00:28.205 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.205 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:00:28.205 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.205 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.205 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.46034.0:105389 9.b 9:da45e8bc:::benchmark_data_smithi151_261954_object2533:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1239)", 2024-09-13T15:00:28.205 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:55:52.869881+0000", 2024-09-13T15:00:28.205 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 275.20546920700002, 2024-09-13T15:00:28.205 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1048247980000001, 2024-09-13T15:00:28.205 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:00:28.205 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:00:28.205 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:28.205 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:00:28.205 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.46034", 2024-09-13T15:00:28.206 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:00:28.206 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 105389 2024-09-13T15:00:28.206 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.206 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:00:28.206 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.206 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:00:28.206 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.869881+0000", 2024-09-13T15:00:28.206 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.206 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.206 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.206 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:00:28.206 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.869881+0000", 2024-09-13T15:00:28.206 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.206 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.207 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:00:28.207 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.869882+0000", 2024-09-13T15:00:28.207 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3659999999999999e-06 2024-09-13T15:00:28.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.207 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:00:28.207 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.869888+0000", 2024-09-13T15:00:28.207 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4480000000000002e-06 2024-09-13T15:00:28.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.207 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:00:28.207 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.869898+0000", 2024-09-13T15:00:28.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0114e-05 2024-09-13T15:00:28.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:00:28.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.869905+0000", 2024-09-13T15:00:28.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.469e-06 2024-09-13T15:00:28.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.869968+0000", 2024-09-13T15:00:28.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2151999999999997e-05 2024-09-13T15:00:28.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:00:28.209 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.869977+0000", 2024-09-13T15:00:28.209 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6649999999999998e-06 2024-09-13T15:00:28.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.209 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.209 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.965912+0000", 2024-09-13T15:00:28.209 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.095934725 2024-09-13T15:00:28.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.209 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:00:28.209 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.966018+0000", 2024-09-13T15:00:28.209 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010648400000000001 2024-09-13T15:00:28.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-13T15:00:28.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.966106+0000", 2024-09-13T15:00:28.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7509999999999994e-05 2024-09-13T15:00:28.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.970681+0000", 2024-09-13T15:00:28.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0045751120000000001 2024-09-13T15:00:28.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.970735+0000", 2024-09-13T15:00:28.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3798999999999998e-05 2024-09-13T15:00:28.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.211 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.211 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:00:28.211 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.974595+0000", 2024-09-13T15:00:28.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0038599870000000001 2024-09-13T15:00:28.211 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.211 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.211 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:00:28.211 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.974652+0000", 2024-09-13T15:00:28.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6968000000000001e-05 2024-09-13T15:00:28.211 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.211 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.211 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:00:28.211 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.974706+0000", 2024-09-13T15:00:28.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3999000000000003e-05 2024-09-13T15:00:28.212 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.212 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:00:28.212 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.212 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.212 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.212 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.46034.0:105398 9.b 9:d51eff01:::benchmark_data_smithi151_261954_object2542:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1239)", 2024-09-13T15:00:28.212 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:55:52.881805+0000", 2024-09-13T15:00:28.212 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 275.19354498199999, 2024-09-13T15:00:28.212 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0997890809999999, 2024-09-13T15:00:28.212 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:00:28.212 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:00:28.212 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:28.212 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:00:28.213 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.46034", 2024-09-13T15:00:28.213 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:00:28.213 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 105398 2024-09-13T15:00:28.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.213 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:00:28.213 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.213 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:00:28.213 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.881805+0000", 2024-09-13T15:00:28.213 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.213 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.213 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:00:28.213 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.881805+0000", 2024-09-13T15:00:28.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:00:28.214 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.881806+0000", 2024-09-13T15:00:28.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2929999999999999e-06 2024-09-13T15:00:28.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:00:28.214 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.881812+0000", 2024-09-13T15:00:28.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5899999999999998e-06 2024-09-13T15:00:28.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.215 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:00:28.215 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.881825+0000", 2024-09-13T15:00:28.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3322000000000001e-05 2024-09-13T15:00:28.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.215 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.215 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:00:28.215 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.881831+0000", 2024-09-13T15:00:28.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.604e-06 2024-09-13T15:00:28.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.215 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.215 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.215 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.881899+0000", 2024-09-13T15:00:28.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8397000000000003e-05 2024-09-13T15:00:28.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.216 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:00:28.216 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.881911+0000", 2024-09-13T15:00:28.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1895999999999999e-05 2024-09-13T15:00:28.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.216 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.216 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.970806+0000", 2024-09-13T15:00:28.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0888942859999999 2024-09-13T15:00:28.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.216 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:00:28.216 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.970953+0000", 2024-09-13T15:00:28.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000147634 2024-09-13T15:00:28.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-13T15:00:28.217 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.971104+0000", 2024-09-13T15:00:28.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015057199999999999 2024-09-13T15:00:28.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.217 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.978202+0000", 2024-09-13T15:00:28.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0070986499999999998 2024-09-13T15:00:28.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.218 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.218 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.978263+0000", 2024-09-13T15:00:28.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1045999999999998e-05 2024-09-13T15:00:28.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.218 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.218 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:00:28.218 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.981499+0000", 2024-09-13T15:00:28.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0032350429999999999 2024-09-13T15:00:28.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.218 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.218 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:00:28.218 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.981551+0000", 2024-09-13T15:00:28.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2373000000000002e-05 2024-09-13T15:00:28.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:00:28.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.981594+0000", 2024-09-13T15:00:28.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3374999999999997e-05 2024-09-13T15:00:28.219 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.219 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:00:28.219 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.219 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.46034.0:105438 9.b 9:d700f1a7:::benchmark_data_smithi151_261954_object2582:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1239)", 2024-09-13T15:00:28.219 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:55:52.920570+0000", 2024-09-13T15:00:28.219 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 275.15477982099998, 2024-09-13T15:00:28.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0611124599999999, 2024-09-13T15:00:28.220 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:00:28.220 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:00:28.220 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:28.220 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:00:28.220 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.46034", 2024-09-13T15:00:28.220 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:00:28.220 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 105438 2024-09-13T15:00:28.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.220 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:00:28.220 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.220 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:00:28.220 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.920570+0000", 2024-09-13T15:00:28.220 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.221 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.221 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:00:28.221 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.920570+0000", 2024-09-13T15:00:28.221 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.221 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.221 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:00:28.221 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.920572+0000", 2024-09-13T15:00:28.221 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2470000000000001e-06 2024-09-13T15:00:28.221 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.221 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:00:28.221 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.920577+0000", 2024-09-13T15:00:28.222 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.118e-06 2024-09-13T15:00:28.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.222 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:00:28.222 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.920588+0000", 2024-09-13T15:00:28.222 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1163e-05 2024-09-13T15:00:28.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.222 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:00:28.222 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.920596+0000", 2024-09-13T15:00:28.222 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7309999999999993e-06 2024-09-13T15:00:28.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.223 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.920663+0000", 2024-09-13T15:00:28.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7185000000000001e-05 2024-09-13T15:00:28.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.223 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:00:28.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.920672+0000", 2024-09-13T15:00:28.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2499999999999995e-06 2024-09-13T15:00:28.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.223 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.974762+0000", 2024-09-13T15:00:28.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0540895530000001 2024-09-13T15:00:28.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.224 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:00:28.224 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.974879+0000", 2024-09-13T15:00:28.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117835 2024-09-13T15:00:28.224 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.224 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-13T15:00:28.224 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.975005+0000", 2024-09-13T15:00:28.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012583000000000001 2024-09-13T15:00:28.224 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.224 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.224 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.978418+0000", 2024-09-13T15:00:28.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034126310000000002 2024-09-13T15:00:28.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.225 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.225 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.978471+0000", 2024-09-13T15:00:28.225 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2893000000000003e-05 2024-09-13T15:00:28.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.225 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:00:28.225 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.981605+0000", 2024-09-13T15:00:28.225 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031339380000000002 2024-09-13T15:00:28.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.225 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:00:28.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.981649+0000", 2024-09-13T15:00:28.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3823999999999999e-05 2024-09-13T15:00:28.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:00:28.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.981683+0000", 2024-09-13T15:00:28.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4261999999999998e-05 2024-09-13T15:00:28.226 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.226 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:00:28.226 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.226 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.46034.0:105449 9.b 9:db14c0d5:::benchmark_data_smithi151_261954_object2593:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1239)", 2024-09-13T15:00:28.226 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:55:52.929012+0000", 2024-09-13T15:00:28.226 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 275.14633772399998, 2024-09-13T15:00:28.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.056517011, 2024-09-13T15:00:28.227 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:00:28.227 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:00:28.227 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:28.227 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:00:28.227 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.46034", 2024-09-13T15:00:28.227 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:00:28.227 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 105449 2024-09-13T15:00:28.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.227 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:00:28.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.227 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:00:28.227 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.929012+0000", 2024-09-13T15:00:28.227 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:00:28.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.929012+0000", 2024-09-13T15:00:28.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:00:28.228 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:00:28.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.929014+0000", 2024-09-13T15:00:28.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.274e-06 2024-09-13T15:00:28.228 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:00:28.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.929019+0000", 2024-09-13T15:00:28.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1170000000000003e-06 2024-09-13T15:00:28.228 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.229 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:00:28.229 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.929031+0000", 2024-09-13T15:00:28.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1737e-05 2024-09-13T15:00:28.229 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.229 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:00:28.229 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.929038+0000", 2024-09-13T15:00:28.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3849999999999996e-06 2024-09-13T15:00:28.229 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.229 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.229 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.929099+0000", 2024-09-13T15:00:28.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1396000000000006e-05 2024-09-13T15:00:28.230 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.230 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.230 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:00:28.230 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:52.929108+0000", 2024-09-13T15:00:28.230 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7350000000000002e-06 2024-09-13T15:00:28.230 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.230 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.230 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:00:28.230 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.978514+0000", 2024-09-13T15:00:28.230 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.049406144 2024-09-13T15:00:28.230 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.230 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.230 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:00:28.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.978594+0000", 2024-09-13T15:00:28.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9479000000000003e-05 2024-09-13T15:00:28.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-13T15:00:28.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.978682+0000", 2024-09-13T15:00:28.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8579999999999996e-05 2024-09-13T15:00:28.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.985342+0000", 2024-09-13T15:00:28.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0066599770000000001 2024-09-13T15:00:28.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:00:28.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.985362+0000", 2024-09-13T15:00:28.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9868e-05 2024-09-13T15:00:28.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.232 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:28.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.985424+0000", 2024-09-13T15:00:28.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2213999999999998e-05 2024-09-13T15:00:28.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.232 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:00:28.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.985477+0000", 2024-09-13T15:00:28.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2178000000000001e-05 2024-09-13T15:00:28.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:00:28.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:00:28.233 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:00:28.233 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:55:53.985529+0000", 2024-09-13T15:00:28.233 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2927000000000003e-05 2024-09-13T15:00:28.233 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.233 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:00:28.233 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.233 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:00:28.233 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:00:28.233 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T15:00:28.234 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:28.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:28.335 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:28.368 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:28.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:28.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:28.470 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:28.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:28.512 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:28.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:28.613 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:28.645 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:28.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:28.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:28.747 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:28.780 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:28.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:28.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:28.882 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:28.930 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:28.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:28.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:29.032 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:29.067 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:29.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:29.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:29.169 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:29.203 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:29.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:29.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:29.305 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:29.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:29.336 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:29.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:29.437 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:29.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:29.477 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:29.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:29.578 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:29.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:29 smithi184 ceph-mon[92527]: pgmap v2127: 119 pgs: 1 active+remapped, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 106 active+clean; 5.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 540 op/s; 106301/276897 objects misplaced (38.390%); 35 MiB/s, 560 objects/s recovering 2024-09-13T15:00:29.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:29 smithi151 ceph-mon[104450]: pgmap v2127: 119 pgs: 1 active+remapped, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 106 active+clean; 5.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 540 op/s; 106301/276897 objects misplaced (38.390%); 35 MiB/s, 560 objects/s recovering 2024-09-13T15:00:29.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:29 smithi151 ceph-mon[108966]: pgmap v2127: 119 pgs: 1 active+remapped, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 106 active+clean; 5.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 540 op/s; 106301/276897 objects misplaced (38.390%); 35 MiB/s, 560 objects/s recovering 2024-09-13T15:00:29.609 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:29.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:29.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:29.712 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:29.740 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:29.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:29.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:29.842 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:29.879 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:29.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:29.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:29.981 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:30.011 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:30.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:30.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:30.113 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:30.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:30.156 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:30.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:30.213 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 6, 7, 1, 5] out_osds: [0, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T15:00:30.213 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:00:30.214 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-13T15:00:30.214 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T15:00:30.214 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T15:00:30.214 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-13T15:00:30.256 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:30.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:30.298 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:30.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:30.356 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T15:00:30.400 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:30.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:30.450 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:30.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:30.552 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:30.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:30.587 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:30.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:30.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:30 smithi184 ceph-mon[92527]: 1.0 scrub starts 2024-09-13T15:00:30.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:00:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:00:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:00:30.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:30 smithi151 ceph-mon[104450]: 1.0 scrub starts 2024-09-13T15:00:30.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:30 smithi151 ceph-mon[108966]: 1.0 scrub starts 2024-09-13T15:00:30.614 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:00:30.688 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:30.718 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:30.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:30.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:30.820 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:30.867 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:30.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:30.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:30.969 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:31.009 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:31.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:31.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:31.111 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:31.153 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:31.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:31.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:31.245 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:31.256 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:31.321 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:31.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:31.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:31.423 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:31.468 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:31.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:31.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:31.570 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:31.590 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:31 smithi151 ceph-mon[104450]: pgmap v2128: 119 pgs: 1 active+remapped, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 106 active+clean; 5.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 8.5 KiB/s rd, 18 KiB/s wr, 542 op/s; 110536/273747 objects misplaced (40.379%); 34 MiB/s, 544 objects/s recovering 2024-09-13T15:00:31.590 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:31 smithi151 ceph-mon[104450]: 1.e scrub starts 2024-09-13T15:00:31.591 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:31 smithi151 ceph-mon[108966]: pgmap v2128: 119 pgs: 1 active+remapped, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 106 active+clean; 5.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 8.5 KiB/s rd, 18 KiB/s wr, 542 op/s; 110536/273747 objects misplaced (40.379%); 34 MiB/s, 544 objects/s recovering 2024-09-13T15:00:31.591 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:31 smithi151 ceph-mon[108966]: 1.e scrub starts 2024-09-13T15:00:31.602 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:31.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:31.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:31.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:31 smithi184 ceph-mon[92527]: pgmap v2128: 119 pgs: 1 active+remapped, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 106 active+clean; 5.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 8.5 KiB/s rd, 18 KiB/s wr, 542 op/s; 110536/273747 objects misplaced (40.379%); 34 MiB/s, 544 objects/s recovering 2024-09-13T15:00:31.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:31 smithi184 ceph-mon[92527]: 1.e scrub starts 2024-09-13T15:00:31.705 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:31.740 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:31.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:31.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:31.842 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:31.884 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:31.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:31.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:31.988 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:32.033 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:32.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:32.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:32.135 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:32.162 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:32.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:32.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:32.263 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:32.293 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:32.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:32.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:32 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/258303322' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:32.395 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:32.430 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:32.430 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:32 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/258303322' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:32.430 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:32.431 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:32 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/258303322' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:32.442 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:32.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:32.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:32.544 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:32.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:32.573 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:32.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:32.674 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:32.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:32.717 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:32.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:32.818 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:32.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:32.840 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:32.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:32.941 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:32.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:32.977 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:32.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:33.079 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:33.113 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:33.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:33.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:33.215 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:33.253 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:33.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:33.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:33.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:33 smithi184 ceph-mon[92527]: pgmap v2129: 119 pgs: 1 active+remapped, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 106 active+clean; 5.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 7.0 KiB/s rd, 15 KiB/s wr, 445 op/s; 110536/273747 objects misplaced (40.379%); 28 MiB/s, 447 objects/s recovering 2024-09-13T15:00:33.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:33 smithi184 ceph-mon[92527]: 1.2 scrub starts 2024-09-13T15:00:33.355 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:33.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:33.389 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:33.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:33.490 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:33.529 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:33.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:33.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:33.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:33 smithi151 ceph-mon[104450]: pgmap v2129: 119 pgs: 1 active+remapped, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 106 active+clean; 5.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 7.0 KiB/s rd, 15 KiB/s wr, 445 op/s; 110536/273747 objects misplaced (40.379%); 28 MiB/s, 447 objects/s recovering 2024-09-13T15:00:33.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:33 smithi151 ceph-mon[104450]: 1.2 scrub starts 2024-09-13T15:00:33.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:33 smithi151 ceph-mon[108966]: pgmap v2129: 119 pgs: 1 active+remapped, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 106 active+clean; 5.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 7.0 KiB/s rd, 15 KiB/s wr, 445 op/s; 110536/273747 objects misplaced (40.379%); 28 MiB/s, 447 objects/s recovering 2024-09-13T15:00:33.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:33 smithi151 ceph-mon[108966]: 1.2 scrub starts 2024-09-13T15:00:33.631 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:33.671 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:33.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:33.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:33.773 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:33.793 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T15:00:33.793 INFO:teuthology.orchestra.run.smithi184.stdout: "success": "" 2024-09-13T15:00:33.793 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T15:00:33.808 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:33.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:33.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:33.910 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:33.947 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:33.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:33.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:34.049 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:34.080 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:34.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:34.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:34.182 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:34.224 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:34.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:34.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:34.326 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:34.368 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:34.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:34.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:34.470 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:34.495 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:34.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:34.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:34.597 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:34.631 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:34.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:34.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:34.733 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:34.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:34.757 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:34.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:34.858 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:34.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:00:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:00:34] "GET /metrics HTTP/1.1" 200 36762 "" "Prometheus/2.51.0" 2024-09-13T15:00:34.899 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:34.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:34.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:35.001 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:35.024 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:35.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:35.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:35.126 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:35.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:35.172 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:35.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:35.273 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:35.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:35.300 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:35.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:35.401 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:35.429 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:35.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:35.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:35.531 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:35.558 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:35.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:35.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:35.596 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:35 smithi151 ceph-mon[104450]: pgmap v2130: 119 pgs: 1 active+remapped, 3 active+clean+scrubbing, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 103 active+clean; 5.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 5.8 KiB/s rd, 13 KiB/s wr, 939 op/s; 103112/258795 objects misplaced (39.843%); 29 MiB/s, 471 objects/s recovering 2024-09-13T15:00:35.596 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:35 smithi151 ceph-mon[104450]: 1.0 scrub starts 2024-09-13T15:00:35.597 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:35 smithi151 ceph-mon[108966]: pgmap v2130: 119 pgs: 1 active+remapped, 3 active+clean+scrubbing, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 103 active+clean; 5.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 5.8 KiB/s rd, 13 KiB/s wr, 939 op/s; 103112/258795 objects misplaced (39.843%); 29 MiB/s, 471 objects/s recovering 2024-09-13T15:00:35.597 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:35 smithi151 ceph-mon[108966]: 1.0 scrub starts 2024-09-13T15:00:35.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:35 smithi184 ceph-mon[92527]: pgmap v2130: 119 pgs: 1 active+remapped, 3 active+clean+scrubbing, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 103 active+clean; 5.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 5.8 KiB/s rd, 13 KiB/s wr, 939 op/s; 103112/258795 objects misplaced (39.843%); 29 MiB/s, 471 objects/s recovering 2024-09-13T15:00:35.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:35 smithi184 ceph-mon[92527]: 1.0 scrub starts 2024-09-13T15:00:35.660 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:35.686 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:35.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:35.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:35.788 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:35.810 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:35.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:35.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:35.912 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:35.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:35.943 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:35.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:36.044 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:36.123 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:36.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:36.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:36.226 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:36.258 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:36.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:36.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:36.360 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:36.382 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:36.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:36.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:36.484 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:36.507 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:36.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:36.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:36.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:36 smithi184 ceph-mon[92527]: 1.e scrub starts 2024-09-13T15:00:36.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:36 smithi151 ceph-mon[104450]: 1.e scrub starts 2024-09-13T15:00:36.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:36 smithi151 ceph-mon[108966]: 1.e scrub starts 2024-09-13T15:00:36.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:36.652 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:36.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:36.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:36.754 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:36.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:36.786 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:36.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:36.887 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:36.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:36.919 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:36.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:37.020 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:37.051 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:37.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:37.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:37.153 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:37.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:37.204 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:37.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:37.258 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T15:00:37.258 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T15:00:37.259 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T15:00:37.259 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T15:00:37.259 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:00:37.259 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:00:37.259 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:00:37.259 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:00:37.259 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:00:37.260 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T15:00:37.260 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T15:00:37.260 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T15:00:37.260 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T15:00:37.260 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:00:37.260 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:00:37.260 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:00:37.305 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:37.339 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:37.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:37.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:37.441 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:37.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:37.472 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:37.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:37.574 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:37.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:37 smithi184 ceph-mon[92527]: pgmap v2131: 119 pgs: 1 active+remapped, 3 active+clean+scrubbing, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 103 active+clean; 5.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 5.1 KiB/s rd, 11 KiB/s wr, 823 op/s; 103112/258795 objects misplaced (39.843%); 26 MiB/s, 413 objects/s recovering 2024-09-13T15:00:37.607 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:37.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:37.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:37.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:37 smithi151 ceph-mon[108966]: pgmap v2131: 119 pgs: 1 active+remapped, 3 active+clean+scrubbing, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 103 active+clean; 5.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 5.1 KiB/s rd, 11 KiB/s wr, 823 op/s; 103112/258795 objects misplaced (39.843%); 26 MiB/s, 413 objects/s recovering 2024-09-13T15:00:37.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:37 smithi151 ceph-mon[104450]: pgmap v2131: 119 pgs: 1 active+remapped, 3 active+clean+scrubbing, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 103 active+clean; 5.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 5.1 KiB/s rd, 11 KiB/s wr, 823 op/s; 103112/258795 objects misplaced (39.843%); 26 MiB/s, 413 objects/s recovering 2024-09-13T15:00:37.710 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:37.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:37.752 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:37.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:37.853 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:37.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:37.893 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:37.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:37.994 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:38.023 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:38.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:38.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:38.125 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:38.166 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:38.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:38.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:38.268 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:38.300 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:38.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:38.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:38.402 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:38.438 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:38.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:38.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:38.540 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:38.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:38.571 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:38.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:38.672 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:38.722 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:38.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:38.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:38.824 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:38.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:38.857 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:38.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:38.958 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:38.994 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:39.019 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:39.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:39.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:39.121 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:39.171 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:39.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:39.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:39.273 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:39.305 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:39.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:39.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:39.336 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-13T15:00:39.407 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:39.408 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:39 smithi151 ceph-mon[104450]: pgmap v2132: 119 pgs: 1 active+remapped, 3 active+clean+scrubbing, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 103 active+clean; 5.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 4.3 KiB/s rd, 9.4 KiB/s wr, 938 op/s; 97648/250383 objects misplaced (38.999%); 28 MiB/s, 450 objects/s recovering 2024-09-13T15:00:39.408 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:39 smithi151 ceph-mon[104450]: 1.2 scrub starts 2024-09-13T15:00:39.408 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:39 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1673951961' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:00:39.408 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:39 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:00:39.409 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:39 smithi151 ceph-mon[108966]: pgmap v2132: 119 pgs: 1 active+remapped, 3 active+clean+scrubbing, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 103 active+clean; 5.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 4.3 KiB/s rd, 9.4 KiB/s wr, 938 op/s; 97648/250383 objects misplaced (38.999%); 28 MiB/s, 450 objects/s recovering 2024-09-13T15:00:39.409 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:39 smithi151 ceph-mon[108966]: 1.2 scrub starts 2024-09-13T15:00:39.409 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:39 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1673951961' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:00:39.409 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:39 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:00:39.456 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:39.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:39.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:39.558 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:39.595 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:39.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:39.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:39.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:39 smithi184 ceph-mon[92527]: pgmap v2132: 119 pgs: 1 active+remapped, 3 active+clean+scrubbing, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 103 active+clean; 5.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 4.3 KiB/s rd, 9.4 KiB/s wr, 938 op/s; 97648/250383 objects misplaced (38.999%); 28 MiB/s, 450 objects/s recovering 2024-09-13T15:00:39.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:39 smithi184 ceph-mon[92527]: 1.2 scrub starts 2024-09-13T15:00:39.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:39 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1673951961' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:00:39.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:39 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:00:39.608 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 6, 7, 1, 5] out_osds: [0, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T15:00:39.608 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:00:39.609 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-09-13T15:00:39.609 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T15:00:39.609 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T15:00:39.609 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-09-13T15:00:39.697 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:39.741 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:39.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:39.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:39.843 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:39.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:39.897 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:39.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:39.998 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:40.022 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:00:40.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:40.036 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:40.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:40.138 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:40.174 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:40.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:40.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:40.276 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:40.310 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:40.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:40.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:40.412 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:40.455 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:40.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:40.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:40.557 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:40.584 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:40 smithi151 ceph-mon[104450]: 1.0 scrub starts 2024-09-13T15:00:40.584 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:40 smithi151 ceph-mon[104450]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:00:40.584 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:40 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:00:40.584 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:40 smithi151 ceph-mon[104450]: osdmap e1291: 8 total, 8 up, 6 in 2024-09-13T15:00:40.585 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:40 smithi151 ceph-mon[108966]: 1.0 scrub starts 2024-09-13T15:00:40.585 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:40 smithi151 ceph-mon[108966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:00:40.585 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:40 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:00:40.585 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:40 smithi151 ceph-mon[108966]: osdmap e1291: 8 total, 8 up, 6 in 2024-09-13T15:00:40.591 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:40.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:40.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:40.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:40 smithi184 ceph-mon[92527]: 1.0 scrub starts 2024-09-13T15:00:40.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:40 smithi184 ceph-mon[92527]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:00:40.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:40 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:00:40.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:40 smithi184 ceph-mon[92527]: osdmap e1291: 8 total, 8 up, 6 in 2024-09-13T15:00:40.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:00:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:00:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:00:40.694 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:40.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:40.727 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:40.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:40.828 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:40.837 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:00:40.852 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:40.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:40.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:40.954 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:40.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:40.999 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:40.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:41.100 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:41.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:41.129 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:41.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:41.230 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:41.265 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:41.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:41.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:41.367 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:41.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:41.421 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:41.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:41.522 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:41.559 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:41.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:41.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:41.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:41 smithi184 ceph-mon[92527]: pgmap v2134: 119 pgs: 1 active+remapped, 3 active+clean+scrubbing, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 103 active+clean; 5.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 909 op/s; 96373/246441 objects misplaced (39.106%); 13 MiB/s, 202 objects/s recovering 2024-09-13T15:00:41.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:41 smithi151 ceph-mon[104450]: pgmap v2134: 119 pgs: 1 active+remapped, 3 active+clean+scrubbing, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 103 active+clean; 5.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 909 op/s; 96373/246441 objects misplaced (39.106%); 13 MiB/s, 202 objects/s recovering 2024-09-13T15:00:41.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:41 smithi151 ceph-mon[108966]: pgmap v2134: 119 pgs: 1 active+remapped, 3 active+clean+scrubbing, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 103 active+clean; 5.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 909 op/s; 96373/246441 objects misplaced (39.106%); 13 MiB/s, 202 objects/s recovering 2024-09-13T15:00:41.661 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:41.699 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:41.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:41.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:41.801 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:41.836 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:41.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:41.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:41.938 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:41.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:41.975 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:41.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:42.076 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:42.115 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:42.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:42.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:42.217 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:42.257 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:42.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:42.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:42.359 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:42.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:42.382 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:42.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:42.483 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:42.523 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:42.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:42.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:42.625 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:42.626 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T15:00:42.659 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:42.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:42.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:42.761 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:42.793 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:42.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:42.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:42.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:42 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:00:42.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:42 smithi184 ceph-mon[92527]: 1.e scrub starts 2024-09-13T15:00:42.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:42 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:00:42.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:42 smithi184 ceph-mon[92527]: 1.e scrub ok 2024-09-13T15:00:42.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:42 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:00:42.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:42 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:00:42.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:42 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:00:42.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:42 smithi184 ceph-mon[92527]: pgmap v2135: 119 pgs: 1 active+remapped, 3 active+clean+scrubbing, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 103 active+clean; 5.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 909 op/s; 96373/246441 objects misplaced (39.106%); 13 MiB/s, 202 objects/s recovering 2024-09-13T15:00:42.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:42 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:00:42.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:42 smithi151 ceph-mon[104450]: 1.e scrub starts 2024-09-13T15:00:42.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:42 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:00:42.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:42 smithi151 ceph-mon[104450]: 1.e scrub ok 2024-09-13T15:00:42.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:42 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:00:42.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:42 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:00:42.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:42 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:00:42.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:42 smithi151 ceph-mon[104450]: pgmap v2135: 119 pgs: 1 active+remapped, 3 active+clean+scrubbing, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 103 active+clean; 5.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 909 op/s; 96373/246441 objects misplaced (39.106%); 13 MiB/s, 202 objects/s recovering 2024-09-13T15:00:42.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:42 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:00:42.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:42 smithi151 ceph-mon[108966]: 1.e scrub starts 2024-09-13T15:00:42.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:42 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:00:42.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:42 smithi151 ceph-mon[108966]: 1.e scrub ok 2024-09-13T15:00:42.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:42 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:00:42.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:42 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:00:42.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:42 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:00:42.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:42 smithi151 ceph-mon[108966]: pgmap v2135: 119 pgs: 1 active+remapped, 3 active+clean+scrubbing, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 103 active+clean; 5.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 909 op/s; 96373/246441 objects misplaced (39.106%); 13 MiB/s, 202 objects/s recovering 2024-09-13T15:00:42.895 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:42.940 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:42.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:42.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:43.044 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:43.073 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:43.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:43.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:43.176 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:43.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:43.212 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:43.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:43.313 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:43.346 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:43.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:43.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:43.447 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:43.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:43.499 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:43.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:43.600 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:43.624 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:43.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:43.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:43.726 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:43.770 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:43.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:43.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:43.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:43.916 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:43.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:43.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:44.017 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:44.053 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:44.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:44.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:44.155 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:44.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:44.226 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:44.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:44.327 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:44.380 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:44.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:44.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:44.400 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:44.428 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:44.439 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T15:00:44.439 INFO:teuthology.orchestra.run.smithi184.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T15:00:44.439 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T15:00:44.482 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:44.520 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:44.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:44.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:44.623 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:44.657 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:44.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:44.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:44.759 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:44.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:44.797 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:44.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:44.854 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:00:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:00:44] "GET /metrics HTTP/1.1" 200 36762 "" "Prometheus/2.51.0" 2024-09-13T15:00:44.897 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:44.922 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:44.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:44.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:45.024 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:45.056 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:45.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:45.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:45.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:44 smithi184 ceph-mon[92527]: 1.2 scrub starts 2024-09-13T15:00:45.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:44 smithi184 ceph-mon[92527]: 1.2 scrub ok 2024-09-13T15:00:45.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:44 smithi184 ceph-mon[92527]: pgmap v2136: 119 pgs: 1 active+remapped, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 106 active+clean; 4.7 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 870 op/s; 89771/232650 objects misplaced (38.586%); 12 MiB/s, 197 objects/s recovering 2024-09-13T15:00:45.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:44 smithi184 ceph-mon[92527]: 1.0 scrub starts 2024-09-13T15:00:45.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:44 smithi184 ceph-mon[92527]: 1.0 scrub ok 2024-09-13T15:00:45.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:44 smithi151 ceph-mon[104450]: 1.2 scrub starts 2024-09-13T15:00:45.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:44 smithi151 ceph-mon[104450]: 1.2 scrub ok 2024-09-13T15:00:45.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:44 smithi151 ceph-mon[104450]: pgmap v2136: 119 pgs: 1 active+remapped, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 106 active+clean; 4.7 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 870 op/s; 89771/232650 objects misplaced (38.586%); 12 MiB/s, 197 objects/s recovering 2024-09-13T15:00:45.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:44 smithi151 ceph-mon[104450]: 1.0 scrub starts 2024-09-13T15:00:45.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:44 smithi151 ceph-mon[104450]: 1.0 scrub ok 2024-09-13T15:00:45.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:44 smithi151 ceph-mon[108966]: 1.2 scrub starts 2024-09-13T15:00:45.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:44 smithi151 ceph-mon[108966]: 1.2 scrub ok 2024-09-13T15:00:45.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:44 smithi151 ceph-mon[108966]: pgmap v2136: 119 pgs: 1 active+remapped, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 106 active+clean; 4.7 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 870 op/s; 89771/232650 objects misplaced (38.586%); 12 MiB/s, 197 objects/s recovering 2024-09-13T15:00:45.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:44 smithi151 ceph-mon[108966]: 1.0 scrub starts 2024-09-13T15:00:45.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:44 smithi151 ceph-mon[108966]: 1.0 scrub ok 2024-09-13T15:00:45.158 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:45.180 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:45.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:45.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:45.282 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:45.310 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:45.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:45.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:45.412 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:45.447 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:45.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:45.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:45.549 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:45.588 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:45.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:45.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:45.689 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:45.728 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:45.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:45.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:45.830 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:45.857 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:45.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:45.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:45.959 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:45.980 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:45.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:45.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:46.082 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:46.112 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:46.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:46.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:46.214 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:46.249 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:46.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:46.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:46.351 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:46.380 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:46.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:46.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:46.482 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:46.514 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:46.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:46.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:46.616 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:46.658 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:46.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:46.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:46.760 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:46.787 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:46.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:46.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:46.889 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:46.905 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:46 smithi151 ceph-mon[104450]: pgmap v2137: 119 pgs: 1 active+remapped, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 106 active+clean; 4.7 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 870 op/s; 89771/232650 objects misplaced (38.586%); 12 MiB/s, 197 objects/s recovering 2024-09-13T15:00:46.906 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:46.906 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:46 smithi151 ceph-mon[108966]: pgmap v2137: 119 pgs: 1 active+remapped, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 106 active+clean; 4.7 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 870 op/s; 89771/232650 objects misplaced (38.586%); 12 MiB/s, 197 objects/s recovering 2024-09-13T15:00:46.906 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:46.926 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:46.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:46.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:47.028 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:47.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:47.081 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:47.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:47.182 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:47.205 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:47.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:47.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:47.307 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:47.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:46 smithi184 ceph-mon[92527]: pgmap v2137: 119 pgs: 1 active+remapped, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 106 active+clean; 4.7 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 870 op/s; 89771/232650 objects misplaced (38.586%); 12 MiB/s, 197 objects/s recovering 2024-09-13T15:00:47.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:47.358 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:47.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:47.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:47.460 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:47.504 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:47.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:47.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:47.606 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:47.629 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:47.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:47.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:47.730 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:47.763 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:47.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:47.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:47.865 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:47.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:47.903 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:47.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:48.004 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:48.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:48.041 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:48.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:48.142 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:48.193 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:48.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:48.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:48.294 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:48.332 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:48.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:48.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:48.434 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:48.475 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:48.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:48.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:48.577 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:48.613 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:48.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:48.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:48.715 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:48.756 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:48.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:48.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:48.858 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:48.880 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:48.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:48.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:48.982 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:49.014 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:49.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:49.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:49.116 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:49.147 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:49.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:49.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:49.155 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:48 smithi184 ceph-mon[92527]: pgmap v2138: 119 pgs: 1 active+remapped, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 106 active+clean; 4.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 894 op/s; 84668/223509 objects misplaced (37.881%); 11 MiB/s, 172 objects/s recovering 2024-09-13T15:00:49.249 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:49.281 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:49.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:49.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:49.327 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T15:00:49.327 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:00:49.327 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:48 smithi151 ceph-mon[104450]: pgmap v2138: 119 pgs: 1 active+remapped, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 106 active+clean; 4.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 894 op/s; 84668/223509 objects misplaced (37.881%); 11 MiB/s, 172 objects/s recovering 2024-09-13T15:00:49.327 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:48 smithi151 ceph-mon[108966]: pgmap v2138: 119 pgs: 1 active+remapped, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 106 active+clean; 4.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 894 op/s; 84668/223509 objects misplaced (37.881%); 11 MiB/s, 172 objects/s recovering 2024-09-13T15:00:49.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:49.427 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:49.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:49.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:49.529 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:49.567 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:49.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:49.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:49.669 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:49.702 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:49.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:49.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:49.805 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:49.839 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:49.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:49.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:49.941 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:49.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:49.980 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:49.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:50.082 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:50.121 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:50.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:50.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:50.223 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:50.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:50.263 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:50.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:50.364 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:50.400 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:50.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:50.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:50.502 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:50.535 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:50.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:50.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:50.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:00:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:00:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:00:50.637 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:50.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:50.677 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:50.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:50.778 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:50.812 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:50.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:50.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:50.860 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 6, 7, 1, 5] out_osds: [0, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T15:00:50.860 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:00:50.860 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_5 2024-09-13T15:00:50.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:00:50.915 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:50.942 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:50.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:50.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:51.001 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:50 smithi151 ceph-mon[104450]: pgmap v2139: 119 pgs: 1 active+remapped, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 106 active+clean; 4.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 854 op/s; 83401/219540 objects misplaced (37.989%); 10 MiB/s, 164 objects/s recovering 2024-09-13T15:00:51.002 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:50 smithi151 ceph-mon[108966]: pgmap v2139: 119 pgs: 1 active+remapped, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 106 active+clean; 4.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 854 op/s; 83401/219540 objects misplaced (37.989%); 10 MiB/s, 164 objects/s recovering 2024-09-13T15:00:51.044 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:51.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:51.102 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:51.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:51.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:50 smithi184 ceph-mon[92527]: pgmap v2139: 119 pgs: 1 active+remapped, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 106 active+clean; 4.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 854 op/s; 83401/219540 objects misplaced (37.989%); 10 MiB/s, 164 objects/s recovering 2024-09-13T15:00:51.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:51.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:51.261 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:51.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:51.362 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:51.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:51.405 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:51.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:51.507 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:51.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:51.540 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:51.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:51.641 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:51.673 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:51.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:51.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:51.775 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:51.801 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:51.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:51.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:51.903 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:51.943 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:51.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:51.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:52.045 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:52.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:52.078 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:52.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:52.179 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:52.203 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:52.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:52.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:52.305 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:52.338 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:52.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:52.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:52.440 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:52.472 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:52.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:52.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:52.574 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:52.607 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:52.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:52.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:52.709 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:52.743 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:52.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:52.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:52.846 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:52.884 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:52.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:52.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:52.986 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:53.023 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:53.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:53.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:53.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:52 smithi184 ceph-mon[92527]: pgmap v2140: 119 pgs: 1 active+remapped, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 106 active+clean; 4.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 745 op/s; 83401/219540 objects misplaced (37.989%); 9.0 MiB/s, 143 objects/s recovering 2024-09-13T15:00:53.125 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:53.155 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:53.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:53.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:53.257 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:53.293 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:53.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:53.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:53.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:52 smithi151 ceph-mon[108966]: pgmap v2140: 119 pgs: 1 active+remapped, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 106 active+clean; 4.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 745 op/s; 83401/219540 objects misplaced (37.989%); 9.0 MiB/s, 143 objects/s recovering 2024-09-13T15:00:53.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:52 smithi151 ceph-mon[104450]: pgmap v2140: 119 pgs: 1 active+remapped, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 106 active+clean; 4.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 745 op/s; 83401/219540 objects misplaced (37.989%); 9.0 MiB/s, 143 objects/s recovering 2024-09-13T15:00:53.395 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:53.425 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:53.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:53.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:53.427 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T15:00:53.527 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:53.576 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:53.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:53.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:53.678 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:53.712 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:53.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:53.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:53.801 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:53.814 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:53.841 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:53.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:53.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:53.943 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:54.000 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:54.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:54.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:54.097 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T15:00:54.098 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:00:54.106 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:54.135 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:54.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:54.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:54.146 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T15:00:54.146 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:00:54.187 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T15:00:54.187 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:00:54.230 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T15:00:54.231 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:00:54.237 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:54.263 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T15:00:54.263 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:00:54.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:54.266 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:54.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:54.293 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T15:00:54.293 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:00:54.326 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T15:00:54.327 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:00:54.358 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T15:00:54.358 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:00:54.367 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:54.399 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:54.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:54.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:54.501 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:54.550 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:54.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:54.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:54.652 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:54.686 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:54.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:54.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:54.788 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:54.822 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:54.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:54.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:54.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:00:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:00:54] "GET /metrics HTTP/1.1" 200 36750 "" "Prometheus/2.51.0" 2024-09-13T15:00:54.925 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:54.953 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:54.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:54.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:55.055 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:55.093 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:55.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:55.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:55.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:54 smithi151 ceph-mon[104450]: pgmap v2141: 119 pgs: 1 active+remapped, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 106 active+clean; 4.2 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 1.13k op/s; 77056/205806 objects misplaced (37.441%); 13 MiB/s, 210 objects/s recovering 2024-09-13T15:00:55.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:54 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/161131859' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:55.134 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:54 smithi151 ceph-mon[108966]: pgmap v2141: 119 pgs: 1 active+remapped, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 106 active+clean; 4.2 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 1.13k op/s; 77056/205806 objects misplaced (37.441%); 13 MiB/s, 210 objects/s recovering 2024-09-13T15:00:55.135 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:54 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/161131859' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:55.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:55.237 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:55.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:55.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:55.339 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:55.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:54 smithi184 ceph-mon[92527]: pgmap v2141: 119 pgs: 1 active+remapped, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 106 active+clean; 4.2 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 1.13k op/s; 77056/205806 objects misplaced (37.441%); 13 MiB/s, 210 objects/s recovering 2024-09-13T15:00:55.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:54 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/161131859' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:55.383 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:55.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:55.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:55.486 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:55.515 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:55.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:55.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:55.617 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:55.647 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:55.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:55.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:55.749 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:55.779 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:55.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:55.791 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:55.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:55.893 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:55.902 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-13T15:00:55.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:55.944 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:55.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:56.045 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:56.074 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:56.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:56.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:56.175 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:56.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:56.214 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:56.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:56.248 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:55 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/313787666' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:00:56.248 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:55 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:00:56.248 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:55 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/313787666' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:00:56.249 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:55 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:00:56.315 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:56.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:55 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/313787666' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:00:56.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:55 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:00:56.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:56.355 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:56.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:56.456 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:56.490 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:56.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:56.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:56.592 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:56.613 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:56.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:56.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:56.715 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:56.747 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:56.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:56.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:56.849 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:56.878 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:56.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:56.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:56.980 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:57.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:57.015 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:57.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:57.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:56 smithi151 ceph-mon[104450]: pgmap v2142: 119 pgs: 1 active+remapped, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 106 active+clean; 4.2 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 744 op/s; 77056/205806 objects misplaced (37.441%); 8.9 MiB/s, 143 objects/s recovering 2024-09-13T15:00:57.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:56 smithi151 ceph-mon[104450]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:00:57.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:56 smithi151 ceph-mon[104450]: Cluster is now healthy 2024-09-13T15:00:57.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:56 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:00:57.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:56 smithi151 ceph-mon[104450]: osdmap e1292: 8 total, 8 up, 6 in 2024-09-13T15:00:57.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:56 smithi151 ceph-mon[108966]: pgmap v2142: 119 pgs: 1 active+remapped, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 106 active+clean; 4.2 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 744 op/s; 77056/205806 objects misplaced (37.441%); 8.9 MiB/s, 143 objects/s recovering 2024-09-13T15:00:57.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:56 smithi151 ceph-mon[108966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:00:57.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:56 smithi151 ceph-mon[108966]: Cluster is now healthy 2024-09-13T15:00:57.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:56 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:00:57.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:56 smithi151 ceph-mon[108966]: osdmap e1292: 8 total, 8 up, 6 in 2024-09-13T15:00:57.116 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:57.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:57.152 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:57.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:57.253 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:57.283 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:57.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:57.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:57.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:56 smithi184 ceph-mon[92527]: pgmap v2142: 119 pgs: 1 active+remapped, 2 active+remapped+backfilling, 10 active+remapped+backfill_wait, 106 active+clean; 4.2 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 744 op/s; 77056/205806 objects misplaced (37.441%); 8.9 MiB/s, 143 objects/s recovering 2024-09-13T15:00:57.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:56 smithi184 ceph-mon[92527]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:00:57.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:56 smithi184 ceph-mon[92527]: Cluster is now healthy 2024-09-13T15:00:57.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:56 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:00:57.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:56 smithi184 ceph-mon[92527]: osdmap e1292: 8 total, 8 up, 6 in 2024-09-13T15:00:57.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:57.434 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:57.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:57.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:57.535 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:57.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:57.565 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:57.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:57.654 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:00:57.666 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:57.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:57.767 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:57.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:57.868 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:57.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:57.930 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:57.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:58.031 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:58.084 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:58.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:58.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:58.185 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:58.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:58.221 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:58.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:58.322 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:58.362 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:58.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:58.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:58.464 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:58.492 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:58.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:58.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:58.594 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:58.631 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:58.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:58.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:58.733 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:58.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:58.771 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:58.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:58.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:58.909 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:58.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:58.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:59.010 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:59.054 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:59.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:59.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:59.156 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:59.186 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:59.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:59.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:59.288 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:59.324 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:59.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:59.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:59.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:00:58 smithi184 ceph-mon[92527]: pgmap v2144: 119 pgs: 2 active+remapped+backfilling, 11 active+remapped+backfill_wait, 106 active+clean; 4.0 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 894 op/s; 72092/196644 objects misplaced (36.661%); 11 MiB/s, 172 objects/s recovering 2024-09-13T15:00:59.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:00:58 smithi151 ceph-mon[104450]: pgmap v2144: 119 pgs: 2 active+remapped+backfilling, 11 active+remapped+backfill_wait, 106 active+clean; 4.0 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 894 op/s; 72092/196644 objects misplaced (36.661%); 11 MiB/s, 172 objects/s recovering 2024-09-13T15:00:59.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:00:58 smithi151 ceph-mon[108966]: pgmap v2144: 119 pgs: 2 active+remapped+backfilling, 11 active+remapped+backfill_wait, 106 active+clean; 4.0 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 894 op/s; 72092/196644 objects misplaced (36.661%); 11 MiB/s, 172 objects/s recovering 2024-09-13T15:00:59.426 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:59.467 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:59.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:59.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:59.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:59.606 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:00:59.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:59.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:59.708 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:59.736 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:59.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:59.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:59.838 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:59.874 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:59.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:59.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:59.976 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:59.998 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:00:59.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:59.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:00.100 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:00.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:00.139 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:00.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:00.240 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:00.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:00.279 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:00.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:00.380 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:00.421 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:00.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:00.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:00.523 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:00.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:00.554 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:00.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:00.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:01:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:01:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:01:00.654 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:00.682 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:00.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:00.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:00.784 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:00.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:00.822 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:00.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:00.923 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:00.948 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:00.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:00.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:01.050 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:01.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:01.088 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:01.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:01.164 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:00 smithi184 ceph-mon[92527]: pgmap v2145: 119 pgs: 2 active+remapped+backfilling, 11 active+remapped+backfill_wait, 106 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 898 op/s; 70769/192564 objects misplaced (36.751%); 11 MiB/s, 172 objects/s recovering 2024-09-13T15:01:01.189 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:01.223 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:01.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:01.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:01.325 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:01.353 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:00 smithi151 ceph-mon[104450]: pgmap v2145: 119 pgs: 2 active+remapped+backfilling, 11 active+remapped+backfill_wait, 106 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 898 op/s; 70769/192564 objects misplaced (36.751%); 11 MiB/s, 172 objects/s recovering 2024-09-13T15:01:01.353 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:00 smithi151 ceph-mon[108966]: pgmap v2145: 119 pgs: 2 active+remapped+backfilling, 11 active+remapped+backfill_wait, 106 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 898 op/s; 70769/192564 objects misplaced (36.751%); 11 MiB/s, 172 objects/s recovering 2024-09-13T15:01:01.370 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:01.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:01.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:01.473 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:01.498 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:01.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:01.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:01.601 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:01.624 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:01.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:01.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:01.726 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:01.758 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:01.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:01.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:01.860 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:01.901 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:01.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:01.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:02.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:02.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:02.048 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:02.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:02.149 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:02.197 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:02.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:02.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:02.299 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:02.335 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:02.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:02.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:02.437 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:02.481 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:02.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:02.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:02.583 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:02.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:01:02.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:02.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:02 smithi151 ceph-mon[104450]: pgmap v2146: 119 pgs: 2 active+remapped+backfilling, 11 active+remapped+backfill_wait, 106 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 898 op/s; 70769/192564 objects misplaced (36.751%); 11 MiB/s, 172 objects/s recovering 2024-09-13T15:01:02.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:01:02.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:02.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:02 smithi151 ceph-mon[108966]: pgmap v2146: 119 pgs: 2 active+remapped+backfilling, 11 active+remapped+backfill_wait, 106 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 898 op/s; 70769/192564 objects misplaced (36.751%); 11 MiB/s, 172 objects/s recovering 2024-09-13T15:01:02.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:02.629 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:02.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:02.730 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:02.745 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:01:02.745 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:02.746 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:02 smithi184 ceph-mon[92527]: pgmap v2146: 119 pgs: 2 active+remapped+backfilling, 11 active+remapped+backfill_wait, 106 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 898 op/s; 70769/192564 objects misplaced (36.751%); 11 MiB/s, 172 objects/s recovering 2024-09-13T15:01:02.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:02.784 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:02.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:02.885 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:02.927 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:02.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:02.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:03.029 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:03.073 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:03.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:03.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:03.177 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:03.226 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:03.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:03.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:03.328 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:03.366 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:03.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:03.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:03.468 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:03.507 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:03.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:03.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:03.609 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:03.639 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:03.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:03.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:03.741 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:03.770 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:03.812 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:03.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:03.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:03.914 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:03.951 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:03.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:03.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:04.053 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:04.079 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:04.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:04.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:04.161 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T15:01:04.164 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T15:01:04.181 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:04.225 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:04.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:04.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:04.264 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T15:01:04.327 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:04.356 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:04.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:04.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:04.458 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:04.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:04.489 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:04.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:04.590 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:04.616 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:04.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:04.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:04.718 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:04.752 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:04.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:04.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:04.854 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:04.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:01:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:01:04] "GET /metrics HTTP/1.1" 200 36739 "" "Prometheus/2.51.0" 2024-09-13T15:01:04.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:04.876 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:04.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:04.977 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:05.003 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:05.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:05.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:05.105 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:05.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:04 smithi151 ceph-mon[104450]: pgmap v2147: 119 pgs: 2 active+remapped+backfilling, 11 active+remapped+backfill_wait, 106 active+clean; 3.7 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 868 op/s; 64726/179742 objects misplaced (36.011%); 11 MiB/s, 177 objects/s recovering 2024-09-13T15:01:05.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:04 smithi151 ceph-mon[104450]: from='client.46367 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:01:05.113 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:04 smithi151 ceph-mon[108966]: pgmap v2147: 119 pgs: 2 active+remapped+backfilling, 11 active+remapped+backfill_wait, 106 active+clean; 3.7 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 868 op/s; 64726/179742 objects misplaced (36.011%); 11 MiB/s, 177 objects/s recovering 2024-09-13T15:01:05.114 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:04 smithi151 ceph-mon[108966]: from='client.46367 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:01:05.129 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:05.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:05.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:05.231 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:05.270 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:05.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:05.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:05.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:04 smithi184 ceph-mon[92527]: pgmap v2147: 119 pgs: 2 active+remapped+backfilling, 11 active+remapped+backfill_wait, 106 active+clean; 3.7 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 868 op/s; 64726/179742 objects misplaced (36.011%); 11 MiB/s, 177 objects/s recovering 2024-09-13T15:01:05.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:04 smithi184 ceph-mon[92527]: from='client.46367 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:01:05.371 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:05.405 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:05.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:05.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:05.507 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:05.538 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:05.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:05.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:05.640 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:05.662 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:05.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:05.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:05.763 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:05.790 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:05.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:05.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:05.892 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:05.933 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:05.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:05.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:06.036 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:06.063 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:06.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:06.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:06.110 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T15:01:06.110 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-13T15:01:06.110 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-13T15:01:06.110 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-13T15:01:06.110 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T15:01:06.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:06.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:06.202 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:06.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:06.303 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:06.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:06.330 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:06.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:06.431 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:06.461 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:06.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:06.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:06.563 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:06.611 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:06.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:06.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:06.713 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:06.743 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:06.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:06.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:06.845 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:06.874 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:06.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:06.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:06.976 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:07.017 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:07.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:07.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:07.120 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:07.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:07.154 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:07.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:07.255 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:07.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:07.303 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:07.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:07.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:07 smithi184 ceph-mon[92527]: pgmap v2148: 119 pgs: 2 active+remapped+backfilling, 11 active+remapped+backfill_wait, 106 active+clean; 3.7 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 868 op/s; 64726/179742 objects misplaced (36.011%); 11 MiB/s, 177 objects/s recovering 2024-09-13T15:01:07.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:07 smithi151 ceph-mon[104450]: pgmap v2148: 119 pgs: 2 active+remapped+backfilling, 11 active+remapped+backfill_wait, 106 active+clean; 3.7 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 868 op/s; 64726/179742 objects misplaced (36.011%); 11 MiB/s, 177 objects/s recovering 2024-09-13T15:01:07.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:07 smithi151 ceph-mon[108966]: pgmap v2148: 119 pgs: 2 active+remapped+backfilling, 11 active+remapped+backfill_wait, 106 active+clean; 3.7 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 868 op/s; 64726/179742 objects misplaced (36.011%); 11 MiB/s, 177 objects/s recovering 2024-09-13T15:01:07.403 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:07.409 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T15:01:07.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:07.440 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:07.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:07.542 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:07.566 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:07.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:07.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:07.668 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:07.698 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:07.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:07.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:07.800 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:07.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:07.834 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:07.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:07.935 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:07.969 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:07.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:07.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:08.072 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:08.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:08.119 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:08.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:08.220 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:08.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:08.253 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:08.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:08.354 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:08.391 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:08.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:08.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:08.485 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":2147,"stamp":"2024-09-13T15:01:03.805909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3925350373,"num_objects":59914,"num_object_clones":0,"num_object_copies":179742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":64726,"num_objects_unfound":0,"num_objects_dirty":59914,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":191397,"num_write_kb":8039072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23138,"num_bytes_recovered":1511077727,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80239,"ondisk_log_size":80239,"up":357,"acting":357,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":360,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":13300500,"kb_used_data":8170724,"kb_used_omap":106,"kb_used_meta":5129429,"kb_avail":549170412,"statfs":{"total":575970213888,"available":562350501888,"internally_reserved":0,"allocated":8366821376,"data_stored":8350255733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":109003,"internal_metadata":5252535861},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,7,7,2,0,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-569376768,"num_objects":-8688,"num_object_clones":0,"num_object_copies":-26064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-12330,"num_objects_unfound":0,"num_objects_dirty":-8688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8688,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1779,"num_bytes_recovered":116588544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004177"},"pg_stats":[{"pgid":"1.1d","version":"261'100","reported_seq":166,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.905017+0000","last_change":"2024-09-13T15:00:25.065803+0000","last_active":"2024-09-13T15:00:55.905017+0000","last_peered":"2024-09-13T15:00:55.905017+0000","last_clean":"2024-09-13T15:00:55.905017+0000","last_became_active":"2024-09-13T15:00:25.063785+0000","last_became_peered":"2024-09-13T15:00:25.063785+0000","last_unstale":"2024-09-13T15:00:55.905017+0000","last_undegraded":"2024-09-13T15:00:55.905017+0000","last_fullsized":"2024-09-13T15:00:55.905017+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:17.988345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:17.988345+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:43.674995+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":1916,"reported_epoch":1291,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.331237+0000","last_change":"2024-09-13T15:00:05.045993+0000","last_active":"2024-09-13T15:00:39.331237+0000","last_peered":"2024-09-13T15:00:39.331237+0000","last_clean":"2024-09-13T15:00:39.331237+0000","last_became_active":"2024-09-13T15:00:05.033311+0000","last_became_peered":"2024-09-13T15:00:05.033311+0000","last_unstale":"2024-09-13T15:00:39.331237+0000","last_undegraded":"2024-09-13T15:00:39.331237+0000","last_fullsized":"2024-09-13T15:00:39.331237+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:27.290733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2280,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":69,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.904443+0000","last_change":"2024-09-13T15:00:25.180918+0000","last_active":"2024-09-13T15:00:55.904443+0000","last_peered":"2024-09-13T15:00:55.904443+0000","last_clean":"2024-09-13T15:00:55.904443+0000","last_became_active":"2024-09-13T15:00:25.180571+0000","last_became_peered":"2024-09-13T15:00:25.180571+0000","last_unstale":"2024-09-13T15:00:55.904443+0000","last_undegraded":"2024-09-13T15:00:55.904443+0000","last_fullsized":"2024-09-13T15:00:55.904443+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:31.846606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":305,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.909977+0000","last_change":"2024-09-13T14:45:19.810203+0000","last_active":"2024-09-13T15:00:55.909977+0000","last_peered":"2024-09-13T15:00:55.909977+0000","last_clean":"2024-09-13T15:00:55.909977+0000","last_became_active":"2024-09-13T14:45:19.809693+0000","last_became_peered":"2024-09-13T14:45:19.809693+0000","last_unstale":"2024-09-13T15:00:55.909977+0000","last_undegraded":"2024-09-13T15:00:55.909977+0000","last_fullsized":"2024-09-13T15:00:55.909977+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:20.219673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":311,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.910685+0000","last_change":"2024-09-13T15:00:05.051083+0000","last_active":"2024-09-13T15:00:55.910685+0000","last_peered":"2024-09-13T15:00:55.910685+0000","last_clean":"2024-09-13T15:00:55.910685+0000","last_became_active":"2024-09-13T15:00:05.050737+0000","last_became_peered":"2024-09-13T15:00:05.050737+0000","last_unstale":"2024-09-13T15:00:55.910685+0000","last_undegraded":"2024-09-13T15:00:55.910685+0000","last_fullsized":"2024-09-13T15:00:55.910685+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:58.282196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"261'100","reported_seq":1350,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.909550+0000","last_change":"2024-09-13T14:59:50.035950+0000","last_active":"2024-09-13T15:00:55.909550+0000","last_peered":"2024-09-13T15:00:55.909550+0000","last_clean":"2024-09-13T15:00:55.909550+0000","last_became_active":"2024-09-13T14:59:50.035558+0000","last_became_peered":"2024-09-13T14:59:50.035558+0000","last_unstale":"2024-09-13T15:00:55.909550+0000","last_undegraded":"2024-09-13T15:00:55.909550+0000","last_fullsized":"2024-09-13T15:00:55.909550+0000","mapping_epoch":1280,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1281,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_clean_scrub_stamp":"2024-09-13T14:44:54.259482+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:18:34.167138+0000","scrub_duration":0.015265569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":83,"reported_epoch":1291,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.330903+0000","last_change":"2024-09-13T15:00:25.072110+0000","last_active":"2024-09-13T15:00:39.330903+0000","last_peered":"2024-09-13T15:00:39.330903+0000","last_clean":"2024-09-13T15:00:39.330903+0000","last_became_active":"2024-09-13T15:00:25.045594+0000","last_became_peered":"2024-09-13T15:00:25.045594+0000","last_unstale":"2024-09-13T15:00:39.330903+0000","last_undegraded":"2024-09-13T15:00:39.330903+0000","last_fullsized":"2024-09-13T15:00:39.330903+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:59:45.210187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":92,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.910085+0000","last_change":"2024-09-13T15:00:25.052537+0000","last_active":"2024-09-13T15:00:55.910085+0000","last_peered":"2024-09-13T15:00:55.910085+0000","last_clean":"2024-09-13T15:00:55.910085+0000","last_became_active":"2024-09-13T15:00:25.052225+0000","last_became_peered":"2024-09-13T15:00:25.052225+0000","last_unstale":"2024-09-13T15:00:55.910085+0000","last_undegraded":"2024-09-13T15:00:55.910085+0000","last_fullsized":"2024-09-13T15:00:55.910085+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:52.039052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1e","version":"261'100","reported_seq":322,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.910178+0000","last_change":"2024-09-13T14:45:35.947332+0000","last_active":"2024-09-13T15:00:55.910178+0000","last_peered":"2024-09-13T15:00:55.910178+0000","last_clean":"2024-09-13T15:00:55.910178+0000","last_became_active":"2024-09-13T14:45:19.810775+0000","last_became_peered":"2024-09-13T14:45:19.810775+0000","last_unstale":"2024-09-13T15:00:55.910178+0000","last_undegraded":"2024-09-13T15:00:55.910178+0000","last_fullsized":"2024-09-13T15:00:55.910178+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:35.947265+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:43:41.569795+0000","last_clean_scrub_stamp":"2024-09-13T14:45:35.947265+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:00.606370+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":83,"reported_epoch":1291,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.330668+0000","last_change":"2024-09-13T15:00:25.192972+0000","last_active":"2024-09-13T15:00:39.330668+0000","last_peered":"2024-09-13T15:00:39.330668+0000","last_clean":"2024-09-13T15:00:39.330668+0000","last_became_active":"2024-09-13T15:00:25.047858+0000","last_became_peered":"2024-09-13T15:00:25.047858+0000","last_unstale":"2024-09-13T15:00:39.330668+0000","last_undegraded":"2024-09-13T15:00:39.330668+0000","last_fullsized":"2024-09-13T15:00:39.330668+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:22:16.795970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1230,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.911280+0000","last_change":"2024-09-13T15:00:25.086077+0000","last_active":"2024-09-13T15:00:55.911280+0000","last_peered":"2024-09-13T15:00:55.911280+0000","last_clean":"2024-09-13T15:00:55.911280+0000","last_became_active":"2024-09-13T15:00:25.085685+0000","last_became_peered":"2024-09-13T15:00:25.085685+0000","last_unstale":"2024-09-13T15:00:55.911280+0000","last_undegraded":"2024-09-13T15:00:55.911280+0000","last_fullsized":"2024-09-13T15:00:55.911280+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:09:24.157291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":1971,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.907758+0000","last_change":"2024-09-13T15:00:25.084862+0000","last_active":"2024-09-13T15:00:55.907758+0000","last_peered":"2024-09-13T15:00:55.907758+0000","last_clean":"2024-09-13T15:00:55.907758+0000","last_became_active":"2024-09-13T15:00:25.083352+0000","last_became_peered":"2024-09-13T15:00:25.083352+0000","last_unstale":"2024-09-13T15:00:55.907758+0000","last_undegraded":"2024-09-13T15:00:55.907758+0000","last_fullsized":"2024-09-13T15:00:55.907758+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:27.541005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1986,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.906427+0000","last_change":"2024-09-13T14:54:05.592484+0000","last_active":"2024-09-13T15:00:55.906427+0000","last_peered":"2024-09-13T15:00:55.906427+0000","last_clean":"2024-09-13T15:00:55.906427+0000","last_became_active":"2024-09-13T14:54:05.592159+0000","last_became_peered":"2024-09-13T14:54:05.592159+0000","last_unstale":"2024-09-13T15:00:55.906427+0000","last_undegraded":"2024-09-13T15:00:55.906427+0000","last_fullsized":"2024-09-13T15:00:55.906427+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:02.607037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"261'100","reported_seq":318,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.910800+0000","last_change":"2024-09-13T15:00:25.083244+0000","last_active":"2024-09-13T15:00:55.910800+0000","last_peered":"2024-09-13T15:00:55.910800+0000","last_clean":"2024-09-13T15:00:55.910800+0000","last_became_active":"2024-09-13T15:00:25.082659+0000","last_became_peered":"2024-09-13T15:00:25.082659+0000","last_unstale":"2024-09-13T15:00:55.910800+0000","last_undegraded":"2024-09-13T15:00:55.910800+0000","last_fullsized":"2024-09-13T15:00:55.910800+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.614340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.614340+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:29.358567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":38,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.919956+0000","last_change":"2024-09-13T15:00:25.078983+0000","last_active":"2024-09-13T15:00:55.919956+0000","last_peered":"2024-09-13T15:00:55.919956+0000","last_clean":"2024-09-13T15:00:55.919956+0000","last_became_active":"2024-09-13T15:00:25.077787+0000","last_became_peered":"2024-09-13T15:00:25.077787+0000","last_unstale":"2024-09-13T15:00:55.919956+0000","last_undegraded":"2024-09-13T15:00:55.919956+0000","last_fullsized":"2024-09-13T15:00:55.919956+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:45.760538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":346,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.913765+0000","last_change":"2024-09-13T15:00:25.052382+0000","last_active":"2024-09-13T15:00:55.913765+0000","last_peered":"2024-09-13T15:00:55.913765+0000","last_clean":"2024-09-13T15:00:55.913765+0000","last_became_active":"2024-09-13T15:00:25.051999+0000","last_became_peered":"2024-09-13T15:00:25.051999+0000","last_unstale":"2024-09-13T15:00:55.913765+0000","last_undegraded":"2024-09-13T15:00:55.913765+0000","last_fullsized":"2024-09-13T15:00:55.913765+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:30.490420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.18","version":"261'100","reported_seq":86,"reported_epoch":1291,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.330613+0000","last_change":"2024-09-13T14:59:50.062546+0000","last_active":"2024-09-13T15:00:39.330613+0000","last_peered":"2024-09-13T15:00:39.330613+0000","last_clean":"2024-09-13T15:00:39.330613+0000","last_became_active":"2024-09-13T14:59:50.035048+0000","last_became_peered":"2024-09-13T14:59:50.035048+0000","last_unstale":"2024-09-13T15:00:39.330613+0000","last_undegraded":"2024-09-13T15:00:39.330613+0000","last_fullsized":"2024-09-13T15:00:39.330613+0000","mapping_epoch":1280,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1281,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:56.548855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:56.548855+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:22:07.889266+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":788064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":166,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.903350+0000","last_change":"2024-09-13T15:00:25.048396+0000","last_active":"2024-09-13T15:00:55.903350+0000","last_peered":"2024-09-13T15:00:55.903350+0000","last_clean":"2024-09-13T15:00:55.903350+0000","last_became_active":"2024-09-13T15:00:25.048084+0000","last_became_peered":"2024-09-13T15:00:25.048084+0000","last_unstale":"2024-09-13T15:00:55.903350+0000","last_undegraded":"2024-09-13T15:00:55.903350+0000","last_fullsized":"2024-09-13T15:00:55.903350+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:05:23.139064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1990,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.909755+0000","last_change":"2024-09-13T15:00:05.046054+0000","last_active":"2024-09-13T15:00:55.909755+0000","last_peered":"2024-09-13T15:00:55.909755+0000","last_clean":"2024-09-13T15:00:55.909755+0000","last_became_active":"2024-09-13T15:00:05.045312+0000","last_became_peered":"2024-09-13T15:00:05.045312+0000","last_unstale":"2024-09-13T15:00:55.909755+0000","last_undegraded":"2024-09-13T15:00:55.909755+0000","last_fullsized":"2024-09-13T15:00:55.909755+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:13:50.497053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"261'100","reported_seq":85,"reported_epoch":1291,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.330778+0000","last_change":"2024-09-13T15:00:25.059447+0000","last_active":"2024-09-13T15:00:39.330778+0000","last_peered":"2024-09-13T15:00:39.330778+0000","last_clean":"2024-09-13T15:00:39.330778+0000","last_became_active":"2024-09-13T15:00:25.057619+0000","last_became_peered":"2024-09-13T15:00:25.057619+0000","last_unstale":"2024-09-13T15:00:39.330778+0000","last_undegraded":"2024-09-13T15:00:39.330778+0000","last_fullsized":"2024-09-13T15:00:39.330778+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:55.988431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:55.988431+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:46:02.664387+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":325,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.910288+0000","last_change":"2024-09-13T15:00:05.047270+0000","last_active":"2024-09-13T15:00:55.910288+0000","last_peered":"2024-09-13T15:00:55.910288+0000","last_clean":"2024-09-13T15:00:55.910288+0000","last_became_active":"2024-09-13T15:00:05.046710+0000","last_became_peered":"2024-09-13T15:00:05.046710+0000","last_unstale":"2024-09-13T15:00:55.910288+0000","last_undegraded":"2024-09-13T15:00:55.910288+0000","last_fullsized":"2024-09-13T15:00:55.910288+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:46:06.874757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":65,"reported_epoch":1291,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.331313+0000","last_change":"2024-09-13T14:55:41.883426+0000","last_active":"2024-09-13T15:00:39.331313+0000","last_peered":"2024-09-13T15:00:39.331313+0000","last_clean":"2024-09-13T15:00:39.331313+0000","last_became_active":"2024-09-13T14:55:41.880724+0000","last_became_peered":"2024-09-13T14:55:41.880724+0000","last_unstale":"2024-09-13T15:00:39.331313+0000","last_undegraded":"2024-09-13T15:00:39.331313+0000","last_fullsized":"2024-09-13T15:00:39.331313+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:17.920508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1a","version":"261'100","reported_seq":119,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.912522+0000","last_change":"2024-09-13T15:00:25.079177+0000","last_active":"2024-09-13T15:00:55.912522+0000","last_peered":"2024-09-13T15:00:55.912522+0000","last_clean":"2024-09-13T15:00:55.912522+0000","last_became_active":"2024-09-13T15:00:25.078209+0000","last_became_peered":"2024-09-13T15:00:25.078209+0000","last_unstale":"2024-09-13T15:00:55.912522+0000","last_undegraded":"2024-09-13T15:00:55.912522+0000","last_fullsized":"2024-09-13T15:00:55.912522+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:46.985510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:46.985510+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:29.828691+0000","scrub_duration":0.0030525999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":325,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.910377+0000","last_change":"2024-09-13T14:54:05.606882+0000","last_active":"2024-09-13T15:00:55.910377+0000","last_peered":"2024-09-13T15:00:55.910377+0000","last_clean":"2024-09-13T15:00:55.910377+0000","last_became_active":"2024-09-13T14:54:05.605912+0000","last_became_peered":"2024-09-13T14:54:05.605912+0000","last_unstale":"2024-09-13T15:00:55.910377+0000","last_undegraded":"2024-09-13T15:00:55.910377+0000","last_fullsized":"2024-09-13T15:00:55.910377+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:05.445329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1931,"reported_epoch":1291,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.331216+0000","last_change":"2024-09-13T15:00:05.037972+0000","last_active":"2024-09-13T15:00:39.331216+0000","last_peered":"2024-09-13T15:00:39.331216+0000","last_clean":"2024-09-13T15:00:39.331216+0000","last_became_active":"2024-09-13T15:00:05.037561+0000","last_became_peered":"2024-09-13T15:00:05.037561+0000","last_unstale":"2024-09-13T15:00:39.331216+0000","last_undegraded":"2024-09-13T15:00:39.331216+0000","last_fullsized":"2024-09-13T15:00:39.331216+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:42:04.578024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.15","version":"261'100","reported_seq":203,"reported_epoch":1291,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.330550+0000","last_change":"2024-09-13T14:59:50.042840+0000","last_active":"2024-09-13T15:00:39.330550+0000","last_peered":"2024-09-13T15:00:39.330550+0000","last_clean":"2024-09-13T15:00:39.330550+0000","last_became_active":"2024-09-13T14:59:50.041801+0000","last_became_peered":"2024-09-13T14:59:50.041801+0000","last_unstale":"2024-09-13T15:00:39.330550+0000","last_undegraded":"2024-09-13T15:00:39.330550+0000","last_fullsized":"2024-09-13T15:00:39.330550+0000","mapping_epoch":1280,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1281,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:38.214865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:38.214865+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:53.905171+0000","scrub_duration":0.0016389639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1883,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.910931+0000","last_change":"2024-09-13T14:59:50.041421+0000","last_active":"2024-09-13T15:00:55.910931+0000","last_peered":"2024-09-13T15:00:55.910931+0000","last_clean":"2024-09-13T15:00:55.910931+0000","last_became_active":"2024-09-13T14:59:50.040879+0000","last_became_peered":"2024-09-13T14:59:50.040879+0000","last_unstale":"2024-09-13T15:00:55.910931+0000","last_undegraded":"2024-09-13T15:00:55.910931+0000","last_fullsized":"2024-09-13T15:00:55.910931+0000","mapping_epoch":1280,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:17.085721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":75,"reported_epoch":1291,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.331064+0000","last_change":"2024-09-13T14:59:50.036785+0000","last_active":"2024-09-13T15:00:39.331064+0000","last_peered":"2024-09-13T15:00:39.331064+0000","last_clean":"2024-09-13T15:00:39.331064+0000","last_became_active":"2024-09-13T14:59:50.036137+0000","last_became_peered":"2024-09-13T14:59:50.036137+0000","last_unstale":"2024-09-13T15:00:39.331064+0000","last_undegraded":"2024-09-13T15:00:39.331064+0000","last_fullsized":"2024-09-13T15:00:39.331064+0000","mapping_epoch":1280,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:45:05.487536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"261'100","reported_seq":395,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.906276+0000","last_change":"2024-09-13T15:00:25.053853+0000","last_active":"2024-09-13T15:00:55.906276+0000","last_peered":"2024-09-13T15:00:55.906276+0000","last_clean":"2024-09-13T15:00:55.906276+0000","last_became_active":"2024-09-13T15:00:25.053479+0000","last_became_peered":"2024-09-13T15:00:25.053479+0000","last_unstale":"2024-09-13T15:00:55.906276+0000","last_undegraded":"2024-09-13T15:00:55.906276+0000","last_fullsized":"2024-09-13T15:00:55.906276+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.037517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.037517+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:18:33.947456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":78,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.913888+0000","last_change":"2024-09-13T15:00:25.104457+0000","last_active":"2024-09-13T15:00:55.913888+0000","last_peered":"2024-09-13T15:00:55.913888+0000","last_clean":"2024-09-13T15:00:55.913888+0000","last_became_active":"2024-09-13T15:00:25.062721+0000","last_became_peered":"2024-09-13T15:00:25.062721+0000","last_unstale":"2024-09-13T15:00:55.913888+0000","last_undegraded":"2024-09-13T15:00:55.913888+0000","last_fullsized":"2024-09-13T15:00:55.913888+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:45.547513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":270,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":152,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.912718+0000","last_change":"2024-09-13T14:54:05.614569+0000","last_active":"2024-09-13T15:00:55.912718+0000","last_peered":"2024-09-13T15:00:55.912718+0000","last_clean":"2024-09-13T15:00:55.912718+0000","last_became_active":"2024-09-13T14:54:05.611810+0000","last_became_peered":"2024-09-13T14:54:05.611810+0000","last_unstale":"2024-09-13T15:00:55.912718+0000","last_undegraded":"2024-09-13T15:00:55.912718+0000","last_fullsized":"2024-09-13T15:00:55.912718+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:46:02.721438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"261'100","reported_seq":75,"reported_epoch":1291,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.331295+0000","last_change":"2024-09-13T15:00:05.039513+0000","last_active":"2024-09-13T15:00:39.331295+0000","last_peered":"2024-09-13T15:00:39.331295+0000","last_clean":"2024-09-13T15:00:39.331295+0000","last_became_active":"2024-09-13T15:00:05.038368+0000","last_became_peered":"2024-09-13T15:00:05.038368+0000","last_unstale":"2024-09-13T15:00:39.331295+0000","last_undegraded":"2024-09-13T15:00:39.331295+0000","last_fullsized":"2024-09-13T15:00:39.331295+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:54.933410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:54.933410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:18.888094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":102,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.906040+0000","last_change":"2024-09-13T15:00:25.097179+0000","last_active":"2024-09-13T15:00:55.906040+0000","last_peered":"2024-09-13T15:00:55.906040+0000","last_clean":"2024-09-13T15:00:55.906040+0000","last_became_active":"2024-09-13T15:00:25.061207+0000","last_became_peered":"2024-09-13T15:00:25.061207+0000","last_unstale":"2024-09-13T15:00:55.906040+0000","last_undegraded":"2024-09-13T15:00:55.906040+0000","last_fullsized":"2024-09-13T15:00:55.906040+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:57.246284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":38,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.908222+0000","last_change":"2024-09-13T15:00:25.091881+0000","last_active":"2024-09-13T15:00:55.908222+0000","last_peered":"2024-09-13T15:00:55.908222+0000","last_clean":"2024-09-13T15:00:55.908222+0000","last_became_active":"2024-09-13T15:00:25.091538+0000","last_became_peered":"2024-09-13T15:00:25.091538+0000","last_unstale":"2024-09-13T15:00:55.908222+0000","last_undegraded":"2024-09-13T15:00:55.908222+0000","last_fullsized":"2024-09-13T15:00:55.908222+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:29:32.889528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.16","version":"261'100","reported_seq":166,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.906676+0000","last_change":"2024-09-13T15:00:25.083905+0000","last_active":"2024-09-13T15:00:55.906676+0000","last_peered":"2024-09-13T15:00:55.906676+0000","last_clean":"2024-09-13T15:00:55.906676+0000","last_became_active":"2024-09-13T15:00:25.082642+0000","last_became_peered":"2024-09-13T15:00:25.082642+0000","last_unstale":"2024-09-13T15:00:55.906676+0000","last_undegraded":"2024-09-13T15:00:55.906676+0000","last_fullsized":"2024-09-13T15:00:55.906676+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:48.363045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:48.363045+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:31:38.562138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.f","version":"1292'11992","reported_seq":12024,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:01:02.389424+0000","last_change":"2024-09-13T14:58:06.200417+0000","last_active":"2024-09-13T15:01:02.389424+0000","last_peered":"2024-09-13T15:01:02.389424+0000","last_clean":"2024-09-13T15:01:02.389424+0000","last_became_active":"2024-09-13T14:58:06.200062+0000","last_became_peered":"2024-09-13T14:58:06.200062+0000","last_unstale":"2024-09-13T15:01:02.389424+0000","last_undegraded":"2024-09-13T15:01:02.389424+0000","last_fullsized":"2024-09-13T15:01:02.389424+0000","mapping_epoch":1268,"log_start":"1279'7697","ondisk_log_start":"1279'7697","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":4295,"log_dups_size":0,"ondisk_log_size":4295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:53:26.095129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251920384,"num_objects":3844,"num_object_clones":0,"num_object_copies":11532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11992,"num_write_kb":506752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":45,"reported_epoch":1291,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.331240+0000","last_change":"2024-09-13T15:00:25.076862+0000","last_active":"2024-09-13T15:00:39.331240+0000","last_peered":"2024-09-13T15:00:39.331240+0000","last_clean":"2024-09-13T15:00:39.331240+0000","last_became_active":"2024-09-13T15:00:25.076536+0000","last_became_peered":"2024-09-13T15:00:25.076536+0000","last_unstale":"2024-09-13T15:00:39.331240+0000","last_undegraded":"2024-09-13T15:00:39.331240+0000","last_fullsized":"2024-09-13T15:00:39.331240+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:24:49.401862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":583,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.903053+0000","last_change":"2024-09-13T14:59:50.049229+0000","last_active":"2024-09-13T15:00:55.903053+0000","last_peered":"2024-09-13T15:00:55.903053+0000","last_clean":"2024-09-13T15:00:55.903053+0000","last_became_active":"2024-09-13T14:59:50.047053+0000","last_became_peered":"2024-09-13T14:59:50.047053+0000","last_unstale":"2024-09-13T15:00:55.903053+0000","last_undegraded":"2024-09-13T15:00:55.903053+0000","last_fullsized":"2024-09-13T15:00:55.903053+0000","mapping_epoch":1280,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:04.011882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":1926,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.906867+0000","last_change":"2024-09-13T15:00:25.158217+0000","last_active":"2024-09-13T15:00:55.906867+0000","last_peered":"2024-09-13T15:00:55.906867+0000","last_clean":"2024-09-13T15:00:55.906867+0000","last_became_active":"2024-09-13T15:00:25.084049+0000","last_became_peered":"2024-09-13T15:00:25.084049+0000","last_unstale":"2024-09-13T15:00:55.906867+0000","last_undegraded":"2024-09-13T15:00:55.906867+0000","last_fullsized":"2024-09-13T15:00:55.906867+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:42:23.145157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"261'100","reported_seq":340,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.912740+0000","last_change":"2024-09-13T14:59:50.045318+0000","last_active":"2024-09-13T15:00:55.912740+0000","last_peered":"2024-09-13T15:00:55.912740+0000","last_clean":"2024-09-13T15:00:55.912740+0000","last_became_active":"2024-09-13T14:59:50.044120+0000","last_became_peered":"2024-09-13T14:59:50.044120+0000","last_unstale":"2024-09-13T15:00:55.912740+0000","last_undegraded":"2024-09-13T15:00:55.912740+0000","last_fullsized":"2024-09-13T15:00:55.912740+0000","mapping_epoch":1280,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1281,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.710192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.710192+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:10.435669+0000","scrub_duration":0.0019250910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"1292'11720","reported_seq":11760,"reported_epoch":1292,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T15:01:02.371275+0000","last_change":"2024-09-13T15:00:26.173324+0000","last_active":"2024-09-13T15:01:02.371275+0000","last_peered":"2024-09-13T15:01:02.371275+0000","last_clean":"2024-09-13T15:00:24.466600+0000","last_became_active":"2024-09-13T15:00:26.128111+0000","last_became_peered":"2024-09-13T15:00:26.128111+0000","last_unstale":"2024-09-13T15:01:02.371275+0000","last_undegraded":"2024-09-13T15:01:02.371275+0000","last_fullsized":"2024-09-13T15:01:02.371275+0000","mapping_epoch":1289,"log_start":"1279'7433","ondisk_log_start":"1279'7433","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":4287,"log_dups_size":0,"ondisk_log_size":4287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:45.983247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240910336,"num_objects":3676,"num_object_clones":0,"num_object_copies":11028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3676,"num_objects_unfound":0,"num_objects_dirty":3676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11720,"num_write_kb":492672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,4],"avail_no_missing":["6","2","4"],"object_location_counts":[{"shards":"2,4,6","objects":3676}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":222,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.910173+0000","last_change":"2024-09-13T15:00:05.047001+0000","last_active":"2024-09-13T15:00:55.910173+0000","last_peered":"2024-09-13T15:00:55.910173+0000","last_clean":"2024-09-13T15:00:55.910173+0000","last_became_active":"2024-09-13T15:00:05.046258+0000","last_became_peered":"2024-09-13T15:00:05.046258+0000","last_unstale":"2024-09-13T15:00:55.910173+0000","last_undegraded":"2024-09-13T15:00:55.910173+0000","last_fullsized":"2024-09-13T15:00:55.910173+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:52:24.589094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":167,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.906763+0000","last_change":"2024-09-13T15:00:25.070169+0000","last_active":"2024-09-13T15:00:55.906763+0000","last_peered":"2024-09-13T15:00:55.906763+0000","last_clean":"2024-09-13T15:00:55.906763+0000","last_became_active":"2024-09-13T15:00:25.069849+0000","last_became_peered":"2024-09-13T15:00:25.069849+0000","last_unstale":"2024-09-13T15:00:55.906763+0000","last_undegraded":"2024-09-13T15:00:55.906763+0000","last_fullsized":"2024-09-13T15:00:55.906763+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:56.491207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":1942,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.911241+0000","last_change":"2024-09-13T15:00:25.100071+0000","last_active":"2024-09-13T15:00:55.911241+0000","last_peered":"2024-09-13T15:00:55.911241+0000","last_clean":"2024-09-13T15:00:55.911241+0000","last_became_active":"2024-09-13T15:00:25.048021+0000","last_became_peered":"2024-09-13T15:00:25.048021+0000","last_unstale":"2024-09-13T15:00:55.911241+0000","last_undegraded":"2024-09-13T15:00:55.911241+0000","last_fullsized":"2024-09-13T15:00:55.911241+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:48.062119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"261'100","reported_seq":159,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.903328+0000","last_change":"2024-09-13T15:00:25.060018+0000","last_active":"2024-09-13T15:00:55.903328+0000","last_peered":"2024-09-13T15:00:55.903328+0000","last_clean":"2024-09-13T15:00:55.903328+0000","last_became_active":"2024-09-13T15:00:25.059438+0000","last_became_peered":"2024-09-13T15:00:25.059438+0000","last_unstale":"2024-09-13T15:00:55.903328+0000","last_undegraded":"2024-09-13T15:00:55.903328+0000","last_fullsized":"2024-09-13T15:00:55.903328+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:51:31.310115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:51:31.310115+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:08.861237+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.d","version":"1292'12123","reported_seq":23700,"reported_epoch":1292,"state":"active+remapped+backfilling","last_fresh":"2024-09-13T15:01:02.469534+0000","last_change":"2024-09-13T15:00:25.200255+0000","last_active":"2024-09-13T15:01:02.469534+0000","last_peered":"2024-09-13T15:01:02.469534+0000","last_clean":"2024-09-13T15:00:04.122591+0000","last_became_active":"2024-09-13T15:00:25.136426+0000","last_became_peered":"2024-09-13T15:00:25.136426+0000","last_unstale":"2024-09-13T15:01:02.469534+0000","last_undegraded":"2024-09-13T15:01:02.469534+0000","last_fullsized":"2024-09-13T15:01:02.469534+0000","mapping_epoch":1288,"log_start":"1275'5297","ondisk_log_start":"1275'5297","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":6826,"log_dups_size":0,"ondisk_log_size":6826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:18.519448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244449304,"num_objects":3731,"num_object_clones":0,"num_object_copies":11193,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1520,"num_objects_unfound":0,"num_objects_dirty":3731,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12123,"num_write_kb":507265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3841,"num_bytes_recovered":251658264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[4,6,0],"avail_no_missing":["4","0","6"],"object_location_counts":[{"shards":"0,4,6","objects":3731}],"blocked_by":[],"up_primary":3,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1979,"reported_epoch":1291,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.330683+0000","last_change":"2024-09-13T15:00:05.044007+0000","last_active":"2024-09-13T15:00:39.330683+0000","last_peered":"2024-09-13T15:00:39.330683+0000","last_clean":"2024-09-13T15:00:39.330683+0000","last_became_active":"2024-09-13T15:00:05.043705+0000","last_became_peered":"2024-09-13T15:00:05.043705+0000","last_unstale":"2024-09-13T15:00:39.330683+0000","last_undegraded":"2024-09-13T15:00:39.330683+0000","last_fullsized":"2024-09-13T15:00:39.330683+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:39:06.976405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1251,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.904703+0000","last_change":"2024-09-13T15:00:25.064155+0000","last_active":"2024-09-13T15:00:55.904703+0000","last_peered":"2024-09-13T15:00:55.904703+0000","last_clean":"2024-09-13T15:00:55.904703+0000","last_became_active":"2024-09-13T15:00:25.063730+0000","last_became_peered":"2024-09-13T15:00:25.063730+0000","last_unstale":"2024-09-13T15:00:55.904703+0000","last_undegraded":"2024-09-13T15:00:55.904703+0000","last_fullsized":"2024-09-13T15:00:55.904703+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:07.209522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":167,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.909910+0000","last_change":"2024-09-13T14:55:41.892125+0000","last_active":"2024-09-13T15:00:55.909910+0000","last_peered":"2024-09-13T15:00:55.909910+0000","last_clean":"2024-09-13T15:00:55.909910+0000","last_became_active":"2024-09-13T14:55:41.891812+0000","last_became_peered":"2024-09-13T14:55:41.891812+0000","last_unstale":"2024-09-13T15:00:55.909910+0000","last_undegraded":"2024-09-13T15:00:55.909910+0000","last_fullsized":"2024-09-13T15:00:55.909910+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:03.359541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"261'100","reported_seq":405,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.909870+0000","last_change":"2024-09-13T15:00:25.052901+0000","last_active":"2024-09-13T15:00:55.909870+0000","last_peered":"2024-09-13T15:00:55.909870+0000","last_clean":"2024-09-13T15:00:55.909870+0000","last_became_active":"2024-09-13T15:00:25.052425+0000","last_became_peered":"2024-09-13T15:00:25.052425+0000","last_unstale":"2024-09-13T15:00:55.909870+0000","last_undegraded":"2024-09-13T15:00:55.909870+0000","last_fullsized":"2024-09-13T15:00:55.909870+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:33.574443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:33.574443+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:12:10.645768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"1292'11862","reported_seq":15761,"reported_epoch":1292,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T15:01:02.400073+0000","last_change":"2024-09-13T15:00:55.222929+0000","last_active":"2024-09-13T15:01:02.400073+0000","last_peered":"2024-09-13T15:01:02.400073+0000","last_clean":"2024-09-13T14:59:49.046145+0000","last_became_active":"2024-09-13T15:00:25.146340+0000","last_became_peered":"2024-09-13T15:00:25.146340+0000","last_unstale":"2024-09-13T15:01:02.400073+0000","last_undegraded":"2024-09-13T15:01:02.400073+0000","last_fullsized":"2024-09-13T15:01:02.400073+0000","mapping_epoch":1288,"log_start":"1275'5059","ondisk_log_start":"1275'5059","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":6803,"log_dups_size":0,"ondisk_log_size":6803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:44.469089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248774656,"num_objects":3796,"num_object_clones":0,"num_object_copies":11388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10398,"num_objects_unfound":0,"num_objects_dirty":3796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11862,"num_write_kb":501056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1915,"num_bytes_recovered":125435904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[4,0,7],"avail_no_missing":["4","0","7"],"object_location_counts":[{"shards":"0,4,7","objects":3796}],"blocked_by":[],"up_primary":3,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":181,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.919855+0000","last_change":"2024-09-13T15:00:25.079609+0000","last_active":"2024-09-13T15:00:55.919855+0000","last_peered":"2024-09-13T15:00:55.919855+0000","last_clean":"2024-09-13T15:00:55.919855+0000","last_became_active":"2024-09-13T15:00:25.077887+0000","last_became_peered":"2024-09-13T15:00:25.077887+0000","last_unstale":"2024-09-13T15:00:55.919855+0000","last_undegraded":"2024-09-13T15:00:55.919855+0000","last_fullsized":"2024-09-13T15:00:55.919855+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:13:04.513297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":167,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.909835+0000","last_change":"2024-09-13T15:00:05.047799+0000","last_active":"2024-09-13T15:00:55.909835+0000","last_peered":"2024-09-13T15:00:55.909835+0000","last_clean":"2024-09-13T15:00:55.909835+0000","last_became_active":"2024-09-13T15:00:05.046576+0000","last_became_peered":"2024-09-13T15:00:05.046576+0000","last_unstale":"2024-09-13T15:00:55.909835+0000","last_undegraded":"2024-09-13T15:00:55.909835+0000","last_fullsized":"2024-09-13T15:00:55.909835+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:19.063120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":65,"reported_epoch":1291,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.330958+0000","last_change":"2024-09-13T14:55:41.839293+0000","last_active":"2024-09-13T15:00:39.330958+0000","last_peered":"2024-09-13T15:00:39.330958+0000","last_clean":"2024-09-13T15:00:39.330958+0000","last_became_active":"2024-09-13T14:55:41.838820+0000","last_became_peered":"2024-09-13T14:55:41.838820+0000","last_unstale":"2024-09-13T15:00:39.330958+0000","last_undegraded":"2024-09-13T15:00:39.330958+0000","last_fullsized":"2024-09-13T15:00:39.330958+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:14:27.811437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"261'100","reported_seq":329,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.911103+0000","last_change":"2024-09-13T14:59:50.042460+0000","last_active":"2024-09-13T15:00:55.911103+0000","last_peered":"2024-09-13T15:00:55.911103+0000","last_clean":"2024-09-13T15:00:55.911103+0000","last_became_active":"2024-09-13T14:59:50.041046+0000","last_became_peered":"2024-09-13T14:59:50.041046+0000","last_unstale":"2024-09-13T15:00:55.911103+0000","last_undegraded":"2024-09-13T15:00:55.911103+0000","last_fullsized":"2024-09-13T15:00:55.911103+0000","mapping_epoch":1280,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1281,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:36.965775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:45:36.965775+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:45.435199+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"1292'11852","reported_seq":17395,"reported_epoch":1292,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T15:01:02.414720+0000","last_change":"2024-09-13T15:00:26.162634+0000","last_active":"2024-09-13T15:01:02.414720+0000","last_peered":"2024-09-13T15:01:02.414720+0000","last_clean":"2024-09-13T15:00:04.119435+0000","last_became_active":"2024-09-13T15:00:06.166229+0000","last_became_peered":"2024-09-13T15:00:06.166229+0000","last_unstale":"2024-09-13T15:01:02.414720+0000","last_undegraded":"2024-09-13T15:01:02.414720+0000","last_fullsized":"2024-09-13T15:01:02.414720+0000","mapping_epoch":1285,"log_start":"1279'7512","ondisk_log_start":"1279'7512","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":4340,"log_dups_size":0,"ondisk_log_size":4340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:43.312119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243924992,"num_objects":3722,"num_object_clones":0,"num_object_copies":11166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1991,"num_objects_unfound":0,"num_objects_dirty":3722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11852,"num_write_kb":498368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2745,"num_bytes_recovered":179896320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,0],"avail_no_missing":["6","0","5"],"object_location_counts":[{"shards":"0,5,6","objects":3722}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":153,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.909680+0000","last_change":"2024-09-13T14:54:05.596068+0000","last_active":"2024-09-13T15:00:55.909680+0000","last_peered":"2024-09-13T15:00:55.909680+0000","last_clean":"2024-09-13T15:00:55.909680+0000","last_became_active":"2024-09-13T14:54:05.595743+0000","last_became_peered":"2024-09-13T14:54:05.595743+0000","last_unstale":"2024-09-13T15:00:55.909680+0000","last_undegraded":"2024-09-13T15:00:55.909680+0000","last_fullsized":"2024-09-13T15:00:55.909680+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:15:39.660370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":604,"reported_epoch":1291,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.328454+0000","last_change":"2024-09-13T15:00:25.089570+0000","last_active":"2024-09-13T15:00:39.328454+0000","last_peered":"2024-09-13T15:00:39.328454+0000","last_clean":"2024-09-13T15:00:39.328454+0000","last_became_active":"2024-09-13T15:00:25.088373+0000","last_became_peered":"2024-09-13T15:00:25.088373+0000","last_unstale":"2024-09-13T15:00:39.328454+0000","last_undegraded":"2024-09-13T15:00:39.328454+0000","last_fullsized":"2024-09-13T15:00:39.328454+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:36:04.220868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":114,"reported_epoch":1291,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.330967+0000","last_change":"2024-09-13T14:55:41.889668+0000","last_active":"2024-09-13T15:00:39.330967+0000","last_peered":"2024-09-13T15:00:39.330967+0000","last_clean":"2024-09-13T15:00:39.330967+0000","last_became_active":"2024-09-13T14:55:41.889195+0000","last_became_peered":"2024-09-13T14:55:41.889195+0000","last_unstale":"2024-09-13T15:00:39.330967+0000","last_undegraded":"2024-09-13T15:00:39.330967+0000","last_fullsized":"2024-09-13T15:00:39.330967+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:52.765208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"261'100","reported_seq":85,"reported_epoch":1291,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.333028+0000","last_change":"2024-09-13T15:00:25.076863+0000","last_active":"2024-09-13T15:00:39.333028+0000","last_peered":"2024-09-13T15:00:39.333028+0000","last_clean":"2024-09-13T15:00:39.333028+0000","last_became_active":"2024-09-13T15:00:25.076542+0000","last_became_peered":"2024-09-13T15:00:25.076542+0000","last_unstale":"2024-09-13T15:00:39.333028+0000","last_undegraded":"2024-09-13T15:00:39.333028+0000","last_fullsized":"2024-09-13T15:00:39.333028+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:57.024823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:57.024823+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:58.411612+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"1292'11848","reported_seq":24963,"reported_epoch":1292,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T15:01:02.455455+0000","last_change":"2024-09-13T15:00:25.206459+0000","last_active":"2024-09-13T15:01:02.455455+0000","last_peered":"2024-09-13T15:01:02.455455+0000","last_clean":"2024-09-13T14:59:49.042471+0000","last_became_active":"2024-09-13T15:00:25.187898+0000","last_became_peered":"2024-09-13T15:00:25.187898+0000","last_unstale":"2024-09-13T15:01:02.455455+0000","last_undegraded":"2024-09-13T15:01:02.455455+0000","last_fullsized":"2024-09-13T15:01:02.455455+0000","mapping_epoch":1288,"log_start":"1279'7659","ondisk_log_start":"1279'7659","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":4189,"log_dups_size":0,"ondisk_log_size":4189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:55:10.086537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237764608,"num_objects":3628,"num_object_clones":0,"num_object_copies":10884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3628,"num_objects_unfound":0,"num_objects_dirty":3628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11848,"num_write_kb":495232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6525,"num_bytes_recovered":427622400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,7],"avail_no_missing":["5","2","4","7"],"object_location_counts":[{"shards":"2,5,7","objects":3628}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":174,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.904832+0000","last_change":"2024-09-13T15:00:05.043740+0000","last_active":"2024-09-13T15:00:55.904832+0000","last_peered":"2024-09-13T15:00:55.904832+0000","last_clean":"2024-09-13T15:00:55.904832+0000","last_became_active":"2024-09-13T15:00:05.043424+0000","last_became_peered":"2024-09-13T15:00:05.043424+0000","last_unstale":"2024-09-13T15:00:55.904832+0000","last_undegraded":"2024-09-13T15:00:55.904832+0000","last_fullsized":"2024-09-13T15:00:55.904832+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:38.139256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":153,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.918368+0000","last_change":"2024-09-13T14:54:05.614107+0000","last_active":"2024-09-13T15:00:55.918368+0000","last_peered":"2024-09-13T15:00:55.918368+0000","last_clean":"2024-09-13T15:00:55.918368+0000","last_became_active":"2024-09-13T14:54:05.613429+0000","last_became_peered":"2024-09-13T14:54:05.613429+0000","last_unstale":"2024-09-13T15:00:55.918368+0000","last_undegraded":"2024-09-13T15:00:55.918368+0000","last_fullsized":"2024-09-13T15:00:55.918368+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:51.312413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1998,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.909698+0000","last_change":"2024-09-13T15:00:05.042993+0000","last_active":"2024-09-13T15:00:55.909698+0000","last_peered":"2024-09-13T15:00:55.909698+0000","last_clean":"2024-09-13T15:00:55.909698+0000","last_became_active":"2024-09-13T15:00:05.041250+0000","last_became_peered":"2024-09-13T15:00:05.041250+0000","last_unstale":"2024-09-13T15:00:55.909698+0000","last_undegraded":"2024-09-13T15:00:55.909698+0000","last_fullsized":"2024-09-13T15:00:55.909698+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:44.484707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"261'100","reported_seq":357,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.903495+0000","last_change":"2024-09-13T15:00:43.767235+0000","last_active":"2024-09-13T15:00:55.903495+0000","last_peered":"2024-09-13T15:00:55.903495+0000","last_clean":"2024-09-13T15:00:55.903495+0000","last_became_active":"2024-09-13T15:00:25.090885+0000","last_became_peered":"2024-09-13T15:00:25.090885+0000","last_unstale":"2024-09-13T15:00:55.903495+0000","last_undegraded":"2024-09-13T15:00:55.903495+0000","last_fullsized":"2024-09-13T15:00:55.903495+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T15:00:43.767173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T15:00:43.767173+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:20.272396+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.8","version":"1292'12007","reported_seq":15191,"reported_epoch":1292,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T15:01:03.212589+0000","last_change":"2024-09-13T15:00:25.196504+0000","last_active":"2024-09-13T15:01:03.212589+0000","last_peered":"2024-09-13T15:01:03.212589+0000","last_clean":"2024-09-13T14:59:49.024499+0000","last_became_active":"2024-09-13T15:00:25.180702+0000","last_became_peered":"2024-09-13T15:00:25.180702+0000","last_unstale":"2024-09-13T15:01:03.212589+0000","last_undegraded":"2024-09-13T15:01:03.212589+0000","last_fullsized":"2024-09-13T15:01:03.212589+0000","mapping_epoch":1288,"log_start":"1278'7184","ondisk_log_start":"1278'7184","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":4823,"log_dups_size":0,"ondisk_log_size":4823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244645888,"num_objects":3733,"num_object_clones":0,"num_object_copies":11199,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6669,"num_objects_unfound":0,"num_objects_dirty":3733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12007,"num_write_kb":503680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1551,"num_bytes_recovered":101580800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,0,7],"avail_no_missing":["2","0","7"],"object_location_counts":[{"shards":"0,2,7","objects":3733}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1874,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.906348+0000","last_change":"2024-09-13T15:00:05.047198+0000","last_active":"2024-09-13T15:00:55.906348+0000","last_peered":"2024-09-13T15:00:55.906348+0000","last_clean":"2024-09-13T15:00:55.906348+0000","last_became_active":"2024-09-13T15:00:05.046822+0000","last_became_peered":"2024-09-13T15:00:05.046822+0000","last_unstale":"2024-09-13T15:00:55.906348+0000","last_undegraded":"2024-09-13T15:00:55.906348+0000","last_fullsized":"2024-09-13T15:00:55.906348+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:47:30.705616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1919,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.910399+0000","last_change":"2024-09-13T15:00:25.082949+0000","last_active":"2024-09-13T15:00:55.910399+0000","last_peered":"2024-09-13T15:00:55.910399+0000","last_clean":"2024-09-13T15:00:55.910399+0000","last_became_active":"2024-09-13T15:00:25.082279+0000","last_became_peered":"2024-09-13T15:00:25.082279+0000","last_unstale":"2024-09-13T15:00:55.910399+0000","last_undegraded":"2024-09-13T15:00:55.910399+0000","last_fullsized":"2024-09-13T15:00:55.910399+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:49.021898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1279,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.918448+0000","last_change":"2024-09-13T15:00:05.047271+0000","last_active":"2024-09-13T15:00:55.918448+0000","last_peered":"2024-09-13T15:00:55.918448+0000","last_clean":"2024-09-13T15:00:55.918448+0000","last_became_active":"2024-09-13T15:00:05.046628+0000","last_became_peered":"2024-09-13T15:00:05.046628+0000","last_unstale":"2024-09-13T15:00:55.918448+0000","last_undegraded":"2024-09-13T15:00:55.918448+0000","last_fullsized":"2024-09-13T15:00:55.918448+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:24:24.255492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"261'100","reported_seq":167,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.908049+0000","last_change":"2024-09-13T15:00:25.085109+0000","last_active":"2024-09-13T15:00:55.908049+0000","last_peered":"2024-09-13T15:00:55.908049+0000","last_clean":"2024-09-13T15:00:55.908049+0000","last_became_active":"2024-09-13T15:00:25.084170+0000","last_became_peered":"2024-09-13T15:00:25.084170+0000","last_unstale":"2024-09-13T15:00:55.908049+0000","last_undegraded":"2024-09-13T15:00:55.908049+0000","last_fullsized":"2024-09-13T15:00:55.908049+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.008264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.008264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:52:58.309554+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"1292'12006","reported_seq":18241,"reported_epoch":1292,"state":"active+remapped+backfilling","last_fresh":"2024-09-13T15:01:02.063539+0000","last_change":"2024-09-13T15:00:26.163583+0000","last_active":"2024-09-13T15:01:02.063539+0000","last_peered":"2024-09-13T15:01:02.063539+0000","last_clean":"2024-09-13T15:00:04.143247+0000","last_became_active":"2024-09-13T15:00:25.161961+0000","last_became_peered":"2024-09-13T15:00:25.161961+0000","last_unstale":"2024-09-13T15:01:02.063539+0000","last_undegraded":"2024-09-13T15:01:02.063539+0000","last_fullsized":"2024-09-13T15:01:02.063539+0000","mapping_epoch":1288,"log_start":"1279'7693","ondisk_log_start":"1279'7693","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":4313,"log_dups_size":0,"ondisk_log_size":4313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252182528,"num_objects":3848,"num_object_clones":0,"num_object_copies":11544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1419,"num_objects_unfound":0,"num_objects_dirty":3848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12006,"num_write_kb":507328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3081,"num_bytes_recovered":201916416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,0],"avail_no_missing":["1","0","6"],"object_location_counts":[{"shards":"0,1,6","objects":3848}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":82,"reported_epoch":1291,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.325262+0000","last_change":"2024-09-13T15:00:25.040768+0000","last_active":"2024-09-13T15:00:39.325262+0000","last_peered":"2024-09-13T15:00:39.325262+0000","last_clean":"2024-09-13T15:00:39.325262+0000","last_became_active":"2024-09-13T15:00:25.040487+0000","last_became_peered":"2024-09-13T15:00:25.040487+0000","last_unstale":"2024-09-13T15:00:39.325262+0000","last_undegraded":"2024-09-13T15:00:39.325262+0000","last_fullsized":"2024-09-13T15:00:39.325262+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:44.500104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":340,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.909991+0000","last_change":"2024-09-13T14:55:41.835887+0000","last_active":"2024-09-13T15:00:55.909991+0000","last_peered":"2024-09-13T15:00:55.909991+0000","last_clean":"2024-09-13T15:00:55.909991+0000","last_became_active":"2024-09-13T14:55:41.834413+0000","last_became_peered":"2024-09-13T14:55:41.834413+0000","last_unstale":"2024-09-13T15:00:55.909991+0000","last_undegraded":"2024-09-13T15:00:55.909991+0000","last_fullsized":"2024-09-13T15:00:55.909991+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:01.850578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":114,"reported_epoch":1291,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.330776+0000","last_change":"2024-09-13T14:59:50.059391+0000","last_active":"2024-09-13T15:00:39.330776+0000","last_peered":"2024-09-13T15:00:39.330776+0000","last_clean":"2024-09-13T15:00:39.330776+0000","last_became_active":"2024-09-13T14:59:50.038165+0000","last_became_peered":"2024-09-13T14:59:50.038165+0000","last_unstale":"2024-09-13T15:00:39.330776+0000","last_undegraded":"2024-09-13T15:00:39.330776+0000","last_fullsized":"2024-09-13T15:00:39.330776+0000","mapping_epoch":1280,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:19:07.523360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"261'100","reported_seq":377,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.918212+0000","last_change":"2024-09-13T15:00:44.119519+0000","last_active":"2024-09-13T15:00:55.918212+0000","last_peered":"2024-09-13T15:00:55.918212+0000","last_clean":"2024-09-13T15:00:55.918212+0000","last_became_active":"2024-09-13T15:00:05.039983+0000","last_became_peered":"2024-09-13T15:00:05.039983+0000","last_unstale":"2024-09-13T15:00:55.918212+0000","last_undegraded":"2024-09-13T15:00:55.918212+0000","last_fullsized":"2024-09-13T15:00:55.918212+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T15:00:44.119431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T15:00:44.119431+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:39.437129+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"1292'12012","reported_seq":12062,"reported_epoch":1292,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T15:01:02.333441+0000","last_change":"2024-09-13T15:00:26.139500+0000","last_active":"2024-09-13T15:01:02.333441+0000","last_peered":"2024-09-13T15:01:02.333441+0000","last_clean":"2024-09-13T15:00:04.129483+0000","last_became_active":"2024-09-13T15:00:26.114854+0000","last_became_peered":"2024-09-13T15:00:26.114854+0000","last_unstale":"2024-09-13T15:01:02.333441+0000","last_undegraded":"2024-09-13T15:01:02.333441+0000","last_fullsized":"2024-09-13T15:01:02.333441+0000","mapping_epoch":1289,"log_start":"1275'5171","ondisk_log_start":"1275'5171","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":6841,"log_dups_size":0,"ondisk_log_size":6841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:44:43.607762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244711424,"num_objects":3734,"num_object_clones":0,"num_object_copies":11202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7468,"num_objects_unfound":0,"num_objects_dirty":3734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12012,"num_write_kb":503872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,4,0],"avail_no_missing":["7","0","4"],"object_location_counts":[{"shards":"0,4,7","objects":3734}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":167,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.905112+0000","last_change":"2024-09-13T15:00:25.059272+0000","last_active":"2024-09-13T15:00:55.905112+0000","last_peered":"2024-09-13T15:00:55.905112+0000","last_clean":"2024-09-13T15:00:55.905112+0000","last_became_active":"2024-09-13T15:00:25.058910+0000","last_became_peered":"2024-09-13T15:00:25.058910+0000","last_unstale":"2024-09-13T15:00:55.905112+0000","last_undegraded":"2024-09-13T15:00:55.905112+0000","last_fullsized":"2024-09-13T15:00:55.905112+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:01:12.921208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":340,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.910465+0000","last_change":"2024-09-13T15:00:25.049881+0000","last_active":"2024-09-13T15:00:55.910465+0000","last_peered":"2024-09-13T15:00:55.910465+0000","last_clean":"2024-09-13T15:00:55.910465+0000","last_became_active":"2024-09-13T15:00:25.048110+0000","last_became_peered":"2024-09-13T15:00:25.048110+0000","last_unstale":"2024-09-13T15:00:55.910465+0000","last_undegraded":"2024-09-13T15:00:55.910465+0000","last_fullsized":"2024-09-13T15:00:55.910465+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:18:03.862509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"261'100","reported_seq":65,"reported_epoch":1291,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.330660+0000","last_change":"2024-09-13T14:55:41.784086+0000","last_active":"2024-09-13T15:00:39.330660+0000","last_peered":"2024-09-13T15:00:39.330660+0000","last_clean":"2024-09-13T15:00:39.330660+0000","last_became_active":"2024-09-13T14:55:41.783492+0000","last_became_peered":"2024-09-13T14:55:41.783492+0000","last_unstale":"2024-09-13T15:00:39.330660+0000","last_undegraded":"2024-09-13T15:00:39.330660+0000","last_fullsized":"2024-09-13T15:00:39.330660+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:49.280478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:49.280478+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:16:55.041094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"1292'11937","reported_seq":11984,"reported_epoch":1292,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T15:01:02.417472+0000","last_change":"2024-09-13T15:00:26.201712+0000","last_active":"2024-09-13T15:01:02.417472+0000","last_peered":"2024-09-13T15:01:02.417472+0000","last_clean":"2024-09-13T15:00:24.483417+0000","last_became_active":"2024-09-13T15:00:26.140552+0000","last_became_peered":"2024-09-13T15:00:26.140552+0000","last_unstale":"2024-09-13T15:01:02.417472+0000","last_undegraded":"2024-09-13T15:01:02.417472+0000","last_fullsized":"2024-09-13T15:01:02.417472+0000","mapping_epoch":1289,"log_start":"1279'7649","ondisk_log_start":"1279'7649","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":4288,"log_dups_size":0,"ondisk_log_size":4288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:05:34.210135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249757696,"num_objects":3811,"num_object_clones":0,"num_object_copies":11433,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3811,"num_objects_unfound":0,"num_objects_dirty":3811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11937,"num_write_kb":503936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,4],"avail_no_missing":["6","1","4"],"object_location_counts":[{"shards":"1,4,6","objects":3811}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":38,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.906594+0000","last_change":"2024-09-13T15:00:25.069428+0000","last_active":"2024-09-13T15:00:55.906594+0000","last_peered":"2024-09-13T15:00:55.906594+0000","last_clean":"2024-09-13T15:00:55.906594+0000","last_became_active":"2024-09-13T15:00:25.068745+0000","last_became_peered":"2024-09-13T15:00:25.068745+0000","last_unstale":"2024-09-13T15:00:55.906594+0000","last_undegraded":"2024-09-13T15:00:55.906594+0000","last_fullsized":"2024-09-13T15:00:55.906594+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:37.264986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":688,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.909438+0000","last_change":"2024-09-13T15:00:25.073732+0000","last_active":"2024-09-13T15:00:55.909438+0000","last_peered":"2024-09-13T15:00:55.909438+0000","last_clean":"2024-09-13T15:00:55.909438+0000","last_became_active":"2024-09-13T15:00:25.073424+0000","last_became_peered":"2024-09-13T15:00:25.073424+0000","last_unstale":"2024-09-13T15:00:55.909438+0000","last_undegraded":"2024-09-13T15:00:55.909438+0000","last_fullsized":"2024-09-13T15:00:55.909438+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:20:59.668905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"261'100","reported_seq":174,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.908132+0000","last_change":"2024-09-13T15:00:05.050225+0000","last_active":"2024-09-13T15:00:55.908132+0000","last_peered":"2024-09-13T15:00:55.908132+0000","last_clean":"2024-09-13T15:00:55.908132+0000","last_became_active":"2024-09-13T15:00:05.049780+0000","last_became_peered":"2024-09-13T15:00:05.049780+0000","last_unstale":"2024-09-13T15:00:55.908132+0000","last_undegraded":"2024-09-13T15:00:55.908132+0000","last_fullsized":"2024-09-13T15:00:55.908132+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:47.405845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:47.405845+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:56:03.644141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3","version":"1292'11898","reported_seq":11958,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:01:03.201487+0000","last_change":"2024-09-13T14:58:06.200017+0000","last_active":"2024-09-13T15:01:03.201487+0000","last_peered":"2024-09-13T15:01:03.201487+0000","last_clean":"2024-09-13T15:01:03.201487+0000","last_became_active":"2024-09-13T14:58:06.199665+0000","last_became_peered":"2024-09-13T14:58:06.199665+0000","last_unstale":"2024-09-13T15:01:03.201487+0000","last_undegraded":"2024-09-13T15:01:03.201487+0000","last_fullsized":"2024-09-13T15:01:03.201487+0000","mapping_epoch":1268,"log_start":"1278'7104","ondisk_log_start":"1278'7104","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":4794,"log_dups_size":0,"ondisk_log_size":4794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:09.742112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248512512,"num_objects":3792,"num_object_clones":0,"num_object_copies":11376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11898,"num_write_kb":502080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1247,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.903585+0000","last_change":"2024-09-13T15:00:25.058894+0000","last_active":"2024-09-13T15:00:55.903585+0000","last_peered":"2024-09-13T15:00:55.903585+0000","last_clean":"2024-09-13T15:00:55.903585+0000","last_became_active":"2024-09-13T15:00:25.058311+0000","last_became_peered":"2024-09-13T15:00:25.058311+0000","last_unstale":"2024-09-13T15:00:55.903585+0000","last_undegraded":"2024-09-13T15:00:55.903585+0000","last_fullsized":"2024-09-13T15:00:55.903585+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:04.030074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":38,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.912626+0000","last_change":"2024-09-13T15:00:25.154369+0000","last_active":"2024-09-13T15:00:55.912626+0000","last_peered":"2024-09-13T15:00:55.912626+0000","last_clean":"2024-09-13T15:00:55.912626+0000","last_became_active":"2024-09-13T15:00:25.154045+0000","last_became_peered":"2024-09-13T15:00:25.154045+0000","last_unstale":"2024-09-13T15:00:55.912626+0000","last_undegraded":"2024-09-13T15:00:55.912626+0000","last_fullsized":"2024-09-13T15:00:55.912626+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:58:48.342494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"261'100","reported_seq":65,"reported_epoch":1291,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.330362+0000","last_change":"2024-09-13T14:55:41.844203+0000","last_active":"2024-09-13T15:00:39.330362+0000","last_peered":"2024-09-13T15:00:39.330362+0000","last_clean":"2024-09-13T15:00:39.330362+0000","last_became_active":"2024-09-13T14:55:41.843897+0000","last_became_peered":"2024-09-13T14:55:41.843897+0000","last_unstale":"2024-09-13T15:00:39.330362+0000","last_undegraded":"2024-09-13T15:00:39.330362+0000","last_fullsized":"2024-09-13T15:00:39.330362+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:55.816857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:55.816857+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:42:54.571599+0000","scrub_duration":270,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2","version":"1292'12086","reported_seq":12162,"reported_epoch":1292,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T15:01:02.018509+0000","last_change":"2024-09-13T15:00:25.199463+0000","last_active":"2024-09-13T15:01:02.018509+0000","last_peered":"2024-09-13T15:01:02.018509+0000","last_clean":"2024-09-13T15:00:04.067491+0000","last_became_active":"2024-09-13T15:00:25.154467+0000","last_became_peered":"2024-09-13T15:00:25.154467+0000","last_unstale":"2024-09-13T15:01:02.018509+0000","last_undegraded":"2024-09-13T15:01:02.018509+0000","last_fullsized":"2024-09-13T15:01:02.018509+0000","mapping_epoch":1288,"log_start":"1279'7758","ondisk_log_start":"1279'7758","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":4328,"log_dups_size":0,"ondisk_log_size":4328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251002880,"num_objects":3830,"num_object_clones":0,"num_object_copies":11490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7660,"num_objects_unfound":0,"num_objects_dirty":3830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12086,"num_write_kb":509312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,4,0],"avail_no_missing":["1","0","4"],"object_location_counts":[{"shards":"0,1,4","objects":3830}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1274,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.912489+0000","last_change":"2024-09-13T15:00:25.045581+0000","last_active":"2024-09-13T15:00:55.912489+0000","last_peered":"2024-09-13T15:00:55.912489+0000","last_clean":"2024-09-13T15:00:55.912489+0000","last_became_active":"2024-09-13T15:00:25.045225+0000","last_became_peered":"2024-09-13T15:00:25.045225+0000","last_unstale":"2024-09-13T15:00:55.912489+0000","last_undegraded":"2024-09-13T15:00:55.912489+0000","last_fullsized":"2024-09-13T15:00:55.912489+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:38.148339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":374,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.911115+0000","last_change":"2024-09-13T15:00:25.164094+0000","last_active":"2024-09-13T15:00:55.911115+0000","last_peered":"2024-09-13T15:00:55.911115+0000","last_clean":"2024-09-13T15:00:55.911115+0000","last_became_active":"2024-09-13T15:00:25.085340+0000","last_became_peered":"2024-09-13T15:00:25.085340+0000","last_unstale":"2024-09-13T15:00:55.911115+0000","last_undegraded":"2024-09-13T15:00:55.911115+0000","last_fullsized":"2024-09-13T15:00:55.911115+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:10:15.121556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"261'100","reported_seq":1305,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.910913+0000","last_change":"2024-09-13T14:59:50.039831+0000","last_active":"2024-09-13T15:00:55.910913+0000","last_peered":"2024-09-13T15:00:55.910913+0000","last_clean":"2024-09-13T15:00:55.910913+0000","last_became_active":"2024-09-13T14:59:50.039483+0000","last_became_peered":"2024-09-13T14:59:50.039483+0000","last_unstale":"2024-09-13T15:00:55.910913+0000","last_undegraded":"2024-09-13T15:00:55.910913+0000","last_fullsized":"2024-09-13T15:00:55.910913+0000","mapping_epoch":1280,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1281,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:29.169325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:29.169325+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:48:22.508457+0000","scrub_duration":0.0019358890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.5","version":"1292'11981","reported_seq":13511,"reported_epoch":1292,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T15:01:03.249233+0000","last_change":"2024-09-13T15:00:26.201193+0000","last_active":"2024-09-13T15:01:03.249233+0000","last_peered":"2024-09-13T15:01:03.249233+0000","last_clean":"2024-09-13T15:00:04.143423+0000","last_became_active":"2024-09-13T15:00:06.195289+0000","last_became_peered":"2024-09-13T15:00:06.195289+0000","last_unstale":"2024-09-13T15:01:03.249233+0000","last_undegraded":"2024-09-13T15:01:03.249233+0000","last_fullsized":"2024-09-13T15:01:03.249233+0000","mapping_epoch":1285,"log_start":"1278'7200","ondisk_log_start":"1278'7200","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":4781,"log_dups_size":0,"ondisk_log_size":4781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239009792,"num_objects":3647,"num_object_clones":0,"num_object_copies":10941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3184,"num_objects_unfound":0,"num_objects_dirty":3647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11981,"num_write_kb":500096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":729,"num_bytes_recovered":47710208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,0],"avail_no_missing":["2","0","3"],"object_location_counts":[{"shards":"0,2,3","objects":3647}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":31,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.905017+0000","last_change":"2024-09-13T15:00:25.060205+0000","last_active":"2024-09-13T15:00:55.905017+0000","last_peered":"2024-09-13T15:00:55.905017+0000","last_clean":"2024-09-13T15:00:55.905017+0000","last_became_active":"2024-09-13T15:00:25.059901+0000","last_became_peered":"2024-09-13T15:00:25.059901+0000","last_unstale":"2024-09-13T15:00:55.905017+0000","last_undegraded":"2024-09-13T15:00:55.905017+0000","last_fullsized":"2024-09-13T15:00:55.905017+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:33.964617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":65,"reported_epoch":1291,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.330563+0000","last_change":"2024-09-13T14:55:41.848811+0000","last_active":"2024-09-13T15:00:39.330563+0000","last_peered":"2024-09-13T15:00:39.330563+0000","last_clean":"2024-09-13T15:00:39.330563+0000","last_became_active":"2024-09-13T14:55:41.848542+0000","last_became_peered":"2024-09-13T14:55:41.848542+0000","last_unstale":"2024-09-13T15:00:39.330563+0000","last_undegraded":"2024-09-13T15:00:39.330563+0000","last_fullsized":"2024-09-13T15:00:39.330563+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:48.534022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"261'100","reported_seq":363,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.910274+0000","last_change":"2024-09-13T15:00:41.463307+0000","last_active":"2024-09-13T15:00:55.910274+0000","last_peered":"2024-09-13T15:00:55.910274+0000","last_clean":"2024-09-13T15:00:55.910274+0000","last_became_active":"2024-09-13T15:00:25.066966+0000","last_became_peered":"2024-09-13T15:00:25.066966+0000","last_unstale":"2024-09-13T15:00:55.910274+0000","last_undegraded":"2024-09-13T15:00:55.910274+0000","last_fullsized":"2024-09-13T15:00:55.910274+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T15:00:41.463224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T15:00:41.463224+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:08.650561+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.4","version":"1292'12127","reported_seq":12202,"reported_epoch":1292,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T15:01:03.657128+0000","last_change":"2024-09-13T15:00:26.171103+0000","last_active":"2024-09-13T15:01:03.657128+0000","last_peered":"2024-09-13T15:01:03.657128+0000","last_clean":"2024-09-13T15:00:04.143229+0000","last_became_active":"2024-09-13T15:00:26.113653+0000","last_became_peered":"2024-09-13T15:00:26.113653+0000","last_unstale":"2024-09-13T15:01:03.657128+0000","last_undegraded":"2024-09-13T15:01:03.657128+0000","last_fullsized":"2024-09-13T15:01:03.657128+0000","mapping_epoch":1289,"log_start":"1282'8110","ondisk_log_start":"1282'8110","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243073024,"num_objects":3709,"num_object_clones":0,"num_object_copies":11127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7418,"num_objects_unfound":0,"num_objects_dirty":3709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12127,"num_write_kb":506752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,4,0],"avail_no_missing":["3","0","4"],"object_location_counts":[{"shards":"0,3,4","objects":3709}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1298,"reported_epoch":1291,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.328104+0000","last_change":"2024-09-13T14:59:50.040361+0000","last_active":"2024-09-13T15:00:39.328104+0000","last_peered":"2024-09-13T15:00:39.328104+0000","last_clean":"2024-09-13T15:00:39.328104+0000","last_became_active":"2024-09-13T14:59:50.040043+0000","last_became_peered":"2024-09-13T14:59:50.040043+0000","last_unstale":"2024-09-13T15:00:39.328104+0000","last_undegraded":"2024-09-13T15:00:39.328104+0000","last_fullsized":"2024-09-13T15:00:39.328104+0000","mapping_epoch":1280,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:22:10.227757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":1246,"reported_epoch":1291,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.330696+0000","last_change":"2024-09-13T14:59:50.054560+0000","last_active":"2024-09-13T15:00:39.330696+0000","last_peered":"2024-09-13T15:00:39.330696+0000","last_clean":"2024-09-13T15:00:39.330696+0000","last_became_active":"2024-09-13T14:59:50.042240+0000","last_became_peered":"2024-09-13T14:59:50.042240+0000","last_unstale":"2024-09-13T15:00:39.330696+0000","last_undegraded":"2024-09-13T15:00:39.330696+0000","last_fullsized":"2024-09-13T15:00:39.330696+0000","mapping_epoch":1280,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:21:58.373058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"261'100","reported_seq":75,"reported_epoch":1291,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.331346+0000","last_change":"2024-09-13T15:00:25.048330+0000","last_active":"2024-09-13T15:00:39.331346+0000","last_peered":"2024-09-13T15:00:39.331346+0000","last_clean":"2024-09-13T15:00:39.331346+0000","last_became_active":"2024-09-13T15:00:25.048001+0000","last_became_peered":"2024-09-13T15:00:25.048001+0000","last_unstale":"2024-09-13T15:00:39.331346+0000","last_undegraded":"2024-09-13T15:00:39.331346+0000","last_fullsized":"2024-09-13T15:00:39.331346+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:49:21.722939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.722939+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:11:39.620088+0000","scrub_duration":0.0016642790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"1292'11952","reported_seq":12012,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:01:03.650663+0000","last_change":"2024-09-13T14:58:06.199838+0000","last_active":"2024-09-13T15:01:03.650663+0000","last_peered":"2024-09-13T15:01:03.650663+0000","last_clean":"2024-09-13T15:01:03.650663+0000","last_became_active":"2024-09-13T14:58:06.199417+0000","last_became_peered":"2024-09-13T14:58:06.199417+0000","last_unstale":"2024-09-13T15:01:03.650663+0000","last_undegraded":"2024-09-13T15:01:03.650663+0000","last_fullsized":"2024-09-13T15:01:03.650663+0000","mapping_epoch":1268,"log_start":"1282'7921","ondisk_log_start":"1282'7921","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":4031,"log_dups_size":0,"ondisk_log_size":4031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:14:49.667560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239861760,"num_objects":3660,"num_object_clones":0,"num_object_copies":10980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11952,"num_write_kb":499584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":69,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.909210+0000","last_change":"2024-09-13T15:00:25.054416+0000","last_active":"2024-09-13T15:00:55.909210+0000","last_peered":"2024-09-13T15:00:55.909210+0000","last_clean":"2024-09-13T15:00:55.909210+0000","last_became_active":"2024-09-13T15:00:25.050641+0000","last_became_peered":"2024-09-13T15:00:25.050641+0000","last_unstale":"2024-09-13T15:00:55.909210+0000","last_undegraded":"2024-09-13T15:00:55.909210+0000","last_fullsized":"2024-09-13T15:00:55.909210+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:35:40.107806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":1273,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.906497+0000","last_change":"2024-09-13T15:00:25.103005+0000","last_active":"2024-09-13T15:00:55.906497+0000","last_peered":"2024-09-13T15:00:55.906497+0000","last_clean":"2024-09-13T15:00:55.906497+0000","last_became_active":"2024-09-13T15:00:25.082153+0000","last_became_peered":"2024-09-13T15:00:25.082153+0000","last_unstale":"2024-09-13T15:00:55.906497+0000","last_undegraded":"2024-09-13T15:00:55.906497+0000","last_fullsized":"2024-09-13T15:00:55.906497+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:36:12.948278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"261'100","reported_seq":333,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.912611+0000","last_change":"2024-09-13T15:00:25.213778+0000","last_active":"2024-09-13T15:00:55.912611+0000","last_peered":"2024-09-13T15:00:55.912611+0000","last_clean":"2024-09-13T15:00:55.912611+0000","last_became_active":"2024-09-13T15:00:25.047003+0000","last_became_peered":"2024-09-13T15:00:25.047003+0000","last_unstale":"2024-09-13T15:00:55.912611+0000","last_undegraded":"2024-09-13T15:00:55.912611+0000","last_fullsized":"2024-09-13T15:00:55.912611+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.597568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.597568+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:20.589087+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"1292'11931","reported_seq":17298,"reported_epoch":1292,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T15:01:02.456870+0000","last_change":"2024-09-13T15:00:26.200343+0000","last_active":"2024-09-13T15:01:02.456870+0000","last_peered":"2024-09-13T15:01:02.456870+0000","last_clean":"2024-09-13T15:00:04.131554+0000","last_became_active":"2024-09-13T15:00:25.156676+0000","last_became_peered":"2024-09-13T15:00:25.156676+0000","last_unstale":"2024-09-13T15:01:02.456870+0000","last_undegraded":"2024-09-13T15:01:02.456870+0000","last_fullsized":"2024-09-13T15:01:02.456870+0000","mapping_epoch":1288,"log_start":"1279'7806","ondisk_log_start":"1279'7806","created":1268,"last_epoch_clean":1269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:58:05.183074+0000","last_clean_scrub_stamp":"2024-09-13T14:58:05.183074+0000","objects_scrubbed":0,"log_size":4125,"log_dups_size":0,"ondisk_log_size":4125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:29:29.616744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244252672,"num_objects":3727,"num_object_clones":0,"num_object_copies":11181,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5884,"num_objects_unfound":0,"num_objects_dirty":3727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11931,"num_write_kb":501056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2649,"num_bytes_recovered":173473792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,4,0],"avail_no_missing":["5","0","4"],"object_location_counts":[{"shards":"0,4,5","objects":3727}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":85,"reported_epoch":1291,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.330531+0000","last_change":"2024-09-13T15:00:25.048401+0000","last_active":"2024-09-13T15:00:39.330531+0000","last_peered":"2024-09-13T15:00:39.330531+0000","last_clean":"2024-09-13T15:00:39.330531+0000","last_became_active":"2024-09-13T15:00:25.048089+0000","last_became_peered":"2024-09-13T15:00:25.048089+0000","last_unstale":"2024-09-13T15:00:39.330531+0000","last_undegraded":"2024-09-13T15:00:39.330531+0000","last_fullsized":"2024-09-13T15:00:39.330531+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:51.258059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":1268,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.904499+0000","last_change":"2024-09-13T15:00:25.063726+0000","last_active":"2024-09-13T15:00:55.904499+0000","last_peered":"2024-09-13T15:00:55.904499+0000","last_clean":"2024-09-13T15:00:55.904499+0000","last_became_active":"2024-09-13T15:00:25.050960+0000","last_became_peered":"2024-09-13T15:00:25.050960+0000","last_unstale":"2024-09-13T15:00:55.904499+0000","last_undegraded":"2024-09-13T15:00:55.904499+0000","last_fullsized":"2024-09-13T15:00:55.904499+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:44:29.236482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"261'100","reported_seq":154,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.904556+0000","last_change":"2024-09-13T15:00:25.066087+0000","last_active":"2024-09-13T15:00:55.904556+0000","last_peered":"2024-09-13T15:00:55.904556+0000","last_clean":"2024-09-13T15:00:55.904556+0000","last_became_active":"2024-09-13T15:00:25.065512+0000","last_became_peered":"2024-09-13T15:00:25.065512+0000","last_unstale":"2024-09-13T15:00:55.904556+0000","last_undegraded":"2024-09-13T15:00:55.904556+0000","last_fullsized":"2024-09-13T15:00:55.904556+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.191326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.191326+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:20:41.915583+0000","scrub_duration":0.0025317360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1337,"reported_epoch":1291,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.327986+0000","last_change":"2024-09-13T15:00:05.058713+0000","last_active":"2024-09-13T15:00:39.327986+0000","last_peered":"2024-09-13T15:00:39.327986+0000","last_clean":"2024-09-13T15:00:39.327986+0000","last_became_active":"2024-09-13T15:00:05.034868+0000","last_became_peered":"2024-09-13T15:00:05.034868+0000","last_unstale":"2024-09-13T15:00:39.327986+0000","last_undegraded":"2024-09-13T15:00:39.327986+0000","last_fullsized":"2024-09-13T15:00:39.327986+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:32:10.399295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":325,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.911024+0000","last_change":"2024-09-13T14:59:50.047806+0000","last_active":"2024-09-13T15:00:55.911024+0000","last_peered":"2024-09-13T15:00:55.911024+0000","last_clean":"2024-09-13T15:00:55.911024+0000","last_became_active":"2024-09-13T14:59:50.047409+0000","last_became_peered":"2024-09-13T14:59:50.047409+0000","last_unstale":"2024-09-13T15:00:55.911024+0000","last_undegraded":"2024-09-13T15:00:55.911024+0000","last_fullsized":"2024-09-13T15:00:55.911024+0000","mapping_epoch":1280,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:23.593917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"261'100","reported_seq":65,"reported_epoch":1291,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.330815+0000","last_change":"2024-09-13T14:55:41.889497+0000","last_active":"2024-09-13T15:00:39.330815+0000","last_peered":"2024-09-13T15:00:39.330815+0000","last_clean":"2024-09-13T15:00:39.330815+0000","last_became_active":"2024-09-13T14:55:41.888766+0000","last_became_peered":"2024-09-13T14:55:41.888766+0000","last_unstale":"2024-09-13T15:00:39.330815+0000","last_undegraded":"2024-09-13T15:00:39.330815+0000","last_fullsized":"2024-09-13T15:00:39.330815+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:55.378404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:55.378404+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:50.289550+0000","scrub_duration":0.0016008439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1214,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.910838+0000","last_change":"2024-09-13T15:00:25.056123+0000","last_active":"2024-09-13T15:00:55.910838+0000","last_peered":"2024-09-13T15:00:55.910838+0000","last_clean":"2024-09-13T15:00:55.910838+0000","last_became_active":"2024-09-13T15:00:25.055741+0000","last_became_peered":"2024-09-13T15:00:25.055741+0000","last_unstale":"2024-09-13T15:00:55.910838+0000","last_undegraded":"2024-09-13T15:00:55.910838+0000","last_fullsized":"2024-09-13T15:00:55.910838+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:11.638358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1973,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.910087+0000","last_change":"2024-09-13T15:00:05.046458+0000","last_active":"2024-09-13T15:00:55.910087+0000","last_peered":"2024-09-13T15:00:55.910087+0000","last_clean":"2024-09-13T15:00:55.910087+0000","last_became_active":"2024-09-13T15:00:05.045762+0000","last_became_peered":"2024-09-13T15:00:05.045762+0000","last_unstale":"2024-09-13T15:00:55.910087+0000","last_undegraded":"2024-09-13T15:00:55.910087+0000","last_fullsized":"2024-09-13T15:00:55.910087+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:04:21.163809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"261'100","reported_seq":166,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.907964+0000","last_change":"2024-09-13T15:00:25.087759+0000","last_active":"2024-09-13T15:00:55.907964+0000","last_peered":"2024-09-13T15:00:55.907964+0000","last_clean":"2024-09-13T15:00:55.907964+0000","last_became_active":"2024-09-13T15:00:25.087315+0000","last_became_peered":"2024-09-13T15:00:25.087315+0000","last_unstale":"2024-09-13T15:00:55.907964+0000","last_undegraded":"2024-09-13T15:00:55.907964+0000","last_fullsized":"2024-09-13T15:00:55.907964+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:19.978464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:19.978464+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:22:19.789250+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":2021,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.910186+0000","last_change":"2024-09-13T15:00:05.045827+0000","last_active":"2024-09-13T15:00:55.910186+0000","last_peered":"2024-09-13T15:00:55.910186+0000","last_clean":"2024-09-13T15:00:55.910186+0000","last_became_active":"2024-09-13T15:00:05.045492+0000","last_became_peered":"2024-09-13T15:00:05.045492+0000","last_unstale":"2024-09-13T15:00:55.910186+0000","last_undegraded":"2024-09-13T15:00:55.910186+0000","last_fullsized":"2024-09-13T15:00:55.910186+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:58.462057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":75,"reported_epoch":1291,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.330999+0000","last_change":"2024-09-13T14:59:50.042678+0000","last_active":"2024-09-13T15:00:39.330999+0000","last_peered":"2024-09-13T15:00:39.330999+0000","last_clean":"2024-09-13T15:00:39.330999+0000","last_became_active":"2024-09-13T14:59:50.041166+0000","last_became_peered":"2024-09-13T14:59:50.041166+0000","last_unstale":"2024-09-13T15:00:39.330999+0000","last_undegraded":"2024-09-13T15:00:39.330999+0000","last_fullsized":"2024-09-13T15:00:39.330999+0000","mapping_epoch":1280,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:39:29.939316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"261'100","reported_seq":85,"reported_epoch":1291,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.330455+0000","last_change":"2024-09-13T15:00:25.070247+0000","last_active":"2024-09-13T15:00:39.330455+0000","last_peered":"2024-09-13T15:00:39.330455+0000","last_clean":"2024-09-13T15:00:39.330455+0000","last_became_active":"2024-09-13T15:00:25.069922+0000","last_became_peered":"2024-09-13T15:00:25.069922+0000","last_unstale":"2024-09-13T15:00:39.330455+0000","last_undegraded":"2024-09-13T15:00:39.330455+0000","last_fullsized":"2024-09-13T15:00:39.330455+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:50.327393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:50.327393+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:39:58.386019+0000","scrub_duration":0.0011508309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":36,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.905318+0000","last_change":"2024-09-13T15:00:25.197990+0000","last_active":"2024-09-13T15:00:55.905318+0000","last_peered":"2024-09-13T15:00:55.905318+0000","last_clean":"2024-09-13T15:00:55.905318+0000","last_became_active":"2024-09-13T15:00:25.089353+0000","last_became_peered":"2024-09-13T15:00:25.089353+0000","last_unstale":"2024-09-13T15:00:55.905318+0000","last_undegraded":"2024-09-13T15:00:55.905318+0000","last_fullsized":"2024-09-13T15:00:55.905318+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:30.870315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1932,"reported_epoch":1292,"state":"active+clean","last_fresh":"2024-09-13T15:00:55.912845+0000","last_change":"2024-09-13T14:59:50.046214+0000","last_active":"2024-09-13T15:00:55.912845+0000","last_peered":"2024-09-13T15:00:55.912845+0000","last_clean":"2024-09-13T15:00:55.912845+0000","last_became_active":"2024-09-13T14:59:50.044194+0000","last_became_peered":"2024-09-13T14:59:50.044194+0000","last_unstale":"2024-09-13T15:00:55.912845+0000","last_undegraded":"2024-09-13T15:00:55.912845+0000","last_fullsized":"2024-09-13T15:00:55.912845+0000","mapping_epoch":1280,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:50.618030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"261'100","reported_seq":613,"reported_epoch":1291,"state":"active+clean","last_fresh":"2024-09-13T15:00:39.333012+0000","last_change":"2024-09-13T15:00:05.038303+0000","last_active":"2024-09-13T15:00:39.333012+0000","last_peered":"2024-09-13T15:00:39.333012+0000","last_clean":"2024-09-13T15:00:39.333012+0000","last_became_active":"2024-09-13T15:00:05.037979+0000","last_became_peered":"2024-09-13T15:00:05.037979+0000","last_unstale":"2024-09-13T15:00:39.333012+0000","last_undegraded":"2024-09-13T15:00:39.333012+0000","last_fullsized":"2024-09-13T15:00:39.333012+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:14.654603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:14.654603+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:28.168890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":3924754456,"num_objects":59888,"num_object_clones":0,"num_object_copies":179664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":64726,"num_objects_unfound":0,"num_objects_dirty":59888,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":191334,"num_write_kb":8039041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23038,"num_bytes_recovered":1509294104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13181997056,"data_stored":13181976696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77081,"ondisk_log_size":77081,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":12462,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":22162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2437120,"data_stored":2426464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430732,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2048860,"kb_used_data":1162260,"kb_used_omap":1,"kb_used_meta":886590,"kb_avail":91696292,"statfs":{"total":95995035648,"available":93897003008,"internally_reserved":0,"allocated":1190154240,"data_stored":1187408691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":907868504},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626961,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2632508,"kb_used_data":1733876,"kb_used_omap":1,"kb_used_meta":898558,"kb_avail":91112644,"statfs":{"total":95995035648,"available":93299347456,"internally_reserved":0,"allocated":1775489024,"data_stored":1772716234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":920123736},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823190,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2184824,"kb_used_data":1297776,"kb_used_omap":1,"kb_used_meta":887038,"kb_avail":91560328,"statfs":{"total":95995035648,"available":93757775872,"internally_reserved":0,"allocated":1328922624,"data_stored":1326154348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":908327256},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019418,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059593076,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2193192,"kb_used_data":1430944,"kb_used_omap":26,"kb_used_meta":762213,"kb_avail":91551960,"statfs":{"total":95995035648,"available":93749207040,"internally_reserved":0,"allocated":1465286656,"data_stored":1462529013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":780506272},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":591,"seq":2538325672314,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2570996,"kb_used_data":1705324,"kb_used_omap":41,"kb_used_meta":865494,"kb_avail":91174156,"statfs":{"total":95995035648,"available":93362335744,"internally_reserved":0,"allocated":1746251776,"data_stored":1743487621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42651,"internal_metadata":886266213},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":557,"seq":2392296784257,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1670120,"kb_used_data":840544,"kb_used_omap":32,"kb_used_meta":829535,"kb_avail":92075032,"statfs":{"total":95995035648,"available":94284832768,"internally_reserved":0,"allocated":860717056,"data_stored":857959826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33752,"internal_metadata":849443880},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":2250562863495,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2584547328,"data_stored":2584543256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":842530816,"data_stored":842530816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1726283776,"data_stored":1726283776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1455886336,"data_stored":1455882264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2331840512,"data_stored":2331836440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1304297472,"data_stored":1304297472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1762332672,"data_stored":1762328600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1174278144,"data_stored":1174274072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:01:08.491 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_5 pgp_num 16 2024-09-13T15:01:08.492 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:08.533 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:08.542 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:08.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:08.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:08.644 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:08.695 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:08.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:08.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:08.798 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:08.832 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:08.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:08.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:08.934 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:08.970 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:08.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:08.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:09.030 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:09.071 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:09.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:09.100 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:09.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:09.202 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:09.240 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:09.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:09.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:09.321 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:09 smithi151 ceph-mon[104450]: pgmap v2149: 119 pgs: 2 active+remapped+backfilling, 11 active+remapped+backfill_wait, 106 active+clean; 3.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 952 op/s; 60158/171765 objects misplaced (35.023%); 15 MiB/s, 235 objects/s recovering 2024-09-13T15:01:09.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:09 smithi151 ceph-mon[108966]: pgmap v2149: 119 pgs: 2 active+remapped+backfilling, 11 active+remapped+backfill_wait, 106 active+clean; 3.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 952 op/s; 60158/171765 objects misplaced (35.023%); 15 MiB/s, 235 objects/s recovering 2024-09-13T15:01:09.342 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:09.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:09 smithi184 ceph-mon[92527]: pgmap v2149: 119 pgs: 2 active+remapped+backfilling, 11 active+remapped+backfill_wait, 106 active+clean; 3.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 952 op/s; 60158/171765 objects misplaced (35.023%); 15 MiB/s, 235 objects/s recovering 2024-09-13T15:01:09.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:09.371 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:09.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:09.472 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:09.508 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:09.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:09.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:09.609 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:09.642 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:09.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:09.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:09.744 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:09.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:09.790 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:09.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:09.891 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:09.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:09.924 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:09.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:10.025 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:10.070 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:10.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:10.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:10.172 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:10.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:10.206 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:10.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:10.307 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:10.340 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:10.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:10.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:10.442 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:10.466 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:10.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:10.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:10.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:10.588 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:10.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:10.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:10.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:01:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:01:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:01:10.691 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:10.716 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:10.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:10.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:10.818 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:10.842 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:10.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:10.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:10.944 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:10.969 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:10.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:10.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:11.071 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:11.092 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:11.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:11.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:11.194 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:11.240 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:11.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:11.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:11.343 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:11.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:11 smithi184 ceph-mon[92527]: pgmap v2150: 119 pgs: 2 active+remapped+backfilling, 11 active+remapped+backfill_wait, 106 active+clean; 3.4 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 792 op/s; 59048/168111 objects misplaced (35.124%); 9.9 MiB/s, 157 objects/s recovering 2024-09-13T15:01:11.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:11 smithi184 ceph-mon[92527]: osdmap e1293: 8 total, 8 up, 6 in 2024-09-13T15:01:11.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:11 smithi151 ceph-mon[104450]: pgmap v2150: 119 pgs: 2 active+remapped+backfilling, 11 active+remapped+backfill_wait, 106 active+clean; 3.4 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 792 op/s; 59048/168111 objects misplaced (35.124%); 9.9 MiB/s, 157 objects/s recovering 2024-09-13T15:01:11.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:11 smithi151 ceph-mon[104450]: osdmap e1293: 8 total, 8 up, 6 in 2024-09-13T15:01:11.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:11 smithi151 ceph-mon[108966]: pgmap v2150: 119 pgs: 2 active+remapped+backfilling, 11 active+remapped+backfill_wait, 106 active+clean; 3.4 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 792 op/s; 59048/168111 objects misplaced (35.124%); 9.9 MiB/s, 157 objects/s recovering 2024-09-13T15:01:11.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:11 smithi151 ceph-mon[108966]: osdmap e1293: 8 total, 8 up, 6 in 2024-09-13T15:01:11.371 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:11.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:11.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:11.473 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:11.498 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:11.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:11.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:11.600 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:11.637 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:11.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:11.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:11.738 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:11.771 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:11.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:11.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:11.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:11.896 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:11.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:11.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:11.998 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:12.034 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:12.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:12.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:12.136 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:12.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:12.169 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:12.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:12.270 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:12.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:12.307 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:12.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:12.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:12 smithi184 ceph-mon[92527]: osdmap e1294: 8 total, 8 up, 6 in 2024-09-13T15:01:12.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:12 smithi151 ceph-mon[104450]: osdmap e1294: 8 total, 8 up, 6 in 2024-09-13T15:01:12.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:12 smithi151 ceph-mon[108966]: osdmap e1294: 8 total, 8 up, 6 in 2024-09-13T15:01:12.408 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:12.445 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:12.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:12.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:12.547 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:12.585 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:12.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:12.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:12.687 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:12.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:12.713 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:12.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:12.814 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:12.836 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:12.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:12.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:12.938 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:12.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:12.986 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:12.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:13.087 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:13.117 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:13.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:13.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:13.219 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:13.251 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:13.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:13.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:13.353 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:13.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:13 smithi184 ceph-mon[92527]: pgmap v2153: 119 pgs: 2 active+remapped+backfilling, 11 active+remapped+backfill_wait, 106 active+clean; 3.4 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 484 op/s; 59048/168111 objects misplaced (35.124%); 8.1 MiB/s, 128 objects/s recovering 2024-09-13T15:01:13.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:13 smithi151 ceph-mon[104450]: pgmap v2153: 119 pgs: 2 active+remapped+backfilling, 11 active+remapped+backfill_wait, 106 active+clean; 3.4 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 484 op/s; 59048/168111 objects misplaced (35.124%); 8.1 MiB/s, 128 objects/s recovering 2024-09-13T15:01:13.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:13 smithi151 ceph-mon[108966]: pgmap v2153: 119 pgs: 2 active+remapped+backfilling, 11 active+remapped+backfill_wait, 106 active+clean; 3.4 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 484 op/s; 59048/168111 objects misplaced (35.124%); 8.1 MiB/s, 128 objects/s recovering 2024-09-13T15:01:13.378 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:13.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:13.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:13.479 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:13.507 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:13.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:13.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:13.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:13.633 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:13.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:13.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:13.734 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:13.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:13.759 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:13.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:13.860 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:13.882 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:13.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:13.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:13.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:14.007 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:14.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:14.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:14.109 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:14.136 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:14.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:14.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:14.238 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:14.266 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:14.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:14.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:14.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:14 smithi184 ceph-mon[92527]: osdmap e1295: 8 total, 8 up, 6 in 2024-09-13T15:01:14.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:14 smithi151 ceph-mon[104450]: osdmap e1295: 8 total, 8 up, 6 in 2024-09-13T15:01:14.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:14 smithi151 ceph-mon[108966]: osdmap e1295: 8 total, 8 up, 6 in 2024-09-13T15:01:14.368 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:14.412 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:14.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:14.416 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:14.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:14.451 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T15:01:14.451 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:01:14.500 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T15:01:14.500 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:01:14.517 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:14.539 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T15:01:14.539 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:01:14.559 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:14.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:14.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:14.575 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T15:01:14.575 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:01:14.606 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T15:01:14.606 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:01:14.646 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T15:01:14.646 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:01:14.651 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:01:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:01:14] "GET /metrics HTTP/1.1" 200 36739 "" "Prometheus/2.51.0" 2024-09-13T15:01:14.662 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:14.693 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T15:01:14.694 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:01:14.721 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:14.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:14.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:14.731 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T15:01:14.731 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:01:14.824 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:14.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:14.863 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:14.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:14.964 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:15.001 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:15.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:15.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:15.105 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:15.143 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:15.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:15.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:15.245 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:15.278 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:15.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:15.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:15.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:15 smithi184 ceph-mon[92527]: pgmap v2155: 119 pgs: 3 active+remapped+backfilling, 9 active+remapped+backfill_wait, 107 active+clean; 3.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 800 op/s; 53487/157359 objects misplaced (33.990%); 17 MiB/s, 277 objects/s recovering 2024-09-13T15:01:15.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:15 smithi184 ceph-mon[92527]: osdmap e1296: 8 total, 8 up, 6 in 2024-09-13T15:01:15.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:15 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3504342367' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:15.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:15 smithi151 ceph-mon[104450]: pgmap v2155: 119 pgs: 3 active+remapped+backfilling, 9 active+remapped+backfill_wait, 107 active+clean; 3.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 800 op/s; 53487/157359 objects misplaced (33.990%); 17 MiB/s, 277 objects/s recovering 2024-09-13T15:01:15.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:15 smithi151 ceph-mon[104450]: osdmap e1296: 8 total, 8 up, 6 in 2024-09-13T15:01:15.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:15 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3504342367' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:15.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:15 smithi151 ceph-mon[108966]: pgmap v2155: 119 pgs: 3 active+remapped+backfilling, 9 active+remapped+backfill_wait, 107 active+clean; 3.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 800 op/s; 53487/157359 objects misplaced (33.990%); 17 MiB/s, 277 objects/s recovering 2024-09-13T15:01:15.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:15 smithi151 ceph-mon[108966]: osdmap e1296: 8 total, 8 up, 6 in 2024-09-13T15:01:15.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:15 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3504342367' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:15.380 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:15.410 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:15.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:15.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:15.512 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:15.539 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:15.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:15.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:15.641 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:15.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:15.673 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:15.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:15.774 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:15.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:15.810 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:15.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:15.911 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:15.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:15.953 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:15.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:16.054 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:16.091 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:16.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:16.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:16.193 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:16.244 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:16.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:16.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:16.347 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:16.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:16 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/830101474' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:01:16.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:16 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/830101474' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:01:16.361 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:16 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/830101474' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:01:16.378 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:16.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:16.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:16.480 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:16.515 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:16.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:16.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:16.617 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:16.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:16.651 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:16.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:16.751 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:16.788 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:16.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:16.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:16.890 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:16.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:16.929 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:16.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:17.030 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:17.067 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:17.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:17.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:17.168 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:17.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:17.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:17.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:17.312 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:17.353 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:17.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:17 smithi184 ceph-mon[92527]: pgmap v2157: 119 pgs: 3 active+remapped+backfilling, 9 active+remapped+backfill_wait, 107 active+clean; 3.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 621 op/s; 53487/157359 objects misplaced (33.990%); 18 MiB/s, 288 objects/s recovering 2024-09-13T15:01:17.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:17 smithi184 ceph-mon[92527]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:01:17.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:17 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/830101474' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:01:17.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:17 smithi184 ceph-mon[92527]: osdmap e1297: 8 total, 8 up, 6 in 2024-09-13T15:01:17.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:01:17.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:17.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:17.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:17.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:17 smithi151 ceph-mon[104450]: pgmap v2157: 119 pgs: 3 active+remapped+backfilling, 9 active+remapped+backfill_wait, 107 active+clean; 3.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 621 op/s; 53487/157359 objects misplaced (33.990%); 18 MiB/s, 288 objects/s recovering 2024-09-13T15:01:17.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:17 smithi151 ceph-mon[104450]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:01:17.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:17 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/830101474' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:01:17.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:17 smithi151 ceph-mon[104450]: osdmap e1297: 8 total, 8 up, 6 in 2024-09-13T15:01:17.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:01:17.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:17.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:17 smithi151 ceph-mon[108966]: pgmap v2157: 119 pgs: 3 active+remapped+backfilling, 9 active+remapped+backfill_wait, 107 active+clean; 3.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 621 op/s; 53487/157359 objects misplaced (33.990%); 18 MiB/s, 288 objects/s recovering 2024-09-13T15:01:17.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:17 smithi151 ceph-mon[108966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:01:17.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:17 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/830101474' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:01:17.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:17 smithi151 ceph-mon[108966]: osdmap e1297: 8 total, 8 up, 6 in 2024-09-13T15:01:17.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:01:17.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:17.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:17.496 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:17.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:17.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:17.598 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:17.633 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:17.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:17.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:17.735 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:17.767 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:17.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:17.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:17.869 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:17.901 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:17.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:17.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:17.936 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-13T15:01:18.002 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:18.036 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:18.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:18.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:18.139 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:18.171 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:18.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:18.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:18.273 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:18.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:18.317 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:18.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:18.418 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:18.459 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:18.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:18.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:18.561 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:18.598 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:18.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:18.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:18.672 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:01:18.700 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:18.730 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:18.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:18.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:18.832 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:18.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:18.864 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:18.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:18.965 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:18.991 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:18.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:18.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:19.093 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:19.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:19.134 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:19.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:19.235 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:19.259 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:19 smithi151 ceph-mon[108966]: pgmap v2159: 119 pgs: 2 active+remapped+backfilling, 9 active+remapped+backfill_wait, 108 active+clean; 3.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 1.00k op/s; 48594/150006 objects misplaced (32.395%); 39 MiB/s, 626 objects/s recovering 2024-09-13T15:01:19.259 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:19 smithi151 ceph-mon[104450]: pgmap v2159: 119 pgs: 2 active+remapped+backfilling, 9 active+remapped+backfill_wait, 108 active+clean; 3.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 1.00k op/s; 48594/150006 objects misplaced (32.395%); 39 MiB/s, 626 objects/s recovering 2024-09-13T15:01:19.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:19.265 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:19.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:19.354 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T15:01:19.355 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:01:19.366 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:19.414 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:19.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:19.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:19.515 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:19.550 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:19.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:19.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:19.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:19 smithi184 ceph-mon[92527]: pgmap v2159: 119 pgs: 2 active+remapped+backfilling, 9 active+remapped+backfill_wait, 108 active+clean; 3.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 1.00k op/s; 48594/150006 objects misplaced (32.395%); 39 MiB/s, 626 objects/s recovering 2024-09-13T15:01:19.651 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:19.681 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:19.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:19.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:19.783 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:19.815 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:19.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:19.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:19.917 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:19.961 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:19.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:19.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:20.063 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:20.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:20.092 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:20.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:20.193 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:20.224 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:20.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:20.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:20.326 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:20.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:20.358 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:20.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:20.459 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:20.498 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:20.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:20.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:20.600 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:20.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:01:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:01:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:01:20.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:20.636 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:20.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:20.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:20.777 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:20.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:20.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:20.879 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:20.909 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:20.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:20.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:20.981 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T15:01:21.011 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:21.035 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:21.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:21.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:21.137 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:21.163 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:21.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:21.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:21.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:21.290 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:21.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:21.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:21.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:21 smithi151 ceph-mon[104450]: pgmap v2160: 119 pgs: 2 active+remapped+backfilling, 9 active+remapped+backfill_wait, 108 active+clean; 3.0 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 789 op/s; 47442/145431 objects misplaced (32.622%); 21 MiB/s, 341 objects/s recovering 2024-09-13T15:01:21.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:21 smithi151 ceph-mon[108966]: pgmap v2160: 119 pgs: 2 active+remapped+backfilling, 9 active+remapped+backfill_wait, 108 active+clean; 3.0 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 789 op/s; 47442/145431 objects misplaced (32.622%); 21 MiB/s, 341 objects/s recovering 2024-09-13T15:01:21.391 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:21.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:21.431 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:21.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:21.532 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:21.550 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T15:01:21.560 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:21.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:21.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:21.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:21 smithi184 ceph-mon[92527]: pgmap v2160: 119 pgs: 2 active+remapped+backfilling, 9 active+remapped+backfill_wait, 108 active+clean; 3.0 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 789 op/s; 47442/145431 objects misplaced (32.622%); 21 MiB/s, 341 objects/s recovering 2024-09-13T15:01:21.662 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:21.701 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:21.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:21.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:21.803 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:21.857 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:21.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:21.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:21.959 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:21.997 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:21.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:21.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:22.099 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:22.126 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:22.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:22.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:22.420 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:22.442 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:22.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:22.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:22.544 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:22.569 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:22.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:22.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:22.671 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:22.696 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:22.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:22.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:22.798 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:22.823 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:22.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:22.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:22.925 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:22.946 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:22.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:22.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:23.048 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:23.072 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:23.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:23.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:23.174 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:23.218 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:23.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:23.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:23.320 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:23.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:23.355 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:23.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:23.457 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:23.499 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:23.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:23.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:23.601 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:23.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:23 smithi184 ceph-mon[92527]: pgmap v2161: 119 pgs: 2 active+remapped+backfilling, 9 active+remapped+backfill_wait, 108 active+clean; 3.0 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 496 op/s; 47442/145431 objects misplaced (32.622%); 16 MiB/s, 261 objects/s recovering 2024-09-13T15:01:23.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:23 smithi184 ceph-mon[92527]: osdmap e1298: 8 total, 8 up, 6 in 2024-09-13T15:01:23.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:23 smithi151 ceph-mon[108966]: pgmap v2161: 119 pgs: 2 active+remapped+backfilling, 9 active+remapped+backfill_wait, 108 active+clean; 3.0 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 496 op/s; 47442/145431 objects misplaced (32.622%); 16 MiB/s, 261 objects/s recovering 2024-09-13T15:01:23.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:23 smithi151 ceph-mon[108966]: osdmap e1298: 8 total, 8 up, 6 in 2024-09-13T15:01:23.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:23 smithi151 ceph-mon[104450]: pgmap v2161: 119 pgs: 2 active+remapped+backfilling, 9 active+remapped+backfill_wait, 108 active+clean; 3.0 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 496 op/s; 47442/145431 objects misplaced (32.622%); 16 MiB/s, 261 objects/s recovering 2024-09-13T15:01:23.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:23 smithi151 ceph-mon[104450]: osdmap e1298: 8 total, 8 up, 6 in 2024-09-13T15:01:23.633 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:23.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:23.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:23.735 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:23.775 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:23.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:23.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:23.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:23.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:23.942 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:23.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:24.009 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:24.044 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:24.047 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:24.049 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:24.071 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:24.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:24.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:24.134 INFO:teuthology.orchestra.run.smithi151.stderr:set pool 10 pgp_num to 16 2024-09-13T15:01:24.173 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:24.206 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:24.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:24.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:24.308 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:24.335 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:24.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:24.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:24.437 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:24.462 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:24.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:24.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:24.564 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:24.588 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:24.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:24.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:24.591 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:24 smithi151 ceph-mon[104450]: osdmap e1299: 8 total, 8 up, 6 in 2024-09-13T15:01:24.591 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:24 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/4250000283' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T15:01:24.591 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:24 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T15:01:24.591 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:24 smithi151 ceph-mon[108966]: osdmap e1299: 8 total, 8 up, 6 in 2024-09-13T15:01:24.591 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:24 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/4250000283' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T15:01:24.591 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:24 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T15:01:24.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:24 smithi184 ceph-mon[92527]: osdmap e1299: 8 total, 8 up, 6 in 2024-09-13T15:01:24.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:24 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/4250000283' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T15:01:24.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:24 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T15:01:24.690 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:24.716 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:24.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:24.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:24.817 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:24.842 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:24.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:24.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:24.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:01:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:01:24] "GET /metrics HTTP/1.1" 200 36729 "" "Prometheus/2.51.0" 2024-09-13T15:01:24.943 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:24.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:24.968 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:24.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:25.069 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:25.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:25.094 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:25.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:25.195 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:25.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:25.235 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:25.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:25.336 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:25.340 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:25 smithi151 ceph-mon[104450]: pgmap v2164: 119 pgs: 1 peering, 3 active+remapped+backfilling, 7 active+remapped+backfill_wait, 108 active+clean; 2.7 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 1.04k op/s; 41987/133392 objects misplaced (31.476%); 30 MiB/s, 474 objects/s recovering 2024-09-13T15:01:25.340 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:25 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]': finished 2024-09-13T15:01:25.340 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:25 smithi151 ceph-mon[104450]: osdmap e1300: 8 total, 8 up, 6 in 2024-09-13T15:01:25.340 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:25 smithi151 ceph-mon[108966]: pgmap v2164: 119 pgs: 1 peering, 3 active+remapped+backfilling, 7 active+remapped+backfill_wait, 108 active+clean; 2.7 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 1.04k op/s; 41987/133392 objects misplaced (31.476%); 30 MiB/s, 474 objects/s recovering 2024-09-13T15:01:25.340 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:25 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]': finished 2024-09-13T15:01:25.340 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:25 smithi151 ceph-mon[108966]: osdmap e1300: 8 total, 8 up, 6 in 2024-09-13T15:01:25.375 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:25.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:25.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:25.477 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:25.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:25.512 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:25.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:25.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:25 smithi184 ceph-mon[92527]: pgmap v2164: 119 pgs: 1 peering, 3 active+remapped+backfilling, 7 active+remapped+backfill_wait, 108 active+clean; 2.7 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 1.04k op/s; 41987/133392 objects misplaced (31.476%); 30 MiB/s, 474 objects/s recovering 2024-09-13T15:01:25.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:25 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]': finished 2024-09-13T15:01:25.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:25 smithi184 ceph-mon[92527]: osdmap e1300: 8 total, 8 up, 6 in 2024-09-13T15:01:25.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:25.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:25.653 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:25.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:25.754 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:25.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:25.790 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:25.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:25.891 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:25.932 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:25.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:25.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:26.035 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:26.068 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:26.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:26.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:26.170 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:26.218 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:26.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:26.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:26.320 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:26.356 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:26.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:26.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:26.458 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:26.496 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:26.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:26.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:26.598 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:26.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:26 smithi184 ceph-mon[92527]: osdmap e1301: 8 total, 8 up, 6 in 2024-09-13T15:01:26.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:26 smithi151 ceph-mon[104450]: osdmap e1301: 8 total, 8 up, 6 in 2024-09-13T15:01:26.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:26 smithi151 ceph-mon[108966]: osdmap e1301: 8 total, 8 up, 6 in 2024-09-13T15:01:26.635 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:26.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:26.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:26.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:26.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:26.769 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:26.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:26.870 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:26.910 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:26.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:26.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:27.012 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:27.045 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:27.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:27.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:27.147 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:27.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:27.204 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:27.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:27.305 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:27.348 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:27.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:27.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:27.450 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:27.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:27.498 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:27.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:27.599 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:27.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:27 smithi184 ceph-mon[92527]: pgmap v2167: 119 pgs: 1 peering, 3 active+remapped+backfilling, 7 active+remapped+backfill_wait, 108 active+clean; 2.7 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 1.00k op/s; 41987/133392 objects misplaced (31.476%); 24 MiB/s, 389 objects/s recovering 2024-09-13T15:01:27.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:27 smithi151 ceph-mon[104450]: pgmap v2167: 119 pgs: 1 peering, 3 active+remapped+backfilling, 7 active+remapped+backfill_wait, 108 active+clean; 2.7 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 1.00k op/s; 41987/133392 objects misplaced (31.476%); 24 MiB/s, 389 objects/s recovering 2024-09-13T15:01:27.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:27 smithi151 ceph-mon[108966]: pgmap v2167: 119 pgs: 1 peering, 3 active+remapped+backfilling, 7 active+remapped+backfill_wait, 108 active+clean; 2.7 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 1.00k op/s; 41987/133392 objects misplaced (31.476%); 24 MiB/s, 389 objects/s recovering 2024-09-13T15:01:27.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:27.629 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:27.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:27.730 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:27.777 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:27.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:27.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:27.879 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:27.912 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:27.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:27.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:28.014 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:28.050 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:28.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:28.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:28.152 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:28.190 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:28.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:28.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:28.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:28.340 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:28.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:28.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:28.442 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:28.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:28.477 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:28.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:28.577 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:28.621 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:28.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:28.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:28.723 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:28.752 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:28.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:28.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:28.855 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:28.886 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:28.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:28.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:28.989 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:29.022 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:29.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:29.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:29.124 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:29.164 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:29.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:29.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:29.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:29.299 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:29.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:29.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:29.401 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:29.432 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:29.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:29.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:29.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:29 smithi151 ceph-mon[104450]: pgmap v2168: 119 pgs: 1 active+remapped+backfilling, 8 active+remapped+backfill_wait, 110 active+clean; 2.6 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 1.05k op/s; 36259/127395 objects misplaced (28.462%); 50 MiB/s, 804 objects/s recovering 2024-09-13T15:01:29.439 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:29 smithi151 ceph-mon[108966]: pgmap v2168: 119 pgs: 1 active+remapped+backfilling, 8 active+remapped+backfill_wait, 110 active+clean; 2.6 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 1.05k op/s; 36259/127395 objects misplaced (28.462%); 50 MiB/s, 804 objects/s recovering 2024-09-13T15:01:29.536 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:29.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:29.591 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:29.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:29.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:29 smithi184 ceph-mon[92527]: pgmap v2168: 119 pgs: 1 active+remapped+backfilling, 8 active+remapped+backfill_wait, 110 active+clean; 2.6 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 1.05k op/s; 36259/127395 objects misplaced (28.462%); 50 MiB/s, 804 objects/s recovering 2024-09-13T15:01:29.692 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:29.721 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:29.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:29.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:29.822 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:29.864 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:29.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:29.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:29.966 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:29.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:29.996 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:29.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:30.096 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:30.100 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 6, 7, 1, 5] out_osds: [0, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T15:01:30.101 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:01:30.101 INFO:tasks.thrashosds.thrasher:Reweighting osd 2 to 0.9672816486885039 2024-09-13T15:01:30.101 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 2 0.9672816486885039 2024-09-13T15:01:30.129 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:30.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:30.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:30.231 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:30.266 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:30.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:30.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:30.368 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:30.395 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:30.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:30.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:30.497 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:30.522 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:30.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:30.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:30.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:01:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:01:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:01:30.623 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:30.650 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:30.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:30.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:30.752 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:30.778 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:30.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:30.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:30.880 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:30.907 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:30.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:30.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:31.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:31.038 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:31.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:31.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:31.123 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:31 smithi151 ceph-mon[104450]: pgmap v2169: 119 pgs: 2 active+remapped+backfilling, 7 active+remapped+backfill_wait, 110 active+clean; 2.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 719 op/s; 34689/122844 objects misplaced (28.238%); 30 MiB/s, 480 objects/s recovering 2024-09-13T15:01:31.123 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:31 smithi151 ceph-mon[104450]: osdmap e1302: 8 total, 8 up, 6 in 2024-09-13T15:01:31.123 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:31 smithi151 ceph-mon[108966]: pgmap v2169: 119 pgs: 2 active+remapped+backfilling, 7 active+remapped+backfill_wait, 110 active+clean; 2.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 719 op/s; 34689/122844 objects misplaced (28.238%); 30 MiB/s, 480 objects/s recovering 2024-09-13T15:01:31.123 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:31 smithi151 ceph-mon[108966]: osdmap e1302: 8 total, 8 up, 6 in 2024-09-13T15:01:31.140 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:31.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:31.182 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:31.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:31.283 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:31.313 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:31.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:31.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:31.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:31 smithi184 ceph-mon[92527]: pgmap v2169: 119 pgs: 2 active+remapped+backfilling, 7 active+remapped+backfill_wait, 110 active+clean; 2.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 719 op/s; 34689/122844 objects misplaced (28.238%); 30 MiB/s, 480 objects/s recovering 2024-09-13T15:01:31.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:31 smithi184 ceph-mon[92527]: osdmap e1302: 8 total, 8 up, 6 in 2024-09-13T15:01:31.415 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:31.446 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:31.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:31.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:31.548 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:31.590 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:31.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:31.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:31.692 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:31.720 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:31.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:31.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:31.821 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:31.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:31.861 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:31.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:31.962 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:32.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:32.017 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:32.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:32.045 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T15:01:32.045 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:01:32.116 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-13T15:01:32.117 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:32.135 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:32 smithi151 ceph-mon[104450]: osdmap e1303: 8 total, 8 up, 6 in 2024-09-13T15:01:32.135 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:32.135 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:32 smithi151 ceph-mon[108966]: osdmap e1303: 8 total, 8 up, 6 in 2024-09-13T15:01:32.136 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:32.143 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:32.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:32.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:32.245 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:32.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:32.285 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:32.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:32 smithi184 ceph-mon[92527]: osdmap e1303: 8 total, 8 up, 6 in 2024-09-13T15:01:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:32.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:32.429 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:32.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:32.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:32.490 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:32.531 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:32.572 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:32.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:32.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:32.674 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:32.714 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:32.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:32.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:32.817 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:32.819 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T15:01:32.820 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:01:32.838 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:32.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:32.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:32.881 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T15:01:32.882 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:01:32.934 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T15:01:32.934 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:01:32.940 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:32.967 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T15:01:32.968 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:01:32.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:32.972 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:32.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:32.999 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T15:01:33.000 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:01:33.031 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T15:01:33.032 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:01:33.064 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T15:01:33.064 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:01:33.073 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:33.095 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T15:01:33.095 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:01:33.101 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:33.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:33.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:33.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:33.246 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:33.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:33.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:33.348 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:33.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:33 smithi184 ceph-mon[92527]: pgmap v2172: 119 pgs: 2 active+remapped+backfilling, 7 active+remapped+backfill_wait, 110 active+clean; 2.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 524 op/s; 34689/122844 objects misplaced (28.238%); 30 MiB/s, 481 objects/s recovering 2024-09-13T15:01:33.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:33 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/904018598' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:01:33.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:33 smithi184 ceph-mon[92527]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:01:33.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:33 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/904018598' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:01:33.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:33 smithi184 ceph-mon[92527]: osdmap e1304: 8 total, 8 up, 6 in 2024-09-13T15:01:33.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:33 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/316101730' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:33.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:33 smithi151 ceph-mon[104450]: pgmap v2172: 119 pgs: 2 active+remapped+backfilling, 7 active+remapped+backfill_wait, 110 active+clean; 2.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 524 op/s; 34689/122844 objects misplaced (28.238%); 30 MiB/s, 481 objects/s recovering 2024-09-13T15:01:33.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:33 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/904018598' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:01:33.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:33 smithi151 ceph-mon[104450]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:01:33.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:33 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/904018598' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:01:33.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:33 smithi151 ceph-mon[104450]: osdmap e1304: 8 total, 8 up, 6 in 2024-09-13T15:01:33.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:33 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/316101730' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:33.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:33 smithi151 ceph-mon[108966]: pgmap v2172: 119 pgs: 2 active+remapped+backfilling, 7 active+remapped+backfill_wait, 110 active+clean; 2.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 524 op/s; 34689/122844 objects misplaced (28.238%); 30 MiB/s, 481 objects/s recovering 2024-09-13T15:01:33.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:33 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/904018598' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:01:33.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:33 smithi151 ceph-mon[108966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:01:33.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:33 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/904018598' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:01:33.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:33 smithi151 ceph-mon[108966]: osdmap e1304: 8 total, 8 up, 6 in 2024-09-13T15:01:33.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:33 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/316101730' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:33.378 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:33.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:33.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:33.480 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:33.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:33.520 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:33.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:33.621 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:33.659 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:33.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:33.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:33.761 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:33.795 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:33.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:33.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:33.897 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:33.928 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:33.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:33.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:34.030 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:34.051 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:34.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:34.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:34.153 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:34.192 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:34.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:34.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:34.294 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:34.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:34.336 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:34.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:34.437 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:34.465 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:34.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:34.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:34.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:34.607 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:34.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:34.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:34.709 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:34.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:34.740 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:34.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:34.841 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:34.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:01:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:01:34] "GET /metrics HTTP/1.1" 200 36737 "" "Prometheus/2.51.0" 2024-09-13T15:01:34.872 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:34.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:34.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:34.974 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:35.014 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:35.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:35.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:35.116 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:35.146 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:35.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:35.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:35.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:35 smithi151 ceph-mon[104450]: pgmap v2174: 119 pgs: 2 active+remapped+backfilling, 6 active+remapped+backfill_wait, 111 active+clean; 2.3 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 856 op/s; 29366/111960 objects misplaced (26.229%); 20 MiB/s, 327 objects/s recovering 2024-09-13T15:01:35.187 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:35 smithi151 ceph-mon[108966]: pgmap v2174: 119 pgs: 2 active+remapped+backfilling, 6 active+remapped+backfill_wait, 111 active+clean; 2.3 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 856 op/s; 29366/111960 objects misplaced (26.229%); 20 MiB/s, 327 objects/s recovering 2024-09-13T15:01:35.247 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:35.283 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:35.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:35.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:35.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:35 smithi184 ceph-mon[92527]: pgmap v2174: 119 pgs: 2 active+remapped+backfilling, 6 active+remapped+backfill_wait, 111 active+clean; 2.3 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 856 op/s; 29366/111960 objects misplaced (26.229%); 20 MiB/s, 327 objects/s recovering 2024-09-13T15:01:35.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:35.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:35.423 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:35.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:35.524 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:35.556 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:35.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:35.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:35.658 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:35.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:35.692 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:35.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:35.793 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:35.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:35.833 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:35.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:35.934 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:35.969 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:35.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:35.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:36.071 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:36.100 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:36.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:36.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:36.202 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:36.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:36.235 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:36.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:36.336 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:36.367 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:36.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:36.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:36.469 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:36.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:36.517 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:36.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:36.527 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T15:01:36.619 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:36.646 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:36.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:36.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:36.749 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:36.801 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:36.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:36.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:36.903 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:36.919 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:01:36.942 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:36.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:36.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:37.044 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:37.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:37.072 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:37.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:37.085 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:37 smithi151 ceph-mon[104450]: pgmap v2175: 119 pgs: 2 active+remapped+backfilling, 6 active+remapped+backfill_wait, 111 active+clean; 2.3 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 603 op/s; 29366/111960 objects misplaced (26.229%); 19 MiB/s, 296 objects/s recovering 2024-09-13T15:01:37.173 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:37.215 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:37.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:37.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:37.317 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:37.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:37 smithi184 ceph-mon[92527]: pgmap v2175: 119 pgs: 2 active+remapped+backfilling, 6 active+remapped+backfill_wait, 111 active+clean; 2.3 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 603 op/s; 29366/111960 objects misplaced (26.229%); 19 MiB/s, 296 objects/s recovering 2024-09-13T15:01:37.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:37 smithi151 ceph-mon[108966]: pgmap v2175: 119 pgs: 2 active+remapped+backfilling, 6 active+remapped+backfill_wait, 111 active+clean; 2.3 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 603 op/s; 29366/111960 objects misplaced (26.229%); 19 MiB/s, 296 objects/s recovering 2024-09-13T15:01:37.364 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:37.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:37.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:37.466 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:37.498 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:37.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:37.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:37.600 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:37.609 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:37.658 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:37.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:37.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:37.761 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:37.785 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:37.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:37.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:37.887 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:37.918 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:37.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:37.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:37.969 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:01:38.020 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:38.046 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:38.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:38.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:38.148 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:38.174 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:38.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:38.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:38.276 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:38.319 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:38.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:38.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:38.422 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:38.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:38.460 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:38.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:38.562 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:38.598 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:38.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:38.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:38.700 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:38.725 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:38.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:38.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:38.826 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:38.856 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:38.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:38.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:38.957 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:39.004 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:39.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:39.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:39.106 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:39.146 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:39.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:39.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:39.248 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:39.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:39.281 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:39.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:39.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:39 smithi184 ceph-mon[92527]: pgmap v2176: 119 pgs: 2 active+remapped+backfilling, 6 active+remapped+backfill_wait, 111 active+clean; 2.1 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 839 op/s; 26789/103260 objects misplaced (25.943%); 14 MiB/s, 228 objects/s recovering 2024-09-13T15:01:39.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:39 smithi151 ceph-mon[104450]: pgmap v2176: 119 pgs: 2 active+remapped+backfilling, 6 active+remapped+backfill_wait, 111 active+clean; 2.1 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 839 op/s; 26789/103260 objects misplaced (25.943%); 14 MiB/s, 228 objects/s recovering 2024-09-13T15:01:39.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:39 smithi151 ceph-mon[108966]: pgmap v2176: 119 pgs: 2 active+remapped+backfilling, 6 active+remapped+backfill_wait, 111 active+clean; 2.1 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 839 op/s; 26789/103260 objects misplaced (25.943%); 14 MiB/s, 228 objects/s recovering 2024-09-13T15:01:39.382 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:39.408 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:39.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:39.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:39.510 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:39.537 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:39.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:39.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:39.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:39.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:39.669 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:39.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:39.770 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:39.800 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:39.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:39.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:39.902 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:39.936 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:39.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:39.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:40.038 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:40.067 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:40.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:40.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:40.168 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:40.203 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:40.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:40.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:40.305 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:40.335 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:40.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:40.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:40.437 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:40.469 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:40.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:40.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:40.571 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:40.603 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:40.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:40.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:40.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:01:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:01:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:01:40.705 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:40.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:40.765 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:40.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:40.866 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:40.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:40.920 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:40.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:41.021 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:41.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:41.045 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:41.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:41.107 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.2 to 0.967282 (f79f) 2024-09-13T15:01:41.146 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:41.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:41.193 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:41.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:41.294 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:41.325 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:41.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:41.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:41.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:41 smithi151 ceph-mon[104450]: pgmap v2177: 119 pgs: 2 active+remapped+backfilling, 6 active+remapped+backfill_wait, 111 active+clean; 2.0 GiB data, 9.9 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 941 op/s; 24519/98043 objects misplaced (25.008%); 17 MiB/s, 277 objects/s recovering 2024-09-13T15:01:41.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:41 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1376900125' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.9672816486885039} : dispatch 2024-09-13T15:01:41.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:41 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.9672816486885039} : dispatch 2024-09-13T15:01:41.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:41 smithi151 ceph-mon[108966]: pgmap v2177: 119 pgs: 2 active+remapped+backfilling, 6 active+remapped+backfill_wait, 111 active+clean; 2.0 GiB data, 9.9 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 941 op/s; 24519/98043 objects misplaced (25.008%); 17 MiB/s, 277 objects/s recovering 2024-09-13T15:01:41.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:41 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1376900125' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.9672816486885039} : dispatch 2024-09-13T15:01:41.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:41 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.9672816486885039} : dispatch 2024-09-13T15:01:41.426 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:41.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:41.459 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:41.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:41.560 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:41.594 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:41.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:41.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:41.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:41 smithi184 ceph-mon[92527]: pgmap v2177: 119 pgs: 2 active+remapped+backfilling, 6 active+remapped+backfill_wait, 111 active+clean; 2.0 GiB data, 9.9 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 941 op/s; 24519/98043 objects misplaced (25.008%); 17 MiB/s, 277 objects/s recovering 2024-09-13T15:01:41.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:41 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1376900125' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.9672816486885039} : dispatch 2024-09-13T15:01:41.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:41 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.9672816486885039} : dispatch 2024-09-13T15:01:41.696 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:41.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:41.742 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:41.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:41.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:41.905 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:41.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:41.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:42.006 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:42.067 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:42.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:42.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:42.169 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:42.210 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:42.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:42.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:42.312 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:42.341 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:42.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:42.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:42.443 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:42.474 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:42.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:42.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:42.576 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:42.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:42 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.9672816486885039}]': finished 2024-09-13T15:01:42.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:42 smithi184 ceph-mon[92527]: osdmap e1305: 8 total, 8 up, 6 in 2024-09-13T15:01:42.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:42 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:01:42.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:42 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.9672816486885039}]': finished 2024-09-13T15:01:42.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:42 smithi151 ceph-mon[104450]: osdmap e1305: 8 total, 8 up, 6 in 2024-09-13T15:01:42.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:42 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:01:42.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:42 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.9672816486885039}]': finished 2024-09-13T15:01:42.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:42 smithi151 ceph-mon[108966]: osdmap e1305: 8 total, 8 up, 6 in 2024-09-13T15:01:42.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:42 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:01:42.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:42.618 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:42.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:42.719 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:42.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:42.761 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:42.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:42.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:42.892 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:42.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:42.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:42.993 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:43.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:43.041 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:43.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:43.142 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:43.165 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:43.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:43.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:43.267 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:43.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:43.313 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:43.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:43.415 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:43.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:43.451 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:43.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:43.552 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:43.595 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:43.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:43.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:43.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:43 smithi184 ceph-mon[92527]: pgmap v2179: 119 pgs: 2 active+remapped+backfilling, 6 active+remapped+backfill_wait, 111 active+clean; 2.0 GiB data, 9.9 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 851 op/s; 24519/98043 objects misplaced (25.008%); 16 MiB/s, 250 objects/s recovering 2024-09-13T15:01:43.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:43 smithi184 ceph-mon[92527]: osdmap e1306: 8 total, 8 up, 6 in 2024-09-13T15:01:43.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:43 smithi151 ceph-mon[104450]: pgmap v2179: 119 pgs: 2 active+remapped+backfilling, 6 active+remapped+backfill_wait, 111 active+clean; 2.0 GiB data, 9.9 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 851 op/s; 24519/98043 objects misplaced (25.008%); 16 MiB/s, 250 objects/s recovering 2024-09-13T15:01:43.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:43 smithi151 ceph-mon[104450]: osdmap e1306: 8 total, 8 up, 6 in 2024-09-13T15:01:43.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:43 smithi151 ceph-mon[108966]: pgmap v2179: 119 pgs: 2 active+remapped+backfilling, 6 active+remapped+backfill_wait, 111 active+clean; 2.0 GiB data, 9.9 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 851 op/s; 24519/98043 objects misplaced (25.008%); 16 MiB/s, 250 objects/s recovering 2024-09-13T15:01:43.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:43 smithi151 ceph-mon[108966]: osdmap e1306: 8 total, 8 up, 6 in 2024-09-13T15:01:43.696 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:43.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:43.746 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:43.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:43.847 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:43.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:43.882 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:43.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:43.983 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:44.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:44.023 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:44.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:44.074 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:44.124 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:44.175 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:44.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:44.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:44.277 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:44.312 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:44.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:44.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:44.414 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:44.451 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:44.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:44.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:44.553 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:44.594 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:44.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:44.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:44.695 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:44.737 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:44.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:44.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:44.839 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:44.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:01:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:01:44] "GET /metrics HTTP/1.1" 200 36737 "" "Prometheus/2.51.0" 2024-09-13T15:01:44.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:44.881 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:44.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:44.982 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:45.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:45.017 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:45.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:45.118 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:45.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:45.149 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:45.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:45.250 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:45.289 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:45.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:45.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:45.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:45 smithi151 ceph-mon[104450]: pgmap v2181: 119 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_wait, 111 active+clean; 1.7 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 1.08k op/s; 20143/85908 objects misplaced (23.447%); 12 MiB/s, 185 objects/s recovering 2024-09-13T15:01:45.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:45 smithi151 ceph-mon[108966]: pgmap v2181: 119 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_wait, 111 active+clean; 1.7 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 1.08k op/s; 20143/85908 objects misplaced (23.447%); 12 MiB/s, 185 objects/s recovering 2024-09-13T15:01:45.391 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:45.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:45.448 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:45.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:45.549 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:45.583 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:45.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:45.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:45.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:45 smithi184 ceph-mon[92527]: pgmap v2181: 119 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_wait, 111 active+clean; 1.7 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 1.08k op/s; 20143/85908 objects misplaced (23.447%); 12 MiB/s, 185 objects/s recovering 2024-09-13T15:01:45.685 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:45.714 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:45.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:45.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:45.816 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:45.856 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:45.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:45.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:45.958 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:45.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:45.997 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:45.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:46.099 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:46.130 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:46.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:46.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:46.232 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:46.270 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:46.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:46.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:46.372 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:46.417 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:46.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:46.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:46.520 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:46.561 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:46.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:46.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:46.570 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:46.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:46.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:46.716 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:46.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:46.817 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:46.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:46.854 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:46.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:46.955 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:46.992 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:46.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:46.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:47.094 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:47.134 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:47.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:47.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:47.237 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:47.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:47.301 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:47.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:47.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:47 smithi184 ceph-mon[92527]: pgmap v2182: 119 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_wait, 111 active+clean; 1.7 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 722 op/s; 20143/85908 objects misplaced (23.447%); 12 MiB/s, 185 objects/s recovering 2024-09-13T15:01:47.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:47 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:47.401 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:47.412 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:47 smithi151 ceph-mon[104450]: pgmap v2182: 119 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_wait, 111 active+clean; 1.7 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 722 op/s; 20143/85908 objects misplaced (23.447%); 12 MiB/s, 185 objects/s recovering 2024-09-13T15:01:47.412 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:47 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:47.413 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:47 smithi151 ceph-mon[108966]: pgmap v2182: 119 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_wait, 111 active+clean; 1.7 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 722 op/s; 20143/85908 objects misplaced (23.447%); 12 MiB/s, 185 objects/s recovering 2024-09-13T15:01:47.413 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:47 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:47.429 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:47.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:47.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:47.530 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:47.572 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:47.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:47.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:47.674 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:47.710 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:47.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:47.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:47.812 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:47.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:47.850 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:47.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:47.952 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:47.964 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T15:01:47.964 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-13T15:01:47.964 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-13T15:01:47.964 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-13T15:01:47.964 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T15:01:47.986 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:47.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:47.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:48.088 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:48.119 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:48.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:48.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:48.220 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:48.264 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:48.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:48.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:48.366 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:48.404 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:48.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:48.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:48.506 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:48.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:48.547 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:48.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:48.648 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:48.689 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:48.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:48.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:48.791 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:48.842 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:48.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:48.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:48.944 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:48.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:48.992 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:48.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:49.093 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:49.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:49.139 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:49.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:49.240 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:49.272 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:49.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:49.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:49.334 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:49 smithi151 ceph-mon[104450]: pgmap v2183: 119 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_wait, 111 active+clean; 1.6 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 846 op/s; 15912/77703 objects misplaced (20.478%); 19 MiB/s, 309 objects/s recovering 2024-09-13T15:01:49.334 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:49 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:01:49.334 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:49 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:01:49.334 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:49 smithi151 ceph-mon[108966]: pgmap v2183: 119 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_wait, 111 active+clean; 1.6 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 846 op/s; 15912/77703 objects misplaced (20.478%); 19 MiB/s, 309 objects/s recovering 2024-09-13T15:01:49.335 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:49 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:01:49.335 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:49 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:01:49.374 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:49.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:49.401 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:49.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:49.502 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:49.506 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:49 smithi184 ceph-mon[92527]: pgmap v2183: 119 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_wait, 111 active+clean; 1.6 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 846 op/s; 15912/77703 objects misplaced (20.478%); 19 MiB/s, 309 objects/s recovering 2024-09-13T15:01:49.506 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:49 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:01:49.506 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:49 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:01:49.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:49.536 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:49.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:49.637 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:49.661 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:49.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:49.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:49.679 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 6, 7, 1, 5] out_osds: [0, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T15:01:49.679 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T15:01:49.763 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:49.822 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:49.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:49.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:49.923 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:49.947 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:49.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:49.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:50.049 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:50.079 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:50.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:50.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:50.181 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:50.202 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:50.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:50.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:50.304 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:50.332 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:50.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:50.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:50.434 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:50.456 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:50.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:50.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:50.558 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:50.573 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:01:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:01:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:01:50.581 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:50.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:50.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:50.583 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-13T15:01:50.683 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:50.723 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:50.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:50.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:50.825 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:50.829 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:50 smithi151 ceph-mon[104450]: osdmap e1307: 8 total, 8 up, 6 in 2024-09-13T15:01:50.829 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:50 smithi151 ceph-mon[104450]: pgmap v2185: 119 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_wait, 111 active+clean; 1.5 GiB data, 8.9 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 14655/72411 objects misplaced (20.239%); 22 MiB/s, 344 objects/s recovering 2024-09-13T15:01:50.829 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:50 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2131978110' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:01:50.829 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:50 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:01:50.830 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:50 smithi151 ceph-mon[108966]: osdmap e1307: 8 total, 8 up, 6 in 2024-09-13T15:01:50.830 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:50 smithi151 ceph-mon[108966]: pgmap v2185: 119 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_wait, 111 active+clean; 1.5 GiB data, 8.9 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 14655/72411 objects misplaced (20.239%); 22 MiB/s, 344 objects/s recovering 2024-09-13T15:01:50.830 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:50 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2131978110' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:01:50.830 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:50 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:01:50.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:50 smithi184 ceph-mon[92527]: osdmap e1307: 8 total, 8 up, 6 in 2024-09-13T15:01:50.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:50 smithi184 ceph-mon[92527]: pgmap v2185: 119 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_wait, 111 active+clean; 1.5 GiB data, 8.9 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 14655/72411 objects misplaced (20.239%); 22 MiB/s, 344 objects/s recovering 2024-09-13T15:01:50.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:50 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2131978110' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:01:50.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:50 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:01:50.857 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:50.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:50.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:50.959 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:50.980 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:50.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:50.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:51.082 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:51.110 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:51.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:51.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:51.212 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:51.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:51.237 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:51.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:51.338 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:51.363 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:51.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:51.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:51.465 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:51.487 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:51.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:51.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:51.589 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:51.619 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:51.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:51.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:51.721 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:51.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:51.767 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:51.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:51.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:51 smithi184 ceph-mon[92527]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:01:51.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:51 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:01:51.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:51 smithi184 ceph-mon[92527]: osdmap e1308: 8 total, 8 up, 6 in 2024-09-13T15:01:51.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:51 smithi151 ceph-mon[104450]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:01:51.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:51 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:01:51.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:51 smithi151 ceph-mon[104450]: osdmap e1308: 8 total, 8 up, 6 in 2024-09-13T15:01:51.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:51 smithi151 ceph-mon[108966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:01:51.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:51 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:01:51.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:51 smithi151 ceph-mon[108966]: osdmap e1308: 8 total, 8 up, 6 in 2024-09-13T15:01:51.868 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:51.904 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T15:01:51.922 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:51.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:51.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:52.025 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:52.068 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:52.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:52.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:52.170 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:52.196 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:52.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:52.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:52.298 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:52.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:52.358 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:52.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:52.459 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:52.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:52.512 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:52.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:52.512 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T15:01:52.513 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:01:52.563 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T15:01:52.564 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:01:52.600 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T15:01:52.600 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:01:52.613 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:52.635 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T15:01:52.635 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:01:52.667 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:52.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:52.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:52.673 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T15:01:52.673 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:01:52.702 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T15:01:52.702 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:01:52.729 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T15:01:52.730 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:01:52.759 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T15:01:52.759 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:01:52.769 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:52.798 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:52.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:52.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:52.833 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:52 smithi151 ceph-mon[104450]: osdmap e1309: 8 total, 8 up, 6 in 2024-09-13T15:01:52.833 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:52 smithi151 ceph-mon[104450]: pgmap v2188: 119 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_wait, 111 active+clean; 1.5 GiB data, 8.9 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 749 op/s; 14655/72411 objects misplaced (20.239%); 20 MiB/s, 321 objects/s recovering 2024-09-13T15:01:52.833 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:52 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/882501703' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:52.833 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:52 smithi151 ceph-mon[108966]: osdmap e1309: 8 total, 8 up, 6 in 2024-09-13T15:01:52.834 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:52 smithi151 ceph-mon[108966]: pgmap v2188: 119 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_wait, 111 active+clean; 1.5 GiB data, 8.9 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 749 op/s; 14655/72411 objects misplaced (20.239%); 20 MiB/s, 321 objects/s recovering 2024-09-13T15:01:52.834 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:52 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/882501703' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:52.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:52 smithi184 ceph-mon[92527]: osdmap e1309: 8 total, 8 up, 6 in 2024-09-13T15:01:52.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:52 smithi184 ceph-mon[92527]: pgmap v2188: 119 pgs: 3 active+remapped+backfilling, 5 active+remapped+backfill_wait, 111 active+clean; 1.5 GiB data, 8.9 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 749 op/s; 14655/72411 objects misplaced (20.239%); 20 MiB/s, 321 objects/s recovering 2024-09-13T15:01:52.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:52 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/882501703' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:52.901 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:52.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:52.930 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:52.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:53.031 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:53.060 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:53.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:53.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:53.162 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:53.204 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:53.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:53.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:53.228 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:53.306 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:53.334 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:53.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:53.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:53.436 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:53.465 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:53.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:53.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:53.568 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:53.604 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:53.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:53.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:53.706 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:53.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:53.786 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:53.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:53.887 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:53.917 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:53.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:53.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:54.019 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:54.044 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:54.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:54.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:54.146 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:54.170 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:54.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:54.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:54.272 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:54.295 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:54.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:54.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:54.397 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:54.420 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:54.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:54.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:54.463 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:54.521 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:54.545 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:54.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:54.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:54.646 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:54.671 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:54.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:54.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:54.773 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:54.796 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:54.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:54.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:54.836 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:01:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:01:54] "GET /metrics HTTP/1.1" 200 36751 "" "Prometheus/2.51.0" 2024-09-13T15:01:54.898 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:54.933 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:54.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:54.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:55.035 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:55.035 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T15:01:55.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:54 smithi184 ceph-mon[92527]: osdmap e1310: 8 total, 8 up, 6 in 2024-09-13T15:01:55.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:54 smithi184 ceph-mon[92527]: pgmap v2190: 119 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_wait, 113 active+clean; 1.3 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 712 op/s; 9120/63267 objects misplaced (14.415%); 39 MiB/s, 626 objects/s recovering 2024-09-13T15:01:55.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:54 smithi151 ceph-mon[108966]: osdmap e1310: 8 total, 8 up, 6 in 2024-09-13T15:01:55.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:54 smithi151 ceph-mon[108966]: pgmap v2190: 119 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_wait, 113 active+clean; 1.3 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 712 op/s; 9120/63267 objects misplaced (14.415%); 39 MiB/s, 626 objects/s recovering 2024-09-13T15:01:55.112 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:54 smithi151 ceph-mon[104450]: osdmap e1310: 8 total, 8 up, 6 in 2024-09-13T15:01:55.112 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:54 smithi151 ceph-mon[104450]: pgmap v2190: 119 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_wait, 113 active+clean; 1.3 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 712 op/s; 9120/63267 objects misplaced (14.415%); 39 MiB/s, 626 objects/s recovering 2024-09-13T15:01:55.114 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:55.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:55.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:55.216 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:55.246 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:55.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:55.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:55.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:55.387 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:55.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:55.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:55.489 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:55.510 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:55.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:55.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:55.612 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:55.635 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:55.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:55.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:55.696 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:01:55.737 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:55.759 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:55.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:55.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:55.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:55.885 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:55.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:55.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:55.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:56.008 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:56.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:56.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:56.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:55 smithi184 ceph-mon[92527]: osdmap e1311: 8 total, 8 up, 6 in 2024-09-13T15:01:56.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:55 smithi184 ceph-mon[92527]: osdmap e1312: 8 total, 8 up, 6 in 2024-09-13T15:01:56.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:55 smithi151 ceph-mon[104450]: osdmap e1311: 8 total, 8 up, 6 in 2024-09-13T15:01:56.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:55 smithi151 ceph-mon[104450]: osdmap e1312: 8 total, 8 up, 6 in 2024-09-13T15:01:56.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:55 smithi151 ceph-mon[108966]: osdmap e1311: 8 total, 8 up, 6 in 2024-09-13T15:01:56.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:55 smithi151 ceph-mon[108966]: osdmap e1312: 8 total, 8 up, 6 in 2024-09-13T15:01:56.110 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:56.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:56.145 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:56.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:56.246 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:56.279 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:56.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:56.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:56.381 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:56.413 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:56.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:56.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:56.514 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:56.541 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:56.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:56.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:56.643 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:56.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:56.676 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:56.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:56.778 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:56.816 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:56.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:56.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:56.918 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:56.950 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:56.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:56.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:57.052 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:57.082 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:57.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:57.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:57.184 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:57.217 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:57.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:57.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:57.319 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:57.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:57 smithi184 ceph-mon[92527]: pgmap v2193: 119 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_wait, 113 active+clean; 1.3 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 714 op/s; 9120/63267 objects misplaced (14.415%); 39 MiB/s, 628 objects/s recovering 2024-09-13T15:01:57.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:57 smithi184 ceph-mon[92527]: osdmap e1313: 8 total, 8 up, 6 in 2024-09-13T15:01:57.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:57.355 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:57.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:57.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:57 smithi151 ceph-mon[104450]: pgmap v2193: 119 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_wait, 113 active+clean; 1.3 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 714 op/s; 9120/63267 objects misplaced (14.415%); 39 MiB/s, 628 objects/s recovering 2024-09-13T15:01:57.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:57 smithi151 ceph-mon[104450]: osdmap e1313: 8 total, 8 up, 6 in 2024-09-13T15:01:57.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:57 smithi151 ceph-mon[108966]: pgmap v2193: 119 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_wait, 113 active+clean; 1.3 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 714 op/s; 9120/63267 objects misplaced (14.415%); 39 MiB/s, 628 objects/s recovering 2024-09-13T15:01:57.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:57 smithi151 ceph-mon[108966]: osdmap e1313: 8 total, 8 up, 6 in 2024-09-13T15:01:57.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:57.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:57.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:57.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:57.595 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:57.632 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:57.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:57.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:57.733 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:57.768 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:57.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:57.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:57.870 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:57.905 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:57.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:57.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:58.007 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:58.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:58.050 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:58.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:58.150 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:58.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:58.184 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:58.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:58.285 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:58.315 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:58.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:58.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:58.417 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:58.441 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:58.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:58.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:58.543 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:58.567 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:58.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:58.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:58.668 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:58.692 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:58.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:58.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:58.794 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:58.816 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:58.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:58.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:58.919 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:58.941 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:58.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:58.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:59.043 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:59.066 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:59.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:59.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:59.168 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:59.203 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:59.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:59.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:59.305 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:59.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:59.330 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:59.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:59.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:59 smithi184 ceph-mon[92527]: pgmap v2195: 119 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_wait, 114 active+clean; 1.2 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 495 op/s; 5916/57297 objects misplaced (10.325%); 20 MiB/s, 323 objects/s recovering 2024-09-13T15:01:59.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:01:59 smithi184 ceph-mon[92527]: osdmap e1314: 8 total, 8 up, 6 in 2024-09-13T15:01:59.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:59 smithi151 ceph-mon[104450]: pgmap v2195: 119 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_wait, 114 active+clean; 1.2 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 495 op/s; 5916/57297 objects misplaced (10.325%); 20 MiB/s, 323 objects/s recovering 2024-09-13T15:01:59.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:01:59 smithi151 ceph-mon[104450]: osdmap e1314: 8 total, 8 up, 6 in 2024-09-13T15:01:59.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:59 smithi151 ceph-mon[108966]: pgmap v2195: 119 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_wait, 114 active+clean; 1.2 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 495 op/s; 5916/57297 objects misplaced (10.325%); 20 MiB/s, 323 objects/s recovering 2024-09-13T15:01:59.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:01:59 smithi151 ceph-mon[108966]: osdmap e1314: 8 total, 8 up, 6 in 2024-09-13T15:01:59.431 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:59.463 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:59.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:59.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:59.565 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:59.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:59.595 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:59.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:59.696 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:59.718 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:59.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:59.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:59.819 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:59.867 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:01:59.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:59.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:59.969 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:59.996 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:01:59.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:59.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:00.098 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:00.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:00.130 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:00.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:00.231 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:00.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:00.253 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:00.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:00.353 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:00.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:00 smithi184 ceph-mon[92527]: osdmap e1315: 8 total, 8 up, 6 in 2024-09-13T15:02:00.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:00 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/4038793385' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:02:00.355 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:02:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:02:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:02:00.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:00 smithi151 ceph-mon[108966]: osdmap e1315: 8 total, 8 up, 6 in 2024-09-13T15:02:00.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:00 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/4038793385' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:02:00.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:00 smithi151 ceph-mon[104450]: osdmap e1315: 8 total, 8 up, 6 in 2024-09-13T15:02:00.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:00 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/4038793385' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:02:00.389 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:00.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:00.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:00.491 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:00.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:00.525 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:00.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:00.551 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T15:02:00.626 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:00.663 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:00.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:00.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:00.765 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:00.805 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:00.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:00.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:00.907 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:00.928 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:00.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:00.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:01.030 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:01.069 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:01.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:01.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:01.170 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:01.206 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:01.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:01.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:01.307 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:01.329 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:01.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:01.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:01.431 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:01.464 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:01.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:01.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:01.566 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:01.602 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:01.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:01.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:01.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:01 smithi184 ceph-mon[92527]: pgmap v2198: 119 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 116 active+clean; 1.1 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 657 op/s; 4868/53853 objects misplaced (9.039%); 24 MiB/s, 382 objects/s recovering 2024-09-13T15:02:01.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:01 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/4038793385' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:02:01.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:01 smithi184 ceph-mon[92527]: osdmap e1316: 8 total, 8 up, 6 in 2024-09-13T15:02:01.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:01 smithi151 ceph-mon[104450]: pgmap v2198: 119 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 116 active+clean; 1.1 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 657 op/s; 4868/53853 objects misplaced (9.039%); 24 MiB/s, 382 objects/s recovering 2024-09-13T15:02:01.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:01 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/4038793385' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:02:01.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:01 smithi151 ceph-mon[104450]: osdmap e1316: 8 total, 8 up, 6 in 2024-09-13T15:02:01.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:01 smithi151 ceph-mon[108966]: pgmap v2198: 119 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 116 active+clean; 1.1 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 657 op/s; 4868/53853 objects misplaced (9.039%); 24 MiB/s, 382 objects/s recovering 2024-09-13T15:02:01.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:01 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/4038793385' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:02:01.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:01 smithi151 ceph-mon[108966]: osdmap e1316: 8 total, 8 up, 6 in 2024-09-13T15:02:01.704 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:01.738 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:01.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:01.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:01.840 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:01.878 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:01.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:01.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:01.980 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:02.002 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:02.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:02.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:02.103 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:02.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:02.139 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:02.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:02.240 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:02.271 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:02.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:02.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:02.373 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:02.400 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:02.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:02.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:02.502 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:02.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:02.533 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:02.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:02.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:02 smithi184 ceph-mon[92527]: osdmap e1317: 8 total, 8 up, 6 in 2024-09-13T15:02:02.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:02.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:02 smithi151 ceph-mon[104450]: osdmap e1317: 8 total, 8 up, 6 in 2024-09-13T15:02:02.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:02.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:02 smithi151 ceph-mon[108966]: osdmap e1317: 8 total, 8 up, 6 in 2024-09-13T15:02:02.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:02.634 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:02.680 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:02.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:02.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:02.781 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:02.810 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:02.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:02.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:02.912 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:02.954 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:02.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:02.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:03.056 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:03.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:03.096 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:03.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:03.197 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:03.247 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:03.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:03.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:03.349 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:03.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:03.381 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:03.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:03.482 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:03.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:03.521 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:03.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:03.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:03 smithi184 ceph-mon[92527]: pgmap v2201: 119 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 116 active+clean; 1.1 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 286 op/s; 4868/53853 objects misplaced (9.039%); 8.2 MiB/s, 131 objects/s recovering 2024-09-13T15:02:03.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:03 smithi151 ceph-mon[108966]: pgmap v2201: 119 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 116 active+clean; 1.1 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 286 op/s; 4868/53853 objects misplaced (9.039%); 8.2 MiB/s, 131 objects/s recovering 2024-09-13T15:02:03.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:03 smithi151 ceph-mon[104450]: pgmap v2201: 119 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 116 active+clean; 1.1 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 286 op/s; 4868/53853 objects misplaced (9.039%); 8.2 MiB/s, 131 objects/s recovering 2024-09-13T15:02:03.622 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:03.655 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:03.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:03.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:03.757 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:03.791 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:03.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:03.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:03.892 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:03.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:03.928 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:03.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:04.029 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:04.068 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:04.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:04.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:04.170 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:04.195 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:04.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:04.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:04.205 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T15:02:04.297 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:04.352 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:04.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:04.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:04.454 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:04.484 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:04.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:04.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:04.586 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:04.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:04.618 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:04.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:04.719 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:04.765 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:04.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:04.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:04.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:02:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:02:04] "GET /metrics HTTP/1.1" 200 36734 "" "Prometheus/2.51.0" 2024-09-13T15:02:04.868 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:04.898 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:04.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:04.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:05.000 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:05.036 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:05.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:05.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:05.138 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:05.180 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:05.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:05.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:05.282 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:05.313 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:05.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:05.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:05.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:05 smithi151 ceph-mon[104450]: pgmap v2202: 119 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 117 active+clean; 858 MiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 927 op/s; 2304/41256 objects misplaced (5.585%); 13 MiB/s, 207 objects/s recovering 2024-09-13T15:02:05.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:05 smithi151 ceph-mon[108966]: pgmap v2202: 119 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 117 active+clean; 858 MiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 927 op/s; 2304/41256 objects misplaced (5.585%); 13 MiB/s, 207 objects/s recovering 2024-09-13T15:02:05.415 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:05.438 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:05.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:05.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:05.540 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:05.576 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:05.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:05.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:05.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:05 smithi184 ceph-mon[92527]: pgmap v2202: 119 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 117 active+clean; 858 MiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 927 op/s; 2304/41256 objects misplaced (5.585%); 13 MiB/s, 207 objects/s recovering 2024-09-13T15:02:05.677 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:05.710 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:05.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:05.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:05.812 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:05.835 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:05.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:05.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:05.937 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:05.963 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:05.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:05.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:06.065 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:06.095 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:06.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:06.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:06.196 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:06.198 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:06.257 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:06.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:06.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:06.359 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:06.390 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:06.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:06.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:06.394 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:06.492 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:06.498 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:06.535 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:06.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:06.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:06.637 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:06.662 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:06.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:06.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:06.763 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:06.791 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:06.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:06.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:06.893 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:06.915 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:06.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:06.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:07.017 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:07.038 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:07.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:07.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:07.141 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:07.164 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:07.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:07.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:07.265 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:07.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:07.300 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:07.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:07.401 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:07.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:07.438 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:07.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:07.438 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T15:02:07.438 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:02:07.439 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:07 smithi151 ceph-mon[108966]: pgmap v2203: 119 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 117 active+clean; 858 MiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 620 op/s; 2304/41256 objects misplaced (5.585%); 6.2 MiB/s, 99 objects/s recovering 2024-09-13T15:02:07.439 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:07 smithi151 ceph-mon[108966]: osdmap e1318: 8 total, 8 up, 6 in 2024-09-13T15:02:07.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:07 smithi151 ceph-mon[104450]: pgmap v2203: 119 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 117 active+clean; 858 MiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 620 op/s; 2304/41256 objects misplaced (5.585%); 6.2 MiB/s, 99 objects/s recovering 2024-09-13T15:02:07.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:07 smithi151 ceph-mon[104450]: osdmap e1318: 8 total, 8 up, 6 in 2024-09-13T15:02:07.539 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:07.561 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:07.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:07.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:07.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:07 smithi184 ceph-mon[92527]: pgmap v2203: 119 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 117 active+clean; 858 MiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 620 op/s; 2304/41256 objects misplaced (5.585%); 6.2 MiB/s, 99 objects/s recovering 2024-09-13T15:02:07.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:07 smithi184 ceph-mon[92527]: osdmap e1318: 8 total, 8 up, 6 in 2024-09-13T15:02:07.663 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:07.684 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:07.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:07.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:07.786 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:07.808 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:07.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:07.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:07.909 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:07.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:07.931 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:07.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:08.033 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:08.053 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:08.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:08.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:08.155 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:08.178 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:08.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:08.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:08.280 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:08.314 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:08 smithi151 ceph-mon[104450]: osdmap e1319: 8 total, 8 up, 6 in 2024-09-13T15:02:08.314 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:08 smithi151 ceph-mon[104450]: osdmap e1320: 8 total, 8 up, 6 in 2024-09-13T15:02:08.314 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:08 smithi151 ceph-mon[108966]: osdmap e1319: 8 total, 8 up, 6 in 2024-09-13T15:02:08.314 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:08 smithi151 ceph-mon[108966]: osdmap e1320: 8 total, 8 up, 6 in 2024-09-13T15:02:08.318 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:08.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:08.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:08.357 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T15:02:08.420 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:08.445 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:08.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:08.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:08.547 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:08.593 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:08.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:08.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:08.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:08 smithi184 ceph-mon[92527]: osdmap e1319: 8 total, 8 up, 6 in 2024-09-13T15:02:08.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:08 smithi184 ceph-mon[92527]: osdmap e1320: 8 total, 8 up, 6 in 2024-09-13T15:02:08.694 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:08.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:08.743 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:08.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:08.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:08.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:08.889 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:08.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:08.990 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:09.017 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:09.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:09.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:09.118 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:09.163 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:09.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:09.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:09.264 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:09.303 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:09.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:09.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:09.405 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:09.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:09.442 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:09.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:09.501 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:09 smithi151 ceph-mon[104450]: pgmap v2206: 119 pgs: 1 peering, 1 active+remapped+backfilling, 117 active+clean; 717 MiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 960 op/s; 1056/34449 objects misplaced (3.065%); 10 MiB/s, 167 objects/s recovering 2024-09-13T15:02:09.502 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:09 smithi151 ceph-mon[108966]: pgmap v2206: 119 pgs: 1 peering, 1 active+remapped+backfilling, 117 active+clean; 717 MiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 960 op/s; 1056/34449 objects misplaced (3.065%); 10 MiB/s, 167 objects/s recovering 2024-09-13T15:02:09.543 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:09.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:09.592 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:09.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:09.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:09 smithi184 ceph-mon[92527]: pgmap v2206: 119 pgs: 1 peering, 1 active+remapped+backfilling, 117 active+clean; 717 MiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 960 op/s; 1056/34449 objects misplaced (3.065%); 10 MiB/s, 167 objects/s recovering 2024-09-13T15:02:09.692 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:09.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:09.735 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:09.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:09.836 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:09.867 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:09.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:09.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:09.968 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:10.006 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:10.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:10.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:10.108 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:10.135 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:10.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:10.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:10.237 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:10.264 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:10.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:10.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:10.366 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:10.393 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:10.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:10.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:10.495 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:10.524 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:10.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:10.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:10.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:02:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:02:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:02:10.626 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:10.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:10.657 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:10.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:10.758 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:10.785 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:10.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:10.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:10.887 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:10.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:10.914 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:10.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:11.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:11.052 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:11.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:11.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:11.154 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:11.176 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:11 smithi151 ceph-mon[104450]: pgmap v2208: 119 pgs: 1 peering, 118 active+clean; 616 MiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 645 op/s; 7.8 MiB/s, 124 objects/s recovering 2024-09-13T15:02:11.184 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:11.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:11.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:11.286 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:11.326 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:11.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:11.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:11.427 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:11.460 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:11.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:11.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:11.562 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:11.589 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:11.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:11.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:11.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:11 smithi184 ceph-mon[92527]: pgmap v2208: 119 pgs: 1 peering, 118 active+clean; 616 MiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 645 op/s; 7.8 MiB/s, 124 objects/s recovering 2024-09-13T15:02:11.611 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:11 smithi151 ceph-mon[108966]: pgmap v2208: 119 pgs: 1 peering, 118 active+clean; 616 MiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 645 op/s; 7.8 MiB/s, 124 objects/s recovering 2024-09-13T15:02:11.691 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:11.728 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:11.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:11.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:11.830 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:11.862 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:11.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:11.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:11.964 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:11.992 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:11.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:11.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:12.094 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:12.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:12.135 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:12.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:12.236 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:12.287 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:12.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:12.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:12.390 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:12.415 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:12.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:12.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:12.516 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:12.548 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:12.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:12.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:12.650 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:12.687 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:12.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:12.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:12.789 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:12.820 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:12.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:12.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:12.922 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:12.959 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:12.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:12.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:13.061 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:13.091 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:13.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:13.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:13.193 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:13.233 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:13.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:13.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:13.335 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:13.369 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:13.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:13.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:13.405 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-13T15:02:13.472 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:13.494 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:13.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:13.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:13.595 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:13.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:13 smithi184 ceph-mon[92527]: pgmap v2209: 119 pgs: 1 peering, 118 active+clean; 616 MiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 645 op/s; 7.8 MiB/s, 124 objects/s recovering 2024-09-13T15:02:13.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:13 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2643143163' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:13.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:13 smithi151 ceph-mon[104450]: pgmap v2209: 119 pgs: 1 peering, 118 active+clean; 616 MiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 645 op/s; 7.8 MiB/s, 124 objects/s recovering 2024-09-13T15:02:13.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:13 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2643143163' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:13.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:13 smithi151 ceph-mon[108966]: pgmap v2209: 119 pgs: 1 peering, 118 active+clean; 616 MiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 645 op/s; 7.8 MiB/s, 124 objects/s recovering 2024-09-13T15:02:13.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:13 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2643143163' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:13.631 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:13.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:13.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:13.732 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:13.753 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:13.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:13.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:13.855 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:13.889 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:13.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:13.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:13.991 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:14.015 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:14.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:14.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:14.117 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:14.152 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:14.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:14.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:14.253 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:14.291 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:14.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:14.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:14.393 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:14.418 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:14.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:14.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:14.520 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:14.558 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:14.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:14.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:14.592 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:14 smithi151 ceph-mon[104450]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:02:14.592 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:14 smithi151 ceph-mon[104450]: Cluster is now healthy 2024-09-13T15:02:14.592 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:14 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2643143163' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:02:14.592 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:14 smithi151 ceph-mon[104450]: osdmap e1321: 8 total, 8 up, 6 in 2024-09-13T15:02:14.593 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:14 smithi151 ceph-mon[108966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:02:14.593 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:14 smithi151 ceph-mon[108966]: Cluster is now healthy 2024-09-13T15:02:14.593 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:14 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2643143163' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:02:14.593 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:14 smithi151 ceph-mon[108966]: osdmap e1321: 8 total, 8 up, 6 in 2024-09-13T15:02:14.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:14 smithi184 ceph-mon[92527]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:02:14.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:14 smithi184 ceph-mon[92527]: Cluster is now healthy 2024-09-13T15:02:14.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:14 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2643143163' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:02:14.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:14 smithi184 ceph-mon[92527]: osdmap e1321: 8 total, 8 up, 6 in 2024-09-13T15:02:14.660 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:14.687 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:14.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:14.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:14.789 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:14.811 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:14.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:14.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:14.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:02:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:02:14] "GET /metrics HTTP/1.1" 200 36734 "" "Prometheus/2.51.0" 2024-09-13T15:02:14.913 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:14.944 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:14.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:14.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:15.046 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:15.073 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:15.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:15.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:15.175 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:15.204 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:15.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:15.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:15.306 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:15.345 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:15.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:15.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:15.447 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:15.474 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:15.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:15.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:15.576 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:15.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:15 smithi184 ceph-mon[92527]: pgmap v2211: 119 pgs: 119 active+clean; 238 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 1.48k op/s; 7.0 MiB/s, 111 objects/s recovering 2024-09-13T15:02:15.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:15.608 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:15.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:15.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:15 smithi151 ceph-mon[104450]: pgmap v2211: 119 pgs: 119 active+clean; 238 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 1.48k op/s; 7.0 MiB/s, 111 objects/s recovering 2024-09-13T15:02:15.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:15 smithi151 ceph-mon[108966]: pgmap v2211: 119 pgs: 119 active+clean; 238 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 1.48k op/s; 7.0 MiB/s, 111 objects/s recovering 2024-09-13T15:02:15.708 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:15.736 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:15.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:15.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:15.838 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:15.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:15.886 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:15.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:15.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:16.018 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:16.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:16.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:16.036 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Removed 125610 objects 2024-09-13T15:02:16.036 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Clean up completed and total clean up time :148.824 2024-09-13T15:02:16.121 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:16.142 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:16.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:16.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:16.244 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:16.265 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:16.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:16.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:16.368 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:16.390 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:16.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:16.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:16.492 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:16.513 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:16.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:16.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:16.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:16.636 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:16.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:16.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:16.738 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:16.760 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:16.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:16.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:16.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:16.883 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:16.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:16.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:16.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:17.007 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:17.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:17.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:17.109 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:17.131 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:17.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:17.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:17.233 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:17.254 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:17.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:17.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:17.356 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:17.377 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:17.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:17.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:17.479 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:17.502 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:17.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:17.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:17.603 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:17.626 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:17.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:17.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:17.709 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:17 smithi151 ceph-mon[104450]: pgmap v2212: 119 pgs: 119 active+clean; 238 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 956 op/s; 2.3 MiB/s, 36 objects/s recovering 2024-09-13T15:02:17.709 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:02:17.710 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:17.710 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:17 smithi151 ceph-mon[108966]: pgmap v2212: 119 pgs: 119 active+clean; 238 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 956 op/s; 2.3 MiB/s, 36 objects/s recovering 2024-09-13T15:02:17.710 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:02:17.710 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:17.728 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:17.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:17.752 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:17.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:17.853 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:17 smithi184 ceph-mon[92527]: pgmap v2212: 119 pgs: 119 active+clean; 238 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 956 op/s; 2.3 MiB/s, 36 objects/s recovering 2024-09-13T15:02:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:02:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:17.876 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:17.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:17.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:17.977 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:17.999 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:17.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:18.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:18.078 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T15:02:18.078 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T15:02:18.078 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:02:18.078 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:02:18.090 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T15:02:18.091 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:02:18.101 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:18.107 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T15:02:18.107 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:02:18.122 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T15:02:18.123 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:02:18.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:18.124 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:18.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:18.138 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T15:02:18.138 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:02:18.153 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T15:02:18.153 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:02:18.168 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T15:02:18.168 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:02:18.225 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:18.246 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:18.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:18.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:18.267 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T15:02:18.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:18.371 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:18.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:18.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:18.472 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:18.494 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:18.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:18.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:18.596 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:18.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:18 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:02:18.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:18 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:02:18.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:18 smithi151 ceph-mon[104450]: pgmap v2213: 119 pgs: 119 active+clean; 81 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 1.05k op/s; 1.9 MiB/s, 30 objects/s recovering 2024-09-13T15:02:18.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:18 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:02:18.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:18 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:02:18.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:18 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:02:18.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:18 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2096712243' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:18.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:18 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:02:18.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:18 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:02:18.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:18 smithi151 ceph-mon[108966]: pgmap v2213: 119 pgs: 119 active+clean; 81 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 1.05k op/s; 1.9 MiB/s, 30 objects/s recovering 2024-09-13T15:02:18.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:18 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:02:18.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:18 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:02:18.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:18 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:02:18.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:18 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2096712243' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:18.619 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:18.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:18.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:18.721 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:18.742 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:18.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:18.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:18.844 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:18.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:18 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:02:18.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:18 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:02:18.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:18 smithi184 ceph-mon[92527]: pgmap v2213: 119 pgs: 119 active+clean; 81 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 1.05k op/s; 1.9 MiB/s, 30 objects/s recovering 2024-09-13T15:02:18.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:18 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:02:18.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:18 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:02:18.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:18 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:02:18.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:18 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2096712243' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:18.866 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:18.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:18.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:18.968 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:18.991 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:18.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:18.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:19.053 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:19.089 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:19.096 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:19.119 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:19.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:19.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:19.221 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:19.243 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:19.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:19.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:19.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:19.367 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:19.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:19.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:19.469 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:19.490 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:19.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:19.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:19.505 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T15:02:19.547 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:19 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3906438164' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:02:19.547 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:19 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:02:19.548 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:19 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3906438164' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:02:19.548 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:19 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:02:19.592 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:19.614 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:19.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:19.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:19.715 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:19.742 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:19.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:19.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:19.754 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:19 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3906438164' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:02:19.754 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:19 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:02:19.844 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:19.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:19.871 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:19.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:19.972 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:19.995 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:19.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:19.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:20.097 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:20.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:20.120 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:20.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:20.222 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:20.244 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:20.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:20.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:20.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:20.369 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:20.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:20.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:20.413 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:02:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:02:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:02:20.470 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:20.492 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:20.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:20.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:20.594 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:20.617 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:20.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:20.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:20.719 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:20.742 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:20.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:20.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:20.769 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:20 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:02:20.769 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:20 smithi151 ceph-mon[104450]: osdmap e1322: 8 total, 8 up, 6 in 2024-09-13T15:02:20.769 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:20 smithi151 ceph-mon[104450]: pgmap v2215: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 1.23k op/s 2024-09-13T15:02:20.769 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:20 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:02:20.769 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:20 smithi151 ceph-mon[108966]: osdmap e1322: 8 total, 8 up, 6 in 2024-09-13T15:02:20.769 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:20 smithi151 ceph-mon[108966]: pgmap v2215: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 1.23k op/s 2024-09-13T15:02:20.793 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T15:02:20.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:20.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:20 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:02:20.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:20 smithi184 ceph-mon[92527]: osdmap e1322: 8 total, 8 up, 6 in 2024-09-13T15:02:20.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:20 smithi184 ceph-mon[92527]: pgmap v2215: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 1.23k op/s 2024-09-13T15:02:20.898 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:20.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:20.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:21.000 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:21.021 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:21.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:21.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:21.031 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:02:21.115 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_5 2024-09-13T15:02:21.115 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_5 unique_pool_5 --yes-i-really-really-mean-it 2024-09-13T15:02:21.122 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:21.143 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:21.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:21.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:21.245 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:21.266 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:21.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:21.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:21.368 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:21.390 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:21.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:21.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:21.492 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:21.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:21.513 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:21.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:21.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:21.637 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:21.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:21.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:21.739 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:21.761 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:21.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:21.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:21.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:21.884 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:21.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:21.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:21.986 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:22.007 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:22.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:22.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:22.109 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:22.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:22.132 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:22.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:22.233 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:22.255 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:22.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:22.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:22.357 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:22.379 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:22.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:22.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:22.481 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:22.502 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:22.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:22.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:22.604 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:22.625 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:22.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:22.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:22.727 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:22.749 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:22.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:22.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:22.851 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:22.873 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:22.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:22.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:22.903 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-13T15:02:22.974 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:22.996 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:22.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:22.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:23.079 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:22 smithi151 ceph-mon[104450]: pgmap v2216: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 765 op/s 2024-09-13T15:02:23.079 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:22 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:02:23.079 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:22 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2962358570' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:02:23.079 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:22 smithi151 ceph-mon[108966]: pgmap v2216: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 765 op/s 2024-09-13T15:02:23.079 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:22 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:02:23.079 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:22 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2962358570' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:02:23.098 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:23.120 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:23.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:23.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:23.221 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:23.243 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:23.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:23.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:23.345 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:23.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:22 smithi184 ceph-mon[92527]: pgmap v2216: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 765 op/s 2024-09-13T15:02:23.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:22 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:02:23.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:22 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2962358570' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:02:23.367 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:23.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:23.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:23.468 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:23.490 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:23.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:23.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:23.592 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:23.614 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:23.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:23.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:23.716 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:23.738 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:23.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:23.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:23.840 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:23.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:23.884 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:23.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:23.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:23 smithi151 ceph-mon[104450]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:02:23.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:23 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2962358570' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:02:23.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:23 smithi151 ceph-mon[104450]: osdmap e1323: 8 total, 8 up, 6 in 2024-09-13T15:02:23.948 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:23 smithi151 ceph-mon[108966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:02:23.948 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:23 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2962358570' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:02:23.948 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:23 smithi151 ceph-mon[108966]: osdmap e1323: 8 total, 8 up, 6 in 2024-09-13T15:02:23.986 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:24.008 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:24.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:24.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:24.038 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:24.039 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:24.039 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:24.050 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T15:02:24.050 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-13T15:02:24.050 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-13T15:02:24.050 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T15:02:24.109 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:24.131 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:24.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:24.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:24.233 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:24.254 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:24.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:24.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:24.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:23 smithi184 ceph-mon[92527]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:02:24.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:23 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2962358570' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:02:24.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:23 smithi184 ceph-mon[92527]: osdmap e1323: 8 total, 8 up, 6 in 2024-09-13T15:02:24.355 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:24.377 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:24.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:24.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:24.479 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:24.500 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:24.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:24.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:24.602 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:24.625 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:24.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:24.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:24.727 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:24.750 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:24.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:24.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:24.794 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:02:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:02:24] "GET /metrics HTTP/1.1" 200 36724 "" "Prometheus/2.51.0" 2024-09-13T15:02:24.821 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T15:02:24.851 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:24.873 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:24.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:24.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:24.975 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:24.997 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:24.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:24.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:25.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:25.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:24 smithi151 ceph-mon[108966]: pgmap v2218: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 475 op/s 2024-09-13T15:02:25.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:24 smithi151 ceph-mon[104450]: pgmap v2218: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 475 op/s 2024-09-13T15:02:25.123 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:25.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:25.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:25.225 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:25.247 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:25.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:25.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:25.349 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:25.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:24 smithi184 ceph-mon[92527]: pgmap v2218: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 475 op/s 2024-09-13T15:02:25.370 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:25.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:25.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:25.472 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:25.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:25.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:25.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:25.595 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:25.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:25.618 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:25.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:25.719 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:25.721 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T15:02:25.742 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:25.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:25.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:25.844 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:25.866 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:25.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:25.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:25.968 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:25.989 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:25.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:25.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:26.091 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:26.114 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:26.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:26.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:26.215 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:26.241 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:26.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:26.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:26.343 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:26.364 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:26.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:26.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:26.466 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:26.488 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:26.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:26.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:26.589 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:26.611 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:26.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:26.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:26.712 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:26.734 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:26.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:26.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:26.836 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:26.857 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:26.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:26.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:26.959 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:26.983 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:26.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:26.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:27.085 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:27.090 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:26 smithi151 ceph-mon[104450]: pgmap v2219: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 160 op/s 2024-09-13T15:02:27.091 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:26 smithi151 ceph-mon[108966]: pgmap v2219: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 160 op/s 2024-09-13T15:02:27.109 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:27.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:27.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:27.211 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:27.236 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:27.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:27.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:27.338 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:27.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:26 smithi184 ceph-mon[92527]: pgmap v2219: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 160 op/s 2024-09-13T15:02:27.360 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:27.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:27.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:27.461 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:27.488 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:27.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:27.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:27.590 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:27.616 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:27.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:27.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:27.718 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:27.743 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:27.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:27.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:27.845 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:27.871 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:27.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:27.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:27.973 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:27.999 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:27.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:28.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:28.101 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:28.126 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:28.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:28.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:28.228 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:28.253 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:28.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:28.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:28.355 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:28.377 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:28.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:28.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:28.416 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:28.479 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:28.501 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:28.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:28.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:28.602 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:28.624 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:28.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:28.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:28.726 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:28.751 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:28.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:28.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:28.814 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:28.853 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:28.876 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:28.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:28.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:28.978 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:29.004 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:29.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:29.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:29.068 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:28 smithi151 ceph-mon[104450]: pgmap v2220: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:29.069 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:28 smithi151 ceph-mon[108966]: pgmap v2220: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:29.106 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:29.132 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:29.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:29.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:29.234 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:29.258 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:29.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:29.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:29.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:28 smithi184 ceph-mon[92527]: pgmap v2220: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:29.360 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:29.381 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:29.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:29.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:29.483 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:29.506 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:29.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:29.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:29.608 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:29.632 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:29.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:29.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:29.734 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:29.756 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:29.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:29.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:29.858 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:29.882 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:29.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:29.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:29.920 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T15:02:29.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:30.005 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:30.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:30.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:30.107 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:30.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:29 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2310552849' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:02:30.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:29 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:02:30.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:29 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2310552849' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:02:30.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:29 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:02:30.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:30.134 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:30.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:30.214 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:29 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2310552849' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:02:30.214 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:29 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:02:30.234 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:30.256 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:30.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:30.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:30.358 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:30.383 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:30.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:30.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:30.485 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:30.509 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:30.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:30.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:30.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:02:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:02:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:02:30.610 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:30.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:30.633 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:30.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:30.734 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:30.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:30.759 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:30.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:30.833 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T15:02:30.859 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:30.897 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:30.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:30.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:30.923 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_5' removed 2024-09-13T15:02:30.999 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:31.024 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:31.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:31.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:31.061 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:30 smithi151 ceph-mon[108966]: pgmap v2221: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:31.061 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:30 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:02:31.061 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:30 smithi151 ceph-mon[108966]: osdmap e1324: 8 total, 8 up, 6 in 2024-09-13T15:02:31.062 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:30 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1975268003' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T15:02:31.062 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:30 smithi151 ceph-mon[104450]: pgmap v2221: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:31.063 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:30 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:02:31.063 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:30 smithi151 ceph-mon[104450]: osdmap e1324: 8 total, 8 up, 6 in 2024-09-13T15:02:31.063 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:30 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1975268003' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T15:02:31.069 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T15:02:31.069 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:02:31.089 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T15:02:31.090 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:02:31.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:30 smithi184 ceph-mon[92527]: pgmap v2221: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:31.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:30 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:02:31.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:30 smithi184 ceph-mon[92527]: osdmap e1324: 8 total, 8 up, 6 in 2024-09-13T15:02:31.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:30 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1975268003' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T15:02:31.109 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T15:02:31.110 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:02:31.126 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:31.136 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T15:02:31.136 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:02:31.154 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T15:02:31.154 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:02:31.161 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:31.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:31.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:31.176 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T15:02:31.177 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:02:31.193 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T15:02:31.193 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:02:31.208 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T15:02:31.209 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:02:31.263 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:31.285 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:31.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:31.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:31.387 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:31.409 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:31.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:31.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:31.511 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:31.532 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:31.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:31.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:31.634 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:31.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:31.657 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:31.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:31.758 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:31.779 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:31.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:31.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:31.881 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:31.902 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:31.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:31.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:32.004 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:32.025 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:32.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:32.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:32.127 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:32.149 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:32.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:32.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:32.229 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:31 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1975268003' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T15:02:32.229 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:31 smithi151 ceph-mon[104450]: osdmap e1325: 8 total, 8 up, 6 in 2024-09-13T15:02:32.229 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:31 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3260096401' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:32.229 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:02:32.229 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:32.230 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:31 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1975268003' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T15:02:32.230 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:31 smithi151 ceph-mon[108966]: osdmap e1325: 8 total, 8 up, 6 in 2024-09-13T15:02:32.230 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:31 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3260096401' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:32.230 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:02:32.230 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:32.250 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:32.271 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:32.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:32.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:31 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1975268003' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T15:02:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:31 smithi184 ceph-mon[92527]: osdmap e1325: 8 total, 8 up, 6 in 2024-09-13T15:02:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:31 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3260096401' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:02:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:32.374 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:32.395 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:32.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:32.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:32.496 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:32.519 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:32.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:32.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:32.621 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:32.644 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:32.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:32.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:32.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:32.768 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:32.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:32.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:32.869 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:32.893 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:32.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:32.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:32.995 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:33.016 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:33.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:33.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:33.119 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:33.142 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:33.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:33.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:33.244 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:33.268 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:33.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:33.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:33.370 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:33.393 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:33.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:33.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:33.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:33.516 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:33.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:33.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:33.579 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:33 smithi151 ceph-mon[104450]: pgmap v2224: 103 pgs: 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:33.580 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:33 smithi151 ceph-mon[108966]: pgmap v2224: 103 pgs: 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:33.618 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:33.641 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:33.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:33.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:33.679 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T15:02:33.679 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:02:33.743 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:33.769 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:33.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:33.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:33.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:33 smithi184 ceph-mon[92527]: pgmap v2224: 103 pgs: 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:33.871 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:33.893 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:33.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:33.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:33.994 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:34.020 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:34.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:34.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:34.121 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:34.146 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:34.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:34.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:34.248 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:34.271 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:34.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:34.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:34.276 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T15:02:34.277 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T15:02:34.278 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_6 2024-09-13T15:02:34.278 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_6 16 2024-09-13T15:02:34.373 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:34.399 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:34.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:34.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:34.501 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:34.526 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:34.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:34.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:34.628 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:34.650 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:34.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:34.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:34.752 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:34.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:34.775 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:34.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:34.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:02:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:02:34] "GET /metrics HTTP/1.1" 200 34282 "" "Prometheus/2.51.0" 2024-09-13T15:02:34.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:34.898 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:34.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:34.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:34.999 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:35.022 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:35.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:35.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:35.124 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:35.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:35.147 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:35.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:35.248 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:35.269 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:35.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:35.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:35.371 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:35.393 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:35.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:35.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:35.494 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:35.514 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T15:02:35.520 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:35.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:35.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:35.622 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:35.649 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:35.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:35.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:35.751 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:35.773 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:35.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:35.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:35.782 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:35 smithi151 ceph-mon[104450]: pgmap v2225: 103 pgs: 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:35.783 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:35 smithi151 ceph-mon[108966]: pgmap v2225: 103 pgs: 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:35.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:35 smithi184 ceph-mon[92527]: pgmap v2225: 103 pgs: 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:35.875 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:35.897 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:35.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:35.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:35.999 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:36.025 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:36.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:36.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:36.043 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:36.070 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:36.089 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:36.126 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:36.148 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:36.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:36.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:36.250 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:36.276 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:36.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:36.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:36.381 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:36.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:36.405 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:36.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:36.443 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-13T15:02:36.506 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:36.529 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:36.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:36.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:36.631 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:36.652 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:36.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:36.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:36.754 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:36.775 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:36.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:36.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:36.829 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:36 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3700133809' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:36.830 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:36 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:36.830 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:36 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3700133809' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:36.830 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:36 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:36.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:36 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3700133809' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:36.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:36 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:36.877 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:36.899 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:36.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:36.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:37.001 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:37.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:37.025 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:37.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:37.077 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:02:37.126 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:37.148 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:37.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:37.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:37.250 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:37.272 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:37.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:37.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:37.374 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:37.395 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:37.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:37.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:37.496 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:37.518 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:37.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:37.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:37.620 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:37.642 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:37.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:37.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:37.666 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:37 smithi151 ceph-mon[104450]: pgmap v2226: 103 pgs: 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:37.666 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:37 smithi151 ceph-mon[104450]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:02:37.667 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:37 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:02:37.667 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:37 smithi151 ceph-mon[104450]: osdmap e1326: 8 total, 8 up, 6 in 2024-09-13T15:02:37.667 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:37 smithi151 ceph-mon[108966]: pgmap v2226: 103 pgs: 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:37.667 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:37 smithi151 ceph-mon[108966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:02:37.667 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:37 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:02:37.667 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:37 smithi151 ceph-mon[108966]: osdmap e1326: 8 total, 8 up, 6 in 2024-09-13T15:02:37.745 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:37.772 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:37.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:37.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:37.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:37 smithi184 ceph-mon[92527]: pgmap v2226: 103 pgs: 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:37.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:37 smithi184 ceph-mon[92527]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:02:37.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:37 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:02:37.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:37 smithi184 ceph-mon[92527]: osdmap e1326: 8 total, 8 up, 6 in 2024-09-13T15:02:37.874 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:37.895 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:37.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:37.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:37.997 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:38.019 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:38.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:38.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:38.121 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:38.142 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:38.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:38.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:38.244 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:38.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:38.266 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:38.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:38.367 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:38.393 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:38.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:38.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:38.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:38.517 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:38.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:38.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:38.563 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:38.619 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:38.641 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:38.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:38.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:38.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:38.765 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:38.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:38.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:38.867 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:38.888 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:38.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:38.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:38.990 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:39.014 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:39.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:39.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:39.116 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:39.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:39.142 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:39.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:39.205 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:02:39.243 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:39.270 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:39.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:39.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:39.371 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:39.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:39.396 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:39.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:39.497 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:39.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:39.520 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:39.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:39.617 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:39 smithi151 ceph-mon[104450]: pgmap v2228: 103 pgs: 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:39.617 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:39 smithi151 ceph-mon[108966]: pgmap v2228: 103 pgs: 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:39.621 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:39.641 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:39.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:39.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:39.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:39.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:39.766 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:39.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:39.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:39 smithi184 ceph-mon[92527]: pgmap v2228: 103 pgs: 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:39.867 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:39.889 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:39.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:39.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:39.991 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:40.017 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:40.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:40.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:40.084 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T15:02:40.084 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:02:40.119 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:40.141 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:40.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:40.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:40.243 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:40.269 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:40.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:40.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:40.371 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:40.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:40.397 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:40.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:40.490 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:02:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:02:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:02:40.498 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:40.522 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:40.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:40.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:40.624 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:40.646 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:40.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:40.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:40.748 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:40.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:40.770 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:40.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:40.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:40 smithi151 ceph-mon[104450]: pgmap v2229: 103 pgs: 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:40.841 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:40 smithi151 ceph-mon[108966]: pgmap v2229: 103 pgs: 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:40.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:40 smithi184 ceph-mon[92527]: pgmap v2229: 103 pgs: 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:40.871 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:40.896 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:40.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:40.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:40.998 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:41.022 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:41.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:41.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:41.124 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:41.147 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:41.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:41.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:41.249 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:41.271 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:41.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:41.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:41.373 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:41.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:41.399 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:41.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:41.500 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:41.522 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:41.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:41.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:41.624 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:41.649 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:41.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:41.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:41.751 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:41.772 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:41.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:41.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:41.868 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T15:02:41.868 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:41 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1423868854' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:02:41.868 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:41 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:02:41.869 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:41 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1423868854' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:02:41.869 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:41 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:02:41.873 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:41.902 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:41.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:41.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:42.004 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:42.028 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:42.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:42.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:42.130 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:42.153 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:42.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:42.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:42.255 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:42.279 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:42.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:42.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:42.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:41 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1423868854' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:02:42.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:41 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:02:42.381 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:42.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:42.404 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:42.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:42.505 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:42.529 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:42.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:42.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:42.631 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:42.653 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:42.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:42.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:42.704 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T15:02:42.755 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:42.776 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:42.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:42.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:42.878 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:42.912 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:42.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:42.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:42.933 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:42 smithi151 ceph-mon[108966]: pgmap v2230: 103 pgs: 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:42.933 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:42 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:02:42.934 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:42 smithi151 ceph-mon[108966]: osdmap e1327: 8 total, 8 up, 6 in 2024-09-13T15:02:42.934 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:42 smithi151 ceph-mon[104450]: pgmap v2230: 103 pgs: 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:42.934 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:42 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:02:42.934 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:42 smithi151 ceph-mon[104450]: osdmap e1327: 8 total, 8 up, 6 in 2024-09-13T15:02:43.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:43.037 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:43.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:43.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:43.139 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:43.161 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:43.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:43.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:43.263 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:43.284 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:43.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:43.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:43.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:42 smithi184 ceph-mon[92527]: pgmap v2230: 103 pgs: 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:43.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:42 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:02:43.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:42 smithi184 ceph-mon[92527]: osdmap e1327: 8 total, 8 up, 6 in 2024-09-13T15:02:43.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:43.408 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:43.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:43.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:43.510 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:43.532 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:43.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:43.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:43.634 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:43.658 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:43.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:43.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:43.760 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:43.781 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:43.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:43.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:43.882 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_6' created 2024-09-13T15:02:43.883 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:43.909 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:43.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:43.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:44.011 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:44.033 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:44.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:44.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:44.079 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:43 smithi151 ceph-mon[104450]: osdmap e1328: 8 total, 8 up, 6 in 2024-09-13T15:02:44.079 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:43 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/422309939' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16} : dispatch 2024-09-13T15:02:44.079 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:43 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16} : dispatch 2024-09-13T15:02:44.080 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:43 smithi151 ceph-mon[108966]: osdmap e1328: 8 total, 8 up, 6 in 2024-09-13T15:02:44.080 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:43 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/422309939' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16} : dispatch 2024-09-13T15:02:44.080 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:43 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16} : dispatch 2024-09-13T15:02:44.134 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:44.156 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:44.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:44.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:44.258 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:44.284 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:44.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:44.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:44.336 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:44.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:43 smithi184 ceph-mon[92527]: osdmap e1328: 8 total, 8 up, 6 in 2024-09-13T15:02:44.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:43 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/422309939' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16} : dispatch 2024-09-13T15:02:44.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:43 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16} : dispatch 2024-09-13T15:02:44.385 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:44.411 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:44.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:44.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:44.513 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:44.539 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:44.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:44.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:44.640 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:44.666 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:44.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:44.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:44.733 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:02:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:02:44] "GET /metrics HTTP/1.1" 200 34282 "" "Prometheus/2.51.0" 2024-09-13T15:02:44.758 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_6 rados --yes-i-really-mean-it || true 2024-09-13T15:02:44.768 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:44.821 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:44.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:44.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:44.923 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:44.949 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:44.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:44.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:45.025 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:44 smithi151 ceph-mon[104450]: pgmap v2233: 103 pgs: 39 peering, 64 active+clean; 577 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:45.025 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:44 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-09-13T15:02:45.025 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:44 smithi151 ceph-mon[104450]: osdmap e1329: 8 total, 8 up, 6 in 2024-09-13T15:02:45.025 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:44 smithi151 ceph-mon[108966]: pgmap v2233: 103 pgs: 39 peering, 64 active+clean; 577 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:45.025 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:44 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-09-13T15:02:45.025 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:44 smithi151 ceph-mon[108966]: osdmap e1329: 8 total, 8 up, 6 in 2024-09-13T15:02:45.051 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:45.072 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:45.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:45.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:45.174 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:45.200 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:45.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:45.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:45.302 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:45.323 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:45.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:45.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:45.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:44 smithi184 ceph-mon[92527]: pgmap v2233: 103 pgs: 39 peering, 64 active+clean; 577 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:45.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:44 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-09-13T15:02:45.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:44 smithi184 ceph-mon[92527]: osdmap e1329: 8 total, 8 up, 6 in 2024-09-13T15:02:45.425 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:45.452 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:45.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:45.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:45.493 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T15:02:45.554 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:45.577 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:45.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:45.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:45.679 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:45.702 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:45.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:45.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:45.754 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:02:45.804 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:45.825 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:45.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:45.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:45.927 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:45.949 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:45.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:45.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:46.051 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:46.072 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:46.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:46.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:46.081 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:45 smithi151 ceph-mon[104450]: osdmap e1330: 8 total, 8 up, 6 in 2024-09-13T15:02:46.081 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:45 smithi151 ceph-mon[108966]: osdmap e1330: 8 total, 8 up, 6 in 2024-09-13T15:02:46.109 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T15:02:46.109 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T15:02:46.109 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:02:46.109 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:02:46.114 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T15:02:46.114 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:02:46.131 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T15:02:46.132 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:02:46.147 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T15:02:46.148 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:02:46.163 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T15:02:46.163 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:02:46.174 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:46.179 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T15:02:46.179 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:02:46.195 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T15:02:46.195 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:02:46.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:46.196 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:46.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:46.204 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:45 smithi184 ceph-mon[92527]: osdmap e1330: 8 total, 8 up, 6 in 2024-09-13T15:02:46.297 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:46.319 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:46.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:46.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:46.421 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:46.443 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:46.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:46.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:46.545 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:46.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:46.567 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:46.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:46.668 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:46.690 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:46.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:46.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:46.792 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:46.813 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:46.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:46.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:46.915 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:46.936 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:46.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:46.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:46.992 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:46 smithi184 ceph-mon[92527]: pgmap v2236: 119 pgs: 16 unknown, 39 peering, 64 active+clean; 577 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:46.992 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:46 smithi184 ceph-mon[92527]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T15:02:46.992 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:46 smithi184 ceph-mon[92527]: osdmap e1331: 8 total, 8 up, 6 in 2024-09-13T15:02:46.992 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:46 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3201234132' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:46.992 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:02:46.993 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:47.038 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:47.061 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:47.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:47.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:47.075 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:46 smithi151 ceph-mon[104450]: pgmap v2236: 119 pgs: 16 unknown, 39 peering, 64 active+clean; 577 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:47.075 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:46 smithi151 ceph-mon[104450]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T15:02:47.075 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:46 smithi151 ceph-mon[104450]: osdmap e1331: 8 total, 8 up, 6 in 2024-09-13T15:02:47.075 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:46 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3201234132' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:47.075 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:02:47.076 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:47.076 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:46 smithi151 ceph-mon[108966]: pgmap v2236: 119 pgs: 16 unknown, 39 peering, 64 active+clean; 577 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:47.076 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:46 smithi151 ceph-mon[108966]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T15:02:47.076 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:46 smithi151 ceph-mon[108966]: osdmap e1331: 8 total, 8 up, 6 in 2024-09-13T15:02:47.076 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:46 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3201234132' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:47.076 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:02:47.076 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:47.163 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:47.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:47.185 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:47.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:47.287 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:47.308 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:47.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:47.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:47.410 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:47.432 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:47.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:47.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:47.470 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T15:02:47.470 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [], 2024-09-13T15:02:47.470 INFO:teuthology.orchestra.run.smithi184.stdout: "num_ops": 0 2024-09-13T15:02:47.470 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T15:02:47.533 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:47.544 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:47.544 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:47.556 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:47.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:47.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:47.658 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:47.679 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:47.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:47.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:47.781 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:47.802 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:47.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:47.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:47.904 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:47.926 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:47.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:47.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:48.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:48.049 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:48.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:48.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:48.089 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T15:02:48.150 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:48.172 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:48.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:48.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:48.273 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:48.296 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:48.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:48.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:48.335 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:02:48.398 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:48.419 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:48.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:48.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:48.521 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:48.543 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:48.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:48.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:48.645 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:48.667 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:48.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:48.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:48.769 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:48.794 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:48.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:48.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:48.896 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:48.917 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:48.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:48.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:49.019 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:49.041 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:49.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:49.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:49.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:02:49.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:48 smithi184 ceph-mon[92527]: pgmap v2238: 119 pgs: 8 unknown, 111 active+clean; 647 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 17 KiB/s rd, 43 KiB/s wr, 16 op/s; 67 KiB/s, 2 keys/s, 8 objects/s recovering 2024-09-13T15:02:49.143 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:49.164 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:49.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:49.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:49.219 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:48 smithi151 ceph-mon[104450]: pgmap v2238: 119 pgs: 8 unknown, 111 active+clean; 647 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 17 KiB/s rd, 43 KiB/s wr, 16 op/s; 67 KiB/s, 2 keys/s, 8 objects/s recovering 2024-09-13T15:02:49.219 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:48 smithi151 ceph-mon[108966]: pgmap v2238: 119 pgs: 8 unknown, 111 active+clean; 647 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 17 KiB/s rd, 43 KiB/s wr, 16 op/s; 67 KiB/s, 2 keys/s, 8 objects/s recovering 2024-09-13T15:02:49.266 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:49.288 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:49.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:49.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:49.390 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:49.411 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:49.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:49.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:49.513 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:49.534 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:49.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:49.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:49.635 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:49.657 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:49.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:49.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:49.759 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:49.781 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:49.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:49.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:49.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:49.906 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:49.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:49.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:49.996 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T15:02:49.996 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [], 2024-09-13T15:02:49.997 INFO:teuthology.orchestra.run.smithi184.stdout: "complaint_time": 30, 2024-09-13T15:02:49.997 INFO:teuthology.orchestra.run.smithi184.stdout: "num_blocked_ops": 0 2024-09-13T15:02:49.997 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T15:02:50.007 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:50.029 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:50.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:50.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:50.131 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:50.152 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:50.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:50.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:50.254 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:50.276 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:50.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:50.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:50.354 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:02:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:02:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:02:50.378 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:50.399 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:50.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:50.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:50.501 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:50.523 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:50.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:50.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:50.624 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:50.646 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:50.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:50.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:50.714 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T15:02:50.747 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:50.769 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:50.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:50.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:50.871 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:50.893 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:50.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:50.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:50.968 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:02:50.995 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:51.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:51.017 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:51.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:51.031 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:50 smithi151 ceph-mon[104450]: pgmap v2239: 119 pgs: 119 active+clean; 647 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 14 KiB/s rd, 35 KiB/s wr, 13 op/s; 56 KiB/s, 1 keys/s, 6 objects/s recovering 2024-09-13T15:02:51.032 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:50 smithi151 ceph-mon[108966]: pgmap v2239: 119 pgs: 119 active+clean; 647 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 14 KiB/s rd, 35 KiB/s wr, 13 op/s; 56 KiB/s, 1 keys/s, 6 objects/s recovering 2024-09-13T15:02:51.089 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:50 smithi184 ceph-mon[92527]: pgmap v2239: 119 pgs: 119 active+clean; 647 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 14 KiB/s rd, 35 KiB/s wr, 13 op/s; 56 KiB/s, 1 keys/s, 6 objects/s recovering 2024-09-13T15:02:51.118 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:51.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:51.141 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:51.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:51.242 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:51.264 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:51.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:51.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:51.366 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:51.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:51.388 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:51.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:51.489 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:51.513 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:51.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:51.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:51.532 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:51.616 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:51.638 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:51.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:51.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:51.740 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:51.761 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:51.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:51.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:51.863 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:51.884 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:51.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:51.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:51.900 INFO:teuthology.orchestra.run.smithi151.stderr:enabled application 'rados' on pool 'unique_pool_6' 2024-09-13T15:02:51.904 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-13T15:02:51.986 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:52.008 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:52.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:52.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:52.110 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:52.131 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:52.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:52.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:52.232 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:52.247 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:51 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/4136987205' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:02:52.248 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:51 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:02:52.248 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:51 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1592483778' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T15:02:52.248 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:51 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/4136987205' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:02:52.248 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:51 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:02:52.248 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:51 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1592483778' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T15:02:52.254 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:52.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:52.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:52.346 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:51 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/4136987205' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:02:52.346 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:51 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:02:52.347 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:51 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1592483778' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T15:02:52.356 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:52.377 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:52.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:52.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:52.479 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:52.500 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:52.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:52.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:52.602 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:52.625 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:52.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:52.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:52.726 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:52.748 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:52.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:52.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:52.852 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:52.863 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T15:02:52.863 INFO:teuthology.orchestra.run.smithi184.stdout: "size": 20, 2024-09-13T15:02:52.863 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 600, 2024-09-13T15:02:52.864 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [ 2024-09-13T15:02:52.864 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.864 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "MOSDPGPush(9.c 1234/1229 [PushOp(9:383b629e:::benchmark_data_smithi151_261954_object8007:head, version: 1225'537, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:383b629e:::benchmark_data_smithi151_261954_object8007:head@1225'537, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T15:02:52.864 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:55:17.970502+0000", 2024-09-13T15:02:52.864 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 454.889938761, 2024-09-13T15:02:52.864 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.21509233799999999, 2024-09-13T15:02:52.864 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:02:52.864 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T15:02:52.864 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:02:52.864 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.864 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:02:52.865 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:17.970502+0000", 2024-09-13T15:02:52.865 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.865 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.865 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.865 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:02:52.865 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:17.970502+0000", 2024-09-13T15:02:52.865 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.865 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.865 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.865 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:02:52.865 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:17.970501+0000", 2024-09-13T15:02:52.865 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:02:52.866 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.866 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.866 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:02:52.866 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:17.970545+0000", 2024-09-13T15:02:52.866 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4351000000000003e-05 2024-09-13T15:02:52.866 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.866 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.866 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:02:52.866 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:18.183178+0000", 2024-09-13T15:02:52.866 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.21263328000000001 2024-09-13T15:02:52.866 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.866 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.867 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:02:52.867 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:18.183203+0000", 2024-09-13T15:02:52.867 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.5021e-05 2024-09-13T15:02:52.867 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.867 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.867 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:02:52.867 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:18.183276+0000", 2024-09-13T15:02:52.867 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.3051000000000002e-05 2024-09-13T15:02:52.867 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.867 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.867 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:02:52.867 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:18.183285+0000", 2024-09-13T15:02:52.867 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.1789999999999997e-06 2024-09-13T15:02:52.867 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.867 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.868 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:02:52.868 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:18.185594+0000", 2024-09-13T15:02:52.868 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0023089360000000001 2024-09-13T15:02:52.868 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.868 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:02:52.868 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.868 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.868 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.868 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:66595 10.e 10:7dc86aed:::benchmark_data_smithi151_288181_object66594:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1275)", 2024-09-13T15:02:52.868 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:58:51.175004+0000", 2024-09-13T15:02:52.868 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 241.68543642099999, 2024-09-13T15:02:52.868 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.16272383100000001, 2024-09-13T15:02:52.868 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:02:52.869 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:52.869 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:02:52.869 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:02:52.869 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:02:52.869 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 66595 2024-09-13T15:02:52.869 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.869 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:02:52.869 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.869 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:02:52.869 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.175004+0000", 2024-09-13T15:02:52.869 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.869 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.869 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.870 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:02:52.870 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.175004+0000", 2024-09-13T15:02:52.870 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.870 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.870 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.870 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:02:52.870 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.175003+0000", 2024-09-13T15:02:52.870 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T15:02:52.870 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.870 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.870 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:02:52.870 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.175062+0000", 2024-09-13T15:02:52.870 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.9284999999999999e-05 2024-09-13T15:02:52.870 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.870 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.871 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:02:52.871 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.175077+0000", 2024-09-13T15:02:52.871 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4917e-05 2024-09-13T15:02:52.871 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.871 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.871 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:02:52.871 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.175093+0000", 2024-09-13T15:02:52.871 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6172e-05 2024-09-13T15:02:52.871 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.871 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.871 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:02:52.871 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.175133+0000", 2024-09-13T15:02:52.871 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9919000000000002e-05 2024-09-13T15:02:52.872 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.872 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.872 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:02:52.872 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.175274+0000", 2024-09-13T15:02:52.872 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014076200000000001 2024-09-13T15:02:52.872 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.872 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.872 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T15:02:52.872 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.175410+0000", 2024-09-13T15:02:52.872 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000136704 2024-09-13T15:02:52.872 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.872 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.872 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:02:52.872 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.178053+0000", 2024-09-13T15:02:52.873 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0026429219999999998 2024-09-13T15:02:52.873 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.873 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.873 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.873 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.178197+0000", 2024-09-13T15:02:52.873 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000143917 2024-09-13T15:02:52.873 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.873 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.873 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.873 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.337669+0000", 2024-09-13T15:02:52.873 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.15947213599999999 2024-09-13T15:02:52.873 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.874 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.874 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:02:52.874 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.337704+0000", 2024-09-13T15:02:52.874 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.4783000000000002e-05 2024-09-13T15:02:52.874 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.874 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.874 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:02:52.874 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:58:51.337728+0000", 2024-09-13T15:02:52.874 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.4049999999999998e-05 2024-09-13T15:02:52.874 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.874 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:02:52.874 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.874 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.875 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.875 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:99184 10.1 10:8b2f0f40:::benchmark_data_smithi151_288181_object99183:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1277)", 2024-09-13T15:02:52.875 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:59:19.844687+0000", 2024-09-13T15:02:52.875 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 213.01575349999999, 2024-09-13T15:02:52.875 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.26746497000000002, 2024-09-13T15:02:52.875 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:02:52.875 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:52.875 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:02:52.875 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:02:52.875 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:02:52.875 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 99184 2024-09-13T15:02:52.875 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.875 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:02:52.876 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.876 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:02:52.876 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.844687+0000", 2024-09-13T15:02:52.876 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.876 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.876 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.876 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:02:52.876 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.844687+0000", 2024-09-13T15:02:52.876 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.876 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.876 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.876 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:02:52.877 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.844685+0000", 2024-09-13T15:02:52.877 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T15:02:52.877 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.877 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.877 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:02:52.877 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.844765+0000", 2024-09-13T15:02:52.877 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.9411000000000003e-05 2024-09-13T15:02:52.877 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.877 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.877 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:02:52.877 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.844792+0000", 2024-09-13T15:02:52.877 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.7487e-05 2024-09-13T15:02:52.878 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.878 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.878 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:02:52.878 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.844804+0000", 2024-09-13T15:02:52.878 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1809e-05 2024-09-13T15:02:52.878 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.878 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.878 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:02:52.878 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.844883+0000", 2024-09-13T15:02:52.878 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.9464999999999999e-05 2024-09-13T15:02:52.878 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.878 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.878 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:02:52.879 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.844984+0000", 2024-09-13T15:02:52.879 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000100375 2024-09-13T15:02:52.879 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.879 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.879 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T15:02:52.879 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.845075+0000", 2024-09-13T15:02:52.879 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.1324999999999997e-05 2024-09-13T15:02:52.879 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.879 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.879 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.879 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.851790+0000", 2024-09-13T15:02:52.879 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0067150109999999999 2024-09-13T15:02:52.879 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.880 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.880 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:02:52.880 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.851797+0000", 2024-09-13T15:02:52.880 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7630000000000003e-06 2024-09-13T15:02:52.880 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.880 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.880 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.880 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:20.112026+0000", 2024-09-13T15:02:52.880 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.26022897499999997 2024-09-13T15:02:52.880 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.880 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.880 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:02:52.880 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:20.112109+0000", 2024-09-13T15:02:52.880 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.2806000000000006e-05 2024-09-13T15:02:52.881 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.881 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.881 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:02:52.881 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:20.112152+0000", 2024-09-13T15:02:52.881 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3639e-05 2024-09-13T15:02:52.881 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.881 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:02:52.881 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.881 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.881 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.881 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:99185 10.1 10:8898143d:::benchmark_data_smithi151_288181_object99184:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1277)", 2024-09-13T15:02:52.881 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:59:19.844830+0000", 2024-09-13T15:02:52.881 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 213.01561058600001, 2024-09-13T15:02:52.881 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.26751710200000001, 2024-09-13T15:02:52.882 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:02:52.882 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:52.882 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:02:52.882 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:02:52.882 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:02:52.882 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 99185 2024-09-13T15:02:52.882 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.882 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:02:52.882 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.882 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:02:52.882 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.844830+0000", 2024-09-13T15:02:52.882 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.882 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.883 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.883 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:02:52.883 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.844830+0000", 2024-09-13T15:02:52.883 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.883 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.883 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:02:52.883 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.844830+0000", 2024-09-13T15:02:52.883 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T15:02:52.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.883 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.883 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:02:52.883 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.844860+0000", 2024-09-13T15:02:52.883 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.9869000000000002e-05 2024-09-13T15:02:52.884 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.884 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.884 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:02:52.884 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.844869+0000", 2024-09-13T15:02:52.884 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.3419999999999997e-06 2024-09-13T15:02:52.884 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.884 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.884 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:02:52.884 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.844888+0000", 2024-09-13T15:02:52.884 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.9638000000000001e-05 2024-09-13T15:02:52.884 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.884 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.884 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:02:52.885 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.847473+0000", 2024-09-13T15:02:52.885 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002584041 2024-09-13T15:02:52.885 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.885 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.885 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:02:52.885 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.847599+0000", 2024-09-13T15:02:52.885 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012684 2024-09-13T15:02:52.885 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.885 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.885 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T15:02:52.885 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.847702+0000", 2024-09-13T15:02:52.885 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000102345 2024-09-13T15:02:52.886 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.886 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.886 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:02:52.886 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.851805+0000", 2024-09-13T15:02:52.886 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0041036629999999996 2024-09-13T15:02:52.886 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.886 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.886 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.886 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.856192+0000", 2024-09-13T15:02:52.886 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0043864220000000001 2024-09-13T15:02:52.886 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.886 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.886 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.886 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:20.112251+0000", 2024-09-13T15:02:52.887 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.25605916899999998 2024-09-13T15:02:52.887 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.887 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.887 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:02:52.887 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:20.112301+0000", 2024-09-13T15:02:52.887 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9861999999999998e-05 2024-09-13T15:02:52.887 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.887 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.887 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:02:52.887 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:20.112347+0000", 2024-09-13T15:02:52.887 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.6474000000000003e-05 2024-09-13T15:02:52.887 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.887 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:02:52.887 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.888 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.888 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.888 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:99189 10.1 10:8b20aa5b:::benchmark_data_smithi151_288181_object99188:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1277)", 2024-09-13T15:02:52.888 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:59:19.845761+0000", 2024-09-13T15:02:52.888 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 213.01467993399999, 2024-09-13T15:02:52.888 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.26711996900000001, 2024-09-13T15:02:52.888 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:02:52.888 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:52.888 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:02:52.888 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:02:52.888 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:02:52.888 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 99189 2024-09-13T15:02:52.888 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.888 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:02:52.889 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.889 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:02:52.889 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.845761+0000", 2024-09-13T15:02:52.889 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.889 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.889 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.889 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:02:52.889 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.845761+0000", 2024-09-13T15:02:52.889 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.889 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.889 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.889 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:02:52.890 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.845760+0000", 2024-09-13T15:02:52.890 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:02:52.890 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.890 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.890 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:02:52.890 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.845815+0000", 2024-09-13T15:02:52.890 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4549999999999998e-05 2024-09-13T15:02:52.890 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.890 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.890 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:02:52.890 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.845828+0000", 2024-09-13T15:02:52.890 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3315999999999999e-05 2024-09-13T15:02:52.891 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.891 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.891 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:02:52.891 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.845833+0000", 2024-09-13T15:02:52.891 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0889999999999999e-06 2024-09-13T15:02:52.891 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.891 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.891 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:02:52.891 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.849719+0000", 2024-09-13T15:02:52.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:52.893 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:52.893 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00388565 2024-09-13T15:02:52.893 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.894 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.894 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:02:52.894 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.849814+0000", 2024-09-13T15:02:52.894 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.5334999999999994e-05 2024-09-13T15:02:52.894 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.894 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.894 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,7", 2024-09-13T15:02:52.894 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.849915+0000", 2024-09-13T15:02:52.894 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000100572 2024-09-13T15:02:52.894 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.894 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.894 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:02:52.894 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.851912+0000", 2024-09-13T15:02:52.894 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0019973920000000002 2024-09-13T15:02:52.895 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.895 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.895 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.895 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:19.856371+0000", 2024-09-13T15:02:52.895 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0044589970000000001 2024-09-13T15:02:52.895 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.895 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.895 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.895 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:20.112782+0000", 2024-09-13T15:02:52.895 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.25641142500000003 2024-09-13T15:02:52.895 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.895 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.895 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:02:52.895 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:20.112838+0000", 2024-09-13T15:02:52.896 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5651e-05 2024-09-13T15:02:52.896 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.896 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.896 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:02:52.896 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:59:20.112881+0000", 2024-09-13T15:02:52.896 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.2753999999999998e-05 2024-09-13T15:02:52.896 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.896 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:02:52.896 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.896 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.896 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.896 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:158524 10.d 10:b0c333ce:::benchmark_data_smithi151_288181_object32911:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1287)", 2024-09-13T15:02:52.896 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:00:24.465167+0000", 2024-09-13T15:02:52.896 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 148.39527369199999, 2024-09-13T15:02:52.897 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.72907941899999995, 2024-09-13T15:02:52.897 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:02:52.897 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "reached pg", 2024-09-13T15:02:52.897 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:02:52.897 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:02:52.897 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:02:52.897 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 158524 2024-09-13T15:02:52.897 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.897 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:02:52.897 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.897 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:02:52.897 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.465167+0000", 2024-09-13T15:02:52.898 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.898 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.898 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.898 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:02:52.898 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.465167+0000", 2024-09-13T15:02:52.898 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.898 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.898 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.898 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:02:52.898 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.465166+0000", 2024-09-13T15:02:52.898 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:02:52.898 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.898 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.898 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:02:52.899 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.465171+0000", 2024-09-13T15:02:52.899 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0420000000000002e-06 2024-09-13T15:02:52.899 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.899 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.899 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:02:52.899 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.465181+0000", 2024-09-13T15:02:52.899 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0103e-05 2024-09-13T15:02:52.899 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.899 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.899 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:02:52.899 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.465186+0000", 2024-09-13T15:02:52.899 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3599999999999998e-06 2024-09-13T15:02:52.899 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.900 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.900 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:02:52.900 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.465216+0000", 2024-09-13T15:02:52.900 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.0139000000000001e-05 2024-09-13T15:02:52.900 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.900 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.900 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:02:52.900 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.465302+0000", 2024-09-13T15:02:52.900 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.6038999999999994e-05 2024-09-13T15:02:52.900 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.900 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.900 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,3,6", 2024-09-13T15:02:52.901 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.465394+0000", 2024-09-13T15:02:52.901 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.1889000000000001e-05 2024-09-13T15:02:52.901 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.901 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.901 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:02:52.901 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.467506+0000", 2024-09-13T15:02:52.901 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0021125290000000001 2024-09-13T15:02:52.901 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.901 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.901 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.901 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.467699+0000", 2024-09-13T15:02:52.901 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000192144 2024-09-13T15:02:52.901 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.902 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.902 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.902 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.478323+0000", 2024-09-13T15:02:52.902 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.010624445999999999 2024-09-13T15:02:52.902 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.902 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.902 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:02:52.902 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.505071+0000", 2024-09-13T15:02:52.902 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.026747736000000001 2024-09-13T15:02:52.902 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.902 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.902 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:02:52.902 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.505081+0000", 2024-09-13T15:02:52.903 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0010000000000001e-05 2024-09-13T15:02:52.903 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.903 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.903 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:02:52.903 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.194158+0000", 2024-09-13T15:02:52.903 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.68907688600000006 2024-09-13T15:02:52.903 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.903 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.903 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:02:52.903 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.194247+0000", 2024-09-13T15:02:52.903 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.8832000000000006e-05 2024-09-13T15:02:52.903 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.903 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:02:52.904 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.904 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.904 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.904 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:158533 10.c 10:3c7de395:::benchmark_data_smithi151_288181_object32920:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1287)", 2024-09-13T15:02:52.904 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:00:24.478129+0000", 2024-09-13T15:02:52.904 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 148.38231170099999, 2024-09-13T15:02:52.904 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.72488107400000001, 2024-09-13T15:02:52.904 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:02:52.904 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "reached pg", 2024-09-13T15:02:52.904 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:02:52.904 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:02:52.904 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:02:52.904 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 158533 2024-09-13T15:02:52.905 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.905 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:02:52.905 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.905 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:02:52.905 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.478129+0000", 2024-09-13T15:02:52.905 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.905 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.905 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.905 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:02:52.905 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.478129+0000", 2024-09-13T15:02:52.905 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.905 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.905 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.906 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:02:52.906 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.478127+0000", 2024-09-13T15:02:52.906 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999976 2024-09-13T15:02:52.906 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.906 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.906 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:02:52.906 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.478135+0000", 2024-09-13T15:02:52.906 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.4290000000000001e-06 2024-09-13T15:02:52.906 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.906 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.906 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:02:52.906 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.478149+0000", 2024-09-13T15:02:52.906 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3230999999999999e-05 2024-09-13T15:02:52.907 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.907 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.907 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:02:52.907 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.478155+0000", 2024-09-13T15:02:52.907 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.2009999999999996e-06 2024-09-13T15:02:52.907 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.907 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.907 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:02:52.907 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.478563+0000", 2024-09-13T15:02:52.907 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00040817099999999998 2024-09-13T15:02:52.907 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.907 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.907 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:02:52.907 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.478695+0000", 2024-09-13T15:02:52.908 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000132399 2024-09-13T15:02:52.908 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.908 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.908 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,3,6,7", 2024-09-13T15:02:52.908 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.478826+0000", 2024-09-13T15:02:52.908 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013099699999999999 2024-09-13T15:02:52.908 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.908 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.908 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:02:52.908 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.481775+0000", 2024-09-13T15:02:52.908 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002948502 2024-09-13T15:02:52.908 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.909 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.909 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.909 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.483348+0000", 2024-09-13T15:02:52.909 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001573103 2024-09-13T15:02:52.909 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.909 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.909 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.909 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.483404+0000", 2024-09-13T15:02:52.909 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6464e-05 2024-09-13T15:02:52.909 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.909 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.909 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:02:52.909 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.511250+0000", 2024-09-13T15:02:52.910 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.027845758000000002 2024-09-13T15:02:52.910 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.910 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.910 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:02:52.910 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.511258+0000", 2024-09-13T15:02:52.910 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.2409999999999995e-06 2024-09-13T15:02:52.910 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.910 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.910 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:02:52.910 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.202885+0000", 2024-09-13T15:02:52.910 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.69162622900000004 2024-09-13T15:02:52.910 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.910 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.910 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:02:52.911 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.203010+0000", 2024-09-13T15:02:52.911 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000125547 2024-09-13T15:02:52.911 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.911 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:02:52.911 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.911 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.911 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.911 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:158534 10.c 10:3ea9c663:::benchmark_data_smithi151_288181_object32921:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1287)", 2024-09-13T15:02:52.911 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:00:24.483557+0000", 2024-09-13T15:02:52.911 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 148.37688328799999, 2024-09-13T15:02:52.911 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.72024967500000003, 2024-09-13T15:02:52.911 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:02:52.911 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "reached pg", 2024-09-13T15:02:52.912 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:02:52.912 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:02:52.912 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:02:52.912 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 158534 2024-09-13T15:02:52.912 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.912 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:02:52.912 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.912 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:02:52.912 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.483557+0000", 2024-09-13T15:02:52.912 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.912 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.912 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.912 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:02:52.912 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.483557+0000", 2024-09-13T15:02:52.912 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.913 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.913 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.913 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:02:52.913 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.483556+0000", 2024-09-13T15:02:52.913 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:02:52.913 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.913 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.913 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:02:52.913 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.483564+0000", 2024-09-13T15:02:52.913 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.5549999999999997e-06 2024-09-13T15:02:52.913 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.913 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.913 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:02:52.914 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.483576+0000", 2024-09-13T15:02:52.914 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2044e-05 2024-09-13T15:02:52.914 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.914 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.914 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:02:52.914 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.483581+0000", 2024-09-13T15:02:52.914 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5849999999999999e-06 2024-09-13T15:02:52.914 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.914 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.914 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:02:52.914 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.483625+0000", 2024-09-13T15:02:52.914 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3581999999999997e-05 2024-09-13T15:02:52.914 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.914 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.915 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:02:52.915 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.483757+0000", 2024-09-13T15:02:52.915 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013194699999999999 2024-09-13T15:02:52.915 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.915 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.915 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,3,6,7", 2024-09-13T15:02:52.915 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.483902+0000", 2024-09-13T15:02:52.915 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014577599999999999 2024-09-13T15:02:52.915 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.915 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.915 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:02:52.915 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.490186+0000", 2024-09-13T15:02:52.915 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0062834229999999998 2024-09-13T15:02:52.915 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.915 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.916 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:02:52.916 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.511406+0000", 2024-09-13T15:02:52.916 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.021220145999999999 2024-09-13T15:02:52.916 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.916 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.916 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:02:52.916 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.511411+0000", 2024-09-13T15:02:52.916 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7299999999999996e-06 2024-09-13T15:02:52.916 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.916 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.916 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:02:52.916 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.203715+0000", 2024-09-13T15:02:52.916 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.69230400599999997 2024-09-13T15:02:52.916 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.916 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.917 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:02:52.917 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.203807+0000", 2024-09-13T15:02:52.917 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.2367999999999994e-05 2024-09-13T15:02:52.917 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.917 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:02:52.917 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.917 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.917 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.917 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:158542 10.4 10.c93ed74 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1288)", 2024-09-13T15:02:52.917 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:00:24.501710+0000", 2024-09-13T15:02:52.917 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 148.358731138, 2024-09-13T15:02:52.917 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.52890606299999998, 2024-09-13T15:02:52.917 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:02:52.917 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "reached pg", 2024-09-13T15:02:52.918 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:02:52.918 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:02:52.918 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:02:52.918 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 158542 2024-09-13T15:02:52.918 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.918 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:02:52.918 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.918 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:02:52.918 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.501710+0000", 2024-09-13T15:02:52.918 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.918 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.918 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.918 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:02:52.919 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.501710+0000", 2024-09-13T15:02:52.919 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.919 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.919 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.919 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:02:52.919 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.501708+0000", 2024-09-13T15:02:52.919 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:02:52.919 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.919 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.919 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:02:52.919 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.501716+0000", 2024-09-13T15:02:52.919 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.4510000000000003e-06 2024-09-13T15:02:52.919 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.919 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.920 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:02:52.920 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.501728+0000", 2024-09-13T15:02:52.920 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1849e-05 2024-09-13T15:02:52.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.920 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.920 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:02:52.920 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.501734+0000", 2024-09-13T15:02:52.920 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3770000000000001e-06 2024-09-13T15:02:52.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.920 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.920 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:02:52.920 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.511308+0000", 2024-09-13T15:02:52.920 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.009574384 2024-09-13T15:02:52.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.920 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.921 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:02:52.921 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.511316+0000", 2024-09-13T15:02:52.921 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.5270000000000001e-06 2024-09-13T15:02:52.921 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.921 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.921 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:02:52.921 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.030607+0000", 2024-09-13T15:02:52.921 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.51929138399999997 2024-09-13T15:02:52.921 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.921 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.921 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:02:52.921 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.030616+0000", 2024-09-13T15:02:52.921 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.5290000000000008e-06 2024-09-13T15:02:52.921 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.921 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:02:52.922 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.922 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.922 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.922 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:158544 10.c 10:3fb067e8:::benchmark_data_smithi151_288181_object32931:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1288)", 2024-09-13T15:02:52.922 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:00:24.512691+0000", 2024-09-13T15:02:52.922 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 148.347749631, 2024-09-13T15:02:52.922 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.70246719700000004, 2024-09-13T15:02:52.922 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:02:52.922 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:52.922 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:02:52.922 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:02:52.922 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:02:52.922 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 158544 2024-09-13T15:02:52.922 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.923 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:02:52.923 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.923 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:02:52.923 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.512691+0000", 2024-09-13T15:02:52.923 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.923 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.923 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.923 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:02:52.923 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.512691+0000", 2024-09-13T15:02:52.923 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.923 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.923 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.923 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:02:52.923 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.512690+0000", 2024-09-13T15:02:52.924 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:02:52.924 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.924 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.924 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:02:52.924 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.512697+0000", 2024-09-13T15:02:52.924 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.9029999999999998e-06 2024-09-13T15:02:52.924 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.924 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.924 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:02:52.924 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.512841+0000", 2024-09-13T15:02:52.924 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000144754 2024-09-13T15:02:52.924 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.924 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.925 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:02:52.925 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.512849+0000", 2024-09-13T15:02:52.925 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.7950000000000008e-06 2024-09-13T15:02:52.925 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.925 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.925 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:02:52.925 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.512926+0000", 2024-09-13T15:02:52.925 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.7317000000000005e-05 2024-09-13T15:02:52.925 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.925 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.925 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:02:52.925 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.512933+0000", 2024-09-13T15:02:52.925 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7050000000000002e-06 2024-09-13T15:02:52.926 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.926 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.926 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:02:52.926 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.203829+0000", 2024-09-13T15:02:52.926 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.69089614700000002 2024-09-13T15:02:52.926 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.926 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.926 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:02:52.926 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.203982+0000", 2024-09-13T15:02:52.926 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00015239799999999999 2024-09-13T15:02:52.926 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.926 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.926 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,1,3,6,7", 2024-09-13T15:02:52.927 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.204116+0000", 2024-09-13T15:02:52.927 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013400799999999999 2024-09-13T15:02:52.927 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.927 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.927 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:02:52.927 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.210291+0000", 2024-09-13T15:02:52.927 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0061756010000000002 2024-09-13T15:02:52.927 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.927 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.927 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.927 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.210357+0000", 2024-09-13T15:02:52.927 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.5915999999999997e-05 2024-09-13T15:02:52.927 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.928 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.928 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.928 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.210788+0000", 2024-09-13T15:02:52.928 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00043070700000000001 2024-09-13T15:02:52.928 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.928 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.928 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.928 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.211189+0000", 2024-09-13T15:02:52.928 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00040090699999999999 2024-09-13T15:02:52.928 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.928 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.928 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.928 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.212915+0000", 2024-09-13T15:02:52.929 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001726333 2024-09-13T15:02:52.929 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.929 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.929 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.929 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.214922+0000", 2024-09-13T15:02:52.929 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0020072269999999999 2024-09-13T15:02:52.929 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.929 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.929 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:02:52.929 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.215022+0000", 2024-09-13T15:02:52.929 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.9653000000000003e-05 2024-09-13T15:02:52.929 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.929 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.929 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:02:52.930 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.215158+0000", 2024-09-13T15:02:52.930 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000136339 2024-09-13T15:02:52.930 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.930 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:02:52.930 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.930 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.930 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.930 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:158550 10.1 10.13a906d1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1288)", 2024-09-13T15:02:52.930 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:00:24.527060+0000", 2024-09-13T15:02:52.930 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 148.33338107, 2024-09-13T15:02:52.930 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.50271600000000005, 2024-09-13T15:02:52.930 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:02:52.930 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "reached pg", 2024-09-13T15:02:52.930 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:02:52.930 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:02:52.931 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:02:52.931 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 158550 2024-09-13T15:02:52.931 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.931 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:02:52.931 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.931 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:02:52.931 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.527060+0000", 2024-09-13T15:02:52.931 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.931 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.931 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.931 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:02:52.931 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.527060+0000", 2024-09-13T15:02:52.931 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.931 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.931 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.932 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:02:52.932 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.527059+0000", 2024-09-13T15:02:52.932 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:02:52.932 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.932 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.932 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:02:52.932 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.527064+0000", 2024-09-13T15:02:52.932 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1950000000000002e-06 2024-09-13T15:02:52.932 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.932 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.932 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:02:52.932 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.527074+0000", 2024-09-13T15:02:52.932 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0054000000000001e-05 2024-09-13T15:02:52.932 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.932 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.933 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:02:52.933 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.527079+0000", 2024-09-13T15:02:52.933 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.006e-06 2024-09-13T15:02:52.933 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.933 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.933 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:02:52.933 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.527125+0000", 2024-09-13T15:02:52.933 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5677000000000003e-05 2024-09-13T15:02:52.933 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.933 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.933 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:02:52.933 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.527133+0000", 2024-09-13T15:02:52.933 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0330000000000008e-06 2024-09-13T15:02:52.933 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.933 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.934 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:02:52.934 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.029761+0000", 2024-09-13T15:02:52.934 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.50262857299999997 2024-09-13T15:02:52.934 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.934 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.934 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:02:52.934 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.029776+0000", 2024-09-13T15:02:52.934 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4489e-05 2024-09-13T15:02:52.934 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.934 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:02:52.934 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.934 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.934 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.934 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:158552 10.c 10:3b8e0666:::benchmark_data_smithi151_288181_object32939:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1288)", 2024-09-13T15:02:52.934 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:00:24.535808+0000", 2024-09-13T15:02:52.935 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 148.324632756, 2024-09-13T15:02:52.935 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.67969992499999998, 2024-09-13T15:02:52.935 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:02:52.935 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:52.935 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:02:52.935 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:02:52.935 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:02:52.935 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 158552 2024-09-13T15:02:52.935 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.935 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:02:52.935 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.935 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:02:52.935 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.535808+0000", 2024-09-13T15:02:52.936 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.936 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.936 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.936 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:02:52.936 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.535808+0000", 2024-09-13T15:02:52.936 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.936 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.936 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.936 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:02:52.936 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.535807+0000", 2024-09-13T15:02:52.936 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:02:52.936 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.936 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.936 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:02:52.937 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.535813+0000", 2024-09-13T15:02:52.937 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.93e-06 2024-09-13T15:02:52.937 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.937 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.937 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:02:52.937 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.535825+0000", 2024-09-13T15:02:52.937 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.222e-05 2024-09-13T15:02:52.937 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.937 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.937 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:02:52.937 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.535830+0000", 2024-09-13T15:02:52.937 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0329999999999999e-06 2024-09-13T15:02:52.937 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.937 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.937 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:02:52.938 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.535870+0000", 2024-09-13T15:02:52.938 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0028999999999999e-05 2024-09-13T15:02:52.938 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.938 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.938 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:02:52.938 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.535885+0000", 2024-09-13T15:02:52.938 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.5089000000000001e-05 2024-09-13T15:02:52.938 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.938 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.938 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:02:52.938 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.207227+0000", 2024-09-13T15:02:52.938 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.67134198499999997 2024-09-13T15:02:52.939 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.939 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.939 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:02:52.939 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.207392+0000", 2024-09-13T15:02:52.939 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00016508399999999999 2024-09-13T15:02:52.939 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.939 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.939 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,1,3,6,7", 2024-09-13T15:02:52.939 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.207595+0000", 2024-09-13T15:02:52.939 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00020257900000000001 2024-09-13T15:02:52.939 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.939 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.940 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:02:52.940 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.210414+0000", 2024-09-13T15:02:52.940 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0028196570000000002 2024-09-13T15:02:52.940 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.940 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.940 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.940 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.210816+0000", 2024-09-13T15:02:52.940 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00040130800000000001 2024-09-13T15:02:52.940 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.940 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.940 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.940 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.211320+0000", 2024-09-13T15:02:52.940 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000504475 2024-09-13T15:02:52.940 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.941 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.941 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.941 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.213156+0000", 2024-09-13T15:02:52.941 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0018360659999999999 2024-09-13T15:02:52.941 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.941 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.941 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.941 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.213737+0000", 2024-09-13T15:02:52.941 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00058048099999999999 2024-09-13T15:02:52.941 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.941 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.941 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.942 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.215343+0000", 2024-09-13T15:02:52.942 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001605839 2024-09-13T15:02:52.942 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.942 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.942 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:02:52.942 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.215387+0000", 2024-09-13T15:02:52.942 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4573999999999997e-05 2024-09-13T15:02:52.942 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.942 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.942 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:02:52.942 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.215508+0000", 2024-09-13T15:02:52.942 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012079299999999999 2024-09-13T15:02:52.942 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.943 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:02:52.943 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.943 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.943 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.943 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:193304 10.d 10:b48a3fa4:::benchmark_data_smithi151_288181_object67691:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1292)", 2024-09-13T15:02:52.943 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:01:04.924699+0000", 2024-09-13T15:02:52.943 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 107.93574228200001, 2024-09-13T15:02:52.943 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.31689917499999998, 2024-09-13T15:02:52.943 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:02:52.943 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:52.943 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:02:52.943 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:02:52.943 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:02:52.944 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 193304 2024-09-13T15:02:52.944 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.944 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:02:52.944 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.944 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:02:52.944 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.924699+0000", 2024-09-13T15:02:52.944 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.944 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.944 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.944 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:02:52.944 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.924699+0000", 2024-09-13T15:02:52.944 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.944 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.945 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.945 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:02:52.945 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.924697+0000", 2024-09-13T15:02:52.945 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:02:52.945 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.945 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.945 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:02:52.945 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.924704+0000", 2024-09-13T15:02:52.945 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.297e-06 2024-09-13T15:02:52.945 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.945 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.945 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:02:52.945 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.924716+0000", 2024-09-13T15:02:52.946 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2034e-05 2024-09-13T15:02:52.946 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.946 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.946 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:02:52.946 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.924722+0000", 2024-09-13T15:02:52.946 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.2689999999999996e-06 2024-09-13T15:02:52.946 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.946 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.946 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:02:52.946 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.924779+0000", 2024-09-13T15:02:52.946 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6854999999999997e-05 2024-09-13T15:02:52.946 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.946 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.946 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:02:52.947 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.924936+0000", 2024-09-13T15:02:52.947 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000157515 2024-09-13T15:02:52.947 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.947 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.947 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,3,6,7", 2024-09-13T15:02:52.947 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.925084+0000", 2024-09-13T15:02:52.947 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014767400000000001 2024-09-13T15:02:52.947 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.947 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.947 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.947 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.933363+0000", 2024-09-13T15:02:52.947 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0082793780000000004 2024-09-13T15:02:52.947 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.947 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.947 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.948 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.933460+0000", 2024-09-13T15:02:52.948 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.6311000000000007e-05 2024-09-13T15:02:52.948 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.948 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.948 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:02:52.948 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.944312+0000", 2024-09-13T15:02:52.948 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.010852379000000001 2024-09-13T15:02:52.948 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.948 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.948 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.948 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.944458+0000", 2024-09-13T15:02:52.948 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000146095 2024-09-13T15:02:52.948 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.949 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.949 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.949 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.241435+0000", 2024-09-13T15:02:52.949 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.296976935 2024-09-13T15:02:52.949 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.949 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.949 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:02:52.949 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.241486+0000", 2024-09-13T15:02:52.949 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1209999999999999e-05 2024-09-13T15:02:52.949 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.949 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.949 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:02:52.949 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.241598+0000", 2024-09-13T15:02:52.949 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000111427 2024-09-13T15:02:52.950 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.950 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:02:52.950 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.950 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.950 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.950 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:193313 10.d 10:bf5ebe87:::benchmark_data_smithi151_288181_object67700:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1292)", 2024-09-13T15:02:52.950 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:01:04.933369+0000", 2024-09-13T15:02:52.950 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 107.927071623, 2024-09-13T15:02:52.950 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.30841534199999998, 2024-09-13T15:02:52.950 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:02:52.950 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:52.950 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:02:52.950 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:02:52.951 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:02:52.951 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 193313 2024-09-13T15:02:52.951 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.951 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:02:52.951 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.951 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:02:52.951 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.933369+0000", 2024-09-13T15:02:52.951 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.951 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.951 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.951 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:02:52.951 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.933369+0000", 2024-09-13T15:02:52.951 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.951 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.952 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.952 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:02:52.952 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.933368+0000", 2024-09-13T15:02:52.952 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:02:52.952 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.952 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.952 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:02:52.952 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.933375+0000", 2024-09-13T15:02:52.952 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.6649999999999997e-06 2024-09-13T15:02:52.952 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.952 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.952 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:02:52.952 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.933386+0000", 2024-09-13T15:02:52.953 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1603999999999999e-05 2024-09-13T15:02:52.953 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.953 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.953 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:02:52.953 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.933392+0000", 2024-09-13T15:02:52.953 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6500000000000001e-06 2024-09-13T15:02:52.953 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.953 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.953 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:02:52.953 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.933491+0000", 2024-09-13T15:02:52.953 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.9259000000000006e-05 2024-09-13T15:02:52.953 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.953 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.954 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:02:52.954 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.933646+0000", 2024-09-13T15:02:52.954 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000154511 2024-09-13T15:02:52.954 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.954 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.954 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,3,6,7", 2024-09-13T15:02:52.954 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.933826+0000", 2024-09-13T15:02:52.954 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00018021700000000001 2024-09-13T15:02:52.954 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.954 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.954 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.954 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.944359+0000", 2024-09-13T15:02:52.954 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.010532781999999999 2024-09-13T15:02:52.954 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.954 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.955 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:02:52.955 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.944410+0000", 2024-09-13T15:02:52.955 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1309e-05 2024-09-13T15:02:52.955 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.955 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.955 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.955 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.944492+0000", 2024-09-13T15:02:52.955 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.2646999999999997e-05 2024-09-13T15:02:52.955 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.955 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.955 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.955 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.946372+0000", 2024-09-13T15:02:52.955 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0018794899999999999 2024-09-13T15:02:52.955 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.956 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.956 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.956 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.241636+0000", 2024-09-13T15:02:52.956 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.29526380899999999 2024-09-13T15:02:52.956 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.956 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.956 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:02:52.956 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.241677+0000", 2024-09-13T15:02:52.956 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1121000000000002e-05 2024-09-13T15:02:52.956 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.956 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.956 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:02:52.956 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.241785+0000", 2024-09-13T15:02:52.956 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000107599 2024-09-13T15:02:52.957 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.957 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:02:52.957 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.957 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.957 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.957 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:193316 10.d 10:b98eba84:::benchmark_data_smithi151_288181_object67703:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1292)", 2024-09-13T15:02:52.957 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:01:04.935949+0000", 2024-09-13T15:02:52.957 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 107.924491773, 2024-09-13T15:02:52.957 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.30602730700000003, 2024-09-13T15:02:52.957 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:02:52.957 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:52.957 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:02:52.957 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:02:52.957 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:02:52.958 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 193316 2024-09-13T15:02:52.958 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.958 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:02:52.958 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.958 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:02:52.958 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.935949+0000", 2024-09-13T15:02:52.958 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.958 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.958 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.958 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:02:52.958 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.935949+0000", 2024-09-13T15:02:52.958 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.958 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.959 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.959 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:02:52.959 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.935948+0000", 2024-09-13T15:02:52.959 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:02:52.959 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.959 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.959 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:02:52.959 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.935953+0000", 2024-09-13T15:02:52.959 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1959999999999999e-06 2024-09-13T15:02:52.959 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.959 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.959 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:02:52.959 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.935964+0000", 2024-09-13T15:02:52.959 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1097999999999999e-05 2024-09-13T15:02:52.960 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.960 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.960 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:02:52.960 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.935970+0000", 2024-09-13T15:02:52.960 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7119999999999997e-06 2024-09-13T15:02:52.960 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.960 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.960 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:02:52.960 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.937426+0000", 2024-09-13T15:02:52.960 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001456043 2024-09-13T15:02:52.960 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.960 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.960 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:02:52.960 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.937585+0000", 2024-09-13T15:02:52.961 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000159226 2024-09-13T15:02:52.961 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.961 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.961 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,3,6,7", 2024-09-13T15:02:52.961 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.937718+0000", 2024-09-13T15:02:52.961 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013257399999999999 2024-09-13T15:02:52.961 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.961 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.961 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.961 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.944403+0000", 2024-09-13T15:02:52.961 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0066853709999999998 2024-09-13T15:02:52.961 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.961 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.961 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:02:52.962 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.944416+0000", 2024-09-13T15:02:52.962 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2227e-05 2024-09-13T15:02:52.962 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.962 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.962 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.962 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.944537+0000", 2024-09-13T15:02:52.962 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000121178 2024-09-13T15:02:52.962 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.962 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.962 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.962 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.946714+0000", 2024-09-13T15:02:52.962 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0021774300000000002 2024-09-13T15:02:52.962 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.962 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.963 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.963 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.241822+0000", 2024-09-13T15:02:52.963 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.29510763299999998 2024-09-13T15:02:52.963 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.963 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.963 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:02:52.963 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.241862+0000", 2024-09-13T15:02:52.963 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9870000000000003e-05 2024-09-13T15:02:52.963 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.963 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.963 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:02:52.963 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.241976+0000", 2024-09-13T15:02:52.963 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00011466 2024-09-13T15:02:52.963 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.964 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:02:52.964 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.964 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.964 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.964 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:193317 10.d 10:ba882176:::benchmark_data_smithi151_288181_object67704:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1292)", 2024-09-13T15:02:52.964 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:01:04.936180+0000", 2024-09-13T15:02:52.964 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 107.92426078699999, 2024-09-13T15:02:52.964 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.30604800399999998, 2024-09-13T15:02:52.964 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:02:52.964 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:52.964 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:02:52.964 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:02:52.964 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:02:52.964 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 193317 2024-09-13T15:02:52.964 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.965 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:02:52.965 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.965 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:02:52.965 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.936180+0000", 2024-09-13T15:02:52.965 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.965 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.965 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.965 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:02:52.965 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.936180+0000", 2024-09-13T15:02:52.965 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.965 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.965 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.965 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:02:52.965 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.936179+0000", 2024-09-13T15:02:52.965 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T15:02:52.966 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.966 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.966 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:02:52.966 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.936183+0000", 2024-09-13T15:02:52.966 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1969999999999998e-06 2024-09-13T15:02:52.966 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.966 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.966 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:02:52.966 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.936192+0000", 2024-09-13T15:02:52.966 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.8289999999999997e-06 2024-09-13T15:02:52.966 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.966 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.966 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:02:52.966 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.936198+0000", 2024-09-13T15:02:52.966 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3020000000000002e-06 2024-09-13T15:02:52.966 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.967 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.967 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:02:52.967 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.940951+0000", 2024-09-13T15:02:52.967 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0047530209999999996 2024-09-13T15:02:52.967 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.967 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.967 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:02:52.967 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.941068+0000", 2024-09-13T15:02:52.967 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00011701000000000001 2024-09-13T15:02:52.967 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.967 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.967 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,3,6,7", 2024-09-13T15:02:52.967 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.941177+0000", 2024-09-13T15:02:52.967 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000109484 2024-09-13T15:02:52.968 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.968 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.968 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.968 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.944587+0000", 2024-09-13T15:02:52.968 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0034096090000000001 2024-09-13T15:02:52.968 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.968 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.968 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:02:52.968 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.944596+0000", 2024-09-13T15:02:52.968 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.7389999999999994e-06 2024-09-13T15:02:52.968 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.968 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.968 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.968 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.944804+0000", 2024-09-13T15:02:52.968 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000207328 2024-09-13T15:02:52.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.969 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.969 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.969 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:04.946739+0000", 2024-09-13T15:02:52.969 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0019347240000000001 2024-09-13T15:02:52.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.969 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.969 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.969 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.242031+0000", 2024-09-13T15:02:52.969 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.295292742 2024-09-13T15:02:52.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.969 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.969 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:02:52.969 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.242105+0000", 2024-09-13T15:02:52.970 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.3392999999999993e-05 2024-09-13T15:02:52.970 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.970 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.970 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:02:52.970 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.242228+0000", 2024-09-13T15:02:52.970 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012332000000000001 2024-09-13T15:02:52.970 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.970 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:02:52.970 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.970 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.970 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.970 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:193547 10.d 10:b6f45b43:::benchmark_data_smithi151_288181_object67934:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1292)", 2024-09-13T15:02:52.970 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:01:05.491522+0000", 2024-09-13T15:02:52.971 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 107.368918515, 2024-09-13T15:02:52.971 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.18128734099999999, 2024-09-13T15:02:52.971 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:02:52.971 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:52.971 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:02:52.971 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:02:52.971 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:02:52.971 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 193547 2024-09-13T15:02:52.971 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.971 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:02:52.971 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.971 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:02:52.971 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.491522+0000", 2024-09-13T15:02:52.971 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.971 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.971 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.972 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:02:52.972 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.491522+0000", 2024-09-13T15:02:52.972 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.972 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.972 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.972 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:02:52.972 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.491521+0000", 2024-09-13T15:02:52.972 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:02:52.972 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.972 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.972 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:02:52.972 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.491527+0000", 2024-09-13T15:02:52.972 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.9059999999999999e-06 2024-09-13T15:02:52.972 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.972 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.973 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:02:52.973 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.491537+0000", 2024-09-13T15:02:52.973 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.893e-06 2024-09-13T15:02:52.973 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.973 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.973 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:02:52.973 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.491541+0000", 2024-09-13T15:02:52.973 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1749999999999998e-06 2024-09-13T15:02:52.973 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.973 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.973 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:02:52.973 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.491585+0000", 2024-09-13T15:02:52.973 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3677999999999998e-05 2024-09-13T15:02:52.973 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.973 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.974 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:02:52.974 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.491673+0000", 2024-09-13T15:02:52.974 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.8698999999999999e-05 2024-09-13T15:02:52.974 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.974 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.974 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,3,6,7", 2024-09-13T15:02:52.974 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.491764+0000", 2024-09-13T15:02:52.974 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.0985999999999999e-05 2024-09-13T15:02:52.974 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.974 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.974 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:02:52.974 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.495643+0000", 2024-09-13T15:02:52.974 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0038782370000000001 2024-09-13T15:02:52.974 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.974 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.974 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.975 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.495818+0000", 2024-09-13T15:02:52.975 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000175611 2024-09-13T15:02:52.975 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.975 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.975 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.975 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.500658+0000", 2024-09-13T15:02:52.975 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0048401360000000001 2024-09-13T15:02:52.975 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.975 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.975 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.975 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.521019+0000", 2024-09-13T15:02:52.975 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.020360385000000002 2024-09-13T15:02:52.975 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.975 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.976 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.976 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.672535+0000", 2024-09-13T15:02:52.976 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.151516177 2024-09-13T15:02:52.976 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.976 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.976 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:02:52.976 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.672701+0000", 2024-09-13T15:02:52.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:52.977 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T15:02:52.977 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000165606 2024-09-13T15:02:52.977 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.978 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.978 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:02:52.978 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.672810+0000", 2024-09-13T15:02:52.978 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000109051 2024-09-13T15:02:52.978 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.978 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:02:52.978 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.978 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.978 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.978 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:193549 10.d 10:b76d56a9:::benchmark_data_smithi151_288181_object67936:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1292)", 2024-09-13T15:02:52.978 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:01:05.496025+0000", 2024-09-13T15:02:52.978 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 107.364415304, 2024-09-13T15:02:52.978 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.17696900600000001, 2024-09-13T15:02:52.979 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:02:52.979 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:52.979 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:02:52.979 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:02:52.979 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:02:52.979 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 193549 2024-09-13T15:02:52.979 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.979 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:02:52.979 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.979 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:02:52.979 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.496025+0000", 2024-09-13T15:02:52.979 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.979 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.980 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.980 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:02:52.980 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.496025+0000", 2024-09-13T15:02:52.980 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.980 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.980 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.980 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:02:52.980 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.496024+0000", 2024-09-13T15:02:52.980 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:02:52.980 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.980 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.980 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:02:52.980 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.496031+0000", 2024-09-13T15:02:52.980 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.0129999999999996e-06 2024-09-13T15:02:52.981 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.981 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.981 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:02:52.981 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.496041+0000", 2024-09-13T15:02:52.981 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.9569999999999997e-06 2024-09-13T15:02:52.981 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.981 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.981 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:02:52.981 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.496046+0000", 2024-09-13T15:02:52.981 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.2610000000000001e-06 2024-09-13T15:02:52.981 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.981 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.981 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:02:52.981 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.496094+0000", 2024-09-13T15:02:52.981 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7574000000000002e-05 2024-09-13T15:02:52.982 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.982 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.982 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:02:52.982 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.496250+0000", 2024-09-13T15:02:52.982 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00015599400000000001 2024-09-13T15:02:52.982 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.982 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.982 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,3,6,7", 2024-09-13T15:02:52.982 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.496394+0000", 2024-09-13T15:02:52.982 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014384099999999999 2024-09-13T15:02:52.982 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.982 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.982 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:02:52.982 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.500084+0000", 2024-09-13T15:02:52.982 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0036903589999999998 2024-09-13T15:02:52.982 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.983 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.983 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.983 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.500212+0000", 2024-09-13T15:02:52.983 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000127551 2024-09-13T15:02:52.983 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.983 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.983 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.983 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.500789+0000", 2024-09-13T15:02:52.983 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00057704900000000005 2024-09-13T15:02:52.983 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.983 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.983 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.983 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.521056+0000", 2024-09-13T15:02:52.983 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.020266948999999999 2024-09-13T15:02:52.983 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.984 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.984 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.984 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.672855+0000", 2024-09-13T15:02:52.984 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.15179975000000001 2024-09-13T15:02:52.984 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.984 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.984 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:02:52.984 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.672906+0000", 2024-09-13T15:02:52.984 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0615000000000002e-05 2024-09-13T15:02:52.984 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.984 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.984 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:02:52.984 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.672994+0000", 2024-09-13T15:02:52.984 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.8577000000000003e-05 2024-09-13T15:02:52.984 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.985 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:02:52.985 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.985 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.985 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.985 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:193560 10.d 10:bc268530:::benchmark_data_smithi151_288181_object67947:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1292)", 2024-09-13T15:02:52.985 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:01:05.508754+0000", 2024-09-13T15:02:52.985 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 107.351686566, 2024-09-13T15:02:52.985 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.164370933, 2024-09-13T15:02:52.985 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:02:52.985 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:52.985 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:02:52.985 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:02:52.985 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:02:52.985 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 193560 2024-09-13T15:02:52.986 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.986 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:02:52.986 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.986 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:02:52.986 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.508754+0000", 2024-09-13T15:02:52.986 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.986 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.986 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.986 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:02:52.986 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.508754+0000", 2024-09-13T15:02:52.986 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.986 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.987 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.987 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:02:52.987 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.508752+0000", 2024-09-13T15:02:52.987 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T15:02:52.987 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.987 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.987 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:02:52.987 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.508759+0000", 2024-09-13T15:02:52.987 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.6959999999999999e-06 2024-09-13T15:02:52.987 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.987 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.987 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:02:52.987 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.508769+0000", 2024-09-13T15:02:52.988 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0196e-05 2024-09-13T15:02:52.988 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.988 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.988 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:02:52.988 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.508775+0000", 2024-09-13T15:02:52.988 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1580000000000002e-06 2024-09-13T15:02:52.988 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.988 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.988 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:02:52.988 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.508834+0000", 2024-09-13T15:02:52.988 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8644000000000003e-05 2024-09-13T15:02:52.988 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.988 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.989 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:02:52.989 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.508977+0000", 2024-09-13T15:02:52.989 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000143342 2024-09-13T15:02:52.989 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.989 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.989 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,3,6,7", 2024-09-13T15:02:52.989 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.509087+0000", 2024-09-13T15:02:52.989 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000109604 2024-09-13T15:02:52.989 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.989 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.989 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:02:52.989 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.512788+0000", 2024-09-13T15:02:52.989 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0037007089999999999 2024-09-13T15:02:52.989 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.990 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.990 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.990 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.512957+0000", 2024-09-13T15:02:52.990 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00016903600000000001 2024-09-13T15:02:52.990 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.990 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.990 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.990 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.513018+0000", 2024-09-13T15:02:52.990 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.0955999999999997e-05 2024-09-13T15:02:52.990 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.990 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.990 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.990 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.526124+0000", 2024-09-13T15:02:52.990 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.013106242000000001 2024-09-13T15:02:52.991 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.991 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.991 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.991 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.673031+0000", 2024-09-13T15:02:52.991 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.14690699800000001 2024-09-13T15:02:52.991 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.991 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.991 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:02:52.991 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.673060+0000", 2024-09-13T15:02:52.991 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.8839e-05 2024-09-13T15:02:52.991 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.991 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.991 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:02:52.992 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:05.673125+0000", 2024-09-13T15:02:52.992 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.5399000000000002e-05 2024-09-13T15:02:52.992 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.992 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:02:52.992 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.992 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.992 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.992 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:219888 10.c 10:39a6a18a:::benchmark_data_smithi151_288181_object94275:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1304)", 2024-09-13T15:02:52.992 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:01:39.348596+0000", 2024-09-13T15:02:52.992 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 73.511845143000002, 2024-09-13T15:02:52.992 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.33839245299999998, 2024-09-13T15:02:52.992 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:02:52.992 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:02:52.992 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:02:52.993 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:02:52.993 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:02:52.993 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 219888 2024-09-13T15:02:52.993 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.993 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:02:52.993 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.993 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:02:52.993 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.348596+0000", 2024-09-13T15:02:52.993 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.993 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.993 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.993 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:02:52.993 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.348596+0000", 2024-09-13T15:02:52.993 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:02:52.994 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.994 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.994 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:02:52.994 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.348594+0000", 2024-09-13T15:02:52.994 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T15:02:52.994 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.994 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.994 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:02:52.994 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.348604+0000", 2024-09-13T15:02:52.994 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.628e-06 2024-09-13T15:02:52.994 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.994 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.994 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:02:52.994 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.348618+0000", 2024-09-13T15:02:52.995 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.438e-05 2024-09-13T15:02:52.995 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.995 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.995 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:02:52.995 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.348624+0000", 2024-09-13T15:02:52.995 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1879999999999999e-06 2024-09-13T15:02:52.995 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.995 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.995 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:02:52.995 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.350557+0000", 2024-09-13T15:02:52.995 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001932731 2024-09-13T15:02:52.995 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.995 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.995 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:02:52.996 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.350676+0000", 2024-09-13T15:02:52.996 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000119067 2024-09-13T15:02:52.996 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.996 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.996 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,1,3,6,7", 2024-09-13T15:02:52.996 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.350793+0000", 2024-09-13T15:02:52.996 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000117379 2024-09-13T15:02:52.996 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.996 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.996 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:02:52.996 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.358783+0000", 2024-09-13T15:02:52.996 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0079894969999999999 2024-09-13T15:02:52.996 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.996 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.996 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.997 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.359859+0000", 2024-09-13T15:02:52.997 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001075866 2024-09-13T15:02:52.997 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.997 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.997 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.997 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.359898+0000", 2024-09-13T15:02:52.997 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8917e-05 2024-09-13T15:02:52.997 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.997 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.997 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.997 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.360821+0000", 2024-09-13T15:02:52.997 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00092319500000000003 2024-09-13T15:02:52.997 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.997 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.998 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.998 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.360838+0000", 2024-09-13T15:02:52.998 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.7119e-05 2024-09-13T15:02:52.998 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.998 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.998 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:02:52.998 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.686755+0000", 2024-09-13T15:02:52.998 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.32591714100000002 2024-09-13T15:02:52.998 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.998 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.998 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:02:52.998 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.686822+0000", 2024-09-13T15:02:52.998 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7020000000000005e-05 2024-09-13T15:02:52.998 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:02:52.999 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:02:52.999 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:02:52.999 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.686988+0000", 2024-09-13T15:02:52.999 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00016599400000000001 2024-09-13T15:02:52.999 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.999 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:02:52.999 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.999 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:02:52.999 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:02:52.999 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T15:02:53.077 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:53.099 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:53.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:53.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:53.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:52 smithi184 ceph-mon[92527]: pgmap v2240: 119 pgs: 119 active+clean; 647 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 11 KiB/s rd, 27 KiB/s wr, 10 op/s; 42 KiB/s, 1 keys/s, 5 objects/s recovering 2024-09-13T15:02:53.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:52 smithi184 ceph-mon[92527]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:02:53.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:52 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:02:53.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:52 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1592483778' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T15:02:53.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:52 smithi184 ceph-mon[92527]: osdmap e1332: 8 total, 8 up, 6 in 2024-09-13T15:02:53.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:52 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3022662480' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:02:53.202 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:53.223 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:53.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:53.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:53.290 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:52 smithi151 ceph-mon[104450]: pgmap v2240: 119 pgs: 119 active+clean; 647 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 11 KiB/s rd, 27 KiB/s wr, 10 op/s; 42 KiB/s, 1 keys/s, 5 objects/s recovering 2024-09-13T15:02:53.290 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:52 smithi151 ceph-mon[104450]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:02:53.290 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:52 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:02:53.290 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:52 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1592483778' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T15:02:53.291 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:52 smithi151 ceph-mon[104450]: osdmap e1332: 8 total, 8 up, 6 in 2024-09-13T15:02:53.291 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:52 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3022662480' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:02:53.291 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:52 smithi151 ceph-mon[108966]: pgmap v2240: 119 pgs: 119 active+clean; 647 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 11 KiB/s rd, 27 KiB/s wr, 10 op/s; 42 KiB/s, 1 keys/s, 5 objects/s recovering 2024-09-13T15:02:53.291 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:52 smithi151 ceph-mon[108966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:02:53.291 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:52 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:02:53.291 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:52 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1592483778' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T15:02:53.291 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:52 smithi151 ceph-mon[108966]: osdmap e1332: 8 total, 8 up, 6 in 2024-09-13T15:02:53.291 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:52 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3022662480' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:02:53.325 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:53.346 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:53.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:53.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:53.448 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:53.469 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:53.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:53.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:53.571 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:53.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:53.593 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:53.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:53.632 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T15:02:53.694 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:53.715 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:53.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:53.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:53.785 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T15:02:53.816 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:53.848 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:53.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:53.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:53.950 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:53.972 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:53.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:53.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:54.032 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:02:54.074 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:54.095 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:54.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:54.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:54.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:53 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3022662480' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:02:54.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:53 smithi184 ceph-mon[92527]: osdmap e1333: 8 total, 8 up, 6 in 2024-09-13T15:02:54.198 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:54.219 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:54.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:54.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:54.231 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:53 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3022662480' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:02:54.231 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:53 smithi151 ceph-mon[104450]: osdmap e1333: 8 total, 8 up, 6 in 2024-09-13T15:02:54.231 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:53 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3022662480' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:02:54.231 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:53 smithi151 ceph-mon[108966]: osdmap e1333: 8 total, 8 up, 6 in 2024-09-13T15:02:54.321 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:54.344 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:54.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:54.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:54.446 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:54.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:54.468 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:54.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:54.569 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:54.653 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:54.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:54.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:54.755 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:54.777 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:54.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:54.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:54.843 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:54.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:02:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:02:54] "GET /metrics HTTP/1.1" 200 36628 "" "Prometheus/2.51.0" 2024-09-13T15:02:54.879 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:54.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:54.901 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:54.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:55.000 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T15:02:55.003 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:55.024 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:55.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:55.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:55.120 DEBUG:teuthology.orchestra.run.smithi151:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_6 bench 90 write' 2024-09-13T15:02:55.123 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T15:02:55.123 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T15:02:55.126 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:55.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:55.149 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:55.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:55.171 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:hints = 1 2024-09-13T15:02:55.172 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T15:02:55.172 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Object prefix: benchmark_data_smithi151_323728 2024-09-13T15:02:55.172 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T15:02:55.172 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 0 0 0 0 0 0 - 0 2024-09-13T15:02:55.188 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:54 smithi184 ceph-mon[92527]: pgmap v2243: 119 pgs: 119 active+clean; 647 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 11 KiB/s rd, 27 KiB/s wr, 10 op/s; 42 KiB/s, 1 keys/s, 5 objects/s recovering 2024-09-13T15:02:55.188 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:54 smithi184 ceph-mon[92527]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T15:02:55.250 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:55.270 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:54 smithi151 ceph-mon[108966]: pgmap v2243: 119 pgs: 119 active+clean; 647 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 11 KiB/s rd, 27 KiB/s wr, 10 op/s; 42 KiB/s, 1 keys/s, 5 objects/s recovering 2024-09-13T15:02:55.271 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:54 smithi151 ceph-mon[108966]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T15:02:55.271 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:54 smithi151 ceph-mon[104450]: pgmap v2243: 119 pgs: 119 active+clean; 647 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 11 KiB/s rd, 27 KiB/s wr, 10 op/s; 42 KiB/s, 1 keys/s, 5 objects/s recovering 2024-09-13T15:02:55.271 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:54 smithi151 ceph-mon[104450]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T15:02:55.292 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:55.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:55.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:55.394 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:55.428 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:55.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:55.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:55.530 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:55.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:55.568 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:55.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:55.669 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:55.703 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:55.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:55.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:55.805 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:55.813 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T15:02:55.813 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [], 2024-09-13T15:02:55.813 INFO:teuthology.orchestra.run.smithi184.stdout: "num_ops": 0 2024-09-13T15:02:55.813 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T15:02:55.842 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:55.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:55.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:55.944 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:55.974 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:55.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:55.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:56.076 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:56.113 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:56.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:56.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:56.172 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 1 15 3925 3910 244.361 244.375 0.00420953 0.00408233 2024-09-13T15:02:56.214 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:56.244 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:56.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:56.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:56.346 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:56.379 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:56.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:56.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:56.481 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:56.516 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:56.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:56.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:56.559 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T15:02:56.618 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:56.652 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:56.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:56.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:56.754 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:56.785 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:56.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:56.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:56.887 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:56.929 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:56.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:56.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:56.946 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:02:57.025 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:56 smithi151 ceph-mon[104450]: pgmap v2244: 119 pgs: 119 active+clean; 647 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:57.025 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:56 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:02:57.025 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:56 smithi151 ceph-mon[108966]: pgmap v2244: 119 pgs: 119 active+clean; 647 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:57.025 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:56 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:02:57.031 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:57.063 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:57.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:57.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:57.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:57.171 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 2 16 7414 7398 231.171 218 0.00785143 0.00430637 2024-09-13T15:02:57.211 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:57.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:57.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:57.235 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:56 smithi184 ceph-mon[92527]: pgmap v2244: 119 pgs: 119 active+clean; 647 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-13T15:02:57.236 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:56 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:02:57.313 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:57.353 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:57.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:57.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:57.455 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:57.498 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:57.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:57.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:57.600 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:57.631 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:57.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:57.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:57.732 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:57.762 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:57.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:57.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:57.864 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:57.900 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:57.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:57.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:57.924 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T15:02:58.003 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:58.031 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:58.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:58.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:58.134 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:58.172 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 3 16 10446 10430 217.275 189.5 0.00600001 0.00459538 2024-09-13T15:02:58.174 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:58.174 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:57 smithi151 ceph-mon[104450]: 1.9 scrub starts 2024-09-13T15:02:58.174 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:57 smithi151 ceph-mon[104450]: 1.9 scrub ok 2024-09-13T15:02:58.175 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:57 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3066996550' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:02:58.175 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:57 smithi151 ceph-mon[108966]: 1.9 scrub starts 2024-09-13T15:02:58.175 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:57 smithi151 ceph-mon[108966]: 1.9 scrub ok 2024-09-13T15:02:58.175 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:57 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3066996550' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:02:58.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:58.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:58.194 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:57 smithi184 ceph-mon[92527]: 1.9 scrub starts 2024-09-13T15:02:58.194 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:57 smithi184 ceph-mon[92527]: 1.9 scrub ok 2024-09-13T15:02:58.194 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:57 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3066996550' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:02:58.277 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:58.314 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:58.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:58.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:58.416 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:58.447 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:58.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:58.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:58.549 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:58.587 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:58.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:58.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:58.689 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:58.723 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:58.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:58.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:58.797 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T15:02:58.797 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:02:58.825 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:58.856 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:58.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:58.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:58.881 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T15:02:58.882 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T15:02:58.882 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T15:02:58.882 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T15:02:58.882 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T15:02:58.882 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T15:02:58.882 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T15:02:58.882 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T15:02:58.882 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:02:58.882 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:02:58.883 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:02:58.883 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:02:58.883 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:02:58.883 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:02:58.883 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:02:58.883 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:02:58.958 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:58.988 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:58.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:58.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:59.090 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:59.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:58 smithi151 ceph-mon[108966]: pgmap v2245: 119 pgs: 119 active+clean; 252 MiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 31 MiB/s wr, 502 op/s 2024-09-13T15:02:59.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:58 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3066996550' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:02:59.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:58 smithi151 ceph-mon[108966]: osdmap e1334: 8 total, 8 up, 6 in 2024-09-13T15:02:59.111 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:02:58 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2614376346' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:59.112 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:58 smithi151 ceph-mon[104450]: pgmap v2245: 119 pgs: 119 active+clean; 252 MiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 31 MiB/s wr, 502 op/s 2024-09-13T15:02:59.112 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:58 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3066996550' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:02:59.112 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:58 smithi151 ceph-mon[104450]: osdmap e1334: 8 total, 8 up, 6 in 2024-09-13T15:02:59.112 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:02:58 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2614376346' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:59.137 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:59.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:59.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:59.172 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 4 16 13292 13276 207.421 177.875 0.00766935 0.00481509 2024-09-13T15:02:59.239 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:59.241 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:58 smithi184 ceph-mon[92527]: pgmap v2245: 119 pgs: 119 active+clean; 252 MiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 31 MiB/s wr, 502 op/s 2024-09-13T15:02:59.241 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:58 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3066996550' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:02:59.241 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:58 smithi184 ceph-mon[92527]: osdmap e1334: 8 total, 8 up, 6 in 2024-09-13T15:02:59.241 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:02:58 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2614376346' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:59.276 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:59.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:59.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:59.378 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:59.395 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T15:02:59.395 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:02:59.412 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:59.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:59.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:59.514 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:59.551 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:59.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:59.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:59.652 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:59.684 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:02:59.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:59.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:59.786 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:59.824 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:59.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:59.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:59.926 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:59.963 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:02:59.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:59.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:00.063 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T15:03:00.065 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:00.101 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:00.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:00.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:00.172 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 5 15 16201 16186 202.308 181.875 0.00417882 0.00493618 2024-09-13T15:03:00.203 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:00.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:00.241 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:00.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:00.342 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:00.354 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:03:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:03:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:03:00.386 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:00.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:00.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:00.440 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:00.460 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:03:00.488 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:00.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:00.538 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:00.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:00.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:00.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:00.682 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:00.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:00.783 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:00.839 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:00.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:00.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:00.941 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:00.962 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:00.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:00.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:01.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:01.100 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:01.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:01.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:01.172 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 6 16 18725 18709 194.869 157.688 0.0022444 0.00512489 2024-09-13T15:03:01.202 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:01.238 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:01.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:01.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:01.340 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:01.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:00 smithi184 ceph-mon[92527]: pgmap v2247: 119 pgs: 119 active+clean; 621 MiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-13T15:03:01.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:00 smithi184 ceph-mon[92527]: 1.2 scrub starts 2024-09-13T15:03:01.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:00 smithi184 ceph-mon[92527]: 1.2 scrub ok 2024-09-13T15:03:01.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:00 smithi151 ceph-mon[104450]: pgmap v2247: 119 pgs: 119 active+clean; 621 MiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-13T15:03:01.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:00 smithi151 ceph-mon[104450]: 1.2 scrub starts 2024-09-13T15:03:01.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:00 smithi151 ceph-mon[104450]: 1.2 scrub ok 2024-09-13T15:03:01.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:00 smithi151 ceph-mon[108966]: pgmap v2247: 119 pgs: 119 active+clean; 621 MiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-13T15:03:01.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:00 smithi151 ceph-mon[108966]: 1.2 scrub starts 2024-09-13T15:03:01.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:00 smithi151 ceph-mon[108966]: 1.2 scrub ok 2024-09-13T15:03:01.372 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:01.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:01.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:01.474 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:01.511 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:01.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:01.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:01.613 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:01.649 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:01.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:01.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:01.751 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:01.781 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:01.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:01.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:01.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:01.918 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:01.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:01.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:02.020 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:02.052 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:02.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:02.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:02.154 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:02.172 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 7 16 21006 20990 187.395 142.562 0.00633404 0.00533081 2024-09-13T15:03:02.186 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:02.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:02.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:02.288 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:02.310 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:02.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:02.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:02.412 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:02.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:03:02.441 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:02.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:02.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:02.543 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:02.571 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:02.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:02.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:02.673 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:02.707 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:02.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:02.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:02.809 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:02.841 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:02.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:02.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:02.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:02 smithi184 ceph-mon[92527]: 1.e scrub starts 2024-09-13T15:03:02.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:02 smithi184 ceph-mon[92527]: 1.e scrub ok 2024-09-13T15:03:02.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:03:02.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:02.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:02 smithi151 ceph-mon[104450]: 1.e scrub starts 2024-09-13T15:03:02.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:02 smithi151 ceph-mon[104450]: 1.e scrub ok 2024-09-13T15:03:02.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:03:02.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:02.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:02 smithi151 ceph-mon[108966]: 1.e scrub starts 2024-09-13T15:03:02.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:02 smithi151 ceph-mon[108966]: 1.e scrub ok 2024-09-13T15:03:02.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:03:02.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:02.905 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:02.942 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:02.973 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:02.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:02.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:03.075 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:03.100 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:03.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:03.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:03.172 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 8 16 23255 23239 181.539 140.562 0.00548556 0.00550325 2024-09-13T15:03:03.202 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:03.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:03.233 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:03.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:03.334 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:03.355 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:03.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:03.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:03.457 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:03.487 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:03.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:03.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:03.589 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:03.623 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:03.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:03.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:03.725 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:03.756 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:03.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:03.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:03.801 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:03 smithi151 ceph-mon[104450]: pgmap v2248: 119 pgs: 119 active+clean; 621 MiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-13T15:03:03.801 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:03 smithi151 ceph-mon[108966]: pgmap v2248: 119 pgs: 119 active+clean; 621 MiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-13T15:03:03.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:03 smithi184 ceph-mon[92527]: pgmap v2248: 119 pgs: 119 active+clean; 621 MiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-13T15:03:03.854 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 15:03:03 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a[78349]: logger=cleanup t=2024-09-13T15:03:03.580650566Z level=info msg="Completed cleanup jobs" duration=290.648932ms 2024-09-13T15:03:03.858 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:03.879 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:03.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:03.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:03.981 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:04.017 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:04.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:04.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:04.119 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:04.147 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:04.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:04.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:04.172 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 9 16 25194 25178 174.832 121.188 0.00234718 0.00569079 2024-09-13T15:03:04.249 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:04.281 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:04.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:04.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:04.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:04.407 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:04.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:04.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:04.509 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:04.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:04.545 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:04.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:04.646 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:04.678 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:04.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:04.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:04.751 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:04 smithi184 ceph-mon[92527]: pgmap v2249: 119 pgs: 119 active+clean; 1.4 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-09-13T15:03:04.780 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:04.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:04.818 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:04.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:04.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:04 smithi151 ceph-mon[104450]: pgmap v2249: 119 pgs: 119 active+clean; 1.4 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-09-13T15:03:04.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:04 smithi151 ceph-mon[108966]: pgmap v2249: 119 pgs: 119 active+clean; 1.4 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-09-13T15:03:04.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:03:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:03:04] "GET /metrics HTTP/1.1" 200 36700 "" "Prometheus/2.51.0" 2024-09-13T15:03:04.920 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:04.944 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:04.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:04.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:05.046 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:05.077 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:05.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:05.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:05.173 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 10 16 27158 27142 169.623 122.75 0.00486227 0.00589016 2024-09-13T15:03:05.179 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:05.201 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:05.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:05.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:05.303 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:05.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:05.344 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:05.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:05.412 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T15:03:05.413 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:03:05.445 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:05.470 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:05.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:05.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:05.572 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:05.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:05.606 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:05.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:05.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:05 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [7, 2]} : dispatch 2024-09-13T15:03:05.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:05 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [7, 2]} : dispatch 2024-09-13T15:03:05.707 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:05.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:05.748 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:05.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:05.850 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:05.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:05 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [7, 2]} : dispatch 2024-09-13T15:03:05.881 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:05.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:05.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:05.984 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:06.015 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:06.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:06.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:06.117 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:06.149 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:06.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:06.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:06.173 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 11 16 29156 29140 165.554 124.875 0.00383985 0.00603501 2024-09-13T15:03:06.250 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:06.283 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:06.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:06.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:06.384 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:06.416 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:06.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:06.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:06.518 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:06.565 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:06.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:06.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:06.667 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:06.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:06.706 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:06.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:06.807 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:06.818 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:06 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [7, 2]}]': finished 2024-09-13T15:03:06.818 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:06 smithi151 ceph-mon[104450]: osdmap e1335: 8 total, 8 up, 6 in 2024-09-13T15:03:06.818 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:06 smithi151 ceph-mon[104450]: pgmap v2251: 119 pgs: 119 active+clean; 1.4 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 147 MiB/s wr, 2.35k op/s 2024-09-13T15:03:06.819 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:06 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [7, 2]}]': finished 2024-09-13T15:03:06.819 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:06 smithi151 ceph-mon[108966]: osdmap e1335: 8 total, 8 up, 6 in 2024-09-13T15:03:06.819 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:06 smithi151 ceph-mon[108966]: pgmap v2251: 119 pgs: 119 active+clean; 1.4 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 147 MiB/s wr, 2.35k op/s 2024-09-13T15:03:06.831 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:06.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:06.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:06.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:06 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [7, 2]}]': finished 2024-09-13T15:03:06.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:06 smithi184 ceph-mon[92527]: osdmap e1335: 8 total, 8 up, 6 in 2024-09-13T15:03:06.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:06 smithi184 ceph-mon[92527]: pgmap v2251: 119 pgs: 119 active+clean; 1.4 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 147 MiB/s wr, 2.35k op/s 2024-09-13T15:03:06.934 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:06.972 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:06.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:06.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:07.031 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T15:03:07.073 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:07.102 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:07.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:07.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:07.173 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 12 16 31252 31236 162.673 131 0.00672112 0.00614232 2024-09-13T15:03:07.204 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:07.226 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:07.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:07.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:07.328 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:07.362 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:07.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:07.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:07.401 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:03:07.464 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:07.496 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:07.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:07.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:07.598 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:07.627 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:07.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:07.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:07.729 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:07.766 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:07.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:07.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:07.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:07 smithi184 ceph-mon[92527]: osdmap e1336: 8 total, 8 up, 6 in 2024-09-13T15:03:07.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:07 smithi151 ceph-mon[104450]: osdmap e1336: 8 total, 8 up, 6 in 2024-09-13T15:03:07.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:07 smithi151 ceph-mon[108966]: osdmap e1336: 8 total, 8 up, 6 in 2024-09-13T15:03:07.868 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:07.890 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:07.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:07.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:07.992 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:08.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:08.027 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:08.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:08.128 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:08.149 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:08.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:08.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:08.172 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 13 16 32772 32756 157.467 95 0.00156649 0.00633801 2024-09-13T15:03:08.251 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:08.281 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:08.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:08.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:08.382 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:08.412 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:08.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:08.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:08.513 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:08.542 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:08.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:08.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:08.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:08 smithi184 ceph-mon[92527]: pgmap v2253: 119 pgs: 119 active+clean; 1.7 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 141 MiB/s wr, 2.25k op/s 2024-09-13T15:03:08.643 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:08.676 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:08.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:08.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:08.778 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:08.818 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:08.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:08.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:08.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:08 smithi151 ceph-mon[104450]: pgmap v2253: 119 pgs: 119 active+clean; 1.7 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 141 MiB/s wr, 2.25k op/s 2024-09-13T15:03:08.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:08 smithi151 ceph-mon[108966]: pgmap v2253: 119 pgs: 119 active+clean; 1.7 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 141 MiB/s wr, 2.25k op/s 2024-09-13T15:03:08.920 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:08.958 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:08.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:08.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:09.060 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:09.087 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:09.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:09.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:09.173 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 14 16 34725 34709 154.937 122.062 0.00780277 0.00644928 2024-09-13T15:03:09.189 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:09.229 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:09.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:09.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:09.286 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T15:03:09.286 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:03:09.331 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:09.368 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:09.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:09.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:09.470 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:09.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:09.507 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:09.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:09.608 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:09.649 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:09.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:09.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:09.751 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:09.789 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:09.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:09.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:09.891 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:09.930 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:09.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:09.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:09.946 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T15:03:10.032 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:10.054 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:10.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:10.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:10.156 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:10.173 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 15 16 36542 36526 152.178 113.562 0.0100024 0.00656608 2024-09-13T15:03:10.200 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:10.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:10.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:10.302 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:10.317 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:03:10.332 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:10.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:10.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:10.434 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:10.462 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:10.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:10.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:10.564 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:10.574 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:03:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:03:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:03:10.602 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:10.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:10.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:10.704 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:10.743 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:10.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:10.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:10.845 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:10.874 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:10.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:10.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:10.976 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:11.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:11.008 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:11.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:11.109 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:11.141 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:11.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:11.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:11.173 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 16 16 38302 38286 149.541 110 0.00509082 0.00668221 2024-09-13T15:03:11.243 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:11.265 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:11.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:11.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:11.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:10 smithi184 ceph-mon[92527]: pgmap v2254: 119 pgs: 119 active+clean; 2.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 177 MiB/s wr, 2.84k op/s 2024-09-13T15:03:11.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:10 smithi151 ceph-mon[104450]: pgmap v2254: 119 pgs: 119 active+clean; 2.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 177 MiB/s wr, 2.84k op/s 2024-09-13T15:03:11.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:10 smithi151 ceph-mon[108966]: pgmap v2254: 119 pgs: 119 active+clean; 2.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 177 MiB/s wr, 2.84k op/s 2024-09-13T15:03:11.367 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:11.401 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:11.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:11.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:11.503 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:11.530 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:11.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:11.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:11.632 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:11.666 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:11.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:11.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:11.768 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:11.801 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:11.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:11.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:11.903 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:11.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:11.940 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:11.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:12.041 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:12.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:12.067 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:12.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:12.168 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:12.173 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 17 16 39899 39883 146.611 99.8125 0.0134379 0.00681511 2024-09-13T15:03:12.204 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:12.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:12.205 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:12.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:12.306 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:12.344 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:12.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:12.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:12.446 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:12.471 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:12.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:12.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:12.573 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:12.610 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:12.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:12.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:12.712 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:12.747 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:12.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:12.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:12.849 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:12.882 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:12.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:12.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:12.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:13.018 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:13.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:13.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:13.120 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:13.160 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:13.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:13.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:13.174 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 18 16 41796 41780 145.052 118.562 0.00793786 0.00688854 2024-09-13T15:03:13.262 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:13.271 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:12 smithi151 ceph-mon[108966]: pgmap v2255: 119 pgs: 119 active+clean; 2.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-13T15:03:13.271 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:12 smithi151 ceph-mon[104450]: pgmap v2255: 119 pgs: 119 active+clean; 2.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-13T15:03:13.300 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:13.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:13.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:13.333 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:12 smithi184 ceph-mon[92527]: pgmap v2255: 119 pgs: 119 active+clean; 2.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-13T15:03:13.402 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:13.447 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:13.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:13.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:13.549 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:13.592 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:13.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:13.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:13.694 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:13.734 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:13.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:13.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:13.764 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T15:03:13.764 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:03:13.836 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:13.866 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:13.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:13.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:13.968 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:13.998 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:13.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:13.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:14.100 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:14.128 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:14.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:14.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:14.174 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 19 16 43584 43568 143.299 111.75 0.0121168 0.00697369 2024-09-13T15:03:14.231 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:14.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:14.269 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:14.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:14.370 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:14.405 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:14.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:14.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:14.492 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T15:03:14.507 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:14.541 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:14.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:14.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:14.643 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:14.676 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:14.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:14.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:14.777 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:14.813 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:14.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:14.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:14.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:03:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:03:14] "GET /metrics HTTP/1.1" 200 36700 "" "Prometheus/2.51.0" 2024-09-13T15:03:14.880 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:03:14.915 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:14.931 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-13T15:03:14.962 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:14.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:14.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:15.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:15.100 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:15.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:15.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:15.174 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:2024-09-13T15:03:15.174409+0000 min lat: 0.00120778 max lat: 0.137226 avg lat: 0.00705068 2024-09-13T15:03:15.175 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T15:03:15.175 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 20 16 45370 45354 141.71 111.625 0.00901723 0.00705068 2024-09-13T15:03:15.202 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:15.247 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:15.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:15.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:15.348 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:15.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:14 smithi184 ceph-mon[92527]: pgmap v2256: 119 pgs: 119 active+clean; 2.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 139 MiB/s wr, 2.23k op/s 2024-09-13T15:03:15.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:14 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1884691338' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:15.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:14 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:15.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:14 smithi151 ceph-mon[104450]: pgmap v2256: 119 pgs: 119 active+clean; 2.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 139 MiB/s wr, 2.23k op/s 2024-09-13T15:03:15.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:14 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1884691338' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:15.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:14 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:15.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:14 smithi151 ceph-mon[108966]: pgmap v2256: 119 pgs: 119 active+clean; 2.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 139 MiB/s wr, 2.23k op/s 2024-09-13T15:03:15.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:14 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1884691338' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:15.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:14 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:15.383 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:15.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:15.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:15.485 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:15.515 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:15.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:15.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:15.616 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:15.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:15.659 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:15.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:15.760 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:15.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:15.798 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:15.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:15.899 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:15.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:15.936 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:15.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:16.037 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:16.070 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:16.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:16.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:16.172 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:16.175 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 21 16 47129 47113 140.197 109.938 0.0060185 0.00712823 2024-09-13T15:03:16.208 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:16.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:16.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:16.310 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:16.351 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:16.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:16.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:16.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:15 smithi184 ceph-mon[92527]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:03:16.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:15 smithi184 ceph-mon[92527]: Cluster is now healthy 2024-09-13T15:03:16.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:15 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:03:16.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:15 smithi184 ceph-mon[92527]: osdmap e1337: 8 total, 8 up, 6 in 2024-09-13T15:03:16.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:15 smithi151 ceph-mon[108966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:03:16.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:15 smithi151 ceph-mon[108966]: Cluster is now healthy 2024-09-13T15:03:16.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:15 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:03:16.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:15 smithi151 ceph-mon[108966]: osdmap e1337: 8 total, 8 up, 6 in 2024-09-13T15:03:16.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:15 smithi151 ceph-mon[104450]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:03:16.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:15 smithi151 ceph-mon[104450]: Cluster is now healthy 2024-09-13T15:03:16.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:15 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:03:16.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:15 smithi151 ceph-mon[104450]: osdmap e1337: 8 total, 8 up, 6 in 2024-09-13T15:03:16.454 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:16.485 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:16.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:16.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:16.586 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:16.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:16.623 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:16.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:16.724 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:16.761 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:16.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:16.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:16.863 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:16.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:16.899 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:16.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:17.000 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:17.034 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:17.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:17.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:17.136 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:17.157 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:17.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:17.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:17.175 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 22 16 48824 48808 138.639 105.938 0.00258644 0.00719257 2024-09-13T15:03:17.259 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:17.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:17.300 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:17.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:17.401 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:17.423 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T15:03:17.423 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:03:17.426 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:17.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:17.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:17.527 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:17.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:17.558 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:17.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:17.659 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:17.682 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:17 smithi151 ceph-mon[108966]: pgmap v2258: 119 pgs: 119 active+clean; 2.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-09-13T15:03:17.682 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:03:17.682 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:17.682 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:17 smithi151 ceph-mon[104450]: pgmap v2258: 119 pgs: 119 active+clean; 2.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-09-13T15:03:17.682 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:03:17.682 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:17.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:17.690 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:17.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:17.791 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:17.805 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:17 smithi184 ceph-mon[92527]: pgmap v2258: 119 pgs: 119 active+clean; 2.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-09-13T15:03:17.805 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:03:17.805 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:17.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:17.828 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:17.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:17.929 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:17.968 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:17.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:17.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:18.070 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:18.101 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:18.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:18.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:18.175 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 23 16 50240 50224 136.459 88.5 0.00769244 0.00732254 2024-09-13T15:03:18.202 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:18.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:18.244 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:18.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:18.345 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:18.388 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:18.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:18.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:18.489 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:18.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:18.519 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:18.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:18.620 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:18.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:18.642 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:18.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:18.743 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:18.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:18.764 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:18.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:18.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:18 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:03:18.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:18 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:03:18.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:18 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:03:18.866 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:18.886 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:18.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:18.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:18.988 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:19.008 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:19.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:19.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:19.110 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:19.131 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:19.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:19.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:19.175 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 24 16 51148 51132 133.138 56.75 0.00236465 0.00750126 2024-09-13T15:03:19.233 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:19.254 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:19.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:19.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:19.356 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:19.396 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:19.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:19.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:19.498 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:19.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:19.537 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:19.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:19.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:19.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:19.677 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:19.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:19.754 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:19 smithi184 ceph-mon[92527]: pgmap v2259: 119 pgs: 119 active+clean; 2.8 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-09-13T15:03:19.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:19 smithi151 ceph-mon[104450]: pgmap v2259: 119 pgs: 119 active+clean; 2.8 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-09-13T15:03:19.768 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:19 smithi151 ceph-mon[108966]: pgmap v2259: 119 pgs: 119 active+clean; 2.8 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-09-13T15:03:19.779 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:19.805 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:19.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:19.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:19.907 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:19.941 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:19.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:19.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:20.021 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T15:03:20.043 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:20.079 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:20.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:20.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:20.175 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 25 16 52405 52389 130.955 78.5625 0.00956132 0.00762973 2024-09-13T15:03:20.222 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:20.262 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T15:03:20.263 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T15:03:20.277 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:20.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:20.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:20.343 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:03:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:03:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:03:20.382 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:20.417 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:20.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:20.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:20.503 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T15:03:20.504 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:03:20.519 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:20.549 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T15:03:20.550 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:03:20.552 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:20.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:20.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:20.583 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T15:03:20.583 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:03:20.614 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T15:03:20.615 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:03:20.645 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T15:03:20.645 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:03:20.654 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:20.677 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T15:03:20.677 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:03:20.699 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:20.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:20.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:20.716 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T15:03:20.716 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:03:20.750 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T15:03:20.750 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:03:20.801 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:20.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:20.850 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:20.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:20.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:20.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:20.992 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:20.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:21.093 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:21.144 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:21.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:21.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:21.155 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:03:21.171 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":2260,"stamp":"2024-09-13T15:03:19.865332+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3278575565,"num_objects":50044,"num_object_clones":0,"num_object_copies":150132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50081,"num_write_kb":3201183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":2051077,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53176,"ondisk_log_size":53176,"up":357,"acting":357,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":357,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":14914464,"kb_used_data":9503088,"kb_used_omap":106,"kb_used_meta":5411221,"kb_avail":547556448,"statfs":{"total":575970213888,"available":560697802752,"internally_reserved":0,"allocated":9731162112,"data_stored":9712958691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":109003,"internal_metadata":5541090869},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1138163712,"num_objects":17367,"num_object_clones":0,"num_object_copies":52101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17367,"num_write_kb":1111488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004726"},"pg_stats":[{"pgid":"1.1d","version":"261'100","reported_seq":257,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.930758+0000","last_change":"2024-09-13T15:00:25.065803+0000","last_active":"2024-09-13T15:03:14.930758+0000","last_peered":"2024-09-13T15:03:14.930758+0000","last_clean":"2024-09-13T15:03:14.930758+0000","last_became_active":"2024-09-13T15:00:25.063785+0000","last_became_peered":"2024-09-13T15:00:25.063785+0000","last_unstale":"2024-09-13T15:03:14.930758+0000","last_undegraded":"2024-09-13T15:03:14.930758+0000","last_fullsized":"2024-09-13T15:03:14.930758+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:17.988345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:17.988345+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:43.674995+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":1961,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552672+0000","last_change":"2024-09-13T15:00:05.045993+0000","last_active":"2024-09-13T15:03:06.552672+0000","last_peered":"2024-09-13T15:03:06.552672+0000","last_clean":"2024-09-13T15:03:06.552672+0000","last_became_active":"2024-09-13T15:00:05.033311+0000","last_became_peered":"2024-09-13T15:00:05.033311+0000","last_unstale":"2024-09-13T15:03:06.552672+0000","last_undegraded":"2024-09-13T15:03:06.552672+0000","last_fullsized":"2024-09-13T15:03:06.552672+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:27.290733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2280,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":160,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.931294+0000","last_change":"2024-09-13T15:00:25.180918+0000","last_active":"2024-09-13T15:03:14.931294+0000","last_peered":"2024-09-13T15:03:14.931294+0000","last_clean":"2024-09-13T15:03:14.931294+0000","last_became_active":"2024-09-13T15:00:25.180571+0000","last_became_peered":"2024-09-13T15:00:25.180571+0000","last_unstale":"2024-09-13T15:03:14.931294+0000","last_undegraded":"2024-09-13T15:03:14.931294+0000","last_fullsized":"2024-09-13T15:03:14.931294+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:31.846606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":396,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.938732+0000","last_change":"2024-09-13T14:45:19.810203+0000","last_active":"2024-09-13T15:03:14.938732+0000","last_peered":"2024-09-13T15:03:14.938732+0000","last_clean":"2024-09-13T15:03:14.938732+0000","last_became_active":"2024-09-13T14:45:19.809693+0000","last_became_peered":"2024-09-13T14:45:19.809693+0000","last_unstale":"2024-09-13T15:03:14.938732+0000","last_undegraded":"2024-09-13T15:03:14.938732+0000","last_fullsized":"2024-09-13T15:03:14.938732+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:20.219673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":402,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.937084+0000","last_change":"2024-09-13T15:00:05.051083+0000","last_active":"2024-09-13T15:03:14.937084+0000","last_peered":"2024-09-13T15:03:14.937084+0000","last_clean":"2024-09-13T15:03:14.937084+0000","last_became_active":"2024-09-13T15:00:05.050737+0000","last_became_peered":"2024-09-13T15:00:05.050737+0000","last_unstale":"2024-09-13T15:03:14.937084+0000","last_undegraded":"2024-09-13T15:03:14.937084+0000","last_fullsized":"2024-09-13T15:03:14.937084+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:58.282196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"261'100","reported_seq":19,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.542812+0000","last_change":"2024-09-13T15:02:42.889565+0000","last_active":"2024-09-13T15:03:06.542812+0000","last_peered":"2024-09-13T15:03:06.542812+0000","last_clean":"2024-09-13T15:03:06.542812+0000","last_became_active":"2024-09-13T15:02:42.888466+0000","last_became_peered":"2024-09-13T15:02:42.888466+0000","last_unstale":"2024-09-13T15:03:06.542812+0000","last_undegraded":"2024-09-13T15:03:06.542812+0000","last_fullsized":"2024-09-13T15:03:06.542812+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_clean_scrub_stamp":"2024-09-13T14:44:54.259482+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:19:31.567604+0000","scrub_duration":0.015265569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":128,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552376+0000","last_change":"2024-09-13T15:00:25.072110+0000","last_active":"2024-09-13T15:03:06.552376+0000","last_peered":"2024-09-13T15:03:06.552376+0000","last_clean":"2024-09-13T15:03:06.552376+0000","last_became_active":"2024-09-13T15:00:25.045594+0000","last_became_peered":"2024-09-13T15:00:25.045594+0000","last_unstale":"2024-09-13T15:03:06.552376+0000","last_undegraded":"2024-09-13T15:03:06.552376+0000","last_fullsized":"2024-09-13T15:03:06.552376+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:59:45.210187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":183,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.940883+0000","last_change":"2024-09-13T15:00:25.052537+0000","last_active":"2024-09-13T15:03:14.940883+0000","last_peered":"2024-09-13T15:03:14.940883+0000","last_clean":"2024-09-13T15:03:14.940883+0000","last_became_active":"2024-09-13T15:00:25.052225+0000","last_became_peered":"2024-09-13T15:00:25.052225+0000","last_unstale":"2024-09-13T15:03:14.940883+0000","last_undegraded":"2024-09-13T15:03:14.940883+0000","last_fullsized":"2024-09-13T15:03:14.940883+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:52.039052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1e","version":"261'100","reported_seq":413,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.939733+0000","last_change":"2024-09-13T14:45:35.947332+0000","last_active":"2024-09-13T15:03:14.939733+0000","last_peered":"2024-09-13T15:03:14.939733+0000","last_clean":"2024-09-13T15:03:14.939733+0000","last_became_active":"2024-09-13T14:45:19.810775+0000","last_became_peered":"2024-09-13T14:45:19.810775+0000","last_unstale":"2024-09-13T15:03:14.939733+0000","last_undegraded":"2024-09-13T15:03:14.939733+0000","last_fullsized":"2024-09-13T15:03:14.939733+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:35.947265+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:43:41.569795+0000","last_clean_scrub_stamp":"2024-09-13T14:45:35.947265+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:00.606370+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":128,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552251+0000","last_change":"2024-09-13T15:00:25.192972+0000","last_active":"2024-09-13T15:03:06.552251+0000","last_peered":"2024-09-13T15:03:06.552251+0000","last_clean":"2024-09-13T15:03:06.552251+0000","last_became_active":"2024-09-13T15:00:25.047858+0000","last_became_peered":"2024-09-13T15:00:25.047858+0000","last_unstale":"2024-09-13T15:03:06.552251+0000","last_undegraded":"2024-09-13T15:03:06.552251+0000","last_fullsized":"2024-09-13T15:03:06.552251+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:22:16.795970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1321,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.937968+0000","last_change":"2024-09-13T15:00:25.086077+0000","last_active":"2024-09-13T15:03:14.937968+0000","last_peered":"2024-09-13T15:03:14.937968+0000","last_clean":"2024-09-13T15:03:14.937968+0000","last_became_active":"2024-09-13T15:00:25.085685+0000","last_became_peered":"2024-09-13T15:00:25.085685+0000","last_unstale":"2024-09-13T15:03:14.937968+0000","last_undegraded":"2024-09-13T15:03:14.937968+0000","last_fullsized":"2024-09-13T15:03:14.937968+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:09:24.157291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":14,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.931568+0000","last_change":"2024-09-13T15:03:06.560718+0000","last_active":"2024-09-13T15:03:14.931568+0000","last_peered":"2024-09-13T15:03:14.931568+0000","last_clean":"2024-09-13T15:03:14.931568+0000","last_became_active":"2024-09-13T15:03:06.560348+0000","last_became_peered":"2024-09-13T15:03:06.560348+0000","last_unstale":"2024-09-13T15:03:14.931568+0000","last_undegraded":"2024-09-13T15:03:14.931568+0000","last_fullsized":"2024-09-13T15:03:14.931568+0000","mapping_epoch":1335,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:19:03.944074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":52,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.933481+0000","last_change":"2024-09-13T15:02:01.133848+0000","last_active":"2024-09-13T15:03:14.933481+0000","last_peered":"2024-09-13T15:03:14.933481+0000","last_clean":"2024-09-13T15:03:14.933481+0000","last_became_active":"2024-09-13T15:02:01.133414+0000","last_became_peered":"2024-09-13T15:02:01.133414+0000","last_unstale":"2024-09-13T15:03:14.933481+0000","last_undegraded":"2024-09-13T15:03:14.933481+0000","last_fullsized":"2024-09-13T15:03:14.933481+0000","mapping_epoch":1316,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:51:09.222615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"261'100","reported_seq":416,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.937186+0000","last_change":"2024-09-13T15:02:42.900466+0000","last_active":"2024-09-13T15:03:14.937186+0000","last_peered":"2024-09-13T15:03:14.937186+0000","last_clean":"2024-09-13T15:03:14.937186+0000","last_became_active":"2024-09-13T15:02:42.900166+0000","last_became_peered":"2024-09-13T15:02:42.900166+0000","last_unstale":"2024-09-13T15:03:14.937186+0000","last_undegraded":"2024-09-13T15:03:14.937186+0000","last_fullsized":"2024-09-13T15:03:14.937186+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.614340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.614340+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:52.087366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":136,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.928463+0000","last_change":"2024-09-13T15:02:42.897194+0000","last_active":"2024-09-13T15:03:14.928463+0000","last_peered":"2024-09-13T15:03:14.928463+0000","last_clean":"2024-09-13T15:03:14.928463+0000","last_became_active":"2024-09-13T15:02:42.896712+0000","last_became_peered":"2024-09-13T15:02:42.896712+0000","last_unstale":"2024-09-13T15:03:14.928463+0000","last_undegraded":"2024-09-13T15:03:14.928463+0000","last_fullsized":"2024-09-13T15:03:14.928463+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:13:29.973653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":437,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.937209+0000","last_change":"2024-09-13T15:00:25.052382+0000","last_active":"2024-09-13T15:03:14.937209+0000","last_peered":"2024-09-13T15:03:14.937209+0000","last_clean":"2024-09-13T15:03:14.937209+0000","last_became_active":"2024-09-13T15:00:25.051999+0000","last_became_peered":"2024-09-13T15:00:25.051999+0000","last_unstale":"2024-09-13T15:03:14.937209+0000","last_undegraded":"2024-09-13T15:03:14.937209+0000","last_fullsized":"2024-09-13T15:03:14.937209+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:30.490420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.18","version":"261'100","reported_seq":145,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552048+0000","last_change":"2024-09-13T15:02:42.914596+0000","last_active":"2024-09-13T15:03:06.552048+0000","last_peered":"2024-09-13T15:03:06.552048+0000","last_clean":"2024-09-13T15:03:06.552048+0000","last_became_active":"2024-09-13T15:02:42.886984+0000","last_became_peered":"2024-09-13T15:02:42.886984+0000","last_unstale":"2024-09-13T15:03:06.552048+0000","last_undegraded":"2024-09-13T15:03:06.552048+0000","last_fullsized":"2024-09-13T15:03:06.552048+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:56.548855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:56.548855+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:30.761139+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1050752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":257,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.929447+0000","last_change":"2024-09-13T15:00:25.048396+0000","last_active":"2024-09-13T15:03:14.929447+0000","last_peered":"2024-09-13T15:03:14.929447+0000","last_clean":"2024-09-13T15:03:14.929447+0000","last_became_active":"2024-09-13T15:00:25.048084+0000","last_became_peered":"2024-09-13T15:00:25.048084+0000","last_unstale":"2024-09-13T15:03:14.929447+0000","last_undegraded":"2024-09-13T15:03:14.929447+0000","last_fullsized":"2024-09-13T15:03:14.929447+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:05:23.139064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.543667+0000","last_change":"2024-09-13T15:02:42.895757+0000","last_active":"2024-09-13T15:03:06.543667+0000","last_peered":"2024-09-13T15:03:06.543667+0000","last_clean":"2024-09-13T15:03:06.543667+0000","last_became_active":"2024-09-13T15:02:42.895416+0000","last_became_peered":"2024-09-13T15:02:42.895416+0000","last_unstale":"2024-09-13T15:03:06.543667+0000","last_undegraded":"2024-09-13T15:03:06.543667+0000","last_fullsized":"2024-09-13T15:03:06.543667+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:50:34.099001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1b","version":"261'100","reported_seq":140,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552687+0000","last_change":"2024-09-13T15:02:42.887234+0000","last_active":"2024-09-13T15:03:06.552687+0000","last_peered":"2024-09-13T15:03:06.552687+0000","last_clean":"2024-09-13T15:03:06.552687+0000","last_became_active":"2024-09-13T15:02:42.886850+0000","last_became_peered":"2024-09-13T15:02:42.886850+0000","last_unstale":"2024-09-13T15:03:06.552687+0000","last_undegraded":"2024-09-13T15:03:06.552687+0000","last_fullsized":"2024-09-13T15:03:06.552687+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:55.988431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:55.988431+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:50:58.012159+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":416,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.936625+0000","last_change":"2024-09-13T15:00:05.047270+0000","last_active":"2024-09-13T15:03:14.936625+0000","last_peered":"2024-09-13T15:03:14.936625+0000","last_clean":"2024-09-13T15:03:14.936625+0000","last_became_active":"2024-09-13T15:00:05.046710+0000","last_became_peered":"2024-09-13T15:00:05.046710+0000","last_unstale":"2024-09-13T15:03:14.936625+0000","last_undegraded":"2024-09-13T15:03:14.936625+0000","last_fullsized":"2024-09-13T15:03:14.936625+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:46:06.874757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":110,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552782+0000","last_change":"2024-09-13T14:55:41.883426+0000","last_active":"2024-09-13T15:03:06.552782+0000","last_peered":"2024-09-13T15:03:06.552782+0000","last_clean":"2024-09-13T15:03:06.552782+0000","last_became_active":"2024-09-13T14:55:41.880724+0000","last_became_peered":"2024-09-13T14:55:41.880724+0000","last_unstale":"2024-09-13T15:03:06.552782+0000","last_undegraded":"2024-09-13T15:03:06.552782+0000","last_fullsized":"2024-09-13T15:03:06.552782+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:17.920508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1a","version":"261'100","reported_seq":19,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.543668+0000","last_change":"2024-09-13T15:02:42.887267+0000","last_active":"2024-09-13T15:03:06.543668+0000","last_peered":"2024-09-13T15:03:06.543668+0000","last_clean":"2024-09-13T15:03:06.543668+0000","last_became_active":"2024-09-13T15:02:42.886976+0000","last_became_peered":"2024-09-13T15:02:42.886976+0000","last_unstale":"2024-09-13T15:03:06.543668+0000","last_undegraded":"2024-09-13T15:03:06.543668+0000","last_fullsized":"2024-09-13T15:03:06.543668+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:46.985510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:46.985510+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:10:55.966984+0000","scrub_duration":0.0030525999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":416,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.936716+0000","last_change":"2024-09-13T14:54:05.606882+0000","last_active":"2024-09-13T15:03:14.936716+0000","last_peered":"2024-09-13T15:03:14.936716+0000","last_clean":"2024-09-13T15:03:14.936716+0000","last_became_active":"2024-09-13T14:54:05.605912+0000","last_became_peered":"2024-09-13T14:54:05.605912+0000","last_unstale":"2024-09-13T15:03:14.936716+0000","last_undegraded":"2024-09-13T15:03:14.936716+0000","last_fullsized":"2024-09-13T15:03:14.936716+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:05.445329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1976,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552657+0000","last_change":"2024-09-13T15:00:05.037972+0000","last_active":"2024-09-13T15:03:06.552657+0000","last_peered":"2024-09-13T15:03:06.552657+0000","last_clean":"2024-09-13T15:03:06.552657+0000","last_became_active":"2024-09-13T15:00:05.037561+0000","last_became_peered":"2024-09-13T15:00:05.037561+0000","last_unstale":"2024-09-13T15:03:06.552657+0000","last_undegraded":"2024-09-13T15:03:06.552657+0000","last_fullsized":"2024-09-13T15:03:06.552657+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:42:04.578024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.15","version":"261'100","reported_seq":19,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.542601+0000","last_change":"2024-09-13T15:02:42.892087+0000","last_active":"2024-09-13T15:03:06.542601+0000","last_peered":"2024-09-13T15:03:06.542601+0000","last_clean":"2024-09-13T15:03:06.542601+0000","last_became_active":"2024-09-13T15:02:42.890787+0000","last_became_peered":"2024-09-13T15:02:42.890787+0000","last_unstale":"2024-09-13T15:03:06.542601+0000","last_undegraded":"2024-09-13T15:03:06.542601+0000","last_fullsized":"2024-09-13T15:03:06.542601+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:38.214865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:38.214865+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:12:04.179567+0000","scrub_duration":0.0016389639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1981,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.942731+0000","last_change":"2024-09-13T15:02:42.901601+0000","last_active":"2024-09-13T15:03:14.942731+0000","last_peered":"2024-09-13T15:03:14.942731+0000","last_clean":"2024-09-13T15:03:14.942731+0000","last_became_active":"2024-09-13T15:02:42.900518+0000","last_became_peered":"2024-09-13T15:02:42.900518+0000","last_unstale":"2024-09-13T15:03:14.942731+0000","last_undegraded":"2024-09-13T15:03:14.942731+0000","last_fullsized":"2024-09-13T15:03:14.942731+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:57:33.494780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":130,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552406+0000","last_change":"2024-09-13T15:02:42.885564+0000","last_active":"2024-09-13T15:03:06.552406+0000","last_peered":"2024-09-13T15:03:06.552406+0000","last_clean":"2024-09-13T15:03:06.552406+0000","last_became_active":"2024-09-13T15:02:42.884990+0000","last_became_peered":"2024-09-13T15:02:42.884990+0000","last_unstale":"2024-09-13T15:03:06.552406+0000","last_undegraded":"2024-09-13T15:03:06.552406+0000","last_fullsized":"2024-09-13T15:03:06.552406+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:51:42.556111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"261'100","reported_seq":493,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.933692+0000","last_change":"2024-09-13T15:02:42.884301+0000","last_active":"2024-09-13T15:03:14.933692+0000","last_peered":"2024-09-13T15:03:14.933692+0000","last_clean":"2024-09-13T15:03:14.933692+0000","last_became_active":"2024-09-13T15:02:42.883961+0000","last_became_peered":"2024-09-13T15:02:42.883961+0000","last_unstale":"2024-09-13T15:03:14.933692+0000","last_undegraded":"2024-09-13T15:03:14.933692+0000","last_fullsized":"2024-09-13T15:03:14.933692+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.037517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.037517+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:09:38.307408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":25,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.542520+0000","last_change":"2024-09-13T15:02:42.922069+0000","last_active":"2024-09-13T15:03:06.542520+0000","last_peered":"2024-09-13T15:03:06.542520+0000","last_clean":"2024-09-13T15:03:06.542520+0000","last_became_active":"2024-09-13T15:02:42.893632+0000","last_became_peered":"2024-09-13T15:02:42.893632+0000","last_unstale":"2024-09-13T15:03:06.542520+0000","last_undegraded":"2024-09-13T15:03:06.542520+0000","last_fullsized":"2024-09-13T15:03:06.542520+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:51:16.796405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":270,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":243,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.928225+0000","last_change":"2024-09-13T14:54:05.614569+0000","last_active":"2024-09-13T15:03:14.928225+0000","last_peered":"2024-09-13T15:03:14.928225+0000","last_clean":"2024-09-13T15:03:14.928225+0000","last_became_active":"2024-09-13T14:54:05.611810+0000","last_became_peered":"2024-09-13T14:54:05.611810+0000","last_unstale":"2024-09-13T15:03:14.928225+0000","last_undegraded":"2024-09-13T15:03:14.928225+0000","last_fullsized":"2024-09-13T15:03:14.928225+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:46:02.721438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"261'100","reported_seq":130,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552769+0000","last_change":"2024-09-13T15:02:42.887347+0000","last_active":"2024-09-13T15:03:06.552769+0000","last_peered":"2024-09-13T15:03:06.552769+0000","last_clean":"2024-09-13T15:03:06.552769+0000","last_became_active":"2024-09-13T15:02:42.886986+0000","last_became_peered":"2024-09-13T15:02:42.886986+0000","last_unstale":"2024-09-13T15:03:06.552769+0000","last_undegraded":"2024-09-13T15:03:06.552769+0000","last_fullsized":"2024-09-13T15:03:06.552769+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:54.933410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:54.933410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:18.197248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":29,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.543657+0000","last_change":"2024-09-13T15:02:42.912172+0000","last_active":"2024-09-13T15:03:06.543657+0000","last_peered":"2024-09-13T15:03:06.543657+0000","last_clean":"2024-09-13T15:03:06.543657+0000","last_became_active":"2024-09-13T15:02:42.880043+0000","last_became_peered":"2024-09-13T15:02:42.880043+0000","last_unstale":"2024-09-13T15:03:06.543657+0000","last_undegraded":"2024-09-13T15:03:06.543657+0000","last_fullsized":"2024-09-13T15:03:06.543657+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:46:15.229979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14232,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":129,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.929176+0000","last_change":"2024-09-13T15:00:25.091881+0000","last_active":"2024-09-13T15:03:14.929176+0000","last_peered":"2024-09-13T15:03:14.929176+0000","last_clean":"2024-09-13T15:03:14.929176+0000","last_became_active":"2024-09-13T15:00:25.091538+0000","last_became_peered":"2024-09-13T15:00:25.091538+0000","last_unstale":"2024-09-13T15:03:14.929176+0000","last_undegraded":"2024-09-13T15:03:14.929176+0000","last_fullsized":"2024-09-13T15:03:14.929176+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:29:32.889528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.16","version":"261'100","reported_seq":257,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.931330+0000","last_change":"2024-09-13T15:00:25.083905+0000","last_active":"2024-09-13T15:03:14.931330+0000","last_peered":"2024-09-13T15:03:14.931330+0000","last_clean":"2024-09-13T15:03:14.931330+0000","last_became_active":"2024-09-13T15:00:25.082642+0000","last_became_peered":"2024-09-13T15:00:25.082642+0000","last_unstale":"2024-09-13T15:03:14.931330+0000","last_undegraded":"2024-09-13T15:03:14.931330+0000","last_fullsized":"2024-09-13T15:03:14.931330+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:48.363045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:48.363045+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:31:38.562138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.e","version":"1337'3033","reported_seq":3059,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:17.286408+0000","last_change":"2024-09-13T15:02:44.891854+0000","last_active":"2024-09-13T15:03:17.286408+0000","last_peered":"2024-09-13T15:03:17.286408+0000","last_clean":"2024-09-13T15:03:17.286408+0000","last_became_active":"2024-09-13T15:02:44.891563+0000","last_became_peered":"2024-09-13T15:02:44.891563+0000","last_unstale":"2024-09-13T15:03:17.286408+0000","last_undegraded":"2024-09-13T15:03:17.286408+0000","last_fullsized":"2024-09-13T15:03:17.286408+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3033,"log_dups_size":0,"ondisk_log_size":3033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:18:05.783586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198770688,"num_objects":3033,"num_object_clones":0,"num_object_copies":9099,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3033,"num_write_kb":194112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":100,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.551906+0000","last_change":"2024-09-13T15:02:42.884334+0000","last_active":"2024-09-13T15:03:06.551906+0000","last_peered":"2024-09-13T15:03:06.551906+0000","last_clean":"2024-09-13T15:03:06.551906+0000","last_became_active":"2024-09-13T15:02:42.884006+0000","last_became_peered":"2024-09-13T15:02:42.884006+0000","last_unstale":"2024-09-13T15:03:06.551906+0000","last_undegraded":"2024-09-13T15:03:06.551906+0000","last_fullsized":"2024-09-13T15:03:06.551906+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:09.872495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.544579+0000","last_change":"2024-09-13T15:02:42.882332+0000","last_active":"2024-09-13T15:03:06.544579+0000","last_peered":"2024-09-13T15:03:06.544579+0000","last_clean":"2024-09-13T15:03:06.544579+0000","last_became_active":"2024-09-13T15:02:42.881175+0000","last_became_peered":"2024-09-13T15:02:42.881175+0000","last_unstale":"2024-09-13T15:03:06.544579+0000","last_undegraded":"2024-09-13T15:03:06.544579+0000","last_fullsized":"2024-09-13T15:03:06.544579+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:52.763730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":2028,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.929733+0000","last_change":"2024-09-13T15:02:42.907810+0000","last_active":"2024-09-13T15:03:14.929733+0000","last_peered":"2024-09-13T15:03:14.929733+0000","last_clean":"2024-09-13T15:03:14.929733+0000","last_became_active":"2024-09-13T15:02:42.901460+0000","last_became_peered":"2024-09-13T15:02:42.901460+0000","last_unstale":"2024-09-13T15:03:14.929733+0000","last_undegraded":"2024-09-13T15:03:14.929733+0000","last_fullsized":"2024-09-13T15:03:14.929733+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:57:16.999581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"261'100","reported_seq":438,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.937642+0000","last_change":"2024-09-13T15:02:42.899622+0000","last_active":"2024-09-13T15:03:14.937642+0000","last_peered":"2024-09-13T15:03:14.937642+0000","last_clean":"2024-09-13T15:03:14.937642+0000","last_became_active":"2024-09-13T15:02:42.897715+0000","last_became_peered":"2024-09-13T15:02:42.897715+0000","last_unstale":"2024-09-13T15:03:14.937642+0000","last_undegraded":"2024-09-13T15:03:14.937642+0000","last_fullsized":"2024-09-13T15:03:14.937642+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.710192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.710192+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:50:03.858097+0000","scrub_duration":0.0019250910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"1337'3039","reported_seq":3055,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:17.393430+0000","last_change":"2024-09-13T15:02:44.893937+0000","last_active":"2024-09-13T15:03:17.393430+0000","last_peered":"2024-09-13T15:03:17.393430+0000","last_clean":"2024-09-13T15:03:17.393430+0000","last_became_active":"2024-09-13T15:02:44.893588+0000","last_became_peered":"2024-09-13T15:02:44.893588+0000","last_unstale":"2024-09-13T15:03:17.393430+0000","last_undegraded":"2024-09-13T15:03:17.393430+0000","last_fullsized":"2024-09-13T15:03:17.393430+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3039,"log_dups_size":0,"ondisk_log_size":3039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:25.119259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199163904,"num_objects":3039,"num_object_clones":0,"num_object_copies":9117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3039,"num_write_kb":194496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":320,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.936484+0000","last_change":"2024-09-13T15:02:42.896729+0000","last_active":"2024-09-13T15:03:14.936484+0000","last_peered":"2024-09-13T15:03:14.936484+0000","last_clean":"2024-09-13T15:03:14.936484+0000","last_became_active":"2024-09-13T15:02:42.896414+0000","last_became_peered":"2024-09-13T15:02:42.896414+0000","last_unstale":"2024-09-13T15:03:14.936484+0000","last_undegraded":"2024-09-13T15:03:14.936484+0000","last_fullsized":"2024-09-13T15:03:14.936484+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:53:32.471811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":258,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.931424+0000","last_change":"2024-09-13T15:00:25.070169+0000","last_active":"2024-09-13T15:03:14.931424+0000","last_peered":"2024-09-13T15:03:14.931424+0000","last_clean":"2024-09-13T15:03:14.931424+0000","last_became_active":"2024-09-13T15:00:25.069849+0000","last_became_peered":"2024-09-13T15:00:25.069849+0000","last_unstale":"2024-09-13T15:03:14.931424+0000","last_undegraded":"2024-09-13T15:03:14.931424+0000","last_fullsized":"2024-09-13T15:03:14.931424+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:56.491207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":2044,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.938439+0000","last_change":"2024-09-13T15:02:42.912751+0000","last_active":"2024-09-13T15:03:14.938439+0000","last_peered":"2024-09-13T15:03:14.938439+0000","last_clean":"2024-09-13T15:03:14.938439+0000","last_became_active":"2024-09-13T15:02:42.903177+0000","last_became_peered":"2024-09-13T15:02:42.903177+0000","last_unstale":"2024-09-13T15:03:14.938439+0000","last_undegraded":"2024-09-13T15:03:14.938439+0000","last_fullsized":"2024-09-13T15:03:14.938439+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:00:52.788057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"261'100","reported_seq":250,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.932353+0000","last_change":"2024-09-13T15:00:25.060018+0000","last_active":"2024-09-13T15:03:14.932353+0000","last_peered":"2024-09-13T15:03:14.932353+0000","last_clean":"2024-09-13T15:03:14.932353+0000","last_became_active":"2024-09-13T15:00:25.059438+0000","last_became_peered":"2024-09-13T15:00:25.059438+0000","last_unstale":"2024-09-13T15:03:14.932353+0000","last_undegraded":"2024-09-13T15:03:14.932353+0000","last_fullsized":"2024-09-13T15:03:14.932353+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:51:31.310115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:51:31.310115+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:08.861237+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.c","version":"1337'3167","reported_seq":3194,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:17.272885+0000","last_change":"2024-09-13T15:02:44.894339+0000","last_active":"2024-09-13T15:03:17.272885+0000","last_peered":"2024-09-13T15:03:17.272885+0000","last_clean":"2024-09-13T15:03:17.272885+0000","last_became_active":"2024-09-13T15:02:44.893967+0000","last_became_peered":"2024-09-13T15:02:44.893967+0000","last_unstale":"2024-09-13T15:03:17.272885+0000","last_undegraded":"2024-09-13T15:03:17.272885+0000","last_fullsized":"2024-09-13T15:03:17.272885+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3167,"log_dups_size":0,"ondisk_log_size":3167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:34.261719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207552512,"num_objects":3167,"num_object_clones":0,"num_object_copies":9501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3167,"num_write_kb":202688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2054,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552575+0000","last_change":"2024-09-13T15:02:42.885186+0000","last_active":"2024-09-13T15:03:06.552575+0000","last_peered":"2024-09-13T15:03:06.552575+0000","last_clean":"2024-09-13T15:03:06.552575+0000","last_became_active":"2024-09-13T15:02:42.884666+0000","last_became_peered":"2024-09-13T15:02:42.884666+0000","last_unstale":"2024-09-13T15:03:06.552575+0000","last_undegraded":"2024-09-13T15:03:06.552575+0000","last_fullsized":"2024-09-13T15:03:06.552575+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:17.386836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.542680+0000","last_change":"2024-09-13T15:02:42.891180+0000","last_active":"2024-09-13T15:03:06.542680+0000","last_peered":"2024-09-13T15:03:06.542680+0000","last_clean":"2024-09-13T15:03:06.542680+0000","last_became_active":"2024-09-13T15:02:42.890141+0000","last_became_peered":"2024-09-13T15:02:42.890141+0000","last_unstale":"2024-09-13T15:03:06.542680+0000","last_undegraded":"2024-09-13T15:03:06.542680+0000","last_fullsized":"2024-09-13T15:03:06.542680+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:28.910151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":258,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.930775+0000","last_change":"2024-09-13T14:55:41.892125+0000","last_active":"2024-09-13T15:03:14.930775+0000","last_peered":"2024-09-13T15:03:14.930775+0000","last_clean":"2024-09-13T15:03:14.930775+0000","last_became_active":"2024-09-13T14:55:41.891812+0000","last_became_peered":"2024-09-13T14:55:41.891812+0000","last_unstale":"2024-09-13T15:03:14.930775+0000","last_undegraded":"2024-09-13T15:03:14.930775+0000","last_fullsized":"2024-09-13T15:03:14.930775+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:03.359541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"261'100","reported_seq":503,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.938641+0000","last_change":"2024-09-13T15:02:42.898674+0000","last_active":"2024-09-13T15:03:14.938641+0000","last_peered":"2024-09-13T15:03:14.938641+0000","last_clean":"2024-09-13T15:03:14.938641+0000","last_became_active":"2024-09-13T15:02:42.898289+0000","last_became_peered":"2024-09-13T15:02:42.898289+0000","last_unstale":"2024-09-13T15:03:14.938641+0000","last_undegraded":"2024-09-13T15:03:14.938641+0000","last_fullsized":"2024-09-13T15:03:14.938641+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:33.574443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:33.574443+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:54:37.955957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"1337'3193","reported_seq":3220,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:18.393966+0000","last_change":"2024-09-13T15:02:45.496583+0000","last_active":"2024-09-13T15:03:18.393966+0000","last_peered":"2024-09-13T15:03:18.393966+0000","last_clean":"2024-09-13T15:03:18.393966+0000","last_became_active":"2024-09-13T15:02:45.496353+0000","last_became_peered":"2024-09-13T15:02:45.496353+0000","last_unstale":"2024-09-13T15:03:18.393966+0000","last_undegraded":"2024-09-13T15:03:18.393966+0000","last_fullsized":"2024-09-13T15:03:18.393966+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3193,"log_dups_size":0,"ondisk_log_size":3193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:09:02.073789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209256448,"num_objects":3193,"num_object_clones":0,"num_object_copies":9579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3193,"num_write_kb":204352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":272,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.928370+0000","last_change":"2024-09-13T15:00:25.079609+0000","last_active":"2024-09-13T15:03:14.928370+0000","last_peered":"2024-09-13T15:03:14.928370+0000","last_clean":"2024-09-13T15:03:14.928370+0000","last_became_active":"2024-09-13T15:00:25.077887+0000","last_became_peered":"2024-09-13T15:00:25.077887+0000","last_unstale":"2024-09-13T15:03:14.928370+0000","last_undegraded":"2024-09-13T15:03:14.928370+0000","last_fullsized":"2024-09-13T15:03:14.928370+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:13:04.513297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":265,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.930602+0000","last_change":"2024-09-13T15:02:42.899923+0000","last_active":"2024-09-13T15:03:14.930602+0000","last_peered":"2024-09-13T15:03:14.930602+0000","last_clean":"2024-09-13T15:03:14.930602+0000","last_became_active":"2024-09-13T15:02:42.898278+0000","last_became_peered":"2024-09-13T15:02:42.898278+0000","last_unstale":"2024-09-13T15:03:14.930602+0000","last_undegraded":"2024-09-13T15:03:14.930602+0000","last_fullsized":"2024-09-13T15:03:14.930602+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:39:54.048710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":110,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552290+0000","last_change":"2024-09-13T14:55:41.839293+0000","last_active":"2024-09-13T15:03:06.552290+0000","last_peered":"2024-09-13T15:03:06.552290+0000","last_clean":"2024-09-13T15:03:06.552290+0000","last_became_active":"2024-09-13T14:55:41.838820+0000","last_became_peered":"2024-09-13T14:55:41.838820+0000","last_unstale":"2024-09-13T15:03:06.552290+0000","last_undegraded":"2024-09-13T15:03:06.552290+0000","last_fullsized":"2024-09-13T15:03:06.552290+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:14:27.811437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"261'100","reported_seq":427,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.938288+0000","last_change":"2024-09-13T15:02:42.903818+0000","last_active":"2024-09-13T15:03:14.938288+0000","last_peered":"2024-09-13T15:03:14.938288+0000","last_clean":"2024-09-13T15:03:14.938288+0000","last_became_active":"2024-09-13T15:02:42.902705+0000","last_became_peered":"2024-09-13T15:02:42.902705+0000","last_unstale":"2024-09-13T15:03:14.938288+0000","last_undegraded":"2024-09-13T15:03:14.938288+0000","last_fullsized":"2024-09-13T15:03:14.938288+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:36.965775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:45:36.965775+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:40:58.317613+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.b","version":"1337'3189","reported_seq":3215,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:18.816646+0000","last_change":"2024-09-13T15:02:44.895572+0000","last_active":"2024-09-13T15:03:18.816646+0000","last_peered":"2024-09-13T15:03:18.816646+0000","last_clean":"2024-09-13T15:03:18.816646+0000","last_became_active":"2024-09-13T15:02:44.895300+0000","last_became_peered":"2024-09-13T15:02:44.895300+0000","last_unstale":"2024-09-13T15:03:18.816646+0000","last_undegraded":"2024-09-13T15:03:18.816646+0000","last_fullsized":"2024-09-13T15:03:18.816646+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3189,"log_dups_size":0,"ondisk_log_size":3189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:46:47.154024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208994304,"num_objects":3189,"num_object_clones":0,"num_object_copies":9567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3189,"num_write_kb":204096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":244,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.930517+0000","last_change":"2024-09-13T14:54:05.596068+0000","last_active":"2024-09-13T15:03:14.930517+0000","last_peered":"2024-09-13T15:03:14.930517+0000","last_clean":"2024-09-13T15:03:14.930517+0000","last_became_active":"2024-09-13T14:54:05.595743+0000","last_became_peered":"2024-09-13T14:54:05.595743+0000","last_unstale":"2024-09-13T15:03:14.930517+0000","last_undegraded":"2024-09-13T15:03:14.930517+0000","last_fullsized":"2024-09-13T15:03:14.930517+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:15:39.660370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":649,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.546483+0000","last_change":"2024-09-13T15:00:25.089570+0000","last_active":"2024-09-13T15:03:06.546483+0000","last_peered":"2024-09-13T15:03:06.546483+0000","last_clean":"2024-09-13T15:03:06.546483+0000","last_became_active":"2024-09-13T15:00:25.088373+0000","last_became_peered":"2024-09-13T15:00:25.088373+0000","last_unstale":"2024-09-13T15:03:06.546483+0000","last_undegraded":"2024-09-13T15:03:06.546483+0000","last_fullsized":"2024-09-13T15:03:06.546483+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:36:04.220868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":159,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552364+0000","last_change":"2024-09-13T14:55:41.889668+0000","last_active":"2024-09-13T15:03:06.552364+0000","last_peered":"2024-09-13T15:03:06.552364+0000","last_clean":"2024-09-13T15:03:06.552364+0000","last_became_active":"2024-09-13T14:55:41.889195+0000","last_became_peered":"2024-09-13T14:55:41.889195+0000","last_unstale":"2024-09-13T15:03:06.552364+0000","last_undegraded":"2024-09-13T15:03:06.552364+0000","last_fullsized":"2024-09-13T15:03:06.552364+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:52.765208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"261'100","reported_seq":140,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552026+0000","last_change":"2024-09-13T15:02:42.878126+0000","last_active":"2024-09-13T15:03:06.552026+0000","last_peered":"2024-09-13T15:03:06.552026+0000","last_clean":"2024-09-13T15:03:06.552026+0000","last_became_active":"2024-09-13T15:02:42.877740+0000","last_became_peered":"2024-09-13T15:02:42.877740+0000","last_unstale":"2024-09-13T15:03:06.552026+0000","last_undegraded":"2024-09-13T15:03:06.552026+0000","last_fullsized":"2024-09-13T15:03:06.552026+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:57.024823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:57.024823+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:16.987315+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"1337'3225","reported_seq":3252,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:18.805771+0000","last_change":"2024-09-13T15:02:45.493849+0000","last_active":"2024-09-13T15:03:18.805771+0000","last_peered":"2024-09-13T15:03:18.805771+0000","last_clean":"2024-09-13T15:03:18.805771+0000","last_became_active":"2024-09-13T15:02:45.493251+0000","last_became_peered":"2024-09-13T15:02:45.493251+0000","last_unstale":"2024-09-13T15:03:18.805771+0000","last_undegraded":"2024-09-13T15:03:18.805771+0000","last_fullsized":"2024-09-13T15:03:18.805771+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3225,"log_dups_size":0,"ondisk_log_size":3225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:15.904351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211353600,"num_objects":3225,"num_object_clones":0,"num_object_copies":9675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3225,"num_write_kb":206400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":272,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.929240+0000","last_change":"2024-09-13T15:02:42.897548+0000","last_active":"2024-09-13T15:03:14.929240+0000","last_peered":"2024-09-13T15:03:14.929240+0000","last_clean":"2024-09-13T15:03:14.929240+0000","last_became_active":"2024-09-13T15:02:42.897144+0000","last_became_peered":"2024-09-13T15:02:42.897144+0000","last_unstale":"2024-09-13T15:03:14.929240+0000","last_undegraded":"2024-09-13T15:03:14.929240+0000","last_fullsized":"2024-09-13T15:03:14.929240+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:57.915560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":244,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.930009+0000","last_change":"2024-09-13T14:54:05.614107+0000","last_active":"2024-09-13T15:03:14.930009+0000","last_peered":"2024-09-13T15:03:14.930009+0000","last_clean":"2024-09-13T15:03:14.930009+0000","last_became_active":"2024-09-13T14:54:05.613429+0000","last_became_peered":"2024-09-13T14:54:05.613429+0000","last_unstale":"2024-09-13T15:03:14.930009+0000","last_undegraded":"2024-09-13T15:03:14.930009+0000","last_fullsized":"2024-09-13T15:03:14.930009+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:51.312413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2089,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.931409+0000","last_change":"2024-09-13T15:00:05.042993+0000","last_active":"2024-09-13T15:03:14.931409+0000","last_peered":"2024-09-13T15:03:14.931409+0000","last_clean":"2024-09-13T15:03:14.931409+0000","last_became_active":"2024-09-13T15:00:05.041250+0000","last_became_peered":"2024-09-13T15:00:05.041250+0000","last_unstale":"2024-09-13T15:03:14.931409+0000","last_undegraded":"2024-09-13T15:03:14.931409+0000","last_fullsized":"2024-09-13T15:03:14.931409+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:44.484707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"261'100","reported_seq":29,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.542393+0000","last_change":"2024-09-13T15:02:59.897496+0000","last_active":"2024-09-13T15:03:06.542393+0000","last_peered":"2024-09-13T15:03:06.542393+0000","last_clean":"2024-09-13T15:03:06.542393+0000","last_became_active":"2024-09-13T15:02:42.883421+0000","last_became_peered":"2024-09-13T15:02:42.883421+0000","last_unstale":"2024-09-13T15:03:06.542393+0000","last_undegraded":"2024-09-13T15:03:06.542393+0000","last_fullsized":"2024-09-13T15:03:06.542393+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T15:02:59.897422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T15:02:59.897422+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:07:39.205660+0000","scrub_duration":0.0032184959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"1337'3204","reported_seq":3231,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:18.800767+0000","last_change":"2024-09-13T15:02:45.495691+0000","last_active":"2024-09-13T15:03:18.800767+0000","last_peered":"2024-09-13T15:03:18.800767+0000","last_clean":"2024-09-13T15:03:18.800767+0000","last_became_active":"2024-09-13T15:02:45.495215+0000","last_became_peered":"2024-09-13T15:02:45.495215+0000","last_unstale":"2024-09-13T15:03:18.800767+0000","last_undegraded":"2024-09-13T15:03:18.800767+0000","last_fullsized":"2024-09-13T15:03:18.800767+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3204,"log_dups_size":0,"ondisk_log_size":3204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:05.547819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209977344,"num_objects":3204,"num_object_clones":0,"num_object_copies":9612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3204,"num_write_kb":205056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1972,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.929906+0000","last_change":"2024-09-13T15:02:42.902299+0000","last_active":"2024-09-13T15:03:14.929906+0000","last_peered":"2024-09-13T15:03:14.929906+0000","last_clean":"2024-09-13T15:03:14.929906+0000","last_became_active":"2024-09-13T15:02:42.898373+0000","last_became_peered":"2024-09-13T15:02:42.898373+0000","last_unstale":"2024-09-13T15:03:14.929906+0000","last_undegraded":"2024-09-13T15:03:14.929906+0000","last_fullsized":"2024-09-13T15:03:14.929906+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:18:54.825311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2010,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.936879+0000","last_change":"2024-09-13T15:00:25.082949+0000","last_active":"2024-09-13T15:03:14.936879+0000","last_peered":"2024-09-13T15:03:14.936879+0000","last_clean":"2024-09-13T15:03:14.936879+0000","last_became_active":"2024-09-13T15:00:25.082279+0000","last_became_peered":"2024-09-13T15:00:25.082279+0000","last_unstale":"2024-09-13T15:03:14.936879+0000","last_undegraded":"2024-09-13T15:03:14.936879+0000","last_fullsized":"2024-09-13T15:03:14.936879+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:49.021898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.542079+0000","last_change":"2024-09-13T15:02:42.895072+0000","last_active":"2024-09-13T15:03:06.542079+0000","last_peered":"2024-09-13T15:03:06.542079+0000","last_clean":"2024-09-13T15:03:06.542079+0000","last_became_active":"2024-09-13T15:02:42.894718+0000","last_became_peered":"2024-09-13T15:02:42.894718+0000","last_unstale":"2024-09-13T15:03:06.542079+0000","last_undegraded":"2024-09-13T15:03:06.542079+0000","last_fullsized":"2024-09-13T15:03:06.542079+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:55:10.117260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"261'100","reported_seq":258,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.929001+0000","last_change":"2024-09-13T15:00:25.085109+0000","last_active":"2024-09-13T15:03:14.929001+0000","last_peered":"2024-09-13T15:03:14.929001+0000","last_clean":"2024-09-13T15:03:14.929001+0000","last_became_active":"2024-09-13T15:00:25.084170+0000","last_became_peered":"2024-09-13T15:00:25.084170+0000","last_unstale":"2024-09-13T15:03:14.929001+0000","last_undegraded":"2024-09-13T15:03:14.929001+0000","last_fullsized":"2024-09-13T15:03:14.929001+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.008264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.008264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:52:58.309554+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"1337'3138","reported_seq":3164,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:18.817557+0000","last_change":"2024-09-13T15:02:45.493742+0000","last_active":"2024-09-13T15:03:18.817557+0000","last_peered":"2024-09-13T15:03:18.817557+0000","last_clean":"2024-09-13T15:03:18.817557+0000","last_became_active":"2024-09-13T15:02:45.493391+0000","last_became_peered":"2024-09-13T15:02:45.493391+0000","last_unstale":"2024-09-13T15:03:18.817557+0000","last_undegraded":"2024-09-13T15:03:18.817557+0000","last_fullsized":"2024-09-13T15:03:18.817557+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3138,"log_dups_size":0,"ondisk_log_size":3138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:26.044885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205651968,"num_objects":3138,"num_object_clones":0,"num_object_copies":9414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3138,"num_write_kb":200832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":127,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.543826+0000","last_change":"2024-09-13T15:00:25.040768+0000","last_active":"2024-09-13T15:03:06.543826+0000","last_peered":"2024-09-13T15:03:06.543826+0000","last_clean":"2024-09-13T15:03:06.543826+0000","last_became_active":"2024-09-13T15:00:25.040487+0000","last_became_peered":"2024-09-13T15:00:25.040487+0000","last_unstale":"2024-09-13T15:03:06.543826+0000","last_undegraded":"2024-09-13T15:03:06.543826+0000","last_fullsized":"2024-09-13T15:03:06.543826+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:44.500104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":431,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.940788+0000","last_change":"2024-09-13T14:55:41.835887+0000","last_active":"2024-09-13T15:03:14.940788+0000","last_peered":"2024-09-13T15:03:14.940788+0000","last_clean":"2024-09-13T15:03:14.940788+0000","last_became_active":"2024-09-13T14:55:41.834413+0000","last_became_peered":"2024-09-13T14:55:41.834413+0000","last_unstale":"2024-09-13T15:03:14.940788+0000","last_undegraded":"2024-09-13T15:03:14.940788+0000","last_fullsized":"2024-09-13T15:03:14.940788+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:01.850578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":176,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552676+0000","last_change":"2024-09-13T15:02:42.915709+0000","last_active":"2024-09-13T15:03:06.552676+0000","last_peered":"2024-09-13T15:03:06.552676+0000","last_clean":"2024-09-13T15:03:06.552676+0000","last_became_active":"2024-09-13T15:02:42.885581+0000","last_became_peered":"2024-09-13T15:02:42.885581+0000","last_unstale":"2024-09-13T15:03:06.552676+0000","last_undegraded":"2024-09-13T15:03:06.552676+0000","last_fullsized":"2024-09-13T15:03:06.552676+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:44:51.824670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"261'100","reported_seq":19,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.542158+0000","last_change":"2024-09-13T15:02:42.884606+0000","last_active":"2024-09-13T15:03:06.542158+0000","last_peered":"2024-09-13T15:03:06.542158+0000","last_clean":"2024-09-13T15:03:06.542158+0000","last_became_active":"2024-09-13T15:02:42.884269+0000","last_became_peered":"2024-09-13T15:02:42.884269+0000","last_unstale":"2024-09-13T15:03:06.542158+0000","last_undegraded":"2024-09-13T15:03:06.542158+0000","last_fullsized":"2024-09-13T15:03:06.542158+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T15:00:44.119431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T15:00:44.119431+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:06:31.424756+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.0","version":"1337'3056","reported_seq":3083,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:17.286627+0000","last_change":"2024-09-13T15:02:44.896248+0000","last_active":"2024-09-13T15:03:17.286627+0000","last_peered":"2024-09-13T15:03:17.286627+0000","last_clean":"2024-09-13T15:03:17.286627+0000","last_became_active":"2024-09-13T15:02:44.895577+0000","last_became_peered":"2024-09-13T15:02:44.895577+0000","last_unstale":"2024-09-13T15:03:17.286627+0000","last_undegraded":"2024-09-13T15:03:17.286627+0000","last_fullsized":"2024-09-13T15:03:17.286627+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3056,"log_dups_size":0,"ondisk_log_size":3056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:20:42.078814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200278016,"num_objects":3056,"num_object_clones":0,"num_object_copies":9168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3056,"num_write_kb":195584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":265,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.929326+0000","last_change":"2024-09-13T15:02:42.896422+0000","last_active":"2024-09-13T15:03:14.929326+0000","last_peered":"2024-09-13T15:03:14.929326+0000","last_clean":"2024-09-13T15:03:14.929326+0000","last_became_active":"2024-09-13T15:02:42.896112+0000","last_became_peered":"2024-09-13T15:02:42.896112+0000","last_unstale":"2024-09-13T15:03:14.929326+0000","last_undegraded":"2024-09-13T15:03:14.929326+0000","last_fullsized":"2024-09-13T15:03:14.929326+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:36:04.129918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":438,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.936859+0000","last_change":"2024-09-13T15:02:42.901428+0000","last_active":"2024-09-13T15:03:14.936859+0000","last_peered":"2024-09-13T15:03:14.936859+0000","last_clean":"2024-09-13T15:03:14.936859+0000","last_became_active":"2024-09-13T15:02:42.901131+0000","last_became_peered":"2024-09-13T15:02:42.901131+0000","last_unstale":"2024-09-13T15:03:14.936859+0000","last_undegraded":"2024-09-13T15:03:14.936859+0000","last_fullsized":"2024-09-13T15:03:14.936859+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:44.152548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"261'100","reported_seq":110,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552258+0000","last_change":"2024-09-13T14:55:41.784086+0000","last_active":"2024-09-13T15:03:06.552258+0000","last_peered":"2024-09-13T15:03:06.552258+0000","last_clean":"2024-09-13T15:03:06.552258+0000","last_became_active":"2024-09-13T14:55:41.783492+0000","last_became_peered":"2024-09-13T14:55:41.783492+0000","last_unstale":"2024-09-13T15:03:06.552258+0000","last_undegraded":"2024-09-13T15:03:06.552258+0000","last_fullsized":"2024-09-13T15:03:06.552258+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:49.280478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:49.280478+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:16:55.041094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"1337'3233","reported_seq":3260,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:18.818945+0000","last_change":"2024-09-13T15:02:45.496010+0000","last_active":"2024-09-13T15:03:18.818945+0000","last_peered":"2024-09-13T15:03:18.818945+0000","last_clean":"2024-09-13T15:03:18.818945+0000","last_became_active":"2024-09-13T15:02:45.495333+0000","last_became_peered":"2024-09-13T15:02:45.495333+0000","last_unstale":"2024-09-13T15:03:18.818945+0000","last_undegraded":"2024-09-13T15:03:18.818945+0000","last_fullsized":"2024-09-13T15:03:18.818945+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3233,"log_dups_size":0,"ondisk_log_size":3233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:03.628293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211877888,"num_objects":3233,"num_object_clones":0,"num_object_copies":9699,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3233,"num_write_kb":206912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":129,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.931234+0000","last_change":"2024-09-13T15:00:25.069428+0000","last_active":"2024-09-13T15:03:14.931234+0000","last_peered":"2024-09-13T15:03:14.931234+0000","last_clean":"2024-09-13T15:03:14.931234+0000","last_became_active":"2024-09-13T15:00:25.068745+0000","last_became_peered":"2024-09-13T15:00:25.068745+0000","last_unstale":"2024-09-13T15:03:14.931234+0000","last_undegraded":"2024-09-13T15:03:14.931234+0000","last_fullsized":"2024-09-13T15:03:14.931234+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:37.264986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":786,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.931321+0000","last_change":"2024-09-13T15:02:42.882826+0000","last_active":"2024-09-13T15:03:14.931321+0000","last_peered":"2024-09-13T15:03:14.931321+0000","last_clean":"2024-09-13T15:03:14.931321+0000","last_became_active":"2024-09-13T15:02:42.882517+0000","last_became_peered":"2024-09-13T15:02:42.882517+0000","last_unstale":"2024-09-13T15:03:14.931321+0000","last_undegraded":"2024-09-13T15:03:14.931321+0000","last_fullsized":"2024-09-13T15:03:14.931321+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:45.948571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"261'100","reported_seq":265,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.929084+0000","last_change":"2024-09-13T15:00:05.050225+0000","last_active":"2024-09-13T15:03:14.929084+0000","last_peered":"2024-09-13T15:03:14.929084+0000","last_clean":"2024-09-13T15:03:14.929084+0000","last_became_active":"2024-09-13T15:00:05.049780+0000","last_became_peered":"2024-09-13T15:00:05.049780+0000","last_unstale":"2024-09-13T15:03:14.929084+0000","last_undegraded":"2024-09-13T15:03:14.929084+0000","last_fullsized":"2024-09-13T15:03:14.929084+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:47.405845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:47.405845+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:56:03.644141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2","version":"1337'3088","reported_seq":3114,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:18.403272+0000","last_change":"2024-09-13T15:02:44.896850+0000","last_active":"2024-09-13T15:03:18.403272+0000","last_peered":"2024-09-13T15:03:18.403272+0000","last_clean":"2024-09-13T15:03:18.403272+0000","last_became_active":"2024-09-13T15:02:44.896509+0000","last_became_peered":"2024-09-13T15:02:44.896509+0000","last_unstale":"2024-09-13T15:03:18.403272+0000","last_undegraded":"2024-09-13T15:03:18.403272+0000","last_fullsized":"2024-09-13T15:03:18.403272+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3088,"log_dups_size":0,"ondisk_log_size":3088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:31.028917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202375168,"num_objects":3088,"num_object_clones":0,"num_object_copies":9264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3088,"num_write_kb":197632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1338,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.932629+0000","last_change":"2024-09-13T15:00:25.058894+0000","last_active":"2024-09-13T15:03:14.932629+0000","last_peered":"2024-09-13T15:03:14.932629+0000","last_clean":"2024-09-13T15:03:14.932629+0000","last_became_active":"2024-09-13T15:00:25.058311+0000","last_became_peered":"2024-09-13T15:00:25.058311+0000","last_unstale":"2024-09-13T15:03:14.932629+0000","last_undegraded":"2024-09-13T15:03:14.932629+0000","last_fullsized":"2024-09-13T15:03:14.932629+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:04.030074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":129,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.928126+0000","last_change":"2024-09-13T15:00:25.154369+0000","last_active":"2024-09-13T15:03:14.928126+0000","last_peered":"2024-09-13T15:03:14.928126+0000","last_clean":"2024-09-13T15:03:14.928126+0000","last_became_active":"2024-09-13T15:00:25.154045+0000","last_became_peered":"2024-09-13T15:00:25.154045+0000","last_unstale":"2024-09-13T15:03:14.928126+0000","last_undegraded":"2024-09-13T15:03:14.928126+0000","last_fullsized":"2024-09-13T15:03:14.928126+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:58:48.342494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"261'100","reported_seq":110,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.551937+0000","last_change":"2024-09-13T14:55:41.844203+0000","last_active":"2024-09-13T15:03:06.551937+0000","last_peered":"2024-09-13T15:03:06.551937+0000","last_clean":"2024-09-13T15:03:06.551937+0000","last_became_active":"2024-09-13T14:55:41.843897+0000","last_became_peered":"2024-09-13T14:55:41.843897+0000","last_unstale":"2024-09-13T15:03:06.551937+0000","last_undegraded":"2024-09-13T15:03:06.551937+0000","last_fullsized":"2024-09-13T15:03:06.551937+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:55.816857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:55.816857+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:42:54.571599+0000","scrub_duration":270,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3","version":"1337'3147","reported_seq":3162,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:17.398935+0000","last_change":"2024-09-13T15:02:45.493751+0000","last_active":"2024-09-13T15:03:17.398935+0000","last_peered":"2024-09-13T15:03:17.398935+0000","last_clean":"2024-09-13T15:03:17.398935+0000","last_became_active":"2024-09-13T15:02:45.493377+0000","last_became_peered":"2024-09-13T15:02:45.493377+0000","last_unstale":"2024-09-13T15:03:17.398935+0000","last_undegraded":"2024-09-13T15:03:17.398935+0000","last_fullsized":"2024-09-13T15:03:17.398935+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3147,"log_dups_size":0,"ondisk_log_size":3147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:49.207437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206241792,"num_objects":3147,"num_object_clones":0,"num_object_copies":9441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3147,"num_write_kb":201408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1365,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.937017+0000","last_change":"2024-09-13T15:00:25.045581+0000","last_active":"2024-09-13T15:03:14.937017+0000","last_peered":"2024-09-13T15:03:14.937017+0000","last_clean":"2024-09-13T15:03:14.937017+0000","last_became_active":"2024-09-13T15:00:25.045225+0000","last_became_peered":"2024-09-13T15:00:25.045225+0000","last_unstale":"2024-09-13T15:03:14.937017+0000","last_undegraded":"2024-09-13T15:03:14.937017+0000","last_fullsized":"2024-09-13T15:03:14.937017+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:38.148339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":476,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.937786+0000","last_change":"2024-09-13T15:02:42.909748+0000","last_active":"2024-09-13T15:03:14.937786+0000","last_peered":"2024-09-13T15:03:14.937786+0000","last_clean":"2024-09-13T15:03:14.937786+0000","last_became_active":"2024-09-13T15:02:42.897824+0000","last_became_peered":"2024-09-13T15:02:42.897824+0000","last_unstale":"2024-09-13T15:03:14.937786+0000","last_undegraded":"2024-09-13T15:03:14.937786+0000","last_fullsized":"2024-09-13T15:03:14.937786+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:05.224186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"261'100","reported_seq":29,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.542088+0000","last_change":"2024-09-13T15:02:56.904746+0000","last_active":"2024-09-13T15:03:06.542088+0000","last_peered":"2024-09-13T15:03:06.542088+0000","last_clean":"2024-09-13T15:03:06.542088+0000","last_became_active":"2024-09-13T15:02:42.894858+0000","last_became_peered":"2024-09-13T15:02:42.894858+0000","last_unstale":"2024-09-13T15:03:06.542088+0000","last_undegraded":"2024-09-13T15:03:06.542088+0000","last_fullsized":"2024-09-13T15:03:06.542088+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T15:02:56.904689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T15:02:56.904689+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:33:17.488947+0000","scrub_duration":0.001699702,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.4","version":"1337'3016","reported_seq":3032,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:17.388264+0000","last_change":"2024-09-13T15:02:45.493795+0000","last_active":"2024-09-13T15:03:17.388264+0000","last_peered":"2024-09-13T15:03:17.388264+0000","last_clean":"2024-09-13T15:03:17.388264+0000","last_became_active":"2024-09-13T15:02:45.493370+0000","last_became_peered":"2024-09-13T15:02:45.493370+0000","last_unstale":"2024-09-13T15:03:17.388264+0000","last_undegraded":"2024-09-13T15:03:17.388264+0000","last_fullsized":"2024-09-13T15:03:17.388264+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3016,"log_dups_size":0,"ondisk_log_size":3016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:32:50.510810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197656576,"num_objects":3016,"num_object_clones":0,"num_object_copies":9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3016,"num_write_kb":193024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":122,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.931771+0000","last_change":"2024-09-13T15:00:25.060205+0000","last_active":"2024-09-13T15:03:14.931771+0000","last_peered":"2024-09-13T15:03:14.931771+0000","last_clean":"2024-09-13T15:03:14.931771+0000","last_became_active":"2024-09-13T15:00:25.059901+0000","last_became_peered":"2024-09-13T15:00:25.059901+0000","last_unstale":"2024-09-13T15:03:14.931771+0000","last_undegraded":"2024-09-13T15:03:14.931771+0000","last_fullsized":"2024-09-13T15:03:14.931771+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:33.964617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":110,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552166+0000","last_change":"2024-09-13T14:55:41.848811+0000","last_active":"2024-09-13T15:03:06.552166+0000","last_peered":"2024-09-13T15:03:06.552166+0000","last_clean":"2024-09-13T15:03:06.552166+0000","last_became_active":"2024-09-13T14:55:41.848542+0000","last_became_peered":"2024-09-13T14:55:41.848542+0000","last_unstale":"2024-09-13T15:03:06.552166+0000","last_undegraded":"2024-09-13T15:03:06.552166+0000","last_fullsized":"2024-09-13T15:03:06.552166+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:48.534022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"261'100","reported_seq":29,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.543984+0000","last_change":"2024-09-13T15:03:00.851297+0000","last_active":"2024-09-13T15:03:06.543984+0000","last_peered":"2024-09-13T15:03:06.543984+0000","last_clean":"2024-09-13T15:03:06.543984+0000","last_became_active":"2024-09-13T15:02:42.894466+0000","last_became_peered":"2024-09-13T15:02:42.894466+0000","last_unstale":"2024-09-13T15:03:06.543984+0000","last_undegraded":"2024-09-13T15:03:06.543984+0000","last_fullsized":"2024-09-13T15:03:06.543984+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T15:03:00.851236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T15:03:00.851236+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:45:21.407912+0000","scrub_duration":0.004159968,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.5","version":"1337'3073","reported_seq":3088,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:17.373833+0000","last_change":"2024-09-13T15:02:45.494451+0000","last_active":"2024-09-13T15:03:17.373833+0000","last_peered":"2024-09-13T15:03:17.373833+0000","last_clean":"2024-09-13T15:03:17.373833+0000","last_became_active":"2024-09-13T15:02:45.494169+0000","last_became_peered":"2024-09-13T15:02:45.494169+0000","last_unstale":"2024-09-13T15:03:17.373833+0000","last_undegraded":"2024-09-13T15:03:17.373833+0000","last_fullsized":"2024-09-13T15:03:17.373833+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3073,"log_dups_size":0,"ondisk_log_size":3073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:45:17.495874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201392128,"num_objects":3073,"num_object_clones":0,"num_object_copies":9219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3073,"num_write_kb":196672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1353,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.546127+0000","last_change":"2024-09-13T15:02:42.880814+0000","last_active":"2024-09-13T15:03:06.546127+0000","last_peered":"2024-09-13T15:03:06.546127+0000","last_clean":"2024-09-13T15:03:06.546127+0000","last_became_active":"2024-09-13T15:02:42.880393+0000","last_became_peered":"2024-09-13T15:02:42.880393+0000","last_unstale":"2024-09-13T15:03:06.546127+0000","last_undegraded":"2024-09-13T15:03:06.546127+0000","last_fullsized":"2024-09-13T15:03:06.546127+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:03.222968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":23,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.543866+0000","last_change":"2024-09-13T15:02:42.914851+0000","last_active":"2024-09-13T15:03:06.543866+0000","last_peered":"2024-09-13T15:03:06.543866+0000","last_clean":"2024-09-13T15:03:06.543866+0000","last_became_active":"2024-09-13T15:02:42.884164+0000","last_became_peered":"2024-09-13T15:02:42.884164+0000","last_unstale":"2024-09-13T15:03:06.543866+0000","last_undegraded":"2024-09-13T15:03:06.543866+0000","last_fullsized":"2024-09-13T15:03:06.543866+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:26:54.412326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"261'100","reported_seq":120,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552862+0000","last_change":"2024-09-13T15:00:25.048330+0000","last_active":"2024-09-13T15:03:06.552862+0000","last_peered":"2024-09-13T15:03:06.552862+0000","last_clean":"2024-09-13T15:03:06.552862+0000","last_became_active":"2024-09-13T15:00:25.048001+0000","last_became_peered":"2024-09-13T15:00:25.048001+0000","last_unstale":"2024-09-13T15:03:06.552862+0000","last_undegraded":"2024-09-13T15:03:06.552862+0000","last_fullsized":"2024-09-13T15:03:06.552862+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:49:21.722939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.722939+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:11:39.620088+0000","scrub_duration":0.0016642790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"1337'3164","reported_seq":3191,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:18.410133+0000","last_change":"2024-09-13T15:02:44.893698+0000","last_active":"2024-09-13T15:03:18.410133+0000","last_peered":"2024-09-13T15:03:18.410133+0000","last_clean":"2024-09-13T15:03:18.410133+0000","last_became_active":"2024-09-13T15:02:44.893386+0000","last_became_peered":"2024-09-13T15:02:44.893386+0000","last_unstale":"2024-09-13T15:03:18.410133+0000","last_undegraded":"2024-09-13T15:03:18.410133+0000","last_fullsized":"2024-09-13T15:03:18.410133+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3164,"log_dups_size":0,"ondisk_log_size":3164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:52.846338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207355904,"num_objects":3164,"num_object_clones":0,"num_object_copies":9492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3164,"num_write_kb":202496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":160,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.931032+0000","last_change":"2024-09-13T15:00:25.054416+0000","last_active":"2024-09-13T15:03:14.931032+0000","last_peered":"2024-09-13T15:03:14.931032+0000","last_clean":"2024-09-13T15:03:14.931032+0000","last_became_active":"2024-09-13T15:00:25.050641+0000","last_became_peered":"2024-09-13T15:00:25.050641+0000","last_unstale":"2024-09-13T15:03:14.931032+0000","last_undegraded":"2024-09-13T15:03:14.931032+0000","last_fullsized":"2024-09-13T15:03:14.931032+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:35:40.107806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":23,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.544178+0000","last_change":"2024-09-13T15:02:42.919448+0000","last_active":"2024-09-13T15:03:06.544178+0000","last_peered":"2024-09-13T15:03:06.544178+0000","last_clean":"2024-09-13T15:03:06.544178+0000","last_became_active":"2024-09-13T15:02:42.889750+0000","last_became_peered":"2024-09-13T15:02:42.889750+0000","last_unstale":"2024-09-13T15:03:06.544178+0000","last_undegraded":"2024-09-13T15:03:06.544178+0000","last_fullsized":"2024-09-13T15:03:06.544178+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:25.834089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"261'100","reported_seq":424,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.937835+0000","last_change":"2024-09-13T15:00:25.213778+0000","last_active":"2024-09-13T15:03:14.937835+0000","last_peered":"2024-09-13T15:03:14.937835+0000","last_clean":"2024-09-13T15:03:14.937835+0000","last_became_active":"2024-09-13T15:00:25.047003+0000","last_became_peered":"2024-09-13T15:00:25.047003+0000","last_unstale":"2024-09-13T15:03:14.937835+0000","last_undegraded":"2024-09-13T15:03:14.937835+0000","last_fullsized":"2024-09-13T15:03:14.937835+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.597568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.597568+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:20.589087+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"1337'3053","reported_seq":3068,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:17.410463+0000","last_change":"2024-09-13T15:02:44.890783+0000","last_active":"2024-09-13T15:03:17.410463+0000","last_peered":"2024-09-13T15:03:17.410463+0000","last_clean":"2024-09-13T15:03:17.410463+0000","last_became_active":"2024-09-13T15:02:44.890448+0000","last_became_peered":"2024-09-13T15:02:44.890448+0000","last_unstale":"2024-09-13T15:03:17.410463+0000","last_undegraded":"2024-09-13T15:03:17.410463+0000","last_fullsized":"2024-09-13T15:03:17.410463+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3053,"log_dups_size":0,"ondisk_log_size":3053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:58.783856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200081408,"num_objects":3053,"num_object_clones":0,"num_object_copies":9159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3053,"num_write_kb":195392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":140,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552013+0000","last_change":"2024-09-13T15:02:42.885662+0000","last_active":"2024-09-13T15:03:06.552013+0000","last_peered":"2024-09-13T15:03:06.552013+0000","last_clean":"2024-09-13T15:03:06.552013+0000","last_became_active":"2024-09-13T15:02:42.885326+0000","last_became_peered":"2024-09-13T15:02:42.885326+0000","last_unstale":"2024-09-13T15:03:06.552013+0000","last_undegraded":"2024-09-13T15:03:06.552013+0000","last_fullsized":"2024-09-13T15:03:06.552013+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:11.631110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":23,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.544583+0000","last_change":"2024-09-13T15:02:42.917286+0000","last_active":"2024-09-13T15:03:06.544583+0000","last_peered":"2024-09-13T15:03:06.544583+0000","last_clean":"2024-09-13T15:03:06.544583+0000","last_became_active":"2024-09-13T15:02:42.888048+0000","last_became_peered":"2024-09-13T15:02:42.888048+0000","last_unstale":"2024-09-13T15:03:06.544583+0000","last_undegraded":"2024-09-13T15:03:06.544583+0000","last_fullsized":"2024-09-13T15:03:06.544583+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:23:37.915522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"261'100","reported_seq":19,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.542765+0000","last_change":"2024-09-13T15:02:42.890854+0000","last_active":"2024-09-13T15:03:06.542765+0000","last_peered":"2024-09-13T15:03:06.542765+0000","last_clean":"2024-09-13T15:03:06.542765+0000","last_became_active":"2024-09-13T15:02:42.889982+0000","last_became_peered":"2024-09-13T15:02:42.889982+0000","last_unstale":"2024-09-13T15:03:06.542765+0000","last_undegraded":"2024-09-13T15:03:06.542765+0000","last_fullsized":"2024-09-13T15:03:06.542765+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.191326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.191326+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:21:40.889165+0000","scrub_duration":0.0025317360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1382,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.546049+0000","last_change":"2024-09-13T15:00:05.058713+0000","last_active":"2024-09-13T15:03:06.546049+0000","last_peered":"2024-09-13T15:03:06.546049+0000","last_clean":"2024-09-13T15:03:06.546049+0000","last_became_active":"2024-09-13T15:00:05.034868+0000","last_became_peered":"2024-09-13T15:00:05.034868+0000","last_unstale":"2024-09-13T15:03:06.546049+0000","last_undegraded":"2024-09-13T15:03:06.546049+0000","last_fullsized":"2024-09-13T15:03:06.546049+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:32:10.399295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":423,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.938202+0000","last_change":"2024-09-13T15:02:42.899236+0000","last_active":"2024-09-13T15:03:14.938202+0000","last_peered":"2024-09-13T15:03:14.938202+0000","last_clean":"2024-09-13T15:03:14.938202+0000","last_became_active":"2024-09-13T15:02:42.897548+0000","last_became_peered":"2024-09-13T15:02:42.897548+0000","last_unstale":"2024-09-13T15:03:14.938202+0000","last_undegraded":"2024-09-13T15:03:14.938202+0000","last_fullsized":"2024-09-13T15:03:14.938202+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:04:04.344226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"261'100","reported_seq":110,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552305+0000","last_change":"2024-09-13T14:55:41.889497+0000","last_active":"2024-09-13T15:03:06.552305+0000","last_peered":"2024-09-13T15:03:06.552305+0000","last_clean":"2024-09-13T15:03:06.552305+0000","last_became_active":"2024-09-13T14:55:41.888766+0000","last_became_peered":"2024-09-13T14:55:41.888766+0000","last_unstale":"2024-09-13T15:03:06.552305+0000","last_undegraded":"2024-09-13T15:03:06.552305+0000","last_fullsized":"2024-09-13T15:03:06.552305+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:55.378404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:55.378404+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:50.289550+0000","scrub_duration":0.0016008439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.542222+0000","last_change":"2024-09-13T15:02:42.894522+0000","last_active":"2024-09-13T15:03:06.542222+0000","last_peered":"2024-09-13T15:03:06.542222+0000","last_clean":"2024-09-13T15:03:06.542222+0000","last_became_active":"2024-09-13T15:02:42.893100+0000","last_became_peered":"2024-09-13T15:02:42.893100+0000","last_unstale":"2024-09-13T15:03:06.542222+0000","last_undegraded":"2024-09-13T15:03:06.542222+0000","last_fullsized":"2024-09-13T15:03:06.542222+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:25.582063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2071,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.938900+0000","last_change":"2024-09-13T15:02:42.901311+0000","last_active":"2024-09-13T15:03:14.938900+0000","last_peered":"2024-09-13T15:03:14.938900+0000","last_clean":"2024-09-13T15:03:14.938900+0000","last_became_active":"2024-09-13T15:02:42.900962+0000","last_became_peered":"2024-09-13T15:02:42.900962+0000","last_unstale":"2024-09-13T15:03:14.938900+0000","last_undegraded":"2024-09-13T15:03:14.938900+0000","last_fullsized":"2024-09-13T15:03:14.938900+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:23:04.122484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"261'100","reported_seq":257,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.928896+0000","last_change":"2024-09-13T15:00:25.087759+0000","last_active":"2024-09-13T15:03:14.928896+0000","last_peered":"2024-09-13T15:03:14.928896+0000","last_clean":"2024-09-13T15:03:14.928896+0000","last_became_active":"2024-09-13T15:00:25.087315+0000","last_became_peered":"2024-09-13T15:00:25.087315+0000","last_unstale":"2024-09-13T15:03:14.928896+0000","last_undegraded":"2024-09-13T15:03:14.928896+0000","last_fullsized":"2024-09-13T15:03:14.928896+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:19.978464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:19.978464+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:22:19.789250+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.542749+0000","last_change":"2024-09-13T15:02:42.894144+0000","last_active":"2024-09-13T15:03:06.542749+0000","last_peered":"2024-09-13T15:03:06.542749+0000","last_clean":"2024-09-13T15:03:06.542749+0000","last_became_active":"2024-09-13T15:02:42.893679+0000","last_became_peered":"2024-09-13T15:02:42.893679+0000","last_unstale":"2024-09-13T15:03:06.542749+0000","last_undegraded":"2024-09-13T15:03:06.542749+0000","last_fullsized":"2024-09-13T15:03:06.542749+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:05.614138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":130,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552385+0000","last_change":"2024-09-13T15:02:42.886038+0000","last_active":"2024-09-13T15:03:06.552385+0000","last_peered":"2024-09-13T15:03:06.552385+0000","last_clean":"2024-09-13T15:03:06.552385+0000","last_became_active":"2024-09-13T15:02:42.884438+0000","last_became_peered":"2024-09-13T15:02:42.884438+0000","last_unstale":"2024-09-13T15:03:06.552385+0000","last_undegraded":"2024-09-13T15:03:06.552385+0000","last_fullsized":"2024-09-13T15:03:06.552385+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:47:39.175275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"261'100","reported_seq":140,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.551936+0000","last_change":"2024-09-13T15:02:42.885131+0000","last_active":"2024-09-13T15:03:06.551936+0000","last_peered":"2024-09-13T15:03:06.551936+0000","last_clean":"2024-09-13T15:03:06.551936+0000","last_became_active":"2024-09-13T15:02:42.883904+0000","last_became_peered":"2024-09-13T15:02:42.883904+0000","last_unstale":"2024-09-13T15:03:06.551936+0000","last_undegraded":"2024-09-13T15:03:06.551936+0000","last_fullsized":"2024-09-13T15:03:06.551936+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:50.327393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:50.327393+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:11:05.603544+0000","scrub_duration":0.0011508309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":25,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.544040+0000","last_change":"2024-09-13T15:02:42.906500+0000","last_active":"2024-09-13T15:03:06.544040+0000","last_peered":"2024-09-13T15:03:06.544040+0000","last_clean":"2024-09-13T15:03:06.544040+0000","last_became_active":"2024-09-13T15:02:42.879187+0000","last_became_peered":"2024-09-13T15:02:42.879187+0000","last_unstale":"2024-09-13T15:03:06.544040+0000","last_undegraded":"2024-09-13T15:03:06.544040+0000","last_fullsized":"2024-09-13T15:03:06.544040+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:19.991287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2030,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.937737+0000","last_change":"2024-09-13T15:02:42.896774+0000","last_active":"2024-09-13T15:03:14.937737+0000","last_peered":"2024-09-13T15:03:14.937737+0000","last_clean":"2024-09-13T15:03:14.937737+0000","last_became_active":"2024-09-13T15:02:42.896459+0000","last_became_peered":"2024-09-13T15:02:42.896459+0000","last_unstale":"2024-09-13T15:03:14.937737+0000","last_undegraded":"2024-09-13T15:03:14.937737+0000","last_fullsized":"2024-09-13T15:03:14.937737+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:35.279246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"261'100","reported_seq":658,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.551995+0000","last_change":"2024-09-13T15:00:05.038303+0000","last_active":"2024-09-13T15:03:06.551995+0000","last_peered":"2024-09-13T15:03:06.551995+0000","last_clean":"2024-09-13T15:03:06.551995+0000","last_became_active":"2024-09-13T15:00:05.037979+0000","last_became_peered":"2024-09-13T15:00:05.037979+0000","last_unstale":"2024-09-13T15:03:06.551995+0000","last_undegraded":"2024-09-13T15:03:06.551995+0000","last_fullsized":"2024-09-13T15:03:06.551995+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:14.654603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:14.654603+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:28.168890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":3277979648,"num_objects":50018,"num_object_clones":0,"num_object_copies":150054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50018,"num_write_kb":3201152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9770237952,"data_stored":9770237952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50018,"ondisk_log_size":50018,"up":48,"acting":48,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":17228,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":22162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2033792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2437120,"data_stored":2426464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430759,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2646524,"kb_used_data":1747444,"kb_used_omap":1,"kb_used_meta":899070,"kb_avail":91098628,"statfs":{"total":95995035648,"available":93284995072,"internally_reserved":0,"allocated":1789382656,"data_stored":1786338916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":920648024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626988,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2485948,"kb_used_data":1587380,"kb_used_omap":1,"kb_used_meta":898558,"kb_avail":91259204,"statfs":{"total":95995035648,"available":93449424896,"internally_reserved":0,"allocated":1625477120,"data_stored":1622433403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":920123736},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823217,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2047216,"kb_used_data":1160168,"kb_used_omap":1,"kb_used_meta":887038,"kb_avail":91697936,"statfs":{"total":95995035648,"available":93898686464,"internally_reserved":0,"allocated":1188012032,"data_stored":1184976301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":908327256},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019445,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059593103,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2470880,"kb_used_data":1632280,"kb_used_omap":26,"kb_used_meta":838565,"kb_avail":91274272,"statfs":{"total":95995035648,"available":93464854528,"internally_reserved":0,"allocated":1671454720,"data_stored":1668434250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":858690720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":591,"seq":2538325672341,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2369844,"kb_used_data":1409900,"kb_used_omap":41,"kb_used_meta":959894,"kb_avail":91375308,"statfs":{"total":95995035648,"available":93568315392,"internally_reserved":0,"allocated":1443737600,"data_stored":1440702452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42651,"internal_metadata":982931813},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":557,"seq":2392296784284,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2894052,"kb_used_data":1965916,"kb_used_omap":32,"kb_used_meta":928095,"kb_avail":90851100,"statfs":{"total":95995035648,"available":93031526400,"internally_reserved":0,"allocated":2013097984,"data_stored":2010073369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33752,"internal_metadata":950369320},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":2250562863523,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2014773248,"data_stored":2014773248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1451360256,"data_stored":1451360256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1666973696,"data_stored":1666973696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1207369728,"data_stored":1207369728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1619329024,"data_stored":1619329024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1810432000,"data_stored":1810432000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:03:21.177 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:03:21.178 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 26 16 53883 53867 129.47 92.375 0.00765756 0.00771819 2024-09-13T15:03:21.246 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:21.284 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:21.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:21.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:21.385 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:21.421 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:21.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:21.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:21.523 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:21.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:21.556 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:21.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:21.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:21 smithi184 ceph-mon[92527]: pgmap v2260: 119 pgs: 119 active+clean; 3.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 108 MiB/s wr, 1.74k op/s 2024-09-13T15:03:21.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:21 smithi184 ceph-mon[92527]: from='client.55795 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:03:21.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:21 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1760694138' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:21.658 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:21.686 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T15:03:21.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:21.695 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:21.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:21.722 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:21 smithi151 ceph-mon[104450]: pgmap v2260: 119 pgs: 119 active+clean; 3.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 108 MiB/s wr, 1.74k op/s 2024-09-13T15:03:21.723 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:21 smithi151 ceph-mon[104450]: from='client.55795 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:03:21.723 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:21 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1760694138' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:21.723 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:21 smithi151 ceph-mon[108966]: pgmap v2260: 119 pgs: 119 active+clean; 3.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 108 MiB/s wr, 1.74k op/s 2024-09-13T15:03:21.723 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:21 smithi151 ceph-mon[108966]: from='client.55795 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:03:21.723 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:21 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1760694138' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:21.796 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:21.842 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:21.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:21.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:21.944 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:21.974 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:21.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:21.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:22.076 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:22.107 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:22.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:22.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:22.175 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 27 16 55310 55294 127.978 89.1875 0.00297555 0.00780533 2024-09-13T15:03:22.209 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:22.241 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:22.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:22.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:22.342 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:22.371 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:22.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:22.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:22.471 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:22.473 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:22.501 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:22.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:22.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:22.603 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:22.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:22 smithi151 ceph-mon[104450]: pgmap v2261: 119 pgs: 119 active+clean; 3.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 108 MiB/s wr, 1.74k op/s 2024-09-13T15:03:22.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:22 smithi151 ceph-mon[108966]: pgmap v2261: 119 pgs: 119 active+clean; 3.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 108 MiB/s wr, 1.74k op/s 2024-09-13T15:03:22.647 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:22.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:22.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:22.746 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:22 smithi184 ceph-mon[92527]: pgmap v2261: 119 pgs: 119 active+clean; 3.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 108 MiB/s wr, 1.74k op/s 2024-09-13T15:03:22.748 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:22.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:22.784 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:22.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:22.885 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:22.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:22.941 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:22.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:23.042 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:23.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:23.073 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:23.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:23.174 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:23.176 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 28 16 56757 56741 126.637 90.4375 0.00542829 0.00789076 2024-09-13T15:03:23.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:23.219 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:23.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:23.320 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:23.360 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:23.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:23.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:23.462 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:23.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:23.503 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:23.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:23.604 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:23.636 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:23.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:23.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:23.738 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:23.775 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:23.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:23.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:23.876 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:23.928 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:03:23.934 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:23.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:23.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:24.035 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:24.084 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:24.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:24.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:24.175 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 29 16 58261 58245 125.511 94 0.0127947 0.00796236 2024-09-13T15:03:24.186 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:24.232 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:24.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:24.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:24.334 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:24.373 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:24.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:24.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:24.456 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T15:03:24.456 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [ 2024-09-13T15:03:24.456 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.456 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:58653 11.5 11:a11a624c:::benchmark_data_smithi151_323728_object58652:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1337)", 2024-09-13T15:03:24.456 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:03:24.442896+0000", 2024-09-13T15:03:24.457 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.0087845630000000004, 2024-09-13T15:03:24.457 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0088184289999999992, 2024-09-13T15:03:24.457 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:03:24.457 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "waiting for sub ops", 2024-09-13T15:03:24.457 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:03:24.457 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:03:24.457 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:03:24.457 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 58653 2024-09-13T15:03:24.457 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.457 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:03:24.457 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.457 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:03:24.457 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.442896+0000", 2024-09-13T15:03:24.458 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:24.458 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.458 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.458 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:03:24.458 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.442896+0000", 2024-09-13T15:03:24.458 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:24.458 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.458 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.458 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:03:24.458 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.442894+0000", 2024-09-13T15:03:24.458 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:03:24.458 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.458 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.458 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:03:24.459 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.442945+0000", 2024-09-13T15:03:24.459 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0659999999999999e-05 2024-09-13T15:03:24.459 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.459 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.459 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:03:24.459 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.442969+0000", 2024-09-13T15:03:24.459 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.3482e-05 2024-09-13T15:03:24.459 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.459 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.459 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:03:24.459 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.442976+0000", 2024-09-13T15:03:24.459 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.7030000000000006e-06 2024-09-13T15:03:24.460 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.460 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.460 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:24.460 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.446787+0000", 2024-09-13T15:03:24.460 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0038103329999999999 2024-09-13T15:03:24.460 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.460 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.460 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:03:24.460 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.446899+0000", 2024-09-13T15:03:24.460 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000112618 2024-09-13T15:03:24.460 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.460 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.460 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 5,6", 2024-09-13T15:03:24.460 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.447023+0000", 2024-09-13T15:03:24.461 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000124101 2024-09-13T15:03:24.461 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.461 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.461 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:03:24.461 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.449767+0000", 2024-09-13T15:03:24.461 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0027437550000000001 2024-09-13T15:03:24.461 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:24.461 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:03:24.461 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:24.461 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.461 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.461 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:58654 11.3 11:ccd338ca:::benchmark_data_smithi151_323728_object58653:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1337)", 2024-09-13T15:03:24.461 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:03:24.442983+0000", 2024-09-13T15:03:24.461 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.0086972339999999999, 2024-09-13T15:03:24.462 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0088109710000000008, 2024-09-13T15:03:24.462 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:03:24.462 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "waiting for sub ops", 2024-09-13T15:03:24.462 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:03:24.462 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:03:24.462 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:03:24.462 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 58654 2024-09-13T15:03:24.462 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.462 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:03:24.462 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.462 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:03:24.462 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.442983+0000", 2024-09-13T15:03:24.462 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:24.462 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.463 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.463 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:03:24.463 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.442983+0000", 2024-09-13T15:03:24.463 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:24.463 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.463 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.463 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:03:24.463 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.442983+0000", 2024-09-13T15:03:24.463 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T15:03:24.463 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.463 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.463 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:03:24.463 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.443015+0000", 2024-09-13T15:03:24.463 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.2577999999999999e-05 2024-09-13T15:03:24.464 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.464 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.464 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:03:24.464 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.443105+0000", 2024-09-13T15:03:24.464 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.9501999999999998e-05 2024-09-13T15:03:24.464 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.464 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.464 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:03:24.464 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.443110+0000", 2024-09-13T15:03:24.464 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8860000000000003e-06 2024-09-13T15:03:24.464 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.464 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.464 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:24.464 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.446174+0000", 2024-09-13T15:03:24.465 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0030644320000000002 2024-09-13T15:03:24.465 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.465 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.465 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:03:24.465 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.446296+0000", 2024-09-13T15:03:24.465 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012224499999999999 2024-09-13T15:03:24.465 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.465 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.465 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,6", 2024-09-13T15:03:24.465 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.446418+0000", 2024-09-13T15:03:24.465 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012215300000000001 2024-09-13T15:03:24.465 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.465 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.466 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:03:24.466 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.448894+0000", 2024-09-13T15:03:24.466 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002475377 2024-09-13T15:03:24.466 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.466 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.466 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:24.466 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.449959+0000", 2024-09-13T15:03:24.466 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0010655009999999999 2024-09-13T15:03:24.466 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:24.466 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:03:24.466 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:24.466 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.466 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.466 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.36666.0:58656 11.e e1337/1329 11:702e1116:::benchmark_data_smithi151_323728_object58655:head v 1337'3625, mlcod=1337'3621)", 2024-09-13T15:03:24.467 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:03:24.447620+0000", 2024-09-13T15:03:24.467 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.0040602160000000002, 2024-09-13T15:03:24.467 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0042491819999999998, 2024-09-13T15:03:24.467 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:03:24.467 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T15:03:24.467 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:03:24.467 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.467 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:03:24.467 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.447620+0000", 2024-09-13T15:03:24.467 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:24.467 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.467 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.467 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:03:24.468 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.447620+0000", 2024-09-13T15:03:24.468 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:24.468 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.468 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.468 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:03:24.468 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.447620+0000", 2024-09-13T15:03:24.468 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T15:03:24.468 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.468 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.468 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:03:24.468 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.447667+0000", 2024-09-13T15:03:24.468 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7824999999999997e-05 2024-09-13T15:03:24.468 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.468 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.469 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:03:24.469 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.447678+0000", 2024-09-13T15:03:24.469 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0114e-05 2024-09-13T15:03:24.469 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.469 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.469 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:03:24.469 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.447682+0000", 2024-09-13T15:03:24.469 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.6999999999999999e-06 2024-09-13T15:03:24.469 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.469 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.469 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:24.469 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.449081+0000", 2024-09-13T15:03:24.470 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0013987509999999999 2024-09-13T15:03:24.470 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.470 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.470 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:03:24.470 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.449101+0000", 2024-09-13T15:03:24.470 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.9658999999999999e-05 2024-09-13T15:03:24.470 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:24.470 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:03:24.470 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:24.470 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.470 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.470 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:58658 11.f 11:fcc0fa87:::benchmark_data_smithi151_323728_object58657:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1337)", 2024-09-13T15:03:24.470 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:03:24.449310+0000", 2024-09-13T15:03:24.471 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.002370788, 2024-09-13T15:03:24.471 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0026247620000000001, 2024-09-13T15:03:24.471 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:03:24.471 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "waiting for sub ops", 2024-09-13T15:03:24.471 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:03:24.471 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:03:24.471 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:03:24.471 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 58658 2024-09-13T15:03:24.471 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.471 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:03:24.471 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.471 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:03:24.471 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.449310+0000", 2024-09-13T15:03:24.472 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:24.472 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.472 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.472 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:03:24.472 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.449310+0000", 2024-09-13T15:03:24.472 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:24.472 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.472 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.472 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:03:24.472 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.449308+0000", 2024-09-13T15:03:24.472 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:03:24.472 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.472 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.473 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:03:24.473 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.449340+0000", 2024-09-13T15:03:24.473 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.1643999999999997e-05 2024-09-13T15:03:24.473 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.473 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.473 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:03:24.473 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.449349+0000", 2024-09-13T15:03:24.473 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.5710000000000004e-06 2024-09-13T15:03:24.473 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.473 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.473 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:03:24.473 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.449355+0000", 2024-09-13T15:03:24.473 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4040000000000003e-06 2024-09-13T15:03:24.473 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.474 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.474 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:24.474 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.449396+0000", 2024-09-13T15:03:24.474 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1600999999999997e-05 2024-09-13T15:03:24.474 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.474 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.474 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:03:24.474 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.449511+0000", 2024-09-13T15:03:24.474 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000114232 2024-09-13T15:03:24.474 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.474 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.474 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:03:24.474 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.449655+0000", 2024-09-13T15:03:24.474 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014412800000000001 2024-09-13T15:03:24.474 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:24.475 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:03:24.475 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:24.475 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.475 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.475 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.36666.0:58657 11.c e1337/1329 11:33f59267:::benchmark_data_smithi151_323728_object58656:head v 1337'3757, mlcod=1337'3755)", 2024-09-13T15:03:24.475 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:03:24.449468+0000", 2024-09-13T15:03:24.475 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.0022120989999999999, 2024-09-13T15:03:24.475 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0025291200000000002, 2024-09-13T15:03:24.475 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:03:24.475 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "started", 2024-09-13T15:03:24.475 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:03:24.475 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.475 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:03:24.475 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.449468+0000", 2024-09-13T15:03:24.476 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:24.476 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.476 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.476 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:03:24.476 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.449468+0000", 2024-09-13T15:03:24.476 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:24.476 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.476 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.476 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:03:24.476 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.449468+0000", 2024-09-13T15:03:24.476 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:03:24.476 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.476 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.476 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:03:24.476 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.449514+0000", 2024-09-13T15:03:24.477 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.6462999999999999e-05 2024-09-13T15:03:24.477 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.477 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.477 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:03:24.477 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.449524+0000", 2024-09-13T15:03:24.477 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.5980000000000003e-06 2024-09-13T15:03:24.477 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.477 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.477 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:03:24.477 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.449528+0000", 2024-09-13T15:03:24.477 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8380000000000001e-06 2024-09-13T15:03:24.477 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.477 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.477 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:24.478 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.449590+0000", 2024-09-13T15:03:24.478 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1103000000000001e-05 2024-09-13T15:03:24.478 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.478 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.478 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:03:24.478 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.449606+0000", 2024-09-13T15:03:24.478 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6407999999999999e-05 2024-09-13T15:03:24.478 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:24.478 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:03:24.478 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:24.478 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.478 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.478 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:58642 11.5 11:a56bac0f:::benchmark_data_smithi151_323728_object58641:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1337)", 2024-09-13T15:03:24.478 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:03:24.437491+0000", 2024-09-13T15:03:24.478 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.014189722, 2024-09-13T15:03:24.479 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.014562841, 2024-09-13T15:03:24.479 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:03:24.479 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "waiting for sub ops", 2024-09-13T15:03:24.479 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:03:24.479 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:03:24.479 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:03:24.479 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 58642 2024-09-13T15:03:24.479 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.479 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:03:24.479 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.479 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:03:24.479 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.437491+0000", 2024-09-13T15:03:24.479 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:24.479 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.480 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.480 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:03:24.480 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.437491+0000", 2024-09-13T15:03:24.480 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:24.480 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.480 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.480 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:03:24.480 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.437490+0000", 2024-09-13T15:03:24.480 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:03:24.480 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.480 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.480 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:03:24.480 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.437548+0000", 2024-09-13T15:03:24.480 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8414e-05 2024-09-13T15:03:24.481 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.481 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.481 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:03:24.481 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.437559+0000", 2024-09-13T15:03:24.481 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1420000000000001e-05 2024-09-13T15:03:24.481 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.481 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.481 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:03:24.481 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.437565+0000", 2024-09-13T15:03:24.481 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5620000000000003e-06 2024-09-13T15:03:24.481 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.481 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.481 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:24.481 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.441474+0000", 2024-09-13T15:03:24.481 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.003908791 2024-09-13T15:03:24.481 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.482 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.482 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:03:24.482 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.441610+0000", 2024-09-13T15:03:24.482 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013657299999999999 2024-09-13T15:03:24.482 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.482 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.482 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 5,6", 2024-09-13T15:03:24.482 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.441742+0000", 2024-09-13T15:03:24.482 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000131462 2024-09-13T15:03:24.482 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.482 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.482 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:24.482 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.446757+0000", 2024-09-13T15:03:24.482 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0050152139999999996 2024-09-13T15:03:24.482 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.483 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.483 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:03:24.483 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.446766+0000", 2024-09-13T15:03:24.483 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.6600000000000001e-06 2024-09-13T15:03:24.483 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:24.483 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:03:24.483 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:24.483 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.483 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.483 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:24.484 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:58660 11.5 11:ac4efca3:::benchmark_data_smithi151_323728_object58659:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1337)", 2024-09-13T15:03:24.484 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:03:24.449811+0000", 2024-09-13T15:03:24.484 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.001869386, 2024-09-13T15:03:24.484 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0023200270000000001, 2024-09-13T15:03:24.485 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:03:24.485 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "waiting for sub ops", 2024-09-13T15:03:24.485 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:03:24.485 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:03:24.485 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:03:24.485 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 58660 2024-09-13T15:03:24.485 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.485 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:03:24.485 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.485 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:03:24.485 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.449811+0000", 2024-09-13T15:03:24.485 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:24.485 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.486 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.486 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:03:24.486 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.449811+0000", 2024-09-13T15:03:24.486 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:24.486 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.486 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.486 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:03:24.486 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.449810+0000", 2024-09-13T15:03:24.486 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T15:03:24.486 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.486 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.486 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:03:24.486 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.449839+0000", 2024-09-13T15:03:24.487 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.8289999999999998e-05 2024-09-13T15:03:24.487 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.487 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.487 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:03:24.487 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.449847+0000", 2024-09-13T15:03:24.487 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.4800000000000001e-06 2024-09-13T15:03:24.487 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.487 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.487 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:03:24.487 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.449852+0000", 2024-09-13T15:03:24.487 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.6129999999999997e-06 2024-09-13T15:03:24.487 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.487 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.488 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:24.488 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.449930+0000", 2024-09-13T15:03:24.488 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.8197999999999999e-05 2024-09-13T15:03:24.488 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.488 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.488 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:03:24.488 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.450051+0000", 2024-09-13T15:03:24.488 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000121131 2024-09-13T15:03:24.488 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.488 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.488 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 5,6", 2024-09-13T15:03:24.488 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.450201+0000", 2024-09-13T15:03:24.488 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000149784 2024-09-13T15:03:24.488 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:24.488 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:03:24.489 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:24.489 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.489 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.489 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:58662 11.4 11:2f7bcc02:::benchmark_data_smithi151_323728_object58661:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1337)", 2024-09-13T15:03:24.489 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:03:24.450055+0000", 2024-09-13T15:03:24.489 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.001625264, 2024-09-13T15:03:24.489 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0021413399999999998, 2024-09-13T15:03:24.489 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:03:24.489 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "waiting for sub ops", 2024-09-13T15:03:24.489 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:03:24.489 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:03:24.489 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:03:24.489 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 58662 2024-09-13T15:03:24.490 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.490 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:03:24.490 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.490 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:03:24.490 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.450055+0000", 2024-09-13T15:03:24.490 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:24.490 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.490 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.490 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:03:24.490 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.450055+0000", 2024-09-13T15:03:24.490 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:24.490 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.490 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.491 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:03:24.491 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.450054+0000", 2024-09-13T15:03:24.491 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:03:24.491 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.491 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.491 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:03:24.491 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.450118+0000", 2024-09-13T15:03:24.491 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3734999999999999e-05 2024-09-13T15:03:24.491 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.491 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.491 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:03:24.491 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.450128+0000", 2024-09-13T15:03:24.491 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.5210000000000008e-06 2024-09-13T15:03:24.492 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.492 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.492 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:03:24.492 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.450133+0000", 2024-09-13T15:03:24.492 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0320000000000003e-06 2024-09-13T15:03:24.492 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.492 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.492 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:24.492 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.450173+0000", 2024-09-13T15:03:24.492 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0250000000000003e-05 2024-09-13T15:03:24.492 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.492 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.492 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:03:24.492 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.450301+0000", 2024-09-13T15:03:24.493 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012817500000000001 2024-09-13T15:03:24.493 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.493 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.493 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T15:03:24.493 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.450430+0000", 2024-09-13T15:03:24.493 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000129066 2024-09-13T15:03:24.493 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:24.493 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:03:24.493 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:24.493 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.493 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.493 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_repop(client.36666.0:58661 11.e e1337/1329)", 2024-09-13T15:03:24.493 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:03:24.450612+0000", 2024-09-13T15:03:24.494 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.0010683120000000001, 2024-09-13T15:03:24.494 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001642789, 2024-09-13T15:03:24.494 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:03:24.494 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "queued for pg", 2024-09-13T15:03:24.494 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:03:24.494 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.494 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:03:24.494 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.450612+0000", 2024-09-13T15:03:24.494 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:24.494 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.494 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.494 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:03:24.494 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.450612+0000", 2024-09-13T15:03:24.494 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:24.495 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.495 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.495 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:03:24.495 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.450612+0000", 2024-09-13T15:03:24.495 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T15:03:24.495 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.495 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.495 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:03:24.495 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.450642+0000", 2024-09-13T15:03:24.495 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.0346e-05 2024-09-13T15:03:24.495 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.495 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.495 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:03:24.496 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.450650+0000", 2024-09-13T15:03:24.496 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.2449999999999998e-06 2024-09-13T15:03:24.496 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.496 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.496 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:03:24.496 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.450655+0000", 2024-09-13T15:03:24.496 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7029999999999997e-06 2024-09-13T15:03:24.496 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:24.496 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:03:24.496 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:24.496 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.496 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.497 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:58643 11.5 11:a8319934:::benchmark_data_smithi151_323728_object58642:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1337)", 2024-09-13T15:03:24.497 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:03:24.439926+0000", 2024-09-13T15:03:24.497 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 0.011754702000000001, 2024-09-13T15:03:24.497 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.012375171000000001, 2024-09-13T15:03:24.497 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:03:24.497 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "waiting for sub ops", 2024-09-13T15:03:24.497 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:03:24.497 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:03:24.497 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:03:24.497 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 58643 2024-09-13T15:03:24.497 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.497 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:03:24.498 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.498 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:03:24.498 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.439926+0000", 2024-09-13T15:03:24.498 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:24.498 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.498 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.498 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:03:24.498 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.439926+0000", 2024-09-13T15:03:24.498 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:24.498 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.498 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.498 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:03:24.498 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.439925+0000", 2024-09-13T15:03:24.499 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:03:24.499 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.499 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.499 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:03:24.499 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.439960+0000", 2024-09-13T15:03:24.499 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.5596999999999998e-05 2024-09-13T15:03:24.499 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.499 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.499 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:03:24.499 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.439969+0000", 2024-09-13T15:03:24.499 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.2660000000000001e-06 2024-09-13T15:03:24.499 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.499 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.500 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:03:24.500 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.439973+0000", 2024-09-13T15:03:24.500 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1869999999999999e-06 2024-09-13T15:03:24.500 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.500 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.500 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:24.500 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.444433+0000", 2024-09-13T15:03:24.500 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0044603150000000003 2024-09-13T15:03:24.500 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.500 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.500 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:03:24.500 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.444542+0000", 2024-09-13T15:03:24.500 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00010863599999999999 2024-09-13T15:03:24.501 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.501 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.501 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 5,6", 2024-09-13T15:03:24.501 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.444668+0000", 2024-09-13T15:03:24.501 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012667700000000001 2024-09-13T15:03:24.501 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.501 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.501 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:24.501 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.449520+0000", 2024-09-13T15:03:24.501 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0048520209999999998 2024-09-13T15:03:24.501 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:24.501 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:24.501 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:03:24.501 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:24.449760+0000", 2024-09-13T15:03:24.501 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00023959400000000001 2024-09-13T15:03:24.502 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:24.502 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:03:24.502 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:24.502 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:24.502 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-13T15:03:24.502 INFO:teuthology.orchestra.run.smithi184.stdout: "num_ops": 10 2024-09-13T15:03:24.502 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T15:03:24.543 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:24.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:24.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:24.645 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:24.681 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:24.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:24.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:24.783 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:24.819 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:24.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:24.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:24.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:03:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:03:24] "GET /metrics HTTP/1.1" 200 36695 "" "Prometheus/2.51.0" 2024-09-13T15:03:24.921 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:24.954 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:24.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:24.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:25.056 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:25.095 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:25.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:25.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:25.176 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 30 16 59785 59769 124.502 95.25 0.00723803 0.00802143 2024-09-13T15:03:25.197 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:25.219 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:25.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:25.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:25.277 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:24 smithi184 ceph-mon[92527]: pgmap v2262: 119 pgs: 119 active+clean; 3.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-09-13T15:03:25.319 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T15:03:25.321 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:25.352 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:25.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:25.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:25.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:24 smithi151 ceph-mon[104450]: pgmap v2262: 119 pgs: 119 active+clean; 3.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-09-13T15:03:25.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:24 smithi151 ceph-mon[108966]: pgmap v2262: 119 pgs: 119 active+clean; 3.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-09-13T15:03:25.454 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:25.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:25.487 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:25.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:25.588 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:25.631 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:25.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:25.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:25.733 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:25.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:25.757 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:25.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:25.858 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:25.889 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:25.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:25.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:25.991 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:26.022 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:26.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:26.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:26.124 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:26.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:26.150 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:26.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:26.176 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 31 16 61034 61018 123.004 78.0625 0.00576096 0.00812485 2024-09-13T15:03:26.250 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:26.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:26.283 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:26.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:26.331 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:03:26.385 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:26.434 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:26.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:26.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:26.536 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:26.576 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:26.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:26.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:26.678 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:26.709 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:26.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:26.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:26.811 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:26.850 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:26.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:26.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:26.946 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:26.952 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:27.014 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:27.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:27.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:27.116 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:27.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:27.149 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:27.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:27.167 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:26 smithi151 ceph-mon[104450]: pgmap v2263: 119 pgs: 119 active+clean; 3.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-13T15:03:27.167 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:26 smithi151 ceph-mon[108966]: pgmap v2263: 119 pgs: 119 active+clean; 3.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-13T15:03:27.176 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 32 16 62510 62494 122.042 92.25 0.0149445 0.00818869 2024-09-13T15:03:27.249 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:27.290 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:27.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:27.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:27.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:26 smithi184 ceph-mon[92527]: pgmap v2263: 119 pgs: 119 active+clean; 3.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-13T15:03:27.391 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:27.413 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:27.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:27.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:27.514 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:27.536 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:27.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:27.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:27.640 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:27.676 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:27.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:27.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:27.778 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:27.819 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:27.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:27.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:27.921 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:27.959 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:27.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:27.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:28.061 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:28.090 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:28.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:28.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:28.176 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 33 16 63659 63643 120.52 71.8125 0.0100016 0.0082925 2024-09-13T15:03:28.192 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:28.232 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:28.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:28.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:28.334 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:28.367 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:28.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:28.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:28.468 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:28.509 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:28.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:28.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:28.611 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:28.650 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:28.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:28.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:28.752 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:28.801 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:28.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:28.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:28.903 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:28.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:28.925 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:28.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:28.982 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:28 smithi151 ceph-mon[104450]: pgmap v2264: 119 pgs: 119 active+clean; 3.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-13T15:03:28.982 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:28 smithi151 ceph-mon[108966]: pgmap v2264: 119 pgs: 119 active+clean; 3.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-13T15:03:29.025 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:29.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:29.052 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:29.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:29.089 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T15:03:29.090 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T15:03:29.154 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:29.176 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 34 16 64655 64639 118.806 62.25 0.0122922 0.00841204 2024-09-13T15:03:29.194 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:29.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:29.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:29.296 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:29.329 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:29.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:29.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:29.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:28 smithi184 ceph-mon[92527]: pgmap v2264: 119 pgs: 119 active+clean; 3.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-13T15:03:29.431 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:29.464 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:29.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:29.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:29.566 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:29.587 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:29.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:29.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:29.689 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:29.715 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:29.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:29.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:29.817 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:29.856 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:29.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:29.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:29.958 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:29.995 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:29.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:29.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:30.097 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:30.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:30.134 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:30.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:30.176 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 35 16 65769 65753 117.4 69.625 0.0105097 0.00851279 2024-09-13T15:03:30.215 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:29 smithi184 ceph-mon[92527]: from='client.46520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:03:30.235 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:30.265 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:30.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:30.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:30.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:29 smithi151 ceph-mon[104450]: from='client.46520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:03:30.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:29 smithi151 ceph-mon[108966]: from='client.46520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:03:30.366 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:30.403 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:30.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:30.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:30.505 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:30.525 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:03:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:03:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:03:30.544 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:30.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:30.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:30.646 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:30.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:30.689 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:30.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:30.790 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:30.828 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:30.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:30.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:30.930 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:30.963 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:30.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:30.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:30.966 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T15:03:30.966 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [], 2024-09-13T15:03:30.966 INFO:teuthology.orchestra.run.smithi184.stdout: "complaint_time": 30, 2024-09-13T15:03:30.967 INFO:teuthology.orchestra.run.smithi184.stdout: "num_blocked_ops": 0 2024-09-13T15:03:30.967 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T15:03:30.992 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:30 smithi151 ceph-mon[104450]: pgmap v2265: 119 pgs: 119 active+clean; 3.9 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 92 MiB/s wr, 1.46k op/s 2024-09-13T15:03:30.992 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:30 smithi151 ceph-mon[108966]: pgmap v2265: 119 pgs: 119 active+clean; 3.9 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 92 MiB/s wr, 1.46k op/s 2024-09-13T15:03:31.066 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:31.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:30 smithi184 ceph-mon[92527]: pgmap v2265: 119 pgs: 119 active+clean; 3.9 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 92 MiB/s wr, 1.46k op/s 2024-09-13T15:03:31.105 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:31.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:31.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:31.176 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 36 16 67134 67118 116.509 85.3125 0.0213826 0.0085782 2024-09-13T15:03:31.207 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:31.237 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:31.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:31.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:31.339 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:31.377 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:31.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:31.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:31.435 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":2264,"stamp":"2024-09-13T15:03:27.867459+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3943110605,"num_objects":60184,"num_object_clones":0,"num_object_copies":180552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60221,"num_write_kb":3850143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":2051077,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63316,"ondisk_log_size":63316,"up":357,"acting":357,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":357,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":17025824,"kb_used_data":11607408,"kb_used_omap":106,"kb_used_meta":5418197,"kb_avail":545445088,"statfs":{"total":575970213888,"available":558535770112,"internally_reserved":0,"allocated":11885985792,"data_stored":11867782371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":109003,"internal_metadata":5548234293},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,2,9,7,6,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1223688192,"num_objects":18672,"num_object_clones":0,"num_object_copies":56016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18672,"num_write_kb":1195008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005087"},"pg_stats":[{"pgid":"1.1d","version":"261'100","reported_seq":257,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.930758+0000","last_change":"2024-09-13T15:00:25.065803+0000","last_active":"2024-09-13T15:03:14.930758+0000","last_peered":"2024-09-13T15:03:14.930758+0000","last_clean":"2024-09-13T15:03:14.930758+0000","last_became_active":"2024-09-13T15:00:25.063785+0000","last_became_peered":"2024-09-13T15:00:25.063785+0000","last_unstale":"2024-09-13T15:03:14.930758+0000","last_undegraded":"2024-09-13T15:03:14.930758+0000","last_fullsized":"2024-09-13T15:03:14.930758+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:17.988345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:17.988345+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:43.674995+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":1961,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552672+0000","last_change":"2024-09-13T15:00:05.045993+0000","last_active":"2024-09-13T15:03:06.552672+0000","last_peered":"2024-09-13T15:03:06.552672+0000","last_clean":"2024-09-13T15:03:06.552672+0000","last_became_active":"2024-09-13T15:00:05.033311+0000","last_became_peered":"2024-09-13T15:00:05.033311+0000","last_unstale":"2024-09-13T15:03:06.552672+0000","last_undegraded":"2024-09-13T15:03:06.552672+0000","last_fullsized":"2024-09-13T15:03:06.552672+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:27.290733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2280,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":160,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.931294+0000","last_change":"2024-09-13T15:00:25.180918+0000","last_active":"2024-09-13T15:03:14.931294+0000","last_peered":"2024-09-13T15:03:14.931294+0000","last_clean":"2024-09-13T15:03:14.931294+0000","last_became_active":"2024-09-13T15:00:25.180571+0000","last_became_peered":"2024-09-13T15:00:25.180571+0000","last_unstale":"2024-09-13T15:03:14.931294+0000","last_undegraded":"2024-09-13T15:03:14.931294+0000","last_fullsized":"2024-09-13T15:03:14.931294+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:31.846606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":396,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.938732+0000","last_change":"2024-09-13T14:45:19.810203+0000","last_active":"2024-09-13T15:03:14.938732+0000","last_peered":"2024-09-13T15:03:14.938732+0000","last_clean":"2024-09-13T15:03:14.938732+0000","last_became_active":"2024-09-13T14:45:19.809693+0000","last_became_peered":"2024-09-13T14:45:19.809693+0000","last_unstale":"2024-09-13T15:03:14.938732+0000","last_undegraded":"2024-09-13T15:03:14.938732+0000","last_fullsized":"2024-09-13T15:03:14.938732+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:20.219673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":402,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.937084+0000","last_change":"2024-09-13T15:00:05.051083+0000","last_active":"2024-09-13T15:03:14.937084+0000","last_peered":"2024-09-13T15:03:14.937084+0000","last_clean":"2024-09-13T15:03:14.937084+0000","last_became_active":"2024-09-13T15:00:05.050737+0000","last_became_peered":"2024-09-13T15:00:05.050737+0000","last_unstale":"2024-09-13T15:03:14.937084+0000","last_undegraded":"2024-09-13T15:03:14.937084+0000","last_fullsized":"2024-09-13T15:03:14.937084+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:58.282196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"261'100","reported_seq":19,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.542812+0000","last_change":"2024-09-13T15:02:42.889565+0000","last_active":"2024-09-13T15:03:06.542812+0000","last_peered":"2024-09-13T15:03:06.542812+0000","last_clean":"2024-09-13T15:03:06.542812+0000","last_became_active":"2024-09-13T15:02:42.888466+0000","last_became_peered":"2024-09-13T15:02:42.888466+0000","last_unstale":"2024-09-13T15:03:06.542812+0000","last_undegraded":"2024-09-13T15:03:06.542812+0000","last_fullsized":"2024-09-13T15:03:06.542812+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_clean_scrub_stamp":"2024-09-13T14:44:54.259482+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:19:31.567604+0000","scrub_duration":0.015265569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":128,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552376+0000","last_change":"2024-09-13T15:00:25.072110+0000","last_active":"2024-09-13T15:03:06.552376+0000","last_peered":"2024-09-13T15:03:06.552376+0000","last_clean":"2024-09-13T15:03:06.552376+0000","last_became_active":"2024-09-13T15:00:25.045594+0000","last_became_peered":"2024-09-13T15:00:25.045594+0000","last_unstale":"2024-09-13T15:03:06.552376+0000","last_undegraded":"2024-09-13T15:03:06.552376+0000","last_fullsized":"2024-09-13T15:03:06.552376+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:59:45.210187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":183,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.940883+0000","last_change":"2024-09-13T15:00:25.052537+0000","last_active":"2024-09-13T15:03:14.940883+0000","last_peered":"2024-09-13T15:03:14.940883+0000","last_clean":"2024-09-13T15:03:14.940883+0000","last_became_active":"2024-09-13T15:00:25.052225+0000","last_became_peered":"2024-09-13T15:00:25.052225+0000","last_unstale":"2024-09-13T15:03:14.940883+0000","last_undegraded":"2024-09-13T15:03:14.940883+0000","last_fullsized":"2024-09-13T15:03:14.940883+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:52.039052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1e","version":"261'100","reported_seq":413,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.939733+0000","last_change":"2024-09-13T14:45:35.947332+0000","last_active":"2024-09-13T15:03:14.939733+0000","last_peered":"2024-09-13T15:03:14.939733+0000","last_clean":"2024-09-13T15:03:14.939733+0000","last_became_active":"2024-09-13T14:45:19.810775+0000","last_became_peered":"2024-09-13T14:45:19.810775+0000","last_unstale":"2024-09-13T15:03:14.939733+0000","last_undegraded":"2024-09-13T15:03:14.939733+0000","last_fullsized":"2024-09-13T15:03:14.939733+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:35.947265+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:43:41.569795+0000","last_clean_scrub_stamp":"2024-09-13T14:45:35.947265+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:00.606370+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":128,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552251+0000","last_change":"2024-09-13T15:00:25.192972+0000","last_active":"2024-09-13T15:03:06.552251+0000","last_peered":"2024-09-13T15:03:06.552251+0000","last_clean":"2024-09-13T15:03:06.552251+0000","last_became_active":"2024-09-13T15:00:25.047858+0000","last_became_peered":"2024-09-13T15:00:25.047858+0000","last_unstale":"2024-09-13T15:03:06.552251+0000","last_undegraded":"2024-09-13T15:03:06.552251+0000","last_fullsized":"2024-09-13T15:03:06.552251+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:22:16.795970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1321,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.937968+0000","last_change":"2024-09-13T15:00:25.086077+0000","last_active":"2024-09-13T15:03:14.937968+0000","last_peered":"2024-09-13T15:03:14.937968+0000","last_clean":"2024-09-13T15:03:14.937968+0000","last_became_active":"2024-09-13T15:00:25.085685+0000","last_became_peered":"2024-09-13T15:00:25.085685+0000","last_unstale":"2024-09-13T15:03:14.937968+0000","last_undegraded":"2024-09-13T15:03:14.937968+0000","last_fullsized":"2024-09-13T15:03:14.937968+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:09:24.157291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":14,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.931568+0000","last_change":"2024-09-13T15:03:06.560718+0000","last_active":"2024-09-13T15:03:14.931568+0000","last_peered":"2024-09-13T15:03:14.931568+0000","last_clean":"2024-09-13T15:03:14.931568+0000","last_became_active":"2024-09-13T15:03:06.560348+0000","last_became_peered":"2024-09-13T15:03:06.560348+0000","last_unstale":"2024-09-13T15:03:14.931568+0000","last_undegraded":"2024-09-13T15:03:14.931568+0000","last_fullsized":"2024-09-13T15:03:14.931568+0000","mapping_epoch":1335,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:19:03.944074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":52,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.933481+0000","last_change":"2024-09-13T15:02:01.133848+0000","last_active":"2024-09-13T15:03:14.933481+0000","last_peered":"2024-09-13T15:03:14.933481+0000","last_clean":"2024-09-13T15:03:14.933481+0000","last_became_active":"2024-09-13T15:02:01.133414+0000","last_became_peered":"2024-09-13T15:02:01.133414+0000","last_unstale":"2024-09-13T15:03:14.933481+0000","last_undegraded":"2024-09-13T15:03:14.933481+0000","last_fullsized":"2024-09-13T15:03:14.933481+0000","mapping_epoch":1316,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:51:09.222615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"261'100","reported_seq":416,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.937186+0000","last_change":"2024-09-13T15:02:42.900466+0000","last_active":"2024-09-13T15:03:14.937186+0000","last_peered":"2024-09-13T15:03:14.937186+0000","last_clean":"2024-09-13T15:03:14.937186+0000","last_became_active":"2024-09-13T15:02:42.900166+0000","last_became_peered":"2024-09-13T15:02:42.900166+0000","last_unstale":"2024-09-13T15:03:14.937186+0000","last_undegraded":"2024-09-13T15:03:14.937186+0000","last_fullsized":"2024-09-13T15:03:14.937186+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.614340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.614340+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:52.087366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":136,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.928463+0000","last_change":"2024-09-13T15:02:42.897194+0000","last_active":"2024-09-13T15:03:14.928463+0000","last_peered":"2024-09-13T15:03:14.928463+0000","last_clean":"2024-09-13T15:03:14.928463+0000","last_became_active":"2024-09-13T15:02:42.896712+0000","last_became_peered":"2024-09-13T15:02:42.896712+0000","last_unstale":"2024-09-13T15:03:14.928463+0000","last_undegraded":"2024-09-13T15:03:14.928463+0000","last_fullsized":"2024-09-13T15:03:14.928463+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:13:29.973653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":437,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.937209+0000","last_change":"2024-09-13T15:00:25.052382+0000","last_active":"2024-09-13T15:03:14.937209+0000","last_peered":"2024-09-13T15:03:14.937209+0000","last_clean":"2024-09-13T15:03:14.937209+0000","last_became_active":"2024-09-13T15:00:25.051999+0000","last_became_peered":"2024-09-13T15:00:25.051999+0000","last_unstale":"2024-09-13T15:03:14.937209+0000","last_undegraded":"2024-09-13T15:03:14.937209+0000","last_fullsized":"2024-09-13T15:03:14.937209+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:30.490420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.18","version":"261'100","reported_seq":145,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552048+0000","last_change":"2024-09-13T15:02:42.914596+0000","last_active":"2024-09-13T15:03:06.552048+0000","last_peered":"2024-09-13T15:03:06.552048+0000","last_clean":"2024-09-13T15:03:06.552048+0000","last_became_active":"2024-09-13T15:02:42.886984+0000","last_became_peered":"2024-09-13T15:02:42.886984+0000","last_unstale":"2024-09-13T15:03:06.552048+0000","last_undegraded":"2024-09-13T15:03:06.552048+0000","last_fullsized":"2024-09-13T15:03:06.552048+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:56.548855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:56.548855+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:30.761139+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1050752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":257,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.929447+0000","last_change":"2024-09-13T15:00:25.048396+0000","last_active":"2024-09-13T15:03:14.929447+0000","last_peered":"2024-09-13T15:03:14.929447+0000","last_clean":"2024-09-13T15:03:14.929447+0000","last_became_active":"2024-09-13T15:00:25.048084+0000","last_became_peered":"2024-09-13T15:00:25.048084+0000","last_unstale":"2024-09-13T15:03:14.929447+0000","last_undegraded":"2024-09-13T15:03:14.929447+0000","last_fullsized":"2024-09-13T15:03:14.929447+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:05:23.139064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.543667+0000","last_change":"2024-09-13T15:02:42.895757+0000","last_active":"2024-09-13T15:03:06.543667+0000","last_peered":"2024-09-13T15:03:06.543667+0000","last_clean":"2024-09-13T15:03:06.543667+0000","last_became_active":"2024-09-13T15:02:42.895416+0000","last_became_peered":"2024-09-13T15:02:42.895416+0000","last_unstale":"2024-09-13T15:03:06.543667+0000","last_undegraded":"2024-09-13T15:03:06.543667+0000","last_fullsized":"2024-09-13T15:03:06.543667+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:50:34.099001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1b","version":"261'100","reported_seq":140,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552687+0000","last_change":"2024-09-13T15:02:42.887234+0000","last_active":"2024-09-13T15:03:06.552687+0000","last_peered":"2024-09-13T15:03:06.552687+0000","last_clean":"2024-09-13T15:03:06.552687+0000","last_became_active":"2024-09-13T15:02:42.886850+0000","last_became_peered":"2024-09-13T15:02:42.886850+0000","last_unstale":"2024-09-13T15:03:06.552687+0000","last_undegraded":"2024-09-13T15:03:06.552687+0000","last_fullsized":"2024-09-13T15:03:06.552687+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:55.988431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:55.988431+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:50:58.012159+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":416,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.936625+0000","last_change":"2024-09-13T15:00:05.047270+0000","last_active":"2024-09-13T15:03:14.936625+0000","last_peered":"2024-09-13T15:03:14.936625+0000","last_clean":"2024-09-13T15:03:14.936625+0000","last_became_active":"2024-09-13T15:00:05.046710+0000","last_became_peered":"2024-09-13T15:00:05.046710+0000","last_unstale":"2024-09-13T15:03:14.936625+0000","last_undegraded":"2024-09-13T15:03:14.936625+0000","last_fullsized":"2024-09-13T15:03:14.936625+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:46:06.874757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":110,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552782+0000","last_change":"2024-09-13T14:55:41.883426+0000","last_active":"2024-09-13T15:03:06.552782+0000","last_peered":"2024-09-13T15:03:06.552782+0000","last_clean":"2024-09-13T15:03:06.552782+0000","last_became_active":"2024-09-13T14:55:41.880724+0000","last_became_peered":"2024-09-13T14:55:41.880724+0000","last_unstale":"2024-09-13T15:03:06.552782+0000","last_undegraded":"2024-09-13T15:03:06.552782+0000","last_fullsized":"2024-09-13T15:03:06.552782+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:17.920508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1a","version":"261'100","reported_seq":19,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.543668+0000","last_change":"2024-09-13T15:02:42.887267+0000","last_active":"2024-09-13T15:03:06.543668+0000","last_peered":"2024-09-13T15:03:06.543668+0000","last_clean":"2024-09-13T15:03:06.543668+0000","last_became_active":"2024-09-13T15:02:42.886976+0000","last_became_peered":"2024-09-13T15:02:42.886976+0000","last_unstale":"2024-09-13T15:03:06.543668+0000","last_undegraded":"2024-09-13T15:03:06.543668+0000","last_fullsized":"2024-09-13T15:03:06.543668+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:46.985510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:46.985510+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:10:55.966984+0000","scrub_duration":0.0030525999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":416,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.936716+0000","last_change":"2024-09-13T14:54:05.606882+0000","last_active":"2024-09-13T15:03:14.936716+0000","last_peered":"2024-09-13T15:03:14.936716+0000","last_clean":"2024-09-13T15:03:14.936716+0000","last_became_active":"2024-09-13T14:54:05.605912+0000","last_became_peered":"2024-09-13T14:54:05.605912+0000","last_unstale":"2024-09-13T15:03:14.936716+0000","last_undegraded":"2024-09-13T15:03:14.936716+0000","last_fullsized":"2024-09-13T15:03:14.936716+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:05.445329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1976,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552657+0000","last_change":"2024-09-13T15:00:05.037972+0000","last_active":"2024-09-13T15:03:06.552657+0000","last_peered":"2024-09-13T15:03:06.552657+0000","last_clean":"2024-09-13T15:03:06.552657+0000","last_became_active":"2024-09-13T15:00:05.037561+0000","last_became_peered":"2024-09-13T15:00:05.037561+0000","last_unstale":"2024-09-13T15:03:06.552657+0000","last_undegraded":"2024-09-13T15:03:06.552657+0000","last_fullsized":"2024-09-13T15:03:06.552657+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:42:04.578024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.15","version":"261'100","reported_seq":19,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.542601+0000","last_change":"2024-09-13T15:02:42.892087+0000","last_active":"2024-09-13T15:03:06.542601+0000","last_peered":"2024-09-13T15:03:06.542601+0000","last_clean":"2024-09-13T15:03:06.542601+0000","last_became_active":"2024-09-13T15:02:42.890787+0000","last_became_peered":"2024-09-13T15:02:42.890787+0000","last_unstale":"2024-09-13T15:03:06.542601+0000","last_undegraded":"2024-09-13T15:03:06.542601+0000","last_fullsized":"2024-09-13T15:03:06.542601+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:38.214865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:38.214865+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:12:04.179567+0000","scrub_duration":0.0016389639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1981,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.942731+0000","last_change":"2024-09-13T15:02:42.901601+0000","last_active":"2024-09-13T15:03:14.942731+0000","last_peered":"2024-09-13T15:03:14.942731+0000","last_clean":"2024-09-13T15:03:14.942731+0000","last_became_active":"2024-09-13T15:02:42.900518+0000","last_became_peered":"2024-09-13T15:02:42.900518+0000","last_unstale":"2024-09-13T15:03:14.942731+0000","last_undegraded":"2024-09-13T15:03:14.942731+0000","last_fullsized":"2024-09-13T15:03:14.942731+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:57:33.494780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":130,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552406+0000","last_change":"2024-09-13T15:02:42.885564+0000","last_active":"2024-09-13T15:03:06.552406+0000","last_peered":"2024-09-13T15:03:06.552406+0000","last_clean":"2024-09-13T15:03:06.552406+0000","last_became_active":"2024-09-13T15:02:42.884990+0000","last_became_peered":"2024-09-13T15:02:42.884990+0000","last_unstale":"2024-09-13T15:03:06.552406+0000","last_undegraded":"2024-09-13T15:03:06.552406+0000","last_fullsized":"2024-09-13T15:03:06.552406+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:51:42.556111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"261'100","reported_seq":493,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.933692+0000","last_change":"2024-09-13T15:02:42.884301+0000","last_active":"2024-09-13T15:03:14.933692+0000","last_peered":"2024-09-13T15:03:14.933692+0000","last_clean":"2024-09-13T15:03:14.933692+0000","last_became_active":"2024-09-13T15:02:42.883961+0000","last_became_peered":"2024-09-13T15:02:42.883961+0000","last_unstale":"2024-09-13T15:03:14.933692+0000","last_undegraded":"2024-09-13T15:03:14.933692+0000","last_fullsized":"2024-09-13T15:03:14.933692+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.037517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.037517+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:09:38.307408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":25,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.542520+0000","last_change":"2024-09-13T15:02:42.922069+0000","last_active":"2024-09-13T15:03:06.542520+0000","last_peered":"2024-09-13T15:03:06.542520+0000","last_clean":"2024-09-13T15:03:06.542520+0000","last_became_active":"2024-09-13T15:02:42.893632+0000","last_became_peered":"2024-09-13T15:02:42.893632+0000","last_unstale":"2024-09-13T15:03:06.542520+0000","last_undegraded":"2024-09-13T15:03:06.542520+0000","last_fullsized":"2024-09-13T15:03:06.542520+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:51:16.796405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":270,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":243,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.928225+0000","last_change":"2024-09-13T14:54:05.614569+0000","last_active":"2024-09-13T15:03:14.928225+0000","last_peered":"2024-09-13T15:03:14.928225+0000","last_clean":"2024-09-13T15:03:14.928225+0000","last_became_active":"2024-09-13T14:54:05.611810+0000","last_became_peered":"2024-09-13T14:54:05.611810+0000","last_unstale":"2024-09-13T15:03:14.928225+0000","last_undegraded":"2024-09-13T15:03:14.928225+0000","last_fullsized":"2024-09-13T15:03:14.928225+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:46:02.721438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"261'100","reported_seq":130,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552769+0000","last_change":"2024-09-13T15:02:42.887347+0000","last_active":"2024-09-13T15:03:06.552769+0000","last_peered":"2024-09-13T15:03:06.552769+0000","last_clean":"2024-09-13T15:03:06.552769+0000","last_became_active":"2024-09-13T15:02:42.886986+0000","last_became_peered":"2024-09-13T15:02:42.886986+0000","last_unstale":"2024-09-13T15:03:06.552769+0000","last_undegraded":"2024-09-13T15:03:06.552769+0000","last_fullsized":"2024-09-13T15:03:06.552769+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:54.933410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:54.933410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:18.197248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":29,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.543657+0000","last_change":"2024-09-13T15:02:42.912172+0000","last_active":"2024-09-13T15:03:06.543657+0000","last_peered":"2024-09-13T15:03:06.543657+0000","last_clean":"2024-09-13T15:03:06.543657+0000","last_became_active":"2024-09-13T15:02:42.880043+0000","last_became_peered":"2024-09-13T15:02:42.880043+0000","last_unstale":"2024-09-13T15:03:06.543657+0000","last_undegraded":"2024-09-13T15:03:06.543657+0000","last_fullsized":"2024-09-13T15:03:06.543657+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:46:15.229979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14232,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":129,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.929176+0000","last_change":"2024-09-13T15:00:25.091881+0000","last_active":"2024-09-13T15:03:14.929176+0000","last_peered":"2024-09-13T15:03:14.929176+0000","last_clean":"2024-09-13T15:03:14.929176+0000","last_became_active":"2024-09-13T15:00:25.091538+0000","last_became_peered":"2024-09-13T15:00:25.091538+0000","last_unstale":"2024-09-13T15:03:14.929176+0000","last_undegraded":"2024-09-13T15:03:14.929176+0000","last_fullsized":"2024-09-13T15:03:14.929176+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:29:32.889528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.16","version":"261'100","reported_seq":257,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.931330+0000","last_change":"2024-09-13T15:00:25.083905+0000","last_active":"2024-09-13T15:03:14.931330+0000","last_peered":"2024-09-13T15:03:14.931330+0000","last_clean":"2024-09-13T15:03:14.931330+0000","last_became_active":"2024-09-13T15:00:25.082642+0000","last_became_peered":"2024-09-13T15:00:25.082642+0000","last_unstale":"2024-09-13T15:03:14.931330+0000","last_undegraded":"2024-09-13T15:03:14.931330+0000","last_fullsized":"2024-09-13T15:03:14.931330+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:48.363045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:48.363045+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:31:38.562138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.e","version":"1337'3898","reported_seq":3925,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:27.271202+0000","last_change":"2024-09-13T15:02:44.891854+0000","last_active":"2024-09-13T15:03:27.271202+0000","last_peered":"2024-09-13T15:03:27.271202+0000","last_clean":"2024-09-13T15:03:27.271202+0000","last_became_active":"2024-09-13T15:02:44.891563+0000","last_became_peered":"2024-09-13T15:02:44.891563+0000","last_unstale":"2024-09-13T15:03:27.271202+0000","last_undegraded":"2024-09-13T15:03:27.271202+0000","last_fullsized":"2024-09-13T15:03:27.271202+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3898,"log_dups_size":0,"ondisk_log_size":3898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:18:05.783586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255459328,"num_objects":3898,"num_object_clones":0,"num_object_copies":11694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3898,"num_write_kb":249472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":100,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.551906+0000","last_change":"2024-09-13T15:02:42.884334+0000","last_active":"2024-09-13T15:03:06.551906+0000","last_peered":"2024-09-13T15:03:06.551906+0000","last_clean":"2024-09-13T15:03:06.551906+0000","last_became_active":"2024-09-13T15:02:42.884006+0000","last_became_peered":"2024-09-13T15:02:42.884006+0000","last_unstale":"2024-09-13T15:03:06.551906+0000","last_undegraded":"2024-09-13T15:03:06.551906+0000","last_fullsized":"2024-09-13T15:03:06.551906+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:09.872495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.544579+0000","last_change":"2024-09-13T15:02:42.882332+0000","last_active":"2024-09-13T15:03:06.544579+0000","last_peered":"2024-09-13T15:03:06.544579+0000","last_clean":"2024-09-13T15:03:06.544579+0000","last_became_active":"2024-09-13T15:02:42.881175+0000","last_became_peered":"2024-09-13T15:02:42.881175+0000","last_unstale":"2024-09-13T15:03:06.544579+0000","last_undegraded":"2024-09-13T15:03:06.544579+0000","last_fullsized":"2024-09-13T15:03:06.544579+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:52.763730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":2028,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.929733+0000","last_change":"2024-09-13T15:02:42.907810+0000","last_active":"2024-09-13T15:03:14.929733+0000","last_peered":"2024-09-13T15:03:14.929733+0000","last_clean":"2024-09-13T15:03:14.929733+0000","last_became_active":"2024-09-13T15:02:42.901460+0000","last_became_peered":"2024-09-13T15:02:42.901460+0000","last_unstale":"2024-09-13T15:03:14.929733+0000","last_undegraded":"2024-09-13T15:03:14.929733+0000","last_fullsized":"2024-09-13T15:03:14.929733+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:57:16.999581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"261'100","reported_seq":438,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.937642+0000","last_change":"2024-09-13T15:02:42.899622+0000","last_active":"2024-09-13T15:03:14.937642+0000","last_peered":"2024-09-13T15:03:14.937642+0000","last_clean":"2024-09-13T15:03:14.937642+0000","last_became_active":"2024-09-13T15:02:42.897715+0000","last_became_peered":"2024-09-13T15:02:42.897715+0000","last_unstale":"2024-09-13T15:03:14.937642+0000","last_undegraded":"2024-09-13T15:03:14.937642+0000","last_fullsized":"2024-09-13T15:03:14.937642+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.710192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.710192+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:50:03.858097+0000","scrub_duration":0.0019250910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"1337'3921","reported_seq":3937,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:27.374675+0000","last_change":"2024-09-13T15:02:44.893937+0000","last_active":"2024-09-13T15:03:27.374675+0000","last_peered":"2024-09-13T15:03:27.374675+0000","last_clean":"2024-09-13T15:03:27.374675+0000","last_became_active":"2024-09-13T15:02:44.893588+0000","last_became_peered":"2024-09-13T15:02:44.893588+0000","last_unstale":"2024-09-13T15:03:27.374675+0000","last_undegraded":"2024-09-13T15:03:27.374675+0000","last_fullsized":"2024-09-13T15:03:27.374675+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3921,"log_dups_size":0,"ondisk_log_size":3921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:25.119259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256966656,"num_objects":3921,"num_object_clones":0,"num_object_copies":11763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3921,"num_write_kb":250944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":320,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.936484+0000","last_change":"2024-09-13T15:02:42.896729+0000","last_active":"2024-09-13T15:03:14.936484+0000","last_peered":"2024-09-13T15:03:14.936484+0000","last_clean":"2024-09-13T15:03:14.936484+0000","last_became_active":"2024-09-13T15:02:42.896414+0000","last_became_peered":"2024-09-13T15:02:42.896414+0000","last_unstale":"2024-09-13T15:03:14.936484+0000","last_undegraded":"2024-09-13T15:03:14.936484+0000","last_fullsized":"2024-09-13T15:03:14.936484+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:53:32.471811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":258,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.931424+0000","last_change":"2024-09-13T15:00:25.070169+0000","last_active":"2024-09-13T15:03:14.931424+0000","last_peered":"2024-09-13T15:03:14.931424+0000","last_clean":"2024-09-13T15:03:14.931424+0000","last_became_active":"2024-09-13T15:00:25.069849+0000","last_became_peered":"2024-09-13T15:00:25.069849+0000","last_unstale":"2024-09-13T15:03:14.931424+0000","last_undegraded":"2024-09-13T15:03:14.931424+0000","last_fullsized":"2024-09-13T15:03:14.931424+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:56.491207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":2044,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.938439+0000","last_change":"2024-09-13T15:02:42.912751+0000","last_active":"2024-09-13T15:03:14.938439+0000","last_peered":"2024-09-13T15:03:14.938439+0000","last_clean":"2024-09-13T15:03:14.938439+0000","last_became_active":"2024-09-13T15:02:42.903177+0000","last_became_peered":"2024-09-13T15:02:42.903177+0000","last_unstale":"2024-09-13T15:03:14.938439+0000","last_undegraded":"2024-09-13T15:03:14.938439+0000","last_fullsized":"2024-09-13T15:03:14.938439+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:00:52.788057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"261'100","reported_seq":250,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.932353+0000","last_change":"2024-09-13T15:00:25.060018+0000","last_active":"2024-09-13T15:03:14.932353+0000","last_peered":"2024-09-13T15:03:14.932353+0000","last_clean":"2024-09-13T15:03:14.932353+0000","last_became_active":"2024-09-13T15:00:25.059438+0000","last_became_peered":"2024-09-13T15:00:25.059438+0000","last_unstale":"2024-09-13T15:03:14.932353+0000","last_undegraded":"2024-09-13T15:03:14.932353+0000","last_fullsized":"2024-09-13T15:03:14.932353+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:51:31.310115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:51:31.310115+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:08.861237+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.c","version":"1337'4018","reported_seq":4042,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:27.273147+0000","last_change":"2024-09-13T15:02:44.894339+0000","last_active":"2024-09-13T15:03:27.273147+0000","last_peered":"2024-09-13T15:03:27.273147+0000","last_clean":"2024-09-13T15:03:27.273147+0000","last_became_active":"2024-09-13T15:02:44.893967+0000","last_became_peered":"2024-09-13T15:02:44.893967+0000","last_unstale":"2024-09-13T15:03:27.273147+0000","last_undegraded":"2024-09-13T15:03:27.273147+0000","last_fullsized":"2024-09-13T15:03:27.273147+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":4018,"log_dups_size":0,"ondisk_log_size":4018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:34.261719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263323648,"num_objects":4018,"num_object_clones":0,"num_object_copies":12054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4018,"num_write_kb":257152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2054,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552575+0000","last_change":"2024-09-13T15:02:42.885186+0000","last_active":"2024-09-13T15:03:06.552575+0000","last_peered":"2024-09-13T15:03:06.552575+0000","last_clean":"2024-09-13T15:03:06.552575+0000","last_became_active":"2024-09-13T15:02:42.884666+0000","last_became_peered":"2024-09-13T15:02:42.884666+0000","last_unstale":"2024-09-13T15:03:06.552575+0000","last_undegraded":"2024-09-13T15:03:06.552575+0000","last_fullsized":"2024-09-13T15:03:06.552575+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:17.386836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.542680+0000","last_change":"2024-09-13T15:02:42.891180+0000","last_active":"2024-09-13T15:03:06.542680+0000","last_peered":"2024-09-13T15:03:06.542680+0000","last_clean":"2024-09-13T15:03:06.542680+0000","last_became_active":"2024-09-13T15:02:42.890141+0000","last_became_peered":"2024-09-13T15:02:42.890141+0000","last_unstale":"2024-09-13T15:03:06.542680+0000","last_undegraded":"2024-09-13T15:03:06.542680+0000","last_fullsized":"2024-09-13T15:03:06.542680+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:28.910151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":258,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.930775+0000","last_change":"2024-09-13T14:55:41.892125+0000","last_active":"2024-09-13T15:03:14.930775+0000","last_peered":"2024-09-13T15:03:14.930775+0000","last_clean":"2024-09-13T15:03:14.930775+0000","last_became_active":"2024-09-13T14:55:41.891812+0000","last_became_peered":"2024-09-13T14:55:41.891812+0000","last_unstale":"2024-09-13T15:03:14.930775+0000","last_undegraded":"2024-09-13T15:03:14.930775+0000","last_fullsized":"2024-09-13T15:03:14.930775+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:03.359541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"261'100","reported_seq":503,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.938641+0000","last_change":"2024-09-13T15:02:42.898674+0000","last_active":"2024-09-13T15:03:14.938641+0000","last_peered":"2024-09-13T15:03:14.938641+0000","last_clean":"2024-09-13T15:03:14.938641+0000","last_became_active":"2024-09-13T15:02:42.898289+0000","last_became_peered":"2024-09-13T15:02:42.898289+0000","last_unstale":"2024-09-13T15:03:14.938641+0000","last_undegraded":"2024-09-13T15:03:14.938641+0000","last_fullsized":"2024-09-13T15:03:14.938641+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:33.574443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:33.574443+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:54:37.955957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"1337'3610","reported_seq":3637,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:23.395802+0000","last_change":"2024-09-13T15:02:45.496583+0000","last_active":"2024-09-13T15:03:23.395802+0000","last_peered":"2024-09-13T15:03:23.395802+0000","last_clean":"2024-09-13T15:03:23.395802+0000","last_became_active":"2024-09-13T15:02:45.496353+0000","last_became_peered":"2024-09-13T15:02:45.496353+0000","last_unstale":"2024-09-13T15:03:23.395802+0000","last_undegraded":"2024-09-13T15:03:23.395802+0000","last_fullsized":"2024-09-13T15:03:23.395802+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3610,"log_dups_size":0,"ondisk_log_size":3610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:09:02.073789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236584960,"num_objects":3610,"num_object_clones":0,"num_object_copies":10830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3610,"num_write_kb":231040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":272,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.928370+0000","last_change":"2024-09-13T15:00:25.079609+0000","last_active":"2024-09-13T15:03:14.928370+0000","last_peered":"2024-09-13T15:03:14.928370+0000","last_clean":"2024-09-13T15:03:14.928370+0000","last_became_active":"2024-09-13T15:00:25.077887+0000","last_became_peered":"2024-09-13T15:00:25.077887+0000","last_unstale":"2024-09-13T15:03:14.928370+0000","last_undegraded":"2024-09-13T15:03:14.928370+0000","last_fullsized":"2024-09-13T15:03:14.928370+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:13:04.513297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":265,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.930602+0000","last_change":"2024-09-13T15:02:42.899923+0000","last_active":"2024-09-13T15:03:14.930602+0000","last_peered":"2024-09-13T15:03:14.930602+0000","last_clean":"2024-09-13T15:03:14.930602+0000","last_became_active":"2024-09-13T15:02:42.898278+0000","last_became_peered":"2024-09-13T15:02:42.898278+0000","last_unstale":"2024-09-13T15:03:14.930602+0000","last_undegraded":"2024-09-13T15:03:14.930602+0000","last_fullsized":"2024-09-13T15:03:14.930602+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:39:54.048710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":110,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552290+0000","last_change":"2024-09-13T14:55:41.839293+0000","last_active":"2024-09-13T15:03:06.552290+0000","last_peered":"2024-09-13T15:03:06.552290+0000","last_clean":"2024-09-13T15:03:06.552290+0000","last_became_active":"2024-09-13T14:55:41.838820+0000","last_became_peered":"2024-09-13T14:55:41.838820+0000","last_unstale":"2024-09-13T15:03:06.552290+0000","last_undegraded":"2024-09-13T15:03:06.552290+0000","last_fullsized":"2024-09-13T15:03:06.552290+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:14:27.811437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"261'100","reported_seq":427,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.938288+0000","last_change":"2024-09-13T15:02:42.903818+0000","last_active":"2024-09-13T15:03:14.938288+0000","last_peered":"2024-09-13T15:03:14.938288+0000","last_clean":"2024-09-13T15:03:14.938288+0000","last_became_active":"2024-09-13T15:02:42.902705+0000","last_became_peered":"2024-09-13T15:02:42.902705+0000","last_unstale":"2024-09-13T15:03:14.938288+0000","last_undegraded":"2024-09-13T15:03:14.938288+0000","last_fullsized":"2024-09-13T15:03:14.938288+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:36.965775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:45:36.965775+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:40:58.317613+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.b","version":"1337'3632","reported_seq":3659,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:23.850728+0000","last_change":"2024-09-13T15:02:44.895572+0000","last_active":"2024-09-13T15:03:23.850728+0000","last_peered":"2024-09-13T15:03:23.850728+0000","last_clean":"2024-09-13T15:03:23.850728+0000","last_became_active":"2024-09-13T15:02:44.895300+0000","last_became_peered":"2024-09-13T15:02:44.895300+0000","last_unstale":"2024-09-13T15:03:23.850728+0000","last_undegraded":"2024-09-13T15:03:23.850728+0000","last_fullsized":"2024-09-13T15:03:23.850728+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3632,"log_dups_size":0,"ondisk_log_size":3632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:46:47.154024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238026752,"num_objects":3632,"num_object_clones":0,"num_object_copies":10896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3632,"num_write_kb":232448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":244,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.930517+0000","last_change":"2024-09-13T14:54:05.596068+0000","last_active":"2024-09-13T15:03:14.930517+0000","last_peered":"2024-09-13T15:03:14.930517+0000","last_clean":"2024-09-13T15:03:14.930517+0000","last_became_active":"2024-09-13T14:54:05.595743+0000","last_became_peered":"2024-09-13T14:54:05.595743+0000","last_unstale":"2024-09-13T15:03:14.930517+0000","last_undegraded":"2024-09-13T15:03:14.930517+0000","last_fullsized":"2024-09-13T15:03:14.930517+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:15:39.660370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":649,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.546483+0000","last_change":"2024-09-13T15:00:25.089570+0000","last_active":"2024-09-13T15:03:06.546483+0000","last_peered":"2024-09-13T15:03:06.546483+0000","last_clean":"2024-09-13T15:03:06.546483+0000","last_became_active":"2024-09-13T15:00:25.088373+0000","last_became_peered":"2024-09-13T15:00:25.088373+0000","last_unstale":"2024-09-13T15:03:06.546483+0000","last_undegraded":"2024-09-13T15:03:06.546483+0000","last_fullsized":"2024-09-13T15:03:06.546483+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:36:04.220868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":159,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552364+0000","last_change":"2024-09-13T14:55:41.889668+0000","last_active":"2024-09-13T15:03:06.552364+0000","last_peered":"2024-09-13T15:03:06.552364+0000","last_clean":"2024-09-13T15:03:06.552364+0000","last_became_active":"2024-09-13T14:55:41.889195+0000","last_became_peered":"2024-09-13T14:55:41.889195+0000","last_unstale":"2024-09-13T15:03:06.552364+0000","last_undegraded":"2024-09-13T15:03:06.552364+0000","last_fullsized":"2024-09-13T15:03:06.552364+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:52.765208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"261'100","reported_seq":140,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552026+0000","last_change":"2024-09-13T15:02:42.878126+0000","last_active":"2024-09-13T15:03:06.552026+0000","last_peered":"2024-09-13T15:03:06.552026+0000","last_clean":"2024-09-13T15:03:06.552026+0000","last_became_active":"2024-09-13T15:02:42.877740+0000","last_became_peered":"2024-09-13T15:02:42.877740+0000","last_unstale":"2024-09-13T15:03:06.552026+0000","last_undegraded":"2024-09-13T15:03:06.552026+0000","last_fullsized":"2024-09-13T15:03:06.552026+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:57.024823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:57.024823+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:16.987315+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"1337'3629","reported_seq":3656,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:23.851659+0000","last_change":"2024-09-13T15:02:45.493849+0000","last_active":"2024-09-13T15:03:23.851659+0000","last_peered":"2024-09-13T15:03:23.851659+0000","last_clean":"2024-09-13T15:03:23.851659+0000","last_became_active":"2024-09-13T15:02:45.493251+0000","last_became_peered":"2024-09-13T15:02:45.493251+0000","last_unstale":"2024-09-13T15:03:23.851659+0000","last_undegraded":"2024-09-13T15:03:23.851659+0000","last_fullsized":"2024-09-13T15:03:23.851659+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3629,"log_dups_size":0,"ondisk_log_size":3629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:15.904351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237830144,"num_objects":3629,"num_object_clones":0,"num_object_copies":10887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3629,"num_write_kb":232256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":272,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.929240+0000","last_change":"2024-09-13T15:02:42.897548+0000","last_active":"2024-09-13T15:03:14.929240+0000","last_peered":"2024-09-13T15:03:14.929240+0000","last_clean":"2024-09-13T15:03:14.929240+0000","last_became_active":"2024-09-13T15:02:42.897144+0000","last_became_peered":"2024-09-13T15:02:42.897144+0000","last_unstale":"2024-09-13T15:03:14.929240+0000","last_undegraded":"2024-09-13T15:03:14.929240+0000","last_fullsized":"2024-09-13T15:03:14.929240+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:57.915560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":244,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.930009+0000","last_change":"2024-09-13T14:54:05.614107+0000","last_active":"2024-09-13T15:03:14.930009+0000","last_peered":"2024-09-13T15:03:14.930009+0000","last_clean":"2024-09-13T15:03:14.930009+0000","last_became_active":"2024-09-13T14:54:05.613429+0000","last_became_peered":"2024-09-13T14:54:05.613429+0000","last_unstale":"2024-09-13T15:03:14.930009+0000","last_undegraded":"2024-09-13T15:03:14.930009+0000","last_fullsized":"2024-09-13T15:03:14.930009+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:51.312413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2089,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.931409+0000","last_change":"2024-09-13T15:00:05.042993+0000","last_active":"2024-09-13T15:03:14.931409+0000","last_peered":"2024-09-13T15:03:14.931409+0000","last_clean":"2024-09-13T15:03:14.931409+0000","last_became_active":"2024-09-13T15:00:05.041250+0000","last_became_peered":"2024-09-13T15:00:05.041250+0000","last_unstale":"2024-09-13T15:03:14.931409+0000","last_undegraded":"2024-09-13T15:03:14.931409+0000","last_fullsized":"2024-09-13T15:03:14.931409+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:44.484707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"261'100","reported_seq":29,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.542393+0000","last_change":"2024-09-13T15:02:59.897496+0000","last_active":"2024-09-13T15:03:06.542393+0000","last_peered":"2024-09-13T15:03:06.542393+0000","last_clean":"2024-09-13T15:03:06.542393+0000","last_became_active":"2024-09-13T15:02:42.883421+0000","last_became_peered":"2024-09-13T15:02:42.883421+0000","last_unstale":"2024-09-13T15:03:06.542393+0000","last_undegraded":"2024-09-13T15:03:06.542393+0000","last_fullsized":"2024-09-13T15:03:06.542393+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T15:02:59.897422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T15:02:59.897422+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:07:39.205660+0000","scrub_duration":0.0032184959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"1337'3594","reported_seq":3621,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:23.850611+0000","last_change":"2024-09-13T15:02:45.495691+0000","last_active":"2024-09-13T15:03:23.850611+0000","last_peered":"2024-09-13T15:03:23.850611+0000","last_clean":"2024-09-13T15:03:23.850611+0000","last_became_active":"2024-09-13T15:02:45.495215+0000","last_became_peered":"2024-09-13T15:02:45.495215+0000","last_unstale":"2024-09-13T15:03:23.850611+0000","last_undegraded":"2024-09-13T15:03:23.850611+0000","last_fullsized":"2024-09-13T15:03:23.850611+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3594,"log_dups_size":0,"ondisk_log_size":3594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:05.547819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235536384,"num_objects":3594,"num_object_clones":0,"num_object_copies":10782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3594,"num_write_kb":230016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1972,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.929906+0000","last_change":"2024-09-13T15:02:42.902299+0000","last_active":"2024-09-13T15:03:14.929906+0000","last_peered":"2024-09-13T15:03:14.929906+0000","last_clean":"2024-09-13T15:03:14.929906+0000","last_became_active":"2024-09-13T15:02:42.898373+0000","last_became_peered":"2024-09-13T15:02:42.898373+0000","last_unstale":"2024-09-13T15:03:14.929906+0000","last_undegraded":"2024-09-13T15:03:14.929906+0000","last_fullsized":"2024-09-13T15:03:14.929906+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:18:54.825311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2010,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.936879+0000","last_change":"2024-09-13T15:00:25.082949+0000","last_active":"2024-09-13T15:03:14.936879+0000","last_peered":"2024-09-13T15:03:14.936879+0000","last_clean":"2024-09-13T15:03:14.936879+0000","last_became_active":"2024-09-13T15:00:25.082279+0000","last_became_peered":"2024-09-13T15:00:25.082279+0000","last_unstale":"2024-09-13T15:03:14.936879+0000","last_undegraded":"2024-09-13T15:03:14.936879+0000","last_fullsized":"2024-09-13T15:03:14.936879+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:49.021898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.542079+0000","last_change":"2024-09-13T15:02:42.895072+0000","last_active":"2024-09-13T15:03:06.542079+0000","last_peered":"2024-09-13T15:03:06.542079+0000","last_clean":"2024-09-13T15:03:06.542079+0000","last_became_active":"2024-09-13T15:02:42.894718+0000","last_became_peered":"2024-09-13T15:02:42.894718+0000","last_unstale":"2024-09-13T15:03:06.542079+0000","last_undegraded":"2024-09-13T15:03:06.542079+0000","last_fullsized":"2024-09-13T15:03:06.542079+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:55:10.117260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"261'100","reported_seq":258,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.929001+0000","last_change":"2024-09-13T15:00:25.085109+0000","last_active":"2024-09-13T15:03:14.929001+0000","last_peered":"2024-09-13T15:03:14.929001+0000","last_clean":"2024-09-13T15:03:14.929001+0000","last_became_active":"2024-09-13T15:00:25.084170+0000","last_became_peered":"2024-09-13T15:00:25.084170+0000","last_unstale":"2024-09-13T15:03:14.929001+0000","last_undegraded":"2024-09-13T15:03:14.929001+0000","last_fullsized":"2024-09-13T15:03:14.929001+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.008264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.008264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:52:58.309554+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"1337'3542","reported_seq":3568,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:23.857237+0000","last_change":"2024-09-13T15:02:45.493742+0000","last_active":"2024-09-13T15:03:23.857237+0000","last_peered":"2024-09-13T15:03:23.857237+0000","last_clean":"2024-09-13T15:03:23.857237+0000","last_became_active":"2024-09-13T15:02:45.493391+0000","last_became_peered":"2024-09-13T15:02:45.493391+0000","last_unstale":"2024-09-13T15:03:23.857237+0000","last_undegraded":"2024-09-13T15:03:23.857237+0000","last_fullsized":"2024-09-13T15:03:23.857237+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3542,"log_dups_size":0,"ondisk_log_size":3542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:26.044885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232128512,"num_objects":3542,"num_object_clones":0,"num_object_copies":10626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3542,"num_write_kb":226688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":127,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.543826+0000","last_change":"2024-09-13T15:00:25.040768+0000","last_active":"2024-09-13T15:03:06.543826+0000","last_peered":"2024-09-13T15:03:06.543826+0000","last_clean":"2024-09-13T15:03:06.543826+0000","last_became_active":"2024-09-13T15:00:25.040487+0000","last_became_peered":"2024-09-13T15:00:25.040487+0000","last_unstale":"2024-09-13T15:03:06.543826+0000","last_undegraded":"2024-09-13T15:03:06.543826+0000","last_fullsized":"2024-09-13T15:03:06.543826+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:44.500104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":431,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.940788+0000","last_change":"2024-09-13T14:55:41.835887+0000","last_active":"2024-09-13T15:03:14.940788+0000","last_peered":"2024-09-13T15:03:14.940788+0000","last_clean":"2024-09-13T15:03:14.940788+0000","last_became_active":"2024-09-13T14:55:41.834413+0000","last_became_peered":"2024-09-13T14:55:41.834413+0000","last_unstale":"2024-09-13T15:03:14.940788+0000","last_undegraded":"2024-09-13T15:03:14.940788+0000","last_fullsized":"2024-09-13T15:03:14.940788+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:01.850578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":176,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552676+0000","last_change":"2024-09-13T15:02:42.915709+0000","last_active":"2024-09-13T15:03:06.552676+0000","last_peered":"2024-09-13T15:03:06.552676+0000","last_clean":"2024-09-13T15:03:06.552676+0000","last_became_active":"2024-09-13T15:02:42.885581+0000","last_became_peered":"2024-09-13T15:02:42.885581+0000","last_unstale":"2024-09-13T15:03:06.552676+0000","last_undegraded":"2024-09-13T15:03:06.552676+0000","last_fullsized":"2024-09-13T15:03:06.552676+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:44:51.824670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"261'100","reported_seq":19,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.542158+0000","last_change":"2024-09-13T15:02:42.884606+0000","last_active":"2024-09-13T15:03:06.542158+0000","last_peered":"2024-09-13T15:03:06.542158+0000","last_clean":"2024-09-13T15:03:06.542158+0000","last_became_active":"2024-09-13T15:02:42.884269+0000","last_became_peered":"2024-09-13T15:02:42.884269+0000","last_unstale":"2024-09-13T15:03:06.542158+0000","last_undegraded":"2024-09-13T15:03:06.542158+0000","last_fullsized":"2024-09-13T15:03:06.542158+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T15:00:44.119431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T15:00:44.119431+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:06:31.424756+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.0","version":"1337'3903","reported_seq":3930,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:27.291457+0000","last_change":"2024-09-13T15:02:44.896248+0000","last_active":"2024-09-13T15:03:27.291457+0000","last_peered":"2024-09-13T15:03:27.291457+0000","last_clean":"2024-09-13T15:03:27.291457+0000","last_became_active":"2024-09-13T15:02:44.895577+0000","last_became_peered":"2024-09-13T15:02:44.895577+0000","last_unstale":"2024-09-13T15:03:27.291457+0000","last_undegraded":"2024-09-13T15:03:27.291457+0000","last_fullsized":"2024-09-13T15:03:27.291457+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3903,"log_dups_size":0,"ondisk_log_size":3903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:20:42.078814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255787008,"num_objects":3903,"num_object_clones":0,"num_object_copies":11709,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3903,"num_write_kb":249792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":265,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.929326+0000","last_change":"2024-09-13T15:02:42.896422+0000","last_active":"2024-09-13T15:03:14.929326+0000","last_peered":"2024-09-13T15:03:14.929326+0000","last_clean":"2024-09-13T15:03:14.929326+0000","last_became_active":"2024-09-13T15:02:42.896112+0000","last_became_peered":"2024-09-13T15:02:42.896112+0000","last_unstale":"2024-09-13T15:03:14.929326+0000","last_undegraded":"2024-09-13T15:03:14.929326+0000","last_fullsized":"2024-09-13T15:03:14.929326+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:36:04.129918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":438,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.936859+0000","last_change":"2024-09-13T15:02:42.901428+0000","last_active":"2024-09-13T15:03:14.936859+0000","last_peered":"2024-09-13T15:03:14.936859+0000","last_clean":"2024-09-13T15:03:14.936859+0000","last_became_active":"2024-09-13T15:02:42.901131+0000","last_became_peered":"2024-09-13T15:02:42.901131+0000","last_unstale":"2024-09-13T15:03:14.936859+0000","last_undegraded":"2024-09-13T15:03:14.936859+0000","last_fullsized":"2024-09-13T15:03:14.936859+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:44.152548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"261'100","reported_seq":110,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552258+0000","last_change":"2024-09-13T14:55:41.784086+0000","last_active":"2024-09-13T15:03:06.552258+0000","last_peered":"2024-09-13T15:03:06.552258+0000","last_clean":"2024-09-13T15:03:06.552258+0000","last_became_active":"2024-09-13T14:55:41.783492+0000","last_became_peered":"2024-09-13T14:55:41.783492+0000","last_unstale":"2024-09-13T15:03:06.552258+0000","last_undegraded":"2024-09-13T15:03:06.552258+0000","last_fullsized":"2024-09-13T15:03:06.552258+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:49.280478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:49.280478+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:16:55.041094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"1337'3692","reported_seq":3717,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:23.847901+0000","last_change":"2024-09-13T15:02:45.496010+0000","last_active":"2024-09-13T15:03:23.847901+0000","last_peered":"2024-09-13T15:03:23.847901+0000","last_clean":"2024-09-13T15:03:23.847901+0000","last_became_active":"2024-09-13T15:02:45.495333+0000","last_became_peered":"2024-09-13T15:02:45.495333+0000","last_unstale":"2024-09-13T15:03:23.847901+0000","last_undegraded":"2024-09-13T15:03:23.847901+0000","last_fullsized":"2024-09-13T15:03:23.847901+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3692,"log_dups_size":0,"ondisk_log_size":3692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:03.628293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241958912,"num_objects":3692,"num_object_clones":0,"num_object_copies":11076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3692,"num_write_kb":236288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":129,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.931234+0000","last_change":"2024-09-13T15:00:25.069428+0000","last_active":"2024-09-13T15:03:14.931234+0000","last_peered":"2024-09-13T15:03:14.931234+0000","last_clean":"2024-09-13T15:03:14.931234+0000","last_became_active":"2024-09-13T15:00:25.068745+0000","last_became_peered":"2024-09-13T15:00:25.068745+0000","last_unstale":"2024-09-13T15:03:14.931234+0000","last_undegraded":"2024-09-13T15:03:14.931234+0000","last_fullsized":"2024-09-13T15:03:14.931234+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:37.264986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":786,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.931321+0000","last_change":"2024-09-13T15:02:42.882826+0000","last_active":"2024-09-13T15:03:14.931321+0000","last_peered":"2024-09-13T15:03:14.931321+0000","last_clean":"2024-09-13T15:03:14.931321+0000","last_became_active":"2024-09-13T15:02:42.882517+0000","last_became_peered":"2024-09-13T15:02:42.882517+0000","last_unstale":"2024-09-13T15:03:14.931321+0000","last_undegraded":"2024-09-13T15:03:14.931321+0000","last_fullsized":"2024-09-13T15:03:14.931321+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:45.948571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"261'100","reported_seq":265,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.929084+0000","last_change":"2024-09-13T15:00:05.050225+0000","last_active":"2024-09-13T15:03:14.929084+0000","last_peered":"2024-09-13T15:03:14.929084+0000","last_clean":"2024-09-13T15:03:14.929084+0000","last_became_active":"2024-09-13T15:00:05.049780+0000","last_became_peered":"2024-09-13T15:00:05.049780+0000","last_unstale":"2024-09-13T15:03:14.929084+0000","last_undegraded":"2024-09-13T15:03:14.929084+0000","last_fullsized":"2024-09-13T15:03:14.929084+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:47.405845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:47.405845+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:56:03.644141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2","version":"1337'3493","reported_seq":3520,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:23.406383+0000","last_change":"2024-09-13T15:02:44.896850+0000","last_active":"2024-09-13T15:03:23.406383+0000","last_peered":"2024-09-13T15:03:23.406383+0000","last_clean":"2024-09-13T15:03:23.406383+0000","last_became_active":"2024-09-13T15:02:44.896509+0000","last_became_peered":"2024-09-13T15:02:44.896509+0000","last_unstale":"2024-09-13T15:03:23.406383+0000","last_undegraded":"2024-09-13T15:03:23.406383+0000","last_fullsized":"2024-09-13T15:03:23.406383+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3493,"log_dups_size":0,"ondisk_log_size":3493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:31.028917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228917248,"num_objects":3493,"num_object_clones":0,"num_object_copies":10479,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3493,"num_write_kb":223552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1338,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.932629+0000","last_change":"2024-09-13T15:00:25.058894+0000","last_active":"2024-09-13T15:03:14.932629+0000","last_peered":"2024-09-13T15:03:14.932629+0000","last_clean":"2024-09-13T15:03:14.932629+0000","last_became_active":"2024-09-13T15:00:25.058311+0000","last_became_peered":"2024-09-13T15:00:25.058311+0000","last_unstale":"2024-09-13T15:03:14.932629+0000","last_undegraded":"2024-09-13T15:03:14.932629+0000","last_fullsized":"2024-09-13T15:03:14.932629+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:04.030074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":129,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.928126+0000","last_change":"2024-09-13T15:00:25.154369+0000","last_active":"2024-09-13T15:03:14.928126+0000","last_peered":"2024-09-13T15:03:14.928126+0000","last_clean":"2024-09-13T15:03:14.928126+0000","last_became_active":"2024-09-13T15:00:25.154045+0000","last_became_peered":"2024-09-13T15:00:25.154045+0000","last_unstale":"2024-09-13T15:03:14.928126+0000","last_undegraded":"2024-09-13T15:03:14.928126+0000","last_fullsized":"2024-09-13T15:03:14.928126+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:58:48.342494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"261'100","reported_seq":110,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.551937+0000","last_change":"2024-09-13T14:55:41.844203+0000","last_active":"2024-09-13T15:03:06.551937+0000","last_peered":"2024-09-13T15:03:06.551937+0000","last_clean":"2024-09-13T15:03:06.551937+0000","last_became_active":"2024-09-13T14:55:41.843897+0000","last_became_peered":"2024-09-13T14:55:41.843897+0000","last_unstale":"2024-09-13T15:03:06.551937+0000","last_undegraded":"2024-09-13T15:03:06.551937+0000","last_fullsized":"2024-09-13T15:03:06.551937+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:55.816857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:55.816857+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:42:54.571599+0000","scrub_duration":270,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3","version":"1337'3973","reported_seq":3989,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:27.388332+0000","last_change":"2024-09-13T15:02:45.493751+0000","last_active":"2024-09-13T15:03:27.388332+0000","last_peered":"2024-09-13T15:03:27.388332+0000","last_clean":"2024-09-13T15:03:27.388332+0000","last_became_active":"2024-09-13T15:02:45.493377+0000","last_became_peered":"2024-09-13T15:02:45.493377+0000","last_unstale":"2024-09-13T15:03:27.388332+0000","last_undegraded":"2024-09-13T15:03:27.388332+0000","last_fullsized":"2024-09-13T15:03:27.388332+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3973,"log_dups_size":0,"ondisk_log_size":3973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:49.207437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260374528,"num_objects":3973,"num_object_clones":0,"num_object_copies":11919,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3973,"num_write_kb":254272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1365,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.937017+0000","last_change":"2024-09-13T15:00:25.045581+0000","last_active":"2024-09-13T15:03:14.937017+0000","last_peered":"2024-09-13T15:03:14.937017+0000","last_clean":"2024-09-13T15:03:14.937017+0000","last_became_active":"2024-09-13T15:00:25.045225+0000","last_became_peered":"2024-09-13T15:00:25.045225+0000","last_unstale":"2024-09-13T15:03:14.937017+0000","last_undegraded":"2024-09-13T15:03:14.937017+0000","last_fullsized":"2024-09-13T15:03:14.937017+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:38.148339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":477,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.937842+0000","last_change":"2024-09-13T15:02:42.909748+0000","last_active":"2024-09-13T15:03:14.937842+0000","last_peered":"2024-09-13T15:03:14.937842+0000","last_clean":"2024-09-13T15:03:14.937842+0000","last_became_active":"2024-09-13T15:02:42.897824+0000","last_became_peered":"2024-09-13T15:02:42.897824+0000","last_unstale":"2024-09-13T15:03:14.937842+0000","last_undegraded":"2024-09-13T15:03:14.937842+0000","last_fullsized":"2024-09-13T15:03:14.937842+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:05.224186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"261'100","reported_seq":29,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.542088+0000","last_change":"2024-09-13T15:02:56.904746+0000","last_active":"2024-09-13T15:03:06.542088+0000","last_peered":"2024-09-13T15:03:06.542088+0000","last_clean":"2024-09-13T15:03:06.542088+0000","last_became_active":"2024-09-13T15:02:42.894858+0000","last_became_peered":"2024-09-13T15:02:42.894858+0000","last_unstale":"2024-09-13T15:03:06.542088+0000","last_undegraded":"2024-09-13T15:03:06.542088+0000","last_fullsized":"2024-09-13T15:03:06.542088+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T15:02:56.904689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T15:02:56.904689+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:33:17.488947+0000","scrub_duration":0.001699702,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.4","version":"1337'3829","reported_seq":3845,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:27.364725+0000","last_change":"2024-09-13T15:02:45.493795+0000","last_active":"2024-09-13T15:03:27.364725+0000","last_peered":"2024-09-13T15:03:27.364725+0000","last_clean":"2024-09-13T15:03:27.364725+0000","last_became_active":"2024-09-13T15:02:45.493370+0000","last_became_peered":"2024-09-13T15:02:45.493370+0000","last_unstale":"2024-09-13T15:03:27.364725+0000","last_undegraded":"2024-09-13T15:03:27.364725+0000","last_fullsized":"2024-09-13T15:03:27.364725+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3829,"log_dups_size":0,"ondisk_log_size":3829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:32:50.510810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250937344,"num_objects":3829,"num_object_clones":0,"num_object_copies":11487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3829,"num_write_kb":245056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":122,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.931771+0000","last_change":"2024-09-13T15:00:25.060205+0000","last_active":"2024-09-13T15:03:14.931771+0000","last_peered":"2024-09-13T15:03:14.931771+0000","last_clean":"2024-09-13T15:03:14.931771+0000","last_became_active":"2024-09-13T15:00:25.059901+0000","last_became_peered":"2024-09-13T15:00:25.059901+0000","last_unstale":"2024-09-13T15:03:14.931771+0000","last_undegraded":"2024-09-13T15:03:14.931771+0000","last_fullsized":"2024-09-13T15:03:14.931771+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:33.964617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":110,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552166+0000","last_change":"2024-09-13T14:55:41.848811+0000","last_active":"2024-09-13T15:03:06.552166+0000","last_peered":"2024-09-13T15:03:06.552166+0000","last_clean":"2024-09-13T15:03:06.552166+0000","last_became_active":"2024-09-13T14:55:41.848542+0000","last_became_peered":"2024-09-13T14:55:41.848542+0000","last_unstale":"2024-09-13T15:03:06.552166+0000","last_undegraded":"2024-09-13T15:03:06.552166+0000","last_fullsized":"2024-09-13T15:03:06.552166+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:48.534022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"261'100","reported_seq":29,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.543984+0000","last_change":"2024-09-13T15:03:00.851297+0000","last_active":"2024-09-13T15:03:06.543984+0000","last_peered":"2024-09-13T15:03:06.543984+0000","last_clean":"2024-09-13T15:03:06.543984+0000","last_became_active":"2024-09-13T15:02:42.894466+0000","last_became_peered":"2024-09-13T15:02:42.894466+0000","last_unstale":"2024-09-13T15:03:06.543984+0000","last_undegraded":"2024-09-13T15:03:06.543984+0000","last_fullsized":"2024-09-13T15:03:06.543984+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T15:03:00.851236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T15:03:00.851236+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:45:21.407912+0000","scrub_duration":0.004159968,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.5","version":"1337'3949","reported_seq":3965,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:27.394141+0000","last_change":"2024-09-13T15:02:45.494451+0000","last_active":"2024-09-13T15:03:27.394141+0000","last_peered":"2024-09-13T15:03:27.394141+0000","last_clean":"2024-09-13T15:03:27.394141+0000","last_became_active":"2024-09-13T15:02:45.494169+0000","last_became_peered":"2024-09-13T15:02:45.494169+0000","last_unstale":"2024-09-13T15:03:27.394141+0000","last_undegraded":"2024-09-13T15:03:27.394141+0000","last_fullsized":"2024-09-13T15:03:27.394141+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3949,"log_dups_size":0,"ondisk_log_size":3949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:45:17.495874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":258801664,"num_objects":3949,"num_object_clones":0,"num_object_copies":11847,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3949,"num_write_kb":252736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1353,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.546127+0000","last_change":"2024-09-13T15:02:42.880814+0000","last_active":"2024-09-13T15:03:06.546127+0000","last_peered":"2024-09-13T15:03:06.546127+0000","last_clean":"2024-09-13T15:03:06.546127+0000","last_became_active":"2024-09-13T15:02:42.880393+0000","last_became_peered":"2024-09-13T15:02:42.880393+0000","last_unstale":"2024-09-13T15:03:06.546127+0000","last_undegraded":"2024-09-13T15:03:06.546127+0000","last_fullsized":"2024-09-13T15:03:06.546127+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:03.222968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":23,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.543866+0000","last_change":"2024-09-13T15:02:42.914851+0000","last_active":"2024-09-13T15:03:06.543866+0000","last_peered":"2024-09-13T15:03:06.543866+0000","last_clean":"2024-09-13T15:03:06.543866+0000","last_became_active":"2024-09-13T15:02:42.884164+0000","last_became_peered":"2024-09-13T15:02:42.884164+0000","last_unstale":"2024-09-13T15:03:06.543866+0000","last_undegraded":"2024-09-13T15:03:06.543866+0000","last_fullsized":"2024-09-13T15:03:06.543866+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:26:54.412326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"261'100","reported_seq":120,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552862+0000","last_change":"2024-09-13T15:00:25.048330+0000","last_active":"2024-09-13T15:03:06.552862+0000","last_peered":"2024-09-13T15:03:06.552862+0000","last_clean":"2024-09-13T15:03:06.552862+0000","last_became_active":"2024-09-13T15:00:25.048001+0000","last_became_peered":"2024-09-13T15:00:25.048001+0000","last_unstale":"2024-09-13T15:03:06.552862+0000","last_undegraded":"2024-09-13T15:03:06.552862+0000","last_fullsized":"2024-09-13T15:03:06.552862+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:49:21.722939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.722939+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:11:39.620088+0000","scrub_duration":0.0016642790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"1337'3562","reported_seq":3589,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:23.416633+0000","last_change":"2024-09-13T15:02:44.893698+0000","last_active":"2024-09-13T15:03:23.416633+0000","last_peered":"2024-09-13T15:03:23.416633+0000","last_clean":"2024-09-13T15:03:23.416633+0000","last_became_active":"2024-09-13T15:02:44.893386+0000","last_became_peered":"2024-09-13T15:02:44.893386+0000","last_unstale":"2024-09-13T15:03:23.416633+0000","last_undegraded":"2024-09-13T15:03:23.416633+0000","last_fullsized":"2024-09-13T15:03:23.416633+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3562,"log_dups_size":0,"ondisk_log_size":3562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:52.846338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233439232,"num_objects":3562,"num_object_clones":0,"num_object_copies":10686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3562,"num_write_kb":227968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":160,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.931032+0000","last_change":"2024-09-13T15:00:25.054416+0000","last_active":"2024-09-13T15:03:14.931032+0000","last_peered":"2024-09-13T15:03:14.931032+0000","last_clean":"2024-09-13T15:03:14.931032+0000","last_became_active":"2024-09-13T15:00:25.050641+0000","last_became_peered":"2024-09-13T15:00:25.050641+0000","last_unstale":"2024-09-13T15:03:14.931032+0000","last_undegraded":"2024-09-13T15:03:14.931032+0000","last_fullsized":"2024-09-13T15:03:14.931032+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:35:40.107806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":23,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.544178+0000","last_change":"2024-09-13T15:02:42.919448+0000","last_active":"2024-09-13T15:03:06.544178+0000","last_peered":"2024-09-13T15:03:06.544178+0000","last_clean":"2024-09-13T15:03:06.544178+0000","last_became_active":"2024-09-13T15:02:42.889750+0000","last_became_peered":"2024-09-13T15:02:42.889750+0000","last_unstale":"2024-09-13T15:03:06.544178+0000","last_undegraded":"2024-09-13T15:03:06.544178+0000","last_fullsized":"2024-09-13T15:03:06.544178+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:25.834089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"261'100","reported_seq":425,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.937882+0000","last_change":"2024-09-13T15:00:25.213778+0000","last_active":"2024-09-13T15:03:14.937882+0000","last_peered":"2024-09-13T15:03:14.937882+0000","last_clean":"2024-09-13T15:03:14.937882+0000","last_became_active":"2024-09-13T15:00:25.047003+0000","last_became_peered":"2024-09-13T15:00:25.047003+0000","last_unstale":"2024-09-13T15:03:14.937882+0000","last_undegraded":"2024-09-13T15:03:14.937882+0000","last_fullsized":"2024-09-13T15:03:14.937882+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.597568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.597568+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:20.589087+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"1337'3913","reported_seq":3929,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:27.385253+0000","last_change":"2024-09-13T15:02:44.890783+0000","last_active":"2024-09-13T15:03:27.385253+0000","last_peered":"2024-09-13T15:03:27.385253+0000","last_clean":"2024-09-13T15:03:27.385253+0000","last_became_active":"2024-09-13T15:02:44.890448+0000","last_became_peered":"2024-09-13T15:02:44.890448+0000","last_unstale":"2024-09-13T15:03:27.385253+0000","last_undegraded":"2024-09-13T15:03:27.385253+0000","last_fullsized":"2024-09-13T15:03:27.385253+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3913,"log_dups_size":0,"ondisk_log_size":3913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:58.783856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256442368,"num_objects":3913,"num_object_clones":0,"num_object_copies":11739,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3913,"num_write_kb":250432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":140,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552013+0000","last_change":"2024-09-13T15:02:42.885662+0000","last_active":"2024-09-13T15:03:06.552013+0000","last_peered":"2024-09-13T15:03:06.552013+0000","last_clean":"2024-09-13T15:03:06.552013+0000","last_became_active":"2024-09-13T15:02:42.885326+0000","last_became_peered":"2024-09-13T15:02:42.885326+0000","last_unstale":"2024-09-13T15:03:06.552013+0000","last_undegraded":"2024-09-13T15:03:06.552013+0000","last_fullsized":"2024-09-13T15:03:06.552013+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:11.631110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":23,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.544583+0000","last_change":"2024-09-13T15:02:42.917286+0000","last_active":"2024-09-13T15:03:06.544583+0000","last_peered":"2024-09-13T15:03:06.544583+0000","last_clean":"2024-09-13T15:03:06.544583+0000","last_became_active":"2024-09-13T15:02:42.888048+0000","last_became_peered":"2024-09-13T15:02:42.888048+0000","last_unstale":"2024-09-13T15:03:06.544583+0000","last_undegraded":"2024-09-13T15:03:06.544583+0000","last_fullsized":"2024-09-13T15:03:06.544583+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:23:37.915522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"261'100","reported_seq":19,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.542765+0000","last_change":"2024-09-13T15:02:42.890854+0000","last_active":"2024-09-13T15:03:06.542765+0000","last_peered":"2024-09-13T15:03:06.542765+0000","last_clean":"2024-09-13T15:03:06.542765+0000","last_became_active":"2024-09-13T15:02:42.889982+0000","last_became_peered":"2024-09-13T15:02:42.889982+0000","last_unstale":"2024-09-13T15:03:06.542765+0000","last_undegraded":"2024-09-13T15:03:06.542765+0000","last_fullsized":"2024-09-13T15:03:06.542765+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.191326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.191326+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:21:40.889165+0000","scrub_duration":0.0025317360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1382,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.546049+0000","last_change":"2024-09-13T15:00:05.058713+0000","last_active":"2024-09-13T15:03:06.546049+0000","last_peered":"2024-09-13T15:03:06.546049+0000","last_clean":"2024-09-13T15:03:06.546049+0000","last_became_active":"2024-09-13T15:00:05.034868+0000","last_became_peered":"2024-09-13T15:00:05.034868+0000","last_unstale":"2024-09-13T15:03:06.546049+0000","last_undegraded":"2024-09-13T15:03:06.546049+0000","last_fullsized":"2024-09-13T15:03:06.546049+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:32:10.399295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":423,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.938202+0000","last_change":"2024-09-13T15:02:42.899236+0000","last_active":"2024-09-13T15:03:14.938202+0000","last_peered":"2024-09-13T15:03:14.938202+0000","last_clean":"2024-09-13T15:03:14.938202+0000","last_became_active":"2024-09-13T15:02:42.897548+0000","last_became_peered":"2024-09-13T15:02:42.897548+0000","last_unstale":"2024-09-13T15:03:14.938202+0000","last_undegraded":"2024-09-13T15:03:14.938202+0000","last_fullsized":"2024-09-13T15:03:14.938202+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:04:04.344226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"261'100","reported_seq":110,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552305+0000","last_change":"2024-09-13T14:55:41.889497+0000","last_active":"2024-09-13T15:03:06.552305+0000","last_peered":"2024-09-13T15:03:06.552305+0000","last_clean":"2024-09-13T15:03:06.552305+0000","last_became_active":"2024-09-13T14:55:41.888766+0000","last_became_peered":"2024-09-13T14:55:41.888766+0000","last_unstale":"2024-09-13T15:03:06.552305+0000","last_undegraded":"2024-09-13T15:03:06.552305+0000","last_fullsized":"2024-09-13T15:03:06.552305+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:55.378404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:55.378404+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:50.289550+0000","scrub_duration":0.0016008439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.542222+0000","last_change":"2024-09-13T15:02:42.894522+0000","last_active":"2024-09-13T15:03:06.542222+0000","last_peered":"2024-09-13T15:03:06.542222+0000","last_clean":"2024-09-13T15:03:06.542222+0000","last_became_active":"2024-09-13T15:02:42.893100+0000","last_became_peered":"2024-09-13T15:02:42.893100+0000","last_unstale":"2024-09-13T15:03:06.542222+0000","last_undegraded":"2024-09-13T15:03:06.542222+0000","last_fullsized":"2024-09-13T15:03:06.542222+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:25.582063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2071,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.938900+0000","last_change":"2024-09-13T15:02:42.901311+0000","last_active":"2024-09-13T15:03:14.938900+0000","last_peered":"2024-09-13T15:03:14.938900+0000","last_clean":"2024-09-13T15:03:14.938900+0000","last_became_active":"2024-09-13T15:02:42.900962+0000","last_became_peered":"2024-09-13T15:02:42.900962+0000","last_unstale":"2024-09-13T15:03:14.938900+0000","last_undegraded":"2024-09-13T15:03:14.938900+0000","last_fullsized":"2024-09-13T15:03:14.938900+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:23:04.122484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"261'100","reported_seq":257,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.928896+0000","last_change":"2024-09-13T15:00:25.087759+0000","last_active":"2024-09-13T15:03:14.928896+0000","last_peered":"2024-09-13T15:03:14.928896+0000","last_clean":"2024-09-13T15:03:14.928896+0000","last_became_active":"2024-09-13T15:00:25.087315+0000","last_became_peered":"2024-09-13T15:00:25.087315+0000","last_unstale":"2024-09-13T15:03:14.928896+0000","last_undegraded":"2024-09-13T15:03:14.928896+0000","last_fullsized":"2024-09-13T15:03:14.928896+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:19.978464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:19.978464+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:22:19.789250+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.542749+0000","last_change":"2024-09-13T15:02:42.894144+0000","last_active":"2024-09-13T15:03:06.542749+0000","last_peered":"2024-09-13T15:03:06.542749+0000","last_clean":"2024-09-13T15:03:06.542749+0000","last_became_active":"2024-09-13T15:02:42.893679+0000","last_became_peered":"2024-09-13T15:02:42.893679+0000","last_unstale":"2024-09-13T15:03:06.542749+0000","last_undegraded":"2024-09-13T15:03:06.542749+0000","last_fullsized":"2024-09-13T15:03:06.542749+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:05.614138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":130,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.552385+0000","last_change":"2024-09-13T15:02:42.886038+0000","last_active":"2024-09-13T15:03:06.552385+0000","last_peered":"2024-09-13T15:03:06.552385+0000","last_clean":"2024-09-13T15:03:06.552385+0000","last_became_active":"2024-09-13T15:02:42.884438+0000","last_became_peered":"2024-09-13T15:02:42.884438+0000","last_unstale":"2024-09-13T15:03:06.552385+0000","last_undegraded":"2024-09-13T15:03:06.552385+0000","last_fullsized":"2024-09-13T15:03:06.552385+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:47:39.175275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"261'100","reported_seq":140,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.551936+0000","last_change":"2024-09-13T15:02:42.885131+0000","last_active":"2024-09-13T15:03:06.551936+0000","last_peered":"2024-09-13T15:03:06.551936+0000","last_clean":"2024-09-13T15:03:06.551936+0000","last_became_active":"2024-09-13T15:02:42.883904+0000","last_became_peered":"2024-09-13T15:02:42.883904+0000","last_unstale":"2024-09-13T15:03:06.551936+0000","last_undegraded":"2024-09-13T15:03:06.551936+0000","last_fullsized":"2024-09-13T15:03:06.551936+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:50.327393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:50.327393+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:11:05.603544+0000","scrub_duration":0.0011508309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":25,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.544040+0000","last_change":"2024-09-13T15:02:42.906500+0000","last_active":"2024-09-13T15:03:06.544040+0000","last_peered":"2024-09-13T15:03:06.544040+0000","last_clean":"2024-09-13T15:03:06.544040+0000","last_became_active":"2024-09-13T15:02:42.879187+0000","last_became_peered":"2024-09-13T15:02:42.879187+0000","last_unstale":"2024-09-13T15:03:06.544040+0000","last_undegraded":"2024-09-13T15:03:06.544040+0000","last_fullsized":"2024-09-13T15:03:06.544040+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:19.991287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2030,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-13T15:03:14.937737+0000","last_change":"2024-09-13T15:02:42.896774+0000","last_active":"2024-09-13T15:03:14.937737+0000","last_peered":"2024-09-13T15:03:14.937737+0000","last_clean":"2024-09-13T15:03:14.937737+0000","last_became_active":"2024-09-13T15:02:42.896459+0000","last_became_peered":"2024-09-13T15:02:42.896459+0000","last_unstale":"2024-09-13T15:03:14.937737+0000","last_undegraded":"2024-09-13T15:03:14.937737+0000","last_fullsized":"2024-09-13T15:03:14.937737+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:35.279246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"261'100","reported_seq":658,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-13T15:03:06.551995+0000","last_change":"2024-09-13T15:00:05.038303+0000","last_active":"2024-09-13T15:03:06.551995+0000","last_peered":"2024-09-13T15:03:06.551995+0000","last_clean":"2024-09-13T15:03:06.551995+0000","last_became_active":"2024-09-13T15:00:05.037979+0000","last_became_peered":"2024-09-13T15:00:05.037979+0000","last_unstale":"2024-09-13T15:03:06.551995+0000","last_undegraded":"2024-09-13T15:03:06.551995+0000","last_fullsized":"2024-09-13T15:03:06.551995+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:14.654603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:14.654603+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:28.168890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":3942514688,"num_objects":60158,"num_object_clones":0,"num_object_copies":180474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60158,"num_write_kb":3850112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12014583808,"data_stored":12014583808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60158,"ondisk_log_size":60158,"up":48,"acting":48,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":17228,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":22162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2033792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2437120,"data_stored":2426464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430761,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3126268,"kb_used_data":2231284,"kb_used_omap":1,"kb_used_meta":894910,"kb_avail":90618884,"statfs":{"total":95995035648,"available":92793737216,"internally_reserved":0,"allocated":2284834816,"data_stored":2281791076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":916388184},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626990,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2888188,"kb_used_data":1985460,"kb_used_omap":1,"kb_used_meta":902718,"kb_avail":90856964,"statfs":{"total":95995035648,"available":93037531136,"internally_reserved":0,"allocated":2033111040,"data_stored":2030067323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":924383576},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823219,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2376624,"kb_used_data":1485416,"kb_used_omap":1,"kb_used_meta":891198,"kb_avail":91368528,"statfs":{"total":95995035648,"available":93561372672,"internally_reserved":0,"allocated":1521065984,"data_stored":1518030253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":912587096},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019447,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059593104,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2693536,"kb_used_data":1843800,"kb_used_omap":26,"kb_used_meta":849701,"kb_avail":91051616,"statfs":{"total":95995035648,"available":93236854784,"internally_reserved":0,"allocated":1888051200,"data_stored":1885030730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":870093984},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":591,"seq":2538325672342,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2545588,"kb_used_data":1589804,"kb_used_omap":41,"kb_used_meta":955734,"kb_avail":91199564,"statfs":{"total":95995035648,"available":93388353536,"internally_reserved":0,"allocated":1627959296,"data_stored":1624924148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42651,"internal_metadata":978671973},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":557,"seq":2392296784286,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3395620,"kb_used_data":2471644,"kb_used_omap":32,"kb_used_meta":923935,"kb_avail":90349532,"statfs":{"total":95995035648,"available":92517920768,"internally_reserved":0,"allocated":2530963456,"data_stored":2527938841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33752,"internal_metadata":946109480},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":2250562863524,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2578055168,"data_stored":2578055168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1641414656,"data_stored":1641414656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1887305728,"data_stored":1887305728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1538719744,"data_stored":1538719744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2060124160,"data_stored":2060124160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2308964352,"data_stored":2308964352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:03:31.440 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T15:03:31.440 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:03:31.441 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [2, 3, 6, 7, 1, 5] 2024-09-13T15:03:31.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd out 2 2024-09-13T15:03:31.479 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:31.508 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:31.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:31.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:31.609 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:31.639 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:31.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:31.640 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:31.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:31.741 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:31.767 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:31.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:31.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:31.869 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:31.901 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:31.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:31.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:32.003 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:32.039 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:32.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:32.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:32.101 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:32.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:32.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:32.140 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:32.177 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 37 16 68406 68390 115.509 79.5 0.0178157 0.00865172 2024-09-13T15:03:32.185 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:32.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:32.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:32.287 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:32.313 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:32.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:32.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:32.414 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:32.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:32.454 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:32.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:32.477 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T15:03:32.555 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:32.599 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:32.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:32.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:32.701 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:32.736 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:32.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:32.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:32.838 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:32.877 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:32.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:32.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:32.951 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:03:32.977 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-13T15:03:32.978 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:33.019 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:33.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:33.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:33.121 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:33.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:33.161 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:33.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:33.177 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 38 15 69725 69710 114.638 82.5 0.0130222 0.00871773 2024-09-13T15:03:33.258 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:32 smithi184 ceph-mon[92527]: pgmap v2266: 119 pgs: 119 active+clean; 3.9 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-13T15:03:33.258 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:32 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/732628562' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:03:33.258 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:32 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:03:33.258 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:03:33.258 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:03:33.262 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:33.314 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:33.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:33.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:33.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:32 smithi151 ceph-mon[104450]: pgmap v2266: 119 pgs: 119 active+clean; 3.9 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-13T15:03:33.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:32 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/732628562' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:03:33.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:32 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:03:33.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:03:33.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:03:33.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:32 smithi151 ceph-mon[108966]: pgmap v2266: 119 pgs: 119 active+clean; 3.9 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-13T15:03:33.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:32 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/732628562' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:03:33.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:32 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:03:33.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:03:33.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:03:33.416 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:33.448 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:33.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:33.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:33.550 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:33.588 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:33.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:33.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:33.690 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:33.724 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:33.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:33.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:33.826 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:33.860 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:33.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:33.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:33.962 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:34.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:34.010 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:34.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:34.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:33 smithi151 ceph-mon[104450]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:03:34.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:33 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:03:34.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:33 smithi151 ceph-mon[104450]: osdmap e1338: 8 total, 8 up, 6 in 2024-09-13T15:03:34.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:33 smithi151 ceph-mon[108966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:03:34.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:33 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:03:34.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:33 smithi151 ceph-mon[108966]: osdmap e1338: 8 total, 8 up, 6 in 2024-09-13T15:03:34.111 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:34.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:34.143 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:34.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:34.177 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 39 16 71113 71097 113.922 86.6875 0.0126326 0.00877304 2024-09-13T15:03:34.244 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:34.245 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:33 smithi184 ceph-mon[92527]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:03:34.245 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:33 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:03:34.245 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:33 smithi184 ceph-mon[92527]: osdmap e1338: 8 total, 8 up, 6 in 2024-09-13T15:03:34.282 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:34.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:34.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:34.384 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:34.410 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:34.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:34.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:34.446 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:34.511 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:34.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:34.557 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:34.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:34.658 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:34.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:34.692 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:34.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:34.720 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:03:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:03:34] "GET /metrics HTTP/1.1" 200 36700 "" "Prometheus/2.51.0" 2024-09-13T15:03:34.793 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:34.832 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:34.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:34.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:34.918 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T15:03:34.919 INFO:teuthology.orchestra.run.smithi184.stdout: "size": 20, 2024-09-13T15:03:34.919 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 600, 2024-09-13T15:03:34.919 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [ 2024-09-13T15:03:34.919 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.919 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:67680 9.e 9:71f21820:::benchmark_data_smithi151_261954_object67679:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1227)", 2024-09-13T15:03:34.919 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:54:57.807154+0000", 2024-09-13T15:03:34.919 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 517.10750345099996, 2024-09-13T15:03:34.919 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.2224513259999998, 2024-09-13T15:03:34.919 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:03:34.920 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:34.920 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:03:34.920 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T15:03:34.920 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:03:34.920 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 67680 2024-09-13T15:03:34.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.920 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:03:34.920 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.920 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:03:34.920 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.807154+0000", 2024-09-13T15:03:34.920 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:34.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.920 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.921 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:03:34.921 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.807154+0000", 2024-09-13T15:03:34.921 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:34.921 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.921 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.921 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:03:34.921 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.807152+0000", 2024-09-13T15:03:34.921 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T15:03:34.921 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.921 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.922 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:03:34.922 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.807195+0000", 2024-09-13T15:03:34.922 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.2524000000000001e-05 2024-09-13T15:03:34.922 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.922 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.922 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:03:34.922 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.807206+0000", 2024-09-13T15:03:34.922 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1532000000000001e-05 2024-09-13T15:03:34.922 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.922 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.923 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:03:34.923 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.807212+0000", 2024-09-13T15:03:34.923 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6280000000000002e-06 2024-09-13T15:03:34.923 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.923 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.923 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:34.923 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.807438+0000", 2024-09-13T15:03:34.923 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000226221 2024-09-13T15:03:34.923 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.923 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.923 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:03:34.923 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.807448+0000", 2024-09-13T15:03:34.923 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0244999999999999e-05 2024-09-13T15:03:34.924 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.924 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.924 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:34.924 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:58.808417+0000", 2024-09-13T15:03:34.924 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0009687190000001 2024-09-13T15:03:34.924 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.924 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.924 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:03:34.924 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:58.808429+0000", 2024-09-13T15:03:34.924 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1841000000000001e-05 2024-09-13T15:03:34.924 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.924 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.925 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:34.925 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.001531+0000", 2024-09-13T15:03:34.925 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.193101934 2024-09-13T15:03:34.925 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.925 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.925 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:03:34.925 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.001691+0000", 2024-09-13T15:03:34.925 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00016000399999999999 2024-09-13T15:03:34.925 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.925 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.925 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,4", 2024-09-13T15:03:34.925 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.001875+0000", 2024-09-13T15:03:34.926 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000184305 2024-09-13T15:03:34.926 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.926 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.926 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:03:34.926 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.011365+0000", 2024-09-13T15:03:34.926 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0094901959999999994 2024-09-13T15:03:34.926 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.926 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.926 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:34.926 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.025618+0000", 2024-09-13T15:03:34.926 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.014252783 2024-09-13T15:03:34.926 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.926 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.926 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:34.927 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.026026+0000", 2024-09-13T15:03:34.927 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00040753499999999999 2024-09-13T15:03:34.927 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.927 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.927 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:03:34.927 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.026062+0000", 2024-09-13T15:03:34.927 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.6086e-05 2024-09-13T15:03:34.927 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.928 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.928 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:03:34.928 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.029605+0000", 2024-09-13T15:03:34.928 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0035434500000000001 2024-09-13T15:03:34.928 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:34.928 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:03:34.929 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:34.929 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.929 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.929 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:67682 9.e 9:7008a8c6:::benchmark_data_smithi151_261954_object67681:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1227)", 2024-09-13T15:03:34.929 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:54:57.807224+0000", 2024-09-13T15:03:34.929 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 517.10743308099995, 2024-09-13T15:03:34.930 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.2258236779999998, 2024-09-13T15:03:34.930 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:03:34.930 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:34.930 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:03:34.930 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T15:03:34.930 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:03:34.931 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 67682 2024-09-13T15:03:34.931 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.931 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:03:34.931 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.931 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:03:34.931 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.807224+0000", 2024-09-13T15:03:34.931 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:34.932 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.932 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.932 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:03:34.932 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.807224+0000", 2024-09-13T15:03:34.932 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:34.932 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.933 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.933 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:03:34.933 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.807224+0000", 2024-09-13T15:03:34.933 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T15:03:34.933 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.933 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.933 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:03:34.933 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.807258+0000", 2024-09-13T15:03:34.933 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3748000000000003e-05 2024-09-13T15:03:34.934 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.934 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.934 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:03:34.934 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.807264+0000", 2024-09-13T15:03:34.934 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7769999999999997e-06 2024-09-13T15:03:34.934 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.934 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.934 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:03:34.934 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.807268+0000", 2024-09-13T15:03:34.934 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.7790000000000002e-06 2024-09-13T15:03:34.934 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.934 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.934 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:34.935 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.807475+0000", 2024-09-13T15:03:34.935 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000207281 2024-09-13T15:03:34.935 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.935 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.935 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:03:34.935 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.807481+0000", 2024-09-13T15:03:34.935 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6899999999999997e-06 2024-09-13T15:03:34.935 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.935 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.935 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:34.935 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:58.808450+0000", 2024-09-13T15:03:34.935 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.000968997 2024-09-13T15:03:34.935 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.935 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.936 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:03:34.936 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:58.808457+0000", 2024-09-13T15:03:34.936 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.075e-06 2024-09-13T15:03:34.936 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.936 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.936 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:34.936 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.004214+0000", 2024-09-13T15:03:34.936 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.19575666 2024-09-13T15:03:34.936 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.936 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.936 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:03:34.936 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.004321+0000", 2024-09-13T15:03:34.936 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000106883 2024-09-13T15:03:34.937 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.937 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.937 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,4", 2024-09-13T15:03:34.937 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.004434+0000", 2024-09-13T15:03:34.937 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000113786 2024-09-13T15:03:34.937 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.937 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.937 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:34.939 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:03:34.939 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.011374+0000", 2024-09-13T15:03:34.939 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0069399520000000001 2024-09-13T15:03:34.939 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.939 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.939 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:34.939 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.025654+0000", 2024-09-13T15:03:34.939 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.014279828 2024-09-13T15:03:34.939 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.939 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.940 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:34.940 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.029665+0000", 2024-09-13T15:03:34.940 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0040107149999999998 2024-09-13T15:03:34.940 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.940 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.940 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:03:34.940 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.029724+0000", 2024-09-13T15:03:34.940 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.9357000000000001e-05 2024-09-13T15:03:34.940 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.940 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.940 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:03:34.940 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.033048+0000", 2024-09-13T15:03:34.941 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0033236390000000002 2024-09-13T15:03:34.941 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:34.941 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:03:34.941 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:34.941 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.941 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.941 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:67685 9.e 9:78d976bf:::benchmark_data_smithi151_261954_object67684:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1228)", 2024-09-13T15:03:34.941 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:54:57.823734+0000", 2024-09-13T15:03:34.941 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 517.09092292000003, 2024-09-13T15:03:34.941 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.2130326760000001, 2024-09-13T15:03:34.941 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:03:34.941 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:34.942 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:03:34.942 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T15:03:34.942 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:03:34.942 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 67685 2024-09-13T15:03:34.942 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.942 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:03:34.942 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.942 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:03:34.942 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.823734+0000", 2024-09-13T15:03:34.942 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:34.942 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.942 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.942 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:03:34.943 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.823734+0000", 2024-09-13T15:03:34.943 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:34.943 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.943 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.943 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:03:34.943 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.823730+0000", 2024-09-13T15:03:34.943 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999962 2024-09-13T15:03:34.944 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.944 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.944 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:03:34.944 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.823779+0000", 2024-09-13T15:03:34.944 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8056000000000002e-05 2024-09-13T15:03:34.945 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.945 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.945 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:03:34.945 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.823796+0000", 2024-09-13T15:03:34.945 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.7493e-05 2024-09-13T15:03:34.946 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.946 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.946 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:03:34.946 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.823803+0000", 2024-09-13T15:03:34.946 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.6150000000000002e-06 2024-09-13T15:03:34.947 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.947 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.947 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:34.947 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.823853+0000", 2024-09-13T15:03:34.947 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0504999999999998e-05 2024-09-13T15:03:34.947 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.948 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.948 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:03:34.948 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.823862+0000", 2024-09-13T15:03:34.948 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.2289999999999992e-06 2024-09-13T15:03:34.948 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.948 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.949 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:34.949 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:58.808474+0000", 2024-09-13T15:03:34.949 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.98461200199999999 2024-09-13T15:03:34.949 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.950 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.950 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:03:34.950 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:58.808480+0000", 2024-09-13T15:03:34.950 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5840000000000002e-06 2024-09-13T15:03:34.950 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.951 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.951 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:34.951 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.006200+0000", 2024-09-13T15:03:34.951 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.19771995 2024-09-13T15:03:34.951 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.951 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.952 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:03:34.952 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.006286+0000", 2024-09-13T15:03:34.952 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.5704999999999993e-05 2024-09-13T15:03:34.952 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.952 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.952 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,4", 2024-09-13T15:03:34.952 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.006399+0000", 2024-09-13T15:03:34.953 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000113408 2024-09-13T15:03:34.953 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.953 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.953 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:03:34.953 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.015211+0000", 2024-09-13T15:03:34.954 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0088117549999999992 2024-09-13T15:03:34.954 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.954 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.954 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:34.954 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.025723+0000", 2024-09-13T15:03:34.954 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.010511866 2024-09-13T15:03:34.955 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.955 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.955 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:34.955 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.033191+0000", 2024-09-13T15:03:34.955 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0074680440000000001 2024-09-13T15:03:34.955 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.955 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.955 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:03:34.956 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.033239+0000", 2024-09-13T15:03:34.956 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8704e-05 2024-09-13T15:03:34.956 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.956 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.956 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:03:34.956 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.036767+0000", 2024-09-13T15:03:34.956 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0035277559999999999 2024-09-13T15:03:34.957 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:34.957 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:03:34.957 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:34.957 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.957 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.957 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:67687 9.e 9:7c15d330:::benchmark_data_smithi151_261954_object67686:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1228)", 2024-09-13T15:03:34.958 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:54:57.826782+0000", 2024-09-13T15:03:34.958 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 517.08787543400001, 2024-09-13T15:03:34.958 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.213488618, 2024-09-13T15:03:34.958 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:03:34.958 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:34.958 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:03:34.958 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T15:03:34.959 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:03:34.959 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 67687 2024-09-13T15:03:34.959 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.959 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:03:34.959 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.959 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:03:34.960 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.826782+0000", 2024-09-13T15:03:34.960 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:34.960 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.960 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.960 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:03:34.961 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.826782+0000", 2024-09-13T15:03:34.961 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:34.961 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.961 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.961 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:03:34.961 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.826780+0000", 2024-09-13T15:03:34.961 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T15:03:34.962 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.962 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.962 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:03:34.962 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.826813+0000", 2024-09-13T15:03:34.962 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.2799000000000002e-05 2024-09-13T15:03:34.962 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.962 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.963 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:03:34.963 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.826825+0000", 2024-09-13T15:03:34.963 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2281e-05 2024-09-13T15:03:34.963 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.963 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.963 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:03:34.964 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.826832+0000", 2024-09-13T15:03:34.964 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.5429999999999999e-06 2024-09-13T15:03:34.964 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.964 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.964 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:34.965 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.827190+0000", 2024-09-13T15:03:34.965 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000357671 2024-09-13T15:03:34.965 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.965 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.965 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:03:34.965 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.827200+0000", 2024-09-13T15:03:34.966 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0180000000000001e-05 2024-09-13T15:03:34.966 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.966 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.966 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:34.966 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:58.808496+0000", 2024-09-13T15:03:34.966 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.98129594499999995 2024-09-13T15:03:34.966 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.967 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.967 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:03:34.967 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:58.808501+0000", 2024-09-13T15:03:34.967 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3800000000000002e-06 2024-09-13T15:03:34.967 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.967 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.967 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:34.968 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.008043+0000", 2024-09-13T15:03:34.968 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1995423080000001 2024-09-13T15:03:34.968 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.968 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.968 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:03:34.968 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.008187+0000", 2024-09-13T15:03:34.968 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014386800000000001 2024-09-13T15:03:34.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.969 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.969 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,4", 2024-09-13T15:03:34.969 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.008361+0000", 2024-09-13T15:03:34.969 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00017340199999999999 2024-09-13T15:03:34.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.970 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.970 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:03:34.970 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.015221+0000", 2024-09-13T15:03:34.970 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0068602369999999999 2024-09-13T15:03:34.970 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.970 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.971 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:34.971 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.025769+0000", 2024-09-13T15:03:34.971 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.010547677 2024-09-13T15:03:34.971 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.971 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.971 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:34.972 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.036845+0000", 2024-09-13T15:03:34.972 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.011076053000000001 2024-09-13T15:03:34.972 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.972 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.972 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:03:34.972 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.036971+0000", 2024-09-13T15:03:34.973 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012642799999999999 2024-09-13T15:03:34.973 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.973 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.973 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:03:34.973 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.040271+0000", 2024-09-13T15:03:34.974 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0032995490000000001 2024-09-13T15:03:34.974 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:34.974 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:03:34.974 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:34.974 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.974 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.975 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:67711 9.e 9:7beb6016:::benchmark_data_smithi151_261954_object67710:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1228)", 2024-09-13T15:03:34.975 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:54:57.850445+0000", 2024-09-13T15:03:34.975 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 517.06421239199994, 2024-09-13T15:03:34.975 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.1933325720000001, 2024-09-13T15:03:34.975 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:03:34.975 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:34.976 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:03:34.976 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T15:03:34.976 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:03:34.976 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 67711 2024-09-13T15:03:34.976 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.977 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:03:34.977 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.977 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:03:34.977 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.850445+0000", 2024-09-13T15:03:34.977 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:34.977 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.978 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.978 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:03:34.978 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.850445+0000", 2024-09-13T15:03:34.978 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:34.978 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.979 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.979 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:03:34.979 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.850444+0000", 2024-09-13T15:03:34.979 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:03:34.979 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.979 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.980 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:03:34.980 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.850544+0000", 2024-09-13T15:03:34.980 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000100169 2024-09-13T15:03:34.980 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.980 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.980 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:03:34.980 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.850555+0000", 2024-09-13T15:03:34.980 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0421000000000001e-05 2024-09-13T15:03:34.980 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.981 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.981 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:03:34.981 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.850559+0000", 2024-09-13T15:03:34.981 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.2390000000000004e-06 2024-09-13T15:03:34.981 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.981 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.981 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:34.981 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.850601+0000", 2024-09-13T15:03:34.981 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1907999999999999e-05 2024-09-13T15:03:34.981 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.982 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.982 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:03:34.982 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.850607+0000", 2024-09-13T15:03:34.982 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8250000000000001e-06 2024-09-13T15:03:34.982 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.982 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.982 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:34.982 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:58.808518+0000", 2024-09-13T15:03:34.982 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.95791166599999999 2024-09-13T15:03:34.982 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.982 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.982 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:03:34.982 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:58.808524+0000", 2024-09-13T15:03:34.983 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6380000000000001e-06 2024-09-13T15:03:34.983 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.983 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.983 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:34.983 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.009820+0000", 2024-09-13T15:03:34.983 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2012963320000001 2024-09-13T15:03:34.983 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.983 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.983 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:03:34.983 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.009919+0000", 2024-09-13T15:03:34.983 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.8408999999999999e-05 2024-09-13T15:03:34.983 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.983 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.983 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,4", 2024-09-13T15:03:34.984 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.010015+0000", 2024-09-13T15:03:34.984 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.6385999999999995e-05 2024-09-13T15:03:34.984 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.984 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.984 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:03:34.984 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.019631+0000", 2024-09-13T15:03:34.984 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0096156520000000006 2024-09-13T15:03:34.984 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.984 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.984 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:34.984 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.025813+0000", 2024-09-13T15:03:34.985 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0061824280000000002 2024-09-13T15:03:34.985 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.985 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.985 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:34.985 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.040710+0000", 2024-09-13T15:03:34.985 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.014896766000000001 2024-09-13T15:03:34.985 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.985 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.985 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:03:34.985 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.040768+0000", 2024-09-13T15:03:34.985 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7790000000000001e-05 2024-09-13T15:03:34.985 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.985 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.986 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:03:34.986 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.043778+0000", 2024-09-13T15:03:34.986 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0030098590000000001 2024-09-13T15:03:34.986 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:34.986 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:03:34.986 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:34.986 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.986 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.986 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:67714 9.e 9:7ef66a5e:::benchmark_data_smithi151_261954_object67713:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1228)", 2024-09-13T15:03:34.986 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:54:57.854719+0000", 2024-09-13T15:03:34.986 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 517.059938794, 2024-09-13T15:03:34.986 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.193210117, 2024-09-13T15:03:34.987 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:03:34.987 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:34.987 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:03:34.987 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T15:03:34.987 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:03:34.987 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 67714 2024-09-13T15:03:34.987 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.987 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:03:34.987 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.987 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:03:34.987 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.854719+0000", 2024-09-13T15:03:34.987 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:34.987 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.988 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.988 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:03:34.988 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.854719+0000", 2024-09-13T15:03:34.988 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:34.988 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.988 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.988 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:03:34.988 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.854711+0000", 2024-09-13T15:03:34.988 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999928 2024-09-13T15:03:34.988 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.988 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.989 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:03:34.989 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.854768+0000", 2024-09-13T15:03:34.989 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6328e-05 2024-09-13T15:03:34.989 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.989 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.989 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:03:34.989 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.854781+0000", 2024-09-13T15:03:34.989 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2965999999999999e-05 2024-09-13T15:03:34.989 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.989 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.989 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:03:34.989 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.854786+0000", 2024-09-13T15:03:34.989 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5890000000000002e-06 2024-09-13T15:03:34.990 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.990 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.990 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:34.990 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.854829+0000", 2024-09-13T15:03:34.990 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3043000000000001e-05 2024-09-13T15:03:34.990 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.990 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.990 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:03:34.991 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.854836+0000", 2024-09-13T15:03:34.991 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.9399999999999996e-06 2024-09-13T15:03:34.991 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.991 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.991 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:34.992 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:58.808540+0000", 2024-09-13T15:03:34.992 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.95370359699999996 2024-09-13T15:03:34.992 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.992 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.992 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:03:34.993 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:58.808545+0000", 2024-09-13T15:03:34.993 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4589999999999997e-06 2024-09-13T15:03:34.993 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.993 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.993 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:34.993 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.011408+0000", 2024-09-13T15:03:34.994 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2028624210000001 2024-09-13T15:03:34.994 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.994 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.994 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:03:34.994 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.011497+0000", 2024-09-13T15:03:34.994 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.9580999999999995e-05 2024-09-13T15:03:34.994 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.994 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.994 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,4", 2024-09-13T15:03:34.995 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.011607+0000", 2024-09-13T15:03:34.995 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000109558 2024-09-13T15:03:34.995 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.995 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.995 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:03:34.995 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.019635+0000", 2024-09-13T15:03:34.995 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0080281199999999997 2024-09-13T15:03:34.996 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.996 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.996 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:34.996 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.025853+0000", 2024-09-13T15:03:34.996 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0062184110000000001 2024-09-13T15:03:34.996 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.996 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.996 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:34.996 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.044382+0000", 2024-09-13T15:03:34.996 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.018528337999999998 2024-09-13T15:03:34.996 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.996 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.997 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:03:34.997 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.044434+0000", 2024-09-13T15:03:34.997 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.2808e-05 2024-09-13T15:03:34.997 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.997 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.997 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:03:34.997 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.047929+0000", 2024-09-13T15:03:34.997 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0034943370000000001 2024-09-13T15:03:34.997 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:34.997 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:03:34.997 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:34.997 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.997 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.998 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:67725 9.e 9:7431c7f7:::benchmark_data_smithi151_261954_object67724:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1228)", 2024-09-13T15:03:34.998 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:54:57.864300+0000", 2024-09-13T15:03:34.998 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 517.05035777199998, 2024-09-13T15:03:34.998 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.194248177, 2024-09-13T15:03:34.998 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:03:34.998 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:34.998 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:03:34.998 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T15:03:34.998 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:03:34.998 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 67725 2024-09-13T15:03:34.998 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.998 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:03:34.998 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.998 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:03:34.999 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.864300+0000", 2024-09-13T15:03:34.999 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:34.999 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.999 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.999 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:03:34.999 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.864300+0000", 2024-09-13T15:03:34.999 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:34.999 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.999 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:34.999 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:03:34.999 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.864299+0000", 2024-09-13T15:03:34.999 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:03:34.999 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:34.999 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.000 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:03:35.000 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.864354+0000", 2024-09-13T15:03:35.000 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4953000000000001e-05 2024-09-13T15:03:35.000 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.000 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.000 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:03:35.000 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.864365+0000", 2024-09-13T15:03:35.000 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1112e-05 2024-09-13T15:03:35.000 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.000 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.000 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:03:35.000 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.864369+0000", 2024-09-13T15:03:35.000 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.6700000000000002e-06 2024-09-13T15:03:35.000 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.001 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.001 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:35.001 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.864426+0000", 2024-09-13T15:03:35.001 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6397000000000002e-05 2024-09-13T15:03:35.001 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.001 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.001 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:03:35.001 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.864435+0000", 2024-09-13T15:03:35.001 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.2420000000000006e-06 2024-09-13T15:03:35.002 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.002 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.002 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:35.002 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:58.808561+0000", 2024-09-13T15:03:35.002 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.94412544600000003 2024-09-13T15:03:35.002 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.002 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.002 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:03:35.002 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:58.808566+0000", 2024-09-13T15:03:35.002 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5090000000000001e-06 2024-09-13T15:03:35.002 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.002 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.003 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:35.003 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.013234+0000", 2024-09-13T15:03:35.003 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.204668359 2024-09-13T15:03:35.003 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.003 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.003 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:03:35.003 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.013367+0000", 2024-09-13T15:03:35.003 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000132967 2024-09-13T15:03:35.003 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.003 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.003 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,4", 2024-09-13T15:03:35.003 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.013469+0000", 2024-09-13T15:03:35.003 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000101964 2024-09-13T15:03:35.004 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.004 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.004 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:03:35.004 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.022044+0000", 2024-09-13T15:03:35.004 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0085750559999999993 2024-09-13T15:03:35.004 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.004 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.004 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:35.004 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.025906+0000", 2024-09-13T15:03:35.004 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.003861555 2024-09-13T15:03:35.004 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.004 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.004 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:35.005 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.049868+0000", 2024-09-13T15:03:35.005 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.023962508 2024-09-13T15:03:35.005 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.005 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.005 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:03:35.005 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.049931+0000", 2024-09-13T15:03:35.005 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.2267999999999994e-05 2024-09-13T15:03:35.005 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.005 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.005 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:03:35.005 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.058548+0000", 2024-09-13T15:03:35.005 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0086170989999999996 2024-09-13T15:03:35.005 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:35.005 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:03:35.006 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:35.006 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.006 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.006 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:67731 9.e 9:70f25959:::benchmark_data_smithi151_261954_object67730:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1228)", 2024-09-13T15:03:35.007 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:54:57.869588+0000", 2024-09-13T15:03:35.007 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 517.04506905000005, 2024-09-13T15:03:35.007 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.1988707280000002, 2024-09-13T15:03:35.007 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:03:35.007 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:35.008 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:03:35.008 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T15:03:35.008 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:03:35.008 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 67731 2024-09-13T15:03:35.008 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.008 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:03:35.008 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.008 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:03:35.008 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.869588+0000", 2024-09-13T15:03:35.008 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:35.008 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.008 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.008 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:03:35.008 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.869588+0000", 2024-09-13T15:03:35.009 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:35.009 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.009 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.009 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:03:35.009 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.869587+0000", 2024-09-13T15:03:35.009 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T15:03:35.009 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.009 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.009 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:03:35.009 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.869646+0000", 2024-09-13T15:03:35.009 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.9036999999999997e-05 2024-09-13T15:03:35.009 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.009 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.010 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:03:35.010 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.869659+0000", 2024-09-13T15:03:35.010 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3849000000000001e-05 2024-09-13T15:03:35.010 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.010 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.010 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:03:35.010 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.869665+0000", 2024-09-13T15:03:35.010 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.096e-06 2024-09-13T15:03:35.010 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.010 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.010 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:35.010 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.869712+0000", 2024-09-13T15:03:35.010 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7423000000000003e-05 2024-09-13T15:03:35.010 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.011 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.011 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:03:35.011 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.869718+0000", 2024-09-13T15:03:35.011 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6720000000000001e-06 2024-09-13T15:03:35.011 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.011 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.011 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:35.011 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:58.808595+0000", 2024-09-13T15:03:35.011 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.93887735000000005 2024-09-13T15:03:35.011 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.011 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.011 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:03:35.011 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:58.808603+0000", 2024-09-13T15:03:35.011 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.3059999999999998e-06 2024-09-13T15:03:35.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.012 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.012 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:35.012 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.015266+0000", 2024-09-13T15:03:35.012 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.206663118 2024-09-13T15:03:35.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.012 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.012 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:03:35.012 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.015407+0000", 2024-09-13T15:03:35.012 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014032699999999999 2024-09-13T15:03:35.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.012 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.012 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,4", 2024-09-13T15:03:35.013 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.015570+0000", 2024-09-13T15:03:35.013 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000163597 2024-09-13T15:03:35.013 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.013 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.013 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:03:35.013 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.022054+0000", 2024-09-13T15:03:35.013 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0064834200000000002 2024-09-13T15:03:35.013 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.013 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.013 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:35.013 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.025947+0000", 2024-09-13T15:03:35.013 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0038930639999999999 2024-09-13T15:03:35.013 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.013 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.014 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:35.014 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.061451+0000", 2024-09-13T15:03:35.014 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.035504352000000003 2024-09-13T15:03:35.014 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.014 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.014 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:03:35.014 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.061541+0000", 2024-09-13T15:03:35.014 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.9351999999999994e-05 2024-09-13T15:03:35.014 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.014 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.014 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:03:35.014 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.068459+0000", 2024-09-13T15:03:35.014 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0069185669999999996 2024-09-13T15:03:35.014 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:35.015 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:03:35.015 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:35.015 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.015 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.015 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:67734 9.e 9:72252bb7:::benchmark_data_smithi151_261954_object67733:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1228)", 2024-09-13T15:03:35.015 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:54:57.870503+0000", 2024-09-13T15:03:35.015 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 517.04415419899999, 2024-09-13T15:03:35.015 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.2017034880000002, 2024-09-13T15:03:35.015 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:03:35.015 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:35.015 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:03:35.015 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T15:03:35.015 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:03:35.015 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 67734 2024-09-13T15:03:35.016 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.016 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:03:35.016 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.016 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:03:35.016 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.870503+0000", 2024-09-13T15:03:35.016 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:35.016 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.016 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.016 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:03:35.016 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.870503+0000", 2024-09-13T15:03:35.016 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:35.016 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.016 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.017 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:03:35.017 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.870502+0000", 2024-09-13T15:03:35.017 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:03:35.017 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.017 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.017 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:03:35.017 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.870578+0000", 2024-09-13T15:03:35.017 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.5992e-05 2024-09-13T15:03:35.017 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.017 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.017 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:03:35.017 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.870593+0000", 2024-09-13T15:03:35.017 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4248e-05 2024-09-13T15:03:35.017 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.018 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.018 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:03:35.018 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.870598+0000", 2024-09-13T15:03:35.018 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7490000000000004e-06 2024-09-13T15:03:35.018 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.018 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.018 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:35.020 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.870648+0000", 2024-09-13T15:03:35.020 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0176999999999997e-05 2024-09-13T15:03:35.020 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.020 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.020 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:03:35.020 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.870659+0000", 2024-09-13T15:03:35.020 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0385e-05 2024-09-13T15:03:35.020 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.021 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.021 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:35.021 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:58.808677+0000", 2024-09-13T15:03:35.021 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.93801768600000002 2024-09-13T15:03:35.021 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.021 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.021 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:03:35.021 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:58.808685+0000", 2024-09-13T15:03:35.021 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.8060000000000001e-06 2024-09-13T15:03:35.021 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.021 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.021 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:35.021 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.017531+0000", 2024-09-13T15:03:35.021 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2088460649999999 2024-09-13T15:03:35.022 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.022 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.022 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:03:35.022 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.017636+0000", 2024-09-13T15:03:35.022 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000104115 2024-09-13T15:03:35.022 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.022 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.022 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,4", 2024-09-13T15:03:35.022 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.017745+0000", 2024-09-13T15:03:35.022 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00010990200000000001 2024-09-13T15:03:35.022 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.022 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.022 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:03:35.022 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.025560+0000", 2024-09-13T15:03:35.023 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0078146420000000001 2024-09-13T15:03:35.023 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.023 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.023 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:35.023 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.025986+0000", 2024-09-13T15:03:35.023 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00042611500000000002 2024-09-13T15:03:35.023 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.023 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.023 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:35.023 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.068507+0000", 2024-09-13T15:03:35.023 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.042520747999999997 2024-09-13T15:03:35.023 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.023 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.024 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:03:35.024 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.068602+0000", 2024-09-13T15:03:35.024 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.5396999999999995e-05 2024-09-13T15:03:35.024 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.024 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.024 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:03:35.024 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.072207+0000", 2024-09-13T15:03:35.024 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.003604342 2024-09-13T15:03:35.024 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:35.024 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:03:35.024 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:35.024 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.024 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.025 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:67736 9.e 9:7621512f:::benchmark_data_smithi151_261954_object67735:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1228)", 2024-09-13T15:03:35.025 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:54:57.873416+0000", 2024-09-13T15:03:35.025 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 517.04124153099997, 2024-09-13T15:03:35.025 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.2024184130000002, 2024-09-13T15:03:35.025 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:03:35.025 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:35.025 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:03:35.025 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T15:03:35.025 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:03:35.025 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 67736 2024-09-13T15:03:35.025 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.025 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:03:35.025 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.026 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:03:35.026 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.873416+0000", 2024-09-13T15:03:35.026 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:35.026 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.026 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.026 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:03:35.026 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.873416+0000", 2024-09-13T15:03:35.026 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:35.026 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.026 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.026 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:03:35.026 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.873415+0000", 2024-09-13T15:03:35.026 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:03:35.026 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.027 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.027 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:03:35.027 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.873443+0000", 2024-09-13T15:03:35.027 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.7826000000000001e-05 2024-09-13T15:03:35.027 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.027 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.027 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:03:35.027 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.873454+0000", 2024-09-13T15:03:35.027 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0903999999999999e-05 2024-09-13T15:03:35.027 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.027 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.027 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:03:35.027 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.873459+0000", 2024-09-13T15:03:35.027 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0139999999999998e-06 2024-09-13T15:03:35.028 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.028 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.028 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:35.028 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.873509+0000", 2024-09-13T15:03:35.028 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0621999999999997e-05 2024-09-13T15:03:35.028 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.028 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.028 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:03:35.028 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:57.873516+0000", 2024-09-13T15:03:35.028 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7390000000000002e-06 2024-09-13T15:03:35.028 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.028 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.028 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:35.029 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:58.809203+0000", 2024-09-13T15:03:35.029 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.93568732399999999 2024-09-13T15:03:35.029 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.029 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.029 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:03:35.029 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:54:58.809212+0000", 2024-09-13T15:03:35.029 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.2679999999999994e-06 2024-09-13T15:03:35.029 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.029 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.029 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:35.029 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.019672+0000", 2024-09-13T15:03:35.029 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2104602390000001 2024-09-13T15:03:35.029 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.029 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.030 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:03:35.030 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.019824+0000", 2024-09-13T15:03:35.030 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000152249 2024-09-13T15:03:35.030 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.030 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.030 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,4", 2024-09-13T15:03:35.030 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.020015+0000", 2024-09-13T15:03:35.030 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000191406 2024-09-13T15:03:35.030 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.030 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.030 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:03:35.030 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.025671+0000", 2024-09-13T15:03:35.030 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0056558449999999996 2024-09-13T15:03:35.030 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.031 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.031 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:35.031 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.033097+0000", 2024-09-13T15:03:35.031 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.007425699 2024-09-13T15:03:35.031 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.031 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.031 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:35.031 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.072269+0000", 2024-09-13T15:03:35.031 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.039171904 2024-09-13T15:03:35.031 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.031 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.031 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:03:35.031 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.072368+0000", 2024-09-13T15:03:35.031 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.8676000000000002e-05 2024-09-13T15:03:35.032 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.032 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.032 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:03:35.032 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:00.075834+0000", 2024-09-13T15:03:35.032 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0034666670000000001 2024-09-13T15:03:35.032 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:35.032 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:03:35.032 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:35.032 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.032 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.032 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:83722 9.7 9:e6e34c82:::benchmark_data_smithi151_261954_object83721:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1232)", 2024-09-13T15:03:35.032 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:55:15.852701+0000", 2024-09-13T15:03:35.032 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 499.061956243, 2024-09-13T15:03:35.033 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.3784974750000001, 2024-09-13T15:03:35.033 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:03:35.033 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "reached pg", 2024-09-13T15:03:35.033 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:03:35.033 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T15:03:35.033 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:03:35.033 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 83722 2024-09-13T15:03:35.033 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.033 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:03:35.033 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.033 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:03:35.033 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:15.852701+0000", 2024-09-13T15:03:35.033 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:35.034 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.034 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.034 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:03:35.034 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:15.852701+0000", 2024-09-13T15:03:35.034 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:35.034 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.034 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.034 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:03:35.034 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:15.852699+0000", 2024-09-13T15:03:35.034 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T15:03:35.034 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.034 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.034 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:03:35.034 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:15.852749+0000", 2024-09-13T15:03:35.034 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9879999999999997e-05 2024-09-13T15:03:35.035 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.035 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.035 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:03:35.035 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:15.852760+0000", 2024-09-13T15:03:35.035 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1005999999999999e-05 2024-09-13T15:03:35.035 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.035 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.035 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:03:35.035 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:15.852766+0000", 2024-09-13T15:03:35.035 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.383e-06 2024-09-13T15:03:35.035 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.035 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.035 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:35.036 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:15.852856+0000", 2024-09-13T15:03:35.036 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.9985e-05 2024-09-13T15:03:35.036 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.036 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.036 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:03:35.036 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:15.852864+0000", 2024-09-13T15:03:35.036 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.4120000000000001e-06 2024-09-13T15:03:35.036 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.036 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.036 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:35.036 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:16.838158+0000", 2024-09-13T15:03:35.036 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.98529345800000001 2024-09-13T15:03:35.036 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.037 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.037 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:03:35.037 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:16.838168+0000", 2024-09-13T15:03:35.037 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0553e-05 2024-09-13T15:03:35.037 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.037 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.037 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:35.037 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:18.229447+0000", 2024-09-13T15:03:35.037 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3912787980000001 2024-09-13T15:03:35.037 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.037 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.037 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:03:35.037 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:55:18.231199+0000", 2024-09-13T15:03:35.038 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0017517089999999999 2024-09-13T15:03:35.038 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:35.038 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:03:35.038 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:35.038 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.038 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.038 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:158550 10.1 10:8b6095c8:::benchmark_data_smithi151_288181_object32937:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1289)", 2024-09-13T15:03:35.038 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:00:25.030663+0000", 2024-09-13T15:03:35.038 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 189.883994403, 2024-09-13T15:03:35.038 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1097155240000001, 2024-09-13T15:03:35.038 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:03:35.038 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:35.038 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:03:35.038 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:03:35.039 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:03:35.039 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 158550 2024-09-13T15:03:35.039 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.039 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:03:35.039 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.039 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:03:35.039 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.030663+0000", 2024-09-13T15:03:35.039 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:35.039 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.039 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.039 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:03:35.039 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.030663+0000", 2024-09-13T15:03:35.039 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:35.040 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.040 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.040 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:03:35.040 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.030661+0000", 2024-09-13T15:03:35.040 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:03:35.040 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.040 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.040 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:03:35.040 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.030667+0000", 2024-09-13T15:03:35.040 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.665e-06 2024-09-13T15:03:35.040 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.040 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.040 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:03:35.041 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.030684+0000", 2024-09-13T15:03:35.041 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6813999999999999e-05 2024-09-13T15:03:35.041 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.041 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.041 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:03:35.041 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.030691+0000", 2024-09-13T15:03:35.041 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.9e-06 2024-09-13T15:03:35.041 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.041 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.041 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:35.041 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.031794+0000", 2024-09-13T15:03:35.041 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0011027089999999999 2024-09-13T15:03:35.041 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.042 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.042 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:03:35.042 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.031801+0000", 2024-09-13T15:03:35.042 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.2969999999999998e-06 2024-09-13T15:03:35.042 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.042 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.042 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:35.042 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.134251+0000", 2024-09-13T15:03:35.042 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1024500559999999 2024-09-13T15:03:35.042 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.042 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.042 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:03:35.042 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.136551+0000", 2024-09-13T15:03:35.042 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002300346 2024-09-13T15:03:35.043 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.043 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.043 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,1,2,4", 2024-09-13T15:03:35.043 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.136703+0000", 2024-09-13T15:03:35.043 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00015136899999999999 2024-09-13T15:03:35.043 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.043 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.043 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:35.043 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.139715+0000", 2024-09-13T15:03:35.043 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0030125540000000002 2024-09-13T15:03:35.043 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.043 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.043 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:35.044 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.139744+0000", 2024-09-13T15:03:35.044 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.8540000000000001e-05 2024-09-13T15:03:35.044 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.044 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.044 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:35.044 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.139998+0000", 2024-09-13T15:03:35.044 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00025390699999999999 2024-09-13T15:03:35.044 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.044 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.044 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:03:35.044 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.140013+0000", 2024-09-13T15:03:35.044 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.5288999999999999e-05 2024-09-13T15:03:35.044 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.045 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.045 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:35.045 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.140199+0000", 2024-09-13T15:03:35.045 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00018655200000000001 2024-09-13T15:03:35.045 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.045 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.045 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:03:35.045 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.140254+0000", 2024-09-13T15:03:35.045 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.414e-05 2024-09-13T15:03:35.045 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.045 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.045 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:03:35.045 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.140379+0000", 2024-09-13T15:03:35.045 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000124936 2024-09-13T15:03:35.046 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:35.046 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:03:35.046 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:35.046 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.046 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.046 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:64463 11.5 11:a79a7a20:::benchmark_data_smithi151_323728_object64462:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1337)", 2024-09-13T15:03:35.046 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:03:28.816354+0000", 2024-09-13T15:03:35.046 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 6.0983030759999997, 2024-09-13T15:03:35.046 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.14040116899999999, 2024-09-13T15:03:35.046 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:03:35.046 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:35.046 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:03:35.046 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:03:35.047 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:03:35.047 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 64463 2024-09-13T15:03:35.047 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.047 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:03:35.047 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.047 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:03:35.047 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:28.816354+0000", 2024-09-13T15:03:35.047 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:35.047 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.047 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.047 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:03:35.047 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:28.816354+0000", 2024-09-13T15:03:35.047 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:35.047 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.047 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.048 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:03:35.048 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:28.816354+0000", 2024-09-13T15:03:35.048 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:03:35.048 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.048 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.048 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:03:35.048 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:28.816412+0000", 2024-09-13T15:03:35.048 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8828e-05 2024-09-13T15:03:35.048 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.048 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.048 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:03:35.048 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:28.816421+0000", 2024-09-13T15:03:35.048 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.6999999999999997e-06 2024-09-13T15:03:35.048 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.049 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.049 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:03:35.049 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:28.816425+0000", 2024-09-13T15:03:35.049 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4499999999999997e-06 2024-09-13T15:03:35.049 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.049 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.049 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:35.049 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:28.816460+0000", 2024-09-13T15:03:35.049 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.4230999999999998e-05 2024-09-13T15:03:35.049 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.049 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.049 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:03:35.049 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:28.816549+0000", 2024-09-13T15:03:35.050 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.9178999999999994e-05 2024-09-13T15:03:35.050 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.050 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.050 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 5,6", 2024-09-13T15:03:35.050 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:28.816638+0000", 2024-09-13T15:03:35.050 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.8695000000000004e-05 2024-09-13T15:03:35.050 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.050 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.050 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:03:35.050 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:28.818386+0000", 2024-09-13T15:03:35.050 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001747984 2024-09-13T15:03:35.050 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.050 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.051 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:35.051 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:28.818607+0000", 2024-09-13T15:03:35.051 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00022137099999999999 2024-09-13T15:03:35.051 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.051 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.051 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:35.051 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:28.956669+0000", 2024-09-13T15:03:35.051 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.13806215699999999 2024-09-13T15:03:35.051 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.051 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.051 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:03:35.051 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:28.956717+0000", 2024-09-13T15:03:35.051 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7790999999999997e-05 2024-09-13T15:03:35.052 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.052 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.052 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:03:35.052 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:28.956755+0000", 2024-09-13T15:03:35.052 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8588999999999999e-05 2024-09-13T15:03:35.052 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:35.052 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:03:35.052 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:35.052 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.052 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.052 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:65032 11.4 11:246a5368:::benchmark_data_smithi151_323728_object65031:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1337)", 2024-09-13T15:03:35.052 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:03:29.444857+0000", 2024-09-13T15:03:35.052 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 5.4698000159999998, 2024-09-13T15:03:35.053 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.258768467, 2024-09-13T15:03:35.053 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:03:35.053 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:35.053 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:03:35.053 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:03:35.053 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:03:35.053 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 65032 2024-09-13T15:03:35.053 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.053 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:03:35.053 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.053 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:03:35.053 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.444857+0000", 2024-09-13T15:03:35.053 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:35.054 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.054 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.054 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:03:35.054 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.444857+0000", 2024-09-13T15:03:35.054 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:35.054 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.054 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.054 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:03:35.054 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.444856+0000", 2024-09-13T15:03:35.054 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T15:03:35.054 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.054 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.054 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:03:35.054 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.444903+0000", 2024-09-13T15:03:35.055 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7034999999999999e-05 2024-09-13T15:03:35.055 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.055 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.055 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:03:35.055 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.444916+0000", 2024-09-13T15:03:35.055 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2996000000000001e-05 2024-09-13T15:03:35.055 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.055 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.055 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:03:35.055 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.444923+0000", 2024-09-13T15:03:35.055 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.2359999999999998e-06 2024-09-13T15:03:35.055 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.055 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.055 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:35.056 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.444980+0000", 2024-09-13T15:03:35.056 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6882999999999997e-05 2024-09-13T15:03:35.056 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.056 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.056 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:03:35.056 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.445114+0000", 2024-09-13T15:03:35.056 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013413699999999999 2024-09-13T15:03:35.056 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.056 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.056 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T15:03:35.056 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.445229+0000", 2024-09-13T15:03:35.056 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000115466 2024-09-13T15:03:35.056 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.057 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.057 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:03:35.057 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.447402+0000", 2024-09-13T15:03:35.057 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0021728419999999999 2024-09-13T15:03:35.057 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.057 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.057 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:35.057 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.447748+0000", 2024-09-13T15:03:35.057 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000345538 2024-09-13T15:03:35.057 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.057 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.057 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:35.057 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.703510+0000", 2024-09-13T15:03:35.058 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.255762092 2024-09-13T15:03:35.058 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.058 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.058 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:03:35.058 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.703575+0000", 2024-09-13T15:03:35.058 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.4760999999999999e-05 2024-09-13T15:03:35.058 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.058 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.058 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:03:35.058 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.703626+0000", 2024-09-13T15:03:35.058 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1150999999999999e-05 2024-09-13T15:03:35.058 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:35.058 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:03:35.058 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:35.059 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.059 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.059 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:65043 11.f 11:f38712a5:::benchmark_data_smithi151_323728_object65042:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1337)", 2024-09-13T15:03:35.059 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:03:29.454037+0000", 2024-09-13T15:03:35.059 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 5.460620413, 2024-09-13T15:03:35.059 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.25816714299999999, 2024-09-13T15:03:35.059 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:03:35.059 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:35.059 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:03:35.059 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:03:35.059 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:03:35.059 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 65043 2024-09-13T15:03:35.059 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.060 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:03:35.060 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.060 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:03:35.060 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.454037+0000", 2024-09-13T15:03:35.060 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:35.060 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.060 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.060 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:03:35.060 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.454037+0000", 2024-09-13T15:03:35.060 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:35.060 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.060 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.060 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:03:35.061 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.454036+0000", 2024-09-13T15:03:35.061 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:03:35.061 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.061 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.061 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:03:35.061 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.454084+0000", 2024-09-13T15:03:35.061 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8523000000000003e-05 2024-09-13T15:03:35.061 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.061 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.061 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:03:35.061 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.454096+0000", 2024-09-13T15:03:35.061 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2244e-05 2024-09-13T15:03:35.061 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.062 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.062 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:03:35.062 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.454101+0000", 2024-09-13T15:03:35.062 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3470000000000001e-06 2024-09-13T15:03:35.062 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.062 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.062 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:35.062 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.454137+0000", 2024-09-13T15:03:35.062 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.5828000000000003e-05 2024-09-13T15:03:35.062 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.062 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.062 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:03:35.062 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.454263+0000", 2024-09-13T15:03:35.063 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012643499999999999 2024-09-13T15:03:35.063 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.063 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.063 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:03:35.063 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.454403+0000", 2024-09-13T15:03:35.063 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013997400000000001 2024-09-13T15:03:35.063 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.063 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.063 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:03:35.063 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.461423+0000", 2024-09-13T15:03:35.063 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0070195730000000003 2024-09-13T15:03:35.063 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.063 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.063 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:35.064 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.461455+0000", 2024-09-13T15:03:35.064 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.2503999999999999e-05 2024-09-13T15:03:35.064 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.064 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.064 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:35.064 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.712155+0000", 2024-09-13T15:03:35.064 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.25069948199999997 2024-09-13T15:03:35.064 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.064 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.064 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:03:35.064 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.712184+0000", 2024-09-13T15:03:35.064 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.9088e-05 2024-09-13T15:03:35.064 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.064 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.065 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:03:35.065 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.712204+0000", 2024-09-13T15:03:35.065 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.0486e-05 2024-09-13T15:03:35.065 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:35.065 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:03:35.065 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:35.065 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.065 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.065 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:65045 11.f 11:f2f3f852:::benchmark_data_smithi151_323728_object65044:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1337)", 2024-09-13T15:03:35.065 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:03:29.455531+0000", 2024-09-13T15:03:35.065 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 5.4591264080000004, 2024-09-13T15:03:35.065 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.26314478899999999, 2024-09-13T15:03:35.065 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:03:35.066 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:35.066 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:03:35.066 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:03:35.066 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:03:35.066 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 65045 2024-09-13T15:03:35.066 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.066 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:03:35.066 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.066 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:03:35.066 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.455531+0000", 2024-09-13T15:03:35.066 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:35.066 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.066 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.067 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:03:35.067 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.455531+0000", 2024-09-13T15:03:35.067 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:35.067 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.067 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.067 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:03:35.067 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.455530+0000", 2024-09-13T15:03:35.067 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:03:35.067 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.067 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.067 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:03:35.067 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.455561+0000", 2024-09-13T15:03:35.067 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.0673000000000003e-05 2024-09-13T15:03:35.067 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.068 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.068 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:03:35.068 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.455576+0000", 2024-09-13T15:03:35.068 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.56e-05 2024-09-13T15:03:35.068 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.068 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.068 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:03:35.068 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.455582+0000", 2024-09-13T15:03:35.068 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3280000000000002e-06 2024-09-13T15:03:35.068 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.068 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.068 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:35.068 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.457070+0000", 2024-09-13T15:03:35.069 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0014876189999999999 2024-09-13T15:03:35.069 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.069 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.069 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:03:35.069 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.457192+0000", 2024-09-13T15:03:35.069 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000121502 2024-09-13T15:03:35.069 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.069 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.069 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:03:35.069 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.457315+0000", 2024-09-13T15:03:35.069 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000123275 2024-09-13T15:03:35.069 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.069 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.070 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:35.070 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.461501+0000", 2024-09-13T15:03:35.070 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0041866059999999998 2024-09-13T15:03:35.070 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.070 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.070 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:03:35.070 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.461515+0000", 2024-09-13T15:03:35.070 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3247e-05 2024-09-13T15:03:35.070 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.070 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.070 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:35.070 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.718530+0000", 2024-09-13T15:03:35.070 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.25701534700000001 2024-09-13T15:03:35.070 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.071 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.071 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:03:35.071 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.718629+0000", 2024-09-13T15:03:35.071 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.9443000000000003e-05 2024-09-13T15:03:35.071 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.071 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.071 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:03:35.071 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.718676+0000", 2024-09-13T15:03:35.071 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.6306e-05 2024-09-13T15:03:35.071 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:35.071 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:03:35.071 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:35.071 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.071 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.072 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:65048 11.f 11:f41a0961:::benchmark_data_smithi151_323728_object65047:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1337)", 2024-09-13T15:03:35.072 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:03:29.456098+0000", 2024-09-13T15:03:35.072 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 5.4585591449999997, 2024-09-13T15:03:35.072 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.262698775, 2024-09-13T15:03:35.072 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:03:35.072 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:35.072 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:03:35.072 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:03:35.072 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:03:35.072 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 65048 2024-09-13T15:03:35.072 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.072 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:03:35.072 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.072 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:03:35.073 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.456098+0000", 2024-09-13T15:03:35.073 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:35.073 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.073 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.073 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:03:35.073 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.456098+0000", 2024-09-13T15:03:35.073 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:35.073 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.073 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.073 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:03:35.073 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.456097+0000", 2024-09-13T15:03:35.073 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:03:35.073 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.074 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.074 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:03:35.074 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.456223+0000", 2024-09-13T15:03:35.074 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012584099999999999 2024-09-13T15:03:35.074 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.074 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.074 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:03:35.074 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.456362+0000", 2024-09-13T15:03:35.074 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013894100000000001 2024-09-13T15:03:35.074 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.074 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.074 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:03:35.074 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.456370+0000", 2024-09-13T15:03:35.075 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0760000000000003e-06 2024-09-13T15:03:35.075 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.075 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.075 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:35.075 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.459233+0000", 2024-09-13T15:03:35.075 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0028623759999999998 2024-09-13T15:03:35.075 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.075 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.075 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:03:35.075 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.459344+0000", 2024-09-13T15:03:35.075 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000111567 2024-09-13T15:03:35.075 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.075 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.075 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:03:35.076 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.459463+0000", 2024-09-13T15:03:35.076 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000118934 2024-09-13T15:03:35.076 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.076 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.076 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:03:35.076 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.461560+0000", 2024-09-13T15:03:35.076 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0020972849999999999 2024-09-13T15:03:35.076 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.076 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.076 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:35.076 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.461890+0000", 2024-09-13T15:03:35.076 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00032947500000000003 2024-09-13T15:03:35.076 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.076 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.077 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:35.077 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.718712+0000", 2024-09-13T15:03:35.077 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.25682233100000001 2024-09-13T15:03:35.077 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.077 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.077 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:03:35.077 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.718764+0000", 2024-09-13T15:03:35.077 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1419999999999999e-05 2024-09-13T15:03:35.077 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.077 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.077 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:03:35.077 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.718797+0000", 2024-09-13T15:03:35.077 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3408999999999998e-05 2024-09-13T15:03:35.077 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:35.078 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:03:35.078 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:35.078 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.078 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.078 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:65050 11.4 11:2d37fcce:::benchmark_data_smithi151_323728_object65049:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1337)", 2024-09-13T15:03:35.078 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:03:29.458797+0000", 2024-09-13T15:03:35.078 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 5.4558602050000005, 2024-09-13T15:03:35.078 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.249468412, 2024-09-13T15:03:35.078 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:03:35.078 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:35.078 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:03:35.078 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:03:35.078 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:03:35.078 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 65050 2024-09-13T15:03:35.079 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.079 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:03:35.080 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.080 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:03:35.080 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.458797+0000", 2024-09-13T15:03:35.080 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:35.080 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.080 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.081 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:03:35.081 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.458797+0000", 2024-09-13T15:03:35.081 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:35.081 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.081 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.081 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:03:35.081 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.458796+0000", 2024-09-13T15:03:35.081 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:03:35.081 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.081 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.081 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:03:35.081 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.458887+0000", 2024-09-13T15:03:35.082 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.1077999999999998e-05 2024-09-13T15:03:35.082 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.082 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.082 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:03:35.082 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.458902+0000", 2024-09-13T15:03:35.082 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4776999999999999e-05 2024-09-13T15:03:35.082 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.082 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.082 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:03:35.082 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.458908+0000", 2024-09-13T15:03:35.082 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1500000000000004e-06 2024-09-13T15:03:35.082 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.082 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.082 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:35.083 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.458960+0000", 2024-09-13T15:03:35.083 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1802999999999999e-05 2024-09-13T15:03:35.083 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.083 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.083 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:03:35.083 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.459086+0000", 2024-09-13T15:03:35.083 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000125986 2024-09-13T15:03:35.083 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.083 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.083 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T15:03:35.083 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.459278+0000", 2024-09-13T15:03:35.083 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00019209899999999999 2024-09-13T15:03:35.083 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.084 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.084 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:03:35.084 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.461801+0000", 2024-09-13T15:03:35.084 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002522472 2024-09-13T15:03:35.084 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.084 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.084 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:35.084 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.461942+0000", 2024-09-13T15:03:35.084 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014171999999999999 2024-09-13T15:03:35.084 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.084 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.084 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:35.084 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.708213+0000", 2024-09-13T15:03:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.24627043500000001 2024-09-13T15:03:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:03:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.708243+0000", 2024-09-13T15:03:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.9731e-05 2024-09-13T15:03:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:03:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.708266+0000", 2024-09-13T15:03:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.3051e-05 2024-09-13T15:03:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:03:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:35.086 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.086 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.086 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:65054 11.f 11:f78e8263:::benchmark_data_smithi151_323728_object65053:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1337)", 2024-09-13T15:03:35.086 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:03:29.462921+0000", 2024-09-13T15:03:35.086 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 5.451736275, 2024-09-13T15:03:35.086 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.25599048499999999, 2024-09-13T15:03:35.086 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:03:35.086 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:35.086 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:03:35.086 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:03:35.086 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:03:35.086 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 65054 2024-09-13T15:03:35.086 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.086 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:03:35.087 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.087 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:03:35.087 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.462921+0000", 2024-09-13T15:03:35.087 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:35.087 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.087 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.087 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:03:35.087 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.462921+0000", 2024-09-13T15:03:35.087 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:35.087 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.087 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.087 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:03:35.087 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.462920+0000", 2024-09-13T15:03:35.088 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:03:35.088 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.088 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.088 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:03:35.088 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.462977+0000", 2024-09-13T15:03:35.088 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7173999999999998e-05 2024-09-13T15:03:35.088 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.088 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.088 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:03:35.088 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.462987+0000", 2024-09-13T15:03:35.088 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.8470000000000008e-06 2024-09-13T15:03:35.088 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.088 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.088 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:03:35.089 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.462991+0000", 2024-09-13T15:03:35.089 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.5860000000000001e-06 2024-09-13T15:03:35.089 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.089 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.089 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:35.089 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.463020+0000", 2024-09-13T15:03:35.089 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.9501000000000001e-05 2024-09-13T15:03:35.089 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.089 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.089 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:03:35.089 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.463100+0000", 2024-09-13T15:03:35.089 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.9427999999999999e-05 2024-09-13T15:03:35.089 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.090 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.090 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:03:35.090 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.463204+0000", 2024-09-13T15:03:35.090 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000104147 2024-09-13T15:03:35.090 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.090 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.090 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:03:35.090 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.465233+0000", 2024-09-13T15:03:35.090 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0020290059999999999 2024-09-13T15:03:35.090 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.090 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.090 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:35.090 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.465649+0000", 2024-09-13T15:03:35.090 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00041594899999999998 2024-09-13T15:03:35.091 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.091 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.091 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:35.091 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.718838+0000", 2024-09-13T15:03:35.091 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.25318934999999998 2024-09-13T15:03:35.091 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.091 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.091 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:03:35.091 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.718883+0000", 2024-09-13T15:03:35.091 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4527999999999998e-05 2024-09-13T15:03:35.091 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.091 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.091 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:03:35.091 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.718912+0000", 2024-09-13T15:03:35.092 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.8804000000000001e-05 2024-09-13T15:03:35.092 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:35.092 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:03:35.092 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:35.092 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.092 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.092 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:65055 11.f 11:f09cec47:::benchmark_data_smithi151_323728_object65054:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1337)", 2024-09-13T15:03:35.092 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:03:29.465531+0000", 2024-09-13T15:03:35.092 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 5.4491264409999998, 2024-09-13T15:03:35.092 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.25715232199999999, 2024-09-13T15:03:35.092 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:03:35.092 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:35.092 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:03:35.092 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:03:35.093 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:03:35.093 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 65055 2024-09-13T15:03:35.093 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.093 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:03:35.093 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.093 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:03:35.093 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.465531+0000", 2024-09-13T15:03:35.093 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:35.093 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.093 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.093 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:03:35.093 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.465531+0000", 2024-09-13T15:03:35.093 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:03:35.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.094 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.094 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:03:35.094 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.465530+0000", 2024-09-13T15:03:35.094 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:03:35.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.094 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.094 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:03:35.094 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.465596+0000", 2024-09-13T15:03:35.094 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.6105999999999994e-05 2024-09-13T15:03:35.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.094 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.094 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:03:35.094 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.465606+0000", 2024-09-13T15:03:35.095 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.8889999999999997e-06 2024-09-13T15:03:35.095 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.095 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.095 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:03:35.095 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.465610+0000", 2024-09-13T15:03:35.095 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3610000000000003e-06 2024-09-13T15:03:35.095 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.095 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.095 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:03:35.095 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.465666+0000", 2024-09-13T15:03:35.095 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5889e-05 2024-09-13T15:03:35.095 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.095 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.095 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:03:35.096 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.465757+0000", 2024-09-13T15:03:35.096 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.0937e-05 2024-09-13T15:03:35.096 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.096 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.096 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:03:35.096 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.465844+0000", 2024-09-13T15:03:35.096 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.7004999999999998e-05 2024-09-13T15:03:35.096 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.096 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.096 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:03:35.096 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.467333+0000", 2024-09-13T15:03:35.096 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001489251 2024-09-13T15:03:35.096 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.096 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.096 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:35.097 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.467440+0000", 2024-09-13T15:03:35.097 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000106969 2024-09-13T15:03:35.097 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.097 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.097 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:35.097 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.722594+0000", 2024-09-13T15:03:35.097 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.25515391100000001 2024-09-13T15:03:35.097 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.097 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.097 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:03:35.097 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.722642+0000", 2024-09-13T15:03:35.097 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7778000000000002e-05 2024-09-13T15:03:35.097 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:03:35.097 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:03:35.098 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:03:35.098 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.722683+0000", 2024-09-13T15:03:35.098 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1414e-05 2024-09-13T15:03:35.098 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:35.098 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:03:35.098 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:35.098 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:03:35.098 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:03:35.098 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T15:03:35.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:35.099 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:35.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:35.177 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:2024-09-13T15:03:35.176972+0000 min lat: 0.00120778 max lat: 0.274253 avg lat: 0.00883293 2024-09-13T15:03:35.178 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T15:03:35.178 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 40 16 72438 72422 113.144 82.8125 0.00734436 0.00883293 2024-09-13T15:03:35.200 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:35.244 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:35.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:35.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:35.268 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:34 smithi151 ceph-mon[104450]: pgmap v2268: 119 pgs: 119 active+clean; 4.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-13T15:03:35.268 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:34 smithi151 ceph-mon[108966]: pgmap v2268: 119 pgs: 119 active+clean; 4.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-13T15:03:35.346 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:35.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:34 smithi184 ceph-mon[92527]: pgmap v2268: 119 pgs: 119 active+clean; 4.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-13T15:03:35.375 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:35.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:35.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:35.476 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:35.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:35.515 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:35.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:35.616 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:35.647 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:35.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:35.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:35.719 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T15:03:35.719 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T15:03:35.719 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T15:03:35.719 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T15:03:35.719 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T15:03:35.719 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:03:35.719 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:03:35.720 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:03:35.720 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:03:35.720 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:03:35.720 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T15:03:35.720 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T15:03:35.720 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T15:03:35.720 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:03:35.721 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:03:35.721 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:03:35.749 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:35.786 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:35.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:35.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:35.826 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T15:03:35.888 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:35.937 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:35.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:35.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:36.039 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:36.065 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T15:03:36.086 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:36.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:36.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:36.128 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:35 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2771023618' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:36.128 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:36 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2771023618' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:36.178 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 41 16 73464 73448 111.946 64.125 0.0164435 0.00892721 2024-09-13T15:03:36.189 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:36.267 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:36.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:36.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:36.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:35 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2771023618' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:36.368 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:36.413 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:36.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:36.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:36.514 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:36.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:36.558 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:36.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:36.659 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:36.703 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:36.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:36.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:36.806 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:36.846 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:36.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:36.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:36.949 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:36.980 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:36.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:36.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:37.082 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:37.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:37.120 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:37.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:37.178 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 42 16 74613 74597 110.99 71.8125 0.0105386 0.00900411 2024-09-13T15:03:37.221 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:37.247 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:37.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:37.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:37.253 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:37 smithi151 ceph-mon[104450]: pgmap v2269: 119 pgs: 119 active+clean; 4.2 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-13T15:03:37.253 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:37 smithi151 ceph-mon[108966]: pgmap v2269: 119 pgs: 119 active+clean; 4.2 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-13T15:03:37.349 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:37.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:36 smithi184 ceph-mon[92527]: pgmap v2269: 119 pgs: 119 active+clean; 4.2 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-13T15:03:37.382 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:37.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:37.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:37.484 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:37.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:37.517 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:37.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:37.617 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:37.653 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:37.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:37.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:37.755 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:37.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:37.782 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:37.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:37.841 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:37.874 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:37.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:37.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:37.937 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:37.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:38.038 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:38.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:38.078 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:38.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:38.179 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 43 16 75916 75900 110.302 81.4375 0.013961 0.00906089 2024-09-13T15:03:38.179 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:38.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:38.228 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:38.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:38.329 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:38.365 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:38.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:38.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:38.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:38.507 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:38.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:38.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:38.609 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:38.641 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:38.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:38.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:38.743 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:38.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:38.778 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:38.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:38.878 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:38.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:38.922 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:38.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:39.023 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:39.049 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:39 smithi151 ceph-mon[104450]: pgmap v2270: 119 pgs: 119 active+clean; 4.4 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-13T15:03:39.049 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:39 smithi151 ceph-mon[108966]: pgmap v2270: 119 pgs: 119 active+clean; 4.4 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-13T15:03:39.055 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:39.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:39.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:39.157 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:39.179 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 44 16 77189 77173 109.604 79.5625 0.00904589 0.00911865 2024-09-13T15:03:39.188 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:39.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:39.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:39.290 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:39.322 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:39.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:39.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:39.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:39 smithi184 ceph-mon[92527]: pgmap v2270: 119 pgs: 119 active+clean; 4.4 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-13T15:03:39.425 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:39.457 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:39.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:39.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:39.559 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:39.588 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:39.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:39.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:39.690 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:39.728 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:39.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:39.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:39.820 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:03:39.830 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:39.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:39 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a[104427]: 2024-09-13T15:03:39.595+0000 7fe9da14f640 -1 mon.a@0(leader).osd e1338 definitely_dead 0 2024-09-13T15:03:39.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:39.913 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:39.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:40.014 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:40.044 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:40.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:40.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:40.146 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:40.169 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:40.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:40.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:40.178 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 45 16 78286 78270 108.691 68.5625 0.00692803 0.0091688 2024-09-13T15:03:40.221 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:40 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/846045527' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-13T15:03:40.221 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:40 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-13T15:03:40.221 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:40 smithi151 ceph-mon[104450]: Client client.admin marked osd.2 out, while it was still marked up 2024-09-13T15:03:40.221 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:40 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/846045527' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-13T15:03:40.221 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:40 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-13T15:03:40.221 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:40 smithi151 ceph-mon[108966]: Client client.admin marked osd.2 out, while it was still marked up 2024-09-13T15:03:40.271 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:40.293 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:40.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:40.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:40.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:40 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/846045527' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-13T15:03:40.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:40 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-13T15:03:40.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:40 smithi184 ceph-mon[92527]: Client client.admin marked osd.2 out, while it was still marked up 2024-09-13T15:03:40.354 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:03:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:03:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:03:40.395 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:40.416 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:40.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:40.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:40.518 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:40.540 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:40.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:40.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:40.642 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:40.694 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:40.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:40.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:40.796 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:40.817 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:40.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:40.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:40.919 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:40.942 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:40.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:40.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:41.044 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:41.067 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:41.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:41.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:41.169 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:41.181 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 46 16 78286 78270 106.324 0 - 0.0091688 2024-09-13T15:03:41.207 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:41 smithi151 ceph-mon[104450]: pgmap v2271: 119 pgs: 119 active+clean; 4.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-13T15:03:41.207 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:41 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-09-13T15:03:41.207 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:41 smithi151 ceph-mon[104450]: osdmap e1339: 8 total, 8 up, 5 in 2024-09-13T15:03:41.207 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:41 smithi151 ceph-mon[104450]: osdmap e1340: 8 total, 8 up, 5 in 2024-09-13T15:03:41.208 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:41 smithi151 ceph-mon[108966]: pgmap v2271: 119 pgs: 119 active+clean; 4.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-13T15:03:41.208 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:41 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-09-13T15:03:41.208 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:41 smithi151 ceph-mon[108966]: osdmap e1339: 8 total, 8 up, 5 in 2024-09-13T15:03:41.208 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:41 smithi151 ceph-mon[108966]: osdmap e1340: 8 total, 8 up, 5 in 2024-09-13T15:03:41.208 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T15:03:41.208 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:03:41.225 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:41.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:41.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:41.284 INFO:teuthology.orchestra.run.smithi151.stderr:marked out osd.2. 2024-09-13T15:03:41.327 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:41.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:41 smithi184 ceph-mon[92527]: pgmap v2271: 119 pgs: 119 active+clean; 4.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-13T15:03:41.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:41 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-09-13T15:03:41.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:41 smithi184 ceph-mon[92527]: osdmap e1339: 8 total, 8 up, 5 in 2024-09-13T15:03:41.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:41 smithi184 ceph-mon[92527]: osdmap e1340: 8 total, 8 up, 5 in 2024-09-13T15:03:41.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:41.363 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:41.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:41.464 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:41.498 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:41.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:41.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:41.601 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:41.637 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:41.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:41.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:41.739 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:41.777 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:41.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:41.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:41.879 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:41.920 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:41.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:41.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:42.025 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:42.054 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:42.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:42.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:42.157 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:42.181 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 47 16 79210 79194 105.29 28.875 0.0093096 0.00949137 2024-09-13T15:03:42.187 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:42.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:42.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:42.288 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:42.330 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:42.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:42.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:42.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:42 smithi184 ceph-mon[92527]: 11.4 starting backfill to osd.1 from (0'0,0'0] MAX to 1338'4805 2024-09-13T15:03:42.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:42 smithi184 ceph-mon[92527]: 11.f starting backfill to osd.3 from (0'0,0'0] MAX to 1338'4843 2024-09-13T15:03:42.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:42 smithi184 ceph-mon[92527]: 11.1 starting backfill to osd.1 from (0'0,0'0] MAX to 1338'5021 2024-09-13T15:03:42.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:42 smithi184 ceph-mon[92527]: osdmap e1341: 8 total, 8 up, 5 in 2024-09-13T15:03:42.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:42 smithi184 ceph-mon[92527]: 11.c starting backfill to osd.5 from (0'0,0'0] MAX to 1338'5027 2024-09-13T15:03:42.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:42 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:03:42.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:42 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:03:42.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:42 smithi151 ceph-mon[104450]: 11.4 starting backfill to osd.1 from (0'0,0'0] MAX to 1338'4805 2024-09-13T15:03:42.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:42 smithi151 ceph-mon[104450]: 11.f starting backfill to osd.3 from (0'0,0'0] MAX to 1338'4843 2024-09-13T15:03:42.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:42 smithi151 ceph-mon[104450]: 11.1 starting backfill to osd.1 from (0'0,0'0] MAX to 1338'5021 2024-09-13T15:03:42.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:42 smithi151 ceph-mon[104450]: osdmap e1341: 8 total, 8 up, 5 in 2024-09-13T15:03:42.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:42 smithi151 ceph-mon[104450]: 11.c starting backfill to osd.5 from (0'0,0'0] MAX to 1338'5027 2024-09-13T15:03:42.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:42 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:03:42.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:42 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:03:42.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:42 smithi151 ceph-mon[108966]: 11.4 starting backfill to osd.1 from (0'0,0'0] MAX to 1338'4805 2024-09-13T15:03:42.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:42 smithi151 ceph-mon[108966]: 11.f starting backfill to osd.3 from (0'0,0'0] MAX to 1338'4843 2024-09-13T15:03:42.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:42 smithi151 ceph-mon[108966]: 11.1 starting backfill to osd.1 from (0'0,0'0] MAX to 1338'5021 2024-09-13T15:03:42.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:42 smithi151 ceph-mon[108966]: osdmap e1341: 8 total, 8 up, 5 in 2024-09-13T15:03:42.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:42 smithi151 ceph-mon[108966]: 11.c starting backfill to osd.5 from (0'0,0'0] MAX to 1338'5027 2024-09-13T15:03:42.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:42 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:03:42.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:42 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:03:42.432 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:42.463 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:42.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:42.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:42.565 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:42.631 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:42.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:42.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:42.733 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:42.763 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:42.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:42.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:42.865 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:42.906 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:42.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:42.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:43.007 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:43.034 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:43.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:43.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:43.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:43.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:43.172 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:43.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:43.181 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 48 16 80190 80174 104.373 61.25 0.00676966 0.00957489 2024-09-13T15:03:43.273 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:43.304 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:43.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:43.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:43.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:43 smithi184 ceph-mon[92527]: pgmap v2275: 119 pgs: 119 active+clean; 4.6 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-13T15:03:43.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:43 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:03:43.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:43 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:03:43.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:43 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:03:43.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:43 smithi151 ceph-mon[104450]: pgmap v2275: 119 pgs: 119 active+clean; 4.6 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-13T15:03:43.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:43 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:03:43.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:43 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:03:43.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:43 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:03:43.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:43 smithi151 ceph-mon[108966]: pgmap v2275: 119 pgs: 119 active+clean; 4.6 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-13T15:03:43.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:43 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:03:43.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:43 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:03:43.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:43 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:03:43.406 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:43.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:43.449 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:43.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:43.553 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:43.585 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:43.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:43.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:43.687 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:43.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:43.723 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:43.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:43.824 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:43.855 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:43.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:43.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:43.903 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:43.916 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T15:03:43.957 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:43.990 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:43.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:43.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:44.092 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:44.118 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:44.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:44.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:44.182 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 49 16 81251 81235 103.595 66.3125 0.0096257 0.00964651 2024-09-13T15:03:44.220 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:44.248 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:44.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:44.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:44.350 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:44.384 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:44.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:44.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:44.486 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:44.522 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:44.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:44.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:44.526 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:44 smithi151 ceph-mon[104450]: Health check failed: Degraded data redundancy: 5001/235305 objects degraded (2.125%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:03:44.526 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:44 smithi151 ceph-mon[108966]: Health check failed: Degraded data redundancy: 5001/235305 objects degraded (2.125%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:03:44.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:44 smithi184 ceph-mon[92527]: Health check failed: Degraded data redundancy: 5001/235305 objects degraded (2.125%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:03:44.624 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:44.655 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:44.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:44.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:44.757 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:44.795 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:44.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:44.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:44.809 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:03:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:03:44] "GET /metrics HTTP/1.1" 200 36700 "" "Prometheus/2.51.0" 2024-09-13T15:03:44.898 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:44.930 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:44.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:44.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:45.032 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:45.067 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:45.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:45.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:45.169 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:45.181 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 50 16 82298 82282 102.832 65.4375 0.0167506 0.00971823 2024-09-13T15:03:45.205 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:45.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:45.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:45.307 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:45.337 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:45.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:45.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:45.439 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:45.473 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:45.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:45.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:45.575 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:45.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:45 smithi184 ceph-mon[92527]: pgmap v2276: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 114 active+clean; 4.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 60 MiB/s wr, 956 op/s; 5001/235305 objects degraded (2.125%); 19562/235305 objects misplaced (8.313%); 2.5 MiB/s, 1 keys/s, 41 objects/s recovering 2024-09-13T15:03:45.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:45 smithi151 ceph-mon[104450]: pgmap v2276: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 114 active+clean; 4.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 60 MiB/s wr, 956 op/s; 5001/235305 objects degraded (2.125%); 19562/235305 objects misplaced (8.313%); 2.5 MiB/s, 1 keys/s, 41 objects/s recovering 2024-09-13T15:03:45.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:45 smithi151 ceph-mon[108966]: pgmap v2276: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 114 active+clean; 4.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 60 MiB/s wr, 956 op/s; 5001/235305 objects degraded (2.125%); 19562/235305 objects misplaced (8.313%); 2.5 MiB/s, 1 keys/s, 41 objects/s recovering 2024-09-13T15:03:45.611 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:45.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:45.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:45.713 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:45.745 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:45.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:45.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:45.847 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:45.870 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:45.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:45.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:45.972 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:46.012 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:46.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:46.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:46.114 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:46.145 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:46.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:46.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:46.172 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-13T15:03:46.182 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 51 16 83270 83254 102.007 60.75 0.015231 0.00979733 2024-09-13T15:03:46.247 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:46.281 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:46.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:46.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:46.383 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:46.408 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:46.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:46.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:46.422 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:46 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2173555855' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:46.423 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:46 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:46.423 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:46 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2173555855' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:46.423 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:46 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:46.510 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:46.548 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:46.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:46.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:46.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:46 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2173555855' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:46.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:46 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:46.650 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:46.684 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:46.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:46.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:46.786 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:46.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:46.825 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:46.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:46.926 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:46.959 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:46.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:46.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:47.061 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:47.093 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:47.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:47.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:47.182 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 52 16 84269 84253 101.246 62.4375 0.00484436 0.00986826 2024-09-13T15:03:47.194 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:47.230 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:47.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:47.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:47.331 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:47.368 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:47.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:47.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:47.469 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:47.490 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:47 smithi151 ceph-mon[104450]: pgmap v2277: 119 pgs: 1 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 4.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 45 MiB/s wr, 715 op/s; 9776/240150 objects degraded (4.071%); 24729/240150 objects misplaced (10.297%); 2.6 MiB/s, 1 keys/s, 43 objects/s recovering 2024-09-13T15:03:47.490 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:47 smithi151 ceph-mon[104450]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:03:47.490 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:47 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:03:47.490 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:47 smithi151 ceph-mon[104450]: osdmap e1342: 8 total, 8 up, 5 in 2024-09-13T15:03:47.490 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:47 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:03:47.490 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:47 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:47.490 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:47 smithi151 ceph-mon[108966]: pgmap v2277: 119 pgs: 1 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 4.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 45 MiB/s wr, 715 op/s; 9776/240150 objects degraded (4.071%); 24729/240150 objects misplaced (10.297%); 2.6 MiB/s, 1 keys/s, 43 objects/s recovering 2024-09-13T15:03:47.491 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:47 smithi151 ceph-mon[108966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:03:47.491 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:47 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:03:47.491 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:47 smithi151 ceph-mon[108966]: osdmap e1342: 8 total, 8 up, 5 in 2024-09-13T15:03:47.491 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:47 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:03:47.491 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:47 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:47.504 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:47.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:47.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:47.553 INFO:tasks.thrashosds.thrasher:in_osds: [3, 6, 7, 1, 5] out_osds: [0, 4, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T15:03:47.553 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:03:47.553 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-13T15:03:47.553 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 1.000000 2024-09-13T15:03:47.554 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 5 1 2024-09-13T15:03:47.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:47 smithi184 ceph-mon[92527]: pgmap v2277: 119 pgs: 1 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 4.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 45 MiB/s wr, 715 op/s; 9776/240150 objects degraded (4.071%); 24729/240150 objects misplaced (10.297%); 2.6 MiB/s, 1 keys/s, 43 objects/s recovering 2024-09-13T15:03:47.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:47 smithi184 ceph-mon[92527]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:03:47.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:47 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:03:47.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:47 smithi184 ceph-mon[92527]: osdmap e1342: 8 total, 8 up, 5 in 2024-09-13T15:03:47.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:47 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:03:47.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:47 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:47.606 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:47.658 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:47.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:47.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:47.760 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:47.792 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:47.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:47.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:47.894 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:47.900 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T15:03:47.901 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:03:47.906 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:47.943 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T15:03:47.943 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:03:47.962 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:47.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:47.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:47.975 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T15:03:47.975 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:03:48.013 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T15:03:48.013 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:03:48.045 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T15:03:48.045 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:03:48.068 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:48.077 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T15:03:48.077 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:03:48.097 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:48.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:48.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:48.108 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T15:03:48.108 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:03:48.137 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T15:03:48.137 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:03:48.182 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 53 16 85243 85227 100.484 60.875 0.00855473 0.00994505 2024-09-13T15:03:48.199 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:48.234 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:48.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:48.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:48.337 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:48.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:48 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/825017775' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:48.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:48 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/825017775' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:48.372 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:48.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:48.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:48.474 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:48.508 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:48.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:48.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:48.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:48 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/825017775' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:48.609 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:48.643 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:48.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:48.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:48.744 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:48.775 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:48.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:48.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:48.877 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:48.906 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:48.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:48.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:49.008 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:49.045 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:49.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:49.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:49.148 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:49.182 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 54 16 86314 86298 99.8624 66.9375 0.0271867 0.0100072 2024-09-13T15:03:49.183 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:49.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:49.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:49.194 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:03:49.260 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:49 smithi151 ceph-mon[104450]: pgmap v2279: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 5.1 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 60 MiB/s wr, 962 op/s; 9776/249756 objects degraded (3.914%); 24716/249756 objects misplaced (9.896%); 9.7 MiB/s, 1 keys/s, 156 objects/s recovering 2024-09-13T15:03:49.260 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:49 smithi151 ceph-mon[108966]: pgmap v2279: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 5.1 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 60 MiB/s wr, 962 op/s; 9776/249756 objects degraded (3.914%); 24716/249756 objects misplaced (9.896%); 9.7 MiB/s, 1 keys/s, 156 objects/s recovering 2024-09-13T15:03:49.285 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:49.313 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:49.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:49.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:49.414 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:49.459 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:49.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:49.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:49.504 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:49 smithi184 ceph-mon[92527]: pgmap v2279: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 5.1 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 60 MiB/s wr, 962 op/s; 9776/249756 objects degraded (3.914%); 24716/249756 objects misplaced (9.896%); 9.7 MiB/s, 1 keys/s, 156 objects/s recovering 2024-09-13T15:03:49.561 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:49.587 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:49.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:49.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:49.689 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:49.723 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:49.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:49.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:49.825 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:49.862 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:49.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:49.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:49.964 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:49.997 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:49.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:49.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:50.048 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:50.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:50.151 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:50.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:50.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:50.183 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 55 16 87267 87251 99.1297 59.5625 0.00909902 0.0100825 2024-09-13T15:03:50.254 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:50.295 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:50.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:50.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:50.344 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:50 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 9776/249756 objects degraded (3.914%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:03:50.344 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:03:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:03:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:03:50.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:50 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 9776/249756 objects degraded (3.914%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:03:50.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:50 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 9776/249756 objects degraded (3.914%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:03:50.397 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:50.432 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:50.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:50.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:50.534 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:50.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:50.573 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:50.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:50.674 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:50.705 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:50.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:50.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:50.808 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:50.851 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:50.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:50.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:50.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:50.989 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:50.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:50.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:51.091 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:51.127 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:51.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:51.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:51.182 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 56 16 88279 88263 98.4889 63.25 0.00915521 0.0101482 2024-09-13T15:03:51.229 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:51.266 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:51.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:51.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:51.368 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:51.401 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:51.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:51.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:51.503 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:51.544 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:51 smithi151 ceph-mon[108966]: pgmap v2280: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 5.2 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 66 MiB/s wr, 1.06k op/s; 9776/255168 objects degraded (3.831%); 24982/255168 objects misplaced (9.790%); 8.6 MiB/s, 1 keys/s, 138 objects/s recovering 2024-09-13T15:03:51.545 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:51 smithi151 ceph-mon[104450]: pgmap v2280: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 5.2 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 66 MiB/s wr, 1.06k op/s; 9776/255168 objects degraded (3.831%); 24982/255168 objects misplaced (9.790%); 8.6 MiB/s, 1 keys/s, 138 objects/s recovering 2024-09-13T15:03:51.549 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:51.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:51.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:51.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:51 smithi184 ceph-mon[92527]: pgmap v2280: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 5.2 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 66 MiB/s wr, 1.06k op/s; 9776/255168 objects degraded (3.831%); 24982/255168 objects misplaced (9.790%); 8.6 MiB/s, 1 keys/s, 138 objects/s recovering 2024-09-13T15:03:51.651 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:51.684 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:51.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:51.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:51.786 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:51.821 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:51.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:51.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:51.923 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:51.958 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:51.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:51.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:52.060 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:52.071 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:03:52.097 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:52.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:52.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:52.182 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 57 16 89317 89301 97.8992 64.875 0.0149342 0.0102073 2024-09-13T15:03:52.200 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:52.233 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:52.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:52.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:52.335 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:52.365 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:52.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:52.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:52.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:52.506 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:52.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:52.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:52.608 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:52.648 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:52.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:52.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:52.750 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:52.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:52.785 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:52.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:52.886 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:52.917 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:52.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:52.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:53.019 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:53.061 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:53.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:53.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:53.163 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:53.182 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 58 16 90344 90328 97.3179 64.1875 0.00710635 0.0102705 2024-09-13T15:03:53.196 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:53.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:53.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:53.298 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:53.334 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:53.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:53.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:53.436 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:53.470 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:53.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:53.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:53.572 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:53.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:53.594 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:53.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:53.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:53 smithi184 ceph-mon[92527]: pgmap v2281: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 5.2 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 58 MiB/s wr, 929 op/s; 9776/255168 objects degraded (3.831%); 24982/255168 objects misplaced (9.790%); 7.6 MiB/s, 0 keys/s, 121 objects/s recovering 2024-09-13T15:03:53.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:53 smithi151 ceph-mon[104450]: pgmap v2281: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 5.2 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 58 MiB/s wr, 929 op/s; 9776/255168 objects degraded (3.831%); 24982/255168 objects misplaced (9.790%); 7.6 MiB/s, 0 keys/s, 121 objects/s recovering 2024-09-13T15:03:53.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:53 smithi151 ceph-mon[108966]: pgmap v2281: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 5.2 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 58 MiB/s wr, 929 op/s; 9776/255168 objects degraded (3.831%); 24982/255168 objects misplaced (9.790%); 7.6 MiB/s, 0 keys/s, 121 objects/s recovering 2024-09-13T15:03:53.695 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:53.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:53.731 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:53.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:53.832 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:53.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:53.866 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:53.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:53.967 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:54.000 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:54.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:54.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:54.102 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:54.106 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T15:03:54.106 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-13T15:03:54.107 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-13T15:03:54.107 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-13T15:03:54.107 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T15:03:54.142 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:54.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:54.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:54.182 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 59 16 91284 91268 96.6641 58.75 0.00345534 0.0103348 2024-09-13T15:03:54.244 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:54.274 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:54.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:54.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:54.377 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:54.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:54.425 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:54.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:54.526 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:54.566 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:54.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:54.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:54.668 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:54.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:54.707 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:54.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:54.808 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:54.836 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:54.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:54.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:54.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:03:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:03:54] "GET /metrics HTTP/1.1" 200 36734 "" "Prometheus/2.51.0" 2024-09-13T15:03:54.938 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:54.967 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:54.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:54.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:55.069 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:55.113 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:55.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:55.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:55.183 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:2024-09-13T15:03:55.182575+0000 min lat: 0.00120778 max lat: 1.27422 avg lat: 0.0104105 2024-09-13T15:03:55.183 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T15:03:55.183 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 60 16 92193 92177 95.9999 56.8125 0.00378235 0.0104105 2024-09-13T15:03:55.215 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:55.248 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:55.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:55.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:55.351 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:55.388 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:55.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:55.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:55.491 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:55.526 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:55.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:55.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:55.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:55 smithi184 ceph-mon[92527]: pgmap v2282: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 5.4 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 61 MiB/s wr, 979 op/s; 9776/264708 objects degraded (3.693%); 24915/264708 objects misplaced (9.412%); 12 MiB/s, 188 objects/s recovering 2024-09-13T15:03:55.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:55 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 9776/264708 objects degraded (3.693%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:03:55.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:55 smithi151 ceph-mon[104450]: pgmap v2282: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 5.4 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 61 MiB/s wr, 979 op/s; 9776/264708 objects degraded (3.693%); 24915/264708 objects misplaced (9.412%); 12 MiB/s, 188 objects/s recovering 2024-09-13T15:03:55.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:55 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 9776/264708 objects degraded (3.693%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:03:55.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:55 smithi151 ceph-mon[108966]: pgmap v2282: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 5.4 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 61 MiB/s wr, 979 op/s; 9776/264708 objects degraded (3.693%); 24915/264708 objects misplaced (9.412%); 12 MiB/s, 188 objects/s recovering 2024-09-13T15:03:55.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:55 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 9776/264708 objects degraded (3.693%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:03:55.629 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:55.658 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:55.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:55.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:55.760 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:55.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:55.796 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:55.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:55.897 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:55.928 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:55.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:55.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:56.030 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:56.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:56.071 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:56.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:56.173 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:56.184 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 61 16 93217 93201 95.4735 64 0.015656 0.0104685 2024-09-13T15:03:56.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:56.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:56.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:56.312 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:56.341 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:56.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:56.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:56.350 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:56.443 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:56.467 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:56.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:56.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:56.569 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:56.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:56.609 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:56.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:56.711 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:56.749 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:56.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:56.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:56.851 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:56.877 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:56.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:56.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:56.980 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:57.019 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:57.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:57.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:57.121 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:57.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:57.149 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:57.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:57.184 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 62 16 94181 94165 94.9054 60.25 0.0193142 0.0105316 2024-09-13T15:03:57.251 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:57.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:57.286 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:57.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:57.387 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:57.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:57.425 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:57.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:57.526 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:57.560 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:57.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:57.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:57.571 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:57 smithi151 ceph-mon[104450]: pgmap v2283: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 5.5 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 63 MiB/s wr, 1.01k op/s; 9776/270387 objects degraded (3.616%); 25244/270387 objects misplaced (9.336%); 12 MiB/s, 187 objects/s recovering 2024-09-13T15:03:57.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:57 smithi151 ceph-mon[108966]: pgmap v2283: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 5.5 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 63 MiB/s wr, 1.01k op/s; 9776/270387 objects degraded (3.616%); 25244/270387 objects misplaced (9.336%); 12 MiB/s, 187 objects/s recovering 2024-09-13T15:03:57.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:57 smithi184 ceph-mon[92527]: pgmap v2283: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 5.5 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 63 MiB/s wr, 1.01k op/s; 9776/270387 objects degraded (3.616%); 25244/270387 objects misplaced (9.336%); 12 MiB/s, 187 objects/s recovering 2024-09-13T15:03:57.665 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:57.706 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:57.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:57.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:57.808 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:57.841 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:57.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:57.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:57.943 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:57.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:57.982 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:57.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:58.083 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:58.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:58.120 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:58.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:58.184 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 63 16 95228 95212 94.4376 65.4375 0.0329634 0.0105834 2024-09-13T15:03:58.221 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:58.255 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:58.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:58.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:58.357 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:58.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:58.381 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:58.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:58.482 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:58.516 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:58.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:58.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:58.618 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:58.644 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:58.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:58.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:58.747 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:58.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:58.784 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:58.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:58.886 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:58.920 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:58.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:58.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:59.023 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:59.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:59.057 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:59.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:59.158 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:59.184 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 64 16 96162 96146 93.8741 58.375 0.0125158 0.0106463 2024-09-13T15:03:59.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:59.190 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:59.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:59.291 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:59.331 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:59.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:59.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:59.433 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:59.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:59.461 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:59.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:59.562 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:59.600 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:59.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:59.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:59.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:03:59 smithi184 ceph-mon[92527]: pgmap v2284: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 5.7 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 70 MiB/s wr, 1.12k op/s; 9776/279459 objects degraded (3.498%); 25164/279459 objects misplaced (9.005%); 15 MiB/s, 238 objects/s recovering 2024-09-13T15:03:59.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:03:59 smithi151 ceph-mon[104450]: pgmap v2284: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 5.7 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 70 MiB/s wr, 1.12k op/s; 9776/279459 objects degraded (3.498%); 25164/279459 objects misplaced (9.005%); 15 MiB/s, 238 objects/s recovering 2024-09-13T15:03:59.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:03:59 smithi151 ceph-mon[108966]: pgmap v2284: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 5.7 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 70 MiB/s wr, 1.12k op/s; 9776/279459 objects degraded (3.498%); 25164/279459 objects misplaced (9.005%); 15 MiB/s, 238 objects/s recovering 2024-09-13T15:03:59.702 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:59.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:59.738 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:59.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:59.839 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:59.871 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:03:59.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:59.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:59.973 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:59.995 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:03:59.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:59.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:00.097 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:00.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:00.134 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:00.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:00.184 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 65 16 97103 97087 93.3346 58.8125 0.00941925 0.0107072 2024-09-13T15:04:00.236 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:00.263 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:00.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:00.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:00.364 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:00.397 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:00.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:00.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:00.499 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:00.534 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:00.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:00.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:00.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:00 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 9776/279459 objects degraded (3.498%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:00.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:04:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:04:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:04:00.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:00 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 9776/279459 objects degraded (3.498%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:00.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:00 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 9776/279459 objects degraded (3.498%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:00.636 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:00.673 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:00.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:00.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:00.775 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:00.813 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:00.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:00.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:00.915 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:00.950 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:00.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:00.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:01.051 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:01.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:01.086 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:01.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:01.185 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 66 16 98060 98044 92.8267 59.8125 0.0129371 0.0107675 2024-09-13T15:04:01.187 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:01.223 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:01.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:01.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:01.324 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:01.359 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:01.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:01.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:01.461 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:01.502 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:01.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:01.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:01.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:01 smithi184 ceph-mon[92527]: pgmap v2285: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 5.8 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 61 MiB/s wr, 979 op/s; 9776/285015 objects degraded (3.430%); 25447/285015 objects misplaced (8.928%); 9.6 MiB/s, 152 objects/s recovering 2024-09-13T15:04:01.604 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:01.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:01 smithi151 ceph-mon[104450]: pgmap v2285: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 5.8 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 61 MiB/s wr, 979 op/s; 9776/285015 objects degraded (3.430%); 25447/285015 objects misplaced (8.928%); 9.6 MiB/s, 152 objects/s recovering 2024-09-13T15:04:01.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:01 smithi151 ceph-mon[108966]: pgmap v2285: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 5.8 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 61 MiB/s wr, 979 op/s; 9776/285015 objects degraded (3.430%); 25447/285015 objects misplaced (8.928%); 9.6 MiB/s, 152 objects/s recovering 2024-09-13T15:04:01.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:01.639 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:01.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:01.740 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:01.776 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:01.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:01.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:01.878 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:01.908 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:01.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:01.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:01.962 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T15:04:02.010 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:02.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:02.054 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:02.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:02.155 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:02.184 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 67 16 99028 99012 92.3442 60.5 0.0129945 0.0108235 2024-09-13T15:04:02.190 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:02.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:02.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:02.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:02.332 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:02.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:02.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:02.434 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:02.465 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:02.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:02.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:02.567 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:02.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:02.603 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:02.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:02.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:02.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:02.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:02.704 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:02.745 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:02.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:02.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:02.847 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:02.871 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:02.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:02.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:02.974 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:03.009 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:03.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:03.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:03.111 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:03.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:03.149 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:03.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:03.185 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 68 16 100073 100057 91.9466 65.3125 0.00362348 0.0108688 2024-09-13T15:04:03.250 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:03.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:03.278 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:03.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:03.379 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:03.417 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:03.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:03.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:03.519 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:03.555 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:03.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:03.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:03.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:03 smithi184 ceph-mon[92527]: pgmap v2286: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 5.8 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 52 MiB/s wr, 828 op/s; 9776/285015 objects degraded (3.430%); 25447/285015 objects misplaced (8.928%); 9.6 MiB/s, 152 objects/s recovering 2024-09-13T15:04:03.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:03 smithi151 ceph-mon[104450]: pgmap v2286: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 5.8 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 52 MiB/s wr, 828 op/s; 9776/285015 objects degraded (3.430%); 25447/285015 objects misplaced (8.928%); 9.6 MiB/s, 152 objects/s recovering 2024-09-13T15:04:03.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:03 smithi151 ceph-mon[108966]: pgmap v2286: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 5.8 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 52 MiB/s wr, 828 op/s; 9776/285015 objects degraded (3.430%); 25447/285015 objects misplaced (8.928%); 9.6 MiB/s, 152 objects/s recovering 2024-09-13T15:04:03.657 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:03.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:03.679 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:03.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:03.780 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:03.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:03.812 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:03.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:03.913 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:03.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:03.945 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:03.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:04.045 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:04.069 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:04.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:04.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:04.171 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:04.186 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 69 16 100909 100893 91.3699 52.25 0.00645075 0.010939 2024-09-13T15:04:04.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:04.215 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:04.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:04.316 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:04.337 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:04.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:04.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:04.439 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:04.481 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:04.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:04.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:04.583 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:04.615 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:04.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:04.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:04.717 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:04.755 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:04.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:04.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:04.857 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:04.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:04:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:04:04] "GET /metrics HTTP/1.1" 200 36722 "" "Prometheus/2.51.0" 2024-09-13T15:04:04.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:04.890 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:04.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:04.991 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:05.026 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:05.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:05.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:05.128 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:05.156 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:05.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:05.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:05.186 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 70 16 101846 101830 90.9011 58.5625 0.00405131 0.0109891 2024-09-13T15:04:05.258 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:05.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:05.295 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:05.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:05.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:05 smithi151 ceph-mon[104450]: pgmap v2287: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 6.0 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 68 MiB/s wr, 1.08k op/s; 9776/294150 objects degraded (3.323%); 25331/294150 objects misplaced (8.612%); 14 MiB/s, 227 objects/s recovering 2024-09-13T15:04:05.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:05 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 9776/294150 objects degraded (3.323%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:05.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:05 smithi151 ceph-mon[108966]: pgmap v2287: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 6.0 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 68 MiB/s wr, 1.08k op/s; 9776/294150 objects degraded (3.323%); 25331/294150 objects misplaced (8.612%); 14 MiB/s, 227 objects/s recovering 2024-09-13T15:04:05.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:05 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 9776/294150 objects degraded (3.323%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:05.396 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:05.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:05.435 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:05.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:05.536 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:05.565 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:05.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:05.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:05.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:05 smithi184 ceph-mon[92527]: pgmap v2287: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 6.0 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 68 MiB/s wr, 1.08k op/s; 9776/294150 objects degraded (3.323%); 25331/294150 objects misplaced (8.612%); 14 MiB/s, 227 objects/s recovering 2024-09-13T15:04:05.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:05 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 9776/294150 objects degraded (3.323%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:05.667 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:05.702 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:05.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:05.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:05.804 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:05.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:05.827 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:05.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:05.928 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:05.965 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:05.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:05.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:06.067 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:06.094 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:06.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:06.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:06.186 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 71 16 102760 102744 90.4253 57.125 0.0094025 0.011053 2024-09-13T15:04:06.196 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:06.245 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:06.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:06.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:06.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:06.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:06.497 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:06.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:06.598 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:06.628 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:06.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:06.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:06.730 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:06.774 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:06.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:06.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:06.876 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:06.902 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:06.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:06.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:07.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:07.007 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:07.088 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:07.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:07.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:07.187 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 72 16 103597 103581 89.8949 52.3125 0.011615 0.0111182 2024-09-13T15:04:07.191 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:07.222 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:07.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:07.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:07.245 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-13T15:04:07.325 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:07.365 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:07.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:07.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:07.467 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:07.503 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:07.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:07.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:07.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:07 smithi184 ceph-mon[92527]: pgmap v2288: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 6.1 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 60 MiB/s wr, 966 op/s; 9776/299514 objects degraded (3.264%); 25637/299514 objects misplaced (8.560%); 9.5 MiB/s, 151 objects/s recovering 2024-09-13T15:04:07.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:07 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1659676254' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:04:07.604 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:07.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:07 smithi151 ceph-mon[104450]: pgmap v2288: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 6.1 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 60 MiB/s wr, 966 op/s; 9776/299514 objects degraded (3.264%); 25637/299514 objects misplaced (8.560%); 9.5 MiB/s, 151 objects/s recovering 2024-09-13T15:04:07.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:07 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1659676254' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:04:07.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:07 smithi151 ceph-mon[108966]: pgmap v2288: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 6.1 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 60 MiB/s wr, 966 op/s; 9776/299514 objects degraded (3.264%); 25637/299514 objects misplaced (8.560%); 9.5 MiB/s, 151 objects/s recovering 2024-09-13T15:04:07.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:07 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1659676254' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:04:07.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:07.644 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:07.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:07.745 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:07.775 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:07.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:07.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:07.877 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:07.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:07.916 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:07.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:07.924 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:08.017 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:08.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:08.056 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:08.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:08.157 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:08.188 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 73 16 104532 104516 89.463 58.4375 0.0127117 0.0111727 2024-09-13T15:04:08.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:08.199 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:08.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:08.300 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:08.340 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:08.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:08.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:08.443 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:08.480 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:08.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:08.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:08.516 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:08 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1659676254' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:04:08.516 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:08 smithi151 ceph-mon[104450]: osdmap e1343: 8 total, 8 up, 5 in 2024-09-13T15:04:08.517 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:08 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1659676254' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:04:08.517 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:08 smithi151 ceph-mon[108966]: osdmap e1343: 8 total, 8 up, 5 in 2024-09-13T15:04:08.582 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:08.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:08 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1659676254' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:04:08.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:08 smithi184 ceph-mon[92527]: osdmap e1343: 8 total, 8 up, 5 in 2024-09-13T15:04:08.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:08.609 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:08.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:08.710 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:08.743 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:08.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:08.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:08.845 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:08.876 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:08.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:08.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:08.978 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:09.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:09.017 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:09.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:09.118 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:09.154 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:09.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:09.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:09.188 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 74 16 105576 105560 89.1358 65.25 0.0497815 0.0112119 2024-09-13T15:04:09.256 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:09.289 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:09.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:09.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:09.391 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:09.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:09.430 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:09.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:09.531 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:09.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:09 smithi151 ceph-mon[104450]: pgmap v2290: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 6.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 60 MiB/s wr, 953 op/s; 9776/308076 objects degraded (3.173%); 25419/308076 objects misplaced (8.251%); 11 MiB/s, 174 objects/s recovering 2024-09-13T15:04:09.555 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:09 smithi151 ceph-mon[108966]: pgmap v2290: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 6.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 60 MiB/s wr, 953 op/s; 9776/308076 objects degraded (3.173%); 25419/308076 objects misplaced (8.251%); 11 MiB/s, 174 objects/s recovering 2024-09-13T15:04:09.580 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:09.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:09.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:09.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:09 smithi184 ceph-mon[92527]: pgmap v2290: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 6.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 60 MiB/s wr, 953 op/s; 9776/308076 objects degraded (3.173%); 25419/308076 objects misplaced (8.251%); 11 MiB/s, 174 objects/s recovering 2024-09-13T15:04:09.682 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:09.712 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:09.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:09.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:09.814 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:09.848 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:09.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:09.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:09.950 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:09.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:09.994 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:09.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:10.095 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:10.122 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:10.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:10.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:10.188 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 75 16 106634 106618 88.829 66.125 0.0339842 0.0112514 2024-09-13T15:04:10.224 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:10.270 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:10.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:10.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:10.372 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:10.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:10.408 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:10.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:10.509 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:10.543 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:10.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:10.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:10.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:10 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 9776/308076 objects degraded (3.173%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:10.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:04:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:04:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:04:10.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:10 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 9776/308076 objects degraded (3.173%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:10.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:10 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 9776/308076 objects degraded (3.173%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:10.645 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:10.681 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:10.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:10.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:10.783 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:10.818 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:10.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:10.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:10.920 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:10.957 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:10.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:10.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:11.059 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:11.092 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:11.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:11.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:11.188 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 76 16 107691 107675 88.5294 66.0625 0.0243588 0.0112903 2024-09-13T15:04:11.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:11.240 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:11.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:11.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:11.289 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-13T15:04:11.342 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:11.386 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:11.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:11.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:11.487 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:11.521 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:11.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:11.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:11.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:11 smithi184 ceph-mon[92527]: pgmap v2291: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 6.4 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 59 MiB/s wr, 938 op/s; 9776/313206 objects degraded (3.121%); 25693/313206 objects misplaced (8.203%); 11 MiB/s, 174 objects/s recovering 2024-09-13T15:04:11.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:11 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3903496006' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:04:11.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:11 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:04:11.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:11 smithi151 ceph-mon[104450]: pgmap v2291: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 6.4 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 59 MiB/s wr, 938 op/s; 9776/313206 objects degraded (3.121%); 25693/313206 objects misplaced (8.203%); 11 MiB/s, 174 objects/s recovering 2024-09-13T15:04:11.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:11 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3903496006' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:04:11.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:11 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:04:11.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:11 smithi151 ceph-mon[108966]: pgmap v2291: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 6.4 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 59 MiB/s wr, 938 op/s; 9776/313206 objects degraded (3.121%); 25693/313206 objects misplaced (8.203%); 11 MiB/s, 174 objects/s recovering 2024-09-13T15:04:11.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:11 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3903496006' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:04:11.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:11 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:04:11.623 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:11.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:11.661 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:11.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:11.762 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:11.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:11.801 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:11.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:11.902 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:11.930 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:11.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:11.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:12.032 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:12.062 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:12.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:12.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:12.165 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:12.188 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 77 16 108718 108702 88.2132 64.1875 0.0132604 0.0113306 2024-09-13T15:04:12.202 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:12.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:12.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:12.304 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:12.336 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:12.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:12.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:12.438 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:12.465 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:12.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:12.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:12.567 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:12.603 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:12.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:12.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:12.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:12 smithi184 ceph-mon[92527]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:04:12.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:12 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:04:12.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:12 smithi184 ceph-mon[92527]: osdmap e1344: 8 total, 8 up, 5 in 2024-09-13T15:04:12.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:12 smithi151 ceph-mon[108966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:04:12.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:12 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:04:12.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:12 smithi151 ceph-mon[108966]: osdmap e1344: 8 total, 8 up, 5 in 2024-09-13T15:04:12.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:12 smithi151 ceph-mon[104450]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:04:12.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:12 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:04:12.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:12 smithi151 ceph-mon[104450]: osdmap e1344: 8 total, 8 up, 5 in 2024-09-13T15:04:12.705 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:12.734 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:12.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:12.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:12.836 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:12.864 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:12.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:12.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:12.966 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:13.007 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:13.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:13.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:13.112 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:13.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:13.154 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:13.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:13.188 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 78 16 109706 109690 87.8739 61.75 0.0169599 0.0113744 2024-09-13T15:04:13.255 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:13.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:13.285 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:13.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:13.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:13.421 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:13.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:13.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:13.523 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:13.547 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T15:04:13.547 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T15:04:13.547 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T15:04:13.547 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T15:04:13.547 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T15:04:13.547 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T15:04:13.548 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T15:04:13.548 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T15:04:13.548 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:04:13.548 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:04:13.548 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:04:13.548 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:04:13.548 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:04:13.548 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:04:13.548 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:04:13.548 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:04:13.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:13.560 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:13.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:13.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:13 smithi184 ceph-mon[92527]: pgmap v2293: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 6.4 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 50 MiB/s wr, 793 op/s; 9776/313206 objects degraded (3.121%); 25693/313206 objects misplaced (8.203%); 6.7 MiB/s, 106 objects/s recovering 2024-09-13T15:04:13.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:13 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1502151838' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:13.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:13 smithi151 ceph-mon[104450]: pgmap v2293: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 6.4 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 50 MiB/s wr, 793 op/s; 9776/313206 objects degraded (3.121%); 25693/313206 objects misplaced (8.203%); 6.7 MiB/s, 106 objects/s recovering 2024-09-13T15:04:13.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:13 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1502151838' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:13.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:13 smithi151 ceph-mon[108966]: pgmap v2293: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 6.4 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 50 MiB/s wr, 793 op/s; 9776/313206 objects degraded (3.121%); 25693/313206 objects misplaced (8.203%); 6.7 MiB/s, 106 objects/s recovering 2024-09-13T15:04:13.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:13 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1502151838' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:13.662 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:13.696 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:13.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:13.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:13.798 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:13.831 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:13.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:13.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:13.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:13.969 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:13.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:13.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:14.071 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:14.109 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:14.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:14.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:14.188 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 79 16 110756 110740 87.5922 65.625 0.00384281 0.0114102 2024-09-13T15:04:14.211 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:14.245 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:14.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:14.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:14.347 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:14.373 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:14.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:14.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:14.476 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:14.503 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:14.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:14.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:14.556 INFO:tasks.thrashosds.thrasher:in_osds: [3, 6, 7, 1, 5] out_osds: [0, 4, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T15:04:14.557 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:04:14.557 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-13T15:04:14.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight-by-utilization 1000 3.0 2024-09-13T15:04:14.588 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T15:04:14.588 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:04:14.604 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:14.611 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:04:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:04:14] "GET /metrics HTTP/1.1" 200 36722 "" "Prometheus/2.51.0" 2024-09-13T15:04:14.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:14.650 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:14.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:14.751 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:14.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:14.781 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:14.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:14.882 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:14.914 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:14.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:14.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:15.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:15.059 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:15.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:15.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:15.161 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:15.188 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:2024-09-13T15:04:15.188283+0000 min lat: 0.00120778 max lat: 1.27422 avg lat: 0.0114497 2024-09-13T15:04:15.188 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T15:04:15.189 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 80 16 111772 111756 87.291 63.5 0.0648139 0.0114497 2024-09-13T15:04:15.200 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:15.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:15.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:15.302 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:15.336 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:15.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:15.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:15.438 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:15.471 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:15.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:15.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:15.573 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:15.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:15 smithi184 ceph-mon[92527]: pgmap v2294: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 6.6 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 61 MiB/s wr, 975 op/s; 9776/322944 objects degraded (3.027%); 25470/322944 objects misplaced (7.887%); 14 MiB/s, 225 objects/s recovering 2024-09-13T15:04:15.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:15 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 9776/322944 objects degraded (3.027%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:15.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:15 smithi151 ceph-mon[104450]: pgmap v2294: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 6.6 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 61 MiB/s wr, 975 op/s; 9776/322944 objects degraded (3.027%); 25470/322944 objects misplaced (7.887%); 14 MiB/s, 225 objects/s recovering 2024-09-13T15:04:15.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:15 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 9776/322944 objects degraded (3.027%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:15.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:15 smithi151 ceph-mon[108966]: pgmap v2294: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 6.6 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 61 MiB/s wr, 975 op/s; 9776/322944 objects degraded (3.027%); 25470/322944 objects misplaced (7.887%); 14 MiB/s, 225 objects/s recovering 2024-09-13T15:04:15.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:15 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 9776/322944 objects degraded (3.027%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:15.612 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:15.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:15.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:15.713 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:15.749 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:15.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:15.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:15.851 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:15.889 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:15.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:15.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:15.991 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:16.017 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:16.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:16.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:16.034 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T15:04:16.067 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:16.118 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:16.148 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:16.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:16.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:16.189 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 81 16 112800 112784 87.0058 64.25 0.00463827 0.0114868 2024-09-13T15:04:16.251 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:16.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:16.296 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:16.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:16.397 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:16.426 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:16.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:16.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:16.525 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:04:16.528 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:16.554 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:16.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:16.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:16.656 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:16.701 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:16.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:16.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:16.803 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:16.829 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:16.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:16.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:16.931 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:16.962 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:16.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:16.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:17.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:17.097 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:17.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:17.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:17.189 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 82 16 113843 113827 86.7396 65.1875 0.00717069 0.0115236 2024-09-13T15:04:17.199 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:17.234 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:17.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:17.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:17.336 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:17.373 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:17.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:17.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:17.475 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:17.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:17.504 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:17.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:17.519 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T15:04:17.577 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:17 smithi151 ceph-mon[104450]: pgmap v2295: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 6.7 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 71 MiB/s wr, 1.13k op/s; 9776/328836 objects degraded (2.973%); 25828/328836 objects misplaced (7.854%); 13 MiB/s, 208 objects/s recovering 2024-09-13T15:04:17.577 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:17.577 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:17 smithi151 ceph-mon[108966]: pgmap v2295: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 6.7 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 71 MiB/s wr, 1.13k op/s; 9776/328836 objects degraded (2.973%); 25828/328836 objects misplaced (7.854%); 13 MiB/s, 208 objects/s recovering 2024-09-13T15:04:17.578 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:17.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:17 smithi184 ceph-mon[92527]: pgmap v2295: 119 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 6.7 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 71 MiB/s wr, 1.13k op/s; 9776/328836 objects degraded (2.973%); 25828/328836 objects misplaced (7.854%); 13 MiB/s, 208 objects/s recovering 2024-09-13T15:04:17.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:17.605 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:17.635 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:17.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:17.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:17.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:17.761 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:17.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:17.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:17.863 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:17.885 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:17.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:17.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:17.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:18.009 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:18.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:18.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:18.110 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:18.132 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:18.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:18.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:18.189 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 83 16 114329 114313 86.0606 30.375 0.00306316 0.0115304 2024-09-13T15:04:18.234 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:18.257 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:18.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:18.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:18.359 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:18.381 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:18.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:18.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:18.483 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:18.513 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:18.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:18.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:18.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:18 smithi184 ceph-mon[92527]: osdmap e1345: 8 total, 8 up, 5 in 2024-09-13T15:04:18.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:18 smithi151 ceph-mon[104450]: osdmap e1345: 8 total, 8 up, 5 in 2024-09-13T15:04:18.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:18 smithi151 ceph-mon[108966]: osdmap e1345: 8 total, 8 up, 5 in 2024-09-13T15:04:18.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:18.649 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:18.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:18.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:18.752 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:18.781 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:18.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:18.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:18.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:18.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:18.921 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:18.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:19.022 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:19.053 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:19.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:19.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:19.156 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:19.190 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 84 15 115130 115115 85.6323 50.125 0.0105553 0.0116721 2024-09-13T15:04:19.199 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:19.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:19.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:19.301 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:19.333 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:19.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:19.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:19.435 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:19.456 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:19.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:19.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:19.558 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:19.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:19.588 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:19.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:19.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:19 smithi151 ceph-mon[104450]: pgmap v2297: 119 pgs: 1 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 6.9 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 66 MiB/s wr, 1.05k op/s; 9776/338391 objects degraded (2.889%); 25590/338391 objects misplaced (7.562%); 14 MiB/s, 224 objects/s recovering 2024-09-13T15:04:19.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:19 smithi151 ceph-mon[104450]: osdmap e1346: 8 total, 8 up, 5 in 2024-09-13T15:04:19.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:19 smithi151 ceph-mon[108966]: pgmap v2297: 119 pgs: 1 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 6.9 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 66 MiB/s wr, 1.05k op/s; 9776/338391 objects degraded (2.889%); 25590/338391 objects misplaced (7.562%); 14 MiB/s, 224 objects/s recovering 2024-09-13T15:04:19.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:19 smithi151 ceph-mon[108966]: osdmap e1346: 8 total, 8 up, 5 in 2024-09-13T15:04:19.690 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:19.720 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:19.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:19.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:19.822 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:19.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:19 smithi184 ceph-mon[92527]: pgmap v2297: 119 pgs: 1 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 6.9 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 66 MiB/s wr, 1.05k op/s; 9776/338391 objects degraded (2.889%); 25590/338391 objects misplaced (7.562%); 14 MiB/s, 224 objects/s recovering 2024-09-13T15:04:19.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:19 smithi184 ceph-mon[92527]: osdmap e1346: 8 total, 8 up, 5 in 2024-09-13T15:04:19.856 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:19.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:19.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:19.958 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:19.992 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:19.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:19.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:20.094 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:20.124 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:20.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:20.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:20.190 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 85 16 116026 116010 85.2829 55.9375 0.0458024 0.0117201 2024-09-13T15:04:20.227 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:20.259 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:20.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:20.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:20.361 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:20.392 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:20.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:20.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:20.494 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:20.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:20.530 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:20.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:20.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:20 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 9776/338391 objects degraded (2.889%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:20.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:04:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:04:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:04:20.630 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:20.670 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:20.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:20.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:20.772 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:20.802 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:20.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:20.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:20.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:20 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 9776/338391 objects degraded (2.889%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:20.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:20 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 9776/338391 objects degraded (2.889%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:20.905 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:20.938 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:20.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:20.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:21.040 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:21.077 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:21.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:21.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:21.178 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:21.190 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 86 16 117131 117115 85.0943 69.0625 0.0084265 0.0117463 2024-09-13T15:04:21.213 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:21.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:21.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:21.315 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:21.348 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:21.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:21.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:21.450 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:21.489 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:21.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:21.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:21.591 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:21.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:21.626 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:21.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:21.727 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:21.758 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:21.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:21.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:21.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:21 smithi184 ceph-mon[92527]: pgmap v2299: 119 pgs: 1 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 7.0 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 78 MiB/s wr, 1.25k op/s; 9776/343260 objects degraded (2.848%); 25866/343260 objects misplaced (7.535%); 14 MiB/s, 224 objects/s recovering 2024-09-13T15:04:21.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:21 smithi151 ceph-mon[104450]: pgmap v2299: 119 pgs: 1 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 7.0 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 78 MiB/s wr, 1.25k op/s; 9776/343260 objects degraded (2.848%); 25866/343260 objects misplaced (7.535%); 14 MiB/s, 224 objects/s recovering 2024-09-13T15:04:21.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:21 smithi151 ceph-mon[108966]: pgmap v2299: 119 pgs: 1 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 7.0 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 78 MiB/s wr, 1.25k op/s; 9776/343260 objects degraded (2.848%); 25866/343260 objects misplaced (7.535%); 14 MiB/s, 224 objects/s recovering 2024-09-13T15:04:21.860 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:21.901 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:21.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:21.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:22.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:22.037 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:22.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:22.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:22.139 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:22.170 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:22.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:22.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:22.190 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 87 16 118160 118144 84.8553 64.3125 0.0243865 0.0117793 2024-09-13T15:04:22.273 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:22.306 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:22.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:22.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:22.408 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:22.439 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:22.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:22.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:22.541 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:22.579 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:22.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:22.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:22.681 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:22.702 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:22.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:22.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:22.803 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:22.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:22.834 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:22.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:22.935 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:22.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:22.974 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:22.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:23.075 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:23.106 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:23.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:23.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:23.191 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 88 16 119204 119188 84.6324 65.25 0.0227184 0.0118099 2024-09-13T15:04:23.208 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:23.242 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:23.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:23.272 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:23.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:23.373 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:23.407 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:23.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:23.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:23.477 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:23.495 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:23.505 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:23 smithi151 ceph-mon[108966]: pgmap v2300: 119 pgs: 1 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 7.0 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 53 MiB/s wr, 846 op/s; 9776/343260 objects degraded (2.848%); 25866/343260 objects misplaced (7.535%); 6.6 MiB/s, 105 objects/s recovering 2024-09-13T15:04:23.509 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:23.509 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:23 smithi151 ceph-mon[104450]: pgmap v2300: 119 pgs: 1 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 7.0 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 53 MiB/s wr, 846 op/s; 9776/343260 objects degraded (2.848%); 25866/343260 objects misplaced (7.535%); 6.6 MiB/s, 105 objects/s recovering 2024-09-13T15:04:23.538 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:23.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:23.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:23.640 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:23.676 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:23.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:23.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:23.778 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:23.819 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:23.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:23.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:23.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:23 smithi184 ceph-mon[92527]: pgmap v2300: 119 pgs: 1 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 112 active+clean; 7.0 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 53 MiB/s wr, 846 op/s; 9776/343260 objects degraded (2.848%); 25866/343260 objects misplaced (7.535%); 6.6 MiB/s, 105 objects/s recovering 2024-09-13T15:04:23.921 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:23.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:23.956 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:23.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:24.057 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:24.088 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:24.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:24.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:24.190 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:24.191 INFO:tasks.radosbench.radosbench.0.smithi151.stdout: 89 16 120210 120194 84.3879 62.875 0.00840895 0.0118445 2024-09-13T15:04:24.221 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:24.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:24.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:24.323 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:24.356 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:24.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:24.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:24.458 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:24.502 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:24.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:24.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:24.605 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:24.640 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:24.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:24.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:24.742 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:24.767 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:24.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:24.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:24.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:04:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:04:24] "GET /metrics HTTP/1.1" 200 36727 "" "Prometheus/2.51.0" 2024-09-13T15:04:24.869 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:24.902 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:24.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:24.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:25.004 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:25.033 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:25.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:25.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:25.134 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:25.167 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:25.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:25.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:25.191 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Total time run: 90.0075 2024-09-13T15:04:25.191 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Total writes made: 121294 2024-09-13T15:04:25.191 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Write size: 65536 2024-09-13T15:04:25.191 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Object size: 65536 2024-09-13T15:04:25.191 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Bandwidth (MB/sec): 84.2249 2024-09-13T15:04:25.191 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Stddev Bandwidth: 39.3396 2024-09-13T15:04:25.191 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Max bandwidth (MB/sec): 244.375 2024-09-13T15:04:25.192 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Min bandwidth (MB/sec): 0 2024-09-13T15:04:25.192 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Average IOPS: 1347 2024-09-13T15:04:25.192 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Stddev IOPS: 629.433 2024-09-13T15:04:25.192 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Max IOPS: 3910 2024-09-13T15:04:25.192 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Min IOPS: 0 2024-09-13T15:04:25.192 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Average Latency(s): 0.0118686 2024-09-13T15:04:25.192 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Stddev Latency(s): 0.0205481 2024-09-13T15:04:25.192 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Max latency(s): 1.27422 2024-09-13T15:04:25.192 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Min latency(s): 0.00120778 2024-09-13T15:04:25.200 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T15:04:25.270 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:25.305 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:25.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:25.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:25.406 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:25.445 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:25.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:25.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:25.547 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:25.579 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:25.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:25.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:25.593 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:25 smithi151 ceph-mon[104450]: pgmap v2301: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 7.1 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 59 MiB/s wr, 942 op/s; 9776/351456 objects degraded (2.782%); 26273/351456 objects misplaced (7.475%); 7.8 MiB/s, 124 objects/s recovering 2024-09-13T15:04:25.593 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:25 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 9776/351456 objects degraded (2.782%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:25.593 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:25 smithi151 ceph-mon[108966]: pgmap v2301: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 7.1 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 59 MiB/s wr, 942 op/s; 9776/351456 objects degraded (2.782%); 26273/351456 objects misplaced (7.475%); 7.8 MiB/s, 124 objects/s recovering 2024-09-13T15:04:25.593 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:25 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 9776/351456 objects degraded (2.782%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:25.681 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:25.722 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:25.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:25.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:25.824 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:25.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:25 smithi184 ceph-mon[92527]: pgmap v2301: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 7.1 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 59 MiB/s wr, 942 op/s; 9776/351456 objects degraded (2.782%); 26273/351456 objects misplaced (7.475%); 7.8 MiB/s, 124 objects/s recovering 2024-09-13T15:04:25.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:25 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 9776/351456 objects degraded (2.782%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:25.860 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:25.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:25.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:25.962 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:25.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:25.993 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:25.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:26.094 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:26.127 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:26.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:26.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:26.229 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:26.265 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:26.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:26.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:26.368 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:26.399 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:26.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:26.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:26.501 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:26.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:26.537 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:26.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:26.638 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:26.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:26.676 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:26.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:26.777 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:26.804 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:26.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:26.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:26.906 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:26.941 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:26.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:26.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:27.043 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:27.074 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:27.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:27.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:27.111 INFO:teuthology.orchestra.run.smithi151.stdout:moved 0 / 357 (0%) 2024-09-13T15:04:27.111 INFO:teuthology.orchestra.run.smithi151.stdout:avg 71.4 2024-09-13T15:04:27.111 INFO:teuthology.orchestra.run.smithi151.stdout:stddev 4.40908 -> 4.40908 (expected baseline 7.55778) 2024-09-13T15:04:27.111 INFO:teuthology.orchestra.run.smithi151.stdout:min osd.7 with 68 -> 68 pgs (0.952381 -> 0.952381 * mean) 2024-09-13T15:04:27.111 INFO:teuthology.orchestra.run.smithi151.stdout:max osd.1 with 80 -> 80 pgs (1.12045 -> 1.12045 * mean) 2024-09-13T15:04:27.111 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T15:04:27.112 INFO:teuthology.orchestra.run.smithi151.stdout:oload 1000 2024-09-13T15:04:27.112 INFO:teuthology.orchestra.run.smithi151.stdout:max_change 3 2024-09-13T15:04:27.112 INFO:teuthology.orchestra.run.smithi151.stdout:max_change_osds 4 2024-09-13T15:04:27.112 INFO:teuthology.orchestra.run.smithi151.stdout:average_utilization 0.0554 2024-09-13T15:04:27.112 INFO:teuthology.orchestra.run.smithi151.stdout:overload_utilization 0.5544 2024-09-13T15:04:27.112 INFO:teuthology.orchestra.run.smithi151.stderr:no change 2024-09-13T15:04:27.177 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:27.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:27.217 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:27.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:27.318 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:27.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:27.352 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:27.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:27.453 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:27.485 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:27.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:27.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:27.587 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:27.618 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:27.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:27.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:27.720 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:27.754 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:27.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:27.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:27.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:27 smithi184 ceph-mon[92527]: pgmap v2302: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 7.3 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 69 MiB/s wr, 1.10k op/s; 9776/357150 objects degraded (2.737%); 26586/357150 objects misplaced (7.444%); 7.3 MiB/s, 116 objects/s recovering 2024-09-13T15:04:27.856 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:27.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:27 smithi151 ceph-mon[104450]: pgmap v2302: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 7.3 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 69 MiB/s wr, 1.10k op/s; 9776/357150 objects degraded (2.737%); 26586/357150 objects misplaced (7.444%); 7.3 MiB/s, 116 objects/s recovering 2024-09-13T15:04:27.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:27 smithi151 ceph-mon[108966]: pgmap v2302: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 7.3 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 69 MiB/s wr, 1.10k op/s; 9776/357150 objects degraded (2.737%); 26586/357150 objects misplaced (7.444%); 7.3 MiB/s, 116 objects/s recovering 2024-09-13T15:04:27.896 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:27.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:27.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:27.998 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:28.035 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:28.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:28.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:28.137 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:28.171 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:28.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:28.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:28.273 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:28.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:28.306 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:28.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:28.407 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:28.441 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:28.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:28.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:28.543 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:28.577 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:28.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:28.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:28.587 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:28 smithi151 ceph-mon[104450]: from='client.46562 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:04:28.587 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:28 smithi151 ceph-mon[108966]: from='client.46562 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:04:28.679 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:28.711 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:28.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:28.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:28.813 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:28.839 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:28.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:28.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:28.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:28 smithi184 ceph-mon[92527]: from='client.46562 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:04:28.941 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:28.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:28.979 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:28.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:29.080 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:29.119 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:29.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:29.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:29.220 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:29.263 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:29.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:29.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:29.365 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:29.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:29.402 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:29.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:29.408 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-13T15:04:29.503 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:29.540 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:29.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:29.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:29.642 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:29.673 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:29.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:29.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:29.775 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:29.807 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:29.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:29.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:29.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:29 smithi184 ceph-mon[92527]: pgmap v2303: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 7.3 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 102 B/s rd, 50 MiB/s wr, 965 op/s; 9776/357582 objects degraded (2.734%); 26528/357582 objects misplaced (7.419%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T15:04:29.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:29 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/4112038649' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:29.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:29 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:29.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:29 smithi151 ceph-mon[104450]: pgmap v2303: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 7.3 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 102 B/s rd, 50 MiB/s wr, 965 op/s; 9776/357582 objects degraded (2.734%); 26528/357582 objects misplaced (7.419%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T15:04:29.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:29 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/4112038649' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:29.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:29 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:29.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:29 smithi151 ceph-mon[108966]: pgmap v2303: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 7.3 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 102 B/s rd, 50 MiB/s wr, 965 op/s; 9776/357582 objects degraded (2.734%); 26528/357582 objects misplaced (7.419%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T15:04:29.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:29 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/4112038649' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:29.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:29 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:29.909 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:29.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:29.943 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:29.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:30.044 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:30.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:30.075 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:30.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:30.176 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:30.214 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:30.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:30.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:30.316 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:30.350 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:30.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:30.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:30.452 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:30.485 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:30.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:30.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:30.587 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:30.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:30 smithi184 ceph-mon[92527]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:04:30.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:30 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:04:30.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:30 smithi184 ceph-mon[92527]: osdmap e1347: 8 total, 8 up, 5 in 2024-09-13T15:04:30.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:30 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 9776/357582 objects degraded (2.734%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:30.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:04:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:04:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:04:30.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:30.621 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:30.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:30.722 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:30.747 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:30.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:30.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:30.849 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:30.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:30 smithi151 ceph-mon[104450]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:04:30.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:30 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:04:30.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:30 smithi151 ceph-mon[104450]: osdmap e1347: 8 total, 8 up, 5 in 2024-09-13T15:04:30.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:30 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 9776/357582 objects degraded (2.734%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:30.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:30 smithi151 ceph-mon[108966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:04:30.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:30 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:04:30.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:30 smithi151 ceph-mon[108966]: osdmap e1347: 8 total, 8 up, 5 in 2024-09-13T15:04:30.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:30 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 9776/357582 objects degraded (2.734%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:30.885 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:30.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:30.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:30.986 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:31.021 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:31.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:31.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:31.123 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:31.153 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:31.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:31.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:31.255 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:31.287 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:31.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:31.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:31.389 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:31.426 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:31.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:31.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:31.528 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:31.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:31.573 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:31.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:31.674 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:31.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:31.711 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:31.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:31.772 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:31 smithi151 ceph-mon[104450]: pgmap v2305: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 7.2 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 102 B/s rd, 43 MiB/s wr, 1.02k op/s; 9776/354123 objects degraded (2.761%); 26324/354123 objects misplaced (7.434%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T15:04:31.773 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:31 smithi151 ceph-mon[108966]: pgmap v2305: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 7.2 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 102 B/s rd, 43 MiB/s wr, 1.02k op/s; 9776/354123 objects degraded (2.761%); 26324/354123 objects misplaced (7.434%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T15:04:31.812 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:31.939 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:31 smithi184 ceph-mon[92527]: pgmap v2305: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 7.2 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 102 B/s rd, 43 MiB/s wr, 1.02k op/s; 9776/354123 objects degraded (2.761%); 26324/354123 objects misplaced (7.434%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T15:04:31.939 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T15:04:31.939 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:04:31.969 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:31.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:31.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:32.071 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:32.109 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:32.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:32.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:32.210 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:32.241 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:32.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:32.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:32.343 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:32.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:32.383 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:32.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:32.484 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:32.525 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:32.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:32.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:32.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:04:32.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:32.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:04:32.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:32.627 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:32.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:32.673 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:32.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:32.774 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:32.804 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:32.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:32.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:04:32.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:32.905 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:32.936 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:32.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:32.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:33.038 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:33.071 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:33.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:33.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:33.173 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:33.208 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:33.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:33.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:33.242 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-13T15:04:33.242 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 0.05 2024-09-13T15:04:33.310 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:33.343 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:33.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:33.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:33.445 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:33.476 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:33.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:33.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:33.578 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:33.603 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:33.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:33.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:33.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:33 smithi151 ceph-mon[104450]: pgmap v2306: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 7.2 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 102 B/s rd, 43 MiB/s wr, 1.02k op/s; 9776/354123 objects degraded (2.761%); 26324/354123 objects misplaced (7.434%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T15:04:33.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:33 smithi151 ceph-mon[108966]: pgmap v2306: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 7.2 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 102 B/s rd, 43 MiB/s wr, 1.02k op/s; 9776/354123 objects degraded (2.761%); 26324/354123 objects misplaced (7.434%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T15:04:33.705 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:33.743 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:33.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:33.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:33.770 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T15:04:33.845 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:33.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:33 smithi184 ceph-mon[92527]: pgmap v2306: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 7.2 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 102 B/s rd, 43 MiB/s wr, 1.02k op/s; 9776/354123 objects degraded (2.761%); 26324/354123 objects misplaced (7.434%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T15:04:33.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:33.897 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:33.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:33.998 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:34.039 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:34.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:34.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:34.119 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T15:04:34.119 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:04:34.141 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:34.143 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:34.171 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T15:04:34.171 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:04:34.206 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T15:04:34.206 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:04:34.226 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:34.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:34.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:34.244 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T15:04:34.244 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:04:34.265 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T15:04:34.266 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:04:34.297 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T15:04:34.297 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:04:34.329 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T15:04:34.329 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:04:34.329 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:34.360 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:34.361 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T15:04:34.361 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:04:34.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:34.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:34.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:34.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:34.499 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:34.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:34.593 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:34 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/491051549' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:34.593 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:34 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/491051549' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:34.600 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:34.635 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:34.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:34.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:34.703 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T15:04:34.737 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:34.761 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:34.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:34.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:34.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:34 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/491051549' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:34.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:04:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:04:34] "GET /metrics HTTP/1.1" 200 36722 "" "Prometheus/2.51.0" 2024-09-13T15:04:34.863 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:34.887 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:34.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:34.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:34.989 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:35.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:35.033 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:35.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:35.134 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:35.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:35.185 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:35.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:35.286 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:35.315 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:35.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:35.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:35.417 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:35.448 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:35.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:35.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:35.550 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:35.586 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:35.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:35.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:35.688 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:35.719 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:35.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:35.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:35.820 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:35.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:35 smithi184 ceph-mon[92527]: pgmap v2307: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 7.0 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 102 B/s rd, 26 MiB/s wr, 1.11k op/s; 9776/343194 objects degraded (2.849%); 25493/343194 objects misplaced (7.428%); 1.4 MiB/s, 21 objects/s recovering 2024-09-13T15:04:35.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:35 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 9776/343194 objects degraded (2.849%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:35.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:35 smithi151 ceph-mon[104450]: pgmap v2307: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 7.0 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 102 B/s rd, 26 MiB/s wr, 1.11k op/s; 9776/343194 objects degraded (2.849%); 25493/343194 objects misplaced (7.428%); 1.4 MiB/s, 21 objects/s recovering 2024-09-13T15:04:35.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:35 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 9776/343194 objects degraded (2.849%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:35.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:35 smithi151 ceph-mon[108966]: pgmap v2307: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 7.0 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 102 B/s rd, 26 MiB/s wr, 1.11k op/s; 9776/343194 objects degraded (2.849%); 25493/343194 objects misplaced (7.428%); 1.4 MiB/s, 21 objects/s recovering 2024-09-13T15:04:35.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:35 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 9776/343194 objects degraded (2.849%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:35.861 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:35.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:35.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:35.963 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:35.996 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:35.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:35.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:36.097 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:36.136 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:36.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:36.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:36.240 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:36.267 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:36.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:36.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:36.369 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:36.409 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:36.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:36.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:36.511 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:36.533 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:36.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:36.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:36.635 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:36.672 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:36.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:36.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:36.777 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:36.814 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:36.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:36.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:36.916 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:36.952 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:36.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:36.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:37.054 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:37.085 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:37.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:37.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:37.187 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:37.226 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:37.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:37.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:37.328 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:37.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:37.360 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:37.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:37.461 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:37.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:37.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:37.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:37.596 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:37.632 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:37.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:37.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:37.734 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:37.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:37.772 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:37.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:37.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:37 smithi184 ceph-mon[92527]: pgmap v2308: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.9 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 102 B/s rd, 14 MiB/s wr, 1.13k op/s; 9776/336834 objects degraded (2.902%); 25162/336834 objects misplaced (7.470%); 1.4 MiB/s, 21 objects/s recovering 2024-09-13T15:04:37.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:37 smithi151 ceph-mon[104450]: pgmap v2308: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.9 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 102 B/s rd, 14 MiB/s wr, 1.13k op/s; 9776/336834 objects degraded (2.902%); 25162/336834 objects misplaced (7.470%); 1.4 MiB/s, 21 objects/s recovering 2024-09-13T15:04:37.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:37 smithi151 ceph-mon[108966]: pgmap v2308: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.9 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 102 B/s rd, 14 MiB/s wr, 1.13k op/s; 9776/336834 objects degraded (2.902%); 25162/336834 objects misplaced (7.470%); 1.4 MiB/s, 21 objects/s recovering 2024-09-13T15:04:37.872 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:37.906 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:37.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:37.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:38.008 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:38.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:38.041 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:38.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:38.142 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:38.175 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:38.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:38.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:38.277 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:38.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:38.311 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:38.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:38.412 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:38.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:38.446 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:38.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:38.547 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:38.586 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:38.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:38.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:38.687 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:38.719 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:38.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:38.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:38.821 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:38.850 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:38.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:38.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:38.952 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:38.987 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:38.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:38.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:39.089 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:39.122 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:39.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:39.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:39.224 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:39.259 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:39.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:39.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:39.361 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:39.382 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:39.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:39.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:39.484 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:39.513 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:39.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:39.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:39.615 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:39.642 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:39.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:39.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:39.744 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:39.778 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:39.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:39.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:39.807 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:39 smithi151 ceph-mon[108966]: pgmap v2309: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.7 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 3.1 MiB/s wr, 1.12k op/s; 9776/327072 objects degraded (2.989%); 24361/327072 objects misplaced (7.448%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T15:04:39.809 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:39 smithi151 ceph-mon[104450]: pgmap v2309: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.7 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 3.1 MiB/s wr, 1.12k op/s; 9776/327072 objects degraded (2.989%); 24361/327072 objects misplaced (7.448%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T15:04:39.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:39 smithi184 ceph-mon[92527]: pgmap v2309: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.7 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 3.1 MiB/s wr, 1.12k op/s; 9776/327072 objects degraded (2.989%); 24361/327072 objects misplaced (7.448%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T15:04:39.880 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:39.901 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:39.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:39.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:40.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:40.034 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:40.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:40.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:40.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:40.172 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:40.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:40.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:40.274 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:40.295 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:40.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:40.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:40.397 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:40.430 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:40.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:40.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:40.482 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:04:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:04:40.215Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:04:40.532 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:40.566 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:40.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:40.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:40.667 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:40.695 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:40.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:40.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:40.797 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:40.836 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:40.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:40.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:40.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:40 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 9776/327072 objects degraded (2.989%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:40.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:40 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 9776/327072 objects degraded (2.989%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:40.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:40 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 9776/327072 objects degraded (2.989%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:04:40.938 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:40.979 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:40.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:40.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:41.081 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:41.121 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:41.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:41.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:41.223 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:41.253 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:41.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:41.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:41.355 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:41.376 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:41.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:41.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:41.478 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:41.516 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:41.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:41.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:41.618 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:41.653 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:41.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:41.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:41.754 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:41.786 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:41.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:41.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:41.814 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:41 smithi151 ceph-mon[104450]: pgmap v2310: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.5 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 0 B/s wr, 1.04k op/s; 9776/321366 objects degraded (3.042%); 24019/321366 objects misplaced (7.474%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T15:04:41.815 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:41 smithi151 ceph-mon[108966]: pgmap v2310: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.5 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 0 B/s wr, 1.04k op/s; 9776/321366 objects degraded (3.042%); 24019/321366 objects misplaced (7.474%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T15:04:41.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:41 smithi184 ceph-mon[92527]: pgmap v2310: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.5 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 0 B/s wr, 1.04k op/s; 9776/321366 objects degraded (3.042%); 24019/321366 objects misplaced (7.474%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T15:04:41.888 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:41.925 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:41.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:41.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:42.027 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:42.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:42.057 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:42.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:42.158 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:42.187 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:42.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:42.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:42.289 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:42.321 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:42.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:42.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:42.352 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:42.423 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:42.485 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:42.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:42.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:42.587 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:42.620 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:42.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:42.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:42.722 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:42.756 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:42.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:42.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:42.770 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:42 smithi151 ceph-mon[104450]: pgmap v2311: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.5 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 0 B/s wr, 909 op/s; 9776/321366 objects degraded (3.042%); 24019/321366 objects misplaced (7.474%); 1.2 MiB/s, 18 objects/s recovering 2024-09-13T15:04:42.770 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:42 smithi151 ceph-mon[108966]: pgmap v2311: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.5 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 0 B/s wr, 909 op/s; 9776/321366 objects degraded (3.042%); 24019/321366 objects misplaced (7.474%); 1.2 MiB/s, 18 objects/s recovering 2024-09-13T15:04:42.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:42 smithi184 ceph-mon[92527]: pgmap v2311: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.5 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 0 B/s wr, 909 op/s; 9776/321366 objects degraded (3.042%); 24019/321366 objects misplaced (7.474%); 1.2 MiB/s, 18 objects/s recovering 2024-09-13T15:04:42.858 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:42.893 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:42.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:42.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:42.995 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:43.022 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:43.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:43.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:43.088 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:43.124 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:43.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:43.202 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:43.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:43.303 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:43.354 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:43.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:43.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:43.455 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:43.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:43.488 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:43.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:43.589 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:43.621 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:43.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:43.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:43.723 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:43.752 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:43.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:43.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:43.854 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:43.855 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:43 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:04:43.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:43 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:04:43.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:43 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:04:43.863 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:04:43.886 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:43.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:43.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:43.988 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:44.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:44.040 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:44.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:44.141 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:44.168 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:44.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:44.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:44.270 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:44.312 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:44.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:44.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:44.414 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:44.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:44.453 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:44.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:44.554 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:44.589 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:44.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:44.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:44.691 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:44.729 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:44.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:44.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:44.831 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:44.836 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:44 smithi184 ceph-mon[92527]: pgmap v2312: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s wr, 1.20k op/s; 9776/310989 objects degraded (3.144%); 23222/310989 objects misplaced (7.467%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T15:04:44.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:44 smithi151 ceph-mon[104450]: pgmap v2312: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s wr, 1.20k op/s; 9776/310989 objects degraded (3.144%); 23222/310989 objects misplaced (7.467%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T15:04:44.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:04:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:04:44] "GET /metrics HTTP/1.1" 200 36722 "" "Prometheus/2.51.0" 2024-09-13T15:04:44.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:44 smithi151 ceph-mon[108966]: pgmap v2312: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s wr, 1.20k op/s; 9776/310989 objects degraded (3.144%); 23222/310989 objects misplaced (7.467%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T15:04:44.865 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:44.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:44.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:44.967 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:45.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:45.002 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:45.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:45.103 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:45.136 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:45.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:45.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:45.239 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:45.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:45.272 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:45.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:45.373 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:45.421 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:45.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:45.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:45.523 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:45.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:45.557 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:45.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:45.659 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:45.702 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:45.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:45.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:45.741 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:45 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 9776/310989 objects degraded (3.144%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:04:45.741 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:45 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 9776/310989 objects degraded (3.144%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:04:45.804 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:45.834 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:45.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:45.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:45.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:45 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 9776/310989 objects degraded (3.144%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:04:45.936 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:45.975 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:45.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:45.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:46.077 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:46.108 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:46.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:46.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:46.210 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:46.247 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:46.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:46.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:46.349 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:46.379 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:46.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:46.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:46.406 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T15:04:46.406 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-13T15:04:46.406 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-13T15:04:46.406 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-13T15:04:46.406 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T15:04:46.481 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:46.515 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:46.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:46.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:46.562 INFO:teuthology.orchestra.run.smithi151.stdout:moved 0 / 357 (0%) 2024-09-13T15:04:46.562 INFO:teuthology.orchestra.run.smithi151.stdout:avg 71.4 2024-09-13T15:04:46.562 INFO:teuthology.orchestra.run.smithi151.stdout:stddev 4.40908 -> 4.40908 (expected baseline 7.55778) 2024-09-13T15:04:46.563 INFO:teuthology.orchestra.run.smithi151.stdout:min osd.7 with 68 -> 68 pgs (0.952381 -> 0.952381 * mean) 2024-09-13T15:04:46.563 INFO:teuthology.orchestra.run.smithi151.stdout:max osd.1 with 80 -> 80 pgs (1.12045 -> 1.12045 * mean) 2024-09-13T15:04:46.563 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T15:04:46.563 INFO:teuthology.orchestra.run.smithi151.stdout:oload 1000 2024-09-13T15:04:46.563 INFO:teuthology.orchestra.run.smithi151.stdout:max_change 0.05 2024-09-13T15:04:46.563 INFO:teuthology.orchestra.run.smithi151.stdout:max_change_osds 4 2024-09-13T15:04:46.563 INFO:teuthology.orchestra.run.smithi151.stdout:average_utilization 0.0499 2024-09-13T15:04:46.563 INFO:teuthology.orchestra.run.smithi151.stdout:overload_utilization 0.4988 2024-09-13T15:04:46.563 INFO:teuthology.orchestra.run.smithi151.stderr:no change 2024-09-13T15:04:46.617 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:46.649 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:46.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:46.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:46.751 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:46.775 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:46.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:46.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:46.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:46 smithi184 ceph-mon[92527]: pgmap v2313: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.2 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s wr, 1.06k op/s; 9776/304929 objects degraded (3.206%); 22879/304929 objects misplaced (7.503%); 1.2 MiB/s, 18 objects/s recovering 2024-09-13T15:04:46.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:46 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:46.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:46 smithi151 ceph-mon[104450]: pgmap v2313: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.2 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s wr, 1.06k op/s; 9776/304929 objects degraded (3.206%); 22879/304929 objects misplaced (7.503%); 1.2 MiB/s, 18 objects/s recovering 2024-09-13T15:04:46.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:46 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:46 smithi151 ceph-mon[108966]: pgmap v2313: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.2 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s wr, 1.06k op/s; 9776/304929 objects degraded (3.206%); 22879/304929 objects misplaced (7.503%); 1.2 MiB/s, 18 objects/s recovering 2024-09-13T15:04:46.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:46 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:46.877 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:46.914 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:46.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:46.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:47.016 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:47.049 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:47.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:47.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:47.151 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:47.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:47.185 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:47.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:47.287 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:47.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:47.326 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:47.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:47.427 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:47.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:47.466 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:47.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:47.566 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:47.603 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:47.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:47.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:47.705 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:47.743 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:47.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:47.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:47.830 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:47 smithi184 ceph-mon[92527]: from='client.36744 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:04:47.845 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:47.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:47 smithi151 ceph-mon[104450]: from='client.36744 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:04:47.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:47 smithi151 ceph-mon[108966]: from='client.36744 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:04:47.885 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:47.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:47.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:47.973 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T15:04:47.976 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:47.987 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:48.012 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:48.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:48.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:48.114 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:48.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:48.164 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:48.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:48.265 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:48.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:48.305 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:48.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:48.406 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:48.434 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:48.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:48.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:48.485 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-13T15:04:48.485 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight-by-utilization 110 1.0 2024-09-13T15:04:48.536 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:48.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:48.577 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:48.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:48.679 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:48.709 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:48.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:48.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:48.736 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:48 smithi151 ceph-mon[104450]: pgmap v2314: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.0 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s wr, 1.17k op/s; 9946/294591 objects degraded (3.376%); 21893/294591 objects misplaced (7.432%); 1.9 MiB/s, 29 objects/s recovering 2024-09-13T15:04:48.737 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:48 smithi151 ceph-mon[108966]: pgmap v2314: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.0 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s wr, 1.17k op/s; 9946/294591 objects degraded (3.376%); 21893/294591 objects misplaced (7.432%); 1.9 MiB/s, 29 objects/s recovering 2024-09-13T15:04:48.811 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:48.847 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:48 smithi184 ceph-mon[92527]: pgmap v2314: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 6.0 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s wr, 1.17k op/s; 9946/294591 objects degraded (3.376%); 21893/294591 objects misplaced (7.432%); 1.9 MiB/s, 29 objects/s recovering 2024-09-13T15:04:48.853 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:48.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:48.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:48.955 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:48.995 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:48.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:48.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:49.097 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:49.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:49.129 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:49.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:49.230 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:49.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:49.265 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:49.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:49.366 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:49.401 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:49.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:49.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:49.502 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:49.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:49.534 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:49.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:49.635 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:49.673 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:49.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:49.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:49.775 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:49.809 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:49.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:49.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:49.911 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:49.946 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:49.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:49.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:50.048 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:50.085 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:50.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:50.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:50.186 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:50.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:50.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:50.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:50.311 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:50.343 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:50.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:50.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:50.445 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:50.474 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:50.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:50.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:50.576 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:50.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:50 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:04:50.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:50 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:04:50.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:50 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 9946/294591 objects degraded (3.376%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:04:50.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:04:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:04:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:04:50.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:50.610 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:50.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:50.711 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:50.737 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:50 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:04:50.737 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:50 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:04:50.737 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:50 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 9946/294591 objects degraded (3.376%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:04:50.738 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:50 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:04:50.738 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:50 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:04:50.738 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:50 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 9946/294591 objects degraded (3.376%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:04:50.742 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:50.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:50.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:50.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:50.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:50.881 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:50.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:50.982 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:51.016 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:51.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:51.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:51.118 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:51.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:51.151 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:51.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:51.252 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:51.280 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:51.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:51.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:51.382 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:51.419 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:51.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:51.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:51.520 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:51.554 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:51.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:51.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:51.656 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:51.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:51.691 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:51.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:51.792 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:51.823 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:51.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:51.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:51.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:51 smithi184 ceph-mon[92527]: pgmap v2315: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.9 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 0 B/s wr, 1.08k op/s; 9946/288303 objects degraded (3.450%); 21507/288303 objects misplaced (7.460%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T15:04:51.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:51 smithi151 ceph-mon[104450]: pgmap v2315: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.9 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 0 B/s wr, 1.08k op/s; 9946/288303 objects degraded (3.450%); 21507/288303 objects misplaced (7.460%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T15:04:51.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:51 smithi151 ceph-mon[108966]: pgmap v2315: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.9 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 0 B/s wr, 1.08k op/s; 9946/288303 objects degraded (3.450%); 21507/288303 objects misplaced (7.460%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T15:04:51.924 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:51.947 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:51.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:51.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:52.049 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:52.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:52.082 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:52.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:52.183 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:52.218 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:52.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:52.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:52.320 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:52.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:52.356 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:52.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:52.457 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:52.478 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:52.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:52.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:52.580 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:52.601 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:52.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:52.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:52.703 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:52.731 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:52.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:52.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:52.833 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:52.872 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:52.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:52.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:52.974 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:52.999 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:53.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:53.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:53.101 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:53.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:53.130 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:53.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:53.231 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:53.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:53.270 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:53.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:53.371 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:53.407 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:53.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:53.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:53.509 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:53.541 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:53.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:53.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:53.643 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:53.677 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:53.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:53.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:53.779 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:53.812 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:53.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:53.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:53.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:53 smithi184 ceph-mon[92527]: pgmap v2316: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.9 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 0 B/s wr, 918 op/s; 9946/288303 objects degraded (3.450%); 21507/288303 objects misplaced (7.460%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T15:04:53.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:53 smithi151 ceph-mon[104450]: pgmap v2316: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.9 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 0 B/s wr, 918 op/s; 9946/288303 objects degraded (3.450%); 21507/288303 objects misplaced (7.460%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T15:04:53.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:53 smithi151 ceph-mon[108966]: pgmap v2316: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.9 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 0 B/s wr, 918 op/s; 9946/288303 objects degraded (3.450%); 21507/288303 objects misplaced (7.460%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T15:04:53.914 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:53.949 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:53.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:53.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:54.051 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:54.075 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:54.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:54.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:54.177 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:54.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:54.216 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:54.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:54.317 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:54.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:54.354 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:54.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:54.455 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:54.487 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:54.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:54.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:54.589 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:54.618 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:54.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:54.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:54.720 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:54.750 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:54.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:54.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:54.852 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:54.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:04:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:04:54] "GET /metrics HTTP/1.1" 200 36725 "" "Prometheus/2.51.0" 2024-09-13T15:04:54.887 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:54.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:54.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:54.989 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:55.054 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:55.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:55.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:55.156 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:55.189 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:55.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:55.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:55.290 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:55.323 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:55.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:55.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:55.425 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:55.460 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:55.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:55.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:55.481 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-13T15:04:55.563 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:55.594 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:55.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:55.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:55.696 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:55.730 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:55.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:55.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:55.831 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:55.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:55 smithi184 ceph-mon[92527]: pgmap v2317: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.7 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 0 B/s wr, 1.19k op/s; 10141/278331 objects degraded (3.644%); 20491/278331 objects misplaced (7.362%); 2.0 MiB/s, 31 objects/s recovering 2024-09-13T15:04:55.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:55 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 10141/278331 objects degraded (3.644%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:04:55.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:55 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1078334698' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:04:55.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:55 smithi151 ceph-mon[104450]: pgmap v2317: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.7 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 0 B/s wr, 1.19k op/s; 10141/278331 objects degraded (3.644%); 20491/278331 objects misplaced (7.362%); 2.0 MiB/s, 31 objects/s recovering 2024-09-13T15:04:55.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:55 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 10141/278331 objects degraded (3.644%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:04:55.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:55 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1078334698' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:04:55.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:55 smithi151 ceph-mon[108966]: pgmap v2317: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.7 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 0 B/s wr, 1.19k op/s; 10141/278331 objects degraded (3.644%); 20491/278331 objects misplaced (7.362%); 2.0 MiB/s, 31 objects/s recovering 2024-09-13T15:04:55.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:55 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 10141/278331 objects degraded (3.644%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:04:55.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:55 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1078334698' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:04:55.865 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:55.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:55.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:55.966 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:56.004 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:56.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:56.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:56.105 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:56.146 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:56.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:56.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:56.248 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:56.280 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:56.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:56.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:56.382 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:56.412 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:56.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:56.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:56.514 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:56.542 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:56.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:56.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:56.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:56 smithi151 ceph-mon[104450]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:04:56.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:56 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1078334698' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:04:56.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:56 smithi151 ceph-mon[104450]: osdmap e1348: 8 total, 8 up, 5 in 2024-09-13T15:04:56.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:56 smithi151 ceph-mon[108966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:04:56.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:56 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1078334698' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:04:56.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:56 smithi151 ceph-mon[108966]: osdmap e1348: 8 total, 8 up, 5 in 2024-09-13T15:04:56.644 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:56.677 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:56.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:56.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:56.779 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:56.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:56.833 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:56.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:56.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:56 smithi184 ceph-mon[92527]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:04:56.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:56 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1078334698' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:04:56.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:56 smithi184 ceph-mon[92527]: osdmap e1348: 8 total, 8 up, 5 in 2024-09-13T15:04:56.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:56.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:56.979 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:56.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:57.080 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:57.117 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:57.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:57.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:57.218 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:57.263 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:57.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:57.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:57.365 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:57.404 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:57.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:57.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:57.596 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T15:04:57.596 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:04:57.597 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T15:04:57.597 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:04:57.597 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T15:04:57.598 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:04:57.598 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T15:04:57.598 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:04:57.599 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T15:04:57.599 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:04:57.609 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T15:04:57.609 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:04:57.638 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T15:04:57.638 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:04:57.664 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T15:04:57.665 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:04:57.665 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:57 smithi151 ceph-mon[104450]: pgmap v2319: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.5 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 0 B/s wr, 1.08k op/s; 10141/272592 objects degraded (3.720%); 20187/272592 objects misplaced (7.406%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T15:04:57.665 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:57 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2894221546' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:57.665 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:57 smithi151 ceph-mon[108966]: pgmap v2319: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.5 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 0 B/s wr, 1.08k op/s; 10141/272592 objects degraded (3.720%); 20187/272592 objects misplaced (7.406%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T15:04:57.665 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:57 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2894221546' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:57.697 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:57.718 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:57.727 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:57.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:57.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:57.829 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:57.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:57 smithi184 ceph-mon[92527]: pgmap v2319: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.5 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 0 B/s wr, 1.08k op/s; 10141/272592 objects degraded (3.720%); 20187/272592 objects misplaced (7.406%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T15:04:57.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:57 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2894221546' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:57.864 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:57.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:57.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:57.966 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:57.994 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:57.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:57.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:58.096 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:58.128 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:58.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:58.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:58.230 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:58.264 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:58.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:58.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:58.364 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T15:04:58.366 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:58.399 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:58.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:58.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:58.456 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:58.501 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:58.551 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:58.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:58.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:58.653 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:58.682 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:58.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:58.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:58.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:04:58 smithi151 ceph-mon[104450]: pgmap v2320: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.3 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 0 B/s wr, 1.07k op/s; 10352/262356 objects degraded (3.946%); 19144/262356 objects misplaced (7.297%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T15:04:58.694 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:04:58 smithi151 ceph-mon[108966]: pgmap v2320: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.3 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 0 B/s wr, 1.07k op/s; 10352/262356 objects degraded (3.946%); 19144/262356 objects misplaced (7.297%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T15:04:58.783 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:58.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:58.827 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:58.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:58.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:04:58 smithi184 ceph-mon[92527]: pgmap v2320: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.3 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 0 B/s wr, 1.07k op/s; 10352/262356 objects degraded (3.946%); 19144/262356 objects misplaced (7.297%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T15:04:58.928 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:58.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:58.974 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:58.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:59.075 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:59.107 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:59.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:59.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:59.209 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:59.245 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:59.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:59.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:59.346 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:59.381 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:59.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:59.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:59.482 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:59.516 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:59.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:59.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:59.618 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:59.653 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:04:59.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:59.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:59.754 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:59.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:59.795 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:59.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:59.896 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:59.930 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:04:59.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:59.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:00.032 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:00.065 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:00.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:00.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:00.167 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:00.204 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:00.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:00.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:00.306 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:00.345 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:00.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:00.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:00.447 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:05:00.447 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:00.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:00.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:00.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:00.596 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:00.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:05:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:05:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:05:00.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:00.638 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:00.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:00.740 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:00.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:00.772 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:00.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:00.874 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:00.909 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:00.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:00.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:01.010 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:01.038 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:01.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:01.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:01.140 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:01.173 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:01.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:01.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:01.275 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:01.304 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:01.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:01.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:01.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:00 smithi184 ceph-mon[92527]: pgmap v2321: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.2 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 0 B/s wr, 1.08k op/s; 10352/255837 objects degraded (4.046%); 18775/255837 objects misplaced (7.339%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T15:05:01.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:00 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 10352/255837 objects degraded (4.046%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:01.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:00 smithi151 ceph-mon[104450]: pgmap v2321: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.2 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 0 B/s wr, 1.08k op/s; 10352/255837 objects degraded (4.046%); 18775/255837 objects misplaced (7.339%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T15:05:01.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:00 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 10352/255837 objects degraded (4.046%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:01.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:00 smithi151 ceph-mon[108966]: pgmap v2321: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.2 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 0 B/s wr, 1.08k op/s; 10352/255837 objects degraded (4.046%); 18775/255837 objects misplaced (7.339%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T15:05:01.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:00 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 10352/255837 objects degraded (4.046%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:01.406 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:01.431 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:01.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:01.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:01.533 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:01.559 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:01.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:01.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:01.661 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:01.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:01.706 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:01.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:01.807 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:01.844 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:01.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:01.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:01.946 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:01.985 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:01.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:01.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:02.087 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:02.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:02.118 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:02.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:02.219 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:02.254 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:02.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:02.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:02.355 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:02.378 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:02.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:02.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:02.480 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:02.518 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:02.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:02.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:02.620 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:02.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:02.661 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:02.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:02.713 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:05:02.713 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:02.713 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:05:02.713 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:02.763 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:02.786 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:02.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:02.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:02.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:05:02.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:02.888 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:02.920 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:02.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:02.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:03.022 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:03.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:03.063 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:03.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:03.163 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:03.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:03.194 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:03.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:03.295 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:03.328 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:03.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:03.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:03.430 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:03.463 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:03.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:03.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:03.565 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:03.601 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:03.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:03.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:03.703 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:03.715 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:03 smithi151 ceph-mon[104450]: pgmap v2322: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.2 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 0 B/s wr, 1.08k op/s; 10352/255837 objects degraded (4.046%); 18775/255837 objects misplaced (7.339%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T15:05:03.715 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:03 smithi151 ceph-mon[108966]: pgmap v2322: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.2 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 0 B/s wr, 1.08k op/s; 10352/255837 objects degraded (4.046%); 18775/255837 objects misplaced (7.339%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T15:05:03.743 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:03.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:03.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:03.845 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:03.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:03 smithi184 ceph-mon[92527]: pgmap v2322: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.2 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 0 B/s wr, 1.08k op/s; 10352/255837 objects degraded (4.046%); 18775/255837 objects misplaced (7.339%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T15:05:03.883 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:03.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:03.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:03.985 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:04.016 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:04.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:04.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:04.117 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:04.146 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:04.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:04.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:04.248 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:04.282 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:04.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:04.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:04.384 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:04.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:04.420 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:04.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:04.521 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:04.554 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:04.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:04.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:04.630 INFO:teuthology.orchestra.run.smithi151.stdout:moved 0 / 357 (0%) 2024-09-13T15:05:04.630 INFO:teuthology.orchestra.run.smithi151.stdout:avg 71.4 2024-09-13T15:05:04.630 INFO:teuthology.orchestra.run.smithi151.stdout:stddev 4.40908 -> 4.40908 (expected baseline 7.55778) 2024-09-13T15:05:04.630 INFO:teuthology.orchestra.run.smithi151.stdout:min osd.7 with 68 -> 68 pgs (0.952381 -> 0.952381 * mean) 2024-09-13T15:05:04.630 INFO:teuthology.orchestra.run.smithi151.stdout:max osd.1 with 80 -> 80 pgs (1.12045 -> 1.12045 * mean) 2024-09-13T15:05:04.630 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T15:05:04.630 INFO:teuthology.orchestra.run.smithi151.stdout:oload 110 2024-09-13T15:05:04.630 INFO:teuthology.orchestra.run.smithi151.stdout:max_change 1 2024-09-13T15:05:04.630 INFO:teuthology.orchestra.run.smithi151.stdout:max_change_osds 4 2024-09-13T15:05:04.630 INFO:teuthology.orchestra.run.smithi151.stdout:average_utilization 0.0438 2024-09-13T15:05:04.630 INFO:teuthology.orchestra.run.smithi151.stdout:overload_utilization 0.0482 2024-09-13T15:05:04.631 INFO:teuthology.orchestra.run.smithi151.stderr:no change 2024-09-13T15:05:04.657 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:04.693 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:04.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:04.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:04.795 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:04.825 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:04.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:04.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:04.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:05:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:05:04] "GET /metrics HTTP/1.1" 200 36722 "" "Prometheus/2.51.0" 2024-09-13T15:05:04.927 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:04.958 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:04.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:04.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:05.060 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:05.088 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:05.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:05.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:05.190 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:05.224 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:05.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:05.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:05.326 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:05.359 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:05.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:05.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:05.461 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:05.495 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:05.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:05.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:05.597 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:05.628 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:05.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:05.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:05.730 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:05.764 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:05.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:05.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:05.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:05 smithi184 ceph-mon[92527]: pgmap v2323: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.0 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 0 B/s wr, 1.11k op/s; 10536/245079 objects degraded (4.299%); 17701/245079 objects misplaced (7.223%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T15:05:05.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:05 smithi184 ceph-mon[92527]: from='client.36759 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:05:05.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:05 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 10536/245079 objects degraded (4.299%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:05.858 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:05 smithi151 ceph-mon[108966]: pgmap v2323: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.0 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 0 B/s wr, 1.11k op/s; 10536/245079 objects degraded (4.299%); 17701/245079 objects misplaced (7.223%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T15:05:05.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:05 smithi151 ceph-mon[108966]: from='client.36759 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:05:05.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:05 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 10536/245079 objects degraded (4.299%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:05.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:05 smithi151 ceph-mon[104450]: pgmap v2323: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 5.0 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 0 B/s wr, 1.11k op/s; 10536/245079 objects degraded (4.299%); 17701/245079 objects misplaced (7.223%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T15:05:05.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:05 smithi151 ceph-mon[104450]: from='client.36759 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:05:05.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:05 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 10536/245079 objects degraded (4.299%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:05.866 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:05.897 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:05.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:05.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:05.906 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:06.000 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:06.033 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:06.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:06.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:06.135 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:06.169 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:06.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:06.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:06.271 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:06.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:06.308 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:06.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:06.409 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:06.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:06.440 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:06.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:06.540 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:06.574 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:06.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:06.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:06.641 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-13T15:05:06.641 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd test-reweight-by-utilization 110 3.0 2024-09-13T15:05:06.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:06.729 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:06.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:06.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:06.831 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:06.854 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:06.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:06.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:06.956 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:07.010 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:07.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:07.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:07.112 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:07.133 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:07.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:07.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:07.235 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:07.276 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:07.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:07.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:07.378 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:07.401 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:07.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:07.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:07.503 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:07.538 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:07.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:07.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:07.640 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:07.685 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:07.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:07.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:07.746 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:07 smithi151 ceph-mon[104450]: pgmap v2324: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.9 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 0 B/s wr, 1.08k op/s; 10536/238818 objects degraded (4.412%); 17354/238818 objects misplaced (7.267%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T15:05:07.747 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:07 smithi151 ceph-mon[108966]: pgmap v2324: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.9 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 0 B/s wr, 1.08k op/s; 10536/238818 objects degraded (4.412%); 17354/238818 objects misplaced (7.267%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T15:05:07.787 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:07.823 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:07.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:07.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:07.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:07 smithi184 ceph-mon[92527]: pgmap v2324: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.9 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 0 B/s wr, 1.08k op/s; 10536/238818 objects degraded (4.412%); 17354/238818 objects misplaced (7.267%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T15:05:07.926 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:07.960 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:07.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:07.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:08.062 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:08.091 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:08.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:08.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:08.193 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:08.216 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:08.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:08.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:08.318 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:08.348 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:08.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:08.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:08.450 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:08.479 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:08.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:08.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:08.581 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:08.610 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:08.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:08.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:08.712 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:08.740 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:08.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:08.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:08.842 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:08.871 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:08.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:08.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:08.972 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:09.011 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:09.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:09.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:09.112 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:09.144 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:09.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:09.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:09.246 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:09.276 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:09.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:09.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:09.378 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:09.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:09.418 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:09.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:09.519 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:09.560 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:09.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:09.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:09.662 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:09.692 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:09.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:09.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:09.794 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:09.826 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:09 smithi151 ceph-mon[104450]: pgmap v2325: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.6 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 0 B/s wr, 1.22k op/s; 10691/228606 objects degraded (4.677%); 16345/228606 objects misplaced (7.150%); 2.5 MiB/s, 39 objects/s recovering 2024-09-13T15:05:09.826 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:09 smithi151 ceph-mon[108966]: pgmap v2325: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.6 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 0 B/s wr, 1.22k op/s; 10691/228606 objects degraded (4.677%); 16345/228606 objects misplaced (7.150%); 2.5 MiB/s, 39 objects/s recovering 2024-09-13T15:05:09.827 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:09.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:09.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:09.841 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:09.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:09 smithi184 ceph-mon[92527]: pgmap v2325: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.6 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 0 B/s wr, 1.22k op/s; 10691/228606 objects degraded (4.677%); 16345/228606 objects misplaced (7.150%); 2.5 MiB/s, 39 objects/s recovering 2024-09-13T15:05:09.929 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:09.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:09.974 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:09.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:10.075 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:10.101 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:10.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:10.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:10.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:10.249 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:10.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:10.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:10.351 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:10.383 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:10.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:10.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:10.485 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:10.518 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:10.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:10.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:10.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:10 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 10691/228606 objects degraded (4.677%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:10.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:05:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:05:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:05:10.619 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:10.673 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:10.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:10.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:10.760 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:10 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 10691/228606 objects degraded (4.677%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:10.760 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:10 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 10691/228606 objects degraded (4.677%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:10.776 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:10.810 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:10.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:10.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:10.912 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:10.944 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:10.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:10.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:11.046 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:11.082 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:11.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:11.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:11.184 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:11.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:11.223 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:11.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:11.324 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:11.366 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:11.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:11.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:11.468 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:11.498 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:11.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:11.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:11.596 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T15:05:11.596 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:05:11.600 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:11.634 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:11.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:11.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:11.736 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:11.768 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:11.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:11.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:11.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:11 smithi184 ceph-mon[92527]: pgmap v2326: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.5 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 1.11k op/s; 10691/222195 objects degraded (4.812%); 15993/222195 objects misplaced (7.198%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T15:05:11.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:11 smithi151 ceph-mon[104450]: pgmap v2326: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.5 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 1.11k op/s; 10691/222195 objects degraded (4.812%); 15993/222195 objects misplaced (7.198%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T15:05:11.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:11 smithi151 ceph-mon[108966]: pgmap v2326: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.5 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 1.11k op/s; 10691/222195 objects degraded (4.812%); 15993/222195 objects misplaced (7.198%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T15:05:11.869 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:11.901 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:11.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:11.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:12.003 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:12.034 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:12.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:12.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:12.136 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:12.172 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:12.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:12.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:12.274 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:12.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:12.318 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:12.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:12.419 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:12.455 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:12.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:12.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:12.557 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:12.586 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:12.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:12.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:12.688 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:12.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:12.721 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:12.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:12.822 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:12.835 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T15:05:12.851 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:12.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:12.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:12.953 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:12.979 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:12.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:12.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:13.080 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:13.111 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:13.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:13.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:13.213 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:13.244 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:13.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:13.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:13.346 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:13.378 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:13.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:13.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:13.480 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:13.514 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:13.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:13.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:13.616 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:13.644 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:13.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:13.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:13.746 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:13.751 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:13 smithi151 ceph-mon[104450]: pgmap v2327: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.5 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 933 op/s; 10691/222195 objects degraded (4.812%); 15993/222195 objects misplaced (7.198%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T15:05:13.752 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:13 smithi151 ceph-mon[108966]: pgmap v2327: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.5 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 933 op/s; 10691/222195 objects degraded (4.812%); 15993/222195 objects misplaced (7.198%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T15:05:13.778 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:13.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:13.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:13.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:13 smithi184 ceph-mon[92527]: pgmap v2327: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.5 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 933 op/s; 10691/222195 objects degraded (4.812%); 15993/222195 objects misplaced (7.198%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T15:05:13.879 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:13.907 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:13.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:13.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:14.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:14.034 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:14.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:14.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:14.135 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:14.172 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:14.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:14.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:14.276 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:14.332 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:14.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:14.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:14.434 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:14.472 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:14.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:14.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:14.577 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:14.607 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:14.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:14.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:14.709 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:14.753 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:14.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:14.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:14.855 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:14.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:05:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:05:14] "GET /metrics HTTP/1.1" 200 36722 "" "Prometheus/2.51.0" 2024-09-13T15:05:14.885 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:14.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:14.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:14.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:15.021 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:15.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:15.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:15.123 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:15.155 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:15.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:15.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:15.257 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:15.295 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:15.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:15.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:15.397 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:15.427 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:15.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:15.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:15.529 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:15.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:15.564 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:15.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:15.664 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:15.695 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:15.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:15.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:15.797 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:15.829 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:15.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:15.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:15.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:15 smithi184 ceph-mon[92527]: pgmap v2328: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.3 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 1.24k op/s; 10895/211182 objects degraded (5.159%); 14922/211182 objects misplaced (7.066%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T15:05:15.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:15 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 10895/211182 objects degraded (5.159%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:15.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:15 smithi151 ceph-mon[104450]: pgmap v2328: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.3 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 1.24k op/s; 10895/211182 objects degraded (5.159%); 14922/211182 objects misplaced (7.066%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T15:05:15.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:15 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 10895/211182 objects degraded (5.159%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:15.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:15 smithi151 ceph-mon[108966]: pgmap v2328: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.3 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 1.24k op/s; 10895/211182 objects degraded (5.159%); 14922/211182 objects misplaced (7.066%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T15:05:15.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:15 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 10895/211182 objects degraded (5.159%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:15.932 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:15.968 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:15.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:15.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:16.070 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:16.101 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:16.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:16.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:16.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:16.230 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:16.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:16.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:16.331 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:16.365 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:16.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:16.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:16.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:16.504 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:16.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:16.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:16.606 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:16.641 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:16.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:16.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:16.652 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-13T15:05:16.672 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:16 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3782197085' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:05:16.672 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:16 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:05:16.672 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:16.672 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:16 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3782197085' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:05:16.672 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:16 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:05:16.672 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:16.744 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:16.782 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:16.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:16.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:16.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:16 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3782197085' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:05:16.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:16 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:05:16.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:16.885 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:16.919 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:16.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:16.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:17.021 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:17.054 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:17.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:17.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:17.155 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:17.185 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:17.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:17.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:17.286 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:17.323 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:17.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:17.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:17.425 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:17.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:17.458 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:17.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:17.558 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:17.597 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:17.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:17.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:17.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:17 smithi151 ceph-mon[104450]: pgmap v2329: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.2 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 1.12k op/s; 10895/204579 objects degraded (5.326%); 14528/204579 objects misplaced (7.101%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T15:05:17.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:17 smithi151 ceph-mon[104450]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:05:17.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:17 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:05:17.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:17 smithi151 ceph-mon[104450]: osdmap e1349: 8 total, 8 up, 5 in 2024-09-13T15:05:17.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:17 smithi151 ceph-mon[108966]: pgmap v2329: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.2 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 1.12k op/s; 10895/204579 objects degraded (5.326%); 14528/204579 objects misplaced (7.101%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T15:05:17.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:17 smithi151 ceph-mon[108966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:05:17.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:17 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:05:17.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:17 smithi151 ceph-mon[108966]: osdmap e1349: 8 total, 8 up, 5 in 2024-09-13T15:05:17.699 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:17.732 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:17.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:17.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:17.834 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:17 smithi184 ceph-mon[92527]: pgmap v2329: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 4.2 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 1.12k op/s; 10895/204579 objects degraded (5.326%); 14528/204579 objects misplaced (7.101%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T15:05:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:17 smithi184 ceph-mon[92527]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:05:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:17 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:05:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:17 smithi184 ceph-mon[92527]: osdmap e1349: 8 total, 8 up, 5 in 2024-09-13T15:05:17.867 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:17.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:17.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:17.968 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:18.018 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:18.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:18.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:18.120 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:18.147 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:18.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:18.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:18.249 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:18.282 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:18.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:18.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:18.384 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:18.417 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:18.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:18.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:18.518 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:18.539 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:18.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:18.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:18.641 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:18.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:18.675 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:18.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:18.776 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:18.799 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:18.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:18.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:18.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:18 smithi184 ceph-mon[92527]: pgmap v2331: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 3.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 1.17k op/s; 11074/193530 objects degraded (5.722%); 13454/193530 objects misplaced (6.952%); 2.2 MiB/s, 35 objects/s recovering 2024-09-13T15:05:18.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:18 smithi151 ceph-mon[108966]: pgmap v2331: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 3.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 1.17k op/s; 11074/193530 objects degraded (5.722%); 13454/193530 objects misplaced (6.952%); 2.2 MiB/s, 35 objects/s recovering 2024-09-13T15:05:18.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:18 smithi151 ceph-mon[104450]: pgmap v2331: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 3.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 1.17k op/s; 11074/193530 objects degraded (5.722%); 13454/193530 objects misplaced (6.952%); 2.2 MiB/s, 35 objects/s recovering 2024-09-13T15:05:18.901 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:18.932 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:18.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:18.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:19.033 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:19.060 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:19.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:19.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:19.162 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:19.199 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:19.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:19.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:19.301 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:19.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:19.338 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:19.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:19.439 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:19.469 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:19.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:19.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:19.571 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:19.603 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:19.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:19.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:19.705 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:19.736 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:19.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:19.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:19.838 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:19.872 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:19.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:19.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:19.974 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:20.010 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:20.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:20.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:20.110 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:20.111 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:20.131 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:20.137 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:20.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:20.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:20.238 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:20.277 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:20.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:20.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:20.379 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:20.409 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:20.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:20.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:20.511 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:20.548 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:20.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:20.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:20.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:05:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:05:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:05:20.650 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:20.691 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:20.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:20.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:20.793 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:20.830 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:20.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:20.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:20.932 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:20.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:20.972 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:20.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:21.013 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:20 smithi151 ceph-mon[108966]: pgmap v2332: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 3.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 1.16k op/s; 11074/187287 objects degraded (5.913%); 13071/187287 objects misplaced (6.979%); 2.2 MiB/s, 35 objects/s recovering 2024-09-13T15:05:21.013 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:20 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 11074/187287 objects degraded (5.913%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:21.013 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:20 smithi151 ceph-mon[104450]: pgmap v2332: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 3.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 1.16k op/s; 11074/187287 objects degraded (5.913%); 13071/187287 objects misplaced (6.979%); 2.2 MiB/s, 35 objects/s recovering 2024-09-13T15:05:21.013 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:20 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 11074/187287 objects degraded (5.913%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:21.073 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:21.109 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:21.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:21.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:21.210 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:21.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:21.245 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:21.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:21.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:21.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:20 smithi184 ceph-mon[92527]: pgmap v2332: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 3.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 1.16k op/s; 11074/187287 objects degraded (5.913%); 13071/187287 objects misplaced (6.979%); 2.2 MiB/s, 35 objects/s recovering 2024-09-13T15:05:21.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:20 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 11074/187287 objects degraded (5.913%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:21.379 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:21.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:21.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:21.481 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:21.520 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:21.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:21.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:21.606 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:05:21.622 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:21.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:21.693 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:21.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:21.793 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:21.829 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:21.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:21.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:21.931 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:21.960 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:21.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:21.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:22.061 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:22.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:22.113 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:22.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:22.213 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:22.254 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:22.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:22.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:22.319 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T15:05:22.320 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:05:22.353 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T15:05:22.353 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:05:22.357 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:22.383 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T15:05:22.383 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:05:22.396 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:22.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:22.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:22.430 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T15:05:22.430 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:05:22.467 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T15:05:22.467 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:05:22.498 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:22.501 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T15:05:22.501 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:05:22.519 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T15:05:22.519 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:05:22.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:22.524 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:22.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:22.625 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:22.648 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:22.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:22.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:22.750 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:22.773 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:22.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:22.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:22.868 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T15:05:22.868 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:05:22.875 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:22.913 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:22.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:22.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:23.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:23.042 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:23.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:23.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:23.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:22 smithi151 ceph-mon[104450]: pgmap v2333: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 3.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 1.16k op/s; 11074/187287 objects degraded (5.913%); 13071/187287 objects misplaced (6.979%); 2.2 MiB/s, 35 objects/s recovering 2024-09-13T15:05:23.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:22 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2147062478' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:23.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:22 smithi151 ceph-mon[108966]: pgmap v2333: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 3.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 1.16k op/s; 11074/187287 objects degraded (5.913%); 13071/187287 objects misplaced (6.979%); 2.2 MiB/s, 35 objects/s recovering 2024-09-13T15:05:23.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:22 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2147062478' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:23.144 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:23.179 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:23.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:23.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:23.281 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:23.318 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:23.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:23.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:23.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:22 smithi184 ceph-mon[92527]: pgmap v2333: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 3.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 1.16k op/s; 11074/187287 objects degraded (5.913%); 13071/187287 objects misplaced (6.979%); 2.2 MiB/s, 35 objects/s recovering 2024-09-13T15:05:23.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:22 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2147062478' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:23.420 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:23.469 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:23.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:23.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:23.571 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:23.605 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:23.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:23.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:23.706 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:23.743 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:23.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:23.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:23.845 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:23.882 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:23.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:23.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:23.984 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:24.021 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:24.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:24.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:24.123 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:24.159 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:24.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:24.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:24.260 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:24.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:24.296 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:24.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:24.397 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:24.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:24.429 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:24.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:24.530 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:24.570 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:24.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:24.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:24.672 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:24.711 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:24.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:24.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:24.813 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:24.848 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:24.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:24.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:24.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:05:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:05:24] "GET /metrics HTTP/1.1" 200 36720 "" "Prometheus/2.51.0" 2024-09-13T15:05:24.949 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:24.976 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:24.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:24.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:25.078 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:25.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:25.116 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:25.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:25.217 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:25.246 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:25.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:25.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:25.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:25.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:24 smithi184 ceph-mon[92527]: pgmap v2334: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 3.6 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 1.15k op/s; 11246/176775 objects degraded (6.362%); 11971/176775 objects misplaced (6.772%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T15:05:25.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:24 smithi151 ceph-mon[104450]: pgmap v2334: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 3.6 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 1.15k op/s; 11246/176775 objects degraded (6.362%); 11971/176775 objects misplaced (6.772%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T15:05:25.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:24 smithi151 ceph-mon[108966]: pgmap v2334: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 3.6 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 1.15k op/s; 11246/176775 objects degraded (6.362%); 11971/176775 objects misplaced (6.772%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T15:05:25.384 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:25.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:25.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:25.486 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:25.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:25.523 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:25.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:25.624 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:25.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:25.646 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:25.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:25.747 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:25.780 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:25.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:25.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:25.882 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:25.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:25.918 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:25.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:26.019 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:26.049 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:26.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:26.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:26.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:25 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 11246/176775 objects degraded (6.362%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:26.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:25 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 11246/176775 objects degraded (6.362%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:26.150 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:26.184 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:26.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:26.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:26.286 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:26.323 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:26.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:26.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:26.356 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:25 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 11246/176775 objects degraded (6.362%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:26.425 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:26.460 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:26.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:26.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:26.562 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:26.594 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:26.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:26.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:26.696 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:26.726 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:26.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:26.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:26.828 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:26.858 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:26.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:26.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:26.960 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:26.994 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:26.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:26.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:27.096 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:27.131 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:27.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:27.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:27.233 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:27.261 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:27.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:27.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:27.269 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:26 smithi151 ceph-mon[104450]: pgmap v2335: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 3.5 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 1.12k op/s; 11246/170826 objects degraded (6.583%); 11614/170826 objects misplaced (6.799%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T15:05:27.270 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:26 smithi151 ceph-mon[108966]: pgmap v2335: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 3.5 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 1.12k op/s; 11246/170826 objects degraded (6.583%); 11614/170826 objects misplaced (6.799%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T15:05:27.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:26 smithi184 ceph-mon[92527]: pgmap v2335: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 3.5 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 1.12k op/s; 11246/170826 objects degraded (6.583%); 11614/170826 objects misplaced (6.799%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T15:05:27.363 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:27.391 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:27.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:27.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:27.493 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:27.527 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:27.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:27.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:27.587 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T15:05:27.587 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-13T15:05:27.587 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-13T15:05:27.587 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T15:05:27.629 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:27.663 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:27.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:27.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:27.765 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:27.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:27.811 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:27.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:27.912 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:27.949 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:27.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:27.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:28.051 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:28.078 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:28.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:28.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:28.180 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:28.215 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:28.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:28.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:28.317 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:28.345 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:28.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:28.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:28.447 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:28.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:28.480 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:28.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:28.582 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:28.614 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:28.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:28.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:28.716 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:28.746 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:28.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:28.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:28.849 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:28.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:28.897 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:28.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:28.999 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:29.068 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:29.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:29.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:29.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:29 smithi151 ceph-mon[104450]: pgmap v2336: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 3.3 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 1.29k op/s; 11363/160161 objects degraded (7.095%); 10563/160161 objects misplaced (6.595%); 3.5 MiB/s, 55 objects/s recovering 2024-09-13T15:05:29.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:29 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:05:29.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:29 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:05:29.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:29 smithi151 ceph-mon[108966]: pgmap v2336: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 3.3 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 1.29k op/s; 11363/160161 objects degraded (7.095%); 10563/160161 objects misplaced (6.595%); 3.5 MiB/s, 55 objects/s recovering 2024-09-13T15:05:29.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:29 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:05:29.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:29 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:05:29.170 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:29.239 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:29.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:29.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:29.341 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:29.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:29 smithi184 ceph-mon[92527]: pgmap v2336: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 3.3 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 1.29k op/s; 11363/160161 objects degraded (7.095%); 10563/160161 objects misplaced (6.595%); 3.5 MiB/s, 55 objects/s recovering 2024-09-13T15:05:29.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:29 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:05:29.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:29 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:05:29.371 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:29.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:29.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:29.473 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:29.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:29.527 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:29.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:29.628 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:29.673 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:29.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:29.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:29.702 INFO:teuthology.orchestra.run.smithi151.stdout:moved 0 / 357 (0%) 2024-09-13T15:05:29.702 INFO:teuthology.orchestra.run.smithi151.stdout:avg 71.4 2024-09-13T15:05:29.702 INFO:teuthology.orchestra.run.smithi151.stdout:stddev 4.40908 -> 4.40908 (expected baseline 7.55778) 2024-09-13T15:05:29.702 INFO:teuthology.orchestra.run.smithi151.stdout:min osd.7 with 68 -> 68 pgs (0.952381 -> 0.952381 * mean) 2024-09-13T15:05:29.703 INFO:teuthology.orchestra.run.smithi151.stdout:max osd.1 with 80 -> 80 pgs (1.12045 -> 1.12045 * mean) 2024-09-13T15:05:29.703 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T15:05:29.703 INFO:teuthology.orchestra.run.smithi151.stdout:oload 110 2024-09-13T15:05:29.703 INFO:teuthology.orchestra.run.smithi151.stdout:max_change 3 2024-09-13T15:05:29.703 INFO:teuthology.orchestra.run.smithi151.stdout:max_change_osds 4 2024-09-13T15:05:29.703 INFO:teuthology.orchestra.run.smithi151.stdout:average_utilization 0.0337 2024-09-13T15:05:29.703 INFO:teuthology.orchestra.run.smithi151.stdout:overload_utilization 0.0370 2024-09-13T15:05:29.703 INFO:teuthology.orchestra.run.smithi151.stderr:no change 2024-09-13T15:05:29.776 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:29.811 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:29.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:29.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:29.913 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:29.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T15:05:29.959 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:29.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:29.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:30.062 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:30.098 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:30.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:30.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:30.145 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:30.200 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:30.270 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:30.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:30.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:30.279 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:30 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:05:30.279 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:30 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:05:30.280 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:30 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:05:30.280 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:30 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:05:30.280 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:30 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:05:30.280 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:30 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:05:30.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:30 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:05:30.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:30 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:05:30.355 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:30 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:05:30.355 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:05:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:05:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:05:30.368 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:05:30.372 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:30.454 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:30.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:30.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:30.556 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:30.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:30.604 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:30.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:30.705 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:30.751 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:30.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:30.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:30.812 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-13T15:05:30.812 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight-by-utilization 110 0.05 2024-09-13T15:05:30.853 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:30.885 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:30.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:30.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:30.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:31.018 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:31 smithi151 ceph-mon[104450]: from='client.55867 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:05:31.018 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:31 smithi151 ceph-mon[104450]: pgmap v2337: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 3.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 1.10k op/s; 11363/153693 objects degraded (7.393%); 10188/153693 objects misplaced (6.629%); 2.3 MiB/s, 37 objects/s recovering 2024-09-13T15:05:31.018 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:31 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 11363/160161 objects degraded (7.095%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:31.021 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:31 smithi151 ceph-mon[108966]: from='client.55867 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:05:31.021 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:31 smithi151 ceph-mon[108966]: pgmap v2337: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 3.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 1.10k op/s; 11363/153693 objects degraded (7.393%); 10188/153693 objects misplaced (6.629%); 2.3 MiB/s, 37 objects/s recovering 2024-09-13T15:05:31.021 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:31 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 11363/160161 objects degraded (7.095%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:31.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:31.057 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:31.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:31.157 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:31.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:31.186 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:31.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:31.287 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:31.323 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:31.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:31.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:31.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:31 smithi184 ceph-mon[92527]: from='client.55867 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:05:31.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:31 smithi184 ceph-mon[92527]: pgmap v2337: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 3.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 1.10k op/s; 11363/153693 objects degraded (7.393%); 10188/153693 objects misplaced (6.629%); 2.3 MiB/s, 37 objects/s recovering 2024-09-13T15:05:31.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:31 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 11363/160161 objects degraded (7.095%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:31.425 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:31.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:31.464 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:31.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:31.547 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:31.565 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:31.574 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:31.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:31.636 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:31.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:31.737 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:31.763 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:31.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:31.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:31.865 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:31.896 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:31.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:31.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:31.997 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:32.036 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:32.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:32.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:32.138 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:32.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:32.183 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:32.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:32.284 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:32.318 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:32.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:32.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:32.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:32.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:32.420 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:32.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:32.454 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:32.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:32.555 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:32.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:32.593 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:32.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:32.694 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:32.725 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:32.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:32.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:32.827 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:32.862 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:32.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:32.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:32.963 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:33.000 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:33.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:33.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:33.103 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:33.136 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:33.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:33.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:33.238 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:33.289 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:33.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:33.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:33.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:33 smithi184 ceph-mon[92527]: pgmap v2338: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 3.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 932 op/s; 11363/153693 objects degraded (7.393%); 10188/153693 objects misplaced (6.629%); 2.3 MiB/s, 37 objects/s recovering 2024-09-13T15:05:33.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:33 smithi151 ceph-mon[104450]: pgmap v2338: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 3.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 932 op/s; 11363/153693 objects degraded (7.393%); 10188/153693 objects misplaced (6.629%); 2.3 MiB/s, 37 objects/s recovering 2024-09-13T15:05:33.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:33 smithi151 ceph-mon[108966]: pgmap v2338: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 3.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 932 op/s; 11363/153693 objects degraded (7.393%); 10188/153693 objects misplaced (6.629%); 2.3 MiB/s, 37 objects/s recovering 2024-09-13T15:05:33.391 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:33.423 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:33.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:33.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:33.525 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:33.560 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:33.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:33.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:33.662 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:33.698 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:33.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:33.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:33.800 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:33.830 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:33.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:33.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:33.932 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:33.965 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:33.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:33.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:34.066 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:34.101 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:34.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:34.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:34.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:34.239 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:34.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:34.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:34.341 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:34.366 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:34.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:34.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:34.468 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:34.489 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:34.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:34.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:34.591 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:34.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:34.631 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:34.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:34.732 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:34.760 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:34.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:34.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:34.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:05:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:05:34] "GET /metrics HTTP/1.1" 200 36724 "" "Prometheus/2.51.0" 2024-09-13T15:05:34.861 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:34.889 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:34.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:34.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:34.991 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:35.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:35.027 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:35.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:35.128 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:35.162 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:35.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:35.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:35.264 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:35.285 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:35.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:35.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:35.312 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:35 smithi151 ceph-mon[104450]: pgmap v2339: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 2.9 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 1.23k op/s; 11442/142839 objects degraded (8.010%); 9075/142839 objects misplaced (6.353%); 3.7 MiB/s, 59 objects/s recovering 2024-09-13T15:05:35.312 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:35 smithi151 ceph-mon[108966]: pgmap v2339: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 2.9 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 1.23k op/s; 11442/142839 objects degraded (8.010%); 9075/142839 objects misplaced (6.353%); 3.7 MiB/s, 59 objects/s recovering 2024-09-13T15:05:35.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:35 smithi184 ceph-mon[92527]: pgmap v2339: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 2.9 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 1.23k op/s; 11442/142839 objects degraded (8.010%); 9075/142839 objects misplaced (6.353%); 3.7 MiB/s, 59 objects/s recovering 2024-09-13T15:05:35.387 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:35.417 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:35.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:35.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:35.496 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:35.519 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:35.540 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:35.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:35.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:35.642 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:35.680 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:35.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:35.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:35.782 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:35.820 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:35.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:35.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:35.921 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:35.966 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:35.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:35.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:36.068 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:36.095 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:36.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:36.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:36.197 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:36.239 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:36.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:36.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:36.286 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:36 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 11442/142839 objects degraded (8.010%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:36.287 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:36 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 11442/142839 objects degraded (8.010%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:36.342 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:36.353 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:36 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 11442/142839 objects degraded (8.010%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:36.379 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:36.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:36.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:36.481 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:36.502 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:36.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:36.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:36.604 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:36.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:36.626 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:36.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:36.727 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:36.750 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:36.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:36.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:36.852 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:36.887 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:36.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:36.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:36.989 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:37.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:37.026 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:37.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:37.127 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:37.167 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:37.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:37.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:37.269 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:37.300 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:37.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:37.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:37.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:37 smithi184 ceph-mon[92527]: pgmap v2340: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 2.8 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 1.12k op/s; 11442/136278 objects degraded (8.396%); 8697/136278 objects misplaced (6.382%); 2.5 MiB/s, 40 objects/s recovering 2024-09-13T15:05:37.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:37 smithi151 ceph-mon[104450]: pgmap v2340: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 2.8 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 1.12k op/s; 11442/136278 objects degraded (8.396%); 8697/136278 objects misplaced (6.382%); 2.5 MiB/s, 40 objects/s recovering 2024-09-13T15:05:37.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:37 smithi151 ceph-mon[108966]: pgmap v2340: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 2.8 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 1.12k op/s; 11442/136278 objects degraded (8.396%); 8697/136278 objects misplaced (6.382%); 2.5 MiB/s, 40 objects/s recovering 2024-09-13T15:05:37.401 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:37.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:37.444 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:37.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:37.545 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:37.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:37.583 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:37.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:37.684 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:37.719 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:37.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:37.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:37.821 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:37.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:37.860 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:37.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:37.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:37.992 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:37.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:37.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:38.081 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-13T15:05:38.093 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:38.133 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:38.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:38.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:38.235 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:38.268 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:38.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:38.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:38.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:38 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3039493894' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:05:38.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:38 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3039493894' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:05:38.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:38 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3039493894' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:05:38.370 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:38.480 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:38.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:38.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:38.582 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:38.616 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:38.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:38.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:38.718 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:38.752 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:38.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:38.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:38.854 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:38.894 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:38.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:38.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:38.995 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:39.043 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:39.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:39.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:39.145 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:39.184 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:39.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:39.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:39.285 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:39.321 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:39.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:39.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:39.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:39 smithi184 ceph-mon[92527]: pgmap v2341: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 2.6 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 1.22k op/s; 11235/126834 objects degraded (8.858%); 7896/126834 objects misplaced (6.225%); 3.4 MiB/s, 74 objects/s recovering 2024-09-13T15:05:39.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:39 smithi184 ceph-mon[92527]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:05:39.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:39 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3039493894' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:05:39.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:39 smithi184 ceph-mon[92527]: osdmap e1350: 8 total, 8 up, 5 in 2024-09-13T15:05:39.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:39 smithi151 ceph-mon[104450]: pgmap v2341: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 2.6 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 1.22k op/s; 11235/126834 objects degraded (8.858%); 7896/126834 objects misplaced (6.225%); 3.4 MiB/s, 74 objects/s recovering 2024-09-13T15:05:39.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:39 smithi151 ceph-mon[104450]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:05:39.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:39 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3039493894' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:05:39.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:39 smithi151 ceph-mon[104450]: osdmap e1350: 8 total, 8 up, 5 in 2024-09-13T15:05:39.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:39 smithi151 ceph-mon[108966]: pgmap v2341: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 2.6 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 1.22k op/s; 11235/126834 objects degraded (8.858%); 7896/126834 objects misplaced (6.225%); 3.4 MiB/s, 74 objects/s recovering 2024-09-13T15:05:39.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:39 smithi151 ceph-mon[108966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:05:39.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:39 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3039493894' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:05:39.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:39 smithi151 ceph-mon[108966]: osdmap e1350: 8 total, 8 up, 5 in 2024-09-13T15:05:39.423 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:39.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:39.468 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:39.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:39.569 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:39.602 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:39.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:39.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:39.703 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:39.735 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:39.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:39.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:39.837 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:39.867 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:39.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:39.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:39.969 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:39.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:40.000 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:40.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:40.101 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:40.139 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:40.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:40.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:40.241 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:40.274 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:40.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:40.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:40.376 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:40.410 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:40.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:40.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:40.512 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:40.538 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:40.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:40.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:40.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:05:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:05:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:05:40.641 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:40.677 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:40.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:40.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:40.779 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:40.813 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:40.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:40.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:40.915 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:40.942 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:40.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:40.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:41.044 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:41.082 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:41.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:41.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:41.184 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:41.220 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:41.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:41.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:41.322 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:41.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:41 smithi184 ceph-mon[92527]: pgmap v2343: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 2.5 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 1.08k op/s; 11235/121221 objects degraded (9.268%); 7611/121221 objects misplaced (6.279%); 2.6 MiB/s, 66 objects/s recovering 2024-09-13T15:05:41.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:41 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 11235/121221 objects degraded (9.268%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:41.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:41 smithi151 ceph-mon[108966]: pgmap v2343: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 2.5 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 1.08k op/s; 11235/121221 objects degraded (9.268%); 7611/121221 objects misplaced (6.279%); 2.6 MiB/s, 66 objects/s recovering 2024-09-13T15:05:41.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:41 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 11235/121221 objects degraded (9.268%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:41.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:41 smithi151 ceph-mon[104450]: pgmap v2343: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 2.5 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 1.08k op/s; 11235/121221 objects degraded (9.268%); 7611/121221 objects misplaced (6.279%); 2.6 MiB/s, 66 objects/s recovering 2024-09-13T15:05:41.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:41 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 11235/121221 objects degraded (9.268%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:41.363 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:41.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:41.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:41.465 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:41.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:41.503 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:41.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:41.604 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:41.638 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:41.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:41.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:41.740 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:41.776 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:41.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:41.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:41.878 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:41.920 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:41.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:41.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:42.021 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:42.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:42.055 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:42.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:42.156 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:42.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:42.188 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:42.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:42.289 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:42.325 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:42.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:42.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:42.427 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:42.465 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:42.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:42.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:42.567 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:42.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:42.606 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:42.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:42.706 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:42.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:42.742 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:42.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:42.843 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:42.882 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:42.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:42.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:42.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:43.018 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:43.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:43.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:43.120 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:43.168 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:43.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:43.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:43.270 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:43.309 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:43.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:43.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:43.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:43 smithi184 ceph-mon[92527]: pgmap v2344: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 2.5 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 1.08k op/s; 11235/121221 objects degraded (9.268%); 7611/121221 objects misplaced (6.279%); 2.6 MiB/s, 66 objects/s recovering 2024-09-13T15:05:43.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:43 smithi151 ceph-mon[104450]: pgmap v2344: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 2.5 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 1.08k op/s; 11235/121221 objects degraded (9.268%); 7611/121221 objects misplaced (6.279%); 2.6 MiB/s, 66 objects/s recovering 2024-09-13T15:05:43.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:43 smithi151 ceph-mon[108966]: pgmap v2344: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 2.5 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 1.08k op/s; 11235/121221 objects degraded (9.268%); 7611/121221 objects misplaced (6.279%); 2.6 MiB/s, 66 objects/s recovering 2024-09-13T15:05:43.411 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:43.448 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:43.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:43.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:43.550 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:43.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:43.590 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:43.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:43.691 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:43.720 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:43.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:43.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:43.822 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:43.855 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:43.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:43.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:43.957 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:43.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:43.986 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:43.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:44.087 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:44.125 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:44.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:44.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:44.227 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:44.276 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:44.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:44.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:44.378 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:44.410 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:44.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:44.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:44.512 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:44.546 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:44.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:44.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:44.648 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:44.694 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:44.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:44.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:44.790 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:05:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:05:44] "GET /metrics HTTP/1.1" 200 36724 "" "Prometheus/2.51.0" 2024-09-13T15:05:44.796 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:44.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:44.831 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:44.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:44.932 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:44.968 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:44.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:44.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:45.073 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:45.108 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:45.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:45.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:45.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T15:05:45.210 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:45.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:45.252 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:45.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:45.353 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:45.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:45 smithi184 ceph-mon[92527]: pgmap v2345: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 2.2 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 1.11k op/s; 10851/109473 objects degraded (9.912%); 6827/109473 objects misplaced (6.236%); 1017 KiB/s, 78 objects/s recovering 2024-09-13T15:05:45.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:45 smithi151 ceph-mon[104450]: pgmap v2345: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 2.2 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 1.11k op/s; 10851/109473 objects degraded (9.912%); 6827/109473 objects misplaced (6.236%); 1017 KiB/s, 78 objects/s recovering 2024-09-13T15:05:45.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:45 smithi151 ceph-mon[108966]: pgmap v2345: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 2.2 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 1.11k op/s; 10851/109473 objects degraded (9.912%); 6827/109473 objects misplaced (6.236%); 1017 KiB/s, 78 objects/s recovering 2024-09-13T15:05:45.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:45.389 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:45.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:45.477 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T15:05:45.477 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-13T15:05:45.477 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-13T15:05:45.477 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-13T15:05:45.478 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T15:05:45.490 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:45.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:45.534 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:45.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:45.635 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:45.674 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:45.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:45.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:45.776 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:45.817 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:45.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:45.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:45.919 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:45.957 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:45.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:45.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:46.059 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:46.087 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:46.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:46.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:46.188 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:46.219 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:46.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:46.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:46.321 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:46.356 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:46 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 10851/109473 objects degraded (9.912%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:46.358 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:46.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:46 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 10851/109473 objects degraded (9.912%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:46.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:46 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 10851/109473 objects degraded (9.912%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:46.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:46.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:46.460 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:46.461 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T15:05:46.488 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:46.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:46.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:46.590 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:46.616 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:46.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:46.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:46.718 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:46.744 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:46.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:46.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:46.846 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:46.899 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:46.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:46.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:47.001 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:47.040 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T15:05:47.041 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:05:47.049 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:47.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:47.058 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:47.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:47.081 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T15:05:47.082 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:05:47.096 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:47 smithi151 ceph-mon[104450]: pgmap v2346: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 2.1 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 1.12k op/s; 10851/102594 objects degraded (10.577%); 6460/102594 objects misplaced (6.297%); 1017 KiB/s, 78 objects/s recovering 2024-09-13T15:05:47.096 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:47 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:47.096 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:47 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/19061003' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:47.096 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:47 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'6':'62260'}"} : dispatch 2024-09-13T15:05:47.097 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:47 smithi151 ceph-mon[108966]: pgmap v2346: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 2.1 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 1.12k op/s; 10851/102594 objects degraded (10.577%); 6460/102594 objects misplaced (6.297%); 1017 KiB/s, 78 objects/s recovering 2024-09-13T15:05:47.097 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:47 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:47.097 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:47 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/19061003' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:47.097 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:47 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'6':'62260'}"} : dispatch 2024-09-13T15:05:47.098 INFO:teuthology.orchestra.run.smithi151.stdout:moved 0 / 357 (0%) 2024-09-13T15:05:47.098 INFO:teuthology.orchestra.run.smithi151.stdout:avg 71.4 2024-09-13T15:05:47.098 INFO:teuthology.orchestra.run.smithi151.stdout:stddev 4.40908 -> 4.40908 (expected baseline 7.55778) 2024-09-13T15:05:47.098 INFO:teuthology.orchestra.run.smithi151.stdout:min osd.7 with 68 -> 68 pgs (0.952381 -> 0.952381 * mean) 2024-09-13T15:05:47.098 INFO:teuthology.orchestra.run.smithi151.stdout:max osd.1 with 80 -> 80 pgs (1.12045 -> 1.12045 * mean) 2024-09-13T15:05:47.098 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T15:05:47.098 INFO:teuthology.orchestra.run.smithi151.stdout:oload 110 2024-09-13T15:05:47.098 INFO:teuthology.orchestra.run.smithi151.stdout:max_change 0.05 2024-09-13T15:05:47.098 INFO:teuthology.orchestra.run.smithi151.stdout:max_change_osds 4 2024-09-13T15:05:47.098 INFO:teuthology.orchestra.run.smithi151.stdout:average_utilization 0.0270 2024-09-13T15:05:47.098 INFO:teuthology.orchestra.run.smithi151.stdout:overload_utilization 0.0297 2024-09-13T15:05:47.099 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6 weight 1.0000 -> 0.9500 2024-09-13T15:05:47.119 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T15:05:47.119 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:05:47.159 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:47.159 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T15:05:47.160 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:05:47.183 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:47.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:47.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:47.195 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T15:05:47.195 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:05:47.226 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T15:05:47.226 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:05:47.262 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T15:05:47.262 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:05:47.285 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:47.293 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T15:05:47.293 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:05:47.328 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:47.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:47.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:47.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:47 smithi184 ceph-mon[92527]: pgmap v2346: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 2.1 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 1.12k op/s; 10851/102594 objects degraded (10.577%); 6460/102594 objects misplaced (6.297%); 1017 KiB/s, 78 objects/s recovering 2024-09-13T15:05:47.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:47 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:47.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:47 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/19061003' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:47.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:47 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'6':'62260'}"} : dispatch 2024-09-13T15:05:47.430 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:47.467 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:47.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:47.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:47.569 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:47.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:47.603 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:47.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:47.704 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:47.740 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:47.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:47.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:47.842 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:47.872 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:47.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:47.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:47.974 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:48.011 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:48.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:48.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:48.113 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:48.147 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:48.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:48.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:48.249 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:48.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:48.282 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:48.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:48.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:48 smithi151 ceph-mon[104450]: from='client.36783 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:05:48.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:48 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'6':'62260'}"}]': finished 2024-09-13T15:05:48.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:48 smithi151 ceph-mon[104450]: osdmap e1351: 8 total, 8 up, 5 in 2024-09-13T15:05:48.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:48 smithi151 ceph-mon[108966]: from='client.36783 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:05:48.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:48 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'6':'62260'}"}]': finished 2024-09-13T15:05:48.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:48 smithi151 ceph-mon[108966]: osdmap e1351: 8 total, 8 up, 5 in 2024-09-13T15:05:48.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:48.415 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:48.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:48.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:48.518 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:48.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:48.551 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:48.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:48.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:48 smithi184 ceph-mon[92527]: from='client.36783 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:05:48.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:48 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'6':'62260'}"}]': finished 2024-09-13T15:05:48.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:48 smithi184 ceph-mon[92527]: osdmap e1351: 8 total, 8 up, 5 in 2024-09-13T15:05:48.652 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:48.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:48.685 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:48.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:48.786 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:48.818 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:48.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:48.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:48.920 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:48.955 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:48.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:48.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:49.057 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:49.090 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:49.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:49.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:49.191 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:49.223 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:49.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:49.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:49.325 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:49.357 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:49.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:49.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:49.397 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:49 smithi151 ceph-mon[108966]: pgmap v2348: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 1.9 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 1.20k op/s; 10405/91473 objects degraded (11.375%); 5676/91473 objects misplaced (6.205%); 0 B/s, 84 objects/s recovering 2024-09-13T15:05:49.398 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:49 smithi151 ceph-mon[104450]: pgmap v2348: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 1.9 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 1.20k op/s; 10405/91473 objects degraded (11.375%); 5676/91473 objects misplaced (6.205%); 0 B/s, 84 objects/s recovering 2024-09-13T15:05:49.460 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:49.509 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:49.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:49.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:49.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:49 smithi184 ceph-mon[92527]: pgmap v2348: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 1.9 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 1.20k op/s; 10405/91473 objects degraded (11.375%); 5676/91473 objects misplaced (6.205%); 0 B/s, 84 objects/s recovering 2024-09-13T15:05:49.611 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:49.646 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:49.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:49.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:49.748 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:49.786 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:49.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:49.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:49.888 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:49.925 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:49.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:49.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:49.927 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:50.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:50.068 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:50.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:50.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:50.170 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:50.198 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:50.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:50.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:50.300 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:50.338 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:50.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:50.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:50.439 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:50.474 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:50.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:50.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:50.505 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:05:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:05:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:05:50.576 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:50.608 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:50.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:50.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:50.710 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:50.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:50.747 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:50.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:50.848 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:50.882 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:50.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:50.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:50.892 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:05:50.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:51.023 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:51.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:51.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:51.125 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:51.155 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:51.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:51.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:51.250 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:51 smithi151 ceph-mon[104450]: pgmap v2349: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 1.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 1.22k op/s; 10405/84597 objects degraded (12.299%); 5320/84597 objects misplaced (6.289%); 0 B/s, 82 objects/s recovering 2024-09-13T15:05:51.250 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:51 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 10405/84597 objects degraded (12.299%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:51.250 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:51 smithi151 ceph-mon[108966]: pgmap v2349: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 1.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 1.22k op/s; 10405/84597 objects degraded (12.299%); 5320/84597 objects misplaced (6.289%); 0 B/s, 82 objects/s recovering 2024-09-13T15:05:51.250 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:51 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 10405/84597 objects degraded (12.299%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:51.258 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:51.284 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:51.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:51.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:51.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:51.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:51.424 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:51.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:51.525 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:51.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:51.563 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:51.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:51.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:51 smithi184 ceph-mon[92527]: pgmap v2349: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 1.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 1.22k op/s; 10405/84597 objects degraded (12.299%); 5320/84597 objects misplaced (6.289%); 0 B/s, 82 objects/s recovering 2024-09-13T15:05:51.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:51 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 10405/84597 objects degraded (12.299%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:51.664 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:51.697 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:51.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:51.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:51.800 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:51.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:51.834 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:51.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:51.934 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:51.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:51.973 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:51.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:52.074 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:52.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:52.110 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:52.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:52.135 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-13T15:05:52.211 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:52.244 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:52.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:52.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:52.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:52.388 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:52.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:52.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:52.490 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:52.521 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:52.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:52.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:52.575 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:52 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/476790633' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:05:52.575 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:52 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:05:52.576 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:52 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/476790633' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:05:52.576 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:52 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:05:52.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:52 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/476790633' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:05:52.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:52 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:05:52.623 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:52.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:52.657 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:52.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:52.758 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:52.790 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:52.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:52.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:52.892 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:52.926 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:52.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:52.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:53.028 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:53.062 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:53.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:53.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:53.164 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:53.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:53.198 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:53.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:53.299 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:53.302 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:53 smithi151 ceph-mon[108966]: pgmap v2350: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 1.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 1.22k op/s; 10405/84597 objects degraded (12.299%); 5320/84597 objects misplaced (6.289%); 0 B/s, 82 objects/s recovering 2024-09-13T15:05:53.302 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:53 smithi151 ceph-mon[108966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:05:53.302 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:53 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:05:53.302 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:53 smithi151 ceph-mon[108966]: osdmap e1352: 8 total, 8 up, 5 in 2024-09-13T15:05:53.303 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:53 smithi151 ceph-mon[104450]: pgmap v2350: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 1.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 1.22k op/s; 10405/84597 objects degraded (12.299%); 5320/84597 objects misplaced (6.289%); 0 B/s, 82 objects/s recovering 2024-09-13T15:05:53.303 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:53 smithi151 ceph-mon[104450]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:05:53.303 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:53 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:05:53.303 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:53 smithi151 ceph-mon[104450]: osdmap e1352: 8 total, 8 up, 5 in 2024-09-13T15:05:53.310 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:53.329 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:53.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:53.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:53.353 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T15:05:53.353 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:05:53.431 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:53.467 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:53.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:53.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:53.569 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:53.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:53 smithi184 ceph-mon[92527]: pgmap v2350: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 1.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 1.22k op/s; 10405/84597 objects degraded (12.299%); 5320/84597 objects misplaced (6.289%); 0 B/s, 82 objects/s recovering 2024-09-13T15:05:53.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:53 smithi184 ceph-mon[92527]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:05:53.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:53 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:05:53.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:53 smithi184 ceph-mon[92527]: osdmap e1352: 8 total, 8 up, 5 in 2024-09-13T15:05:53.606 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:53.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:53.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:53.708 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:53.736 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:53.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:53.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:53.838 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:53.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:53.873 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:53.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:53.974 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:54.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:54.009 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:54.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:54.110 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:54.147 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:54.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:54.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:54.249 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:54.276 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:54.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:54.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:54.378 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:54.400 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:54.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:54.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:54.438 INFO:tasks.thrashosds.thrasher:in_osds: [3, 6, 7, 1, 5] out_osds: [0, 4, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T15:05:54.438 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:05:54.439 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_6 2024-09-13T15:05:54.439 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:05:54.501 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:54.525 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:54.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:54.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:54.627 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:54.648 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:54.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:54.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:54.750 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:54.771 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:54.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:54.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:54.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:05:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:05:54] "GET /metrics HTTP/1.1" 200 36721 "" "Prometheus/2.51.0" 2024-09-13T15:05:54.873 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:54.895 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:54.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:54.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:54.997 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:55.016 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T15:05:55.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:55.019 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:55.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:55.120 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:55.142 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:55.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:55.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:55.244 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:55.275 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:55.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:55.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:55.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:55 smithi151 ceph-mon[104450]: pgmap v2352: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 1.5 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 1.24k op/s; 9852/72909 objects degraded (13.513%); 4494/72909 objects misplaced (6.164%); 0 B/s, 124 objects/s recovering 2024-09-13T15:05:55.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:55 smithi151 ceph-mon[104450]: osdmap e1353: 8 total, 8 up, 5 in 2024-09-13T15:05:55.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:55 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 9852/72909 objects degraded (13.513%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:55.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:55 smithi151 ceph-mon[108966]: pgmap v2352: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 1.5 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 1.24k op/s; 9852/72909 objects degraded (13.513%); 4494/72909 objects misplaced (6.164%); 0 B/s, 124 objects/s recovering 2024-09-13T15:05:55.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:55 smithi151 ceph-mon[108966]: osdmap e1353: 8 total, 8 up, 5 in 2024-09-13T15:05:55.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:55 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 9852/72909 objects degraded (13.513%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:55.376 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:55.411 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:55.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:55.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:55.513 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:55.549 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:55.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:55.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:55.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:55 smithi184 ceph-mon[92527]: pgmap v2352: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 1.5 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 1.24k op/s; 9852/72909 objects degraded (13.513%); 4494/72909 objects misplaced (6.164%); 0 B/s, 124 objects/s recovering 2024-09-13T15:05:55.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:55 smithi184 ceph-mon[92527]: osdmap e1353: 8 total, 8 up, 5 in 2024-09-13T15:05:55.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:55 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 9852/72909 objects degraded (13.513%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T15:05:55.651 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:55.661 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T15:05:55.692 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:55.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:55.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:55.797 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:55.842 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:55.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:55.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:55.944 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:55.973 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:55.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:55.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:56.075 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:56.095 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:56.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:56.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:56.185 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:56 smithi151 ceph-mon[104450]: osdmap e1354: 8 total, 8 up, 5 in 2024-09-13T15:05:56.186 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:56 smithi151 ceph-mon[108966]: osdmap e1354: 8 total, 8 up, 5 in 2024-09-13T15:05:56.196 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:56.203 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:56.219 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:56.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:56.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:56.231 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T15:05:56.231 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:05:56.269 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T15:05:56.269 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:05:56.290 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:56.309 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T15:05:56.310 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:05:56.321 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:56.348 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T15:05:56.349 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:05:56.361 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:56.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:56.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:56.376 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T15:05:56.376 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:05:56.408 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T15:05:56.409 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:05:56.438 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T15:05:56.438 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:05:56.463 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:56.469 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T15:05:56.469 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:05:56.496 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:56.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:56.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:56.598 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:56.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:56 smithi184 ceph-mon[92527]: osdmap e1354: 8 total, 8 up, 5 in 2024-09-13T15:05:56.631 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:56.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:56.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:56.733 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:56.772 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:56.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:56.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:56.874 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:56.909 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:56.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:56.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:57.011 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:57.045 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:57.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:57.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:57.147 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:57.167 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:57.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:57.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:57.269 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:57.303 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:57.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:57.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:57.404 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:57.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:57.436 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:57.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:57.537 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:57.565 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:57.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:57.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:57.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:57 smithi184 ceph-mon[92527]: pgmap v2355: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 1.3 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 1.04k op/s; 9852/65922 objects degraded (14.945%); 4093/65922 objects misplaced (6.209%); 0 B/s, 92 objects/s recovering 2024-09-13T15:05:57.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:57 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3034331221' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:57.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:57 smithi151 ceph-mon[104450]: pgmap v2355: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 1.3 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 1.04k op/s; 9852/65922 objects degraded (14.945%); 4093/65922 objects misplaced (6.209%); 0 B/s, 92 objects/s recovering 2024-09-13T15:05:57.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:57 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3034331221' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:57.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:57 smithi151 ceph-mon[108966]: pgmap v2355: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 113 active+clean; 1.3 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 1.04k op/s; 9852/65922 objects degraded (14.945%); 4093/65922 objects misplaced (6.209%); 0 B/s, 92 objects/s recovering 2024-09-13T15:05:57.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:57 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3034331221' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:57.667 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:57.700 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:57.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:57.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:57.802 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:57.837 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:57.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:57.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:57.939 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:57.969 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:57.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:57.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:58.071 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:58.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:58.109 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:58.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:58.210 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:58.231 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:58.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:58.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:58.333 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:58.373 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:58.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:58.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:58.475 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:58.508 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:58.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:58.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:58.610 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:58.644 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:58.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:58.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:58.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:58.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:58.779 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:58.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:58.880 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:58.911 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:58.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:58.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:59.013 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:59.047 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:59.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:59.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:59.149 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:59.182 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:59.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:59.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:59.284 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:59.317 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:59.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:59.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:59.419 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:59.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:05:59 smithi151 ceph-mon[104450]: pgmap v2356: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 1.2 GiB data, 9.6 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 1.54k op/s; 9552/56913 objects degraded (16.784%); 3544/56913 objects misplaced (6.227%); 0 B/s, 142 objects/s recovering 2024-09-13T15:05:59.437 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:05:59 smithi151 ceph-mon[108966]: pgmap v2356: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 1.2 GiB data, 9.6 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 1.54k op/s; 9552/56913 objects degraded (16.784%); 3544/56913 objects misplaced (6.227%); 0 B/s, 142 objects/s recovering 2024-09-13T15:05:59.455 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:59.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:59.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:59.557 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:59.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:59.599 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:59.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:59.603 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:05:59 smithi184 ceph-mon[92527]: pgmap v2356: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 1.2 GiB data, 9.6 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 1.54k op/s; 9552/56913 objects degraded (16.784%); 3544/56913 objects misplaced (6.227%); 0 B/s, 142 objects/s recovering 2024-09-13T15:05:59.701 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:59.733 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:05:59.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:59.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:59.835 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:59.869 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:05:59.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:59.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:59.970 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:00.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:00.006 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:00.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:00.107 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:00.143 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:00.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:00.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:00.245 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:00.277 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:00.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:00.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:00.379 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:00.413 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:00.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:00.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:00.515 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:00.549 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:00.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:00.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:00.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:00 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 9552/56913 objects degraded (16.784%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T15:06:00.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:06:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:06:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:06:00.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:00 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 9552/56913 objects degraded (16.784%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T15:06:00.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:00 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 9552/56913 objects degraded (16.784%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T15:06:00.651 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:00.684 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:00.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:00.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:00.786 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:00.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:00.824 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:00.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:00.925 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:00.958 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:00.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:00.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:01.060 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:01.095 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:01.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:01.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:01.197 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:01.219 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:01.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:01.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:01.321 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:01.351 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:01.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:01.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:01.453 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:01.479 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:01.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:01.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:01.581 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:01.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:01 smithi184 ceph-mon[92527]: pgmap v2357: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 1.1 GiB data, 9.5 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 1.39k op/s; 9552/51957 objects degraded (18.384%); 3278/51957 objects misplaced (6.309%); 0 B/s, 109 objects/s recovering 2024-09-13T15:06:01.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:01 smithi151 ceph-mon[104450]: pgmap v2357: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 1.1 GiB data, 9.5 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 1.39k op/s; 9552/51957 objects degraded (18.384%); 3278/51957 objects misplaced (6.309%); 0 B/s, 109 objects/s recovering 2024-09-13T15:06:01.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:01 smithi151 ceph-mon[108966]: pgmap v2357: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 1.1 GiB data, 9.5 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 1.39k op/s; 9552/51957 objects degraded (18.384%); 3278/51957 objects misplaced (6.309%); 0 B/s, 109 objects/s recovering 2024-09-13T15:06:01.617 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:01.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:01.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:01.719 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:01.755 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:01.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:01.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:01.857 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:01.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:01.896 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:01.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:01.997 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:02.018 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:02.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:02.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:02.120 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:02.151 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:02.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:02.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:02.253 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:02.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:02.280 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:02.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:02.381 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:02.412 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:02.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:02.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:02.514 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:02.545 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:02.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:02.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:02.647 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:02.683 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:02.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:02.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:02.785 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:02.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:02.824 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:02.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:02.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:06:02.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:02.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:06:02.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:02.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:06:02.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:02.875 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:02.925 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:02.949 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:02.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:02.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:03.051 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:03.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:03.088 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:03.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:03.189 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:03.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:03.222 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:03.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:03.323 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:03.355 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:03.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:03.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:03.457 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:03.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:03.492 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:03.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:03.593 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:03.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:03.627 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:03.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:03.728 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:03.749 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:03.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:03.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:03.851 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:03.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:03 smithi184 ceph-mon[92527]: pgmap v2358: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 1.1 GiB data, 9.5 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 872 op/s; 9552/51957 objects degraded (18.384%); 3278/51957 objects misplaced (6.309%); 0 B/s, 37 objects/s recovering 2024-09-13T15:06:03.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:03 smithi151 ceph-mon[104450]: pgmap v2358: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 1.1 GiB data, 9.5 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 872 op/s; 9552/51957 objects degraded (18.384%); 3278/51957 objects misplaced (6.309%); 0 B/s, 37 objects/s recovering 2024-09-13T15:06:03.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:03 smithi151 ceph-mon[108966]: pgmap v2358: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 1.1 GiB data, 9.5 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 872 op/s; 9552/51957 objects degraded (18.384%); 3278/51957 objects misplaced (6.309%); 0 B/s, 37 objects/s recovering 2024-09-13T15:06:03.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:03.888 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:03.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:03.989 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:04.025 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:04.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:04.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:04.127 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:04.158 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:04.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:04.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:04.260 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:04.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:04.301 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:04.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:04.402 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:04.424 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:04.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:04.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:04.525 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:04.559 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:04.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:04.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:04.661 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:04.697 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:04.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:04.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:04.800 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:04.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:04.834 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:04.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:04.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:06:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:06:04] "GET /metrics HTTP/1.1" 200 36710 "" "Prometheus/2.51.0" 2024-09-13T15:06:04.935 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:04.957 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:04.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:04.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:05.059 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:05.094 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:05.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:05.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:05.196 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:05.229 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:05.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:05.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:05.331 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:05.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:05.364 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:05.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:05.465 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:05.489 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:05.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:05.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:05.592 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:05.630 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:05.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:05.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:05.731 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:05.766 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:05.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:05.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:05.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:05 smithi184 ceph-mon[92527]: pgmap v2359: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 849 MiB data, 8.7 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 1.09k op/s; 9355/40797 objects degraded (22.931%); 2516/40797 objects misplaced (6.167%); 0 B/s, 50 objects/s recovering 2024-09-13T15:06:05.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:05 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 9355/40797 objects degraded (22.931%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T15:06:05.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:05 smithi151 ceph-mon[104450]: pgmap v2359: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 849 MiB data, 8.7 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 1.09k op/s; 9355/40797 objects degraded (22.931%); 2516/40797 objects misplaced (6.167%); 0 B/s, 50 objects/s recovering 2024-09-13T15:06:05.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:05 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 9355/40797 objects degraded (22.931%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T15:06:05.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:05 smithi151 ceph-mon[108966]: pgmap v2359: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 849 MiB data, 8.7 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 1.09k op/s; 9355/40797 objects degraded (22.931%); 2516/40797 objects misplaced (6.167%); 0 B/s, 50 objects/s recovering 2024-09-13T15:06:05.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:05 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 9355/40797 objects degraded (22.931%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T15:06:05.868 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:05.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:05.903 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:05.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:06.004 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:06.038 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:06.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:06.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:06.139 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:06.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:06.174 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:06.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:06.275 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:06.310 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:06.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:06.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:06.411 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:06.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:06.448 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:06.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:06.549 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:06.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:06.583 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:06.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:06.684 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:06.715 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:06.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:06.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:06.816 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:06.851 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:06.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:06.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:06.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:06.990 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:06.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:06.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:07.091 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:07.134 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:07.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:07.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:07.236 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:07.275 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:07.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:07.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:07.377 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:07.412 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:07.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:07.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:07.514 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:07.536 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:07.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:07.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:07.638 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:07.673 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:07.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:07.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:07.775 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:07.810 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:07.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:07.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:07.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:07 smithi184 ceph-mon[92527]: pgmap v2360: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 706 MiB data, 8.5 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 986 op/s; 9355/33954 objects degraded (27.552%); 2111/33954 objects misplaced (6.217%); 0 B/s, 45 objects/s recovering 2024-09-13T15:06:07.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:07 smithi151 ceph-mon[104450]: pgmap v2360: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 706 MiB data, 8.5 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 986 op/s; 9355/33954 objects degraded (27.552%); 2111/33954 objects misplaced (6.217%); 0 B/s, 45 objects/s recovering 2024-09-13T15:06:07.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:07 smithi151 ceph-mon[108966]: pgmap v2360: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 706 MiB data, 8.5 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 986 op/s; 9355/33954 objects degraded (27.552%); 2111/33954 objects misplaced (6.217%); 0 B/s, 45 objects/s recovering 2024-09-13T15:06:07.912 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:07.945 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:07.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:07.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:08.047 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:08.082 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:08.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:08.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:08.184 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:08.221 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:08.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:08.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:08.325 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:08.353 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:08.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:08.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:08.456 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:08.495 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:08.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:08.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:08.597 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:08.640 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:08.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:08.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:08.742 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:08.784 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:08.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:08.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:08.886 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:08.918 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:08.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:08.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:09.020 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:09.055 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:09.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:09.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:09.157 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:09.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:09.193 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:09.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:09.294 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:09.329 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:09.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:09.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:09.432 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:09.468 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:09.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:09.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:09.570 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:09.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:09.605 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:09.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:09.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:09 smithi151 ceph-mon[104450]: pgmap v2361: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 484 MiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 1.18k op/s; 9105/23277 objects degraded (39.116%); 1433/23277 objects misplaced (6.156%); 0 B/s, 62 objects/s recovering 2024-09-13T15:06:09.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:09 smithi151 ceph-mon[108966]: pgmap v2361: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 484 MiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 1.18k op/s; 9105/23277 objects degraded (39.116%); 1433/23277 objects misplaced (6.156%); 0 B/s, 62 objects/s recovering 2024-09-13T15:06:09.706 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:09.744 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:09.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:09.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:09.846 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:09.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:09 smithi184 ceph-mon[92527]: pgmap v2361: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 484 MiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 1.18k op/s; 9105/23277 objects degraded (39.116%); 1433/23277 objects misplaced (6.156%); 0 B/s, 62 objects/s recovering 2024-09-13T15:06:09.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:09.885 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:09.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:09.986 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:10.023 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:10.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:10.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:10.125 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:10.161 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:10.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:10.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:10.263 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:10.300 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:10.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:10.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:10.402 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:10.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:10.440 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:10.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:10.541 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:10.578 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:10.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:10.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:10.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:10 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 9105/23277 objects degraded (39.116%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T15:06:10.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:06:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:06:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:06:10.680 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:10.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:10.722 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:10.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:10.823 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:10.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:10 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 9105/23277 objects degraded (39.116%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T15:06:10.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:10 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 9105/23277 objects degraded (39.116%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T15:06:10.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:10.860 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:10.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:10.961 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:10.997 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:10.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:10.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:11.100 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:11.151 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:11.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:11.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:11.253 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:11.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:11.287 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:11.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:11.349 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:06:11.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:11.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:11.442 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:11.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:11.543 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:11.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:11.604 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:11.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:11.705 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:11.712 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:11 smithi151 ceph-mon[108966]: pgmap v2362: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 355 MiB data, 7.5 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 1.10k op/s; 9105/17112 objects degraded (53.208%); 1081/17112 objects misplaced (6.317%); 0 B/s, 37 objects/s recovering 2024-09-13T15:06:11.714 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:11 smithi151 ceph-mon[104450]: pgmap v2362: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 355 MiB data, 7.5 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 1.10k op/s; 9105/17112 objects degraded (53.208%); 1081/17112 objects misplaced (6.317%); 0 B/s, 37 objects/s recovering 2024-09-13T15:06:11.736 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:11.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:11.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:11.838 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:11.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:11 smithi184 ceph-mon[92527]: pgmap v2362: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 355 MiB data, 7.5 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 1.10k op/s; 9105/17112 objects degraded (53.208%); 1081/17112 objects misplaced (6.317%); 0 B/s, 37 objects/s recovering 2024-09-13T15:06:11.873 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:11.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:11.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:11.975 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:12.020 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:12.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:12.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:12.122 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:12.134 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T15:06:12.136 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T15:06:12.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:12.157 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:12.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:12.258 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:12.294 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:12.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:12.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:12.396 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:12.428 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:12.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:12.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:12.530 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:12.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:12.592 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:12.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:12.694 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:12.724 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:12.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:12.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:12.826 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:12.859 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:12.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:12.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:12.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:12.997 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:12.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:12.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:13.003 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Removed 121294 objects 2024-09-13T15:06:13.003 INFO:tasks.radosbench.radosbench.0.smithi151.stdout:Clean up completed and total clean up time :107.803 2024-09-13T15:06:13.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:13.120 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:13.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:13.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:13.182 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":2363,"stamp":"2024-09-13T15:06:11.948983+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":372643813,"num_objects":5704,"num_object_clones":0,"num_object_copies":17112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9105,"num_objects_misplaced":1081,"num_objects_unfound":0,"num_objects_dirty":5704,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":236975,"num_write_kb":7762848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11647,"num_bytes_recovered":602170121,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71391,"ondisk_log_size":71391,"up":357,"acting":355,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":357,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":7875192,"kb_used_data":1216656,"kb_used_omap":64,"kb_used_meta":6658431,"kb_avail":460850568,"statfs":{"total":479975178240,"available":471910981632,"internally_reserved":0,"allocated":1245855744,"data_stored":1230030991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":66352,"internal_metadata":6818233552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,4,8,4,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-761200640,"num_objects":-11615,"num_object_clones":0,"num_object_copies":-34845,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-447,"num_objects_misplaced":-2197,"num_objects_unfound":0,"num_objects_dirty":-11615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11615,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":447,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003818"},"pg_stats":[{"pgid":"1.1d","version":"261'100","reported_seq":299,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.144242+0000","last_change":"2024-09-13T15:03:40.112651+0000","last_active":"2024-09-13T15:05:55.144242+0000","last_peered":"2024-09-13T15:05:55.144242+0000","last_clean":"2024-09-13T15:05:55.144242+0000","last_became_active":"2024-09-13T15:03:40.111841+0000","last_became_peered":"2024-09-13T15:03:40.111841+0000","last_unstale":"2024-09-13T15:05:55.144242+0000","last_undegraded":"2024-09-13T15:05:55.144242+0000","last_fullsized":"2024-09-13T15:05:55.144242+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:17.988345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:17.988345+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:05:01.634451+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":1994,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.138837+0000","last_change":"2024-09-13T15:03:40.110517+0000","last_active":"2024-09-13T15:05:54.138837+0000","last_peered":"2024-09-13T15:05:54.138837+0000","last_clean":"2024-09-13T15:05:54.138837+0000","last_became_active":"2024-09-13T15:03:40.100427+0000","last_became_peered":"2024-09-13T15:03:40.100427+0000","last_unstale":"2024-09-13T15:05:54.138837+0000","last_undegraded":"2024-09-13T15:05:54.138837+0000","last_fullsized":"2024-09-13T15:05:54.138837+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:11:25.489060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2850,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":41,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.143668+0000","last_change":"2024-09-13T15:03:40.110517+0000","last_active":"2024-09-13T15:05:55.143668+0000","last_peered":"2024-09-13T15:05:55.143668+0000","last_clean":"2024-09-13T15:05:55.143668+0000","last_became_active":"2024-09-13T15:03:40.109724+0000","last_became_peered":"2024-09-13T15:03:40.109724+0000","last_unstale":"2024-09-13T15:05:55.143668+0000","last_undegraded":"2024-09-13T15:05:55.143668+0000","last_fullsized":"2024-09-13T15:05:55.143668+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:05:04.511989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":439,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.180517+0000","last_change":"2024-09-13T15:03:40.108006+0000","last_active":"2024-09-13T15:05:55.180517+0000","last_peered":"2024-09-13T15:05:55.180517+0000","last_clean":"2024-09-13T15:05:55.180517+0000","last_became_active":"2024-09-13T15:03:40.107644+0000","last_became_peered":"2024-09-13T15:03:40.107644+0000","last_unstale":"2024-09-13T15:05:55.180517+0000","last_undegraded":"2024-09-13T15:05:55.180517+0000","last_fullsized":"2024-09-13T15:05:55.180517+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:16:37.263599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":444,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.179009+0000","last_change":"2024-09-13T15:03:40.107810+0000","last_active":"2024-09-13T15:05:55.179009+0000","last_peered":"2024-09-13T15:05:55.179009+0000","last_clean":"2024-09-13T15:05:55.179009+0000","last_became_active":"2024-09-13T15:03:40.107502+0000","last_became_peered":"2024-09-13T15:03:40.107502+0000","last_unstale":"2024-09-13T15:05:55.179009+0000","last_undegraded":"2024-09-13T15:05:55.179009+0000","last_fullsized":"2024-09-13T15:05:55.179009+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:27.287016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"261'100","reported_seq":36,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.133764+0000","last_change":"2024-09-13T15:02:42.889565+0000","last_active":"2024-09-13T15:05:54.133764+0000","last_peered":"2024-09-13T15:05:54.133764+0000","last_clean":"2024-09-13T15:05:54.133764+0000","last_became_active":"2024-09-13T15:02:42.888466+0000","last_became_peered":"2024-09-13T15:02:42.888466+0000","last_unstale":"2024-09-13T15:05:54.133764+0000","last_undegraded":"2024-09-13T15:05:54.133764+0000","last_fullsized":"2024-09-13T15:05:54.133764+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_clean_scrub_stamp":"2024-09-13T14:44:54.259482+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:19:31.567604+0000","scrub_duration":0.015265569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":145,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.145621+0000","last_change":"2024-09-13T15:00:25.072110+0000","last_active":"2024-09-13T15:05:54.145621+0000","last_peered":"2024-09-13T15:05:54.145621+0000","last_clean":"2024-09-13T15:05:54.145621+0000","last_became_active":"2024-09-13T15:00:25.045594+0000","last_became_peered":"2024-09-13T15:00:25.045594+0000","last_unstale":"2024-09-13T15:05:54.145621+0000","last_undegraded":"2024-09-13T15:05:54.145621+0000","last_fullsized":"2024-09-13T15:05:54.145621+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:59:45.210187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":225,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.180740+0000","last_change":"2024-09-13T15:03:40.104746+0000","last_active":"2024-09-13T15:05:55.180740+0000","last_peered":"2024-09-13T15:05:55.180740+0000","last_clean":"2024-09-13T15:05:55.180740+0000","last_became_active":"2024-09-13T15:03:40.104192+0000","last_became_peered":"2024-09-13T15:03:40.104192+0000","last_unstale":"2024-09-13T15:05:55.180740+0000","last_undegraded":"2024-09-13T15:05:55.180740+0000","last_fullsized":"2024-09-13T15:05:55.180740+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:32:40.343578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1e","version":"261'100","reported_seq":449,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.180761+0000","last_change":"2024-09-13T14:45:35.947332+0000","last_active":"2024-09-13T15:05:55.180761+0000","last_peered":"2024-09-13T15:05:55.180761+0000","last_clean":"2024-09-13T15:05:55.180761+0000","last_became_active":"2024-09-13T14:45:19.810775+0000","last_became_peered":"2024-09-13T14:45:19.810775+0000","last_unstale":"2024-09-13T15:05:55.180761+0000","last_undegraded":"2024-09-13T15:05:55.180761+0000","last_fullsized":"2024-09-13T15:05:55.180761+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:35.947265+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:43:41.569795+0000","last_clean_scrub_stamp":"2024-09-13T14:45:35.947265+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:00.606370+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":145,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.138144+0000","last_change":"2024-09-13T15:00:25.192972+0000","last_active":"2024-09-13T15:05:54.138144+0000","last_peered":"2024-09-13T15:05:54.138144+0000","last_clean":"2024-09-13T15:05:54.138144+0000","last_became_active":"2024-09-13T15:00:25.047858+0000","last_became_peered":"2024-09-13T15:00:25.047858+0000","last_unstale":"2024-09-13T15:05:54.138144+0000","last_undegraded":"2024-09-13T15:05:54.138144+0000","last_fullsized":"2024-09-13T15:05:54.138144+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:22:16.795970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1363,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.179142+0000","last_change":"2024-09-13T15:03:40.107685+0000","last_active":"2024-09-13T15:05:55.179142+0000","last_peered":"2024-09-13T15:05:55.179142+0000","last_clean":"2024-09-13T15:05:55.179142+0000","last_became_active":"2024-09-13T15:03:40.107360+0000","last_became_peered":"2024-09-13T15:03:40.107360+0000","last_unstale":"2024-09-13T15:05:55.179142+0000","last_undegraded":"2024-09-13T15:05:55.179142+0000","last_fullsized":"2024-09-13T15:05:55.179142+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:53.665230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":24,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.133666+0000","last_change":"2024-09-13T15:03:40.108424+0000","last_active":"2024-09-13T15:05:54.133666+0000","last_peered":"2024-09-13T15:05:54.133666+0000","last_clean":"2024-09-13T15:05:54.133666+0000","last_became_active":"2024-09-13T15:03:40.108060+0000","last_became_peered":"2024-09-13T15:03:40.108060+0000","last_unstale":"2024-09-13T15:05:54.133666+0000","last_undegraded":"2024-09-13T15:05:54.133666+0000","last_fullsized":"2024-09-13T15:05:54.133666+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:35:47.475045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1906,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.144037+0000","last_change":"2024-09-13T15:03:40.118810+0000","last_active":"2024-09-13T15:05:55.144037+0000","last_peered":"2024-09-13T15:05:55.144037+0000","last_clean":"2024-09-13T15:05:55.144037+0000","last_became_active":"2024-09-13T15:03:40.118326+0000","last_became_peered":"2024-09-13T15:03:40.118326+0000","last_unstale":"2024-09-13T15:05:55.144037+0000","last_undegraded":"2024-09-13T15:05:55.144037+0000","last_fullsized":"2024-09-13T15:05:55.144037+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:02.375911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"261'100","reported_seq":451,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.179042+0000","last_change":"2024-09-13T15:02:42.900466+0000","last_active":"2024-09-13T15:05:55.179042+0000","last_peered":"2024-09-13T15:05:55.179042+0000","last_clean":"2024-09-13T15:05:55.179042+0000","last_became_active":"2024-09-13T15:02:42.900166+0000","last_became_peered":"2024-09-13T15:02:42.900166+0000","last_unstale":"2024-09-13T15:05:55.179042+0000","last_undegraded":"2024-09-13T15:05:55.179042+0000","last_fullsized":"2024-09-13T15:05:55.179042+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.614340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.614340+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:52.087366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":178,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.143958+0000","last_change":"2024-09-13T15:03:40.112364+0000","last_active":"2024-09-13T15:05:55.143958+0000","last_peered":"2024-09-13T15:05:55.143958+0000","last_clean":"2024-09-13T15:05:55.143958+0000","last_became_active":"2024-09-13T15:03:40.111497+0000","last_became_peered":"2024-09-13T15:03:40.111497+0000","last_unstale":"2024-09-13T15:05:55.143958+0000","last_undegraded":"2024-09-13T15:05:55.143958+0000","last_fullsized":"2024-09-13T15:05:55.143958+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:05:48.315858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":472,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.180067+0000","last_change":"2024-09-13T15:00:25.052382+0000","last_active":"2024-09-13T15:05:55.180067+0000","last_peered":"2024-09-13T15:05:55.180067+0000","last_clean":"2024-09-13T15:05:55.180067+0000","last_became_active":"2024-09-13T15:00:25.051999+0000","last_became_peered":"2024-09-13T15:00:25.051999+0000","last_unstale":"2024-09-13T15:05:55.180067+0000","last_undegraded":"2024-09-13T15:05:55.180067+0000","last_fullsized":"2024-09-13T15:05:55.180067+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:30.490420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.18","version":"261'100","reported_seq":176,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.137723+0000","last_change":"2024-09-13T15:03:40.130464+0000","last_active":"2024-09-13T15:05:54.137723+0000","last_peered":"2024-09-13T15:05:54.137723+0000","last_clean":"2024-09-13T15:05:54.137723+0000","last_became_active":"2024-09-13T15:03:40.111914+0000","last_became_peered":"2024-09-13T15:03:40.111914+0000","last_unstale":"2024-09-13T15:05:54.137723+0000","last_undegraded":"2024-09-13T15:05:54.137723+0000","last_fullsized":"2024-09-13T15:05:54.137723+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:56.548855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:56.548855+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:28:46.404144+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1313440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":292,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.144458+0000","last_change":"2024-09-13T15:00:25.048396+0000","last_active":"2024-09-13T15:05:55.144458+0000","last_peered":"2024-09-13T15:05:55.144458+0000","last_clean":"2024-09-13T15:05:55.144458+0000","last_became_active":"2024-09-13T15:00:25.048084+0000","last_became_peered":"2024-09-13T15:00:25.048084+0000","last_unstale":"2024-09-13T15:05:55.144458+0000","last_undegraded":"2024-09-13T15:05:55.144458+0000","last_fullsized":"2024-09-13T15:05:55.144458+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:05:23.139064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":36,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.134169+0000","last_change":"2024-09-13T15:02:42.895757+0000","last_active":"2024-09-13T15:05:54.134169+0000","last_peered":"2024-09-13T15:05:54.134169+0000","last_clean":"2024-09-13T15:05:54.134169+0000","last_became_active":"2024-09-13T15:02:42.895416+0000","last_became_peered":"2024-09-13T15:02:42.895416+0000","last_unstale":"2024-09-13T15:05:54.134169+0000","last_undegraded":"2024-09-13T15:05:54.134169+0000","last_fullsized":"2024-09-13T15:05:54.134169+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:50:34.099001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1b","version":"261'100","reported_seq":167,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.138270+0000","last_change":"2024-09-13T15:03:40.097377+0000","last_active":"2024-09-13T15:05:54.138270+0000","last_peered":"2024-09-13T15:05:54.138270+0000","last_clean":"2024-09-13T15:05:54.138270+0000","last_became_active":"2024-09-13T15:03:40.097043+0000","last_became_peered":"2024-09-13T15:03:40.097043+0000","last_unstale":"2024-09-13T15:05:54.138270+0000","last_undegraded":"2024-09-13T15:05:54.138270+0000","last_fullsized":"2024-09-13T15:05:54.138270+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:55.988431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:55.988431+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:51.437501+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":458,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.178717+0000","last_change":"2024-09-13T15:03:40.114391+0000","last_active":"2024-09-13T15:05:55.178717+0000","last_peered":"2024-09-13T15:05:55.178717+0000","last_clean":"2024-09-13T15:05:55.178717+0000","last_became_active":"2024-09-13T15:03:40.113947+0000","last_became_peered":"2024-09-13T15:03:40.113947+0000","last_unstale":"2024-09-13T15:05:55.178717+0000","last_undegraded":"2024-09-13T15:05:55.178717+0000","last_fullsized":"2024-09-13T15:05:55.178717+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:16:01.556456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":127,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.138971+0000","last_change":"2024-09-13T14:55:41.883426+0000","last_active":"2024-09-13T15:05:54.138971+0000","last_peered":"2024-09-13T15:05:54.138971+0000","last_clean":"2024-09-13T15:05:54.138971+0000","last_became_active":"2024-09-13T14:55:41.880724+0000","last_became_peered":"2024-09-13T14:55:41.880724+0000","last_unstale":"2024-09-13T15:05:54.138971+0000","last_undegraded":"2024-09-13T15:05:54.138971+0000","last_fullsized":"2024-09-13T15:05:54.138971+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:17.920508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1a","version":"261'100","reported_seq":36,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.132702+0000","last_change":"2024-09-13T15:02:42.887267+0000","last_active":"2024-09-13T15:05:54.132702+0000","last_peered":"2024-09-13T15:05:54.132702+0000","last_clean":"2024-09-13T15:05:54.132702+0000","last_became_active":"2024-09-13T15:02:42.886976+0000","last_became_peered":"2024-09-13T15:02:42.886976+0000","last_unstale":"2024-09-13T15:05:54.132702+0000","last_undegraded":"2024-09-13T15:05:54.132702+0000","last_fullsized":"2024-09-13T15:05:54.132702+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:46.985510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:46.985510+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:10:55.966984+0000","scrub_duration":0.0030525999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":451,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.178809+0000","last_change":"2024-09-13T14:54:05.606882+0000","last_active":"2024-09-13T15:05:55.178809+0000","last_peered":"2024-09-13T15:05:55.178809+0000","last_clean":"2024-09-13T15:05:55.178809+0000","last_became_active":"2024-09-13T14:54:05.605912+0000","last_became_peered":"2024-09-13T14:54:05.605912+0000","last_unstale":"2024-09-13T15:05:55.178809+0000","last_undegraded":"2024-09-13T15:05:55.178809+0000","last_fullsized":"2024-09-13T15:05:55.178809+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:05.445329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1993,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.138768+0000","last_change":"2024-09-13T15:00:05.037972+0000","last_active":"2024-09-13T15:05:54.138768+0000","last_peered":"2024-09-13T15:05:54.138768+0000","last_clean":"2024-09-13T15:05:54.138768+0000","last_became_active":"2024-09-13T15:00:05.037561+0000","last_became_peered":"2024-09-13T15:00:05.037561+0000","last_unstale":"2024-09-13T15:05:54.138768+0000","last_undegraded":"2024-09-13T15:05:54.138768+0000","last_fullsized":"2024-09-13T15:05:54.138768+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:42:04.578024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.15","version":"261'100","reported_seq":36,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.134014+0000","last_change":"2024-09-13T15:02:42.892087+0000","last_active":"2024-09-13T15:05:54.134014+0000","last_peered":"2024-09-13T15:05:54.134014+0000","last_clean":"2024-09-13T15:05:54.134014+0000","last_became_active":"2024-09-13T15:02:42.890787+0000","last_became_peered":"2024-09-13T15:02:42.890787+0000","last_unstale":"2024-09-13T15:05:54.134014+0000","last_undegraded":"2024-09-13T15:05:54.134014+0000","last_fullsized":"2024-09-13T15:05:54.134014+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:38.214865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:38.214865+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:12:04.179567+0000","scrub_duration":0.0016389639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2016,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.179129+0000","last_change":"2024-09-13T15:02:42.901601+0000","last_active":"2024-09-13T15:05:55.179129+0000","last_peered":"2024-09-13T15:05:55.179129+0000","last_clean":"2024-09-13T15:05:55.179129+0000","last_became_active":"2024-09-13T15:02:42.900518+0000","last_became_peered":"2024-09-13T15:02:42.900518+0000","last_unstale":"2024-09-13T15:05:55.179129+0000","last_undegraded":"2024-09-13T15:05:55.179129+0000","last_fullsized":"2024-09-13T15:05:55.179129+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:57:33.494780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":147,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.138884+0000","last_change":"2024-09-13T15:02:42.885564+0000","last_active":"2024-09-13T15:05:54.138884+0000","last_peered":"2024-09-13T15:05:54.138884+0000","last_clean":"2024-09-13T15:05:54.138884+0000","last_became_active":"2024-09-13T15:02:42.884990+0000","last_became_peered":"2024-09-13T15:02:42.884990+0000","last_unstale":"2024-09-13T15:05:54.138884+0000","last_undegraded":"2024-09-13T15:05:54.138884+0000","last_fullsized":"2024-09-13T15:05:54.138884+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:51:42.556111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"261'100","reported_seq":492,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.133635+0000","last_change":"2024-09-13T15:03:40.109293+0000","last_active":"2024-09-13T15:05:54.133635+0000","last_peered":"2024-09-13T15:05:54.133635+0000","last_clean":"2024-09-13T15:05:54.133635+0000","last_became_active":"2024-09-13T15:03:40.108941+0000","last_became_peered":"2024-09-13T15:03:40.108941+0000","last_unstale":"2024-09-13T15:05:54.133635+0000","last_undegraded":"2024-09-13T15:05:54.133635+0000","last_fullsized":"2024-09-13T15:05:54.133635+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.037517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.037517+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:52.673005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":42,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.133932+0000","last_change":"2024-09-13T15:02:42.922069+0000","last_active":"2024-09-13T15:05:54.133932+0000","last_peered":"2024-09-13T15:05:54.133932+0000","last_clean":"2024-09-13T15:05:54.133932+0000","last_became_active":"2024-09-13T15:02:42.893632+0000","last_became_peered":"2024-09-13T15:02:42.893632+0000","last_unstale":"2024-09-13T15:05:54.133932+0000","last_undegraded":"2024-09-13T15:05:54.133932+0000","last_fullsized":"2024-09-13T15:05:54.133932+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:51:16.796405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":270,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":286,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.143779+0000","last_change":"2024-09-13T15:03:40.115076+0000","last_active":"2024-09-13T15:05:55.143779+0000","last_peered":"2024-09-13T15:05:55.143779+0000","last_clean":"2024-09-13T15:05:55.143779+0000","last_became_active":"2024-09-13T15:03:40.111986+0000","last_became_peered":"2024-09-13T15:03:40.111986+0000","last_unstale":"2024-09-13T15:05:55.143779+0000","last_undegraded":"2024-09-13T15:05:55.143779+0000","last_fullsized":"2024-09-13T15:05:55.143779+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:27.947373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"261'100","reported_seq":147,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.138914+0000","last_change":"2024-09-13T15:02:42.887347+0000","last_active":"2024-09-13T15:05:54.138914+0000","last_peered":"2024-09-13T15:05:54.138914+0000","last_clean":"2024-09-13T15:05:54.138914+0000","last_became_active":"2024-09-13T15:02:42.886986+0000","last_became_peered":"2024-09-13T15:02:42.886986+0000","last_unstale":"2024-09-13T15:05:54.138914+0000","last_undegraded":"2024-09-13T15:05:54.138914+0000","last_fullsized":"2024-09-13T15:05:54.138914+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:54.933410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:54.933410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:18.197248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":66,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.133857+0000","last_change":"2024-09-13T15:03:40.104796+0000","last_active":"2024-09-13T15:05:54.133857+0000","last_peered":"2024-09-13T15:05:54.133857+0000","last_clean":"2024-09-13T15:05:54.133857+0000","last_became_active":"2024-09-13T15:03:40.096411+0000","last_became_peered":"2024-09-13T15:03:40.096411+0000","last_unstale":"2024-09-13T15:05:54.133857+0000","last_undegraded":"2024-09-13T15:05:54.133857+0000","last_fullsized":"2024-09-13T15:05:54.133857+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:27:52.898363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":18976,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":164,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.144873+0000","last_change":"2024-09-13T15:00:25.091881+0000","last_active":"2024-09-13T15:05:55.144873+0000","last_peered":"2024-09-13T15:05:55.144873+0000","last_clean":"2024-09-13T15:05:55.144873+0000","last_became_active":"2024-09-13T15:00:25.091538+0000","last_became_peered":"2024-09-13T15:00:25.091538+0000","last_unstale":"2024-09-13T15:05:55.144873+0000","last_undegraded":"2024-09-13T15:05:55.144873+0000","last_fullsized":"2024-09-13T15:05:55.144873+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:29:32.889528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.16","version":"261'100","reported_seq":292,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.145239+0000","last_change":"2024-09-13T15:00:25.083905+0000","last_active":"2024-09-13T15:05:55.145239+0000","last_peered":"2024-09-13T15:05:55.145239+0000","last_clean":"2024-09-13T15:05:55.145239+0000","last_became_active":"2024-09-13T15:00:25.082642+0000","last_became_peered":"2024-09-13T15:00:25.082642+0000","last_unstale":"2024-09-13T15:05:55.145239+0000","last_undegraded":"2024-09-13T15:05:55.145239+0000","last_fullsized":"2024-09-13T15:05:55.145239+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:48.363045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:48.363045+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:31:38.562138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.e","version":"1354'14895","reported_seq":14957,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:06:07.793387+0000","last_change":"2024-09-13T15:02:44.891854+0000","last_active":"2024-09-13T15:06:07.793387+0000","last_peered":"2024-09-13T15:06:07.793387+0000","last_clean":"2024-09-13T15:06:07.793387+0000","last_became_active":"2024-09-13T15:02:44.891563+0000","last_became_peered":"2024-09-13T15:02:44.891563+0000","last_unstale":"2024-09-13T15:06:07.793387+0000","last_undegraded":"2024-09-13T15:06:07.793387+0000","last_fullsized":"2024-09-13T15:06:07.793387+0000","mapping_epoch":1329,"log_start":"1348'11070","ondisk_log_start":"1348'11070","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3825,"log_dups_size":0,"ondisk_log_size":3825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:18:05.783586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23658496,"num_objects":361,"num_object_clones":0,"num_object_copies":1083,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14895,"num_write_kb":488192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":117,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.138263+0000","last_change":"2024-09-13T15:02:42.884334+0000","last_active":"2024-09-13T15:05:54.138263+0000","last_peered":"2024-09-13T15:05:54.138263+0000","last_clean":"2024-09-13T15:05:54.138263+0000","last_became_active":"2024-09-13T15:02:42.884006+0000","last_became_peered":"2024-09-13T15:02:42.884006+0000","last_unstale":"2024-09-13T15:05:54.138263+0000","last_undegraded":"2024-09-13T15:05:54.138263+0000","last_fullsized":"2024-09-13T15:05:54.138263+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:09.872495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":46,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.135466+0000","last_change":"2024-09-13T15:03:40.101482+0000","last_active":"2024-09-13T15:05:54.135466+0000","last_peered":"2024-09-13T15:05:54.135466+0000","last_clean":"2024-09-13T15:05:54.135466+0000","last_became_active":"2024-09-13T15:03:40.101161+0000","last_became_peered":"2024-09-13T15:03:40.101161+0000","last_unstale":"2024-09-13T15:05:54.135466+0000","last_undegraded":"2024-09-13T15:05:54.135466+0000","last_fullsized":"2024-09-13T15:05:54.135466+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:40.808257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":2063,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.144344+0000","last_change":"2024-09-13T15:02:42.907810+0000","last_active":"2024-09-13T15:05:55.144344+0000","last_peered":"2024-09-13T15:05:55.144344+0000","last_clean":"2024-09-13T15:05:55.144344+0000","last_became_active":"2024-09-13T15:02:42.901460+0000","last_became_peered":"2024-09-13T15:02:42.901460+0000","last_unstale":"2024-09-13T15:05:55.144344+0000","last_undegraded":"2024-09-13T15:05:55.144344+0000","last_fullsized":"2024-09-13T15:05:55.144344+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:57:16.999581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"261'100","reported_seq":473,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.180983+0000","last_change":"2024-09-13T15:02:42.899622+0000","last_active":"2024-09-13T15:05:55.180983+0000","last_peered":"2024-09-13T15:05:55.180983+0000","last_clean":"2024-09-13T15:05:55.180983+0000","last_became_active":"2024-09-13T15:02:42.897715+0000","last_became_peered":"2024-09-13T15:02:42.897715+0000","last_unstale":"2024-09-13T15:05:55.180983+0000","last_undegraded":"2024-09-13T15:05:55.180983+0000","last_fullsized":"2024-09-13T15:05:55.180983+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.710192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.710192+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:50:03.858097+0000","scrub_duration":0.0019250910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"1354'14627","reported_seq":26528,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:06:07.404057+0000","last_change":"2024-09-13T15:04:18.382544+0000","last_active":"2024-09-13T15:06:07.404057+0000","last_peered":"2024-09-13T15:06:07.404057+0000","last_clean":"2024-09-13T15:06:07.404057+0000","last_became_active":"2024-09-13T15:04:18.356959+0000","last_became_peered":"2024-09-13T15:04:18.356959+0000","last_unstale":"2024-09-13T15:06:07.404057+0000","last_undegraded":"2024-09-13T15:06:07.404057+0000","last_fullsized":"2024-09-13T15:06:07.404057+0000","mapping_epoch":1345,"log_start":"1348'10132","ondisk_log_start":"1348'10132","created":1329,"last_epoch_clean":1346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":4495,"log_dups_size":0,"ondisk_log_size":4495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:53:13.985260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23527424,"num_objects":359,"num_object_clones":0,"num_object_copies":1077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14627,"num_write_kb":479552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5922,"num_bytes_recovered":388104192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":355,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.178570+0000","last_change":"2024-09-13T15:02:42.896729+0000","last_active":"2024-09-13T15:05:55.178570+0000","last_peered":"2024-09-13T15:05:55.178570+0000","last_clean":"2024-09-13T15:05:55.178570+0000","last_became_active":"2024-09-13T15:02:42.896414+0000","last_became_peered":"2024-09-13T15:02:42.896414+0000","last_unstale":"2024-09-13T15:05:55.178570+0000","last_undegraded":"2024-09-13T15:05:55.178570+0000","last_fullsized":"2024-09-13T15:05:55.178570+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:53:32.471811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":293,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.145261+0000","last_change":"2024-09-13T15:00:25.070169+0000","last_active":"2024-09-13T15:05:55.145261+0000","last_peered":"2024-09-13T15:05:55.145261+0000","last_clean":"2024-09-13T15:05:55.145261+0000","last_became_active":"2024-09-13T15:00:25.069849+0000","last_became_peered":"2024-09-13T15:00:25.069849+0000","last_unstale":"2024-09-13T15:05:55.145261+0000","last_undegraded":"2024-09-13T15:05:55.145261+0000","last_fullsized":"2024-09-13T15:05:55.145261+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:56.491207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":2079,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.181643+0000","last_change":"2024-09-13T15:02:42.912751+0000","last_active":"2024-09-13T15:05:55.181643+0000","last_peered":"2024-09-13T15:05:55.181643+0000","last_clean":"2024-09-13T15:05:55.181643+0000","last_became_active":"2024-09-13T15:02:42.903177+0000","last_became_peered":"2024-09-13T15:02:42.903177+0000","last_unstale":"2024-09-13T15:05:55.181643+0000","last_undegraded":"2024-09-13T15:05:55.181643+0000","last_fullsized":"2024-09-13T15:05:55.181643+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:00:52.788057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"261'100","reported_seq":169,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.145542+0000","last_change":"2024-09-13T15:03:40.102844+0000","last_active":"2024-09-13T15:05:54.145542+0000","last_peered":"2024-09-13T15:05:54.145542+0000","last_clean":"2024-09-13T15:05:54.145542+0000","last_became_active":"2024-09-13T15:03:40.102439+0000","last_became_peered":"2024-09-13T15:03:40.102439+0000","last_unstale":"2024-09-13T15:05:54.145542+0000","last_undegraded":"2024-09-13T15:05:54.145542+0000","last_fullsized":"2024-09-13T15:05:54.145542+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:51:31.310115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:51:31.310115+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:04:56.751492+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"1354'15120","reported_seq":15199,"reported_epoch":1354,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T15:06:07.828641+0000","last_change":"2024-09-13T15:03:41.268200+0000","last_active":"2024-09-13T15:06:07.828641+0000","last_peered":"2024-09-13T15:06:07.828641+0000","last_clean":"2024-09-13T15:03:40.019127+0000","last_became_active":"2024-09-13T15:03:41.208822+0000","last_became_peered":"2024-09-13T15:03:41.208822+0000","last_unstale":"2024-09-13T15:06:07.828641+0000","last_undegraded":"2024-09-13T15:06:07.828641+0000","last_fullsized":"2024-09-13T15:06:07.828641+0000","mapping_epoch":1340,"log_start":"1348'11277","ondisk_log_start":"1348'11277","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3843,"log_dups_size":0,"ondisk_log_size":3843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24510464,"num_objects":374,"num_object_clones":0,"num_object_copies":1122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":374,"num_objects_unfound":0,"num_objects_dirty":374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15120,"num_write_kb":495808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,2],"avail_no_missing":["1","2","7"],"object_location_counts":[{"shards":"1,2,7","objects":374}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2071,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.138099+0000","last_change":"2024-09-13T15:02:42.885186+0000","last_active":"2024-09-13T15:05:54.138099+0000","last_peered":"2024-09-13T15:05:54.138099+0000","last_clean":"2024-09-13T15:05:54.138099+0000","last_became_active":"2024-09-13T15:02:42.884666+0000","last_became_peered":"2024-09-13T15:02:42.884666+0000","last_unstale":"2024-09-13T15:05:54.138099+0000","last_undegraded":"2024-09-13T15:05:54.138099+0000","last_fullsized":"2024-09-13T15:05:54.138099+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:17.386836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":36,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.134088+0000","last_change":"2024-09-13T15:02:42.891180+0000","last_active":"2024-09-13T15:05:54.134088+0000","last_peered":"2024-09-13T15:05:54.134088+0000","last_clean":"2024-09-13T15:05:54.134088+0000","last_became_active":"2024-09-13T15:02:42.890141+0000","last_became_peered":"2024-09-13T15:02:42.890141+0000","last_unstale":"2024-09-13T15:05:54.134088+0000","last_undegraded":"2024-09-13T15:05:54.134088+0000","last_fullsized":"2024-09-13T15:05:54.134088+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:28.910151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":300,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.144274+0000","last_change":"2024-09-13T15:03:40.116490+0000","last_active":"2024-09-13T15:05:55.144274+0000","last_peered":"2024-09-13T15:05:55.144274+0000","last_clean":"2024-09-13T15:05:55.144274+0000","last_became_active":"2024-09-13T15:03:40.115296+0000","last_became_peered":"2024-09-13T15:03:40.115296+0000","last_unstale":"2024-09-13T15:05:55.144274+0000","last_undegraded":"2024-09-13T15:05:55.144274+0000","last_fullsized":"2024-09-13T15:05:55.144274+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:56:13.539892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"261'100","reported_seq":538,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.180416+0000","last_change":"2024-09-13T15:02:42.898674+0000","last_active":"2024-09-13T15:05:55.180416+0000","last_peered":"2024-09-13T15:05:55.180416+0000","last_clean":"2024-09-13T15:05:55.180416+0000","last_became_active":"2024-09-13T15:02:42.898289+0000","last_became_peered":"2024-09-13T15:02:42.898289+0000","last_unstale":"2024-09-13T15:05:55.180416+0000","last_undegraded":"2024-09-13T15:05:55.180416+0000","last_fullsized":"2024-09-13T15:05:55.180416+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:33.574443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:33.574443+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:54:37.955957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"1354'14961","reported_seq":16359,"reported_epoch":1354,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T15:06:07.656606+0000","last_change":"2024-09-13T15:05:53.324373+0000","last_active":"2024-09-13T15:06:07.656606+0000","last_peered":"2024-09-13T15:06:07.656606+0000","last_clean":"2024-09-13T15:03:39.943522+0000","last_became_active":"2024-09-13T15:03:41.281675+0000","last_became_peered":"2024-09-13T15:03:41.281675+0000","last_unstale":"2024-09-13T15:06:07.656606+0000","last_undegraded":"2024-09-13T15:03:41.117384+0000","last_fullsized":"2024-09-13T15:03:41.087538+0000","mapping_epoch":1340,"log_start":"1348'10676","ondisk_log_start":"1348'10676","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":4285,"log_dups_size":0,"ondisk_log_size":4285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:10.953680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28049432,"num_objects":429,"num_object_clones":0,"num_object_copies":1287,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4330,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":429,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14961,"num_write_kb":492417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":671,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1],"avail_no_missing":["6","1","2"],"object_location_counts":[{"shards":"1,6","objects":-3901}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":314,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.143889+0000","last_change":"2024-09-13T15:03:40.115373+0000","last_active":"2024-09-13T15:05:55.143889+0000","last_peered":"2024-09-13T15:05:55.143889+0000","last_clean":"2024-09-13T15:05:55.143889+0000","last_became_active":"2024-09-13T15:03:40.111437+0000","last_became_peered":"2024-09-13T15:03:40.111437+0000","last_unstale":"2024-09-13T15:05:55.143889+0000","last_undegraded":"2024-09-13T15:05:55.143889+0000","last_fullsized":"2024-09-13T15:05:55.143889+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:09.500663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":307,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.144185+0000","last_change":"2024-09-13T15:03:40.117138+0000","last_active":"2024-09-13T15:05:55.144185+0000","last_peered":"2024-09-13T15:05:55.144185+0000","last_clean":"2024-09-13T15:05:55.144185+0000","last_became_active":"2024-09-13T15:03:40.116175+0000","last_became_peered":"2024-09-13T15:03:40.116175+0000","last_unstale":"2024-09-13T15:05:55.144185+0000","last_undegraded":"2024-09-13T15:05:55.144185+0000","last_fullsized":"2024-09-13T15:05:55.144185+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:11.356995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":137,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.145460+0000","last_change":"2024-09-13T15:03:40.101072+0000","last_active":"2024-09-13T15:05:54.145460+0000","last_peered":"2024-09-13T15:05:54.145460+0000","last_clean":"2024-09-13T15:05:54.145460+0000","last_became_active":"2024-09-13T15:03:40.100744+0000","last_became_peered":"2024-09-13T15:03:40.100744+0000","last_unstale":"2024-09-13T15:05:54.145460+0000","last_undegraded":"2024-09-13T15:05:54.145460+0000","last_fullsized":"2024-09-13T15:05:54.145460+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:55:53.279389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"261'100","reported_seq":462,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.181458+0000","last_change":"2024-09-13T15:02:42.903818+0000","last_active":"2024-09-13T15:05:55.181458+0000","last_peered":"2024-09-13T15:05:55.181458+0000","last_clean":"2024-09-13T15:05:55.181458+0000","last_became_active":"2024-09-13T15:02:42.902705+0000","last_became_peered":"2024-09-13T15:02:42.902705+0000","last_unstale":"2024-09-13T15:05:55.181458+0000","last_undegraded":"2024-09-13T15:05:55.181458+0000","last_fullsized":"2024-09-13T15:05:55.181458+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:36.965775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:45:36.965775+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:40:58.317613+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.b","version":"1354'14882","reported_seq":14944,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:06:09.217491+0000","last_change":"2024-09-13T15:02:44.895572+0000","last_active":"2024-09-13T15:06:09.217491+0000","last_peered":"2024-09-13T15:06:09.217491+0000","last_clean":"2024-09-13T15:06:09.217491+0000","last_became_active":"2024-09-13T15:02:44.895300+0000","last_became_peered":"2024-09-13T15:02:44.895300+0000","last_unstale":"2024-09-13T15:06:09.217491+0000","last_undegraded":"2024-09-13T15:06:09.217491+0000","last_fullsized":"2024-09-13T15:06:09.217491+0000","mapping_epoch":1329,"log_start":"1348'10517","ondisk_log_start":"1348'10517","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":4365,"log_dups_size":0,"ondisk_log_size":4365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:46:47.154024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":320,"num_object_clones":0,"num_object_copies":960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14882,"num_write_kb":486464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":286,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.144053+0000","last_change":"2024-09-13T15:03:40.116772+0000","last_active":"2024-09-13T15:05:55.144053+0000","last_peered":"2024-09-13T15:05:55.144053+0000","last_clean":"2024-09-13T15:05:55.144053+0000","last_became_active":"2024-09-13T15:03:40.115696+0000","last_became_peered":"2024-09-13T15:03:40.115696+0000","last_unstale":"2024-09-13T15:05:55.144053+0000","last_undegraded":"2024-09-13T15:05:55.144053+0000","last_fullsized":"2024-09-13T15:05:55.144053+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:11:14.723716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":666,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.136959+0000","last_change":"2024-09-13T15:00:25.089570+0000","last_active":"2024-09-13T15:05:54.136959+0000","last_peered":"2024-09-13T15:05:54.136959+0000","last_clean":"2024-09-13T15:05:54.136959+0000","last_became_active":"2024-09-13T15:00:25.088373+0000","last_became_peered":"2024-09-13T15:00:25.088373+0000","last_unstale":"2024-09-13T15:05:54.136959+0000","last_undegraded":"2024-09-13T15:05:54.136959+0000","last_fullsized":"2024-09-13T15:05:54.136959+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:36:04.220868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":176,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.138176+0000","last_change":"2024-09-13T14:55:41.889668+0000","last_active":"2024-09-13T15:05:54.138176+0000","last_peered":"2024-09-13T15:05:54.138176+0000","last_clean":"2024-09-13T15:05:54.138176+0000","last_became_active":"2024-09-13T14:55:41.889195+0000","last_became_peered":"2024-09-13T14:55:41.889195+0000","last_unstale":"2024-09-13T15:05:54.138176+0000","last_undegraded":"2024-09-13T15:05:54.138176+0000","last_fullsized":"2024-09-13T15:05:54.138176+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:52.765208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"261'100","reported_seq":167,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.138479+0000","last_change":"2024-09-13T15:03:40.096093+0000","last_active":"2024-09-13T15:05:54.138479+0000","last_peered":"2024-09-13T15:05:54.138479+0000","last_clean":"2024-09-13T15:05:54.138479+0000","last_became_active":"2024-09-13T15:03:40.095709+0000","last_became_peered":"2024-09-13T15:03:40.095709+0000","last_unstale":"2024-09-13T15:05:54.138479+0000","last_undegraded":"2024-09-13T15:05:54.138479+0000","last_fullsized":"2024-09-13T15:05:54.138479+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:57.024823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:57.024823+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:16.724516+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"1354'14740","reported_seq":14802,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:06:09.212560+0000","last_change":"2024-09-13T15:02:45.493849+0000","last_active":"2024-09-13T15:06:09.212560+0000","last_peered":"2024-09-13T15:06:09.212560+0000","last_clean":"2024-09-13T15:06:09.212560+0000","last_became_active":"2024-09-13T15:02:45.493251+0000","last_became_peered":"2024-09-13T15:02:45.493251+0000","last_unstale":"2024-09-13T15:06:09.212560+0000","last_undegraded":"2024-09-13T15:06:09.212560+0000","last_fullsized":"2024-09-13T15:06:09.212560+0000","mapping_epoch":1329,"log_start":"1348'10324","ondisk_log_start":"1348'10324","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":4416,"log_dups_size":0,"ondisk_log_size":4416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:15.904351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17825792,"num_objects":272,"num_object_clones":0,"num_object_copies":816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14740,"num_write_kb":480384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":307,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.144735+0000","last_change":"2024-09-13T15:02:42.897548+0000","last_active":"2024-09-13T15:05:55.144735+0000","last_peered":"2024-09-13T15:05:55.144735+0000","last_clean":"2024-09-13T15:05:55.144735+0000","last_became_active":"2024-09-13T15:02:42.897144+0000","last_became_peered":"2024-09-13T15:02:42.897144+0000","last_unstale":"2024-09-13T15:05:55.144735+0000","last_undegraded":"2024-09-13T15:05:55.144735+0000","last_fullsized":"2024-09-13T15:05:55.144735+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:57.915560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":286,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.143616+0000","last_change":"2024-09-13T15:03:40.117679+0000","last_active":"2024-09-13T15:05:55.143616+0000","last_peered":"2024-09-13T15:05:55.143616+0000","last_clean":"2024-09-13T15:05:55.143616+0000","last_became_active":"2024-09-13T15:03:40.117207+0000","last_became_peered":"2024-09-13T15:03:40.117207+0000","last_unstale":"2024-09-13T15:05:55.143616+0000","last_undegraded":"2024-09-13T15:05:55.143616+0000","last_fullsized":"2024-09-13T15:05:55.143616+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:42.675352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2015,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.178901+0000","last_change":"2024-09-13T15:03:40.109066+0000","last_active":"2024-09-13T15:05:55.178901+0000","last_peered":"2024-09-13T15:05:55.178901+0000","last_clean":"2024-09-13T15:05:55.178901+0000","last_became_active":"2024-09-13T15:03:40.108422+0000","last_became_peered":"2024-09-13T15:03:40.108422+0000","last_unstale":"2024-09-13T15:05:55.178901+0000","last_undegraded":"2024-09-13T15:05:55.178901+0000","last_fullsized":"2024-09-13T15:05:55.178901+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:59:09.415893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"261'100","reported_seq":56,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.132499+0000","last_change":"2024-09-13T15:03:40.107901+0000","last_active":"2024-09-13T15:05:54.132499+0000","last_peered":"2024-09-13T15:05:54.132499+0000","last_clean":"2024-09-13T15:05:54.132499+0000","last_became_active":"2024-09-13T15:03:40.107539+0000","last_became_peered":"2024-09-13T15:03:40.107539+0000","last_unstale":"2024-09-13T15:05:54.132499+0000","last_undegraded":"2024-09-13T15:05:54.132499+0000","last_fullsized":"2024-09-13T15:05:54.132499+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T15:02:59.897422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T15:02:59.897422+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:08:06.713489+0000","scrub_duration":0.0032184959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"1354'14773","reported_seq":14835,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:06:09.231014+0000","last_change":"2024-09-13T15:02:45.495691+0000","last_active":"2024-09-13T15:06:09.231014+0000","last_peered":"2024-09-13T15:06:09.231014+0000","last_clean":"2024-09-13T15:06:09.231014+0000","last_became_active":"2024-09-13T15:02:45.495215+0000","last_became_peered":"2024-09-13T15:02:45.495215+0000","last_unstale":"2024-09-13T15:06:09.231014+0000","last_undegraded":"2024-09-13T15:06:09.231014+0000","last_fullsized":"2024-09-13T15:06:09.231014+0000","mapping_epoch":1329,"log_start":"1348'10351","ondisk_log_start":"1348'10351","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":4422,"log_dups_size":0,"ondisk_log_size":4422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:05.547819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19595264,"num_objects":299,"num_object_clones":0,"num_object_copies":897,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14773,"num_write_kb":482304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":2014,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.144627+0000","last_change":"2024-09-13T15:03:40.114732+0000","last_active":"2024-09-13T15:05:55.144627+0000","last_peered":"2024-09-13T15:05:55.144627+0000","last_clean":"2024-09-13T15:05:55.144627+0000","last_became_active":"2024-09-13T15:03:40.113562+0000","last_became_peered":"2024-09-13T15:03:40.113562+0000","last_unstale":"2024-09-13T15:05:55.144627+0000","last_undegraded":"2024-09-13T15:05:55.144627+0000","last_fullsized":"2024-09-13T15:05:55.144627+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:09:34.976604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2045,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.178813+0000","last_change":"2024-09-13T15:00:25.082949+0000","last_active":"2024-09-13T15:05:55.178813+0000","last_peered":"2024-09-13T15:05:55.178813+0000","last_clean":"2024-09-13T15:05:55.178813+0000","last_became_active":"2024-09-13T15:00:25.082279+0000","last_became_peered":"2024-09-13T15:00:25.082279+0000","last_unstale":"2024-09-13T15:05:55.178813+0000","last_undegraded":"2024-09-13T15:05:55.178813+0000","last_fullsized":"2024-09-13T15:05:55.178813+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:49.021898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":36,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.133051+0000","last_change":"2024-09-13T15:02:42.895072+0000","last_active":"2024-09-13T15:05:54.133051+0000","last_peered":"2024-09-13T15:05:54.133051+0000","last_clean":"2024-09-13T15:05:54.133051+0000","last_became_active":"2024-09-13T15:02:42.894718+0000","last_became_peered":"2024-09-13T15:02:42.894718+0000","last_unstale":"2024-09-13T15:05:54.133051+0000","last_undegraded":"2024-09-13T15:05:54.133051+0000","last_fullsized":"2024-09-13T15:05:54.133051+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:55:10.117260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"261'100","reported_seq":293,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.144661+0000","last_change":"2024-09-13T15:00:25.085109+0000","last_active":"2024-09-13T15:05:55.144661+0000","last_peered":"2024-09-13T15:05:55.144661+0000","last_clean":"2024-09-13T15:05:55.144661+0000","last_became_active":"2024-09-13T15:00:25.084170+0000","last_became_peered":"2024-09-13T15:00:25.084170+0000","last_unstale":"2024-09-13T15:05:55.144661+0000","last_undegraded":"2024-09-13T15:05:55.144661+0000","last_fullsized":"2024-09-13T15:05:55.144661+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.008264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.008264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:52:58.309554+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"1354'14635","reported_seq":14697,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:06:09.236262+0000","last_change":"2024-09-13T15:02:45.493742+0000","last_active":"2024-09-13T15:06:09.236262+0000","last_peered":"2024-09-13T15:06:09.236262+0000","last_clean":"2024-09-13T15:06:09.236262+0000","last_became_active":"2024-09-13T15:02:45.493391+0000","last_became_peered":"2024-09-13T15:02:45.493391+0000","last_unstale":"2024-09-13T15:06:09.236262+0000","last_undegraded":"2024-09-13T15:06:09.236262+0000","last_fullsized":"2024-09-13T15:06:09.236262+0000","mapping_epoch":1329,"log_start":"1348'10252","ondisk_log_start":"1348'10252","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":4383,"log_dups_size":0,"ondisk_log_size":4383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:26.044885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18284544,"num_objects":279,"num_object_clones":0,"num_object_copies":837,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14635,"num_write_kb":477248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":144,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.134696+0000","last_change":"2024-09-13T15:00:25.040768+0000","last_active":"2024-09-13T15:05:54.134696+0000","last_peered":"2024-09-13T15:05:54.134696+0000","last_clean":"2024-09-13T15:05:54.134696+0000","last_became_active":"2024-09-13T15:00:25.040487+0000","last_became_peered":"2024-09-13T15:00:25.040487+0000","last_unstale":"2024-09-13T15:05:54.134696+0000","last_undegraded":"2024-09-13T15:05:54.134696+0000","last_fullsized":"2024-09-13T15:05:54.134696+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:44.500104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":466,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.180645+0000","last_change":"2024-09-13T14:55:41.835887+0000","last_active":"2024-09-13T15:05:55.180645+0000","last_peered":"2024-09-13T15:05:55.180645+0000","last_clean":"2024-09-13T15:05:55.180645+0000","last_became_active":"2024-09-13T14:55:41.834413+0000","last_became_peered":"2024-09-13T14:55:41.834413+0000","last_unstale":"2024-09-13T15:05:55.180645+0000","last_undegraded":"2024-09-13T15:05:55.180645+0000","last_fullsized":"2024-09-13T15:05:55.180645+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:01.850578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":210,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.138292+0000","last_change":"2024-09-13T15:03:40.111604+0000","last_active":"2024-09-13T15:05:54.138292+0000","last_peered":"2024-09-13T15:05:54.138292+0000","last_clean":"2024-09-13T15:05:54.138292+0000","last_became_active":"2024-09-13T15:03:40.100430+0000","last_became_peered":"2024-09-13T15:03:40.100430+0000","last_unstale":"2024-09-13T15:05:54.138292+0000","last_undegraded":"2024-09-13T15:05:54.138292+0000","last_fullsized":"2024-09-13T15:05:54.138292+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:06:45.959941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"261'100","reported_seq":36,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.133177+0000","last_change":"2024-09-13T15:02:42.884606+0000","last_active":"2024-09-13T15:05:54.133177+0000","last_peered":"2024-09-13T15:05:54.133177+0000","last_clean":"2024-09-13T15:05:54.133177+0000","last_became_active":"2024-09-13T15:02:42.884269+0000","last_became_peered":"2024-09-13T15:02:42.884269+0000","last_unstale":"2024-09-13T15:05:54.133177+0000","last_undegraded":"2024-09-13T15:05:54.133177+0000","last_fullsized":"2024-09-13T15:05:54.133177+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T15:00:44.119431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T15:00:44.119431+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:06:31.424756+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.0","version":"1354'14587","reported_seq":14649,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:06:07.822537+0000","last_change":"2024-09-13T15:02:44.896248+0000","last_active":"2024-09-13T15:06:07.822537+0000","last_peered":"2024-09-13T15:06:07.822537+0000","last_clean":"2024-09-13T15:06:07.822537+0000","last_became_active":"2024-09-13T15:02:44.895577+0000","last_became_peered":"2024-09-13T15:02:44.895577+0000","last_unstale":"2024-09-13T15:06:07.822537+0000","last_undegraded":"2024-09-13T15:06:07.822537+0000","last_fullsized":"2024-09-13T15:06:07.822537+0000","mapping_epoch":1329,"log_start":"1348'10780","ondisk_log_start":"1348'10780","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3807,"log_dups_size":0,"ondisk_log_size":3807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:20:42.078814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23658496,"num_objects":361,"num_object_clones":0,"num_object_copies":1083,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14587,"num_write_kb":478336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":300,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.144742+0000","last_change":"2024-09-13T15:02:42.896422+0000","last_active":"2024-09-13T15:05:55.144742+0000","last_peered":"2024-09-13T15:05:55.144742+0000","last_clean":"2024-09-13T15:05:55.144742+0000","last_became_active":"2024-09-13T15:02:42.896112+0000","last_became_peered":"2024-09-13T15:02:42.896112+0000","last_unstale":"2024-09-13T15:05:55.144742+0000","last_undegraded":"2024-09-13T15:05:55.144742+0000","last_fullsized":"2024-09-13T15:05:55.144742+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:36:04.129918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":473,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.178954+0000","last_change":"2024-09-13T15:02:42.901428+0000","last_active":"2024-09-13T15:05:55.178954+0000","last_peered":"2024-09-13T15:05:55.178954+0000","last_clean":"2024-09-13T15:05:55.178954+0000","last_became_active":"2024-09-13T15:02:42.901131+0000","last_became_peered":"2024-09-13T15:02:42.901131+0000","last_unstale":"2024-09-13T15:05:55.178954+0000","last_undegraded":"2024-09-13T15:05:55.178954+0000","last_fullsized":"2024-09-13T15:05:55.178954+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:44.152548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"261'100","reported_seq":137,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.138073+0000","last_change":"2024-09-13T15:03:40.107555+0000","last_active":"2024-09-13T15:05:54.138073+0000","last_peered":"2024-09-13T15:05:54.138073+0000","last_clean":"2024-09-13T15:05:54.138073+0000","last_became_active":"2024-09-13T15:03:40.107230+0000","last_became_peered":"2024-09-13T15:03:40.107230+0000","last_unstale":"2024-09-13T15:05:54.138073+0000","last_undegraded":"2024-09-13T15:05:54.138073+0000","last_fullsized":"2024-09-13T15:05:54.138073+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:49.280478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:49.280478+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:13:24.031951+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"1354'15052","reported_seq":15145,"reported_epoch":1354,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T15:06:09.222507+0000","last_change":"2024-09-13T15:03:41.291109+0000","last_active":"2024-09-13T15:06:09.222507+0000","last_peered":"2024-09-13T15:06:09.222507+0000","last_clean":"2024-09-13T15:03:40.030901+0000","last_became_active":"2024-09-13T15:03:41.221238+0000","last_became_peered":"2024-09-13T15:03:41.221238+0000","last_unstale":"2024-09-13T15:06:09.222507+0000","last_undegraded":"2024-09-13T15:06:09.222507+0000","last_fullsized":"2024-09-13T15:06:09.222507+0000","mapping_epoch":1340,"log_start":"1348'10674","ondisk_log_start":"1348'10674","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":4378,"log_dups_size":0,"ondisk_log_size":4378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18087936,"num_objects":276,"num_object_clones":0,"num_object_copies":828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":275,"num_objects_unfound":0,"num_objects_dirty":276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15052,"num_write_kb":490496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,2],"avail_no_missing":["3","2","6"],"object_location_counts":[{"shards":"2,3,6","objects":276}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":164,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.145181+0000","last_change":"2024-09-13T15:00:25.069428+0000","last_active":"2024-09-13T15:05:55.145181+0000","last_peered":"2024-09-13T15:05:55.145181+0000","last_clean":"2024-09-13T15:05:55.145181+0000","last_became_active":"2024-09-13T15:00:25.068745+0000","last_became_peered":"2024-09-13T15:00:25.068745+0000","last_unstale":"2024-09-13T15:05:55.145181+0000","last_undegraded":"2024-09-13T15:05:55.145181+0000","last_fullsized":"2024-09-13T15:05:55.145181+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:37.264986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":785,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.131776+0000","last_change":"2024-09-13T15:03:40.106188+0000","last_active":"2024-09-13T15:05:54.131776+0000","last_peered":"2024-09-13T15:05:54.131776+0000","last_clean":"2024-09-13T15:05:54.131776+0000","last_became_active":"2024-09-13T15:03:40.105764+0000","last_became_peered":"2024-09-13T15:03:40.105764+0000","last_unstale":"2024-09-13T15:05:54.131776+0000","last_undegraded":"2024-09-13T15:05:54.131776+0000","last_fullsized":"2024-09-13T15:05:54.131776+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:36:28.533954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"261'100","reported_seq":307,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.144757+0000","last_change":"2024-09-13T15:03:40.111680+0000","last_active":"2024-09-13T15:05:55.144757+0000","last_peered":"2024-09-13T15:05:55.144757+0000","last_clean":"2024-09-13T15:05:55.144757+0000","last_became_active":"2024-09-13T15:03:40.111289+0000","last_became_peered":"2024-09-13T15:03:40.111289+0000","last_unstale":"2024-09-13T15:05:55.144757+0000","last_undegraded":"2024-09-13T15:05:55.144757+0000","last_fullsized":"2024-09-13T15:05:55.144757+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:47.405845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:47.405845+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:14.271742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2","version":"1354'14607","reported_seq":14670,"reported_epoch":1354,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T15:06:09.224476+0000","last_change":"2024-09-13T15:03:41.290486+0000","last_active":"2024-09-13T15:06:09.224476+0000","last_peered":"2024-09-13T15:06:09.224476+0000","last_clean":"2024-09-13T15:03:40.019998+0000","last_became_active":"2024-09-13T15:03:41.277434+0000","last_became_peered":"2024-09-13T15:03:41.277434+0000","last_unstale":"2024-09-13T15:06:09.224476+0000","last_undegraded":"2024-09-13T15:03:41.124667+0000","last_fullsized":"2024-09-13T15:03:41.087359+0000","mapping_epoch":1340,"log_start":"1348'10228","ondisk_log_start":"1348'10228","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":4379,"log_dups_size":0,"ondisk_log_size":4379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18415616,"num_objects":281,"num_object_clones":0,"num_object_copies":843,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14607,"num_write_kb":476416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7],"avail_no_missing":["3","2","7"],"object_location_counts":[{"shards":"3,7","objects":-4494}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1272,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.144083+0000","last_change":"2024-09-13T15:03:40.114962+0000","last_active":"2024-09-13T15:05:55.144083+0000","last_peered":"2024-09-13T15:05:55.144083+0000","last_clean":"2024-09-13T15:05:55.144083+0000","last_became_active":"2024-09-13T15:03:40.112298+0000","last_became_peered":"2024-09-13T15:03:40.112298+0000","last_unstale":"2024-09-13T15:05:55.144083+0000","last_undegraded":"2024-09-13T15:05:55.144083+0000","last_fullsized":"2024-09-13T15:05:55.144083+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:35:05.699786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":164,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.143771+0000","last_change":"2024-09-13T15:00:25.154369+0000","last_active":"2024-09-13T15:05:55.143771+0000","last_peered":"2024-09-13T15:05:55.143771+0000","last_clean":"2024-09-13T15:05:55.143771+0000","last_became_active":"2024-09-13T15:00:25.154045+0000","last_became_peered":"2024-09-13T15:00:25.154045+0000","last_unstale":"2024-09-13T15:05:55.143771+0000","last_undegraded":"2024-09-13T15:05:55.143771+0000","last_fullsized":"2024-09-13T15:05:55.143771+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:58:48.342494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"261'100","reported_seq":137,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.137488+0000","last_change":"2024-09-13T15:03:40.102353+0000","last_active":"2024-09-13T15:05:54.137488+0000","last_peered":"2024-09-13T15:05:54.137488+0000","last_clean":"2024-09-13T15:05:54.137488+0000","last_became_active":"2024-09-13T15:03:40.101656+0000","last_became_peered":"2024-09-13T15:03:40.101656+0000","last_unstale":"2024-09-13T15:05:54.137488+0000","last_undegraded":"2024-09-13T15:05:54.137488+0000","last_fullsized":"2024-09-13T15:05:54.137488+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:55.816857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:55.816857+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:54.772898+0000","scrub_duration":270,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3","version":"1354'14812","reported_seq":14845,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:06:07.377880+0000","last_change":"2024-09-13T15:02:45.493751+0000","last_active":"2024-09-13T15:06:07.377880+0000","last_peered":"2024-09-13T15:06:07.377880+0000","last_clean":"2024-09-13T15:06:07.377880+0000","last_became_active":"2024-09-13T15:02:45.493377+0000","last_became_peered":"2024-09-13T15:02:45.493377+0000","last_unstale":"2024-09-13T15:06:07.377880+0000","last_undegraded":"2024-09-13T15:06:07.377880+0000","last_fullsized":"2024-09-13T15:06:07.377880+0000","mapping_epoch":1329,"log_start":"1348'10311","ondisk_log_start":"1348'10311","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":4501,"log_dups_size":0,"ondisk_log_size":4501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:49.207437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26476544,"num_objects":404,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14812,"num_write_kb":486912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1400,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.179324+0000","last_change":"2024-09-13T15:00:25.045581+0000","last_active":"2024-09-13T15:05:55.179324+0000","last_peered":"2024-09-13T15:05:55.179324+0000","last_clean":"2024-09-13T15:05:55.179324+0000","last_became_active":"2024-09-13T15:00:25.045225+0000","last_became_peered":"2024-09-13T15:00:25.045225+0000","last_unstale":"2024-09-13T15:05:55.179324+0000","last_undegraded":"2024-09-13T15:05:55.179324+0000","last_fullsized":"2024-09-13T15:05:55.179324+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:38.148339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":512,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.178953+0000","last_change":"2024-09-13T15:02:42.909748+0000","last_active":"2024-09-13T15:05:55.178953+0000","last_peered":"2024-09-13T15:05:55.178953+0000","last_clean":"2024-09-13T15:05:55.178953+0000","last_became_active":"2024-09-13T15:02:42.897824+0000","last_became_peered":"2024-09-13T15:02:42.897824+0000","last_unstale":"2024-09-13T15:05:55.178953+0000","last_undegraded":"2024-09-13T15:05:55.178953+0000","last_fullsized":"2024-09-13T15:05:55.178953+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:05.224186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"261'100","reported_seq":56,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.131984+0000","last_change":"2024-09-13T15:03:40.101983+0000","last_active":"2024-09-13T15:05:54.131984+0000","last_peered":"2024-09-13T15:05:54.131984+0000","last_clean":"2024-09-13T15:05:54.131984+0000","last_became_active":"2024-09-13T15:03:40.101141+0000","last_became_peered":"2024-09-13T15:03:40.101141+0000","last_unstale":"2024-09-13T15:05:54.131984+0000","last_undegraded":"2024-09-13T15:05:54.131984+0000","last_fullsized":"2024-09-13T15:05:54.131984+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T15:02:56.904689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T15:02:56.904689+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:20:01.949481+0000","scrub_duration":0.001699702,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.4","version":"1354'14598","reported_seq":14642,"reported_epoch":1354,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T15:06:07.378112+0000","last_change":"2024-09-13T15:03:41.226259+0000","last_active":"2024-09-13T15:06:07.378112+0000","last_peered":"2024-09-13T15:06:07.378112+0000","last_clean":"2024-09-13T15:03:40.015690+0000","last_became_active":"2024-09-13T15:03:41.221116+0000","last_became_peered":"2024-09-13T15:03:41.221116+0000","last_unstale":"2024-09-13T15:06:07.378112+0000","last_undegraded":"2024-09-13T15:06:07.378112+0000","last_fullsized":"2024-09-13T15:06:07.378112+0000","mapping_epoch":1340,"log_start":"1348'10094","ondisk_log_start":"1348'10094","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":4504,"log_dups_size":0,"ondisk_log_size":4504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:47:40.568978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28311552,"num_objects":432,"num_object_clones":0,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":432,"num_objects_unfound":0,"num_objects_dirty":432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14598,"num_write_kb":480960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,2],"avail_no_missing":["7","2","6"],"object_location_counts":[{"shards":"2,6,7","objects":432}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":41,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.144613+0000","last_change":"2024-09-13T15:03:40.115749+0000","last_active":"2024-09-13T15:05:55.144613+0000","last_peered":"2024-09-13T15:05:55.144613+0000","last_clean":"2024-09-13T15:05:55.144613+0000","last_became_active":"2024-09-13T15:03:40.114427+0000","last_became_peered":"2024-09-13T15:03:40.114427+0000","last_unstale":"2024-09-13T15:05:55.144613+0000","last_undegraded":"2024-09-13T15:05:55.144613+0000","last_fullsized":"2024-09-13T15:05:55.144613+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:45:15.328660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":127,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.138008+0000","last_change":"2024-09-13T14:55:41.848811+0000","last_active":"2024-09-13T15:05:54.138008+0000","last_peered":"2024-09-13T15:05:54.138008+0000","last_clean":"2024-09-13T15:05:54.138008+0000","last_became_active":"2024-09-13T14:55:41.848542+0000","last_became_peered":"2024-09-13T14:55:41.848542+0000","last_unstale":"2024-09-13T15:05:54.138008+0000","last_undegraded":"2024-09-13T15:05:54.138008+0000","last_fullsized":"2024-09-13T15:05:54.138008+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:48.534022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"261'100","reported_seq":56,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.134377+0000","last_change":"2024-09-13T15:03:40.108123+0000","last_active":"2024-09-13T15:05:54.134377+0000","last_peered":"2024-09-13T15:05:54.134377+0000","last_clean":"2024-09-13T15:05:54.134377+0000","last_became_active":"2024-09-13T15:03:40.107745+0000","last_became_peered":"2024-09-13T15:03:40.107745+0000","last_unstale":"2024-09-13T15:05:54.134377+0000","last_undegraded":"2024-09-13T15:05:54.134377+0000","last_fullsized":"2024-09-13T15:05:54.134377+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T15:03:00.851236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T15:03:00.851236+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:52.994128+0000","scrub_duration":0.004159968,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.5","version":"1354'14961","reported_seq":14994,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:06:07.408915+0000","last_change":"2024-09-13T15:02:45.494451+0000","last_active":"2024-09-13T15:06:07.408915+0000","last_peered":"2024-09-13T15:06:07.408915+0000","last_clean":"2024-09-13T15:06:07.408915+0000","last_became_active":"2024-09-13T15:02:45.494169+0000","last_became_peered":"2024-09-13T15:02:45.494169+0000","last_unstale":"2024-09-13T15:06:07.408915+0000","last_undegraded":"2024-09-13T15:06:07.408915+0000","last_fullsized":"2024-09-13T15:06:07.408915+0000","mapping_epoch":1329,"log_start":"1348'10547","ondisk_log_start":"1348'10547","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":4414,"log_dups_size":0,"ondisk_log_size":4414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:45:17.495874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28770304,"num_objects":439,"num_object_clones":0,"num_object_copies":1317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14961,"num_write_kb":492800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1370,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.136342+0000","last_change":"2024-09-13T15:02:42.880814+0000","last_active":"2024-09-13T15:05:54.136342+0000","last_peered":"2024-09-13T15:05:54.136342+0000","last_clean":"2024-09-13T15:05:54.136342+0000","last_became_active":"2024-09-13T15:02:42.880393+0000","last_became_peered":"2024-09-13T15:02:42.880393+0000","last_unstale":"2024-09-13T15:05:54.136342+0000","last_undegraded":"2024-09-13T15:05:54.136342+0000","last_fullsized":"2024-09-13T15:05:54.136342+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:03.222968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":40,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.134619+0000","last_change":"2024-09-13T15:02:42.914851+0000","last_active":"2024-09-13T15:05:54.134619+0000","last_peered":"2024-09-13T15:05:54.134619+0000","last_clean":"2024-09-13T15:05:54.134619+0000","last_became_active":"2024-09-13T15:02:42.884164+0000","last_became_peered":"2024-09-13T15:02:42.884164+0000","last_unstale":"2024-09-13T15:05:54.134619+0000","last_undegraded":"2024-09-13T15:05:54.134619+0000","last_fullsized":"2024-09-13T15:05:54.134619+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:26:54.412326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"261'100","reported_seq":137,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.139030+0000","last_change":"2024-09-13T15:00:25.048330+0000","last_active":"2024-09-13T15:05:54.139030+0000","last_peered":"2024-09-13T15:05:54.139030+0000","last_clean":"2024-09-13T15:05:54.139030+0000","last_became_active":"2024-09-13T15:00:25.048001+0000","last_became_peered":"2024-09-13T15:00:25.048001+0000","last_unstale":"2024-09-13T15:05:54.139030+0000","last_undegraded":"2024-09-13T15:05:54.139030+0000","last_fullsized":"2024-09-13T15:05:54.139030+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:49:21.722939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:49:21.722939+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:11:39.620088+0000","scrub_duration":0.0016642790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"1354'14785","reported_seq":24697,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:06:07.802852+0000","last_change":"2024-09-13T15:05:55.174189+0000","last_active":"2024-09-13T15:06:07.802852+0000","last_peered":"2024-09-13T15:06:07.802852+0000","last_clean":"2024-09-13T15:06:07.802852+0000","last_became_active":"2024-09-13T15:05:55.148549+0000","last_became_peered":"2024-09-13T15:05:55.148549+0000","last_unstale":"2024-09-13T15:06:07.802852+0000","last_undegraded":"2024-09-13T15:06:07.802852+0000","last_fullsized":"2024-09-13T15:06:07.802852+0000","mapping_epoch":1353,"log_start":"1348'10969","ondisk_log_start":"1348'10969","created":1329,"last_epoch_clean":1354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":3816,"log_dups_size":0,"ondisk_log_size":3816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:35.600932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26279936,"num_objects":401,"num_object_clones":0,"num_object_copies":1203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14785,"num_write_kb":485952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4919,"num_bytes_recovered":210829312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":41,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.144359+0000","last_change":"2024-09-13T15:03:40.116623+0000","last_active":"2024-09-13T15:05:55.144359+0000","last_peered":"2024-09-13T15:05:55.144359+0000","last_clean":"2024-09-13T15:05:55.144359+0000","last_became_active":"2024-09-13T15:03:40.116283+0000","last_became_peered":"2024-09-13T15:03:40.116283+0000","last_unstale":"2024-09-13T15:05:55.144359+0000","last_undegraded":"2024-09-13T15:05:55.144359+0000","last_fullsized":"2024-09-13T15:05:55.144359+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:00.215206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":40,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.134250+0000","last_change":"2024-09-13T15:02:42.919448+0000","last_active":"2024-09-13T15:05:54.134250+0000","last_peered":"2024-09-13T15:05:54.134250+0000","last_clean":"2024-09-13T15:05:54.134250+0000","last_became_active":"2024-09-13T15:02:42.889750+0000","last_became_peered":"2024-09-13T15:02:42.889750+0000","last_unstale":"2024-09-13T15:05:54.134250+0000","last_undegraded":"2024-09-13T15:05:54.134250+0000","last_fullsized":"2024-09-13T15:05:54.134250+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:25.834089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"261'100","reported_seq":460,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.179471+0000","last_change":"2024-09-13T15:00:25.213778+0000","last_active":"2024-09-13T15:05:55.179471+0000","last_peered":"2024-09-13T15:05:55.179471+0000","last_clean":"2024-09-13T15:05:55.179471+0000","last_became_active":"2024-09-13T15:00:25.047003+0000","last_became_peered":"2024-09-13T15:00:25.047003+0000","last_unstale":"2024-09-13T15:05:55.179471+0000","last_undegraded":"2024-09-13T15:05:55.179471+0000","last_fullsized":"2024-09-13T15:05:55.179471+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.597568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.597568+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:20.589087+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"1354'14877","reported_seq":14910,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:06:07.570926+0000","last_change":"2024-09-13T15:02:44.890783+0000","last_active":"2024-09-13T15:06:07.570926+0000","last_peered":"2024-09-13T15:06:07.570926+0000","last_clean":"2024-09-13T15:06:07.570926+0000","last_became_active":"2024-09-13T15:02:44.890448+0000","last_became_peered":"2024-09-13T15:02:44.890448+0000","last_unstale":"2024-09-13T15:06:07.570926+0000","last_undegraded":"2024-09-13T15:06:07.570926+0000","last_fullsized":"2024-09-13T15:06:07.570926+0000","mapping_epoch":1329,"log_start":"1348'10477","ondisk_log_start":"1348'10477","created":1329,"last_epoch_clean":1330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:02:43.870821+0000","last_clean_scrub_stamp":"2024-09-13T15:02:43.870821+0000","objects_scrubbed":0,"log_size":4400,"log_dups_size":0,"ondisk_log_size":4400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:58.783856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624576,"num_objects":391,"num_object_clones":0,"num_object_copies":1173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14877,"num_write_kb":488576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":157,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.137644+0000","last_change":"2024-09-13T15:02:42.885662+0000","last_active":"2024-09-13T15:05:54.137644+0000","last_peered":"2024-09-13T15:05:54.137644+0000","last_clean":"2024-09-13T15:05:54.137644+0000","last_became_active":"2024-09-13T15:02:42.885326+0000","last_became_peered":"2024-09-13T15:02:42.885326+0000","last_unstale":"2024-09-13T15:05:54.137644+0000","last_undegraded":"2024-09-13T15:05:54.137644+0000","last_fullsized":"2024-09-13T15:05:54.137644+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:11.631110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":40,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.135469+0000","last_change":"2024-09-13T15:02:42.917286+0000","last_active":"2024-09-13T15:05:54.135469+0000","last_peered":"2024-09-13T15:05:54.135469+0000","last_clean":"2024-09-13T15:05:54.135469+0000","last_became_active":"2024-09-13T15:02:42.888048+0000","last_became_peered":"2024-09-13T15:02:42.888048+0000","last_unstale":"2024-09-13T15:05:54.135469+0000","last_undegraded":"2024-09-13T15:05:54.135469+0000","last_fullsized":"2024-09-13T15:05:54.135469+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:23:37.915522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"261'100","reported_seq":46,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.134167+0000","last_change":"2024-09-13T15:03:40.103528+0000","last_active":"2024-09-13T15:05:54.134167+0000","last_peered":"2024-09-13T15:05:54.134167+0000","last_clean":"2024-09-13T15:05:54.134167+0000","last_became_active":"2024-09-13T15:03:40.102933+0000","last_became_peered":"2024-09-13T15:03:40.102933+0000","last_unstale":"2024-09-13T15:05:54.134167+0000","last_undegraded":"2024-09-13T15:05:54.134167+0000","last_fullsized":"2024-09-13T15:05:54.134167+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.191326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.191326+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:10.236210+0000","scrub_duration":0.0025317360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1399,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.136425+0000","last_change":"2024-09-13T15:00:05.058713+0000","last_active":"2024-09-13T15:05:54.136425+0000","last_peered":"2024-09-13T15:05:54.136425+0000","last_clean":"2024-09-13T15:05:54.136425+0000","last_became_active":"2024-09-13T15:00:05.034868+0000","last_became_peered":"2024-09-13T15:00:05.034868+0000","last_unstale":"2024-09-13T15:05:54.136425+0000","last_undegraded":"2024-09-13T15:05:54.136425+0000","last_fullsized":"2024-09-13T15:05:54.136425+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:32:10.399295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":458,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.181378+0000","last_change":"2024-09-13T15:02:42.899236+0000","last_active":"2024-09-13T15:05:55.181378+0000","last_peered":"2024-09-13T15:05:55.181378+0000","last_clean":"2024-09-13T15:05:55.181378+0000","last_became_active":"2024-09-13T15:02:42.897548+0000","last_became_peered":"2024-09-13T15:02:42.897548+0000","last_unstale":"2024-09-13T15:05:55.181378+0000","last_undegraded":"2024-09-13T15:05:55.181378+0000","last_fullsized":"2024-09-13T15:05:55.181378+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:04:04.344226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"261'100","reported_seq":127,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.138685+0000","last_change":"2024-09-13T14:55:41.889497+0000","last_active":"2024-09-13T15:05:54.138685+0000","last_peered":"2024-09-13T15:05:54.138685+0000","last_clean":"2024-09-13T15:05:54.138685+0000","last_became_active":"2024-09-13T14:55:41.888766+0000","last_became_peered":"2024-09-13T14:55:41.888766+0000","last_unstale":"2024-09-13T15:05:54.138685+0000","last_undegraded":"2024-09-13T15:05:54.138685+0000","last_fullsized":"2024-09-13T15:05:54.138685+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:55.378404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:55.378404+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:50.289550+0000","scrub_duration":0.0016008439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":36,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.132112+0000","last_change":"2024-09-13T15:02:42.894522+0000","last_active":"2024-09-13T15:05:54.132112+0000","last_peered":"2024-09-13T15:05:54.132112+0000","last_clean":"2024-09-13T15:05:54.132112+0000","last_became_active":"2024-09-13T15:02:42.893100+0000","last_became_peered":"2024-09-13T15:02:42.893100+0000","last_unstale":"2024-09-13T15:05:54.132112+0000","last_undegraded":"2024-09-13T15:05:54.132112+0000","last_fullsized":"2024-09-13T15:05:54.132112+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:25.582063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2113,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.180683+0000","last_change":"2024-09-13T15:03:40.101348+0000","last_active":"2024-09-13T15:05:55.180683+0000","last_peered":"2024-09-13T15:05:55.180683+0000","last_clean":"2024-09-13T15:05:55.180683+0000","last_became_active":"2024-09-13T15:03:40.100990+0000","last_became_peered":"2024-09-13T15:03:40.100990+0000","last_unstale":"2024-09-13T15:05:55.180683+0000","last_undegraded":"2024-09-13T15:05:55.180683+0000","last_fullsized":"2024-09-13T15:05:55.180683+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:20:40.865047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"261'100","reported_seq":293,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.144477+0000","last_change":"2024-09-13T15:00:25.087759+0000","last_active":"2024-09-13T15:05:55.144477+0000","last_peered":"2024-09-13T15:05:55.144477+0000","last_clean":"2024-09-13T15:05:55.144477+0000","last_became_active":"2024-09-13T15:00:25.087315+0000","last_became_peered":"2024-09-13T15:00:25.087315+0000","last_unstale":"2024-09-13T15:05:55.144477+0000","last_undegraded":"2024-09-13T15:05:55.144477+0000","last_fullsized":"2024-09-13T15:05:55.144477+0000","mapping_epoch":1288,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1289,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:19.978464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:19.978464+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:22:19.789250+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":36,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.132574+0000","last_change":"2024-09-13T15:02:42.894144+0000","last_active":"2024-09-13T15:05:54.132574+0000","last_peered":"2024-09-13T15:05:54.132574+0000","last_clean":"2024-09-13T15:05:54.132574+0000","last_became_active":"2024-09-13T15:02:42.893679+0000","last_became_peered":"2024-09-13T15:02:42.893679+0000","last_unstale":"2024-09-13T15:05:54.132574+0000","last_undegraded":"2024-09-13T15:05:54.132574+0000","last_fullsized":"2024-09-13T15:05:54.132574+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:05.614138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":147,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.145918+0000","last_change":"2024-09-13T15:02:42.886038+0000","last_active":"2024-09-13T15:05:54.145918+0000","last_peered":"2024-09-13T15:05:54.145918+0000","last_clean":"2024-09-13T15:05:54.145918+0000","last_became_active":"2024-09-13T15:02:42.884438+0000","last_became_peered":"2024-09-13T15:02:42.884438+0000","last_unstale":"2024-09-13T15:05:54.145918+0000","last_undegraded":"2024-09-13T15:05:54.145918+0000","last_fullsized":"2024-09-13T15:05:54.145918+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:47:39.175275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"261'100","reported_seq":157,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.137570+0000","last_change":"2024-09-13T15:02:42.885131+0000","last_active":"2024-09-13T15:05:54.137570+0000","last_peered":"2024-09-13T15:05:54.137570+0000","last_clean":"2024-09-13T15:05:54.137570+0000","last_became_active":"2024-09-13T15:02:42.883904+0000","last_became_peered":"2024-09-13T15:02:42.883904+0000","last_unstale":"2024-09-13T15:05:54.137570+0000","last_undegraded":"2024-09-13T15:05:54.137570+0000","last_fullsized":"2024-09-13T15:05:54.137570+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:50.327393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:50.327393+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:11:05.603544+0000","scrub_duration":0.0011508309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":58,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.134060+0000","last_change":"2024-09-13T15:03:40.127299+0000","last_active":"2024-09-13T15:05:54.134060+0000","last_peered":"2024-09-13T15:05:54.134060+0000","last_clean":"2024-09-13T15:05:54.134060+0000","last_became_active":"2024-09-13T15:03:40.108555+0000","last_became_peered":"2024-09-13T15:03:40.108555+0000","last_unstale":"2024-09-13T15:05:54.134060+0000","last_undegraded":"2024-09-13T15:05:54.134060+0000","last_fullsized":"2024-09-13T15:05:54.134060+0000","mapping_epoch":1339,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:13.009809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2065,"reported_epoch":1354,"state":"active+clean","last_fresh":"2024-09-13T15:05:55.181068+0000","last_change":"2024-09-13T15:02:42.896774+0000","last_active":"2024-09-13T15:05:55.181068+0000","last_peered":"2024-09-13T15:05:55.181068+0000","last_clean":"2024-09-13T15:05:55.181068+0000","last_became_active":"2024-09-13T15:02:42.896459+0000","last_became_peered":"2024-09-13T15:02:42.896459+0000","last_unstale":"2024-09-13T15:05:55.181068+0000","last_undegraded":"2024-09-13T15:05:55.181068+0000","last_fullsized":"2024-09-13T15:05:55.181068+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:35.279246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"261'100","reported_seq":675,"reported_epoch":1353,"state":"active+clean","last_fresh":"2024-09-13T15:05:54.138397+0000","last_change":"2024-09-13T15:00:05.038303+0000","last_active":"2024-09-13T15:05:54.138397+0000","last_peered":"2024-09-13T15:05:54.138397+0000","last_clean":"2024-09-13T15:05:54.138397+0000","last_became_active":"2024-09-13T15:00:05.037979+0000","last_became_peered":"2024-09-13T15:00:05.037979+0000","last_unstale":"2024-09-13T15:05:54.138397+0000","last_undegraded":"2024-09-13T15:05:54.138397+0000","last_fullsized":"2024-09-13T15:05:54.138397+0000","mapping_epoch":1284,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1285,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:14.654603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:14.654603+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:28.168890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":372047896,"num_objects":5678,"num_object_clones":0,"num_object_copies":17034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9105,"num_objects_misplaced":1081,"num_objects_unfound":0,"num_objects_dirty":5678,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":236912,"num_write_kb":7762817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11526,"num_bytes_recovered":599851008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2357080064,"data_stored":2357067848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68233,"ondisk_log_size":68233,"up":48,"acting":46,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":22576,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":27510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2296480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2703360,"data_stored":2689152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430793,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1751236,"kb_used_data":261948,"kb_used_omap":1,"kb_used_meta":1489278,"kb_avail":91993916,"statfs":{"total":95995035648,"available":94201769984,"internally_reserved":0,"allocated":268234752,"data_stored":265065896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1525021016},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388627022,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1757192,"kb_used_data":270208,"kb_used_omap":1,"kb_used_meta":1486974,"kb_avail":91987960,"statfs":{"total":95995035648,"available":94195671040,"internally_reserved":0,"allocated":276692992,"data_stored":273520087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1522661720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823251,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1715528,"kb_used_data":221568,"kb_used_omap":1,"kb_used_meta":1493950,"kb_avail":92029624,"statfs":{"total":95995035648,"available":94238334976,"internally_reserved":0,"allocated":226885632,"data_stored":223713169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1529805144},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019479,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059593137,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1207852,"kb_used_data":173476,"kb_used_omap":26,"kb_used_meta":1034341,"kb_avail":92537300,"statfs":{"total":95995035648,"available":94758195200,"internally_reserved":0,"allocated":177639424,"data_stored":174490312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":1059165344},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2538325672375,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":557,"seq":2392296784318,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1443384,"kb_used_data":289456,"kb_used_omap":32,"kb_used_meta":1153887,"kb_avail":92301768,"statfs":{"total":95995035648,"available":94517010432,"internally_reserved":0,"allocated":296402944,"data_stored":293241527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33752,"internal_metadata":1181580328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":2250562863557,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":276959232,"data_stored":276955160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1251606528,"data_stored":1251606528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":166133760,"data_stored":166133760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":188354560,"data_stored":188350488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":240390144,"data_stored":240386072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":233635840,"data_stored":233635840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:06:13.188 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_6 pgp_num 16 2024-09-13T15:06:13.222 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:13.244 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:13.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:13.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:13.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:13.368 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:13.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:13.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:13.470 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:13.481 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-13T15:06:13.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:13.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:13.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:13.596 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:13.618 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:13.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:13.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:13.720 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:13.741 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:13.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:13.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:13.759 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:13.843 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:13.849 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:13 smithi151 ceph-mon[104450]: pgmap v2363: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 355 MiB data, 7.5 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 967 op/s; 9105/17112 objects degraded (53.208%); 1081/17112 objects misplaced (6.317%); 0 B/s, 37 objects/s recovering 2024-09-13T15:06:13.849 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:13 smithi151 ceph-mon[104450]: from='client.36804 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:06:13.849 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:13 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2704523020' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:06:13.849 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:13 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:06:13.850 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:13 smithi151 ceph-mon[108966]: pgmap v2363: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 355 MiB data, 7.5 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 967 op/s; 9105/17112 objects degraded (53.208%); 1081/17112 objects misplaced (6.317%); 0 B/s, 37 objects/s recovering 2024-09-13T15:06:13.850 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:13 smithi151 ceph-mon[108966]: from='client.36804 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:06:13.850 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:13 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2704523020' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:06:13.850 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:13 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:06:13.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:13 smithi184 ceph-mon[92527]: pgmap v2363: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 355 MiB data, 7.5 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 967 op/s; 9105/17112 objects degraded (53.208%); 1081/17112 objects misplaced (6.317%); 0 B/s, 37 objects/s recovering 2024-09-13T15:06:13.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:13 smithi184 ceph-mon[92527]: from='client.36804 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:06:13.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:13 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2704523020' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:06:13.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:13 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:06:13.865 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:13.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:13.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:13.967 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:13.988 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:13.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:13.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:14.090 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:14.111 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:14.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:14.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:14.213 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:14.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:14.241 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:14.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:14.342 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:14.363 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:14.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:14.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:14.465 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:14.487 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:14.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:14.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:14.589 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:14.610 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:14.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:14.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:14.712 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:14.734 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:14 smithi151 ceph-mon[104450]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:06:14.734 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:14 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:06:14.734 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:14 smithi151 ceph-mon[104450]: osdmap e1355: 8 total, 8 up, 5 in 2024-09-13T15:06:14.734 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:06:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:06:14] "GET /metrics HTTP/1.1" 200 36710 "" "Prometheus/2.51.0" 2024-09-13T15:06:14.734 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:14 smithi151 ceph-mon[108966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:06:14.735 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:14 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:06:14.735 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:14 smithi151 ceph-mon[108966]: osdmap e1355: 8 total, 8 up, 5 in 2024-09-13T15:06:14.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:14.736 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:14.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:14.837 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:14.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:14 smithi184 ceph-mon[92527]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:06:14.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:14 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:06:14.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:14 smithi184 ceph-mon[92527]: osdmap e1355: 8 total, 8 up, 5 in 2024-09-13T15:06:14.859 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:14.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:14.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:14.961 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:14.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:14.984 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:14.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:15.086 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:15.109 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:15.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:15.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:15.210 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:15.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:15.234 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:15.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:15.335 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:15.356 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:15.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:15.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:15.424 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T15:06:15.424 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-13T15:06:15.425 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-13T15:06:15.425 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T15:06:15.458 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:15.480 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:15.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:15.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:15.582 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:15.605 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:15.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:15.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:15.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:15 smithi151 ceph-mon[108966]: pgmap v2365: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 127 MiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 1.15k op/s; 8885/6141 objects degraded (144.683%); 327/6141 objects misplaced (5.325%); 0 B/s, 46 objects/s recovering 2024-09-13T15:06:15.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:15 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 8885/6141 objects degraded (144.683%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T15:06:15.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:15 smithi151 ceph-mon[104450]: pgmap v2365: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 127 MiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 1.15k op/s; 8885/6141 objects degraded (144.683%); 327/6141 objects misplaced (5.325%); 0 B/s, 46 objects/s recovering 2024-09-13T15:06:15.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:15 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 8885/6141 objects degraded (144.683%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T15:06:15.707 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:15.728 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:15.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:15.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:15.830 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:15.853 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:15.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:15 smithi184 ceph-mon[92527]: pgmap v2365: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 127 MiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 1.15k op/s; 8885/6141 objects degraded (144.683%); 327/6141 objects misplaced (5.325%); 0 B/s, 46 objects/s recovering 2024-09-13T15:06:15.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:15 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 8885/6141 objects degraded (144.683%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T15:06:15.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:15.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:15.955 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:15.977 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:15.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:15.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:16.078 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:16.101 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:16.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:16.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:16.160 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T15:06:16.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:16.208 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T15:06:16.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:16.226 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:16.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:16.235 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:16.327 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:16.351 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:16.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:16.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:16.453 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:16.475 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:16.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:16.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:16.577 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:16.598 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:16.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:16.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:16.700 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:16.721 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:16.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:16.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:16.823 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:16.845 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:16.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:16.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:16.947 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:16.971 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:16.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:16.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:17.073 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:17.095 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:17.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:17.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:17.197 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:17.219 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:17.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:17.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:17.303 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T15:06:17.303 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:06:17.311 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T15:06:17.312 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:06:17.320 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:17.340 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T15:06:17.340 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:06:17.343 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:17.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:17.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:17.347 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:17.357 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T15:06:17.357 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:06:17.373 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T15:06:17.373 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:06:17.389 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T15:06:17.389 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:06:17.404 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T15:06:17.404 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:06:17.420 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T15:06:17.420 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:06:17.445 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:17.467 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:17.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:17.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:17.569 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:17.591 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:17.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:17.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:17.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:17 smithi151 ceph-mon[104450]: pgmap v2366: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 19 MiB data, 6.5 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 1.10k op/s; 8885/960 objects degraded (925.521%); 52/960 objects misplaced (5.417%); 0 B/s, 46 objects/s recovering 2024-09-13T15:06:17.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:06:17.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:17.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:17 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2285888882' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:06:17.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:17 smithi151 ceph-mon[108966]: pgmap v2366: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 19 MiB data, 6.5 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 1.10k op/s; 8885/960 objects degraded (925.521%); 52/960 objects misplaced (5.417%); 0 B/s, 46 objects/s recovering 2024-09-13T15:06:17.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:06:17.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:17.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:17 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2285888882' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:06:17.692 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:17.713 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:17.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:17.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:17.815 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:17.837 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:17.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:17.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:17 smithi184 ceph-mon[92527]: pgmap v2366: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 19 MiB data, 6.5 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 1.10k op/s; 8885/960 objects degraded (925.521%); 52/960 objects misplaced (5.417%); 0 B/s, 46 objects/s recovering 2024-09-13T15:06:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:06:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:17.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:17 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2285888882' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:06:17.940 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:17.962 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:17.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:17.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:18.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:18.085 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:18.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:18.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:18.187 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:18.209 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:18.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:18.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:18.310 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:18.333 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:18.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:18.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:18.438 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:18.460 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:18.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:18.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:18.562 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:18.585 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:18.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:18.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:18.686 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:18.709 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:18.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:18.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:18.811 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:18.832 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:18.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:18.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:18.934 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:18.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:18.957 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:18.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:19.058 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:19.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:19.080 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:19.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:19.181 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:19.203 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:19.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:19.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:19.305 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:19.327 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:19.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:19.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:19.429 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:19.452 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:19.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:19.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:19.554 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:19.576 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:19.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:19.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:19.678 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:19.700 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:19.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:19.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:19.802 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:19.825 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:19.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:19.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:19.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:19 smithi184 ceph-mon[92527]: pgmap v2367: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 773 op/s; 7926/78 objects degraded (10161.538%); 0 B/s, 117 objects/s recovering 2024-09-13T15:06:19.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:19 smithi151 ceph-mon[104450]: pgmap v2367: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 773 op/s; 7926/78 objects degraded (10161.538%); 0 B/s, 117 objects/s recovering 2024-09-13T15:06:19.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:19 smithi151 ceph-mon[108966]: pgmap v2367: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 773 op/s; 7926/78 objects degraded (10161.538%); 0 B/s, 117 objects/s recovering 2024-09-13T15:06:19.927 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:19.950 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:19.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:19.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:20.052 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:20.074 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:20.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:20.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:20.177 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:20.201 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:20.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:20.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:20.303 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:20.326 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:20.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:20.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:20.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:20.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:20.452 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:20.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:20.553 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:20.575 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:20.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:20.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:20.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:20 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 7926/78 objects degraded (10161.538%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T15:06:20.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:06:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:06:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:06:20.677 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:20.700 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:20.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:20.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:20.802 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:20.826 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:20.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:20.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:20.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:20 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 7926/78 objects degraded (10161.538%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T15:06:20.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:20 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 7926/78 objects degraded (10161.538%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T15:06:20.928 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:20.951 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:20.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:20.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:21.052 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:21.075 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:21.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:21.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:21.173 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:21.177 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:21.199 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:21.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:21.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:21.301 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:21.323 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:21.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:21.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:21.424 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:21.446 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:21.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:21.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:21.451 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-13T15:06:21.523 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:21 smithi151 ceph-mon[104450]: pgmap v2368: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 567 op/s; 7926/78 objects degraded (10161.538%); 0 B/s, 117 objects/s recovering 2024-09-13T15:06:21.523 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:21 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1510700352' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:21.523 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:21 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:21.524 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:21 smithi151 ceph-mon[108966]: pgmap v2368: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 567 op/s; 7926/78 objects degraded (10161.538%); 0 B/s, 117 objects/s recovering 2024-09-13T15:06:21.524 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:21 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1510700352' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:21.524 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:21 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:21.548 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:21.570 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:21.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:21.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:21.672 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:21.693 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:21.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:21.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:21.795 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:21.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:21.819 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:21.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:21.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:21 smithi184 ceph-mon[92527]: pgmap v2368: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 567 op/s; 7926/78 objects degraded (10161.538%); 0 B/s, 117 objects/s recovering 2024-09-13T15:06:21.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:21 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1510700352' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:21.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:21 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:21.920 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:21.942 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:21.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:21.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:22.030 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:06:22.044 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:22.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:22.066 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:22.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:22.167 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:22.189 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:22.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:22.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:22.291 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:22.313 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:22.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:22.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:22.415 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:22.436 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:22.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:22.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:22.457 INFO:teuthology.orchestra.run.smithi151.stderr:set pool 11 pgp_num to 16 2024-09-13T15:06:22.539 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:22 smithi151 ceph-mon[104450]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:06:22.539 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:22 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:06:22.539 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:22 smithi151 ceph-mon[104450]: osdmap e1356: 8 total, 8 up, 5 in 2024-09-13T15:06:22.539 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:22 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/4216495429' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T15:06:22.539 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:22 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T15:06:22.539 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:22 smithi151 ceph-mon[108966]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:06:22.540 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:22 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:06:22.540 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:22 smithi151 ceph-mon[108966]: osdmap e1356: 8 total, 8 up, 5 in 2024-09-13T15:06:22.540 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:22 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/4216495429' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T15:06:22.540 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:22 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T15:06:22.540 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:22.562 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:22.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:22.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:22.664 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:22.686 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:22.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:22.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:22.788 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:22.810 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:22.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:22.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:22.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:22 smithi184 ceph-mon[92527]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:06:22.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:22 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:06:22.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:22 smithi184 ceph-mon[92527]: osdmap e1356: 8 total, 8 up, 5 in 2024-09-13T15:06:22.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:22 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/4216495429' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T15:06:22.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:22 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T15:06:22.912 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:22.933 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:22.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:22.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:23.035 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:23.057 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:23.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:23.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:23.159 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:23.182 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:23.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:23.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:23.284 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:23.286 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_6 2024-09-13T15:06:23.286 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_6 unique_pool_6 --yes-i-really-really-mean-it 2024-09-13T15:06:23.347 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:23.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:23.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:23.449 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:23.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:23.476 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:23.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:23.526 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:23 smithi151 ceph-mon[104450]: pgmap v2370: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 238 op/s; 7926/78 objects degraded (10161.538%); 0 B/s, 113 objects/s recovering 2024-09-13T15:06:23.527 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:23 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"}]': finished 2024-09-13T15:06:23.527 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:23 smithi151 ceph-mon[104450]: osdmap e1357: 8 total, 8 up, 5 in 2024-09-13T15:06:23.527 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:23 smithi151 ceph-mon[108966]: pgmap v2370: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 238 op/s; 7926/78 objects degraded (10161.538%); 0 B/s, 113 objects/s recovering 2024-09-13T15:06:23.527 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:23 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"}]': finished 2024-09-13T15:06:23.527 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:23 smithi151 ceph-mon[108966]: osdmap e1357: 8 total, 8 up, 5 in 2024-09-13T15:06:23.577 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:23.599 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:23.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:23.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:23.701 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:23.722 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:23.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:23.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:23.787 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:23.824 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:23.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:23.847 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:23.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:23.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:23 smithi184 ceph-mon[92527]: pgmap v2370: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 238 op/s; 7926/78 objects degraded (10161.538%); 0 B/s, 113 objects/s recovering 2024-09-13T15:06:23.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:23 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"}]': finished 2024-09-13T15:06:23.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:23 smithi184 ceph-mon[92527]: osdmap e1357: 8 total, 8 up, 5 in 2024-09-13T15:06:23.949 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:23.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:23.973 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:23.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:24.074 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:24.097 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:24.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:24.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:24.198 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:24.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:24.221 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:24.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:24.323 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:24.345 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:24.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:24.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:24.447 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:24.468 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:24.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:24.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:24.562 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:06:24.570 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:24.592 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:24.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:24.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:24.694 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:24.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:24.717 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:24.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:24.809 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:06:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:06:24] "GET /metrics HTTP/1.1" 200 36713 "" "Prometheus/2.51.0" 2024-09-13T15:06:24.818 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:24.839 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:24.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:24.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:24.941 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:24.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:24.964 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:24.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:25.065 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:25.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:25.087 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:25.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:25.110 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-13T15:06:25.110 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:06:25.189 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:25.210 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:25.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:25.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:25.311 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:25.333 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:25.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:25.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:25.435 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:25.457 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:25.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:25.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:25.559 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:25.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:25.581 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:25.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:25.682 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:25.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:25.706 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:25.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:25.808 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:25.831 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:25.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:25.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:25.853 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:25 smithi184 ceph-mon[92527]: pgmap v2372: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 36 op/s; 6360/78 objects degraded (8153.846%); 0 B/s, 315 objects/s recovering 2024-09-13T15:06:25.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:25 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 6360/78 objects degraded (8153.846%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T15:06:25.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:25 smithi151 ceph-mon[104450]: pgmap v2372: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 36 op/s; 6360/78 objects degraded (8153.846%); 0 B/s, 315 objects/s recovering 2024-09-13T15:06:25.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:25 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 6360/78 objects degraded (8153.846%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T15:06:25.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:25 smithi151 ceph-mon[108966]: pgmap v2372: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 36 op/s; 6360/78 objects degraded (8153.846%); 0 B/s, 315 objects/s recovering 2024-09-13T15:06:25.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:25 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 6360/78 objects degraded (8153.846%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T15:06:25.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:25.955 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:25.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:25.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:26.057 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:26.078 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:26.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:26.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:26.180 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:26.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:26.201 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:26.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:26.302 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:26.324 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:26.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:26.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:26.426 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:26.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:26.449 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:26.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:26.550 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:26.571 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:26.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:26.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:26.674 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:26.695 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:26.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:26.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:26.797 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:26.820 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:26.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:26.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:26.827 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:26.854 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T15:06:26.922 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:26.943 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:26.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:26.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:27.045 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:27.068 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:27.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:27.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:27.170 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:27.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:27.197 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:27.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:27.298 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:27.319 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:27.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:27.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:27.421 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:27.444 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:27.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:27.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:27.548 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:27.571 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:27.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:27.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:27.673 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:27.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:27 smithi151 ceph-mon[104450]: pgmap v2373: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 6360/78 objects degraded (8153.846%); 0 B/s, 195 objects/s recovering 2024-09-13T15:06:27.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:27 smithi151 ceph-mon[104450]: osdmap e1358: 8 total, 8 up, 5 in 2024-09-13T15:06:27.693 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:27 smithi151 ceph-mon[108966]: pgmap v2373: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 6360/78 objects degraded (8153.846%); 0 B/s, 195 objects/s recovering 2024-09-13T15:06:27.694 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:27 smithi151 ceph-mon[108966]: osdmap e1358: 8 total, 8 up, 5 in 2024-09-13T15:06:27.695 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:27.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:27.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:27.797 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:27.804 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T15:06:27.804 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:06:27.816 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T15:06:27.817 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:06:27.819 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:27.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:27.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:27.832 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T15:06:27.833 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:06:27.849 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T15:06:27.849 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:06:27.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:27 smithi184 ceph-mon[92527]: pgmap v2373: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 114 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 6360/78 objects degraded (8153.846%); 0 B/s, 195 objects/s recovering 2024-09-13T15:06:27.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:27 smithi184 ceph-mon[92527]: osdmap e1358: 8 total, 8 up, 5 in 2024-09-13T15:06:27.866 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T15:06:27.867 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:06:27.882 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T15:06:27.882 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:06:27.898 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T15:06:27.898 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:06:27.914 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T15:06:27.914 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:06:27.920 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:27.942 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:27.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:27.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:28.044 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:28.066 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:28.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:28.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:28.163 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:28.168 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:28.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:28.190 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:28.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:28.286 INFO:tasks.thrashosds.thrasher:in_osds: [3, 6, 7, 1, 5] out_osds: [0, 4, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T15:06:28.286 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:06:28.287 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-09-13T15:06:28.287 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd in 0 2024-09-13T15:06:28.291 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:28.312 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:28.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:28.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:28.413 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:28.434 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:28.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:28.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:28.536 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:28.548 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:28 smithi151 ceph-mon[104450]: osdmap e1359: 8 total, 8 up, 5 in 2024-09-13T15:06:28.548 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:28 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1271165465' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:06:28.548 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:28 smithi151 ceph-mon[108966]: osdmap e1359: 8 total, 8 up, 5 in 2024-09-13T15:06:28.548 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:28 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1271165465' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:06:28.561 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:28.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:28.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:28.663 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:28.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:28.686 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:28.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:28.787 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:28.808 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:28.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:28.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:28.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:28 smithi184 ceph-mon[92527]: osdmap e1359: 8 total, 8 up, 5 in 2024-09-13T15:06:28.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:28 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1271165465' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:06:28.910 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:28.932 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:28.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:28.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:29.034 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:29.056 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:29.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:29.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:29.158 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:29.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:29.180 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:29.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:29.282 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:29.303 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:29.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:29.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:29.405 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:29.427 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:29.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:29.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:29.529 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:29.551 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:29.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:29.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:29.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:29 smithi151 ceph-mon[104450]: pgmap v2376: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 4775/78 objects degraded (6121.795%); 0 B/s, 525 objects/s recovering 2024-09-13T15:06:29.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:29 smithi151 ceph-mon[108966]: pgmap v2376: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 4775/78 objects degraded (6121.795%); 0 B/s, 525 objects/s recovering 2024-09-13T15:06:29.653 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:29.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:29.677 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:29.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:29.778 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:29.803 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:29.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:29.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:29.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:29 smithi184 ceph-mon[92527]: pgmap v2376: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 4775/78 objects degraded (6121.795%); 0 B/s, 525 objects/s recovering 2024-09-13T15:06:29.905 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:29.927 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:29.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:29.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:30.028 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:30.050 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:30.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:30.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:30.152 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:30.177 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:30.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:30.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:30.279 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:30.301 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:30.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:30.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:30.403 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:30.425 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:30.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:30.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:30.486 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:06:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:06:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:06:30.527 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:30.549 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:30.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:30.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:30.651 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:30.673 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:30.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:30.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:30.775 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:30.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:30.801 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:30.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:30.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:30 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:06:30.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:30 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 4775/78 objects degraded (6121.795%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:06:30.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:30 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:06:30.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:30 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:06:30.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:30 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:06:30.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:30 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 4775/78 objects degraded (6121.795%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:06:30.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:30 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:06:30.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:30 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:06:30.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:30 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:06:30.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:30 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 4775/78 objects degraded (6121.795%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:06:30.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:30 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:06:30.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:30 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:06:30.903 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:30.926 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:30.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:30.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:31.028 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:31.050 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:31.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:31.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:31.152 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:31.173 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:31.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:31.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:31.275 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:31.301 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:31.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:31.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:31.403 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:31.428 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:31.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:31.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:31.530 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:31.552 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:31.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:31.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:31.612 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:31.654 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:31.680 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:31.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:31.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:31.762 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:31 smithi184 ceph-mon[92527]: pgmap v2377: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 4453/78 objects degraded (5708.974%); 0 B/s, 463 objects/s recovering 2024-09-13T15:06:31.763 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:31 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:31.783 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:31.808 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:31.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:31.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:31.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:31 smithi151 ceph-mon[104450]: pgmap v2377: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 4453/78 objects degraded (5708.974%); 0 B/s, 463 objects/s recovering 2024-09-13T15:06:31.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:31 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:31.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:31 smithi151 ceph-mon[108966]: pgmap v2377: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 4453/78 objects degraded (5708.974%); 0 B/s, 463 objects/s recovering 2024-09-13T15:06:31.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:31 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:31.910 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:31.933 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:31.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:31.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:32.035 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:32.057 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:32.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:32.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:32.159 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:32.181 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:32.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:32.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:32.283 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:32.305 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:32.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:32.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:32.407 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:32.428 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:32.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:32.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:32.530 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:32.551 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:32.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:32.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:32.653 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:32.674 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:32.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:32.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:32.776 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:32.797 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:32.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:32.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:32.899 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:32.921 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:32.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:32.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:33.024 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:33.046 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:33.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:33.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:33.148 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:33.170 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:33.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:33.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:33.273 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:33.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:33.296 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:33.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:33.397 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:33.419 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:33.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:33.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:33.520 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:33.542 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:33.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:33.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:33.644 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:33.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:33.667 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:33.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:33.768 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:33.790 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:33.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:33.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:33.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:33 smithi184 ceph-mon[92527]: pgmap v2378: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 4453/78 objects degraded (5708.974%); 0 B/s, 238 objects/s recovering 2024-09-13T15:06:33.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:33 smithi151 ceph-mon[104450]: pgmap v2378: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 4453/78 objects degraded (5708.974%); 0 B/s, 238 objects/s recovering 2024-09-13T15:06:33.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:33 smithi151 ceph-mon[108966]: pgmap v2378: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 4453/78 objects degraded (5708.974%); 0 B/s, 238 objects/s recovering 2024-09-13T15:06:33.892 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:33.913 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:33.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:33.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:34.015 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:34.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:34.036 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:34.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:34.137 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:34.158 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:34.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:34.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:34.260 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:34.281 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:34.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:34.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:34.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:34.405 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:34.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:34.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:34.506 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:34.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:34.528 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:34.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:34.629 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:34.651 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:34.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:34.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:34.753 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:34.775 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:34.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:34.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:34.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:06:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:06:34] "GET /metrics HTTP/1.1" 200 36709 "" "Prometheus/2.51.0" 2024-09-13T15:06:34.876 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:34.898 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:34.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:34.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:35.000 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:35.024 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:35.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:35.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:35.126 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:35.148 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:35.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:35.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:35.250 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:35.272 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:35.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:35.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:35.374 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:35.395 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:35.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:35.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:35.497 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:35.519 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:35.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:35.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:35.621 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:35.642 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:35.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:35.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:35.744 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:35.765 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:35.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:35.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:35.846 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:35 smithi151 ceph-mon[104450]: pgmap v2379: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 4453/78 objects degraded (5708.974%); 0 B/s, 238 objects/s recovering 2024-09-13T15:06:35.847 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:35 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 4453/78 objects degraded (5708.974%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:06:35.847 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:35 smithi151 ceph-mon[108966]: pgmap v2379: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 4453/78 objects degraded (5708.974%); 0 B/s, 238 objects/s recovering 2024-09-13T15:06:35.847 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:35 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 4453/78 objects degraded (5708.974%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:06:35.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:35 smithi184 ceph-mon[92527]: pgmap v2379: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 4453/78 objects degraded (5708.974%); 0 B/s, 238 objects/s recovering 2024-09-13T15:06:35.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:35 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 4453/78 objects degraded (5708.974%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:06:35.867 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:35.889 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:35.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:35.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:35.991 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:36.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:36.013 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:36.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:36.114 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:36.135 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:36.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:36.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:36.237 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:36.261 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:36.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:36.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:36.363 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:36.385 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:36.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:36.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:36.486 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:36.508 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:36.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:36.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:36.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:36 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:06:36.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:36 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:06:36.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:36 smithi151 ceph-mon[108966]: pgmap v2380: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 3851/78 objects degraded (4937.179%); 0 B/s, 264 objects/s recovering 2024-09-13T15:06:36.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:36 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:06:36.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:36 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:06:36.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:36 smithi151 ceph-mon[104450]: pgmap v2380: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 3851/78 objects degraded (4937.179%); 0 B/s, 264 objects/s recovering 2024-09-13T15:06:36.610 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:36.631 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:36.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:36.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:36.733 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:36.754 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:36.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:36.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:36.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:36 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:06:36.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:36 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:06:36.854 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:36 smithi184 ceph-mon[92527]: pgmap v2380: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 3851/78 objects degraded (4937.179%); 0 B/s, 264 objects/s recovering 2024-09-13T15:06:36.855 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:36.878 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:36.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:36.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:36.980 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:37.001 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:37.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:37.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:37.103 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:37.125 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:37.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:37.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:37.227 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:37.250 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:37.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:37.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:37.352 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:37.375 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:37.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:37.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:37.477 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:37.499 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:37.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:37.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:37.601 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:37.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:37.624 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:37.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:37.725 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:37.746 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:37.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:37.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:37.848 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:37.871 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:37.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:37.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:37.973 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:37.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:37.996 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:37.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:38.097 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:38.120 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:38.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:38.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:38.222 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:38.244 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:38.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:38.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:38.346 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:38.368 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:38.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:38.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:38.470 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:38.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:38.493 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:38.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:38.594 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:38.616 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:38.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:38.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:38.718 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:38.739 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:38.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:38.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:38.841 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:38.867 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:38.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:38.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:38.969 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:38.994 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:38.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:38.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:39.097 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:39.123 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:39.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:39.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:39.225 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:39.249 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:39.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:39.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:39.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:39.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:39 smithi184 ceph-mon[92527]: pgmap v2381: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 3851/78 objects degraded (4937.179%); 0 B/s, 239 objects/s recovering 2024-09-13T15:06:39.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:39 smithi151 ceph-mon[104450]: pgmap v2381: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 3851/78 objects degraded (4937.179%); 0 B/s, 239 objects/s recovering 2024-09-13T15:06:39.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:39 smithi151 ceph-mon[108966]: pgmap v2381: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 3851/78 objects degraded (4937.179%); 0 B/s, 239 objects/s recovering 2024-09-13T15:06:39.372 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:39.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:39.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:39.474 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:39.496 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:39.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:39.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:39.598 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:39.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:39.623 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:39.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:39.724 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:39.751 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:39.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:39.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:39.853 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:39.903 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:39.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:39.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:40.005 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:40.032 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:40.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:40.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:40.133 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:40.156 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:40.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:40.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:40.258 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:40.279 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:40.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:40.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:40.380 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:40.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:40.407 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:40.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:40.508 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:40.534 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:40.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:40.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:40.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:06:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:06:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:06:40.635 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:40.660 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:40.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:40.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:40.762 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:40.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:40.785 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:40.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:40.886 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:40.907 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:40.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:40.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:41.009 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:41.033 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:41.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:41.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:41.135 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:41.158 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:41.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:41.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:41.260 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:41.281 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:41.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:41.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:41.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:41 smithi184 ceph-mon[92527]: pgmap v2382: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 3174/78 objects degraded (4069.231%); 0 B/s, 133 objects/s recovering 2024-09-13T15:06:41.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:41 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 3174/78 objects degraded (4069.231%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:06:41.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:41 smithi151 ceph-mon[104450]: pgmap v2382: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 3174/78 objects degraded (4069.231%); 0 B/s, 133 objects/s recovering 2024-09-13T15:06:41.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:41 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 3174/78 objects degraded (4069.231%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:06:41.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:41 smithi151 ceph-mon[108966]: pgmap v2382: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 3174/78 objects degraded (4069.231%); 0 B/s, 133 objects/s recovering 2024-09-13T15:06:41.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:41 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 3174/78 objects degraded (4069.231%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:06:41.383 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:41.407 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:41.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:41.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:41.509 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:41.532 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:41.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:41.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:41.634 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:41.658 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:41.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:41.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:41.760 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:41.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:41.782 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:41.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:41.883 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:41.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:41.907 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:41.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:42.008 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:42.031 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:42.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:42.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:42.133 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:42.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:42.156 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:42.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:42.257 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:42.279 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:42.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:42.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:42.381 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:42.403 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:42.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:42.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:42.505 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:42.526 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:42.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:42.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:42.628 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:42.649 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:42.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:42.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:42.708 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:42.751 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:42.773 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:42.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:42.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:42.874 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:42.897 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:42.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:42.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:42.980 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:06:42.999 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:43.020 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-13T15:06:43.045 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:43.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:43.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:43.147 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:43.169 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:43.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:43.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:43.271 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:43.297 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:43.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:43.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:43.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:43 smithi184 ceph-mon[92527]: pgmap v2383: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 3174/78 objects degraded (4069.231%); 0 B/s, 106 objects/s recovering 2024-09-13T15:06:43.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:43 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1548149431' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:06:43.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:43 smithi151 ceph-mon[104450]: pgmap v2383: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 3174/78 objects degraded (4069.231%); 0 B/s, 106 objects/s recovering 2024-09-13T15:06:43.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:43 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1548149431' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:06:43.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:43 smithi151 ceph-mon[108966]: pgmap v2383: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 3174/78 objects degraded (4069.231%); 0 B/s, 106 objects/s recovering 2024-09-13T15:06:43.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:43 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1548149431' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:06:43.399 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:43.422 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:43.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:43.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:43.524 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:43.546 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:43.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:43.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:43.647 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:43.670 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:43.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:43.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:43.771 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:43.797 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:43.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:43.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:43.899 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:43.921 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:43.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:43.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:44.022 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:44.048 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:44.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:44.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:44.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:44 smithi151 ceph-mon[104450]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:06:44.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:44 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1548149431' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:06:44.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:44 smithi151 ceph-mon[104450]: osdmap e1360: 8 total, 8 up, 5 in 2024-09-13T15:06:44.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:44 smithi151 ceph-mon[108966]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:06:44.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:44 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1548149431' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:06:44.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:44 smithi151 ceph-mon[108966]: osdmap e1360: 8 total, 8 up, 5 in 2024-09-13T15:06:44.150 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:44.173 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:44.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:44.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:44.275 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:44.296 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:44.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:44.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:44.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:44 smithi184 ceph-mon[92527]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:06:44.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:44 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1548149431' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:06:44.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:44 smithi184 ceph-mon[92527]: osdmap e1360: 8 total, 8 up, 5 in 2024-09-13T15:06:44.399 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:44.422 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:44.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:44.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:44.524 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:44.550 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:44.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:44.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:44.652 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:44.675 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:44.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:44.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:44.777 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:44.803 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:44.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:44.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:44.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:06:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:06:44] "GET /metrics HTTP/1.1" 200 36709 "" "Prometheus/2.51.0" 2024-09-13T15:06:44.905 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:44.927 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:44.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:44.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:45.029 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:45.051 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:45.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:45.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:45.153 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:45.179 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:45.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:45.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:45.281 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:45.302 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:45.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:45.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:45.312 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:45 smithi151 ceph-mon[104450]: pgmap v2385: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 3174/78 objects degraded (4069.231%); 0 B/s, 127 objects/s recovering 2024-09-13T15:06:45.312 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:45 smithi151 ceph-mon[108966]: pgmap v2385: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 3174/78 objects degraded (4069.231%); 0 B/s, 127 objects/s recovering 2024-09-13T15:06:45.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:45 smithi184 ceph-mon[92527]: pgmap v2385: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 3174/78 objects degraded (4069.231%); 0 B/s, 127 objects/s recovering 2024-09-13T15:06:45.404 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:45.430 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:45.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:45.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:45.532 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:45.558 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:45.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:45.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:45.659 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:45.685 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:45.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:45.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:45.787 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:45.813 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:45.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:45.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:45.915 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:45.941 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:45.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:45.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:46.029 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_6' removed 2024-09-13T15:06:46.043 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:46.066 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:46.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:46.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:46.169 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:46.190 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:46.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:46.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:46.292 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:46.313 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:46.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:46.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:46.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:46 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2551415158' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T15:06:46.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:46 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2551415158' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T15:06:46.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:46 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2551415158' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T15:06:46.415 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:46.437 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:46.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:46.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:46.539 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:46.560 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:46.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:46.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:46.662 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:46.684 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:46.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:46.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:46.786 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:46.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:46.810 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:46.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:46.911 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:46.933 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:46.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:46.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:47.035 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:47.056 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:47.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:47.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:47.158 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:47.180 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:47.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:47.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:47.281 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:47.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:47.304 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:47.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:47.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:47 smithi184 ceph-mon[92527]: pgmap v2386: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 2332/78 objects degraded (2989.744%); 0 B/s, 151 objects/s recovering 2024-09-13T15:06:47.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:47 smithi184 ceph-mon[92527]: Health check update: Degraded data redundancy: 2332/78 objects degraded (2989.744%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:06:47.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:47 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2551415158' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T15:06:47.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:47 smithi184 ceph-mon[92527]: osdmap e1361: 8 total, 8 up, 5 in 2024-09-13T15:06:47.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:47 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:47.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:47 smithi151 ceph-mon[104450]: pgmap v2386: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 2332/78 objects degraded (2989.744%); 0 B/s, 151 objects/s recovering 2024-09-13T15:06:47.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:47 smithi151 ceph-mon[104450]: Health check update: Degraded data redundancy: 2332/78 objects degraded (2989.744%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:06:47.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:47 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2551415158' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T15:06:47.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:47 smithi151 ceph-mon[104450]: osdmap e1361: 8 total, 8 up, 5 in 2024-09-13T15:06:47.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:47 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:47.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:47 smithi151 ceph-mon[108966]: pgmap v2386: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 116 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 2332/78 objects degraded (2989.744%); 0 B/s, 151 objects/s recovering 2024-09-13T15:06:47.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:47 smithi151 ceph-mon[108966]: Health check update: Degraded data redundancy: 2332/78 objects degraded (2989.744%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T15:06:47.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:47 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2551415158' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T15:06:47.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:47 smithi151 ceph-mon[108966]: osdmap e1361: 8 total, 8 up, 5 in 2024-09-13T15:06:47.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:47 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:47.405 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:47.426 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:47.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:47.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:47.528 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:47.549 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:47.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:47.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:47.651 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:47.675 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:47.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:47.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:47.777 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:47.801 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:47.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:47.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:47.903 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:47.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:47.927 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:47.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:48.028 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:48.051 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:48.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:48.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:48.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:48 smithi151 ceph-mon[104450]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2332/78 objects degraded (2989.744%), 1 pg degraded, 1 pg undersized) 2024-09-13T15:06:48.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:48 smithi151 ceph-mon[108966]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2332/78 objects degraded (2989.744%), 1 pg degraded, 1 pg undersized) 2024-09-13T15:06:48.153 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:48.175 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:48.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:48.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:48.277 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:48.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:48.301 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:48.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:48.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:48 smithi184 ceph-mon[92527]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2332/78 objects degraded (2989.744%), 1 pg degraded, 1 pg undersized) 2024-09-13T15:06:48.402 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:48.424 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:48.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:48.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:48.526 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:48.547 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:48.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:48.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:48.649 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:48.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:48.673 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:48.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:48.775 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:48.799 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:48.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:48.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:48.804 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-13T15:06:48.804 INFO:teuthology.task.print:**** done end radosbench.yaml 2024-09-13T15:06:48.805 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-09-13T15:06:48.812 INFO:tasks.workunit:Pulling workunits from ref quincy 2024-09-13T15:06:48.812 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-13T15:06:48.813 DEBUG:teuthology.orchestra.run.smithi151:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-13T15:06:48.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:48.866 INFO:teuthology.orchestra.run.smithi151.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-13T15:06:48.867 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-13T15:06:48.901 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:48.922 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-13T15:06:48.922 DEBUG:teuthology.orchestra.run.smithi151:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-13T15:06:48.924 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:48.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:48.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:48.977 INFO:tasks.workunit:timeout=3h 2024-09-13T15:06:48.978 INFO:tasks.workunit:cleanup=True 2024-09-13T15:06:48.978 DEBUG:teuthology.orchestra.run.smithi151:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch quincy https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 2024-09-13T15:06:49.026 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:49.057 INFO:tasks.workunit.client.0.smithi151.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-13T15:06:49.077 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:49.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:49.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:49.179 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:49.200 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:49.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:49.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:49.302 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:49.329 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:49.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:49.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:49.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:49 smithi184 ceph-mon[92527]: pgmap v2388: 103 pgs: 103 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail 2024-09-13T15:06:49.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:49 smithi151 ceph-mon[104450]: pgmap v2388: 103 pgs: 103 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail 2024-09-13T15:06:49.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:49 smithi151 ceph-mon[108966]: pgmap v2388: 103 pgs: 103 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail 2024-09-13T15:06:49.431 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:49.454 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:49.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:49.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:49.556 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:49.582 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:49.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:49.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:49.587 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T15:06:49.683 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:49.708 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:49.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:49.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:49.811 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:49.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:49.834 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:49.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:49.935 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:49.956 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:49.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:49.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:50.058 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:50.085 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:50.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:50.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:50.187 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:50.211 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:50.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:50.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:50.313 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:50.335 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:50.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:50.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:50.422 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:50.437 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:50.460 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:50.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:50.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:50.515 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:06:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:06:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:06:50.562 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:50.584 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:50.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:50.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:50.686 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:50.708 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:50.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:50.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:50.810 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:50.833 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:50.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:50.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:50.935 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:50.957 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:50.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:50.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:51.058 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:51.081 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:51.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:51.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:51.182 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:51.205 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:51.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:51.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:51.308 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:51.329 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:51.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:51.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:51.431 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:51.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:51.453 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:51.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:51.553 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:51.575 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:51.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:51.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:51.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:51 smithi184 ceph-mon[92527]: pgmap v2389: 103 pgs: 103 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail 2024-09-13T15:06:51.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:51 smithi151 ceph-mon[104450]: pgmap v2389: 103 pgs: 103 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail 2024-09-13T15:06:51.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:51 smithi151 ceph-mon[108966]: pgmap v2389: 103 pgs: 103 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail 2024-09-13T15:06:51.676 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:51.698 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:51.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:51.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:51.800 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:51.821 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:51.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:51.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:51.923 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:51.945 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:51.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:51.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:52.047 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:52.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:52.070 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:52.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:52.171 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:52.193 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:52.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:52.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:52.295 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:52.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:52.318 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:52.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:52.419 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:52.445 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:52.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:52.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:52.546 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:52.569 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:52.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:52.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:52.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:52 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a[104427]: 2024-09-13T15:06:52.303+0000 7fe9da14f640 -1 mon.a@0(leader).osd e1361 definitely_dead 0 2024-09-13T15:06:52.671 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:52.693 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:52.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:52.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:52.727 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-13T15:06:52.727 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-13T15:06:52.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-13T15:06:52.727 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-13T15:06:52.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.727 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.46034.0:200313 9.8 9:14628d6c:::benchmark_data_smithi151_261954_object97457:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1253)", 2024-09-13T15:06:52.727 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:57:34.978598+0000", 2024-09-13T15:06:52.727 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 557.72653907300003, 2024-09-13T15:06:52.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0422188750000001, 2024-09-13T15:06:52.727 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:06:52.727 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:06:52.728 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-13T15:06:52.728 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:06:52.728 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.46034", 2024-09-13T15:06:52.728 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:06:52.728 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 200313 2024-09-13T15:06:52.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.728 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:06:52.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:06:52.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:34.978598+0000", 2024-09-13T15:06:52.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:06:52.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:34.978598+0000", 2024-09-13T15:06:52.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:06:52.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:34.978600+0000", 2024-09-13T15:06:52.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2899999999999999e-06 2024-09-13T15:06:52.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:06:52.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:34.978604+0000", 2024-09-13T15:06:52.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8749999999999999e-06 2024-09-13T15:06:52.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:06:52.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:34.978661+0000", 2024-09-13T15:06:52.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6752999999999997e-05 2024-09-13T15:06:52.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:06:52.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:34.978684+0000", 2024-09-13T15:06:52.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2993000000000002e-05 2024-09-13T15:06:52.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.731 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:34.994896+0000", 2024-09-13T15:06:52.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016211817999999999 2024-09-13T15:06:52.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.732 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:06:52.732 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:34.995038+0000", 2024-09-13T15:06:52.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014216000000000001 2024-09-13T15:06:52.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.732 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,5", 2024-09-13T15:06:52.732 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:34.995168+0000", 2024-09-13T15:06:52.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013023000000000001 2024-09-13T15:06:52.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.732 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.009534+0000", 2024-09-13T15:06:52.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014365559999999999 2024-09-13T15:06:52.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:06:52.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.009540+0000", 2024-09-13T15:06:52.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9839999999999998e-06 2024-09-13T15:06:52.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.020738+0000", 2024-09-13T15:06:52.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0111981430000001 2024-09-13T15:06:52.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:06:52.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.020817+0000", 2024-09-13T15:06:52.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9068999999999998e-05 2024-09-13T15:06:52.734 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.734 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:06:52.734 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.735 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.46034.0:200368 9.8 9:1614d357:::benchmark_data_smithi151_261954_object97512:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1254)", 2024-09-13T15:06:52.735 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:57:35.039528+0000", 2024-09-13T15:06:52.735 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 557.66560953400005, 2024-09-13T15:06:52.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.98955691000000001, 2024-09-13T15:06:52.735 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:06:52.735 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:06:52.735 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:52.735 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:06:52.735 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.46034", 2024-09-13T15:06:52.735 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:06:52.735 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 200368 2024-09-13T15:06:52.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.735 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:06:52.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.736 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:06:52.736 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.039528+0000", 2024-09-13T15:06:52.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.736 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.736 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.736 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:06:52.736 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.039528+0000", 2024-09-13T15:06:52.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.736 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.736 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.736 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:06:52.736 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.039529+0000", 2024-09-13T15:06:52.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.19e-06 2024-09-13T15:06:52.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.737 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:06:52.737 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.039534+0000", 2024-09-13T15:06:52.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9250000000000003e-06 2024-09-13T15:06:52.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.737 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:06:52.737 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.039550+0000", 2024-09-13T15:06:52.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5747e-05 2024-09-13T15:06:52.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:06:52.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.039557+0000", 2024-09-13T15:06:52.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.537e-06 2024-09-13T15:06:52.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.039634+0000", 2024-09-13T15:06:52.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6333000000000003e-05 2024-09-13T15:06:52.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:06:52.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.039643+0000", 2024-09-13T15:06:52.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5650000000000007e-06 2024-09-13T15:06:52.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.021229+0000", 2024-09-13T15:06:52.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.98158553299999995 2024-09-13T15:06:52.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:06:52.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.021339+0000", 2024-09-13T15:06:52.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000110402 2024-09-13T15:06:52.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,5", 2024-09-13T15:06:52.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.021458+0000", 2024-09-13T15:06:52.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118445 2024-09-13T15:06:52.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.028852+0000", 2024-09-13T15:06:52.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0073948210000000002 2024-09-13T15:06:52.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.741 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.028918+0000", 2024-09-13T15:06:52.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5522e-05 2024-09-13T15:06:52.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:06:52.741 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.028987+0000", 2024-09-13T15:06:52.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9136999999999999e-05 2024-09-13T15:06:52.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:06:52.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.029041+0000", 2024-09-13T15:06:52.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3943000000000002e-05 2024-09-13T15:06:52.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:06:52.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.029085+0000", 2024-09-13T15:06:52.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3810000000000002e-05 2024-09-13T15:06:52.743 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.743 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:06:52.743 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.743 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.46034.0:200370 9.8 9:176c4228:::benchmark_data_smithi151_261954_object97514:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1254)", 2024-09-13T15:06:52.743 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:57:35.040542+0000", 2024-09-13T15:06:52.743 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 557.664595265, 2024-09-13T15:06:52.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.98866303600000005, 2024-09-13T15:06:52.743 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:06:52.743 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:06:52.744 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:52.744 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:06:52.744 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.46034", 2024-09-13T15:06:52.744 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:06:52.744 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 200370 2024-09-13T15:06:52.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.744 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:06:52.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:06:52.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.040542+0000", 2024-09-13T15:06:52.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:06:52.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.040542+0000", 2024-09-13T15:06:52.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:06:52.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.040543+0000", 2024-09-13T15:06:52.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6499999999999999e-07 2024-09-13T15:06:52.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.746 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:06:52.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.040546+0000", 2024-09-13T15:06:52.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8459999999999999e-06 2024-09-13T15:06:52.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:06:52.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.040555+0000", 2024-09-13T15:06:52.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8039999999999996e-06 2024-09-13T15:06:52.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:06:52.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.040561+0000", 2024-09-13T15:06:52.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0039999999999996e-06 2024-09-13T15:06:52.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.040605+0000", 2024-09-13T15:06:52.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.392e-05 2024-09-13T15:06:52.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:06:52.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.040613+0000", 2024-09-13T15:06:52.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6510000000000001e-06 2024-09-13T15:06:52.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.023922+0000", 2024-09-13T15:06:52.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.98330905899999999 2024-09-13T15:06:52.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.750 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:06:52.750 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.024037+0000", 2024-09-13T15:06:52.750 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000115202 2024-09-13T15:06:52.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.750 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,5", 2024-09-13T15:06:52.750 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.024104+0000", 2024-09-13T15:06:52.750 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6537000000000003e-05 2024-09-13T15:06:52.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.750 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.750 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.028973+0000", 2024-09-13T15:06:52.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0048694189999999998 2024-09-13T15:06:52.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:06:52.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.029095+0000", 2024-09-13T15:06:52.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012183799999999999 2024-09-13T15:06:52.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.029148+0000", 2024-09-13T15:06:52.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2549000000000001e-05 2024-09-13T15:06:52.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.752 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:06:52.752 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.029180+0000", 2024-09-13T15:06:52.752 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2777000000000001e-05 2024-09-13T15:06:52.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.752 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:06:52.752 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.029205+0000", 2024-09-13T15:06:52.752 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4765000000000001e-05 2024-09-13T15:06:52.752 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.752 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:06:52.752 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.753 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.46034.0:200377 9.8 9:16aeef32:::benchmark_data_smithi151_261954_object97521:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1254)", 2024-09-13T15:06:52.753 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T14:57:35.045522+0000", 2024-09-13T15:06:52.753 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 557.65961565800001, 2024-09-13T15:06:52.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.99413063400000001, 2024-09-13T15:06:52.753 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:06:52.753 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:06:52.753 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:52.753 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:06:52.753 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.46034", 2024-09-13T15:06:52.753 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:06:52.753 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 200377 2024-09-13T15:06:52.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.754 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:06:52.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:06:52.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.045522+0000", 2024-09-13T15:06:52.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:06:52.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.045522+0000", 2024-09-13T15:06:52.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:06:52.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.045523+0000", 2024-09-13T15:06:52.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.31e-06 2024-09-13T15:06:52.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:06:52.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.045528+0000", 2024-09-13T15:06:52.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3430000000000003e-06 2024-09-13T15:06:52.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:06:52.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.045542+0000", 2024-09-13T15:06:52.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3465999999999999e-05 2024-09-13T15:06:52.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.756 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:06:52.756 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.045549+0000", 2024-09-13T15:06:52.756 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3939999999999998e-06 2024-09-13T15:06:52.756 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.756 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.756 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.756 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.045609+0000", 2024-09-13T15:06:52.756 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9346999999999999e-05 2024-09-13T15:06:52.756 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.756 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.756 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:06:52.756 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:35.045618+0000", 2024-09-13T15:06:52.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4180000000000003e-06 2024-09-13T15:06:52.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.029477+0000", 2024-09-13T15:06:52.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.98385874100000004 2024-09-13T15:06:52.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:06:52.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.029585+0000", 2024-09-13T15:06:52.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000107936 2024-09-13T15:06:52.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,5", 2024-09-13T15:06:52.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.029686+0000", 2024-09-13T15:06:52.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010132399999999999 2024-09-13T15:06:52.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:06:52.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.039442+0000", 2024-09-13T15:06:52.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0097556729999999994 2024-09-13T15:06:52.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.039477+0000", 2024-09-13T15:06:52.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4866999999999997e-05 2024-09-13T15:06:52.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.039546+0000", 2024-09-13T15:06:52.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9882999999999995e-05 2024-09-13T15:06:52.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:06:52.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.039606+0000", 2024-09-13T15:06:52.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9621999999999999e-05 2024-09-13T15:06:52.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:06:52.760 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T14:57:36.039652+0000", 2024-09-13T15:06:52.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6310000000000002e-05 2024-09-13T15:06:52.760 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.760 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:06:52.760 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.761 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.36396.0:158542 10.4 10:2eb7c930:::benchmark_data_smithi151_288181_object32929:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1289)", 2024-09-13T15:06:52.761 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T15:00:25.033415+0000", 2024-09-13T15:06:52.761 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 387.671722273, 2024-09-13T15:06:52.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1486585330000001, 2024-09-13T15:06:52.761 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:06:52.761 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:06:52.761 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:52.761 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:06:52.761 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.36396", 2024-09-13T15:06:52.761 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:06:52.761 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 158542 2024-09-13T15:06:52.761 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.762 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:06:52.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:06:52.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:00:25.033415+0000", 2024-09-13T15:06:52.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:06:52.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:00:25.033415+0000", 2024-09-13T15:06:52.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:06:52.763 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:00:25.033416+0000", 2024-09-13T15:06:52.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.266e-06 2024-09-13T15:06:52.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.763 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:06:52.763 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:00:25.033420+0000", 2024-09-13T15:06:52.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8539999999999999e-06 2024-09-13T15:06:52.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.763 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:06:52.763 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:00:25.033431+0000", 2024-09-13T15:06:52.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0788000000000001e-05 2024-09-13T15:06:52.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:06:52.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:00:25.033441+0000", 2024-09-13T15:06:52.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.011e-05 2024-09-13T15:06:52.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:00:25.036506+0000", 2024-09-13T15:06:52.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030646409999999999 2024-09-13T15:06:52.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.765 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:06:52.765 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:00:25.036516+0000", 2024-09-13T15:06:52.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0315e-05 2024-09-13T15:06:52.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.765 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:00:26.163126+0000", 2024-09-13T15:06:52.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1266100269999999 2024-09-13T15:06:52.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:06:52.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:00:26.163268+0000", 2024-09-13T15:06:52.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014178500000000001 2024-09-13T15:06:52.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,1,2,4", 2024-09-13T15:06:52.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:00:26.163695+0000", 2024-09-13T15:06:52.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00042752799999999999 2024-09-13T15:06:52.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:00:26.175299+0000", 2024-09-13T15:06:52.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011603542999999999 2024-09-13T15:06:52.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:06:52.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:00:26.178331+0000", 2024-09-13T15:06:52.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030319420000000001 2024-09-13T15:06:52.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:00:26.179013+0000", 2024-09-13T15:06:52.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00068187300000000005 2024-09-13T15:06:52.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:00:26.181603+0000", 2024-09-13T15:06:52.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002590111 2024-09-13T15:06:52.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.769 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.769 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:00:26.181833+0000", 2024-09-13T15:06:52.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000230008 2024-09-13T15:06:52.769 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.769 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:06:52.769 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:00:26.181915+0000", 2024-09-13T15:06:52.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1779000000000004e-05 2024-09-13T15:06:52.769 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.769 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:06:52.770 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:00:26.182074+0000", 2024-09-13T15:06:52.770 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000158963 2024-09-13T15:06:52.770 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.770 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:06:52.770 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.770 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.771 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.36396.0:197261 10.d 10:babcec4c:::benchmark_data_smithi151_288181_object71648:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1293)", 2024-09-13T15:06:52.771 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T15:01:10.089241+0000", 2024-09-13T15:06:52.771 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 342.61589647099999, 2024-09-13T15:06:52.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0503124660000001, 2024-09-13T15:06:52.771 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:06:52.771 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:06:52.772 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:52.772 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:06:52.772 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.36396", 2024-09-13T15:06:52.772 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:06:52.772 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 197261 2024-09-13T15:06:52.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.773 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:06:52.773 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.773 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:06:52.773 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.089241+0000", 2024-09-13T15:06:52.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.773 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.773 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:06:52.773 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.089241+0000", 2024-09-13T15:06:52.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.774 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:06:52.774 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.089242+0000", 2024-09-13T15:06:52.774 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.55e-06 2024-09-13T15:06:52.774 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.774 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:06:52.774 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.089247+0000", 2024-09-13T15:06:52.774 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9860000000000002e-06 2024-09-13T15:06:52.774 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.774 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:06:52.774 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.089261+0000", 2024-09-13T15:06:52.775 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3937000000000001e-05 2024-09-13T15:06:52.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:06:52.775 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.089269+0000", 2024-09-13T15:06:52.775 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2139999999999999e-06 2024-09-13T15:06:52.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.775 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.101115+0000", 2024-09-13T15:06:52.775 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011846867000000001 2024-09-13T15:06:52.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:06:52.776 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.101124+0000", 2024-09-13T15:06:52.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.789e-06 2024-09-13T15:06:52.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.776 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.776 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.776 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.110277+0000", 2024-09-13T15:06:52.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.009152952 2024-09-13T15:06:52.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.776 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.776 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:06:52.776 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.110413+0000", 2024-09-13T15:06:52.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013595 2024-09-13T15:06:52.777 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.777 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T15:06:52.777 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.110964+0000", 2024-09-13T15:06:52.777 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00055111600000000004 2024-09-13T15:06:52.777 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.777 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:06:52.777 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.138175+0000", 2024-09-13T15:06:52.777 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.027210673000000001 2024-09-13T15:06:52.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.139445+0000", 2024-09-13T15:06:52.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001269795 2024-09-13T15:06:52.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.139479+0000", 2024-09-13T15:06:52.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4109000000000002e-05 2024-09-13T15:06:52.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:06:52.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.139525+0000", 2024-09-13T15:06:52.779 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6121000000000001e-05 2024-09-13T15:06:52.779 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.779 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:06:52.779 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.139553+0000", 2024-09-13T15:06:52.779 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8407000000000001e-05 2024-09-13T15:06:52.779 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.779 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:06:52.779 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.779 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.779 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.36396.0:197286 10.d 10:bc85818d:::benchmark_data_smithi151_288181_object71673:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1293)", 2024-09-13T15:06:52.779 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T15:01:10.115739+0000", 2024-09-13T15:06:52.779 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 342.58939833699998, 2024-09-13T15:06:52.780 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0239408320000001, 2024-09-13T15:06:52.780 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:06:52.780 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:06:52.780 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:52.780 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:06:52.780 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.36396", 2024-09-13T15:06:52.780 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:06:52.780 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 197286 2024-09-13T15:06:52.780 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.780 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:06:52.780 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.780 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:06:52.780 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.115739+0000", 2024-09-13T15:06:52.781 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.781 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:06:52.781 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.115739+0000", 2024-09-13T15:06:52.781 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.781 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:06:52.781 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.115741+0000", 2024-09-13T15:06:52.781 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5290000000000001e-06 2024-09-13T15:06:52.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.782 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:06:52.782 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.115751+0000", 2024-09-13T15:06:52.782 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0305999999999999e-05 2024-09-13T15:06:52.782 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.782 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:06:52.782 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.115763+0000", 2024-09-13T15:06:52.782 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1909999999999999e-05 2024-09-13T15:06:52.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.783 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.783 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:06:52.783 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.115778+0000", 2024-09-13T15:06:52.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5013e-05 2024-09-13T15:06:52.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.784 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.784 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.117685+0000", 2024-09-13T15:06:52.784 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019076080000000001 2024-09-13T15:06:52.784 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:06:52.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.117695+0000", 2024-09-13T15:06:52.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1419999999999999e-06 2024-09-13T15:06:52.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.132909+0000", 2024-09-13T15:06:52.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0152146799999999 2024-09-13T15:06:52.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.786 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:06:52.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.133022+0000", 2024-09-13T15:06:52.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011234999999999999 2024-09-13T15:06:52.787 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.787 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.787 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T15:06:52.787 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.133143+0000", 2024-09-13T15:06:52.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000121752 2024-09-13T15:06:52.787 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.787 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.788 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:06:52.788 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.139408+0000", 2024-09-13T15:06:52.788 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0062648529999999999 2024-09-13T15:06:52.788 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.788 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.788 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.789 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.139596+0000", 2024-09-13T15:06:52.789 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018741999999999999 2024-09-13T15:06:52.789 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.789 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.789 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.139629+0000", 2024-09-13T15:06:52.790 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3594999999999999e-05 2024-09-13T15:06:52.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.790 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:06:52.790 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.139655+0000", 2024-09-13T15:06:52.790 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6259e-05 2024-09-13T15:06:52.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.791 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:06:52.791 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.139680+0000", 2024-09-13T15:06:52.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4414999999999999e-05 2024-09-13T15:06:52.791 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.792 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:06:52.792 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.792 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.792 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.792 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.36396.0:197290 10.d 10:b1decba0:::benchmark_data_smithi151_288181_object71677:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1293)", 2024-09-13T15:06:52.792 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T15:01:10.121541+0000", 2024-09-13T15:06:52.792 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 342.58359651900003, 2024-09-13T15:06:52.793 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.018303918, 2024-09-13T15:06:52.793 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:06:52.793 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:06:52.793 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:52.793 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:06:52.793 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.36396", 2024-09-13T15:06:52.793 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:06:52.794 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 197290 2024-09-13T15:06:52.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.794 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:06:52.794 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.794 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:06:52.794 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.121541+0000", 2024-09-13T15:06:52.794 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.795 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.795 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:06:52.795 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.121541+0000", 2024-09-13T15:06:52.795 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.795 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.796 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:06:52.796 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.121542+0000", 2024-09-13T15:06:52.796 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9800000000000002e-07 2024-09-13T15:06:52.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.796 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:06:52.797 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.121545+0000", 2024-09-13T15:06:52.797 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.411e-06 2024-09-13T15:06:52.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.797 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:06:52.797 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.121560+0000", 2024-09-13T15:06:52.798 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5064e-05 2024-09-13T15:06:52.798 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.798 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.798 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:06:52.798 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.121568+0000", 2024-09-13T15:06:52.798 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2060000000000001e-06 2024-09-13T15:06:52.799 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.799 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.799 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.799 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.121668+0000", 2024-09-13T15:06:52.799 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000100506 2024-09-13T15:06:52.799 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.800 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.800 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:06:52.800 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.121676+0000", 2024-09-13T15:06:52.800 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0039999999999999e-06 2024-09-13T15:06:52.800 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.800 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.800 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.801 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.134615+0000", 2024-09-13T15:06:52.801 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0129393790000001 2024-09-13T15:06:52.801 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.801 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.801 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:06:52.801 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.134675+0000", 2024-09-13T15:06:52.801 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9862999999999999e-05 2024-09-13T15:06:52.801 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.802 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.802 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T15:06:52.802 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.134765+0000", 2024-09-13T15:06:52.802 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9928999999999999e-05 2024-09-13T15:06:52.802 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.802 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.802 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:06:52.802 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.139563+0000", 2024-09-13T15:06:52.802 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0047982160000000001 2024-09-13T15:06:52.803 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.803 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.803 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.803 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.139730+0000", 2024-09-13T15:06:52.803 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000166844 2024-09-13T15:06:52.803 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.803 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.804 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.804 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.139771+0000", 2024-09-13T15:06:52.804 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0435000000000002e-05 2024-09-13T15:06:52.804 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.804 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.805 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:06:52.805 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.139815+0000", 2024-09-13T15:06:52.805 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4168000000000001e-05 2024-09-13T15:06:52.805 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.805 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.805 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:06:52.806 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.139845+0000", 2024-09-13T15:06:52.806 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9895000000000001e-05 2024-09-13T15:06:52.806 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.806 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:06:52.806 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.806 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.807 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.807 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.36396.0:197306 10.d 10:bf3adaf7:::benchmark_data_smithi151_288181_object71693:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1293)", 2024-09-13T15:06:52.807 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T15:01:10.133096+0000", 2024-09-13T15:06:52.807 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 342.57204127900002, 2024-09-13T15:06:52.808 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0434554810000001, 2024-09-13T15:06:52.808 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:06:52.808 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:06:52.808 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:52.808 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:06:52.808 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.36396", 2024-09-13T15:06:52.808 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:06:52.808 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 197306 2024-09-13T15:06:52.808 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.808 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:06:52.808 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.808 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:06:52.808 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.133096+0000", 2024-09-13T15:06:52.809 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.809 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.809 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.809 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:06:52.809 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.133096+0000", 2024-09-13T15:06:52.809 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.809 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.809 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.809 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:06:52.809 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.133098+0000", 2024-09-13T15:06:52.809 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.468e-06 2024-09-13T15:06:52.809 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.809 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.810 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:06:52.810 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.133102+0000", 2024-09-13T15:06:52.810 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6129999999999997e-06 2024-09-13T15:06:52.810 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.810 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.810 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:06:52.810 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.133113+0000", 2024-09-13T15:06:52.810 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0801999999999999e-05 2024-09-13T15:06:52.810 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.810 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.810 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:06:52.810 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.133120+0000", 2024-09-13T15:06:52.810 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4150000000000002e-06 2024-09-13T15:06:52.810 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.811 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.811 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.811 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.134969+0000", 2024-09-13T15:06:52.811 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001848812 2024-09-13T15:06:52.811 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.811 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.811 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:06:52.811 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.134979+0000", 2024-09-13T15:06:52.811 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7999999999999993e-06 2024-09-13T15:06:52.811 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.811 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.811 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.811 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.135938+0000", 2024-09-13T15:06:52.812 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0009589139999999 2024-09-13T15:06:52.812 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.812 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.812 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:06:52.812 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.136025+0000", 2024-09-13T15:06:52.812 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6699000000000005e-05 2024-09-13T15:06:52.812 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.812 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.812 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T15:06:52.812 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.136131+0000", 2024-09-13T15:06:52.812 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000106585 2024-09-13T15:06:52.812 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.812 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.812 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:06:52.813 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.139568+0000", 2024-09-13T15:06:52.813 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034369919999999998 2024-09-13T15:06:52.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.813 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.813 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.813 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.139887+0000", 2024-09-13T15:06:52.813 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000319124 2024-09-13T15:06:52.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.813 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.813 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.813 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.139924+0000", 2024-09-13T15:06:52.813 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6285000000000002e-05 2024-09-13T15:06:52.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.814 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.814 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:06:52.814 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.139949+0000", 2024-09-13T15:06:52.814 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5344e-05 2024-09-13T15:06:52.814 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.814 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.814 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:06:52.814 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.176552+0000", 2024-09-13T15:06:52.814 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.036602627999999998 2024-09-13T15:06:52.814 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.814 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:06:52.814 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.814 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.814 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.815 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.36396.0:197307 10.d 10:b5551dde:::benchmark_data_smithi151_288181_object71694:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1293)", 2024-09-13T15:06:52.815 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T15:01:10.133166+0000", 2024-09-13T15:06:52.815 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 342.57197178899997, 2024-09-13T15:06:52.815 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0449684429999999, 2024-09-13T15:06:52.815 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:06:52.815 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:06:52.815 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:52.815 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:06:52.815 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.36396", 2024-09-13T15:06:52.815 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:06:52.815 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 197307 2024-09-13T15:06:52.815 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.815 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:06:52.815 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.816 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:06:52.816 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.133166+0000", 2024-09-13T15:06:52.816 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.816 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.816 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.816 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:06:52.816 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.133166+0000", 2024-09-13T15:06:52.816 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.816 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.816 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.816 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:06:52.816 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.133166+0000", 2024-09-13T15:06:52.816 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8999999999999996e-07 2024-09-13T15:06:52.817 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.817 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.817 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:06:52.817 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.133169+0000", 2024-09-13T15:06:52.817 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8700000000000001e-06 2024-09-13T15:06:52.817 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.817 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.817 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:06:52.817 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.133177+0000", 2024-09-13T15:06:52.817 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1259999999999998e-06 2024-09-13T15:06:52.817 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.817 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.817 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:06:52.817 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.133184+0000", 2024-09-13T15:06:52.818 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9630000000000001e-06 2024-09-13T15:06:52.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.818 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.818 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.818 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.135008+0000", 2024-09-13T15:06:52.818 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018237049999999999 2024-09-13T15:06:52.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.818 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.818 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:06:52.818 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.135014+0000", 2024-09-13T15:06:52.818 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7749999999999998e-06 2024-09-13T15:06:52.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.818 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.819 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.819 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.137062+0000", 2024-09-13T15:06:52.819 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0020480620000001 2024-09-13T15:06:52.819 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.819 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.819 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:06:52.819 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.137123+0000", 2024-09-13T15:06:52.819 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1520999999999996e-05 2024-09-13T15:06:52.819 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.819 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.819 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T15:06:52.819 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.137184+0000", 2024-09-13T15:06:52.819 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1017000000000002e-05 2024-09-13T15:06:52.819 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.820 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.820 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:06:52.820 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.139573+0000", 2024-09-13T15:06:52.820 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0023883659999999998 2024-09-13T15:06:52.820 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.820 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.820 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.820 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.177072+0000", 2024-09-13T15:06:52.820 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.037499313999999999 2024-09-13T15:06:52.820 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.820 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.820 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.821 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.178009+0000", 2024-09-13T15:06:52.821 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00093690899999999998 2024-09-13T15:06:52.821 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.821 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.821 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:06:52.821 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.178088+0000", 2024-09-13T15:06:52.821 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9301000000000005e-05 2024-09-13T15:06:52.821 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.821 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.821 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:06:52.821 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.178134+0000", 2024-09-13T15:06:52.821 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5924000000000003e-05 2024-09-13T15:06:52.821 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.821 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:06:52.822 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.822 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.822 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.822 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.36396.0:197320 10.d 10:bae18b2d:::benchmark_data_smithi151_288181_object71707:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1293)", 2024-09-13T15:06:52.822 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T15:01:10.147327+0000", 2024-09-13T15:06:52.822 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 342.55781069400001, 2024-09-13T15:06:52.822 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.030913051, 2024-09-13T15:06:52.822 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:06:52.822 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:06:52.822 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:52.822 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:06:52.822 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.36396", 2024-09-13T15:06:52.822 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:06:52.823 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 197320 2024-09-13T15:06:52.823 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.823 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:06:52.823 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.823 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:06:52.823 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.147327+0000", 2024-09-13T15:06:52.823 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.823 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.823 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.823 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:06:52.824 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.147327+0000", 2024-09-13T15:06:52.824 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.824 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.824 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.824 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:06:52.824 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.147328+0000", 2024-09-13T15:06:52.824 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.798e-06 2024-09-13T15:06:52.824 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.824 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.824 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:06:52.825 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.147334+0000", 2024-09-13T15:06:52.825 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3000000000000001e-06 2024-09-13T15:06:52.825 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.825 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.825 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:06:52.825 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.147346+0000", 2024-09-13T15:06:52.825 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2224e-05 2024-09-13T15:06:52.825 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.825 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.825 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:06:52.825 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.147353+0000", 2024-09-13T15:06:52.825 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1459999999999999e-06 2024-09-13T15:06:52.825 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.825 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.826 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.826 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.147565+0000", 2024-09-13T15:06:52.826 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000212127 2024-09-13T15:06:52.826 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.826 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.826 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-13T15:06:52.826 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:10.147585+0000", 2024-09-13T15:06:52.826 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0020000000000001e-05 2024-09-13T15:06:52.826 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.826 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.826 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.826 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.138194+0000", 2024-09-13T15:06:52.826 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.99060846499999999 2024-09-13T15:06:52.826 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.827 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:06:52.827 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.138284+0000", 2024-09-13T15:06:52.827 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0218999999999998e-05 2024-09-13T15:06:52.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.827 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-13T15:06:52.827 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.138373+0000", 2024-09-13T15:06:52.827 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8774000000000001e-05 2024-09-13T15:06:52.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.828 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:06:52.828 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.139577+0000", 2024-09-13T15:06:52.828 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012043850000000001 2024-09-13T15:06:52.828 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.828 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.828 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.829 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.177892+0000", 2024-09-13T15:06:52.829 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038315183000000003 2024-09-13T15:06:52.829 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.829 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.829 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.829 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.178179+0000", 2024-09-13T15:06:52.829 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028668699999999999 2024-09-13T15:06:52.829 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.829 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.829 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:06:52.830 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.178207+0000", 2024-09-13T15:06:52.830 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7902000000000002e-05 2024-09-13T15:06:52.830 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.830 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.830 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:06:52.830 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:01:11.178240+0000", 2024-09-13T15:06:52.830 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2821000000000003e-05 2024-09-13T15:06:52.830 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.830 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:06:52.830 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.830 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.830 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.830 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.36666.0:185104 11.2 11:4a55edbf:::benchmark_data_smithi151_323728_object63807:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1349)", 2024-09-13T15:06:52.830 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T15:05:22.485337+0000", 2024-09-13T15:06:52.831 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 90.219800859000003, 2024-09-13T15:06:52.831 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35852753999999998, 2024-09-13T15:06:52.831 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:06:52.831 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:06:52.831 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:52.831 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:06:52.831 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.36666", 2024-09-13T15:06:52.831 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:06:52.831 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 185104 2024-09-13T15:06:52.831 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.831 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:06:52.831 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.831 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:06:52.831 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:22.485337+0000", 2024-09-13T15:06:52.832 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.832 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.832 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.832 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:06:52.832 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:22.485337+0000", 2024-09-13T15:06:52.832 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.832 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.832 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.832 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:06:52.832 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:22.485338+0000", 2024-09-13T15:06:52.832 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.186e-06 2024-09-13T15:06:52.832 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.832 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.832 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:06:52.833 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:22.485343+0000", 2024-09-13T15:06:52.833 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9370000000000003e-06 2024-09-13T15:06:52.833 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.833 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.833 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:06:52.833 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:22.485356+0000", 2024-09-13T15:06:52.833 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3179999999999999e-05 2024-09-13T15:06:52.833 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.833 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.833 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:06:52.833 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:22.485363+0000", 2024-09-13T15:06:52.833 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5440000000000001e-06 2024-09-13T15:06:52.833 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.833 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.834 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.834 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:22.486868+0000", 2024-09-13T15:06:52.834 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015042129999999999 2024-09-13T15:06:52.834 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.834 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.834 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:06:52.834 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:22.487042+0000", 2024-09-13T15:06:52.834 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000173936 2024-09-13T15:06:52.834 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.834 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.834 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-13T15:06:52.834 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:22.487195+0000", 2024-09-13T15:06:52.834 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000153986 2024-09-13T15:06:52.834 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.834 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.835 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:06:52.835 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:22.496031+0000", 2024-09-13T15:06:52.835 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0088350070000000006 2024-09-13T15:06:52.835 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.835 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.835 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.835 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:22.511883+0000", 2024-09-13T15:06:52.835 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015852026000000002 2024-09-13T15:06:52.835 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.835 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.835 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.835 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:22.839967+0000", 2024-09-13T15:06:52.835 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32808457099999999 2024-09-13T15:06:52.835 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.836 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.836 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:06:52.836 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:52.837 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:22.840031+0000", 2024-09-13T15:06:52.837 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3948000000000005e-05 2024-09-13T15:06:52.837 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.838 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.838 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:06:52.838 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:22.843864+0000", 2024-09-13T15:06:52.838 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0038330059999999999 2024-09-13T15:06:52.838 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.838 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:06:52.838 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.838 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.838 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.838 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.36666.0:185110 11.2 11:408f4fb6:::benchmark_data_smithi151_323728_object63813:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1349)", 2024-09-13T15:06:52.838 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T15:05:22.489277+0000", 2024-09-13T15:06:52.838 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 90.215860245000002, 2024-09-13T15:06:52.838 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35673818200000001, 2024-09-13T15:06:52.838 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:06:52.839 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:06:52.839 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:52.839 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:06:52.839 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.36666", 2024-09-13T15:06:52.839 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:06:52.839 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 185110 2024-09-13T15:06:52.839 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.839 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:06:52.839 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.839 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:06:52.839 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:22.489277+0000", 2024-09-13T15:06:52.839 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.839 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.839 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.840 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:06:52.840 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:22.489277+0000", 2024-09-13T15:06:52.840 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.840 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.840 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.840 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:06:52.840 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:22.489278+0000", 2024-09-13T15:06:52.840 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1899999999999998e-07 2024-09-13T15:06:52.840 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.840 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.840 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:06:52.840 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:22.489280+0000", 2024-09-13T15:06:52.840 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2579999999999998e-06 2024-09-13T15:06:52.841 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.841 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:06:52.841 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:22.489286+0000", 2024-09-13T15:06:52.841 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3439999999999997e-06 2024-09-13T15:06:52.841 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.841 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:06:52.841 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:22.489291+0000", 2024-09-13T15:06:52.841 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0370000000000002e-06 2024-09-13T15:06:52.841 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.841 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.841 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:22.491396+0000", 2024-09-13T15:06:52.842 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021052269999999999 2024-09-13T15:06:52.842 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.842 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:06:52.842 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:22.491536+0000", 2024-09-13T15:06:52.842 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013999100000000001 2024-09-13T15:06:52.842 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.842 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-13T15:06:52.842 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:22.491680+0000", 2024-09-13T15:06:52.842 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014398899999999999 2024-09-13T15:06:52.842 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.842 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:06:52.843 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:22.496136+0000", 2024-09-13T15:06:52.843 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0044551979999999996 2024-09-13T15:06:52.843 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.843 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.843 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:22.511936+0000", 2024-09-13T15:06:52.843 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015800440999999998 2024-09-13T15:06:52.843 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.843 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.843 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:22.843916+0000", 2024-09-13T15:06:52.843 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33198006400000002 2024-09-13T15:06:52.843 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.843 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:06:52.844 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:22.843984+0000", 2024-09-13T15:06:52.844 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7900999999999999e-05 2024-09-13T15:06:52.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.844 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.844 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:06:52.844 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:22.846015+0000", 2024-09-13T15:06:52.844 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020313129999999999 2024-09-13T15:06:52.844 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.844 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:06:52.844 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.844 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.844 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.36666.0:200580 11.2 11:465b12ab:::benchmark_data_smithi151_323728_object79283:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1349)", 2024-09-13T15:06:52.845 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T15:05:36.457683+0000", 2024-09-13T15:06:52.845 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 76.247454169999997, 2024-09-13T15:06:52.845 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33145375199999999, 2024-09-13T15:06:52.845 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:06:52.845 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:06:52.845 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:52.845 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:06:52.845 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.36666", 2024-09-13T15:06:52.845 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:06:52.845 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 200580 2024-09-13T15:06:52.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.845 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:06:52.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.845 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:06:52.846 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.457683+0000", 2024-09-13T15:06:52.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.846 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:06:52.846 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.457683+0000", 2024-09-13T15:06:52.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.846 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:06:52.846 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.457684+0000", 2024-09-13T15:06:52.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4200000000000004e-07 2024-09-13T15:06:52.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.847 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:06:52.847 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.457688+0000", 2024-09-13T15:06:52.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4510000000000001e-06 2024-09-13T15:06:52.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.847 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:06:52.847 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.457698+0000", 2024-09-13T15:06:52.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0519999999999999e-05 2024-09-13T15:06:52.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.847 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:06:52.847 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.457705+0000", 2024-09-13T15:06:52.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8689999999999998e-06 2024-09-13T15:06:52.848 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.848 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.848 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.460931+0000", 2024-09-13T15:06:52.848 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0032255640000000002 2024-09-13T15:06:52.848 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.848 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:06:52.848 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.461062+0000", 2024-09-13T15:06:52.848 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013173199999999999 2024-09-13T15:06:52.848 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.848 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-13T15:06:52.848 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.461202+0000", 2024-09-13T15:06:52.849 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000139925 2024-09-13T15:06:52.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.849 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.849 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.849 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.525316+0000", 2024-09-13T15:06:52.849 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.064113376999999999 2024-09-13T15:06:52.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.849 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.849 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.849 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.786965+0000", 2024-09-13T15:06:52.849 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.261649364 2024-09-13T15:06:52.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.849 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.849 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:06:52.849 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.786978+0000", 2024-09-13T15:06:52.850 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2832e-05 2024-09-13T15:06:52.850 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.850 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.850 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:06:52.850 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.787025+0000", 2024-09-13T15:06:52.850 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7447000000000002e-05 2024-09-13T15:06:52.850 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.850 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.850 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:06:52.850 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.789137+0000", 2024-09-13T15:06:52.850 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021117290000000001 2024-09-13T15:06:52.850 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.850 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:06:52.850 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.851 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.851 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.851 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.36666.0:200584 11.b 11:d070a024:::benchmark_data_smithi151_323728_object79287:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1349)", 2024-09-13T15:06:52.851 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T15:05:36.460730+0000", 2024-09-13T15:06:52.851 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 76.244407824000007, 2024-09-13T15:06:52.851 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.323865244, 2024-09-13T15:06:52.851 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:06:52.851 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:06:52.851 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:52.851 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:06:52.851 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.36666", 2024-09-13T15:06:52.851 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:06:52.851 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 200584 2024-09-13T15:06:52.852 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.852 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:06:52.852 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.852 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:06:52.852 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.460730+0000", 2024-09-13T15:06:52.852 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.852 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.852 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.852 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:06:52.852 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.460730+0000", 2024-09-13T15:06:52.852 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.852 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.852 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.852 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:06:52.853 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.460730+0000", 2024-09-13T15:06:52.853 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5099999999999999e-07 2024-09-13T15:06:52.853 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.853 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.853 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:06:52.853 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.460734+0000", 2024-09-13T15:06:52.853 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4230000000000001e-06 2024-09-13T15:06:52.853 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.853 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.853 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:06:52.853 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.460742+0000", 2024-09-13T15:06:52.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2519999999999999e-06 2024-09-13T15:06:52.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:06:52.854 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.460749+0000", 2024-09-13T15:06:52.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.579e-06 2024-09-13T15:06:52.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.854 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.519304+0000", 2024-09-13T15:06:52.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.058555426000000001 2024-09-13T15:06:52.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:06:52.855 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.519440+0000", 2024-09-13T15:06:52.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013613699999999999 2024-09-13T15:06:52.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.855 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:06:52.855 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.519572+0000", 2024-09-13T15:06:52.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000131617 2024-09-13T15:06:52.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.855 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:06:52.855 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.525195+0000", 2024-09-13T15:06:52.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0056228110000000001 2024-09-13T15:06:52.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.856 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.856 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.525241+0000", 2024-09-13T15:06:52.856 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6640000000000001e-05 2024-09-13T15:06:52.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.856 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.856 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.856 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.784345+0000", 2024-09-13T15:06:52.856 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25910340999999998 2024-09-13T15:06:52.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.856 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.856 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:06:52.856 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.784556+0000", 2024-09-13T15:06:52.856 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000211447 2024-09-13T15:06:52.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.857 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.857 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:06:52.857 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.784595+0000", 2024-09-13T15:06:52.857 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8751000000000002e-05 2024-09-13T15:06:52.857 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.857 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:06:52.857 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.857 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.857 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.857 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.36666.0:200585 11.b 11:dccd30a5:::benchmark_data_smithi151_323728_object79288:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1349)", 2024-09-13T15:06:52.857 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T15:05:36.460830+0000", 2024-09-13T15:06:52.857 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 76.244306988999995, 2024-09-13T15:06:52.857 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32390214099999998, 2024-09-13T15:06:52.857 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:06:52.858 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:06:52.858 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:52.858 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:06:52.858 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.36666", 2024-09-13T15:06:52.858 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:06:52.858 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 200585 2024-09-13T15:06:52.858 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.858 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:06:52.858 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.858 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:06:52.858 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.460830+0000", 2024-09-13T15:06:52.858 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.858 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.858 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.859 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:06:52.859 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.460830+0000", 2024-09-13T15:06:52.859 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.859 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.859 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.859 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:06:52.859 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.460831+0000", 2024-09-13T15:06:52.859 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1500000000000003e-07 2024-09-13T15:06:52.859 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.859 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.859 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:06:52.859 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.460836+0000", 2024-09-13T15:06:52.859 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4800000000000003e-06 2024-09-13T15:06:52.859 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.859 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.860 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:06:52.860 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.460848+0000", 2024-09-13T15:06:52.860 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2264e-05 2024-09-13T15:06:52.860 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.860 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.860 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:06:52.860 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.460855+0000", 2024-09-13T15:06:52.860 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2490000000000004e-06 2024-09-13T15:06:52.860 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.860 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.860 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.860 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.521772+0000", 2024-09-13T15:06:52.860 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.060916861000000003 2024-09-13T15:06:52.861 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.861 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:06:52.861 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.521865+0000", 2024-09-13T15:06:52.861 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3159999999999996e-05 2024-09-13T15:06:52.861 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.861 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:06:52.861 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.522049+0000", 2024-09-13T15:06:52.861 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000183647 2024-09-13T15:06:52.861 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.861 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:06:52.861 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.525199+0000", 2024-09-13T15:06:52.862 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031502549999999998 2024-09-13T15:06:52.862 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.862 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.862 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.862 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.525280+0000", 2024-09-13T15:06:52.862 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0600999999999996e-05 2024-09-13T15:06:52.862 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.862 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.862 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.862 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.784649+0000", 2024-09-13T15:06:52.862 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25936910299999999 2024-09-13T15:06:52.862 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.862 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.862 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:06:52.863 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.784700+0000", 2024-09-13T15:06:52.863 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0766000000000001e-05 2024-09-13T15:06:52.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.863 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:06:52.863 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.784733+0000", 2024-09-13T15:06:52.863 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2839999999999997e-05 2024-09-13T15:06:52.863 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.863 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:06:52.863 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.863 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.36666.0:200589 11.2 11:43c20e00:::benchmark_data_smithi151_323728_object79292:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1349)", 2024-09-13T15:06:52.863 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T15:05:36.519032+0000", 2024-09-13T15:06:52.864 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 76.186105190999996, 2024-09-13T15:06:52.864 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27986589000000001, 2024-09-13T15:06:52.864 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:06:52.864 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:06:52.864 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:52.864 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:06:52.864 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.36666", 2024-09-13T15:06:52.864 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:06:52.864 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 200589 2024-09-13T15:06:52.864 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.864 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:06:52.864 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.864 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:06:52.864 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.519032+0000", 2024-09-13T15:06:52.864 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.865 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.865 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:06:52.865 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.519032+0000", 2024-09-13T15:06:52.865 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.865 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.865 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:06:52.865 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.519033+0000", 2024-09-13T15:06:52.865 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2200000000000004e-07 2024-09-13T15:06:52.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.865 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.865 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:06:52.866 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.519035+0000", 2024-09-13T15:06:52.866 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.543e-06 2024-09-13T15:06:52.866 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.866 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:06:52.866 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.519042+0000", 2024-09-13T15:06:52.866 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8530000000000003e-06 2024-09-13T15:06:52.866 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.866 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:06:52.866 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.519047+0000", 2024-09-13T15:06:52.866 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3669999999999996e-06 2024-09-13T15:06:52.866 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.867 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.867 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.535606+0000", 2024-09-13T15:06:52.867 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016558658 2024-09-13T15:06:52.867 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.867 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.867 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:06:52.867 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.535734+0000", 2024-09-13T15:06:52.867 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000128248 2024-09-13T15:06:52.868 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.868 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.868 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-13T15:06:52.868 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.784167+0000", 2024-09-13T15:06:52.868 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24843293599999999 2024-09-13T15:06:52.868 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.868 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.868 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:06:52.868 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.792575+0000", 2024-09-13T15:06:52.868 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0084077000000000006 2024-09-13T15:06:52.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.869 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.869 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.792618+0000", 2024-09-13T15:06:52.869 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3238999999999997e-05 2024-09-13T15:06:52.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.869 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.869 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.796447+0000", 2024-09-13T15:06:52.869 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00382855 2024-09-13T15:06:52.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.869 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:06:52.870 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.796693+0000", 2024-09-13T15:06:52.870 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024590499999999998 2024-09-13T15:06:52.870 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.870 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.870 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:06:52.870 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.798898+0000", 2024-09-13T15:06:52.870 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002205369 2024-09-13T15:06:52.870 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.870 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:06:52.870 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.870 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.870 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.870 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.36666.0:200590 11.a 11:581d1477:::benchmark_data_smithi151_323728_object79293:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1349)", 2024-09-13T15:06:52.870 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T15:05:36.519055+0000", 2024-09-13T15:06:52.871 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 76.186082795999994, 2024-09-13T15:06:52.871 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27653478300000001, 2024-09-13T15:06:52.871 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:06:52.871 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:06:52.871 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:52.871 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:06:52.871 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.36666", 2024-09-13T15:06:52.871 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:06:52.871 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 200590 2024-09-13T15:06:52.871 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.871 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:06:52.871 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.871 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:06:52.871 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.519055+0000", 2024-09-13T15:06:52.872 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.872 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.872 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:06:52.872 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.519055+0000", 2024-09-13T15:06:52.872 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.872 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.872 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:06:52.872 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.519055+0000", 2024-09-13T15:06:52.872 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4200000000000002e-07 2024-09-13T15:06:52.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.872 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.872 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:06:52.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.519057+0000", 2024-09-13T15:06:52.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.322e-06 2024-09-13T15:06:52.873 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.873 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.873 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:06:52.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.519063+0000", 2024-09-13T15:06:52.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8969999999999996e-06 2024-09-13T15:06:52.873 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.873 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.873 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:06:52.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.519068+0000", 2024-09-13T15:06:52.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.972e-06 2024-09-13T15:06:52.873 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.873 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.784102+0000", 2024-09-13T15:06:52.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26503353600000001 2024-09-13T15:06:52.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:06:52.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.784235+0000", 2024-09-13T15:06:52.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013353 2024-09-13T15:06:52.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-13T15:06:52.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.784355+0000", 2024-09-13T15:06:52.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011957 2024-09-13T15:06:52.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.875 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.875 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.789179+0000", 2024-09-13T15:06:52.875 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.004823763 2024-09-13T15:06:52.875 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.875 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.875 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.789211+0000", 2024-09-13T15:06:52.875 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2163999999999999e-05 2024-09-13T15:06:52.875 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.875 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:06:52.875 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.795489+0000", 2024-09-13T15:06:52.876 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0062779389999999997 2024-09-13T15:06:52.876 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.876 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.876 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:06:52.876 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.795542+0000", 2024-09-13T15:06:52.876 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3220000000000002e-05 2024-09-13T15:06:52.876 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.876 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.876 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:06:52.876 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.795589+0000", 2024-09-13T15:06:52.876 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7528000000000003e-05 2024-09-13T15:06:52.876 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.876 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:06:52.876 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.876 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.877 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.877 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.36666.0:200599 11.0 e1349/1329 11:0a156054:::benchmark_data_smithi151_323728_object79302:head v 1349'12366, mlcod=1349'12363)", 2024-09-13T15:06:52.877 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T15:05:36.534886+0000", 2024-09-13T15:06:52.877 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 76.170251449000006, 2024-09-13T15:06:52.877 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26893484099999998, 2024-09-13T15:06:52.877 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:06:52.877 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:06:52.877 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:52.877 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:06:52.877 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.877 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:06:52.877 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.534886+0000", 2024-09-13T15:06:52.877 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.877 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.878 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.878 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:06:52.878 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.534886+0000", 2024-09-13T15:06:52.878 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.878 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.878 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.878 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:06:52.878 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.534887+0000", 2024-09-13T15:06:52.878 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7599999999999997e-07 2024-09-13T15:06:52.878 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.878 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.878 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:06:52.878 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.534892+0000", 2024-09-13T15:06:52.878 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9660000000000004e-06 2024-09-13T15:06:52.878 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.879 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:06:52.879 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.534899+0000", 2024-09-13T15:06:52.879 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9009999999999994e-06 2024-09-13T15:06:52.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.879 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:06:52.879 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.534905+0000", 2024-09-13T15:06:52.879 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2329999999999997e-06 2024-09-13T15:06:52.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.879 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.879 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.784357+0000", 2024-09-13T15:06:52.879 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.249452282 2024-09-13T15:06:52.880 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.880 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.880 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:06:52.880 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.784376+0000", 2024-09-13T15:06:52.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9355999999999999e-05 2024-09-13T15:06:52.880 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.880 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.880 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:06:52.880 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.803769+0000", 2024-09-13T15:06:52.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019392665 2024-09-13T15:06:52.880 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.880 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.880 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:06:52.880 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.803821+0000", 2024-09-13T15:06:52.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1761999999999997e-05 2024-09-13T15:06:52.881 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.881 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:06:52.881 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.881 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.881 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.881 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.36666.0:200606 11.2 11:44fb8a6b:::benchmark_data_smithi151_323728_object79309:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1349)", 2024-09-13T15:06:52.881 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-13T15:05:36.536137+0000", 2024-09-13T15:06:52.881 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 76.169000284999996, 2024-09-13T15:06:52.881 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27369212300000001, 2024-09-13T15:06:52.881 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-13T15:06:52.881 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-13T15:06:52.881 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:52.881 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-13T15:06:52.881 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.36666", 2024-09-13T15:06:52.882 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:06:52.882 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 200606 2024-09-13T15:06:52.882 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.882 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-13T15:06:52.882 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.882 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-13T15:06:52.882 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.536137+0000", 2024-09-13T15:06:52.882 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.882 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.882 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.882 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-13T15:06:52.882 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.536137+0000", 2024-09-13T15:06:52.882 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-13T15:06:52.882 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.883 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.883 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-13T15:06:52.883 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.536187+0000", 2024-09-13T15:06:52.883 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9490000000000002e-05 2024-09-13T15:06:52.883 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.883 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.883 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-13T15:06:52.883 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.536192+0000", 2024-09-13T15:06:52.883 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0640000000000001e-06 2024-09-13T15:06:52.883 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.883 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.883 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-13T15:06:52.883 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.784069+0000", 2024-09-13T15:06:52.883 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24787767499999999 2024-09-13T15:06:52.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.884 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.884 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-13T15:06:52.884 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.784081+0000", 2024-09-13T15:06:52.884 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1864e-05 2024-09-13T15:06:52.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.884 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.884 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-13T15:06:52.884 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.789240+0000", 2024-09-13T15:06:52.884 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.005158775 2024-09-13T15:06:52.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.884 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.884 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-13T15:06:52.884 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.789356+0000", 2024-09-13T15:06:52.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000115639 2024-09-13T15:06:52.885 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.885 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.885 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-13T15:06:52.885 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.789478+0000", 2024-09-13T15:06:52.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012214699999999999 2024-09-13T15:06:52.885 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.885 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.885 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.885 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.792829+0000", 2024-09-13T15:06:52.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033513319999999998 2024-09-13T15:06:52.885 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.885 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.885 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-13T15:06:52.886 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.795612+0000", 2024-09-13T15:06:52.886 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002782475 2024-09-13T15:06:52.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.886 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.886 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:52.886 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.804201+0000", 2024-09-13T15:06:52.886 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.008589658 2024-09-13T15:06:52.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.886 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.886 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-13T15:06:52.886 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.804264+0000", 2024-09-13T15:06:52.886 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2532000000000004e-05 2024-09-13T15:06:52.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-13T15:06:52.886 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-13T15:06:52.887 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-13T15:06:52.887 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-13T15:05:36.809829+0000", 2024-09-13T15:06:52.887 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0055654720000000001 2024-09-13T15:06:52.887 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.887 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:06:52.887 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.887 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-13T15:06:52.887 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-13T15:06:52.887 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-13T15:06:52.908 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:52.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:52.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:52.971 INFO:teuthology.orchestra.run.smithi151.stderr:marked in osd.0. 2024-09-13T15:06:53.010 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:53.033 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:53.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:53.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:53.135 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:53.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:53.158 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:53.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:53.259 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:53.284 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:53.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:53.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:53.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:52 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:06:53.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:52 smithi184 ceph-mon[92527]: pgmap v2390: 103 pgs: 103 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail 2024-09-13T15:06:53.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:52 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/933167792' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-13T15:06:53.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:52 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-13T15:06:53.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:52 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:06:53.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:52 smithi151 ceph-mon[104450]: pgmap v2390: 103 pgs: 103 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail 2024-09-13T15:06:53.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:52 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/933167792' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-13T15:06:53.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:52 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-13T15:06:53.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:52 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:06:53.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:52 smithi151 ceph-mon[108966]: pgmap v2390: 103 pgs: 103 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail 2024-09-13T15:06:53.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:52 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/933167792' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-13T15:06:53.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:52 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-13T15:06:53.386 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:53.412 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:53.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:53.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:53.514 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:53.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:53.536 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:53.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:53.637 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:53.663 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:53.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:53.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:53.764 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:53.789 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:53.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:53.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:53.803 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:53.890 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:53.912 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:53.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:53.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:54.013 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:54.037 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:54.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:54.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:54.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:53 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-09-13T15:06:54.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:53 smithi151 ceph-mon[104450]: osdmap e1362: 8 total, 8 up, 6 in 2024-09-13T15:06:54.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:53 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-09-13T15:06:54.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:53 smithi151 ceph-mon[108966]: osdmap e1362: 8 total, 8 up, 6 in 2024-09-13T15:06:54.139 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:54.162 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:54.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:54.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:54.263 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:54.285 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:54.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:54.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:54.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:53 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-09-13T15:06:54.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:53 smithi184 ceph-mon[92527]: osdmap e1362: 8 total, 8 up, 6 in 2024-09-13T15:06:54.387 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:54.411 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:54.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:54.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:54.513 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:54.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:54.536 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:54.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:54.637 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:54.663 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:54.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:54.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:54.718 DEBUG:teuthology.orchestra.run.smithi151:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-13T15:06:54.765 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:54.791 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:54.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:54.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:54.800 INFO:tasks.workunit.client.0.smithi151.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-13T15:06:54.804 INFO:tasks.workunit.client.0.smithi151.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-13T15:06:54.804 INFO:tasks.workunit.client.0.smithi151.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-13T15:06:54.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:06:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:06:54] "GET /metrics HTTP/1.1" 200 34291 "" "Prometheus/2.51.0" 2024-09-13T15:06:54.893 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:54.915 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:54.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:54.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:54.950 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T15:06:55.017 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:55.039 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:55.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:55.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:55.141 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:55.161 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:55.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:55.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:55.209 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:06:55.263 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:55.285 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:55.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:55.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:55.293 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:54 smithi151 ceph-mon[108966]: pgmap v2392: 103 pgs: 103 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail 2024-09-13T15:06:55.294 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:54 smithi151 ceph-mon[108966]: osdmap e1363: 8 total, 8 up, 6 in 2024-09-13T15:06:55.294 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:54 smithi151 ceph-mon[104450]: pgmap v2392: 103 pgs: 103 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail 2024-09-13T15:06:55.294 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:54 smithi151 ceph-mon[104450]: osdmap e1363: 8 total, 8 up, 6 in 2024-09-13T15:06:55.329 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:54 smithi184 ceph-mon[92527]: pgmap v2392: 103 pgs: 103 active+clean; 582 KiB data, 6.4 GiB used, 441 GiB / 447 GiB avail 2024-09-13T15:06:55.329 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:54 smithi184 ceph-mon[92527]: osdmap e1363: 8 total, 8 up, 6 in 2024-09-13T15:06:55.388 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:55.409 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:55.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:55.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:55.511 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:55.533 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:55.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:55.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:55.635 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:55.656 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:55.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:55.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:55.758 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:55.780 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:55.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:55.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:55.881 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:55.903 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:55.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:55.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:55.937 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-09-13T15:06:56.005 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:56.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:56.029 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:56.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:56.130 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:56.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:56.153 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:56.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:56.254 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:56.276 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:56.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:56.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:56.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:56 smithi184 ceph-mon[92527]: Health check failed: Reduced data availability: 7 pgs inactive (PG_AVAILABILITY) 2024-09-13T15:06:56.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:56 smithi184 ceph-mon[92527]: Health check failed: Degraded data redundancy: 3/78 objects degraded (3.846%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:06:56.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:56 smithi151 ceph-mon[104450]: Health check failed: Reduced data availability: 7 pgs inactive (PG_AVAILABILITY) 2024-09-13T15:06:56.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:56 smithi151 ceph-mon[104450]: Health check failed: Degraded data redundancy: 3/78 objects degraded (3.846%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:06:56.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:56 smithi151 ceph-mon[108966]: Health check failed: Reduced data availability: 7 pgs inactive (PG_AVAILABILITY) 2024-09-13T15:06:56.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:56 smithi151 ceph-mon[108966]: Health check failed: Degraded data redundancy: 3/78 objects degraded (3.846%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:06:56.378 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:56.399 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:56.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:56.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:56.501 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:56.523 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:56.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:56.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:56.624 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:56.646 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:56.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:56.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:56.748 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:56.769 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:56.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:56.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:56.871 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:56.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:56.894 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:56.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:56.995 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:57.018 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:57.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:57.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:57.119 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:57.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:57.142 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:57.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:57.159 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T15:06:57.160 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [], 2024-09-13T15:06:57.160 INFO:teuthology.orchestra.run.smithi184.stdout: "num_ops": 0 2024-09-13T15:06:57.160 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T15:06:57.243 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:57.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:57.266 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:57.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:57.356 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:57 smithi184 ceph-mon[92527]: pgmap v2394: 103 pgs: 2 activating+degraded, 12 activating, 89 active+clean; 582 KiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 3/78 objects degraded (3.846%); 40 KiB/s, 0 objects/s recovering 2024-09-13T15:06:57.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:57 smithi151 ceph-mon[104450]: pgmap v2394: 103 pgs: 2 activating+degraded, 12 activating, 89 active+clean; 582 KiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 3/78 objects degraded (3.846%); 40 KiB/s, 0 objects/s recovering 2024-09-13T15:06:57.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:57 smithi151 ceph-mon[108966]: pgmap v2394: 103 pgs: 2 activating+degraded, 12 activating, 89 active+clean; 582 KiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 3/78 objects degraded (3.846%); 40 KiB/s, 0 objects/s recovering 2024-09-13T15:06:57.367 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:57.388 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:57.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:57.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:57.490 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:57.511 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:57.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:57.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:57.613 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:57.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:57.637 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:57.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:57.638 INFO:tasks.workunit.client.0.smithi151.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-13T15:06:57.728 INFO:tasks.workunit.client.0.smithi151.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-13T15:06:57.737 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:57.759 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:57.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:57.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:57.771 INFO:tasks.workunit.client.0.smithi151.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-13T15:06:57.773 INFO:tasks.workunit.client.0.smithi151.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-13T15:06:57.773 INFO:tasks.workunit.client.0.smithi151.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-13T15:06:57.787 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T15:06:57.812 INFO:tasks.workunit.client.0.smithi151.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-13T15:06:57.858 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-13T15:06:57.858 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-13T15:06:57.860 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:57.887 INFO:tasks.workunit:Running workunits matching cls/test_cls_rbd.sh on client.0... 2024-09-13T15:06:57.889 INFO:tasks.workunit:Running workunit cls/test_cls_rbd.sh... 2024-09-13T15:06:57.889 DEBUG:teuthology.orchestra.run.smithi151:workunit test cls/test_cls_rbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh 2024-09-13T15:06:57.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:57.893 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:57.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:57.994 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:58.006 INFO:tasks.workunit.client.0.smithi151.stdout:Running main() from gmock_main.cc 2024-09-13T15:06:58.007 INFO:tasks.workunit.client.0.smithi151.stdout:[==========] Running 70 tests from 1 test suite. 2024-09-13T15:06:58.007 INFO:tasks.workunit.client.0.smithi151.stdout:[----------] Global test environment set-up. 2024-09-13T15:06:58.007 INFO:tasks.workunit.client.0.smithi151.stdout:[----------] 70 tests from TestClsRbd 2024-09-13T15:06:58.017 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:58.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:58.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:58.073 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:06:58.118 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:58.140 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:58.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:58.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:58.242 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:58.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:58.264 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:58.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:58.365 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:58.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:58.387 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:58.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:58.488 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:58.510 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:58.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:58.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:58.611 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:58.633 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:58.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:58.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:58.734 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:58.756 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:58.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:58.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:58.858 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:58.878 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:58.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:58.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:58.980 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:59.002 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:59.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:59.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:59.034 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-13T15:06:59.104 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:59.126 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:59.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:59.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:59.205 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:59 smithi184 ceph-mon[92527]: pgmap v2395: 103 pgs: 2 activating+degraded, 12 activating, 89 active+clean; 582 KiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 3/78 objects degraded (3.846%); 41 KiB/s, 1 keys/s, 1 objects/s recovering 2024-09-13T15:06:59.205 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:06:59 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/827215399' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:59.228 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:59.249 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:59.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:59.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:59.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:59.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:59 smithi151 ceph-mon[104450]: pgmap v2395: 103 pgs: 2 activating+degraded, 12 activating, 89 active+clean; 582 KiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 3/78 objects degraded (3.846%); 41 KiB/s, 1 keys/s, 1 objects/s recovering 2024-09-13T15:06:59.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:06:59 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/827215399' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:59.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:59 smithi151 ceph-mon[108966]: pgmap v2395: 103 pgs: 2 activating+degraded, 12 activating, 89 active+clean; 582 KiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 3/78 objects degraded (3.846%); 41 KiB/s, 1 keys/s, 1 objects/s recovering 2024-09-13T15:06:59.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:06:59 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/827215399' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:59.372 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:59.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:59.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:59.474 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:59.495 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:59.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:59.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:59.597 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:59.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:59.620 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:06:59.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:59.629 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T15:06:59.630 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:06:59.647 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T15:06:59.647 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:06:59.663 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T15:06:59.663 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:06:59.679 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T15:06:59.680 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:06:59.695 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T15:06:59.695 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:06:59.714 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T15:06:59.714 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:06:59.721 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:59.729 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T15:06:59.730 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:06:59.743 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:59.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:59.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:59.745 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T15:06:59.745 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:06:59.758 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T15:06:59.758 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:06:59.845 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:59.866 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:59.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:59.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:59.968 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:59.989 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:06:59.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:59.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:00.052 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.get_all_features 2024-09-13T15:07:00.091 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:00.093 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.get_all_features (41 ms) 2024-09-13T15:07:00.093 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.copyup 2024-09-13T15:07:00.108 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:00 smithi151 ceph-mon[108966]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:07:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:00 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/827215399' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:07:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:00 smithi151 ceph-mon[108966]: osdmap e1364: 8 total, 8 up, 6 in 2024-09-13T15:07:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:00 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1422272862' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi151-350457-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-13T15:07:00.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:00 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2169974088' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:07:00.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:00 smithi151 ceph-mon[104450]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:07:00.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:00 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/827215399' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:07:00.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:00 smithi151 ceph-mon[104450]: osdmap e1364: 8 total, 8 up, 6 in 2024-09-13T15:07:00.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:00 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1422272862' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi151-350457-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-13T15:07:00.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:00 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2169974088' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:07:00.114 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:07:00.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:00.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:00.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:00.237 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:07:00.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:00.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:00.279 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.copyup (186 ms) 2024-09-13T15:07:00.279 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.sparse_copyup 2024-09-13T15:07:00.309 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.sparse_copyup (30 ms) 2024-09-13T15:07:00.309 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.get_and_set_id 2024-09-13T15:07:00.316 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.get_and_set_id (8 ms) 2024-09-13T15:07:00.316 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.add_remove_child 2024-09-13T15:07:00.326 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.add_remove_child (9 ms) 2024-09-13T15:07:00.326 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.directory_methods 2024-09-13T15:07:00.339 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:00.350 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.directory_methods (25 ms) 2024-09-13T15:07:00.351 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.create 2024-09-13T15:07:00.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:00 smithi184 ceph-mon[92527]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:07:00.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:00 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/827215399' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:07:00.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:00 smithi184 ceph-mon[92527]: osdmap e1364: 8 total, 8 up, 6 in 2024-09-13T15:07:00.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:00 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1422272862' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi151-350457-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-13T15:07:00.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:00 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2169974088' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:07:00.354 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:07:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:07:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:07:00.361 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:07:00.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:00.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:00.373 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.create (23 ms) 2024-09-13T15:07:00.373 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.get_features 2024-09-13T15:07:00.374 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.get_features (1 ms) 2024-09-13T15:07:00.374 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.get_object_prefix 2024-09-13T15:07:00.376 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.get_object_prefix (2 ms) 2024-09-13T15:07:00.376 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.get_create_timestamp 2024-09-13T15:07:00.377 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.get_create_timestamp (1 ms) 2024-09-13T15:07:00.377 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.get_access_timestamp 2024-09-13T15:07:00.378 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.get_access_timestamp (1 ms) 2024-09-13T15:07:00.379 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.get_modify_timestamp 2024-09-13T15:07:00.380 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.get_modify_timestamp (1 ms) 2024-09-13T15:07:00.380 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.get_data_pool 2024-09-13T15:07:00.383 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.get_data_pool (4 ms) 2024-09-13T15:07:00.383 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.get_size 2024-09-13T15:07:00.387 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.get_size (4 ms) 2024-09-13T15:07:00.387 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.set_size 2024-09-13T15:07:00.392 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.set_size (4 ms) 2024-09-13T15:07:00.392 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.protection_status 2024-09-13T15:07:00.398 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T15:07:00.409 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.protection_status (18 ms) 2024-09-13T15:07:00.410 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.snapshot_limits 2024-09-13T15:07:00.417 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.snapshot_limits (7 ms) 2024-09-13T15:07:00.417 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.parents_v1 2024-09-13T15:07:00.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:00.466 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.parents_v1 (50 ms) 2024-09-13T15:07:00.466 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.parents_v2 2024-09-13T15:07:00.485 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:07:00.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:00.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:00.495 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.parents_v2 (29 ms) 2024-09-13T15:07:00.496 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.snapshots 2024-09-13T15:07:00.517 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.snapshots (21 ms) 2024-09-13T15:07:00.517 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.snapid_race 2024-09-13T15:07:00.523 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.snapid_race (6 ms) 2024-09-13T15:07:00.523 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.stripingv2 2024-09-13T15:07:00.531 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.stripingv2 (9 ms) 2024-09-13T15:07:00.531 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.object_map_save 2024-09-13T15:07:00.533 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.object_map_save (2 ms) 2024-09-13T15:07:00.533 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.object_map_resize 2024-09-13T15:07:00.538 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.object_map_resize (5 ms) 2024-09-13T15:07:00.538 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.object_map_update 2024-09-13T15:07:00.547 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.object_map_update (8 ms) 2024-09-13T15:07:00.547 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.object_map_load_enoent 2024-09-13T15:07:00.547 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.object_map_load_enoent (1 ms) 2024-09-13T15:07:00.547 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.object_map_snap_add 2024-09-13T15:07:00.554 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.object_map_snap_add (7 ms) 2024-09-13T15:07:00.554 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.object_map_snap_remove 2024-09-13T15:07:00.560 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.object_map_snap_remove (5 ms) 2024-09-13T15:07:00.560 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.flags 2024-09-13T15:07:00.564 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.flags (5 ms) 2024-09-13T15:07:00.564 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.metadata 2024-09-13T15:07:00.588 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:00.588 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.metadata (23 ms) 2024-09-13T15:07:00.589 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.set_features 2024-09-13T15:07:00.596 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.set_features (9 ms) 2024-09-13T15:07:00.596 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.mirror 2024-09-13T15:07:00.611 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:07:00.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:00.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:00.628 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.mirror (32 ms) 2024-09-13T15:07:00.628 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.mirror_image 2024-09-13T15:07:00.643 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.mirror_image (15 ms) 2024-09-13T15:07:00.643 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.mirror_image_status 2024-09-13T15:07:00.657 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:07:00.713 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:00.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:00.736 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:07:00.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:00.837 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:00.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:00.860 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:07:00.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:00.938 INFO:tasks.thrashosds.thrasher:in_osds: [3, 6, 7, 1, 5, 0] out_osds: [4, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T15:07:00.938 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:07:00.938 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-13T15:07:00.938 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T15:07:00.938 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T15:07:00.939 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-13T15:07:00.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:01.016 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:07:01.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:01.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:01.121 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:01.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:01.146 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:07:01.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:01.247 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:01.271 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:07:01.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:01.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:01.334 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:01 smithi184 ceph-mon[92527]: pgmap v2397: 135 pgs: 10 creating+peering, 22 unknown, 103 active+clean; 582 KiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 46 KiB/s, 1 keys/s, 2 objects/s recovering 2024-09-13T15:07:01.334 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:01 smithi184 ceph-mon[92527]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T15:07:01.334 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:01 smithi184 ceph-mon[92527]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive) 2024-09-13T15:07:01.334 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:01 smithi184 ceph-mon[92527]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/78 objects degraded (3.846%), 2 pgs degraded) 2024-09-13T15:07:01.334 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:01 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1422272862' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi151-350457-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-13T15:07:01.334 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:01 smithi184 ceph-mon[92527]: osdmap e1365: 8 total, 8 up, 6 in 2024-09-13T15:07:01.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:01 smithi151 ceph-mon[104450]: pgmap v2397: 135 pgs: 10 creating+peering, 22 unknown, 103 active+clean; 582 KiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 46 KiB/s, 1 keys/s, 2 objects/s recovering 2024-09-13T15:07:01.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:01 smithi151 ceph-mon[104450]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T15:07:01.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:01 smithi151 ceph-mon[104450]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive) 2024-09-13T15:07:01.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:01 smithi151 ceph-mon[104450]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/78 objects degraded (3.846%), 2 pgs degraded) 2024-09-13T15:07:01.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:01 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1422272862' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi151-350457-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-13T15:07:01.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:01 smithi151 ceph-mon[104450]: osdmap e1365: 8 total, 8 up, 6 in 2024-09-13T15:07:01.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:01 smithi151 ceph-mon[108966]: pgmap v2397: 135 pgs: 10 creating+peering, 22 unknown, 103 active+clean; 582 KiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 46 KiB/s, 1 keys/s, 2 objects/s recovering 2024-09-13T15:07:01.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:01 smithi151 ceph-mon[108966]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T15:07:01.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:01 smithi151 ceph-mon[108966]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive) 2024-09-13T15:07:01.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:01 smithi151 ceph-mon[108966]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/78 objects degraded (3.846%), 2 pgs degraded) 2024-09-13T15:07:01.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:01 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1422272862' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi151-350457-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-13T15:07:01.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:01 smithi151 ceph-mon[108966]: osdmap e1365: 8 total, 8 up, 6 in 2024-09-13T15:07:01.373 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:01.398 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:07:01.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:01.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:01.500 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:01.522 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:07:01.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:01.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:01.624 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:01.632 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:07:01.647 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:07:01.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:01.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:01.750 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:01.774 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:07:01.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:01.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:01.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:01.900 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:07:01.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:01.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:02.002 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:02.023 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:07:02.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:02.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:02.125 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:02.149 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:07:02.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:02.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:02.177 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T15:07:02.250 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:02.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:02.272 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:07:02.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:02.317 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:02 smithi151 ceph-mon[104450]: osdmap e1366: 8 total, 8 up, 6 in 2024-09-13T15:07:02.317 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:07:02.317 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:02.317 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:02 smithi151 ceph-mon[108966]: osdmap e1366: 8 total, 8 up, 6 in 2024-09-13T15:07:02.317 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:07:02.318 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:02.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:02 smithi184 ceph-mon[92527]: osdmap e1366: 8 total, 8 up, 6 in 2024-09-13T15:07:02.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:07:02.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:02.377 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:02.403 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:07:02.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:02.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:02.505 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:02.526 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:07:02.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:02.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:02.628 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:02.651 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:07:02.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:02.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:02.752 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:02.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:02.775 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:07:02.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:02.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:02.902 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:07:02.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:02.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:02.929 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:02.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:07:03.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:03.032 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:07:03.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:03.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:03.102 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.mirror_image_status (2459 ms) 2024-09-13T15:07:03.102 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.mirror_image_map 2024-09-13T15:07:03.136 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:03.158 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:07:03.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:03.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:03.197 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.mirror_image_map (94 ms) 2024-09-13T15:07:03.197 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.mirror_instances 2024-09-13T15:07:03.204 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.mirror_instances (8 ms) 2024-09-13T15:07:03.204 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.mirror_snapshot 2024-09-13T15:07:03.223 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.mirror_snapshot (18 ms) 2024-09-13T15:07:03.223 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.group_dir_list 2024-09-13T15:07:03.225 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.group_dir_list (3 ms) 2024-09-13T15:07:03.225 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.group_dir_add 2024-09-13T15:07:03.227 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.group_dir_add (2 ms) 2024-09-13T15:07:03.227 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.dir_add_already_existing 2024-09-13T15:07:03.230 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.dir_add_already_existing (3 ms) 2024-09-13T15:07:03.230 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.group_dir_rename 2024-09-13T15:07:03.231 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:03 smithi151 ceph-mon[104450]: pgmap v2400: 135 pgs: 10 creating+peering, 22 unknown, 103 active+clean; 582 KiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 909 B/s, 1 keys/s, 2 objects/s recovering 2024-09-13T15:07:03.231 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:03 smithi151 ceph-mon[104450]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T15:07:03.231 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:03 smithi151 ceph-mon[104450]: Cluster is now healthy 2024-09-13T15:07:03.231 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:03 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:07:03.231 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:03 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:07:03.231 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:03 smithi151 ceph-mon[108966]: pgmap v2400: 135 pgs: 10 creating+peering, 22 unknown, 103 active+clean; 582 KiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 909 B/s, 1 keys/s, 2 objects/s recovering 2024-09-13T15:07:03.231 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:03 smithi151 ceph-mon[108966]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T15:07:03.231 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:03 smithi151 ceph-mon[108966]: Cluster is now healthy 2024-09-13T15:07:03.231 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:03 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:07:03.231 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:03 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:07:03.236 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.group_dir_rename (6 ms) 2024-09-13T15:07:03.236 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.group_dir_remove 2024-09-13T15:07:03.239 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.group_dir_remove (3 ms) 2024-09-13T15:07:03.239 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.group_dir_remove_missing 2024-09-13T15:07:03.243 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.group_dir_remove_missing (4 ms) 2024-09-13T15:07:03.243 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.group_image_add 2024-09-13T15:07:03.246 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.group_image_add (2 ms) 2024-09-13T15:07:03.246 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.group_image_remove 2024-09-13T15:07:03.249 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.group_image_remove (4 ms) 2024-09-13T15:07:03.249 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.group_image_list 2024-09-13T15:07:03.252 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.group_image_list (3 ms) 2024-09-13T15:07:03.252 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.group_image_clean 2024-09-13T15:07:03.257 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.group_image_clean (5 ms) 2024-09-13T15:07:03.257 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.image_group_add 2024-09-13T15:07:03.259 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:03.260 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.image_group_add (2 ms) 2024-09-13T15:07:03.260 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.image_group_remove 2024-09-13T15:07:03.264 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.image_group_remove (4 ms) 2024-09-13T15:07:03.264 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.image_group_get 2024-09-13T15:07:03.267 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.image_group_get (4 ms) 2024-09-13T15:07:03.267 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_name 2024-09-13T15:07:03.269 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.group_snap_set_empty_name (1 ms) 2024-09-13T15:07:03.269 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_id 2024-09-13T15:07:03.271 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.group_snap_set_empty_id (3 ms) 2024-09-13T15:07:03.271 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_id 2024-09-13T15:07:03.274 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_id (3 ms) 2024-09-13T15:07:03.274 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_name 2024-09-13T15:07:03.277 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_name (3 ms) 2024-09-13T15:07:03.277 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.group_snap_set 2024-09-13T15:07:03.280 INFO:tasks.workunit.client.0.smithi151.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/17.2.7-1627-gbbdadfa3/rpm/el9/BUILD/ceph-17.2.7-1627-gbbdadfa3/src/test/cls_rbd/test_cls_rbd.cc:2681: Failure 2024-09-13T15:07:03.280 INFO:tasks.workunit.client.0.smithi151.stdout:Expected equality of these values: 2024-09-13T15:07:03.280 INFO:tasks.workunit.client.0.smithi151.stdout: 1U 2024-09-13T15:07:03.280 INFO:tasks.workunit.client.0.smithi151.stdout: Which is: 1 2024-09-13T15:07:03.280 INFO:tasks.workunit.client.0.smithi151.stdout: keys.size() 2024-09-13T15:07:03.280 INFO:tasks.workunit.client.0.smithi151.stdout: Which is: 3 2024-09-13T15:07:03.280 INFO:tasks.workunit.client.0.smithi151.stdout:[ FAILED ] TestClsRbd.group_snap_set (2 ms) 2024-09-13T15:07:03.280 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.group_snap_list 2024-09-13T15:07:03.284 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.group_snap_list (4 ms) 2024-09-13T15:07:03.284 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.group_snap_list_max_return 2024-09-13T15:07:03.284 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:07:03.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:03.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:03.301 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.group_snap_list_max_return (17 ms) 2024-09-13T15:07:03.301 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.group_snap_list_max_read 2024-09-13T15:07:03.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:03 smithi184 ceph-mon[92527]: pgmap v2400: 135 pgs: 10 creating+peering, 22 unknown, 103 active+clean; 582 KiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 909 B/s, 1 keys/s, 2 objects/s recovering 2024-09-13T15:07:03.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:03 smithi184 ceph-mon[92527]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T15:07:03.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:03 smithi184 ceph-mon[92527]: Cluster is now healthy 2024-09-13T15:07:03.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:03 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:07:03.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:03 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:07:03.386 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:03.409 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:07:03.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:03.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:03.465 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T15:07:03.465 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:07:03.511 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:03.515 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.group_snap_list_max_read (215 ms) 2024-09-13T15:07:03.515 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.group_snap_remove 2024-09-13T15:07:03.519 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.group_snap_remove (4 ms) 2024-09-13T15:07:03.519 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.group_snap_get_by_id 2024-09-13T15:07:03.521 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.group_snap_get_by_id (2 ms) 2024-09-13T15:07:03.521 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.trash_methods 2024-09-13T15:07:03.528 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.trash_methods (7 ms) 2024-09-13T15:07:03.528 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.op_features 2024-09-13T15:07:03.534 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:07:03.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:03.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:03.536 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.op_features (7 ms) 2024-09-13T15:07:03.536 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.clone_parent 2024-09-13T15:07:03.553 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.clone_parent (18 ms) 2024-09-13T15:07:03.554 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.clone_parent_ns 2024-09-13T15:07:03.561 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.clone_parent_ns (7 ms) 2024-09-13T15:07:03.561 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.clone_child 2024-09-13T15:07:03.571 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.clone_child (11 ms) 2024-09-13T15:07:03.571 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.namespace_methods 2024-09-13T15:07:03.577 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.namespace_methods (6 ms) 2024-09-13T15:07:03.577 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.migration 2024-09-13T15:07:03.587 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.migration (9 ms) 2024-09-13T15:07:03.587 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.migration_v1 2024-09-13T15:07:03.597 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.migration_v1 (10 ms) 2024-09-13T15:07:03.597 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.assert_snapc_seq 2024-09-13T15:07:03.636 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:03.657 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:07:03.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:03.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:03.759 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:03.780 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:07:03.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:03.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:03.882 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:03.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:03.904 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:07:03.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:04.008 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:04.029 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:07:04.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:04.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:04.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:04 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:07:04.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:04 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:07:04.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:04 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:07:04.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:04 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:07:04.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:04 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:07:04.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:04 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:07:04.131 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:04.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:04.153 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:07:04.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:04.254 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:04.278 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:04.278 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:07:04.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:04.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:04.317 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:04 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:07:04.317 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:04 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:07:04.318 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:04 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:07:04.380 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:04.402 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:07:04.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:04.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:04.503 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:04.526 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:07:04.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:04.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:04.627 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:04.630 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:07:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:07:04] "GET /metrics HTTP/1.1" 200 36680 "" "Prometheus/2.51.0" 2024-09-13T15:07:04.649 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:07:04.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:04.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:04.669 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T15:07:04.669 INFO:teuthology.orchestra.run.smithi184.stdout: "success": "" 2024-09-13T15:07:04.669 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T15:07:04.750 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:04.772 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:07:04.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:04.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:04.874 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:04.897 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:07:04.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:04.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:04.998 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:05.022 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:07:05.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:05.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:05.123 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:05.145 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:07:05.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:05.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:05.247 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:05.269 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:07:05.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:05.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:05.363 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.assert_snapc_seq (1766 ms) 2024-09-13T15:07:05.363 INFO:tasks.workunit.client.0.smithi151.stdout:[ RUN ] TestClsRbd.sparsify 2024-09-13T15:07:05.363 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-13T15:07:05.371 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:05.393 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:07:05.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:05.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:05.414 INFO:tasks.workunit.client.0.smithi151.stdout:[ OK ] TestClsRbd.sparsify (52 ms) 2024-09-13T15:07:05.495 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:05.516 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:07:05.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:05.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:05.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:05 smithi184 ceph-mon[92527]: pgmap v2401: 135 pgs: 10 creating+peering, 8 unknown, 117 active+clean; 582 KiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 2.1 MiB/s rd, 1.2 MiB/s wr, 1.27k op/s; 94 B/s, 0 objects/s recovering 2024-09-13T15:07:05.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:05 smithi184 ceph-mon[92527]: osdmap e1367: 8 total, 8 up, 6 in 2024-09-13T15:07:05.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:05 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2297014002' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:07:05.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:05 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:07:05.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:05 smithi151 ceph-mon[104450]: pgmap v2401: 135 pgs: 10 creating+peering, 8 unknown, 117 active+clean; 582 KiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 2.1 MiB/s rd, 1.2 MiB/s wr, 1.27k op/s; 94 B/s, 0 objects/s recovering 2024-09-13T15:07:05.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:05 smithi151 ceph-mon[104450]: osdmap e1367: 8 total, 8 up, 6 in 2024-09-13T15:07:05.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:05 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2297014002' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:07:05.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:05 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:07:05.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:05 smithi151 ceph-mon[108966]: pgmap v2401: 135 pgs: 10 creating+peering, 8 unknown, 117 active+clean; 582 KiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 2.1 MiB/s rd, 1.2 MiB/s wr, 1.27k op/s; 94 B/s, 0 objects/s recovering 2024-09-13T15:07:05.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:05 smithi151 ceph-mon[108966]: osdmap e1367: 8 total, 8 up, 6 in 2024-09-13T15:07:05.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:05 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2297014002' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:07:05.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:05 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:07:05.618 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:05.636 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T15:07:05.640 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:07:05.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:05.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:05.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:05.764 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:07:05.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:05.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:05.865 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:05.885 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:07:05.888 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:07:05.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:05.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:05.990 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:06.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:06.013 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:07:06.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:06.114 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:06.135 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:07:06.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:06.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:06.237 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:06.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:06.261 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:07:06.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:06.362 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:06.375 INFO:tasks.workunit.client.0.smithi151.stdout:[----------] 70 tests from TestClsRbd (5363 ms total) 2024-09-13T15:07:06.375 INFO:tasks.workunit.client.0.smithi151.stdout: 2024-09-13T15:07:06.375 INFO:tasks.workunit.client.0.smithi151.stdout:[----------] Global test environment tear-down 2024-09-13T15:07:06.375 INFO:tasks.workunit.client.0.smithi151.stdout:[==========] 70 tests from 1 test suite ran. (8369 ms total) 2024-09-13T15:07:06.375 INFO:tasks.workunit.client.0.smithi151.stdout:[ PASSED ] 69 tests. 2024-09-13T15:07:06.376 INFO:tasks.workunit.client.0.smithi151.stdout:[ FAILED ] 1 test, listed below: 2024-09-13T15:07:06.376 INFO:tasks.workunit.client.0.smithi151.stdout:[ FAILED ] TestClsRbd.group_snap_set 2024-09-13T15:07:06.376 INFO:tasks.workunit.client.0.smithi151.stdout: 2024-09-13T15:07:06.376 INFO:tasks.workunit.client.0.smithi151.stdout: 1 FAILED TEST 2024-09-13T15:07:06.388 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:07:06.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:06.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:06.426 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:06 smithi151 ceph-mon[108966]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:07:06.426 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:06 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:07:06.426 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:06 smithi151 ceph-mon[108966]: osdmap e1368: 8 total, 8 up, 6 in 2024-09-13T15:07:06.426 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:06 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 5]} : dispatch 2024-09-13T15:07:06.426 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:06 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 0]} : dispatch 2024-09-13T15:07:06.426 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:06 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [3, 5]} : dispatch 2024-09-13T15:07:06.427 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:06 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1", "id": [3, 0]} : dispatch 2024-09-13T15:07:06.427 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:06 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.b", "id": [6, 0]} : dispatch 2024-09-13T15:07:06.427 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:06 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.e", "id": [7, 5]} : dispatch 2024-09-13T15:07:06.427 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:06 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1c", "id": [7, 1]} : dispatch 2024-09-13T15:07:06.427 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:06 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1d", "id": [7, 1]} : dispatch 2024-09-13T15:07:06.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:06 smithi151 ceph-mon[104450]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:07:06.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:06 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:07:06.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:06 smithi151 ceph-mon[104450]: osdmap e1368: 8 total, 8 up, 6 in 2024-09-13T15:07:06.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:06 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 5]} : dispatch 2024-09-13T15:07:06.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:06 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 0]} : dispatch 2024-09-13T15:07:06.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:06 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [3, 5]} : dispatch 2024-09-13T15:07:06.428 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:06 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1", "id": [3, 0]} : dispatch 2024-09-13T15:07:06.428 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:06 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.b", "id": [6, 0]} : dispatch 2024-09-13T15:07:06.428 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:06 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.e", "id": [7, 5]} : dispatch 2024-09-13T15:07:06.428 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:06 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1c", "id": [7, 1]} : dispatch 2024-09-13T15:07:06.428 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:06 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1d", "id": [7, 1]} : dispatch 2024-09-13T15:07:06.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:06.443 INFO:tasks.workunit:Stopping ['cls/test_cls_rbd.sh'] on client.0... 2024-09-13T15:07:06.443 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-13T15:07:06.489 DEBUG:teuthology.orchestra.run.smithi184:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:06.510 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: docker: command not found 2024-09-13T15:07:06.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:06.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi184 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:06.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:06 smithi184 ceph-mon[92527]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:07:06.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:06 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:07:06.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:06 smithi184 ceph-mon[92527]: osdmap e1368: 8 total, 8 up, 6 in 2024-09-13T15:07:06.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:06 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 5]} : dispatch 2024-09-13T15:07:06.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:06 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 0]} : dispatch 2024-09-13T15:07:06.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:06 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [3, 5]} : dispatch 2024-09-13T15:07:06.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:06 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1", "id": [3, 0]} : dispatch 2024-09-13T15:07:06.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:06 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.b", "id": [6, 0]} : dispatch 2024-09-13T15:07:06.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:06 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.e", "id": [7, 5]} : dispatch 2024-09-13T15:07:06.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:06 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1c", "id": [7, 1]} : dispatch 2024-09-13T15:07:06.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:06 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1d", "id": [7, 1]} : dispatch 2024-09-13T15:07:06.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:06.633 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-13T15:07:06.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:06.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:06.650 INFO:tasks.thrashosds:joining thrashosds 2024-09-13T15:07:07.557 INFO:teuthology.orchestra.run.smithi184.stdout:ERROR: (22) Invalid argument 2024-09-13T15:07:07.557 INFO:teuthology.orchestra.run.smithi184.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:07:07.565 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:07 smithi184 ceph-mon[92527]: pgmap v2404: 135 pgs: 135 active+clean; 582 KiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 2.4 MiB/s rd, 1.3 MiB/s wr, 1.59k op/s 2024-09-13T15:07:07.565 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:07 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 5]}]': finished 2024-09-13T15:07:07.565 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:07 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 0]}]': finished 2024-09-13T15:07:07.565 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:07 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [3, 5]}]': finished 2024-09-13T15:07:07.565 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:07 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1", "id": [3, 0]}]': finished 2024-09-13T15:07:07.565 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:07 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.b", "id": [6, 0]}]': finished 2024-09-13T15:07:07.565 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:07 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.e", "id": [7, 5]}]': finished 2024-09-13T15:07:07.565 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:07 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1c", "id": [7, 1]}]': finished 2024-09-13T15:07:07.565 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:07 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1d", "id": [7, 1]}]': finished 2024-09-13T15:07:07.565 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:07 smithi184 ceph-mon[92527]: osdmap e1369: 8 total, 8 up, 6 in 2024-09-13T15:07:07.643 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:07 smithi151 ceph-mon[104450]: pgmap v2404: 135 pgs: 135 active+clean; 582 KiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 2.4 MiB/s rd, 1.3 MiB/s wr, 1.59k op/s 2024-09-13T15:07:07.643 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:07 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 5]}]': finished 2024-09-13T15:07:07.643 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:07 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 0]}]': finished 2024-09-13T15:07:07.643 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:07 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [3, 5]}]': finished 2024-09-13T15:07:07.643 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:07 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1", "id": [3, 0]}]': finished 2024-09-13T15:07:07.643 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:07 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.b", "id": [6, 0]}]': finished 2024-09-13T15:07:07.643 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:07 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.e", "id": [7, 5]}]': finished 2024-09-13T15:07:07.643 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:07 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1c", "id": [7, 1]}]': finished 2024-09-13T15:07:07.643 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:07 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1d", "id": [7, 1]}]': finished 2024-09-13T15:07:07.643 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:07 smithi151 ceph-mon[104450]: osdmap e1369: 8 total, 8 up, 6 in 2024-09-13T15:07:07.644 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:07 smithi151 ceph-mon[108966]: pgmap v2404: 135 pgs: 135 active+clean; 582 KiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 2.4 MiB/s rd, 1.3 MiB/s wr, 1.59k op/s 2024-09-13T15:07:07.644 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:07 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 5]}]': finished 2024-09-13T15:07:07.644 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:07 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 0]}]': finished 2024-09-13T15:07:07.644 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:07 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [3, 5]}]': finished 2024-09-13T15:07:07.644 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:07 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1", "id": [3, 0]}]': finished 2024-09-13T15:07:07.644 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:07 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.b", "id": [6, 0]}]': finished 2024-09-13T15:07:07.644 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:07 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.e", "id": [7, 5]}]': finished 2024-09-13T15:07:07.644 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:07 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1c", "id": [7, 1]}]': finished 2024-09-13T15:07:07.644 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:07 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "12.1d", "id": [7, 1]}]': finished 2024-09-13T15:07:07.644 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:07 smithi151 ceph-mon[108966]: osdmap e1369: 8 total, 8 up, 6 in 2024-09-13T15:07:07.766 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-13T15:07:07.766 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-13T15:07:07.767 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:07:07.767 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:07:07.775 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-13T15:07:07.775 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:07:07.791 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-13T15:07:07.791 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:07:07.806 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-13T15:07:07.806 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:07:07.822 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-13T15:07:07.822 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:07:07.838 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-13T15:07:07.838 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:07:07.853 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-13T15:07:07.853 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:07:08.249 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T15:07:08.492 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:07:08.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:08 smithi184 ceph-mon[92527]: osdmap e1370: 8 total, 8 up, 6 in 2024-09-13T15:07:08.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:08 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2482070900' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:07:08.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:08 smithi151 ceph-mon[104450]: osdmap e1370: 8 total, 8 up, 6 in 2024-09-13T15:07:08.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:08 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2482070900' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:07:08.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:08 smithi151 ceph-mon[108966]: osdmap e1370: 8 total, 8 up, 6 in 2024-09-13T15:07:08.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:08 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2482070900' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:07:09.573 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:09 smithi184 ceph-mon[92527]: pgmap v2407: 103 pgs: 103 active+clean; 582 KiB data, 7.2 GiB used, 529 GiB / 536 GiB avail 2024-09-13T15:07:09.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:09 smithi151 ceph-mon[104450]: pgmap v2407: 103 pgs: 103 active+clean; 582 KiB data, 7.2 GiB used, 529 GiB / 536 GiB avail 2024-09-13T15:07:09.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:09 smithi151 ceph-mon[108966]: pgmap v2407: 103 pgs: 103 active+clean; 582 KiB data, 7.2 GiB used, 529 GiB / 536 GiB avail 2024-09-13T15:07:10.076 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T15:07:10.076 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [], 2024-09-13T15:07:10.076 INFO:teuthology.orchestra.run.smithi184.stdout: "complaint_time": 30, 2024-09-13T15:07:10.076 INFO:teuthology.orchestra.run.smithi184.stdout: "num_blocked_ops": 0 2024-09-13T15:07:10.077 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T15:07:10.240 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-13T15:07:10.240 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-09-13T15:07:10.240 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd in 4 2024-09-13T15:07:10.499 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:10.525 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:07:10.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:07:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:07:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:07:10.730 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T15:07:10.976 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:07:11.584 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:11.704 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:11 smithi151 ceph-mon[104450]: pgmap v2408: 103 pgs: 103 active+clean; 582 KiB data, 7.2 GiB used, 529 GiB / 536 GiB avail 2024-09-13T15:07:11.705 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:11 smithi151 ceph-mon[108966]: pgmap v2408: 103 pgs: 103 active+clean; 582 KiB data, 7.2 GiB used, 529 GiB / 536 GiB avail 2024-09-13T15:07:11.770 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:11 smithi184 ceph-mon[92527]: pgmap v2408: 103 pgs: 103 active+clean; 582 KiB data, 7.2 GiB used, 529 GiB / 536 GiB avail 2024-09-13T15:07:12.616 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T15:07:12.617 INFO:teuthology.orchestra.run.smithi184.stdout: "size": 20, 2024-09-13T15:07:12.617 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 600, 2024-09-13T15:07:12.617 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [ 2024-09-13T15:07:12.617 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.617 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197603 9.6 9:67850b99:::benchmark_data_smithi151_261954_object94747:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T15:07:12.617 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.242955+0000", 2024-09-13T15:07:12.617 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 581.36983995900005, 2024-09-13T15:07:12.617 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1339466739999999, 2024-09-13T15:07:12.617 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:12.617 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "reached pg", 2024-09-13T15:07:12.617 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:12.617 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T15:07:12.617 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:07:12.618 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197603 2024-09-13T15:07:12.618 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.618 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:12.618 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.618 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:12.618 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.242955+0000", 2024-09-13T15:07:12.618 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.618 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.618 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.618 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:12.618 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.242955+0000", 2024-09-13T15:07:12.618 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.618 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.619 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.619 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:12.619 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.242953+0000", 2024-09-13T15:07:12.619 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:07:12.619 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.619 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.619 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:12.619 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.242966+0000", 2024-09-13T15:07:12.619 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2500999999999999e-05 2024-09-13T15:07:12.619 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.619 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.619 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:12.619 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.242979+0000", 2024-09-13T15:07:12.620 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3421000000000001e-05 2024-09-13T15:07:12.620 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.620 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.620 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:12.620 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.242985+0000", 2024-09-13T15:07:12.620 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5960000000000003e-06 2024-09-13T15:07:12.620 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.620 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.620 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.620 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.245536+0000", 2024-09-13T15:07:12.620 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002551598 2024-09-13T15:07:12.620 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.620 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.621 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op must wait for map", 2024-09-13T15:07:12.621 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.245541+0000", 2024-09-13T15:07:12.621 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3510000000000004e-06 2024-09-13T15:07:12.621 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.621 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.621 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.621 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.246376+0000", 2024-09-13T15:07:12.621 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00083585899999999999 2024-09-13T15:07:12.621 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.621 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.621 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:12.621 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.246385+0000", 2024-09-13T15:07:12.621 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.8689999999999993e-06 2024-09-13T15:07:12.622 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.622 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.622 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.622 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.303041+0000", 2024-09-13T15:07:12.622 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0566552010000001 2024-09-13T15:07:12.622 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.622 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.622 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for missing object", 2024-09-13T15:07:12.622 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.303149+0000", 2024-09-13T15:07:12.622 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00010892599999999999 2024-09-13T15:07:12.622 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.622 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.622 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.622 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.376838+0000", 2024-09-13T15:07:12.623 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.073688742000000002 2024-09-13T15:07:12.623 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.623 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.623 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:12.623 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.376901+0000", 2024-09-13T15:07:12.623 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3144000000000004e-05 2024-09-13T15:07:12.623 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.623 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:12.623 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.623 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.623 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.623 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197608 9.6 9:664669c4:::benchmark_data_smithi151_261954_object94752:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T15:07:12.623 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.248591+0000", 2024-09-13T15:07:12.623 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 581.36420326799998, 2024-09-13T15:07:12.624 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1402739749999999, 2024-09-13T15:07:12.624 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:12.624 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:12.624 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:12.624 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T15:07:12.624 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:07:12.624 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197608 2024-09-13T15:07:12.624 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.624 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:12.624 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.624 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:12.624 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.248591+0000", 2024-09-13T15:07:12.624 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.624 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.625 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.625 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:12.625 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.248591+0000", 2024-09-13T15:07:12.625 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.625 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.625 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.625 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:12.625 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.248590+0000", 2024-09-13T15:07:12.625 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:07:12.625 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.625 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.625 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:12.625 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.248597+0000", 2024-09-13T15:07:12.625 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.4560000000000003e-06 2024-09-13T15:07:12.626 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.626 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.626 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:12.626 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.248610+0000", 2024-09-13T15:07:12.626 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.276e-05 2024-09-13T15:07:12.626 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.626 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.626 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:12.626 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.248615+0000", 2024-09-13T15:07:12.626 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7820000000000001e-06 2024-09-13T15:07:12.626 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.626 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.626 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.626 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.248958+0000", 2024-09-13T15:07:12.627 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00034373999999999999 2024-09-13T15:07:12.627 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.627 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.627 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:12.627 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.248985+0000", 2024-09-13T15:07:12.627 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.6902000000000001e-05 2024-09-13T15:07:12.627 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.627 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.627 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.627 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.303170+0000", 2024-09-13T15:07:12.627 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0541845750000001 2024-09-13T15:07:12.627 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.627 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.627 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:12.628 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.303270+0000", 2024-09-13T15:07:12.628 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00010008700000000001 2024-09-13T15:07:12.628 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.628 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.628 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:07:12.628 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.303848+0000", 2024-09-13T15:07:12.628 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00057828600000000001 2024-09-13T15:07:12.628 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.628 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.628 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:12.628 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.325700+0000", 2024-09-13T15:07:12.628 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.021852136000000001 2024-09-13T15:07:12.628 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.628 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.629 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.629 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.384239+0000", 2024-09-13T15:07:12.629 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.058538245000000003 2024-09-13T15:07:12.629 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.629 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.629 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.629 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.388686+0000", 2024-09-13T15:07:12.629 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0044472849999999996 2024-09-13T15:07:12.629 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.629 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.629 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:12.629 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.388740+0000", 2024-09-13T15:07:12.629 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3934999999999998e-05 2024-09-13T15:07:12.630 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.630 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.630 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:12.630 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.388865+0000", 2024-09-13T15:07:12.630 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012541200000000001 2024-09-13T15:07:12.630 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.630 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:12.630 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.630 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.630 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.630 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197611 9.6 9:6ed0f261:::benchmark_data_smithi151_261954_object94755:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T15:07:12.630 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.250949+0000", 2024-09-13T15:07:12.630 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 581.36184569299996, 2024-09-13T15:07:12.631 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1381295499999999, 2024-09-13T15:07:12.631 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:12.631 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:12.631 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:12.631 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T15:07:12.631 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:07:12.631 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197611 2024-09-13T15:07:12.631 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.631 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:12.631 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.631 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:12.631 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.250949+0000", 2024-09-13T15:07:12.632 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.632 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.632 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.632 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:12.632 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.250949+0000", 2024-09-13T15:07:12.632 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.632 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.632 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.632 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:12.632 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.250948+0000", 2024-09-13T15:07:12.632 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:07:12.632 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.632 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.633 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:12.633 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.250953+0000", 2024-09-13T15:07:12.633 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8359999999999997e-06 2024-09-13T15:07:12.633 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.633 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.633 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:12.633 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.250967+0000", 2024-09-13T15:07:12.633 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3585e-05 2024-09-13T15:07:12.633 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.633 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.633 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:12.633 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.250973+0000", 2024-09-13T15:07:12.633 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8000000000000004e-06 2024-09-13T15:07:12.634 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.634 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.634 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.634 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.253272+0000", 2024-09-13T15:07:12.634 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002299125 2024-09-13T15:07:12.634 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.634 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.634 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:12.634 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.253280+0000", 2024-09-13T15:07:12.634 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.8630000000000008e-06 2024-09-13T15:07:12.634 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.634 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.634 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.634 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.316907+0000", 2024-09-13T15:07:12.634 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.063627122 2024-09-13T15:07:12.635 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.635 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.635 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:12.635 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.316994+0000", 2024-09-13T15:07:12.635 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.7138000000000005e-05 2024-09-13T15:07:12.635 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.635 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.635 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:07:12.635 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.317078+0000", 2024-09-13T15:07:12.635 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.3681000000000001e-05 2024-09-13T15:07:12.635 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.635 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.635 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:12.635 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.325728+0000", 2024-09-13T15:07:12.636 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0086505990000000001 2024-09-13T15:07:12.636 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.636 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.636 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.636 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.384264+0000", 2024-09-13T15:07:12.636 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.058535255000000001 2024-09-13T15:07:12.636 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.636 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.636 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.636 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.388929+0000", 2024-09-13T15:07:12.636 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0046659789999999998 2024-09-13T15:07:12.636 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.637 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.637 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:12.637 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.388978+0000", 2024-09-13T15:07:12.637 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8105000000000001e-05 2024-09-13T15:07:12.637 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.637 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.637 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:12.637 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.389079+0000", 2024-09-13T15:07:12.637 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000100908 2024-09-13T15:07:12.637 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.637 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:12.637 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.637 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.637 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.638 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197615 9.6 9:6509112f:::benchmark_data_smithi151_261954_object94759:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T15:07:12.638 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.252664+0000", 2024-09-13T15:07:12.638 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 581.36013065899999, 2024-09-13T15:07:12.638 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1437066060000001, 2024-09-13T15:07:12.638 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:12.638 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:12.638 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:12.638 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T15:07:12.638 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:07:12.638 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197615 2024-09-13T15:07:12.638 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.638 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:12.638 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.638 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:12.639 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.252664+0000", 2024-09-13T15:07:12.639 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.639 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.639 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.639 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:12.639 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.252664+0000", 2024-09-13T15:07:12.639 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.639 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.639 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.639 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:12.639 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.252663+0000", 2024-09-13T15:07:12.639 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:07:12.639 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.640 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.640 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:12.640 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.252672+0000", 2024-09-13T15:07:12.640 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.8130000000000002e-06 2024-09-13T15:07:12.640 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.640 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.640 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:12.640 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.252686+0000", 2024-09-13T15:07:12.640 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4126e-05 2024-09-13T15:07:12.640 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.640 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.640 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:12.640 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.252692+0000", 2024-09-13T15:07:12.641 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.4620000000000001e-06 2024-09-13T15:07:12.641 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.641 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.641 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.641 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.253301+0000", 2024-09-13T15:07:12.641 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00060926600000000002 2024-09-13T15:07:12.641 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.641 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.641 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:12.641 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.253308+0000", 2024-09-13T15:07:12.641 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3550000000000001e-06 2024-09-13T15:07:12.641 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.641 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.641 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.642 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.319292+0000", 2024-09-13T15:07:12.642 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.065983992 2024-09-13T15:07:12.642 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.642 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.642 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:12.642 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.319387+0000", 2024-09-13T15:07:12.642 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.4790000000000006e-05 2024-09-13T15:07:12.642 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.642 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.642 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:07:12.642 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.319457+0000", 2024-09-13T15:07:12.642 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.0302999999999994e-05 2024-09-13T15:07:12.642 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.642 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.643 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:12.643 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.329056+0000", 2024-09-13T15:07:12.643 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0095986660000000005 2024-09-13T15:07:12.643 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.643 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.643 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.643 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.390210+0000", 2024-09-13T15:07:12.643 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.061154497000000002 2024-09-13T15:07:12.643 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.643 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.643 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.643 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.396285+0000", 2024-09-13T15:07:12.643 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0060744579999999996 2024-09-13T15:07:12.644 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.644 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.644 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:12.644 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.396330+0000", 2024-09-13T15:07:12.644 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5677999999999999e-05 2024-09-13T15:07:12.644 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.644 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.644 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:12.644 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.396371+0000", 2024-09-13T15:07:12.644 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0364999999999997e-05 2024-09-13T15:07:12.644 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.644 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:12.644 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.644 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.645 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.646 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197628 9.6 9:6d4e4d04:::benchmark_data_smithi151_261954_object94772:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T15:07:12.646 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.268145+0000", 2024-09-13T15:07:12.646 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 581.34465013500005, 2024-09-13T15:07:12.646 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.128359632, 2024-09-13T15:07:12.646 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:12.646 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:12.646 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:12.646 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T15:07:12.646 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:07:12.646 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197628 2024-09-13T15:07:12.646 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.646 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:12.647 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.647 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:12.647 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.268145+0000", 2024-09-13T15:07:12.647 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.647 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.647 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.647 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:12.647 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.268145+0000", 2024-09-13T15:07:12.647 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.647 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.647 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.647 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:12.647 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.268143+0000", 2024-09-13T15:07:12.648 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T15:07:12.648 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.648 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.648 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:12.648 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.268150+0000", 2024-09-13T15:07:12.648 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.2069999999999998e-06 2024-09-13T15:07:12.648 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.648 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.648 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:12.648 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.268166+0000", 2024-09-13T15:07:12.648 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.5611999999999999e-05 2024-09-13T15:07:12.648 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.648 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.648 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:12.649 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.268176+0000", 2024-09-13T15:07:12.649 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0108999999999999e-05 2024-09-13T15:07:12.649 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.649 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.649 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.649 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.274003+0000", 2024-09-13T15:07:12.649 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0058268690000000001 2024-09-13T15:07:12.649 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.649 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.649 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:12.649 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.274013+0000", 2024-09-13T15:07:12.649 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0909e-05 2024-09-13T15:07:12.649 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.650 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.650 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.650 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.321130+0000", 2024-09-13T15:07:12.650 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.047116344 2024-09-13T15:07:12.650 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.650 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.650 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:12.650 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.321198+0000", 2024-09-13T15:07:12.650 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7824000000000007e-05 2024-09-13T15:07:12.650 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.650 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.650 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:07:12.650 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.321265+0000", 2024-09-13T15:07:12.650 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7514000000000004e-05 2024-09-13T15:07:12.651 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.651 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.651 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:12.651 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.329063+0000", 2024-09-13T15:07:12.651 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0077980139999999998 2024-09-13T15:07:12.651 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.651 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.651 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.651 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.390255+0000", 2024-09-13T15:07:12.651 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.061192150000000001 2024-09-13T15:07:12.651 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.651 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.651 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.652 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.396426+0000", 2024-09-13T15:07:12.652 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0061704300000000002 2024-09-13T15:07:12.652 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.652 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.652 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:12.652 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.396465+0000", 2024-09-13T15:07:12.652 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9487000000000003e-05 2024-09-13T15:07:12.652 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.652 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.652 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:12.652 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.396504+0000", 2024-09-13T15:07:12.652 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8927999999999997e-05 2024-09-13T15:07:12.652 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.652 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:12.653 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.653 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.653 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.653 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197632 9.6 9:650414d3:::benchmark_data_smithi151_261954_object94776:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T15:07:12.653 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.269941+0000", 2024-09-13T15:07:12.653 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 581.34285339300004, 2024-09-13T15:07:12.653 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.130399621, 2024-09-13T15:07:12.653 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:12.653 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:12.653 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:12.653 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T15:07:12.653 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:07:12.653 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197632 2024-09-13T15:07:12.654 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.654 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:12.654 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.654 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:12.654 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.269941+0000", 2024-09-13T15:07:12.654 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.654 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.654 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.654 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:12.654 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.269941+0000", 2024-09-13T15:07:12.654 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.654 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.654 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.655 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:12.655 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.269940+0000", 2024-09-13T15:07:12.655 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:07:12.655 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.655 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.655 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:12.655 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.269948+0000", 2024-09-13T15:07:12.655 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.6009999999999997e-06 2024-09-13T15:07:12.655 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.655 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.655 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:12.655 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.269961+0000", 2024-09-13T15:07:12.655 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3560000000000001e-05 2024-09-13T15:07:12.655 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.656 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.656 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:12.656 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.269967+0000", 2024-09-13T15:07:12.656 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8109999999999999e-06 2024-09-13T15:07:12.656 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.656 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.656 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.656 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.274093+0000", 2024-09-13T15:07:12.656 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0041263710000000002 2024-09-13T15:07:12.656 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.656 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.656 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:12.656 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.274101+0000", 2024-09-13T15:07:12.656 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.5009999999999998e-06 2024-09-13T15:07:12.657 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.657 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.657 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.657 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.322701+0000", 2024-09-13T15:07:12.657 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0486003740000001 2024-09-13T15:07:12.657 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.657 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.657 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:12.657 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.322765+0000", 2024-09-13T15:07:12.657 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3381999999999997e-05 2024-09-13T15:07:12.657 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.657 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.657 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:07:12.657 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.322834+0000", 2024-09-13T15:07:12.658 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.9497000000000002e-05 2024-09-13T15:07:12.658 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.658 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.658 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:12.658 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.329070+0000", 2024-09-13T15:07:12.658 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0062355609999999997 2024-09-13T15:07:12.658 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.658 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.658 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.658 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.397613+0000", 2024-09-13T15:07:12.658 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.068543525999999994 2024-09-13T15:07:12.658 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.658 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.658 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.659 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.400251+0000", 2024-09-13T15:07:12.659 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002637805 2024-09-13T15:07:12.659 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.659 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.659 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:12.659 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.400302+0000", 2024-09-13T15:07:12.659 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0559999999999997e-05 2024-09-13T15:07:12.659 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.659 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.659 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:12.659 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.400341+0000", 2024-09-13T15:07:12.659 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9203999999999999e-05 2024-09-13T15:07:12.659 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.659 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:12.660 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.660 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.660 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.660 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197636 9.6 9:623919ba:::benchmark_data_smithi151_261954_object94780:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T15:07:12.660 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.275235+0000", 2024-09-13T15:07:12.660 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 581.33755921700003, 2024-09-13T15:07:12.660 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.126014793, 2024-09-13T15:07:12.660 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:12.660 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:12.660 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:12.660 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T15:07:12.660 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:07:12.661 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197636 2024-09-13T15:07:12.661 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.661 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:12.661 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.661 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:12.661 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.275235+0000", 2024-09-13T15:07:12.661 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.661 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.661 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.661 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:12.661 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.275235+0000", 2024-09-13T15:07:12.661 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.661 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.661 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.662 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:12.662 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.275234+0000", 2024-09-13T15:07:12.662 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T15:07:12.662 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.662 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.662 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:12.662 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.275239+0000", 2024-09-13T15:07:12.662 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1749999999999996e-06 2024-09-13T15:07:12.662 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.662 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.662 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:12.662 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.275248+0000", 2024-09-13T15:07:12.662 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.662e-06 2024-09-13T15:07:12.662 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.663 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.663 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:12.663 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.275253+0000", 2024-09-13T15:07:12.663 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9910000000000002e-06 2024-09-13T15:07:12.663 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.663 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.663 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.663 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.278310+0000", 2024-09-13T15:07:12.663 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0030561 2024-09-13T15:07:12.663 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.663 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.663 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:12.663 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.278320+0000", 2024-09-13T15:07:12.663 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.042e-05 2024-09-13T15:07:12.664 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.664 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.664 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.664 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.324240+0000", 2024-09-13T15:07:12.664 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.045920113 2024-09-13T15:07:12.664 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.664 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.664 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:12.664 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.324321+0000", 2024-09-13T15:07:12.664 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.1411999999999999e-05 2024-09-13T15:07:12.664 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.664 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.664 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:07:12.665 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.324396+0000", 2024-09-13T15:07:12.665 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.4723999999999998e-05 2024-09-13T15:07:12.665 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.665 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.665 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:12.665 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.333518+0000", 2024-09-13T15:07:12.665 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0091214390000000003 2024-09-13T15:07:12.665 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.665 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.665 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.665 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.400387+0000", 2024-09-13T15:07:12.665 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.066869291999999997 2024-09-13T15:07:12.665 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.665 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.666 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.666 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.401174+0000", 2024-09-13T15:07:12.666 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00078730300000000005 2024-09-13T15:07:12.666 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.666 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.666 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:12.666 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.401217+0000", 2024-09-13T15:07:12.666 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.2880999999999998e-05 2024-09-13T15:07:12.666 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.666 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.666 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:12.666 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.401250+0000", 2024-09-13T15:07:12.666 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3105999999999998e-05 2024-09-13T15:07:12.666 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.667 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:12.667 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.667 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.667 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.667 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197641 9.6 9:6b2553dd:::benchmark_data_smithi151_261954_object94785:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T15:07:12.667 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.278888+0000", 2024-09-13T15:07:12.667 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 581.33390626200003, 2024-09-13T15:07:12.667 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1274783509999999, 2024-09-13T15:07:12.667 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:12.667 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:12.667 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:12.667 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T15:07:12.667 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:07:12.667 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197641 2024-09-13T15:07:12.668 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.668 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:12.668 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.668 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:12.668 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.278888+0000", 2024-09-13T15:07:12.668 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.668 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.668 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.668 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:12.668 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.278888+0000", 2024-09-13T15:07:12.668 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.668 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.668 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.668 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:12.669 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.278887+0000", 2024-09-13T15:07:12.669 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:07:12.669 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.669 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.669 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:12.669 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.278896+0000", 2024-09-13T15:07:12.669 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.3260000000000002e-06 2024-09-13T15:07:12.669 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.669 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.669 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:12.669 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.278910+0000", 2024-09-13T15:07:12.669 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4234e-05 2024-09-13T15:07:12.669 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.669 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.670 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:12.670 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.278916+0000", 2024-09-13T15:07:12.670 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5160000000000002e-06 2024-09-13T15:07:12.670 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.670 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.670 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.670 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.278966+0000", 2024-09-13T15:07:12.670 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9969999999999998e-05 2024-09-13T15:07:12.670 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.670 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.670 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:12.670 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.278976+0000", 2024-09-13T15:07:12.670 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.7880000000000001e-06 2024-09-13T15:07:12.670 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.671 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.671 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.671 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.325763+0000", 2024-09-13T15:07:12.671 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0467872410000001 2024-09-13T15:07:12.671 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.671 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.671 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:12.671 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.325830+0000", 2024-09-13T15:07:12.671 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.6948999999999999e-05 2024-09-13T15:07:12.671 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.671 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.671 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:07:12.671 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.325905+0000", 2024-09-13T15:07:12.671 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.5026999999999998e-05 2024-09-13T15:07:12.671 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.672 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.672 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:12.672 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.333522+0000", 2024-09-13T15:07:12.672 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0076173600000000001 2024-09-13T15:07:12.672 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.672 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.672 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.672 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.401291+0000", 2024-09-13T15:07:12.672 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.067768703999999999 2024-09-13T15:07:12.672 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.672 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.672 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.672 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.406235+0000", 2024-09-13T15:07:12.673 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0049436469999999998 2024-09-13T15:07:12.673 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.673 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.673 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:12.673 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.406316+0000", 2024-09-13T15:07:12.673 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.1417999999999999e-05 2024-09-13T15:07:12.673 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.673 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.673 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:12.673 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.406367+0000", 2024-09-13T15:07:12.673 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0739000000000003e-05 2024-09-13T15:07:12.673 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.673 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:12.673 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.674 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.674 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.674 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197648 9.6 9:680e8575:::benchmark_data_smithi151_261954_object94792:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T15:07:12.674 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.286808+0000", 2024-09-13T15:07:12.674 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 581.325986597, 2024-09-13T15:07:12.674 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.11968004, 2024-09-13T15:07:12.674 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:12.674 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:12.674 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:12.674 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T15:07:12.674 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:07:12.674 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197648 2024-09-13T15:07:12.674 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.674 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:12.675 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.675 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:12.675 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.286808+0000", 2024-09-13T15:07:12.675 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.675 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.675 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.675 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:12.675 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.286808+0000", 2024-09-13T15:07:12.675 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.675 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.675 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.675 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:12.675 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.286806+0000", 2024-09-13T15:07:12.676 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T15:07:12.676 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.676 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.676 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:12.676 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.286813+0000", 2024-09-13T15:07:12.676 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.3490000000000003e-06 2024-09-13T15:07:12.676 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.676 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.676 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:12.676 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.286826+0000", 2024-09-13T15:07:12.676 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2367e-05 2024-09-13T15:07:12.676 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.676 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.676 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:12.677 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.286832+0000", 2024-09-13T15:07:12.677 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8640000000000001e-06 2024-09-13T15:07:12.677 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.677 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.677 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.677 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.287248+0000", 2024-09-13T15:07:12.677 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00041674399999999998 2024-09-13T15:07:12.677 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.677 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.677 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:12.677 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.287312+0000", 2024-09-13T15:07:12.677 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3938999999999999e-05 2024-09-13T15:07:12.677 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.678 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.678 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.678 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.327320+0000", 2024-09-13T15:07:12.678 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0400074850000001 2024-09-13T15:07:12.678 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.678 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.678 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:12.678 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.327407+0000", 2024-09-13T15:07:12.678 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.7183999999999997e-05 2024-09-13T15:07:12.678 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.678 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.678 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:07:12.678 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.327486+0000", 2024-09-13T15:07:12.678 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.9269000000000003e-05 2024-09-13T15:07:12.679 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.679 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.679 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:12.679 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.335020+0000", 2024-09-13T15:07:12.679 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.007533891 2024-09-13T15:07:12.679 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.679 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.679 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.679 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.405114+0000", 2024-09-13T15:07:12.679 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.070093875 2024-09-13T15:07:12.679 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.679 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.679 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.679 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.406409+0000", 2024-09-13T15:07:12.680 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0012952319999999999 2024-09-13T15:07:12.680 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.680 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.680 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:12.680 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.406454+0000", 2024-09-13T15:07:12.680 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4342999999999999e-05 2024-09-13T15:07:12.680 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.680 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.680 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:12.680 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.406488+0000", 2024-09-13T15:07:12.680 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.4419999999999999e-05 2024-09-13T15:07:12.680 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.680 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:12.681 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.681 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.681 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.681 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197655 9.6 9:6ad42e07:::benchmark_data_smithi151_261954_object94799:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T15:07:12.681 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.294574+0000", 2024-09-13T15:07:12.681 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 581.31822113500004, 2024-09-13T15:07:12.681 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1215945810000001, 2024-09-13T15:07:12.681 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:12.681 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:12.681 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:12.681 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T15:07:12.681 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:07:12.681 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197655 2024-09-13T15:07:12.682 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.682 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:12.682 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.682 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:12.682 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.294574+0000", 2024-09-13T15:07:12.682 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.682 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.682 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.682 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:12.682 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.294574+0000", 2024-09-13T15:07:12.682 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.682 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.682 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.682 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:12.682 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.294572+0000", 2024-09-13T15:07:12.683 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:07:12.683 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.683 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.683 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:12.683 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.294586+0000", 2024-09-13T15:07:12.683 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.402e-05 2024-09-13T15:07:12.683 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.683 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.683 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:12.683 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.294597+0000", 2024-09-13T15:07:12.683 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1353e-05 2024-09-13T15:07:12.683 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.683 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.684 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:12.684 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.294603+0000", 2024-09-13T15:07:12.684 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4110000000000004e-06 2024-09-13T15:07:12.684 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.684 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.684 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.684 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.294671+0000", 2024-09-13T15:07:12.684 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.8596000000000005e-05 2024-09-13T15:07:12.684 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.684 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.684 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:12.684 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.294690+0000", 2024-09-13T15:07:12.684 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.8448e-05 2024-09-13T15:07:12.684 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.685 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.685 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.685 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.329097+0000", 2024-09-13T15:07:12.685 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.034406905 2024-09-13T15:07:12.685 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.685 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.685 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:12.685 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.329164+0000", 2024-09-13T15:07:12.685 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.6860999999999996e-05 2024-09-13T15:07:12.685 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.685 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.685 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:07:12.685 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.329229+0000", 2024-09-13T15:07:12.686 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.5568000000000006e-05 2024-09-13T15:07:12.686 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.686 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.686 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:12.686 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.335029+0000", 2024-09-13T15:07:12.686 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0057995770000000002 2024-09-13T15:07:12.686 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.686 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.686 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.686 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.405257+0000", 2024-09-13T15:07:12.686 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.070228192999999994 2024-09-13T15:07:12.686 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.686 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.687 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.687 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.416082+0000", 2024-09-13T15:07:12.687 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.010825128999999999 2024-09-13T15:07:12.687 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.687 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.687 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:12.687 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.416130+0000", 2024-09-13T15:07:12.687 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8223000000000002e-05 2024-09-13T15:07:12.687 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.687 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.688 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:12.688 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.416168+0000", 2024-09-13T15:07:12.688 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.7808999999999997e-05 2024-09-13T15:07:12.688 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.688 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:12.688 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.688 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.688 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.688 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197661 9.6 9:6ea81a69:::benchmark_data_smithi151_261954_object94805:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T15:07:12.688 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.305951+0000", 2024-09-13T15:07:12.688 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 581.30684373899999, 2024-09-13T15:07:12.688 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.114294699, 2024-09-13T15:07:12.688 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:12.689 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:12.689 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:12.689 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T15:07:12.689 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:07:12.689 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197661 2024-09-13T15:07:12.689 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.689 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:12.689 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.689 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:12.689 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.305951+0000", 2024-09-13T15:07:12.689 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.689 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.689 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.690 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:12.690 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.305951+0000", 2024-09-13T15:07:12.690 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.690 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.690 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.690 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:12.690 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.305949+0000", 2024-09-13T15:07:12.690 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T15:07:12.690 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.690 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.690 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:12.690 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.305957+0000", 2024-09-13T15:07:12.690 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.994e-06 2024-09-13T15:07:12.691 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.691 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.691 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:12.691 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.305976+0000", 2024-09-13T15:07:12.691 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.8349999999999999e-05 2024-09-13T15:07:12.691 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.691 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.691 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:12.691 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.305982+0000", 2024-09-13T15:07:12.691 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.336e-06 2024-09-13T15:07:12.691 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.691 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.691 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.691 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.321123+0000", 2024-09-13T15:07:12.692 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.015140614 2024-09-13T15:07:12.692 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.692 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.692 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:12.692 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.321132+0000", 2024-09-13T15:07:12.692 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.533e-06 2024-09-13T15:07:12.692 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.692 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.692 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.692 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.330506+0000", 2024-09-13T15:07:12.692 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0093742370000001 2024-09-13T15:07:12.692 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.692 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.692 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:12.693 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.330565+0000", 2024-09-13T15:07:12.693 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8239000000000003e-05 2024-09-13T15:07:12.693 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.693 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.693 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:07:12.693 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.330629+0000", 2024-09-13T15:07:12.693 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.4053999999999994e-05 2024-09-13T15:07:12.693 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.693 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.693 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:12.693 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.335035+0000", 2024-09-13T15:07:12.693 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0044066849999999996 2024-09-13T15:07:12.693 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.693 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.694 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.694 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.415527+0000", 2024-09-13T15:07:12.694 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.080491317000000007 2024-09-13T15:07:12.694 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.694 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.694 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.694 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.420160+0000", 2024-09-13T15:07:12.694 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0046338359999999997 2024-09-13T15:07:12.694 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.694 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.694 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:12.694 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.420204+0000", 2024-09-13T15:07:12.694 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3850000000000002e-05 2024-09-13T15:07:12.695 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.695 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.695 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:12.695 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.420246+0000", 2024-09-13T15:07:12.695 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1322999999999997e-05 2024-09-13T15:07:12.695 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.695 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:12.695 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.695 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.695 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.695 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197663 9.6 9:6cc501ca:::benchmark_data_smithi151_261954_object94807:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T15:07:12.695 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.306393+0000", 2024-09-13T15:07:12.695 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 581.30640203300004, 2024-09-13T15:07:12.695 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1249737289999999, 2024-09-13T15:07:12.696 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:12.696 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:12.696 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:12.696 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T15:07:12.696 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:07:12.696 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197663 2024-09-13T15:07:12.696 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.696 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:12.696 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.696 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:12.696 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.306393+0000", 2024-09-13T15:07:12.696 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.696 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.697 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.697 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:12.697 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.306393+0000", 2024-09-13T15:07:12.697 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.697 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.697 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.697 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:12.697 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.306392+0000", 2024-09-13T15:07:12.697 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:07:12.697 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.697 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.697 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:12.697 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.306400+0000", 2024-09-13T15:07:12.697 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.4349999999999997e-06 2024-09-13T15:07:12.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.698 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.698 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:12.698 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.306412+0000", 2024-09-13T15:07:12.698 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2131e-05 2024-09-13T15:07:12.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.698 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.698 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:12.698 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.306420+0000", 2024-09-13T15:07:12.698 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.644e-06 2024-09-13T15:07:12.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.698 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.698 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.698 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.321161+0000", 2024-09-13T15:07:12.699 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.014741291 2024-09-13T15:07:12.699 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.699 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.699 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:12.699 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.321167+0000", 2024-09-13T15:07:12.699 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1399999999999997e-06 2024-09-13T15:07:12.699 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.699 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.699 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.699 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.331935+0000", 2024-09-13T15:07:12.699 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.010767926 2024-09-13T15:07:12.699 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.699 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.699 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:12.700 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.332015+0000", 2024-09-13T15:07:12.700 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.9986000000000003e-05 2024-09-13T15:07:12.700 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.700 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.700 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:07:12.700 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.332091+0000", 2024-09-13T15:07:12.700 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.6066e-05 2024-09-13T15:07:12.700 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.700 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.700 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:12.700 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.338252+0000", 2024-09-13T15:07:12.700 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0061609170000000001 2024-09-13T15:07:12.700 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.701 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.701 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.701 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.427278+0000", 2024-09-13T15:07:12.701 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.089025608000000006 2024-09-13T15:07:12.701 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.701 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.701 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.701 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.431274+0000", 2024-09-13T15:07:12.701 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.003996108 2024-09-13T15:07:12.701 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.701 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.701 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:12.701 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.431324+0000", 2024-09-13T15:07:12.702 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9848000000000001e-05 2024-09-13T15:07:12.702 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.702 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.702 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:12.702 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.431366+0000", 2024-09-13T15:07:12.702 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.2619e-05 2024-09-13T15:07:12.702 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.702 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:12.702 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.702 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.702 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.702 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197673 9.6 9:6e5dc1cf:::benchmark_data_smithi151_261954_object94817:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T15:07:12.702 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.319349+0000", 2024-09-13T15:07:12.703 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 581.29344552999999, 2024-09-13T15:07:12.703 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1161776160000001, 2024-09-13T15:07:12.703 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:12.703 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:12.703 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:12.703 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T15:07:12.703 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:07:12.703 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197673 2024-09-13T15:07:12.703 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.703 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:12.703 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.703 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:12.703 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.319349+0000", 2024-09-13T15:07:12.704 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.704 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.704 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.704 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:12.704 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.319349+0000", 2024-09-13T15:07:12.704 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.704 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.704 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.704 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:12.704 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.319347+0000", 2024-09-13T15:07:12.704 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T15:07:12.704 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.704 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.704 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:12.705 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.319354+0000", 2024-09-13T15:07:12.705 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.1110000000000002e-06 2024-09-13T15:07:12.705 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.705 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.705 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:12.705 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.319369+0000", 2024-09-13T15:07:12.705 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4137e-05 2024-09-13T15:07:12.705 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.705 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.705 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:12.705 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.319375+0000", 2024-09-13T15:07:12.705 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.4080000000000003e-06 2024-09-13T15:07:12.705 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.706 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.706 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.706 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.321187+0000", 2024-09-13T15:07:12.706 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001811937 2024-09-13T15:07:12.706 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.706 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.706 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:12.706 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.321193+0000", 2024-09-13T15:07:12.706 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1120000000000001e-06 2024-09-13T15:07:12.706 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.706 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.706 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.706 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.333564+0000", 2024-09-13T15:07:12.706 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0123709649999999 2024-09-13T15:07:12.707 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.707 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.707 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:12.707 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.333648+0000", 2024-09-13T15:07:12.707 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.3926999999999998e-05 2024-09-13T15:07:12.707 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.707 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.707 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:07:12.707 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.333724+0000", 2024-09-13T15:07:12.707 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.6155000000000005e-05 2024-09-13T15:07:12.707 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.707 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.707 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:12.707 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.339722+0000", 2024-09-13T15:07:12.708 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0059980229999999999 2024-09-13T15:07:12.708 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.708 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.708 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.708 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.427375+0000", 2024-09-13T15:07:12.708 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.087653178999999998 2024-09-13T15:07:12.708 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.708 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.708 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.708 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.435449+0000", 2024-09-13T15:07:12.708 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.008073172 2024-09-13T15:07:12.708 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.708 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.709 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:12.709 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.435490+0000", 2024-09-13T15:07:12.709 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1168000000000003e-05 2024-09-13T15:07:12.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.709 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.709 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:12.709 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.435527+0000", 2024-09-13T15:07:12.709 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.7069e-05 2024-09-13T15:07:12.709 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.709 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:12.709 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.709 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.710 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197677 9.6 9:6ff7b7d6:::benchmark_data_smithi151_261954_object94821:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T15:07:12.710 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.324907+0000", 2024-09-13T15:07:12.710 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 581.28788730700001, 2024-09-13T15:07:12.710 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.110717366, 2024-09-13T15:07:12.710 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:12.710 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:12.710 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:12.710 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T15:07:12.710 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:07:12.710 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197677 2024-09-13T15:07:12.710 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.710 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:12.710 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.710 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:12.711 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.324907+0000", 2024-09-13T15:07:12.711 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.711 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.711 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.711 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:12.711 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.324907+0000", 2024-09-13T15:07:12.711 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.711 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.711 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.711 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:12.711 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.324906+0000", 2024-09-13T15:07:12.711 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:07:12.711 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.711 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.712 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:12.712 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.324915+0000", 2024-09-13T15:07:12.712 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.2089999999999994e-06 2024-09-13T15:07:12.712 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.712 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.712 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:12.712 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.324927+0000", 2024-09-13T15:07:12.712 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1759999999999999e-05 2024-09-13T15:07:12.712 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.712 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.712 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:12.712 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.324932+0000", 2024-09-13T15:07:12.712 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0950000000000003e-06 2024-09-13T15:07:12.712 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.713 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.713 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.713 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.324978+0000", 2024-09-13T15:07:12.713 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5920000000000001e-05 2024-09-13T15:07:12.713 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.713 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.713 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:12.713 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.324985+0000", 2024-09-13T15:07:12.713 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.6399999999999997e-06 2024-09-13T15:07:12.713 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.713 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.713 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.713 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.335071+0000", 2024-09-13T15:07:12.714 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0100860730000001 2024-09-13T15:07:12.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.714 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.714 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:12.714 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.335145+0000", 2024-09-13T15:07:12.714 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.3349000000000005e-05 2024-09-13T15:07:12.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.714 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.714 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:07:12.714 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.335213+0000", 2024-09-13T15:07:12.714 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7719000000000007e-05 2024-09-13T15:07:12.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.714 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.714 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:12.715 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.339729+0000", 2024-09-13T15:07:12.715 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.004516066 2024-09-13T15:07:12.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.715 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.715 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.715 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.427485+0000", 2024-09-13T15:07:12.715 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.087756043000000006 2024-09-13T15:07:12.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.715 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.715 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.715 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.435557+0000", 2024-09-13T15:07:12.715 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.008072648 2024-09-13T15:07:12.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.715 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.716 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:12.716 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.435593+0000", 2024-09-13T15:07:12.717 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.5373999999999997e-05 2024-09-13T15:07:12.717 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.717 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.717 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:12.717 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.435625+0000", 2024-09-13T15:07:12.717 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.2020000000000002e-05 2024-09-13T15:07:12.717 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.717 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:12.717 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.717 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.717 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.717 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197692 9.6 9:6ccfa49f:::benchmark_data_smithi151_261954_object94836:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T15:07:12.718 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.354825+0000", 2024-09-13T15:07:12.718 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 581.25796916700006, 2024-09-13T15:07:12.718 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0808906490000001, 2024-09-13T15:07:12.718 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:12.718 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:12.718 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:12.718 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T15:07:12.718 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:07:12.718 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197692 2024-09-13T15:07:12.718 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.718 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:12.718 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.718 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:12.719 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.354825+0000", 2024-09-13T15:07:12.719 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.719 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.719 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.719 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:12.719 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.354825+0000", 2024-09-13T15:07:12.719 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.719 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.719 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.719 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:12.719 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.354824+0000", 2024-09-13T15:07:12.719 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:07:12.719 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.719 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.720 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:12.720 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.354829+0000", 2024-09-13T15:07:12.720 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.2789999999999998e-06 2024-09-13T15:07:12.720 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.720 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.720 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:12.720 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.354839+0000", 2024-09-13T15:07:12.720 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0052e-05 2024-09-13T15:07:12.720 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.720 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.720 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:12.720 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.354844+0000", 2024-09-13T15:07:12.720 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5140000000000003e-06 2024-09-13T15:07:12.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.721 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.721 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.721 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.354895+0000", 2024-09-13T15:07:12.721 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0875999999999999e-05 2024-09-13T15:07:12.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.721 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.721 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:12.721 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.354902+0000", 2024-09-13T15:07:12.721 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.6899999999999992e-06 2024-09-13T15:07:12.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.721 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.721 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.721 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.336702+0000", 2024-09-13T15:07:12.722 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.98179912499999999 2024-09-13T15:07:12.722 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.722 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.722 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:12.722 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.336790+0000", 2024-09-13T15:07:12.722 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.7943999999999997e-05 2024-09-13T15:07:12.722 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.722 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.722 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:07:12.722 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.336870+0000", 2024-09-13T15:07:12.722 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0018000000000005e-05 2024-09-13T15:07:12.722 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.722 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.722 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:12.723 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.339822+0000", 2024-09-13T15:07:12.723 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0029526470000000001 2024-09-13T15:07:12.723 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.723 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.723 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.723 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.427588+0000", 2024-09-13T15:07:12.723 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.087765459000000004 2024-09-13T15:07:12.723 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.723 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.723 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.723 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.435654+0000", 2024-09-13T15:07:12.723 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0080660839999999994 2024-09-13T15:07:12.723 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.723 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.724 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:12.724 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.435687+0000", 2024-09-13T15:07:12.724 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3231000000000001e-05 2024-09-13T15:07:12.724 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.724 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.724 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:12.724 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.435716+0000", 2024-09-13T15:07:12.724 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.9193999999999999e-05 2024-09-13T15:07:12.724 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.724 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:12.724 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.724 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.724 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.725 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.46034.0:197694 9.6 9:63889f28:::benchmark_data_smithi151_261954_object94838:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1252)", 2024-09-13T15:07:12.725 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T14:57:31.360247+0000", 2024-09-13T15:07:12.725 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 581.25254790400004, 2024-09-13T15:07:12.725 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.075543398, 2024-09-13T15:07:12.725 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:12.725 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:12.725 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:12.725 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.46034", 2024-09-13T15:07:12.725 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/2720818260", 2024-09-13T15:07:12.725 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 197694 2024-09-13T15:07:12.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.725 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:12.725 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.726 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:12.726 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.360247+0000", 2024-09-13T15:07:12.726 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.726 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.726 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:12.726 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.360247+0000", 2024-09-13T15:07:12.726 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.726 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.726 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:12.726 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.360245+0000", 2024-09-13T15:07:12.726 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:07:12.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.726 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.727 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:12.727 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.360250+0000", 2024-09-13T15:07:12.727 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8670000000000002e-06 2024-09-13T15:07:12.727 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.727 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.727 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:12.727 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.360270+0000", 2024-09-13T15:07:12.727 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.9613e-05 2024-09-13T15:07:12.727 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.727 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.727 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:12.727 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.360275+0000", 2024-09-13T15:07:12.727 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9400000000000001e-06 2024-09-13T15:07:12.727 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.728 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.728 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.728 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.360331+0000", 2024-09-13T15:07:12.728 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6437000000000002e-05 2024-09-13T15:07:12.728 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.728 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.728 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:12.728 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:31.360341+0000", 2024-09-13T15:07:12.728 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0128e-05 2024-09-13T15:07:12.728 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.728 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.728 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.728 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.338291+0000", 2024-09-13T15:07:12.729 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.97794926900000001 2024-09-13T15:07:12.729 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.729 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.729 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:12.729 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.338371+0000", 2024-09-13T15:07:12.729 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0735999999999994e-05 2024-09-13T15:07:12.729 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.729 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.729 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:07:12.729 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.338449+0000", 2024-09-13T15:07:12.729 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.8010999999999996e-05 2024-09-13T15:07:12.729 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.729 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.729 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:12.730 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.366705+0000", 2024-09-13T15:07:12.730 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.028255118999999999 2024-09-13T15:07:12.730 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.730 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.730 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.730 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.427612+0000", 2024-09-13T15:07:12.730 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.060906982999999998 2024-09-13T15:07:12.730 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.730 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.730 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.730 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.435740+0000", 2024-09-13T15:07:12.730 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0081282839999999995 2024-09-13T15:07:12.730 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.730 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.731 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:12.731 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.435767+0000", 2024-09-13T15:07:12.731 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.7004000000000001e-05 2024-09-13T15:07:12.731 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.731 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.731 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:12.731 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T14:57:32.435790+0000", 2024-09-13T15:07:12.731 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.3312999999999999e-05 2024-09-13T15:07:12.731 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.731 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:12.731 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.731 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.731 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.731 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:158539 10.6 10:61feb804:::benchmark_data_smithi151_288181_object32926:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1287)", 2024-09-13T15:07:12.732 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:00:24.491156+0000", 2024-09-13T15:07:12.732 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 408.12163836299999, 2024-09-13T15:07:12.732 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.72429242400000005, 2024-09-13T15:07:12.732 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:12.732 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "reached pg", 2024-09-13T15:07:12.732 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:12.732 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:07:12.732 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:07:12.732 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 158539 2024-09-13T15:07:12.732 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.732 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:12.732 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.732 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:12.733 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.491156+0000", 2024-09-13T15:07:12.733 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.733 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.733 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.733 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:12.733 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.491156+0000", 2024-09-13T15:07:12.733 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.733 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.733 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.733 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:12.733 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.491154+0000", 2024-09-13T15:07:12.733 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T15:07:12.733 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.733 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.734 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:12.734 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.491162+0000", 2024-09-13T15:07:12.734 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.2790000000000001e-06 2024-09-13T15:07:12.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.734 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.734 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:12.734 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.491175+0000", 2024-09-13T15:07:12.734 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3699999999999999e-05 2024-09-13T15:07:12.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.734 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.734 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:12.734 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.491182+0000", 2024-09-13T15:07:12.734 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.2929999999999997e-06 2024-09-13T15:07:12.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.734 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.735 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.735 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.491233+0000", 2024-09-13T15:07:12.735 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1668000000000001e-05 2024-09-13T15:07:12.735 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.735 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.735 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:12.735 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.491356+0000", 2024-09-13T15:07:12.735 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012310199999999999 2024-09-13T15:07:12.735 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.735 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.735 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,1,4", 2024-09-13T15:07:12.735 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.491489+0000", 2024-09-13T15:07:12.735 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000132943 2024-09-13T15:07:12.735 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.736 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.736 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.736 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.500964+0000", 2024-09-13T15:07:12.736 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0094743299999999996 2024-09-13T15:07:12.736 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.736 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.736 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:12.736 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.500974+0000", 2024-09-13T15:07:12.736 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0064000000000001e-05 2024-09-13T15:07:12.736 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.736 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.736 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.736 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.200251+0000", 2024-09-13T15:07:12.737 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.69927723500000005 2024-09-13T15:07:12.737 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.737 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.737 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for missing object", 2024-09-13T15:07:12.737 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.200401+0000", 2024-09-13T15:07:12.737 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000150335 2024-09-13T15:07:12.737 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.737 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.737 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.737 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.215359+0000", 2024-09-13T15:07:12.737 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.014957227999999999 2024-09-13T15:07:12.737 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.737 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.738 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:12.738 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.215449+0000", 2024-09-13T15:07:12.738 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.0146000000000001e-05 2024-09-13T15:07:12.738 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.738 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:12.738 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.738 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.738 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.738 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:158547 10.6 10:6a35454c:::benchmark_data_smithi151_288181_object32934:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1288)", 2024-09-13T15:07:12.738 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:00:24.524836+0000", 2024-09-13T15:07:12.738 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 408.087958238, 2024-09-13T15:07:12.738 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.69194671100000005, 2024-09-13T15:07:12.738 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:12.738 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:12.739 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:12.739 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:07:12.739 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:07:12.739 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 158547 2024-09-13T15:07:12.739 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.739 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:12.739 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.739 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:12.739 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.524836+0000", 2024-09-13T15:07:12.739 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.739 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.739 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.739 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:12.739 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.524836+0000", 2024-09-13T15:07:12.740 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.740 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.740 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.740 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:12.740 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.524835+0000", 2024-09-13T15:07:12.740 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:07:12.740 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.740 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.740 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:12.740 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.524842+0000", 2024-09-13T15:07:12.740 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.1729999999999998e-06 2024-09-13T15:07:12.740 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.740 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.741 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:12.741 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.524855+0000", 2024-09-13T15:07:12.741 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2846e-05 2024-09-13T15:07:12.741 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.741 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.741 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:12.741 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.524863+0000", 2024-09-13T15:07:12.741 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.5690000000000005e-06 2024-09-13T15:07:12.741 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.741 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.741 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.741 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.524909+0000", 2024-09-13T15:07:12.741 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5578999999999998e-05 2024-09-13T15:07:12.741 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.742 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.742 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:12.742 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.524916+0000", 2024-09-13T15:07:12.742 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.2919999999999998e-06 2024-09-13T15:07:12.742 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.742 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.742 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.742 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.200444+0000", 2024-09-13T15:07:12.742 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.67552759900000003 2024-09-13T15:07:12.742 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.742 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.742 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:12.742 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.200566+0000", 2024-09-13T15:07:12.742 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000121679 2024-09-13T15:07:12.743 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.743 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.743 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,1,4,6", 2024-09-13T15:07:12.743 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.201025+0000", 2024-09-13T15:07:12.743 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00045963599999999997 2024-09-13T15:07:12.743 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.743 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.743 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:12.743 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.211766+0000", 2024-09-13T15:07:12.743 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.010740669 2024-09-13T15:07:12.743 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.743 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.743 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.743 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.212741+0000", 2024-09-13T15:07:12.744 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00097493600000000003 2024-09-13T15:07:12.744 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.744 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.744 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.744 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.213917+0000", 2024-09-13T15:07:12.744 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0011761709999999999 2024-09-13T15:07:12.744 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.744 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.744 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.744 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.215887+0000", 2024-09-13T15:07:12.744 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001970238 2024-09-13T15:07:12.744 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.744 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.745 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.745 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.216613+0000", 2024-09-13T15:07:12.745 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00072576499999999996 2024-09-13T15:07:12.745 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.745 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.745 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:12.745 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.216666+0000", 2024-09-13T15:07:12.745 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.2759000000000001e-05 2024-09-13T15:07:12.745 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.745 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.745 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:12.745 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.216783+0000", 2024-09-13T15:07:12.745 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000117374 2024-09-13T15:07:12.745 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.746 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:12.746 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.746 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.746 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.746 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:219906 10.6 10:6064ed3c:::benchmark_data_smithi151_288181_object94293:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1304)", 2024-09-13T15:07:12.746 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:01:39.366236+0000", 2024-09-13T15:07:12.746 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 333.24655857200003, 2024-09-13T15:07:12.746 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.32339694899999999, 2024-09-13T15:07:12.746 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:12.746 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:12.746 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:12.746 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:07:12.746 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:07:12.746 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 219906 2024-09-13T15:07:12.747 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.747 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:12.747 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.747 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:12.747 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.366236+0000", 2024-09-13T15:07:12.747 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.747 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.747 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.747 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:12.747 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.366236+0000", 2024-09-13T15:07:12.747 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.747 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.747 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.747 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:12.747 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.366235+0000", 2024-09-13T15:07:12.748 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:07:12.748 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.748 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.748 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:12.748 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.366241+0000", 2024-09-13T15:07:12.748 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.0739999999999998e-06 2024-09-13T15:07:12.748 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.748 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.748 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:12.748 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.366253+0000", 2024-09-13T15:07:12.748 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2405999999999999e-05 2024-09-13T15:07:12.748 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.748 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.749 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:12.749 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.366259+0000", 2024-09-13T15:07:12.749 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7239999999999997e-06 2024-09-13T15:07:12.749 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.749 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.749 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.749 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.366324+0000", 2024-09-13T15:07:12.749 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.5598000000000005e-05 2024-09-13T15:07:12.749 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.749 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.749 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:12.749 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.366427+0000", 2024-09-13T15:07:12.749 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000102456 2024-09-13T15:07:12.749 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.750 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.750 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,1,4,6", 2024-09-13T15:07:12.750 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.366507+0000", 2024-09-13T15:07:12.750 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.9777000000000005e-05 2024-09-13T15:07:12.750 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.750 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.750 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.750 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.375503+0000", 2024-09-13T15:07:12.750 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0089965840000000002 2024-09-13T15:07:12.750 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.750 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.750 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.750 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.375553+0000", 2024-09-13T15:07:12.750 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0062999999999998e-05 2024-09-13T15:07:12.751 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.751 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.751 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.751 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.375611+0000", 2024-09-13T15:07:12.751 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7988000000000001e-05 2024-09-13T15:07:12.751 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.751 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.751 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:12.751 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.378119+0000", 2024-09-13T15:07:12.751 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002507951 2024-09-13T15:07:12.751 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.751 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.751 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.751 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.689441+0000", 2024-09-13T15:07:12.751 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.31132204899999999 2024-09-13T15:07:12.752 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.752 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.752 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:12.752 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.689496+0000", 2024-09-13T15:07:12.752 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4644000000000001e-05 2024-09-13T15:07:12.752 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.752 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.752 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:12.752 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.689633+0000", 2024-09-13T15:07:12.752 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013703599999999999 2024-09-13T15:07:12.752 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.752 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:12.752 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.753 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.753 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.753 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:219908 10.6 10:6f67220c:::benchmark_data_smithi151_288181_object94295:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1304)", 2024-09-13T15:07:12.753 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:01:39.367273+0000", 2024-09-13T15:07:12.753 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 333.24552128900001, 2024-09-13T15:07:12.753 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.32259187299999997, 2024-09-13T15:07:12.753 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:12.753 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:12.753 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:12.753 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:07:12.753 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:07:12.753 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 219908 2024-09-13T15:07:12.753 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.754 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:12.754 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.754 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:12.754 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.367273+0000", 2024-09-13T15:07:12.754 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.754 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.754 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.754 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:12.754 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.367273+0000", 2024-09-13T15:07:12.754 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:12.754 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.754 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.754 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:12.754 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.367272+0000", 2024-09-13T15:07:12.755 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:07:12.755 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.755 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.755 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:12.755 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.367277+0000", 2024-09-13T15:07:12.755 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9790000000000001e-06 2024-09-13T15:07:12.755 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.755 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.755 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:12.755 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.367292+0000", 2024-09-13T15:07:12.755 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4304000000000001e-05 2024-09-13T15:07:12.755 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.755 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.755 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:12.756 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.367298+0000", 2024-09-13T15:07:12.756 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.9150000000000001e-06 2024-09-13T15:07:12.756 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.756 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.756 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:12.756 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.368675+0000", 2024-09-13T15:07:12.756 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001377629 2024-09-13T15:07:12.756 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.756 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.756 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:12.756 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.368784+0000", 2024-09-13T15:07:12.756 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000109005 2024-09-13T15:07:12.756 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.756 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.757 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,1,4,6", 2024-09-13T15:07:12.757 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.368911+0000", 2024-09-13T15:07:12.757 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000126864 2024-09-13T15:07:12.757 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.757 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.757 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.757 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.375582+0000", 2024-09-13T15:07:12.757 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0066708460000000002 2024-09-13T15:07:12.757 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.757 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.757 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.757 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.375641+0000", 2024-09-13T15:07:12.757 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8743000000000003e-05 2024-09-13T15:07:12.757 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.758 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.758 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.758 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.376448+0000", 2024-09-13T15:07:12.758 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00080714000000000003 2024-09-13T15:07:12.758 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.758 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.758 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:12.758 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.378123+0000", 2024-09-13T15:07:12.758 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0016755610000000001 2024-09-13T15:07:12.758 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.758 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.758 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:12.758 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.689706+0000", 2024-09-13T15:07:12.758 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.31158232200000002 2024-09-13T15:07:12.759 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.759 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.759 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:12.759 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.689790+0000", 2024-09-13T15:07:12.759 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.4468999999999994e-05 2024-09-13T15:07:12.759 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:12.759 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:12.759 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:12.759 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:39.689865+0000", 2024-09-13T15:07:12.759 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.4993000000000005e-05 2024-09-13T15:07:12.759 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.759 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:12.759 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.759 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:12.760 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:12.760 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T15:07:13.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:12 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:07:13.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:12 smithi184 ceph-mon[92527]: pgmap v2409: 103 pgs: 103 active+clean; 582 KiB data, 7.2 GiB used, 529 GiB / 536 GiB avail 2024-09-13T15:07:13.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:12 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:07:13.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:12 smithi151 ceph-mon[104450]: pgmap v2409: 103 pgs: 103 active+clean; 582 KiB data, 7.2 GiB used, 529 GiB / 536 GiB avail 2024-09-13T15:07:13.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:12 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:07:13.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:12 smithi151 ceph-mon[108966]: pgmap v2409: 103 pgs: 103 active+clean; 582 KiB data, 7.2 GiB used, 529 GiB / 536 GiB avail 2024-09-13T15:07:13.432 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T15:07:13.678 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:07:14.590 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a[104427]: 2024-09-13T15:07:14.499+0000 7fe9da14f640 -1 mon.a@0(leader).osd e1370 definitely_dead 0 2024-09-13T15:07:14.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:07:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:07:14] "GET /metrics HTTP/1.1" 200 36680 "" "Prometheus/2.51.0" 2024-09-13T15:07:14.975 INFO:teuthology.orchestra.run.smithi151.stderr:marked in osd.4. 2024-09-13T15:07:14.987 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-13T15:07:15.065 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:14 smithi184 ceph-mon[92527]: pgmap v2410: 103 pgs: 103 active+clean; 582 KiB data, 7.2 GiB used, 529 GiB / 536 GiB avail 2024-09-13T15:07:15.065 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:14 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/844262093' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-13T15:07:15.065 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:14 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3848611177' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:07:15.065 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:14 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:07:15.236 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:14 smithi151 ceph-mon[104450]: pgmap v2410: 103 pgs: 103 active+clean; 582 KiB data, 7.2 GiB used, 529 GiB / 536 GiB avail 2024-09-13T15:07:15.236 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:14 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/844262093' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-13T15:07:15.236 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:14 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3848611177' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:07:15.236 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:14 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:07:15.237 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:14 smithi151 ceph-mon[108966]: pgmap v2410: 103 pgs: 103 active+clean; 582 KiB data, 7.2 GiB used, 529 GiB / 536 GiB avail 2024-09-13T15:07:15.237 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:14 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/844262093' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-13T15:07:15.237 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:14 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3848611177' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:07:15.237 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:14 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:07:15.370 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T15:07:15.371 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [], 2024-09-13T15:07:15.371 INFO:teuthology.orchestra.run.smithi184.stdout: "num_ops": 0 2024-09-13T15:07:15.371 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T15:07:16.022 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T15:07:16.032 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T15:07:16.277 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:07:16.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:15 smithi184 ceph-mon[92527]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-09-13T15:07:16.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:15 smithi184 ceph-mon[92527]: Cluster is now healthy 2024-09-13T15:07:16.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:15 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/844262093' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-13T15:07:16.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:15 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:07:16.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:15 smithi184 ceph-mon[92527]: osdmap e1371: 8 total, 8 up, 7 in 2024-09-13T15:07:16.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:15 smithi151 ceph-mon[104450]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-09-13T15:07:16.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:15 smithi151 ceph-mon[104450]: Cluster is now healthy 2024-09-13T15:07:16.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:15 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/844262093' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-13T15:07:16.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:15 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:07:16.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:15 smithi151 ceph-mon[104450]: osdmap e1371: 8 total, 8 up, 7 in 2024-09-13T15:07:16.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:15 smithi151 ceph-mon[108966]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-09-13T15:07:16.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:15 smithi151 ceph-mon[108966]: Cluster is now healthy 2024-09-13T15:07:16.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:15 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/844262093' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-13T15:07:16.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:15 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:07:16.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:15 smithi151 ceph-mon[108966]: osdmap e1371: 8 total, 8 up, 7 in 2024-09-13T15:07:16.417 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-09-13T15:07:16.418 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-13T15:07:16.418 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-09-13T15:07:16.418 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd in 2 2024-09-13T15:07:16.466 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:17.346 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:16 smithi184 ceph-mon[92527]: pgmap v2412: 103 pgs: 103 active+clean; 582 KiB data, 7.2 GiB used, 529 GiB / 536 GiB avail 2024-09-13T15:07:17.346 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:16 smithi184 ceph-mon[92527]: osdmap e1372: 8 total, 8 up, 7 in 2024-09-13T15:07:17.346 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:07:17.346 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:16 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:17.356 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:17.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:16 smithi151 ceph-mon[104450]: pgmap v2412: 103 pgs: 103 active+clean; 582 KiB data, 7.2 GiB used, 529 GiB / 536 GiB avail 2024-09-13T15:07:17.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:16 smithi151 ceph-mon[104450]: osdmap e1372: 8 total, 8 up, 7 in 2024-09-13T15:07:17.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:07:17.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:16 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:17.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:16 smithi151 ceph-mon[108966]: pgmap v2412: 103 pgs: 103 active+clean; 582 KiB data, 7.2 GiB used, 529 GiB / 536 GiB avail 2024-09-13T15:07:17.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:16 smithi151 ceph-mon[108966]: osdmap e1372: 8 total, 8 up, 7 in 2024-09-13T15:07:17.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:07:17.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:16 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:17.898 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T15:07:17.898 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [], 2024-09-13T15:07:17.899 INFO:teuthology.orchestra.run.smithi184.stdout: "complaint_time": 30, 2024-09-13T15:07:17.899 INFO:teuthology.orchestra.run.smithi184.stdout: "num_blocked_ops": 0 2024-09-13T15:07:17.899 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T15:07:18.493 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T15:07:18.741 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:07:19.105 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:18 smithi184 ceph-mon[92527]: pgmap v2414: 103 pgs: 103 active+clean; 582 KiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 18 B/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:07:19.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:18 smithi151 ceph-mon[104450]: pgmap v2414: 103 pgs: 103 active+clean; 582 KiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 18 B/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:07:19.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:18 smithi151 ceph-mon[108966]: pgmap v2414: 103 pgs: 103 active+clean; 582 KiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 18 B/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:07:19.989 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-13T15:07:20.088 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:19 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3400746752' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:07:20.286 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T15:07:20.287 INFO:teuthology.orchestra.run.smithi184.stdout: "size": 20, 2024-09-13T15:07:20.287 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 600, 2024-09-13T15:07:20.287 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [ 2024-09-13T15:07:20.287 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.287 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:158537 10.0 10:03b2b38f:::benchmark_data_smithi151_288181_object32924:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1287)", 2024-09-13T15:07:20.287 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:00:24.486293+0000", 2024-09-13T15:07:20.287 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 415.79722984199998, 2024-09-13T15:07:20.287 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6740325299999999, 2024-09-13T15:07:20.287 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:20.287 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "reached pg", 2024-09-13T15:07:20.287 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:20.288 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:07:20.288 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:07:20.288 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 158537 2024-09-13T15:07:20.288 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.288 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:20.288 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.288 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:20.288 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.486293+0000", 2024-09-13T15:07:20.288 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.288 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.288 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.288 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:20.288 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.486293+0000", 2024-09-13T15:07:20.289 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.289 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.289 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.289 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:20.289 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.486292+0000", 2024-09-13T15:07:20.289 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:07:20.289 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.289 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.289 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:20.289 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.486296+0000", 2024-09-13T15:07:20.289 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.6969999999999999e-06 2024-09-13T15:07:20.289 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.289 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.290 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:20.290 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.486304+0000", 2024-09-13T15:07:20.290 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.8000000000000004e-06 2024-09-13T15:07:20.290 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.290 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.290 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:20.290 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.486309+0000", 2024-09-13T15:07:20.290 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4850000000000003e-06 2024-09-13T15:07:20.290 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.290 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.290 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.290 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.486414+0000", 2024-09-13T15:07:20.290 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00010559400000000001 2024-09-13T15:07:20.290 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.291 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.291 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:20.291 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.486519+0000", 2024-09-13T15:07:20.291 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000104269 2024-09-13T15:07:20.291 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.291 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.291 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,4", 2024-09-13T15:07:20.291 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.486629+0000", 2024-09-13T15:07:20.291 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00011013799999999999 2024-09-13T15:07:20.291 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.291 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.291 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:20.291 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.492797+0000", 2024-09-13T15:07:20.291 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0061678449999999999 2024-09-13T15:07:20.291 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.507902+0000", 2024-09-13T15:07:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.015104935999999999 2024-09-13T15:07:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.507910+0000", 2024-09-13T15:07:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.4610000000000008e-06 2024-09-13T15:07:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.039073+0000", 2024-09-13T15:07:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.53116252399999997 2024-09-13T15:07:20.293 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.293 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.293 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:20.293 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.039083+0000", 2024-09-13T15:07:20.293 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0651e-05 2024-09-13T15:07:20.293 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.293 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.293 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.293 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.160259+0000", 2024-09-13T15:07:20.293 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.121175413 2024-09-13T15:07:20.293 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.293 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.293 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:20.294 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.160325+0000", 2024-09-13T15:07:20.294 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.6439000000000005e-05 2024-09-13T15:07:20.294 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.294 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:20.294 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.294 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.294 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.294 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:158538 10.0 10:01770849:::benchmark_data_smithi151_288181_object32925:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1287)", 2024-09-13T15:07:20.294 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:00:24.487042+0000", 2024-09-13T15:07:20.294 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 415.79648018500001, 2024-09-13T15:07:20.294 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6930981539999999, 2024-09-13T15:07:20.294 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:20.294 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "reached pg", 2024-09-13T15:07:20.294 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:20.295 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:07:20.295 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:07:20.295 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 158538 2024-09-13T15:07:20.295 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.295 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:20.295 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.295 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:20.295 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.487042+0000", 2024-09-13T15:07:20.295 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.295 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.295 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.295 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:20.295 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.487042+0000", 2024-09-13T15:07:20.295 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.295 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.296 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.296 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:20.296 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.487041+0000", 2024-09-13T15:07:20.296 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:07:20.296 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.296 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.296 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:20.296 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.487046+0000", 2024-09-13T15:07:20.296 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9799999999999998e-06 2024-09-13T15:07:20.296 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.296 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.296 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:20.296 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.487061+0000", 2024-09-13T15:07:20.296 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4866e-05 2024-09-13T15:07:20.297 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.297 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.297 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:20.297 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.487067+0000", 2024-09-13T15:07:20.297 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7230000000000001e-06 2024-09-13T15:07:20.297 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.297 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.297 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.297 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.489436+0000", 2024-09-13T15:07:20.297 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0023688329999999999 2024-09-13T15:07:20.297 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.297 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.297 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:20.297 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.489566+0000", 2024-09-13T15:07:20.298 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000130357 2024-09-13T15:07:20.298 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.298 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.298 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,4", 2024-09-13T15:07:20.298 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.489682+0000", 2024-09-13T15:07:20.298 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00011586000000000001 2024-09-13T15:07:20.298 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.298 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.298 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:20.298 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.492902+0000", 2024-09-13T15:07:20.298 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0032196579999999998 2024-09-13T15:07:20.298 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.298 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.298 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.299 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.507963+0000", 2024-09-13T15:07:20.299 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.015061105999999999 2024-09-13T15:07:20.299 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.299 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.299 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:20.299 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.507994+0000", 2024-09-13T15:07:20.299 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.0864999999999997e-05 2024-09-13T15:07:20.299 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.299 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.299 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.299 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.039763+0000", 2024-09-13T15:07:20.299 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.53176926599999996 2024-09-13T15:07:20.299 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.299 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.299 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:20.300 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.039770+0000", 2024-09-13T15:07:20.300 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.418e-06 2024-09-13T15:07:20.300 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.300 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.300 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.300 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.160812+0000", 2024-09-13T15:07:20.300 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1210413770000001 2024-09-13T15:07:20.300 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.300 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.300 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for missing object", 2024-09-13T15:07:20.300 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.160930+0000", 2024-09-13T15:07:20.300 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00011856199999999999 2024-09-13T15:07:20.300 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.301 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.301 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.301 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.180056+0000", 2024-09-13T15:07:20.301 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.019126059000000001 2024-09-13T15:07:20.301 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.301 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.301 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:20.301 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.180140+0000", 2024-09-13T15:07:20.301 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.4133000000000002e-05 2024-09-13T15:07:20.301 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.301 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:20.301 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.301 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.301 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.302 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:158536 10.e 10:72aa543a:::benchmark_data_smithi151_288181_object32923:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1288)", 2024-09-13T15:07:20.302 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:00:24.501582+0000", 2024-09-13T15:07:20.302 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 415.78194050500002, 2024-09-13T15:07:20.302 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.647897843, 2024-09-13T15:07:20.302 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:20.302 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "reached pg", 2024-09-13T15:07:20.302 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:20.302 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:07:20.302 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:07:20.302 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 158536 2024-09-13T15:07:20.302 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.302 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:20.302 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.302 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:20.303 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.501582+0000", 2024-09-13T15:07:20.303 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.303 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.303 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.303 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:20.303 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.501582+0000", 2024-09-13T15:07:20.303 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.303 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.303 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.303 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:20.303 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.501580+0000", 2024-09-13T15:07:20.303 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T15:07:20.303 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.303 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.304 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:20.304 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.501587+0000", 2024-09-13T15:07:20.304 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7710000000000001e-06 2024-09-13T15:07:20.304 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.304 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.304 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:20.304 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.501597+0000", 2024-09-13T15:07:20.304 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0472000000000001e-05 2024-09-13T15:07:20.304 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.304 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.304 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:20.304 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.501605+0000", 2024-09-13T15:07:20.304 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.1699999999999997e-06 2024-09-13T15:07:20.304 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.305 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.305 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.305 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.501714+0000", 2024-09-13T15:07:20.305 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000108838 2024-09-13T15:07:20.305 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.305 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.305 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:20.305 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.501723+0000", 2024-09-13T15:07:20.305 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.4060000000000005e-06 2024-09-13T15:07:20.305 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.305 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.305 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.305 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.040012+0000", 2024-09-13T15:07:20.305 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.53828984700000004 2024-09-13T15:07:20.306 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.306 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.306 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:20.306 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.040022+0000", 2024-09-13T15:07:20.306 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.5449999999999992e-06 2024-09-13T15:07:20.306 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.306 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.306 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.306 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.147912+0000", 2024-09-13T15:07:20.306 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.107890386 2024-09-13T15:07:20.306 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.306 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.306 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:20.306 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.149480+0000", 2024-09-13T15:07:20.306 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001567393 2024-09-13T15:07:20.307 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.307 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:20.307 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.307 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.307 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.307 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:158541 10.e 10:72218e81:::benchmark_data_smithi151_288181_object32928:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1288)", 2024-09-13T15:07:20.307 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:00:24.501618+0000", 2024-09-13T15:07:20.307 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 415.78190408299997, 2024-09-13T15:07:20.307 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.672070291, 2024-09-13T15:07:20.307 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:20.307 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:20.307 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:20.307 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:07:20.307 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:07:20.308 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 158541 2024-09-13T15:07:20.308 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.308 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:20.308 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.308 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:20.308 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.501618+0000", 2024-09-13T15:07:20.308 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.308 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.308 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.308 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:20.308 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.501618+0000", 2024-09-13T15:07:20.308 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.308 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.308 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.309 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:20.309 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.501618+0000", 2024-09-13T15:07:20.309 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T15:07:20.309 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.309 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.309 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:20.309 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.501621+0000", 2024-09-13T15:07:20.309 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.4429999999999999e-06 2024-09-13T15:07:20.309 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.309 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.309 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:20.309 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.501628+0000", 2024-09-13T15:07:20.309 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.8689999999999998e-06 2024-09-13T15:07:20.309 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.310 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.310 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:20.310 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.501632+0000", 2024-09-13T15:07:20.310 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3699999999999997e-06 2024-09-13T15:07:20.310 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.310 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.310 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.310 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.501743+0000", 2024-09-13T15:07:20.310 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000110888 2024-09-13T15:07:20.310 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.310 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.310 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:20.310 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.501749+0000", 2024-09-13T15:07:20.310 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6629999999999998e-06 2024-09-13T15:07:20.311 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.311 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.311 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.311 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.040044+0000", 2024-09-13T15:07:20.311 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.53829492899999998 2024-09-13T15:07:20.311 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.311 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.311 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:20.311 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.040050+0000", 2024-09-13T15:07:20.311 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.2659999999999998e-06 2024-09-13T15:07:20.311 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.311 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.311 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.311 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.149514+0000", 2024-09-13T15:07:20.312 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1094636170000001 2024-09-13T15:07:20.312 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.312 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.312 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:20.312 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.149912+0000", 2024-09-13T15:07:20.312 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00039853099999999998 2024-09-13T15:07:20.312 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.312 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.312 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,4,5", 2024-09-13T15:07:20.312 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.150152+0000", 2024-09-13T15:07:20.312 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000239941 2024-09-13T15:07:20.312 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.312 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.312 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:20.313 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.173181+0000", 2024-09-13T15:07:20.313 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.023028251999999999 2024-09-13T15:07:20.313 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.313 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.313 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.313 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.173436+0000", 2024-09-13T15:07:20.313 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00025518399999999999 2024-09-13T15:07:20.313 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.313 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.313 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.313 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.173500+0000", 2024-09-13T15:07:20.313 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.4560000000000005e-05 2024-09-13T15:07:20.313 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.313 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.314 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.314 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.173532+0000", 2024-09-13T15:07:20.314 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.1288999999999998e-05 2024-09-13T15:07:20.314 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.314 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.315 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:20.316 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.173578+0000", 2024-09-13T15:07:20.316 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.6714e-05 2024-09-13T15:07:20.316 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.316 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.316 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:20.316 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.173689+0000", 2024-09-13T15:07:20.316 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000110315 2024-09-13T15:07:20.316 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.317 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:20.317 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.317 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.317 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.317 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:158543 10.0 10:074ac37e:::benchmark_data_smithi151_288181_object32930:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1288)", 2024-09-13T15:07:20.317 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:00:24.506480+0000", 2024-09-13T15:07:20.317 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 415.77704283999998, 2024-09-13T15:07:20.317 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.676456087, 2024-09-13T15:07:20.317 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:20.317 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:20.317 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:20.317 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:07:20.317 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:07:20.318 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 158543 2024-09-13T15:07:20.318 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.318 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:20.318 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.318 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:20.318 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.506480+0000", 2024-09-13T15:07:20.318 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.318 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.318 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.318 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:20.319 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.506480+0000", 2024-09-13T15:07:20.319 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.319 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.319 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.319 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:20.319 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.506478+0000", 2024-09-13T15:07:20.319 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T15:07:20.319 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.319 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.319 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:20.319 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.506483+0000", 2024-09-13T15:07:20.319 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6319999999999996e-06 2024-09-13T15:07:20.319 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.319 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.320 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:20.320 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.506495+0000", 2024-09-13T15:07:20.320 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1647999999999999e-05 2024-09-13T15:07:20.320 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.320 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.320 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:20.320 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.506501+0000", 2024-09-13T15:07:20.320 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4739999999999996e-06 2024-09-13T15:07:20.320 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.320 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.320 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.320 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.508017+0000", 2024-09-13T15:07:20.320 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0015164289999999999 2024-09-13T15:07:20.321 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.321 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.321 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:20.321 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.508023+0000", 2024-09-13T15:07:20.321 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1589999999999998e-06 2024-09-13T15:07:20.321 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.321 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.321 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.321 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.039788+0000", 2024-09-13T15:07:20.321 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.53176479899999995 2024-09-13T15:07:20.321 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.321 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.321 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:20.322 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.039794+0000", 2024-09-13T15:07:20.322 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6130000000000003e-06 2024-09-13T15:07:20.322 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.322 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.322 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.322 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.160956+0000", 2024-09-13T15:07:20.322 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1211620840000001 2024-09-13T15:07:20.322 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.322 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.322 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:20.322 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.161052+0000", 2024-09-13T15:07:20.322 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.6434999999999994e-05 2024-09-13T15:07:20.322 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.322 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.323 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,4,5", 2024-09-13T15:07:20.323 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.162249+0000", 2024-09-13T15:07:20.323 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001196484 2024-09-13T15:07:20.323 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.323 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.323 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:20.323 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.179745+0000", 2024-09-13T15:07:20.323 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.017496748999999999 2024-09-13T15:07:20.323 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.323 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.323 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.323 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.180173+0000", 2024-09-13T15:07:20.323 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00042771699999999998 2024-09-13T15:07:20.323 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.324 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.324 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.324 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.180203+0000", 2024-09-13T15:07:20.324 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.9598999999999999e-05 2024-09-13T15:07:20.324 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.324 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.324 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.324 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.182783+0000", 2024-09-13T15:07:20.324 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0025803570000000001 2024-09-13T15:07:20.324 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.324 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.324 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:20.324 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.182826+0000", 2024-09-13T15:07:20.324 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3183999999999999e-05 2024-09-13T15:07:20.325 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.325 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.325 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:20.325 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.182936+0000", 2024-09-13T15:07:20.325 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000109477 2024-09-13T15:07:20.325 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.325 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:20.325 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.325 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.325 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.325 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:158548 10.0 10:06fda424:::benchmark_data_smithi151_288181_object32935:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1288)", 2024-09-13T15:07:20.325 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:00:24.522042+0000", 2024-09-13T15:07:20.325 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 415.76148041300002, 2024-09-13T15:07:20.326 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.670566674, 2024-09-13T15:07:20.326 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:20.326 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:20.326 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:20.326 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:07:20.326 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:07:20.326 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 158548 2024-09-13T15:07:20.326 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.326 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:20.326 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.326 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:20.326 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.522042+0000", 2024-09-13T15:07:20.326 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.326 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.326 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.327 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:20.327 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.522042+0000", 2024-09-13T15:07:20.327 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.327 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.327 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.327 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:20.327 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.522041+0000", 2024-09-13T15:07:20.327 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:07:20.327 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.327 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.327 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:20.327 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.522049+0000", 2024-09-13T15:07:20.327 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.551e-06 2024-09-13T15:07:20.328 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.328 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.328 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:20.328 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.522062+0000", 2024-09-13T15:07:20.328 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2798e-05 2024-09-13T15:07:20.328 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.328 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.328 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:20.328 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.522069+0000", 2024-09-13T15:07:20.328 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.5019999999999998e-06 2024-09-13T15:07:20.328 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.328 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.328 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.328 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.522133+0000", 2024-09-13T15:07:20.329 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.4264999999999995e-05 2024-09-13T15:07:20.329 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.329 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.329 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:20.329 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.522143+0000", 2024-09-13T15:07:20.329 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.8670000000000006e-06 2024-09-13T15:07:20.329 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.329 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.329 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.329 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.039810+0000", 2024-09-13T15:07:20.329 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.51766684600000001 2024-09-13T15:07:20.329 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.329 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.329 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:20.330 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.039815+0000", 2024-09-13T15:07:20.330 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3870000000000003e-06 2024-09-13T15:07:20.330 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.330 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.330 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.330 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.172694+0000", 2024-09-13T15:07:20.330 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1328790200000001 2024-09-13T15:07:20.330 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.330 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.330 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:20.330 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.172809+0000", 2024-09-13T15:07:20.330 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000114678 2024-09-13T15:07:20.330 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.331 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.331 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,4,5", 2024-09-13T15:07:20.331 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.172938+0000", 2024-09-13T15:07:20.331 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012925799999999999 2024-09-13T15:07:20.331 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.331 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.331 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.331 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.180235+0000", 2024-09-13T15:07:20.331 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0072970370000000001 2024-09-13T15:07:20.331 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.331 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.331 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.331 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.182969+0000", 2024-09-13T15:07:20.331 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0027342489999999998 2024-09-13T15:07:20.332 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.332 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.332 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.332 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.185499+0000", 2024-09-13T15:07:20.332 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0025295909999999999 2024-09-13T15:07:20.332 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.332 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.332 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:20.332 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.192453+0000", 2024-09-13T15:07:20.332 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0069544029999999996 2024-09-13T15:07:20.332 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.332 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.332 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:20.332 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.192504+0000", 2024-09-13T15:07:20.333 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0324999999999997e-05 2024-09-13T15:07:20.333 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.333 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.333 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:20.333 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.192609+0000", 2024-09-13T15:07:20.333 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00010510600000000001 2024-09-13T15:07:20.333 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.333 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:20.333 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.333 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.333 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.334 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:158549 10.0 10:0ff2ec5d:::benchmark_data_smithi151_288181_object32936:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1288)", 2024-09-13T15:07:20.334 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:00:24.522084+0000", 2024-09-13T15:07:20.334 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 415.76143886900002, 2024-09-13T15:07:20.334 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.670897399, 2024-09-13T15:07:20.334 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:20.334 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:20.334 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:20.334 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:07:20.334 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:07:20.334 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 158549 2024-09-13T15:07:20.334 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.334 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:20.334 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.334 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:20.335 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.522084+0000", 2024-09-13T15:07:20.335 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.335 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.335 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.335 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:20.335 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.522084+0000", 2024-09-13T15:07:20.335 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.335 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.335 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.335 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:20.335 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.522083+0000", 2024-09-13T15:07:20.335 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T15:07:20.335 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.335 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.336 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:20.336 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.522086+0000", 2024-09-13T15:07:20.336 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.9720000000000001e-06 2024-09-13T15:07:20.336 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.336 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.336 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:20.336 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.522092+0000", 2024-09-13T15:07:20.336 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.2430000000000002e-06 2024-09-13T15:07:20.336 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.336 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.336 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:20.336 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.522096+0000", 2024-09-13T15:07:20.336 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8369999999999999e-06 2024-09-13T15:07:20.337 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.337 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.337 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.337 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.522671+0000", 2024-09-13T15:07:20.337 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000574574 2024-09-13T15:07:20.337 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.337 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.337 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:20.337 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:24.522678+0000", 2024-09-13T15:07:20.337 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.4490000000000002e-06 2024-09-13T15:07:20.337 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.337 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.337 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.337 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.039833+0000", 2024-09-13T15:07:20.338 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.51715474800000005 2024-09-13T15:07:20.338 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.338 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.338 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:20.338 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.039839+0000", 2024-09-13T15:07:20.338 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.2570000000000004e-06 2024-09-13T15:07:20.338 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.338 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.338 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.338 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.176350+0000", 2024-09-13T15:07:20.338 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1365113710000001 2024-09-13T15:07:20.338 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.338 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.339 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:20.339 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.176617+0000", 2024-09-13T15:07:20.339 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00026680900000000001 2024-09-13T15:07:20.339 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.339 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.339 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,4,5", 2024-09-13T15:07:20.339 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.176976+0000", 2024-09-13T15:07:20.339 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00035866700000000001 2024-09-13T15:07:20.339 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.339 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.339 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.339 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.192663+0000", 2024-09-13T15:07:20.339 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.015686757999999999 2024-09-13T15:07:20.340 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.340 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.340 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:20.340 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.192669+0000", 2024-09-13T15:07:20.340 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.2879999999999998e-06 2024-09-13T15:07:20.340 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.340 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.340 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.340 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.192791+0000", 2024-09-13T15:07:20.340 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000122057 2024-09-13T15:07:20.340 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.340 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.340 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.340 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.192834+0000", 2024-09-13T15:07:20.341 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3469e-05 2024-09-13T15:07:20.341 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.341 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.341 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:20.341 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.192882+0000", 2024-09-13T15:07:20.341 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7503999999999998e-05 2024-09-13T15:07:20.341 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.341 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.341 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:20.341 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.192981+0000", 2024-09-13T15:07:20.341 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.8967000000000003e-05 2024-09-13T15:07:20.341 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.341 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:20.342 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.342 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.342 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.342 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:228643 10.0 10:0fcaf389:::benchmark_data_smithi151_288181_object103030:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1307)", 2024-09-13T15:07:20.342 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:01:49.597140+0000", 2024-09-13T15:07:20.342 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 330.68638221399999, 2024-09-13T15:07:20.342 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0130530419999999, 2024-09-13T15:07:20.342 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:20.342 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:20.342 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:20.342 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:07:20.342 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:07:20.342 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 228643 2024-09-13T15:07:20.342 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.342 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:20.343 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.343 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:20.343 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:49.597140+0000", 2024-09-13T15:07:20.343 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.343 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.343 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.343 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:20.343 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:49.597140+0000", 2024-09-13T15:07:20.343 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.343 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.343 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.343 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:20.343 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:49.597138+0000", 2024-09-13T15:07:20.344 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T15:07:20.344 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.344 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.344 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:20.344 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:49.597145+0000", 2024-09-13T15:07:20.344 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7329999999999997e-06 2024-09-13T15:07:20.344 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.344 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.344 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:20.344 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:49.597158+0000", 2024-09-13T15:07:20.344 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3127e-05 2024-09-13T15:07:20.344 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.344 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.345 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:20.345 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:49.597164+0000", 2024-09-13T15:07:20.345 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5090000000000001e-06 2024-09-13T15:07:20.345 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.345 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.345 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.345 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:49.597208+0000", 2024-09-13T15:07:20.345 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4611999999999999e-05 2024-09-13T15:07:20.345 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.345 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.345 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:20.345 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:49.597217+0000", 2024-09-13T15:07:20.345 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.5329999999999994e-06 2024-09-13T15:07:20.345 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.346 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.346 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.346 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:50.604289+0000", 2024-09-13T15:07:20.346 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.00707251 2024-09-13T15:07:20.346 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.346 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.346 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:20.346 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:50.604419+0000", 2024-09-13T15:07:20.346 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012969500000000001 2024-09-13T15:07:20.346 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.346 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.346 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:07:20.346 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:50.604554+0000", 2024-09-13T15:07:20.347 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013478000000000001 2024-09-13T15:07:20.347 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.347 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.347 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:20.347 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:50.606761+0000", 2024-09-13T15:07:20.347 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002206927 2024-09-13T15:07:20.347 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.347 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.347 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.347 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:50.609932+0000", 2024-09-13T15:07:20.347 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0031714899999999999 2024-09-13T15:07:20.347 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.347 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.347 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.348 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:50.610108+0000", 2024-09-13T15:07:20.348 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00017563600000000001 2024-09-13T15:07:20.348 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.348 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.348 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:20.348 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:50.610156+0000", 2024-09-13T15:07:20.348 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7852000000000002e-05 2024-09-13T15:07:20.348 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.348 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.348 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:20.348 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:01:50.610193+0000", 2024-09-13T15:07:20.348 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.7347e-05 2024-09-13T15:07:20.348 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.348 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:20.349 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.349 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.349 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.349 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:225236 11.d 11:b6a86611:::benchmark_data_smithi151_323728_object103939:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1354)", 2024-09-13T15:07:20.349 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:05:58.120704+0000", 2024-09-13T15:07:20.349 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 82.162818153000003, 2024-09-13T15:07:20.349 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.12615722500000001, 2024-09-13T15:07:20.349 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:20.349 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:20.349 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:20.349 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:07:20.349 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:07:20.349 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 225236 2024-09-13T15:07:20.349 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.350 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:20.350 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.350 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:20.350 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.120704+0000", 2024-09-13T15:07:20.350 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.350 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.350 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.350 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:20.350 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.120704+0000", 2024-09-13T15:07:20.350 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.350 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.350 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.350 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:20.350 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.120703+0000", 2024-09-13T15:07:20.351 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T15:07:20.351 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.351 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.351 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:20.351 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.120717+0000", 2024-09-13T15:07:20.351 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4773000000000001e-05 2024-09-13T15:07:20.351 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.351 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.351 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:20.351 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.120731+0000", 2024-09-13T15:07:20.351 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3312000000000001e-05 2024-09-13T15:07:20.351 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.351 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.351 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:20.352 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.120737+0000", 2024-09-13T15:07:20.352 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.207e-06 2024-09-13T15:07:20.352 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.352 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.352 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.352 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.125804+0000", 2024-09-13T15:07:20.352 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0050668689999999999 2024-09-13T15:07:20.352 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.352 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.352 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:20.352 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.125952+0000", 2024-09-13T15:07:20.352 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014858699999999999 2024-09-13T15:07:20.352 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.352 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.353 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:07:20.353 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.126099+0000", 2024-09-13T15:07:20.353 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014713500000000001 2024-09-13T15:07:20.353 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.353 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.353 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.353 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.130219+0000", 2024-09-13T15:07:20.353 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0041196460000000002 2024-09-13T15:07:20.353 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.353 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.353 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:20.353 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.133354+0000", 2024-09-13T15:07:20.353 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0031345700000000002 2024-09-13T15:07:20.353 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.354 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.354 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.354 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.245222+0000", 2024-09-13T15:07:20.354 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.11186804 2024-09-13T15:07:20.354 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.354 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.354 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:20.354 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.245275+0000", 2024-09-13T15:07:20.354 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.2975e-05 2024-09-13T15:07:20.354 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.354 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.354 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:20.354 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.246861+0000", 2024-09-13T15:07:20.354 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0015868010000000001 2024-09-13T15:07:20.355 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.355 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:20.355 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.355 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.355 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.355 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:225247 11.d 11:b2022588:::benchmark_data_smithi151_323728_object103950:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1354)", 2024-09-13T15:07:20.355 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:05:58.128438+0000", 2024-09-13T15:07:20.355 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 82.155084522999999, 2024-09-13T15:07:20.355 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.120973357, 2024-09-13T15:07:20.355 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:20.355 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:20.355 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:20.355 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:07:20.356 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:07:20.356 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 225247 2024-09-13T15:07:20.356 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.356 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:20.356 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.356 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:20.356 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.128438+0000", 2024-09-13T15:07:20.356 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.356 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.356 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.356 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:20.356 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.128438+0000", 2024-09-13T15:07:20.356 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.357 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.357 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.357 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:20.357 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.128437+0000", 2024-09-13T15:07:20.357 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:07:20.357 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.357 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.357 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:20.357 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.128442+0000", 2024-09-13T15:07:20.357 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7069999999999997e-06 2024-09-13T15:07:20.357 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.357 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.357 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:20.357 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.128453+0000", 2024-09-13T15:07:20.358 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0754000000000001e-05 2024-09-13T15:07:20.358 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.358 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.358 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:20.358 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.128458+0000", 2024-09-13T15:07:20.358 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3270000000000002e-06 2024-09-13T15:07:20.358 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.358 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.358 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.358 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.131012+0000", 2024-09-13T15:07:20.358 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002554171 2024-09-13T15:07:20.358 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.359 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.359 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:20.359 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.131146+0000", 2024-09-13T15:07:20.359 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013437099999999999 2024-09-13T15:07:20.359 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.359 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.359 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:07:20.359 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.131259+0000", 2024-09-13T15:07:20.359 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000112658 2024-09-13T15:07:20.359 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.359 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.359 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.359 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.141837+0000", 2024-09-13T15:07:20.359 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.010578550000000001 2024-09-13T15:07:20.360 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.360 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.360 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:20.360 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.143748+0000", 2024-09-13T15:07:20.360 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001910494 2024-09-13T15:07:20.360 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.360 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.360 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.360 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.247816+0000", 2024-09-13T15:07:20.360 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.10406843 2024-09-13T15:07:20.360 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.360 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.360 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:20.361 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.247868+0000", 2024-09-13T15:07:20.361 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1442e-05 2024-09-13T15:07:20.361 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.361 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.361 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:20.361 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.249411+0000", 2024-09-13T15:07:20.361 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0015435539999999999 2024-09-13T15:07:20.361 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.361 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:20.361 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.361 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.361 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.361 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:225259 11.d 11:b7beac66:::benchmark_data_smithi151_323728_object103962:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1354)", 2024-09-13T15:07:20.362 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:05:58.138242+0000", 2024-09-13T15:07:20.362 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 82.145279966999993, 2024-09-13T15:07:20.362 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.11368415799999999, 2024-09-13T15:07:20.362 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:20.362 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:20.362 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:20.362 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:07:20.362 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:07:20.362 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 225259 2024-09-13T15:07:20.362 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.362 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:20.362 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.362 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:20.362 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.138242+0000", 2024-09-13T15:07:20.363 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.363 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.363 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.363 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:20.363 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.138242+0000", 2024-09-13T15:07:20.363 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.363 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.363 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.363 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:20.363 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.138229+0000", 2024-09-13T15:07:20.363 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999862 2024-09-13T15:07:20.363 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.363 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.363 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:20.364 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.138247+0000", 2024-09-13T15:07:20.364 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.8546000000000002e-05 2024-09-13T15:07:20.364 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.364 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.364 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:20.364 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.138260+0000", 2024-09-13T15:07:20.364 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.327e-05 2024-09-13T15:07:20.364 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.364 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.364 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:20.364 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.138266+0000", 2024-09-13T15:07:20.364 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7769999999999999e-06 2024-09-13T15:07:20.364 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.364 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.365 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.365 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.143764+0000", 2024-09-13T15:07:20.365 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.005497923 2024-09-13T15:07:20.365 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.365 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.365 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:20.365 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.143843+0000", 2024-09-13T15:07:20.365 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.8770000000000006e-05 2024-09-13T15:07:20.365 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.365 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.365 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:07:20.365 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.143935+0000", 2024-09-13T15:07:20.365 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.2157999999999994e-05 2024-09-13T15:07:20.365 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.366 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.366 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:20.366 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.148891+0000", 2024-09-13T15:07:20.366 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.004956078 2024-09-13T15:07:20.366 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.366 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.366 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.366 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.148918+0000", 2024-09-13T15:07:20.366 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.6551e-05 2024-09-13T15:07:20.366 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.366 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.366 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.366 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.250104+0000", 2024-09-13T15:07:20.367 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.101186025 2024-09-13T15:07:20.367 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.367 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.367 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:20.367 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.250137+0000", 2024-09-13T15:07:20.367 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.2951999999999999e-05 2024-09-13T15:07:20.367 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.367 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.367 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:20.367 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.251927+0000", 2024-09-13T15:07:20.367 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0017899770000000001 2024-09-13T15:07:20.367 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.367 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:20.367 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.368 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.368 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.368 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:225278 11.d 11:bf437b02:::benchmark_data_smithi151_323728_object103981:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1354)", 2024-09-13T15:07:20.368 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:05:58.153876+0000", 2024-09-13T15:07:20.368 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 82.12964667, 2024-09-13T15:07:20.368 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.103156673, 2024-09-13T15:07:20.368 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:20.368 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:20.368 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:20.368 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:07:20.368 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:07:20.368 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 225278 2024-09-13T15:07:20.368 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.368 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:20.369 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.369 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:20.369 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.153876+0000", 2024-09-13T15:07:20.369 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.369 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.369 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.369 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:20.369 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.153876+0000", 2024-09-13T15:07:20.369 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.369 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.369 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.369 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:20.369 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.153874+0000", 2024-09-13T15:07:20.369 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:07:20.370 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.370 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.370 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:20.370 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.153881+0000", 2024-09-13T15:07:20.370 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.2609999999999999e-06 2024-09-13T15:07:20.370 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.370 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.370 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:20.370 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.234230+0000", 2024-09-13T15:07:20.371 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.080348939999999994 2024-09-13T15:07:20.371 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.371 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.371 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:20.371 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.234241+0000", 2024-09-13T15:07:20.371 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1092e-05 2024-09-13T15:07:20.371 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.371 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.371 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.371 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.234300+0000", 2024-09-13T15:07:20.371 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.9679000000000002e-05 2024-09-13T15:07:20.371 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.371 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.372 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:20.372 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.234463+0000", 2024-09-13T15:07:20.372 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00016275400000000001 2024-09-13T15:07:20.372 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.372 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.372 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:07:20.372 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.234597+0000", 2024-09-13T15:07:20.372 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000133781 2024-09-13T15:07:20.372 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.372 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.372 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.372 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.237238+0000", 2024-09-13T15:07:20.372 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0026408870000000002 2024-09-13T15:07:20.373 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.373 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.373 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:20.373 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.237693+0000", 2024-09-13T15:07:20.373 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00045530499999999998 2024-09-13T15:07:20.373 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.373 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.373 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.373 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.255627+0000", 2024-09-13T15:07:20.374 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.017933595 2024-09-13T15:07:20.374 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.374 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.374 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:20.374 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.255670+0000", 2024-09-13T15:07:20.374 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.2997000000000002e-05 2024-09-13T15:07:20.374 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.374 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.374 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:20.374 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.257032+0000", 2024-09-13T15:07:20.374 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001362678 2024-09-13T15:07:20.374 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.374 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:20.374 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.375 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.375 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.375 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:231877 11.d 11:b3a81ec5:::benchmark_data_smithi151_323728_object110580:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1354)", 2024-09-13T15:07:20.375 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:06:03.712878+0000", 2024-09-13T15:07:20.375 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 76.570643889999999, 2024-09-13T15:07:20.375 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.099330226999999993, 2024-09-13T15:07:20.375 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:20.375 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:20.375 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:20.375 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:07:20.375 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:07:20.375 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 231877 2024-09-13T15:07:20.375 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.376 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:20.376 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.376 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:20.376 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:03.712878+0000", 2024-09-13T15:07:20.376 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.376 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.376 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.376 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:20.376 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:03.712878+0000", 2024-09-13T15:07:20.376 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.376 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.376 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.376 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:20.377 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:03.712870+0000", 2024-09-13T15:07:20.377 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999914 2024-09-13T15:07:20.377 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.377 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.377 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:20.377 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:03.712882+0000", 2024-09-13T15:07:20.377 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1669e-05 2024-09-13T15:07:20.377 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.377 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.377 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:20.377 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:03.712889+0000", 2024-09-13T15:07:20.377 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.7170000000000008e-06 2024-09-13T15:07:20.377 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.377 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.377 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:20.378 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:03.712893+0000", 2024-09-13T15:07:20.378 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0910000000000003e-06 2024-09-13T15:07:20.378 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.378 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.378 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.378 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:03.720744+0000", 2024-09-13T15:07:20.378 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0078503759999999992 2024-09-13T15:07:20.378 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.378 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.378 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:20.378 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:03.720890+0000", 2024-09-13T15:07:20.378 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014592699999999999 2024-09-13T15:07:20.378 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.378 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.379 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:07:20.379 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:03.721041+0000", 2024-09-13T15:07:20.379 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00015167199999999999 2024-09-13T15:07:20.379 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.379 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.379 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.379 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:03.728152+0000", 2024-09-13T15:07:20.379 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0071108730000000002 2024-09-13T15:07:20.379 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.379 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.379 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:20.379 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:03.728172+0000", 2024-09-13T15:07:20.379 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.0027999999999999e-05 2024-09-13T15:07:20.379 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.380 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.380 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.380 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:03.809329+0000", 2024-09-13T15:07:20.380 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.081156502000000005 2024-09-13T15:07:20.380 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.380 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.380 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:20.380 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:03.809380+0000", 2024-09-13T15:07:20.380 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1688999999999999e-05 2024-09-13T15:07:20.380 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.380 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.380 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:20.380 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:03.812209+0000", 2024-09-13T15:07:20.380 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002828283 2024-09-13T15:07:20.381 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.381 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:20.381 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.381 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.381 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.381 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:232549 11.d 11:bc2267d2:::benchmark_data_smithi151_323728_object111252:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1354)", 2024-09-13T15:07:20.381 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:06:04.332439+0000", 2024-09-13T15:07:20.381 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 75.951083116999996, 2024-09-13T15:07:20.381 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.11551895399999999, 2024-09-13T15:07:20.381 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:20.381 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:20.381 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:20.381 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:07:20.381 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:07:20.382 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 232549 2024-09-13T15:07:20.382 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.382 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:20.382 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.382 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:20.382 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.332439+0000", 2024-09-13T15:07:20.382 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.382 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.382 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.382 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:20.382 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.332439+0000", 2024-09-13T15:07:20.382 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.382 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.383 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.383 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:20.383 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.332438+0000", 2024-09-13T15:07:20.383 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:07:20.383 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.383 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.383 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:20.383 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.332443+0000", 2024-09-13T15:07:20.383 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3970000000000002e-06 2024-09-13T15:07:20.383 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.383 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.383 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:20.383 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.332454+0000", 2024-09-13T15:07:20.383 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0831e-05 2024-09-13T15:07:20.384 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.384 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.384 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:20.384 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.332459+0000", 2024-09-13T15:07:20.384 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1150000000000002e-06 2024-09-13T15:07:20.384 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.384 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.384 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.384 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.340027+0000", 2024-09-13T15:07:20.384 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.007568183 2024-09-13T15:07:20.384 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.384 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.384 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:20.384 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.340169+0000", 2024-09-13T15:07:20.385 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014205000000000001 2024-09-13T15:07:20.385 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.385 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.385 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:07:20.385 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.340309+0000", 2024-09-13T15:07:20.385 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013925600000000001 2024-09-13T15:07:20.385 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.385 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.385 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:20.385 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.347898+0000", 2024-09-13T15:07:20.385 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0075898290000000002 2024-09-13T15:07:20.385 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.385 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.385 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.386 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.370295+0000", 2024-09-13T15:07:20.386 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.022396633999999999 2024-09-13T15:07:20.386 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.386 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.386 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.386 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.446105+0000", 2024-09-13T15:07:20.386 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.075810078000000003 2024-09-13T15:07:20.386 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.386 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.386 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:20.386 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.446156+0000", 2024-09-13T15:07:20.386 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0423999999999997e-05 2024-09-13T15:07:20.386 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.387 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:19 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3400746752' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:07:20.388 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:19 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3400746752' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:07:20.388 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:20 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a[104427]: 2024-09-13T15:07:20.107+0000 7fe9da14f640 -1 mon.a@0(leader).osd e1373 definitely_dead 0 2024-09-13T15:07:20.388 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.388 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:20.388 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.447958+0000", 2024-09-13T15:07:20.388 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0018026540000000001 2024-09-13T15:07:20.388 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.388 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:20.389 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.389 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.389 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.389 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:232552 11.d 11:b74bdb27:::benchmark_data_smithi151_323728_object111255:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1354)", 2024-09-13T15:07:20.389 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:06:04.334208+0000", 2024-09-13T15:07:20.389 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 75.949314255000004, 2024-09-13T15:07:20.389 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.11551228600000001, 2024-09-13T15:07:20.389 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:20.389 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:20.389 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:20.389 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:07:20.389 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:07:20.389 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 232552 2024-09-13T15:07:20.389 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.390 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:20.390 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.390 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:20.390 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.334208+0000", 2024-09-13T15:07:20.390 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.390 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.390 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.390 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:20.390 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.334208+0000", 2024-09-13T15:07:20.390 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.390 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.390 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.390 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:20.391 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.334207+0000", 2024-09-13T15:07:20.391 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:07:20.391 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.391 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.391 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:20.391 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.334216+0000", 2024-09-13T15:07:20.391 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.7730000000000005e-06 2024-09-13T15:07:20.391 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.391 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.391 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:20.391 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.334227+0000", 2024-09-13T15:07:20.391 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1528999999999999e-05 2024-09-13T15:07:20.391 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.391 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.391 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:20.392 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.334233+0000", 2024-09-13T15:07:20.392 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.2569999999999998e-06 2024-09-13T15:07:20.392 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.392 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.392 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.392 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.342717+0000", 2024-09-13T15:07:20.392 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0084841120000000003 2024-09-13T15:07:20.392 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.392 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.392 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:20.392 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.342866+0000", 2024-09-13T15:07:20.392 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014982600000000001 2024-09-13T15:07:20.392 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.393 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.393 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:07:20.393 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.343106+0000", 2024-09-13T15:07:20.393 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00023951699999999999 2024-09-13T15:07:20.393 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.393 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.393 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:20.393 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.350564+0000", 2024-09-13T15:07:20.393 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0074579249999999998 2024-09-13T15:07:20.393 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.393 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.393 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.393 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.370341+0000", 2024-09-13T15:07:20.393 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.019777395999999999 2024-09-13T15:07:20.394 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.394 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.394 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.394 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.448006+0000", 2024-09-13T15:07:20.394 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.077664423999999996 2024-09-13T15:07:20.394 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.394 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.394 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:20.394 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.448067+0000", 2024-09-13T15:07:20.394 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1032000000000001e-05 2024-09-13T15:07:20.394 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.394 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.394 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:20.394 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.449720+0000", 2024-09-13T15:07:20.395 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001653648 2024-09-13T15:07:20.395 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.395 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:20.395 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.395 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.395 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.395 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:232554 11.d 11:b20b471a:::benchmark_data_smithi151_323728_object111257:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1354)", 2024-09-13T15:07:20.395 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:06:04.334244+0000", 2024-09-13T15:07:20.395 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 75.949278045, 2024-09-13T15:07:20.395 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.11714150299999999, 2024-09-13T15:07:20.395 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:20.395 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:20.395 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:20.395 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:07:20.396 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:07:20.396 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 232554 2024-09-13T15:07:20.396 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.396 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:20.396 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.396 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:20.396 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.334244+0000", 2024-09-13T15:07:20.396 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.396 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.396 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.396 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:20.396 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.334244+0000", 2024-09-13T15:07:20.396 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.396 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.397 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.397 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:20.397 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.334244+0000", 2024-09-13T15:07:20.397 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T15:07:20.397 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.397 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.397 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:20.397 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.334247+0000", 2024-09-13T15:07:20.397 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.8619999999999999e-06 2024-09-13T15:07:20.397 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.397 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.397 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:20.397 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.334255+0000", 2024-09-13T15:07:20.397 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.7169999999999997e-06 2024-09-13T15:07:20.398 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.398 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.398 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:20.398 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.334259+0000", 2024-09-13T15:07:20.398 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.5899999999999999e-06 2024-09-13T15:07:20.398 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.398 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.398 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.398 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.345402+0000", 2024-09-13T15:07:20.398 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.011142529 2024-09-13T15:07:20.398 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.398 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.398 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:20.398 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.345518+0000", 2024-09-13T15:07:20.399 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00011666400000000001 2024-09-13T15:07:20.399 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.399 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.399 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:07:20.399 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.345643+0000", 2024-09-13T15:07:20.399 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012422600000000001 2024-09-13T15:07:20.399 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.399 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.399 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:20.399 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.353301+0000", 2024-09-13T15:07:20.399 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0076584979999999997 2024-09-13T15:07:20.399 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.399 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.399 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.400 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.370380+0000", 2024-09-13T15:07:20.400 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.017078923999999999 2024-09-13T15:07:20.400 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.400 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.400 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.400 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.449770+0000", 2024-09-13T15:07:20.400 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.079390085999999999 2024-09-13T15:07:20.400 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.400 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.400 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:20.400 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.449828+0000", 2024-09-13T15:07:20.400 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8304000000000003e-05 2024-09-13T15:07:20.400 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.400 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.401 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:20.401 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.451386+0000", 2024-09-13T15:07:20.401 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001557529 2024-09-13T15:07:20.401 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.401 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:20.401 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.401 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.401 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.401 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:232556 11.d 11:be6904bb:::benchmark_data_smithi151_323728_object111259:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1354)", 2024-09-13T15:07:20.401 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:06:04.334389+0000", 2024-09-13T15:07:20.401 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 75.949132969000004, 2024-09-13T15:07:20.401 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.120223238, 2024-09-13T15:07:20.401 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:20.401 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:20.402 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:20.402 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:07:20.402 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:07:20.402 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 232556 2024-09-13T15:07:20.402 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.402 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:20.402 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.402 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:20.402 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.334389+0000", 2024-09-13T15:07:20.402 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.402 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.402 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.402 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:20.403 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.334389+0000", 2024-09-13T15:07:20.403 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.403 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.403 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.403 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:20.403 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.334389+0000", 2024-09-13T15:07:20.403 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T15:07:20.403 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.403 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.403 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:20.403 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.334397+0000", 2024-09-13T15:07:20.403 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.6790000000000005e-06 2024-09-13T15:07:20.403 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.403 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.404 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:20.404 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.334407+0000", 2024-09-13T15:07:20.404 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0601000000000001e-05 2024-09-13T15:07:20.404 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.404 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.404 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:20.404 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.334411+0000", 2024-09-13T15:07:20.404 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.233e-06 2024-09-13T15:07:20.404 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.404 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.404 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.404 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.347933+0000", 2024-09-13T15:07:20.404 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.013521347 2024-09-13T15:07:20.405 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.405 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.405 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:20.405 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.348075+0000", 2024-09-13T15:07:20.405 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014210199999999999 2024-09-13T15:07:20.405 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.405 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.405 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:07:20.405 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.348269+0000", 2024-09-13T15:07:20.405 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00019431099999999999 2024-09-13T15:07:20.405 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.405 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.405 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:20.405 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.355344+0000", 2024-09-13T15:07:20.406 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0070749289999999998 2024-09-13T15:07:20.406 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.406 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.406 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.406 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.370422+0000", 2024-09-13T15:07:20.406 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.015078029 2024-09-13T15:07:20.406 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.406 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.406 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.406 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.453760+0000", 2024-09-13T15:07:20.406 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.083337724000000002 2024-09-13T15:07:20.406 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.406 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.406 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:20.406 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.453825+0000", 2024-09-13T15:07:20.407 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.5679000000000006e-05 2024-09-13T15:07:20.407 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.407 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.407 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:20.407 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.454613+0000", 2024-09-13T15:07:20.407 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00078719599999999999 2024-09-13T15:07:20.407 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.407 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:20.407 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.407 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.407 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.407 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:232574 11.d 11:b912aca1:::benchmark_data_smithi151_323728_object111277:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1354)", 2024-09-13T15:07:20.407 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:06:04.346954+0000", 2024-09-13T15:07:20.407 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 75.936567933999996, 2024-09-13T15:07:20.408 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.10856834999999999, 2024-09-13T15:07:20.408 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:20.408 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:20.408 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:20.408 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:07:20.408 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:07:20.408 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 232574 2024-09-13T15:07:20.408 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.408 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:20.408 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.408 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:20.408 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.346954+0000", 2024-09-13T15:07:20.408 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.409 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.409 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.409 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:20.409 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.346954+0000", 2024-09-13T15:07:20.409 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.409 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.409 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.409 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:20.409 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.346953+0000", 2024-09-13T15:07:20.409 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:07:20.409 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.409 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.409 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:20.409 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.346960+0000", 2024-09-13T15:07:20.410 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.0500000000000003e-06 2024-09-13T15:07:20.410 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.410 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.410 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:20.410 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.347087+0000", 2024-09-13T15:07:20.410 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000127138 2024-09-13T15:07:20.410 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.410 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.410 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:20.410 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.347096+0000", 2024-09-13T15:07:20.410 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.4419999999999998e-06 2024-09-13T15:07:20.410 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.410 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.410 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.411 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.350898+0000", 2024-09-13T15:07:20.411 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0038021439999999999 2024-09-13T15:07:20.411 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.411 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.411 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:20.411 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.351021+0000", 2024-09-13T15:07:20.411 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012263899999999999 2024-09-13T15:07:20.411 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.411 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.411 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:07:20.411 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.351150+0000", 2024-09-13T15:07:20.411 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012981000000000001 2024-09-13T15:07:20.411 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.411 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.412 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:20.412 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.355377+0000", 2024-09-13T15:07:20.412 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0042270500000000004 2024-09-13T15:07:20.412 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.412 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.412 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.412 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.370453+0000", 2024-09-13T15:07:20.412 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.015075539000000001 2024-09-13T15:07:20.412 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.412 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.412 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.412 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.454664+0000", 2024-09-13T15:07:20.412 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.084210853000000002 2024-09-13T15:07:20.412 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.413 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.413 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:20.413 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.454740+0000", 2024-09-13T15:07:20.413 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.6409000000000007e-05 2024-09-13T15:07:20.413 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.413 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.413 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:20.413 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.455523+0000", 2024-09-13T15:07:20.413 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00078263699999999998 2024-09-13T15:07:20.413 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.413 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:20.413 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.413 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.414 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.414 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:232580 11.d 11:bf70b94b:::benchmark_data_smithi151_323728_object111283:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1354)", 2024-09-13T15:07:20.414 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:06:04.349890+0000", 2024-09-13T15:07:20.414 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 75.933632719000002, 2024-09-13T15:07:20.414 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.106509468, 2024-09-13T15:07:20.414 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:20.414 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:20.414 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:20.414 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:07:20.414 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:07:20.414 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 232580 2024-09-13T15:07:20.414 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.414 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:20.414 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.415 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:20.415 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.349890+0000", 2024-09-13T15:07:20.415 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.415 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.415 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.415 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:20.415 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.349890+0000", 2024-09-13T15:07:20.415 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.415 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.415 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.415 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:20.415 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.349888+0000", 2024-09-13T15:07:20.415 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:07:20.415 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.415 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.416 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:20.416 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.349896+0000", 2024-09-13T15:07:20.416 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.8809999999999996e-06 2024-09-13T15:07:20.416 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.416 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.416 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:20.416 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.349907+0000", 2024-09-13T15:07:20.416 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0429e-05 2024-09-13T15:07:20.416 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.416 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.416 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:20.416 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.349912+0000", 2024-09-13T15:07:20.416 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5079999999999996e-06 2024-09-13T15:07:20.417 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.417 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.417 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.417 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.353324+0000", 2024-09-13T15:07:20.417 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.003412305 2024-09-13T15:07:20.417 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.417 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.417 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:20.417 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.353419+0000", 2024-09-13T15:07:20.417 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.4857000000000003e-05 2024-09-13T15:07:20.417 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.417 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.417 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:07:20.417 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.353502+0000", 2024-09-13T15:07:20.418 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.3279e-05 2024-09-13T15:07:20.418 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.418 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.418 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:20.418 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.355575+0000", 2024-09-13T15:07:20.418 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0020721730000000001 2024-09-13T15:07:20.418 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.418 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.418 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.418 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.370473+0000", 2024-09-13T15:07:20.419 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.014898672 2024-09-13T15:07:20.419 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.419 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.419 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.419 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.455568+0000", 2024-09-13T15:07:20.419 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.085094415000000007 2024-09-13T15:07:20.419 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.419 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.419 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:20.419 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.455642+0000", 2024-09-13T15:07:20.419 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.4343000000000003e-05 2024-09-13T15:07:20.419 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.419 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.420 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:20.420 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:04.456399+0000", 2024-09-13T15:07:20.420 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00075707499999999996 2024-09-13T15:07:20.420 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.420 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:20.420 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.420 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.420 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.420 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:236166 11.d 11:b7b1b494:::benchmark_data_smithi151_323728_object114869:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1354)", 2024-09-13T15:07:20.420 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:06:07.411370+0000", 2024-09-13T15:07:20.420 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 72.872152107999995, 2024-09-13T15:07:20.420 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.16085361400000001, 2024-09-13T15:07:20.420 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:20.420 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:20.421 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:20.421 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:07:20.421 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:07:20.421 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 236166 2024-09-13T15:07:20.421 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.421 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:20.421 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.421 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:20.421 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.411370+0000", 2024-09-13T15:07:20.421 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.421 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.421 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.421 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:20.421 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.411370+0000", 2024-09-13T15:07:20.422 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:20.422 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.422 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.422 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:20.422 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.411369+0000", 2024-09-13T15:07:20.422 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:07:20.422 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.422 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.422 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:20.422 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.411374+0000", 2024-09-13T15:07:20.422 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8590000000000004e-06 2024-09-13T15:07:20.422 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.422 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.423 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:20.423 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.411385+0000", 2024-09-13T15:07:20.423 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0757000000000001e-05 2024-09-13T15:07:20.423 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.423 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.423 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:20.423 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.411389+0000", 2024-09-13T15:07:20.423 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3900000000000003e-06 2024-09-13T15:07:20.423 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.423 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.423 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:20.423 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.412666+0000", 2024-09-13T15:07:20.423 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0012769579999999999 2024-09-13T15:07:20.423 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.424 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.424 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:20.424 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.412788+0000", 2024-09-13T15:07:20.424 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000121688 2024-09-13T15:07:20.424 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.424 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.424 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:07:20.424 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.412933+0000", 2024-09-13T15:07:20.424 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000144986 2024-09-13T15:07:20.424 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.424 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.424 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:20.424 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.415503+0000", 2024-09-13T15:07:20.424 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0025707989999999999 2024-09-13T15:07:20.425 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.425 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.425 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.425 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.419699+0000", 2024-09-13T15:07:20.425 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0041951439999999996 2024-09-13T15:07:20.425 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.425 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.425 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:20.425 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.570952+0000", 2024-09-13T15:07:20.425 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.15125360700000001 2024-09-13T15:07:20.425 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.425 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.425 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:20.425 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.571001+0000", 2024-09-13T15:07:20.426 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9079999999999998e-05 2024-09-13T15:07:20.426 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:20.426 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:20.426 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:20.426 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.572224+0000", 2024-09-13T15:07:20.426 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001222585 2024-09-13T15:07:20.426 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.426 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:20.426 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.426 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:20.426 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:20.426 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T15:07:20.427 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:07:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:07:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:07:20.998 INFO:teuthology.orchestra.run.smithi151.stderr:marked in osd.2. 2024-09-13T15:07:21.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:21 smithi184 ceph-mon[92527]: pgmap v2415: 103 pgs: 103 active+clean; 582 KiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 18 B/s, 1 keys/s, 1 objects/s recovering 2024-09-13T15:07:21.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:21 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3400746752' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:07:21.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:21 smithi184 ceph-mon[92527]: osdmap e1373: 8 total, 8 up, 7 in 2024-09-13T15:07:21.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:21 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1331404200' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-13T15:07:21.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:21 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-13T15:07:21.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:21 smithi151 ceph-mon[104450]: pgmap v2415: 103 pgs: 103 active+clean; 582 KiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 18 B/s, 1 keys/s, 1 objects/s recovering 2024-09-13T15:07:21.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:21 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3400746752' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:07:21.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:21 smithi151 ceph-mon[104450]: osdmap e1373: 8 total, 8 up, 7 in 2024-09-13T15:07:21.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:21 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1331404200' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-13T15:07:21.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:21 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-13T15:07:21.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:21 smithi151 ceph-mon[108966]: pgmap v2415: 103 pgs: 103 active+clean; 582 KiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 18 B/s, 1 keys/s, 1 objects/s recovering 2024-09-13T15:07:21.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:21 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3400746752' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:07:21.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:21 smithi151 ceph-mon[108966]: osdmap e1373: 8 total, 8 up, 7 in 2024-09-13T15:07:21.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:21 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1331404200' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-13T15:07:21.110 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:21 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-13T15:07:21.270 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T15:07:21.522 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:07:21.847 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-09-13T15:07:21.847 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-13T15:07:22.066 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:22 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-09-13T15:07:22.066 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:22 smithi184 ceph-mon[92527]: osdmap e1374: 8 total, 8 up, 8 in 2024-09-13T15:07:22.067 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:22 smithi184 ceph-mon[92527]: osdmap e1375: 8 total, 8 up, 8 in 2024-09-13T15:07:22.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:22 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-09-13T15:07:22.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:22 smithi151 ceph-mon[108966]: osdmap e1374: 8 total, 8 up, 8 in 2024-09-13T15:07:22.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:22 smithi151 ceph-mon[108966]: osdmap e1375: 8 total, 8 up, 8 in 2024-09-13T15:07:22.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:22 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-09-13T15:07:22.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:22 smithi151 ceph-mon[104450]: osdmap e1374: 8 total, 8 up, 8 in 2024-09-13T15:07:22.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:22 smithi151 ceph-mon[104450]: osdmap e1375: 8 total, 8 up, 8 in 2024-09-13T15:07:22.116 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:23.274 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:23 smithi184 ceph-mon[92527]: pgmap v2418: 103 pgs: 103 active+clean; 582 KiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 24 B/s, 1 keys/s, 1 objects/s recovering 2024-09-13T15:07:23.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:23 smithi151 ceph-mon[104450]: pgmap v2418: 103 pgs: 103 active+clean; 582 KiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 24 B/s, 1 keys/s, 1 objects/s recovering 2024-09-13T15:07:23.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:23 smithi151 ceph-mon[108966]: pgmap v2418: 103 pgs: 103 active+clean; 582 KiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 24 B/s, 1 keys/s, 1 objects/s recovering 2024-09-13T15:07:23.382 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T15:07:23.382 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [], 2024-09-13T15:07:23.382 INFO:teuthology.orchestra.run.smithi184.stdout: "num_ops": 0 2024-09-13T15:07:23.382 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T15:07:24.002 INFO:teuthology.orchestra.run.smithi151.stdout:mon.a: {} 2024-09-13T15:07:24.003 INFO:teuthology.orchestra.run.smithi151.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T15:07:24.003 INFO:teuthology.orchestra.run.smithi151.stdout:mon.b: {} 2024-09-13T15:07:24.003 INFO:teuthology.orchestra.run.smithi151.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T15:07:24.006 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T15:07:24.011 INFO:teuthology.orchestra.run.smithi151.stdout:mon.c: {} 2024-09-13T15:07:24.011 INFO:teuthology.orchestra.run.smithi151.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T15:07:24.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:24 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/77268609' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T15:07:24.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:24 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/77268609' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T15:07:24.251 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:07:24.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:24 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/77268609' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T15:07:24.766 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:07:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:07:24] "GET /metrics HTTP/1.1" 200 34292 "" "Prometheus/2.51.0" 2024-09-13T15:07:24.792 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T15:07:25.039 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:25.293 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:25 smithi151 ceph-mon[104450]: pgmap v2420: 103 pgs: 103 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 43 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:07:25.294 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:25 smithi151 ceph-mon[108966]: pgmap v2420: 103 pgs: 103 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 43 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:07:25.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:25 smithi184 ceph-mon[92527]: pgmap v2420: 103 pgs: 103 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 43 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:07:25.862 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T15:07:25.863 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [], 2024-09-13T15:07:25.863 INFO:teuthology.orchestra.run.smithi184.stdout: "complaint_time": 30, 2024-09-13T15:07:25.863 INFO:teuthology.orchestra.run.smithi184.stdout: "num_blocked_ops": 0 2024-09-13T15:07:25.863 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T15:07:26.618 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T15:07:26.862 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:07:27.033 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T15:07:27.233 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:27 smithi184 ceph-mon[92527]: pgmap v2421: 103 pgs: 103 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 43 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:07:27.233 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:27 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3481577851' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:07:27.233 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:27 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:07:27.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:27 smithi151 ceph-mon[104450]: pgmap v2421: 103 pgs: 103 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 43 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:07:27.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:27 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3481577851' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:07:27.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:27 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:07:27.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:27 smithi151 ceph-mon[108966]: pgmap v2421: 103 pgs: 103 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 43 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:07:27.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:27 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3481577851' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:07:27.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:27 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:07:27.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 0 1 2024-09-13T15:07:28.203 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:28.267 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:28 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T15:07:28.267 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:28 smithi184 ceph-mon[92527]: osdmap e1376: 8 total, 8 up, 8 in 2024-09-13T15:07:28.324 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:28 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T15:07:28.324 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:28 smithi151 ceph-mon[104450]: osdmap e1376: 8 total, 8 up, 8 in 2024-09-13T15:07:28.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:28 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T15:07:28.324 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:28 smithi151 ceph-mon[108966]: osdmap e1376: 8 total, 8 up, 8 in 2024-09-13T15:07:28.490 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-13T15:07:28.490 INFO:teuthology.orchestra.run.smithi184.stdout: "size": 20, 2024-09-13T15:07:28.490 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 600, 2024-09-13T15:07:28.490 INFO:teuthology.orchestra.run.smithi184.stdout: "ops": [ 2024-09-13T15:07:28.490 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.490 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36396.0:158550 10.1 10:8b6095c8:::benchmark_data_smithi151_288181_object32937:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1289)", 2024-09-13T15:07:28.490 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:00:25.030663+0000", 2024-09-13T15:07:28.491 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 423.45639365, 2024-09-13T15:07:28.491 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1097155240000001, 2024-09-13T15:07:28.491 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:28.491 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:28.491 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:28.491 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36396", 2024-09-13T15:07:28.491 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/1396538002", 2024-09-13T15:07:28.491 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 158550 2024-09-13T15:07:28.491 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.491 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:28.491 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.491 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:28.492 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.030663+0000", 2024-09-13T15:07:28.492 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.492 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.492 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.492 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:28.492 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.030663+0000", 2024-09-13T15:07:28.492 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.492 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.492 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.492 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:28.492 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.030661+0000", 2024-09-13T15:07:28.492 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:07:28.493 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.493 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.493 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:28.493 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.030667+0000", 2024-09-13T15:07:28.493 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.665e-06 2024-09-13T15:07:28.493 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.493 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.493 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:28.493 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.030684+0000", 2024-09-13T15:07:28.493 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.6813999999999999e-05 2024-09-13T15:07:28.493 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.493 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.493 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:28.493 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.030691+0000", 2024-09-13T15:07:28.494 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.9e-06 2024-09-13T15:07:28.494 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.494 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.494 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:28.494 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.031794+0000", 2024-09-13T15:07:28.494 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0011027089999999999 2024-09-13T15:07:28.494 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.494 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.494 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for peered", 2024-09-13T15:07:28.494 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:25.031801+0000", 2024-09-13T15:07:28.495 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.2969999999999998e-06 2024-09-13T15:07:28.495 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.495 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.495 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:28.495 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.134251+0000", 2024-09-13T15:07:28.495 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1024500559999999 2024-09-13T15:07:28.495 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.495 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.495 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:28.495 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.136551+0000", 2024-09-13T15:07:28.495 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002300346 2024-09-13T15:07:28.495 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.496 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.496 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 0,1,2,4", 2024-09-13T15:07:28.496 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.136703+0000", 2024-09-13T15:07:28.496 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00015136899999999999 2024-09-13T15:07:28.496 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.496 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.496 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.496 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.139715+0000", 2024-09-13T15:07:28.496 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0030125540000000002 2024-09-13T15:07:28.496 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.496 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.497 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.497 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.139744+0000", 2024-09-13T15:07:28.497 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.8540000000000001e-05 2024-09-13T15:07:28.497 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.497 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.497 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.497 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.139998+0000", 2024-09-13T15:07:28.497 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00025390699999999999 2024-09-13T15:07:28.497 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.497 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.497 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:28.497 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.140013+0000", 2024-09-13T15:07:28.498 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.5288999999999999e-05 2024-09-13T15:07:28.498 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.498 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.498 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.498 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.140199+0000", 2024-09-13T15:07:28.498 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00018655200000000001 2024-09-13T15:07:28.498 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.498 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.498 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:28.498 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.140254+0000", 2024-09-13T15:07:28.498 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.414e-05 2024-09-13T15:07:28.498 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.498 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.498 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:28.499 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:00:26.140379+0000", 2024-09-13T15:07:28.499 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000124936 2024-09-13T15:07:28.499 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.499 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:28.499 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.499 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.499 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.499 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:65032 11.4 11:246a5368:::benchmark_data_smithi151_323728_object65031:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1337)", 2024-09-13T15:07:28.499 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:03:29.444857+0000", 2024-09-13T15:07:28.499 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 239.04219926299999, 2024-09-13T15:07:28.499 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.258768467, 2024-09-13T15:07:28.499 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:28.499 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:28.500 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:28.500 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:07:28.500 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:07:28.500 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 65032 2024-09-13T15:07:28.500 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.500 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:28.500 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.500 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:28.500 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.444857+0000", 2024-09-13T15:07:28.500 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.500 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.500 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.500 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:28.501 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.444857+0000", 2024-09-13T15:07:28.501 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.501 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.501 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.501 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:28.501 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.444856+0000", 2024-09-13T15:07:28.501 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T15:07:28.501 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.501 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.501 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:28.501 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.444903+0000", 2024-09-13T15:07:28.501 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7034999999999999e-05 2024-09-13T15:07:28.502 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.502 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.502 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:28.502 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.444916+0000", 2024-09-13T15:07:28.502 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2996000000000001e-05 2024-09-13T15:07:28.502 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.502 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.502 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:28.502 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.444923+0000", 2024-09-13T15:07:28.502 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.2359999999999998e-06 2024-09-13T15:07:28.502 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.502 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.503 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:28.503 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.444980+0000", 2024-09-13T15:07:28.503 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6882999999999997e-05 2024-09-13T15:07:28.503 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.503 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.503 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:28.503 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.445114+0000", 2024-09-13T15:07:28.503 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013413699999999999 2024-09-13T15:07:28.503 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.503 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.503 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T15:07:28.503 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.445229+0000", 2024-09-13T15:07:28.503 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000115466 2024-09-13T15:07:28.503 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.503 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.504 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:28.504 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.447402+0000", 2024-09-13T15:07:28.504 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0021728419999999999 2024-09-13T15:07:28.504 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.504 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.504 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.504 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.447748+0000", 2024-09-13T15:07:28.504 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000345538 2024-09-13T15:07:28.504 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.504 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.504 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.504 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.703510+0000", 2024-09-13T15:07:28.504 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.255762092 2024-09-13T15:07:28.505 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.505 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.505 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:28.505 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.703575+0000", 2024-09-13T15:07:28.505 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.4760999999999999e-05 2024-09-13T15:07:28.505 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.505 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.505 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:28.505 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.703626+0000", 2024-09-13T15:07:28.505 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1150999999999999e-05 2024-09-13T15:07:28.505 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.505 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:28.505 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.505 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.506 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.506 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:65043 11.f 11:f38712a5:::benchmark_data_smithi151_323728_object65042:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1337)", 2024-09-13T15:07:28.506 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:03:29.454037+0000", 2024-09-13T15:07:28.506 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 239.03301966000001, 2024-09-13T15:07:28.506 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.25816714299999999, 2024-09-13T15:07:28.506 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:28.506 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:28.506 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:28.506 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:07:28.506 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:07:28.506 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 65043 2024-09-13T15:07:28.507 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.507 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:28.507 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.507 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:28.507 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.454037+0000", 2024-09-13T15:07:28.507 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.507 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.507 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.507 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:28.507 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.454037+0000", 2024-09-13T15:07:28.507 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.507 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.507 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.508 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:28.508 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.454036+0000", 2024-09-13T15:07:28.508 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:07:28.508 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.508 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.508 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:28.508 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.454084+0000", 2024-09-13T15:07:28.508 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8523000000000003e-05 2024-09-13T15:07:28.508 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.508 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.508 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:28.508 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.454096+0000", 2024-09-13T15:07:28.508 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2244e-05 2024-09-13T15:07:28.509 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.509 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.509 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:28.509 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.454101+0000", 2024-09-13T15:07:28.509 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3470000000000001e-06 2024-09-13T15:07:28.509 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.509 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.509 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:28.509 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.454137+0000", 2024-09-13T15:07:28.509 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.5828000000000003e-05 2024-09-13T15:07:28.509 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.509 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.510 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:28.510 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.454263+0000", 2024-09-13T15:07:28.510 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012643499999999999 2024-09-13T15:07:28.510 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.510 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.510 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:07:28.510 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.454403+0000", 2024-09-13T15:07:28.510 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013997400000000001 2024-09-13T15:07:28.510 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.510 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.510 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:28.510 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.461423+0000", 2024-09-13T15:07:28.510 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0070195730000000003 2024-09-13T15:07:28.511 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.511 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.511 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.511 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.461455+0000", 2024-09-13T15:07:28.511 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.2503999999999999e-05 2024-09-13T15:07:28.511 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.511 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.511 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.511 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.712155+0000", 2024-09-13T15:07:28.511 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.25069948199999997 2024-09-13T15:07:28.511 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.511 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.511 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:28.512 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.712184+0000", 2024-09-13T15:07:28.512 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.9088e-05 2024-09-13T15:07:28.512 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.512 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.512 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:28.512 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.712204+0000", 2024-09-13T15:07:28.512 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.0486e-05 2024-09-13T15:07:28.512 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.512 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:28.512 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.512 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.512 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.512 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:65045 11.f 11:f2f3f852:::benchmark_data_smithi151_323728_object65044:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1337)", 2024-09-13T15:07:28.513 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:03:29.455531+0000", 2024-09-13T15:07:28.513 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 239.031525655, 2024-09-13T15:07:28.513 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.26314478899999999, 2024-09-13T15:07:28.513 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:28.513 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:28.513 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:28.513 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:07:28.513 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:07:28.513 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 65045 2024-09-13T15:07:28.513 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.513 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:28.513 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.513 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:28.513 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.455531+0000", 2024-09-13T15:07:28.514 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.514 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.514 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.514 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:28.514 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.455531+0000", 2024-09-13T15:07:28.514 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.514 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.514 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.514 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:28.514 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.455530+0000", 2024-09-13T15:07:28.514 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:07:28.514 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.514 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.514 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:28.515 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.455561+0000", 2024-09-13T15:07:28.515 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.0673000000000003e-05 2024-09-13T15:07:28.515 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.515 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.515 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:28.515 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.455576+0000", 2024-09-13T15:07:28.515 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.56e-05 2024-09-13T15:07:28.515 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.515 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.515 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:28.515 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.455582+0000", 2024-09-13T15:07:28.515 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3280000000000002e-06 2024-09-13T15:07:28.515 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.515 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.516 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:28.516 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.457070+0000", 2024-09-13T15:07:28.516 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0014876189999999999 2024-09-13T15:07:28.516 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.516 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.516 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:28.516 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.457192+0000", 2024-09-13T15:07:28.516 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000121502 2024-09-13T15:07:28.516 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.517 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.517 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:07:28.517 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.457315+0000", 2024-09-13T15:07:28.517 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000123275 2024-09-13T15:07:28.517 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.517 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.517 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.517 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.461501+0000", 2024-09-13T15:07:28.517 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0041866059999999998 2024-09-13T15:07:28.517 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.517 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.517 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:28.518 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.461515+0000", 2024-09-13T15:07:28.518 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3247e-05 2024-09-13T15:07:28.518 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.518 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.518 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.518 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.718530+0000", 2024-09-13T15:07:28.518 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.25701534700000001 2024-09-13T15:07:28.518 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.518 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.519 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:28.519 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.718629+0000", 2024-09-13T15:07:28.519 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.9443000000000003e-05 2024-09-13T15:07:28.519 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.519 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.519 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:28.519 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.718676+0000", 2024-09-13T15:07:28.519 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.6306e-05 2024-09-13T15:07:28.519 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.519 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:28.519 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.520 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.520 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.520 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:65048 11.f 11:f41a0961:::benchmark_data_smithi151_323728_object65047:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1337)", 2024-09-13T15:07:28.520 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:03:29.456098+0000", 2024-09-13T15:07:28.520 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 239.030958392, 2024-09-13T15:07:28.520 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.262698775, 2024-09-13T15:07:28.520 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:28.520 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:28.521 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:28.521 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:07:28.521 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:07:28.521 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 65048 2024-09-13T15:07:28.521 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.521 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:28.522 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.522 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:28.522 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.456098+0000", 2024-09-13T15:07:28.522 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.522 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.522 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.522 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:28.522 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.456098+0000", 2024-09-13T15:07:28.522 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.522 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.522 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.522 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:28.522 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.456097+0000", 2024-09-13T15:07:28.522 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:07:28.523 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.523 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.523 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:28.523 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.456223+0000", 2024-09-13T15:07:28.523 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012584099999999999 2024-09-13T15:07:28.523 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.523 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.523 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:28.523 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.456362+0000", 2024-09-13T15:07:28.523 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013894100000000001 2024-09-13T15:07:28.523 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.523 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.523 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:28.524 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.456370+0000", 2024-09-13T15:07:28.524 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0760000000000003e-06 2024-09-13T15:07:28.524 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.524 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.524 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:28.524 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.459233+0000", 2024-09-13T15:07:28.524 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0028623759999999998 2024-09-13T15:07:28.524 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.524 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.524 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:28.524 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.459344+0000", 2024-09-13T15:07:28.524 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000111567 2024-09-13T15:07:28.524 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.525 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.525 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:07:28.525 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.459463+0000", 2024-09-13T15:07:28.525 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000118934 2024-09-13T15:07:28.525 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.525 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.525 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:28.525 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.461560+0000", 2024-09-13T15:07:28.525 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0020972849999999999 2024-09-13T15:07:28.525 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.525 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.525 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.525 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.461890+0000", 2024-09-13T15:07:28.526 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00032947500000000003 2024-09-13T15:07:28.526 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.526 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.526 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.526 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.718712+0000", 2024-09-13T15:07:28.526 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.25682233100000001 2024-09-13T15:07:28.526 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.526 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.526 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:28.526 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.718764+0000", 2024-09-13T15:07:28.526 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1419999999999999e-05 2024-09-13T15:07:28.526 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.527 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.527 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:28.527 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.718797+0000", 2024-09-13T15:07:28.527 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.3408999999999998e-05 2024-09-13T15:07:28.527 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.527 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:28.527 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.527 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.527 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.527 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:65050 11.4 11:2d37fcce:::benchmark_data_smithi151_323728_object65049:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1337)", 2024-09-13T15:07:28.527 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:03:29.458797+0000", 2024-09-13T15:07:28.527 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 239.02825945199999, 2024-09-13T15:07:28.527 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.249468412, 2024-09-13T15:07:28.527 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:28.528 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:28.528 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:28.528 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:07:28.528 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:07:28.528 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 65050 2024-09-13T15:07:28.528 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.528 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:28.528 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.528 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:28.528 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.458797+0000", 2024-09-13T15:07:28.528 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.528 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.528 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.529 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:28.529 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.458797+0000", 2024-09-13T15:07:28.529 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.529 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.529 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.529 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:28.529 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.458796+0000", 2024-09-13T15:07:28.529 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:07:28.529 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.529 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.529 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:28.529 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.458887+0000", 2024-09-13T15:07:28.529 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.1077999999999998e-05 2024-09-13T15:07:28.530 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.530 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.530 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:28.530 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.458902+0000", 2024-09-13T15:07:28.530 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.4776999999999999e-05 2024-09-13T15:07:28.530 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.530 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.530 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:28.530 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.458908+0000", 2024-09-13T15:07:28.530 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1500000000000004e-06 2024-09-13T15:07:28.530 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.530 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.531 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:28.531 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.458960+0000", 2024-09-13T15:07:28.531 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1802999999999999e-05 2024-09-13T15:07:28.531 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.531 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.531 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:28.531 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.459086+0000", 2024-09-13T15:07:28.531 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000125986 2024-09-13T15:07:28.531 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.531 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.531 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 2,6", 2024-09-13T15:07:28.531 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.459278+0000", 2024-09-13T15:07:28.531 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00019209899999999999 2024-09-13T15:07:28.531 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.531 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.532 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:28.532 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.461801+0000", 2024-09-13T15:07:28.532 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.002522472 2024-09-13T15:07:28.532 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.532 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.532 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.532 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.461942+0000", 2024-09-13T15:07:28.532 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00014171999999999999 2024-09-13T15:07:28.532 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.532 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.532 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.532 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.708213+0000", 2024-09-13T15:07:28.532 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.24627043500000001 2024-09-13T15:07:28.532 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.533 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.533 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:28.533 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.708243+0000", 2024-09-13T15:07:28.533 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.9731e-05 2024-09-13T15:07:28.533 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.533 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.533 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:28.533 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.708266+0000", 2024-09-13T15:07:28.533 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.3051e-05 2024-09-13T15:07:28.533 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.533 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:28.533 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.533 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.533 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.533 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:65054 11.f 11:f78e8263:::benchmark_data_smithi151_323728_object65053:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1337)", 2024-09-13T15:07:28.534 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:03:29.462921+0000", 2024-09-13T15:07:28.534 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 239.02413552199999, 2024-09-13T15:07:28.534 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.25599048499999999, 2024-09-13T15:07:28.534 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:28.534 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:28.534 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:28.534 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:07:28.534 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:07:28.534 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 65054 2024-09-13T15:07:28.534 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.534 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:28.534 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.534 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:28.535 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.462921+0000", 2024-09-13T15:07:28.535 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.535 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.535 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.535 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:28.535 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.462921+0000", 2024-09-13T15:07:28.535 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.535 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.535 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.535 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:28.536 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.462920+0000", 2024-09-13T15:07:28.536 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:07:28.536 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.536 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.536 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:28.536 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.462977+0000", 2024-09-13T15:07:28.536 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7173999999999998e-05 2024-09-13T15:07:28.536 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.536 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.536 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:28.536 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.462987+0000", 2024-09-13T15:07:28.536 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.8470000000000008e-06 2024-09-13T15:07:28.537 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.537 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.537 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:28.537 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.462991+0000", 2024-09-13T15:07:28.537 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.5860000000000001e-06 2024-09-13T15:07:28.537 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.537 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.537 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:28.537 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.463020+0000", 2024-09-13T15:07:28.538 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.9501000000000001e-05 2024-09-13T15:07:28.538 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.538 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.538 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:28.538 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.463100+0000", 2024-09-13T15:07:28.538 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.9427999999999999e-05 2024-09-13T15:07:28.538 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.538 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.538 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:07:28.538 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.463204+0000", 2024-09-13T15:07:28.538 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000104147 2024-09-13T15:07:28.538 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.538 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.538 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:28.539 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.465233+0000", 2024-09-13T15:07:28.539 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0020290059999999999 2024-09-13T15:07:28.539 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.539 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.539 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.539 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.465649+0000", 2024-09-13T15:07:28.539 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00041594899999999998 2024-09-13T15:07:28.539 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.539 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.539 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.539 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.718838+0000", 2024-09-13T15:07:28.539 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.25318934999999998 2024-09-13T15:07:28.539 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.540 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.540 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:28.540 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.718883+0000", 2024-09-13T15:07:28.540 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.4527999999999998e-05 2024-09-13T15:07:28.540 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.540 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.540 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:28.540 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.718912+0000", 2024-09-13T15:07:28.540 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.8804000000000001e-05 2024-09-13T15:07:28.540 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.540 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:28.540 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.541 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.541 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.541 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:65055 11.f 11:f09cec47:::benchmark_data_smithi151_323728_object65054:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1337)", 2024-09-13T15:07:28.541 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:03:29.465531+0000", 2024-09-13T15:07:28.541 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 239.021525688, 2024-09-13T15:07:28.541 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.25715232199999999, 2024-09-13T15:07:28.541 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:28.541 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:28.541 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:28.541 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:07:28.541 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:07:28.541 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 65055 2024-09-13T15:07:28.541 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.542 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:28.542 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.542 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:28.542 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.465531+0000", 2024-09-13T15:07:28.542 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.542 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.542 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.542 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:28.542 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.465531+0000", 2024-09-13T15:07:28.542 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.542 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.542 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.542 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:28.543 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.465530+0000", 2024-09-13T15:07:28.543 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:07:28.543 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.543 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.543 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:28.543 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.465596+0000", 2024-09-13T15:07:28.543 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.6105999999999994e-05 2024-09-13T15:07:28.543 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.543 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.543 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:28.543 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.465606+0000", 2024-09-13T15:07:28.543 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.8889999999999997e-06 2024-09-13T15:07:28.543 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.543 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.544 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:28.544 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.465610+0000", 2024-09-13T15:07:28.544 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3610000000000003e-06 2024-09-13T15:07:28.544 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.544 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.544 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:28.544 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.465666+0000", 2024-09-13T15:07:28.544 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.5889e-05 2024-09-13T15:07:28.544 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.544 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.544 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:28.544 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.465757+0000", 2024-09-13T15:07:28.544 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.0937e-05 2024-09-13T15:07:28.544 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.545 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.545 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:07:28.545 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.465844+0000", 2024-09-13T15:07:28.545 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.7004999999999998e-05 2024-09-13T15:07:28.545 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.545 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.545 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:28.545 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.467333+0000", 2024-09-13T15:07:28.545 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001489251 2024-09-13T15:07:28.545 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.545 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.545 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.545 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.467440+0000", 2024-09-13T15:07:28.545 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000106969 2024-09-13T15:07:28.546 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.546 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.546 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.546 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.722594+0000", 2024-09-13T15:07:28.546 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.25515391100000001 2024-09-13T15:07:28.546 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.546 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.547 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:28.547 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.722642+0000", 2024-09-13T15:07:28.547 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7778000000000002e-05 2024-09-13T15:07:28.547 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.547 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.547 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:28.547 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:03:29.722683+0000", 2024-09-13T15:07:28.547 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.1414e-05 2024-09-13T15:07:28.547 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.547 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:28.547 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.547 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.547 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.548 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:100458 11.f 11:f7d29eb5:::benchmark_data_smithi151_323728_object100457:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1342)", 2024-09-13T15:07:28.548 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:04:03.604011+0000", 2024-09-13T15:07:28.548 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 204.88304602100001, 2024-09-13T15:07:28.548 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.14455447599999999, 2024-09-13T15:07:28.548 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:28.548 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:28.548 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:28.548 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:07:28.548 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:07:28.548 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 100458 2024-09-13T15:07:28.548 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.548 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:28.549 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.549 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:28.549 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:04:03.604011+0000", 2024-09-13T15:07:28.549 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.549 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.549 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.549 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:28.549 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:04:03.604011+0000", 2024-09-13T15:07:28.549 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.549 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.549 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.549 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:28.549 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:04:03.604009+0000", 2024-09-13T15:07:28.549 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:07:28.550 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.550 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.550 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:28.550 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:04:03.604056+0000", 2024-09-13T15:07:28.550 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7193e-05 2024-09-13T15:07:28.550 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.550 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.550 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:28.550 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:04:03.604069+0000", 2024-09-13T15:07:28.550 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.2218999999999999e-05 2024-09-13T15:07:28.550 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.550 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.550 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:28.551 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:04:03.604074+0000", 2024-09-13T15:07:28.551 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4589999999999997e-06 2024-09-13T15:07:28.551 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.551 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.551 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:28.551 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:04:03.604120+0000", 2024-09-13T15:07:28.551 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5599999999999997e-05 2024-09-13T15:07:28.551 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.551 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.551 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:28.551 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:04:03.604250+0000", 2024-09-13T15:07:28.551 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013029799999999999 2024-09-13T15:07:28.551 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.551 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.552 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2,3", 2024-09-13T15:07:28.552 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:04:03.604471+0000", 2024-09-13T15:07:28.552 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00022085099999999999 2024-09-13T15:07:28.552 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.552 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.552 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:28.552 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:04:03.606613+0000", 2024-09-13T15:07:28.552 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0021419400000000002 2024-09-13T15:07:28.552 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.552 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.552 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.553 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:04:03.607527+0000", 2024-09-13T15:07:28.553 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000914352 2024-09-13T15:07:28.553 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.553 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.553 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.553 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:04:03.607939+0000", 2024-09-13T15:07:28.553 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00041191900000000001 2024-09-13T15:07:28.553 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.553 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.553 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.553 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:04:03.748399+0000", 2024-09-13T15:07:28.553 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.140459468 2024-09-13T15:07:28.554 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.554 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.554 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:28.554 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:04:03.748461+0000", 2024-09-13T15:07:28.554 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1971999999999996e-05 2024-09-13T15:07:28.554 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.554 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.554 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:28.554 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:04:03.748565+0000", 2024-09-13T15:07:28.554 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000104545 2024-09-13T15:07:28.554 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.554 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:28.554 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.555 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.555 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.555 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:198596 11.4 11:26945646:::benchmark_data_smithi151_323728_object77299:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1349)", 2024-09-13T15:07:28.555 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:05:34.317529+0000", 2024-09-13T15:07:28.555 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 114.169527644, 2024-09-13T15:07:28.555 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.19232540200000001, 2024-09-13T15:07:28.556 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:28.557 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:28.557 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:28.557 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:07:28.557 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:07:28.557 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 198596 2024-09-13T15:07:28.557 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.557 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:28.557 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.557 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:28.557 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.317529+0000", 2024-09-13T15:07:28.557 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.557 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.557 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.558 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:28.558 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.317529+0000", 2024-09-13T15:07:28.558 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.558 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.558 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.558 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:28.558 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.317529+0000", 2024-09-13T15:07:28.558 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T15:07:28.558 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.558 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.558 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:28.558 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.317531+0000", 2024-09-13T15:07:28.558 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.6850000000000001e-06 2024-09-13T15:07:28.559 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.559 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.559 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:28.559 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.317537+0000", 2024-09-13T15:07:28.559 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.2970000000000003e-06 2024-09-13T15:07:28.559 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.559 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.559 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:28.559 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.317541+0000", 2024-09-13T15:07:28.559 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9770000000000002e-06 2024-09-13T15:07:28.559 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.559 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.559 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:28.559 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.318238+0000", 2024-09-13T15:07:28.560 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00069773100000000005 2024-09-13T15:07:28.560 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.560 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.560 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:28.560 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.318363+0000", 2024-09-13T15:07:28.560 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012423800000000001 2024-09-13T15:07:28.560 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.560 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.560 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2,6", 2024-09-13T15:07:28.560 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.318478+0000", 2024-09-13T15:07:28.560 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00011582900000000001 2024-09-13T15:07:28.560 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.560 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.560 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:28.561 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.321129+0000", 2024-09-13T15:07:28.561 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0026503270000000001 2024-09-13T15:07:28.561 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.561 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.561 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.561 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.321348+0000", 2024-09-13T15:07:28.561 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000219244 2024-09-13T15:07:28.561 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.561 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.561 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.561 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.326255+0000", 2024-09-13T15:07:28.561 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0049071890000000002 2024-09-13T15:07:28.561 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.561 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.562 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.562 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.509673+0000", 2024-09-13T15:07:28.562 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.18341793000000001 2024-09-13T15:07:28.562 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.562 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.562 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:28.562 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.509728+0000", 2024-09-13T15:07:28.562 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4855000000000002e-05 2024-09-13T15:07:28.562 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.562 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.562 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:28.562 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.509854+0000", 2024-09-13T15:07:28.562 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012648999999999999 2024-09-13T15:07:28.562 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.563 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:28.563 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.563 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.563 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.563 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:198632 11.4 11:2be2c230:::benchmark_data_smithi151_323728_object77335:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1349)", 2024-09-13T15:07:28.563 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:05:34.348002+0000", 2024-09-13T15:07:28.563 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 114.13905423, 2024-09-13T15:07:28.563 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.16599768200000001, 2024-09-13T15:07:28.563 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:28.563 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:28.563 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:28.563 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:07:28.563 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:07:28.564 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 198632 2024-09-13T15:07:28.564 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.564 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:28.564 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.564 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:28.564 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.348002+0000", 2024-09-13T15:07:28.564 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.564 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.564 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.564 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:28.564 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.348002+0000", 2024-09-13T15:07:28.564 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.564 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.564 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.565 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:28.565 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.348001+0000", 2024-09-13T15:07:28.565 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:07:28.565 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.565 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.565 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:28.565 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.348010+0000", 2024-09-13T15:07:28.565 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.4660000000000005e-06 2024-09-13T15:07:28.565 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.565 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.565 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:28.565 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.348024+0000", 2024-09-13T15:07:28.565 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3478e-05 2024-09-13T15:07:28.565 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.566 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.566 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:28.566 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.348029+0000", 2024-09-13T15:07:28.566 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.6550000000000001e-06 2024-09-13T15:07:28.566 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.566 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.566 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:28.566 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.348081+0000", 2024-09-13T15:07:28.566 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.2011000000000001e-05 2024-09-13T15:07:28.566 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.566 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.566 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:28.566 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.348196+0000", 2024-09-13T15:07:28.566 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000114931 2024-09-13T15:07:28.567 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.567 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.567 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2,6", 2024-09-13T15:07:28.567 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.348318+0000", 2024-09-13T15:07:28.567 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012166499999999999 2024-09-13T15:07:28.567 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.567 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.567 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:28.567 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.350858+0000", 2024-09-13T15:07:28.568 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0025404059999999998 2024-09-13T15:07:28.568 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.568 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.568 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.568 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.351185+0000", 2024-09-13T15:07:28.568 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00032622700000000001 2024-09-13T15:07:28.568 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.568 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.568 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.568 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.351223+0000", 2024-09-13T15:07:28.568 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8309999999999997e-05 2024-09-13T15:07:28.568 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.568 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.568 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.569 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.513820+0000", 2024-09-13T15:07:28.569 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.16259710599999999 2024-09-13T15:07:28.569 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.569 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.572 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:28.572 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.513878+0000", 2024-09-13T15:07:28.572 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.8362999999999997e-05 2024-09-13T15:07:28.572 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.572 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.572 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:28.572 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.514000+0000", 2024-09-13T15:07:28.572 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000121622 2024-09-13T15:07:28.573 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.573 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:28.573 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.573 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.573 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.573 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:198639 11.4 11:2d4f28ca:::benchmark_data_smithi151_323728_object77342:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1349)", 2024-09-13T15:07:28.573 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:05:34.355944+0000", 2024-09-13T15:07:28.573 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 114.131113118, 2024-09-13T15:07:28.573 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.161472484, 2024-09-13T15:07:28.573 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:28.573 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:28.573 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:28.573 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:07:28.573 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:07:28.574 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 198639 2024-09-13T15:07:28.574 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.574 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:28.574 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.574 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:28.574 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.355944+0000", 2024-09-13T15:07:28.574 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.574 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.574 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.574 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:28.574 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.355944+0000", 2024-09-13T15:07:28.574 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.574 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.574 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.575 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:28.575 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.355943+0000", 2024-09-13T15:07:28.575 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999995 2024-09-13T15:07:28.575 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.575 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.575 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:28.575 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.355947+0000", 2024-09-13T15:07:28.575 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0409999999999999e-06 2024-09-13T15:07:28.575 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.575 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.575 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:28.575 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.356000+0000", 2024-09-13T15:07:28.575 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3167000000000001e-05 2024-09-13T15:07:28.576 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.576 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.576 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:28.576 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.356008+0000", 2024-09-13T15:07:28.576 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.8709999999999997e-06 2024-09-13T15:07:28.576 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.576 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.576 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:28.576 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.356082+0000", 2024-09-13T15:07:28.576 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.3915e-05 2024-09-13T15:07:28.576 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.576 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.576 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:28.576 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.356237+0000", 2024-09-13T15:07:28.577 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00015517800000000001 2024-09-13T15:07:28.577 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.577 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.577 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2,6", 2024-09-13T15:07:28.577 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.356404+0000", 2024-09-13T15:07:28.577 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00016660700000000001 2024-09-13T15:07:28.577 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.577 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.577 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.578 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.358991+0000", 2024-09-13T15:07:28.578 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0025872249999999999 2024-09-13T15:07:28.578 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.578 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.578 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:28.578 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.359084+0000", 2024-09-13T15:07:28.578 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.3187000000000001e-05 2024-09-13T15:07:28.578 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.578 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.578 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.578 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.360665+0000", 2024-09-13T15:07:28.578 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0015812280000000001 2024-09-13T15:07:28.578 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.578 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.579 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.579 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.517231+0000", 2024-09-13T15:07:28.579 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.156566076 2024-09-13T15:07:28.579 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.579 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.579 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:28.579 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.517283+0000", 2024-09-13T15:07:28.579 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1780999999999998e-05 2024-09-13T15:07:28.579 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.579 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.579 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:28.580 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.517416+0000", 2024-09-13T15:07:28.580 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013289700000000001 2024-09-13T15:07:28.580 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.580 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:28.580 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.580 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.580 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.580 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:198650 11.4 11:20fa7131:::benchmark_data_smithi151_323728_object77353:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1349)", 2024-09-13T15:07:28.580 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:05:34.369524+0000", 2024-09-13T15:07:28.580 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 114.117533123, 2024-09-13T15:07:28.580 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.149911922, 2024-09-13T15:07:28.580 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:28.580 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:28.581 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:28.581 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:07:28.581 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:07:28.581 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 198650 2024-09-13T15:07:28.581 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.581 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:28.581 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.581 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:28.581 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.369524+0000", 2024-09-13T15:07:28.581 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.581 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.581 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.581 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:28.581 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.369524+0000", 2024-09-13T15:07:28.581 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.582 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.582 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.582 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:28.582 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.369522+0000", 2024-09-13T15:07:28.582 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:07:28.582 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.582 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.582 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:28.582 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.369528+0000", 2024-09-13T15:07:28.582 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.9499999999999998e-06 2024-09-13T15:07:28.582 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.582 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.582 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:28.582 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.369538+0000", 2024-09-13T15:07:28.583 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0227000000000001e-05 2024-09-13T15:07:28.583 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.583 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.583 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:28.583 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.369543+0000", 2024-09-13T15:07:28.583 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7280000000000003e-06 2024-09-13T15:07:28.583 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.583 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.583 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:28.583 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.369593+0000", 2024-09-13T15:07:28.583 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.9767e-05 2024-09-13T15:07:28.583 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.583 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.584 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:28.584 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.369720+0000", 2024-09-13T15:07:28.584 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00012700299999999999 2024-09-13T15:07:28.584 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.584 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.584 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2,6", 2024-09-13T15:07:28.584 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.369866+0000", 2024-09-13T15:07:28.584 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000146124 2024-09-13T15:07:28.584 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.584 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.584 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:28.584 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.372298+0000", 2024-09-13T15:07:28.584 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0024316569999999998 2024-09-13T15:07:28.585 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.585 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.585 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.585 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.372430+0000", 2024-09-13T15:07:28.585 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000132204 2024-09-13T15:07:28.585 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.585 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.585 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.585 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.372528+0000", 2024-09-13T15:07:28.585 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.7873000000000003e-05 2024-09-13T15:07:28.585 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.585 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.586 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.586 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.519273+0000", 2024-09-13T15:07:28.586 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.14674548700000001 2024-09-13T15:07:28.586 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.586 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.586 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:28.586 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.519321+0000", 2024-09-13T15:07:28.586 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.7568999999999998e-05 2024-09-13T15:07:28.586 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.586 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.586 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:28.586 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.519435+0000", 2024-09-13T15:07:28.586 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000114707 2024-09-13T15:07:28.586 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.587 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:28.587 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.587 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.587 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.587 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:198662 11.4 11:22b2845c:::benchmark_data_smithi151_323728_object77365:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1349)", 2024-09-13T15:07:28.587 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:05:34.381753+0000", 2024-09-13T15:07:28.587 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 114.105303667, 2024-09-13T15:07:28.587 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.147044443, 2024-09-13T15:07:28.587 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:28.587 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:28.587 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:28.587 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:07:28.587 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:07:28.588 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 198662 2024-09-13T15:07:28.588 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.588 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:28.588 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.588 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:28.588 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.381753+0000", 2024-09-13T15:07:28.588 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.588 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.588 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.588 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:28.588 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.381753+0000", 2024-09-13T15:07:28.588 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.588 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.588 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.589 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:28.589 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.381752+0000", 2024-09-13T15:07:28.589 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:07:28.589 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.589 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.589 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:28.589 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.381759+0000", 2024-09-13T15:07:28.589 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.7170000000000002e-06 2024-09-13T15:07:28.589 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.589 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.589 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:28.589 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.381770+0000", 2024-09-13T15:07:28.589 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1725999999999999e-05 2024-09-13T15:07:28.589 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.590 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.590 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:28.590 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.381775+0000", 2024-09-13T15:07:28.590 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.6739999999999996e-06 2024-09-13T15:07:28.590 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.590 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.590 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:28.590 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.381811+0000", 2024-09-13T15:07:28.590 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.6273000000000003e-05 2024-09-13T15:07:28.590 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.590 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.590 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:28.590 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.381912+0000", 2024-09-13T15:07:28.591 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000101106 2024-09-13T15:07:28.591 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.591 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.591 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2,6", 2024-09-13T15:07:28.591 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.382005+0000", 2024-09-13T15:07:28.591 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.2158999999999996e-05 2024-09-13T15:07:28.591 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.591 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.591 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.592 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.384258+0000", 2024-09-13T15:07:28.592 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0022530670000000001 2024-09-13T15:07:28.592 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.592 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.592 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:28.592 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.384281+0000", 2024-09-13T15:07:28.592 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.3022000000000001e-05 2024-09-13T15:07:28.592 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.592 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.592 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.592 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.384591+0000", 2024-09-13T15:07:28.592 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00031076600000000003 2024-09-13T15:07:28.592 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.593 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.593 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.593 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.528616+0000", 2024-09-13T15:07:28.593 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.144024234 2024-09-13T15:07:28.593 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.593 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.593 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:28.593 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.528677+0000", 2024-09-13T15:07:28.593 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.1309000000000005e-05 2024-09-13T15:07:28.593 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.593 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.593 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:28.593 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:34.528797+0000", 2024-09-13T15:07:28.593 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000120501 2024-09-13T15:07:28.594 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.594 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:28.594 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.594 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.594 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.594 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:200600 11.f 11:fd44a013:::benchmark_data_smithi151_323728_object79303:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1349)", 2024-09-13T15:07:28.594 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:05:36.527954+0000", 2024-09-13T15:07:28.594 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 111.959102871, 2024-09-13T15:07:28.594 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.25635716400000003, 2024-09-13T15:07:28.594 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:28.594 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:28.594 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:28.594 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:07:28.594 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:07:28.595 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 200600 2024-09-13T15:07:28.595 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.595 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:28.595 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.595 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:28.595 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:36.527954+0000", 2024-09-13T15:07:28.595 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.595 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.595 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.595 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:28.595 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:36.527954+0000", 2024-09-13T15:07:28.595 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.595 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.595 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.595 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:28.596 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:36.527952+0000", 2024-09-13T15:07:28.596 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999981 2024-09-13T15:07:28.596 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.596 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.596 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:28.596 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:36.527961+0000", 2024-09-13T15:07:28.596 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.721e-06 2024-09-13T15:07:28.596 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.596 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.596 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:28.596 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:36.527974+0000", 2024-09-13T15:07:28.596 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3234000000000001e-05 2024-09-13T15:07:28.596 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.596 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.597 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:28.597 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:36.527979+0000", 2024-09-13T15:07:28.597 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.3900000000000001e-06 2024-09-13T15:07:28.597 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.597 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.597 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:28.597 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:36.528020+0000", 2024-09-13T15:07:28.597 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.0604e-05 2024-09-13T15:07:28.597 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.597 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.597 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:28.597 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:36.528096+0000", 2024-09-13T15:07:28.597 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.6262999999999998e-05 2024-09-13T15:07:28.597 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.598 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.598 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3", 2024-09-13T15:07:28.598 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:36.528168+0000", 2024-09-13T15:07:28.598 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.2026999999999993e-05 2024-09-13T15:07:28.598 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.598 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.598 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:28.598 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:36.530782+0000", 2024-09-13T15:07:28.598 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0026135080000000001 2024-09-13T15:07:28.599 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.599 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.599 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.600 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:36.534732+0000", 2024-09-13T15:07:28.600 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0039502879999999997 2024-09-13T15:07:28.600 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.600 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.600 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.600 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:36.784236+0000", 2024-09-13T15:07:28.600 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.24950430700000001 2024-09-13T15:07:28.600 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.600 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.600 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:28.600 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:36.784282+0000", 2024-09-13T15:07:28.600 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.5726999999999998e-05 2024-09-13T15:07:28.600 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.601 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.601 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:28.601 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:36.784311+0000", 2024-09-13T15:07:28.601 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.8789000000000002e-05 2024-09-13T15:07:28.601 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.601 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:28.601 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.601 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.601 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.601 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:200608 11.f 11:ffa24b44:::benchmark_data_smithi151_323728_object79311:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1349)", 2024-09-13T15:07:28.601 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:05:36.543254+0000", 2024-09-13T15:07:28.601 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 111.943802486, 2024-09-13T15:07:28.601 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.25239693800000002, 2024-09-13T15:07:28.602 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:28.602 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:28.602 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:28.602 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:07:28.602 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:07:28.602 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 200608 2024-09-13T15:07:28.602 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.602 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:28.602 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.602 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:28.602 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:36.543254+0000", 2024-09-13T15:07:28.602 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.602 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.602 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.603 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:28.603 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:36.543254+0000", 2024-09-13T15:07:28.603 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.603 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.603 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.603 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:28.603 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:36.543253+0000", 2024-09-13T15:07:28.603 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.9999986 2024-09-13T15:07:28.603 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.603 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.603 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:28.603 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:36.543260+0000", 2024-09-13T15:07:28.603 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.1339999999999998e-06 2024-09-13T15:07:28.603 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.604 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.604 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:28.604 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:36.543272+0000", 2024-09-13T15:07:28.604 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.1632e-05 2024-09-13T15:07:28.604 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.604 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.604 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:28.604 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:36.543277+0000", 2024-09-13T15:07:28.604 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.8760000000000004e-06 2024-09-13T15:07:28.604 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.604 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.604 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:28.605 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:36.543311+0000", 2024-09-13T15:07:28.605 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.4681000000000002e-05 2024-09-13T15:07:28.605 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.605 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.605 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:28.605 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:36.543394+0000", 2024-09-13T15:07:28.605 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.2769000000000007e-05 2024-09-13T15:07:28.605 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.605 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.605 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,3", 2024-09-13T15:07:28.605 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:36.543468+0000", 2024-09-13T15:07:28.606 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.4314999999999996e-05 2024-09-13T15:07:28.606 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.606 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.606 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:28.606 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:36.545432+0000", 2024-09-13T15:07:28.606 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.001963925 2024-09-13T15:07:28.606 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.606 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.606 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.606 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:36.545668+0000", 2024-09-13T15:07:28.606 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00023541400000000001 2024-09-13T15:07:28.606 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.606 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.607 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.607 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:36.795591+0000", 2024-09-13T15:07:28.607 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.249923218 2024-09-13T15:07:28.607 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.607 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.607 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:28.607 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:36.795625+0000", 2024-09-13T15:07:28.607 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.4289999999999999e-05 2024-09-13T15:07:28.607 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.607 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.607 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:28.607 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:36.795651+0000", 2024-09-13T15:07:28.607 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 2.5948e-05 2024-09-13T15:07:28.607 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.607 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:28.608 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.608 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.608 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.608 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:225229 11.5 11:a93c6574:::benchmark_data_smithi151_323728_object103932:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1354)", 2024-09-13T15:07:28.608 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:05:58.114651+0000", 2024-09-13T15:07:28.608 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 90.372405790000002, 2024-09-13T15:07:28.608 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.12236100900000001, 2024-09-13T15:07:28.608 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:28.608 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:28.608 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:28.608 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:07:28.608 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:07:28.608 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 225229 2024-09-13T15:07:28.609 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.609 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:28.609 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.609 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:28.609 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.114651+0000", 2024-09-13T15:07:28.609 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.609 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.609 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.609 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:28.609 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.114651+0000", 2024-09-13T15:07:28.609 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.609 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.609 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.609 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:28.610 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.114650+0000", 2024-09-13T15:07:28.610 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:07:28.610 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.610 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.610 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:28.610 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.114657+0000", 2024-09-13T15:07:28.610 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.588e-06 2024-09-13T15:07:28.610 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.610 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.610 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:28.610 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.114672+0000", 2024-09-13T15:07:28.610 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.473e-05 2024-09-13T15:07:28.610 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.610 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.611 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:28.611 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.114678+0000", 2024-09-13T15:07:28.611 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7699999999999998e-06 2024-09-13T15:07:28.611 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.611 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.611 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:28.611 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.114952+0000", 2024-09-13T15:07:28.611 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00027359599999999997 2024-09-13T15:07:28.611 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.611 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.611 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:28.611 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.115060+0000", 2024-09-13T15:07:28.611 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000108408 2024-09-13T15:07:28.611 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.611 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.612 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 5,6", 2024-09-13T15:07:28.612 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.115169+0000", 2024-09-13T15:07:28.612 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00010869099999999999 2024-09-13T15:07:28.612 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.612 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.612 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:28.612 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.120805+0000", 2024-09-13T15:07:28.612 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0056364539999999999 2024-09-13T15:07:28.612 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.612 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.612 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.612 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.131283+0000", 2024-09-13T15:07:28.612 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.010478024000000001 2024-09-13T15:07:28.612 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.613 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.613 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.613 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.236884+0000", 2024-09-13T15:07:28.613 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.105600813 2024-09-13T15:07:28.613 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.613 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.613 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:28.613 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.236972+0000", 2024-09-13T15:07:28.613 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.7998999999999996e-05 2024-09-13T15:07:28.613 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.613 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.613 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:28.613 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:05:58.237012+0000", 2024-09-13T15:07:28.614 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.9947000000000002e-05 2024-09-13T15:07:28.614 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.614 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:28.614 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.614 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.614 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.614 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:236164 11.4 11:240df607:::benchmark_data_smithi151_323728_object114867:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1354)", 2024-09-13T15:07:28.614 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:06:07.409131+0000", 2024-09-13T15:07:28.614 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 81.077925363999995, 2024-09-13T15:07:28.614 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.21810183599999999, 2024-09-13T15:07:28.614 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:28.615 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:28.615 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:28.615 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:07:28.615 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:07:28.615 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 236164 2024-09-13T15:07:28.615 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.615 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:28.615 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.615 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:28.615 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.409131+0000", 2024-09-13T15:07:28.615 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.615 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.615 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.615 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:28.616 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.409131+0000", 2024-09-13T15:07:28.616 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.616 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.616 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.616 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:28.616 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.409130+0000", 2024-09-13T15:07:28.616 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:07:28.616 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.616 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.616 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:28.616 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.409136+0000", 2024-09-13T15:07:28.616 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 6.3539999999999996e-06 2024-09-13T15:07:28.616 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.617 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.617 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:28.617 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.409147+0000", 2024-09-13T15:07:28.617 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.0249e-05 2024-09-13T15:07:28.617 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.617 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.617 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:28.617 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.409152+0000", 2024-09-13T15:07:28.617 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.4190000000000001e-06 2024-09-13T15:07:28.617 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.617 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.617 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:28.617 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.409195+0000", 2024-09-13T15:07:28.618 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.2664999999999999e-05 2024-09-13T15:07:28.618 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.618 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.618 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:28.618 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.409325+0000", 2024-09-13T15:07:28.618 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00013043899999999999 2024-09-13T15:07:28.618 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.618 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.618 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2,6", 2024-09-13T15:07:28.618 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.409463+0000", 2024-09-13T15:07:28.618 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000137408 2024-09-13T15:07:28.618 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.618 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.618 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.619 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.414572+0000", 2024-09-13T15:07:28.619 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0051095569999999998 2024-09-13T15:07:28.619 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.619 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.619 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.619 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.414590+0000", 2024-09-13T15:07:28.619 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.747e-05 2024-09-13T15:07:28.619 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.619 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.619 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:28.619 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.416866+0000", 2024-09-13T15:07:28.619 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0022765229999999999 2024-09-13T15:07:28.619 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.620 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.620 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.620 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.627023+0000", 2024-09-13T15:07:28.620 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.210156804 2024-09-13T15:07:28.620 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.620 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.620 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:28.620 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.627099+0000", 2024-09-13T15:07:28.620 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.6333000000000003e-05 2024-09-13T15:07:28.620 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.620 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.620 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:28.620 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.627233+0000", 2024-09-13T15:07:28.620 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000133783 2024-09-13T15:07:28.621 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.621 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:28.621 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.621 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.621 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.621 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:236168 11.4 11:216bfd7e:::benchmark_data_smithi151_323728_object114871:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1354)", 2024-09-13T15:07:28.621 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:06:07.411450+0000", 2024-09-13T15:07:28.621 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 81.075606484000005, 2024-09-13T15:07:28.621 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.21804651799999999, 2024-09-13T15:07:28.621 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:28.621 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:28.621 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:28.621 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:07:28.622 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:07:28.622 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 236168 2024-09-13T15:07:28.622 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.622 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:28.622 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.622 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:28.622 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.411450+0000", 2024-09-13T15:07:28.622 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.622 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.622 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.622 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:28.622 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.411450+0000", 2024-09-13T15:07:28.622 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.622 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.623 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.623 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:28.623 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.411449+0000", 2024-09-13T15:07:28.623 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:07:28.623 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.623 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.623 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:28.623 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.411454+0000", 2024-09-13T15:07:28.623 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.369e-06 2024-09-13T15:07:28.623 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.623 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.623 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:28.623 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.411463+0000", 2024-09-13T15:07:28.623 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.0280000000000006e-06 2024-09-13T15:07:28.624 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.624 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.624 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:28.624 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.411467+0000", 2024-09-13T15:07:28.624 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 3.8310000000000003e-06 2024-09-13T15:07:28.624 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.624 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.624 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:28.624 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.411868+0000", 2024-09-13T15:07:28.624 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00040137000000000001 2024-09-13T15:07:28.624 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.624 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.624 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:28.625 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.412011+0000", 2024-09-13T15:07:28.625 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000142762 2024-09-13T15:07:28.625 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.625 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.625 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2,6", 2024-09-13T15:07:28.625 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.412156+0000", 2024-09-13T15:07:28.625 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.000145527 2024-09-13T15:07:28.625 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.625 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.625 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.625 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.416920+0000", 2024-09-13T15:07:28.625 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0047633989999999998 2024-09-13T15:07:28.625 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.626 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.626 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.626 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.416971+0000", 2024-09-13T15:07:28.626 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.1307000000000002e-05 2024-09-13T15:07:28.626 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.626 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.626 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:28.626 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.416980+0000", 2024-09-13T15:07:28.626 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.6109999999999994e-06 2024-09-13T15:07:28.626 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.626 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.626 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.626 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.629335+0000", 2024-09-13T15:07:28.626 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.21235552899999999 2024-09-13T15:07:28.627 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.627 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.627 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:28.627 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.629386+0000", 2024-09-13T15:07:28.627 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.0927000000000002e-05 2024-09-13T15:07:28.627 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.627 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.627 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:28.627 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.629497+0000", 2024-09-13T15:07:28.627 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00011070599999999999 2024-09-13T15:07:28.627 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.627 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:28.627 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.628 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.628 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.628 INFO:teuthology.orchestra.run.smithi184.stdout: "description": "osd_op(client.36666.0:236171 11.4 11:2bea7180:::benchmark_data_smithi151_323728_object114874:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1354)", 2024-09-13T15:07:28.628 INFO:teuthology.orchestra.run.smithi184.stdout: "initiated_at": "2024-09-13T15:06:07.412547+0000", 2024-09-13T15:07:28.628 INFO:teuthology.orchestra.run.smithi184.stdout: "age": 81.074509497999998, 2024-09-13T15:07:28.628 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.21711403900000001, 2024-09-13T15:07:28.628 INFO:teuthology.orchestra.run.smithi184.stdout: "type_data": { 2024-09-13T15:07:28.628 INFO:teuthology.orchestra.run.smithi184.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:28.628 INFO:teuthology.orchestra.run.smithi184.stdout: "client_info": { 2024-09-13T15:07:28.628 INFO:teuthology.orchestra.run.smithi184.stdout: "client": "client.36666", 2024-09-13T15:07:28.628 INFO:teuthology.orchestra.run.smithi184.stdout: "client_addr": "172.21.15.151:0/3950305307", 2024-09-13T15:07:28.628 INFO:teuthology.orchestra.run.smithi184.stdout: "tid": 236171 2024-09-13T15:07:28.628 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.629 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2024-09-13T15:07:28.629 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.629 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "initiated", 2024-09-13T15:07:28.629 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.412547+0000", 2024-09-13T15:07:28.629 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.629 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.629 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.629 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "throttled", 2024-09-13T15:07:28.629 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.412547+0000", 2024-09-13T15:07:28.629 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0 2024-09-13T15:07:28.629 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.629 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.629 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "header_read", 2024-09-13T15:07:28.629 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.412546+0000", 2024-09-13T15:07:28.630 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4294967295.999999 2024-09-13T15:07:28.630 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.630 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.630 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "all_read", 2024-09-13T15:07:28.630 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.412551+0000", 2024-09-13T15:07:28.630 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.2399999999999998e-06 2024-09-13T15:07:28.630 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.630 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.630 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "dispatched", 2024-09-13T15:07:28.630 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.412565+0000", 2024-09-13T15:07:28.630 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 1.3516000000000001e-05 2024-09-13T15:07:28.630 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.630 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.631 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "queued_for_pg", 2024-09-13T15:07:28.631 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.412571+0000", 2024-09-13T15:07:28.631 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 5.7599999999999999e-06 2024-09-13T15:07:28.631 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.631 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.631 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "reached_pg", 2024-09-13T15:07:28.631 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.414602+0000", 2024-09-13T15:07:28.631 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0020317690000000001 2024-09-13T15:07:28.631 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.631 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.631 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "started", 2024-09-13T15:07:28.631 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.414683+0000", 2024-09-13T15:07:28.631 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0656999999999997e-05 2024-09-13T15:07:28.632 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.632 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.632 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "waiting for subops from 1,2,6", 2024-09-13T15:07:28.632 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.414760+0000", 2024-09-13T15:07:28.632 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 7.7291000000000002e-05 2024-09-13T15:07:28.632 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.632 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.632 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.632 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.417018+0000", 2024-09-13T15:07:28.632 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.0022580759999999999 2024-09-13T15:07:28.632 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.632 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.632 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "op_commit", 2024-09-13T15:07:28.633 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.417028+0000", 2024-09-13T15:07:28.633 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 9.482e-06 2024-09-13T15:07:28.633 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.633 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.633 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.633 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.417240+0000", 2024-09-13T15:07:28.633 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.00021168200000000001 2024-09-13T15:07:28.633 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.633 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.633 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:28.633 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.629537+0000", 2024-09-13T15:07:28.633 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 0.21229753900000001 2024-09-13T15:07:28.633 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.633 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.634 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "commit_sent", 2024-09-13T15:07:28.634 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.629581+0000", 2024-09-13T15:07:28.634 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 4.3754000000000002e-05 2024-09-13T15:07:28.634 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-13T15:07:28.634 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-13T15:07:28.634 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "done", 2024-09-13T15:07:28.634 INFO:teuthology.orchestra.run.smithi184.stdout: "time": "2024-09-13T15:06:07.629661+0000", 2024-09-13T15:07:28.634 INFO:teuthology.orchestra.run.smithi184.stdout: "duration": 8.0352999999999995e-05 2024-09-13T15:07:28.634 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.634 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:28.634 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.634 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-13T15:07:28.634 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-13T15:07:28.634 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-13T15:07:29.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:29 smithi184 ceph-mon[92527]: pgmap v2423: 103 pgs: 103 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 37 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:07:29.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:29 smithi151 ceph-mon[104450]: pgmap v2423: 103 pgs: 103 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 37 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:07:29.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:29 smithi151 ceph-mon[108966]: pgmap v2423: 103 pgs: 103 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 37 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:07:30.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:07:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:07:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:07:31.053 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-13T15:07:31.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:31 smithi184 ceph-mon[92527]: pgmap v2424: 103 pgs: 103 active+clean; 582 KiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 32 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:07:31.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:31 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/572165353' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:07:31.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:31 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:07:31.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:31 smithi151 ceph-mon[104450]: pgmap v2424: 103 pgs: 103 active+clean; 582 KiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 32 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:07:31.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:31 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/572165353' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:07:31.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:31 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:07:31.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:31 smithi151 ceph-mon[108966]: pgmap v2424: 103 pgs: 103 active+clean; 582 KiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 32 KiB/s, 1 keys/s, 0 objects/s recovering 2024-09-13T15:07:31.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:31 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/572165353' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:07:31.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:31 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:07:31.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T15:07:32.101 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:32.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:32 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-13T15:07:32.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:32 smithi151 ceph-mon[104450]: osdmap e1377: 8 total, 8 up, 8 in 2024-09-13T15:07:32.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:32.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:32 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-13T15:07:32.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:32 smithi151 ceph-mon[108966]: osdmap e1377: 8 total, 8 up, 8 in 2024-09-13T15:07:32.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:32 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-13T15:07:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:32 smithi184 ceph-mon[92527]: osdmap e1377: 8 total, 8 up, 8 in 2024-09-13T15:07:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:33.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:33 smithi184 ceph-mon[92527]: pgmap v2426: 103 pgs: 103 active+clean; 582 KiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 71 B/s, 0 objects/s recovering 2024-09-13T15:07:33.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:33 smithi151 ceph-mon[104450]: pgmap v2426: 103 pgs: 103 active+clean; 582 KiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 71 B/s, 0 objects/s recovering 2024-09-13T15:07:33.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:33 smithi151 ceph-mon[108966]: pgmap v2426: 103 pgs: 103 active+clean; 582 KiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 71 B/s, 0 objects/s recovering 2024-09-13T15:07:34.609 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:07:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:07:34] "GET /metrics HTTP/1.1" 200 34316 "" "Prometheus/2.51.0" 2024-09-13T15:07:35.071 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T15:07:35.321 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:35 smithi151 ceph-mon[104450]: pgmap v2427: 103 pgs: 103 active+clean; 582 KiB data, 9.7 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:35.322 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:35 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2706233150' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:07:35.322 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:35 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:07:35.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:35 smithi151 ceph-mon[108966]: pgmap v2427: 103 pgs: 103 active+clean; 582 KiB data, 9.7 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:35.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:35 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2706233150' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:07:35.322 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:35 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:07:35.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:35 smithi184 ceph-mon[92527]: pgmap v2427: 103 pgs: 103 active+clean; 582 KiB data, 9.7 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:35.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:35 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2706233150' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:07:35.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:35 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:07:35.816 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 1 1 2024-09-13T15:07:36.068 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:36.339 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:36 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:07:36.340 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:36 smithi151 ceph-mon[104450]: osdmap e1378: 8 total, 8 up, 8 in 2024-09-13T15:07:36.340 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:36 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:07:36.340 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:36 smithi151 ceph-mon[108966]: osdmap e1378: 8 total, 8 up, 8 in 2024-09-13T15:07:36.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:36 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:07:36.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:36 smithi184 ceph-mon[92527]: osdmap e1378: 8 total, 8 up, 8 in 2024-09-13T15:07:37.289 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:37 smithi151 ceph-mon[108966]: pgmap v2429: 103 pgs: 103 active+clean; 582 KiB data, 9.7 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:37.290 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:37 smithi151 ceph-mon[104450]: pgmap v2429: 103 pgs: 103 active+clean; 582 KiB data, 9.7 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:37.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:37 smithi184 ceph-mon[92527]: pgmap v2429: 103 pgs: 103 active+clean; 582 KiB data, 9.7 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:38.079 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-13T15:07:38.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:38 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3803688444' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:07:38.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:38 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3803688444' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:07:38.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:38 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3803688444' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:07:38.882 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T15:07:39.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:39 smithi151 ceph-mon[104450]: pgmap v2430: 103 pgs: 103 active+clean; 582 KiB data, 9.7 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:39.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:39 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3803688444' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:07:39.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:39 smithi151 ceph-mon[104450]: osdmap e1379: 8 total, 8 up, 8 in 2024-09-13T15:07:39.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:39 smithi151 ceph-mon[108966]: pgmap v2430: 103 pgs: 103 active+clean; 582 KiB data, 9.7 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:39.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:39 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3803688444' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:07:39.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:39 smithi151 ceph-mon[108966]: osdmap e1379: 8 total, 8 up, 8 in 2024-09-13T15:07:39.131 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:39.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:39 smithi184 ceph-mon[92527]: pgmap v2430: 103 pgs: 103 active+clean; 582 KiB data, 9.7 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:39.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:39 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3803688444' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:07:39.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:39 smithi184 ceph-mon[92527]: osdmap e1379: 8 total, 8 up, 8 in 2024-09-13T15:07:40.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:07:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:07:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:07:41.141 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T15:07:41.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:41 smithi151 ceph-mon[108966]: pgmap v2432: 103 pgs: 103 active+clean; 582 KiB data, 9.7 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:41.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:41 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/4152175027' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:07:41.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:41 smithi151 ceph-mon[104450]: pgmap v2432: 103 pgs: 103 active+clean; 582 KiB data, 9.7 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:41.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:41 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/4152175027' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:07:41.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:41 smithi184 ceph-mon[92527]: pgmap v2432: 103 pgs: 103 active+clean; 582 KiB data, 9.7 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:41.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:41 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/4152175027' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:07:41.999 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 2 1 2024-09-13T15:07:42.247 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:42.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:42 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/4152175027' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:07:42.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:42 smithi151 ceph-mon[104450]: osdmap e1380: 8 total, 8 up, 8 in 2024-09-13T15:07:42.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:42 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/4152175027' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:07:42.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:42 smithi151 ceph-mon[108966]: osdmap e1380: 8 total, 8 up, 8 in 2024-09-13T15:07:42.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:42 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/4152175027' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:07:42.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:42 smithi184 ceph-mon[92527]: osdmap e1380: 8 total, 8 up, 8 in 2024-09-13T15:07:43.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:43 smithi151 ceph-mon[104450]: pgmap v2434: 103 pgs: 103 active+clean; 582 KiB data, 9.7 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:43.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:43 smithi151 ceph-mon[108966]: pgmap v2434: 103 pgs: 103 active+clean; 582 KiB data, 9.7 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:43.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:43 smithi184 ceph-mon[92527]: pgmap v2434: 103 pgs: 103 active+clean; 582 KiB data, 9.7 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:44.160 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-13T15:07:44.464 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:44 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/957866036' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:07:44.464 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:44 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/957866036' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:07:44.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:44 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/957866036' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:07:44.822 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:07:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:07:44] "GET /metrics HTTP/1.1" 200 34316 "" "Prometheus/2.51.0" 2024-09-13T15:07:44.905 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T15:07:45.156 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:45.458 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:45 smithi151 ceph-mon[104450]: pgmap v2435: 103 pgs: 103 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:45.459 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:45 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/957866036' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:07:45.459 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:45 smithi151 ceph-mon[104450]: osdmap e1381: 8 total, 8 up, 8 in 2024-09-13T15:07:45.459 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:45 smithi151 ceph-mon[108966]: pgmap v2435: 103 pgs: 103 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:45.459 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:45 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/957866036' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:07:45.459 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:45 smithi151 ceph-mon[108966]: osdmap e1381: 8 total, 8 up, 8 in 2024-09-13T15:07:45.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:45 smithi184 ceph-mon[92527]: pgmap v2435: 103 pgs: 103 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:45.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:45 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/957866036' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:07:45.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:45 smithi184 ceph-mon[92527]: osdmap e1381: 8 total, 8 up, 8 in 2024-09-13T15:07:47.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:47 smithi151 ceph-mon[104450]: pgmap v2437: 103 pgs: 103 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:47.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:47 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:47.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:47 smithi151 ceph-mon[108966]: pgmap v2437: 103 pgs: 103 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:47.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:47 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:47.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:47 smithi184 ceph-mon[92527]: pgmap v2437: 103 pgs: 103 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:47.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:47 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:48.169 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T15:07:48.546 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:48 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2396099808' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:07:48.547 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:48 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2396099808' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:07:48.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:48 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2396099808' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:07:48.960 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 3 1 2024-09-13T15:07:49.213 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:49.334 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:49 smithi151 ceph-mon[104450]: pgmap v2438: 103 pgs: 103 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:49.334 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:49 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2396099808' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:07:49.335 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:49 smithi151 ceph-mon[104450]: osdmap e1382: 8 total, 8 up, 8 in 2024-09-13T15:07:49.335 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:49 smithi151 ceph-mon[108966]: pgmap v2438: 103 pgs: 103 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:49.335 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:49 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2396099808' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:07:49.335 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:49 smithi151 ceph-mon[108966]: osdmap e1382: 8 total, 8 up, 8 in 2024-09-13T15:07:49.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:49 smithi184 ceph-mon[92527]: pgmap v2438: 103 pgs: 103 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:49.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:49 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2396099808' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:07:49.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:49 smithi184 ceph-mon[92527]: osdmap e1382: 8 total, 8 up, 8 in 2024-09-13T15:07:50.506 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:07:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:07:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:07:51.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:51 smithi184 ceph-mon[92527]: pgmap v2440: 103 pgs: 103 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:51.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:51 smithi151 ceph-mon[104450]: pgmap v2440: 103 pgs: 103 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:51.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:51 smithi151 ceph-mon[108966]: pgmap v2440: 103 pgs: 103 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:52.180 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-13T15:07:52.546 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:52 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/352646763' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:07:52.546 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:52 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/352646763' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:07:52.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:52 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/352646763' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:07:53.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-13T15:07:53.255 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:53.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:53 smithi151 ceph-mon[104450]: pgmap v2441: 103 pgs: 103 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:53.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:53 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/352646763' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:07:53.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:53 smithi151 ceph-mon[104450]: osdmap e1383: 8 total, 8 up, 8 in 2024-09-13T15:07:53.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:53 smithi151 ceph-mon[108966]: pgmap v2441: 103 pgs: 103 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:53.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:53 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/352646763' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:07:53.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:53 smithi151 ceph-mon[108966]: osdmap e1383: 8 total, 8 up, 8 in 2024-09-13T15:07:53.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:53 smithi184 ceph-mon[92527]: pgmap v2441: 103 pgs: 103 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:53.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:53 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/352646763' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:07:53.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:53 smithi184 ceph-mon[92527]: osdmap e1383: 8 total, 8 up, 8 in 2024-09-13T15:07:54.817 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:07:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:07:54] "GET /metrics HTTP/1.1" 200 34295 "" "Prometheus/2.51.0" 2024-09-13T15:07:55.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:55 smithi184 ceph-mon[92527]: pgmap v2443: 103 pgs: 103 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:55.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:55 smithi151 ceph-mon[104450]: pgmap v2443: 103 pgs: 103 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:55.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:55 smithi151 ceph-mon[108966]: pgmap v2443: 103 pgs: 103 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:56.195 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T15:07:56.537 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:56 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/821891139' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:07:56.538 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:56 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:07:56.538 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:56 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/821891139' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:07:56.538 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:56 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:07:56.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:56 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/821891139' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:07:56.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:56 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:07:56.918 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 4 1 2024-09-13T15:07:57.170 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:57.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:57 smithi151 ceph-mon[104450]: pgmap v2444: 103 pgs: 103 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:57.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:57 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:07:57.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:57 smithi151 ceph-mon[104450]: osdmap e1384: 8 total, 8 up, 8 in 2024-09-13T15:07:57.450 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:57 smithi151 ceph-mon[108966]: pgmap v2444: 103 pgs: 103 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:57.450 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:57 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:07:57.450 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:57 smithi151 ceph-mon[108966]: osdmap e1384: 8 total, 8 up, 8 in 2024-09-13T15:07:57.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:57 smithi184 ceph-mon[92527]: pgmap v2444: 103 pgs: 103 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:57.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:57 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:07:57.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:57 smithi184 ceph-mon[92527]: osdmap e1384: 8 total, 8 up, 8 in 2024-09-13T15:07:59.199 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-13T15:07:59.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:59 smithi151 ceph-mon[108966]: pgmap v2446: 103 pgs: 103 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:59.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:59 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/667769594' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:07:59.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:07:59 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:07:59.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:59 smithi151 ceph-mon[104450]: pgmap v2446: 103 pgs: 103 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:59.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:59 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/667769594' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:07:59.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:07:59 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:07:59.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:59 smithi184 ceph-mon[92527]: pgmap v2446: 103 pgs: 103 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:07:59.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:59 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/667769594' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:07:59.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:07:59 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:07:59.970 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T15:08:00.215 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:00.468 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:00 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:08:00.468 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:00 smithi151 ceph-mon[104450]: osdmap e1385: 8 total, 8 up, 8 in 2024-09-13T15:08:00.468 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:00 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:08:00.468 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:00 smithi151 ceph-mon[108966]: osdmap e1385: 8 total, 8 up, 8 in 2024-09-13T15:08:00.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:00 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:08:00.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:00 smithi184 ceph-mon[92527]: osdmap e1385: 8 total, 8 up, 8 in 2024-09-13T15:08:00.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:08:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:08:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:08:01.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:01 smithi151 ceph-mon[104450]: pgmap v2448: 103 pgs: 103 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:01.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:01 smithi151 ceph-mon[108966]: pgmap v2448: 103 pgs: 103 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:01.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:01 smithi184 ceph-mon[92527]: pgmap v2448: 103 pgs: 103 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:02.215 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T15:08:02.593 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:02.593 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:02 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2928623198' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:08:02.593 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:02.593 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:02 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2928623198' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:08:02.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:02.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:02 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2928623198' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:08:03.037 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 5 1 2024-09-13T15:08:03.285 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:03.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:03 smithi151 ceph-mon[108966]: pgmap v2449: 103 pgs: 103 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:03.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:03 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2928623198' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:08:03.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:03 smithi151 ceph-mon[108966]: osdmap e1386: 8 total, 8 up, 8 in 2024-09-13T15:08:03.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:03 smithi151 ceph-mon[104450]: pgmap v2449: 103 pgs: 103 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:03.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:03 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2928623198' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:08:03.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:03 smithi151 ceph-mon[104450]: osdmap e1386: 8 total, 8 up, 8 in 2024-09-13T15:08:03.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:03 smithi184 ceph-mon[92527]: pgmap v2449: 103 pgs: 103 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:03.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:03 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2928623198' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:08:03.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:03 smithi184 ceph-mon[92527]: osdmap e1386: 8 total, 8 up, 8 in 2024-09-13T15:08:04.528 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:04 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:08:04.528 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:04 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:08:04.528 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:04 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:08:04.535 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:04 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:08:04.536 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:04 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:08:04.536 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:04 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:08:04.536 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:04 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:08:04.536 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:04 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:08:04.536 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:04 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:08:04.817 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:08:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:08:04] "GET /metrics HTTP/1.1" 200 34300 "" "Prometheus/2.51.0" 2024-09-13T15:08:05.491 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:05 smithi184 ceph-mon[92527]: pgmap v2451: 103 pgs: 103 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:05.494 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:05 smithi151 ceph-mon[104450]: pgmap v2451: 103 pgs: 103 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:05.494 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:05 smithi151 ceph-mon[108966]: pgmap v2451: 103 pgs: 103 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:06.233 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-13T15:08:06.471 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:06 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2309245442' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:08:06.471 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:06 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:08:06.471 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:06 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2309245442' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:08:06.471 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:06 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:08:06.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:06 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2309245442' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:08:06.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:06 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:08:07.141 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T15:08:07.502 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:07.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:07 smithi184 ceph-mon[92527]: pgmap v2452: 103 pgs: 103 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:07.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:07 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:08:07.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:07 smithi184 ceph-mon[92527]: osdmap e1387: 8 total, 8 up, 8 in 2024-09-13T15:08:07.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:07 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]} : dispatch 2024-09-13T15:08:07.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:07 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]} : dispatch 2024-09-13T15:08:07.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:07 smithi151 ceph-mon[104450]: pgmap v2452: 103 pgs: 103 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:07.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:07 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:08:07.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:07 smithi151 ceph-mon[104450]: osdmap e1387: 8 total, 8 up, 8 in 2024-09-13T15:08:07.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:07 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]} : dispatch 2024-09-13T15:08:07.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:07 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]} : dispatch 2024-09-13T15:08:07.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:07 smithi151 ceph-mon[108966]: pgmap v2452: 103 pgs: 103 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:07.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:07 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:08:07.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:07 smithi151 ceph-mon[108966]: osdmap e1387: 8 total, 8 up, 8 in 2024-09-13T15:08:07.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:07 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]} : dispatch 2024-09-13T15:08:07.610 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:07 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]} : dispatch 2024-09-13T15:08:08.542 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:08 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-09-13T15:08:08.543 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:08 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-13T15:08:08.543 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:08 smithi151 ceph-mon[104450]: osdmap e1388: 8 total, 8 up, 8 in 2024-09-13T15:08:08.543 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:08 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-09-13T15:08:08.543 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:08 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-13T15:08:08.543 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:08 smithi151 ceph-mon[108966]: osdmap e1388: 8 total, 8 up, 8 in 2024-09-13T15:08:08.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:08 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-09-13T15:08:08.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:08 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-13T15:08:08.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:08 smithi184 ceph-mon[92527]: osdmap e1388: 8 total, 8 up, 8 in 2024-09-13T15:08:09.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:09 smithi184 ceph-mon[92527]: pgmap v2455: 103 pgs: 103 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:09.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:09 smithi184 ceph-mon[92527]: osdmap e1389: 8 total, 8 up, 8 in 2024-09-13T15:08:09.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:09 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:08:09.605 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:09 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:08:09.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:09 smithi151 ceph-mon[104450]: pgmap v2455: 103 pgs: 103 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:09.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:09 smithi151 ceph-mon[104450]: osdmap e1389: 8 total, 8 up, 8 in 2024-09-13T15:08:09.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:09 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:08:09.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:09 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:08:09.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:09 smithi151 ceph-mon[108966]: pgmap v2455: 103 pgs: 103 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:09.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:09 smithi151 ceph-mon[108966]: osdmap e1389: 8 total, 8 up, 8 in 2024-09-13T15:08:09.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:09 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:08:09.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:09 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:08:10.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:08:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:08:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:08:10.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:10 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:08:10.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:10 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:08:10.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:10 smithi151 ceph-mon[104450]: pgmap v2457: 103 pgs: 103 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:10.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:10 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:08:10.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:10 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:08:10.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:10 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:08:10.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:10 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:08:10.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:10 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:08:10.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:10 smithi151 ceph-mon[108966]: pgmap v2457: 103 pgs: 103 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:10.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:10 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:08:10.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:10 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:08:10.860 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:10 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:08:11.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:10 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:08:11.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:10 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:08:11.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:10 smithi184 ceph-mon[92527]: pgmap v2457: 103 pgs: 103 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:11.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:10 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:08:11.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:10 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:08:11.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:10 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' 2024-09-13T15:08:11.739 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T15:08:12.101 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:11 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2418610268' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:08:12.102 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:11 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2418610268' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:08:12.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:11 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2418610268' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:08:12.616 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 6 1 2024-09-13T15:08:12.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:12 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/2418610268' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:08:12.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:12 smithi151 ceph-mon[104450]: osdmap e1390: 8 total, 8 up, 8 in 2024-09-13T15:08:12.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:12 smithi151 ceph-mon[104450]: pgmap v2459: 103 pgs: 103 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:12.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:12 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/2418610268' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:08:12.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:12 smithi151 ceph-mon[108966]: osdmap e1390: 8 total, 8 up, 8 in 2024-09-13T15:08:12.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:12 smithi151 ceph-mon[108966]: pgmap v2459: 103 pgs: 103 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:12.870 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:13.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:12 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/2418610268' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:08:13.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:12 smithi184 ceph-mon[92527]: osdmap e1390: 8 total, 8 up, 8 in 2024-09-13T15:08:13.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:12 smithi184 ceph-mon[92527]: pgmap v2459: 103 pgs: 103 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:14.609 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:08:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:08:14] "GET /metrics HTTP/1.1" 200 34300 "" "Prometheus/2.51.0" 2024-09-13T15:08:15.037 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-13T15:08:15.343 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:15 smithi151 ceph-mon[104450]: pgmap v2460: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:15.343 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:15 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1232599381' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:08:15.343 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:15 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:08:15.343 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:15 smithi151 ceph-mon[108966]: pgmap v2460: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:15.343 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:15 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1232599381' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:08:15.344 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:15 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:08:15.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:15 smithi184 ceph-mon[92527]: pgmap v2460: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:15.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:15 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1232599381' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:08:15.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:15 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:08:15.770 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T15:08:16.020 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:16.307 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:16 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:08:16.307 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:16 smithi151 ceph-mon[104450]: osdmap e1391: 8 total, 8 up, 8 in 2024-09-13T15:08:16.307 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:16 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:08:16.307 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:16 smithi151 ceph-mon[108966]: osdmap e1391: 8 total, 8 up, 8 in 2024-09-13T15:08:16.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:16 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:08:16.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:16 smithi184 ceph-mon[92527]: osdmap e1391: 8 total, 8 up, 8 in 2024-09-13T15:08:17.218 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:17 smithi151 ceph-mon[104450]: pgmap v2462: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:17.218 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:17 smithi151 ceph-mon[104450]: osdmap e1392: 8 total, 8 up, 8 in 2024-09-13T15:08:17.218 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:17 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:17.219 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:17 smithi151 ceph-mon[108966]: pgmap v2462: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:17.219 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:17 smithi151 ceph-mon[108966]: osdmap e1392: 8 total, 8 up, 8 in 2024-09-13T15:08:17.219 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:17 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:17.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:17 smithi184 ceph-mon[92527]: pgmap v2462: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:17.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:17 smithi184 ceph-mon[92527]: osdmap e1392: 8 total, 8 up, 8 in 2024-09-13T15:08:17.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:17 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:18.063 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T15:08:18.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:18 smithi184 ceph-mon[92527]: 1.7 scrub starts 2024-09-13T15:08:18.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:18 smithi184 ceph-mon[92527]: 1.7 scrub ok 2024-09-13T15:08:18.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:18 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/496264763' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:08:18.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:18 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:08:18.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:18 smithi151 ceph-mon[108966]: 1.7 scrub starts 2024-09-13T15:08:18.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:18 smithi151 ceph-mon[108966]: 1.7 scrub ok 2024-09-13T15:08:18.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:18 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/496264763' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:08:18.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:18 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:08:18.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:18 smithi151 ceph-mon[104450]: 1.7 scrub starts 2024-09-13T15:08:18.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:18 smithi151 ceph-mon[104450]: 1.7 scrub ok 2024-09-13T15:08:18.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:18 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/496264763' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:08:18.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:18 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:08:18.736 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 7 1 2024-09-13T15:08:18.982 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:19.059 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:19 smithi151 ceph-mon[104450]: pgmap v2464: 103 pgs: 103 active+clean; 608 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 5.4 KiB/s rd, 12 KiB/s wr, 3 op/s; 4.2 KiB/s, 0 objects/s recovering 2024-09-13T15:08:19.059 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:19 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:08:19.059 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:19 smithi151 ceph-mon[104450]: osdmap e1393: 8 total, 8 up, 8 in 2024-09-13T15:08:19.321 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:19 smithi151 ceph-mon[108966]: pgmap v2464: 103 pgs: 103 active+clean; 608 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 5.4 KiB/s rd, 12 KiB/s wr, 3 op/s; 4.2 KiB/s, 0 objects/s recovering 2024-09-13T15:08:19.321 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:19 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:08:19.321 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:19 smithi151 ceph-mon[108966]: osdmap e1393: 8 total, 8 up, 8 in 2024-09-13T15:08:19.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:19 smithi184 ceph-mon[92527]: pgmap v2464: 103 pgs: 103 active+clean; 608 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 5.4 KiB/s rd, 12 KiB/s wr, 3 op/s; 4.2 KiB/s, 0 objects/s recovering 2024-09-13T15:08:19.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:19 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:08:19.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:19 smithi184 ceph-mon[92527]: osdmap e1393: 8 total, 8 up, 8 in 2024-09-13T15:08:20.527 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:08:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:08:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:08:21.069 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-13T15:08:21.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:21 smithi184 ceph-mon[92527]: pgmap v2466: 103 pgs: 103 active+clean; 608 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 5.7 KiB/s rd, 12 KiB/s wr, 3 op/s; 4.4 KiB/s, 0 objects/s recovering 2024-09-13T15:08:21.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:21 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1868945933' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:08:21.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:21 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:08:21.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:21 smithi151 ceph-mon[104450]: pgmap v2466: 103 pgs: 103 active+clean; 608 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 5.7 KiB/s rd, 12 KiB/s wr, 3 op/s; 4.4 KiB/s, 0 objects/s recovering 2024-09-13T15:08:21.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:21 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1868945933' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:08:21.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:21 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:08:21.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:21 smithi151 ceph-mon[108966]: pgmap v2466: 103 pgs: 103 active+clean; 608 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 5.7 KiB/s rd, 12 KiB/s wr, 3 op/s; 4.4 KiB/s, 0 objects/s recovering 2024-09-13T15:08:21.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:21 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1868945933' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:08:21.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:21 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:08:21.842 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-13T15:08:21.842 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-13T15:08:21.842 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-13T15:08:21.843 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-13T15:08:21.843 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-13T15:08:21.843 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T15:08:22.093 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:22.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:22 smithi184 ceph-mon[92527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:08:22.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:22 smithi184 ceph-mon[92527]: osdmap e1394: 8 total, 8 up, 8 in 2024-09-13T15:08:22.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:22 smithi151 ceph-mon[104450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:08:22.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:22 smithi151 ceph-mon[104450]: osdmap e1394: 8 total, 8 up, 8 in 2024-09-13T15:08:22.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:22 smithi151 ceph-mon[108966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:08:22.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:22 smithi151 ceph-mon[108966]: osdmap e1394: 8 total, 8 up, 8 in 2024-09-13T15:08:23.231 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:23 smithi151 ceph-mon[104450]: pgmap v2468: 103 pgs: 103 active+clean; 608 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 5.7 KiB/s rd, 12 KiB/s wr, 3 op/s; 4.4 KiB/s, 0 objects/s recovering 2024-09-13T15:08:23.232 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:23 smithi151 ceph-mon[108966]: pgmap v2468: 103 pgs: 103 active+clean; 608 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 5.7 KiB/s rd, 12 KiB/s wr, 3 op/s; 4.4 KiB/s, 0 objects/s recovering 2024-09-13T15:08:23.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:23 smithi184 ceph-mon[92527]: pgmap v2468: 103 pgs: 103 active+clean; 608 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 5.7 KiB/s rd, 12 KiB/s wr, 3 op/s; 4.4 KiB/s, 0 objects/s recovering 2024-09-13T15:08:23.949 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T15:08:23.949 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":1394,"fsid":"2401fc50-71da-11ef-bceb-c7b262605968","created":"2024-09-13T14:12:36.168205+0000","modified":"2024-09-13T15:08:21.059478+0000","last_up_change":"2024-09-13T14:30:14.547371+0000","last_in_change":"2024-09-13T15:07:20.987934+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:15:41.780748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"1128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"1121","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.059999942779541,"score_stable":2.059999942779541,"optimal_score":1,"raw_score_acting":2.059999942779541,"raw_score_stable":2.059999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:18:09.030957+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":56,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T14:19:24.791384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T14:19:25.101452+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0ad945ee-af7d-4f26-a1e9-18ed963194ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":511,"up_from":524,"up_thru":1374,"down_at":512,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2137494296},{"type":"v1","addr":"172.21.15.151:6803","nonce":2137494296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2137494296},{"type":"v1","addr":"172.21.15.151:6805","nonce":2137494296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":2137494296},{"type":"v1","addr":"172.21.15.151:6809","nonce":2137494296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2137494296},{"type":"v1","addr":"172.21.15.151:6807","nonce":2137494296}]},"public_addr":"172.21.15.151:6803/2137494296","cluster_addr":"172.21.15.151:6805/2137494296","heartbeat_back_addr":"172.21.15.151:6809/2137494296","heartbeat_front_addr":"172.21.15.151:6807/2137494296","state":["exists","up"]},{"osd":1,"uuid":"09f5d705-684f-4e86-80a2-ec2292af09d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":544,"up_from":557,"up_thru":1374,"down_at":545,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":3884881052},{"type":"v1","addr":"172.21.15.151:6811","nonce":3884881052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3884881052},{"type":"v1","addr":"172.21.15.151:6813","nonce":3884881052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3884881052},{"type":"v1","addr":"172.21.15.151:6817","nonce":3884881052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":3884881052},{"type":"v1","addr":"172.21.15.151:6815","nonce":3884881052}]},"public_addr":"172.21.15.151:6811/3884881052","cluster_addr":"172.21.15.151:6813/3884881052","heartbeat_back_addr":"172.21.15.151:6817/3884881052","heartbeat_front_addr":"172.21.15.151:6815/3884881052","state":["exists","up"]},{"osd":2,"uuid":"6a63f8af-ff69-46bb-830f-03ea89fc78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":578,"up_from":591,"up_thru":1388,"down_at":579,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":322435630},{"type":"v1","addr":"172.21.15.151:6819","nonce":322435630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":322435630},{"type":"v1","addr":"172.21.15.151:6821","nonce":322435630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":322435630},{"type":"v1","addr":"172.21.15.151:6825","nonce":322435630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":322435630},{"type":"v1","addr":"172.21.15.151:6823","nonce":322435630}]},"public_addr":"172.21.15.151:6819/322435630","cluster_addr":"172.21.15.151:6821/322435630","heartbeat_back_addr":"172.21.15.151:6825/322435630","heartbeat_front_addr":"172.21.15.151:6823/322435630","state":["exists","up"]},{"osd":3,"uuid":"770e8334-3b3a-46bc-9782-964b2a696660","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":611,"up_from":624,"up_thru":1388,"down_at":612,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2997668861},{"type":"v1","addr":"172.21.15.151:6827","nonce":2997668861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":2997668861},{"type":"v1","addr":"172.21.15.151:6829","nonce":2997668861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":2997668861},{"type":"v1","addr":"172.21.15.151:6833","nonce":2997668861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":2997668861},{"type":"v1","addr":"172.21.15.151:6831","nonce":2997668861}]},"public_addr":"172.21.15.151:6827/2997668861","cluster_addr":"172.21.15.151:6829/2997668861","heartbeat_back_addr":"172.21.15.151:6833/2997668861","heartbeat_front_addr":"172.21.15.151:6831/2997668861","state":["exists","up"]},{"osd":4,"uuid":"75e258c0-1145-4aee-86aa-db2b63082cc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":1374,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6801","nonce":3275553935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6803","nonce":3275553935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6807","nonce":3275553935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6805","nonce":3275553935}]},"public_addr":"172.21.15.184:6801/3275553935","cluster_addr":"172.21.15.184:6803/3275553935","heartbeat_back_addr":"172.21.15.184:6807/3275553935","heartbeat_front_addr":"172.21.15.184:6805/3275553935","state":["exists","up"]},{"osd":5,"uuid":"f56a5366-9d22-4960-933d-340aa280fbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":1374,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6809","nonce":2446419621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6811","nonce":2446419621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6815","nonce":2446419621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6813","nonce":2446419621}]},"public_addr":"172.21.15.184:6809/2446419621","cluster_addr":"172.21.15.184:6811/2446419621","heartbeat_back_addr":"172.21.15.184:6815/2446419621","heartbeat_front_addr":"172.21.15.184:6813/2446419621","state":["exists","up"]},{"osd":6,"uuid":"e9826568-a728-45ab-8180-9777fe31f93a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":1391,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6817","nonce":2214837203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6819","nonce":2214837203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6823","nonce":2214837203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6821","nonce":2214837203}]},"public_addr":"172.21.15.184:6817/2214837203","cluster_addr":"172.21.15.184:6819/2214837203","heartbeat_back_addr":"172.21.15.184:6823/2214837203","heartbeat_front_addr":"172.21.15.184:6821/2214837203","state":["exists","up"]},{"osd":7,"uuid":"d76869fe-5121-4467-8b56-1aadef0eaf65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":1374,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6825","nonce":3378735997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6827","nonce":3378735997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6831","nonce":3378735997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6829","nonce":3378735997}]},"public_addr":"172.21.15.184:6825/3378735997","cluster_addr":"172.21.15.184:6827/3378735997","heartbeat_back_addr":"172.21.15.184:6831/3378735997","heartbeat_front_addr":"172.21.15.184:6829/3378735997","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:14:49.835715+0000","dead_epoch":511},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:15:14.682882+0000","dead_epoch":544},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:15:39.423890+0000","dead_epoch":578},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:16:04.527460+0000","dead_epoch":611},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:16:27.283661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:16:51.893868+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:16.152592+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:39.763533+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":3,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":0}]},{"pgid":"4.1b","mappings":[{"from":3,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.151:0/1783031268":"2024-09-14T14:24:01.073069+0000","172.21.15.151:0/1775071943":"2024-09-14T14:24:01.073069+0000","172.21.15.151:6801/42840":"2024-09-14T14:24:01.073069+0000","172.21.15.151:0/841256485":"2024-09-14T14:24:01.073069+0000","172.21.15.151:0/1182681341":"2024-09-14T14:13:04.761383+0000","172.21.15.151:6800/1057983082":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/2889050140":"2024-09-14T14:18:18.344788+0000","172.21.15.151:6801/4025698662":"2024-09-14T14:22:15.389006+0000","172.21.15.151:0/2800235450":"2024-09-14T14:18:18.344788+0000","172.21.15.184:0/3929228938":"2024-09-14T14:23:20.772174+0000","172.21.15.151:0/1409232061":"2024-09-14T14:24:01.073069+0000","172.21.15.151:6800/4025698662":"2024-09-14T14:22:15.389006+0000","172.21.15.184:0/1999058007":"2024-09-14T14:23:20.772174+0000","172.21.15.151:6801/1057983082":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/514781427":"2024-09-14T14:22:15.389006+0000","172.21.15.151:0/663506338":"2024-09-14T14:24:01.073069+0000","172.21.15.151:6800/3383065173":"2024-09-14T14:13:04.761383+0000","172.21.15.184:6834/205203765":"2024-09-14T14:23:20.772174+0000","172.21.15.151:0/1530349337":"2024-09-14T14:22:15.389006+0000","172.21.15.151:0/1427922964":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/2666829207":"2024-09-14T14:13:27.478847+0000","172.21.15.151:6801/2335297718":"2024-09-14T14:18:18.344788+0000","172.21.15.151:0/1351186656":"2024-09-14T14:18:18.344788+0000","172.21.15.184:6835/205203765":"2024-09-14T14:23:20.772174+0000","172.21.15.184:0/3003411098":"2024-09-14T14:23:20.772174+0000","172.21.15.151:0/4115173791":"2024-09-14T14:18:18.344788+0000","172.21.15.151:0/3689756337":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/1517910843":"2024-09-14T14:24:01.073069+0000","172.21.15.184:0/3370242044":"2024-09-14T14:23:20.772174+0000","172.21.15.151:6800/2335297718":"2024-09-14T14:18:18.344788+0000","172.21.15.151:6800/42840":"2024-09-14T14:24:01.073069+0000","172.21.15.151:0/1488764319":"2024-09-14T14:22:15.389006+0000","172.21.15.184:0/2995983270":"2024-09-14T14:23:20.772174+0000","172.21.15.151:0/3946777511":"2024-09-14T14:13:04.761383+0000","172.21.15.151:0/624042464":"2024-09-14T14:13:04.761383+0000","172.21.15.151:0/2201280183":"2024-09-14T14:22:15.389006+0000","172.21.15.151:6801/3383065173":"2024-09-14T14:13:04.761383+0000","172.21.15.184:0/3047962606":"2024-09-14T14:23:20.772174+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":12,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T15:08:24.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:24 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/295155005' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T15:08:24.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:24 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/295155005' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T15:08:24.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:24 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/295155005' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T15:08:24.777 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-13T15:08:24.777 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T15:08:24.858 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:08:24 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:08:24] "GET /metrics HTTP/1.1" 200 34285 "" "Prometheus/2.51.0" 2024-09-13T15:08:25.029 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:25.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:25 smithi184 ceph-mon[92527]: pgmap v2469: 103 pgs: 103 active+clean; 608 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 4.3 KiB/s rd, 9.3 KiB/s wr, 2 op/s; 3.3 KiB/s, 0 objects/s recovering 2024-09-13T15:08:25.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:25 smithi184 ceph-mon[92527]: 1.f scrub starts 2024-09-13T15:08:25.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:25 smithi184 ceph-mon[92527]: 1.f scrub ok 2024-09-13T15:08:25.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:25 smithi151 ceph-mon[104450]: pgmap v2469: 103 pgs: 103 active+clean; 608 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 4.3 KiB/s rd, 9.3 KiB/s wr, 2 op/s; 3.3 KiB/s, 0 objects/s recovering 2024-09-13T15:08:25.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:25 smithi151 ceph-mon[104450]: 1.f scrub starts 2024-09-13T15:08:25.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:25 smithi151 ceph-mon[104450]: 1.f scrub ok 2024-09-13T15:08:25.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:25 smithi151 ceph-mon[108966]: pgmap v2469: 103 pgs: 103 active+clean; 608 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 4.3 KiB/s rd, 9.3 KiB/s wr, 2 op/s; 3.3 KiB/s, 0 objects/s recovering 2024-09-13T15:08:25.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:25 smithi151 ceph-mon[108966]: 1.f scrub starts 2024-09-13T15:08:25.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:25 smithi151 ceph-mon[108966]: 1.f scrub ok 2024-09-13T15:08:26.935 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T15:08:26.936 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":1394,"fsid":"2401fc50-71da-11ef-bceb-c7b262605968","created":"2024-09-13T14:12:36.168205+0000","modified":"2024-09-13T15:08:21.059478+0000","last_up_change":"2024-09-13T14:30:14.547371+0000","last_in_change":"2024-09-13T15:07:20.987934+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:15:41.780748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"1128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"1121","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.059999942779541,"score_stable":2.059999942779541,"optimal_score":1,"raw_score_acting":2.059999942779541,"raw_score_stable":2.059999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:18:09.030957+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":56,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T14:19:24.791384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T14:19:25.101452+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0ad945ee-af7d-4f26-a1e9-18ed963194ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":511,"up_from":524,"up_thru":1374,"down_at":512,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2137494296},{"type":"v1","addr":"172.21.15.151:6803","nonce":2137494296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2137494296},{"type":"v1","addr":"172.21.15.151:6805","nonce":2137494296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":2137494296},{"type":"v1","addr":"172.21.15.151:6809","nonce":2137494296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2137494296},{"type":"v1","addr":"172.21.15.151:6807","nonce":2137494296}]},"public_addr":"172.21.15.151:6803/2137494296","cluster_addr":"172.21.15.151:6805/2137494296","heartbeat_back_addr":"172.21.15.151:6809/2137494296","heartbeat_front_addr":"172.21.15.151:6807/2137494296","state":["exists","up"]},{"osd":1,"uuid":"09f5d705-684f-4e86-80a2-ec2292af09d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":544,"up_from":557,"up_thru":1374,"down_at":545,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":3884881052},{"type":"v1","addr":"172.21.15.151:6811","nonce":3884881052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3884881052},{"type":"v1","addr":"172.21.15.151:6813","nonce":3884881052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3884881052},{"type":"v1","addr":"172.21.15.151:6817","nonce":3884881052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":3884881052},{"type":"v1","addr":"172.21.15.151:6815","nonce":3884881052}]},"public_addr":"172.21.15.151:6811/3884881052","cluster_addr":"172.21.15.151:6813/3884881052","heartbeat_back_addr":"172.21.15.151:6817/3884881052","heartbeat_front_addr":"172.21.15.151:6815/3884881052","state":["exists","up"]},{"osd":2,"uuid":"6a63f8af-ff69-46bb-830f-03ea89fc78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":578,"up_from":591,"up_thru":1388,"down_at":579,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":322435630},{"type":"v1","addr":"172.21.15.151:6819","nonce":322435630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":322435630},{"type":"v1","addr":"172.21.15.151:6821","nonce":322435630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":322435630},{"type":"v1","addr":"172.21.15.151:6825","nonce":322435630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":322435630},{"type":"v1","addr":"172.21.15.151:6823","nonce":322435630}]},"public_addr":"172.21.15.151:6819/322435630","cluster_addr":"172.21.15.151:6821/322435630","heartbeat_back_addr":"172.21.15.151:6825/322435630","heartbeat_front_addr":"172.21.15.151:6823/322435630","state":["exists","up"]},{"osd":3,"uuid":"770e8334-3b3a-46bc-9782-964b2a696660","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":611,"up_from":624,"up_thru":1388,"down_at":612,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2997668861},{"type":"v1","addr":"172.21.15.151:6827","nonce":2997668861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":2997668861},{"type":"v1","addr":"172.21.15.151:6829","nonce":2997668861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":2997668861},{"type":"v1","addr":"172.21.15.151:6833","nonce":2997668861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":2997668861},{"type":"v1","addr":"172.21.15.151:6831","nonce":2997668861}]},"public_addr":"172.21.15.151:6827/2997668861","cluster_addr":"172.21.15.151:6829/2997668861","heartbeat_back_addr":"172.21.15.151:6833/2997668861","heartbeat_front_addr":"172.21.15.151:6831/2997668861","state":["exists","up"]},{"osd":4,"uuid":"75e258c0-1145-4aee-86aa-db2b63082cc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":1374,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6801","nonce":3275553935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6803","nonce":3275553935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6807","nonce":3275553935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3275553935},{"type":"v1","addr":"172.21.15.184:6805","nonce":3275553935}]},"public_addr":"172.21.15.184:6801/3275553935","cluster_addr":"172.21.15.184:6803/3275553935","heartbeat_back_addr":"172.21.15.184:6807/3275553935","heartbeat_front_addr":"172.21.15.184:6805/3275553935","state":["exists","up"]},{"osd":5,"uuid":"f56a5366-9d22-4960-933d-340aa280fbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":1374,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6809","nonce":2446419621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6811","nonce":2446419621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6815","nonce":2446419621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2446419621},{"type":"v1","addr":"172.21.15.184:6813","nonce":2446419621}]},"public_addr":"172.21.15.184:6809/2446419621","cluster_addr":"172.21.15.184:6811/2446419621","heartbeat_back_addr":"172.21.15.184:6815/2446419621","heartbeat_front_addr":"172.21.15.184:6813/2446419621","state":["exists","up"]},{"osd":6,"uuid":"e9826568-a728-45ab-8180-9777fe31f93a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":1391,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6817","nonce":2214837203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6819","nonce":2214837203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6823","nonce":2214837203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2214837203},{"type":"v1","addr":"172.21.15.184:6821","nonce":2214837203}]},"public_addr":"172.21.15.184:6817/2214837203","cluster_addr":"172.21.15.184:6819/2214837203","heartbeat_back_addr":"172.21.15.184:6823/2214837203","heartbeat_front_addr":"172.21.15.184:6821/2214837203","state":["exists","up"]},{"osd":7,"uuid":"d76869fe-5121-4467-8b56-1aadef0eaf65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":1374,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6825","nonce":3378735997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6827","nonce":3378735997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6831","nonce":3378735997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":3378735997},{"type":"v1","addr":"172.21.15.184:6829","nonce":3378735997}]},"public_addr":"172.21.15.184:6825/3378735997","cluster_addr":"172.21.15.184:6827/3378735997","heartbeat_back_addr":"172.21.15.184:6831/3378735997","heartbeat_front_addr":"172.21.15.184:6829/3378735997","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:14:49.835715+0000","dead_epoch":511},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:15:14.682882+0000","dead_epoch":544},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:15:39.423890+0000","dead_epoch":578},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:16:04.527460+0000","dead_epoch":611},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:16:27.283661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:16:51.893868+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:16.152592+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:17:39.763533+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":3,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":0}]},{"pgid":"4.1b","mappings":[{"from":3,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.151:0/1783031268":"2024-09-14T14:24:01.073069+0000","172.21.15.151:0/1775071943":"2024-09-14T14:24:01.073069+0000","172.21.15.151:6801/42840":"2024-09-14T14:24:01.073069+0000","172.21.15.151:0/841256485":"2024-09-14T14:24:01.073069+0000","172.21.15.151:0/1182681341":"2024-09-14T14:13:04.761383+0000","172.21.15.151:6800/1057983082":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/2889050140":"2024-09-14T14:18:18.344788+0000","172.21.15.151:6801/4025698662":"2024-09-14T14:22:15.389006+0000","172.21.15.151:0/2800235450":"2024-09-14T14:18:18.344788+0000","172.21.15.184:0/3929228938":"2024-09-14T14:23:20.772174+0000","172.21.15.151:0/1409232061":"2024-09-14T14:24:01.073069+0000","172.21.15.151:6800/4025698662":"2024-09-14T14:22:15.389006+0000","172.21.15.184:0/1999058007":"2024-09-14T14:23:20.772174+0000","172.21.15.151:6801/1057983082":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/514781427":"2024-09-14T14:22:15.389006+0000","172.21.15.151:0/663506338":"2024-09-14T14:24:01.073069+0000","172.21.15.151:6800/3383065173":"2024-09-14T14:13:04.761383+0000","172.21.15.184:6834/205203765":"2024-09-14T14:23:20.772174+0000","172.21.15.151:0/1530349337":"2024-09-14T14:22:15.389006+0000","172.21.15.151:0/1427922964":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/2666829207":"2024-09-14T14:13:27.478847+0000","172.21.15.151:6801/2335297718":"2024-09-14T14:18:18.344788+0000","172.21.15.151:0/1351186656":"2024-09-14T14:18:18.344788+0000","172.21.15.184:6835/205203765":"2024-09-14T14:23:20.772174+0000","172.21.15.184:0/3003411098":"2024-09-14T14:23:20.772174+0000","172.21.15.151:0/4115173791":"2024-09-14T14:18:18.344788+0000","172.21.15.151:0/3689756337":"2024-09-14T14:13:27.478847+0000","172.21.15.151:0/1517910843":"2024-09-14T14:24:01.073069+0000","172.21.15.184:0/3370242044":"2024-09-14T14:23:20.772174+0000","172.21.15.151:6800/2335297718":"2024-09-14T14:18:18.344788+0000","172.21.15.151:6800/42840":"2024-09-14T14:24:01.073069+0000","172.21.15.151:0/1488764319":"2024-09-14T14:22:15.389006+0000","172.21.15.184:0/2995983270":"2024-09-14T14:23:20.772174+0000","172.21.15.151:0/3946777511":"2024-09-14T14:13:04.761383+0000","172.21.15.151:0/624042464":"2024-09-14T14:13:04.761383+0000","172.21.15.151:0/2201280183":"2024-09-14T14:22:15.389006+0000","172.21.15.151:6801/3383065173":"2024-09-14T14:13:04.761383+0000","172.21.15.184:0/3047962606":"2024-09-14T14:23:20.772174+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":12,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T15:08:27.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:27 smithi184 ceph-mon[92527]: pgmap v2470: 103 pgs: 103 active+clean; 608 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:27.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:27 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/569268339' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T15:08:27.358 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:27 smithi151 ceph-mon[108966]: pgmap v2470: 103 pgs: 103 active+clean; 608 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:27.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:27 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/569268339' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T15:08:27.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:27 smithi151 ceph-mon[104450]: pgmap v2470: 103 pgs: 103 active+clean; 608 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:27.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:27 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/569268339' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T15:08:27.580 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-13T15:08:27.580 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-13T15:08:27.581 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-13T15:08:27.581 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-13T15:08:27.582 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-13T15:08:27.582 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-13T15:08:27.582 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-13T15:08:27.583 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-13T15:08:28.129 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:28.129 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:28.130 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:28.131 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:28.135 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:28.135 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:28.135 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:28.140 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:29.301 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:29 smithi151 ceph-mon[104450]: pgmap v2471: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:29.301 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:29 smithi151 ceph-mon[108966]: pgmap v2471: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:29.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:29 smithi184 ceph-mon[92527]: pgmap v2471: 103 pgs: 103 active+clean; 582 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:30.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:08:30 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:08:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:08:31.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:31 smithi151 ceph-mon[104450]: pgmap v2472: 103 pgs: 103 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:31.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:31 smithi151 ceph-mon[108966]: pgmap v2472: 103 pgs: 103 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:31.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:31 smithi184 ceph-mon[92527]: pgmap v2472: 103 pgs: 103 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:32.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:32 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:32.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:32 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:32.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:32 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:33.354 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:33 smithi184 ceph-mon[92527]: pgmap v2473: 103 pgs: 103 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:33.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:33 smithi151 ceph-mon[104450]: pgmap v2473: 103 pgs: 103 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:33.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:33 smithi151 ceph-mon[108966]: pgmap v2473: 103 pgs: 103 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:34.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:08:34 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:08:34] "GET /metrics HTTP/1.1" 200 34284 "" "Prometheus/2.51.0" 2024-09-13T15:08:35.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:35 smithi151 ceph-mon[104450]: pgmap v2474: 103 pgs: 103 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:35.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:35 smithi151 ceph-mon[108966]: pgmap v2474: 103 pgs: 103 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:35.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:35 smithi184 ceph-mon[92527]: pgmap v2474: 103 pgs: 103 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:37.337 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:37 smithi151 ceph-mon[104450]: pgmap v2475: 103 pgs: 103 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:37.337 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:37 smithi151 ceph-mon[108966]: pgmap v2475: 103 pgs: 103 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:37.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:37 smithi184 ceph-mon[92527]: pgmap v2475: 103 pgs: 103 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:38.905 INFO:teuthology.orchestra.run.smithi151.stdout:128849019510 2024-09-13T15:08:38.906 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-13T15:08:39.145 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:39 smithi151 ceph-mon[104450]: pgmap v2476: 103 pgs: 103 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:39.146 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:39 smithi151 ceph-mon[108966]: pgmap v2476: 103 pgs: 103 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:39.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:39 smithi184 ceph-mon[92527]: pgmap v2476: 103 pgs: 103 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:40.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:08:40 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:08:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:08:40.648 INFO:teuthology.orchestra.run.smithi151.stdout:180388627053 2024-09-13T15:08:40.648 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-13T15:08:41.553 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:41 smithi151 ceph-mon[104450]: pgmap v2477: 103 pgs: 103 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:41.553 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:41 smithi151 ceph-mon[108966]: pgmap v2477: 103 pgs: 103 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:41.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:41 smithi184 ceph-mon[92527]: pgmap v2477: 103 pgs: 103 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:42.736 INFO:teuthology.orchestra.run.smithi151.stdout:2392296784349 2024-09-13T15:08:42.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-13T15:08:43.187 INFO:teuthology.orchestra.run.smithi151.stdout:154618823282 2024-09-13T15:08:43.187 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-13T15:08:43.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:43 smithi184 ceph-mon[92527]: pgmap v2478: 103 pgs: 103 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:43.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:43 smithi151 ceph-mon[104450]: pgmap v2478: 103 pgs: 103 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:43.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:43 smithi151 ceph-mon[108966]: pgmap v2478: 103 pgs: 103 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:44.283 INFO:teuthology.orchestra.run.smithi151.stdout:2538325672406 2024-09-13T15:08:44.283 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-13T15:08:44.714 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:44.715 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:44.740 INFO:teuthology.orchestra.run.smithi151.stdout:2680059593168 2024-09-13T15:08:44.740 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-13T15:08:44.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:08:44 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:08:44] "GET /metrics HTTP/1.1" 200 34284 "" "Prometheus/2.51.0" 2024-09-13T15:08:45.252 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:45 smithi151 ceph-mon[104450]: pgmap v2479: 103 pgs: 103 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:45.252 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:45 smithi151 ceph-mon[108966]: pgmap v2479: 103 pgs: 103 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:45.258 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:45.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:45 smithi184 ceph-mon[92527]: pgmap v2479: 103 pgs: 103 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:47.114 INFO:teuthology.orchestra.run.smithi151.stdout:206158430825 2024-09-13T15:08:47.115 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-13T15:08:47.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:47 smithi151 ceph-mon[104450]: pgmap v2480: 103 pgs: 103 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:47.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:47 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:47.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:47 smithi151 ceph-mon[108966]: pgmap v2480: 103 pgs: 103 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:47.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:47 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:47.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:47 smithi184 ceph-mon[92527]: pgmap v2480: 103 pgs: 103 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:47.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:47 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:47.794 INFO:teuthology.orchestra.run.smithi151.stdout:2250562863589 2024-09-13T15:08:47.794 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-13T15:08:49.124 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:49.571 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:49 smithi151 ceph-mon[104450]: pgmap v2481: 103 pgs: 103 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:49.572 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:49 smithi151 ceph-mon[108966]: pgmap v2481: 103 pgs: 103 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:49.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:49 smithi184 ceph-mon[92527]: pgmap v2481: 103 pgs: 103 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:49.831 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:50.505 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:08:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:08:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:08:51.292 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:51 smithi151 ceph-mon[104450]: pgmap v2482: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:51.292 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:51 smithi151 ceph-mon[108966]: pgmap v2482: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:51.416 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:51.527 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:51 smithi184 ceph-mon[92527]: pgmap v2482: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:51.567 INFO:teuthology.orchestra.run.smithi151.stdout:154618823284 2024-09-13T15:08:52.489 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:52 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1844231234' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-13T15:08:52.490 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:52 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1844231234' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-13T15:08:52.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:52 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1844231234' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-13T15:08:53.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:53 smithi151 ceph-mon[104450]: pgmap v2483: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:53.359 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:53 smithi151 ceph-mon[108966]: pgmap v2483: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:53.408 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618823282 got 154618823284 for osd.5 2024-09-13T15:08:53.409 DEBUG:teuthology.parallel:result is None 2024-09-13T15:08:53.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:53 smithi184 ceph-mon[92527]: pgmap v2483: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:54.078 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:54.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:08:54 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:08:54] "GET /metrics HTTP/1.1" 200 34277 "" "Prometheus/2.51.0" 2024-09-13T15:08:54.904 INFO:teuthology.orchestra.run.smithi151.stdout:128849019513 2024-09-13T15:08:55.414 INFO:teuthology.orchestra.run.smithi151.stdout:180388627056 2024-09-13T15:08:55.415 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:55 smithi151 ceph-mon[104450]: pgmap v2484: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:55.415 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:55 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3189197366' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-13T15:08:55.415 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:55 smithi151 ceph-mon[108966]: pgmap v2484: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:55.415 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:55 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3189197366' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-13T15:08:55.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:55 smithi184 ceph-mon[92527]: pgmap v2484: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:55.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:55 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3189197366' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-13T15:08:56.064 INFO:teuthology.orchestra.run.smithi151.stdout:2392296784352 2024-09-13T15:08:56.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:56 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/795654867' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-13T15:08:56.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:56 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/3459894121' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-13T15:08:56.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:56 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/795654867' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-13T15:08:56.360 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:56 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/3459894121' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-13T15:08:56.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:56 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/795654867' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-13T15:08:56.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:56 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/3459894121' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-13T15:08:57.352 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:57 smithi151 ceph-mon[104450]: pgmap v2485: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:57.352 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:57 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/138084171' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-13T15:08:57.353 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:57 smithi151 ceph-mon[108966]: pgmap v2485: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:57.353 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:57 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/138084171' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-13T15:08:57.354 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2392296784349 got 2392296784352 for osd.1 2024-09-13T15:08:57.354 DEBUG:teuthology.parallel:result is None 2024-09-13T15:08:57.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:57 smithi184 ceph-mon[92527]: pgmap v2485: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:57.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:57 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/138084171' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-13T15:08:57.944 INFO:teuthology.orchestra.run.smithi151.stdout:2680059593171 2024-09-13T15:08:58.499 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2680059593168 got 2680059593171 for osd.3 2024-09-13T15:08:58.500 DEBUG:teuthology.parallel:result is None 2024-09-13T15:08:58.897 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:58.929 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388627053 got 180388627056 for osd.6 2024-09-13T15:08:58.929 DEBUG:teuthology.parallel:result is None 2024-09-13T15:08:59.557 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:08:59 smithi151 ceph-mon[104450]: pgmap v2486: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:59.557 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:08:59 smithi151 ceph-mon[108966]: pgmap v2486: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:08:59.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:08:59 smithi184 ceph-mon[92527]: pgmap v2486: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:09:00.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:09:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:09:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:09:01.344 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849019510 got 128849019513 for osd.4 2024-09-13T15:09:01.345 DEBUG:teuthology.parallel:result is None 2024-09-13T15:09:01.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:09:01 smithi151 ceph-mon[104450]: pgmap v2487: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:09:01.449 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:09:01 smithi151 ceph-mon[108966]: pgmap v2487: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:09:01.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:09:01 smithi184 ceph-mon[92527]: pgmap v2487: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:09:02.558 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:09:02 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:09:02.558 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:09:02 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:09:02.597 INFO:teuthology.orchestra.run.smithi151.stdout:2538325672410 2024-09-13T15:09:02.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:09:02 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:09:02.833 INFO:teuthology.orchestra.run.smithi151.stdout:206158430828 2024-09-13T15:09:03.532 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:09:03 smithi151 ceph-mon[104450]: pgmap v2488: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:09:03.532 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:09:03 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/1078571874' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-13T15:09:03.532 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:09:03 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/4274672738' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-13T15:09:03.532 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:09:03 smithi151 ceph-mon[108966]: pgmap v2488: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:09:03.532 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:09:03 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/1078571874' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-13T15:09:03.533 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:09:03 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/4274672738' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-13T15:09:03.556 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430825 got 206158430828 for osd.7 2024-09-13T15:09:03.557 DEBUG:teuthology.parallel:result is None 2024-09-13T15:09:03.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:09:03 smithi184 ceph-mon[92527]: pgmap v2488: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:09:03.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:09:03 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/1078571874' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-13T15:09:03.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:09:03 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/4274672738' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-13T15:09:04.181 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2538325672406 got 2538325672410 for osd.2 2024-09-13T15:09:04.181 DEBUG:teuthology.parallel:result is None 2024-09-13T15:09:04.601 INFO:teuthology.orchestra.run.smithi151.stdout:2250562863593 2024-09-13T15:09:04.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:09:04 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:09:04] "GET /metrics HTTP/1.1" 200 34280 "" "Prometheus/2.51.0" 2024-09-13T15:09:05.240 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2250562863589 got 2250562863593 for osd.0 2024-09-13T15:09:05.240 DEBUG:teuthology.parallel:result is None 2024-09-13T15:09:05.240 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T15:09:05.241 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:09:05.498 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:05.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:09:05 smithi184 ceph-mon[92527]: pgmap v2489: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:09:05.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:09:05 smithi184 ceph-mon[92527]: from='client.? 172.21.15.151:0/173694537' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-13T15:09:05.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:09:05 smithi151 ceph-mon[104450]: pgmap v2489: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:09:05.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:09:05 smithi151 ceph-mon[104450]: from='client.? 172.21.15.151:0/173694537' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-13T15:09:05.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:09:05 smithi151 ceph-mon[108966]: pgmap v2489: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:09:05.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:09:05 smithi151 ceph-mon[108966]: from='client.? 172.21.15.151:0/173694537' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-13T15:09:07.404 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:09:07 smithi151 ceph-mon[104450]: pgmap v2490: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:09:07.404 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:09:07 smithi151 ceph-mon[108966]: pgmap v2490: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:09:07.571 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T15:09:07.573 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T15:09:07.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:09:07 smithi184 ceph-mon[92527]: pgmap v2490: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:09:08.268 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":2490,"stamp":"2024-09-13T15:09:05.992363+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":2915121,"num_keys_recovered":50,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3158,"ondisk_log_size":3158,"up":309,"acting":309,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":311,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":9638736,"kb_used_data":136952,"kb_used_omap":149,"kb_used_meta":9501546,"kb_avail":740322480,"statfs":{"total":767960285184,"available":758090219520,"internally_reserved":0,"allocated":140238848,"data_stored":113553903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":153401,"internal_metadata":9729583303},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003034"},"pg_stats":[{"pgid":"1.1d","version":"261'100","reported_seq":385,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.007628+0000","last_change":"2024-09-13T15:07:15.998428+0000","last_active":"2024-09-13T15:08:22.007628+0000","last_peered":"2024-09-13T15:08:22.007628+0000","last_clean":"2024-09-13T15:08:22.007628+0000","last_became_active":"2024-09-13T15:07:15.997925+0000","last_became_peered":"2024-09-13T15:07:15.997925+0000","last_unstale":"2024-09-13T15:08:22.007628+0000","last_undegraded":"2024-09-13T15:08:22.007628+0000","last_fullsized":"2024-09-13T15:08:22.007628+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:17.988345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:17.988345+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:31:34.794143+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":93,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.037259+0000","last_change":"2024-09-13T15:07:22.018488+0000","last_active":"2024-09-13T15:08:22.037259+0000","last_peered":"2024-09-13T15:08:22.037259+0000","last_clean":"2024-09-13T15:08:22.037259+0000","last_became_active":"2024-09-13T15:07:22.005489+0000","last_became_peered":"2024-09-13T15:07:22.005489+0000","last_unstale":"2024-09-13T15:08:22.037259+0000","last_undegraded":"2024-09-13T15:08:22.037259+0000","last_fullsized":"2024-09-13T15:08:22.037259+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:44:23.083231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3420,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":31,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.058429+0000","last_change":"2024-09-13T15:07:16.006235+0000","last_active":"2024-09-13T15:08:18.058429+0000","last_peered":"2024-09-13T15:08:18.058429+0000","last_clean":"2024-09-13T15:08:18.058429+0000","last_became_active":"2024-09-13T15:07:16.005374+0000","last_became_peered":"2024-09-13T15:07:16.005374+0000","last_unstale":"2024-09-13T15:08:18.058429+0000","last_undegraded":"2024-09-13T15:08:18.058429+0000","last_fullsized":"2024-09-13T15:08:18.058429+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:54:10.663527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":526,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089685+0000","last_change":"2024-09-13T15:07:22.013535+0000","last_active":"2024-09-13T15:08:22.089685+0000","last_peered":"2024-09-13T15:08:22.089685+0000","last_clean":"2024-09-13T15:08:22.089685+0000","last_became_active":"2024-09-13T15:07:22.013156+0000","last_became_peered":"2024-09-13T15:07:22.013156+0000","last_unstale":"2024-09-13T15:08:22.089685+0000","last_undegraded":"2024-09-13T15:08:22.089685+0000","last_fullsized":"2024-09-13T15:08:22.089685+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:10.552250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":538,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089256+0000","last_change":"2024-09-13T15:07:22.008177+0000","last_active":"2024-09-13T15:08:22.089256+0000","last_peered":"2024-09-13T15:08:22.089256+0000","last_clean":"2024-09-13T15:08:22.089256+0000","last_became_active":"2024-09-13T15:07:22.007855+0000","last_became_peered":"2024-09-13T15:07:22.007855+0000","last_unstale":"2024-09-13T15:08:22.089256+0000","last_undegraded":"2024-09-13T15:08:22.089256+0000","last_fullsized":"2024-09-13T15:08:22.089256+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:15:30.634430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"261'100","reported_seq":76,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.061387+0000","last_change":"2024-09-13T15:02:42.889565+0000","last_active":"2024-09-13T15:08:18.061387+0000","last_peered":"2024-09-13T15:08:18.061387+0000","last_clean":"2024-09-13T15:08:18.061387+0000","last_became_active":"2024-09-13T15:02:42.888466+0000","last_became_peered":"2024-09-13T15:02:42.888466+0000","last_unstale":"2024-09-13T15:08:18.061387+0000","last_undegraded":"2024-09-13T15:08:18.061387+0000","last_fullsized":"2024-09-13T15:08:18.061387+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_clean_scrub_stamp":"2024-09-13T14:44:54.259482+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:19:31.567604+0000","scrub_duration":0.015265569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":198,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.682151+0000","last_change":"2024-09-13T15:07:16.019036+0000","last_active":"2024-09-13T15:08:18.682151+0000","last_peered":"2024-09-13T15:08:18.682151+0000","last_clean":"2024-09-13T15:08:18.682151+0000","last_became_active":"2024-09-13T15:07:15.997656+0000","last_became_peered":"2024-09-13T15:07:15.997656+0000","last_unstale":"2024-09-13T15:08:18.682151+0000","last_undegraded":"2024-09-13T15:08:18.682151+0000","last_fullsized":"2024-09-13T15:08:18.682151+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:46.516312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":31,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.058575+0000","last_change":"2024-09-13T15:07:16.001171+0000","last_active":"2024-09-13T15:08:18.058575+0000","last_peered":"2024-09-13T15:08:18.058575+0000","last_clean":"2024-09-13T15:08:18.058575+0000","last_became_active":"2024-09-13T15:07:15.997226+0000","last_became_peered":"2024-09-13T15:07:15.997226+0000","last_unstale":"2024-09-13T15:08:18.058575+0000","last_undegraded":"2024-09-13T15:08:18.058575+0000","last_fullsized":"2024-09-13T15:08:18.058575+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:57:22.574830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"261'100","reported_seq":529,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089762+0000","last_change":"2024-09-13T14:45:35.947332+0000","last_active":"2024-09-13T15:08:22.089762+0000","last_peered":"2024-09-13T15:08:22.089762+0000","last_clean":"2024-09-13T15:08:22.089762+0000","last_became_active":"2024-09-13T14:45:19.810775+0000","last_became_peered":"2024-09-13T14:45:19.810775+0000","last_unstale":"2024-09-13T15:08:22.089762+0000","last_undegraded":"2024-09-13T15:08:22.089762+0000","last_fullsized":"2024-09-13T15:08:22.089762+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:35.947265+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:43:41.569795+0000","last_clean_scrub_stamp":"2024-09-13T14:45:35.947265+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:00.606370+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":198,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.682123+0000","last_change":"2024-09-13T15:07:16.023529+0000","last_active":"2024-09-13T15:08:18.682123+0000","last_peered":"2024-09-13T15:08:18.682123+0000","last_clean":"2024-09-13T15:08:18.682123+0000","last_became_active":"2024-09-13T15:07:15.998436+0000","last_became_peered":"2024-09-13T15:07:15.998436+0000","last_unstale":"2024-09-13T15:08:18.682123+0000","last_undegraded":"2024-09-13T15:08:18.682123+0000","last_fullsized":"2024-09-13T15:08:18.682123+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:03.197620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":31,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.059031+0000","last_change":"2024-09-13T15:07:16.003785+0000","last_active":"2024-09-13T15:08:18.059031+0000","last_peered":"2024-09-13T15:08:18.059031+0000","last_clean":"2024-09-13T15:08:18.059031+0000","last_became_active":"2024-09-13T15:07:16.003380+0000","last_became_peered":"2024-09-13T15:07:16.003380+0000","last_unstale":"2024-09-13T15:08:18.059031+0000","last_undegraded":"2024-09-13T15:08:18.059031+0000","last_fullsized":"2024-09-13T15:08:18.059031+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:00.677156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":80,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.037139+0000","last_change":"2024-09-13T15:07:15.993955+0000","last_active":"2024-09-13T15:08:22.037139+0000","last_peered":"2024-09-13T15:08:22.037139+0000","last_clean":"2024-09-13T15:08:22.037139+0000","last_became_active":"2024-09-13T15:07:15.993625+0000","last_became_peered":"2024-09-13T15:07:15.993625+0000","last_unstale":"2024-09-13T15:08:22.037139+0000","last_undegraded":"2024-09-13T15:08:22.037139+0000","last_fullsized":"2024-09-13T15:08:22.037139+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:00.369075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":50,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:21.201903+0000","last_change":"2024-09-13T15:07:22.005002+0000","last_active":"2024-09-13T15:08:21.201903+0000","last_peered":"2024-09-13T15:08:21.201903+0000","last_clean":"2024-09-13T15:08:21.201903+0000","last_became_active":"2024-09-13T15:07:22.004605+0000","last_became_peered":"2024-09-13T15:07:22.004605+0000","last_unstale":"2024-09-13T15:08:21.201903+0000","last_undegraded":"2024-09-13T15:08:21.201903+0000","last_fullsized":"2024-09-13T15:08:21.201903+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:56:34.368320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"261'100","reported_seq":538,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089267+0000","last_change":"2024-09-13T15:06:53.991044+0000","last_active":"2024-09-13T15:08:22.089267+0000","last_peered":"2024-09-13T15:08:22.089267+0000","last_clean":"2024-09-13T15:08:22.089267+0000","last_became_active":"2024-09-13T15:06:53.987755+0000","last_became_peered":"2024-09-13T15:06:53.987755+0000","last_unstale":"2024-09-13T15:08:22.089267+0000","last_undegraded":"2024-09-13T15:08:22.089267+0000","last_fullsized":"2024-09-13T15:08:22.089267+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.614340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.614340+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:26:58.871459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":41,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.058735+0000","last_change":"2024-09-13T15:07:22.008506+0000","last_active":"2024-09-13T15:08:18.058735+0000","last_peered":"2024-09-13T15:08:18.058735+0000","last_clean":"2024-09-13T15:08:18.058735+0000","last_became_active":"2024-09-13T15:07:22.007790+0000","last_became_peered":"2024-09-13T15:07:22.007790+0000","last_unstale":"2024-09-13T15:08:18.058735+0000","last_undegraded":"2024-09-13T15:08:18.058735+0000","last_fullsized":"2024-09-13T15:08:18.058735+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:05.415174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":566,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089658+0000","last_change":"2024-09-13T15:08:08.306111+0000","last_active":"2024-09-13T15:08:22.089658+0000","last_peered":"2024-09-13T15:08:22.089658+0000","last_clean":"2024-09-13T15:08:22.089658+0000","last_became_active":"2024-09-13T15:08:08.305751+0000","last_became_peered":"2024-09-13T15:08:08.305751+0000","last_unstale":"2024-09-13T15:08:22.089658+0000","last_undegraded":"2024-09-13T15:08:22.089658+0000","last_fullsized":"2024-09-13T15:08:22.089658+0000","mapping_epoch":1388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:10:05.198808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.18","version":"261'100","reported_seq":229,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.681715+0000","last_change":"2024-09-13T15:07:22.024346+0000","last_active":"2024-09-13T15:08:18.681715+0000","last_peered":"2024-09-13T15:08:18.681715+0000","last_clean":"2024-09-13T15:08:18.681715+0000","last_became_active":"2024-09-13T15:07:22.008541+0000","last_became_peered":"2024-09-13T15:07:22.008541+0000","last_unstale":"2024-09-13T15:08:18.681715+0000","last_undegraded":"2024-09-13T15:08:18.681715+0000","last_fullsized":"2024-09-13T15:08:18.681715+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:56.548855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:56.548855+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:43:23.766548+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1576128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":379,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.007208+0000","last_change":"2024-09-13T15:07:15.997884+0000","last_active":"2024-09-13T15:08:22.007208+0000","last_peered":"2024-09-13T15:08:22.007208+0000","last_clean":"2024-09-13T15:08:22.007208+0000","last_became_active":"2024-09-13T15:07:15.997533+0000","last_became_peered":"2024-09-13T15:07:15.997533+0000","last_unstale":"2024-09-13T15:08:22.007208+0000","last_undegraded":"2024-09-13T15:08:22.007208+0000","last_fullsized":"2024-09-13T15:08:22.007208+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:08:37.966159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":74,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.036653+0000","last_change":"2024-09-13T15:06:53.984474+0000","last_active":"2024-09-13T15:08:22.036653+0000","last_peered":"2024-09-13T15:08:22.036653+0000","last_clean":"2024-09-13T15:08:22.036653+0000","last_became_active":"2024-09-13T15:06:53.983733+0000","last_became_peered":"2024-09-13T15:06:53.983733+0000","last_unstale":"2024-09-13T15:08:22.036653+0000","last_undegraded":"2024-09-13T15:08:22.036653+0000","last_fullsized":"2024-09-13T15:08:22.036653+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:19:24.594408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1b","version":"261'100","reported_seq":226,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.681906+0000","last_change":"2024-09-13T15:07:16.002022+0000","last_active":"2024-09-13T15:08:18.681906+0000","last_peered":"2024-09-13T15:08:18.681906+0000","last_clean":"2024-09-13T15:08:18.681906+0000","last_became_active":"2024-09-13T15:07:16.001656+0000","last_became_peered":"2024-09-13T15:07:16.001656+0000","last_unstale":"2024-09-13T15:08:18.681906+0000","last_undegraded":"2024-09-13T15:08:18.681906+0000","last_fullsized":"2024-09-13T15:08:18.681906+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:55.988431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:55.988431+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:36:09.915389+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":545,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089117+0000","last_change":"2024-09-13T15:06:53.990683+0000","last_active":"2024-09-13T15:08:22.089117+0000","last_peered":"2024-09-13T15:08:22.089117+0000","last_clean":"2024-09-13T15:08:22.089117+0000","last_became_active":"2024-09-13T15:06:53.988100+0000","last_became_peered":"2024-09-13T15:06:53.988100+0000","last_unstale":"2024-09-13T15:08:22.089117+0000","last_undegraded":"2024-09-13T15:08:22.089117+0000","last_fullsized":"2024-09-13T15:08:22.089117+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:03.162701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1100,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:19.004385+0000","last_change":"2024-09-13T15:08:16.108837+0000","last_active":"2024-09-13T15:08:19.004385+0000","last_peered":"2024-09-13T15:08:19.004385+0000","last_clean":"2024-09-13T15:08:19.004385+0000","last_became_active":"2024-09-13T15:08:16.108299+0000","last_became_peered":"2024-09-13T15:08:16.108299+0000","last_unstale":"2024-09-13T15:08:19.004385+0000","last_undegraded":"2024-09-13T15:08:19.004385+0000","last_fullsized":"2024-09-13T15:08:19.004385+0000","mapping_epoch":1391,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:11:37.949098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1a","version":"261'100","reported_seq":31,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.058700+0000","last_change":"2024-09-13T15:07:15.998441+0000","last_active":"2024-09-13T15:08:18.058700+0000","last_peered":"2024-09-13T15:08:18.058700+0000","last_clean":"2024-09-13T15:08:18.058700+0000","last_became_active":"2024-09-13T15:07:15.998094+0000","last_became_peered":"2024-09-13T15:07:15.998094+0000","last_unstale":"2024-09-13T15:08:18.058700+0000","last_undegraded":"2024-09-13T15:08:18.058700+0000","last_fullsized":"2024-09-13T15:08:18.058700+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:46.985510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:46.985510+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:54:26.582536+0000","scrub_duration":0.0030525999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":532,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089216+0000","last_change":"2024-09-13T14:54:05.606882+0000","last_active":"2024-09-13T15:08:22.089216+0000","last_peered":"2024-09-13T15:08:22.089216+0000","last_clean":"2024-09-13T15:08:22.089216+0000","last_became_active":"2024-09-13T14:54:05.605912+0000","last_became_peered":"2024-09-13T14:54:05.605912+0000","last_unstale":"2024-09-13T15:08:22.089216+0000","last_undegraded":"2024-09-13T15:08:22.089216+0000","last_fullsized":"2024-09-13T15:08:22.089216+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:05.445329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":73,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.036484+0000","last_change":"2024-09-13T15:06:53.989318+0000","last_active":"2024-09-13T15:08:22.036484+0000","last_peered":"2024-09-13T15:08:22.036484+0000","last_clean":"2024-09-13T15:08:22.036484+0000","last_became_active":"2024-09-13T15:06:53.986672+0000","last_became_peered":"2024-09-13T15:06:53.986672+0000","last_unstale":"2024-09-13T15:08:22.036484+0000","last_undegraded":"2024-09-13T15:08:22.036484+0000","last_fullsized":"2024-09-13T15:08:22.036484+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:17.142773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"261'100","reported_seq":76,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.061445+0000","last_change":"2024-09-13T15:02:42.892087+0000","last_active":"2024-09-13T15:08:18.061445+0000","last_peered":"2024-09-13T15:08:18.061445+0000","last_clean":"2024-09-13T15:08:18.061445+0000","last_became_active":"2024-09-13T15:02:42.890787+0000","last_became_peered":"2024-09-13T15:02:42.890787+0000","last_unstale":"2024-09-13T15:08:18.061445+0000","last_undegraded":"2024-09-13T15:08:18.061445+0000","last_fullsized":"2024-09-13T15:08:18.061445+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:38.214865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:38.214865+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:12:04.179567+0000","scrub_duration":0.0016389639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2096,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089395+0000","last_change":"2024-09-13T15:02:42.901601+0000","last_active":"2024-09-13T15:08:22.089395+0000","last_peered":"2024-09-13T15:08:22.089395+0000","last_clean":"2024-09-13T15:08:22.089395+0000","last_became_active":"2024-09-13T15:02:42.900518+0000","last_became_peered":"2024-09-13T15:02:42.900518+0000","last_unstale":"2024-09-13T15:08:22.089395+0000","last_undegraded":"2024-09-13T15:08:22.089395+0000","last_fullsized":"2024-09-13T15:08:22.089395+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:57:33.494780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":186,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.682095+0000","last_change":"2024-09-13T15:02:42.885564+0000","last_active":"2024-09-13T15:08:18.682095+0000","last_peered":"2024-09-13T15:08:18.682095+0000","last_clean":"2024-09-13T15:08:18.682095+0000","last_became_active":"2024-09-13T15:02:42.884990+0000","last_became_peered":"2024-09-13T15:02:42.884990+0000","last_unstale":"2024-09-13T15:08:18.682095+0000","last_undegraded":"2024-09-13T15:08:18.682095+0000","last_fullsized":"2024-09-13T15:08:18.682095+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:51:42.556111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"261'100","reported_seq":50,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:21.201930+0000","last_change":"2024-09-13T15:07:22.003039+0000","last_active":"2024-09-13T15:08:21.201930+0000","last_peered":"2024-09-13T15:08:21.201930+0000","last_clean":"2024-09-13T15:08:21.201930+0000","last_became_active":"2024-09-13T15:07:22.002458+0000","last_became_peered":"2024-09-13T15:07:22.002458+0000","last_unstale":"2024-09-13T15:08:21.201930+0000","last_undegraded":"2024-09-13T15:08:21.201930+0000","last_fullsized":"2024-09-13T15:08:21.201930+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.037517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.037517+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:30:14.518182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":114,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.061401+0000","last_change":"2024-09-13T15:07:16.027851+0000","last_active":"2024-09-13T15:08:18.061401+0000","last_peered":"2024-09-13T15:08:18.061401+0000","last_clean":"2024-09-13T15:08:18.061401+0000","last_became_active":"2024-09-13T15:07:15.998722+0000","last_became_peered":"2024-09-13T15:07:15.998722+0000","last_unstale":"2024-09-13T15:08:18.061401+0000","last_undegraded":"2024-09-13T15:08:18.061401+0000","last_fullsized":"2024-09-13T15:08:18.061401+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:14.744690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":450,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":379,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.007322+0000","last_change":"2024-09-13T15:07:22.008311+0000","last_active":"2024-09-13T15:08:22.007322+0000","last_peered":"2024-09-13T15:08:22.007322+0000","last_clean":"2024-09-13T15:08:22.007322+0000","last_became_active":"2024-09-13T15:07:22.008012+0000","last_became_peered":"2024-09-13T15:07:22.008012+0000","last_unstale":"2024-09-13T15:08:22.007322+0000","last_undegraded":"2024-09-13T15:08:22.007322+0000","last_fullsized":"2024-09-13T15:08:22.007322+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:57.605334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"261'100","reported_seq":196,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.682242+0000","last_change":"2024-09-13T15:06:53.998536+0000","last_active":"2024-09-13T15:08:18.682242+0000","last_peered":"2024-09-13T15:08:18.682242+0000","last_clean":"2024-09-13T15:08:18.682242+0000","last_became_active":"2024-09-13T15:06:53.998173+0000","last_became_peered":"2024-09-13T15:06:53.998173+0000","last_unstale":"2024-09-13T15:08:18.682242+0000","last_undegraded":"2024-09-13T15:08:18.682242+0000","last_fullsized":"2024-09-13T15:08:18.682242+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:54.933410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:54.933410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:23.765857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":41,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.059000+0000","last_change":"2024-09-13T15:07:16.017541+0000","last_active":"2024-09-13T15:08:18.059000+0000","last_peered":"2024-09-13T15:08:18.059000+0000","last_clean":"2024-09-13T15:08:18.059000+0000","last_became_active":"2024-09-13T15:07:16.006537+0000","last_became_peered":"2024-09-13T15:07:16.006537+0000","last_unstale":"2024-09-13T15:08:18.059000+0000","last_undegraded":"2024-09-13T15:08:18.059000+0000","last_fullsized":"2024-09-13T15:08:18.059000+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:49.741517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":23720,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":31,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.058824+0000","last_change":"2024-09-13T15:07:16.006027+0000","last_active":"2024-09-13T15:08:18.058824+0000","last_peered":"2024-09-13T15:08:18.058824+0000","last_clean":"2024-09-13T15:08:18.058824+0000","last_became_active":"2024-09-13T15:07:16.005747+0000","last_became_peered":"2024-09-13T15:07:16.005747+0000","last_unstale":"2024-09-13T15:08:18.058824+0000","last_undegraded":"2024-09-13T15:08:18.058824+0000","last_fullsized":"2024-09-13T15:08:18.058824+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:28.371504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"261'100","reported_seq":379,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.007839+0000","last_change":"2024-09-13T15:06:53.990651+0000","last_active":"2024-09-13T15:08:22.007839+0000","last_peered":"2024-09-13T15:08:22.007839+0000","last_clean":"2024-09-13T15:08:22.007839+0000","last_became_active":"2024-09-13T15:06:53.989985+0000","last_became_peered":"2024-09-13T15:06:53.989985+0000","last_unstale":"2024-09-13T15:08:22.007839+0000","last_undegraded":"2024-09-13T15:08:22.007839+0000","last_fullsized":"2024-09-13T15:08:22.007839+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:48.363045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:48.363045+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:51.271154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":31,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.058614+0000","last_change":"2024-09-13T15:07:16.005932+0000","last_active":"2024-09-13T15:08:18.058614+0000","last_peered":"2024-09-13T15:08:18.058614+0000","last_clean":"2024-09-13T15:08:18.058614+0000","last_became_active":"2024-09-13T15:07:16.005634+0000","last_became_peered":"2024-09-13T15:07:16.005634+0000","last_unstale":"2024-09-13T15:08:18.058614+0000","last_undegraded":"2024-09-13T15:08:18.058614+0000","last_fullsized":"2024-09-13T15:08:18.058614+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:56:13.989988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1094,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:19.004437+0000","last_change":"2024-09-13T15:08:16.109153+0000","last_active":"2024-09-13T15:08:19.004437+0000","last_peered":"2024-09-13T15:08:19.004437+0000","last_clean":"2024-09-13T15:08:19.004437+0000","last_became_active":"2024-09-13T15:08:16.108623+0000","last_became_peered":"2024-09-13T15:08:16.108623+0000","last_unstale":"2024-09-13T15:08:19.004437+0000","last_undegraded":"2024-09-13T15:08:19.004437+0000","last_fullsized":"2024-09-13T15:08:19.004437+0000","mapping_epoch":1391,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:06.341946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":89,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.037155+0000","last_change":"2024-09-13T15:07:16.005034+0000","last_active":"2024-09-13T15:08:22.037155+0000","last_peered":"2024-09-13T15:08:22.037155+0000","last_clean":"2024-09-13T15:08:22.037155+0000","last_became_active":"2024-09-13T15:07:15.996110+0000","last_became_peered":"2024-09-13T15:07:15.996110+0000","last_unstale":"2024-09-13T15:08:22.037155+0000","last_undegraded":"2024-09-13T15:08:22.037155+0000","last_fullsized":"2024-09-13T15:08:22.037155+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:17:23.430425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"261'100","reported_seq":554,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089635+0000","last_change":"2024-09-13T15:02:42.899622+0000","last_active":"2024-09-13T15:08:22.089635+0000","last_peered":"2024-09-13T15:08:22.089635+0000","last_clean":"2024-09-13T15:08:22.089635+0000","last_became_active":"2024-09-13T15:02:42.897715+0000","last_became_peered":"2024-09-13T15:02:42.897715+0000","last_unstale":"2024-09-13T15:08:22.089635+0000","last_undegraded":"2024-09-13T15:08:22.089635+0000","last_fullsized":"2024-09-13T15:08:22.089635+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.710192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.710192+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:50:03.858097+0000","scrub_duration":0.0019250910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":443,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089038+0000","last_change":"2024-09-13T15:06:53.990088+0000","last_active":"2024-09-13T15:08:22.089038+0000","last_peered":"2024-09-13T15:08:22.089038+0000","last_clean":"2024-09-13T15:08:22.089038+0000","last_became_active":"2024-09-13T15:06:53.988653+0000","last_became_peered":"2024-09-13T15:06:53.988653+0000","last_unstale":"2024-09-13T15:08:22.089038+0000","last_undegraded":"2024-09-13T15:08:22.089038+0000","last_fullsized":"2024-09-13T15:08:22.089038+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:27:26.438276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":379,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.007801+0000","last_change":"2024-09-13T15:07:15.999008+0000","last_active":"2024-09-13T15:08:22.007801+0000","last_peered":"2024-09-13T15:08:22.007801+0000","last_clean":"2024-09-13T15:08:22.007801+0000","last_became_active":"2024-09-13T15:07:15.998575+0000","last_became_peered":"2024-09-13T15:07:15.998575+0000","last_unstale":"2024-09-13T15:08:22.007801+0000","last_undegraded":"2024-09-13T15:08:22.007801+0000","last_fullsized":"2024-09-13T15:08:22.007801+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:32.354104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":2170,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089021+0000","last_change":"2024-09-13T15:06:54.000491+0000","last_active":"2024-09-13T15:08:22.089021+0000","last_peered":"2024-09-13T15:08:22.089021+0000","last_clean":"2024-09-13T15:08:22.089021+0000","last_became_active":"2024-09-13T15:06:53.987425+0000","last_became_peered":"2024-09-13T15:06:53.987425+0000","last_unstale":"2024-09-13T15:08:22.089021+0000","last_undegraded":"2024-09-13T15:08:22.089021+0000","last_fullsized":"2024-09-13T15:08:22.089021+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:14.972231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"261'100","reported_seq":41,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.059280+0000","last_change":"2024-09-13T15:07:22.007080+0000","last_active":"2024-09-13T15:08:18.059280+0000","last_peered":"2024-09-13T15:08:18.059280+0000","last_clean":"2024-09-13T15:08:18.059280+0000","last_became_active":"2024-09-13T15:07:22.006645+0000","last_became_peered":"2024-09-13T15:07:22.006645+0000","last_unstale":"2024-09-13T15:08:18.059280+0000","last_undegraded":"2024-09-13T15:08:18.059280+0000","last_fullsized":"2024-09-13T15:08:18.059280+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:51:31.310115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:51:31.310115+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:25.468316+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":74,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.036491+0000","last_change":"2024-09-13T15:06:53.987052+0000","last_active":"2024-09-13T15:08:22.036491+0000","last_peered":"2024-09-13T15:08:22.036491+0000","last_clean":"2024-09-13T15:08:22.036491+0000","last_became_active":"2024-09-13T15:06:53.986723+0000","last_became_peered":"2024-09-13T15:06:53.986723+0000","last_unstale":"2024-09-13T15:08:22.036491+0000","last_undegraded":"2024-09-13T15:08:22.036491+0000","last_fullsized":"2024-09-13T15:08:22.036491+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:39.492490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":96,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.061475+0000","last_change":"2024-09-13T15:07:16.000241+0000","last_active":"2024-09-13T15:08:18.061475+0000","last_peered":"2024-09-13T15:08:18.061475+0000","last_clean":"2024-09-13T15:08:18.061475+0000","last_became_active":"2024-09-13T15:07:15.999629+0000","last_became_peered":"2024-09-13T15:07:15.999629+0000","last_unstale":"2024-09-13T15:08:18.061475+0000","last_undegraded":"2024-09-13T15:08:18.061475+0000","last_fullsized":"2024-09-13T15:08:18.061475+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:02.423223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":394,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.007677+0000","last_change":"2024-09-13T15:07:22.010344+0000","last_active":"2024-09-13T15:08:22.007677+0000","last_peered":"2024-09-13T15:08:22.007677+0000","last_clean":"2024-09-13T15:08:22.007677+0000","last_became_active":"2024-09-13T15:07:22.010045+0000","last_became_peered":"2024-09-13T15:07:22.010045+0000","last_unstale":"2024-09-13T15:08:22.007677+0000","last_undegraded":"2024-09-13T15:08:22.007677+0000","last_fullsized":"2024-09-13T15:08:22.007677+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:33.588135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"261'100","reported_seq":416,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:19.004268+0000","last_change":"2024-09-13T15:08:16.056633+0000","last_active":"2024-09-13T15:08:19.004268+0000","last_peered":"2024-09-13T15:08:19.004268+0000","last_clean":"2024-09-13T15:08:19.004268+0000","last_became_active":"2024-09-13T15:08:16.056135+0000","last_became_peered":"2024-09-13T15:08:16.056135+0000","last_unstale":"2024-09-13T15:08:19.004268+0000","last_undegraded":"2024-09-13T15:08:19.004268+0000","last_fullsized":"2024-09-13T15:08:19.004268+0000","mapping_epoch":1391,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1392,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:33.574443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:33.574443+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:03.166364+0000","scrub_duration":0.0079574959999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":22,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:21.201685+0000","last_change":"2024-09-13T15:08:08.404465+0000","last_active":"2024-09-13T15:08:21.201685+0000","last_peered":"2024-09-13T15:08:21.201685+0000","last_clean":"2024-09-13T15:08:21.201685+0000","last_became_active":"2024-09-13T15:08:08.404180+0000","last_became_peered":"2024-09-13T15:08:08.404180+0000","last_unstale":"2024-09-13T15:08:21.201685+0000","last_undegraded":"2024-09-13T15:08:21.201685+0000","last_fullsized":"2024-09-13T15:08:21.201685+0000","mapping_epoch":1388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:33.254058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":394,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.007083+0000","last_change":"2024-09-13T15:06:53.989169+0000","last_active":"2024-09-13T15:08:22.007083+0000","last_peered":"2024-09-13T15:08:22.007083+0000","last_clean":"2024-09-13T15:08:22.007083+0000","last_became_active":"2024-09-13T15:06:53.988812+0000","last_became_peered":"2024-09-13T15:06:53.988812+0000","last_unstale":"2024-09-13T15:08:22.007083+0000","last_undegraded":"2024-09-13T15:08:22.007083+0000","last_fullsized":"2024-09-13T15:08:22.007083+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:55:02.815779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":186,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.682136+0000","last_change":"2024-09-13T15:07:22.008756+0000","last_active":"2024-09-13T15:08:18.682136+0000","last_peered":"2024-09-13T15:08:18.682136+0000","last_clean":"2024-09-13T15:08:18.682136+0000","last_became_active":"2024-09-13T15:07:22.008411+0000","last_became_peered":"2024-09-13T15:07:22.008411+0000","last_unstale":"2024-09-13T15:08:18.682136+0000","last_undegraded":"2024-09-13T15:08:18.682136+0000","last_fullsized":"2024-09-13T15:08:18.682136+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:51.064483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"261'100","reported_seq":412,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:19.004313+0000","last_change":"2024-09-13T15:08:16.986734+0000","last_active":"2024-09-13T15:08:19.004313+0000","last_peered":"2024-09-13T15:08:19.004313+0000","last_clean":"2024-09-13T15:08:19.004313+0000","last_became_active":"2024-09-13T15:08:16.055903+0000","last_became_peered":"2024-09-13T15:08:16.055903+0000","last_unstale":"2024-09-13T15:08:19.004313+0000","last_undegraded":"2024-09-13T15:08:19.004313+0000","last_fullsized":"2024-09-13T15:08:19.004313+0000","mapping_epoch":1391,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1392,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T15:08:16.986669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T15:08:16.986669+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:24:30.288349+0000","scrub_duration":0.0021550900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":373,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.007642+0000","last_change":"2024-09-13T15:07:22.009592+0000","last_active":"2024-09-13T15:08:22.007642+0000","last_peered":"2024-09-13T15:08:22.007642+0000","last_clean":"2024-09-13T15:08:22.007642+0000","last_became_active":"2024-09-13T15:07:22.009269+0000","last_became_peered":"2024-09-13T15:07:22.009269+0000","last_unstale":"2024-09-13T15:08:22.007642+0000","last_undegraded":"2024-09-13T15:08:22.007642+0000","last_fullsized":"2024-09-13T15:08:22.007642+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:47:53.357130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":31,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.058938+0000","last_change":"2024-09-13T15:07:16.006172+0000","last_active":"2024-09-13T15:08:18.058938+0000","last_peered":"2024-09-13T15:08:18.058938+0000","last_clean":"2024-09-13T15:08:18.058938+0000","last_became_active":"2024-09-13T15:07:16.005881+0000","last_became_peered":"2024-09-13T15:07:16.005881+0000","last_unstale":"2024-09-13T15:08:18.058938+0000","last_undegraded":"2024-09-13T15:08:18.058938+0000","last_fullsized":"2024-09-13T15:08:18.058938+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:56.281641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":215,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.681932+0000","last_change":"2024-09-13T14:55:41.889668+0000","last_active":"2024-09-13T15:08:18.681932+0000","last_peered":"2024-09-13T15:08:18.681932+0000","last_clean":"2024-09-13T15:08:18.681932+0000","last_became_active":"2024-09-13T14:55:41.889195+0000","last_became_peered":"2024-09-13T14:55:41.889195+0000","last_unstale":"2024-09-13T15:08:18.681932+0000","last_undegraded":"2024-09-13T15:08:18.681932+0000","last_fullsized":"2024-09-13T15:08:18.681932+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:52.765208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"261'100","reported_seq":236,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.681690+0000","last_change":"2024-09-13T15:07:22.013394+0000","last_active":"2024-09-13T15:08:18.681690+0000","last_peered":"2024-09-13T15:08:18.681690+0000","last_clean":"2024-09-13T15:08:18.681690+0000","last_became_active":"2024-09-13T15:07:22.012974+0000","last_became_peered":"2024-09-13T15:07:22.012974+0000","last_unstale":"2024-09-13T15:08:18.681690+0000","last_undegraded":"2024-09-13T15:08:18.681690+0000","last_fullsized":"2024-09-13T15:08:18.681690+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:57.024823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:57.024823+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:11.466865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":393,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.007209+0000","last_change":"2024-09-13T15:06:53.987932+0000","last_active":"2024-09-13T15:08:22.007209+0000","last_peered":"2024-09-13T15:08:22.007209+0000","last_clean":"2024-09-13T15:08:22.007209+0000","last_became_active":"2024-09-13T15:06:53.987612+0000","last_became_peered":"2024-09-13T15:06:53.987612+0000","last_unstale":"2024-09-13T15:08:22.007209+0000","last_undegraded":"2024-09-13T15:08:22.007209+0000","last_fullsized":"2024-09-13T15:08:22.007209+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:31:01.073349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":380,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.007227+0000","last_change":"2024-09-13T15:07:22.006243+0000","last_active":"2024-09-13T15:08:22.007227+0000","last_peered":"2024-09-13T15:08:22.007227+0000","last_clean":"2024-09-13T15:08:22.007227+0000","last_became_active":"2024-09-13T15:07:22.005921+0000","last_became_peered":"2024-09-13T15:07:22.005921+0000","last_unstale":"2024-09-13T15:08:22.007227+0000","last_undegraded":"2024-09-13T15:08:22.007227+0000","last_fullsized":"2024-09-13T15:08:22.007227+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:55.155198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":50,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:21.201733+0000","last_change":"2024-09-13T15:07:22.004071+0000","last_active":"2024-09-13T15:08:21.201733+0000","last_peered":"2024-09-13T15:08:21.201733+0000","last_clean":"2024-09-13T15:08:21.201733+0000","last_became_active":"2024-09-13T15:07:22.003746+0000","last_became_peered":"2024-09-13T15:07:22.003746+0000","last_unstale":"2024-09-13T15:08:21.201733+0000","last_undegraded":"2024-09-13T15:08:21.201733+0000","last_fullsized":"2024-09-13T15:08:21.201733+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:36.228833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"261'100","reported_seq":116,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.061042+0000","last_change":"2024-09-13T15:07:15.998303+0000","last_active":"2024-09-13T15:08:18.061042+0000","last_peered":"2024-09-13T15:08:18.061042+0000","last_clean":"2024-09-13T15:08:18.061042+0000","last_became_active":"2024-09-13T15:07:15.997919+0000","last_became_peered":"2024-09-13T15:07:15.997919+0000","last_unstale":"2024-09-13T15:08:18.061042+0000","last_undegraded":"2024-09-13T15:08:18.061042+0000","last_fullsized":"2024-09-13T15:08:18.061042+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T15:02:59.897422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T15:02:59.897422+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:23.129996+0000","scrub_duration":0.0032184959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":81,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.037069+0000","last_change":"2024-09-13T15:07:22.005527+0000","last_active":"2024-09-13T15:08:22.037069+0000","last_peered":"2024-09-13T15:08:22.037069+0000","last_clean":"2024-09-13T15:08:22.037069+0000","last_became_active":"2024-09-13T15:07:22.005237+0000","last_became_peered":"2024-09-13T15:07:22.005237+0000","last_unstale":"2024-09-13T15:08:22.037069+0000","last_undegraded":"2024-09-13T15:08:22.037069+0000","last_fullsized":"2024-09-13T15:08:22.037069+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:13:44.182858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":81,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.036860+0000","last_change":"2024-09-13T15:07:15.995884+0000","last_active":"2024-09-13T15:08:22.036860+0000","last_peered":"2024-09-13T15:08:22.036860+0000","last_clean":"2024-09-13T15:08:22.036860+0000","last_became_active":"2024-09-13T15:07:15.995475+0000","last_became_peered":"2024-09-13T15:07:15.995475+0000","last_unstale":"2024-09-13T15:08:22.036860+0000","last_undegraded":"2024-09-13T15:08:22.036860+0000","last_fullsized":"2024-09-13T15:08:22.036860+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:27.705697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":86,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.061080+0000","last_change":"2024-09-13T15:06:53.982720+0000","last_active":"2024-09-13T15:08:18.061080+0000","last_peered":"2024-09-13T15:08:18.061080+0000","last_clean":"2024-09-13T15:08:18.061080+0000","last_became_active":"2024-09-13T15:06:53.982103+0000","last_became_peered":"2024-09-13T15:06:53.982103+0000","last_unstale":"2024-09-13T15:08:18.061080+0000","last_undegraded":"2024-09-13T15:08:18.061080+0000","last_fullsized":"2024-09-13T15:08:18.061080+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:46:54.675404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"261'100","reported_seq":380,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.007518+0000","last_change":"2024-09-13T15:07:15.998937+0000","last_active":"2024-09-13T15:08:22.007518+0000","last_peered":"2024-09-13T15:08:22.007518+0000","last_clean":"2024-09-13T15:08:22.007518+0000","last_became_active":"2024-09-13T15:07:15.998454+0000","last_became_peered":"2024-09-13T15:07:15.998454+0000","last_unstale":"2024-09-13T15:08:22.007518+0000","last_undegraded":"2024-09-13T15:08:22.007518+0000","last_fullsized":"2024-09-13T15:08:22.007518+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.008264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.008264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:51:19.930403+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1063,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:19.004226+0000","last_change":"2024-09-13T15:08:16.058775+0000","last_active":"2024-09-13T15:08:19.004226+0000","last_peered":"2024-09-13T15:08:19.004226+0000","last_clean":"2024-09-13T15:08:19.004226+0000","last_became_active":"2024-09-13T15:08:16.058426+0000","last_became_peered":"2024-09-13T15:08:16.058426+0000","last_unstale":"2024-09-13T15:08:19.004226+0000","last_undegraded":"2024-09-13T15:08:19.004226+0000","last_fullsized":"2024-09-13T15:08:19.004226+0000","mapping_epoch":1391,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:16.361720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":547,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089258+0000","last_change":"2024-09-13T14:55:41.835887+0000","last_active":"2024-09-13T15:08:22.089258+0000","last_peered":"2024-09-13T15:08:22.089258+0000","last_clean":"2024-09-13T15:08:22.089258+0000","last_became_active":"2024-09-13T14:55:41.834413+0000","last_became_peered":"2024-09-13T14:55:41.834413+0000","last_unstale":"2024-09-13T15:08:22.089258+0000","last_undegraded":"2024-09-13T15:08:22.089258+0000","last_fullsized":"2024-09-13T15:08:22.089258+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:01.850578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":263,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.681952+0000","last_change":"2024-09-13T15:07:22.025908+0000","last_active":"2024-09-13T15:08:18.681952+0000","last_peered":"2024-09-13T15:08:18.681952+0000","last_clean":"2024-09-13T15:08:18.681952+0000","last_became_active":"2024-09-13T15:07:22.013151+0000","last_became_peered":"2024-09-13T15:07:22.013151+0000","last_unstale":"2024-09-13T15:08:18.681952+0000","last_undegraded":"2024-09-13T15:08:18.681952+0000","last_fullsized":"2024-09-13T15:08:18.681952+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:02:47.167554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"261'100","reported_seq":86,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.061147+0000","last_change":"2024-09-13T15:06:53.983024+0000","last_active":"2024-09-13T15:08:18.061147+0000","last_peered":"2024-09-13T15:08:18.061147+0000","last_clean":"2024-09-13T15:08:18.061147+0000","last_became_active":"2024-09-13T15:06:53.982602+0000","last_became_peered":"2024-09-13T15:06:53.982602+0000","last_unstale":"2024-09-13T15:08:18.061147+0000","last_undegraded":"2024-09-13T15:08:18.061147+0000","last_fullsized":"2024-09-13T15:08:18.061147+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T15:00:44.119431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T15:00:44.119431+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:03:24.572217+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":386,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.007781+0000","last_change":"2024-09-13T15:07:15.998107+0000","last_active":"2024-09-13T15:08:22.007781+0000","last_peered":"2024-09-13T15:08:22.007781+0000","last_clean":"2024-09-13T15:08:22.007781+0000","last_became_active":"2024-09-13T15:07:15.997717+0000","last_became_peered":"2024-09-13T15:07:15.997717+0000","last_unstale":"2024-09-13T15:08:22.007781+0000","last_undegraded":"2024-09-13T15:08:22.007781+0000","last_fullsized":"2024-09-13T15:08:22.007781+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:40:14.840576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":553,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089209+0000","last_change":"2024-09-13T15:02:42.901428+0000","last_active":"2024-09-13T15:08:22.089209+0000","last_peered":"2024-09-13T15:08:22.089209+0000","last_clean":"2024-09-13T15:08:22.089209+0000","last_became_active":"2024-09-13T15:02:42.901131+0000","last_became_peered":"2024-09-13T15:02:42.901131+0000","last_unstale":"2024-09-13T15:08:22.089209+0000","last_undegraded":"2024-09-13T15:08:22.089209+0000","last_fullsized":"2024-09-13T15:08:22.089209+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:44.152548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"261'100","reported_seq":206,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.681766+0000","last_change":"2024-09-13T15:07:22.013339+0000","last_active":"2024-09-13T15:08:18.681766+0000","last_peered":"2024-09-13T15:08:18.681766+0000","last_clean":"2024-09-13T15:08:18.681766+0000","last_became_active":"2024-09-13T15:07:22.012961+0000","last_became_peered":"2024-09-13T15:07:22.012961+0000","last_unstale":"2024-09-13T15:08:18.681766+0000","last_undegraded":"2024-09-13T15:08:18.681766+0000","last_fullsized":"2024-09-13T15:08:18.681766+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:49.280478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:49.280478+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:42.191955+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":31,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.059187+0000","last_change":"2024-09-13T15:07:16.005652+0000","last_active":"2024-09-13T15:08:18.059187+0000","last_peered":"2024-09-13T15:08:18.059187+0000","last_clean":"2024-09-13T15:08:18.059187+0000","last_became_active":"2024-09-13T15:07:16.005341+0000","last_became_peered":"2024-09-13T15:07:16.005341+0000","last_unstale":"2024-09-13T15:08:18.059187+0000","last_undegraded":"2024-09-13T15:08:18.059187+0000","last_fullsized":"2024-09-13T15:08:18.059187+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:43.855261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":41,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.058785+0000","last_change":"2024-09-13T15:07:22.007141+0000","last_active":"2024-09-13T15:08:18.058785+0000","last_peered":"2024-09-13T15:08:18.058785+0000","last_clean":"2024-09-13T15:08:18.058785+0000","last_became_active":"2024-09-13T15:07:22.006715+0000","last_became_peered":"2024-09-13T15:07:22.006715+0000","last_unstale":"2024-09-13T15:08:18.058785+0000","last_undegraded":"2024-09-13T15:08:18.058785+0000","last_fullsized":"2024-09-13T15:08:18.058785+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:38:53.023680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"261'100","reported_seq":393,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.007014+0000","last_change":"2024-09-13T15:06:53.991344+0000","last_active":"2024-09-13T15:08:22.007014+0000","last_peered":"2024-09-13T15:08:22.007014+0000","last_clean":"2024-09-13T15:08:22.007014+0000","last_became_active":"2024-09-13T15:06:53.987397+0000","last_became_peered":"2024-09-13T15:06:53.987397+0000","last_unstale":"2024-09-13T15:08:22.007014+0000","last_undegraded":"2024-09-13T15:08:22.007014+0000","last_fullsized":"2024-09-13T15:08:22.007014+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:47.405845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:47.405845+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:57:33.181023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":41,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.059341+0000","last_change":"2024-09-13T15:07:22.002515+0000","last_active":"2024-09-13T15:08:18.059341+0000","last_peered":"2024-09-13T15:08:18.059341+0000","last_clean":"2024-09-13T15:08:18.059341+0000","last_became_active":"2024-09-13T15:07:22.002149+0000","last_became_peered":"2024-09-13T15:07:22.002149+0000","last_unstale":"2024-09-13T15:08:18.059341+0000","last_undegraded":"2024-09-13T15:08:18.059341+0000","last_fullsized":"2024-09-13T15:08:18.059341+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:26:12.683816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1122,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:19.003998+0000","last_change":"2024-09-13T15:08:16.108248+0000","last_active":"2024-09-13T15:08:19.003998+0000","last_peered":"2024-09-13T15:08:19.003998+0000","last_clean":"2024-09-13T15:08:19.003998+0000","last_became_active":"2024-09-13T15:08:16.107920+0000","last_became_peered":"2024-09-13T15:08:16.107920+0000","last_unstale":"2024-09-13T15:08:19.003998+0000","last_undegraded":"2024-09-13T15:08:19.003998+0000","last_fullsized":"2024-09-13T15:08:19.003998+0000","mapping_epoch":1391,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:49.517472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"261'100","reported_seq":186,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.681535+0000","last_change":"2024-09-13T15:07:22.013902+0000","last_active":"2024-09-13T15:08:18.681535+0000","last_peered":"2024-09-13T15:08:18.681535+0000","last_clean":"2024-09-13T15:08:18.681535+0000","last_became_active":"2024-09-13T15:07:22.013483+0000","last_became_peered":"2024-09-13T15:07:22.013483+0000","last_unstale":"2024-09-13T15:08:18.681535+0000","last_undegraded":"2024-09-13T15:08:18.681535+0000","last_fullsized":"2024-09-13T15:08:18.681535+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:55.816857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:55.816857+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:01.392722+0000","scrub_duration":270,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":31,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.059004+0000","last_change":"2024-09-13T15:07:16.005711+0000","last_active":"2024-09-13T15:08:18.059004+0000","last_peered":"2024-09-13T15:08:18.059004+0000","last_clean":"2024-09-13T15:08:18.059004+0000","last_became_active":"2024-09-13T15:07:16.005412+0000","last_became_peered":"2024-09-13T15:07:16.005412+0000","last_unstale":"2024-09-13T15:08:18.059004+0000","last_undegraded":"2024-09-13T15:08:18.059004+0000","last_fullsized":"2024-09-13T15:08:18.059004+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:57:22.027709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":614,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089317+0000","last_change":"2024-09-13T15:07:16.020821+0000","last_active":"2024-09-13T15:08:22.089317+0000","last_peered":"2024-09-13T15:08:22.089317+0000","last_clean":"2024-09-13T15:08:22.089317+0000","last_became_active":"2024-09-13T15:07:16.006351+0000","last_became_peered":"2024-09-13T15:07:16.006351+0000","last_unstale":"2024-09-13T15:08:22.089317+0000","last_undegraded":"2024-09-13T15:08:22.089317+0000","last_fullsized":"2024-09-13T15:08:22.089317+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:26.498892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"261'100","reported_seq":106,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.061075+0000","last_change":"2024-09-13T15:07:22.004190+0000","last_active":"2024-09-13T15:08:18.061075+0000","last_peered":"2024-09-13T15:08:18.061075+0000","last_clean":"2024-09-13T15:08:18.061075+0000","last_became_active":"2024-09-13T15:07:22.003796+0000","last_became_peered":"2024-09-13T15:07:22.003796+0000","last_unstale":"2024-09-13T15:08:18.061075+0000","last_undegraded":"2024-09-13T15:08:18.061075+0000","last_fullsized":"2024-09-13T15:08:18.061075+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T15:02:56.904689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T15:02:56.904689+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:01:18.911718+0000","scrub_duration":0.001699702,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":80,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.036972+0000","last_change":"2024-09-13T15:07:15.996439+0000","last_active":"2024-09-13T15:08:22.036972+0000","last_peered":"2024-09-13T15:08:22.036972+0000","last_clean":"2024-09-13T15:08:22.036972+0000","last_became_active":"2024-09-13T15:07:15.996083+0000","last_became_peered":"2024-09-13T15:07:15.996083+0000","last_unstale":"2024-09-13T15:08:22.036972+0000","last_undegraded":"2024-09-13T15:08:22.036972+0000","last_fullsized":"2024-09-13T15:08:22.036972+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:12:21.867991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":166,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.682008+0000","last_change":"2024-09-13T14:55:41.848811+0000","last_active":"2024-09-13T15:08:18.682008+0000","last_peered":"2024-09-13T15:08:18.682008+0000","last_clean":"2024-09-13T15:08:18.682008+0000","last_became_active":"2024-09-13T14:55:41.848542+0000","last_became_peered":"2024-09-13T14:55:41.848542+0000","last_unstale":"2024-09-13T15:08:18.682008+0000","last_undegraded":"2024-09-13T15:08:18.682008+0000","last_fullsized":"2024-09-13T15:08:18.682008+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:48.534022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"261'100","reported_seq":31,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.059236+0000","last_change":"2024-09-13T15:07:16.000211+0000","last_active":"2024-09-13T15:08:18.059236+0000","last_peered":"2024-09-13T15:08:18.059236+0000","last_clean":"2024-09-13T15:08:18.059236+0000","last_became_active":"2024-09-13T15:07:15.998798+0000","last_became_peered":"2024-09-13T15:07:15.998798+0000","last_unstale":"2024-09-13T15:08:18.059236+0000","last_undegraded":"2024-09-13T15:08:18.059236+0000","last_fullsized":"2024-09-13T15:08:18.059236+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T15:03:00.851236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T15:03:00.851236+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:50:35.250688+0000","scrub_duration":0.004159968,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1409,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:19.003414+0000","last_change":"2024-09-13T15:02:42.880814+0000","last_active":"2024-09-13T15:08:19.003414+0000","last_peered":"2024-09-13T15:08:19.003414+0000","last_clean":"2024-09-13T15:08:19.003414+0000","last_became_active":"2024-09-13T15:02:42.880393+0000","last_became_peered":"2024-09-13T15:02:42.880393+0000","last_unstale":"2024-09-13T15:08:19.003414+0000","last_undegraded":"2024-09-13T15:08:19.003414+0000","last_fullsized":"2024-09-13T15:08:19.003414+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:03.222968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":80,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.061596+0000","last_change":"2024-09-13T15:02:42.914851+0000","last_active":"2024-09-13T15:08:18.061596+0000","last_peered":"2024-09-13T15:08:18.061596+0000","last_clean":"2024-09-13T15:08:18.061596+0000","last_became_active":"2024-09-13T15:02:42.884164+0000","last_became_peered":"2024-09-13T15:02:42.884164+0000","last_unstale":"2024-09-13T15:08:18.061596+0000","last_undegraded":"2024-09-13T15:08:18.061596+0000","last_fullsized":"2024-09-13T15:08:18.061596+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:26:54.412326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"261'100","reported_seq":408,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:24.001118+0000","last_change":"2024-09-13T15:08:24.001087+0000","last_active":"2024-09-13T15:08:24.001118+0000","last_peered":"2024-09-13T15:08:24.001118+0000","last_clean":"2024-09-13T15:08:24.001118+0000","last_became_active":"2024-09-13T15:08:16.058475+0000","last_became_peered":"2024-09-13T15:08:16.058475+0000","last_unstale":"2024-09-13T15:08:24.001118+0000","last_undegraded":"2024-09-13T15:08:24.001118+0000","last_fullsized":"2024-09-13T15:08:24.001118+0000","mapping_epoch":1391,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1392,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T15:08:24.001016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T15:08:24.001016+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:54:01.192884+0000","scrub_duration":0.0015859839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":31,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.058590+0000","last_change":"2024-09-13T15:07:16.003278+0000","last_active":"2024-09-13T15:08:18.058590+0000","last_peered":"2024-09-13T15:08:18.058590+0000","last_clean":"2024-09-13T15:08:18.058590+0000","last_became_active":"2024-09-13T15:07:16.002938+0000","last_became_peered":"2024-09-13T15:07:16.002938+0000","last_unstale":"2024-09-13T15:08:18.058590+0000","last_undegraded":"2024-09-13T15:08:18.058590+0000","last_fullsized":"2024-09-13T15:08:18.058590+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:52.151311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":108,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.061565+0000","last_change":"2024-09-13T15:07:16.021982+0000","last_active":"2024-09-13T15:08:18.061565+0000","last_peered":"2024-09-13T15:08:18.061565+0000","last_clean":"2024-09-13T15:08:18.061565+0000","last_became_active":"2024-09-13T15:07:15.998901+0000","last_became_peered":"2024-09-13T15:07:15.998901+0000","last_unstale":"2024-09-13T15:08:18.061565+0000","last_undegraded":"2024-09-13T15:08:18.061565+0000","last_fullsized":"2024-09-13T15:08:18.061565+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:28:38.132787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"261'100","reported_seq":554,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089536+0000","last_change":"2024-09-13T15:06:54.015766+0000","last_active":"2024-09-13T15:08:22.089536+0000","last_peered":"2024-09-13T15:08:22.089536+0000","last_clean":"2024-09-13T15:08:22.089536+0000","last_became_active":"2024-09-13T15:06:53.989009+0000","last_became_peered":"2024-09-13T15:06:53.989009+0000","last_unstale":"2024-09-13T15:08:22.089536+0000","last_undegraded":"2024-09-13T15:08:22.089536+0000","last_fullsized":"2024-09-13T15:08:22.089536+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.597568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.597568+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:25.025847+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":196,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.681633+0000","last_change":"2024-09-13T15:02:42.885662+0000","last_active":"2024-09-13T15:08:18.681633+0000","last_peered":"2024-09-13T15:08:18.681633+0000","last_clean":"2024-09-13T15:08:18.681633+0000","last_became_active":"2024-09-13T15:02:42.885326+0000","last_became_peered":"2024-09-13T15:02:42.885326+0000","last_unstale":"2024-09-13T15:08:18.681633+0000","last_undegraded":"2024-09-13T15:08:18.681633+0000","last_fullsized":"2024-09-13T15:08:18.681633+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:11.631110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":108,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.061714+0000","last_change":"2024-09-13T15:07:16.025552+0000","last_active":"2024-09-13T15:08:18.061714+0000","last_peered":"2024-09-13T15:08:18.061714+0000","last_clean":"2024-09-13T15:08:18.061714+0000","last_became_active":"2024-09-13T15:07:15.999224+0000","last_became_peered":"2024-09-13T15:07:15.999224+0000","last_unstale":"2024-09-13T15:08:18.061714+0000","last_undegraded":"2024-09-13T15:08:18.061714+0000","last_fullsized":"2024-09-13T15:08:18.061714+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:23.250695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"261'100","reported_seq":96,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.061571+0000","last_change":"2024-09-13T15:07:16.000056+0000","last_active":"2024-09-13T15:08:18.061571+0000","last_peered":"2024-09-13T15:08:18.061571+0000","last_clean":"2024-09-13T15:08:18.061571+0000","last_became_active":"2024-09-13T15:07:15.999193+0000","last_became_peered":"2024-09-13T15:07:15.999193+0000","last_unstale":"2024-09-13T15:08:18.061571+0000","last_undegraded":"2024-09-13T15:08:18.061571+0000","last_fullsized":"2024-09-13T15:08:18.061571+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.191326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.191326+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:09:34.018627+0000","scrub_duration":0.0025317360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1458,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:19.003715+0000","last_change":"2024-09-13T15:06:54.005515+0000","last_active":"2024-09-13T15:08:19.003715+0000","last_peered":"2024-09-13T15:08:19.003715+0000","last_clean":"2024-09-13T15:08:19.003715+0000","last_became_active":"2024-09-13T15:06:53.982127+0000","last_became_peered":"2024-09-13T15:06:53.982127+0000","last_unstale":"2024-09-13T15:08:19.003715+0000","last_undegraded":"2024-09-13T15:08:19.003715+0000","last_fullsized":"2024-09-13T15:08:19.003715+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:47.964603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":545,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089748+0000","last_change":"2024-09-13T15:07:07.603412+0000","last_active":"2024-09-13T15:08:22.089748+0000","last_peered":"2024-09-13T15:08:22.089748+0000","last_clean":"2024-09-13T15:08:22.089748+0000","last_became_active":"2024-09-13T15:07:07.603069+0000","last_became_peered":"2024-09-13T15:07:07.603069+0000","last_unstale":"2024-09-13T15:08:22.089748+0000","last_undegraded":"2024-09-13T15:08:22.089748+0000","last_fullsized":"2024-09-13T15:08:22.089748+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:54:02.275494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"261'100","reported_seq":166,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.681896+0000","last_change":"2024-09-13T14:55:41.889497+0000","last_active":"2024-09-13T15:08:18.681896+0000","last_peered":"2024-09-13T15:08:18.681896+0000","last_clean":"2024-09-13T15:08:18.681896+0000","last_became_active":"2024-09-13T14:55:41.888766+0000","last_became_peered":"2024-09-13T14:55:41.888766+0000","last_unstale":"2024-09-13T15:08:18.681896+0000","last_undegraded":"2024-09-13T15:08:18.681896+0000","last_fullsized":"2024-09-13T15:08:18.681896+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:55.378404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:55.378404+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:50.289550+0000","scrub_duration":0.0016008439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":96,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.061074+0000","last_change":"2024-09-13T15:07:15.990866+0000","last_active":"2024-09-13T15:08:18.061074+0000","last_peered":"2024-09-13T15:08:18.061074+0000","last_clean":"2024-09-13T15:08:18.061074+0000","last_became_active":"2024-09-13T15:07:15.990549+0000","last_became_peered":"2024-09-13T15:07:15.990549+0000","last_unstale":"2024-09-13T15:08:18.061074+0000","last_undegraded":"2024-09-13T15:08:18.061074+0000","last_fullsized":"2024-09-13T15:08:18.061074+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:41:03.478002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":73,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.036469+0000","last_change":"2024-09-13T15:06:53.995604+0000","last_active":"2024-09-13T15:08:22.036469+0000","last_peered":"2024-09-13T15:08:22.036469+0000","last_clean":"2024-09-13T15:08:22.036469+0000","last_became_active":"2024-09-13T15:06:53.995306+0000","last_became_peered":"2024-09-13T15:06:53.995306+0000","last_unstale":"2024-09-13T15:08:22.036469+0000","last_undegraded":"2024-09-13T15:08:22.036469+0000","last_fullsized":"2024-09-13T15:08:22.036469+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:09:34.918000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"261'100","reported_seq":386,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.007425+0000","last_change":"2024-09-13T15:07:15.999275+0000","last_active":"2024-09-13T15:08:22.007425+0000","last_peered":"2024-09-13T15:08:22.007425+0000","last_clean":"2024-09-13T15:08:22.007425+0000","last_became_active":"2024-09-13T15:07:15.997916+0000","last_became_peered":"2024-09-13T15:07:15.997916+0000","last_unstale":"2024-09-13T15:08:22.007425+0000","last_undegraded":"2024-09-13T15:08:22.007425+0000","last_fullsized":"2024-09-13T15:08:22.007425+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:19.978464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:19.978464+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:57:24.878718+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":73,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.036426+0000","last_change":"2024-09-13T15:06:53.982939+0000","last_active":"2024-09-13T15:08:22.036426+0000","last_peered":"2024-09-13T15:08:22.036426+0000","last_clean":"2024-09-13T15:08:22.036426+0000","last_became_active":"2024-09-13T15:06:53.982631+0000","last_became_peered":"2024-09-13T15:06:53.982631+0000","last_unstale":"2024-09-13T15:08:22.036426+0000","last_undegraded":"2024-09-13T15:08:22.036426+0000","last_fullsized":"2024-09-13T15:08:22.036426+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:43:11.173393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":186,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.682223+0000","last_change":"2024-09-13T15:02:42.886038+0000","last_active":"2024-09-13T15:08:18.682223+0000","last_peered":"2024-09-13T15:08:18.682223+0000","last_clean":"2024-09-13T15:08:18.682223+0000","last_became_active":"2024-09-13T15:02:42.884438+0000","last_became_peered":"2024-09-13T15:02:42.884438+0000","last_unstale":"2024-09-13T15:08:18.682223+0000","last_undegraded":"2024-09-13T15:08:18.682223+0000","last_fullsized":"2024-09-13T15:08:18.682223+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:47:39.175275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"261'100","reported_seq":196,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.681593+0000","last_change":"2024-09-13T15:02:42.885131+0000","last_active":"2024-09-13T15:08:18.681593+0000","last_peered":"2024-09-13T15:08:18.681593+0000","last_clean":"2024-09-13T15:08:18.681593+0000","last_became_active":"2024-09-13T15:02:42.883904+0000","last_became_peered":"2024-09-13T15:02:42.883904+0000","last_unstale":"2024-09-13T15:08:18.681593+0000","last_undegraded":"2024-09-13T15:08:18.681593+0000","last_fullsized":"2024-09-13T15:08:18.681593+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:50.327393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:50.327393+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:11:05.603544+0000","scrub_duration":0.0011508309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":146,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.061285+0000","last_change":"2024-09-13T15:07:22.021811+0000","last_active":"2024-09-13T15:08:18.061285+0000","last_peered":"2024-09-13T15:08:18.061285+0000","last_clean":"2024-09-13T15:08:18.061285+0000","last_became_active":"2024-09-13T15:07:22.007340+0000","last_became_peered":"2024-09-13T15:07:22.007340+0000","last_unstale":"2024-09-13T15:08:18.061285+0000","last_undegraded":"2024-09-13T15:08:18.061285+0000","last_fullsized":"2024-09-13T15:08:18.061285+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:35:59.082302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":170,"num_keys_recovered":50,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":34,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.681888+0000","last_change":"2024-09-13T15:07:07.379986+0000","last_active":"2024-09-13T15:08:18.681888+0000","last_peered":"2024-09-13T15:08:18.681888+0000","last_clean":"2024-09-13T15:08:18.681888+0000","last_became_active":"2024-09-13T15:07:07.379648+0000","last_became_peered":"2024-09-13T15:07:07.379648+0000","last_unstale":"2024-09-13T15:08:18.681888+0000","last_undegraded":"2024-09-13T15:08:18.681888+0000","last_fullsized":"2024-09-13T15:08:18.681888+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:18.037587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"261'100","reported_seq":73,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.036430+0000","last_change":"2024-09-13T15:06:53.986158+0000","last_active":"2024-09-13T15:08:22.036430+0000","last_peered":"2024-09-13T15:08:22.036430+0000","last_clean":"2024-09-13T15:08:22.036430+0000","last_became_active":"2024-09-13T15:06:53.985844+0000","last_became_peered":"2024-09-13T15:06:53.985844+0000","last_unstale":"2024-09-13T15:08:22.036430+0000","last_undegraded":"2024-09-13T15:08:22.036430+0000","last_fullsized":"2024-09-13T15:08:22.036430+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:14.654603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:14.654603+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:50:16.010750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":28216,"num_keys_recovered":50,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2886848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430829,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1490236,"kb_used_data":9264,"kb_used_omap":1,"kb_used_meta":1480958,"kb_avail":92254916,"statfs":{"total":95995035648,"available":94469033984,"internally_reserved":0,"allocated":9486336,"data_stored":6136523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1516501336},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388627058,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1496300,"kb_used_data":9316,"kb_used_omap":1,"kb_used_meta":1486974,"kb_avail":92248852,"statfs":{"total":95995035648,"available":94462824448,"internally_reserved":0,"allocated":9539584,"data_stored":6201388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1522661720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823287,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1503212,"kb_used_data":9248,"kb_used_omap":1,"kb_used_meta":1493950,"kb_avail":92241940,"statfs":{"total":95995035648,"available":94455746560,"internally_reserved":0,"allocated":9469952,"data_stored":6132222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1529805144},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019515,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":913012,"kb_used_data":9000,"kb_used_omap":1,"kb_used_meta":903998,"kb_avail":92832140,"statfs":{"total":95995035648,"available":95060111360,"internally_reserved":0,"allocated":9216000,"data_stored":5873765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":925694296},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059593173,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1039520,"kb_used_data":9300,"kb_used_omap":26,"kb_used_meta":1030181,"kb_avail":92705632,"statfs":{"total":95995035648,"available":94930567168,"internally_reserved":0,"allocated":9523200,"data_stored":6196657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":1054905504},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":591,"seq":2538325672411,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1139364,"kb_used_data":72540,"kb_used_omap":41,"kb_used_meta":1066774,"kb_avail":92605788,"statfs":{"total":95995035648,"available":94828326912,"internally_reserved":0,"allocated":74280960,"data_stored":70947283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42651,"internal_metadata":1092376933},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":557,"seq":2392296784354,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1185504,"kb_used_data":8980,"kb_used_omap":32,"kb_used_meta":1176479,"kb_avail":92559648,"statfs":{"total":95995035648,"available":94781079552,"internally_reserved":0,"allocated":9195520,"data_stored":5868906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33752,"internal_metadata":1204714536},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":524,"seq":2250562863593,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":871588,"kb_used_data":9304,"kb_used_omap":41,"kb_used_meta":862230,"kb_avail":92873564,"statfs":{"total":95995035648,"available":95102529536,"internally_reserved":0,"allocated":9527296,"data_stored":6197159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42694,"internal_metadata":882923834},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:09:08.274 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2401fc50-71da-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:09:08.519 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:09.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:09:09 smithi184 ceph-mon[92527]: from='client.46958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:09:09.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:09:09 smithi184 ceph-mon[92527]: pgmap v2491: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:09:09.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:09:09 smithi151 ceph-mon[104450]: from='client.46958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:09:09.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:09:09 smithi151 ceph-mon[104450]: pgmap v2491: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:09:09.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:09:09 smithi151 ceph-mon[108966]: from='client.46958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:09:09.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:09:09 smithi151 ceph-mon[108966]: pgmap v2491: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:09:10.424 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-13T15:09:10.425 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-13T15:09:10.604 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:09:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:09:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:09:11.264 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":2492,"stamp":"2024-09-13T15:09:09.993250+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":2915121,"num_keys_recovered":50,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3158,"ondisk_log_size":3158,"up":309,"acting":309,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":311,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":9619536,"kb_used_data":117752,"kb_used_omap":149,"kb_used_meta":9501546,"kb_avail":740341680,"statfs":{"total":767960285184,"available":758109880320,"internally_reserved":0,"allocated":120578048,"data_stored":93893103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":153401,"internal_metadata":9729583303},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002791"},"pg_stats":[{"pgid":"1.1d","version":"261'100","reported_seq":385,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.007628+0000","last_change":"2024-09-13T15:07:15.998428+0000","last_active":"2024-09-13T15:08:22.007628+0000","last_peered":"2024-09-13T15:08:22.007628+0000","last_clean":"2024-09-13T15:08:22.007628+0000","last_became_active":"2024-09-13T15:07:15.997925+0000","last_became_peered":"2024-09-13T15:07:15.997925+0000","last_unstale":"2024-09-13T15:08:22.007628+0000","last_undegraded":"2024-09-13T15:08:22.007628+0000","last_fullsized":"2024-09-13T15:08:22.007628+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:17.988345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:17.988345+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:31:34.794143+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":93,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.037259+0000","last_change":"2024-09-13T15:07:22.018488+0000","last_active":"2024-09-13T15:08:22.037259+0000","last_peered":"2024-09-13T15:08:22.037259+0000","last_clean":"2024-09-13T15:08:22.037259+0000","last_became_active":"2024-09-13T15:07:22.005489+0000","last_became_peered":"2024-09-13T15:07:22.005489+0000","last_unstale":"2024-09-13T15:08:22.037259+0000","last_undegraded":"2024-09-13T15:08:22.037259+0000","last_fullsized":"2024-09-13T15:08:22.037259+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:44:23.083231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3420,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":31,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.058429+0000","last_change":"2024-09-13T15:07:16.006235+0000","last_active":"2024-09-13T15:08:18.058429+0000","last_peered":"2024-09-13T15:08:18.058429+0000","last_clean":"2024-09-13T15:08:18.058429+0000","last_became_active":"2024-09-13T15:07:16.005374+0000","last_became_peered":"2024-09-13T15:07:16.005374+0000","last_unstale":"2024-09-13T15:08:18.058429+0000","last_undegraded":"2024-09-13T15:08:18.058429+0000","last_fullsized":"2024-09-13T15:08:18.058429+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:54:10.663527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":526,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089685+0000","last_change":"2024-09-13T15:07:22.013535+0000","last_active":"2024-09-13T15:08:22.089685+0000","last_peered":"2024-09-13T15:08:22.089685+0000","last_clean":"2024-09-13T15:08:22.089685+0000","last_became_active":"2024-09-13T15:07:22.013156+0000","last_became_peered":"2024-09-13T15:07:22.013156+0000","last_unstale":"2024-09-13T15:08:22.089685+0000","last_undegraded":"2024-09-13T15:08:22.089685+0000","last_fullsized":"2024-09-13T15:08:22.089685+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:10.552250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":538,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089256+0000","last_change":"2024-09-13T15:07:22.008177+0000","last_active":"2024-09-13T15:08:22.089256+0000","last_peered":"2024-09-13T15:08:22.089256+0000","last_clean":"2024-09-13T15:08:22.089256+0000","last_became_active":"2024-09-13T15:07:22.007855+0000","last_became_peered":"2024-09-13T15:07:22.007855+0000","last_unstale":"2024-09-13T15:08:22.089256+0000","last_undegraded":"2024-09-13T15:08:22.089256+0000","last_fullsized":"2024-09-13T15:08:22.089256+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:15:30.634430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"261'100","reported_seq":76,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.061387+0000","last_change":"2024-09-13T15:02:42.889565+0000","last_active":"2024-09-13T15:08:18.061387+0000","last_peered":"2024-09-13T15:08:18.061387+0000","last_clean":"2024-09-13T15:08:18.061387+0000","last_became_active":"2024-09-13T15:02:42.888466+0000","last_became_peered":"2024-09-13T15:02:42.888466+0000","last_unstale":"2024-09-13T15:08:18.061387+0000","last_undegraded":"2024-09-13T15:08:18.061387+0000","last_fullsized":"2024-09-13T15:08:18.061387+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:44:54.259482+0000","last_clean_scrub_stamp":"2024-09-13T14:44:54.259482+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:19:31.567604+0000","scrub_duration":0.015265569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":198,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.682151+0000","last_change":"2024-09-13T15:07:16.019036+0000","last_active":"2024-09-13T15:08:18.682151+0000","last_peered":"2024-09-13T15:08:18.682151+0000","last_clean":"2024-09-13T15:08:18.682151+0000","last_became_active":"2024-09-13T15:07:15.997656+0000","last_became_peered":"2024-09-13T15:07:15.997656+0000","last_unstale":"2024-09-13T15:08:18.682151+0000","last_undegraded":"2024-09-13T15:08:18.682151+0000","last_fullsized":"2024-09-13T15:08:18.682151+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:46.516312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":31,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.058575+0000","last_change":"2024-09-13T15:07:16.001171+0000","last_active":"2024-09-13T15:08:18.058575+0000","last_peered":"2024-09-13T15:08:18.058575+0000","last_clean":"2024-09-13T15:08:18.058575+0000","last_became_active":"2024-09-13T15:07:15.997226+0000","last_became_peered":"2024-09-13T15:07:15.997226+0000","last_unstale":"2024-09-13T15:08:18.058575+0000","last_undegraded":"2024-09-13T15:08:18.058575+0000","last_fullsized":"2024-09-13T15:08:18.058575+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:57:22.574830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"261'100","reported_seq":529,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089762+0000","last_change":"2024-09-13T14:45:35.947332+0000","last_active":"2024-09-13T15:08:22.089762+0000","last_peered":"2024-09-13T15:08:22.089762+0000","last_clean":"2024-09-13T15:08:22.089762+0000","last_became_active":"2024-09-13T14:45:19.810775+0000","last_became_peered":"2024-09-13T14:45:19.810775+0000","last_unstale":"2024-09-13T15:08:22.089762+0000","last_undegraded":"2024-09-13T15:08:22.089762+0000","last_fullsized":"2024-09-13T15:08:22.089762+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:45:35.947265+0000","last_deep_scrub":"261'100","last_deep_scrub_stamp":"2024-09-13T14:43:41.569795+0000","last_clean_scrub_stamp":"2024-09-13T14:45:35.947265+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:00.606370+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":198,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.682123+0000","last_change":"2024-09-13T15:07:16.023529+0000","last_active":"2024-09-13T15:08:18.682123+0000","last_peered":"2024-09-13T15:08:18.682123+0000","last_clean":"2024-09-13T15:08:18.682123+0000","last_became_active":"2024-09-13T15:07:15.998436+0000","last_became_peered":"2024-09-13T15:07:15.998436+0000","last_unstale":"2024-09-13T15:08:18.682123+0000","last_undegraded":"2024-09-13T15:08:18.682123+0000","last_fullsized":"2024-09-13T15:08:18.682123+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:03.197620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":31,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.059031+0000","last_change":"2024-09-13T15:07:16.003785+0000","last_active":"2024-09-13T15:08:18.059031+0000","last_peered":"2024-09-13T15:08:18.059031+0000","last_clean":"2024-09-13T15:08:18.059031+0000","last_became_active":"2024-09-13T15:07:16.003380+0000","last_became_peered":"2024-09-13T15:07:16.003380+0000","last_unstale":"2024-09-13T15:08:18.059031+0000","last_undegraded":"2024-09-13T15:08:18.059031+0000","last_fullsized":"2024-09-13T15:08:18.059031+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:00.677156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":80,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.037139+0000","last_change":"2024-09-13T15:07:15.993955+0000","last_active":"2024-09-13T15:08:22.037139+0000","last_peered":"2024-09-13T15:08:22.037139+0000","last_clean":"2024-09-13T15:08:22.037139+0000","last_became_active":"2024-09-13T15:07:15.993625+0000","last_became_peered":"2024-09-13T15:07:15.993625+0000","last_unstale":"2024-09-13T15:08:22.037139+0000","last_undegraded":"2024-09-13T15:08:22.037139+0000","last_fullsized":"2024-09-13T15:08:22.037139+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:00.369075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":50,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:21.201903+0000","last_change":"2024-09-13T15:07:22.005002+0000","last_active":"2024-09-13T15:08:21.201903+0000","last_peered":"2024-09-13T15:08:21.201903+0000","last_clean":"2024-09-13T15:08:21.201903+0000","last_became_active":"2024-09-13T15:07:22.004605+0000","last_became_peered":"2024-09-13T15:07:22.004605+0000","last_unstale":"2024-09-13T15:08:21.201903+0000","last_undegraded":"2024-09-13T15:08:21.201903+0000","last_fullsized":"2024-09-13T15:08:21.201903+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:56:34.368320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"261'100","reported_seq":538,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089267+0000","last_change":"2024-09-13T15:06:53.991044+0000","last_active":"2024-09-13T15:08:22.089267+0000","last_peered":"2024-09-13T15:08:22.089267+0000","last_clean":"2024-09-13T15:08:22.089267+0000","last_became_active":"2024-09-13T15:06:53.987755+0000","last_became_peered":"2024-09-13T15:06:53.987755+0000","last_unstale":"2024-09-13T15:08:22.089267+0000","last_undegraded":"2024-09-13T15:08:22.089267+0000","last_fullsized":"2024-09-13T15:08:22.089267+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.614340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.614340+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:26:58.871459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":41,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.058735+0000","last_change":"2024-09-13T15:07:22.008506+0000","last_active":"2024-09-13T15:08:18.058735+0000","last_peered":"2024-09-13T15:08:18.058735+0000","last_clean":"2024-09-13T15:08:18.058735+0000","last_became_active":"2024-09-13T15:07:22.007790+0000","last_became_peered":"2024-09-13T15:07:22.007790+0000","last_unstale":"2024-09-13T15:08:18.058735+0000","last_undegraded":"2024-09-13T15:08:18.058735+0000","last_fullsized":"2024-09-13T15:08:18.058735+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:05.415174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":566,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089658+0000","last_change":"2024-09-13T15:08:08.306111+0000","last_active":"2024-09-13T15:08:22.089658+0000","last_peered":"2024-09-13T15:08:22.089658+0000","last_clean":"2024-09-13T15:08:22.089658+0000","last_became_active":"2024-09-13T15:08:08.305751+0000","last_became_peered":"2024-09-13T15:08:08.305751+0000","last_unstale":"2024-09-13T15:08:22.089658+0000","last_undegraded":"2024-09-13T15:08:22.089658+0000","last_fullsized":"2024-09-13T15:08:22.089658+0000","mapping_epoch":1388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:10:05.198808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.18","version":"261'100","reported_seq":229,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.681715+0000","last_change":"2024-09-13T15:07:22.024346+0000","last_active":"2024-09-13T15:08:18.681715+0000","last_peered":"2024-09-13T15:08:18.681715+0000","last_clean":"2024-09-13T15:08:18.681715+0000","last_became_active":"2024-09-13T15:07:22.008541+0000","last_became_peered":"2024-09-13T15:07:22.008541+0000","last_unstale":"2024-09-13T15:08:18.681715+0000","last_undegraded":"2024-09-13T15:08:18.681715+0000","last_fullsized":"2024-09-13T15:08:18.681715+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:56.548855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:56.548855+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:43:23.766548+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1576128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":379,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.007208+0000","last_change":"2024-09-13T15:07:15.997884+0000","last_active":"2024-09-13T15:08:22.007208+0000","last_peered":"2024-09-13T15:08:22.007208+0000","last_clean":"2024-09-13T15:08:22.007208+0000","last_became_active":"2024-09-13T15:07:15.997533+0000","last_became_peered":"2024-09-13T15:07:15.997533+0000","last_unstale":"2024-09-13T15:08:22.007208+0000","last_undegraded":"2024-09-13T15:08:22.007208+0000","last_fullsized":"2024-09-13T15:08:22.007208+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:08:37.966159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":74,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.036653+0000","last_change":"2024-09-13T15:06:53.984474+0000","last_active":"2024-09-13T15:08:22.036653+0000","last_peered":"2024-09-13T15:08:22.036653+0000","last_clean":"2024-09-13T15:08:22.036653+0000","last_became_active":"2024-09-13T15:06:53.983733+0000","last_became_peered":"2024-09-13T15:06:53.983733+0000","last_unstale":"2024-09-13T15:08:22.036653+0000","last_undegraded":"2024-09-13T15:08:22.036653+0000","last_fullsized":"2024-09-13T15:08:22.036653+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:19:24.594408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1b","version":"261'100","reported_seq":226,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.681906+0000","last_change":"2024-09-13T15:07:16.002022+0000","last_active":"2024-09-13T15:08:18.681906+0000","last_peered":"2024-09-13T15:08:18.681906+0000","last_clean":"2024-09-13T15:08:18.681906+0000","last_became_active":"2024-09-13T15:07:16.001656+0000","last_became_peered":"2024-09-13T15:07:16.001656+0000","last_unstale":"2024-09-13T15:08:18.681906+0000","last_undegraded":"2024-09-13T15:08:18.681906+0000","last_fullsized":"2024-09-13T15:08:18.681906+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:55.988431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:55.988431+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:36:09.915389+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":545,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089117+0000","last_change":"2024-09-13T15:06:53.990683+0000","last_active":"2024-09-13T15:08:22.089117+0000","last_peered":"2024-09-13T15:08:22.089117+0000","last_clean":"2024-09-13T15:08:22.089117+0000","last_became_active":"2024-09-13T15:06:53.988100+0000","last_became_peered":"2024-09-13T15:06:53.988100+0000","last_unstale":"2024-09-13T15:08:22.089117+0000","last_undegraded":"2024-09-13T15:08:22.089117+0000","last_fullsized":"2024-09-13T15:08:22.089117+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:03.162701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1100,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:19.004385+0000","last_change":"2024-09-13T15:08:16.108837+0000","last_active":"2024-09-13T15:08:19.004385+0000","last_peered":"2024-09-13T15:08:19.004385+0000","last_clean":"2024-09-13T15:08:19.004385+0000","last_became_active":"2024-09-13T15:08:16.108299+0000","last_became_peered":"2024-09-13T15:08:16.108299+0000","last_unstale":"2024-09-13T15:08:19.004385+0000","last_undegraded":"2024-09-13T15:08:19.004385+0000","last_fullsized":"2024-09-13T15:08:19.004385+0000","mapping_epoch":1391,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:11:37.949098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1a","version":"261'100","reported_seq":31,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.058700+0000","last_change":"2024-09-13T15:07:15.998441+0000","last_active":"2024-09-13T15:08:18.058700+0000","last_peered":"2024-09-13T15:08:18.058700+0000","last_clean":"2024-09-13T15:08:18.058700+0000","last_became_active":"2024-09-13T15:07:15.998094+0000","last_became_peered":"2024-09-13T15:07:15.998094+0000","last_unstale":"2024-09-13T15:08:18.058700+0000","last_undegraded":"2024-09-13T15:08:18.058700+0000","last_fullsized":"2024-09-13T15:08:18.058700+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:46.985510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:46.985510+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:54:26.582536+0000","scrub_duration":0.0030525999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":532,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089216+0000","last_change":"2024-09-13T14:54:05.606882+0000","last_active":"2024-09-13T15:08:22.089216+0000","last_peered":"2024-09-13T15:08:22.089216+0000","last_clean":"2024-09-13T15:08:22.089216+0000","last_became_active":"2024-09-13T14:54:05.605912+0000","last_became_peered":"2024-09-13T14:54:05.605912+0000","last_unstale":"2024-09-13T15:08:22.089216+0000","last_undegraded":"2024-09-13T15:08:22.089216+0000","last_fullsized":"2024-09-13T15:08:22.089216+0000","mapping_epoch":1222,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:05.445329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":73,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.036484+0000","last_change":"2024-09-13T15:06:53.989318+0000","last_active":"2024-09-13T15:08:22.036484+0000","last_peered":"2024-09-13T15:08:22.036484+0000","last_clean":"2024-09-13T15:08:22.036484+0000","last_became_active":"2024-09-13T15:06:53.986672+0000","last_became_peered":"2024-09-13T15:06:53.986672+0000","last_unstale":"2024-09-13T15:08:22.036484+0000","last_undegraded":"2024-09-13T15:08:22.036484+0000","last_fullsized":"2024-09-13T15:08:22.036484+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:17.142773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"261'100","reported_seq":76,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.061445+0000","last_change":"2024-09-13T15:02:42.892087+0000","last_active":"2024-09-13T15:08:18.061445+0000","last_peered":"2024-09-13T15:08:18.061445+0000","last_clean":"2024-09-13T15:08:18.061445+0000","last_became_active":"2024-09-13T15:02:42.890787+0000","last_became_peered":"2024-09-13T15:02:42.890787+0000","last_unstale":"2024-09-13T15:08:18.061445+0000","last_undegraded":"2024-09-13T15:08:18.061445+0000","last_fullsized":"2024-09-13T15:08:18.061445+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:38.214865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:38.214865+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:12:04.179567+0000","scrub_duration":0.0016389639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2096,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089395+0000","last_change":"2024-09-13T15:02:42.901601+0000","last_active":"2024-09-13T15:08:22.089395+0000","last_peered":"2024-09-13T15:08:22.089395+0000","last_clean":"2024-09-13T15:08:22.089395+0000","last_became_active":"2024-09-13T15:02:42.900518+0000","last_became_peered":"2024-09-13T15:02:42.900518+0000","last_unstale":"2024-09-13T15:08:22.089395+0000","last_undegraded":"2024-09-13T15:08:22.089395+0000","last_fullsized":"2024-09-13T15:08:22.089395+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:57:33.494780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":186,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.682095+0000","last_change":"2024-09-13T15:02:42.885564+0000","last_active":"2024-09-13T15:08:18.682095+0000","last_peered":"2024-09-13T15:08:18.682095+0000","last_clean":"2024-09-13T15:08:18.682095+0000","last_became_active":"2024-09-13T15:02:42.884990+0000","last_became_peered":"2024-09-13T15:02:42.884990+0000","last_unstale":"2024-09-13T15:08:18.682095+0000","last_undegraded":"2024-09-13T15:08:18.682095+0000","last_fullsized":"2024-09-13T15:08:18.682095+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:51:42.556111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"261'100","reported_seq":50,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:21.201930+0000","last_change":"2024-09-13T15:07:22.003039+0000","last_active":"2024-09-13T15:08:21.201930+0000","last_peered":"2024-09-13T15:08:21.201930+0000","last_clean":"2024-09-13T15:08:21.201930+0000","last_became_active":"2024-09-13T15:07:22.002458+0000","last_became_peered":"2024-09-13T15:07:22.002458+0000","last_unstale":"2024-09-13T15:08:21.201930+0000","last_undegraded":"2024-09-13T15:08:21.201930+0000","last_fullsized":"2024-09-13T15:08:21.201930+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:15.037517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:15.037517+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:30:14.518182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":114,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.061401+0000","last_change":"2024-09-13T15:07:16.027851+0000","last_active":"2024-09-13T15:08:18.061401+0000","last_peered":"2024-09-13T15:08:18.061401+0000","last_clean":"2024-09-13T15:08:18.061401+0000","last_became_active":"2024-09-13T15:07:15.998722+0000","last_became_peered":"2024-09-13T15:07:15.998722+0000","last_unstale":"2024-09-13T15:08:18.061401+0000","last_undegraded":"2024-09-13T15:08:18.061401+0000","last_fullsized":"2024-09-13T15:08:18.061401+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:14.744690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":450,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":379,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.007322+0000","last_change":"2024-09-13T15:07:22.008311+0000","last_active":"2024-09-13T15:08:22.007322+0000","last_peered":"2024-09-13T15:08:22.007322+0000","last_clean":"2024-09-13T15:08:22.007322+0000","last_became_active":"2024-09-13T15:07:22.008012+0000","last_became_peered":"2024-09-13T15:07:22.008012+0000","last_unstale":"2024-09-13T15:08:22.007322+0000","last_undegraded":"2024-09-13T15:08:22.007322+0000","last_fullsized":"2024-09-13T15:08:22.007322+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:57.605334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"261'100","reported_seq":196,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.682242+0000","last_change":"2024-09-13T15:06:53.998536+0000","last_active":"2024-09-13T15:08:18.682242+0000","last_peered":"2024-09-13T15:08:18.682242+0000","last_clean":"2024-09-13T15:08:18.682242+0000","last_became_active":"2024-09-13T15:06:53.998173+0000","last_became_peered":"2024-09-13T15:06:53.998173+0000","last_unstale":"2024-09-13T15:08:18.682242+0000","last_undegraded":"2024-09-13T15:08:18.682242+0000","last_fullsized":"2024-09-13T15:08:18.682242+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:54.933410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:54.933410+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:23.765857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"343'22","reported_seq":41,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.059000+0000","last_change":"2024-09-13T15:07:16.017541+0000","last_active":"2024-09-13T15:08:18.059000+0000","last_peered":"2024-09-13T15:08:18.059000+0000","last_clean":"2024-09-13T15:08:18.059000+0000","last_became_active":"2024-09-13T15:07:16.006537+0000","last_became_peered":"2024-09-13T15:07:16.006537+0000","last_unstale":"2024-09-13T15:08:18.059000+0000","last_undegraded":"2024-09-13T15:08:18.059000+0000","last_fullsized":"2024-09-13T15:08:18.059000+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:49.741517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":23720,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":31,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.058824+0000","last_change":"2024-09-13T15:07:16.006027+0000","last_active":"2024-09-13T15:08:18.058824+0000","last_peered":"2024-09-13T15:08:18.058824+0000","last_clean":"2024-09-13T15:08:18.058824+0000","last_became_active":"2024-09-13T15:07:16.005747+0000","last_became_peered":"2024-09-13T15:07:16.005747+0000","last_unstale":"2024-09-13T15:08:18.058824+0000","last_undegraded":"2024-09-13T15:08:18.058824+0000","last_fullsized":"2024-09-13T15:08:18.058824+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:28.371504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"261'100","reported_seq":379,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.007839+0000","last_change":"2024-09-13T15:06:53.990651+0000","last_active":"2024-09-13T15:08:22.007839+0000","last_peered":"2024-09-13T15:08:22.007839+0000","last_clean":"2024-09-13T15:08:22.007839+0000","last_became_active":"2024-09-13T15:06:53.989985+0000","last_became_peered":"2024-09-13T15:06:53.989985+0000","last_unstale":"2024-09-13T15:08:22.007839+0000","last_undegraded":"2024-09-13T15:08:22.007839+0000","last_fullsized":"2024-09-13T15:08:22.007839+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":1121,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:48.363045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:48.363045+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:51.271154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":31,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.058614+0000","last_change":"2024-09-13T15:07:16.005932+0000","last_active":"2024-09-13T15:08:18.058614+0000","last_peered":"2024-09-13T15:08:18.058614+0000","last_clean":"2024-09-13T15:08:18.058614+0000","last_became_active":"2024-09-13T15:07:16.005634+0000","last_became_peered":"2024-09-13T15:07:16.005634+0000","last_unstale":"2024-09-13T15:08:18.058614+0000","last_undegraded":"2024-09-13T15:08:18.058614+0000","last_fullsized":"2024-09-13T15:08:18.058614+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:56:13.989988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1094,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:19.004437+0000","last_change":"2024-09-13T15:08:16.109153+0000","last_active":"2024-09-13T15:08:19.004437+0000","last_peered":"2024-09-13T15:08:19.004437+0000","last_clean":"2024-09-13T15:08:19.004437+0000","last_became_active":"2024-09-13T15:08:16.108623+0000","last_became_peered":"2024-09-13T15:08:16.108623+0000","last_unstale":"2024-09-13T15:08:19.004437+0000","last_undegraded":"2024-09-13T15:08:19.004437+0000","last_fullsized":"2024-09-13T15:08:19.004437+0000","mapping_epoch":1391,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:06.341946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025489000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":89,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.037155+0000","last_change":"2024-09-13T15:07:16.005034+0000","last_active":"2024-09-13T15:08:22.037155+0000","last_peered":"2024-09-13T15:08:22.037155+0000","last_clean":"2024-09-13T15:08:22.037155+0000","last_became_active":"2024-09-13T15:07:15.996110+0000","last_became_peered":"2024-09-13T15:07:15.996110+0000","last_unstale":"2024-09-13T15:08:22.037155+0000","last_undegraded":"2024-09-13T15:08:22.037155+0000","last_fullsized":"2024-09-13T15:08:22.037155+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:17:23.430425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"261'100","reported_seq":554,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089635+0000","last_change":"2024-09-13T15:02:42.899622+0000","last_active":"2024-09-13T15:08:22.089635+0000","last_peered":"2024-09-13T15:08:22.089635+0000","last_clean":"2024-09-13T15:08:22.089635+0000","last_became_active":"2024-09-13T15:02:42.897715+0000","last_became_peered":"2024-09-13T15:02:42.897715+0000","last_unstale":"2024-09-13T15:08:22.089635+0000","last_undegraded":"2024-09-13T15:08:22.089635+0000","last_fullsized":"2024-09-13T15:08:22.089635+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.710192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.710192+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:50:03.858097+0000","scrub_duration":0.0019250910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":443,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089038+0000","last_change":"2024-09-13T15:06:53.990088+0000","last_active":"2024-09-13T15:08:22.089038+0000","last_peered":"2024-09-13T15:08:22.089038+0000","last_clean":"2024-09-13T15:08:22.089038+0000","last_became_active":"2024-09-13T15:06:53.988653+0000","last_became_peered":"2024-09-13T15:06:53.988653+0000","last_unstale":"2024-09-13T15:08:22.089038+0000","last_undegraded":"2024-09-13T15:08:22.089038+0000","last_fullsized":"2024-09-13T15:08:22.089038+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:27:26.438276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":379,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.007801+0000","last_change":"2024-09-13T15:07:15.999008+0000","last_active":"2024-09-13T15:08:22.007801+0000","last_peered":"2024-09-13T15:08:22.007801+0000","last_clean":"2024-09-13T15:08:22.007801+0000","last_became_active":"2024-09-13T15:07:15.998575+0000","last_became_peered":"2024-09-13T15:07:15.998575+0000","last_unstale":"2024-09-13T15:08:22.007801+0000","last_undegraded":"2024-09-13T15:08:22.007801+0000","last_fullsized":"2024-09-13T15:08:22.007801+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:32.354104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":2170,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089021+0000","last_change":"2024-09-13T15:06:54.000491+0000","last_active":"2024-09-13T15:08:22.089021+0000","last_peered":"2024-09-13T15:08:22.089021+0000","last_clean":"2024-09-13T15:08:22.089021+0000","last_became_active":"2024-09-13T15:06:53.987425+0000","last_became_peered":"2024-09-13T15:06:53.987425+0000","last_unstale":"2024-09-13T15:08:22.089021+0000","last_undegraded":"2024-09-13T15:08:22.089021+0000","last_fullsized":"2024-09-13T15:08:22.089021+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:14.972231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"261'100","reported_seq":41,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.059280+0000","last_change":"2024-09-13T15:07:22.007080+0000","last_active":"2024-09-13T15:08:18.059280+0000","last_peered":"2024-09-13T15:08:18.059280+0000","last_clean":"2024-09-13T15:08:18.059280+0000","last_became_active":"2024-09-13T15:07:22.006645+0000","last_became_peered":"2024-09-13T15:07:22.006645+0000","last_unstale":"2024-09-13T15:08:18.059280+0000","last_undegraded":"2024-09-13T15:08:18.059280+0000","last_fullsized":"2024-09-13T15:08:18.059280+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:51:31.310115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:51:31.310115+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:25.468316+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":74,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.036491+0000","last_change":"2024-09-13T15:06:53.987052+0000","last_active":"2024-09-13T15:08:22.036491+0000","last_peered":"2024-09-13T15:08:22.036491+0000","last_clean":"2024-09-13T15:08:22.036491+0000","last_became_active":"2024-09-13T15:06:53.986723+0000","last_became_peered":"2024-09-13T15:06:53.986723+0000","last_unstale":"2024-09-13T15:08:22.036491+0000","last_undegraded":"2024-09-13T15:08:22.036491+0000","last_fullsized":"2024-09-13T15:08:22.036491+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:39.492490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":96,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.061475+0000","last_change":"2024-09-13T15:07:16.000241+0000","last_active":"2024-09-13T15:08:18.061475+0000","last_peered":"2024-09-13T15:08:18.061475+0000","last_clean":"2024-09-13T15:08:18.061475+0000","last_became_active":"2024-09-13T15:07:15.999629+0000","last_became_peered":"2024-09-13T15:07:15.999629+0000","last_unstale":"2024-09-13T15:08:18.061475+0000","last_undegraded":"2024-09-13T15:08:18.061475+0000","last_fullsized":"2024-09-13T15:08:18.061475+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:02.423223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026676800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":394,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.007677+0000","last_change":"2024-09-13T15:07:22.010344+0000","last_active":"2024-09-13T15:08:22.007677+0000","last_peered":"2024-09-13T15:08:22.007677+0000","last_clean":"2024-09-13T15:08:22.007677+0000","last_became_active":"2024-09-13T15:07:22.010045+0000","last_became_peered":"2024-09-13T15:07:22.010045+0000","last_unstale":"2024-09-13T15:08:22.007677+0000","last_undegraded":"2024-09-13T15:08:22.007677+0000","last_fullsized":"2024-09-13T15:08:22.007677+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:33.588135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"261'100","reported_seq":416,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:19.004268+0000","last_change":"2024-09-13T15:08:16.056633+0000","last_active":"2024-09-13T15:08:19.004268+0000","last_peered":"2024-09-13T15:08:19.004268+0000","last_clean":"2024-09-13T15:08:19.004268+0000","last_became_active":"2024-09-13T15:08:16.056135+0000","last_became_peered":"2024-09-13T15:08:16.056135+0000","last_unstale":"2024-09-13T15:08:19.004268+0000","last_undegraded":"2024-09-13T15:08:19.004268+0000","last_fullsized":"2024-09-13T15:08:19.004268+0000","mapping_epoch":1391,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1392,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:33.574443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:33.574443+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:03.166364+0000","scrub_duration":0.0079574959999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":22,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:21.201685+0000","last_change":"2024-09-13T15:08:08.404465+0000","last_active":"2024-09-13T15:08:21.201685+0000","last_peered":"2024-09-13T15:08:21.201685+0000","last_clean":"2024-09-13T15:08:21.201685+0000","last_became_active":"2024-09-13T15:08:08.404180+0000","last_became_peered":"2024-09-13T15:08:08.404180+0000","last_unstale":"2024-09-13T15:08:21.201685+0000","last_undegraded":"2024-09-13T15:08:21.201685+0000","last_fullsized":"2024-09-13T15:08:21.201685+0000","mapping_epoch":1388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:33.254058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":394,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.007083+0000","last_change":"2024-09-13T15:06:53.989169+0000","last_active":"2024-09-13T15:08:22.007083+0000","last_peered":"2024-09-13T15:08:22.007083+0000","last_clean":"2024-09-13T15:08:22.007083+0000","last_became_active":"2024-09-13T15:06:53.988812+0000","last_became_peered":"2024-09-13T15:06:53.988812+0000","last_unstale":"2024-09-13T15:08:22.007083+0000","last_undegraded":"2024-09-13T15:08:22.007083+0000","last_fullsized":"2024-09-13T15:08:22.007083+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:55:02.815779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034029399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":186,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.682136+0000","last_change":"2024-09-13T15:07:22.008756+0000","last_active":"2024-09-13T15:08:18.682136+0000","last_peered":"2024-09-13T15:08:18.682136+0000","last_clean":"2024-09-13T15:08:18.682136+0000","last_became_active":"2024-09-13T15:07:22.008411+0000","last_became_peered":"2024-09-13T15:07:22.008411+0000","last_unstale":"2024-09-13T15:08:18.682136+0000","last_undegraded":"2024-09-13T15:08:18.682136+0000","last_fullsized":"2024-09-13T15:08:18.682136+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:51.064483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"261'100","reported_seq":412,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:19.004313+0000","last_change":"2024-09-13T15:08:16.986734+0000","last_active":"2024-09-13T15:08:19.004313+0000","last_peered":"2024-09-13T15:08:19.004313+0000","last_clean":"2024-09-13T15:08:19.004313+0000","last_became_active":"2024-09-13T15:08:16.055903+0000","last_became_peered":"2024-09-13T15:08:16.055903+0000","last_unstale":"2024-09-13T15:08:19.004313+0000","last_undegraded":"2024-09-13T15:08:19.004313+0000","last_fullsized":"2024-09-13T15:08:19.004313+0000","mapping_epoch":1391,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1392,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T15:08:16.986669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T15:08:16.986669+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:24:30.288349+0000","scrub_duration":0.0021550900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":373,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.007642+0000","last_change":"2024-09-13T15:07:22.009592+0000","last_active":"2024-09-13T15:08:22.007642+0000","last_peered":"2024-09-13T15:08:22.007642+0000","last_clean":"2024-09-13T15:08:22.007642+0000","last_became_active":"2024-09-13T15:07:22.009269+0000","last_became_peered":"2024-09-13T15:07:22.009269+0000","last_unstale":"2024-09-13T15:08:22.007642+0000","last_undegraded":"2024-09-13T15:08:22.007642+0000","last_fullsized":"2024-09-13T15:08:22.007642+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:47:53.357130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":31,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.058938+0000","last_change":"2024-09-13T15:07:16.006172+0000","last_active":"2024-09-13T15:08:18.058938+0000","last_peered":"2024-09-13T15:08:18.058938+0000","last_clean":"2024-09-13T15:08:18.058938+0000","last_became_active":"2024-09-13T15:07:16.005881+0000","last_became_peered":"2024-09-13T15:07:16.005881+0000","last_unstale":"2024-09-13T15:08:18.058938+0000","last_undegraded":"2024-09-13T15:08:18.058938+0000","last_fullsized":"2024-09-13T15:08:18.058938+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:56.281641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":215,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.681932+0000","last_change":"2024-09-13T14:55:41.889668+0000","last_active":"2024-09-13T15:08:18.681932+0000","last_peered":"2024-09-13T15:08:18.681932+0000","last_clean":"2024-09-13T15:08:18.681932+0000","last_became_active":"2024-09-13T14:55:41.889195+0000","last_became_peered":"2024-09-13T14:55:41.889195+0000","last_unstale":"2024-09-13T15:08:18.681932+0000","last_undegraded":"2024-09-13T15:08:18.681932+0000","last_fullsized":"2024-09-13T15:08:18.681932+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:52.765208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081834900000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"261'100","reported_seq":236,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.681690+0000","last_change":"2024-09-13T15:07:22.013394+0000","last_active":"2024-09-13T15:08:18.681690+0000","last_peered":"2024-09-13T15:08:18.681690+0000","last_clean":"2024-09-13T15:08:18.681690+0000","last_became_active":"2024-09-13T15:07:22.012974+0000","last_became_peered":"2024-09-13T15:07:22.012974+0000","last_unstale":"2024-09-13T15:08:18.681690+0000","last_undegraded":"2024-09-13T15:08:18.681690+0000","last_fullsized":"2024-09-13T15:08:18.681690+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:57.024823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:57.024823+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:11.466865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":393,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.007209+0000","last_change":"2024-09-13T15:06:53.987932+0000","last_active":"2024-09-13T15:08:22.007209+0000","last_peered":"2024-09-13T15:08:22.007209+0000","last_clean":"2024-09-13T15:08:22.007209+0000","last_became_active":"2024-09-13T15:06:53.987612+0000","last_became_peered":"2024-09-13T15:06:53.987612+0000","last_unstale":"2024-09-13T15:08:22.007209+0000","last_undegraded":"2024-09-13T15:08:22.007209+0000","last_fullsized":"2024-09-13T15:08:22.007209+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:31:01.073349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":380,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.007227+0000","last_change":"2024-09-13T15:07:22.006243+0000","last_active":"2024-09-13T15:08:22.007227+0000","last_peered":"2024-09-13T15:08:22.007227+0000","last_clean":"2024-09-13T15:08:22.007227+0000","last_became_active":"2024-09-13T15:07:22.005921+0000","last_became_peered":"2024-09-13T15:07:22.005921+0000","last_unstale":"2024-09-13T15:08:22.007227+0000","last_undegraded":"2024-09-13T15:08:22.007227+0000","last_fullsized":"2024-09-13T15:08:22.007227+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:55.155198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":50,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:21.201733+0000","last_change":"2024-09-13T15:07:22.004071+0000","last_active":"2024-09-13T15:08:21.201733+0000","last_peered":"2024-09-13T15:08:21.201733+0000","last_clean":"2024-09-13T15:08:21.201733+0000","last_became_active":"2024-09-13T15:07:22.003746+0000","last_became_peered":"2024-09-13T15:07:22.003746+0000","last_unstale":"2024-09-13T15:08:21.201733+0000","last_undegraded":"2024-09-13T15:08:21.201733+0000","last_fullsized":"2024-09-13T15:08:21.201733+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:36.228833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025774199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"261'100","reported_seq":116,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.061042+0000","last_change":"2024-09-13T15:07:15.998303+0000","last_active":"2024-09-13T15:08:18.061042+0000","last_peered":"2024-09-13T15:08:18.061042+0000","last_clean":"2024-09-13T15:08:18.061042+0000","last_became_active":"2024-09-13T15:07:15.997919+0000","last_became_peered":"2024-09-13T15:07:15.997919+0000","last_unstale":"2024-09-13T15:08:18.061042+0000","last_undegraded":"2024-09-13T15:08:18.061042+0000","last_fullsized":"2024-09-13T15:08:18.061042+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T15:02:59.897422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T15:02:59.897422+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:23.129996+0000","scrub_duration":0.0032184959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":81,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.037069+0000","last_change":"2024-09-13T15:07:22.005527+0000","last_active":"2024-09-13T15:08:22.037069+0000","last_peered":"2024-09-13T15:08:22.037069+0000","last_clean":"2024-09-13T15:08:22.037069+0000","last_became_active":"2024-09-13T15:07:22.005237+0000","last_became_peered":"2024-09-13T15:07:22.005237+0000","last_unstale":"2024-09-13T15:08:22.037069+0000","last_undegraded":"2024-09-13T15:08:22.037069+0000","last_fullsized":"2024-09-13T15:08:22.037069+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:13:44.182858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":81,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.036860+0000","last_change":"2024-09-13T15:07:15.995884+0000","last_active":"2024-09-13T15:08:22.036860+0000","last_peered":"2024-09-13T15:08:22.036860+0000","last_clean":"2024-09-13T15:08:22.036860+0000","last_became_active":"2024-09-13T15:07:15.995475+0000","last_became_peered":"2024-09-13T15:07:15.995475+0000","last_unstale":"2024-09-13T15:08:22.036860+0000","last_undegraded":"2024-09-13T15:08:22.036860+0000","last_fullsized":"2024-09-13T15:08:22.036860+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:27.705697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":86,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.061080+0000","last_change":"2024-09-13T15:06:53.982720+0000","last_active":"2024-09-13T15:08:18.061080+0000","last_peered":"2024-09-13T15:08:18.061080+0000","last_clean":"2024-09-13T15:08:18.061080+0000","last_became_active":"2024-09-13T15:06:53.982103+0000","last_became_peered":"2024-09-13T15:06:53.982103+0000","last_unstale":"2024-09-13T15:08:18.061080+0000","last_undegraded":"2024-09-13T15:08:18.061080+0000","last_fullsized":"2024-09-13T15:08:18.061080+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:46:54.675404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000237383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"261'100","reported_seq":380,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.007518+0000","last_change":"2024-09-13T15:07:15.998937+0000","last_active":"2024-09-13T15:08:22.007518+0000","last_peered":"2024-09-13T15:08:22.007518+0000","last_clean":"2024-09-13T15:08:22.007518+0000","last_became_active":"2024-09-13T15:07:15.998454+0000","last_became_peered":"2024-09-13T15:07:15.998454+0000","last_unstale":"2024-09-13T15:08:22.007518+0000","last_undegraded":"2024-09-13T15:08:22.007518+0000","last_fullsized":"2024-09-13T15:08:22.007518+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":3,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.008264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.008264+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:51:19.930403+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1063,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:19.004226+0000","last_change":"2024-09-13T15:08:16.058775+0000","last_active":"2024-09-13T15:08:19.004226+0000","last_peered":"2024-09-13T15:08:19.004226+0000","last_clean":"2024-09-13T15:08:19.004226+0000","last_became_active":"2024-09-13T15:08:16.058426+0000","last_became_peered":"2024-09-13T15:08:16.058426+0000","last_unstale":"2024-09-13T15:08:19.004226+0000","last_undegraded":"2024-09-13T15:08:19.004226+0000","last_fullsized":"2024-09-13T15:08:19.004226+0000","mapping_epoch":1391,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:16.361720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":547,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089258+0000","last_change":"2024-09-13T14:55:41.835887+0000","last_active":"2024-09-13T15:08:22.089258+0000","last_peered":"2024-09-13T15:08:22.089258+0000","last_clean":"2024-09-13T15:08:22.089258+0000","last_became_active":"2024-09-13T14:55:41.834413+0000","last_became_peered":"2024-09-13T14:55:41.834413+0000","last_unstale":"2024-09-13T15:08:22.089258+0000","last_undegraded":"2024-09-13T15:08:22.089258+0000","last_fullsized":"2024-09-13T15:08:22.089258+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:01.850578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"54'1","reported_seq":263,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.681952+0000","last_change":"2024-09-13T15:07:22.025908+0000","last_active":"2024-09-13T15:08:18.681952+0000","last_peered":"2024-09-13T15:08:18.681952+0000","last_clean":"2024-09-13T15:08:18.681952+0000","last_became_active":"2024-09-13T15:07:22.013151+0000","last_became_peered":"2024-09-13T15:07:22.013151+0000","last_unstale":"2024-09-13T15:08:18.681952+0000","last_undegraded":"2024-09-13T15:08:18.681952+0000","last_fullsized":"2024-09-13T15:08:18.681952+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:18:09.595157+0000","last_clean_scrub_stamp":"2024-09-13T14:18:09.595157+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:02:47.167554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"261'100","reported_seq":86,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.061147+0000","last_change":"2024-09-13T15:06:53.983024+0000","last_active":"2024-09-13T15:08:18.061147+0000","last_peered":"2024-09-13T15:08:18.061147+0000","last_clean":"2024-09-13T15:08:18.061147+0000","last_became_active":"2024-09-13T15:06:53.982602+0000","last_became_peered":"2024-09-13T15:06:53.982602+0000","last_unstale":"2024-09-13T15:08:18.061147+0000","last_undegraded":"2024-09-13T15:08:18.061147+0000","last_fullsized":"2024-09-13T15:08:18.061147+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T15:00:44.119431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T15:00:44.119431+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:03:24.572217+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":386,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.007781+0000","last_change":"2024-09-13T15:07:15.998107+0000","last_active":"2024-09-13T15:08:22.007781+0000","last_peered":"2024-09-13T15:08:22.007781+0000","last_clean":"2024-09-13T15:08:22.007781+0000","last_became_active":"2024-09-13T15:07:15.997717+0000","last_became_peered":"2024-09-13T15:07:15.997717+0000","last_unstale":"2024-09-13T15:08:22.007781+0000","last_undegraded":"2024-09-13T15:08:22.007781+0000","last_fullsized":"2024-09-13T15:08:22.007781+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:40:14.840576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":553,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089209+0000","last_change":"2024-09-13T15:02:42.901428+0000","last_active":"2024-09-13T15:08:22.089209+0000","last_peered":"2024-09-13T15:08:22.089209+0000","last_clean":"2024-09-13T15:08:22.089209+0000","last_became_active":"2024-09-13T15:02:42.901131+0000","last_became_peered":"2024-09-13T15:02:42.901131+0000","last_unstale":"2024-09-13T15:08:22.089209+0000","last_undegraded":"2024-09-13T15:08:22.089209+0000","last_fullsized":"2024-09-13T15:08:22.089209+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:44.152548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"261'100","reported_seq":206,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.681766+0000","last_change":"2024-09-13T15:07:22.013339+0000","last_active":"2024-09-13T15:08:18.681766+0000","last_peered":"2024-09-13T15:08:18.681766+0000","last_clean":"2024-09-13T15:08:18.681766+0000","last_became_active":"2024-09-13T15:07:22.012961+0000","last_became_peered":"2024-09-13T15:07:22.012961+0000","last_unstale":"2024-09-13T15:08:18.681766+0000","last_undegraded":"2024-09-13T15:08:18.681766+0000","last_fullsized":"2024-09-13T15:08:18.681766+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:49.280478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:49.280478+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:42.191955+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":31,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.059187+0000","last_change":"2024-09-13T15:07:16.005652+0000","last_active":"2024-09-13T15:08:18.059187+0000","last_peered":"2024-09-13T15:08:18.059187+0000","last_clean":"2024-09-13T15:08:18.059187+0000","last_became_active":"2024-09-13T15:07:16.005341+0000","last_became_peered":"2024-09-13T15:07:16.005341+0000","last_unstale":"2024-09-13T15:08:18.059187+0000","last_undegraded":"2024-09-13T15:08:18.059187+0000","last_fullsized":"2024-09-13T15:08:18.059187+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:43.855261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":41,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.058785+0000","last_change":"2024-09-13T15:07:22.007141+0000","last_active":"2024-09-13T15:08:18.058785+0000","last_peered":"2024-09-13T15:08:18.058785+0000","last_clean":"2024-09-13T15:08:18.058785+0000","last_became_active":"2024-09-13T15:07:22.006715+0000","last_became_peered":"2024-09-13T15:07:22.006715+0000","last_unstale":"2024-09-13T15:08:18.058785+0000","last_undegraded":"2024-09-13T15:08:18.058785+0000","last_fullsized":"2024-09-13T15:08:18.058785+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:38:53.023680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"261'100","reported_seq":393,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.007014+0000","last_change":"2024-09-13T15:06:53.991344+0000","last_active":"2024-09-13T15:08:22.007014+0000","last_peered":"2024-09-13T15:08:22.007014+0000","last_clean":"2024-09-13T15:08:22.007014+0000","last_became_active":"2024-09-13T15:06:53.987397+0000","last_became_peered":"2024-09-13T15:06:53.987397+0000","last_unstale":"2024-09-13T15:08:22.007014+0000","last_undegraded":"2024-09-13T15:08:22.007014+0000","last_fullsized":"2024-09-13T15:08:22.007014+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:47.405845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:47.405845+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:57:33.181023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":41,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.059341+0000","last_change":"2024-09-13T15:07:22.002515+0000","last_active":"2024-09-13T15:08:18.059341+0000","last_peered":"2024-09-13T15:08:18.059341+0000","last_clean":"2024-09-13T15:08:18.059341+0000","last_became_active":"2024-09-13T15:07:22.002149+0000","last_became_peered":"2024-09-13T15:07:22.002149+0000","last_unstale":"2024-09-13T15:08:18.059341+0000","last_undegraded":"2024-09-13T15:08:18.059341+0000","last_fullsized":"2024-09-13T15:08:18.059341+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:26:12.683816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1122,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:19.003998+0000","last_change":"2024-09-13T15:08:16.108248+0000","last_active":"2024-09-13T15:08:19.003998+0000","last_peered":"2024-09-13T15:08:19.003998+0000","last_clean":"2024-09-13T15:08:19.003998+0000","last_became_active":"2024-09-13T15:08:16.107920+0000","last_became_peered":"2024-09-13T15:08:16.107920+0000","last_unstale":"2024-09-13T15:08:19.003998+0000","last_undegraded":"2024-09-13T15:08:19.003998+0000","last_fullsized":"2024-09-13T15:08:19.003998+0000","mapping_epoch":1391,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:49.517472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"261'100","reported_seq":186,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.681535+0000","last_change":"2024-09-13T15:07:22.013902+0000","last_active":"2024-09-13T15:08:18.681535+0000","last_peered":"2024-09-13T15:08:18.681535+0000","last_clean":"2024-09-13T15:08:18.681535+0000","last_became_active":"2024-09-13T15:07:22.013483+0000","last_became_peered":"2024-09-13T15:07:22.013483+0000","last_unstale":"2024-09-13T15:08:18.681535+0000","last_undegraded":"2024-09-13T15:08:18.681535+0000","last_fullsized":"2024-09-13T15:08:18.681535+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:52:55.816857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:52:55.816857+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:01.392722+0000","scrub_duration":270,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":31,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.059004+0000","last_change":"2024-09-13T15:07:16.005711+0000","last_active":"2024-09-13T15:08:18.059004+0000","last_peered":"2024-09-13T15:08:18.059004+0000","last_clean":"2024-09-13T15:08:18.059004+0000","last_became_active":"2024-09-13T15:07:16.005412+0000","last_became_peered":"2024-09-13T15:07:16.005412+0000","last_unstale":"2024-09-13T15:08:18.059004+0000","last_undegraded":"2024-09-13T15:08:18.059004+0000","last_fullsized":"2024-09-13T15:08:18.059004+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:57:22.027709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":614,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089317+0000","last_change":"2024-09-13T15:07:16.020821+0000","last_active":"2024-09-13T15:08:22.089317+0000","last_peered":"2024-09-13T15:08:22.089317+0000","last_clean":"2024-09-13T15:08:22.089317+0000","last_became_active":"2024-09-13T15:07:16.006351+0000","last_became_peered":"2024-09-13T15:07:16.006351+0000","last_unstale":"2024-09-13T15:08:22.089317+0000","last_undegraded":"2024-09-13T15:08:22.089317+0000","last_fullsized":"2024-09-13T15:08:22.089317+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:26.498892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"261'100","reported_seq":106,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.061075+0000","last_change":"2024-09-13T15:07:22.004190+0000","last_active":"2024-09-13T15:08:18.061075+0000","last_peered":"2024-09-13T15:08:18.061075+0000","last_clean":"2024-09-13T15:08:18.061075+0000","last_became_active":"2024-09-13T15:07:22.003796+0000","last_became_peered":"2024-09-13T15:07:22.003796+0000","last_unstale":"2024-09-13T15:08:18.061075+0000","last_undegraded":"2024-09-13T15:08:18.061075+0000","last_fullsized":"2024-09-13T15:08:18.061075+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":1097,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T15:02:56.904689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T15:02:56.904689+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:01:18.911718+0000","scrub_duration":0.001699702,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":81,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.037016+0000","last_change":"2024-09-13T15:07:15.996439+0000","last_active":"2024-09-13T15:08:22.037016+0000","last_peered":"2024-09-13T15:08:22.037016+0000","last_clean":"2024-09-13T15:08:22.037016+0000","last_became_active":"2024-09-13T15:07:15.996083+0000","last_became_peered":"2024-09-13T15:07:15.996083+0000","last_unstale":"2024-09-13T15:08:22.037016+0000","last_undegraded":"2024-09-13T15:08:22.037016+0000","last_fullsized":"2024-09-13T15:08:22.037016+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:12:21.867991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":166,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.682008+0000","last_change":"2024-09-13T14:55:41.848811+0000","last_active":"2024-09-13T15:08:18.682008+0000","last_peered":"2024-09-13T15:08:18.682008+0000","last_clean":"2024-09-13T15:08:18.682008+0000","last_became_active":"2024-09-13T14:55:41.848542+0000","last_became_peered":"2024-09-13T14:55:41.848542+0000","last_unstale":"2024-09-13T15:08:18.682008+0000","last_undegraded":"2024-09-13T15:08:18.682008+0000","last_fullsized":"2024-09-13T15:08:18.682008+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:48.534022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"261'100","reported_seq":31,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.059236+0000","last_change":"2024-09-13T15:07:16.000211+0000","last_active":"2024-09-13T15:08:18.059236+0000","last_peered":"2024-09-13T15:08:18.059236+0000","last_clean":"2024-09-13T15:08:18.059236+0000","last_became_active":"2024-09-13T15:07:15.998798+0000","last_became_peered":"2024-09-13T15:07:15.998798+0000","last_unstale":"2024-09-13T15:08:18.059236+0000","last_undegraded":"2024-09-13T15:08:18.059236+0000","last_fullsized":"2024-09-13T15:08:18.059236+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T15:03:00.851236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T15:03:00.851236+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:50:35.250688+0000","scrub_duration":0.004159968,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1409,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:19.003414+0000","last_change":"2024-09-13T15:02:42.880814+0000","last_active":"2024-09-13T15:08:19.003414+0000","last_peered":"2024-09-13T15:08:19.003414+0000","last_clean":"2024-09-13T15:08:19.003414+0000","last_became_active":"2024-09-13T15:02:42.880393+0000","last_became_peered":"2024-09-13T15:02:42.880393+0000","last_unstale":"2024-09-13T15:08:19.003414+0000","last_undegraded":"2024-09-13T15:08:19.003414+0000","last_fullsized":"2024-09-13T15:08:19.003414+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:03.222968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":80,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.061596+0000","last_change":"2024-09-13T15:02:42.914851+0000","last_active":"2024-09-13T15:08:18.061596+0000","last_peered":"2024-09-13T15:08:18.061596+0000","last_clean":"2024-09-13T15:08:18.061596+0000","last_became_active":"2024-09-13T15:02:42.884164+0000","last_became_peered":"2024-09-13T15:02:42.884164+0000","last_unstale":"2024-09-13T15:08:18.061596+0000","last_undegraded":"2024-09-13T15:08:18.061596+0000","last_fullsized":"2024-09-13T15:08:18.061596+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:26:54.412326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"261'100","reported_seq":408,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:24.001118+0000","last_change":"2024-09-13T15:08:24.001087+0000","last_active":"2024-09-13T15:08:24.001118+0000","last_peered":"2024-09-13T15:08:24.001118+0000","last_clean":"2024-09-13T15:08:24.001118+0000","last_became_active":"2024-09-13T15:08:16.058475+0000","last_became_peered":"2024-09-13T15:08:16.058475+0000","last_unstale":"2024-09-13T15:08:24.001118+0000","last_undegraded":"2024-09-13T15:08:24.001118+0000","last_fullsized":"2024-09-13T15:08:24.001118+0000","mapping_epoch":1391,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1392,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T15:08:24.001016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T15:08:24.001016+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:54:01.192884+0000","scrub_duration":0.0015859839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":31,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.058590+0000","last_change":"2024-09-13T15:07:16.003278+0000","last_active":"2024-09-13T15:08:18.058590+0000","last_peered":"2024-09-13T15:08:18.058590+0000","last_clean":"2024-09-13T15:08:18.058590+0000","last_became_active":"2024-09-13T15:07:16.002938+0000","last_became_peered":"2024-09-13T15:07:16.002938+0000","last_unstale":"2024-09-13T15:08:18.058590+0000","last_undegraded":"2024-09-13T15:08:18.058590+0000","last_fullsized":"2024-09-13T15:08:18.058590+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:52.151311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":108,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.061565+0000","last_change":"2024-09-13T15:07:16.021982+0000","last_active":"2024-09-13T15:08:18.061565+0000","last_peered":"2024-09-13T15:08:18.061565+0000","last_clean":"2024-09-13T15:08:18.061565+0000","last_became_active":"2024-09-13T15:07:15.998901+0000","last_became_peered":"2024-09-13T15:07:15.998901+0000","last_unstale":"2024-09-13T15:08:18.061565+0000","last_undegraded":"2024-09-13T15:08:18.061565+0000","last_fullsized":"2024-09-13T15:08:18.061565+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:28:38.132787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"261'100","reported_seq":554,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089536+0000","last_change":"2024-09-13T15:06:54.015766+0000","last_active":"2024-09-13T15:08:22.089536+0000","last_peered":"2024-09-13T15:08:22.089536+0000","last_clean":"2024-09-13T15:08:22.089536+0000","last_became_active":"2024-09-13T15:06:53.989009+0000","last_became_peered":"2024-09-13T15:06:53.989009+0000","last_unstale":"2024-09-13T15:08:22.089536+0000","last_undegraded":"2024-09-13T15:08:22.089536+0000","last_fullsized":"2024-09-13T15:08:22.089536+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:16.597568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:16.597568+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:25.025847+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":196,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.681633+0000","last_change":"2024-09-13T15:02:42.885662+0000","last_active":"2024-09-13T15:08:18.681633+0000","last_peered":"2024-09-13T15:08:18.681633+0000","last_clean":"2024-09-13T15:08:18.681633+0000","last_became_active":"2024-09-13T15:02:42.885326+0000","last_became_peered":"2024-09-13T15:02:42.885326+0000","last_unstale":"2024-09-13T15:08:18.681633+0000","last_undegraded":"2024-09-13T15:08:18.681633+0000","last_fullsized":"2024-09-13T15:08:18.681633+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:11.631110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":108,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.061714+0000","last_change":"2024-09-13T15:07:16.025552+0000","last_active":"2024-09-13T15:08:18.061714+0000","last_peered":"2024-09-13T15:08:18.061714+0000","last_clean":"2024-09-13T15:08:18.061714+0000","last_became_active":"2024-09-13T15:07:15.999224+0000","last_became_peered":"2024-09-13T15:07:15.999224+0000","last_unstale":"2024-09-13T15:08:18.061714+0000","last_undegraded":"2024-09-13T15:08:18.061714+0000","last_fullsized":"2024-09-13T15:08:18.061714+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:23.250695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"261'100","reported_seq":96,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.061571+0000","last_change":"2024-09-13T15:07:16.000056+0000","last_active":"2024-09-13T15:08:18.061571+0000","last_peered":"2024-09-13T15:08:18.061571+0000","last_clean":"2024-09-13T15:08:18.061571+0000","last_became_active":"2024-09-13T15:07:15.999193+0000","last_became_peered":"2024-09-13T15:07:15.999193+0000","last_unstale":"2024-09-13T15:08:18.061571+0000","last_undegraded":"2024-09-13T15:08:18.061571+0000","last_fullsized":"2024-09-13T15:08:18.061571+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":4,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:44:31.191326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:44:31.191326+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:09:34.018627+0000","scrub_duration":0.0025317360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":1458,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:19.003715+0000","last_change":"2024-09-13T15:06:54.005515+0000","last_active":"2024-09-13T15:08:19.003715+0000","last_peered":"2024-09-13T15:08:19.003715+0000","last_clean":"2024-09-13T15:08:19.003715+0000","last_became_active":"2024-09-13T15:06:53.982127+0000","last_became_peered":"2024-09-13T15:06:53.982127+0000","last_unstale":"2024-09-13T15:08:19.003715+0000","last_undegraded":"2024-09-13T15:08:19.003715+0000","last_fullsized":"2024-09-13T15:08:19.003715+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:47.964603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":545,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.089748+0000","last_change":"2024-09-13T15:07:07.603412+0000","last_active":"2024-09-13T15:08:22.089748+0000","last_peered":"2024-09-13T15:08:22.089748+0000","last_clean":"2024-09-13T15:08:22.089748+0000","last_became_active":"2024-09-13T15:07:07.603069+0000","last_became_peered":"2024-09-13T15:07:07.603069+0000","last_unstale":"2024-09-13T15:08:22.089748+0000","last_undegraded":"2024-09-13T15:08:22.089748+0000","last_fullsized":"2024-09-13T15:08:22.089748+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:54:02.275494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"261'100","reported_seq":166,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.681896+0000","last_change":"2024-09-13T14:55:41.889497+0000","last_active":"2024-09-13T15:08:18.681896+0000","last_peered":"2024-09-13T15:08:18.681896+0000","last_clean":"2024-09-13T15:08:18.681896+0000","last_became_active":"2024-09-13T14:55:41.888766+0000","last_became_peered":"2024-09-13T14:55:41.888766+0000","last_unstale":"2024-09-13T15:08:18.681896+0000","last_undegraded":"2024-09-13T15:08:18.681896+0000","last_fullsized":"2024-09-13T15:08:18.681896+0000","mapping_epoch":1236,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:55.378404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:55.378404+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:50.289550+0000","scrub_duration":0.0016008439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":96,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.061074+0000","last_change":"2024-09-13T15:07:15.990866+0000","last_active":"2024-09-13T15:08:18.061074+0000","last_peered":"2024-09-13T15:08:18.061074+0000","last_clean":"2024-09-13T15:08:18.061074+0000","last_became_active":"2024-09-13T15:07:15.990549+0000","last_became_peered":"2024-09-13T15:07:15.990549+0000","last_unstale":"2024-09-13T15:08:18.061074+0000","last_undegraded":"2024-09-13T15:08:18.061074+0000","last_fullsized":"2024-09-13T15:08:18.061074+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:41:03.478002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":73,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.036469+0000","last_change":"2024-09-13T15:06:53.995604+0000","last_active":"2024-09-13T15:08:22.036469+0000","last_peered":"2024-09-13T15:08:22.036469+0000","last_clean":"2024-09-13T15:08:22.036469+0000","last_became_active":"2024-09-13T15:06:53.995306+0000","last_became_peered":"2024-09-13T15:06:53.995306+0000","last_unstale":"2024-09-13T15:08:22.036469+0000","last_undegraded":"2024-09-13T15:08:22.036469+0000","last_fullsized":"2024-09-13T15:08:22.036469+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:09:34.918000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"261'100","reported_seq":386,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.007425+0000","last_change":"2024-09-13T15:07:15.999275+0000","last_active":"2024-09-13T15:08:22.007425+0000","last_peered":"2024-09-13T15:08:22.007425+0000","last_clean":"2024-09-13T15:08:22.007425+0000","last_became_active":"2024-09-13T15:07:15.997916+0000","last_became_peered":"2024-09-13T15:07:15.997916+0000","last_unstale":"2024-09-13T15:08:22.007425+0000","last_undegraded":"2024-09-13T15:08:22.007425+0000","last_fullsized":"2024-09-13T15:08:22.007425+0000","mapping_epoch":1371,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1372,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:19.978464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:19.978464+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:57:24.878718+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":73,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.036426+0000","last_change":"2024-09-13T15:06:53.982939+0000","last_active":"2024-09-13T15:08:22.036426+0000","last_peered":"2024-09-13T15:08:22.036426+0000","last_clean":"2024-09-13T15:08:22.036426+0000","last_became_active":"2024-09-13T15:06:53.982631+0000","last_became_peered":"2024-09-13T15:06:53.982631+0000","last_unstale":"2024-09-13T15:08:22.036426+0000","last_undegraded":"2024-09-13T15:08:22.036426+0000","last_fullsized":"2024-09-13T15:08:22.036426+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:43:11.173393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":186,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.682223+0000","last_change":"2024-09-13T15:02:42.886038+0000","last_active":"2024-09-13T15:08:18.682223+0000","last_peered":"2024-09-13T15:08:18.682223+0000","last_clean":"2024-09-13T15:08:18.682223+0000","last_became_active":"2024-09-13T15:02:42.884438+0000","last_became_peered":"2024-09-13T15:02:42.884438+0000","last_unstale":"2024-09-13T15:08:18.682223+0000","last_undegraded":"2024-09-13T15:08:18.682223+0000","last_fullsized":"2024-09-13T15:08:18.682223+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:47:39.175275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"261'100","reported_seq":196,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.681593+0000","last_change":"2024-09-13T15:02:42.885131+0000","last_active":"2024-09-13T15:08:18.681593+0000","last_peered":"2024-09-13T15:08:18.681593+0000","last_clean":"2024-09-13T15:08:18.681593+0000","last_became_active":"2024-09-13T15:02:42.883904+0000","last_became_peered":"2024-09-13T15:02:42.883904+0000","last_unstale":"2024-09-13T15:08:18.681593+0000","last_undegraded":"2024-09-13T15:08:18.681593+0000","last_fullsized":"2024-09-13T15:08:18.681593+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:53:50.327393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:53:50.327393+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:11:05.603544+0000","scrub_duration":0.0011508309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":146,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.061285+0000","last_change":"2024-09-13T15:07:22.021811+0000","last_active":"2024-09-13T15:08:18.061285+0000","last_peered":"2024-09-13T15:08:18.061285+0000","last_clean":"2024-09-13T15:08:18.061285+0000","last_became_active":"2024-09-13T15:07:22.007340+0000","last_became_peered":"2024-09-13T15:07:22.007340+0000","last_unstale":"2024-09-13T15:08:18.061285+0000","last_undegraded":"2024-09-13T15:08:18.061285+0000","last_fullsized":"2024-09-13T15:08:18.061285+0000","mapping_epoch":1374,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:24.847134+0000","last_clean_scrub_stamp":"2024-09-13T14:19:24.847134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:35:59.082302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":170,"num_keys_recovered":50,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":34,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-13T15:08:18.681888+0000","last_change":"2024-09-13T15:07:07.379986+0000","last_active":"2024-09-13T15:08:18.681888+0000","last_peered":"2024-09-13T15:08:18.681888+0000","last_clean":"2024-09-13T15:08:18.681888+0000","last_became_active":"2024-09-13T15:07:07.379648+0000","last_became_peered":"2024-09-13T15:07:07.379648+0000","last_unstale":"2024-09-13T15:08:18.681888+0000","last_undegraded":"2024-09-13T15:08:18.681888+0000","last_fullsized":"2024-09-13T15:08:18.681888+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:19:25.852456+0000","last_clean_scrub_stamp":"2024-09-13T14:19:25.852456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:18.037587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"261'100","reported_seq":73,"reported_epoch":1394,"state":"active+clean","last_fresh":"2024-09-13T15:08:22.036430+0000","last_change":"2024-09-13T15:06:53.986158+0000","last_active":"2024-09-13T15:08:22.036430+0000","last_peered":"2024-09-13T15:08:22.036430+0000","last_clean":"2024-09-13T15:08:22.036430+0000","last_became_active":"2024-09-13T15:06:53.985844+0000","last_became_peered":"2024-09-13T15:06:53.985844+0000","last_unstale":"2024-09-13T15:08:22.036430+0000","last_undegraded":"2024-09-13T15:08:22.036430+0000","last_fullsized":"2024-09-13T15:08:22.036430+0000","mapping_epoch":1362,"log_start":"0'0","ondisk_log_start":"0'0","created":1116,"last_epoch_clean":1363,"parent":"0.0","parent_split_bits":5,"last_scrub":"261'100","last_scrub_stamp":"2024-09-13T14:43:14.654603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:15:42.069756+0000","last_clean_scrub_stamp":"2024-09-13T14:43:14.654603+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:50:16.010750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":28216,"num_keys_recovered":50,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15334,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2886848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430830,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1490236,"kb_used_data":9264,"kb_used_omap":1,"kb_used_meta":1480958,"kb_avail":92254916,"statfs":{"total":95995035648,"available":94469033984,"internally_reserved":0,"allocated":9486336,"data_stored":6136523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1516501336},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388627059,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1496300,"kb_used_data":9316,"kb_used_omap":1,"kb_used_meta":1486974,"kb_avail":92248852,"statfs":{"total":95995035648,"available":94462824448,"internally_reserved":0,"allocated":9539584,"data_stored":6201388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1522661720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823288,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1503212,"kb_used_data":9248,"kb_used_omap":1,"kb_used_meta":1493950,"kb_avail":92241940,"statfs":{"total":95995035648,"available":94455746560,"internally_reserved":0,"allocated":9469952,"data_stored":6132222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1529805144},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019516,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":913012,"kb_used_data":9000,"kb_used_omap":1,"kb_used_meta":903998,"kb_avail":92832140,"statfs":{"total":95995035648,"available":95060111360,"internally_reserved":0,"allocated":9216000,"data_stored":5873765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":925694296},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":624,"seq":2680059593174,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1039520,"kb_used_data":9300,"kb_used_omap":26,"kb_used_meta":1030181,"kb_avail":92705632,"statfs":{"total":95995035648,"available":94930567168,"internally_reserved":0,"allocated":9523200,"data_stored":6196657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27488,"internal_metadata":1054905504},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":591,"seq":2538325672412,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1120164,"kb_used_data":53340,"kb_used_omap":41,"kb_used_meta":1066774,"kb_avail":92624988,"statfs":{"total":95995035648,"available":94847987712,"internally_reserved":0,"allocated":54620160,"data_stored":51286483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42651,"internal_metadata":1092376933},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":557,"seq":2392296784355,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1185504,"kb_used_data":8980,"kb_used_omap":32,"kb_used_meta":1176479,"kb_avail":92559648,"statfs":{"total":95995035648,"available":94781079552,"internally_reserved":0,"allocated":9195520,"data_stored":5868906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33752,"internal_metadata":1204714536},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":524,"seq":2250562863594,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":871588,"kb_used_data":9304,"kb_used_omap":41,"kb_used_meta":862230,"kb_avail":92873564,"statfs":{"total":95995035648,"available":95102529536,"internally_reserved":0,"allocated":9527296,"data_stored":6197159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42694,"internal_metadata":882923834},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5498,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5906,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2000,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1930,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:09:11.269 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T15:09:11.269 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 48, in task with parallel.parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cls/test_cls_rbd.sh) on smithi151 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2024-09-13T15:09:11.446 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ac70f570e58942088455127a5ce00c3d Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 48, in task with parallel.parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cls/test_cls_rbd.sh) on smithi151 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2024-09-13T15:09:11.450 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-13T15:09:11.460 INFO:tasks.cephadm:Teardown begin 2024-09-13T15:09:11.461 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-13T15:09:11.496 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:09:11 smithi151 ceph-mon[104450]: pgmap v2492: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:09:11.496 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:09:11 smithi151 ceph-mon[104450]: from='client.46964 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:09:11.496 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:09:11 smithi151 ceph-mon[104450]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:09:11.497 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:09:11 smithi151 ceph-mon[108966]: pgmap v2492: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:09:11.497 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:09:11 smithi151 ceph-mon[108966]: from='client.46964 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:09:11.497 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:09:11 smithi151 ceph-mon[108966]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:09:11.514 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-13T15:09:11.538 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:09:11 smithi184 ceph-mon[92527]: pgmap v2492: 103 pgs: 103 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-09-13T15:09:11.538 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:09:11 smithi184 ceph-mon[92527]: from='client.46964 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:09:11.538 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:09:11 smithi184 ceph-mon[92527]: from='mgr.25952 172.21.15.151:0/2482405164' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:09:11.556 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-13T15:09:11.556 DEBUG:teuthology.orchestra.run.smithi151:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-13T15:09:11.572 DEBUG:teuthology.orchestra.run.smithi184:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-13T15:09:11.612 INFO:tasks.cephadm:Stopping all daemons... 2024-09-13T15:09:11.612 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-13T15:09:11.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-2401fc50-71da-11ef-bceb-c7b262605968@mon.a 2024-09-13T15:09:11.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:09:11 smithi151 systemd[1]: Stopping Ceph mon.a for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T15:09:12.598 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:09:12 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: [13/Sep/2024:15:09:12] ENGINE Bus STOPPING 2024-09-13T15:09:12.599 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:09:12 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a[104427]: 2024-09-13T15:09:12.333+0000 7fe9df95a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:09:12.599 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:09:12 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a[104427]: 2024-09-13T15:09:12.333+0000 7fe9df95a640 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-09-13T15:09:12.599 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:09:12 smithi151 podman[361423]: 2024-09-13 15:09:12.497556615 +0000 UTC m=+0.281154938 container died 694de6b945cd657e48dd990549243d0260bafc754240ec51555fb5c26362a11a (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True) 2024-09-13T15:09:12.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:09:12 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: [13/Sep/2024:15:09:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T15:09:12.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:09:12 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: [13/Sep/2024:15:09:12] ENGINE Bus STOPPED 2024-09-13T15:09:12.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:09:12 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: [13/Sep/2024:15:09:12] ENGINE Bus STARTING 2024-09-13T15:09:12.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:09:12 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: [13/Sep/2024:15:09:12] ENGINE Serving on http://:::9283 2024-09-13T15:09:12.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:09:12 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: [13/Sep/2024:15:09:12] ENGINE Bus STARTED 2024-09-13T15:09:13.320 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:09:12 smithi151 podman[361423]: 2024-09-13 15:09:12.958581654 +0000 UTC m=+0.742179972 container cleanup 694de6b945cd657e48dd990549243d0260bafc754240ec51555fb5c26362a11a (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731) 2024-09-13T15:09:13.320 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:09:12 smithi151 bash[361423]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a 2024-09-13T15:09:13.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 13 15:09:13 smithi151 podman[361522]: 2024-09-13 15:09:13.319292431 +0000 UTC m=+0.818794566 container remove 694de6b945cd657e48dd990549243d0260bafc754240ec51555fb5c26362a11a (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-a, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-13T15:09:13.761 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@mon.a.service' 2024-09-13T15:09:13.826 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:09:13.826 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-13T15:09:13.827 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-13T15:09:13.827 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-2401fc50-71da-11ef-bceb-c7b262605968@mon.c 2024-09-13T15:09:14.109 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:09:13 smithi151 systemd[1]: Stopping Ceph mon.c for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T15:09:14.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:09:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: ::ffff:172.21.15.184 - - [13/Sep/2024:15:09:14] "GET /metrics HTTP/1.1" 200 34280 "" "Prometheus/2.51.0" 2024-09-13T15:09:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:09:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-c[108943]: 2024-09-13T15:09:14.605+0000 7f32b81ab640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:09:14.859 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:09:14 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-c[108943]: 2024-09-13T15:09:14.605+0000 7f32b81ab640 -1 mon.c@2(peon) e4 *** Got Signal Terminated *** 2024-09-13T15:09:15.235 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:09:14 smithi151 podman[361808]: 2024-09-13 15:09:14.966982947 +0000 UTC m=+0.531154243 container died e548fceed2f9a4804e53d2f49ccc63cb9cc0cd3fb315e9189c6066ba685cf405 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-c, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730) 2024-09-13T15:09:15.608 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:09:15 smithi151 podman[361808]: 2024-09-13 15:09:15.416892542 +0000 UTC m=+0.981063884 container cleanup e548fceed2f9a4804e53d2f49ccc63cb9cc0cd3fb315e9189c6066ba685cf405 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-13T15:09:15.609 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:09:15 smithi151 bash[361808]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-c 2024-09-13T15:09:15.872 INFO:journalctl@ceph.mon.c.smithi151.stdout:Sep 13 15:09:15 smithi151 podman[361835]: 2024-09-13 15:09:15.610424582 +0000 UTC m=+0.964039583 container remove e548fceed2f9a4804e53d2f49ccc63cb9cc0cd3fb315e9189c6066ba685cf405 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-13T15:09:16.430 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@mon.c.service' 2024-09-13T15:09:16.469 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:09:16.470 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-13T15:09:16.470 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-13T15:09:16.471 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-2401fc50-71da-11ef-bceb-c7b262605968@mon.b 2024-09-13T15:09:16.811 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:09:16 smithi184 systemd[1]: Stopping Ceph mon.b for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T15:09:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:09:16 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-b[92504]: 2024-09-13T15:09:16.808+0000 7faaa06f6640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:09:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:09:16 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-b[92504]: 2024-09-13T15:09:16.808+0000 7faaa06f6640 -1 mon.b@1(peon) e4 *** Got Signal Terminated *** 2024-09-13T15:09:17.104 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:09:16 smithi184 podman[246154]: 2024-09-13 15:09:16.941799007 +0000 UTC m=+0.266883107 container died 12b6c0ccb635e1e1f30dc0613122cee5a4dd58d8baaa259c2746a4305c361546 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-13T15:09:17.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:09:17 smithi184 podman[246154]: 2024-09-13 15:09:17.423597801 +0000 UTC m=+0.748681894 container cleanup 12b6c0ccb635e1e1f30dc0613122cee5a4dd58d8baaa259c2746a4305c361546 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-b, io.buildah.version=1.37.2, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux ) 2024-09-13T15:09:17.604 INFO:journalctl@ceph.mon.b.smithi184.stdout:Sep 13 15:09:17 smithi184 bash[246154]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-mon-b 2024-09-13T15:09:17.856 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@mon.b.service' 2024-09-13T15:09:17.901 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:09:17.902 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-13T15:09:17.902 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-13T15:09:17.902 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-2401fc50-71da-11ef-bceb-c7b262605968@mgr.y 2024-09-13T15:09:18.221 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:09:17 smithi151 systemd[1]: Stopping Ceph mgr.y for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T15:09:18.563 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:09:18 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y[94858]: 2024-09-13T15:09:18.217+0000 7fc9a9d76640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:09:18.563 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:09:18 smithi151 podman[362019]: 2024-09-13 15:09:18.260246576 +0000 UTC m=+0.172159604 container died d8c169a4f36ef4d11ce368dd05af6808863a872fc2e5e325f786a7b17a622f81 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-13T15:09:18.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:09:18 smithi151 podman[362019]: 2024-09-13 15:09:18.72082033 +0000 UTC m=+0.632733402 container cleanup d8c169a4f36ef4d11ce368dd05af6808863a872fc2e5e325f786a7b17a622f81 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-13T15:09:18.859 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 13 15:09:18 smithi151 bash[362019]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-y 2024-09-13T15:09:19.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@mgr.y.service' 2024-09-13T15:09:19.232 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:09:19.232 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-13T15:09:19.232 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-13T15:09:19.232 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-2401fc50-71da-11ef-bceb-c7b262605968@mgr.x 2024-09-13T15:09:19.579 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 15:09:19 smithi184 systemd[1]: Stopping Ceph mgr.x for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T15:09:19.854 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 15:09:19 smithi184 podman[246285]: 2024-09-13 15:09:19.577814364 +0000 UTC m=+0.153537400 container died 8a0f7d440b6e957af920c6c95c7e6875804ca27fb6c0762d585b79ba5935b226 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730) 2024-09-13T15:09:20.213 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 15:09:19 smithi184 podman[246285]: 2024-09-13 15:09:19.96098241 +0000 UTC m=+0.536705360 container cleanup 8a0f7d440b6e957af920c6c95c7e6875804ca27fb6c0762d585b79ba5935b226 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T15:09:20.213 INFO:journalctl@ceph.mgr.x.smithi184.stdout:Sep 13 15:09:19 smithi184 bash[246285]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-mgr-x 2024-09-13T15:09:20.434 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@mgr.x.service' 2024-09-13T15:09:20.467 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:09:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:09:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2401fc50-71da-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-13T15:09:20.469 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:09:20.469 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-13T15:09:20.469 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-13T15:09:20.469 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.0 2024-09-13T15:09:20.779 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 15:09:20 smithi151 systemd[1]: Stopping Ceph osd.0 for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T15:09:21.109 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 15:09:20 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0[118742]: 2024-09-13T15:09:20.777+0000 7f18b7342640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:09:21.109 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 15:09:20 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0[118742]: 2024-09-13T15:09:20.777+0000 7f18b7342640 -1 osd.0 1394 *** Got signal Terminated *** 2024-09-13T15:09:21.109 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 15:09:20 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0[118742]: 2024-09-13T15:09:20.777+0000 7f18b7342640 -1 osd.0 1394 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:09:26.602 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 15:09:26 smithi151 podman[362151]: 2024-09-13 15:09:26.222132088 +0000 UTC m=+5.553042901 container died 018b463f93c20c7f1986053c11b443c104ae16f40983b2abdd8e881257657c94 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-13T15:09:26.859 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 15:09:26 smithi151 podman[362151]: 2024-09-13 15:09:26.601625577 +0000 UTC m=+5.932536433 container cleanup 018b463f93c20c7f1986053c11b443c104ae16f40983b2abdd8e881257657c94 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730) 2024-09-13T15:09:26.859 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 15:09:26 smithi151 bash[362151]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0 2024-09-13T15:09:27.197 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 15:09:26 smithi151 podman[362171]: 2024-09-13 15:09:26.887056417 +0000 UTC m=+0.665102371 container remove 018b463f93c20c7f1986053c11b443c104ae16f40983b2abdd8e881257657c94 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True) 2024-09-13T15:09:27.517 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 15:09:27 smithi151 podman[362268]: 2024-09-13 15:09:27.09888516 +0000 UTC m=+0.015807338 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T15:09:27.517 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 15:09:27 smithi151 podman[362268]: 2024-09-13 15:09:27.213409124 +0000 UTC m=+0.130331335 container create 5030bd7d4c40a14a4401d522fb155ad359c8f8ed62255b857e10efc9b970d838 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-13T15:09:27.517 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 15:09:27 smithi151 podman[362268]: 2024-09-13 15:09:27.51623562 +0000 UTC m=+0.433157786 container init 5030bd7d4c40a14a4401d522fb155ad359c8f8ed62255b857e10efc9b970d838 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-13T15:09:27.859 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 15:09:27 smithi151 podman[362268]: 2024-09-13 15:09:27.520997206 +0000 UTC m=+0.437919372 container start 5030bd7d4c40a14a4401d522fb155ad359c8f8ed62255b857e10efc9b970d838 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=) 2024-09-13T15:09:27.859 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 15:09:27 smithi151 podman[362268]: 2024-09-13 15:09:27.624496817 +0000 UTC m=+0.541419000 container attach 5030bd7d4c40a14a4401d522fb155ad359c8f8ed62255b857e10efc9b970d838 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T15:09:28.196 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 13 15:09:27 smithi151 podman[362268]: 2024-09-13 15:09:27.902581139 +0000 UTC m=+0.819503317 container died 5030bd7d4c40a14a4401d522fb155ad359c8f8ed62255b857e10efc9b970d838 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-13T15:09:28.649 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.0.service' 2024-09-13T15:09:28.686 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:09:28.687 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-13T15:09:28.687 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-13T15:09:28.687 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.1 2024-09-13T15:09:29.004 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 15:09:28 smithi151 systemd[1]: Stopping Ceph osd.1 for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T15:09:29.359 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 15:09:29 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1[124715]: 2024-09-13T15:09:29.002+0000 7fd1834dd640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:09:29.359 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 15:09:29 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1[124715]: 2024-09-13T15:09:29.002+0000 7fd1834dd640 -1 osd.1 1394 *** Got signal Terminated *** 2024-09-13T15:09:29.359 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 15:09:29 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1[124715]: 2024-09-13T15:09:29.002+0000 7fd1834dd640 -1 osd.1 1394 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:09:34.953 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 15:09:34 smithi151 podman[362394]: 2024-09-13 15:09:34.675798531 +0000 UTC m=+5.769885113 container died 04d393df5f431646e8f0f3b823329442e7889de20762f2dbe043bb127d91b0db (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T15:09:35.357 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 15:09:35 smithi151 podman[362394]: 2024-09-13 15:09:35.151115419 +0000 UTC m=+6.245201997 container cleanup 04d393df5f431646e8f0f3b823329442e7889de20762f2dbe043bb127d91b0db (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-13T15:09:35.358 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 15:09:35 smithi151 bash[362394]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1 2024-09-13T15:09:35.609 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 15:09:35 smithi151 podman[362414]: 2024-09-13 15:09:35.453256148 +0000 UTC m=+0.774469243 container remove 04d393df5f431646e8f0f3b823329442e7889de20762f2dbe043bb127d91b0db (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-13T15:09:36.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:09:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:09:35.639Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://172.21.15.151:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 172.21.15.151:8765: connect: connection refused" 2024-09-13T15:09:36.104 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:09:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:09:35.641Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.151:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.151:8765: connect: connection refused" 2024-09-13T15:09:36.105 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:09:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:09:35.642Z caller=refresh.go:90 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.151:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.151:8765: connect: connection refused" 2024-09-13T15:09:36.105 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:09:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:09:35.643Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.151:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.151:8765: connect: connection refused" 2024-09-13T15:09:36.105 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:09:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:09:35.643Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.151:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.151:8765: connect: connection refused" 2024-09-13T15:09:36.105 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:09:35 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:09:35.643Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.151:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.151:8765: connect: connection refused" 2024-09-13T15:09:36.108 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 15:09:35 smithi151 podman[362515]: 2024-09-13 15:09:35.748181725 +0000 UTC m=+0.016426784 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T15:09:36.109 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 15:09:35 smithi151 podman[362515]: 2024-09-13 15:09:35.85457479 +0000 UTC m=+0.122819837 container create aed5789178a373f5d9265588bf4c62a3c1f3bb9bc5d407f7b6fa2e1f8ffdb99a (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-13T15:09:36.609 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 15:09:36 smithi151 podman[362515]: 2024-09-13 15:09:36.132540124 +0000 UTC m=+0.400785175 container init aed5789178a373f5d9265588bf4c62a3c1f3bb9bc5d407f7b6fa2e1f8ffdb99a (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, ceph=True) 2024-09-13T15:09:36.609 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 15:09:36 smithi151 podman[362515]: 2024-09-13 15:09:36.137867517 +0000 UTC m=+0.406112568 container start aed5789178a373f5d9265588bf4c62a3c1f3bb9bc5d407f7b6fa2e1f8ffdb99a (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-13T15:09:36.609 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 15:09:36 smithi151 podman[362515]: 2024-09-13 15:09:36.249447551 +0000 UTC m=+0.517692620 container attach aed5789178a373f5d9265588bf4c62a3c1f3bb9bc5d407f7b6fa2e1f8ffdb99a (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T15:09:36.609 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 13 15:09:36 smithi151 podman[362515]: 2024-09-13 15:09:36.328252456 +0000 UTC m=+0.596497504 container died aed5789178a373f5d9265588bf4c62a3c1f3bb9bc5d407f7b6fa2e1f8ffdb99a (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-13T15:09:37.016 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.1.service' 2024-09-13T15:09:37.057 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:09:37.057 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-13T15:09:37.057 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-13T15:09:37.057 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.2 2024-09-13T15:09:37.359 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 15:09:37 smithi151 systemd[1]: Stopping Ceph osd.2 for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T15:09:37.859 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 15:09:37 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2[130271]: 2024-09-13T15:09:37.388+0000 7f07d81bf640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:09:37.859 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 15:09:37 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2[130271]: 2024-09-13T15:09:37.388+0000 7f07d81bf640 -1 osd.2 1394 *** Got signal Terminated *** 2024-09-13T15:09:37.859 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 15:09:37 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2[130271]: 2024-09-13T15:09:37.388+0000 7f07d81bf640 -1 osd.2 1394 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:09:43.169 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 15:09:42 smithi151 podman[362644]: 2024-09-13 15:09:42.893530841 +0000 UTC m=+5.615316548 container died 2faaa919b127de81ef46defdf4e920eafa41865f7e62ec05a4d61ecaed748007 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-13T15:09:43.536 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 15:09:43 smithi151 podman[362644]: 2024-09-13 15:09:43.342009927 +0000 UTC m=+6.063795635 container cleanup 2faaa919b127de81ef46defdf4e920eafa41865f7e62ec05a4d61ecaed748007 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, GIT_CLEAN=True) 2024-09-13T15:09:43.537 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 15:09:43 smithi151 bash[362644]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2 2024-09-13T15:09:43.858 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 15:09:43 smithi151 podman[362666]: 2024-09-13 15:09:43.535562524 +0000 UTC m=+0.633883651 container remove 2faaa919b127de81ef46defdf4e920eafa41865f7e62ec05a4d61ecaed748007 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T15:09:44.182 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 15:09:43 smithi151 podman[362756]: 2024-09-13 15:09:43.828531669 +0000 UTC m=+0.015723731 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T15:09:44.182 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 15:09:43 smithi151 podman[362756]: 2024-09-13 15:09:43.953665042 +0000 UTC m=+0.140857092 container create fded42afb3a49561dedf2f7c274b78d63c490950deb746626cb16f3faa7b2704 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-13T15:09:44.609 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 15:09:44 smithi151 podman[362756]: 2024-09-13 15:09:44.181364748 +0000 UTC m=+0.368556797 container init fded42afb3a49561dedf2f7c274b78d63c490950deb746626cb16f3faa7b2704 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-13T15:09:44.609 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 15:09:44 smithi151 podman[362756]: 2024-09-13 15:09:44.186798541 +0000 UTC m=+0.373990593 container start fded42afb3a49561dedf2f7c274b78d63c490950deb746626cb16f3faa7b2704 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T15:09:44.609 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 15:09:44 smithi151 podman[362756]: 2024-09-13 15:09:44.273118124 +0000 UTC m=+0.460310174 container attach fded42afb3a49561dedf2f7c274b78d63c490950deb746626cb16f3faa7b2704 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T15:09:44.609 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 13 15:09:44 smithi151 podman[362756]: 2024-09-13 15:09:44.380113226 +0000 UTC m=+0.567305291 container died fded42afb3a49561dedf2f7c274b78d63c490950deb746626cb16f3faa7b2704 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2) 2024-09-13T15:09:45.237 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.2.service' 2024-09-13T15:09:45.279 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:09:45.279 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-13T15:09:45.279 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-13T15:09:45.279 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.3 2024-09-13T15:09:45.609 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 15:09:45 smithi151 systemd[1]: Stopping Ceph osd.3 for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T15:09:46.109 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 15:09:45 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3[135997]: 2024-09-13T15:09:45.628+0000 7f3b96019640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:09:46.109 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 15:09:45 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3[135997]: 2024-09-13T15:09:45.628+0000 7f3b96019640 -1 osd.3 1394 *** Got signal Terminated *** 2024-09-13T15:09:46.109 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 15:09:45 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3[135997]: 2024-09-13T15:09:45.628+0000 7f3b96019640 -1 osd.3 1394 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:09:48.104 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:09:47 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:09:47.611+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:09:48.359 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 15:09:47 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3[135997]: 2024-09-13T15:09:47.985+0000 7f3b9262e640 -1 osd.3 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:24.120170+0000 front 2024-09-13T15:09:24.120422+0000 (oldest deadline 2024-09-13T15:09:47.620313+0000) 2024-09-13T15:09:48.854 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:09:48 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:09:48.564+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:09:49.359 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 15:09:49 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3[135997]: 2024-09-13T15:09:49.027+0000 7f3b9262e640 -1 osd.3 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:24.120170+0000 front 2024-09-13T15:09:24.120422+0000 (oldest deadline 2024-09-13T15:09:47.620313+0000) 2024-09-13T15:09:49.854 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:09:49 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:09:49.557+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:09:50.359 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 15:09:50 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3[135997]: 2024-09-13T15:09:50.074+0000 7f3b9262e640 -1 osd.3 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:24.120170+0000 front 2024-09-13T15:09:24.120422+0000 (oldest deadline 2024-09-13T15:09:47.620313+0000) 2024-09-13T15:09:50.478 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:09:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T15:09:50.135+0000 7f267d4ad640 -1 osd.5 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:25.590984+0000 front 2024-09-13T15:09:25.591009+0000 (oldest deadline 2024-09-13T15:09:49.690853+0000) 2024-09-13T15:09:50.479 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:09:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:09:50.299+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:24.292807+0000 front 2024-09-13T15:09:24.292974+0000 (oldest deadline 2024-09-13T15:09:50.192777+0000) 2024-09-13T15:09:50.854 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:09:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:09:50.508+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:09:50.854 INFO:journalctl@ceph.osd.4.smithi184.stdout:Sep 13 15:09:50 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-4[38671]: 2024-09-13T15:09:50.476+0000 7fac48e3d640 -1 osd.4 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:23.714526+0000 front 2024-09-13T15:09:23.714823+0000 (oldest deadline 2024-09-13T15:09:49.614557+0000) 2024-09-13T15:09:51.450 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 15:09:51 smithi151 podman[362884]: 2024-09-13 15:09:51.142305746 +0000 UTC m=+5.629042906 container died 31c786beb443da9ab78c6900e0bd0446703939d8bee7cc8d502953400544a7d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-13T15:09:51.458 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:09:51 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T15:09:51.147+0000 7f267d4ad640 -1 osd.5 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:25.590984+0000 front 2024-09-13T15:09:25.591009+0000 (oldest deadline 2024-09-13T15:09:49.690853+0000) 2024-09-13T15:09:51.458 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:09:51 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:09:51.290+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:24.292807+0000 front 2024-09-13T15:09:24.292974+0000 (oldest deadline 2024-09-13T15:09:50.192777+0000) 2024-09-13T15:09:51.856 INFO:journalctl@ceph.osd.4.smithi184.stdout:Sep 13 15:09:51 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-4[38671]: 2024-09-13T15:09:51.456+0000 7fac48e3d640 -1 osd.4 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:23.714526+0000 front 2024-09-13T15:09:23.714823+0000 (oldest deadline 2024-09-13T15:09:49.614557+0000) 2024-09-13T15:09:51.857 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:09:51 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:09:51.527+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:09:51.858 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 15:09:51 smithi151 podman[362884]: 2024-09-13 15:09:51.623440073 +0000 UTC m=+6.110177267 container cleanup 31c786beb443da9ab78c6900e0bd0446703939d8bee7cc8d502953400544a7d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T15:09:51.858 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 15:09:51 smithi151 bash[362884]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3 2024-09-13T15:09:52.234 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 15:09:51 smithi151 podman[362907]: 2024-09-13 15:09:51.891720301 +0000 UTC m=+0.741106622 container remove 31c786beb443da9ab78c6900e0bd0446703939d8bee7cc8d502953400544a7d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T15:09:52.480 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:09:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T15:09:52.154+0000 7f267d4ad640 -1 osd.5 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:25.590984+0000 front 2024-09-13T15:09:25.591009+0000 (oldest deadline 2024-09-13T15:09:49.690853+0000) 2024-09-13T15:09:52.481 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:09:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:09:52.279+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:24.292807+0000 front 2024-09-13T15:09:24.292974+0000 (oldest deadline 2024-09-13T15:09:50.192777+0000) 2024-09-13T15:09:52.555 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 15:09:52 smithi151 podman[362998]: 2024-09-13 15:09:52.136598895 +0000 UTC m=+0.015555364 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T15:09:52.555 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 15:09:52 smithi151 podman[362998]: 2024-09-13 15:09:52.234262303 +0000 UTC m=+0.113218763 container create 541564dc2f3f3d90a368adfb6af7aa873ec817b4e045849cc7b15baaccc480e8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T15:09:52.555 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 15:09:52 smithi151 podman[362998]: 2024-09-13 15:09:52.462657947 +0000 UTC m=+0.341614402 container init 541564dc2f3f3d90a368adfb6af7aa873ec817b4e045849cc7b15baaccc480e8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD) 2024-09-13T15:09:52.555 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 15:09:52 smithi151 podman[362998]: 2024-09-13 15:09:52.467817256 +0000 UTC m=+0.346773716 container start 541564dc2f3f3d90a368adfb6af7aa873ec817b4e045849cc7b15baaccc480e8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-13T15:09:52.555 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 15:09:52 smithi151 podman[362998]: 2024-09-13 15:09:52.554123246 +0000 UTC m=+0.433079716 container attach 541564dc2f3f3d90a368adfb6af7aa873ec817b4e045849cc7b15baaccc480e8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-13T15:09:52.854 INFO:journalctl@ceph.osd.4.smithi184.stdout:Sep 13 15:09:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-4[38671]: 2024-09-13T15:09:52.476+0000 7fac48e3d640 -1 osd.4 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:23.714526+0000 front 2024-09-13T15:09:23.714823+0000 (oldest deadline 2024-09-13T15:09:49.614557+0000) 2024-09-13T15:09:52.854 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:09:52 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:09:52.542+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:09:52.859 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 13 15:09:52 smithi151 podman[362998]: 2024-09-13 15:09:52.662017705 +0000 UTC m=+0.540974173 container died 541564dc2f3f3d90a368adfb6af7aa873ec817b4e045849cc7b15baaccc480e8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T15:09:53.230 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.3.service' 2024-09-13T15:09:53.271 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:09:53.272 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-13T15:09:53.272 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-13T15:09:53.272 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.4 2024-09-13T15:09:53.442 INFO:journalctl@ceph.osd.4.smithi184.stdout:Sep 13 15:09:53 smithi184 systemd[1]: Stopping Ceph osd.4 for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T15:09:53.443 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:09:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:09:53.256+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:24.292807+0000 front 2024-09-13T15:09:24.292974+0000 (oldest deadline 2024-09-13T15:09:50.192777+0000) 2024-09-13T15:09:53.443 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:09:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T15:09:53.113+0000 7f267d4ad640 -1 osd.5 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:25.590984+0000 front 2024-09-13T15:09:25.591009+0000 (oldest deadline 2024-09-13T15:09:49.690853+0000) 2024-09-13T15:09:53.854 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:09:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:09:53.591+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:09:53.854 INFO:journalctl@ceph.osd.4.smithi184.stdout:Sep 13 15:09:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-4[38671]: 2024-09-13T15:09:53.441+0000 7fac48e3d640 -1 osd.4 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:23.714526+0000 front 2024-09-13T15:09:23.714823+0000 (oldest deadline 2024-09-13T15:09:49.614557+0000) 2024-09-13T15:09:53.854 INFO:journalctl@ceph.osd.4.smithi184.stdout:Sep 13 15:09:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-4[38671]: 2024-09-13T15:09:53.573+0000 7fac4d827640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:09:53.855 INFO:journalctl@ceph.osd.4.smithi184.stdout:Sep 13 15:09:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-4[38671]: 2024-09-13T15:09:53.573+0000 7fac4d827640 -1 osd.4 1394 *** Got signal Terminated *** 2024-09-13T15:09:53.855 INFO:journalctl@ceph.osd.4.smithi184.stdout:Sep 13 15:09:53 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-4[38671]: 2024-09-13T15:09:53.573+0000 7fac4d827640 -1 osd.4 1394 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:09:54.354 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:09:54 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T15:09:54.091+0000 7f267d4ad640 -1 osd.5 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:25.590984+0000 front 2024-09-13T15:09:25.591009+0000 (oldest deadline 2024-09-13T15:09:49.690853+0000) 2024-09-13T15:09:54.354 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:09:54 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:09:54.296+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:24.292807+0000 front 2024-09-13T15:09:24.292974+0000 (oldest deadline 2024-09-13T15:09:50.192777+0000) 2024-09-13T15:09:54.854 INFO:journalctl@ceph.osd.4.smithi184.stdout:Sep 13 15:09:54 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-4[38671]: 2024-09-13T15:09:54.469+0000 7fac48e3d640 -1 osd.4 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:23.714526+0000 front 2024-09-13T15:09:23.714823+0000 (oldest deadline 2024-09-13T15:09:49.614557+0000) 2024-09-13T15:09:54.854 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:09:54 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:09:54.636+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:09:55.494 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:09:55 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T15:09:55.105+0000 7f267d4ad640 -1 osd.5 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:25.590984+0000 front 2024-09-13T15:09:25.591009+0000 (oldest deadline 2024-09-13T15:09:49.690853+0000) 2024-09-13T15:09:55.495 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:09:55 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:09:55.308+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:24.292807+0000 front 2024-09-13T15:09:24.292974+0000 (oldest deadline 2024-09-13T15:09:50.192777+0000) 2024-09-13T15:09:55.854 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:09:55 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:09:55.617+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:09:55.854 INFO:journalctl@ceph.osd.4.smithi184.stdout:Sep 13 15:09:55 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-4[38671]: 2024-09-13T15:09:55.492+0000 7fac48e3d640 -1 osd.4 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:23.714526+0000 front 2024-09-13T15:09:23.714823+0000 (oldest deadline 2024-09-13T15:09:49.614557+0000) 2024-09-13T15:09:55.854 INFO:journalctl@ceph.osd.4.smithi184.stdout:Sep 13 15:09:55 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-4[38671]: 2024-09-13T15:09:55.492+0000 7fac48e3d640 -1 osd.4 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:33.115661+0000 front 2024-09-13T15:09:33.115631+0000 (oldest deadline 2024-09-13T15:09:54.815664+0000) 2024-09-13T15:09:56.470 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:09:56 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:09:56.350+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:24.292807+0000 front 2024-09-13T15:09:24.292974+0000 (oldest deadline 2024-09-13T15:09:50.192777+0000) 2024-09-13T15:09:56.470 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:09:56 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T15:09:56.125+0000 7f267d4ad640 -1 osd.5 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:25.590984+0000 front 2024-09-13T15:09:25.591009+0000 (oldest deadline 2024-09-13T15:09:49.690853+0000) 2024-09-13T15:09:56.471 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:09:56 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T15:09:56.125+0000 7f267d4ad640 -1 osd.5 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:29.691546+0000 front 2024-09-13T15:09:29.691498+0000 (oldest deadline 2024-09-13T15:09:55.591497+0000) 2024-09-13T15:09:56.854 INFO:journalctl@ceph.osd.4.smithi184.stdout:Sep 13 15:09:56 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-4[38671]: 2024-09-13T15:09:56.469+0000 7fac48e3d640 -1 osd.4 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:23.714526+0000 front 2024-09-13T15:09:23.714823+0000 (oldest deadline 2024-09-13T15:09:49.614557+0000) 2024-09-13T15:09:56.854 INFO:journalctl@ceph.osd.4.smithi184.stdout:Sep 13 15:09:56 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-4[38671]: 2024-09-13T15:09:56.469+0000 7fac48e3d640 -1 osd.4 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:33.115661+0000 front 2024-09-13T15:09:33.115631+0000 (oldest deadline 2024-09-13T15:09:54.815664+0000) 2024-09-13T15:09:56.854 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:09:56 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:09:56.601+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:09:56.855 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:09:56 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:09:56.601+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.572819+0000 front 2024-09-13T15:09:30.572762+0000 (oldest deadline 2024-09-13T15:09:55.872702+0000) 2024-09-13T15:09:57.499 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:09:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:09:57.303+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:24.292807+0000 front 2024-09-13T15:09:24.292974+0000 (oldest deadline 2024-09-13T15:09:50.192777+0000) 2024-09-13T15:09:57.499 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:09:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T15:09:57.160+0000 7f267d4ad640 -1 osd.5 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:25.590984+0000 front 2024-09-13T15:09:25.591009+0000 (oldest deadline 2024-09-13T15:09:49.690853+0000) 2024-09-13T15:09:57.499 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:09:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T15:09:57.160+0000 7f267d4ad640 -1 osd.5 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:29.691546+0000 front 2024-09-13T15:09:29.691498+0000 (oldest deadline 2024-09-13T15:09:55.591497+0000) 2024-09-13T15:09:57.854 INFO:journalctl@ceph.osd.4.smithi184.stdout:Sep 13 15:09:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-4[38671]: 2024-09-13T15:09:57.497+0000 7fac48e3d640 -1 osd.4 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:23.714526+0000 front 2024-09-13T15:09:23.714823+0000 (oldest deadline 2024-09-13T15:09:49.614557+0000) 2024-09-13T15:09:57.854 INFO:journalctl@ceph.osd.4.smithi184.stdout:Sep 13 15:09:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-4[38671]: 2024-09-13T15:09:57.497+0000 7fac48e3d640 -1 osd.4 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:33.115661+0000 front 2024-09-13T15:09:33.115631+0000 (oldest deadline 2024-09-13T15:09:54.815664+0000) 2024-09-13T15:09:57.854 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:09:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:09:57.583+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:09:57.855 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:09:57 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:09:57.583+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.572819+0000 front 2024-09-13T15:09:30.572762+0000 (oldest deadline 2024-09-13T15:09:55.872702+0000) 2024-09-13T15:09:58.505 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:09:58 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:09:58.313+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:24.292807+0000 front 2024-09-13T15:09:24.292974+0000 (oldest deadline 2024-09-13T15:09:50.192777+0000) 2024-09-13T15:09:58.506 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:09:58 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T15:09:58.135+0000 7f267d4ad640 -1 osd.5 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:25.590984+0000 front 2024-09-13T15:09:25.591009+0000 (oldest deadline 2024-09-13T15:09:49.690853+0000) 2024-09-13T15:09:58.506 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:09:58 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T15:09:58.135+0000 7f267d4ad640 -1 osd.5 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:29.691546+0000 front 2024-09-13T15:09:29.691498+0000 (oldest deadline 2024-09-13T15:09:55.591497+0000) 2024-09-13T15:09:58.854 INFO:journalctl@ceph.osd.4.smithi184.stdout:Sep 13 15:09:58 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-4[38671]: 2024-09-13T15:09:58.503+0000 7fac48e3d640 -1 osd.4 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:23.714526+0000 front 2024-09-13T15:09:23.714823+0000 (oldest deadline 2024-09-13T15:09:49.614557+0000) 2024-09-13T15:09:58.854 INFO:journalctl@ceph.osd.4.smithi184.stdout:Sep 13 15:09:58 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-4[38671]: 2024-09-13T15:09:58.503+0000 7fac48e3d640 -1 osd.4 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:33.115661+0000 front 2024-09-13T15:09:33.115631+0000 (oldest deadline 2024-09-13T15:09:54.815664+0000) 2024-09-13T15:09:58.855 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:09:58 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:09:58.610+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:09:58.855 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:09:58 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:09:58.610+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.572819+0000 front 2024-09-13T15:09:30.572762+0000 (oldest deadline 2024-09-13T15:09:55.872702+0000) 2024-09-13T15:09:59.304 INFO:journalctl@ceph.osd.4.smithi184.stdout:Sep 13 15:09:59 smithi184 podman[246429]: 2024-09-13 15:09:59.043265761 +0000 UTC m=+5.569880867 container died 6e70699b8084a8bdc434725021b3d1e0e48ed5112b76c8d91c3003e3aaa5a4a5 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-4, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-bbdadfa, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-13T15:09:59.304 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:09:59 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T15:09:59.172+0000 7f267d4ad640 -1 osd.5 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:25.590984+0000 front 2024-09-13T15:09:25.591009+0000 (oldest deadline 2024-09-13T15:09:49.690853+0000) 2024-09-13T15:09:59.304 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:09:59 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T15:09:59.172+0000 7f267d4ad640 -1 osd.5 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:29.691546+0000 front 2024-09-13T15:09:29.691498+0000 (oldest deadline 2024-09-13T15:09:55.591497+0000) 2024-09-13T15:09:59.604 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:09:59 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:09:59.302+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:24.292807+0000 front 2024-09-13T15:09:24.292974+0000 (oldest deadline 2024-09-13T15:09:50.192777+0000) 2024-09-13T15:09:59.604 INFO:journalctl@ceph.osd.4.smithi184.stdout:Sep 13 15:09:59 smithi184 podman[246429]: 2024-09-13 15:09:59.404451497 +0000 UTC m=+5.931066603 container cleanup 6e70699b8084a8bdc434725021b3d1e0e48ed5112b76c8d91c3003e3aaa5a4a5 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-4, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=quincy-bbdadfa, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T15:09:59.604 INFO:journalctl@ceph.osd.4.smithi184.stdout:Sep 13 15:09:59 smithi184 bash[246429]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-4 2024-09-13T15:10:00.046 INFO:journalctl@ceph.osd.4.smithi184.stdout:Sep 13 15:09:59 smithi184 podman[246451]: 2024-09-13 15:09:59.705681931 +0000 UTC m=+0.657530699 container remove 6e70699b8084a8bdc434725021b3d1e0e48ed5112b76c8d91c3003e3aaa5a4a5 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-4, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, RELEASE=quincy-bbdadfa) 2024-09-13T15:10:00.047 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:09:59 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:09:59.633+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:10:00.047 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:09:59 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:09:59.633+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.572819+0000 front 2024-09-13T15:09:30.572762+0000 (oldest deadline 2024-09-13T15:09:55.872702+0000) 2024-09-13T15:10:00.298 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:10:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T15:10:00.187+0000 7f267d4ad640 -1 osd.5 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:25.590984+0000 front 2024-09-13T15:09:25.591009+0000 (oldest deadline 2024-09-13T15:09:49.690853+0000) 2024-09-13T15:10:00.299 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:10:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T15:10:00.187+0000 7f267d4ad640 -1 osd.5 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:29.691546+0000 front 2024-09-13T15:09:29.691498+0000 (oldest deadline 2024-09-13T15:09:55.591497+0000) 2024-09-13T15:10:00.299 INFO:journalctl@ceph.osd.4.smithi184.stdout:Sep 13 15:10:00 smithi184 podman[246548]: 2024-09-13 15:09:59.94857699 +0000 UTC m=+0.015889711 image pull a994b00deb3a4a8eeb484a456d3f3207aae0401a31d251b199b2c9dbc1c4036c quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be 2024-09-13T15:10:00.299 INFO:journalctl@ceph.osd.4.smithi184.stdout:Sep 13 15:10:00 smithi184 podman[246548]: 2024-09-13 15:10:00.073952814 +0000 UTC m=+0.141265569 container create 5ef1052db2a36a651c02feb0af66b59747cc314c96c8d5edd2f803adfa53d7ae (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=quincy-bbdadfa, maintainer=Guillaume Abrioux ) 2024-09-13T15:10:00.604 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:00.295+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:24.292807+0000 front 2024-09-13T15:09:24.292974+0000 (oldest deadline 2024-09-13T15:09:50.192777+0000) 2024-09-13T15:10:00.604 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:00.295+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.193348+0000 front 2024-09-13T15:09:30.193379+0000 (oldest deadline 2024-09-13T15:10:00.193870+0000) 2024-09-13T15:10:00.604 INFO:journalctl@ceph.osd.4.smithi184.stdout:Sep 13 15:10:00 smithi184 podman[246548]: 2024-09-13 15:10:00.359591124 +0000 UTC m=+0.426903849 container init 5ef1052db2a36a651c02feb0af66b59747cc314c96c8d5edd2f803adfa53d7ae (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=quincy-bbdadfa, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-13T15:10:00.605 INFO:journalctl@ceph.osd.4.smithi184.stdout:Sep 13 15:10:00 smithi184 podman[246548]: 2024-09-13 15:10:00.364483797 +0000 UTC m=+0.431796519 container start 5ef1052db2a36a651c02feb0af66b59747cc314c96c8d5edd2f803adfa53d7ae (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-4-deactivate, ceph=True, org.label-schema.build-date=20240731, RELEASE=quincy-bbdadfa, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T15:10:00.605 INFO:journalctl@ceph.osd.4.smithi184.stdout:Sep 13 15:10:00 smithi184 podman[246548]: 2024-09-13 15:10:00.467854322 +0000 UTC m=+0.535167094 container attach 5ef1052db2a36a651c02feb0af66b59747cc314c96c8d5edd2f803adfa53d7ae (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=quincy-bbdadfa) 2024-09-13T15:10:00.938 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:00.640+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:10:00.939 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:00 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:00.640+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.572819+0000 front 2024-09-13T15:09:30.572762+0000 (oldest deadline 2024-09-13T15:09:55.872702+0000) 2024-09-13T15:10:00.939 INFO:journalctl@ceph.osd.4.smithi184.stdout:Sep 13 15:10:00 smithi184 podman[246548]: 2024-09-13 15:10:00.626345608 +0000 UTC m=+0.693658334 container died 5ef1052db2a36a651c02feb0af66b59747cc314c96c8d5edd2f803adfa53d7ae (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=quincy-bbdadfa, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T15:10:01.236 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.4.service' 2024-09-13T15:10:01.281 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:10:01.281 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-13T15:10:01.281 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-13T15:10:01.282 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.5 2024-09-13T15:10:01.604 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:01 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:01.311+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:24.292807+0000 front 2024-09-13T15:09:24.292974+0000 (oldest deadline 2024-09-13T15:09:50.192777+0000) 2024-09-13T15:10:01.604 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:01 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:01.311+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.193348+0000 front 2024-09-13T15:09:30.193379+0000 (oldest deadline 2024-09-13T15:10:00.193870+0000) 2024-09-13T15:10:01.605 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:10:01 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T15:10:01.221+0000 7f267d4ad640 -1 osd.5 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:25.590984+0000 front 2024-09-13T15:09:25.591009+0000 (oldest deadline 2024-09-13T15:09:49.690853+0000) 2024-09-13T15:10:01.605 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:10:01 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T15:10:01.221+0000 7f267d4ad640 -1 osd.5 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:29.691546+0000 front 2024-09-13T15:09:29.691498+0000 (oldest deadline 2024-09-13T15:09:55.591497+0000) 2024-09-13T15:10:01.605 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:10:01 smithi184 systemd[1]: Stopping Ceph osd.5 for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T15:10:02.104 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:10:01 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T15:10:01.644+0000 7f2682e99640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:10:02.105 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:10:01 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T15:10:01.644+0000 7f2682e99640 -1 osd.5 1394 *** Got signal Terminated *** 2024-09-13T15:10:02.105 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:10:01 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T15:10:01.644+0000 7f2682e99640 -1 osd.5 1394 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:10:02.105 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:01 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:01.638+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:10:02.105 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:01 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:01.638+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.572819+0000 front 2024-09-13T15:09:30.572762+0000 (oldest deadline 2024-09-13T15:09:55.872702+0000) 2024-09-13T15:10:02.604 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:10:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T15:10:02.172+0000 7f267d4ad640 -1 osd.5 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:25.590984+0000 front 2024-09-13T15:09:25.591009+0000 (oldest deadline 2024-09-13T15:09:49.690853+0000) 2024-09-13T15:10:02.604 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:10:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T15:10:02.172+0000 7f267d4ad640 -1 osd.5 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:29.691546+0000 front 2024-09-13T15:09:29.691498+0000 (oldest deadline 2024-09-13T15:09:55.591497+0000) 2024-09-13T15:10:02.604 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:02.338+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:24.292807+0000 front 2024-09-13T15:09:24.292974+0000 (oldest deadline 2024-09-13T15:09:50.192777+0000) 2024-09-13T15:10:02.605 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:02.338+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.193348+0000 front 2024-09-13T15:09:30.193379+0000 (oldest deadline 2024-09-13T15:10:00.193870+0000) 2024-09-13T15:10:03.104 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:02.642+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:10:03.104 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:02 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:02.642+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.572819+0000 front 2024-09-13T15:09:30.572762+0000 (oldest deadline 2024-09-13T15:09:55.872702+0000) 2024-09-13T15:10:03.604 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:10:03 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T15:10:03.219+0000 7f267d4ad640 -1 osd.5 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:25.590984+0000 front 2024-09-13T15:09:25.591009+0000 (oldest deadline 2024-09-13T15:09:49.690853+0000) 2024-09-13T15:10:03.604 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:10:03 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T15:10:03.219+0000 7f267d4ad640 -1 osd.5 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:29.691546+0000 front 2024-09-13T15:09:29.691498+0000 (oldest deadline 2024-09-13T15:09:55.591497+0000) 2024-09-13T15:10:03.605 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:03 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:03.323+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:24.292807+0000 front 2024-09-13T15:09:24.292974+0000 (oldest deadline 2024-09-13T15:09:50.192777+0000) 2024-09-13T15:10:03.605 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:03 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:03.323+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.193348+0000 front 2024-09-13T15:09:30.193379+0000 (oldest deadline 2024-09-13T15:10:00.193870+0000) 2024-09-13T15:10:03.605 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:03 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:03.323+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-13T15:09:40.194496+0000 front 2024-09-13T15:09:40.194474+0000 (oldest deadline 2024-09-13T15:10:03.094276+0000) 2024-09-13T15:10:04.104 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:03 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:03.608+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:10:04.104 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:03 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:03.608+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.572819+0000 front 2024-09-13T15:09:30.572762+0000 (oldest deadline 2024-09-13T15:09:55.872702+0000) 2024-09-13T15:10:04.599 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:10:04 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T15:10:04.266+0000 7f267d4ad640 -1 osd.5 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:25.590984+0000 front 2024-09-13T15:09:25.591009+0000 (oldest deadline 2024-09-13T15:09:49.690853+0000) 2024-09-13T15:10:04.600 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:10:04 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T15:10:04.266+0000 7f267d4ad640 -1 osd.5 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:29.691546+0000 front 2024-09-13T15:09:29.691498+0000 (oldest deadline 2024-09-13T15:09:55.591497+0000) 2024-09-13T15:10:04.600 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:04 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:04.344+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:24.292807+0000 front 2024-09-13T15:09:24.292974+0000 (oldest deadline 2024-09-13T15:09:50.192777+0000) 2024-09-13T15:10:04.600 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:04 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:04.344+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.193348+0000 front 2024-09-13T15:09:30.193379+0000 (oldest deadline 2024-09-13T15:10:00.193870+0000) 2024-09-13T15:10:04.600 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:04 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:04.344+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-13T15:09:40.194496+0000 front 2024-09-13T15:09:40.194474+0000 (oldest deadline 2024-09-13T15:10:03.094276+0000) 2024-09-13T15:10:04.854 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:04 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:04.597+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:10:04.854 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:04 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:04.597+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.572819+0000 front 2024-09-13T15:09:30.572762+0000 (oldest deadline 2024-09-13T15:09:55.872702+0000) 2024-09-13T15:10:05.604 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:10:05 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T15:10:05.293+0000 7f267d4ad640 -1 osd.5 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:25.590984+0000 front 2024-09-13T15:09:25.591009+0000 (oldest deadline 2024-09-13T15:09:49.690853+0000) 2024-09-13T15:10:05.604 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:10:05 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T15:10:05.293+0000 7f267d4ad640 -1 osd.5 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:29.691546+0000 front 2024-09-13T15:09:29.691498+0000 (oldest deadline 2024-09-13T15:09:55.591497+0000) 2024-09-13T15:10:05.605 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:05 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:05.346+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:24.292807+0000 front 2024-09-13T15:09:24.292974+0000 (oldest deadline 2024-09-13T15:09:50.192777+0000) 2024-09-13T15:10:05.605 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:05 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:05.346+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.193348+0000 front 2024-09-13T15:09:30.193379+0000 (oldest deadline 2024-09-13T15:10:00.193870+0000) 2024-09-13T15:10:05.605 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:05 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:05.346+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-13T15:09:40.194496+0000 front 2024-09-13T15:09:40.194474+0000 (oldest deadline 2024-09-13T15:10:03.094276+0000) 2024-09-13T15:10:06.104 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:05 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:05.621+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:10:06.104 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:05 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:05.621+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.572819+0000 front 2024-09-13T15:09:30.572762+0000 (oldest deadline 2024-09-13T15:09:55.872702+0000) 2024-09-13T15:10:06.104 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:05 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:05.621+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-13T15:09:40.573748+0000 front 2024-09-13T15:09:40.573668+0000 (oldest deadline 2024-09-13T15:10:05.273646+0000) 2024-09-13T15:10:06.604 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:10:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T15:10:06.302+0000 7f267d4ad640 -1 osd.5 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:25.590984+0000 front 2024-09-13T15:09:25.591009+0000 (oldest deadline 2024-09-13T15:09:49.690853+0000) 2024-09-13T15:10:06.604 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:10:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5[45042]: 2024-09-13T15:10:06.302+0000 7f267d4ad640 -1 osd.5 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:29.691546+0000 front 2024-09-13T15:09:29.691498+0000 (oldest deadline 2024-09-13T15:09:55.591497+0000) 2024-09-13T15:10:06.604 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:06.302+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:24.292807+0000 front 2024-09-13T15:09:24.292974+0000 (oldest deadline 2024-09-13T15:09:50.192777+0000) 2024-09-13T15:10:06.605 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:06.302+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.193348+0000 front 2024-09-13T15:09:30.193379+0000 (oldest deadline 2024-09-13T15:10:00.193870+0000) 2024-09-13T15:10:06.605 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:06.302+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-13T15:09:40.194496+0000 front 2024-09-13T15:09:40.194474+0000 (oldest deadline 2024-09-13T15:10:03.094276+0000) 2024-09-13T15:10:07.100 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:06.625+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:10:07.101 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:06.625+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.572819+0000 front 2024-09-13T15:09:30.572762+0000 (oldest deadline 2024-09-13T15:09:55.872702+0000) 2024-09-13T15:10:07.101 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:06 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:06.625+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-13T15:09:40.573748+0000 front 2024-09-13T15:09:40.573668+0000 (oldest deadline 2024-09-13T15:10:05.273646+0000) 2024-09-13T15:10:07.354 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:10:07 smithi184 podman[246675]: 2024-09-13 15:10:07.099831795 +0000 UTC m=+5.596337995 container died 36b82c9e7230fea4e2b8395e85473435808d2eb6d1b1a72edc32410e68854470 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5, RELEASE=quincy-bbdadfa, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-13T15:10:07.354 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:07 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:07.344+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:24.292807+0000 front 2024-09-13T15:09:24.292974+0000 (oldest deadline 2024-09-13T15:09:50.192777+0000) 2024-09-13T15:10:07.355 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:07 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:07.344+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.193348+0000 front 2024-09-13T15:09:30.193379+0000 (oldest deadline 2024-09-13T15:10:00.193870+0000) 2024-09-13T15:10:07.355 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:07 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:07.344+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-13T15:09:40.194496+0000 front 2024-09-13T15:09:40.194474+0000 (oldest deadline 2024-09-13T15:10:03.094276+0000) 2024-09-13T15:10:07.665 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:10:07 smithi184 podman[246675]: 2024-09-13 15:10:07.474123901 +0000 UTC m=+5.970630089 container cleanup 36b82c9e7230fea4e2b8395e85473435808d2eb6d1b1a72edc32410e68854470 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=quincy-bbdadfa, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-13T15:10:07.666 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:10:07 smithi184 bash[246675]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5 2024-09-13T15:10:08.104 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:10:07 smithi184 podman[246698]: 2024-09-13 15:10:07.800427262 +0000 UTC m=+0.689621022 container remove 36b82c9e7230fea4e2b8395e85473435808d2eb6d1b1a72edc32410e68854470 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=quincy-bbdadfa, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T15:10:08.104 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:07 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:07.663+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:10:08.105 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:07 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:07.663+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.572819+0000 front 2024-09-13T15:09:30.572762+0000 (oldest deadline 2024-09-13T15:09:55.872702+0000) 2024-09-13T15:10:08.105 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:07 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:07.663+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-13T15:09:40.573748+0000 front 2024-09-13T15:09:40.573668+0000 (oldest deadline 2024-09-13T15:10:05.273646+0000) 2024-09-13T15:10:08.429 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:08 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:08.309+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:24.292807+0000 front 2024-09-13T15:09:24.292974+0000 (oldest deadline 2024-09-13T15:09:50.192777+0000) 2024-09-13T15:10:08.430 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:08 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:08.309+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.193348+0000 front 2024-09-13T15:09:30.193379+0000 (oldest deadline 2024-09-13T15:10:00.193870+0000) 2024-09-13T15:10:08.430 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:08 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:08.309+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-13T15:09:40.194496+0000 front 2024-09-13T15:09:40.194474+0000 (oldest deadline 2024-09-13T15:10:03.094276+0000) 2024-09-13T15:10:08.430 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:10:08 smithi184 podman[246791]: 2024-09-13 15:10:08.118007014 +0000 UTC m=+0.105910557 container create c1f1a0e2dbadeac9232c865ec7ee6e899225df40af831554e93d2e0042f6e4a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5-deactivate, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=quincy-bbdadfa, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T15:10:08.430 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:10:08 smithi184 podman[246791]: 2024-09-13 15:10:08.027670156 +0000 UTC m=+0.015573704 image pull a994b00deb3a4a8eeb484a456d3f3207aae0401a31d251b199b2c9dbc1c4036c quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be 2024-09-13T15:10:08.430 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:10:08 smithi184 podman[246791]: 2024-09-13 15:10:08.337123702 +0000 UTC m=+0.325027245 container init c1f1a0e2dbadeac9232c865ec7ee6e899225df40af831554e93d2e0042f6e4a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=quincy-bbdadfa, CEPH_POINT_RELEASE=) 2024-09-13T15:10:08.430 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:10:08 smithi184 podman[246791]: 2024-09-13 15:10:08.342131189 +0000 UTC m=+0.330034732 container start c1f1a0e2dbadeac9232c865ec7ee6e899225df40af831554e93d2e0042f6e4a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-bbdadfa) 2024-09-13T15:10:08.431 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:10:08 smithi184 podman[246791]: 2024-09-13 15:10:08.428557757 +0000 UTC m=+0.416461312 container attach c1f1a0e2dbadeac9232c865ec7ee6e899225df40af831554e93d2e0042f6e4a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=quincy-bbdadfa, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-13T15:10:08.816 INFO:journalctl@ceph.osd.5.smithi184.stdout:Sep 13 15:10:08 smithi184 podman[246791]: 2024-09-13 15:10:08.512736296 +0000 UTC m=+0.500639845 container died c1f1a0e2dbadeac9232c865ec7ee6e899225df40af831554e93d2e0042f6e4a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-5-deactivate, RELEASE=quincy-bbdadfa, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-13T15:10:08.816 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:08 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:08.653+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:10:08.817 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:08 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:08.653+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.572819+0000 front 2024-09-13T15:09:30.572762+0000 (oldest deadline 2024-09-13T15:09:55.872702+0000) 2024-09-13T15:10:08.817 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:08 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:08.653+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-13T15:09:40.573748+0000 front 2024-09-13T15:09:40.573668+0000 (oldest deadline 2024-09-13T15:10:05.273646+0000) 2024-09-13T15:10:09.105 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.5.service' 2024-09-13T15:10:09.146 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:10:09.146 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-13T15:10:09.147 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-13T15:10:09.147 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.6 2024-09-13T15:10:09.354 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:09 smithi184 systemd[1]: Stopping Ceph osd.6 for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T15:10:09.354 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:09 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:09.297+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:24.292807+0000 front 2024-09-13T15:09:24.292974+0000 (oldest deadline 2024-09-13T15:09:50.192777+0000) 2024-09-13T15:10:09.354 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:09 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:09.297+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.193348+0000 front 2024-09-13T15:09:30.193379+0000 (oldest deadline 2024-09-13T15:10:00.193870+0000) 2024-09-13T15:10:09.354 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:09 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:09.297+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-13T15:09:40.194496+0000 front 2024-09-13T15:09:40.194474+0000 (oldest deadline 2024-09-13T15:10:03.094276+0000) 2024-09-13T15:10:09.854 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:09 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:09.470+0000 7f9f3e365640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:10:09.854 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:09 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:09.470+0000 7f9f3e365640 -1 osd.6 1394 *** Got signal Terminated *** 2024-09-13T15:10:09.854 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:09 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:09.470+0000 7f9f3e365640 -1 osd.6 1394 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:10:09.855 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:09 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:09.689+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:10:09.855 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:09 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:09.689+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.572819+0000 front 2024-09-13T15:09:30.572762+0000 (oldest deadline 2024-09-13T15:09:55.872702+0000) 2024-09-13T15:10:09.855 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:09 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:09.689+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-13T15:09:40.573748+0000 front 2024-09-13T15:09:40.573668+0000 (oldest deadline 2024-09-13T15:10:05.273646+0000) 2024-09-13T15:10:10.604 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:10.261+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:24.292807+0000 front 2024-09-13T15:09:24.292974+0000 (oldest deadline 2024-09-13T15:09:50.192777+0000) 2024-09-13T15:10:10.604 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:10.261+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.193348+0000 front 2024-09-13T15:09:30.193379+0000 (oldest deadline 2024-09-13T15:10:00.193870+0000) 2024-09-13T15:10:10.604 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:10.261+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-13T15:09:40.194496+0000 front 2024-09-13T15:09:40.194474+0000 (oldest deadline 2024-09-13T15:10:03.094276+0000) 2024-09-13T15:10:11.104 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:10.697+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:10:11.104 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:10.697+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.572819+0000 front 2024-09-13T15:09:30.572762+0000 (oldest deadline 2024-09-13T15:09:55.872702+0000) 2024-09-13T15:10:11.104 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:10 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:10.697+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-13T15:09:40.573748+0000 front 2024-09-13T15:09:40.573668+0000 (oldest deadline 2024-09-13T15:10:05.273646+0000) 2024-09-13T15:10:11.604 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:11 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:11.296+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:24.292807+0000 front 2024-09-13T15:09:24.292974+0000 (oldest deadline 2024-09-13T15:09:50.192777+0000) 2024-09-13T15:10:11.604 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:11 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:11.296+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.193348+0000 front 2024-09-13T15:09:30.193379+0000 (oldest deadline 2024-09-13T15:10:00.193870+0000) 2024-09-13T15:10:11.604 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:11 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:11.296+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-13T15:09:40.194496+0000 front 2024-09-13T15:09:40.194474+0000 (oldest deadline 2024-09-13T15:10:03.094276+0000) 2024-09-13T15:10:12.104 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:11 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:11.652+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:10:12.104 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:11 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:11.652+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.572819+0000 front 2024-09-13T15:09:30.572762+0000 (oldest deadline 2024-09-13T15:09:55.872702+0000) 2024-09-13T15:10:12.104 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:11 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:11.652+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-13T15:09:40.573748+0000 front 2024-09-13T15:09:40.573668+0000 (oldest deadline 2024-09-13T15:10:05.273646+0000) 2024-09-13T15:10:12.604 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:12 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:12.254+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:24.292807+0000 front 2024-09-13T15:09:24.292974+0000 (oldest deadline 2024-09-13T15:09:50.192777+0000) 2024-09-13T15:10:12.604 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:12 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:12.254+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.193348+0000 front 2024-09-13T15:09:30.193379+0000 (oldest deadline 2024-09-13T15:10:00.193870+0000) 2024-09-13T15:10:12.604 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:12 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:12.254+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-13T15:09:40.194496+0000 front 2024-09-13T15:09:40.194474+0000 (oldest deadline 2024-09-13T15:10:03.094276+0000) 2024-09-13T15:10:13.104 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:12 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:12.661+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:10:13.104 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:12 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:12.661+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.572819+0000 front 2024-09-13T15:09:30.572762+0000 (oldest deadline 2024-09-13T15:09:55.872702+0000) 2024-09-13T15:10:13.104 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:12 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:12.661+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-13T15:09:40.573748+0000 front 2024-09-13T15:09:40.573668+0000 (oldest deadline 2024-09-13T15:10:05.273646+0000) 2024-09-13T15:10:13.604 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:13 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:13.267+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:24.292807+0000 front 2024-09-13T15:09:24.292974+0000 (oldest deadline 2024-09-13T15:09:50.192777+0000) 2024-09-13T15:10:13.604 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:13 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:13.267+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.193348+0000 front 2024-09-13T15:09:30.193379+0000 (oldest deadline 2024-09-13T15:10:00.193870+0000) 2024-09-13T15:10:13.604 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:13 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:13.267+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-13T15:09:40.194496+0000 front 2024-09-13T15:09:40.194474+0000 (oldest deadline 2024-09-13T15:10:03.094276+0000) 2024-09-13T15:10:14.104 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:13 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:13.616+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:10:14.104 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:13 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:13.616+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.572819+0000 front 2024-09-13T15:09:30.572762+0000 (oldest deadline 2024-09-13T15:09:55.872702+0000) 2024-09-13T15:10:14.104 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:13 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:13.616+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-13T15:09:40.573748+0000 front 2024-09-13T15:09:40.573668+0000 (oldest deadline 2024-09-13T15:10:05.273646+0000) 2024-09-13T15:10:14.104 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:13 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:13.616+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-13T15:09:49.374582+0000 front 2024-09-13T15:09:49.374522+0000 (oldest deadline 2024-09-13T15:10:12.874435+0000) 2024-09-13T15:10:14.584 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:14 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:14.261+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:24.292807+0000 front 2024-09-13T15:09:24.292974+0000 (oldest deadline 2024-09-13T15:09:50.192777+0000) 2024-09-13T15:10:14.585 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:14 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:14.261+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.193348+0000 front 2024-09-13T15:09:30.193379+0000 (oldest deadline 2024-09-13T15:10:00.193870+0000) 2024-09-13T15:10:14.585 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:14 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6[51256]: 2024-09-13T15:10:14.261+0000 7f9f38178640 -1 osd.6 1394 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-13T15:09:40.194496+0000 front 2024-09-13T15:09:40.194474+0000 (oldest deadline 2024-09-13T15:10:03.094276+0000) 2024-09-13T15:10:14.854 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:14 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:14.582+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:10:14.854 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:14 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:14.582+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.572819+0000 front 2024-09-13T15:09:30.572762+0000 (oldest deadline 2024-09-13T15:09:55.872702+0000) 2024-09-13T15:10:14.854 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:14 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:14.582+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-13T15:09:40.573748+0000 front 2024-09-13T15:09:40.573668+0000 (oldest deadline 2024-09-13T15:10:05.273646+0000) 2024-09-13T15:10:14.855 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:14 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:14.582+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-13T15:09:49.374582+0000 front 2024-09-13T15:09:49.374522+0000 (oldest deadline 2024-09-13T15:10:12.874435+0000) 2024-09-13T15:10:15.445 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:15 smithi184 podman[246919]: 2024-09-13 15:10:15.143293976 +0000 UTC m=+5.778234490 container died f9379cec75045b42da38abac047241cbddae361ccd40209e3c3cd67baf0fe483 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6, RELEASE=quincy-bbdadfa, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-13T15:10:15.837 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:15 smithi184 podman[246919]: 2024-09-13 15:10:15.534706516 +0000 UTC m=+6.169647020 container cleanup f9379cec75045b42da38abac047241cbddae361ccd40209e3c3cd67baf0fe483 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=quincy-bbdadfa, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-13T15:10:15.837 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:15 smithi184 bash[246919]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6 2024-09-13T15:10:15.838 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:15.551+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:10:15.838 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:15.551+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.572819+0000 front 2024-09-13T15:09:30.572762+0000 (oldest deadline 2024-09-13T15:09:55.872702+0000) 2024-09-13T15:10:15.838 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:15.551+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-13T15:09:40.573748+0000 front 2024-09-13T15:09:40.573668+0000 (oldest deadline 2024-09-13T15:10:05.273646+0000) 2024-09-13T15:10:15.838 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:15 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:15.551+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-13T15:09:49.374582+0000 front 2024-09-13T15:09:49.374522+0000 (oldest deadline 2024-09-13T15:10:12.874435+0000) 2024-09-13T15:10:16.104 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:15 smithi184 podman[246940]: 2024-09-13 15:10:15.835983864 +0000 UTC m=+0.688340898 container remove f9379cec75045b42da38abac047241cbddae361ccd40209e3c3cd67baf0fe483 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-bbdadfa, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-13T15:10:16.473 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:16 smithi184 podman[247033]: 2024-09-13 15:10:16.051766257 +0000 UTC m=+0.016059439 image pull a994b00deb3a4a8eeb484a456d3f3207aae0401a31d251b199b2c9dbc1c4036c quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be 2024-09-13T15:10:16.474 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:16 smithi184 podman[247033]: 2024-09-13 15:10:16.153621365 +0000 UTC m=+0.117914583 container create e02696e4e2153906a812027fcb7c056b57fef07e89d768cf83795839483fd103 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=quincy-bbdadfa, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T15:10:16.474 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:16 smithi184 podman[247033]: 2024-09-13 15:10:16.372331046 +0000 UTC m=+0.336624223 container init e02696e4e2153906a812027fcb7c056b57fef07e89d768cf83795839483fd103 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6-deactivate, RELEASE=quincy-bbdadfa, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T15:10:16.474 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:16 smithi184 podman[247033]: 2024-09-13 15:10:16.37713549 +0000 UTC m=+0.341428668 container start e02696e4e2153906a812027fcb7c056b57fef07e89d768cf83795839483fd103 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=quincy-bbdadfa, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-13T15:10:16.474 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:16 smithi184 podman[247033]: 2024-09-13 15:10:16.472474366 +0000 UTC m=+0.436767539 container attach e02696e4e2153906a812027fcb7c056b57fef07e89d768cf83795839483fd103 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=quincy-bbdadfa, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True) 2024-09-13T15:10:16.854 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:16 smithi184 podman[247033]: 2024-09-13 15:10:16.555479392 +0000 UTC m=+0.519772576 container died e02696e4e2153906a812027fcb7c056b57fef07e89d768cf83795839483fd103 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=quincy-bbdadfa) 2024-09-13T15:10:16.854 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:16 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:16.571+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:10:16.855 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:16 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:16.571+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.572819+0000 front 2024-09-13T15:09:30.572762+0000 (oldest deadline 2024-09-13T15:09:55.872702+0000) 2024-09-13T15:10:16.855 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:16 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:16.571+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-13T15:09:40.573748+0000 front 2024-09-13T15:09:40.573668+0000 (oldest deadline 2024-09-13T15:10:05.273646+0000) 2024-09-13T15:10:16.855 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:16 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:16.571+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-13T15:09:49.374582+0000 front 2024-09-13T15:09:49.374522+0000 (oldest deadline 2024-09-13T15:10:12.874435+0000) 2024-09-13T15:10:17.120 INFO:journalctl@ceph.osd.6.smithi184.stdout:Sep 13 15:10:17 smithi184 podman[247076]: 2024-09-13 15:10:17.117295217 +0000 UTC m=+0.552815299 container remove e02696e4e2153906a812027fcb7c056b57fef07e89d768cf83795839483fd103 (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-6-deactivate, RELEASE=quincy-bbdadfa, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T15:10:17.139 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.6.service' 2024-09-13T15:10:17.176 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:10:17.177 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-13T15:10:17.177 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-13T15:10:17.177 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.7 2024-09-13T15:10:17.507 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:17 smithi184 systemd[1]: Stopping Ceph osd.7 for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T15:10:17.507 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:17.504+0000 7fce63da3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:10:17.507 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:17.504+0000 7fce63da3640 -1 osd.7 1394 *** Got signal Terminated *** 2024-09-13T15:10:17.507 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:17.504+0000 7fce63da3640 -1 osd.7 1394 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:10:17.854 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:17.608+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:10:17.854 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:17.608+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.572819+0000 front 2024-09-13T15:09:30.572762+0000 (oldest deadline 2024-09-13T15:09:55.872702+0000) 2024-09-13T15:10:17.854 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:17.608+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-13T15:09:40.573748+0000 front 2024-09-13T15:09:40.573668+0000 (oldest deadline 2024-09-13T15:10:05.273646+0000) 2024-09-13T15:10:17.854 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:17 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:17.608+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-13T15:09:49.374582+0000 front 2024-09-13T15:09:49.374522+0000 (oldest deadline 2024-09-13T15:10:12.874435+0000) 2024-09-13T15:10:18.854 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:18 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:18.579+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:10:18.855 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:18 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:18.579+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.572819+0000 front 2024-09-13T15:09:30.572762+0000 (oldest deadline 2024-09-13T15:09:55.872702+0000) 2024-09-13T15:10:18.855 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:18 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:18.579+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-13T15:09:40.573748+0000 front 2024-09-13T15:09:40.573668+0000 (oldest deadline 2024-09-13T15:10:05.273646+0000) 2024-09-13T15:10:18.855 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:18 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:18.579+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-13T15:09:49.374582+0000 front 2024-09-13T15:09:49.374522+0000 (oldest deadline 2024-09-13T15:10:12.874435+0000) 2024-09-13T15:10:20.104 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:19 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:19.628+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:10:20.104 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:19 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:19.628+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.572819+0000 front 2024-09-13T15:09:30.572762+0000 (oldest deadline 2024-09-13T15:09:55.872702+0000) 2024-09-13T15:10:20.105 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:19 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:19.628+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-13T15:09:40.573748+0000 front 2024-09-13T15:09:40.573668+0000 (oldest deadline 2024-09-13T15:10:05.273646+0000) 2024-09-13T15:10:20.105 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:19 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:19.628+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-13T15:09:49.374582+0000 front 2024-09-13T15:09:49.374522+0000 (oldest deadline 2024-09-13T15:10:12.874435+0000) 2024-09-13T15:10:21.005 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:20.668+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:10:21.005 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:20.668+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.572819+0000 front 2024-09-13T15:09:30.572762+0000 (oldest deadline 2024-09-13T15:09:55.872702+0000) 2024-09-13T15:10:21.005 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:20.668+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-13T15:09:40.573748+0000 front 2024-09-13T15:09:40.573668+0000 (oldest deadline 2024-09-13T15:10:05.273646+0000) 2024-09-13T15:10:21.005 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:20 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:20.668+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-13T15:09:49.374582+0000 front 2024-09-13T15:09:49.374522+0000 (oldest deadline 2024-09-13T15:10:12.874435+0000) 2024-09-13T15:10:22.104 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:21 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:21.652+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-13T15:09:21.871117+0000 front 2024-09-13T15:09:21.871153+0000 (oldest deadline 2024-09-13T15:09:46.571162+0000) 2024-09-13T15:10:22.104 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:21 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:21.652+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-13T15:09:30.572819+0000 front 2024-09-13T15:09:30.572762+0000 (oldest deadline 2024-09-13T15:09:55.872702+0000) 2024-09-13T15:10:22.104 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:21 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:21.652+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-13T15:09:40.573748+0000 front 2024-09-13T15:09:40.573668+0000 (oldest deadline 2024-09-13T15:10:05.273646+0000) 2024-09-13T15:10:22.105 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:21 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7[57630]: 2024-09-13T15:10:21.652+0000 7fce603bb640 -1 osd.7 1394 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-13T15:09:49.374582+0000 front 2024-09-13T15:09:49.374522+0000 (oldest deadline 2024-09-13T15:10:12.874435+0000) 2024-09-13T15:10:23.464 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:23 smithi184 podman[247161]: 2024-09-13 15:10:23.155224348 +0000 UTC m=+5.754090724 container died f37cc947097b57db019877c725c4be6731702db18ef25d6f88a4189a9d0420fd (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=quincy-bbdadfa, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True) 2024-09-13T15:10:23.821 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:23 smithi184 podman[247161]: 2024-09-13 15:10:23.569737635 +0000 UTC m=+6.168604015 container cleanup f37cc947097b57db019877c725c4be6731702db18ef25d6f88a4189a9d0420fd (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7, org.label-schema.schema-version=1.0, RELEASE=quincy-bbdadfa, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.vendor=CentOS) 2024-09-13T15:10:23.822 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:23 smithi184 bash[247161]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7 2024-09-13T15:10:24.104 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:23 smithi184 podman[247195]: 2024-09-13 15:10:23.820811839 +0000 UTC m=+0.644255167 container remove f37cc947097b57db019877c725c4be6731702db18ef25d6f88a4189a9d0420fd (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7, org.label-schema.license=GPLv2, RELEASE=quincy-bbdadfa, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-13T15:10:24.416 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:24 smithi184 podman[247292]: 2024-09-13 15:10:24.028818562 +0000 UTC m=+0.015637363 image pull a994b00deb3a4a8eeb484a456d3f3207aae0401a31d251b199b2c9dbc1c4036c quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be 2024-09-13T15:10:24.417 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:24 smithi184 podman[247292]: 2024-09-13 15:10:24.130170458 +0000 UTC m=+0.116989252 container create bdbf2c14c6e881375941788278dee8526b74b1e0d349ebbd6720cd7b2f86d9ee (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=quincy-bbdadfa, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-13T15:10:24.417 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:24 smithi184 podman[247292]: 2024-09-13 15:10:24.332645412 +0000 UTC m=+0.319464211 container init bdbf2c14c6e881375941788278dee8526b74b1e0d349ebbd6720cd7b2f86d9ee (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7-deactivate, RELEASE=quincy-bbdadfa, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T15:10:24.417 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:24 smithi184 podman[247292]: 2024-09-13 15:10:24.33719919 +0000 UTC m=+0.324017985 container start bdbf2c14c6e881375941788278dee8526b74b1e0d349ebbd6720cd7b2f86d9ee (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=quincy-bbdadfa, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T15:10:24.417 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:24 smithi184 podman[247292]: 2024-09-13 15:10:24.415662435 +0000 UTC m=+0.402481227 container attach bdbf2c14c6e881375941788278dee8526b74b1e0d349ebbd6720cd7b2f86d9ee (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=quincy-bbdadfa, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T15:10:24.803 INFO:journalctl@ceph.osd.7.smithi184.stdout:Sep 13 15:10:24 smithi184 podman[247292]: 2024-09-13 15:10:24.511997814 +0000 UTC m=+0.498816618 container died bdbf2c14c6e881375941788278dee8526b74b1e0d349ebbd6720cd7b2f86d9ee (image=quay.ceph.io/ceph-ci/ceph@sha256:20745bd452f31fe48c67c092910f9b2c550f9dc62d239ff3e2953415a85590be, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, RELEASE=quincy-bbdadfa, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T15:10:25.106 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@osd.7.service' 2024-09-13T15:10:25.145 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:10:25.145 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-13T15:10:25.145 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-13T15:10:25.146 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-2401fc50-71da-11ef-bceb-c7b262605968@prometheus.a 2024-09-13T15:10:25.354 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:10:25 smithi184 systemd[1]: Stopping Ceph prometheus.a for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T15:10:25.771 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:10:25 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:10:25.465Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-13T15:10:25.771 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:10:25 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:10:25.465Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-13T15:10:25.772 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:10:25 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:10:25.465Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-13T15:10:25.772 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:10:25 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:10:25.465Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-13T15:10:25.772 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:10:25 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:10:25.465Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-13T15:10:25.772 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:10:25 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:10:25.465Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-13T15:10:25.772 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:10:25 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:10:25.465Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-13T15:10:25.772 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:10:25 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:10:25.465Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-13T15:10:25.772 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:10:25 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:10:25.465Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-13T15:10:25.773 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:10:25 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:10:25.467Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-13T15:10:25.773 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:10:25 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:10:25.467Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-13T15:10:25.773 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:10:25 smithi184 ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a[81491]: ts=2024-09-13T15:10:25.467Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-13T15:10:25.773 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:10:25 smithi184 podman[247419]: 2024-09-13 15:10:25.475907248 +0000 UTC m=+0.115769384 container died 3da9265b8f14fc104ad3d01a00c82acbe684385d0380509cde4281b20a65563a (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T15:10:26.076 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:10:25 smithi184 podman[247419]: 2024-09-13 15:10:25.888640861 +0000 UTC m=+0.528502996 container cleanup 3da9265b8f14fc104ad3d01a00c82acbe684385d0380509cde4281b20a65563a (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T15:10:26.077 INFO:journalctl@ceph.prometheus.a.smithi184.stdout:Sep 13 15:10:25 smithi184 bash[247419]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-prometheus-a 2024-09-13T15:10:26.204 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2401fc50-71da-11ef-bceb-c7b262605968@prometheus.a.service' 2024-09-13T15:10:26.237 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:10:26.238 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-13T15:10:26.238 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2401fc50-71da-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-13T15:10:28.038 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 15:10:27 smithi151 systemd[1]: Stopping Ceph alertmanager.a for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T15:10:28.354 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 15:10:28 smithi151 ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a[93024]: ts=2024-09-13T15:10:28.037Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-13T15:10:28.354 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 15:10:28 smithi151 podman[363426]: 2024-09-13 15:10:28.078069137 +0000 UTC m=+0.144584499 container died ed70c3d751072ecc08fe78b35eb20e6407c7ae03a131f6a3030aaef8d0b11d2d (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T15:10:28.609 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 15:10:28 smithi151 podman[363426]: 2024-09-13 15:10:28.544345636 +0000 UTC m=+0.610860998 container cleanup ed70c3d751072ecc08fe78b35eb20e6407c7ae03a131f6a3030aaef8d0b11d2d (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T15:10:28.609 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 15:10:28 smithi151 bash[363426]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a 2024-09-13T15:10:29.109 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 15:10:28 smithi151 podman[363444]: 2024-09-13 15:10:28.854235831 +0000 UTC m=+0.772700572 container remove ed70c3d751072ecc08fe78b35eb20e6407c7ae03a131f6a3030aaef8d0b11d2d (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T15:10:29.109 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 15:10:28 smithi151 podman[363444]: 2024-09-13 15:10:28.946248958 +0000 UTC m=+0.864713701 volume remove c6414bb6d522b88697c95b79ed90224b825961ea410115f794924cf09daade91 2024-09-13T15:10:29.109 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 15:10:29 smithi151 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-13T15:10:29.109 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 15:10:29 smithi151 systemd[1]: Stopped Ceph alertmanager.a for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T15:10:29.109 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Sep 13 15:10:29 smithi151 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@alertmanager.a.service: Consumed 4.122s CPU time. 2024-09-13T15:10:29.609 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 15:10:29 smithi151 systemd[1]: Stopping Ceph node-exporter.a for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T15:10:29.609 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 15:10:29 smithi151 podman[363558]: 2024-09-13 15:10:29.487380935 +0000 UTC m=+0.110449204 container died d91e4cb09e9abc98c127ea34498db567a9a6dbd00d90a8be381e26fffd4beeba (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T15:10:30.109 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 15:10:29 smithi151 podman[363558]: 2024-09-13 15:10:29.908860847 +0000 UTC m=+0.531929113 container cleanup d91e4cb09e9abc98c127ea34498db567a9a6dbd00d90a8be381e26fffd4beeba (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T15:10:30.109 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 15:10:29 smithi151 bash[363558]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a 2024-09-13T15:10:30.391 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 15:10:30 smithi151 podman[363575]: 2024-09-13 15:10:30.127150295 +0000 UTC m=+0.643816978 container remove d91e4cb09e9abc98c127ea34498db567a9a6dbd00d90a8be381e26fffd4beeba (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T15:10:30.392 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 15:10:30 smithi151 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-13T15:10:30.392 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 15:10:30 smithi151 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-13T15:10:30.392 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 15:10:30 smithi151 systemd[1]: Stopped Ceph node-exporter.a for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T15:10:30.393 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Sep 13 15:10:30 smithi151 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@node-exporter.a.service: Consumed 8.448s CPU time. 2024-09-13T15:10:38.502 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2401fc50-71da-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-13T15:10:40.281 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 15:10:40 smithi184 systemd[1]: Stopping Ceph grafana.a for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T15:10:40.548 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 15:10:40 smithi184 podman[247837]: 2024-09-13 15:10:40.279627765 +0000 UTC m=+0.112494217 container died 4ba18fb1f1fc82cd0aa5b0d7bd519a9b28340496730b2a332a9c65a8349e2af8 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.expose-services=, release=239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-04-29T11:48:11.055897, architecture=x86_64, version=8.5, vcs-type=git, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, distribution-scope=public, maintainer=Paul Cuzner , description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com) 2024-09-13T15:10:40.854 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 15:10:40 smithi184 podman[247837]: 2024-09-13 15:10:40.633371634 +0000 UTC m=+0.466238123 container cleanup 4ba18fb1f1fc82cd0aa5b0d7bd519a9b28340496730b2a332a9c65a8349e2af8 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a, architecture=x86_64, description=Ceph Grafana Container, vendor=Red Hat, Inc., version=8.5, release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, build-date=2022-04-29T11:48:11.055897, vcs-type=git, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.expose-services=, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com) 2024-09-13T15:10:40.854 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 15:10:40 smithi184 bash[247837]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a 2024-09-13T15:10:41.249 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 15:10:40 smithi184 podman[247854]: 2024-09-13 15:10:40.951036867 +0000 UTC m=+0.674022822 container remove 4ba18fb1f1fc82cd0aa5b0d7bd519a9b28340496730b2a332a9c65a8349e2af8 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-grafana-a, version=8.5, io.openshift.expose-services=, release=239.1651231664, build-date=2022-04-29T11:48:11.055897, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, name=ubi8, vcs-type=git, architecture=x86_64, io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.32.0, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , vendor=Red Hat, Inc., distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container) 2024-09-13T15:10:41.249 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 15:10:40 smithi184 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-09-13T15:10:41.249 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 15:10:41 smithi184 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-09-13T15:10:41.249 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 15:10:41 smithi184 systemd[1]: Stopped Ceph grafana.a for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T15:10:41.249 INFO:journalctl@ceph.grafana.a.smithi184.stdout:Sep 13 15:10:41 smithi184 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@grafana.a.service: Consumed 30.471s CPU time. 2024-09-13T15:10:41.509 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 15:10:41 smithi184 systemd[1]: Stopping Ceph node-exporter.b for 2401fc50-71da-11ef-bceb-c7b262605968... 2024-09-13T15:10:41.773 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 15:10:41 smithi184 podman[247968]: 2024-09-13 15:10:41.508382504 +0000 UTC m=+0.124119943 container died 1e1a9dd643a0fb3d8c295b4aa417b9292f72e15efd8cc3c20df9f5514ac076b4 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T15:10:42.104 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 15:10:41 smithi184 podman[247968]: 2024-09-13 15:10:41.870573068 +0000 UTC m=+0.486310498 container cleanup 1e1a9dd643a0fb3d8c295b4aa417b9292f72e15efd8cc3c20df9f5514ac076b4 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T15:10:42.104 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 15:10:41 smithi184 bash[247968]: ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b 2024-09-13T15:10:42.555 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 15:10:42 smithi184 podman[247986]: 2024-09-13 15:10:42.146692477 +0000 UTC m=+0.642058457 container remove 1e1a9dd643a0fb3d8c295b4aa417b9292f72e15efd8cc3c20df9f5514ac076b4 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2401fc50-71da-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T15:10:42.555 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 15:10:42 smithi184 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-13T15:10:42.555 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 15:10:42 smithi184 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-13T15:10:42.555 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 15:10:42 smithi184 systemd[1]: Stopped Ceph node-exporter.b for 2401fc50-71da-11ef-bceb-c7b262605968. 2024-09-13T15:10:42.555 INFO:journalctl@ceph.node-exporter.b.smithi184.stdout:Sep 13 15:10:42 smithi184 systemd[1]: ceph-2401fc50-71da-11ef-bceb-c7b262605968@node-exporter.b.service: Consumed 7.957s CPU time. 2024-09-13T15:10:48.435 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-13T15:10:48.469 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-13T15:10:48.497 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-13T15:10:48.499 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903555/remote/smithi151/crash 2024-09-13T15:10:48.500 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/crash -- . 2024-09-13T15:10:48.554 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-13T15:10:48.554 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2024-09-13T15:10:48.556 DEBUG:teuthology.misc:Transferring archived files from smithi184:/var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903555/remote/smithi184/crash 2024-09-13T15:10:48.557 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/crash -- . 2024-09-13T15:10:48.597 INFO:teuthology.orchestra.run.smithi184.stderr:tar: /var/lib/ceph/2401fc50-71da-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-13T15:10:48.597 INFO:teuthology.orchestra.run.smithi184.stderr:tar: Error is not recoverable: exiting now 2024-09-13T15:10:48.598 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-13T15:10:48.598 DEBUG:teuthology.orchestra.run.smithi151:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v OSD_DOWN | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v MON_DOWN | egrep -v 'out of quorum' | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v OSDMAP_FLAGS | egrep -v PG_AVAILABILITY | head -n 1 2024-09-13T15:10:48.726 INFO:teuthology.orchestra.run.smithi151.stdout:1726237200.0000975 mon.a (mon.0) 772 : cluster [ERR] Health detail: HEALTH_ERR 1 filesystem is offline; 1 filesystem is online with fewer MDS than max_mds 2024-09-13T15:10:48.727 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-09-13T15:10:48.727 INFO:tasks.cephadm:Compressing logs... 2024-09-13T15:10:48.727 DEBUG:teuthology.orchestra.run.smithi151:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T15:10:48.729 DEBUG:teuthology.orchestra.run.smithi184:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T15:10:48.763 INFO:teuthology.orchestra.run.smithi151.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-13T15:10:48.771 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-osd.0.log 2024-09-13T15:10:48.776 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-osd.1.log 2024-09-13T15:10:48.776 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-13T15:10:48.776 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-13T15:10:48.776 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-mon.c.log 2024-09-13T15:10:48.776 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-13T15:10:48.776 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-mon.a.log 2024-09-13T15:10:48.777 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-osd.2.log 2024-09-13T15:10:48.777 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-mgr.y.log 2024-09-13T15:10:48.777 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph.log 2024-09-13T15:10:48.777 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-mds.foo.smithi151.rqqklw.log 2024-09-13T15:10:48.777 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-osd.3.log 2024-09-13T15:10:48.777 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-mds.foo.smithi151.rqqklw.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-13T15:10:48.777 INFO:teuthology.orchestra.run.smithi151.stderr: 82.3% -- replaced with /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-mds.foo.smithi151.rqqklw.log.gz 2024-09-13T15:10:48.788 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-osd.3.log: /var/log/ceph/cephadm.log: 92.3% -- replaced with /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-13T15:10:48.791 INFO:teuthology.orchestra.run.smithi184.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-mds.foo.smithi184.jismsi.log 2024-09-13T15:10:48.791 INFO:teuthology.orchestra.run.smithi184.stderr:: No such file or directory 2024-09-13T15:10:48.791 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-osd.7.log 2024-09-13T15:10:48.792 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-mds.foo.smithi184.jismsi.log: gzip -5 --verbose -- /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-13T15:10:48.792 INFO:teuthology.orchestra.run.smithi184.stderr: 62.4% -- replaced with /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-mds.foo.smithi184.jismsi.log.gz 2024-09-13T15:10:48.792 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-osd.4.log 2024-09-13T15:10:48.793 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-13T15:10:48.793 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-13T15:10:48.793 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-osd.6.log 2024-09-13T15:10:48.793 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-osd.5.log 2024-09-13T15:10:48.793 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph.log 2024-09-13T15:10:48.795 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-mgr.x.log 2024-09-13T15:10:48.795 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-osd.5.log: /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-mon.b.log 2024-09-13T15:10:48.795 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-13T15:10:48.923 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-mon.b.log: /var/log/ceph/cephadm.log: 84.2% -- replaced with /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-13T15:10:48.942 INFO:teuthology.orchestra.run.smithi151.stderr: 91.2% -- replaced with /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-13T15:10:48.999 INFO:teuthology.orchestra.run.smithi184.stderr: 90.0% -- replaced with /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-13T15:10:49.153 INFO:teuthology.orchestra.run.smithi151.stderr: 95.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-13T15:10:49.164 INFO:teuthology.orchestra.run.smithi184.stderr: 91.3% -- replaced with /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-13T15:10:49.214 INFO:teuthology.orchestra.run.smithi184.stderr: 86.3% -- replaced with /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-13T15:10:49.288 INFO:teuthology.orchestra.run.smithi184.stderr: 92.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-13T15:10:49.350 INFO:teuthology.orchestra.run.smithi151.stderr: 94.0% -- replaced with /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-13T15:10:49.489 INFO:teuthology.orchestra.run.smithi151.stderr: 91.0% -- replaced with /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-13T15:10:50.005 INFO:teuthology.orchestra.run.smithi184.stderr: 90.2% -- replaced with /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-mgr.x.log.gz 2024-09-13T15:11:01.326 INFO:teuthology.orchestra.run.smithi151.stderr: 89.3% -- replaced with /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-mgr.y.log.gz 2024-09-13T15:11:02.694 INFO:teuthology.orchestra.run.smithi184.stderr: 92.2% -- replaced with /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-mon.b.log.gz 2024-09-13T15:11:13.114 INFO:teuthology.orchestra.run.smithi151.stderr: 91.3% -- replaced with /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-mon.c.log.gz 2024-09-13T15:11:31.288 INFO:teuthology.orchestra.run.smithi151.stderr: 90.2% -- replaced with /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-mon.a.log.gz 2024-09-13T15:17:02.568 INFO:teuthology.orchestra.run.smithi184.stderr: 93.2% -- replaced with /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-osd.7.log.gz 2024-09-13T15:17:38.096 INFO:teuthology.orchestra.run.smithi151.stderr: 92.8% -- replaced with /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-osd.2.log.gz 2024-09-13T15:19:54.156 INFO:teuthology.orchestra.run.smithi184.stderr: 93.5% -- replaced with /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-osd.4.log.gz 2024-09-13T15:21:37.398 INFO:teuthology.orchestra.run.smithi184.stderr: 92.8% -- replaced with /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-osd.6.log.gz 2024-09-13T15:22:06.115 INFO:teuthology.orchestra.run.smithi184.stderr: 93.2% -- replaced with /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-osd.5.log.gz 2024-09-13T15:22:06.117 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-09-13T15:22:06.117 INFO:teuthology.orchestra.run.smithi184.stderr:real 11m17.373s 2024-09-13T15:22:06.118 INFO:teuthology.orchestra.run.smithi184.stderr:user 6m8.084s 2024-09-13T15:22:06.118 INFO:teuthology.orchestra.run.smithi184.stderr:sys 0m18.346s 2024-09-13T15:30:13.556 INFO:teuthology.orchestra.run.smithi151.stderr: 93.4% -- replaced with /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-osd.0.log.gz 2024-09-13T15:35:57.335 INFO:teuthology.orchestra.run.smithi151.stderr: 93.0% -- replaced with /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-osd.1.log.gz 2024-09-13T15:36:29.068 INFO:teuthology.orchestra.run.smithi151.stderr: 92.4% -- replaced with /var/log/ceph/2401fc50-71da-11ef-bceb-c7b262605968/ceph-osd.3.log.gz 2024-09-13T15:36:29.070 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-09-13T15:36:29.070 INFO:teuthology.orchestra.run.smithi151.stderr:real 25m40.327s 2024-09-13T15:36:29.070 INFO:teuthology.orchestra.run.smithi151.stderr:user 9m31.087s 2024-09-13T15:36:29.070 INFO:teuthology.orchestra.run.smithi151.stderr:sys 0m28.750s 2024-09-13T15:36:29.071 INFO:tasks.cephadm:Archiving logs... 2024-09-13T15:36:29.071 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/log/ceph to /home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903555/remote/smithi151/log 2024-09-13T15:36:29.072 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T15:37:59.964 DEBUG:teuthology.misc:Transferring archived files from smithi184:/var/log/ceph to /home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903555/remote/smithi184/log 2024-09-13T15:37:59.966 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T15:38:56.547 INFO:tasks.cephadm:Removing cluster... 2024-09-13T15:38:56.548 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2401fc50-71da-11ef-bceb-c7b262605968 --force 2024-09-13T15:38:58.054 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2401fc50-71da-11ef-bceb-c7b262605968 --force 2024-09-13T15:38:59.000 INFO:tasks.cephadm:Removing cephadm ... 2024-09-13T15:38:59.000 DEBUG:teuthology.orchestra.run.smithi151:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-13T15:38:59.030 DEBUG:teuthology.orchestra.run.smithi184:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-13T15:38:59.046 INFO:tasks.cephadm:Teardown complete 2024-09-13T15:38:59.047 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-13T15:38:59.057 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-13T15:38:59.057 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T15:38:59.072 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T15:38:59.209 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-13T15:38:59.221 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-13T15:38:59.222 DEBUG:teuthology.orchestra.run.smithi151:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T15:38:59.225 DEBUG:teuthology.orchestra.run.smithi184:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T15:38:59.239 INFO:teuthology.orchestra.run.smithi151.stderr:bash: line 1: ntpq: command not found 2024-09-13T15:38:59.239 INFO:teuthology.orchestra.run.smithi184.stderr:bash: line 1: ntpq: command not found 2024-09-13T15:38:59.257 INFO:teuthology.orchestra.run.smithi184.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T15:38:59.257 INFO:teuthology.orchestra.run.smithi184.stdout:=============================================================================== 2024-09-13T15:38:59.257 INFO:teuthology.orchestra.run.smithi184.stdout:^* hv01.front.sepia.ceph.com 3 6 377 43 +299us[ +302us] +/- 40ms 2024-09-13T15:38:59.258 INFO:teuthology.orchestra.run.smithi184.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 46 -791us[ -788us] +/- 56ms 2024-09-13T15:38:59.258 INFO:teuthology.orchestra.run.smithi184.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 42 +423us[ +423us] +/- 67ms 2024-09-13T15:38:59.258 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T15:38:59.354 INFO:teuthology.orchestra.run.smithi151.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T15:38:59.354 INFO:teuthology.orchestra.run.smithi151.stdout:=============================================================================== 2024-09-13T15:38:59.354 INFO:teuthology.orchestra.run.smithi151.stdout:^* hv01.front.sepia.ceph.com 3 6 377 42 +303us[ +303us] +/- 40ms 2024-09-13T15:38:59.354 INFO:teuthology.orchestra.run.smithi151.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 42 -805us[ -805us] +/- 56ms 2024-09-13T15:38:59.355 INFO:teuthology.orchestra.run.smithi151.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 41 +430us[ +430us] +/- 67ms 2024-09-13T15:38:59.355 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T15:38:59.364 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-13T15:38:59.374 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-13T15:38:59.375 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-13T15:38:59.384 DEBUG:teuthology.orchestra.run.smithi151:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T15:38:59.518 DEBUG:teuthology.orchestra.run.smithi184:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T15:38:59.618 DEBUG:teuthology.orchestra.run.smithi151:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-13T15:38:59.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:38:59.649 DEBUG:teuthology.orchestra.run.smithi184:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-13T15:38:59.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:38:59.678 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-13T15:38:59.688 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-13T15:38:59.697 INFO:teuthology.task.internal:Duration was 5583.541607 seconds 2024-09-13T15:38:59.697 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-13T15:38:59.707 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-13T15:38:59.707 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T15:38:59.710 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T15:38:59.768 INFO:teuthology.orchestra.run.smithi184.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T15:38:59.772 INFO:teuthology.orchestra.run.smithi151.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T15:39:00.187 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-13T15:39:00.188 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi151.front.sepia.ceph.com 2024-09-13T15:39:00.188 DEBUG:teuthology.orchestra.run.smithi151:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T15:39:00.211 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi184.front.sepia.ceph.com 2024-09-13T15:39:00.212 DEBUG:teuthology.orchestra.run.smithi184:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T15:39:00.239 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-13T15:39:00.240 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T15:39:00.254 DEBUG:teuthology.orchestra.run.smithi184:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T15:39:00.307 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-13T15:39:00.307 DEBUG:teuthology.orchestra.run.smithi151:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T15:39:00.325 DEBUG:teuthology.orchestra.run.smithi184:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T15:39:01.861 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-13T15:39:01.886 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-13T15:39:01.886 DEBUG:teuthology.orchestra.run.smithi151:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T15:39:01.915 DEBUG:teuthology.orchestra.run.smithi184:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T15:39:01.956 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-13T15:39:02.039 DEBUG:teuthology.orchestra.run.smithi151:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:39:02.042 DEBUG:teuthology.orchestra.run.smithi184:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:39:02.075 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = core 2024-09-13T15:39:02.101 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern = core 2024-09-13T15:39:02.119 DEBUG:teuthology.orchestra.run.smithi151:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:39:02.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:39:02.144 DEBUG:teuthology.orchestra.run.smithi184:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:39:02.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:39:02.160 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-13T15:39:02.173 INFO:teuthology.task.internal:Transferring archived files... 2024-09-13T15:39:02.174 DEBUG:teuthology.misc:Transferring archived files from smithi151:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903555/remote/smithi151 2024-09-13T15:39:02.174 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T15:39:02.261 DEBUG:teuthology.misc:Transferring archived files from smithi184:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903555/remote/smithi184 2024-09-13T15:39:02.261 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T15:39:02.308 INFO:teuthology.task.internal:Removing archive directory... 2024-09-13T15:39:02.308 DEBUG:teuthology.orchestra.run.smithi151:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T15:39:02.311 DEBUG:teuthology.orchestra.run.smithi184:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T15:39:02.355 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-13T15:39:02.377 INFO:teuthology.task.internal:Not uploading archives. 2024-09-13T15:39:02.377 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-13T15:39:02.388 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-13T15:39:02.388 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T15:39:02.391 DEBUG:teuthology.orchestra.run.smithi184:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T15:39:02.410 INFO:teuthology.orchestra.run.smithi184.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 15:39 /home/ubuntu/cephtest 2024-09-13T15:39:02.417 INFO:teuthology.orchestra.run.smithi151.stdout: 262169 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 13 15:39 /home/ubuntu/cephtest 2024-09-13T15:39:02.417 INFO:teuthology.orchestra.run.smithi151.stdout: 5111814 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 13 15:06 /home/ubuntu/cephtest/mnt.0 2024-09-13T15:39:02.417 INFO:teuthology.orchestra.run.smithi151.stdout: 5111821 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 13 15:06 /home/ubuntu/cephtest/mnt.0/client.0 2024-09-13T15:39:02.418 INFO:teuthology.orchestra.run.smithi151.stdout: 5111852 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 15:06 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-13T15:39:02.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:39:02.419 INFO:teuthology.orchestra.run.smithi151.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-09-13T15:39:02.419 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi151 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-13T15:39:02.425 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-13T15:39:02.436 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-13T15:39:02.446 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi151 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-13T15:39:02.450 INFO:teuthology.run:Summary data: description: upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} duration: 5583.541606903076 failure_reason: 'Command failed (workunit test cls/test_cls_rbd.sh) on smithi151 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh''' flavor: default owner: scheduled_akupczyk@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=ac70f570e58942088455127a5ce00c3d status: fail success: false 2024-09-13T15:39:02.451 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T15:39:02.568 INFO:teuthology.run:FAIL